2024-02-13T00:14:53.809 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-13T00:14:53.810 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} --name lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi --owner scheduled_lflores@teuthology --verbose --archive /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557148 -- /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557148/orig.config.yaml 2024-02-13T00:14:53.861 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-13T00:14:53.951 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557148 branch: wip-yuri10-testing-2024-02-08-0854-pacific description: rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} email: null first_in_suite: false job_id: '7557148' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-02-08-0854-pacific ceph: conf: client: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mgrc: 20 debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon warn on pool no app: false paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 crush_tunables: jewel flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(MON_DOWN\) - \(OSDMAP_FLAGS\) - flag\(s\) set - \(CACHE_POOL_NO_HIT_SET\) - \(PG_ - \(OSD_ - mons down: null - reached quota - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-tracker-64343-pacific sha1: 84e714279aa684636ac645daf0a1a85c2094cfb3 owner: scheduled_lflores@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 964 sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-tracker-64343-pacific suite_path: /home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 84e714279aa684636ac645daf0a1a85c2094cfb3 targets: smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEXH5Amm+A+g8MvSb9PRCKMns56HmLlUD/QFloAsBJ1Hzoj4z30vb+z498JHtCUNexF8/DZeVnn+d+sgeu0iHpo= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPk7kbpnJ5UoTYejzsFP9Z4RpDIbvfOfvJ+d3SOBX5z+BTz8zZrsDvSL/iAsFZ1zjIzk8ddi5qyqh+Soevp8eo0= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-12_21:56:27 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-13T00:14:53.951 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa; will attempt to use it 2024-02-13T00:14:53.952 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks 2024-02-13T00:14:53.952 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-13T00:14:53.953 INFO:teuthology.task.internal:Checking packages... 2024-02-13T00:14:53.978 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0e714d9a4bd2a821113e6318adb87bd06cf81ec1' 2024-02-13T00:14:53.978 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-13T00:14:53.978 INFO:teuthology.packaging:ref: None 2024-02-13T00:14:53.978 INFO:teuthology.packaging:tag: None 2024-02-13T00:14:53.978 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-13T00:14:53.978 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T00:14:53.979 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-13T00:14:54.205 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-521.g0e714d9a 2024-02-13T00:14:54.207 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-13T00:14:54.214 INFO:teuthology.task.internal:no buildpackages task found 2024-02-13T00:14:54.214 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-13T00:14:54.220 INFO:teuthology.task.internal:Saving configuration 2024-02-13T00:14:54.234 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-13T00:14:54.238 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T00:14:54.263 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 00:09:36.879680', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEXH5Amm+A+g8MvSb9PRCKMns56HmLlUD/QFloAsBJ1Hzoj4z30vb+z498JHtCUNexF8/DZeVnn+d+sgeu0iHpo='} 2024-02-13T00:14:54.282 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 00:09:36.881062', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPk7kbpnJ5UoTYejzsFP9Z4RpDIbvfOfvJ+d3SOBX5z+BTz8zZrsDvSL/iAsFZ1zjIzk8ddi5qyqh+Soevp8eo0='} 2024-02-13T00:14:54.282 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-13T00:14:54.288 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-13T00:14:54.288 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-02-13T00:14:54.288 INFO:teuthology.run_tasks:Running task console_log... 2024-02-13T00:14:54.357 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f931c56ef70>, signals=[15]) 2024-02-13T00:14:54.358 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-13T00:14:54.368 INFO:teuthology.task.internal:Opening connections... 2024-02-13T00:14:54.368 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-02-13T00:14:54.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T00:14:54.449 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-02-13T00:14:54.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T00:14:54.521 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-13T00:14:54.528 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-02-13T00:14:54.551 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-02-13T00:14:54.551 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-02-13T00:14:54.612 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-02-13T00:14:54.612 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="8" 2024-02-13T00:14:54.613 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-02-13T00:14:54.613 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-02-13T00:14:54.613 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="8" 2024-02-13T00:14:54.613 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el8" 2024-02-13T00:14:54.613 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-13T00:14:54.613 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-02-13T00:14:54.613 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-13T00:14:54.613 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-02-13T00:14:54.613 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-13T00:14:54.613 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-13T00:14:54.613 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-13T00:14:54.614 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-02-13T00:14:54.637 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-02-13T00:14:54.659 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-02-13T00:14:54.660 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-02-13T00:14:54.720 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-02-13T00:14:54.720 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="8" 2024-02-13T00:14:54.720 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-02-13T00:14:54.720 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-02-13T00:14:54.720 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="8" 2024-02-13T00:14:54.720 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el8" 2024-02-13T00:14:54.721 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-13T00:14:54.721 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-02-13T00:14:54.723 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-13T00:14:54.723 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-02-13T00:14:54.723 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-13T00:14:54.723 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-13T00:14:54.724 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-13T00:14:54.724 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-02-13T00:14:54.745 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-13T00:14:54.753 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-13T00:14:54.758 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-13T00:14:54.758 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-02-13T00:14:54.761 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-02-13T00:14:54.781 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-13T00:14:54.786 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-13T00:14:54.786 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-02-13T00:14:54.822 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-02-13T00:14:54.851 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-13T00:14:54.914 INFO:teuthology.run_tasks:Running task kernel... 2024-02-13T00:14:54.931 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-13T00:14:54.931 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-13T00:14:54.932 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-13T00:14:54.932 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-13T00:14:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T00:14:54.953 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-02-13T00:14:54.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T00:14:54.954 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-02-13T00:14:55.015 INFO:teuthology.orchestra.run.smithi162.stdout:4.18.0-489.el8.x86_64 2024-02-13T00:14:55.015 INFO:teuthology.task.kernel:Running kernel on smithi162: 4.18.0-489.el8.x86_64 2024-02-13T00:14:55.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-02-13T00:14:55.016 INFO:teuthology.orchestra.run.smithi181.stdout:4.18.0-489.el8.x86_64 2024-02-13T00:14:55.016 INFO:teuthology.task.kernel:Running kernel on smithi181: 4.18.0-489.el8.x86_64 2024-02-13T00:14:55.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-02-13T00:14:59.891 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 8 - AppStream 12 MB/s | 27 MB 00:02 2024-02-13T00:15:00.243 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - AppStream 9.9 MB/s | 27 MB 00:02 2024-02-13T00:15:06.383 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 8 - BaseOS 5.9 MB/s | 10 MB 00:01 2024-02-13T00:15:07.526 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - BaseOS 4.1 MB/s | 10 MB 00:02 2024-02-13T00:15:09.549 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 8 - Extras 11 kB/s | 18 kB 00:01 2024-02-13T00:15:09.685 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-02-13T00:15:10.369 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2024-02-13T00:15:10.483 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 8 - Extras common packages 8.5 kB/s | 6.9 kB 00:00 2024-02-13T00:15:10.604 INFO:teuthology.orchestra.run.smithi162.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-02-13T00:15:10.718 INFO:teuthology.orchestra.run.smithi181.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-02-13T00:15:12.637 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 16 MB 00:01 2024-02-13T00:15:16.402 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 270 kB/s | 24 kB 00:00 2024-02-13T00:15:19.349 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-13T00:15:19.349 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-13T00:15:19.470 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:15:19.471 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:15:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-02-13T00:15:19.472 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:15:19.472 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:15:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-13T00:15:19.472 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-13T00:15:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-13T00:15:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-13T00:15:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:15:19.473 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:15:19.473 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:15:19.473 INFO:teuthology.orchestra.run.smithi181.stdout:Install 3 Packages 2024-02-13T00:15:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:15:19.473 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 90 M 2024-02-13T00:15:19.474 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 96 M 2024-02-13T00:15:19.474 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:15:20.762 INFO:teuthology.orchestra.run.smithi181.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 8.3 MB/s | 10 MB 00:01 2024-02-13T00:15:20.772 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 1.6 MB/s | 16 MB 00:10 2024-02-13T00:15:22.028 INFO:teuthology.orchestra.run.smithi181.stdout:(2/3): kernel-core-4.18.0-536.el8.x86_64.rpm 17 MB/s | 44 MB 00:02 2024-02-13T00:15:22.454 INFO:teuthology.orchestra.run.smithi181.stdout:(3/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 12 MB/s | 36 MB 00:02 2024-02-13T00:15:22.454 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:15:22.454 INFO:teuthology.orchestra.run.smithi181.stdout:Total 30 MB/s | 90 MB 00:02 2024-02-13T00:15:23.136 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:15:23.567 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:15:23.567 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:15:24.560 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 569 kB/s | 24 kB 00:00 2024-02-13T00:15:24.801 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:15:24.831 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:15:27.482 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-13T00:15:27.483 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-13T00:15:27.595 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:15:27.596 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:15:27.596 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-02-13T00:15:27.596 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:15:27.596 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:15:27.596 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-13T00:15:27.596 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-13T00:15:27.596 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-13T00:15:27.596 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-13T00:15:27.596 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:15:27.596 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:15:27.596 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:15:27.596 INFO:teuthology.orchestra.run.smithi162.stdout:Install 3 Packages 2024-02-13T00:15:27.597 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:15:27.597 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 90 M 2024-02-13T00:15:27.597 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 96 M 2024-02-13T00:15:27.597 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:15:27.750 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:15:28.370 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-13T00:15:28.636 INFO:teuthology.orchestra.run.smithi162.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 16 MB/s | 10 MB 00:00 2024-02-13T00:15:29.862 INFO:teuthology.orchestra.run.smithi162.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 19 MB/s | 36 MB 00:01 2024-02-13T00:15:30.237 INFO:teuthology.orchestra.run.smithi162.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 19 MB/s | 44 MB 00:02 2024-02-13T00:15:30.237 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:15:30.237 INFO:teuthology.orchestra.run.smithi162.stdout:Total 34 MB/s | 90 MB 00:02 2024-02-13T00:15:30.918 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:15:31.349 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:15:31.350 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:15:32.035 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-13T00:15:32.428 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-13T00:15:32.594 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:15:32.612 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:15:35.386 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:15:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-13T00:15:35.856 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-13T00:15:36.133 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-13T00:15:39.417 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-13T00:15:39.892 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-13T00:15:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-13T00:15:43.608 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-13T00:15:57.923 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-13T00:15:57.947 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-13T00:15:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-13T00:15:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-13T00:15:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-13T00:15:58.792 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-13T00:15:58.792 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:15:58.792 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:15:58.792 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-13T00:15:58.792 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-13T00:15:58.792 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:15:58.792 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:15:58.971 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T00:15:59.191 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-13T00:15:59.191 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-13T00:15:59.192 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-13T00:15:59.192 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-13T00:15:59.192 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-13T00:15:59.192 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-13T00:15:59.192 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-02-13T00:15:59.192 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T00:15:59.402 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-13T00:15:59.402 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-13T00:15:59.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-02-13T00:15:59.436 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-02-13T00:15:59.436 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.02 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 129.el8 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Group : System Environment/Base 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 9265586 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Relocations : (not relocatable) 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : CentOS BuildSystem 2024-02-13T00:15:59.437 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-02-13T00:15:59.438 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-13T00:15:59.438 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-02-13T00:15:59.438 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-02-13T00:15:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:15:59.438 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-13T00:15:59.438 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-13T00:15:59.438 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-13T00:15:59.438 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-02-13T00:15:59.438 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:15:59.438 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-02-13T00:15:59.439 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-13T00:15:59.439 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 4.18.0-536.el8.x86_64 2024-02-13T00:15:59.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-13T00:16:01.258 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-02-13T00:16:03.514 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-02-13T00:16:03.515 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-02-13T00:16:03.538 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.lOni3z5TWW 2024-02-13T00:16:03.539 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lOni3z5TWW 2024-02-13T00:16:03.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.lOni3z5TWW 2024-02-13T00:16:03.767 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.lOni3z5TWW is 6KB 2024-02-13T00:16:03.815 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.lOni3z5TWW 2024-02-13T00:16:03.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-02-13T00:16:03.908 INFO:teuthology.orchestra.run.smithi181.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-13T00:16:03.908 INFO:teuthology.orchestra.run.smithi181.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-13T00:16:03.908 INFO:teuthology.orchestra.run.smithi181.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-13T00:16:03.908 INFO:teuthology.orchestra.run.smithi181.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-13T00:16:03.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-13T00:16:04.071 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-02-13T00:16:05.789 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-13T00:16:05.813 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-13T00:16:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-13T00:16:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-13T00:16:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-13T00:16:06.922 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-13T00:16:06.922 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:16:06.922 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:16:06.922 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-13T00:16:06.922 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-13T00:16:06.923 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:16:06.923 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:16:07.063 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T00:16:07.282 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-13T00:16:07.282 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-13T00:16:07.282 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-13T00:16:07.282 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-13T00:16:07.282 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-13T00:16:07.282 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-13T00:16:07.282 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2024-02-13T00:16:07.283 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T00:16:07.493 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-13T00:16:07.493 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-13T00:16:07.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -qi grub2-tools 2024-02-13T00:16:07.527 INFO:teuthology.orchestra.run.smithi162.stdout:Name : grub2-tools 2024-02-13T00:16:07.527 INFO:teuthology.orchestra.run.smithi162.stdout:Epoch : 1 2024-02-13T00:16:07.527 INFO:teuthology.orchestra.run.smithi162.stdout:Version : 2.02 2024-02-13T00:16:07.527 INFO:teuthology.orchestra.run.smithi162.stdout:Release : 129.el8 2024-02-13T00:16:07.527 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: x86_64 2024-02-13T00:16:07.527 INFO:teuthology.orchestra.run.smithi162.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:Group : System Environment/Base 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:Size : 9265586 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:License : GPLv3+ 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:Relocations : (not relocatable) 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:Packager : CentOS BuildSystem 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:Vendor : CentOS 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:Summary : Support tools for GRUB. 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout:Description : 2024-02-13T00:16:07.528 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:16:07.529 INFO:teuthology.orchestra.run.smithi162.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-13T00:16:07.529 INFO:teuthology.orchestra.run.smithi162.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-13T00:16:07.529 INFO:teuthology.orchestra.run.smithi162.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-13T00:16:07.529 INFO:teuthology.orchestra.run.smithi162.stdout:hardware devices. 2024-02-13T00:16:07.529 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:16:07.529 INFO:teuthology.orchestra.run.smithi162.stdout:This subpackage provides tools for support of all platforms. 2024-02-13T00:16:07.530 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-13T00:16:07.530 INFO:teuthology.task.kernel:Updating grub on smithi162 to boot 4.18.0-536.el8.x86_64 2024-02-13T00:16:07.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-13T00:16:08.994 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-02-13T00:16:11.477 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-02-13T00:16:11.479 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-02-13T00:16:11.501 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.otAnZ2NAeP 2024-02-13T00:16:11.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.otAnZ2NAeP 2024-02-13T00:16:11.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.otAnZ2NAeP 2024-02-13T00:16:11.747 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.otAnZ2NAeP is 6KB 2024-02-13T00:16:11.796 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.otAnZ2NAeP 2024-02-13T00:16:11.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo /bin/ls /boot/loader/entries || true 2024-02-13T00:16:11.885 INFO:teuthology.orchestra.run.smithi162.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-13T00:16:11.885 INFO:teuthology.orchestra.run.smithi162.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-13T00:16:11.885 INFO:teuthology.orchestra.run.smithi162.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-13T00:16:11.885 INFO:teuthology.orchestra.run.smithi162.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-13T00:16:11.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-13T00:16:12.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-02-13T00:16:34.095 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-13T00:16:34.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-13T00:16:34.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T00:16:42.044 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-13T00:16:42.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-13T00:16:42.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T00:17:08.668 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-13T00:17:16.604 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-02-13T00:17:17.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-13T00:17:17.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T00:17:25.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-13T00:17:25.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T00:17:36.064 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-02-13T00:17:48.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-13T00:17:48.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T00:17:48.407 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-02-13T00:17:48.639 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-13T00:17:48.639 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-536.el8.x86_64"... 2024-02-13T00:17:48.639 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-02-13T00:17:48.700 INFO:teuthology.orchestra.run.smithi181.stdout:4.18.0-536.el8.x86_64 2024-02-13T00:17:48.700 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-13T00:17:48.700 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-13T00:17:48.700 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-13T00:17:49.701 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-13T00:17:49.701 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-13T00:17:49.933 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-02-13T00:17:49.951 DEBUG:teuthology.parallel:result is None 2024-02-13T00:18:25.616 DEBUG:teuthology.orchestra.remote:timed out 2024-02-13T00:18:37.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-13T00:18:37.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T00:18:37.896 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-02-13T00:18:38.175 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-13T00:18:38.175 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-536.el8.x86_64"... 2024-02-13T00:18:38.176 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-02-13T00:18:38.195 INFO:teuthology.orchestra.run.smithi162.stdout:4.18.0-536.el8.x86_64 2024-02-13T00:18:38.196 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-13T00:18:38.196 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-13T00:18:38.196 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-13T00:18:39.197 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-13T00:18:39.197 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-13T00:18:39.266 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-02-13T00:18:39.285 DEBUG:teuthology.parallel:result is None 2024-02-13T00:18:39.285 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-13T00:18:39.293 INFO:teuthology.task.internal:Creating test directory... 2024-02-13T00:18:39.293 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-13T00:18:39.296 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-13T00:18:39.320 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-13T00:18:39.327 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-13T00:18:39.336 INFO:teuthology.task.internal:Creating archive directory... 2024-02-13T00:18:39.336 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-13T00:18:39.356 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-13T00:18:39.405 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-13T00:18:39.414 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-13T00:18:39.415 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-13T00:18:39.418 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-13T00:18:39.451 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T00:18:39.462 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T00:18:39.462 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T00:18:39.473 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T00:18:39.475 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-13T00:18:39.490 INFO:teuthology.task.internal:Configuring sudo... 2024-02-13T00:18:39.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-13T00:18:39.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-13T00:18:39.546 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-13T00:18:39.556 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-13T00:18:39.556 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-13T00:18:39.576 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-13T00:18:39.605 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T00:18:39.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T00:18:39.731 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T00:18:39.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T00:18:39.858 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:18:39.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-13T00:18:39.925 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T00:18:39.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T00:18:40.026 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T00:18:40.111 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T00:18:40.154 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-13T00:18:40.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-13T00:18:40.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-02-13T00:18:40.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-02-13T00:18:40.280 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T00:18:40.333 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T00:18:40.590 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-13T00:18:40.600 INFO:teuthology.task.internal:Starting timer... 2024-02-13T00:18:40.600 INFO:teuthology.run_tasks:Running task pcp... 2024-02-13T00:18:40.615 INFO:teuthology.run_tasks:Running task selinux... 2024-02-13T00:18:40.625 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-13T00:18:40.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-02-13T00:18:40.729 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-02-13T00:18:40.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-02-13T00:18:40.807 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-02-13T00:18:40.809 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-13T00:18:40.809 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-02-13T00:18:40.834 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-02-13T00:18:40.834 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-02-13T00:18:40.873 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-02-13T00:18:40.873 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi162.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-02-13T00:18:40.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T00:18:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T00:18:40.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T00:18:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T00:18:40.941 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-13T00:18:40.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-02-13T00:18:40.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-02-13T00:18:41.007 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-13T00:18:41.018 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-13T00:18:41.023 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-13T00:18:41.144 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-13T00:18:41.161 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-13T00:18:41.162 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi162.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-13T00:24:32.190 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi162.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-02-13T00:24:32.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-13T00:24:32.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T00:24:32.284 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-02-13T00:24:32.359 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-13T00:24:32.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-13T00:24:32.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T00:24:32.438 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-02-13T00:24:32.509 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-02-13T00:24:32.509 INFO:teuthology.run_tasks:Running task clock... 2024-02-13T00:24:32.522 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-13T00:24:32.522 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-13T00:24:32.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T00:24:32.525 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-13T00:24:32.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T00:24:32.556 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-13T00:24:32.569 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-13T00:24:32.579 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-13T00:24:32.593 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-13T00:24:32.638 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-02-13T00:24:32.649 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-02-13T00:24:32.655 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-02-13T00:24:32.662 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-13T00:24:32.666 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-02-13T00:24:32.676 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-13T00:24:32.680 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-13T00:24:32.693 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-13T00:24:32.742 INFO:teuthology.orchestra.run.smithi162.stderr:bash: ntpq: command not found 2024-02-13T00:24:32.746 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T00:24:32.746 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-02-13T00:24:32.746 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T00:24:32.746 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T00:24:32.746 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T00:24:32.746 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T00:24:32.760 INFO:teuthology.orchestra.run.smithi181.stderr:bash: ntpq: command not found 2024-02-13T00:24:32.765 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T00:24:32.765 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-02-13T00:24:32.765 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T00:24:32.765 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T00:24:32.765 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T00:24:32.765 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T00:24:32.766 INFO:teuthology.run_tasks:Running task pexec... 2024-02-13T00:24:32.775 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-13T00:24:32.775 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi162.front.sepia.ceph.com 2024-02-13T00:24:32.775 DEBUG:teuthology.orchestra.run.smithi162:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-13T00:24:32.776 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi181.front.sepia.ceph.com 2024-02-13T00:24:32.776 DEBUG:teuthology.orchestra.run.smithi181:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-13T00:24:33.570 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:26 ago on Tue 13 Feb 2024 12:23:07 AM UTC. 2024-02-13T00:24:33.594 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:24 ago on Tue 13 Feb 2024 12:23:09 AM UTC. 2024-02-13T00:24:35.543 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:24:35.544 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:24:35.544 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-02-13T00:24:35.544 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:24:35.544 INFO:teuthology.orchestra.run.smithi162.stdout:Resetting modules: 2024-02-13T00:24:35.544 INFO:teuthology.orchestra.run.smithi162.stdout: container-tools 2024-02-13T00:24:35.544 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:24:35.544 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:24:35.544 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:24:35.544 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:24:35.589 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:24:35.590 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:24:35.590 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-02-13T00:24:35.590 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:24:35.590 INFO:teuthology.orchestra.run.smithi181.stdout:Resetting modules: 2024-02-13T00:24:35.590 INFO:teuthology.orchestra.run.smithi181.stdout: container-tools 2024-02-13T00:24:35.590 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:24:35.590 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:24:35.590 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:24:35.591 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:24:35.794 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:24:35.847 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:24:36.647 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:29 ago on Tue 13 Feb 2024 12:23:07 AM UTC. 2024-02-13T00:24:36.717 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:27 ago on Tue 13 Feb 2024 12:23:09 AM UTC. 2024-02-13T00:24:38.592 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:24:38.627 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:24:38.627 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-13T00:24:38.627 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:24:38.627 INFO:teuthology.orchestra.run.smithi162.stdout:Installing group/module packages: 2024-02-13T00:24:38.627 INFO:teuthology.orchestra.run.smithi162.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-13T00:24:38.627 INFO:teuthology.orchestra.run.smithi162.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-13T00:24:38.627 INFO:teuthology.orchestra.run.smithi162.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-13T00:24:38.628 INFO:teuthology.orchestra.run.smithi162.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-13T00:24:38.628 INFO:teuthology.orchestra.run.smithi162.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-13T00:24:38.628 INFO:teuthology.orchestra.run.smithi162.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-13T00:24:38.628 INFO:teuthology.orchestra.run.smithi162.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-13T00:24:38.628 INFO:teuthology.orchestra.run.smithi162.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-13T00:24:38.628 INFO:teuthology.orchestra.run.smithi162.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-13T00:24:38.628 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-13T00:24:38.628 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-13T00:24:38.628 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-13T00:24:38.628 INFO:teuthology.orchestra.run.smithi162.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-13T00:24:38.628 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-02-13T00:24:38.628 INFO:teuthology.orchestra.run.smithi162.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-13T00:24:38.629 INFO:teuthology.orchestra.run.smithi162.stdout:Installing module profiles: 2024-02-13T00:24:38.629 INFO:teuthology.orchestra.run.smithi162.stdout: container-tools/common 2024-02-13T00:24:38.629 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:24:38.629 INFO:teuthology.orchestra.run.smithi162.stdout:Enabling module streams: 2024-02-13T00:24:38.629 INFO:teuthology.orchestra.run.smithi162.stdout: container-tools rhel8 2024-02-13T00:24:38.629 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:24:38.629 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:24:38.629 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:24:38.629 INFO:teuthology.orchestra.run.smithi162.stdout:Install 13 Packages 2024-02-13T00:24:38.629 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:24:38.630 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 27 M 2024-02-13T00:24:38.630 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 91 M 2024-02-13T00:24:38.630 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:24:38.690 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:24:38.724 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:24:38.724 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-13T00:24:38.724 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:24:38.724 INFO:teuthology.orchestra.run.smithi181.stdout:Installing group/module packages: 2024-02-13T00:24:38.724 INFO:teuthology.orchestra.run.smithi181.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-13T00:24:38.724 INFO:teuthology.orchestra.run.smithi181.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-13T00:24:38.724 INFO:teuthology.orchestra.run.smithi181.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-13T00:24:38.724 INFO:teuthology.orchestra.run.smithi181.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-13T00:24:38.724 INFO:teuthology.orchestra.run.smithi181.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-13T00:24:38.724 INFO:teuthology.orchestra.run.smithi181.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout:Installing module profiles: 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout: container-tools/common 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout:Enabling module streams: 2024-02-13T00:24:38.725 INFO:teuthology.orchestra.run.smithi181.stdout: container-tools rhel8 2024-02-13T00:24:38.726 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:24:38.726 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:24:38.726 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:24:38.726 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-02-13T00:24:38.726 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:24:38.726 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 27 M 2024-02-13T00:24:38.727 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 91 M 2024-02-13T00:24:38.727 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:24:38.926 INFO:teuthology.orchestra.run.smithi162.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 2.9 MB/s | 759 kB 00:00 2024-02-13T00:24:38.959 INFO:teuthology.orchestra.run.smithi162.stdout:(2/13): aardvark-dns-1.9.0-1.module_el8+804+f13 3.5 MB/s | 1.0 MB 00:00 2024-02-13T00:24:38.984 INFO:teuthology.orchestra.run.smithi162.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 646 kB/s | 38 kB 00:00 2024-02-13T00:24:39.018 INFO:teuthology.orchestra.run.smithi162.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 4.1 MB/s | 248 kB 00:00 2024-02-13T00:24:39.085 INFO:teuthology.orchestra.run.smithi162.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 2.6 MB/s | 179 kB 00:00 2024-02-13T00:24:39.152 INFO:teuthology.orchestra.run.smithi162.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 371 kB/s | 25 kB 00:00 2024-02-13T00:24:39.218 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 1.6 MB/s | 759 kB 00:00 2024-02-13T00:24:39.261 INFO:teuthology.orchestra.run.smithi162.stdout:(7/13): buildah-1.34.0-1.module_el8+804+f131391 16 MB/s | 9.7 MB 00:00 2024-02-13T00:24:39.277 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 651 kB/s | 38 kB 00:00 2024-02-13T00:24:39.286 INFO:teuthology.orchestra.run.smithi162.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 703 kB/s | 94 kB 00:00 2024-02-13T00:24:39.353 INFO:teuthology.orchestra.run.smithi162.stdout:(9/13): netavark-1.9.0-1.module_el8+804+f131391 11 MB/s | 4.0 MB 00:00 2024-02-13T00:24:39.360 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): aardvark-dns-1.9.0-1.module_el8+804+f13 1.7 MB/s | 1.0 MB 00:00 2024-02-13T00:24:39.412 INFO:teuthology.orchestra.run.smithi162.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 903 kB/s | 53 kB 00:00 2024-02-13T00:24:39.419 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 1.7 MB/s | 248 kB 00:00 2024-02-13T00:24:39.479 INFO:teuthology.orchestra.run.smithi162.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 615 kB/s | 41 kB 00:00 2024-02-13T00:24:39.528 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 1.6 MB/s | 179 kB 00:00 2024-02-13T00:24:39.529 INFO:teuthology.orchestra.run.smithi162.stdout:(12/13): toolbox-0.0.99.4-8.module_el8+804+f131 10 MB/s | 2.5 MB 00:00 2024-02-13T00:24:39.586 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 424 kB/s | 25 kB 00:00 2024-02-13T00:24:39.629 INFO:teuthology.orchestra.run.smithi162.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 24 MB/s | 8.8 MB 00:00 2024-02-13T00:24:39.629 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:24:39.629 INFO:teuthology.orchestra.run.smithi162.stdout:Total 27 MB/s | 27 MB 00:01 2024-02-13T00:24:39.645 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2024-02-13T00:24:39.898 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:24:39.919 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:24:39.919 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:24:40.185 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:24:40.188 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:24:40.195 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): netavark-1.9.0-1.module_el8+804+f131391 4.7 MB/s | 4.0 MB 00:00 2024-02-13T00:24:40.496 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): buildah-1.34.0-1.module_el8+804+f131391 5.6 MB/s | 9.7 MB 00:01 2024-02-13T00:24:40.571 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): toolbox-0.0.99.4-8.module_el8+804+f131 6.7 MB/s | 2.5 MB 00:00 2024-02-13T00:24:40.596 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 525 kB/s | 53 kB 00:00 2024-02-13T00:24:40.630 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 701 kB/s | 41 kB 00:00 2024-02-13T00:24:40.869 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:24:40.897 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 7.0 MB/s | 8.8 MB 00:01 2024-02-13T00:24:40.897 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:24:40.897 INFO:teuthology.orchestra.run.smithi181.stdout:Total 13 MB/s | 27 MB 00:02 2024-02-13T00:24:41.089 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-13T00:24:41.165 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:24:41.186 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:24:41.186 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:24:41.242 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-13T00:24:41.456 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:24:41.457 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-13T00:24:41.459 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:24:41.686 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-13T00:24:42.179 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:24:42.200 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-13T00:24:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-13T00:24:42.582 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-13T00:24:42.759 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-13T00:24:42.974 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-13T00:24:43.171 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-13T00:24:43.439 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-13T00:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-13T00:24:43.659 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-13T00:24:43.954 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-13T00:24:44.377 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-13T00:24:44.475 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-13T00:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-13T00:24:44.987 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-13T00:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-13T00:24:45.326 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-13T00:24:45.533 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-13T00:24:45.643 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-13T00:24:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-13T00:24:46.134 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-13T00:24:46.134 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-13T00:24:46.134 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-13T00:24:46.135 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-13T00:24:46.135 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-13T00:24:46.135 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-13T00:24:46.135 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-13T00:24:46.135 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-13T00:24:46.135 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-13T00:24:46.135 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-13T00:24:46.135 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-13T00:24:46.135 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-13T00:24:46.135 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-13T00:24:46.591 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-13T00:24:46.652 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-13T00:24:46.652 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:24:46.652 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:24:46.652 INFO:teuthology.orchestra.run.smithi162.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-13T00:24:46.652 INFO:teuthology.orchestra.run.smithi162.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:24:46.653 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:24:46.803 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-13T00:24:46.918 DEBUG:teuthology.parallel:result is None 2024-02-13T00:24:46.997 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-13T00:24:47.414 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-13T00:24:47.414 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-13T00:24:47.414 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-13T00:24:47.414 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-13T00:24:47.414 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-13T00:24:47.414 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-13T00:24:47.414 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-13T00:24:47.414 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-13T00:24:47.414 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-13T00:24:47.414 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-13T00:24:47.415 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-13T00:24:47.415 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-13T00:24:47.415 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-13T00:24:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-13T00:24:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:24:47.953 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:24:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-13T00:24:47.953 INFO:teuthology.orchestra.run.smithi181.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:24:47.954 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:24:48.242 DEBUG:teuthology.parallel:result is None 2024-02-13T00:24:48.242 INFO:teuthology.run_tasks:Running task install... 2024-02-13T00:24:48.254 DEBUG:teuthology.task.install:project ceph 2024-02-13T00:24:48.254 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}} 2024-02-13T00:24:48.254 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-13T00:24:48.254 INFO:teuthology.task.install:Using flavor: default 2024-02-13T00:24:48.260 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-13T00:24:48.260 INFO:teuthology.task.install:extra packages: [] 2024-02-13T00:24:48.260 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': None, 'wait_for_package': False} 2024-02-13T00:24:48.261 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T00:24:48.263 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': None, 'wait_for_package': False} 2024-02-13T00:24:48.263 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T00:24:48.402 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/ 2024-02-13T00:24:48.403 INFO:teuthology.task.install.rpm:Package version is 16.2.14-521.g0e714d9a 2024-02-13T00:24:48.500 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/ 2024-02-13T00:24:48.500 INFO:teuthology.task.install.rpm:Package version is 16.2.14-521.g0e714d9a 2024-02-13T00:24:48.508 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-13T00:24:48.508 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:24:48.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-13T00:24:48.541 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-13T00:24:48.541 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-13T00:24:48.603 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-02-08-0854-pacific/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-13T00:24:48.604 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-13T00:24:48.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-13T00:24:48.611 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-13T00:24:48.632 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-13T00:24:48.632 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-13T00:24:48.691 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-13T00:24:48.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-13T00:24:48.760 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-02-13T00:24:48.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-02-13T00:24:48.776 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-13T00:24:48.841 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-02-13T00:24:48.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-02-13T00:24:49.029 INFO:teuthology.orchestra.run.smithi162.stdout:76 files removed 2024-02-13T00:24:49.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw 2024-02-13T00:24:49.120 INFO:teuthology.orchestra.run.smithi181.stdout:76 files removed 2024-02-13T00:24:49.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-02-13T00:24:50.138 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 114 kB/s | 76 kB 00:00 2024-02-13T00:24:50.231 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 112 kB/s | 76 kB 00:00 2024-02-13T00:24:50.597 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 36 kB/s | 15 kB 00:00 2024-02-13T00:24:50.615 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 45 kB/s | 15 kB 00:00 2024-02-13T00:24:51.041 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 3.6 kB/s | 1.4 kB 00:00 2024-02-13T00:24:51.206 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 2.5 kB/s | 1.4 kB 00:00 2024-02-13T00:24:52.625 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 8 - AppStream 17 MB/s | 27 MB 00:01 2024-02-13T00:24:54.773 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - AppStream 7.6 MB/s | 27 MB 00:03 2024-02-13T00:24:58.542 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 8 - BaseOS 9.7 MB/s | 10 MB 00:01 2024-02-13T00:25:00.451 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 8 - Extras 64 kB/s | 18 kB 00:00 2024-02-13T00:25:00.698 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - BaseOS 9.0 MB/s | 10 MB 00:01 2024-02-13T00:25:01.068 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2024-02-13T00:25:02.865 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2024-02-13T00:25:03.186 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 8 - PowerTools 2.3 MB/s | 4.8 MB 00:02 2024-02-13T00:25:03.582 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - Extras common packages 11 kB/s | 6.9 kB 00:00 2024-02-13T00:25:04.530 INFO:teuthology.orchestra.run.smithi181.stdout:Copr repo for python3-asyncssh owned by ceph 8.3 kB/s | 3.5 kB 00:00 2024-02-13T00:25:05.709 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - PowerTools 2.3 MB/s | 4.8 MB 00:02 2024-02-13T00:25:06.918 INFO:teuthology.orchestra.run.smithi162.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2024-02-13T00:25:07.083 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 16 MB 00:02 2024-02-13T00:25:10.938 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 190 kB/s | 24 kB 00:00 2024-02-13T00:25:14.499 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:25:14.501 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:25:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-13T00:25:14.502 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:25:14.502 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:25:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 12 M 2024-02-13T00:25:14.502 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-02-13T00:25:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.7 M 2024-02-13T00:25:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 4.2 M 2024-02-13T00:25:14.502 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-13T00:25:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 6.1 M 2024-02-13T00:25:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 24 M 2024-02-13T00:25:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 26 k 2024-02-13T00:25:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-13T00:25:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-13T00:25:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 825 k 2024-02-13T00:25:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-13T00:25:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-13T00:25:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 494 k 2024-02-13T00:25:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-13T00:25:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.8 M 2024-02-13T00:25:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-13T00:25:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-13T00:25:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 45 k 2024-02-13T00:25:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 99 k 2024-02-13T00:25:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 215 k 2024-02-13T00:25:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-13T00:25:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 389 k 2024-02-13T00:25:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 369 k 2024-02-13T00:25:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 114 k 2024-02-13T00:25:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:25:14.504 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:25:14.504 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:25:14.504 INFO:teuthology.orchestra.run.smithi181.stdout:Install 21 Packages 2024-02-13T00:25:14.504 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-02-13T00:25:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:25:14.505 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 58 M 2024-02-13T00:25:14.505 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout:(1/23): ceph-radosgw-16.2.14-521.g0e714d9a.el8. 9.4 MB/s | 12 MB 00:01 2024-02-13T00:25:17.024 INFO:teuthology.orchestra.run.smithi181.stdout:(2/23): ceph-selinux-16.2.14-521.g0e714d9a.el8. 84 kB/s | 26 kB 00:00 2024-02-13T00:25:17.133 INFO:teuthology.orchestra.run.smithi181.stdout:(3/23): libcephfs2-16.2.14-521.g0e714d9a.el8.x8 7.4 MB/s | 825 kB 00:00 2024-02-13T00:25:17.183 INFO:teuthology.orchestra.run.smithi181.stdout:(4/23): libradosstriper1-16.2.14-521.g0e714d9a. 9.7 MB/s | 494 kB 00:00 2024-02-13T00:25:17.408 INFO:teuthology.orchestra.run.smithi181.stdout:(5/23): librgw2-16.2.14-521.g0e714d9a.el8.x86_6 17 MB/s | 3.8 MB 00:00 2024-02-13T00:25:17.442 INFO:teuthology.orchestra.run.smithi181.stdout:(6/23): python3-ceph-argparse-16.2.14-521.g0e71 1.3 MB/s | 45 kB 00:00 2024-02-13T00:25:17.483 INFO:teuthology.orchestra.run.smithi181.stdout:(7/23): python3-ceph-common-16.2.14-521.g0e714d 2.3 MB/s | 99 kB 00:00 2024-02-13T00:25:17.525 INFO:teuthology.orchestra.run.smithi181.stdout:(8/23): python3-cephfs-16.2.14-521.g0e714d9a.el 5.1 MB/s | 215 kB 00:00 2024-02-13T00:25:17.576 INFO:teuthology.orchestra.run.smithi181.stdout:(9/23): python3-rados-16.2.14-521.g0e714d9a.el8 7.6 MB/s | 389 kB 00:00 2024-02-13T00:25:17.644 INFO:teuthology.orchestra.run.smithi181.stdout:(10/23): python3-rbd-16.2.14-521.g0e714d9a.el8. 5.3 MB/s | 369 kB 00:00 2024-02-13T00:25:17.693 INFO:teuthology.orchestra.run.smithi181.stdout:(11/23): python3-rgw-16.2.14-521.g0e714d9a.el8. 2.3 MB/s | 114 kB 00:00 2024-02-13T00:25:18.035 INFO:teuthology.orchestra.run.smithi181.stdout:(12/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.9 MB/s | 677 kB 00:00 2024-02-13T00:25:18.110 INFO:teuthology.orchestra.run.smithi181.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.4 MB/s | 259 kB 00:00 2024-02-13T00:25:18.177 INFO:teuthology.orchestra.run.smithi181.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 659 kB/s | 44 kB 00:00 2024-02-13T00:25:18.294 INFO:teuthology.orchestra.run.smithi181.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 403 kB/s | 47 kB 00:00 2024-02-13T00:25:18.562 INFO:teuthology.orchestra.run.smithi181.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2024-02-13T00:25:18.654 INFO:teuthology.orchestra.run.smithi181.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 1.9 MB/s | 181 kB 00:00 2024-02-13T00:25:18.745 INFO:teuthology.orchestra.run.smithi181.stdout:(18/23): ceph-base-16.2.14-521.g0e714d9a.el8.x8 1.9 MB/s | 6.1 MB 00:03 2024-02-13T00:25:18.771 INFO:teuthology.orchestra.run.smithi181.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 503 kB/s | 59 kB 00:00 2024-02-13T00:25:19.004 INFO:teuthology.orchestra.run.smithi181.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 290 kB/s | 75 kB 00:00 2024-02-13T00:25:19.338 INFO:teuthology.orchestra.run.smithi181.stdout:(21/23): librbd1-16.2.14-521.g0e714d9a.el8.x86_ 13 MB/s | 4.2 MB 00:00 2024-02-13T00:25:20.430 INFO:teuthology.orchestra.run.smithi181.stdout:(22/23): librados2-16.2.14-521.g0e714d9a.el8.x8 2.3 MB/s | 3.7 MB 00:01 2024-02-13T00:25:24.136 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 966 kB/s | 16 MB 00:17 2024-02-13T00:25:27.890 INFO:teuthology.orchestra.run.smithi181.stdout:(23/23): ceph-common-16.2.14-521.g0e714d9a.el8. 2.0 MB/s | 24 MB 00:12 2024-02-13T00:25:27.892 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:25:27.893 INFO:teuthology.orchestra.run.smithi181.stdout:Total 4.3 MB/s | 58 MB 00:13 2024-02-13T00:25:27.916 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 361 kB/s | 24 kB 00:00 2024-02-13T00:25:27.936 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:25:27.984 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:25:27.984 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:25:28.335 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:25:28.339 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:25:29.349 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:25:29.548 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-13T00:25:29.716 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-13T00:25:30.130 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-13T00:25:30.300 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-13T00:25:30.398 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-13T00:25:30.647 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 4/25 2024-02-13T00:25:30.816 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-13T00:25:31.233 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-13T00:25:31.416 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 6/25 2024-02-13T00:25:31.486 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 6/25 2024-02-13T00:25:31.587 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:25:31.590 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:25:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-13T00:25:31.590 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:25:31.590 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:25:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 12 M 2024-02-13T00:25:31.590 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-02-13T00:25:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.7 M 2024-02-13T00:25:31.590 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 4.2 M 2024-02-13T00:25:31.590 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-13T00:25:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 6.1 M 2024-02-13T00:25:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 24 M 2024-02-13T00:25:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 26 k 2024-02-13T00:25:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-13T00:25:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-13T00:25:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 825 k 2024-02-13T00:25:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-13T00:25:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-13T00:25:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 494 k 2024-02-13T00:25:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-13T00:25:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.8 M 2024-02-13T00:25:31.591 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-13T00:25:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-13T00:25:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 45 k 2024-02-13T00:25:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 99 k 2024-02-13T00:25:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 215 k 2024-02-13T00:25:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-13T00:25:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 389 k 2024-02-13T00:25:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 369 k 2024-02-13T00:25:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 114 k 2024-02-13T00:25:31.592 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:25:31.592 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:25:31.592 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:25:31.593 INFO:teuthology.orchestra.run.smithi162.stdout:Install 21 Packages 2024-02-13T00:25:31.593 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 2 Packages 2024-02-13T00:25:31.593 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:25:31.593 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 58 M 2024-02-13T00:25:31.593 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:25:31.684 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-13T00:25:32.180 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-13T00:25:32.308 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 9/25 2024-02-13T00:25:32.410 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 9/25 2024-02-13T00:25:32.588 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-13T00:25:32.746 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8. 11/25 2024-02-13T00:25:32.793 INFO:teuthology.orchestra.run.smithi162.stdout:(1/23): ceph-base-16.2.14-521.g0e714d9a.el8.x86 9.0 MB/s | 6.1 MB 00:00 2024-02-13T00:25:32.859 INFO:teuthology.orchestra.run.smithi162.stdout:(2/23): ceph-selinux-16.2.14-521.g0e714d9a.el8. 387 kB/s | 26 kB 00:00 2024-02-13T00:25:32.903 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 12/25 2024-02-13T00:25:32.959 INFO:teuthology.orchestra.run.smithi162.stdout:(3/23): libcephfs2-16.2.14-521.g0e714d9a.el8.x8 8.1 MB/s | 825 kB 00:00 2024-02-13T00:25:33.060 INFO:teuthology.orchestra.run.smithi162.stdout:(4/23): libradosstriper1-16.2.14-521.g0e714d9a. 4.8 MB/s | 494 kB 00:00 2024-02-13T00:25:33.409 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 13/25 2024-02-13T00:25:33.468 INFO:teuthology.orchestra.run.smithi162.stdout:(5/23): librgw2-16.2.14-521.g0e714d9a.el8.x86_6 9.3 MB/s | 3.8 MB 00:00 2024-02-13T00:25:33.535 INFO:teuthology.orchestra.run.smithi162.stdout:(6/23): python3-ceph-argparse-16.2.14-521.g0e71 676 kB/s | 45 kB 00:00 2024-02-13T00:25:33.599 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 14/25 2024-02-13T00:25:33.758 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 15/25 2024-02-13T00:25:33.777 INFO:teuthology.orchestra.run.smithi162.stdout:(7/23): ceph-common-16.2.14-521.g0e714d9a.el8.x 15 MB/s | 24 MB 00:01 2024-02-13T00:25:33.802 INFO:teuthology.orchestra.run.smithi162.stdout:(8/23): python3-ceph-common-16.2.14-521.g0e714d 370 kB/s | 99 kB 00:00 2024-02-13T00:25:33.828 INFO:teuthology.orchestra.run.smithi162.stdout:(9/23): python3-cephfs-16.2.14-521.g0e714d9a.el 4.2 MB/s | 215 kB 00:00 2024-02-13T00:25:33.830 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 15/25 2024-02-13T00:25:33.870 INFO:teuthology.orchestra.run.smithi162.stdout:(10/23): python3-rados-16.2.14-521.g0e714d9a.el 5.7 MB/s | 389 kB 00:00 2024-02-13T00:25:33.916 INFO:teuthology.orchestra.run.smithi162.stdout:(11/23): python3-rbd-16.2.14-521.g0e714d9a.el8. 4.1 MB/s | 369 kB 00:00 2024-02-13T00:25:33.954 INFO:teuthology.orchestra.run.smithi162.stdout:(12/23): python3-rgw-16.2.14-521.g0e714d9a.el8. 1.3 MB/s | 114 kB 00:00 2024-02-13T00:25:34.033 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-13T00:25:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-13T00:25:34.304 INFO:teuthology.orchestra.run.smithi162.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 742 kB/s | 259 kB 00:00 2024-02-13T00:25:34.403 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-13T00:25:34.412 INFO:teuthology.orchestra.run.smithi162.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 405 kB/s | 44 kB 00:00 2024-02-13T00:25:34.552 INFO:teuthology.orchestra.run.smithi162.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 340 kB/s | 47 kB 00:00 2024-02-13T00:25:34.616 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x8 19/25 2024-02-13T00:25:34.638 INFO:teuthology.orchestra.run.smithi162.stdout:(16/23): librdkafka-1.6.1-1.el8.x86_64.rpm 937 kB/s | 677 kB 00:00 2024-02-13T00:25:34.864 INFO:teuthology.orchestra.run.smithi162.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 983 kB/s | 306 kB 00:00 2024-02-13T00:25:34.947 INFO:teuthology.orchestra.run.smithi162.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 706 kB/s | 59 kB 00:00 2024-02-13T00:25:34.972 INFO:teuthology.orchestra.run.smithi162.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 542 kB/s | 181 kB 00:00 2024-02-13T00:25:35.031 INFO:teuthology.orchestra.run.smithi162.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 899 kB/s | 75 kB 00:00 2024-02-13T00:25:35.306 INFO:teuthology.orchestra.run.smithi162.stdout:(21/23): librados2-16.2.14-521.g0e714d9a.el8.x8 11 MB/s | 3.7 MB 00:00 2024-02-13T00:25:35.398 INFO:teuthology.orchestra.run.smithi162.stdout:(22/23): librbd1-16.2.14-521.g0e714d9a.el8.x86_ 11 MB/s | 4.2 MB 00:00 2024-02-13T00:25:37.154 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-13T00:25:37.383 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-13T00:25:38.087 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-13T00:25:38.396 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 21/25 2024-02-13T00:25:38.504 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 21/25 2024-02-13T00:25:38.715 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-13T00:25:38.999 INFO:teuthology.orchestra.run.smithi162.stdout:(23/23): ceph-radosgw-16.2.14-521.g0e714d9a.el8 1.7 MB/s | 12 MB 00:06 2024-02-13T00:25:39.000 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:25:39.001 INFO:teuthology.orchestra.run.smithi162.stdout:Total 7.8 MB/s | 58 MB 00:07 2024-02-13T00:25:39.043 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:25:39.091 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:25:39.091 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:25:39.463 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:25:39.466 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:25:40.484 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:25:40.700 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-13T00:25:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-13T00:25:41.310 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-13T00:25:41.471 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-13T00:25:41.572 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-13T00:25:41.859 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 4/25 2024-02-13T00:25:42.046 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-13T00:25:42.463 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-13T00:25:42.646 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 6/25 2024-02-13T00:25:42.711 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 6/25 2024-02-13T00:25:42.940 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-13T00:25:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-13T00:25:43.571 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 9/25 2024-02-13T00:25:43.643 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 9/25 2024-02-13T00:25:43.810 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-13T00:25:43.959 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8. 11/25 2024-02-13T00:25:44.118 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 12/25 2024-02-13T00:25:44.355 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 13/25 2024-02-13T00:25:44.586 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 14/25 2024-02-13T00:25:44.745 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 15/25 2024-02-13T00:25:44.850 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 15/25 2024-02-13T00:25:45.046 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-13T00:25:45.236 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-13T00:25:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-13T00:25:45.612 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x8 19/25 2024-02-13T00:25:48.051 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-13T00:25:48.253 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-13T00:25:48.964 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 20/25 2024-02-13T00:25:49.246 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 21/25 2024-02-13T00:25:49.374 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 21/25 2024-02-13T00:25:49.503 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-13T00:25:57.806 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-13T00:25:57.806 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-02-13T00:25:57.806 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-02-13T00:25:57.806 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /dev 2024-02-13T00:25:57.806 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /run 2024-02-13T00:25:57.806 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-02-13T00:25:57.807 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-02-13T00:25:57.807 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-02-13T00:25:57.807 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-02-13T00:25:57.807 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /dev 2024-02-13T00:25:57.807 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:25:58.993 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 23/25 2024-02-13T00:25:59.071 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 23/25 2024-02-13T00:25:59.072 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-13T00:25:59.297 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-13T00:25:59.297 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-13T00:26:00.050 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-13T00:26:00.051 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 1/25 2024-02-13T00:26:00.051 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2/25 2024-02-13T00:26:00.051 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-13T00:26:00.051 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 4/25 2024-02-13T00:26:00.051 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-13T00:26:00.051 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 6/25 2024-02-13T00:26:00.051 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 7/25 2024-02-13T00:26:00.051 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8. 8/25 2024-02-13T00:26:00.051 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x8 9/25 2024-02-13T00:26:00.051 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 10/25 2024-02-13T00:26:00.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 11/25 2024-02-13T00:26:00.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 12/25 2024-02-13T00:26:00.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 13/25 2024-02-13T00:26:00.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-13T00:26:00.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-13T00:26:00.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-13T00:26:00.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-13T00:26:00.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-13T00:26:00.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-13T00:26:00.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-13T00:26:00.052 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-13T00:26:00.053 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-13T00:26:00.053 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-13T00:26:00.053 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 24/25 2024-02-13T00:26:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-13T00:26:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:26:00.919 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-02-13T00:26:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.920 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:26:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-13T00:26:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: leveldb-1.22-1.el8.x86_64 2024-02-13T00:26:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-13T00:26:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-13T00:26:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-13T00:26:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-13T00:26:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-13T00:26:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-13T00:26:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:26:00.921 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:26:01.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-02-13T00:26:01.968 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:51 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:26:03.483 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:26:03.484 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:26:03.484 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-13T00:26:03.484 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:26:03.484 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:26:03.484 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 52 M 2024-02-13T00:26:03.484 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-13T00:26:03.484 INFO:teuthology.orchestra.run.smithi181.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-13T00:26:03.484 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 174 k 2024-02-13T00:26:03.484 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-13T00:26:03.484 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-13T00:26:03.484 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-13T00:26:03.485 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:26:03.485 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:26:03.485 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:26:03.485 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-02-13T00:26:03.485 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:26:03.485 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 53 M 2024-02-13T00:26:03.485 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 197 M 2024-02-13T00:26:03.485 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:26:03.904 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): libcephsqlite-16.2.14-521.g0e714d9a.el8. 924 kB/s | 174 kB 00:00 2024-02-13T00:26:04.112 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 512 kB/s | 203 kB 00:00 2024-02-13T00:26:04.246 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 550 kB/s | 188 kB 00:00 2024-02-13T00:26:04.279 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.9 MB/s | 323 kB 00:00 2024-02-13T00:26:04.338 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 770 kB/s | 71 kB 00:00 2024-02-13T00:26:08.485 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-13T00:26:08.485 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-02-13T00:26:08.485 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-02-13T00:26:08.485 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /dev 2024-02-13T00:26:08.485 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /run 2024-02-13T00:26:08.485 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-02-13T00:26:08.485 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-02-13T00:26:08.485 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-02-13T00:26:08.485 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-02-13T00:26:08.485 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /dev 2024-02-13T00:26:08.486 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:09.624 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 23/25 2024-02-13T00:26:09.703 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 23/25 2024-02-13T00:26:09.703 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-13T00:26:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-13T00:26:09.942 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-13T00:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-13T00:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 1/25 2024-02-13T00:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2/25 2024-02-13T00:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 3/25 2024-02-13T00:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 4/25 2024-02-13T00:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 5/25 2024-02-13T00:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_6 6/25 2024-02-13T00:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 7/25 2024-02-13T00:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8. 8/25 2024-02-13T00:26:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x8 9/25 2024-02-13T00:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 10/25 2024-02-13T00:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 11/25 2024-02-13T00:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 12/25 2024-02-13T00:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 13/25 2024-02-13T00:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-13T00:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-13T00:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-13T00:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-13T00:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-13T00:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-13T00:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-13T00:26:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-13T00:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 22/25 2024-02-13T00:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-13T00:26:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 24/25 2024-02-13T00:26:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-13T00:26:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:11.384 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: leveldb-1.22-1.el8.x86_64 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-13T00:26:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-13T00:26:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-13T00:26:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-13T00:26:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:11.386 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:26:11.725 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-test 2024-02-13T00:26:12.487 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:45 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:26:13.999 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:26:14.000 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:26:14.000 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-13T00:26:14.000 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:26:14.000 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:26:14.000 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 52 M 2024-02-13T00:26:14.001 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-13T00:26:14.001 INFO:teuthology.orchestra.run.smithi162.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-13T00:26:14.001 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 174 k 2024-02-13T00:26:14.001 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-13T00:26:14.001 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-13T00:26:14.001 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-13T00:26:14.001 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:14.001 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:26:14.001 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:26:14.001 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-02-13T00:26:14.001 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:14.002 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 53 M 2024-02-13T00:26:14.002 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 197 M 2024-02-13T00:26:14.002 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:26:14.436 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): libcephsqlite-16.2.14-521.g0e714d9a.el8. 945 kB/s | 174 kB 00:00 2024-02-13T00:26:14.595 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 593 kB/s | 203 kB 00:00 2024-02-13T00:26:14.737 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.2 MB/s | 323 kB 00:00 2024-02-13T00:26:14.770 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 563 kB/s | 188 kB 00:00 2024-02-13T00:26:14.820 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 846 kB/s | 71 kB 00:00 2024-02-13T00:26:16.721 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): ceph-test-16.2.14-521.g0e714d9a.el8.x86_ 21 MB/s | 52 MB 00:02 2024-02-13T00:26:16.722 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:26:16.723 INFO:teuthology.orchestra.run.smithi162.stdout:Total 20 MB/s | 53 MB 00:02 2024-02-13T00:26:16.765 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:26:16.785 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:26:16.785 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:26:17.073 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:26:17.077 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:26:17.573 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:26:17.797 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-13T00:26:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-13T00:26:18.146 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-13T00:26:18.225 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-13T00:26:18.419 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-13T00:26:18.571 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 5/6 2024-02-13T00:26:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 5/6 2024-02-13T00:26:23.469 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 6/6 2024-02-13T00:26:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 6/6 2024-02-13T00:26:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 1/6 2024-02-13T00:26:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 2/6 2024-02-13T00:26:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-13T00:26:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-13T00:26:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-13T00:26:24.142 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-13T00:26:24.142 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:24.142 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:26:24.142 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:24.142 INFO:teuthology.orchestra.run.smithi162.stdout: jq-1.6-8.el8.x86_64 2024-02-13T00:26:24.142 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:24.143 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-13T00:26:24.143 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-13T00:26:24.143 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-13T00:26:24.143 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:24.143 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:26:24.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph 2024-02-13T00:26:25.248 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:58 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:26:26.761 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:26:26.764 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:26:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 6.4 k 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 2.4 M 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 1.6 M 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 245 k 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 5.2 M 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 19 M 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-13T00:26:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-13T00:26:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout:Install 32 Packages 2024-02-13T00:26:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:26.768 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 32 M 2024-02-13T00:26:26.768 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 104 M 2024-02-13T00:26:26.768 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:26:27.556 INFO:teuthology.orchestra.run.smithi162.stdout:(1/32): ceph-16.2.14-521.g0e714d9a.el8.x86_64.r 24 kB/s | 6.4 kB 00:00 2024-02-13T00:26:27.605 INFO:teuthology.orchestra.run.smithi162.stdout:(2/32): ceph-mds-16.2.14-521.g0e714d9a.el8.x86_ 7.6 MB/s | 2.4 MB 00:00 2024-02-13T00:26:28.523 INFO:teuthology.orchestra.run.smithi162.stdout:(3/32): ceph-osd-16.2.14-521.g0e714d9a.el8.x86_ 20 MB/s | 19 MB 00:00 2024-02-13T00:26:28.564 INFO:teuthology.orchestra.run.smithi162.stdout:(4/32): ceph-mgr-16.2.14-521.g0e714d9a.el8.x86_ 1.3 MB/s | 1.6 MB 00:01 2024-02-13T00:26:28.590 INFO:teuthology.orchestra.run.smithi162.stdout:(5/32): ceph-mgr-modules-core-16.2.14-521.g0e71 3.6 MB/s | 245 kB 00:00 2024-02-13T00:26:28.782 INFO:teuthology.orchestra.run.smithi162.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 202 kB/s | 39 kB 00:00 2024-02-13T00:26:28.882 INFO:teuthology.orchestra.run.smithi162.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 495 kB/s | 157 kB 00:00 2024-02-13T00:26:28.933 INFO:teuthology.orchestra.run.smithi162.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 688 kB/s | 103 kB 00:00 2024-02-13T00:26:28.958 INFO:teuthology.orchestra.run.smithi162.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 759 kB/s | 57 kB 00:00 2024-02-13T00:26:29.091 INFO:teuthology.orchestra.run.smithi162.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.8 MB/s | 457 kB 00:00 2024-02-13T00:26:29.291 INFO:teuthology.orchestra.run.smithi162.stdout:(11/32): python3-cryptography-3.2.1-7.el8.x86_6 3.1 MB/s | 641 kB 00:00 2024-02-13T00:26:29.334 INFO:teuthology.orchestra.run.smithi162.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 2.6 MB/s | 109 kB 00:00 2024-02-13T00:26:29.359 INFO:teuthology.orchestra.run.smithi162.stdout:(13/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 620 kB/s | 249 kB 00:00 2024-02-13T00:26:29.901 INFO:teuthology.orchestra.run.smithi162.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 77 kB/s | 44 kB 00:00 2024-02-13T00:26:30.134 INFO:teuthology.orchestra.run.smithi162.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 239 kB/s | 185 kB 00:00 2024-02-13T00:26:30.218 INFO:teuthology.orchestra.run.smithi162.stdout:(16/32): ceph-mon-16.2.14-521.g0e714d9a.el8.x86 2.0 MB/s | 5.2 MB 00:02 2024-02-13T00:26:30.310 INFO:teuthology.orchestra.run.smithi162.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 422 kB/s | 173 kB 00:00 2024-02-13T00:26:30.460 INFO:teuthology.orchestra.run.smithi162.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 117 kB/s | 18 kB 00:00 2024-02-13T00:26:30.485 INFO:teuthology.orchestra.run.smithi162.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2024-02-13T00:26:30.644 INFO:teuthology.orchestra.run.smithi162.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 266 kB/s | 49 kB 00:00 2024-02-13T00:26:30.669 INFO:teuthology.orchestra.run.smithi162.stdout:(21/32): python3-jaraco-6.2-6.el8.noarch.rpm 25 kB/s | 11 kB 00:00 2024-02-13T00:26:30.695 INFO:teuthology.orchestra.run.smithi162.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 281 kB/s | 59 kB 00:00 2024-02-13T00:26:30.828 INFO:teuthology.orchestra.run.smithi162.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 140 kB/s | 19 kB 00:00 2024-02-13T00:26:30.853 INFO:teuthology.orchestra.run.smithi162.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 85 kB/s | 16 kB 00:00 2024-02-13T00:26:30.895 INFO:teuthology.orchestra.run.smithi162.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2024-02-13T00:26:30.963 INFO:teuthology.orchestra.run.smithi162.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 179 kB/s | 24 kB 00:00 2024-02-13T00:26:31.030 INFO:teuthology.orchestra.run.smithi162.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 206 kB/s | 27 kB 00:00 2024-02-13T00:26:31.063 INFO:teuthology.orchestra.run.smithi162.stdout:(28/32): python3-tempora-1.14.1-5.el8.noarch.rp 140 kB/s | 29 kB 00:00 2024-02-13T00:26:31.155 INFO:teuthology.orchestra.run.smithi162.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.3 MB/s | 255 kB 00:00 2024-02-13T00:26:31.222 INFO:teuthology.orchestra.run.smithi162.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2024-02-13T00:26:31.297 INFO:teuthology.orchestra.run.smithi162.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 162 kB/s | 23 kB 00:00 2024-02-13T00:26:31.389 INFO:teuthology.orchestra.run.smithi162.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 262 kB/s | 85 kB 00:00 2024-02-13T00:26:31.393 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:26:31.393 INFO:teuthology.orchestra.run.smithi162.stdout:Total 6.9 MB/s | 32 MB 00:04 2024-02-13T00:26:31.475 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:26:31.514 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:26:31.514 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:26:31.838 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:26:31.841 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:26:32.948 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:26:33.164 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-13T00:26:33.375 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-13T00:26:33.658 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-13T00:26:33.874 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-13T00:26:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-13T00:26:34.101 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): ceph-test-16.2.14-521.g0e714d9a.el8.x86_ 1.7 MB/s | 52 MB 00:30 2024-02-13T00:26:34.103 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:26:34.103 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.7 MB/s | 53 MB 00:30 2024-02-13T00:26:34.143 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:26:34.164 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:26:34.165 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:26:34.310 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-13T00:26:34.452 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:26:34.456 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:26:34.503 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-13T00:26:34.698 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-13T00:26:34.936 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:26:34.958 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-13T00:26:35.152 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-13T00:26:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-13T00:26:35.355 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-13T00:26:35.415 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-13T00:26:35.489 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-13T00:26:35.556 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-13T00:26:35.631 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-13T00:26:35.738 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-13T00:26:35.864 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 5/6 2024-02-13T00:26:35.979 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-13T00:26:36.261 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-13T00:26:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-13T00:26:36.758 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-13T00:26:36.984 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-13T00:26:37.300 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-13T00:26:37.539 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-13T00:26:37.686 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-13T00:26:37.895 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-13T00:26:38.164 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-13T00:26:38.396 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-13T00:26:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-13T00:26:38.941 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-13T00:26:39.245 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-13T00:26:39.616 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8. 27/32 2024-02-13T00:26:39.817 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 28/32 2024-02-13T00:26:40.462 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 5/6 2024-02-13T00:26:40.712 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 6/6 2024-02-13T00:26:41.004 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 6/6 2024-02-13T00:26:41.005 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 1/6 2024-02-13T00:26:41.005 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 2/6 2024-02-13T00:26:41.005 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-13T00:26:41.005 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-13T00:26:41.005 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-13T00:26:41.345 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 28/32 2024-02-13T00:26:41.403 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-13T00:26:41.403 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:26:41.404 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:26:41.404 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:41.404 INFO:teuthology.orchestra.run.smithi181.stdout: jq-1.6-8.el8.x86_64 2024-02-13T00:26:41.404 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:41.404 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-13T00:26:41.404 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-13T00:26:41.404 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-13T00:26:41.404 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:26:41.404 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:26:41.533 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 29/32 2024-02-13T00:26:41.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-02-13T00:26:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 29/32 2024-02-13T00:26:42.158 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 30/32 2024-02-13T00:26:42.374 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 30/32 2024-02-13T00:26:42.412 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:32 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:26:42.591 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 31/32 2024-02-13T00:26:42.609 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 31/32 2024-02-13T00:26:42.732 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 32/32 2024-02-13T00:26:43.147 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 32/32 2024-02-13T00:26:43.148 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 1/32 2024-02-13T00:26:43.148 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 2/32 2024-02-13T00:26:43.148 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 3/32 2024-02-13T00:26:43.148 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 4/32 2024-02-13T00:26:43.148 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 5/32 2024-02-13T00:26:43.148 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8. 6/32 2024-02-13T00:26:43.148 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-13T00:26:43.148 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-13T00:26:43.148 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-13T00:26:43.148 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-13T00:26:43.148 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-13T00:26:43.148 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-13T00:26:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-13T00:26:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-13T00:26:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-13T00:26:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-13T00:26:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-13T00:26:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-13T00:26:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-13T00:26:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-13T00:26:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-13T00:26:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-13T00:26:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-13T00:26:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-13T00:26:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-13T00:26:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-13T00:26:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-13T00:26:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-13T00:26:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-13T00:26:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-13T00:26:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-13T00:26:43.932 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:26:43.935 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:26:43.935 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-13T00:26:43.935 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:26:43.935 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:26:43.936 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 6.4 k 2024-02-13T00:26:43.936 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-13T00:26:43.936 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 2.4 M 2024-02-13T00:26:43.936 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 1.6 M 2024-02-13T00:26:43.936 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 245 k 2024-02-13T00:26:43.936 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 5.2 M 2024-02-13T00:26:43.936 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 19 M 2024-02-13T00:26:43.936 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-13T00:26:43.936 INFO:teuthology.orchestra.run.smithi181.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-13T00:26:43.936 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-13T00:26:43.936 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-13T00:26:43.936 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-13T00:26:43.937 INFO:teuthology.orchestra.run.smithi181.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-13T00:26:43.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-13T00:26:43.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-13T00:26:43.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-13T00:26:43.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-13T00:26:43.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-13T00:26:43.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-13T00:26:43.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-13T00:26:43.938 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:26:43.938 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:26:43.938 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:26:43.938 INFO:teuthology.orchestra.run.smithi181.stdout:Install 32 Packages 2024-02-13T00:26:43.938 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:26:43.939 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 32 M 2024-02-13T00:26:43.939 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 104 M 2024-02-13T00:26:43.939 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:26:44.295 INFO:teuthology.orchestra.run.smithi181.stdout:(1/32): ceph-16.2.14-521.g0e714d9a.el8.x86_64.r 50 kB/s | 6.4 kB 00:00 2024-02-13T00:26:44.520 INFO:teuthology.orchestra.run.smithi181.stdout:(2/32): ceph-mds-16.2.14-521.g0e714d9a.el8.x86_ 6.9 MB/s | 2.4 MB 00:00 2024-02-13T00:26:44.521 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-13T00:26:44.521 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-13T00:26:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-2.6-1.el8.noarch 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-13T00:26:44.523 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-13T00:26:44.524 INFO:teuthology.orchestra.run.smithi162.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-13T00:26:44.524 INFO:teuthology.orchestra.run.smithi162.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-13T00:26:44.524 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-13T00:26:44.524 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-13T00:26:44.524 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-13T00:26:44.524 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-13T00:26:44.524 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:44.524 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:26:44.562 INFO:teuthology.orchestra.run.smithi181.stdout:(3/32): ceph-mgr-16.2.14-521.g0e714d9a.el8.x86_ 4.1 MB/s | 1.6 MB 00:00 2024-02-13T00:26:44.654 INFO:teuthology.orchestra.run.smithi181.stdout:(4/32): ceph-mgr-modules-core-16.2.14-521.g0e71 2.6 MB/s | 245 kB 00:00 2024-02-13T00:26:44.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base 2024-02-13T00:26:44.954 INFO:teuthology.orchestra.run.smithi181.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 523 kB/s | 157 kB 00:00 2024-02-13T00:26:45.038 INFO:teuthology.orchestra.run.smithi181.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 466 kB/s | 39 kB 00:00 2024-02-13T00:26:45.113 INFO:teuthology.orchestra.run.smithi181.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.3 MB/s | 103 kB 00:00 2024-02-13T00:26:45.188 INFO:teuthology.orchestra.run.smithi181.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 762 kB/s | 57 kB 00:00 2024-02-13T00:26:45.364 INFO:teuthology.orchestra.run.smithi181.stdout:(9/32): ceph-mon-16.2.14-521.g0e714d9a.el8.x86_ 4.9 MB/s | 5.2 MB 00:01 2024-02-13T00:26:45.398 INFO:teuthology.orchestra.run.smithi181.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.1 MB/s | 457 kB 00:00 2024-02-13T00:26:45.524 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:18 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:26:45.598 INFO:teuthology.orchestra.run.smithi181.stdout:(11/32): ceph-osd-16.2.14-521.g0e714d9a.el8.x86 17 MB/s | 19 MB 00:01 2024-02-13T00:26:45.665 INFO:teuthology.orchestra.run.smithi181.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 828 kB/s | 249 kB 00:00 2024-02-13T00:26:45.765 INFO:teuthology.orchestra.run.smithi181.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-02-13T00:26:45.790 INFO:teuthology.orchestra.run.smithi181.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 568 kB/s | 109 kB 00:00 2024-02-13T00:26:45.982 INFO:teuthology.orchestra.run.smithi181.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 137 kB/s | 44 kB 00:00 2024-02-13T00:26:46.124 INFO:teuthology.orchestra.run.smithi181.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 523 kB/s | 185 kB 00:00 2024-02-13T00:26:46.150 INFO:teuthology.orchestra.run.smithi181.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 481 kB/s | 173 kB 00:00 2024-02-13T00:26:46.192 INFO:teuthology.orchestra.run.smithi181.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2024-02-13T00:26:46.217 INFO:teuthology.orchestra.run.smithi181.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 263 kB/s | 18 kB 00:00 2024-02-13T00:26:46.242 INFO:teuthology.orchestra.run.smithi181.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-02-13T00:26:46.284 INFO:teuthology.orchestra.run.smithi181.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 529 kB/s | 49 kB 00:00 2024-02-13T00:26:46.301 INFO:teuthology.orchestra.run.smithi181.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 702 kB/s | 59 kB 00:00 2024-02-13T00:26:46.351 INFO:teuthology.orchestra.run.smithi181.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2024-02-13T00:26:46.368 INFO:teuthology.orchestra.run.smithi181.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2024-02-13T00:26:46.385 INFO:teuthology.orchestra.run.smithi181.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 223 kB/s | 19 kB 00:00 2024-02-13T00:26:46.419 INFO:teuthology.orchestra.run.smithi181.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 356 kB/s | 24 kB 00:00 2024-02-13T00:26:46.444 INFO:teuthology.orchestra.run.smithi181.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2024-02-13T00:26:46.461 INFO:teuthology.orchestra.run.smithi181.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2024-02-13T00:26:46.528 INFO:teuthology.orchestra.run.smithi181.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2024-02-13T00:26:46.553 INFO:teuthology.orchestra.run.smithi181.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 928 kB/s | 85 kB 00:00 2024-02-13T00:26:46.578 INFO:teuthology.orchestra.run.smithi181.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2024-02-13T00:26:46.595 INFO:teuthology.orchestra.run.smithi181.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 344 kB/s | 23 kB 00:00 2024-02-13T00:26:46.599 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:26:46.599 INFO:teuthology.orchestra.run.smithi181.stdout:Total 12 MB/s | 32 MB 00:02 2024-02-13T00:26:46.681 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:26:46.718 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:26:46.718 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:26:46.978 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:26:47.024 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:26:47.025 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-13T00:26:47.025 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:26:47.031 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:26:47.034 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:26:47.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install cephadm 2024-02-13T00:26:47.885 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:20 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:26:48.139 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:26:48.400 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-13T00:26:48.612 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-13T00:26:48.854 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-13T00:26:49.077 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-13T00:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-13T00:26:49.399 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:26:49.400 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:26:49.400 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-13T00:26:49.400 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:26:49.400 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:26:49.400 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 85 k 2024-02-13T00:26:49.400 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:49.400 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:26:49.400 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:26:49.401 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-13T00:26:49.401 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:49.401 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 85 k 2024-02-13T00:26:49.401 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 352 k 2024-02-13T00:26:49.401 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:26:49.514 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-13T00:26:49.591 INFO:teuthology.orchestra.run.smithi162.stdout:cephadm-16.2.14-521.g0e714d9a.el8.noarch.rpm 449 kB/s | 85 kB 00:00 2024-02-13T00:26:49.592 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:26:49.592 INFO:teuthology.orchestra.run.smithi162.stdout:Total 446 kB/s | 85 kB 00:00 2024-02-13T00:26:49.592 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:26:49.599 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:26:49.599 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:26:49.601 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:26:49.604 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:26:49.650 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:26:49.698 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-13T00:26:49.884 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-13T00:26:50.094 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-13T00:26:50.105 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-13T00:26:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-13T00:26:50.364 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-13T00:26:50.573 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-13T00:26:50.577 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-13T00:26:50.843 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-13T00:26:50.986 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-13T00:26:50.986 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:50.986 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:26:50.986 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:26:50.986 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:50.986 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:26:51.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-immutable-object-cache 2024-02-13T00:26:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-13T00:26:51.498 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-13T00:26:51.695 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-13T00:26:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-13T00:26:51.970 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:24 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:26:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-13T00:26:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-13T00:26:52.742 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-13T00:26:52.947 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-13T00:26:53.144 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-13T00:26:53.377 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-13T00:26:53.447 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:26:53.448 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:26:53.448 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-02-13T00:26:53.448 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:26:53.448 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:26:53.448 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 161 k 2024-02-13T00:26:53.448 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:53.448 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:26:53.448 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:26:53.449 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-13T00:26:53.449 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:53.449 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 161 k 2024-02-13T00:26:53.449 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 470 k 2024-02-13T00:26:53.449 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:26:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-13T00:26:53.630 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-immutable-object-cache-16.2.14-521.g0e714d 891 kB/s | 161 kB 00:00 2024-02-13T00:26:53.630 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:26:53.631 INFO:teuthology.orchestra.run.smithi162.stdout:Total 885 kB/s | 161 kB 00:00 2024-02-13T00:26:53.631 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:26:53.641 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:26:53.641 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:26:53.687 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:26:53.690 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:26:53.821 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:26:53.829 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-13T00:26:54.160 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-13T00:26:54.183 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-13T00:26:54.464 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-13T00:26:54.857 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8. 27/32 2024-02-13T00:26:55.051 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 28/32 2024-02-13T00:26:55.240 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-13T00:26:55.906 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-13T00:26:55.906 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:55.906 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:26:55.906 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:55.906 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:26:55.906 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:26:56.093 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr 2024-02-13T00:26:56.576 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 28/32 2024-02-13T00:26:56.736 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 29/32 2024-02-13T00:26:56.860 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:29 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:26:57.181 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 29/32 2024-02-13T00:26:57.361 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 30/32 2024-02-13T00:26:57.578 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 30/32 2024-02-13T00:26:57.803 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 31/32 2024-02-13T00:26:57.821 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 31/32 2024-02-13T00:26:57.944 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 32/32 2024-02-13T00:26:58.323 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:26:58.357 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 32/32 2024-02-13T00:26:58.357 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 1/32 2024-02-13T00:26:58.357 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 2/32 2024-02-13T00:26:58.357 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 3/32 2024-02-13T00:26:58.357 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 4/32 2024-02-13T00:26:58.357 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 5/32 2024-02-13T00:26:58.357 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8. 6/32 2024-02-13T00:26:58.357 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-13T00:26:58.357 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-13T00:26:58.357 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-13T00:26:58.357 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-13T00:26:58.358 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-13T00:26:58.359 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-13T00:26:58.359 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-13T00:26:58.359 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-13T00:26:58.359 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-13T00:26:58.359 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-13T00:26:58.359 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-13T00:26:58.370 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:26:58.371 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-13T00:26:58.371 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:26:58.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-dashboard 2024-02-13T00:26:59.233 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-13T00:26:59.234 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-2.6-1.el8.noarch 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-13T00:26:59.235 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-13T00:26:59.236 INFO:teuthology.orchestra.run.smithi181.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-13T00:26:59.236 INFO:teuthology.orchestra.run.smithi181.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-13T00:26:59.236 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-13T00:26:59.236 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-13T00:26:59.236 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-13T00:26:59.236 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-13T00:26:59.236 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:26:59.236 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:26:59.239 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:32 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:26:59.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-02-13T00:27:00.269 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:50 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:27:00.759 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:27:00.760 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:00.760 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-13T00:27:00.760 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:00.760 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:27:00.760 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 3.1 M 2024-02-13T00:27:00.761 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-13T00:27:00.761 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 24 k 2024-02-13T00:27:00.761 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 15 k 2024-02-13T00:27:00.761 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-13T00:27:00.761 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-13T00:27:00.761 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-13T00:27:00.761 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:00.761 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:27:00.761 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:00.761 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-02-13T00:27:00.761 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:00.762 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 3.4 M 2024-02-13T00:27:00.762 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 22 M 2024-02-13T00:27:00.762 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:27:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): ceph-grafana-dashboards-16.2.14-521.g0e7 171 kB/s | 24 kB 00:00 2024-02-13T00:27:01.285 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): ceph-prometheus-alerts-16.2.14-521.g0e71 74 kB/s | 15 kB 00:00 2024-02-13T00:27:01.411 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 232 kB/s | 43 kB 00:00 2024-02-13T00:27:01.735 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:27:01.782 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:27:01.783 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-13T00:27:01.783 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:27:01.869 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 58 kB/s | 34 kB 00:00 2024-02-13T00:27:01.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-02-13T00:27:02.494 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): ceph-mgr-dashboard-16.2.14-521.g0e714d9a 2.2 MB/s | 3.1 MB 00:01 2024-02-13T00:27:02.652 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:52 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:27:02.769 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 145 kB/s | 196 kB 00:01 2024-02-13T00:27:02.773 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:02.774 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.7 MB/s | 3.4 MB 00:02 2024-02-13T00:27:02.785 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:27:02.796 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:27:02.796 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:27:02.966 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:27:02.969 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:27:03.235 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:27:03.442 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-13T00:27:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-13T00:27:03.955 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-13T00:27:04.067 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.n 4/6 2024-02-13T00:27:04.156 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:27:04.157 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:04.157 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-13T00:27:04.157 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:04.157 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:27:04.157 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 85 k 2024-02-13T00:27:04.157 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:04.157 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:27:04.157 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:04.158 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-13T00:27:04.158 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:04.158 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 85 k 2024-02-13T00:27:04.158 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 352 k 2024-02-13T00:27:04.158 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:27:04.320 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-16.2.14-521.g0e714d9a.el8.noarch.rpm 525 kB/s | 85 kB 00:00 2024-02-13T00:27:04.321 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:04.321 INFO:teuthology.orchestra.run.smithi181.stdout:Total 521 kB/s | 85 kB 00:00 2024-02-13T00:27:04.321 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:27:04.328 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:27:04.328 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:27:04.330 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:27:04.333 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:27:04.376 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:27:04.890 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-13T00:27:05.090 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8. 5/6 2024-02-13T00:27:05.095 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-13T00:27:05.327 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-13T00:27:05.361 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 6/6 2024-02-13T00:27:05.646 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 6/6 2024-02-13T00:27:05.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8. 1/6 2024-02-13T00:27:05.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 2/6 2024-02-13T00:27:05.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.n 3/6 2024-02-13T00:27:05.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-13T00:27:05.646 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-13T00:27:05.708 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/1 2024-02-13T00:27:05.709 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:05.709 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:27:05.709 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:05.709 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:05.709 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:27:05.928 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-02-13T00:27:06.039 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-13T00:27:06.040 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:06.040 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:27:06.040 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:06.040 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:06.040 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:06.040 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-13T00:27:06.040 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-13T00:27:06.040 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-13T00:27:06.040 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:06.040 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:27:06.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-13T00:27:06.692 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:56 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:27:06.924 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:39 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:27:08.214 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:27:08.215 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:08.215 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-02-13T00:27:08.215 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:08.215 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:27:08.215 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 161 k 2024-02-13T00:27:08.215 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:08.215 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:27:08.215 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:08.215 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-13T00:27:08.216 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:08.216 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 161 k 2024-02-13T00:27:08.216 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 470 k 2024-02-13T00:27:08.216 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:27:08.400 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-16.2.14-521.g0e714d 875 kB/s | 161 kB 00:00 2024-02-13T00:27:08.401 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:08.401 INFO:teuthology.orchestra.run.smithi181.stdout:Total 869 kB/s | 161 kB 00:00 2024-02-13T00:27:08.401 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:27:08.411 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:27:08.412 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:27:08.412 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:27:08.413 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-02-13T00:27:08.413 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-13T00:27:08.413 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-02-13T00:27:08.413 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:27:08.413 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 8.6 M 2024-02-13T00:27:08.413 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-13T00:27:08.413 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-13T00:27:08.413 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-13T00:27:08.413 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-13T00:27:08.414 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-13T00:27:08.414 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-13T00:27:08.414 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-13T00:27:08.414 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-13T00:27:08.414 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:08.414 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:27:08.414 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-02-13T00:27:08.414 INFO:teuthology.orchestra.run.smithi162.stdout:Install 8 Packages 2024-02-13T00:27:08.414 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:08.415 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 37 M 2024-02-13T00:27:08.415 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 210 M 2024-02-13T00:27:08.415 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:27:08.459 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:27:08.462 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:27:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:27:08.801 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-13T00:27:09.202 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-13T00:27:09.734 INFO:teuthology.orchestra.run.smithi162.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.1 MB/s | 4.9 MB 00:00 2024-02-13T00:27:10.034 INFO:teuthology.orchestra.run.smithi162.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 12 MB/s | 3.7 MB 00:00 2024-02-13T00:27:10.134 INFO:teuthology.orchestra.run.smithi162.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 3.9 MB/s | 4.7 MB 00:01 2024-02-13T00:27:10.161 INFO:teuthology.orchestra.run.smithi162.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.8 MB/s | 225 kB 00:00 2024-02-13T00:27:10.232 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a. 1/1 2024-02-13T00:27:10.232 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:10.232 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:27:10.232 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:27:10.232 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:10.232 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:27:10.352 INFO:teuthology.orchestra.run.smithi162.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.3 MB/s | 645 kB 00:00 2024-02-13T00:27:10.402 INFO:teuthology.orchestra.run.smithi162.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 3.4 MB/s | 172 kB 00:00 2024-02-13T00:27:11.227 INFO:teuthology.orchestra.run.smithi162.stdout:(7/8): ceph-mgr-diskprediction-local-16.2.14-52 3.8 MB/s | 8.6 MB 00:02 2024-02-13T00:27:11.704 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-02-13T00:27:12.470 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:02 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:27:12.621 INFO:teuthology.orchestra.run.smithi162.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 5.7 MB/s | 14 MB 00:02 2024-02-13T00:27:12.622 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:12.622 INFO:teuthology.orchestra.run.smithi162.stdout:Total 8.8 MB/s | 37 MB 00:04 2024-02-13T00:27:12.854 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:27:12.873 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:27:12.873 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:27:13.212 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:27:13.215 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:27:13.769 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:27:13.917 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-13T00:27:13.943 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:27:13.989 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:27:13.990 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-13T00:27:13.990 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:27:14.060 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-13T00:27:14.107 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-02-13T00:27:14.217 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-13T00:27:14.828 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-13T00:27:14.862 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:04 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:27:14.983 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-13T00:27:15.596 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-13T00:27:15.766 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-13T00:27:16.368 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:27:16.370 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:16.370 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-13T00:27:16.370 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:16.370 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:27:16.370 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 3.1 M 2024-02-13T00:27:16.370 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-13T00:27:16.370 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 24 k 2024-02-13T00:27:16.370 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 15 k 2024-02-13T00:27:16.370 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-13T00:27:16.370 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-13T00:27:16.371 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-13T00:27:16.371 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:16.371 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:27:16.371 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:16.371 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-02-13T00:27:16.371 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:16.371 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.4 M 2024-02-13T00:27:16.371 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 22 M 2024-02-13T00:27:16.372 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:27:16.473 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-13T00:27:16.836 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-13T00:27:16.863 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): ceph-prometheus-alerts-16.2.14-521.g0e71 93 kB/s | 15 kB 00:00 2024-02-13T00:27:16.888 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): ceph-grafana-dashboards-16.2.14-521.g0e7 131 kB/s | 24 kB 00:00 2024-02-13T00:27:17.014 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 283 kB/s | 43 kB 00:00 2024-02-13T00:27:17.049 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-13T00:27:17.098 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): ceph-mgr-dashboard-16.2.14-521.g0e714d9a 7.9 MB/s | 3.1 MB 00:00 2024-02-13T00:27:17.531 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 52 kB/s | 34 kB 00:00 2024-02-13T00:27:18.656 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 120 kB/s | 196 kB 00:01 2024-02-13T00:27:18.658 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:18.659 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.5 MB/s | 3.4 MB 00:02 2024-02-13T00:27:18.671 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:27:18.682 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:27:18.682 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:27:18.802 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-13T00:27:18.849 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:27:18.852 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:27:19.135 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:27:19.388 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-13T00:27:19.658 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-13T00:27:19.885 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-13T00:27:19.988 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.n 4/6 2024-02-13T00:27:20.079 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-13T00:27:20.282 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 8/8 2024-02-13T00:27:20.620 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 8/8 2024-02-13T00:27:20.621 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 1/8 2024-02-13T00:27:20.621 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-13T00:27:20.621 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-13T00:27:20.621 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-13T00:27:20.621 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-13T00:27:20.621 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-13T00:27:20.621 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-13T00:27:21.003 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8. 5/6 2024-02-13T00:27:21.019 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-13T00:27:21.019 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:21.020 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:27:21.020 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:21.020 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-13T00:27:21.020 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-13T00:27:21.020 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-13T00:27:21.020 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-13T00:27:21.020 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-13T00:27:21.020 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-13T00:27:21.020 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-13T00:27:21.020 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:21.020 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:27:21.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-rook 2024-02-13T00:27:21.263 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 6/6 2024-02-13T00:27:21.552 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 6/6 2024-02-13T00:27:21.552 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8. 1/6 2024-02-13T00:27:21.553 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarc 2/6 2024-02-13T00:27:21.553 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.n 3/6 2024-02-13T00:27:21.553 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-13T00:27:21.553 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-13T00:27:21.901 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:54 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:27:21.940 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-13T00:27:21.940 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:21.940 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:27:21.940 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:21.940 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:21.940 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:21.940 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-13T00:27:21.940 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-13T00:27:21.941 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-13T00:27:21.941 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:21.941 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:27:22.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-13T00:27:22.835 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:12 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:27:23.421 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:27:23.423 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:23.423 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-13T00:27:23.423 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:23.423 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:27:23.423 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 27 k 2024-02-13T00:27:23.423 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-13T00:27:23.423 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-13T00:27:23.423 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-13T00:27:23.423 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:23.424 INFO:teuthology.orchestra.run.smithi162.stdout:Install 13 Packages 2024-02-13T00:27:23.425 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:23.425 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 1.9 M 2024-02-13T00:27:23.425 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 24 M 2024-02-13T00:27:23.425 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:27:23.878 INFO:teuthology.orchestra.run.smithi162.stdout:(1/13): ceph-mgr-rook-16.2.14-521.g0e714d9a.el8 183 kB/s | 27 kB 00:00 2024-02-13T00:27:23.903 INFO:teuthology.orchestra.run.smithi162.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 152 kB/s | 27 kB 00:00 2024-02-13T00:27:23.929 INFO:teuthology.orchestra.run.smithi162.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 100 kB/s | 20 kB 00:00 2024-02-13T00:27:24.062 INFO:teuthology.orchestra.run.smithi162.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 687 kB/s | 126 kB 00:00 2024-02-13T00:27:24.088 INFO:teuthology.orchestra.run.smithi162.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 976 kB/s | 155 kB 00:00 2024-02-13T00:27:24.113 INFO:teuthology.orchestra.run.smithi162.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 597 kB/s | 110 kB 00:00 2024-02-13T00:27:24.172 INFO:teuthology.orchestra.run.smithi162.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 390 kB/s | 43 kB 00:00 2024-02-13T00:27:24.338 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:27:24.339 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================================ 2024-02-13T00:27:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-13T00:27:24.339 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================================ 2024-02-13T00:27:24.340 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:27:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 8.6 M 2024-02-13T00:27:24.340 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-13T00:27:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-13T00:27:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-13T00:27:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-13T00:27:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-13T00:27:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-13T00:27:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-13T00:27:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-13T00:27:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:24.341 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:27:24.341 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================================ 2024-02-13T00:27:24.341 INFO:teuthology.orchestra.run.smithi181.stdout:Install 8 Packages 2024-02-13T00:27:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:24.341 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 37 M 2024-02-13T00:27:24.342 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 210 M 2024-02-13T00:27:24.342 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:27:24.389 INFO:teuthology.orchestra.run.smithi162.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 110 kB/s | 33 kB 00:00 2024-02-13T00:27:24.406 INFO:teuthology.orchestra.run.smithi162.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 56 kB/s | 16 kB 00:00 2024-02-13T00:27:24.540 INFO:teuthology.orchestra.run.smithi162.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 227 kB/s | 81 kB 00:00 2024-02-13T00:27:24.582 INFO:teuthology.orchestra.run.smithi162.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 366 kB/s | 64 kB 00:00 2024-02-13T00:27:24.707 INFO:teuthology.orchestra.run.smithi162.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 367 kB/s | 61 kB 00:00 2024-02-13T00:27:25.521 INFO:teuthology.orchestra.run.smithi181.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-52 10 MB/s | 8.6 MB 00:00 2024-02-13T00:27:26.790 INFO:teuthology.orchestra.run.smithi162.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 497 kB/s | 1.2 MB 00:02 2024-02-13T00:27:26.792 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:26.793 INFO:teuthology.orchestra.run.smithi162.stdout:Total 581 kB/s | 1.9 MB 00:03 2024-02-13T00:27:26.850 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:27:26.866 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:27:26.922 INFO:teuthology.orchestra.run.smithi181.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 2.7 MB/s | 3.7 MB 00:01 2024-02-13T00:27:26.954 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:27:26.957 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:27:26.996 INFO:teuthology.orchestra.run.smithi181.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.9 MB/s | 225 kB 00:00 2024-02-13T00:27:27.544 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:27:27.816 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-13T00:27:27.997 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-13T00:27:28.207 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-13T00:27:28.420 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-13T00:27:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-13T00:27:28.673 INFO:teuthology.orchestra.run.smithi181.stdout:(4/8): python3-scipy-1.0.0-21.module_el8+762+77 8.5 MB/s | 14 MB 00:01 2024-02-13T00:27:28.808 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-13T00:27:28.890 INFO:teuthology.orchestra.run.smithi181.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 1.1 MB/s | 4.7 MB 00:04 2024-02-13T00:27:28.940 INFO:teuthology.orchestra.run.smithi181.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.4 MB/s | 645 kB 00:00 2024-02-13T00:27:29.032 INFO:teuthology.orchestra.run.smithi181.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.2 MB/s | 172 kB 00:00 2024-02-13T00:27:29.124 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-13T00:27:29.367 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-13T00:27:30.428 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-13T00:27:30.590 INFO:teuthology.orchestra.run.smithi181.stdout:(8/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 842 kB/s | 4.9 MB 00:05 2024-02-13T00:27:30.592 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:30.592 INFO:teuthology.orchestra.run.smithi181.stdout:Total 5.9 MB/s | 37 MB 00:06 2024-02-13T00:27:30.686 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-13T00:27:30.825 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:27:30.843 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:27:30.843 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:27:30.902 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-13T00:27:31.095 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-13T00:27:31.182 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:27:31.185 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:27:31.262 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 13/13 2024-02-13T00:27:31.554 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 13/13 2024-02-13T00:27:31.554 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 1/13 2024-02-13T00:27:31.554 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-13T00:27:31.555 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-13T00:27:31.555 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-13T00:27:31.555 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-13T00:27:31.555 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-13T00:27:31.555 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-13T00:27:31.555 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-13T00:27:31.555 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-13T00:27:31.555 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-13T00:27:31.555 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-13T00:27:31.555 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-13T00:27:31.747 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:27:31.864 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-13T00:27:32.001 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-13T00:27:32.002 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-13T00:27:32.003 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-13T00:27:32.003 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:32.003 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:27:32.003 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-13T00:27:32.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-cephadm 2024-02-13T00:27:32.180 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-13T00:27:32.780 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-13T00:27:32.895 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:05 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:27:32.939 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-13T00:27:33.558 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-13T00:27:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-13T00:27:34.402 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:27:34.404 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:34.404 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-13T00:27:34.404 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:34.404 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:27:34.404 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 116 k 2024-02-13T00:27:34.404 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-13T00:27:34.404 INFO:teuthology.orchestra.run.smithi162.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-13T00:27:34.404 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-13T00:27:34.404 INFO:teuthology.orchestra.run.smithi162.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-13T00:27:34.404 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-13T00:27:34.405 INFO:teuthology.orchestra.run.smithi162.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-13T00:27:34.405 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:34.405 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:27:34.405 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:34.405 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-02-13T00:27:34.405 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:34.405 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 5.8 M 2024-02-13T00:27:34.405 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 24 M 2024-02-13T00:27:34.406 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:27:34.407 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-13T00:27:34.750 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-13T00:27:34.940 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): ceph-mgr-cephadm-16.2.14-521.g0e714d9a.e 637 kB/s | 116 kB 00:00 2024-02-13T00:27:34.979 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-13T00:27:35.140 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 106 kB/s | 21 kB 00:00 2024-02-13T00:27:35.165 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.3 MB/s | 538 kB 00:00 2024-02-13T00:27:35.424 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 189 kB/s | 49 kB 00:00 2024-02-13T00:27:35.449 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.1 MB/s | 355 kB 00:00 2024-02-13T00:27:35.708 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.0 MB/s | 4.8 MB 00:00 2024-02-13T00:27:35.710 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:35.710 INFO:teuthology.orchestra.run.smithi162.stdout:Total 4.5 MB/s | 5.8 MB 00:01 2024-02-13T00:27:35.762 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:27:35.772 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:27:35.772 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:27:35.870 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:27:35.873 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:27:36.166 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:27:36.412 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-13T00:27:36.690 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-13T00:27:36.739 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-13T00:27:37.610 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-13T00:27:37.990 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-13T00:27:38.033 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-13T00:27:38.204 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 8/8 2024-02-13T00:27:38.520 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-13T00:27:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 8/8 2024-02-13T00:27:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9 1/8 2024-02-13T00:27:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-13T00:27:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-13T00:27:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-13T00:27:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-13T00:27:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-13T00:27:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-13T00:27:38.712 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 6/6 2024-02-13T00:27:38.938 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-13T00:27:38.939 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:38.939 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:27:38.939 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:38.939 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-13T00:27:38.939 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-13T00:27:38.939 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-13T00:27:38.940 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-13T00:27:38.940 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-13T00:27:38.940 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-13T00:27:38.940 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-13T00:27:38.940 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:38.940 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:27:38.989 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 6/6 2024-02-13T00:27:38.989 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/6 2024-02-13T00:27:38.989 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-13T00:27:38.989 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-13T00:27:38.989 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-13T00:27:38.989 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-13T00:27:39.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-02-13T00:27:39.384 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-13T00:27:39.385 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:39.385 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:27:39.385 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:39.385 INFO:teuthology.orchestra.run.smithi162.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-13T00:27:39.385 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-13T00:27:39.385 INFO:teuthology.orchestra.run.smithi162.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-13T00:27:39.385 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-02-13T00:27:39.385 INFO:teuthology.orchestra.run.smithi162.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-13T00:27:39.385 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:39.385 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:27:39.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-fuse 2024-02-13T00:27:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:29 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:27:40.280 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:13 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:27:41.353 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:27:41.355 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:41.355 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-13T00:27:41.355 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:41.355 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:27:41.355 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 27 k 2024-02-13T00:27:41.355 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-13T00:27:41.355 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-13T00:27:41.355 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-13T00:27:41.355 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-13T00:27:41.355 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-02-13T00:27:41.356 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:41.357 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 1.9 M 2024-02-13T00:27:41.357 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 24 M 2024-02-13T00:27:41.357 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:27:41.791 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:27:41.791 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-13T00:27:41.792 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:41.792 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:27:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 911 k 2024-02-13T00:27:41.792 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-13T00:27:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-13T00:27:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:41.792 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:27:41.792 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:41.792 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-02-13T00:27:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:41.793 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 994 k 2024-02-13T00:27:41.793 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.8 M 2024-02-13T00:27:41.793 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:27:41.813 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): ceph-mgr-rook-16.2.14-521.g0e714d9a.el8 194 kB/s | 27 kB 00:00 2024-02-13T00:27:41.838 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 159 kB/s | 27 kB 00:00 2024-02-13T00:27:41.864 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 104 kB/s | 20 kB 00:00 2024-02-13T00:27:41.956 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 889 kB/s | 126 kB 00:00 2024-02-13T00:27:41.986 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 665 kB/s | 83 kB 00:00 2024-02-13T00:27:41.998 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.1 MB/s | 155 kB 00:00 2024-02-13T00:27:42.023 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 692 kB/s | 110 kB 00:00 2024-02-13T00:27:42.090 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 318 kB/s | 43 kB 00:00 2024-02-13T00:27:42.157 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 208 kB/s | 33 kB 00:00 2024-02-13T00:27:42.182 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 103 kB/s | 16 kB 00:00 2024-02-13T00:27:42.258 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 857 kB/s | 64 kB 00:00 2024-02-13T00:27:42.291 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 407 kB/s | 81 kB 00:00 2024-02-13T00:27:42.353 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2024-02-13T00:27:42.383 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.2 MB/s | 1.2 MB 00:00 2024-02-13T00:27:42.386 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:42.386 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.9 MB/s | 1.9 MB 00:01 2024-02-13T00:27:42.420 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-fuse-16.2.14-521.g0e714d9a.el8.x86_ 1.6 MB/s | 911 kB 00:00 2024-02-13T00:27:42.420 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:42.421 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.5 MB/s | 994 kB 00:00 2024-02-13T00:27:42.431 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:27:42.443 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:27:42.451 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:27:42.451 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:27:42.459 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:27:42.459 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:27:42.537 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:27:42.540 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:27:42.548 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:27:42.552 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:27:42.721 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:27:43.003 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-13T00:27:43.185 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2/2 2024-02-13T00:27:43.262 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:27:43.574 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2/2 2024-02-13T00:27:43.574 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/2 2024-02-13T00:27:43.682 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-13T00:27:43.880 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-13T00:27:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-13T00:27:44.227 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-13T00:27:44.227 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:44.227 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:27:44.227 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-13T00:27:44.228 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:44.228 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:27:44.335 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-13T00:27:44.456 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados-devel 2024-02-13T00:27:44.594 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-13T00:27:44.810 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-13T00:27:45.134 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-13T00:27:45.223 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:18 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:27:45.367 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-13T00:27:46.399 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-13T00:27:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-13T00:27:46.719 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:27:46.720 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:46.720 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-02-13T00:27:46.720 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:46.720 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:27:46.720 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 131 k 2024-02-13T00:27:46.720 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:46.720 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:27:46.720 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:46.720 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-13T00:27:46.721 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:46.721 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 131 k 2024-02-13T00:27:46.722 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 432 k 2024-02-13T00:27:46.722 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:27:46.834 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-13T00:27:46.917 INFO:teuthology.orchestra.run.smithi162.stdout:librados-devel-16.2.14-521.g0e714d9a.el8.x86_64 669 kB/s | 131 kB 00:00 2024-02-13T00:27:46.917 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:46.918 INFO:teuthology.orchestra.run.smithi162.stdout:Total 665 kB/s | 131 kB 00:00 2024-02-13T00:27:46.918 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:27:46.923 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:27:46.923 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:27:46.975 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:27:46.978 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:27:47.027 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-13T00:27:47.089 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:27:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 13/13 2024-02-13T00:27:47.317 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:27:47.453 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 13/13 2024-02-13T00:27:47.454 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 1/13 2024-02-13T00:27:47.454 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-13T00:27:47.454 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-13T00:27:47.454 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-13T00:27:47.454 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-13T00:27:47.454 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-13T00:27:47.454 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-13T00:27:47.454 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-13T00:27:47.454 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-13T00:27:47.454 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-13T00:27:47.454 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-13T00:27:47.455 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-13T00:27:47.595 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-13T00:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-13T00:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-13T00:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-13T00:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-13T00:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-13T00:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-13T00:27:47.924 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-13T00:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-13T00:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-13T00:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-13T00:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-13T00:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-13T00:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:47.925 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:27:47.996 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:27:47.996 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:47.997 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:27:47.997 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:27:47.997 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:47.997 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:27:48.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-02-13T00:27:48.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs2 2024-02-13T00:27:48.798 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:38 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:27:48.999 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:21 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:27:50.315 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:27:50.316 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:50.316 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-13T00:27:50.316 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:50.316 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:27:50.316 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:16.2.14-521.g0e714d9a.el8 ceph-noarch 116 k 2024-02-13T00:27:50.316 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-13T00:27:50.316 INFO:teuthology.orchestra.run.smithi181.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-13T00:27:50.317 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-13T00:27:50.317 INFO:teuthology.orchestra.run.smithi181.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-13T00:27:50.317 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-13T00:27:50.317 INFO:teuthology.orchestra.run.smithi181.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-13T00:27:50.317 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:50.317 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:27:50.317 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:50.317 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-02-13T00:27:50.317 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:50.318 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 5.8 M 2024-02-13T00:27:50.318 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 24 M 2024-02-13T00:27:50.318 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:27:50.476 INFO:teuthology.orchestra.run.smithi162.stdout:Package libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:27:50.521 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:27:50.522 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-13T00:27:50.522 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:27:51.083 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.4 MB/s | 538 kB 00:00 2024-02-13T00:27:51.125 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): ceph-mgr-cephadm-16.2.14-521.g0e714d9a.e 282 kB/s | 116 kB 00:00 2024-02-13T00:27:51.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs-devel 2024-02-13T00:27:51.334 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.7 MB/s | 4.8 MB 00:00 2024-02-13T00:27:51.426 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 62 kB/s | 21 kB 00:00 2024-02-13T00:27:51.726 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 125 kB/s | 49 kB 00:00 2024-02-13T00:27:51.768 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 553 kB/s | 355 kB 00:00 2024-02-13T00:27:51.771 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:51.771 INFO:teuthology.orchestra.run.smithi181.stdout:Total 4.0 MB/s | 5.8 MB 00:01 2024-02-13T00:27:51.825 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:27:51.835 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:27:51.835 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:27:51.940 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:27:51.943 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:27:51.964 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:24 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:27:52.244 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:27:52.501 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-13T00:27:52.804 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-13T00:27:53.464 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:27:53.465 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:53.465 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-02-13T00:27:53.465 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:53.465 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:27:53.465 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 25 k 2024-02-13T00:27:53.465 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:53.465 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:27:53.465 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:27:53.465 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-13T00:27:53.465 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:53.466 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 25 k 2024-02-13T00:27:53.466 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 107 k 2024-02-13T00:27:53.466 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:27:53.721 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-13T00:27:53.874 INFO:teuthology.orchestra.run.smithi162.stdout:libcephfs-devel-16.2.14-521.g0e714d9a.el8.x86_6 63 kB/s | 25 kB 00:00 2024-02-13T00:27:53.875 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:53.875 INFO:teuthology.orchestra.run.smithi162.stdout:Total 62 kB/s | 25 kB 00:00 2024-02-13T00:27:53.875 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:27:53.879 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:27:53.879 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:27:53.882 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:27:53.885 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:27:53.938 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:27:54.140 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:27:54.393 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-13T00:27:54.416 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:27:54.618 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-13T00:27:54.757 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:27:54.757 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:54.757 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:27:54.757 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:27:54.758 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:27:54.758 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:27:54.818 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 6/6 2024-02-13T00:27:54.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados2 2024-02-13T00:27:55.115 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 6/6 2024-02-13T00:27:55.115 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 1/6 2024-02-13T00:27:55.115 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-13T00:27:55.116 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-13T00:27:55.116 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-13T00:27:55.116 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-13T00:27:55.509 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-13T00:27:55.509 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:55.509 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:27:55.509 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:16.2.14-521.g0e714d9a.el8.noarch 2024-02-13T00:27:55.509 INFO:teuthology.orchestra.run.smithi181.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-13T00:27:55.509 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-13T00:27:55.509 INFO:teuthology.orchestra.run.smithi181.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-13T00:27:55.509 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-02-13T00:27:55.509 INFO:teuthology.orchestra.run.smithi181.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-13T00:27:55.509 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:55.509 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:27:55.633 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:28 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:27:55.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-02-13T00:27:56.395 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:46 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:27:57.170 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:27:57.216 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:27:57.218 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-13T00:27:57.218 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:27:57.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librbd1 2024-02-13T00:27:57.879 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:27:57.879 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:57.880 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-02-13T00:27:57.880 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:57.880 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:27:57.880 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 911 k 2024-02-13T00:27:57.880 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-02-13T00:27:57.880 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-13T00:27:57.880 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:57.880 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:27:57.880 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:27:57.880 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-02-13T00:27:57.880 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:57.881 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 994 k 2024-02-13T00:27:57.881 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 2.8 M 2024-02-13T00:27:57.881 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:27:58.067 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:31 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:27:58.271 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 642 kB/s | 83 kB 00:00 2024-02-13T00:27:58.396 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-16.2.14-521.g0e714d9a.el8.x86_ 3.5 MB/s | 911 kB 00:00 2024-02-13T00:27:58.397 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:27:58.397 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.9 MB/s | 994 kB 00:00 2024-02-13T00:27:58.408 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:27:58.429 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:27:58.429 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:27:58.517 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:27:58.520 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:27:58.698 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:27:58.950 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-13T00:27:59.160 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2/2 2024-02-13T00:27:59.539 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:27:59.569 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2/2 2024-02-13T00:27:59.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/2 2024-02-13T00:27:59.585 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:27:59.586 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-13T00:27:59.586 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:27:59.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rados 2024-02-13T00:27:59.973 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-13T00:27:59.973 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:59.973 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:27:59.974 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-13T00:27:59.974 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:27:59.974 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:00.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-02-13T00:28:00.438 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:33 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:28:00.981 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:50 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:28:01.910 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:28:01.955 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:28:01.957 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-13T00:28:01.957 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:28:02.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rgw 2024-02-13T00:28:02.495 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:28:02.496 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:02.496 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-02-13T00:28:02.496 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:02.496 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:28:02.496 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 131 k 2024-02-13T00:28:02.496 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:02.496 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:28:02.497 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:02.497 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-13T00:28:02.497 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:02.497 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 131 k 2024-02-13T00:28:02.497 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 432 k 2024-02-13T00:28:02.497 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:28:02.803 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:35 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:28:03.019 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-16.2.14-521.g0e714d9a.el8.x86_64 251 kB/s | 131 kB 00:00 2024-02-13T00:28:03.020 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:28:03.020 INFO:teuthology.orchestra.run.smithi181.stdout:Total 250 kB/s | 131 kB 00:00 2024-02-13T00:28:03.020 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:28:03.025 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:28:03.026 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:28:03.076 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:28:03.079 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:28:03.193 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:28:03.391 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:03.680 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:04.020 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:04.020 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:04.020 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:28:04.020 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:28:04.020 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:04.020 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:04.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-02-13T00:28:04.275 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:28:04.322 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:28:04.322 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-13T00:28:04.322 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:28:04.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-cephfs 2024-02-13T00:28:05.013 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:55 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:28:05.180 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:38 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:28:06.475 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:28:06.520 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:28:06.522 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-13T00:28:06.522 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:06.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-02-13T00:28:06.653 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:28:06.699 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:28:06.700 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-13T00:28:06.700 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:28:06.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rbd 2024-02-13T00:28:07.392 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:57 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:28:07.549 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:40 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:28:08.915 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:28:08.916 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-02-13T00:28:08.916 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:08.916 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:28:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 25 k 2024-02-13T00:28:08.916 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:08.916 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:28:08.917 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:08.917 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-13T00:28:08.917 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:08.917 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 25 k 2024-02-13T00:28:08.917 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 107 k 2024-02-13T00:28:08.917 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:28:08.994 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:28:09.039 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:28:09.041 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-13T00:28:09.041 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:28:09.067 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-16.2.14-521.g0e714d9a.el8.x86_6 170 kB/s | 25 kB 00:00 2024-02-13T00:28:09.067 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:28:09.068 INFO:teuthology.orchestra.run.smithi181.stdout:Total 169 kB/s | 25 kB 00:00 2024-02-13T00:28:09.068 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:28:09.072 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:28:09.072 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:28:09.074 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:28:09.077 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:28:09.125 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:28:09.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-fuse 2024-02-13T00:28:09.315 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:09.582 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:09.896 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:42 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:28:09.922 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:09.922 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:09.922 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:28:09.922 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:28:09.922 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:09.922 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:10.047 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-02-13T00:28:10.802 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:03:00 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:28:11.412 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:28:11.413 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:28:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-02-13T00:28:11.413 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:28:11.413 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:28:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 88 k 2024-02-13T00:28:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:28:11.413 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:28:11.413 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:28:11.413 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-13T00:28:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:28:11.414 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 88 k 2024-02-13T00:28:11.414 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 240 k 2024-02-13T00:28:11.414 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:28:11.579 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-fuse-16.2.14-521.g0e714d9a.el8.x86_64.rpm 536 kB/s | 88 kB 00:00 2024-02-13T00:28:11.579 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:28:11.579 INFO:teuthology.orchestra.run.smithi162.stdout:Total 532 kB/s | 88 kB 00:00 2024-02-13T00:28:11.580 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:28:11.589 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:28:11.589 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:28:11.639 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:28:11.642 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:28:11.744 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:28:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:12.267 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:28:12.313 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:28:12.315 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-13T00:28:12.315 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:12.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-02-13T00:28:12.463 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:12.819 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:12.819 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:28:12.819 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:28:12.820 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:28:12.820 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:28:12.820 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:28:13.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-mirror 2024-02-13T00:28:13.184 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:03:03 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:28:13.797 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:46 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:28:14.663 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:28:14.708 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:28:14.709 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-13T00:28:14.710 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:14.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-02-13T00:28:15.280 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:28:15.281 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:28:15.281 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-02-13T00:28:15.281 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:28:15.281 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:28:15.281 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.7 M 2024-02-13T00:28:15.281 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:28:15.281 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:28:15.281 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:28:15.281 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-13T00:28:15.281 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:28:15.281 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 3.7 M 2024-02-13T00:28:15.281 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 15 M 2024-02-13T00:28:15.282 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:28:15.596 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:03:05 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:28:16.250 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-mirror-16.2.14-521.g0e714d9a.el8.x86_64.rpm 3.8 MB/s | 3.7 MB 00:00 2024-02-13T00:28:16.251 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:28:16.251 INFO:teuthology.orchestra.run.smithi162.stdout:Total 3.8 MB/s | 3.7 MB 00:00 2024-02-13T00:28:16.251 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:28:16.262 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:28:16.263 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:28:16.324 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:28:16.328 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:28:16.773 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:28:16.961 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:17.051 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:28:17.097 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:28:17.099 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-13T00:28:17.099 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:17.385 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:17.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-02-13T00:28:17.887 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:17.888 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:28:17.888 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:28:17.888 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:28:17.888 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:28:17.888 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:28:18.137 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-nbd 2024-02-13T00:28:18.275 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:03:08 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:28:18.971 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:51 ago on Tue 13 Feb 2024 12:25:27 AM UTC. 2024-02-13T00:28:19.716 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:28:19.763 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:28:19.764 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-13T00:28:19.764 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:19.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-02-13T00:28:20.480 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-13T00:28:20.481 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:28:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-02-13T00:28:20.481 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:28:20.481 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-13T00:28:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 175 k 2024-02-13T00:28:20.481 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:28:20.481 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-13T00:28:20.481 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-13T00:28:20.482 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-13T00:28:20.482 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:28:20.482 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 175 k 2024-02-13T00:28:20.482 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 523 k 2024-02-13T00:28:20.482 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-13T00:28:20.625 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:03:10 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:28:20.870 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-nbd-16.2.14-521.g0e714d9a.el8.x86_64.rpm 451 kB/s | 175 kB 00:00 2024-02-13T00:28:20.871 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:28:20.871 INFO:teuthology.orchestra.run.smithi162.stdout:Total 450 kB/s | 175 kB 00:00 2024-02-13T00:28:20.871 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-13T00:28:20.882 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-13T00:28:20.882 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-13T00:28:20.931 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-13T00:28:20.934 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-13T00:28:21.043 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-13T00:28:21.259 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:21.552 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:21.938 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:21.939 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:28:21.939 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-13T00:28:21.939 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:28:21.939 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:28:21.939 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-13T00:28:22.103 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:28:22.148 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:28:22.149 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-13T00:28:22.149 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:22.154 DEBUG:teuthology.parallel:result is None 2024-02-13T00:28:22.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-02-13T00:28:23.014 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:03:13 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:28:24.453 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:16.2.14-521.g0e714d9a.el8.x86_64 is already installed. 2024-02-13T00:28:24.499 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:28:24.500 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-02-13T00:28:24.500 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:24.614 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-02-13T00:28:25.365 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:03:15 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:28:26.905 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:28:26.905 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:26.905 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-02-13T00:28:26.905 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:26.905 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:28:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 88 k 2024-02-13T00:28:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:26.906 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:28:26.906 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:26.906 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-13T00:28:26.906 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:26.907 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 88 k 2024-02-13T00:28:26.907 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 240 k 2024-02-13T00:28:26.907 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:28:27.565 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-16.2.14-521.g0e714d9a.el8.x86_64.rpm 134 kB/s | 88 kB 00:00 2024-02-13T00:28:27.566 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:28:27.566 INFO:teuthology.orchestra.run.smithi181.stdout:Total 134 kB/s | 88 kB 00:00 2024-02-13T00:28:27.566 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:28:27.577 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:28:27.577 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:28:27.631 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:28:27.634 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:28:27.973 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:28:28.154 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:28.448 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:28.813 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:28.813 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:28.813 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:28:28.813 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:28:28.813 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:28.813 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:29.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-02-13T00:28:29.779 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:03:19 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:28:31.287 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:28:31.287 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:31.287 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-02-13T00:28:31.288 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:31.288 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:28:31.288 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 3.7 M 2024-02-13T00:28:31.288 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:31.288 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:28:31.288 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:31.288 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-13T00:28:31.288 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:31.289 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.7 M 2024-02-13T00:28:31.289 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 15 M 2024-02-13T00:28:31.289 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:28:31.694 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-16.2.14-521.g0e714d9a.el8.x86_64.rpm 9.1 MB/s | 3.7 MB 00:00 2024-02-13T00:28:31.695 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:28:31.695 INFO:teuthology.orchestra.run.smithi181.stdout:Total 9.0 MB/s | 3.7 MB 00:00 2024-02-13T00:28:31.696 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:28:31.708 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:28:31.708 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:28:31.771 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:28:31.774 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:28:32.212 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:28:32.404 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:32.835 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:33.229 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:33.229 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:33.229 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:28:33.229 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:28:33.229 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:33.229 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:33.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-02-13T00:28:34.227 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:03:24 ago on Tue 13 Feb 2024 12:25:10 AM UTC. 2024-02-13T00:28:35.748 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-02-13T00:28:35.749 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:35.749 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-02-13T00:28:35.749 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:35.749 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-02-13T00:28:35.749 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:16.2.14-521.g0e714d9a.el8 ceph 175 k 2024-02-13T00:28:35.749 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:35.749 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-02-13T00:28:35.749 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-02-13T00:28:35.749 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-02-13T00:28:35.749 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:35.750 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 175 k 2024-02-13T00:28:35.750 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 523 k 2024-02-13T00:28:35.750 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-02-13T00:28:36.257 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-16.2.14-521.g0e714d9a.el8.x86_64.rpm 345 kB/s | 175 kB 00:00 2024-02-13T00:28:36.257 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-02-13T00:28:36.258 INFO:teuthology.orchestra.run.smithi181.stdout:Total 344 kB/s | 175 kB 00:00 2024-02-13T00:28:36.258 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-02-13T00:28:36.271 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-02-13T00:28:36.271 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-02-13T00:28:36.322 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-02-13T00:28:36.325 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-02-13T00:28:36.440 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-02-13T00:28:36.612 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:36.930 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:37.301 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 1/1 2024-02-13T00:28:37.302 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:37.302 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-02-13T00:28:37.302 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:16.2.14-521.g0e714d9a.el8.x86_64 2024-02-13T00:28:37.302 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:28:37.302 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-02-13T00:28:37.531 DEBUG:teuthology.parallel:result is None 2024-02-13T00:28:37.531 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T00:28:37.676 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-13T00:28:37.702 INFO:teuthology.orchestra.run.smithi162.stdout:16.2.14-521.g0e714d9a.el8 2024-02-13T00:28:37.703 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-521.g0e714d9a.el8 2024-02-13T00:28:37.703 INFO:teuthology.task.install:The correct ceph version 16.2.14-521.g0e714d9a is installed. 2024-02-13T00:28:37.705 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T00:28:37.848 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-13T00:28:37.871 INFO:teuthology.orchestra.run.smithi181.stdout:16.2.14-521.g0e714d9a.el8 2024-02-13T00:28:37.871 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-521.g0e714d9a.el8 2024-02-13T00:28:37.872 INFO:teuthology.task.install:The correct ceph version 16.2.14-521.g0e714d9a is installed. 2024-02-13T00:28:37.874 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-13T00:28:37.874 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:28:37.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-13T00:28:37.911 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-13T00:28:37.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-13T00:28:37.940 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-13T00:28:37.940 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:28:37.940 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-02-13T00:28:37.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-13T00:28:38.053 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-13T00:28:38.053 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-02-13T00:28:38.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-13T00:28:38.146 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-13T00:28:38.146 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:28:38.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-13T00:28:38.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-13T00:28:38.244 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-13T00:28:38.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-13T00:28:38.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-13T00:28:38.337 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-13T00:28:38.338 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:28:38.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-02-13T00:28:38.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-13T00:28:38.433 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-13T00:28:38.433 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-02-13T00:28:38.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-13T00:28:38.527 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-13T00:28:38.604 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', 'flag\\(s\\) set', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(PG_', '\\(OSD_', {'mons down': None}, 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'cephadm_mode': 'root'} 2024-02-13T00:28:38.604 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T00:28:38.604 INFO:tasks.cephadm:Cluster fsid is d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:28:38.604 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-13T00:28:38.605 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.162', 'mon.c': '[v2:172.21.15.162:3301,v1:172.21.15.162:6790]', 'mon.b': '172.21.15.181'} 2024-02-13T00:28:38.605 INFO:tasks.cephadm:First mon is mon.a on smithi162 2024-02-13T00:28:38.605 INFO:tasks.cephadm:First mgr is y 2024-02-13T00:28:38.605 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-13T00:28:38.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname $(hostname -s) 2024-02-13T00:28:38.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname $(hostname -s) 2024-02-13T00:28:38.660 INFO:tasks.cephadm:Downloading cephadm (repo https://git.ceph.com/ceph-ci.git ref 0e714d9a4bd2a821113e6318adb87bd06cf81ec1)... 2024-02-13T00:28:38.661 DEBUG:teuthology.orchestra.run.smithi162:> git clone https://git.ceph.com/ceph-ci.git testrepo && cd testrepo && git show 0e714d9a4bd2a821113e6318adb87bd06cf81ec1:src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-13T00:28:38.692 INFO:teuthology.orchestra.run.smithi162.stderr:Cloning into 'testrepo'... 2024-02-13T00:29:49.521 INFO:teuthology.orchestra.run.smithi162.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 13 00:29 /home/ubuntu/cephtest/cephadm 2024-02-13T00:29:49.522 DEBUG:teuthology.orchestra.run.smithi181:> git clone https://git.ceph.com/ceph-ci.git testrepo && cd testrepo && git show 0e714d9a4bd2a821113e6318adb87bd06cf81ec1:src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-13T00:29:49.546 INFO:teuthology.orchestra.run.smithi181.stderr:Cloning into 'testrepo'... 2024-02-13T00:31:18.993 INFO:teuthology.orchestra.run.smithi181.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 13 00:31 /home/ubuntu/cephtest/cephadm 2024-02-13T00:31:18.994 DEBUG:teuthology.orchestra.run.smithi162:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-13T00:31:19.018 DEBUG:teuthology.orchestra.run.smithi181:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-13T00:31:19.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph 2024-02-13T00:31:19.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph 2024-02-13T00:31:19.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /etc/ceph 2024-02-13T00:31:19.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /etc/ceph 2024-02-13T00:31:19.192 INFO:tasks.cephadm:Writing seed config... 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [client] debug objecter = 20 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [client] debug rados = 20 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [global] ms type = async 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [mon] debug mgrc = 20 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-13T00:31:19.194 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-13T00:31:19.195 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-02-13T00:31:19.196 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:31:19.196 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-13T00:31:19.226 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d4ee01fc-ca06-11ee-95b9-87774f69a715 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mgrc = 20 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon warn on pool no app = False paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 debug objecter = 20 debug rados = 20 2024-02-13T00:31:19.226 DEBUG:teuthology.orchestra.run.smithi162:mon.a> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.a.service 2024-02-13T00:31:19.268 DEBUG:teuthology.orchestra.run.smithi162:mgr.y> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mgr.y.service 2024-02-13T00:31:19.295 INFO:journalctl@ceph.mon.a.smithi162.stdout:-- Logs begin at Tue 2024-02-13 00:17:45 UTC. -- 2024-02-13T00:31:19.297 INFO:tasks.cephadm:Bootstrapping... 2024-02-13T00:31:19.297 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -v bootstrap --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.162 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-13T00:31:19.323 INFO:journalctl@ceph.mgr.y.smithi162.stdout:-- Logs begin at Tue 2024-02-13 00:17:45 UTC. -- 2024-02-13T00:31:19.583 INFO:teuthology.orchestra.run.smithi162.stderr:-------------------------------------------------------------------------------- 2024-02-13T00:31:19.583 INFO:teuthology.orchestra.run.smithi162.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1', '-v', 'bootstrap', '--fsid', 'd4ee01fc-ca06-11ee-95b9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.162', '--skip-admin-label'] 2024-02-13T00:31:20.298 INFO:teuthology.orchestra.run.smithi162.stderr:/bin/podman: 4.8.3 2024-02-13T00:31:20.306 INFO:teuthology.orchestra.run.smithi162.stderr:Verifying podman|docker is present... 2024-02-13T00:31:20.326 INFO:teuthology.orchestra.run.smithi162.stderr:/bin/podman: 4.8.3 2024-02-13T00:31:20.335 INFO:teuthology.orchestra.run.smithi162.stderr:Verifying lvm2 is present... 2024-02-13T00:31:20.335 INFO:teuthology.orchestra.run.smithi162.stderr:Verifying time synchronization is in place... 2024-02-13T00:31:20.342 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-13T00:31:20.348 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: inactive 2024-02-13T00:31:20.354 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: enabled 2024-02-13T00:31:20.359 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: active 2024-02-13T00:31:20.360 INFO:teuthology.orchestra.run.smithi162.stderr:Unit chronyd.service is enabled and running 2024-02-13T00:31:20.360 INFO:teuthology.orchestra.run.smithi162.stderr:Repeating the final host check... 2024-02-13T00:31:20.380 INFO:teuthology.orchestra.run.smithi162.stderr:/bin/podman: 4.8.3 2024-02-13T00:31:20.389 INFO:teuthology.orchestra.run.smithi162.stderr:podman (/bin/podman) version 4.8.3 is present 2024-02-13T00:31:20.389 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl is present 2024-02-13T00:31:20.389 INFO:teuthology.orchestra.run.smithi162.stderr:lvcreate is present 2024-02-13T00:31:20.395 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-13T00:31:20.400 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: inactive 2024-02-13T00:31:20.406 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: enabled 2024-02-13T00:31:20.412 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: active 2024-02-13T00:31:20.412 INFO:teuthology.orchestra.run.smithi162.stderr:Unit chronyd.service is enabled and running 2024-02-13T00:31:20.412 INFO:teuthology.orchestra.run.smithi162.stderr:Host looks OK 2024-02-13T00:31:20.412 INFO:teuthology.orchestra.run.smithi162.stderr:Cluster fsid: d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:31:20.413 INFO:teuthology.orchestra.run.smithi162.stderr:Acquiring lock 140698848287208 on /run/cephadm/d4ee01fc-ca06-11ee-95b9-87774f69a715.lock 2024-02-13T00:31:20.413 INFO:teuthology.orchestra.run.smithi162.stderr:Lock 140698848287208 acquired on /run/cephadm/d4ee01fc-ca06-11ee-95b9-87774f69a715.lock 2024-02-13T00:31:20.413 INFO:teuthology.orchestra.run.smithi162.stderr:Verifying IP 172.21.15.162 port 3300 ... 2024-02-13T00:31:20.413 INFO:teuthology.orchestra.run.smithi162.stderr:Verifying IP 172.21.15.162 port 6789 ... 2024-02-13T00:31:20.413 INFO:teuthology.orchestra.run.smithi162.stderr:Base mon IP(s) is [172.21.15.162:3300, 172.21.15.162:6789], mon addrv is [v2:172.21.15.162:3300,v1:172.21.15.162:6789] 2024-02-13T00:31:20.417 INFO:teuthology.orchestra.run.smithi162.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.162 metric 100 2024-02-13T00:31:20.417 INFO:teuthology.orchestra.run.smithi162.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.162 metric 100 2024-02-13T00:31:20.420 INFO:teuthology.orchestra.run.smithi162.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-13T00:31:20.420 INFO:teuthology.orchestra.run.smithi162.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-13T00:31:20.420 INFO:teuthology.orchestra.run.smithi162.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1793sec hoplimit 64 pref medium 2024-02-13T00:31:20.422 INFO:teuthology.orchestra.run.smithi162.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-13T00:31:20.422 INFO:teuthology.orchestra.run.smithi162.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-13T00:31:20.423 INFO:teuthology.orchestra.run.smithi162.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-13T00:31:20.423 INFO:teuthology.orchestra.run.smithi162.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-13T00:31:20.423 INFO:teuthology.orchestra.run.smithi162.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7f41/64 scope link 2024-02-13T00:31:20.423 INFO:teuthology.orchestra.run.smithi162.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-13T00:31:20.423 INFO:teuthology.orchestra.run.smithi162.stderr:Mon IP `172.21.15.162` is in CIDR network `172.21.0.0/20` 2024-02-13T00:31:20.423 INFO:teuthology.orchestra.run.smithi162.stderr:Mon IP `172.21.15.162` is in CIDR network `172.21.0.0/20` 2024-02-13T00:31:20.424 INFO:teuthology.orchestra.run.smithi162.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-13T00:31:20.424 INFO:teuthology.orchestra.run.smithi162.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-13T00:31:20.425 INFO:teuthology.orchestra.run.smithi162.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1... 2024-02-13T00:31:20.522 INFO:teuthology.orchestra.run.smithi162.stderr:/bin/podman: Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1... 2024-02-13T00:31:20.733 INFO:teuthology.orchestra.run.smithi162.stderr:/bin/podman: Getting image source signatures 2024-02-13T00:31:20.739 INFO:teuthology.orchestra.run.smithi162.stderr:/bin/podman: Copying blob sha256:955bfa9dba4a4380f93be7209d227ce2b057a09dc4ade787f31ef12d8b977cea 2024-02-13T00:31:20.739 INFO:teuthology.orchestra.run.smithi162.stderr:/bin/podman: Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-13T00:31:42.256 INFO:teuthology.orchestra.run.smithi162.stderr:/bin/podman: Copying config sha256:1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5 2024-02-13T00:31:42.465 INFO:teuthology.orchestra.run.smithi162.stderr:/bin/podman: Writing manifest to image destination 2024-02-13T00:31:42.718 INFO:teuthology.orchestra.run.smithi162.stderr:/bin/podman: 1fbc92c859dc0947ffca020b41c01a6ac9859691b5303dd3e8b8634d23f540d5 2024-02-13T00:31:43.363 INFO:teuthology.orchestra.run.smithi162.stderr:ceph: ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable) 2024-02-13T00:31:43.974 INFO:teuthology.orchestra.run.smithi162.stderr:Ceph version: ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable) 2024-02-13T00:31:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:Extracting ceph user uid/gid from container image... 2024-02-13T00:31:44.502 INFO:teuthology.orchestra.run.smithi162.stderr:stat: 167 167 2024-02-13T00:31:45.028 INFO:teuthology.orchestra.run.smithi162.stderr:Creating initial keys... 2024-02-13T00:31:45.491 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-authtool: AQBxuMplzbpVFxAAOtgNculykx4kM2erowa/7A== 2024-02-13T00:31:46.614 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-authtool: AQByuMpl1/6yHxAAZAzg8oTHbpZjH2PierqJ2w== 2024-02-13T00:31:47.729 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-authtool: AQBzuMplEyqHJRAAAP9b0huOq3FZetWzQWpW1w== 2024-02-13T00:31:48.317 INFO:teuthology.orchestra.run.smithi162.stderr:Creating initial monmap... 2024-02-13T00:31:48.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-13T00:31:48.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:31:48.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-13T00:31:49.356 INFO:teuthology.orchestra.run.smithi162.stderr:monmaptool for a [v2:172.21.15.162:3300,v1:172.21.15.162:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-13T00:31:49.356 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/monmaptool: set fsid to d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:31:49.356 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-13T00:31:49.356 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-02-13T00:31:49.356 INFO:teuthology.orchestra.run.smithi162.stderr:Creating mon... 2024-02-13T00:31:49.858 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.779+0000 7f8239594880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-13T00:31:49.858 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.779+0000 7f8239594880 1 imported monmap: 2024-02-13T00:31:49.858 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-13T00:31:49.858 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:31:49.858 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: last_changed 2024-02-13T00:31:48.749492+0000 2024-02-13T00:31:49.858 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: created 2024-02-13T00:31:48.749492+0000 2024-02-13T00:31:49.858 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-13T00:31:49.858 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-13T00:31:49.858 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.a 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.779+0000 7f8239594880 0 /usr/bin/ceph-mon: set fsid to d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: RocksDB version: 6.8.1 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Compile date Feb 8 2024 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: DB SUMMARY 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.error_if_exists: 0 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.create_if_missing: 1 2024-02-13T00:31:49.859 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.paranoid_checks: 1 2024-02-13T00:31:49.860 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.env: 0x5607ab6a0080 2024-02-13T00:31:49.860 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.fs: Posix File System 2024-02-13T00:31:49.860 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.info_log: 0x5607acbc5a80 2024-02-13T00:31:49.860 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-13T00:31:49.860 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.statistics: (nil) 2024-02-13T00:31:49.860 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.use_fsync: 0 2024-02-13T00:31:49.860 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.max_log_file_size: 0 2024-02-13T00:31:49.860 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-13T00:31:49.860 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-13T00:31:49.860 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-13T00:31:49.860 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.allow_fallocate: 1 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.use_direct_reads: 0 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.db_log_dir: 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.max_subcompactions: 1 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.max_background_flushes: -1 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-13T00:31:49.861 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.write_buffer_manager: 0x5607acbcb950 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-13T00:31:49.862 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.unordered_write: 0 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.row_cache: None 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.wal_filter: None 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.preserve_deletes: 0 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.two_write_queues: 0 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-13T00:31:49.863 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.atomic_flush: 0 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.log_readahead_size: 0 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.max_background_jobs: 2 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.max_background_compactions: -1 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-13T00:31:49.864 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.max_open_files: -1 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Compression algorithms supported: 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: kZSTD supported: 0 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: kXpressCompression supported: 0 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: kLZ4Compression supported: 1 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: kBZip2Compression supported: 0 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: kZlibCompression supported: 1 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: kSnappyCompression supported: 1 2024-02-13T00:31:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.780+0000 7f8239594880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.merge_operator: 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compaction_filter: None 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5607acadbd30) 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-13T00:31:49.866 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-13T00:31:49.867 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-13T00:31:49.867 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-13T00:31:49.867 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-13T00:31:49.867 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-13T00:31:49.867 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-13T00:31:49.867 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-13T00:31:49.867 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-13T00:31:49.867 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-13T00:31:49.867 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-13T00:31:49.867 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: block_cache: 0x5607acb10d10 2024-02-13T00:31:49.867 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-13T00:31:49.867 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-13T00:31:49.868 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compression: NoCompression 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-13T00:31:49.869 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.num_levels: 7 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-13T00:31:49.870 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-13T00:31:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-13T00:31:49.872 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.table_properties_collectors: 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.inplace_update_support: 0 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-13T00:31:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.bloom_locality: 0 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.max_successive_merges: 0 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.ttl: 2592000 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.781+0000 7f8239594880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.782+0000 7f8239594880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.782+0000 7f8239594880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.782+0000 7f8239594880 4 rocksdb: DB pointer 0x5607acbd9800 2024-02-13T00:31:49.874 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.782+0000 7f8222406700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.782+0000 7f8222406700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T00:31:49.875 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T00:31:49.876 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-13T00:31:49.876 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-13T00:31:49.876 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.876 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-13T00:31:49.876 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T00:31:49.876 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T00:31:49.876 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T00:31:49.876 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T00:31:49.876 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T00:31:49.877 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T00:31:49.877 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T00:31:49.877 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-13T00:31:49.877 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T00:31:49.877 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T00:31:49.877 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T00:31:49.877 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.878 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-13T00:31:49.878 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.878 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-13T00:31:49.878 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T00:31:49.878 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T00:31:49.878 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-13T00:31:49.878 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-13T00:31:49.878 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.878 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-13T00:31:49.878 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T00:31:49.879 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T00:31:49.879 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T00:31:49.879 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T00:31:49.879 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T00:31:49.879 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T00:31:49.879 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T00:31:49.879 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-13T00:31:49.880 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T00:31:49.880 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T00:31:49.880 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T00:31:49.880 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.880 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-13T00:31:49.880 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: 2024-02-13T00:31:49.880 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.784+0000 7f8239594880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-13T00:31:49.880 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.784+0000 7f8239594880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-13T00:31:49.880 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph-mon: debug 2024-02-13T00:31:49.784+0000 7f8239594880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-02-13T00:31:50.439 INFO:teuthology.orchestra.run.smithi162.stderr:create mon.a on 2024-02-13T00:31:50.452 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-13T00:31:50.544 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-13T00:31:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715.target. 2024-02-13T00:31:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715.target. 2024-02-13T00:31:50.836 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: Failed to reset failed state of unit ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.a.service: Unit ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.a.service not loaded. 2024-02-13T00:31:50.843 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: Created symlink /etc/systemd/system/ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715.target.wants/ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@.service. 2024-02-13T00:31:51.862 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: disabled 2024-02-13T00:31:51.869 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: inactive 2024-02-13T00:31:51.870 INFO:teuthology.orchestra.run.smithi162.stderr:firewalld.service is not enabled 2024-02-13T00:31:51.870 INFO:teuthology.orchestra.run.smithi162.stderr:Not possible to enable service . firewalld.service is not available 2024-02-13T00:31:51.870 INFO:teuthology.orchestra.run.smithi162.stderr:Waiting for mon to start... 2024-02-13T00:31:51.870 INFO:teuthology.orchestra.run.smithi162.stderr:Waiting for mon... 2024-02-13T00:31:52.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:51 smithi162 bash[39090]: 233c7f25c51df5aedf74e845ef26c7fc5a0b60f26fde087a87531123e9ebfa5b 2024-02-13T00:31:52.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:51 smithi162 systemd[1]: Started Ceph mon.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:31:52.430 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.423+0000 7fb598ab2700 1 Processor -- start 2024-02-13T00:31:52.430 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.423+0000 7fb598ab2700 1 -- start start 2024-02-13T00:31:52.430 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.424+0000 7fb598ab2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 0x7fb5940fc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:31:52.430 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.424+0000 7fb598ab2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5940fca20 con 0x7fb5940fc110 2024-02-13T00:31:52.430 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.425+0000 7fb59259c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 0x7fb5940fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:31:52.430 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.425+0000 7fb59259c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 0x7fb5940fc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46950/0 (socket says 172.21.15.162:46950) 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.425+0000 7fb59259c700 1 -- 172.21.15.162:0/2717114301 learned_addr learned my addr 172.21.15.162:0/2717114301 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.425+0000 7fb59259c700 1 -- 172.21.15.162:0/2717114301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5940fcb60 con 0x7fb5940fc110 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.425+0000 7fb59259c700 1 --2- 172.21.15.162:0/2717114301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 0x7fb5940fc530 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb588009730 tx=0x7fb5880090f0).ready entity=mon.0 client_cookie=5be4e0f62f488f9d server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.425+0000 7fb59159a700 1 -- 172.21.15.162:0/2717114301 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb58800b580 con 0x7fb5940fc110 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.425+0000 7fb59159a700 1 -- 172.21.15.162:0/2717114301 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58800b6e0 con 0x7fb5940fc110 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.425+0000 7fb59159a700 1 -- 172.21.15.162:0/2717114301 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb58800b990 con 0x7fb5940fc110 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.426+0000 7fb598ab2700 1 -- 172.21.15.162:0/2717114301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 msgr2=0x7fb5940fc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.426+0000 7fb598ab2700 1 --2- 172.21.15.162:0/2717114301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 0x7fb5940fc530 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb588009730 tx=0x7fb5880090f0).stop 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.426+0000 7fb598ab2700 1 -- 172.21.15.162:0/2717114301 shutdown_connections 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.426+0000 7fb598ab2700 1 --2- 172.21.15.162:0/2717114301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 0x7fb5940fc530 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.426+0000 7fb598ab2700 1 -- 172.21.15.162:0/2717114301 >> 172.21.15.162:0/2717114301 conn(0x7fb5940f7930 msgr2=0x7fb5940f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.426+0000 7fb598ab2700 1 -- 172.21.15.162:0/2717114301 shutdown_connections 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.426+0000 7fb598ab2700 1 -- 172.21.15.162:0/2717114301 wait complete. 2024-02-13T00:31:52.431 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.426+0000 7fb598ab2700 1 librados: starting msgr at 2024-02-13T00:31:52.432 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.426+0000 7fb598ab2700 1 librados: starting objecter 2024-02-13T00:31:52.434 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb598ab2700 1 Processor -- start 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb598ab2700 1 -- start start 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb598ab2700 1 librados: setting wanted keys 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb598ab2700 1 librados: calling monclient init 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb598ab2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 0x7fb59410a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb598ab2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59410aa40 con 0x7fb5940fc110 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb59259c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 0x7fb59410a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb59259c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 0x7fb59410a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46952/0 (socket says 172.21.15.162:46952) 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb59259c700 1 -- 172.21.15.162:0/1199584877 learned_addr learned my addr 172.21.15.162:0/1199584877 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb59259c700 1 -- 172.21.15.162:0/1199584877 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb588009370 con 0x7fb5940fc110 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb59259c700 1 --2- 172.21.15.162:0/1199584877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 0x7fb59410a550 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb588009130 tx=0x7fb588017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb5837fe700 10 client.?.objecter ms_handle_connect 0x7fb5940fc110 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.427+0000 7fb5837fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:31:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb5837fe700 1 -- 172.21.15.162:0/1199584877 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb588011cd0 con 0x7fb5940fc110 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb5837fe700 1 -- 172.21.15.162:0/1199584877 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb588011e30 con 0x7fb5940fc110 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb5837fe700 1 -- 172.21.15.162:0/1199584877 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb58801d570 con 0x7fb5940fc110 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb598ab2700 1 -- 172.21.15.162:0/1199584877 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59410ac40 con 0x7fb5940fc110 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb598ab2700 10 client.4098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb598ab2700 1 -- 172.21.15.162:0/1199584877 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59410b0d0 con 0x7fb5940fc110 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb598ab2700 1 librados: init done 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb5837fe700 1 -- 172.21.15.162:0/1199584877 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7fb58801da00 con 0x7fb5940fc110 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb5837fe700 1 -- 172.21.15.162:0/1199584877 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb588026710 con 0x7fb5940fc110 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb5837fe700 10 client.4098.objecter ms_dispatch 0x7fb594001550 osd_map(1..1 src has 1..1) v4 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb5837fe700 3 client.4098.objecter handle_osd_map got epochs [1,1] > 0 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb5837fe700 3 client.4098.objecter handle_osd_map decoding full epoch 1 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.428+0000 7fb5837fe700 20 client.4098.objecter dump_active .. 0 homeless 2024-02-13T00:31:52.436 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.429+0000 7fb598ab2700 1 -- 172.21.15.162:0/1199584877 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb594046d80 con 0x7fb5940fc110 2024-02-13T00:31:52.437 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.431+0000 7fb5837fe700 1 -- 172.21.15.162:0/1199584877 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7fb588021030 con 0x7fb5940fc110 2024-02-13T00:31:52.504 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.497+0000 7fb598ab2700 1 -- 172.21.15.162:0/1199584877 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fb594103a80 con 0x7fb5940fc110 2024-02-13T00:31:52.504 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.498+0000 7fb5837fe700 1 -- 172.21.15.162:0/1199584877 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fb588026950 con 0x7fb5940fc110 2024-02-13T00:31:52.504 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: cluster: 2024-02-13T00:31:52.504 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: id: d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:31:52.504 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-13T00:31:52.504 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.504 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: services: 2024-02-13T00:31:52.504 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.704015s) 2024-02-13T00:31:52.504 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-13T00:31:52.504 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: data: 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: pgs: 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.498+0000 7fb598ab2700 10 librados: watch_flush enter 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.498+0000 7fb598ab2700 10 librados: watch_flush exit 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.498+0000 7fb598ab2700 20 client.4098.objecter shutdown clearing up homeless session... 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.498+0000 7fb598ab2700 10 client.4098.objecter successfully canceled tick 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.498+0000 7fb598ab2700 1 -- 172.21.15.162:0/1199584877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 msgr2=0x7fb59410a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.499+0000 7fb598ab2700 1 --2- 172.21.15.162:0/1199584877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 0x7fb59410a550 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb588009130 tx=0x7fb588017000).stop 2024-02-13T00:31:52.505 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.499+0000 7fb598ab2700 1 -- 172.21.15.162:0/1199584877 shutdown_connections 2024-02-13T00:31:52.506 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.499+0000 7fb598ab2700 1 --2- 172.21.15.162:0/1199584877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5940fc110 0x7fb59410a550 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:31:52.506 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.499+0000 7fb598ab2700 1 -- 172.21.15.162:0/1199584877 >> 172.21.15.162:0/1199584877 conn(0x7fb5940f7930 msgr2=0x7fb5940fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:31:52.506 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.499+0000 7fb598ab2700 1 -- 172.21.15.162:0/1199584877 shutdown_connections 2024-02-13T00:31:52.506 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.499+0000 7fb598ab2700 1 -- 172.21.15.162:0/1199584877 wait complete. 2024-02-13T00:31:52.506 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:52.499+0000 7fb598ab2700 1 librados: shutdown 2024-02-13T00:31:53.060 INFO:teuthology.orchestra.run.smithi162.stderr:mon is available 2024-02-13T00:31:53.060 INFO:teuthology.orchestra.run.smithi162.stderr:Assimilating anything we can from ceph.conf... 2024-02-13T00:31:53.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39105]: cluster 2024-02-13T00:31:51.795759+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-13T00:31:53.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39105]: cluster 2024-02-13T00:31:51.796122+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0]} removed_ranks: {} 2024-02-13T00:31:53.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39105]: cluster 2024-02-13T00:31:51.799266+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-02-13T00:31:53.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39105]: cluster 2024-02-13T00:31:51.801890+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-13T00:31:53.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39105]: cluster 2024-02-13T00:31:51.802355+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-13T00:31:53.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39105]: audit 2024-02-13T00:31:52.499712+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.162:0/1199584877' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-13T00:31:53.563 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.557+0000 7fb8dcb96700 1 Processor -- start 2024-02-13T00:31:53.563 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.557+0000 7fb8dcb96700 1 -- start start 2024-02-13T00:31:53.563 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.557+0000 7fb8dcb96700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 0x7fb8d8101170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:31:53.563 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.557+0000 7fb8dcb96700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d80fcb60 con 0x7fb8d8070bc0 2024-02-13T00:31:53.563 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.557+0000 7fb8d659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 0x7fb8d8101170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:31:53.563 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.557+0000 7fb8d659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 0x7fb8d8101170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46964/0 (socket says 172.21.15.162:46964) 2024-02-13T00:31:53.563 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.557+0000 7fb8d659c700 1 -- 172.21.15.162:0/1635348960 learned_addr learned my addr 172.21.15.162:0/1635348960 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:31:53.563 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.557+0000 7fb8d659c700 1 -- 172.21.15.162:0/1635348960 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d8101660 con 0x7fb8d8070bc0 2024-02-13T00:31:53.563 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.557+0000 7fb8d659c700 1 --2- 172.21.15.162:0/1635348960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 0x7fb8d8101170 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb8cc009730 tx=0x7fb8cc0090f0).ready entity=mon.0 client_cookie=f1af2a067f27436c server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:31:53.564 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.558+0000 7fb8d559a700 1 -- 172.21.15.162:0/1635348960 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8cc00b580 con 0x7fb8d8070bc0 2024-02-13T00:31:53.564 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.558+0000 7fb8d559a700 1 -- 172.21.15.162:0/1635348960 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc00b6e0 con 0x7fb8d8070bc0 2024-02-13T00:31:53.564 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.558+0000 7fb8d559a700 1 -- 172.21.15.162:0/1635348960 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8cc00b990 con 0x7fb8d8070bc0 2024-02-13T00:31:53.564 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.558+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1635348960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 msgr2=0x7fb8d8101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:31:53.564 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.558+0000 7fb8dcb96700 1 --2- 172.21.15.162:0/1635348960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 0x7fb8d8101170 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb8cc009730 tx=0x7fb8cc0090f0).stop 2024-02-13T00:31:53.564 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.558+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1635348960 shutdown_connections 2024-02-13T00:31:53.564 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.558+0000 7fb8dcb96700 1 --2- 172.21.15.162:0/1635348960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 0x7fb8d8101170 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:31:53.564 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.558+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1635348960 >> 172.21.15.162:0/1635348960 conn(0x7fb8d80f6770 msgr2=0x7fb8d80f8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:31:53.564 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.558+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1635348960 shutdown_connections 2024-02-13T00:31:53.564 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.558+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1635348960 wait complete. 2024-02-13T00:31:53.564 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.559+0000 7fb8dcb96700 1 librados: starting msgr at 2024-02-13T00:31:53.564 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.559+0000 7fb8dcb96700 1 librados: starting objecter 2024-02-13T00:31:53.566 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.559+0000 7fb8dcb96700 1 Processor -- start 2024-02-13T00:31:53.566 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.559+0000 7fb8dcb96700 1 -- start start 2024-02-13T00:31:53.566 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.559+0000 7fb8dcb96700 1 librados: setting wanted keys 2024-02-13T00:31:53.566 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.559+0000 7fb8dcb96700 1 librados: calling monclient init 2024-02-13T00:31:53.566 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8dcb96700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 0x7fb8d810c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:31:53.566 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8dcb96700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d810ccc0 con 0x7fb8d8070bc0 2024-02-13T00:31:53.566 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8d659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 0x7fb8d810c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:31:53.566 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8d659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 0x7fb8d810c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46970/0 (socket says 172.21.15.162:46970) 2024-02-13T00:31:53.566 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8d659c700 1 -- 172.21.15.162:0/1639870950 learned_addr learned my addr 172.21.15.162:0/1639870950 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8d659c700 1 -- 172.21.15.162:0/1639870950 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8cc009370 con 0x7fb8d8070bc0 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8d659c700 1 --2- 172.21.15.162:0/1639870950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 0x7fb8d810c7d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fb8cc009130 tx=0x7fb8cc017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8c77fe700 10 client.?.objecter ms_handle_connect 0x7fb8d8070bc0 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8c77fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8c77fe700 1 -- 172.21.15.162:0/1639870950 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8cc011cd0 con 0x7fb8d8070bc0 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8c77fe700 1 -- 172.21.15.162:0/1639870950 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc011e30 con 0x7fb8d8070bc0 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8c77fe700 1 -- 172.21.15.162:0/1639870950 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8cc01d570 con 0x7fb8d8070bc0 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1639870950 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d810cec0 con 0x7fb8d8070bc0 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8dcb96700 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1639870950 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d810d2d0 con 0x7fb8d8070bc0 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.560+0000 7fb8dcb96700 1 librados: init done 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.561+0000 7fb8c77fe700 1 -- 172.21.15.162:0/1639870950 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7fb8cc01da00 con 0x7fb8d8070bc0 2024-02-13T00:31:53.567 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.561+0000 7fb8c77fe700 1 -- 172.21.15.162:0/1639870950 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb8cc026710 con 0x7fb8d8070bc0 2024-02-13T00:31:53.568 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.561+0000 7fb8c77fe700 10 client.4100.objecter ms_dispatch 0x7fb8d8001550 osd_map(1..1 src has 1..1) v4 2024-02-13T00:31:53.568 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.561+0000 7fb8c77fe700 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-02-13T00:31:53.568 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.561+0000 7fb8c77fe700 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-02-13T00:31:53.568 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.561+0000 7fb8c77fe700 20 client.4100.objecter dump_active .. 0 homeless 2024-02-13T00:31:53.568 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.561+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1639870950 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d805a3b0 con 0x7fb8d8070bc0 2024-02-13T00:31:53.568 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.564+0000 7fb8c77fe700 1 -- 172.21.15.162:0/1639870950 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7fb8cc021030 con 0x7fb8d8070bc0 2024-02-13T00:31:53.631 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.627+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1639870950 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb8d8046d80 con 0x7fb8d8070bc0 2024-02-13T00:31:53.637 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.630+0000 7fb8c77fe700 1 -- 172.21.15.162:0/1639870950 <== mon.0 v2:172.21.15.162:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8cc01de90 con 0x7fb8d8070bc0 2024-02-13T00:31:53.637 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.630+0000 7fb8c77fe700 1 -- 172.21.15.162:0/1639870950 <== mon.0 v2:172.21.15.162:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+365 (secure 0 0 0) 0x7fb8cc0269f0 con 0x7fb8d8070bc0 2024-02-13T00:31:53.637 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.637 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: [global] 2024-02-13T00:31:53.637 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: fsid = d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:31:53.637 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-13T00:31:53.637 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.162:3300,v1:172.21.15.162:6789] 2024-02-13T00:31:53.637 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-13T00:31:53.637 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-13T00:31:53.637 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-13T00:31:53.637 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-13T00:31:53.637 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: [mgr] 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: [osd] 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.631+0000 7fb8dcb96700 10 librados: watch_flush enter 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.631+0000 7fb8dcb96700 10 librados: watch_flush exit 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.631+0000 7fb8dcb96700 20 client.4100.objecter shutdown clearing up homeless session... 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.631+0000 7fb8dcb96700 10 client.4100.objecter successfully canceled tick 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.631+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1639870950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 msgr2=0x7fb8d810c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.631+0000 7fb8dcb96700 1 --2- 172.21.15.162:0/1639870950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 0x7fb8d810c7d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fb8cc009130 tx=0x7fb8cc017000).stop 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.631+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1639870950 shutdown_connections 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.631+0000 7fb8dcb96700 1 --2- 172.21.15.162:0/1639870950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d8070bc0 0x7fb8d810c7d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.631+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1639870950 >> 172.21.15.162:0/1639870950 conn(0x7fb8d80f6770 msgr2=0x7fb8d80f8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:31:53.638 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.631+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1639870950 shutdown_connections 2024-02-13T00:31:53.639 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.631+0000 7fb8dcb96700 1 -- 172.21.15.162:0/1639870950 wait complete. 2024-02-13T00:31:53.639 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:53.631+0000 7fb8dcb96700 1 librados: shutdown 2024-02-13T00:31:54.017 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39105]: audit 2024-02-13T00:31:53.629415+0000 mon.a (mon.0) 8 : audit [INF] from='client.? 172.21.15.162:0/1639870950' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-13T00:31:54.018 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39105]: audit 2024-02-13T00:31:53.632387+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.162:0/1639870950' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-13T00:31:54.282 INFO:teuthology.orchestra.run.smithi162.stderr:Generating new minimal ceph.conf... 2024-02-13T00:31:54.817 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.810+0000 7f18665fd700 1 Processor -- start 2024-02-13T00:31:54.817 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.811+0000 7f18665fd700 1 -- start start 2024-02-13T00:31:54.817 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.811+0000 7f18665fd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 0x7f18600fede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:31:54.817 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.811+0000 7f18665fd700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18600ff2d0 con 0x7f1860070c00 2024-02-13T00:31:54.817 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.811+0000 7f185ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 0x7f18600fede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.811+0000 7f185ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 0x7f18600fede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59572/0 (socket says 172.21.15.162:59572) 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.811+0000 7f185ffff700 1 -- 172.21.15.162:0/4102191400 learned_addr learned my addr 172.21.15.162:0/4102191400 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.811+0000 7f185ffff700 1 -- 172.21.15.162:0/4102191400 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18600ff410 con 0x7f1860070c00 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.811+0000 7f185ffff700 1 --2- 172.21.15.162:0/4102191400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 0x7f18600fede0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1850009730 tx=0x7f18500090f0).ready entity=mon.0 client_cookie=b8b4f5317dc9ff76 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.811+0000 7f185effd700 1 -- 172.21.15.162:0/4102191400 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f185000b580 con 0x7f1860070c00 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.812+0000 7f185effd700 1 -- 172.21.15.162:0/4102191400 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f185000bb90 con 0x7f1860070c00 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.812+0000 7f185effd700 1 -- 172.21.15.162:0/4102191400 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1850011c30 con 0x7f1860070c00 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.812+0000 7f18665fd700 1 -- 172.21.15.162:0/4102191400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 msgr2=0x7f18600fede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.812+0000 7f18665fd700 1 --2- 172.21.15.162:0/4102191400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 0x7f18600fede0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1850009730 tx=0x7f18500090f0).stop 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.812+0000 7f18665fd700 1 -- 172.21.15.162:0/4102191400 shutdown_connections 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.812+0000 7f18665fd700 1 --2- 172.21.15.162:0/4102191400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 0x7f18600fede0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.812+0000 7f18665fd700 1 -- 172.21.15.162:0/4102191400 >> 172.21.15.162:0/4102191400 conn(0x7f18600f6820 msgr2=0x7f18600f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.812+0000 7f18665fd700 1 -- 172.21.15.162:0/4102191400 shutdown_connections 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.812+0000 7f18665fd700 1 -- 172.21.15.162:0/4102191400 wait complete. 2024-02-13T00:31:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.813+0000 7f18665fd700 1 librados: starting msgr at 2024-02-13T00:31:54.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.813+0000 7f18665fd700 1 librados: starting objecter 2024-02-13T00:31:54.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.813+0000 7f18665fd700 1 Processor -- start 2024-02-13T00:31:54.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.813+0000 7f18665fd700 1 -- start start 2024-02-13T00:31:54.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.813+0000 7f18665fd700 1 librados: setting wanted keys 2024-02-13T00:31:54.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.813+0000 7f18665fd700 1 librados: calling monclient init 2024-02-13T00:31:54.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.813+0000 7f18665fd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 0x7f186010d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:31:54.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.813+0000 7f18665fd700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186010d610 con 0x7f1860070c00 2024-02-13T00:31:54.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.813+0000 7f185ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 0x7f186010d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:31:54.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.813+0000 7f185ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 0x7f186010d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59586/0 (socket says 172.21.15.162:59586) 2024-02-13T00:31:54.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.813+0000 7f185ffff700 1 -- 172.21.15.162:0/935080022 learned_addr learned my addr 172.21.15.162:0/935080022 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:31:54.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.813+0000 7f185ffff700 1 -- 172.21.15.162:0/935080022 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1850009370 con 0x7f1860070c00 2024-02-13T00:31:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f185ffff700 1 --2- 172.21.15.162:0/935080022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 0x7f186010d120 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1850011630 tx=0x7f18500090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:31:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f185d7fa700 10 client.?.objecter ms_handle_connect 0x7f1860070c00 2024-02-13T00:31:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f185d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:31:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f185d7fa700 1 -- 172.21.15.162:0/935080022 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f185001a430 con 0x7f1860070c00 2024-02-13T00:31:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f185d7fa700 1 -- 172.21.15.162:0/935080022 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f185001aa40 con 0x7f1860070c00 2024-02-13T00:31:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f18665fd700 1 -- 172.21.15.162:0/935080022 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186010d810 con 0x7f1860070c00 2024-02-13T00:31:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f185d7fa700 1 -- 172.21.15.162:0/935080022 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18500239a0 con 0x7f1860070c00 2024-02-13T00:31:54.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f18665fd700 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:31:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f18665fd700 1 -- 172.21.15.162:0/935080022 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186010dca0 con 0x7f1860070c00 2024-02-13T00:31:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f18665fd700 1 librados: init done 2024-02-13T00:31:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f185d7fa700 1 -- 172.21.15.162:0/935080022 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f185002c3f0 con 0x7f1860070c00 2024-02-13T00:31:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f185d7fa700 1 -- 172.21.15.162:0/935080022 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1850022730 con 0x7f1860070c00 2024-02-13T00:31:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f185d7fa700 10 client.4102.objecter ms_dispatch 0x7f1860001550 osd_map(1..1 src has 1..1) v4 2024-02-13T00:31:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f185d7fa700 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-02-13T00:31:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f185d7fa700 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-02-13T00:31:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.814+0000 7f185d7fa700 20 client.4102.objecter dump_active .. 0 homeless 2024-02-13T00:31:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.815+0000 7f18665fd700 1 -- 172.21.15.162:0/935080022 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1860046d80 con 0x7f1860070c00 2024-02-13T00:31:54.821 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.817+0000 7f185d7fa700 1 -- 172.21.15.162:0/935080022 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f185001e030 con 0x7f1860070c00 2024-02-13T00:31:54.885 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.882+0000 7f18665fd700 1 -- 172.21.15.162:0/935080022 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f18601001c0 con 0x7f1860070c00 2024-02-13T00:31:54.885 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.882+0000 7f185d7fa700 1 -- 172.21.15.162:0/935080022 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f1850023410 con 0x7f1860070c00 2024-02-13T00:31:54.887 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.883+0000 7f18665fd700 10 librados: watch_flush enter 2024-02-13T00:31:54.887 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.883+0000 7f18665fd700 10 librados: watch_flush exit 2024-02-13T00:31:54.887 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.883+0000 7f18665fd700 20 client.4102.objecter shutdown clearing up homeless session... 2024-02-13T00:31:54.887 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.883+0000 7f18665fd700 10 client.4102.objecter successfully canceled tick 2024-02-13T00:31:54.887 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.883+0000 7f18665fd700 1 -- 172.21.15.162:0/935080022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 msgr2=0x7f186010d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:31:54.887 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.883+0000 7f18665fd700 1 --2- 172.21.15.162:0/935080022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 0x7f186010d120 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1850011630 tx=0x7f18500090f0).stop 2024-02-13T00:31:54.887 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.883+0000 7f18665fd700 1 -- 172.21.15.162:0/935080022 shutdown_connections 2024-02-13T00:31:54.888 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.883+0000 7f18665fd700 1 --2- 172.21.15.162:0/935080022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1860070c00 0x7f186010d120 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:31:54.888 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.883+0000 7f18665fd700 1 -- 172.21.15.162:0/935080022 >> 172.21.15.162:0/935080022 conn(0x7f18600f6820 msgr2=0x7f18600f80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:31:54.888 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.883+0000 7f18665fd700 1 -- 172.21.15.162:0/935080022 shutdown_connections 2024-02-13T00:31:54.888 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.884+0000 7f18665fd700 1 -- 172.21.15.162:0/935080022 wait complete. 2024-02-13T00:31:54.888 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:54.884+0000 7f18665fd700 1 librados: shutdown 2024-02-13T00:31:55.245 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39105]: audit 2024-02-13T00:31:54.883560+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.162:0/935080022' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:31:55.532 INFO:teuthology.orchestra.run.smithi162.stderr:Restarting the monitor... 2024-02-13T00:31:55.788 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:55 smithi162 systemd[1]: Stopping Ceph mon.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:31:55.788 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:55 smithi162 bash[39426]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon.a" found: no such container 2024-02-13T00:31:56.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39105]: debug 2024-02-13T00:31:55.785+0000 7fac085c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-13T00:31:56.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39105]: debug 2024-02-13T00:31:55.785+0000 7fac085c0700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-13T00:31:56.524 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:56 smithi162 bash[39467]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a 2024-02-13T00:31:56.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:56 smithi162 systemd[1]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.a.service: Succeeded. 2024-02-13T00:31:56.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:56 smithi162 systemd[1]: Stopped Ceph mon.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:31:56.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:56 smithi162 systemd[1]: Starting Ceph mon.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:31:59.171 INFO:teuthology.orchestra.run.smithi162.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-13T00:31:59.454 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.113+0000 7f5f7b279880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-13T00:31:59.454 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.113+0000 7f5f7b279880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 7 2024-02-13T00:31:59.454 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.113+0000 7f5f7b279880 0 pidfile_write: ignore empty --pid-file 2024-02-13T00:31:59.454 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.118+0000 7f5f7b279880 0 load: jerasure load: lrc load: isa 2024-02-13T00:31:59.454 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: RocksDB version: 6.8.1 2024-02-13T00:31:59.454 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.454 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-13T00:31:59.454 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Compile date Feb 8 2024 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: DB SUMMARY 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: CURRENT file: CURRENT 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 94619 ; 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.error_if_exists: 0 2024-02-13T00:31:59.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.create_if_missing: 0 2024-02-13T00:31:59.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.paranoid_checks: 1 2024-02-13T00:31:59.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.env: 0x5594e63a5080 2024-02-13T00:31:59.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.fs: Posix File System 2024-02-13T00:31:59.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.info_log: 0x5594e7f96260 2024-02-13T00:31:59.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-13T00:31:59.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.statistics: (nil) 2024-02-13T00:31:59.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.use_fsync: 0 2024-02-13T00:31:59.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_log_file_size: 0 2024-02-13T00:31:59.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-13T00:31:59.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-13T00:31:59.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-13T00:31:59.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.allow_fallocate: 1 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.use_direct_reads: 0 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.db_log_dir: 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_subcompactions: 1 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_background_flushes: -1 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-13T00:31:59.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.write_buffer_manager: 0x5594e8c88390 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.unordered_write: 0 2024-02-13T00:31:59.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.row_cache: None 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.wal_filter: None 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.preserve_deletes: 0 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.two_write_queues: 0 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.atomic_flush: 0 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-13T00:31:59.459 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.log_readahead_size: 0 2024-02-13T00:31:59.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-13T00:31:59.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_background_jobs: 2 2024-02-13T00:31:59.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_background_compactions: -1 2024-02-13T00:31:59.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-13T00:31:59.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-13T00:31:59.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-13T00:31:59.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-13T00:31:59.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-13T00:31:59.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_open_files: -1 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Compression algorithms supported: 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: kZSTD supported: 0 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: kXpressCompression supported: 0 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: kLZ4Compression supported: 1 2024-02-13T00:31:59.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: kBZip2Compression supported: 0 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: kZlibCompression supported: 1 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: kSnappyCompression supported: 1 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.merge_operator: 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_filter: None 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5594e7f3bd28) 2024-02-13T00:31:59.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cache_index_and_filter_blocks: 1 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: pin_top_level_index_and_filter: 1 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: index_type: 0 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: data_block_index_type: 0 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: index_shortening: 1 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: data_block_hash_table_util_ratio: 0.750000 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: hash_index_allow_collision: 1 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: checksum: 1 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: no_block_cache: 0 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: block_cache: 0x5594e7f70f10 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: block_cache_name: BinnedLRUCache 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: block_cache_options: 2024-02-13T00:31:59.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: capacity : 536870912 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: num_shard_bits : 4 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: strict_capacity_limit : 0 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: high_pri_pool_ratio: 0.000 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: block_cache_compressed: (nil) 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: persistent_cache: (nil) 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: block_size: 4096 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: block_size_deviation: 10 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: block_restart_interval: 16 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: index_block_restart_interval: 1 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: metadata_block_size: 4096 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: partition_filters: 0 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: use_delta_encoding: 1 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: whole_key_filtering: 1 2024-02-13T00:31:59.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: verify_compression: 0 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: read_amp_bytes_per_bit: 0 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: format_version: 2 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: enable_index_compression: 1 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: block_align: 0 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compression: NoCompression 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.num_levels: 7 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-13T00:31:59.466 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-13T00:31:59.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-13T00:31:59.468 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-13T00:31:59.469 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.table_properties_collectors: 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.inplace_update_support: 0 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.bloom_locality: 0 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.max_successive_merges: 0 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.ttl: 2592000 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.119+0000 7f5f7b279880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-13T00:31:59.470 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.120+0000 7f5f7b279880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.120+0000 7f5f7b279880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.120+0000 7f5f7b279880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707784319121007, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.120+0000 7f5f7b279880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.120+0000 7f5f7b279880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.121+0000 7f5f7b279880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707784319122018, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 90459, "table_properties": {"data_size": 88561, "index_size": 375, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 11592, "raw_average_key_size": 49, "raw_value_size": 82200, "raw_average_value_size": 348, "num_data_blocks": 12, "num_entries": 236, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707784319, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.121+0000 7f5f7b279880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.121+0000 7f5f7b279880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707784319122491, "job": 1, "event": "recovery_finished"} 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.122+0000 7f5f7b279880 4 rocksdb: DB pointer 0x5594e8039800 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.122+0000 7f5f6206a700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.122+0000 7f5f6206a700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-13T00:31:59.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: ** DB Stats ** 2024-02-13T00:31:59.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T00:31:59.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-13T00:31:59.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T00:31:59.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T00:31:59.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-13T00:31:59.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-13T00:31:59.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T00:31:59.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: ** Compaction Stats [default] ** 2024-02-13T00:31:59.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T00:31:59.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T00:31:59.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: L0 2/0 89.99 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.9 0.00 0.00 1 0.001 0 0 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Sum 2/0 89.99 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.9 0.00 0.00 1 0.001 0 0 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.9 0.00 0.00 1 0.001 0 0 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: ** Compaction Stats [default] ** 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.9 0.00 0.00 1 0.001 0 0 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: AddFile(Keys): cumulative 0, interval 0 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Cumulative compaction: 0.00 GB write, 26.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Interval compaction: 0.00 GB write, 26.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T00:31:59.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: ** File Read Latency Histogram By Level [default] ** 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: ** Compaction Stats [default] ** 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: L0 2/0 89.99 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.9 0.00 0.00 1 0.001 0 0 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Sum 2/0 89.99 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.9 0.00 0.00 1 0.001 0 0 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: ** Compaction Stats [default] ** 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.9 0.00 0.00 1 0.001 0 0 2024-02-13T00:31:59.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: AddFile(Keys): cumulative 0, interval 0 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Cumulative compaction: 0.00 GB write, 26.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: ** File Read Latency Histogram By Level [default] ** 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.123+0000 7f5f7b279880 0 starting mon.a rank 0 at public addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] at bind addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.124+0000 7f5f7b279880 1 mon.a@-1(???) e1 preinit fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.124+0000 7f5f7b279880 0 mon.a@-1(???).mds e1 new map 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.124+0000 7f5f7b279880 0 mon.a@-1(???).mds e1 print_map 2024-02-13T00:31:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: e1 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: legacy client fscid: -1 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: No filesystems configured 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.124+0000 7f5f7b279880 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.124+0000 7f5f7b279880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.124+0000 7f5f7b279880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.124+0000 7f5f7b279880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:31:54.883560+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.162:0/935080022' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:31:59.124+0000 7f5f7b279880 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 bash[39591]: 2026d4f5c831482ba35e12f4996e384de0a533e803b005d28952f1fda1d3f31c 2024-02-13T00:31:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 systemd[1]: Started Ceph mon.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:31:59.477 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:31:59.141430+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-13T00:31:59.477 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:31:59.141492+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0]} removed_ranks: {} 2024-02-13T00:31:59.477 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:31:59.141881+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2024-02-13T00:31:59.477 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:31:59.141913+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-13T00:31:59.477 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:31:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:31:59.142335+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-13T00:31:59.735 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.731+0000 7f476cd6e700 1 Processor -- start 2024-02-13T00:31:59.735 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.731+0000 7f476cd6e700 1 -- start start 2024-02-13T00:31:59.739 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.732+0000 7f476cd6e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 0x7f47681011d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:31:59.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.732+0000 7f476cd6e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47680fcbc0 con 0x7f4768070be0 2024-02-13T00:31:59.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.732+0000 7f476659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 0x7f47681011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:31:59.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.732+0000 7f476659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 0x7f47681011d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59600/0 (socket says 172.21.15.162:59600) 2024-02-13T00:31:59.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.732+0000 7f476659c700 1 -- 172.21.15.162:0/1055574688 learned_addr learned my addr 172.21.15.162:0/1055574688 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:31:59.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.732+0000 7f476659c700 1 -- 172.21.15.162:0/1055574688 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47681016c0 con 0x7f4768070be0 2024-02-13T00:31:59.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.733+0000 7f476659c700 1 --2- 172.21.15.162:0/1055574688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 0x7f47681011d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f4750009730 tx=0x7f47500090f0).ready entity=mon.0 client_cookie=6c3f17d0136e0967 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:31:59.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.733+0000 7f476559a700 1 -- 172.21.15.162:0/1055574688 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f475000b580 con 0x7f4768070be0 2024-02-13T00:31:59.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.733+0000 7f476559a700 1 -- 172.21.15.162:0/1055574688 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f475000bb90 con 0x7f4768070be0 2024-02-13T00:31:59.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.733+0000 7f476559a700 1 -- 172.21.15.162:0/1055574688 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4750011c30 con 0x7f4768070be0 2024-02-13T00:31:59.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.733+0000 7f476cd6e700 1 -- 172.21.15.162:0/1055574688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 msgr2=0x7f47681011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:31:59.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.733+0000 7f476cd6e700 1 --2- 172.21.15.162:0/1055574688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 0x7f47681011d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f4750009730 tx=0x7f47500090f0).stop 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.733+0000 7f476cd6e700 1 -- 172.21.15.162:0/1055574688 shutdown_connections 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.733+0000 7f476cd6e700 1 --2- 172.21.15.162:0/1055574688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 0x7f47681011d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.733+0000 7f476cd6e700 1 -- 172.21.15.162:0/1055574688 >> 172.21.15.162:0/1055574688 conn(0x7f47680f67d0 msgr2=0x7f47680f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.733+0000 7f476cd6e700 1 -- 172.21.15.162:0/1055574688 shutdown_connections 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.733+0000 7f476cd6e700 1 -- 172.21.15.162:0/1055574688 wait complete. 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.734+0000 7f476cd6e700 1 librados: starting msgr at 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.734+0000 7f476cd6e700 1 librados: starting objecter 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.734+0000 7f476cd6e700 1 Processor -- start 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.734+0000 7f476cd6e700 1 -- start start 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.734+0000 7f476cd6e700 1 librados: setting wanted keys 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.734+0000 7f476cd6e700 1 librados: calling monclient init 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.734+0000 7f476cd6e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 0x7f476810d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:31:59.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.734+0000 7f476cd6e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476810d550 con 0x7f4768070be0 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f476659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 0x7f476810d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f476659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 0x7f476810d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59608/0 (socket says 172.21.15.162:59608) 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f476659c700 1 -- 172.21.15.162:0/4113511663 learned_addr learned my addr 172.21.15.162:0/4113511663 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f476659c700 1 -- 172.21.15.162:0/4113511663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4750009370 con 0x7f4768070be0 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f476659c700 1 --2- 172.21.15.162:0/4113511663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 0x7f476810d060 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f475000bf60 tx=0x7f47500090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f475f7fe700 10 client.?.objecter ms_handle_connect 0x7f4768070be0 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f475f7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f475f7fe700 1 -- 172.21.15.162:0/4113511663 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4750011bf0 con 0x7f4768070be0 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f476cd6e700 1 -- 172.21.15.162:0/4113511663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476810d750 con 0x7f4768070be0 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f475f7fe700 1 -- 172.21.15.162:0/4113511663 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f475001aaf0 con 0x7f4768070be0 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f475f7fe700 1 -- 172.21.15.162:0/4113511663 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47500229c0 con 0x7f4768070be0 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f476cd6e700 10 client.14098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:31:59.742 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f476cd6e700 1 -- 172.21.15.162:0/4113511663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f476810dbe0 con 0x7f4768070be0 2024-02-13T00:31:59.743 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f476cd6e700 1 librados: init done 2024-02-13T00:31:59.743 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.735+0000 7f475f7fe700 1 -- 172.21.15.162:0/4113511663 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f4750023450 con 0x7f4768070be0 2024-02-13T00:31:59.743 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.736+0000 7f475f7fe700 1 -- 172.21.15.162:0/4113511663 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4750019700 con 0x7f4768070be0 2024-02-13T00:31:59.743 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.736+0000 7f475f7fe700 10 client.14098.objecter ms_dispatch 0x7f4768001550 osd_map(1..1 src has 1..1) v4 2024-02-13T00:31:59.743 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.736+0000 7f475f7fe700 3 client.14098.objecter handle_osd_map got epochs [1,1] > 0 2024-02-13T00:31:59.743 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.736+0000 7f475f7fe700 3 client.14098.objecter handle_osd_map decoding full epoch 1 2024-02-13T00:31:59.743 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.736+0000 7f475f7fe700 20 client.14098.objecter dump_active .. 0 homeless 2024-02-13T00:31:59.743 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.736+0000 7f476cd6e700 1 -- 172.21.15.162:0/4113511663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4768046d80 con 0x7f4768070be0 2024-02-13T00:31:59.743 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.738+0000 7f475f7fe700 1 -- 172.21.15.162:0/4113511663 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f4750028060 con 0x7f4768070be0 2024-02-13T00:31:59.805 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.802+0000 7f476cd6e700 1 -- 172.21.15.162:0/4113511663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f47681023b0 con 0x7f4768070be0 2024-02-13T00:31:59.807 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.804+0000 7f475f7fe700 1 -- 172.21.15.162:0/4113511663 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f475001e070 con 0x7f4768070be0 2024-02-13T00:31:59.809 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.805+0000 7f476cd6e700 10 librados: watch_flush enter 2024-02-13T00:31:59.809 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.805+0000 7f476cd6e700 10 librados: watch_flush exit 2024-02-13T00:31:59.809 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.805+0000 7f476cd6e700 20 client.14098.objecter shutdown clearing up homeless session... 2024-02-13T00:31:59.809 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.805+0000 7f476cd6e700 10 client.14098.objecter successfully canceled tick 2024-02-13T00:31:59.809 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.805+0000 7f476cd6e700 1 -- 172.21.15.162:0/4113511663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 msgr2=0x7f476810d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:31:59.809 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.805+0000 7f476cd6e700 1 --2- 172.21.15.162:0/4113511663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 0x7f476810d060 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f475000bf60 tx=0x7f47500090f0).stop 2024-02-13T00:31:59.809 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.805+0000 7f476cd6e700 1 -- 172.21.15.162:0/4113511663 shutdown_connections 2024-02-13T00:31:59.809 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.805+0000 7f476cd6e700 1 --2- 172.21.15.162:0/4113511663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4768070be0 0x7f476810d060 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:31:59.809 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.805+0000 7f476cd6e700 1 -- 172.21.15.162:0/4113511663 >> 172.21.15.162:0/4113511663 conn(0x7f47680f67d0 msgr2=0x7f47680f8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:31:59.809 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.805+0000 7f476cd6e700 1 -- 172.21.15.162:0/4113511663 shutdown_connections 2024-02-13T00:31:59.809 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.805+0000 7f476cd6e700 1 -- 172.21.15.162:0/4113511663 wait complete. 2024-02-13T00:31:59.809 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:31:59.805+0000 7f476cd6e700 1 librados: shutdown 2024-02-13T00:32:00.351 INFO:teuthology.orchestra.run.smithi162.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-13T00:32:00.351 INFO:teuthology.orchestra.run.smithi162.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-13T00:32:00.351 INFO:teuthology.orchestra.run.smithi162.stderr:Creating mgr... 2024-02-13T00:32:00.352 INFO:teuthology.orchestra.run.smithi162.stderr:Verifying port 9283 ... 2024-02-13T00:32:00.478 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: Failed to reset failed state of unit ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mgr.y.service: Unit ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mgr.y.service not loaded. 2024-02-13T00:32:00.484 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: Created symlink /etc/systemd/system/ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715.target.wants/ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@.service. 2024-02-13T00:32:00.807 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:00 smithi162 systemd[1]: Starting Ceph mgr.y for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:32:01.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:31:59.805166+0000 mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.162:0/4113511663' entity='client.admin' 2024-02-13T00:32:01.274 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: disabled 2024-02-13T00:32:01.282 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: inactive 2024-02-13T00:32:01.283 INFO:teuthology.orchestra.run.smithi162.stderr:firewalld.service is not enabled 2024-02-13T00:32:01.283 INFO:teuthology.orchestra.run.smithi162.stderr:Not possible to enable service . firewalld.service is not available 2024-02-13T00:32:01.290 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: disabled 2024-02-13T00:32:01.299 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: inactive 2024-02-13T00:32:01.300 INFO:teuthology.orchestra.run.smithi162.stderr:firewalld.service is not enabled 2024-02-13T00:32:01.300 INFO:teuthology.orchestra.run.smithi162.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-13T00:32:01.300 INFO:teuthology.orchestra.run.smithi162.stderr:Waiting for mgr to start... 2024-02-13T00:32:01.300 INFO:teuthology.orchestra.run.smithi162.stderr:Waiting for mgr... 2024-02-13T00:32:01.539 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:01 smithi162 bash[39866]: 3f99519322879978b8b9250332c6b016ca211080a0cc91d19d0bd2655adb0ee6 2024-02-13T00:32:01.539 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:01 smithi162 systemd[1]: Started Ceph mgr.y for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:32:01.539 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:01.276+0000 7f8804d9a680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T00:32:01.539 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:01.369+0000 7f8804d9a680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T00:32:01.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:01.753+0000 7f8804d9a680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T00:32:01.902 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.854+0000 7f06120a3700 1 Processor -- start 2024-02-13T00:32:01.902 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.854+0000 7f06120a3700 1 -- start start 2024-02-13T00:32:01.902 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.854+0000 7f06120a3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 0x7f060c0fdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:01.902 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.854+0000 7f06120a3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c0fe030 con 0x7f060c0fd720 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.854+0000 7f060b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 0x7f060c0fdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.854+0000 7f060b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 0x7f060c0fdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59648/0 (socket says 172.21.15.162:59648) 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.854+0000 7f060b7fe700 1 -- 172.21.15.162:0/2576096685 learned_addr learned my addr 172.21.15.162:0/2576096685 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.855+0000 7f060b7fe700 1 -- 172.21.15.162:0/2576096685 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c0fe170 con 0x7f060c0fd720 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.855+0000 7f060b7fe700 1 --2- 172.21.15.162:0/2576096685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 0x7f060c0fdb40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f05fc00f9b0 tx=0x7f05fc0090f0).ready entity=mon.0 client_cookie=24d6b2840369cf3e server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.855+0000 7f060a7fc700 1 -- 172.21.15.162:0/2576096685 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f05fc0099f0 con 0x7f060c0fd720 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.855+0000 7f060a7fc700 1 -- 172.21.15.162:0/2576096685 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05fc018440 con 0x7f060c0fd720 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.855+0000 7f060a7fc700 1 -- 172.21.15.162:0/2576096685 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f05fc0175a0 con 0x7f060c0fd720 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.856+0000 7f06120a3700 1 -- 172.21.15.162:0/2576096685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 msgr2=0x7f060c0fdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.856+0000 7f06120a3700 1 --2- 172.21.15.162:0/2576096685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 0x7f060c0fdb40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f05fc00f9b0 tx=0x7f05fc0090f0).stop 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.856+0000 7f06120a3700 1 -- 172.21.15.162:0/2576096685 shutdown_connections 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.856+0000 7f06120a3700 1 --2- 172.21.15.162:0/2576096685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 0x7f060c0fdb40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.856+0000 7f06120a3700 1 -- 172.21.15.162:0/2576096685 >> 172.21.15.162:0/2576096685 conn(0x7f060c0f5f60 msgr2=0x7f060c0f83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.856+0000 7f06120a3700 1 -- 172.21.15.162:0/2576096685 shutdown_connections 2024-02-13T00:32:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.856+0000 7f06120a3700 1 -- 172.21.15.162:0/2576096685 wait complete. 2024-02-13T00:32:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.856+0000 7f06120a3700 1 librados: starting msgr at 2024-02-13T00:32:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.856+0000 7f06120a3700 1 librados: starting objecter 2024-02-13T00:32:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.856+0000 7f06120a3700 1 Processor -- start 2024-02-13T00:32:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f06120a3700 1 -- start start 2024-02-13T00:32:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f06120a3700 1 librados: setting wanted keys 2024-02-13T00:32:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f06120a3700 1 librados: calling monclient init 2024-02-13T00:32:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f06120a3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 0x7f060c108cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f06120a3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c1091e0 con 0x7f060c0fd720 2024-02-13T00:32:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f060b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 0x7f060c108cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f060b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 0x7f060c108cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59650/0 (socket says 172.21.15.162:59650) 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f060b7fe700 1 -- 172.21.15.162:0/2927714874 learned_addr learned my addr 172.21.15.162:0/2927714874 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f060b7fe700 1 -- 172.21.15.162:0/2927714874 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc00f5f0 con 0x7f060c0fd720 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f060b7fe700 1 --2- 172.21.15.162:0/2927714874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 0x7f060c108cf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f05fc0187e0 tx=0x7f05fc0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f0608ff9700 10 client.?.objecter ms_handle_connect 0x7f060c0fd720 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f0608ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f0608ff9700 1 -- 172.21.15.162:0/2927714874 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f05fc009c00 con 0x7f060c0fd720 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f06120a3700 1 -- 172.21.15.162:0/2927714874 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060c1093e0 con 0x7f060c0fd720 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f0608ff9700 1 -- 172.21.15.162:0/2927714874 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05fc009d60 con 0x7f060c0fd720 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.857+0000 7f0608ff9700 1 -- 172.21.15.162:0/2927714874 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f05fc020920 con 0x7f060c0fd720 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.858+0000 7f06120a3700 10 client.14102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.858+0000 7f06120a3700 1 -- 172.21.15.162:0/2927714874 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060c109f90 con 0x7f060c0fd720 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.858+0000 7f06120a3700 1 librados: init done 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.858+0000 7f0608ff9700 1 -- 172.21.15.162:0/2927714874 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f05fc029400 con 0x7f060c0fd720 2024-02-13T00:32:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.858+0000 7f0608ff9700 1 -- 172.21.15.162:0/2927714874 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f05fc01f710 con 0x7f060c0fd720 2024-02-13T00:32:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.858+0000 7f0608ff9700 10 client.14102.objecter ms_dispatch 0x7f060c001550 osd_map(1..1 src has 1..1) v4 2024-02-13T00:32:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.858+0000 7f0608ff9700 3 client.14102.objecter handle_osd_map got epochs [1,1] > 0 2024-02-13T00:32:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.858+0000 7f0608ff9700 3 client.14102.objecter handle_osd_map decoding full epoch 1 2024-02-13T00:32:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.858+0000 7f0608ff9700 20 client.14102.objecter dump_active .. 0 homeless 2024-02-13T00:32:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.858+0000 7f06120a3700 1 -- 172.21.15.162:0/2927714874 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060c05a3b0 con 0x7f060c0fd720 2024-02-13T00:32:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.861+0000 7f0608ff9700 1 -- 172.21.15.162:0/2927714874 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f05fc029840 con 0x7f060c0fd720 2024-02-13T00:32:01.941 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.930+0000 7f06120a3700 1 -- 172.21.15.162:0/2927714874 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f060c0fad20 con 0x7f060c0fd720 2024-02-13T00:32:01.941 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.931+0000 7f0608ff9700 1 -- 172.21.15.162:0/2927714874 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (secure 0 0 0) 0x7f05fc038090 con 0x7f060c0fd720 2024-02-13T00:32:01.941 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.941 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: { 2024-02-13T00:32:01.941 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:32:01.941 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "health": { 2024-02-13T00:32:01.941 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "checks": {}, 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "mutes": [] 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "quorum": [ 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 0 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: ], 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "a" 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: ], 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "monmap": { 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T00:32:01.942 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "osdmap": { 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "pgmap": { 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-13T00:32:01.943 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "fsmap": { 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-13T00:32:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "available": false, 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "modules": [ 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "iostat", 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "nfs", 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "restful" 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: ], 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "services": {} 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "servicemap": { 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "modified": "2024-02-13T00:31:51.797490+0000", 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "services": {} 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: } 2024-02-13T00:32:01.945 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.932+0000 7f06120a3700 10 librados: watch_flush enter 2024-02-13T00:32:01.946 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.932+0000 7f06120a3700 10 librados: watch_flush exit 2024-02-13T00:32:01.946 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.932+0000 7f06120a3700 20 client.14102.objecter shutdown clearing up homeless session... 2024-02-13T00:32:01.946 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.932+0000 7f06120a3700 10 client.14102.objecter successfully canceled tick 2024-02-13T00:32:01.946 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.932+0000 7f06120a3700 1 -- 172.21.15.162:0/2927714874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 msgr2=0x7f060c108cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:01.946 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.932+0000 7f06120a3700 1 --2- 172.21.15.162:0/2927714874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 0x7f060c108cf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f05fc0187e0 tx=0x7f05fc0090f0).stop 2024-02-13T00:32:01.946 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.932+0000 7f06120a3700 1 -- 172.21.15.162:0/2927714874 shutdown_connections 2024-02-13T00:32:01.946 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.932+0000 7f06120a3700 1 --2- 172.21.15.162:0/2927714874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0fd720 0x7f060c108cf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:01.946 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.932+0000 7f06120a3700 1 -- 172.21.15.162:0/2927714874 >> 172.21.15.162:0/2927714874 conn(0x7f060c0f5f60 msgr2=0x7f060c0f7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:01.946 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.932+0000 7f06120a3700 1 -- 172.21.15.162:0/2927714874 shutdown_connections 2024-02-13T00:32:01.946 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.932+0000 7f06120a3700 1 -- 172.21.15.162:0/2927714874 wait complete. 2024-02-13T00:32:01.946 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:01.932+0000 7f06120a3700 1 librados: shutdown 2024-02-13T00:32:02.275 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:01.931984+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.162:0/2927714874' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-13T00:32:02.578 INFO:teuthology.orchestra.run.smithi162.stderr:mgr not available, waiting (1/15)... 2024-02-13T00:32:02.828 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:02.530+0000 7f8804d9a680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T00:32:02.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:02.615+0000 7f8804d9a680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T00:32:03.079 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:02.825+0000 7f8804d9a680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T00:32:04.065 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:03.766+0000 7f8804d9a680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T00:32:04.065 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:03.893+0000 7f8804d9a680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T00:32:04.065 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:03.972+0000 7f8804d9a680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T00:32:04.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:04.062+0000 7f8804d9a680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T00:32:04.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:04.257+0000 7f8804d9a680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T00:32:04.820 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:04.534+0000 7f8804d9a680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T00:32:04.820 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:04.636+0000 7f8804d9a680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T00:32:05.167 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.151+0000 7fe02332e700 1 Processor -- start 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.151+0000 7fe02332e700 1 -- start start 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.151+0000 7fe02332e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 0x7fe01c0fedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.151+0000 7fe02332e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0ff2b0 con 0x7fe01c070c20 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.151+0000 7fe0210ca700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 0x7fe01c0fedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.151+0000 7fe0210ca700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 0x7fe01c0fedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44668/0 (socket says 172.21.15.162:44668) 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.151+0000 7fe0210ca700 1 -- 172.21.15.162:0/3079222426 learned_addr learned my addr 172.21.15.162:0/3079222426 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.152+0000 7fe0210ca700 1 -- 172.21.15.162:0/3079222426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c0ff3f0 con 0x7fe01c070c20 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.152+0000 7fe0210ca700 1 --2- 172.21.15.162:0/3079222426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 0x7fe01c0fedc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe018009730 tx=0x7fe0180090f0).ready entity=mon.0 client_cookie=8c2071a69b8b68e8 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.152+0000 7fe013fff700 1 -- 172.21.15.162:0/3079222426 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe01800b610 con 0x7fe01c070c20 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.152+0000 7fe013fff700 1 -- 172.21.15.162:0/3079222426 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe01800bc20 con 0x7fe01c070c20 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.152+0000 7fe013fff700 1 -- 172.21.15.162:0/3079222426 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe018011ca0 con 0x7fe01c070c20 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.152+0000 7fe02332e700 1 -- 172.21.15.162:0/3079222426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 msgr2=0x7fe01c0fedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:05.168 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.152+0000 7fe02332e700 1 --2- 172.21.15.162:0/3079222426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 0x7fe01c0fedc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe018009730 tx=0x7fe0180090f0).stop 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 -- 172.21.15.162:0/3079222426 shutdown_connections 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 --2- 172.21.15.162:0/3079222426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 0x7fe01c0fedc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 -- 172.21.15.162:0/3079222426 >> 172.21.15.162:0/3079222426 conn(0x7fe01c0f6820 msgr2=0x7fe01c0f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 -- 172.21.15.162:0/3079222426 shutdown_connections 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 -- 172.21.15.162:0/3079222426 wait complete. 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 librados: starting msgr at 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 librados: starting objecter 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 Processor -- start 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 -- start start 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 librados: setting wanted keys 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 librados: calling monclient init 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 0x7fe01c0fc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.153+0000 7fe02332e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0facc0 con 0x7fe01c070c20 2024-02-13T00:32:05.169 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0210ca700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 0x7fe01c0fc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0210ca700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 0x7fe01c0fc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44674/0 (socket says 172.21.15.162:44674) 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0210ca700 1 -- 172.21.15.162:0/1710881369 learned_addr learned my addr 172.21.15.162:0/1710881369 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0210ca700 1 -- 172.21.15.162:0/1710881369 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe018009370 con 0x7fe01c070c20 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0210ca700 1 --2- 172.21.15.162:0/1710881369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 0x7fe01c0fc840 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe018000c00 tx=0x7fe0180090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0127fc700 10 client.?.objecter ms_handle_connect 0x7fe01c070c20 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0127fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0127fc700 1 -- 172.21.15.162:0/1710881369 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe01801a430 con 0x7fe01c070c20 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0127fc700 1 -- 172.21.15.162:0/1710881369 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe01801aa40 con 0x7fe01c070c20 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0127fc700 1 -- 172.21.15.162:0/1710881369 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe018023b90 con 0x7fe01c070c20 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe02332e700 1 -- 172.21.15.162:0/1710881369 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01c0fae60 con 0x7fe01c070c20 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe02332e700 10 client.14104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe02332e700 1 -- 172.21.15.162:0/1710881369 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c0fb2f0 con 0x7fe01c070c20 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe02332e700 1 librados: init done 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0127fc700 1 -- 172.21.15.162:0/1710881369 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7fe018022830 con 0x7fe01c070c20 2024-02-13T00:32:05.170 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0127fc700 1 -- 172.21.15.162:0/1710881369 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe01802c7a0 con 0x7fe01c070c20 2024-02-13T00:32:05.171 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0127fc700 10 client.14104.objecter ms_dispatch 0x7fe01c001550 osd_map(1..1 src has 1..1) v4 2024-02-13T00:32:05.171 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0127fc700 3 client.14104.objecter handle_osd_map got epochs [1,1] > 0 2024-02-13T00:32:05.171 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0127fc700 3 client.14104.objecter handle_osd_map decoding full epoch 1 2024-02-13T00:32:05.171 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.154+0000 7fe0127fc700 20 client.14104.objecter dump_active .. 0 homeless 2024-02-13T00:32:05.171 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.155+0000 7fe02332e700 1 -- 172.21.15.162:0/1710881369 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01c05a3b0 con 0x7fe01c070c20 2024-02-13T00:32:05.171 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.157+0000 7fe0127fc700 1 -- 172.21.15.162:0/1710881369 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7fe01801e030 con 0x7fe01c070c20 2024-02-13T00:32:05.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.227+0000 7fe02332e700 1 -- 172.21.15.162:0/1710881369 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe01c104a00 con 0x7fe01c070c20 2024-02-13T00:32:05.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.227+0000 7fe0127fc700 1 -- 172.21.15.162:0/1710881369 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (secure 0 0 0) 0x7fe018023590 con 0x7fe01c070c20 2024-02-13T00:32:05.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: { 2024-02-13T00:32:05.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:32:05.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "health": { 2024-02-13T00:32:05.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-13T00:32:05.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "checks": {}, 2024-02-13T00:32:05.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "mutes": [] 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "quorum": [ 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 0 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: ], 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "a" 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: ], 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "monmap": { 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-13T00:32:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "osdmap": { 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "pgmap": { 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-13T00:32:05.239 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "fsmap": { 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "available": false, 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "modules": [ 2024-02-13T00:32:05.240 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "iostat", 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "nfs", 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "restful" 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: ], 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "services": {} 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "servicemap": { 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "modified": "2024-02-13T00:31:51.797490+0000", 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "services": {} 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: } 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.228+0000 7fe02332e700 10 librados: watch_flush enter 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.228+0000 7fe02332e700 10 librados: watch_flush exit 2024-02-13T00:32:05.241 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.228+0000 7fe02332e700 20 client.14104.objecter shutdown clearing up homeless session... 2024-02-13T00:32:05.242 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.228+0000 7fe02332e700 10 client.14104.objecter successfully canceled tick 2024-02-13T00:32:05.242 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.228+0000 7fe02332e700 1 -- 172.21.15.162:0/1710881369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 msgr2=0x7fe01c0fc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:05.242 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.228+0000 7fe02332e700 1 --2- 172.21.15.162:0/1710881369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 0x7fe01c0fc840 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe018000c00 tx=0x7fe0180090f0).stop 2024-02-13T00:32:05.242 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.229+0000 7fe02332e700 1 -- 172.21.15.162:0/1710881369 shutdown_connections 2024-02-13T00:32:05.242 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.229+0000 7fe02332e700 1 --2- 172.21.15.162:0/1710881369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c070c20 0x7fe01c0fc840 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:05.242 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.229+0000 7fe02332e700 1 -- 172.21.15.162:0/1710881369 >> 172.21.15.162:0/1710881369 conn(0x7fe01c0f6820 msgr2=0x7fe01c0f80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:05.242 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.229+0000 7fe02332e700 1 -- 172.21.15.162:0/1710881369 shutdown_connections 2024-02-13T00:32:05.242 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.229+0000 7fe02332e700 1 -- 172.21.15.162:0/1710881369 wait complete. 2024-02-13T00:32:05.242 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:05.229+0000 7fe02332e700 1 librados: shutdown 2024-02-13T00:32:05.570 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:05.391+0000 7f8804d9a680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T00:32:05.570 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:05.473+0000 7f8804d9a680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T00:32:05.570 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:05.228529+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? 172.21.15.162:0/1710881369' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-13T00:32:05.794 INFO:teuthology.orchestra.run.smithi162.stderr:mgr not available, waiting (2/15)... 2024-02-13T00:32:05.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:05.567+0000 7f8804d9a680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T00:32:05.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:05.739+0000 7f8804d9a680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T00:32:06.272 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:06.005+0000 7f8804d9a680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T00:32:06.272 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:06.136+0000 7f8804d9a680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T00:32:06.525 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:06.269+0000 7f8804d9a680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T00:32:06.525 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:06.440+0000 7f8804d9a680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T00:32:06.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:06.522+0000 7f8804d9a680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T00:32:06.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:06.536683+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2024-02-13T00:32:06.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:06.539319+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00272134s) 2024-02-13T00:32:06.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:06.540053+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.162:0/3622772605' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T00:32:06.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:06.540166+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.162:0/3622772605' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T00:32:06.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:06.540277+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.162:0/3622772605' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T00:32:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:06.540384+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.162:0/3622772605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T00:32:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:06.540756+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.162:0/3622772605' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T00:32:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:06.542601+0000 mon.a (mon.0) 16 : cluster [INF] Manager daemon y is now available 2024-02-13T00:32:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:06.550861+0000 mon.a (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.162:0/3622772605' entity='mgr.y' 2024-02-13T00:32:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:06.556171+0000 mon.a (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.162:0/3622772605' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T00:32:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:06.558535+0000 mon.a (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.162:0/3622772605' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T00:32:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:06.561409+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.162:0/3622772605' entity='mgr.y' 2024-02-13T00:32:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:06.563331+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.162:0/3622772605' entity='mgr.y' 2024-02-13T00:32:08.457 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.450+0000 7f36ef776700 1 Processor -- start 2024-02-13T00:32:08.457 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.450+0000 7f36ef776700 1 -- start start 2024-02-13T00:32:08.457 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.450+0000 7f36ef776700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 0x7f36e80fedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:08.457 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.450+0000 7f36ef776700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e80ff2b0 con 0x7f36e8070c20 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.450+0000 7f36ed512700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 0x7f36e80fedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.451+0000 7f36ed512700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 0x7f36e80fedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44736/0 (socket says 172.21.15.162:44736) 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.451+0000 7f36ed512700 1 -- 172.21.15.162:0/3107335938 learned_addr learned my addr 172.21.15.162:0/3107335938 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.451+0000 7f36ed512700 1 -- 172.21.15.162:0/3107335938 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e80ff3f0 con 0x7f36e8070c20 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.451+0000 7f36ed512700 1 --2- 172.21.15.162:0/3107335938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 0x7f36e80fedc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f36e4009730 tx=0x7f36e40090f0).ready entity=mon.0 client_cookie=e8172186ed16c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.451+0000 7f36dffff700 1 -- 172.21.15.162:0/3107335938 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36e400b610 con 0x7f36e8070c20 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.451+0000 7f36dffff700 1 -- 172.21.15.162:0/3107335938 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36e400bc20 con 0x7f36e8070c20 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.451+0000 7f36dffff700 1 -- 172.21.15.162:0/3107335938 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36e4011ca0 con 0x7f36e8070c20 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.452+0000 7f36ef776700 1 -- 172.21.15.162:0/3107335938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 msgr2=0x7f36e80fedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.452+0000 7f36ef776700 1 --2- 172.21.15.162:0/3107335938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 0x7f36e80fedc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f36e4009730 tx=0x7f36e40090f0).stop 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.452+0000 7f36ef776700 1 -- 172.21.15.162:0/3107335938 shutdown_connections 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.452+0000 7f36ef776700 1 --2- 172.21.15.162:0/3107335938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 0x7f36e80fedc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.452+0000 7f36ef776700 1 -- 172.21.15.162:0/3107335938 >> 172.21.15.162:0/3107335938 conn(0x7f36e80f6820 msgr2=0x7f36e80f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.452+0000 7f36ef776700 1 -- 172.21.15.162:0/3107335938 shutdown_connections 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.452+0000 7f36ef776700 1 -- 172.21.15.162:0/3107335938 wait complete. 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.452+0000 7f36ef776700 1 librados: starting msgr at 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.452+0000 7f36ef776700 1 librados: starting objecter 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.452+0000 7f36ef776700 1 Processor -- start 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36ef776700 1 -- start start 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36ef776700 1 librados: setting wanted keys 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36ef776700 1 librados: calling monclient init 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36ef776700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 0x7f36e810d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36ef776700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e810d5a0 con 0x7f36e8070c20 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36ed512700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 0x7f36e810d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36ed512700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 0x7f36e810d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44740/0 (socket says 172.21.15.162:44740) 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36ed512700 1 -- 172.21.15.162:0/668852116 learned_addr learned my addr 172.21.15.162:0/668852116 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36ed512700 1 -- 172.21.15.162:0/668852116 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e4009370 con 0x7f36e8070c20 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36ed512700 1 --2- 172.21.15.162:0/668852116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 0x7f36e810d0b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f36e400bfc0 tx=0x7f36e40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:08.459 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36de7fc700 10 client.?.objecter ms_handle_connect 0x7f36e8070c20 2024-02-13T00:32:08.460 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36de7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:08.460 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.453+0000 7f36de7fc700 1 -- 172.21.15.162:0/668852116 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36e401a640 con 0x7f36e8070c20 2024-02-13T00:32:08.460 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.454+0000 7f36de7fc700 1 -- 172.21.15.162:0/668852116 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36e401ac50 con 0x7f36e8070c20 2024-02-13T00:32:08.460 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.454+0000 7f36de7fc700 1 -- 172.21.15.162:0/668852116 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36e4023b30 con 0x7f36e8070c20 2024-02-13T00:32:08.460 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.454+0000 7f36ef776700 1 -- 172.21.15.162:0/668852116 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e810d7a0 con 0x7f36e8070c20 2024-02-13T00:32:08.460 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.454+0000 7f36ef776700 10 client.14110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:08.460 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.454+0000 7f36ef776700 1 -- 172.21.15.162:0/668852116 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e810dc30 con 0x7f36e8070c20 2024-02-13T00:32:08.460 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.454+0000 7f36ef776700 1 librados: init done 2024-02-13T00:32:08.460 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.454+0000 7f36de7fc700 1 -- 172.21.15.162:0/668852116 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 3) v1 ==== 38465+0+0 (secure 0 0 0) 0x7f36e4023c90 con 0x7f36e8070c20 2024-02-13T00:32:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.454+0000 7f36de7fc700 1 --2- 172.21.15.162:0/668852116 >> v2:172.21.15.162:6800/2775563314 conn(0x7f36d4033e00 0x7f36d40362c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.455+0000 7f36de7fc700 1 -- 172.21.15.162:0/668852116 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f36e4044d60 con 0x7f36e8070c20 2024-02-13T00:32:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.455+0000 7f36de7fc700 10 client.14110.objecter ms_dispatch 0x7f36e8001550 osd_map(1..1 src has 1..1) v4 2024-02-13T00:32:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.455+0000 7f36de7fc700 3 client.14110.objecter handle_osd_map got epochs [1,1] > 0 2024-02-13T00:32:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.455+0000 7f36de7fc700 3 client.14110.objecter handle_osd_map decoding full epoch 1 2024-02-13T00:32:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.455+0000 7f36ef776700 1 -- 172.21.15.162:0/668852116 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e8046d80 con 0x7f36e8070c20 2024-02-13T00:32:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.455+0000 7f36de7fc700 20 client.14110.objecter dump_active .. 0 homeless 2024-02-13T00:32:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.455+0000 7f36ecd11700 1 --2- 172.21.15.162:0/668852116 >> v2:172.21.15.162:6800/2775563314 conn(0x7f36d4033e00 0x7f36d40362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.455+0000 7f36de7fc700 10 client.14110.objecter ms_handle_connect 0x7f36d4033e00 2024-02-13T00:32:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.455+0000 7f36ecd11700 1 --2- 172.21.15.162:0/668852116 >> v2:172.21.15.162:6800/2775563314 conn(0x7f36d4033e00 0x7f36d40362c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f36d8009780 tx=0x7f36d8006cc0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:08.462 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.460+0000 7f36de7fc700 1 -- 172.21.15.162:0/668852116 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f36e401e070 con 0x7f36e8070c20 2024-02-13T00:32:08.693 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.687+0000 7f36ef776700 1 -- 172.21.15.162:0/668852116 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f36e81001a0 con 0x7f36e8070c20 2024-02-13T00:32:08.693 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.687+0000 7f36de7fc700 1 -- 172.21.15.162:0/668852116 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f36e401e020 con 0x7f36e8070c20 2024-02-13T00:32:08.693 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.693 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: { 2024-02-13T00:32:08.693 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:32:08.693 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "health": { 2024-02-13T00:32:08.693 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-13T00:32:08.693 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "checks": {}, 2024-02-13T00:32:08.693 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "mutes": [] 2024-02-13T00:32:08.693 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:08.693 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "quorum": [ 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 0 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: ], 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "a" 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: ], 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "monmap": { 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "osdmap": { 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T00:32:08.694 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-13T00:32:08.699 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-13T00:32:08.699 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-13T00:32:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-13T00:32:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-13T00:32:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-13T00:32:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "pgmap": { 2024-02-13T00:32:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-13T00:32:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-13T00:32:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-13T00:32:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-13T00:32:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-13T00:32:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-13T00:32:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "fsmap": { 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "available": true, 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "modules": [ 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "iostat", 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "nfs", 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "restful" 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: ], 2024-02-13T00:32:08.701 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "services": {} 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "servicemap": { 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "modified": "2024-02-13T00:31:51.797490+0000", 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "services": {} 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: }, 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: } 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 10 librados: watch_flush enter 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 10 librados: watch_flush exit 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 20 client.14110.objecter shutdown clearing up homeless session... 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 10 client.14110.objecter successfully canceled tick 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 1 -- 172.21.15.162:0/668852116 >> v2:172.21.15.162:6800/2775563314 conn(0x7f36d4033e00 msgr2=0x7f36d40362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:08.702 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 1 --2- 172.21.15.162:0/668852116 >> v2:172.21.15.162:6800/2775563314 conn(0x7f36d4033e00 0x7f36d40362c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f36d8009780 tx=0x7f36d8006cc0).stop 2024-02-13T00:32:08.703 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 1 -- 172.21.15.162:0/668852116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 msgr2=0x7f36e810d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:08.703 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 1 --2- 172.21.15.162:0/668852116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 0x7f36e810d0b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f36e400bfc0 tx=0x7f36e40090f0).stop 2024-02-13T00:32:08.703 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 1 -- 172.21.15.162:0/668852116 shutdown_connections 2024-02-13T00:32:08.703 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 1 --2- 172.21.15.162:0/668852116 >> v2:172.21.15.162:6800/2775563314 conn(0x7f36d4033e00 0x7f36d40362c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:08.703 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 1 --2- 172.21.15.162:0/668852116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8070c20 0x7f36e810d0b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:08.703 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 1 -- 172.21.15.162:0/668852116 >> 172.21.15.162:0/668852116 conn(0x7f36e80f6820 msgr2=0x7f36e80f80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:08.703 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 1 -- 172.21.15.162:0/668852116 shutdown_connections 2024-02-13T00:32:08.703 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 1 -- 172.21.15.162:0/668852116 wait complete. 2024-02-13T00:32:08.703 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:08.689+0000 7f36ef776700 1 librados: shutdown 2024-02-13T00:32:08.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:07.542106+0000 mon.a (mon.0) 22 : cluster [DBG] mgrmap e3: y(active, since 1.00551s) 2024-02-13T00:32:09.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:08.688089+0000 mon.a (mon.0) 23 : audit [DBG] from='client.? 172.21.15.162:0/668852116' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-13T00:32:10.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:09.542655+0000 mon.a (mon.0) 24 : cluster [DBG] mgrmap e4: y(active, since 3s) 2024-02-13T00:32:11.560 INFO:teuthology.orchestra.run.smithi162.stderr:mgr is available 2024-02-13T00:32:15.214 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.701+0000 7f24cfd69700 1 Processor -- start 2024-02-13T00:32:15.222 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.701+0000 7f24cfd69700 1 -- start start 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.701+0000 7f24cfd69700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 0x7f24c80fb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.701+0000 7f24cfd69700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c80fb930 con 0x7f24c80fb020 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.701+0000 7f24cdb05700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 0x7f24c80fb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.701+0000 7f24cdb05700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 0x7f24c80fb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38830/0 (socket says 172.21.15.162:38830) 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.701+0000 7f24cdb05700 1 -- 172.21.15.162:0/3648751473 learned_addr learned my addr 172.21.15.162:0/3648751473 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.701+0000 7f24cdb05700 1 -- 172.21.15.162:0/3648751473 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c80fba70 con 0x7f24c80fb020 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.702+0000 7f24cdb05700 1 --2- 172.21.15.162:0/3648751473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 0x7f24c80fb440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f24c4009730 tx=0x7f24c40090f0).ready entity=mon.0 client_cookie=dc1873302207d81 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.702+0000 7f24ccb03700 1 -- 172.21.15.162:0/3648751473 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24c400b580 con 0x7f24c80fb020 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.702+0000 7f24ccb03700 1 -- 172.21.15.162:0/3648751473 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24c400bb90 con 0x7f24c80fb020 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.702+0000 7f24ccb03700 1 -- 172.21.15.162:0/3648751473 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24c4011c30 con 0x7f24c80fb020 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.702+0000 7f24cfd69700 1 -- 172.21.15.162:0/3648751473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 msgr2=0x7f24c80fb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.702+0000 7f24cfd69700 1 --2- 172.21.15.162:0/3648751473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 0x7f24c80fb440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f24c4009730 tx=0x7f24c40090f0).stop 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.702+0000 7f24cfd69700 1 -- 172.21.15.162:0/3648751473 shutdown_connections 2024-02-13T00:32:15.223 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.702+0000 7f24cfd69700 1 --2- 172.21.15.162:0/3648751473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 0x7f24c80fb440 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.702+0000 7f24cfd69700 1 -- 172.21.15.162:0/3648751473 >> 172.21.15.162:0/3648751473 conn(0x7f24c80f6820 msgr2=0x7f24c80f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.702+0000 7f24cfd69700 1 -- 172.21.15.162:0/3648751473 shutdown_connections 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.702+0000 7f24cfd69700 1 -- 172.21.15.162:0/3648751473 wait complete. 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.703+0000 7f24cfd69700 1 librados: starting msgr at 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.703+0000 7f24cfd69700 1 librados: starting objecter 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.703+0000 7f24cfd69700 1 Processor -- start 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.703+0000 7f24cfd69700 1 -- start start 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.703+0000 7f24cfd69700 1 librados: setting wanted keys 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.703+0000 7f24cfd69700 1 librados: calling monclient init 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.703+0000 7f24cfd69700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 0x7f24c810d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.703+0000 7f24cfd69700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c810d570 con 0x7f24c80fb020 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.703+0000 7f24cdb05700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 0x7f24c810d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:15.224 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.703+0000 7f24cdb05700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 0x7f24c810d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38844/0 (socket says 172.21.15.162:38844) 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.703+0000 7f24cdb05700 1 -- 172.21.15.162:0/2884813135 learned_addr learned my addr 172.21.15.162:0/2884813135 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.703+0000 7f24cdb05700 1 -- 172.21.15.162:0/2884813135 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c4009370 con 0x7f24c80fb020 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.704+0000 7f24cdb05700 1 --2- 172.21.15.162:0/2884813135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 0x7f24c810d080 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f24c4000c00 tx=0x7f24c40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.704+0000 7f24beffd700 10 client.?.objecter ms_handle_connect 0x7f24c80fb020 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.704+0000 7f24beffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.704+0000 7f24beffd700 1 -- 172.21.15.162:0/2884813135 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24c401a430 con 0x7f24c80fb020 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.704+0000 7f24beffd700 1 -- 172.21.15.162:0/2884813135 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24c401aa40 con 0x7f24c80fb020 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.704+0000 7f24beffd700 1 -- 172.21.15.162:0/2884813135 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24c4023b00 con 0x7f24c80fb020 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.704+0000 7f24cfd69700 1 -- 172.21.15.162:0/2884813135 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c810d770 con 0x7f24c80fb020 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.704+0000 7f24cfd69700 10 client.14112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.704+0000 7f24cfd69700 1 -- 172.21.15.162:0/2884813135 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c810dc00 con 0x7f24c80fb020 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.704+0000 7f24cfd69700 1 librados: init done 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.705+0000 7f24beffd700 1 -- 172.21.15.162:0/2884813135 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 4) v1 ==== 38571+0+0 (secure 0 0 0) 0x7f24c4023410 con 0x7f24c80fb020 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.705+0000 7f24beffd700 1 --2- 172.21.15.162:0/2884813135 >> v2:172.21.15.162:6800/2775563314 conn(0x7f24b4033f20 0x7f24b40363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:15.225 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.705+0000 7f24beffd700 1 -- 172.21.15.162:0/2884813135 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f24c4044d90 con 0x7f24c80fb020 2024-02-13T00:32:15.226 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.705+0000 7f24beffd700 10 client.14112.objecter ms_dispatch 0x7f24c8001550 osd_map(1..1 src has 1..1) v4 2024-02-13T00:32:15.226 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.705+0000 7f24beffd700 3 client.14112.objecter handle_osd_map got epochs [1,1] > 0 2024-02-13T00:32:15.226 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.705+0000 7f24beffd700 3 client.14112.objecter handle_osd_map decoding full epoch 1 2024-02-13T00:32:15.226 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.705+0000 7f24cfd69700 1 -- 172.21.15.162:0/2884813135 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c8046d80 con 0x7f24c80fb020 2024-02-13T00:32:15.226 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.705+0000 7f24beffd700 20 client.14112.objecter dump_active .. 0 homeless 2024-02-13T00:32:15.226 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.705+0000 7f24cd304700 1 --2- 172.21.15.162:0/2884813135 >> v2:172.21.15.162:6800/2775563314 conn(0x7f24b4033f20 0x7f24b40363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:15.226 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.705+0000 7f24beffd700 10 client.14112.objecter ms_handle_connect 0x7f24b4033f20 2024-02-13T00:32:15.226 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.705+0000 7f24cd304700 1 --2- 172.21.15.162:0/2884813135 >> v2:172.21.15.162:6800/2775563314 conn(0x7f24b4033f20 0x7f24b40363e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f24b8009780 tx=0x7f24b8006cc0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:15.226 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.709+0000 7f24beffd700 1 -- 172.21.15.162:0/2884813135 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f24c401e020 con 0x7f24c80fb020 2024-02-13T00:32:15.226 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.875+0000 7f24cfd69700 1 -- 172.21.15.162:0/2884813135 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f24c80fce60 con 0x7f24c80fb020 2024-02-13T00:32:15.226 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.876+0000 7f24beffd700 1 -- 172.21.15.162:0/2884813135 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+307 (secure 0 0 0) 0x7f24c4019570 con 0x7f24c80fb020 2024-02-13T00:32:15.226 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 10 librados: watch_flush enter 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 10 librados: watch_flush exit 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 20 client.14112.objecter shutdown clearing up homeless session... 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 10 client.14112.objecter successfully canceled tick 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 1 -- 172.21.15.162:0/2884813135 >> v2:172.21.15.162:6800/2775563314 conn(0x7f24b4033f20 msgr2=0x7f24b40363e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 1 --2- 172.21.15.162:0/2884813135 >> v2:172.21.15.162:6800/2775563314 conn(0x7f24b4033f20 0x7f24b40363e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f24b8009780 tx=0x7f24b8006cc0).stop 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 1 -- 172.21.15.162:0/2884813135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 msgr2=0x7f24c810d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 1 --2- 172.21.15.162:0/2884813135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 0x7f24c810d080 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f24c4000c00 tx=0x7f24c40090f0).stop 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 1 -- 172.21.15.162:0/2884813135 shutdown_connections 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 1 --2- 172.21.15.162:0/2884813135 >> v2:172.21.15.162:6800/2775563314 conn(0x7f24b4033f20 0x7f24b40363e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 1 --2- 172.21.15.162:0/2884813135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24c80fb020 0x7f24c810d080 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 1 -- 172.21.15.162:0/2884813135 >> 172.21.15.162:0/2884813135 conn(0x7f24c80f6820 msgr2=0x7f24c8070850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 1 -- 172.21.15.162:0/2884813135 shutdown_connections 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 1 -- 172.21.15.162:0/2884813135 wait complete. 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:14.878+0000 7f24cfd69700 1 librados: shutdown 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:15.227 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: [global] 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: fsid = d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: [mgr] 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: [osd] 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-13T00:32:15.228 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-13T00:32:15.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:14.877021+0000 mon.a (mon.0) 25 : audit [INF] from='client.? 172.21.15.162:0/2884813135' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-13T00:32:18.214 INFO:teuthology.orchestra.run.smithi162.stderr:Enabling cephadm module... 2024-02-13T00:32:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.771+0000 7fcbde52a700 1 Processor -- start 2024-02-13T00:32:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.771+0000 7fcbde52a700 1 -- start start 2024-02-13T00:32:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.772+0000 7fcbde52a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 0x7fcbd81011f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.772+0000 7fcbde52a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd80fcbe0 con 0x7fcbd8070bc0 2024-02-13T00:32:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.772+0000 7fcbd7fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 0x7fcbd81011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.772+0000 7fcbd7fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 0x7fcbd81011f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38860/0 (socket says 172.21.15.162:38860) 2024-02-13T00:32:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.772+0000 7fcbd7fff700 1 -- 172.21.15.162:0/3979118396 learned_addr learned my addr 172.21.15.162:0/3979118396 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.772+0000 7fcbd7fff700 1 -- 172.21.15.162:0/3979118396 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd81016e0 con 0x7fcbd8070bc0 2024-02-13T00:32:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.772+0000 7fcbd7fff700 1 --2- 172.21.15.162:0/3979118396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 0x7fcbd81011f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fcbc8009730 tx=0x7fcbc80090f0).ready entity=mon.0 client_cookie=67c2bbf7a47baeb8 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.772+0000 7fcbd6ffd700 1 -- 172.21.15.162:0/3979118396 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbc800b580 con 0x7fcbd8070bc0 2024-02-13T00:32:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.772+0000 7fcbd6ffd700 1 -- 172.21.15.162:0/3979118396 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbc800bb90 con 0x7fcbd8070bc0 2024-02-13T00:32:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.772+0000 7fcbd6ffd700 1 -- 172.21.15.162:0/3979118396 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbc8011c30 con 0x7fcbd8070bc0 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.773+0000 7fcbde52a700 1 -- 172.21.15.162:0/3979118396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 msgr2=0x7fcbd81011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.773+0000 7fcbde52a700 1 --2- 172.21.15.162:0/3979118396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 0x7fcbd81011f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fcbc8009730 tx=0x7fcbc80090f0).stop 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.773+0000 7fcbde52a700 1 -- 172.21.15.162:0/3979118396 shutdown_connections 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.773+0000 7fcbde52a700 1 --2- 172.21.15.162:0/3979118396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 0x7fcbd81011f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.773+0000 7fcbde52a700 1 -- 172.21.15.162:0/3979118396 >> 172.21.15.162:0/3979118396 conn(0x7fcbd80f67d0 msgr2=0x7fcbd80f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.773+0000 7fcbde52a700 1 -- 172.21.15.162:0/3979118396 shutdown_connections 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.773+0000 7fcbde52a700 1 -- 172.21.15.162:0/3979118396 wait complete. 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.773+0000 7fcbde52a700 1 librados: starting msgr at 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.773+0000 7fcbde52a700 1 librados: starting objecter 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.774+0000 7fcbde52a700 1 Processor -- start 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.774+0000 7fcbde52a700 1 -- start start 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.774+0000 7fcbde52a700 1 librados: setting wanted keys 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.774+0000 7fcbde52a700 1 librados: calling monclient init 2024-02-13T00:32:18.780 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.774+0000 7fcbde52a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 0x7fcbd810d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.774+0000 7fcbde52a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd810d530 con 0x7fcbd8070bc0 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.774+0000 7fcbd7fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 0x7fcbd810d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.774+0000 7fcbd7fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 0x7fcbd810d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38868/0 (socket says 172.21.15.162:38868) 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.774+0000 7fcbd7fff700 1 -- 172.21.15.162:0/3520461277 learned_addr learned my addr 172.21.15.162:0/3520461277 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.774+0000 7fcbd7fff700 1 -- 172.21.15.162:0/3520461277 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc8009370 con 0x7fcbd8070bc0 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.775+0000 7fcbd7fff700 1 --2- 172.21.15.162:0/3520461277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 0x7fcbd810d040 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fcbc800bf60 tx=0x7fcbc80090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.775+0000 7fcbd57fa700 10 client.?.objecter ms_handle_connect 0x7fcbd8070bc0 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.775+0000 7fcbd57fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.775+0000 7fcbd57fa700 1 -- 172.21.15.162:0/3520461277 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbc8011bf0 con 0x7fcbd8070bc0 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.775+0000 7fcbd57fa700 1 -- 172.21.15.162:0/3520461277 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbc801aaf0 con 0x7fcbd8070bc0 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.775+0000 7fcbd57fa700 1 -- 172.21.15.162:0/3520461277 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbc80229c0 con 0x7fcbd8070bc0 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.775+0000 7fcbde52a700 1 -- 172.21.15.162:0/3520461277 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd810d730 con 0x7fcbd8070bc0 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.775+0000 7fcbde52a700 10 client.14114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.775+0000 7fcbde52a700 1 -- 172.21.15.162:0/3520461277 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd810dbc0 con 0x7fcbd8070bc0 2024-02-13T00:32:18.781 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.775+0000 7fcbde52a700 1 librados: init done 2024-02-13T00:32:18.782 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.776+0000 7fcbde52a700 1 -- 172.21.15.162:0/3520461277 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd8046d80 con 0x7fcbd8070bc0 2024-02-13T00:32:18.783 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.776+0000 7fcbd57fa700 1 -- 172.21.15.162:0/3520461277 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 4) v1 ==== 38571+0+0 (secure 0 0 0) 0x7fcbc801ac60 con 0x7fcbd8070bc0 2024-02-13T00:32:18.783 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.777+0000 7fcbd57fa700 1 --2- 172.21.15.162:0/3520461277 >> v2:172.21.15.162:6800/2775563314 conn(0x7fcbc0033ed0 0x7fcbc0036390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:18.783 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.777+0000 7fcbd57fa700 1 -- 172.21.15.162:0/3520461277 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fcbc8044c60 con 0x7fcbd8070bc0 2024-02-13T00:32:18.783 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.777+0000 7fcbd57fa700 10 client.14114.objecter ms_dispatch 0x7fcbd8001550 osd_map(1..1 src has 1..1) v4 2024-02-13T00:32:18.783 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.777+0000 7fcbd57fa700 3 client.14114.objecter handle_osd_map got epochs [1,1] > 0 2024-02-13T00:32:18.783 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.777+0000 7fcbd57fa700 3 client.14114.objecter handle_osd_map decoding full epoch 1 2024-02-13T00:32:18.783 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.777+0000 7fcbd57fa700 20 client.14114.objecter dump_active .. 0 homeless 2024-02-13T00:32:18.783 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.777+0000 7fcbd77fe700 1 --2- 172.21.15.162:0/3520461277 >> v2:172.21.15.162:6800/2775563314 conn(0x7fcbc0033ed0 0x7fcbc0036390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:18.783 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.777+0000 7fcbd57fa700 10 client.14114.objecter ms_handle_connect 0x7fcbc0033ed0 2024-02-13T00:32:18.783 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.777+0000 7fcbd77fe700 1 --2- 172.21.15.162:0/3520461277 >> v2:172.21.15.162:6800/2775563314 conn(0x7fcbc0033ed0 0x7fcbc0036390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fcbcc009780 tx=0x7fcbcc006cc0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:18.784 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.781+0000 7fcbd57fa700 1 -- 172.21.15.162:0/3520461277 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fcbc80223b0 con 0x7fcbd8070bc0 2024-02-13T00:32:18.985 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:18.982+0000 7fcbde52a700 1 -- 172.21.15.162:0/3520461277 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fcbd8102370 con 0x7fcbd8070bc0 2024-02-13T00:32:19.037 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.034+0000 7fcbd57fa700 1 -- 172.21.15.162:0/3520461277 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mgrmap(e 5) v1 ==== 38582+0+0 (secure 0 0 0) 0x7fcbc8023c70 con 0x7fcbd8070bc0 2024-02-13T00:32:19.038 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.035+0000 7fcbd57fa700 1 -- 172.21.15.162:0/3520461277 <== mon.0 v2:172.21.15.162:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fcbc802b590 con 0x7fcbd8070bc0 2024-02-13T00:32:19.041 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.037+0000 7fcbde52a700 10 librados: watch_flush enter 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.037+0000 7fcbde52a700 10 librados: watch_flush exit 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.037+0000 7fcbde52a700 20 client.14114.objecter shutdown clearing up homeless session... 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.037+0000 7fcbde52a700 10 client.14114.objecter successfully canceled tick 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.037+0000 7fcbde52a700 1 -- 172.21.15.162:0/3520461277 >> v2:172.21.15.162:6800/2775563314 conn(0x7fcbc0033ed0 msgr2=0x7fcbc0036390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.037+0000 7fcbde52a700 1 --2- 172.21.15.162:0/3520461277 >> v2:172.21.15.162:6800/2775563314 conn(0x7fcbc0033ed0 0x7fcbc0036390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fcbcc009780 tx=0x7fcbcc006cc0).stop 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.037+0000 7fcbde52a700 1 -- 172.21.15.162:0/3520461277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 msgr2=0x7fcbd810d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.037+0000 7fcbde52a700 1 --2- 172.21.15.162:0/3520461277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 0x7fcbd810d040 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fcbc800bf60 tx=0x7fcbc80090f0).stop 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.037+0000 7fcbde52a700 1 -- 172.21.15.162:0/3520461277 shutdown_connections 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.037+0000 7fcbde52a700 1 --2- 172.21.15.162:0/3520461277 >> v2:172.21.15.162:6800/2775563314 conn(0x7fcbc0033ed0 0x7fcbc0036390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.037+0000 7fcbde52a700 1 --2- 172.21.15.162:0/3520461277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbd8070bc0 0x7fcbd810d040 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.037+0000 7fcbde52a700 1 -- 172.21.15.162:0/3520461277 >> 172.21.15.162:0/3520461277 conn(0x7fcbd80f67d0 msgr2=0x7fcbd80f80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.038+0000 7fcbde52a700 1 -- 172.21.15.162:0/3520461277 shutdown_connections 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.038+0000 7fcbde52a700 1 -- 172.21.15.162:0/3520461277 wait complete. 2024-02-13T00:32:19.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:19.038+0000 7fcbde52a700 1 librados: shutdown 2024-02-13T00:32:19.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:18.983233+0000 mon.a (mon.0) 26 : audit [INF] from='client.? 172.21.15.162:0/3520461277' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-13T00:32:19.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ignoring --setuser ceph since I am not root 2024-02-13T00:32:19.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ignoring --setgroup ceph since I am not root 2024-02-13T00:32:19.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:19.187+0000 7ff2db0da680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T00:32:19.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:19.268+0000 7ff2db0da680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T00:32:19.715 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:19.632+0000 7ff2db0da680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T00:32:20.316 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.311+0000 7f550d146700 1 Processor -- start 2024-02-13T00:32:20.316 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.311+0000 7f550d146700 1 -- start start 2024-02-13T00:32:20.316 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.311+0000 7f550d146700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 0x7f55080fb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:20.316 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.311+0000 7f550d146700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55080fb930 con 0x7f55080fb020 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.311+0000 7f5506d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 0x7f55080fb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.311+0000 7f5506d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 0x7f55080fb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38896/0 (socket says 172.21.15.162:38896) 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.311+0000 7f5506d9d700 1 -- 172.21.15.162:0/4264844322 learned_addr learned my addr 172.21.15.162:0/4264844322 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.312+0000 7f5506d9d700 1 -- 172.21.15.162:0/4264844322 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55080fba70 con 0x7f55080fb020 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.312+0000 7f5506d9d700 1 --2- 172.21.15.162:0/4264844322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 0x7f55080fb440 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f54f800b9d0 tx=0x7f54f800b3b0).ready entity=mon.0 client_cookie=c4233a53b90af050 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.312+0000 7f5505d9b700 1 -- 172.21.15.162:0/4264844322 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54f800da70 con 0x7f55080fb020 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.312+0000 7f5505d9b700 1 -- 172.21.15.162:0/4264844322 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54f800dbd0 con 0x7f55080fb020 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.312+0000 7f5505d9b700 1 -- 172.21.15.162:0/4264844322 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54f800f430 con 0x7f55080fb020 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.313+0000 7f550d146700 1 -- 172.21.15.162:0/4264844322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 msgr2=0x7f55080fb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.313+0000 7f550d146700 1 --2- 172.21.15.162:0/4264844322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 0x7f55080fb440 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f54f800b9d0 tx=0x7f54f800b3b0).stop 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.313+0000 7f550d146700 1 -- 172.21.15.162:0/4264844322 shutdown_connections 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.313+0000 7f550d146700 1 --2- 172.21.15.162:0/4264844322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 0x7f55080fb440 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.313+0000 7f550d146700 1 -- 172.21.15.162:0/4264844322 >> 172.21.15.162:0/4264844322 conn(0x7f55080f6820 msgr2=0x7f55080f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:20.317 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.313+0000 7f550d146700 1 -- 172.21.15.162:0/4264844322 shutdown_connections 2024-02-13T00:32:20.318 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.313+0000 7f550d146700 1 -- 172.21.15.162:0/4264844322 wait complete. 2024-02-13T00:32:20.318 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.313+0000 7f550d146700 1 librados: starting msgr at 2024-02-13T00:32:20.318 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.313+0000 7f550d146700 1 librados: starting objecter 2024-02-13T00:32:20.318 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.314+0000 7f550d146700 1 Processor -- start 2024-02-13T00:32:20.320 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.314+0000 7f550d146700 1 -- start start 2024-02-13T00:32:20.320 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.314+0000 7f550d146700 1 librados: setting wanted keys 2024-02-13T00:32:20.320 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.314+0000 7f550d146700 1 librados: calling monclient init 2024-02-13T00:32:20.320 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.314+0000 7f550d146700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 0x7f5508108bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:20.320 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.314+0000 7f550d146700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55081090e0 con 0x7f55080fb020 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.314+0000 7f5506d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 0x7f5508108bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.314+0000 7f5506d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 0x7f5508108bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38902/0 (socket says 172.21.15.162:38902) 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.314+0000 7f5506d9d700 1 -- 172.21.15.162:0/631031476 learned_addr learned my addr 172.21.15.162:0/631031476 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.315+0000 7f5506d9d700 1 -- 172.21.15.162:0/631031476 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f800b610 con 0x7f55080fb020 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.315+0000 7f5506d9d700 1 --2- 172.21.15.162:0/631031476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 0x7f5508108bf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f54f800f850 tx=0x7f54f800b3b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.315+0000 7f54effff700 10 client.?.objecter ms_handle_connect 0x7f55080fb020 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.315+0000 7f54effff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.315+0000 7f54effff700 1 -- 172.21.15.162:0/631031476 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54f800da70 con 0x7f55080fb020 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.315+0000 7f54effff700 1 -- 172.21.15.162:0/631031476 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54f8010c20 con 0x7f55080fb020 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.315+0000 7f550d146700 1 -- 172.21.15.162:0/631031476 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55081092e0 con 0x7f55080fb020 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.315+0000 7f54effff700 1 -- 172.21.15.162:0/631031476 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54f8019d10 con 0x7f55080fb020 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.315+0000 7f550d146700 10 client.14118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.315+0000 7f550d146700 1 -- 172.21.15.162:0/631031476 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5508109770 con 0x7f55080fb020 2024-02-13T00:32:20.321 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.315+0000 7f550d146700 1 librados: init done 2024-02-13T00:32:20.322 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.316+0000 7f54effff700 1 -- 172.21.15.162:0/631031476 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f54f802d440 con 0x7f55080fb020 2024-02-13T00:32:20.322 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.316+0000 7f54effff700 1 --2- 172.21.15.162:0/631031476 >> v2:172.21.15.162:6800/2775563314 conn(0x7f54f0033f20 0x7f54f00363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:20.322 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.316+0000 7f54effff700 1 -- 172.21.15.162:0/631031476 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f54f8047680 con 0x7f55080fb020 2024-02-13T00:32:20.322 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.316+0000 7f54effff700 10 client.14118.objecter ms_dispatch 0x7f5508001550 osd_map(1..1 src has 1..1) v4 2024-02-13T00:32:20.322 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.316+0000 7f54effff700 3 client.14118.objecter handle_osd_map got epochs [1,1] > 0 2024-02-13T00:32:20.322 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.316+0000 7f54effff700 3 client.14118.objecter handle_osd_map decoding full epoch 1 2024-02-13T00:32:20.322 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.316+0000 7f550659c700 1 -- 172.21.15.162:0/631031476 >> v2:172.21.15.162:6800/2775563314 conn(0x7f54f0033f20 msgr2=0x7f54f00363e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/2775563314 2024-02-13T00:32:20.322 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.316+0000 7f550659c700 1 --2- 172.21.15.162:0/631031476 >> v2:172.21.15.162:6800/2775563314 conn(0x7f54f0033f20 0x7f54f00363e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-13T00:32:20.322 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.316+0000 7f54effff700 20 client.14118.objecter dump_active .. 0 homeless 2024-02-13T00:32:20.322 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.316+0000 7f550d146700 1 -- 172.21.15.162:0/631031476 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550805a3b0 con 0x7f55080fb020 2024-02-13T00:32:20.324 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.321+0000 7f54effff700 1 -- 172.21.15.162:0/631031476 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f54f802b980 con 0x7f55080fb020 2024-02-13T00:32:20.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:19.036296+0000 mon.a (mon.0) 27 : audit [INF] from='client.? 172.21.15.162:0/3520461277' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-13T00:32:20.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:19.036363+0000 mon.a (mon.0) 28 : cluster [DBG] mgrmap e5: y(active, since 12s) 2024-02-13T00:32:20.520 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.517+0000 7f550659c700 1 -- 172.21.15.162:0/631031476 >> v2:172.21.15.162:6800/2775563314 conn(0x7f54f0033f20 msgr2=0x7f54f00363e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/2775563314 2024-02-13T00:32:20.520 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.517+0000 7f550659c700 1 --2- 172.21.15.162:0/631031476 >> v2:172.21.15.162:6800/2775563314 conn(0x7f54f0033f20 0x7f54f00363e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-13T00:32:20.573 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.569+0000 7f550d146700 1 -- 172.21.15.162:0/631031476 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f5508046d80 con 0x7f55080fb020 2024-02-13T00:32:20.573 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.570+0000 7f54effff700 1 -- 172.21.15.162:0/631031476 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f54f801f290 con 0x7f55080fb020 2024-02-13T00:32:20.573 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: { 2024-02-13T00:32:20.573 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-13T00:32:20.573 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "available": true, 2024-02-13T00:32:20.573 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-13T00:32:20.573 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-13T00:32:20.574 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: } 2024-02-13T00:32:20.576 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 10 librados: watch_flush enter 2024-02-13T00:32:20.577 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 10 librados: watch_flush exit 2024-02-13T00:32:20.577 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 20 client.14118.objecter shutdown clearing up homeless session... 2024-02-13T00:32:20.577 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 10 client.14118.objecter successfully canceled tick 2024-02-13T00:32:20.577 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 1 -- 172.21.15.162:0/631031476 >> v2:172.21.15.162:6800/2775563314 conn(0x7f54f0033f20 msgr2=0x7f54f00363e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T00:32:20.577 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 1 --2- 172.21.15.162:0/631031476 >> v2:172.21.15.162:6800/2775563314 conn(0x7f54f0033f20 0x7f54f00363e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:32:20.577 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 1 -- 172.21.15.162:0/631031476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 msgr2=0x7f5508108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:20.577 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 1 --2- 172.21.15.162:0/631031476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 0x7f5508108bf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f54f800f850 tx=0x7f54f800b3b0).stop 2024-02-13T00:32:20.577 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 1 -- 172.21.15.162:0/631031476 shutdown_connections 2024-02-13T00:32:20.577 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 1 --2- 172.21.15.162:0/631031476 >> v2:172.21.15.162:6800/2775563314 conn(0x7f54f0033f20 0x7f54f00363e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:32:20.577 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 1 --2- 172.21.15.162:0/631031476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55080fb020 0x7f5508108bf0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:20.577 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 1 -- 172.21.15.162:0/631031476 >> 172.21.15.162:0/631031476 conn(0x7f55080f6820 msgr2=0x7f55080ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:20.577 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 1 -- 172.21.15.162:0/631031476 shutdown_connections 2024-02-13T00:32:20.578 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 1 -- 172.21.15.162:0/631031476 wait complete. 2024-02-13T00:32:20.578 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:20.572+0000 7f550d146700 1 librados: shutdown 2024-02-13T00:32:20.715 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:20.402+0000 7ff2db0da680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T00:32:20.715 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:20.488+0000 7ff2db0da680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T00:32:21.042 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:20.712+0000 7ff2db0da680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T00:32:21.253 INFO:teuthology.orchestra.run.smithi162.stderr:Waiting for the mgr to restart... 2024-02-13T00:32:21.253 INFO:teuthology.orchestra.run.smithi162.stderr:Waiting for mgr epoch 5... 2024-02-13T00:32:21.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:20.570852+0000 mon.a (mon.0) 29 : audit [DBG] from='client.? 172.21.15.162:0/631031476' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-13T00:32:21.815 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.795+0000 7fa04a4e6700 1 Processor -- start 2024-02-13T00:32:21.815 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.795+0000 7fa04a4e6700 1 -- start start 2024-02-13T00:32:21.815 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.795+0000 7fa04a4e6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 0x7fa0440fede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:21.815 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.795+0000 7fa04a4e6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0440ff2d0 con 0x7fa044070c00 2024-02-13T00:32:21.815 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.795+0000 7fa043fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 0x7fa0440fede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:21.815 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.795+0000 7fa043fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 0x7fa0440fede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38912/0 (socket says 172.21.15.162:38912) 2024-02-13T00:32:21.815 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.795+0000 7fa043fff700 1 -- 172.21.15.162:0/2790285410 learned_addr learned my addr 172.21.15.162:0/2790285410 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.795+0000 7fa043fff700 1 -- 172.21.15.162:0/2790285410 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0440ff410 con 0x7fa044070c00 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.795+0000 7fa043fff700 1 --2- 172.21.15.162:0/2790285410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 0x7fa0440fede0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa034009730 tx=0x7fa0340090f0).ready entity=mon.0 client_cookie=105387ae7693d657 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.795+0000 7fa042ffd700 1 -- 172.21.15.162:0/2790285410 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa03400b580 con 0x7fa044070c00 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.796+0000 7fa042ffd700 1 -- 172.21.15.162:0/2790285410 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa03400bb90 con 0x7fa044070c00 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.796+0000 7fa042ffd700 1 -- 172.21.15.162:0/2790285410 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa034011c30 con 0x7fa044070c00 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.796+0000 7fa04a4e6700 1 -- 172.21.15.162:0/2790285410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 msgr2=0x7fa0440fede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.796+0000 7fa04a4e6700 1 --2- 172.21.15.162:0/2790285410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 0x7fa0440fede0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa034009730 tx=0x7fa0340090f0).stop 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.796+0000 7fa04a4e6700 1 -- 172.21.15.162:0/2790285410 shutdown_connections 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.796+0000 7fa04a4e6700 1 --2- 172.21.15.162:0/2790285410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 0x7fa0440fede0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.796+0000 7fa04a4e6700 1 -- 172.21.15.162:0/2790285410 >> 172.21.15.162:0/2790285410 conn(0x7fa0440f6820 msgr2=0x7fa0440f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.796+0000 7fa04a4e6700 1 -- 172.21.15.162:0/2790285410 shutdown_connections 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.796+0000 7fa04a4e6700 1 -- 172.21.15.162:0/2790285410 wait complete. 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.797+0000 7fa04a4e6700 1 librados: starting msgr at 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.797+0000 7fa04a4e6700 1 librados: starting objecter 2024-02-13T00:32:21.816 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.797+0000 7fa04a4e6700 1 Processor -- start 2024-02-13T00:32:21.817 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.797+0000 7fa04a4e6700 1 -- start start 2024-02-13T00:32:21.817 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.797+0000 7fa04a4e6700 1 librados: setting wanted keys 2024-02-13T00:32:21.817 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.797+0000 7fa04a4e6700 1 librados: calling monclient init 2024-02-13T00:32:21.817 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.797+0000 7fa04a4e6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 0x7fa0440fc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:21.817 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.797+0000 7fa04a4e6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0440face0 con 0x7fa044070c00 2024-02-13T00:32:21.817 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.797+0000 7fa043fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 0x7fa0440fc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:21.817 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:21.675+0000 7ff2db0da680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T00:32:21.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.797+0000 7fa043fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 0x7fa0440fc860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38928/0 (socket says 172.21.15.162:38928) 2024-02-13T00:32:21.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.797+0000 7fa043fff700 1 -- 172.21.15.162:0/3710722353 learned_addr learned my addr 172.21.15.162:0/3710722353 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:21.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.798+0000 7fa043fff700 1 -- 172.21.15.162:0/3710722353 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa034009370 con 0x7fa044070c00 2024-02-13T00:32:21.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.798+0000 7fa043fff700 1 --2- 172.21.15.162:0/3710722353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 0x7fa0440fc860 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fa034011630 tx=0x7fa0340090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:21.819 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.798+0000 7fa0417fa700 10 client.?.objecter ms_handle_connect 0x7fa044070c00 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.798+0000 7fa0417fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.798+0000 7fa0417fa700 1 -- 172.21.15.162:0/3710722353 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa03401a430 con 0x7fa044070c00 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.798+0000 7fa0417fa700 1 -- 172.21.15.162:0/3710722353 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa03401aa40 con 0x7fa044070c00 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.798+0000 7fa04a4e6700 1 -- 172.21.15.162:0/3710722353 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0440fae80 con 0x7fa044070c00 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.798+0000 7fa0417fa700 1 -- 172.21.15.162:0/3710722353 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0340239a0 con 0x7fa044070c00 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.798+0000 7fa04a4e6700 10 client.14120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.798+0000 7fa04a4e6700 1 -- 172.21.15.162:0/3710722353 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0440fb310 con 0x7fa044070c00 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.798+0000 7fa04a4e6700 1 librados: init done 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.799+0000 7fa0417fa700 1 -- 172.21.15.162:0/3710722353 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 38582+0+0 (secure 0 0 0) 0x7fa034023c20 con 0x7fa044070c00 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.799+0000 7fa0417fa700 1 --2- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/2775563314 conn(0x7fa02c033f20 0x7fa02c0363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.799+0000 7fa0417fa700 1 -- 172.21.15.162:0/3710722353 --> v2:172.21.15.162:6800/2775563314 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa02c036a60 con 0x7fa02c033f20 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.799+0000 7fa0417fa700 1 -- 172.21.15.162:0/3710722353 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa034044b70 con 0x7fa044070c00 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.799+0000 7fa0417fa700 10 client.14120.objecter ms_dispatch 0x7fa044001550 osd_map(1..1 src has 1..1) v4 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.799+0000 7fa0417fa700 3 client.14120.objecter handle_osd_map got epochs [1,1] > 0 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.799+0000 7fa0417fa700 3 client.14120.objecter handle_osd_map decoding full epoch 1 2024-02-13T00:32:21.820 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.799+0000 7fa0437fe700 1 -- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/2775563314 conn(0x7fa02c033f20 msgr2=0x7fa02c0363e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/2775563314 2024-02-13T00:32:21.821 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.799+0000 7fa0417fa700 20 client.14120.objecter dump_active .. 0 homeless 2024-02-13T00:32:21.821 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:21.799+0000 7fa0437fe700 1 --2- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/2775563314 conn(0x7fa02c033f20 0x7fa02c0363e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-13T00:32:22.003 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:22.000+0000 7fa0437fe700 1 -- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/2775563314 conn(0x7fa02c033f20 msgr2=0x7fa02c0363e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/2775563314 2024-02-13T00:32:22.003 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:22.000+0000 7fa0437fe700 1 --2- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/2775563314 conn(0x7fa02c033f20 0x7fa02c0363e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-13T00:32:22.079 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:21.811+0000 7ff2db0da680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T00:32:22.079 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:21.890+0000 7ff2db0da680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T00:32:22.079 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:21.977+0000 7ff2db0da680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T00:32:22.404 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:22.401+0000 7fa0437fe700 1 -- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/2775563314 conn(0x7fa02c033f20 msgr2=0x7fa02c0363e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/2775563314 2024-02-13T00:32:22.404 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:22.401+0000 7fa0437fe700 1 --2- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/2775563314 conn(0x7fa02c033f20 0x7fa02c0363e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-13T00:32:22.450 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:22.173+0000 7ff2db0da680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T00:32:22.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:22.447+0000 7ff2db0da680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T00:32:22.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:22.543+0000 7ff2db0da680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T00:32:23.205 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:23.202+0000 7fa0437fe700 1 -- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/2775563314 conn(0x7fa02c033f20 msgr2=0x7fa02c0363e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/2775563314 2024-02-13T00:32:23.205 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:23.202+0000 7fa0437fe700 1 --2- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/2775563314 conn(0x7fa02c033f20 0x7fa02c0363e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-13T00:32:23.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:23.256+0000 7ff2db0da680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T00:32:23.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:23.337+0000 7ff2db0da680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T00:32:23.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:23.428+0000 7ff2db0da680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T00:32:23.860 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:23.598+0000 7ff2db0da680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T00:32:24.119 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:23.857+0000 7ff2db0da680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T00:32:24.119 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:23.984+0000 7ff2db0da680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T00:32:24.370 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:24.116+0000 7ff2db0da680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T00:32:24.370 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:24.284+0000 7ff2db0da680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T00:32:24.385 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:24.382+0000 7fa0417fa700 1 -- 172.21.15.162:0/3710722353 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mgrmap(e 6) v1 ==== 38441+0+0 (secure 0 0 0) 0x7fa03402c3f0 con 0x7fa044070c00 2024-02-13T00:32:24.385 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:24.382+0000 7fa0417fa700 1 -- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/2775563314 conn(0x7fa02c033f20 msgr2=0x7fa02c0363e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T00:32:24.385 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:24.382+0000 7fa0417fa700 1 --2- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/2775563314 conn(0x7fa02c033f20 0x7fa02c0363e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:32:25.386 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.382+0000 7fa0417fa700 1 -- 172.21.15.162:0/3710722353 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mgrmap(e 7) v1 ==== 38476+0+0 (secure 0 0 0) 0x7fa03402b350 con 0x7fa044070c00 2024-02-13T00:32:25.386 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.383+0000 7fa0417fa700 1 --2- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa02c0379d0 0x7fa02c039dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:25.386 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.383+0000 7fa0417fa700 1 -- 172.21.15.162:0/3710722353 --> v2:172.21.15.162:6800/4121673848 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa02c036a60 con 0x7fa02c0379d0 2024-02-13T00:32:25.386 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.383+0000 7fa0437fe700 1 --2- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa02c0379d0 0x7fa02c039dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:25.387 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.384+0000 7fa0437fe700 1 --2- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa02c0379d0 0x7fa02c039dc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa038003850 tx=0x7fa038006e20).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:25.387 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.384+0000 7fa0417fa700 10 client.14120.objecter ms_handle_connect 0x7fa02c0379d0 2024-02-13T00:32:25.387 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.384+0000 7fa0417fa700 1 -- 172.21.15.162:0/3710722353 <== mgr.14116 v2:172.21.15.162:6800/4121673848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fa02c036a60 con 0x7fa02c0379d0 2024-02-13T00:32:25.393 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.390+0000 7fa04a4e6700 1 -- 172.21.15.162:0/3710722353 --> v2:172.21.15.162:6800/4121673848 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa04405a3b0 con 0x7fa02c0379d0 2024-02-13T00:32:25.393 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa0417fa700 1 -- 172.21.15.162:0/3710722353 <== mgr.14116 v2:172.21.15.162:6800/4121673848 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa04405a3b0 con 0x7fa02c0379d0 2024-02-13T00:32:25.395 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: { 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "initialized": true 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: } 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 10 librados: watch_flush enter 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 10 librados: watch_flush exit 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 20 client.14120.objecter shutdown clearing up homeless session... 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 10 client.14120.objecter successfully canceled tick 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 1 -- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa02c0379d0 msgr2=0x7fa02c039dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 1 --2- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa02c0379d0 0x7fa02c039dc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa038003850 tx=0x7fa038006e20).stop 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 1 -- 172.21.15.162:0/3710722353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 msgr2=0x7fa0440fc860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 1 --2- 172.21.15.162:0/3710722353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 0x7fa0440fc860 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fa034011630 tx=0x7fa0340090f0).stop 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 1 -- 172.21.15.162:0/3710722353 shutdown_connections 2024-02-13T00:32:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 1 --2- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa02c0379d0 0x7fa02c039dc0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:25.397 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 1 --2- 172.21.15.162:0/3710722353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa044070c00 0x7fa0440fc860 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:25.397 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 1 --2- 172.21.15.162:0/3710722353 >> v2:172.21.15.162:6800/2775563314 conn(0x7fa02c033f20 0x7fa02c0363e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:32:25.397 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 1 -- 172.21.15.162:0/3710722353 >> 172.21.15.162:0/3710722353 conn(0x7fa0440f6820 msgr2=0x7fa0440f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:25.397 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 1 -- 172.21.15.162:0/3710722353 shutdown_connections 2024-02-13T00:32:25.397 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 1 -- 172.21.15.162:0/3710722353 wait complete. 2024-02-13T00:32:25.397 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:25.391+0000 7fa04a4e6700 1 librados: shutdown 2024-02-13T00:32:25.422 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:24.367+0000 7ff2db0da680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T00:32:25.422 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:24.380606+0000 mon.a (mon.0) 30 : cluster [INF] Active manager daemon y restarted 2024-02-13T00:32:25.422 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:24.380828+0000 mon.a (mon.0) 31 : cluster [INF] Activating manager daemon y 2024-02-13T00:32:25.422 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:24.383736+0000 mon.a (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-13T00:32:25.422 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:24.383858+0000 mon.a (mon.0) 33 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00311967s) 2024-02-13T00:32:25.422 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:24.384765+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T00:32:25.422 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:24.385323+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T00:32:25.422 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:24.385896+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T00:32:25.422 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:24.386038+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T00:32:25.423 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:24.386144+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T00:32:25.423 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:24.401136+0000 mon.a (mon.0) 39 : cluster [INF] Manager daemon y is now available 2024-02-13T00:32:25.423 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:32:24.410470+0000 mgr.y (mgr.14116) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-02-13T00:32:25.423 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:24.413029+0000 mon.a (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' 2024-02-13T00:32:25.423 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:24.497018+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' 2024-02-13T00:32:25.423 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:24.498040+0000 mon.a (mon.0) 42 : audit [DBG] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:32:25.423 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:24.499447+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:32:25.423 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:24.501400+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:32:25.423 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:24.514819+0000 mon.a (mon.0) 45 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T00:32:25.423 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:24.517420+0000 mon.a (mon.0) 46 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T00:32:26.035 INFO:teuthology.orchestra.run.smithi162.stderr:mgr epoch 5 is available 2024-02-13T00:32:26.036 INFO:teuthology.orchestra.run.smithi162.stderr:Setting orchestrator backend to cephadm... 2024-02-13T00:32:26.579 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.574+0000 7fc38f4b9700 1 Processor -- start 2024-02-13T00:32:26.579 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.575+0000 7fc38f4b9700 1 -- start start 2024-02-13T00:32:26.579 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.575+0000 7fc38f4b9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 0x7fc3880fed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:26.579 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.575+0000 7fc38f4b9700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3880ff280 con 0x7fc388070c20 2024-02-13T00:32:26.579 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.575+0000 7fc38d255700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 0x7fc3880fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:26.580 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.575+0000 7fc38d255700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 0x7fc3880fed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35604/0 (socket says 172.21.15.162:35604) 2024-02-13T00:32:26.580 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.575+0000 7fc38d255700 1 -- 172.21.15.162:0/3257886552 learned_addr learned my addr 172.21.15.162:0/3257886552 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:26.580 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.575+0000 7fc38d255700 1 -- 172.21.15.162:0/3257886552 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3880ff3c0 con 0x7fc388070c20 2024-02-13T00:32:26.580 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.575+0000 7fc38d255700 1 --2- 172.21.15.162:0/3257886552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 0x7fc3880fed90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc378009730 tx=0x7fc3780090f0).ready entity=mon.0 client_cookie=53b875bd17931c1b server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:26.580 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.576+0000 7fc37ffff700 1 -- 172.21.15.162:0/3257886552 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc37800b610 con 0x7fc388070c20 2024-02-13T00:32:26.580 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.576+0000 7fc37ffff700 1 -- 172.21.15.162:0/3257886552 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc37800bc20 con 0x7fc388070c20 2024-02-13T00:32:26.580 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.576+0000 7fc37ffff700 1 -- 172.21.15.162:0/3257886552 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc378011ca0 con 0x7fc388070c20 2024-02-13T00:32:26.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:25.385104+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-13T00:32:26.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:25.385722+0000 mon.a (mon.0) 47 : cluster [DBG] mgrmap e7: y(active, since 1.00498s) 2024-02-13T00:32:26.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:25.391778+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-13T00:32:26.584 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.576+0000 7fc38f4b9700 1 -- 172.21.15.162:0/3257886552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 msgr2=0x7fc3880fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:26.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.576+0000 7fc38f4b9700 1 --2- 172.21.15.162:0/3257886552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 0x7fc3880fed90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc378009730 tx=0x7fc3780090f0).stop 2024-02-13T00:32:26.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.576+0000 7fc38f4b9700 1 -- 172.21.15.162:0/3257886552 shutdown_connections 2024-02-13T00:32:26.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.576+0000 7fc38f4b9700 1 --2- 172.21.15.162:0/3257886552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 0x7fc3880fed90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:26.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.576+0000 7fc38f4b9700 1 -- 172.21.15.162:0/3257886552 >> 172.21.15.162:0/3257886552 conn(0x7fc3880f67d0 msgr2=0x7fc3880f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:26.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.576+0000 7fc38f4b9700 1 -- 172.21.15.162:0/3257886552 shutdown_connections 2024-02-13T00:32:26.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.576+0000 7fc38f4b9700 1 -- 172.21.15.162:0/3257886552 wait complete. 2024-02-13T00:32:26.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.577+0000 7fc38f4b9700 1 librados: starting msgr at 2024-02-13T00:32:26.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.577+0000 7fc38f4b9700 1 librados: starting objecter 2024-02-13T00:32:26.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.577+0000 7fc38f4b9700 1 Processor -- start 2024-02-13T00:32:26.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.577+0000 7fc38f4b9700 1 -- start start 2024-02-13T00:32:26.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.577+0000 7fc38f4b9700 1 librados: setting wanted keys 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.577+0000 7fc38f4b9700 1 librados: calling monclient init 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.577+0000 7fc38f4b9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 0x7fc3880fc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.577+0000 7fc38f4b9700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3880fac90 con 0x7fc388070c20 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.577+0000 7fc38d255700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 0x7fc3880fc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.577+0000 7fc38d255700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 0x7fc3880fc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35608/0 (socket says 172.21.15.162:35608) 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.577+0000 7fc38d255700 1 -- 172.21.15.162:0/1522868094 learned_addr learned my addr 172.21.15.162:0/1522868094 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.578+0000 7fc38d255700 1 -- 172.21.15.162:0/1522868094 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc378009370 con 0x7fc388070c20 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.578+0000 7fc38d255700 1 --2- 172.21.15.162:0/1522868094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 0x7fc3880fc810 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fc37800bfc0 tx=0x7fc3780090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.578+0000 7fc37e7fc700 10 client.?.objecter ms_handle_connect 0x7fc388070c20 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.578+0000 7fc37e7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.578+0000 7fc37e7fc700 1 -- 172.21.15.162:0/1522868094 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc37801a640 con 0x7fc388070c20 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.578+0000 7fc38f4b9700 1 -- 172.21.15.162:0/1522868094 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3880fae30 con 0x7fc388070c20 2024-02-13T00:32:26.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.578+0000 7fc37e7fc700 1 -- 172.21.15.162:0/1522868094 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc37801ac50 con 0x7fc388070c20 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.578+0000 7fc38f4b9700 10 client.14126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.578+0000 7fc37e7fc700 1 -- 172.21.15.162:0/1522868094 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc378023b30 con 0x7fc388070c20 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.578+0000 7fc38f4b9700 1 -- 172.21.15.162:0/1522868094 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3880fb2c0 con 0x7fc388070c20 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.578+0000 7fc38f4b9700 1 librados: init done 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.579+0000 7fc38f4b9700 1 -- 172.21.15.162:0/1522868094 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38805a3b0 con 0x7fc388070c20 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.579+0000 7fc37e7fc700 1 -- 172.21.15.162:0/1522868094 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 7) v1 ==== 38476+0+0 (secure 0 0 0) 0x7fc37801a7a0 con 0x7fc388070c20 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.580+0000 7fc37e7fc700 1 --2- 172.21.15.162:0/1522868094 >> v2:172.21.15.162:6800/4121673848 conn(0x7fc374033e00 0x7fc3740362c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.580+0000 7fc37e7fc700 1 -- 172.21.15.162:0/1522868094 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7fc378044c40 con 0x7fc388070c20 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.580+0000 7fc37e7fc700 10 client.14126.objecter ms_dispatch 0x7fc388001550 osd_map(2..2 src has 1..2) v4 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.580+0000 7fc37e7fc700 3 client.14126.objecter handle_osd_map got epochs [2,2] > 0 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.580+0000 7fc37e7fc700 3 client.14126.objecter handle_osd_map decoding full epoch 2 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.580+0000 7fc37e7fc700 20 client.14126.objecter dump_active .. 0 homeless 2024-02-13T00:32:26.587 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.580+0000 7fc38ca54700 1 --2- 172.21.15.162:0/1522868094 >> v2:172.21.15.162:6800/4121673848 conn(0x7fc374033e00 0x7fc3740362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:26.588 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.580+0000 7fc38ca54700 1 --2- 172.21.15.162:0/1522868094 >> v2:172.21.15.162:6800/4121673848 conn(0x7fc374033e00 0x7fc3740362c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc384009780 tx=0x7fc384006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:26.588 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.580+0000 7fc37e7fc700 10 client.14126.objecter ms_handle_connect 0x7fc374033e00 2024-02-13T00:32:26.588 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.584+0000 7fc37e7fc700 1 -- 172.21.15.162:0/1522868094 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc378023590 con 0x7fc388070c20 2024-02-13T00:32:26.759 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.756+0000 7fc38f4b9700 1 -- 172.21.15.162:0/1522868094 --> v2:172.21.15.162:6800/4121673848 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3880fb5c0 con 0x7fc374033e00 2024-02-13T00:32:26.764 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.761+0000 7fc37e7fc700 1 -- 172.21.15.162:0/1522868094 <== mgr.14116 v2:172.21.15.162:6800/4121673848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc3880fb5c0 con 0x7fc374033e00 2024-02-13T00:32:26.766 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.763+0000 7fc38f4b9700 10 librados: watch_flush enter 2024-02-13T00:32:26.766 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.763+0000 7fc38f4b9700 10 librados: watch_flush exit 2024-02-13T00:32:26.766 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.763+0000 7fc38f4b9700 20 client.14126.objecter shutdown clearing up homeless session... 2024-02-13T00:32:26.767 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.763+0000 7fc38f4b9700 10 client.14126.objecter successfully canceled tick 2024-02-13T00:32:26.767 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.763+0000 7fc38f4b9700 1 -- 172.21.15.162:0/1522868094 >> v2:172.21.15.162:6800/4121673848 conn(0x7fc374033e00 msgr2=0x7fc3740362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:26.767 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.763+0000 7fc38f4b9700 1 --2- 172.21.15.162:0/1522868094 >> v2:172.21.15.162:6800/4121673848 conn(0x7fc374033e00 0x7fc3740362c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc384009780 tx=0x7fc384006cc0).stop 2024-02-13T00:32:26.767 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.763+0000 7fc38f4b9700 1 -- 172.21.15.162:0/1522868094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 msgr2=0x7fc3880fc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:26.767 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.763+0000 7fc38f4b9700 1 --2- 172.21.15.162:0/1522868094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 0x7fc3880fc810 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fc37800bfc0 tx=0x7fc3780090f0).stop 2024-02-13T00:32:26.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.763+0000 7fc38f4b9700 1 -- 172.21.15.162:0/1522868094 shutdown_connections 2024-02-13T00:32:26.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.763+0000 7fc38f4b9700 1 --2- 172.21.15.162:0/1522868094 >> v2:172.21.15.162:6800/4121673848 conn(0x7fc374033e00 0x7fc3740362c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:26.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.763+0000 7fc38f4b9700 1 --2- 172.21.15.162:0/1522868094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc388070c20 0x7fc3880fc810 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:26.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.763+0000 7fc38f4b9700 1 -- 172.21.15.162:0/1522868094 >> 172.21.15.162:0/1522868094 conn(0x7fc3880f67d0 msgr2=0x7fc3880f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:26.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.764+0000 7fc38f4b9700 1 -- 172.21.15.162:0/1522868094 shutdown_connections 2024-02-13T00:32:26.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.764+0000 7fc38f4b9700 1 -- 172.21.15.162:0/1522868094 wait complete. 2024-02-13T00:32:26.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:26.764+0000 7fc38f4b9700 1 librados: shutdown 2024-02-13T00:32:27.895 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.889+0000 7fd8662ca700 1 Processor -- start 2024-02-13T00:32:27.895 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.889+0000 7fd8662ca700 1 -- start start 2024-02-13T00:32:27.895 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.890+0000 7fd8662ca700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 0x7fd8600fedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:27.895 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.890+0000 7fd8662ca700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600ff2b0 con 0x7fd860070c20 2024-02-13T00:32:27.895 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.890+0000 7fd85ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 0x7fd8600fedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:27.895 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.890+0000 7fd85ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 0x7fd8600fedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35620/0 (socket says 172.21.15.162:35620) 2024-02-13T00:32:27.895 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.890+0000 7fd85ffff700 1 -- 172.21.15.162:0/4111654734 learned_addr learned my addr 172.21.15.162:0/4111654734 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:27.895 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.890+0000 7fd85ffff700 1 -- 172.21.15.162:0/4111654734 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8600ff3f0 con 0x7fd860070c20 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.890+0000 7fd85ffff700 1 --2- 172.21.15.162:0/4111654734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 0x7fd8600fedc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd850009730 tx=0x7fd8500090f0).ready entity=mon.0 client_cookie=607513cacafa0338 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.890+0000 7fd85f7fe700 1 -- 172.21.15.162:0/4111654734 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd85000b610 con 0x7fd860070c20 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.890+0000 7fd85f7fe700 1 -- 172.21.15.162:0/4111654734 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd85000bc20 con 0x7fd860070c20 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.890+0000 7fd85f7fe700 1 -- 172.21.15.162:0/4111654734 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd850011ca0 con 0x7fd860070c20 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.891+0000 7fd8662ca700 1 -- 172.21.15.162:0/4111654734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 msgr2=0x7fd8600fedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.891+0000 7fd8662ca700 1 --2- 172.21.15.162:0/4111654734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 0x7fd8600fedc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd850009730 tx=0x7fd8500090f0).stop 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.891+0000 7fd8662ca700 1 -- 172.21.15.162:0/4111654734 shutdown_connections 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.891+0000 7fd8662ca700 1 --2- 172.21.15.162:0/4111654734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 0x7fd8600fedc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.891+0000 7fd8662ca700 1 -- 172.21.15.162:0/4111654734 >> 172.21.15.162:0/4111654734 conn(0x7fd8600f6820 msgr2=0x7fd8600f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.891+0000 7fd8662ca700 1 -- 172.21.15.162:0/4111654734 shutdown_connections 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.891+0000 7fd8662ca700 1 -- 172.21.15.162:0/4111654734 wait complete. 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd8662ca700 1 librados: starting msgr at 2024-02-13T00:32:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd8662ca700 1 librados: starting objecter 2024-02-13T00:32:27.897 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd8662ca700 1 Processor -- start 2024-02-13T00:32:27.897 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd8662ca700 1 -- start start 2024-02-13T00:32:27.897 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd8662ca700 1 librados: setting wanted keys 2024-02-13T00:32:27.897 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd8662ca700 1 librados: calling monclient init 2024-02-13T00:32:27.898 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd8662ca700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 0x7fd86010aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:27.898 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd8662ca700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86010b3a0 con 0x7fd860070c20 2024-02-13T00:32:27.898 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd85ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 0x7fd86010aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:27.898 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd85ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 0x7fd86010aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35634/0 (socket says 172.21.15.162:35634) 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd85ffff700 1 -- 172.21.15.162:0/4233849435 learned_addr learned my addr 172.21.15.162:0/4233849435 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd85ffff700 1 -- 172.21.15.162:0/4233849435 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd850009370 con 0x7fd860070c20 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd85ffff700 1 --2- 172.21.15.162:0/4233849435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 0x7fd86010aeb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd85000bfc0 tx=0x7fd8500090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd85df76700 10 client.?.objecter ms_handle_connect 0x7fd860070c20 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd85df76700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.892+0000 7fd85df76700 1 -- 172.21.15.162:0/4233849435 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd85001a640 con 0x7fd860070c20 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.893+0000 7fd85df76700 1 -- 172.21.15.162:0/4233849435 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd85001ac50 con 0x7fd860070c20 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.893+0000 7fd8662ca700 1 -- 172.21.15.162:0/4233849435 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86010b5a0 con 0x7fd860070c20 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.893+0000 7fd85df76700 1 -- 172.21.15.162:0/4233849435 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd850023b30 con 0x7fd860070c20 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.893+0000 7fd8662ca700 10 client.14128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.893+0000 7fd8662ca700 1 -- 172.21.15.162:0/4233849435 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86010ba30 con 0x7fd860070c20 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.893+0000 7fd8662ca700 1 librados: init done 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.893+0000 7fd8662ca700 1 -- 172.21.15.162:0/4233849435 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd860046d80 con 0x7fd860070c20 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.894+0000 7fd85df76700 1 -- 172.21.15.162:0/4233849435 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7fd85001a7a0 con 0x7fd860070c20 2024-02-13T00:32:27.899 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.894+0000 7fd85df76700 1 --2- 172.21.15.162:0/4233849435 >> v2:172.21.15.162:6800/4121673848 conn(0x7fd84c033f70 0x7fd84c036430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:27.900 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.894+0000 7fd85df76700 1 -- 172.21.15.162:0/4233849435 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7fd850044ef0 con 0x7fd860070c20 2024-02-13T00:32:27.900 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.894+0000 7fd85df76700 10 client.14128.objecter ms_dispatch 0x7fd860001550 osd_map(2..2 src has 1..2) v4 2024-02-13T00:32:27.900 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.894+0000 7fd85df76700 3 client.14128.objecter handle_osd_map got epochs [2,2] > 0 2024-02-13T00:32:27.900 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.894+0000 7fd85df76700 3 client.14128.objecter handle_osd_map decoding full epoch 2 2024-02-13T00:32:27.900 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.894+0000 7fd85df76700 20 client.14128.objecter dump_active .. 0 homeless 2024-02-13T00:32:27.900 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.894+0000 7fd857fff700 1 --2- 172.21.15.162:0/4233849435 >> v2:172.21.15.162:6800/4121673848 conn(0x7fd84c033f70 0x7fd84c036430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:27.900 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.894+0000 7fd85df76700 10 client.14128.objecter ms_handle_connect 0x7fd84c033f70 2024-02-13T00:32:27.900 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.894+0000 7fd857fff700 1 --2- 172.21.15.162:0/4233849435 >> v2:172.21.15.162:6800/4121673848 conn(0x7fd84c033f70 0x7fd84c036430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd848009780 tx=0x7fd848006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:27.901 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:27.898+0000 7fd85df76700 1 -- 172.21.15.162:0/4233849435 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd850023590 con 0x7fd860070c20 2024-02-13T00:32:28.066 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.063+0000 7fd8662ca700 1 -- 172.21.15.162:0/4233849435 --> v2:172.21.15.162:6800/4121673848 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fd860104a70 con 0x7fd84c033f70 2024-02-13T00:32:28.066 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.064+0000 7fd85df76700 1 -- 172.21.15.162:0/4233849435 <== mgr.14116 v2:172.21.15.162:6800/4121673848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fd860104a70 con 0x7fd84c033f70 2024-02-13T00:32:28.066 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: value unchanged 2024-02-13T00:32:28.069 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.065+0000 7fd8662ca700 10 librados: watch_flush enter 2024-02-13T00:32:28.069 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.065+0000 7fd8662ca700 10 librados: watch_flush exit 2024-02-13T00:32:28.069 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.065+0000 7fd8662ca700 20 client.14128.objecter shutdown clearing up homeless session... 2024-02-13T00:32:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.065+0000 7fd8662ca700 10 client.14128.objecter successfully canceled tick 2024-02-13T00:32:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.065+0000 7fd8662ca700 1 -- 172.21.15.162:0/4233849435 >> v2:172.21.15.162:6800/4121673848 conn(0x7fd84c033f70 msgr2=0x7fd84c036430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.065+0000 7fd8662ca700 1 --2- 172.21.15.162:0/4233849435 >> v2:172.21.15.162:6800/4121673848 conn(0x7fd84c033f70 0x7fd84c036430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd848009780 tx=0x7fd848006cc0).stop 2024-02-13T00:32:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.065+0000 7fd8662ca700 1 -- 172.21.15.162:0/4233849435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 msgr2=0x7fd86010aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.065+0000 7fd8662ca700 1 --2- 172.21.15.162:0/4233849435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 0x7fd86010aeb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd85000bfc0 tx=0x7fd8500090f0).stop 2024-02-13T00:32:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.065+0000 7fd8662ca700 1 -- 172.21.15.162:0/4233849435 shutdown_connections 2024-02-13T00:32:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.065+0000 7fd8662ca700 1 --2- 172.21.15.162:0/4233849435 >> v2:172.21.15.162:6800/4121673848 conn(0x7fd84c033f70 0x7fd84c036430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.065+0000 7fd8662ca700 1 --2- 172.21.15.162:0/4233849435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd860070c20 0x7fd86010aeb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.065+0000 7fd8662ca700 1 -- 172.21.15.162:0/4233849435 >> 172.21.15.162:0/4233849435 conn(0x7fd8600f6820 msgr2=0x7fd8600f80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.066+0000 7fd8662ca700 1 -- 172.21.15.162:0/4233849435 shutdown_connections 2024-02-13T00:32:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.066+0000 7fd8662ca700 1 -- 172.21.15.162:0/4233849435 wait complete. 2024-02-13T00:32:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:28.066+0000 7fd8662ca700 1 librados: shutdown 2024-02-13T00:32:28.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:26.757871+0000 mgr.y (mgr.14116) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:32:28.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:26.761900+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' 2024-02-13T00:32:28.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:26.776112+0000 mon.a (mon.0) 49 : audit [DBG] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:32:28.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:27.387374+0000 mon.a (mon.0) 50 : cluster [DBG] mgrmap e8: y(active, since 3s) 2024-02-13T00:32:28.750 INFO:teuthology.orchestra.run.smithi162.stderr:Generating ssh key... 2024-02-13T00:32:29.023 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:28.064575+0000 mgr.y (mgr.14116) 5 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:32:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.330+0000 7fe60e57e700 1 Processor -- start 2024-02-13T00:32:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.330+0000 7fe60e57e700 1 -- start start 2024-02-13T00:32:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.330+0000 7fe60e57e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 0x7fe608101220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.330+0000 7fe60e57e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080fcc10 con 0x7fe608070bc0 2024-02-13T00:32:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.330+0000 7fe607fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 0x7fe608101220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.330+0000 7fe607fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 0x7fe608101220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35642/0 (socket says 172.21.15.162:35642) 2024-02-13T00:32:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.330+0000 7fe607fff700 1 -- 172.21.15.162:0/596983145 learned_addr learned my addr 172.21.15.162:0/596983145 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.330+0000 7fe607fff700 1 -- 172.21.15.162:0/596983145 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe608101710 con 0x7fe608070bc0 2024-02-13T00:32:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.331+0000 7fe607fff700 1 --2- 172.21.15.162:0/596983145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 0x7fe608101220 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe5f8009730 tx=0x7fe5f80090f0).ready entity=mon.0 client_cookie=91a37239bcb513b3 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.331+0000 7fe606ffd700 1 -- 172.21.15.162:0/596983145 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5f800b580 con 0x7fe608070bc0 2024-02-13T00:32:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.331+0000 7fe606ffd700 1 -- 172.21.15.162:0/596983145 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5f800bb90 con 0x7fe608070bc0 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.331+0000 7fe606ffd700 1 -- 172.21.15.162:0/596983145 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5f8011c30 con 0x7fe608070bc0 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.331+0000 7fe60e57e700 1 -- 172.21.15.162:0/596983145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 msgr2=0x7fe608101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.331+0000 7fe60e57e700 1 --2- 172.21.15.162:0/596983145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 0x7fe608101220 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe5f8009730 tx=0x7fe5f80090f0).stop 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.331+0000 7fe60e57e700 1 -- 172.21.15.162:0/596983145 shutdown_connections 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.331+0000 7fe60e57e700 1 --2- 172.21.15.162:0/596983145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 0x7fe608101220 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.331+0000 7fe60e57e700 1 -- 172.21.15.162:0/596983145 >> 172.21.15.162:0/596983145 conn(0x7fe6080f6820 msgr2=0x7fe6080f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.331+0000 7fe60e57e700 1 -- 172.21.15.162:0/596983145 shutdown_connections 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.331+0000 7fe60e57e700 1 -- 172.21.15.162:0/596983145 wait complete. 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.332+0000 7fe60e57e700 1 librados: starting msgr at 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.332+0000 7fe60e57e700 1 librados: starting objecter 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.332+0000 7fe60e57e700 1 Processor -- start 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.332+0000 7fe60e57e700 1 -- start start 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.332+0000 7fe60e57e700 1 librados: setting wanted keys 2024-02-13T00:32:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.332+0000 7fe60e57e700 1 librados: calling monclient init 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe60e57e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 0x7fe60810d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe60e57e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60810d540 con 0x7fe608070bc0 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe607fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 0x7fe60810d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe607fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 0x7fe60810d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35646/0 (socket says 172.21.15.162:35646) 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe607fff700 1 -- 172.21.15.162:0/143185663 learned_addr learned my addr 172.21.15.162:0/143185663 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe607fff700 1 -- 172.21.15.162:0/143185663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f8009370 con 0x7fe608070bc0 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe607fff700 1 --2- 172.21.15.162:0/143185663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 0x7fe60810d050 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fe5f8000c00 tx=0x7fe5f80090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe6057fa700 10 client.?.objecter ms_handle_connect 0x7fe608070bc0 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe6057fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe6057fa700 1 -- 172.21.15.162:0/143185663 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5f800b9a0 con 0x7fe608070bc0 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe6057fa700 1 -- 172.21.15.162:0/143185663 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5f801aaf0 con 0x7fe608070bc0 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe6057fa700 1 -- 172.21.15.162:0/143185663 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5f8023ac0 con 0x7fe608070bc0 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe60e57e700 1 -- 172.21.15.162:0/143185663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60810d740 con 0x7fe608070bc0 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe60e57e700 10 client.14130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe60e57e700 1 -- 172.21.15.162:0/143185663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60810dbd0 con 0x7fe608070bc0 2024-02-13T00:32:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.333+0000 7fe60e57e700 1 librados: init done 2024-02-13T00:32:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.334+0000 7fe6057fa700 1 -- 172.21.15.162:0/143185663 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7fe5f801ac60 con 0x7fe608070bc0 2024-02-13T00:32:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.334+0000 7fe60e57e700 1 -- 172.21.15.162:0/143185663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe608046d80 con 0x7fe608070bc0 2024-02-13T00:32:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.334+0000 7fe6057fa700 1 --2- 172.21.15.162:0/143185663 >> v2:172.21.15.162:6800/4121673848 conn(0x7fe5f0033f70 0x7fe5f0036430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.334+0000 7fe6057fa700 1 -- 172.21.15.162:0/143185663 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7fe5f80448f0 con 0x7fe608070bc0 2024-02-13T00:32:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.334+0000 7fe6057fa700 10 client.14130.objecter ms_dispatch 0x7fe608001550 osd_map(2..2 src has 1..2) v4 2024-02-13T00:32:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.334+0000 7fe6057fa700 3 client.14130.objecter handle_osd_map got epochs [2,2] > 0 2024-02-13T00:32:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.334+0000 7fe6057fa700 3 client.14130.objecter handle_osd_map decoding full epoch 2 2024-02-13T00:32:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.334+0000 7fe6057fa700 20 client.14130.objecter dump_active .. 0 homeless 2024-02-13T00:32:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.334+0000 7fe6077fe700 1 --2- 172.21.15.162:0/143185663 >> v2:172.21.15.162:6800/4121673848 conn(0x7fe5f0033f70 0x7fe5f0036430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.335+0000 7fe6077fe700 1 --2- 172.21.15.162:0/143185663 >> v2:172.21.15.162:6800/4121673848 conn(0x7fe5f0033f70 0x7fe5f0036430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe5fc009780 tx=0x7fe5fc006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.335+0000 7fe6057fa700 10 client.14130.objecter ms_handle_connect 0x7fe5f0033f70 2024-02-13T00:32:29.343 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.339+0000 7fe6057fa700 1 -- 172.21.15.162:0/143185663 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe5f801a640 con 0x7fe608070bc0 2024-02-13T00:32:29.506 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.503+0000 7fe60e57e700 1 -- 172.21.15.162:0/143185663 --> v2:172.21.15.162:6800/4121673848 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe608106c30 con 0x7fe5f0033f70 2024-02-13T00:32:29.920 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.917+0000 7fe6057fa700 1 -- 172.21.15.162:0/143185663 <== mgr.14116 v2:172.21.15.162:6800/4121673848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe608106c30 con 0x7fe5f0033f70 2024-02-13T00:32:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 10 librados: watch_flush enter 2024-02-13T00:32:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 10 librados: watch_flush exit 2024-02-13T00:32:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 20 client.14130.objecter shutdown clearing up homeless session... 2024-02-13T00:32:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 10 client.14130.objecter successfully canceled tick 2024-02-13T00:32:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 1 -- 172.21.15.162:0/143185663 >> v2:172.21.15.162:6800/4121673848 conn(0x7fe5f0033f70 msgr2=0x7fe5f0036430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 1 --2- 172.21.15.162:0/143185663 >> v2:172.21.15.162:6800/4121673848 conn(0x7fe5f0033f70 0x7fe5f0036430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe5fc009780 tx=0x7fe5fc006cc0).stop 2024-02-13T00:32:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 1 -- 172.21.15.162:0/143185663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 msgr2=0x7fe60810d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 1 --2- 172.21.15.162:0/143185663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 0x7fe60810d050 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fe5f8000c00 tx=0x7fe5f80090f0).stop 2024-02-13T00:32:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 1 -- 172.21.15.162:0/143185663 shutdown_connections 2024-02-13T00:32:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 1 --2- 172.21.15.162:0/143185663 >> v2:172.21.15.162:6800/4121673848 conn(0x7fe5f0033f70 0x7fe5f0036430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 1 --2- 172.21.15.162:0/143185663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe608070bc0 0x7fe60810d050 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 1 -- 172.21.15.162:0/143185663 >> 172.21.15.162:0/143185663 conn(0x7fe6080f6820 msgr2=0x7fe6080f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 1 -- 172.21.15.162:0/143185663 shutdown_connections 2024-02-13T00:32:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 1 -- 172.21.15.162:0/143185663 wait complete. 2024-02-13T00:32:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:29.919+0000 7fe60e57e700 1 librados: shutdown 2024-02-13T00:32:30.187 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: Generating public/private rsa key pair. 2024-02-13T00:32:30.187 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: Your identification has been saved in /tmp/tmppfxhkw9k/key. 2024-02-13T00:32:30.187 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: Your public key has been saved in /tmp/tmppfxhkw9k/key.pub. 2024-02-13T00:32:30.187 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: The key fingerprint is: 2024-02-13T00:32:30.187 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: SHA256:2ohLzjTV+mCRP7qgL1x9EBQZ1THxQoHSjxuPhOCUL2s ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:32:30.188 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: The key's randomart image is: 2024-02-13T00:32:30.188 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: +---[RSA 3072]----+ 2024-02-13T00:32:30.188 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: | o+*.o*+ | 2024-02-13T00:32:30.188 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: | + + o..o | 2024-02-13T00:32:30.188 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: | o o + o. . | 2024-02-13T00:32:30.188 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: | o +o+ .. | 2024-02-13T00:32:30.188 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: | ++oS= | 2024-02-13T00:32:30.188 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: | Eo.B+ . | 2024-02-13T00:32:30.188 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: | . o* *.+ | 2024-02-13T00:32:30.188 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: | o* = + . | 2024-02-13T00:32:30.188 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: | .o= o.. | 2024-02-13T00:32:30.188 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: +----[SHA256]-----+ 2024-02-13T00:32:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.963+0000 7fcc019c0700 1 Processor -- start 2024-02-13T00:32:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.963+0000 7fcc019c0700 1 -- start start 2024-02-13T00:32:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.963+0000 7fcc019c0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 0x7fcbfc0fca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.963+0000 7fcc019c0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc0fcf10 con 0x7fcbfc0fc600 2024-02-13T00:32:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.963+0000 7fcbfaffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 0x7fcbfc0fca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.963+0000 7fcbfaffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 0x7fcbfc0fca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35658/0 (socket says 172.21.15.162:35658) 2024-02-13T00:32:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.963+0000 7fcbfaffd700 1 -- 172.21.15.162:0/2765467022 learned_addr learned my addr 172.21.15.162:0/2765467022 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.963+0000 7fcbfaffd700 1 -- 172.21.15.162:0/2765467022 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbfc0fd050 con 0x7fcbfc0fc600 2024-02-13T00:32:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.963+0000 7fcbfaffd700 1 --2- 172.21.15.162:0/2765467022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 0x7fcbfc0fca20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fcbec009730 tx=0x7fcbec0090f0).ready entity=mon.0 client_cookie=dda31187831edb80 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.964+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/2765467022 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbec00b580 con 0x7fcbfc0fc600 2024-02-13T00:32:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.964+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/2765467022 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbec00bb90 con 0x7fcbfc0fc600 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.964+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/2765467022 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbec011c30 con 0x7fcbfc0fc600 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.964+0000 7fcc019c0700 1 -- 172.21.15.162:0/2765467022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 msgr2=0x7fcbfc0fca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.964+0000 7fcc019c0700 1 --2- 172.21.15.162:0/2765467022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 0x7fcbfc0fca20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fcbec009730 tx=0x7fcbec0090f0).stop 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.964+0000 7fcc019c0700 1 -- 172.21.15.162:0/2765467022 shutdown_connections 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.964+0000 7fcc019c0700 1 --2- 172.21.15.162:0/2765467022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 0x7fcbfc0fca20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.964+0000 7fcc019c0700 1 -- 172.21.15.162:0/2765467022 >> 172.21.15.162:0/2765467022 conn(0x7fcbfc0f5f60 msgr2=0x7fcbfc0f83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.964+0000 7fcc019c0700 1 -- 172.21.15.162:0/2765467022 shutdown_connections 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.964+0000 7fcc019c0700 1 -- 172.21.15.162:0/2765467022 wait complete. 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.965+0000 7fcc019c0700 1 librados: starting msgr at 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.965+0000 7fcc019c0700 1 librados: starting objecter 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.965+0000 7fcc019c0700 1 Processor -- start 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.965+0000 7fcc019c0700 1 -- start start 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.965+0000 7fcc019c0700 1 librados: setting wanted keys 2024-02-13T00:32:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.965+0000 7fcc019c0700 1 librados: calling monclient init 2024-02-13T00:32:30.971 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.965+0000 7fcc019c0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 0x7fcbfc10dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:30.971 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.965+0000 7fcc019c0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc0fafd0 con 0x7fcbfc0fc600 2024-02-13T00:32:30.971 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.965+0000 7fcbfaffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 0x7fcbfc10dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:30.971 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.965+0000 7fcbfaffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 0x7fcbfc10dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35662/0 (socket says 172.21.15.162:35662) 2024-02-13T00:32:30.971 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.965+0000 7fcbfaffd700 1 -- 172.21.15.162:0/4104341783 learned_addr learned my addr 172.21.15.162:0/4104341783 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:30.971 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.966+0000 7fcbfaffd700 1 -- 172.21.15.162:0/4104341783 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec009370 con 0x7fcbfc0fc600 2024-02-13T00:32:30.971 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.966+0000 7fcbfaffd700 1 --2- 172.21.15.162:0/4104341783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 0x7fcbfc10dae0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fcbec000c00 tx=0x7fcbec0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:30.971 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.966+0000 7fcbe3fff700 10 client.?.objecter ms_handle_connect 0x7fcbfc0fc600 2024-02-13T00:32:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.966+0000 7fcbe3fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.966+0000 7fcbe3fff700 1 -- 172.21.15.162:0/4104341783 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbec01a430 con 0x7fcbfc0fc600 2024-02-13T00:32:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.966+0000 7fcbe3fff700 1 -- 172.21.15.162:0/4104341783 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbec01aa40 con 0x7fcbfc0fc600 2024-02-13T00:32:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.966+0000 7fcbe3fff700 1 -- 172.21.15.162:0/4104341783 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbec023b00 con 0x7fcbfc0fc600 2024-02-13T00:32:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.966+0000 7fcc019c0700 1 -- 172.21.15.162:0/4104341783 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbfc0fb1d0 con 0x7fcbfc0fc600 2024-02-13T00:32:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.966+0000 7fcc019c0700 10 client.14132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.966+0000 7fcc019c0700 1 -- 172.21.15.162:0/4104341783 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbfc10e210 con 0x7fcbfc0fc600 2024-02-13T00:32:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.966+0000 7fcc019c0700 1 librados: init done 2024-02-13T00:32:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.967+0000 7fcbe3fff700 1 -- 172.21.15.162:0/4104341783 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7fcbec02c3f0 con 0x7fcbfc0fc600 2024-02-13T00:32:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.967+0000 7fcbe3fff700 1 --2- 172.21.15.162:0/4104341783 >> v2:172.21.15.162:6800/4121673848 conn(0x7fcbe4033f70 0x7fcbe4036430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.967+0000 7fcc019c0700 1 -- 172.21.15.162:0/4104341783 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbfc046d80 con 0x7fcbfc0fc600 2024-02-13T00:32:30.973 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.967+0000 7fcbe3fff700 1 -- 172.21.15.162:0/4104341783 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7fcbec044d40 con 0x7fcbfc0fc600 2024-02-13T00:32:30.974 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.967+0000 7fcbe3fff700 10 client.14132.objecter ms_dispatch 0x7fcbfc001550 osd_map(2..2 src has 1..2) v4 2024-02-13T00:32:30.974 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.967+0000 7fcbe3fff700 3 client.14132.objecter handle_osd_map got epochs [2,2] > 0 2024-02-13T00:32:30.974 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.967+0000 7fcbe3fff700 3 client.14132.objecter handle_osd_map decoding full epoch 2 2024-02-13T00:32:30.974 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.967+0000 7fcbe3fff700 20 client.14132.objecter dump_active .. 0 homeless 2024-02-13T00:32:30.974 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.967+0000 7fcbfa7fc700 1 --2- 172.21.15.162:0/4104341783 >> v2:172.21.15.162:6800/4121673848 conn(0x7fcbe4033f70 0x7fcbe4036430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:30.974 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.967+0000 7fcbe3fff700 10 client.14132.objecter ms_handle_connect 0x7fcbe4033f70 2024-02-13T00:32:30.974 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.967+0000 7fcbfa7fc700 1 --2- 172.21.15.162:0/4104341783 >> v2:172.21.15.162:6800/4121673848 conn(0x7fcbe4033f70 0x7fcbe4036430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fcbf0009780 tx=0x7fcbf0006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:30.974 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:30.972+0000 7fcbe3fff700 1 -- 172.21.15.162:0/4104341783 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fcbec01e070 con 0x7fcbfc0fc600 2024-02-13T00:32:31.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.133+0000 7fcc019c0700 1 -- 172.21.15.162:0/4104341783 --> v2:172.21.15.162:6800/4121673848 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbfc103db0 con 0x7fcbe4033f70 2024-02-13T00:32:31.137 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.134+0000 7fcbe3fff700 1 -- 172.21.15.162:0/4104341783 <== mgr.14116 v2:172.21.15.162:6800/4121673848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fcbfc103db0 con 0x7fcbe4033f70 2024-02-13T00:32:31.137 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTWxtRczCpQs9KnbDWy9LKnJJ833ey5M77SdBAE66n3+d7Xu19kaT2BNWfMBd40BPDBJ45NH+dGh+4tKDBicqT0JD1S+j5QSa1J6Txnn7CAx2rwSUJoZiGLpKAQ+A8CaVF/2/ZoTNbhHsQJHWos13q7yGWvEO5fqYOc6+DcjVoJkoAqproDKtnpOcoYSEWsm51nMAYZMd8OqMk+IsP2kzxPBwXyV6Q+I1lU6SjTtWwyjvImSUPfbzwAtR9ALkP+5hm6C76VhMRdFd6ztBEK9Xq2eTWxkUY9Z6vhxyQsTQSjz1kBmfQypLzuYGr8vPHIHMy9k/dwbLFt6u3kGN+cNrfWVrN9VZgkjZmO9itStZCBDbecHElIrEQ25ZLgP/ct/0h4FAdy7nH7jYehxiNpCZNJ76NN3FoPRVBKTjX8IEtfalirMIeLi9wiaZ1W/6qIpg64J2mfIBi6Z3pVeaG5KAUlDKIo1V11bsiLsM/IkSnWDilUaGEVGJwB/aaWuz8vRU= ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:32:31.140 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.135+0000 7fcc019c0700 10 librados: watch_flush enter 2024-02-13T00:32:31.140 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.135+0000 7fcc019c0700 10 librados: watch_flush exit 2024-02-13T00:32:31.140 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 20 client.14132.objecter shutdown clearing up homeless session... 2024-02-13T00:32:31.140 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 10 client.14132.objecter successfully canceled tick 2024-02-13T00:32:31.140 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 1 -- 172.21.15.162:0/4104341783 >> v2:172.21.15.162:6800/4121673848 conn(0x7fcbe4033f70 msgr2=0x7fcbe4036430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:31.140 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 1 --2- 172.21.15.162:0/4104341783 >> v2:172.21.15.162:6800/4121673848 conn(0x7fcbe4033f70 0x7fcbe4036430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fcbf0009780 tx=0x7fcbf0006cc0).stop 2024-02-13T00:32:31.140 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 1 -- 172.21.15.162:0/4104341783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 msgr2=0x7fcbfc10dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:31.140 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 1 --2- 172.21.15.162:0/4104341783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 0x7fcbfc10dae0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fcbec000c00 tx=0x7fcbec0090f0).stop 2024-02-13T00:32:31.140 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 1 -- 172.21.15.162:0/4104341783 shutdown_connections 2024-02-13T00:32:31.140 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 1 --2- 172.21.15.162:0/4104341783 >> v2:172.21.15.162:6800/4121673848 conn(0x7fcbe4033f70 0x7fcbe4036430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:31.140 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 1 --2- 172.21.15.162:0/4104341783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fc600 0x7fcbfc10dae0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:31.141 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 1 -- 172.21.15.162:0/4104341783 >> 172.21.15.162:0/4104341783 conn(0x7fcbfc0f5f60 msgr2=0x7fcbfc0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:31.141 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 1 -- 172.21.15.162:0/4104341783 shutdown_connections 2024-02-13T00:32:31.141 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 1 -- 172.21.15.162:0/4104341783 wait complete. 2024-02-13T00:32:31.141 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:31.136+0000 7fcc019c0700 1 librados: shutdown 2024-02-13T00:32:31.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:29.505021+0000 mgr.y (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:32:31.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:32:29.505224+0000 mgr.y (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2024-02-13T00:32:31.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:29.914757+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' 2024-02-13T00:32:31.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:29.917190+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' 2024-02-13T00:32:31.690 INFO:teuthology.orchestra.run.smithi162.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-13T00:32:31.690 INFO:teuthology.orchestra.run.smithi162.stderr:Adding key to root@localhost authorized_keys... 2024-02-13T00:32:31.691 INFO:teuthology.orchestra.run.smithi162.stderr:Adding host smithi162... 2024-02-13T00:32:32.255 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.249+0000 7f9ae0019700 1 Processor -- start 2024-02-13T00:32:32.255 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.249+0000 7f9ae0019700 1 -- start start 2024-02-13T00:32:32.255 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.249+0000 7f9ae0019700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 0x7f9ad80fa950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.249+0000 7f9ae0019700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad80fae40 con 0x7f9ad80fa530 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.249+0000 7f9adddb5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 0x7f9ad80fa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.249+0000 7f9adddb5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 0x7f9ad80fa950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35672/0 (socket says 172.21.15.162:35672) 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.249+0000 7f9adddb5700 1 -- 172.21.15.162:0/724423259 learned_addr learned my addr 172.21.15.162:0/724423259 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.249+0000 7f9adddb5700 1 -- 172.21.15.162:0/724423259 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad80faf80 con 0x7f9ad80fa530 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.250+0000 7f9adddb5700 1 --2- 172.21.15.162:0/724423259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 0x7f9ad80fa950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9acc009730 tx=0x7f9acc0090f0).ready entity=mon.0 client_cookie=4db259c939812565 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.250+0000 7f9adcdb3700 1 -- 172.21.15.162:0/724423259 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9acc00b580 con 0x7f9ad80fa530 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.250+0000 7f9adcdb3700 1 -- 172.21.15.162:0/724423259 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9acc00bb90 con 0x7f9ad80fa530 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.250+0000 7f9adcdb3700 1 -- 172.21.15.162:0/724423259 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9acc011c30 con 0x7f9ad80fa530 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.250+0000 7f9ae0019700 1 -- 172.21.15.162:0/724423259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 msgr2=0x7f9ad80fa950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.250+0000 7f9ae0019700 1 --2- 172.21.15.162:0/724423259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 0x7f9ad80fa950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9acc009730 tx=0x7f9acc0090f0).stop 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.250+0000 7f9ae0019700 1 -- 172.21.15.162:0/724423259 shutdown_connections 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.250+0000 7f9ae0019700 1 --2- 172.21.15.162:0/724423259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 0x7f9ad80fa950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:32.256 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.250+0000 7f9ae0019700 1 -- 172.21.15.162:0/724423259 >> 172.21.15.162:0/724423259 conn(0x7f9ad80f5f30 msgr2=0x7f9ad80f8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.250+0000 7f9ae0019700 1 -- 172.21.15.162:0/724423259 shutdown_connections 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.250+0000 7f9ae0019700 1 -- 172.21.15.162:0/724423259 wait complete. 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.251+0000 7f9ae0019700 1 librados: starting msgr at 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.251+0000 7f9ae0019700 1 librados: starting objecter 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.251+0000 7f9ae0019700 1 Processor -- start 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.251+0000 7f9ae0019700 1 -- start start 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.251+0000 7f9ae0019700 1 librados: setting wanted keys 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.251+0000 7f9ae0019700 1 librados: calling monclient init 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.251+0000 7f9ae0019700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 0x7f9ad810ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.251+0000 7f9ae0019700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad810b360 con 0x7f9ad80fa530 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.251+0000 7f9adddb5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 0x7f9ad810ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.251+0000 7f9adddb5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 0x7f9ad810ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35684/0 (socket says 172.21.15.162:35684) 2024-02-13T00:32:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.251+0000 7f9adddb5700 1 -- 172.21.15.162:0/3100529374 learned_addr learned my addr 172.21.15.162:0/3100529374 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:32.258 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.252+0000 7f9adddb5700 1 -- 172.21.15.162:0/3100529374 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc009370 con 0x7f9ad80fa530 2024-02-13T00:32:32.258 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.252+0000 7f9adddb5700 1 --2- 172.21.15.162:0/3100529374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 0x7f9ad810ae70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9acc006940 tx=0x7f9acc0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:32.258 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.252+0000 7f9acaffd700 10 client.?.objecter ms_handle_connect 0x7f9ad80fa530 2024-02-13T00:32:32.258 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.252+0000 7f9acaffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:32.258 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.252+0000 7f9acaffd700 1 -- 172.21.15.162:0/3100529374 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9acc01a430 con 0x7f9ad80fa530 2024-02-13T00:32:32.258 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.252+0000 7f9acaffd700 1 -- 172.21.15.162:0/3100529374 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9acc01aa40 con 0x7f9ad80fa530 2024-02-13T00:32:32.258 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.252+0000 7f9ae0019700 1 -- 172.21.15.162:0/3100529374 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad810b560 con 0x7f9ad80fa530 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.252+0000 7f9acaffd700 1 -- 172.21.15.162:0/3100529374 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9acc023b90 con 0x7f9ad80fa530 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.252+0000 7f9ae0019700 10 client.14134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.252+0000 7f9ae0019700 1 -- 172.21.15.162:0/3100529374 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad810b9f0 con 0x7f9ad80fa530 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.252+0000 7f9ae0019700 1 librados: init done 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.253+0000 7f9acaffd700 1 -- 172.21.15.162:0/3100529374 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f9acc023cf0 con 0x7f9ad80fa530 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.253+0000 7f9acaffd700 1 --2- 172.21.15.162:0/3100529374 >> v2:172.21.15.162:6800/4121673848 conn(0x7f9ac4033f70 0x7f9ac4036430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.253+0000 7f9acaffd700 1 -- 172.21.15.162:0/3100529374 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f9acc044a40 con 0x7f9ad80fa530 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.253+0000 7f9acaffd700 10 client.14134.objecter ms_dispatch 0x7f9ad8001550 osd_map(2..2 src has 1..2) v4 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.253+0000 7f9acaffd700 3 client.14134.objecter handle_osd_map got epochs [2,2] > 0 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.253+0000 7f9acaffd700 3 client.14134.objecter handle_osd_map decoding full epoch 2 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.253+0000 7f9acaffd700 20 client.14134.objecter dump_active .. 0 homeless 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.253+0000 7f9add5b4700 1 --2- 172.21.15.162:0/3100529374 >> v2:172.21.15.162:6800/4121673848 conn(0x7f9ac4033f70 0x7f9ac4036430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.253+0000 7f9acaffd700 10 client.14134.objecter ms_handle_connect 0x7f9ac4033f70 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.253+0000 7f9add5b4700 1 --2- 172.21.15.162:0/3100529374 >> v2:172.21.15.162:6800/4121673848 conn(0x7f9ac4033f70 0x7f9ac4036430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f9ad4009780 tx=0x7f9ad4006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:32.259 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.254+0000 7f9ae0019700 1 -- 172.21.15.162:0/3100529374 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad805a3b0 con 0x7f9ad80fa530 2024-02-13T00:32:32.261 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.259+0000 7f9acaffd700 1 -- 172.21.15.162:0/3100529374 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9acc01e070 con 0x7f9ad80fa530 2024-02-13T00:32:32.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:31.134978+0000 mgr.y (mgr.14116) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:32:32.428 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:32.425+0000 7f9ae0019700 1 -- 172.21.15.162:0/3100529374 --> v2:172.21.15.162:6800/4121673848 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi162", "addr": "172.21.15.162", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ad810c250 con 0x7f9ac4033f70 2024-02-13T00:32:32.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [80B blob data] 2024-02-13T00:32:33.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.039+0000 7f9acaffd700 1 -- 172.21.15.162:0/3100529374 <== mgr.14116 v2:172.21.15.162:6800/4121673848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9ad810c250 con 0x7f9ac4033f70 2024-02-13T00:32:33.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: Added host 'smithi162' with addr '172.21.15.162' 2024-02-13T00:32:33.046 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.041+0000 7f9ae0019700 10 librados: watch_flush enter 2024-02-13T00:32:33.046 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.041+0000 7f9ae0019700 10 librados: watch_flush exit 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.041+0000 7f9ae0019700 20 client.14134.objecter shutdown clearing up homeless session... 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.041+0000 7f9ae0019700 10 client.14134.objecter successfully canceled tick 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.041+0000 7f9ae0019700 1 -- 172.21.15.162:0/3100529374 >> v2:172.21.15.162:6800/4121673848 conn(0x7f9ac4033f70 msgr2=0x7f9ac4036430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.041+0000 7f9ae0019700 1 --2- 172.21.15.162:0/3100529374 >> v2:172.21.15.162:6800/4121673848 conn(0x7f9ac4033f70 0x7f9ac4036430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f9ad4009780 tx=0x7f9ad4006cc0).stop 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.041+0000 7f9ae0019700 1 -- 172.21.15.162:0/3100529374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 msgr2=0x7f9ad810ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.041+0000 7f9ae0019700 1 --2- 172.21.15.162:0/3100529374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 0x7f9ad810ae70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9acc006940 tx=0x7f9acc0090f0).stop 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.041+0000 7f9ae0019700 1 -- 172.21.15.162:0/3100529374 shutdown_connections 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.041+0000 7f9ae0019700 1 --2- 172.21.15.162:0/3100529374 >> v2:172.21.15.162:6800/4121673848 conn(0x7f9ac4033f70 0x7f9ac4036430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.041+0000 7f9ae0019700 1 --2- 172.21.15.162:0/3100529374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ad80fa530 0x7f9ad810ae70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.041+0000 7f9ae0019700 1 -- 172.21.15.162:0/3100529374 >> 172.21.15.162:0/3100529374 conn(0x7f9ad80f5f30 msgr2=0x7f9ad80f78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.042+0000 7f9ae0019700 1 -- 172.21.15.162:0/3100529374 shutdown_connections 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.042+0000 7f9ae0019700 1 -- 172.21.15.162:0/3100529374 wait complete. 2024-02-13T00:32:33.047 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:33.042+0000 7f9ae0019700 1 librados: shutdown 2024-02-13T00:32:33.841 INFO:teuthology.orchestra.run.smithi162.stderr:Deploying unmanaged mon service... 2024-02-13T00:32:34.141 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:32.427119+0000 mgr.y (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "addr": "172.21.15.162", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:32:34.141 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:33.040783+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' 2024-02-13T00:32:34.141 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:32:33.041067+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Added host smithi162 2024-02-13T00:32:34.141 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:33.056597+0000 mon.a (mon.0) 54 : audit [DBG] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:32:34.522 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.488+0000 7fb7bee24700 1 Processor -- start 2024-02-13T00:32:34.522 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.488+0000 7fb7bee24700 1 -- start start 2024-02-13T00:32:34.522 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.488+0000 7fb7bee24700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 0x7fb7b80fb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:34.522 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.488+0000 7fb7bee24700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b80fb900 con 0x7fb7b80faff0 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.489+0000 7fb7bcbc0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 0x7fb7b80fb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.489+0000 7fb7bcbc0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 0x7fb7b80fb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41624/0 (socket says 172.21.15.162:41624) 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.489+0000 7fb7bcbc0700 1 -- 172.21.15.162:0/1669682254 learned_addr learned my addr 172.21.15.162:0/1669682254 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.489+0000 7fb7bcbc0700 1 -- 172.21.15.162:0/1669682254 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b80fba40 con 0x7fb7b80faff0 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.489+0000 7fb7bcbc0700 1 --2- 172.21.15.162:0/1669682254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 0x7fb7b80fb410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fb7ac009730 tx=0x7fb7ac0090f0).ready entity=mon.0 client_cookie=583842dc4a7b8a54 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.489+0000 7fb7b77fe700 1 -- 172.21.15.162:0/1669682254 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7ac00b580 con 0x7fb7b80faff0 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.489+0000 7fb7b77fe700 1 -- 172.21.15.162:0/1669682254 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7ac00bb90 con 0x7fb7b80faff0 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.489+0000 7fb7b77fe700 1 -- 172.21.15.162:0/1669682254 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7ac011c30 con 0x7fb7b80faff0 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 -- 172.21.15.162:0/1669682254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 msgr2=0x7fb7b80fb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 --2- 172.21.15.162:0/1669682254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 0x7fb7b80fb410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fb7ac009730 tx=0x7fb7ac0090f0).stop 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 -- 172.21.15.162:0/1669682254 shutdown_connections 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 --2- 172.21.15.162:0/1669682254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 0x7fb7b80fb410 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:34.523 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 -- 172.21.15.162:0/1669682254 >> 172.21.15.162:0/1669682254 conn(0x7fb7b80f67d0 msgr2=0x7fb7b80f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 -- 172.21.15.162:0/1669682254 shutdown_connections 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 -- 172.21.15.162:0/1669682254 wait complete. 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 librados: starting msgr at 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 librados: starting objecter 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 Processor -- start 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 -- start start 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 librados: setting wanted keys 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 librados: calling monclient init 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 0x7fb7b8109da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.490+0000 7fb7bee24700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b810a290 con 0x7fb7b80faff0 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7bcbc0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 0x7fb7b8109da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7bcbc0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 0x7fb7b8109da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41640/0 (socket says 172.21.15.162:41640) 2024-02-13T00:32:34.524 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7bcbc0700 1 -- 172.21.15.162:0/4216299339 learned_addr learned my addr 172.21.15.162:0/4216299339 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:34.525 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7bcbc0700 1 -- 172.21.15.162:0/4216299339 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac009370 con 0x7fb7b80faff0 2024-02-13T00:32:34.525 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7bcbc0700 1 --2- 172.21.15.162:0/4216299339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 0x7fb7b8109da0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb7ac00bf30 tx=0x7fb7ac0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:34.525 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7b5ffb700 10 client.?.objecter ms_handle_connect 0x7fb7b80faff0 2024-02-13T00:32:34.525 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7b5ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:34.525 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7b5ffb700 1 -- 172.21.15.162:0/4216299339 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7ac01a640 con 0x7fb7b80faff0 2024-02-13T00:32:34.526 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7b5ffb700 1 -- 172.21.15.162:0/4216299339 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7ac01ac50 con 0x7fb7b80faff0 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7b5ffb700 1 -- 172.21.15.162:0/4216299339 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7ac023ae0 con 0x7fb7b80faff0 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7bee24700 1 -- 172.21.15.162:0/4216299339 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b810a490 con 0x7fb7b80faff0 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7bee24700 10 client.14136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7bee24700 1 -- 172.21.15.162:0/4216299339 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b810b0d0 con 0x7fb7b80faff0 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.491+0000 7fb7bee24700 1 librados: init done 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.492+0000 7fb7bee24700 1 -- 172.21.15.162:0/4216299339 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b810a920 con 0x7fb7b80faff0 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.492+0000 7fb7b5ffb700 1 -- 172.21.15.162:0/4216299339 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7fb7ac01a7a0 con 0x7fb7b80faff0 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.493+0000 7fb7b5ffb700 1 --2- 172.21.15.162:0/4216299339 >> v2:172.21.15.162:6800/4121673848 conn(0x7fb7a8033f70 0x7fb7a8036430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.493+0000 7fb7b5ffb700 1 -- 172.21.15.162:0/4216299339 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7fb7ac044df0 con 0x7fb7b80faff0 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.493+0000 7fb7b5ffb700 10 client.14136.objecter ms_dispatch 0x7fb7b8001550 osd_map(2..2 src has 1..2) v4 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.493+0000 7fb7b5ffb700 3 client.14136.objecter handle_osd_map got epochs [2,2] > 0 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.493+0000 7fb7b5ffb700 3 client.14136.objecter handle_osd_map decoding full epoch 2 2024-02-13T00:32:34.527 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.493+0000 7fb7b5ffb700 20 client.14136.objecter dump_active .. 0 homeless 2024-02-13T00:32:34.528 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.493+0000 7fb7b7fff700 1 --2- 172.21.15.162:0/4216299339 >> v2:172.21.15.162:6800/4121673848 conn(0x7fb7a8033f70 0x7fb7a8036430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:34.528 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.493+0000 7fb7b5ffb700 10 client.14136.objecter ms_handle_connect 0x7fb7a8033f70 2024-02-13T00:32:34.528 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.493+0000 7fb7b7fff700 1 --2- 172.21.15.162:0/4216299339 >> v2:172.21.15.162:6800/4121673848 conn(0x7fb7a8033f70 0x7fb7a8036430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb7a4009780 tx=0x7fb7a4006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:34.528 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.497+0000 7fb7b5ffb700 1 -- 172.21.15.162:0/4216299339 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb7ac023520 con 0x7fb7b80faff0 2024-02-13T00:32:34.674 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.670+0000 7fb7bee24700 1 -- 172.21.15.162:0/4216299339 --> v2:172.21.15.162:6800/4121673848 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb7b80fcfb0 con 0x7fb7a8033f70 2024-02-13T00:32:34.679 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.675+0000 7fb7b5ffb700 1 -- 172.21.15.162:0/4216299339 <== mgr.14116 v2:172.21.15.162:6800/4121673848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb7b80fcfb0 con 0x7fb7a8033f70 2024-02-13T00:32:34.679 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-13T00:32:34.682 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 10 librados: watch_flush enter 2024-02-13T00:32:34.682 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 10 librados: watch_flush exit 2024-02-13T00:32:34.682 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 20 client.14136.objecter shutdown clearing up homeless session... 2024-02-13T00:32:34.682 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 10 client.14136.objecter successfully canceled tick 2024-02-13T00:32:34.682 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 1 -- 172.21.15.162:0/4216299339 >> v2:172.21.15.162:6800/4121673848 conn(0x7fb7a8033f70 msgr2=0x7fb7a8036430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:34.682 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 1 --2- 172.21.15.162:0/4216299339 >> v2:172.21.15.162:6800/4121673848 conn(0x7fb7a8033f70 0x7fb7a8036430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb7a4009780 tx=0x7fb7a4006cc0).stop 2024-02-13T00:32:34.682 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 1 -- 172.21.15.162:0/4216299339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 msgr2=0x7fb7b8109da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:34.682 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 1 --2- 172.21.15.162:0/4216299339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 0x7fb7b8109da0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb7ac00bf30 tx=0x7fb7ac0090f0).stop 2024-02-13T00:32:34.682 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 1 -- 172.21.15.162:0/4216299339 shutdown_connections 2024-02-13T00:32:34.682 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 1 --2- 172.21.15.162:0/4216299339 >> v2:172.21.15.162:6800/4121673848 conn(0x7fb7a8033f70 0x7fb7a8036430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:34.683 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 1 --2- 172.21.15.162:0/4216299339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7b80faff0 0x7fb7b8109da0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:34.683 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 1 -- 172.21.15.162:0/4216299339 >> 172.21.15.162:0/4216299339 conn(0x7fb7b80f67d0 msgr2=0x7fb7b80f80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:34.683 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 1 -- 172.21.15.162:0/4216299339 shutdown_connections 2024-02-13T00:32:34.683 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 1 -- 172.21.15.162:0/4216299339 wait complete. 2024-02-13T00:32:34.683 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:34.677+0000 7fb7bee24700 1 librados: shutdown 2024-02-13T00:32:35.333 INFO:teuthology.orchestra.run.smithi162.stderr:Deploying unmanaged mgr service... 2024-02-13T00:32:36.041 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.010+0000 7f3f97f45700 1 Processor -- start 2024-02-13T00:32:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.011+0000 7f3f97f45700 1 -- start start 2024-02-13T00:32:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.011+0000 7f3f97f45700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 0x7f3f900fed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.011+0000 7f3f97f45700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f900ff280 con 0x7f3f90070c20 2024-02-13T00:32:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.011+0000 7f3f95ce1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 0x7f3f900fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.011+0000 7f3f95ce1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 0x7f3f900fed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41648/0 (socket says 172.21.15.162:41648) 2024-02-13T00:32:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.011+0000 7f3f95ce1700 1 -- 172.21.15.162:0/2203573955 learned_addr learned my addr 172.21.15.162:0/2203573955 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.011+0000 7f3f95ce1700 1 -- 172.21.15.162:0/2203573955 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f900ff3c0 con 0x7f3f90070c20 2024-02-13T00:32:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.011+0000 7f3f95ce1700 1 --2- 172.21.15.162:0/2203573955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 0x7f3f900fed90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f3f8c009730 tx=0x7f3f8c0090f0).ready entity=mon.0 client_cookie=86cc8945a1609467 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.012+0000 7f3f94cdf700 1 -- 172.21.15.162:0/2203573955 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f8c00b610 con 0x7f3f90070c20 2024-02-13T00:32:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.012+0000 7f3f94cdf700 1 -- 172.21.15.162:0/2203573955 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f8c00bc20 con 0x7f3f90070c20 2024-02-13T00:32:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.012+0000 7f3f94cdf700 1 -- 172.21.15.162:0/2203573955 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f8c011ca0 con 0x7f3f90070c20 2024-02-13T00:32:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.012+0000 7f3f97f45700 1 -- 172.21.15.162:0/2203573955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 msgr2=0x7f3f900fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.012+0000 7f3f97f45700 1 --2- 172.21.15.162:0/2203573955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 0x7f3f900fed90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f3f8c009730 tx=0x7f3f8c0090f0).stop 2024-02-13T00:32:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.012+0000 7f3f97f45700 1 -- 172.21.15.162:0/2203573955 shutdown_connections 2024-02-13T00:32:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.012+0000 7f3f97f45700 1 --2- 172.21.15.162:0/2203573955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 0x7f3f900fed90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.012+0000 7f3f97f45700 1 -- 172.21.15.162:0/2203573955 >> 172.21.15.162:0/2203573955 conn(0x7f3f900f67d0 msgr2=0x7f3f900f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.012+0000 7f3f97f45700 1 -- 172.21.15.162:0/2203573955 shutdown_connections 2024-02-13T00:32:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.012+0000 7f3f97f45700 1 -- 172.21.15.162:0/2203573955 wait complete. 2024-02-13T00:32:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.013+0000 7f3f97f45700 1 librados: starting msgr at 2024-02-13T00:32:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.013+0000 7f3f97f45700 1 librados: starting objecter 2024-02-13T00:32:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.013+0000 7f3f97f45700 1 Processor -- start 2024-02-13T00:32:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.013+0000 7f3f97f45700 1 -- start start 2024-02-13T00:32:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.013+0000 7f3f97f45700 1 librados: setting wanted keys 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.013+0000 7f3f97f45700 1 librados: calling monclient init 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.013+0000 7f3f97f45700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 0x7f3f9010aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.013+0000 7f3f97f45700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9010b3c0 con 0x7f3f90070c20 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.013+0000 7f3f95ce1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 0x7f3f9010aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.013+0000 7f3f95ce1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 0x7f3f9010aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41660/0 (socket says 172.21.15.162:41660) 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.013+0000 7f3f95ce1700 1 -- 172.21.15.162:0/3682347594 learned_addr learned my addr 172.21.15.162:0/3682347594 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.013+0000 7f3f95ce1700 1 -- 172.21.15.162:0/3682347594 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8c009370 con 0x7f3f90070c20 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.014+0000 7f3f95ce1700 1 --2- 172.21.15.162:0/3682347594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 0x7f3f9010aed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f3f8c00bfc0 tx=0x7f3f8c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.014+0000 7f3f86ffd700 10 client.?.objecter ms_handle_connect 0x7f3f90070c20 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.014+0000 7f3f86ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.014+0000 7f3f86ffd700 1 -- 172.21.15.162:0/3682347594 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f8c01a640 con 0x7f3f90070c20 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.014+0000 7f3f86ffd700 1 -- 172.21.15.162:0/3682347594 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f8c01ac50 con 0x7f3f90070c20 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.014+0000 7f3f97f45700 1 -- 172.21.15.162:0/3682347594 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9010b5c0 con 0x7f3f90070c20 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.014+0000 7f3f86ffd700 1 -- 172.21.15.162:0/3682347594 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f8c023b30 con 0x7f3f90070c20 2024-02-13T00:32:36.044 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.014+0000 7f3f97f45700 10 client.14138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.014+0000 7f3f97f45700 1 -- 172.21.15.162:0/3682347594 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9010c1d0 con 0x7f3f90070c20 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.014+0000 7f3f97f45700 1 librados: init done 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.014+0000 7f3f86ffd700 1 -- 172.21.15.162:0/3682347594 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f3f8c023c90 con 0x7f3f90070c20 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.015+0000 7f3f86ffd700 1 --2- 172.21.15.162:0/3682347594 >> v2:172.21.15.162:6800/4121673848 conn(0x7f3f7c033f70 0x7f3f7c036430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.015+0000 7f3f86ffd700 1 -- 172.21.15.162:0/3682347594 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f3f8c044980 con 0x7f3f90070c20 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.015+0000 7f3f86ffd700 10 client.14138.objecter ms_dispatch 0x7f3f90001550 osd_map(2..2 src has 1..2) v4 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.015+0000 7f3f86ffd700 3 client.14138.objecter handle_osd_map got epochs [2,2] > 0 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.015+0000 7f3f86ffd700 3 client.14138.objecter handle_osd_map decoding full epoch 2 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.015+0000 7f3f86ffd700 20 client.14138.objecter dump_active .. 0 homeless 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.015+0000 7f3f954e0700 1 --2- 172.21.15.162:0/3682347594 >> v2:172.21.15.162:6800/4121673848 conn(0x7f3f7c033f70 0x7f3f7c036430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.015+0000 7f3f97f45700 1 -- 172.21.15.162:0/3682347594 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9010ba50 con 0x7f3f90070c20 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.015+0000 7f3f86ffd700 10 client.14138.objecter ms_handle_connect 0x7f3f7c033f70 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.015+0000 7f3f954e0700 1 --2- 172.21.15.162:0/3682347594 >> v2:172.21.15.162:6800/4121673848 conn(0x7f3f7c033f70 0x7f3f7c036430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f3f80009780 tx=0x7f3f80006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:36.045 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.020+0000 7f3f86ffd700 1 -- 172.21.15.162:0/3682347594 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3f9010ba50 con 0x7f3f90070c20 2024-02-13T00:32:36.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:34.673295+0000 mgr.y (mgr.14116) 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:32:36.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:32:34.674470+0000 mgr.y (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-13T00:32:36.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:34.676674+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' 2024-02-13T00:32:36.230 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.226+0000 7f3f97f45700 1 -- 172.21.15.162:0/3682347594 --> v2:172.21.15.162:6800/4121673848 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3f901001e0 con 0x7f3f7c033f70 2024-02-13T00:32:36.234 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.230+0000 7f3f86ffd700 1 -- 172.21.15.162:0/3682347594 <== mgr.14116 v2:172.21.15.162:6800/4121673848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3f901001e0 con 0x7f3f7c033f70 2024-02-13T00:32:36.234 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-13T00:32:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 10 librados: watch_flush enter 2024-02-13T00:32:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 10 librados: watch_flush exit 2024-02-13T00:32:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 20 client.14138.objecter shutdown clearing up homeless session... 2024-02-13T00:32:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 10 client.14138.objecter successfully canceled tick 2024-02-13T00:32:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 1 -- 172.21.15.162:0/3682347594 >> v2:172.21.15.162:6800/4121673848 conn(0x7f3f7c033f70 msgr2=0x7f3f7c036430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 1 --2- 172.21.15.162:0/3682347594 >> v2:172.21.15.162:6800/4121673848 conn(0x7f3f7c033f70 0x7f3f7c036430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f3f80009780 tx=0x7f3f80006cc0).stop 2024-02-13T00:32:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 1 -- 172.21.15.162:0/3682347594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 msgr2=0x7f3f9010aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 1 --2- 172.21.15.162:0/3682347594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 0x7f3f9010aed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f3f8c00bfc0 tx=0x7f3f8c0090f0).stop 2024-02-13T00:32:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 1 -- 172.21.15.162:0/3682347594 shutdown_connections 2024-02-13T00:32:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 1 --2- 172.21.15.162:0/3682347594 >> v2:172.21.15.162:6800/4121673848 conn(0x7f3f7c033f70 0x7f3f7c036430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 1 --2- 172.21.15.162:0/3682347594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f90070c20 0x7f3f9010aed0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 1 -- 172.21.15.162:0/3682347594 >> 172.21.15.162:0/3682347594 conn(0x7f3f900f67d0 msgr2=0x7f3f900f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 1 -- 172.21.15.162:0/3682347594 shutdown_connections 2024-02-13T00:32:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 1 -- 172.21.15.162:0/3682347594 wait complete. 2024-02-13T00:32:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:36.232+0000 7f3f97f45700 1 librados: shutdown 2024-02-13T00:32:37.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:36.228982+0000 mgr.y (mgr.14116) 13 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:32:37.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:32:36.229843+0000 mgr.y (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-13T00:32:37.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:36.231975+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' 2024-02-13T00:32:37.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:36.840542+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' 2024-02-13T00:32:37.738 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.733+0000 7feee1f01700 1 Processor -- start 2024-02-13T00:32:37.738 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.734+0000 7feee1f01700 1 -- start start 2024-02-13T00:32:37.738 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.734+0000 7feee1f01700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 0x7feedc0fdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:37.738 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.734+0000 7feee1f01700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc0fe0d0 con 0x7feedc0fd7c0 2024-02-13T00:32:37.738 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.734+0000 7feedb7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 0x7feedc0fdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:37.739 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.734+0000 7feedb7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 0x7feedc0fdbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41664/0 (socket says 172.21.15.162:41664) 2024-02-13T00:32:37.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.734+0000 7feedb7fe700 1 -- 172.21.15.162:0/301535218 learned_addr learned my addr 172.21.15.162:0/301535218 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:37.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.734+0000 7feedb7fe700 1 -- 172.21.15.162:0/301535218 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc0fe210 con 0x7feedc0fd7c0 2024-02-13T00:32:37.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.734+0000 7feedb7fe700 1 --2- 172.21.15.162:0/301535218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 0x7feedc0fdbe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7feecc009730 tx=0x7feecc0090f0).ready entity=mon.0 client_cookie=3bb75d3b97b767e6 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:37.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.734+0000 7feeda7fc700 1 -- 172.21.15.162:0/301535218 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feecc00b580 con 0x7feedc0fd7c0 2024-02-13T00:32:37.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.735+0000 7feeda7fc700 1 -- 172.21.15.162:0/301535218 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feecc00bbb0 con 0x7feedc0fd7c0 2024-02-13T00:32:37.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.735+0000 7feeda7fc700 1 -- 172.21.15.162:0/301535218 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feecc011bc0 con 0x7feedc0fd7c0 2024-02-13T00:32:37.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.735+0000 7feee1f01700 1 -- 172.21.15.162:0/301535218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 msgr2=0x7feedc0fdbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:37.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.735+0000 7feee1f01700 1 --2- 172.21.15.162:0/301535218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 0x7feedc0fdbe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7feecc009730 tx=0x7feecc0090f0).stop 2024-02-13T00:32:37.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.736+0000 7feee1f01700 1 -- 172.21.15.162:0/301535218 shutdown_connections 2024-02-13T00:32:37.740 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.736+0000 7feee1f01700 1 --2- 172.21.15.162:0/301535218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 0x7feedc0fdbe0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:37.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.736+0000 7feee1f01700 1 -- 172.21.15.162:0/301535218 >> 172.21.15.162:0/301535218 conn(0x7feedc0f5f60 msgr2=0x7feedc0f83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:37.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.736+0000 7feee1f01700 1 -- 172.21.15.162:0/301535218 shutdown_connections 2024-02-13T00:32:37.741 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.736+0000 7feee1f01700 1 -- 172.21.15.162:0/301535218 wait complete. 2024-02-13T00:32:37.744 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.736+0000 7feee1f01700 1 librados: starting msgr at 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.736+0000 7feee1f01700 1 librados: starting objecter 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.736+0000 7feee1f01700 1 Processor -- start 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feee1f01700 1 -- start start 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feee1f01700 1 librados: setting wanted keys 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feee1f01700 1 librados: calling monclient init 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feee1f01700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 0x7feedc10d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feee1f01700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc10d540 con 0x7feedc0fd7c0 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feedb7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 0x7feedc10d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feedb7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 0x7feedc10d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41666/0 (socket says 172.21.15.162:41666) 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feedb7fe700 1 -- 172.21.15.162:0/1706034598 learned_addr learned my addr 172.21.15.162:0/1706034598 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feedb7fe700 1 -- 172.21.15.162:0/1706034598 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feecc009370 con 0x7feedc0fd7c0 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feedb7fe700 1 --2- 172.21.15.162:0/1706034598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 0x7feedc10d050 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7feecc006e90 tx=0x7feecc0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feed8ff9700 10 client.?.objecter ms_handle_connect 0x7feedc0fd7c0 2024-02-13T00:32:37.745 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feed8ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.737+0000 7feed8ff9700 1 -- 172.21.15.162:0/1706034598 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feecc01a640 con 0x7feedc0fd7c0 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.738+0000 7feee1f01700 1 -- 172.21.15.162:0/1706034598 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feedc10d740 con 0x7feedc0fd7c0 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.738+0000 7feed8ff9700 1 -- 172.21.15.162:0/1706034598 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feecc01ac70 con 0x7feedc0fd7c0 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.738+0000 7feee1f01700 10 client.14140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.738+0000 7feed8ff9700 1 -- 172.21.15.162:0/1706034598 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feecc023b90 con 0x7feedc0fd7c0 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.738+0000 7feee1f01700 1 -- 172.21.15.162:0/1706034598 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feedc10dbd0 con 0x7feedc0fd7c0 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.738+0000 7feee1f01700 1 librados: init done 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.739+0000 7feed8ff9700 1 -- 172.21.15.162:0/1706034598 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7feecc02c3f0 con 0x7feedc0fd7c0 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.739+0000 7feee1f01700 1 -- 172.21.15.162:0/1706034598 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feedc046d80 con 0x7feedc0fd7c0 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.739+0000 7feed8ff9700 1 --2- 172.21.15.162:0/1706034598 >> v2:172.21.15.162:6800/4121673848 conn(0x7feec4033f20 0x7feec40363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.739+0000 7feed8ff9700 1 -- 172.21.15.162:0/1706034598 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7feecc044910 con 0x7feedc0fd7c0 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.739+0000 7feed8ff9700 10 client.14140.objecter ms_dispatch 0x7feedc001550 osd_map(2..2 src has 1..2) v4 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.739+0000 7feed8ff9700 3 client.14140.objecter handle_osd_map got epochs [2,2] > 0 2024-02-13T00:32:37.746 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.739+0000 7feed8ff9700 3 client.14140.objecter handle_osd_map decoding full epoch 2 2024-02-13T00:32:37.747 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.739+0000 7feed8ff9700 20 client.14140.objecter dump_active .. 0 homeless 2024-02-13T00:32:37.747 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.739+0000 7feedaffd700 1 --2- 172.21.15.162:0/1706034598 >> v2:172.21.15.162:6800/4121673848 conn(0x7feec4033f20 0x7feec40363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:37.747 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.739+0000 7feedaffd700 1 --2- 172.21.15.162:0/1706034598 >> v2:172.21.15.162:6800/4121673848 conn(0x7feec4033f20 0x7feec40363e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7feed0009780 tx=0x7feed0006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:37.747 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.739+0000 7feed8ff9700 10 client.14140.objecter ms_handle_connect 0x7feec4033f20 2024-02-13T00:32:37.748 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.744+0000 7feed8ff9700 1 -- 172.21.15.162:0/1706034598 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7feecc01a7a0 con 0x7feedc0fd7c0 2024-02-13T00:32:37.944 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.940+0000 7feee1f01700 1 -- 172.21.15.162:0/1706034598 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7feedc0fad70 con 0x7feedc0fd7c0 2024-02-13T00:32:37.946 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.943+0000 7feed8ff9700 1 -- 172.21.15.162:0/1706034598 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7feecc01e070 con 0x7feedc0fd7c0 2024-02-13T00:32:37.950 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.944+0000 7feee1f01700 10 librados: watch_flush enter 2024-02-13T00:32:37.951 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.944+0000 7feee1f01700 10 librados: watch_flush exit 2024-02-13T00:32:37.951 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.944+0000 7feee1f01700 20 client.14140.objecter shutdown clearing up homeless session... 2024-02-13T00:32:37.951 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.944+0000 7feee1f01700 10 client.14140.objecter successfully canceled tick 2024-02-13T00:32:37.951 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.945+0000 7feee1f01700 1 -- 172.21.15.162:0/1706034598 >> v2:172.21.15.162:6800/4121673848 conn(0x7feec4033f20 msgr2=0x7feec40363e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:37.951 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.945+0000 7feee1f01700 1 --2- 172.21.15.162:0/1706034598 >> v2:172.21.15.162:6800/4121673848 conn(0x7feec4033f20 0x7feec40363e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7feed0009780 tx=0x7feed0006cc0).stop 2024-02-13T00:32:37.951 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.945+0000 7feee1f01700 1 -- 172.21.15.162:0/1706034598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 msgr2=0x7feedc10d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:37.951 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.945+0000 7feee1f01700 1 --2- 172.21.15.162:0/1706034598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 0x7feedc10d050 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7feecc006e90 tx=0x7feecc0090f0).stop 2024-02-13T00:32:37.951 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.945+0000 7feee1f01700 1 -- 172.21.15.162:0/1706034598 shutdown_connections 2024-02-13T00:32:37.951 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.945+0000 7feee1f01700 1 --2- 172.21.15.162:0/1706034598 >> v2:172.21.15.162:6800/4121673848 conn(0x7feec4033f20 0x7feec40363e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:37.951 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.945+0000 7feee1f01700 1 --2- 172.21.15.162:0/1706034598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feedc0fd7c0 0x7feedc10d050 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:37.951 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.945+0000 7feee1f01700 1 -- 172.21.15.162:0/1706034598 >> 172.21.15.162:0/1706034598 conn(0x7feedc0f5f60 msgr2=0x7feedc0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:37.951 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.945+0000 7feee1f01700 1 -- 172.21.15.162:0/1706034598 shutdown_connections 2024-02-13T00:32:37.952 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.945+0000 7feee1f01700 1 -- 172.21.15.162:0/1706034598 wait complete. 2024-02-13T00:32:37.952 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:37.945+0000 7feee1f01700 1 librados: shutdown 2024-02-13T00:32:38.616 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:37.522112+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' 2024-02-13T00:32:38.616 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:37.944820+0000 mon.a (mon.0) 59 : audit [INF] from='client.? 172.21.15.162:0/1706034598' entity='client.admin' 2024-02-13T00:32:39.466 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.461+0000 7f37fa834700 1 Processor -- start 2024-02-13T00:32:39.466 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.461+0000 7f37fa834700 1 -- start start 2024-02-13T00:32:39.466 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.461+0000 7f37fa834700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 0x7f37f40fed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:39.466 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.461+0000 7f37fa834700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f40ff280 con 0x7f37f4070c20 2024-02-13T00:32:39.466 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.461+0000 7f37f3fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 0x7f37f40fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:39.466 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.461+0000 7f37f3fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 0x7f37f40fed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41674/0 (socket says 172.21.15.162:41674) 2024-02-13T00:32:39.467 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.461+0000 7f37f3fff700 1 -- 172.21.15.162:0/896297767 learned_addr learned my addr 172.21.15.162:0/896297767 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:39.467 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.462+0000 7f37f3fff700 1 -- 172.21.15.162:0/896297767 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f40ff3c0 con 0x7f37f4070c20 2024-02-13T00:32:39.467 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.462+0000 7f37f3fff700 1 --2- 172.21.15.162:0/896297767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 0x7f37f40fed90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f37e8009730 tx=0x7f37e80090f0).ready entity=mon.0 client_cookie=34fa95080ab2e5f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:39.467 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.462+0000 7f37f2ffd700 1 -- 172.21.15.162:0/896297767 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37e800b610 con 0x7f37f4070c20 2024-02-13T00:32:39.467 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.462+0000 7f37f2ffd700 1 -- 172.21.15.162:0/896297767 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37e800bc40 con 0x7f37f4070c20 2024-02-13T00:32:39.467 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.462+0000 7f37f2ffd700 1 -- 172.21.15.162:0/896297767 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37e8011c10 con 0x7f37f4070c20 2024-02-13T00:32:39.467 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.463+0000 7f37fa834700 1 -- 172.21.15.162:0/896297767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 msgr2=0x7f37f40fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:39.467 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.463+0000 7f37fa834700 1 --2- 172.21.15.162:0/896297767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 0x7f37f40fed90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f37e8009730 tx=0x7f37e80090f0).stop 2024-02-13T00:32:39.470 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.463+0000 7f37fa834700 1 -- 172.21.15.162:0/896297767 shutdown_connections 2024-02-13T00:32:39.470 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.463+0000 7f37fa834700 1 --2- 172.21.15.162:0/896297767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 0x7f37f40fed90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:39.470 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.463+0000 7f37fa834700 1 -- 172.21.15.162:0/896297767 >> 172.21.15.162:0/896297767 conn(0x7f37f40f67d0 msgr2=0x7f37f40f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:39.470 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.463+0000 7f37fa834700 1 -- 172.21.15.162:0/896297767 shutdown_connections 2024-02-13T00:32:39.470 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.463+0000 7f37fa834700 1 -- 172.21.15.162:0/896297767 wait complete. 2024-02-13T00:32:39.470 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.464+0000 7f37fa834700 1 librados: starting msgr at 2024-02-13T00:32:39.470 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.464+0000 7f37fa834700 1 librados: starting objecter 2024-02-13T00:32:39.470 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.464+0000 7f37fa834700 1 Processor -- start 2024-02-13T00:32:39.470 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.464+0000 7f37fa834700 1 -- start start 2024-02-13T00:32:39.470 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.464+0000 7f37fa834700 1 librados: setting wanted keys 2024-02-13T00:32:39.470 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.464+0000 7f37fa834700 1 librados: calling monclient init 2024-02-13T00:32:39.470 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.464+0000 7f37fa834700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 0x7f37f410ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.464+0000 7f37fa834700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f410b370 con 0x7f37f4070c20 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37f3fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 0x7f37f410ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37f3fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 0x7f37f410ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41676/0 (socket says 172.21.15.162:41676) 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37f3fff700 1 -- 172.21.15.162:0/3531734528 learned_addr learned my addr 172.21.15.162:0/3531734528 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37f3fff700 1 -- 172.21.15.162:0/3531734528 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e8009370 con 0x7f37f4070c20 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37f3fff700 1 --2- 172.21.15.162:0/3531734528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 0x7f37f410ae80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f37e8009f00 tx=0x7f37e80090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37f17fa700 10 client.?.objecter ms_handle_connect 0x7f37f4070c20 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37f17fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37f17fa700 1 -- 172.21.15.162:0/3531734528 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37e801a430 con 0x7f37f4070c20 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37f17fa700 1 -- 172.21.15.162:0/3531734528 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37e801aa60 con 0x7f37f4070c20 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37fa834700 1 -- 172.21.15.162:0/3531734528 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f410b570 con 0x7f37f4070c20 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37f17fa700 1 -- 172.21.15.162:0/3531734528 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37e8023a10 con 0x7f37f4070c20 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37fa834700 10 client.14142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:39.471 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37fa834700 1 -- 172.21.15.162:0/3531734528 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f410ba00 con 0x7f37f4070c20 2024-02-13T00:32:39.472 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.465+0000 7f37fa834700 1 librados: init done 2024-02-13T00:32:39.472 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.466+0000 7f37f17fa700 1 -- 172.21.15.162:0/3531734528 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f37e8023ca0 con 0x7f37f4070c20 2024-02-13T00:32:39.472 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.466+0000 7f37f17fa700 1 --2- 172.21.15.162:0/3531734528 >> v2:172.21.15.162:6800/4121673848 conn(0x7f37dc033f70 0x7f37dc036430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.466+0000 7f37f17fa700 1 -- 172.21.15.162:0/3531734528 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f37e80447a0 con 0x7f37f4070c20 2024-02-13T00:32:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.466+0000 7f37f17fa700 10 client.14142.objecter ms_dispatch 0x7f37f4001550 osd_map(2..2 src has 1..2) v4 2024-02-13T00:32:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.466+0000 7f37f17fa700 3 client.14142.objecter handle_osd_map got epochs [2,2] > 0 2024-02-13T00:32:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.466+0000 7f37f17fa700 3 client.14142.objecter handle_osd_map decoding full epoch 2 2024-02-13T00:32:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.466+0000 7f37f17fa700 20 client.14142.objecter dump_active .. 0 homeless 2024-02-13T00:32:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.466+0000 7f37f37fe700 1 --2- 172.21.15.162:0/3531734528 >> v2:172.21.15.162:6800/4121673848 conn(0x7f37dc033f70 0x7f37dc036430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.466+0000 7f37fa834700 1 -- 172.21.15.162:0/3531734528 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f405a3b0 con 0x7f37f4070c20 2024-02-13T00:32:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.467+0000 7f37f17fa700 10 client.14142.objecter ms_handle_connect 0x7f37dc033f70 2024-02-13T00:32:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.467+0000 7f37f37fe700 1 --2- 172.21.15.162:0/3531734528 >> v2:172.21.15.162:6800/4121673848 conn(0x7f37dc033f70 0x7f37dc036430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f37e4009780 tx=0x7f37e4006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:39.475 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.471+0000 7f37f17fa700 1 -- 172.21.15.162:0/3531734528 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f37e801e080 con 0x7f37f4070c20 2024-02-13T00:32:39.654 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.649+0000 7f37fa834700 1 -- 172.21.15.162:0/3531734528 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f37f4046d80 con 0x7f37f4070c20 2024-02-13T00:32:39.656 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.652+0000 7f37f17fa700 1 -- 172.21.15.162:0/3531734528 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f37e8027070 con 0x7f37f4070c20 2024-02-13T00:32:39.658 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.654+0000 7f37fa834700 10 librados: watch_flush enter 2024-02-13T00:32:39.658 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.654+0000 7f37fa834700 10 librados: watch_flush exit 2024-02-13T00:32:39.660 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.654+0000 7f37fa834700 20 client.14142.objecter shutdown clearing up homeless session... 2024-02-13T00:32:39.660 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.654+0000 7f37fa834700 10 client.14142.objecter successfully canceled tick 2024-02-13T00:32:39.660 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.654+0000 7f37fa834700 1 -- 172.21.15.162:0/3531734528 >> v2:172.21.15.162:6800/4121673848 conn(0x7f37dc033f70 msgr2=0x7f37dc036430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:39.660 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.654+0000 7f37fa834700 1 --2- 172.21.15.162:0/3531734528 >> v2:172.21.15.162:6800/4121673848 conn(0x7f37dc033f70 0x7f37dc036430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f37e4009780 tx=0x7f37e4006cc0).stop 2024-02-13T00:32:39.660 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.654+0000 7f37fa834700 1 -- 172.21.15.162:0/3531734528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 msgr2=0x7f37f410ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:39.660 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.654+0000 7f37fa834700 1 --2- 172.21.15.162:0/3531734528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 0x7f37f410ae80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f37e8009f00 tx=0x7f37e80090f0).stop 2024-02-13T00:32:39.660 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.655+0000 7f37fa834700 1 -- 172.21.15.162:0/3531734528 shutdown_connections 2024-02-13T00:32:39.660 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.655+0000 7f37fa834700 1 --2- 172.21.15.162:0/3531734528 >> v2:172.21.15.162:6800/4121673848 conn(0x7f37dc033f70 0x7f37dc036430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:39.660 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.655+0000 7f37fa834700 1 --2- 172.21.15.162:0/3531734528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37f4070c20 0x7f37f410ae80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:39.660 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.655+0000 7f37fa834700 1 -- 172.21.15.162:0/3531734528 >> 172.21.15.162:0/3531734528 conn(0x7f37f40f67d0 msgr2=0x7f37f40f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:39.660 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.655+0000 7f37fa834700 1 -- 172.21.15.162:0/3531734528 shutdown_connections 2024-02-13T00:32:39.661 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.655+0000 7f37fa834700 1 -- 172.21.15.162:0/3531734528 wait complete. 2024-02-13T00:32:39.661 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:39.655+0000 7f37fa834700 1 librados: shutdown 2024-02-13T00:32:40.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:39.280071+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.162:0/2551652524' entity='mgr.y' 2024-02-13T00:32:40.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:39.654455+0000 mon.a (mon.0) 61 : audit [INF] from='client.? 172.21.15.162:0/3531734528' entity='client.admin' 2024-02-13T00:32:41.188 INFO:teuthology.orchestra.run.smithi162.stderr:Enabling the dashboard module... 2024-02-13T00:32:41.793 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.784+0000 7f38230b0700 1 Processor -- start 2024-02-13T00:32:41.793 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.784+0000 7f38230b0700 1 -- start start 2024-02-13T00:32:41.793 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.784+0000 7f38230b0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 0x7f381c0fc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:41.793 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.784+0000 7f38230b0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0fca20 con 0x7f381c0fc110 2024-02-13T00:32:41.793 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.784+0000 7f3820e4c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 0x7f381c0fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:41.793 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.784+0000 7f3820e4c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 0x7f381c0fc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41692/0 (socket says 172.21.15.162:41692) 2024-02-13T00:32:41.793 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.784+0000 7f3820e4c700 1 -- 172.21.15.162:0/2535385551 learned_addr learned my addr 172.21.15.162:0/2535385551 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:41.793 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.785+0000 7f3820e4c700 1 -- 172.21.15.162:0/2535385551 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c0fcb60 con 0x7f381c0fc110 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.785+0000 7f3820e4c700 1 --2- 172.21.15.162:0/2535385551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 0x7f381c0fc530 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f3810009730 tx=0x7f38100090f0).ready entity=mon.0 client_cookie=8118802adaad1bbf server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.785+0000 7f381b7fe700 1 -- 172.21.15.162:0/2535385551 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f381000b580 con 0x7f381c0fc110 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.785+0000 7f381b7fe700 1 -- 172.21.15.162:0/2535385551 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f381000bbb0 con 0x7f381c0fc110 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.785+0000 7f381b7fe700 1 -- 172.21.15.162:0/2535385551 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3810011bc0 con 0x7f381c0fc110 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.785+0000 7f38230b0700 1 -- 172.21.15.162:0/2535385551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 msgr2=0x7f381c0fc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.785+0000 7f38230b0700 1 --2- 172.21.15.162:0/2535385551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 0x7f381c0fc530 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f3810009730 tx=0x7f38100090f0).stop 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.785+0000 7f38230b0700 1 -- 172.21.15.162:0/2535385551 shutdown_connections 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f38230b0700 1 --2- 172.21.15.162:0/2535385551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 0x7f381c0fc530 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f38230b0700 1 -- 172.21.15.162:0/2535385551 >> 172.21.15.162:0/2535385551 conn(0x7f381c0f7930 msgr2=0x7f381c0f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f38230b0700 1 -- 172.21.15.162:0/2535385551 shutdown_connections 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f38230b0700 1 -- 172.21.15.162:0/2535385551 wait complete. 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f38230b0700 1 librados: starting msgr at 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f38230b0700 1 librados: starting objecter 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f38230b0700 1 Processor -- start 2024-02-13T00:32:41.794 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f38230b0700 1 -- start start 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f38230b0700 1 librados: setting wanted keys 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f38230b0700 1 librados: calling monclient init 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f38230b0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 0x7f381c10ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f38230b0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c10b360 con 0x7f381c0fc110 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.786+0000 7f3820e4c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 0x7f381c10ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f3820e4c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 0x7f381c10ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41706/0 (socket says 172.21.15.162:41706) 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f3820e4c700 1 -- 172.21.15.162:0/2425723569 learned_addr learned my addr 172.21.15.162:0/2425723569 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f3820e4c700 1 -- 172.21.15.162:0/2425723569 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3810009370 con 0x7f381c0fc110 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f3820e4c700 1 --2- 172.21.15.162:0/2425723569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 0x7f381c10ae70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f3810006e90 tx=0x7f38100090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f3819ffb700 10 client.?.objecter ms_handle_connect 0x7f381c0fc110 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f3819ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f3819ffb700 1 -- 172.21.15.162:0/2425723569 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f381001a640 con 0x7f381c0fc110 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f38230b0700 1 -- 172.21.15.162:0/2425723569 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381c10b560 con 0x7f381c0fc110 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f3819ffb700 1 -- 172.21.15.162:0/2425723569 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f381001ac70 con 0x7f381c0fc110 2024-02-13T00:32:41.795 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f3819ffb700 1 -- 172.21.15.162:0/2425723569 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3810023b90 con 0x7f381c0fc110 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f38230b0700 10 client.14144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f38230b0700 1 -- 172.21.15.162:0/2425723569 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381c10b9f0 con 0x7f381c0fc110 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.787+0000 7f38230b0700 1 librados: init done 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.788+0000 7f3819ffb700 1 -- 172.21.15.162:0/2425723569 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f381002c3f0 con 0x7f381c0fc110 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.788+0000 7f3819ffb700 1 --2- 172.21.15.162:0/2425723569 >> v2:172.21.15.162:6800/4121673848 conn(0x7f380c033f20 0x7f380c0363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.788+0000 7f3819ffb700 1 -- 172.21.15.162:0/2425723569 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f3810044c80 con 0x7f381c0fc110 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.788+0000 7f3819ffb700 10 client.14144.objecter ms_dispatch 0x7f381c001550 osd_map(2..2 src has 1..2) v4 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.788+0000 7f3819ffb700 3 client.14144.objecter handle_osd_map got epochs [2,2] > 0 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.788+0000 7f3819ffb700 3 client.14144.objecter handle_osd_map decoding full epoch 2 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.788+0000 7f3819ffb700 20 client.14144.objecter dump_active .. 0 homeless 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.788+0000 7f381bfff700 1 --2- 172.21.15.162:0/2425723569 >> v2:172.21.15.162:6800/4121673848 conn(0x7f380c033f20 0x7f380c0363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.789+0000 7f38230b0700 1 -- 172.21.15.162:0/2425723569 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381c05a3b0 con 0x7f381c0fc110 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.789+0000 7f3819ffb700 10 client.14144.objecter ms_handle_connect 0x7f380c033f20 2024-02-13T00:32:41.796 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.789+0000 7f381bfff700 1 --2- 172.21.15.162:0/2425723569 >> v2:172.21.15.162:6800/4121673848 conn(0x7f380c033f20 0x7f380c0363e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f3808009780 tx=0x7f3808006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:41.797 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:41.793+0000 7f3819ffb700 1 -- 172.21.15.162:0/2425723569 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f381001e070 con 0x7f381c0fc110 2024-02-13T00:32:42.006 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.002+0000 7f38230b0700 1 -- 172.21.15.162:0/2425723569 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f381c046d80 con 0x7f381c0fc110 2024-02-13T00:32:42.058 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.054+0000 7f3819ffb700 1 -- 172.21.15.162:0/2425723569 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mgrmap(e 9) v1 ==== 38595+0+0 (secure 0 0 0) 0x7f381002c6a0 con 0x7f381c0fc110 2024-02-13T00:32:42.060 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.056+0000 7f3819ffb700 1 -- 172.21.15.162:0/2425723569 <== mon.0 v2:172.21.15.162:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f38100103a0 con 0x7f381c0fc110 2024-02-13T00:32:42.063 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 10 librados: watch_flush enter 2024-02-13T00:32:42.063 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 10 librados: watch_flush exit 2024-02-13T00:32:42.063 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 20 client.14144.objecter shutdown clearing up homeless session... 2024-02-13T00:32:42.063 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 10 client.14144.objecter successfully canceled tick 2024-02-13T00:32:42.063 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 1 -- 172.21.15.162:0/2425723569 >> v2:172.21.15.162:6800/4121673848 conn(0x7f380c033f20 msgr2=0x7f380c0363e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 1 --2- 172.21.15.162:0/2425723569 >> v2:172.21.15.162:6800/4121673848 conn(0x7f380c033f20 0x7f380c0363e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f3808009780 tx=0x7f3808006cc0).stop 2024-02-13T00:32:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 1 -- 172.21.15.162:0/2425723569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 msgr2=0x7f381c10ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 1 --2- 172.21.15.162:0/2425723569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 0x7f381c10ae70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f3810006e90 tx=0x7f38100090f0).stop 2024-02-13T00:32:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 1 -- 172.21.15.162:0/2425723569 shutdown_connections 2024-02-13T00:32:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 1 --2- 172.21.15.162:0/2425723569 >> v2:172.21.15.162:6800/4121673848 conn(0x7f380c033f20 0x7f380c0363e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 1 --2- 172.21.15.162:0/2425723569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f381c0fc110 0x7f381c10ae70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 1 -- 172.21.15.162:0/2425723569 >> 172.21.15.162:0/2425723569 conn(0x7f381c0f7930 msgr2=0x7f381c0f91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 1 -- 172.21.15.162:0/2425723569 shutdown_connections 2024-02-13T00:32:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 1 -- 172.21.15.162:0/2425723569 wait complete. 2024-02-13T00:32:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:42.058+0000 7f38230b0700 1 librados: shutdown 2024-02-13T00:32:42.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:42.004630+0000 mon.a (mon.0) 62 : audit [INF] from='client.? 172.21.15.162:0/2425723569' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-13T00:32:42.330 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ignoring --setuser ceph since I am not root 2024-02-13T00:32:42.330 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ignoring --setgroup ceph since I am not root 2024-02-13T00:32:42.330 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:42.214+0000 7f00edfc7680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T00:32:42.330 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:42.295+0000 7f00edfc7680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T00:32:43.065 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:42.665+0000 7f00edfc7680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T00:32:43.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:42.058035+0000 mon.a (mon.0) 63 : audit [INF] from='client.? 172.21.15.162:0/2425723569' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-13T00:32:43.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:42.058077+0000 mon.a (mon.0) 64 : cluster [DBG] mgrmap e9: y(active, since 17s) 2024-02-13T00:32:43.541 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.537+0000 7fa5d2fda700 1 Processor -- start 2024-02-13T00:32:43.542 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.537+0000 7fa5d2fda700 1 -- start start 2024-02-13T00:32:43.542 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.537+0000 7fa5d2fda700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 0x7fa5cc0fb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:43.543 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.537+0000 7fa5d2fda700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0fb930 con 0x7fa5cc0fb020 2024-02-13T00:32:43.543 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.537+0000 7fa5d0d76700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 0x7fa5cc0fb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:43.543 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.538+0000 7fa5d0d76700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 0x7fa5cc0fb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41728/0 (socket says 172.21.15.162:41728) 2024-02-13T00:32:43.543 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.538+0000 7fa5d0d76700 1 -- 172.21.15.162:0/147511899 learned_addr learned my addr 172.21.15.162:0/147511899 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:43.543 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.538+0000 7fa5d0d76700 1 -- 172.21.15.162:0/147511899 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc0fba70 con 0x7fa5cc0fb020 2024-02-13T00:32:43.543 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.538+0000 7fa5d0d76700 1 --2- 172.21.15.162:0/147511899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 0x7fa5cc0fb440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa5c0009730 tx=0x7fa5c00090f0).ready entity=mon.0 client_cookie=3c0dd5c4f0645ad8 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:43.543 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.538+0000 7fa5cb7fe700 1 -- 172.21.15.162:0/147511899 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5c000b580 con 0x7fa5cc0fb020 2024-02-13T00:32:43.545 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.538+0000 7fa5cb7fe700 1 -- 172.21.15.162:0/147511899 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5c000bbb0 con 0x7fa5cc0fb020 2024-02-13T00:32:43.545 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.538+0000 7fa5cb7fe700 1 -- 172.21.15.162:0/147511899 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5c0011bc0 con 0x7fa5cc0fb020 2024-02-13T00:32:43.545 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 -- 172.21.15.162:0/147511899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 msgr2=0x7fa5cc0fb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:43.545 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 --2- 172.21.15.162:0/147511899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 0x7fa5cc0fb440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa5c0009730 tx=0x7fa5c00090f0).stop 2024-02-13T00:32:43.545 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 -- 172.21.15.162:0/147511899 shutdown_connections 2024-02-13T00:32:43.545 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 --2- 172.21.15.162:0/147511899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 0x7fa5cc0fb440 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:43.545 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 -- 172.21.15.162:0/147511899 >> 172.21.15.162:0/147511899 conn(0x7fa5cc0f6820 msgr2=0x7fa5cc0f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:43.545 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 -- 172.21.15.162:0/147511899 shutdown_connections 2024-02-13T00:32:43.545 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 -- 172.21.15.162:0/147511899 wait complete. 2024-02-13T00:32:43.545 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 librados: starting msgr at 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 librados: starting objecter 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 Processor -- start 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 -- start start 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 librados: setting wanted keys 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.539+0000 7fa5d2fda700 1 librados: calling monclient init 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5d2fda700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 0x7fa5cc10d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5d2fda700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc10d5a0 con 0x7fa5cc0fb020 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5d0d76700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 0x7fa5cc10d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5d0d76700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 0x7fa5cc10d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41738/0 (socket says 172.21.15.162:41738) 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5d0d76700 1 -- 172.21.15.162:0/319174083 learned_addr learned my addr 172.21.15.162:0/319174083 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5d0d76700 1 -- 172.21.15.162:0/319174083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c0009370 con 0x7fa5cc0fb020 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5d0d76700 1 --2- 172.21.15.162:0/319174083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 0x7fa5cc10d0b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa5c0006e90 tx=0x7fa5c00090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5c9ffb700 10 client.?.objecter ms_handle_connect 0x7fa5cc0fb020 2024-02-13T00:32:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5c9ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:43.547 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5c9ffb700 1 -- 172.21.15.162:0/319174083 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5c001a640 con 0x7fa5cc0fb020 2024-02-13T00:32:43.547 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5d2fda700 1 -- 172.21.15.162:0/319174083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc10d7a0 con 0x7fa5cc0fb020 2024-02-13T00:32:43.547 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5c9ffb700 1 -- 172.21.15.162:0/319174083 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5c001ac70 con 0x7fa5cc0fb020 2024-02-13T00:32:43.547 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5c9ffb700 1 -- 172.21.15.162:0/319174083 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5c0023b90 con 0x7fa5cc0fb020 2024-02-13T00:32:43.547 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5d2fda700 10 client.14148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:43.547 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5d2fda700 1 -- 172.21.15.162:0/319174083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc10dc30 con 0x7fa5cc0fb020 2024-02-13T00:32:43.547 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.540+0000 7fa5d2fda700 1 librados: init done 2024-02-13T00:32:43.548 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.541+0000 7fa5d2fda700 1 -- 172.21.15.162:0/319174083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc046d80 con 0x7fa5cc0fb020 2024-02-13T00:32:43.548 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.542+0000 7fa5c9ffb700 1 -- 172.21.15.162:0/319174083 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 9) v1 ==== 38595+0+0 (secure 0 0 0) 0x7fa5c001a7a0 con 0x7fa5cc0fb020 2024-02-13T00:32:43.548 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.542+0000 7fa5c9ffb700 1 --2- 172.21.15.162:0/319174083 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa5bc033f80 0x7fa5bc036440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:43.548 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.542+0000 7fa5c9ffb700 1 -- 172.21.15.162:0/319174083 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7fa5c0044d60 con 0x7fa5cc0fb020 2024-02-13T00:32:43.549 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.542+0000 7fa5c9ffb700 10 client.14148.objecter ms_dispatch 0x7fa5cc001550 osd_map(2..2 src has 1..2) v4 2024-02-13T00:32:43.549 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.542+0000 7fa5c9ffb700 3 client.14148.objecter handle_osd_map got epochs [2,2] > 0 2024-02-13T00:32:43.549 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.542+0000 7fa5c9ffb700 3 client.14148.objecter handle_osd_map decoding full epoch 2 2024-02-13T00:32:43.549 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.542+0000 7fa5cbfff700 1 -- 172.21.15.162:0/319174083 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa5bc033f80 msgr2=0x7fa5bc036440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/4121673848 2024-02-13T00:32:43.549 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.542+0000 7fa5cbfff700 1 --2- 172.21.15.162:0/319174083 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa5bc033f80 0x7fa5bc036440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-13T00:32:43.549 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.542+0000 7fa5c9ffb700 20 client.14148.objecter dump_active .. 0 homeless 2024-02-13T00:32:43.554 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.550+0000 7fa5c9ffb700 1 -- 172.21.15.162:0/319174083 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa5c001ade0 con 0x7fa5cc0fb020 2024-02-13T00:32:43.747 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.743+0000 7fa5cbfff700 1 -- 172.21.15.162:0/319174083 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa5bc033f80 msgr2=0x7fa5bc036440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/4121673848 2024-02-13T00:32:43.747 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.743+0000 7fa5cbfff700 1 --2- 172.21.15.162:0/319174083 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa5bc033f80 0x7fa5bc036440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-13T00:32:43.756 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:43.429+0000 7f00edfc7680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T00:32:43.756 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:43.531+0000 7f00edfc7680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T00:32:43.767 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.763+0000 7fa5d2fda700 1 -- 172.21.15.162:0/319174083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa5cc102650 con 0x7fa5cc0fb020 2024-02-13T00:32:43.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.763+0000 7fa5c9ffb700 1 -- 172.21.15.162:0/319174083 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fa5c001e030 con 0x7fa5cc0fb020 2024-02-13T00:32:43.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: { 2024-02-13T00:32:43.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-13T00:32:43.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "available": true, 2024-02-13T00:32:43.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-13T00:32:43.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-13T00:32:43.768 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: } 2024-02-13T00:32:43.771 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.765+0000 7fa5d2fda700 10 librados: watch_flush enter 2024-02-13T00:32:43.771 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.765+0000 7fa5d2fda700 10 librados: watch_flush exit 2024-02-13T00:32:43.771 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.765+0000 7fa5d2fda700 20 client.14148.objecter shutdown clearing up homeless session... 2024-02-13T00:32:43.771 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.765+0000 7fa5d2fda700 10 client.14148.objecter successfully canceled tick 2024-02-13T00:32:43.771 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.766+0000 7fa5d2fda700 1 -- 172.21.15.162:0/319174083 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa5bc033f80 msgr2=0x7fa5bc036440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T00:32:43.771 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.766+0000 7fa5d2fda700 1 --2- 172.21.15.162:0/319174083 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa5bc033f80 0x7fa5bc036440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:32:43.771 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.766+0000 7fa5d2fda700 1 -- 172.21.15.162:0/319174083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 msgr2=0x7fa5cc10d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:43.772 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.766+0000 7fa5d2fda700 1 --2- 172.21.15.162:0/319174083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 0x7fa5cc10d0b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa5c0006e90 tx=0x7fa5c00090f0).stop 2024-02-13T00:32:43.772 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.766+0000 7fa5d2fda700 1 -- 172.21.15.162:0/319174083 shutdown_connections 2024-02-13T00:32:43.772 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.766+0000 7fa5d2fda700 1 --2- 172.21.15.162:0/319174083 >> v2:172.21.15.162:6800/4121673848 conn(0x7fa5bc033f80 0x7fa5bc036440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:32:43.772 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.766+0000 7fa5d2fda700 1 --2- 172.21.15.162:0/319174083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5cc0fb020 0x7fa5cc10d0b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:43.772 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.766+0000 7fa5d2fda700 1 -- 172.21.15.162:0/319174083 >> 172.21.15.162:0/319174083 conn(0x7fa5cc0f6820 msgr2=0x7fa5cc0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:43.772 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.766+0000 7fa5d2fda700 1 -- 172.21.15.162:0/319174083 shutdown_connections 2024-02-13T00:32:43.772 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.766+0000 7fa5d2fda700 1 -- 172.21.15.162:0/319174083 wait complete. 2024-02-13T00:32:43.772 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:43.766+0000 7fa5d2fda700 1 librados: shutdown 2024-02-13T00:32:44.066 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:43.752+0000 7f00edfc7680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T00:32:44.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:43.765412+0000 mon.a (mon.0) 65 : audit [DBG] from='client.? 172.21.15.162:0/319174083' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-13T00:32:44.993 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:44.695+0000 7f00edfc7680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T00:32:44.993 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:44.822+0000 7f00edfc7680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T00:32:44.993 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:44.901+0000 7f00edfc7680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T00:32:45.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:44.990+0000 7f00edfc7680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T00:32:45.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:45.188+0000 7f00edfc7680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T00:32:45.456 INFO:teuthology.orchestra.run.smithi162.stderr:Waiting for the mgr to restart... 2024-02-13T00:32:45.456 INFO:teuthology.orchestra.run.smithi162.stderr:Waiting for mgr epoch 9... 2024-02-13T00:32:45.727 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:45.471+0000 7f00edfc7680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T00:32:45.727 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:45.572+0000 7f00edfc7680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T00:32:46.015 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.010+0000 7f5bc05db700 1 Processor -- start 2024-02-13T00:32:46.016 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.010+0000 7f5bc05db700 1 -- start start 2024-02-13T00:32:46.016 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.010+0000 7f5bc05db700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 0x7f5bb80fedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:46.016 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.010+0000 7f5bc05db700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb80ff2b0 con 0x7f5bb8070c20 2024-02-13T00:32:46.016 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.010+0000 7f5bbe377700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 0x7f5bb80fedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:46.017 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.010+0000 7f5bbe377700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 0x7f5bb80fedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:57338/0 (socket says 172.21.15.162:57338) 2024-02-13T00:32:46.017 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.010+0000 7f5bbe377700 1 -- 172.21.15.162:0/1536033799 learned_addr learned my addr 172.21.15.162:0/1536033799 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:46.017 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.010+0000 7f5bbe377700 1 -- 172.21.15.162:0/1536033799 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb80ff3f0 con 0x7f5bb8070c20 2024-02-13T00:32:46.017 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.011+0000 7f5bbe377700 1 --2- 172.21.15.162:0/1536033799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 0x7f5bb80fedc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f5bb4009730 tx=0x7f5bb40090f0).ready entity=mon.0 client_cookie=7eeb6aec27c5b84c server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:46.017 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.011+0000 7f5bbd375700 1 -- 172.21.15.162:0/1536033799 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bb400b610 con 0x7f5bb8070c20 2024-02-13T00:32:46.017 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.011+0000 7f5bbd375700 1 -- 172.21.15.162:0/1536033799 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bb400bc40 con 0x7f5bb8070c20 2024-02-13T00:32:46.018 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.011+0000 7f5bbd375700 1 -- 172.21.15.162:0/1536033799 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bb4011c10 con 0x7f5bb8070c20 2024-02-13T00:32:46.019 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.011+0000 7f5bc05db700 1 -- 172.21.15.162:0/1536033799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 msgr2=0x7f5bb80fedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:46.019 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.011+0000 7f5bc05db700 1 --2- 172.21.15.162:0/1536033799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 0x7f5bb80fedc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f5bb4009730 tx=0x7f5bb40090f0).stop 2024-02-13T00:32:46.019 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.012+0000 7f5bc05db700 1 -- 172.21.15.162:0/1536033799 shutdown_connections 2024-02-13T00:32:46.019 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.012+0000 7f5bc05db700 1 --2- 172.21.15.162:0/1536033799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 0x7f5bb80fedc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:46.019 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.012+0000 7f5bc05db700 1 -- 172.21.15.162:0/1536033799 >> 172.21.15.162:0/1536033799 conn(0x7f5bb80f6820 msgr2=0x7f5bb80f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:46.019 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.012+0000 7f5bc05db700 1 -- 172.21.15.162:0/1536033799 shutdown_connections 2024-02-13T00:32:46.019 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.012+0000 7f5bc05db700 1 -- 172.21.15.162:0/1536033799 wait complete. 2024-02-13T00:32:46.020 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.012+0000 7f5bc05db700 1 librados: starting msgr at 2024-02-13T00:32:46.020 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.012+0000 7f5bc05db700 1 librados: starting objecter 2024-02-13T00:32:46.020 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.012+0000 7f5bc05db700 1 Processor -- start 2024-02-13T00:32:46.020 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.012+0000 7f5bc05db700 1 -- start start 2024-02-13T00:32:46.020 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.012+0000 7f5bc05db700 1 librados: setting wanted keys 2024-02-13T00:32:46.020 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.012+0000 7f5bc05db700 1 librados: calling monclient init 2024-02-13T00:32:46.020 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5bc05db700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 0x7f5bb810aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:46.020 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5bc05db700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb810b3c0 con 0x7f5bb8070c20 2024-02-13T00:32:46.020 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5bbe377700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 0x7f5bb810aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:46.020 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5bbe377700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 0x7f5bb810aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:57342/0 (socket says 172.21.15.162:57342) 2024-02-13T00:32:46.020 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5bbe377700 1 -- 172.21.15.162:0/85475621 learned_addr learned my addr 172.21.15.162:0/85475621 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5bbe377700 1 -- 172.21.15.162:0/85475621 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb4009370 con 0x7f5bb8070c20 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5bbe377700 1 --2- 172.21.15.162:0/85475621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 0x7f5bb810aed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5bb4009f00 tx=0x7f5bb40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5baf7fe700 10 client.?.objecter ms_handle_connect 0x7f5bb8070c20 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5baf7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5baf7fe700 1 -- 172.21.15.162:0/85475621 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bb401a430 con 0x7f5bb8070c20 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5baf7fe700 1 -- 172.21.15.162:0/85475621 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bb401aa60 con 0x7f5bb8070c20 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5bc05db700 1 -- 172.21.15.162:0/85475621 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb810b5c0 con 0x7f5bb8070c20 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5baf7fe700 1 -- 172.21.15.162:0/85475621 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bb4023a10 con 0x7f5bb8070c20 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5bc05db700 10 client.14150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5bc05db700 1 -- 172.21.15.162:0/85475621 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb810ba50 con 0x7f5bb8070c20 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.013+0000 7f5bc05db700 1 librados: init done 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.014+0000 7f5baf7fe700 1 -- 172.21.15.162:0/85475621 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 9) v1 ==== 38595+0+0 (secure 0 0 0) 0x7f5bb4023cf0 con 0x7f5bb8070c20 2024-02-13T00:32:46.021 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.014+0000 7f5baf7fe700 1 --2- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/4121673848 conn(0x7f5ba4033f70 0x7f5ba4036430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:46.022 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.014+0000 7f5baf7fe700 1 -- 172.21.15.162:0/85475621 --> v2:172.21.15.162:6800/4121673848 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ba4036ab0 con 0x7f5ba4033f70 2024-02-13T00:32:46.022 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.014+0000 7f5baf7fe700 1 -- 172.21.15.162:0/85475621 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f5bb4044870 con 0x7f5bb8070c20 2024-02-13T00:32:46.022 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.014+0000 7f5bbdb76700 1 -- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/4121673848 conn(0x7f5ba4033f70 msgr2=0x7f5ba4036430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/4121673848 2024-02-13T00:32:46.022 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.014+0000 7f5baf7fe700 10 client.14150.objecter ms_dispatch 0x7f5bb8001550 osd_map(2..2 src has 1..2) v4 2024-02-13T00:32:46.022 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.014+0000 7f5baf7fe700 3 client.14150.objecter handle_osd_map got epochs [2,2] > 0 2024-02-13T00:32:46.022 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.014+0000 7f5baf7fe700 3 client.14150.objecter handle_osd_map decoding full epoch 2 2024-02-13T00:32:46.022 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.014+0000 7f5bbdb76700 1 --2- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/4121673848 conn(0x7f5ba4033f70 0x7f5ba4036430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-13T00:32:46.022 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.014+0000 7f5baf7fe700 20 client.14150.objecter dump_active .. 0 homeless 2024-02-13T00:32:46.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.215+0000 7f5bbdb76700 1 -- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/4121673848 conn(0x7f5ba4033f70 msgr2=0x7f5ba4036430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/4121673848 2024-02-13T00:32:46.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.215+0000 7f5bbdb76700 1 --2- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/4121673848 conn(0x7f5ba4033f70 0x7f5ba4036430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-13T00:32:46.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:46.311+0000 7f00edfc7680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T00:32:46.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:46.392+0000 7f00edfc7680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T00:32:46.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:46.485+0000 7f00edfc7680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T00:32:46.619 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.616+0000 7f5bbdb76700 1 -- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/4121673848 conn(0x7f5ba4033f70 msgr2=0x7f5ba4036430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/4121673848 2024-02-13T00:32:46.619 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:46.616+0000 7f5bbdb76700 1 --2- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/4121673848 conn(0x7f5ba4033f70 0x7f5ba4036430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-13T00:32:46.935 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:46.659+0000 7f00edfc7680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T00:32:47.202 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:46.932+0000 7f00edfc7680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T00:32:47.203 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:47.063+0000 7f00edfc7680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T00:32:47.203 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:47.199+0000 7f00edfc7680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T00:32:47.420 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:47.417+0000 7f5bbdb76700 1 -- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/4121673848 conn(0x7f5ba4033f70 msgr2=0x7f5ba4036430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/4121673848 2024-02-13T00:32:47.420 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:47.417+0000 7f5bbdb76700 1 --2- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/4121673848 conn(0x7f5ba4033f70 0x7f5ba4036430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-13T00:32:47.458 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:47.371+0000 7f00edfc7680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T00:32:47.474 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:47.471+0000 7f5baf7fe700 1 -- 172.21.15.162:0/85475621 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mgrmap(e 10) v1 ==== 38454+0+0 (secure 0 0 0) 0x7f5bb401fcd0 con 0x7f5bb8070c20 2024-02-13T00:32:47.474 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:47.471+0000 7f5baf7fe700 1 -- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/4121673848 conn(0x7f5ba4033f70 msgr2=0x7f5ba4036430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T00:32:47.474 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:47.471+0000 7f5baf7fe700 1 --2- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/4121673848 conn(0x7f5ba4033f70 0x7f5ba4036430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:32:47.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:32:47.456+0000 7f00edfc7680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T00:32:47.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [80B blob data] 2024-02-13T00:32:47.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:47.469867+0000 mon.a (mon.0) 66 : cluster [INF] Active manager daemon y restarted 2024-02-13T00:32:47.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:47.470179+0000 mon.a (mon.0) 67 : cluster [INF] Activating manager daemon y 2024-02-13T00:32:47.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:47.473048+0000 mon.a (mon.0) 68 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-13T00:32:47.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:47.473152+0000 mon.a (mon.0) 69 : cluster [DBG] mgrmap e10: y(active, starting, since 0.00311466s) 2024-02-13T00:32:47.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:47.474121+0000 mon.a (mon.0) 70 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T00:32:47.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:47.474685+0000 mon.a (mon.0) 71 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T00:32:47.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:47.475454+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T00:32:47.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:47.475594+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T00:32:47.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:47.475724+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T00:32:47.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:47.489181+0000 mon.a (mon.0) 75 : cluster [INF] Manager daemon y is now available 2024-02-13T00:32:48.477 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.473+0000 7f5baf7fe700 1 -- 172.21.15.162:0/85475621 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mgrmap(e 11) v1 ==== 38489+0+0 (secure 0 0 0) 0x7f5bb4022460 con 0x7f5bb8070c20 2024-02-13T00:32:48.477 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.473+0000 7f5baf7fe700 1 --2- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5ba4037b10 0x7f5ba4039f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:48.477 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.473+0000 7f5baf7fe700 1 -- 172.21.15.162:0/85475621 --> v2:172.21.15.162:6800/1909610488 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ba4036ab0 con 0x7f5ba4037b10 2024-02-13T00:32:48.477 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.473+0000 7f5bbdb76700 1 --2- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5ba4037b10 0x7f5ba4039f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:48.477 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.474+0000 7f5bbdb76700 1 --2- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5ba4037b10 0x7f5ba4039f00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f5ba8003850 tx=0x7f5ba8006e60).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:48.478 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.474+0000 7f5baf7fe700 10 client.14150.objecter ms_handle_connect 0x7f5ba4037b10 2024-02-13T00:32:48.478 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.474+0000 7f5baf7fe700 1 -- 172.21.15.162:0/85475621 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f5ba4036ab0 con 0x7f5ba4037b10 2024-02-13T00:32:48.485 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.480+0000 7f5bc05db700 1 -- 172.21.15.162:0/85475621 --> v2:172.21.15.162:6800/1909610488 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5bb805a3b0 con 0x7f5ba4037b10 2024-02-13T00:32:48.485 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.480+0000 7f5baf7fe700 1 -- 172.21.15.162:0/85475621 <== mgr.14146 v2:172.21.15.162:6800/1909610488 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f5bb805a3b0 con 0x7f5ba4037b10 2024-02-13T00:32:48.485 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: { 2024-02-13T00:32:48.485 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-13T00:32:48.485 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: "initialized": true 2024-02-13T00:32:48.485 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: } 2024-02-13T00:32:48.485 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.480+0000 7f5bc05db700 10 librados: watch_flush enter 2024-02-13T00:32:48.485 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.480+0000 7f5bc05db700 10 librados: watch_flush exit 2024-02-13T00:32:48.485 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.480+0000 7f5bc05db700 20 client.14150.objecter shutdown clearing up homeless session... 2024-02-13T00:32:48.485 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.480+0000 7f5bc05db700 10 client.14150.objecter successfully canceled tick 2024-02-13T00:32:48.486 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.480+0000 7f5bc05db700 1 -- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5ba4037b10 msgr2=0x7f5ba4039f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:48.486 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.480+0000 7f5bc05db700 1 --2- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5ba4037b10 0x7f5ba4039f00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f5ba8003850 tx=0x7f5ba8006e60).stop 2024-02-13T00:32:48.486 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.480+0000 7f5bc05db700 1 -- 172.21.15.162:0/85475621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 msgr2=0x7f5bb810aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:48.486 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.480+0000 7f5bc05db700 1 --2- 172.21.15.162:0/85475621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 0x7f5bb810aed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5bb4009f00 tx=0x7f5bb40090f0).stop 2024-02-13T00:32:48.486 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.481+0000 7f5bc05db700 1 -- 172.21.15.162:0/85475621 shutdown_connections 2024-02-13T00:32:48.486 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.481+0000 7f5bc05db700 1 --2- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5ba4037b10 0x7f5ba4039f00 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:48.486 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.481+0000 7f5bc05db700 1 --2- 172.21.15.162:0/85475621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8070c20 0x7f5bb810aed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:48.486 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.481+0000 7f5bc05db700 1 --2- 172.21.15.162:0/85475621 >> v2:172.21.15.162:6800/4121673848 conn(0x7f5ba4033f70 0x7f5ba4036430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:32:48.486 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.481+0000 7f5bc05db700 1 -- 172.21.15.162:0/85475621 >> 172.21.15.162:0/85475621 conn(0x7f5bb80f6820 msgr2=0x7f5bb80f80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:48.486 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.481+0000 7f5bc05db700 1 -- 172.21.15.162:0/85475621 shutdown_connections 2024-02-13T00:32:48.486 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.481+0000 7f5bc05db700 1 -- 172.21.15.162:0/85475621 wait complete. 2024-02-13T00:32:48.486 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:48.481+0000 7f5bc05db700 1 librados: shutdown 2024-02-13T00:32:48.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:47.581453+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:32:48.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:47.582948+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:32:48.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:47.608370+0000 mon.a (mon.0) 78 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T00:32:48.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:47.611398+0000 mon.a (mon.0) 79 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T00:32:48.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:48.475598+0000 mon.a (mon.0) 80 : cluster [DBG] mgrmap e11: y(active, since 1.00555s) 2024-02-13T00:32:49.549 INFO:teuthology.orchestra.run.smithi162.stderr:mgr epoch 9 is available 2024-02-13T00:32:49.549 INFO:teuthology.orchestra.run.smithi162.stderr:Generating a dashboard self-signed certificate... 2024-02-13T00:32:50.304 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:48.475101+0000 mgr.y (mgr.14146) 1 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-13T00:32:50.305 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:48.481316+0000 mgr.y (mgr.14146) 2 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-13T00:32:50.305 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:48.875324+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:32:51.132 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.291+0000 7feee590e700 1 Processor -- start 2024-02-13T00:32:51.132 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.291+0000 7feee590e700 1 -- start start 2024-02-13T00:32:51.132 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.291+0000 7feee590e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 0x7feee00fb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:51.132 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.291+0000 7feee590e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee00fb930 con 0x7feee00fb020 2024-02-13T00:32:51.132 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.291+0000 7feedeffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 0x7feee00fb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:51.132 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.291+0000 7feedeffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 0x7feee00fb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:57402/0 (socket says 172.21.15.162:57402) 2024-02-13T00:32:51.132 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.291+0000 7feedeffd700 1 -- 172.21.15.162:0/626319144 learned_addr learned my addr 172.21.15.162:0/626319144 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:51.132 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.292+0000 7feedeffd700 1 -- 172.21.15.162:0/626319144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee00fba70 con 0x7feee00fb020 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.292+0000 7feedeffd700 1 --2- 172.21.15.162:0/626319144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 0x7feee00fb440 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7feed0009730 tx=0x7feed00090f0).ready entity=mon.0 client_cookie=3129ae2a8a2ccd83 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.292+0000 7feeddffb700 1 -- 172.21.15.162:0/626319144 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed000b580 con 0x7feee00fb020 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.292+0000 7feeddffb700 1 -- 172.21.15.162:0/626319144 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feed000bbb0 con 0x7feee00fb020 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.292+0000 7feeddffb700 1 -- 172.21.15.162:0/626319144 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed0011bc0 con 0x7feee00fb020 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.292+0000 7feee590e700 1 -- 172.21.15.162:0/626319144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 msgr2=0x7feee00fb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.292+0000 7feee590e700 1 --2- 172.21.15.162:0/626319144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 0x7feee00fb440 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7feed0009730 tx=0x7feed00090f0).stop 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.292+0000 7feee590e700 1 -- 172.21.15.162:0/626319144 shutdown_connections 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.292+0000 7feee590e700 1 --2- 172.21.15.162:0/626319144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 0x7feee00fb440 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.292+0000 7feee590e700 1 -- 172.21.15.162:0/626319144 >> 172.21.15.162:0/626319144 conn(0x7feee00f6820 msgr2=0x7feee00f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.293+0000 7feee590e700 1 -- 172.21.15.162:0/626319144 shutdown_connections 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.293+0000 7feee590e700 1 -- 172.21.15.162:0/626319144 wait complete. 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.293+0000 7feee590e700 1 librados: starting msgr at 2024-02-13T00:32:51.133 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.293+0000 7feee590e700 1 librados: starting objecter 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.293+0000 7feee590e700 1 Processor -- start 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.293+0000 7feee590e700 1 -- start start 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.293+0000 7feee590e700 1 librados: setting wanted keys 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.293+0000 7feee590e700 1 librados: calling monclient init 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.293+0000 7feee590e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 0x7feee010dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.293+0000 7feee590e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee0070c80 con 0x7feee00fb020 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feedeffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 0x7feee010dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feedeffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 0x7feee010dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:57416/0 (socket says 172.21.15.162:57416) 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feedeffd700 1 -- 172.21.15.162:0/3197459045 learned_addr learned my addr 172.21.15.162:0/3197459045 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feedeffd700 1 -- 172.21.15.162:0/3197459045 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0009370 con 0x7feee00fb020 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feedeffd700 1 --2- 172.21.15.162:0/3197459045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 0x7feee010dba0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7feed0006e90 tx=0x7feed00090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feec7fff700 10 client.?.objecter ms_handle_connect 0x7feee00fb020 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feec7fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feec7fff700 1 -- 172.21.15.162:0/3197459045 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed001a640 con 0x7feee00fb020 2024-02-13T00:32:51.134 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feee590e700 1 -- 172.21.15.162:0/3197459045 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee0070e80 con 0x7feee00fb020 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feec7fff700 1 -- 172.21.15.162:0/3197459045 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feed001ac70 con 0x7feee00fb020 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feec7fff700 1 -- 172.21.15.162:0/3197459045 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed0023b90 con 0x7feee00fb020 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feee590e700 10 client.14156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feee590e700 1 -- 172.21.15.162:0/3197459045 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee010e250 con 0x7feee00fb020 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.294+0000 7feee590e700 1 librados: init done 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.295+0000 7feec7fff700 1 -- 172.21.15.162:0/3197459045 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 12) v1 ==== 38595+0+0 (secure 0 0 0) 0x7feed002c3f0 con 0x7feee00fb020 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.295+0000 7feec7fff700 1 --2- 172.21.15.162:0/3197459045 >> v2:172.21.15.162:6800/1909610488 conn(0x7feec8033f70 0x7feec8036430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.295+0000 7feec7fff700 1 -- 172.21.15.162:0/3197459045 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7feed0044e50 con 0x7feee00fb020 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.295+0000 7feec7fff700 10 client.14156.objecter ms_dispatch 0x7feee0001550 osd_map(3..3 src has 1..3) v4 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.295+0000 7feec7fff700 3 client.14156.objecter handle_osd_map got epochs [3,3] > 0 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.295+0000 7feec7fff700 3 client.14156.objecter handle_osd_map decoding full epoch 3 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.295+0000 7feee590e700 1 -- 172.21.15.162:0/3197459045 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee005a3b0 con 0x7feee00fb020 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.295+0000 7feec7fff700 20 client.14156.objecter dump_active .. 0 homeless 2024-02-13T00:32:51.135 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.295+0000 7feede7fc700 1 --2- 172.21.15.162:0/3197459045 >> v2:172.21.15.162:6800/1909610488 conn(0x7feec8033f70 0x7feec8036430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.295+0000 7feec7fff700 10 client.14156.objecter ms_handle_connect 0x7feec8033f70 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.295+0000 7feede7fc700 1 --2- 172.21.15.162:0/3197459045 >> v2:172.21.15.162:6800/1909610488 conn(0x7feec8033f70 0x7feec8036430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7feed4009780 tx=0x7feed4006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.300+0000 7feec7fff700 1 -- 172.21.15.162:0/3197459045 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7feed001e070 con 0x7feee00fb020 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.472+0000 7feee590e700 1 -- 172.21.15.162:0/3197459045 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7feee00591c0 con 0x7feec8033f70 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.563+0000 7feec7fff700 1 -- 172.21.15.162:0/3197459045 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7feee00591c0 con 0x7feec8033f70 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 10 librados: watch_flush enter 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 10 librados: watch_flush exit 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 20 client.14156.objecter shutdown clearing up homeless session... 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 10 client.14156.objecter successfully canceled tick 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 1 -- 172.21.15.162:0/3197459045 >> v2:172.21.15.162:6800/1909610488 conn(0x7feec8033f70 msgr2=0x7feec8036430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 1 --2- 172.21.15.162:0/3197459045 >> v2:172.21.15.162:6800/1909610488 conn(0x7feec8033f70 0x7feec8036430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7feed4009780 tx=0x7feed4006cc0).stop 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 1 -- 172.21.15.162:0/3197459045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 msgr2=0x7feee010dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 1 --2- 172.21.15.162:0/3197459045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 0x7feee010dba0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7feed0006e90 tx=0x7feed00090f0).stop 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 1 -- 172.21.15.162:0/3197459045 shutdown_connections 2024-02-13T00:32:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 1 --2- 172.21.15.162:0/3197459045 >> v2:172.21.15.162:6800/1909610488 conn(0x7feec8033f70 0x7feec8036430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:51.137 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 1 --2- 172.21.15.162:0/3197459045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee00fb020 0x7feee010dba0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:51.137 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 1 -- 172.21.15.162:0/3197459045 >> 172.21.15.162:0/3197459045 conn(0x7feee00f6820 msgr2=0x7feee00f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:51.137 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 1 -- 172.21.15.162:0/3197459045 shutdown_connections 2024-02-13T00:32:51.137 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 1 -- 172.21.15.162:0/3197459045 wait complete. 2024-02-13T00:32:51.137 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:50.565+0000 7feee590e700 1 librados: shutdown 2024-02-13T00:32:51.137 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-13T00:32:51.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:50.143495+0000 mon.a (mon.0) 82 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-02-13T00:32:51.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:50.473670+0000 mgr.y (mgr.14146) 3 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:32:51.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:50.562403+0000 mon.a (mon.0) 83 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:32:51.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:50.564266+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:32:51.790 INFO:teuthology.orchestra.run.smithi162.stderr:Creating initial admin user... 2024-02-13T00:32:52.387 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.378+0000 7f4605bda700 1 Processor -- start 2024-02-13T00:32:52.387 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.378+0000 7f4605bda700 1 -- start start 2024-02-13T00:32:52.387 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.379+0000 7f4605bda700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 0x7f46000fb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:52.387 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.379+0000 7f4605bda700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46000fb930 con 0x7f46000fb020 2024-02-13T00:32:52.387 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.379+0000 7f45ff7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 0x7f46000fb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:52.388 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.379+0000 7f45ff7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 0x7f46000fb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:57426/0 (socket says 172.21.15.162:57426) 2024-02-13T00:32:52.388 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.379+0000 7f45ff7fe700 1 -- 172.21.15.162:0/723014070 learned_addr learned my addr 172.21.15.162:0/723014070 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:52.388 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.379+0000 7f45ff7fe700 1 -- 172.21.15.162:0/723014070 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46000fba70 con 0x7f46000fb020 2024-02-13T00:32:52.388 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.379+0000 7f45ff7fe700 1 --2- 172.21.15.162:0/723014070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 0x7f46000fb440 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f45f4009730 tx=0x7f45f40090f0).ready entity=mon.0 client_cookie=aabcd530fbb06f7d server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:52.388 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.379+0000 7f45fe7fc700 1 -- 172.21.15.162:0/723014070 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45f400b580 con 0x7f46000fb020 2024-02-13T00:32:52.388 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.379+0000 7f45fe7fc700 1 -- 172.21.15.162:0/723014070 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45f400bbb0 con 0x7f46000fb020 2024-02-13T00:32:52.388 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.379+0000 7f45fe7fc700 1 -- 172.21.15.162:0/723014070 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45f4011bc0 con 0x7f46000fb020 2024-02-13T00:32:52.388 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.380+0000 7f4605bda700 1 -- 172.21.15.162:0/723014070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 msgr2=0x7f46000fb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:52.388 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.380+0000 7f4605bda700 1 --2- 172.21.15.162:0/723014070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 0x7f46000fb440 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f45f4009730 tx=0x7f45f40090f0).stop 2024-02-13T00:32:52.388 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.380+0000 7f4605bda700 1 -- 172.21.15.162:0/723014070 shutdown_connections 2024-02-13T00:32:52.388 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.380+0000 7f4605bda700 1 --2- 172.21.15.162:0/723014070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 0x7f46000fb440 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.380+0000 7f4605bda700 1 -- 172.21.15.162:0/723014070 >> 172.21.15.162:0/723014070 conn(0x7f46000f6820 msgr2=0x7f46000f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.380+0000 7f4605bda700 1 -- 172.21.15.162:0/723014070 shutdown_connections 2024-02-13T00:32:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.380+0000 7f4605bda700 1 -- 172.21.15.162:0/723014070 wait complete. 2024-02-13T00:32:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.381+0000 7f4605bda700 1 librados: starting msgr at 2024-02-13T00:32:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.381+0000 7f4605bda700 1 librados: starting objecter 2024-02-13T00:32:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.381+0000 7f4605bda700 1 Processor -- start 2024-02-13T00:32:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.381+0000 7f4605bda700 1 -- start start 2024-02-13T00:32:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.381+0000 7f4605bda700 1 librados: setting wanted keys 2024-02-13T00:32:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.381+0000 7f4605bda700 1 librados: calling monclient init 2024-02-13T00:32:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.381+0000 7f4605bda700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 0x7f460010db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.381+0000 7f4605bda700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4600070ca0 con 0x7f46000fb020 2024-02-13T00:32:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.381+0000 7f45ff7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 0x7f460010db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:52.390 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.381+0000 7f45ff7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 0x7f460010db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:57434/0 (socket says 172.21.15.162:57434) 2024-02-13T00:32:52.390 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.381+0000 7f45ff7fe700 1 -- 172.21.15.162:0/1150762833 learned_addr learned my addr 172.21.15.162:0/1150762833 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:52.390 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.382+0000 7f45ff7fe700 1 -- 172.21.15.162:0/1150762833 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f4009370 con 0x7f46000fb020 2024-02-13T00:32:52.390 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.382+0000 7f45ff7fe700 1 --2- 172.21.15.162:0/1150762833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 0x7f460010db80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f45f4006e90 tx=0x7f45f40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:52.390 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.382+0000 7f45fcff9700 10 client.?.objecter ms_handle_connect 0x7f46000fb020 2024-02-13T00:32:52.390 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.382+0000 7f45fcff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:52.390 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.382+0000 7f45fcff9700 1 -- 172.21.15.162:0/1150762833 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45f401a640 con 0x7f46000fb020 2024-02-13T00:32:52.390 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.382+0000 7f45fcff9700 1 -- 172.21.15.162:0/1150762833 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45f401ac70 con 0x7f46000fb020 2024-02-13T00:32:52.390 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.382+0000 7f4605bda700 1 -- 172.21.15.162:0/1150762833 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4600070ea0 con 0x7f46000fb020 2024-02-13T00:32:52.390 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.382+0000 7f45fcff9700 1 -- 172.21.15.162:0/1150762833 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45f4023b90 con 0x7f46000fb020 2024-02-13T00:32:52.390 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.382+0000 7f4605bda700 10 client.14158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:52.390 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.382+0000 7f4605bda700 1 -- 172.21.15.162:0/1150762833 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f460010e230 con 0x7f46000fb020 2024-02-13T00:32:52.391 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.382+0000 7f4605bda700 1 librados: init done 2024-02-13T00:32:52.391 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.383+0000 7f45fcff9700 1 -- 172.21.15.162:0/1150762833 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 12) v1 ==== 38595+0+0 (secure 0 0 0) 0x7f45f402c3f0 con 0x7f46000fb020 2024-02-13T00:32:52.391 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.383+0000 7f45fcff9700 1 --2- 172.21.15.162:0/1150762833 >> v2:172.21.15.162:6800/1909610488 conn(0x7f45e8033fc0 0x7f45e8036480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:52.391 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.383+0000 7f45fcff9700 1 -- 172.21.15.162:0/1150762833 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f45f4045c30 con 0x7f46000fb020 2024-02-13T00:32:52.391 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.383+0000 7f45fcff9700 10 client.14158.objecter ms_dispatch 0x7f4600001550 osd_map(3..3 src has 1..3) v4 2024-02-13T00:32:52.391 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.383+0000 7f45fcff9700 3 client.14158.objecter handle_osd_map got epochs [3,3] > 0 2024-02-13T00:32:52.391 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.383+0000 7f45fcff9700 3 client.14158.objecter handle_osd_map decoding full epoch 3 2024-02-13T00:32:52.391 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.383+0000 7f45fcff9700 20 client.14158.objecter dump_active .. 0 homeless 2024-02-13T00:32:52.391 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.383+0000 7f45feffd700 1 --2- 172.21.15.162:0/1150762833 >> v2:172.21.15.162:6800/1909610488 conn(0x7f45e8033fc0 0x7f45e8036480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:52.391 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.383+0000 7f4605bda700 1 -- 172.21.15.162:0/1150762833 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460005a3b0 con 0x7f46000fb020 2024-02-13T00:32:52.391 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.383+0000 7f45fcff9700 10 client.14158.objecter ms_handle_connect 0x7f45e8033fc0 2024-02-13T00:32:52.391 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.383+0000 7f45feffd700 1 --2- 172.21.15.162:0/1150762833 >> v2:172.21.15.162:6800/1909610488 conn(0x7f45e8033fc0 0x7f45e8036480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f45f0009780 tx=0x7f45f0006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:52.392 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.388+0000 7f45fcff9700 1 -- 172.21.15.162:0/1150762833 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f45f401e070 con 0x7f46000fb020 2024-02-13T00:32:52.568 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.564+0000 7f4605bda700 1 -- 172.21.15.162:0/1150762833 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f460005a1d0 con 0x7f45e8033fc0 2024-02-13T00:32:52.798 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.794+0000 7f45fcff9700 1 -- 172.21.15.162:0/1150762833 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f460005a1d0 con 0x7f45e8033fc0 2024-02-13T00:32:52.798 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$XRLhd8gntbKIYybOn7QwOOm9DqV89N6IvTUwEOi.VjmuUlS2R/ETe", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707784372, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-13T00:32:52.800 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.796+0000 7f4605bda700 10 librados: watch_flush enter 2024-02-13T00:32:52.800 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.796+0000 7f4605bda700 10 librados: watch_flush exit 2024-02-13T00:32:52.800 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.796+0000 7f4605bda700 20 client.14158.objecter shutdown clearing up homeless session... 2024-02-13T00:32:52.800 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.796+0000 7f4605bda700 10 client.14158.objecter successfully canceled tick 2024-02-13T00:32:52.800 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.796+0000 7f4605bda700 1 -- 172.21.15.162:0/1150762833 >> v2:172.21.15.162:6800/1909610488 conn(0x7f45e8033fc0 msgr2=0x7f45e8036480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:52.801 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.796+0000 7f4605bda700 1 --2- 172.21.15.162:0/1150762833 >> v2:172.21.15.162:6800/1909610488 conn(0x7f45e8033fc0 0x7f45e8036480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f45f0009780 tx=0x7f45f0006cc0).stop 2024-02-13T00:32:52.801 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.796+0000 7f4605bda700 1 -- 172.21.15.162:0/1150762833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 msgr2=0x7f460010db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:52.801 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.796+0000 7f4605bda700 1 --2- 172.21.15.162:0/1150762833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 0x7f460010db80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f45f4006e90 tx=0x7f45f40090f0).stop 2024-02-13T00:32:52.801 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.796+0000 7f4605bda700 1 -- 172.21.15.162:0/1150762833 shutdown_connections 2024-02-13T00:32:52.801 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.796+0000 7f4605bda700 1 --2- 172.21.15.162:0/1150762833 >> v2:172.21.15.162:6800/1909610488 conn(0x7f45e8033fc0 0x7f45e8036480 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:52.801 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.796+0000 7f4605bda700 1 --2- 172.21.15.162:0/1150762833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46000fb020 0x7f460010db80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:52.801 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.796+0000 7f4605bda700 1 -- 172.21.15.162:0/1150762833 >> 172.21.15.162:0/1150762833 conn(0x7f46000f6820 msgr2=0x7f46000f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:52.801 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.797+0000 7f4605bda700 1 -- 172.21.15.162:0/1150762833 shutdown_connections 2024-02-13T00:32:52.801 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.797+0000 7f4605bda700 1 -- 172.21.15.162:0/1150762833 wait complete. 2024-02-13T00:32:52.801 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:52.797+0000 7f4605bda700 1 librados: shutdown 2024-02-13T00:32:53.430 INFO:teuthology.orchestra.run.smithi162.stderr:Fetching dashboard port number... 2024-02-13T00:32:54.054 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.038+0000 7f5294df8700 1 Processor -- start 2024-02-13T00:32:54.054 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.038+0000 7f5294df8700 1 -- start start 2024-02-13T00:32:54.054 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.038+0000 7f5294df8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 0x7f52900fed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:54.054 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.038+0000 7f5294df8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52900ff240 con 0x7f5290070c60 2024-02-13T00:32:54.054 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.038+0000 7f528e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 0x7f52900fed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:54.054 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.038+0000 7f528e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 0x7f52900fed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:57448/0 (socket says 172.21.15.162:57448) 2024-02-13T00:32:54.054 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.038+0000 7f528e59c700 1 -- 172.21.15.162:0/3386928393 learned_addr learned my addr 172.21.15.162:0/3386928393 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:54.054 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.038+0000 7f528e59c700 1 -- 172.21.15.162:0/3386928393 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52900ff380 con 0x7f5290070c60 2024-02-13T00:32:54.054 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.038+0000 7f528e59c700 1 --2- 172.21.15.162:0/3386928393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 0x7f52900fed50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f5280009e50 tx=0x7f5280009840).ready entity=mon.0 client_cookie=82ac355c6517d22c server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.039+0000 7f528d59a700 1 -- 172.21.15.162:0/3386928393 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f528000e070 con 0x7f5290070c60 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.039+0000 7f528d59a700 1 -- 172.21.15.162:0/3386928393 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5280013420 con 0x7f5290070c60 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.039+0000 7f528d59a700 1 -- 172.21.15.162:0/3386928393 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5280007400 con 0x7f5290070c60 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.039+0000 7f5294df8700 1 -- 172.21.15.162:0/3386928393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 msgr2=0x7f52900fed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.039+0000 7f5294df8700 1 --2- 172.21.15.162:0/3386928393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 0x7f52900fed50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f5280009e50 tx=0x7f5280009840).stop 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.039+0000 7f5294df8700 1 -- 172.21.15.162:0/3386928393 shutdown_connections 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.039+0000 7f5294df8700 1 --2- 172.21.15.162:0/3386928393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 0x7f52900fed50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.039+0000 7f5294df8700 1 -- 172.21.15.162:0/3386928393 >> 172.21.15.162:0/3386928393 conn(0x7f52900f67d0 msgr2=0x7f52900f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.039+0000 7f5294df8700 1 -- 172.21.15.162:0/3386928393 shutdown_connections 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.039+0000 7f5294df8700 1 -- 172.21.15.162:0/3386928393 wait complete. 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.040+0000 7f5294df8700 1 librados: starting msgr at 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.040+0000 7f5294df8700 1 librados: starting objecter 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.040+0000 7f5294df8700 1 Processor -- start 2024-02-13T00:32:54.055 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.040+0000 7f5294df8700 1 -- start start 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.040+0000 7f5294df8700 1 librados: setting wanted keys 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.040+0000 7f5294df8700 1 librados: calling monclient init 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.040+0000 7f5294df8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 0x7f529010aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.040+0000 7f5294df8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529010b3c0 con 0x7f5290070c60 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.040+0000 7f528e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 0x7f529010aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f528e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 0x7f529010aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:57462/0 (socket says 172.21.15.162:57462) 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f528e59c700 1 -- 172.21.15.162:0/544381513 learned_addr learned my addr 172.21.15.162:0/544381513 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f528e59c700 1 -- 172.21.15.162:0/544381513 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5280009ae0 con 0x7f5290070c60 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f528e59c700 1 --2- 172.21.15.162:0/544381513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 0x7f529010aed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f5280009fb0 tx=0x7f5280009840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f52877fe700 10 client.?.objecter ms_handle_connect 0x7f5290070c60 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f52877fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f52877fe700 1 -- 172.21.15.162:0/544381513 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f528000e040 con 0x7f5290070c60 2024-02-13T00:32:54.056 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f5294df8700 1 -- 172.21.15.162:0/544381513 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529010b5c0 con 0x7f5290070c60 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f52877fe700 1 -- 172.21.15.162:0/544381513 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f528000bc70 con 0x7f5290070c60 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f52877fe700 1 -- 172.21.15.162:0/544381513 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f528001c770 con 0x7f5290070c60 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f5294df8700 10 client.14160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f5294df8700 1 -- 172.21.15.162:0/544381513 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529010c1d0 con 0x7f5290070c60 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.041+0000 7f5294df8700 1 librados: init done 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.042+0000 7f52877fe700 1 -- 172.21.15.162:0/544381513 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7f5280003040 con 0x7f5290070c60 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.042+0000 7f52877fe700 1 --2- 172.21.15.162:0/544381513 >> v2:172.21.15.162:6800/1909610488 conn(0x7f527c034010 0x7f527c0364d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.042+0000 7f52877fe700 1 -- 172.21.15.162:0/544381513 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f5280044800 con 0x7f5290070c60 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.042+0000 7f52877fe700 10 client.14160.objecter ms_dispatch 0x7f5290001550 osd_map(3..3 src has 1..3) v4 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.042+0000 7f52877fe700 3 client.14160.objecter handle_osd_map got epochs [3,3] > 0 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.042+0000 7f52877fe700 3 client.14160.objecter handle_osd_map decoding full epoch 3 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.042+0000 7f5294df8700 1 -- 172.21.15.162:0/544381513 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529010ba50 con 0x7f5290070c60 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.042+0000 7f52877fe700 20 client.14160.objecter dump_active .. 0 homeless 2024-02-13T00:32:54.057 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.042+0000 7f528dd9b700 1 --2- 172.21.15.162:0/544381513 >> v2:172.21.15.162:6800/1909610488 conn(0x7f527c034010 0x7f527c0364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:54.058 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.042+0000 7f52877fe700 10 client.14160.objecter ms_handle_connect 0x7f527c034010 2024-02-13T00:32:54.058 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.042+0000 7f528dd9b700 1 --2- 172.21.15.162:0/544381513 >> v2:172.21.15.162:6800/1909610488 conn(0x7f527c034010 0x7f527c0364d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f5278009780 tx=0x7f5278006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:54.058 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.048+0000 7f52877fe700 1 -- 172.21.15.162:0/544381513 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f529010ba50 con 0x7f5290070c60 2024-02-13T00:32:54.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:52.566231+0000 mgr.y (mgr.14146) 4 : audit [DBG] from='client.14158 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:32:54.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:52.795275+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:32:54.214 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.211+0000 7f5294df8700 1 -- 172.21.15.162:0/544381513 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f529010ba50 con 0x7f5290070c60 2024-02-13T00:32:54.214 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.211+0000 7f52877fe700 1 -- 172.21.15.162:0/544381513 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f529010ba50 con 0x7f5290070c60 2024-02-13T00:32:54.214 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 8443 2024-02-13T00:32:54.217 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 10 librados: watch_flush enter 2024-02-13T00:32:54.217 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 10 librados: watch_flush exit 2024-02-13T00:32:54.217 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 20 client.14160.objecter shutdown clearing up homeless session... 2024-02-13T00:32:54.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 10 client.14160.objecter successfully canceled tick 2024-02-13T00:32:54.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 1 -- 172.21.15.162:0/544381513 >> v2:172.21.15.162:6800/1909610488 conn(0x7f527c034010 msgr2=0x7f527c0364d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:54.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 1 --2- 172.21.15.162:0/544381513 >> v2:172.21.15.162:6800/1909610488 conn(0x7f527c034010 0x7f527c0364d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f5278009780 tx=0x7f5278006cc0).stop 2024-02-13T00:32:54.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 1 -- 172.21.15.162:0/544381513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 msgr2=0x7f529010aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:54.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 1 --2- 172.21.15.162:0/544381513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 0x7f529010aed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f5280009fb0 tx=0x7f5280009840).stop 2024-02-13T00:32:54.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 1 -- 172.21.15.162:0/544381513 shutdown_connections 2024-02-13T00:32:54.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 1 --2- 172.21.15.162:0/544381513 >> v2:172.21.15.162:6800/1909610488 conn(0x7f527c034010 0x7f527c0364d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:54.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 1 --2- 172.21.15.162:0/544381513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5290070c60 0x7f529010aed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:54.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 1 -- 172.21.15.162:0/544381513 >> 172.21.15.162:0/544381513 conn(0x7f52900f67d0 msgr2=0x7f52900f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:54.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 1 -- 172.21.15.162:0/544381513 shutdown_connections 2024-02-13T00:32:54.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 1 -- 172.21.15.162:0/544381513 wait complete. 2024-02-13T00:32:54.218 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:54.213+0000 7f5294df8700 1 librados: shutdown 2024-02-13T00:32:54.916 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:32:53.796588+0000 mon.a (mon.0) 86 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-02-13T00:32:54.916 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:54.212376+0000 mon.a (mon.0) 87 : audit [DBG] from='client.? 172.21.15.162:0/544381513' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-13T00:32:54.922 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: disabled 2024-02-13T00:32:54.930 INFO:teuthology.orchestra.run.smithi162.stderr:systemctl: inactive 2024-02-13T00:32:54.930 INFO:teuthology.orchestra.run.smithi162.stderr:firewalld.service is not enabled 2024-02-13T00:32:54.930 INFO:teuthology.orchestra.run.smithi162.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-13T00:32:54.933 INFO:teuthology.orchestra.run.smithi162.stderr:Ceph Dashboard is now available at: 2024-02-13T00:32:54.933 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-02-13T00:32:54.933 INFO:teuthology.orchestra.run.smithi162.stderr: URL: https://smithi162.front.sepia.ceph.com:8443/ 2024-02-13T00:32:54.934 INFO:teuthology.orchestra.run.smithi162.stderr: User: admin 2024-02-13T00:32:54.934 INFO:teuthology.orchestra.run.smithi162.stderr: Password: 1it4t1y7e5 2024-02-13T00:32:54.934 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-02-13T00:32:54.934 INFO:teuthology.orchestra.run.smithi162.stderr:Enabling autotune for osd_memory_target 2024-02-13T00:32:55.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.580+0000 7f7ae8258700 1 Processor -- start 2024-02-13T00:32:55.585 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.581+0000 7f7ae8258700 1 -- start start 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.581+0000 7f7ae8258700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 0x7f7ae00fb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.581+0000 7f7ae8258700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae00fb900 con 0x7f7ae00faff0 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.581+0000 7f7ae5ff4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 0x7f7ae00fb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.581+0000 7f7ae5ff4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 0x7f7ae00fb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:51200/0 (socket says 172.21.15.162:51200) 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.581+0000 7f7ae5ff4700 1 -- 172.21.15.162:0/2942457490 learned_addr learned my addr 172.21.15.162:0/2942457490 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.581+0000 7f7ae5ff4700 1 -- 172.21.15.162:0/2942457490 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae00fba40 con 0x7f7ae00faff0 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.582+0000 7f7ae5ff4700 1 --2- 172.21.15.162:0/2942457490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 0x7f7ae00fb410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f7ad0009730 tx=0x7f7ad00090f0).ready entity=mon.0 client_cookie=ad40b1adafa1e38e server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.582+0000 7f7ae4ff2700 1 -- 172.21.15.162:0/2942457490 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ad000b580 con 0x7f7ae00faff0 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.582+0000 7f7ae4ff2700 1 -- 172.21.15.162:0/2942457490 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ad000bbb0 con 0x7f7ae00faff0 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.582+0000 7f7ae4ff2700 1 -- 172.21.15.162:0/2942457490 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ad0011bc0 con 0x7f7ae00faff0 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.582+0000 7f7ae8258700 1 -- 172.21.15.162:0/2942457490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 msgr2=0x7f7ae00fb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.582+0000 7f7ae8258700 1 --2- 172.21.15.162:0/2942457490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 0x7f7ae00fb410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f7ad0009730 tx=0x7f7ad00090f0).stop 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.582+0000 7f7ae8258700 1 -- 172.21.15.162:0/2942457490 shutdown_connections 2024-02-13T00:32:55.586 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.582+0000 7f7ae8258700 1 --2- 172.21.15.162:0/2942457490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 0x7f7ae00fb410 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:55.589 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.582+0000 7f7ae8258700 1 -- 172.21.15.162:0/2942457490 >> 172.21.15.162:0/2942457490 conn(0x7f7ae00f67d0 msgr2=0x7f7ae00f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:55.589 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.582+0000 7f7ae8258700 1 -- 172.21.15.162:0/2942457490 shutdown_connections 2024-02-13T00:32:55.589 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.583+0000 7f7ae8258700 1 -- 172.21.15.162:0/2942457490 wait complete. 2024-02-13T00:32:55.589 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.583+0000 7f7ae8258700 1 librados: starting msgr at 2024-02-13T00:32:55.589 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.583+0000 7f7ae8258700 1 librados: starting objecter 2024-02-13T00:32:55.589 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.583+0000 7f7ae8258700 1 Processor -- start 2024-02-13T00:32:55.589 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.583+0000 7f7ae8258700 1 -- start start 2024-02-13T00:32:55.589 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.583+0000 7f7ae8258700 1 librados: setting wanted keys 2024-02-13T00:32:55.589 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.583+0000 7f7ae8258700 1 librados: calling monclient init 2024-02-13T00:32:55.589 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ae8258700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 0x7f7ae010db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:55.589 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ae8258700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae0070ca0 con 0x7f7ae00faff0 2024-02-13T00:32:55.590 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ae5ff4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 0x7f7ae010db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:55.590 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ae5ff4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 0x7f7ae010db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:51210/0 (socket says 172.21.15.162:51210) 2024-02-13T00:32:55.590 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ae5ff4700 1 -- 172.21.15.162:0/2621187497 learned_addr learned my addr 172.21.15.162:0/2621187497 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:55.590 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ae5ff4700 1 -- 172.21.15.162:0/2621187497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad0009370 con 0x7f7ae00faff0 2024-02-13T00:32:55.590 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ae5ff4700 1 --2- 172.21.15.162:0/2621187497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 0x7f7ae010db60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f7ad0006e90 tx=0x7f7ad00090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:55.590 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ad6ffd700 10 client.?.objecter ms_handle_connect 0x7f7ae00faff0 2024-02-13T00:32:55.590 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ad6ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:55.590 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ad6ffd700 1 -- 172.21.15.162:0/2621187497 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ad001a640 con 0x7f7ae00faff0 2024-02-13T00:32:55.590 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ae8258700 1 -- 172.21.15.162:0/2621187497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae0070ea0 con 0x7f7ae00faff0 2024-02-13T00:32:55.590 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ad6ffd700 1 -- 172.21.15.162:0/2621187497 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ad001ac70 con 0x7f7ae00faff0 2024-02-13T00:32:55.590 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ad6ffd700 1 -- 172.21.15.162:0/2621187497 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ad0023b90 con 0x7f7ae00faff0 2024-02-13T00:32:55.590 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ae8258700 10 client.14162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:55.591 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ae8258700 1 -- 172.21.15.162:0/2621187497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae010e210 con 0x7f7ae00faff0 2024-02-13T00:32:55.591 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.584+0000 7f7ae8258700 1 librados: init done 2024-02-13T00:32:55.591 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.585+0000 7f7ad6ffd700 1 -- 172.21.15.162:0/2621187497 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7f7ad002c3f0 con 0x7f7ae00faff0 2024-02-13T00:32:55.591 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.585+0000 7f7ad6ffd700 1 --2- 172.21.15.162:0/2621187497 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7acc034060 0x7f7acc036520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:55.591 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.585+0000 7f7ad6ffd700 1 -- 172.21.15.162:0/2621187497 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f7ad0045230 con 0x7f7ae00faff0 2024-02-13T00:32:55.591 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.585+0000 7f7ad6ffd700 10 client.14162.objecter ms_dispatch 0x7f7ae0001550 osd_map(3..3 src has 1..3) v4 2024-02-13T00:32:55.591 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.585+0000 7f7ad6ffd700 3 client.14162.objecter handle_osd_map got epochs [3,3] > 0 2024-02-13T00:32:55.591 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.585+0000 7f7ad6ffd700 3 client.14162.objecter handle_osd_map decoding full epoch 3 2024-02-13T00:32:55.591 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.585+0000 7f7ad6ffd700 20 client.14162.objecter dump_active .. 0 homeless 2024-02-13T00:32:55.591 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.585+0000 7f7ae57f3700 1 --2- 172.21.15.162:0/2621187497 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7acc034060 0x7f7acc036520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:55.591 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.586+0000 7f7ad6ffd700 10 client.14162.objecter ms_handle_connect 0x7f7acc034060 2024-02-13T00:32:55.591 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.586+0000 7f7ae57f3700 1 --2- 172.21.15.162:0/2621187497 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7acc034060 0x7f7acc036520 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7adc009780 tx=0x7f7adc006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:55.592 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.587+0000 7f7ae8258700 1 -- 172.21.15.162:0/2621187497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae005a3b0 con 0x7f7ae00faff0 2024-02-13T00:32:55.594 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.592+0000 7f7ad6ffd700 1 -- 172.21.15.162:0/2621187497 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7ad002bbb0 con 0x7f7ae00faff0 2024-02-13T00:32:55.758 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.754+0000 7f7ae8258700 1 -- 172.21.15.162:0/2621187497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f7ae0046d80 con 0x7f7ae00faff0 2024-02-13T00:32:55.758 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.755+0000 7f7ad6ffd700 1 -- 172.21.15.162:0/2621187497 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f7ad002b7c0 con 0x7f7ae00faff0 2024-02-13T00:32:55.761 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.756+0000 7f7ae8258700 10 librados: watch_flush enter 2024-02-13T00:32:55.761 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.756+0000 7f7ae8258700 10 librados: watch_flush exit 2024-02-13T00:32:55.761 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.756+0000 7f7ae8258700 20 client.14162.objecter shutdown clearing up homeless session... 2024-02-13T00:32:55.761 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.756+0000 7f7ae8258700 10 client.14162.objecter successfully canceled tick 2024-02-13T00:32:55.761 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.757+0000 7f7ae8258700 1 -- 172.21.15.162:0/2621187497 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7acc034060 msgr2=0x7f7acc036520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:55.761 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.757+0000 7f7ae8258700 1 --2- 172.21.15.162:0/2621187497 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7acc034060 0x7f7acc036520 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7adc009780 tx=0x7f7adc006cc0).stop 2024-02-13T00:32:55.761 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.757+0000 7f7ae8258700 1 -- 172.21.15.162:0/2621187497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 msgr2=0x7f7ae010db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:55.761 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.757+0000 7f7ae8258700 1 --2- 172.21.15.162:0/2621187497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 0x7f7ae010db60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f7ad0006e90 tx=0x7f7ad00090f0).stop 2024-02-13T00:32:55.761 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.757+0000 7f7ae8258700 1 -- 172.21.15.162:0/2621187497 shutdown_connections 2024-02-13T00:32:55.761 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.757+0000 7f7ae8258700 1 --2- 172.21.15.162:0/2621187497 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7acc034060 0x7f7acc036520 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:55.761 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.757+0000 7f7ae8258700 1 --2- 172.21.15.162:0/2621187497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ae00faff0 0x7f7ae010db60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:55.761 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.757+0000 7f7ae8258700 1 -- 172.21.15.162:0/2621187497 >> 172.21.15.162:0/2621187497 conn(0x7f7ae00f67d0 msgr2=0x7f7ae00f80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:55.762 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.757+0000 7f7ae8258700 1 -- 172.21.15.162:0/2621187497 shutdown_connections 2024-02-13T00:32:55.762 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.757+0000 7f7ae8258700 1 -- 172.21.15.162:0/2621187497 wait complete. 2024-02-13T00:32:55.762 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:55.757+0000 7f7ae8258700 1 librados: shutdown 2024-02-13T00:32:57.119 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.087+0000 7fafa56c0700 1 Processor -- start 2024-02-13T00:32:57.119 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.087+0000 7fafa56c0700 1 -- start start 2024-02-13T00:32:57.120 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.087+0000 7fafa56c0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 0x7fafa00fed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:57.120 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.087+0000 7fafa56c0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa00ff280 con 0x7fafa0070c20 2024-02-13T00:32:57.120 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.087+0000 7faf9effd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 0x7fafa00fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:57.120 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.087+0000 7faf9effd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 0x7fafa00fed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:51216/0 (socket says 172.21.15.162:51216) 2024-02-13T00:32:57.120 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.087+0000 7faf9effd700 1 -- 172.21.15.162:0/381305350 learned_addr learned my addr 172.21.15.162:0/381305350 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:57.120 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.087+0000 7faf9effd700 1 -- 172.21.15.162:0/381305350 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa00ff3c0 con 0x7fafa0070c20 2024-02-13T00:32:57.120 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.088+0000 7faf9effd700 1 --2- 172.21.15.162:0/381305350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 0x7fafa00fed90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7faf90009730 tx=0x7faf900090f0).ready entity=mon.0 client_cookie=7861b9c44e510407 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:57.120 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.088+0000 7faf9e7fc700 1 -- 172.21.15.162:0/381305350 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf9000b610 con 0x7fafa0070c20 2024-02-13T00:32:57.120 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.088+0000 7faf9e7fc700 1 -- 172.21.15.162:0/381305350 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf9000bc40 con 0x7fafa0070c20 2024-02-13T00:32:57.120 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.088+0000 7faf9e7fc700 1 -- 172.21.15.162:0/381305350 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf90011c10 con 0x7fafa0070c20 2024-02-13T00:32:57.120 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.088+0000 7fafa56c0700 1 -- 172.21.15.162:0/381305350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 msgr2=0x7fafa00fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.088+0000 7fafa56c0700 1 --2- 172.21.15.162:0/381305350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 0x7fafa00fed90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7faf90009730 tx=0x7faf900090f0).stop 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.088+0000 7fafa56c0700 1 -- 172.21.15.162:0/381305350 shutdown_connections 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.088+0000 7fafa56c0700 1 --2- 172.21.15.162:0/381305350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 0x7fafa00fed90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.088+0000 7fafa56c0700 1 -- 172.21.15.162:0/381305350 >> 172.21.15.162:0/381305350 conn(0x7fafa00f67d0 msgr2=0x7fafa00f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.088+0000 7fafa56c0700 1 -- 172.21.15.162:0/381305350 shutdown_connections 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.088+0000 7fafa56c0700 1 -- 172.21.15.162:0/381305350 wait complete. 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7fafa56c0700 1 librados: starting msgr at 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7fafa56c0700 1 librados: starting objecter 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7fafa56c0700 1 Processor -- start 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7fafa56c0700 1 -- start start 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7fafa56c0700 1 librados: setting wanted keys 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7fafa56c0700 1 librados: calling monclient init 2024-02-13T00:32:57.121 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7fafa56c0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 0x7fafa010ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7fafa56c0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa010b370 con 0x7fafa0070c20 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7faf9effd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 0x7fafa010ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7faf9effd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 0x7fafa010ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:51232/0 (socket says 172.21.15.162:51232) 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7faf9effd700 1 -- 172.21.15.162:0/3333329835 learned_addr learned my addr 172.21.15.162:0/3333329835 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7faf9effd700 1 -- 172.21.15.162:0/3333329835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf90009370 con 0x7fafa0070c20 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7faf9effd700 1 --2- 172.21.15.162:0/3333329835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 0x7fafa010ae80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7faf90006eb0 tx=0x7faf900090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7faf9cff9700 10 client.?.objecter ms_handle_connect 0x7fafa0070c20 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.089+0000 7faf9cff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7faf9cff9700 1 -- 172.21.15.162:0/3333329835 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf90011bd0 con 0x7fafa0070c20 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7faf9cff9700 1 -- 172.21.15.162:0/3333329835 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf9001ab10 con 0x7fafa0070c20 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7fafa56c0700 1 -- 172.21.15.162:0/3333329835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa010b570 con 0x7fafa0070c20 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7faf9cff9700 1 -- 172.21.15.162:0/3333329835 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf90023a80 con 0x7fafa0070c20 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7fafa56c0700 10 client.14164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7fafa56c0700 1 -- 172.21.15.162:0/3333329835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa010ba00 con 0x7fafa0070c20 2024-02-13T00:32:57.122 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7fafa56c0700 1 librados: init done 2024-02-13T00:32:57.123 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7faf9cff9700 1 -- 172.21.15.162:0/3333329835 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7faf90023cf0 con 0x7fafa0070c20 2024-02-13T00:32:57.123 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7faf9cff9700 1 --2- 172.21.15.162:0/3333329835 >> v2:172.21.15.162:6800/1909610488 conn(0x7faf8c034010 0x7faf8c0364d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:32:57.123 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7faf9cff9700 1 -- 172.21.15.162:0/3333329835 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7faf90044f90 con 0x7fafa0070c20 2024-02-13T00:32:57.123 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7faf9cff9700 10 client.14164.objecter ms_dispatch 0x7fafa0001550 osd_map(3..3 src has 1..3) v4 2024-02-13T00:32:57.123 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7faf9cff9700 3 client.14164.objecter handle_osd_map got epochs [3,3] > 0 2024-02-13T00:32:57.123 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7faf9cff9700 3 client.14164.objecter handle_osd_map decoding full epoch 3 2024-02-13T00:32:57.123 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.091+0000 7faf9cff9700 20 client.14164.objecter dump_active .. 0 homeless 2024-02-13T00:32:57.123 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.090+0000 7fafa56c0700 1 -- 172.21.15.162:0/3333329835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa0046d80 con 0x7fafa0070c20 2024-02-13T00:32:57.123 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.091+0000 7faf97fff700 1 --2- 172.21.15.162:0/3333329835 >> v2:172.21.15.162:6800/1909610488 conn(0x7faf8c034010 0x7faf8c0364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:32:57.123 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.091+0000 7faf9cff9700 10 client.14164.objecter ms_handle_connect 0x7faf8c034010 2024-02-13T00:32:57.123 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.091+0000 7faf97fff700 1 --2- 172.21.15.162:0/3333329835 >> v2:172.21.15.162:6800/1909610488 conn(0x7faf8c034010 0x7faf8c0364d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7faf88009780 tx=0x7faf88006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:32:57.123 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.095+0000 7faf9cff9700 1 -- 172.21.15.162:0/3333329835 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7faf9002b9a0 con 0x7fafa0070c20 2024-02-13T00:32:57.353 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.350+0000 7fafa56c0700 1 -- 172.21.15.162:0/3333329835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fafa0100220 con 0x7fafa0070c20 2024-02-13T00:32:57.354 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.352+0000 7faf9cff9700 1 -- 172.21.15.162:0/3333329835 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v24)=0 set mgr/dashboard/cluster/status v24) v1 ==== 153+0+0 (secure 0 0 0) 0x7faf9002b350 con 0x7fafa0070c20 2024-02-13T00:32:57.358 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 10 librados: watch_flush enter 2024-02-13T00:32:57.358 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 10 librados: watch_flush exit 2024-02-13T00:32:57.358 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 20 client.14164.objecter shutdown clearing up homeless session... 2024-02-13T00:32:57.358 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 10 client.14164.objecter successfully canceled tick 2024-02-13T00:32:57.358 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 1 -- 172.21.15.162:0/3333329835 >> v2:172.21.15.162:6800/1909610488 conn(0x7faf8c034010 msgr2=0x7faf8c0364d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:57.358 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 1 --2- 172.21.15.162:0/3333329835 >> v2:172.21.15.162:6800/1909610488 conn(0x7faf8c034010 0x7faf8c0364d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7faf88009780 tx=0x7faf88006cc0).stop 2024-02-13T00:32:57.359 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 1 -- 172.21.15.162:0/3333329835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 msgr2=0x7fafa010ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:32:57.359 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 1 --2- 172.21.15.162:0/3333329835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 0x7fafa010ae80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7faf90006eb0 tx=0x7faf900090f0).stop 2024-02-13T00:32:57.359 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 1 -- 172.21.15.162:0/3333329835 shutdown_connections 2024-02-13T00:32:57.359 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 1 --2- 172.21.15.162:0/3333329835 >> v2:172.21.15.162:6800/1909610488 conn(0x7faf8c034010 0x7faf8c0364d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:57.359 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 1 --2- 172.21.15.162:0/3333329835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa0070c20 0x7fafa010ae80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:32:57.359 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 1 -- 172.21.15.162:0/3333329835 >> 172.21.15.162:0/3333329835 conn(0x7fafa00f67d0 msgr2=0x7fafa00f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:32:57.359 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 1 -- 172.21.15.162:0/3333329835 shutdown_connections 2024-02-13T00:32:57.359 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 1 -- 172.21.15.162:0/3333329835 wait complete. 2024-02-13T00:32:57.359 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: 2024-02-13T00:32:57.354+0000 7fafa56c0700 1 librados: shutdown 2024-02-13T00:32:57.359 INFO:teuthology.orchestra.run.smithi162.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-13T00:32:58.080 INFO:teuthology.orchestra.run.smithi162.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-13T00:32:58.080 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-02-13T00:32:58.080 INFO:teuthology.orchestra.run.smithi162.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr:Or, if you are only running a single cluster on this host: 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr: ceph telemetry on 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr:For more information see: 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-02-13T00:32:58.081 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-02-13T00:32:58.082 INFO:teuthology.orchestra.run.smithi162.stderr:Bootstrap complete. 2024-02-13T00:32:58.114 INFO:tasks.cephadm:Fetching config... 2024-02-13T00:32:58.115 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:32:58.115 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-13T00:32:58.172 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-13T00:32:58.173 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:32:58.173 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-13T00:32:58.230 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-13T00:32:58.230 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:32:58.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-13T00:32:58.297 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-13T00:32:58.297 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:32:58.297 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-13T00:32:58.356 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-13T00:32:58.357 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTWxtRczCpQs9KnbDWy9LKnJJ833ey5M77SdBAE66n3+d7Xu19kaT2BNWfMBd40BPDBJ45NH+dGh+4tKDBicqT0JD1S+j5QSa1J6Txnn7CAx2rwSUJoZiGLpKAQ+A8CaVF/2/ZoTNbhHsQJHWos13q7yGWvEO5fqYOc6+DcjVoJkoAqproDKtnpOcoYSEWsm51nMAYZMd8OqMk+IsP2kzxPBwXyV6Q+I1lU6SjTtWwyjvImSUPfbzwAtR9ALkP+5hm6C76VhMRdFd6ztBEK9Xq2eTWxkUY9Z6vhxyQsTQSjz1kBmfQypLzuYGr8vPHIHMy9k/dwbLFt6u3kGN+cNrfWVrN9VZgkjZmO9itStZCBDbecHElIrEQ25ZLgP/ct/0h4FAdy7nH7jYehxiNpCZNJ76NN3FoPRVBKTjX8IEtfalirMIeLi9wiaZ1W/6qIpg64J2mfIBi6Z3pVeaG5KAUlDKIo1V11bsiLsM/IkSnWDilUaGEVGJwB/aaWuz8vRU= ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-13T00:32:58.434 INFO:teuthology.orchestra.run.smithi162.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTWxtRczCpQs9KnbDWy9LKnJJ833ey5M77SdBAE66n3+d7Xu19kaT2BNWfMBd40BPDBJ45NH+dGh+4tKDBicqT0JD1S+j5QSa1J6Txnn7CAx2rwSUJoZiGLpKAQ+A8CaVF/2/ZoTNbhHsQJHWos13q7yGWvEO5fqYOc6+DcjVoJkoAqproDKtnpOcoYSEWsm51nMAYZMd8OqMk+IsP2kzxPBwXyV6Q+I1lU6SjTtWwyjvImSUPfbzwAtR9ALkP+5hm6C76VhMRdFd6ztBEK9Xq2eTWxkUY9Z6vhxyQsTQSjz1kBmfQypLzuYGr8vPHIHMy9k/dwbLFt6u3kGN+cNrfWVrN9VZgkjZmO9itStZCBDbecHElIrEQ25ZLgP/ct/0h4FAdy7nH7jYehxiNpCZNJ76NN3FoPRVBKTjX8IEtfalirMIeLi9wiaZ1W/6qIpg64J2mfIBi6Z3pVeaG5KAUlDKIo1V11bsiLsM/IkSnWDilUaGEVGJwB/aaWuz8vRU= ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:32:58.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTWxtRczCpQs9KnbDWy9LKnJJ833ey5M77SdBAE66n3+d7Xu19kaT2BNWfMBd40BPDBJ45NH+dGh+4tKDBicqT0JD1S+j5QSa1J6Txnn7CAx2rwSUJoZiGLpKAQ+A8CaVF/2/ZoTNbhHsQJHWos13q7yGWvEO5fqYOc6+DcjVoJkoAqproDKtnpOcoYSEWsm51nMAYZMd8OqMk+IsP2kzxPBwXyV6Q+I1lU6SjTtWwyjvImSUPfbzwAtR9ALkP+5hm6C76VhMRdFd6ztBEK9Xq2eTWxkUY9Z6vhxyQsTQSjz1kBmfQypLzuYGr8vPHIHMy9k/dwbLFt6u3kGN+cNrfWVrN9VZgkjZmO9itStZCBDbecHElIrEQ25ZLgP/ct/0h4FAdy7nH7jYehxiNpCZNJ76NN3FoPRVBKTjX8IEtfalirMIeLi9wiaZ1W/6qIpg64J2mfIBi6Z3pVeaG5KAUlDKIo1V11bsiLsM/IkSnWDilUaGEVGJwB/aaWuz8vRU= ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-13T00:32:58.491 INFO:teuthology.orchestra.run.smithi181.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTWxtRczCpQs9KnbDWy9LKnJJ833ey5M77SdBAE66n3+d7Xu19kaT2BNWfMBd40BPDBJ45NH+dGh+4tKDBicqT0JD1S+j5QSa1J6Txnn7CAx2rwSUJoZiGLpKAQ+A8CaVF/2/ZoTNbhHsQJHWos13q7yGWvEO5fqYOc6+DcjVoJkoAqproDKtnpOcoYSEWsm51nMAYZMd8OqMk+IsP2kzxPBwXyV6Q+I1lU6SjTtWwyjvImSUPfbzwAtR9ALkP+5hm6C76VhMRdFd6ztBEK9Xq2eTWxkUY9Z6vhxyQsTQSjz1kBmfQypLzuYGr8vPHIHMy9k/dwbLFt6u3kGN+cNrfWVrN9VZgkjZmO9itStZCBDbecHElIrEQ25ZLgP/ct/0h4FAdy7nH7jYehxiNpCZNJ76NN3FoPRVBKTjX8IEtfalirMIeLi9wiaZ1W/6qIpg64J2mfIBi6Z3pVeaG5KAUlDKIo1V11bsiLsM/IkSnWDilUaGEVGJwB/aaWuz8vRU= ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:32:58.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-13T00:32:58.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:32:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:32:57.352991+0000 mon.a (mon.0) 88 : audit [INF] from='client.? 172.21.15.162:0/3333329835' entity='client.admin' 2024-02-13T00:33:01.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.579+0000 7f5caba8a700 1 -- 172.21.15.162:0/1600736742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ca4061290 msgr2=0x7f5ca40fa230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:01.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.579+0000 7f5caba8a700 1 --2- 172.21.15.162:0/1600736742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ca4061290 0x7f5ca40fa230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f5ca0004770 tx=0x7f5ca0009cb0).stop 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.579+0000 7f5caba8a700 1 -- 172.21.15.162:0/1600736742 shutdown_connections 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.579+0000 7f5caba8a700 1 --2- 172.21.15.162:0/1600736742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ca4061290 0x7f5ca40fa230 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.579+0000 7f5caba8a700 1 -- 172.21.15.162:0/1600736742 >> 172.21.15.162:0/1600736742 conn(0x7f5ca40f1cb0 msgr2=0x7f5ca40f40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.579+0000 7f5caba8a700 1 -- 172.21.15.162:0/1600736742 shutdown_connections 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.579+0000 7f5caba8a700 1 -- 172.21.15.162:0/1600736742 wait complete. 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.579+0000 7f5caba8a700 1 librados: starting msgr at 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.579+0000 7f5caba8a700 1 librados: starting objecter 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.579+0000 7f5caba8a700 1 Processor -- start 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5caba8a700 1 -- start start 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5caba8a700 1 librados: setting wanted keys 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5caba8a700 1 librados: calling monclient init 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5caba8a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ca4061290 0x7f5ca4106f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:01.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5caba8a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca4107450 con 0x7f5ca4061290 2024-02-13T00:33:01.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5ca9826700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ca4061290 0x7f5ca4106f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:01.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5ca9826700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ca4061290 0x7f5ca4106f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:51240/0 (socket says 172.21.15.162:51240) 2024-02-13T00:33:01.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5ca9826700 1 -- 172.21.15.162:0/2431562572 learned_addr learned my addr 172.21.15.162:0/2431562572 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:33:01.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5ca9826700 1 -- 172.21.15.162:0/2431562572 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca0005040 con 0x7f5ca4061290 2024-02-13T00:33:01.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5ca9826700 1 --2- 172.21.15.162:0/2431562572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ca4061290 0x7f5ca4106f60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5ca0013560 tx=0x7f5ca0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:01.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5c9af78700 10 client.?.objecter ms_handle_connect 0x7f5ca4061290 2024-02-13T00:33:01.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5c9af78700 10 client.?.objecter resend_mon_ops 2024-02-13T00:33:01.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5c9af78700 1 -- 172.21.15.162:0/2431562572 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ca000e030 con 0x7f5ca4061290 2024-02-13T00:33:01.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5c9af78700 1 -- 172.21.15.162:0/2431562572 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ca0013630 con 0x7f5ca4061290 2024-02-13T00:33:01.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5c9af78700 1 -- 172.21.15.162:0/2431562572 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ca0018070 con 0x7f5ca4061290 2024-02-13T00:33:01.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5caba8a700 1 -- 172.21.15.162:0/2431562572 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca4107650 con 0x7f5ca4061290 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5caba8a700 10 client.14166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5caba8a700 1 -- 172.21.15.162:0/2431562572 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca4107ae0 con 0x7f5ca4061290 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.580+0000 7f5caba8a700 1 librados: init done 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.581+0000 7f5caba8a700 1 -- 172.21.15.162:0/2431562572 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca405a3b0 con 0x7f5ca4061290 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.581+0000 7f5c9af78700 1 -- 172.21.15.162:0/2431562572 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7f5ca001c440 con 0x7f5ca4061290 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.582+0000 7f5c9af78700 1 --2- 172.21.15.162:0/2431562572 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5c90033fb0 0x7f5c90036470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.582+0000 7f5c9af78700 1 -- 172.21.15.162:0/2431562572 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f5ca0043de0 con 0x7f5ca4061290 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.582+0000 7f5c9af78700 10 client.14166.objecter ms_dispatch 0x7f5ca4001550 osd_map(3..3 src has 1..3) v4 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.582+0000 7f5c9af78700 3 client.14166.objecter handle_osd_map got epochs [3,3] > 0 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.582+0000 7f5c9af78700 3 client.14166.objecter handle_osd_map decoding full epoch 3 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.582+0000 7f5c9af78700 20 client.14166.objecter dump_active .. 0 homeless 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.582+0000 7f5ca9025700 1 --2- 172.21.15.162:0/2431562572 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5c90033fb0 0x7f5c90036470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.582+0000 7f5c9af78700 10 client.14166.objecter ms_handle_connect 0x7f5c90033fb0 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.582+0000 7f5ca9025700 1 --2- 172.21.15.162:0/2431562572 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5c90033fb0 0x7f5c90036470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5c94009780 tx=0x7f5c94006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:01.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.590+0000 7f5c9af78700 1 -- 172.21.15.162:0/2431562572 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5ca001cdb0 con 0x7f5ca4061290 2024-02-13T00:33:01.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.761+0000 7f5caba8a700 1 -- 172.21.15.162:0/2431562572 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f5ca4046d80 con 0x7f5ca4061290 2024-02-13T00:33:01.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.763+0000 7f5c9af78700 1 -- 172.21.15.162:0/2431562572 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f5ca00242f0 con 0x7f5ca4061290 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.765+0000 7f5caba8a700 10 librados: watch_flush enter 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.765+0000 7f5caba8a700 10 librados: watch_flush exit 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.765+0000 7f5caba8a700 20 client.14166.objecter shutdown clearing up homeless session... 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.765+0000 7f5caba8a700 10 client.14166.objecter successfully canceled tick 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.766+0000 7f5caba8a700 1 -- 172.21.15.162:0/2431562572 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5c90033fb0 msgr2=0x7f5c90036470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.766+0000 7f5caba8a700 1 --2- 172.21.15.162:0/2431562572 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5c90033fb0 0x7f5c90036470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5c94009780 tx=0x7f5c94006cc0).stop 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.766+0000 7f5caba8a700 1 -- 172.21.15.162:0/2431562572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ca4061290 msgr2=0x7f5ca4106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.766+0000 7f5caba8a700 1 --2- 172.21.15.162:0/2431562572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ca4061290 0x7f5ca4106f60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5ca0013560 tx=0x7f5ca0009cb0).stop 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.766+0000 7f5caba8a700 1 -- 172.21.15.162:0/2431562572 shutdown_connections 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.766+0000 7f5caba8a700 1 --2- 172.21.15.162:0/2431562572 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5c90033fb0 0x7f5c90036470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.766+0000 7f5caba8a700 1 --2- 172.21.15.162:0/2431562572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ca4061290 0x7f5ca4106f60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.766+0000 7f5caba8a700 1 -- 172.21.15.162:0/2431562572 >> 172.21.15.162:0/2431562572 conn(0x7f5ca40f1cb0 msgr2=0x7f5ca40f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:01.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.766+0000 7f5caba8a700 1 -- 172.21.15.162:0/2431562572 shutdown_connections 2024-02-13T00:33:01.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.766+0000 7f5caba8a700 1 -- 172.21.15.162:0/2431562572 wait complete. 2024-02-13T00:33:01.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:01.766+0000 7f5caba8a700 1 librados: shutdown 2024-02-13T00:33:02.509 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-13T00:33:02.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-13T00:33:03.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:01.764553+0000 mon.a (mon.0) 89 : audit [INF] from='client.? 172.21.15.162:0/2431562572' entity='client.admin' 2024-02-13T00:33:04.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.663+0000 7fc600c5a700 1 -- 172.21.15.162:0/2200996948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5fc0f6550 msgr2=0x7fc5fc0f6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:04.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.663+0000 7fc600c5a700 1 --2- 172.21.15.162:0/2200996948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5fc0f6550 0x7fc5fc0f6970 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fc5ec004770 tx=0x7fc5ec009cb0).stop 2024-02-13T00:33:04.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.663+0000 7fc600c5a700 1 -- 172.21.15.162:0/2200996948 shutdown_connections 2024-02-13T00:33:04.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.663+0000 7fc600c5a700 1 --2- 172.21.15.162:0/2200996948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5fc0f6550 0x7fc5fc0f6970 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:04.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.663+0000 7fc600c5a700 1 -- 172.21.15.162:0/2200996948 >> 172.21.15.162:0/2200996948 conn(0x7fc5fc0f1cb0 msgr2=0x7fc5fc0f40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:04.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.663+0000 7fc600c5a700 1 -- 172.21.15.162:0/2200996948 shutdown_connections 2024-02-13T00:33:04.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.663+0000 7fc600c5a700 1 -- 172.21.15.162:0/2200996948 wait complete. 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.664+0000 7fc600c5a700 1 librados: starting msgr at 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.664+0000 7fc600c5a700 1 librados: starting objecter 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.664+0000 7fc600c5a700 1 Processor -- start 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.664+0000 7fc600c5a700 1 -- start start 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.664+0000 7fc600c5a700 1 librados: setting wanted keys 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.664+0000 7fc600c5a700 1 librados: calling monclient init 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.664+0000 7fc600c5a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5fc0f6550 0x7fc5fc107030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.664+0000 7fc600c5a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc107520 con 0x7fc5fc0f6550 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.664+0000 7fc5fa59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5fc0f6550 0x7fc5fc107030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.664+0000 7fc5fa59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5fc0f6550 0x7fc5fc107030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:42388/0 (socket says 172.21.15.162:42388) 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.664+0000 7fc5fa59c700 1 -- 172.21.15.162:0/1668613761 learned_addr learned my addr 172.21.15.162:0/1668613761 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.665+0000 7fc5fa59c700 1 -- 172.21.15.162:0/1668613761 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec005040 con 0x7fc5fc0f6550 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.665+0000 7fc5fa59c700 1 --2- 172.21.15.162:0/1668613761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5fc0f6550 0x7fc5fc107030 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fc5ec013560 tx=0x7fc5ec009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:04.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.665+0000 7fc5eb7fe700 10 client.?.objecter ms_handle_connect 0x7fc5fc0f6550 2024-02-13T00:33:04.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.665+0000 7fc5eb7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:33:04.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.665+0000 7fc5eb7fe700 1 -- 172.21.15.162:0/1668613761 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5ec00e030 con 0x7fc5fc0f6550 2024-02-13T00:33:04.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.665+0000 7fc600c5a700 1 -- 172.21.15.162:0/1668613761 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5fc107720 con 0x7fc5fc0f6550 2024-02-13T00:33:04.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.665+0000 7fc5eb7fe700 1 -- 172.21.15.162:0/1668613761 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5ec009710 con 0x7fc5fc0f6550 2024-02-13T00:33:04.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.665+0000 7fc5eb7fe700 1 -- 172.21.15.162:0/1668613761 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5ec018070 con 0x7fc5fc0f6550 2024-02-13T00:33:04.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.665+0000 7fc600c5a700 10 client.14168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:33:04.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.665+0000 7fc600c5a700 1 -- 172.21.15.162:0/1668613761 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5fc107bb0 con 0x7fc5fc0f6550 2024-02-13T00:33:04.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.665+0000 7fc600c5a700 1 librados: init done 2024-02-13T00:33:04.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.666+0000 7fc5eb7fe700 1 -- 172.21.15.162:0/1668613761 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7fc5ec01c450 con 0x7fc5fc0f6550 2024-02-13T00:33:04.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.666+0000 7fc5eb7fe700 1 --2- 172.21.15.162:0/1668613761 >> v2:172.21.15.162:6800/1909610488 conn(0x7fc5e4033f60 0x7fc5e4036420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:04.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.666+0000 7fc5eb7fe700 1 -- 172.21.15.162:0/1668613761 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fc5ec044240 con 0x7fc5fc0f6550 2024-02-13T00:33:04.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.666+0000 7fc5eb7fe700 10 client.14168.objecter ms_dispatch 0x7fc5fc001550 osd_map(3..3 src has 1..3) v4 2024-02-13T00:33:04.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.666+0000 7fc5eb7fe700 3 client.14168.objecter handle_osd_map got epochs [3,3] > 0 2024-02-13T00:33:04.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.666+0000 7fc5eb7fe700 3 client.14168.objecter handle_osd_map decoding full epoch 3 2024-02-13T00:33:04.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.666+0000 7fc600c5a700 1 -- 172.21.15.162:0/1668613761 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5fc05a3b0 con 0x7fc5fc0f6550 2024-02-13T00:33:04.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.666+0000 7fc5eb7fe700 20 client.14168.objecter dump_active .. 0 homeless 2024-02-13T00:33:04.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.666+0000 7fc5f9d9b700 1 --2- 172.21.15.162:0/1668613761 >> v2:172.21.15.162:6800/1909610488 conn(0x7fc5e4033f60 0x7fc5e4036420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:04.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.666+0000 7fc5eb7fe700 10 client.14168.objecter ms_handle_connect 0x7fc5e4033f60 2024-02-13T00:33:04.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.666+0000 7fc5f9d9b700 1 --2- 172.21.15.162:0/1668613761 >> v2:172.21.15.162:6800/1909610488 conn(0x7fc5e4033f60 0x7fc5e4036420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fc5f0009780 tx=0x7fc5f0006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:04.673 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.671+0000 7fc5eb7fe700 1 -- 172.21.15.162:0/1668613761 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc5ec01b760 con 0x7fc5fc0f6550 2024-02-13T00:33:04.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:03.444505+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:04.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.837+0000 7fc600c5a700 1 -- 172.21.15.162:0/1668613761 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5fc0f2cc0 con 0x7fc5e4033f60 2024-02-13T00:33:04.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.841+0000 7fc5eb7fe700 1 -- 172.21.15.162:0/1668613761 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc5fc0f2cc0 con 0x7fc5e4033f60 2024-02-13T00:33:04.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 10 librados: watch_flush enter 2024-02-13T00:33:04.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 10 librados: watch_flush exit 2024-02-13T00:33:04.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 20 client.14168.objecter shutdown clearing up homeless session... 2024-02-13T00:33:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 10 client.14168.objecter successfully canceled tick 2024-02-13T00:33:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 1 -- 172.21.15.162:0/1668613761 >> v2:172.21.15.162:6800/1909610488 conn(0x7fc5e4033f60 msgr2=0x7fc5e4036420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 1 --2- 172.21.15.162:0/1668613761 >> v2:172.21.15.162:6800/1909610488 conn(0x7fc5e4033f60 0x7fc5e4036420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fc5f0009780 tx=0x7fc5f0006cc0).stop 2024-02-13T00:33:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 1 -- 172.21.15.162:0/1668613761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5fc0f6550 msgr2=0x7fc5fc107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 1 --2- 172.21.15.162:0/1668613761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5fc0f6550 0x7fc5fc107030 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fc5ec013560 tx=0x7fc5ec009cb0).stop 2024-02-13T00:33:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 1 -- 172.21.15.162:0/1668613761 shutdown_connections 2024-02-13T00:33:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 1 --2- 172.21.15.162:0/1668613761 >> v2:172.21.15.162:6800/1909610488 conn(0x7fc5e4033f60 0x7fc5e4036420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 1 --2- 172.21.15.162:0/1668613761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5fc0f6550 0x7fc5fc107030 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 1 -- 172.21.15.162:0/1668613761 >> 172.21.15.162:0/1668613761 conn(0x7fc5fc0f1cb0 msgr2=0x7fc5fc0f25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.842+0000 7fc600c5a700 1 -- 172.21.15.162:0/1668613761 shutdown_connections 2024-02-13T00:33:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.843+0000 7fc600c5a700 1 -- 172.21.15.162:0/1668613761 wait complete. 2024-02-13T00:33:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:04.843+0000 7fc600c5a700 1 librados: shutdown 2024-02-13T00:33:05.550 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:04.571556+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:33:05.550 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:04.575849+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:05.550 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:04.577598+0000 mon.a (mon.0) 93 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:33:05.550 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:04.581646+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:05.550 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:04.841457+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:05.550 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:04.846265+0000 mon.a (mon.0) 96 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:33:05.550 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:04.847510+0000 mon.a (mon.0) 97 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:33:05.550 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:04.848065+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:33:05.550 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:33:04.848984+0000 mgr.y (mgr.14146) 6 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-02-13T00:33:05.551 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:33:04.880466+0000 mgr.y (mgr.14146) 7 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-02-13T00:33:05.551 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:04.935649+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:05.551 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:04.938594+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:05.571 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi181 2024-02-13T00:33:05.571 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-13T00:33:05.571 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.conf 2024-02-13T00:33:05.593 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-13T00:33:05.593 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-13T00:33:05.654 INFO:tasks.cephadm:Adding host smithi181 to orchestrator... 2024-02-13T00:33:05.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch host add smithi181 2024-02-13T00:33:08.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:07.476929+0000 mgr.y (mgr.14146) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:10.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:09.477227+0000 mgr.y (mgr.14146) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:12.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:11.477508+0000 mgr.y (mgr.14146) 10 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:14.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:13.477843+0000 mgr.y (mgr.14146) 11 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:16.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:15.478071+0000 mgr.y (mgr.14146) 12 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:18.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:17.478335+0000 mgr.y (mgr.14146) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:20.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:19.478624+0000 mgr.y (mgr.14146) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:22.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:21.478970+0000 mgr.y (mgr.14146) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:24.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:23.479305+0000 mgr.y (mgr.14146) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:26.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:25.479611+0000 mgr.y (mgr.14146) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:28.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:27.479857+0000 mgr.y (mgr.14146) 18 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:30.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:29.480121+0000 mgr.y (mgr.14146) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:32.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:31.480452+0000 mgr.y (mgr.14146) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:34.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:33.480694+0000 mgr.y (mgr.14146) 21 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:35.997 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.993+0000 7f0864ccb700 1 -- 172.21.15.181:0/1903813522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08600f6550 msgr2=0x7f08600f6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:35.997 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.993+0000 7f0864ccb700 1 --2- 172.21.15.181:0/1903813522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08600f6550 0x7f08600f6970 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f0854004770 tx=0x7f0854009cb0).stop 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.993+0000 7f0864ccb700 1 -- 172.21.15.181:0/1903813522 shutdown_connections 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.993+0000 7f0864ccb700 1 --2- 172.21.15.181:0/1903813522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08600f6550 0x7f08600f6970 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.993+0000 7f0864ccb700 1 -- 172.21.15.181:0/1903813522 >> 172.21.15.181:0/1903813522 conn(0x7f08600f1cb0 msgr2=0x7f08600f40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.993+0000 7f0864ccb700 1 -- 172.21.15.181:0/1903813522 shutdown_connections 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.993+0000 7f0864ccb700 1 -- 172.21.15.181:0/1903813522 wait complete. 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.993+0000 7f0864ccb700 1 librados: starting msgr at 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.993+0000 7f0864ccb700 1 librados: starting objecter 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.994+0000 7f0864ccb700 1 Processor -- start 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.994+0000 7f0864ccb700 1 -- start start 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.994+0000 7f0864ccb700 1 librados: setting wanted keys 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.994+0000 7f0864ccb700 1 librados: calling monclient init 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.994+0000 7f0864ccb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08600f6550 0x7f0860109d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:35.998 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.994+0000 7f0864ccb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0860061230 con 0x7f08600f6550 2024-02-13T00:33:35.999 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.994+0000 7f085e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08600f6550 0x7f0860109d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:35.999 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.994+0000 7f085e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08600f6550 0x7f0860109d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.181:36738/0 (socket says 172.21.15.181:36738) 2024-02-13T00:33:35.999 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.994+0000 7f085e59c700 1 -- 172.21.15.181:0/764485170 learned_addr learned my addr 172.21.15.181:0/764485170 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:33:36.001 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.995+0000 7f085e59c700 1 -- 172.21.15.181:0/764485170 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0854005040 con 0x7f08600f6550 2024-02-13T00:33:36.001 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.995+0000 7f085e59c700 1 --2- 172.21.15.181:0/764485170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08600f6550 0x7f0860109d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f0854013a00 tx=0x7f0854009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:36.001 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.995+0000 7f084f7fe700 10 client.?.objecter ms_handle_connect 0x7f08600f6550 2024-02-13T00:33:36.001 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.995+0000 7f084f7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:33:36.001 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.995+0000 7f084f7fe700 1 -- 172.21.15.181:0/764485170 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f085400e030 con 0x7f08600f6550 2024-02-13T00:33:36.001 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.995+0000 7f0864ccb700 1 -- 172.21.15.181:0/764485170 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0860061430 con 0x7f08600f6550 2024-02-13T00:33:36.001 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.995+0000 7f084f7fe700 1 -- 172.21.15.181:0/764485170 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0854009710 con 0x7f08600f6550 2024-02-13T00:33:36.001 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.995+0000 7f084f7fe700 1 -- 172.21.15.181:0/764485170 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0854018070 con 0x7f08600f6550 2024-02-13T00:33:36.001 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.995+0000 7f0864ccb700 10 client.14170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:33:36.001 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.995+0000 7f0864ccb700 1 -- 172.21.15.181:0/764485170 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086010a3f0 con 0x7f08600f6550 2024-02-13T00:33:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.995+0000 7f0864ccb700 1 librados: init done 2024-02-13T00:33:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.996+0000 7f084f7fe700 1 -- 172.21.15.181:0/764485170 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7f085401d420 con 0x7f08600f6550 2024-02-13T00:33:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.996+0000 7f0864ccb700 1 -- 172.21.15.181:0/764485170 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f086005a3b0 con 0x7f08600f6550 2024-02-13T00:33:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.996+0000 7f084f7fe700 1 --2- 172.21.15.181:0/764485170 >> v2:172.21.15.162:6800/1909610488 conn(0x7f0848033fc0 0x7f0848036480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.996+0000 7f084f7fe700 1 -- 172.21.15.181:0/764485170 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f08540442a0 con 0x7f08600f6550 2024-02-13T00:33:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.996+0000 7f084f7fe700 10 client.14170.objecter ms_dispatch 0x7f0860001550 osd_map(3..3 src has 1..3) v4 2024-02-13T00:33:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.996+0000 7f084f7fe700 3 client.14170.objecter handle_osd_map got epochs [3,3] > 0 2024-02-13T00:33:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.996+0000 7f084f7fe700 3 client.14170.objecter handle_osd_map decoding full epoch 3 2024-02-13T00:33:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.996+0000 7f084f7fe700 20 client.14170.objecter dump_active .. 0 homeless 2024-02-13T00:33:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.996+0000 7f085dd9b700 1 --2- 172.21.15.181:0/764485170 >> v2:172.21.15.162:6800/1909610488 conn(0x7f0848033fc0 0x7f0848036480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.997+0000 7f084f7fe700 10 client.14170.objecter ms_handle_connect 0x7f0848033fc0 2024-02-13T00:33:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:35.997+0000 7f085dd9b700 1 --2- 172.21.15.181:0/764485170 >> v2:172.21.15.162:6800/1909610488 conn(0x7f0848033fc0 0x7f0848036480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0850009780 tx=0x7f0850006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:36.004 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.001+0000 7f084f7fe700 1 -- 172.21.15.181:0/764485170 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0854025ce0 con 0x7f08600f6550 2024-02-13T00:33:36.170 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.167+0000 7f0864ccb700 1 -- 172.21.15.181:0/764485170 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}) v1 -- 0x7f08600f2cc0 con 0x7f0848033fc0 2024-02-13T00:33:36.537 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:33:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [90B blob data] 2024-02-13T00:33:36.791 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.787+0000 7f084f7fe700 1 -- 172.21.15.181:0/764485170 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f08600f2cc0 con 0x7f0848033fc0 2024-02-13T00:33:36.791 INFO:teuthology.orchestra.run.smithi181.stdout:Added host 'smithi181' with addr '172.21.15.181' 2024-02-13T00:33:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 10 librados: watch_flush enter 2024-02-13T00:33:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 10 librados: watch_flush exit 2024-02-13T00:33:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 20 client.14170.objecter shutdown clearing up homeless session... 2024-02-13T00:33:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 10 client.14170.objecter successfully canceled tick 2024-02-13T00:33:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 1 -- 172.21.15.181:0/764485170 >> v2:172.21.15.162:6800/1909610488 conn(0x7f0848033fc0 msgr2=0x7f0848036480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 1 --2- 172.21.15.181:0/764485170 >> v2:172.21.15.162:6800/1909610488 conn(0x7f0848033fc0 0x7f0848036480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0850009780 tx=0x7f0850006cc0).stop 2024-02-13T00:33:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 1 -- 172.21.15.181:0/764485170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08600f6550 msgr2=0x7f0860109d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 1 --2- 172.21.15.181:0/764485170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08600f6550 0x7f0860109d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f0854013a00 tx=0x7f0854009cb0).stop 2024-02-13T00:33:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 1 -- 172.21.15.181:0/764485170 shutdown_connections 2024-02-13T00:33:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 1 --2- 172.21.15.181:0/764485170 >> v2:172.21.15.162:6800/1909610488 conn(0x7f0848033fc0 0x7f0848036480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 1 --2- 172.21.15.181:0/764485170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08600f6550 0x7f0860109d40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 1 -- 172.21.15.181:0/764485170 >> 172.21.15.181:0/764485170 conn(0x7f08600f1cb0 msgr2=0x7f08600f25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:36.794 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 1 -- 172.21.15.181:0/764485170 shutdown_connections 2024-02-13T00:33:36.794 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 1 -- 172.21.15.181:0/764485170 wait complete. 2024-02-13T00:33:36.794 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:36.789+0000 7f0864ccb700 1 librados: shutdown 2024-02-13T00:33:36.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:35.480953+0000 mgr.y (mgr.14146) 22 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:37.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:33:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [80B blob data] 2024-02-13T00:33:37.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-13T00:33:38.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:36.168984+0000 mgr.y (mgr.14146) 23 : audit [DBG] from='client.14170 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:33:38.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:36.787553+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:38.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:33:36.787931+0000 mgr.y (mgr.14146) 24 : cephadm [INF] Added host smithi181 2024-02-13T00:33:38.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:36.830315+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:33:38.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:37.612360+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:39.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:37.481338+0000 mgr.y (mgr.14146) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:39.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:38.467746+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:39.471 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.466+0000 7fe7c99b0700 1 -- 172.21.15.181:0/1526543429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7c40f7660 msgr2=0x7fe7c40f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:39.471 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.466+0000 7fe7c99b0700 1 --2- 172.21.15.181:0/1526543429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7c40f7660 0x7fe7c40f7a80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fe7b4004770 tx=0x7fe7b4009cb0).stop 2024-02-13T00:33:39.471 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.466+0000 7fe7c99b0700 1 -- 172.21.15.181:0/1526543429 shutdown_connections 2024-02-13T00:33:39.471 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.466+0000 7fe7c99b0700 1 --2- 172.21.15.181:0/1526543429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7c40f7660 0x7fe7c40f7a80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:39.471 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.466+0000 7fe7c99b0700 1 -- 172.21.15.181:0/1526543429 >> 172.21.15.181:0/1526543429 conn(0x7fe7c40f2de0 msgr2=0x7fe7c40f5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:39.471 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.466+0000 7fe7c99b0700 1 -- 172.21.15.181:0/1526543429 shutdown_connections 2024-02-13T00:33:39.471 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.466+0000 7fe7c99b0700 1 -- 172.21.15.181:0/1526543429 wait complete. 2024-02-13T00:33:39.471 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.466+0000 7fe7c99b0700 1 librados: starting msgr at 2024-02-13T00:33:39.471 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.466+0000 7fe7c99b0700 1 librados: starting objecter 2024-02-13T00:33:39.471 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.466+0000 7fe7c99b0700 1 Processor -- start 2024-02-13T00:33:39.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.467+0000 7fe7c99b0700 1 -- start start 2024-02-13T00:33:39.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.467+0000 7fe7c99b0700 1 librados: setting wanted keys 2024-02-13T00:33:39.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.467+0000 7fe7c99b0700 1 librados: calling monclient init 2024-02-13T00:33:39.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.467+0000 7fe7c99b0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7c40f7660 0x7fe7c4106fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:39.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.467+0000 7fe7c99b0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c41074c0 con 0x7fe7c40f7660 2024-02-13T00:33:39.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.467+0000 7fe7c2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7c40f7660 0x7fe7c4106fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:39.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.467+0000 7fe7c2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7c40f7660 0x7fe7c4106fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.181:39504/0 (socket says 172.21.15.181:39504) 2024-02-13T00:33:39.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.467+0000 7fe7c2ffd700 1 -- 172.21.15.181:0/2009575089 learned_addr learned my addr 172.21.15.181:0/2009575089 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:33:39.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.467+0000 7fe7c2ffd700 1 -- 172.21.15.181:0/2009575089 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b4005040 con 0x7fe7c40f7660 2024-02-13T00:33:39.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.468+0000 7fe7c2ffd700 1 --2- 172.21.15.181:0/2009575089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7c40f7660 0x7fe7c4106fd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fe7b4013560 tx=0x7fe7b4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:39.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.468+0000 7fe7abfff700 10 client.?.objecter ms_handle_connect 0x7fe7c40f7660 2024-02-13T00:33:39.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.468+0000 7fe7abfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:33:39.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.468+0000 7fe7abfff700 1 -- 172.21.15.181:0/2009575089 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe7b400e030 con 0x7fe7c40f7660 2024-02-13T00:33:39.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.468+0000 7fe7abfff700 1 -- 172.21.15.181:0/2009575089 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7b4009710 con 0x7fe7c40f7660 2024-02-13T00:33:39.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.468+0000 7fe7c99b0700 1 -- 172.21.15.181:0/2009575089 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c41076c0 con 0x7fe7c40f7660 2024-02-13T00:33:39.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.468+0000 7fe7abfff700 1 -- 172.21.15.181:0/2009575089 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe7b4018070 con 0x7fe7c40f7660 2024-02-13T00:33:39.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.468+0000 7fe7c99b0700 10 client.14172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:33:39.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.468+0000 7fe7c99b0700 1 -- 172.21.15.181:0/2009575089 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c4107b50 con 0x7fe7c40f7660 2024-02-13T00:33:39.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.468+0000 7fe7c99b0700 1 librados: init done 2024-02-13T00:33:39.474 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.469+0000 7fe7abfff700 1 -- 172.21.15.181:0/2009575089 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7fe7b401c450 con 0x7fe7c40f7660 2024-02-13T00:33:39.474 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.469+0000 7fe7abfff700 1 --2- 172.21.15.181:0/2009575089 >> v2:172.21.15.162:6800/1909610488 conn(0x7fe7ac033f60 0x7fe7ac036420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:39.474 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.469+0000 7fe7abfff700 1 -- 172.21.15.181:0/2009575089 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fe7b40451d0 con 0x7fe7c40f7660 2024-02-13T00:33:39.474 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.469+0000 7fe7abfff700 10 client.14172.objecter ms_dispatch 0x7fe7c4001550 osd_map(3..3 src has 1..3) v4 2024-02-13T00:33:39.474 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.469+0000 7fe7abfff700 3 client.14172.objecter handle_osd_map got epochs [3,3] > 0 2024-02-13T00:33:39.474 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.469+0000 7fe7abfff700 3 client.14172.objecter handle_osd_map decoding full epoch 3 2024-02-13T00:33:39.474 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.469+0000 7fe7abfff700 20 client.14172.objecter dump_active .. 0 homeless 2024-02-13T00:33:39.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.469+0000 7fe7c99b0700 1 -- 172.21.15.181:0/2009575089 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c405a3b0 con 0x7fe7c40f7660 2024-02-13T00:33:39.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.469+0000 7fe7c27fc700 1 --2- 172.21.15.181:0/2009575089 >> v2:172.21.15.162:6800/1909610488 conn(0x7fe7ac033f60 0x7fe7ac036420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:39.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.470+0000 7fe7abfff700 10 client.14172.objecter ms_handle_connect 0x7fe7ac033f60 2024-02-13T00:33:39.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.470+0000 7fe7c27fc700 1 --2- 172.21.15.181:0/2009575089 >> v2:172.21.15.162:6800/1909610488 conn(0x7fe7ac033f60 0x7fe7ac036420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe7b8009780 tx=0x7fe7b8006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:39.477 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.474+0000 7fe7abfff700 1 -- 172.21.15.181:0/2009575089 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe7b4027440 con 0x7fe7c40f7660 2024-02-13T00:33:39.655 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.651+0000 7fe7c99b0700 1 -- 172.21.15.181:0/2009575089 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe7c40fcb30 con 0x7fe7ac033f60 2024-02-13T00:33:39.656 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.654+0000 7fe7abfff700 1 -- 172.21.15.181:0/2009575089 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fe7c40fcb30 con 0x7fe7ac033f60 2024-02-13T00:33:39.656 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:33:39.657 INFO:teuthology.orchestra.run.smithi181.stdout:[{"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}, {"addr": "172.21.15.181", "hostname": "smithi181", "labels": [], "status": ""}] 2024-02-13T00:33:39.658 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 10 librados: watch_flush enter 2024-02-13T00:33:39.658 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 10 librados: watch_flush exit 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 20 client.14172.objecter shutdown clearing up homeless session... 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 10 client.14172.objecter successfully canceled tick 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 1 -- 172.21.15.181:0/2009575089 >> v2:172.21.15.162:6800/1909610488 conn(0x7fe7ac033f60 msgr2=0x7fe7ac036420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 1 --2- 172.21.15.181:0/2009575089 >> v2:172.21.15.162:6800/1909610488 conn(0x7fe7ac033f60 0x7fe7ac036420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe7b8009780 tx=0x7fe7b8006cc0).stop 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 1 -- 172.21.15.181:0/2009575089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7c40f7660 msgr2=0x7fe7c4106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 1 --2- 172.21.15.181:0/2009575089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7c40f7660 0x7fe7c4106fd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fe7b4013560 tx=0x7fe7b4009cb0).stop 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 1 -- 172.21.15.181:0/2009575089 shutdown_connections 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 1 --2- 172.21.15.181:0/2009575089 >> v2:172.21.15.162:6800/1909610488 conn(0x7fe7ac033f60 0x7fe7ac036420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 1 --2- 172.21.15.181:0/2009575089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7c40f7660 0x7fe7c4106fd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 1 -- 172.21.15.181:0/2009575089 >> 172.21.15.181:0/2009575089 conn(0x7fe7c40f2de0 msgr2=0x7fe7c40f36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 1 -- 172.21.15.181:0/2009575089 shutdown_connections 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 1 -- 172.21.15.181:0/2009575089 wait complete. 2024-02-13T00:33:39.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:39.655+0000 7fe7c99b0700 1 librados: shutdown 2024-02-13T00:33:40.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:39.481621+0000 mgr.y (mgr.14146) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:40.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:39.653924+0000 mgr.y (mgr.14146) 27 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:33:41.185 INFO:tasks.cephadm:Setting crush tunables to jewel 2024-02-13T00:33:41.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd crush tunables jewel 2024-02-13T00:33:42.788 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:41.481965+0000 mgr.y (mgr.14146) 28 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:42.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.977+0000 7f3b3f88c700 1 -- 172.21.15.162:0/2812201134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b380f8730 msgr2=0x7f3b380f8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.977+0000 7f3b3f88c700 1 --2- 172.21.15.162:0/2812201134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b380f8730 0x7f3b380f8b10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3b28004770 tx=0x7f3b28009cb0).stop 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.977+0000 7f3b3f88c700 1 -- 172.21.15.162:0/2812201134 shutdown_connections 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.977+0000 7f3b3f88c700 1 --2- 172.21.15.162:0/2812201134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b380f8730 0x7f3b380f8b10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.977+0000 7f3b3f88c700 1 -- 172.21.15.162:0/2812201134 >> 172.21.15.162:0/2812201134 conn(0x7f3b3806c4e0 msgr2=0x7f3b3806c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.977+0000 7f3b3f88c700 1 -- 172.21.15.162:0/2812201134 shutdown_connections 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.977+0000 7f3b3f88c700 1 -- 172.21.15.162:0/2812201134 wait complete. 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3f88c700 1 librados: starting msgr at 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3f88c700 1 librados: starting objecter 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3f88c700 1 Processor -- start 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3f88c700 1 -- start start 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3f88c700 1 librados: setting wanted keys 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3f88c700 1 librados: calling monclient init 2024-02-13T00:33:42.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3f88c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b380f8730 0x7f3b3810d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3f88c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3810d750 con 0x7f3b380f8730 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3d628700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b380f8730 0x7f3b3810d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3d628700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b380f8730 0x7f3b3810d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43754/0 (socket says 172.21.15.162:43754) 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3d628700 1 -- 172.21.15.162:0/1383130995 learned_addr learned my addr 172.21.15.162:0/1383130995 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3d628700 1 -- 172.21.15.162:0/1383130995 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b28005040 con 0x7f3b380f8730 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b3d628700 1 --2- 172.21.15.162:0/1383130995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b380f8730 0x7f3b3810d260 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3b28009b10 tx=0x7f3b28009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b2e7fc700 10 client.?.objecter ms_handle_connect 0x7f3b380f8730 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.978+0000 7f3b2e7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.979+0000 7f3b2e7fc700 1 -- 172.21.15.162:0/1383130995 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3b2800e030 con 0x7f3b380f8730 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.979+0000 7f3b3f88c700 1 -- 172.21.15.162:0/1383130995 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3810d950 con 0x7f3b380f8730 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.979+0000 7f3b2e7fc700 1 -- 172.21.15.162:0/1383130995 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b28009430 con 0x7f3b380f8730 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.979+0000 7f3b2e7fc700 1 -- 172.21.15.162:0/1383130995 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3b28018070 con 0x7f3b380f8730 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.979+0000 7f3b3f88c700 10 client.14174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.979+0000 7f3b3f88c700 1 -- 172.21.15.162:0/1383130995 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3810dd60 con 0x7f3b380f8730 2024-02-13T00:33:42.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.979+0000 7f3b3f88c700 1 librados: init done 2024-02-13T00:33:42.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.979+0000 7f3b2e7fc700 1 -- 172.21.15.162:0/1383130995 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7f3b2801d420 con 0x7f3b380f8730 2024-02-13T00:33:42.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.980+0000 7f3b2e7fc700 1 --2- 172.21.15.162:0/1383130995 >> v2:172.21.15.162:6800/1909610488 conn(0x7f3b24034000 0x7f3b240364c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:42.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.979+0000 7f3b3f88c700 1 -- 172.21.15.162:0/1383130995 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3805f290 con 0x7f3b380f8730 2024-02-13T00:33:42.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.980+0000 7f3b2e7fc700 1 -- 172.21.15.162:0/1383130995 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f3b280441a0 con 0x7f3b380f8730 2024-02-13T00:33:42.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.980+0000 7f3b2e7fc700 10 client.14174.objecter ms_dispatch 0x7f3b38001550 osd_map(3..3 src has 1..3) v4 2024-02-13T00:33:42.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.980+0000 7f3b2e7fc700 3 client.14174.objecter handle_osd_map got epochs [3,3] > 0 2024-02-13T00:33:42.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.980+0000 7f3b2e7fc700 3 client.14174.objecter handle_osd_map decoding full epoch 3 2024-02-13T00:33:42.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.980+0000 7f3b3ce27700 1 --2- 172.21.15.162:0/1383130995 >> v2:172.21.15.162:6800/1909610488 conn(0x7f3b24034000 0x7f3b240364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:42.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.980+0000 7f3b2e7fc700 20 client.14174.objecter dump_active .. 0 homeless 2024-02-13T00:33:42.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.980+0000 7f3b2e7fc700 10 client.14174.objecter ms_handle_connect 0x7f3b24034000 2024-02-13T00:33:42.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.980+0000 7f3b3ce27700 1 --2- 172.21.15.162:0/1383130995 >> v2:172.21.15.162:6800/1909610488 conn(0x7f3b24034000 0x7f3b240364c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f3b34009780 tx=0x7f3b34006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:42.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:42.984+0000 7f3b2e7fc700 1 -- 172.21.15.162:0/1383130995 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3b28025b80 con 0x7f3b380f8730 2024-02-13T00:33:43.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.149+0000 7f3b3f88c700 1 -- 172.21.15.162:0/1383130995 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "jewel"} v 0) v1 -- 0x7f3b38046d80 con 0x7f3b380f8730 2024-02-13T00:33:43.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.938+0000 7f3b2e7fc700 1 -- 172.21.15.162:0/1383130995 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "jewel"}]=0 adjusted tunables profile to jewel v4) v1 ==== 120+0+0 (secure 0 0 0) 0x7f3b280290c0 con 0x7f3b380f8730 2024-02-13T00:33:43.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 10 librados: watch_flush enter 2024-02-13T00:33:43.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 10 librados: watch_flush exit 2024-02-13T00:33:43.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 20 client.14174.objecter shutdown clearing up homeless session... 2024-02-13T00:33:43.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 10 client.14174.objecter successfully canceled tick 2024-02-13T00:33:43.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 1 -- 172.21.15.162:0/1383130995 >> v2:172.21.15.162:6800/1909610488 conn(0x7f3b24034000 msgr2=0x7f3b240364c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:43.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 1 --2- 172.21.15.162:0/1383130995 >> v2:172.21.15.162:6800/1909610488 conn(0x7f3b24034000 0x7f3b240364c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f3b34009780 tx=0x7f3b34006cc0).stop 2024-02-13T00:33:43.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 1 -- 172.21.15.162:0/1383130995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b380f8730 msgr2=0x7f3b3810d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:43.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 1 --2- 172.21.15.162:0/1383130995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b380f8730 0x7f3b3810d260 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3b28009b10 tx=0x7f3b28009cb0).stop 2024-02-13T00:33:43.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 1 -- 172.21.15.162:0/1383130995 shutdown_connections 2024-02-13T00:33:43.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 1 --2- 172.21.15.162:0/1383130995 >> v2:172.21.15.162:6800/1909610488 conn(0x7f3b24034000 0x7f3b240364c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:43.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 1 --2- 172.21.15.162:0/1383130995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b380f8730 0x7f3b3810d260 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:43.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 1 -- 172.21.15.162:0/1383130995 >> 172.21.15.162:0/1383130995 conn(0x7f3b3806c4e0 msgr2=0x7f3b380fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:43.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 1 -- 172.21.15.162:0/1383130995 shutdown_connections 2024-02-13T00:33:43.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 1 -- 172.21.15.162:0/1383130995 wait complete. 2024-02-13T00:33:43.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:33:43.941+0000 7f3b3f88c700 1 librados: shutdown 2024-02-13T00:33:43.945 INFO:teuthology.orchestra.run.smithi162.stderr:adjusted tunables profile to jewel 2024-02-13T00:33:44.241 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:42.936506+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:44.241 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:43.150192+0000 mon.a (mon.0) 106 : audit [INF] from='client.? 172.21.15.162:0/1383130995' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-02-13T00:33:44.241 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:43.258771+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:44.241 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:43.259327+0000 mon.a (mon.0) 108 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:33:44.241 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:43.260109+0000 mon.a (mon.0) 109 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:33:44.241 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:43.260708+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:33:44.242 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:43.327750+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:44.242 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:43.332986+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:44.529 INFO:tasks.cephadm:Adding mon.a on smithi162 2024-02-13T00:33:44.529 INFO:tasks.cephadm:Adding mon.c on smithi162 2024-02-13T00:33:44.529 INFO:tasks.cephadm:Adding mon.b on smithi181 2024-02-13T00:33:44.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch apply mon '3;smithi162:172.21.15.162=a;smithi162:[v2:172.21.15.162:3301,v1:172.21.15.162:6790]=c;smithi181:172.21.15.181=b' 2024-02-13T00:33:45.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:33:43.261673+0000 mgr.y (mgr.14146) 29 : cephadm [INF] Updating smithi181:/etc/ceph/ceph.conf 2024-02-13T00:33:45.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:33:43.298179+0000 mgr.y (mgr.14146) 30 : cephadm [INF] Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-02-13T00:33:45.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:43.482414+0000 mgr.y (mgr.14146) 31 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:45.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:43.939566+0000 mon.a (mon.0) 113 : audit [INF] from='client.? 172.21.15.162:0/1383130995' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-02-13T00:33:45.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:43.939629+0000 mon.a (mon.0) 114 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-13T00:33:46.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.281+0000 7f8553b57700 1 -- 172.21.15.181:0/944425623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f854c0fc5e0 msgr2=0x7f854c0fc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:46.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.281+0000 7f8553b57700 1 --2- 172.21.15.181:0/944425623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f854c0fc5e0 0x7f854c0fc9c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f853c004770 tx=0x7f853c009cb0).stop 2024-02-13T00:33:46.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.281+0000 7f8553b57700 1 -- 172.21.15.181:0/944425623 shutdown_connections 2024-02-13T00:33:46.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.281+0000 7f8553b57700 1 --2- 172.21.15.181:0/944425623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f854c0fc5e0 0x7f854c0fc9c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:46.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.281+0000 7f8553b57700 1 -- 172.21.15.181:0/944425623 >> 172.21.15.181:0/944425623 conn(0x7f854c06c4e0 msgr2=0x7f854c06c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:46.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.282+0000 7f8553b57700 1 -- 172.21.15.181:0/944425623 shutdown_connections 2024-02-13T00:33:46.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.282+0000 7f8553b57700 1 -- 172.21.15.181:0/944425623 wait complete. 2024-02-13T00:33:46.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.282+0000 7f8553b57700 1 librados: starting msgr at 2024-02-13T00:33:46.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.282+0000 7f8553b57700 1 librados: starting objecter 2024-02-13T00:33:46.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.282+0000 7f8553b57700 1 Processor -- start 2024-02-13T00:33:46.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.282+0000 7f8553b57700 1 -- start start 2024-02-13T00:33:46.287 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.282+0000 7f8553b57700 1 librados: setting wanted keys 2024-02-13T00:33:46.287 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.282+0000 7f8553b57700 1 librados: calling monclient init 2024-02-13T00:33:46.287 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.283+0000 7f8553b57700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f854c0fc5e0 0x7f854c108160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:46.287 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.283+0000 7f8553b57700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c108650 con 0x7f854c0fc5e0 2024-02-13T00:33:46.289 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.283+0000 7f85518f3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f854c0fc5e0 0x7f854c108160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:46.289 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.283+0000 7f85518f3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f854c0fc5e0 0x7f854c108160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.181:39530/0 (socket says 172.21.15.181:39530) 2024-02-13T00:33:46.289 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.283+0000 7f85518f3700 1 -- 172.21.15.181:0/763885887 learned_addr learned my addr 172.21.15.181:0/763885887 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:33:46.289 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.283+0000 7f85518f3700 1 -- 172.21.15.181:0/763885887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c005040 con 0x7f854c0fc5e0 2024-02-13T00:33:46.289 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.283+0000 7f85518f3700 1 --2- 172.21.15.181:0/763885887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f854c0fc5e0 0x7f854c108160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f853c013a30 tx=0x7f853c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:46.289 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.283+0000 7f8542ffd700 10 client.?.objecter ms_handle_connect 0x7f854c0fc5e0 2024-02-13T00:33:46.289 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.283+0000 7f8542ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:33:46.289 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.284+0000 7f8542ffd700 1 -- 172.21.15.181:0/763885887 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f853c00e070 con 0x7f854c0fc5e0 2024-02-13T00:33:46.289 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.284+0000 7f8542ffd700 1 -- 172.21.15.181:0/763885887 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f853c013630 con 0x7f854c0fc5e0 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.284+0000 7f8553b57700 1 -- 172.21.15.181:0/763885887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c108850 con 0x7f854c0fc5e0 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.284+0000 7f8542ffd700 1 -- 172.21.15.181:0/763885887 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f853c018070 con 0x7f854c0fc5e0 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.284+0000 7f8553b57700 10 client.14176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.284+0000 7f8553b57700 1 -- 172.21.15.181:0/763885887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c108ce0 con 0x7f854c0fc5e0 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.284+0000 7f8553b57700 1 librados: init done 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.284+0000 7f8542ffd700 1 -- 172.21.15.181:0/763885887 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7f853c01d420 con 0x7f854c0fc5e0 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.284+0000 7f8542ffd700 1 --2- 172.21.15.181:0/763885887 >> v2:172.21.15.162:6800/1909610488 conn(0x7f8538034010 0x7f85380364d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.284+0000 7f8542ffd700 1 -- 172.21.15.181:0/763885887 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f853c043b00 con 0x7f854c0fc5e0 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.285+0000 7f8542ffd700 10 client.14176.objecter ms_dispatch 0x7f854c001550 osd_map(4..4 src has 1..4) v4 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.285+0000 7f8542ffd700 3 client.14176.objecter handle_osd_map got epochs [4,4] > 0 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.285+0000 7f8542ffd700 3 client.14176.objecter handle_osd_map decoding full epoch 4 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.284+0000 7f8553b57700 1 -- 172.21.15.181:0/763885887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854c05f290 con 0x7f854c0fc5e0 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.285+0000 7f8542ffd700 20 client.14176.objecter dump_active .. 0 homeless 2024-02-13T00:33:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.285+0000 7f85510f2700 1 --2- 172.21.15.181:0/763885887 >> v2:172.21.15.162:6800/1909610488 conn(0x7f8538034010 0x7f85380364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:46.291 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.285+0000 7f8542ffd700 10 client.14176.objecter ms_handle_connect 0x7f8538034010 2024-02-13T00:33:46.291 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.285+0000 7f85510f2700 1 --2- 172.21.15.181:0/763885887 >> v2:172.21.15.162:6800/1909610488 conn(0x7f8538034010 0x7f85380364d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f8548009780 tx=0x7f8548006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:46.293 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.290+0000 7f8542ffd700 1 -- 172.21.15.181:0/763885887 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f853c043e60 con 0x7f854c0fc5e0 2024-02-13T00:33:46.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.461+0000 7f8553b57700 1 -- 172.21.15.181:0/763885887 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi162:172.21.15.162=a;smithi162:[v2:172.21.15.162:3301,v1:172.21.15.162:6790]=c;smithi181:172.21.15.181=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f854c0fff20 con 0x7f8538034010 2024-02-13T00:33:46.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.467+0000 7f8542ffd700 1 -- 172.21.15.181:0/763885887 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f854c0fff20 con 0x7f8538034010 2024-02-13T00:33:46.470 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mon update... 2024-02-13T00:33:46.472 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 10 librados: watch_flush enter 2024-02-13T00:33:46.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 10 librados: watch_flush exit 2024-02-13T00:33:46.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 20 client.14176.objecter shutdown clearing up homeless session... 2024-02-13T00:33:46.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 10 client.14176.objecter successfully canceled tick 2024-02-13T00:33:46.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 1 -- 172.21.15.181:0/763885887 >> v2:172.21.15.162:6800/1909610488 conn(0x7f8538034010 msgr2=0x7f85380364d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:46.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 1 --2- 172.21.15.181:0/763885887 >> v2:172.21.15.162:6800/1909610488 conn(0x7f8538034010 0x7f85380364d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f8548009780 tx=0x7f8548006cc0).stop 2024-02-13T00:33:46.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 1 -- 172.21.15.181:0/763885887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f854c0fc5e0 msgr2=0x7f854c108160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:46.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 1 --2- 172.21.15.181:0/763885887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f854c0fc5e0 0x7f854c108160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f853c013a30 tx=0x7f853c009cb0).stop 2024-02-13T00:33:46.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 1 -- 172.21.15.181:0/763885887 shutdown_connections 2024-02-13T00:33:46.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 1 --2- 172.21.15.181:0/763885887 >> v2:172.21.15.162:6800/1909610488 conn(0x7f8538034010 0x7f85380364d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:46.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 1 --2- 172.21.15.181:0/763885887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f854c0fc5e0 0x7f854c108160 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:46.473 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 1 -- 172.21.15.181:0/763885887 >> 172.21.15.181:0/763885887 conn(0x7f854c06c4e0 msgr2=0x7f854c0fba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:46.474 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 1 -- 172.21.15.181:0/763885887 shutdown_connections 2024-02-13T00:33:46.474 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 1 -- 172.21.15.181:0/763885887 wait complete. 2024-02-13T00:33:46.474 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:46.469+0000 7f8553b57700 1 librados: shutdown 2024-02-13T00:33:47.108 DEBUG:teuthology.orchestra.run.smithi162:mon.c> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.c.service 2024-02-13T00:33:47.112 DEBUG:teuthology.orchestra.run.smithi181:mon.b> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.b.service 2024-02-13T00:33:47.115 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-13T00:33:47.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T00:33:47.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:-- Logs begin at Tue 2024-02-13 00:17:30 UTC. -- 2024-02-13T00:33:47.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:-- Logs begin at Tue 2024-02-13 00:17:45 UTC. -- 2024-02-13T00:33:47.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:45.482669+0000 mgr.y (mgr.14146) 32 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:47.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:46.468024+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:47.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:46.503991+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:33:47.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:46.505680+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:33:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:46.506465+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:33:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:46.509947+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:33:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:46.511670+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T00:33:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:46.512357+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:33:48.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:46.463179+0000 mgr.y (mgr.14146) 33 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi162:172.21.15.162=a;smithi162:[v2:172.21.15.162:3301,v1:172.21.15.162:6790]=c;smithi181:172.21.15.181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:33:48.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:33:46.465322+0000 mgr.y (mgr.14146) 34 : cephadm [INF] Saving service mon spec with placement smithi162:172.21.15.162=a;smithi162:[v2:172.21.15.162:3301,v1:172.21.15.162:6790]=c;smithi181:172.21.15.181=b;count:3 2024-02-13T00:33:48.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:33:46.513037+0000 mgr.y (mgr.14146) 35 : cephadm [INF] Deploying daemon mon.b on smithi181 2024-02-13T00:33:49.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:47.482959+0000 mgr.y (mgr.14146) 36 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:50.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:49.483358+0000 mgr.y (mgr.14146) 37 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:52.213 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.210+0000 7f8bb22f6700 1 -- 172.21.15.181:0/142779385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bac0fc5e0 msgr2=0x7f8bac0fc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:52.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.210+0000 7f8bb22f6700 1 --2- 172.21.15.181:0/142779385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bac0fc5e0 0x7f8bac0fc9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f8b94004770 tx=0x7f8b94009cb0).stop 2024-02-13T00:33:52.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.210+0000 7f8bb22f6700 1 -- 172.21.15.181:0/142779385 shutdown_connections 2024-02-13T00:33:52.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.210+0000 7f8bb22f6700 1 --2- 172.21.15.181:0/142779385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bac0fc5e0 0x7f8bac0fc9c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:52.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.210+0000 7f8bb22f6700 1 -- 172.21.15.181:0/142779385 >> 172.21.15.181:0/142779385 conn(0x7f8bac06c4e0 msgr2=0x7f8bac06c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:52.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.210+0000 7f8bb22f6700 1 -- 172.21.15.181:0/142779385 shutdown_connections 2024-02-13T00:33:52.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.210+0000 7f8bb22f6700 1 -- 172.21.15.181:0/142779385 wait complete. 2024-02-13T00:33:52.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.210+0000 7f8bb22f6700 1 librados: starting msgr at 2024-02-13T00:33:52.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.210+0000 7f8bb22f6700 1 librados: starting objecter 2024-02-13T00:33:52.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.210+0000 7f8bb22f6700 1 Processor -- start 2024-02-13T00:33:52.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.211+0000 7f8bb22f6700 1 -- start start 2024-02-13T00:33:52.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.211+0000 7f8bb22f6700 1 librados: setting wanted keys 2024-02-13T00:33:52.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.211+0000 7f8bb22f6700 1 librados: calling monclient init 2024-02-13T00:33:52.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.211+0000 7f8bb22f6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bac0fc5e0 0x7f8bac10d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:52.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.211+0000 7f8bb22f6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac10d620 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.211+0000 7f8babfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bac0fc5e0 0x7f8bac10d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:52.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.211+0000 7f8babfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bac0fc5e0 0x7f8bac10d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.181:46342/0 (socket says 172.21.15.181:46342) 2024-02-13T00:33:52.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.211+0000 7f8babfff700 1 -- 172.21.15.181:0/2969678923 learned_addr learned my addr 172.21.15.181:0/2969678923 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:33:52.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.211+0000 7f8babfff700 1 -- 172.21.15.181:0/2969678923 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b94005040 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.211+0000 7f8babfff700 1 --2- 172.21.15.181:0/2969678923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bac0fc5e0 0x7f8bac10d130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f8b94009b10 tx=0x7f8b94009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:52.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.211+0000 7f8ba97fa700 10 client.?.objecter ms_handle_connect 0x7f8bac0fc5e0 2024-02-13T00:33:52.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.211+0000 7f8ba97fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:33:52.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.212+0000 7f8ba97fa700 1 -- 172.21.15.181:0/2969678923 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b9400e030 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.212+0000 7f8bb22f6700 1 -- 172.21.15.181:0/2969678923 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bac10d820 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.212+0000 7f8ba97fa700 1 -- 172.21.15.181:0/2969678923 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b94009430 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.212+0000 7f8ba97fa700 1 -- 172.21.15.181:0/2969678923 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b94018070 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.212+0000 7f8bb22f6700 10 client.14178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.212+0000 7f8bb22f6700 1 -- 172.21.15.181:0/2969678923 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bac10dc30 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.212+0000 7f8bb22f6700 1 librados: init done 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.212+0000 7f8ba97fa700 1 -- 172.21.15.181:0/2969678923 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7f8b9401d420 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.213+0000 7f8ba97fa700 1 --2- 172.21.15.181:0/2969678923 >> v2:172.21.15.162:6800/1909610488 conn(0x7f8b98034050 0x7f8b98036510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.213+0000 7f8ba97fa700 1 -- 172.21.15.181:0/2969678923 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f8b94044120 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.213+0000 7f8ba97fa700 10 client.14178.objecter ms_dispatch 0x7f8bac001550 osd_map(4..4 src has 1..4) v4 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.213+0000 7f8ba97fa700 3 client.14178.objecter handle_osd_map got epochs [4,4] > 0 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.213+0000 7f8ba97fa700 3 client.14178.objecter handle_osd_map decoding full epoch 4 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.213+0000 7f8ba97fa700 20 client.14178.objecter dump_active .. 0 homeless 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.213+0000 7f8bb22f6700 1 -- 172.21.15.181:0/2969678923 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bac05f290 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.213+0000 7f8bab7fe700 1 --2- 172.21.15.181:0/2969678923 >> v2:172.21.15.162:6800/1909610488 conn(0x7f8b98034050 0x7f8b98036510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:52.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.213+0000 7f8ba97fa700 10 client.14178.objecter ms_handle_connect 0x7f8b98034050 2024-02-13T00:33:52.219 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.213+0000 7f8bab7fe700 1 --2- 172.21.15.181:0/2969678923 >> v2:172.21.15.162:6800/1909610488 conn(0x7f8b98034050 0x7f8b98036510 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f8b9c009780 tx=0x7f8b9c006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:52.221 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.218+0000 7f8ba97fa700 1 -- 172.21.15.181:0/2969678923 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8b94025b80 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.451 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.448+0000 7f8bb22f6700 1 -- 172.21.15.181:0/2969678923 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8bac05a3b0 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.452 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.449+0000 7f8ba97fa700 1 -- 172.21.15.181:0/2969678923 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+727 (secure 0 0 0) 0x7f8b94025b80 con 0x7f8bac0fc5e0 2024-02-13T00:33:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:33:52.452 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"d4ee01fc-ca06-11ee-95b9-87774f69a715","modified":"2024-02-13T00:31:48.749492Z","created":"2024-02-13T00:31:48.749492Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T00:33:52.454 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.450+0000 7f8bb22f6700 10 librados: watch_flush enter 2024-02-13T00:33:52.455 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.450+0000 7f8bb22f6700 10 librados: watch_flush exit 2024-02-13T00:33:52.455 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.450+0000 7f8bb22f6700 20 client.14178.objecter shutdown clearing up homeless session... 2024-02-13T00:33:52.455 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.450+0000 7f8bb22f6700 10 client.14178.objecter successfully canceled tick 2024-02-13T00:33:52.455 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.450+0000 7f8bb22f6700 1 -- 172.21.15.181:0/2969678923 >> v2:172.21.15.162:6800/1909610488 conn(0x7f8b98034050 msgr2=0x7f8b98036510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:52.455 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.451+0000 7f8bb22f6700 1 --2- 172.21.15.181:0/2969678923 >> v2:172.21.15.162:6800/1909610488 conn(0x7f8b98034050 0x7f8b98036510 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f8b9c009780 tx=0x7f8b9c006cc0).stop 2024-02-13T00:33:52.455 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.451+0000 7f8bb22f6700 1 -- 172.21.15.181:0/2969678923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bac0fc5e0 msgr2=0x7f8bac10d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:52.455 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.451+0000 7f8bb22f6700 1 --2- 172.21.15.181:0/2969678923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bac0fc5e0 0x7f8bac10d130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f8b94009b10 tx=0x7f8b94009cb0).stop 2024-02-13T00:33:52.455 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.451+0000 7f8bb22f6700 1 -- 172.21.15.181:0/2969678923 shutdown_connections 2024-02-13T00:33:52.455 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.451+0000 7f8bb22f6700 1 --2- 172.21.15.181:0/2969678923 >> v2:172.21.15.162:6800/1909610488 conn(0x7f8b98034050 0x7f8b98036510 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:52.456 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.451+0000 7f8bb22f6700 1 --2- 172.21.15.181:0/2969678923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bac0fc5e0 0x7f8bac10d130 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:52.456 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.451+0000 7f8bb22f6700 1 -- 172.21.15.181:0/2969678923 >> 172.21.15.181:0/2969678923 conn(0x7f8bac06c4e0 msgr2=0x7f8bac060da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:52.456 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.451+0000 7f8bb22f6700 1 -- 172.21.15.181:0/2969678923 shutdown_connections 2024-02-13T00:33:52.456 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.451+0000 7f8bb22f6700 1 -- 172.21.15.181:0/2969678923 wait complete. 2024-02-13T00:33:52.456 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:52.451+0000 7f8bb22f6700 1 librados: shutdown 2024-02-13T00:33:52.456 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-02-13T00:33:52.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:51.483672+0000 mgr.y (mgr.14146) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:33:52.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:52.449906+0000 mon.a (mon.0) 122 : audit [DBG] from='client.? 172.21.15.181:0/2969678923' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T00:33:54.692 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-13T00:33:54.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T00:33:54.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:54 smithi181 bash[40895]: 2274e34fa0f8d672af842d7daf90233af1e05c6687d476ec08082a2311732f4a 2024-02-13T00:33:54.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:54 smithi181 systemd[1]: Started Ceph mon.b for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:33:59.567 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.564+0000 7f5267898700 1 -- 172.21.15.181:0/3318350803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f524c003400 msgr2=0x7f524c005890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:59.567 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.564+0000 7f5267898700 1 --2- 172.21.15.181:0/3318350803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f524c003400 0x7f524c005890 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5260060880 tx=0x7f525c00b040).stop 2024-02-13T00:33:59.567 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.564+0000 7f5267898700 1 -- 172.21.15.181:0/3318350803 shutdown_connections 2024-02-13T00:33:59.568 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.564+0000 7f5267898700 1 --2- 172.21.15.181:0/3318350803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f524c003400 0x7f524c005890 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:59.568 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.564+0000 7f5267898700 1 -- 172.21.15.181:0/3318350803 >> 172.21.15.181:0/3318350803 conn(0x7f526006c4e0 msgr2=0x7f526006c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:59.568 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.564+0000 7f5267898700 1 -- 172.21.15.181:0/3318350803 shutdown_connections 2024-02-13T00:33:59.568 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.564+0000 7f5267898700 1 -- 172.21.15.181:0/3318350803 wait complete. 2024-02-13T00:33:59.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.564+0000 7f5267898700 1 librados: starting msgr at 2024-02-13T00:33:59.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.564+0000 7f5267898700 1 librados: starting objecter 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5267898700 1 Processor -- start 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5267898700 1 -- start start 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5267898700 1 librados: setting wanted keys 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5267898700 1 librados: calling monclient init 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5267898700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f524c003400 0x7f52600f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5267898700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52600f6bd0 0x7f52600f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5267898700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52601134d0 con 0x7f524c003400 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5267898700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5260113610 con 0x7f52600f6bd0 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5264e33700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52600f6bd0 0x7f52600f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5265634700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f524c003400 0x7f52600f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5264e33700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52600f6bd0 0x7f52600f7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:36716/0 (socket says 172.21.15.181:36716) 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5265634700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f524c003400 0x7f52600f66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.181:60462/0 (socket says 172.21.15.181:60462) 2024-02-13T00:33:59.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5264e33700 1 -- 172.21.15.181:0/2863760144 learned_addr learned my addr 172.21.15.181:0/2863760144 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:33:59.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5264e33700 1 -- 172.21.15.181:0/2863760144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52600f6bd0 msgr2=0x7f52600f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-02-13T00:33:59.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5264e33700 1 -- 172.21.15.181:0/2863760144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52600f6bd0 msgr2=0x7f52600f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-13T00:33:59.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5264e33700 1 --2- 172.21.15.181:0/2863760144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52600f6bd0 0x7f52600f7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-02-13T00:33:59.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.565+0000 7f5264e33700 1 --2- 172.21.15.181:0/2863760144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52600f6bd0 0x7f52600f7050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-02-13T00:33:59.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.566+0000 7f5265634700 1 -- 172.21.15.181:0/2863760144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52600f6bd0 msgr2=0x7f52600f7050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T00:33:59.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.566+0000 7f5265634700 1 --2- 172.21.15.181:0/2863760144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52600f6bd0 0x7f52600f7050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:59.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.566+0000 7f5265634700 1 -- 172.21.15.181:0/2863760144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c005040 con 0x7f524c003400 2024-02-13T00:33:59.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.566+0000 7f5265634700 1 --2- 172.21.15.181:0/2863760144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f524c003400 0x7f52600f66e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f5260060880 tx=0x7f525c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:59.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.566+0000 7f52567fc700 10 client.?.objecter ms_handle_connect 0x7f524c003400 2024-02-13T00:33:59.571 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.566+0000 7f52567fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:33:59.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.567+0000 7f52567fc700 1 -- 172.21.15.181:0/2863760144 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f525c004ac0 con 0x7f524c003400 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.567+0000 7f5267898700 1 -- 172.21.15.181:0/2863760144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52601137b0 con 0x7f524c003400 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.567+0000 7f52567fc700 1 -- 172.21.15.181:0/2863760144 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f525c004c20 con 0x7f524c003400 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.567+0000 7f52567fc700 1 -- 172.21.15.181:0/2863760144 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f525c0203c0 con 0x7f524c003400 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.567+0000 7f5267898700 10 client.14182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.567+0000 7f5267898700 1 -- 172.21.15.181:0/2863760144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5260113cc0 con 0x7f524c003400 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.567+0000 7f5267898700 1 librados: init done 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.568+0000 7f52567fc700 1 -- 172.21.15.181:0/2863760144 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7f525c020640 con 0x7f524c003400 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.568+0000 7f52567fc700 1 --2- 172.21.15.181:0/2863760144 >> v2:172.21.15.162:6800/1909610488 conn(0x7f52440341a0 0x7f5244036660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.568+0000 7f52567fc700 1 -- 172.21.15.181:0/2863760144 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f525c0453d0 con 0x7f524c003400 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.568+0000 7f52567fc700 10 client.14182.objecter ms_dispatch 0x7f5260001550 osd_map(4..4 src has 1..4) v4 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.568+0000 7f52567fc700 3 client.14182.objecter handle_osd_map got epochs [4,4] > 0 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.568+0000 7f52567fc700 3 client.14182.objecter handle_osd_map decoding full epoch 4 2024-02-13T00:33:59.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.568+0000 7f52567fc700 20 client.14182.objecter dump_active .. 0 homeless 2024-02-13T00:33:59.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.568+0000 7f5267898700 1 -- 172.21.15.181:0/2863760144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524c001f20 con 0x7f524c003400 2024-02-13T00:33:59.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.568+0000 7f5264e33700 1 --2- 172.21.15.181:0/2863760144 >> v2:172.21.15.162:6800/1909610488 conn(0x7f52440341a0 0x7f5244036660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:33:59.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.569+0000 7f5264e33700 1 --2- 172.21.15.181:0/2863760144 >> v2:172.21.15.162:6800/1909610488 conn(0x7f52440341a0 0x7f5244036660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f5250006fd0 tx=0x7f5250008040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:33:59.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.569+0000 7f52567fc700 10 client.14182.objecter ms_handle_connect 0x7f52440341a0 2024-02-13T00:33:59.577 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.574+0000 7f52567fc700 1 -- 172.21.15.181:0/2863760144 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f525c01b020 con 0x7f524c003400 2024-02-13T00:33:59.809 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.806+0000 7f5267898700 1 -- 172.21.15.181:0/2863760144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f524c002a00 con 0x7f524c003400 2024-02-13T00:33:59.810 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.807+0000 7f52567fc700 1 -- 172.21.15.181:0/2863760144 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+997 (secure 0 0 0) 0x7f525c026070 con 0x7f524c003400 2024-02-13T00:33:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:33:59.810 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"d4ee01fc-ca06-11ee-95b9-87774f69a715","modified":"2024-02-13T00:33:54.524798Z","created":"2024-02-13T00:31:48.749492Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-13T00:33:59.812 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 10 librados: watch_flush enter 2024-02-13T00:33:59.812 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 10 librados: watch_flush exit 2024-02-13T00:33:59.812 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 20 client.14182.objecter shutdown clearing up homeless session... 2024-02-13T00:33:59.812 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 10 client.14182.objecter successfully canceled tick 2024-02-13T00:33:59.812 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 1 -- 172.21.15.181:0/2863760144 >> v2:172.21.15.162:6800/1909610488 conn(0x7f52440341a0 msgr2=0x7f5244036660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:59.812 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 1 --2- 172.21.15.181:0/2863760144 >> v2:172.21.15.162:6800/1909610488 conn(0x7f52440341a0 0x7f5244036660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f5250006fd0 tx=0x7f5250008040).stop 2024-02-13T00:33:59.812 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 1 -- 172.21.15.181:0/2863760144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f524c003400 msgr2=0x7f52600f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:33:59.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 1 --2- 172.21.15.181:0/2863760144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f524c003400 0x7f52600f66e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f5260060880 tx=0x7f525c00b040).stop 2024-02-13T00:33:59.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 1 -- 172.21.15.181:0/2863760144 shutdown_connections 2024-02-13T00:33:59.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 1 --2- 172.21.15.181:0/2863760144 >> v2:172.21.15.162:6800/1909610488 conn(0x7f52440341a0 0x7f5244036660 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:59.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 1 --2- 172.21.15.181:0/2863760144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f524c003400 0x7f52600f66e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:59.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 1 --2- 172.21.15.181:0/2863760144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52600f6bd0 0x7f52600f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:33:59.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 1 -- 172.21.15.181:0/2863760144 >> 172.21.15.181:0/2863760144 conn(0x7f526006c4e0 msgr2=0x7f52600f9770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:33:59.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 1 -- 172.21.15.181:0/2863760144 shutdown_connections 2024-02-13T00:33:59.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.808+0000 7f5267898700 1 -- 172.21.15.181:0/2863760144 wait complete. 2024-02-13T00:33:59.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:33:59.809+0000 7f5267898700 1 librados: shutdown 2024-02-13T00:33:59.813 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2024-02-13T00:34:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:33:54.529935+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T00:34:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:33:54.530126+0000 mon.a (mon.0) 125 : cluster [INF] mon.a calling monitor election 2024-02-13T00:34:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:33:54.534003+0000 mon.a (mon.0) 126 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:33:55.484307+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:33:55.524935+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:33:56.524859+0000 mon.a (mon.0) 128 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:33:56.525620+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-02-13T00:34:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:33:57.484654+0000 mgr.y (mgr.14146) 41 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:00.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:33:57.525031+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:00.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:33:58.524921+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:00.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:33:59.525169+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:00.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:33:59.547319+0000 mon.a (mon.0) 132 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-13T00:34:00.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:33:59.550400+0000 mon.a (mon.0) 133 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} 2024-02-13T00:34:00.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:33:59.562354+0000 mon.a (mon.0) 134 : cluster [DBG] fsmap 2024-02-13T00:34:00.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:33:59.562385+0000 mon.a (mon.0) 135 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-13T00:34:00.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:33:59.562582+0000 mon.a (mon.0) 136 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-02-13T00:34:00.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:33:59.565897+0000 mon.a (mon.0) 137 : cluster [INF] overall HEALTH_OK 2024-02-13T00:34:00.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:33:59.568289+0000 mon.a (mon.0) 138 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:00.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:33:59.569273+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T00:34:00.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:33:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:33:59.570041+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:00.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:54.529935+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T00:34:00.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:54.530126+0000 mon.a (mon.0) 125 : cluster [INF] mon.a calling monitor election 2024-02-13T00:34:00.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:54.534003+0000 mon.a (mon.0) 126 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:55.484307+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:55.524935+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:56.524859+0000 mon.a (mon.0) 128 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:56.525620+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:57.484654+0000 mgr.y (mgr.14146) 41 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:57.525031+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:58.524921+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:59.525169+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:59.547319+0000 mon.a (mon.0) 132 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:59.550400+0000 mon.a (mon.0) 133 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:59.562354+0000 mon.a (mon.0) 134 : cluster [DBG] fsmap 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:59.562385+0000 mon.a (mon.0) 135 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-13T00:34:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:59.562582+0000 mon.a (mon.0) 136 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-02-13T00:34:00.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:59.565897+0000 mon.a (mon.0) 137 : cluster [INF] overall HEALTH_OK 2024-02-13T00:34:00.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:59.568289+0000 mon.a (mon.0) 138 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:00.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:59.569273+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T00:34:00.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:33:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:59.570041+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:00.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:33:59.484971+0000 mgr.y (mgr.14146) 42 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:00.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:33:59.575223+0000 mgr.y (mgr.14146) 43 : cephadm [INF] Deploying daemon mon.c on smithi162 2024-02-13T00:34:00.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:33:59.807739+0000 mon.a (mon.0) 141 : audit [DBG] from='client.? 172.21.15.181:0/2863760144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T00:34:00.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:00.525806+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:01.058 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:33:59.484971+0000 mgr.y (mgr.14146) 42 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:01.058 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:33:59.575223+0000 mgr.y (mgr.14146) 43 : cephadm [INF] Deploying daemon mon.c on smithi162 2024-02-13T00:34:01.058 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:33:59.807739+0000 mon.a (mon.0) 141 : audit [DBG] from='client.? 172.21.15.181:0/2863760144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T00:34:01.059 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:00.525806+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:01.411 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-13T00:34:01.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T00:34:02.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 systemd[1]: Starting Ceph mon.c for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:34:02.916 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:01.485292+0000 mgr.y (mgr.14146) 44 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:03.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:01.485292+0000 mgr.y (mgr.14146) 44 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:03.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.913+0000 7f7b729a8880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-13T00:34:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.913+0000 7f7b729a8880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 7 2024-02-13T00:34:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.913+0000 7f7b729a8880 0 pidfile_write: ignore empty --pid-file 2024-02-13T00:34:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.917+0000 7f7b729a8880 0 load: jerasure load: lrc load: isa 2024-02-13T00:34:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: RocksDB version: 6.8.1 2024-02-13T00:34:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-13T00:34:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Compile date Feb 8 2024 2024-02-13T00:34:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: DB SUMMARY 2024-02-13T00:34:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: CURRENT file: CURRENT 2024-02-13T00:34:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000003.log size: 636 ; 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.error_if_exists: 0 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.create_if_missing: 0 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.paranoid_checks: 1 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.env: 0x559b23ef2080 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.fs: Posix File System 2024-02-13T00:34:03.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.info_log: 0x559b26390260 2024-02-13T00:34:03.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-13T00:34:03.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.statistics: (nil) 2024-02-13T00:34:03.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.use_fsync: 0 2024-02-13T00:34:03.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_log_file_size: 0 2024-02-13T00:34:03.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-13T00:34:03.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-13T00:34:03.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-13T00:34:03.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-13T00:34:03.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.allow_fallocate: 1 2024-02-13T00:34:03.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-13T00:34:03.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-13T00:34:03.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.use_direct_reads: 0 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.db_log_dir: 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_subcompactions: 1 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_background_flushes: -1 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-13T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.write_buffer_manager: 0x559b270823c0 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.unordered_write: 0 2024-02-13T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.row_cache: None 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.wal_filter: None 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.preserve_deletes: 0 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.two_write_queues: 0 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.atomic_flush: 0 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-13T00:34:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.log_readahead_size: 0 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_background_jobs: 2 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_background_compactions: -1 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-13T00:34:03.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_open_files: -1 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Compression algorithms supported: 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: kZSTD supported: 0 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: kXpressCompression supported: 0 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: kLZ4Compression supported: 1 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: kBZip2Compression supported: 0 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: kZlibCompression supported: 1 2024-02-13T00:34:03.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: kSnappyCompression supported: 1 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.merge_operator: 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_filter: None 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559b26335d28) 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cache_index_and_filter_blocks: 1 2024-02-13T00:34:03.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: pin_top_level_index_and_filter: 1 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: index_type: 0 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: data_block_index_type: 0 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: index_shortening: 1 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: data_block_hash_table_util_ratio: 0.750000 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: hash_index_allow_collision: 1 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: checksum: 1 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: no_block_cache: 0 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: block_cache: 0x559b2636af10 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: block_cache_name: BinnedLRUCache 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: block_cache_options: 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: capacity : 536870912 2024-02-13T00:34:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: num_shard_bits : 4 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: strict_capacity_limit : 0 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: high_pri_pool_ratio: 0.000 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: block_cache_compressed: (nil) 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: persistent_cache: (nil) 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: block_size: 4096 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: block_size_deviation: 10 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: block_restart_interval: 16 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: index_block_restart_interval: 1 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: metadata_block_size: 4096 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: partition_filters: 0 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: use_delta_encoding: 1 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-13T00:34:03.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: whole_key_filtering: 1 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: verify_compression: 0 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: read_amp_bytes_per_bit: 0 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: format_version: 2 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: enable_index_compression: 1 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: block_align: 0 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compression: NoCompression 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.num_levels: 7 2024-02-13T00:34:03.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-13T00:34:03.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-13T00:34:03.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-13T00:34:03.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.table_properties_collectors: 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.inplace_update_support: 0 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.bloom_locality: 0 2024-02-13T00:34:03.346 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.max_successive_merges: 0 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.ttl: 2592000 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.918+0000 7f7b729a8880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.919+0000 7f7b729a8880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707784442919843, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.919+0000 7f7b729a8880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-02-13T00:34:03.347 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.919+0000 7f7b729a8880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-13T00:34:03.349 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.919+0000 7f7b729a8880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707784442920258, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1562, "table_properties": {"data_size": 648, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707784442, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-13T00:34:03.349 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.919+0000 7f7b729a8880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-02-13T00:34:03.349 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.349 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.919+0000 7f7b729a8880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707784442920621, "job": 1, "event": "recovery_finished"} 2024-02-13T00:34:03.349 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.920+0000 7f7b729a8880 4 rocksdb: DB pointer 0x559b26433800 2024-02-13T00:34:03.349 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.920+0000 7f7b59799700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-13T00:34:03.349 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.920+0000 7f7b59799700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-13T00:34:03.349 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: ** DB Stats ** 2024-02-13T00:34:03.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T00:34:03.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-13T00:34:03.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T00:34:03.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T00:34:03.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-13T00:34:03.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-13T00:34:03.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T00:34:03.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: ** Compaction Stats [default] ** 2024-02-13T00:34:03.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T00:34:03.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T00:34:03.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: ** Compaction Stats [default] ** 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: AddFile(Keys): cumulative 0, interval 0 2024-02-13T00:34:03.351 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: ** File Read Latency Histogram By Level [default] ** 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: ** Compaction Stats [default] ** 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.352 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: ** Compaction Stats [default] ** 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: AddFile(Keys): cumulative 0, interval 0 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T00:34:03.353 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.354 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: ** File Read Latency Histogram By Level [default] ** 2024-02-13T00:34:03.354 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.354 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.920+0000 7f7b729a8880 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-13T00:34:03.354 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.920+0000 7f7b729a8880 0 using public_addrv [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] 2024-02-13T00:34:03.354 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.921+0000 7f7b729a8880 0 starting mon.c rank -1 at public addrs [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] at bind addrs [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:34:03.354 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.922+0000 7f7b729a8880 1 mon.c@-1(???) e0 preinit fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:34:03.354 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 0 mon.c@-1(synchronizing).mds e1 new map 2024-02-13T00:34:03.354 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 0 mon.c@-1(synchronizing).mds e1 print_map 2024-02-13T00:34:03.354 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: e1 2024-02-13T00:34:03.354 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-13T00:34:03.354 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-13T00:34:03.354 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: legacy client fscid: -1 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: No filesystems configured 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b57795700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b617a9700 4 rocksdb: (Original Log Time 2024/02/13-00:34:02.938898) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b617a9700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b617a9700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707784442938948, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1101, "num_deletes": 14, "total_data_size": 3964727, "memory_usage": 3985480, "flush_reason": "Manual Compaction"} 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b617a9700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-13T00:34:03.355 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-13T00:34:03.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-13T00:34:03.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-13T00:34:03.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-13T00:34:03.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-13T00:34:03.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T00:34:03.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T00:34:03.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T00:34:03.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:01.485292+0000 mgr.y (mgr.14146) 44 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:03.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:34:02.938+0000 7f7b5bf9e700 1 mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-13T00:34:03.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 bash[46361]: f70507b1fef7c7fffc5ef0100c1a2601de24290484530d52e0e8de9555cab8b1 2024-02-13T00:34:03.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:02 smithi162 systemd[1]: Started Ceph mon.c for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:34:08.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:02.947633+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:02.947847+0000 mon.a (mon.0) 145 : cluster [INF] mon.a calling monitor election 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:02.950704+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:02.960715+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:02.960831+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:03.485652+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:03.942649+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:04.942377+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:04.944212+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:05.485951+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:05.942635+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:06.942551+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:07.942675+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:07.952722+0000 mon.a (mon.0) 153 : cluster [INF] mon.a calling monitor election 2024-02-13T00:34:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:07.977375+0000 mon.a (mon.0) 154 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-13T00:34:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:07.983061+0000 mon.a (mon.0) 155 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0]} removed_ranks: {} 2024-02-13T00:34:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:07.983584+0000 mon.a (mon.0) 156 : cluster [DBG] fsmap 2024-02-13T00:34:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:07.983613+0000 mon.a (mon.0) 157 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-13T00:34:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:07.983830+0000 mon.a (mon.0) 158 : cluster [DBG] mgrmap e13: y(active, since 80s) 2024-02-13T00:34:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:07.997723+0000 mon.a (mon.0) 159 : cluster [INF] overall HEALTH_OK 2024-02-13T00:34:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:08.003315+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:08.004618+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:08.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:02.947633+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T00:34:08.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:02.947847+0000 mon.a (mon.0) 145 : cluster [INF] mon.a calling monitor election 2024-02-13T00:34:08.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:02.950704+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-13T00:34:08.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:02.960715+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:02.960831+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:03.485652+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:03.942649+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:04.942377+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:04.944212+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-13T00:34:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:05.485951+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:05.942635+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:06.942551+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:07.942675+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:07.952722+0000 mon.a (mon.0) 153 : cluster [INF] mon.a calling monitor election 2024-02-13T00:34:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:07.977375+0000 mon.a (mon.0) 154 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-13T00:34:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:07.983061+0000 mon.a (mon.0) 155 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0]} removed_ranks: {} 2024-02-13T00:34:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:07.983584+0000 mon.a (mon.0) 156 : cluster [DBG] fsmap 2024-02-13T00:34:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:07.983613+0000 mon.a (mon.0) 157 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-13T00:34:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:07.983830+0000 mon.a (mon.0) 158 : cluster [DBG] mgrmap e13: y(active, since 80s) 2024-02-13T00:34:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:07.997723+0000 mon.a (mon.0) 159 : cluster [INF] overall HEALTH_OK 2024-02-13T00:34:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:08.003315+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:08.004618+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:08.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:02.947633+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T00:34:08.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:02.947847+0000 mon.a (mon.0) 145 : cluster [INF] mon.a calling monitor election 2024-02-13T00:34:08.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:02.950704+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-13T00:34:08.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:02.960715+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:34:08.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:02.960831+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:03.485652+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:08.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:03.942649+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:04.942377+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:04.944212+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-13T00:34:08.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:05.485951+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:08.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:05.942635+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:06.942551+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:07.942675+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:08.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:07.952722+0000 mon.a (mon.0) 153 : cluster [INF] mon.a calling monitor election 2024-02-13T00:34:08.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:07.977375+0000 mon.a (mon.0) 154 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-13T00:34:08.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:07.983061+0000 mon.a (mon.0) 155 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0]} removed_ranks: {} 2024-02-13T00:34:08.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:07.983584+0000 mon.a (mon.0) 156 : cluster [DBG] fsmap 2024-02-13T00:34:08.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:07.983613+0000 mon.a (mon.0) 157 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-13T00:34:08.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:07.983830+0000 mon.a (mon.0) 158 : cluster [DBG] mgrmap e13: y(active, since 80s) 2024-02-13T00:34:08.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:07.997723+0000 mon.a (mon.0) 159 : cluster [INF] overall HEALTH_OK 2024-02-13T00:34:08.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:08.003315+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:08.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:08.004618+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:09.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:07.486182+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:09.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:08.942825+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:09.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:09.037810+0000 mon.a (mon.0) 163 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:09.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:07.486182+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:09.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:08.942825+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:09.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:09.037810+0000 mon.a (mon.0) 163 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:09.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:07.486182+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:09.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:08.942825+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:34:09.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:09.037810+0000 mon.a (mon.0) 163 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:10.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:09.486478+0000 mgr.y (mgr.14146) 48 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:10.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:09.486478+0000 mgr.y (mgr.14146) 48 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:10.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:09.486478+0000 mgr.y (mgr.14146) 48 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:11.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.173+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/1743833237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc0fd6c0 msgr2=0x7ff1bc0fdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:11.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.173+0000 7ff1c0ce5700 1 --2- 172.21.15.181:0/1743833237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc0fd6c0 0x7ff1bc0fdaa0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff1ac004770 tx=0x7ff1ac009cb0).stop 2024-02-13T00:34:11.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.173+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/1743833237 shutdown_connections 2024-02-13T00:34:11.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.173+0000 7ff1c0ce5700 1 --2- 172.21.15.181:0/1743833237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc0fd6c0 0x7ff1bc0fdaa0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:11.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.173+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/1743833237 >> 172.21.15.181:0/1743833237 conn(0x7ff1bc0f32c0 msgr2=0x7ff1bc0f56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:34:11.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.173+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/1743833237 shutdown_connections 2024-02-13T00:34:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.173+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/1743833237 wait complete. 2024-02-13T00:34:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1c0ce5700 1 librados: starting msgr at 2024-02-13T00:34:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1c0ce5700 1 librados: starting objecter 2024-02-13T00:34:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1c0ce5700 1 Processor -- start 2024-02-13T00:34:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1c0ce5700 1 -- start start 2024-02-13T00:34:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1c0ce5700 1 librados: setting wanted keys 2024-02-13T00:34:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1c0ce5700 1 librados: calling monclient init 2024-02-13T00:34:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1c0ce5700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0fd6c0 0x7ff1bc06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1c0ce5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc06b660 0x7ff1bc06bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1c0ce5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc117dd0 0x7ff1bc11a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1c0ce5700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc10b430 con 0x7ff1bc06b660 2024-02-13T00:34:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1c0ce5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc11a6b0 con 0x7ff1bc117dd0 2024-02-13T00:34:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1c0ce5700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc11a820 con 0x7ff1bc0fd6c0 2024-02-13T00:34:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1bad9d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc117dd0 0x7ff1bc11a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1bad9d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc117dd0 0x7ff1bc11a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:59640/0 (socket says 172.21.15.181:59640) 2024-02-13T00:34:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1bad9d700 1 -- 172.21.15.181:0/2929513030 learned_addr learned my addr 172.21.15.181:0/2929513030 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:34:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.174+0000 7ff1b9d9b700 1 --2- 172.21.15.181:0/2929513030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc06b660 0x7ff1bc06bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1ba59c700 1 --2- 172.21.15.181:0/2929513030 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0fd6c0 0x7ff1bc06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1bad9d700 1 -- 172.21.15.181:0/2929513030 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0fd6c0 msgr2=0x7ff1bc06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1bad9d700 1 --2- 172.21.15.181:0/2929513030 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0fd6c0 0x7ff1bc06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1bad9d700 1 -- 172.21.15.181:0/2929513030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc06b660 msgr2=0x7ff1bc06bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1bad9d700 1 --2- 172.21.15.181:0/2929513030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc06b660 0x7ff1bc06bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1bad9d700 1 -- 172.21.15.181:0/2929513030 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac005040 con 0x7ff1bc117dd0 2024-02-13T00:34:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1b9d9b700 1 --2- 172.21.15.181:0/2929513030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc06b660 0x7ff1bc06bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:34:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1bad9d700 1 --2- 172.21.15.181:0/2929513030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc117dd0 0x7ff1bc11a1c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff1b400bc30 tx=0x7ff1b4009820).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:34:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1ab7fe700 10 client.?.objecter ms_handle_connect 0x7ff1bc117dd0 2024-02-13T00:34:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1ab7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:34:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1ab7fe700 1 -- 172.21.15.181:0/2929513030 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b400c9b0 con 0x7ff1bc117dd0 2024-02-13T00:34:11.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/2929513030 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1bc11ab00 con 0x7ff1bc117dd0 2024-02-13T00:34:11.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1ab7fe700 1 -- 172.21.15.181:0/2929513030 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1b4015420 con 0x7ff1bc117dd0 2024-02-13T00:34:11.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1ab7fe700 1 -- 172.21.15.181:0/2929513030 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b40073e0 con 0x7ff1bc117dd0 2024-02-13T00:34:11.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1c0ce5700 10 client.24100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:34:11.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/2929513030 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc11afa0 con 0x7ff1bc117dd0 2024-02-13T00:34:11.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.175+0000 7ff1c0ce5700 1 librados: init done 2024-02-13T00:34:11.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.176+0000 7ff1ab7fe700 1 -- 172.21.15.181:0/2929513030 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7ff1b4012020 con 0x7ff1bc117dd0 2024-02-13T00:34:11.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.176+0000 7ff1ab7fe700 1 --2- 172.21.15.181:0/2929513030 >> v2:172.21.15.162:6800/1909610488 conn(0x7ff1a40344f0 0x7ff1a40369b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:11.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.176+0000 7ff1ab7fe700 1 -- 172.21.15.181:0/2929513030 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7ff1b4044130 con 0x7ff1bc117dd0 2024-02-13T00:34:11.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.176+0000 7ff1ab7fe700 10 client.24100.objecter ms_dispatch 0x7ff1bc001550 osd_map(4..4 src has 1..4) v4 2024-02-13T00:34:11.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.176+0000 7ff1ab7fe700 3 client.24100.objecter handle_osd_map got epochs [4,4] > 0 2024-02-13T00:34:11.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.176+0000 7ff1ab7fe700 3 client.24100.objecter handle_osd_map decoding full epoch 4 2024-02-13T00:34:11.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.176+0000 7ff1ab7fe700 20 client.24100.objecter dump_active .. 0 homeless 2024-02-13T00:34:11.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.176+0000 7ff1ba59c700 1 --2- 172.21.15.181:0/2929513030 >> v2:172.21.15.162:6800/1909610488 conn(0x7ff1a40344f0 0x7ff1a40369b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:11.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.176+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/2929513030 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc05f1f0 con 0x7ff1bc117dd0 2024-02-13T00:34:11.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.177+0000 7ff1ab7fe700 10 client.24100.objecter ms_handle_connect 0x7ff1a40344f0 2024-02-13T00:34:11.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.177+0000 7ff1ba59c700 1 --2- 172.21.15.181:0/2929513030 >> v2:172.21.15.162:6800/1909610488 conn(0x7ff1a40344f0 0x7ff1a40369b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7ff1ac009f20 tx=0x7ff1ac00b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:34:11.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.181+0000 7ff1ab7fe700 1 -- 172.21.15.181:0/2929513030 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff1b4009890 con 0x7ff1bc117dd0 2024-02-13T00:34:11.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.421+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/2929513030 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1bc05a3b0 con 0x7ff1bc117dd0 2024-02-13T00:34:11.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.421+0000 7ff1ab7fe700 1 -- 172.21.15.181:0/2929513030 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1267 (secure 0 0 0) 0x7ff1b40191c0 con 0x7ff1bc117dd0 2024-02-13T00:34:11.424 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-02-13T00:34:11.424 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":3,"fsid":"d4ee01fc-ca06-11ee-95b9-87774f69a715","modified":"2024-02-13T00:34:02.943127Z","created":"2024-02-13T00:31:48.749492Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3301","nonce":0},{"type":"v1","addr":"172.21.15.162:6790","nonce":0}]},"addr":"172.21.15.162:6790/0","public_addr":"172.21.15.162:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-02-13T00:34:11.426 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 10 librados: watch_flush enter 2024-02-13T00:34:11.427 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 10 librados: watch_flush exit 2024-02-13T00:34:11.427 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 20 client.24100.objecter shutdown clearing up homeless session... 2024-02-13T00:34:11.427 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 10 client.24100.objecter successfully canceled tick 2024-02-13T00:34:11.427 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/2929513030 >> v2:172.21.15.162:6800/1909610488 conn(0x7ff1a40344f0 msgr2=0x7ff1a40369b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:11.427 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 --2- 172.21.15.181:0/2929513030 >> v2:172.21.15.162:6800/1909610488 conn(0x7ff1a40344f0 0x7ff1a40369b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7ff1ac009f20 tx=0x7ff1ac00b040).stop 2024-02-13T00:34:11.427 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/2929513030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc117dd0 msgr2=0x7ff1bc11a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:11.427 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 --2- 172.21.15.181:0/2929513030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc117dd0 0x7ff1bc11a1c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff1b400bc30 tx=0x7ff1b4009820).stop 2024-02-13T00:34:11.427 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/2929513030 shutdown_connections 2024-02-13T00:34:11.427 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 --2- 172.21.15.181:0/2929513030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc117dd0 0x7ff1bc11a1c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:11.427 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 --2- 172.21.15.181:0/2929513030 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0fd6c0 0x7ff1bc06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:11.427 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 --2- 172.21.15.181:0/2929513030 >> v2:172.21.15.162:6800/1909610488 conn(0x7ff1a40344f0 0x7ff1a40369b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:11.428 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 --2- 172.21.15.181:0/2929513030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc06b660 0x7ff1bc06bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:11.428 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/2929513030 >> 172.21.15.181:0/2929513030 conn(0x7ff1bc0f32c0 msgr2=0x7ff1bc0f3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:34:11.428 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/2929513030 shutdown_connections 2024-02-13T00:34:11.428 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 -- 172.21.15.181:0/2929513030 wait complete. 2024-02-13T00:34:11.428 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:11.423+0000 7ff1c0ce5700 1 librados: shutdown 2024-02-13T00:34:11.428 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 3 2024-02-13T00:34:11.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:11.422205+0000 mon.b (mon.1) 3 : audit [DBG] from='client.? 172.21.15.181:0/2929513030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T00:34:11.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:11.422205+0000 mon.b (mon.1) 3 : audit [DBG] from='client.? 172.21.15.181:0/2929513030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T00:34:11.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:11.422205+0000 mon.b (mon.1) 3 : audit [DBG] from='client.? 172.21.15.181:0/2929513030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T00:34:12.542 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-13T00:34:12.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph config generate-minimal-conf 2024-02-13T00:34:13.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:11.486871+0000 mgr.y (mgr.14146) 49 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:13.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:12.106294+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:12.107017+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:13.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:12.107864+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:13.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:12.108927+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-02-13T00:34:13.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:12.109235+0000 mgr.y (mgr.14146) 51 : cephadm [INF] Updating smithi181:/etc/ceph/ceph.conf 2024-02-13T00:34:13.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:12.133203+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:12.137597+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:12.143191+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:12.144677+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T00:34:13.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:12.145396+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T00:34:13.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:12.146043+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:13.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:12.615998+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.281 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:11.486871+0000 mgr.y (mgr.14146) 49 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:13.281 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:12.106294+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.281 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:12.107017+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:13.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:12.107864+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:13.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:12.108927+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-02-13T00:34:13.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:12.109235+0000 mgr.y (mgr.14146) 51 : cephadm [INF] Updating smithi181:/etc/ceph/ceph.conf 2024-02-13T00:34:13.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:12.133203+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:12.137597+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:12.143191+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:12.144677+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T00:34:13.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:12.145396+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T00:34:13.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:12.146043+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:13.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:12.615998+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:11.486871+0000 mgr.y (mgr.14146) 49 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:13.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:12.106294+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:12.107017+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:12.107864+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:12.108927+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-02-13T00:34:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:12.109235+0000 mgr.y (mgr.14146) 51 : cephadm [INF] Updating smithi181:/etc/ceph/ceph.conf 2024-02-13T00:34:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:12.133203+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:12.137597+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:12.143191+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:12.144677+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T00:34:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:12.145396+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T00:34:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:12.146043+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:12.615998+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:14.508 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:12.144382+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-13T00:34:14.508 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:12.146590+0000 mgr.y (mgr.14146) 53 : cephadm [INF] Reconfiguring daemon mon.a on smithi162 2024-02-13T00:34:14.509 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:13.817977+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:14.509 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:13.818625+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T00:34:14.509 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:13.819234+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T00:34:14.509 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:13.820075+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:14.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:12.144382+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-13T00:34:14.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:12.146590+0000 mgr.y (mgr.14146) 53 : cephadm [INF] Reconfiguring daemon mon.a on smithi162 2024-02-13T00:34:14.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:13.817977+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:14.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:13.818625+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T00:34:14.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:13.819234+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T00:34:14.510 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:13.820075+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:14.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:12.144382+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-13T00:34:14.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:12.146590+0000 mgr.y (mgr.14146) 53 : cephadm [INF] Reconfiguring daemon mon.a on smithi162 2024-02-13T00:34:14.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:13.817977+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:14.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:13.818625+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T00:34:14.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:13.819234+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T00:34:14.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:13.820075+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:15.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:13.487139+0000 mgr.y (mgr.14146) 54 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:15.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:13.818408+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-13T00:34:15.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:13.820675+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring daemon mon.c on smithi162 2024-02-13T00:34:15.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:13.487139+0000 mgr.y (mgr.14146) 54 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:15.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:13.818408+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-13T00:34:15.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:13.820675+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring daemon mon.c on smithi162 2024-02-13T00:34:15.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:13.487139+0000 mgr.y (mgr.14146) 54 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:15.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:13.818408+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-13T00:34:15.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:13.820675+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring daemon mon.c on smithi162 2024-02-13T00:34:15.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.213+0000 7f3715c9c700 1 -- 172.21.15.162:0/1915043786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37100fdbc0 msgr2=0x7f37100fdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:15.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.213+0000 7f3715c9c700 1 --2- 172.21.15.162:0/1915043786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37100fdbc0 0x7f37100fdfa0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f3700004870 tx=0x7f3700009cb0).stop 2024-02-13T00:34:15.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.213+0000 7f3715c9c700 1 -- 172.21.15.162:0/1915043786 shutdown_connections 2024-02-13T00:34:15.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.213+0000 7f3715c9c700 1 --2- 172.21.15.162:0/1915043786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37100f8430 0x7f3710104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:15.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.213+0000 7f3715c9c700 1 --2- 172.21.15.162:0/1915043786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37100fdbc0 0x7f37100fdfa0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:15.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.213+0000 7f3715c9c700 1 --2- 172.21.15.162:0/1915043786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37100fe520 0x7f37100f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:15.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.213+0000 7f3715c9c700 1 -- 172.21.15.162:0/1915043786 >> 172.21.15.162:0/1915043786 conn(0x7f37100f3810 msgr2=0x7f37100f5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:34:15.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.213+0000 7f3715c9c700 1 -- 172.21.15.162:0/1915043786 shutdown_connections 2024-02-13T00:34:15.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.213+0000 7f3715c9c700 1 -- 172.21.15.162:0/1915043786 wait complete. 2024-02-13T00:34:15.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.214+0000 7f3715c9c700 1 librados: starting msgr at 2024-02-13T00:34:15.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.214+0000 7f3715c9c700 1 librados: starting objecter 2024-02-13T00:34:15.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.214+0000 7f3715c9c700 1 Processor -- start 2024-02-13T00:34:15.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.214+0000 7f3715c9c700 1 -- start start 2024-02-13T00:34:15.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.214+0000 7f3715c9c700 1 librados: setting wanted keys 2024-02-13T00:34:15.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.214+0000 7f3715c9c700 1 librados: calling monclient init 2024-02-13T00:34:15.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.214+0000 7f3715c9c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37100f8430 0x7f371006b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:15.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.214+0000 7f3715c9c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37100fdbc0 0x7f371006b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:15.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.214+0000 7f3715c9c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37100fe520 0x7f371006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:15.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.214+0000 7f3715c9c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37101165e0 con 0x7f37100f8430 2024-02-13T00:34:15.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.214+0000 7f3715c9c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3710116750 con 0x7f37100fe520 2024-02-13T00:34:15.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.214+0000 7f3715c9c700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3710116910 con 0x7f37100fdbc0 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370f7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37100f8430 0x7f371006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370effd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37100fdbc0 0x7f371006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370f7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37100f8430 0x7f371006b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46448/0 (socket says 172.21.15.162:46448) 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370effd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37100fdbc0 0x7f371006b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:54476/0 (socket says 172.21.15.162:54476) 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370f7fe700 1 -- 172.21.15.162:0/772346263 learned_addr learned my addr 172.21.15.162:0/772346263 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370f7fe700 1 -- 172.21.15.162:0/772346263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37100fdbc0 msgr2=0x7f371006b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370f7fe700 1 --2- 172.21.15.162:0/772346263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37100fdbc0 0x7f371006b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370f7fe700 1 -- 172.21.15.162:0/772346263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37100fe520 msgr2=0x7f371006bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370f7fe700 1 --2- 172.21.15.162:0/772346263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37100fe520 0x7f371006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370f7fe700 1 -- 172.21.15.162:0/772346263 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3704005040 con 0x7f37100f8430 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370effd700 1 --2- 172.21.15.162:0/772346263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37100fdbc0 0x7f371006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370f7fe700 1 --2- 172.21.15.162:0/772346263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37100f8430 0x7f371006b170 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f37000043a0 tx=0x7f3700009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370cff9700 10 client.?.objecter ms_handle_connect 0x7f37100f8430 2024-02-13T00:34:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370cff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:34:15.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370cff9700 1 -- 172.21.15.162:0/772346263 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370000e030 con 0x7f37100f8430 2024-02-13T00:34:15.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370cff9700 1 -- 172.21.15.162:0/772346263 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3700013da0 con 0x7f37100f8430 2024-02-13T00:34:15.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f3715c9c700 1 -- 172.21.15.162:0/772346263 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3700005040 con 0x7f37100f8430 2024-02-13T00:34:15.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f370cff9700 1 -- 172.21.15.162:0/772346263 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700018070 con 0x7f37100f8430 2024-02-13T00:34:15.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f3715c9c700 10 client.14193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:34:15.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f3715c9c700 1 -- 172.21.15.162:0/772346263 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3710116e90 con 0x7f37100f8430 2024-02-13T00:34:15.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.215+0000 7f3715c9c700 1 librados: init done 2024-02-13T00:34:15.722 INFO:teuthology.orchestra.run.smithi162.stdout:# minimal ceph.conf for d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:34:15.723 INFO:teuthology.orchestra.run.smithi162.stdout:[global] 2024-02-13T00:34:15.723 INFO:teuthology.orchestra.run.smithi162.stdout: fsid = d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:34:15.723 INFO:teuthology.orchestra.run.smithi162.stdout: mon_host = [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] 2024-02-13T00:34:15.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.216+0000 7f370cff9700 1 -- 172.21.15.162:0/772346263 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7f3700027440 con 0x7f37100f8430 2024-02-13T00:34:15.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.216+0000 7f370cff9700 1 --2- 172.21.15.162:0/772346263 >> v2:172.21.15.162:6800/1909610488 conn(0x7f36f8034230 0x7f36f80366f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:15.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.216+0000 7f370cff9700 1 -- 172.21.15.162:0/772346263 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f3700044f10 con 0x7f37100f8430 2024-02-13T00:34:15.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.216+0000 7f370cff9700 10 client.14193.objecter ms_dispatch 0x7f3710001550 osd_map(4..4 src has 1..4) v4 2024-02-13T00:34:15.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.216+0000 7f370cff9700 3 client.14193.objecter handle_osd_map got epochs [4,4] > 0 2024-02-13T00:34:15.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.216+0000 7f370cff9700 3 client.14193.objecter handle_osd_map decoding full epoch 4 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.216+0000 7f3715c9c700 1 -- 172.21.15.162:0/772346263 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3710060ba0 con 0x7f37100f8430 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.216+0000 7f370cff9700 20 client.14193.objecter dump_active .. 0 homeless 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.216+0000 7f370effd700 1 --2- 172.21.15.162:0/772346263 >> v2:172.21.15.162:6800/1909610488 conn(0x7f36f8034230 0x7f36f80366f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.217+0000 7f370cff9700 10 client.14193.objecter ms_handle_connect 0x7f36f8034230 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.217+0000 7f370effd700 1 --2- 172.21.15.162:0/772346263 >> v2:172.21.15.162:6800/1909610488 conn(0x7f36f8034230 0x7f36f80366f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f371006c910 tx=0x7f370400b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.221+0000 7f370cff9700 1 -- 172.21.15.162:0/772346263 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f370001b940 con 0x7f37100f8430 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.388+0000 7f3715c9c700 1 -- 172.21.15.162:0/772346263 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f371006c650 con 0x7f37100f8430 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.388+0000 7f370cff9700 1 -- 172.21.15.162:0/772346263 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f370000f270 con 0x7f37100f8430 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 10 librados: watch_flush enter 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 10 librados: watch_flush exit 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 20 client.14193.objecter shutdown clearing up homeless session... 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 10 client.14193.objecter successfully canceled tick 2024-02-13T00:34:15.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 -- 172.21.15.162:0/772346263 >> v2:172.21.15.162:6800/1909610488 conn(0x7f36f8034230 msgr2=0x7f36f80366f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:15.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 --2- 172.21.15.162:0/772346263 >> v2:172.21.15.162:6800/1909610488 conn(0x7f36f8034230 0x7f36f80366f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f371006c910 tx=0x7f370400b040).stop 2024-02-13T00:34:15.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 -- 172.21.15.162:0/772346263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37100f8430 msgr2=0x7f371006b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:15.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 --2- 172.21.15.162:0/772346263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37100f8430 0x7f371006b170 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f37000043a0 tx=0x7f3700009cb0).stop 2024-02-13T00:34:15.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 -- 172.21.15.162:0/772346263 shutdown_connections 2024-02-13T00:34:15.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 --2- 172.21.15.162:0/772346263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37100fe520 0x7f371006bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:34:15.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 --2- 172.21.15.162:0/772346263 >> v2:172.21.15.162:6800/1909610488 conn(0x7f36f8034230 0x7f36f80366f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:15.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 --2- 172.21.15.162:0/772346263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37100f8430 0x7f371006b170 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:15.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 --2- 172.21.15.162:0/772346263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37100fdbc0 0x7f371006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:15.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 -- 172.21.15.162:0/772346263 >> 172.21.15.162:0/772346263 conn(0x7f37100f3810 msgr2=0x7f37100fb950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:34:15.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 -- 172.21.15.162:0/772346263 shutdown_connections 2024-02-13T00:34:15.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 -- 172.21.15.162:0/772346263 wait complete. 2024-02-13T00:34:15.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:15.390+0000 7f3715c9c700 1 librados: shutdown 2024-02-13T00:34:16.238 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:15.390513+0000 mon.a (mon.0) 178 : audit [DBG] from='client.? 172.21.15.162:0/772346263' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:16.238 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:15.487391+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:16.239 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:15.390513+0000 mon.a (mon.0) 178 : audit [DBG] from='client.? 172.21.15.162:0/772346263' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:16.239 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:15.487391+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:16.240 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-13T00:34:16.240 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:34:16.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.conf 2024-02-13T00:34:16.273 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:34:16.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-13T00:34:16.342 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-13T00:34:16.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.conf 2024-02-13T00:34:16.374 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:15.390513+0000 mon.a (mon.0) 178 : audit [DBG] from='client.? 172.21.15.162:0/772346263' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:16.374 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:15.487391+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:16.379 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-13T00:34:16.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-13T00:34:16.450 INFO:tasks.cephadm:Adding mgr.y on smithi162 2024-02-13T00:34:16.450 INFO:tasks.cephadm:Adding mgr.x on smithi181 2024-02-13T00:34:16.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch apply mgr '2;smithi162=y;smithi181=x' 2024-02-13T00:34:18.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:16.679096+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:18.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:16.679543+0000 mgr.y (mgr.14146) 58 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-13T00:34:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:16.679745+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T00:34:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:16.680317+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T00:34:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:16.680855+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:16.681314+0000 mgr.y (mgr.14146) 59 : cephadm [INF] Reconfiguring daemon mon.b on smithi181 2024-02-13T00:34:18.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:16.679096+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:18.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:16.679543+0000 mgr.y (mgr.14146) 58 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-13T00:34:18.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:16.679745+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T00:34:18.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:16.680317+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T00:34:18.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:16.680855+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:18.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:16.681314+0000 mgr.y (mgr.14146) 59 : cephadm [INF] Reconfiguring daemon mon.b on smithi181 2024-02-13T00:34:18.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:16.679096+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:18.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:16.679543+0000 mgr.y (mgr.14146) 58 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-13T00:34:18.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:16.679745+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T00:34:18.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:16.680317+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T00:34:18.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:16.680855+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:18.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:16.681314+0000 mgr.y (mgr.14146) 59 : cephadm [INF] Reconfiguring daemon mon.b on smithi181 2024-02-13T00:34:18.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:17.487771+0000 mgr.y (mgr.14146) 60 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:18.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:18.520205+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:18.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:18.521081+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:18.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:18.522651+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:18.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:18.523380+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:18.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:18.527711+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:19.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:17.487771+0000 mgr.y (mgr.14146) 60 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:19.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:18.520205+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:19.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:18.521081+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:19.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:18.522651+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:19.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:18.523380+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:19.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:18.527711+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:17.487771+0000 mgr.y (mgr.14146) 60 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:18.520205+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:18.521081+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:18.522651+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:18.523380+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:18.527711+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:19.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.301+0000 7f0b94fe2700 1 -- 172.21.15.181:0/1620445536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b900f67a0 msgr2=0x7f0b900f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:19.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.301+0000 7f0b94fe2700 1 --2- 172.21.15.181:0/1620445536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b900f67a0 0x7f0b900f8f20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f0b80004870 tx=0x7f0b80009cb0).stop 2024-02-13T00:34:19.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.301+0000 7f0b94fe2700 1 -- 172.21.15.181:0/1620445536 shutdown_connections 2024-02-13T00:34:19.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.301+0000 7f0b94fe2700 1 --2- 172.21.15.181:0/1620445536 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b900f9540 0x7f0b900f99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:19.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.301+0000 7f0b94fe2700 1 --2- 172.21.15.181:0/1620445536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b900f5e00 0x7f0b900f6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.301+0000 7f0b94fe2700 1 --2- 172.21.15.181:0/1620445536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b900f67a0 0x7f0b900f8f20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.301+0000 7f0b94fe2700 1 -- 172.21.15.181:0/1620445536 >> 172.21.15.181:0/1620445536 conn(0x7f0b900f19a0 msgr2=0x7f0b900f3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.301+0000 7f0b94fe2700 1 -- 172.21.15.181:0/1620445536 shutdown_connections 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.301+0000 7f0b94fe2700 1 -- 172.21.15.181:0/1620445536 wait complete. 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b94fe2700 1 librados: starting msgr at 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b94fe2700 1 librados: starting objecter 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b94fe2700 1 Processor -- start 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b94fe2700 1 -- start start 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b94fe2700 1 librados: setting wanted keys 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b94fe2700 1 librados: calling monclient init 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b94fe2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b900f5e00 0x7f0b901081a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b94fe2700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b900f67a0 0x7f0b90108690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b94fe2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b900f9540 0x7f0b9010f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:19.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b94fe2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9010fc60 con 0x7f0b900f5e00 2024-02-13T00:34:19.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b94fe2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9010fda0 con 0x7f0b900f9540 2024-02-13T00:34:19.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b94fe2700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9010fee0 con 0x7f0b900f67a0 2024-02-13T00:34:19.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b8e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b900f5e00 0x7f0b901081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:19.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b8ed9d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b900f9540 0x7f0b9010f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:19.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b8e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b900f5e00 0x7f0b901081a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.181:49994/0 (socket says 172.21.15.181:49994) 2024-02-13T00:34:19.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b8ed9d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b900f9540 0x7f0b9010f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:54100/0 (socket says 172.21.15.181:54100) 2024-02-13T00:34:19.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b8e59c700 1 -- 172.21.15.181:0/3855058238 learned_addr learned my addr 172.21.15.181:0/3855058238 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:34:19.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.302+0000 7f0b8dd9b700 1 --2- 172.21.15.181:0/3855058238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b900f67a0 0x7f0b90108690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:19.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b8ed9d700 1 -- 172.21.15.181:0/3855058238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b900f67a0 msgr2=0x7f0b90108690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:19.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b8ed9d700 1 --2- 172.21.15.181:0/3855058238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b900f67a0 0x7f0b90108690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:19.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b8ed9d700 1 -- 172.21.15.181:0/3855058238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b900f5e00 msgr2=0x7f0b901081a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:19.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b8ed9d700 1 --2- 172.21.15.181:0/3855058238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b900f5e00 0x7f0b901081a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:19.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b8ed9d700 1 -- 172.21.15.181:0/3855058238 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b80005040 con 0x7f0b900f9540 2024-02-13T00:34:19.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b8ed9d700 1 --2- 172.21.15.181:0/3855058238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b900f9540 0x7f0b9010f770 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0b8800d8a0 tx=0x7f0b88004db0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:34:19.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b7f7fe700 10 client.?.objecter ms_handle_connect 0x7f0b900f9540 2024-02-13T00:34:19.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b7f7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:34:19.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b7f7fe700 1 -- 172.21.15.181:0/3855058238 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88004020 con 0x7f0b900f9540 2024-02-13T00:34:19.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b7f7fe700 1 -- 172.21.15.181:0/3855058238 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b8800f040 con 0x7f0b900f9540 2024-02-13T00:34:19.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b7f7fe700 1 -- 172.21.15.181:0/3855058238 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b880036c0 con 0x7f0b900f9540 2024-02-13T00:34:19.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b94fe2700 1 -- 172.21.15.181:0/3855058238 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b901101c0 con 0x7f0b900f9540 2024-02-13T00:34:19.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b94fe2700 10 client.24109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:34:19.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b94fe2700 1 -- 172.21.15.181:0/3855058238 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b901106e0 con 0x7f0b900f9540 2024-02-13T00:34:19.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.303+0000 7f0b94fe2700 1 librados: init done 2024-02-13T00:34:19.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.304+0000 7f0b7f7fe700 1 -- 172.21.15.181:0/3855058238 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7f0b88003af0 con 0x7f0b900f9540 2024-02-13T00:34:19.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.304+0000 7f0b7f7fe700 1 --2- 172.21.15.181:0/3855058238 >> v2:172.21.15.162:6800/1909610488 conn(0x7f0b78034230 0x7f0b780366f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:19.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.304+0000 7f0b7f7fe700 1 -- 172.21.15.181:0/3855058238 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f0b880427a0 con 0x7f0b900f9540 2024-02-13T00:34:19.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.304+0000 7f0b7f7fe700 10 client.24109.objecter ms_dispatch 0x7f0b90001550 osd_map(4..4 src has 1..4) v4 2024-02-13T00:34:19.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.304+0000 7f0b7f7fe700 3 client.24109.objecter handle_osd_map got epochs [4,4] > 0 2024-02-13T00:34:19.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.304+0000 7f0b7f7fe700 3 client.24109.objecter handle_osd_map decoding full epoch 4 2024-02-13T00:34:19.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.304+0000 7f0b94fe2700 1 -- 172.21.15.181:0/3855058238 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b90102a30 con 0x7f0b900f9540 2024-02-13T00:34:19.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.304+0000 7f0b7f7fe700 20 client.24109.objecter dump_active .. 0 homeless 2024-02-13T00:34:19.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.304+0000 7f0b8e59c700 1 --2- 172.21.15.181:0/3855058238 >> v2:172.21.15.162:6800/1909610488 conn(0x7f0b78034230 0x7f0b780366f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:19.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.304+0000 7f0b7f7fe700 10 client.24109.objecter ms_handle_connect 0x7f0b78034230 2024-02-13T00:34:19.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.304+0000 7f0b8e59c700 1 --2- 172.21.15.181:0/3855058238 >> v2:172.21.15.162:6800/1909610488 conn(0x7f0b78034230 0x7f0b780366f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f0b84009fd0 tx=0x7f0b8400b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:34:19.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.308+0000 7f0b7f7fe700 1 -- 172.21.15.181:0/3855058238 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0b88011d30 con 0x7f0b900f9540 2024-02-13T00:34:19.483 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.480+0000 7f0b94fe2700 1 -- 172.21.15.181:0/3855058238 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi162=y;smithi181=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b900febc0 con 0x7f0b78034230 2024-02-13T00:34:19.489 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.486+0000 7f0b7f7fe700 1 -- 172.21.15.181:0/3855058238 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0b900febc0 con 0x7f0b78034230 2024-02-13T00:34:19.489 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mgr update... 2024-02-13T00:34:19.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.487+0000 7f0b94fe2700 10 librados: watch_flush enter 2024-02-13T00:34:19.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 10 librados: watch_flush exit 2024-02-13T00:34:19.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 20 client.24109.objecter shutdown clearing up homeless session... 2024-02-13T00:34:19.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 10 client.24109.objecter successfully canceled tick 2024-02-13T00:34:19.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 -- 172.21.15.181:0/3855058238 >> v2:172.21.15.162:6800/1909610488 conn(0x7f0b78034230 msgr2=0x7f0b780366f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:19.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 --2- 172.21.15.181:0/3855058238 >> v2:172.21.15.162:6800/1909610488 conn(0x7f0b78034230 0x7f0b780366f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f0b84009fd0 tx=0x7f0b8400b040).stop 2024-02-13T00:34:19.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 -- 172.21.15.181:0/3855058238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b900f9540 msgr2=0x7f0b9010f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:19.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 --2- 172.21.15.181:0/3855058238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b900f9540 0x7f0b9010f770 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0b8800d8a0 tx=0x7f0b88004db0).stop 2024-02-13T00:34:19.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 -- 172.21.15.181:0/3855058238 shutdown_connections 2024-02-13T00:34:19.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 --2- 172.21.15.181:0/3855058238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b900f9540 0x7f0b9010f770 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:19.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 --2- 172.21.15.181:0/3855058238 >> v2:172.21.15.162:6800/1909610488 conn(0x7f0b78034230 0x7f0b780366f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:19.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 --2- 172.21.15.181:0/3855058238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b900f5e00 0x7f0b901081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:19.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 --2- 172.21.15.181:0/3855058238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b900f67a0 0x7f0b90108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:19.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 -- 172.21.15.181:0/3855058238 >> 172.21.15.181:0/3855058238 conn(0x7f0b900f19a0 msgr2=0x7f0b900f3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:34:19.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 -- 172.21.15.181:0/3855058238 shutdown_connections 2024-02-13T00:34:19.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 -- 172.21.15.181:0/3855058238 wait complete. 2024-02-13T00:34:19.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:34:19.488+0000 7f0b94fe2700 1 librados: shutdown 2024-02-13T00:34:20.162 DEBUG:teuthology.orchestra.run.smithi181:mgr.x> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mgr.x.service 2024-02-13T00:34:20.165 INFO:tasks.cephadm:Deploying OSDs... 2024-02-13T00:34:20.165 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:34:20.165 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T00:34:20.188 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-13T00:34:20.189 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-02-13T00:34:20.196 INFO:journalctl@ceph.mgr.x.smithi181.stdout:-- Logs begin at Tue 2024-02-13 00:17:30 UTC. -- 2024-02-13T00:34:20.249 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-13T00:34:20.249 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T00:34:20.249 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 167249 Links: 1 2024-02-13T00:34:20.250 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T00:34:20.250 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T00:34:20.250 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-02-13 00:34:02.346189883 +0000 2024-02-13T00:34:20.250 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-02-13 00:24:01.470111658 +0000 2024-02-13T00:34:20.250 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-02-13 00:24:01.470111658 +0000 2024-02-13T00:34:20.250 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-02-13T00:34:20.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-13T00:34:20.319 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-02-13T00:34:20.319 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-02-13T00:34:20.319 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.00024006 s, 2.1 MB/s 2024-02-13T00:34:20.320 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-13T00:34:20.381 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-02-13T00:34:20.440 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-13T00:34:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T00:34:20.441 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 158537 Links: 1 2024-02-13T00:34:20.441 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T00:34:20.441 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T00:34:20.441 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-02-13 00:34:02.346189883 +0000 2024-02-13T00:34:20.441 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-02-13 00:24:01.819104822 +0000 2024-02-13T00:34:20.441 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-02-13 00:24:01.819104822 +0000 2024-02-13T00:34:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-02-13T00:34:20.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-13T00:34:20.509 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-02-13T00:34:20.509 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-02-13T00:34:20.509 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000233564 s, 2.2 MB/s 2024-02-13T00:34:20.510 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-13T00:34:20.570 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-02-13T00:34:20.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:19.482026+0000 mgr.y (mgr.14146) 61 : audit [DBG] from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi162=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:34:20.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:19.483422+0000 mgr.y (mgr.14146) 62 : cephadm [INF] Saving service mgr spec with placement smithi162=y;smithi181=x;count:2 2024-02-13T00:34:20.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:19.486752+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:20.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:19.488011+0000 mgr.y (mgr.14146) 63 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:20.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:19.495450+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:20.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:19.497031+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:20.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:19.497882+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:20.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:19.502484+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:20.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:19.504153+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T00:34:20.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:19.506833+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-13T00:34:20.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:19.507545+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T00:34:20.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:19.508366+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:20.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:19.509011+0000 mgr.y (mgr.14146) 64 : cephadm [INF] Deploying daemon mgr.x on smithi181 2024-02-13T00:34:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:19.482026+0000 mgr.y (mgr.14146) 61 : audit [DBG] from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi162=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:34:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:19.483422+0000 mgr.y (mgr.14146) 62 : cephadm [INF] Saving service mgr spec with placement smithi162=y;smithi181=x;count:2 2024-02-13T00:34:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:19.486752+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:19.488011+0000 mgr.y (mgr.14146) 63 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:19.495450+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:19.497031+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:19.497882+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:19.502484+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:19.504153+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T00:34:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:19.506833+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-13T00:34:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:19.507545+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T00:34:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:19.508366+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:19.509011+0000 mgr.y (mgr.14146) 64 : cephadm [INF] Deploying daemon mgr.x on smithi181 2024-02-13T00:34:20.599 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-13T00:34:20.599 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T00:34:20.599 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 166733 Links: 1 2024-02-13T00:34:20.599 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T00:34:20.599 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T00:34:20.599 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-02-13 00:34:02.346189883 +0000 2024-02-13T00:34:20.599 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-02-13 00:24:02.160098142 +0000 2024-02-13T00:34:20.599 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-02-13 00:24:02.160098142 +0000 2024-02-13T00:34:20.599 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-02-13T00:34:20.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-13T00:34:20.661 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:19.482026+0000 mgr.y (mgr.14146) 61 : audit [DBG] from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi162=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:34:20.661 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:19.483422+0000 mgr.y (mgr.14146) 62 : cephadm [INF] Saving service mgr spec with placement smithi162=y;smithi181=x;count:2 2024-02-13T00:34:20.662 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:19.486752+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:20.662 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:19.488011+0000 mgr.y (mgr.14146) 63 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:20.662 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:19.495450+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:20.662 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:19.497031+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:20.662 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:19.497882+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:20.662 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:19.502484+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:20.662 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:19.504153+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T00:34:20.662 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:19.506833+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-13T00:34:20.662 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:19.507545+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T00:34:20.662 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:19.508366+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:20.663 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:19.509011+0000 mgr.y (mgr.14146) 64 : cephadm [INF] Deploying daemon mgr.x on smithi181 2024-02-13T00:34:20.664 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-02-13T00:34:20.664 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-02-13T00:34:20.664 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000204084 s, 2.5 MB/s 2024-02-13T00:34:20.665 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-13T00:34:20.724 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-02-13T00:34:20.782 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-13T00:34:20.783 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T00:34:20.783 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 169288 Links: 1 2024-02-13T00:34:20.783 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T00:34:20.783 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T00:34:20.783 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-02-13 00:34:02.346189883 +0000 2024-02-13T00:34:20.783 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-02-13 00:24:02.498091521 +0000 2024-02-13T00:34:20.783 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-02-13 00:24:02.498091521 +0000 2024-02-13T00:34:20.783 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-02-13T00:34:20.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-13T00:34:20.848 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-02-13T00:34:20.848 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-02-13T00:34:20.848 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000233853 s, 2.2 MB/s 2024-02-13T00:34:20.849 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-13T00:34:20.908 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-13T00:34:20.908 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T00:34:20.927 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-13T00:34:20.927 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-02-13T00:34:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-13T00:34:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T00:34:20.988 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 162580 Links: 1 2024-02-13T00:34:20.988 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T00:34:20.988 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T00:34:20.988 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-02-13 00:33:53.957741692 +0000 2024-02-13T00:34:20.988 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-02-13 00:23:59.208345708 +0000 2024-02-13T00:34:20.988 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-02-13 00:23:59.208345708 +0000 2024-02-13T00:34:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-02-13T00:34:20.989 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-13T00:34:21.055 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-02-13T00:34:21.055 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-02-13T00:34:21.056 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000337748 s, 1.5 MB/s 2024-02-13T00:34:21.057 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-13T00:34:21.116 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-02-13T00:34:21.174 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-13T00:34:21.174 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T00:34:21.174 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 168242 Links: 1 2024-02-13T00:34:21.174 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T00:34:21.174 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T00:34:21.175 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-02-13 00:33:53.957741692 +0000 2024-02-13T00:34:21.175 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-02-13 00:23:59.562339023 +0000 2024-02-13T00:34:21.175 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-02-13 00:23:59.562339023 +0000 2024-02-13T00:34:21.175 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-02-13T00:34:21.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-13T00:34:21.240 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-02-13T00:34:21.240 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-02-13T00:34:21.240 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000308671 s, 1.7 MB/s 2024-02-13T00:34:21.241 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-13T00:34:21.301 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-02-13T00:34:21.359 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-13T00:34:21.359 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T00:34:21.359 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 166294 Links: 1 2024-02-13T00:34:21.359 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T00:34:21.359 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T00:34:21.359 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-02-13 00:33:53.957741692 +0000 2024-02-13T00:34:21.359 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-02-13 00:23:59.888332867 +0000 2024-02-13T00:34:21.359 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-02-13 00:23:59.888332867 +0000 2024-02-13T00:34:21.359 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-02-13T00:34:21.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-13T00:34:21.425 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-02-13T00:34:21.425 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-02-13T00:34:21.425 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000166709 s, 3.1 MB/s 2024-02-13T00:34:21.426 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-13T00:34:21.486 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-02-13T00:34:21.545 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-13T00:34:21.545 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T00:34:21.545 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 166353 Links: 1 2024-02-13T00:34:21.545 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T00:34:21.545 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T00:34:21.545 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-02-13 00:34:21.479202871 +0000 2024-02-13T00:34:21.545 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-02-13 00:24:00.234326333 +0000 2024-02-13T00:34:21.545 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-02-13 00:24:00.234326333 +0000 2024-02-13T00:34:21.545 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-02-13T00:34:21.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-13T00:34:21.658 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-02-13T00:34:21.658 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-02-13T00:34:21.658 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000198412 s, 2.6 MB/s 2024-02-13T00:34:21.659 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-13T00:34:21.677 INFO:tasks.cephadm:Deploying osd.0 on smithi162 with /dev/vg_nvme/lv_4... 2024-02-13T00:34:21.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-13T00:34:21.794 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:21 smithi181 systemd[1]: Starting Ceph mgr.x for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:34:22.496 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:22.205+0000 7f7a9547d700 1 -- 172.21.15.181:0/1457972102 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55944ac324e0 con 0x55944ac06c00 2024-02-13T00:34:22.496 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:22 smithi181 bash[42359]: 550ae696a07528877e02b8c43b87794a4af9e8f62b661435b054a5e3205e4332 2024-02-13T00:34:22.496 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:22 smithi181 systemd[1]: Started Ceph mgr.x for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:34:22.496 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:22.336+0000 7f7aa5a79680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T00:34:22.497 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:22.421+0000 7f7aa5a79680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T00:34:22.746 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:21.488284+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:22.746 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:22.311094+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:22.746 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:22.312520+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:22.746 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:21.488284+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:22.747 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:22.311094+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:22.747 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:22.312520+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:22.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:21.488284+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:22.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:22.311094+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:22.312520+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:23.294 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:22.820+0000 7f7aa5a79680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T00:34:23.816 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-13T00:34:23.816 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-13T00:34:23.816 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-02-13T00:34:23.816 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-02-13T00:34:23.816 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124356 s, 843 MB/s 2024-02-13T00:34:23.817 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-02-13T00:34:23.886 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:23.590+0000 7f7aa5a79680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T00:34:23.887 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:23.674+0000 7f7aa5a79680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T00:34:23.887 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:23.883+0000 7f7aa5a79680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T00:34:23.887 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:22.622192+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:23.887 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:23.296791+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:23.887 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:23.297294+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:23.887 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:23.297884+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:23.887 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:23.303752+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:23.888 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:23.305215+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T00:34:23.888 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:23.305875+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T00:34:23.888 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:23.306402+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:24.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:22.622192+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:24.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:23.296791+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:24.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:23.297294+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:24.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:23.297884+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:24.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:23.303752+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:24.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:23.305215+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T00:34:24.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:23.305875+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T00:34:24.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:23.306402+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:22.622192+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:23.296791+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:23.297294+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:23.297884+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:23.303752+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:23.305215+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T00:34:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:23.305875+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T00:34:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:23.306402+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:24.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:23.304986+0000 mgr.y (mgr.14146) 66 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-13T00:34:24.781 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:23.306867+0000 mgr.y (mgr.14146) 67 : cephadm [INF] Reconfiguring daemon mgr.y on smithi162 2024-02-13T00:34:24.781 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:23.488531+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:24.781 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:23.304986+0000 mgr.y (mgr.14146) 66 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-13T00:34:24.781 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:23.306867+0000 mgr.y (mgr.14146) 67 : cephadm [INF] Reconfiguring daemon mgr.y on smithi162 2024-02-13T00:34:24.781 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:23.488531+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:24.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_4 2024-02-13T00:34:24.951 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:23.304986+0000 mgr.y (mgr.14146) 66 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-13T00:34:24.951 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:23.306867+0000 mgr.y (mgr.14146) 67 : cephadm [INF] Reconfiguring daemon mgr.y on smithi162 2024-02-13T00:34:24.951 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:23.488531+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:24.951 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:24.822+0000 7f7aa5a79680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T00:34:25.294 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:24.947+0000 7f7aa5a79680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T00:34:25.294 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:25.025+0000 7f7aa5a79680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T00:34:25.294 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:25.113+0000 7f7aa5a79680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T00:34:25.585 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:25.308+0000 7f7aa5a79680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T00:34:26.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:25.581+0000 7f7aa5a79680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T00:34:26.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:25.679+0000 7f7aa5a79680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T00:34:26.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:25.274500+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:26.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:25.275460+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:26.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:25.276626+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:26.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:25.277169+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:26.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:25.281134+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:26.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:25.488830+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:25.274500+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:25.275460+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:25.276626+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:25.277169+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:25.281134+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:25.488830+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:26.544 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:26.398+0000 7f7aa5a79680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T00:34:26.544 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:26.477+0000 7f7aa5a79680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T00:34:26.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:25.274500+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:26.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:25.275460+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:26.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:25.276626+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:26.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:25.277169+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:26.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:25.281134+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:26.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:25.488830+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:26.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.842+0000 7efcce302700 1 -- 172.21.15.162:0/1624912375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcc80f6a30 msgr2=0x7efcc80f6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:26.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.842+0000 7efcce302700 1 --2- 172.21.15.162:0/1624912375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcc80f6a30 0x7efcc80f6e50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7efcb8004870 tx=0x7efcb8009cb0).stop 2024-02-13T00:34:26.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.842+0000 7efcce302700 1 -- 172.21.15.162:0/1624912375 shutdown_connections 2024-02-13T00:34:26.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.842+0000 7efcce302700 1 --2- 172.21.15.162:0/1624912375 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efcc80f8450 0x7efcc80fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:26.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.842+0000 7efcce302700 1 --2- 172.21.15.162:0/1624912375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcc80f6a30 0x7efcc80f6e50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:26.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.842+0000 7efcce302700 1 --2- 172.21.15.162:0/1624912375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcc80f7340 0x7efcc80f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:26.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.842+0000 7efcce302700 1 -- 172.21.15.162:0/1624912375 >> 172.21.15.162:0/1624912375 conn(0x7efcc80f2230 msgr2=0x7efcc80f4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:34:26.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.842+0000 7efcce302700 1 -- 172.21.15.162:0/1624912375 shutdown_connections 2024-02-13T00:34:26.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 -- 172.21.15.162:0/1624912375 wait complete. 2024-02-13T00:34:26.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 librados: starting msgr at 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 librados: starting objecter 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 Processor -- start 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 -- start start 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 librados: setting wanted keys 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 librados: calling monclient init 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcc80f6a30 0x7efcc8061b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcc80f7340 0x7efcc810d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efcc80f8450 0x7efcc810f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc810fdb0 con 0x7efcc80f6a30 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc810ff20 con 0x7efcc80f7340 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcce302700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc81100e0 con 0x7efcc80f8450 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.843+0000 7efcc7fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcc80f6a30 0x7efcc8061b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:26.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc7fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcc80f6a30 0x7efcc8061b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:36548/0 (socket says 172.21.15.162:36548) 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc7fff700 1 -- 172.21.15.162:0/824109071 learned_addr learned my addr 172.21.15.162:0/824109071 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efccc89f700 1 --2- 172.21.15.162:0/824109071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efcc80f8450 0x7efcc810f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc77fe700 1 --2- 172.21.15.162:0/824109071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcc80f7340 0x7efcc810d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc7fff700 1 -- 172.21.15.162:0/824109071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efcc80f8450 msgr2=0x7efcc810f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc7fff700 1 --2- 172.21.15.162:0/824109071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efcc80f8450 0x7efcc810f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc7fff700 1 -- 172.21.15.162:0/824109071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcc80f7340 msgr2=0x7efcc810d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc7fff700 1 --2- 172.21.15.162:0/824109071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcc80f7340 0x7efcc810d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc7fff700 1 -- 172.21.15.162:0/824109071 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc00097c0 con 0x7efcc80f6a30 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc7fff700 1 --2- 172.21.15.162:0/824109071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcc80f6a30 0x7efcc8061b80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7efcb8004710 tx=0x7efcb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc57fa700 10 client.?.objecter ms_handle_connect 0x7efcc80f6a30 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc57fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc57fa700 1 -- 172.21.15.162:0/824109071 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb800e030 con 0x7efcc80f6a30 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc57fa700 1 -- 172.21.15.162:0/824109071 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcb8007d50 con 0x7efcc80f6a30 2024-02-13T00:34:26.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcce302700 1 -- 172.21.15.162:0/824109071 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb8005040 con 0x7efcc80f6a30 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcc57fa700 1 -- 172.21.15.162:0/824109071 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb8019070 con 0x7efcc80f6a30 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcce302700 10 client.14211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcce302700 1 -- 172.21.15.162:0/824109071 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc8110580 con 0x7efcc80f6a30 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.844+0000 7efcce302700 1 librados: init done 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.845+0000 7efcc57fa700 1 -- 172.21.15.162:0/824109071 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) v1 ==== 38639+0+0 (secure 0 0 0) 0x7efcb801d5d0 con 0x7efcc80f6a30 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.845+0000 7efcc57fa700 1 --2- 172.21.15.162:0/824109071 >> v2:172.21.15.162:6800/1909610488 conn(0x7efcb0034230 0x7efcb00366f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.845+0000 7efcc57fa700 1 -- 172.21.15.162:0/824109071 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7efcb80443a0 con 0x7efcc80f6a30 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.845+0000 7efcc57fa700 10 client.14211.objecter ms_dispatch 0x7efcc8001550 osd_map(4..4 src has 1..4) v4 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.845+0000 7efcc57fa700 3 client.14211.objecter handle_osd_map got epochs [4,4] > 0 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.845+0000 7efcc57fa700 3 client.14211.objecter handle_osd_map decoding full epoch 4 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.845+0000 7efcce302700 1 -- 172.21.15.162:0/824109071 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc8060070 con 0x7efcc80f6a30 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.845+0000 7efcc57fa700 20 client.14211.objecter dump_active .. 0 homeless 2024-02-13T00:34:26.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.845+0000 7efcc77fe700 1 --2- 172.21.15.162:0/824109071 >> v2:172.21.15.162:6800/1909610488 conn(0x7efcb0034230 0x7efcb00366f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:26.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.846+0000 7efcc77fe700 1 --2- 172.21.15.162:0/824109071 >> v2:172.21.15.162:6800/1909610488 conn(0x7efcb0034230 0x7efcb00366f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7efcbc004e60 tx=0x7efcbc00b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:34:26.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.846+0000 7efcc57fa700 10 client.14211.objecter ms_handle_connect 0x7efcb0034230 2024-02-13T00:34:26.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:26.850+0000 7efcc57fa700 1 -- 172.21.15.162:0/824109071 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7efcb801c730 con 0x7efcc80f6a30 2024-02-13T00:34:27.005 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:26.567+0000 7f7aa5a79680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T00:34:27.005 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:26.738+0000 7f7aa5a79680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T00:34:27.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:27.021+0000 7efcce302700 1 -- 172.21.15.162:0/824109071 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7efcc81018c0 con 0x7efcb0034230 2024-02-13T00:34:27.264 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:27.001+0000 7f7aa5a79680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T00:34:27.264 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:27.130+0000 7f7aa5a79680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T00:34:27.516 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:27.261+0000 7f7aa5a79680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T00:34:27.516 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:27.431+0000 7f7aa5a79680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T00:34:27.516 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:27.022861+0000 mgr.y (mgr.14146) 70 : audit [DBG] from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:34:27.516 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:27.024993+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:34:27.516 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:27.026974+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:34:27.516 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:27.027510+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:27.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:27.022861+0000 mgr.y (mgr.14146) 70 : audit [DBG] from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:34:27.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:27.024993+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:34:27.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:27.026974+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:34:27.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:27.027510+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:27.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:27.022861+0000 mgr.y (mgr.14146) 70 : audit [DBG] from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:34:27.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:27.024993+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:34:27.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:27.026974+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:34:27.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:27.027510+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:27.794 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:34:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:34:27.512+0000 7f7aa5a79680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T00:34:28.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:28.289+0000 7efcc57fa700 1 -- 172.21.15.162:0/824109071 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7efcb8028440 con 0x7efcc80f6a30 2024-02-13T00:34:28.414 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:27.489045+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:28.414 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:27.527146+0000 mon.a (mon.0) 215 : cluster [DBG] Standby manager daemon x started 2024-02-13T00:34:28.414 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:27.528575+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.? 172.21.15.181:0/654134991' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T00:34:28.414 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:27.528889+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.? 172.21.15.181:0/654134991' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T00:34:28.414 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:27.529797+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.? 172.21.15.181:0/654134991' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T00:34:28.414 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:27.530055+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.? 172.21.15.181:0/654134991' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T00:34:28.414 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:27.489045+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:28.414 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:27.527146+0000 mon.a (mon.0) 215 : cluster [DBG] Standby manager daemon x started 2024-02-13T00:34:28.414 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:27.528575+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.? 172.21.15.181:0/654134991' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T00:34:28.414 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:27.528889+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.? 172.21.15.181:0/654134991' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T00:34:28.415 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:27.529797+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.? 172.21.15.181:0/654134991' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T00:34:28.415 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:27.530055+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.? 172.21.15.181:0/654134991' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T00:34:28.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:27.489045+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:28.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:27.527146+0000 mon.a (mon.0) 215 : cluster [DBG] Standby manager daemon x started 2024-02-13T00:34:28.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:27.528575+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.? 172.21.15.181:0/654134991' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T00:34:28.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:27.528889+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.? 172.21.15.181:0/654134991' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T00:34:28.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:27.529797+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.? 172.21.15.181:0/654134991' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T00:34:28.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:27.530055+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.? 172.21.15.181:0/654134991' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T00:34:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:28.291996+0000 mon.a (mon.0) 220 : cluster [DBG] mgrmap e14: y(active, since 100s), standbys: x 2024-02-13T00:34:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:28.292112+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T00:34:29.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:28.291996+0000 mon.a (mon.0) 220 : cluster [DBG] mgrmap e14: y(active, since 100s), standbys: x 2024-02-13T00:34:29.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:28.292112+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T00:34:29.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:28.291996+0000 mon.a (mon.0) 220 : cluster [DBG] mgrmap e14: y(active, since 100s), standbys: x 2024-02-13T00:34:29.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:28.292112+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T00:34:30.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:29.434312+0000 mon.c (mon.2) 2 : audit [INF] from='client.? 172.21.15.162:0/323967589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99"}]: dispatch 2024-02-13T00:34:30.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:29.434637+0000 mon.a (mon.0) 222 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99"}]: dispatch 2024-02-13T00:34:30.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:29.438238+0000 mon.a (mon.0) 223 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99"}]': finished 2024-02-13T00:34:30.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:29.438292+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-13T00:34:30.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:29.438385+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:30.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:29.489398+0000 mgr.y (mgr.14146) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:30.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:30.008223+0000 mon.a (mon.0) 226 : audit [DBG] from='client.? 172.21.15.162:0/2514728700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:34:30.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:29.434312+0000 mon.c (mon.2) 2 : audit [INF] from='client.? 172.21.15.162:0/323967589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99"}]: dispatch 2024-02-13T00:34:30.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:29.434637+0000 mon.a (mon.0) 222 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99"}]: dispatch 2024-02-13T00:34:30.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:29.438238+0000 mon.a (mon.0) 223 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99"}]': finished 2024-02-13T00:34:30.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:29.438292+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-13T00:34:30.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:29.438385+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:30.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:29.489398+0000 mgr.y (mgr.14146) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:30.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:30.008223+0000 mon.a (mon.0) 226 : audit [DBG] from='client.? 172.21.15.162:0/2514728700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:34:30.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:29.434312+0000 mon.c (mon.2) 2 : audit [INF] from='client.? 172.21.15.162:0/323967589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99"}]: dispatch 2024-02-13T00:34:30.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:29.434637+0000 mon.a (mon.0) 222 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99"}]: dispatch 2024-02-13T00:34:30.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:29.438238+0000 mon.a (mon.0) 223 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99"}]': finished 2024-02-13T00:34:30.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:29.438292+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-13T00:34:30.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:29.438385+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:30.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:29.489398+0000 mgr.y (mgr.14146) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:30.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:30.008223+0000 mon.a (mon.0) 226 : audit [DBG] from='client.? 172.21.15.162:0/2514728700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:34:31.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:31.845+0000 7efccdb01700 10 client.14211.objecter tick 2024-02-13T00:34:32.815 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:31.489715+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:32.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:31.489715+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:33.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:31.489715+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:34.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:33.489999+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:34.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:33.489999+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:35.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:33.489999+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:36.743 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:35.490412+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:36.743 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:35.631097+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T00:34:36.743 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:35.631771+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:36.743 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:35.632318+0000 mgr.y (mgr.14146) 76 : cephadm [INF] Deploying daemon osd.0 on smithi162 2024-02-13T00:34:36.743 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:35.490412+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:36.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:35.631097+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T00:34:36.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:35.631771+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:36.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:35.632318+0000 mgr.y (mgr.14146) 76 : cephadm [INF] Deploying daemon osd.0 on smithi162 2024-02-13T00:34:36.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:36.846+0000 7efccdb01700 10 client.14211.objecter tick 2024-02-13T00:34:37.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:35.490412+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:37.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:35.631097+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T00:34:37.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:35.631771+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:37.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:35.632318+0000 mgr.y (mgr.14146) 76 : cephadm [INF] Deploying daemon osd.0 on smithi162 2024-02-13T00:34:38.775 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:37.490671+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:38.775 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:37.490671+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:39.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:37.490671+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:40.418 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:39.490938+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:40.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:39.490938+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:40.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:39.490938+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:41.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:40.460005+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:41.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:40.476713+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:41.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:40.460005+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:41.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:40.476713+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:41.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:40.460005+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:41.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:40.476713+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:41.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:41.847+0000 7efccdb01700 10 client.14211.objecter tick 2024-02-13T00:34:42.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:41.491179+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:42.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:41.491179+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:42.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:41.491179+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:43.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.733+0000 7efcc57fa700 1 -- 172.21.15.162:0/824109071 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efcc81018c0 con 0x7efcb0034230 2024-02-13T00:34:43.736 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 0 on host 'smithi162' 2024-02-13T00:34:43.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.734+0000 7efcce302700 10 librados: watch_flush enter 2024-02-13T00:34:43.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.734+0000 7efcce302700 10 librados: watch_flush exit 2024-02-13T00:34:43.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.734+0000 7efcce302700 20 client.14211.objecter shutdown clearing up homeless session... 2024-02-13T00:34:43.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.734+0000 7efcce302700 10 client.14211.objecter successfully canceled tick 2024-02-13T00:34:43.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 -- 172.21.15.162:0/824109071 >> v2:172.21.15.162:6800/1909610488 conn(0x7efcb0034230 msgr2=0x7efcb00366f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:43.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 --2- 172.21.15.162:0/824109071 >> v2:172.21.15.162:6800/1909610488 conn(0x7efcb0034230 0x7efcb00366f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7efcbc004e60 tx=0x7efcbc00b040).stop 2024-02-13T00:34:43.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 -- 172.21.15.162:0/824109071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcc80f6a30 msgr2=0x7efcc8061b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:43.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 --2- 172.21.15.162:0/824109071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcc80f6a30 0x7efcc8061b80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7efcb8004710 tx=0x7efcb800b040).stop 2024-02-13T00:34:43.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 -- 172.21.15.162:0/824109071 shutdown_connections 2024-02-13T00:34:43.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 --2- 172.21.15.162:0/824109071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efcc80f8450 0x7efcc810f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:43.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 --2- 172.21.15.162:0/824109071 >> v2:172.21.15.162:6800/1909610488 conn(0x7efcb0034230 0x7efcb00366f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:43.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 --2- 172.21.15.162:0/824109071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efcc80f6a30 0x7efcc8061b80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:43.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 --2- 172.21.15.162:0/824109071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcc80f7340 0x7efcc810d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:43.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 -- 172.21.15.162:0/824109071 >> 172.21.15.162:0/824109071 conn(0x7efcc80f2230 msgr2=0x7efcc80fb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:34:43.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 -- 172.21.15.162:0/824109071 shutdown_connections 2024-02-13T00:34:43.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 -- 172.21.15.162:0/824109071 wait complete. 2024-02-13T00:34:43.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:43.735+0000 7efcce302700 1 librados: shutdown 2024-02-13T00:34:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:42.978256+0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:42.978933+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:42.979782+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:42.986672+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:43.307898+0000 mon.c (mon.2) 3 : audit [INF] from='osd.0 v2:172.21.15.162:6801/2103743338' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T00:34:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:43.308180+0000 mon.a (mon.0) 235 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T00:34:44.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:43.733217+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:44.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:43.753494+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:44.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:42.978256+0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:44.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:42.978933+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:44.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:42.979782+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:44.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:42.986672+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:44.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:43.307898+0000 mon.c (mon.2) 3 : audit [INF] from='osd.0 v2:172.21.15.162:6801/2103743338' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T00:34:44.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:43.308180+0000 mon.a (mon.0) 235 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T00:34:44.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:43.733217+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:44.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:43.753494+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:44.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:42.978256+0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:44.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:42.978933+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:44.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:42.979782+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:44.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:42.986672+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:44.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:43.307898+0000 mon.c (mon.2) 3 : audit [INF] from='osd.0 v2:172.21.15.162:6801/2103743338' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T00:34:44.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:43.308180+0000 mon.a (mon.0) 235 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T00:34:44.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:43.733217+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:44.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:43.753494+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:34:44.452 DEBUG:teuthology.orchestra.run.smithi162:osd.0> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.0.service 2024-02-13T00:34:44.454 INFO:tasks.cephadm:Deploying osd.1 on smithi162 with /dev/vg_nvme/lv_3... 2024-02-13T00:34:44.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-13T00:34:44.484 INFO:journalctl@ceph.osd.0.smithi162.stdout:-- Logs begin at Tue 2024-02-13 00:17:45 UTC. -- 2024-02-13T00:34:45.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:43.491459+0000 mgr.y (mgr.14146) 80 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:43.993650+0000 mon.a (mon.0) 238 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-13T00:34:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:43.993700+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-13T00:34:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:43.993818+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:43.995004+0000 mon.c (mon.2) 4 : audit [INF] from='osd.0 v2:172.21.15.162:6801/2103743338' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:34:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:43.995741+0000 mon.a (mon.0) 241 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:34:45.330 INFO:journalctl@ceph.osd.0.smithi162.stdout:Feb 13 00:34:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-0[49887]: debug 2024-02-13T00:34:45.000+0000 7f2399543700 -1 osd.0 0 waiting for initial osdmap 2024-02-13T00:34:45.330 INFO:journalctl@ceph.osd.0.smithi162.stdout:Feb 13 00:34:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-0[49887]: debug 2024-02-13T00:34:45.002+0000 7f23934ba700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T00:34:45.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:43.491459+0000 mgr.y (mgr.14146) 80 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:45.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:43.993650+0000 mon.a (mon.0) 238 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-13T00:34:45.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:43.993700+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-13T00:34:45.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:43.993818+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:45.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:43.995004+0000 mon.c (mon.2) 4 : audit [INF] from='osd.0 v2:172.21.15.162:6801/2103743338' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:34:45.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:43.995741+0000 mon.a (mon.0) 241 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:34:45.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:43.491459+0000 mgr.y (mgr.14146) 80 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:45.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:43.993650+0000 mon.a (mon.0) 238 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-13T00:34:45.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:43.993700+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-13T00:34:45.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:43.993818+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:45.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:43.995004+0000 mon.c (mon.2) 4 : audit [INF] from='osd.0 v2:172.21.15.162:6801/2103743338' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:34:45.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:43.995741+0000 mon.a (mon.0) 241 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:34:46.262 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:44.995249+0000 mon.a (mon.0) 242 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-13T00:34:46.262 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:44.995335+0000 mon.a (mon.0) 243 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-13T00:34:46.262 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:44.995688+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:46.262 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:45.001756+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:46.262 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:44.995249+0000 mon.a (mon.0) 242 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-13T00:34:46.262 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:44.995335+0000 mon.a (mon.0) 243 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-13T00:34:46.263 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:44.995688+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:46.263 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:45.001756+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:46.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:44.995249+0000 mon.a (mon.0) 242 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-13T00:34:46.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:44.995335+0000 mon.a (mon.0) 243 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-13T00:34:46.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:44.995688+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:45.001756+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:47.191 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:44.292029+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:34:47.192 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:44.292082+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:34:47.192 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:45.491720+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:47.192 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:46.000439+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:47.192 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:46.003875+0000 mon.a (mon.0) 247 : cluster [INF] osd.0 v2:172.21.15.162:6801/2103743338 boot 2024-02-13T00:34:47.192 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:46.003916+0000 mon.a (mon.0) 248 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-13T00:34:47.192 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:46.004145+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:47.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:44.292029+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:34:47.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:44.292082+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:34:47.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:45.491720+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:47.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:46.000439+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:47.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:46.003875+0000 mon.a (mon.0) 247 : cluster [INF] osd.0 v2:172.21.15.162:6801/2103743338 boot 2024-02-13T00:34:47.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:46.003916+0000 mon.a (mon.0) 248 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-13T00:34:47.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:46.004145+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:47.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:44.292029+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:34:47.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:44.292082+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:34:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:45.491720+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T00:34:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:46.000439+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:46.003875+0000 mon.a (mon.0) 247 : cluster [INF] osd.0 v2:172.21.15.162:6801/2103743338 boot 2024-02-13T00:34:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:46.003916+0000 mon.a (mon.0) 248 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-13T00:34:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:46.004145+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:34:48.274 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-13T00:34:48.274 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-13T00:34:48.274 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-02-13T00:34:48.274 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-02-13T00:34:48.274 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011341 s, 925 MB/s 2024-02-13T00:34:48.274 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-02-13T00:34:48.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:47.008240+0000 mon.a (mon.0) 250 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-13T00:34:48.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:47.974475+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:48.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:47.975260+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:34:48.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:47.979207+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:47.980423+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:47.981070+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:47.985490+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:48.310 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:47.008240+0000 mon.a (mon.0) 250 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-13T00:34:48.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:47.974475+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:48.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:47.975260+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:34:48.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:47.979207+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:48.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:47.980423+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:48.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:47.981070+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:48.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:47.985490+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:48.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:47.008240+0000 mon.a (mon.0) 250 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-13T00:34:48.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:47.974475+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:48.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:47.975260+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:34:48.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:47.979207+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:48.312 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:47.980423+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:48.312 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:47.981070+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:34:48.312 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:47.985490+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:34:48.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_3 2024-02-13T00:34:49.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:47.491974+0000 mgr.y (mgr.14146) 82 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:49.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:47.969549+0000 mgr.y (mgr.14146) 83 : cephadm [INF] Detected new or changed devices on smithi162 2024-02-13T00:34:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:47.975710+0000 mgr.y (mgr.14146) 84 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 16098M 2024-02-13T00:34:49.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:47.491974+0000 mgr.y (mgr.14146) 82 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:49.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:47.969549+0000 mgr.y (mgr.14146) 83 : cephadm [INF] Detected new or changed devices on smithi162 2024-02-13T00:34:49.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:47.975710+0000 mgr.y (mgr.14146) 84 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 16098M 2024-02-13T00:34:49.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:47.491974+0000 mgr.y (mgr.14146) 82 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:49.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:47.969549+0000 mgr.y (mgr.14146) 83 : cephadm [INF] Detected new or changed devices on smithi162 2024-02-13T00:34:49.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:47.975710+0000 mgr.y (mgr.14146) 84 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 16098M 2024-02-13T00:34:50.301 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:49.492223+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:50.302 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:49.492223+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:50.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:49.492223+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:50.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.809+0000 7f59d848d700 1 -- 172.21.15.162:0/3856133548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59d00f8620 msgr2=0x7f59d00fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.809+0000 7f59d848d700 1 --2- 172.21.15.162:0/3856133548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59d00f8620 0x7f59d00fef20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f59c8009be0 tx=0x7f59c8004d10).stop 2024-02-13T00:34:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.809+0000 7f59d848d700 1 -- 172.21.15.162:0/3856133548 shutdown_connections 2024-02-13T00:34:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.809+0000 7f59d848d700 1 --2- 172.21.15.162:0/3856133548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59d00f8620 0x7f59d00fef20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.809+0000 7f59d848d700 1 --2- 172.21.15.162:0/3856133548 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59d00f6a30 0x7f59d00f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.809+0000 7f59d848d700 1 --2- 172.21.15.162:0/3856133548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59d00f7cd0 0x7f59d00f8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.809+0000 7f59d848d700 1 -- 172.21.15.162:0/3856133548 >> 172.21.15.162:0/3856133548 conn(0x7f59d00f2230 msgr2=0x7f59d00f4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:34:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.809+0000 7f59d848d700 1 -- 172.21.15.162:0/3856133548 shutdown_connections 2024-02-13T00:34:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.810+0000 7f59d848d700 1 -- 172.21.15.162:0/3856133548 wait complete. 2024-02-13T00:34:50.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.810+0000 7f59d848d700 1 librados: starting msgr at 2024-02-13T00:34:50.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.810+0000 7f59d848d700 1 librados: starting objecter 2024-02-13T00:34:50.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.810+0000 7f59d848d700 1 Processor -- start 2024-02-13T00:34:50.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.810+0000 7f59d848d700 1 -- start start 2024-02-13T00:34:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.810+0000 7f59d848d700 1 librados: setting wanted keys 2024-02-13T00:34:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.810+0000 7f59d848d700 1 librados: calling monclient init 2024-02-13T00:34:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d848d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59d00f6a30 0x7f59d0106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d848d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59d00f7cd0 0x7f59d0107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d848d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59d00f8620 0x7f59d010e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d848d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d010e820 con 0x7f59d00f8620 2024-02-13T00:34:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d848d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d010e960 con 0x7f59d00f6a30 2024-02-13T00:34:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d848d700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d010eaf0 con 0x7f59d00f7cd0 2024-02-13T00:34:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d5a28700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59d00f7cd0 0x7f59d0107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d5a28700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59d00f7cd0 0x7f59d0107290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:50768/0 (socket says 172.21.15.162:50768) 2024-02-13T00:34:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d5a28700 1 -- 172.21.15.162:0/3595702330 learned_addr learned my addr 172.21.15.162:0/3595702330 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:34:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d6a2a700 1 --2- 172.21.15.162:0/3595702330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59d00f8620 0x7f59d010e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d6229700 1 --2- 172.21.15.162:0/3595702330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59d00f6a30 0x7f59d0106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d5a28700 1 -- 172.21.15.162:0/3595702330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59d00f6a30 msgr2=0x7f59d0106da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d5a28700 1 --2- 172.21.15.162:0/3595702330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59d00f6a30 0x7f59d0106da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d5a28700 1 -- 172.21.15.162:0/3595702330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59d00f8620 msgr2=0x7f59d010e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d5a28700 1 --2- 172.21.15.162:0/3595702330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59d00f8620 0x7f59d010e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d5a28700 1 -- 172.21.15.162:0/3595702330 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc005040 con 0x7f59d00f7cd0 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d6a2a700 1 --2- 172.21.15.162:0/3595702330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59d00f8620 0x7f59d010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d6229700 1 --2- 172.21.15.162:0/3595702330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59d00f6a30 0x7f59d0106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d5a28700 1 --2- 172.21.15.162:0/3595702330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59d00f7cd0 0x7f59d0107290 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f59c400c7c0 tx=0x7f59c4007ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59c37fe700 10 client.?.objecter ms_handle_connect 0x7f59d00f7cd0 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59c37fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59c37fe700 1 -- 172.21.15.162:0/3595702330 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c4004430 con 0x7f59d00f7cd0 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59c37fe700 1 -- 172.21.15.162:0/3595702330 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f59c400e970 con 0x7f59d00f7cd0 2024-02-13T00:34:50.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59c37fe700 1 -- 172.21.15.162:0/3595702330 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c4013a50 con 0x7f59d00f7cd0 2024-02-13T00:34:50.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d848d700 1 -- 172.21.15.162:0/3595702330 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c80097c0 con 0x7f59d00f7cd0 2024-02-13T00:34:50.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d848d700 10 client.24149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:34:50.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d848d700 1 -- 172.21.15.162:0/3595702330 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d010f130 con 0x7f59d00f7cd0 2024-02-13T00:34:50.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.811+0000 7f59d848d700 1 librados: init done 2024-02-13T00:34:50.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.812+0000 7f59d848d700 1 -- 172.21.15.162:0/3595702330 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d005f400 con 0x7f59d00f7cd0 2024-02-13T00:34:50.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.813+0000 7f59c37fe700 1 -- 172.21.15.162:0/3595702330 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7f59c400eae0 con 0x7f59d00f7cd0 2024-02-13T00:34:50.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.813+0000 7f59c37fe700 1 --2- 172.21.15.162:0/3595702330 >> v2:172.21.15.162:6800/1909610488 conn(0x7f59bc062bd0 0x7f59bc065090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:34:50.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.813+0000 7f59c37fe700 1 -- 172.21.15.162:0/3595702330 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1484+0+0 (secure 0 0 0) 0x7f59c407b840 con 0x7f59d00f7cd0 2024-02-13T00:34:50.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.813+0000 7f59c37fe700 10 client.24149.objecter ms_dispatch 0x7f59d0001550 osd_map(9..9 src has 1..9) v4 2024-02-13T00:34:50.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.813+0000 7f59c37fe700 3 client.24149.objecter handle_osd_map got epochs [9,9] > 0 2024-02-13T00:34:50.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.813+0000 7f59c37fe700 3 client.24149.objecter handle_osd_map decoding full epoch 9 2024-02-13T00:34:50.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.813+0000 7f59d6229700 1 --2- 172.21.15.162:0/3595702330 >> v2:172.21.15.162:6800/1909610488 conn(0x7f59bc062bd0 0x7f59bc065090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:34:50.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.813+0000 7f59c37fe700 20 client.24149.objecter dump_active .. 0 homeless 2024-02-13T00:34:50.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.813+0000 7f59c37fe700 10 client.24149.objecter ms_handle_connect 0x7f59bc062bd0 2024-02-13T00:34:50.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.813+0000 7f59d6229700 1 --2- 172.21.15.162:0/3595702330 >> v2:172.21.15.162:6800/1909610488 conn(0x7f59bc062bd0 0x7f59bc065090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f59cc007fd0 tx=0x7f59cc009cb0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:34:50.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.817+0000 7f59c37fe700 1 -- 172.21.15.162:0/3595702330 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f59c404cd70 con 0x7f59d00f7cd0 2024-02-13T00:34:50.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:50.987+0000 7f59d848d700 1 -- 172.21.15.162:0/3595702330 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f59d00fdd80 con 0x7f59bc062bd0 2024-02-13T00:34:51.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:50.989341+0000 mgr.y (mgr.14146) 86 : audit [DBG] from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:34:51.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:50.991389+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:34:51.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:50.993361+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:34:51.461 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:50.993841+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:51.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:50.989341+0000 mgr.y (mgr.14146) 86 : audit [DBG] from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:34:51.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:50.991389+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:34:51.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:50.993361+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:34:51.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:50.993841+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:51.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:50.989341+0000 mgr.y (mgr.14146) 86 : audit [DBG] from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:34:51.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:50.991389+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:34:51.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:50.993361+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:34:51.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:50.993841+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:34:52.319 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:51.492693+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:52.320 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:51.492693+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:52.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:51.492693+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:53.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:53.322496+0000 mon.a (mon.0) 260 : audit [INF] from='client.? 172.21.15.162:0/3901658563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886"}]: dispatch 2024-02-13T00:34:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:53.326654+0000 mon.a (mon.0) 261 : audit [INF] from='client.? 172.21.15.162:0/3901658563' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886"}]': finished 2024-02-13T00:34:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:53.326700+0000 mon.a (mon.0) 262 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-13T00:34:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:53.326823+0000 mon.a (mon.0) 263 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:34:53.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:53.322496+0000 mon.a (mon.0) 260 : audit [INF] from='client.? 172.21.15.162:0/3901658563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886"}]: dispatch 2024-02-13T00:34:53.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:53.326654+0000 mon.a (mon.0) 261 : audit [INF] from='client.? 172.21.15.162:0/3901658563' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886"}]': finished 2024-02-13T00:34:53.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:53.326700+0000 mon.a (mon.0) 262 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-13T00:34:53.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:53.326823+0000 mon.a (mon.0) 263 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:34:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:53.322496+0000 mon.a (mon.0) 260 : audit [INF] from='client.? 172.21.15.162:0/3901658563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886"}]: dispatch 2024-02-13T00:34:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:53.326654+0000 mon.a (mon.0) 261 : audit [INF] from='client.? 172.21.15.162:0/3901658563' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886"}]': finished 2024-02-13T00:34:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:53.326700+0000 mon.a (mon.0) 262 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-13T00:34:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:53.326823+0000 mon.a (mon.0) 263 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:34:54.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:53.492951+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:53.886184+0000 mon.a (mon.0) 264 : audit [DBG] from='client.? 172.21.15.162:0/1089048401' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:34:54.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:53.492951+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:54.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:53.886184+0000 mon.a (mon.0) 264 : audit [DBG] from='client.? 172.21.15.162:0/1089048401' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:34:54.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:53.492951+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:54.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:53.886184+0000 mon.a (mon.0) 264 : audit [DBG] from='client.? 172.21.15.162:0/1089048401' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:34:55.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:34:55.812+0000 7f59d7c8c700 10 client.24149.objecter tick 2024-02-13T00:34:56.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:55.493186+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:55.493186+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:57.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:55.493186+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:58.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:34:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:57.493553+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:58.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:34:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:57.493553+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:34:59.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:34:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:57.493553+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:00.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:59.410327+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T00:35:00.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:34:59.410925+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:00.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:34:59.411441+0000 mgr.y (mgr.14146) 91 : cephadm [INF] Deploying daemon osd.1 on smithi162 2024-02-13T00:35:00.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:34:59.493859+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:00.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:59.410327+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T00:35:00.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:34:59.410925+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:00.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:34:59.411441+0000 mgr.y (mgr.14146) 91 : cephadm [INF] Deploying daemon osd.1 on smithi162 2024-02-13T00:35:00.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:34:59.493859+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:00.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:59.410327+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T00:35:00.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:34:59.410925+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:00.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:34:59.411441+0000 mgr.y (mgr.14146) 91 : cephadm [INF] Deploying daemon osd.1 on smithi162 2024-02-13T00:35:00.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:34:59.493859+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:00.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:00.813+0000 7f59d7c8c700 10 client.24149.objecter tick 2024-02-13T00:35:02.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:01.494221+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:02.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:01.494221+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:03.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:01.494221+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:04.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:03.494708+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:04.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:03.494708+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:05.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:03.494708+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:05.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:05.813+0000 7f59d7c8c700 10 client.24149.objecter tick 2024-02-13T00:35:06.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:04.655073+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:06.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:04.688704+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:06.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:04.655073+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:06.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:04.688704+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:06.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:04.655073+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:06.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:04.688704+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:07.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:05.495046+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:07.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:05.495046+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:07.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:05.495046+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:07.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.903+0000 7f59c37fe700 1 -- 172.21.15.162:0/3595702330 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f59d00fdd80 con 0x7f59bc062bd0 2024-02-13T00:35:07.908 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 1 on host 'smithi162' 2024-02-13T00:35:07.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 10 librados: watch_flush enter 2024-02-13T00:35:07.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 10 librados: watch_flush exit 2024-02-13T00:35:07.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 20 client.24149.objecter shutdown clearing up homeless session... 2024-02-13T00:35:07.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 10 client.24149.objecter successfully canceled tick 2024-02-13T00:35:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 1 -- 172.21.15.162:0/3595702330 >> v2:172.21.15.162:6800/1909610488 conn(0x7f59bc062bd0 msgr2=0x7f59bc065090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:35:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 1 --2- 172.21.15.162:0/3595702330 >> v2:172.21.15.162:6800/1909610488 conn(0x7f59bc062bd0 0x7f59bc065090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f59cc007fd0 tx=0x7f59cc009cb0).stop 2024-02-13T00:35:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 1 -- 172.21.15.162:0/3595702330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59d00f7cd0 msgr2=0x7f59d0107290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:35:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 1 --2- 172.21.15.162:0/3595702330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59d00f7cd0 0x7f59d0107290 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f59c400c7c0 tx=0x7f59c4007ef0).stop 2024-02-13T00:35:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 1 -- 172.21.15.162:0/3595702330 shutdown_connections 2024-02-13T00:35:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 1 --2- 172.21.15.162:0/3595702330 >> v2:172.21.15.162:6800/1909610488 conn(0x7f59bc062bd0 0x7f59bc065090 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 1 --2- 172.21.15.162:0/3595702330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59d00f8620 0x7f59d010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 1 --2- 172.21.15.162:0/3595702330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59d00f6a30 0x7f59d0106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 1 --2- 172.21.15.162:0/3595702330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59d00f7cd0 0x7f59d0107290 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 1 -- 172.21.15.162:0/3595702330 >> 172.21.15.162:0/3595702330 conn(0x7f59d00f2230 msgr2=0x7f59d00faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:35:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 1 -- 172.21.15.162:0/3595702330 shutdown_connections 2024-02-13T00:35:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.905+0000 7f59d848d700 1 -- 172.21.15.162:0/3595702330 wait complete. 2024-02-13T00:35:07.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:07.906+0000 7f59d848d700 1 librados: shutdown 2024-02-13T00:35:08.543 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:07.172040+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:08.543 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:07.172720+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:08.543 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:07.173327+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:35:08.543 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:07.178348+0000 mon.a (mon.0) 272 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:08.543 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:07.495329+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:08.544 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:07.739062+0000 mon.c (mon.2) 5 : audit [INF] from='osd.1 v2:172.21.15.162:6805/150599780' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T00:35:08.544 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:07.739381+0000 mon.a (mon.0) 273 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T00:35:08.544 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:07.904775+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:08.544 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:07.945349+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:08.544 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:07.172040+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:08.544 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:07.172720+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:08.544 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:07.173327+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:35:08.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:07.178348+0000 mon.a (mon.0) 272 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:08.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:07.495329+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:08.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:07.739062+0000 mon.c (mon.2) 5 : audit [INF] from='osd.1 v2:172.21.15.162:6805/150599780' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T00:35:08.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:07.739381+0000 mon.a (mon.0) 273 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T00:35:08.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:07.904775+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:08.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:07.945349+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:08.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:07.172040+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:08.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:07.172720+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:08.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:07.173327+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:35:08.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:07.178348+0000 mon.a (mon.0) 272 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:08.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:07.495329+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:08.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:07.739062+0000 mon.c (mon.2) 5 : audit [INF] from='osd.1 v2:172.21.15.162:6805/150599780' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T00:35:08.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:07.739381+0000 mon.a (mon.0) 273 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T00:35:08.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:07.904775+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:08.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:07.945349+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:08.571 DEBUG:teuthology.orchestra.run.smithi162:osd.1> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.1.service 2024-02-13T00:35:08.573 INFO:tasks.cephadm:Deploying osd.2 on smithi162 with /dev/vg_nvme/lv_2... 2024-02-13T00:35:08.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-13T00:35:08.606 INFO:journalctl@ceph.osd.1.smithi162.stdout:-- Logs begin at Tue 2024-02-13 00:17:45 UTC. -- 2024-02-13T00:35:09.329 INFO:journalctl@ceph.osd.1.smithi162.stdout:Feb 13 00:35:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-1[54323]: debug 2024-02-13T00:35:09.192+0000 7fe463558700 -1 osd.1 0 waiting for initial osdmap 2024-02-13T00:35:09.329 INFO:journalctl@ceph.osd.1.smithi162.stdout:Feb 13 00:35:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-1[54323]: debug 2024-02-13T00:35:09.195+0000 7fe45ccce700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T00:35:09.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:08.188215+0000 mon.a (mon.0) 276 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-13T00:35:09.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:08.188261+0000 mon.a (mon.0) 277 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-13T00:35:09.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:08.188350+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:35:09.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:08.191304+0000 mon.c (mon.2) 6 : audit [INF] from='osd.1 v2:172.21.15.162:6805/150599780' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:35:09.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:08.191583+0000 mon.a (mon.0) 279 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:35:09.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:08.188215+0000 mon.a (mon.0) 276 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-13T00:35:09.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:08.188261+0000 mon.a (mon.0) 277 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-13T00:35:09.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:08.188350+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:35:09.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:08.191304+0000 mon.c (mon.2) 6 : audit [INF] from='osd.1 v2:172.21.15.162:6805/150599780' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:35:09.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:08.191583+0000 mon.a (mon.0) 279 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:35:09.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:08.188215+0000 mon.a (mon.0) 276 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-13T00:35:09.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:08.188261+0000 mon.a (mon.0) 277 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-13T00:35:09.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:08.188350+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:35:09.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:08.191304+0000 mon.c (mon.2) 6 : audit [INF] from='osd.1 v2:172.21.15.162:6805/150599780' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:35:09.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:08.191583+0000 mon.a (mon.0) 279 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:35:10.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:09.188135+0000 mon.a (mon.0) 280 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-13T00:35:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:09.188223+0000 mon.a (mon.0) 281 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-13T00:35:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:09.188588+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:35:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:09.194623+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:35:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:09.495724+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:10.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:09.188135+0000 mon.a (mon.0) 280 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-13T00:35:10.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:09.188223+0000 mon.a (mon.0) 281 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-13T00:35:10.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:09.188588+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:35:10.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:09.194623+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:35:10.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:09.495724+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:10.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:09.188135+0000 mon.a (mon.0) 280 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-13T00:35:10.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:09.188223+0000 mon.a (mon.0) 281 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-13T00:35:10.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:09.188588+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:35:10.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:09.194623+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:35:10.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:09.495724+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-13T00:35:11.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:08.720808+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:35:11.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:08.720852+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:35:11.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:10.195322+0000 mon.a (mon.0) 284 : cluster [INF] osd.1 v2:172.21.15.162:6805/150599780 boot 2024-02-13T00:35:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:10.195370+0000 mon.a (mon.0) 285 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-13T00:35:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:10.195810+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:35:11.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:08.720808+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:35:11.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:08.720852+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:35:11.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:10.195322+0000 mon.a (mon.0) 284 : cluster [INF] osd.1 v2:172.21.15.162:6805/150599780 boot 2024-02-13T00:35:11.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:10.195370+0000 mon.a (mon.0) 285 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-13T00:35:11.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:10.195810+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:35:11.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:08.720808+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:35:11.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:08.720852+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:35:11.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:10.195322+0000 mon.a (mon.0) 284 : cluster [INF] osd.1 v2:172.21.15.162:6805/150599780 boot 2024-02-13T00:35:11.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:10.195370+0000 mon.a (mon.0) 285 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-13T00:35:11.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:10.195810+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:35:11.858 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-13T00:35:11.859 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-13T00:35:11.859 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-02-13T00:35:11.859 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-02-13T00:35:11.859 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113655 s, 923 MB/s 2024-02-13T00:35:11.859 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-02-13T00:35:12.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:11.200608+0000 mon.a (mon.0) 287 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-13T00:35:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:11.496068+0000 mgr.y (mgr.14146) 98 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:12.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:11.200608+0000 mon.a (mon.0) 287 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-13T00:35:12.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:11.496068+0000 mgr.y (mgr.14146) 98 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:11.200608+0000 mon.a (mon.0) 287 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-13T00:35:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:11.496068+0000 mgr.y (mgr.14146) 98 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:13.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_2 2024-02-13T00:35:14.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:35:13.180367+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Detected new or changed devices on smithi162 2024-02-13T00:35:14.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:13.184599+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:14.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:13.496530+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:14.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:13.706351+0000 mon.a (mon.0) 289 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:14.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:13.706929+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:14.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:35:13.707308+0000 mgr.y (mgr.14146) 101 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 8049M 2024-02-13T00:35:14.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:13.711068+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:14.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:13.711741+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:14.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:13.712264+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:35:14.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:13.716126+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:14.538 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:35:13.180367+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Detected new or changed devices on smithi162 2024-02-13T00:35:14.539 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:13.184599+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:14.539 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:13.496530+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:14.539 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:13.706351+0000 mon.a (mon.0) 289 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:14.539 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:13.706929+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:14.539 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:35:13.707308+0000 mgr.y (mgr.14146) 101 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 8049M 2024-02-13T00:35:14.539 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:13.711068+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:14.539 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:13.711741+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:14.539 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:13.712264+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:35:14.539 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:13.716126+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:14.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:35:13.180367+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Detected new or changed devices on smithi162 2024-02-13T00:35:14.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:13.184599+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:14.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:13.496530+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:14.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:13.706351+0000 mon.a (mon.0) 289 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:14.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:13.706929+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:14.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:35:13.707308+0000 mgr.y (mgr.14146) 101 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 8049M 2024-02-13T00:35:14.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:13.711068+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:14.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:13.711741+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:14.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:13.712264+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:35:14.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:13.716126+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.985+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/4259228746 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7d00f7320 msgr2=0x7fb7d00f7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:35:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.985+0000 7fb7d4f8d700 1 --2- 172.21.15.162:0/4259228746 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7d00f7320 0x7fb7d00f7d50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb7b8004870 tx=0x7fb7b8009cb0).stop 2024-02-13T00:35:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.985+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/4259228746 shutdown_connections 2024-02-13T00:35:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.985+0000 7fb7d4f8d700 1 --2- 172.21.15.162:0/4259228746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d00f8430 0x7fb7d00fefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:35:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.985+0000 7fb7d4f8d700 1 --2- 172.21.15.162:0/4259228746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7d00f6a10 0x7fb7d00f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.985+0000 7fb7d4f8d700 1 --2- 172.21.15.162:0/4259228746 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7d00f7320 0x7fb7d00f7d50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.985+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/4259228746 >> 172.21.15.162:0/4259228746 conn(0x7fb7d00f2210 msgr2=0x7fb7d00f4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:35:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.986+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/4259228746 shutdown_connections 2024-02-13T00:35:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.986+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/4259228746 wait complete. 2024-02-13T00:35:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.986+0000 7fb7d4f8d700 1 librados: starting msgr at 2024-02-13T00:35:14.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.986+0000 7fb7d4f8d700 1 librados: starting objecter 2024-02-13T00:35:14.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.986+0000 7fb7d4f8d700 1 Processor -- start 2024-02-13T00:35:14.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.986+0000 7fb7d4f8d700 1 -- start start 2024-02-13T00:35:14.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.986+0000 7fb7d4f8d700 1 librados: setting wanted keys 2024-02-13T00:35:14.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.986+0000 7fb7d4f8d700 1 librados: calling monclient init 2024-02-13T00:35:14.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.986+0000 7fb7d4f8d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7d00f6a10 0x7fb7d010a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:35:14.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.986+0000 7fb7d4f8d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7d00f7320 0x7fb7d010ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.986+0000 7fb7d4f8d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d00f8430 0x7fb7d0111cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7d4f8d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d01121e0 con 0x7fb7d00f6a10 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7d4f8d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0112320 con 0x7fb7d00f8430 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7d4f8d700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0112460 con 0x7fb7d00f7320 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7cdd9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7d00f7320 0x7fb7d010ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7cdd9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7d00f7320 0x7fb7d010ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:39992/0 (socket says 172.21.15.162:39992) 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7ce59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7d00f6a10 0x7fb7d010a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7cdd9b700 1 -- 172.21.15.162:0/3509933360 learned_addr learned my addr 172.21.15.162:0/3509933360 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7ce59c700 1 -- 172.21.15.162:0/3509933360 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7d00f7320 msgr2=0x7fb7d010ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7ce59c700 1 --2- 172.21.15.162:0/3509933360 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7d00f7320 0x7fb7d010ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7ced9d700 1 --2- 172.21.15.162:0/3509933360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d00f8430 0x7fb7d0111cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7ce59c700 1 -- 172.21.15.162:0/3509933360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d00f8430 msgr2=0x7fb7d0111cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:35:14.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7ce59c700 1 --2- 172.21.15.162:0/3509933360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d00f8430 0x7fb7d0111cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7ce59c700 1 -- 172.21.15.162:0/3509933360 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c0005040 con 0x7fb7d00f6a10 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7cdd9b700 1 --2- 172.21.15.162:0/3509933360 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7d00f7320 0x7fb7d010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7ced9d700 1 --2- 172.21.15.162:0/3509933360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d00f8430 0x7fb7d0111cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7ce59c700 1 --2- 172.21.15.162:0/3509933360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7d00f6a10 0x7fb7d010a720 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb7c0004920 tx=0x7fb7c0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7c77fe700 10 client.?.objecter ms_handle_connect 0x7fb7d00f6a10 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7c77fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7c77fe700 1 -- 172.21.15.162:0/3509933360 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0007a40 con 0x7fb7d00f6a10 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/3509933360 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b8005040 con 0x7fb7d00f6a10 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7c77fe700 1 -- 172.21.15.162:0/3509933360 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7c000f040 con 0x7fb7d00f6a10 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7c77fe700 1 -- 172.21.15.162:0/3509933360 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0014a70 con 0x7fb7d00f6a10 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7d4f8d700 10 client.14265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/3509933360 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d0112af0 con 0x7fb7d00f6a10 2024-02-13T00:35:14.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.987+0000 7fb7d4f8d700 1 librados: init done 2024-02-13T00:35:14.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.988+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/3509933360 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d0060070 con 0x7fb7d00f6a10 2024-02-13T00:35:14.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.989+0000 7fb7c77fe700 1 -- 172.21.15.162:0/3509933360 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7fb7c0010530 con 0x7fb7d00f6a10 2024-02-13T00:35:14.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.989+0000 7fb7c77fe700 1 --2- 172.21.15.162:0/3509933360 >> v2:172.21.15.162:6800/1909610488 conn(0x7fb7bc062ba0 0x7fb7bc065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:35:14.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.989+0000 7fb7c77fe700 1 -- 172.21.15.162:0/3509933360 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 1776+0+0 (secure 0 0 0) 0x7fb7c007b360 con 0x7fb7d00f6a10 2024-02-13T00:35:14.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.989+0000 7fb7c77fe700 10 client.14265.objecter ms_dispatch 0x7fb7d0001550 osd_map(14..14 src has 1..14) v4 2024-02-13T00:35:14.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.989+0000 7fb7c77fe700 3 client.14265.objecter handle_osd_map got epochs [14,14] > 0 2024-02-13T00:35:14.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.989+0000 7fb7c77fe700 3 client.14265.objecter handle_osd_map decoding full epoch 14 2024-02-13T00:35:14.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.989+0000 7fb7cdd9b700 1 --2- 172.21.15.162:0/3509933360 >> v2:172.21.15.162:6800/1909610488 conn(0x7fb7bc062ba0 0x7fb7bc065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:35:14.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.989+0000 7fb7c77fe700 20 client.14265.objecter dump_active .. 0 homeless 2024-02-13T00:35:14.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.989+0000 7fb7c77fe700 10 client.14265.objecter ms_handle_connect 0x7fb7bc062ba0 2024-02-13T00:35:14.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.989+0000 7fb7cdd9b700 1 --2- 172.21.15.162:0/3509933360 >> v2:172.21.15.162:6800/1909610488 conn(0x7fb7bc062ba0 0x7fb7bc065060 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fb7b8004540 tx=0x7fb7b8016040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:35:14.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:14.993+0000 7fb7c77fe700 1 -- 172.21.15.162:0/3509933360 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb7c0049f60 con 0x7fb7d00f6a10 2024-02-13T00:35:15.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:15.163+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/3509933360 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7d0101a20 con 0x7fb7bc062ba0 2024-02-13T00:35:15.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:15.167068+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:35:15.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:15.169022+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:35:15.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:15.169562+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:15.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:15.167068+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:35:15.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:15.169022+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:35:15.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:15.169562+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:15.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:15.167068+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:35:15.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:15.169022+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:35:15.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:15.169562+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:16.538 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:15.165099+0000 mgr.y (mgr.14146) 102 : audit [DBG] from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:35:16.538 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:15.496885+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:16.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:15.165099+0000 mgr.y (mgr.14146) 102 : audit [DBG] from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:35:16.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:15.496885+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:16.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:15.165099+0000 mgr.y (mgr.14146) 102 : audit [DBG] from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:35:16.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:15.496885+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:18.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:17.497150+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:18.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:17.556742+0000 mon.a (mon.0) 298 : audit [INF] from='client.? 172.21.15.162:0/427637141' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e"}]: dispatch 2024-02-13T00:35:18.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:17.560129+0000 mon.a (mon.0) 299 : audit [INF] from='client.? 172.21.15.162:0/427637141' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e"}]': finished 2024-02-13T00:35:18.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:17.560170+0000 mon.a (mon.0) 300 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-13T00:35:18.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:17.560287+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:18.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:18.155428+0000 mon.c (mon.2) 7 : audit [DBG] from='client.? 172.21.15.162:0/3207630399' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:35:18.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:17.497150+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:18.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:17.556742+0000 mon.a (mon.0) 298 : audit [INF] from='client.? 172.21.15.162:0/427637141' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e"}]: dispatch 2024-02-13T00:35:18.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:17.560129+0000 mon.a (mon.0) 299 : audit [INF] from='client.? 172.21.15.162:0/427637141' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e"}]': finished 2024-02-13T00:35:18.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:17.560170+0000 mon.a (mon.0) 300 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-13T00:35:18.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:17.560287+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:18.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:18.155428+0000 mon.c (mon.2) 7 : audit [DBG] from='client.? 172.21.15.162:0/3207630399' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:35:19.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:17.497150+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:19.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:17.556742+0000 mon.a (mon.0) 298 : audit [INF] from='client.? 172.21.15.162:0/427637141' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e"}]: dispatch 2024-02-13T00:35:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:17.560129+0000 mon.a (mon.0) 299 : audit [INF] from='client.? 172.21.15.162:0/427637141' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e"}]': finished 2024-02-13T00:35:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:17.560170+0000 mon.a (mon.0) 300 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-13T00:35:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:17.560287+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:18.155428+0000 mon.c (mon.2) 7 : audit [DBG] from='client.? 172.21.15.162:0/3207630399' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:35:19.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:19.988+0000 7fb7cffff700 10 client.14265.objecter tick 2024-02-13T00:35:20.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:19.497432+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:20.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:19.497432+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:20.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:19.497432+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:22.687 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:21.497701+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:22.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:21.497701+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:23.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:21.497701+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:24.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:23.498011+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:24.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:23.701133+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T00:35:24.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:23.701806+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:35:23.702333+0000 mgr.y (mgr.14146) 108 : cephadm [INF] Deploying daemon osd.2 on smithi162 2024-02-13T00:35:24.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:23.498011+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:24.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:23.701133+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T00:35:24.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:23.701806+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:24.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:35:23.702333+0000 mgr.y (mgr.14146) 108 : cephadm [INF] Deploying daemon osd.2 on smithi162 2024-02-13T00:35:24.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:24.989+0000 7fb7cffff700 10 client.14265.objecter tick 2024-02-13T00:35:25.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:23.498011+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:25.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:23.701133+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T00:35:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:23.701806+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:35:23.702333+0000 mgr.y (mgr.14146) 108 : cephadm [INF] Deploying daemon osd.2 on smithi162 2024-02-13T00:35:26.626 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:25.498287+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:26.627 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:25.498287+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:27.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:25.498287+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:28.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:27.498881+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:28.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:27.498881+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:27.498881+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:29.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:29.990+0000 7fb7cffff700 10 client.14265.objecter tick 2024-02-13T00:35:30.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:28.734854+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:30.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:28.761109+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:30.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:28.734854+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:30.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:28.761109+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:30.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:28.734854+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:30.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:28.761109+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:31.017 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:29.499144+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:31.017 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:29.499144+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:31.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:29.499144+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:31.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.939+0000 7fb7c77fe700 1 -- 172.21.15.162:0/3509933360 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb7d0101a20 con 0x7fb7bc062ba0 2024-02-13T00:35:31.943 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 2 on host 'smithi162' 2024-02-13T00:35:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 10 librados: watch_flush enter 2024-02-13T00:35:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 10 librados: watch_flush exit 2024-02-13T00:35:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 20 client.14265.objecter shutdown clearing up homeless session... 2024-02-13T00:35:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 10 client.14265.objecter successfully canceled tick 2024-02-13T00:35:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/3509933360 >> v2:172.21.15.162:6800/1909610488 conn(0x7fb7bc062ba0 msgr2=0x7fb7bc065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:35:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 --2- 172.21.15.162:0/3509933360 >> v2:172.21.15.162:6800/1909610488 conn(0x7fb7bc062ba0 0x7fb7bc065060 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fb7b8004540 tx=0x7fb7b8016040).stop 2024-02-13T00:35:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/3509933360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7d00f6a10 msgr2=0x7fb7d010a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:35:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 --2- 172.21.15.162:0/3509933360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7d00f6a10 0x7fb7d010a720 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb7c0004920 tx=0x7fb7c0009cb0).stop 2024-02-13T00:35:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/3509933360 shutdown_connections 2024-02-13T00:35:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 --2- 172.21.15.162:0/3509933360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7d00f8430 0x7fb7d0111cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 --2- 172.21.15.162:0/3509933360 >> v2:172.21.15.162:6800/1909610488 conn(0x7fb7bc062ba0 0x7fb7bc065060 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 --2- 172.21.15.162:0/3509933360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7d00f6a10 0x7fb7d010a720 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 --2- 172.21.15.162:0/3509933360 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7d00f7320 0x7fb7d010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/3509933360 >> 172.21.15.162:0/3509933360 conn(0x7fb7d00f2210 msgr2=0x7fb7d00f4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:35:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/3509933360 shutdown_connections 2024-02-13T00:35:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 -- 172.21.15.162:0/3509933360 wait complete. 2024-02-13T00:35:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:31.941+0000 7fb7d4f8d700 1 librados: shutdown 2024-02-13T00:35:32.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:31.280256+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:32.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:31.280941+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:31.281505+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:35:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:31.285323+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:31.499519+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:31.806150+0000 mon.a (mon.0) 310 : audit [INF] from='osd.2 v2:172.21.15.162:6809/3689719376' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T00:35:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:31.939458+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:31.951864+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:32.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:31.280256+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:32.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:31.280941+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:32.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:31.281505+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:35:32.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:31.285323+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:32.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:31.499519+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:32.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:31.806150+0000 mon.a (mon.0) 310 : audit [INF] from='osd.2 v2:172.21.15.162:6809/3689719376' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T00:35:32.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:31.939458+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:32.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:31.951864+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:32.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:31.280256+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:32.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:31.280941+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:32.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:31.281505+0000 mon.a (mon.0) 308 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:35:32.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:31.285323+0000 mon.a (mon.0) 309 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:32.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:31.499519+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:32.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:31.806150+0000 mon.a (mon.0) 310 : audit [INF] from='osd.2 v2:172.21.15.162:6809/3689719376' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T00:35:32.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:31.939458+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:32.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:31.951864+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:32.697 DEBUG:teuthology.orchestra.run.smithi162:osd.2> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.2.service 2024-02-13T00:35:32.699 INFO:tasks.cephadm:Deploying osd.3 on smithi162 with /dev/vg_nvme/lv_1... 2024-02-13T00:35:32.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-13T00:35:32.731 INFO:journalctl@ceph.osd.2.smithi162.stdout:-- Logs begin at Tue 2024-02-13 00:17:45 UTC. -- 2024-02-13T00:35:33.494 INFO:journalctl@ceph.osd.2.smithi162.stdout:Feb 13 00:35:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-2[58993]: debug 2024-02-13T00:35:33.292+0000 7f7a14bc0700 -1 osd.2 0 waiting for initial osdmap 2024-02-13T00:35:33.494 INFO:journalctl@ceph.osd.2.smithi162.stdout:Feb 13 00:35:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-2[58993]: debug 2024-02-13T00:35:33.297+0000 7f7a0f338700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T00:35:33.495 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:32.287903+0000 mon.a (mon.0) 313 : audit [INF] from='osd.2 v2:172.21.15.162:6809/3689719376' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-13T00:35:33.495 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:32.287954+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-13T00:35:33.495 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:32.288059+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:33.495 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:32.288377+0000 mon.a (mon.0) 316 : audit [INF] from='osd.2 v2:172.21.15.162:6809/3689719376' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:35:33.495 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:32.287903+0000 mon.a (mon.0) 313 : audit [INF] from='osd.2 v2:172.21.15.162:6809/3689719376' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-13T00:35:33.495 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:32.287954+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-13T00:35:33.495 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:32.288059+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:33.495 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:32.288377+0000 mon.a (mon.0) 316 : audit [INF] from='osd.2 v2:172.21.15.162:6809/3689719376' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:35:33.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:32.287903+0000 mon.a (mon.0) 313 : audit [INF] from='osd.2 v2:172.21.15.162:6809/3689719376' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-13T00:35:33.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:32.287954+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-13T00:35:33.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:32.288059+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:33.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:32.288377+0000 mon.a (mon.0) 316 : audit [INF] from='osd.2 v2:172.21.15.162:6809/3689719376' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:35:34.452 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:33.290496+0000 mon.a (mon.0) 317 : audit [INF] from='osd.2 v2:172.21.15.162:6809/3689719376' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-13T00:35:34.452 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:33.290557+0000 mon.a (mon.0) 318 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-13T00:35:34.452 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:33.290879+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:34.452 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:33.295471+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:34.453 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:33.499857+0000 mgr.y (mgr.14146) 113 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:34.453 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:33.290496+0000 mon.a (mon.0) 317 : audit [INF] from='osd.2 v2:172.21.15.162:6809/3689719376' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-13T00:35:34.453 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:33.290557+0000 mon.a (mon.0) 318 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-13T00:35:34.453 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:33.290879+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:34.453 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:33.295471+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:34.453 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:33.499857+0000 mgr.y (mgr.14146) 113 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:34.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:33.290496+0000 mon.a (mon.0) 317 : audit [INF] from='osd.2 v2:172.21.15.162:6809/3689719376' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-13T00:35:34.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:33.290557+0000 mon.a (mon.0) 318 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-13T00:35:34.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:33.290879+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:33.295471+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:33.499857+0000 mgr.y (mgr.14146) 113 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-13T00:35:35.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:32.810640+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:35:35.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:32.810692+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:35:35.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:34.295350+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:35.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:34.299529+0000 mon.a (mon.0) 322 : cluster [INF] osd.2 v2:172.21.15.162:6809/3689719376 boot 2024-02-13T00:35:35.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:34.299562+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-13T00:35:35.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:34.300010+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:35.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:34.300260+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-13T00:35:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:32.810640+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:35:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:32.810692+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:35:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:34.295350+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:34.299529+0000 mon.a (mon.0) 322 : cluster [INF] osd.2 v2:172.21.15.162:6809/3689719376 boot 2024-02-13T00:35:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:34.299562+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-13T00:35:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:34.300010+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:35.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:34.300260+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-13T00:35:35.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:32.810640+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:35:35.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:32.810692+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:35:35.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:34.295350+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:34.299529+0000 mon.a (mon.0) 322 : cluster [INF] osd.2 v2:172.21.15.162:6809/3689719376 boot 2024-02-13T00:35:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:34.299562+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-13T00:35:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:34.300010+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:35:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:34.300260+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-13T00:35:36.447 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-13T00:35:36.447 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-13T00:35:36.447 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-02-13T00:35:36.447 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-02-13T00:35:36.447 INFO:teuthology.orchestra.run.smithi162.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112688 s, 931 MB/s 2024-02-13T00:35:36.447 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 2024-02-13T00:35:36.447 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-02-13T00:35:36.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:35.303866+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-13T00:35:36.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:35.303910+0000 mon.a (mon.0) 327 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-13T00:35:36.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:35.304542+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-13T00:35:36.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:35.500146+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:36.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:35.303866+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-13T00:35:36.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:35.303910+0000 mon.a (mon.0) 327 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-13T00:35:36.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:35.304542+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-13T00:35:36.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:35.500146+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:36.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:35.303866+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-13T00:35:36.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:35.303910+0000 mon.a (mon.0) 327 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-13T00:35:36.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:35.304542+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-13T00:35:36.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:35.500146+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:37.378 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:36.305687+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-13T00:35:37.378 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:36.305791+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-13T00:35:37.378 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:35:37.033223+0000 mgr.y (mgr.14146) 115 : cephadm [INF] Detected new or changed devices on smithi162 2024-02-13T00:35:37.378 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:37.037456+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:37.378 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:37.038243+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:37.378 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:37.038804+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:37.378 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:37.039271+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:37.378 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:35:37.039630+0000 mgr.y (mgr.14146) 116 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 5366M 2024-02-13T00:35:37.378 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:37.043042+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:37.379 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:37.043937+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:37.379 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:37.044543+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:35:37.379 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:37.049180+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:37.379 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:36.305687+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-13T00:35:37.379 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:36.305791+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-13T00:35:37.379 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:35:37.033223+0000 mgr.y (mgr.14146) 115 : cephadm [INF] Detected new or changed devices on smithi162 2024-02-13T00:35:37.379 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:37.037456+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:37.379 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:37.038243+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:37.379 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:37.038804+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:37.379 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:37.039271+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:37.379 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:35:37.039630+0000 mgr.y (mgr.14146) 116 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 5366M 2024-02-13T00:35:37.379 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:37.043042+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:37.380 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:37.043937+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:37.380 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:37.044543+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:35:37.380 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:37.049180+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:37.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_1 2024-02-13T00:35:37.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:36.305687+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-13T00:35:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:36.305791+0000 mon.a (mon.0) 330 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-13T00:35:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:35:37.033223+0000 mgr.y (mgr.14146) 115 : cephadm [INF] Detected new or changed devices on smithi162 2024-02-13T00:35:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:37.037456+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:37.038243+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:37.038804+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:37.039271+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:35:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:35:37.039630+0000 mgr.y (mgr.14146) 116 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 5366M 2024-02-13T00:35:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:37.043042+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:37.043937+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:37.044543+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:35:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:37.049180+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:38.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:37.321392+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-13T00:35:38.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:37.500444+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:38.562 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:37.321392+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-13T00:35:38.562 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:37.500444+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:38.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:37.321392+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-13T00:35:38.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:37.500444+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:39.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.330+0000 7f5f08b3b700 1 -- 172.21.15.162:0/322501616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f040f7cd0 msgr2=0x7f5f040f8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:35:39.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.330+0000 7f5f08b3b700 1 --2- 172.21.15.162:0/322501616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f040f7cd0 0x7f5f040f8130 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f5ef4004870 tx=0x7f5ef4009cb0).stop 2024-02-13T00:35:39.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.330+0000 7f5f08b3b700 1 -- 172.21.15.162:0/322501616 shutdown_connections 2024-02-13T00:35:39.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.330+0000 7f5f08b3b700 1 --2- 172.21.15.162:0/322501616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5f040f8620 0x7f5f040fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:39.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.330+0000 7f5f08b3b700 1 --2- 172.21.15.162:0/322501616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f040f6a30 0x7f5f040f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:39.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.330+0000 7f5f08b3b700 1 --2- 172.21.15.162:0/322501616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f040f7cd0 0x7f5f040f8130 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:39.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.331+0000 7f5f08b3b700 1 -- 172.21.15.162:0/322501616 >> 172.21.15.162:0/322501616 conn(0x7f5f040f2230 msgr2=0x7f5f040f4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:35:39.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.331+0000 7f5f08b3b700 1 -- 172.21.15.162:0/322501616 shutdown_connections 2024-02-13T00:35:39.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.331+0000 7f5f08b3b700 1 -- 172.21.15.162:0/322501616 wait complete. 2024-02-13T00:35:39.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.331+0000 7f5f08b3b700 1 librados: starting msgr at 2024-02-13T00:35:39.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.331+0000 7f5f08b3b700 1 librados: starting objecter 2024-02-13T00:35:39.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.331+0000 7f5f08b3b700 1 Processor -- start 2024-02-13T00:35:39.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.331+0000 7f5f08b3b700 1 -- start start 2024-02-13T00:35:39.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.331+0000 7f5f08b3b700 1 librados: setting wanted keys 2024-02-13T00:35:39.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.331+0000 7f5f08b3b700 1 librados: calling monclient init 2024-02-13T00:35:39.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.331+0000 7f5f08b3b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f040f6a30 0x7f5f04101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:35:39.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f08b3b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f040f7cd0 0x7f5f040ff860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:35:39.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f08b3b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5f040f8620 0x7f5f040ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:35:39.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f08b3b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f04100370 con 0x7f5f040f6a30 2024-02-13T00:35:39.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f08b3b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f041004e0 con 0x7f5f040f7cd0 2024-02-13T00:35:39.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f08b3b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f041006a0 con 0x7f5f040f8620 2024-02-13T00:35:39.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f0259c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f040f6a30 0x7f5f04101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:35:39.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f02d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5f040f8620 0x7f5f040ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:35:39.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f0259c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f040f6a30 0x7f5f04101200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34624/0 (socket says 172.21.15.162:34624) 2024-02-13T00:35:39.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f02d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5f040f8620 0x7f5f040ffd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:33246/0 (socket says 172.21.15.162:33246) 2024-02-13T00:35:39.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f0259c700 1 -- 172.21.15.162:0/3625107615 learned_addr learned my addr 172.21.15.162:0/3625107615 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:35:39.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f01d9b700 1 --2- 172.21.15.162:0/3625107615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f040f7cd0 0x7f5f040ff860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:35:39.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f0259c700 1 -- 172.21.15.162:0/3625107615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5f040f8620 msgr2=0x7f5f040ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:35:39.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f0259c700 1 --2- 172.21.15.162:0/3625107615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5f040f8620 0x7f5f040ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f0259c700 1 -- 172.21.15.162:0/3625107615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f040f7cd0 msgr2=0x7f5f040ff860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f0259c700 1 --2- 172.21.15.162:0/3625107615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f040f7cd0 0x7f5f040ff860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f0259c700 1 -- 172.21.15.162:0/3625107615 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc0097c0 con 0x7f5f040f6a30 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f01d9b700 1 --2- 172.21.15.162:0/3625107615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f040f7cd0 0x7f5f040ff860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f02d9d700 1 --2- 172.21.15.162:0/3625107615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5f040f8620 0x7f5f040ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f0259c700 1 --2- 172.21.15.162:0/3625107615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f040f6a30 0x7f5f04101200 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f5eec00c840 tx=0x7f5eec007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5efb7fe700 10 client.?.objecter ms_handle_connect 0x7f5f040f6a30 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5efb7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5efb7fe700 1 -- 172.21.15.162:0/3625107615 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec004430 con 0x7f5f040f6a30 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f08b3b700 1 -- 172.21.15.162:0/3625107615 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef4005040 con 0x7f5f040f6a30 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5efb7fe700 1 -- 172.21.15.162:0/3625107615 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5eec00e970 con 0x7f5f040f6a30 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5efb7fe700 1 -- 172.21.15.162:0/3625107615 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec013a90 con 0x7f5f040f6a30 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f08b3b700 10 client.14292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f08b3b700 1 -- 172.21.15.162:0/3625107615 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f041126c0 con 0x7f5f040f6a30 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.332+0000 7f5f08b3b700 1 librados: init done 2024-02-13T00:35:39.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.333+0000 7f5f08b3b700 1 -- 172.21.15.162:0/3625107615 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0405f400 con 0x7f5f040f6a30 2024-02-13T00:35:39.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.334+0000 7f5efb7fe700 1 -- 172.21.15.162:0/3625107615 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7f5eec00e4a0 con 0x7f5f040f6a30 2024-02-13T00:35:39.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.334+0000 7f5efb7fe700 1 --2- 172.21.15.162:0/3625107615 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5ef0062ba0 0x7f5ef0065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:35:39.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.334+0000 7f5efb7fe700 1 -- 172.21.15.162:0/3625107615 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 2484+0+0 (secure 0 0 0) 0x7f5eec07bcd0 con 0x7f5f040f6a30 2024-02-13T00:35:39.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.334+0000 7f5efb7fe700 10 client.14292.objecter ms_dispatch 0x7f5f04001550 osd_map(21..21 src has 1..21) v4 2024-02-13T00:35:39.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.334+0000 7f5efb7fe700 3 client.14292.objecter handle_osd_map got epochs [21,21] > 0 2024-02-13T00:35:39.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.334+0000 7f5f01d9b700 1 --2- 172.21.15.162:0/3625107615 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5ef0062ba0 0x7f5ef0065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:35:39.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.334+0000 7f5efb7fe700 3 client.14292.objecter handle_osd_map decoding full epoch 21 2024-02-13T00:35:39.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.334+0000 7f5efb7fe700 20 client.14292.objecter dump_active .. 0 homeless 2024-02-13T00:35:39.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.335+0000 7f5efb7fe700 10 client.14292.objecter ms_handle_connect 0x7f5ef0062ba0 2024-02-13T00:35:39.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.335+0000 7f5f01d9b700 1 --2- 172.21.15.162:0/3625107615 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5ef0062ba0 0x7f5ef0065060 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f5ef4005010 tx=0x7f5ef400b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:35:39.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.339+0000 7f5efb7fe700 1 -- 172.21.15.162:0/3625107615 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5eec04cd70 con 0x7f5f040f6a30 2024-02-13T00:35:39.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:39.509+0000 7f5f08b3b700 1 -- 172.21.15.162:0/3625107615 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f040fdd80 con 0x7f5ef0062ba0 2024-02-13T00:35:40.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:39.500771+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v97: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:40.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:39.511001+0000 mgr.y (mgr.14146) 119 : audit [DBG] from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:35:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:39.512991+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:35:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:39.514685+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:35:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:39.515147+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:40.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:39.500771+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v97: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:40.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:39.511001+0000 mgr.y (mgr.14146) 119 : audit [DBG] from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:35:40.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:39.512991+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:35:40.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:39.514685+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:35:40.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:39.515147+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:40.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:39.500771+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v97: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:40.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:39.511001+0000 mgr.y (mgr.14146) 119 : audit [DBG] from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:35:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:39.512991+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:35:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:39.514685+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:35:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:39.515147+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:42.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:41.501073+0000 mgr.y (mgr.14146) 120 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:42.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:41.990063+0000 mon.c (mon.2) 8 : audit [INF] from='client.? 172.21.15.162:0/4061340614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001"}]: dispatch 2024-02-13T00:35:42.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:41.990373+0000 mon.a (mon.0) 343 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001"}]: dispatch 2024-02-13T00:35:42.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:41.998708+0000 mon.a (mon.0) 344 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001"}]': finished 2024-02-13T00:35:42.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:41.998772+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-13T00:35:42.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:41.998869+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:41.501073+0000 mgr.y (mgr.14146) 120 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:41.990063+0000 mon.c (mon.2) 8 : audit [INF] from='client.? 172.21.15.162:0/4061340614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001"}]: dispatch 2024-02-13T00:35:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:41.990373+0000 mon.a (mon.0) 343 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001"}]: dispatch 2024-02-13T00:35:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:41.998708+0000 mon.a (mon.0) 344 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001"}]': finished 2024-02-13T00:35:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:41.998772+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-13T00:35:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:41.998869+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:43.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:41.501073+0000 mgr.y (mgr.14146) 120 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:43.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:41.990063+0000 mon.c (mon.2) 8 : audit [INF] from='client.? 172.21.15.162:0/4061340614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001"}]: dispatch 2024-02-13T00:35:43.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:41.990373+0000 mon.a (mon.0) 343 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001"}]: dispatch 2024-02-13T00:35:43.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:41.998708+0000 mon.a (mon.0) 344 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001"}]': finished 2024-02-13T00:35:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:41.998772+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-13T00:35:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:41.998869+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:43.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:42.584207+0000 mon.c (mon.2) 9 : audit [DBG] from='client.? 172.21.15.162:0/90270288' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:35:43.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:42.584207+0000 mon.c (mon.2) 9 : audit [DBG] from='client.? 172.21.15.162:0/90270288' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:35:44.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:42.584207+0000 mon.c (mon.2) 9 : audit [DBG] from='client.? 172.21.15.162:0/90270288' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:35:44.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:44.333+0000 7f5f03fff700 10 client.14292.objecter tick 2024-02-13T00:35:44.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:43.501380+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:44.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:43.501380+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:45.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:43.501380+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:46.814 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:45.501680+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:46.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:45.501680+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:47.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:45.501680+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:47.502072+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:48.140342+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-13T00:35:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:48.140998+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:48.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:47.502072+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:48.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:48.140342+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-13T00:35:48.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:48.140998+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:47.502072+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:48.140342+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-13T00:35:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:48.140998+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:35:49.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:49.334+0000 7f5f03fff700 10 client.14292.objecter tick 2024-02-13T00:35:49.767 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:35:48.141500+0000 mgr.y (mgr.14146) 124 : cephadm [INF] Deploying daemon osd.3 on smithi162 2024-02-13T00:35:49.767 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:35:48.141500+0000 mgr.y (mgr.14146) 124 : cephadm [INF] Deploying daemon osd.3 on smithi162 2024-02-13T00:35:50.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:35:48.141500+0000 mgr.y (mgr.14146) 124 : cephadm [INF] Deploying daemon osd.3 on smithi162 2024-02-13T00:35:50.792 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:49.502372+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:50.792 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:49.502372+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:51.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:49.502372+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:52.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:51.502661+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:51.502661+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:53.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:51.502661+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:54.263 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:53.256999+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:54.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:54.334+0000 7f5f03fff700 10 client.14292.objecter tick 2024-02-13T00:35:54.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:53.256999+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:54.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:53.298455+0000 mon.a (mon.0) 350 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:54.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:53.502945+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:54.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:53.298455+0000 mon.a (mon.0) 350 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:54.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:53.502945+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:54.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:53.256999+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:54.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:53.298455+0000 mon.a (mon.0) 350 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:35:54.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:53.502945+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:56.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.500+0000 7f5efb7fe700 1 -- 172.21.15.162:0/3625107615 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5f040fdd80 con 0x7f5ef0062ba0 2024-02-13T00:35:56.505 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 3 on host 'smithi162' 2024-02-13T00:35:56.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.502+0000 7f5f08b3b700 10 librados: watch_flush enter 2024-02-13T00:35:56.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.502+0000 7f5f08b3b700 10 librados: watch_flush exit 2024-02-13T00:35:56.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.502+0000 7f5f08b3b700 20 client.14292.objecter shutdown clearing up homeless session... 2024-02-13T00:35:56.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.502+0000 7f5f08b3b700 10 client.14292.objecter successfully canceled tick 2024-02-13T00:35:56.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.502+0000 7f5f08b3b700 1 -- 172.21.15.162:0/3625107615 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5ef0062ba0 msgr2=0x7f5ef0065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:35:56.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.502+0000 7f5f08b3b700 1 --2- 172.21.15.162:0/3625107615 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5ef0062ba0 0x7f5ef0065060 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f5ef4005010 tx=0x7f5ef400b040).stop 2024-02-13T00:35:56.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.503+0000 7f5f08b3b700 1 -- 172.21.15.162:0/3625107615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f040f6a30 msgr2=0x7f5f04101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:35:56.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.503+0000 7f5f08b3b700 1 --2- 172.21.15.162:0/3625107615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f040f6a30 0x7f5f04101200 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f5eec00c840 tx=0x7f5eec007f10).stop 2024-02-13T00:35:56.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.503+0000 7f5f08b3b700 1 -- 172.21.15.162:0/3625107615 shutdown_connections 2024-02-13T00:35:56.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.503+0000 7f5f08b3b700 1 --2- 172.21.15.162:0/3625107615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5f040f8620 0x7f5f040ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:56.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.503+0000 7f5f08b3b700 1 --2- 172.21.15.162:0/3625107615 >> v2:172.21.15.162:6800/1909610488 conn(0x7f5ef0062ba0 0x7f5ef0065060 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:56.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.503+0000 7f5f08b3b700 1 --2- 172.21.15.162:0/3625107615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f040f6a30 0x7f5f04101200 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:56.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.503+0000 7f5f08b3b700 1 --2- 172.21.15.162:0/3625107615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f040f7cd0 0x7f5f040ff860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:35:56.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.503+0000 7f5f08b3b700 1 -- 172.21.15.162:0/3625107615 >> 172.21.15.162:0/3625107615 conn(0x7f5f040f2230 msgr2=0x7f5f040f4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:35:56.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.503+0000 7f5f08b3b700 1 -- 172.21.15.162:0/3625107615 shutdown_connections 2024-02-13T00:35:56.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.503+0000 7f5f08b3b700 1 -- 172.21.15.162:0/3625107615 wait complete. 2024-02-13T00:35:56.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:35:56.503+0000 7f5f08b3b700 1 librados: shutdown 2024-02-13T00:35:56.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:55.503230+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:56.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:56.334476+0000 mon.a (mon.0) 351 : audit [INF] from='osd.3 v2:172.21.15.162:6813/604971011' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-13T00:35:56.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:56.501785+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:56.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:55.503230+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:56.334476+0000 mon.a (mon.0) 351 : audit [INF] from='osd.3 v2:172.21.15.162:6813/604971011' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-13T00:35:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:56.501785+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:57.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:55.503230+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:56.334476+0000 mon.a (mon.0) 351 : audit [INF] from='osd.3 v2:172.21.15.162:6813/604971011' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-13T00:35:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:56.501785+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:57.242 DEBUG:teuthology.orchestra.run.smithi162:osd.3> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.3.service 2024-02-13T00:35:57.245 INFO:tasks.cephadm:Deploying osd.4 on smithi181 with /dev/vg_nvme/lv_4... 2024-02-13T00:35:57.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-13T00:35:57.274 INFO:journalctl@ceph.osd.3.smithi162.stdout:-- Logs begin at Tue 2024-02-13 00:17:45 UTC. -- 2024-02-13T00:35:57.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:56.566124+0000 mon.a (mon.0) 353 : audit [INF] from='osd.3 v2:172.21.15.162:6813/604971011' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-13T00:35:57.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:56.566173+0000 mon.a (mon.0) 354 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-13T00:35:57.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:56.566313+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:57.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:56.566649+0000 mon.a (mon.0) 356 : audit [INF] from='osd.3 v2:172.21.15.162:6813/604971011' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:35:57.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:57.373726+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:57.575 INFO:journalctl@ceph.osd.3.smithi162.stdout:Feb 13 00:35:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-3[63463]: debug 2024-02-13T00:35:57.569+0000 7fab6f885700 -1 osd.3 0 waiting for initial osdmap 2024-02-13T00:35:57.829 INFO:journalctl@ceph.osd.3.smithi162.stdout:Feb 13 00:35:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-3[63463]: debug 2024-02-13T00:35:57.575+0000 7fab687fa700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T00:35:57.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:56.566124+0000 mon.a (mon.0) 353 : audit [INF] from='osd.3 v2:172.21.15.162:6813/604971011' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-13T00:35:57.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:56.566173+0000 mon.a (mon.0) 354 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-13T00:35:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:56.566313+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:56.566649+0000 mon.a (mon.0) 356 : audit [INF] from='osd.3 v2:172.21.15.162:6813/604971011' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:35:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:57.373726+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:57.916 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:56.566124+0000 mon.a (mon.0) 353 : audit [INF] from='osd.3 v2:172.21.15.162:6813/604971011' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-13T00:35:57.916 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:56.566173+0000 mon.a (mon.0) 354 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-13T00:35:57.916 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:56.566313+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:57.916 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:56.566649+0000 mon.a (mon.0) 356 : audit [INF] from='osd.3 v2:172.21.15.162:6813/604971011' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-13T00:35:57.916 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:57.373726+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:35:58.688 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:57.503537+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:58.688 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:57.567736+0000 mon.a (mon.0) 358 : audit [INF] from='osd.3 v2:172.21.15.162:6813/604971011' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-13T00:35:58.688 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:57.567808+0000 mon.a (mon.0) 359 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-13T00:35:58.688 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:57.568354+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:58.689 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:57.573488+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:58.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:57.503537+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:58.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:57.567736+0000 mon.a (mon.0) 358 : audit [INF] from='osd.3 v2:172.21.15.162:6813/604971011' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-13T00:35:58.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:57.567808+0000 mon.a (mon.0) 359 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-13T00:35:58.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:57.568354+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:58.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:57.573488+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:58.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:57.503537+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:35:58.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:57.567736+0000 mon.a (mon.0) 358 : audit [INF] from='osd.3 v2:172.21.15.162:6813/604971011' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-13T00:35:58.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:57.567808+0000 mon.a (mon.0) 359 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-13T00:35:58.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:57.568354+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:58.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:57.573488+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:59.462 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-13T00:35:59.462 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-13T00:35:59.462 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-02-13T00:35:59.462 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-02-13T00:35:59.462 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111544 s, 940 MB/s 2024-02-13T00:35:59.462 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-02-13T00:35:59.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:57.380347+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:35:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:57.380400+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:35:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:58.571325+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:35:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:58.574256+0000 mon.a (mon.0) 363 : cluster [INF] osd.3 v2:172.21.15.162:6813/604971011 boot 2024-02-13T00:35:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:58.574293+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-02-13T00:35:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:35:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:35:58.574812+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:36:00.067 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi181:vg_nvme/lv_4 2024-02-13T00:36:00.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:57.380347+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:00.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:57.380400+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:00.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:58.571325+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:36:00.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:58.574256+0000 mon.a (mon.0) 363 : cluster [INF] osd.3 v2:172.21.15.162:6813/604971011 boot 2024-02-13T00:36:00.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:58.574293+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-02-13T00:36:00.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:35:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:35:58.574812+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:36:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:57.380347+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:57.380400+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:58.571325+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:36:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:58.574256+0000 mon.a (mon.0) 363 : cluster [INF] osd.3 v2:172.21.15.162:6813/604971011 boot 2024-02-13T00:36:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:58.574293+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-02-13T00:36:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:35:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:35:58.574812+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:36:00.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:35:59.503870+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:36:00.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:00.181004+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-13T00:36:00.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:00.205819+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:00.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:00.205872+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:00.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:35:59.503870+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:36:00.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:00.181004+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-13T00:36:00.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:00.205819+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:00.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:00.205872+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:35:59.503870+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-13T00:36:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:00.181004+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-13T00:36:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:00.205819+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:00.205872+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:01.774 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.769+0000 7fddfd8ab700 1 -- 172.21.15.181:0/3975648038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddf80fd720 msgr2=0x7fddf80ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:01.774 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.769+0000 7fddfd8ab700 1 --2- 172.21.15.181:0/3975648038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddf80fd720 0x7fddf80ffbb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fddf0009be0 tx=0x7fddf0004d10).stop 2024-02-13T00:36:01.774 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.769+0000 7fddfd8ab700 1 -- 172.21.15.181:0/3975648038 shutdown_connections 2024-02-13T00:36:01.774 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.769+0000 7fddfd8ab700 1 --2- 172.21.15.181:0/3975648038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddf80fd720 0x7fddf80ffbb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:01.774 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.769+0000 7fddfd8ab700 1 --2- 172.21.15.181:0/3975648038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddf8061850 0x7fddf80fa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:01.774 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.769+0000 7fddfd8ab700 1 --2- 172.21.15.181:0/3975648038 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fddf80fac90 0x7fddf80fd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:01.774 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.769+0000 7fddfd8ab700 1 -- 172.21.15.181:0/3975648038 >> 172.21.15.181:0/3975648038 conn(0x7fddf80f2250 msgr2=0x7fddf80f4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:36:01.774 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.769+0000 7fddfd8ab700 1 -- 172.21.15.181:0/3975648038 shutdown_connections 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.769+0000 7fddfd8ab700 1 -- 172.21.15.181:0/3975648038 wait complete. 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.769+0000 7fddfd8ab700 1 librados: starting msgr at 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.769+0000 7fddfd8ab700 1 librados: starting objecter 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.769+0000 7fddfd8ab700 1 Processor -- start 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 1 -- start start 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 1 librados: setting wanted keys 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 1 librados: calling monclient init 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddf8061850 0x7fddf80f7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fddf80fac90 0x7fddf80f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddf80fd720 0x7fddf80f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80f7120 con 0x7fddf80fd720 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80f7290 con 0x7fddf8061850 2024-02-13T00:36:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80f7450 con 0x7fddf80fac90 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddf6ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddf8061850 0x7fddf80f7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddf6ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddf8061850 0x7fddf80f7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:60316/0 (socket says 172.21.15.181:60316) 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddf6ffd700 1 -- 172.21.15.181:0/2855517821 learned_addr learned my addr 172.21.15.181:0/2855517821 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddf67fc700 1 --2- 172.21.15.181:0/2855517821 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fddf80fac90 0x7fddf80f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddf77fe700 1 --2- 172.21.15.181:0/2855517821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddf80fd720 0x7fddf80f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddf6ffd700 1 -- 172.21.15.181:0/2855517821 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fddf80fac90 msgr2=0x7fddf80f6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddf6ffd700 1 --2- 172.21.15.181:0/2855517821 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fddf80fac90 0x7fddf80f6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddf6ffd700 1 -- 172.21.15.181:0/2855517821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddf80fd720 msgr2=0x7fddf80f6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddf6ffd700 1 --2- 172.21.15.181:0/2855517821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddf80fd720 0x7fddf80f6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddf6ffd700 1 -- 172.21.15.181:0/2855517821 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf00097c0 con 0x7fddf8061850 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddf6ffd700 1 --2- 172.21.15.181:0/2855517821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddf8061850 0x7fddf80f7fb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fddec0028a0 tx=0x7fddec009f60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fdddffff700 10 client.?.objecter ms_handle_connect 0x7fddf8061850 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fdddffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:36:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fdddffff700 1 -- 172.21.15.181:0/2855517821 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec007b40 con 0x7fddf8061850 2024-02-13T00:36:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 1 -- 172.21.15.181:0/2855517821 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf81120d0 con 0x7fddf8061850 2024-02-13T00:36:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fdddffff700 1 -- 172.21.15.181:0/2855517821 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddec00f040 con 0x7fddf8061850 2024-02-13T00:36:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fdddffff700 1 -- 172.21.15.181:0/2855517821 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec014730 con 0x7fddf8061850 2024-02-13T00:36:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 10 client.24193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:36:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 1 -- 172.21.15.181:0/2855517821 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf8112690 con 0x7fddf8061850 2024-02-13T00:36:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.770+0000 7fddfd8ab700 1 librados: init done 2024-02-13T00:36:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.771+0000 7fddfd8ab700 1 -- 172.21.15.181:0/2855517821 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf805f400 con 0x7fddf8061850 2024-02-13T00:36:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.772+0000 7fdddffff700 1 -- 172.21.15.181:0/2855517821 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7fddec009110 con 0x7fddf8061850 2024-02-13T00:36:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.772+0000 7fdddffff700 1 --2- 172.21.15.181:0/2855517821 >> v2:172.21.15.162:6800/1909610488 conn(0x7fdde0062ba0 0x7fdde0065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:36:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.772+0000 7fdddffff700 1 -- 172.21.15.181:0/2855517821 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 2776+0+0 (secure 0 0 0) 0x7fddec07bbd0 con 0x7fddf8061850 2024-02-13T00:36:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.772+0000 7fdddffff700 10 client.24193.objecter ms_dispatch 0x7fddf8001550 osd_map(27..27 src has 1..27) v4 2024-02-13T00:36:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.772+0000 7fdddffff700 3 client.24193.objecter handle_osd_map got epochs [27,27] > 0 2024-02-13T00:36:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.772+0000 7fdddffff700 3 client.24193.objecter handle_osd_map decoding full epoch 27 2024-02-13T00:36:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.772+0000 7fddf67fc700 1 --2- 172.21.15.181:0/2855517821 >> v2:172.21.15.162:6800/1909610488 conn(0x7fdde0062ba0 0x7fdde0065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:36:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.772+0000 7fdddffff700 20 client.24193.objecter dump_active .. 0 homeless 2024-02-13T00:36:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.773+0000 7fdddffff700 10 client.24193.objecter ms_handle_connect 0x7fdde0062ba0 2024-02-13T00:36:01.779 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.773+0000 7fddf67fc700 1 --2- 172.21.15.181:0/2855517821 >> v2:172.21.15.162:6800/1909610488 conn(0x7fdde0062ba0 0x7fdde0065060 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fdde80041d0 tx=0x7fdde8009040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:36:01.779 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.777+0000 7fdddffff700 1 -- 172.21.15.181:0/2855517821 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fddec04cb50 con 0x7fddf8061850 2024-02-13T00:36:01.949 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:01.945+0000 7fddfd8ab700 1 -- 172.21.15.181:0/2855517821 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fddf80f4c90 con 0x7fdde0062ba0 2024-02-13T00:36:02.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:36:00.724250+0000 mgr.y (mgr.14146) 131 : cephadm [INF] Detected new or changed devices on smithi162 2024-02-13T00:36:02.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.728717+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:02.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.729556+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.730126+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.730650+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.731176+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:36:00.731580+0000 mgr.y (mgr.14146) 132 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 4024M 2024-02-13T00:36:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.737546+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:02.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.738511+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:02.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.739041+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:02.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.745318+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:02.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.747063+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:02.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.748171+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:02.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.748632+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:02.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:00.754452+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:02.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:01.185334+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-02-13T00:36:02.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:36:00.724250+0000 mgr.y (mgr.14146) 131 : cephadm [INF] Detected new or changed devices on smithi162 2024-02-13T00:36:02.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.728717+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:02.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.729556+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:02.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.730126+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:02.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.730650+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:02.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.731176+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:02.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:36:00.731580+0000 mgr.y (mgr.14146) 132 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 4024M 2024-02-13T00:36:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.737546+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.738511+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.739041+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.745318+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.747063+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.748171+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.748632+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:00.754452+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:01.185334+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-02-13T00:36:02.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:36:00.724250+0000 mgr.y (mgr.14146) 131 : cephadm [INF] Detected new or changed devices on smithi162 2024-02-13T00:36:02.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.728717+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:02.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.729556+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:02.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.730126+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:02.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.730650+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:02.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.731176+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:02.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:36:00.731580+0000 mgr.y (mgr.14146) 132 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 4024M 2024-02-13T00:36:02.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.737546+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:02.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.738511+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:02.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.739041+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:02.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.745318+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:02.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.747063+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:02.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.748171+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:02.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.748632+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:02.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:00.754452+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:02.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:01.185334+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-02-13T00:36:03.031 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:01.504263+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:03.031 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:01.947831+0000 mgr.y (mgr.14146) 134 : audit [DBG] from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:36:03.031 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:01.950398+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:36:03.032 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:01.952813+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:36:03.032 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:01.953516+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:03.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:01.504263+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:03.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:01.947831+0000 mgr.y (mgr.14146) 134 : audit [DBG] from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:36:03.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:01.950398+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:36:03.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:01.952813+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:36:03.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:01.953516+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:03.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:01.504263+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:03.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:01.947831+0000 mgr.y (mgr.14146) 134 : audit [DBG] from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:36:03.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:01.950398+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:36:03.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:01.952813+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:36:03.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:01.953516+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:05.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:03.504739+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:05.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:04.294374+0000 mon.b (mon.1) 4 : audit [INF] from='client.? 172.21.15.181:0/1943640832' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3"}]: dispatch 2024-02-13T00:36:05.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:04.295155+0000 mon.a (mon.0) 384 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3"}]: dispatch 2024-02-13T00:36:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:04.304875+0000 mon.a (mon.0) 385 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3"}]': finished 2024-02-13T00:36:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:04.304936+0000 mon.a (mon.0) 386 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-13T00:36:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:04.305120+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:05.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:03.504739+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:05.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:04.294374+0000 mon.b (mon.1) 4 : audit [INF] from='client.? 172.21.15.181:0/1943640832' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3"}]: dispatch 2024-02-13T00:36:05.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:04.295155+0000 mon.a (mon.0) 384 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3"}]: dispatch 2024-02-13T00:36:05.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:04.304875+0000 mon.a (mon.0) 385 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3"}]': finished 2024-02-13T00:36:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:04.304936+0000 mon.a (mon.0) 386 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-13T00:36:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:04.305120+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:03.504739+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:04.294374+0000 mon.b (mon.1) 4 : audit [INF] from='client.? 172.21.15.181:0/1943640832' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3"}]: dispatch 2024-02-13T00:36:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:04.295155+0000 mon.a (mon.0) 384 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3"}]: dispatch 2024-02-13T00:36:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:04.304875+0000 mon.a (mon.0) 385 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3"}]': finished 2024-02-13T00:36:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:04.304936+0000 mon.a (mon.0) 386 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-13T00:36:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:04.305120+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:06.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:04.959960+0000 mon.b (mon.1) 5 : audit [DBG] from='client.? 172.21.15.181:0/1643954991' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:36:06.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:04.959960+0000 mon.b (mon.1) 5 : audit [DBG] from='client.? 172.21.15.181:0/1643954991' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:36:06.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:04.959960+0000 mon.b (mon.1) 5 : audit [DBG] from='client.? 172.21.15.181:0/1643954991' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:36:06.774 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:06.771+0000 7fddfd0aa700 10 client.24193.objecter tick 2024-02-13T00:36:07.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:05.505131+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:07.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:05.505131+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:07.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:05.505131+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:09.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:07.505502+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:09.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:07.505502+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:09.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:07.505502+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:10.269 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:09.505922+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:10.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:09.505922+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:10.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:09.505922+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:11.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:10.557898+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-13T00:36:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:10.558678+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:36:10.559349+0000 mgr.y (mgr.14146) 139 : cephadm [INF] Deploying daemon osd.4 on smithi181 2024-02-13T00:36:11.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:10.557898+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-13T00:36:11.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:10.558678+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:11.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:36:10.559349+0000 mgr.y (mgr.14146) 139 : cephadm [INF] Deploying daemon osd.4 on smithi181 2024-02-13T00:36:11.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:10.557898+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-13T00:36:11.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:10.558678+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:11.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:36:10.559349+0000 mgr.y (mgr.14146) 139 : cephadm [INF] Deploying daemon osd.4 on smithi181 2024-02-13T00:36:11.775 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:11.772+0000 7fddfd0aa700 10 client.24193.objecter tick 2024-02-13T00:36:12.529 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:11.506307+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:12.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:11.506307+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:12.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:11.506307+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:14.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:13.506876+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:14.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:13.506876+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:14.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:13.506876+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:16.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:15.226877+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:16.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:15.255350+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:16.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:15.507228+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:16.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:15.226877+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:16.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:15.255350+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:16.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:15.507228+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:15.226877+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:15.255350+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:15.507228+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:16.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:16.772+0000 7fddfd0aa700 10 client.24193.objecter tick 2024-02-13T00:36:18.473 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 4 on host 'smithi181' 2024-02-13T00:36:18.474 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.469+0000 7fdddffff700 1 -- 172.21.15.181:0/2855517821 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fddf80f4c90 con 0x7fdde0062ba0 2024-02-13T00:36:18.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.471+0000 7fddfd8ab700 10 librados: watch_flush enter 2024-02-13T00:36:18.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.471+0000 7fddfd8ab700 10 librados: watch_flush exit 2024-02-13T00:36:18.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.471+0000 7fddfd8ab700 20 client.24193.objecter shutdown clearing up homeless session... 2024-02-13T00:36:18.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.471+0000 7fddfd8ab700 10 client.24193.objecter successfully canceled tick 2024-02-13T00:36:18.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 -- 172.21.15.181:0/2855517821 >> v2:172.21.15.162:6800/1909610488 conn(0x7fdde0062ba0 msgr2=0x7fdde0065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:18.475 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 --2- 172.21.15.181:0/2855517821 >> v2:172.21.15.162:6800/1909610488 conn(0x7fdde0062ba0 0x7fdde0065060 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fdde80041d0 tx=0x7fdde8009040).stop 2024-02-13T00:36:18.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 -- 172.21.15.181:0/2855517821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddf8061850 msgr2=0x7fddf80f7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:18.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 --2- 172.21.15.181:0/2855517821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddf8061850 0x7fddf80f7fb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fddec0028a0 tx=0x7fddec009f60).stop 2024-02-13T00:36:18.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 -- 172.21.15.181:0/2855517821 shutdown_connections 2024-02-13T00:36:18.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 --2- 172.21.15.181:0/2855517821 >> v2:172.21.15.162:6800/1909610488 conn(0x7fdde0062ba0 0x7fdde0065060 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:18.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 --2- 172.21.15.181:0/2855517821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddf80fd720 0x7fddf80f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:18.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 --2- 172.21.15.181:0/2855517821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddf8061850 0x7fddf80f7fb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:18.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 --2- 172.21.15.181:0/2855517821 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fddf80fac90 0x7fddf80f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:18.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 -- 172.21.15.181:0/2855517821 >> 172.21.15.181:0/2855517821 conn(0x7fddf80f2250 msgr2=0x7fddf80f9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:36:18.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 -- 172.21.15.181:0/2855517821 shutdown_connections 2024-02-13T00:36:18.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 -- 172.21.15.181:0/2855517821 wait complete. 2024-02-13T00:36:18.476 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:18.472+0000 7fddfd8ab700 1 librados: shutdown 2024-02-13T00:36:19.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:17.507646+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:17.744966+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:17.745749+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:17.746546+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:17.754036+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:18.354876+0000 mon.a (mon.0) 396 : audit [INF] from='osd.4 v2:172.21.15.181:6800/3344263388' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T00:36:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:18.469853+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:18.471680+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:17.507646+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:17.744966+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:17.745749+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:17.746546+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:17.754036+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:18.354876+0000 mon.a (mon.0) 396 : audit [INF] from='osd.4 v2:172.21.15.181:6800/3344263388' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T00:36:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:18.469853+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:18.471680+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:17.507646+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:17.744966+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:17.745749+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:17.746546+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:17.754036+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:18.354876+0000 mon.a (mon.0) 396 : audit [INF] from='osd.4 v2:172.21.15.181:6800/3344263388' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-13T00:36:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:18.469853+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:18.471680+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:19.148 DEBUG:teuthology.orchestra.run.smithi181:osd.4> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.4.service 2024-02-13T00:36:19.151 INFO:tasks.cephadm:Deploying osd.5 on smithi181 with /dev/vg_nvme/lv_3... 2024-02-13T00:36:19.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-13T00:36:19.183 INFO:journalctl@ceph.osd.4.smithi181.stdout:-- Logs begin at Tue 2024-02-13 00:17:30 UTC. -- 2024-02-13T00:36:20.045 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:36:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:36:19.762+0000 7f54d38df700 -1 osd.4 0 waiting for initial osdmap 2024-02-13T00:36:20.045 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:36:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:36:19.768+0000 7f54cc854700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T00:36:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:18.755669+0000 mon.a (mon.0) 399 : audit [INF] from='osd.4 v2:172.21.15.181:6800/3344263388' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-13T00:36:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:18.755744+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-13T00:36:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:18.755876+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:18.756268+0000 mon.a (mon.0) 402 : audit [INF] from='osd.4 v2:172.21.15.181:6800/3344263388' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:36:20.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:18.755669+0000 mon.a (mon.0) 399 : audit [INF] from='osd.4 v2:172.21.15.181:6800/3344263388' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-13T00:36:20.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:18.755744+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-13T00:36:20.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:18.755876+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:20.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:18.756268+0000 mon.a (mon.0) 402 : audit [INF] from='osd.4 v2:172.21.15.181:6800/3344263388' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:36:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:18.755669+0000 mon.a (mon.0) 399 : audit [INF] from='osd.4 v2:172.21.15.181:6800/3344263388' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-13T00:36:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:18.755744+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-13T00:36:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:18.755876+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:18.756268+0000 mon.a (mon.0) 402 : audit [INF] from='osd.4 v2:172.21.15.181:6800/3344263388' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:36:21.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:19.508024+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:21.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:19.757392+0000 mon.a (mon.0) 403 : audit [INF] from='osd.4 v2:172.21.15.181:6800/3344263388' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-13T00:36:21.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:19.757453+0000 mon.a (mon.0) 404 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-13T00:36:21.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:19.760504+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:21.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:19.765375+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:21.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:19.508024+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:21.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:19.757392+0000 mon.a (mon.0) 403 : audit [INF] from='osd.4 v2:172.21.15.181:6800/3344263388' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-13T00:36:21.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:19.757453+0000 mon.a (mon.0) 404 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-13T00:36:21.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:19.760504+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:21.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:19.765375+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:21.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:19.508024+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:21.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:19.757392+0000 mon.a (mon.0) 403 : audit [INF] from='osd.4 v2:172.21.15.181:6800/3344263388' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-13T00:36:21.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:19.757453+0000 mon.a (mon.0) 404 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-13T00:36:21.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:19.760504+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:21.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:19.765375+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:21.564 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-13T00:36:21.565 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-13T00:36:21.565 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-02-13T00:36:21.565 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-02-13T00:36:21.565 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123726 s, 847 MB/s 2024-02-13T00:36:21.565 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-02-13T00:36:21.941 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:19.357237+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:21.941 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:19.357285+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:21.941 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:20.766102+0000 mon.a (mon.0) 407 : cluster [INF] osd.4 v2:172.21.15.181:6800/3344263388 boot 2024-02-13T00:36:21.941 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:20.766140+0000 mon.a (mon.0) 408 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-02-13T00:36:21.941 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:20.767002+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:22.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:19.357237+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:22.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:19.357285+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:22.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:20.766102+0000 mon.a (mon.0) 407 : cluster [INF] osd.4 v2:172.21.15.181:6800/3344263388 boot 2024-02-13T00:36:22.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:20.766140+0000 mon.a (mon.0) 408 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-02-13T00:36:22.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:20.767002+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:19.357237+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:19.357285+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:20.766102+0000 mon.a (mon.0) 407 : cluster [INF] osd.4 v2:172.21.15.181:6800/3344263388 boot 2024-02-13T00:36:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:20.766140+0000 mon.a (mon.0) 408 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-02-13T00:36:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:20.767002+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:36:22.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi181:vg_nvme/lv_3 2024-02-13T00:36:23.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:21.508474+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v128: 1 pgs: 1 remapped+peering; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:21.780987+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-13T00:36:23.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:21.508474+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v128: 1 pgs: 1 remapped+peering; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:23.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:21.780987+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-13T00:36:23.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:21.508474+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v128: 1 pgs: 1 remapped+peering; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:36:23.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:21.780987+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-13T00:36:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:22.785656+0000 mon.a (mon.0) 411 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-13T00:36:24.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:22.785656+0000 mon.a (mon.0) 411 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-13T00:36:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:22.785656+0000 mon.a (mon.0) 411 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-13T00:36:24.932 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.917+0000 7f3118f2c700 1 -- 172.21.15.181:0/2463712799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31140f6a40 msgr2=0x7f31140f6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:24.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.917+0000 7f3118f2c700 1 --2- 172.21.15.181:0/2463712799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31140f6a40 0x7f31140f6e60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f31040048a0 tx=0x7f3104009cb0).stop 2024-02-13T00:36:24.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.917+0000 7f3118f2c700 1 -- 172.21.15.181:0/2463712799 shutdown_connections 2024-02-13T00:36:24.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.917+0000 7f3118f2c700 1 --2- 172.21.15.181:0/2463712799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31140f8630 0x7f31140fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:24.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.917+0000 7f3118f2c700 1 --2- 172.21.15.181:0/2463712799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31140f6a40 0x7f31140f6e60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:24.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.917+0000 7f3118f2c700 1 --2- 172.21.15.181:0/2463712799 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31140f7ce0 0x7f31140f8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:24.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.917+0000 7f3118f2c700 1 -- 172.21.15.181:0/2463712799 >> 172.21.15.181:0/2463712799 conn(0x7f31140f2240 msgr2=0x7f31140f4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:36:24.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.917+0000 7f3118f2c700 1 -- 172.21.15.181:0/2463712799 shutdown_connections 2024-02-13T00:36:24.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.917+0000 7f3118f2c700 1 -- 172.21.15.181:0/2463712799 wait complete. 2024-02-13T00:36:24.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3118f2c700 1 librados: starting msgr at 2024-02-13T00:36:24.934 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3118f2c700 1 librados: starting objecter 2024-02-13T00:36:24.934 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3118f2c700 1 Processor -- start 2024-02-13T00:36:24.934 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3118f2c700 1 -- start start 2024-02-13T00:36:24.934 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3118f2c700 1 librados: setting wanted keys 2024-02-13T00:36:24.934 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3118f2c700 1 librados: calling monclient init 2024-02-13T00:36:24.934 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3118f2c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31140f6a40 0x7f3114106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:36:24.934 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3118f2c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31140f7ce0 0x7f31141073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:36:24.934 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3118f2c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31140f8630 0x7f311410e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:36:24.934 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3118f2c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311410e9a0 con 0x7f31140f6a40 2024-02-13T00:36:24.934 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3118f2c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311410eae0 con 0x7f31140f7ce0 2024-02-13T00:36:24.935 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3118f2c700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311410ec20 con 0x7f31140f8630 2024-02-13T00:36:24.935 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f311259c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31140f6a40 0x7f3114106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:36:24.935 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3112d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31140f8630 0x7f311410e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:36:24.935 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3111d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31140f7ce0 0x7f31141073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:36:24.935 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f3112d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31140f8630 0x7f311410e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.181:51444/0 (socket says 172.21.15.181:51444) 2024-02-13T00:36:24.936 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f3112d9d700 1 -- 172.21.15.181:0/4284626907 learned_addr learned my addr 172.21.15.181:0/4284626907 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:36:24.936 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.918+0000 7f311259c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31140f6a40 0x7f3114106ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.181:55390/0 (socket says 172.21.15.181:55390) 2024-02-13T00:36:24.936 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f3111d9b700 1 -- 172.21.15.181:0/4284626907 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31140f8630 msgr2=0x7f311410e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:24.936 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f3111d9b700 1 --2- 172.21.15.181:0/4284626907 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31140f8630 0x7f311410e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:24.937 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f3111d9b700 1 -- 172.21.15.181:0/4284626907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31140f6a40 msgr2=0x7f3114106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:24.937 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f3111d9b700 1 --2- 172.21.15.181:0/4284626907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31140f6a40 0x7f3114106ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:24.938 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f3111d9b700 1 -- 172.21.15.181:0/4284626907 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3104005040 con 0x7f31140f7ce0 2024-02-13T00:36:24.938 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f3111d9b700 1 --2- 172.21.15.181:0/4284626907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31140f7ce0 0x7f31141073d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f3108002760 tx=0x7f3108009e80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:36:24.938 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f31037fe700 10 client.?.objecter ms_handle_connect 0x7f31140f7ce0 2024-02-13T00:36:24.938 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f31037fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:36:24.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f31037fe700 1 -- 172.21.15.181:0/4284626907 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108007b40 con 0x7f31140f7ce0 2024-02-13T00:36:24.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f31037fe700 1 -- 172.21.15.181:0/4284626907 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f310800f040 con 0x7f31140f7ce0 2024-02-13T00:36:24.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f31037fe700 1 -- 172.21.15.181:0/4284626907 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108014790 con 0x7f31140f7ce0 2024-02-13T00:36:24.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f3118f2c700 1 -- 172.21.15.181:0/4284626907 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3114060b40 con 0x7f31140f7ce0 2024-02-13T00:36:24.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f3118f2c700 10 client.24220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:36:24.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f3118f2c700 1 -- 172.21.15.181:0/4284626907 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31140610b0 con 0x7f31140f7ce0 2024-02-13T00:36:24.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.919+0000 7f3118f2c700 1 librados: init done 2024-02-13T00:36:24.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.920+0000 7f3118f2c700 1 -- 172.21.15.181:0/4284626907 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311405f400 con 0x7f31140f7ce0 2024-02-13T00:36:24.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.921+0000 7f31037fe700 1 -- 172.21.15.181:0/4284626907 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7f3108007ca0 con 0x7f31140f7ce0 2024-02-13T00:36:24.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.921+0000 7f31037fe700 1 --2- 172.21.15.181:0/4284626907 >> v2:172.21.15.162:6800/1909610488 conn(0x7f30fc062ba0 0x7f30fc065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:36:24.940 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.921+0000 7f31037fe700 1 -- 172.21.15.181:0/4284626907 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 3178+0+0 (secure 0 0 0) 0x7f310807b6d0 con 0x7f31140f7ce0 2024-02-13T00:36:24.940 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.921+0000 7f31037fe700 10 client.24220.objecter ms_dispatch 0x7f3114001550 osd_map(33..33 src has 1..33) v4 2024-02-13T00:36:24.940 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.921+0000 7f31037fe700 3 client.24220.objecter handle_osd_map got epochs [33,33] > 0 2024-02-13T00:36:24.940 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.921+0000 7f31037fe700 3 client.24220.objecter handle_osd_map decoding full epoch 33 2024-02-13T00:36:24.940 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.921+0000 7f31037fe700 20 client.24220.objecter dump_active .. 0 homeless 2024-02-13T00:36:24.940 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.921+0000 7f311259c700 1 --2- 172.21.15.181:0/4284626907 >> v2:172.21.15.162:6800/1909610488 conn(0x7f30fc062ba0 0x7f30fc065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:36:24.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.922+0000 7f31037fe700 10 client.24220.objecter ms_handle_connect 0x7f30fc062ba0 2024-02-13T00:36:24.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.922+0000 7f311259c700 1 --2- 172.21.15.181:0/4284626907 >> v2:172.21.15.162:6800/1909610488 conn(0x7f30fc062ba0 0x7f30fc065060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f3104009da0 tx=0x7f3104009cb0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:36:24.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:24.925+0000 7f31037fe700 1 -- 172.21.15.181:0/4284626907 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f310804c560 con 0x7f31140f7ce0 2024-02-13T00:36:25.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:23.508914+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v131: 1 pgs: 1 remapped+peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:25.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:23.508914+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v131: 1 pgs: 1 remapped+peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:25.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:23.508914+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v131: 1 pgs: 1 remapped+peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:25.103 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:25.099+0000 7f3118f2c700 1 -- 172.21.15.181:0/4284626907 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f31140fd9d0 con 0x7f30fc062ba0 2024-02-13T00:36:26.280 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:36:24.969612+0000 mgr.y (mgr.14146) 147 : cephadm [INF] Detected new or changed devices on smithi181 2024-02-13T00:36:26.280 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:24.975351+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:26.280 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:24.976244+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:26.280 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:36:24.976845+0000 mgr.y (mgr.14146) 148 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 17122M 2024-02-13T00:36:26.280 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:24.981775+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:26.280 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:24.982694+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:26.280 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:24.983386+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:26.281 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:24.988455+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:26.281 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:25.101881+0000 mgr.y (mgr.14146) 149 : audit [DBG] from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:36:26.281 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:25.104218+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:36:26.281 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:25.106426+0000 mon.a (mon.0) 419 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:36:26.281 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:25.107187+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:26.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:36:24.969612+0000 mgr.y (mgr.14146) 147 : cephadm [INF] Detected new or changed devices on smithi181 2024-02-13T00:36:26.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:24.975351+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:24.976244+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:36:24.976845+0000 mgr.y (mgr.14146) 148 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 17122M 2024-02-13T00:36:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:24.981775+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:24.982694+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:24.983386+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:24.988455+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:25.101881+0000 mgr.y (mgr.14146) 149 : audit [DBG] from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:36:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:25.104218+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:36:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:25.106426+0000 mon.a (mon.0) 419 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:36:26.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:25.107187+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:26.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:36:24.969612+0000 mgr.y (mgr.14146) 147 : cephadm [INF] Detected new or changed devices on smithi181 2024-02-13T00:36:26.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:24.975351+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:26.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:24.976244+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:26.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:36:24.976845+0000 mgr.y (mgr.14146) 148 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 17122M 2024-02-13T00:36:26.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:24.981775+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:26.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:24.982694+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:26.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:24.983386+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:26.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:24.988455+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:26.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:25.101881+0000 mgr.y (mgr.14146) 149 : audit [DBG] from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:36:26.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:25.104218+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:36:26.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:25.106426+0000 mon.a (mon.0) 419 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:36:26.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:25.107187+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:27.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:25.509310+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v132: 1 pgs: 1 remapped+peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:27.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:25.509310+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v132: 1 pgs: 1 remapped+peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:27.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:25.509310+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v132: 1 pgs: 1 remapped+peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:27.561520+0000 mon.b (mon.1) 6 : audit [INF] from='client.? 172.21.15.181:0/3361604997' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723"}]: dispatch 2024-02-13T00:36:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:27.562305+0000 mon.a (mon.0) 421 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723"}]: dispatch 2024-02-13T00:36:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:27.568327+0000 mon.a (mon.0) 422 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723"}]': finished 2024-02-13T00:36:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:27.568457+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-13T00:36:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:27.568601+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:28.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:27.561520+0000 mon.b (mon.1) 6 : audit [INF] from='client.? 172.21.15.181:0/3361604997' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723"}]: dispatch 2024-02-13T00:36:28.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:27.562305+0000 mon.a (mon.0) 421 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723"}]: dispatch 2024-02-13T00:36:28.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:27.568327+0000 mon.a (mon.0) 422 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723"}]': finished 2024-02-13T00:36:28.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:27.568457+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-13T00:36:28.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:27.568601+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:28.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:27.561520+0000 mon.b (mon.1) 6 : audit [INF] from='client.? 172.21.15.181:0/3361604997' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723"}]: dispatch 2024-02-13T00:36:28.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:27.562305+0000 mon.a (mon.0) 421 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723"}]: dispatch 2024-02-13T00:36:28.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:27.568327+0000 mon.a (mon.0) 422 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723"}]': finished 2024-02-13T00:36:28.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:27.568457+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-13T00:36:28.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:27.568601+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:29.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:27.509869+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:29.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:28.136471+0000 mon.b (mon.1) 7 : audit [DBG] from='client.? 172.21.15.181:0/3571672252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:36:29.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:27.509869+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:29.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:28.136471+0000 mon.b (mon.1) 7 : audit [DBG] from='client.? 172.21.15.181:0/3571672252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:36:29.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:27.509869+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:29.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:28.136471+0000 mon.b (mon.1) 7 : audit [DBG] from='client.? 172.21.15.181:0/3571672252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:36:29.923 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:29.920+0000 7f3113fff700 10 client.24220.objecter tick 2024-02-13T00:36:30.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:29.510206+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:30.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:29.510206+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:30.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:29.510206+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:32.748 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:31.510732+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:32.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:31.510732+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:32.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:31.510732+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:34.606 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:33.511227+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:34.606 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:33.857499+0000 mon.a (mon.0) 425 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-13T00:36:34.606 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:33.858333+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:34.606 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:36:33.859032+0000 mgr.y (mgr.14146) 155 : cephadm [INF] Deploying daemon osd.5 on smithi181 2024-02-13T00:36:34.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:33.511227+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:34.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:33.857499+0000 mon.a (mon.0) 425 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-13T00:36:34.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:33.858333+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:34.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:36:33.859032+0000 mgr.y (mgr.14146) 155 : cephadm [INF] Deploying daemon osd.5 on smithi181 2024-02-13T00:36:34.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:33.511227+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:34.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:33.857499+0000 mon.a (mon.0) 425 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-13T00:36:34.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:33.858333+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:34.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:36:33.859032+0000 mgr.y (mgr.14146) 155 : cephadm [INF] Deploying daemon osd.5 on smithi181 2024-02-13T00:36:34.924 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:34.921+0000 7f3113fff700 10 client.24220.objecter tick 2024-02-13T00:36:35.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:34.511448+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:34.511575+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:35.184462+0000 mon.a (mon.0) 427 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-13T00:36:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:35.184617+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:35.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:34.511448+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:35.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:34.511575+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:35.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:35.184462+0000 mon.a (mon.0) 427 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-13T00:36:35.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:35.184617+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:35.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:34.511448+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:35.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:34.511575+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:35.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:35.184462+0000 mon.a (mon.0) 427 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-13T00:36:35.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:35.184617+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:36.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:35.511811+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:36.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:35.511811+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:36.853 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:35.511811+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:37.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:36.245441+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:37.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:36.245519+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:37.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:37.186934+0000 mon.a (mon.0) 429 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-13T00:36:37.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:37.187097+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:37.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:36.245441+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:37.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:36.245519+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:37.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:37.186934+0000 mon.a (mon.0) 429 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-13T00:36:37.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:37.187097+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:38.038 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:36.245441+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:38.038 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:36.245519+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:38.038 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:37.186934+0000 mon.a (mon.0) 429 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-13T00:36:38.038 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:37.187097+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:38.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:37.512331+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:38.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:37.512331+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:38.842 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:37.512331+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:39.926 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:39.922+0000 7f3113fff700 10 client.24220.objecter tick 2024-02-13T00:36:40.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:38.884148+0000 mon.a (mon.0) 431 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:40.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:38.888227+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:40.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:38.884148+0000 mon.a (mon.0) 431 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:40.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:38.888227+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:40.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:38.884148+0000 mon.a (mon.0) 431 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:40.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:38.888227+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:41.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:39.512825+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:39.512825+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:41.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:39.512825+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:42.086 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.083+0000 7f31037fe700 1 -- 172.21.15.181:0/4284626907 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f31140fd9d0 con 0x7f30fc062ba0 2024-02-13T00:36:42.087 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 5 on host 'smithi181' 2024-02-13T00:36:42.088 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 10 librados: watch_flush enter 2024-02-13T00:36:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 10 librados: watch_flush exit 2024-02-13T00:36:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 20 client.24220.objecter shutdown clearing up homeless session... 2024-02-13T00:36:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 10 client.24220.objecter successfully canceled tick 2024-02-13T00:36:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 -- 172.21.15.181:0/4284626907 >> v2:172.21.15.162:6800/1909610488 conn(0x7f30fc062ba0 msgr2=0x7f30fc065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 --2- 172.21.15.181:0/4284626907 >> v2:172.21.15.162:6800/1909610488 conn(0x7f30fc062ba0 0x7f30fc065060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f3104009da0 tx=0x7f3104009cb0).stop 2024-02-13T00:36:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 -- 172.21.15.181:0/4284626907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31140f7ce0 msgr2=0x7f31141073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 --2- 172.21.15.181:0/4284626907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31140f7ce0 0x7f31141073d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f3108002760 tx=0x7f3108009e80).stop 2024-02-13T00:36:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 -- 172.21.15.181:0/4284626907 shutdown_connections 2024-02-13T00:36:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 --2- 172.21.15.181:0/4284626907 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31140f8630 0x7f311410e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 --2- 172.21.15.181:0/4284626907 >> v2:172.21.15.162:6800/1909610488 conn(0x7f30fc062ba0 0x7f30fc065060 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 --2- 172.21.15.181:0/4284626907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31140f6a40 0x7f3114106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:42.089 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 --2- 172.21.15.181:0/4284626907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31140f7ce0 0x7f31141073d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:42.090 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 -- 172.21.15.181:0/4284626907 >> 172.21.15.181:0/4284626907 conn(0x7f31140f2240 msgr2=0x7f31140f4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:36:42.090 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 -- 172.21.15.181:0/4284626907 shutdown_connections 2024-02-13T00:36:42.090 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 -- 172.21.15.181:0/4284626907 wait complete. 2024-02-13T00:36:42.090 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:42.085+0000 7f3118f2c700 1 librados: shutdown 2024-02-13T00:36:42.773 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:41.411500+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:42.774 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:41.412400+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:42.774 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:41.413317+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:42.774 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:41.418898+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:42.774 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:41.513241+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:42.774 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:41.993276+0000 mon.b (mon.1) 8 : audit [INF] from='osd.5 v2:172.21.15.181:6804/2212296748' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T00:36:42.774 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:41.993891+0000 mon.a (mon.0) 437 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T00:36:42.774 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:42.083379+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:42.774 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:42.086632+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:42.801 DEBUG:teuthology.orchestra.run.smithi181:osd.5> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.5.service 2024-02-13T00:36:42.804 INFO:tasks.cephadm:Deploying osd.6 on smithi181 with /dev/vg_nvme/lv_2... 2024-02-13T00:36:42.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-13T00:36:42.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:41.411500+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:42.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:41.412400+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:42.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:41.413317+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:42.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:41.418898+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:41.513241+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:41.993276+0000 mon.b (mon.1) 8 : audit [INF] from='osd.5 v2:172.21.15.181:6804/2212296748' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T00:36:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:41.993891+0000 mon.a (mon.0) 437 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T00:36:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:42.083379+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:42.086632+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:42.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:41.411500+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:42.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:41.412400+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:42.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:41.413317+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:41.418898+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:41.513241+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:41.993276+0000 mon.b (mon.1) 8 : audit [INF] from='osd.5 v2:172.21.15.181:6804/2212296748' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T00:36:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:41.993891+0000 mon.a (mon.0) 437 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T00:36:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:42.083379+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:42.086632+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:36:42.833 INFO:journalctl@ceph.osd.5.smithi181.stdout:-- Logs begin at Tue 2024-02-13 00:17:30 UTC. -- 2024-02-13T00:36:43.795 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:36:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[49633]: debug 2024-02-13T00:36:43.429+0000 7f6c06e7c700 -1 osd.5 0 waiting for initial osdmap 2024-02-13T00:36:43.795 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:36:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[49633]: debug 2024-02-13T00:36:43.434+0000 7f6c015f4700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T00:36:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:42.422744+0000 mon.a (mon.0) 440 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-13T00:36:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:42.422839+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2024-02-13T00:36:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:42.422995+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:42.423244+0000 mon.b (mon.1) 9 : audit [INF] from='osd.5 v2:172.21.15.181:6804/2212296748' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:36:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:42.423744+0000 mon.a (mon.0) 443 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:36:43.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:42.422744+0000 mon.a (mon.0) 440 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-13T00:36:43.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:42.422839+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2024-02-13T00:36:43.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:42.422995+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:43.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:42.423244+0000 mon.b (mon.1) 9 : audit [INF] from='osd.5 v2:172.21.15.181:6804/2212296748' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:36:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:42.423744+0000 mon.a (mon.0) 443 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:36:43.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:42.422744+0000 mon.a (mon.0) 440 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-13T00:36:43.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:42.422839+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2024-02-13T00:36:43.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:42.422995+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:43.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:42.423244+0000 mon.b (mon.1) 9 : audit [INF] from='osd.5 v2:172.21.15.181:6804/2212296748' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:36:43.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:42.423744+0000 mon.a (mon.0) 443 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:36:44.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:43.424923+0000 mon.a (mon.0) 444 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-13T00:36:44.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:43.425032+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e38: 6 total, 5 up, 6 in 2024-02-13T00:36:44.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:43.425480+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:44.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:43.430961+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:44.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:43.513591+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:44.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:44.188244+0000 mon.a (mon.0) 448 : cluster [INF] osd.5 v2:172.21.15.181:6804/2212296748 boot 2024-02-13T00:36:44.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:44.188287+0000 mon.a (mon.0) 449 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-13T00:36:44.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:44.191298+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:44.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:43.424923+0000 mon.a (mon.0) 444 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-13T00:36:44.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:43.425032+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e38: 6 total, 5 up, 6 in 2024-02-13T00:36:44.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:43.425480+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:44.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:43.430961+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:44.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:43.513591+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:44.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:44.188244+0000 mon.a (mon.0) 448 : cluster [INF] osd.5 v2:172.21.15.181:6804/2212296748 boot 2024-02-13T00:36:44.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:44.188287+0000 mon.a (mon.0) 449 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-13T00:36:44.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:44.191298+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:44.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:43.424923+0000 mon.a (mon.0) 444 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-13T00:36:44.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:43.425032+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e38: 6 total, 5 up, 6 in 2024-02-13T00:36:44.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:43.425480+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:44.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:43.430961+0000 mon.a (mon.0) 447 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:44.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:43.513591+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:36:44.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:44.188244+0000 mon.a (mon.0) 448 : cluster [INF] osd.5 v2:172.21.15.181:6804/2212296748 boot 2024-02-13T00:36:44.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:44.188287+0000 mon.a (mon.0) 449 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-13T00:36:44.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:44.191298+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:36:45.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:42.981297+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:45.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:42.981352+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:45.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:45.204308+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-02-13T00:36:45.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:42.981297+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:45.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:42.981352+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:45.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:45.204308+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-02-13T00:36:45.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:42.981297+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:36:45.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:42.981352+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:36:45.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:45.204308+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-02-13T00:36:46.757 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-13T00:36:46.758 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-13T00:36:46.758 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-02-13T00:36:46.758 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-02-13T00:36:46.758 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011118 s, 943 MB/s 2024-02-13T00:36:46.758 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-02-13T00:36:47.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:45.514027+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:47.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:36:45.984107+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Detected new or changed devices on smithi181 2024-02-13T00:36:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:45.989257+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:45.990323+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:45.991093+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:36:45.991597+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 8561M 2024-02-13T00:36:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:45.998024+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:45.999006+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:45.999703+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:46.006821+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:46.208074+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2024-02-13T00:36:47.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:45.514027+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:47.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:36:45.984107+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Detected new or changed devices on smithi181 2024-02-13T00:36:47.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:45.989257+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:47.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:45.990323+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:45.991093+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:36:45.991597+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 8561M 2024-02-13T00:36:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:45.998024+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:45.999006+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:45.999703+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:46.006821+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:46.208074+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2024-02-13T00:36:47.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:45.514027+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:47.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:36:45.984107+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Detected new or changed devices on smithi181 2024-02-13T00:36:47.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:45.989257+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:47.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:45.990323+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:47.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:45.991093+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:36:47.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:36:45.991597+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 8561M 2024-02-13T00:36:47.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:45.998024+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:47.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:45.999006+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:47.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:45.999703+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:36:47.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:46.006821+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:36:47.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:46.208074+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2024-02-13T00:36:47.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi181:vg_nvme/lv_2 2024-02-13T00:36:48.141 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:47.062622+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:36:48.141 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:47.063564+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:36:48.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:47.062622+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:36:48.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:47.063564+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:36:48.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:47.062622+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:36:48.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:47.063564+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:36:49.108 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:47.514367+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:49.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:47.514367+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:49.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:47.514367+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:49.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 -- 172.21.15.181:0/3770309638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c0f8dc0 msgr2=0x7f762c0f9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:49.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 --2- 172.21.15.181:0/3770309638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c0f8dc0 0x7f762c0f9220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f76280048a0 tx=0x7f7628009cb0).stop 2024-02-13T00:36:49.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 -- 172.21.15.181:0/3770309638 shutdown_connections 2024-02-13T00:36:49.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 --2- 172.21.15.181:0/3770309638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f762c0f9710 0x7f762c100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 --2- 172.21.15.181:0/3770309638 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f762c0f7b20 0x7f762c0f7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 --2- 172.21.15.181:0/3770309638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c0f8dc0 0x7f762c0f9220 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 -- 172.21.15.181:0/3770309638 >> 172.21.15.181:0/3770309638 conn(0x7f762c0f3340 msgr2=0x7f762c0f5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 -- 172.21.15.181:0/3770309638 shutdown_connections 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 -- 172.21.15.181:0/3770309638 wait complete. 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 librados: starting msgr at 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 librados: starting objecter 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 Processor -- start 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 -- start start 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 librados: setting wanted keys 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 librados: calling monclient init 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c0f7b20 0x7f762c108060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f762c0f8dc0 0x7f762c108550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:36:49.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f762c0f9710 0x7f762c10f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:36:49.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c10fb20 con 0x7f762c0f9710 2024-02-13T00:36:49.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c10fc60 con 0x7f762c0f7b20 2024-02-13T00:36:49.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.375+0000 7f7633e05700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762c10fda0 con 0x7f762c0f8dc0 2024-02-13T00:36:49.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7631ba1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c0f7b20 0x7f762c108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:36:49.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7631ba1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c0f7b20 0x7f762c108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:40270/0 (socket says 172.21.15.181:40270) 2024-02-13T00:36:49.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7631ba1700 1 -- 172.21.15.181:0/1136476547 learned_addr learned my addr 172.21.15.181:0/1136476547 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:36:49.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f76323a2700 1 --2- 172.21.15.181:0/1136476547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f762c0f9710 0x7f762c10f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:36:49.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f76313a0700 1 --2- 172.21.15.181:0/1136476547 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f762c0f8dc0 0x7f762c108550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:36:49.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7631ba1700 1 -- 172.21.15.181:0/1136476547 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f762c0f8dc0 msgr2=0x7f762c108550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:49.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7631ba1700 1 --2- 172.21.15.181:0/1136476547 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f762c0f8dc0 0x7f762c108550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:49.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7631ba1700 1 -- 172.21.15.181:0/1136476547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f762c0f9710 msgr2=0x7f762c10f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:36:49.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7631ba1700 1 --2- 172.21.15.181:0/1136476547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f762c0f9710 0x7f762c10f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7631ba1700 1 -- 172.21.15.181:0/1136476547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7628005040 con 0x7f762c0f7b20 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f76313a0700 1 --2- 172.21.15.181:0/1136476547 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f762c0f8dc0 0x7f762c108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7631ba1700 1 --2- 172.21.15.181:0/1136476547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c0f7b20 0x7f762c108060 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f761c00c810 tx=0x7f761c007f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7622ffd700 10 client.?.objecter ms_handle_connect 0x7f762c0f7b20 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7622ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7622ffd700 1 -- 172.21.15.181:0/1136476547 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c00e4a0 con 0x7f762c0f7b20 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7622ffd700 1 -- 172.21.15.181:0/1136476547 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f761c00ead0 con 0x7f762c0f7b20 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7633e05700 1 -- 172.21.15.181:0/1136476547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762c110080 con 0x7f762c0f7b20 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7622ffd700 1 -- 172.21.15.181:0/1136476547 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c012a00 con 0x7f762c0f7b20 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7633e05700 10 client.24247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7633e05700 1 -- 172.21.15.181:0/1136476547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c1105f0 con 0x7f762c0f7b20 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.376+0000 7f7633e05700 1 librados: init done 2024-02-13T00:36:49.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.377+0000 7f7633e05700 1 -- 172.21.15.181:0/1136476547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762c05f400 con 0x7f762c0f7b20 2024-02-13T00:36:49.416 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.377+0000 7f7622ffd700 1 -- 172.21.15.181:0/1136476547 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7f761c00ec40 con 0x7f762c0f7b20 2024-02-13T00:36:49.416 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.378+0000 7f7622ffd700 1 --2- 172.21.15.181:0/1136476547 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7618062ba0 0x7f7618065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:36:49.416 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.378+0000 7f7622ffd700 1 -- 172.21.15.181:0/1136476547 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 3470+0+0 (secure 0 0 0) 0x7f761c07b490 con 0x7f762c0f7b20 2024-02-13T00:36:49.416 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.378+0000 7f7622ffd700 10 client.24247.objecter ms_dispatch 0x7f762c001550 osd_map(41..41 src has 1..41) v4 2024-02-13T00:36:49.416 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.378+0000 7f7622ffd700 3 client.24247.objecter handle_osd_map got epochs [41,41] > 0 2024-02-13T00:36:49.416 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.378+0000 7f7622ffd700 3 client.24247.objecter handle_osd_map decoding full epoch 41 2024-02-13T00:36:49.416 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.378+0000 7f76313a0700 1 --2- 172.21.15.181:0/1136476547 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7618062ba0 0x7f7618065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:36:49.416 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.378+0000 7f7622ffd700 20 client.24247.objecter dump_active .. 0 homeless 2024-02-13T00:36:49.416 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.378+0000 7f7622ffd700 10 client.24247.objecter ms_handle_connect 0x7f7618062ba0 2024-02-13T00:36:49.416 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.378+0000 7f76313a0700 1 --2- 172.21.15.181:0/1136476547 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7618062ba0 0x7f7618065060 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f7628005010 tx=0x7f7628009cb0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:36:49.416 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.382+0000 7f7622ffd700 1 -- 172.21.15.181:0/1136476547 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f761c047f60 con 0x7f762c0f7b20 2024-02-13T00:36:49.557 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:49.554+0000 7f7633e05700 1 -- 172.21.15.181:0/1136476547 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f762c0fef20 con 0x7f7618062ba0 2024-02-13T00:36:50.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:49.514866+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:50.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:49.556414+0000 mgr.y (mgr.14146) 166 : audit [DBG] from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:36:50.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:49.558795+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:36:50.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:49.561160+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:36:50.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:49.561978+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:50.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:49.514866+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:50.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:49.556414+0000 mgr.y (mgr.14146) 166 : audit [DBG] from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:36:50.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:49.558795+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:36:50.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:49.561160+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:36:50.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:49.561978+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:49.514866+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:49.556414+0000 mgr.y (mgr.14146) 166 : audit [DBG] from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:36:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:49.558795+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:36:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:49.561160+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:36:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:49.561978+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:52.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:51.515162+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:52.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:52.058315+0000 mon.b (mon.1) 10 : audit [INF] from='client.? 172.21.15.181:0/1325877246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474"}]: dispatch 2024-02-13T00:36:52.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:52.058991+0000 mon.a (mon.0) 463 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474"}]: dispatch 2024-02-13T00:36:52.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:52.063797+0000 mon.a (mon.0) 464 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474"}]': finished 2024-02-13T00:36:52.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:52.063867+0000 mon.a (mon.0) 465 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-13T00:36:52.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:52.064001+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:36:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:51.515162+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:52.058315+0000 mon.b (mon.1) 10 : audit [INF] from='client.? 172.21.15.181:0/1325877246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474"}]: dispatch 2024-02-13T00:36:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:52.058991+0000 mon.a (mon.0) 463 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474"}]: dispatch 2024-02-13T00:36:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:52.063797+0000 mon.a (mon.0) 464 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474"}]': finished 2024-02-13T00:36:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:52.063867+0000 mon.a (mon.0) 465 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-13T00:36:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:52.064001+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:36:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:51.515162+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:52.058315+0000 mon.b (mon.1) 10 : audit [INF] from='client.? 172.21.15.181:0/1325877246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474"}]: dispatch 2024-02-13T00:36:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:52.058991+0000 mon.a (mon.0) 463 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474"}]: dispatch 2024-02-13T00:36:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:52.063797+0000 mon.a (mon.0) 464 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474"}]': finished 2024-02-13T00:36:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:52.063867+0000 mon.a (mon.0) 465 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-13T00:36:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:52.064001+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:36:53.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:52.662763+0000 mon.b (mon.1) 11 : audit [DBG] from='client.? 172.21.15.181:0/844747173' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:36:53.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:52.662763+0000 mon.b (mon.1) 11 : audit [DBG] from='client.? 172.21.15.181:0/844747173' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:36:54.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:52.662763+0000 mon.b (mon.1) 11 : audit [DBG] from='client.? 172.21.15.181:0/844747173' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:36:54.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:54.377+0000 7f7633604700 10 client.24247.objecter tick 2024-02-13T00:36:54.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:53.515618+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:54.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:53.515618+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:54.928 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:53.515618+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:56.994 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:55.516034+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:57.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:55.516034+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:57.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:55.516034+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:58.944 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:57.516498+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:58.944 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:58.317460+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-13T00:36:58.944 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:36:58.318214+0000 mon.a (mon.0) 468 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:59.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:57.516498+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:59.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:58.317460+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-13T00:36:59.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:36:58.318214+0000 mon.a (mon.0) 468 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:59.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:57.516498+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:36:59.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:58.317460+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-13T00:36:59.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:36:58.318214+0000 mon.a (mon.0) 468 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:36:59.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:36:59.378+0000 7f7633604700 10 client.24247.objecter tick 2024-02-13T00:36:59.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:36:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:36:58.318828+0000 mgr.y (mgr.14146) 171 : cephadm [INF] Deploying daemon osd.6 on smithi181 2024-02-13T00:37:00.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:36:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:36:58.318828+0000 mgr.y (mgr.14146) 171 : cephadm [INF] Deploying daemon osd.6 on smithi181 2024-02-13T00:37:00.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:36:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:36:58.318828+0000 mgr.y (mgr.14146) 171 : cephadm [INF] Deploying daemon osd.6 on smithi181 2024-02-13T00:37:00.997 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:36:59.516958+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:01.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:36:59.516958+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:01.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:36:59.516958+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:01.517412+0000 mgr.y (mgr.14146) 173 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:03.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:01.517412+0000 mgr.y (mgr.14146) 173 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:03.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:01.517412+0000 mgr.y (mgr.14146) 173 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:04.382 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:04.378+0000 7f7633604700 10 client.24247.objecter tick 2024-02-13T00:37:04.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:03.348613+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:04.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:03.365261+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:04.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:03.517887+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:04.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:03.348613+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:04.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:03.365261+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:04.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:03.517887+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:04.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:03.348613+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:04.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:03.365261+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:04.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:03.517887+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:06.430 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.427+0000 7f7622ffd700 1 -- 172.21.15.181:0/1136476547 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f762c0fef20 con 0x7f7618062ba0 2024-02-13T00:37:06.431 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 6 on host 'smithi181' 2024-02-13T00:37:06.433 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 10 librados: watch_flush enter 2024-02-13T00:37:06.433 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 10 librados: watch_flush exit 2024-02-13T00:37:06.433 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 20 client.24247.objecter shutdown clearing up homeless session... 2024-02-13T00:37:06.433 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 10 client.24247.objecter successfully canceled tick 2024-02-13T00:37:06.433 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 1 -- 172.21.15.181:0/1136476547 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7618062ba0 msgr2=0x7f7618065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:06.433 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 1 --2- 172.21.15.181:0/1136476547 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7618062ba0 0x7f7618065060 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f7628005010 tx=0x7f7628009cb0).stop 2024-02-13T00:37:06.433 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 1 -- 172.21.15.181:0/1136476547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c0f7b20 msgr2=0x7f762c108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:06.433 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 1 --2- 172.21.15.181:0/1136476547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c0f7b20 0x7f762c108060 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f761c00c810 tx=0x7f761c007f70).stop 2024-02-13T00:37:06.433 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 1 -- 172.21.15.181:0/1136476547 shutdown_connections 2024-02-13T00:37:06.433 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 1 --2- 172.21.15.181:0/1136476547 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7618062ba0 0x7f7618065060 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:06.434 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 1 --2- 172.21.15.181:0/1136476547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f762c0f9710 0x7f762c10f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:06.434 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 1 --2- 172.21.15.181:0/1136476547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c0f7b20 0x7f762c108060 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:06.434 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 1 --2- 172.21.15.181:0/1136476547 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f762c0f8dc0 0x7f762c108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:06.434 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 1 -- 172.21.15.181:0/1136476547 >> 172.21.15.181:0/1136476547 conn(0x7f762c0f3340 msgr2=0x7f762c0f6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:06.434 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 1 -- 172.21.15.181:0/1136476547 shutdown_connections 2024-02-13T00:37:06.434 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.428+0000 7f7633e05700 1 -- 172.21.15.181:0/1136476547 wait complete. 2024-02-13T00:37:06.434 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:06.429+0000 7f7633e05700 1 librados: shutdown 2024-02-13T00:37:06.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:05.518352+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:06.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:06.427453+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:06.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:06.471817+0000 mon.b (mon.1) 12 : audit [INF] from='osd.6 v2:172.21.15.181:6808/1506640341' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T00:37:06.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:06.472400+0000 mon.a (mon.0) 472 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T00:37:06.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:05.518352+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:06.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:06.427453+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:06.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:06.471817+0000 mon.b (mon.1) 12 : audit [INF] from='osd.6 v2:172.21.15.181:6808/1506640341' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T00:37:06.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:06.472400+0000 mon.a (mon.0) 472 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T00:37:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:05.518352+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:06.427453+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:06.471817+0000 mon.b (mon.1) 12 : audit [INF] from='osd.6 v2:172.21.15.181:6808/1506640341' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T00:37:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:06.472400+0000 mon.a (mon.0) 472 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-13T00:37:07.138 DEBUG:teuthology.orchestra.run.smithi181:osd.6> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.6.service 2024-02-13T00:37:07.140 INFO:tasks.cephadm:Deploying osd.7 on smithi181 with /dev/vg_nvme/lv_1... 2024-02-13T00:37:07.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-13T00:37:07.169 INFO:journalctl@ceph.osd.6.smithi181.stdout:-- Logs begin at Tue 2024-02-13 00:17:30 UTC. -- 2024-02-13T00:37:08.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:07.234624+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:08.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:07.436949+0000 mon.a (mon.0) 474 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-13T00:37:08.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:07.437022+0000 mon.a (mon.0) 475 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-13T00:37:08.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:07.437176+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:37:08.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:07.437461+0000 mon.b (mon.1) 13 : audit [INF] from='osd.6 v2:172.21.15.181:6808/1506640341' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:37:08.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:07.437955+0000 mon.a (mon.0) 477 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:37:08.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:07.518809+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:08.545 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:37:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:37:08.439+0000 7f8a796ec700 -1 osd.6 0 waiting for initial osdmap 2024-02-13T00:37:08.545 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:37:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:37:08.445+0000 7f8a72661700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T00:37:08.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:07.234624+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:08.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:07.436949+0000 mon.a (mon.0) 474 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-13T00:37:08.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:07.437022+0000 mon.a (mon.0) 475 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-13T00:37:08.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:07.437176+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:37:08.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:07.437461+0000 mon.b (mon.1) 13 : audit [INF] from='osd.6 v2:172.21.15.181:6808/1506640341' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:37:08.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:07.437955+0000 mon.a (mon.0) 477 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:37:08.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:07.518809+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:08.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:07.234624+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:08.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:07.436949+0000 mon.a (mon.0) 474 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-13T00:37:08.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:07.437022+0000 mon.a (mon.0) 475 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-13T00:37:08.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:07.437176+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:37:08.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:07.437461+0000 mon.b (mon.1) 13 : audit [INF] from='osd.6 v2:172.21.15.181:6808/1506640341' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:37:08.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:07.437955+0000 mon.a (mon.0) 477 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:37:08.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:07.518809+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:37:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:08.438185+0000 mon.a (mon.0) 478 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-13T00:37:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:08.438294+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2024-02-13T00:37:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:08.438431+0000 mon.a (mon.0) 480 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:37:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:08.441477+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:37:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:09.191104+0000 mon.a (mon.0) 482 : cluster [INF] osd.6 v2:172.21.15.181:6808/1506640341 boot 2024-02-13T00:37:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:09.191165+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-13T00:37:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:09.193610+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:37:09.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:08.438185+0000 mon.a (mon.0) 478 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-13T00:37:09.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:08.438294+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2024-02-13T00:37:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:08.438431+0000 mon.a (mon.0) 480 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:37:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:08.441477+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:37:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:09.191104+0000 mon.a (mon.0) 482 : cluster [INF] osd.6 v2:172.21.15.181:6808/1506640341 boot 2024-02-13T00:37:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:09.191165+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-13T00:37:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:09.193610+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:37:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:08.438185+0000 mon.a (mon.0) 478 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-13T00:37:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:08.438294+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2024-02-13T00:37:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:08.438431+0000 mon.a (mon.0) 480 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:37:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:08.441477+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:37:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:09.191104+0000 mon.a (mon.0) 482 : cluster [INF] osd.6 v2:172.21.15.181:6808/1506640341 boot 2024-02-13T00:37:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:09.191165+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-13T00:37:09.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:09.193610+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:37:10.165 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-13T00:37:10.165 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-13T00:37:10.165 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-02-13T00:37:10.165 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-02-13T00:37:10.165 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122664 s, 855 MB/s 2024-02-13T00:37:10.165 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-02-13T00:37:10.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:07.501260+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:07.501353+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:09.519195+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:10.201140+0000 mon.a (mon.0) 485 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-13T00:37:10.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:07.501260+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:10.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:07.501353+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:10.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:09.519195+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:10.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:10.201140+0000 mon.a (mon.0) 485 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-13T00:37:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:07.501260+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:07.501353+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:09.519195+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:10.201140+0000 mon.a (mon.0) 485 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-13T00:37:10.843 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi181:vg_nvme/lv_1 2024-02-13T00:37:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:11.145819+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:11.145927+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:11.201880+0000 mon.a (mon.0) 486 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-13T00:37:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:11.519639+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:12.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:11.145819+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:12.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:11.145927+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:12.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:11.201880+0000 mon.a (mon.0) 486 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-13T00:37:12.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:11.519639+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:11.145819+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:11.145927+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:11.201880+0000 mon.a (mon.0) 486 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-13T00:37:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:11.519639+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:13.526 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.523+0000 7fa605776700 1 -- 172.21.15.181:0/761746339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6000f8640 msgr2=0x7fa6000fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:13.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.523+0000 7fa605776700 1 --2- 172.21.15.181:0/761746339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6000f8640 0x7fa6000fef40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa5f0004a00 tx=0x7fa5f0009ca0).stop 2024-02-13T00:37:13.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.523+0000 7fa605776700 1 -- 172.21.15.181:0/761746339 shutdown_connections 2024-02-13T00:37:13.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.523+0000 7fa605776700 1 --2- 172.21.15.181:0/761746339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6000f8640 0x7fa6000fef40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:13.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.523+0000 7fa605776700 1 --2- 172.21.15.181:0/761746339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6000f6a50 0x7fa6000f6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:13.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.523+0000 7fa605776700 1 --2- 172.21.15.181:0/761746339 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6000f7cf0 0x7fa6000f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:13.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.523+0000 7fa605776700 1 -- 172.21.15.181:0/761746339 >> 172.21.15.181:0/761746339 conn(0x7fa6000f2230 msgr2=0x7fa6000f4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:13.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.523+0000 7fa605776700 1 -- 172.21.15.181:0/761746339 shutdown_connections 2024-02-13T00:37:13.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.523+0000 7fa605776700 1 -- 172.21.15.181:0/761746339 wait complete. 2024-02-13T00:37:13.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.523+0000 7fa605776700 1 librados: starting msgr at 2024-02-13T00:37:13.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.523+0000 7fa605776700 1 librados: starting objecter 2024-02-13T00:37:13.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.523+0000 7fa605776700 1 Processor -- start 2024-02-13T00:37:13.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa605776700 1 -- start start 2024-02-13T00:37:13.528 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa605776700 1 librados: setting wanted keys 2024-02-13T00:37:13.528 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa605776700 1 librados: calling monclient init 2024-02-13T00:37:13.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa605776700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6000f6a50 0x7fa6000fa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:13.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa605776700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6000f7cf0 0x7fa6000f9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:13.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa605776700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6000f8640 0x7fa6000f9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:13.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa605776700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6000f9b40 con 0x7fa6000f6a50 2024-02-13T00:37:13.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa605776700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6000f9cb0 con 0x7fa6000f8640 2024-02-13T00:37:13.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa605776700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6000f9e70 con 0x7fa6000f7cf0 2024-02-13T00:37:13.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5ff7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6000f8640 0x7fa6000f9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:13.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5ff7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6000f8640 0x7fa6000f9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:34748/0 (socket says 172.21.15.181:34748) 2024-02-13T00:37:13.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5ff7fe700 1 -- 172.21.15.181:0/3676255039 learned_addr learned my addr 172.21.15.181:0/3676255039 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:37:13.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5fe7fc700 1 --2- 172.21.15.181:0/3676255039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6000f7cf0 0x7fa6000f9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5feffd700 1 --2- 172.21.15.181:0/3676255039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6000f6a50 0x7fa6000fa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5ff7fe700 1 -- 172.21.15.181:0/3676255039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6000f7cf0 msgr2=0x7fa6000f9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5ff7fe700 1 --2- 172.21.15.181:0/3676255039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6000f7cf0 0x7fa6000f9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5ff7fe700 1 -- 172.21.15.181:0/3676255039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6000f6a50 msgr2=0x7fa6000fa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5ff7fe700 1 --2- 172.21.15.181:0/3676255039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6000f6a50 0x7fa6000fa9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5ff7fe700 1 -- 172.21.15.181:0/3676255039 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f0004620 con 0x7fa6000f8640 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5ff7fe700 1 --2- 172.21.15.181:0/3676255039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6000f8640 0x7fa6000f9520 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa5f0004390 tx=0x7fa5f0009ca0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5e7fff700 10 client.?.objecter ms_handle_connect 0x7fa6000f8640 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5e7fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.524+0000 7fa5e7fff700 1 -- 172.21.15.181:0/3676255039 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f000e070 con 0x7fa6000f8640 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.525+0000 7fa5e7fff700 1 -- 172.21.15.181:0/3676255039 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5f000f040 con 0x7fa6000f8640 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.525+0000 7fa5e7fff700 1 -- 172.21.15.181:0/3676255039 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0014e90 con 0x7fa6000f8640 2024-02-13T00:37:13.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.525+0000 7fa605776700 1 -- 172.21.15.181:0/3676255039 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa600112020 con 0x7fa6000f8640 2024-02-13T00:37:13.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.525+0000 7fa605776700 10 client.24274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:37:13.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.525+0000 7fa605776700 1 -- 172.21.15.181:0/3676255039 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa600112450 con 0x7fa6000f8640 2024-02-13T00:37:13.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.525+0000 7fa605776700 1 librados: init done 2024-02-13T00:37:13.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.526+0000 7fa605776700 1 -- 172.21.15.181:0/3676255039 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa600101850 con 0x7fa6000f8640 2024-02-13T00:37:13.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.526+0000 7fa5e7fff700 1 -- 172.21.15.181:0/3676255039 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7fa5f0014590 con 0x7fa6000f8640 2024-02-13T00:37:13.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.527+0000 7fa5e7fff700 1 --2- 172.21.15.181:0/3676255039 >> v2:172.21.15.162:6800/1909610488 conn(0x7fa5e8062ba0 0x7fa5e8065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:13.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.527+0000 7fa5e7fff700 1 -- 172.21.15.181:0/3676255039 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 3762+0+0 (secure 0 0 0) 0x7fa5f007cdc0 con 0x7fa6000f8640 2024-02-13T00:37:13.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.527+0000 7fa5e7fff700 10 client.24274.objecter ms_dispatch 0x7fa600001550 osd_map(47..47 src has 1..47) v4 2024-02-13T00:37:13.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.527+0000 7fa5e7fff700 3 client.24274.objecter handle_osd_map got epochs [47,47] > 0 2024-02-13T00:37:13.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.527+0000 7fa5e7fff700 3 client.24274.objecter handle_osd_map decoding full epoch 47 2024-02-13T00:37:13.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.527+0000 7fa5e7fff700 20 client.24274.objecter dump_active .. 0 homeless 2024-02-13T00:37:13.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.527+0000 7fa5feffd700 1 --2- 172.21.15.181:0/3676255039 >> v2:172.21.15.162:6800/1909610488 conn(0x7fa5e8062ba0 0x7fa5e8065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:13.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.527+0000 7fa5e7fff700 10 client.24274.objecter ms_handle_connect 0x7fa5e8062ba0 2024-02-13T00:37:13.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.527+0000 7fa5feffd700 1 --2- 172.21.15.181:0/3676255039 >> v2:172.21.15.162:6800/1909610488 conn(0x7fa5e8062ba0 0x7fa5e8065060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fa6000f7b50 tx=0x7fa5f8009040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:13.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.530+0000 7fa5e7fff700 1 -- 172.21.15.181:0/3676255039 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa5f004da00 con 0x7fa6000f8640 2024-02-13T00:37:13.704 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:13.700+0000 7fa605776700 1 -- 172.21.15.181:0/3676255039 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6000611d0 con 0x7fa5e8062ba0 2024-02-13T00:37:14.454 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:13.087880+0000 mgr.y (mgr.14146) 179 : cephadm [INF] Detected new or changed devices on smithi181 2024-02-13T00:37:14.454 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.092806+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.373194+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.374082+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.374808+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.380117+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.381101+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.381839+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.386262+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.388968+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.390554+0000 mon.a (mon.0) 496 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.391227+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.396313+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.705123+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:37:14.455 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.707515+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:37:14.456 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.708221+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:14.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:13.087880+0000 mgr.y (mgr.14146) 179 : cephadm [INF] Detected new or changed devices on smithi181 2024-02-13T00:37:14.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.092806+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:14.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.373194+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.374082+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.374808+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.380117+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.381101+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.381839+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.386262+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.388968+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.390554+0000 mon.a (mon.0) 496 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.391227+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.396313+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.705123+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:37:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.707515+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:37:14.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.708221+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:14.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:13.087880+0000 mgr.y (mgr.14146) 179 : cephadm [INF] Detected new or changed devices on smithi181 2024-02-13T00:37:14.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.092806+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:14.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.373194+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:14.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.374082+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:14.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.374808+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:14.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.380117+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:14.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.381101+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:14.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.381839+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:14.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.386262+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:14.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.388968+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:14.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.390554+0000 mon.a (mon.0) 496 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:14.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.391227+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:14.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.396313+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:14.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.705123+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T00:37:14.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.707515+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T00:37:14.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.708221+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:15.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:13.375352+0000 mgr.y (mgr.14146) 180 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 5707M 2024-02-13T00:37:15.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:13.487429+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:15.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:13.487534+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:15.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:13.520032+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:15.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:13.702801+0000 mgr.y (mgr.14146) 182 : audit [DBG] from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:15.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:14.408851+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-13T00:37:15.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:13.375352+0000 mgr.y (mgr.14146) 180 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 5707M 2024-02-13T00:37:15.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:13.487429+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:15.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:13.487534+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:15.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:13.520032+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:15.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:13.702801+0000 mgr.y (mgr.14146) 182 : audit [DBG] from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:15.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:14.408851+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-13T00:37:15.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:13.375352+0000 mgr.y (mgr.14146) 180 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 5707M 2024-02-13T00:37:15.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:13.487429+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:15.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:13.487534+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:15.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:13.520032+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:15.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:13.702801+0000 mgr.y (mgr.14146) 182 : audit [DBG] from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:15.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:14.408851+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-13T00:37:17.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:15.520377+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:17.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:16.201292+0000 mon.b (mon.1) 14 : audit [INF] from='client.? 172.21.15.181:0/4191408426' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b"}]: dispatch 2024-02-13T00:37:17.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:16.201839+0000 mon.a (mon.0) 503 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b"}]: dispatch 2024-02-13T00:37:17.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:16.206426+0000 mon.a (mon.0) 504 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b"}]': finished 2024-02-13T00:37:17.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:16.206505+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2024-02-13T00:37:17.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:16.206672+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:17.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:16.767667+0000 mon.b (mon.1) 15 : audit [DBG] from='client.? 172.21.15.181:0/3845541510' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:37:17.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:15.520377+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:17.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:16.201292+0000 mon.b (mon.1) 14 : audit [INF] from='client.? 172.21.15.181:0/4191408426' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b"}]: dispatch 2024-02-13T00:37:17.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:16.201839+0000 mon.a (mon.0) 503 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b"}]: dispatch 2024-02-13T00:37:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:16.206426+0000 mon.a (mon.0) 504 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b"}]': finished 2024-02-13T00:37:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:16.206505+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2024-02-13T00:37:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:16.206672+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:16.767667+0000 mon.b (mon.1) 15 : audit [DBG] from='client.? 172.21.15.181:0/3845541510' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:37:17.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:15.520377+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:17.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:16.201292+0000 mon.b (mon.1) 14 : audit [INF] from='client.? 172.21.15.181:0/4191408426' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b"}]: dispatch 2024-02-13T00:37:17.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:16.201839+0000 mon.a (mon.0) 503 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b"}]: dispatch 2024-02-13T00:37:17.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:16.206426+0000 mon.a (mon.0) 504 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b"}]': finished 2024-02-13T00:37:17.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:16.206505+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2024-02-13T00:37:17.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:16.206672+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:17.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:16.767667+0000 mon.b (mon.1) 15 : audit [DBG] from='client.? 172.21.15.181:0/3845541510' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T00:37:18.529 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:18.526+0000 7fa604f75700 10 client.24274.objecter tick 2024-02-13T00:37:19.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:17.520757+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:19.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:17.520757+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:19.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:17.520757+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:20.450 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:19.521095+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:20.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:19.521095+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:20.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:19.521095+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:22.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:21.521479+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:22.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:21.521479+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:23.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:21.521479+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:23.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:23.527+0000 7fa604f75700 10 client.24274.objecter tick 2024-02-13T00:37:24.041 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:22.596734+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-13T00:37:24.041 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:22.597483+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:24.041 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:22.598117+0000 mgr.y (mgr.14146) 187 : cephadm [INF] Deploying daemon osd.7 on smithi181 2024-02-13T00:37:24.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:22.596734+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-13T00:37:24.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:22.597483+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:24.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:22.598117+0000 mgr.y (mgr.14146) 187 : cephadm [INF] Deploying daemon osd.7 on smithi181 2024-02-13T00:37:24.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:22.596734+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-13T00:37:24.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:22.597483+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:24.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:22.598117+0000 mgr.y (mgr.14146) 187 : cephadm [INF] Deploying daemon osd.7 on smithi181 2024-02-13T00:37:25.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:23.521838+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:25.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:23.521838+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:25.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:23.521838+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:26.926 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:25.522148+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:27.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:25.522148+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:27.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:25.522148+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:28.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:28.528+0000 7fa604f75700 10 client.24274.objecter tick 2024-02-13T00:37:29.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:27.522546+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:27.839205+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:27.848462+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:29.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:27.522546+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:29.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:27.839205+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:29.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:27.848462+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:29.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:27.522546+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:29.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:27.839205+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:29.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:27.848462+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:29.972 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:28.458087+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:29.972 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:28.458196+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:29.973 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:28.841262+0000 mon.a (mon.0) 511 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-13T00:37:29.973 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:28.841550+0000 mon.a (mon.0) 512 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:30.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:28.458087+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:30.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:28.458196+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:30.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:28.841262+0000 mon.a (mon.0) 511 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-13T00:37:30.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:28.841550+0000 mon.a (mon.0) 512 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:30.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:28.458087+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:30.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:28.458196+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:30.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:28.841262+0000 mon.a (mon.0) 511 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-13T00:37:30.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:28.841550+0000 mon.a (mon.0) 512 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:29.523021+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:30.337760+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:30.338610+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:30.339549+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:30.344446+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:31.104 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 7 on host 'smithi181' 2024-02-13T00:37:31.104 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.100+0000 7fa5e7fff700 1 -- 172.21.15.181:0/3676255039 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa6000611d0 con 0x7fa5e8062ba0 2024-02-13T00:37:31.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 10 librados: watch_flush enter 2024-02-13T00:37:31.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 10 librados: watch_flush exit 2024-02-13T00:37:31.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 20 client.24274.objecter shutdown clearing up homeless session... 2024-02-13T00:37:31.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 10 client.24274.objecter successfully canceled tick 2024-02-13T00:37:31.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 -- 172.21.15.181:0/3676255039 >> v2:172.21.15.162:6800/1909610488 conn(0x7fa5e8062ba0 msgr2=0x7fa5e8065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:31.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 --2- 172.21.15.181:0/3676255039 >> v2:172.21.15.162:6800/1909610488 conn(0x7fa5e8062ba0 0x7fa5e8065060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fa6000f7b50 tx=0x7fa5f8009040).stop 2024-02-13T00:37:31.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 -- 172.21.15.181:0/3676255039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6000f8640 msgr2=0x7fa6000f9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:31.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 --2- 172.21.15.181:0/3676255039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6000f8640 0x7fa6000f9520 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa5f0004390 tx=0x7fa5f0009ca0).stop 2024-02-13T00:37:31.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 -- 172.21.15.181:0/3676255039 shutdown_connections 2024-02-13T00:37:31.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 --2- 172.21.15.181:0/3676255039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6000f8640 0x7fa6000f9520 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:31.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 --2- 172.21.15.181:0/3676255039 >> v2:172.21.15.162:6800/1909610488 conn(0x7fa5e8062ba0 0x7fa5e8065060 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:31.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 --2- 172.21.15.181:0/3676255039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6000f6a50 0x7fa6000fa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:31.107 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 --2- 172.21.15.181:0/3676255039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6000f7cf0 0x7fa6000f9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:31.107 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 -- 172.21.15.181:0/3676255039 >> 172.21.15.181:0/3676255039 conn(0x7fa6000f2230 msgr2=0x7fa600060ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:31.107 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 -- 172.21.15.181:0/3676255039 shutdown_connections 2024-02-13T00:37:31.107 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 -- 172.21.15.181:0/3676255039 wait complete. 2024-02-13T00:37:31.107 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:31.102+0000 7fa605776700 1 librados: shutdown 2024-02-13T00:37:31.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:29.523021+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:31.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:30.337760+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:31.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:30.338610+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:31.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:30.339549+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:30.344446+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:31.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:29.523021+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:31.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:30.337760+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:31.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:30.338610+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:31.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:30.339549+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:31.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:30.344446+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:31.822 DEBUG:teuthology.orchestra.run.smithi181:osd.7> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.7.service 2024-02-13T00:37:31.824 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-13T00:37:31.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd stat -f json 2024-02-13T00:37:31.856 INFO:journalctl@ceph.osd.7.smithi181.stdout:-- Logs begin at Tue 2024-02-13 00:17:30 UTC. -- 2024-02-13T00:37:32.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:30.947367+0000 mon.b (mon.1) 16 : audit [INF] from='osd.7 v2:172.21.15.181:6812/1964192772' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T00:37:32.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:30.947914+0000 mon.a (mon.0) 517 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T00:37:32.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:31.100384+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:32.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:31.111841+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:32.273 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:30.947367+0000 mon.b (mon.1) 16 : audit [INF] from='osd.7 v2:172.21.15.181:6812/1964192772' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T00:37:32.273 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:30.947914+0000 mon.a (mon.0) 517 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T00:37:32.273 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:31.100384+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:32.273 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:31.111841+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:32.274 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:30.947367+0000 mon.b (mon.1) 16 : audit [INF] from='osd.7 v2:172.21.15.181:6812/1964192772' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T00:37:32.274 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:30.947914+0000 mon.a (mon.0) 517 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-13T00:37:32.274 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:31.100384+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:32.274 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:31.111841+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:32.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:31.523452+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:32.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:31.858024+0000 mon.a (mon.0) 520 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-13T00:37:32.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:31.858098+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2024-02-13T00:37:32.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:31.858270+0000 mon.a (mon.0) 522 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:32.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:31.858433+0000 mon.b (mon.1) 17 : audit [INF] from='osd.7 v2:172.21.15.181:6812/1964192772' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:37:32.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:31.858943+0000 mon.a (mon.0) 523 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:37:32.929 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:37:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:37:32.870+0000 7f92789c2700 -1 osd.7 0 waiting for initial osdmap 2024-02-13T00:37:32.929 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:37:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:37:32.876+0000 7f927313a700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T00:37:32.929 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:31.523452+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:32.930 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:31.858024+0000 mon.a (mon.0) 520 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-13T00:37:32.930 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:31.858098+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2024-02-13T00:37:32.930 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:31.858270+0000 mon.a (mon.0) 522 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:32.930 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:31.858433+0000 mon.b (mon.1) 17 : audit [INF] from='osd.7 v2:172.21.15.181:6812/1964192772' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:37:32.930 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:31.858943+0000 mon.a (mon.0) 523 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:37:33.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:31.523452+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:33.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:31.858024+0000 mon.a (mon.0) 520 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-13T00:37:33.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:31.858098+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2024-02-13T00:37:33.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:31.858270+0000 mon.a (mon.0) 522 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:33.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:31.858433+0000 mon.b (mon.1) 17 : audit [INF] from='osd.7 v2:172.21.15.181:6812/1964192772' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:37:33.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:31.858943+0000 mon.a (mon.0) 523 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:37:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.818+0000 7f9dd26eb700 1 -- 172.21.15.162:0/3307851789 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9dcc0f7cd0 msgr2=0x7f9dcc0f8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.818+0000 7f9dd26eb700 1 --2- 172.21.15.162:0/3307851789 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9dcc0f7cd0 0x7f9dcc0f8130 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9dbc0048a0 tx=0x7f9dbc009cb0).stop 2024-02-13T00:37:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 -- 172.21.15.162:0/3307851789 shutdown_connections 2024-02-13T00:37:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 --2- 172.21.15.162:0/3307851789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9dcc0f8620 0x7f9dcc0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 --2- 172.21.15.162:0/3307851789 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dcc0f6a30 0x7f9dcc0f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 --2- 172.21.15.162:0/3307851789 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9dcc0f7cd0 0x7f9dcc0f8130 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 -- 172.21.15.162:0/3307851789 >> 172.21.15.162:0/3307851789 conn(0x7f9dcc0f2230 msgr2=0x7f9dcc0f4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 -- 172.21.15.162:0/3307851789 shutdown_connections 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 -- 172.21.15.162:0/3307851789 wait complete. 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 librados: starting msgr at 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 librados: starting objecter 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 Processor -- start 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 -- start start 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 librados: setting wanted keys 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.819+0000 7f9dd26eb700 1 librados: calling monclient init 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dd26eb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dcc0f6a30 0x7f9dcc061530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dd26eb700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9dcc0f7cd0 0x7f9dcc10f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dd26eb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9dcc0f8620 0x7f9dcc111880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dd26eb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc111db0 con 0x7f9dcc0f8620 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dd26eb700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc111ef0 con 0x7f9dcc0f6a30 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dd26eb700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc1120b0 con 0x7f9dcc0f7cd0 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dcb7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9dcc0f7cd0 0x7f9dcc10f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dcb7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9dcc0f7cd0 0x7f9dcc10f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:35656/0 (socket says 172.21.15.162:35656) 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dcb7fe700 1 -- 172.21.15.162:0/1709676103 learned_addr learned my addr 172.21.15.162:0/1709676103 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dcbfff700 1 --2- 172.21.15.162:0/1709676103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dcc0f6a30 0x7f9dcc061530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dd0c88700 1 --2- 172.21.15.162:0/1709676103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9dcc0f8620 0x7f9dcc111880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dcb7fe700 1 -- 172.21.15.162:0/1709676103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dcc0f6a30 msgr2=0x7f9dcc061530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dcb7fe700 1 --2- 172.21.15.162:0/1709676103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dcc0f6a30 0x7f9dcc061530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dcb7fe700 1 -- 172.21.15.162:0/1709676103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9dcc0f8620 msgr2=0x7f9dcc111880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dcb7fe700 1 --2- 172.21.15.162:0/1709676103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9dcc0f8620 0x7f9dcc111880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:33.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dcb7fe700 1 -- 172.21.15.162:0/1709676103 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc005040 con 0x7f9dcc0f7cd0 2024-02-13T00:37:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dcbfff700 1 --2- 172.21.15.162:0/1709676103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dcc0f6a30 0x7f9dcc061530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:37:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dd0c88700 1 --2- 172.21.15.162:0/1709676103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9dcc0f8620 0x7f9dcc111880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:37:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dcb7fe700 1 --2- 172.21.15.162:0/1709676103 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9dcc0f7cd0 0x7f9dcc10f380 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f9dbc004ed0 tx=0x7f9dbc00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dc97fa700 10 client.?.objecter ms_handle_connect 0x7f9dcc0f7cd0 2024-02-13T00:37:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dc97fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:37:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dc97fa700 1 -- 172.21.15.162:0/1709676103 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc019070 con 0x7f9dcc0f7cd0 2024-02-13T00:37:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dc97fa700 1 -- 172.21.15.162:0/1709676103 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9dbc01e420 con 0x7f9dcc0f7cd0 2024-02-13T00:37:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dd26eb700 1 -- 172.21.15.162:0/1709676103 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dcc112330 con 0x7f9dcc0f7cd0 2024-02-13T00:37:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dc97fa700 1 -- 172.21.15.162:0/1709676103 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc007d50 con 0x7f9dcc0f7cd0 2024-02-13T00:37:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dd26eb700 10 client.24314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:37:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dd26eb700 1 -- 172.21.15.162:0/1709676103 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dcc1127b0 con 0x7f9dcc0f7cd0 2024-02-13T00:37:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.820+0000 7f9dd26eb700 1 librados: init done 2024-02-13T00:37:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.822+0000 7f9dd26eb700 1 -- 172.21.15.162:0/1709676103 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dcc05f400 con 0x7f9dcc0f7cd0 2024-02-13T00:37:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.822+0000 7f9dc97fa700 1 -- 172.21.15.162:0/1709676103 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7f9dbc01e850 con 0x7f9dcc0f7cd0 2024-02-13T00:37:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.822+0000 7f9dc97fa700 1 --2- 172.21.15.162:0/1709676103 >> v2:172.21.15.162:6800/1909610488 conn(0x7f9db4062ba0 0x7f9db4065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.822+0000 7f9dc97fa700 1 -- 172.21.15.162:0/1709676103 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 3914+0+0 (secure 0 0 0) 0x7f9dbc07cbc0 con 0x7f9dcc0f7cd0 2024-02-13T00:37:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.822+0000 7f9dc97fa700 10 client.24314.objecter ms_dispatch 0x7f9dcc001550 osd_map(52..52 src has 1..52) v4 2024-02-13T00:37:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.822+0000 7f9dc97fa700 3 client.24314.objecter handle_osd_map got epochs [52,52] > 0 2024-02-13T00:37:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.822+0000 7f9dc97fa700 3 client.24314.objecter handle_osd_map decoding full epoch 52 2024-02-13T00:37:33.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.822+0000 7f9dcbfff700 1 --2- 172.21.15.162:0/1709676103 >> v2:172.21.15.162:6800/1909610488 conn(0x7f9db4062ba0 0x7f9db4065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:33.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.822+0000 7f9dc97fa700 20 client.24314.objecter dump_active .. 0 homeless 2024-02-13T00:37:33.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.822+0000 7f9dcbfff700 1 --2- 172.21.15.162:0/1709676103 >> v2:172.21.15.162:6800/1909610488 conn(0x7f9db4062ba0 0x7f9db4065060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9dcc0f7b30 tx=0x7f9dc000b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:33.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.822+0000 7f9dc97fa700 10 client.24314.objecter ms_handle_connect 0x7f9db4062ba0 2024-02-13T00:37:33.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.826+0000 7f9dc97fa700 1 -- 172.21.15.162:0/1709676103 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9dbc04d780 con 0x7f9dcc0f7cd0 2024-02-13T00:37:33.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.995+0000 7f9dd26eb700 1 -- 172.21.15.162:0/1709676103 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f9dcc0fe480 con 0x7f9dcc0f7cd0 2024-02-13T00:37:34.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.996+0000 7f9dc97fa700 1 -- 172.21.15.162:0/1709676103 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v53) v1 ==== 74+0+130 (secure 0 0 0) 0x7f9dbc023030 con 0x7f9dcc0f7cd0 2024-02-13T00:37:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:37:34.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 10 librados: watch_flush enter 2024-02-13T00:37:34.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 10 librados: watch_flush exit 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 20 client.24314.objecter shutdown clearing up homeless session... 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 10 client.24314.objecter successfully canceled tick 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 1 -- 172.21.15.162:0/1709676103 >> v2:172.21.15.162:6800/1909610488 conn(0x7f9db4062ba0 msgr2=0x7f9db4065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 1 --2- 172.21.15.162:0/1709676103 >> v2:172.21.15.162:6800/1909610488 conn(0x7f9db4062ba0 0x7f9db4065060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9dcc0f7b30 tx=0x7f9dc000b040).stop 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 1 -- 172.21.15.162:0/1709676103 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9dcc0f7cd0 msgr2=0x7f9dcc10f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 1 --2- 172.21.15.162:0/1709676103 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9dcc0f7cd0 0x7f9dcc10f380 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f9dbc004ed0 tx=0x7f9dbc00b040).stop 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 1 -- 172.21.15.162:0/1709676103 shutdown_connections 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 1 --2- 172.21.15.162:0/1709676103 >> v2:172.21.15.162:6800/1909610488 conn(0x7f9db4062ba0 0x7f9db4065060 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 1 --2- 172.21.15.162:0/1709676103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9dcc0f8620 0x7f9dcc111880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 1 --2- 172.21.15.162:0/1709676103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dcc0f6a30 0x7f9dcc061530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 1 --2- 172.21.15.162:0/1709676103 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9dcc0f7cd0 0x7f9dcc10f380 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 1 -- 172.21.15.162:0/1709676103 >> 172.21.15.162:0/1709676103 conn(0x7f9dcc0f2230 msgr2=0x7f9dcc0fb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 1 -- 172.21.15.162:0/1709676103 shutdown_connections 2024-02-13T00:37:34.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.997+0000 7f9dd26eb700 1 -- 172.21.15.162:0/1709676103 wait complete. 2024-02-13T00:37:34.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:37:33.998+0000 7f9dd26eb700 1 librados: shutdown 2024-02-13T00:37:34.247 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:32.863772+0000 mon.a (mon.0) 524 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-13T00:37:34.248 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:32.863859+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2024-02-13T00:37:34.248 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:32.864269+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:34.249 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:32.867343+0000 mon.a (mon.0) 527 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:34.249 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:33.860149+0000 mon.a (mon.0) 528 : cluster [INF] osd.7 v2:172.21.15.181:6812/1964192772 boot 2024-02-13T00:37:34.249 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:33.860194+0000 mon.a (mon.0) 529 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-13T00:37:34.249 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:33.860327+0000 mon.a (mon.0) 530 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:34.316 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:32.863772+0000 mon.a (mon.0) 524 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-13T00:37:34.316 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:32.863859+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2024-02-13T00:37:34.316 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:32.864269+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:34.317 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:32.867343+0000 mon.a (mon.0) 527 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:34.317 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:33.860149+0000 mon.a (mon.0) 528 : cluster [INF] osd.7 v2:172.21.15.181:6812/1964192772 boot 2024-02-13T00:37:34.317 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:33.860194+0000 mon.a (mon.0) 529 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-13T00:37:34.317 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:33.860327+0000 mon.a (mon.0) 530 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:34.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:32.863772+0000 mon.a (mon.0) 524 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-02-13T00:37:34.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:32.863859+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2024-02-13T00:37:34.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:32.864269+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:34.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:32.867343+0000 mon.a (mon.0) 527 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:34.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:33.860149+0000 mon.a (mon.0) 528 : cluster [INF] osd.7 v2:172.21.15.181:6812/1964192772 boot 2024-02-13T00:37:34.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:33.860194+0000 mon.a (mon.0) 529 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-13T00:37:34.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:33.860327+0000 mon.a (mon.0) 530 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:37:34.718 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":53,"num_osds":8,"num_up_osds":8,"osd_up_since":1707784653,"num_in_osds":8,"osd_in_since":1707784636,"num_remapped_pgs":1} 2024-02-13T00:37:34.718 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi162 2024-02-13T00:37:34.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi162=foo.a' 2024-02-13T00:37:35.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:31.903075+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:35.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:31.903161+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:35.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:33.524012+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:35.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:33.997634+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? 172.21.15.162:0/1709676103' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T00:37:35.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:34.196370+0000 mon.a (mon.0) 531 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-13T00:37:35.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:34.794542+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:35.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:34.795389+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:35.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:34.795979+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:35.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:34.796469+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:35.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:34.797002+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:35.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:34.800850+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:35.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:34.801534+0000 mon.a (mon.0) 538 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:35.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:34.802068+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:35.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:34.806096+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:35.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:31.903075+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:35.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:31.903161+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:33.524012+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:33.997634+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? 172.21.15.162:0/1709676103' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T00:37:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:34.196370+0000 mon.a (mon.0) 531 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-13T00:37:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:34.794542+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:34.795389+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:34.795979+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:34.796469+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:34.797002+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:34.800850+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:34.801534+0000 mon.a (mon.0) 538 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:34.802068+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:34.806096+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:35.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:31.903075+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:35.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:31.903161+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:35.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:33.524012+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:37:35.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:33.997634+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? 172.21.15.162:0/1709676103' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T00:37:35.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:34.196370+0000 mon.a (mon.0) 531 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-13T00:37:35.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:34.794542+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:35.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:34.795389+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:35.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:34.795979+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:35.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:34.796469+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:35.159 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:34.797002+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:37:35.159 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:34.800850+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:35.159 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:34.801534+0000 mon.a (mon.0) 538 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:35.159 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:34.802068+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:35.159 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:34.806096+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:36.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:34.789882+0000 mgr.y (mgr.14146) 194 : cephadm [INF] Detected new or changed devices on smithi181 2024-02-13T00:37:36.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:34.797395+0000 mgr.y (mgr.14146) 195 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 4280M 2024-02-13T00:37:36.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:35.201080+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-13T00:37:36.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:34.789882+0000 mgr.y (mgr.14146) 194 : cephadm [INF] Detected new or changed devices on smithi181 2024-02-13T00:37:36.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:34.797395+0000 mgr.y (mgr.14146) 195 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 4280M 2024-02-13T00:37:36.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:35.201080+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-13T00:37:36.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:34.789882+0000 mgr.y (mgr.14146) 194 : cephadm [INF] Detected new or changed devices on smithi181 2024-02-13T00:37:36.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:34.797395+0000 mgr.y (mgr.14146) 195 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 4280M 2024-02-13T00:37:36.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:35.201080+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-13T00:37:36.536 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.531+0000 7f9e7f381700 1 -- 172.21.15.181:0/1979933369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e780f7b20 msgr2=0x7f9e780f7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:36.536 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.531+0000 7f9e7f381700 1 --2- 172.21.15.181:0/1979933369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e780f7b20 0x7f9e780f7f40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9e680048a0 tx=0x7f9e68009cb0).stop 2024-02-13T00:37:36.536 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.531+0000 7f9e7f381700 1 -- 172.21.15.181:0/1979933369 shutdown_connections 2024-02-13T00:37:36.536 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.531+0000 7f9e7f381700 1 --2- 172.21.15.181:0/1979933369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e780f9710 0x7f9e78100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:36.536 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.531+0000 7f9e7f381700 1 --2- 172.21.15.181:0/1979933369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e780f7b20 0x7f9e780f7f40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:36.536 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.531+0000 7f9e7f381700 1 --2- 172.21.15.181:0/1979933369 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e780f8dc0 0x7f9e780f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:36.536 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.531+0000 7f9e7f381700 1 -- 172.21.15.181:0/1979933369 >> 172.21.15.181:0/1979933369 conn(0x7f9e780f3340 msgr2=0x7f9e780f5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:36.536 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.531+0000 7f9e7f381700 1 -- 172.21.15.181:0/1979933369 shutdown_connections 2024-02-13T00:37:36.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.531+0000 7f9e7f381700 1 -- 172.21.15.181:0/1979933369 wait complete. 2024-02-13T00:37:36.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.531+0000 7f9e7f381700 1 librados: starting msgr at 2024-02-13T00:37:36.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.531+0000 7f9e7f381700 1 librados: starting objecter 2024-02-13T00:37:36.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.531+0000 7f9e7f381700 1 Processor -- start 2024-02-13T00:37:36.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 1 -- start start 2024-02-13T00:37:36.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 1 librados: setting wanted keys 2024-02-13T00:37:36.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 1 librados: calling monclient init 2024-02-13T00:37:36.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e780f7b20 0x7f9e78108050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:36.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e780f8dc0 0x7f9e78108540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:36.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e780f9710 0x7f9e7810f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:36.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7810fb10 con 0x7f9e780f8dc0 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7810fc50 con 0x7f9e780f7b20 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7810fd90 con 0x7f9e780f9710 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7d11d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e780f7b20 0x7f9e78108050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7d11d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e780f7b20 0x7f9e78108050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:51306/0 (socket says 172.21.15.181:51306) 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7d11d700 1 -- 172.21.15.181:0/1399921812 learned_addr learned my addr 172.21.15.181:0/1399921812 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7c91c700 1 --2- 172.21.15.181:0/1399921812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e780f8dc0 0x7f9e78108540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7d91e700 1 --2- 172.21.15.181:0/1399921812 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e780f9710 0x7f9e7810f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7d11d700 1 -- 172.21.15.181:0/1399921812 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e780f9710 msgr2=0x7f9e7810f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7d11d700 1 --2- 172.21.15.181:0/1399921812 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e780f9710 0x7f9e7810f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7d11d700 1 -- 172.21.15.181:0/1399921812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e780f8dc0 msgr2=0x7f9e78108540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7d11d700 1 --2- 172.21.15.181:0/1399921812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e780f8dc0 0x7f9e78108540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7d11d700 1 -- 172.21.15.181:0/1399921812 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e68005040 con 0x7f9e780f7b20 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7d11d700 1 --2- 172.21.15.181:0/1399921812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e780f7b20 0x7f9e78108050 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f9e6800b1a0 tx=0x7f9e68009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:36.538 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e6e7fc700 10 client.?.objecter ms_handle_connect 0x7f9e780f7b20 2024-02-13T00:37:36.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e6e7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:37:36.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e6e7fc700 1 -- 172.21.15.181:0/1399921812 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6800e030 con 0x7f9e780f7b20 2024-02-13T00:37:36.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 1 -- 172.21.15.181:0/1399921812 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e78110010 con 0x7f9e780f7b20 2024-02-13T00:37:36.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e6e7fc700 1 -- 172.21.15.181:0/1399921812 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e68004100 con 0x7f9e780f7b20 2024-02-13T00:37:36.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e6e7fc700 1 -- 172.21.15.181:0/1399921812 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68018070 con 0x7f9e780f7b20 2024-02-13T00:37:36.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 10 client.24307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:37:36.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 1 -- 172.21.15.181:0/1399921812 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e781104f0 con 0x7f9e780f7b20 2024-02-13T00:37:36.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.532+0000 7f9e7f381700 1 librados: init done 2024-02-13T00:37:36.540 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.533+0000 7f9e7f381700 1 -- 172.21.15.181:0/1399921812 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7805f400 con 0x7f9e780f7b20 2024-02-13T00:37:36.541 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.534+0000 7f9e6e7fc700 1 -- 172.21.15.181:0/1399921812 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7f9e6801d420 con 0x7f9e780f7b20 2024-02-13T00:37:36.541 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.535+0000 7f9e6e7fc700 1 --2- 172.21.15.181:0/1399921812 >> v2:172.21.15.162:6800/1909610488 conn(0x7f9e64062ba0 0x7f9e64065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:36.541 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.535+0000 7f9e6e7fc700 1 -- 172.21.15.181:0/1399921812 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4054+0+0 (secure 0 0 0) 0x7f9e6807ded0 con 0x7f9e780f7b20 2024-02-13T00:37:36.541 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.535+0000 7f9e6e7fc700 10 client.24307.objecter ms_dispatch 0x7f9e78001550 osd_map(55..55 src has 1..55) v4 2024-02-13T00:37:36.541 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.535+0000 7f9e6e7fc700 3 client.24307.objecter handle_osd_map got epochs [55,55] > 0 2024-02-13T00:37:36.541 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.535+0000 7f9e6e7fc700 3 client.24307.objecter handle_osd_map decoding full epoch 55 2024-02-13T00:37:36.541 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.535+0000 7f9e6e7fc700 20 client.24307.objecter dump_active .. 0 homeless 2024-02-13T00:37:36.541 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.535+0000 7f9e7c91c700 1 --2- 172.21.15.181:0/1399921812 >> v2:172.21.15.162:6800/1909610488 conn(0x7f9e64062ba0 0x7f9e64065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:36.541 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.535+0000 7f9e6e7fc700 10 client.24307.objecter ms_handle_connect 0x7f9e64062ba0 2024-02-13T00:37:36.541 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.535+0000 7f9e7c91c700 1 --2- 172.21.15.181:0/1399921812 >> v2:172.21.15.162:6800/1909610488 conn(0x7f9e64062ba0 0x7f9e64065060 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9e74004de0 tx=0x7f9e7400b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:36.542 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.539+0000 7f9e6e7fc700 1 -- 172.21.15.181:0/1399921812 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9e6804e9f0 con 0x7f9e780f7b20 2024-02-13T00:37:36.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.711+0000 7f9e7f381700 1 -- 172.21.15.181:0/1399921812 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi162=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e780fef60 con 0x7f9e64062ba0 2024-02-13T00:37:36.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.719+0000 7f9e6e7fc700 1 -- 172.21.15.181:0/1399921812 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (secure 0 0 0) 0x7f9e780fef60 con 0x7f9e64062ba0 2024-02-13T00:37:36.722 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled rgw.foo.a update... 2024-02-13T00:37:36.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.720+0000 7f9e7f381700 10 librados: watch_flush enter 2024-02-13T00:37:36.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 10 librados: watch_flush exit 2024-02-13T00:37:36.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 20 client.24307.objecter shutdown clearing up homeless session... 2024-02-13T00:37:36.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 10 client.24307.objecter successfully canceled tick 2024-02-13T00:37:36.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 -- 172.21.15.181:0/1399921812 >> v2:172.21.15.162:6800/1909610488 conn(0x7f9e64062ba0 msgr2=0x7f9e64065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:36.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 --2- 172.21.15.181:0/1399921812 >> v2:172.21.15.162:6800/1909610488 conn(0x7f9e64062ba0 0x7f9e64065060 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9e74004de0 tx=0x7f9e7400b040).stop 2024-02-13T00:37:36.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 -- 172.21.15.181:0/1399921812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e780f7b20 msgr2=0x7f9e78108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:36.726 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 --2- 172.21.15.181:0/1399921812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e780f7b20 0x7f9e78108050 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f9e6800b1a0 tx=0x7f9e68009cb0).stop 2024-02-13T00:37:36.726 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 -- 172.21.15.181:0/1399921812 shutdown_connections 2024-02-13T00:37:36.726 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 --2- 172.21.15.181:0/1399921812 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e780f9710 0x7f9e7810f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:36.726 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 --2- 172.21.15.181:0/1399921812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e780f7b20 0x7f9e78108050 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:36.726 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 --2- 172.21.15.181:0/1399921812 >> v2:172.21.15.162:6800/1909610488 conn(0x7f9e64062ba0 0x7f9e64065060 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:36.726 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 --2- 172.21.15.181:0/1399921812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e780f8dc0 0x7f9e78108540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:36.727 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 -- 172.21.15.181:0/1399921812 >> 172.21.15.181:0/1399921812 conn(0x7f9e780f3340 msgr2=0x7f9e780f6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:36.727 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 -- 172.21.15.181:0/1399921812 shutdown_connections 2024-02-13T00:37:36.727 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 -- 172.21.15.181:0/1399921812 wait complete. 2024-02-13T00:37:36.727 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:36.721+0000 7f9e7f381700 1 librados: shutdown 2024-02-13T00:37:37.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:35.524378+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:36.719542+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:36.725864+0000 mon.a (mon.0) 543 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:36.727476+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:36.728159+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:36.733512+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:36.739134+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:36.739771+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T00:37:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:36.743566+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T00:37:37.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:36.748287+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:37.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:36.749055+0000 mon.a (mon.0) 551 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:37.244 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:35.524378+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:37.244 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:36.719542+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:37.244 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:36.725864+0000 mon.a (mon.0) 543 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:37.244 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:36.727476+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:37.244 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:36.728159+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:37.244 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:36.733512+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:37.244 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:36.739134+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:37.244 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:36.739771+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T00:37:37.244 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:36.743566+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T00:37:37.244 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:36.748287+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:37.245 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:36.749055+0000 mon.a (mon.0) 551 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:37.245 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:35.524378+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:37.245 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:36.719542+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:37.245 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:36.725864+0000 mon.a (mon.0) 543 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:37.245 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:36.727476+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:37.245 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:36.728159+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:37.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:36.733512+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:37.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:36.739134+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:37.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:36.739771+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T00:37:37.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:36.743566+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T00:37:37.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:36.748287+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:37.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:36.749055+0000 mon.a (mon.0) 551 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:37.438 DEBUG:teuthology.orchestra.run.smithi162:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@rgw.foo.a.service 2024-02-13T00:37:37.441 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi181 2024-02-13T00:37:37.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool create iscsi 3 3 replicated 2024-02-13T00:37:37.471 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:-- Logs begin at Tue 2024-02-13 00:17:45 UTC. -- 2024-02-13T00:37:38.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:36.713805+0000 mgr.y (mgr.14146) 197 : audit [DBG] from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi162=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:36.715483+0000 mgr.y (mgr.14146) 198 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi162=foo.a;count:1 2024-02-13T00:37:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:36.735555+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi162=foo.a;count:1 2024-02-13T00:37:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:36.749654+0000 mgr.y (mgr.14146) 200 : cephadm [INF] Deploying daemon rgw.foo.a on smithi162 2024-02-13T00:37:38.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:36.713805+0000 mgr.y (mgr.14146) 197 : audit [DBG] from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi162=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:38.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:36.715483+0000 mgr.y (mgr.14146) 198 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi162=foo.a;count:1 2024-02-13T00:37:38.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:36.735555+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi162=foo.a;count:1 2024-02-13T00:37:38.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:36.749654+0000 mgr.y (mgr.14146) 200 : cephadm [INF] Deploying daemon rgw.foo.a on smithi162 2024-02-13T00:37:38.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:36.713805+0000 mgr.y (mgr.14146) 197 : audit [DBG] from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi162=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:38.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:36.715483+0000 mgr.y (mgr.14146) 198 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi162=foo.a;count:1 2024-02-13T00:37:38.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:36.735555+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi162=foo.a;count:1 2024-02-13T00:37:38.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:36.749654+0000 mgr.y (mgr.14146) 200 : cephadm [INF] Deploying daemon rgw.foo.a on smithi162 2024-02-13T00:37:38.830 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:37:38 smithi162 systemd[1]: Starting Ceph rgw.foo.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:37:39.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:37.524700+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:39.263 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:37.524700+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:39.263 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:37.524700+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:39.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.323+0000 7fe180cd6700 1 -- 172.21.15.181:0/860981128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0f8dc0 msgr2=0x7fe17c0f9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:39.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.323+0000 7fe180cd6700 1 --2- 172.21.15.181:0/860981128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0f8dc0 0x7fe17c0f9220 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe16c0048a0 tx=0x7fe16c009cb0).stop 2024-02-13T00:37:39.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.323+0000 7fe180cd6700 1 -- 172.21.15.181:0/860981128 shutdown_connections 2024-02-13T00:37:39.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.323+0000 7fe180cd6700 1 --2- 172.21.15.181:0/860981128 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe17c0f9710 0x7fe17c100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:39.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.323+0000 7fe180cd6700 1 --2- 172.21.15.181:0/860981128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe17c0f7b20 0x7fe17c0f7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:39.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.323+0000 7fe180cd6700 1 --2- 172.21.15.181:0/860981128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0f8dc0 0x7fe17c0f9220 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:39.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.323+0000 7fe180cd6700 1 -- 172.21.15.181:0/860981128 >> 172.21.15.181:0/860981128 conn(0x7fe17c0f3340 msgr2=0x7fe17c0f5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:39.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.323+0000 7fe180cd6700 1 -- 172.21.15.181:0/860981128 shutdown_connections 2024-02-13T00:37:39.328 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.323+0000 7fe180cd6700 1 -- 172.21.15.181:0/860981128 wait complete. 2024-02-13T00:37:39.328 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe180cd6700 1 librados: starting msgr at 2024-02-13T00:37:39.328 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe180cd6700 1 librados: starting objecter 2024-02-13T00:37:39.334 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe180cd6700 1 Processor -- start 2024-02-13T00:37:39.334 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe180cd6700 1 -- start start 2024-02-13T00:37:39.334 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe180cd6700 1 librados: setting wanted keys 2024-02-13T00:37:39.334 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe180cd6700 1 librados: calling monclient init 2024-02-13T00:37:39.334 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe180cd6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe17c0f7b20 0x7fe17c108010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:39.334 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe180cd6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0f8dc0 0x7fe17c108500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:39.334 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe180cd6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe17c0f9710 0x7fe17c10f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:39.334 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe180cd6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c10fad0 con 0x7fe17c0f9710 2024-02-13T00:37:39.335 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe180cd6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c10fc10 con 0x7fe17c0f8dc0 2024-02-13T00:37:39.335 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe180cd6700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c10fd50 con 0x7fe17c0f7b20 2024-02-13T00:37:39.335 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe179d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0f8dc0 0x7fe17c108500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:39.335 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe179d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0f8dc0 0x7fe17c108500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:33386/0 (socket says 172.21.15.181:33386) 2024-02-13T00:37:39.335 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe179d9b700 1 -- 172.21.15.181:0/102668268 learned_addr learned my addr 172.21.15.181:0/102668268 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:37:39.335 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.324+0000 7fe17a59c700 1 --2- 172.21.15.181:0/102668268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe17c0f7b20 0x7fe17c108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:39.335 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe17ad9d700 1 --2- 172.21.15.181:0/102668268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe17c0f9710 0x7fe17c10f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:39.335 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe179d9b700 1 -- 172.21.15.181:0/102668268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe17c0f7b20 msgr2=0x7fe17c108010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe179d9b700 1 --2- 172.21.15.181:0/102668268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe17c0f7b20 0x7fe17c108010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe179d9b700 1 -- 172.21.15.181:0/102668268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe17c0f9710 msgr2=0x7fe17c10f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe179d9b700 1 --2- 172.21.15.181:0/102668268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe17c0f9710 0x7fe17c10f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe179d9b700 1 -- 172.21.15.181:0/102668268 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c005040 con 0x7fe17c0f8dc0 2024-02-13T00:37:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe179d9b700 1 --2- 172.21.15.181:0/102668268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0f8dc0 0x7fe17c108500 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fe16c009da0 tx=0x7fe16c009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe16b7fe700 10 client.?.objecter ms_handle_connect 0x7fe17c0f8dc0 2024-02-13T00:37:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe16b7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:37:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe16b7fe700 1 -- 172.21.15.181:0/102668268 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c00e070 con 0x7fe17c0f8dc0 2024-02-13T00:37:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe180cd6700 1 -- 172.21.15.181:0/102668268 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17c10ffd0 con 0x7fe17c0f8dc0 2024-02-13T00:37:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe16b7fe700 1 -- 172.21.15.181:0/102668268 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe16c004100 con 0x7fe17c0f8dc0 2024-02-13T00:37:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe16b7fe700 1 -- 172.21.15.181:0/102668268 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c018070 con 0x7fe17c0f8dc0 2024-02-13T00:37:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe180cd6700 10 client.24322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe180cd6700 1 -- 172.21.15.181:0/102668268 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17c110450 con 0x7fe17c0f8dc0 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.325+0000 7fe180cd6700 1 librados: init done 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.326+0000 7fe180cd6700 1 -- 172.21.15.181:0/102668268 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe17c05f400 con 0x7fe17c0f8dc0 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.327+0000 7fe16b7fe700 1 -- 172.21.15.181:0/102668268 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7fe16c012920 con 0x7fe17c0f8dc0 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.327+0000 7fe16b7fe700 1 --2- 172.21.15.181:0/102668268 >> v2:172.21.15.162:6800/1909610488 conn(0x7fe164062ba0 0x7fe164065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.327+0000 7fe16b7fe700 1 -- 172.21.15.181:0/102668268 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4054+0+0 (secure 0 0 0) 0x7fe16c07d070 con 0x7fe17c0f8dc0 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.327+0000 7fe16b7fe700 10 client.24322.objecter ms_dispatch 0x7fe17c001550 osd_map(55..55 src has 1..55) v4 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.327+0000 7fe16b7fe700 3 client.24322.objecter handle_osd_map got epochs [55,55] > 0 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.327+0000 7fe16b7fe700 3 client.24322.objecter handle_osd_map decoding full epoch 55 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.327+0000 7fe16b7fe700 20 client.24322.objecter dump_active .. 0 homeless 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.327+0000 7fe17a59c700 1 --2- 172.21.15.181:0/102668268 >> v2:172.21.15.162:6800/1909610488 conn(0x7fe164062ba0 0x7fe164065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.327+0000 7fe16b7fe700 10 client.24322.objecter ms_handle_connect 0x7fe164062ba0 2024-02-13T00:37:39.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.327+0000 7fe17a59c700 1 --2- 172.21.15.181:0/102668268 >> v2:172.21.15.162:6800/1909610488 conn(0x7fe164062ba0 0x7fe164065060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fe17c0f8c20 tx=0x7fe17000b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:39.338 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.331+0000 7fe16b7fe700 1 -- 172.21.15.181:0/102668268 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe16c04daf0 con 0x7fe17c0f8dc0 2024-02-13T00:37:39.496 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:39.493+0000 7fe180cd6700 1 -- 172.21.15.181:0/102668268 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7fe17c0008d0 con 0x7fe17c0f8dc0 2024-02-13T00:37:39.579 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:37:39 smithi162 bash[66280]: e4ac7361cee00c32e070bf7c0a41e6d59d13830eb0a7fe0aa50c942a2f52b3fd 2024-02-13T00:37:39.579 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:37:39 smithi162 systemd[1]: Started Ceph rgw.foo.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:37:40.210 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.207+0000 7fe16b7fe700 1 -- 172.21.15.181:0/102668268 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'iscsi' created v56) v1 ==== 154+0+0 (secure 0 0 0) 0x7fe16c022070 con 0x7fe17c0f8dc0 2024-02-13T00:37:40.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 10 librados: watch_flush enter 2024-02-13T00:37:40.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 10 librados: watch_flush exit 2024-02-13T00:37:40.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 20 client.24322.objecter shutdown clearing up homeless session... 2024-02-13T00:37:40.214 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 10 client.24322.objecter successfully canceled tick 2024-02-13T00:37:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 1 -- 172.21.15.181:0/102668268 >> v2:172.21.15.162:6800/1909610488 conn(0x7fe164062ba0 msgr2=0x7fe164065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 1 --2- 172.21.15.181:0/102668268 >> v2:172.21.15.162:6800/1909610488 conn(0x7fe164062ba0 0x7fe164065060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fe17c0f8c20 tx=0x7fe17000b040).stop 2024-02-13T00:37:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 1 -- 172.21.15.181:0/102668268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0f8dc0 msgr2=0x7fe17c108500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 1 --2- 172.21.15.181:0/102668268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0f8dc0 0x7fe17c108500 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fe16c009da0 tx=0x7fe16c009cb0).stop 2024-02-13T00:37:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 1 -- 172.21.15.181:0/102668268 shutdown_connections 2024-02-13T00:37:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 1 --2- 172.21.15.181:0/102668268 >> v2:172.21.15.162:6800/1909610488 conn(0x7fe164062ba0 0x7fe164065060 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 1 --2- 172.21.15.181:0/102668268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe17c0f9710 0x7fe17c10f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:40.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 1 --2- 172.21.15.181:0/102668268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe17c0f7b20 0x7fe17c108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:40.216 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 1 --2- 172.21.15.181:0/102668268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0f8dc0 0x7fe17c108500 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:40.216 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.211+0000 7fe180cd6700 1 -- 172.21.15.181:0/102668268 >> 172.21.15.181:0/102668268 conn(0x7fe17c0f3340 msgr2=0x7fe17c0f6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:40.216 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.212+0000 7fe180cd6700 1 -- 172.21.15.181:0/102668268 shutdown_connections 2024-02-13T00:37:40.216 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.212+0000 7fe180cd6700 1 -- 172.21.15.181:0/102668268 wait complete. 2024-02-13T00:37:40.216 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:40.212+0000 7fe180cd6700 1 librados: shutdown 2024-02-13T00:37:40.216 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'iscsi' created 2024-02-13T00:37:40.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:39.464414+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:40.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:39.465475+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:39.494335+0000 mon.b (mon.1) 18 : audit [INF] from='client.? 172.21.15.181:0/102668268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-13T00:37:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:39.494939+0000 mon.a (mon.0) 554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-13T00:37:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:39.525070+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:40.207078+0000 mon.a (mon.0) 555 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-13T00:37:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:40.207160+0000 mon.a (mon.0) 556 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-13T00:37:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:40.211266+0000 mon.c (mon.2) 11 : audit [INF] from='client.? 172.21.15.162:0/3408777347' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T00:37:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:40.211645+0000 mon.a (mon.0) 557 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T00:37:40.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:39.464414+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:40.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:39.465475+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:39.494335+0000 mon.b (mon.1) 18 : audit [INF] from='client.? 172.21.15.181:0/102668268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-13T00:37:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:39.494939+0000 mon.a (mon.0) 554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-13T00:37:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:39.525070+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:40.207078+0000 mon.a (mon.0) 555 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-13T00:37:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:40.207160+0000 mon.a (mon.0) 556 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-13T00:37:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:40.211266+0000 mon.c (mon.2) 11 : audit [INF] from='client.? 172.21.15.162:0/3408777347' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T00:37:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:40.211645+0000 mon.a (mon.0) 557 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T00:37:40.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:39.464414+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:39.465475+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:39.494335+0000 mon.b (mon.1) 18 : audit [INF] from='client.? 172.21.15.181:0/102668268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-13T00:37:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:39.494939+0000 mon.a (mon.0) 554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-13T00:37:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:39.525070+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:40.207078+0000 mon.a (mon.0) 555 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-13T00:37:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:40.207160+0000 mon.a (mon.0) 556 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-13T00:37:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:40.211266+0000 mon.c (mon.2) 11 : audit [INF] from='client.? 172.21.15.162:0/3408777347' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T00:37:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:40.211645+0000 mon.a (mon.0) 557 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T00:37:40.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool application enable iscsi rbd 2024-02-13T00:37:42.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:40.751571+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:40.752380+0000 mon.a (mon.0) 559 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:40.752970+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:40.756790+0000 mgr.y (mgr.14146) 203 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-13T00:37:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:41.208920+0000 mon.a (mon.0) 561 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-13T00:37:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:41.209032+0000 mon.a (mon.0) 562 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-13T00:37:42.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:40.751571+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:42.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:40.752380+0000 mon.a (mon.0) 559 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:42.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:40.752970+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:42.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:40.756790+0000 mgr.y (mgr.14146) 203 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-13T00:37:42.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:41.208920+0000 mon.a (mon.0) 561 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-13T00:37:42.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:41.209032+0000 mon.a (mon.0) 562 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-13T00:37:42.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:40.751571+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:42.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:40.752380+0000 mon.a (mon.0) 559 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:42.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:40.752970+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:42.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:40.756790+0000 mgr.y (mgr.14146) 203 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-13T00:37:42.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:41.208920+0000 mon.a (mon.0) 561 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-13T00:37:42.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:41.209032+0000 mon.a (mon.0) 562 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-13T00:37:42.780 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.774+0000 7f94fe72b700 1 -- 172.21.15.181:0/64009099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f80f7cd0 msgr2=0x7f94f80f8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:42.780 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 --2- 172.21.15.181:0/64009099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f80f7cd0 0x7f94f80f8130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f94e80048a0 tx=0x7f94e8009cb0).stop 2024-02-13T00:37:42.780 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 -- 172.21.15.181:0/64009099 shutdown_connections 2024-02-13T00:37:42.780 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 --2- 172.21.15.181:0/64009099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94f80f8620 0x7f94f80fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:42.780 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 --2- 172.21.15.181:0/64009099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94f80f6a30 0x7f94f80f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 --2- 172.21.15.181:0/64009099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f80f7cd0 0x7f94f80f8130 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 -- 172.21.15.181:0/64009099 >> 172.21.15.181:0/64009099 conn(0x7f94f80f2230 msgr2=0x7f94f80f4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 -- 172.21.15.181:0/64009099 shutdown_connections 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 -- 172.21.15.181:0/64009099 wait complete. 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 librados: starting msgr at 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 librados: starting objecter 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 Processor -- start 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 -- start start 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 librados: setting wanted keys 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.775+0000 7f94fe72b700 1 librados: calling monclient init 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94fe72b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94f80f6a30 0x7f94f8108020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94fe72b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f80f7cd0 0x7f94f8108510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94fe72b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94f80f8620 0x7f94f810f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94fe72b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f810fae0 con 0x7f94f80f8620 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94fe72b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f810fc20 con 0x7f94f80f7cd0 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94fe72b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f810fd60 con 0x7f94f80f6a30 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f77fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f80f7cd0 0x7f94f8108510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f77fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f80f7cd0 0x7f94f8108510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:33404/0 (socket says 172.21.15.181:33404) 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f77fe700 1 -- 172.21.15.181:0/2876934122 learned_addr learned my addr 172.21.15.181:0/2876934122 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94fccc8700 1 --2- 172.21.15.181:0/2876934122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94f80f8620 0x7f94f810f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f7fff700 1 --2- 172.21.15.181:0/2876934122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94f80f6a30 0x7f94f8108020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f77fe700 1 -- 172.21.15.181:0/2876934122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94f80f6a30 msgr2=0x7f94f8108020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f77fe700 1 --2- 172.21.15.181:0/2876934122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94f80f6a30 0x7f94f8108020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f77fe700 1 -- 172.21.15.181:0/2876934122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94f80f8620 msgr2=0x7f94f810f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f77fe700 1 --2- 172.21.15.181:0/2876934122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94f80f8620 0x7f94f810f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:42.782 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f77fe700 1 -- 172.21.15.181:0/2876934122 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e8005040 con 0x7f94f80f7cd0 2024-02-13T00:37:42.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f77fe700 1 --2- 172.21.15.181:0/2876934122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f80f7cd0 0x7f94f8108510 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f94e80045a0 tx=0x7f94e8009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:42.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f57fa700 10 client.?.objecter ms_handle_connect 0x7f94f80f7cd0 2024-02-13T00:37:42.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f57fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:37:42.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f57fa700 1 -- 172.21.15.181:0/2876934122 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e800e030 con 0x7f94f80f7cd0 2024-02-13T00:37:42.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f57fa700 1 -- 172.21.15.181:0/2876934122 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94e80093b0 con 0x7f94f80f7cd0 2024-02-13T00:37:42.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94fe72b700 1 -- 172.21.15.181:0/2876934122 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f810ffe0 con 0x7f94f80f7cd0 2024-02-13T00:37:42.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94f57fa700 1 -- 172.21.15.181:0/2876934122 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e8018070 con 0x7f94f80f7cd0 2024-02-13T00:37:42.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94fe72b700 10 client.24352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:37:42.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94fe72b700 1 -- 172.21.15.181:0/2876934122 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f8110390 con 0x7f94f80f7cd0 2024-02-13T00:37:42.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.776+0000 7f94fe72b700 1 librados: init done 2024-02-13T00:37:42.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.777+0000 7f94fe72b700 1 -- 172.21.15.181:0/2876934122 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f805f400 con 0x7f94f80f7cd0 2024-02-13T00:37:42.784 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.778+0000 7f94f57fa700 1 -- 172.21.15.181:0/2876934122 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7f94e801c920 con 0x7f94f80f7cd0 2024-02-13T00:37:42.784 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.778+0000 7f94f57fa700 1 --2- 172.21.15.181:0/2876934122 >> v2:172.21.15.162:6800/1909610488 conn(0x7f94e0062ba0 0x7f94e0065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:42.784 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.778+0000 7f94f57fa700 1 -- 172.21.15.181:0/2876934122 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5111+0+0 (secure 0 0 0) 0x7f94e807de70 con 0x7f94f80f7cd0 2024-02-13T00:37:42.784 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.778+0000 7f94f57fa700 10 client.24352.objecter ms_dispatch 0x7f94f8001550 osd_map(58..58 src has 1..58) v4 2024-02-13T00:37:42.785 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.778+0000 7f94f57fa700 3 client.24352.objecter handle_osd_map got epochs [58,58] > 0 2024-02-13T00:37:42.785 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.778+0000 7f94f57fa700 3 client.24352.objecter handle_osd_map decoding full epoch 58 2024-02-13T00:37:42.785 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.778+0000 7f94f57fa700 20 client.24352.objecter dump_active .. 0 homeless 2024-02-13T00:37:42.785 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.778+0000 7f94f7fff700 1 --2- 172.21.15.181:0/2876934122 >> v2:172.21.15.162:6800/1909610488 conn(0x7f94e0062ba0 0x7f94e0065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:42.785 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.779+0000 7f94f57fa700 10 client.24352.objecter ms_handle_connect 0x7f94e0062ba0 2024-02-13T00:37:42.785 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.779+0000 7f94f7fff700 1 --2- 172.21.15.181:0/2876934122 >> v2:172.21.15.162:6800/1909610488 conn(0x7f94e0062ba0 0x7f94e0065060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f94f80f7b30 tx=0x7f94ec007b10).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:42.785 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.782+0000 7f94f57fa700 1 -- 172.21.15.181:0/2876934122 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f94e804e470 con 0x7f94f80f7cd0 2024-02-13T00:37:42.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:41.525461+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v194: 36 pgs: 11 creating+peering, 1 active+clean, 24 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:42.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:42.218028+0000 mon.a (mon.0) 563 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-13T00:37:42.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:42.222757+0000 mon.c (mon.2) 12 : audit [INF] from='client.? 172.21.15.162:0/3408777347' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T00:37:42.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:42.223215+0000 mon.a (mon.0) 564 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T00:37:42.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:42.226904+0000 mon.c (mon.2) 13 : audit [INF] from='client.? 172.21.15.162:0/396906212' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T00:37:42.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:42.227199+0000 mon.a (mon.0) 565 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T00:37:42.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:42.655613+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:42.948 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:42.945+0000 7f94fe72b700 1 -- 172.21.15.181:0/2876934122 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"} v 0) v1 -- 0x7f94f8101830 con 0x7f94f80f7cd0 2024-02-13T00:37:43.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:41.525461+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v194: 36 pgs: 11 creating+peering, 1 active+clean, 24 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:43.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:42.218028+0000 mon.a (mon.0) 563 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-13T00:37:43.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:42.222757+0000 mon.c (mon.2) 12 : audit [INF] from='client.? 172.21.15.162:0/3408777347' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T00:37:43.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:42.223215+0000 mon.a (mon.0) 564 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T00:37:43.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:42.226904+0000 mon.c (mon.2) 13 : audit [INF] from='client.? 172.21.15.162:0/396906212' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T00:37:43.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:42.227199+0000 mon.a (mon.0) 565 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T00:37:43.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:42.655613+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:43.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:41.525461+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v194: 36 pgs: 11 creating+peering, 1 active+clean, 24 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:43.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:42.218028+0000 mon.a (mon.0) 563 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-13T00:37:43.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:42.222757+0000 mon.c (mon.2) 12 : audit [INF] from='client.? 172.21.15.162:0/3408777347' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T00:37:43.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:42.223215+0000 mon.a (mon.0) 564 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T00:37:43.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:42.226904+0000 mon.c (mon.2) 13 : audit [INF] from='client.? 172.21.15.162:0/396906212' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T00:37:43.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:42.227199+0000 mon.a (mon.0) 565 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T00:37:43.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:42.655613+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:43.222 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.218+0000 7f94f57fa700 1 -- 172.21.15.181:0/2876934122 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]=0 enabled application 'rbd' on pool 'iscsi' v59) v1 ==== 147+0+0 (secure 0 0 0) 0x7f94e8051a90 con 0x7f94f80f7cd0 2024-02-13T00:37:43.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.221+0000 7f94fe72b700 10 librados: watch_flush enter 2024-02-13T00:37:43.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.222+0000 7f94fe72b700 10 librados: watch_flush exit 2024-02-13T00:37:43.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.222+0000 7f94fe72b700 20 client.24352.objecter shutdown clearing up homeless session... 2024-02-13T00:37:43.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.222+0000 7f94fe72b700 10 client.24352.objecter successfully canceled tick 2024-02-13T00:37:43.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.222+0000 7f94fe72b700 1 -- 172.21.15.181:0/2876934122 >> v2:172.21.15.162:6800/1909610488 conn(0x7f94e0062ba0 msgr2=0x7f94e0065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:43.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.222+0000 7f94fe72b700 1 --2- 172.21.15.181:0/2876934122 >> v2:172.21.15.162:6800/1909610488 conn(0x7f94e0062ba0 0x7f94e0065060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f94f80f7b30 tx=0x7f94ec007b10).stop 2024-02-13T00:37:43.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.222+0000 7f94fe72b700 1 -- 172.21.15.181:0/2876934122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f80f7cd0 msgr2=0x7f94f8108510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:43.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.222+0000 7f94fe72b700 1 --2- 172.21.15.181:0/2876934122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f80f7cd0 0x7f94f8108510 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f94e80045a0 tx=0x7f94e8009cb0).stop 2024-02-13T00:37:43.227 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.223+0000 7f94fe72b700 1 -- 172.21.15.181:0/2876934122 shutdown_connections 2024-02-13T00:37:43.227 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.223+0000 7f94fe72b700 1 --2- 172.21.15.181:0/2876934122 >> v2:172.21.15.162:6800/1909610488 conn(0x7f94e0062ba0 0x7f94e0065060 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:43.227 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.223+0000 7f94fe72b700 1 --2- 172.21.15.181:0/2876934122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94f80f8620 0x7f94f810f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:43.227 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.223+0000 7f94fe72b700 1 --2- 172.21.15.181:0/2876934122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94f80f6a30 0x7f94f8108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:43.227 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.223+0000 7f94fe72b700 1 --2- 172.21.15.181:0/2876934122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94f80f7cd0 0x7f94f8108510 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:43.227 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.223+0000 7f94fe72b700 1 -- 172.21.15.181:0/2876934122 >> 172.21.15.181:0/2876934122 conn(0x7f94f80f2230 msgr2=0x7f94f8060a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:43.228 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.223+0000 7f94fe72b700 1 -- 172.21.15.181:0/2876934122 shutdown_connections 2024-02-13T00:37:43.228 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.223+0000 7f94fe72b700 1 -- 172.21.15.181:0/2876934122 wait complete. 2024-02-13T00:37:43.228 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:43.223+0000 7f94fe72b700 1 librados: shutdown 2024-02-13T00:37:43.228 INFO:teuthology.orchestra.run.smithi181.stderr:enabled application 'rbd' on pool 'iscsi' 2024-02-13T00:37:43.861 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:42.946834+0000 mon.b (mon.1) 19 : audit [INF] from='client.? 172.21.15.181:0/2876934122' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-13T00:37:43.861 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:42.947523+0000 mon.a (mon.0) 567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-13T00:37:43.861 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:43.218978+0000 mon.a (mon.0) 568 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-13T00:37:43.861 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:43.219182+0000 mon.a (mon.0) 569 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-13T00:37:43.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:43.219284+0000 mon.a (mon.0) 570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-13T00:37:43.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:43.219359+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-13T00:37:43.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch apply iscsi iscsi user password --placement '1;smithi181=iscsi.a' 2024-02-13T00:37:44.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:42.946834+0000 mon.b (mon.1) 19 : audit [INF] from='client.? 172.21.15.181:0/2876934122' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-13T00:37:44.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:42.947523+0000 mon.a (mon.0) 567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-13T00:37:44.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:43.218978+0000 mon.a (mon.0) 568 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-13T00:37:44.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:43.219182+0000 mon.a (mon.0) 569 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-13T00:37:44.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:43.219284+0000 mon.a (mon.0) 570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-13T00:37:44.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:43.219359+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-13T00:37:44.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:42.946834+0000 mon.b (mon.1) 19 : audit [INF] from='client.? 172.21.15.181:0/2876934122' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-13T00:37:44.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:42.947523+0000 mon.a (mon.0) 567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-13T00:37:44.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:43.218978+0000 mon.a (mon.0) 568 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-13T00:37:44.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:43.219182+0000 mon.a (mon.0) 569 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-13T00:37:44.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:43.219284+0000 mon.a (mon.0) 570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-13T00:37:44.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:43.219359+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-13T00:37:45.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:43.525840+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v197: 68 pgs: 11 creating+peering, 5 active+clean, 52 unknown; 92 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:44.230097+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-13T00:37:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:44.233577+0000 mon.c (mon.2) 14 : audit [INF] from='client.? 172.21.15.162:0/396906212' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T00:37:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:44.233912+0000 mon.c (mon.2) 15 : audit [INF] from='client.? 172.21.15.162:0/3408777347' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T00:37:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:44.235350+0000 mon.a (mon.0) 573 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T00:37:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:44.235617+0000 mon.a (mon.0) 574 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T00:37:45.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:43.525840+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v197: 68 pgs: 11 creating+peering, 5 active+clean, 52 unknown; 92 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:45.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:44.230097+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-13T00:37:45.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:44.233577+0000 mon.c (mon.2) 14 : audit [INF] from='client.? 172.21.15.162:0/396906212' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T00:37:45.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:44.233912+0000 mon.c (mon.2) 15 : audit [INF] from='client.? 172.21.15.162:0/3408777347' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T00:37:45.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:44.235350+0000 mon.a (mon.0) 573 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T00:37:45.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:44.235617+0000 mon.a (mon.0) 574 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T00:37:45.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:43.525840+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v197: 68 pgs: 11 creating+peering, 5 active+clean, 52 unknown; 92 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:37:45.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:44.230097+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-13T00:37:45.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:44.233577+0000 mon.c (mon.2) 14 : audit [INF] from='client.? 172.21.15.162:0/396906212' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T00:37:45.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:44.233912+0000 mon.c (mon.2) 15 : audit [INF] from='client.? 172.21.15.162:0/3408777347' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T00:37:45.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:44.235350+0000 mon.a (mon.0) 573 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T00:37:45.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:44.235617+0000 mon.a (mon.0) 574 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T00:37:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.875+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3576668596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b40f8620 msgr2=0x7f76b40fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.875+0000 7f76bbe7e700 1 --2- 172.21.15.181:0/3576668596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b40f8620 0x7f76b40fef20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f76ac0048a0 tx=0x7f76ac009cb0).stop 2024-02-13T00:37:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.875+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3576668596 shutdown_connections 2024-02-13T00:37:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.875+0000 7f76bbe7e700 1 --2- 172.21.15.181:0/3576668596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b40f8620 0x7f76b40fef20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.875+0000 7f76bbe7e700 1 --2- 172.21.15.181:0/3576668596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76b40f6a30 0x7f76b40f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.875+0000 7f76bbe7e700 1 --2- 172.21.15.181:0/3576668596 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76b40f7cd0 0x7f76b40f8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.875+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3576668596 >> 172.21.15.181:0/3576668596 conn(0x7f76b40f2230 msgr2=0x7f76b40f4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.875+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3576668596 shutdown_connections 2024-02-13T00:37:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.875+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3576668596 wait complete. 2024-02-13T00:37:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76bbe7e700 1 librados: starting msgr at 2024-02-13T00:37:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76bbe7e700 1 librados: starting objecter 2024-02-13T00:37:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76bbe7e700 1 Processor -- start 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76bbe7e700 1 -- start start 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76bbe7e700 1 librados: setting wanted keys 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76bbe7e700 1 librados: calling monclient init 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76bbe7e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76b40f6a30 0x7f76b4061550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76bbe7e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b40f7cd0 0x7f76b410ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76bbe7e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76b40f8620 0x7f76b410f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76bbe7e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b410f8f0 con 0x7f76b40f8620 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76bbe7e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b410fa60 con 0x7f76b40f7cd0 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76bbe7e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b410fc20 con 0x7f76b40f6a30 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76b9419700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b40f7cd0 0x7f76b410ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76b9419700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b40f7cd0 0x7f76b410ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:33412/0 (socket says 172.21.15.181:33412) 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76b9419700 1 -- 172.21.15.181:0/3288564758 learned_addr learned my addr 172.21.15.181:0/3288564758 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76ba41b700 1 --2- 172.21.15.181:0/3288564758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76b40f8620 0x7f76b410f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:45.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.876+0000 7f76b9c1a700 1 --2- 172.21.15.181:0/3288564758 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76b40f6a30 0x7f76b4061550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76b9419700 1 -- 172.21.15.181:0/3288564758 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76b40f6a30 msgr2=0x7f76b4061550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76b9419700 1 --2- 172.21.15.181:0/3288564758 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76b40f6a30 0x7f76b4061550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76b9419700 1 -- 172.21.15.181:0/3288564758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76b40f8620 msgr2=0x7f76b410f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76b9419700 1 --2- 172.21.15.181:0/3288564758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76b40f8620 0x7f76b410f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76b9419700 1 -- 172.21.15.181:0/3288564758 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac005040 con 0x7f76b40f7cd0 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76b9419700 1 --2- 172.21.15.181:0/3288564758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b40f7cd0 0x7f76b410ccd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f76a80027f0 tx=0x7f76a8002b00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76b2ffd700 10 client.?.objecter ms_handle_connect 0x7f76b40f7cd0 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76b2ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76b2ffd700 1 -- 172.21.15.181:0/3288564758 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a8007b40 con 0x7f76b40f7cd0 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76b2ffd700 1 -- 172.21.15.181:0/3288564758 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76a800f040 con 0x7f76b40f7cd0 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3288564758 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b410ff00 con 0x7f76b40f7cd0 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76b2ffd700 1 -- 172.21.15.181:0/3288564758 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a80146f0 con 0x7f76b40f7cd0 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76bbe7e700 10 client.24370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:37:45.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3288564758 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b4110360 con 0x7f76b40f7cd0 2024-02-13T00:37:45.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.877+0000 7f76bbe7e700 1 librados: init done 2024-02-13T00:37:45.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.878+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3288564758 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b405f400 con 0x7f76b40f7cd0 2024-02-13T00:37:45.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.878+0000 7f76b2ffd700 1 -- 172.21.15.181:0/3288564758 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7f76a8009110 con 0x7f76b40f7cd0 2024-02-13T00:37:45.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.879+0000 7f76b2ffd700 1 --2- 172.21.15.181:0/3288564758 >> v2:172.21.15.162:6800/1909610488 conn(0x7f769c062ba0 0x7f769c065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:45.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.879+0000 7f76b2ffd700 1 -- 172.21.15.181:0/3288564758 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f76a807c420 con 0x7f76b40f7cd0 2024-02-13T00:37:45.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.879+0000 7f76b2ffd700 10 client.24370.objecter ms_dispatch 0x7f76b4001550 osd_map(61..61 src has 1..61) v4 2024-02-13T00:37:45.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.879+0000 7f76b2ffd700 3 client.24370.objecter handle_osd_map got epochs [61,61] > 0 2024-02-13T00:37:45.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.879+0000 7f76b2ffd700 3 client.24370.objecter handle_osd_map decoding full epoch 61 2024-02-13T00:37:45.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.879+0000 7f76b2ffd700 20 client.24370.objecter dump_active .. 0 homeless 2024-02-13T00:37:45.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.879+0000 7f76b9c1a700 1 --2- 172.21.15.181:0/3288564758 >> v2:172.21.15.162:6800/1909610488 conn(0x7f769c062ba0 0x7f769c065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:45.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.879+0000 7f76b2ffd700 10 client.24370.objecter ms_handle_connect 0x7f769c062ba0 2024-02-13T00:37:45.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.879+0000 7f76b9c1a700 1 --2- 172.21.15.181:0/3288564758 >> v2:172.21.15.162:6800/1909610488 conn(0x7f769c062ba0 0x7f769c065060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f76b40f7b30 tx=0x7f76a4009040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:45.886 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:45.883+0000 7f76b2ffd700 1 -- 172.21.15.181:0/3288564758 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f76a804c8a0 con 0x7f76b40f7cd0 2024-02-13T00:37:46.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:44.566874+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:46.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:44.566922+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:46.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:45.235081+0000 mon.a (mon.0) 575 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-13T00:37:46.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:45.235173+0000 mon.a (mon.0) 576 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-13T00:37:46.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:45.235237+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-13T00:37:46.056 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.053+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3288564758 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f76b40fdec0 con 0x7f769c062ba0 2024-02-13T00:37:46.065 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.062+0000 7f76b2ffd700 1 -- 172.21.15.181:0/3288564758 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f76b40fdec0 con 0x7f769c062ba0 2024-02-13T00:37:46.065 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled iscsi.iscsi update... 2024-02-13T00:37:46.067 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 10 librados: watch_flush enter 2024-02-13T00:37:46.067 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 10 librados: watch_flush exit 2024-02-13T00:37:46.067 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 20 client.24370.objecter shutdown clearing up homeless session... 2024-02-13T00:37:46.067 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 10 client.24370.objecter successfully canceled tick 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3288564758 >> v2:172.21.15.162:6800/1909610488 conn(0x7f769c062ba0 msgr2=0x7f769c065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 --2- 172.21.15.181:0/3288564758 >> v2:172.21.15.162:6800/1909610488 conn(0x7f769c062ba0 0x7f769c065060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f76b40f7b30 tx=0x7f76a4009040).stop 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3288564758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b40f7cd0 msgr2=0x7f76b410ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 --2- 172.21.15.181:0/3288564758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b40f7cd0 0x7f76b410ccd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f76a80027f0 tx=0x7f76a8002b00).stop 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3288564758 shutdown_connections 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 --2- 172.21.15.181:0/3288564758 >> v2:172.21.15.162:6800/1909610488 conn(0x7f769c062ba0 0x7f769c065060 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 --2- 172.21.15.181:0/3288564758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76b40f8620 0x7f76b410f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 --2- 172.21.15.181:0/3288564758 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76b40f6a30 0x7f76b4061550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 --2- 172.21.15.181:0/3288564758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b40f7cd0 0x7f76b410ccd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3288564758 >> 172.21.15.181:0/3288564758 conn(0x7f76b40f2230 msgr2=0x7f76b40f4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3288564758 shutdown_connections 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 -- 172.21.15.181:0/3288564758 wait complete. 2024-02-13T00:37:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:46.064+0000 7f76bbe7e700 1 librados: shutdown 2024-02-13T00:37:46.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:44.566874+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:46.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:44.566922+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:46.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:45.235081+0000 mon.a (mon.0) 575 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-13T00:37:46.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:45.235173+0000 mon.a (mon.0) 576 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-13T00:37:46.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:45.235237+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-13T00:37:46.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:44.566874+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:46.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:44.566922+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:46.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:45.235081+0000 mon.a (mon.0) 575 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-13T00:37:46.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:45.235173+0000 mon.a (mon.0) 576 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-13T00:37:46.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:45.235237+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-13T00:37:46.710 DEBUG:teuthology.orchestra.run.smithi181:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@iscsi.iscsi.a.service 2024-02-13T00:37:46.712 INFO:tasks.cephadm:Adding prometheus.a on smithi181 2024-02-13T00:37:46.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch apply prometheus '1;smithi181=a' 2024-02-13T00:37:46.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:-- Logs begin at Tue 2024-02-13 00:17:30 UTC. -- 2024-02-13T00:37:47.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:45.526254+0000 mgr.y (mgr.14146) 206 : cluster [DBG] pgmap v200: 100 pgs: 19 creating+peering, 47 active+clean, 34 unknown; 965 B data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 309 B/s wr, 1 op/s 2024-02-13T00:37:47.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:46.055695+0000 mgr.y (mgr.14146) 207 : audit [DBG] from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:47.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:46.057576+0000 mgr.y (mgr.14146) 208 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi181=iscsi.a;count:1 2024-02-13T00:37:47.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:46.063160+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:47.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:46.243630+0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-13T00:37:47.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:46.247682+0000 mon.c (mon.2) 16 : audit [INF] from='client.? 172.21.15.162:0/81002204' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T00:37:47.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:46.247886+0000 mon.c (mon.2) 17 : audit [INF] from='client.? 172.21.15.162:0/2326056861' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T00:37:47.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:46.249050+0000 mon.a (mon.0) 580 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T00:37:47.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:46.249341+0000 mon.a (mon.0) 581 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T00:37:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:45.526254+0000 mgr.y (mgr.14146) 206 : cluster [DBG] pgmap v200: 100 pgs: 19 creating+peering, 47 active+clean, 34 unknown; 965 B data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 309 B/s wr, 1 op/s 2024-02-13T00:37:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:46.055695+0000 mgr.y (mgr.14146) 207 : audit [DBG] from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:46.057576+0000 mgr.y (mgr.14146) 208 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi181=iscsi.a;count:1 2024-02-13T00:37:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:46.063160+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:47.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:46.243630+0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-13T00:37:47.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:46.247682+0000 mon.c (mon.2) 16 : audit [INF] from='client.? 172.21.15.162:0/81002204' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T00:37:47.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:46.247886+0000 mon.c (mon.2) 17 : audit [INF] from='client.? 172.21.15.162:0/2326056861' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T00:37:47.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:46.249050+0000 mon.a (mon.0) 580 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T00:37:47.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:46.249341+0000 mon.a (mon.0) 581 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T00:37:47.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:45.526254+0000 mgr.y (mgr.14146) 206 : cluster [DBG] pgmap v200: 100 pgs: 19 creating+peering, 47 active+clean, 34 unknown; 965 B data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 309 B/s wr, 1 op/s 2024-02-13T00:37:47.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:46.055695+0000 mgr.y (mgr.14146) 207 : audit [DBG] from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:47.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:46.057576+0000 mgr.y (mgr.14146) 208 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi181=iscsi.a;count:1 2024-02-13T00:37:47.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:46.063160+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:47.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:46.243630+0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-13T00:37:47.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:46.247682+0000 mon.c (mon.2) 16 : audit [INF] from='client.? 172.21.15.162:0/81002204' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T00:37:47.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:46.247886+0000 mon.c (mon.2) 17 : audit [INF] from='client.? 172.21.15.162:0/2326056861' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T00:37:47.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:46.249050+0000 mon.a (mon.0) 580 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T00:37:47.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:46.249341+0000 mon.a (mon.0) 581 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T00:37:48.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:47.245429+0000 mon.a (mon.0) 582 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-13T00:37:48.323 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:47.245533+0000 mon.a (mon.0) 583 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-13T00:37:48.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:47.245635+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-13T00:37:48.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:47.249324+0000 mon.c (mon.2) 18 : audit [INF] from='client.? 172.21.15.162:0/2326056861' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T00:37:48.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:47.252199+0000 mon.a (mon.0) 585 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T00:37:48.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:47.253393+0000 mon.c (mon.2) 19 : audit [INF] from='client.? 172.21.15.162:0/81002204' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T00:37:48.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:47.255195+0000 mon.a (mon.0) 586 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T00:37:48.324 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:47.526812+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v203: 132 pgs: 22 creating+peering, 84 active+clean, 26 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-13T00:37:48.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.565+0000 7f80cd7b5700 1 -- 172.21.15.181:0/2940952908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80c80f6eb0 msgr2=0x7f80c80f72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:48.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.565+0000 7f80cd7b5700 1 --2- 172.21.15.181:0/2940952908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80c80f6eb0 0x7f80c80f72d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f80b8004870 tx=0x7f80b8009cb0).stop 2024-02-13T00:37:48.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.565+0000 7f80cd7b5700 1 -- 172.21.15.181:0/2940952908 shutdown_connections 2024-02-13T00:37:48.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.565+0000 7f80cd7b5700 1 --2- 172.21.15.181:0/2940952908 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80c80f6570 0x7f80c80fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:48.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.565+0000 7f80cd7b5700 1 --2- 172.21.15.181:0/2940952908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80c80f6eb0 0x7f80c80f72d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:48.569 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.565+0000 7f80cd7b5700 1 --2- 172.21.15.181:0/2940952908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80c80f5c00 0x7f80c80f6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:48.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.565+0000 7f80cd7b5700 1 -- 172.21.15.181:0/2940952908 >> 172.21.15.181:0/2940952908 conn(0x7f80c80f19a0 msgr2=0x7f80c80f3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:48.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.565+0000 7f80cd7b5700 1 -- 172.21.15.181:0/2940952908 shutdown_connections 2024-02-13T00:37:48.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.565+0000 7f80cd7b5700 1 -- 172.21.15.181:0/2940952908 wait complete. 2024-02-13T00:37:48.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80cd7b5700 1 librados: starting msgr at 2024-02-13T00:37:48.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80cd7b5700 1 librados: starting objecter 2024-02-13T00:37:48.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80cd7b5700 1 Processor -- start 2024-02-13T00:37:48.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80cd7b5700 1 -- start start 2024-02-13T00:37:48.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80cd7b5700 1 librados: setting wanted keys 2024-02-13T00:37:48.570 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80cd7b5700 1 librados: calling monclient init 2024-02-13T00:37:48.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80cd7b5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80c80f5c00 0x7f80c810e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:48.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80cd7b5700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80c80f6570 0x7f80c810ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:48.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80cd7b5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80c80f6eb0 0x7f80c8115d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:48.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80cd7b5700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c8116280 con 0x7f80c80f5c00 2024-02-13T00:37:48.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80cd7b5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c81163c0 con 0x7f80c80f6eb0 2024-02-13T00:37:48.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80cd7b5700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c8116500 con 0x7f80c80f6570 2024-02-13T00:37:48.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80c77fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80c80f6eb0 0x7f80c8115d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:48.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80c77fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80c80f6eb0 0x7f80c8115d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:33430/0 (socket says 172.21.15.181:33430) 2024-02-13T00:37:48.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80c77fe700 1 -- 172.21.15.181:0/688317170 learned_addr learned my addr 172.21.15.181:0/688317170 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:37:48.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.566+0000 7f80c6ffd700 1 --2- 172.21.15.181:0/688317170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80c80f5c00 0x7f80c810e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:48.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80c67fc700 1 --2- 172.21.15.181:0/688317170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80c80f6570 0x7f80c810ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80c77fe700 1 -- 172.21.15.181:0/688317170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80c80f6570 msgr2=0x7f80c810ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80c77fe700 1 --2- 172.21.15.181:0/688317170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80c80f6570 0x7f80c810ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80c77fe700 1 -- 172.21.15.181:0/688317170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80c80f5c00 msgr2=0x7f80c810e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80c77fe700 1 --2- 172.21.15.181:0/688317170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80c80f5c00 0x7f80c810e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80c77fe700 1 -- 172.21.15.181:0/688317170 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b8005040 con 0x7f80c80f6eb0 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80c77fe700 1 --2- 172.21.15.181:0/688317170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80c80f6eb0 0x7f80c8115d90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f80c000ec20 tx=0x7f80c000b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80affff700 10 client.?.objecter ms_handle_connect 0x7f80c80f6eb0 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80affff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80affff700 1 -- 172.21.15.181:0/688317170 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c0009950 con 0x7f80c80f6eb0 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80cd7b5700 1 -- 172.21.15.181:0/688317170 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c8116780 con 0x7f80c80f6eb0 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80affff700 1 -- 172.21.15.181:0/688317170 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80c000f040 con 0x7f80c80f6eb0 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80affff700 1 -- 172.21.15.181:0/688317170 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c0007df0 con 0x7f80c80f6eb0 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80cd7b5700 10 client.24391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80cd7b5700 1 -- 172.21.15.181:0/688317170 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c8116cf0 con 0x7f80c80f6eb0 2024-02-13T00:37:48.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.567+0000 7f80cd7b5700 1 librados: init done 2024-02-13T00:37:48.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.568+0000 7f80cd7b5700 1 -- 172.21.15.181:0/688317170 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c8109030 con 0x7f80c80f6eb0 2024-02-13T00:37:48.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.569+0000 7f80affff700 1 -- 172.21.15.181:0/688317170 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 14) v1 ==== 77030+0+0 (secure 0 0 0) 0x7f80c00092e0 con 0x7f80c80f6eb0 2024-02-13T00:37:48.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.569+0000 7f80affff700 1 --2- 172.21.15.181:0/688317170 >> v2:172.21.15.162:6800/1909610488 conn(0x7f80b0062ba0 0x7f80b0065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:48.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.569+0000 7f80affff700 1 -- 172.21.15.181:0/688317170 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5884+0+0 (secure 0 0 0) 0x7f80c007c3f0 con 0x7f80c80f6eb0 2024-02-13T00:37:48.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.569+0000 7f80affff700 10 client.24391.objecter ms_dispatch 0x7f80c8001550 osd_map(64..64 src has 1..64) v4 2024-02-13T00:37:48.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.569+0000 7f80affff700 3 client.24391.objecter handle_osd_map got epochs [64,64] > 0 2024-02-13T00:37:48.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.569+0000 7f80affff700 3 client.24391.objecter handle_osd_map decoding full epoch 64 2024-02-13T00:37:48.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.569+0000 7f80affff700 20 client.24391.objecter dump_active .. 0 homeless 2024-02-13T00:37:48.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.569+0000 7f80c6ffd700 1 --2- 172.21.15.181:0/688317170 >> v2:172.21.15.162:6800/1909610488 conn(0x7f80b0062ba0 0x7f80b0065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:48.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.570+0000 7f80affff700 10 client.24391.objecter ms_handle_connect 0x7f80b0062ba0 2024-02-13T00:37:48.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.570+0000 7f80c6ffd700 1 --2- 172.21.15.181:0/688317170 >> v2:172.21.15.162:6800/1909610488 conn(0x7f80b0062ba0 0x7f80b0065060 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f80b8009da0 tx=0x7f80b800b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:48.577 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.574+0000 7f80affff700 1 -- 172.21.15.181:0/688317170 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f80c004c740 con 0x7f80c80f6eb0 2024-02-13T00:37:48.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:47.245429+0000 mon.a (mon.0) 582 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-13T00:37:48.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:47.245533+0000 mon.a (mon.0) 583 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-13T00:37:48.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:47.245635+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-13T00:37:48.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:47.249324+0000 mon.c (mon.2) 18 : audit [INF] from='client.? 172.21.15.162:0/2326056861' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T00:37:48.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:47.252199+0000 mon.a (mon.0) 585 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T00:37:48.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:47.253393+0000 mon.c (mon.2) 19 : audit [INF] from='client.? 172.21.15.162:0/81002204' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T00:37:48.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:47.255195+0000 mon.a (mon.0) 586 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T00:37:48.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:47.526812+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v203: 132 pgs: 22 creating+peering, 84 active+clean, 26 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-13T00:37:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:47.245429+0000 mon.a (mon.0) 582 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-13T00:37:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:47.245533+0000 mon.a (mon.0) 583 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-13T00:37:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:47.245635+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-13T00:37:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:47.249324+0000 mon.c (mon.2) 18 : audit [INF] from='client.? 172.21.15.162:0/2326056861' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T00:37:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:47.252199+0000 mon.a (mon.0) 585 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T00:37:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:47.253393+0000 mon.c (mon.2) 19 : audit [INF] from='client.? 172.21.15.162:0/81002204' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T00:37:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:47.255195+0000 mon.a (mon.0) 586 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T00:37:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:47.526812+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v203: 132 pgs: 22 creating+peering, 84 active+clean, 26 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-13T00:37:48.779 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.776+0000 7f80cd7b5700 1 -- 172.21.15.181:0/688317170 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f80c80f3cc0 con 0x7f80b0062ba0 2024-02-13T00:37:48.785 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.782+0000 7f80affff700 1 -- 172.21.15.181:0/688317170 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f80c80f3cc0 con 0x7f80b0062ba0 2024-02-13T00:37:48.785 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled prometheus update... 2024-02-13T00:37:48.787 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 10 librados: watch_flush enter 2024-02-13T00:37:48.787 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 10 librados: watch_flush exit 2024-02-13T00:37:48.787 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 20 client.24391.objecter shutdown clearing up homeless session... 2024-02-13T00:37:48.787 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 10 client.24391.objecter successfully canceled tick 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 -- 172.21.15.181:0/688317170 >> v2:172.21.15.162:6800/1909610488 conn(0x7f80b0062ba0 msgr2=0x7f80b0065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 --2- 172.21.15.181:0/688317170 >> v2:172.21.15.162:6800/1909610488 conn(0x7f80b0062ba0 0x7f80b0065060 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f80b8009da0 tx=0x7f80b800b040).stop 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 -- 172.21.15.181:0/688317170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80c80f6eb0 msgr2=0x7f80c8115d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 --2- 172.21.15.181:0/688317170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80c80f6eb0 0x7f80c8115d90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f80c000ec20 tx=0x7f80c000b040).stop 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 -- 172.21.15.181:0/688317170 shutdown_connections 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 --2- 172.21.15.181:0/688317170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80c80f6eb0 0x7f80c8115d90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 --2- 172.21.15.181:0/688317170 >> v2:172.21.15.162:6800/1909610488 conn(0x7f80b0062ba0 0x7f80b0065060 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 --2- 172.21.15.181:0/688317170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80c80f5c00 0x7f80c810e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 --2- 172.21.15.181:0/688317170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80c80f6570 0x7f80c810ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 -- 172.21.15.181:0/688317170 >> 172.21.15.181:0/688317170 conn(0x7f80c80f19a0 msgr2=0x7f80c80f25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 -- 172.21.15.181:0/688317170 shutdown_connections 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 -- 172.21.15.181:0/688317170 wait complete. 2024-02-13T00:37:48.788 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:48.784+0000 7f80cd7b5700 1 librados: shutdown 2024-02-13T00:37:49.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:48.249373+0000 mon.a (mon.0) 587 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T00:37:49.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:48.249509+0000 mon.a (mon.0) 588 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T00:37:49.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:48.249614+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-13T00:37:49.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:48.425932+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:49.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:48.433407+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:49.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:48.439584+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:49.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:48.445826+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-13T00:37:49.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:48.454803+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T00:37:49.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:48.461657+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-13T00:37:49.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:48.471458+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:49.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:48.472366+0000 mgr.y (mgr.14146) 211 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi181 2024-02-13T00:37:49.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:48.778465+0000 mgr.y (mgr.14146) 212 : audit [DBG] from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:49.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:48.779893+0000 mgr.y (mgr.14146) 213 : cephadm [INF] Saving service prometheus spec with placement smithi181=a;count:1 2024-02-13T00:37:49.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:48.782908+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:49.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:48.249373+0000 mon.a (mon.0) 587 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T00:37:49.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:48.249509+0000 mon.a (mon.0) 588 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T00:37:49.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:48.249614+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-13T00:37:49.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:48.425932+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:48.433407+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:48.439584+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:48.445826+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-13T00:37:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:48.454803+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T00:37:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:48.461657+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-13T00:37:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:48.471458+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:48.472366+0000 mgr.y (mgr.14146) 211 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi181 2024-02-13T00:37:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:48.778465+0000 mgr.y (mgr.14146) 212 : audit [DBG] from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:48.779893+0000 mgr.y (mgr.14146) 213 : cephadm [INF] Saving service prometheus spec with placement smithi181=a;count:1 2024-02-13T00:37:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:48.782908+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:49.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:48.249373+0000 mon.a (mon.0) 587 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T00:37:49.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:48.249509+0000 mon.a (mon.0) 588 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T00:37:49.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:48.249614+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-13T00:37:49.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:48.425932+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:49.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:48.433407+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:49.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:48.439584+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:49.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:48.445826+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-13T00:37:49.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:48.454803+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-13T00:37:49.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:48.461657+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-13T00:37:49.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:48.471458+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:49.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:48.472366+0000 mgr.y (mgr.14146) 211 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi181 2024-02-13T00:37:49.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:48.778465+0000 mgr.y (mgr.14146) 212 : audit [DBG] from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:49.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:48.779893+0000 mgr.y (mgr.14146) 213 : cephadm [INF] Saving service prometheus spec with placement smithi181=a;count:1 2024-02-13T00:37:49.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:48.782908+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:49.911 DEBUG:teuthology.orchestra.run.smithi181:prometheus.a> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@prometheus.a.service 2024-02-13T00:37:49.914 INFO:tasks.cephadm:Adding node-exporter.a on smithi162 2024-02-13T00:37:49.914 INFO:tasks.cephadm:Adding node-exporter.b on smithi181 2024-02-13T00:37:49.914 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch apply node-exporter '2;smithi162=a;smithi181=b' 2024-02-13T00:37:49.945 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:-- Logs begin at Tue 2024-02-13 00:17:30 UTC. -- 2024-02-13T00:37:50.618 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:37:50 smithi181 systemd[1]: Starting Ceph iscsi.iscsi.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:37:50.971 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:49.527413+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v205: 132 pgs: 14 creating+peering, 106 active+clean, 12 unknown; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 1.9 KiB/s wr, 61 op/s 2024-02-13T00:37:50.971 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:49.790251+0000 mon.a (mon.0) 597 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-02-13T00:37:51.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:49.527413+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v205: 132 pgs: 14 creating+peering, 106 active+clean, 12 unknown; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 1.9 KiB/s wr, 61 op/s 2024-02-13T00:37:51.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:49.790251+0000 mon.a (mon.0) 597 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-02-13T00:37:51.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:49.527413+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v205: 132 pgs: 14 creating+peering, 106 active+clean, 12 unknown; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 1.9 KiB/s wr, 61 op/s 2024-02-13T00:37:51.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:49.790251+0000 mon.a (mon.0) 597 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-02-13T00:37:52.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:37:52 smithi181 bash[62193]: log file path now is '/var/log/tcmu-runner.log' 2024-02-13T00:37:52.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.485+0000 7f2e13391700 1 -- 172.21.15.181:0/2426784555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e0c0f6a00 msgr2=0x7f2e0c0f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:52.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.485+0000 7f2e13391700 1 --2- 172.21.15.181:0/2426784555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e0c0f6a00 0x7f2e0c0f6e20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f2e00004870 tx=0x7f2e00009cb0).stop 2024-02-13T00:37:52.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.485+0000 7f2e13391700 1 -- 172.21.15.181:0/2426784555 shutdown_connections 2024-02-13T00:37:52.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.485+0000 7f2e13391700 1 --2- 172.21.15.181:0/2426784555 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2e0c0f8420 0x7f2e0c0fef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:52.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.485+0000 7f2e13391700 1 --2- 172.21.15.181:0/2426784555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e0c0f6a00 0x7f2e0c0f6e20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:52.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.485+0000 7f2e13391700 1 --2- 172.21.15.181:0/2426784555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0c0f7310 0x7f2e0c0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:52.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.485+0000 7f2e13391700 1 -- 172.21.15.181:0/2426784555 >> 172.21.15.181:0/2426784555 conn(0x7f2e0c0f2220 msgr2=0x7f2e0c0f4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:52.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.485+0000 7f2e13391700 1 -- 172.21.15.181:0/2426784555 shutdown_connections 2024-02-13T00:37:52.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.485+0000 7f2e13391700 1 -- 172.21.15.181:0/2426784555 wait complete. 2024-02-13T00:37:52.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e13391700 1 librados: starting msgr at 2024-02-13T00:37:52.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e13391700 1 librados: starting objecter 2024-02-13T00:37:52.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e13391700 1 Processor -- start 2024-02-13T00:37:52.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e13391700 1 -- start start 2024-02-13T00:37:52.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e13391700 1 librados: setting wanted keys 2024-02-13T00:37:52.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e13391700 1 librados: calling monclient init 2024-02-13T00:37:52.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e13391700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0c0f6a00 0x7f2e0c061b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:52.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e13391700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e0c0f7310 0x7f2e0c10d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:52.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e13391700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2e0c0f8420 0x7f2e0c10f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:52.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e13391700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c10fd80 con 0x7f2e0c0f6a00 2024-02-13T00:37:52.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e13391700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c10fef0 con 0x7f2e0c0f7310 2024-02-13T00:37:52.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e13391700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c1100b0 con 0x7f2e0c0f8420 2024-02-13T00:37:52.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e1092c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e0c0f7310 0x7f2e0c10d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:52.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e1092c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e0c0f7310 0x7f2e0c10d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44748/0 (socket says 172.21.15.181:44748) 2024-02-13T00:37:52.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e1112d700 1 --2- 172.21.15.181:0/2672812920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0c0f6a00 0x7f2e0c061b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:52.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e1092c700 1 -- 172.21.15.181:0/2672812920 learned_addr learned my addr 172.21.15.181:0/2672812920 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:37:52.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.486+0000 7f2e1192e700 1 --2- 172.21.15.181:0/2672812920 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2e0c0f8420 0x7f2e0c10f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:52.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2e1092c700 1 -- 172.21.15.181:0/2672812920 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2e0c0f8420 msgr2=0x7f2e0c10f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:52.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2e1092c700 1 --2- 172.21.15.181:0/2672812920 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2e0c0f8420 0x7f2e0c10f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:52.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2e1092c700 1 -- 172.21.15.181:0/2672812920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0c0f6a00 msgr2=0x7f2e0c061b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:52.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2e1092c700 1 --2- 172.21.15.181:0/2672812920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0c0f6a00 0x7f2e0c061b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2e1092c700 1 -- 172.21.15.181:0/2672812920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e00005040 con 0x7f2e0c0f7310 2024-02-13T00:37:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2e1112d700 1 --2- 172.21.15.181:0/2672812920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0c0f6a00 0x7f2e0c061b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:37:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2e1092c700 1 --2- 172.21.15.181:0/2672812920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e0c0f7310 0x7f2e0c10d180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f2e080041d0 tx=0x7f2e08009f60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2dfe7fc700 10 client.?.objecter ms_handle_connect 0x7f2e0c0f7310 2024-02-13T00:37:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2dfe7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:37:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2dfe7fc700 1 -- 172.21.15.181:0/2672812920 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08007b40 con 0x7f2e0c0f7310 2024-02-13T00:37:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2dfe7fc700 1 -- 172.21.15.181:0/2672812920 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e0800f040 con 0x7f2e0c0f7310 2024-02-13T00:37:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2e13391700 1 -- 172.21.15.181:0/2672812920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0c110390 con 0x7f2e0c0f7310 2024-02-13T00:37:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2dfe7fc700 1 -- 172.21.15.181:0/2672812920 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08014810 con 0x7f2e0c0f7310 2024-02-13T00:37:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2e13391700 10 client.24397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:37:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2e13391700 1 -- 172.21.15.181:0/2672812920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0c110950 con 0x7f2e0c0f7310 2024-02-13T00:37:52.494 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.487+0000 7f2e13391700 1 librados: init done 2024-02-13T00:37:52.495 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.488+0000 7f2e13391700 1 -- 172.21.15.181:0/2672812920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0c060070 con 0x7f2e0c0f7310 2024-02-13T00:37:52.495 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.488+0000 7f2dfe7fc700 1 -- 172.21.15.181:0/2672812920 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 15) v1 ==== 77081+0+0 (secure 0 0 0) 0x7f2e08009110 con 0x7f2e0c0f7310 2024-02-13T00:37:52.495 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.489+0000 7f2dfe7fc700 1 --2- 172.21.15.181:0/2672812920 >> v2:172.21.15.162:6800/1909610488 conn(0x7f2df8062c20 0x7f2df80650e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:52.496 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.489+0000 7f2dfe7fc700 1 -- 172.21.15.181:0/2672812920 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5884+0+0 (secure 0 0 0) 0x7f2e0807c8c0 con 0x7f2e0c0f7310 2024-02-13T00:37:52.496 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.489+0000 7f2dfe7fc700 10 client.24397.objecter ms_dispatch 0x7f2e0c001550 osd_map(64..64 src has 1..64) v4 2024-02-13T00:37:52.496 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.489+0000 7f2dfe7fc700 3 client.24397.objecter handle_osd_map got epochs [64,64] > 0 2024-02-13T00:37:52.496 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.489+0000 7f2dfe7fc700 3 client.24397.objecter handle_osd_map decoding full epoch 64 2024-02-13T00:37:52.496 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.489+0000 7f2dfe7fc700 20 client.24397.objecter dump_active .. 0 homeless 2024-02-13T00:37:52.496 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.489+0000 7f2e1112d700 1 --2- 172.21.15.181:0/2672812920 >> v2:172.21.15.162:6800/1909610488 conn(0x7f2df8062c20 0x7f2df80650e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:52.496 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.489+0000 7f2dfe7fc700 10 client.24397.objecter ms_handle_connect 0x7f2df8062c20 2024-02-13T00:37:52.496 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.489+0000 7f2e1112d700 1 --2- 172.21.15.181:0/2672812920 >> v2:172.21.15.162:6800/1909610488 conn(0x7f2df8062c20 0x7f2df80650e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2e00004560 tx=0x7f2e00009cb0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:52.497 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.493+0000 7f2dfe7fc700 1 -- 172.21.15.181:0/2672812920 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2e0804cbd0 con 0x7f2e0c0f7310 2024-02-13T00:37:52.677 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.674+0000 7f2e13391700 1 -- 172.21.15.181:0/2672812920 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi162=a;smithi181=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e0c0008d0 con 0x7f2df8062c20 2024-02-13T00:37:52.684 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.681+0000 7f2dfe7fc700 1 -- 172.21.15.181:0/2672812920 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f2e0c0008d0 con 0x7f2df8062c20 2024-02-13T00:37:52.685 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled node-exporter update... 2024-02-13T00:37:52.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 10 librados: watch_flush enter 2024-02-13T00:37:52.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 10 librados: watch_flush exit 2024-02-13T00:37:52.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 20 client.24397.objecter shutdown clearing up homeless session... 2024-02-13T00:37:52.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 10 client.24397.objecter successfully canceled tick 2024-02-13T00:37:52.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 -- 172.21.15.181:0/2672812920 >> v2:172.21.15.162:6800/1909610488 conn(0x7f2df8062c20 msgr2=0x7f2df80650e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:52.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 --2- 172.21.15.181:0/2672812920 >> v2:172.21.15.162:6800/1909610488 conn(0x7f2df8062c20 0x7f2df80650e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2e00004560 tx=0x7f2e00009cb0).stop 2024-02-13T00:37:52.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 -- 172.21.15.181:0/2672812920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e0c0f7310 msgr2=0x7f2e0c10d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:52.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 --2- 172.21.15.181:0/2672812920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e0c0f7310 0x7f2e0c10d180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f2e080041d0 tx=0x7f2e08009f60).stop 2024-02-13T00:37:52.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 -- 172.21.15.181:0/2672812920 shutdown_connections 2024-02-13T00:37:52.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 --2- 172.21.15.181:0/2672812920 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2e0c0f8420 0x7f2e0c10f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:52.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 --2- 172.21.15.181:0/2672812920 >> v2:172.21.15.162:6800/1909610488 conn(0x7f2df8062c20 0x7f2df80650e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:52.687 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 --2- 172.21.15.181:0/2672812920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e0c0f6a00 0x7f2e0c061b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:52.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 --2- 172.21.15.181:0/2672812920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e0c0f7310 0x7f2e0c10d180 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:52.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 -- 172.21.15.181:0/2672812920 >> 172.21.15.181:0/2672812920 conn(0x7f2e0c0f2220 msgr2=0x7f2e0c0f4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:52.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 -- 172.21.15.181:0/2672812920 shutdown_connections 2024-02-13T00:37:52.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 -- 172.21.15.181:0/2672812920 wait complete. 2024-02-13T00:37:52.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:52.683+0000 7f2e13391700 1 librados: shutdown 2024-02-13T00:37:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:51.528137+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v206: 132 pgs: 10 creating+peering, 122 active+clean; 3.9 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 3.8 KiB/s wr, 150 op/s 2024-02-13T00:37:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:52.126599+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:52.126664+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:53.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:52.682166+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:53.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:51.528137+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v206: 132 pgs: 10 creating+peering, 122 active+clean; 3.9 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 3.8 KiB/s wr, 150 op/s 2024-02-13T00:37:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:52.126599+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:52.126664+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:52.682166+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:51.528137+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v206: 132 pgs: 10 creating+peering, 122 active+clean; 3.9 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 3.8 KiB/s wr, 150 op/s 2024-02-13T00:37:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:52.126599+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:52.126664+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:52.682166+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:53.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:37:53 smithi181 bash[62387]: 8aa5b3fda83751def9ba1ff3ab51d28b915bdced4ec7f1dd52ee6e297260b2e3 2024-02-13T00:37:53.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:37:53 smithi181 systemd[1]: Started Ceph iscsi.iscsi.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:37:53.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:37:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug Started the configuration object watcher 2024-02-13T00:37:53.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:37:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug Checking for config object changes every 1s 2024-02-13T00:37:53.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:37:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug Processing osd blocklist entries for this node 2024-02-13T00:37:53.695 DEBUG:teuthology.orchestra.run.smithi162:node-exporter.a> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@node-exporter.a.service 2024-02-13T00:37:53.700 DEBUG:teuthology.orchestra.run.smithi181:node-exporter.b> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@node-exporter.b.service 2024-02-13T00:37:53.703 INFO:tasks.cephadm:Adding alertmanager.a on smithi162 2024-02-13T00:37:53.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch apply alertmanager '1;smithi162=a' 2024-02-13T00:37:53.733 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:-- Logs begin at Tue 2024-02-13 00:17:30 UTC. -- 2024-02-13T00:37:53.734 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:-- Logs begin at Tue 2024-02-13 00:17:45 UTC. -- 2024-02-13T00:37:54.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:37:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug Reading the configuration object to update local LIO configuration 2024-02-13T00:37:54.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:37:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug Configuration does not have an entry for this host(smithi181.front.sepia.ceph.com) - nothing to define to LIO 2024-02-13T00:37:54.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:37:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-02-13T00:37:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:52.676778+0000 mgr.y (mgr.14146) 216 : audit [DBG] from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi162=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:52.678201+0000 mgr.y (mgr.14146) 217 : cephadm [INF] Saving service node-exporter spec with placement smithi162=a;smithi181=b;count:2 2024-02-13T00:37:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:52.794543+0000 mon.a (mon.0) 599 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-13T00:37:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:53.205574+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:53.206993+0000 mon.a (mon.0) 601 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:53.624968+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? 172.21.15.181:0/571452612' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T00:37:54.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:52.676778+0000 mgr.y (mgr.14146) 216 : audit [DBG] from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi162=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:54.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:52.678201+0000 mgr.y (mgr.14146) 217 : cephadm [INF] Saving service node-exporter spec with placement smithi162=a;smithi181=b;count:2 2024-02-13T00:37:54.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:52.794543+0000 mon.a (mon.0) 599 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-13T00:37:54.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:53.205574+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:54.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:53.206993+0000 mon.a (mon.0) 601 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:54.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:53.624968+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? 172.21.15.181:0/571452612' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T00:37:54.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:52.676778+0000 mgr.y (mgr.14146) 216 : audit [DBG] from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi162=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:54.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:52.678201+0000 mgr.y (mgr.14146) 217 : cephadm [INF] Saving service node-exporter spec with placement smithi162=a;smithi181=b;count:2 2024-02-13T00:37:54.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:52.794543+0000 mon.a (mon.0) 599 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-13T00:37:54.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:53.205574+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:54.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:53.206993+0000 mon.a (mon.0) 601 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:37:54.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:53.624968+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? 172.21.15.181:0/571452612' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-13T00:37:55.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:53.397136+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:55.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:53.397182+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:55.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:53.528507+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v208: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 5.6 KiB/s wr, 240 op/s 2024-02-13T00:37:55.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:54.218206+0000 mon.a (mon.0) 602 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-13T00:37:55.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:53.397136+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:55.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:53.397182+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:55.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:53.528507+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v208: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 5.6 KiB/s wr, 240 op/s 2024-02-13T00:37:55.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:54.218206+0000 mon.a (mon.0) 602 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-13T00:37:55.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:53.397136+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:37:55.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:53.397182+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:37:55.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:53.528507+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v208: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 5.6 KiB/s wr, 240 op/s 2024-02-13T00:37:55.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:54.218206+0000 mon.a (mon.0) 602 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-13T00:37:56.477 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.447+0000 7f00793d6700 1 -- 172.21.15.181:0/3961571028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00740f6270 msgr2=0x7f00740f66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:56.477 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.447+0000 7f00793d6700 1 --2- 172.21.15.181:0/3961571028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00740f6270 0x7f00740f66f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f006c009be0 tx=0x7f006c004d10).stop 2024-02-13T00:37:56.477 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.447+0000 7f00793d6700 1 -- 172.21.15.181:0/3961571028 shutdown_connections 2024-02-13T00:37:56.477 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.447+0000 7f00793d6700 1 --2- 172.21.15.181:0/3961571028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00740f6270 0x7f00740f66f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:56.477 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.447+0000 7f00793d6700 1 --2- 172.21.15.181:0/3961571028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00740f9180 0x7f00740f95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:56.477 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.447+0000 7f00793d6700 1 --2- 172.21.15.181:0/3961571028 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00740f9a90 0x7f00740f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:56.477 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.447+0000 7f00793d6700 1 -- 172.21.15.181:0/3961571028 >> 172.21.15.181:0/3961571028 conn(0x7f00740f19c0 msgr2=0x7f00740f3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:56.477 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.447+0000 7f00793d6700 1 -- 172.21.15.181:0/3961571028 shutdown_connections 2024-02-13T00:37:56.477 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.447+0000 7f00793d6700 1 -- 172.21.15.181:0/3961571028 wait complete. 2024-02-13T00:37:56.477 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.447+0000 7f00793d6700 1 librados: starting msgr at 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.447+0000 7f00793d6700 1 librados: starting objecter 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00793d6700 1 Processor -- start 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00793d6700 1 -- start start 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00793d6700 1 librados: setting wanted keys 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00793d6700 1 librados: calling monclient init 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00793d6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00740f6270 0x7f0074107e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00793d6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00740f9180 0x7f0074108380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00793d6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00740f9a90 0x7f007410f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00793d6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007410f950 con 0x7f00740f6270 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00793d6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007410fa90 con 0x7f00740f9180 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00793d6700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007410fbd0 con 0x7f00740f9a90 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00727fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00740f9180 0x7f0074108380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00727fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00740f9180 0x7f0074108380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:44824/0 (socket says 172.21.15.181:44824) 2024-02-13T00:37:56.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00727fc700 1 -- 172.21.15.181:0/1500641082 learned_addr learned my addr 172.21.15.181:0/1500641082 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f0072ffd700 1 --2- 172.21.15.181:0/1500641082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00740f6270 0x7f0074107e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00737fe700 1 --2- 172.21.15.181:0/1500641082 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00740f9a90 0x7f007410f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00727fc700 1 -- 172.21.15.181:0/1500641082 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00740f9a90 msgr2=0x7f007410f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00727fc700 1 --2- 172.21.15.181:0/1500641082 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00740f9a90 0x7f007410f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00727fc700 1 -- 172.21.15.181:0/1500641082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00740f6270 msgr2=0x7f0074107e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00727fc700 1 --2- 172.21.15.181:0/1500641082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00740f6270 0x7f0074107e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00727fc700 1 -- 172.21.15.181:0/1500641082 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c0097c0 con 0x7f00740f9180 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f0072ffd700 1 --2- 172.21.15.181:0/1500641082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00740f6270 0x7f0074107e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f00727fc700 1 --2- 172.21.15.181:0/1500641082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00740f9180 0x7f0074108380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f006400c810 tx=0x7f006400cbd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f005bfff700 10 client.?.objecter ms_handle_connect 0x7f00740f9180 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.448+0000 7f005bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.449+0000 7f005bfff700 1 -- 172.21.15.181:0/1500641082 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064010510 con 0x7f00740f9180 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.449+0000 7f005bfff700 1 -- 172.21.15.181:0/1500641082 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0064010b40 con 0x7f00740f9180 2024-02-13T00:37:56.479 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.449+0000 7f00793d6700 1 -- 172.21.15.181:0/1500641082 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007410feb0 con 0x7f00740f9180 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.449+0000 7f005bfff700 1 -- 172.21.15.181:0/1500641082 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064012b00 con 0x7f00740f9180 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.449+0000 7f00793d6700 10 client.24424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.449+0000 7f00793d6700 1 -- 172.21.15.181:0/1500641082 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0074110340 con 0x7f00740f9180 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.449+0000 7f00793d6700 1 librados: init done 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.449+0000 7f00793d6700 1 -- 172.21.15.181:0/1500641082 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0074102ab0 con 0x7f00740f9180 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.450+0000 7f005bfff700 1 -- 172.21.15.181:0/1500641082 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 15) v1 ==== 77081+0+0 (secure 0 0 0) 0x7f0064010cb0 con 0x7f00740f9180 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.450+0000 7f005bfff700 1 --2- 172.21.15.181:0/1500641082 >> v2:172.21.15.162:6800/1909610488 conn(0x7f005c062c20 0x7f005c0650e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.450+0000 7f005bfff700 1 -- 172.21.15.181:0/1500641082 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5884+0+0 (secure 0 0 0) 0x7f006407c930 con 0x7f00740f9180 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.450+0000 7f005bfff700 10 client.24424.objecter ms_dispatch 0x7f0074001550 osd_map(66..66 src has 1..66) v4 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.450+0000 7f005bfff700 3 client.24424.objecter handle_osd_map got epochs [66,66] > 0 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.450+0000 7f005bfff700 3 client.24424.objecter handle_osd_map decoding full epoch 66 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.450+0000 7f005bfff700 20 client.24424.objecter dump_active .. 0 homeless 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.450+0000 7f0072ffd700 1 --2- 172.21.15.181:0/1500641082 >> v2:172.21.15.162:6800/1909610488 conn(0x7f005c062c20 0x7f005c0650e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:56.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.451+0000 7f005bfff700 10 client.24424.objecter ms_handle_connect 0x7f005c062c20 2024-02-13T00:37:56.481 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.451+0000 7f0072ffd700 1 --2- 172.21.15.181:0/1500641082 >> v2:172.21.15.162:6800/1909610488 conn(0x7f005c062c20 0x7f005c0650e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f0068004b80 tx=0x7f0068009040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:56.481 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.454+0000 7f005bfff700 1 -- 172.21.15.181:0/1500641082 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f006404cb80 con 0x7f00740f9180 2024-02-13T00:37:56.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.637+0000 7f00793d6700 1 -- 172.21.15.181:0/1500641082 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f00740fea40 con 0x7f005c062c20 2024-02-13T00:37:56.649 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.645+0000 7f005bfff700 1 -- 172.21.15.181:0/1500641082 <== mgr.14146 v2:172.21.15.162:6800/1909610488 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f00740fea40 con 0x7f005c062c20 2024-02-13T00:37:56.649 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled alertmanager update... 2024-02-13T00:37:56.651 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 10 librados: watch_flush enter 2024-02-13T00:37:56.651 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 10 librados: watch_flush exit 2024-02-13T00:37:56.651 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 20 client.24424.objecter shutdown clearing up homeless session... 2024-02-13T00:37:56.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 10 client.24424.objecter successfully canceled tick 2024-02-13T00:37:56.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 -- 172.21.15.181:0/1500641082 >> v2:172.21.15.162:6800/1909610488 conn(0x7f005c062c20 msgr2=0x7f005c0650e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:56.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 --2- 172.21.15.181:0/1500641082 >> v2:172.21.15.162:6800/1909610488 conn(0x7f005c062c20 0x7f005c0650e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f0068004b80 tx=0x7f0068009040).stop 2024-02-13T00:37:56.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 -- 172.21.15.181:0/1500641082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00740f9180 msgr2=0x7f0074108380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:56.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 --2- 172.21.15.181:0/1500641082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00740f9180 0x7f0074108380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f006400c810 tx=0x7f006400cbd0).stop 2024-02-13T00:37:56.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 -- 172.21.15.181:0/1500641082 shutdown_connections 2024-02-13T00:37:56.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 --2- 172.21.15.181:0/1500641082 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00740f9a90 0x7f007410f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:56.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 --2- 172.21.15.181:0/1500641082 >> v2:172.21.15.162:6800/1909610488 conn(0x7f005c062c20 0x7f005c0650e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:56.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 --2- 172.21.15.181:0/1500641082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00740f6270 0x7f0074107e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:56.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 --2- 172.21.15.181:0/1500641082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00740f9180 0x7f0074108380 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:56.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 -- 172.21.15.181:0/1500641082 >> 172.21.15.181:0/1500641082 conn(0x7f00740f19c0 msgr2=0x7f00740f3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:56.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 -- 172.21.15.181:0/1500641082 shutdown_connections 2024-02-13T00:37:56.653 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 -- 172.21.15.181:0/1500641082 wait complete. 2024-02-13T00:37:56.653 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:56.647+0000 7f00793d6700 1 librados: shutdown 2024-02-13T00:37:56.998 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:55.529244+0000 mgr.y (mgr.14146) 219 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 KiB/s rd, 4.8 KiB/s wr, 220 op/s 2024-02-13T00:37:56.998 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:56.647097+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:57.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:55.529244+0000 mgr.y (mgr.14146) 219 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 KiB/s rd, 4.8 KiB/s wr, 220 op/s 2024-02-13T00:37:57.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:56.647097+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:57.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:55.529244+0000 mgr.y (mgr.14146) 219 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 KiB/s rd, 4.8 KiB/s wr, 220 op/s 2024-02-13T00:37:57.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:56.647097+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:57.356 DEBUG:teuthology.orchestra.run.smithi162:alertmanager.a> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@alertmanager.a.service 2024-02-13T00:37:57.358 INFO:tasks.cephadm:Adding grafana.a on smithi181 2024-02-13T00:37:57.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph orch apply grafana '1;smithi181=a' 2024-02-13T00:37:57.390 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:-- Logs begin at Tue 2024-02-13 00:17:45 UTC. -- 2024-02-13T00:37:58.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:56.640808+0000 mgr.y (mgr.14146) 220 : audit [DBG] from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:58.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:37:56.642244+0000 mgr.y (mgr.14146) 221 : cephadm [INF] Saving service alertmanager spec with placement smithi162=a;count:1 2024-02-13T00:37:58.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:57.666471+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:58.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:56.640808+0000 mgr.y (mgr.14146) 220 : audit [DBG] from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:58.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:37:56.642244+0000 mgr.y (mgr.14146) 221 : cephadm [INF] Saving service alertmanager spec with placement smithi162=a;count:1 2024-02-13T00:37:58.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:57.666471+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:58.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:56.640808+0000 mgr.y (mgr.14146) 220 : audit [DBG] from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T00:37:58.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:37:56.642244+0000 mgr.y (mgr.14146) 221 : cephadm [INF] Saving service alertmanager spec with placement smithi162=a;count:1 2024-02-13T00:37:58.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:57.666471+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:58.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:57.530037+0000 mgr.y (mgr.14146) 222 : cluster [DBG] pgmap v211: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 KiB/s rd, 3.7 KiB/s wr, 181 op/s 2024-02-13T00:37:58.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:57.819650+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:58.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:57.820537+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:58.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:57.821365+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:58.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:57.829058+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:58.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:57.833972+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-13T00:37:58.947 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:37:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ignoring --setuser ceph since I am not root 2024-02-13T00:37:58.947 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:37:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ignoring --setgroup ceph since I am not root 2024-02-13T00:37:58.947 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:37:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:37:58.869+0000 7f2293521700 1 -- 172.21.15.181:0/4069105348 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c985bda4e0 con 0x55c985bb1400 2024-02-13T00:37:59.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:57.530037+0000 mgr.y (mgr.14146) 222 : cluster [DBG] pgmap v211: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 KiB/s rd, 3.7 KiB/s wr, 181 op/s 2024-02-13T00:37:59.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:57.819650+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:59.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:57.820537+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:59.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:57.821365+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:59.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:57.829058+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:59.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:57.833972+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-13T00:37:59.080 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ignoring --setuser ceph since I am not root 2024-02-13T00:37:59.080 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ignoring --setgroup ceph since I am not root 2024-02-13T00:37:59.080 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:37:58.876+0000 7f1779075700 1 -- 172.21.15.162:0/1743657853 <== mon.2 v2:172.21.15.162:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556a36b444e0 con 0x556a36b1b000 2024-02-13T00:37:59.081 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:37:58.998+0000 7f1789253680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T00:37:59.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:57.530037+0000 mgr.y (mgr.14146) 222 : cluster [DBG] pgmap v211: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 KiB/s rd, 3.7 KiB/s wr, 181 op/s 2024-02-13T00:37:59.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:57.819650+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:59.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:57.820537+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:37:59.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:57.821365+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:37:59.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:57.829058+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' 2024-02-13T00:37:59.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:57.833972+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-13T00:37:59.198 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:37:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:37:58.997+0000 7f22a36ff680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T00:37:59.198 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:37:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:37:59.080+0000 7f22a36ff680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T00:37:59.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.438+0000 7f714b01a700 1 -- 172.21.15.181:0/631924399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7144061710 msgr2=0x7f71440fa770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:59.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.438+0000 7f714b01a700 1 --2- 172.21.15.181:0/631924399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7144061710 0x7f71440fa770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f7138004870 tx=0x7f7138009cb0).stop 2024-02-13T00:37:59.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.438+0000 7f714b01a700 1 -- 172.21.15.181:0/631924399 shutdown_connections 2024-02-13T00:37:59.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.438+0000 7f714b01a700 1 --2- 172.21.15.181:0/631924399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71440fd720 0x7f71440ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:59.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.438+0000 7f714b01a700 1 --2- 172.21.15.181:0/631924399 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7144061710 0x7f71440fa770 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:59.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.438+0000 7f714b01a700 1 --2- 172.21.15.181:0/631924399 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71440facf0 0x7f71440fd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:59.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.438+0000 7f714b01a700 1 -- 172.21.15.181:0/631924399 >> 172.21.15.181:0/631924399 conn(0x7f71440f2230 msgr2=0x7f71440f4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.438+0000 7f714b01a700 1 -- 172.21.15.181:0/631924399 shutdown_connections 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.438+0000 7f714b01a700 1 -- 172.21.15.181:0/631924399 wait complete. 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f714b01a700 1 librados: starting msgr at 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f714b01a700 1 librados: starting objecter 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f714b01a700 1 Processor -- start 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f714b01a700 1 -- start start 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f714b01a700 1 librados: setting wanted keys 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f714b01a700 1 librados: calling monclient init 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f714b01a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7144061710 0x7f7144108100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f714b01a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71440facf0 0x7f71441085f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f714b01a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71440fd720 0x7f714410f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f714b01a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714410fbc0 con 0x7f7144061710 2024-02-13T00:37:59.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f714b01a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714410fd00 con 0x7f71440fd720 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f714b01a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714410fe40 con 0x7f71440facf0 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f7148db6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7144061710 0x7f7144108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f71495b7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71440fd720 0x7f714410f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f7143fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71440facf0 0x7f71441085f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f7148db6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7144061710 0x7f7144108100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.181:53604/0 (socket says 172.21.15.181:53604) 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.439+0000 7f7148db6700 1 -- 172.21.15.181:0/1761066479 learned_addr learned my addr 172.21.15.181:0/1761066479 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f71495b7700 1 -- 172.21.15.181:0/1761066479 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71440facf0 msgr2=0x7f71441085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f71495b7700 1 --2- 172.21.15.181:0/1761066479 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71440facf0 0x7f71441085f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f71495b7700 1 -- 172.21.15.181:0/1761066479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7144061710 msgr2=0x7f7144108100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f71495b7700 1 --2- 172.21.15.181:0/1761066479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7144061710 0x7f7144108100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f71495b7700 1 -- 172.21.15.181:0/1761066479 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7138005040 con 0x7f71440fd720 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f7148db6700 1 --2- 172.21.15.181:0/1761066479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7144061710 0x7f7144108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f71495b7700 1 --2- 172.21.15.181:0/1761066479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71440fd720 0x7f714410f6d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f713c00eb70 tx=0x7f713c00b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f7141ffb700 10 client.?.objecter ms_handle_connect 0x7f71440fd720 2024-02-13T00:37:59.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f7141ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:37:59.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f7141ffb700 1 -- 172.21.15.181:0/1761066479 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c009870 con 0x7f71440fd720 2024-02-13T00:37:59.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f7141ffb700 1 -- 172.21.15.181:0/1761066479 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f713c00f040 con 0x7f71440fd720 2024-02-13T00:37:59.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f7141ffb700 1 -- 172.21.15.181:0/1761066479 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c007da0 con 0x7f71440fd720 2024-02-13T00:37:59.446 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f714b01a700 1 -- 172.21.15.181:0/1761066479 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71441100c0 con 0x7f71440fd720 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f714b01a700 10 client.24442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f714b01a700 1 -- 172.21.15.181:0/1761066479 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7144110560 con 0x7f71440fd720 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.440+0000 7f714b01a700 1 librados: init done 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.441+0000 7f714b01a700 1 -- 172.21.15.181:0/1761066479 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714405f400 con 0x7f71440fd720 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.442+0000 7f7141ffb700 1 -- 172.21.15.181:0/1761066479 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 77095+0+0 (secure 0 0 0) 0x7f713c0040b0 con 0x7f71440fd720 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.442+0000 7f7141ffb700 1 --2- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 0x7f7134065130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.442+0000 7f7141ffb700 1 -- 172.21.15.181:0/1761066479 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5884+0+0 (secure 0 0 0) 0x7f713c07c690 con 0x7f71440fd720 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.442+0000 7f7148db6700 1 -- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 msgr2=0x7f7134065130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/1909610488 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.442+0000 7f7141ffb700 10 client.24442.objecter ms_dispatch 0x7f7144001550 osd_map(67..67 src has 1..67) v4 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.442+0000 7f7141ffb700 3 client.24442.objecter handle_osd_map got epochs [67,67] > 0 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.442+0000 7f7141ffb700 3 client.24442.objecter handle_osd_map decoding full epoch 67 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.442+0000 7f7148db6700 1 --2- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 0x7f7134065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-13T00:37:59.447 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.442+0000 7f7141ffb700 20 client.24442.objecter dump_active .. 0 homeless 2024-02-13T00:37:59.449 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.446+0000 7f7141ffb700 1 -- 172.21.15.181:0/1761066479 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f713c049f60 con 0x7f71440fd720 2024-02-13T00:37:59.451 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:37:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:37:59.079+0000 7f1789253680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T00:37:59.467 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:37:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:37:59.463+0000 7f22a36ff680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T00:37:59.634 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.631+0000 7f714b01a700 1 -- 172.21.15.181:0/1761066479 --> v2:172.21.15.162:6800/1909610488 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f71440f3cc0 con 0x7f7134062c70 2024-02-13T00:37:59.646 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.643+0000 7f7148db6700 1 -- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 msgr2=0x7f7134065130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/1909610488 2024-02-13T00:37:59.646 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:37:59.643+0000 7f7148db6700 1 --2- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 0x7f7134065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-13T00:37:59.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:37:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:37:59.448+0000 7f1789253680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T00:38:00.047 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:00.044+0000 7f7148db6700 1 -- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 msgr2=0x7f7134065130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/1909610488 2024-02-13T00:38:00.047 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:00.044+0000 7f7148db6700 1 --2- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 0x7f7134065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-13T00:38:00.194 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:37:58.839045+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-13T00:38:00.194 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:58.839106+0000 mon.a (mon.0) 611 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-13T00:38:00.194 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:37:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:37:59.205455+0000 mon.a (mon.0) 612 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-13T00:38:00.194 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:37:58.839045+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-13T00:38:00.194 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:58.839106+0000 mon.a (mon.0) 611 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-13T00:38:00.194 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:37:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:37:59.205455+0000 mon.a (mon.0) 612 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-13T00:38:00.216 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:37:58.839045+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 172.21.15.162:0/4292543960' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-13T00:38:00.216 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:58.839106+0000 mon.a (mon.0) 611 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-13T00:38:00.216 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:37:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:37:59.205455+0000 mon.a (mon.0) 612 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-13T00:38:00.216 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:00.212+0000 7f22a36ff680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T00:38:00.487 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:00.190+0000 7f1789253680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T00:38:00.487 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:00.274+0000 7f1789253680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T00:38:00.510 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:00.297+0000 7f22a36ff680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T00:38:00.794 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:00.507+0000 7f22a36ff680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T00:38:00.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:00.484+0000 7f1789253680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T00:38:00.848 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:00.845+0000 7f7148db6700 1 -- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 msgr2=0x7f7134065130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/1909610488 2024-02-13T00:38:00.848 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:00.845+0000 7f7148db6700 1 --2- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 0x7f7134065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-13T00:38:01.716 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:01.417+0000 7f1789253680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T00:38:01.716 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:01.544+0000 7f1789253680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T00:38:01.716 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:01.623+0000 7f1789253680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T00:38:01.736 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:01.439+0000 7f22a36ff680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T00:38:01.736 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:01.566+0000 7f22a36ff680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T00:38:01.736 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:01.644+0000 7f22a36ff680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T00:38:01.736 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:01.733+0000 7f22a36ff680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T00:38:02.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:01.930+0000 7f22a36ff680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T00:38:02.079 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:01.713+0000 7f1789253680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T00:38:02.079 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:01.910+0000 7f1789253680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T00:38:02.450 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:02.446+0000 7f7148db6700 1 -- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 msgr2=0x7f7134065130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.162:6800/1909610488 2024-02-13T00:38:02.450 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:02.446+0000 7f7148db6700 1 --2- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 0x7f7134065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-13T00:38:02.544 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:02.205+0000 7f22a36ff680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T00:38:02.544 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:02.302+0000 7f22a36ff680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T00:38:02.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:02.187+0000 7f1789253680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T00:38:02.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:02.284+0000 7f1789253680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T00:38:03.294 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:03.020+0000 7f22a36ff680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T00:38:03.294 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:03.101+0000 7f22a36ff680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T00:38:03.294 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:03.191+0000 7f22a36ff680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T00:38:03.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:03.004+0000 7f1789253680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T00:38:03.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:03.086+0000 7f1789253680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T00:38:03.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:03.177+0000 7f1789253680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T00:38:03.615 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:03.353+0000 7f1789253680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T00:38:03.629 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:03.364+0000 7f22a36ff680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T00:38:03.879 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:03.612+0000 7f1789253680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T00:38:03.879 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:03.742+0000 7f1789253680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T00:38:03.888 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:03.624+0000 7f22a36ff680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T00:38:03.888 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:03.753+0000 7f22a36ff680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T00:38:04.132 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:03.875+0000 7f1789253680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T00:38:04.132 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:04.046+0000 7f1789253680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T00:38:04.139 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:03.884+0000 7f22a36ff680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T00:38:04.139 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:04.053+0000 7f22a36ff680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T00:38:04.151 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:04.148+0000 7f7141ffb700 1 -- 172.21.15.181:0/1761066479 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mgrmap(e 17) v1 ==== 76859+0+0 (secure 0 0 0) 0x7f713c07a500 con 0x7f71440fd720 2024-02-13T00:38:04.152 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:04.148+0000 7f7141ffb700 1 -- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 msgr2=0x7f7134065130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T00:38:04.152 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:04.148+0000 7f7141ffb700 1 --2- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/1909610488 conn(0x7f7134062c70 0x7f7134065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:38:04.394 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: debug 2024-02-13T00:38:04.135+0000 7f22a36ff680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T00:38:04.394 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: [13/Feb/2024:00:38:04] ENGINE Bus STARTING 2024-02-13T00:38:04.394 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: CherryPy Checker: 2024-02-13T00:38:04.394 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: The Application mounted at '' has an empty config. 2024-02-13T00:38:04.394 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: 2024-02-13T00:38:04.395 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: [13/Feb/2024:00:38:04] ENGINE Serving on http://:::9283 2024-02-13T00:38:04.395 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: [13/Feb/2024:00:38:04] ENGINE Bus STARTED 2024-02-13T00:38:04.395 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:04.142872+0000 mon.a (mon.0) 613 : cluster [INF] Active manager daemon y restarted 2024-02-13T00:38:04.395 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:04.143308+0000 mon.a (mon.0) 614 : cluster [INF] Activating manager daemon y 2024-02-13T00:38:04.396 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.152085+0000 mon.b (mon.1) 21 : audit [DBG] from='mgr.? 172.21.15.181:0/3073881578' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T00:38:04.396 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:04.152349+0000 mon.a (mon.0) 615 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-13T00:38:04.396 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.152700+0000 mon.b (mon.1) 22 : audit [DBG] from='mgr.? 172.21.15.181:0/3073881578' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T00:38:04.396 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:04.152892+0000 mon.a (mon.0) 616 : cluster [DBG] mgrmap e17: y(active, starting, since 0.00968779s), standbys: x 2024-02-13T00:38:04.396 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:04.154220+0000 mon.a (mon.0) 617 : cluster [DBG] Standby manager daemon x restarted 2024-02-13T00:38:04.396 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:04.154325+0000 mon.a (mon.0) 618 : cluster [DBG] Standby manager daemon x started 2024-02-13T00:38:04.396 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.154903+0000 mon.b (mon.1) 23 : audit [DBG] from='mgr.? 172.21.15.181:0/3073881578' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T00:38:04.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.155175+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T00:38:04.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.155321+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:38:04.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.155485+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:38:04.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.155903+0000 mon.b (mon.1) 24 : audit [DBG] from='mgr.? 172.21.15.181:0/3073881578' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T00:38:04.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.156367+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T00:38:04.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.156584+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T00:38:04.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.156821+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:38:04.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.157046+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:38:04.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.157259+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:38:04.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.157527+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:38:04.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.157861+0000 mon.c (mon.2) 29 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:38:04.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.158170+0000 mon.c (mon.2) 30 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:38:04.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.158422+0000 mon.c (mon.2) 31 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:38:04.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.158720+0000 mon.c (mon.2) 32 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:38:04.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.159025+0000 mon.c (mon.2) 33 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T00:38:04.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.159224+0000 mon.c (mon.2) 34 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T00:38:04.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.159700+0000 mon.c (mon.2) 35 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T00:38:04.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:04.174130+0000 mon.a (mon.0) 619 : cluster [INF] Manager daemon y is now available 2024-02-13T00:38:04.400 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:38:04.129+0000 7f1789253680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T00:38:04.400 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:38:04] ENGINE Bus STARTING 2024-02-13T00:38:04.400 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [80B blob data] 2024-02-13T00:38:04.400 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [80B blob data] 2024-02-13T00:38:04.400 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:04.142872+0000 mon.a (mon.0) 613 : cluster [INF] Active manager daemon y restarted 2024-02-13T00:38:04.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:04.143308+0000 mon.a (mon.0) 614 : cluster [INF] Activating manager daemon y 2024-02-13T00:38:04.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.152085+0000 mon.b (mon.1) 21 : audit [DBG] from='mgr.? 172.21.15.181:0/3073881578' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T00:38:04.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:04.152349+0000 mon.a (mon.0) 615 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-13T00:38:04.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.152700+0000 mon.b (mon.1) 22 : audit [DBG] from='mgr.? 172.21.15.181:0/3073881578' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T00:38:04.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:04.152892+0000 mon.a (mon.0) 616 : cluster [DBG] mgrmap e17: y(active, starting, since 0.00968779s), standbys: x 2024-02-13T00:38:04.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:04.154220+0000 mon.a (mon.0) 617 : cluster [DBG] Standby manager daemon x restarted 2024-02-13T00:38:04.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:04.154325+0000 mon.a (mon.0) 618 : cluster [DBG] Standby manager daemon x started 2024-02-13T00:38:04.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.154903+0000 mon.b (mon.1) 23 : audit [DBG] from='mgr.? 172.21.15.181:0/3073881578' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T00:38:04.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.155175+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T00:38:04.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.155321+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:38:04.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.155485+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:38:04.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.155903+0000 mon.b (mon.1) 24 : audit [DBG] from='mgr.? 172.21.15.181:0/3073881578' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T00:38:04.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.156367+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T00:38:04.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.156584+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T00:38:04.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.156821+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:38:04.403 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:04.142872+0000 mon.a (mon.0) 613 : cluster [INF] Active manager daemon y restarted 2024-02-13T00:38:04.403 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:04.143308+0000 mon.a (mon.0) 614 : cluster [INF] Activating manager daemon y 2024-02-13T00:38:04.403 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.152085+0000 mon.b (mon.1) 21 : audit [DBG] from='mgr.? 172.21.15.181:0/3073881578' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T00:38:04.403 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:04.152349+0000 mon.a (mon.0) 615 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-13T00:38:04.404 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.152700+0000 mon.b (mon.1) 22 : audit [DBG] from='mgr.? 172.21.15.181:0/3073881578' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T00:38:04.404 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:04.152892+0000 mon.a (mon.0) 616 : cluster [DBG] mgrmap e17: y(active, starting, since 0.00968779s), standbys: x 2024-02-13T00:38:04.404 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:04.154220+0000 mon.a (mon.0) 617 : cluster [DBG] Standby manager daemon x restarted 2024-02-13T00:38:04.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:04.154325+0000 mon.a (mon.0) 618 : cluster [DBG] Standby manager daemon x started 2024-02-13T00:38:04.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.154903+0000 mon.b (mon.1) 23 : audit [DBG] from='mgr.? 172.21.15.181:0/3073881578' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T00:38:04.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.155175+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T00:38:04.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.155321+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T00:38:04.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.155485+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T00:38:04.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.155903+0000 mon.b (mon.1) 24 : audit [DBG] from='mgr.? 172.21.15.181:0/3073881578' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T00:38:04.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.156367+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T00:38:04.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.156584+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T00:38:04.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.156821+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T00:38:04.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.157046+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:38:04.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.157259+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:38:04.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.157527+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:38:04.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.157861+0000 mon.c (mon.2) 29 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:38:04.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.158170+0000 mon.c (mon.2) 30 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:38:04.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.158422+0000 mon.c (mon.2) 31 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:38:04.407 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.158720+0000 mon.c (mon.2) 32 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:38:04.407 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.159025+0000 mon.c (mon.2) 33 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T00:38:04.407 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.159224+0000 mon.c (mon.2) 34 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T00:38:04.407 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.159700+0000 mon.c (mon.2) 35 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T00:38:04.407 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:04.174130+0000 mon.a (mon.0) 619 : cluster [INF] Manager daemon y is now available 2024-02-13T00:38:04.407 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.157046+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T00:38:04.407 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.157259+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T00:38:04.407 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.157527+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-13T00:38:04.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.157861+0000 mon.c (mon.2) 29 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-13T00:38:04.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.158170+0000 mon.c (mon.2) 30 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:38:04.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.158422+0000 mon.c (mon.2) 31 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-13T00:38:04.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.158720+0000 mon.c (mon.2) 32 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-13T00:38:04.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.159025+0000 mon.c (mon.2) 33 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T00:38:04.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.159224+0000 mon.c (mon.2) 34 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T00:38:04.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.159700+0000 mon.c (mon.2) 35 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T00:38:04.408 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:04.174130+0000 mon.a (mon.0) 619 : cluster [INF] Manager daemon y is now available 2024-02-13T00:38:04.444 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:04.441+0000 7f714a819700 10 client.24442.objecter tick 2024-02-13T00:38:04.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: CherryPy Checker: 2024-02-13T00:38:04.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: The Application mounted at '' has an empty config. 2024-02-13T00:38:04.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: 2024-02-13T00:38:04.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:38:04] ENGINE Serving on http://:::9283 2024-02-13T00:38:04.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:38:04] ENGINE Bus STARTED 2024-02-13T00:38:05.156 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.152+0000 7f7141ffb700 1 -- 172.21.15.181:0/1761066479 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mgrmap(e 18) v1 ==== 76894+0+0 (secure 0 0 0) 0x7f713c07a880 con 0x7f71440fd720 2024-02-13T00:38:05.156 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.153+0000 7f7141ffb700 1 --2- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/2866534053 conn(0x7f71340323c0 0x7f713406e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:05.156 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.153+0000 7f7141ffb700 1 -- 172.21.15.181:0/1761066479 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f71440f3cc0 con 0x7f71340323c0 2024-02-13T00:38:05.157 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.154+0000 7f7148db6700 1 --2- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/2866534053 conn(0x7f71340323c0 0x7f713406e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:05.157 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.154+0000 7f7141ffb700 10 client.24442.objecter ms_handle_connect 0x7f71340323c0 2024-02-13T00:38:05.157 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.154+0000 7f7148db6700 1 --2- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/2866534053 conn(0x7f71340323c0 0x7f713406e1e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f7138004ee0 tx=0x7f7138016040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:05.165 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.162+0000 7f7141ffb700 1 -- 172.21.15.181:0/1761066479 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f71440f3cc0 con 0x7f71340323c0 2024-02-13T00:38:05.166 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled grafana update... 2024-02-13T00:38:05.167 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.164+0000 7f714b01a700 10 librados: watch_flush enter 2024-02-13T00:38:05.167 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.164+0000 7f714b01a700 10 librados: watch_flush exit 2024-02-13T00:38:05.167 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.164+0000 7f714b01a700 20 client.24442.objecter shutdown clearing up homeless session... 2024-02-13T00:38:05.168 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.164+0000 7f714b01a700 10 client.24442.objecter successfully canceled tick 2024-02-13T00:38:05.168 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.165+0000 7f714b01a700 1 -- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/2866534053 conn(0x7f71340323c0 msgr2=0x7f713406e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:05.169 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.165+0000 7f714b01a700 1 --2- 172.21.15.181:0/1761066479 >> v2:172.21.15.162:6800/2866534053 conn(0x7f71340323c0 0x7f713406e1e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f7138004ee0 tx=0x7f7138016040).stop 2024-02-13T00:38:05.169 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.165+0000 7f714b01a700 1 -- 172.21.15.181:0/1761066479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71440fd720 msgr2=0x7f714410f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:05.169 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.165+0000 7f714b01a700 1 --2- 172.21.15.181:0/1761066479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71440fd720 0x7f714410f6d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f713c00eb70 tx=0x7f713c00b040).stop 2024-02-13T00:38:05.169 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.165+0000 7f71495b7700 1 -- 172.21.15.181:0/1761066479 reap_dead start 2024-02-13T00:38:05.169 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.165+0000 7f714b01a700 1 -- 172.21.15.181:0/1761066479 shutdown_connections 2024-02-13T00:38:05.169 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.165+0000 7f714b01a700 1 -- 172.21.15.181:0/1761066479 >> 172.21.15.181:0/1761066479 conn(0x7f71440f2230 msgr2=0x7f71440f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:05.169 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.165+0000 7f714b01a700 1 -- 172.21.15.181:0/1761066479 shutdown_connections 2024-02-13T00:38:05.169 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.165+0000 7f714b01a700 1 -- 172.21.15.181:0/1761066479 wait complete. 2024-02-13T00:38:05.169 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:05.165+0000 7f714b01a700 1 librados: shutdown 2024-02-13T00:38:05.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:38:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:38:05.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:04.211866+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-13T00:38:05.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.274581+0000 mon.c (mon.2) 36 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:38:05.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.276124+0000 mon.c (mon.2) 37 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:38:05.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.306305+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:05.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.319280+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T00:38:05.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.319750+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T00:38:05.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.341105+0000 mon.c (mon.2) 39 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T00:38:05.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:04.341446+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T00:38:05.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:05.153525+0000 mon.a (mon.0) 624 : cluster [DBG] mgrmap e18: y(active, since 1.01032s), standbys: x 2024-02-13T00:38:05.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:05.162987+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:05.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:04.211866+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-13T00:38:05.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.274581+0000 mon.c (mon.2) 36 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:38:05.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.276124+0000 mon.c (mon.2) 37 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:38:05.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.306305+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:05.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.319280+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T00:38:05.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.319750+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T00:38:05.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.341105+0000 mon.c (mon.2) 39 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T00:38:05.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:04.341446+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T00:38:05.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:05.153525+0000 mon.a (mon.0) 624 : cluster [DBG] mgrmap e18: y(active, since 1.01032s), standbys: x 2024-02-13T00:38:05.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:05.162987+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:05.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:04.211866+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-13T00:38:05.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.274581+0000 mon.c (mon.2) 36 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:38:05.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.276124+0000 mon.c (mon.2) 37 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:38:05.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.306305+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:05.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.319280+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T00:38:05.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.319750+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T00:38:05.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.341105+0000 mon.c (mon.2) 39 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T00:38:05.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:04.341446+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T00:38:05.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:05.153525+0000 mon.a (mon.0) 624 : cluster [DBG] mgrmap e18: y(active, since 1.01032s), standbys: x 2024-02-13T00:38:05.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:05.162987+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:06.227 DEBUG:teuthology.orchestra.run.smithi181:grafana.a> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@grafana.a.service 2024-02-13T00:38:06.230 INFO:tasks.cephadm:Setting up client nodes... 2024-02-13T00:38:06.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-13T00:38:06.263 INFO:journalctl@ceph.grafana.a.smithi181.stdout:-- Logs begin at Tue 2024-02-13 00:17:30 UTC. -- 2024-02-13T00:38:06.937 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:38:05.159614+0000 mgr.y (mgr.14544) 3 : cephadm [INF] Saving service grafana spec with placement smithi181=a;count:1 2024-02-13T00:38:06.937 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:05.160913+0000 mgr.y (mgr.14544) 4 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:38:06.938 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:05.798211+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:06.938 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:06.068316+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.938 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:06.068644+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.938 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:06.069415+0000 mon.c (mon.2) 41 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.938 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:06.069633+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.938 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:06.070237+0000 mon.c (mon.2) 42 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.938 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:06.070436+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.938 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:06.071040+0000 mon.c (mon.2) 43 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.938 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:06.071275+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.938 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:06.076504+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:06.938 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:38:05.159614+0000 mgr.y (mgr.14544) 3 : cephadm [INF] Saving service grafana spec with placement smithi181=a;count:1 2024-02-13T00:38:06.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:05.160913+0000 mgr.y (mgr.14544) 4 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:38:06.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:05.798211+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:06.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:06.068316+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:06.068644+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:06.069415+0000 mon.c (mon.2) 41 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:06.069633+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:06.070237+0000 mon.c (mon.2) 42 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:06.070436+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:06.071040+0000 mon.c (mon.2) 43 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:06.071275+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:06.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:06.076504+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:38:05.159614+0000 mgr.y (mgr.14544) 3 : cephadm [INF] Saving service grafana spec with placement smithi181=a;count:1 2024-02-13T00:38:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:05.160913+0000 mgr.y (mgr.14544) 4 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:38:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:05.798211+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:06.068316+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:06.068644+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:06.069415+0000 mon.c (mon.2) 41 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:06.069633+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:06.070237+0000 mon.c (mon.2) 42 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:06.070436+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:06.071040+0000 mon.c (mon.2) 43 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:06.071275+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:07.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:06.076504+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.054 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:38:06.071783+0000 mgr.y (mgr.14544) 5 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3768M 2024-02-13T00:38:08.054 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:06.157759+0000 mgr.y (mgr.14544) 6 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:38:08.054 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:07.080634+0000 mon.a (mon.0) 632 : cluster [DBG] mgrmap e19: y(active, since 2s), standbys: x 2024-02-13T00:38:08.054 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.103842+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.054 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.388180+0000 mon.c (mon.2) 44 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.054 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.388497+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.054 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.389211+0000 mon.c (mon.2) 45 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.054 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.389438+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.054 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.390068+0000 mon.c (mon.2) 46 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.054 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.390288+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.054 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.390923+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.055 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.391135+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.055 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.396056+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.055 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.397053+0000 mon.c (mon.2) 48 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:38:08.055 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.397619+0000 mon.c (mon.2) 49 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:38:08.055 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.443254+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:38:06.071783+0000 mgr.y (mgr.14544) 5 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3768M 2024-02-13T00:38:08.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:06.157759+0000 mgr.y (mgr.14544) 6 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:38:08.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:07.080634+0000 mon.a (mon.0) 632 : cluster [DBG] mgrmap e19: y(active, since 2s), standbys: x 2024-02-13T00:38:08.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.103842+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.388180+0000 mon.c (mon.2) 44 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.388497+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.389211+0000 mon.c (mon.2) 45 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.389438+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.390068+0000 mon.c (mon.2) 46 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.390288+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.390923+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.391135+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.396056+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.397053+0000 mon.c (mon.2) 48 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.397619+0000 mon.c (mon.2) 49 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.443254+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.484027+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:07.487966+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.057 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.484027+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.057 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:07.487966+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:38:06.071783+0000 mgr.y (mgr.14544) 5 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3768M 2024-02-13T00:38:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:06.157759+0000 mgr.y (mgr.14544) 6 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:38:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:07.080634+0000 mon.a (mon.0) 632 : cluster [DBG] mgrmap e19: y(active, since 2s), standbys: x 2024-02-13T00:38:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.103842+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.388180+0000 mon.c (mon.2) 44 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.388497+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.389211+0000 mon.c (mon.2) 45 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.389438+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.390068+0000 mon.c (mon.2) 46 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.390288+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.390923+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.391135+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:38:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.396056+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.397053+0000 mon.c (mon.2) 48 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:38:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.397619+0000 mon.c (mon.2) 49 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:38:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.443254+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.484027+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:07.487966+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:08.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.299+0000 7fc0e07e4700 1 -- 172.21.15.162:0/3400864297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d80f7240 msgr2=0x7fc0d8103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:08.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.299+0000 7fc0e07e4700 1 --2- 172.21.15.162:0/3400864297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d80f7240 0x7fc0d8103810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc0d0009c30 tx=0x7fc0d0004c50).stop 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.299+0000 7fc0e07e4700 1 -- 172.21.15.162:0/3400864297 shutdown_connections 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.299+0000 7fc0e07e4700 1 --2- 172.21.15.162:0/3400864297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d80f7240 0x7fc0d8103810 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.299+0000 7fc0e07e4700 1 --2- 172.21.15.162:0/3400864297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d80fca50 0x7fc0d80fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.299+0000 7fc0e07e4700 1 --2- 172.21.15.162:0/3400864297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d80fd320 0x7fc0d80f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.299+0000 7fc0e07e4700 1 -- 172.21.15.162:0/3400864297 >> 172.21.15.162:0/3400864297 conn(0x7fc0d80f2640 msgr2=0x7fc0d80f4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.299+0000 7fc0e07e4700 1 -- 172.21.15.162:0/3400864297 shutdown_connections 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 -- 172.21.15.162:0/3400864297 wait complete. 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 librados: starting msgr at 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 librados: starting objecter 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 Processor -- start 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 -- start start 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 librados: setting wanted keys 2024-02-13T00:38:08.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 librados: calling monclient init 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d80f7240 0x7fc0d8106e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d80fca50 0x7fc0d81072f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d80fd320 0x7fc0d81077e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d8116480 con 0x7fc0d80f7240 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d81165c0 con 0x7fc0d80fca50 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.300+0000 7fc0e07e4700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d8116730 con 0x7fc0d80fd320 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0de580700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d80f7240 0x7fc0d8106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0de580700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d80f7240 0x7fc0d8106e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:52130/0 (socket says 172.21.15.162:52130) 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0de580700 1 -- 172.21.15.162:0/1785477175 learned_addr learned my addr 172.21.15.162:0/1785477175 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0ded81700 1 --2- 172.21.15.162:0/1785477175 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d80fd320 0x7fc0d81077e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0ddd7f700 1 --2- 172.21.15.162:0/1785477175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d80fca50 0x7fc0d81072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0de580700 1 -- 172.21.15.162:0/1785477175 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d80fd320 msgr2=0x7fc0d81077e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0de580700 1 --2- 172.21.15.162:0/1785477175 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d80fd320 0x7fc0d81077e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:08.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0de580700 1 -- 172.21.15.162:0/1785477175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d80fca50 msgr2=0x7fc0d81072f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:08.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0de580700 1 --2- 172.21.15.162:0/1785477175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d80fca50 0x7fc0d81072f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:08.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0de580700 1 -- 172.21.15.162:0/1785477175 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c8005040 con 0x7fc0d80f7240 2024-02-13T00:38:08.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0ded81700 1 --2- 172.21.15.162:0/1785477175 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d80fd320 0x7fc0d81077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:08.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0de580700 1 --2- 172.21.15.162:0/1785477175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d80f7240 0x7fc0d8106e00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc0c8009400 tx=0x7fc0c8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:08.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0cf7fe700 10 client.?.objecter ms_handle_connect 0x7fc0d80f7240 2024-02-13T00:38:08.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0cf7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:08.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0cf7fe700 1 -- 172.21.15.162:0/1785477175 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8007a40 con 0x7fc0d80f7240 2024-02-13T00:38:08.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0cf7fe700 1 -- 172.21.15.162:0/1785477175 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0c800f040 con 0x7fc0d80f7240 2024-02-13T00:38:08.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0cf7fe700 1 -- 172.21.15.162:0/1785477175 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8014a70 con 0x7fc0d80f7240 2024-02-13T00:38:08.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0e07e4700 1 -- 172.21.15.162:0/1785477175 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d00097c0 con 0x7fc0d80f7240 2024-02-13T00:38:08.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0e07e4700 10 client.14565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0e07e4700 1 -- 172.21.15.162:0/1785477175 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d8116d70 con 0x7fc0d80f7240 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.301+0000 7fc0e07e4700 1 librados: init done 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.302+0000 7fc0e07e4700 1 -- 172.21.15.162:0/1785477175 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d805f990 con 0x7fc0d80f7240 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.302+0000 7fc0cf7fe700 1 -- 172.21.15.162:0/1785477175 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc0c8010530 con 0x7fc0d80f7240 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.303+0000 7fc0cf7fe700 1 --2- 172.21.15.162:0/1785477175 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0c4062c90 0x7fc0c4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.303+0000 7fc0ddd7f700 1 --2- 172.21.15.162:0/1785477175 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0c4062c90 0x7fc0c4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.303+0000 7fc0cf7fe700 1 -- 172.21.15.162:0/1785477175 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6056+0+0 (secure 0 0 0) 0x7fc0c807c4f0 con 0x7fc0d80f7240 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.303+0000 7fc0cf7fe700 10 client.14565.objecter ms_dispatch 0x7fc0d8001550 osd_map(69..69 src has 1..69) v4 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.303+0000 7fc0cf7fe700 3 client.14565.objecter handle_osd_map got epochs [69,69] > 0 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.303+0000 7fc0cf7fe700 3 client.14565.objecter handle_osd_map decoding full epoch 69 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.303+0000 7fc0cf7fe700 20 client.14565.objecter dump_active .. 0 homeless 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.303+0000 7fc0cf7fe700 10 client.14565.objecter ms_handle_connect 0x7fc0c4062c90 2024-02-13T00:38:08.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.303+0000 7fc0ddd7f700 1 --2- 172.21.15.162:0/1785477175 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0c4062c90 0x7fc0c4065150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fc0d8061250 tx=0x7fc0d400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:08.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.307+0000 7fc0cf7fe700 1 -- 172.21.15.162:0/1785477175 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc0c8049f60 con 0x7fc0d80f7240 2024-02-13T00:38:08.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.546+0000 7fc0e07e4700 1 -- 172.21.15.162:0/1785477175 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc0d8060f90 con 0x7fc0d80f7240 2024-02-13T00:38:08.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.550+0000 7fc0cf7fe700 1 -- 172.21.15.162:0/1785477175 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc0c8050270 con 0x7fc0d80f7240 2024-02-13T00:38:08.553 INFO:teuthology.orchestra.run.smithi162.stdout:[client.0] 2024-02-13T00:38:08.553 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQDwucplQ22dIBAAHfdbDw5r3RbSW/2I7rULWg== 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.551+0000 7fc0e07e4700 10 librados: watch_flush enter 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.551+0000 7fc0e07e4700 10 librados: watch_flush exit 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.551+0000 7fc0e07e4700 20 client.14565.objecter shutdown clearing up homeless session... 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.551+0000 7fc0e07e4700 10 client.14565.objecter successfully canceled tick 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 -- 172.21.15.162:0/1785477175 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0c4062c90 msgr2=0x7fc0c4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 --2- 172.21.15.162:0/1785477175 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0c4062c90 0x7fc0c4065150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fc0d8061250 tx=0x7fc0d400b040).stop 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 -- 172.21.15.162:0/1785477175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d80f7240 msgr2=0x7fc0d8106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 --2- 172.21.15.162:0/1785477175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d80f7240 0x7fc0d8106e00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc0c8009400 tx=0x7fc0c8009cb0).stop 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 -- 172.21.15.162:0/1785477175 shutdown_connections 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 --2- 172.21.15.162:0/1785477175 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0c4062c90 0x7fc0c4065150 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 --2- 172.21.15.162:0/1785477175 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d80fd320 0x7fc0d81077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 --2- 172.21.15.162:0/1785477175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d80f7240 0x7fc0d8106e00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:08.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 --2- 172.21.15.162:0/1785477175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d80fca50 0x7fc0d81072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:08.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 -- 172.21.15.162:0/1785477175 >> 172.21.15.162:0/1785477175 conn(0x7fc0d80f2640 msgr2=0x7fc0d80fa870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:08.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 -- 172.21.15.162:0/1785477175 shutdown_connections 2024-02-13T00:38:08.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 -- 172.21.15.162:0/1785477175 wait complete. 2024-02-13T00:38:08.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:08.552+0000 7fc0e07e4700 1 librados: shutdown 2024-02-13T00:38:08.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:38:07.391631+0000 mgr.y (mgr.14544) 7 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 4024M 2024-02-13T00:38:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:38:07.398626+0000 mgr.y (mgr.14544) 8 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-02-13T00:38:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:38:07.398867+0000 mgr.y (mgr.14544) 9 : cephadm [INF] Updating smithi181:/etc/ceph/ceph.conf 2024-02-13T00:38:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:38:07.412706+0000 mgr.y (mgr.14544) 10 : cephadm [INF] Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-02-13T00:38:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:38:07.419740+0000 mgr.y (mgr.14544) 11 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-02-13T00:38:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:38:07.498176+0000 mgr.y (mgr.14544) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi162 2024-02-13T00:38:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:08.547031+0000 mon.a (mon.0) 642 : audit [INF] from='client.? 172.21.15.162:0/1785477175' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T00:38:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:08.550695+0000 mon.a (mon.0) 643 : audit [INF] from='client.? 172.21.15.162:0/1785477175' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T00:38:08.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:38:07.391631+0000 mgr.y (mgr.14544) 7 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 4024M 2024-02-13T00:38:08.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:38:07.398626+0000 mgr.y (mgr.14544) 8 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-02-13T00:38:08.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:38:07.398867+0000 mgr.y (mgr.14544) 9 : cephadm [INF] Updating smithi181:/etc/ceph/ceph.conf 2024-02-13T00:38:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:38:07.412706+0000 mgr.y (mgr.14544) 10 : cephadm [INF] Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-02-13T00:38:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:38:07.419740+0000 mgr.y (mgr.14544) 11 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-02-13T00:38:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:38:07.498176+0000 mgr.y (mgr.14544) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi162 2024-02-13T00:38:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:08.547031+0000 mon.a (mon.0) 642 : audit [INF] from='client.? 172.21.15.162:0/1785477175' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T00:38:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:08.550695+0000 mon.a (mon.0) 643 : audit [INF] from='client.? 172.21.15.162:0/1785477175' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T00:38:09.289 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:38:09.289 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-13T00:38:09.289 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-13T00:38:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:38:07.391631+0000 mgr.y (mgr.14544) 7 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 4024M 2024-02-13T00:38:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:38:07.398626+0000 mgr.y (mgr.14544) 8 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-02-13T00:38:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:38:07.398867+0000 mgr.y (mgr.14544) 9 : cephadm [INF] Updating smithi181:/etc/ceph/ceph.conf 2024-02-13T00:38:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:38:07.412706+0000 mgr.y (mgr.14544) 10 : cephadm [INF] Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-02-13T00:38:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:38:07.419740+0000 mgr.y (mgr.14544) 11 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-02-13T00:38:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:38:07.498176+0000 mgr.y (mgr.14544) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi162 2024-02-13T00:38:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:08.547031+0000 mon.a (mon.0) 642 : audit [INF] from='client.? 172.21.15.162:0/1785477175' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T00:38:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:08.550695+0000 mon.a (mon.0) 643 : audit [INF] from='client.? 172.21.15.162:0/1785477175' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T00:38:09.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-13T00:38:10.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:08.158264+0000 mgr.y (mgr.14544) 13 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:38:10.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:09.041855+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:10.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:09.041986+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:10.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:09.212798+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-13T00:38:10.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:09.212997+0000 mon.c (mon.2) 50 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-13T00:38:10.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:09.213508+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-13T00:38:10.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:09.221501+0000 mon.c (mon.2) 51 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-13T00:38:10.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:09.221923+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-13T00:38:10.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:08.158264+0000 mgr.y (mgr.14544) 13 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:38:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:09.041855+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:09.041986+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:09.212798+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-13T00:38:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:09.212997+0000 mon.c (mon.2) 50 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-13T00:38:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:09.213508+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-13T00:38:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:09.221501+0000 mon.c (mon.2) 51 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-13T00:38:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:09.221923+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-13T00:38:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:08.158264+0000 mgr.y (mgr.14544) 13 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:38:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:09.041855+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:09.041986+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:09.212798+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-13T00:38:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:09.212997+0000 mon.c (mon.2) 50 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-13T00:38:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:09.213508+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-13T00:38:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:09.221501+0000 mon.c (mon.2) 51 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-13T00:38:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:09.221923+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-13T00:38:11.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:10.158823+0000 mgr.y (mgr.14544) 14 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-02-13T00:38:11.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:10.219494+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-13T00:38:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:10.219555+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-13T00:38:11.350 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 -- 172.21.15.181:0/3225728919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47bc0f6800 msgr2=0x7f47bc104b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:11.350 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 --2- 172.21.15.181:0/3225728919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47bc0f6800 0x7f47bc104b10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f47b4009be0 tx=0x7f47b4004d10).stop 2024-02-13T00:38:11.350 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 -- 172.21.15.181:0/3225728919 shutdown_connections 2024-02-13T00:38:11.350 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 --2- 172.21.15.181:0/3225728919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47bc0f6800 0x7f47bc104b10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:11.350 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 --2- 172.21.15.181:0/3225728919 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc0fdd10 0x7f47bc0fe0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:11.350 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 --2- 172.21.15.181:0/3225728919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47bc0fe670 0x7f47bc0f6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:11.350 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 -- 172.21.15.181:0/3225728919 >> 172.21.15.181:0/3225728919 conn(0x7f47bc070030 msgr2=0x7f47bc0f4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:11.350 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 -- 172.21.15.181:0/3225728919 shutdown_connections 2024-02-13T00:38:11.350 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 -- 172.21.15.181:0/3225728919 wait complete. 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 librados: starting msgr at 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 librados: starting objecter 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 Processor -- start 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.294+0000 7f47c4173700 1 -- start start 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c4173700 1 librados: setting wanted keys 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c4173700 1 librados: calling monclient init 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c4173700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc0f6800 0x7f47bc10e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c4173700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47bc0fdd10 0x7f47bc10e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c4173700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47bc0fe670 0x7f47bc1081b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c4173700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc1086a0 con 0x7f47bc0fe670 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c4173700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc108810 con 0x7f47bc0fdd10 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c4173700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc1089d0 con 0x7f47bc0f6800 2024-02-13T00:38:11.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c170e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47bc0fdd10 0x7f47bc10e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:11.352 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c170e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47bc0fdd10 0x7f47bc10e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43556/0 (socket says 172.21.15.181:43556) 2024-02-13T00:38:11.352 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c170e700 1 -- 172.21.15.181:0/1334015358 learned_addr learned my addr 172.21.15.181:0/1334015358 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-02-13T00:38:11.352 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c2710700 1 --2- 172.21.15.181:0/1334015358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47bc0fe670 0x7f47bc1081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:11.352 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c1f0f700 1 --2- 172.21.15.181:0/1334015358 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc0f6800 0x7f47bc10e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:11.352 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c170e700 1 -- 172.21.15.181:0/1334015358 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc0f6800 msgr2=0x7f47bc10e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:11.352 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c170e700 1 --2- 172.21.15.181:0/1334015358 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc0f6800 0x7f47bc10e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:11.352 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c170e700 1 -- 172.21.15.181:0/1334015358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47bc0fe670 msgr2=0x7f47bc1081b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:11.352 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c170e700 1 --2- 172.21.15.181:0/1334015358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47bc0fe670 0x7f47bc1081b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:11.352 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c170e700 1 -- 172.21.15.181:0/1334015358 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b40097c0 con 0x7f47bc0fdd10 2024-02-13T00:38:11.352 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c170e700 1 --2- 172.21.15.181:0/1334015358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47bc0fdd10 0x7f47bc10e500 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f47ac0028a0 tx=0x7f47ac009f20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:11.352 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47b2ffd700 10 client.?.objecter ms_handle_connect 0x7f47bc0fdd10 2024-02-13T00:38:11.352 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47b2ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:11.353 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47b2ffd700 1 -- 172.21.15.181:0/1334015358 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac007b40 con 0x7f47bc0fdd10 2024-02-13T00:38:11.353 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47b2ffd700 1 -- 172.21.15.181:0/1334015358 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47ac00f040 con 0x7f47bc0fdd10 2024-02-13T00:38:11.353 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c4173700 1 -- 172.21.15.181:0/1334015358 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc108cb0 con 0x7f47bc0fdd10 2024-02-13T00:38:11.353 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47b2ffd700 1 -- 172.21.15.181:0/1334015358 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac014820 con 0x7f47bc0fdd10 2024-02-13T00:38:11.353 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c4173700 10 client.24463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:11.353 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c4173700 1 -- 172.21.15.181:0/1334015358 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc109030 con 0x7f47bc0fdd10 2024-02-13T00:38:11.353 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.295+0000 7f47c4173700 1 librados: init done 2024-02-13T00:38:11.353 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.296+0000 7f47c4173700 1 -- 172.21.15.181:0/1334015358 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc060070 con 0x7f47bc0fdd10 2024-02-13T00:38:11.353 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.297+0000 7f47b2ffd700 1 -- 172.21.15.181:0/1334015358 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f47ac007ca0 con 0x7f47bc0fdd10 2024-02-13T00:38:11.353 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.297+0000 7f47b2ffd700 1 --2- 172.21.15.181:0/1334015358 >> v2:172.21.15.162:6800/2866534053 conn(0x7f47a8062c90 0x7f47a8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:11.354 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.297+0000 7f47b2ffd700 1 -- 172.21.15.181:0/1334015358 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f47ac07c640 con 0x7f47bc0fdd10 2024-02-13T00:38:11.354 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.297+0000 7f47b2ffd700 10 client.24463.objecter ms_dispatch 0x7f47bc001550 osd_map(71..71 src has 1..71) v4 2024-02-13T00:38:11.354 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.297+0000 7f47b2ffd700 3 client.24463.objecter handle_osd_map got epochs [71,71] > 0 2024-02-13T00:38:11.354 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.297+0000 7f47b2ffd700 3 client.24463.objecter handle_osd_map decoding full epoch 71 2024-02-13T00:38:11.354 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.297+0000 7f47c1f0f700 1 --2- 172.21.15.181:0/1334015358 >> v2:172.21.15.162:6800/2866534053 conn(0x7f47a8062c90 0x7f47a8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:11.354 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.297+0000 7f47b2ffd700 20 client.24463.objecter dump_active .. 0 homeless 2024-02-13T00:38:11.354 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.298+0000 7f47b2ffd700 10 client.24463.objecter ms_handle_connect 0x7f47a8062c90 2024-02-13T00:38:11.354 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.298+0000 7f47c1f0f700 1 --2- 172.21.15.181:0/1334015358 >> v2:172.21.15.162:6800/2866534053 conn(0x7f47a8062c90 0x7f47a8065150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f47b8004e60 tx=0x7f47b800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:11.354 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.301+0000 7f47b2ffd700 1 -- 172.21.15.181:0/1334015358 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f47ac0503c0 con 0x7f47bc0fdd10 2024-02-13T00:38:11.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.529+0000 7f47c4173700 1 -- 172.21.15.181:0/1334015358 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f47bc109690 con 0x7f47bc0fdd10 2024-02-13T00:38:11.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.534+0000 7f47b2ffd700 1 -- 172.21.15.181:0/1334015358 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f47ac04ffb0 con 0x7f47bc0fdd10 2024-02-13T00:38:11.537 INFO:teuthology.orchestra.run.smithi181.stdout:[client.1] 2024-02-13T00:38:11.537 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDzucplohOuHxAA2OWTTrM6UH256+qE+8r5pw== 2024-02-13T00:38:11.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.535+0000 7f47c4173700 10 librados: watch_flush enter 2024-02-13T00:38:11.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.535+0000 7f47c4173700 10 librados: watch_flush exit 2024-02-13T00:38:11.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.535+0000 7f47c4173700 20 client.24463.objecter shutdown clearing up homeless session... 2024-02-13T00:38:11.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.535+0000 7f47c4173700 10 client.24463.objecter successfully canceled tick 2024-02-13T00:38:11.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 -- 172.21.15.181:0/1334015358 >> v2:172.21.15.162:6800/2866534053 conn(0x7f47a8062c90 msgr2=0x7f47a8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:11.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 --2- 172.21.15.181:0/1334015358 >> v2:172.21.15.162:6800/2866534053 conn(0x7f47a8062c90 0x7f47a8065150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f47b8004e60 tx=0x7f47b800b040).stop 2024-02-13T00:38:11.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 -- 172.21.15.181:0/1334015358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47bc0fdd10 msgr2=0x7f47bc10e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:11.540 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 --2- 172.21.15.181:0/1334015358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47bc0fdd10 0x7f47bc10e500 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f47ac0028a0 tx=0x7f47ac009f20).stop 2024-02-13T00:38:11.540 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 -- 172.21.15.181:0/1334015358 shutdown_connections 2024-02-13T00:38:11.540 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 --2- 172.21.15.181:0/1334015358 >> v2:172.21.15.162:6800/2866534053 conn(0x7f47a8062c90 0x7f47a8065150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:11.540 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 --2- 172.21.15.181:0/1334015358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47bc0fe670 0x7f47bc1081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:11.540 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 --2- 172.21.15.181:0/1334015358 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc0f6800 0x7f47bc10e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:11.540 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 --2- 172.21.15.181:0/1334015358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47bc0fdd10 0x7f47bc10e500 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:11.540 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 -- 172.21.15.181:0/1334015358 >> 172.21.15.181:0/1334015358 conn(0x7f47bc070030 msgr2=0x7f47bc06c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:11.540 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 -- 172.21.15.181:0/1334015358 shutdown_connections 2024-02-13T00:38:11.540 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 -- 172.21.15.181:0/1334015358 wait complete. 2024-02-13T00:38:11.540 INFO:teuthology.orchestra.run.smithi181.stderr:2024-02-13T00:38:11.536+0000 7f47c4173700 1 librados: shutdown 2024-02-13T00:38:11.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:10.158823+0000 mgr.y (mgr.14544) 14 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-02-13T00:38:11.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:10.219494+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-13T00:38:11.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:10.219555+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-13T00:38:11.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:10.158823+0000 mgr.y (mgr.14544) 14 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-02-13T00:38:11.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:10.219494+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-13T00:38:11.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:10.219555+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-13T00:38:12.187 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-02-13T00:38:12.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-13T00:38:12.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-13T00:38:12.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-13T00:38:12.489 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:12.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:11.530699+0000 mon.b (mon.1) 25 : audit [INF] from='client.? 172.21.15.181:0/1334015358' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T00:38:12.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:11.531356+0000 mon.a (mon.0) 649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T00:38:12.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:11.534900+0000 mon.a (mon.0) 650 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T00:38:12.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:11.530699+0000 mon.b (mon.1) 25 : audit [INF] from='client.? 172.21.15.181:0/1334015358' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T00:38:12.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:11.531356+0000 mon.a (mon.0) 649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T00:38:12.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:11.534900+0000 mon.a (mon.0) 650 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T00:38:12.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:11.530699+0000 mon.b (mon.1) 25 : audit [INF] from='client.? 172.21.15.181:0/1334015358' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T00:38:12.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:11.531356+0000 mon.a (mon.0) 649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T00:38:12.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:11.534900+0000 mon.a (mon.0) 650 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T00:38:13.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:12.159231+0000 mgr.y (mgr.14544) 15 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-13T00:38:13.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:12.159231+0000 mgr.y (mgr.14544) 15 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-13T00:38:13.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:12.159231+0000 mgr.y (mgr.14544) 15 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-13T00:38:14.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:12.977816+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:38:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:12.978860+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:38:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:14.208398+0000 mon.a (mon.0) 651 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-13T00:38:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:14.231993+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:14.232054+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:14.809 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:12.977816+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:38:14.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:12.978860+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:38:14.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:14.208398+0000 mon.a (mon.0) 651 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-13T00:38:14.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:14.231993+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:14.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:14.232054+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:14.810 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:12.977816+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:38:14.810 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:12.978860+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:38:14.810 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:14.208398+0000 mon.a (mon.0) 651 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-13T00:38:14.810 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:14.231993+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:14.810 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:14.232054+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:15.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:38:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:38:15.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:14.159666+0000 mgr.y (mgr.14544) 16 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:38:15.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:15.214772+0000 mon.a (mon.0) 652 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-13T00:38:15.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:15.533424+0000 mon.a (mon.0) 653 : audit [DBG] from='client.? 172.21.15.162:0/3580863160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:38:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:14.159666+0000 mgr.y (mgr.14544) 16 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:38:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:15.214772+0000 mon.a (mon.0) 652 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-13T00:38:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:15.533424+0000 mon.a (mon.0) 653 : audit [DBG] from='client.? 172.21.15.162:0/3580863160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:38:16.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:14.159666+0000 mgr.y (mgr.14544) 16 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:38:16.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:15.214772+0000 mon.a (mon.0) 652 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-13T00:38:16.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:15.533424+0000 mon.a (mon.0) 653 : audit [DBG] from='client.? 172.21.15.162:0/3580863160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:38:16.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.357+0000 7fc0f06aa700 1 -- 172.21.15.162:0/2399957128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0e80f95e0 msgr2=0x7fc0e8101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:16.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.357+0000 7fc0f06aa700 1 --2- 172.21.15.162:0/2399957128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0e80f95e0 0x7fc0e8101d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fc0d80047b0 tx=0x7fc0d8009cb0).stop 2024-02-13T00:38:16.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.357+0000 7fc0f06aa700 1 -- 172.21.15.162:0/2399957128 shutdown_connections 2024-02-13T00:38:16.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.357+0000 7fc0f06aa700 1 --2- 172.21.15.162:0/2399957128 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0e8102270 0x7fc0e8104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:16.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.357+0000 7fc0f06aa700 1 --2- 172.21.15.162:0/2399957128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0e80f8d10 0x7fc0e80f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:16.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.357+0000 7fc0f06aa700 1 --2- 172.21.15.162:0/2399957128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0e80f95e0 0x7fc0e8101d80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:16.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.357+0000 7fc0f06aa700 1 -- 172.21.15.162:0/2399957128 >> 172.21.15.162:0/2399957128 conn(0x7fc0e806eb90 msgr2=0x7fc0e80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:16.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.357+0000 7fc0f06aa700 1 -- 172.21.15.162:0/2399957128 shutdown_connections 2024-02-13T00:38:16.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.357+0000 7fc0f06aa700 1 -- 172.21.15.162:0/2399957128 wait complete. 2024-02-13T00:38:16.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0f06aa700 1 librados: starting msgr at 2024-02-13T00:38:16.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0f06aa700 1 librados: starting objecter 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0f06aa700 1 Processor -- start 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0f06aa700 1 -- start start 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0f06aa700 1 librados: setting wanted keys 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0f06aa700 1 librados: calling monclient init 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0f06aa700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0e80f8d10 0x7fc0e80f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0f06aa700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0e80f95e0 0x7fc0e80f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0f06aa700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0e8102270 0x7fc0e80f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0f06aa700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e8116620 con 0x7fc0e8102270 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0f06aa700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e8116790 con 0x7fc0e80f95e0 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0f06aa700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e8116950 con 0x7fc0e80f8d10 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0ee446700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0e80f8d10 0x7fc0e80f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0eec47700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0e8102270 0x7fc0e80f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0ee446700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0e80f8d10 0x7fc0e80f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:39948/0 (socket says 172.21.15.162:39948) 2024-02-13T00:38:16.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0ee446700 1 -- 172.21.15.162:0/3580863160 learned_addr learned my addr 172.21.15.162:0/3580863160 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.358+0000 7fc0edc45700 1 --2- 172.21.15.162:0/3580863160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0e80f95e0 0x7fc0e80f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0eec47700 1 -- 172.21.15.162:0/3580863160 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0e80f8d10 msgr2=0x7fc0e80f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0eec47700 1 --2- 172.21.15.162:0/3580863160 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0e80f8d10 0x7fc0e80f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0eec47700 1 -- 172.21.15.162:0/3580863160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0e80f95e0 msgr2=0x7fc0e80f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0eec47700 1 --2- 172.21.15.162:0/3580863160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0e80f95e0 0x7fc0e80f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0eec47700 1 -- 172.21.15.162:0/3580863160 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e00097c0 con 0x7fc0e8102270 2024-02-13T00:38:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0edc45700 1 --2- 172.21.15.162:0/3580863160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0e80f95e0 0x7fc0e80f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:38:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0ee446700 1 --2- 172.21.15.162:0/3580863160 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0e80f8d10 0x7fc0e80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0eec47700 1 --2- 172.21.15.162:0/3580863160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0e8102270 0x7fc0e80f76a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fc0e0001dc0 tx=0x7fc0e0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0df7fe700 10 client.?.objecter ms_handle_connect 0x7fc0e8102270 2024-02-13T00:38:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0df7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0df7fe700 1 -- 172.21.15.162:0/3580863160 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0009a10 con 0x7fc0e8102270 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0f06aa700 1 -- 172.21.15.162:0/3580863160 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d8005040 con 0x7fc0e8102270 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0df7fe700 1 -- 172.21.15.162:0/3580863160 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0e00095a0 con 0x7fc0e8102270 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0df7fe700 1 -- 172.21.15.162:0/3580863160 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0007d50 con 0x7fc0e8102270 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0f06aa700 10 client.14577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0f06aa700 1 -- 172.21.15.162:0/3580863160 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e8116f40 con 0x7fc0e8102270 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.359+0000 7fc0f06aa700 1 librados: init done 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.360+0000 7fc0f06aa700 1 -- 172.21.15.162:0/3580863160 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e80475e0 con 0x7fc0e8102270 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.360+0000 7fc0df7fe700 1 -- 172.21.15.162:0/3580863160 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc0e00090d0 con 0x7fc0e8102270 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.361+0000 7fc0df7fe700 1 --2- 172.21.15.162:0/3580863160 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0d4062c90 0x7fc0d4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.361+0000 7fc0df7fe700 1 -- 172.21.15.162:0/3580863160 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6056+0+0 (secure 0 0 0) 0x7fc0e007d340 con 0x7fc0e8102270 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.361+0000 7fc0df7fe700 10 client.14577.objecter ms_dispatch 0x7fc0e8001550 osd_map(73..73 src has 1..73) v4 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.361+0000 7fc0df7fe700 3 client.14577.objecter handle_osd_map got epochs [73,73] > 0 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.361+0000 7fc0df7fe700 3 client.14577.objecter handle_osd_map decoding full epoch 73 2024-02-13T00:38:16.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.361+0000 7fc0ee446700 1 --2- 172.21.15.162:0/3580863160 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0d4062c90 0x7fc0d4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:16.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.361+0000 7fc0df7fe700 20 client.14577.objecter dump_active .. 0 homeless 2024-02-13T00:38:16.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.361+0000 7fc0df7fe700 10 client.14577.objecter ms_handle_connect 0x7fc0d4062c90 2024-02-13T00:38:16.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.361+0000 7fc0ee446700 1 --2- 172.21.15.162:0/3580863160 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0d4062c90 0x7fc0d4065150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fc0e4004fd0 tx=0x7fc0e400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:16.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.365+0000 7fc0df7fe700 1 -- 172.21.15.162:0/3580863160 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc0e0050fc0 con 0x7fc0e8102270 2024-02-13T00:38:16.102 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:38:16.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.532+0000 7fc0f06aa700 1 -- 172.21.15.162:0/3580863160 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0e81071b0 con 0x7fc0e8102270 2024-02-13T00:38:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.533+0000 7fc0df7fe700 1 -- 172.21.15.162:0/3580863160 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+19063 (secure 0 0 0) 0x7fc0e0050bb0 con 0x7fc0e8102270 2024-02-13T00:38:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 10 librados: watch_flush enter 2024-02-13T00:38:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 10 librados: watch_flush exit 2024-02-13T00:38:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 20 client.14577.objecter shutdown clearing up homeless session... 2024-02-13T00:38:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 10 client.14577.objecter successfully canceled tick 2024-02-13T00:38:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 -- 172.21.15.162:0/3580863160 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0d4062c90 msgr2=0x7fc0d4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 --2- 172.21.15.162:0/3580863160 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0d4062c90 0x7fc0d4065150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fc0e4004fd0 tx=0x7fc0e400b040).stop 2024-02-13T00:38:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 -- 172.21.15.162:0/3580863160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0e8102270 msgr2=0x7fc0e80f76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 --2- 172.21.15.162:0/3580863160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0e8102270 0x7fc0e80f76a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fc0e0001dc0 tx=0x7fc0e0004d10).stop 2024-02-13T00:38:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 -- 172.21.15.162:0/3580863160 shutdown_connections 2024-02-13T00:38:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 --2- 172.21.15.162:0/3580863160 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0d4062c90 0x7fc0d4065150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 --2- 172.21.15.162:0/3580863160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0e8102270 0x7fc0e80f76a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:16.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 --2- 172.21.15.162:0/3580863160 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0e80f8d10 0x7fc0e80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:16.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 --2- 172.21.15.162:0/3580863160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0e80f95e0 0x7fc0e80f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:16.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 -- 172.21.15.162:0/3580863160 >> 172.21.15.162:0/3580863160 conn(0x7fc0e806eb90 msgr2=0x7fc0e8102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:16.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 -- 172.21.15.162:0/3580863160 shutdown_connections 2024-02-13T00:38:16.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 -- 172.21.15.162:0/3580863160 wait complete. 2024-02-13T00:38:16.104 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":73,"fsid":"d4ee01fc-ca06-11ee-95b9-87774f69a715","created":"2024-02-13T00:31:51.797017+0000","modified":"2024-02-13T00:38:15.211027+0000","last_up_change":"2024-02-13T00:37:33.855829+0000","last_in_change":"2024-02-13T00:37:16.202456+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-13T00:35:34.545390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-13T00:37:39.459980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-13T00:37:39.740219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-13T00:37:41.489292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-13T00:37:43.466304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-13T00:37:45.542442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6801","nonce":2103743338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2103743338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2103743338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6803","nonce":2103743338}]},"public_addr":"172.21.15.162:6801/2103743338","cluster_addr":"172.21.15.162:6802/2103743338","heartbeat_back_addr":"172.21.15.162:6804/2103743338","heartbeat_front_addr":"172.21.15.162:6803/2103743338","state":["exists","up"]},{"osd":1,"uuid":"39e9d2ef-9aa4-45bb-8a47-b0e0ec222886","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6805","nonce":150599780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":150599780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":150599780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6807","nonce":150599780}]},"public_addr":"172.21.15.162:6805/150599780","cluster_addr":"172.21.15.162:6806/150599780","heartbeat_back_addr":"172.21.15.162:6808/150599780","heartbeat_front_addr":"172.21.15.162:6807/150599780","state":["exists","up"]},{"osd":2,"uuid":"86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6809","nonce":3689719376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3689719376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3689719376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6811","nonce":3689719376}]},"public_addr":"172.21.15.162:6809/3689719376","cluster_addr":"172.21.15.162:6810/3689719376","heartbeat_back_addr":"172.21.15.162:6812/3689719376","heartbeat_front_addr":"172.21.15.162:6811/3689719376","state":["exists","up"]},{"osd":3,"uuid":"2e6b9f0e-7a18-4cf8-9331-1a973fadc001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6813","nonce":604971011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":604971011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":604971011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6815","nonce":604971011}]},"public_addr":"172.21.15.162:6813/604971011","cluster_addr":"172.21.15.162:6814/604971011","heartbeat_back_addr":"172.21.15.162:6816/604971011","heartbeat_front_addr":"172.21.15.162:6815/604971011","state":["exists","up"]},{"osd":4,"uuid":"c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3344263388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":3344263388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":3344263388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3344263388}]},"public_addr":"172.21.15.181:6800/3344263388","cluster_addr":"172.21.15.181:6801/3344263388","heartbeat_back_addr":"172.21.15.181:6803/3344263388","heartbeat_front_addr":"172.21.15.181:6802/3344263388","state":["exists","up"]},{"osd":5,"uuid":"6aa2bd68-7f4b-4973-ac49-7fbc57c1a723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2212296748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":2212296748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":2212296748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2212296748}]},"public_addr":"172.21.15.181:6804/2212296748","cluster_addr":"172.21.15.181:6805/2212296748","heartbeat_back_addr":"172.21.15.181:6807/2212296748","heartbeat_front_addr":"172.21.15.181:6806/2212296748","state":["exists","up"]},{"osd":6,"uuid":"791ba814-5284-432e-addc-d6b7f5f3d474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1506640341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1506640341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1506640341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1506640341}]},"public_addr":"172.21.15.181:6808/1506640341","cluster_addr":"172.21.15.181:6809/1506640341","heartbeat_back_addr":"172.21.15.181:6811/1506640341","heartbeat_front_addr":"172.21.15.181:6810/1506640341","state":["exists","up"]},{"osd":7,"uuid":"1f369c65-e663-4eb2-87a5-f2900f15b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1964192772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":1964192772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":1964192772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1964192772}]},"public_addr":"172.21.15.181:6812/1964192772","cluster_addr":"172.21.15.181:6813/1964192772","heartbeat_back_addr":"172.21.15.181:6815/1964192772","heartbeat_front_addr":"172.21.15.181:6814/1964192772","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:38:14.232055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:52.126669+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:53.397185+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:11.145930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:28.458200+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:44.566925+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:38:09.041989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:31.903163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.162:6800/1909610488":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/2370739667":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/1502590187":"2024-02-14T00:32:47.469889+0000","172.21.15.162:6800/4121673848":"2024-02-14T00:32:47.469889+0000","172.21.15.162:0/2412382714":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/3041562701":"2024-02-14T00:32:47.469889+0000","172.21.15.162:6800/2775563314":"2024-02-14T00:32:24.380625+0000","172.21.15.162:0/3135632190":"2024-02-14T00:32:24.380625+0000","172.21.15.162:0/197307636":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/2653247729":"2024-02-14T00:32:24.380625+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-13T00:38:16.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:15.535+0000 7fc0f06aa700 1 librados: shutdown 2024-02-13T00:38:16.648 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:15.167010+0000 mgr.y (mgr.14544) 17 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:16.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:15.167010+0000 mgr.y (mgr.14544) 17 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:16.671 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-13T00:35:34.545390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-13T00:37:39.459980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-13T00:37:39.740219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-13T00:37:41.489292+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-13T00:37:43.466304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-13T00:37:45.542442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}] 2024-02-13T00:38:16.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool get device_health_metrics pg_num 2024-02-13T00:38:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:15.167010+0000 mgr.y (mgr.14544) 17 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:17.423 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:17.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:16.160033+0000 mgr.y (mgr.14544) 18 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:17.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:16.160033+0000 mgr.y (mgr.14544) 18 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:16.160033+0000 mgr.y (mgr.14544) 18 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:18.829 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:18 smithi162 systemd[1]: Starting Ceph alertmanager.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:38:19.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.651+0000 7f0d633d1700 1 -- 172.21.15.162:0/1327891215 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d5c0f9b70 msgr2=0x7f0d5c0f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:19.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.651+0000 7f0d633d1700 1 --2- 172.21.15.162:0/1327891215 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d5c0f9b70 0x7f0d5c0f9f50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0d580047b0 tx=0x7f0d58009cb0).stop 2024-02-13T00:38:19.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.652+0000 7f0d633d1700 1 -- 172.21.15.162:0/1327891215 shutdown_connections 2024-02-13T00:38:19.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.652+0000 7f0d633d1700 1 --2- 172.21.15.162:0/1327891215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d5c0f8500 0x7f0d5c104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:19.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.652+0000 7f0d633d1700 1 --2- 172.21.15.162:0/1327891215 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d5c0f9b70 0x7f0d5c0f9f50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:19.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.652+0000 7f0d633d1700 1 --2- 172.21.15.162:0/1327891215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d5c0fa440 0x7f0d5c0f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:19.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.652+0000 7f0d633d1700 1 -- 172.21.15.162:0/1327891215 >> 172.21.15.162:0/1327891215 conn(0x7f0d5c0f38c0 msgr2=0x7f0d5c0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:19.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.652+0000 7f0d633d1700 1 -- 172.21.15.162:0/1327891215 shutdown_connections 2024-02-13T00:38:19.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.652+0000 7f0d633d1700 1 -- 172.21.15.162:0/1327891215 wait complete. 2024-02-13T00:38:19.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d633d1700 1 librados: starting msgr at 2024-02-13T00:38:19.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d633d1700 1 librados: starting objecter 2024-02-13T00:38:19.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d633d1700 1 Processor -- start 2024-02-13T00:38:19.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d633d1700 1 -- start start 2024-02-13T00:38:19.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d633d1700 1 librados: setting wanted keys 2024-02-13T00:38:19.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d633d1700 1 librados: calling monclient init 2024-02-13T00:38:19.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d633d1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d5c0f8500 0x7f0d5c10e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:19.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d633d1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d5c0f9b70 0x7f0d5c10e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:19.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d633d1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d5c0fa440 0x7f0d5c108410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:19.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d633d1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c108900 con 0x7f0d5c0fa440 2024-02-13T00:38:19.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d633d1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c108a70 con 0x7f0d5c0f8500 2024-02-13T00:38:19.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d633d1700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c108c30 con 0x7f0d5c0f9b70 2024-02-13T00:38:19.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d6196e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d5c0fa440 0x7f0d5c108410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:19.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d6096c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d5c0f9b70 0x7f0d5c10e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:19.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d6196e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d5c0fa440 0x7f0d5c108410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:37566/0 (socket says 172.21.15.162:37566) 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d6096c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d5c0f9b70 0x7f0d5c10e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:39964/0 (socket says 172.21.15.162:39964) 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.653+0000 7f0d6196e700 1 -- 172.21.15.162:0/3852533948 learned_addr learned my addr 172.21.15.162:0/3852533948 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d6116d700 1 --2- 172.21.15.162:0/3852533948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d5c0f8500 0x7f0d5c10e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d6196e700 1 -- 172.21.15.162:0/3852533948 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d5c0f9b70 msgr2=0x7f0d5c10e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d6196e700 1 --2- 172.21.15.162:0/3852533948 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d5c0f9b70 0x7f0d5c10e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d6196e700 1 -- 172.21.15.162:0/3852533948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d5c0f8500 msgr2=0x7f0d5c10e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d6196e700 1 --2- 172.21.15.162:0/3852533948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d5c0f8500 0x7f0d5c10e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d6196e700 1 -- 172.21.15.162:0/3852533948 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c005040 con 0x7f0d5c0fa440 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d6096c700 1 --2- 172.21.15.162:0/3852533948 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d5c0f9b70 0x7f0d5c10e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d6196e700 1 --2- 172.21.15.162:0/3852533948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d5c0fa440 0x7f0d5c108410 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f0d5400d970 tx=0x7f0d54004e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d527fc700 10 client.?.objecter ms_handle_connect 0x7f0d5c0fa440 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d527fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:19.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d527fc700 1 -- 172.21.15.162:0/3852533948 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5400de60 con 0x7f0d5c0fa440 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d527fc700 1 -- 172.21.15.162:0/3852533948 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d5400f040 con 0x7f0d5c0fa440 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d633d1700 1 -- 172.21.15.162:0/3852533948 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d58005040 con 0x7f0d5c0fa440 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d527fc700 1 -- 172.21.15.162:0/3852533948 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d540036c0 con 0x7f0d5c0fa440 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d633d1700 10 client.14583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d633d1700 1 -- 172.21.15.162:0/3852533948 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c109220 con 0x7f0d5c0fa440 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.654+0000 7f0d633d1700 1 librados: init done 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.655+0000 7f0d633d1700 1 -- 172.21.15.162:0/3852533948 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5c0475e0 con 0x7f0d5c0fa440 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.655+0000 7f0d527fc700 1 -- 172.21.15.162:0/3852533948 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0d54003ab0 con 0x7f0d5c0fa440 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.656+0000 7f0d527fc700 1 --2- 172.21.15.162:0/3852533948 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0d48062c90 0x7f0d48065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.656+0000 7f0d527fc700 1 -- 172.21.15.162:0/3852533948 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f0d5407c570 con 0x7f0d5c0fa440 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.656+0000 7f0d527fc700 10 client.14583.objecter ms_dispatch 0x7f0d5c001550 osd_map(74..74 src has 1..74) v4 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.656+0000 7f0d527fc700 3 client.14583.objecter handle_osd_map got epochs [74,74] > 0 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.656+0000 7f0d527fc700 3 client.14583.objecter handle_osd_map decoding full epoch 74 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.656+0000 7f0d6116d700 1 --2- 172.21.15.162:0/3852533948 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0d48062c90 0x7f0d48065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.656+0000 7f0d527fc700 20 client.14583.objecter dump_active .. 0 homeless 2024-02-13T00:38:19.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.656+0000 7f0d527fc700 10 client.14583.objecter ms_handle_connect 0x7f0d48062c90 2024-02-13T00:38:19.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.656+0000 7f0d6116d700 1 --2- 172.21.15.162:0/3852533948 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0d48062c90 0x7f0d48065150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f0d58005010 tx=0x7f0d58007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:19.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.660+0000 7f0d527fc700 1 -- 172.21.15.162:0/3852533948 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0d54048f60 con 0x7f0d5c0fa440 2024-02-13T00:38:19.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.824+0000 7f0d633d1700 1 -- 172.21.15.162:0/3852533948 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f0d5c1098f0 con 0x7f0d5c0fa440 2024-02-13T00:38:19.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.824+0000 7f0d527fc700 1 -- 172.21.15.162:0/3852533948 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v74) v1 ==== 110+0+10 (secure 0 0 0) 0x7f0d54050250 con 0x7f0d5c0fa440 2024-02-13T00:38:19.828 INFO:teuthology.orchestra.run.smithi162.stdout:pg_num: 1 2024-02-13T00:38:19.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:18.160486+0000 mgr.y (mgr.14544) 19 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-02-13T00:38:19.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:19.215345+0000 mon.a (mon.0) 654 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-13T00:38:19.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:18.160486+0000 mgr.y (mgr.14544) 19 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-02-13T00:38:19.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:19.215345+0000 mon.a (mon.0) 654 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-13T00:38:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.826+0000 7f0d633d1700 10 librados: watch_flush enter 2024-02-13T00:38:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.826+0000 7f0d633d1700 10 librados: watch_flush exit 2024-02-13T00:38:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.826+0000 7f0d633d1700 20 client.14583.objecter shutdown clearing up homeless session... 2024-02-13T00:38:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.826+0000 7f0d633d1700 10 client.14583.objecter successfully canceled tick 2024-02-13T00:38:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.826+0000 7f0d633d1700 1 -- 172.21.15.162:0/3852533948 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0d48062c90 msgr2=0x7f0d48065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.826+0000 7f0d633d1700 1 --2- 172.21.15.162:0/3852533948 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0d48062c90 0x7f0d48065150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f0d58005010 tx=0x7f0d58007f90).stop 2024-02-13T00:38:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.826+0000 7f0d633d1700 1 -- 172.21.15.162:0/3852533948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d5c0fa440 msgr2=0x7f0d5c108410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.826+0000 7f0d633d1700 1 --2- 172.21.15.162:0/3852533948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d5c0fa440 0x7f0d5c108410 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f0d5400d970 tx=0x7f0d54004e60).stop 2024-02-13T00:38:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.827+0000 7f0d633d1700 1 -- 172.21.15.162:0/3852533948 shutdown_connections 2024-02-13T00:38:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.827+0000 7f0d633d1700 1 --2- 172.21.15.162:0/3852533948 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0d48062c90 0x7f0d48065150 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:19.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.827+0000 7f0d633d1700 1 --2- 172.21.15.162:0/3852533948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d5c0fa440 0x7f0d5c108410 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:19.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.827+0000 7f0d633d1700 1 --2- 172.21.15.162:0/3852533948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d5c0f8500 0x7f0d5c10e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:19.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.827+0000 7f0d633d1700 1 --2- 172.21.15.162:0/3852533948 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d5c0f9b70 0x7f0d5c10e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:19.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.827+0000 7f0d633d1700 1 -- 172.21.15.162:0/3852533948 >> 172.21.15.162:0/3852533948 conn(0x7f0d5c0f38c0 msgr2=0x7f0d5c0f54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:19.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.827+0000 7f0d633d1700 1 -- 172.21.15.162:0/3852533948 shutdown_connections 2024-02-13T00:38:19.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.827+0000 7f0d633d1700 1 -- 172.21.15.162:0/3852533948 wait complete. 2024-02-13T00:38:19.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:19.827+0000 7f0d633d1700 1 librados: shutdown 2024-02-13T00:38:20.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:18.160486+0000 mgr.y (mgr.14544) 19 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-02-13T00:38:20.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:19.215345+0000 mon.a (mon.0) 654 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-13T00:38:20.764 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:19.825597+0000 mon.a (mon.0) 655 : audit [DBG] from='client.? 172.21.15.162:0/3852533948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-13T00:38:20.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:19.825597+0000 mon.a (mon.0) 655 : audit [DBG] from='client.? 172.21.15.162:0/3852533948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-13T00:38:20.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool get .rgw.root pg_num 2024-02-13T00:38:21.049 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:19.825597+0000 mon.a (mon.0) 655 : audit [DBG] from='client.? 172.21.15.162:0/3852533948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-13T00:38:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:21.079 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[68995]: level=info ts=2024-02-13T00:38:20.901Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-13T00:38:21.079 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[68995]: level=info ts=2024-02-13T00:38:20.901Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-13T00:38:21.079 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[68995]: level=info ts=2024-02-13T00:38:20.902Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.162 port=9094 2024-02-13T00:38:21.079 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[68995]: level=info ts=2024-02-13T00:38:20.903Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-13T00:38:21.079 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[68995]: level=info ts=2024-02-13T00:38:20.929Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T00:38:21.079 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[68995]: level=info ts=2024-02-13T00:38:20.929Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T00:38:21.079 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[68995]: level=info ts=2024-02-13T00:38:20.931Z caller=main.go:518 msg=Listening address=:9093 2024-02-13T00:38:21.080 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[68995]: level=info ts=2024-02-13T00:38:20.931Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-13T00:38:21.080 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:20 smithi162 bash[68939]: 8aec9f20e0f13dbda99cf24b766216bd0dabf2e306e48a04a7f13b92c3d6a2f6 2024-02-13T00:38:21.080 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:20 smithi162 systemd[1]: Started Ceph alertmanager.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:38:21.672 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:20.161007+0000 mgr.y (mgr.14544) 20 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:38:21.672 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:21.052351+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:21.673 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:20.161007+0000 mgr.y (mgr.14544) 20 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:38:21.673 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:21.052351+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:21.966 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:21 smithi162 systemd[1]: Starting Ceph node-exporter.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:38:22.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:20.161007+0000 mgr.y (mgr.14544) 20 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:38:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:21.052351+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:22.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.768+0000 7f7c76945700 1 -- 172.21.15.162:0/294980281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c700fe590 msgr2=0x7f7c700f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:22.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.768+0000 7f7c76945700 1 --2- 172.21.15.162:0/294980281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c700fe590 0x7f7c700f7dc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7c64004870 tx=0x7f7c64009cb0).stop 2024-02-13T00:38:22.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.768+0000 7f7c76945700 1 -- 172.21.15.162:0/294980281 shutdown_connections 2024-02-13T00:38:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.768+0000 7f7c76945700 1 --2- 172.21.15.162:0/294980281 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c700f84a0 0x7f7c70104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.768+0000 7f7c76945700 1 --2- 172.21.15.162:0/294980281 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c700fdc30 0x7f7c700fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.768+0000 7f7c76945700 1 --2- 172.21.15.162:0/294980281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c700fe590 0x7f7c700f7dc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.768+0000 7f7c76945700 1 -- 172.21.15.162:0/294980281 >> 172.21.15.162:0/294980281 conn(0x7f7c700f3880 msgr2=0x7f7c700f5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.768+0000 7f7c76945700 1 -- 172.21.15.162:0/294980281 shutdown_connections 2024-02-13T00:38:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.768+0000 7f7c76945700 1 -- 172.21.15.162:0/294980281 wait complete. 2024-02-13T00:38:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.768+0000 7f7c76945700 1 librados: starting msgr at 2024-02-13T00:38:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.768+0000 7f7c76945700 1 librados: starting objecter 2024-02-13T00:38:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.768+0000 7f7c76945700 1 Processor -- start 2024-02-13T00:38:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c76945700 1 -- start start 2024-02-13T00:38:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c76945700 1 librados: setting wanted keys 2024-02-13T00:38:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c76945700 1 librados: calling monclient init 2024-02-13T00:38:22.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c76945700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c700f84a0 0x7f7c7006b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:22.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c76945700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c700fdc30 0x7f7c7006b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:22.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c76945700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c700fe590 0x7f7c7006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:22.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c76945700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70116550 con 0x7f7c700fe590 2024-02-13T00:38:22.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c76945700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c701166c0 con 0x7f7c700f84a0 2024-02-13T00:38:22.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c76945700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70116880 con 0x7f7c700fdc30 2024-02-13T00:38:22.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c6f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c700fdc30 0x7f7c7006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:22.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c74ee2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c700fe590 0x7f7c7006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:22.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c6f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c700fdc30 0x7f7c7006b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:39980/0 (socket says 172.21.15.162:39980) 2024-02-13T00:38:22.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c6f7fe700 1 -- 172.21.15.162:0/1048315260 learned_addr learned my addr 172.21.15.162:0/1048315260 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:22.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c74ee2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c700fe590 0x7f7c7006bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:37594/0 (socket says 172.21.15.162:37594) 2024-02-13T00:38:22.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c6ffff700 1 --2- 172.21.15.162:0/1048315260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c700f84a0 0x7f7c7006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:22.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c74ee2700 1 -- 172.21.15.162:0/1048315260 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c700fdc30 msgr2=0x7f7c7006b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c74ee2700 1 --2- 172.21.15.162:0/1048315260 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c700fdc30 0x7f7c7006b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c74ee2700 1 -- 172.21.15.162:0/1048315260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c700f84a0 msgr2=0x7f7c7006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c74ee2700 1 --2- 172.21.15.162:0/1048315260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c700f84a0 0x7f7c7006b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c74ee2700 1 -- 172.21.15.162:0/1048315260 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c680097c0 con 0x7f7c700fe590 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c6f7fe700 1 --2- 172.21.15.162:0/1048315260 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c700fdc30 0x7f7c7006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c74ee2700 1 --2- 172.21.15.162:0/1048315260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c700fe590 0x7f7c7006bb50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f7c68001cb0 tx=0x7f7c68004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c6d7fa700 10 client.?.objecter ms_handle_connect 0x7f7c700fe590 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.769+0000 7f7c6d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.770+0000 7f7c6d7fa700 1 -- 172.21.15.162:0/1048315260 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c680079b0 con 0x7f7c700fe590 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.770+0000 7f7c6d7fa700 1 -- 172.21.15.162:0/1048315260 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c680090d0 con 0x7f7c700fe590 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.770+0000 7f7c76945700 1 -- 172.21.15.162:0/1048315260 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c64005040 con 0x7f7c700fe590 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.770+0000 7f7c6d7fa700 1 -- 172.21.15.162:0/1048315260 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6800fc20 con 0x7f7c700fe590 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.770+0000 7f7c76945700 10 client.14589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:22.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.770+0000 7f7c76945700 1 -- 172.21.15.162:0/1048315260 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c70116ef0 con 0x7f7c700fe590 2024-02-13T00:38:22.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.770+0000 7f7c76945700 1 librados: init done 2024-02-13T00:38:22.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.771+0000 7f7c76945700 1 -- 172.21.15.162:0/1048315260 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7005fae0 con 0x7f7c700fe590 2024-02-13T00:38:22.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.771+0000 7f7c6d7fa700 1 -- 172.21.15.162:0/1048315260 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f7c6800fd80 con 0x7f7c700fe590 2024-02-13T00:38:22.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.771+0000 7f7c6d7fa700 1 --2- 172.21.15.162:0/1048315260 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7c58062c90 0x7f7c58065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:22.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.772+0000 7f7c6d7fa700 1 -- 172.21.15.162:0/1048315260 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f7c6807d230 con 0x7f7c700fe590 2024-02-13T00:38:22.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.772+0000 7f7c6d7fa700 10 client.14589.objecter ms_dispatch 0x7f7c70001550 osd_map(74..74 src has 1..74) v4 2024-02-13T00:38:22.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.772+0000 7f7c6ffff700 1 --2- 172.21.15.162:0/1048315260 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7c58062c90 0x7f7c58065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:22.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.772+0000 7f7c6d7fa700 3 client.14589.objecter handle_osd_map got epochs [74,74] > 0 2024-02-13T00:38:22.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.772+0000 7f7c6d7fa700 3 client.14589.objecter handle_osd_map decoding full epoch 74 2024-02-13T00:38:22.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.772+0000 7f7c6d7fa700 20 client.14589.objecter dump_active .. 0 homeless 2024-02-13T00:38:22.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.772+0000 7f7c6d7fa700 10 client.14589.objecter ms_handle_connect 0x7f7c58062c90 2024-02-13T00:38:22.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.772+0000 7f7c6ffff700 1 --2- 172.21.15.162:0/1048315260 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7c58062c90 0x7f7c58065150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f7c600041d0 tx=0x7f7c60009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:22.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.776+0000 7f7c6d7fa700 1 -- 172.21.15.162:0/1048315260 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7c68050f10 con 0x7f7c700fe590 2024-02-13T00:38:22.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:38:21.057092+0000 mgr.y (mgr.14544) 21 : cephadm [INF] Deploying daemon node-exporter.a on smithi162 2024-02-13T00:38:22.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:38:21.057092+0000 mgr.y (mgr.14544) 21 : cephadm [INF] Deploying daemon node-exporter.a on smithi162 2024-02-13T00:38:22.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.953+0000 7f7c76945700 1 -- 172.21.15.162:0/1048315260 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"} v 0) v1 -- 0x7f7c7006c650 con 0x7f7c700fe590 2024-02-13T00:38:22.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.953+0000 7f7c6d7fa700 1 -- 172.21.15.162:0/1048315260 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]=0 v74) v1 ==== 98+0+11 (secure 0 0 0) 0x7f7c68050b00 con 0x7f7c700fe590 2024-02-13T00:38:22.958 INFO:teuthology.orchestra.run.smithi162.stdout:pg_num: 32 2024-02-13T00:38:22.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 10 librados: watch_flush enter 2024-02-13T00:38:22.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 10 librados: watch_flush exit 2024-02-13T00:38:22.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 20 client.14589.objecter shutdown clearing up homeless session... 2024-02-13T00:38:22.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 10 client.14589.objecter successfully canceled tick 2024-02-13T00:38:22.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 -- 172.21.15.162:0/1048315260 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7c58062c90 msgr2=0x7f7c58065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:22.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 --2- 172.21.15.162:0/1048315260 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7c58062c90 0x7f7c58065150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f7c600041d0 tx=0x7f7c60009040).stop 2024-02-13T00:38:22.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 -- 172.21.15.162:0/1048315260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c700fe590 msgr2=0x7f7c7006bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:22.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 --2- 172.21.15.162:0/1048315260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c700fe590 0x7f7c7006bb50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f7c68001cb0 tx=0x7f7c68004c50).stop 2024-02-13T00:38:22.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 -- 172.21.15.162:0/1048315260 shutdown_connections 2024-02-13T00:38:22.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 --2- 172.21.15.162:0/1048315260 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7c58062c90 0x7f7c58065150 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:22.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 --2- 172.21.15.162:0/1048315260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c700fe590 0x7f7c7006bb50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:22.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 --2- 172.21.15.162:0/1048315260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c700f84a0 0x7f7c7006b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:22.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 --2- 172.21.15.162:0/1048315260 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c700fdc30 0x7f7c7006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:22.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 -- 172.21.15.162:0/1048315260 >> 172.21.15.162:0/1048315260 conn(0x7f7c700f3880 msgr2=0x7f7c700f5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:22.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 -- 172.21.15.162:0/1048315260 shutdown_connections 2024-02-13T00:38:22.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 -- 172.21.15.162:0/1048315260 wait complete. 2024-02-13T00:38:22.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:22.955+0000 7f7c76945700 1 librados: shutdown 2024-02-13T00:38:23.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:38:21.057092+0000 mgr.y (mgr.14544) 21 : cephadm [INF] Deploying daemon node-exporter.a on smithi162 2024-02-13T00:38:23.307 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:22 smithi162 bash[69400]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-13T00:38:23.308 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[68995]: level=info ts=2024-02-13T00:38:22.903Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000221255s 2024-02-13T00:38:23.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool get iscsi pg_num 2024-02-13T00:38:23.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:22.161455+0000 mgr.y (mgr.14544) 22 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:38:23.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:22.955455+0000 mon.a (mon.0) 657 : audit [DBG] from='client.? 172.21.15.162:0/1048315260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-13T00:38:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:22.161455+0000 mgr.y (mgr.14544) 22 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:38:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:22.955455+0000 mon.a (mon.0) 657 : audit [DBG] from='client.? 172.21.15.162:0/1048315260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-13T00:38:23.830 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:23 smithi162 bash[69400]: Getting image source signatures 2024-02-13T00:38:23.830 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:23 smithi162 bash[69400]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-13T00:38:23.830 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:23 smithi162 bash[69400]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-13T00:38:23.830 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:23 smithi162 bash[69400]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-13T00:38:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:24.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:22.161455+0000 mgr.y (mgr.14544) 22 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:38:24.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:22.955455+0000 mon.a (mon.0) 657 : audit [DBG] from='client.? 172.21.15.162:0/1048315260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-13T00:38:25.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:38:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:38:25.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:24.161795+0000 mgr.y (mgr.14544) 23 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-02-13T00:38:25.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:24.323054+0000 mon.a (mon.0) 658 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:25.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:24.161795+0000 mgr.y (mgr.14544) 23 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-02-13T00:38:25.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:24.323054+0000 mon.a (mon.0) 658 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:25.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:24.161795+0000 mgr.y (mgr.14544) 23 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-02-13T00:38:25.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:24.323054+0000 mon.a (mon.0) 658 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:26.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.052+0000 7f520ba69700 1 -- 172.21.15.162:0/4125059391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52040f8500 msgr2=0x7f5204104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:26.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.052+0000 7f520ba69700 1 --2- 172.21.15.162:0/4125059391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52040f8500 0x7f5204104bd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f51fc009c30 tx=0x7f51fc004c50).stop 2024-02-13T00:38:26.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.052+0000 7f520ba69700 1 -- 172.21.15.162:0/4125059391 shutdown_connections 2024-02-13T00:38:26.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.052+0000 7f520ba69700 1 --2- 172.21.15.162:0/4125059391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52040f8500 0x7f5204104bd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:26.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.052+0000 7f520ba69700 1 --2- 172.21.15.162:0/4125059391 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52040f9b70 0x7f52040f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:26.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.052+0000 7f520ba69700 1 --2- 172.21.15.162:0/4125059391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52040fa440 0x7f52040f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:26.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.052+0000 7f520ba69700 1 -- 172.21.15.162:0/4125059391 >> 172.21.15.162:0/4125059391 conn(0x7f52040f38c0 msgr2=0x7f52040f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:26.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.052+0000 7f520ba69700 1 -- 172.21.15.162:0/4125059391 shutdown_connections 2024-02-13T00:38:26.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.052+0000 7f520ba69700 1 -- 172.21.15.162:0/4125059391 wait complete. 2024-02-13T00:38:26.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520ba69700 1 librados: starting msgr at 2024-02-13T00:38:26.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520ba69700 1 librados: starting objecter 2024-02-13T00:38:26.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520ba69700 1 Processor -- start 2024-02-13T00:38:26.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520ba69700 1 -- start start 2024-02-13T00:38:26.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520ba69700 1 librados: setting wanted keys 2024-02-13T00:38:26.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520ba69700 1 librados: calling monclient init 2024-02-13T00:38:26.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520ba69700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52040f8500 0x7f52041102c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:26.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520ba69700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52040f9b70 0x7f52041107b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:26.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520ba69700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52040fa440 0x7f520410a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:26.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520ba69700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520410a950 con 0x7f52040f8500 2024-02-13T00:38:26.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520ba69700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520410aac0 con 0x7f52040fa440 2024-02-13T00:38:26.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520ba69700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520410ac80 con 0x7f52040f9b70 2024-02-13T00:38:26.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f5209805700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52040f8500 0x7f52041102c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:26.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f5209805700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52040f8500 0x7f52041102c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:54738/0 (socket says 172.21.15.162:54738) 2024-02-13T00:38:26.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f5209805700 1 -- 172.21.15.162:0/4077092214 learned_addr learned my addr 172.21.15.162:0/4077092214 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:26.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f5209004700 1 --2- 172.21.15.162:0/4077092214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52040f9b70 0x7f52041107b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:26.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.053+0000 7f520a006700 1 --2- 172.21.15.162:0/4077092214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52040fa440 0x7f520410a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:26.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f5209805700 1 -- 172.21.15.162:0/4077092214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52040f9b70 msgr2=0x7f52041107b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:26.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f5209805700 1 --2- 172.21.15.162:0/4077092214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52040f9b70 0x7f52041107b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:26.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f5209805700 1 -- 172.21.15.162:0/4077092214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52040fa440 msgr2=0x7f520410a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:26.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f5209805700 1 --2- 172.21.15.162:0/4077092214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52040fa440 0x7f520410a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:26.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f5209805700 1 -- 172.21.15.162:0/4077092214 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc0097c0 con 0x7f52040f8500 2024-02-13T00:38:26.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f520a006700 1 --2- 172.21.15.162:0/4077092214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52040fa440 0x7f520410a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:38:26.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f5209805700 1 --2- 172.21.15.162:0/4077092214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52040f8500 0x7f52041102c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f520000e990 tx=0x7f520000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:26.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f51faf78700 10 client.?.objecter ms_handle_connect 0x7f52040f8500 2024-02-13T00:38:26.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f51faf78700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:26.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f51faf78700 1 -- 172.21.15.162:0/4077092214 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52000046e0 con 0x7f52040f8500 2024-02-13T00:38:26.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f51faf78700 1 -- 172.21.15.162:0/4077092214 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f520000f040 con 0x7f52040f8500 2024-02-13T00:38:26.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f520ba69700 1 -- 172.21.15.162:0/4077092214 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520410af00 con 0x7f52040f8500 2024-02-13T00:38:26.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f51faf78700 1 -- 172.21.15.162:0/4077092214 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520000c840 con 0x7f52040f8500 2024-02-13T00:38:26.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f520ba69700 10 client.14595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:26.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f520ba69700 1 -- 172.21.15.162:0/4077092214 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52041170f0 con 0x7f52040f8500 2024-02-13T00:38:26.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.054+0000 7f520ba69700 1 librados: init done 2024-02-13T00:38:26.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.055+0000 7f520ba69700 1 -- 172.21.15.162:0/4077092214 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52040475e0 con 0x7f52040f8500 2024-02-13T00:38:26.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.055+0000 7f51faf78700 1 -- 172.21.15.162:0/4077092214 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f520000cc70 con 0x7f52040f8500 2024-02-13T00:38:26.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.056+0000 7f51faf78700 1 --2- 172.21.15.162:0/4077092214 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51f0062c90 0x7f51f0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:26.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.056+0000 7f5209004700 1 --2- 172.21.15.162:0/4077092214 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51f0062c90 0x7f51f0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:26.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.056+0000 7f51faf78700 1 -- 172.21.15.162:0/4077092214 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f520007c5d0 con 0x7f52040f8500 2024-02-13T00:38:26.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.056+0000 7f51faf78700 10 client.14595.objecter ms_dispatch 0x7f5204001550 osd_map(74..74 src has 1..74) v4 2024-02-13T00:38:26.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.056+0000 7f51faf78700 3 client.14595.objecter handle_osd_map got epochs [74,74] > 0 2024-02-13T00:38:26.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.056+0000 7f51faf78700 3 client.14595.objecter handle_osd_map decoding full epoch 74 2024-02-13T00:38:26.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.056+0000 7f51faf78700 20 client.14595.objecter dump_active .. 0 homeless 2024-02-13T00:38:26.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.056+0000 7f51faf78700 10 client.14595.objecter ms_handle_connect 0x7f51f0062c90 2024-02-13T00:38:26.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.056+0000 7f5209004700 1 --2- 172.21.15.162:0/4077092214 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51f0062c90 0x7f51f0065150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f520410bc00 tx=0x7f51f4009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:26.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.060+0000 7f51faf78700 1 -- 172.21.15.162:0/4077092214 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5200049f60 con 0x7f52040f8500 2024-02-13T00:38:26.084 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:25 smithi162 bash[69400]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-13T00:38:26.084 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:25 smithi162 bash[69400]: Writing manifest to image destination 2024-02-13T00:38:26.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.225+0000 7f520ba69700 1 -- 172.21.15.162:0/4077092214 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"} v 0) v1 -- 0x7f520410b940 con 0x7f52040f8500 2024-02-13T00:38:26.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.226+0000 7f51faf78700 1 -- 172.21.15.162:0/4077092214 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]=0 v74) v1 ==== 94+0+10 (secure 0 0 0) 0x7f52000502b0 con 0x7f52040f8500 2024-02-13T00:38:26.230 INFO:teuthology.orchestra.run.smithi162.stdout:pg_num: 3 2024-02-13T00:38:26.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 10 librados: watch_flush enter 2024-02-13T00:38:26.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 10 librados: watch_flush exit 2024-02-13T00:38:26.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 20 client.14595.objecter shutdown clearing up homeless session... 2024-02-13T00:38:26.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 10 client.14595.objecter successfully canceled tick 2024-02-13T00:38:26.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 1 -- 172.21.15.162:0/4077092214 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51f0062c90 msgr2=0x7f51f0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:26.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 1 --2- 172.21.15.162:0/4077092214 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51f0062c90 0x7f51f0065150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f520410bc00 tx=0x7f51f4009040).stop 2024-02-13T00:38:26.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 1 -- 172.21.15.162:0/4077092214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52040f8500 msgr2=0x7f52041102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:26.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 1 --2- 172.21.15.162:0/4077092214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52040f8500 0x7f52041102c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f520000e990 tx=0x7f520000b040).stop 2024-02-13T00:38:26.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 1 -- 172.21.15.162:0/4077092214 shutdown_connections 2024-02-13T00:38:26.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 1 --2- 172.21.15.162:0/4077092214 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51f0062c90 0x7f51f0065150 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:26.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 1 --2- 172.21.15.162:0/4077092214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52040fa440 0x7f520410a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:26.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 1 --2- 172.21.15.162:0/4077092214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52040f8500 0x7f52041102c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:26.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 1 --2- 172.21.15.162:0/4077092214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52040f9b70 0x7f52041107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:26.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 1 -- 172.21.15.162:0/4077092214 >> 172.21.15.162:0/4077092214 conn(0x7f52040f38c0 msgr2=0x7f5204102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:26.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 1 -- 172.21.15.162:0/4077092214 shutdown_connections 2024-02-13T00:38:26.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.227+0000 7f520ba69700 1 -- 172.21.15.162:0/4077092214 wait complete. 2024-02-13T00:38:26.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:26.228+0000 7f520ba69700 1 librados: shutdown 2024-02-13T00:38:26.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:25.177900+0000 mgr.y (mgr.14544) 24 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:26.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:26.227609+0000 mon.a (mon.0) 659 : audit [DBG] from='client.? 172.21.15.162:0/4077092214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-13T00:38:26.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:25.177900+0000 mgr.y (mgr.14544) 24 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:26.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:26.227609+0000 mon.a (mon.0) 659 : audit [DBG] from='client.? 172.21.15.162:0/4077092214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-13T00:38:26.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:25.177900+0000 mgr.y (mgr.14544) 24 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:26.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:26.227609+0000 mon.a (mon.0) 659 : audit [DBG] from='client.? 172.21.15.162:0/4077092214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-13T00:38:27.080 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.675Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-13T00:38:27.080 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.675Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-13T00:38:27.081 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.676Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-13T00:38:27.081 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.676Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-13T00:38:27.081 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-13T00:38:27.081 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=arp 2024-02-13T00:38:27.081 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-13T00:38:27.081 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-13T00:38:27.082 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-13T00:38:27.082 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-13T00:38:27.082 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-13T00:38:27.082 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-13T00:38:27.082 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-13T00:38:27.082 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-13T00:38:27.082 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=edac 2024-02-13T00:38:27.083 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-13T00:38:27.083 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-13T00:38:27.083 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-13T00:38:27.083 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-13T00:38:27.083 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-13T00:38:27.083 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-13T00:38:27.084 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-13T00:38:27.084 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-13T00:38:27.084 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-13T00:38:27.084 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-13T00:38:27.084 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-13T00:38:27.084 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-13T00:38:27.085 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-13T00:38:27.085 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-13T00:38:27.085 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-13T00:38:27.085 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-13T00:38:27.085 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=os 2024-02-13T00:38:27.085 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-13T00:38:27.085 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-13T00:38:27.086 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-13T00:38:27.086 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-13T00:38:27.086 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-13T00:38:27.086 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-13T00:38:27.086 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=stat 2024-02-13T00:38:27.086 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-13T00:38:27.086 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-13T00:38:27.087 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-13T00:38:27.087 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=time 2024-02-13T00:38:27.087 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-13T00:38:27.087 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=uname 2024-02-13T00:38:27.087 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-13T00:38:27.087 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-13T00:38:27.088 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-13T00:38:27.088 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-13T00:38:27.088 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a[69660]: ts=2024-02-13T00:38:26.677Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-13T00:38:27.088 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 bash[69400]: 86785512b37b994f232c1a4c65d2cc96cdc1124ea63eecee04b4515de8ae83f6 2024-02-13T00:38:27.088 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:38:26 smithi162 systemd[1]: Started Ceph node-exporter.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:38:27.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool get default.rgw.log pg_num 2024-02-13T00:38:27.505 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:27.794 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:27 smithi181 systemd[1]: Starting Ceph node-exporter.b for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:38:27.795 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:27 smithi181 bash[64042]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-13T00:38:28.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:26.162348+0000 mgr.y (mgr.14544) 25 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:38:28.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:26.811940+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:28.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:38:26.813285+0000 mgr.y (mgr.14544) 26 : cephadm [INF] Deploying daemon node-exporter.b on smithi181 2024-02-13T00:38:28.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:26.162348+0000 mgr.y (mgr.14544) 25 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:38:28.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:26.811940+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:28.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:38:26.813285+0000 mgr.y (mgr.14544) 26 : cephadm [INF] Deploying daemon node-exporter.b on smithi181 2024-02-13T00:38:28.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:26.162348+0000 mgr.y (mgr.14544) 25 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:38:28.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:26.811940+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:28.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:38:26.813285+0000 mgr.y (mgr.14544) 26 : cephadm [INF] Deploying daemon node-exporter.b on smithi181 2024-02-13T00:38:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.330+0000 7f7df67cb700 1 -- 172.21.15.162:0/3271865148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7df00f73a0 msgr2=0x7f7df01039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.330+0000 7f7df67cb700 1 --2- 172.21.15.162:0/3271865148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7df00f73a0 0x7f7df01039a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f7de8009c30 tx=0x7f7de8004c50).stop 2024-02-13T00:38:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.330+0000 7f7df67cb700 1 -- 172.21.15.162:0/3271865148 shutdown_connections 2024-02-13T00:38:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.330+0000 7f7df67cb700 1 --2- 172.21.15.162:0/3271865148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7df00f73a0 0x7f7df01039a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.330+0000 7f7df67cb700 1 --2- 172.21.15.162:0/3271865148 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7df00fcb30 0x7f7df00fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.330+0000 7f7df67cb700 1 --2- 172.21.15.162:0/3271865148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7df00fd490 0x7f7df00f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:29.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.330+0000 7f7df67cb700 1 -- 172.21.15.162:0/3271865148 >> 172.21.15.162:0/3271865148 conn(0x7f7df006eb90 msgr2=0x7f7df00f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.330+0000 7f7df67cb700 1 -- 172.21.15.162:0/3271865148 shutdown_connections 2024-02-13T00:38:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.330+0000 7f7df67cb700 1 -- 172.21.15.162:0/3271865148 wait complete. 2024-02-13T00:38:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df67cb700 1 librados: starting msgr at 2024-02-13T00:38:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df67cb700 1 librados: starting objecter 2024-02-13T00:38:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df67cb700 1 Processor -- start 2024-02-13T00:38:29.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df67cb700 1 -- start start 2024-02-13T00:38:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df67cb700 1 librados: setting wanted keys 2024-02-13T00:38:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df67cb700 1 librados: calling monclient init 2024-02-13T00:38:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df67cb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7df00f73a0 0x7f7df010ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df67cb700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7df00fcb30 0x7f7df010d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:29.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df67cb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7df00fd490 0x7f7df0107030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:29.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df67cb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df0107520 con 0x7f7df00fd490 2024-02-13T00:38:29.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df67cb700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df0107690 con 0x7f7df00f73a0 2024-02-13T00:38:29.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df67cb700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df0107850 con 0x7f7df00fcb30 2024-02-13T00:38:29.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7def7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7df00fcb30 0x7f7df010d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:29.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df4d68700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7df00fd490 0x7f7df0107030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:29.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df4d68700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7df00fd490 0x7f7df0107030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:54764/0 (socket says 172.21.15.162:54764) 2024-02-13T00:38:29.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7def7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7df00fcb30 0x7f7df010d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:53900/0 (socket says 172.21.15.162:53900) 2024-02-13T00:38:29.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.331+0000 7f7df4d68700 1 -- 172.21.15.162:0/2290128336 learned_addr learned my addr 172.21.15.162:0/2290128336 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:29.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7deffff700 1 --2- 172.21.15.162:0/2290128336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7df00f73a0 0x7f7df010ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:29.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7df4d68700 1 -- 172.21.15.162:0/2290128336 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7df00fcb30 msgr2=0x7f7df010d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:29.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7df4d68700 1 --2- 172.21.15.162:0/2290128336 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7df00fcb30 0x7f7df010d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:29.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7df4d68700 1 -- 172.21.15.162:0/2290128336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7df00f73a0 msgr2=0x7f7df010ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:29.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7df4d68700 1 --2- 172.21.15.162:0/2290128336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7df00f73a0 0x7f7df010ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:29.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7df4d68700 1 -- 172.21.15.162:0/2290128336 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de4005040 con 0x7f7df00fd490 2024-02-13T00:38:29.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7df4d68700 1 --2- 172.21.15.162:0/2290128336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7df00fd490 0x7f7df0107030 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f7de8004860 tx=0x7f7de8004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:29.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7ded7fa700 10 client.?.objecter ms_handle_connect 0x7f7df00fd490 2024-02-13T00:38:29.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7ded7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:29.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7ded7fa700 1 -- 172.21.15.162:0/2290128336 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de800e040 con 0x7f7df00fd490 2024-02-13T00:38:29.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7ded7fa700 1 -- 172.21.15.162:0/2290128336 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7de8018040 con 0x7f7df00fd490 2024-02-13T00:38:29.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7ded7fa700 1 -- 172.21.15.162:0/2290128336 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de801c3f0 con 0x7f7df00fd490 2024-02-13T00:38:29.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7df67cb700 1 -- 172.21.15.162:0/2290128336 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de80097c0 con 0x7f7df00fd490 2024-02-13T00:38:29.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7df67cb700 10 client.14601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:29.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7df67cb700 1 -- 172.21.15.162:0/2290128336 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df0107d70 con 0x7f7df00fd490 2024-02-13T00:38:29.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.332+0000 7f7df67cb700 1 librados: init done 2024-02-13T00:38:29.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.333+0000 7f7df67cb700 1 -- 172.21.15.162:0/2290128336 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df00475e0 con 0x7f7df00fd490 2024-02-13T00:38:29.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.333+0000 7f7ded7fa700 1 -- 172.21.15.162:0/2290128336 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f7de8013420 con 0x7f7df00fd490 2024-02-13T00:38:29.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.334+0000 7f7ded7fa700 1 --2- 172.21.15.162:0/2290128336 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7dd8062c90 0x7f7dd8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:29.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.334+0000 7f7ded7fa700 1 -- 172.21.15.162:0/2290128336 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f7de807d990 con 0x7f7df00fd490 2024-02-13T00:38:29.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.334+0000 7f7ded7fa700 10 client.14601.objecter ms_dispatch 0x7f7df0001550 osd_map(74..74 src has 1..74) v4 2024-02-13T00:38:29.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.334+0000 7f7ded7fa700 3 client.14601.objecter handle_osd_map got epochs [74,74] > 0 2024-02-13T00:38:29.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.334+0000 7f7ded7fa700 3 client.14601.objecter handle_osd_map decoding full epoch 74 2024-02-13T00:38:29.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.334+0000 7f7deffff700 1 --2- 172.21.15.162:0/2290128336 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7dd8062c90 0x7f7dd8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:29.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.334+0000 7f7ded7fa700 20 client.14601.objecter dump_active .. 0 homeless 2024-02-13T00:38:29.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.334+0000 7f7ded7fa700 10 client.14601.objecter ms_handle_connect 0x7f7dd8062c90 2024-02-13T00:38:29.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.334+0000 7f7deffff700 1 --2- 172.21.15.162:0/2290128336 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7dd8062c90 0x7f7dd8065150 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f7de0004de0 tx=0x7f7de000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:29.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.338+0000 7f7ded7fa700 1 -- 172.21.15.162:0/2290128336 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7de80517c0 con 0x7f7df00fd490 2024-02-13T00:38:29.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.500+0000 7f7df67cb700 1 -- 172.21.15.162:0/2290128336 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"} v 0) v1 -- 0x7f7df0106090 con 0x7f7df00fd490 2024-02-13T00:38:29.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.501+0000 7f7ded7fa700 1 -- 172.21.15.162:0/2290128336 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]=0 v74) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7de80210a0 con 0x7f7df00fd490 2024-02-13T00:38:29.505 INFO:teuthology.orchestra.run.smithi162.stdout:pg_num: 32 2024-02-13T00:38:29.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.502+0000 7f7df67cb700 10 librados: watch_flush enter 2024-02-13T00:38:29.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.502+0000 7f7df67cb700 10 librados: watch_flush exit 2024-02-13T00:38:29.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.502+0000 7f7df67cb700 20 client.14601.objecter shutdown clearing up homeless session... 2024-02-13T00:38:29.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.502+0000 7f7df67cb700 10 client.14601.objecter successfully canceled tick 2024-02-13T00:38:29.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.502+0000 7f7df67cb700 1 -- 172.21.15.162:0/2290128336 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7dd8062c90 msgr2=0x7f7dd8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:29.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.502+0000 7f7df67cb700 1 --2- 172.21.15.162:0/2290128336 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7dd8062c90 0x7f7dd8065150 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f7de0004de0 tx=0x7f7de000b040).stop 2024-02-13T00:38:29.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.502+0000 7f7df67cb700 1 -- 172.21.15.162:0/2290128336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7df00fd490 msgr2=0x7f7df0107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:29.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.502+0000 7f7df67cb700 1 --2- 172.21.15.162:0/2290128336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7df00fd490 0x7f7df0107030 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f7de8004860 tx=0x7f7de8004c50).stop 2024-02-13T00:38:29.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.503+0000 7f7df67cb700 1 -- 172.21.15.162:0/2290128336 shutdown_connections 2024-02-13T00:38:29.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.503+0000 7f7df67cb700 1 --2- 172.21.15.162:0/2290128336 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7dd8062c90 0x7f7dd8065150 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:29.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.503+0000 7f7df67cb700 1 --2- 172.21.15.162:0/2290128336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7df00fd490 0x7f7df0107030 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:29.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.503+0000 7f7df67cb700 1 --2- 172.21.15.162:0/2290128336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7df00f73a0 0x7f7df010ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:29.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.503+0000 7f7df67cb700 1 --2- 172.21.15.162:0/2290128336 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7df00fcb30 0x7f7df010d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:29.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.503+0000 7f7df67cb700 1 -- 172.21.15.162:0/2290128336 >> 172.21.15.162:0/2290128336 conn(0x7f7df006eb90 msgr2=0x7f7df00fab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:29.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.503+0000 7f7df67cb700 1 -- 172.21.15.162:0/2290128336 shutdown_connections 2024-02-13T00:38:29.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.503+0000 7f7df67cb700 1 -- 172.21.15.162:0/2290128336 wait complete. 2024-02-13T00:38:29.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:29.503+0000 7f7df67cb700 1 librados: shutdown 2024-02-13T00:38:29.794 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:29 smithi181 bash[64042]: Getting image source signatures 2024-02-13T00:38:29.795 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:29 smithi181 bash[64042]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-13T00:38:29.795 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:29 smithi181 bash[64042]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-13T00:38:29.795 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:29 smithi181 bash[64042]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-13T00:38:30.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:28.162849+0000 mgr.y (mgr.14544) 27 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:38:30.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:29.502806+0000 mon.a (mon.0) 661 : audit [DBG] from='client.? 172.21.15.162:0/2290128336' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-13T00:38:30.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:28.162849+0000 mgr.y (mgr.14544) 27 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:38:30.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:29.502806+0000 mon.a (mon.0) 661 : audit [DBG] from='client.? 172.21.15.162:0/2290128336' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-13T00:38:30.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool get default.rgw.control pg_num 2024-02-13T00:38:30.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:28.162849+0000 mgr.y (mgr.14544) 27 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:38:30.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:29.502806+0000 mon.a (mon.0) 661 : audit [DBG] from='client.? 172.21.15.162:0/2290128336' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-13T00:38:30.487 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:31.023 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:38:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[68995]: level=info ts=2024-02-13T00:38:30.905Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002088009s 2024-02-13T00:38:32.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:30.163335+0000 mgr.y (mgr.14544) 28 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:38:32.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:30.163335+0000 mgr.y (mgr.14544) 28 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:38:32.111 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:30.163335+0000 mgr.y (mgr.14544) 28 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:38:32.111 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:31 smithi181 bash[64042]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-13T00:38:32.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.338+0000 7f98014a9700 1 -- 172.21.15.162:0/2515482335 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f97fc0fcb00 msgr2=0x7f97fc0fcee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:32.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.338+0000 7f98014a9700 1 --2- 172.21.15.162:0/2515482335 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f97fc0fcb00 0x7f97fc0fcee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f97e40047b0 tx=0x7f97e4009cb0).stop 2024-02-13T00:38:32.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.338+0000 7f98014a9700 1 -- 172.21.15.162:0/2515482335 shutdown_connections 2024-02-13T00:38:32.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.338+0000 7f98014a9700 1 --2- 172.21.15.162:0/2515482335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97fc0f7370 0x7f97fc103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:32.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.338+0000 7f98014a9700 1 --2- 172.21.15.162:0/2515482335 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f97fc0fcb00 0x7f97fc0fcee0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:32.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.338+0000 7f98014a9700 1 --2- 172.21.15.162:0/2515482335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97fc0fd460 0x7f97fc0f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:32.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.338+0000 7f98014a9700 1 -- 172.21.15.162:0/2515482335 >> 172.21.15.162:0/2515482335 conn(0x7f97fc0f2710 msgr2=0x7f97fc0f4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:32.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.338+0000 7f98014a9700 1 -- 172.21.15.162:0/2515482335 shutdown_connections 2024-02-13T00:38:32.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.338+0000 7f98014a9700 1 -- 172.21.15.162:0/2515482335 wait complete. 2024-02-13T00:38:32.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f98014a9700 1 librados: starting msgr at 2024-02-13T00:38:32.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f98014a9700 1 librados: starting objecter 2024-02-13T00:38:32.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f98014a9700 1 Processor -- start 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f98014a9700 1 -- start start 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f98014a9700 1 librados: setting wanted keys 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f98014a9700 1 librados: calling monclient init 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f98014a9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97fc0f7370 0x7f97fc10e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f98014a9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f97fc0fcb00 0x7f97fc10e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f98014a9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97fc0fd460 0x7f97fc108340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f98014a9700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc108830 con 0x7f97fc0f7370 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f98014a9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc1089a0 con 0x7f97fc0fd460 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f98014a9700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc108b60 con 0x7f97fc0fcb00 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f97faffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97fc0f7370 0x7f97fc10e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f97fa7fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f97fc0fcb00 0x7f97fc10e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f97faffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97fc0f7370 0x7f97fc10e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:54790/0 (socket says 172.21.15.162:54790) 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f97fa7fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f97fc0fcb00 0x7f97fc10e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:53928/0 (socket says 172.21.15.162:53928) 2024-02-13T00:38:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f97faffd700 1 -- 172.21.15.162:0/1985031349 learned_addr learned my addr 172.21.15.162:0/1985031349 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f97fb7fe700 1 --2- 172.21.15.162:0/1985031349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97fc0fd460 0x7f97fc108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f97fa7fc700 1 -- 172.21.15.162:0/1985031349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97fc0fd460 msgr2=0x7f97fc108340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f97fa7fc700 1 --2- 172.21.15.162:0/1985031349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97fc0fd460 0x7f97fc108340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f97fa7fc700 1 -- 172.21.15.162:0/1985031349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97fc0f7370 msgr2=0x7f97fc10e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f97fa7fc700 1 --2- 172.21.15.162:0/1985031349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97fc0f7370 0x7f97fc10e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.339+0000 7f97fa7fc700 1 -- 172.21.15.162:0/1985031349 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec005040 con 0x7f97fc0fcb00 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.340+0000 7f97faffd700 1 --2- 172.21.15.162:0/1985031349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97fc0f7370 0x7f97fc10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.340+0000 7f97fa7fc700 1 --2- 172.21.15.162:0/1985031349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f97fc0fcb00 0x7f97fc10e690 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f97ec0091f0 tx=0x7f97ec009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.340+0000 7f97f3fff700 10 client.?.objecter ms_handle_connect 0x7f97fc0fcb00 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.340+0000 7f97f3fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.340+0000 7f97f3fff700 1 -- 172.21.15.162:0/1985031349 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec00e030 con 0x7f97fc0fcb00 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.340+0000 7f98014a9700 1 -- 172.21.15.162:0/1985031349 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e4005040 con 0x7f97fc0fcb00 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.340+0000 7f97f3fff700 1 -- 172.21.15.162:0/1985031349 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97ec00f040 con 0x7f97fc0fcb00 2024-02-13T00:38:32.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.340+0000 7f97f3fff700 1 -- 172.21.15.162:0/1985031349 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec0147e0 con 0x7f97fc0fcb00 2024-02-13T00:38:32.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.340+0000 7f98014a9700 10 client.24506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:32.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.340+0000 7f98014a9700 1 -- 172.21.15.162:0/1985031349 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97fc1091a0 con 0x7f97fc0fcb00 2024-02-13T00:38:32.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.340+0000 7f98014a9700 1 librados: init done 2024-02-13T00:38:32.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.341+0000 7f98014a9700 1 -- 172.21.15.162:0/1985031349 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97fc0607d0 con 0x7f97fc0fcb00 2024-02-13T00:38:32.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.341+0000 7f97f3fff700 1 -- 172.21.15.162:0/1985031349 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f97ec0104f0 con 0x7f97fc0fcb00 2024-02-13T00:38:32.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.342+0000 7f97f3fff700 1 --2- 172.21.15.162:0/1985031349 >> v2:172.21.15.162:6800/2866534053 conn(0x7f97e8062c90 0x7f97e8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:32.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.342+0000 7f97faffd700 1 --2- 172.21.15.162:0/1985031349 >> v2:172.21.15.162:6800/2866534053 conn(0x7f97e8062c90 0x7f97e8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:32.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.342+0000 7f97f3fff700 1 -- 172.21.15.162:0/1985031349 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f97ec07ca40 con 0x7f97fc0fcb00 2024-02-13T00:38:32.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.342+0000 7f97f3fff700 10 client.24506.objecter ms_dispatch 0x7f97fc001550 osd_map(74..74 src has 1..74) v4 2024-02-13T00:38:32.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.342+0000 7f97f3fff700 3 client.24506.objecter handle_osd_map got epochs [74,74] > 0 2024-02-13T00:38:32.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.342+0000 7f97f3fff700 3 client.24506.objecter handle_osd_map decoding full epoch 74 2024-02-13T00:38:32.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.342+0000 7f97f3fff700 20 client.24506.objecter dump_active .. 0 homeless 2024-02-13T00:38:32.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.342+0000 7f97f3fff700 10 client.24506.objecter ms_handle_connect 0x7f97e8062c90 2024-02-13T00:38:32.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.342+0000 7f97faffd700 1 --2- 172.21.15.162:0/1985031349 >> v2:172.21.15.162:6800/2866534053 conn(0x7f97e8062c90 0x7f97e8065150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f97e4009bc0 tx=0x7f97e4007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:32.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.346+0000 7f97f3fff700 1 -- 172.21.15.162:0/1985031349 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f97ec07ee60 con 0x7f97fc0fcb00 2024-02-13T00:38:32.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.507+0000 7f98014a9700 1 -- 172.21.15.162:0/1985031349 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"} v 0) v1 -- 0x7f97fc1060b0 con 0x7f97fc0fcb00 2024-02-13T00:38:32.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.508+0000 7f97f3fff700 1 -- 172.21.15.162:0/1985031349 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]=0 v74) v1 ==== 108+0+11 (secure 0 0 0) 0x7f97ec050720 con 0x7f97fc0fcb00 2024-02-13T00:38:32.511 INFO:teuthology.orchestra.run.smithi162.stdout:pg_num: 32 2024-02-13T00:38:32.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.509+0000 7f98014a9700 10 librados: watch_flush enter 2024-02-13T00:38:32.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.509+0000 7f98014a9700 10 librados: watch_flush exit 2024-02-13T00:38:32.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.509+0000 7f98014a9700 20 client.24506.objecter shutdown clearing up homeless session... 2024-02-13T00:38:32.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.509+0000 7f98014a9700 10 client.24506.objecter successfully canceled tick 2024-02-13T00:38:32.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.509+0000 7f98014a9700 1 -- 172.21.15.162:0/1985031349 >> v2:172.21.15.162:6800/2866534053 conn(0x7f97e8062c90 msgr2=0x7f97e8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:32.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.509+0000 7f98014a9700 1 --2- 172.21.15.162:0/1985031349 >> v2:172.21.15.162:6800/2866534053 conn(0x7f97e8062c90 0x7f97e8065150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f97e4009bc0 tx=0x7f97e4007f90).stop 2024-02-13T00:38:32.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.510+0000 7f98014a9700 1 -- 172.21.15.162:0/1985031349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f97fc0fcb00 msgr2=0x7f97fc10e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:32.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.510+0000 7f98014a9700 1 --2- 172.21.15.162:0/1985031349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f97fc0fcb00 0x7f97fc10e690 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f97ec0091f0 tx=0x7f97ec009cb0).stop 2024-02-13T00:38:32.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.510+0000 7f98014a9700 1 -- 172.21.15.162:0/1985031349 shutdown_connections 2024-02-13T00:38:32.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.510+0000 7f98014a9700 1 --2- 172.21.15.162:0/1985031349 >> v2:172.21.15.162:6800/2866534053 conn(0x7f97e8062c90 0x7f97e8065150 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:32.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.510+0000 7f98014a9700 1 --2- 172.21.15.162:0/1985031349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97fc0fd460 0x7f97fc108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:32.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.510+0000 7f98014a9700 1 --2- 172.21.15.162:0/1985031349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97fc0f7370 0x7f97fc10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:32.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.510+0000 7f98014a9700 1 --2- 172.21.15.162:0/1985031349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f97fc0fcb00 0x7f97fc10e690 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:32.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.510+0000 7f98014a9700 1 -- 172.21.15.162:0/1985031349 >> 172.21.15.162:0/1985031349 conn(0x7f97fc0f2710 msgr2=0x7f97fc0f41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:32.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.510+0000 7f98014a9700 1 -- 172.21.15.162:0/1985031349 shutdown_connections 2024-02-13T00:38:32.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.510+0000 7f98014a9700 1 -- 172.21.15.162:0/1985031349 wait complete. 2024-02-13T00:38:32.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:32.510+0000 7f98014a9700 1 librados: shutdown 2024-02-13T00:38:32.544 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 bash[64042]: Writing manifest to image destination 2024-02-13T00:38:33.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:32.508850+0000 mon.c (mon.2) 52 : audit [DBG] from='client.? 172.21.15.162:0/1985031349' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-13T00:38:33.045 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.787Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-13T00:38:33.045 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.787Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-13T00:38:33.045 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-13T00:38:33.045 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-13T00:38:33.045 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-13T00:38:33.045 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=arp 2024-02-13T00:38:33.045 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-13T00:38:33.045 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-13T00:38:33.045 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=edac 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-13T00:38:33.046 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=os 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-13T00:38:33.047 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=stat 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=time 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=uname 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-13T00:38:33.048 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-13T00:38:33.049 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b[64101]: ts=2024-02-13T00:38:32.788Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-13T00:38:33.049 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 bash[64042]: 9c9767158575233a3489a83d360993c9a6200b1dfc98f447fba75f0d09a69cf6 2024-02-13T00:38:33.049 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:38:32 smithi181 systemd[1]: Started Ceph node-exporter.b for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:38:33.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:32.508850+0000 mon.c (mon.2) 52 : audit [DBG] from='client.? 172.21.15.162:0/1985031349' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-13T00:38:33.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:32.508850+0000 mon.c (mon.2) 52 : audit [DBG] from='client.? 172.21.15.162:0/1985031349' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-13T00:38:33.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool get default.rgw.meta pg_num 2024-02-13T00:38:33.403 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:32.163736+0000 mgr.y (mgr.14544) 29 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:32.921804+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:34.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:32.163736+0000 mgr.y (mgr.14544) 29 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:34.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:32.921804+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:32.163736+0000 mgr.y (mgr.14544) 29 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:32.921804+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:35.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:38:32.932178+0000 mgr.y (mgr.14544) 30 : cephadm [INF] Deploying daemon prometheus.a on smithi181 2024-02-13T00:38:35.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:34.341115+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:35.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:38:32.932178+0000 mgr.y (mgr.14544) 30 : cephadm [INF] Deploying daemon prometheus.a on smithi181 2024-02-13T00:38:35.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:34.341115+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:35.191 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:38:32.932178+0000 mgr.y (mgr.14544) 30 : cephadm [INF] Deploying daemon prometheus.a on smithi181 2024-02-13T00:38:35.191 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:34.341115+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:35.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.360+0000 7f166a574700 1 -- 172.21.15.162:0/4046490652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16640fdc50 msgr2=0x7f16640fe030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:35.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.360+0000 7f166a574700 1 --2- 172.21.15.162:0/4046490652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16640fdc50 0x7f16640fe030 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f16540047b0 tx=0x7f1654009cb0).stop 2024-02-13T00:38:35.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.360+0000 7f166a574700 1 -- 172.21.15.162:0/4046490652 shutdown_connections 2024-02-13T00:38:35.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.360+0000 7f166a574700 1 --2- 172.21.15.162:0/4046490652 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16640f84c0 0x7f1664104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:35.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.360+0000 7f166a574700 1 --2- 172.21.15.162:0/4046490652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16640fdc50 0x7f16640fe030 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:35.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.360+0000 7f166a574700 1 --2- 172.21.15.162:0/4046490652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16640fe5b0 0x7f16640f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:35.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.360+0000 7f166a574700 1 -- 172.21.15.162:0/4046490652 >> 172.21.15.162:0/4046490652 conn(0x7f16640f38a0 msgr2=0x7f16640f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.360+0000 7f166a574700 1 -- 172.21.15.162:0/4046490652 shutdown_connections 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.360+0000 7f166a574700 1 -- 172.21.15.162:0/4046490652 wait complete. 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f166a574700 1 librados: starting msgr at 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f166a574700 1 librados: starting objecter 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f166a574700 1 Processor -- start 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f166a574700 1 -- start start 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f166a574700 1 librados: setting wanted keys 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f166a574700 1 librados: calling monclient init 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f166a574700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16640f84c0 0x7f166406b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f166a574700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16640fdc50 0x7f166406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f166a574700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16640fe5b0 0x7f166406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f166a574700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1664116580 con 0x7f16640f84c0 2024-02-13T00:38:35.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f166a574700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16641166f0 con 0x7f16640fdc50 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f166a574700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16641168b0 con 0x7f16640fe5b0 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.361+0000 7f1663fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16640f84c0 0x7f166406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f1663fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16640f84c0 0x7f166406b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38510/0 (socket says 172.21.15.162:38510) 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f1663fff700 1 -- 172.21.15.162:0/3332876444 learned_addr learned my addr 172.21.15.162:0/3332876444 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f1668b11700 1 --2- 172.21.15.162:0/3332876444 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16640fe5b0 0x7f166406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f16637fe700 1 --2- 172.21.15.162:0/3332876444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16640fdc50 0x7f166406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f1663fff700 1 -- 172.21.15.162:0/3332876444 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16640fe5b0 msgr2=0x7f166406bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f1663fff700 1 --2- 172.21.15.162:0/3332876444 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16640fe5b0 0x7f166406bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f1663fff700 1 -- 172.21.15.162:0/3332876444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16640fdc50 msgr2=0x7f166406b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f1663fff700 1 --2- 172.21.15.162:0/3332876444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16640fdc50 0x7f166406b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f1663fff700 1 -- 172.21.15.162:0/3332876444 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1654005040 con 0x7f16640f84c0 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f16637fe700 1 --2- 172.21.15.162:0/3332876444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16640fdc50 0x7f166406b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f1663fff700 1 --2- 172.21.15.162:0/3332876444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16640f84c0 0x7f166406b170 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f16540044e0 tx=0x7f1654007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f16617fa700 10 client.?.objecter ms_handle_connect 0x7f16640f84c0 2024-02-13T00:38:35.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f16617fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:35.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f16617fa700 1 -- 172.21.15.162:0/3332876444 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654018070 con 0x7f16640f84c0 2024-02-13T00:38:35.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f16617fa700 1 -- 172.21.15.162:0/3332876444 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16540093b0 con 0x7f16640f84c0 2024-02-13T00:38:35.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f166a574700 1 -- 172.21.15.162:0/3332876444 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1664116b30 con 0x7f16640f84c0 2024-02-13T00:38:35.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f16617fa700 1 -- 172.21.15.162:0/3332876444 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165401de40 con 0x7f16640f84c0 2024-02-13T00:38:35.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f166a574700 10 client.14613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:35.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f166a574700 1 -- 172.21.15.162:0/3332876444 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1664116ee0 con 0x7f16640f84c0 2024-02-13T00:38:35.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.362+0000 7f166a574700 1 librados: init done 2024-02-13T00:38:35.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.363+0000 7f166a574700 1 -- 172.21.15.162:0/3332876444 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1664107170 con 0x7f16640f84c0 2024-02-13T00:38:35.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.363+0000 7f16617fa700 1 -- 172.21.15.162:0/3332876444 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1654013ad0 con 0x7f16640f84c0 2024-02-13T00:38:35.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.364+0000 7f16617fa700 1 --2- 172.21.15.162:0/3332876444 >> v2:172.21.15.162:6800/2866534053 conn(0x7f164c062c90 0x7f164c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:35.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.364+0000 7f16617fa700 1 -- 172.21.15.162:0/3332876444 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f165407df30 con 0x7f16640f84c0 2024-02-13T00:38:35.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.364+0000 7f16617fa700 10 client.14613.objecter ms_dispatch 0x7f1664001550 osd_map(74..74 src has 1..74) v4 2024-02-13T00:38:35.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.364+0000 7f16617fa700 3 client.14613.objecter handle_osd_map got epochs [74,74] > 0 2024-02-13T00:38:35.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.364+0000 7f16617fa700 3 client.14613.objecter handle_osd_map decoding full epoch 74 2024-02-13T00:38:35.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.364+0000 7f16637fe700 1 --2- 172.21.15.162:0/3332876444 >> v2:172.21.15.162:6800/2866534053 conn(0x7f164c062c90 0x7f164c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:35.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.364+0000 7f16617fa700 20 client.14613.objecter dump_active .. 0 homeless 2024-02-13T00:38:35.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.364+0000 7f16617fa700 10 client.14613.objecter ms_handle_connect 0x7f164c062c90 2024-02-13T00:38:35.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.364+0000 7f16637fe700 1 --2- 172.21.15.162:0/3332876444 >> v2:172.21.15.162:6800/2866534053 conn(0x7f164c062c90 0x7f164c065150 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f166406c910 tx=0x7f1658009de0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:35.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.368+0000 7f16617fa700 1 -- 172.21.15.162:0/3332876444 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1654051bb0 con 0x7f16640f84c0 2024-02-13T00:38:35.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.530+0000 7f166a574700 1 -- 172.21.15.162:0/3332876444 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"} v 0) v1 -- 0x7f166405fb50 con 0x7f16640f84c0 2024-02-13T00:38:35.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.531+0000 7f16617fa700 1 -- 172.21.15.162:0/3332876444 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]=0 v74) v1 ==== 105+0+11 (secure 0 0 0) 0x7f16540517a0 con 0x7f16640f84c0 2024-02-13T00:38:35.534 INFO:teuthology.orchestra.run.smithi162.stdout:pg_num: 32 2024-02-13T00:38:35.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.532+0000 7f166a574700 10 librados: watch_flush enter 2024-02-13T00:38:35.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 10 librados: watch_flush exit 2024-02-13T00:38:35.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 20 client.14613.objecter shutdown clearing up homeless session... 2024-02-13T00:38:35.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 10 client.14613.objecter successfully canceled tick 2024-02-13T00:38:35.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 -- 172.21.15.162:0/3332876444 >> v2:172.21.15.162:6800/2866534053 conn(0x7f164c062c90 msgr2=0x7f164c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:35.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 --2- 172.21.15.162:0/3332876444 >> v2:172.21.15.162:6800/2866534053 conn(0x7f164c062c90 0x7f164c065150 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f166406c910 tx=0x7f1658009de0).stop 2024-02-13T00:38:35.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 -- 172.21.15.162:0/3332876444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16640f84c0 msgr2=0x7f166406b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:35.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 --2- 172.21.15.162:0/3332876444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16640f84c0 0x7f166406b170 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f16540044e0 tx=0x7f1654007f90).stop 2024-02-13T00:38:35.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 -- 172.21.15.162:0/3332876444 shutdown_connections 2024-02-13T00:38:35.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 --2- 172.21.15.162:0/3332876444 >> v2:172.21.15.162:6800/2866534053 conn(0x7f164c062c90 0x7f164c065150 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:35.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 --2- 172.21.15.162:0/3332876444 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16640fe5b0 0x7f166406bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:35.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 --2- 172.21.15.162:0/3332876444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16640f84c0 0x7f166406b170 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:35.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 --2- 172.21.15.162:0/3332876444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16640fdc50 0x7f166406b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:35.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 -- 172.21.15.162:0/3332876444 >> 172.21.15.162:0/3332876444 conn(0x7f16640f38a0 msgr2=0x7f16640f53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:35.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 -- 172.21.15.162:0/3332876444 shutdown_connections 2024-02-13T00:38:35.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 -- 172.21.15.162:0/3332876444 wait complete. 2024-02-13T00:38:35.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:35.533+0000 7f166a574700 1 librados: shutdown 2024-02-13T00:38:35.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:38:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:38:36.154 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:34.164094+0000 mgr.y (mgr.14544) 31 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:36.154 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:35.531954+0000 mon.a (mon.0) 664 : audit [DBG] from='client.? 172.21.15.162:0/3332876444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-13T00:38:36.154 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:34.164094+0000 mgr.y (mgr.14544) 31 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:36.154 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:35.531954+0000 mon.a (mon.0) 664 : audit [DBG] from='client.? 172.21.15.162:0/3332876444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-13T00:38:36.174 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-13T00:38:36.174 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-13T00:38:36.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph mgr dump --format=json 2024-02-13T00:38:36.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:34.164094+0000 mgr.y (mgr.14544) 31 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:36.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:35.531954+0000 mon.a (mon.0) 664 : audit [DBG] from='client.? 172.21.15.162:0/3332876444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-13T00:38:36.443 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:36.960 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:35.188573+0000 mgr.y (mgr.14544) 32 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:36.961 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:35.188573+0000 mgr.y (mgr.14544) 32 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:37.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:35.188573+0000 mgr.y (mgr.14544) 32 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:37.934 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:36.164710+0000 mgr.y (mgr.14544) 33 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:37.935 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:37.239307+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:37.935 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:37.239349+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:37.935 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:36.164710+0000 mgr.y (mgr.14544) 33 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:37.935 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:37.239307+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:37.935 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:37.239349+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:38.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.169+0000 7f4e3855d700 1 -- 172.21.15.162:0/3169843650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e300f9b70 msgr2=0x7f4e300f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:38.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.169+0000 7f4e3855d700 1 --2- 172.21.15.162:0/3169843650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e300f9b70 0x7f4e300f9f50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f4e24004870 tx=0x7f4e24009cb0).stop 2024-02-13T00:38:38.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.169+0000 7f4e3855d700 1 -- 172.21.15.162:0/3169843650 shutdown_connections 2024-02-13T00:38:38.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.169+0000 7f4e3855d700 1 --2- 172.21.15.162:0/3169843650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e300f8500 0x7f4e30104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.169+0000 7f4e3855d700 1 --2- 172.21.15.162:0/3169843650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e300f9b70 0x7f4e300f9f50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.169+0000 7f4e3855d700 1 --2- 172.21.15.162:0/3169843650 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e300fa440 0x7f4e300f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.169+0000 7f4e3855d700 1 -- 172.21.15.162:0/3169843650 >> 172.21.15.162:0/3169843650 conn(0x7f4e300f38c0 msgr2=0x7f4e300f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 -- 172.21.15.162:0/3169843650 shutdown_connections 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 -- 172.21.15.162:0/3169843650 wait complete. 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 librados: starting msgr at 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 librados: starting objecter 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 Processor -- start 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 -- start start 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 librados: setting wanted keys 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 librados: calling monclient init 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e300f8500 0x7f4e3006b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e300f9b70 0x7f4e3006b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:38.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e300fa440 0x7f4e3006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e301166b0 con 0x7f4e300fa440 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e30116820 con 0x7f4e300f9b70 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.170+0000 7f4e3855d700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e301169e0 con 0x7f4e300f8500 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e362f9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e300f8500 0x7f4e3006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e36afa700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e300fa440 0x7f4e3006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e362f9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e300f8500 0x7f4e3006b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:51032/0 (socket says 172.21.15.162:51032) 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e362f9700 1 -- 172.21.15.162:0/3004042451 learned_addr learned my addr 172.21.15.162:0/3004042451 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e36afa700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e300fa440 0x7f4e3006bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38528/0 (socket says 172.21.15.162:38528) 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e35af8700 1 --2- 172.21.15.162:0/3004042451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e300f9b70 0x7f4e3006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e36afa700 1 -- 172.21.15.162:0/3004042451 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e300f8500 msgr2=0x7f4e3006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e36afa700 1 --2- 172.21.15.162:0/3004042451 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e300f8500 0x7f4e3006b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e36afa700 1 -- 172.21.15.162:0/3004042451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e300f9b70 msgr2=0x7f4e3006b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e36afa700 1 --2- 172.21.15.162:0/3004042451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e300f9b70 0x7f4e3006b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:38.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e36afa700 1 -- 172.21.15.162:0/3004042451 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c005040 con 0x7f4e300fa440 2024-02-13T00:38:38.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e362f9700 1 --2- 172.21.15.162:0/3004042451 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e300f8500 0x7f4e3006b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:38.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e36afa700 1 --2- 172.21.15.162:0/3004042451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e300fa440 0x7f4e3006bb50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4e2800c8e0 tx=0x7f4e28004ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:38.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e237fe700 10 client.?.objecter ms_handle_connect 0x7f4e300fa440 2024-02-13T00:38:38.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e237fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:38.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e237fe700 1 -- 172.21.15.162:0/3004042451 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2800e4a0 con 0x7f4e300fa440 2024-02-13T00:38:38.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e237fe700 1 -- 172.21.15.162:0/3004042451 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e2800ead0 con 0x7f4e300fa440 2024-02-13T00:38:38.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e237fe700 1 -- 172.21.15.162:0/3004042451 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28012a70 con 0x7f4e300fa440 2024-02-13T00:38:38.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e3855d700 1 -- 172.21.15.162:0/3004042451 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e24005040 con 0x7f4e300fa440 2024-02-13T00:38:38.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e3855d700 10 client.14619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:38.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e3855d700 1 -- 172.21.15.162:0/3004042451 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e301170a0 con 0x7f4e300fa440 2024-02-13T00:38:38.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.171+0000 7f4e3855d700 1 librados: init done 2024-02-13T00:38:38.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.172+0000 7f4e3855d700 1 -- 172.21.15.162:0/3004042451 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e300475e0 con 0x7f4e300fa440 2024-02-13T00:38:38.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.173+0000 7f4e237fe700 1 -- 172.21.15.162:0/3004042451 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4e2800e600 con 0x7f4e300fa440 2024-02-13T00:38:38.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.173+0000 7f4e237fe700 1 --2- 172.21.15.162:0/3004042451 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4e1c062c90 0x7f4e1c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:38.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.173+0000 7f4e237fe700 1 -- 172.21.15.162:0/3004042451 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f4e2807c3d0 con 0x7f4e300fa440 2024-02-13T00:38:38.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.173+0000 7f4e237fe700 10 client.14619.objecter ms_dispatch 0x7f4e30001550 osd_map(75..75 src has 1..75) v4 2024-02-13T00:38:38.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.173+0000 7f4e237fe700 3 client.14619.objecter handle_osd_map got epochs [75,75] > 0 2024-02-13T00:38:38.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.173+0000 7f4e237fe700 3 client.14619.objecter handle_osd_map decoding full epoch 75 2024-02-13T00:38:38.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.173+0000 7f4e362f9700 1 --2- 172.21.15.162:0/3004042451 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4e1c062c90 0x7f4e1c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:38.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.173+0000 7f4e237fe700 20 client.14619.objecter dump_active .. 0 homeless 2024-02-13T00:38:38.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.173+0000 7f4e237fe700 10 client.14619.objecter ms_handle_connect 0x7f4e1c062c90 2024-02-13T00:38:38.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.173+0000 7f4e362f9700 1 --2- 172.21.15.162:0/3004042451 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4e1c062c90 0x7f4e1c065150 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f4e24009a00 tx=0x7f4e2400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:38.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.177+0000 7f4e237fe700 1 -- 172.21.15.162:0/3004042451 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4e28048f60 con 0x7f4e300fa440 2024-02-13T00:38:38.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:36.164710+0000 mgr.y (mgr.14544) 33 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:38.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:37.239307+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:38.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:37.239349+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:38.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.386+0000 7f4e3855d700 1 -- 172.21.15.162:0/3004042451 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4e3006c650 con 0x7f4e300fa440 2024-02-13T00:38:38.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.389+0000 7f4e237fe700 1 -- 172.21.15.162:0/3004042451 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+151942 (secure 0 0 0) 0x7f4e280461a0 con 0x7f4e300fa440 2024-02-13T00:38:38.394 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:38:38.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 10 librados: watch_flush enter 2024-02-13T00:38:38.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 10 librados: watch_flush exit 2024-02-13T00:38:38.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 20 client.14619.objecter shutdown clearing up homeless session... 2024-02-13T00:38:38.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 10 client.14619.objecter successfully canceled tick 2024-02-13T00:38:38.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 -- 172.21.15.162:0/3004042451 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4e1c062c90 msgr2=0x7f4e1c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:38.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 --2- 172.21.15.162:0/3004042451 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4e1c062c90 0x7f4e1c065150 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f4e24009a00 tx=0x7f4e2400b040).stop 2024-02-13T00:38:38.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 -- 172.21.15.162:0/3004042451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e300fa440 msgr2=0x7f4e3006bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:38.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 --2- 172.21.15.162:0/3004042451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e300fa440 0x7f4e3006bb50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4e2800c8e0 tx=0x7f4e28004ca0).stop 2024-02-13T00:38:38.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 -- 172.21.15.162:0/3004042451 shutdown_connections 2024-02-13T00:38:38.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 --2- 172.21.15.162:0/3004042451 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4e1c062c90 0x7f4e1c065150 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:38.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 --2- 172.21.15.162:0/3004042451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e300fa440 0x7f4e3006bb50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:38.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 --2- 172.21.15.162:0/3004042451 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e300f8500 0x7f4e3006b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:38.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 --2- 172.21.15.162:0/3004042451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e300f9b70 0x7f4e3006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:38.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 -- 172.21.15.162:0/3004042451 >> 172.21.15.162:0/3004042451 conn(0x7f4e300f38c0 msgr2=0x7f4e300fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:38.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 -- 172.21.15.162:0/3004042451 shutdown_connections 2024-02-13T00:38:38.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 -- 172.21.15.162:0/3004042451 wait complete. 2024-02-13T00:38:38.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:38.391+0000 7f4e3855d700 1 librados: shutdown 2024-02-13T00:38:38.996 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:37.838767+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-13T00:38:38.996 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:38.387256+0000 mon.a (mon.0) 666 : audit [DBG] from='client.? 172.21.15.162:0/3004042451' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-13T00:38:38.996 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:37.838767+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-13T00:38:38.996 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:38.387256+0000 mon.a (mon.0) 666 : audit [DBG] from='client.? 172.21.15.162:0/3004042451' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-13T00:38:39.016 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":19,"active_gid":14544,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2866534053}]},"active_addr":"172.21.15.162:6800/2866534053","active_change":"2024-02-13T00:38:04.143193+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[{"gid":24430,"name":"x","mgr_features":4540138314316775423,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.162:8443/","prometheus":"http://172.21.15.162:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":68,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":529667480}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":1771981399}]}]} 2024-02-13T00:38:39.019 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-13T00:38:39.019 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-13T00:38:39.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-13T00:38:39.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:37.838767+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-13T00:38:39.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:38.387256+0000 mon.a (mon.0) 666 : audit [DBG] from='client.? 172.21.15.162:0/3004042451' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-13T00:38:39.300 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:40.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:38.165018+0000 mgr.y (mgr.14544) 34 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:38:40.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:39.215059+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-13T00:38:40.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:38.165018+0000 mgr.y (mgr.14544) 34 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:38:40.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:39.215059+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-13T00:38:40.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:38.165018+0000 mgr.y (mgr.14544) 34 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:38:40.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:39.215059+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-13T00:38:41.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/1891049162 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d340f84c0 msgr2=0x7f0d34104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:41.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 --2- 172.21.15.162:0/1891049162 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d340f84c0 0x7f0d34104ac0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f0d2c009be0 tx=0x7f0d2c004d10).stop 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/1891049162 shutdown_connections 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 --2- 172.21.15.162:0/1891049162 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d340f84c0 0x7f0d34104ac0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 --2- 172.21.15.162:0/1891049162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d340fdc50 0x7f0d340fe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 --2- 172.21.15.162:0/1891049162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d340fe5b0 0x7f0d340f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/1891049162 >> 172.21.15.162:0/1891049162 conn(0x7f0d340f38a0 msgr2=0x7f0d340f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/1891049162 shutdown_connections 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/1891049162 wait complete. 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 librados: starting msgr at 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 librados: starting objecter 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 Processor -- start 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 -- start start 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 librados: setting wanted keys 2024-02-13T00:38:41.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.069+0000 7f0d3b3fd700 1 librados: calling monclient init 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d3b3fd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d340f84c0 0x7f0d3410dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d3b3fd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d340fdc50 0x7f0d3410e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d3b3fd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d340fe5b0 0x7f0d34108160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d3b3fd700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d34108650 con 0x7f0d340f84c0 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d3b3fd700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341087c0 con 0x7f0d340fdc50 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d3b3fd700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d34108980 con 0x7f0d340fe5b0 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d39199700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d340f84c0 0x7f0d3410dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d39199700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d340f84c0 0x7f0d3410dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38538/0 (socket says 172.21.15.162:38538) 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d3999a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d340fe5b0 0x7f0d34108160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d39199700 1 -- 172.21.15.162:0/3267751313 learned_addr learned my addr 172.21.15.162:0/3267751313 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d38998700 1 --2- 172.21.15.162:0/3267751313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d340fdc50 0x7f0d3410e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d39199700 1 -- 172.21.15.162:0/3267751313 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d340fe5b0 msgr2=0x7f0d34108160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d39199700 1 --2- 172.21.15.162:0/3267751313 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d340fe5b0 0x7f0d34108160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:41.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d39199700 1 -- 172.21.15.162:0/3267751313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d340fdc50 msgr2=0x7f0d3410e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:41.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d39199700 1 --2- 172.21.15.162:0/3267751313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d340fdc50 0x7f0d3410e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:41.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d39199700 1 -- 172.21.15.162:0/3267751313 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d30005040 con 0x7f0d340f84c0 2024-02-13T00:38:41.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d3999a700 1 --2- 172.21.15.162:0/3267751313 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d340fe5b0 0x7f0d34108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:41.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d39199700 1 --2- 172.21.15.162:0/3267751313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d340f84c0 0x7f0d3410dfc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f0d2400b530 tx=0x7f0d240090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:41.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d2a7fc700 10 client.?.objecter ms_handle_connect 0x7f0d340f84c0 2024-02-13T00:38:41.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d2a7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:41.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d2a7fc700 1 -- 172.21.15.162:0/3267751313 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24002320 con 0x7f0d340f84c0 2024-02-13T00:38:41.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/3267751313 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d2c0097c0 con 0x7f0d340f84c0 2024-02-13T00:38:41.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d2a7fc700 1 -- 172.21.15.162:0/3267751313 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d24002950 con 0x7f0d340f84c0 2024-02-13T00:38:41.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d2a7fc700 1 -- 172.21.15.162:0/3267751313 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24012950 con 0x7f0d340f84c0 2024-02-13T00:38:41.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d3b3fd700 10 client.14625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:41.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/3267751313 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d34108fc0 con 0x7f0d340f84c0 2024-02-13T00:38:41.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.070+0000 7f0d3b3fd700 1 librados: init done 2024-02-13T00:38:41.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.071+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/3267751313 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d340475e0 con 0x7f0d340f84c0 2024-02-13T00:38:41.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.072+0000 7f0d2a7fc700 1 -- 172.21.15.162:0/3267751313 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0d24002480 con 0x7f0d340f84c0 2024-02-13T00:38:41.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.072+0000 7f0d2a7fc700 1 --2- 172.21.15.162:0/3267751313 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0d20062c90 0x7f0d20065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:41.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.072+0000 7f0d38998700 1 --2- 172.21.15.162:0/3267751313 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0d20062c90 0x7f0d20065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:41.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.072+0000 7f0d2a7fc700 1 -- 172.21.15.162:0/3267751313 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f0d2407ca40 con 0x7f0d340f84c0 2024-02-13T00:38:41.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.072+0000 7f0d2a7fc700 10 client.14625.objecter ms_dispatch 0x7f0d34001550 osd_map(76..76 src has 1..76) v4 2024-02-13T00:38:41.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.072+0000 7f0d2a7fc700 3 client.14625.objecter handle_osd_map got epochs [76,76] > 0 2024-02-13T00:38:41.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.072+0000 7f0d2a7fc700 3 client.14625.objecter handle_osd_map decoding full epoch 76 2024-02-13T00:38:41.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.072+0000 7f0d2a7fc700 20 client.14625.objecter dump_active .. 0 homeless 2024-02-13T00:38:41.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.072+0000 7f0d2a7fc700 10 client.14625.objecter ms_handle_connect 0x7f0d20062c90 2024-02-13T00:38:41.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.072+0000 7f0d38998700 1 --2- 172.21.15.162:0/3267751313 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0d20062c90 0x7f0d20065150 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f0d34109900 tx=0x7f0d3000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:41.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.077+0000 7f0d2a7fc700 1 -- 172.21.15.162:0/3267751313 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0d2407ee60 con 0x7f0d340f84c0 2024-02-13T00:38:41.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.241+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/3267751313 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d3405fac0 con 0x7f0d340f84c0 2024-02-13T00:38:41.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.242+0000 7f0d2a7fc700 1 -- 172.21.15.162:0/3267751313 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+19063 (secure 0 0 0) 0x7f0d24050720 con 0x7f0d340f84c0 2024-02-13T00:38:41.245 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:38:41.245 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":76,"fsid":"d4ee01fc-ca06-11ee-95b9-87774f69a715","created":"2024-02-13T00:31:51.797017+0000","modified":"2024-02-13T00:38:39.210651+0000","last_up_change":"2024-02-13T00:37:33.855829+0000","last_in_change":"2024-02-13T00:37:16.202456+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-13T00:35:34.545390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-13T00:37:39.459980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-13T00:37:39.740219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-13T00:37:41.489292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-13T00:37:43.466304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-13T00:37:45.542442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6801","nonce":2103743338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2103743338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2103743338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6803","nonce":2103743338}]},"public_addr":"172.21.15.162:6801/2103743338","cluster_addr":"172.21.15.162:6802/2103743338","heartbeat_back_addr":"172.21.15.162:6804/2103743338","heartbeat_front_addr":"172.21.15.162:6803/2103743338","state":["exists","up"]},{"osd":1,"uuid":"39e9d2ef-9aa4-45bb-8a47-b0e0ec222886","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6805","nonce":150599780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":150599780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":150599780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6807","nonce":150599780}]},"public_addr":"172.21.15.162:6805/150599780","cluster_addr":"172.21.15.162:6806/150599780","heartbeat_back_addr":"172.21.15.162:6808/150599780","heartbeat_front_addr":"172.21.15.162:6807/150599780","state":["exists","up"]},{"osd":2,"uuid":"86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6809","nonce":3689719376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3689719376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3689719376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6811","nonce":3689719376}]},"public_addr":"172.21.15.162:6809/3689719376","cluster_addr":"172.21.15.162:6810/3689719376","heartbeat_back_addr":"172.21.15.162:6812/3689719376","heartbeat_front_addr":"172.21.15.162:6811/3689719376","state":["exists","up"]},{"osd":3,"uuid":"2e6b9f0e-7a18-4cf8-9331-1a973fadc001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6813","nonce":604971011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":604971011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":604971011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6815","nonce":604971011}]},"public_addr":"172.21.15.162:6813/604971011","cluster_addr":"172.21.15.162:6814/604971011","heartbeat_back_addr":"172.21.15.162:6816/604971011","heartbeat_front_addr":"172.21.15.162:6815/604971011","state":["exists","up"]},{"osd":4,"uuid":"c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3344263388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":3344263388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":3344263388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3344263388}]},"public_addr":"172.21.15.181:6800/3344263388","cluster_addr":"172.21.15.181:6801/3344263388","heartbeat_back_addr":"172.21.15.181:6803/3344263388","heartbeat_front_addr":"172.21.15.181:6802/3344263388","state":["exists","up"]},{"osd":5,"uuid":"6aa2bd68-7f4b-4973-ac49-7fbc57c1a723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2212296748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":2212296748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":2212296748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2212296748}]},"public_addr":"172.21.15.181:6804/2212296748","cluster_addr":"172.21.15.181:6805/2212296748","heartbeat_back_addr":"172.21.15.181:6807/2212296748","heartbeat_front_addr":"172.21.15.181:6806/2212296748","state":["exists","up"]},{"osd":6,"uuid":"791ba814-5284-432e-addc-d6b7f5f3d474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1506640341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1506640341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1506640341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1506640341}]},"public_addr":"172.21.15.181:6808/1506640341","cluster_addr":"172.21.15.181:6809/1506640341","heartbeat_back_addr":"172.21.15.181:6811/1506640341","heartbeat_front_addr":"172.21.15.181:6810/1506640341","state":["exists","up"]},{"osd":7,"uuid":"1f369c65-e663-4eb2-87a5-f2900f15b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1964192772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":1964192772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":1964192772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1964192772}]},"public_addr":"172.21.15.181:6812/1964192772","cluster_addr":"172.21.15.181:6813/1964192772","heartbeat_back_addr":"172.21.15.181:6815/1964192772","heartbeat_front_addr":"172.21.15.181:6814/1964192772","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:38:14.232055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:52.126669+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:53.397185+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:38:37.239351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:28.458200+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:44.566925+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:38:09.041989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:31.903163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.162:6800/1909610488":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/2370739667":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/1502590187":"2024-02-14T00:32:47.469889+0000","172.21.15.162:6800/4121673848":"2024-02-14T00:32:47.469889+0000","172.21.15.162:0/2412382714":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/3041562701":"2024-02-14T00:32:47.469889+0000","172.21.15.162:6800/2775563314":"2024-02-14T00:32:24.380625+0000","172.21.15.162:0/3135632190":"2024-02-14T00:32:24.380625+0000","172.21.15.162:0/197307636":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/2653247729":"2024-02-14T00:32:24.380625+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-13T00:38:41.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 10 librados: watch_flush enter 2024-02-13T00:38:41.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 10 librados: watch_flush exit 2024-02-13T00:38:41.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 20 client.14625.objecter shutdown clearing up homeless session... 2024-02-13T00:38:41.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 10 client.14625.objecter successfully canceled tick 2024-02-13T00:38:41.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/3267751313 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0d20062c90 msgr2=0x7f0d20065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:41.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 --2- 172.21.15.162:0/3267751313 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0d20062c90 0x7f0d20065150 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f0d34109900 tx=0x7f0d3000b040).stop 2024-02-13T00:38:41.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/3267751313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d340f84c0 msgr2=0x7f0d3410dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:41.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 --2- 172.21.15.162:0/3267751313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d340f84c0 0x7f0d3410dfc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f0d2400b530 tx=0x7f0d240090f0).stop 2024-02-13T00:38:41.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/3267751313 shutdown_connections 2024-02-13T00:38:41.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 --2- 172.21.15.162:0/3267751313 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0d20062c90 0x7f0d20065150 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:41.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 --2- 172.21.15.162:0/3267751313 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d340fe5b0 0x7f0d34108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:41.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 --2- 172.21.15.162:0/3267751313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d340f84c0 0x7f0d3410dfc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:41.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 --2- 172.21.15.162:0/3267751313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d340fdc50 0x7f0d3410e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:41.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/3267751313 >> 172.21.15.162:0/3267751313 conn(0x7f0d340f38a0 msgr2=0x7f0d340f5150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:41.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/3267751313 shutdown_connections 2024-02-13T00:38:41.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 -- 172.21.15.162:0/3267751313 wait complete. 2024-02-13T00:38:41.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:41.244+0000 7f0d3b3fd700 1 librados: shutdown 2024-02-13T00:38:41.925 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-13T00:38:41.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-13T00:38:42.193 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:42.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:40.165802+0000 mgr.y (mgr.14544) 35 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:42.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:41.242249+0000 mon.a (mon.0) 668 : audit [DBG] from='client.? 172.21.15.162:0/3267751313' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:38:42.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:40.165802+0000 mgr.y (mgr.14544) 35 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:42.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:41.242249+0000 mon.a (mon.0) 668 : audit [DBG] from='client.? 172.21.15.162:0/3267751313' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:38:42.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:40.165802+0000 mgr.y (mgr.14544) 35 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:42.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:41.242249+0000 mon.a (mon.0) 668 : audit [DBG] from='client.? 172.21.15.162:0/3267751313' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:38:43.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:42.526088+0000 osd.1 (osd.1) 7 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:38:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:42.529663+0000 osd.1 (osd.1) 8 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:38:43.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:42.526088+0000 osd.1 (osd.1) 7 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:38:43.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:42.529663+0000 osd.1 (osd.1) 8 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:38:43.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:42.526088+0000 osd.1 (osd.1) 7 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:38:43.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:42.529663+0000 osd.1 (osd.1) 8 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:38:43.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.969+0000 7ff7881db700 1 -- 172.21.15.162:0/3053246118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff7800f73a0 msgr2=0x7ff7801039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:43.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.969+0000 7ff7881db700 1 --2- 172.21.15.162:0/3053246118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff7800f73a0 0x7ff7801039a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7ff778009c30 tx=0x7ff778004c50).stop 2024-02-13T00:38:43.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.969+0000 7ff7881db700 1 -- 172.21.15.162:0/3053246118 shutdown_connections 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.969+0000 7ff7881db700 1 --2- 172.21.15.162:0/3053246118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff7800f73a0 0x7ff7801039a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.969+0000 7ff7881db700 1 --2- 172.21.15.162:0/3053246118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7800fcb30 0x7ff7800fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.969+0000 7ff7881db700 1 --2- 172.21.15.162:0/3053246118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7800fd490 0x7ff7800f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.969+0000 7ff7881db700 1 -- 172.21.15.162:0/3053246118 >> 172.21.15.162:0/3053246118 conn(0x7ff78006eb90 msgr2=0x7ff7800f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.969+0000 7ff7881db700 1 -- 172.21.15.162:0/3053246118 shutdown_connections 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.969+0000 7ff7881db700 1 -- 172.21.15.162:0/3053246118 wait complete. 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff7881db700 1 librados: starting msgr at 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff7881db700 1 librados: starting objecter 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff7881db700 1 Processor -- start 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff7881db700 1 -- start start 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff7881db700 1 librados: setting wanted keys 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff7881db700 1 librados: calling monclient init 2024-02-13T00:38:43.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff7881db700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7800f73a0 0x7ff78010dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff7881db700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7800fcb30 0x7ff78010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff7881db700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff7800fd490 0x7ff7801081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff7881db700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7801086b0 con 0x7ff7800f73a0 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff7881db700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff780108820 con 0x7ff7800fcb30 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff7881db700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff780108990 con 0x7ff7800fd490 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff785f77700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7800f73a0 0x7ff78010dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff785f77700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7800f73a0 0x7ff78010dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38550/0 (socket says 172.21.15.162:38550) 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff785f77700 1 -- 172.21.15.162:0/2525593711 learned_addr learned my addr 172.21.15.162:0/2525593711 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff786778700 1 --2- 172.21.15.162:0/2525593711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff7800fd490 0x7ff7801081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.970+0000 7ff785776700 1 --2- 172.21.15.162:0/2525593711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7800fcb30 0x7ff78010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff785f77700 1 -- 172.21.15.162:0/2525593711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff7800fd490 msgr2=0x7ff7801081c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff785f77700 1 --2- 172.21.15.162:0/2525593711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff7800fd490 0x7ff7801081c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff785f77700 1 -- 172.21.15.162:0/2525593711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7800fcb30 msgr2=0x7ff78010e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:43.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff785f77700 1 --2- 172.21.15.162:0/2525593711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7800fcb30 0x7ff78010e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:43.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff785f77700 1 -- 172.21.15.162:0/2525593711 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c005040 con 0x7ff7800f73a0 2024-02-13T00:38:43.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff786778700 1 --2- 172.21.15.162:0/2525593711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff7800fd490 0x7ff7801081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:43.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff785f77700 1 --2- 172.21.15.162:0/2525593711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7800f73a0 0x7ff78010dfe0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7ff77000e990 tx=0x7ff77000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:43.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff776ffd700 10 client.?.objecter ms_handle_connect 0x7ff7800f73a0 2024-02-13T00:38:43.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff776ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:43.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff776ffd700 1 -- 172.21.15.162:0/2525593711 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff770004680 con 0x7ff7800f73a0 2024-02-13T00:38:43.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff7881db700 1 -- 172.21.15.162:0/2525593711 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7780097c0 con 0x7ff7800f73a0 2024-02-13T00:38:43.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff776ffd700 1 -- 172.21.15.162:0/2525593711 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff77000f040 con 0x7ff7800f73a0 2024-02-13T00:38:43.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff776ffd700 1 -- 172.21.15.162:0/2525593711 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7700146b0 con 0x7ff7800f73a0 2024-02-13T00:38:43.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff7881db700 10 client.14631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:43.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff7881db700 1 -- 172.21.15.162:0/2525593711 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff780108fa0 con 0x7ff7800f73a0 2024-02-13T00:38:43.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.971+0000 7ff7881db700 1 librados: init done 2024-02-13T00:38:43.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.972+0000 7ff7881db700 1 -- 172.21.15.162:0/2525593711 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7800475e0 con 0x7ff7800f73a0 2024-02-13T00:38:43.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.972+0000 7ff776ffd700 1 -- 172.21.15.162:0/2525593711 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff770014ae0 con 0x7ff7800f73a0 2024-02-13T00:38:43.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.973+0000 7ff776ffd700 1 --2- 172.21.15.162:0/2525593711 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff76c062c90 0x7ff76c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:43.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.973+0000 7ff785776700 1 --2- 172.21.15.162:0/2525593711 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff76c062c90 0x7ff76c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:43.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.973+0000 7ff776ffd700 1 -- 172.21.15.162:0/2525593711 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6056+0+0 (secure 0 0 0) 0x7ff77007c540 con 0x7ff7800f73a0 2024-02-13T00:38:43.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.973+0000 7ff776ffd700 10 client.14631.objecter ms_dispatch 0x7ff780001550 osd_map(76..76 src has 1..76) v4 2024-02-13T00:38:43.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.973+0000 7ff776ffd700 3 client.14631.objecter handle_osd_map got epochs [76,76] > 0 2024-02-13T00:38:43.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.973+0000 7ff776ffd700 3 client.14631.objecter handle_osd_map decoding full epoch 76 2024-02-13T00:38:43.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.973+0000 7ff776ffd700 20 client.14631.objecter dump_active .. 0 homeless 2024-02-13T00:38:43.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.973+0000 7ff776ffd700 10 client.14631.objecter ms_handle_connect 0x7ff76c062c90 2024-02-13T00:38:43.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.973+0000 7ff785776700 1 --2- 172.21.15.162:0/2525593711 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff76c062c90 0x7ff76c065150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff780109960 tx=0x7ff77c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:43.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:43.977+0000 7ff776ffd700 1 -- 172.21.15.162:0/2525593711 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff770049f60 con 0x7ff7800f73a0 2024-02-13T00:38:44.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:42.166176+0000 mgr.y (mgr.14544) 36 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:44.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:42.166176+0000 mgr.y (mgr.14544) 36 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:44.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.140+0000 7ff7881db700 1 -- 172.21.15.162:0/2525593711 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff78005fb80 con 0x7ff7800f73a0 2024-02-13T00:38:44.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.141+0000 7ff776ffd700 1 -- 172.21.15.162:0/2525593711 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+19063 (secure 0 0 0) 0x7ff770050060 con 0x7ff7800f73a0 2024-02-13T00:38:44.144 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:38:44.144 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":76,"fsid":"d4ee01fc-ca06-11ee-95b9-87774f69a715","created":"2024-02-13T00:31:51.797017+0000","modified":"2024-02-13T00:38:39.210651+0000","last_up_change":"2024-02-13T00:37:33.855829+0000","last_in_change":"2024-02-13T00:37:16.202456+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-13T00:35:34.545390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-13T00:37:39.459980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-13T00:37:39.740219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-13T00:37:41.489292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-13T00:37:43.466304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-13T00:37:45.542442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6801","nonce":2103743338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2103743338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2103743338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6803","nonce":2103743338}]},"public_addr":"172.21.15.162:6801/2103743338","cluster_addr":"172.21.15.162:6802/2103743338","heartbeat_back_addr":"172.21.15.162:6804/2103743338","heartbeat_front_addr":"172.21.15.162:6803/2103743338","state":["exists","up"]},{"osd":1,"uuid":"39e9d2ef-9aa4-45bb-8a47-b0e0ec222886","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6805","nonce":150599780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":150599780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":150599780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6807","nonce":150599780}]},"public_addr":"172.21.15.162:6805/150599780","cluster_addr":"172.21.15.162:6806/150599780","heartbeat_back_addr":"172.21.15.162:6808/150599780","heartbeat_front_addr":"172.21.15.162:6807/150599780","state":["exists","up"]},{"osd":2,"uuid":"86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6809","nonce":3689719376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3689719376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3689719376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6811","nonce":3689719376}]},"public_addr":"172.21.15.162:6809/3689719376","cluster_addr":"172.21.15.162:6810/3689719376","heartbeat_back_addr":"172.21.15.162:6812/3689719376","heartbeat_front_addr":"172.21.15.162:6811/3689719376","state":["exists","up"]},{"osd":3,"uuid":"2e6b9f0e-7a18-4cf8-9331-1a973fadc001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6813","nonce":604971011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":604971011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":604971011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6815","nonce":604971011}]},"public_addr":"172.21.15.162:6813/604971011","cluster_addr":"172.21.15.162:6814/604971011","heartbeat_back_addr":"172.21.15.162:6816/604971011","heartbeat_front_addr":"172.21.15.162:6815/604971011","state":["exists","up"]},{"osd":4,"uuid":"c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3344263388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":3344263388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":3344263388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3344263388}]},"public_addr":"172.21.15.181:6800/3344263388","cluster_addr":"172.21.15.181:6801/3344263388","heartbeat_back_addr":"172.21.15.181:6803/3344263388","heartbeat_front_addr":"172.21.15.181:6802/3344263388","state":["exists","up"]},{"osd":5,"uuid":"6aa2bd68-7f4b-4973-ac49-7fbc57c1a723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2212296748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":2212296748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":2212296748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2212296748}]},"public_addr":"172.21.15.181:6804/2212296748","cluster_addr":"172.21.15.181:6805/2212296748","heartbeat_back_addr":"172.21.15.181:6807/2212296748","heartbeat_front_addr":"172.21.15.181:6806/2212296748","state":["exists","up"]},{"osd":6,"uuid":"791ba814-5284-432e-addc-d6b7f5f3d474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1506640341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1506640341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1506640341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1506640341}]},"public_addr":"172.21.15.181:6808/1506640341","cluster_addr":"172.21.15.181:6809/1506640341","heartbeat_back_addr":"172.21.15.181:6811/1506640341","heartbeat_front_addr":"172.21.15.181:6810/1506640341","state":["exists","up"]},{"osd":7,"uuid":"1f369c65-e663-4eb2-87a5-f2900f15b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1964192772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":1964192772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":1964192772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1964192772}]},"public_addr":"172.21.15.181:6812/1964192772","cluster_addr":"172.21.15.181:6813/1964192772","heartbeat_back_addr":"172.21.15.181:6815/1964192772","heartbeat_front_addr":"172.21.15.181:6814/1964192772","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:38:14.232055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:52.126669+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:53.397185+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:38:37.239351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:28.458200+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:44.566925+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:38:09.041989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:37:31.903163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.162:6800/1909610488":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/2370739667":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/1502590187":"2024-02-14T00:32:47.469889+0000","172.21.15.162:6800/4121673848":"2024-02-14T00:32:47.469889+0000","172.21.15.162:0/2412382714":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/3041562701":"2024-02-14T00:32:47.469889+0000","172.21.15.162:6800/2775563314":"2024-02-14T00:32:24.380625+0000","172.21.15.162:0/3135632190":"2024-02-14T00:32:24.380625+0000","172.21.15.162:0/197307636":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/2653247729":"2024-02-14T00:32:24.380625+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-13T00:38:44.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 10 librados: watch_flush enter 2024-02-13T00:38:44.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 10 librados: watch_flush exit 2024-02-13T00:38:44.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 20 client.14631.objecter shutdown clearing up homeless session... 2024-02-13T00:38:44.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 10 client.14631.objecter successfully canceled tick 2024-02-13T00:38:44.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 -- 172.21.15.162:0/2525593711 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff76c062c90 msgr2=0x7ff76c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:44.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 --2- 172.21.15.162:0/2525593711 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff76c062c90 0x7ff76c065150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff780109960 tx=0x7ff77c00b040).stop 2024-02-13T00:38:44.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 -- 172.21.15.162:0/2525593711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7800f73a0 msgr2=0x7ff78010dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:44.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 --2- 172.21.15.162:0/2525593711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7800f73a0 0x7ff78010dfe0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7ff77000e990 tx=0x7ff77000b040).stop 2024-02-13T00:38:44.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 -- 172.21.15.162:0/2525593711 shutdown_connections 2024-02-13T00:38:44.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 --2- 172.21.15.162:0/2525593711 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff76c062c90 0x7ff76c065150 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:44.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 --2- 172.21.15.162:0/2525593711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff7800fd490 0x7ff7801081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:44.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 --2- 172.21.15.162:0/2525593711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7800f73a0 0x7ff78010dfe0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:44.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 --2- 172.21.15.162:0/2525593711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7800fcb30 0x7ff78010e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:44.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 -- 172.21.15.162:0/2525593711 >> 172.21.15.162:0/2525593711 conn(0x7ff78006eb90 msgr2=0x7ff780060e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:44.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 -- 172.21.15.162:0/2525593711 shutdown_connections 2024-02-13T00:38:44.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 -- 172.21.15.162:0/2525593711 wait complete. 2024-02-13T00:38:44.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:44.143+0000 7ff7881db700 1 librados: shutdown 2024-02-13T00:38:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:42.166176+0000 mgr.y (mgr.14544) 36 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:45.205 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:44.141345+0000 mon.a (mon.0) 669 : audit [DBG] from='client.? 172.21.15.162:0/2525593711' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:38:45.306 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:44.141345+0000 mon.a (mon.0) 669 : audit [DBG] from='client.? 172.21.15.162:0/2525593711' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:38:45.307 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:44.141345+0000 mon.a (mon.0) 669 : audit [DBG] from='client.? 172.21.15.162:0/2525593711' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:38:45.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-13T00:38:45.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:38:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:38:45.600 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:46.075 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:44.166572+0000 mgr.y (mgr.14544) 37 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:38:46.075 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:44.892806+0000 osd.7 (osd.7) 5 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:38:46.075 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:44.893692+0000 osd.7 (osd.7) 6 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:38:46.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:44.166572+0000 mgr.y (mgr.14544) 37 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:38:46.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:44.892806+0000 osd.7 (osd.7) 5 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:38:46.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:44.893692+0000 osd.7 (osd.7) 6 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:38:46.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:44.166572+0000 mgr.y (mgr.14544) 37 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:38:46.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:44.892806+0000 osd.7 (osd.7) 5 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:38:46.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:44.893692+0000 osd.7 (osd.7) 6 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:38:47.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:45.199673+0000 mgr.y (mgr.14544) 38 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:47.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:45.893261+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:38:47.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:45.896339+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:38:47.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:46.163424+0000 osd.3 (osd.3) 7 : cluster [DBG] 4.14 deep-scrub starts 2024-02-13T00:38:47.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:46.169340+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.14 deep-scrub ok 2024-02-13T00:38:47.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:45.199673+0000 mgr.y (mgr.14544) 38 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:47.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:45.893261+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:38:47.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:45.896339+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:38:47.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:46.163424+0000 osd.3 (osd.3) 7 : cluster [DBG] 4.14 deep-scrub starts 2024-02-13T00:38:47.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:46.169340+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.14 deep-scrub ok 2024-02-13T00:38:47.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:45.199673+0000 mgr.y (mgr.14544) 38 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:45.893261+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:38:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:45.896339+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:38:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:46.163424+0000 osd.3 (osd.3) 7 : cluster [DBG] 4.14 deep-scrub starts 2024-02-13T00:38:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:46.169340+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.14 deep-scrub ok 2024-02-13T00:38:47.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.286+0000 7f1d79253700 1 -- 172.21.15.162:0/2352879002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d740fd490 msgr2=0x7f1d740f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:47.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.286+0000 7f1d79253700 1 --2- 172.21.15.162:0/2352879002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d740fd490 0x7f1d740f6cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1d64004870 tx=0x7f1d64009cb0).stop 2024-02-13T00:38:47.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.286+0000 7f1d79253700 1 -- 172.21.15.162:0/2352879002 shutdown_connections 2024-02-13T00:38:47.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.286+0000 7f1d79253700 1 --2- 172.21.15.162:0/2352879002 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d740f73a0 0x7f1d741039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:47.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.286+0000 7f1d79253700 1 --2- 172.21.15.162:0/2352879002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d740fcb30 0x7f1d740fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:47.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.286+0000 7f1d79253700 1 --2- 172.21.15.162:0/2352879002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d740fd490 0x7f1d740f6cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:47.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.286+0000 7f1d79253700 1 -- 172.21.15.162:0/2352879002 >> 172.21.15.162:0/2352879002 conn(0x7f1d7406eb90 msgr2=0x7f1d740f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.286+0000 7f1d79253700 1 -- 172.21.15.162:0/2352879002 shutdown_connections 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.286+0000 7f1d79253700 1 -- 172.21.15.162:0/2352879002 wait complete. 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.286+0000 7f1d79253700 1 librados: starting msgr at 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d79253700 1 librados: starting objecter 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d79253700 1 Processor -- start 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d79253700 1 -- start start 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d79253700 1 librados: setting wanted keys 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d79253700 1 librados: calling monclient init 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d79253700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d740f73a0 0x7f1d74109120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d79253700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d740fcb30 0x7f1d74109610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d79253700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d740fd490 0x7f1d74109bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d79253700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d74114300 con 0x7f1d740f73a0 2024-02-13T00:38:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d79253700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d74114470 con 0x7f1d740fcb30 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d79253700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d74114630 con 0x7f1d740fd490 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d72d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d740f73a0 0x7f1d74109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d7359e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d740fd490 0x7f1d74109bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d7359e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d740fd490 0x7f1d74109bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48402/0 (socket says 172.21.15.162:48402) 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.287+0000 7f1d7359e700 1 -- 172.21.15.162:0/172828821 learned_addr learned my addr 172.21.15.162:0/172828821 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d7259c700 1 --2- 172.21.15.162:0/172828821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d740fcb30 0x7f1d74109610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d7359e700 1 -- 172.21.15.162:0/172828821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d740fcb30 msgr2=0x7f1d74109610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d7359e700 1 --2- 172.21.15.162:0/172828821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d740fcb30 0x7f1d74109610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d7359e700 1 -- 172.21.15.162:0/172828821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d740f73a0 msgr2=0x7f1d74109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d7359e700 1 --2- 172.21.15.162:0/172828821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d740f73a0 0x7f1d74109120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d7359e700 1 -- 172.21.15.162:0/172828821 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d64005040 con 0x7f1d740fd490 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d72d9d700 1 --2- 172.21.15.162:0/172828821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d740f73a0 0x7f1d74109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d7359e700 1 --2- 172.21.15.162:0/172828821 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d740fd490 0x7f1d74109bf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f1d6c00ec20 tx=0x7f1d6c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:47.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d5bfff700 10 client.?.objecter ms_handle_connect 0x7f1d740fd490 2024-02-13T00:38:47.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d5bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:47.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d5bfff700 1 -- 172.21.15.162:0/172828821 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c009950 con 0x7f1d740fd490 2024-02-13T00:38:47.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d5bfff700 1 -- 172.21.15.162:0/172828821 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d6c00f040 con 0x7f1d740fd490 2024-02-13T00:38:47.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d79253700 1 -- 172.21.15.162:0/172828821 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d741148b0 con 0x7f1d740fd490 2024-02-13T00:38:47.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d5bfff700 1 -- 172.21.15.162:0/172828821 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c00c6e0 con 0x7f1d740fd490 2024-02-13T00:38:47.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d79253700 10 client.24536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:47.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d79253700 1 -- 172.21.15.162:0/172828821 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d74114e20 con 0x7f1d740fd490 2024-02-13T00:38:47.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.288+0000 7f1d79253700 1 librados: init done 2024-02-13T00:38:47.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.289+0000 7f1d79253700 10 client.24536.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.289+0000 7f1d79253700 15 client.24536.objecter _session_command_op_assign -1 1 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.289+0000 7f1d79253700 20 client.24536.objecter _get_session osd=-1 returning homeless 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.289+0000 7f1d79253700 10 client.24536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.289+0000 7f1d79253700 1 -- 172.21.15.162:0/172828821 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1d740475e0 con 0x7f1d740fd490 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.289+0000 7f1d5bfff700 1 -- 172.21.15.162:0/172828821 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1d6c0092e0 con 0x7f1d740fd490 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 1 --2- 172.21.15.162:0/172828821 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1d5c062c90 0x7f1d5c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 1 -- 172.21.15.162:0/172828821 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f1d6c07c720 con 0x7f1d740fd490 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 10 client.24536.objecter ms_dispatch 0x7f1d74001550 osd_map(76..76 src has 1..76) v4 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 3 client.24536.objecter handle_osd_map got epochs [76,76] > 0 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 3 client.24536.objecter handle_osd_map decoding full epoch 76 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d72d9d700 1 --2- 172.21.15.162:0/172828821 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1d5c062c90 0x7f1d5c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 10 client.24536.objecter checking command 1 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 1 --2- 172.21.15.162:0/172828821 >> v2:172.21.15.162:6801/2103743338 conn(0x7f1d5c06c4b0 0x7f1d5c06e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:47.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 20 client.24536.objecter _get_session s=0x7f1d5c06be00 osd=0 3 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 20 client.24536.objecter put_session s=0x7f1d5c06be00 osd=0 3 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 15 client.24536.objecter _session_command_op_remove -1 1 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 20 client.24536.objecter _get_session s=0x7f1d5c06be00 osd=0 3 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 20 client.24536.objecter get_session s=0x7f1d5c06be00 osd=0 3 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 15 client.24536.objecter _session_command_op_assign 0 1 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 20 client.24536.objecter put_session s=0x7f1d5c06be00 osd=0 4 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 10 client.24536.objecter _send_command 1 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 1 -- 172.21.15.162:0/172828821 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d5c06ef90 con 0x7f1d5c06c4b0 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 20 client.24536.objecter dump_active .. 0 homeless 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 1 -- 172.21.15.162:0/172828821 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d6c07ed40 con 0x7f1d740fd490 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d7259c700 1 --2- 172.21.15.162:0/172828821 >> v2:172.21.15.162:6801/2103743338 conn(0x7f1d5c06c4b0 0x7f1d5c06e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 10 client.24536.objecter ms_handle_connect 0x7f1d5c062c90 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d72d9d700 1 --2- 172.21.15.162:0/172828821 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1d5c062c90 0x7f1d5c065150 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f1d6800f2f0 tx=0x7f1d68009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:47.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d5bfff700 10 client.24536.objecter ms_handle_connect 0x7f1d5c06c4b0 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.290+0000 7f1d7259c700 1 --2- 172.21.15.162:0/172828821 >> v2:172.21.15.162:6801/2103743338 conn(0x7f1d5c06c4b0 0x7f1d5c06e950 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.291+0000 7f1d5bfff700 1 -- 172.21.15.162:0/172828821 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1d5c06ef90 con 0x7f1d5c06c4b0 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.291+0000 7f1d5bfff700 10 client.24536.objecter ms_dispatch 0x7f1d74001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.291+0000 7f1d5bfff700 10 client.24536.objecter _finish_command 1 = system:0 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.291+0000 7f1d5bfff700 20 client.24536.objecter put_session s=0x7f1d5c06be00 osd=0 4 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.291+0000 7f1d5bfff700 15 client.24536.objecter _session_command_op_remove 0 1 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 10 client.24536.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 15 client.24536.objecter _session_command_op_assign -1 2 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 20 client.24536.objecter _get_session s=0x7f1d5c06be00 osd=0 3 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 20 client.24536.objecter put_session s=0x7f1d5c06be00 osd=0 3 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 20 client.24536.objecter _get_session s=0x7f1d5c06be00 osd=0 3 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 15 client.24536.objecter _session_command_op_remove -1 2 2024-02-13T00:38:47.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 20 client.24536.objecter get_session s=0x7f1d5c06be00 osd=0 3 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 15 client.24536.objecter _session_command_op_assign 0 2 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 20 client.24536.objecter put_session s=0x7f1d5c06be00 osd=0 4 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 10 client.24536.objecter _send_command 2 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 1 -- 172.21.15.162:0/172828821 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1d74105fe0 con 0x7f1d5c06c4b0 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d5bfff700 1 -- 172.21.15.162:0/172828821 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1d74105fe0 con 0x7f1d5c06c4b0 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d5bfff700 10 client.24536.objecter ms_dispatch 0x7f1d74001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d5bfff700 10 client.24536.objecter _finish_command 2 = system:0 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d5bfff700 20 client.24536.objecter put_session s=0x7f1d5c06be00 osd=0 4 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d5bfff700 15 client.24536.objecter _session_command_op_remove 0 2 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 10 librados: watch_flush enter 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 10 librados: watch_flush exit 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 10 client.24536.objecter close_session for osd.0 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 1 -- 172.21.15.162:0/172828821 >> v2:172.21.15.162:6801/2103743338 conn(0x7f1d5c06c4b0 msgr2=0x7f1d5c06e950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 1 --2- 172.21.15.162:0/172828821 >> v2:172.21.15.162:6801/2103743338 conn(0x7f1d5c06c4b0 0x7f1d5c06e950 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 20 client.24536.objecter put_session s=0x7f1d5c06be00 osd=0 1 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 20 client.24536.objecter shutdown clearing up homeless session... 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 10 client.24536.objecter successfully canceled tick 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 1 -- 172.21.15.162:0/172828821 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1d5c062c90 msgr2=0x7f1d5c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 1 --2- 172.21.15.162:0/172828821 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1d5c062c90 0x7f1d5c065150 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f1d6800f2f0 tx=0x7f1d68009040).stop 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 1 -- 172.21.15.162:0/172828821 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d740fd490 msgr2=0x7f1d74109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 1 --2- 172.21.15.162:0/172828821 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d740fd490 0x7f1d74109bf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f1d6c00ec20 tx=0x7f1d6c00b040).stop 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d7359e700 1 -- 172.21.15.162:0/172828821 reap_dead start 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 1 -- 172.21.15.162:0/172828821 shutdown_connections 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 1 -- 172.21.15.162:0/172828821 >> 172.21.15.162:0/172828821 conn(0x7f1d7406eb90 msgr2=0x7f1d74070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 1 -- 172.21.15.162:0/172828821 shutdown_connections 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 1 -- 172.21.15.162:0/172828821 wait complete. 2024-02-13T00:38:47.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:47.305+0000 7f1d79253700 1 librados: shutdown 2024-02-13T00:38:48.022 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:46.167271+0000 mgr.y (mgr.14544) 39 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:48.022 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:46.926237+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:38:48.022 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:46.927063+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:38:48.022 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:47.206355+0000 osd.3 (osd.3) 9 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:38:48.022 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:47.209995+0000 osd.3 (osd.3) 10 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:38:48.022 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:46.167271+0000 mgr.y (mgr.14544) 39 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:48.022 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:46.926237+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:38:48.022 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:46.927063+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:38:48.022 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:47.206355+0000 osd.3 (osd.3) 9 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:38:48.022 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:47.209995+0000 osd.3 (osd.3) 10 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:38:48.042 INFO:teuthology.orchestra.run.smithi162.stdout:34359738418 2024-02-13T00:38:48.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-13T00:38:48.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:46.167271+0000 mgr.y (mgr.14544) 39 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:46.926237+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:38:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:46.927063+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:38:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:47.206355+0000 osd.3 (osd.3) 9 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:38:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:47.209995+0000 osd.3 (osd.3) 10 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:38:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:49.232 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:47.390306+0000 osd.0 (osd.0) 9 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:38:49.233 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:47.393473+0000 osd.0 (osd.0) 10 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:38:49.233 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:47.601389+0000 osd.7 (osd.7) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:49.233 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:47.601472+0000 osd.7 (osd.7) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:49.233 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:47.879053+0000 mon.a (mon.0) 670 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-13T00:38:49.234 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:47.390306+0000 osd.0 (osd.0) 9 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:38:49.234 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:47.393473+0000 osd.0 (osd.0) 10 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:38:49.234 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:47.601389+0000 osd.7 (osd.7) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:49.234 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:47.601472+0000 osd.7 (osd.7) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:49.234 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:47.879053+0000 mon.a (mon.0) 670 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-13T00:38:49.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:47.390306+0000 osd.0 (osd.0) 9 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:38:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:47.393473+0000 osd.0 (osd.0) 10 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:38:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:47.601389+0000 osd.7 (osd.7) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:47.601472+0000 osd.7 (osd.7) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:47.879053+0000 mon.a (mon.0) 670 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-13T00:38:50.044 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:49 smithi181 systemd[1]: Starting Ceph prometheus.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:38:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:48.167649+0000 mgr.y (mgr.14544) 40 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:38:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:48.377416+0000 osd.4 (osd.4) 5 : cluster [DBG] 2.f scrub starts 2024-02-13T00:38:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:48.379173+0000 osd.4 (osd.4) 6 : cluster [DBG] 2.f scrub ok 2024-02-13T00:38:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:48.617901+0000 osd.5 (osd.5) 7 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:38:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:48.618823+0000 osd.5 (osd.5) 8 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:38:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:49.221365+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-13T00:38:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:49.615070+0000 osd.1 (osd.1) 9 : cluster [DBG] 2.4 deep-scrub starts 2024-02-13T00:38:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:49.617980+0000 osd.1 (osd.1) 10 : cluster [DBG] 2.4 deep-scrub ok 2024-02-13T00:38:50.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.078+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3098590169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e380f73a0 msgr2=0x7f3e381039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:50.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.078+0000 7f3e3fed2700 1 --2- 172.21.15.162:0/3098590169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e380f73a0 0x7f3e381039a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3e30009be0 tx=0x7f3e30004d10).stop 2024-02-13T00:38:50.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.078+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3098590169 shutdown_connections 2024-02-13T00:38:50.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.078+0000 7f3e3fed2700 1 --2- 172.21.15.162:0/3098590169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e380f73a0 0x7f3e381039a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:50.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.078+0000 7f3e3fed2700 1 --2- 172.21.15.162:0/3098590169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e380fcb30 0x7f3e380fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:50.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.078+0000 7f3e3fed2700 1 --2- 172.21.15.162:0/3098590169 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3e380fd490 0x7f3e380f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:50.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.078+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3098590169 >> 172.21.15.162:0/3098590169 conn(0x7f3e3806eb90 msgr2=0x7f3e380f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:50.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.078+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3098590169 shutdown_connections 2024-02-13T00:38:50.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.079+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3098590169 wait complete. 2024-02-13T00:38:50.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.079+0000 7f3e3fed2700 1 librados: starting msgr at 2024-02-13T00:38:50.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.079+0000 7f3e3fed2700 1 librados: starting objecter 2024-02-13T00:38:50.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.079+0000 7f3e3fed2700 1 Processor -- start 2024-02-13T00:38:50.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.079+0000 7f3e3fed2700 1 -- start start 2024-02-13T00:38:50.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.079+0000 7f3e3fed2700 1 librados: setting wanted keys 2024-02-13T00:38:50.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.079+0000 7f3e3fed2700 1 librados: calling monclient init 2024-02-13T00:38:50.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3fed2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e380f73a0 0x7f3e380f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:50.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3fed2700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3e380fcb30 0x7f3e380f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:50.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3fed2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e380fd490 0x7f3e380f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:50.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3fed2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e38116390 con 0x7f3e380fd490 2024-02-13T00:38:50.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3fed2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e38116500 con 0x7f3e380f73a0 2024-02-13T00:38:50.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3fed2700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e381166c0 con 0x7f3e380fcb30 2024-02-13T00:38:50.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3d46d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3e380fcb30 0x7f3e380f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:50.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3e46f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e380fd490 0x7f3e380f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:50.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3d46d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3e380fcb30 0x7f3e380f8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48432/0 (socket says 172.21.15.162:48432) 2024-02-13T00:38:50.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3e46f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e380fd490 0x7f3e380f9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60498/0 (socket says 172.21.15.162:60498) 2024-02-13T00:38:50.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3d46d700 1 -- 172.21.15.162:0/3572398382 learned_addr learned my addr 172.21.15.162:0/3572398382 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:50.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3dc6e700 1 --2- 172.21.15.162:0/3572398382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e380f73a0 0x7f3e380f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:50.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3e46f700 1 -- 172.21.15.162:0/3572398382 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3e380fcb30 msgr2=0x7f3e380f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:50.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3e46f700 1 --2- 172.21.15.162:0/3572398382 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3e380fcb30 0x7f3e380f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:50.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3e46f700 1 -- 172.21.15.162:0/3572398382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e380f73a0 msgr2=0x7f3e380f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:50.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3e46f700 1 --2- 172.21.15.162:0/3572398382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e380f73a0 0x7f3e380f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:50.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3e46f700 1 -- 172.21.15.162:0/3572398382 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e34005040 con 0x7f3e380fd490 2024-02-13T00:38:50.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3d46d700 1 --2- 172.21.15.162:0/3572398382 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3e380fcb30 0x7f3e380f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:50.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3e46f700 1 --2- 172.21.15.162:0/3572398382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e380fd490 0x7f3e380f9310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f3e3000b6d0 tx=0x7f3e30004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:50.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e2effd700 10 client.?.objecter ms_handle_connect 0x7f3e380fd490 2024-02-13T00:38:50.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e2effd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:50.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e2effd700 1 -- 172.21.15.162:0/3572398382 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3000e040 con 0x7f3e380fd490 2024-02-13T00:38:50.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3572398382 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e300097c0 con 0x7f3e380fd490 2024-02-13T00:38:50.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e2effd700 1 -- 172.21.15.162:0/3572398382 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e30018040 con 0x7f3e380fd490 2024-02-13T00:38:50.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e2effd700 1 -- 172.21.15.162:0/3572398382 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3001c3f0 con 0x7f3e380fd490 2024-02-13T00:38:50.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3fed2700 10 client.14643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:50.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3572398382 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e38116c30 con 0x7f3e380fd490 2024-02-13T00:38:50.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.080+0000 7f3e3fed2700 1 librados: init done 2024-02-13T00:38:50.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.081+0000 7f3e3fed2700 10 client.14643.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:38:50.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.081+0000 7f3e3fed2700 15 client.14643.objecter _session_command_op_assign -1 1 2024-02-13T00:38:50.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.081+0000 7f3e3fed2700 20 client.14643.objecter _get_session osd=-1 returning homeless 2024-02-13T00:38:50.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.081+0000 7f3e3fed2700 10 client.14643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:50.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.081+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3572398382 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3e380475e0 con 0x7f3e380fd490 2024-02-13T00:38:50.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 1 -- 172.21.15.162:0/3572398382 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f3e30013420 con 0x7f3e380fd490 2024-02-13T00:38:50.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 1 --2- 172.21.15.162:0/3572398382 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3e24062c90 0x7f3e24065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:50.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 1 -- 172.21.15.162:0/3572398382 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f3e3007e200 con 0x7f3e380fd490 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 10 client.14643.objecter ms_dispatch 0x7f3e38001550 osd_map(78..78 src has 1..78) v4 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 3 client.14643.objecter handle_osd_map got epochs [78,78] > 0 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 3 client.14643.objecter handle_osd_map decoding full epoch 78 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e3dc6e700 1 --2- 172.21.15.162:0/3572398382 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3e24062c90 0x7f3e24065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 10 client.14643.objecter checking command 1 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 1 --2- 172.21.15.162:0/3572398382 >> v2:172.21.15.162:6805/150599780 conn(0x7f3e2406c4b0 0x7f3e2406e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 20 client.14643.objecter _get_session s=0x7f3e2406be00 osd=1 3 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 20 client.14643.objecter put_session s=0x7f3e2406be00 osd=1 3 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 15 client.14643.objecter _session_command_op_remove -1 1 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 20 client.14643.objecter _get_session s=0x7f3e2406be00 osd=1 3 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 20 client.14643.objecter get_session s=0x7f3e2406be00 osd=1 3 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 15 client.14643.objecter _session_command_op_assign 1 1 2024-02-13T00:38:50.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 20 client.14643.objecter put_session s=0x7f3e2406be00 osd=1 4 2024-02-13T00:38:50.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 10 client.14643.objecter _send_command 1 2024-02-13T00:38:50.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 1 -- 172.21.15.162:0/3572398382 --> v2:172.21.15.162:6805/150599780 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e2406ef90 con 0x7f3e2406c4b0 2024-02-13T00:38:50.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 20 client.14643.objecter dump_active .. 0 homeless 2024-02-13T00:38:50.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e2effd700 1 -- 172.21.15.162:0/3572398382 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e3007e5b0 con 0x7f3e380fd490 2024-02-13T00:38:50.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.082+0000 7f3e3d46d700 1 --2- 172.21.15.162:0/3572398382 >> v2:172.21.15.162:6805/150599780 conn(0x7f3e2406c4b0 0x7f3e2406e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:50.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.083+0000 7f3e3dc6e700 1 --2- 172.21.15.162:0/3572398382 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3e24062c90 0x7f3e24065150 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f3e28004b60 tx=0x7f3e28009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:50.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.083+0000 7f3e2effd700 10 client.14643.objecter ms_handle_connect 0x7f3e24062c90 2024-02-13T00:38:50.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.083+0000 7f3e2effd700 10 client.14643.objecter ms_handle_connect 0x7f3e2406c4b0 2024-02-13T00:38:50.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.083+0000 7f3e3d46d700 1 --2- 172.21.15.162:0/3572398382 >> v2:172.21.15.162:6805/150599780 conn(0x7f3e2406c4b0 0x7f3e2406e950 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:50.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.083+0000 7f3e2effd700 1 -- 172.21.15.162:0/3572398382 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3e2406ef90 con 0x7f3e2406c4b0 2024-02-13T00:38:50.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.083+0000 7f3e2effd700 10 client.14643.objecter ms_dispatch 0x7f3e38001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:38:50.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.083+0000 7f3e2effd700 10 client.14643.objecter _finish_command 1 = system:0 2024-02-13T00:38:50.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.083+0000 7f3e2effd700 20 client.14643.objecter put_session s=0x7f3e2406be00 osd=1 4 2024-02-13T00:38:50.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.083+0000 7f3e2effd700 15 client.14643.objecter _session_command_op_remove 1 1 2024-02-13T00:38:50.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.097+0000 7f3e3fed2700 10 client.14643.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:38:50.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.097+0000 7f3e3fed2700 15 client.14643.objecter _session_command_op_assign -1 2 2024-02-13T00:38:50.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.097+0000 7f3e3fed2700 20 client.14643.objecter _get_session s=0x7f3e2406be00 osd=1 3 2024-02-13T00:38:50.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.097+0000 7f3e3fed2700 20 client.14643.objecter put_session s=0x7f3e2406be00 osd=1 3 2024-02-13T00:38:50.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.097+0000 7f3e3fed2700 20 client.14643.objecter _get_session s=0x7f3e2406be00 osd=1 3 2024-02-13T00:38:50.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.097+0000 7f3e3fed2700 15 client.14643.objecter _session_command_op_remove -1 2 2024-02-13T00:38:50.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.097+0000 7f3e3fed2700 20 client.14643.objecter get_session s=0x7f3e2406be00 osd=1 3 2024-02-13T00:38:50.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.097+0000 7f3e3fed2700 15 client.14643.objecter _session_command_op_assign 1 2 2024-02-13T00:38:50.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.097+0000 7f3e3fed2700 20 client.14643.objecter put_session s=0x7f3e2406be00 osd=1 4 2024-02-13T00:38:50.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.097+0000 7f3e3fed2700 10 client.14643.objecter _send_command 2 2024-02-13T00:38:50.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.097+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3572398382 --> v2:172.21.15.162:6805/150599780 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3e38106090 con 0x7f3e2406c4b0 2024-02-13T00:38:50.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e2effd700 1 -- 172.21.15.162:0/3572398382 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3e38106090 con 0x7f3e2406c4b0 2024-02-13T00:38:50.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e2effd700 10 client.14643.objecter ms_dispatch 0x7f3e38001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:38:50.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e2effd700 10 client.14643.objecter _finish_command 2 = system:0 2024-02-13T00:38:50.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e2effd700 20 client.14643.objecter put_session s=0x7f3e2406be00 osd=1 4 2024-02-13T00:38:50.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e2effd700 15 client.14643.objecter _session_command_op_remove 1 2 2024-02-13T00:38:50.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 10 librados: watch_flush enter 2024-02-13T00:38:50.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 10 librados: watch_flush exit 2024-02-13T00:38:50.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 10 client.14643.objecter close_session for osd.1 2024-02-13T00:38:50.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3572398382 >> v2:172.21.15.162:6805/150599780 conn(0x7f3e2406c4b0 msgr2=0x7f3e2406e950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:50.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 1 --2- 172.21.15.162:0/3572398382 >> v2:172.21.15.162:6805/150599780 conn(0x7f3e2406c4b0 0x7f3e2406e950 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 20 client.14643.objecter put_session s=0x7f3e2406be00 osd=1 1 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 20 client.14643.objecter shutdown clearing up homeless session... 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 10 client.14643.objecter successfully canceled tick 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3572398382 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3e24062c90 msgr2=0x7f3e24065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 1 --2- 172.21.15.162:0/3572398382 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3e24062c90 0x7f3e24065150 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f3e28004b60 tx=0x7f3e28009040).stop 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3572398382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e380fd490 msgr2=0x7f3e380f9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 1 --2- 172.21.15.162:0/3572398382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e380fd490 0x7f3e380f9310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f3e3000b6d0 tx=0x7f3e30004d10).stop 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3e46f700 1 -- 172.21.15.162:0/3572398382 reap_dead start 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3572398382 shutdown_connections 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3572398382 >> 172.21.15.162:0/3572398382 conn(0x7f3e3806eb90 msgr2=0x7f3e38070160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3572398382 shutdown_connections 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 1 -- 172.21.15.162:0/3572398382 wait complete. 2024-02-13T00:38:50.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:50.098+0000 7f3e3fed2700 1 librados: shutdown 2024-02-13T00:38:50.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:48.167649+0000 mgr.y (mgr.14544) 40 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:38:50.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:48.377416+0000 osd.4 (osd.4) 5 : cluster [DBG] 2.f scrub starts 2024-02-13T00:38:50.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:48.379173+0000 osd.4 (osd.4) 6 : cluster [DBG] 2.f scrub ok 2024-02-13T00:38:50.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:48.617901+0000 osd.5 (osd.5) 7 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:38:50.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:48.618823+0000 osd.5 (osd.5) 8 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:38:50.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:49.221365+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-13T00:38:50.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:49.615070+0000 osd.1 (osd.1) 9 : cluster [DBG] 2.4 deep-scrub starts 2024-02-13T00:38:50.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:49.617980+0000 osd.1 (osd.1) 10 : cluster [DBG] 2.4 deep-scrub ok 2024-02-13T00:38:50.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:48.167649+0000 mgr.y (mgr.14544) 40 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:38:50.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:48.377416+0000 osd.4 (osd.4) 5 : cluster [DBG] 2.f scrub starts 2024-02-13T00:38:50.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:48.379173+0000 osd.4 (osd.4) 6 : cluster [DBG] 2.f scrub ok 2024-02-13T00:38:50.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:48.617901+0000 osd.5 (osd.5) 7 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:38:50.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:48.618823+0000 osd.5 (osd.5) 8 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:38:50.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:49.221365+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-13T00:38:50.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:49.615070+0000 osd.1 (osd.1) 9 : cluster [DBG] 2.4 deep-scrub starts 2024-02-13T00:38:50.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:49.617980+0000 osd.1 (osd.1) 10 : cluster [DBG] 2.4 deep-scrub ok 2024-02-13T00:38:50.453 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.374Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-13T00:38:50.454 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.374Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-13T00:38:50.454 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.374Z caller=main.go:518 level=info host_details="(Linux 4.18.0-536.el8.x86_64 #1 SMP Thu Jan 18 15:15:29 UTC 2024 x86_64 smithi181 (none))" 2024-02-13T00:38:50.454 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.374Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-13T00:38:50.454 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.374Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-13T00:38:50.454 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.375Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-13T00:38:50.454 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.376Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-13T00:38:50.454 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.377Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-13T00:38:50.454 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.379Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-13T00:38:50.454 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.379Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.508µs 2024-02-13T00:38:50.455 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.379Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-13T00:38:50.455 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.379Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-13T00:38:50.455 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.379Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=34.554µs wal_replay_duration=104.978µs total_replay_duration=153.315µs 2024-02-13T00:38:50.455 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.379Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-13T00:38:50.455 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.379Z caller=main.go:947 level=info msg="TSDB started" 2024-02-13T00:38:50.455 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.380Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-13T00:38:50.455 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.394Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.644522ms db_storage=551ns remote_storage=1.043µs web_handler=527ns query_engine=583ns scrape=428.475µs scrape_sd=21.104µs notify=18.034µs notify_sd=5.591µs rules=13.780685ms 2024-02-13T00:38:50.455 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:38:50.394Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-13T00:38:50.455 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 bash[64561]: 3a26c33fc0550e616c14854754b340a50a1924af58e770444e5d1c55fb7983fa 2024-02-13T00:38:50.755 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:38:50] ENGINE Bus STOPPING 2024-02-13T00:38:50.755 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:38:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T00:38:50.755 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:38:50] ENGINE Bus STOPPED 2024-02-13T00:38:50.755 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:38:50] ENGINE Bus STARTING 2024-02-13T00:38:50.773 INFO:teuthology.orchestra.run.smithi162.stdout:55834574894 2024-02-13T00:38:50.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-13T00:38:50.794 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:38:50 smithi181 systemd[1]: Started Ceph prometheus.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:38:51.048 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:51.079 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:38:50] ENGINE Serving on http://:::9283 2024-02-13T00:38:51.079 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:38:50] ENGINE Bus STARTED 2024-02-13T00:38:51.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:49.499806+0000 osd.5 (osd.5) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:51.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:49.499871+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:51.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:49.605360+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:38:51.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:49.606350+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:38:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:50.168294+0000 mgr.y (mgr.14544) 41 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:50.229949+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-13T00:38:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:50.514677+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:50.661316+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:50.665732+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:50.668234+0000 mon.c (mon.2) 53 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T00:38:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:50.668614+0000 mgr.y (mgr.14544) 42 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T00:38:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:50.672494+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:38:50.680605+0000 mgr.y (mgr.14544) 43 : cephadm [INF] Deploying daemon grafana.a on smithi181 2024-02-13T00:38:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:49.499806+0000 osd.5 (osd.5) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:49.499871+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:49.605360+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:38:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:49.606350+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:38:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:50.168294+0000 mgr.y (mgr.14544) 41 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:50.229949+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-13T00:38:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:50.514677+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:50.661316+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:50.665732+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:50.668234+0000 mon.c (mon.2) 53 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T00:38:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:50.668614+0000 mgr.y (mgr.14544) 42 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T00:38:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:50.672494+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:38:50.680605+0000 mgr.y (mgr.14544) 43 : cephadm [INF] Deploying daemon grafana.a on smithi181 2024-02-13T00:38:51.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:49.499806+0000 osd.5 (osd.5) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:51.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:49.499871+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:51.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:49.605360+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:38:51.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:49.606350+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:38:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:50.168294+0000 mgr.y (mgr.14544) 41 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:38:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:50.229949+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-13T00:38:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:50.514677+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:50.661316+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:50.665732+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:50.668234+0000 mon.c (mon.2) 53 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T00:38:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:50.668614+0000 mgr.y (mgr.14544) 42 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T00:38:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:50.672494+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:38:50.680605+0000 mgr.y (mgr.14544) 43 : cephadm [INF] Deploying daemon grafana.a on smithi181 2024-02-13T00:38:52.255 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:50.862863+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:38:52.255 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:50.866319+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:38:52.255 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:50.862863+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:38:52.255 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:50.866319+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:38:52.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:50.862863+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:38:52.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:50.866319+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:38:52.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.769+0000 7f414beb4700 1 -- 172.21.15.162:0/4004317842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41440f73a0 msgr2=0x7f41441039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:52.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.769+0000 7f414beb4700 1 --2- 172.21.15.162:0/4004317842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41440f73a0 0x7f41441039a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f413c009be0 tx=0x7f413c004d10).stop 2024-02-13T00:38:52.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.769+0000 7f414beb4700 1 -- 172.21.15.162:0/4004317842 shutdown_connections 2024-02-13T00:38:52.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.769+0000 7f414beb4700 1 --2- 172.21.15.162:0/4004317842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41440f73a0 0x7f41441039a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:52.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.769+0000 7f414beb4700 1 --2- 172.21.15.162:0/4004317842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41440fcb30 0x7f41440fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:52.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.769+0000 7f414beb4700 1 --2- 172.21.15.162:0/4004317842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41440fd490 0x7f41440f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.769+0000 7f414beb4700 1 -- 172.21.15.162:0/4004317842 >> 172.21.15.162:0/4004317842 conn(0x7f414406eb90 msgr2=0x7f41440f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 -- 172.21.15.162:0/4004317842 shutdown_connections 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 -- 172.21.15.162:0/4004317842 wait complete. 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 librados: starting msgr at 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 librados: starting objecter 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 Processor -- start 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 -- start start 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 librados: setting wanted keys 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 librados: calling monclient init 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41440f73a0 0x7f4144109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41440fcb30 0x7f41441096f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41440fd490 0x7f4144109c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4144114390 con 0x7f41440f73a0 2024-02-13T00:38:52.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4144114500 con 0x7f41440fd490 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f414beb4700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41441146c0 con 0x7f41440fcb30 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f4149c50700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41440f73a0 0x7f4144109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f4149c50700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41440f73a0 0x7f4144109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60512/0 (socket says 172.21.15.162:60512) 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.770+0000 7f4149c50700 1 -- 172.21.15.162:0/1445747110 learned_addr learned my addr 172.21.15.162:0/1445747110 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f414944f700 1 --2- 172.21.15.162:0/1445747110 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41440fcb30 0x7f41441096f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f414a451700 1 --2- 172.21.15.162:0/1445747110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41440fd490 0x7f4144109c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f4149c50700 1 -- 172.21.15.162:0/1445747110 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41440fcb30 msgr2=0x7f41441096f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f4149c50700 1 --2- 172.21.15.162:0/1445747110 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41440fcb30 0x7f41441096f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f4149c50700 1 -- 172.21.15.162:0/1445747110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41440fd490 msgr2=0x7f4144109c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f4149c50700 1 --2- 172.21.15.162:0/1445747110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41440fd490 0x7f4144109c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f4149c50700 1 -- 172.21.15.162:0/1445747110 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4140005040 con 0x7f41440f73a0 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f414a451700 1 --2- 172.21.15.162:0/1445747110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41440fd490 0x7f4144109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f4149c50700 1 --2- 172.21.15.162:0/1445747110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41440f73a0 0x7f4144109200 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f41400048f0 tx=0x7f4140009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:52.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f413affd700 10 client.?.objecter ms_handle_connect 0x7f41440f73a0 2024-02-13T00:38:52.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f413affd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:52.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f413affd700 1 -- 172.21.15.162:0/1445747110 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4140007b00 con 0x7f41440f73a0 2024-02-13T00:38:52.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f413affd700 1 -- 172.21.15.162:0/1445747110 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f414000f040 con 0x7f41440f73a0 2024-02-13T00:38:52.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f414beb4700 1 -- 172.21.15.162:0/1445747110 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413c0097c0 con 0x7f41440f73a0 2024-02-13T00:38:52.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f413affd700 1 -- 172.21.15.162:0/1445747110 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4140014990 con 0x7f41440f73a0 2024-02-13T00:38:52.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f414beb4700 10 client.14649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:52.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f414beb4700 1 -- 172.21.15.162:0/1445747110 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4144114d00 con 0x7f41440f73a0 2024-02-13T00:38:52.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.771+0000 7f414beb4700 1 librados: init done 2024-02-13T00:38:52.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.772+0000 7f414beb4700 10 client.14649.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.772+0000 7f414beb4700 15 client.14649.objecter _session_command_op_assign -1 1 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.772+0000 7f414beb4700 20 client.14649.objecter _get_session osd=-1 returning homeless 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.772+0000 7f414beb4700 10 client.14649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.772+0000 7f414beb4700 1 -- 172.21.15.162:0/1445747110 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f41440475e0 con 0x7f41440f73a0 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.772+0000 7f413affd700 1 -- 172.21.15.162:0/1445747110 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4140010500 con 0x7f41440f73a0 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 1 --2- 172.21.15.162:0/1445747110 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4130062c90 0x7f4130065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 1 -- 172.21.15.162:0/1445747110 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f414007c830 con 0x7f41440f73a0 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 10 client.14649.objecter ms_dispatch 0x7f4144001550 osd_map(79..79 src has 1..79) v4 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 3 client.14649.objecter handle_osd_map got epochs [79,79] > 0 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 3 client.14649.objecter handle_osd_map decoding full epoch 79 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f414944f700 1 --2- 172.21.15.162:0/1445747110 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4130062c90 0x7f4130065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 10 client.14649.objecter checking command 1 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 1 --2- 172.21.15.162:0/1445747110 >> v2:172.21.15.162:6809/3689719376 conn(0x7f413006c4b0 0x7f413006e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:52.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 20 client.14649.objecter _get_session s=0x7f413006be00 osd=2 3 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 20 client.14649.objecter put_session s=0x7f413006be00 osd=2 3 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 15 client.14649.objecter _session_command_op_remove -1 1 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 20 client.14649.objecter _get_session s=0x7f413006be00 osd=2 3 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 20 client.14649.objecter get_session s=0x7f413006be00 osd=2 3 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 15 client.14649.objecter _session_command_op_assign 2 1 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 20 client.14649.objecter put_session s=0x7f413006be00 osd=2 4 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 10 client.14649.objecter _send_command 1 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 1 -- 172.21.15.162:0/1445747110 --> v2:172.21.15.162:6809/3689719376 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f413006ef90 con 0x7f413006c4b0 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 20 client.14649.objecter dump_active .. 0 homeless 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 1 -- 172.21.15.162:0/1445747110 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f414007cbe0 con 0x7f41440f73a0 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f414a451700 1 --2- 172.21.15.162:0/1445747110 >> v2:172.21.15.162:6809/3689719376 conn(0x7f413006c4b0 0x7f413006e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 10 client.14649.objecter ms_handle_connect 0x7f4130062c90 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f414944f700 1 --2- 172.21.15.162:0/1445747110 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4130062c90 0x7f4130065150 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4144061760 tx=0x7f413400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f413affd700 10 client.14649.objecter ms_handle_connect 0x7f413006c4b0 2024-02-13T00:38:52.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.773+0000 7f414a451700 1 --2- 172.21.15.162:0/1445747110 >> v2:172.21.15.162:6809/3689719376 conn(0x7f413006c4b0 0x7f413006e950 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:52.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.774+0000 7f413affd700 1 -- 172.21.15.162:0/1445747110 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f413006ef90 con 0x7f413006c4b0 2024-02-13T00:38:52.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.774+0000 7f413affd700 10 client.14649.objecter ms_dispatch 0x7f4144001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:38:52.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.774+0000 7f413affd700 10 client.14649.objecter _finish_command 1 = system:0 2024-02-13T00:38:52.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.774+0000 7f413affd700 20 client.14649.objecter put_session s=0x7f413006be00 osd=2 4 2024-02-13T00:38:52.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.774+0000 7f413affd700 15 client.14649.objecter _session_command_op_remove 2 1 2024-02-13T00:38:52.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f414beb4700 10 client.14649.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:38:52.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f414beb4700 15 client.14649.objecter _session_command_op_assign -1 2 2024-02-13T00:38:52.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f414beb4700 20 client.14649.objecter _get_session s=0x7f413006be00 osd=2 3 2024-02-13T00:38:52.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f414beb4700 20 client.14649.objecter put_session s=0x7f413006be00 osd=2 3 2024-02-13T00:38:52.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f414beb4700 20 client.14649.objecter _get_session s=0x7f413006be00 osd=2 3 2024-02-13T00:38:52.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f414beb4700 15 client.14649.objecter _session_command_op_remove -1 2 2024-02-13T00:38:52.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f414beb4700 20 client.14649.objecter get_session s=0x7f413006be00 osd=2 3 2024-02-13T00:38:52.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f414beb4700 15 client.14649.objecter _session_command_op_assign 2 2 2024-02-13T00:38:52.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f414beb4700 20 client.14649.objecter put_session s=0x7f413006be00 osd=2 4 2024-02-13T00:38:52.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f414beb4700 10 client.14649.objecter _send_command 2 2024-02-13T00:38:52.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f414beb4700 1 -- 172.21.15.162:0/1445747110 --> v2:172.21.15.162:6809/3689719376 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4144106090 con 0x7f413006c4b0 2024-02-13T00:38:52.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f413affd700 1 -- 172.21.15.162:0/1445747110 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4144106090 con 0x7f413006c4b0 2024-02-13T00:38:52.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f413affd700 10 client.14649.objecter ms_dispatch 0x7f4144001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:38:52.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f413affd700 10 client.14649.objecter _finish_command 2 = system:0 2024-02-13T00:38:52.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f413affd700 20 client.14649.objecter put_session s=0x7f413006be00 osd=2 4 2024-02-13T00:38:52.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.788+0000 7f413affd700 15 client.14649.objecter _session_command_op_remove 2 2 2024-02-13T00:38:52.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 10 librados: watch_flush enter 2024-02-13T00:38:52.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 10 librados: watch_flush exit 2024-02-13T00:38:52.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 10 client.14649.objecter close_session for osd.2 2024-02-13T00:38:52.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 1 -- 172.21.15.162:0/1445747110 >> v2:172.21.15.162:6809/3689719376 conn(0x7f413006c4b0 msgr2=0x7f413006e950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 1 --2- 172.21.15.162:0/1445747110 >> v2:172.21.15.162:6809/3689719376 conn(0x7f413006c4b0 0x7f413006e950 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 20 client.14649.objecter put_session s=0x7f413006be00 osd=2 1 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 20 client.14649.objecter shutdown clearing up homeless session... 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 10 client.14649.objecter successfully canceled tick 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 1 -- 172.21.15.162:0/1445747110 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4130062c90 msgr2=0x7f4130065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 1 --2- 172.21.15.162:0/1445747110 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4130062c90 0x7f4130065150 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4144061760 tx=0x7f413400b040).stop 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 1 -- 172.21.15.162:0/1445747110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41440f73a0 msgr2=0x7f4144109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 1 --2- 172.21.15.162:0/1445747110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41440f73a0 0x7f4144109200 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f41400048f0 tx=0x7f4140009cb0).stop 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414a451700 1 -- 172.21.15.162:0/1445747110 reap_dead start 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 1 -- 172.21.15.162:0/1445747110 shutdown_connections 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 1 -- 172.21.15.162:0/1445747110 >> 172.21.15.162:0/1445747110 conn(0x7f414406eb90 msgr2=0x7f414406fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 1 -- 172.21.15.162:0/1445747110 shutdown_connections 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 1 -- 172.21.15.162:0/1445747110 wait complete. 2024-02-13T00:38:52.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:52.789+0000 7f414beb4700 1 librados: shutdown 2024-02-13T00:38:53.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.168804+0000 mgr.y (mgr.14544) 44 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:53.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.168804+0000 mgr.y (mgr.14544) 44 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:53.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.168804+0000 mgr.y (mgr.14544) 44 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:53.383 INFO:teuthology.orchestra.run.smithi162.stdout:77309411370 2024-02-13T00:38:53.383 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-13T00:38:53.649 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:54.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.374756+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.1e deep-scrub starts 2024-02-13T00:38:54.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.380364+0000 osd.0 (osd.0) 12 : cluster [DBG] 4.1e deep-scrub ok 2024-02-13T00:38:54.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.385194+0000 osd.4 (osd.4) 7 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:38:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.388089+0000 osd.4 (osd.4) 8 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:38:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.490338+0000 osd.6 (osd.6) 5 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:38:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.493233+0000 osd.6 (osd.6) 6 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:38:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.615802+0000 osd.5 (osd.5) 13 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:38:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.618692+0000 osd.5 (osd.5) 14 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:38:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.678304+0000 osd.4 (osd.4) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.678352+0000 osd.4 (osd.4) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.856153+0000 osd.7 (osd.7) 15 : cluster [DBG] 5.1a scrub starts 2024-02-13T00:38:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.859082+0000 osd.7 (osd.7) 16 : cluster [DBG] 5.1a scrub ok 2024-02-13T00:38:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:52.907437+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-13T00:38:54.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.374756+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.1e deep-scrub starts 2024-02-13T00:38:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.380364+0000 osd.0 (osd.0) 12 : cluster [DBG] 4.1e deep-scrub ok 2024-02-13T00:38:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.385194+0000 osd.4 (osd.4) 7 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:38:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.388089+0000 osd.4 (osd.4) 8 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:38:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.490338+0000 osd.6 (osd.6) 5 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:38:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.493233+0000 osd.6 (osd.6) 6 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:38:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.615802+0000 osd.5 (osd.5) 13 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:38:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.618692+0000 osd.5 (osd.5) 14 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:38:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.678304+0000 osd.4 (osd.4) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.678352+0000 osd.4 (osd.4) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.856153+0000 osd.7 (osd.7) 15 : cluster [DBG] 5.1a scrub starts 2024-02-13T00:38:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.859082+0000 osd.7 (osd.7) 16 : cluster [DBG] 5.1a scrub ok 2024-02-13T00:38:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:52.907437+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-13T00:38:54.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.374756+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.1e deep-scrub starts 2024-02-13T00:38:54.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.380364+0000 osd.0 (osd.0) 12 : cluster [DBG] 4.1e deep-scrub ok 2024-02-13T00:38:54.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.385194+0000 osd.4 (osd.4) 7 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:38:54.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.388089+0000 osd.4 (osd.4) 8 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:38:54.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.490338+0000 osd.6 (osd.6) 5 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:38:54.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.493233+0000 osd.6 (osd.6) 6 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:38:54.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.615802+0000 osd.5 (osd.5) 13 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:38:54.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.618692+0000 osd.5 (osd.5) 14 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:38:54.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.678304+0000 osd.4 (osd.4) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:38:54.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.678352+0000 osd.4 (osd.4) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:38:54.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.856153+0000 osd.7 (osd.7) 15 : cluster [DBG] 5.1a scrub starts 2024-02-13T00:38:54.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.859082+0000 osd.7 (osd.7) 16 : cluster [DBG] 5.1a scrub ok 2024-02-13T00:38:54.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:52.907437+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-13T00:38:55.212 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:53.597530+0000 osd.5 (osd.5) 15 : cluster [DBG] 6.19 deep-scrub starts 2024-02-13T00:38:55.212 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:53.600420+0000 osd.5 (osd.5) 16 : cluster [DBG] 6.19 deep-scrub ok 2024-02-13T00:38:55.212 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:54.169159+0000 mgr.y (mgr.14544) 45 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:55.212 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:54.350822+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:55.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:53.597530+0000 osd.5 (osd.5) 15 : cluster [DBG] 6.19 deep-scrub starts 2024-02-13T00:38:55.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:53.600420+0000 osd.5 (osd.5) 16 : cluster [DBG] 6.19 deep-scrub ok 2024-02-13T00:38:55.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:54.169159+0000 mgr.y (mgr.14544) 45 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:55.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:54.350822+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:55.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:53.597530+0000 osd.5 (osd.5) 15 : cluster [DBG] 6.19 deep-scrub starts 2024-02-13T00:38:55.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:53.600420+0000 osd.5 (osd.5) 16 : cluster [DBG] 6.19 deep-scrub ok 2024-02-13T00:38:55.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:54.169159+0000 mgr.y (mgr.14544) 45 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:38:55.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:54.350822+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:38:55.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:38:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:38:55.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.672+0000 7f3597ecc700 1 -- 172.21.15.162:0/3314831578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35900f84c0 msgr2=0x7f3590104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:55.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.672+0000 7f3597ecc700 1 --2- 172.21.15.162:0/3314831578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35900f84c0 0x7f3590104ac0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f3588009be0 tx=0x7f3588004d10).stop 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.672+0000 7f3597ecc700 1 -- 172.21.15.162:0/3314831578 shutdown_connections 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.672+0000 7f3597ecc700 1 --2- 172.21.15.162:0/3314831578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35900f84c0 0x7f3590104ac0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.672+0000 7f3597ecc700 1 --2- 172.21.15.162:0/3314831578 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35900fdc50 0x7f35900fe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.672+0000 7f3597ecc700 1 --2- 172.21.15.162:0/3314831578 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35900fe5b0 0x7f35900f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.672+0000 7f3597ecc700 1 -- 172.21.15.162:0/3314831578 >> 172.21.15.162:0/3314831578 conn(0x7f35900f38a0 msgr2=0x7f35900f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.672+0000 7f3597ecc700 1 -- 172.21.15.162:0/3314831578 shutdown_connections 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.672+0000 7f3597ecc700 1 -- 172.21.15.162:0/3314831578 wait complete. 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3597ecc700 1 librados: starting msgr at 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3597ecc700 1 librados: starting objecter 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3597ecc700 1 Processor -- start 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3597ecc700 1 -- start start 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3597ecc700 1 librados: setting wanted keys 2024-02-13T00:38:55.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3597ecc700 1 librados: calling monclient init 2024-02-13T00:38:55.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3597ecc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35900f84c0 0x7f359006b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:55.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3597ecc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35900fdc50 0x7f359006b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:55.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3597ecc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35900fe5b0 0x7f359006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:55.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3597ecc700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3590116540 con 0x7f35900fe5b0 2024-02-13T00:38:55.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3597ecc700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35901166b0 con 0x7f35900fdc50 2024-02-13T00:38:55.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3597ecc700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3590116870 con 0x7f35900f84c0 2024-02-13T00:38:55.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3595c68700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35900f84c0 0x7f359006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:55.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3595c68700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35900f84c0 0x7f359006b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:54610/0 (socket says 172.21.15.162:54610) 2024-02-13T00:38:55.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3595c68700 1 -- 172.21.15.162:0/3161348999 learned_addr learned my addr 172.21.15.162:0/3161348999 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3596469700 1 --2- 172.21.15.162:0/3161348999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35900fe5b0 0x7f359006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3595467700 1 --2- 172.21.15.162:0/3161348999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35900fdc50 0x7f359006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3595c68700 1 -- 172.21.15.162:0/3161348999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35900fdc50 msgr2=0x7f359006b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3595c68700 1 --2- 172.21.15.162:0/3161348999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35900fdc50 0x7f359006b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3595c68700 1 -- 172.21.15.162:0/3161348999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35900fe5b0 msgr2=0x7f359006bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3595c68700 1 --2- 172.21.15.162:0/3161348999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35900fe5b0 0x7f359006bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3595c68700 1 -- 172.21.15.162:0/3161348999 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c005040 con 0x7f35900f84c0 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.673+0000 7f3596469700 1 --2- 172.21.15.162:0/3161348999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35900fe5b0 0x7f359006bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3595c68700 1 --2- 172.21.15.162:0/3161348999 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35900f84c0 0x7f359006b170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f358000e970 tx=0x7f358000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3586ffd700 10 client.?.objecter ms_handle_connect 0x7f35900f84c0 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3586ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3586ffd700 1 -- 172.21.15.162:0/3161348999 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580004660 con 0x7f35900f84c0 2024-02-13T00:38:55.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3597ecc700 1 -- 172.21.15.162:0/3161348999 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35880097c0 con 0x7f35900f84c0 2024-02-13T00:38:55.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3586ffd700 1 -- 172.21.15.162:0/3161348999 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f358000f040 con 0x7f35900f84c0 2024-02-13T00:38:55.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3586ffd700 1 -- 172.21.15.162:0/3161348999 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580014790 con 0x7f35900f84c0 2024-02-13T00:38:55.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3597ecc700 10 client.24554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:55.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3597ecc700 1 -- 172.21.15.162:0/3161348999 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3590116f00 con 0x7f35900f84c0 2024-02-13T00:38:55.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3597ecc700 1 librados: init done 2024-02-13T00:38:55.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3597ecc700 10 client.24554.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:38:55.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3597ecc700 15 client.24554.objecter _session_command_op_assign -1 1 2024-02-13T00:38:55.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3597ecc700 20 client.24554.objecter _get_session osd=-1 returning homeless 2024-02-13T00:38:55.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3597ecc700 10 client.24554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:55.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.674+0000 7f3597ecc700 1 -- 172.21.15.162:0/3161348999 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f35900475e0 con 0x7f35900f84c0 2024-02-13T00:38:55.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.675+0000 7f3586ffd700 1 -- 172.21.15.162:0/3161348999 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f35800090d0 con 0x7f35900f84c0 2024-02-13T00:38:55.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 1 --2- 172.21.15.162:0/3161348999 >> v2:172.21.15.162:6800/2866534053 conn(0x7f357c062c90 0x7f357c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:55.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 1 -- 172.21.15.162:0/3161348999 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(80..80 src has 30..80) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f358007cab0 con 0x7f35900f84c0 2024-02-13T00:38:55.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 10 client.24554.objecter ms_dispatch 0x7f3590001550 osd_map(80..80 src has 30..80) v4 2024-02-13T00:38:55.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 3 client.24554.objecter handle_osd_map got epochs [80,80] > 0 2024-02-13T00:38:55.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3595467700 1 --2- 172.21.15.162:0/3161348999 >> v2:172.21.15.162:6800/2866534053 conn(0x7f357c062c90 0x7f357c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:55.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 3 client.24554.objecter handle_osd_map decoding full epoch 80 2024-02-13T00:38:55.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 10 client.24554.objecter checking command 1 2024-02-13T00:38:55.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 1 --2- 172.21.15.162:0/3161348999 >> v2:172.21.15.162:6813/604971011 conn(0x7f357c06c4b0 0x7f357c06e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:55.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 20 client.24554.objecter _get_session s=0x7f357c06be00 osd=3 3 2024-02-13T00:38:55.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 20 client.24554.objecter put_session s=0x7f357c06be00 osd=3 3 2024-02-13T00:38:55.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 15 client.24554.objecter _session_command_op_remove -1 1 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 20 client.24554.objecter _get_session s=0x7f357c06be00 osd=3 3 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 20 client.24554.objecter get_session s=0x7f357c06be00 osd=3 3 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 15 client.24554.objecter _session_command_op_assign 3 1 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 20 client.24554.objecter put_session s=0x7f357c06be00 osd=3 4 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 10 client.24554.objecter _send_command 1 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 1 -- 172.21.15.162:0/3161348999 --> v2:172.21.15.162:6813/604971011 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f357c06ef90 con 0x7f357c06c4b0 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 20 client.24554.objecter dump_active .. 0 homeless 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 1 -- 172.21.15.162:0/3161348999 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f358007ce60 con 0x7f35900f84c0 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3596469700 1 --2- 172.21.15.162:0/3161348999 >> v2:172.21.15.162:6813/604971011 conn(0x7f357c06c4b0 0x7f357c06e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 10 client.24554.objecter ms_handle_connect 0x7f357c062c90 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3595467700 1 --2- 172.21.15.162:0/3161348999 >> v2:172.21.15.162:6800/2866534053 conn(0x7f357c062c90 0x7f357c065150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f359006c910 tx=0x7f358c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3586ffd700 10 client.24554.objecter ms_handle_connect 0x7f357c06c4b0 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.676+0000 7f3596469700 1 --2- 172.21.15.162:0/3161348999 >> v2:172.21.15.162:6813/604971011 conn(0x7f357c06c4b0 0x7f357c06e950 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.677+0000 7f3586ffd700 1 -- 172.21.15.162:0/3161348999 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f357c06ef90 con 0x7f357c06c4b0 2024-02-13T00:38:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.677+0000 7f3586ffd700 10 client.24554.objecter ms_dispatch 0x7f3590001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:38:55.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.677+0000 7f3586ffd700 10 client.24554.objecter _finish_command 1 = system:0 2024-02-13T00:38:55.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.677+0000 7f3586ffd700 20 client.24554.objecter put_session s=0x7f357c06be00 osd=3 4 2024-02-13T00:38:55.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.677+0000 7f3586ffd700 15 client.24554.objecter _session_command_op_remove 3 1 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 10 client.24554.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 15 client.24554.objecter _session_command_op_assign -1 2 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 20 client.24554.objecter _get_session s=0x7f357c06be00 osd=3 3 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 20 client.24554.objecter put_session s=0x7f357c06be00 osd=3 3 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 20 client.24554.objecter _get_session s=0x7f357c06be00 osd=3 3 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 15 client.24554.objecter _session_command_op_remove -1 2 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 20 client.24554.objecter get_session s=0x7f357c06be00 osd=3 3 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 15 client.24554.objecter _session_command_op_assign 3 2 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 20 client.24554.objecter put_session s=0x7f357c06be00 osd=3 4 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 10 client.24554.objecter _send_command 2 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 1 -- 172.21.15.162:0/3161348999 --> v2:172.21.15.162:6813/604971011 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f35901071b0 con 0x7f357c06c4b0 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3586ffd700 1 -- 172.21.15.162:0/3161348999 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f35901071b0 con 0x7f357c06c4b0 2024-02-13T00:38:55.694 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3586ffd700 10 client.24554.objecter ms_dispatch 0x7f3590001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:38:55.695 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3586ffd700 10 client.24554.objecter _finish_command 2 = system:0 2024-02-13T00:38:55.695 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3586ffd700 20 client.24554.objecter put_session s=0x7f357c06be00 osd=3 4 2024-02-13T00:38:55.695 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3586ffd700 15 client.24554.objecter _session_command_op_remove 3 2 2024-02-13T00:38:55.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 10 librados: watch_flush enter 2024-02-13T00:38:55.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 10 librados: watch_flush exit 2024-02-13T00:38:55.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 10 client.24554.objecter close_session for osd.3 2024-02-13T00:38:55.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 1 -- 172.21.15.162:0/3161348999 >> v2:172.21.15.162:6813/604971011 conn(0x7f357c06c4b0 msgr2=0x7f357c06e950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:55.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 1 --2- 172.21.15.162:0/3161348999 >> v2:172.21.15.162:6813/604971011 conn(0x7f357c06c4b0 0x7f357c06e950 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:55.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 20 client.24554.objecter put_session s=0x7f357c06be00 osd=3 1 2024-02-13T00:38:55.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 20 client.24554.objecter shutdown clearing up homeless session... 2024-02-13T00:38:55.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 10 client.24554.objecter successfully canceled tick 2024-02-13T00:38:55.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 1 -- 172.21.15.162:0/3161348999 >> v2:172.21.15.162:6800/2866534053 conn(0x7f357c062c90 msgr2=0x7f357c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:55.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 1 --2- 172.21.15.162:0/3161348999 >> v2:172.21.15.162:6800/2866534053 conn(0x7f357c062c90 0x7f357c065150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f359006c910 tx=0x7f358c00b040).stop 2024-02-13T00:38:55.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 1 -- 172.21.15.162:0/3161348999 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35900f84c0 msgr2=0x7f359006b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:55.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 1 --2- 172.21.15.162:0/3161348999 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35900f84c0 0x7f359006b170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f358000e970 tx=0x7f358000b040).stop 2024-02-13T00:38:55.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3596469700 1 -- 172.21.15.162:0/3161348999 reap_dead start 2024-02-13T00:38:55.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 1 -- 172.21.15.162:0/3161348999 shutdown_connections 2024-02-13T00:38:55.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 1 -- 172.21.15.162:0/3161348999 >> 172.21.15.162:0/3161348999 conn(0x7f35900f38a0 msgr2=0x7f35900f50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:55.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 1 -- 172.21.15.162:0/3161348999 shutdown_connections 2024-02-13T00:38:55.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 1 -- 172.21.15.162:0/3161348999 wait complete. 2024-02-13T00:38:55.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:55.691+0000 7f3597ecc700 1 librados: shutdown 2024-02-13T00:38:56.023 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:54.348334+0000 osd.0 (osd.0) 13 : cluster [DBG] 6.0 scrub starts 2024-02-13T00:38:56.023 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:54.349167+0000 osd.0 (osd.0) 14 : cluster [DBG] 6.0 scrub ok 2024-02-13T00:38:56.024 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:54.598183+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.d deep-scrub starts 2024-02-13T00:38:56.024 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:54.599052+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.d deep-scrub ok 2024-02-13T00:38:56.024 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:38:55.209503+0000 mgr.y (mgr.14544) 46 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:56.024 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:54.348334+0000 osd.0 (osd.0) 13 : cluster [DBG] 6.0 scrub starts 2024-02-13T00:38:56.024 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:54.349167+0000 osd.0 (osd.0) 14 : cluster [DBG] 6.0 scrub ok 2024-02-13T00:38:56.024 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:54.598183+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.d deep-scrub starts 2024-02-13T00:38:56.024 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:54.599052+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.d deep-scrub ok 2024-02-13T00:38:56.024 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:38:55.209503+0000 mgr.y (mgr.14544) 46 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:56.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:54.348334+0000 osd.0 (osd.0) 13 : cluster [DBG] 6.0 scrub starts 2024-02-13T00:38:56.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:54.349167+0000 osd.0 (osd.0) 14 : cluster [DBG] 6.0 scrub ok 2024-02-13T00:38:56.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:54.598183+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.d deep-scrub starts 2024-02-13T00:38:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:54.599052+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.d deep-scrub ok 2024-02-13T00:38:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:38:55.209503+0000 mgr.y (mgr.14544) 46 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:38:56.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:38:56] "GET /metrics HTTP/1.1" 200 172163 "" "Prometheus/2.33.4" 2024-02-13T00:38:56.358 INFO:teuthology.orchestra.run.smithi162.stdout:107374182437 2024-02-13T00:38:56.358 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-13T00:38:56.625 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:38:57.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:55.352273+0000 osd.0 (osd.0) 15 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:38:57.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:55.353151+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:38:57.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:55.572871+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:38:57.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:55.573624+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:38:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:56.169774+0000 mgr.y (mgr.14544) 47 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:38:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:56.328227+0000 osd.0 (osd.0) 17 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:38:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:56.331698+0000 osd.0 (osd.0) 18 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:38:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:56.367359+0000 osd.6 (osd.6) 7 : cluster [DBG] 4.a scrub starts 2024-02-13T00:38:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:56.370994+0000 osd.6 (osd.6) 8 : cluster [DBG] 4.a scrub ok 2024-02-13T00:38:57.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:55.352273+0000 osd.0 (osd.0) 15 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:38:57.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:55.353151+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:38:57.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:55.572871+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:38:57.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:55.573624+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:38:57.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:56.169774+0000 mgr.y (mgr.14544) 47 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:38:57.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:56.328227+0000 osd.0 (osd.0) 17 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:38:57.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:56.331698+0000 osd.0 (osd.0) 18 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:38:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:56.367359+0000 osd.6 (osd.6) 7 : cluster [DBG] 4.a scrub starts 2024-02-13T00:38:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:56.370994+0000 osd.6 (osd.6) 8 : cluster [DBG] 4.a scrub ok 2024-02-13T00:38:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:55.352273+0000 osd.0 (osd.0) 15 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:38:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:55.353151+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:38:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:55.572871+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:38:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:55.573624+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:38:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:56.169774+0000 mgr.y (mgr.14544) 47 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:38:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:56.328227+0000 osd.0 (osd.0) 17 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:38:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:56.331698+0000 osd.0 (osd.0) 18 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:38:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:56.367359+0000 osd.6 (osd.6) 7 : cluster [DBG] 4.a scrub starts 2024-02-13T00:38:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:56.370994+0000 osd.6 (osd.6) 8 : cluster [DBG] 4.a scrub ok 2024-02-13T00:38:58.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:56.854627+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:38:58.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:56.857528+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:38:58.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:57.528967+0000 osd.1 (osd.1) 11 : cluster [DBG] 4.4 deep-scrub starts 2024-02-13T00:38:58.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:57.537303+0000 osd.1 (osd.1) 12 : cluster [DBG] 4.4 deep-scrub ok 2024-02-13T00:38:58.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:56.854627+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:38:58.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:56.857528+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:38:58.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:57.528967+0000 osd.1 (osd.1) 11 : cluster [DBG] 4.4 deep-scrub starts 2024-02-13T00:38:58.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:57.537303+0000 osd.1 (osd.1) 12 : cluster [DBG] 4.4 deep-scrub ok 2024-02-13T00:38:58.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:56.854627+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:38:58.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:56.857528+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:38:58.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:57.528967+0000 osd.1 (osd.1) 11 : cluster [DBG] 4.4 deep-scrub starts 2024-02-13T00:38:58.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:57.537303+0000 osd.1 (osd.1) 12 : cluster [DBG] 4.4 deep-scrub ok 2024-02-13T00:38:58.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 -- 172.21.15.162:0/3475393561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4280f73a0 msgr2=0x7ff4281039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:58.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 --2- 172.21.15.162:0/3475393561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4280f73a0 0x7ff4281039a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7ff420009be0 tx=0x7ff420004d10).stop 2024-02-13T00:38:58.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 -- 172.21.15.162:0/3475393561 shutdown_connections 2024-02-13T00:38:58.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 --2- 172.21.15.162:0/3475393561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4280f73a0 0x7ff4281039a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:58.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 --2- 172.21.15.162:0/3475393561 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4280fcb30 0x7ff4280fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:58.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 --2- 172.21.15.162:0/3475393561 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4280fd490 0x7ff4280f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:58.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 -- 172.21.15.162:0/3475393561 >> 172.21.15.162:0/3475393561 conn(0x7ff42806eb90 msgr2=0x7ff4280f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:58.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 -- 172.21.15.162:0/3475393561 shutdown_connections 2024-02-13T00:38:58.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 -- 172.21.15.162:0/3475393561 wait complete. 2024-02-13T00:38:58.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 librados: starting msgr at 2024-02-13T00:38:58.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 librados: starting objecter 2024-02-13T00:38:58.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 Processor -- start 2024-02-13T00:38:58.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 -- start start 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 librados: setting wanted keys 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.392+0000 7ff42fcd9700 1 librados: calling monclient init 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42fcd9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4280f73a0 0x7ff42810def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42fcd9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4280fcb30 0x7ff42810e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42fcd9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4280fd490 0x7ff428108090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42fcd9700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff428108580 con 0x7ff4280fcb30 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42fcd9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4281086f0 con 0x7ff4280f73a0 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42fcd9700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4281088b0 con 0x7ff4280fd490 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42d274700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4280fcb30 0x7ff42810e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42e276700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4280fd490 0x7ff428108090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42e276700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4280fd490 0x7ff428108090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:54632/0 (socket says 172.21.15.162:54632) 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42e276700 1 -- 172.21.15.162:0/1890466215 learned_addr learned my addr 172.21.15.162:0/1890466215 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:38:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42da75700 1 --2- 172.21.15.162:0/1890466215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4280f73a0 0x7ff42810def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:58.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42d274700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4280fcb30 0x7ff42810e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:49942/0 (socket says 172.21.15.162:49942) 2024-02-13T00:38:58.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42e276700 1 -- 172.21.15.162:0/1890466215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4280f73a0 msgr2=0x7ff42810def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:58.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42e276700 1 --2- 172.21.15.162:0/1890466215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4280f73a0 0x7ff42810def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:58.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42e276700 1 -- 172.21.15.162:0/1890466215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4280fcb30 msgr2=0x7ff42810e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:58.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42e276700 1 --2- 172.21.15.162:0/1890466215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4280fcb30 0x7ff42810e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:58.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42e276700 1 -- 172.21.15.162:0/1890466215 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4200097c0 con 0x7ff4280fd490 2024-02-13T00:38:58.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42d274700 1 --2- 172.21.15.162:0/1890466215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4280fcb30 0x7ff42810e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:38:58.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42e276700 1 --2- 172.21.15.162:0/1890466215 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4280fd490 0x7ff428108090 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7ff420004a30 tx=0x7ff420004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:58.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff41effd700 10 client.?.objecter ms_handle_connect 0x7ff4280fd490 2024-02-13T00:38:58.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff41effd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:38:58.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff41effd700 1 -- 172.21.15.162:0/1890466215 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42000e040 con 0x7ff4280fd490 2024-02-13T00:38:58.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42fcd9700 1 -- 172.21.15.162:0/1890466215 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff428108b30 con 0x7ff4280fd490 2024-02-13T00:38:58.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff41effd700 1 -- 172.21.15.162:0/1890466215 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff420018040 con 0x7ff4280fd490 2024-02-13T00:38:58.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff41effd700 1 -- 172.21.15.162:0/1890466215 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42001c3f0 con 0x7ff4280fd490 2024-02-13T00:38:58.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42fcd9700 10 client.24560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:58.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42fcd9700 1 -- 172.21.15.162:0/1890466215 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff428114d20 con 0x7ff4280fd490 2024-02-13T00:38:58.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.393+0000 7ff42fcd9700 1 librados: init done 2024-02-13T00:38:58.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.394+0000 7ff42fcd9700 10 client.24560.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:38:58.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.394+0000 7ff42fcd9700 15 client.24560.objecter _session_command_op_assign -1 1 2024-02-13T00:38:58.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.394+0000 7ff42fcd9700 20 client.24560.objecter _get_session osd=-1 returning homeless 2024-02-13T00:38:58.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.394+0000 7ff42fcd9700 10 client.24560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:38:58.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.394+0000 7ff42fcd9700 1 -- 172.21.15.162:0/1890466215 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff4280475e0 con 0x7ff4280fd490 2024-02-13T00:38:58.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 1 -- 172.21.15.162:0/1890466215 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff420007b20 con 0x7ff4280fd490 2024-02-13T00:38:58.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 1 --2- 172.21.15.162:0/1890466215 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff414062c90 0x7ff414065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:58.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 1 -- 172.21.15.162:0/1890466215 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(80..80 src has 30..80) v4 ==== 6056+0+0 (secure 0 0 0) 0x7ff42007da20 con 0x7ff4280fd490 2024-02-13T00:38:58.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 10 client.24560.objecter ms_dispatch 0x7ff428001550 osd_map(80..80 src has 30..80) v4 2024-02-13T00:38:58.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 3 client.24560.objecter handle_osd_map got epochs [80,80] > 0 2024-02-13T00:38:58.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 3 client.24560.objecter handle_osd_map decoding full epoch 80 2024-02-13T00:38:58.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff42da75700 1 --2- 172.21.15.162:0/1890466215 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff414062c90 0x7ff414065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:58.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 10 client.24560.objecter checking command 1 2024-02-13T00:38:58.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 1 --2- 172.21.15.162:0/1890466215 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff41406c4b0 0x7ff41406e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:38:58.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 20 client.24560.objecter _get_session s=0x7ff41406be00 osd=4 3 2024-02-13T00:38:58.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 20 client.24560.objecter put_session s=0x7ff41406be00 osd=4 3 2024-02-13T00:38:58.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 15 client.24560.objecter _session_command_op_remove -1 1 2024-02-13T00:38:58.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 20 client.24560.objecter _get_session s=0x7ff41406be00 osd=4 3 2024-02-13T00:38:58.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 20 client.24560.objecter get_session s=0x7ff41406be00 osd=4 3 2024-02-13T00:38:58.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 15 client.24560.objecter _session_command_op_assign 4 1 2024-02-13T00:38:58.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 20 client.24560.objecter put_session s=0x7ff41406be00 osd=4 4 2024-02-13T00:38:58.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 10 client.24560.objecter _send_command 1 2024-02-13T00:38:58.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 1 -- 172.21.15.162:0/1890466215 --> v2:172.21.15.181:6800/3344263388 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff41406ef90 con 0x7ff41406c4b0 2024-02-13T00:38:58.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 20 client.24560.objecter dump_active .. 0 homeless 2024-02-13T00:38:58.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 1 -- 172.21.15.162:0/1890466215 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff420082220 con 0x7ff4280fd490 2024-02-13T00:38:58.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff42d274700 1 --2- 172.21.15.162:0/1890466215 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff41406c4b0 0x7ff41406e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:38:58.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff41effd700 10 client.24560.objecter ms_handle_connect 0x7ff414062c90 2024-02-13T00:38:58.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.395+0000 7ff42da75700 1 --2- 172.21.15.162:0/1890466215 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff414062c90 0x7ff414065150 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ff424004e60 tx=0x7ff42400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:58.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.396+0000 7ff41effd700 10 client.24560.objecter ms_handle_connect 0x7ff41406c4b0 2024-02-13T00:38:58.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.396+0000 7ff42d274700 1 --2- 172.21.15.162:0/1890466215 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff41406c4b0 0x7ff41406e950 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:38:58.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.397+0000 7ff41effd700 1 -- 172.21.15.162:0/1890466215 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff41406ef90 con 0x7ff41406c4b0 2024-02-13T00:38:58.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.397+0000 7ff41effd700 10 client.24560.objecter ms_dispatch 0x7ff428001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:38:58.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.397+0000 7ff41effd700 10 client.24560.objecter _finish_command 1 = system:0 2024-02-13T00:38:58.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.397+0000 7ff41effd700 20 client.24560.objecter put_session s=0x7ff41406be00 osd=4 4 2024-02-13T00:38:58.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.397+0000 7ff41effd700 15 client.24560.objecter _session_command_op_remove 4 1 2024-02-13T00:38:58.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 10 client.24560.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:38:58.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 15 client.24560.objecter _session_command_op_assign -1 2 2024-02-13T00:38:58.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 20 client.24560.objecter _get_session s=0x7ff41406be00 osd=4 3 2024-02-13T00:38:58.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 20 client.24560.objecter put_session s=0x7ff41406be00 osd=4 3 2024-02-13T00:38:58.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 20 client.24560.objecter _get_session s=0x7ff41406be00 osd=4 3 2024-02-13T00:38:58.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 15 client.24560.objecter _session_command_op_remove -1 2 2024-02-13T00:38:58.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 20 client.24560.objecter get_session s=0x7ff41406be00 osd=4 3 2024-02-13T00:38:58.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 15 client.24560.objecter _session_command_op_assign 4 2 2024-02-13T00:38:58.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 20 client.24560.objecter put_session s=0x7ff41406be00 osd=4 4 2024-02-13T00:38:58.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 10 client.24560.objecter _send_command 2 2024-02-13T00:38:58.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 1 -- 172.21.15.162:0/1890466215 --> v2:172.21.15.181:6800/3344263388 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff428106090 con 0x7ff41406c4b0 2024-02-13T00:38:58.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff41effd700 1 -- 172.21.15.162:0/1890466215 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff428106090 con 0x7ff41406c4b0 2024-02-13T00:38:58.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff41effd700 10 client.24560.objecter ms_dispatch 0x7ff428001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:38:58.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff41effd700 10 client.24560.objecter _finish_command 2 = system:0 2024-02-13T00:38:58.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff41effd700 20 client.24560.objecter put_session s=0x7ff41406be00 osd=4 4 2024-02-13T00:38:58.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff41effd700 15 client.24560.objecter _session_command_op_remove 4 2 2024-02-13T00:38:58.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 10 librados: watch_flush enter 2024-02-13T00:38:58.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 10 librados: watch_flush exit 2024-02-13T00:38:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 10 client.24560.objecter close_session for osd.4 2024-02-13T00:38:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 1 -- 172.21.15.162:0/1890466215 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff41406c4b0 msgr2=0x7ff41406e950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 1 --2- 172.21.15.162:0/1890466215 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff41406c4b0 0x7ff41406e950 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:38:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 20 client.24560.objecter put_session s=0x7ff41406be00 osd=4 1 2024-02-13T00:38:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 20 client.24560.objecter shutdown clearing up homeless session... 2024-02-13T00:38:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 10 client.24560.objecter successfully canceled tick 2024-02-13T00:38:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 1 -- 172.21.15.162:0/1890466215 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff414062c90 msgr2=0x7ff414065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 1 --2- 172.21.15.162:0/1890466215 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff414062c90 0x7ff414065150 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ff424004e60 tx=0x7ff42400b040).stop 2024-02-13T00:38:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 1 -- 172.21.15.162:0/1890466215 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4280fd490 msgr2=0x7ff428108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:38:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42fcd9700 1 --2- 172.21.15.162:0/1890466215 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4280fd490 0x7ff428108090 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7ff420004a30 tx=0x7ff420004d10).stop 2024-02-13T00:38:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.411+0000 7ff42e276700 1 -- 172.21.15.162:0/1890466215 reap_dead start 2024-02-13T00:38:58.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.412+0000 7ff42fcd9700 1 -- 172.21.15.162:0/1890466215 shutdown_connections 2024-02-13T00:38:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.412+0000 7ff42fcd9700 1 -- 172.21.15.162:0/1890466215 >> 172.21.15.162:0/1890466215 conn(0x7ff42806eb90 msgr2=0x7ff4280701c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:38:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.412+0000 7ff42fcd9700 1 -- 172.21.15.162:0/1890466215 shutdown_connections 2024-02-13T00:38:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.412+0000 7ff42fcd9700 1 -- 172.21.15.162:0/1890466215 wait complete. 2024-02-13T00:38:58.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:38:58.412+0000 7ff42fcd9700 1 librados: shutdown 2024-02-13T00:38:58.933 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:57.493722+0000 osd.5 (osd.5) 21 : cluster [DBG] 5.f deep-scrub starts 2024-02-13T00:38:58.933 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:57.494645+0000 osd.5 (osd.5) 22 : cluster [DBG] 5.f deep-scrub ok 2024-02-13T00:38:58.934 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:57.815264+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:38:58.934 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:57.818195+0000 osd.7 (osd.7) 20 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:38:58.934 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:58.170204+0000 mgr.y (mgr.14544) 48 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:38:58.934 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:58.235058+0000 osd.3 (osd.3) 11 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:38:58.934 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:58.238521+0000 osd.3 (osd.3) 12 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:38:58.934 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:58.311647+0000 osd.0 (osd.0) 19 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:38:58.934 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:58.314538+0000 osd.0 (osd.0) 20 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:38:58.934 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:58.339858+0000 osd.6 (osd.6) 9 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:38:58.934 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:58.341395+0000 osd.6 (osd.6) 10 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:38:58.934 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:58.509154+0000 osd.1 (osd.1) 13 : cluster [DBG] 5.12 deep-scrub starts 2024-02-13T00:38:58.934 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:58.512048+0000 osd.1 (osd.1) 14 : cluster [DBG] 5.12 deep-scrub ok 2024-02-13T00:38:58.935 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:38:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:38:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:38:59.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:57.493722+0000 osd.5 (osd.5) 21 : cluster [DBG] 5.f deep-scrub starts 2024-02-13T00:38:59.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:57.494645+0000 osd.5 (osd.5) 22 : cluster [DBG] 5.f deep-scrub ok 2024-02-13T00:38:59.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:57.815264+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:38:59.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:57.818195+0000 osd.7 (osd.7) 20 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:38:59.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:58.170204+0000 mgr.y (mgr.14544) 48 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:38:59.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:58.235058+0000 osd.3 (osd.3) 11 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:38:59.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:58.238521+0000 osd.3 (osd.3) 12 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:38:59.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:58.311647+0000 osd.0 (osd.0) 19 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:58.314538+0000 osd.0 (osd.0) 20 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:58.339858+0000 osd.6 (osd.6) 9 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:58.341395+0000 osd.6 (osd.6) 10 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:58.509154+0000 osd.1 (osd.1) 13 : cluster [DBG] 5.12 deep-scrub starts 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:58.512048+0000 osd.1 (osd.1) 14 : cluster [DBG] 5.12 deep-scrub ok 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:57.493722+0000 osd.5 (osd.5) 21 : cluster [DBG] 5.f deep-scrub starts 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:57.494645+0000 osd.5 (osd.5) 22 : cluster [DBG] 5.f deep-scrub ok 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:57.815264+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:57.818195+0000 osd.7 (osd.7) 20 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:58.170204+0000 mgr.y (mgr.14544) 48 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:58.235058+0000 osd.3 (osd.3) 11 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:58.238521+0000 osd.3 (osd.3) 12 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:38:59.139 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:58.311647+0000 osd.0 (osd.0) 19 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:38:59.140 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:58.314538+0000 osd.0 (osd.0) 20 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:38:59.140 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:58.339858+0000 osd.6 (osd.6) 9 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:38:59.140 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:58.341395+0000 osd.6 (osd.6) 10 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:38:59.140 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:58.509154+0000 osd.1 (osd.1) 13 : cluster [DBG] 5.12 deep-scrub starts 2024-02-13T00:38:59.140 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:58.512048+0000 osd.1 (osd.1) 14 : cluster [DBG] 5.12 deep-scrub ok 2024-02-13T00:38:59.161 INFO:teuthology.orchestra.run.smithi162.stdout:133143986209 2024-02-13T00:38:59.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-13T00:38:59.437 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:00.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:58.525676+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:39:00.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:58.528577+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:39:00.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:59.185968+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.c scrub starts 2024-02-13T00:39:00.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:38:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:59.188909+0000 osd.3 (osd.3) 14 : cluster [DBG] 6.c scrub ok 2024-02-13T00:39:00.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:58.525676+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:39:00.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:58.528577+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:39:00.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:59.185968+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.c scrub starts 2024-02-13T00:39:00.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:38:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:59.188909+0000 osd.3 (osd.3) 14 : cluster [DBG] 6.c scrub ok 2024-02-13T00:39:00.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:58.525676+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:39:00.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:58.528577+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:39:00.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:59.185968+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.c scrub starts 2024-02-13T00:39:00.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:38:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:59.188909+0000 osd.3 (osd.3) 14 : cluster [DBG] 6.c scrub ok 2024-02-13T00:39:01.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:59.543501+0000 osd.5 (osd.5) 25 : cluster [DBG] 4.8 deep-scrub starts 2024-02-13T00:39:01.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:38:59.548868+0000 osd.5 (osd.5) 26 : cluster [DBG] 4.8 deep-scrub ok 2024-02-13T00:39:01.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:00.171038+0000 mgr.y (mgr.14544) 49 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:39:01.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:00.212851+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:39:01.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:00.215752+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:39:01.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:00.392384+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:39:01.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:00.395377+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:39:01.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:00.448996+0000 osd.1 (osd.1) 15 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:39:01.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:00.452080+0000 osd.1 (osd.1) 16 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:39:01.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:59.543501+0000 osd.5 (osd.5) 25 : cluster [DBG] 4.8 deep-scrub starts 2024-02-13T00:39:01.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:38:59.548868+0000 osd.5 (osd.5) 26 : cluster [DBG] 4.8 deep-scrub ok 2024-02-13T00:39:01.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:00.171038+0000 mgr.y (mgr.14544) 49 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:39:01.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:00.212851+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:39:01.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:00.215752+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:39:01.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:00.392384+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:39:01.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:00.395377+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:39:01.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:00.448996+0000 osd.1 (osd.1) 15 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:39:01.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:00.452080+0000 osd.1 (osd.1) 16 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:39:01.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:59.543501+0000 osd.5 (osd.5) 25 : cluster [DBG] 4.8 deep-scrub starts 2024-02-13T00:39:01.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:38:59.548868+0000 osd.5 (osd.5) 26 : cluster [DBG] 4.8 deep-scrub ok 2024-02-13T00:39:01.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:00.171038+0000 mgr.y (mgr.14544) 49 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:39:01.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:00.212851+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:39:01.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:00.215752+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:39:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:00.392384+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:39:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:00.395377+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:39:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:00.448996+0000 osd.1 (osd.1) 15 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:39:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:00.452080+0000 osd.1 (osd.1) 16 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:39:01.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.547+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/3272450737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb880fd490 msgr2=0x7fcb880f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:01.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.547+0000 7fcb8eeb8700 1 --2- 172.21.15.162:0/3272450737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb880fd490 0x7fcb880f6cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fcb740047b0 tx=0x7fcb74009cb0).stop 2024-02-13T00:39:01.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.547+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/3272450737 shutdown_connections 2024-02-13T00:39:01.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.547+0000 7fcb8eeb8700 1 --2- 172.21.15.162:0/3272450737 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb880f73a0 0x7fcb881039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:01.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.547+0000 7fcb8eeb8700 1 --2- 172.21.15.162:0/3272450737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb880fcb30 0x7fcb880fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:01.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.547+0000 7fcb8eeb8700 1 --2- 172.21.15.162:0/3272450737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb880fd490 0x7fcb880f6cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:01.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.547+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/3272450737 >> 172.21.15.162:0/3272450737 conn(0x7fcb8806eb90 msgr2=0x7fcb880f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:01.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.547+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/3272450737 shutdown_connections 2024-02-13T00:39:01.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.547+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/3272450737 wait complete. 2024-02-13T00:39:01.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8eeb8700 1 librados: starting msgr at 2024-02-13T00:39:01.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8eeb8700 1 librados: starting objecter 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8eeb8700 1 Processor -- start 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8eeb8700 1 -- start start 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8eeb8700 1 librados: setting wanted keys 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8eeb8700 1 librados: calling monclient init 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8eeb8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb880f73a0 0x7fcb880f8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8eeb8700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb880fcb30 0x7fcb880f8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8eeb8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb880fd490 0x7fcb880f9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8eeb8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb88116570 con 0x7fcb880fd490 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8eeb8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb881166b0 con 0x7fcb880f73a0 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8eeb8700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb88116870 con 0x7fcb880fcb30 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb87fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb880fcb30 0x7fcb880f8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8d455700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb880fd490 0x7fcb880f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb87fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb880fcb30 0x7fcb880f8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:54640/0 (socket says 172.21.15.162:54640) 2024-02-13T00:39:01.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8d455700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb880fd490 0x7fcb880f9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:49950/0 (socket says 172.21.15.162:49950) 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb87fff700 1 -- 172.21.15.162:0/448870655 learned_addr learned my addr 172.21.15.162:0/448870655 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8cc54700 1 --2- 172.21.15.162:0/448870655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb880f73a0 0x7fcb880f8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb87fff700 1 -- 172.21.15.162:0/448870655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb880f73a0 msgr2=0x7fcb880f8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb87fff700 1 --2- 172.21.15.162:0/448870655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb880f73a0 0x7fcb880f8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb87fff700 1 -- 172.21.15.162:0/448870655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb880fd490 msgr2=0x7fcb880f9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb87fff700 1 --2- 172.21.15.162:0/448870655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb880fd490 0x7fcb880f9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb87fff700 1 -- 172.21.15.162:0/448870655 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb800097c0 con 0x7fcb880fcb30 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8cc54700 1 --2- 172.21.15.162:0/448870655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb880f73a0 0x7fcb880f8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.548+0000 7fcb8d455700 1 --2- 172.21.15.162:0/448870655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb880fd490 0x7fcb880f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.549+0000 7fcb87fff700 1 --2- 172.21.15.162:0/448870655 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb880fcb30 0x7fcb880f8d30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fcb740044e0 tx=0x7fcb74007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.549+0000 7fcb85ffb700 10 client.?.objecter ms_handle_connect 0x7fcb880fcb30 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.549+0000 7fcb85ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.549+0000 7fcb85ffb700 1 -- 172.21.15.162:0/448870655 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb74018070 con 0x7fcb880fcb30 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.549+0000 7fcb85ffb700 1 -- 172.21.15.162:0/448870655 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb740093b0 con 0x7fcb880fcb30 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.549+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/448870655 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb74005040 con 0x7fcb880fcb30 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.549+0000 7fcb85ffb700 1 -- 172.21.15.162:0/448870655 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7401de40 con 0x7fcb880fcb30 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.549+0000 7fcb8eeb8700 10 client.24566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.549+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/448870655 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb88116d90 con 0x7fcb880fcb30 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.549+0000 7fcb8eeb8700 1 librados: init done 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.550+0000 7fcb8eeb8700 10 client.24566.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.550+0000 7fcb8eeb8700 15 client.24566.objecter _session_command_op_assign -1 1 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.550+0000 7fcb8eeb8700 20 client.24566.objecter _get_session osd=-1 returning homeless 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.550+0000 7fcb8eeb8700 10 client.24566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.550+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/448870655 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcb880475e0 con 0x7fcb880fcb30 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.550+0000 7fcb85ffb700 1 -- 172.21.15.162:0/448870655 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fcb74013ad0 con 0x7fcb880fcb30 2024-02-13T00:39:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 1 --2- 172.21.15.162:0/448870655 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcb78062c90 0x7fcb78065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 1 -- 172.21.15.162:0/448870655 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(80..80 src has 30..80) v4 ==== 6056+0+0 (secure 0 0 0) 0x7fcb7407dfd0 con 0x7fcb880fcb30 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 10 client.24566.objecter ms_dispatch 0x7fcb88001550 osd_map(80..80 src has 30..80) v4 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 3 client.24566.objecter handle_osd_map got epochs [80,80] > 0 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 3 client.24566.objecter handle_osd_map decoding full epoch 80 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb8cc54700 1 --2- 172.21.15.162:0/448870655 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcb78062c90 0x7fcb78065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 10 client.24566.objecter checking command 1 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 1 --2- 172.21.15.162:0/448870655 >> v2:172.21.15.181:6804/2212296748 conn(0x7fcb7806c530 0x7fcb7806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 20 client.24566.objecter _get_session s=0x7fcb7806be80 osd=5 3 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 20 client.24566.objecter put_session s=0x7fcb7806be80 osd=5 3 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 15 client.24566.objecter _session_command_op_remove -1 1 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 20 client.24566.objecter _get_session s=0x7fcb7806be80 osd=5 3 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 20 client.24566.objecter get_session s=0x7fcb7806be80 osd=5 3 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 15 client.24566.objecter _session_command_op_assign 5 1 2024-02-13T00:39:01.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 20 client.24566.objecter put_session s=0x7fcb7806be80 osd=5 4 2024-02-13T00:39:01.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 10 client.24566.objecter _send_command 1 2024-02-13T00:39:01.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 1 -- 172.21.15.162:0/448870655 --> v2:172.21.15.181:6804/2212296748 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb7806f010 con 0x7fcb7806c530 2024-02-13T00:39:01.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 20 client.24566.objecter dump_active .. 0 homeless 2024-02-13T00:39:01.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 1 -- 172.21.15.162:0/448870655 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb740804f0 con 0x7fcb880fcb30 2024-02-13T00:39:01.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb85ffb700 10 client.24566.objecter ms_handle_connect 0x7fcb78062c90 2024-02-13T00:39:01.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb8cc54700 1 --2- 172.21.15.162:0/448870655 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcb78062c90 0x7fcb78065150 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fcb7c004b80 tx=0x7fcb7c009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:01.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.551+0000 7fcb8d455700 1 --2- 172.21.15.162:0/448870655 >> v2:172.21.15.181:6804/2212296748 conn(0x7fcb7806c530 0x7fcb7806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:01.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.552+0000 7fcb85ffb700 10 client.24566.objecter ms_handle_connect 0x7fcb7806c530 2024-02-13T00:39:01.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.552+0000 7fcb8d455700 1 --2- 172.21.15.162:0/448870655 >> v2:172.21.15.181:6804/2212296748 conn(0x7fcb7806c530 0x7fcb7806e9d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:01.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.552+0000 7fcb85ffb700 1 -- 172.21.15.162:0/448870655 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcb7806f010 con 0x7fcb7806c530 2024-02-13T00:39:01.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.552+0000 7fcb85ffb700 10 client.24566.objecter ms_dispatch 0x7fcb88001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:39:01.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.552+0000 7fcb85ffb700 10 client.24566.objecter _finish_command 1 = system:0 2024-02-13T00:39:01.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.552+0000 7fcb85ffb700 20 client.24566.objecter put_session s=0x7fcb7806be80 osd=5 4 2024-02-13T00:39:01.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.552+0000 7fcb85ffb700 15 client.24566.objecter _session_command_op_remove 5 1 2024-02-13T00:39:01.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.567+0000 7fcb8eeb8700 10 client.24566.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:39:01.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.567+0000 7fcb8eeb8700 15 client.24566.objecter _session_command_op_assign -1 2 2024-02-13T00:39:01.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.567+0000 7fcb8eeb8700 20 client.24566.objecter _get_session s=0x7fcb7806be80 osd=5 3 2024-02-13T00:39:01.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.567+0000 7fcb8eeb8700 20 client.24566.objecter put_session s=0x7fcb7806be80 osd=5 3 2024-02-13T00:39:01.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.567+0000 7fcb8eeb8700 20 client.24566.objecter _get_session s=0x7fcb7806be80 osd=5 3 2024-02-13T00:39:01.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.567+0000 7fcb8eeb8700 15 client.24566.objecter _session_command_op_remove -1 2 2024-02-13T00:39:01.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.567+0000 7fcb8eeb8700 20 client.24566.objecter get_session s=0x7fcb7806be80 osd=5 3 2024-02-13T00:39:01.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.567+0000 7fcb8eeb8700 15 client.24566.objecter _session_command_op_assign 5 2 2024-02-13T00:39:01.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.567+0000 7fcb8eeb8700 20 client.24566.objecter put_session s=0x7fcb7806be80 osd=5 4 2024-02-13T00:39:01.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.567+0000 7fcb8eeb8700 10 client.24566.objecter _send_command 2 2024-02-13T00:39:01.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.567+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/448870655 --> v2:172.21.15.181:6804/2212296748 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcb88106050 con 0x7fcb7806c530 2024-02-13T00:39:01.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb85ffb700 1 -- 172.21.15.162:0/448870655 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcb88106050 con 0x7fcb7806c530 2024-02-13T00:39:01.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb85ffb700 10 client.24566.objecter ms_dispatch 0x7fcb88001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:39:01.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb85ffb700 10 client.24566.objecter _finish_command 2 = system:0 2024-02-13T00:39:01.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb85ffb700 20 client.24566.objecter put_session s=0x7fcb7806be80 osd=5 4 2024-02-13T00:39:01.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb85ffb700 15 client.24566.objecter _session_command_op_remove 5 2 2024-02-13T00:39:01.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 10 librados: watch_flush enter 2024-02-13T00:39:01.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 10 librados: watch_flush exit 2024-02-13T00:39:01.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 10 client.24566.objecter close_session for osd.5 2024-02-13T00:39:01.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/448870655 >> v2:172.21.15.181:6804/2212296748 conn(0x7fcb7806c530 msgr2=0x7fcb7806e9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:01.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 1 --2- 172.21.15.162:0/448870655 >> v2:172.21.15.181:6804/2212296748 conn(0x7fcb7806c530 0x7fcb7806e9d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:01.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 20 client.24566.objecter put_session s=0x7fcb7806be80 osd=5 1 2024-02-13T00:39:01.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 20 client.24566.objecter shutdown clearing up homeless session... 2024-02-13T00:39:01.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 10 client.24566.objecter successfully canceled tick 2024-02-13T00:39:01.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/448870655 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcb78062c90 msgr2=0x7fcb78065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:01.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 1 --2- 172.21.15.162:0/448870655 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcb78062c90 0x7fcb78065150 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fcb7c004b80 tx=0x7fcb7c009040).stop 2024-02-13T00:39:01.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/448870655 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb880fcb30 msgr2=0x7fcb880f8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:01.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 1 --2- 172.21.15.162:0/448870655 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb880fcb30 0x7fcb880f8d30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fcb740044e0 tx=0x7fcb74007f90).stop 2024-02-13T00:39:01.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8d455700 1 -- 172.21.15.162:0/448870655 reap_dead start 2024-02-13T00:39:01.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/448870655 shutdown_connections 2024-02-13T00:39:01.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/448870655 >> 172.21.15.162:0/448870655 conn(0x7fcb8806eb90 msgr2=0x7fcb8806ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:01.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/448870655 shutdown_connections 2024-02-13T00:39:01.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.568+0000 7fcb8eeb8700 1 -- 172.21.15.162:0/448870655 wait complete. 2024-02-13T00:39:01.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:01.569+0000 7fcb8eeb8700 1 librados: shutdown 2024-02-13T00:39:02.228 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:01.194407+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:39:02.228 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:01.197306+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:39:02.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:01.436512+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.11 deep-scrub starts 2024-02-13T00:39:02.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:01.439505+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 deep-scrub ok 2024-02-13T00:39:02.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:01.452823+0000 osd.1 (osd.1) 17 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:39:02.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:01.456500+0000 osd.1 (osd.1) 18 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:39:02.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:01.566208+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:39:02.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:01.569103+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:39:02.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:01.194407+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:39:02.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:01.197306+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:39:02.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:01.436512+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.11 deep-scrub starts 2024-02-13T00:39:02.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:01.439505+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 deep-scrub ok 2024-02-13T00:39:02.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:01.452823+0000 osd.1 (osd.1) 17 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:39:02.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:01.456500+0000 osd.1 (osd.1) 18 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:39:02.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:01.566208+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:39:02.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:01.569103+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:39:02.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:01.194407+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:39:02.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:01.197306+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:39:02.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:01.436512+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.11 deep-scrub starts 2024-02-13T00:39:02.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:01.439505+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 deep-scrub ok 2024-02-13T00:39:02.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:01.452823+0000 osd.1 (osd.1) 17 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:39:02.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:01.456500+0000 osd.1 (osd.1) 18 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:39:02.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:01.566208+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:39:02.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:01.569103+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:39:02.412 INFO:teuthology.orchestra.run.smithi162.stdout:167503724573 2024-02-13T00:39:02.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-13T00:39:02.683 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:03.102 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 systemd[1]: Starting Ceph grafana.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:39:03.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:01.558177+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:39:03.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:01.561147+0000 osd.5 (osd.5) 28 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:39:03.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:02.171550+0000 mgr.y (mgr.14544) 50 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:03.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:02.184515+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.d scrub starts 2024-02-13T00:39:03.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:02.188092+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.d scrub ok 2024-02-13T00:39:03.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:02.222831+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:39:03.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:02.223819+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:39:03.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:02.451703+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-02-13T00:39:03.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:02.452750+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-02-13T00:39:03.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:02.542483+0000 osd.2 (osd.2) 9 : cluster [DBG] 2.e scrub starts 2024-02-13T00:39:03.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:02.545490+0000 osd.2 (osd.2) 10 : cluster [DBG] 2.e scrub ok 2024-02-13T00:39:03.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:01.558177+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:39:03.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:01.561147+0000 osd.5 (osd.5) 28 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:39:03.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:02.171550+0000 mgr.y (mgr.14544) 50 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:03.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:02.184515+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.d scrub starts 2024-02-13T00:39:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:02.188092+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.d scrub ok 2024-02-13T00:39:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:02.222831+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:39:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:02.223819+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:39:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:02.451703+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-02-13T00:39:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:02.452750+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-02-13T00:39:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:02.542483+0000 osd.2 (osd.2) 9 : cluster [DBG] 2.e scrub starts 2024-02-13T00:39:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:02.545490+0000 osd.2 (osd.2) 10 : cluster [DBG] 2.e scrub ok 2024-02-13T00:39:03.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:01.558177+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:39:03.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:01.561147+0000 osd.5 (osd.5) 28 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:39:03.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:02.171550+0000 mgr.y (mgr.14544) 50 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:03.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:02.184515+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.d scrub starts 2024-02-13T00:39:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:02.188092+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.d scrub ok 2024-02-13T00:39:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:02.222831+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:39:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:02.223819+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:39:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:02.451703+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-02-13T00:39:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:02.452750+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-02-13T00:39:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:02.542483+0000 osd.2 (osd.2) 9 : cluster [DBG] 2.e scrub starts 2024-02-13T00:39:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:02.545490+0000 osd.2 (osd.2) 10 : cluster [DBG] 2.e scrub ok 2024-02-13T00:39:03.957 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:03.183589+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.13 deep-scrub starts 2024-02-13T00:39:03.957 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:03.189233+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.13 deep-scrub ok 2024-02-13T00:39:03.957 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:03.814423+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:03.957 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:03.818026+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:39:03.959 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-13T00:39:03.960 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-13T00:39:03.960 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-13T00:39:03.960 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-13T00:39:03.960 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-13T00:39:03.960 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-13T00:39:03.960 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-13T00:39:03.960 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-13T00:39:03.960 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-13T00:39:03.960 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-13T00:39:03.960 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-13T00:39:03.960 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-13T00:39:03.960 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="App mode production" logger=settings 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-02-13T00:39:03.961 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-02-13T00:39:03.962 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 bash[64927]: 31ce363ce9c18139c01385964a9b47d02b32d67542c9449acac64ab1aed13c80 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 systemd[1]: Started Ceph grafana.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-02-13T00:39:03.963 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-02-13T00:39:03.964 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-02-13T00:39:03.965 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-02-13T00:39:03.966 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-02-13T00:39:03.967 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-02-13T00:39:03.968 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-02-13T00:39:03.969 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-02-13T00:39:03.970 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-02-13T00:39:03.971 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-02-13T00:39:03.972 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-02-13T00:39:03.973 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-02-13T00:39:03.974 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-02-13T00:39:03.975 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-02-13T00:39:03.975 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-02-13T00:39:03.975 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-02-13T00:39:03.975 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-02-13T00:39:04.209 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-02-13T00:39:04.210 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-02-13T00:39:04.210 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-02-13T00:39:04.210 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-02-13T00:39:04.210 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-02-13T00:39:04.210 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-02-13T00:39:04.210 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-02-13T00:39:04.210 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-02-13T00:39:04.210 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-02-13T00:39:04.210 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-02-13T00:39:04.211 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-02-13T00:39:04.211 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-02-13T00:39:04.211 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-02-13T00:39:04.211 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-02-13T00:39:04.211 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-02-13T00:39:04.211 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-13T00:39:04.211 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-13T00:39:04.211 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-13T00:39:04.211 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-02-13T00:39:04.211 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:03+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-13T00:39:04.212 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-02-13T00:39:04.212 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-02-13T00:39:04.212 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-02-13T00:39:04.212 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-02-13T00:39:04.212 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-02-13T00:39:04.212 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-02-13T00:39:04.212 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-02-13T00:39:04.213 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-13T00:39:04.213 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-13T00:39:04.213 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-13T00:39:04.213 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-13T00:39:04.213 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-13T00:39:04.213 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-13T00:39:04.213 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-13T00:39:04.213 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-13T00:39:04.213 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-13T00:39:04.213 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-13T00:39:04.213 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-02-13T00:39:04.213 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-13T00:39:04.214 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-13T00:39:04.214 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-13T00:39:04.214 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-13T00:39:04.214 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-13T00:39:04.214 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-13T00:39:04.214 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-13T00:39:04.214 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-13T00:39:04.214 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-13T00:39:04.214 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-13T00:39:04.214 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-13T00:39:04.215 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-13T00:39:04.215 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-13T00:39:04.215 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-02-13T00:39:04.215 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-13T00:39:04.215 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-13T00:39:04.215 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-13T00:39:04.215 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-13T00:39:04.215 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-13T00:39:04.216 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-13T00:39:04.216 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-13T00:39:04.216 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-02-13T00:39:04.216 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-02-13T00:39:04.216 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-02-13T00:39:04.216 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-02-13T00:39:04.216 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-13T00:39:04.216 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-02-13T00:39:04.216 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-13T00:39:04.216 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-13T00:39:04.216 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-13T00:39:04.216 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-13T00:39:04.217 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-02-13T00:39:04.218 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-13T00:39:04.219 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-02-13T00:39:04.220 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-02-13T00:39:04.221 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-02-13T00:39:04.222 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-02-13T00:39:04.223 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-02-13T00:39:04.224 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-02-13T00:39:04.224 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-02-13T00:39:04.224 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-02-13T00:39:04.224 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:03.183589+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.13 deep-scrub starts 2024-02-13T00:39:04.224 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:03.189233+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.13 deep-scrub ok 2024-02-13T00:39:04.224 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:03.814423+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:04.224 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:03.818026+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:39:04.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:03.183589+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.13 deep-scrub starts 2024-02-13T00:39:04.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:03.189233+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.13 deep-scrub ok 2024-02-13T00:39:04.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:03.814423+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:04.225 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:03.818026+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:39:04.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949710397 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a140f8500 msgr2=0x7f9a14104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:04.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 --2- 172.21.15.162:0/2949710397 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a140f8500 0x7f9a14104bd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9a0c009be0 tx=0x7f9a0c004d10).stop 2024-02-13T00:39:04.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949710397 shutdown_connections 2024-02-13T00:39:04.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 --2- 172.21.15.162:0/2949710397 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a140f8500 0x7f9a14104bd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:04.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 --2- 172.21.15.162:0/2949710397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a140f9b70 0x7f9a140f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:04.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 --2- 172.21.15.162:0/2949710397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a140fa440 0x7f9a140f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:04.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949710397 >> 172.21.15.162:0/2949710397 conn(0x7f9a140f38c0 msgr2=0x7f9a140f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:04.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949710397 shutdown_connections 2024-02-13T00:39:04.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949710397 wait complete. 2024-02-13T00:39:04.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 librados: starting msgr at 2024-02-13T00:39:04.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 librados: starting objecter 2024-02-13T00:39:04.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 Processor -- start 2024-02-13T00:39:04.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 -- start start 2024-02-13T00:39:04.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 librados: setting wanted keys 2024-02-13T00:39:04.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.439+0000 7f9a1b9cd700 1 librados: calling monclient init 2024-02-13T00:39:04.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a1b9cd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a140f8500 0x7f9a1410e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:04.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a1b9cd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a140f9b70 0x7f9a1410e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:04.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a1b9cd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a140fa440 0x7f9a141081f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:04.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a1b9cd700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a141086e0 con 0x7f9a140f8500 2024-02-13T00:39:04.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a1b9cd700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a14108850 con 0x7f9a140f9b70 2024-02-13T00:39:04.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a1b9cd700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a141089c0 con 0x7f9a140fa440 2024-02-13T00:39:04.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a19769700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a140f8500 0x7f9a1410e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:04.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a19f6a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a140fa440 0x7f9a141081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:04.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a19769700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a140f8500 0x7f9a1410e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:40876/0 (socket says 172.21.15.162:40876) 2024-02-13T00:39:04.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a19769700 1 -- 172.21.15.162:0/2949767306 learned_addr learned my addr 172.21.15.162:0/2949767306 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:04.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a18f68700 1 --2- 172.21.15.162:0/2949767306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a140f9b70 0x7f9a1410e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:04.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a19f6a700 1 -- 172.21.15.162:0/2949767306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a140f9b70 msgr2=0x7f9a1410e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:04.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a19f6a700 1 --2- 172.21.15.162:0/2949767306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a140f9b70 0x7f9a1410e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:04.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a19f6a700 1 -- 172.21.15.162:0/2949767306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a140f8500 msgr2=0x7f9a1410e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:04.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a19f6a700 1 --2- 172.21.15.162:0/2949767306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a140f8500 0x7f9a1410e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:04.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a19f6a700 1 -- 172.21.15.162:0/2949767306 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a10005040 con 0x7f9a140fa440 2024-02-13T00:39:04.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a19769700 1 --2- 172.21.15.162:0/2949767306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a140f8500 0x7f9a1410e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:39:04.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a19f6a700 1 --2- 172.21.15.162:0/2949767306 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a140fa440 0x7f9a141081f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f9a0c00b6d0 tx=0x7f9a0c004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:04.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a0a7fc700 10 client.?.objecter ms_handle_connect 0x7f9a140fa440 2024-02-13T00:39:04.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a0a7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:04.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a0a7fc700 1 -- 172.21.15.162:0/2949767306 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c00e040 con 0x7f9a140fa440 2024-02-13T00:39:04.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a0a7fc700 1 -- 172.21.15.162:0/2949767306 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a0c018040 con 0x7f9a140fa440 2024-02-13T00:39:04.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949767306 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a0c0097c0 con 0x7f9a140fa440 2024-02-13T00:39:04.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a0a7fc700 1 -- 172.21.15.162:0/2949767306 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c01c3f0 con 0x7f9a140fa440 2024-02-13T00:39:04.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a1b9cd700 10 client.24572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:04.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949767306 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a14108f30 con 0x7f9a140fa440 2024-02-13T00:39:04.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.440+0000 7f9a1b9cd700 1 librados: init done 2024-02-13T00:39:04.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.441+0000 7f9a1b9cd700 10 client.24572.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:39:04.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.441+0000 7f9a1b9cd700 15 client.24572.objecter _session_command_op_assign -1 1 2024-02-13T00:39:04.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.441+0000 7f9a1b9cd700 20 client.24572.objecter _get_session osd=-1 returning homeless 2024-02-13T00:39:04.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.441+0000 7f9a1b9cd700 10 client.24572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:04.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.441+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949767306 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9a140475e0 con 0x7f9a140fa440 2024-02-13T00:39:04.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 1 -- 172.21.15.162:0/2949767306 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f9a0c013420 con 0x7f9a140fa440 2024-02-13T00:39:04.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 1 --2- 172.21.15.162:0/2949767306 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9a00062c90 0x7f9a00065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:04.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 1 -- 172.21.15.162:0/2949767306 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(80..80 src has 30..80) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f9a0c07da30 con 0x7f9a140fa440 2024-02-13T00:39:04.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 10 client.24572.objecter ms_dispatch 0x7f9a14001550 osd_map(80..80 src has 30..80) v4 2024-02-13T00:39:04.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 3 client.24572.objecter handle_osd_map got epochs [80,80] > 0 2024-02-13T00:39:04.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 3 client.24572.objecter handle_osd_map decoding full epoch 80 2024-02-13T00:39:04.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a19769700 1 --2- 172.21.15.162:0/2949767306 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9a00062c90 0x7f9a00065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:04.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 10 client.24572.objecter checking command 1 2024-02-13T00:39:04.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 1 --2- 172.21.15.162:0/2949767306 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9a0006c530 0x7f9a0006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:04.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 20 client.24572.objecter _get_session s=0x7f9a0006be80 osd=6 3 2024-02-13T00:39:04.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 20 client.24572.objecter put_session s=0x7f9a0006be80 osd=6 3 2024-02-13T00:39:04.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 15 client.24572.objecter _session_command_op_remove -1 1 2024-02-13T00:39:04.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 20 client.24572.objecter _get_session s=0x7f9a0006be80 osd=6 3 2024-02-13T00:39:04.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 20 client.24572.objecter get_session s=0x7f9a0006be80 osd=6 3 2024-02-13T00:39:04.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 15 client.24572.objecter _session_command_op_assign 6 1 2024-02-13T00:39:04.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 20 client.24572.objecter put_session s=0x7f9a0006be80 osd=6 4 2024-02-13T00:39:04.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 10 client.24572.objecter _send_command 1 2024-02-13T00:39:04.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 1 -- 172.21.15.162:0/2949767306 --> v2:172.21.15.181:6808/1506640341 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a0006f010 con 0x7f9a0006c530 2024-02-13T00:39:04.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 20 client.24572.objecter dump_active .. 0 homeless 2024-02-13T00:39:04.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a0a7fc700 1 -- 172.21.15.162:0/2949767306 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a0c07fea0 con 0x7f9a140fa440 2024-02-13T00:39:04.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a18f68700 1 --2- 172.21.15.162:0/2949767306 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9a0006c530 0x7f9a0006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:04.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.442+0000 7f9a19769700 1 --2- 172.21.15.162:0/2949767306 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9a00062c90 0x7f9a00065150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9a10009810 tx=0x7f9a1000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:04.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.443+0000 7f9a0a7fc700 10 client.24572.objecter ms_handle_connect 0x7f9a00062c90 2024-02-13T00:39:04.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.443+0000 7f9a0a7fc700 10 client.24572.objecter ms_handle_connect 0x7f9a0006c530 2024-02-13T00:39:04.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.443+0000 7f9a18f68700 1 --2- 172.21.15.162:0/2949767306 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9a0006c530 0x7f9a0006e9d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:04.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.443+0000 7f9a0a7fc700 1 -- 172.21.15.162:0/2949767306 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9a0006f010 con 0x7f9a0006c530 2024-02-13T00:39:04.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.443+0000 7f9a0a7fc700 10 client.24572.objecter ms_dispatch 0x7f9a14001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:39:04.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.443+0000 7f9a0a7fc700 10 client.24572.objecter _finish_command 1 = system:0 2024-02-13T00:39:04.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.443+0000 7f9a0a7fc700 20 client.24572.objecter put_session s=0x7f9a0006be80 osd=6 4 2024-02-13T00:39:04.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.443+0000 7f9a0a7fc700 15 client.24572.objecter _session_command_op_remove 6 1 2024-02-13T00:39:04.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.460+0000 7f9a1b9cd700 10 client.24572.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:39:04.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.460+0000 7f9a1b9cd700 15 client.24572.objecter _session_command_op_assign -1 2 2024-02-13T00:39:04.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.460+0000 7f9a1b9cd700 20 client.24572.objecter _get_session s=0x7f9a0006be80 osd=6 3 2024-02-13T00:39:04.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.460+0000 7f9a1b9cd700 20 client.24572.objecter put_session s=0x7f9a0006be80 osd=6 3 2024-02-13T00:39:04.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.460+0000 7f9a1b9cd700 20 client.24572.objecter _get_session s=0x7f9a0006be80 osd=6 3 2024-02-13T00:39:04.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.460+0000 7f9a1b9cd700 15 client.24572.objecter _session_command_op_remove -1 2 2024-02-13T00:39:04.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.460+0000 7f9a1b9cd700 20 client.24572.objecter get_session s=0x7f9a0006be80 osd=6 3 2024-02-13T00:39:04.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.460+0000 7f9a1b9cd700 15 client.24572.objecter _session_command_op_assign 6 2 2024-02-13T00:39:04.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.460+0000 7f9a1b9cd700 20 client.24572.objecter put_session s=0x7f9a0006be80 osd=6 4 2024-02-13T00:39:04.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.460+0000 7f9a1b9cd700 10 client.24572.objecter _send_command 2 2024-02-13T00:39:04.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.460+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949767306 --> v2:172.21.15.181:6808/1506640341 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9a141072c0 con 0x7f9a0006c530 2024-02-13T00:39:04.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a0a7fc700 1 -- 172.21.15.162:0/2949767306 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9a141072c0 con 0x7f9a0006c530 2024-02-13T00:39:04.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a0a7fc700 10 client.24572.objecter ms_dispatch 0x7f9a14001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:39:04.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a0a7fc700 10 client.24572.objecter _finish_command 2 = system:0 2024-02-13T00:39:04.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a0a7fc700 20 client.24572.objecter put_session s=0x7f9a0006be80 osd=6 4 2024-02-13T00:39:04.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a0a7fc700 15 client.24572.objecter _session_command_op_remove 6 2 2024-02-13T00:39:04.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a1b9cd700 10 librados: watch_flush enter 2024-02-13T00:39:04.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a1b9cd700 10 librados: watch_flush exit 2024-02-13T00:39:04.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a1b9cd700 10 client.24572.objecter close_session for osd.6 2024-02-13T00:39:04.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949767306 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9a0006c530 msgr2=0x7f9a0006e9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:04.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a1b9cd700 1 --2- 172.21.15.162:0/2949767306 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9a0006c530 0x7f9a0006e9d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:04.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a1b9cd700 20 client.24572.objecter put_session s=0x7f9a0006be80 osd=6 1 2024-02-13T00:39:04.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a1b9cd700 20 client.24572.objecter shutdown clearing up homeless session... 2024-02-13T00:39:04.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a1b9cd700 10 client.24572.objecter successfully canceled tick 2024-02-13T00:39:04.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949767306 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9a00062c90 msgr2=0x7f9a00065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:04.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a1b9cd700 1 --2- 172.21.15.162:0/2949767306 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9a00062c90 0x7f9a00065150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9a10009810 tx=0x7f9a1000b040).stop 2024-02-13T00:39:04.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949767306 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a140fa440 msgr2=0x7f9a141081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:04.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a1b9cd700 1 --2- 172.21.15.162:0/2949767306 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a140fa440 0x7f9a141081f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f9a0c00b6d0 tx=0x7f9a0c004d10).stop 2024-02-13T00:39:04.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.461+0000 7f9a19f6a700 1 -- 172.21.15.162:0/2949767306 reap_dead start 2024-02-13T00:39:04.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.462+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949767306 shutdown_connections 2024-02-13T00:39:04.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.462+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949767306 >> 172.21.15.162:0/2949767306 conn(0x7f9a140f38c0 msgr2=0x7f9a140f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:04.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.462+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949767306 shutdown_connections 2024-02-13T00:39:04.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.462+0000 7f9a1b9cd700 1 -- 172.21.15.162:0/2949767306 wait complete. 2024-02-13T00:39:04.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:04.462+0000 7f9a1b9cd700 1 librados: shutdown 2024-02-13T00:39:04.545 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-02-13T00:39:04.545 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-02-13T00:39:04.545 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-02-13T00:39:04.545 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-02-13T00:39:04.545 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-02-13T00:39:04.546 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-02-13T00:39:04.546 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-02-13T00:39:04.546 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-02-13T00:39:04.546 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-02-13T00:39:04.546 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-02-13T00:39:04.546 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-02-13T00:39:04.546 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-02-13T00:39:04.546 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-02-13T00:39:04.546 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-02-13T00:39:04.546 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-02-13T00:39:04.546 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=537.137227ms 2024-02-13T00:39:04.546 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-13T00:39:04.547 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-13T00:39:04.547 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-13T00:39:04.547 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-13T00:39:04.547 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-13T00:39:04.547 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-13T00:39:04.547 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-13T00:39:04.547 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-13T00:39:04.547 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-13T00:39:04.547 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-13T00:39:04.547 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:39:04+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-13T00:39:05.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:04.172077+0000 mgr.y (mgr.14544) 51 : cluster [DBG] pgmap v45: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-02-13T00:39:05.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.177429+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-13T00:39:05.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.177633+0000 mon.c (mon.2) 56 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-13T00:39:05.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.177791+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-13T00:39:05.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.177799+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-13T00:39:05.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.178057+0000 mon.c (mon.2) 58 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-13T00:39:05.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.178254+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-13T00:39:05.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.178517+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-13T00:39:05.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.178686+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-13T00:39:05.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:04.218567+0000 osd.3 (osd.3) 21 : cluster [DBG] 3.2 deep-scrub starts 2024-02-13T00:39:05.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:04.221497+0000 osd.3 (osd.3) 22 : cluster [DBG] 3.2 deep-scrub ok 2024-02-13T00:39:05.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:04.312100+0000 osd.0 (osd.0) 21 : cluster [DBG] 4.3 deep-scrub starts 2024-02-13T00:39:05.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:04.318917+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.3 deep-scrub ok 2024-02-13T00:39:05.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.360060+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:05.193 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:04.172077+0000 mgr.y (mgr.14544) 51 : cluster [DBG] pgmap v45: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-02-13T00:39:05.193 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.177429+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-13T00:39:05.193 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.177633+0000 mon.c (mon.2) 56 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-13T00:39:05.193 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.177791+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-13T00:39:05.193 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.177799+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-13T00:39:05.193 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.178057+0000 mon.c (mon.2) 58 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-13T00:39:05.194 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.178254+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-13T00:39:05.194 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.178517+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-13T00:39:05.194 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.178686+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-13T00:39:05.194 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:04.218567+0000 osd.3 (osd.3) 21 : cluster [DBG] 3.2 deep-scrub starts 2024-02-13T00:39:05.194 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:04.221497+0000 osd.3 (osd.3) 22 : cluster [DBG] 3.2 deep-scrub ok 2024-02-13T00:39:05.194 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:04.312100+0000 osd.0 (osd.0) 21 : cluster [DBG] 4.3 deep-scrub starts 2024-02-13T00:39:05.194 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:04.318917+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.3 deep-scrub ok 2024-02-13T00:39:05.194 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.360060+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:05.210 INFO:teuthology.orchestra.run.smithi162.stdout:193273528344 2024-02-13T00:39:05.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-13T00:39:05.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:04.172077+0000 mgr.y (mgr.14544) 51 : cluster [DBG] pgmap v45: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-02-13T00:39:05.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.177429+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-13T00:39:05.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.177633+0000 mon.c (mon.2) 56 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-13T00:39:05.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.177791+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-02-13T00:39:05.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.177799+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-13T00:39:05.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.178057+0000 mon.c (mon.2) 58 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-13T00:39:05.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.178254+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-13T00:39:05.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.178517+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-13T00:39:05.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.178686+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-13T00:39:05.224 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:04.218567+0000 osd.3 (osd.3) 21 : cluster [DBG] 3.2 deep-scrub starts 2024-02-13T00:39:05.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:04.221497+0000 osd.3 (osd.3) 22 : cluster [DBG] 3.2 deep-scrub ok 2024-02-13T00:39:05.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:04.312100+0000 osd.0 (osd.0) 21 : cluster [DBG] 4.3 deep-scrub starts 2024-02-13T00:39:05.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:04.318917+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.3 deep-scrub ok 2024-02-13T00:39:05.225 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.360060+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:05.500 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:39:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:39:05.929 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:06.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.954685+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-02-13T00:39:06.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.954769+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-13T00:39:06.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.954814+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-02-13T00:39:06.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:04.954856+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-13T00:39:06.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:04.954888+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-13T00:39:06.213 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:05.220170+0000 mgr.y (mgr.14544) 52 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:06.213 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.954685+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-02-13T00:39:06.213 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.954769+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-13T00:39:06.213 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.954814+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-02-13T00:39:06.213 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:04.954856+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-13T00:39:06.214 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:04.954888+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-13T00:39:06.214 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:05.220170+0000 mgr.y (mgr.14544) 52 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:06.214 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:39:06] "GET /metrics HTTP/1.1" 200 172171 "" "Prometheus/2.33.4" 2024-02-13T00:39:06.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.954685+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-02-13T00:39:06.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.954769+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-13T00:39:06.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.954814+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-02-13T00:39:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:04.954856+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-13T00:39:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:04.954888+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-13T00:39:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:05.220170+0000 mgr.y (mgr.14544) 52 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:07.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:05.595698+0000 osd.5 (osd.5) 29 : cluster [DBG] 6.7 deep-scrub starts 2024-02-13T00:39:07.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:05.596459+0000 osd.5 (osd.5) 30 : cluster [DBG] 6.7 deep-scrub ok 2024-02-13T00:39:07.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:05.961843+0000 mon.a (mon.0) 690 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-13T00:39:07.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:06.172464+0000 mgr.y (mgr.14544) 53 : cluster [DBG] pgmap v48: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:39:07.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:05.595698+0000 osd.5 (osd.5) 29 : cluster [DBG] 6.7 deep-scrub starts 2024-02-13T00:39:07.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:05.596459+0000 osd.5 (osd.5) 30 : cluster [DBG] 6.7 deep-scrub ok 2024-02-13T00:39:07.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:05.961843+0000 mon.a (mon.0) 690 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-13T00:39:07.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:06.172464+0000 mgr.y (mgr.14544) 53 : cluster [DBG] pgmap v48: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:39:07.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:05.595698+0000 osd.5 (osd.5) 29 : cluster [DBG] 6.7 deep-scrub starts 2024-02-13T00:39:07.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:05.596459+0000 osd.5 (osd.5) 30 : cluster [DBG] 6.7 deep-scrub ok 2024-02-13T00:39:07.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:05.961843+0000 mon.a (mon.0) 690 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-13T00:39:07.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:06.172464+0000 mgr.y (mgr.14544) 53 : cluster [DBG] pgmap v48: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:39:07.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.593+0000 7f1a25e69700 1 -- 172.21.15.162:0/912186349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a200fa440 msgr2=0x7f1a200f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:07.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.593+0000 7f1a25e69700 1 --2- 172.21.15.162:0/912186349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a200fa440 0x7f1a200f7e20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f1a140047b0 tx=0x7f1a14009cb0).stop 2024-02-13T00:39:07.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.593+0000 7f1a25e69700 1 -- 172.21.15.162:0/912186349 shutdown_connections 2024-02-13T00:39:07.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.593+0000 7f1a25e69700 1 --2- 172.21.15.162:0/912186349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a200f8500 0x7f1a20104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.593+0000 7f1a25e69700 1 --2- 172.21.15.162:0/912186349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a200f9b70 0x7f1a200f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.593+0000 7f1a25e69700 1 --2- 172.21.15.162:0/912186349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a200fa440 0x7f1a200f7e20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.593+0000 7f1a25e69700 1 -- 172.21.15.162:0/912186349 >> 172.21.15.162:0/912186349 conn(0x7f1a200f38c0 msgr2=0x7f1a200f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.593+0000 7f1a25e69700 1 -- 172.21.15.162:0/912186349 shutdown_connections 2024-02-13T00:39:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.593+0000 7f1a25e69700 1 -- 172.21.15.162:0/912186349 wait complete. 2024-02-13T00:39:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.594+0000 7f1a25e69700 1 librados: starting msgr at 2024-02-13T00:39:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.594+0000 7f1a25e69700 1 librados: starting objecter 2024-02-13T00:39:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.594+0000 7f1a25e69700 1 Processor -- start 2024-02-13T00:39:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.594+0000 7f1a25e69700 1 -- start start 2024-02-13T00:39:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.594+0000 7f1a25e69700 1 librados: setting wanted keys 2024-02-13T00:39:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.594+0000 7f1a25e69700 1 librados: calling monclient init 2024-02-13T00:39:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.594+0000 7f1a25e69700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a200f8500 0x7f1a2010e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.594+0000 7f1a25e69700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a200f9b70 0x7f1a2010e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.594+0000 7f1a25e69700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a200fa440 0x7f1a201084c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.594+0000 7f1a25e69700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a201089b0 con 0x7f1a200f9b70 2024-02-13T00:39:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.594+0000 7f1a25e69700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a20108b20 con 0x7f1a200f8500 2024-02-13T00:39:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.594+0000 7f1a25e69700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a20108c90 con 0x7f1a200fa440 2024-02-13T00:39:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1effd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a200f9b70 0x7f1a2010e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1ffff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a200fa440 0x7f1a201084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1f7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a200f8500 0x7f1a2010e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1effd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a200f9b70 0x7f1a2010e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:40888/0 (socket says 172.21.15.162:40888) 2024-02-13T00:39:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1ffff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a200fa440 0x7f1a201084c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:37936/0 (socket says 172.21.15.162:37936) 2024-02-13T00:39:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1effd700 1 -- 172.21.15.162:0/2920640942 learned_addr learned my addr 172.21.15.162:0/2920640942 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:07.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1effd700 1 -- 172.21.15.162:0/2920640942 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a200fa440 msgr2=0x7f1a201084c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1effd700 1 --2- 172.21.15.162:0/2920640942 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a200fa440 0x7f1a201084c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1effd700 1 -- 172.21.15.162:0/2920640942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a200f8500 msgr2=0x7f1a2010e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1effd700 1 --2- 172.21.15.162:0/2920640942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a200f8500 0x7f1a2010e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1effd700 1 -- 172.21.15.162:0/2920640942 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a10005040 con 0x7f1a200f9b70 2024-02-13T00:39:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1ffff700 1 --2- 172.21.15.162:0/2920640942 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a200fa440 0x7f1a201084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:39:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1effd700 1 --2- 172.21.15.162:0/2920640942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a200f9b70 0x7f1a2010e780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f1a140044e0 tx=0x7f1a14007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1cff9700 10 client.?.objecter ms_handle_connect 0x7f1a200f9b70 2024-02-13T00:39:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1cff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1cff9700 1 -- 172.21.15.162:0/2920640942 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a14018070 con 0x7f1a200f9b70 2024-02-13T00:39:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1cff9700 1 -- 172.21.15.162:0/2920640942 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a140093b0 con 0x7f1a200f9b70 2024-02-13T00:39:07.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a25e69700 1 -- 172.21.15.162:0/2920640942 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a14005040 con 0x7f1a200f9b70 2024-02-13T00:39:07.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a1cff9700 1 -- 172.21.15.162:0/2920640942 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1401dd90 con 0x7f1a200f9b70 2024-02-13T00:39:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a25e69700 10 client.14679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a25e69700 1 -- 172.21.15.162:0/2920640942 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a20109130 con 0x7f1a200f9b70 2024-02-13T00:39:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.595+0000 7f1a25e69700 1 librados: init done 2024-02-13T00:39:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.596+0000 7f1a25e69700 10 client.14679.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:39:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.596+0000 7f1a25e69700 15 client.14679.objecter _session_command_op_assign -1 1 2024-02-13T00:39:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.596+0000 7f1a25e69700 20 client.14679.objecter _get_session osd=-1 returning homeless 2024-02-13T00:39:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.596+0000 7f1a25e69700 10 client.14679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.596+0000 7f1a25e69700 1 -- 172.21.15.162:0/2920640942 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1a20109350 con 0x7f1a200f9b70 2024-02-13T00:39:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 1 -- 172.21.15.162:0/2920640942 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1a1401d420 con 0x7f1a200f9b70 2024-02-13T00:39:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 1 --2- 172.21.15.162:0/2920640942 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1a08062c90 0x7f1a08065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 1 -- 172.21.15.162:0/2920640942 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(82..82 src has 30..82) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f1a1407de70 con 0x7f1a200f9b70 2024-02-13T00:39:07.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 10 client.14679.objecter ms_dispatch 0x7f1a20001550 osd_map(82..82 src has 30..82) v4 2024-02-13T00:39:07.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 3 client.14679.objecter handle_osd_map got epochs [82,82] > 0 2024-02-13T00:39:07.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 3 client.14679.objecter handle_osd_map decoding full epoch 82 2024-02-13T00:39:07.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1f7fe700 1 --2- 172.21.15.162:0/2920640942 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1a08062c90 0x7f1a08065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:07.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 10 client.14679.objecter checking command 1 2024-02-13T00:39:07.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 1 --2- 172.21.15.162:0/2920640942 >> v2:172.21.15.181:6812/1964192772 conn(0x7f1a0806c650 0x7f1a0806eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:07.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 20 client.14679.objecter _get_session s=0x7f1a0806bfa0 osd=7 3 2024-02-13T00:39:07.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 20 client.14679.objecter put_session s=0x7f1a0806bfa0 osd=7 3 2024-02-13T00:39:07.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 15 client.14679.objecter _session_command_op_remove -1 1 2024-02-13T00:39:07.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 20 client.14679.objecter _get_session s=0x7f1a0806bfa0 osd=7 3 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 20 client.14679.objecter get_session s=0x7f1a0806bfa0 osd=7 3 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 15 client.14679.objecter _session_command_op_assign 7 1 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 20 client.14679.objecter put_session s=0x7f1a0806bfa0 osd=7 4 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 10 client.14679.objecter _send_command 1 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 1 -- 172.21.15.162:0/2920640942 --> v2:172.21.15.181:6812/1964192772 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a0806f130 con 0x7f1a0806c650 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 20 client.14679.objecter dump_active .. 0 homeless 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 1 -- 172.21.15.162:0/2920640942 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a14080490 con 0x7f1a200f9b70 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1ffff700 1 --2- 172.21.15.162:0/2920640942 >> v2:172.21.15.181:6812/1964192772 conn(0x7f1a0806c650 0x7f1a0806eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1f7fe700 1 --2- 172.21.15.162:0/2920640942 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1a08062c90 0x7f1a08065150 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f1a10004870 tx=0x7f1a1000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.597+0000 7f1a1cff9700 10 client.14679.objecter ms_handle_connect 0x7f1a08062c90 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.598+0000 7f1a1cff9700 10 client.14679.objecter ms_handle_connect 0x7f1a0806c650 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.598+0000 7f1a1ffff700 1 --2- 172.21.15.162:0/2920640942 >> v2:172.21.15.181:6812/1964192772 conn(0x7f1a0806c650 0x7f1a0806eaf0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.598+0000 7f1a1cff9700 1 -- 172.21.15.162:0/2920640942 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1a0806f130 con 0x7f1a0806c650 2024-02-13T00:39:07.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.598+0000 7f1a1cff9700 10 client.14679.objecter ms_dispatch 0x7f1a20001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:39:07.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.598+0000 7f1a1cff9700 10 client.14679.objecter _finish_command 1 = system:0 2024-02-13T00:39:07.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.598+0000 7f1a1cff9700 20 client.14679.objecter put_session s=0x7f1a0806bfa0 osd=7 4 2024-02-13T00:39:07.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.598+0000 7f1a1cff9700 15 client.14679.objecter _session_command_op_remove 7 1 2024-02-13T00:39:07.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.613+0000 7f1a25e69700 10 client.14679.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:39:07.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 15 client.14679.objecter _session_command_op_assign -1 2 2024-02-13T00:39:07.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 20 client.14679.objecter _get_session s=0x7f1a0806bfa0 osd=7 3 2024-02-13T00:39:07.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 20 client.14679.objecter put_session s=0x7f1a0806bfa0 osd=7 3 2024-02-13T00:39:07.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 20 client.14679.objecter _get_session s=0x7f1a0806bfa0 osd=7 3 2024-02-13T00:39:07.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 15 client.14679.objecter _session_command_op_remove -1 2 2024-02-13T00:39:07.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 20 client.14679.objecter get_session s=0x7f1a0806bfa0 osd=7 3 2024-02-13T00:39:07.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 15 client.14679.objecter _session_command_op_assign 7 2 2024-02-13T00:39:07.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 20 client.14679.objecter put_session s=0x7f1a0806bfa0 osd=7 4 2024-02-13T00:39:07.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 10 client.14679.objecter _send_command 2 2024-02-13T00:39:07.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 1 -- 172.21.15.162:0/2920640942 --> v2:172.21.15.181:6812/1964192772 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1a200475e0 con 0x7f1a0806c650 2024-02-13T00:39:07.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a1cff9700 1 -- 172.21.15.162:0/2920640942 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1a200475e0 con 0x7f1a0806c650 2024-02-13T00:39:07.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a1cff9700 10 client.14679.objecter ms_dispatch 0x7f1a20001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:39:07.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a1cff9700 10 client.14679.objecter _finish_command 2 = system:0 2024-02-13T00:39:07.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a1cff9700 20 client.14679.objecter put_session s=0x7f1a0806bfa0 osd=7 4 2024-02-13T00:39:07.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a1cff9700 15 client.14679.objecter _session_command_op_remove 7 2 2024-02-13T00:39:07.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 10 librados: watch_flush enter 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 10 librados: watch_flush exit 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 10 client.14679.objecter close_session for osd.7 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 1 -- 172.21.15.162:0/2920640942 >> v2:172.21.15.181:6812/1964192772 conn(0x7f1a0806c650 msgr2=0x7f1a0806eaf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 1 --2- 172.21.15.162:0/2920640942 >> v2:172.21.15.181:6812/1964192772 conn(0x7f1a0806c650 0x7f1a0806eaf0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 20 client.14679.objecter put_session s=0x7f1a0806bfa0 osd=7 1 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 20 client.14679.objecter shutdown clearing up homeless session... 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.614+0000 7f1a25e69700 10 client.14679.objecter successfully canceled tick 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.615+0000 7f1a25e69700 1 -- 172.21.15.162:0/2920640942 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1a08062c90 msgr2=0x7f1a08065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.615+0000 7f1a25e69700 1 --2- 172.21.15.162:0/2920640942 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1a08062c90 0x7f1a08065150 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f1a10004870 tx=0x7f1a1000b040).stop 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.615+0000 7f1a25e69700 1 -- 172.21.15.162:0/2920640942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a200f9b70 msgr2=0x7f1a2010e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.615+0000 7f1a25e69700 1 --2- 172.21.15.162:0/2920640942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a200f9b70 0x7f1a2010e780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f1a140044e0 tx=0x7f1a14007f90).stop 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.615+0000 7f1a1ffff700 1 -- 172.21.15.162:0/2920640942 reap_dead start 2024-02-13T00:39:07.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.615+0000 7f1a25e69700 1 -- 172.21.15.162:0/2920640942 shutdown_connections 2024-02-13T00:39:07.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.615+0000 7f1a25e69700 1 -- 172.21.15.162:0/2920640942 >> 172.21.15.162:0/2920640942 conn(0x7f1a200f38c0 msgr2=0x7f1a200f43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:07.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.615+0000 7f1a25e69700 1 -- 172.21.15.162:0/2920640942 shutdown_connections 2024-02-13T00:39:07.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.615+0000 7f1a25e69700 1 -- 172.21.15.162:0/2920640942 wait complete. 2024-02-13T00:39:07.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:07.615+0000 7f1a25e69700 1 librados: shutdown 2024-02-13T00:39:08.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:06.578712+0000 osd.5 (osd.5) 31 : cluster [DBG] 5.1b scrub starts 2024-02-13T00:39:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:06.581604+0000 osd.5 (osd.5) 32 : cluster [DBG] 5.1b scrub ok 2024-02-13T00:39:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:07.357719+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:39:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:07.358526+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:39:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:07.560533+0000 osd.2 (osd.2) 11 : cluster [DBG] 6.f scrub starts 2024-02-13T00:39:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:07.563436+0000 osd.2 (osd.2) 12 : cluster [DBG] 6.f scrub ok 2024-02-13T00:39:08.297 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:06.578712+0000 osd.5 (osd.5) 31 : cluster [DBG] 5.1b scrub starts 2024-02-13T00:39:08.297 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:06.581604+0000 osd.5 (osd.5) 32 : cluster [DBG] 5.1b scrub ok 2024-02-13T00:39:08.297 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:07.357719+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:39:08.297 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:07.358526+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:39:08.297 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:07.560533+0000 osd.2 (osd.2) 11 : cluster [DBG] 6.f scrub starts 2024-02-13T00:39:08.298 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:07.563436+0000 osd.2 (osd.2) 12 : cluster [DBG] 6.f scrub ok 2024-02-13T00:39:08.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:06.578712+0000 osd.5 (osd.5) 31 : cluster [DBG] 5.1b scrub starts 2024-02-13T00:39:08.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:06.581604+0000 osd.5 (osd.5) 32 : cluster [DBG] 5.1b scrub ok 2024-02-13T00:39:08.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:07.357719+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:39:08.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:07.358526+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:39:08.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:07.560533+0000 osd.2 (osd.2) 11 : cluster [DBG] 6.f scrub starts 2024-02-13T00:39:08.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:07.563436+0000 osd.2 (osd.2) 12 : cluster [DBG] 6.f scrub ok 2024-02-13T00:39:08.599 INFO:teuthology.orchestra.run.smithi162.stdout:227633266707 2024-02-13T00:39:08.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-13T00:39:08.971 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:39:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:39:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:39:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:07.555944+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.16 scrub starts 2024-02-13T00:39:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:07.556765+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.16 scrub ok 2024-02-13T00:39:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:08.172881+0000 mgr.y (mgr.14544) 54 : cluster [DBG] pgmap v49: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:39:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:08.275497+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:08.580563+0000 osd.2 (osd.2) 13 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:39:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:08.582473+0000 osd.2 (osd.2) 14 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:39:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:08.929508+0000 osd.7 (osd.7) 21 : cluster [DBG] 6.5 scrub starts 2024-02-13T00:39:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:08.930281+0000 osd.7 (osd.7) 22 : cluster [DBG] 6.5 scrub ok 2024-02-13T00:39:09.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:07.555944+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.16 scrub starts 2024-02-13T00:39:09.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:07.556765+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.16 scrub ok 2024-02-13T00:39:09.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:08.172881+0000 mgr.y (mgr.14544) 54 : cluster [DBG] pgmap v49: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:39:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:08.275497+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:08.580563+0000 osd.2 (osd.2) 13 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:39:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:08.582473+0000 osd.2 (osd.2) 14 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:39:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:08.929508+0000 osd.7 (osd.7) 21 : cluster [DBG] 6.5 scrub starts 2024-02-13T00:39:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:08.930281+0000 osd.7 (osd.7) 22 : cluster [DBG] 6.5 scrub ok 2024-02-13T00:39:09.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:07.555944+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.16 scrub starts 2024-02-13T00:39:09.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:07.556765+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.16 scrub ok 2024-02-13T00:39:09.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:08.172881+0000 mgr.y (mgr.14544) 54 : cluster [DBG] pgmap v49: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:39:09.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:08.275497+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:09.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:08.580563+0000 osd.2 (osd.2) 13 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:39:09.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:08.582473+0000 osd.2 (osd.2) 14 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:39:09.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:08.929508+0000 osd.7 (osd.7) 21 : cluster [DBG] 6.5 scrub starts 2024-02-13T00:39:09.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:08.930281+0000 osd.7 (osd.7) 22 : cluster [DBG] 6.5 scrub ok 2024-02-13T00:39:09.810 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:10.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:08.340659+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:39:10.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:08.344125+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:39:10.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:09.234053+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.2 scrub starts 2024-02-13T00:39:10.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:09.237309+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.2 scrub ok 2024-02-13T00:39:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:09.393642+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:39:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:09.393713+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:39:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:09.586918+0000 osd.2 (osd.2) 17 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:39:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:09.589809+0000 osd.2 (osd.2) 18 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:39:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:09.838356+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:10.146526+0000 mon.c (mon.2) 59 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:39:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:10.147202+0000 mon.c (mon.2) 60 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:39:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:10.151673+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:10.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:08.340659+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:39:10.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:08.344125+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:39:10.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:09.234053+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.2 scrub starts 2024-02-13T00:39:10.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:09.237309+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.2 scrub ok 2024-02-13T00:39:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:09.393642+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:39:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:09.393713+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:39:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:09.586918+0000 osd.2 (osd.2) 17 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:39:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:09.589809+0000 osd.2 (osd.2) 18 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:39:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:09.838356+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:10.146526+0000 mon.c (mon.2) 59 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:39:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:10.147202+0000 mon.c (mon.2) 60 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:39:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:10.151673+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:10.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:08.340659+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:39:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:08.344125+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:39:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:09.234053+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.2 scrub starts 2024-02-13T00:39:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:09.237309+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.2 scrub ok 2024-02-13T00:39:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:09.393642+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:39:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:09.393713+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:39:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:09.586918+0000 osd.2 (osd.2) 17 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:39:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:09.589809+0000 osd.2 (osd.2) 18 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:39:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:09.838356+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:10.146526+0000 mon.c (mon.2) 59 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:39:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:10.147202+0000 mon.c (mon.2) 60 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:39:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:10.151673+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:11.393 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:39:10.155103+0000 mgr.y (mgr.14544) 55 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-13T00:39:11.394 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:39:10.157784+0000 mgr.y (mgr.14544) 56 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi162 2024-02-13T00:39:11.394 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:10.173394+0000 mgr.y (mgr.14544) 57 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 2 objects/s recovering 2024-02-13T00:39:11.394 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:10.227680+0000 mon.a (mon.0) 694 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-13T00:39:11.394 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:10.283705+0000 osd.1 (osd.1) 19 : cluster [DBG] 3.0 scrub starts 2024-02-13T00:39:11.394 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:10.287292+0000 osd.1 (osd.1) 20 : cluster [DBG] 3.0 scrub ok 2024-02-13T00:39:11.394 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:10.361488+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:39:11.394 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:10.364860+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:39:11.394 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:10.547019+0000 osd.5 (osd.5) 35 : cluster [DBG] 4.15 scrub starts 2024-02-13T00:39:11.394 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:10.549028+0000 osd.5 (osd.5) 36 : cluster [DBG] 4.15 scrub ok 2024-02-13T00:39:11.395 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:39:10.155103+0000 mgr.y (mgr.14544) 55 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-13T00:39:11.395 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:39:10.157784+0000 mgr.y (mgr.14544) 56 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi162 2024-02-13T00:39:11.395 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:10.173394+0000 mgr.y (mgr.14544) 57 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 2 objects/s recovering 2024-02-13T00:39:11.395 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:10.227680+0000 mon.a (mon.0) 694 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-13T00:39:11.395 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:10.283705+0000 osd.1 (osd.1) 19 : cluster [DBG] 3.0 scrub starts 2024-02-13T00:39:11.395 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:10.287292+0000 osd.1 (osd.1) 20 : cluster [DBG] 3.0 scrub ok 2024-02-13T00:39:11.395 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:10.361488+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:39:11.396 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:10.364860+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:39:11.396 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:10.547019+0000 osd.5 (osd.5) 35 : cluster [DBG] 4.15 scrub starts 2024-02-13T00:39:11.396 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:10.549028+0000 osd.5 (osd.5) 36 : cluster [DBG] 4.15 scrub ok 2024-02-13T00:39:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:39:10.155103+0000 mgr.y (mgr.14544) 55 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-13T00:39:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:39:10.157784+0000 mgr.y (mgr.14544) 56 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi162 2024-02-13T00:39:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:10.173394+0000 mgr.y (mgr.14544) 57 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 2 objects/s recovering 2024-02-13T00:39:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:10.227680+0000 mon.a (mon.0) 694 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-13T00:39:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:10.283705+0000 osd.1 (osd.1) 19 : cluster [DBG] 3.0 scrub starts 2024-02-13T00:39:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:10.287292+0000 osd.1 (osd.1) 20 : cluster [DBG] 3.0 scrub ok 2024-02-13T00:39:11.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:10.361488+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:39:11.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:10.364860+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:39:11.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:10.547019+0000 osd.5 (osd.5) 35 : cluster [DBG] 4.15 scrub starts 2024-02-13T00:39:11.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:10.549028+0000 osd.5 (osd.5) 36 : cluster [DBG] 4.15 scrub ok 2024-02-13T00:39:11.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.762+0000 7ff1c42a3700 1 -- 172.21.15.162:0/895253726 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0f6b20 msgr2=0x7ff1bc103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:11.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.762+0000 7ff1c42a3700 1 --2- 172.21.15.162:0/895253726 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0f6b20 0x7ff1bc103120 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7ff1b4009be0 tx=0x7ff1b4004d10).stop 2024-02-13T00:39:11.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.762+0000 7ff1c42a3700 1 -- 172.21.15.162:0/895253726 shutdown_connections 2024-02-13T00:39:11.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.762+0000 7ff1c42a3700 1 --2- 172.21.15.162:0/895253726 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0f6b20 0x7ff1bc103120 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:11.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.762+0000 7ff1c42a3700 1 --2- 172.21.15.162:0/895253726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc0fc2b0 0x7ff1bc0fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:11.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.762+0000 7ff1c42a3700 1 --2- 172.21.15.162:0/895253726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc0fcc10 0x7ff1bc0f6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:11.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.762+0000 7ff1c42a3700 1 -- 172.21.15.162:0/895253726 >> 172.21.15.162:0/895253726 conn(0x7ff1bc070080 msgr2=0x7ff1bc0f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:11.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.762+0000 7ff1c42a3700 1 -- 172.21.15.162:0/895253726 shutdown_connections 2024-02-13T00:39:11.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 -- 172.21.15.162:0/895253726 wait complete. 2024-02-13T00:39:11.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 librados: starting msgr at 2024-02-13T00:39:11.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 librados: starting objecter 2024-02-13T00:39:11.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 Processor -- start 2024-02-13T00:39:11.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 -- start start 2024-02-13T00:39:11.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 librados: setting wanted keys 2024-02-13T00:39:11.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 librados: calling monclient init 2024-02-13T00:39:11.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc0f6b20 0x7ff1bc10ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:11.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc0fc2b0 0x7ff1bc10e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:11.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0fcc10 0x7ff1bc1080e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:11.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc1085d0 con 0x7ff1bc0fc2b0 2024-02-13T00:39:11.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc108740 con 0x7ff1bc0f6b20 2024-02-13T00:39:11.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c42a3700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc108900 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c2840700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0fcc10 0x7ff1bc1080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:11.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.763+0000 7ff1c183e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc0fc2b0 0x7ff1bc10e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:11.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c2840700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0fcc10 0x7ff1bc1080e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:37956/0 (socket says 172.21.15.162:37956) 2024-02-13T00:39:11.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c183e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc0fc2b0 0x7ff1bc10e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:40894/0 (socket says 172.21.15.162:40894) 2024-02-13T00:39:11.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c2840700 1 -- 172.21.15.162:0/1126313890 learned_addr learned my addr 172.21.15.162:0/1126313890 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:11.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c203f700 1 --2- 172.21.15.162:0/1126313890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc0f6b20 0x7ff1bc10ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:11.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c2840700 1 -- 172.21.15.162:0/1126313890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc0f6b20 msgr2=0x7ff1bc10ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:11.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c2840700 1 --2- 172.21.15.162:0/1126313890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc0f6b20 0x7ff1bc10ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:11.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c2840700 1 -- 172.21.15.162:0/1126313890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc0fc2b0 msgr2=0x7ff1bc10e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:11.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c2840700 1 --2- 172.21.15.162:0/1126313890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc0fc2b0 0x7ff1bc10e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:11.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c2840700 1 -- 172.21.15.162:0/1126313890 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac005040 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c183e700 1 --2- 172.21.15.162:0/1126313890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc0fc2b0 0x7ff1bc10e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:39:11.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c2840700 1 --2- 172.21.15.162:0/1126313890 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0fcc10 0x7ff1bc1080e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff1b4004a30 tx=0x7ff1b4004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:11.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1b3779700 10 client.?.objecter ms_handle_connect 0x7ff1bc0fcc10 2024-02-13T00:39:11.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1b3779700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:11.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1b3779700 1 -- 172.21.15.162:0/1126313890 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b400e040 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c42a3700 1 -- 172.21.15.162:0/1126313890 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b40097c0 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1b3779700 1 -- 172.21.15.162:0/1126313890 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1b4018040 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1b3779700 1 -- 172.21.15.162:0/1126313890 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b401c3f0 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c42a3700 10 client.24584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:11.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c42a3700 1 -- 172.21.15.162:0/1126313890 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc108e20 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.764+0000 7ff1c42a3700 1 librados: init done 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.765+0000 7ff1c42a3700 1 -- 172.21.15.162:0/1126313890 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc05fb80 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.765+0000 7ff1b3779700 1 -- 172.21.15.162:0/1126313890 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff1b4007b20 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.766+0000 7ff1b3779700 1 --2- 172.21.15.162:0/1126313890 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff1a8062c90 0x7ff1a8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.766+0000 7ff1b3779700 1 -- 172.21.15.162:0/1126313890 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(83..83 src has 30..83) v4 ==== 6172+0+0 (secure 0 0 0) 0x7ff1b407db30 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.766+0000 7ff1b3779700 10 client.24584.objecter ms_dispatch 0x7ff1bc001550 osd_map(83..83 src has 30..83) v4 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.766+0000 7ff1b3779700 3 client.24584.objecter handle_osd_map got epochs [83,83] > 0 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.766+0000 7ff1b3779700 3 client.24584.objecter handle_osd_map decoding full epoch 83 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.766+0000 7ff1c203f700 1 --2- 172.21.15.162:0/1126313890 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff1a8062c90 0x7ff1a8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.766+0000 7ff1b3779700 20 client.24584.objecter dump_active .. 0 homeless 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.766+0000 7ff1b3779700 10 client.24584.objecter ms_handle_connect 0x7ff1a8062c90 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.766+0000 7ff1c203f700 1 --2- 172.21.15.162:0/1126313890 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff1a8062c90 0x7ff1a8065150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff1ac004970 tx=0x7ff1ac00c040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:11.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.770+0000 7ff1b3779700 1 -- 172.21.15.162:0/1126313890 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff1b4051850 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.937+0000 7ff1c42a3700 1 -- 172.21.15.162:0/1126313890 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff1bc104f80 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.937+0000 7ff1b3779700 1 -- 172.21.15.162:0/1126313890 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff1b4021020 con 0x7ff1bc0fcc10 2024-02-13T00:39:11.942 INFO:teuthology.orchestra.run.smithi162.stdout:34359738423 2024-02-13T00:39:11.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 10 librados: watch_flush enter 2024-02-13T00:39:11.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 10 librados: watch_flush exit 2024-02-13T00:39:11.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 20 client.24584.objecter shutdown clearing up homeless session... 2024-02-13T00:39:11.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 10 client.24584.objecter successfully canceled tick 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 1 -- 172.21.15.162:0/1126313890 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff1a8062c90 msgr2=0x7ff1a8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 1 --2- 172.21.15.162:0/1126313890 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff1a8062c90 0x7ff1a8065150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff1ac004970 tx=0x7ff1ac00c040).stop 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 1 -- 172.21.15.162:0/1126313890 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0fcc10 msgr2=0x7ff1bc1080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 1 --2- 172.21.15.162:0/1126313890 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0fcc10 0x7ff1bc1080e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff1b4004a30 tx=0x7ff1b4004d10).stop 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 1 -- 172.21.15.162:0/1126313890 shutdown_connections 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 1 --2- 172.21.15.162:0/1126313890 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff1a8062c90 0x7ff1a8065150 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 1 --2- 172.21.15.162:0/1126313890 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1bc0fcc10 0x7ff1bc1080e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 1 --2- 172.21.15.162:0/1126313890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1bc0f6b20 0x7ff1bc10ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 1 --2- 172.21.15.162:0/1126313890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1bc0fc2b0 0x7ff1bc10e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.939+0000 7ff1c42a3700 1 -- 172.21.15.162:0/1126313890 >> 172.21.15.162:0/1126313890 conn(0x7ff1bc070080 msgr2=0x7ff1bc06d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.940+0000 7ff1c42a3700 1 -- 172.21.15.162:0/1126313890 shutdown_connections 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.940+0000 7ff1c42a3700 1 -- 172.21.15.162:0/1126313890 wait complete. 2024-02-13T00:39:11.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:11.940+0000 7ff1c42a3700 1 librados: shutdown 2024-02-13T00:39:12.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:11.516558+0000 osd.5 (osd.5) 37 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:39:12.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:11.518354+0000 osd.5 (osd.5) 38 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:39:12.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:11.939620+0000 mon.c (mon.2) 61 : audit [DBG] from='client.? 172.21.15.162:0/1126313890' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-13T00:39:12.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:11.516558+0000 osd.5 (osd.5) 37 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:39:12.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:11.518354+0000 osd.5 (osd.5) 38 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:39:12.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:11.939620+0000 mon.c (mon.2) 61 : audit [DBG] from='client.? 172.21.15.162:0/1126313890' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-13T00:39:12.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:11.516558+0000 osd.5 (osd.5) 37 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:39:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:11.518354+0000 osd.5 (osd.5) 38 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:39:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:11.939620+0000 mon.c (mon.2) 61 : audit [DBG] from='client.? 172.21.15.162:0/1126313890' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-13T00:39:13.161 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738423 for osd.0 2024-02-13T00:39:13.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-13T00:39:13.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:12.109408+0000 osd.6 (osd.6) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:39:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:12.109455+0000 osd.6 (osd.6) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:39:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:12.173812+0000 mgr.y (mgr.14544) 58 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 15 B/s, 2 objects/s recovering 2024-02-13T00:39:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:12.240420+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-13T00:39:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:12.342551+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.12 deep-scrub starts 2024-02-13T00:39:13.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:12.348475+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.12 deep-scrub ok 2024-02-13T00:39:13.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:12.109408+0000 osd.6 (osd.6) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:39:13.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:12.109455+0000 osd.6 (osd.6) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:39:13.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:12.173812+0000 mgr.y (mgr.14544) 58 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 15 B/s, 2 objects/s recovering 2024-02-13T00:39:13.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:12.240420+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-13T00:39:13.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:12.342551+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.12 deep-scrub starts 2024-02-13T00:39:13.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:12.348475+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.12 deep-scrub ok 2024-02-13T00:39:13.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:12.109408+0000 osd.6 (osd.6) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:39:13.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:12.109455+0000 osd.6 (osd.6) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:39:13.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:12.173812+0000 mgr.y (mgr.14544) 58 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 15 B/s, 2 objects/s recovering 2024-02-13T00:39:13.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:12.240420+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-13T00:39:13.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:12.342551+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.12 deep-scrub starts 2024-02-13T00:39:13.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:12.348475+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.12 deep-scrub ok 2024-02-13T00:39:14.444 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:13.025347+0000 osd.7 (osd.7) 23 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:39:14.445 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:13.027295+0000 osd.7 (osd.7) 24 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:39:14.445 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:13.025347+0000 osd.7 (osd.7) 23 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:39:14.445 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:13.027295+0000 osd.7 (osd.7) 24 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:39:14.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:13.025347+0000 osd.7 (osd.7) 23 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:39:14.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:13.027295+0000 osd.7 (osd.7) 24 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:39:14.715 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:15.518 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:15 smithi162 systemd[1]: Stopping Ceph alertmanager.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:39:15.518 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:14.174155+0000 mgr.y (mgr.14544) 59 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s; 13 B/s, 2 objects/s recovering 2024-02-13T00:39:15.518 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:14.366060+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:39:15.519 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:14.367109+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:39:15.519 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:14.608416+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.d scrub starts 2024-02-13T00:39:15.519 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:14.609839+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.d scrub ok 2024-02-13T00:39:15.519 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:14.174155+0000 mgr.y (mgr.14544) 59 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s; 13 B/s, 2 objects/s recovering 2024-02-13T00:39:15.519 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:14.366060+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:39:15.519 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:14.367109+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:39:15.519 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:14.608416+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.d scrub starts 2024-02-13T00:39:15.519 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:14.609839+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.d scrub ok 2024-02-13T00:39:15.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:39:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:39:15.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:14.174155+0000 mgr.y (mgr.14544) 59 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s; 13 B/s, 2 objects/s recovering 2024-02-13T00:39:15.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:14.366060+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:39:15.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:14.367109+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:39:15.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:14.608416+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.d scrub starts 2024-02-13T00:39:15.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:14.609839+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.d scrub ok 2024-02-13T00:39:15.829 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:15 smithi162 bash[73695]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-13T00:39:16.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:15.076831+0000 osd.7 (osd.7) 25 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:39:16.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:15.080310+0000 osd.7 (osd.7) 26 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:39:16.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:15.225507+0000 mgr.y (mgr.14544) 60 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:16.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:15.324950+0000 osd.0 (osd.0) 25 : cluster [DBG] 5.1e scrub starts 2024-02-13T00:39:16.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:15.327967+0000 osd.0 (osd.0) 26 : cluster [DBG] 5.1e scrub ok 2024-02-13T00:39:16.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:15.411452+0000 osd.6 (osd.6) 25 : cluster [DBG] 4.5 scrub starts 2024-02-13T00:39:16.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:15.414997+0000 osd.6 (osd.6) 26 : cluster [DBG] 4.5 scrub ok 2024-02-13T00:39:16.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:15.602712+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.a scrub starts 2024-02-13T00:39:16.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:15.603805+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.a scrub ok 2024-02-13T00:39:16.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:39:16] "GET /metrics HTTP/1.1" 200 172171 "" "Prometheus/2.33.4" 2024-02-13T00:39:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:15.076831+0000 osd.7 (osd.7) 25 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:39:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:15.080310+0000 osd.7 (osd.7) 26 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:39:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:15.225507+0000 mgr.y (mgr.14544) 60 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:15.324950+0000 osd.0 (osd.0) 25 : cluster [DBG] 5.1e scrub starts 2024-02-13T00:39:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:15.327967+0000 osd.0 (osd.0) 26 : cluster [DBG] 5.1e scrub ok 2024-02-13T00:39:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:15.411452+0000 osd.6 (osd.6) 25 : cluster [DBG] 4.5 scrub starts 2024-02-13T00:39:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:15.414997+0000 osd.6 (osd.6) 26 : cluster [DBG] 4.5 scrub ok 2024-02-13T00:39:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:15.602712+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.a scrub starts 2024-02-13T00:39:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:15.603805+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.a scrub ok 2024-02-13T00:39:16.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:15.076831+0000 osd.7 (osd.7) 25 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:39:16.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:15.080310+0000 osd.7 (osd.7) 26 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:39:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:15.225507+0000 mgr.y (mgr.14544) 60 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:15.324950+0000 osd.0 (osd.0) 25 : cluster [DBG] 5.1e scrub starts 2024-02-13T00:39:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:15.327967+0000 osd.0 (osd.0) 26 : cluster [DBG] 5.1e scrub ok 2024-02-13T00:39:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:15.411452+0000 osd.6 (osd.6) 25 : cluster [DBG] 4.5 scrub starts 2024-02-13T00:39:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:15.414997+0000 osd.6 (osd.6) 26 : cluster [DBG] 4.5 scrub ok 2024-02-13T00:39:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:15.602712+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.a scrub starts 2024-02-13T00:39:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:15.603805+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.a scrub ok 2024-02-13T00:39:16.964 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[68995]: level=info ts=2024-02-13T00:39:16.707Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-13T00:39:17.258 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:16.085181+0000 osd.7 (osd.7) 27 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:39:17.258 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:16.086199+0000 osd.7 (osd.7) 28 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:39:17.258 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:16.174881+0000 mgr.y (mgr.14544) 61 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 6 op/s; 13 B/s, 2 objects/s recovering 2024-02-13T00:39:17.258 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:16.259515+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.d scrub starts 2024-02-13T00:39:17.258 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:16.262377+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.d scrub ok 2024-02-13T00:39:17.258 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:16.300614+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.0 scrub starts 2024-02-13T00:39:17.258 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:16.304313+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.0 scrub ok 2024-02-13T00:39:17.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:16.085181+0000 osd.7 (osd.7) 27 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:39:17.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:16.086199+0000 osd.7 (osd.7) 28 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:39:17.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:16.174881+0000 mgr.y (mgr.14544) 61 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 6 op/s; 13 B/s, 2 objects/s recovering 2024-02-13T00:39:17.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:16.259515+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.d scrub starts 2024-02-13T00:39:17.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:16.262377+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.d scrub ok 2024-02-13T00:39:17.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:16.300614+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.0 scrub starts 2024-02-13T00:39:17.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:16.304313+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.0 scrub ok 2024-02-13T00:39:17.579 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:17 smithi162 bash[73761]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a 2024-02-13T00:39:17.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:16.085181+0000 osd.7 (osd.7) 27 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:39:17.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:16.086199+0000 osd.7 (osd.7) 28 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:39:17.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:16.174881+0000 mgr.y (mgr.14544) 61 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 6 op/s; 13 B/s, 2 objects/s recovering 2024-02-13T00:39:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:16.259515+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.d scrub starts 2024-02-13T00:39:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:16.262377+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.d scrub ok 2024-02-13T00:39:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:16.300614+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.0 scrub starts 2024-02-13T00:39:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:16.304313+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.0 scrub ok 2024-02-13T00:39:17.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.823+0000 7f3eec639700 1 -- 172.21.15.162:0/1281782131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ee40f84a0 msgr2=0x7f3ee4104aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:17.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.823+0000 7f3eec639700 1 --2- 172.21.15.162:0/1281782131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ee40f84a0 0x7f3ee4104aa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f3edc009c30 tx=0x7f3edc004c50).stop 2024-02-13T00:39:17.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.823+0000 7f3eec639700 1 -- 172.21.15.162:0/1281782131 shutdown_connections 2024-02-13T00:39:17.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.823+0000 7f3eec639700 1 --2- 172.21.15.162:0/1281782131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ee40f84a0 0x7f3ee4104aa0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:17.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.823+0000 7f3eec639700 1 --2- 172.21.15.162:0/1281782131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee40fdc30 0x7f3ee40fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:17.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.823+0000 7f3eec639700 1 --2- 172.21.15.162:0/1281782131 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ee40fe590 0x7f3ee40f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:17.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.823+0000 7f3eec639700 1 -- 172.21.15.162:0/1281782131 >> 172.21.15.162:0/1281782131 conn(0x7f3ee40f3880 msgr2=0x7f3ee40f5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:17.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.823+0000 7f3eec639700 1 -- 172.21.15.162:0/1281782131 shutdown_connections 2024-02-13T00:39:17.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.823+0000 7f3eec639700 1 -- 172.21.15.162:0/1281782131 wait complete. 2024-02-13T00:39:17.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.823+0000 7f3eec639700 1 librados: starting msgr at 2024-02-13T00:39:17.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.823+0000 7f3eec639700 1 librados: starting objecter 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.823+0000 7f3eec639700 1 Processor -- start 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3eec639700 1 -- start start 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3eec639700 1 librados: setting wanted keys 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3eec639700 1 librados: calling monclient init 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3eec639700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ee40f84a0 0x7f3ee410df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3eec639700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ee40fdc30 0x7f3ee410e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3eec639700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee40fe590 0x7f3ee4108100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3eec639700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee41085f0 con 0x7f3ee40fdc30 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3eec639700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee4108760 con 0x7f3ee40fe590 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3eec639700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee4108920 con 0x7f3ee40f84a0 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3ee9bd4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ee40fdc30 0x7f3ee410e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3ee9bd4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ee40fdc30 0x7f3ee410e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:36848/0 (socket says 172.21.15.162:36848) 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3ee9bd4700 1 -- 172.21.15.162:0/1169217575 learned_addr learned my addr 172.21.15.162:0/1169217575 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3eea3d5700 1 --2- 172.21.15.162:0/1169217575 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ee40f84a0 0x7f3ee410df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3eeabd6700 1 --2- 172.21.15.162:0/1169217575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee40fe590 0x7f3ee4108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3ee9bd4700 1 -- 172.21.15.162:0/1169217575 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ee40f84a0 msgr2=0x7f3ee410df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3ee9bd4700 1 --2- 172.21.15.162:0/1169217575 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ee40f84a0 0x7f3ee410df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3ee9bd4700 1 -- 172.21.15.162:0/1169217575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee40fe590 msgr2=0x7f3ee4108100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3ee9bd4700 1 --2- 172.21.15.162:0/1169217575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee40fe590 0x7f3ee4108100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3ee9bd4700 1 -- 172.21.15.162:0/1169217575 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee0005040 con 0x7f3ee40fdc30 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3ee9bd4700 1 --2- 172.21.15.162:0/1169217575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ee40fdc30 0x7f3ee410e450 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3ee0009400 tx=0x7f3ee0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3edb7fe700 10 client.?.objecter ms_handle_connect 0x7f3ee40fdc30 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3edb7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.824+0000 7f3edb7fe700 1 -- 172.21.15.162:0/1169217575 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0007a40 con 0x7f3ee40fdc30 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.825+0000 7f3edb7fe700 1 -- 172.21.15.162:0/1169217575 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ee000f040 con 0x7f3ee40fdc30 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.825+0000 7f3eec639700 1 -- 172.21.15.162:0/1169217575 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc0097c0 con 0x7f3ee40fdc30 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.825+0000 7f3edb7fe700 1 -- 172.21.15.162:0/1169217575 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0014a70 con 0x7f3ee40fdc30 2024-02-13T00:39:17.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.825+0000 7f3eec639700 10 client.14691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:17.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.825+0000 7f3eec639700 1 -- 172.21.15.162:0/1169217575 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee4114d90 con 0x7f3ee40fdc30 2024-02-13T00:39:17.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.825+0000 7f3eec639700 1 librados: init done 2024-02-13T00:39:17.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.825+0000 7f3eec639700 1 -- 172.21.15.162:0/1169217575 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee405fae0 con 0x7f3ee40fdc30 2024-02-13T00:39:17.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.826+0000 7f3edb7fe700 1 -- 172.21.15.162:0/1169217575 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f3ee0010530 con 0x7f3ee40fdc30 2024-02-13T00:39:17.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.826+0000 7f3edb7fe700 1 --2- 172.21.15.162:0/1169217575 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ed0062c90 0x7f3ed0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:17.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.826+0000 7f3edb7fe700 1 -- 172.21.15.162:0/1169217575 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(84..84 src has 30..84) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f3ee007c560 con 0x7f3ee40fdc30 2024-02-13T00:39:17.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.826+0000 7f3edb7fe700 10 client.14691.objecter ms_dispatch 0x7f3ee4001550 osd_map(84..84 src has 30..84) v4 2024-02-13T00:39:17.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.826+0000 7f3edb7fe700 3 client.14691.objecter handle_osd_map got epochs [84,84] > 0 2024-02-13T00:39:17.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.826+0000 7f3edb7fe700 3 client.14691.objecter handle_osd_map decoding full epoch 84 2024-02-13T00:39:17.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.826+0000 7f3eea3d5700 1 --2- 172.21.15.162:0/1169217575 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ed0062c90 0x7f3ed0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:17.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.826+0000 7f3edb7fe700 20 client.14691.objecter dump_active .. 0 homeless 2024-02-13T00:39:17.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.827+0000 7f3edb7fe700 10 client.14691.objecter ms_handle_connect 0x7f3ed0062c90 2024-02-13T00:39:17.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.827+0000 7f3eea3d5700 1 --2- 172.21.15.162:0/1169217575 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ed0062c90 0x7f3ed0065150 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f3ed4004dc0 tx=0x7f3ed400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:17.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:17.831+0000 7f3edb7fe700 1 -- 172.21.15.162:0/1169217575 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3ee0049f60 con 0x7f3ee40fdc30 2024-02-13T00:39:18.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.032+0000 7f3eec639700 1 -- 172.21.15.162:0/1169217575 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3ee4107190 con 0x7f3ee40fdc30 2024-02-13T00:39:18.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.032+0000 7f3edb7fe700 1 -- 172.21.15.162:0/1169217575 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3ee0050270 con 0x7f3ee40fdc30 2024-02-13T00:39:18.036 INFO:teuthology.orchestra.run.smithi162.stdout:55834574900 2024-02-13T00:39:18.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 10 librados: watch_flush enter 2024-02-13T00:39:18.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 10 librados: watch_flush exit 2024-02-13T00:39:18.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 20 client.14691.objecter shutdown clearing up homeless session... 2024-02-13T00:39:18.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 10 client.14691.objecter successfully canceled tick 2024-02-13T00:39:18.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 -- 172.21.15.162:0/1169217575 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ed0062c90 msgr2=0x7f3ed0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:18.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 --2- 172.21.15.162:0/1169217575 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ed0062c90 0x7f3ed0065150 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f3ed4004dc0 tx=0x7f3ed400b040).stop 2024-02-13T00:39:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 -- 172.21.15.162:0/1169217575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ee40fdc30 msgr2=0x7f3ee410e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 --2- 172.21.15.162:0/1169217575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ee40fdc30 0x7f3ee410e450 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3ee0009400 tx=0x7f3ee0009cb0).stop 2024-02-13T00:39:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 -- 172.21.15.162:0/1169217575 shutdown_connections 2024-02-13T00:39:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 --2- 172.21.15.162:0/1169217575 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ed0062c90 0x7f3ed0065150 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 --2- 172.21.15.162:0/1169217575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee40fe590 0x7f3ee4108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 --2- 172.21.15.162:0/1169217575 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ee40f84a0 0x7f3ee410df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 --2- 172.21.15.162:0/1169217575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ee40fdc30 0x7f3ee410e450 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 -- 172.21.15.162:0/1169217575 >> 172.21.15.162:0/1169217575 conn(0x7f3ee40f3880 msgr2=0x7f3ee40f50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 -- 172.21.15.162:0/1169217575 shutdown_connections 2024-02-13T00:39:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 -- 172.21.15.162:0/1169217575 wait complete. 2024-02-13T00:39:18.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:18.034+0000 7f3eec639700 1 librados: shutdown 2024-02-13T00:39:18.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:17.280818+0000 osd.1 (osd.1) 25 : cluster [DBG] 2.a scrub starts 2024-02-13T00:39:18.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:17.283893+0000 osd.1 (osd.1) 26 : cluster [DBG] 2.a scrub ok 2024-02-13T00:39:18.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:17.630655+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.b deep-scrub starts 2024-02-13T00:39:18.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:17.631868+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.b deep-scrub ok 2024-02-13T00:39:18.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:17.710051+0000 osd.5 (osd.5) 39 : cluster [DBG] 6.a scrub starts 2024-02-13T00:39:18.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:17.712960+0000 osd.5 (osd.5) 40 : cluster [DBG] 6.a scrub ok 2024-02-13T00:39:18.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:18.034180+0000 mon.a (mon.0) 696 : audit [DBG] from='client.? 172.21.15.162:0/1169217575' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-13T00:39:18.332 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:18 smithi162 systemd[1]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@alertmanager.a.service: Succeeded. 2024-02-13T00:39:18.332 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:18 smithi162 systemd[1]: Stopped Ceph alertmanager.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:39:18.332 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:18 smithi162 systemd[1]: Starting Ceph alertmanager.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:39:18.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:17.280818+0000 osd.1 (osd.1) 25 : cluster [DBG] 2.a scrub starts 2024-02-13T00:39:18.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:17.283893+0000 osd.1 (osd.1) 26 : cluster [DBG] 2.a scrub ok 2024-02-13T00:39:18.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:17.630655+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.b deep-scrub starts 2024-02-13T00:39:18.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:17.631868+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.b deep-scrub ok 2024-02-13T00:39:18.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:17.710051+0000 osd.5 (osd.5) 39 : cluster [DBG] 6.a scrub starts 2024-02-13T00:39:18.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:17.712960+0000 osd.5 (osd.5) 40 : cluster [DBG] 6.a scrub ok 2024-02-13T00:39:18.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:18.034180+0000 mon.a (mon.0) 696 : audit [DBG] from='client.? 172.21.15.162:0/1169217575' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-13T00:39:18.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:17.280818+0000 osd.1 (osd.1) 25 : cluster [DBG] 2.a scrub starts 2024-02-13T00:39:18.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:17.283893+0000 osd.1 (osd.1) 26 : cluster [DBG] 2.a scrub ok 2024-02-13T00:39:18.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:17.630655+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.b deep-scrub starts 2024-02-13T00:39:18.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:17.631868+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.b deep-scrub ok 2024-02-13T00:39:18.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:17.710051+0000 osd.5 (osd.5) 39 : cluster [DBG] 6.a scrub starts 2024-02-13T00:39:18.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:17.712960+0000 osd.5 (osd.5) 40 : cluster [DBG] 6.a scrub ok 2024-02-13T00:39:18.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:18.034180+0000 mon.a (mon.0) 696 : audit [DBG] from='client.? 172.21.15.162:0/1169217575' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-13T00:39:19.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:39:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:39:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:39:19.394 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Bus STOPPING 2024-02-13T00:39:19.394 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T00:39:19.394 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Bus STOPPED 2024-02-13T00:39:19.394 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Bus STARTING 2024-02-13T00:39:19.394 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:18.175444+0000 mgr.y (mgr.14544) 62 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:39:19.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:18.308597+0000 osd.3 (osd.3) 25 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:39:19.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:18.311661+0000 osd.3 (osd.3) 26 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:39:19.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:18.671409+0000 osd.2 (osd.2) 25 : cluster [DBG] 2.c scrub starts 2024-02-13T00:39:19.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:18.672512+0000 osd.2 (osd.2) 26 : cluster [DBG] 2.c scrub ok 2024-02-13T00:39:19.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.238555+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:19.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.243544+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T00:39:19.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.244664+0000 mon.c (mon.2) 63 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi162.front.sepia.ceph.com:9093"}]: dispatch 2024-02-13T00:39:19.395 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:18.175444+0000 mgr.y (mgr.14544) 62 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:39:19.395 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:18.308597+0000 osd.3 (osd.3) 25 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:39:19.395 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:18.311661+0000 osd.3 (osd.3) 26 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:39:19.396 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:18.671409+0000 osd.2 (osd.2) 25 : cluster [DBG] 2.c scrub starts 2024-02-13T00:39:19.396 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:18.672512+0000 osd.2 (osd.2) 26 : cluster [DBG] 2.c scrub ok 2024-02-13T00:39:19.396 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.238555+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:19.396 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.243544+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T00:39:19.396 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.244664+0000 mon.c (mon.2) 63 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi162.front.sepia.ceph.com:9093"}]: dispatch 2024-02-13T00:39:19.396 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=info ts=2024-02-13T00:39:19.123Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-13T00:39:19.396 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=info ts=2024-02-13T00:39:19.123Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-13T00:39:19.396 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=info ts=2024-02-13T00:39:19.124Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.162 port=9094 2024-02-13T00:39:19.396 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=info ts=2024-02-13T00:39:19.125Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-13T00:39:19.396 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=info ts=2024-02-13T00:39:19.150Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T00:39:19.396 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=info ts=2024-02-13T00:39:19.150Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T00:39:19.397 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=info ts=2024-02-13T00:39:19.152Z caller=main.go:518 msg=Listening address=:9093 2024-02-13T00:39:19.397 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=info ts=2024-02-13T00:39:19.152Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-13T00:39:19.397 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:19 smithi162 bash[73985]: f8ac756ae8a12479500b43d7488f0ae91c7848a08d27fac33ca7e1e5b6f96458 2024-02-13T00:39:19.397 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:19 smithi162 systemd[1]: Started Ceph alertmanager.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:39:19.398 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574900 for osd.1 2024-02-13T00:39:19.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-13T00:39:19.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:18.175444+0000 mgr.y (mgr.14544) 62 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:39:19.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:18.308597+0000 osd.3 (osd.3) 25 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:39:19.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:18.311661+0000 osd.3 (osd.3) 26 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:39:19.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:18.671409+0000 osd.2 (osd.2) 25 : cluster [DBG] 2.c scrub starts 2024-02-13T00:39:19.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:18.672512+0000 osd.2 (osd.2) 26 : cluster [DBG] 2.c scrub ok 2024-02-13T00:39:19.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.238555+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:19.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.243544+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T00:39:19.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.244664+0000 mon.c (mon.2) 63 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi162.front.sepia.ceph.com:9093"}]: dispatch 2024-02-13T00:39:19.696 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:19.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Serving on http://:::9283 2024-02-13T00:39:19.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Bus STARTED 2024-02-13T00:39:19.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Bus STOPPING 2024-02-13T00:39:19.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T00:39:19.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Bus STOPPED 2024-02-13T00:39:19.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Bus STARTING 2024-02-13T00:39:19.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Serving on http://:::9283 2024-02-13T00:39:19.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Bus STARTED 2024-02-13T00:39:19.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Bus STOPPING 2024-02-13T00:39:19.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T00:39:19.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Bus STOPPED 2024-02-13T00:39:19.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Bus STARTING 2024-02-13T00:39:19.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Serving on http://:::9283 2024-02-13T00:39:19.831 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:19] ENGINE Bus STARTED 2024-02-13T00:39:20.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:18.358735+0000 osd.0 (osd.0) 27 : cluster [DBG] 6.9 deep-scrub starts 2024-02-13T00:39:20.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:18.359599+0000 osd.0 (osd.0) 28 : cluster [DBG] 6.9 deep-scrub ok 2024-02-13T00:39:20.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.243949+0000 mgr.y (mgr.14544) 63 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T00:39:20.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.244953+0000 mgr.y (mgr.14544) 64 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi162.front.sepia.ceph.com:9093"}]: dispatch 2024-02-13T00:39:20.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:19.265404+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:39:20.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:19.266723+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:39:20.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.277634+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:20.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.285561+0000 mon.c (mon.2) 64 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T00:39:20.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.286013+0000 mgr.y (mgr.14544) 65 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T00:39:20.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.287320+0000 mon.c (mon.2) 65 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-02-13T00:39:20.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.287669+0000 mgr.y (mgr.14544) 66 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-02-13T00:39:20.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.292284+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:20.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.297275+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T00:39:20.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.297680+0000 mgr.y (mgr.14544) 67 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T00:39:20.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.299391+0000 mon.c (mon.2) 67 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-02-13T00:39:20.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.299703+0000 mgr.y (mgr.14544) 68 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-02-13T00:39:20.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.304982+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:20.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:19.307156+0000 mon.c (mon.2) 68 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:39:20.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:18.358735+0000 osd.0 (osd.0) 27 : cluster [DBG] 6.9 deep-scrub starts 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:18.359599+0000 osd.0 (osd.0) 28 : cluster [DBG] 6.9 deep-scrub ok 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.243949+0000 mgr.y (mgr.14544) 63 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.244953+0000 mgr.y (mgr.14544) 64 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi162.front.sepia.ceph.com:9093"}]: dispatch 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:19.265404+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:19.266723+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.277634+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.285561+0000 mon.c (mon.2) 64 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.286013+0000 mgr.y (mgr.14544) 65 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.287320+0000 mon.c (mon.2) 65 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.287669+0000 mgr.y (mgr.14544) 66 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.292284+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.297275+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T00:39:20.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.297680+0000 mgr.y (mgr.14544) 67 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T00:39:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.299391+0000 mon.c (mon.2) 67 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-02-13T00:39:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.299703+0000 mgr.y (mgr.14544) 68 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-02-13T00:39:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.304982+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:20.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:19.307156+0000 mon.c (mon.2) 68 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:39:20.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:18.358735+0000 osd.0 (osd.0) 27 : cluster [DBG] 6.9 deep-scrub starts 2024-02-13T00:39:20.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:18.359599+0000 osd.0 (osd.0) 28 : cluster [DBG] 6.9 deep-scrub ok 2024-02-13T00:39:20.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.243949+0000 mgr.y (mgr.14544) 63 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-13T00:39:20.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.244953+0000 mgr.y (mgr.14544) 64 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi162.front.sepia.ceph.com:9093"}]: dispatch 2024-02-13T00:39:20.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:19.265404+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:39:20.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:19.266723+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:39:20.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.277634+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:20.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.285561+0000 mon.c (mon.2) 64 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T00:39:20.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.286013+0000 mgr.y (mgr.14544) 65 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-13T00:39:20.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.287320+0000 mon.c (mon.2) 65 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-02-13T00:39:20.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.287669+0000 mgr.y (mgr.14544) 66 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-02-13T00:39:20.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.292284+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:20.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.297275+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T00:39:20.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.297680+0000 mgr.y (mgr.14544) 67 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-13T00:39:20.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.299391+0000 mon.c (mon.2) 67 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-02-13T00:39:20.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.299703+0000 mgr.y (mgr.14544) 68 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-02-13T00:39:20.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.304982+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:20.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:19.307156+0000 mon.c (mon.2) 68 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:39:21.479 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=info ts=2024-02-13T00:39:21.125Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000119686s 2024-02-13T00:39:21.479 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:20.176021+0000 mgr.y (mgr.14544) 69 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:39:21.479 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:20.448134+0000 osd.6 (osd.6) 27 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:39:21.479 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:20.451084+0000 osd.6 (osd.6) 28 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:39:21.479 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:20.700813+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:39:21.480 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:20.703696+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:39:21.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:20.176021+0000 mgr.y (mgr.14544) 69 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:39:21.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:20.448134+0000 osd.6 (osd.6) 27 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:39:21.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:20.451084+0000 osd.6 (osd.6) 28 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:39:21.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:20.700813+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:39:21.480 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:20.703696+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:39:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:20.176021+0000 mgr.y (mgr.14544) 69 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:39:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:20.448134+0000 osd.6 (osd.6) 27 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:39:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:20.451084+0000 osd.6 (osd.6) 28 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:39:21.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:20.700813+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:39:21.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:20.703696+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:39:22.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.021+0000 7fb3fcb33700 1 -- 172.21.15.162:0/1833862052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3f80fd490 msgr2=0x7fb3f80f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:22.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.021+0000 7fb3fcb33700 1 --2- 172.21.15.162:0/1833862052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3f80fd490 0x7fb3f80f6cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fb3ec004870 tx=0x7fb3ec009cb0).stop 2024-02-13T00:39:22.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 -- 172.21.15.162:0/1833862052 shutdown_connections 2024-02-13T00:39:22.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 --2- 172.21.15.162:0/1833862052 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3f80f73a0 0x7fb3f81039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:22.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 --2- 172.21.15.162:0/1833862052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f80fcb30 0x7fb3f80fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:22.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 --2- 172.21.15.162:0/1833862052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3f80fd490 0x7fb3f80f6cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 -- 172.21.15.162:0/1833862052 >> 172.21.15.162:0/1833862052 conn(0x7fb3f806eb90 msgr2=0x7fb3f80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 -- 172.21.15.162:0/1833862052 shutdown_connections 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 -- 172.21.15.162:0/1833862052 wait complete. 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 librados: starting msgr at 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 librados: starting objecter 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 Processor -- start 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 -- start start 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 librados: setting wanted keys 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.022+0000 7fb3fcb33700 1 librados: calling monclient init 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3fcb33700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3f80f73a0 0x7fb3f810d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3fcb33700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f80fcb30 0x7fb3f810d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3fcb33700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3f80fd490 0x7fb3f8107220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3fcb33700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8107710 con 0x7fb3f80fd490 2024-02-13T00:39:22.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3fcb33700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8107880 con 0x7fb3f80fcb30 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3fcb33700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8107a40 con 0x7fb3f80f73a0 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3f659c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3f80f73a0 0x7fb3f810d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3f659c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3f80f73a0 0x7fb3f810d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:50348/0 (socket says 172.21.15.162:50348) 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3f659c700 1 -- 172.21.15.162:0/180883131 learned_addr learned my addr 172.21.15.162:0/180883131 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3f6d9d700 1 --2- 172.21.15.162:0/180883131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3f80fd490 0x7fb3f8107220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3f5d9b700 1 --2- 172.21.15.162:0/180883131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f80fcb30 0x7fb3f810d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3f659c700 1 -- 172.21.15.162:0/180883131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f80fcb30 msgr2=0x7fb3f810d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3f659c700 1 --2- 172.21.15.162:0/180883131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f80fcb30 0x7fb3f810d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3f659c700 1 -- 172.21.15.162:0/180883131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3f80fd490 msgr2=0x7fb3f8107220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3f659c700 1 --2- 172.21.15.162:0/180883131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3f80fd490 0x7fb3f8107220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3f659c700 1 -- 172.21.15.162:0/180883131 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec005040 con 0x7fb3f80f73a0 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3f5d9b700 1 --2- 172.21.15.162:0/180883131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f80fcb30 0x7fb3f810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3f659c700 1 --2- 172.21.15.162:0/180883131 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3f80f73a0 0x7fb3f810d080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb3e8010b80 tx=0x7fb3e800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:22.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3e77fe700 10 client.?.objecter ms_handle_connect 0x7fb3f80f73a0 2024-02-13T00:39:22.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.023+0000 7fb3e77fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:22.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.024+0000 7fb3e77fe700 1 -- 172.21.15.162:0/180883131 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e8004660 con 0x7fb3f80f73a0 2024-02-13T00:39:22.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.024+0000 7fb3e77fe700 1 -- 172.21.15.162:0/180883131 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3e800ed60 con 0x7fb3f80f73a0 2024-02-13T00:39:22.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.024+0000 7fb3fcb33700 1 -- 172.21.15.162:0/180883131 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f8107cc0 con 0x7fb3f80f73a0 2024-02-13T00:39:22.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.024+0000 7fb3e77fe700 1 -- 172.21.15.162:0/180883131 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e80095c0 con 0x7fb3f80f73a0 2024-02-13T00:39:22.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.024+0000 7fb3fcb33700 10 client.24596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:22.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.024+0000 7fb3fcb33700 1 -- 172.21.15.162:0/180883131 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f8060ee0 con 0x7fb3f80f73a0 2024-02-13T00:39:22.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.024+0000 7fb3fcb33700 1 librados: init done 2024-02-13T00:39:22.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.025+0000 7fb3fcb33700 1 -- 172.21.15.162:0/180883131 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f8106070 con 0x7fb3f80f73a0 2024-02-13T00:39:22.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.026+0000 7fb3e77fe700 1 -- 172.21.15.162:0/180883131 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fb3e800e890 con 0x7fb3f80f73a0 2024-02-13T00:39:22.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.026+0000 7fb3e77fe700 1 --2- 172.21.15.162:0/180883131 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb3e0062c90 0x7fb3e0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:22.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.026+0000 7fb3e77fe700 1 -- 172.21.15.162:0/180883131 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(84..84 src has 30..84) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fb3e807f250 con 0x7fb3f80f73a0 2024-02-13T00:39:22.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.026+0000 7fb3f5d9b700 1 --2- 172.21.15.162:0/180883131 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb3e0062c90 0x7fb3e0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:22.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.026+0000 7fb3e77fe700 10 client.24596.objecter ms_dispatch 0x7fb3f8001550 osd_map(84..84 src has 30..84) v4 2024-02-13T00:39:22.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.026+0000 7fb3e77fe700 3 client.24596.objecter handle_osd_map got epochs [84,84] > 0 2024-02-13T00:39:22.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.026+0000 7fb3e77fe700 3 client.24596.objecter handle_osd_map decoding full epoch 84 2024-02-13T00:39:22.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.026+0000 7fb3e77fe700 20 client.24596.objecter dump_active .. 0 homeless 2024-02-13T00:39:22.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.026+0000 7fb3f5d9b700 1 --2- 172.21.15.162:0/180883131 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb3e0062c90 0x7fb3e0065150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb3ec0043d0 tx=0x7fb3ec009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:22.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.026+0000 7fb3e77fe700 10 client.24596.objecter ms_handle_connect 0x7fb3e0062c90 2024-02-13T00:39:22.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.030+0000 7fb3e77fe700 1 -- 172.21.15.162:0/180883131 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb3e8052f60 con 0x7fb3f80f73a0 2024-02-13T00:39:22.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.213+0000 7fb3fcb33700 1 -- 172.21.15.162:0/180883131 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb3f8108700 con 0x7fb3f80f73a0 2024-02-13T00:39:22.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.214+0000 7fb3e77fe700 1 -- 172.21.15.162:0/180883131 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb3e8020090 con 0x7fb3f80f73a0 2024-02-13T00:39:22.218 INFO:teuthology.orchestra.run.smithi162.stdout:77309411376 2024-02-13T00:39:22.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 10 librados: watch_flush enter 2024-02-13T00:39:22.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 10 librados: watch_flush exit 2024-02-13T00:39:22.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 20 client.24596.objecter shutdown clearing up homeless session... 2024-02-13T00:39:22.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 10 client.24596.objecter successfully canceled tick 2024-02-13T00:39:22.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 -- 172.21.15.162:0/180883131 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb3e0062c90 msgr2=0x7fb3e0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:22.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 --2- 172.21.15.162:0/180883131 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb3e0062c90 0x7fb3e0065150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb3ec0043d0 tx=0x7fb3ec009cb0).stop 2024-02-13T00:39:22.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 -- 172.21.15.162:0/180883131 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3f80f73a0 msgr2=0x7fb3f810d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:22.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 --2- 172.21.15.162:0/180883131 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3f80f73a0 0x7fb3f810d080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb3e8010b80 tx=0x7fb3e800b040).stop 2024-02-13T00:39:22.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 -- 172.21.15.162:0/180883131 shutdown_connections 2024-02-13T00:39:22.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 --2- 172.21.15.162:0/180883131 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb3e0062c90 0x7fb3e0065150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:22.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 --2- 172.21.15.162:0/180883131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3f80fd490 0x7fb3f8107220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:22.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 --2- 172.21.15.162:0/180883131 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3f80f73a0 0x7fb3f810d080 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:22.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 --2- 172.21.15.162:0/180883131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3f80fcb30 0x7fb3f810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:22.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 -- 172.21.15.162:0/180883131 >> 172.21.15.162:0/180883131 conn(0x7fb3f806eb90 msgr2=0x7fb3f80fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:22.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 -- 172.21.15.162:0/180883131 shutdown_connections 2024-02-13T00:39:22.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 -- 172.21.15.162:0/180883131 wait complete. 2024-02-13T00:39:22.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:22.216+0000 7fb3fcb33700 1 librados: shutdown 2024-02-13T00:39:22.505 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:21.164479+0000 osd.7 (osd.7) 29 : cluster [DBG] 2.1d deep-scrub starts 2024-02-13T00:39:22.505 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:21.165241+0000 osd.7 (osd.7) 30 : cluster [DBG] 2.1d deep-scrub ok 2024-02-13T00:39:22.506 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:21.240624+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.13 deep-scrub starts 2024-02-13T00:39:22.506 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:21.243523+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.13 deep-scrub ok 2024-02-13T00:39:22.506 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:21.298339+0000 osd.4 (osd.4) 17 : cluster [DBG] 5.1c deep-scrub starts 2024-02-13T00:39:22.506 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:21.301279+0000 osd.4 (osd.4) 18 : cluster [DBG] 5.1c deep-scrub ok 2024-02-13T00:39:22.506 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:21.328371+0000 osd.1 (osd.1) 27 : cluster [DBG] 2.6 scrub starts 2024-02-13T00:39:22.506 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:21.329170+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.6 scrub ok 2024-02-13T00:39:22.506 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:21.341867+0000 osd.0 (osd.0) 29 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:39:22.506 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:21.651828+0000 osd.2 (osd.2) 29 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:39:22.506 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:21.654726+0000 osd.2 (osd.2) 30 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:39:22.506 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:22.215135+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? 172.21.15.162:0/180883131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-13T00:39:22.506 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:21.164479+0000 osd.7 (osd.7) 29 : cluster [DBG] 2.1d deep-scrub starts 2024-02-13T00:39:22.506 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:21.165241+0000 osd.7 (osd.7) 30 : cluster [DBG] 2.1d deep-scrub ok 2024-02-13T00:39:22.507 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:21.240624+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.13 deep-scrub starts 2024-02-13T00:39:22.507 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:21.243523+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.13 deep-scrub ok 2024-02-13T00:39:22.507 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:21.298339+0000 osd.4 (osd.4) 17 : cluster [DBG] 5.1c deep-scrub starts 2024-02-13T00:39:22.507 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:21.301279+0000 osd.4 (osd.4) 18 : cluster [DBG] 5.1c deep-scrub ok 2024-02-13T00:39:22.507 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:21.328371+0000 osd.1 (osd.1) 27 : cluster [DBG] 2.6 scrub starts 2024-02-13T00:39:22.507 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:21.329170+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.6 scrub ok 2024-02-13T00:39:22.507 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:21.341867+0000 osd.0 (osd.0) 29 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:39:22.507 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:21.651828+0000 osd.2 (osd.2) 29 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:39:22.507 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:21.654726+0000 osd.2 (osd.2) 30 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:39:22.507 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:22.215135+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? 172.21.15.162:0/180883131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-13T00:39:22.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:21.164479+0000 osd.7 (osd.7) 29 : cluster [DBG] 2.1d deep-scrub starts 2024-02-13T00:39:22.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:21.165241+0000 osd.7 (osd.7) 30 : cluster [DBG] 2.1d deep-scrub ok 2024-02-13T00:39:22.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:21.240624+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.13 deep-scrub starts 2024-02-13T00:39:22.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:21.243523+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.13 deep-scrub ok 2024-02-13T00:39:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:21.298339+0000 osd.4 (osd.4) 17 : cluster [DBG] 5.1c deep-scrub starts 2024-02-13T00:39:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:21.301279+0000 osd.4 (osd.4) 18 : cluster [DBG] 5.1c deep-scrub ok 2024-02-13T00:39:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:21.328371+0000 osd.1 (osd.1) 27 : cluster [DBG] 2.6 scrub starts 2024-02-13T00:39:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:21.329170+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.6 scrub ok 2024-02-13T00:39:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:21.341867+0000 osd.0 (osd.0) 29 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:39:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:21.651828+0000 osd.2 (osd.2) 29 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:39:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:21.654726+0000 osd.2 (osd.2) 30 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:39:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:22.215135+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? 172.21.15.162:0/180883131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-13T00:39:23.256 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411370 got 77309411376 for osd.2 2024-02-13T00:39:23.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-13T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:21.344758+0000 osd.0 (osd.0) 30 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:22.176354+0000 mgr.y (mgr.14544) 70 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:22.264999+0000 osd.3 (osd.3) 31 : cluster [DBG] 5.14 deep-scrub starts 2024-02-13T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:22.269094+0000 osd.3 (osd.3) 32 : cluster [DBG] 5.14 deep-scrub ok 2024-02-13T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:22.337144+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.18 deep-scrub starts 2024-02-13T00:39:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:22.338034+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.18 deep-scrub ok 2024-02-13T00:39:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:22.614723+0000 osd.2 (osd.2) 31 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:39:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:22.615758+0000 osd.2 (osd.2) 32 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:39:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:22.622471+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:39:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:22.622506+0000 osd.1 (osd.1) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:39:23.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:21.344758+0000 osd.0 (osd.0) 30 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:39:23.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:22.176354+0000 mgr.y (mgr.14544) 70 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:22.264999+0000 osd.3 (osd.3) 31 : cluster [DBG] 5.14 deep-scrub starts 2024-02-13T00:39:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:22.269094+0000 osd.3 (osd.3) 32 : cluster [DBG] 5.14 deep-scrub ok 2024-02-13T00:39:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:22.337144+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.18 deep-scrub starts 2024-02-13T00:39:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:22.338034+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.18 deep-scrub ok 2024-02-13T00:39:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:22.614723+0000 osd.2 (osd.2) 31 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:39:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:22.615758+0000 osd.2 (osd.2) 32 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:39:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:22.622471+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:39:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:22.622506+0000 osd.1 (osd.1) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:39:23.649 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:23.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:21.344758+0000 osd.0 (osd.0) 30 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:39:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:22.176354+0000 mgr.y (mgr.14544) 70 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:22.264999+0000 osd.3 (osd.3) 31 : cluster [DBG] 5.14 deep-scrub starts 2024-02-13T00:39:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:22.269094+0000 osd.3 (osd.3) 32 : cluster [DBG] 5.14 deep-scrub ok 2024-02-13T00:39:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:22.337144+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.18 deep-scrub starts 2024-02-13T00:39:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:22.338034+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.18 deep-scrub ok 2024-02-13T00:39:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:22.614723+0000 osd.2 (osd.2) 31 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:39:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:22.615758+0000 osd.2 (osd.2) 32 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:39:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:22.622471+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:39:23.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:22.622506+0000 osd.1 (osd.1) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:39:24.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:23.227804+0000 osd.3 (osd.3) 33 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:39:24.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:23.231390+0000 osd.3 (osd.3) 34 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:39:24.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:23.270435+0000 osd.4 (osd.4) 19 : cluster [DBG] 5.18 deep-scrub starts 2024-02-13T00:39:24.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:23.273342+0000 osd.4 (osd.4) 20 : cluster [DBG] 5.18 deep-scrub ok 2024-02-13T00:39:24.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:23.320076+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-13T00:39:24.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:23.369145+0000 osd.1 (osd.1) 31 : cluster [DBG] 4.f scrub starts 2024-02-13T00:39:24.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:23.370443+0000 osd.1 (osd.1) 32 : cluster [DBG] 4.f scrub ok 2024-02-13T00:39:24.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:24.008383+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:24.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:23.227804+0000 osd.3 (osd.3) 33 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:39:24.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:23.231390+0000 osd.3 (osd.3) 34 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:39:24.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:23.270435+0000 osd.4 (osd.4) 19 : cluster [DBG] 5.18 deep-scrub starts 2024-02-13T00:39:24.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:23.273342+0000 osd.4 (osd.4) 20 : cluster [DBG] 5.18 deep-scrub ok 2024-02-13T00:39:24.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:23.320076+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-13T00:39:24.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:23.369145+0000 osd.1 (osd.1) 31 : cluster [DBG] 4.f scrub starts 2024-02-13T00:39:24.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:23.370443+0000 osd.1 (osd.1) 32 : cluster [DBG] 4.f scrub ok 2024-02-13T00:39:24.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:24.008383+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:24.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:23.227804+0000 osd.3 (osd.3) 33 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:39:24.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:23.231390+0000 osd.3 (osd.3) 34 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:39:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:23.270435+0000 osd.4 (osd.4) 19 : cluster [DBG] 5.18 deep-scrub starts 2024-02-13T00:39:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:23.273342+0000 osd.4 (osd.4) 20 : cluster [DBG] 5.18 deep-scrub ok 2024-02-13T00:39:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:23.320076+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-13T00:39:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:23.369145+0000 osd.1 (osd.1) 31 : cluster [DBG] 4.f scrub starts 2024-02-13T00:39:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:23.370443+0000 osd.1 (osd.1) 32 : cluster [DBG] 4.f scrub ok 2024-02-13T00:39:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:24.008383+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:25.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:39:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:39:25.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:24.134830+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.b scrub starts 2024-02-13T00:39:25.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:24.138624+0000 osd.7 (osd.7) 32 : cluster [DBG] 2.b scrub ok 2024-02-13T00:39:25.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:24.176910+0000 mgr.y (mgr.14544) 71 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:25.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:24.329086+0000 osd.1 (osd.1) 33 : cluster [DBG] 5.19 scrub starts 2024-02-13T00:39:25.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:24.332128+0000 osd.1 (osd.1) 34 : cluster [DBG] 5.19 scrub ok 2024-02-13T00:39:25.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:25.008497+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:25.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:25.011611+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:39:25.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:25.012356+0000 mon.c (mon.2) 71 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:39:25.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:25.016658+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:25.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:25.021422+0000 mon.c (mon.2) 72 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:39:25.555 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:24.134830+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.b scrub starts 2024-02-13T00:39:25.555 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:24.138624+0000 osd.7 (osd.7) 32 : cluster [DBG] 2.b scrub ok 2024-02-13T00:39:25.555 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:24.176910+0000 mgr.y (mgr.14544) 71 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:25.556 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:24.329086+0000 osd.1 (osd.1) 33 : cluster [DBG] 5.19 scrub starts 2024-02-13T00:39:25.556 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:24.332128+0000 osd.1 (osd.1) 34 : cluster [DBG] 5.19 scrub ok 2024-02-13T00:39:25.556 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:25.008497+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:25.556 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:25.011611+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:39:25.556 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:25.012356+0000 mon.c (mon.2) 71 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:39:25.556 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:25.016658+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:25.556 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:25.021422+0000 mon.c (mon.2) 72 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:39:25.556 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:24.134830+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.b scrub starts 2024-02-13T00:39:25.556 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:24.138624+0000 osd.7 (osd.7) 32 : cluster [DBG] 2.b scrub ok 2024-02-13T00:39:25.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:24.176910+0000 mgr.y (mgr.14544) 71 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:25.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:24.329086+0000 osd.1 (osd.1) 33 : cluster [DBG] 5.19 scrub starts 2024-02-13T00:39:25.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:24.332128+0000 osd.1 (osd.1) 34 : cluster [DBG] 5.19 scrub ok 2024-02-13T00:39:25.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:25.008497+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:25.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:25.011611+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:39:25.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:25.012356+0000 mon.c (mon.2) 71 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:39:25.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:25.016658+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:25.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:25.021422+0000 mon.c (mon.2) 72 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:39:26.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.090+0000 7fb72c55c700 1 -- 172.21.15.162:0/2515911076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7240f8d10 msgr2=0x7fb7240f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:26.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.090+0000 7fb72c55c700 1 --2- 172.21.15.162:0/2515911076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7240f8d10 0x7fb7240f90f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb720004870 tx=0x7fb720009cb0).stop 2024-02-13T00:39:26.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.091+0000 7fb72c55c700 1 -- 172.21.15.162:0/2515911076 shutdown_connections 2024-02-13T00:39:26.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.091+0000 7fb72c55c700 1 --2- 172.21.15.162:0/2515911076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb724102270 0x7fb724104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:26.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.091+0000 7fb72c55c700 1 --2- 172.21.15.162:0/2515911076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7240f8d10 0x7fb7240f90f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:26.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.091+0000 7fb72c55c700 1 --2- 172.21.15.162:0/2515911076 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7240f95e0 0x7fb724101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:26.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.091+0000 7fb72c55c700 1 -- 172.21.15.162:0/2515911076 >> 172.21.15.162:0/2515911076 conn(0x7fb72406eb90 msgr2=0x7fb7240f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:26.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.091+0000 7fb72c55c700 1 -- 172.21.15.162:0/2515911076 shutdown_connections 2024-02-13T00:39:26.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.091+0000 7fb72c55c700 1 -- 172.21.15.162:0/2515911076 wait complete. 2024-02-13T00:39:26.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.091+0000 7fb72c55c700 1 librados: starting msgr at 2024-02-13T00:39:26.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.091+0000 7fb72c55c700 1 librados: starting objecter 2024-02-13T00:39:26.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.091+0000 7fb72c55c700 1 Processor -- start 2024-02-13T00:39:26.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72c55c700 1 -- start start 2024-02-13T00:39:26.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72c55c700 1 librados: setting wanted keys 2024-02-13T00:39:26.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72c55c700 1 librados: calling monclient init 2024-02-13T00:39:26.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72c55c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7240f8d10 0x7fb72410e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:26.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72c55c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7240f95e0 0x7fb72410e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:26.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72c55c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb724102270 0x7fb7241083d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72c55c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7241088c0 con 0x7fb7240f95e0 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72c55c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb724108a30 con 0x7fb724102270 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72c55c700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb724108bf0 con 0x7fb7240f8d10 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72a2f8700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7240f8d10 0x7fb72410e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72a2f8700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7240f8d10 0x7fb72410e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:56156/0 (socket says 172.21.15.162:56156) 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb729af7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7240f95e0 0x7fb72410e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72a2f8700 1 -- 172.21.15.162:0/4044515274 learned_addr learned my addr 172.21.15.162:0/4044515274 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72aaf9700 1 --2- 172.21.15.162:0/4044515274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb724102270 0x7fb7241083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb729af7700 1 -- 172.21.15.162:0/4044515274 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7240f8d10 msgr2=0x7fb72410e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb729af7700 1 --2- 172.21.15.162:0/4044515274 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7240f8d10 0x7fb72410e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb729af7700 1 -- 172.21.15.162:0/4044515274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb724102270 msgr2=0x7fb7241083d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb729af7700 1 --2- 172.21.15.162:0/4044515274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb724102270 0x7fb7241083d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb729af7700 1 -- 172.21.15.162:0/4044515274 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb714005040 con 0x7fb7240f95e0 2024-02-13T00:39:26.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72a2f8700 1 --2- 172.21.15.162:0/4044515274 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7240f8d10 0x7fb72410e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:39:26.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.092+0000 7fb72aaf9700 1 --2- 172.21.15.162:0/4044515274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb724102270 0x7fb7241083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:39:26.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.093+0000 7fb729af7700 1 --2- 172.21.15.162:0/4044515274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7240f95e0 0x7fb72410e720 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fb714004920 tx=0x7fb714009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:26.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.093+0000 7fb71b7fe700 10 client.?.objecter ms_handle_connect 0x7fb7240f95e0 2024-02-13T00:39:26.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.093+0000 7fb71b7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:26.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.093+0000 7fb71b7fe700 1 -- 172.21.15.162:0/4044515274 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71400e030 con 0x7fb7240f95e0 2024-02-13T00:39:26.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.093+0000 7fb72c55c700 1 -- 172.21.15.162:0/4044515274 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb720005040 con 0x7fb7240f95e0 2024-02-13T00:39:26.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.093+0000 7fb72c55c700 10 client.14703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:26.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.093+0000 7fb72c55c700 1 -- 172.21.15.162:0/4044515274 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb724115060 con 0x7fb7240f95e0 2024-02-13T00:39:26.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.093+0000 7fb72c55c700 1 librados: init done 2024-02-13T00:39:26.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.093+0000 7fb71b7fe700 1 -- 172.21.15.162:0/4044515274 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb71400f040 con 0x7fb7240f95e0 2024-02-13T00:39:26.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.093+0000 7fb71b7fe700 1 -- 172.21.15.162:0/4044515274 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb714014850 con 0x7fb7240f95e0 2024-02-13T00:39:26.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.094+0000 7fb72c55c700 1 -- 172.21.15.162:0/4044515274 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb708004f50 con 0x7fb7240f95e0 2024-02-13T00:39:26.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.094+0000 7fb71b7fe700 1 -- 172.21.15.162:0/4044515274 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fb714014c80 con 0x7fb7240f95e0 2024-02-13T00:39:26.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.095+0000 7fb71b7fe700 1 --2- 172.21.15.162:0/4044515274 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb710062c90 0x7fb710065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:26.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.095+0000 7fb71b7fe700 1 -- 172.21.15.162:0/4044515274 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(85..85 src has 30..85) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fb71407cb50 con 0x7fb7240f95e0 2024-02-13T00:39:26.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.095+0000 7fb71b7fe700 10 client.14703.objecter ms_dispatch 0x7fb724001550 osd_map(85..85 src has 30..85) v4 2024-02-13T00:39:26.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.095+0000 7fb71b7fe700 3 client.14703.objecter handle_osd_map got epochs [85,85] > 0 2024-02-13T00:39:26.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.095+0000 7fb71b7fe700 3 client.14703.objecter handle_osd_map decoding full epoch 85 2024-02-13T00:39:26.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.095+0000 7fb71b7fe700 20 client.14703.objecter dump_active .. 0 homeless 2024-02-13T00:39:26.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.095+0000 7fb72a2f8700 1 --2- 172.21.15.162:0/4044515274 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb710062c90 0x7fb710065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:26.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.095+0000 7fb71b7fe700 10 client.14703.objecter ms_handle_connect 0x7fb710062c90 2024-02-13T00:39:26.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.095+0000 7fb72a2f8700 1 --2- 172.21.15.162:0/4044515274 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb710062c90 0x7fb710065150 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb720009a00 tx=0x7fb72000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:26.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.099+0000 7fb71b7fe700 1 -- 172.21.15.162:0/4044515274 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb7140508d0 con 0x7fb7240f95e0 2024-02-13T00:39:26.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.277+0000 7fb72c55c700 1 -- 172.21.15.162:0/4044515274 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb708005200 con 0x7fb7240f95e0 2024-02-13T00:39:26.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.277+0000 7fb71b7fe700 1 -- 172.21.15.162:0/4044515274 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb7140504c0 con 0x7fb7240f95e0 2024-02-13T00:39:26.280 INFO:teuthology.orchestra.run.smithi162.stdout:107374182442 2024-02-13T00:39:26.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 10 librados: watch_flush enter 2024-02-13T00:39:26.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 10 librados: watch_flush exit 2024-02-13T00:39:26.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 20 client.14703.objecter shutdown clearing up homeless session... 2024-02-13T00:39:26.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 10 client.14703.objecter successfully canceled tick 2024-02-13T00:39:26.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 -- 172.21.15.162:0/4044515274 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb710062c90 msgr2=0x7fb710065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:26.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 --2- 172.21.15.162:0/4044515274 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb710062c90 0x7fb710065150 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb720009a00 tx=0x7fb72000b040).stop 2024-02-13T00:39:26.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 -- 172.21.15.162:0/4044515274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7240f95e0 msgr2=0x7fb72410e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:26.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 --2- 172.21.15.162:0/4044515274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7240f95e0 0x7fb72410e720 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fb714004920 tx=0x7fb714009cb0).stop 2024-02-13T00:39:26.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 -- 172.21.15.162:0/4044515274 shutdown_connections 2024-02-13T00:39:26.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 --2- 172.21.15.162:0/4044515274 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb710062c90 0x7fb710065150 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:26.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 --2- 172.21.15.162:0/4044515274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb724102270 0x7fb7241083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:26.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 --2- 172.21.15.162:0/4044515274 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7240f8d10 0x7fb72410e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:26.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 --2- 172.21.15.162:0/4044515274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7240f95e0 0x7fb72410e720 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:26.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 -- 172.21.15.162:0/4044515274 >> 172.21.15.162:0/4044515274 conn(0x7fb72406eb90 msgr2=0x7fb724102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:26.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 -- 172.21.15.162:0/4044515274 shutdown_connections 2024-02-13T00:39:26.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 -- 172.21.15.162:0/4044515274 wait complete. 2024-02-13T00:39:26.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:26.279+0000 7fb72c55c700 1 librados: shutdown 2024-02-13T00:39:26.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:39:26] "GET /metrics HTTP/1.1" 200 172174 "" "Prometheus/2.33.4" 2024-02-13T00:39:26.791 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:24.372629+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.b scrub starts 2024-02-13T00:39:26.792 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:24.376097+0000 osd.0 (osd.0) 34 : cluster [DBG] 4.b scrub ok 2024-02-13T00:39:26.792 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:25.165897+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:39:26.792 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:25.166772+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:39:26.792 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:25.236469+0000 mgr.y (mgr.14544) 72 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:26.792 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:25.317387+0000 osd.1 (osd.1) 35 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:39:26.792 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:25.319056+0000 osd.1 (osd.1) 36 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:39:26.792 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:26.278171+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? 172.21.15.162:0/4044515274' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-13T00:39:26.793 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:24.372629+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.b scrub starts 2024-02-13T00:39:26.793 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:24.376097+0000 osd.0 (osd.0) 34 : cluster [DBG] 4.b scrub ok 2024-02-13T00:39:26.793 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:25.165897+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:39:26.793 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:25.166772+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:39:26.793 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:25.236469+0000 mgr.y (mgr.14544) 72 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:26.793 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:25.317387+0000 osd.1 (osd.1) 35 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:39:26.793 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:25.319056+0000 osd.1 (osd.1) 36 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:39:26.794 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:26.278171+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? 172.21.15.162:0/4044515274' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-13T00:39:26.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:24.372629+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.b scrub starts 2024-02-13T00:39:26.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:24.376097+0000 osd.0 (osd.0) 34 : cluster [DBG] 4.b scrub ok 2024-02-13T00:39:26.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:25.165897+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:39:26.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:25.166772+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:39:26.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:25.236469+0000 mgr.y (mgr.14544) 72 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:26.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:25.317387+0000 osd.1 (osd.1) 35 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:39:26.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:25.319056+0000 osd.1 (osd.1) 36 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:39:26.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:26.278171+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? 172.21.15.162:0/4044515274' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-13T00:39:27.201 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182437 got 107374182442 for osd.3 2024-02-13T00:39:27.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-13T00:39:27.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:26.177722+0000 mgr.y (mgr.14544) 73 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:27.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:26.210575+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:39:27.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:26.214079+0000 osd.3 (osd.3) 36 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:39:27.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:26.177722+0000 mgr.y (mgr.14544) 73 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:27.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:26.210575+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:39:27.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:26.214079+0000 osd.3 (osd.3) 36 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:39:27.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:26.177722+0000 mgr.y (mgr.14544) 73 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:27.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:26.210575+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:39:27.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:26.214079+0000 osd.3 (osd.3) 36 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:39:27.661 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:28.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:27.294221+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:39:28.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:27.297104+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:39:28.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:27.294221+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:39:28.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:27.297104+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:39:28.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:27.294221+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:39:28.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:27.297104+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:39:28.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:39:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:39:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:39:29.284 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=info ts=2024-02-13T00:39:29.126Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001595943s 2024-02-13T00:39:29.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.462+0000 7f6a6fb71700 1 -- 172.21.15.162:0/78316855 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a680fd490 msgr2=0x7f6a680f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:29.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.462+0000 7f6a6fb71700 1 --2- 172.21.15.162:0/78316855 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a680fd490 0x7f6a680f6cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f6a580047b0 tx=0x7f6a58009cb0).stop 2024-02-13T00:39:29.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 -- 172.21.15.162:0/78316855 shutdown_connections 2024-02-13T00:39:29.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 --2- 172.21.15.162:0/78316855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a680f73a0 0x7f6a681039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:29.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 --2- 172.21.15.162:0/78316855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a680fcb30 0x7f6a680fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:29.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 --2- 172.21.15.162:0/78316855 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a680fd490 0x7f6a680f6cc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:29.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 -- 172.21.15.162:0/78316855 >> 172.21.15.162:0/78316855 conn(0x7f6a6806eb90 msgr2=0x7f6a680f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:29.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 -- 172.21.15.162:0/78316855 shutdown_connections 2024-02-13T00:39:29.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 -- 172.21.15.162:0/78316855 wait complete. 2024-02-13T00:39:29.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 librados: starting msgr at 2024-02-13T00:39:29.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 librados: starting objecter 2024-02-13T00:39:29.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 Processor -- start 2024-02-13T00:39:29.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 -- start start 2024-02-13T00:39:29.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 librados: setting wanted keys 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 librados: calling monclient init 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a680f73a0 0x7f6a680f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a680fcb30 0x7f6a680f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a680fd490 0x7f6a680f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68116630 con 0x7f6a680fd490 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.463+0000 7f6a6fb71700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681167a0 con 0x7f6a680fcb30 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6fb71700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68116960 con 0x7f6a680f73a0 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6e10e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a680fd490 0x7f6a680f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6e10e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a680fd490 0x7f6a680f9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34470/0 (socket says 172.21.15.162:34470) 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6e10e700 1 -- 172.21.15.162:0/3554362041 learned_addr learned my addr 172.21.15.162:0/3554362041 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6d10c700 1 --2- 172.21.15.162:0/3554362041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a680fcb30 0x7f6a680f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6d90d700 1 --2- 172.21.15.162:0/3554362041 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a680f73a0 0x7f6a680f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6e10e700 1 -- 172.21.15.162:0/3554362041 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a680f73a0 msgr2=0x7f6a680f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:29.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6e10e700 1 --2- 172.21.15.162:0/3554362041 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a680f73a0 0x7f6a680f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6e10e700 1 -- 172.21.15.162:0/3554362041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a680fcb30 msgr2=0x7f6a680f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6e10e700 1 --2- 172.21.15.162:0/3554362041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a680fcb30 0x7f6a680f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6e10e700 1 -- 172.21.15.162:0/3554362041 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a600097c0 con 0x7f6a680fd490 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6e10e700 1 --2- 172.21.15.162:0/3554362041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a680fd490 0x7f6a680f9310 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f6a60001df0 tx=0x7f6a60004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a5effd700 10 client.?.objecter ms_handle_connect 0x7f6a680fd490 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a5effd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a5effd700 1 -- 172.21.15.162:0/3554362041 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60009a10 con 0x7f6a680fd490 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6fb71700 1 -- 172.21.15.162:0/3554362041 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a58005040 con 0x7f6a680fd490 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6fb71700 10 client.14709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a5effd700 1 -- 172.21.15.162:0/3554362041 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a600090d0 con 0x7f6a680fd490 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6fb71700 1 -- 172.21.15.162:0/3554362041 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a68116fa0 con 0x7f6a680fd490 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a6fb71700 1 librados: init done 2024-02-13T00:39:29.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.464+0000 7f6a5effd700 1 -- 172.21.15.162:0/3554362041 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60007d30 con 0x7f6a680fd490 2024-02-13T00:39:29.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.465+0000 7f6a6fb71700 1 -- 172.21.15.162:0/3554362041 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4c004f50 con 0x7f6a680fd490 2024-02-13T00:39:29.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.466+0000 7f6a5effd700 1 -- 172.21.15.162:0/3554362041 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6a6000fb30 con 0x7f6a680fd490 2024-02-13T00:39:29.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.466+0000 7f6a5effd700 1 --2- 172.21.15.162:0/3554362041 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6a54062c90 0x7f6a54065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:29.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.466+0000 7f6a5effd700 1 -- 172.21.15.162:0/3554362041 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(85..85 src has 30..85) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f6a6007d0c0 con 0x7f6a680fd490 2024-02-13T00:39:29.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.466+0000 7f6a5effd700 10 client.14709.objecter ms_dispatch 0x7f6a68001550 osd_map(85..85 src has 30..85) v4 2024-02-13T00:39:29.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.466+0000 7f6a5effd700 3 client.14709.objecter handle_osd_map got epochs [85,85] > 0 2024-02-13T00:39:29.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.466+0000 7f6a5effd700 3 client.14709.objecter handle_osd_map decoding full epoch 85 2024-02-13T00:39:29.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.466+0000 7f6a6d90d700 1 --2- 172.21.15.162:0/3554362041 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6a54062c90 0x7f6a54065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:29.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.466+0000 7f6a5effd700 20 client.14709.objecter dump_active .. 0 homeless 2024-02-13T00:39:29.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.466+0000 7f6a6d90d700 1 --2- 172.21.15.162:0/3554362041 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6a54062c90 0x7f6a54065150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6a640041d0 tx=0x7f6a64009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:29.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.466+0000 7f6a5effd700 10 client.14709.objecter ms_handle_connect 0x7f6a54062c90 2024-02-13T00:39:29.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.470+0000 7f6a5effd700 1 -- 172.21.15.162:0/3554362041 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6a60050dd0 con 0x7f6a680fd490 2024-02-13T00:39:29.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.642+0000 7f6a6fb71700 1 -- 172.21.15.162:0/3554362041 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f6a4c005200 con 0x7f6a680fd490 2024-02-13T00:39:29.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.642+0000 7f6a5effd700 1 -- 172.21.15.162:0/3554362041 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6a6001c300 con 0x7f6a680fd490 2024-02-13T00:39:29.645 INFO:teuthology.orchestra.run.smithi162.stdout:133143986215 2024-02-13T00:39:29.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 10 librados: watch_flush enter 2024-02-13T00:39:29.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 10 librados: watch_flush exit 2024-02-13T00:39:29.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 20 client.14709.objecter shutdown clearing up homeless session... 2024-02-13T00:39:29.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 10 client.14709.objecter successfully canceled tick 2024-02-13T00:39:29.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 -- 172.21.15.162:0/3554362041 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6a54062c90 msgr2=0x7f6a54065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:29.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 --2- 172.21.15.162:0/3554362041 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6a54062c90 0x7f6a54065150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6a640041d0 tx=0x7f6a64009040).stop 2024-02-13T00:39:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 -- 172.21.15.162:0/3554362041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a680fd490 msgr2=0x7f6a680f9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 --2- 172.21.15.162:0/3554362041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a680fd490 0x7f6a680f9310 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f6a60001df0 tx=0x7f6a60004d10).stop 2024-02-13T00:39:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 -- 172.21.15.162:0/3554362041 shutdown_connections 2024-02-13T00:39:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 --2- 172.21.15.162:0/3554362041 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6a54062c90 0x7f6a54065150 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 --2- 172.21.15.162:0/3554362041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a680fd490 0x7f6a680f9310 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 --2- 172.21.15.162:0/3554362041 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a680f73a0 0x7f6a680f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 --2- 172.21.15.162:0/3554362041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a680fcb30 0x7f6a680f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 -- 172.21.15.162:0/3554362041 >> 172.21.15.162:0/3554362041 conn(0x7f6a6806eb90 msgr2=0x7f6a68070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 -- 172.21.15.162:0/3554362041 shutdown_connections 2024-02-13T00:39:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 -- 172.21.15.162:0/3554362041 wait complete. 2024-02-13T00:39:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:29.644+0000 7f6a6fb71700 1 librados: shutdown 2024-02-13T00:39:29.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:28.168830+0000 osd.7 (osd.7) 35 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:39:29.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:28.171737+0000 osd.7 (osd.7) 36 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:39:29.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:28.178093+0000 mgr.y (mgr.14544) 74 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:29.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:28.285890+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.10 scrub starts 2024-02-13T00:39:29.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:28.286570+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.10 scrub ok 2024-02-13T00:39:29.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:28.300172+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.c deep-scrub starts 2024-02-13T00:39:29.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:28.305175+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.c deep-scrub ok 2024-02-13T00:39:29.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:29.208288+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:29.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:28.168830+0000 osd.7 (osd.7) 35 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:39:29.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:28.171737+0000 osd.7 (osd.7) 36 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:39:29.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:28.178093+0000 mgr.y (mgr.14544) 74 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:29.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:28.285890+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.10 scrub starts 2024-02-13T00:39:29.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:28.286570+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.10 scrub ok 2024-02-13T00:39:29.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:28.300172+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.c deep-scrub starts 2024-02-13T00:39:29.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:28.305175+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.c deep-scrub ok 2024-02-13T00:39:29.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:29.208288+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:29.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:28.168830+0000 osd.7 (osd.7) 35 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:39:29.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:28.171737+0000 osd.7 (osd.7) 36 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:39:29.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:28.178093+0000 mgr.y (mgr.14544) 74 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:29.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:28.285890+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.10 scrub starts 2024-02-13T00:39:29.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:28.286570+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.10 scrub ok 2024-02-13T00:39:29.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:28.300172+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.c deep-scrub starts 2024-02-13T00:39:29.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:28.305175+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.c deep-scrub ok 2024-02-13T00:39:29.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:29.208288+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:30.487 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:29.158086+0000 osd.7 (osd.7) 37 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:39:30.487 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:29.159073+0000 osd.7 (osd.7) 38 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:39:30.487 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:29.226789+0000 osd.3 (osd.3) 37 : cluster [DBG] 5.13 deep-scrub starts 2024-02-13T00:39:30.487 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:29.229672+0000 osd.3 (osd.3) 38 : cluster [DBG] 5.13 deep-scrub ok 2024-02-13T00:39:30.487 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:29.271663+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.e scrub starts 2024-02-13T00:39:30.487 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:29.275130+0000 osd.4 (osd.4) 24 : cluster [DBG] 4.e scrub ok 2024-02-13T00:39:30.487 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:29.643378+0000 mon.a (mon.0) 707 : audit [DBG] from='client.? 172.21.15.162:0/3554362041' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-13T00:39:30.487 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:30.269587+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.3 scrub starts 2024-02-13T00:39:30.488 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:30.273161+0000 osd.0 (osd.0) 38 : cluster [DBG] 5.3 scrub ok 2024-02-13T00:39:30.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:29.158086+0000 osd.7 (osd.7) 37 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:39:30.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:29.159073+0000 osd.7 (osd.7) 38 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:39:30.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:29.226789+0000 osd.3 (osd.3) 37 : cluster [DBG] 5.13 deep-scrub starts 2024-02-13T00:39:30.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:29.229672+0000 osd.3 (osd.3) 38 : cluster [DBG] 5.13 deep-scrub ok 2024-02-13T00:39:30.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:29.271663+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.e scrub starts 2024-02-13T00:39:30.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:29.275130+0000 osd.4 (osd.4) 24 : cluster [DBG] 4.e scrub ok 2024-02-13T00:39:30.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:29.643378+0000 mon.a (mon.0) 707 : audit [DBG] from='client.? 172.21.15.162:0/3554362041' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-13T00:39:30.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:30.269587+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.3 scrub starts 2024-02-13T00:39:30.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:30.273161+0000 osd.0 (osd.0) 38 : cluster [DBG] 5.3 scrub ok 2024-02-13T00:39:30.506 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986209 got 133143986215 for osd.4 2024-02-13T00:39:30.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-13T00:39:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:29.158086+0000 osd.7 (osd.7) 37 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:39:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:29.159073+0000 osd.7 (osd.7) 38 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:39:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:29.226789+0000 osd.3 (osd.3) 37 : cluster [DBG] 5.13 deep-scrub starts 2024-02-13T00:39:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:29.229672+0000 osd.3 (osd.3) 38 : cluster [DBG] 5.13 deep-scrub ok 2024-02-13T00:39:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:29.271663+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.e scrub starts 2024-02-13T00:39:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:29.275130+0000 osd.4 (osd.4) 24 : cluster [DBG] 4.e scrub ok 2024-02-13T00:39:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:29.643378+0000 mon.a (mon.0) 707 : audit [DBG] from='client.? 172.21.15.162:0/3554362041' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-13T00:39:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:30.269587+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.3 scrub starts 2024-02-13T00:39:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:30.273161+0000 osd.0 (osd.0) 38 : cluster [DBG] 5.3 scrub ok 2024-02-13T00:39:30.870 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:31.454 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:30.178652+0000 mgr.y (mgr.14544) 75 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:31.454 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:30.242518+0000 osd.1 (osd.1) 39 : cluster [DBG] 5.1d scrub starts 2024-02-13T00:39:31.454 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:30.245503+0000 osd.1 (osd.1) 40 : cluster [DBG] 5.1d scrub ok 2024-02-13T00:39:31.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:30.178652+0000 mgr.y (mgr.14544) 75 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:31.455 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:30.242518+0000 osd.1 (osd.1) 39 : cluster [DBG] 5.1d scrub starts 2024-02-13T00:39:31.455 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:30.245503+0000 osd.1 (osd.1) 40 : cluster [DBG] 5.1d scrub ok 2024-02-13T00:39:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:30.178652+0000 mgr.y (mgr.14544) 75 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:30.242518+0000 osd.1 (osd.1) 39 : cluster [DBG] 5.1d scrub starts 2024-02-13T00:39:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:30.245503+0000 osd.1 (osd.1) 40 : cluster [DBG] 5.1d scrub ok 2024-02-13T00:39:33.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:31.971454+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:33.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:31.972608+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:39:33.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:31.973241+0000 mon.c (mon.2) 74 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:39:33.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:31.977090+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:33.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:32.179115+0000 mgr.y (mgr.14544) 76 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:33.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:32.252112+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.4 deep-scrub starts 2024-02-13T00:39:33.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:32.255257+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.4 deep-scrub ok 2024-02-13T00:39:33.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:32.310583+0000 osd.0 (osd.0) 39 : cluster [DBG] 2.1f deep-scrub starts 2024-02-13T00:39:33.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:32.313834+0000 osd.0 (osd.0) 40 : cluster [DBG] 2.1f deep-scrub ok 2024-02-13T00:39:33.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:31.971454+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:33.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:31.972608+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:39:33.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:31.973241+0000 mon.c (mon.2) 74 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:39:33.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:31.977090+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:33.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:32.179115+0000 mgr.y (mgr.14544) 76 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:33.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:32.252112+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.4 deep-scrub starts 2024-02-13T00:39:33.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:32.255257+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.4 deep-scrub ok 2024-02-13T00:39:33.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:32.310583+0000 osd.0 (osd.0) 39 : cluster [DBG] 2.1f deep-scrub starts 2024-02-13T00:39:33.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:32.313834+0000 osd.0 (osd.0) 40 : cluster [DBG] 2.1f deep-scrub ok 2024-02-13T00:39:33.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.282+0000 7fe4e6217700 1 -- 172.21.15.162:0/1727756134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4e00fcb30 msgr2=0x7fe4e00fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:33.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.282+0000 7fe4e6217700 1 --2- 172.21.15.162:0/1727756134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4e00fcb30 0x7fe4e00fcf10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fe4d8009be0 tx=0x7fe4d8004d10).stop 2024-02-13T00:39:33.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.282+0000 7fe4e6217700 1 -- 172.21.15.162:0/1727756134 shutdown_connections 2024-02-13T00:39:33.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.282+0000 7fe4e6217700 1 --2- 172.21.15.162:0/1727756134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4e00f73a0 0x7fe4e01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:33.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.282+0000 7fe4e6217700 1 --2- 172.21.15.162:0/1727756134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4e00fcb30 0x7fe4e00fcf10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:33.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.282+0000 7fe4e6217700 1 --2- 172.21.15.162:0/1727756134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4e00fd490 0x7fe4e00f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:33.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.282+0000 7fe4e6217700 1 -- 172.21.15.162:0/1727756134 >> 172.21.15.162:0/1727756134 conn(0x7fe4e006eb90 msgr2=0x7fe4e00f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:33.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.282+0000 7fe4e6217700 1 -- 172.21.15.162:0/1727756134 shutdown_connections 2024-02-13T00:39:33.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.282+0000 7fe4e6217700 1 -- 172.21.15.162:0/1727756134 wait complete. 2024-02-13T00:39:33.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.283+0000 7fe4e6217700 1 librados: starting msgr at 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.283+0000 7fe4e6217700 1 librados: starting objecter 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.283+0000 7fe4e6217700 1 Processor -- start 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.283+0000 7fe4e6217700 1 -- start start 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.283+0000 7fe4e6217700 1 librados: setting wanted keys 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.283+0000 7fe4e6217700 1 librados: calling monclient init 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.283+0000 7fe4e6217700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4e00f73a0 0x7fe4e010d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.283+0000 7fe4e6217700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4e00fcb30 0x7fe4e010d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.283+0000 7fe4e6217700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4e00fd490 0x7fe4e01072f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.283+0000 7fe4e6217700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e01077e0 con 0x7fe4e00f73a0 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.283+0000 7fe4e6217700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0107950 con 0x7fe4e00fcb30 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.283+0000 7fe4e6217700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0107b10 con 0x7fe4e00fd490 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4df7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4e00f73a0 0x7fe4e010d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4df7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4e00f73a0 0x7fe4e010d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34482/0 (socket says 172.21.15.162:34482) 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4df7fe700 1 -- 172.21.15.162:0/2373908490 learned_addr learned my addr 172.21.15.162:0/2373908490 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4dffff700 1 --2- 172.21.15.162:0/2373908490 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4e00fd490 0x7fe4e01072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4deffd700 1 --2- 172.21.15.162:0/2373908490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4e00fcb30 0x7fe4e010d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4df7fe700 1 -- 172.21.15.162:0/2373908490 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4e00fd490 msgr2=0x7fe4e01072f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4df7fe700 1 --2- 172.21.15.162:0/2373908490 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4e00fd490 0x7fe4e01072f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4df7fe700 1 -- 172.21.15.162:0/2373908490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4e00fcb30 msgr2=0x7fe4e010d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4df7fe700 1 --2- 172.21.15.162:0/2373908490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4e00fcb30 0x7fe4e010d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4df7fe700 1 -- 172.21.15.162:0/2373908490 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c8005040 con 0x7fe4e00f73a0 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4df7fe700 1 --2- 172.21.15.162:0/2373908490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4e00f73a0 0x7fe4e010d150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fe4d800b6d0 tx=0x7fe4d8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4dcff9700 10 client.?.objecter ms_handle_connect 0x7fe4e00f73a0 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4dcff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4dcff9700 1 -- 172.21.15.162:0/2373908490 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d800e030 con 0x7fe4e00f73a0 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4dcff9700 1 -- 172.21.15.162:0/2373908490 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe4d800be70 con 0x7fe4e00f73a0 2024-02-13T00:39:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4e6217700 1 -- 172.21.15.162:0/2373908490 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d80097c0 con 0x7fe4e00f73a0 2024-02-13T00:39:33.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4e6217700 10 client.14715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:33.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4dcff9700 1 -- 172.21.15.162:0/2373908490 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8018070 con 0x7fe4e00f73a0 2024-02-13T00:39:33.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4e6217700 1 -- 172.21.15.162:0/2373908490 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e0108030 con 0x7fe4e00f73a0 2024-02-13T00:39:33.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.284+0000 7fe4e6217700 1 librados: init done 2024-02-13T00:39:33.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.286+0000 7fe4dcff9700 1 -- 172.21.15.162:0/2373908490 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe4d8007b20 con 0x7fe4e00f73a0 2024-02-13T00:39:33.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.286+0000 7fe4e6217700 1 -- 172.21.15.162:0/2373908490 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e00475e0 con 0x7fe4e00f73a0 2024-02-13T00:39:33.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.286+0000 7fe4dcff9700 1 --2- 172.21.15.162:0/2373908490 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe4cc062c90 0x7fe4cc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:33.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.286+0000 7fe4dcff9700 1 -- 172.21.15.162:0/2373908490 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(85..85 src has 30..85) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fe4d807e070 con 0x7fe4e00f73a0 2024-02-13T00:39:33.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.286+0000 7fe4dcff9700 10 client.14715.objecter ms_dispatch 0x7fe4e0001550 osd_map(85..85 src has 30..85) v4 2024-02-13T00:39:33.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.286+0000 7fe4dcff9700 3 client.14715.objecter handle_osd_map got epochs [85,85] > 0 2024-02-13T00:39:33.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.286+0000 7fe4dcff9700 3 client.14715.objecter handle_osd_map decoding full epoch 85 2024-02-13T00:39:33.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.286+0000 7fe4deffd700 1 --2- 172.21.15.162:0/2373908490 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe4cc062c90 0x7fe4cc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:33.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.286+0000 7fe4dcff9700 20 client.14715.objecter dump_active .. 0 homeless 2024-02-13T00:39:33.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.286+0000 7fe4dcff9700 10 client.14715.objecter ms_handle_connect 0x7fe4cc062c90 2024-02-13T00:39:33.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.286+0000 7fe4deffd700 1 --2- 172.21.15.162:0/2373908490 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe4cc062c90 0x7fe4cc065150 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe4e0108a90 tx=0x7fe4c800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:33.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.290+0000 7fe4dcff9700 1 -- 172.21.15.162:0/2373908490 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe4d8051ce0 con 0x7fe4e00f73a0 2024-02-13T00:39:33.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:31.971454+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:33.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:31.972608+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:39:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:31.973241+0000 mon.c (mon.2) 74 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:39:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:31.977090+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:32.179115+0000 mgr.y (mgr.14544) 76 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:32.252112+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.4 deep-scrub starts 2024-02-13T00:39:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:32.255257+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.4 deep-scrub ok 2024-02-13T00:39:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:32.310583+0000 osd.0 (osd.0) 39 : cluster [DBG] 2.1f deep-scrub starts 2024-02-13T00:39:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:32.313834+0000 osd.0 (osd.0) 40 : cluster [DBG] 2.1f deep-scrub ok 2024-02-13T00:39:33.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.454+0000 7fe4e6217700 1 -- 172.21.15.162:0/2373908490 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe4e01088e0 con 0x7fe4e00f73a0 2024-02-13T00:39:33.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.455+0000 7fe4dcff9700 1 -- 172.21.15.162:0/2373908490 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe4d8004810 con 0x7fe4e00f73a0 2024-02-13T00:39:33.458 INFO:teuthology.orchestra.run.smithi162.stdout:167503724579 2024-02-13T00:39:33.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 10 librados: watch_flush enter 2024-02-13T00:39:33.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 10 librados: watch_flush exit 2024-02-13T00:39:33.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 20 client.14715.objecter shutdown clearing up homeless session... 2024-02-13T00:39:33.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 10 client.14715.objecter successfully canceled tick 2024-02-13T00:39:33.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 1 -- 172.21.15.162:0/2373908490 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe4cc062c90 msgr2=0x7fe4cc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:33.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 1 --2- 172.21.15.162:0/2373908490 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe4cc062c90 0x7fe4cc065150 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe4e0108a90 tx=0x7fe4c800b040).stop 2024-02-13T00:39:33.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 1 -- 172.21.15.162:0/2373908490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4e00f73a0 msgr2=0x7fe4e010d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:33.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 1 --2- 172.21.15.162:0/2373908490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4e00f73a0 0x7fe4e010d150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fe4d800b6d0 tx=0x7fe4d8004d10).stop 2024-02-13T00:39:33.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 1 -- 172.21.15.162:0/2373908490 shutdown_connections 2024-02-13T00:39:33.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 1 --2- 172.21.15.162:0/2373908490 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe4cc062c90 0x7fe4cc065150 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:33.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 1 --2- 172.21.15.162:0/2373908490 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4e00fd490 0x7fe4e01072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:33.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 1 --2- 172.21.15.162:0/2373908490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4e00f73a0 0x7fe4e010d150 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:33.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.456+0000 7fe4e6217700 1 --2- 172.21.15.162:0/2373908490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4e00fcb30 0x7fe4e010d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:33.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.457+0000 7fe4e6217700 1 -- 172.21.15.162:0/2373908490 >> 172.21.15.162:0/2373908490 conn(0x7fe4e006eb90 msgr2=0x7fe4e00700b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:33.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.457+0000 7fe4e6217700 1 -- 172.21.15.162:0/2373908490 shutdown_connections 2024-02-13T00:39:33.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.457+0000 7fe4e6217700 1 -- 172.21.15.162:0/2373908490 wait complete. 2024-02-13T00:39:33.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:33.457+0000 7fe4e6217700 1 librados: shutdown 2024-02-13T00:39:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:33.175979+0000 osd.3 (osd.3) 39 : cluster [DBG] 6.b scrub starts 2024-02-13T00:39:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:33.179221+0000 osd.3 (osd.3) 40 : cluster [DBG] 6.b scrub ok 2024-02-13T00:39:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:33.300802+0000 osd.1 (osd.1) 43 : cluster [DBG] 6.1 scrub starts 2024-02-13T00:39:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:33.303974+0000 osd.1 (osd.1) 44 : cluster [DBG] 6.1 scrub ok 2024-02-13T00:39:34.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:33.455869+0000 mon.a (mon.0) 710 : audit [DBG] from='client.? 172.21.15.162:0/2373908490' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-13T00:39:34.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:33.175979+0000 osd.3 (osd.3) 39 : cluster [DBG] 6.b scrub starts 2024-02-13T00:39:34.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:33.179221+0000 osd.3 (osd.3) 40 : cluster [DBG] 6.b scrub ok 2024-02-13T00:39:34.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:33.300802+0000 osd.1 (osd.1) 43 : cluster [DBG] 6.1 scrub starts 2024-02-13T00:39:34.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:33.303974+0000 osd.1 (osd.1) 44 : cluster [DBG] 6.1 scrub ok 2024-02-13T00:39:34.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:33.455869+0000 mon.a (mon.0) 710 : audit [DBG] from='client.? 172.21.15.162:0/2373908490' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-13T00:39:34.243 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724573 got 167503724579 for osd.5 2024-02-13T00:39:34.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-13T00:39:34.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:33.175979+0000 osd.3 (osd.3) 39 : cluster [DBG] 6.b scrub starts 2024-02-13T00:39:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:33.179221+0000 osd.3 (osd.3) 40 : cluster [DBG] 6.b scrub ok 2024-02-13T00:39:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:33.300802+0000 osd.1 (osd.1) 43 : cluster [DBG] 6.1 scrub starts 2024-02-13T00:39:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:33.303974+0000 osd.1 (osd.1) 44 : cluster [DBG] 6.1 scrub ok 2024-02-13T00:39:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:33.455869+0000 mon.a (mon.0) 710 : audit [DBG] from='client.? 172.21.15.162:0/2373908490' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-13T00:39:34.519 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:35.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:39:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:39:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:34.179595+0000 mgr.y (mgr.14544) 77 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-02-13T00:39:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:34.200964+0000 osd.3 (osd.3) 41 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:39:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:34.204884+0000 osd.3 (osd.3) 42 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:39:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:34.209526+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:39:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:34.213070+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:39:35.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:34.179595+0000 mgr.y (mgr.14544) 77 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-02-13T00:39:35.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:34.200964+0000 osd.3 (osd.3) 41 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:39:35.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:34.204884+0000 osd.3 (osd.3) 42 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:39:35.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:34.209526+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:39:35.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:34.213070+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:39:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:34.179595+0000 mgr.y (mgr.14544) 77 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-02-13T00:39:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:34.200964+0000 osd.3 (osd.3) 41 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:39:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:34.204884+0000 osd.3 (osd.3) 42 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:39:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:34.209526+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:39:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:34.213070+0000 osd.4 (osd.4) 26 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:39:36.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.249+0000 7fd84a56a700 1 -- 172.21.15.162:0/2061922430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8440fcb30 msgr2=0x7fd8440fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:36.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.249+0000 7fd84a56a700 1 --2- 172.21.15.162:0/2061922430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8440fcb30 0x7fd8440fcf10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fd834004870 tx=0x7fd834009cb0).stop 2024-02-13T00:39:36.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.249+0000 7fd84a56a700 1 -- 172.21.15.162:0/2061922430 shutdown_connections 2024-02-13T00:39:36.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.249+0000 7fd84a56a700 1 --2- 172.21.15.162:0/2061922430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8440f73a0 0x7fd8441039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:36.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.249+0000 7fd84a56a700 1 --2- 172.21.15.162:0/2061922430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8440fcb30 0x7fd8440fcf10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:36.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.249+0000 7fd84a56a700 1 --2- 172.21.15.162:0/2061922430 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8440fd490 0x7fd8440f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:36.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.249+0000 7fd84a56a700 1 -- 172.21.15.162:0/2061922430 >> 172.21.15.162:0/2061922430 conn(0x7fd84406eb90 msgr2=0x7fd8440f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:36.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.249+0000 7fd84a56a700 1 -- 172.21.15.162:0/2061922430 shutdown_connections 2024-02-13T00:39:36.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.250+0000 7fd84a56a700 1 -- 172.21.15.162:0/2061922430 wait complete. 2024-02-13T00:39:36.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.250+0000 7fd84a56a700 1 librados: starting msgr at 2024-02-13T00:39:36.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.250+0000 7fd84a56a700 1 librados: starting objecter 2024-02-13T00:39:36.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.250+0000 7fd84a56a700 1 Processor -- start 2024-02-13T00:39:36.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.250+0000 7fd84a56a700 1 -- start start 2024-02-13T00:39:36.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.250+0000 7fd84a56a700 1 librados: setting wanted keys 2024-02-13T00:39:36.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.250+0000 7fd84a56a700 1 librados: calling monclient init 2024-02-13T00:39:36.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.250+0000 7fd84a56a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8440f73a0 0x7fd8441091f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.250+0000 7fd84a56a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8440fcb30 0x7fd8441096e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd84a56a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8440fd490 0x7fd844109cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd84a56a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8441143d0 con 0x7fd8440fd490 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd84a56a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd844114540 con 0x7fd8440fcb30 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd84a56a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd844114700 con 0x7fd8440f73a0 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd843fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8440f73a0 0x7fd8441091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd848b07700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8440fd490 0x7fd844109cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd843fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8440f73a0 0x7fd8441091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:44708/0 (socket says 172.21.15.162:44708) 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd843fff700 1 -- 172.21.15.162:0/3811792811 learned_addr learned my addr 172.21.15.162:0/3811792811 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd848b07700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8440fd490 0x7fd844109cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46314/0 (socket says 172.21.15.162:46314) 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd8437fe700 1 --2- 172.21.15.162:0/3811792811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8440fcb30 0x7fd8441096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd843fff700 1 -- 172.21.15.162:0/3811792811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8440fcb30 msgr2=0x7fd8441096e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd843fff700 1 --2- 172.21.15.162:0/3811792811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8440fcb30 0x7fd8441096e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd843fff700 1 -- 172.21.15.162:0/3811792811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8440fd490 msgr2=0x7fd844109cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd843fff700 1 --2- 172.21.15.162:0/3811792811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8440fd490 0x7fd844109cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd843fff700 1 -- 172.21.15.162:0/3811792811 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838005040 con 0x7fd8440f73a0 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd8437fe700 1 --2- 172.21.15.162:0/3811792811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8440fcb30 0x7fd8441096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd848b07700 1 --2- 172.21.15.162:0/3811792811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8440fd490 0x7fd844109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd843fff700 1 --2- 172.21.15.162:0/3811792811 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8440f73a0 0x7fd8441091f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fd834009eb0 tx=0x7fd83400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd8417fa700 10 client.?.objecter ms_handle_connect 0x7fd8440f73a0 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd8417fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd8417fa700 1 -- 172.21.15.162:0/3811792811 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83400e070 con 0x7fd8440f73a0 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd84a56a700 1 -- 172.21.15.162:0/3811792811 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd834005040 con 0x7fd8440f73a0 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd8417fa700 1 -- 172.21.15.162:0/3811792811 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd834013da0 con 0x7fd8440f73a0 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd8417fa700 1 -- 172.21.15.162:0/3811792811 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd834019070 con 0x7fd8440f73a0 2024-02-13T00:39:36.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd84a56a700 10 client.24620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd84a56a700 1 -- 172.21.15.162:0/3811792811 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd844114cd0 con 0x7fd8440f73a0 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.251+0000 7fd84a56a700 1 librados: init done 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.252+0000 7fd84a56a700 1 -- 172.21.15.162:0/3811792811 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8440475e0 con 0x7fd8440f73a0 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.253+0000 7fd8417fa700 1 -- 172.21.15.162:0/3811792811 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fd83401e420 con 0x7fd8440f73a0 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.253+0000 7fd8417fa700 1 --2- 172.21.15.162:0/3811792811 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd82c062c90 0x7fd82c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.253+0000 7fd8417fa700 1 -- 172.21.15.162:0/3811792811 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(85..85 src has 30..85) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fd83407d5f0 con 0x7fd8440f73a0 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.253+0000 7fd8417fa700 10 client.24620.objecter ms_dispatch 0x7fd844001550 osd_map(85..85 src has 30..85) v4 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.253+0000 7fd8417fa700 3 client.24620.objecter handle_osd_map got epochs [85,85] > 0 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.253+0000 7fd8417fa700 3 client.24620.objecter handle_osd_map decoding full epoch 85 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.253+0000 7fd8437fe700 1 --2- 172.21.15.162:0/3811792811 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd82c062c90 0x7fd82c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.253+0000 7fd8417fa700 20 client.24620.objecter dump_active .. 0 homeless 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.254+0000 7fd8417fa700 10 client.24620.objecter ms_handle_connect 0x7fd82c062c90 2024-02-13T00:39:36.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.254+0000 7fd8437fe700 1 --2- 172.21.15.162:0/3811792811 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd82c062c90 0x7fd82c065150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fd838007fd0 tx=0x7fd838009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:36.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.258+0000 7fd8417fa700 1 -- 172.21.15.162:0/3811792811 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd834051280 con 0x7fd8440f73a0 2024-02-13T00:39:36.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.420+0000 7fd84a56a700 1 -- 172.21.15.162:0/3811792811 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd844105fe0 con 0x7fd8440f73a0 2024-02-13T00:39:36.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.420+0000 7fd8417fa700 1 -- 172.21.15.162:0/3811792811 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd834050e70 con 0x7fd8440f73a0 2024-02-13T00:39:36.423 INFO:teuthology.orchestra.run.smithi162.stdout:193273528351 2024-02-13T00:39:36.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 10 librados: watch_flush enter 2024-02-13T00:39:36.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 10 librados: watch_flush exit 2024-02-13T00:39:36.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 20 client.24620.objecter shutdown clearing up homeless session... 2024-02-13T00:39:36.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 10 client.24620.objecter successfully canceled tick 2024-02-13T00:39:36.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 -- 172.21.15.162:0/3811792811 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd82c062c90 msgr2=0x7fd82c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:36.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 --2- 172.21.15.162:0/3811792811 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd82c062c90 0x7fd82c065150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fd838007fd0 tx=0x7fd838009cb0).stop 2024-02-13T00:39:36.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 -- 172.21.15.162:0/3811792811 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8440f73a0 msgr2=0x7fd8441091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:36.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 --2- 172.21.15.162:0/3811792811 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8440f73a0 0x7fd8441091f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fd834009eb0 tx=0x7fd83400b040).stop 2024-02-13T00:39:36.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 -- 172.21.15.162:0/3811792811 shutdown_connections 2024-02-13T00:39:36.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 --2- 172.21.15.162:0/3811792811 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd82c062c90 0x7fd82c065150 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:36.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 --2- 172.21.15.162:0/3811792811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8440fd490 0x7fd844109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:36.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 --2- 172.21.15.162:0/3811792811 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8440f73a0 0x7fd8441091f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:36.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 --2- 172.21.15.162:0/3811792811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8440fcb30 0x7fd8441096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:36.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 -- 172.21.15.162:0/3811792811 >> 172.21.15.162:0/3811792811 conn(0x7fd84406eb90 msgr2=0x7fd8440fa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:36.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 -- 172.21.15.162:0/3811792811 shutdown_connections 2024-02-13T00:39:36.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 -- 172.21.15.162:0/3811792811 wait complete. 2024-02-13T00:39:36.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:36.422+0000 7fd84a56a700 1 librados: shutdown 2024-02-13T00:39:36.448 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:35.242516+0000 mgr.y (mgr.14544) 78 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:36.448 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:35.266442+0000 osd.1 (osd.1) 45 : cluster [DBG] 5.c scrub starts 2024-02-13T00:39:36.448 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:35.269534+0000 osd.1 (osd.1) 46 : cluster [DBG] 5.c scrub ok 2024-02-13T00:39:36.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:35.242516+0000 mgr.y (mgr.14544) 78 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:36.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:35.266442+0000 osd.1 (osd.1) 45 : cluster [DBG] 5.c scrub starts 2024-02-13T00:39:36.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:35.269534+0000 osd.1 (osd.1) 46 : cluster [DBG] 5.c scrub ok 2024-02-13T00:39:36.449 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:39:36] "GET /metrics HTTP/1.1" 200 172174 "" "Prometheus/2.33.4" 2024-02-13T00:39:36.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:35.242516+0000 mgr.y (mgr.14544) 78 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:36.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:35.266442+0000 osd.1 (osd.1) 45 : cluster [DBG] 5.c scrub starts 2024-02-13T00:39:36.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:35.269534+0000 osd.1 (osd.1) 46 : cluster [DBG] 5.c scrub ok 2024-02-13T00:39:37.226 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528344 got 193273528351 for osd.6 2024-02-13T00:39:37.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-13T00:39:37.499 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:37.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:36.154474+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.6 scrub starts 2024-02-13T00:39:37.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:36.157689+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.6 scrub ok 2024-02-13T00:39:37.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:36.180405+0000 mgr.y (mgr.14544) 79 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:39:37.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:36.421105+0000 mon.c (mon.2) 75 : audit [DBG] from='client.? 172.21.15.162:0/3811792811' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-13T00:39:37.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:36.154474+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.6 scrub starts 2024-02-13T00:39:37.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:36.157689+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.6 scrub ok 2024-02-13T00:39:37.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:36.180405+0000 mgr.y (mgr.14544) 79 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:39:37.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:36.421105+0000 mon.c (mon.2) 75 : audit [DBG] from='client.? 172.21.15.162:0/3811792811' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-13T00:39:37.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:36.154474+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.6 scrub starts 2024-02-13T00:39:37.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:36.157689+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.6 scrub ok 2024-02-13T00:39:37.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:36.180405+0000 mgr.y (mgr.14544) 79 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:39:37.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:36.421105+0000 mon.c (mon.2) 75 : audit [DBG] from='client.? 172.21.15.162:0/3811792811' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-13T00:39:38.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:37.207497+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:39:38.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:37.210403+0000 osd.4 (osd.4) 28 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:39:38.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:37.207497+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:39:38.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:37.210403+0000 osd.4 (osd.4) 28 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:39:38.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:37.207497+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:39:38.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:37.210403+0000 osd.4 (osd.4) 28 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:39:39.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:39:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:39:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:39:39.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.208+0000 7f5c51fb2700 1 -- 172.21.15.162:0/77805393 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c4c0fa440 msgr2=0x7f5c4c0f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:39.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.208+0000 7f5c51fb2700 1 --2- 172.21.15.162:0/77805393 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c4c0fa440 0x7f5c4c0f7e20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f5c40004870 tx=0x7f5c40009cb0).stop 2024-02-13T00:39:39.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.208+0000 7f5c51fb2700 1 -- 172.21.15.162:0/77805393 shutdown_connections 2024-02-13T00:39:39.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.208+0000 7f5c51fb2700 1 --2- 172.21.15.162:0/77805393 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c4c0f8500 0x7f5c4c104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:39.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.208+0000 7f5c51fb2700 1 --2- 172.21.15.162:0/77805393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c0f9b70 0x7f5c4c0f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:39.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.208+0000 7f5c51fb2700 1 --2- 172.21.15.162:0/77805393 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c4c0fa440 0x7f5c4c0f7e20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:39.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.208+0000 7f5c51fb2700 1 -- 172.21.15.162:0/77805393 >> 172.21.15.162:0/77805393 conn(0x7f5c4c0f38c0 msgr2=0x7f5c4c0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:39.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.208+0000 7f5c51fb2700 1 -- 172.21.15.162:0/77805393 shutdown_connections 2024-02-13T00:39:39.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 -- 172.21.15.162:0/77805393 wait complete. 2024-02-13T00:39:39.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 librados: starting msgr at 2024-02-13T00:39:39.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 librados: starting objecter 2024-02-13T00:39:39.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 Processor -- start 2024-02-13T00:39:39.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 -- start start 2024-02-13T00:39:39.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 librados: setting wanted keys 2024-02-13T00:39:39.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 librados: calling monclient init 2024-02-13T00:39:39.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c0f8500 0x7f5c4c10e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:39.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c4c0f9b70 0x7f5c4c10e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:39.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c4c0fa440 0x7f5c4c108410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:39.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c108900 con 0x7f5c4c0f8500 2024-02-13T00:39:39.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c108a70 con 0x7f5c4c0fa440 2024-02-13T00:39:39.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.209+0000 7f5c51fb2700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c108c30 con 0x7f5c4c0f9b70 2024-02-13T00:39:39.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c4b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c0f8500 0x7f5c4c10e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:39.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c4affd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c4c0f9b70 0x7f5c4c10e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:39.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c4b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c0f8500 0x7f5c4c10e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46328/0 (socket says 172.21.15.162:46328) 2024-02-13T00:39:39.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c4b7fe700 1 -- 172.21.15.162:0/2627246520 learned_addr learned my addr 172.21.15.162:0/2627246520 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:39.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c4bfff700 1 --2- 172.21.15.162:0/2627246520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c4c0fa440 0x7f5c4c108410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:39.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c4b7fe700 1 -- 172.21.15.162:0/2627246520 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c4c0f9b70 msgr2=0x7f5c4c10e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:39.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c4b7fe700 1 --2- 172.21.15.162:0/2627246520 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c4c0f9b70 0x7f5c4c10e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:39.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c4b7fe700 1 -- 172.21.15.162:0/2627246520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c4c0fa440 msgr2=0x7f5c4c108410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:39.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c4b7fe700 1 --2- 172.21.15.162:0/2627246520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c4c0fa440 0x7f5c4c108410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:39.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c4b7fe700 1 -- 172.21.15.162:0/2627246520 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c005040 con 0x7f5c4c0f8500 2024-02-13T00:39:39.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c4affd700 1 --2- 172.21.15.162:0/2627246520 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c4c0f9b70 0x7f5c4c10e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:39:39.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c4b7fe700 1 --2- 172.21.15.162:0/2627246520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c0f8500 0x7f5c4c10e220 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f5c3c0048f0 tx=0x7f5c3c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:39.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c48ff9700 10 client.?.objecter ms_handle_connect 0x7f5c4c0f8500 2024-02-13T00:39:39.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c48ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:39.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c48ff9700 1 -- 172.21.15.162:0/2627246520 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c007b00 con 0x7f5c4c0f8500 2024-02-13T00:39:39.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c48ff9700 1 -- 172.21.15.162:0/2627246520 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c3c00f040 con 0x7f5c4c0f8500 2024-02-13T00:39:39.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c51fb2700 1 -- 172.21.15.162:0/2627246520 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c40005040 con 0x7f5c4c0f8500 2024-02-13T00:39:39.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c48ff9700 1 -- 172.21.15.162:0/2627246520 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c014990 con 0x7f5c4c0f8500 2024-02-13T00:39:39.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c51fb2700 10 client.14727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:39.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c51fb2700 1 -- 172.21.15.162:0/2627246520 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4c109270 con 0x7f5c4c0f8500 2024-02-13T00:39:39.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.210+0000 7f5c51fb2700 1 librados: init done 2024-02-13T00:39:39.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.211+0000 7f5c51fb2700 1 -- 172.21.15.162:0/2627246520 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c4c0475e0 con 0x7f5c4c0f8500 2024-02-13T00:39:39.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.211+0000 7f5c48ff9700 1 -- 172.21.15.162:0/2627246520 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f5c3c010500 con 0x7f5c4c0f8500 2024-02-13T00:39:39.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.212+0000 7f5c48ff9700 1 --2- 172.21.15.162:0/2627246520 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5c34062c90 0x7f5c34065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:39.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.212+0000 7f5c48ff9700 1 -- 172.21.15.162:0/2627246520 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(85..85 src has 30..85) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f5c3c07c5a0 con 0x7f5c4c0f8500 2024-02-13T00:39:39.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.212+0000 7f5c48ff9700 10 client.14727.objecter ms_dispatch 0x7f5c4c001550 osd_map(85..85 src has 30..85) v4 2024-02-13T00:39:39.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.212+0000 7f5c48ff9700 3 client.14727.objecter handle_osd_map got epochs [85,85] > 0 2024-02-13T00:39:39.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.212+0000 7f5c4affd700 1 --2- 172.21.15.162:0/2627246520 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5c34062c90 0x7f5c34065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:39.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.212+0000 7f5c48ff9700 3 client.14727.objecter handle_osd_map decoding full epoch 85 2024-02-13T00:39:39.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.212+0000 7f5c48ff9700 20 client.14727.objecter dump_active .. 0 homeless 2024-02-13T00:39:39.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.212+0000 7f5c48ff9700 10 client.14727.objecter ms_handle_connect 0x7f5c34062c90 2024-02-13T00:39:39.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.212+0000 7f5c4affd700 1 --2- 172.21.15.162:0/2627246520 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5c34062c90 0x7f5c34065150 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f5c40004ee0 tx=0x7f5c4000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:39.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.216+0000 7f5c48ff9700 1 -- 172.21.15.162:0/2627246520 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5c3c049f60 con 0x7f5c4c0f8500 2024-02-13T00:39:39.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:38.181003+0000 mgr.y (mgr.14544) 80 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:39:39.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:38.181003+0000 mgr.y (mgr.14544) 80 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:39:39.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.390+0000 7f5c51fb2700 1 -- 172.21.15.162:0/2627246520 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f5c4c1098a0 con 0x7f5c4c0f8500 2024-02-13T00:39:39.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.391+0000 7f5c48ff9700 1 -- 172.21.15.162:0/2627246520 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5c3c050210 con 0x7f5c4c0f8500 2024-02-13T00:39:39.394 INFO:teuthology.orchestra.run.smithi162.stdout:227633266713 2024-02-13T00:39:39.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.392+0000 7f5c51fb2700 10 librados: watch_flush enter 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.392+0000 7f5c51fb2700 10 librados: watch_flush exit 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.392+0000 7f5c51fb2700 20 client.14727.objecter shutdown clearing up homeless session... 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.392+0000 7f5c51fb2700 10 client.14727.objecter successfully canceled tick 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.392+0000 7f5c51fb2700 1 -- 172.21.15.162:0/2627246520 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5c34062c90 msgr2=0x7f5c34065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.392+0000 7f5c51fb2700 1 --2- 172.21.15.162:0/2627246520 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5c34062c90 0x7f5c34065150 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f5c40004ee0 tx=0x7f5c4000b040).stop 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.392+0000 7f5c51fb2700 1 -- 172.21.15.162:0/2627246520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c0f8500 msgr2=0x7f5c4c10e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.392+0000 7f5c51fb2700 1 --2- 172.21.15.162:0/2627246520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c0f8500 0x7f5c4c10e220 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f5c3c0048f0 tx=0x7f5c3c009cb0).stop 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.393+0000 7f5c51fb2700 1 -- 172.21.15.162:0/2627246520 shutdown_connections 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.393+0000 7f5c51fb2700 1 --2- 172.21.15.162:0/2627246520 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5c34062c90 0x7f5c34065150 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.393+0000 7f5c51fb2700 1 --2- 172.21.15.162:0/2627246520 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c4c0fa440 0x7f5c4c108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.393+0000 7f5c51fb2700 1 --2- 172.21.15.162:0/2627246520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c0f8500 0x7f5c4c10e220 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.393+0000 7f5c51fb2700 1 --2- 172.21.15.162:0/2627246520 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c4c0f9b70 0x7f5c4c10e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:39.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.393+0000 7f5c51fb2700 1 -- 172.21.15.162:0/2627246520 >> 172.21.15.162:0/2627246520 conn(0x7f5c4c0f38c0 msgr2=0x7f5c4c0f44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:39.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.393+0000 7f5c51fb2700 1 -- 172.21.15.162:0/2627246520 shutdown_connections 2024-02-13T00:39:39.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.393+0000 7f5c51fb2700 1 -- 172.21.15.162:0/2627246520 wait complete. 2024-02-13T00:39:39.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:39.393+0000 7f5c51fb2700 1 librados: shutdown 2024-02-13T00:39:39.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:38.181003+0000 mgr.y (mgr.14544) 80 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:39:40.277 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266707 got 227633266713 for osd.7 2024-02-13T00:39:40.277 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-13T00:39:40.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:39:40.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:39.391677+0000 mon.a (mon.0) 711 : audit [DBG] from='client.? 172.21.15.162:0/2627246520' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-13T00:39:40.550 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:40.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:39.391677+0000 mon.a (mon.0) 711 : audit [DBG] from='client.? 172.21.15.162:0/2627246520' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-13T00:39:40.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:39.391677+0000 mon.a (mon.0) 711 : audit [DBG] from='client.? 172.21.15.162:0/2627246520' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-13T00:39:41.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:40.181836+0000 mgr.y (mgr.14544) 81 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:39:41.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:40.277265+0000 osd.3 (osd.3) 45 : cluster [DBG] 2.1e deep-scrub starts 2024-02-13T00:39:41.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:40.280298+0000 osd.3 (osd.3) 46 : cluster [DBG] 2.1e deep-scrub ok 2024-02-13T00:39:41.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:40.181836+0000 mgr.y (mgr.14544) 81 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:39:41.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:40.277265+0000 osd.3 (osd.3) 45 : cluster [DBG] 2.1e deep-scrub starts 2024-02-13T00:39:41.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:40.280298+0000 osd.3 (osd.3) 46 : cluster [DBG] 2.1e deep-scrub ok 2024-02-13T00:39:41.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:40.181836+0000 mgr.y (mgr.14544) 81 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:39:41.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:40.277265+0000 osd.3 (osd.3) 45 : cluster [DBG] 2.1e deep-scrub starts 2024-02-13T00:39:41.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:40.280298+0000 osd.3 (osd.3) 46 : cluster [DBG] 2.1e deep-scrub ok 2024-02-13T00:39:42.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.332+0000 7f808c365700 1 -- 172.21.15.162:0/451093095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80840fcb30 msgr2=0x7f80840fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:42.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.332+0000 7f808c365700 1 --2- 172.21.15.162:0/451093095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80840fcb30 0x7f80840fcf10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f8080004870 tx=0x7f8080009cb0).stop 2024-02-13T00:39:42.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.332+0000 7f808c365700 1 -- 172.21.15.162:0/451093095 shutdown_connections 2024-02-13T00:39:42.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.332+0000 7f808c365700 1 --2- 172.21.15.162:0/451093095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80840f73a0 0x7f80841039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:39:42.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.332+0000 7f808c365700 1 --2- 172.21.15.162:0/451093095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80840fcb30 0x7f80840fcf10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:42.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.332+0000 7f808c365700 1 --2- 172.21.15.162:0/451093095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80840fd490 0x7f80840f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:42.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.332+0000 7f808c365700 1 -- 172.21.15.162:0/451093095 >> 172.21.15.162:0/451093095 conn(0x7f808406eb90 msgr2=0x7f80840f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:42.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.332+0000 7f808c365700 1 -- 172.21.15.162:0/451093095 shutdown_connections 2024-02-13T00:39:42.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.332+0000 7f808c365700 1 -- 172.21.15.162:0/451093095 wait complete. 2024-02-13T00:39:42.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808c365700 1 librados: starting msgr at 2024-02-13T00:39:42.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808c365700 1 librados: starting objecter 2024-02-13T00:39:42.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808c365700 1 Processor -- start 2024-02-13T00:39:42.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808c365700 1 -- start start 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808c365700 1 librados: setting wanted keys 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808c365700 1 librados: calling monclient init 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808c365700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80840f73a0 0x7f8084109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808c365700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80840fcb30 0x7f8084109800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808c365700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80840fd490 0x7f8084109dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808c365700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8084114500 con 0x7f80840fd490 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808c365700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8084114670 con 0x7f80840fcb30 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808c365700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8084114830 con 0x7f80840f73a0 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808a101700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80840f73a0 0x7f8084109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808a101700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80840f73a0 0x7f8084109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:44748/0 (socket says 172.21.15.162:44748) 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808a101700 1 -- 172.21.15.162:0/101676147 learned_addr learned my addr 172.21.15.162:0/101676147 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.333+0000 7f808a902700 1 --2- 172.21.15.162:0/101676147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80840fd490 0x7f8084109dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:42.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f8089900700 1 --2- 172.21.15.162:0/101676147 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80840fcb30 0x7f8084109800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f808a101700 1 -- 172.21.15.162:0/101676147 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80840fcb30 msgr2=0x7f8084109800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f808a101700 1 --2- 172.21.15.162:0/101676147 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80840fcb30 0x7f8084109800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f808a101700 1 -- 172.21.15.162:0/101676147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80840fd490 msgr2=0x7f8084109dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f808a101700 1 --2- 172.21.15.162:0/101676147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80840fd490 0x7f8084109dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f808a101700 1 -- 172.21.15.162:0/101676147 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8074005040 con 0x7f80840f73a0 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f8089900700 1 --2- 172.21.15.162:0/101676147 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80840fcb30 0x7f8084109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f808a101700 1 --2- 172.21.15.162:0/101676147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80840f73a0 0x7f8084109310 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f8080009eb0 tx=0x7f808000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f807b7fe700 10 client.?.objecter ms_handle_connect 0x7f80840f73a0 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f807b7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f807b7fe700 1 -- 172.21.15.162:0/101676147 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808000e070 con 0x7f80840f73a0 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f807b7fe700 1 -- 172.21.15.162:0/101676147 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8080013da0 con 0x7f80840f73a0 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f808c365700 1 -- 172.21.15.162:0/101676147 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8080005040 con 0x7f80840f73a0 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f807b7fe700 1 -- 172.21.15.162:0/101676147 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080019070 con 0x7f80840f73a0 2024-02-13T00:39:42.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f808c365700 10 client.24632.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f808c365700 1 -- 172.21.15.162:0/101676147 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8084114d80 con 0x7f80840f73a0 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.334+0000 7f808c365700 1 librados: init done 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.335+0000 7f808c365700 1 -- 172.21.15.162:0/101676147 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8084106020 con 0x7f80840f73a0 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.335+0000 7f807b7fe700 1 -- 172.21.15.162:0/101676147 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f808001e420 con 0x7f80840f73a0 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.336+0000 7f807b7fe700 1 --2- 172.21.15.162:0/101676147 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8070062c90 0x7f8070065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.336+0000 7f8089900700 1 --2- 172.21.15.162:0/101676147 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8070062c90 0x7f8070065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.336+0000 7f807b7fe700 1 -- 172.21.15.162:0/101676147 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(85..85 src has 30..85) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f808007d540 con 0x7f80840f73a0 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.336+0000 7f807b7fe700 10 client.24632.objecter ms_dispatch 0x7f8084001550 osd_map(85..85 src has 30..85) v4 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.336+0000 7f807b7fe700 3 client.24632.objecter handle_osd_map got epochs [85,85] > 0 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.336+0000 7f807b7fe700 3 client.24632.objecter handle_osd_map decoding full epoch 85 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.336+0000 7f807b7fe700 20 client.24632.objecter dump_active .. 0 homeless 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.336+0000 7f807b7fe700 10 client.24632.objecter ms_handle_connect 0x7f8070062c90 2024-02-13T00:39:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.336+0000 7f8089900700 1 --2- 172.21.15.162:0/101676147 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8070062c90 0x7f8070065150 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f8074007fd0 tx=0x7f8074009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:42.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.340+0000 7f807b7fe700 1 -- 172.21.15.162:0/101676147 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f80800511b0 con 0x7f80840f73a0 2024-02-13T00:39:42.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.500+0000 7f808c365700 1 -- 172.21.15.162:0/101676147 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f808405fc70 con 0x7f8070062c90 2024-02-13T00:39:42.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.510+0000 7f807b7fe700 1 -- 172.21.15.162:0/101676147 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357119 (secure 0 0 0) 0x7f808405fc70 con 0x7f8070062c90 2024-02-13T00:39:42.518 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:39:42.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 10 librados: watch_flush enter 2024-02-13T00:39:42.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 10 librados: watch_flush exit 2024-02-13T00:39:42.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 20 client.24632.objecter shutdown clearing up homeless session... 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 10 client.24632.objecter successfully canceled tick 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 -- 172.21.15.162:0/101676147 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8070062c90 msgr2=0x7f8070065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 --2- 172.21.15.162:0/101676147 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8070062c90 0x7f8070065150 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f8074007fd0 tx=0x7f8074009cb0).stop 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 -- 172.21.15.162:0/101676147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80840f73a0 msgr2=0x7f8084109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 --2- 172.21.15.162:0/101676147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80840f73a0 0x7f8084109310 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f8080009eb0 tx=0x7f808000b040).stop 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 -- 172.21.15.162:0/101676147 shutdown_connections 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 --2- 172.21.15.162:0/101676147 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8070062c90 0x7f8070065150 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 --2- 172.21.15.162:0/101676147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80840fd490 0x7f8084109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 --2- 172.21.15.162:0/101676147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f80840f73a0 0x7f8084109310 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 --2- 172.21.15.162:0/101676147 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80840fcb30 0x7f8084109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 -- 172.21.15.162:0/101676147 >> 172.21.15.162:0/101676147 conn(0x7f808406eb90 msgr2=0x7f80840fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 -- 172.21.15.162:0/101676147 shutdown_connections 2024-02-13T00:39:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 -- 172.21.15.162:0/101676147 wait complete. 2024-02-13T00:39:42.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:42.513+0000 7f808c365700 1 librados: shutdown 2024-02-13T00:39:42.523 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:39:42.535 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:41.228187+0000 osd.3 (osd.3) 47 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:39:42.536 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:41.231237+0000 osd.3 (osd.3) 48 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:39:42.536 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:41.254819+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 deep-scrub starts 2024-02-13T00:39:42.536 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:41.255779+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 deep-scrub ok 2024-02-13T00:39:42.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:41.228187+0000 osd.3 (osd.3) 47 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:39:42.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:41.231237+0000 osd.3 (osd.3) 48 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:39:42.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:41.254819+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 deep-scrub starts 2024-02-13T00:39:42.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:41.255779+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 deep-scrub ok 2024-02-13T00:39:42.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:41.228187+0000 osd.3 (osd.3) 47 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:39:42.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:41.231237+0000 osd.3 (osd.3) 48 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:39:42.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:41.254819+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 deep-scrub starts 2024-02-13T00:39:42.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:41.255779+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 deep-scrub ok 2024-02-13T00:39:43.351 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-02-13T00:39:42.182012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1216,"num_read_kb":936,"num_write":620,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":716,"ondisk_log_size":716,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381848,"kb_used_data":5080,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579368,"statfs":{"total":767960285184,"available":765521272832,"internally_reserved":0,"allocated":5201920,"data_stored":2376237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003705"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:18.311814+0000","last_change":"2024-02-13T00:39:18.311781+0000","last_active":"2024-02-13T00:39:18.311814+0000","last_peered":"2024-02-13T00:39:18.311814+0000","last_clean":"2024-02-13T00:39:18.311814+0000","last_became_active":"2024-02-13T00:37:47.552890+0000","last_became_peered":"2024-02-13T00:37:47.552890+0000","last_unstale":"2024-02-13T00:39:18.311814+0000","last_undegraded":"2024-02-13T00:39:18.311814+0000","last_fullsized":"2024-02-13T00:39:18.311814+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.311669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.311669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.273436+0000","last_change":"2024-02-13T00:39:23.273409+0000","last_active":"2024-02-13T00:39:23.273436+0000","last_peered":"2024-02-13T00:39:23.273436+0000","last_clean":"2024-02-13T00:39:23.273436+0000","last_became_active":"2024-02-13T00:37:45.245133+0000","last_became_peered":"2024-02-13T00:37:45.245133+0000","last_unstale":"2024-02-13T00:39:23.273436+0000","last_undegraded":"2024-02-13T00:39:23.273436+0000","last_fullsized":"2024-02-13T00:39:23.273436+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.273346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:32.313930+0000","last_change":"2024-02-13T00:39:32.313930+0000","last_active":"2024-02-13T00:39:32.313930+0000","last_peered":"2024-02-13T00:39:32.313930+0000","last_clean":"2024-02-13T00:39:32.313930+0000","last_became_active":"2024-02-13T00:37:41.220583+0000","last_became_peered":"2024-02-13T00:37:41.220583+0000","last_unstale":"2024-02-13T00:39:32.313930+0000","last_undegraded":"2024-02-13T00:39:32.313930+0000","last_fullsized":"2024-02-13T00:39:32.313930+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.313841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'39","reported_seq":124,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239928+0000","last_change":"2024-02-13T00:38:47.210073+0000","last_active":"2024-02-13T00:39:12.239928+0000","last_peered":"2024-02-13T00:39:12.239928+0000","last_clean":"2024-02-13T00:39:12.239928+0000","last_became_active":"2024-02-13T00:37:43.228728+0000","last_became_peered":"2024-02-13T00:37:43.228728+0000","last_unstale":"2024-02-13T00:39:12.239928+0000","last_undegraded":"2024-02-13T00:39:12.239928+0000","last_fullsized":"2024-02-13T00:39:12.239928+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'39","last_scrub_stamp":"2024-02-13T00:38:47.210000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:47.210000+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:37.210485+0000","last_change":"2024-02-13T00:39:37.210485+0000","last_active":"2024-02-13T00:39:37.210485+0000","last_peered":"2024-02-13T00:39:37.210485+0000","last_clean":"2024-02-13T00:39:37.210485+0000","last_became_active":"2024-02-13T00:37:47.259555+0000","last_became_peered":"2024-02-13T00:37:47.259555+0000","last_unstale":"2024-02-13T00:39:37.210485+0000","last_undegraded":"2024-02-13T00:39:37.210485+0000","last_fullsized":"2024-02-13T00:39:37.210485+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:37.210406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:37.210406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.332217+0000","last_change":"2024-02-13T00:39:24.332217+0000","last_active":"2024-02-13T00:39:24.332217+0000","last_peered":"2024-02-13T00:39:24.332217+0000","last_clean":"2024-02-13T00:39:24.332217+0000","last_became_active":"2024-02-13T00:37:45.249914+0000","last_became_peered":"2024-02-13T00:37:45.249914+0000","last_unstale":"2024-02-13T00:39:24.332217+0000","last_undegraded":"2024-02-13T00:39:24.332217+0000","last_fullsized":"2024-02-13T00:39:24.332217+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:24.332133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.332133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239919+0000","last_change":"2024-02-13T00:37:41.217193+0000","last_active":"2024-02-13T00:39:12.239919+0000","last_peered":"2024-02-13T00:39:12.239919+0000","last_clean":"2024-02-13T00:39:12.239919+0000","last_became_active":"2024-02-13T00:37:41.216896+0000","last_became_peered":"2024-02-13T00:37:41.216896+0000","last_unstale":"2024-02-13T00:39:12.239919+0000","last_undegraded":"2024-02-13T00:39:12.239919+0000","last_fullsized":"2024-02-13T00:39:12.239919+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:37:40.201175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028519+0000","last_change":"2024-02-13T00:37:43.223162+0000","last_active":"2024-02-13T00:39:13.028519+0000","last_peered":"2024-02-13T00:39:13.028519+0000","last_clean":"2024-02-13T00:39:13.028519+0000","last_became_active":"2024-02-13T00:37:43.222849+0000","last_became_peered":"2024-02-13T00:37:43.222849+0000","last_unstale":"2024-02-13T00:39:13.028519+0000","last_undegraded":"2024-02-13T00:39:13.028519+0000","last_fullsized":"2024-02-13T00:39:13.028519+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239660+0000","last_change":"2024-02-13T00:38:53.600488+0000","last_active":"2024-02-13T00:39:12.239660+0000","last_peered":"2024-02-13T00:39:12.239660+0000","last_clean":"2024-02-13T00:39:12.239660+0000","last_became_active":"2024-02-13T00:37:47.260025+0000","last_became_peered":"2024-02-13T00:37:47.260025+0000","last_unstale":"2024-02-13T00:39:12.239660+0000","last_undegraded":"2024-02-13T00:39:12.239660+0000","last_fullsized":"2024-02-13T00:39:12.239660+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:38:53.600425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.026600+0000","last_change":"2024-02-13T00:38:52.859190+0000","last_active":"2024-02-13T00:39:13.026600+0000","last_peered":"2024-02-13T00:39:13.026600+0000","last_clean":"2024-02-13T00:39:13.026600+0000","last_became_active":"2024-02-13T00:37:45.246184+0000","last_became_peered":"2024-02-13T00:37:45.246184+0000","last_unstale":"2024-02-13T00:39:13.026600+0000","last_undegraded":"2024-02-13T00:39:13.026600+0000","last_fullsized":"2024-02-13T00:39:13.026600+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.859089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.859089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:21.165337+0000","last_change":"2024-02-13T00:39:21.165313+0000","last_active":"2024-02-13T00:39:21.165337+0000","last_peered":"2024-02-13T00:39:21.165337+0000","last_clean":"2024-02-13T00:39:21.165337+0000","last_became_active":"2024-02-13T00:37:41.221025+0000","last_became_peered":"2024-02-13T00:37:41.221025+0000","last_unstale":"2024-02-13T00:39:21.165337+0000","last_undegraded":"2024-02-13T00:39:21.165337+0000","last_fullsized":"2024-02-13T00:39:21.165337+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.165246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'11","reported_seq":59,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028619+0000","last_change":"2024-02-13T00:37:43.230514+0000","last_active":"2024-02-13T00:39:13.028619+0000","last_peered":"2024-02-13T00:39:13.028619+0000","last_clean":"2024-02-13T00:39:13.028619+0000","last_became_active":"2024-02-13T00:37:43.230223+0000","last_became_peered":"2024-02-13T00:37:43.230223+0000","last_unstale":"2024-02-13T00:39:13.028619+0000","last_undegraded":"2024-02-13T00:39:13.028619+0000","last_fullsized":"2024-02-13T00:39:13.028619+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:22.338171+0000","last_change":"2024-02-13T00:39:22.338139+0000","last_active":"2024-02-13T00:39:22.338171+0000","last_peered":"2024-02-13T00:39:22.338171+0000","last_clean":"2024-02-13T00:39:22.338171+0000","last_became_active":"2024-02-13T00:37:47.259688+0000","last_became_peered":"2024-02-13T00:37:47.259688+0000","last_unstale":"2024-02-13T00:39:22.338171+0000","last_undegraded":"2024-02-13T00:39:22.338171+0000","last_fullsized":"2024-02-13T00:39:22.338171+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.338042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240052+0000","last_change":"2024-02-13T00:39:06.581693+0000","last_active":"2024-02-13T00:39:12.240052+0000","last_peered":"2024-02-13T00:39:12.240052+0000","last_clean":"2024-02-13T00:39:12.240052+0000","last_became_active":"2024-02-13T00:37:45.243961+0000","last_became_peered":"2024-02-13T00:37:45.243961+0000","last_unstale":"2024-02-13T00:39:12.240052+0000","last_undegraded":"2024-02-13T00:39:12.240052+0000","last_fullsized":"2024-02-13T00:39:12.240052+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:06.581608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:06.581608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.026764+0000","last_change":"2024-02-13T00:38:45.896411+0000","last_active":"2024-02-13T00:39:13.026764+0000","last_peered":"2024-02-13T00:39:13.026764+0000","last_clean":"2024-02-13T00:39:13.026764+0000","last_became_active":"2024-02-13T00:37:41.222506+0000","last_became_peered":"2024-02-13T00:37:41.222506+0000","last_unstale":"2024-02-13T00:39:13.026764+0000","last_undegraded":"2024-02-13T00:39:13.026764+0000","last_fullsized":"2024-02-13T00:39:13.026764+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:45.896343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:45.896343+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":85,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.213153+0000","last_change":"2024-02-13T00:39:34.213153+0000","last_active":"2024-02-13T00:39:34.213153+0000","last_peered":"2024-02-13T00:39:34.213153+0000","last_clean":"2024-02-13T00:39:34.213153+0000","last_became_active":"2024-02-13T00:37:43.228966+0000","last_became_peered":"2024-02-13T00:37:43.228966+0000","last_unstale":"2024-02-13T00:39:34.213153+0000","last_undegraded":"2024-02-13T00:39:34.213153+0000","last_fullsized":"2024-02-13T00:39:34.213153+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:39:34.213074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.213074+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239640+0000","last_change":"2024-02-13T00:39:02.223894+0000","last_active":"2024-02-13T00:39:12.239640+0000","last_peered":"2024-02-13T00:39:12.239640+0000","last_clean":"2024-02-13T00:39:12.239640+0000","last_became_active":"2024-02-13T00:37:47.553853+0000","last_became_peered":"2024-02-13T00:37:47.553853+0000","last_unstale":"2024-02-13T00:39:12.239640+0000","last_undegraded":"2024-02-13T00:39:12.239640+0000","last_fullsized":"2024-02-13T00:39:12.239640+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:02.223823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.223823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:21.301368+0000","last_change":"2024-02-13T00:39:21.301346+0000","last_active":"2024-02-13T00:39:21.301368+0000","last_peered":"2024-02-13T00:39:21.301368+0000","last_clean":"2024-02-13T00:39:21.301368+0000","last_became_active":"2024-02-13T00:37:45.242917+0000","last_became_peered":"2024-02-13T00:37:45.242917+0000","last_unstale":"2024-02-13T00:39:21.301368+0000","last_undegraded":"2024-02-13T00:39:21.301368+0000","last_fullsized":"2024-02-13T00:39:21.301368+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.301282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239489+0000","last_change":"2024-02-13T00:39:00.215830+0000","last_active":"2024-02-13T00:39:12.239489+0000","last_peered":"2024-02-13T00:39:12.239489+0000","last_clean":"2024-02-13T00:39:12.239489+0000","last_became_active":"2024-02-13T00:37:41.218333+0000","last_became_peered":"2024-02-13T00:37:41.218333+0000","last_unstale":"2024-02-13T00:39:12.239489+0000","last_undegraded":"2024-02-13T00:39:12.239489+0000","last_fullsized":"2024-02-13T00:39:12.239489+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.215755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.215755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":77,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:26.214150+0000","last_change":"2024-02-13T00:39:26.214150+0000","last_active":"2024-02-13T00:39:26.214150+0000","last_peered":"2024-02-13T00:39:26.214150+0000","last_clean":"2024-02-13T00:39:26.214150+0000","last_became_active":"2024-02-13T00:37:43.236026+0000","last_became_peered":"2024-02-13T00:37:43.236026+0000","last_unstale":"2024-02-13T00:39:26.214150+0000","last_undegraded":"2024-02-13T00:39:26.214150+0000","last_fullsized":"2024-02-13T00:39:26.214150+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:39:26.214084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:26.214084+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.029224+0000","last_change":"2024-02-13T00:37:47.552844+0000","last_active":"2024-02-13T00:39:13.029224+0000","last_peered":"2024-02-13T00:39:13.029224+0000","last_clean":"2024-02-13T00:39:13.029224+0000","last_became_active":"2024-02-13T00:37:47.552547+0000","last_became_peered":"2024-02-13T00:37:47.552547+0000","last_unstale":"2024-02-13T00:39:13.029224+0000","last_undegraded":"2024-02-13T00:39:13.029224+0000","last_fullsized":"2024-02-13T00:39:13.029224+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:37:46.228444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:30.245586+0000","last_change":"2024-02-13T00:39:30.245586+0000","last_active":"2024-02-13T00:39:30.245586+0000","last_peered":"2024-02-13T00:39:30.245586+0000","last_clean":"2024-02-13T00:39:30.245586+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:39:30.245586+0000","last_undegraded":"2024-02-13T00:39:30.245586+0000","last_fullsized":"2024-02-13T00:39:30.245586+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:30.245508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.245508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.684215+0000","last_change":"2024-02-13T00:39:10.364972+0000","last_active":"2024-02-13T00:39:12.684215+0000","last_peered":"2024-02-13T00:39:12.684215+0000","last_clean":"2024-02-13T00:39:12.684215+0000","last_became_active":"2024-02-13T00:37:41.218334+0000","last_became_peered":"2024-02-13T00:37:41.218334+0000","last_unstale":"2024-02-13T00:39:12.684215+0000","last_undegraded":"2024-02-13T00:39:12.684215+0000","last_fullsized":"2024-02-13T00:39:12.684215+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:10.364865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.364865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115735+0000","last_change":"2024-02-13T00:39:08.582559+0000","last_active":"2024-02-13T00:39:13.115735+0000","last_peered":"2024-02-13T00:39:13.115735+0000","last_clean":"2024-02-13T00:39:13.115735+0000","last_became_active":"2024-02-13T00:37:43.232938+0000","last_became_peered":"2024-02-13T00:37:43.232938+0000","last_unstale":"2024-02-13T00:39:13.115735+0000","last_undegraded":"2024-02-13T00:39:13.115735+0000","last_fullsized":"2024-02-13T00:39:13.115735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:39:08.582479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.582479+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115363+0000","last_change":"2024-02-13T00:39:00.452185+0000","last_active":"2024-02-13T00:39:13.115363+0000","last_peered":"2024-02-13T00:39:13.115363+0000","last_clean":"2024-02-13T00:39:13.115363+0000","last_became_active":"2024-02-13T00:37:47.260182+0000","last_became_peered":"2024-02-13T00:37:47.260182+0000","last_unstale":"2024-02-13T00:39:13.115363+0000","last_undegraded":"2024-02-13T00:39:13.115363+0000","last_fullsized":"2024-02-13T00:39:13.115363+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.452084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.452084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:15.328059+0000","last_change":"2024-02-13T00:39:15.328037+0000","last_active":"2024-02-13T00:39:15.328059+0000","last_peered":"2024-02-13T00:39:15.328059+0000","last_clean":"2024-02-13T00:39:15.328059+0000","last_became_active":"2024-02-13T00:37:45.249064+0000","last_became_peered":"2024-02-13T00:37:45.249064+0000","last_unstale":"2024-02-13T00:39:15.328059+0000","last_undegraded":"2024-02-13T00:39:15.328059+0000","last_fullsized":"2024-02-13T00:39:15.328059+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.327971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.327971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239811+0000","last_change":"2024-02-13T00:37:41.216878+0000","last_active":"2024-02-13T00:39:12.239811+0000","last_peered":"2024-02-13T00:39:12.239811+0000","last_clean":"2024-02-13T00:39:12.239811+0000","last_became_active":"2024-02-13T00:37:41.216574+0000","last_became_peered":"2024-02-13T00:37:41.216574+0000","last_unstale":"2024-02-13T00:39:12.239811+0000","last_undegraded":"2024-02-13T00:39:12.239811+0000","last_fullsized":"2024-02-13T00:39:12.239811+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:37:40.201175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":74,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.684608+0000","last_change":"2024-02-13T00:38:58.341512+0000","last_active":"2024-02-13T00:39:12.684608+0000","last_peered":"2024-02-13T00:39:12.684608+0000","last_clean":"2024-02-13T00:39:12.684608+0000","last_became_active":"2024-02-13T00:37:43.230491+0000","last_became_peered":"2024-02-13T00:37:43.230491+0000","last_unstale":"2024-02-13T00:39:12.684608+0000","last_undegraded":"2024-02-13T00:39:12.684608+0000","last_fullsized":"2024-02-13T00:39:12.684608+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:38:58.341406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.341406+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.027218+0000","last_change":"2024-02-13T00:38:50.866411+0000","last_active":"2024-02-13T00:39:13.027218+0000","last_peered":"2024-02-13T00:39:13.027218+0000","last_clean":"2024-02-13T00:39:13.027218+0000","last_became_active":"2024-02-13T00:37:47.254586+0000","last_became_peered":"2024-02-13T00:37:47.254586+0000","last_unstale":"2024-02-13T00:39:13.027218+0000","last_undegraded":"2024-02-13T00:39:13.027218+0000","last_fullsized":"2024-02-13T00:39:13.027218+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:38:50.866323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:50.866323+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"64'8","reported_seq":73,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233398+0000","last_change":"2024-02-13T00:39:02.452828+0000","last_active":"2024-02-13T00:39:34.233398+0000","last_peered":"2024-02-13T00:39:34.233398+0000","last_clean":"2024-02-13T00:39:34.233398+0000","last_became_active":"2024-02-13T00:37:45.249902+0000","last_became_peered":"2024-02-13T00:37:45.249902+0000","last_unstale":"2024-02-13T00:39:34.233398+0000","last_undegraded":"2024-02-13T00:39:34.233398+0000","last_fullsized":"2024-02-13T00:39:34.233398+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:02.452755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.452755+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239588+0000","last_change":"2024-02-13T00:38:49.606456+0000","last_active":"2024-02-13T00:39:12.239588+0000","last_peered":"2024-02-13T00:39:12.239588+0000","last_clean":"2024-02-13T00:39:12.239588+0000","last_became_active":"2024-02-13T00:37:41.214965+0000","last_became_peered":"2024-02-13T00:37:41.214965+0000","last_unstale":"2024-02-13T00:39:12.239588+0000","last_undegraded":"2024-02-13T00:39:12.239588+0000","last_fullsized":"2024-02-13T00:39:12.239588+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:49.606357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:49.606357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":75,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.242507+0000","last_change":"2024-02-13T00:38:52.380430+0000","last_active":"2024-02-13T00:39:12.242507+0000","last_peered":"2024-02-13T00:39:12.242507+0000","last_clean":"2024-02-13T00:39:12.242507+0000","last_became_active":"2024-02-13T00:37:43.324221+0000","last_became_peered":"2024-02-13T00:37:43.324221+0000","last_unstale":"2024-02-13T00:39:12.242507+0000","last_undegraded":"2024-02-13T00:39:12.242507+0000","last_fullsized":"2024-02-13T00:39:12.242507+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.380369+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:21.243629+0000","last_change":"2024-02-13T00:39:21.243606+0000","last_active":"2024-02-13T00:39:21.243629+0000","last_peered":"2024-02-13T00:39:21.243629+0000","last_clean":"2024-02-13T00:39:21.243629+0000","last_became_active":"2024-02-13T00:37:47.552917+0000","last_became_peered":"2024-02-13T00:37:47.552917+0000","last_unstale":"2024-02-13T00:39:21.243629+0000","last_undegraded":"2024-02-13T00:39:21.243629+0000","last_fullsized":"2024-02-13T00:39:21.243629+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.243527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.027438+0000","last_change":"2024-02-13T00:39:13.027403+0000","last_active":"2024-02-13T00:39:13.027438+0000","last_peered":"2024-02-13T00:39:13.027438+0000","last_clean":"2024-02-13T00:39:13.027438+0000","last_became_active":"2024-02-13T00:37:45.248592+0000","last_became_peered":"2024-02-13T00:37:45.248592+0000","last_unstale":"2024-02-13T00:39:13.027438+0000","last_undegraded":"2024-02-13T00:39:13.027438+0000","last_fullsized":"2024-02-13T00:39:13.027438+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:13.027299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:13.027299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.684722+0000","last_change":"2024-02-13T00:39:07.358630+0000","last_active":"2024-02-13T00:39:12.684722+0000","last_peered":"2024-02-13T00:39:12.684722+0000","last_clean":"2024-02-13T00:39:12.684722+0000","last_became_active":"2024-02-13T00:37:41.220344+0000","last_became_peered":"2024-02-13T00:37:41.220344+0000","last_unstale":"2024-02-13T00:39:12.684722+0000","last_undegraded":"2024-02-13T00:39:12.684722+0000","last_fullsized":"2024-02-13T00:39:12.684722+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.358534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.358534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":73,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.204969+0000","last_change":"2024-02-13T00:39:34.204969+0000","last_active":"2024-02-13T00:39:34.204969+0000","last_peered":"2024-02-13T00:39:34.204969+0000","last_clean":"2024-02-13T00:39:34.204969+0000","last_became_active":"2024-02-13T00:37:43.324196+0000","last_became_peered":"2024-02-13T00:37:43.324196+0000","last_unstale":"2024-02-13T00:39:34.204969+0000","last_undegraded":"2024-02-13T00:39:34.204969+0000","last_fullsized":"2024-02-13T00:39:34.204969+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:34.204888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.204888+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.026416+0000","last_change":"2024-02-13T00:38:57.818305+0000","last_active":"2024-02-13T00:39:13.026416+0000","last_peered":"2024-02-13T00:39:13.026416+0000","last_clean":"2024-02-13T00:39:13.026416+0000","last_became_active":"2024-02-13T00:37:47.254667+0000","last_became_peered":"2024-02-13T00:37:47.254667+0000","last_unstale":"2024-02-13T00:39:13.026416+0000","last_undegraded":"2024-02-13T00:39:13.026416+0000","last_fullsized":"2024-02-13T00:39:13.026416+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:57.818202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.818202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.683998+0000","last_change":"2024-02-13T00:38:52.493308+0000","last_active":"2024-02-13T00:39:12.683998+0000","last_peered":"2024-02-13T00:39:12.683998+0000","last_clean":"2024-02-13T00:39:12.683998+0000","last_became_active":"2024-02-13T00:37:45.249556+0000","last_became_peered":"2024-02-13T00:37:45.249556+0000","last_unstale":"2024-02-13T00:39:12.683998+0000","last_undegraded":"2024-02-13T00:39:12.683998+0000","last_fullsized":"2024-02-13T00:39:12.683998+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.493237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.493237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240300+0000","last_change":"2024-02-13T00:38:55.573704+0000","last_active":"2024-02-13T00:39:12.240300+0000","last_peered":"2024-02-13T00:39:12.240300+0000","last_clean":"2024-02-13T00:39:12.240300+0000","last_became_active":"2024-02-13T00:37:41.213033+0000","last_became_peered":"2024-02-13T00:37:41.213033+0000","last_unstale":"2024-02-13T00:39:12.240300+0000","last_undegraded":"2024-02-13T00:39:12.240300+0000","last_fullsized":"2024-02-13T00:39:12.240300+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:55.573637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:55.573637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.231582+0000","last_change":"2024-02-13T00:39:23.231537+0000","last_active":"2024-02-13T00:39:23.231582+0000","last_peered":"2024-02-13T00:39:23.231582+0000","last_clean":"2024-02-13T00:39:23.231582+0000","last_became_active":"2024-02-13T00:37:43.234395+0000","last_became_peered":"2024-02-13T00:37:43.234395+0000","last_unstale":"2024-02-13T00:39:23.231582+0000","last_undegraded":"2024-02-13T00:39:23.231582+0000","last_fullsized":"2024-02-13T00:39:23.231582+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:39:23.231401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.231401+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239725+0000","last_change":"2024-02-13T00:39:01.197382+0000","last_active":"2024-02-13T00:39:12.239725+0000","last_peered":"2024-02-13T00:39:12.239725+0000","last_clean":"2024-02-13T00:39:12.239725+0000","last_became_active":"2024-02-13T00:37:47.254519+0000","last_became_peered":"2024-02-13T00:37:47.254519+0000","last_unstale":"2024-02-13T00:39:12.239725+0000","last_undegraded":"2024-02-13T00:39:12.239725+0000","last_fullsized":"2024-02-13T00:39:12.239725+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.197310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.197310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.116198+0000","last_change":"2024-02-13T00:38:58.512117+0000","last_active":"2024-02-13T00:39:13.116198+0000","last_peered":"2024-02-13T00:39:13.116198+0000","last_clean":"2024-02-13T00:39:13.116198+0000","last_became_active":"2024-02-13T00:37:45.247114+0000","last_became_peered":"2024-02-13T00:37:45.247114+0000","last_unstale":"2024-02-13T00:39:13.116198+0000","last_undegraded":"2024-02-13T00:39:13.116198+0000","last_fullsized":"2024-02-13T00:39:13.116198+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.512053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:27.297182+0000","last_change":"2024-02-13T00:39:27.297182+0000","last_active":"2024-02-13T00:39:27.297182+0000","last_peered":"2024-02-13T00:39:27.297182+0000","last_clean":"2024-02-13T00:39:27.297182+0000","last_became_active":"2024-02-13T00:37:41.215531+0000","last_became_peered":"2024-02-13T00:37:41.215531+0000","last_unstale":"2024-02-13T00:39:27.297182+0000","last_undegraded":"2024-02-13T00:39:27.297182+0000","last_fullsized":"2024-02-13T00:39:27.297182+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:27.297107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:27.297107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":84,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028714+0000","last_change":"2024-02-13T00:39:06.398068+0000","last_active":"2024-02-13T00:39:13.028714+0000","last_peered":"2024-02-13T00:39:13.028714+0000","last_clean":"2024-02-13T00:39:13.028714+0000","last_became_active":"2024-02-13T00:39:05.965554+0000","last_became_peered":"2024-02-13T00:39:05.965554+0000","last_unstale":"2024-02-13T00:39:13.028714+0000","last_undegraded":"2024-02-13T00:39:13.028714+0000","last_fullsized":"2024-02-13T00:39:13.028714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:39:03.189237+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:28.286635+0000","last_change":"2024-02-13T00:39:28.286635+0000","last_active":"2024-02-13T00:39:28.286635+0000","last_peered":"2024-02-13T00:39:28.286635+0000","last_clean":"2024-02-13T00:39:28.286635+0000","last_became_active":"2024-02-13T00:37:47.259792+0000","last_became_peered":"2024-02-13T00:37:47.259792+0000","last_unstale":"2024-02-13T00:39:28.286635+0000","last_undegraded":"2024-02-13T00:39:28.286635+0000","last_fullsized":"2024-02-13T00:39:28.286635+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.286574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.286574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:29.229760+0000","last_change":"2024-02-13T00:39:29.229760+0000","last_active":"2024-02-13T00:39:29.229760+0000","last_peered":"2024-02-13T00:39:29.229760+0000","last_clean":"2024-02-13T00:39:29.229760+0000","last_became_active":"2024-02-13T00:37:45.243594+0000","last_became_peered":"2024-02-13T00:37:45.243594+0000","last_unstale":"2024-02-13T00:39:29.229760+0000","last_undegraded":"2024-02-13T00:39:29.229760+0000","last_fullsized":"2024-02-13T00:39:29.229760+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.229676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.684249+0000","last_change":"2024-02-13T00:39:00.395451+0000","last_active":"2024-02-13T00:39:12.684249+0000","last_peered":"2024-02-13T00:39:12.684249+0000","last_clean":"2024-02-13T00:39:12.684249+0000","last_became_active":"2024-02-13T00:37:41.216390+0000","last_became_peered":"2024-02-13T00:37:41.216390+0000","last_unstale":"2024-02-13T00:39:12.684249+0000","last_undegraded":"2024-02-13T00:39:12.684249+0000","last_fullsized":"2024-02-13T00:39:12.684249+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.395381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.395381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":60,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.116148+0000","last_change":"2024-02-13T00:39:12.348547+0000","last_active":"2024-02-13T00:39:13.116148+0000","last_peered":"2024-02-13T00:39:13.116148+0000","last_clean":"2024-02-13T00:39:13.116148+0000","last_became_active":"2024-02-13T00:37:43.225347+0000","last_became_peered":"2024-02-13T00:37:43.225347+0000","last_unstale":"2024-02-13T00:39:13.116148+0000","last_undegraded":"2024-02-13T00:39:13.116148+0000","last_fullsized":"2024-02-13T00:39:13.116148+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:39:12.348480+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.029316+0000","last_change":"2024-02-13T00:37:47.254258+0000","last_active":"2024-02-13T00:39:13.029316+0000","last_peered":"2024-02-13T00:39:13.029316+0000","last_clean":"2024-02-13T00:39:13.029316+0000","last_became_active":"2024-02-13T00:37:47.253945+0000","last_became_peered":"2024-02-13T00:37:47.253945+0000","last_unstale":"2024-02-13T00:39:13.029316+0000","last_undegraded":"2024-02-13T00:39:13.029316+0000","last_fullsized":"2024-02-13T00:39:13.029316+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:37:46.228444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"84'9","reported_seq":73,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233379+0000","last_change":"2024-02-13T00:39:22.269192+0000","last_active":"2024-02-13T00:39:34.233379+0000","last_peered":"2024-02-13T00:39:34.233379+0000","last_clean":"2024-02-13T00:39:34.233379+0000","last_became_active":"2024-02-13T00:37:45.250598+0000","last_became_peered":"2024-02-13T00:37:45.250598+0000","last_unstale":"2024-02-13T00:39:34.233379+0000","last_undegraded":"2024-02-13T00:39:34.233379+0000","last_fullsized":"2024-02-13T00:39:34.233379+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.269104+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.242178+0000","last_change":"2024-02-13T00:38:58.314612+0000","last_active":"2024-02-13T00:39:12.242178+0000","last_peered":"2024-02-13T00:39:12.242178+0000","last_clean":"2024-02-13T00:39:12.242178+0000","last_became_active":"2024-02-13T00:37:41.217953+0000","last_became_peered":"2024-02-13T00:37:41.217953+0000","last_unstale":"2024-02-13T00:39:12.242178+0000","last_undegraded":"2024-02-13T00:39:12.242178+0000","last_fullsized":"2024-02-13T00:39:12.242178+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.314542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.314542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":71,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240366+0000","last_change":"2024-02-13T00:39:10.549128+0000","last_active":"2024-02-13T00:39:12.240366+0000","last_peered":"2024-02-13T00:39:12.240366+0000","last_clean":"2024-02-13T00:39:12.240366+0000","last_became_active":"2024-02-13T00:37:43.322357+0000","last_became_peered":"2024-02-13T00:37:43.322357+0000","last_unstale":"2024-02-13T00:39:12.240366+0000","last_undegraded":"2024-02-13T00:39:12.240366+0000","last_fullsized":"2024-02-13T00:39:12.240366+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:39:10.549045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.549045+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:21.344861+0000","last_change":"2024-02-13T00:39:21.344836+0000","last_active":"2024-02-13T00:39:21.344861+0000","last_peered":"2024-02-13T00:39:21.344861+0000","last_clean":"2024-02-13T00:39:21.344861+0000","last_became_active":"2024-02-13T00:37:47.252722+0000","last_became_peered":"2024-02-13T00:37:47.252722+0000","last_unstale":"2024-02-13T00:39:21.344861+0000","last_undegraded":"2024-02-13T00:39:21.344861+0000","last_fullsized":"2024-02-13T00:39:21.344861+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.344771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.344771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"64'8","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233394+0000","last_change":"2024-02-13T00:39:11.518468+0000","last_active":"2024-02-13T00:39:34.233394+0000","last_peered":"2024-02-13T00:39:34.233394+0000","last_clean":"2024-02-13T00:39:34.233394+0000","last_became_active":"2024-02-13T00:37:45.246757+0000","last_became_peered":"2024-02-13T00:37:45.246757+0000","last_unstale":"2024-02-13T00:39:34.233394+0000","last_undegraded":"2024-02-13T00:39:34.233394+0000","last_fullsized":"2024-02-13T00:39:34.233394+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:11.518359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:11.518359+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239760+0000","last_change":"2024-02-13T00:39:01.561276+0000","last_active":"2024-02-13T00:39:12.239760+0000","last_peered":"2024-02-13T00:39:12.239760+0000","last_clean":"2024-02-13T00:39:12.239760+0000","last_became_active":"2024-02-13T00:37:41.219309+0000","last_became_peered":"2024-02-13T00:37:41.219309+0000","last_unstale":"2024-02-13T00:39:12.239760+0000","last_undegraded":"2024-02-13T00:39:12.239760+0000","last_fullsized":"2024-02-13T00:39:12.239760+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.561157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.561157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":91,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239464+0000","last_change":"2024-02-13T00:38:46.169426+0000","last_active":"2024-02-13T00:39:12.239464+0000","last_peered":"2024-02-13T00:39:12.239464+0000","last_clean":"2024-02-13T00:39:12.239464+0000","last_became_active":"2024-02-13T00:37:43.324313+0000","last_became_peered":"2024-02-13T00:37:43.324313+0000","last_unstale":"2024-02-13T00:39:12.239464+0000","last_undegraded":"2024-02-13T00:39:12.239464+0000","last_fullsized":"2024-02-13T00:39:12.239464+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:38:46.169344+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:25.166852+0000","last_change":"2024-02-13T00:39:25.166852+0000","last_active":"2024-02-13T00:39:25.166852+0000","last_peered":"2024-02-13T00:39:25.166852+0000","last_clean":"2024-02-13T00:39:25.166852+0000","last_became_active":"2024-02-13T00:37:47.553612+0000","last_became_peered":"2024-02-13T00:37:47.553612+0000","last_unstale":"2024-02-13T00:39:25.166852+0000","last_undegraded":"2024-02-13T00:39:25.166852+0000","last_fullsized":"2024-02-13T00:39:25.166852+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:25.166777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.166777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240270+0000","last_change":"2024-02-13T00:39:07.556868+0000","last_active":"2024-02-13T00:39:12.240270+0000","last_peered":"2024-02-13T00:39:12.240270+0000","last_clean":"2024-02-13T00:39:12.240270+0000","last_became_active":"2024-02-13T00:37:45.246907+0000","last_became_peered":"2024-02-13T00:37:45.246907+0000","last_unstale":"2024-02-13T00:39:12.240270+0000","last_undegraded":"2024-02-13T00:39:12.240270+0000","last_fullsized":"2024-02-13T00:39:12.240270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.556772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.556772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.684024+0000","last_change":"2024-02-13T00:39:01.439623+0000","last_active":"2024-02-13T00:39:12.684024+0000","last_peered":"2024-02-13T00:39:12.684024+0000","last_clean":"2024-02-13T00:39:12.684024+0000","last_became_active":"2024-02-13T00:37:41.212866+0000","last_became_peered":"2024-02-13T00:37:41.212866+0000","last_unstale":"2024-02-13T00:39:12.684024+0000","last_undegraded":"2024-02-13T00:39:12.684024+0000","last_fullsized":"2024-02-13T00:39:12.684024+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.439513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":63,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:19.266831+0000","last_change":"2024-02-13T00:39:19.266801+0000","last_active":"2024-02-13T00:39:19.266831+0000","last_peered":"2024-02-13T00:39:19.266831+0000","last_clean":"2024-02-13T00:39:19.266831+0000","last_became_active":"2024-02-13T00:37:43.324510+0000","last_became_peered":"2024-02-13T00:37:43.324510+0000","last_unstale":"2024-02-13T00:39:19.266831+0000","last_undegraded":"2024-02-13T00:39:19.266831+0000","last_fullsized":"2024-02-13T00:39:19.266831+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:39:19.266729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:19.266729+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:22.615899+0000","last_change":"2024-02-13T00:39:22.615864+0000","last_active":"2024-02-13T00:39:22.615899+0000","last_peered":"2024-02-13T00:39:22.615899+0000","last_clean":"2024-02-13T00:39:22.615899+0000","last_became_active":"2024-02-13T00:37:47.254803+0000","last_became_peered":"2024-02-13T00:37:47.254803+0000","last_unstale":"2024-02-13T00:39:22.615899+0000","last_undegraded":"2024-02-13T00:39:22.615899+0000","last_fullsized":"2024-02-13T00:39:22.615899+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.615773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.615773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":32,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239687+0000","last_change":"2024-02-13T00:37:45.250983+0000","last_active":"2024-02-13T00:39:12.239687+0000","last_peered":"2024-02-13T00:39:12.239687+0000","last_clean":"2024-02-13T00:39:12.239687+0000","last_became_active":"2024-02-13T00:37:45.250632+0000","last_became_peered":"2024-02-13T00:37:45.250632+0000","last_unstale":"2024-02-13T00:39:12.239687+0000","last_undegraded":"2024-02-13T00:39:12.239687+0000","last_fullsized":"2024-02-13T00:39:12.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:37:44.215312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115163+0000","last_change":"2024-02-13T00:39:01.569173+0000","last_active":"2024-02-13T00:39:13.115163+0000","last_peered":"2024-02-13T00:39:13.115163+0000","last_clean":"2024-02-13T00:39:13.115163+0000","last_became_active":"2024-02-13T00:37:41.214091+0000","last_became_peered":"2024-02-13T00:37:41.214091+0000","last_unstale":"2024-02-13T00:39:13.115163+0000","last_undegraded":"2024-02-13T00:39:13.115163+0000","last_fullsized":"2024-02-13T00:39:13.115163+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.569109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.569109+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":85,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.242392+0000","last_change":"2024-02-13T00:38:56.331799+0000","last_active":"2024-02-13T00:39:12.242392+0000","last_peered":"2024-02-13T00:39:12.242392+0000","last_clean":"2024-02-13T00:39:12.242392+0000","last_became_active":"2024-02-13T00:37:43.322401+0000","last_became_peered":"2024-02-13T00:37:43.322401+0000","last_unstale":"2024-02-13T00:39:12.242392+0000","last_undegraded":"2024-02-13T00:39:12.242392+0000","last_fullsized":"2024-02-13T00:39:12.242392+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:38:56.331703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.331703+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:33.179309+0000","last_change":"2024-02-13T00:39:33.179309+0000","last_active":"2024-02-13T00:39:33.179309+0000","last_peered":"2024-02-13T00:39:33.179309+0000","last_clean":"2024-02-13T00:39:33.179309+0000","last_became_active":"2024-02-13T00:37:47.260242+0000","last_became_peered":"2024-02-13T00:37:47.260242+0000","last_unstale":"2024-02-13T00:39:33.179309+0000","last_undegraded":"2024-02-13T00:39:33.179309+0000","last_fullsized":"2024-02-13T00:39:33.179309+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.179225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.179225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.029231+0000","last_change":"2024-02-13T00:38:48.379316+0000","last_active":"2024-02-13T00:39:13.029231+0000","last_peered":"2024-02-13T00:39:13.029231+0000","last_clean":"2024-02-13T00:39:13.029231+0000","last_became_active":"2024-02-13T00:37:41.219249+0000","last_became_peered":"2024-02-13T00:37:41.219249+0000","last_unstale":"2024-02-13T00:39:13.029231+0000","last_undegraded":"2024-02-13T00:39:13.029231+0000","last_fullsized":"2024-02-13T00:39:13.029231+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:38:48.379184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:48.379184+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":87,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028724+0000","last_change":"2024-02-13T00:37:43.229199+0000","last_active":"2024-02-13T00:39:13.028724+0000","last_peered":"2024-02-13T00:39:13.028724+0000","last_clean":"2024-02-13T00:39:13.028724+0000","last_became_active":"2024-02-13T00:37:43.228898+0000","last_became_peered":"2024-02-13T00:37:43.228898+0000","last_unstale":"2024-02-13T00:39:13.028724+0000","last_undegraded":"2024-02-13T00:39:13.028724+0000","last_fullsized":"2024-02-13T00:39:13.028724+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:21.654867+0000","last_change":"2024-02-13T00:39:21.654834+0000","last_active":"2024-02-13T00:39:21.654867+0000","last_peered":"2024-02-13T00:39:21.654867+0000","last_clean":"2024-02-13T00:39:21.654867+0000","last_became_active":"2024-02-13T00:37:45.244408+0000","last_became_peered":"2024-02-13T00:37:45.244408+0000","last_unstale":"2024-02-13T00:39:21.654867+0000","last_undegraded":"2024-02-13T00:39:21.654867+0000","last_fullsized":"2024-02-13T00:39:21.654867+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.654733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.654733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.241786+0000","last_change":"2024-02-13T00:38:54.349240+0000","last_active":"2024-02-13T00:39:12.241786+0000","last_peered":"2024-02-13T00:39:12.241786+0000","last_clean":"2024-02-13T00:39:12.241786+0000","last_became_active":"2024-02-13T00:37:47.255928+0000","last_became_peered":"2024-02-13T00:37:47.255928+0000","last_unstale":"2024-02-13T00:39:12.241786+0000","last_undegraded":"2024-02-13T00:39:12.241786+0000","last_fullsized":"2024-02-13T00:39:12.241786+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:54.349171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:54.349171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.116015+0000","last_change":"2024-02-13T00:38:49.618099+0000","last_active":"2024-02-13T00:39:13.116015+0000","last_peered":"2024-02-13T00:39:13.116015+0000","last_clean":"2024-02-13T00:39:13.116015+0000","last_became_active":"2024-02-13T00:37:41.217777+0000","last_became_peered":"2024-02-13T00:37:41.217777+0000","last_unstale":"2024-02-13T00:39:13.116015+0000","last_undegraded":"2024-02-13T00:39:13.116015+0000","last_fullsized":"2024-02-13T00:39:13.116015+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:38:49.617988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115544+0000","last_change":"2024-02-13T00:39:05.974172+0000","last_active":"2024-02-13T00:39:13.115544+0000","last_peered":"2024-02-13T00:39:13.115544+0000","last_clean":"2024-02-13T00:39:13.115544+0000","last_became_active":"2024-02-13T00:39:05.965560+0000","last_became_peered":"2024-02-13T00:39:05.965560+0000","last_unstale":"2024-02-13T00:39:13.115544+0000","last_undegraded":"2024-02-13T00:39:13.115544+0000","last_fullsized":"2024-02-13T00:39:13.115544+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:39:01.456507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.456507+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"64'8","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233379+0000","last_change":"2024-02-13T00:39:30.273237+0000","last_active":"2024-02-13T00:39:34.233379+0000","last_peered":"2024-02-13T00:39:34.233379+0000","last_clean":"2024-02-13T00:39:34.233379+0000","last_became_active":"2024-02-13T00:37:45.241274+0000","last_became_peered":"2024-02-13T00:37:45.241274+0000","last_unstale":"2024-02-13T00:39:34.233379+0000","last_undegraded":"2024-02-13T00:39:34.233379+0000","last_fullsized":"2024-02-13T00:39:34.233379+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:30.273165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.273165+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:36.157777+0000","last_change":"2024-02-13T00:39:36.157777+0000","last_active":"2024-02-13T00:39:36.157777+0000","last_peered":"2024-02-13T00:39:36.157777+0000","last_clean":"2024-02-13T00:39:36.157777+0000","last_became_active":"2024-02-13T00:37:47.252270+0000","last_became_peered":"2024-02-13T00:37:47.252270+0000","last_unstale":"2024-02-13T00:39:36.157777+0000","last_undegraded":"2024-02-13T00:39:36.157777+0000","last_fullsized":"2024-02-13T00:39:36.157777+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:36.157693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:36.157693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239861+0000","last_change":"2024-02-13T00:38:58.528669+0000","last_active":"2024-02-13T00:39:12.239861+0000","last_peered":"2024-02-13T00:39:12.239861+0000","last_clean":"2024-02-13T00:39:12.239861+0000","last_became_active":"2024-02-13T00:37:41.212924+0000","last_became_peered":"2024-02-13T00:37:41.212924+0000","last_unstale":"2024-02-13T00:39:12.239861+0000","last_undegraded":"2024-02-13T00:39:12.239861+0000","last_fullsized":"2024-02-13T00:39:12.239861+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.528580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.528580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"80'45","reported_seq":138,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233366+0000","last_change":"2024-02-13T00:38:57.537373+0000","last_active":"2024-02-13T00:39:34.233366+0000","last_peered":"2024-02-13T00:39:34.233366+0000","last_clean":"2024-02-13T00:39:34.233366+0000","last_became_active":"2024-02-13T00:37:43.232014+0000","last_became_peered":"2024-02-13T00:37:43.232014+0000","last_unstale":"2024-02-13T00:39:34.233366+0000","last_undegraded":"2024-02-13T00:39:34.233366+0000","last_fullsized":"2024-02-13T00:39:34.233366+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'45","last_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_deep_scrub":"80'45","last_deep_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.537309+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.242412+0000","last_change":"2024-02-13T00:38:47.393545+0000","last_active":"2024-02-13T00:39:12.242412+0000","last_peered":"2024-02-13T00:39:12.242412+0000","last_clean":"2024-02-13T00:39:12.242412+0000","last_became_active":"2024-02-13T00:37:45.247012+0000","last_became_peered":"2024-02-13T00:37:45.247012+0000","last_unstale":"2024-02-13T00:39:12.242412+0000","last_undegraded":"2024-02-13T00:39:12.242412+0000","last_fullsized":"2024-02-13T00:39:12.242412+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:47.393477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:47.393477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.026985+0000","last_change":"2024-02-13T00:39:08.930351+0000","last_active":"2024-02-13T00:39:13.026985+0000","last_peered":"2024-02-13T00:39:13.026985+0000","last_clean":"2024-02-13T00:39:13.026985+0000","last_became_active":"2024-02-13T00:37:47.552693+0000","last_became_peered":"2024-02-13T00:37:47.552693+0000","last_unstale":"2024-02-13T00:39:13.026985+0000","last_undegraded":"2024-02-13T00:39:13.026985+0000","last_fullsized":"2024-02-13T00:39:13.026985+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:08.930286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.930286+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":159,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:38.515414+0000","last_change":"2024-02-13T00:39:10.287406+0000","last_active":"2024-02-13T00:39:38.515414+0000","last_peered":"2024-02-13T00:39:38.515414+0000","last_clean":"2024-02-13T00:39:38.515414+0000","last_became_active":"2024-02-13T00:37:41.215137+0000","last_became_peered":"2024-02-13T00:37:41.215137+0000","last_unstale":"2024-02-13T00:39:38.515414+0000","last_undegraded":"2024-02-13T00:39:38.515414+0000","last_fullsized":"2024-02-13T00:39:38.515414+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-13T00:39:10.287299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.287299+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:20.703811+0000","last_change":"2024-02-13T00:39:20.703787+0000","last_active":"2024-02-13T00:39:20.703811+0000","last_peered":"2024-02-13T00:39:20.703811+0000","last_clean":"2024-02-13T00:39:20.703811+0000","last_became_active":"2024-02-13T00:37:41.214442+0000","last_became_peered":"2024-02-13T00:37:41.214442+0000","last_unstale":"2024-02-13T00:39:20.703811+0000","last_undegraded":"2024-02-13T00:39:20.703811+0000","last_fullsized":"2024-02-13T00:39:20.703811+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:20.703700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:20.703700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":82,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:25.319173+0000","last_change":"2024-02-13T00:39:25.319173+0000","last_active":"2024-02-13T00:39:25.319173+0000","last_peered":"2024-02-13T00:39:25.319173+0000","last_clean":"2024-02-13T00:39:25.319173+0000","last_became_active":"2024-02-13T00:37:43.231120+0000","last_became_peered":"2024-02-13T00:37:43.231120+0000","last_unstale":"2024-02-13T00:39:25.319173+0000","last_undegraded":"2024-02-13T00:39:25.319173+0000","last_fullsized":"2024-02-13T00:39:25.319173+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:39:25.319066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.319066+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115331+0000","last_change":"2024-02-13T00:39:09.589876+0000","last_active":"2024-02-13T00:39:13.115331+0000","last_peered":"2024-02-13T00:39:13.115331+0000","last_clean":"2024-02-13T00:39:13.115331+0000","last_became_active":"2024-02-13T00:37:45.248383+0000","last_became_peered":"2024-02-13T00:37:45.248383+0000","last_unstale":"2024-02-13T00:39:13.115331+0000","last_undegraded":"2024-02-13T00:39:13.115331+0000","last_fullsized":"2024-02-13T00:39:13.115331+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:09.589813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:09.589813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240574+0000","last_change":"2024-02-13T00:39:05.596529+0000","last_active":"2024-02-13T00:39:12.240574+0000","last_peered":"2024-02-13T00:39:12.240574+0000","last_clean":"2024-02-13T00:39:12.240574+0000","last_became_active":"2024-02-13T00:37:47.253996+0000","last_became_peered":"2024-02-13T00:37:47.253996+0000","last_unstale":"2024-02-13T00:39:12.240574+0000","last_undegraded":"2024-02-13T00:39:12.240574+0000","last_fullsized":"2024-02-13T00:39:12.240574+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:39:05.596462+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239260+0000","last_change":"2024-02-13T00:39:04.221630+0000","last_active":"2024-02-13T00:39:12.239260+0000","last_peered":"2024-02-13T00:39:12.239260+0000","last_clean":"2024-02-13T00:39:12.239260+0000","last_became_active":"2024-02-13T00:37:41.216367+0000","last_became_peered":"2024-02-13T00:37:41.216367+0000","last_unstale":"2024-02-13T00:39:12.239260+0000","last_undegraded":"2024-02-13T00:39:12.239260+0000","last_fullsized":"2024-02-13T00:39:12.239260+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:39:04.221506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239887+0000","last_change":"2024-02-13T00:38:52.618768+0000","last_active":"2024-02-13T00:39:12.239887+0000","last_peered":"2024-02-13T00:39:12.239887+0000","last_clean":"2024-02-13T00:39:12.239887+0000","last_became_active":"2024-02-13T00:37:41.216101+0000","last_became_peered":"2024-02-13T00:37:41.216101+0000","last_unstale":"2024-02-13T00:39:12.239887+0000","last_undegraded":"2024-02-13T00:39:12.239887+0000","last_fullsized":"2024-02-13T00:39:12.239887+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.618696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.618696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.026331+0000","last_change":"2024-02-13T00:38:12.978952+0000","last_active":"2024-02-13T00:39:13.026331+0000","last_peered":"2024-02-13T00:39:13.026331+0000","last_clean":"2024-02-13T00:39:13.026331+0000","last_became_active":"2024-02-13T00:37:35.635369+0000","last_became_peered":"2024-02-13T00:37:35.635369+0000","last_unstale":"2024-02-13T00:39:13.026331+0000","last_undegraded":"2024-02-13T00:39:13.026331+0000","last_fullsized":"2024-02-13T00:39:13.026331+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:12.978865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:35:35.296696+0000","last_clean_scrub_stamp":"2024-02-13T00:38:12.978865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'14","reported_seq":62,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:15.415117+0000","last_change":"2024-02-13T00:39:15.415087+0000","last_active":"2024-02-13T00:39:15.415117+0000","last_peered":"2024-02-13T00:39:15.415117+0000","last_clean":"2024-02-13T00:39:15.415117+0000","last_became_active":"2024-02-13T00:37:43.324209+0000","last_became_peered":"2024-02-13T00:37:43.324209+0000","last_unstale":"2024-02-13T00:39:15.415117+0000","last_undegraded":"2024-02-13T00:39:15.415117+0000","last_fullsized":"2024-02-13T00:39:15.415117+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'14","last_scrub_stamp":"2024-02-13T00:39:15.415004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.415004+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:28.171808+0000","last_change":"2024-02-13T00:39:28.171808+0000","last_active":"2024-02-13T00:39:28.171808+0000","last_peered":"2024-02-13T00:39:28.171808+0000","last_clean":"2024-02-13T00:39:28.171808+0000","last_became_active":"2024-02-13T00:37:45.247708+0000","last_became_peered":"2024-02-13T00:37:45.247708+0000","last_unstale":"2024-02-13T00:39:28.171808+0000","last_undegraded":"2024-02-13T00:39:28.171808+0000","last_fullsized":"2024-02-13T00:39:28.171808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.171740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.171740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:32.255327+0000","last_change":"2024-02-13T00:39:32.255327+0000","last_active":"2024-02-13T00:39:32.255327+0000","last_peered":"2024-02-13T00:39:32.255327+0000","last_clean":"2024-02-13T00:39:32.255327+0000","last_became_active":"2024-02-13T00:37:47.262024+0000","last_became_peered":"2024-02-13T00:37:47.262024+0000","last_unstale":"2024-02-13T00:39:32.255327+0000","last_undegraded":"2024-02-13T00:39:32.255327+0000","last_fullsized":"2024-02-13T00:39:32.255327+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.255260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.241978+0000","last_change":"2024-02-13T00:38:55.353229+0000","last_active":"2024-02-13T00:39:12.241978+0000","last_peered":"2024-02-13T00:39:12.241978+0000","last_clean":"2024-02-13T00:39:12.241978+0000","last_became_active":"2024-02-13T00:37:41.217577+0000","last_became_peered":"2024-02-13T00:37:41.217577+0000","last_unstale":"2024-02-13T00:39:12.241978+0000","last_undegraded":"2024-02-13T00:39:12.241978+0000","last_fullsized":"2024-02-13T00:39:12.241978+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:55.353157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:55.353157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.114906+0000","last_change":"2024-02-13T00:39:05.968028+0000","last_active":"2024-02-13T00:39:13.114906+0000","last_peered":"2024-02-13T00:39:13.114906+0000","last_clean":"2024-02-13T00:39:13.114906+0000","last_became_active":"2024-02-13T00:39:05.967772+0000","last_became_peered":"2024-02-13T00:39:05.967772+0000","last_unstale":"2024-02-13T00:39:13.114906+0000","last_undegraded":"2024-02-13T00:39:13.114906+0000","last_fullsized":"2024-02-13T00:39:13.114906+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:44.893697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:44.893697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":80,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.242362+0000","last_change":"2024-02-13T00:39:08.344202+0000","last_active":"2024-02-13T00:39:12.242362+0000","last_peered":"2024-02-13T00:39:12.242362+0000","last_clean":"2024-02-13T00:39:12.242362+0000","last_became_active":"2024-02-13T00:39:05.968155+0000","last_became_peered":"2024-02-13T00:39:05.968155+0000","last_unstale":"2024-02-13T00:39:12.242362+0000","last_undegraded":"2024-02-13T00:39:12.242362+0000","last_fullsized":"2024-02-13T00:39:12.242362+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:39:08.344129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.344129+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240565+0000","last_change":"2024-02-13T00:38:48.618900+0000","last_active":"2024-02-13T00:39:12.240565+0000","last_peered":"2024-02-13T00:39:12.240565+0000","last_clean":"2024-02-13T00:39:12.240565+0000","last_became_active":"2024-02-13T00:37:45.245956+0000","last_became_peered":"2024-02-13T00:37:45.245956+0000","last_unstale":"2024-02-13T00:39:12.240565+0000","last_undegraded":"2024-02-13T00:39:12.240565+0000","last_fullsized":"2024-02-13T00:39:12.240565+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:48.618830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:48.618830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:33.304107+0000","last_change":"2024-02-13T00:39:33.304107+0000","last_active":"2024-02-13T00:39:33.304107+0000","last_peered":"2024-02-13T00:39:33.304107+0000","last_clean":"2024-02-13T00:39:33.304107+0000","last_became_active":"2024-02-13T00:37:47.552672+0000","last_became_peered":"2024-02-13T00:37:47.552672+0000","last_unstale":"2024-02-13T00:39:33.304107+0000","last_undegraded":"2024-02-13T00:39:33.304107+0000","last_fullsized":"2024-02-13T00:39:33.304107+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.303982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.303982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.027005+0000","last_change":"2024-02-13T00:38:46.927138+0000","last_active":"2024-02-13T00:39:13.027005+0000","last_peered":"2024-02-13T00:39:13.027005+0000","last_clean":"2024-02-13T00:39:13.027005+0000","last_became_active":"2024-02-13T00:37:41.221636+0000","last_became_peered":"2024-02-13T00:37:41.221636+0000","last_unstale":"2024-02-13T00:39:13.027005+0000","last_undegraded":"2024-02-13T00:39:13.027005+0000","last_fullsized":"2024-02-13T00:39:13.027005+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:46.927068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:46.927068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":99,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.242163+0000","last_change":"2024-02-13T00:39:04.318990+0000","last_active":"2024-02-13T00:39:12.242163+0000","last_peered":"2024-02-13T00:39:12.242163+0000","last_clean":"2024-02-13T00:39:12.242163+0000","last_became_active":"2024-02-13T00:37:43.322450+0000","last_became_peered":"2024-02-13T00:37:43.322450+0000","last_unstale":"2024-02-13T00:39:12.242163+0000","last_undegraded":"2024-02-13T00:39:12.242163+0000","last_fullsized":"2024-02-13T00:39:12.242163+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_clean_scrub_stamp":"2024-02-13T00:39:04.318923+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:20.451172+0000","last_change":"2024-02-13T00:39:20.451151+0000","last_active":"2024-02-13T00:39:20.451172+0000","last_peered":"2024-02-13T00:39:20.451172+0000","last_clean":"2024-02-13T00:39:20.451172+0000","last_became_active":"2024-02-13T00:37:45.245974+0000","last_became_peered":"2024-02-13T00:37:45.245974+0000","last_unstale":"2024-02-13T00:39:20.451172+0000","last_undegraded":"2024-02-13T00:39:20.451172+0000","last_fullsized":"2024-02-13T00:39:20.451172+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:20.451088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:20.451088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028568+0000","last_change":"2024-02-13T00:39:09.237388+0000","last_active":"2024-02-13T00:39:13.028568+0000","last_peered":"2024-02-13T00:39:13.028568+0000","last_clean":"2024-02-13T00:39:13.028568+0000","last_became_active":"2024-02-13T00:37:47.250291+0000","last_became_peered":"2024-02-13T00:37:47.250291+0000","last_unstale":"2024-02-13T00:39:13.028568+0000","last_undegraded":"2024-02-13T00:39:13.028568+0000","last_fullsized":"2024-02-13T00:39:13.028568+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:09.237312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:09.237312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:21.329271+0000","last_change":"2024-02-13T00:39:21.329248+0000","last_active":"2024-02-13T00:39:21.329271+0000","last_peered":"2024-02-13T00:39:21.329271+0000","last_clean":"2024-02-13T00:39:21.329271+0000","last_became_active":"2024-02-13T00:37:41.214202+0000","last_became_peered":"2024-02-13T00:37:41.214202+0000","last_unstale":"2024-02-13T00:39:21.329271+0000","last_undegraded":"2024-02-13T00:39:21.329271+0000","last_fullsized":"2024-02-13T00:39:21.329271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.329173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.329173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":101,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239164+0000","last_change":"2024-02-13T00:38:58.238608+0000","last_active":"2024-02-13T00:39:12.239164+0000","last_peered":"2024-02-13T00:39:12.239164+0000","last_clean":"2024-02-13T00:39:12.239164+0000","last_became_active":"2024-02-13T00:37:43.324275+0000","last_became_peered":"2024-02-13T00:37:43.324275+0000","last_unstale":"2024-02-13T00:39:12.239164+0000","last_undegraded":"2024-02-13T00:39:12.239164+0000","last_fullsized":"2024-02-13T00:39:12.239164+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:38:58.238526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.238526+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028541+0000","last_change":"2024-02-13T00:38:52.388152+0000","last_active":"2024-02-13T00:39:13.028541+0000","last_peered":"2024-02-13T00:39:13.028541+0000","last_clean":"2024-02-13T00:39:13.028541+0000","last_became_active":"2024-02-13T00:37:45.244203+0000","last_became_peered":"2024-02-13T00:37:45.244203+0000","last_unstale":"2024-02-13T00:39:13.028541+0000","last_undegraded":"2024-02-13T00:39:13.028541+0000","last_fullsized":"2024-02-13T00:39:13.028541+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.388093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.388093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.027224+0000","last_change":"2024-02-13T00:38:56.857630+0000","last_active":"2024-02-13T00:39:13.027224+0000","last_peered":"2024-02-13T00:39:13.027224+0000","last_clean":"2024-02-13T00:39:13.027224+0000","last_became_active":"2024-02-13T00:37:47.553554+0000","last_became_peered":"2024-02-13T00:37:47.553554+0000","last_unstale":"2024-02-13T00:39:13.027224+0000","last_undegraded":"2024-02-13T00:39:13.027224+0000","last_fullsized":"2024-02-13T00:39:13.027224+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:56.857534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.857534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:14.367248+0000","last_change":"2024-02-13T00:39:14.367203+0000","last_active":"2024-02-13T00:39:14.367248+0000","last_peered":"2024-02-13T00:39:14.367248+0000","last_clean":"2024-02-13T00:39:14.367248+0000","last_became_active":"2024-02-13T00:37:41.218313+0000","last_became_peered":"2024-02-13T00:37:41.218313+0000","last_unstale":"2024-02-13T00:39:14.367248+0000","last_undegraded":"2024-02-13T00:39:14.367248+0000","last_fullsized":"2024-02-13T00:39:14.367248+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:14.367115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:14.367115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028670+0000","last_change":"2024-02-13T00:37:43.222098+0000","last_active":"2024-02-13T00:39:13.028670+0000","last_peered":"2024-02-13T00:39:13.028670+0000","last_clean":"2024-02-13T00:39:13.028670+0000","last_became_active":"2024-02-13T00:37:43.221753+0000","last_became_peered":"2024-02-13T00:37:43.221753+0000","last_unstale":"2024-02-13T00:39:13.028670+0000","last_undegraded":"2024-02-13T00:39:13.028670+0000","last_fullsized":"2024-02-13T00:39:13.028670+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"64'8","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233394+0000","last_change":"2024-02-13T00:39:16.304400+0000","last_active":"2024-02-13T00:39:34.233394+0000","last_peered":"2024-02-13T00:39:34.233394+0000","last_clean":"2024-02-13T00:39:34.233394+0000","last_became_active":"2024-02-13T00:37:45.245528+0000","last_became_peered":"2024-02-13T00:37:45.245528+0000","last_unstale":"2024-02-13T00:39:34.233394+0000","last_undegraded":"2024-02-13T00:39:34.233394+0000","last_fullsized":"2024-02-13T00:39:34.233394+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:16.304316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.304316+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239289+0000","last_change":"2024-02-13T00:38:59.188989+0000","last_active":"2024-02-13T00:39:12.239289+0000","last_peered":"2024-02-13T00:39:12.239289+0000","last_clean":"2024-02-13T00:39:12.239289+0000","last_became_active":"2024-02-13T00:37:47.552591+0000","last_became_peered":"2024-02-13T00:37:47.552591+0000","last_unstale":"2024-02-13T00:39:12.239289+0000","last_undegraded":"2024-02-13T00:39:12.239289+0000","last_fullsized":"2024-02-13T00:39:12.239289+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:59.188913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:59.188913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240485+0000","last_change":"2024-02-13T00:38:57.494754+0000","last_active":"2024-02-13T00:39:12.240485+0000","last_peered":"2024-02-13T00:39:12.240485+0000","last_clean":"2024-02-13T00:39:12.240485+0000","last_became_active":"2024-02-13T00:37:45.245897+0000","last_became_peered":"2024-02-13T00:37:45.245897+0000","last_unstale":"2024-02-13T00:39:12.240485+0000","last_undegraded":"2024-02-13T00:39:12.240485+0000","last_fullsized":"2024-02-13T00:39:12.240485+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.494653+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:29.159172+0000","last_change":"2024-02-13T00:39:29.159172+0000","last_active":"2024-02-13T00:39:29.159172+0000","last_peered":"2024-02-13T00:39:29.159172+0000","last_clean":"2024-02-13T00:39:29.159172+0000","last_became_active":"2024-02-13T00:37:41.221627+0000","last_became_peered":"2024-02-13T00:37:41.221627+0000","last_unstale":"2024-02-13T00:39:29.159172+0000","last_undegraded":"2024-02-13T00:39:29.159172+0000","last_fullsized":"2024-02-13T00:39:29.159172+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.159080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.159080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":73,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:29.275198+0000","last_change":"2024-02-13T00:39:29.275198+0000","last_active":"2024-02-13T00:39:29.275198+0000","last_peered":"2024-02-13T00:39:29.275198+0000","last_clean":"2024-02-13T00:39:29.275198+0000","last_became_active":"2024-02-13T00:37:43.224302+0000","last_became_peered":"2024-02-13T00:37:43.224302+0000","last_unstale":"2024-02-13T00:39:29.275198+0000","last_undegraded":"2024-02-13T00:39:29.275198+0000","last_fullsized":"2024-02-13T00:39:29.275198+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:29.275134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.275134+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240172+0000","last_change":"2024-02-13T00:38:54.599127+0000","last_active":"2024-02-13T00:39:12.240172+0000","last_peered":"2024-02-13T00:39:12.240172+0000","last_clean":"2024-02-13T00:39:12.240172+0000","last_became_active":"2024-02-13T00:37:47.260477+0000","last_became_peered":"2024-02-13T00:37:47.260477+0000","last_unstale":"2024-02-13T00:39:12.240172+0000","last_undegraded":"2024-02-13T00:39:12.240172+0000","last_fullsized":"2024-02-13T00:39:12.240172+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:54.599057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:54.599057+0000","last_clean_scrub_stamp":"2024-02-13T00:38:54.599057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"64'8","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233393+0000","last_change":"2024-02-13T00:37:45.245491+0000","last_active":"2024-02-13T00:39:34.233393+0000","last_peered":"2024-02-13T00:39:34.233393+0000","last_clean":"2024-02-13T00:39:34.233393+0000","last_became_active":"2024-02-13T00:37:45.245183+0000","last_became_peered":"2024-02-13T00:37:45.245183+0000","last_unstale":"2024-02-13T00:39:34.233393+0000","last_undegraded":"2024-02-13T00:39:34.233393+0000","last_fullsized":"2024-02-13T00:39:34.233393+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:37:44.215312+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":48,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115712+0000","last_change":"2024-02-13T00:38:42.529745+0000","last_active":"2024-02-13T00:39:13.115712+0000","last_peered":"2024-02-13T00:39:13.115712+0000","last_clean":"2024-02-13T00:39:13.115712+0000","last_became_active":"2024-02-13T00:37:41.217979+0000","last_became_peered":"2024-02-13T00:37:41.217979+0000","last_unstale":"2024-02-13T00:39:13.115712+0000","last_undegraded":"2024-02-13T00:39:13.115712+0000","last_fullsized":"2024-02-13T00:39:13.115712+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:38:42.529666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:42.529666+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.370512+0000","last_change":"2024-02-13T00:39:23.370512+0000","last_active":"2024-02-13T00:39:23.370512+0000","last_peered":"2024-02-13T00:39:23.370512+0000","last_clean":"2024-02-13T00:39:23.370512+0000","last_became_active":"2024-02-13T00:37:43.232161+0000","last_became_peered":"2024-02-13T00:37:43.232161+0000","last_unstale":"2024-02-13T00:39:23.370512+0000","last_undegraded":"2024-02-13T00:39:23.370512+0000","last_fullsized":"2024-02-13T00:39:23.370512+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:39:23.370447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.370447+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.029022+0000","last_change":"2024-02-13T00:37:47.259820+0000","last_active":"2024-02-13T00:39:13.029022+0000","last_peered":"2024-02-13T00:39:13.029022+0000","last_clean":"2024-02-13T00:39:13.029022+0000","last_became_active":"2024-02-13T00:37:47.259517+0000","last_became_peered":"2024-02-13T00:37:47.259517+0000","last_unstale":"2024-02-13T00:39:13.029022+0000","last_undegraded":"2024-02-13T00:39:13.029022+0000","last_fullsized":"2024-02-13T00:39:13.029022+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:37:46.228444+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"64'8","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233366+0000","last_change":"2024-02-13T00:39:14.609925+0000","last_active":"2024-02-13T00:39:34.233366+0000","last_peered":"2024-02-13T00:39:34.233366+0000","last_clean":"2024-02-13T00:39:34.233366+0000","last_became_active":"2024-02-13T00:37:45.249684+0000","last_became_peered":"2024-02-13T00:37:45.249684+0000","last_unstale":"2024-02-13T00:39:34.233366+0000","last_undegraded":"2024-02-13T00:39:34.233366+0000","last_fullsized":"2024-02-13T00:39:34.233366+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:14.609850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:14.609850+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:17.284029+0000","last_change":"2024-02-13T00:39:17.283996+0000","last_active":"2024-02-13T00:39:17.284029+0000","last_peered":"2024-02-13T00:39:17.284029+0000","last_clean":"2024-02-13T00:39:17.284029+0000","last_became_active":"2024-02-13T00:37:41.218346+0000","last_became_peered":"2024-02-13T00:37:41.218346+0000","last_unstale":"2024-02-13T00:39:17.284029+0000","last_undegraded":"2024-02-13T00:39:17.284029+0000","last_fullsized":"2024-02-13T00:39:17.284029+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.283902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.283902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:28.305239+0000","last_change":"2024-02-13T00:39:28.305239+0000","last_active":"2024-02-13T00:39:28.305239+0000","last_peered":"2024-02-13T00:39:28.305239+0000","last_clean":"2024-02-13T00:39:28.305239+0000","last_became_active":"2024-02-13T00:37:43.227980+0000","last_became_peered":"2024-02-13T00:37:43.227980+0000","last_unstale":"2024-02-13T00:39:28.305239+0000","last_undegraded":"2024-02-13T00:39:28.305239+0000","last_fullsized":"2024-02-13T00:39:28.305239+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.305179+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115273+0000","last_change":"2024-02-13T00:39:07.563502+0000","last_active":"2024-02-13T00:39:13.115273+0000","last_peered":"2024-02-13T00:39:13.115273+0000","last_clean":"2024-02-13T00:39:13.115273+0000","last_became_active":"2024-02-13T00:37:47.256807+0000","last_became_peered":"2024-02-13T00:37:47.256807+0000","last_unstale":"2024-02-13T00:39:13.115273+0000","last_undegraded":"2024-02-13T00:39:13.115273+0000","last_fullsized":"2024-02-13T00:39:13.115273+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.563440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.563440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:35.269607+0000","last_change":"2024-02-13T00:39:35.269607+0000","last_active":"2024-02-13T00:39:35.269607+0000","last_peered":"2024-02-13T00:39:35.269607+0000","last_clean":"2024-02-13T00:39:35.269607+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:39:35.269607+0000","last_undegraded":"2024-02-13T00:39:35.269607+0000","last_fullsized":"2024-02-13T00:39:35.269607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:35.269538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:35.269538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138710+0000","last_change":"2024-02-13T00:39:24.138710+0000","last_active":"2024-02-13T00:39:24.138710+0000","last_peered":"2024-02-13T00:39:24.138710+0000","last_clean":"2024-02-13T00:39:24.138710+0000","last_became_active":"2024-02-13T00:37:41.220874+0000","last_became_peered":"2024-02-13T00:37:41.220874+0000","last_unstale":"2024-02-13T00:39:24.138710+0000","last_undegraded":"2024-02-13T00:39:24.138710+0000","last_fullsized":"2024-02-13T00:39:24.138710+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:39:24.138629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.138629+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.029183+0000","last_change":"2024-02-13T00:39:02.188162+0000","last_active":"2024-02-13T00:39:13.029183+0000","last_peered":"2024-02-13T00:39:13.029183+0000","last_clean":"2024-02-13T00:39:13.029183+0000","last_became_active":"2024-02-13T00:37:43.223574+0000","last_became_peered":"2024-02-13T00:37:43.223574+0000","last_unstale":"2024-02-13T00:39:13.029183+0000","last_undegraded":"2024-02-13T00:39:13.029183+0000","last_fullsized":"2024-02-13T00:39:13.029183+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:39:02.188096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.188096+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:16.086309+0000","last_change":"2024-02-13T00:39:16.086278+0000","last_active":"2024-02-13T00:39:16.086309+0000","last_peered":"2024-02-13T00:39:16.086309+0000","last_clean":"2024-02-13T00:39:16.086309+0000","last_became_active":"2024-02-13T00:37:47.254705+0000","last_became_peered":"2024-02-13T00:37:47.254705+0000","last_unstale":"2024-02-13T00:39:16.086309+0000","last_undegraded":"2024-02-13T00:39:16.086309+0000","last_fullsized":"2024-02-13T00:39:16.086309+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.086207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.086207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:17.632011+0000","last_change":"2024-02-13T00:39:17.631975+0000","last_active":"2024-02-13T00:39:17.632011+0000","last_peered":"2024-02-13T00:39:17.632011+0000","last_clean":"2024-02-13T00:39:17.632011+0000","last_became_active":"2024-02-13T00:37:45.239867+0000","last_became_peered":"2024-02-13T00:37:45.239867+0000","last_unstale":"2024-02-13T00:39:17.632011+0000","last_undegraded":"2024-02-13T00:39:17.632011+0000","last_fullsized":"2024-02-13T00:39:17.632011+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.631877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:18.672624+0000","last_change":"2024-02-13T00:39:18.672595+0000","last_active":"2024-02-13T00:39:18.672624+0000","last_peered":"2024-02-13T00:39:18.672624+0000","last_clean":"2024-02-13T00:39:18.672624+0000","last_became_active":"2024-02-13T00:37:41.219457+0000","last_became_peered":"2024-02-13T00:37:41.219457+0000","last_unstale":"2024-02-13T00:39:18.672624+0000","last_undegraded":"2024-02-13T00:39:18.672624+0000","last_fullsized":"2024-02-13T00:39:18.672624+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.672516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.672516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'25","reported_seq":86,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.684152+0000","last_change":"2024-02-13T00:38:56.371071+0000","last_active":"2024-02-13T00:39:12.684152+0000","last_peered":"2024-02-13T00:39:12.684152+0000","last_clean":"2024-02-13T00:39:12.684152+0000","last_became_active":"2024-02-13T00:37:43.322957+0000","last_became_peered":"2024-02-13T00:37:43.322957+0000","last_unstale":"2024-02-13T00:39:12.684152+0000","last_undegraded":"2024-02-13T00:39:12.684152+0000","last_fullsized":"2024-02-13T00:39:12.684152+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'25","last_scrub_stamp":"2024-02-13T00:38:56.370998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.370998+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:18.359703+0000","last_change":"2024-02-13T00:39:18.359681+0000","last_active":"2024-02-13T00:39:18.359703+0000","last_peered":"2024-02-13T00:39:18.359703+0000","last_clean":"2024-02-13T00:39:18.359703+0000","last_became_active":"2024-02-13T00:37:47.249556+0000","last_became_peered":"2024-02-13T00:37:47.249556+0000","last_unstale":"2024-02-13T00:39:18.359703+0000","last_undegraded":"2024-02-13T00:39:18.359703+0000","last_fullsized":"2024-02-13T00:39:18.359703+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.359604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:15.603914+0000","last_change":"2024-02-13T00:39:15.603878+0000","last_active":"2024-02-13T00:39:15.603914+0000","last_peered":"2024-02-13T00:39:15.603914+0000","last_clean":"2024-02-13T00:39:15.603914+0000","last_became_active":"2024-02-13T00:37:45.240923+0000","last_became_peered":"2024-02-13T00:37:45.240923+0000","last_unstale":"2024-02-13T00:39:15.603914+0000","last_undegraded":"2024-02-13T00:39:15.603914+0000","last_fullsized":"2024-02-13T00:39:15.603914+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.603808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.603808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:16.262471+0000","last_change":"2024-02-13T00:39:16.262449+0000","last_active":"2024-02-13T00:39:16.262471+0000","last_peered":"2024-02-13T00:39:16.262471+0000","last_clean":"2024-02-13T00:39:16.262471+0000","last_became_active":"2024-02-13T00:37:41.212429+0000","last_became_peered":"2024-02-13T00:37:41.212429+0000","last_unstale":"2024-02-13T00:39:16.262471+0000","last_undegraded":"2024-02-13T00:39:16.262471+0000","last_fullsized":"2024-02-13T00:39:16.262471+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.262380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.262380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.376175+0000","last_change":"2024-02-13T00:39:24.376175+0000","last_active":"2024-02-13T00:39:24.376175+0000","last_peered":"2024-02-13T00:39:24.376175+0000","last_clean":"2024-02-13T00:39:24.376175+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:39:24.376175+0000","last_undegraded":"2024-02-13T00:39:24.376175+0000","last_fullsized":"2024-02-13T00:39:24.376175+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:39:24.376100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.376100+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:17.713064+0000","last_change":"2024-02-13T00:39:17.713034+0000","last_active":"2024-02-13T00:39:17.713064+0000","last_peered":"2024-02-13T00:39:17.713064+0000","last_clean":"2024-02-13T00:39:17.713064+0000","last_became_active":"2024-02-13T00:37:47.552384+0000","last_became_peered":"2024-02-13T00:37:47.552384+0000","last_unstale":"2024-02-13T00:39:17.713064+0000","last_undegraded":"2024-02-13T00:39:17.713064+0000","last_fullsized":"2024-02-13T00:39:17.713064+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.712965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.712965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115491+0000","last_change":"2024-02-13T00:39:02.545556+0000","last_active":"2024-02-13T00:39:13.115491+0000","last_peered":"2024-02-13T00:39:13.115491+0000","last_clean":"2024-02-13T00:39:13.115491+0000","last_became_active":"2024-02-13T00:37:41.219093+0000","last_became_peered":"2024-02-13T00:37:41.219093+0000","last_unstale":"2024-02-13T00:39:13.115491+0000","last_undegraded":"2024-02-13T00:39:13.115491+0000","last_fullsized":"2024-02-13T00:39:13.115491+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:02.545495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.545495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":103,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239641+0000","last_change":"2024-02-13T00:38:59.548948+0000","last_active":"2024-02-13T00:39:12.239641+0000","last_peered":"2024-02-13T00:39:12.239641+0000","last_clean":"2024-02-13T00:39:12.239641+0000","last_became_active":"2024-02-13T00:37:43.322374+0000","last_became_peered":"2024-02-13T00:37:43.322374+0000","last_unstale":"2024-02-13T00:39:12.239641+0000","last_undegraded":"2024-02-13T00:39:12.239641+0000","last_fullsized":"2024-02-13T00:39:12.239641+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_clean_scrub_stamp":"2024-02-13T00:38:59.548872+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"64'8","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233399+0000","last_change":"2024-02-13T00:39:15.080406+0000","last_active":"2024-02-13T00:39:34.233399+0000","last_peered":"2024-02-13T00:39:34.233399+0000","last_clean":"2024-02-13T00:39:34.233399+0000","last_became_active":"2024-02-13T00:37:45.247905+0000","last_became_peered":"2024-02-13T00:37:45.247905+0000","last_unstale":"2024-02-13T00:39:34.233399+0000","last_undegraded":"2024-02-13T00:39:34.233399+0000","last_fullsized":"2024-02-13T00:39:34.233399+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:15.080315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.080315+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":640,"ondisk_log_size":640,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266714,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297732,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447420,"statfs":{"total":95995035648,"available":95690158080,"internally_reserved":0,"allocated":651264,"data_stored":297684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.498,"15min":0.501},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.63800000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.505},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52300000000000002,"15min":0.52800000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.63,"5min":1.028,"15min":1.028},"last":0.625}]},{"osd":1,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.49099999999999999,"15min":0.49399999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.59899999999999998,"15min":0.61499999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64200000000000002,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.58199999999999996}]},{"osd":2,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.628,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61099999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.52800000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.55800000000000005,"15min":0.56799999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.64100000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.65700000000000003,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.57499999999999996}]},{"osd":4,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.46100000000000002,"15min":0.46999999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.65400000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.45600000000000002,"15min":0.46400000000000002},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67000000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.40200000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.63600000000000001,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46800000000000003,"15min":0.47199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65800000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.438}]},{"osd":6,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.49199999999999999,"15min":0.497},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68899999999999995,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.45400000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528352,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297708,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447444,"statfs":{"total":95995035648,"available":95690182656,"internally_reserved":0,"allocated":626688,"data_stored":296161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.46899999999999997,"15min":0.46800000000000003},"min":{"1min":0.35899999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.52900000000000003}]},{"osd":1,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.504,"15min":0.50600000000000001},"min":{"1min":0.41499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61099999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.67800000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.38}]},{"osd":2,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.497,"15min":0.5},"min":{"1min":0.34899999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.64100000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.439},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.626,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34399999999999997}]},{"osd":3,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.53200000000000003,"15min":0.54300000000000004},"min":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68999999999999995,"5min":1.181,"15min":1.181},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.42499999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.49099999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.39500000000000002},"min":{"1min":0.29999999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72199999999999998,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41699999999999998},"min":{"1min":0.31,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66700000000000004,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.47399999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.312,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61199999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.44700000000000001,"15min":0.442},"min":{"1min":0.34000000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.76300000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.51800000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.499,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67700000000000005,"5min":1.173,"15min":1.173},"last":0.504},{"interface":"front","average":{"1min":0.503,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79000000000000004,"5min":1.262,"15min":1.262},"last":0.54900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297760,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447392,"statfs":{"total":95995035648,"available":95690129408,"internally_reserved":0,"allocated":679936,"data_stored":297411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.32000000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.371},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.313},"min":{"1min":0.221,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.49099999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.28799999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:38:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.376,"15min":0.372},"min":{"1min":0.26200000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.624,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.28999999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.313}]},{"osd":3,"last update":"Tue Feb 13 00:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65000000000000002,"5min":0.748,"15min":0.748},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.45000000000000001,"15min":0.46500000000000002},"min":{"1min":0.29299999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.53900000000000003,"5min":0.745,"15min":0.745},"last":0.34100000000000003}]},{"osd":4,"last update":"Tue Feb 13 00:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.81299999999999994,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82499999999999996,"5min":1.02,"15min":1.02},"last":0.53500000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:38:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80100000000000005,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58299999999999996,"15min":0.58399999999999996},"min":{"1min":0.41599999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.497}]},{"osd":6,"last update":"Tue Feb 13 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78700000000000003,"5min":1.054,"15min":1.054},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86799999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.59399999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.78500000000000003,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.62,"15min":0.623},"min":{"1min":0.41599999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81899999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.61499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":647168,"data_stored":296465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:39:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34599999999999997,"15min":0.32800000000000001},"min":{"1min":0.254,"5min":0.192,"15min":0.192},"max":{"1min":0.52000000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.378,"5min":0.376,"15min":0.34399999999999997},"min":{"1min":0.23499999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.58899999999999997,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.59999999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:38:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38400000000000001,"15min":0.378},"min":{"1min":0.28999999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.746}]},{"osd":3,"last update":"Tue Feb 13 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.64900000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45600000000000002,"15min":0.46999999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.71499999999999997}]},{"osd":4,"last update":"Tue Feb 13 00:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.75600000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72099999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.89500000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.65800000000000003,"5min":0.88,"15min":0.88},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.74099999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.92100000000000004}]},{"osd":6,"last update":"Tue Feb 13 00:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.41599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54200000000000004,"15min":0.53700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.84299999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:39:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.80800000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411380,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":296732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:38:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.36599999999999999,"15min":0.35799999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.34499999999999997},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.44700000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:38:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.35799999999999998,"15min":0.34899999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.52100000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.38400000000000001,"15min":0.374},"min":{"1min":0.245,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47199999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:39:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.28000000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.52800000000000002,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.442},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59099999999999997,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.504}]},{"osd":4,"last update":"Tue Feb 13 00:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73699999999999999,"5min":0.873,"15min":0.873},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.58799999999999997,"15min":0.59199999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71699999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.55800000000000005}]},{"osd":5,"last update":"Tue Feb 13 00:38:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.61499999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58099999999999996,"15min":0.58399999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79400000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.495}]},{"osd":7,"last update":"Tue Feb 13 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58299999999999996,"15min":0.59099999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.748,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70999999999999996,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.59699999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182446,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297744,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447408,"statfs":{"total":95995035648,"available":95690145792,"internally_reserved":0,"allocated":663552,"data_stored":296994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.36099999999999999,"15min":0.376},"min":{"1min":0.221,"5min":0.191,"15min":0.191},"max":{"1min":0.55300000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.39300000000000002,"15min":0.40799999999999997},"min":{"1min":0.27800000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46200000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.22600000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38500000000000001,"15min":0.39800000000000002},"min":{"1min":0.28100000000000003,"5min":0.217,"15min":0.217},"max":{"1min":0.5,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.39100000000000001,"15min":0.39900000000000002},"min":{"1min":0.24399999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.56200000000000006,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.41399999999999998,"15min":0.42099999999999999},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58899999999999997,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39200000000000002,"15min":0.40300000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.52900000000000003,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.247}]},{"osd":4,"last update":"Tue Feb 13 00:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53400000000000003,"15min":0.54000000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.64700000000000002,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.53600000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.75,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.46800000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:38:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.50600000000000001,"15min":0.499},"min":{"1min":0.38200000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.498},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.52100000000000002,"15min":0.51600000000000001},"min":{"1min":0.41199999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.41799999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.46100000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:39:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.54400000000000004},"min":{"1min":0.372,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77800000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.435}]}]},{"osd":4,"up_from":31,"seq":133143986218,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297740,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447412,"statfs":{"total":95995035648,"available":95690149888,"internally_reserved":0,"allocated":659456,"data_stored":297505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67900000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.435},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.66500000000000004,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53500000000000003}]},{"osd":1,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.52100000000000002,"15min":0.53000000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.61499999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.47599999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69899999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52000000000000002,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.62,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.498}]},{"osd":3,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.498,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.38500000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.45800000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:38:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.35699999999999998,"15min":0.34699999999999998},"min":{"1min":0.28199999999999997,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.375,"15min":0.36199999999999999},"min":{"1min":0.316,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.40600000000000003}]},{"osd":6,"last update":"Tue Feb 13 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.32900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.41299999999999998,"15min":0.40400000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.57699999999999996}]},{"osd":7,"last update":"Tue Feb 13 00:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46100000000000002,"15min":0.45900000000000002},"min":{"1min":0.34200000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.33000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.63800000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.443}]}]},{"osd":5,"up_from":39,"seq":167503724581,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297716,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":634880,"data_stored":297285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.48699999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.53100000000000003,"15min":0.51200000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.0369999999999999,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.55800000000000005}]},{"osd":1,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.5,"15min":0.496},"min":{"1min":0.34599999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.52400000000000002,"15min":0.502},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":4.0839999999999996,"5min":4.0839999999999996,"15min":4.0839999999999996},"last":0.503}]},{"osd":2,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.51400000000000001,"15min":0.50600000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.52300000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.51400000000000001,"15min":0.505},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.38300000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.55400000000000005}]},{"osd":4,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.40600000000000003,"15min":0.36199999999999999},"min":{"1min":0.246,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.1210000000000004,"5min":4.1210000000000004,"15min":4.1210000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.46700000000000003,"15min":0.40699999999999997},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.6950000000000003,"5min":5.6950000000000003,"15min":5.6950000000000003},"last":0.48699999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.441,"15min":0.42699999999999999},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.873,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.46400000000000002,"15min":0.44700000000000001},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.327,"5min":1.377,"15min":1.377},"last":0.40899999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.755,"15min":0.80300000000000005},"min":{"1min":0.33200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.58499999999999996,"5min":7.5519999999999996,"15min":7.5519999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.55800000000000005,"15min":0.57399999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.64800000000000002,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.42899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:39:43.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:39:43.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:42.150682+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.1b deep-scrub starts 2024-02-13T00:39:43.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:42.156251+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.1b deep-scrub ok 2024-02-13T00:39:43.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:42.182240+0000 mgr.y (mgr.14544) 82 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:39:43.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:42.501031+0000 mgr.y (mgr.14544) 83 : audit [DBG] from='client.24632 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:39:43.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:42.150682+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.1b deep-scrub starts 2024-02-13T00:39:43.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:42.156251+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.1b deep-scrub ok 2024-02-13T00:39:43.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:42.182240+0000 mgr.y (mgr.14544) 82 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:39:43.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:42.501031+0000 mgr.y (mgr.14544) 83 : audit [DBG] from='client.24632 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:39:43.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:42.150682+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.1b deep-scrub starts 2024-02-13T00:39:43.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:42.156251+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.1b deep-scrub ok 2024-02-13T00:39:43.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:42.182240+0000 mgr.y (mgr.14544) 82 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:39:43.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:42.501031+0000 mgr.y (mgr.14544) 83 : audit [DBG] from='client.24632 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:39:43.683 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:45.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 -- 172.21.15.162:0/4186932999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86500fd4c0 msgr2=0x7f86500f7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:45.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 --2- 172.21.15.162:0/4186932999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86500fd4c0 0x7f86500f7300 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f863c004870 tx=0x7f863c009cb0).stop 2024-02-13T00:39:45.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 -- 172.21.15.162:0/4186932999 shutdown_connections 2024-02-13T00:39:45.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 --2- 172.21.15.162:0/4186932999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86500f79e0 0x7f86501039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:39:45.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 --2- 172.21.15.162:0/4186932999 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86500fcb60 0x7f86500fcf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:45.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 --2- 172.21.15.162:0/4186932999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86500fd4c0 0x7f86500f7300 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:45.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 -- 172.21.15.162:0/4186932999 >> 172.21.15.162:0/4186932999 conn(0x7f8650070060 msgr2=0x7f86500f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:45.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 -- 172.21.15.162:0/4186932999 shutdown_connections 2024-02-13T00:39:45.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 -- 172.21.15.162:0/4186932999 wait complete. 2024-02-13T00:39:45.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 librados: starting msgr at 2024-02-13T00:39:45.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 librados: starting objecter 2024-02-13T00:39:45.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 Processor -- start 2024-02-13T00:39:45.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 -- start start 2024-02-13T00:39:45.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 librados: setting wanted keys 2024-02-13T00:39:45.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.422+0000 7f8656d7e700 1 librados: calling monclient init 2024-02-13T00:39:45.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8656d7e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86500f79e0 0x7f86500f8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:45.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8656d7e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86500fcb60 0x7f86500f8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:45.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8656d7e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86500fd4c0 0x7f86500f9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:45.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8656d7e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86501166d0 con 0x7f86500f79e0 2024-02-13T00:39:45.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8656d7e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8650116840 con 0x7f86500fcb60 2024-02-13T00:39:45.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8656d7e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8650116a00 con 0x7f86500fd4c0 2024-02-13T00:39:45.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8654b1a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86500f79e0 0x7f86500f8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:45.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8654b1a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86500f79e0 0x7f86500f8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46298/0 (socket says 172.21.15.162:46298) 2024-02-13T00:39:45.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8654b1a700 1 -- 172.21.15.162:0/2972830119 learned_addr learned my addr 172.21.15.162:0/2972830119 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:45.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f865531b700 1 --2- 172.21.15.162:0/2972830119 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86500fd4c0 0x7f86500f9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:45.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f864ffff700 1 --2- 172.21.15.162:0/2972830119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86500fcb60 0x7f86500f8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:45.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8654b1a700 1 -- 172.21.15.162:0/2972830119 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86500fd4c0 msgr2=0x7f86500f9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:45.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8654b1a700 1 --2- 172.21.15.162:0/2972830119 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86500fd4c0 0x7f86500f9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:45.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8654b1a700 1 -- 172.21.15.162:0/2972830119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86500fcb60 msgr2=0x7f86500f8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:45.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8654b1a700 1 --2- 172.21.15.162:0/2972830119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86500fcb60 0x7f86500f8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:45.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8654b1a700 1 -- 172.21.15.162:0/2972830119 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8644005040 con 0x7f86500f79e0 2024-02-13T00:39:45.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8654b1a700 1 --2- 172.21.15.162:0/2972830119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86500f79e0 0x7f86500f8960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f86440048f0 tx=0x7f8644009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:45.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f864dffb700 10 client.?.objecter ms_handle_connect 0x7f86500f79e0 2024-02-13T00:39:45.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f864dffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:45.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f864dffb700 1 -- 172.21.15.162:0/2972830119 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644007b00 con 0x7f86500f79e0 2024-02-13T00:39:45.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f864dffb700 1 -- 172.21.15.162:0/2972830119 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f864400f040 con 0x7f86500f79e0 2024-02-13T00:39:45.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f8656d7e700 1 -- 172.21.15.162:0/2972830119 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863c005040 con 0x7f86500f79e0 2024-02-13T00:39:45.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.423+0000 7f864dffb700 1 -- 172.21.15.162:0/2972830119 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644014a00 con 0x7f86500f79e0 2024-02-13T00:39:45.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.424+0000 7f8656d7e700 10 client.14739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:45.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.424+0000 7f8656d7e700 1 -- 172.21.15.162:0/2972830119 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8650117040 con 0x7f86500f79e0 2024-02-13T00:39:45.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.424+0000 7f8656d7e700 1 librados: init done 2024-02-13T00:39:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.425+0000 7f864dffb700 1 -- 172.21.15.162:0/2972830119 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f864400e580 con 0x7f86500f79e0 2024-02-13T00:39:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.425+0000 7f8656d7e700 1 -- 172.21.15.162:0/2972830119 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865005f960 con 0x7f86500f79e0 2024-02-13T00:39:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.425+0000 7f864dffb700 1 --2- 172.21.15.162:0/2972830119 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8640062c90 0x7f8640065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.425+0000 7f864dffb700 1 -- 172.21.15.162:0/2972830119 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(86..86 src has 30..86) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f864407c620 con 0x7f86500f79e0 2024-02-13T00:39:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.425+0000 7f864dffb700 10 client.14739.objecter ms_dispatch 0x7f8650001550 osd_map(86..86 src has 30..86) v4 2024-02-13T00:39:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.425+0000 7f864dffb700 3 client.14739.objecter handle_osd_map got epochs [86,86] > 0 2024-02-13T00:39:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.425+0000 7f864dffb700 3 client.14739.objecter handle_osd_map decoding full epoch 86 2024-02-13T00:39:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.425+0000 7f864ffff700 1 --2- 172.21.15.162:0/2972830119 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8640062c90 0x7f8640065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.425+0000 7f864dffb700 20 client.14739.objecter dump_active .. 0 homeless 2024-02-13T00:39:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.426+0000 7f864dffb700 10 client.14739.objecter ms_handle_connect 0x7f8640062c90 2024-02-13T00:39:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.426+0000 7f864ffff700 1 --2- 172.21.15.162:0/2972830119 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8640062c90 0x7f8640065150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f863c004ee0 tx=0x7f863c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:45.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.430+0000 7f864dffb700 1 -- 172.21.15.162:0/2972830119 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8644049f60 con 0x7f86500f79e0 2024-02-13T00:39:45.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:39:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:39:45.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:43.731358+0000 osd.0 (osd.0) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:39:45.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:43.731415+0000 osd.0 (osd.0) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:39:45.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:44.182744+0000 mgr.y (mgr.14544) 84 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:39:45.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:44.260503+0000 mon.a (mon.0) 712 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-13T00:39:45.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:43.731358+0000 osd.0 (osd.0) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:39:45.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:43.731415+0000 osd.0 (osd.0) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:39:45.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:44.182744+0000 mgr.y (mgr.14544) 84 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:39:45.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:44.260503+0000 mon.a (mon.0) 712 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-13T00:39:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:43.731358+0000 osd.0 (osd.0) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:39:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:43.731415+0000 osd.0 (osd.0) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:39:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:44.182744+0000 mgr.y (mgr.14544) 84 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:39:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:44.260503+0000 mon.a (mon.0) 712 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-13T00:39:45.592 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.589+0000 7f8656d7e700 1 -- 172.21.15.162:0/2972830119 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f86500f9eb0 con 0x7f8640062c90 2024-02-13T00:39:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.599+0000 7f864dffb700 1 -- 172.21.15.162:0/2972830119 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357119 (secure 0 0 0) 0x7f86500f9eb0 con 0x7f8640062c90 2024-02-13T00:39:45.606 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:39:45.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 10 librados: watch_flush enter 2024-02-13T00:39:45.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 10 librados: watch_flush exit 2024-02-13T00:39:45.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 20 client.14739.objecter shutdown clearing up homeless session... 2024-02-13T00:39:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 10 client.14739.objecter successfully canceled tick 2024-02-13T00:39:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 -- 172.21.15.162:0/2972830119 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8640062c90 msgr2=0x7f8640065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 --2- 172.21.15.162:0/2972830119 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8640062c90 0x7f8640065150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f863c004ee0 tx=0x7f863c00b040).stop 2024-02-13T00:39:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 -- 172.21.15.162:0/2972830119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86500f79e0 msgr2=0x7f86500f8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 --2- 172.21.15.162:0/2972830119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86500f79e0 0x7f86500f8960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f86440048f0 tx=0x7f8644009cb0).stop 2024-02-13T00:39:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 -- 172.21.15.162:0/2972830119 shutdown_connections 2024-02-13T00:39:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 --2- 172.21.15.162:0/2972830119 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8640062c90 0x7f8640065150 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 --2- 172.21.15.162:0/2972830119 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86500fd4c0 0x7f86500f9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 --2- 172.21.15.162:0/2972830119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86500f79e0 0x7f86500f8960 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 --2- 172.21.15.162:0/2972830119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86500fcb60 0x7f86500f8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 -- 172.21.15.162:0/2972830119 >> 172.21.15.162:0/2972830119 conn(0x7f8650070060 msgr2=0x7f86500f62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 -- 172.21.15.162:0/2972830119 shutdown_connections 2024-02-13T00:39:45.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 -- 172.21.15.162:0/2972830119 wait complete. 2024-02-13T00:39:45.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:45.602+0000 7f8656d7e700 1 librados: shutdown 2024-02-13T00:39:45.611 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:39:46.309 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-02-13T00:39:44.260151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1216,"num_read_kb":936,"num_write":620,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":716,"ondisk_log_size":716,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381848,"kb_used_data":5080,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579368,"statfs":{"total":767960285184,"available":765521272832,"internally_reserved":0,"allocated":5201920,"data_stored":2376237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.080778"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:18.311814+0000","last_change":"2024-02-13T00:39:18.311781+0000","last_active":"2024-02-13T00:39:18.311814+0000","last_peered":"2024-02-13T00:39:18.311814+0000","last_clean":"2024-02-13T00:39:18.311814+0000","last_became_active":"2024-02-13T00:37:47.552890+0000","last_became_peered":"2024-02-13T00:37:47.552890+0000","last_unstale":"2024-02-13T00:39:18.311814+0000","last_undegraded":"2024-02-13T00:39:18.311814+0000","last_fullsized":"2024-02-13T00:39:18.311814+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.311669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.311669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.273436+0000","last_change":"2024-02-13T00:39:23.273409+0000","last_active":"2024-02-13T00:39:23.273436+0000","last_peered":"2024-02-13T00:39:23.273436+0000","last_clean":"2024-02-13T00:39:23.273436+0000","last_became_active":"2024-02-13T00:37:45.245133+0000","last_became_peered":"2024-02-13T00:37:45.245133+0000","last_unstale":"2024-02-13T00:39:23.273436+0000","last_undegraded":"2024-02-13T00:39:23.273436+0000","last_fullsized":"2024-02-13T00:39:23.273436+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.273346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:32.313930+0000","last_change":"2024-02-13T00:39:32.313930+0000","last_active":"2024-02-13T00:39:32.313930+0000","last_peered":"2024-02-13T00:39:32.313930+0000","last_clean":"2024-02-13T00:39:32.313930+0000","last_became_active":"2024-02-13T00:37:41.220583+0000","last_became_peered":"2024-02-13T00:37:41.220583+0000","last_unstale":"2024-02-13T00:39:32.313930+0000","last_undegraded":"2024-02-13T00:39:32.313930+0000","last_fullsized":"2024-02-13T00:39:32.313930+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.313841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'39","reported_seq":124,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239928+0000","last_change":"2024-02-13T00:38:47.210073+0000","last_active":"2024-02-13T00:39:12.239928+0000","last_peered":"2024-02-13T00:39:12.239928+0000","last_clean":"2024-02-13T00:39:12.239928+0000","last_became_active":"2024-02-13T00:37:43.228728+0000","last_became_peered":"2024-02-13T00:37:43.228728+0000","last_unstale":"2024-02-13T00:39:12.239928+0000","last_undegraded":"2024-02-13T00:39:12.239928+0000","last_fullsized":"2024-02-13T00:39:12.239928+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'39","last_scrub_stamp":"2024-02-13T00:38:47.210000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:47.210000+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:37.210485+0000","last_change":"2024-02-13T00:39:37.210485+0000","last_active":"2024-02-13T00:39:37.210485+0000","last_peered":"2024-02-13T00:39:37.210485+0000","last_clean":"2024-02-13T00:39:37.210485+0000","last_became_active":"2024-02-13T00:37:47.259555+0000","last_became_peered":"2024-02-13T00:37:47.259555+0000","last_unstale":"2024-02-13T00:39:37.210485+0000","last_undegraded":"2024-02-13T00:39:37.210485+0000","last_fullsized":"2024-02-13T00:39:37.210485+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:37.210406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:37.210406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.332217+0000","last_change":"2024-02-13T00:39:24.332217+0000","last_active":"2024-02-13T00:39:24.332217+0000","last_peered":"2024-02-13T00:39:24.332217+0000","last_clean":"2024-02-13T00:39:24.332217+0000","last_became_active":"2024-02-13T00:37:45.249914+0000","last_became_peered":"2024-02-13T00:37:45.249914+0000","last_unstale":"2024-02-13T00:39:24.332217+0000","last_undegraded":"2024-02-13T00:39:24.332217+0000","last_fullsized":"2024-02-13T00:39:24.332217+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:24.332133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.332133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239919+0000","last_change":"2024-02-13T00:37:41.217193+0000","last_active":"2024-02-13T00:39:12.239919+0000","last_peered":"2024-02-13T00:39:12.239919+0000","last_clean":"2024-02-13T00:39:12.239919+0000","last_became_active":"2024-02-13T00:37:41.216896+0000","last_became_peered":"2024-02-13T00:37:41.216896+0000","last_unstale":"2024-02-13T00:39:12.239919+0000","last_undegraded":"2024-02-13T00:39:12.239919+0000","last_fullsized":"2024-02-13T00:39:12.239919+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:37:40.201175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028519+0000","last_change":"2024-02-13T00:37:43.223162+0000","last_active":"2024-02-13T00:39:13.028519+0000","last_peered":"2024-02-13T00:39:13.028519+0000","last_clean":"2024-02-13T00:39:13.028519+0000","last_became_active":"2024-02-13T00:37:43.222849+0000","last_became_peered":"2024-02-13T00:37:43.222849+0000","last_unstale":"2024-02-13T00:39:13.028519+0000","last_undegraded":"2024-02-13T00:39:13.028519+0000","last_fullsized":"2024-02-13T00:39:13.028519+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239660+0000","last_change":"2024-02-13T00:38:53.600488+0000","last_active":"2024-02-13T00:39:12.239660+0000","last_peered":"2024-02-13T00:39:12.239660+0000","last_clean":"2024-02-13T00:39:12.239660+0000","last_became_active":"2024-02-13T00:37:47.260025+0000","last_became_peered":"2024-02-13T00:37:47.260025+0000","last_unstale":"2024-02-13T00:39:12.239660+0000","last_undegraded":"2024-02-13T00:39:12.239660+0000","last_fullsized":"2024-02-13T00:39:12.239660+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:38:53.600425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.026600+0000","last_change":"2024-02-13T00:38:52.859190+0000","last_active":"2024-02-13T00:39:13.026600+0000","last_peered":"2024-02-13T00:39:13.026600+0000","last_clean":"2024-02-13T00:39:13.026600+0000","last_became_active":"2024-02-13T00:37:45.246184+0000","last_became_peered":"2024-02-13T00:37:45.246184+0000","last_unstale":"2024-02-13T00:39:13.026600+0000","last_undegraded":"2024-02-13T00:39:13.026600+0000","last_fullsized":"2024-02-13T00:39:13.026600+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.859089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.859089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:21.165337+0000","last_change":"2024-02-13T00:39:21.165313+0000","last_active":"2024-02-13T00:39:21.165337+0000","last_peered":"2024-02-13T00:39:21.165337+0000","last_clean":"2024-02-13T00:39:21.165337+0000","last_became_active":"2024-02-13T00:37:41.221025+0000","last_became_peered":"2024-02-13T00:37:41.221025+0000","last_unstale":"2024-02-13T00:39:21.165337+0000","last_undegraded":"2024-02-13T00:39:21.165337+0000","last_fullsized":"2024-02-13T00:39:21.165337+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.165246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'11","reported_seq":59,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028619+0000","last_change":"2024-02-13T00:37:43.230514+0000","last_active":"2024-02-13T00:39:13.028619+0000","last_peered":"2024-02-13T00:39:13.028619+0000","last_clean":"2024-02-13T00:39:13.028619+0000","last_became_active":"2024-02-13T00:37:43.230223+0000","last_became_peered":"2024-02-13T00:37:43.230223+0000","last_unstale":"2024-02-13T00:39:13.028619+0000","last_undegraded":"2024-02-13T00:39:13.028619+0000","last_fullsized":"2024-02-13T00:39:13.028619+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:22.338171+0000","last_change":"2024-02-13T00:39:22.338139+0000","last_active":"2024-02-13T00:39:22.338171+0000","last_peered":"2024-02-13T00:39:22.338171+0000","last_clean":"2024-02-13T00:39:22.338171+0000","last_became_active":"2024-02-13T00:37:47.259688+0000","last_became_peered":"2024-02-13T00:37:47.259688+0000","last_unstale":"2024-02-13T00:39:22.338171+0000","last_undegraded":"2024-02-13T00:39:22.338171+0000","last_fullsized":"2024-02-13T00:39:22.338171+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.338042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240052+0000","last_change":"2024-02-13T00:39:06.581693+0000","last_active":"2024-02-13T00:39:12.240052+0000","last_peered":"2024-02-13T00:39:12.240052+0000","last_clean":"2024-02-13T00:39:12.240052+0000","last_became_active":"2024-02-13T00:37:45.243961+0000","last_became_peered":"2024-02-13T00:37:45.243961+0000","last_unstale":"2024-02-13T00:39:12.240052+0000","last_undegraded":"2024-02-13T00:39:12.240052+0000","last_fullsized":"2024-02-13T00:39:12.240052+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:06.581608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:06.581608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.026764+0000","last_change":"2024-02-13T00:38:45.896411+0000","last_active":"2024-02-13T00:39:13.026764+0000","last_peered":"2024-02-13T00:39:13.026764+0000","last_clean":"2024-02-13T00:39:13.026764+0000","last_became_active":"2024-02-13T00:37:41.222506+0000","last_became_peered":"2024-02-13T00:37:41.222506+0000","last_unstale":"2024-02-13T00:39:13.026764+0000","last_undegraded":"2024-02-13T00:39:13.026764+0000","last_fullsized":"2024-02-13T00:39:13.026764+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:45.896343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:45.896343+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":85,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.213153+0000","last_change":"2024-02-13T00:39:34.213153+0000","last_active":"2024-02-13T00:39:34.213153+0000","last_peered":"2024-02-13T00:39:34.213153+0000","last_clean":"2024-02-13T00:39:34.213153+0000","last_became_active":"2024-02-13T00:37:43.228966+0000","last_became_peered":"2024-02-13T00:37:43.228966+0000","last_unstale":"2024-02-13T00:39:34.213153+0000","last_undegraded":"2024-02-13T00:39:34.213153+0000","last_fullsized":"2024-02-13T00:39:34.213153+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:39:34.213074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.213074+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239640+0000","last_change":"2024-02-13T00:39:02.223894+0000","last_active":"2024-02-13T00:39:12.239640+0000","last_peered":"2024-02-13T00:39:12.239640+0000","last_clean":"2024-02-13T00:39:12.239640+0000","last_became_active":"2024-02-13T00:37:47.553853+0000","last_became_peered":"2024-02-13T00:37:47.553853+0000","last_unstale":"2024-02-13T00:39:12.239640+0000","last_undegraded":"2024-02-13T00:39:12.239640+0000","last_fullsized":"2024-02-13T00:39:12.239640+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:02.223823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.223823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:21.301368+0000","last_change":"2024-02-13T00:39:21.301346+0000","last_active":"2024-02-13T00:39:21.301368+0000","last_peered":"2024-02-13T00:39:21.301368+0000","last_clean":"2024-02-13T00:39:21.301368+0000","last_became_active":"2024-02-13T00:37:45.242917+0000","last_became_peered":"2024-02-13T00:37:45.242917+0000","last_unstale":"2024-02-13T00:39:21.301368+0000","last_undegraded":"2024-02-13T00:39:21.301368+0000","last_fullsized":"2024-02-13T00:39:21.301368+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.301282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239489+0000","last_change":"2024-02-13T00:39:00.215830+0000","last_active":"2024-02-13T00:39:12.239489+0000","last_peered":"2024-02-13T00:39:12.239489+0000","last_clean":"2024-02-13T00:39:12.239489+0000","last_became_active":"2024-02-13T00:37:41.218333+0000","last_became_peered":"2024-02-13T00:37:41.218333+0000","last_unstale":"2024-02-13T00:39:12.239489+0000","last_undegraded":"2024-02-13T00:39:12.239489+0000","last_fullsized":"2024-02-13T00:39:12.239489+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.215755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.215755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":77,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:26.214150+0000","last_change":"2024-02-13T00:39:26.214150+0000","last_active":"2024-02-13T00:39:26.214150+0000","last_peered":"2024-02-13T00:39:26.214150+0000","last_clean":"2024-02-13T00:39:26.214150+0000","last_became_active":"2024-02-13T00:37:43.236026+0000","last_became_peered":"2024-02-13T00:37:43.236026+0000","last_unstale":"2024-02-13T00:39:26.214150+0000","last_undegraded":"2024-02-13T00:39:26.214150+0000","last_fullsized":"2024-02-13T00:39:26.214150+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:39:26.214084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:26.214084+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.029224+0000","last_change":"2024-02-13T00:37:47.552844+0000","last_active":"2024-02-13T00:39:13.029224+0000","last_peered":"2024-02-13T00:39:13.029224+0000","last_clean":"2024-02-13T00:39:13.029224+0000","last_became_active":"2024-02-13T00:37:47.552547+0000","last_became_peered":"2024-02-13T00:37:47.552547+0000","last_unstale":"2024-02-13T00:39:13.029224+0000","last_undegraded":"2024-02-13T00:39:13.029224+0000","last_fullsized":"2024-02-13T00:39:13.029224+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:37:46.228444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:30.245586+0000","last_change":"2024-02-13T00:39:30.245586+0000","last_active":"2024-02-13T00:39:30.245586+0000","last_peered":"2024-02-13T00:39:30.245586+0000","last_clean":"2024-02-13T00:39:30.245586+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:39:30.245586+0000","last_undegraded":"2024-02-13T00:39:30.245586+0000","last_fullsized":"2024-02-13T00:39:30.245586+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:30.245508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.245508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.684215+0000","last_change":"2024-02-13T00:39:10.364972+0000","last_active":"2024-02-13T00:39:12.684215+0000","last_peered":"2024-02-13T00:39:12.684215+0000","last_clean":"2024-02-13T00:39:12.684215+0000","last_became_active":"2024-02-13T00:37:41.218334+0000","last_became_peered":"2024-02-13T00:37:41.218334+0000","last_unstale":"2024-02-13T00:39:12.684215+0000","last_undegraded":"2024-02-13T00:39:12.684215+0000","last_fullsized":"2024-02-13T00:39:12.684215+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:10.364865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.364865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115735+0000","last_change":"2024-02-13T00:39:08.582559+0000","last_active":"2024-02-13T00:39:13.115735+0000","last_peered":"2024-02-13T00:39:13.115735+0000","last_clean":"2024-02-13T00:39:13.115735+0000","last_became_active":"2024-02-13T00:37:43.232938+0000","last_became_peered":"2024-02-13T00:37:43.232938+0000","last_unstale":"2024-02-13T00:39:13.115735+0000","last_undegraded":"2024-02-13T00:39:13.115735+0000","last_fullsized":"2024-02-13T00:39:13.115735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:39:08.582479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.582479+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115363+0000","last_change":"2024-02-13T00:39:00.452185+0000","last_active":"2024-02-13T00:39:13.115363+0000","last_peered":"2024-02-13T00:39:13.115363+0000","last_clean":"2024-02-13T00:39:13.115363+0000","last_became_active":"2024-02-13T00:37:47.260182+0000","last_became_peered":"2024-02-13T00:37:47.260182+0000","last_unstale":"2024-02-13T00:39:13.115363+0000","last_undegraded":"2024-02-13T00:39:13.115363+0000","last_fullsized":"2024-02-13T00:39:13.115363+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.452084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.452084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:15.328059+0000","last_change":"2024-02-13T00:39:15.328037+0000","last_active":"2024-02-13T00:39:15.328059+0000","last_peered":"2024-02-13T00:39:15.328059+0000","last_clean":"2024-02-13T00:39:15.328059+0000","last_became_active":"2024-02-13T00:37:45.249064+0000","last_became_peered":"2024-02-13T00:37:45.249064+0000","last_unstale":"2024-02-13T00:39:15.328059+0000","last_undegraded":"2024-02-13T00:39:15.328059+0000","last_fullsized":"2024-02-13T00:39:15.328059+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.327971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.327971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239811+0000","last_change":"2024-02-13T00:37:41.216878+0000","last_active":"2024-02-13T00:39:12.239811+0000","last_peered":"2024-02-13T00:39:12.239811+0000","last_clean":"2024-02-13T00:39:12.239811+0000","last_became_active":"2024-02-13T00:37:41.216574+0000","last_became_peered":"2024-02-13T00:37:41.216574+0000","last_unstale":"2024-02-13T00:39:12.239811+0000","last_undegraded":"2024-02-13T00:39:12.239811+0000","last_fullsized":"2024-02-13T00:39:12.239811+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:37:40.201175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":74,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.684608+0000","last_change":"2024-02-13T00:38:58.341512+0000","last_active":"2024-02-13T00:39:12.684608+0000","last_peered":"2024-02-13T00:39:12.684608+0000","last_clean":"2024-02-13T00:39:12.684608+0000","last_became_active":"2024-02-13T00:37:43.230491+0000","last_became_peered":"2024-02-13T00:37:43.230491+0000","last_unstale":"2024-02-13T00:39:12.684608+0000","last_undegraded":"2024-02-13T00:39:12.684608+0000","last_fullsized":"2024-02-13T00:39:12.684608+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:38:58.341406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.341406+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.027218+0000","last_change":"2024-02-13T00:38:50.866411+0000","last_active":"2024-02-13T00:39:13.027218+0000","last_peered":"2024-02-13T00:39:13.027218+0000","last_clean":"2024-02-13T00:39:13.027218+0000","last_became_active":"2024-02-13T00:37:47.254586+0000","last_became_peered":"2024-02-13T00:37:47.254586+0000","last_unstale":"2024-02-13T00:39:13.027218+0000","last_undegraded":"2024-02-13T00:39:13.027218+0000","last_fullsized":"2024-02-13T00:39:13.027218+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:38:50.866323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:50.866323+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"64'8","reported_seq":73,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233398+0000","last_change":"2024-02-13T00:39:02.452828+0000","last_active":"2024-02-13T00:39:34.233398+0000","last_peered":"2024-02-13T00:39:34.233398+0000","last_clean":"2024-02-13T00:39:34.233398+0000","last_became_active":"2024-02-13T00:37:45.249902+0000","last_became_peered":"2024-02-13T00:37:45.249902+0000","last_unstale":"2024-02-13T00:39:34.233398+0000","last_undegraded":"2024-02-13T00:39:34.233398+0000","last_fullsized":"2024-02-13T00:39:34.233398+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:02.452755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.452755+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239588+0000","last_change":"2024-02-13T00:38:49.606456+0000","last_active":"2024-02-13T00:39:12.239588+0000","last_peered":"2024-02-13T00:39:12.239588+0000","last_clean":"2024-02-13T00:39:12.239588+0000","last_became_active":"2024-02-13T00:37:41.214965+0000","last_became_peered":"2024-02-13T00:37:41.214965+0000","last_unstale":"2024-02-13T00:39:12.239588+0000","last_undegraded":"2024-02-13T00:39:12.239588+0000","last_fullsized":"2024-02-13T00:39:12.239588+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:49.606357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:49.606357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":75,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.242507+0000","last_change":"2024-02-13T00:38:52.380430+0000","last_active":"2024-02-13T00:39:12.242507+0000","last_peered":"2024-02-13T00:39:12.242507+0000","last_clean":"2024-02-13T00:39:12.242507+0000","last_became_active":"2024-02-13T00:37:43.324221+0000","last_became_peered":"2024-02-13T00:37:43.324221+0000","last_unstale":"2024-02-13T00:39:12.242507+0000","last_undegraded":"2024-02-13T00:39:12.242507+0000","last_fullsized":"2024-02-13T00:39:12.242507+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.380369+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:21.243629+0000","last_change":"2024-02-13T00:39:21.243606+0000","last_active":"2024-02-13T00:39:21.243629+0000","last_peered":"2024-02-13T00:39:21.243629+0000","last_clean":"2024-02-13T00:39:21.243629+0000","last_became_active":"2024-02-13T00:37:47.552917+0000","last_became_peered":"2024-02-13T00:37:47.552917+0000","last_unstale":"2024-02-13T00:39:21.243629+0000","last_undegraded":"2024-02-13T00:39:21.243629+0000","last_fullsized":"2024-02-13T00:39:21.243629+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.243527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.027438+0000","last_change":"2024-02-13T00:39:13.027403+0000","last_active":"2024-02-13T00:39:13.027438+0000","last_peered":"2024-02-13T00:39:13.027438+0000","last_clean":"2024-02-13T00:39:13.027438+0000","last_became_active":"2024-02-13T00:37:45.248592+0000","last_became_peered":"2024-02-13T00:37:45.248592+0000","last_unstale":"2024-02-13T00:39:13.027438+0000","last_undegraded":"2024-02-13T00:39:13.027438+0000","last_fullsized":"2024-02-13T00:39:13.027438+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:13.027299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:13.027299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.684722+0000","last_change":"2024-02-13T00:39:07.358630+0000","last_active":"2024-02-13T00:39:12.684722+0000","last_peered":"2024-02-13T00:39:12.684722+0000","last_clean":"2024-02-13T00:39:12.684722+0000","last_became_active":"2024-02-13T00:37:41.220344+0000","last_became_peered":"2024-02-13T00:37:41.220344+0000","last_unstale":"2024-02-13T00:39:12.684722+0000","last_undegraded":"2024-02-13T00:39:12.684722+0000","last_fullsized":"2024-02-13T00:39:12.684722+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.358534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.358534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":73,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.204969+0000","last_change":"2024-02-13T00:39:34.204969+0000","last_active":"2024-02-13T00:39:34.204969+0000","last_peered":"2024-02-13T00:39:34.204969+0000","last_clean":"2024-02-13T00:39:34.204969+0000","last_became_active":"2024-02-13T00:37:43.324196+0000","last_became_peered":"2024-02-13T00:37:43.324196+0000","last_unstale":"2024-02-13T00:39:34.204969+0000","last_undegraded":"2024-02-13T00:39:34.204969+0000","last_fullsized":"2024-02-13T00:39:34.204969+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:34.204888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.204888+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.026416+0000","last_change":"2024-02-13T00:38:57.818305+0000","last_active":"2024-02-13T00:39:13.026416+0000","last_peered":"2024-02-13T00:39:13.026416+0000","last_clean":"2024-02-13T00:39:13.026416+0000","last_became_active":"2024-02-13T00:37:47.254667+0000","last_became_peered":"2024-02-13T00:37:47.254667+0000","last_unstale":"2024-02-13T00:39:13.026416+0000","last_undegraded":"2024-02-13T00:39:13.026416+0000","last_fullsized":"2024-02-13T00:39:13.026416+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:57.818202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.818202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.683998+0000","last_change":"2024-02-13T00:38:52.493308+0000","last_active":"2024-02-13T00:39:12.683998+0000","last_peered":"2024-02-13T00:39:12.683998+0000","last_clean":"2024-02-13T00:39:12.683998+0000","last_became_active":"2024-02-13T00:37:45.249556+0000","last_became_peered":"2024-02-13T00:37:45.249556+0000","last_unstale":"2024-02-13T00:39:12.683998+0000","last_undegraded":"2024-02-13T00:39:12.683998+0000","last_fullsized":"2024-02-13T00:39:12.683998+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.493237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.493237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240300+0000","last_change":"2024-02-13T00:38:55.573704+0000","last_active":"2024-02-13T00:39:12.240300+0000","last_peered":"2024-02-13T00:39:12.240300+0000","last_clean":"2024-02-13T00:39:12.240300+0000","last_became_active":"2024-02-13T00:37:41.213033+0000","last_became_peered":"2024-02-13T00:37:41.213033+0000","last_unstale":"2024-02-13T00:39:12.240300+0000","last_undegraded":"2024-02-13T00:39:12.240300+0000","last_fullsized":"2024-02-13T00:39:12.240300+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:55.573637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:55.573637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":45,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.231582+0000","last_change":"2024-02-13T00:39:23.231537+0000","last_active":"2024-02-13T00:39:23.231582+0000","last_peered":"2024-02-13T00:39:23.231582+0000","last_clean":"2024-02-13T00:39:23.231582+0000","last_became_active":"2024-02-13T00:37:43.234395+0000","last_became_peered":"2024-02-13T00:37:43.234395+0000","last_unstale":"2024-02-13T00:39:23.231582+0000","last_undegraded":"2024-02-13T00:39:23.231582+0000","last_fullsized":"2024-02-13T00:39:23.231582+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:39:23.231401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.231401+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239725+0000","last_change":"2024-02-13T00:39:01.197382+0000","last_active":"2024-02-13T00:39:12.239725+0000","last_peered":"2024-02-13T00:39:12.239725+0000","last_clean":"2024-02-13T00:39:12.239725+0000","last_became_active":"2024-02-13T00:37:47.254519+0000","last_became_peered":"2024-02-13T00:37:47.254519+0000","last_unstale":"2024-02-13T00:39:12.239725+0000","last_undegraded":"2024-02-13T00:39:12.239725+0000","last_fullsized":"2024-02-13T00:39:12.239725+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.197310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.197310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.116198+0000","last_change":"2024-02-13T00:38:58.512117+0000","last_active":"2024-02-13T00:39:13.116198+0000","last_peered":"2024-02-13T00:39:13.116198+0000","last_clean":"2024-02-13T00:39:13.116198+0000","last_became_active":"2024-02-13T00:37:45.247114+0000","last_became_peered":"2024-02-13T00:37:45.247114+0000","last_unstale":"2024-02-13T00:39:13.116198+0000","last_undegraded":"2024-02-13T00:39:13.116198+0000","last_fullsized":"2024-02-13T00:39:13.116198+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.512053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:27.297182+0000","last_change":"2024-02-13T00:39:27.297182+0000","last_active":"2024-02-13T00:39:27.297182+0000","last_peered":"2024-02-13T00:39:27.297182+0000","last_clean":"2024-02-13T00:39:27.297182+0000","last_became_active":"2024-02-13T00:37:41.215531+0000","last_became_peered":"2024-02-13T00:37:41.215531+0000","last_unstale":"2024-02-13T00:39:27.297182+0000","last_undegraded":"2024-02-13T00:39:27.297182+0000","last_fullsized":"2024-02-13T00:39:27.297182+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:27.297107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:27.297107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":84,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028714+0000","last_change":"2024-02-13T00:39:06.398068+0000","last_active":"2024-02-13T00:39:13.028714+0000","last_peered":"2024-02-13T00:39:13.028714+0000","last_clean":"2024-02-13T00:39:13.028714+0000","last_became_active":"2024-02-13T00:39:05.965554+0000","last_became_peered":"2024-02-13T00:39:05.965554+0000","last_unstale":"2024-02-13T00:39:13.028714+0000","last_undegraded":"2024-02-13T00:39:13.028714+0000","last_fullsized":"2024-02-13T00:39:13.028714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:39:03.189237+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:28.286635+0000","last_change":"2024-02-13T00:39:28.286635+0000","last_active":"2024-02-13T00:39:28.286635+0000","last_peered":"2024-02-13T00:39:28.286635+0000","last_clean":"2024-02-13T00:39:28.286635+0000","last_became_active":"2024-02-13T00:37:47.259792+0000","last_became_peered":"2024-02-13T00:37:47.259792+0000","last_unstale":"2024-02-13T00:39:28.286635+0000","last_undegraded":"2024-02-13T00:39:28.286635+0000","last_fullsized":"2024-02-13T00:39:28.286635+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.286574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.286574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:29.229760+0000","last_change":"2024-02-13T00:39:29.229760+0000","last_active":"2024-02-13T00:39:29.229760+0000","last_peered":"2024-02-13T00:39:29.229760+0000","last_clean":"2024-02-13T00:39:29.229760+0000","last_became_active":"2024-02-13T00:37:45.243594+0000","last_became_peered":"2024-02-13T00:37:45.243594+0000","last_unstale":"2024-02-13T00:39:29.229760+0000","last_undegraded":"2024-02-13T00:39:29.229760+0000","last_fullsized":"2024-02-13T00:39:29.229760+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.229676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.684249+0000","last_change":"2024-02-13T00:39:00.395451+0000","last_active":"2024-02-13T00:39:12.684249+0000","last_peered":"2024-02-13T00:39:12.684249+0000","last_clean":"2024-02-13T00:39:12.684249+0000","last_became_active":"2024-02-13T00:37:41.216390+0000","last_became_peered":"2024-02-13T00:37:41.216390+0000","last_unstale":"2024-02-13T00:39:12.684249+0000","last_undegraded":"2024-02-13T00:39:12.684249+0000","last_fullsized":"2024-02-13T00:39:12.684249+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.395381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.395381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":60,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.116148+0000","last_change":"2024-02-13T00:39:12.348547+0000","last_active":"2024-02-13T00:39:13.116148+0000","last_peered":"2024-02-13T00:39:13.116148+0000","last_clean":"2024-02-13T00:39:13.116148+0000","last_became_active":"2024-02-13T00:37:43.225347+0000","last_became_peered":"2024-02-13T00:37:43.225347+0000","last_unstale":"2024-02-13T00:39:13.116148+0000","last_undegraded":"2024-02-13T00:39:13.116148+0000","last_fullsized":"2024-02-13T00:39:13.116148+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:39:12.348480+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.029316+0000","last_change":"2024-02-13T00:37:47.254258+0000","last_active":"2024-02-13T00:39:13.029316+0000","last_peered":"2024-02-13T00:39:13.029316+0000","last_clean":"2024-02-13T00:39:13.029316+0000","last_became_active":"2024-02-13T00:37:47.253945+0000","last_became_peered":"2024-02-13T00:37:47.253945+0000","last_unstale":"2024-02-13T00:39:13.029316+0000","last_undegraded":"2024-02-13T00:39:13.029316+0000","last_fullsized":"2024-02-13T00:39:13.029316+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:37:46.228444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"84'9","reported_seq":73,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233379+0000","last_change":"2024-02-13T00:39:22.269192+0000","last_active":"2024-02-13T00:39:34.233379+0000","last_peered":"2024-02-13T00:39:34.233379+0000","last_clean":"2024-02-13T00:39:34.233379+0000","last_became_active":"2024-02-13T00:37:45.250598+0000","last_became_peered":"2024-02-13T00:37:45.250598+0000","last_unstale":"2024-02-13T00:39:34.233379+0000","last_undegraded":"2024-02-13T00:39:34.233379+0000","last_fullsized":"2024-02-13T00:39:34.233379+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.269104+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.242178+0000","last_change":"2024-02-13T00:38:58.314612+0000","last_active":"2024-02-13T00:39:12.242178+0000","last_peered":"2024-02-13T00:39:12.242178+0000","last_clean":"2024-02-13T00:39:12.242178+0000","last_became_active":"2024-02-13T00:37:41.217953+0000","last_became_peered":"2024-02-13T00:37:41.217953+0000","last_unstale":"2024-02-13T00:39:12.242178+0000","last_undegraded":"2024-02-13T00:39:12.242178+0000","last_fullsized":"2024-02-13T00:39:12.242178+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.314542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.314542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":71,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240366+0000","last_change":"2024-02-13T00:39:10.549128+0000","last_active":"2024-02-13T00:39:12.240366+0000","last_peered":"2024-02-13T00:39:12.240366+0000","last_clean":"2024-02-13T00:39:12.240366+0000","last_became_active":"2024-02-13T00:37:43.322357+0000","last_became_peered":"2024-02-13T00:37:43.322357+0000","last_unstale":"2024-02-13T00:39:12.240366+0000","last_undegraded":"2024-02-13T00:39:12.240366+0000","last_fullsized":"2024-02-13T00:39:12.240366+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:39:10.549045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.549045+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:21.344861+0000","last_change":"2024-02-13T00:39:21.344836+0000","last_active":"2024-02-13T00:39:21.344861+0000","last_peered":"2024-02-13T00:39:21.344861+0000","last_clean":"2024-02-13T00:39:21.344861+0000","last_became_active":"2024-02-13T00:37:47.252722+0000","last_became_peered":"2024-02-13T00:37:47.252722+0000","last_unstale":"2024-02-13T00:39:21.344861+0000","last_undegraded":"2024-02-13T00:39:21.344861+0000","last_fullsized":"2024-02-13T00:39:21.344861+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.344771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.344771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"64'8","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233394+0000","last_change":"2024-02-13T00:39:11.518468+0000","last_active":"2024-02-13T00:39:34.233394+0000","last_peered":"2024-02-13T00:39:34.233394+0000","last_clean":"2024-02-13T00:39:34.233394+0000","last_became_active":"2024-02-13T00:37:45.246757+0000","last_became_peered":"2024-02-13T00:37:45.246757+0000","last_unstale":"2024-02-13T00:39:34.233394+0000","last_undegraded":"2024-02-13T00:39:34.233394+0000","last_fullsized":"2024-02-13T00:39:34.233394+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:11.518359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:11.518359+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239760+0000","last_change":"2024-02-13T00:39:01.561276+0000","last_active":"2024-02-13T00:39:12.239760+0000","last_peered":"2024-02-13T00:39:12.239760+0000","last_clean":"2024-02-13T00:39:12.239760+0000","last_became_active":"2024-02-13T00:37:41.219309+0000","last_became_peered":"2024-02-13T00:37:41.219309+0000","last_unstale":"2024-02-13T00:39:12.239760+0000","last_undegraded":"2024-02-13T00:39:12.239760+0000","last_fullsized":"2024-02-13T00:39:12.239760+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.561157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.561157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":91,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239464+0000","last_change":"2024-02-13T00:38:46.169426+0000","last_active":"2024-02-13T00:39:12.239464+0000","last_peered":"2024-02-13T00:39:12.239464+0000","last_clean":"2024-02-13T00:39:12.239464+0000","last_became_active":"2024-02-13T00:37:43.324313+0000","last_became_peered":"2024-02-13T00:37:43.324313+0000","last_unstale":"2024-02-13T00:39:12.239464+0000","last_undegraded":"2024-02-13T00:39:12.239464+0000","last_fullsized":"2024-02-13T00:39:12.239464+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:38:46.169344+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:25.166852+0000","last_change":"2024-02-13T00:39:25.166852+0000","last_active":"2024-02-13T00:39:25.166852+0000","last_peered":"2024-02-13T00:39:25.166852+0000","last_clean":"2024-02-13T00:39:25.166852+0000","last_became_active":"2024-02-13T00:37:47.553612+0000","last_became_peered":"2024-02-13T00:37:47.553612+0000","last_unstale":"2024-02-13T00:39:25.166852+0000","last_undegraded":"2024-02-13T00:39:25.166852+0000","last_fullsized":"2024-02-13T00:39:25.166852+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:25.166777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.166777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240270+0000","last_change":"2024-02-13T00:39:07.556868+0000","last_active":"2024-02-13T00:39:12.240270+0000","last_peered":"2024-02-13T00:39:12.240270+0000","last_clean":"2024-02-13T00:39:12.240270+0000","last_became_active":"2024-02-13T00:37:45.246907+0000","last_became_peered":"2024-02-13T00:37:45.246907+0000","last_unstale":"2024-02-13T00:39:12.240270+0000","last_undegraded":"2024-02-13T00:39:12.240270+0000","last_fullsized":"2024-02-13T00:39:12.240270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.556772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.556772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.684024+0000","last_change":"2024-02-13T00:39:01.439623+0000","last_active":"2024-02-13T00:39:12.684024+0000","last_peered":"2024-02-13T00:39:12.684024+0000","last_clean":"2024-02-13T00:39:12.684024+0000","last_became_active":"2024-02-13T00:37:41.212866+0000","last_became_peered":"2024-02-13T00:37:41.212866+0000","last_unstale":"2024-02-13T00:39:12.684024+0000","last_undegraded":"2024-02-13T00:39:12.684024+0000","last_fullsized":"2024-02-13T00:39:12.684024+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.439513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":63,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:19.266831+0000","last_change":"2024-02-13T00:39:19.266801+0000","last_active":"2024-02-13T00:39:19.266831+0000","last_peered":"2024-02-13T00:39:19.266831+0000","last_clean":"2024-02-13T00:39:19.266831+0000","last_became_active":"2024-02-13T00:37:43.324510+0000","last_became_peered":"2024-02-13T00:37:43.324510+0000","last_unstale":"2024-02-13T00:39:19.266831+0000","last_undegraded":"2024-02-13T00:39:19.266831+0000","last_fullsized":"2024-02-13T00:39:19.266831+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:39:19.266729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:19.266729+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:22.615899+0000","last_change":"2024-02-13T00:39:22.615864+0000","last_active":"2024-02-13T00:39:22.615899+0000","last_peered":"2024-02-13T00:39:22.615899+0000","last_clean":"2024-02-13T00:39:22.615899+0000","last_became_active":"2024-02-13T00:37:47.254803+0000","last_became_peered":"2024-02-13T00:37:47.254803+0000","last_unstale":"2024-02-13T00:39:22.615899+0000","last_undegraded":"2024-02-13T00:39:22.615899+0000","last_fullsized":"2024-02-13T00:39:22.615899+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.615773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.615773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":32,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239687+0000","last_change":"2024-02-13T00:37:45.250983+0000","last_active":"2024-02-13T00:39:12.239687+0000","last_peered":"2024-02-13T00:39:12.239687+0000","last_clean":"2024-02-13T00:39:12.239687+0000","last_became_active":"2024-02-13T00:37:45.250632+0000","last_became_peered":"2024-02-13T00:37:45.250632+0000","last_unstale":"2024-02-13T00:39:12.239687+0000","last_undegraded":"2024-02-13T00:39:12.239687+0000","last_fullsized":"2024-02-13T00:39:12.239687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:37:44.215312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115163+0000","last_change":"2024-02-13T00:39:01.569173+0000","last_active":"2024-02-13T00:39:13.115163+0000","last_peered":"2024-02-13T00:39:13.115163+0000","last_clean":"2024-02-13T00:39:13.115163+0000","last_became_active":"2024-02-13T00:37:41.214091+0000","last_became_peered":"2024-02-13T00:37:41.214091+0000","last_unstale":"2024-02-13T00:39:13.115163+0000","last_undegraded":"2024-02-13T00:39:13.115163+0000","last_fullsized":"2024-02-13T00:39:13.115163+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.569109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.569109+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":85,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.242392+0000","last_change":"2024-02-13T00:38:56.331799+0000","last_active":"2024-02-13T00:39:12.242392+0000","last_peered":"2024-02-13T00:39:12.242392+0000","last_clean":"2024-02-13T00:39:12.242392+0000","last_became_active":"2024-02-13T00:37:43.322401+0000","last_became_peered":"2024-02-13T00:37:43.322401+0000","last_unstale":"2024-02-13T00:39:12.242392+0000","last_undegraded":"2024-02-13T00:39:12.242392+0000","last_fullsized":"2024-02-13T00:39:12.242392+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:38:56.331703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.331703+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:33.179309+0000","last_change":"2024-02-13T00:39:33.179309+0000","last_active":"2024-02-13T00:39:33.179309+0000","last_peered":"2024-02-13T00:39:33.179309+0000","last_clean":"2024-02-13T00:39:33.179309+0000","last_became_active":"2024-02-13T00:37:47.260242+0000","last_became_peered":"2024-02-13T00:37:47.260242+0000","last_unstale":"2024-02-13T00:39:33.179309+0000","last_undegraded":"2024-02-13T00:39:33.179309+0000","last_fullsized":"2024-02-13T00:39:33.179309+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.179225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.179225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.029231+0000","last_change":"2024-02-13T00:38:48.379316+0000","last_active":"2024-02-13T00:39:13.029231+0000","last_peered":"2024-02-13T00:39:13.029231+0000","last_clean":"2024-02-13T00:39:13.029231+0000","last_became_active":"2024-02-13T00:37:41.219249+0000","last_became_peered":"2024-02-13T00:37:41.219249+0000","last_unstale":"2024-02-13T00:39:13.029231+0000","last_undegraded":"2024-02-13T00:39:13.029231+0000","last_fullsized":"2024-02-13T00:39:13.029231+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:38:48.379184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:48.379184+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":87,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028724+0000","last_change":"2024-02-13T00:37:43.229199+0000","last_active":"2024-02-13T00:39:13.028724+0000","last_peered":"2024-02-13T00:39:13.028724+0000","last_clean":"2024-02-13T00:39:13.028724+0000","last_became_active":"2024-02-13T00:37:43.228898+0000","last_became_peered":"2024-02-13T00:37:43.228898+0000","last_unstale":"2024-02-13T00:39:13.028724+0000","last_undegraded":"2024-02-13T00:39:13.028724+0000","last_fullsized":"2024-02-13T00:39:13.028724+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:21.654867+0000","last_change":"2024-02-13T00:39:21.654834+0000","last_active":"2024-02-13T00:39:21.654867+0000","last_peered":"2024-02-13T00:39:21.654867+0000","last_clean":"2024-02-13T00:39:21.654867+0000","last_became_active":"2024-02-13T00:37:45.244408+0000","last_became_peered":"2024-02-13T00:37:45.244408+0000","last_unstale":"2024-02-13T00:39:21.654867+0000","last_undegraded":"2024-02-13T00:39:21.654867+0000","last_fullsized":"2024-02-13T00:39:21.654867+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.654733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.654733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.241786+0000","last_change":"2024-02-13T00:38:54.349240+0000","last_active":"2024-02-13T00:39:12.241786+0000","last_peered":"2024-02-13T00:39:12.241786+0000","last_clean":"2024-02-13T00:39:12.241786+0000","last_became_active":"2024-02-13T00:37:47.255928+0000","last_became_peered":"2024-02-13T00:37:47.255928+0000","last_unstale":"2024-02-13T00:39:12.241786+0000","last_undegraded":"2024-02-13T00:39:12.241786+0000","last_fullsized":"2024-02-13T00:39:12.241786+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:54.349171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:54.349171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.116015+0000","last_change":"2024-02-13T00:38:49.618099+0000","last_active":"2024-02-13T00:39:13.116015+0000","last_peered":"2024-02-13T00:39:13.116015+0000","last_clean":"2024-02-13T00:39:13.116015+0000","last_became_active":"2024-02-13T00:37:41.217777+0000","last_became_peered":"2024-02-13T00:37:41.217777+0000","last_unstale":"2024-02-13T00:39:13.116015+0000","last_undegraded":"2024-02-13T00:39:13.116015+0000","last_fullsized":"2024-02-13T00:39:13.116015+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:38:49.617988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115544+0000","last_change":"2024-02-13T00:39:05.974172+0000","last_active":"2024-02-13T00:39:13.115544+0000","last_peered":"2024-02-13T00:39:13.115544+0000","last_clean":"2024-02-13T00:39:13.115544+0000","last_became_active":"2024-02-13T00:39:05.965560+0000","last_became_peered":"2024-02-13T00:39:05.965560+0000","last_unstale":"2024-02-13T00:39:13.115544+0000","last_undegraded":"2024-02-13T00:39:13.115544+0000","last_fullsized":"2024-02-13T00:39:13.115544+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:39:01.456507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.456507+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"64'8","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233379+0000","last_change":"2024-02-13T00:39:30.273237+0000","last_active":"2024-02-13T00:39:34.233379+0000","last_peered":"2024-02-13T00:39:34.233379+0000","last_clean":"2024-02-13T00:39:34.233379+0000","last_became_active":"2024-02-13T00:37:45.241274+0000","last_became_peered":"2024-02-13T00:37:45.241274+0000","last_unstale":"2024-02-13T00:39:34.233379+0000","last_undegraded":"2024-02-13T00:39:34.233379+0000","last_fullsized":"2024-02-13T00:39:34.233379+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:30.273165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.273165+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:36.157777+0000","last_change":"2024-02-13T00:39:36.157777+0000","last_active":"2024-02-13T00:39:36.157777+0000","last_peered":"2024-02-13T00:39:36.157777+0000","last_clean":"2024-02-13T00:39:36.157777+0000","last_became_active":"2024-02-13T00:37:47.252270+0000","last_became_peered":"2024-02-13T00:37:47.252270+0000","last_unstale":"2024-02-13T00:39:36.157777+0000","last_undegraded":"2024-02-13T00:39:36.157777+0000","last_fullsized":"2024-02-13T00:39:36.157777+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:36.157693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:36.157693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239861+0000","last_change":"2024-02-13T00:38:58.528669+0000","last_active":"2024-02-13T00:39:12.239861+0000","last_peered":"2024-02-13T00:39:12.239861+0000","last_clean":"2024-02-13T00:39:12.239861+0000","last_became_active":"2024-02-13T00:37:41.212924+0000","last_became_peered":"2024-02-13T00:37:41.212924+0000","last_unstale":"2024-02-13T00:39:12.239861+0000","last_undegraded":"2024-02-13T00:39:12.239861+0000","last_fullsized":"2024-02-13T00:39:12.239861+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.528580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.528580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"80'45","reported_seq":138,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233366+0000","last_change":"2024-02-13T00:38:57.537373+0000","last_active":"2024-02-13T00:39:34.233366+0000","last_peered":"2024-02-13T00:39:34.233366+0000","last_clean":"2024-02-13T00:39:34.233366+0000","last_became_active":"2024-02-13T00:37:43.232014+0000","last_became_peered":"2024-02-13T00:37:43.232014+0000","last_unstale":"2024-02-13T00:39:34.233366+0000","last_undegraded":"2024-02-13T00:39:34.233366+0000","last_fullsized":"2024-02-13T00:39:34.233366+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'45","last_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_deep_scrub":"80'45","last_deep_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.537309+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.242412+0000","last_change":"2024-02-13T00:38:47.393545+0000","last_active":"2024-02-13T00:39:12.242412+0000","last_peered":"2024-02-13T00:39:12.242412+0000","last_clean":"2024-02-13T00:39:12.242412+0000","last_became_active":"2024-02-13T00:37:45.247012+0000","last_became_peered":"2024-02-13T00:37:45.247012+0000","last_unstale":"2024-02-13T00:39:12.242412+0000","last_undegraded":"2024-02-13T00:39:12.242412+0000","last_fullsized":"2024-02-13T00:39:12.242412+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:47.393477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:47.393477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.026985+0000","last_change":"2024-02-13T00:39:08.930351+0000","last_active":"2024-02-13T00:39:13.026985+0000","last_peered":"2024-02-13T00:39:13.026985+0000","last_clean":"2024-02-13T00:39:13.026985+0000","last_became_active":"2024-02-13T00:37:47.552693+0000","last_became_peered":"2024-02-13T00:37:47.552693+0000","last_unstale":"2024-02-13T00:39:13.026985+0000","last_undegraded":"2024-02-13T00:39:13.026985+0000","last_fullsized":"2024-02-13T00:39:13.026985+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:08.930286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.930286+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":159,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:38.515414+0000","last_change":"2024-02-13T00:39:10.287406+0000","last_active":"2024-02-13T00:39:38.515414+0000","last_peered":"2024-02-13T00:39:38.515414+0000","last_clean":"2024-02-13T00:39:38.515414+0000","last_became_active":"2024-02-13T00:37:41.215137+0000","last_became_peered":"2024-02-13T00:37:41.215137+0000","last_unstale":"2024-02-13T00:39:38.515414+0000","last_undegraded":"2024-02-13T00:39:38.515414+0000","last_fullsized":"2024-02-13T00:39:38.515414+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-13T00:39:10.287299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.287299+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:20.703811+0000","last_change":"2024-02-13T00:39:20.703787+0000","last_active":"2024-02-13T00:39:20.703811+0000","last_peered":"2024-02-13T00:39:20.703811+0000","last_clean":"2024-02-13T00:39:20.703811+0000","last_became_active":"2024-02-13T00:37:41.214442+0000","last_became_peered":"2024-02-13T00:37:41.214442+0000","last_unstale":"2024-02-13T00:39:20.703811+0000","last_undegraded":"2024-02-13T00:39:20.703811+0000","last_fullsized":"2024-02-13T00:39:20.703811+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:20.703700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:20.703700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":82,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:25.319173+0000","last_change":"2024-02-13T00:39:25.319173+0000","last_active":"2024-02-13T00:39:25.319173+0000","last_peered":"2024-02-13T00:39:25.319173+0000","last_clean":"2024-02-13T00:39:25.319173+0000","last_became_active":"2024-02-13T00:37:43.231120+0000","last_became_peered":"2024-02-13T00:37:43.231120+0000","last_unstale":"2024-02-13T00:39:25.319173+0000","last_undegraded":"2024-02-13T00:39:25.319173+0000","last_fullsized":"2024-02-13T00:39:25.319173+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:39:25.319066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.319066+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115331+0000","last_change":"2024-02-13T00:39:09.589876+0000","last_active":"2024-02-13T00:39:13.115331+0000","last_peered":"2024-02-13T00:39:13.115331+0000","last_clean":"2024-02-13T00:39:13.115331+0000","last_became_active":"2024-02-13T00:37:45.248383+0000","last_became_peered":"2024-02-13T00:37:45.248383+0000","last_unstale":"2024-02-13T00:39:13.115331+0000","last_undegraded":"2024-02-13T00:39:13.115331+0000","last_fullsized":"2024-02-13T00:39:13.115331+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:09.589813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:09.589813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240574+0000","last_change":"2024-02-13T00:39:05.596529+0000","last_active":"2024-02-13T00:39:12.240574+0000","last_peered":"2024-02-13T00:39:12.240574+0000","last_clean":"2024-02-13T00:39:12.240574+0000","last_became_active":"2024-02-13T00:37:47.253996+0000","last_became_peered":"2024-02-13T00:37:47.253996+0000","last_unstale":"2024-02-13T00:39:12.240574+0000","last_undegraded":"2024-02-13T00:39:12.240574+0000","last_fullsized":"2024-02-13T00:39:12.240574+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:39:05.596462+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239260+0000","last_change":"2024-02-13T00:39:04.221630+0000","last_active":"2024-02-13T00:39:12.239260+0000","last_peered":"2024-02-13T00:39:12.239260+0000","last_clean":"2024-02-13T00:39:12.239260+0000","last_became_active":"2024-02-13T00:37:41.216367+0000","last_became_peered":"2024-02-13T00:37:41.216367+0000","last_unstale":"2024-02-13T00:39:12.239260+0000","last_undegraded":"2024-02-13T00:39:12.239260+0000","last_fullsized":"2024-02-13T00:39:12.239260+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:39:04.221506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239887+0000","last_change":"2024-02-13T00:38:52.618768+0000","last_active":"2024-02-13T00:39:12.239887+0000","last_peered":"2024-02-13T00:39:12.239887+0000","last_clean":"2024-02-13T00:39:12.239887+0000","last_became_active":"2024-02-13T00:37:41.216101+0000","last_became_peered":"2024-02-13T00:37:41.216101+0000","last_unstale":"2024-02-13T00:39:12.239887+0000","last_undegraded":"2024-02-13T00:39:12.239887+0000","last_fullsized":"2024-02-13T00:39:12.239887+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.618696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.618696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.026331+0000","last_change":"2024-02-13T00:38:12.978952+0000","last_active":"2024-02-13T00:39:13.026331+0000","last_peered":"2024-02-13T00:39:13.026331+0000","last_clean":"2024-02-13T00:39:13.026331+0000","last_became_active":"2024-02-13T00:37:35.635369+0000","last_became_peered":"2024-02-13T00:37:35.635369+0000","last_unstale":"2024-02-13T00:39:13.026331+0000","last_undegraded":"2024-02-13T00:39:13.026331+0000","last_fullsized":"2024-02-13T00:39:13.026331+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:12.978865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:35:35.296696+0000","last_clean_scrub_stamp":"2024-02-13T00:38:12.978865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'14","reported_seq":62,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:15.415117+0000","last_change":"2024-02-13T00:39:15.415087+0000","last_active":"2024-02-13T00:39:15.415117+0000","last_peered":"2024-02-13T00:39:15.415117+0000","last_clean":"2024-02-13T00:39:15.415117+0000","last_became_active":"2024-02-13T00:37:43.324209+0000","last_became_peered":"2024-02-13T00:37:43.324209+0000","last_unstale":"2024-02-13T00:39:15.415117+0000","last_undegraded":"2024-02-13T00:39:15.415117+0000","last_fullsized":"2024-02-13T00:39:15.415117+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'14","last_scrub_stamp":"2024-02-13T00:39:15.415004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.415004+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:28.171808+0000","last_change":"2024-02-13T00:39:28.171808+0000","last_active":"2024-02-13T00:39:28.171808+0000","last_peered":"2024-02-13T00:39:28.171808+0000","last_clean":"2024-02-13T00:39:28.171808+0000","last_became_active":"2024-02-13T00:37:45.247708+0000","last_became_peered":"2024-02-13T00:37:45.247708+0000","last_unstale":"2024-02-13T00:39:28.171808+0000","last_undegraded":"2024-02-13T00:39:28.171808+0000","last_fullsized":"2024-02-13T00:39:28.171808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.171740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.171740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:32.255327+0000","last_change":"2024-02-13T00:39:32.255327+0000","last_active":"2024-02-13T00:39:32.255327+0000","last_peered":"2024-02-13T00:39:32.255327+0000","last_clean":"2024-02-13T00:39:32.255327+0000","last_became_active":"2024-02-13T00:37:47.262024+0000","last_became_peered":"2024-02-13T00:37:47.262024+0000","last_unstale":"2024-02-13T00:39:32.255327+0000","last_undegraded":"2024-02-13T00:39:32.255327+0000","last_fullsized":"2024-02-13T00:39:32.255327+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.255260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.241978+0000","last_change":"2024-02-13T00:38:55.353229+0000","last_active":"2024-02-13T00:39:12.241978+0000","last_peered":"2024-02-13T00:39:12.241978+0000","last_clean":"2024-02-13T00:39:12.241978+0000","last_became_active":"2024-02-13T00:37:41.217577+0000","last_became_peered":"2024-02-13T00:37:41.217577+0000","last_unstale":"2024-02-13T00:39:12.241978+0000","last_undegraded":"2024-02-13T00:39:12.241978+0000","last_fullsized":"2024-02-13T00:39:12.241978+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:55.353157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:55.353157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.114906+0000","last_change":"2024-02-13T00:39:05.968028+0000","last_active":"2024-02-13T00:39:13.114906+0000","last_peered":"2024-02-13T00:39:13.114906+0000","last_clean":"2024-02-13T00:39:13.114906+0000","last_became_active":"2024-02-13T00:39:05.967772+0000","last_became_peered":"2024-02-13T00:39:05.967772+0000","last_unstale":"2024-02-13T00:39:13.114906+0000","last_undegraded":"2024-02-13T00:39:13.114906+0000","last_fullsized":"2024-02-13T00:39:13.114906+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:44.893697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:44.893697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":80,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.242362+0000","last_change":"2024-02-13T00:39:08.344202+0000","last_active":"2024-02-13T00:39:12.242362+0000","last_peered":"2024-02-13T00:39:12.242362+0000","last_clean":"2024-02-13T00:39:12.242362+0000","last_became_active":"2024-02-13T00:39:05.968155+0000","last_became_peered":"2024-02-13T00:39:05.968155+0000","last_unstale":"2024-02-13T00:39:12.242362+0000","last_undegraded":"2024-02-13T00:39:12.242362+0000","last_fullsized":"2024-02-13T00:39:12.242362+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:39:08.344129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.344129+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240565+0000","last_change":"2024-02-13T00:38:48.618900+0000","last_active":"2024-02-13T00:39:12.240565+0000","last_peered":"2024-02-13T00:39:12.240565+0000","last_clean":"2024-02-13T00:39:12.240565+0000","last_became_active":"2024-02-13T00:37:45.245956+0000","last_became_peered":"2024-02-13T00:37:45.245956+0000","last_unstale":"2024-02-13T00:39:12.240565+0000","last_undegraded":"2024-02-13T00:39:12.240565+0000","last_fullsized":"2024-02-13T00:39:12.240565+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:48.618830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:48.618830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:33.304107+0000","last_change":"2024-02-13T00:39:33.304107+0000","last_active":"2024-02-13T00:39:33.304107+0000","last_peered":"2024-02-13T00:39:33.304107+0000","last_clean":"2024-02-13T00:39:33.304107+0000","last_became_active":"2024-02-13T00:37:47.552672+0000","last_became_peered":"2024-02-13T00:37:47.552672+0000","last_unstale":"2024-02-13T00:39:33.304107+0000","last_undegraded":"2024-02-13T00:39:33.304107+0000","last_fullsized":"2024-02-13T00:39:33.304107+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.303982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.303982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.027005+0000","last_change":"2024-02-13T00:38:46.927138+0000","last_active":"2024-02-13T00:39:13.027005+0000","last_peered":"2024-02-13T00:39:13.027005+0000","last_clean":"2024-02-13T00:39:13.027005+0000","last_became_active":"2024-02-13T00:37:41.221636+0000","last_became_peered":"2024-02-13T00:37:41.221636+0000","last_unstale":"2024-02-13T00:39:13.027005+0000","last_undegraded":"2024-02-13T00:39:13.027005+0000","last_fullsized":"2024-02-13T00:39:13.027005+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:46.927068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:46.927068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":99,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.242163+0000","last_change":"2024-02-13T00:39:04.318990+0000","last_active":"2024-02-13T00:39:12.242163+0000","last_peered":"2024-02-13T00:39:12.242163+0000","last_clean":"2024-02-13T00:39:12.242163+0000","last_became_active":"2024-02-13T00:37:43.322450+0000","last_became_peered":"2024-02-13T00:37:43.322450+0000","last_unstale":"2024-02-13T00:39:12.242163+0000","last_undegraded":"2024-02-13T00:39:12.242163+0000","last_fullsized":"2024-02-13T00:39:12.242163+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_clean_scrub_stamp":"2024-02-13T00:39:04.318923+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:20.451172+0000","last_change":"2024-02-13T00:39:20.451151+0000","last_active":"2024-02-13T00:39:20.451172+0000","last_peered":"2024-02-13T00:39:20.451172+0000","last_clean":"2024-02-13T00:39:20.451172+0000","last_became_active":"2024-02-13T00:37:45.245974+0000","last_became_peered":"2024-02-13T00:37:45.245974+0000","last_unstale":"2024-02-13T00:39:20.451172+0000","last_undegraded":"2024-02-13T00:39:20.451172+0000","last_fullsized":"2024-02-13T00:39:20.451172+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:20.451088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:20.451088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028568+0000","last_change":"2024-02-13T00:39:09.237388+0000","last_active":"2024-02-13T00:39:13.028568+0000","last_peered":"2024-02-13T00:39:13.028568+0000","last_clean":"2024-02-13T00:39:13.028568+0000","last_became_active":"2024-02-13T00:37:47.250291+0000","last_became_peered":"2024-02-13T00:37:47.250291+0000","last_unstale":"2024-02-13T00:39:13.028568+0000","last_undegraded":"2024-02-13T00:39:13.028568+0000","last_fullsized":"2024-02-13T00:39:13.028568+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:09.237312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:09.237312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:21.329271+0000","last_change":"2024-02-13T00:39:21.329248+0000","last_active":"2024-02-13T00:39:21.329271+0000","last_peered":"2024-02-13T00:39:21.329271+0000","last_clean":"2024-02-13T00:39:21.329271+0000","last_became_active":"2024-02-13T00:37:41.214202+0000","last_became_peered":"2024-02-13T00:37:41.214202+0000","last_unstale":"2024-02-13T00:39:21.329271+0000","last_undegraded":"2024-02-13T00:39:21.329271+0000","last_fullsized":"2024-02-13T00:39:21.329271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.329173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.329173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":101,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239164+0000","last_change":"2024-02-13T00:38:58.238608+0000","last_active":"2024-02-13T00:39:12.239164+0000","last_peered":"2024-02-13T00:39:12.239164+0000","last_clean":"2024-02-13T00:39:12.239164+0000","last_became_active":"2024-02-13T00:37:43.324275+0000","last_became_peered":"2024-02-13T00:37:43.324275+0000","last_unstale":"2024-02-13T00:39:12.239164+0000","last_undegraded":"2024-02-13T00:39:12.239164+0000","last_fullsized":"2024-02-13T00:39:12.239164+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:38:58.238526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.238526+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028541+0000","last_change":"2024-02-13T00:38:52.388152+0000","last_active":"2024-02-13T00:39:13.028541+0000","last_peered":"2024-02-13T00:39:13.028541+0000","last_clean":"2024-02-13T00:39:13.028541+0000","last_became_active":"2024-02-13T00:37:45.244203+0000","last_became_peered":"2024-02-13T00:37:45.244203+0000","last_unstale":"2024-02-13T00:39:13.028541+0000","last_undegraded":"2024-02-13T00:39:13.028541+0000","last_fullsized":"2024-02-13T00:39:13.028541+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.388093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.388093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.027224+0000","last_change":"2024-02-13T00:38:56.857630+0000","last_active":"2024-02-13T00:39:13.027224+0000","last_peered":"2024-02-13T00:39:13.027224+0000","last_clean":"2024-02-13T00:39:13.027224+0000","last_became_active":"2024-02-13T00:37:47.553554+0000","last_became_peered":"2024-02-13T00:37:47.553554+0000","last_unstale":"2024-02-13T00:39:13.027224+0000","last_undegraded":"2024-02-13T00:39:13.027224+0000","last_fullsized":"2024-02-13T00:39:13.027224+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:56.857534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.857534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:14.367248+0000","last_change":"2024-02-13T00:39:14.367203+0000","last_active":"2024-02-13T00:39:14.367248+0000","last_peered":"2024-02-13T00:39:14.367248+0000","last_clean":"2024-02-13T00:39:14.367248+0000","last_became_active":"2024-02-13T00:37:41.218313+0000","last_became_peered":"2024-02-13T00:37:41.218313+0000","last_unstale":"2024-02-13T00:39:14.367248+0000","last_undegraded":"2024-02-13T00:39:14.367248+0000","last_fullsized":"2024-02-13T00:39:14.367248+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:14.367115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:14.367115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.028670+0000","last_change":"2024-02-13T00:37:43.222098+0000","last_active":"2024-02-13T00:39:13.028670+0000","last_peered":"2024-02-13T00:39:13.028670+0000","last_clean":"2024-02-13T00:39:13.028670+0000","last_became_active":"2024-02-13T00:37:43.221753+0000","last_became_peered":"2024-02-13T00:37:43.221753+0000","last_unstale":"2024-02-13T00:39:13.028670+0000","last_undegraded":"2024-02-13T00:39:13.028670+0000","last_fullsized":"2024-02-13T00:39:13.028670+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"64'8","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233394+0000","last_change":"2024-02-13T00:39:16.304400+0000","last_active":"2024-02-13T00:39:34.233394+0000","last_peered":"2024-02-13T00:39:34.233394+0000","last_clean":"2024-02-13T00:39:34.233394+0000","last_became_active":"2024-02-13T00:37:45.245528+0000","last_became_peered":"2024-02-13T00:37:45.245528+0000","last_unstale":"2024-02-13T00:39:34.233394+0000","last_undegraded":"2024-02-13T00:39:34.233394+0000","last_fullsized":"2024-02-13T00:39:34.233394+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:16.304316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.304316+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239289+0000","last_change":"2024-02-13T00:38:59.188989+0000","last_active":"2024-02-13T00:39:12.239289+0000","last_peered":"2024-02-13T00:39:12.239289+0000","last_clean":"2024-02-13T00:39:12.239289+0000","last_became_active":"2024-02-13T00:37:47.552591+0000","last_became_peered":"2024-02-13T00:37:47.552591+0000","last_unstale":"2024-02-13T00:39:12.239289+0000","last_undegraded":"2024-02-13T00:39:12.239289+0000","last_fullsized":"2024-02-13T00:39:12.239289+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:59.188913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:59.188913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240485+0000","last_change":"2024-02-13T00:38:57.494754+0000","last_active":"2024-02-13T00:39:12.240485+0000","last_peered":"2024-02-13T00:39:12.240485+0000","last_clean":"2024-02-13T00:39:12.240485+0000","last_became_active":"2024-02-13T00:37:45.245897+0000","last_became_peered":"2024-02-13T00:37:45.245897+0000","last_unstale":"2024-02-13T00:39:12.240485+0000","last_undegraded":"2024-02-13T00:39:12.240485+0000","last_fullsized":"2024-02-13T00:39:12.240485+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.494653+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:29.159172+0000","last_change":"2024-02-13T00:39:29.159172+0000","last_active":"2024-02-13T00:39:29.159172+0000","last_peered":"2024-02-13T00:39:29.159172+0000","last_clean":"2024-02-13T00:39:29.159172+0000","last_became_active":"2024-02-13T00:37:41.221627+0000","last_became_peered":"2024-02-13T00:37:41.221627+0000","last_unstale":"2024-02-13T00:39:29.159172+0000","last_undegraded":"2024-02-13T00:39:29.159172+0000","last_fullsized":"2024-02-13T00:39:29.159172+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.159080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.159080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":73,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:29.275198+0000","last_change":"2024-02-13T00:39:29.275198+0000","last_active":"2024-02-13T00:39:29.275198+0000","last_peered":"2024-02-13T00:39:29.275198+0000","last_clean":"2024-02-13T00:39:29.275198+0000","last_became_active":"2024-02-13T00:37:43.224302+0000","last_became_peered":"2024-02-13T00:37:43.224302+0000","last_unstale":"2024-02-13T00:39:29.275198+0000","last_undegraded":"2024-02-13T00:39:29.275198+0000","last_fullsized":"2024-02-13T00:39:29.275198+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:29.275134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.275134+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.240172+0000","last_change":"2024-02-13T00:38:54.599127+0000","last_active":"2024-02-13T00:39:12.240172+0000","last_peered":"2024-02-13T00:39:12.240172+0000","last_clean":"2024-02-13T00:39:12.240172+0000","last_became_active":"2024-02-13T00:37:47.260477+0000","last_became_peered":"2024-02-13T00:37:47.260477+0000","last_unstale":"2024-02-13T00:39:12.240172+0000","last_undegraded":"2024-02-13T00:39:12.240172+0000","last_fullsized":"2024-02-13T00:39:12.240172+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:54.599057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:54.599057+0000","last_clean_scrub_stamp":"2024-02-13T00:38:54.599057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"64'8","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233393+0000","last_change":"2024-02-13T00:37:45.245491+0000","last_active":"2024-02-13T00:39:34.233393+0000","last_peered":"2024-02-13T00:39:34.233393+0000","last_clean":"2024-02-13T00:39:34.233393+0000","last_became_active":"2024-02-13T00:37:45.245183+0000","last_became_peered":"2024-02-13T00:37:45.245183+0000","last_unstale":"2024-02-13T00:39:34.233393+0000","last_undegraded":"2024-02-13T00:39:34.233393+0000","last_fullsized":"2024-02-13T00:39:34.233393+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:37:44.215312+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":48,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115712+0000","last_change":"2024-02-13T00:38:42.529745+0000","last_active":"2024-02-13T00:39:13.115712+0000","last_peered":"2024-02-13T00:39:13.115712+0000","last_clean":"2024-02-13T00:39:13.115712+0000","last_became_active":"2024-02-13T00:37:41.217979+0000","last_became_peered":"2024-02-13T00:37:41.217979+0000","last_unstale":"2024-02-13T00:39:13.115712+0000","last_undegraded":"2024-02-13T00:39:13.115712+0000","last_fullsized":"2024-02-13T00:39:13.115712+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:38:42.529666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:42.529666+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.370512+0000","last_change":"2024-02-13T00:39:23.370512+0000","last_active":"2024-02-13T00:39:23.370512+0000","last_peered":"2024-02-13T00:39:23.370512+0000","last_clean":"2024-02-13T00:39:23.370512+0000","last_became_active":"2024-02-13T00:37:43.232161+0000","last_became_peered":"2024-02-13T00:37:43.232161+0000","last_unstale":"2024-02-13T00:39:23.370512+0000","last_undegraded":"2024-02-13T00:39:23.370512+0000","last_fullsized":"2024-02-13T00:39:23.370512+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:39:23.370447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.370447+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.029022+0000","last_change":"2024-02-13T00:37:47.259820+0000","last_active":"2024-02-13T00:39:13.029022+0000","last_peered":"2024-02-13T00:39:13.029022+0000","last_clean":"2024-02-13T00:39:13.029022+0000","last_became_active":"2024-02-13T00:37:47.259517+0000","last_became_peered":"2024-02-13T00:37:47.259517+0000","last_unstale":"2024-02-13T00:39:13.029022+0000","last_undegraded":"2024-02-13T00:39:13.029022+0000","last_fullsized":"2024-02-13T00:39:13.029022+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:37:46.228444+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"64'8","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233366+0000","last_change":"2024-02-13T00:39:14.609925+0000","last_active":"2024-02-13T00:39:34.233366+0000","last_peered":"2024-02-13T00:39:34.233366+0000","last_clean":"2024-02-13T00:39:34.233366+0000","last_became_active":"2024-02-13T00:37:45.249684+0000","last_became_peered":"2024-02-13T00:37:45.249684+0000","last_unstale":"2024-02-13T00:39:34.233366+0000","last_undegraded":"2024-02-13T00:39:34.233366+0000","last_fullsized":"2024-02-13T00:39:34.233366+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:14.609850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:14.609850+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:17.284029+0000","last_change":"2024-02-13T00:39:17.283996+0000","last_active":"2024-02-13T00:39:17.284029+0000","last_peered":"2024-02-13T00:39:17.284029+0000","last_clean":"2024-02-13T00:39:17.284029+0000","last_became_active":"2024-02-13T00:37:41.218346+0000","last_became_peered":"2024-02-13T00:37:41.218346+0000","last_unstale":"2024-02-13T00:39:17.284029+0000","last_undegraded":"2024-02-13T00:39:17.284029+0000","last_fullsized":"2024-02-13T00:39:17.284029+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.283902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.283902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:28.305239+0000","last_change":"2024-02-13T00:39:28.305239+0000","last_active":"2024-02-13T00:39:28.305239+0000","last_peered":"2024-02-13T00:39:28.305239+0000","last_clean":"2024-02-13T00:39:28.305239+0000","last_became_active":"2024-02-13T00:37:43.227980+0000","last_became_peered":"2024-02-13T00:37:43.227980+0000","last_unstale":"2024-02-13T00:39:28.305239+0000","last_undegraded":"2024-02-13T00:39:28.305239+0000","last_fullsized":"2024-02-13T00:39:28.305239+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.305179+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115273+0000","last_change":"2024-02-13T00:39:07.563502+0000","last_active":"2024-02-13T00:39:13.115273+0000","last_peered":"2024-02-13T00:39:13.115273+0000","last_clean":"2024-02-13T00:39:13.115273+0000","last_became_active":"2024-02-13T00:37:47.256807+0000","last_became_peered":"2024-02-13T00:37:47.256807+0000","last_unstale":"2024-02-13T00:39:13.115273+0000","last_undegraded":"2024-02-13T00:39:13.115273+0000","last_fullsized":"2024-02-13T00:39:13.115273+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.563440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.563440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:35.269607+0000","last_change":"2024-02-13T00:39:35.269607+0000","last_active":"2024-02-13T00:39:35.269607+0000","last_peered":"2024-02-13T00:39:35.269607+0000","last_clean":"2024-02-13T00:39:35.269607+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:39:35.269607+0000","last_undegraded":"2024-02-13T00:39:35.269607+0000","last_fullsized":"2024-02-13T00:39:35.269607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:35.269538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:35.269538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138710+0000","last_change":"2024-02-13T00:39:24.138710+0000","last_active":"2024-02-13T00:39:24.138710+0000","last_peered":"2024-02-13T00:39:24.138710+0000","last_clean":"2024-02-13T00:39:24.138710+0000","last_became_active":"2024-02-13T00:37:41.220874+0000","last_became_peered":"2024-02-13T00:37:41.220874+0000","last_unstale":"2024-02-13T00:39:24.138710+0000","last_undegraded":"2024-02-13T00:39:24.138710+0000","last_fullsized":"2024-02-13T00:39:24.138710+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:39:24.138629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.138629+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":67,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.029183+0000","last_change":"2024-02-13T00:39:02.188162+0000","last_active":"2024-02-13T00:39:13.029183+0000","last_peered":"2024-02-13T00:39:13.029183+0000","last_clean":"2024-02-13T00:39:13.029183+0000","last_became_active":"2024-02-13T00:37:43.223574+0000","last_became_peered":"2024-02-13T00:37:43.223574+0000","last_unstale":"2024-02-13T00:39:13.029183+0000","last_undegraded":"2024-02-13T00:39:13.029183+0000","last_fullsized":"2024-02-13T00:39:13.029183+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:39:02.188096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.188096+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:16.086309+0000","last_change":"2024-02-13T00:39:16.086278+0000","last_active":"2024-02-13T00:39:16.086309+0000","last_peered":"2024-02-13T00:39:16.086309+0000","last_clean":"2024-02-13T00:39:16.086309+0000","last_became_active":"2024-02-13T00:37:47.254705+0000","last_became_peered":"2024-02-13T00:37:47.254705+0000","last_unstale":"2024-02-13T00:39:16.086309+0000","last_undegraded":"2024-02-13T00:39:16.086309+0000","last_fullsized":"2024-02-13T00:39:16.086309+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.086207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.086207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:17.632011+0000","last_change":"2024-02-13T00:39:17.631975+0000","last_active":"2024-02-13T00:39:17.632011+0000","last_peered":"2024-02-13T00:39:17.632011+0000","last_clean":"2024-02-13T00:39:17.632011+0000","last_became_active":"2024-02-13T00:37:45.239867+0000","last_became_peered":"2024-02-13T00:37:45.239867+0000","last_unstale":"2024-02-13T00:39:17.632011+0000","last_undegraded":"2024-02-13T00:39:17.632011+0000","last_fullsized":"2024-02-13T00:39:17.632011+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.631877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:18.672624+0000","last_change":"2024-02-13T00:39:18.672595+0000","last_active":"2024-02-13T00:39:18.672624+0000","last_peered":"2024-02-13T00:39:18.672624+0000","last_clean":"2024-02-13T00:39:18.672624+0000","last_became_active":"2024-02-13T00:37:41.219457+0000","last_became_peered":"2024-02-13T00:37:41.219457+0000","last_unstale":"2024-02-13T00:39:18.672624+0000","last_undegraded":"2024-02-13T00:39:18.672624+0000","last_fullsized":"2024-02-13T00:39:18.672624+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.672516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.672516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'25","reported_seq":86,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.684152+0000","last_change":"2024-02-13T00:38:56.371071+0000","last_active":"2024-02-13T00:39:12.684152+0000","last_peered":"2024-02-13T00:39:12.684152+0000","last_clean":"2024-02-13T00:39:12.684152+0000","last_became_active":"2024-02-13T00:37:43.322957+0000","last_became_peered":"2024-02-13T00:37:43.322957+0000","last_unstale":"2024-02-13T00:39:12.684152+0000","last_undegraded":"2024-02-13T00:39:12.684152+0000","last_fullsized":"2024-02-13T00:39:12.684152+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'25","last_scrub_stamp":"2024-02-13T00:38:56.370998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.370998+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:18.359703+0000","last_change":"2024-02-13T00:39:18.359681+0000","last_active":"2024-02-13T00:39:18.359703+0000","last_peered":"2024-02-13T00:39:18.359703+0000","last_clean":"2024-02-13T00:39:18.359703+0000","last_became_active":"2024-02-13T00:37:47.249556+0000","last_became_peered":"2024-02-13T00:37:47.249556+0000","last_unstale":"2024-02-13T00:39:18.359703+0000","last_undegraded":"2024-02-13T00:39:18.359703+0000","last_fullsized":"2024-02-13T00:39:18.359703+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.359604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:15.603914+0000","last_change":"2024-02-13T00:39:15.603878+0000","last_active":"2024-02-13T00:39:15.603914+0000","last_peered":"2024-02-13T00:39:15.603914+0000","last_clean":"2024-02-13T00:39:15.603914+0000","last_became_active":"2024-02-13T00:37:45.240923+0000","last_became_peered":"2024-02-13T00:37:45.240923+0000","last_unstale":"2024-02-13T00:39:15.603914+0000","last_undegraded":"2024-02-13T00:39:15.603914+0000","last_fullsized":"2024-02-13T00:39:15.603914+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.603808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.603808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:16.262471+0000","last_change":"2024-02-13T00:39:16.262449+0000","last_active":"2024-02-13T00:39:16.262471+0000","last_peered":"2024-02-13T00:39:16.262471+0000","last_clean":"2024-02-13T00:39:16.262471+0000","last_became_active":"2024-02-13T00:37:41.212429+0000","last_became_peered":"2024-02-13T00:37:41.212429+0000","last_unstale":"2024-02-13T00:39:16.262471+0000","last_undegraded":"2024-02-13T00:39:16.262471+0000","last_fullsized":"2024-02-13T00:39:16.262471+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.262380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.262380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.376175+0000","last_change":"2024-02-13T00:39:24.376175+0000","last_active":"2024-02-13T00:39:24.376175+0000","last_peered":"2024-02-13T00:39:24.376175+0000","last_clean":"2024-02-13T00:39:24.376175+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:39:24.376175+0000","last_undegraded":"2024-02-13T00:39:24.376175+0000","last_fullsized":"2024-02-13T00:39:24.376175+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:39:24.376100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.376100+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:17.713064+0000","last_change":"2024-02-13T00:39:17.713034+0000","last_active":"2024-02-13T00:39:17.713064+0000","last_peered":"2024-02-13T00:39:17.713064+0000","last_clean":"2024-02-13T00:39:17.713064+0000","last_became_active":"2024-02-13T00:37:47.552384+0000","last_became_peered":"2024-02-13T00:37:47.552384+0000","last_unstale":"2024-02-13T00:39:17.713064+0000","last_undegraded":"2024-02-13T00:39:17.713064+0000","last_fullsized":"2024-02-13T00:39:17.713064+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.712965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.712965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:13.115491+0000","last_change":"2024-02-13T00:39:02.545556+0000","last_active":"2024-02-13T00:39:13.115491+0000","last_peered":"2024-02-13T00:39:13.115491+0000","last_clean":"2024-02-13T00:39:13.115491+0000","last_became_active":"2024-02-13T00:37:41.219093+0000","last_became_peered":"2024-02-13T00:37:41.219093+0000","last_unstale":"2024-02-13T00:39:13.115491+0000","last_undegraded":"2024-02-13T00:39:13.115491+0000","last_fullsized":"2024-02-13T00:39:13.115491+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:02.545495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.545495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":103,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-13T00:39:12.239641+0000","last_change":"2024-02-13T00:38:59.548948+0000","last_active":"2024-02-13T00:39:12.239641+0000","last_peered":"2024-02-13T00:39:12.239641+0000","last_clean":"2024-02-13T00:39:12.239641+0000","last_became_active":"2024-02-13T00:37:43.322374+0000","last_became_peered":"2024-02-13T00:37:43.322374+0000","last_unstale":"2024-02-13T00:39:12.239641+0000","last_undegraded":"2024-02-13T00:39:12.239641+0000","last_fullsized":"2024-02-13T00:39:12.239641+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_clean_scrub_stamp":"2024-02-13T00:38:59.548872+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"64'8","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.233399+0000","last_change":"2024-02-13T00:39:15.080406+0000","last_active":"2024-02-13T00:39:34.233399+0000","last_peered":"2024-02-13T00:39:34.233399+0000","last_clean":"2024-02-13T00:39:34.233399+0000","last_became_active":"2024-02-13T00:37:45.247905+0000","last_became_peered":"2024-02-13T00:37:45.247905+0000","last_unstale":"2024-02-13T00:39:34.233399+0000","last_undegraded":"2024-02-13T00:39:34.233399+0000","last_fullsized":"2024-02-13T00:39:34.233399+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:15.080315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.080315+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":640,"ondisk_log_size":640,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266714,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297732,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447420,"statfs":{"total":95995035648,"available":95690158080,"internally_reserved":0,"allocated":651264,"data_stored":297684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.498,"15min":0.501},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.63800000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.505},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52300000000000002,"15min":0.52800000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.63,"5min":1.028,"15min":1.028},"last":0.625}]},{"osd":1,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.49099999999999999,"15min":0.49399999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.59899999999999998,"15min":0.61499999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64200000000000002,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.58199999999999996}]},{"osd":2,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.628,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61099999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.52800000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.55800000000000005,"15min":0.56799999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.64100000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.65700000000000003,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.57499999999999996}]},{"osd":4,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.46100000000000002,"15min":0.46999999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.65400000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.45600000000000002,"15min":0.46400000000000002},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67000000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.40200000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.63600000000000001,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46800000000000003,"15min":0.47199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65800000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.438}]},{"osd":6,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.49199999999999999,"15min":0.497},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68899999999999995,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.45400000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528352,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297708,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447444,"statfs":{"total":95995035648,"available":95690182656,"internally_reserved":0,"allocated":626688,"data_stored":296161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.46899999999999997,"15min":0.46800000000000003},"min":{"1min":0.35899999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.52900000000000003}]},{"osd":1,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.504,"15min":0.50600000000000001},"min":{"1min":0.41499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61099999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.67800000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.38}]},{"osd":2,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.497,"15min":0.5},"min":{"1min":0.34899999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.64100000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.439},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.626,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34399999999999997}]},{"osd":3,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.53200000000000003,"15min":0.54300000000000004},"min":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68999999999999995,"5min":1.181,"15min":1.181},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.42499999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.49099999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.39500000000000002},"min":{"1min":0.29999999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72199999999999998,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41699999999999998},"min":{"1min":0.31,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66700000000000004,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.47399999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.312,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61199999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.44700000000000001,"15min":0.442},"min":{"1min":0.34000000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.76300000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.51800000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.499,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67700000000000005,"5min":1.173,"15min":1.173},"last":0.504},{"interface":"front","average":{"1min":0.503,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79000000000000004,"5min":1.262,"15min":1.262},"last":0.54900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297760,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447392,"statfs":{"total":95995035648,"available":95690129408,"internally_reserved":0,"allocated":679936,"data_stored":297411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.32000000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.371},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.313},"min":{"1min":0.221,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.49099999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.28799999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:38:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.376,"15min":0.372},"min":{"1min":0.26200000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.624,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.28999999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.313}]},{"osd":3,"last update":"Tue Feb 13 00:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65000000000000002,"5min":0.748,"15min":0.748},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.45000000000000001,"15min":0.46500000000000002},"min":{"1min":0.29299999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.53900000000000003,"5min":0.745,"15min":0.745},"last":0.34100000000000003}]},{"osd":4,"last update":"Tue Feb 13 00:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.81299999999999994,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82499999999999996,"5min":1.02,"15min":1.02},"last":0.53500000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:38:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80100000000000005,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58299999999999996,"15min":0.58399999999999996},"min":{"1min":0.41599999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.497}]},{"osd":6,"last update":"Tue Feb 13 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78700000000000003,"5min":1.054,"15min":1.054},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86799999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.59399999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.78500000000000003,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.62,"15min":0.623},"min":{"1min":0.41599999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81899999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.61499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":647168,"data_stored":296465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:39:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34599999999999997,"15min":0.32800000000000001},"min":{"1min":0.254,"5min":0.192,"15min":0.192},"max":{"1min":0.52000000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.378,"5min":0.376,"15min":0.34399999999999997},"min":{"1min":0.23499999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.58899999999999997,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.59999999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:38:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38400000000000001,"15min":0.378},"min":{"1min":0.28999999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.746}]},{"osd":3,"last update":"Tue Feb 13 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.64900000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45600000000000002,"15min":0.46999999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.71499999999999997}]},{"osd":4,"last update":"Tue Feb 13 00:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.75600000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72099999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.89500000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.65800000000000003,"5min":0.88,"15min":0.88},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.74099999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.92100000000000004}]},{"osd":6,"last update":"Tue Feb 13 00:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.41599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54200000000000004,"15min":0.53700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.84299999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:39:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.80800000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411380,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":296732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:38:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.36599999999999999,"15min":0.35799999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.34499999999999997},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.44700000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:38:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.35799999999999998,"15min":0.34899999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.52100000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.38400000000000001,"15min":0.374},"min":{"1min":0.245,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47199999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:39:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.28000000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.52800000000000002,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.442},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59099999999999997,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.504}]},{"osd":4,"last update":"Tue Feb 13 00:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73699999999999999,"5min":0.873,"15min":0.873},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.58799999999999997,"15min":0.59199999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71699999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.55800000000000005}]},{"osd":5,"last update":"Tue Feb 13 00:38:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.61499999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58099999999999996,"15min":0.58399999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79400000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.495}]},{"osd":7,"last update":"Tue Feb 13 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58299999999999996,"15min":0.59099999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.748,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70999999999999996,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.59699999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182446,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297744,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447408,"statfs":{"total":95995035648,"available":95690145792,"internally_reserved":0,"allocated":663552,"data_stored":296994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.36099999999999999,"15min":0.376},"min":{"1min":0.221,"5min":0.191,"15min":0.191},"max":{"1min":0.55300000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.39300000000000002,"15min":0.40799999999999997},"min":{"1min":0.27800000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46200000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.22600000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38500000000000001,"15min":0.39800000000000002},"min":{"1min":0.28100000000000003,"5min":0.217,"15min":0.217},"max":{"1min":0.5,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.39100000000000001,"15min":0.39900000000000002},"min":{"1min":0.24399999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.56200000000000006,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.41399999999999998,"15min":0.42099999999999999},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58899999999999997,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39200000000000002,"15min":0.40300000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.52900000000000003,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.247}]},{"osd":4,"last update":"Tue Feb 13 00:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53400000000000003,"15min":0.54000000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.64700000000000002,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.53600000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.75,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.46800000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:38:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.50600000000000001,"15min":0.499},"min":{"1min":0.38200000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.498},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.52100000000000002,"15min":0.51600000000000001},"min":{"1min":0.41199999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.41799999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.46100000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:39:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.54400000000000004},"min":{"1min":0.372,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77800000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.435}]}]},{"osd":4,"up_from":31,"seq":133143986218,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297740,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447412,"statfs":{"total":95995035648,"available":95690149888,"internally_reserved":0,"allocated":659456,"data_stored":297505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67900000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.435},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.66500000000000004,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53500000000000003}]},{"osd":1,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.52100000000000002,"15min":0.53000000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.61499999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.47599999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69899999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52000000000000002,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.62,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.498}]},{"osd":3,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.498,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.38500000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.45800000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:38:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.35699999999999998,"15min":0.34699999999999998},"min":{"1min":0.28199999999999997,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.375,"15min":0.36199999999999999},"min":{"1min":0.316,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.40600000000000003}]},{"osd":6,"last update":"Tue Feb 13 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.32900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.41299999999999998,"15min":0.40400000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.57699999999999996}]},{"osd":7,"last update":"Tue Feb 13 00:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46100000000000002,"15min":0.45900000000000002},"min":{"1min":0.34200000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.33000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.63800000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.443}]}]},{"osd":5,"up_from":39,"seq":167503724581,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297716,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":634880,"data_stored":297285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.48699999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.53100000000000003,"15min":0.51200000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.0369999999999999,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.55800000000000005}]},{"osd":1,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.5,"15min":0.496},"min":{"1min":0.34599999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.52400000000000002,"15min":0.502},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":4.0839999999999996,"5min":4.0839999999999996,"15min":4.0839999999999996},"last":0.503}]},{"osd":2,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.51400000000000001,"15min":0.50600000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.52300000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.51400000000000001,"15min":0.505},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.38300000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.55400000000000005}]},{"osd":4,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.40600000000000003,"15min":0.36199999999999999},"min":{"1min":0.246,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.1210000000000004,"5min":4.1210000000000004,"15min":4.1210000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.46700000000000003,"15min":0.40699999999999997},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.6950000000000003,"5min":5.6950000000000003,"15min":5.6950000000000003},"last":0.48699999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.441,"15min":0.42699999999999999},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.873,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.46400000000000002,"15min":0.44700000000000001},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.327,"5min":1.377,"15min":1.377},"last":0.40899999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.755,"15min":0.80300000000000005},"min":{"1min":0.33200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.58499999999999996,"5min":7.5519999999999996,"15min":7.5519999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.55800000000000005,"15min":0.57399999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.64800000000000002,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.42899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:39:46.320 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-13T00:39:46.320 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-13T00:39:46.320 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-13T00:39:46.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph health --format=json 2024-02-13T00:39:46.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:45.252433+0000 mgr.y (mgr.14544) 85 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:46.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:45.590704+0000 mgr.y (mgr.14544) 86 : audit [DBG] from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:39:46.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:39:46] "GET /metrics HTTP/1.1" 200 172174 "" "Prometheus/2.33.4" 2024-02-13T00:39:46.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:45.252433+0000 mgr.y (mgr.14544) 85 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:46.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:45.590704+0000 mgr.y (mgr.14544) 86 : audit [DBG] from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:39:46.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:45.252433+0000 mgr.y (mgr.14544) 85 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:46.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:45.590704+0000 mgr.y (mgr.14544) 86 : audit [DBG] from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:39:46.586 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:47.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:46.154694+0000 osd.3 (osd.3) 49 : cluster [DBG] 5.17 scrub starts 2024-02-13T00:39:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:46.155712+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.17 scrub ok 2024-02-13T00:39:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:46.183309+0000 mgr.y (mgr.14544) 87 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:47.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:46.154694+0000 osd.3 (osd.3) 49 : cluster [DBG] 5.17 scrub starts 2024-02-13T00:39:47.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:46.155712+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.17 scrub ok 2024-02-13T00:39:47.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:46.183309+0000 mgr.y (mgr.14544) 87 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:46.154694+0000 osd.3 (osd.3) 49 : cluster [DBG] 5.17 scrub starts 2024-02-13T00:39:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:46.155712+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.17 scrub ok 2024-02-13T00:39:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:46.183309+0000 mgr.y (mgr.14544) 87 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:48.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.490+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/3775028205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f40f8500 msgr2=0x7ff6f4104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:48.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.490+0000 7ff6f9ea7700 1 --2- 172.21.15.162:0/3775028205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f40f8500 0x7ff6f4104bd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ff6ec009be0 tx=0x7ff6ec004d10).stop 2024-02-13T00:39:48.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.490+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/3775028205 shutdown_connections 2024-02-13T00:39:48.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.490+0000 7ff6f9ea7700 1 --2- 172.21.15.162:0/3775028205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f40f8500 0x7ff6f4104bd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:48.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.490+0000 7ff6f9ea7700 1 --2- 172.21.15.162:0/3775028205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6f40f9b70 0x7ff6f40f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:48.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.490+0000 7ff6f9ea7700 1 --2- 172.21.15.162:0/3775028205 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f40fa440 0x7ff6f40f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:48.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.490+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/3775028205 >> 172.21.15.162:0/3775028205 conn(0x7ff6f40f38c0 msgr2=0x7ff6f40f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.490+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/3775028205 shutdown_connections 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.490+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/3775028205 wait complete. 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.490+0000 7ff6f9ea7700 1 librados: starting msgr at 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.490+0000 7ff6f9ea7700 1 librados: starting objecter 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.490+0000 7ff6f9ea7700 1 Processor -- start 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f9ea7700 1 -- start start 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f9ea7700 1 librados: setting wanted keys 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f9ea7700 1 librados: calling monclient init 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f9ea7700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f40f8500 0x7ff6f410e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f9ea7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f40f9b70 0x7ff6f410e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f9ea7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6f40fa440 0x7ff6f41081d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f9ea7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f41086c0 con 0x7ff6f40fa440 2024-02-13T00:39:48.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f9ea7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f4108830 con 0x7ff6f40f9b70 2024-02-13T00:39:48.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f9ea7700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f41089f0 con 0x7ff6f40f8500 2024-02-13T00:39:48.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f37fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f40f8500 0x7ff6f410e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:48.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f37fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f40f8500 0x7ff6f410e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:34584/0 (socket says 172.21.15.162:34584) 2024-02-13T00:39:48.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f37fe700 1 -- 172.21.15.162:0/1814893605 learned_addr learned my addr 172.21.15.162:0/1814893605 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:48.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f3fff700 1 --2- 172.21.15.162:0/1814893605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6f40fa440 0x7ff6f41081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:48.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f2ffd700 1 --2- 172.21.15.162:0/1814893605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f40f9b70 0x7ff6f410e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:48.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f37fe700 1 -- 172.21.15.162:0/1814893605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f40f9b70 msgr2=0x7ff6f410e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:48.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f37fe700 1 --2- 172.21.15.162:0/1814893605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f40f9b70 0x7ff6f410e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:48.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f37fe700 1 -- 172.21.15.162:0/1814893605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6f40fa440 msgr2=0x7ff6f41081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:48.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f37fe700 1 --2- 172.21.15.162:0/1814893605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6f40fa440 0x7ff6f41081d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:48.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f37fe700 1 -- 172.21.15.162:0/1814893605 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc005040 con 0x7ff6f40f8500 2024-02-13T00:39:48.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f2ffd700 1 --2- 172.21.15.162:0/1814893605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f40f9b70 0x7ff6f410e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:39:48.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f3fff700 1 --2- 172.21.15.162:0/1814893605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6f40fa440 0x7ff6f41081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:39:48.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f37fe700 1 --2- 172.21.15.162:0/1814893605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f40f8500 0x7ff6f410e030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7ff6dc0048f0 tx=0x7ff6dc009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:48.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f0ff9700 10 client.?.objecter ms_handle_connect 0x7ff6f40f8500 2024-02-13T00:39:48.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f0ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:48.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f0ff9700 1 -- 172.21.15.162:0/1814893605 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc007b00 con 0x7ff6f40f8500 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/1814893605 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ec0097c0 con 0x7ff6f40f8500 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.491+0000 7ff6f0ff9700 1 -- 172.21.15.162:0/1814893605 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6dc00f040 con 0x7ff6f40f8500 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.492+0000 7ff6f0ff9700 1 -- 172.21.15.162:0/1814893605 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc014a00 con 0x7ff6f40f8500 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.492+0000 7ff6f9ea7700 10 client.24644.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.492+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/1814893605 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f4109030 con 0x7ff6f40f8500 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.492+0000 7ff6f9ea7700 1 librados: init done 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.493+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/1814893605 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f40475e0 con 0x7ff6f40f8500 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.493+0000 7ff6f0ff9700 1 -- 172.21.15.162:0/1814893605 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff6dc00e580 con 0x7ff6f40f8500 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.493+0000 7ff6f0ff9700 1 --2- 172.21.15.162:0/1814893605 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff6e0062c90 0x7ff6e0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.493+0000 7ff6f0ff9700 1 -- 172.21.15.162:0/1814893605 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(86..86 src has 30..86) v4 ==== 6172+0+0 (secure 0 0 0) 0x7ff6dc07c5e0 con 0x7ff6f40f8500 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.493+0000 7ff6f0ff9700 10 client.24644.objecter ms_dispatch 0x7ff6f4001550 osd_map(86..86 src has 30..86) v4 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.493+0000 7ff6f0ff9700 3 client.24644.objecter handle_osd_map got epochs [86,86] > 0 2024-02-13T00:39:48.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.493+0000 7ff6f0ff9700 3 client.24644.objecter handle_osd_map decoding full epoch 86 2024-02-13T00:39:48.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.493+0000 7ff6f2ffd700 1 --2- 172.21.15.162:0/1814893605 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff6e0062c90 0x7ff6e0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:48.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.493+0000 7ff6f0ff9700 20 client.24644.objecter dump_active .. 0 homeless 2024-02-13T00:39:48.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.494+0000 7ff6f0ff9700 10 client.24644.objecter ms_handle_connect 0x7ff6e0062c90 2024-02-13T00:39:48.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.494+0000 7ff6f2ffd700 1 --2- 172.21.15.162:0/1814893605 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff6e0062c90 0x7ff6e0065150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff6e4007fd0 tx=0x7ff6e4009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:48.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.498+0000 7ff6f0ff9700 1 -- 172.21.15.162:0/1814893605 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff6dc049f60 con 0x7ff6f40f8500 2024-02-13T00:39:48.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.698+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/1814893605 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff6f41096b0 con 0x7ff6f40f8500 2024-02-13T00:39:48.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.699+0000 7ff6f0ff9700 1 -- 172.21.15.162:0/1814893605 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7ff6dc050250 con 0x7ff6f40f8500 2024-02-13T00:39:48.702 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:39:48.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 10 librados: watch_flush enter 2024-02-13T00:39:48.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 10 librados: watch_flush exit 2024-02-13T00:39:48.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 20 client.24644.objecter shutdown clearing up homeless session... 2024-02-13T00:39:48.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 10 client.24644.objecter successfully canceled tick 2024-02-13T00:39:48.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/1814893605 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff6e0062c90 msgr2=0x7ff6e0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:48.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 1 --2- 172.21.15.162:0/1814893605 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff6e0062c90 0x7ff6e0065150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff6e4007fd0 tx=0x7ff6e4009040).stop 2024-02-13T00:39:48.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/1814893605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f40f8500 msgr2=0x7ff6f410e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:48.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 1 --2- 172.21.15.162:0/1814893605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f40f8500 0x7ff6f410e030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7ff6dc0048f0 tx=0x7ff6dc009cb0).stop 2024-02-13T00:39:48.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/1814893605 shutdown_connections 2024-02-13T00:39:48.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 1 --2- 172.21.15.162:0/1814893605 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff6e0062c90 0x7ff6e0065150 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:48.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 1 --2- 172.21.15.162:0/1814893605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6f40fa440 0x7ff6f41081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:48.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 1 --2- 172.21.15.162:0/1814893605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f40f8500 0x7ff6f410e030 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:48.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 1 --2- 172.21.15.162:0/1814893605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f40f9b70 0x7ff6f410e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:48.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.700+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/1814893605 >> 172.21.15.162:0/1814893605 conn(0x7ff6f40f38c0 msgr2=0x7ff6f40f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:48.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.701+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/1814893605 shutdown_connections 2024-02-13T00:39:48.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.701+0000 7ff6f9ea7700 1 -- 172.21.15.162:0/1814893605 wait complete. 2024-02-13T00:39:48.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:48.701+0000 7ff6f9ea7700 1 librados: shutdown 2024-02-13T00:39:49.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:39:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:39:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:39:49.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:48.183752+0000 mgr.y (mgr.14544) 88 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:49.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:48.699675+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? 172.21.15.162:0/1814893605' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-13T00:39:49.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:48.183752+0000 mgr.y (mgr.14544) 88 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:49.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:48.699675+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? 172.21.15.162:0/1814893605' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-13T00:39:49.464 INFO:teuthology.orchestra.run.smithi162.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-13T00:39:49.464 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-13T00:39:49.464 INFO:tasks.cephadm:Setup complete, yielding 2024-02-13T00:39:49.465 INFO:teuthology.run_tasks:Running task thrashosds... 2024-02-13T00:39:49.479 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-13T00:39:49.479 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri10-testing-2024-02-08-0854-pacific'}, 'ceph': {'conf': {'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', 'flag\\(s\\) set', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(PG_', '\\(OSD_', {'mons down': None}, 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-tracker-64343-pacific', 'sha1': '84e714279aa684636ac645daf0a1a85c2094cfb3'}} 2024-02-13T00:39:49.479 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-13T00:39:49.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-02-13T00:39:49.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:48.183752+0000 mgr.y (mgr.14544) 88 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:49.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:48.699675+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? 172.21.15.162:0/1814893605' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-13T00:39:49.746 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:51.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.442+0000 7fc81670e700 1 -- 172.21.15.162:0/2655665056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8100fd490 msgr2=0x7fc8100f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:51.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.442+0000 7fc81670e700 1 --2- 172.21.15.162:0/2655665056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8100fd490 0x7fc8100f6cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fc804004870 tx=0x7fc804009cb0).stop 2024-02-13T00:39:51.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.442+0000 7fc81670e700 1 -- 172.21.15.162:0/2655665056 shutdown_connections 2024-02-13T00:39:51.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.442+0000 7fc81670e700 1 --2- 172.21.15.162:0/2655665056 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8100f73a0 0x7fc8101039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:51.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.442+0000 7fc81670e700 1 --2- 172.21.15.162:0/2655665056 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8100fcb30 0x7fc8100fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:51.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.442+0000 7fc81670e700 1 --2- 172.21.15.162:0/2655665056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8100fd490 0x7fc8100f6cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.442+0000 7fc81670e700 1 -- 172.21.15.162:0/2655665056 >> 172.21.15.162:0/2655665056 conn(0x7fc81006eb90 msgr2=0x7fc8100f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.442+0000 7fc81670e700 1 -- 172.21.15.162:0/2655665056 shutdown_connections 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.442+0000 7fc81670e700 1 -- 172.21.15.162:0/2655665056 wait complete. 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.443+0000 7fc81670e700 1 librados: starting msgr at 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.443+0000 7fc81670e700 1 librados: starting objecter 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.443+0000 7fc81670e700 1 Processor -- start 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.443+0000 7fc81670e700 1 -- start start 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.443+0000 7fc81670e700 1 librados: setting wanted keys 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.443+0000 7fc81670e700 1 librados: calling monclient init 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.443+0000 7fc81670e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8100f73a0 0x7fc8101091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.443+0000 7fc81670e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8100fcb30 0x7fc8101096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.443+0000 7fc81670e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8100fd490 0x7fc810109c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.443+0000 7fc81670e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8101143a0 con 0x7fc8100f73a0 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.443+0000 7fc81670e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc810114510 con 0x7fc8100fd490 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.443+0000 7fc81670e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8101146d0 con 0x7fc8100fcb30 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8100fcb30 0x7fc8101096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8100fcb30 0x7fc8101096a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:34612/0 (socket says 172.21.15.162:34612) 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80f7fe700 1 -- 172.21.15.162:0/30397619 learned_addr learned my addr 172.21.15.162:0/30397619 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80ffff700 1 --2- 172.21.15.162:0/30397619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8100f73a0 0x7fc8101091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc814cab700 1 --2- 172.21.15.162:0/30397619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8100fd490 0x7fc810109c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80f7fe700 1 -- 172.21.15.162:0/30397619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8100fd490 msgr2=0x7fc810109c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80f7fe700 1 --2- 172.21.15.162:0/30397619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8100fd490 0x7fc810109c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80f7fe700 1 -- 172.21.15.162:0/30397619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8100f73a0 msgr2=0x7fc8101091b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80f7fe700 1 --2- 172.21.15.162:0/30397619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8100f73a0 0x7fc8101091b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80f7fe700 1 -- 172.21.15.162:0/30397619 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8080097c0 con 0x7fc8100fcb30 2024-02-13T00:39:51.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80ffff700 1 --2- 172.21.15.162:0/30397619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8100f73a0 0x7fc8101091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80f7fe700 1 --2- 172.21.15.162:0/30397619 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8100fcb30 0x7fc8101096a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fc804009eb0 tx=0x7fc80400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80d7fa700 10 client.?.objecter ms_handle_connect 0x7fc8100fcb30 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80d7fa700 1 -- 172.21.15.162:0/30397619 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80400e070 con 0x7fc8100fcb30 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80d7fa700 1 -- 172.21.15.162:0/30397619 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc804013da0 con 0x7fc8100fcb30 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc81670e700 1 -- 172.21.15.162:0/30397619 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc804005040 con 0x7fc8100fcb30 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc80d7fa700 1 -- 172.21.15.162:0/30397619 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc804019070 con 0x7fc8100fcb30 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc81670e700 10 client.24650.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc81670e700 1 -- 172.21.15.162:0/30397619 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc810114ca0 con 0x7fc8100fcb30 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.444+0000 7fc81670e700 1 librados: init done 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.445+0000 7fc81670e700 1 -- 172.21.15.162:0/30397619 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8100475e0 con 0x7fc8100fcb30 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.445+0000 7fc80d7fa700 1 -- 172.21.15.162:0/30397619 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc8040090d0 con 0x7fc8100fcb30 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.446+0000 7fc80d7fa700 1 --2- 172.21.15.162:0/30397619 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc7f8062c90 0x7fc7f8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:51.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.446+0000 7fc80d7fa700 1 -- 172.21.15.162:0/30397619 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(86..86 src has 30..86) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fc80407d630 con 0x7fc8100fcb30 2024-02-13T00:39:51.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.446+0000 7fc80d7fa700 10 client.24650.objecter ms_dispatch 0x7fc810001550 osd_map(86..86 src has 30..86) v4 2024-02-13T00:39:51.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.446+0000 7fc80d7fa700 3 client.24650.objecter handle_osd_map got epochs [86,86] > 0 2024-02-13T00:39:51.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.446+0000 7fc80d7fa700 3 client.24650.objecter handle_osd_map decoding full epoch 86 2024-02-13T00:39:51.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.446+0000 7fc80ffff700 1 --2- 172.21.15.162:0/30397619 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc7f8062c90 0x7fc7f8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:51.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.446+0000 7fc80d7fa700 20 client.24650.objecter dump_active .. 0 homeless 2024-02-13T00:39:51.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.446+0000 7fc80d7fa700 10 client.24650.objecter ms_handle_connect 0x7fc7f8062c90 2024-02-13T00:39:51.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.446+0000 7fc80ffff700 1 --2- 172.21.15.162:0/30397619 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc7f8062c90 0x7fc7f8065150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fc8000041d0 tx=0x7fc800009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:51.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.450+0000 7fc80d7fa700 1 -- 172.21.15.162:0/30397619 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc8040512a0 con 0x7fc8100fcb30 2024-02-13T00:39:51.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:50.184349+0000 mgr.y (mgr.14544) 89 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:51.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:50.184349+0000 mgr.y (mgr.14544) 89 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:51.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:50.184349+0000 mgr.y (mgr.14544) 89 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:51.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.618+0000 7fc81670e700 1 -- 172.21.15.162:0/30397619 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) v1 -- 0x7fc81005fb80 con 0x7fc8100fcb30 2024-02-13T00:39:51.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.625+0000 7fc80d7fa700 1 -- 172.21.15.162:0/30397619 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v27)=0 v27) v1 ==== 136+0+0 (secure 0 0 0) 0x7fc8040230c0 con 0x7fc8100fcb30 2024-02-13T00:39:51.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 10 librados: watch_flush enter 2024-02-13T00:39:51.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 10 librados: watch_flush exit 2024-02-13T00:39:51.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 20 client.24650.objecter shutdown clearing up homeless session... 2024-02-13T00:39:51.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 10 client.24650.objecter successfully canceled tick 2024-02-13T00:39:51.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 -- 172.21.15.162:0/30397619 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc7f8062c90 msgr2=0x7fc7f8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:51.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 --2- 172.21.15.162:0/30397619 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc7f8062c90 0x7fc7f8065150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fc8000041d0 tx=0x7fc800009040).stop 2024-02-13T00:39:51.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 -- 172.21.15.162:0/30397619 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8100fcb30 msgr2=0x7fc8101096a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:51.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 --2- 172.21.15.162:0/30397619 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8100fcb30 0x7fc8101096a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fc804009eb0 tx=0x7fc80400b040).stop 2024-02-13T00:39:51.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 -- 172.21.15.162:0/30397619 shutdown_connections 2024-02-13T00:39:51.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 --2- 172.21.15.162:0/30397619 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc7f8062c90 0x7fc7f8065150 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:51.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 --2- 172.21.15.162:0/30397619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8100fd490 0x7fc810109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:51.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 --2- 172.21.15.162:0/30397619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8100f73a0 0x7fc8101091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:51.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 --2- 172.21.15.162:0/30397619 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8100fcb30 0x7fc8101096a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:51.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 -- 172.21.15.162:0/30397619 >> 172.21.15.162:0/30397619 conn(0x7fc81006eb90 msgr2=0x7fc8100fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:51.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 -- 172.21.15.162:0/30397619 shutdown_connections 2024-02-13T00:39:51.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 -- 172.21.15.162:0/30397619 wait complete. 2024-02-13T00:39:51.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:51.628+0000 7fc81670e700 1 librados: shutdown 2024-02-13T00:39:52.063 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:51] ENGINE Bus STOPPING 2024-02-13T00:39:52.064 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T00:39:52.064 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:51] ENGINE Bus STOPPED 2024-02-13T00:39:52.064 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:51] ENGINE Bus STARTING 2024-02-13T00:39:52.064 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:51] ENGINE Serving on http://:::9283 2024-02-13T00:39:52.064 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:39:51] ENGINE Bus STARTED 2024-02-13T00:39:52.461 INFO:tasks.thrashosds:Beginning thrashosds... 2024-02-13T00:39:52.461 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-13T00:39:52.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:39:52.733 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:52.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:51.623596+0000 mon.a (mon.0) 713 : audit [INF] from='client.? ' entity='client.admin' 2024-02-13T00:39:52.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:51.664974+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:39:52.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:51.666358+0000 mon.c (mon.2) 78 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:39:52.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:51.666915+0000 mon.c (mon.2) 79 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:39:52.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:51.673982+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:51.623596+0000 mon.a (mon.0) 713 : audit [INF] from='client.? ' entity='client.admin' 2024-02-13T00:39:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:51.664974+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:39:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:51.666358+0000 mon.c (mon.2) 78 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:39:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:51.666915+0000 mon.c (mon.2) 79 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:39:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:51.673982+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:53.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:51.623596+0000 mon.a (mon.0) 713 : audit [INF] from='client.? ' entity='client.admin' 2024-02-13T00:39:53.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:51.664974+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:39:53.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:51.666358+0000 mon.c (mon.2) 78 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:39:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:51.666915+0000 mon.c (mon.2) 79 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:39:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:51.673982+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:39:53.949 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:52.184798+0000 mgr.y (mgr.14544) 90 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:53.950 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:52.202363+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:39:53.950 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:52.205919+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:39:53.950 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:53.110164+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.19 deep-scrub starts 2024-02-13T00:39:53.950 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:53.118984+0000 osd.3 (osd.3) 52 : cluster [DBG] 4.19 deep-scrub ok 2024-02-13T00:39:53.950 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:52.184798+0000 mgr.y (mgr.14544) 90 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:53.950 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:52.202363+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:39:53.950 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:52.205919+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:39:53.951 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:53.110164+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.19 deep-scrub starts 2024-02-13T00:39:53.951 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:53.118984+0000 osd.3 (osd.3) 52 : cluster [DBG] 4.19 deep-scrub ok 2024-02-13T00:39:54.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:52.184798+0000 mgr.y (mgr.14544) 90 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:52.202363+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:39:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:52.205919+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:39:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:53.110164+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.19 deep-scrub starts 2024-02-13T00:39:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:53.118984+0000 osd.3 (osd.3) 52 : cluster [DBG] 4.19 deep-scrub ok 2024-02-13T00:39:54.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.439+0000 7f1bacdcf700 1 -- 172.21.15.162:0/654486051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ba80f73a0 msgr2=0x7f1ba81039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:54.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.439+0000 7f1bacdcf700 1 --2- 172.21.15.162:0/654486051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ba80f73a0 0x7f1ba81039a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f1ba0009be0 tx=0x7f1ba0004d10).stop 2024-02-13T00:39:54.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.439+0000 7f1bacdcf700 1 -- 172.21.15.162:0/654486051 shutdown_connections 2024-02-13T00:39:54.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.439+0000 7f1bacdcf700 1 --2- 172.21.15.162:0/654486051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ba80f73a0 0x7f1ba81039a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:54.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.439+0000 7f1bacdcf700 1 --2- 172.21.15.162:0/654486051 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ba80fcb30 0x7f1ba80fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:54.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.439+0000 7f1bacdcf700 1 --2- 172.21.15.162:0/654486051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba80fd490 0x7f1ba80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:54.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.439+0000 7f1bacdcf700 1 -- 172.21.15.162:0/654486051 >> 172.21.15.162:0/654486051 conn(0x7f1ba806eb90 msgr2=0x7f1ba80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.439+0000 7f1bacdcf700 1 -- 172.21.15.162:0/654486051 shutdown_connections 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.439+0000 7f1bacdcf700 1 -- 172.21.15.162:0/654486051 wait complete. 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1bacdcf700 1 librados: starting msgr at 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1bacdcf700 1 librados: starting objecter 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1bacdcf700 1 Processor -- start 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1bacdcf700 1 -- start start 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1bacdcf700 1 librados: setting wanted keys 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1bacdcf700 1 librados: calling monclient init 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1bacdcf700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba80f73a0 0x7f1ba8106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1bacdcf700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ba80fcb30 0x7f1ba81073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1bacdcf700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ba80fd490 0x7f1ba81078b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1bacdcf700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba8116550 con 0x7f1ba80fd490 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1bacdcf700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba8116690 con 0x7f1ba80f73a0 2024-02-13T00:39:54.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1bacdcf700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba8116820 con 0x7f1ba80fcb30 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1ba6d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ba80fd490 0x7f1ba81078b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1ba5d9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ba80fcb30 0x7f1ba81073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1ba6d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ba80fd490 0x7f1ba81078b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43666/0 (socket says 172.21.15.162:43666) 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1ba5d9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ba80fcb30 0x7f1ba81073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:41922/0 (socket says 172.21.15.162:41922) 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.440+0000 7f1ba6d9d700 1 -- 172.21.15.162:0/223188234 learned_addr learned my addr 172.21.15.162:0/223188234 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1ba6d9d700 1 -- 172.21.15.162:0/223188234 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ba80fcb30 msgr2=0x7f1ba81073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1ba6d9d700 1 --2- 172.21.15.162:0/223188234 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ba80fcb30 0x7f1ba81073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1ba659c700 1 --2- 172.21.15.162:0/223188234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba80f73a0 0x7f1ba8106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1ba6d9d700 1 -- 172.21.15.162:0/223188234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba80f73a0 msgr2=0x7f1ba8106ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1ba6d9d700 1 --2- 172.21.15.162:0/223188234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba80f73a0 0x7f1ba8106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1ba6d9d700 1 -- 172.21.15.162:0/223188234 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c005040 con 0x7f1ba80fd490 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1ba5d9b700 1 --2- 172.21.15.162:0/223188234 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ba80fcb30 0x7f1ba81073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1ba6d9d700 1 --2- 172.21.15.162:0/223188234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ba80fd490 0x7f1ba81078b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f1ba000b6d0 tx=0x7f1ba0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:54.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1b977fe700 10 client.?.objecter ms_handle_connect 0x7f1ba80fd490 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1b977fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1b977fe700 1 -- 172.21.15.162:0/223188234 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba000e040 con 0x7f1ba80fd490 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1bacdcf700 1 -- 172.21.15.162:0/223188234 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba00097c0 con 0x7f1ba80fd490 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1b977fe700 1 -- 172.21.15.162:0/223188234 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ba0018040 con 0x7f1ba80fd490 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1b977fe700 1 -- 172.21.15.162:0/223188234 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba001c3f0 con 0x7f1ba80fd490 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1bacdcf700 10 client.14757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1bacdcf700 1 -- 172.21.15.162:0/223188234 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba8116d10 con 0x7f1ba80fd490 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.441+0000 7f1bacdcf700 1 librados: init done 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.442+0000 7f1bacdcf700 1 -- 172.21.15.162:0/223188234 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba80475e0 con 0x7f1ba80fd490 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.442+0000 7f1b977fe700 1 -- 172.21.15.162:0/223188234 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1ba0013420 con 0x7f1ba80fd490 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.443+0000 7f1b977fe700 1 --2- 172.21.15.162:0/223188234 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1b90062c90 0x7f1b90065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.443+0000 7f1b977fe700 1 -- 172.21.15.162:0/223188234 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(86..86 src has 30..86) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f1ba007dba0 con 0x7f1ba80fd490 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.443+0000 7f1b977fe700 10 client.14757.objecter ms_dispatch 0x7f1ba8001550 osd_map(86..86 src has 30..86) v4 2024-02-13T00:39:54.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.443+0000 7f1b977fe700 3 client.14757.objecter handle_osd_map got epochs [86,86] > 0 2024-02-13T00:39:54.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.443+0000 7f1b977fe700 3 client.14757.objecter handle_osd_map decoding full epoch 86 2024-02-13T00:39:54.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.443+0000 7f1ba659c700 1 --2- 172.21.15.162:0/223188234 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1b90062c90 0x7f1b90065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:54.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.443+0000 7f1b977fe700 20 client.14757.objecter dump_active .. 0 homeless 2024-02-13T00:39:54.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.443+0000 7f1b977fe700 10 client.14757.objecter ms_handle_connect 0x7f1b90062c90 2024-02-13T00:39:54.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.443+0000 7f1ba659c700 1 --2- 172.21.15.162:0/223188234 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1b90062c90 0x7f1b90065150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f1b9c009f20 tx=0x7f1b9c009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:54.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.447+0000 7f1b977fe700 1 -- 172.21.15.162:0/223188234 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1ba00518c0 con 0x7f1ba80fd490 2024-02-13T00:39:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.613+0000 7f1bacdcf700 1 -- 172.21.15.162:0/223188234 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1ba805fb80 con 0x7f1b90062c90 2024-02-13T00:39:54.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.623+0000 7f1b977fe700 1 -- 172.21.15.162:0/223188234 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357124 (secure 0 0 0) 0x7f1ba805fb80 con 0x7f1b90062c90 2024-02-13T00:39:54.630 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:39:54.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 10 librados: watch_flush enter 2024-02-13T00:39:54.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 10 librados: watch_flush exit 2024-02-13T00:39:54.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 20 client.14757.objecter shutdown clearing up homeless session... 2024-02-13T00:39:54.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 10 client.14757.objecter successfully canceled tick 2024-02-13T00:39:54.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 -- 172.21.15.162:0/223188234 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1b90062c90 msgr2=0x7f1b90065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:54.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 --2- 172.21.15.162:0/223188234 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1b90062c90 0x7f1b90065150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f1b9c009f20 tx=0x7f1b9c009cb0).stop 2024-02-13T00:39:54.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 -- 172.21.15.162:0/223188234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ba80fd490 msgr2=0x7f1ba81078b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:54.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 --2- 172.21.15.162:0/223188234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ba80fd490 0x7f1ba81078b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f1ba000b6d0 tx=0x7f1ba0004d10).stop 2024-02-13T00:39:54.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 -- 172.21.15.162:0/223188234 shutdown_connections 2024-02-13T00:39:54.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 --2- 172.21.15.162:0/223188234 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1b90062c90 0x7f1b90065150 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:54.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 --2- 172.21.15.162:0/223188234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ba80fd490 0x7f1ba81078b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:54.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 --2- 172.21.15.162:0/223188234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba80f73a0 0x7f1ba8106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:54.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 --2- 172.21.15.162:0/223188234 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ba80fcb30 0x7f1ba81073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:54.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 -- 172.21.15.162:0/223188234 >> 172.21.15.162:0/223188234 conn(0x7f1ba806eb90 msgr2=0x7f1ba80fab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:54.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 -- 172.21.15.162:0/223188234 shutdown_connections 2024-02-13T00:39:54.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 -- 172.21.15.162:0/223188234 wait complete. 2024-02-13T00:39:54.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:54.626+0000 7f1bacdcf700 1 librados: shutdown 2024-02-13T00:39:54.634 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:39:55.289 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-02-13T00:39:54.184951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1226,"num_read_kb":946,"num_write":620,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":719,"ondisk_log_size":719,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381944,"kb_used_data":5176,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579272,"statfs":{"total":767960285184,"available":765521174528,"internally_reserved":0,"allocated":5300224,"data_stored":2427549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002449"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368327+0000","last_change":"2024-02-13T00:39:18.311781+0000","last_active":"2024-02-13T00:39:23.368327+0000","last_peered":"2024-02-13T00:39:23.368327+0000","last_clean":"2024-02-13T00:39:23.368327+0000","last_became_active":"2024-02-13T00:37:47.552890+0000","last_became_peered":"2024-02-13T00:37:47.552890+0000","last_unstale":"2024-02-13T00:39:23.368327+0000","last_undegraded":"2024-02-13T00:39:23.368327+0000","last_fullsized":"2024-02-13T00:39:23.368327+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.311669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.311669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.367866+0000","last_change":"2024-02-13T00:39:23.273409+0000","last_active":"2024-02-13T00:39:23.367866+0000","last_peered":"2024-02-13T00:39:23.367866+0000","last_clean":"2024-02-13T00:39:23.367866+0000","last_became_active":"2024-02-13T00:37:45.245133+0000","last_became_peered":"2024-02-13T00:37:45.245133+0000","last_unstale":"2024-02-13T00:39:23.367866+0000","last_undegraded":"2024-02-13T00:39:23.367866+0000","last_fullsized":"2024-02-13T00:39:23.367866+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.273346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:32.313966+0000","last_change":"2024-02-13T00:39:32.313930+0000","last_active":"2024-02-13T00:39:32.313966+0000","last_peered":"2024-02-13T00:39:32.313966+0000","last_clean":"2024-02-13T00:39:32.313966+0000","last_became_active":"2024-02-13T00:37:41.220583+0000","last_became_peered":"2024-02-13T00:37:41.220583+0000","last_unstale":"2024-02-13T00:39:32.313966+0000","last_undegraded":"2024-02-13T00:39:32.313966+0000","last_fullsized":"2024-02-13T00:39:32.313966+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.313841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'39","reported_seq":125,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368464+0000","last_change":"2024-02-13T00:38:47.210073+0000","last_active":"2024-02-13T00:39:23.368464+0000","last_peered":"2024-02-13T00:39:23.368464+0000","last_clean":"2024-02-13T00:39:23.368464+0000","last_became_active":"2024-02-13T00:37:43.228728+0000","last_became_peered":"2024-02-13T00:37:43.228728+0000","last_unstale":"2024-02-13T00:39:23.368464+0000","last_undegraded":"2024-02-13T00:39:23.368464+0000","last_fullsized":"2024-02-13T00:39:23.368464+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'39","last_scrub_stamp":"2024-02-13T00:38:47.210000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:47.210000+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:37.210510+0000","last_change":"2024-02-13T00:39:37.210485+0000","last_active":"2024-02-13T00:39:37.210510+0000","last_peered":"2024-02-13T00:39:37.210510+0000","last_clean":"2024-02-13T00:39:37.210510+0000","last_became_active":"2024-02-13T00:37:47.259555+0000","last_became_peered":"2024-02-13T00:37:47.259555+0000","last_unstale":"2024-02-13T00:39:37.210510+0000","last_undegraded":"2024-02-13T00:39:37.210510+0000","last_fullsized":"2024-02-13T00:39:37.210510+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:37.210406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:37.210406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.332240+0000","last_change":"2024-02-13T00:39:24.332217+0000","last_active":"2024-02-13T00:39:24.332240+0000","last_peered":"2024-02-13T00:39:24.332240+0000","last_clean":"2024-02-13T00:39:24.332240+0000","last_became_active":"2024-02-13T00:37:45.249914+0000","last_became_peered":"2024-02-13T00:37:45.249914+0000","last_unstale":"2024-02-13T00:39:24.332240+0000","last_undegraded":"2024-02-13T00:39:24.332240+0000","last_fullsized":"2024-02-13T00:39:24.332240+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:24.332133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.332133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:40.280413+0000","last_change":"2024-02-13T00:39:40.280385+0000","last_active":"2024-02-13T00:39:40.280413+0000","last_peered":"2024-02-13T00:39:40.280413+0000","last_clean":"2024-02-13T00:39:40.280413+0000","last_became_active":"2024-02-13T00:37:41.216896+0000","last_became_peered":"2024-02-13T00:37:41.216896+0000","last_unstale":"2024-02-13T00:39:40.280413+0000","last_undegraded":"2024-02-13T00:39:40.280413+0000","last_fullsized":"2024-02-13T00:39:40.280413+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:39:40.280305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368028+0000","last_change":"2024-02-13T00:37:43.223162+0000","last_active":"2024-02-13T00:39:23.368028+0000","last_peered":"2024-02-13T00:39:23.368028+0000","last_clean":"2024-02-13T00:39:23.368028+0000","last_became_active":"2024-02-13T00:37:43.222849+0000","last_became_peered":"2024-02-13T00:37:43.222849+0000","last_unstale":"2024-02-13T00:39:23.368028+0000","last_undegraded":"2024-02-13T00:39:23.368028+0000","last_fullsized":"2024-02-13T00:39:23.368028+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313360+0000","last_change":"2024-02-13T00:38:53.600488+0000","last_active":"2024-02-13T00:39:23.313360+0000","last_peered":"2024-02-13T00:39:23.313360+0000","last_clean":"2024-02-13T00:39:23.313360+0000","last_became_active":"2024-02-13T00:37:47.260025+0000","last_became_peered":"2024-02-13T00:37:47.260025+0000","last_unstale":"2024-02-13T00:39:23.313360+0000","last_undegraded":"2024-02-13T00:39:23.313360+0000","last_fullsized":"2024-02-13T00:39:23.313360+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:38:53.600425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135669+0000","last_change":"2024-02-13T00:38:52.859190+0000","last_active":"2024-02-13T00:39:24.135669+0000","last_peered":"2024-02-13T00:39:24.135669+0000","last_clean":"2024-02-13T00:39:24.135669+0000","last_became_active":"2024-02-13T00:37:45.246184+0000","last_became_peered":"2024-02-13T00:37:45.246184+0000","last_unstale":"2024-02-13T00:39:24.135669+0000","last_undegraded":"2024-02-13T00:39:24.135669+0000","last_fullsized":"2024-02-13T00:39:24.135669+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.859089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.859089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135863+0000","last_change":"2024-02-13T00:39:21.165313+0000","last_active":"2024-02-13T00:39:24.135863+0000","last_peered":"2024-02-13T00:39:24.135863+0000","last_clean":"2024-02-13T00:39:24.135863+0000","last_became_active":"2024-02-13T00:37:41.221025+0000","last_became_peered":"2024-02-13T00:37:41.221025+0000","last_unstale":"2024-02-13T00:39:24.135863+0000","last_undegraded":"2024-02-13T00:39:24.135863+0000","last_fullsized":"2024-02-13T00:39:24.135863+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.165246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"85'14","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:42.156392+0000","last_change":"2024-02-13T00:39:42.156362+0000","last_active":"2024-02-13T00:39:42.156392+0000","last_peered":"2024-02-13T00:39:42.156392+0000","last_clean":"2024-02-13T00:39:42.156392+0000","last_became_active":"2024-02-13T00:37:43.230223+0000","last_became_peered":"2024-02-13T00:37:43.230223+0000","last_unstale":"2024-02-13T00:39:42.156392+0000","last_undegraded":"2024-02-13T00:39:42.156392+0000","last_fullsized":"2024-02-13T00:39:42.156392+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:39:42.156267+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:39:42.156267+0000","last_clean_scrub_stamp":"2024-02-13T00:39:42.156267+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040408+0000","last_change":"2024-02-13T00:39:22.338139+0000","last_active":"2024-02-13T00:39:24.040408+0000","last_peered":"2024-02-13T00:39:24.040408+0000","last_clean":"2024-02-13T00:39:24.040408+0000","last_became_active":"2024-02-13T00:37:47.259688+0000","last_became_peered":"2024-02-13T00:37:47.259688+0000","last_unstale":"2024-02-13T00:39:24.040408+0000","last_undegraded":"2024-02-13T00:39:24.040408+0000","last_fullsized":"2024-02-13T00:39:24.040408+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.338042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313713+0000","last_change":"2024-02-13T00:39:06.581693+0000","last_active":"2024-02-13T00:39:23.313713+0000","last_peered":"2024-02-13T00:39:23.313713+0000","last_clean":"2024-02-13T00:39:23.313713+0000","last_became_active":"2024-02-13T00:37:45.243961+0000","last_became_peered":"2024-02-13T00:37:45.243961+0000","last_unstale":"2024-02-13T00:39:23.313713+0000","last_undegraded":"2024-02-13T00:39:23.313713+0000","last_fullsized":"2024-02-13T00:39:23.313713+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:06.581608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:06.581608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135987+0000","last_change":"2024-02-13T00:38:45.896411+0000","last_active":"2024-02-13T00:39:24.135987+0000","last_peered":"2024-02-13T00:39:24.135987+0000","last_clean":"2024-02-13T00:39:24.135987+0000","last_became_active":"2024-02-13T00:37:41.222506+0000","last_became_peered":"2024-02-13T00:37:41.222506+0000","last_unstale":"2024-02-13T00:39:24.135987+0000","last_undegraded":"2024-02-13T00:39:24.135987+0000","last_fullsized":"2024-02-13T00:39:24.135987+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:45.896343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:45.896343+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":86,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.213185+0000","last_change":"2024-02-13T00:39:34.213153+0000","last_active":"2024-02-13T00:39:34.213185+0000","last_peered":"2024-02-13T00:39:34.213185+0000","last_clean":"2024-02-13T00:39:34.213185+0000","last_became_active":"2024-02-13T00:37:43.228966+0000","last_became_peered":"2024-02-13T00:37:43.228966+0000","last_unstale":"2024-02-13T00:39:34.213185+0000","last_undegraded":"2024-02-13T00:39:34.213185+0000","last_fullsized":"2024-02-13T00:39:34.213185+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:39:34.213074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.213074+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368823+0000","last_change":"2024-02-13T00:39:02.223894+0000","last_active":"2024-02-13T00:39:23.368823+0000","last_peered":"2024-02-13T00:39:23.368823+0000","last_clean":"2024-02-13T00:39:23.368823+0000","last_became_active":"2024-02-13T00:37:47.553853+0000","last_became_peered":"2024-02-13T00:37:47.553853+0000","last_unstale":"2024-02-13T00:39:23.368823+0000","last_undegraded":"2024-02-13T00:39:23.368823+0000","last_fullsized":"2024-02-13T00:39:23.368823+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:02.223823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.223823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368202+0000","last_change":"2024-02-13T00:39:21.301346+0000","last_active":"2024-02-13T00:39:23.368202+0000","last_peered":"2024-02-13T00:39:23.368202+0000","last_clean":"2024-02-13T00:39:23.368202+0000","last_became_active":"2024-02-13T00:37:45.242917+0000","last_became_peered":"2024-02-13T00:37:45.242917+0000","last_unstale":"2024-02-13T00:39:23.368202+0000","last_undegraded":"2024-02-13T00:39:23.368202+0000","last_fullsized":"2024-02-13T00:39:23.368202+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.301282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368528+0000","last_change":"2024-02-13T00:39:00.215830+0000","last_active":"2024-02-13T00:39:23.368528+0000","last_peered":"2024-02-13T00:39:23.368528+0000","last_clean":"2024-02-13T00:39:23.368528+0000","last_became_active":"2024-02-13T00:37:41.218333+0000","last_became_peered":"2024-02-13T00:37:41.218333+0000","last_unstale":"2024-02-13T00:39:23.368528+0000","last_undegraded":"2024-02-13T00:39:23.368528+0000","last_fullsized":"2024-02-13T00:39:23.368528+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.215755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.215755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:26.214174+0000","last_change":"2024-02-13T00:39:26.214150+0000","last_active":"2024-02-13T00:39:26.214174+0000","last_peered":"2024-02-13T00:39:26.214174+0000","last_clean":"2024-02-13T00:39:26.214174+0000","last_became_active":"2024-02-13T00:37:43.236026+0000","last_became_peered":"2024-02-13T00:37:43.236026+0000","last_unstale":"2024-02-13T00:39:26.214174+0000","last_undegraded":"2024-02-13T00:39:26.214174+0000","last_fullsized":"2024-02-13T00:39:26.214174+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:39:26.214084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:26.214084+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368515+0000","last_change":"2024-02-13T00:37:47.552844+0000","last_active":"2024-02-13T00:39:23.368515+0000","last_peered":"2024-02-13T00:39:23.368515+0000","last_clean":"2024-02-13T00:39:23.368515+0000","last_became_active":"2024-02-13T00:37:47.552547+0000","last_became_peered":"2024-02-13T00:37:47.552547+0000","last_unstale":"2024-02-13T00:39:23.368515+0000","last_undegraded":"2024-02-13T00:39:23.368515+0000","last_fullsized":"2024-02-13T00:39:23.368515+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:37:46.228444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:30.245605+0000","last_change":"2024-02-13T00:39:30.245586+0000","last_active":"2024-02-13T00:39:30.245605+0000","last_peered":"2024-02-13T00:39:30.245605+0000","last_clean":"2024-02-13T00:39:30.245605+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:39:30.245605+0000","last_undegraded":"2024-02-13T00:39:30.245605+0000","last_fullsized":"2024-02-13T00:39:30.245605+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:30.245508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.245508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138197+0000","last_change":"2024-02-13T00:39:10.364972+0000","last_active":"2024-02-13T00:39:24.138197+0000","last_peered":"2024-02-13T00:39:24.138197+0000","last_clean":"2024-02-13T00:39:24.138197+0000","last_became_active":"2024-02-13T00:37:41.218334+0000","last_became_peered":"2024-02-13T00:37:41.218334+0000","last_unstale":"2024-02-13T00:39:24.138197+0000","last_undegraded":"2024-02-13T00:39:24.138197+0000","last_fullsized":"2024-02-13T00:39:24.138197+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:10.364865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.364865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.320654+0000","last_change":"2024-02-13T00:39:08.582559+0000","last_active":"2024-02-13T00:39:23.320654+0000","last_peered":"2024-02-13T00:39:23.320654+0000","last_clean":"2024-02-13T00:39:23.320654+0000","last_became_active":"2024-02-13T00:37:43.232938+0000","last_became_peered":"2024-02-13T00:37:43.232938+0000","last_unstale":"2024-02-13T00:39:23.320654+0000","last_undegraded":"2024-02-13T00:39:23.320654+0000","last_fullsized":"2024-02-13T00:39:23.320654+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:39:08.582479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.582479+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319675+0000","last_change":"2024-02-13T00:39:00.452185+0000","last_active":"2024-02-13T00:39:23.319675+0000","last_peered":"2024-02-13T00:39:23.319675+0000","last_clean":"2024-02-13T00:39:23.319675+0000","last_became_active":"2024-02-13T00:37:47.260182+0000","last_became_peered":"2024-02-13T00:37:47.260182+0000","last_unstale":"2024-02-13T00:39:23.319675+0000","last_undegraded":"2024-02-13T00:39:23.319675+0000","last_fullsized":"2024-02-13T00:39:23.319675+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.452084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.452084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040900+0000","last_change":"2024-02-13T00:39:15.328037+0000","last_active":"2024-02-13T00:39:24.040900+0000","last_peered":"2024-02-13T00:39:24.040900+0000","last_clean":"2024-02-13T00:39:24.040900+0000","last_became_active":"2024-02-13T00:37:45.249064+0000","last_became_peered":"2024-02-13T00:37:45.249064+0000","last_unstale":"2024-02-13T00:39:24.040900+0000","last_undegraded":"2024-02-13T00:39:24.040900+0000","last_fullsized":"2024-02-13T00:39:24.040900+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.327971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.327971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:41.231368+0000","last_change":"2024-02-13T00:39:41.231336+0000","last_active":"2024-02-13T00:39:41.231368+0000","last_peered":"2024-02-13T00:39:41.231368+0000","last_clean":"2024-02-13T00:39:41.231368+0000","last_became_active":"2024-02-13T00:37:41.216574+0000","last_became_peered":"2024-02-13T00:37:41.216574+0000","last_unstale":"2024-02-13T00:39:41.231368+0000","last_undegraded":"2024-02-13T00:39:41.231368+0000","last_fullsized":"2024-02-13T00:39:41.231368+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:41.231244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:41.231244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138586+0000","last_change":"2024-02-13T00:38:58.341512+0000","last_active":"2024-02-13T00:39:24.138586+0000","last_peered":"2024-02-13T00:39:24.138586+0000","last_clean":"2024-02-13T00:39:24.138586+0000","last_became_active":"2024-02-13T00:37:43.230491+0000","last_became_peered":"2024-02-13T00:37:43.230491+0000","last_unstale":"2024-02-13T00:39:24.138586+0000","last_undegraded":"2024-02-13T00:39:24.138586+0000","last_fullsized":"2024-02-13T00:39:24.138586+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:38:58.341406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.341406+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.136647+0000","last_change":"2024-02-13T00:38:50.866411+0000","last_active":"2024-02-13T00:39:24.136647+0000","last_peered":"2024-02-13T00:39:24.136647+0000","last_clean":"2024-02-13T00:39:24.136647+0000","last_became_active":"2024-02-13T00:37:47.254586+0000","last_became_peered":"2024-02-13T00:37:47.254586+0000","last_unstale":"2024-02-13T00:39:24.136647+0000","last_undegraded":"2024-02-13T00:39:24.136647+0000","last_fullsized":"2024-02-13T00:39:24.136647+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:38:50.866323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:50.866323+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"64'8","reported_seq":76,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:44.619527+0000","last_change":"2024-02-13T00:39:02.452828+0000","last_active":"2024-02-13T00:39:44.619527+0000","last_peered":"2024-02-13T00:39:44.619527+0000","last_clean":"2024-02-13T00:39:44.619527+0000","last_became_active":"2024-02-13T00:37:45.249902+0000","last_became_peered":"2024-02-13T00:37:45.249902+0000","last_unstale":"2024-02-13T00:39:44.619527+0000","last_undegraded":"2024-02-13T00:39:44.619527+0000","last_fullsized":"2024-02-13T00:39:44.619527+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:02.452755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.452755+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313321+0000","last_change":"2024-02-13T00:38:49.606456+0000","last_active":"2024-02-13T00:39:23.313321+0000","last_peered":"2024-02-13T00:39:23.313321+0000","last_clean":"2024-02-13T00:39:23.313321+0000","last_became_active":"2024-02-13T00:37:41.214965+0000","last_became_peered":"2024-02-13T00:37:41.214965+0000","last_unstale":"2024-02-13T00:39:23.313321+0000","last_undegraded":"2024-02-13T00:39:23.313321+0000","last_fullsized":"2024-02-13T00:39:23.313321+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:49.606357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:49.606357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040877+0000","last_change":"2024-02-13T00:38:52.380430+0000","last_active":"2024-02-13T00:39:24.040877+0000","last_peered":"2024-02-13T00:39:24.040877+0000","last_clean":"2024-02-13T00:39:24.040877+0000","last_became_active":"2024-02-13T00:37:43.324221+0000","last_became_peered":"2024-02-13T00:37:43.324221+0000","last_unstale":"2024-02-13T00:39:24.040877+0000","last_undegraded":"2024-02-13T00:39:24.040877+0000","last_fullsized":"2024-02-13T00:39:24.040877+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.380369+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368642+0000","last_change":"2024-02-13T00:39:21.243606+0000","last_active":"2024-02-13T00:39:23.368642+0000","last_peered":"2024-02-13T00:39:23.368642+0000","last_clean":"2024-02-13T00:39:23.368642+0000","last_became_active":"2024-02-13T00:37:47.552917+0000","last_became_peered":"2024-02-13T00:37:47.552917+0000","last_unstale":"2024-02-13T00:39:23.368642+0000","last_undegraded":"2024-02-13T00:39:23.368642+0000","last_fullsized":"2024-02-13T00:39:23.368642+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.243527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135696+0000","last_change":"2024-02-13T00:39:13.027403+0000","last_active":"2024-02-13T00:39:24.135696+0000","last_peered":"2024-02-13T00:39:24.135696+0000","last_clean":"2024-02-13T00:39:24.135696+0000","last_became_active":"2024-02-13T00:37:45.248592+0000","last_became_peered":"2024-02-13T00:37:45.248592+0000","last_unstale":"2024-02-13T00:39:24.135696+0000","last_undegraded":"2024-02-13T00:39:24.135696+0000","last_fullsized":"2024-02-13T00:39:24.135696+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:13.027299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:13.027299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138660+0000","last_change":"2024-02-13T00:39:07.358630+0000","last_active":"2024-02-13T00:39:24.138660+0000","last_peered":"2024-02-13T00:39:24.138660+0000","last_clean":"2024-02-13T00:39:24.138660+0000","last_became_active":"2024-02-13T00:37:41.220344+0000","last_became_peered":"2024-02-13T00:37:41.220344+0000","last_unstale":"2024-02-13T00:39:24.138660+0000","last_undegraded":"2024-02-13T00:39:24.138660+0000","last_fullsized":"2024-02-13T00:39:24.138660+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.358534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.358534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.205013+0000","last_change":"2024-02-13T00:39:34.204969+0000","last_active":"2024-02-13T00:39:34.205013+0000","last_peered":"2024-02-13T00:39:34.205013+0000","last_clean":"2024-02-13T00:39:34.205013+0000","last_became_active":"2024-02-13T00:37:43.324196+0000","last_became_peered":"2024-02-13T00:37:43.324196+0000","last_unstale":"2024-02-13T00:39:34.205013+0000","last_undegraded":"2024-02-13T00:39:34.205013+0000","last_fullsized":"2024-02-13T00:39:34.205013+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:34.204888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.204888+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135581+0000","last_change":"2024-02-13T00:38:57.818305+0000","last_active":"2024-02-13T00:39:24.135581+0000","last_peered":"2024-02-13T00:39:24.135581+0000","last_clean":"2024-02-13T00:39:24.135581+0000","last_became_active":"2024-02-13T00:37:47.254667+0000","last_became_peered":"2024-02-13T00:37:47.254667+0000","last_unstale":"2024-02-13T00:39:24.135581+0000","last_undegraded":"2024-02-13T00:39:24.135581+0000","last_fullsized":"2024-02-13T00:39:24.135581+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:57.818202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.818202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138007+0000","last_change":"2024-02-13T00:38:52.493308+0000","last_active":"2024-02-13T00:39:24.138007+0000","last_peered":"2024-02-13T00:39:24.138007+0000","last_clean":"2024-02-13T00:39:24.138007+0000","last_became_active":"2024-02-13T00:37:45.249556+0000","last_became_peered":"2024-02-13T00:37:45.249556+0000","last_unstale":"2024-02-13T00:39:24.138007+0000","last_undegraded":"2024-02-13T00:39:24.138007+0000","last_fullsized":"2024-02-13T00:39:24.138007+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.493237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.493237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313839+0000","last_change":"2024-02-13T00:38:55.573704+0000","last_active":"2024-02-13T00:39:23.313839+0000","last_peered":"2024-02-13T00:39:23.313839+0000","last_clean":"2024-02-13T00:39:23.313839+0000","last_became_active":"2024-02-13T00:37:41.213033+0000","last_became_peered":"2024-02-13T00:37:41.213033+0000","last_unstale":"2024-02-13T00:39:23.313839+0000","last_undegraded":"2024-02-13T00:39:23.313839+0000","last_fullsized":"2024-02-13T00:39:23.313839+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:55.573637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:55.573637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368248+0000","last_change":"2024-02-13T00:39:23.231537+0000","last_active":"2024-02-13T00:39:23.368248+0000","last_peered":"2024-02-13T00:39:23.368248+0000","last_clean":"2024-02-13T00:39:23.368248+0000","last_became_active":"2024-02-13T00:37:43.234395+0000","last_became_peered":"2024-02-13T00:37:43.234395+0000","last_unstale":"2024-02-13T00:39:23.368248+0000","last_undegraded":"2024-02-13T00:39:23.368248+0000","last_fullsized":"2024-02-13T00:39:23.368248+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:39:23.231401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.231401+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368160+0000","last_change":"2024-02-13T00:39:01.197382+0000","last_active":"2024-02-13T00:39:23.368160+0000","last_peered":"2024-02-13T00:39:23.368160+0000","last_clean":"2024-02-13T00:39:23.368160+0000","last_became_active":"2024-02-13T00:37:47.254519+0000","last_became_peered":"2024-02-13T00:37:47.254519+0000","last_unstale":"2024-02-13T00:39:23.368160+0000","last_undegraded":"2024-02-13T00:39:23.368160+0000","last_fullsized":"2024-02-13T00:39:23.368160+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.197310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.197310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319556+0000","last_change":"2024-02-13T00:38:58.512117+0000","last_active":"2024-02-13T00:39:23.319556+0000","last_peered":"2024-02-13T00:39:23.319556+0000","last_clean":"2024-02-13T00:39:23.319556+0000","last_became_active":"2024-02-13T00:37:45.247114+0000","last_became_peered":"2024-02-13T00:37:45.247114+0000","last_unstale":"2024-02-13T00:39:23.319556+0000","last_undegraded":"2024-02-13T00:39:23.319556+0000","last_fullsized":"2024-02-13T00:39:23.319556+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.512053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:27.297205+0000","last_change":"2024-02-13T00:39:27.297182+0000","last_active":"2024-02-13T00:39:27.297205+0000","last_peered":"2024-02-13T00:39:27.297205+0000","last_clean":"2024-02-13T00:39:27.297205+0000","last_became_active":"2024-02-13T00:37:41.215531+0000","last_became_peered":"2024-02-13T00:37:41.215531+0000","last_unstale":"2024-02-13T00:39:27.297205+0000","last_undegraded":"2024-02-13T00:39:27.297205+0000","last_fullsized":"2024-02-13T00:39:27.297205+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:27.297107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:27.297107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":85,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368294+0000","last_change":"2024-02-13T00:39:06.398068+0000","last_active":"2024-02-13T00:39:23.368294+0000","last_peered":"2024-02-13T00:39:23.368294+0000","last_clean":"2024-02-13T00:39:23.368294+0000","last_became_active":"2024-02-13T00:39:05.965554+0000","last_became_peered":"2024-02-13T00:39:05.965554+0000","last_unstale":"2024-02-13T00:39:23.368294+0000","last_undegraded":"2024-02-13T00:39:23.368294+0000","last_fullsized":"2024-02-13T00:39:23.368294+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:39:03.189237+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:28.286655+0000","last_change":"2024-02-13T00:39:28.286635+0000","last_active":"2024-02-13T00:39:28.286655+0000","last_peered":"2024-02-13T00:39:28.286655+0000","last_clean":"2024-02-13T00:39:28.286655+0000","last_became_active":"2024-02-13T00:37:47.259792+0000","last_became_peered":"2024-02-13T00:37:47.259792+0000","last_unstale":"2024-02-13T00:39:28.286655+0000","last_undegraded":"2024-02-13T00:39:28.286655+0000","last_fullsized":"2024-02-13T00:39:28.286655+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.286574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.286574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:29.229804+0000","last_change":"2024-02-13T00:39:29.229760+0000","last_active":"2024-02-13T00:39:29.229804+0000","last_peered":"2024-02-13T00:39:29.229804+0000","last_clean":"2024-02-13T00:39:29.229804+0000","last_became_active":"2024-02-13T00:37:45.243594+0000","last_became_peered":"2024-02-13T00:37:45.243594+0000","last_unstale":"2024-02-13T00:39:29.229804+0000","last_undegraded":"2024-02-13T00:39:29.229804+0000","last_fullsized":"2024-02-13T00:39:29.229804+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.229676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138404+0000","last_change":"2024-02-13T00:39:00.395451+0000","last_active":"2024-02-13T00:39:24.138404+0000","last_peered":"2024-02-13T00:39:24.138404+0000","last_clean":"2024-02-13T00:39:24.138404+0000","last_became_active":"2024-02-13T00:37:41.216390+0000","last_became_peered":"2024-02-13T00:37:41.216390+0000","last_unstale":"2024-02-13T00:39:24.138404+0000","last_undegraded":"2024-02-13T00:39:24.138404+0000","last_fullsized":"2024-02-13T00:39:24.138404+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.395381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.395381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319472+0000","last_change":"2024-02-13T00:39:12.348547+0000","last_active":"2024-02-13T00:39:23.319472+0000","last_peered":"2024-02-13T00:39:23.319472+0000","last_clean":"2024-02-13T00:39:23.319472+0000","last_became_active":"2024-02-13T00:37:43.225347+0000","last_became_peered":"2024-02-13T00:37:43.225347+0000","last_unstale":"2024-02-13T00:39:23.319472+0000","last_undegraded":"2024-02-13T00:39:23.319472+0000","last_fullsized":"2024-02-13T00:39:23.319472+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:39:12.348480+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368585+0000","last_change":"2024-02-13T00:37:47.254258+0000","last_active":"2024-02-13T00:39:23.368585+0000","last_peered":"2024-02-13T00:39:23.368585+0000","last_clean":"2024-02-13T00:39:23.368585+0000","last_became_active":"2024-02-13T00:37:47.253945+0000","last_became_peered":"2024-02-13T00:37:47.253945+0000","last_unstale":"2024-02-13T00:39:23.368585+0000","last_undegraded":"2024-02-13T00:39:23.368585+0000","last_fullsized":"2024-02-13T00:39:23.368585+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:37:46.228444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"84'9","reported_seq":76,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:44.620412+0000","last_change":"2024-02-13T00:39:22.269192+0000","last_active":"2024-02-13T00:39:44.620412+0000","last_peered":"2024-02-13T00:39:44.620412+0000","last_clean":"2024-02-13T00:39:44.620412+0000","last_became_active":"2024-02-13T00:37:45.250598+0000","last_became_peered":"2024-02-13T00:37:45.250598+0000","last_unstale":"2024-02-13T00:39:44.620412+0000","last_undegraded":"2024-02-13T00:39:44.620412+0000","last_fullsized":"2024-02-13T00:39:44.620412+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.269104+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040525+0000","last_change":"2024-02-13T00:38:58.314612+0000","last_active":"2024-02-13T00:39:24.040525+0000","last_peered":"2024-02-13T00:39:24.040525+0000","last_clean":"2024-02-13T00:39:24.040525+0000","last_became_active":"2024-02-13T00:37:41.217953+0000","last_became_peered":"2024-02-13T00:37:41.217953+0000","last_unstale":"2024-02-13T00:39:24.040525+0000","last_undegraded":"2024-02-13T00:39:24.040525+0000","last_fullsized":"2024-02-13T00:39:24.040525+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.314542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.314542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313975+0000","last_change":"2024-02-13T00:39:10.549128+0000","last_active":"2024-02-13T00:39:23.313975+0000","last_peered":"2024-02-13T00:39:23.313975+0000","last_clean":"2024-02-13T00:39:23.313975+0000","last_became_active":"2024-02-13T00:37:43.322357+0000","last_became_peered":"2024-02-13T00:37:43.322357+0000","last_unstale":"2024-02-13T00:39:23.313975+0000","last_undegraded":"2024-02-13T00:39:23.313975+0000","last_fullsized":"2024-02-13T00:39:23.313975+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:39:10.549045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.549045+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040713+0000","last_change":"2024-02-13T00:39:21.344836+0000","last_active":"2024-02-13T00:39:24.040713+0000","last_peered":"2024-02-13T00:39:24.040713+0000","last_clean":"2024-02-13T00:39:24.040713+0000","last_became_active":"2024-02-13T00:37:47.252722+0000","last_became_peered":"2024-02-13T00:37:47.252722+0000","last_unstale":"2024-02-13T00:39:24.040713+0000","last_undegraded":"2024-02-13T00:39:24.040713+0000","last_fullsized":"2024-02-13T00:39:24.040713+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.344771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.344771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"64'8","reported_seq":72,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:44.265505+0000","last_change":"2024-02-13T00:39:11.518468+0000","last_active":"2024-02-13T00:39:44.265505+0000","last_peered":"2024-02-13T00:39:44.265505+0000","last_clean":"2024-02-13T00:39:44.265505+0000","last_became_active":"2024-02-13T00:37:45.246757+0000","last_became_peered":"2024-02-13T00:37:45.246757+0000","last_unstale":"2024-02-13T00:39:44.265505+0000","last_undegraded":"2024-02-13T00:39:44.265505+0000","last_fullsized":"2024-02-13T00:39:44.265505+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:11.518359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:11.518359+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313484+0000","last_change":"2024-02-13T00:39:01.561276+0000","last_active":"2024-02-13T00:39:23.313484+0000","last_peered":"2024-02-13T00:39:23.313484+0000","last_clean":"2024-02-13T00:39:23.313484+0000","last_became_active":"2024-02-13T00:37:41.219309+0000","last_became_peered":"2024-02-13T00:37:41.219309+0000","last_unstale":"2024-02-13T00:39:23.313484+0000","last_undegraded":"2024-02-13T00:39:23.313484+0000","last_fullsized":"2024-02-13T00:39:23.313484+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.561157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.561157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":92,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368503+0000","last_change":"2024-02-13T00:38:46.169426+0000","last_active":"2024-02-13T00:39:23.368503+0000","last_peered":"2024-02-13T00:39:23.368503+0000","last_clean":"2024-02-13T00:39:23.368503+0000","last_became_active":"2024-02-13T00:37:43.324313+0000","last_became_peered":"2024-02-13T00:37:43.324313+0000","last_unstale":"2024-02-13T00:39:23.368503+0000","last_undegraded":"2024-02-13T00:39:23.368503+0000","last_fullsized":"2024-02-13T00:39:23.368503+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:38:46.169344+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:25.166875+0000","last_change":"2024-02-13T00:39:25.166852+0000","last_active":"2024-02-13T00:39:25.166875+0000","last_peered":"2024-02-13T00:39:25.166875+0000","last_clean":"2024-02-13T00:39:25.166875+0000","last_became_active":"2024-02-13T00:37:47.553612+0000","last_became_peered":"2024-02-13T00:37:47.553612+0000","last_unstale":"2024-02-13T00:39:25.166875+0000","last_undegraded":"2024-02-13T00:39:25.166875+0000","last_fullsized":"2024-02-13T00:39:25.166875+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:25.166777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.166777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313787+0000","last_change":"2024-02-13T00:39:07.556868+0000","last_active":"2024-02-13T00:39:23.313787+0000","last_peered":"2024-02-13T00:39:23.313787+0000","last_clean":"2024-02-13T00:39:23.313787+0000","last_became_active":"2024-02-13T00:37:45.246907+0000","last_became_peered":"2024-02-13T00:37:45.246907+0000","last_unstale":"2024-02-13T00:39:23.313787+0000","last_undegraded":"2024-02-13T00:39:23.313787+0000","last_fullsized":"2024-02-13T00:39:23.313787+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.556772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.556772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138075+0000","last_change":"2024-02-13T00:39:01.439623+0000","last_active":"2024-02-13T00:39:24.138075+0000","last_peered":"2024-02-13T00:39:24.138075+0000","last_clean":"2024-02-13T00:39:24.138075+0000","last_became_active":"2024-02-13T00:37:41.212866+0000","last_became_peered":"2024-02-13T00:37:41.212866+0000","last_unstale":"2024-02-13T00:39:24.138075+0000","last_undegraded":"2024-02-13T00:39:24.138075+0000","last_fullsized":"2024-02-13T00:39:24.138075+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.439513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368814+0000","last_change":"2024-02-13T00:39:19.266801+0000","last_active":"2024-02-13T00:39:23.368814+0000","last_peered":"2024-02-13T00:39:23.368814+0000","last_clean":"2024-02-13T00:39:23.368814+0000","last_became_active":"2024-02-13T00:37:43.324510+0000","last_became_peered":"2024-02-13T00:37:43.324510+0000","last_unstale":"2024-02-13T00:39:23.368814+0000","last_undegraded":"2024-02-13T00:39:23.368814+0000","last_fullsized":"2024-02-13T00:39:23.368814+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:39:19.266729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:19.266729+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.320477+0000","last_change":"2024-02-13T00:39:22.615864+0000","last_active":"2024-02-13T00:39:23.320477+0000","last_peered":"2024-02-13T00:39:23.320477+0000","last_clean":"2024-02-13T00:39:23.320477+0000","last_became_active":"2024-02-13T00:37:47.254803+0000","last_became_peered":"2024-02-13T00:37:47.254803+0000","last_unstale":"2024-02-13T00:39:23.320477+0000","last_undegraded":"2024-02-13T00:39:23.320477+0000","last_fullsized":"2024-02-13T00:39:23.320477+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.615773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.615773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:46.155792+0000","last_change":"2024-02-13T00:39:46.155792+0000","last_active":"2024-02-13T00:39:46.155792+0000","last_peered":"2024-02-13T00:39:46.155792+0000","last_clean":"2024-02-13T00:39:46.155792+0000","last_became_active":"2024-02-13T00:37:45.250632+0000","last_became_peered":"2024-02-13T00:37:45.250632+0000","last_unstale":"2024-02-13T00:39:46.155792+0000","last_undegraded":"2024-02-13T00:39:46.155792+0000","last_fullsized":"2024-02-13T00:39:46.155792+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:46.155718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:46.155718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.318963+0000","last_change":"2024-02-13T00:39:01.569173+0000","last_active":"2024-02-13T00:39:23.318963+0000","last_peered":"2024-02-13T00:39:23.318963+0000","last_clean":"2024-02-13T00:39:23.318963+0000","last_became_active":"2024-02-13T00:37:41.214091+0000","last_became_peered":"2024-02-13T00:37:41.214091+0000","last_unstale":"2024-02-13T00:39:23.318963+0000","last_undegraded":"2024-02-13T00:39:23.318963+0000","last_fullsized":"2024-02-13T00:39:23.318963+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.569109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.569109+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":86,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040815+0000","last_change":"2024-02-13T00:38:56.331799+0000","last_active":"2024-02-13T00:39:24.040815+0000","last_peered":"2024-02-13T00:39:24.040815+0000","last_clean":"2024-02-13T00:39:24.040815+0000","last_became_active":"2024-02-13T00:37:43.322401+0000","last_became_peered":"2024-02-13T00:37:43.322401+0000","last_unstale":"2024-02-13T00:39:24.040815+0000","last_undegraded":"2024-02-13T00:39:24.040815+0000","last_fullsized":"2024-02-13T00:39:24.040815+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:38:56.331703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.331703+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:33.179336+0000","last_change":"2024-02-13T00:39:33.179309+0000","last_active":"2024-02-13T00:39:33.179336+0000","last_peered":"2024-02-13T00:39:33.179336+0000","last_clean":"2024-02-13T00:39:33.179336+0000","last_became_active":"2024-02-13T00:37:47.260242+0000","last_became_peered":"2024-02-13T00:37:47.260242+0000","last_unstale":"2024-02-13T00:39:33.179336+0000","last_undegraded":"2024-02-13T00:39:33.179336+0000","last_fullsized":"2024-02-13T00:39:33.179336+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.179225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.179225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":55,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368502+0000","last_change":"2024-02-13T00:38:48.379316+0000","last_active":"2024-02-13T00:39:23.368502+0000","last_peered":"2024-02-13T00:39:23.368502+0000","last_clean":"2024-02-13T00:39:23.368502+0000","last_became_active":"2024-02-13T00:37:41.219249+0000","last_became_peered":"2024-02-13T00:37:41.219249+0000","last_unstale":"2024-02-13T00:39:23.368502+0000","last_undegraded":"2024-02-13T00:39:23.368502+0000","last_fullsized":"2024-02-13T00:39:23.368502+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:38:48.379184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:48.379184+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":88,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368158+0000","last_change":"2024-02-13T00:37:43.229199+0000","last_active":"2024-02-13T00:39:23.368158+0000","last_peered":"2024-02-13T00:39:23.368158+0000","last_clean":"2024-02-13T00:39:23.368158+0000","last_became_active":"2024-02-13T00:37:43.228898+0000","last_became_peered":"2024-02-13T00:37:43.228898+0000","last_unstale":"2024-02-13T00:39:23.368158+0000","last_undegraded":"2024-02-13T00:39:23.368158+0000","last_fullsized":"2024-02-13T00:39:23.368158+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.318820+0000","last_change":"2024-02-13T00:39:21.654834+0000","last_active":"2024-02-13T00:39:23.318820+0000","last_peered":"2024-02-13T00:39:23.318820+0000","last_clean":"2024-02-13T00:39:23.318820+0000","last_became_active":"2024-02-13T00:37:45.244408+0000","last_became_peered":"2024-02-13T00:37:45.244408+0000","last_unstale":"2024-02-13T00:39:23.318820+0000","last_undegraded":"2024-02-13T00:39:23.318820+0000","last_fullsized":"2024-02-13T00:39:23.318820+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.654733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.654733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040159+0000","last_change":"2024-02-13T00:38:54.349240+0000","last_active":"2024-02-13T00:39:24.040159+0000","last_peered":"2024-02-13T00:39:24.040159+0000","last_clean":"2024-02-13T00:39:24.040159+0000","last_became_active":"2024-02-13T00:37:47.255928+0000","last_became_peered":"2024-02-13T00:37:47.255928+0000","last_unstale":"2024-02-13T00:39:24.040159+0000","last_undegraded":"2024-02-13T00:39:24.040159+0000","last_fullsized":"2024-02-13T00:39:24.040159+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:54.349171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:54.349171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319474+0000","last_change":"2024-02-13T00:38:49.618099+0000","last_active":"2024-02-13T00:39:23.319474+0000","last_peered":"2024-02-13T00:39:23.319474+0000","last_clean":"2024-02-13T00:39:23.319474+0000","last_became_active":"2024-02-13T00:37:41.217777+0000","last_became_peered":"2024-02-13T00:37:41.217777+0000","last_unstale":"2024-02-13T00:39:23.319474+0000","last_undegraded":"2024-02-13T00:39:23.319474+0000","last_fullsized":"2024-02-13T00:39:23.319474+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:38:49.617988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":28,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.320080+0000","last_change":"2024-02-13T00:39:05.974172+0000","last_active":"2024-02-13T00:39:23.320080+0000","last_peered":"2024-02-13T00:39:23.320080+0000","last_clean":"2024-02-13T00:39:23.320080+0000","last_became_active":"2024-02-13T00:39:05.965560+0000","last_became_peered":"2024-02-13T00:39:05.965560+0000","last_unstale":"2024-02-13T00:39:23.320080+0000","last_undegraded":"2024-02-13T00:39:23.320080+0000","last_fullsized":"2024-02-13T00:39:23.320080+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:39:01.456507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.456507+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"64'8","reported_seq":72,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:44.261976+0000","last_change":"2024-02-13T00:39:30.273237+0000","last_active":"2024-02-13T00:39:44.261976+0000","last_peered":"2024-02-13T00:39:44.261976+0000","last_clean":"2024-02-13T00:39:44.261976+0000","last_became_active":"2024-02-13T00:37:45.241274+0000","last_became_peered":"2024-02-13T00:37:45.241274+0000","last_unstale":"2024-02-13T00:39:44.261976+0000","last_undegraded":"2024-02-13T00:39:44.261976+0000","last_fullsized":"2024-02-13T00:39:44.261976+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:30.273165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.273165+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:36.157813+0000","last_change":"2024-02-13T00:39:36.157777+0000","last_active":"2024-02-13T00:39:36.157813+0000","last_peered":"2024-02-13T00:39:36.157813+0000","last_clean":"2024-02-13T00:39:36.157813+0000","last_became_active":"2024-02-13T00:37:47.252270+0000","last_became_peered":"2024-02-13T00:37:47.252270+0000","last_unstale":"2024-02-13T00:39:36.157813+0000","last_undegraded":"2024-02-13T00:39:36.157813+0000","last_fullsized":"2024-02-13T00:39:36.157813+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:36.157693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:36.157693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313530+0000","last_change":"2024-02-13T00:38:58.528669+0000","last_active":"2024-02-13T00:39:23.313530+0000","last_peered":"2024-02-13T00:39:23.313530+0000","last_clean":"2024-02-13T00:39:23.313530+0000","last_became_active":"2024-02-13T00:37:41.212924+0000","last_became_peered":"2024-02-13T00:37:41.212924+0000","last_unstale":"2024-02-13T00:39:23.313530+0000","last_undegraded":"2024-02-13T00:39:23.313530+0000","last_fullsized":"2024-02-13T00:39:23.313530+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.528580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.528580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"80'45","reported_seq":141,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:44.621126+0000","last_change":"2024-02-13T00:38:57.537373+0000","last_active":"2024-02-13T00:39:44.621126+0000","last_peered":"2024-02-13T00:39:44.621126+0000","last_clean":"2024-02-13T00:39:44.621126+0000","last_became_active":"2024-02-13T00:37:43.232014+0000","last_became_peered":"2024-02-13T00:37:43.232014+0000","last_unstale":"2024-02-13T00:39:44.621126+0000","last_undegraded":"2024-02-13T00:39:44.621126+0000","last_fullsized":"2024-02-13T00:39:44.621126+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'45","last_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_deep_scrub":"80'45","last_deep_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.537309+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040804+0000","last_change":"2024-02-13T00:38:47.393545+0000","last_active":"2024-02-13T00:39:24.040804+0000","last_peered":"2024-02-13T00:39:24.040804+0000","last_clean":"2024-02-13T00:39:24.040804+0000","last_became_active":"2024-02-13T00:37:45.247012+0000","last_became_peered":"2024-02-13T00:37:45.247012+0000","last_unstale":"2024-02-13T00:39:24.040804+0000","last_undegraded":"2024-02-13T00:39:24.040804+0000","last_fullsized":"2024-02-13T00:39:24.040804+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:47.393477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:47.393477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135417+0000","last_change":"2024-02-13T00:39:08.930351+0000","last_active":"2024-02-13T00:39:24.135417+0000","last_peered":"2024-02-13T00:39:24.135417+0000","last_clean":"2024-02-13T00:39:24.135417+0000","last_became_active":"2024-02-13T00:37:47.552693+0000","last_became_peered":"2024-02-13T00:37:47.552693+0000","last_unstale":"2024-02-13T00:39:24.135417+0000","last_undegraded":"2024-02-13T00:39:24.135417+0000","last_fullsized":"2024-02-13T00:39:24.135417+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:08.930286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.930286+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":170,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:48.534290+0000","last_change":"2024-02-13T00:39:10.287406+0000","last_active":"2024-02-13T00:39:48.534290+0000","last_peered":"2024-02-13T00:39:48.534290+0000","last_clean":"2024-02-13T00:39:48.534290+0000","last_became_active":"2024-02-13T00:37:41.215137+0000","last_became_peered":"2024-02-13T00:37:41.215137+0000","last_unstale":"2024-02-13T00:39:48.534290+0000","last_undegraded":"2024-02-13T00:39:48.534290+0000","last_fullsized":"2024-02-13T00:39:48.534290+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-13T00:39:10.287299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.287299+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":117,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.318856+0000","last_change":"2024-02-13T00:39:20.703787+0000","last_active":"2024-02-13T00:39:23.318856+0000","last_peered":"2024-02-13T00:39:23.318856+0000","last_clean":"2024-02-13T00:39:23.318856+0000","last_became_active":"2024-02-13T00:37:41.214442+0000","last_became_peered":"2024-02-13T00:37:41.214442+0000","last_unstale":"2024-02-13T00:39:23.318856+0000","last_undegraded":"2024-02-13T00:39:23.318856+0000","last_fullsized":"2024-02-13T00:39:23.318856+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:20.703700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:20.703700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":83,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:25.319209+0000","last_change":"2024-02-13T00:39:25.319173+0000","last_active":"2024-02-13T00:39:25.319209+0000","last_peered":"2024-02-13T00:39:25.319209+0000","last_clean":"2024-02-13T00:39:25.319209+0000","last_became_active":"2024-02-13T00:37:43.231120+0000","last_became_peered":"2024-02-13T00:37:43.231120+0000","last_unstale":"2024-02-13T00:39:25.319209+0000","last_undegraded":"2024-02-13T00:39:25.319209+0000","last_fullsized":"2024-02-13T00:39:25.319209+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:39:25.319066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.319066+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319718+0000","last_change":"2024-02-13T00:39:09.589876+0000","last_active":"2024-02-13T00:39:23.319718+0000","last_peered":"2024-02-13T00:39:23.319718+0000","last_clean":"2024-02-13T00:39:23.319718+0000","last_became_active":"2024-02-13T00:37:45.248383+0000","last_became_peered":"2024-02-13T00:37:45.248383+0000","last_unstale":"2024-02-13T00:39:23.319718+0000","last_undegraded":"2024-02-13T00:39:23.319718+0000","last_fullsized":"2024-02-13T00:39:23.319718+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:09.589813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:09.589813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.314207+0000","last_change":"2024-02-13T00:39:05.596529+0000","last_active":"2024-02-13T00:39:23.314207+0000","last_peered":"2024-02-13T00:39:23.314207+0000","last_clean":"2024-02-13T00:39:23.314207+0000","last_became_active":"2024-02-13T00:37:47.253996+0000","last_became_peered":"2024-02-13T00:37:47.253996+0000","last_unstale":"2024-02-13T00:39:23.314207+0000","last_undegraded":"2024-02-13T00:39:23.314207+0000","last_fullsized":"2024-02-13T00:39:23.314207+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:39:05.596462+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368285+0000","last_change":"2024-02-13T00:39:04.221630+0000","last_active":"2024-02-13T00:39:23.368285+0000","last_peered":"2024-02-13T00:39:23.368285+0000","last_clean":"2024-02-13T00:39:23.368285+0000","last_became_active":"2024-02-13T00:37:41.216367+0000","last_became_peered":"2024-02-13T00:37:41.216367+0000","last_unstale":"2024-02-13T00:39:23.368285+0000","last_undegraded":"2024-02-13T00:39:23.368285+0000","last_fullsized":"2024-02-13T00:39:23.368285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:39:04.221506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313651+0000","last_change":"2024-02-13T00:38:52.618768+0000","last_active":"2024-02-13T00:39:23.313651+0000","last_peered":"2024-02-13T00:39:23.313651+0000","last_clean":"2024-02-13T00:39:23.313651+0000","last_became_active":"2024-02-13T00:37:41.216101+0000","last_became_peered":"2024-02-13T00:37:41.216101+0000","last_unstale":"2024-02-13T00:39:23.313651+0000","last_undegraded":"2024-02-13T00:39:23.313651+0000","last_fullsized":"2024-02-13T00:39:23.313651+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.618696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.618696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:41.255887+0000","last_change":"2024-02-13T00:39:41.255860+0000","last_active":"2024-02-13T00:39:41.255887+0000","last_peered":"2024-02-13T00:39:41.255887+0000","last_clean":"2024-02-13T00:39:41.255887+0000","last_became_active":"2024-02-13T00:37:35.635369+0000","last_became_peered":"2024-02-13T00:37:35.635369+0000","last_unstale":"2024-02-13T00:39:41.255887+0000","last_undegraded":"2024-02-13T00:39:41.255887+0000","last_fullsized":"2024-02-13T00:39:41.255887+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:39:41.255785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'14","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138522+0000","last_change":"2024-02-13T00:39:15.415087+0000","last_active":"2024-02-13T00:39:24.138522+0000","last_peered":"2024-02-13T00:39:24.138522+0000","last_clean":"2024-02-13T00:39:24.138522+0000","last_became_active":"2024-02-13T00:37:43.324209+0000","last_became_peered":"2024-02-13T00:37:43.324209+0000","last_unstale":"2024-02-13T00:39:24.138522+0000","last_undegraded":"2024-02-13T00:39:24.138522+0000","last_fullsized":"2024-02-13T00:39:24.138522+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'14","last_scrub_stamp":"2024-02-13T00:39:15.415004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.415004+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:28.171840+0000","last_change":"2024-02-13T00:39:28.171808+0000","last_active":"2024-02-13T00:39:28.171840+0000","last_peered":"2024-02-13T00:39:28.171840+0000","last_clean":"2024-02-13T00:39:28.171840+0000","last_became_active":"2024-02-13T00:37:45.247708+0000","last_became_peered":"2024-02-13T00:37:45.247708+0000","last_unstale":"2024-02-13T00:39:28.171840+0000","last_undegraded":"2024-02-13T00:39:28.171840+0000","last_fullsized":"2024-02-13T00:39:28.171840+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.171740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.171740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:32.255351+0000","last_change":"2024-02-13T00:39:32.255327+0000","last_active":"2024-02-13T00:39:32.255351+0000","last_peered":"2024-02-13T00:39:32.255351+0000","last_clean":"2024-02-13T00:39:32.255351+0000","last_became_active":"2024-02-13T00:37:47.262024+0000","last_became_peered":"2024-02-13T00:37:47.262024+0000","last_unstale":"2024-02-13T00:39:32.255351+0000","last_undegraded":"2024-02-13T00:39:32.255351+0000","last_fullsized":"2024-02-13T00:39:32.255351+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.255260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040433+0000","last_change":"2024-02-13T00:38:55.353229+0000","last_active":"2024-02-13T00:39:24.040433+0000","last_peered":"2024-02-13T00:39:24.040433+0000","last_clean":"2024-02-13T00:39:24.040433+0000","last_became_active":"2024-02-13T00:37:41.217577+0000","last_became_peered":"2024-02-13T00:37:41.217577+0000","last_unstale":"2024-02-13T00:39:24.040433+0000","last_undegraded":"2024-02-13T00:39:24.040433+0000","last_fullsized":"2024-02-13T00:39:24.040433+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:55.353157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:55.353157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.318596+0000","last_change":"2024-02-13T00:39:05.968028+0000","last_active":"2024-02-13T00:39:23.318596+0000","last_peered":"2024-02-13T00:39:23.318596+0000","last_clean":"2024-02-13T00:39:23.318596+0000","last_became_active":"2024-02-13T00:39:05.967772+0000","last_became_peered":"2024-02-13T00:39:05.967772+0000","last_unstale":"2024-02-13T00:39:23.318596+0000","last_undegraded":"2024-02-13T00:39:23.318596+0000","last_fullsized":"2024-02-13T00:39:23.318596+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:44.893697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:44.893697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":81,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040735+0000","last_change":"2024-02-13T00:39:08.344202+0000","last_active":"2024-02-13T00:39:24.040735+0000","last_peered":"2024-02-13T00:39:24.040735+0000","last_clean":"2024-02-13T00:39:24.040735+0000","last_became_active":"2024-02-13T00:39:05.968155+0000","last_became_peered":"2024-02-13T00:39:05.968155+0000","last_unstale":"2024-02-13T00:39:24.040735+0000","last_undegraded":"2024-02-13T00:39:24.040735+0000","last_fullsized":"2024-02-13T00:39:24.040735+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:39:08.344129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.344129+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.314146+0000","last_change":"2024-02-13T00:38:48.618900+0000","last_active":"2024-02-13T00:39:23.314146+0000","last_peered":"2024-02-13T00:39:23.314146+0000","last_clean":"2024-02-13T00:39:23.314146+0000","last_became_active":"2024-02-13T00:37:45.245956+0000","last_became_peered":"2024-02-13T00:37:45.245956+0000","last_unstale":"2024-02-13T00:39:23.314146+0000","last_undegraded":"2024-02-13T00:39:23.314146+0000","last_fullsized":"2024-02-13T00:39:23.314146+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:48.618830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:48.618830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:33.304159+0000","last_change":"2024-02-13T00:39:33.304107+0000","last_active":"2024-02-13T00:39:33.304159+0000","last_peered":"2024-02-13T00:39:33.304159+0000","last_clean":"2024-02-13T00:39:33.304159+0000","last_became_active":"2024-02-13T00:37:47.552672+0000","last_became_peered":"2024-02-13T00:37:47.552672+0000","last_unstale":"2024-02-13T00:39:33.304159+0000","last_undegraded":"2024-02-13T00:39:33.304159+0000","last_fullsized":"2024-02-13T00:39:33.304159+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.303982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.303982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.136021+0000","last_change":"2024-02-13T00:38:46.927138+0000","last_active":"2024-02-13T00:39:24.136021+0000","last_peered":"2024-02-13T00:39:24.136021+0000","last_clean":"2024-02-13T00:39:24.136021+0000","last_became_active":"2024-02-13T00:37:41.221636+0000","last_became_peered":"2024-02-13T00:37:41.221636+0000","last_unstale":"2024-02-13T00:39:24.136021+0000","last_undegraded":"2024-02-13T00:39:24.136021+0000","last_fullsized":"2024-02-13T00:39:24.136021+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:46.927068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:46.927068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":100,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040454+0000","last_change":"2024-02-13T00:39:04.318990+0000","last_active":"2024-02-13T00:39:24.040454+0000","last_peered":"2024-02-13T00:39:24.040454+0000","last_clean":"2024-02-13T00:39:24.040454+0000","last_became_active":"2024-02-13T00:37:43.322450+0000","last_became_peered":"2024-02-13T00:37:43.322450+0000","last_unstale":"2024-02-13T00:39:24.040454+0000","last_undegraded":"2024-02-13T00:39:24.040454+0000","last_fullsized":"2024-02-13T00:39:24.040454+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_clean_scrub_stamp":"2024-02-13T00:39:04.318923+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138172+0000","last_change":"2024-02-13T00:39:20.451151+0000","last_active":"2024-02-13T00:39:24.138172+0000","last_peered":"2024-02-13T00:39:24.138172+0000","last_clean":"2024-02-13T00:39:24.138172+0000","last_became_active":"2024-02-13T00:37:45.245974+0000","last_became_peered":"2024-02-13T00:37:45.245974+0000","last_unstale":"2024-02-13T00:39:24.138172+0000","last_undegraded":"2024-02-13T00:39:24.138172+0000","last_fullsized":"2024-02-13T00:39:24.138172+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:20.451088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:20.451088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368220+0000","last_change":"2024-02-13T00:39:09.237388+0000","last_active":"2024-02-13T00:39:23.368220+0000","last_peered":"2024-02-13T00:39:23.368220+0000","last_clean":"2024-02-13T00:39:23.368220+0000","last_became_active":"2024-02-13T00:37:47.250291+0000","last_became_peered":"2024-02-13T00:37:47.250291+0000","last_unstale":"2024-02-13T00:39:23.368220+0000","last_undegraded":"2024-02-13T00:39:23.368220+0000","last_fullsized":"2024-02-13T00:39:23.368220+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:09.237312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:09.237312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319820+0000","last_change":"2024-02-13T00:39:21.329248+0000","last_active":"2024-02-13T00:39:23.319820+0000","last_peered":"2024-02-13T00:39:23.319820+0000","last_clean":"2024-02-13T00:39:23.319820+0000","last_became_active":"2024-02-13T00:37:41.214202+0000","last_became_peered":"2024-02-13T00:37:41.214202+0000","last_unstale":"2024-02-13T00:39:23.319820+0000","last_undegraded":"2024-02-13T00:39:23.319820+0000","last_fullsized":"2024-02-13T00:39:23.319820+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.329173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.329173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":102,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368161+0000","last_change":"2024-02-13T00:38:58.238608+0000","last_active":"2024-02-13T00:39:23.368161+0000","last_peered":"2024-02-13T00:39:23.368161+0000","last_clean":"2024-02-13T00:39:23.368161+0000","last_became_active":"2024-02-13T00:37:43.324275+0000","last_became_peered":"2024-02-13T00:37:43.324275+0000","last_unstale":"2024-02-13T00:39:23.368161+0000","last_undegraded":"2024-02-13T00:39:23.368161+0000","last_fullsized":"2024-02-13T00:39:23.368161+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:38:58.238526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.238526+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.367995+0000","last_change":"2024-02-13T00:38:52.388152+0000","last_active":"2024-02-13T00:39:23.367995+0000","last_peered":"2024-02-13T00:39:23.367995+0000","last_clean":"2024-02-13T00:39:23.367995+0000","last_became_active":"2024-02-13T00:37:45.244203+0000","last_became_peered":"2024-02-13T00:37:45.244203+0000","last_unstale":"2024-02-13T00:39:23.367995+0000","last_undegraded":"2024-02-13T00:39:23.367995+0000","last_fullsized":"2024-02-13T00:39:23.367995+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.388093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.388093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135371+0000","last_change":"2024-02-13T00:38:56.857630+0000","last_active":"2024-02-13T00:39:24.135371+0000","last_peered":"2024-02-13T00:39:24.135371+0000","last_clean":"2024-02-13T00:39:24.135371+0000","last_became_active":"2024-02-13T00:37:47.553554+0000","last_became_peered":"2024-02-13T00:37:47.553554+0000","last_unstale":"2024-02-13T00:39:24.135371+0000","last_undegraded":"2024-02-13T00:39:24.135371+0000","last_fullsized":"2024-02-13T00:39:24.135371+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:56.857534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.857534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138676+0000","last_change":"2024-02-13T00:39:14.367203+0000","last_active":"2024-02-13T00:39:24.138676+0000","last_peered":"2024-02-13T00:39:24.138676+0000","last_clean":"2024-02-13T00:39:24.138676+0000","last_became_active":"2024-02-13T00:37:41.218313+0000","last_became_peered":"2024-02-13T00:37:41.218313+0000","last_unstale":"2024-02-13T00:39:24.138676+0000","last_undegraded":"2024-02-13T00:39:24.138676+0000","last_fullsized":"2024-02-13T00:39:24.138676+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:14.367115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:14.367115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368081+0000","last_change":"2024-02-13T00:37:43.222098+0000","last_active":"2024-02-13T00:39:23.368081+0000","last_peered":"2024-02-13T00:39:23.368081+0000","last_clean":"2024-02-13T00:39:23.368081+0000","last_became_active":"2024-02-13T00:37:43.221753+0000","last_became_peered":"2024-02-13T00:37:43.221753+0000","last_unstale":"2024-02-13T00:39:23.368081+0000","last_undegraded":"2024-02-13T00:39:23.368081+0000","last_fullsized":"2024-02-13T00:39:23.368081+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"64'8","reported_seq":74,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:44.619138+0000","last_change":"2024-02-13T00:39:16.304400+0000","last_active":"2024-02-13T00:39:44.619138+0000","last_peered":"2024-02-13T00:39:44.619138+0000","last_clean":"2024-02-13T00:39:44.619138+0000","last_became_active":"2024-02-13T00:37:45.245528+0000","last_became_peered":"2024-02-13T00:37:45.245528+0000","last_unstale":"2024-02-13T00:39:44.619138+0000","last_undegraded":"2024-02-13T00:39:44.619138+0000","last_fullsized":"2024-02-13T00:39:44.619138+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:16.304316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.304316+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368370+0000","last_change":"2024-02-13T00:38:59.188989+0000","last_active":"2024-02-13T00:39:23.368370+0000","last_peered":"2024-02-13T00:39:23.368370+0000","last_clean":"2024-02-13T00:39:23.368370+0000","last_became_active":"2024-02-13T00:37:47.552591+0000","last_became_peered":"2024-02-13T00:37:47.552591+0000","last_unstale":"2024-02-13T00:39:23.368370+0000","last_undegraded":"2024-02-13T00:39:23.368370+0000","last_fullsized":"2024-02-13T00:39:23.368370+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:59.188913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:59.188913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.314084+0000","last_change":"2024-02-13T00:38:57.494754+0000","last_active":"2024-02-13T00:39:23.314084+0000","last_peered":"2024-02-13T00:39:23.314084+0000","last_clean":"2024-02-13T00:39:23.314084+0000","last_became_active":"2024-02-13T00:37:45.245897+0000","last_became_peered":"2024-02-13T00:37:45.245897+0000","last_unstale":"2024-02-13T00:39:23.314084+0000","last_undegraded":"2024-02-13T00:39:23.314084+0000","last_fullsized":"2024-02-13T00:39:23.314084+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.494653+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:29.159204+0000","last_change":"2024-02-13T00:39:29.159172+0000","last_active":"2024-02-13T00:39:29.159204+0000","last_peered":"2024-02-13T00:39:29.159204+0000","last_clean":"2024-02-13T00:39:29.159204+0000","last_became_active":"2024-02-13T00:37:41.221627+0000","last_became_peered":"2024-02-13T00:37:41.221627+0000","last_unstale":"2024-02-13T00:39:29.159204+0000","last_undegraded":"2024-02-13T00:39:29.159204+0000","last_fullsized":"2024-02-13T00:39:29.159204+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.159080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.159080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:29.275226+0000","last_change":"2024-02-13T00:39:29.275198+0000","last_active":"2024-02-13T00:39:29.275226+0000","last_peered":"2024-02-13T00:39:29.275226+0000","last_clean":"2024-02-13T00:39:29.275226+0000","last_became_active":"2024-02-13T00:37:43.224302+0000","last_became_peered":"2024-02-13T00:37:43.224302+0000","last_unstale":"2024-02-13T00:39:29.275226+0000","last_undegraded":"2024-02-13T00:39:29.275226+0000","last_fullsized":"2024-02-13T00:39:29.275226+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:29.275134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.275134+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313776+0000","last_change":"2024-02-13T00:38:54.599127+0000","last_active":"2024-02-13T00:39:23.313776+0000","last_peered":"2024-02-13T00:39:23.313776+0000","last_clean":"2024-02-13T00:39:23.313776+0000","last_became_active":"2024-02-13T00:37:47.260477+0000","last_became_peered":"2024-02-13T00:37:47.260477+0000","last_unstale":"2024-02-13T00:39:23.313776+0000","last_undegraded":"2024-02-13T00:39:23.313776+0000","last_fullsized":"2024-02-13T00:39:23.313776+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:54.599057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:54.599057+0000","last_clean_scrub_stamp":"2024-02-13T00:38:54.599057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"64'8","reported_seq":67,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:44.264511+0000","last_change":"2024-02-13T00:37:45.245491+0000","last_active":"2024-02-13T00:39:44.264511+0000","last_peered":"2024-02-13T00:39:44.264511+0000","last_clean":"2024-02-13T00:39:44.264511+0000","last_became_active":"2024-02-13T00:37:45.245183+0000","last_became_peered":"2024-02-13T00:37:45.245183+0000","last_unstale":"2024-02-13T00:39:44.264511+0000","last_undegraded":"2024-02-13T00:39:44.264511+0000","last_fullsized":"2024-02-13T00:39:44.264511+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:37:44.215312+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319112+0000","last_change":"2024-02-13T00:38:42.529745+0000","last_active":"2024-02-13T00:39:23.319112+0000","last_peered":"2024-02-13T00:39:23.319112+0000","last_clean":"2024-02-13T00:39:23.319112+0000","last_became_active":"2024-02-13T00:37:41.217979+0000","last_became_peered":"2024-02-13T00:37:41.217979+0000","last_unstale":"2024-02-13T00:39:23.319112+0000","last_undegraded":"2024-02-13T00:39:23.319112+0000","last_fullsized":"2024-02-13T00:39:23.319112+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:38:42.529666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:42.529666+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.370534+0000","last_change":"2024-02-13T00:39:23.370512+0000","last_active":"2024-02-13T00:39:23.370534+0000","last_peered":"2024-02-13T00:39:23.370534+0000","last_clean":"2024-02-13T00:39:23.370534+0000","last_became_active":"2024-02-13T00:37:43.232161+0000","last_became_peered":"2024-02-13T00:37:43.232161+0000","last_unstale":"2024-02-13T00:39:23.370534+0000","last_undegraded":"2024-02-13T00:39:23.370534+0000","last_fullsized":"2024-02-13T00:39:23.370534+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:39:23.370447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.370447+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368383+0000","last_change":"2024-02-13T00:37:47.259820+0000","last_active":"2024-02-13T00:39:23.368383+0000","last_peered":"2024-02-13T00:39:23.368383+0000","last_clean":"2024-02-13T00:39:23.368383+0000","last_became_active":"2024-02-13T00:37:47.259517+0000","last_became_peered":"2024-02-13T00:37:47.259517+0000","last_unstale":"2024-02-13T00:39:23.368383+0000","last_undegraded":"2024-02-13T00:39:23.368383+0000","last_fullsized":"2024-02-13T00:39:23.368383+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:37:46.228444+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"64'8","reported_seq":73,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:44.621284+0000","last_change":"2024-02-13T00:39:14.609925+0000","last_active":"2024-02-13T00:39:44.621284+0000","last_peered":"2024-02-13T00:39:44.621284+0000","last_clean":"2024-02-13T00:39:44.621284+0000","last_became_active":"2024-02-13T00:37:45.249684+0000","last_became_peered":"2024-02-13T00:37:45.249684+0000","last_unstale":"2024-02-13T00:39:44.621284+0000","last_undegraded":"2024-02-13T00:39:44.621284+0000","last_fullsized":"2024-02-13T00:39:44.621284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:14.609850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:14.609850+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319382+0000","last_change":"2024-02-13T00:39:17.283996+0000","last_active":"2024-02-13T00:39:23.319382+0000","last_peered":"2024-02-13T00:39:23.319382+0000","last_clean":"2024-02-13T00:39:23.319382+0000","last_became_active":"2024-02-13T00:37:41.218346+0000","last_became_peered":"2024-02-13T00:37:41.218346+0000","last_unstale":"2024-02-13T00:39:23.319382+0000","last_undegraded":"2024-02-13T00:39:23.319382+0000","last_fullsized":"2024-02-13T00:39:23.319382+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.283902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.283902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:28.305261+0000","last_change":"2024-02-13T00:39:28.305239+0000","last_active":"2024-02-13T00:39:28.305261+0000","last_peered":"2024-02-13T00:39:28.305261+0000","last_clean":"2024-02-13T00:39:28.305261+0000","last_became_active":"2024-02-13T00:37:43.227980+0000","last_became_peered":"2024-02-13T00:37:43.227980+0000","last_unstale":"2024-02-13T00:39:28.305261+0000","last_undegraded":"2024-02-13T00:39:28.305261+0000","last_fullsized":"2024-02-13T00:39:28.305261+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.305179+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319487+0000","last_change":"2024-02-13T00:39:07.563502+0000","last_active":"2024-02-13T00:39:23.319487+0000","last_peered":"2024-02-13T00:39:23.319487+0000","last_clean":"2024-02-13T00:39:23.319487+0000","last_became_active":"2024-02-13T00:37:47.256807+0000","last_became_peered":"2024-02-13T00:37:47.256807+0000","last_unstale":"2024-02-13T00:39:23.319487+0000","last_undegraded":"2024-02-13T00:39:23.319487+0000","last_fullsized":"2024-02-13T00:39:23.319487+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.563440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.563440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:35.269627+0000","last_change":"2024-02-13T00:39:35.269607+0000","last_active":"2024-02-13T00:39:35.269627+0000","last_peered":"2024-02-13T00:39:35.269627+0000","last_clean":"2024-02-13T00:39:35.269627+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:39:35.269627+0000","last_undegraded":"2024-02-13T00:39:35.269627+0000","last_fullsized":"2024-02-13T00:39:35.269627+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:35.269538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:35.269538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138738+0000","last_change":"2024-02-13T00:39:24.138710+0000","last_active":"2024-02-13T00:39:24.138738+0000","last_peered":"2024-02-13T00:39:24.138738+0000","last_clean":"2024-02-13T00:39:24.138738+0000","last_became_active":"2024-02-13T00:37:41.220874+0000","last_became_peered":"2024-02-13T00:37:41.220874+0000","last_unstale":"2024-02-13T00:39:24.138738+0000","last_undegraded":"2024-02-13T00:39:24.138738+0000","last_fullsized":"2024-02-13T00:39:24.138738+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:39:24.138629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.138629+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368495+0000","last_change":"2024-02-13T00:39:02.188162+0000","last_active":"2024-02-13T00:39:23.368495+0000","last_peered":"2024-02-13T00:39:23.368495+0000","last_clean":"2024-02-13T00:39:23.368495+0000","last_became_active":"2024-02-13T00:37:43.223574+0000","last_became_peered":"2024-02-13T00:37:43.223574+0000","last_unstale":"2024-02-13T00:39:23.368495+0000","last_undegraded":"2024-02-13T00:39:23.368495+0000","last_fullsized":"2024-02-13T00:39:23.368495+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:39:02.188096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.188096+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135466+0000","last_change":"2024-02-13T00:39:16.086278+0000","last_active":"2024-02-13T00:39:24.135466+0000","last_peered":"2024-02-13T00:39:24.135466+0000","last_clean":"2024-02-13T00:39:24.135466+0000","last_became_active":"2024-02-13T00:37:47.254705+0000","last_became_peered":"2024-02-13T00:37:47.254705+0000","last_unstale":"2024-02-13T00:39:24.135466+0000","last_undegraded":"2024-02-13T00:39:24.135466+0000","last_fullsized":"2024-02-13T00:39:24.135466+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.086207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.086207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319190+0000","last_change":"2024-02-13T00:39:17.631975+0000","last_active":"2024-02-13T00:39:23.319190+0000","last_peered":"2024-02-13T00:39:23.319190+0000","last_clean":"2024-02-13T00:39:23.319190+0000","last_became_active":"2024-02-13T00:37:45.239867+0000","last_became_peered":"2024-02-13T00:37:45.239867+0000","last_unstale":"2024-02-13T00:39:23.319190+0000","last_undegraded":"2024-02-13T00:39:23.319190+0000","last_fullsized":"2024-02-13T00:39:23.319190+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.631877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.320609+0000","last_change":"2024-02-13T00:39:18.672595+0000","last_active":"2024-02-13T00:39:23.320609+0000","last_peered":"2024-02-13T00:39:23.320609+0000","last_clean":"2024-02-13T00:39:23.320609+0000","last_became_active":"2024-02-13T00:37:41.219457+0000","last_became_peered":"2024-02-13T00:37:41.219457+0000","last_unstale":"2024-02-13T00:39:23.320609+0000","last_undegraded":"2024-02-13T00:39:23.320609+0000","last_fullsized":"2024-02-13T00:39:23.320609+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.672516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.672516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'25","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138107+0000","last_change":"2024-02-13T00:38:56.371071+0000","last_active":"2024-02-13T00:39:24.138107+0000","last_peered":"2024-02-13T00:39:24.138107+0000","last_clean":"2024-02-13T00:39:24.138107+0000","last_became_active":"2024-02-13T00:37:43.322957+0000","last_became_peered":"2024-02-13T00:37:43.322957+0000","last_unstale":"2024-02-13T00:39:24.138107+0000","last_undegraded":"2024-02-13T00:39:24.138107+0000","last_fullsized":"2024-02-13T00:39:24.138107+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'25","last_scrub_stamp":"2024-02-13T00:38:56.370998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.370998+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040202+0000","last_change":"2024-02-13T00:39:18.359681+0000","last_active":"2024-02-13T00:39:24.040202+0000","last_peered":"2024-02-13T00:39:24.040202+0000","last_clean":"2024-02-13T00:39:24.040202+0000","last_became_active":"2024-02-13T00:37:47.249556+0000","last_became_peered":"2024-02-13T00:37:47.249556+0000","last_unstale":"2024-02-13T00:39:24.040202+0000","last_undegraded":"2024-02-13T00:39:24.040202+0000","last_fullsized":"2024-02-13T00:39:24.040202+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.359604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.320337+0000","last_change":"2024-02-13T00:39:15.603878+0000","last_active":"2024-02-13T00:39:23.320337+0000","last_peered":"2024-02-13T00:39:23.320337+0000","last_clean":"2024-02-13T00:39:23.320337+0000","last_became_active":"2024-02-13T00:37:45.240923+0000","last_became_peered":"2024-02-13T00:37:45.240923+0000","last_unstale":"2024-02-13T00:39:23.320337+0000","last_undegraded":"2024-02-13T00:39:23.320337+0000","last_fullsized":"2024-02-13T00:39:23.320337+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.603808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.603808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319903+0000","last_change":"2024-02-13T00:39:16.262449+0000","last_active":"2024-02-13T00:39:23.319903+0000","last_peered":"2024-02-13T00:39:23.319903+0000","last_clean":"2024-02-13T00:39:23.319903+0000","last_became_active":"2024-02-13T00:37:41.212429+0000","last_became_peered":"2024-02-13T00:37:41.212429+0000","last_unstale":"2024-02-13T00:39:23.319903+0000","last_undegraded":"2024-02-13T00:39:23.319903+0000","last_fullsized":"2024-02-13T00:39:23.319903+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.262380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.262380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.376199+0000","last_change":"2024-02-13T00:39:24.376175+0000","last_active":"2024-02-13T00:39:24.376199+0000","last_peered":"2024-02-13T00:39:24.376199+0000","last_clean":"2024-02-13T00:39:24.376199+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:39:24.376199+0000","last_undegraded":"2024-02-13T00:39:24.376199+0000","last_fullsized":"2024-02-13T00:39:24.376199+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:39:24.376100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.376100+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313416+0000","last_change":"2024-02-13T00:39:17.713034+0000","last_active":"2024-02-13T00:39:23.313416+0000","last_peered":"2024-02-13T00:39:23.313416+0000","last_clean":"2024-02-13T00:39:23.313416+0000","last_became_active":"2024-02-13T00:37:47.552384+0000","last_became_peered":"2024-02-13T00:37:47.552384+0000","last_unstale":"2024-02-13T00:39:23.313416+0000","last_undegraded":"2024-02-13T00:39:23.313416+0000","last_fullsized":"2024-02-13T00:39:23.313416+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.712965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.712965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319980+0000","last_change":"2024-02-13T00:39:02.545556+0000","last_active":"2024-02-13T00:39:23.319980+0000","last_peered":"2024-02-13T00:39:23.319980+0000","last_clean":"2024-02-13T00:39:23.319980+0000","last_became_active":"2024-02-13T00:37:41.219093+0000","last_became_peered":"2024-02-13T00:37:41.219093+0000","last_unstale":"2024-02-13T00:39:23.319980+0000","last_undegraded":"2024-02-13T00:39:23.319980+0000","last_fullsized":"2024-02-13T00:39:23.319980+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:02.545495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.545495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":104,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313402+0000","last_change":"2024-02-13T00:38:59.548948+0000","last_active":"2024-02-13T00:39:23.313402+0000","last_peered":"2024-02-13T00:39:23.313402+0000","last_clean":"2024-02-13T00:39:23.313402+0000","last_became_active":"2024-02-13T00:37:43.322374+0000","last_became_peered":"2024-02-13T00:37:43.322374+0000","last_unstale":"2024-02-13T00:39:23.313402+0000","last_undegraded":"2024-02-13T00:39:23.313402+0000","last_fullsized":"2024-02-13T00:39:23.313402+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_clean_scrub_stamp":"2024-02-13T00:38:59.548872+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"64'8","reported_seq":72,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:44.619437+0000","last_change":"2024-02-13T00:39:15.080406+0000","last_active":"2024-02-13T00:39:44.619437+0000","last_peered":"2024-02-13T00:39:44.619437+0000","last_clean":"2024-02-13T00:39:44.619437+0000","last_became_active":"2024-02-13T00:37:45.247905+0000","last_became_peered":"2024-02-13T00:37:45.247905+0000","last_unstale":"2024-02-13T00:39:44.619437+0000","last_undegraded":"2024-02-13T00:39:44.619437+0000","last_fullsized":"2024-02-13T00:39:44.619437+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:15.080315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.080315+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":643,"ondisk_log_size":643,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":117,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266716,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297744,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447408,"statfs":{"total":95995035648,"available":95690145792,"internally_reserved":0,"allocated":663552,"data_stored":304098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.498,"15min":0.501},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.63800000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52300000000000002,"15min":0.52800000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.63,"5min":1.028,"15min":1.028},"last":0.63500000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.49099999999999999,"15min":0.49399999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.622},{"interface":"front","average":{"1min":0.503,"5min":0.59899999999999998,"15min":0.61499999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64200000000000002,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.56399999999999995}]},{"osd":2,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.628,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61099999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.55200000000000005}]},{"osd":3,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.55800000000000005,"15min":0.56799999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.64100000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.65700000000000003,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.58599999999999997}]},{"osd":4,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.46100000000000002,"15min":0.46999999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.65400000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.45600000000000002,"15min":0.46400000000000002},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67000000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.437}]},{"osd":5,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.63600000000000001,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46800000000000003,"15min":0.47199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65800000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.53100000000000003}]},{"osd":6,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.49199999999999999,"15min":0.497},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68899999999999995,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.499}]}]},{"osd":6,"up_from":45,"seq":193273528354,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":302575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.496},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.46899999999999997,"15min":0.46800000000000003},"min":{"1min":0.35899999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.61399999999999999}]},{"osd":1,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.504,"15min":0.50600000000000001},"min":{"1min":0.41499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61099999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.67800000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.51000000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.497,"15min":0.5},"min":{"1min":0.34899999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.64100000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.626,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.59199999999999997}]},{"osd":3,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.53200000000000003,"15min":0.54300000000000004},"min":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68999999999999995,"5min":1.181,"15min":1.181},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.42499999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.57799999999999996}]},{"osd":4,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.39500000000000002},"min":{"1min":0.29999999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72199999999999998,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41699999999999998},"min":{"1min":0.31,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66700000000000004,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.44600000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.312,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61199999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.44700000000000001,"15min":0.442},"min":{"1min":0.34000000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.76300000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.38200000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.499,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67700000000000005,"5min":1.173,"15min":1.173},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79000000000000004,"5min":1.262,"15min":1.262},"last":0.42299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297772,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447380,"statfs":{"total":95995035648,"available":95690117120,"internally_reserved":0,"allocated":692224,"data_stored":303825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.32000000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.313},"min":{"1min":0.221,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.49099999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.25900000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.372},"min":{"1min":0.255,"5min":0.215,"15min":0.215},"max":{"1min":0.57999999999999996,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.313},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.27300000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.69499999999999995,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.27300000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65000000000000002,"5min":0.748,"15min":0.748},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.45000000000000001,"15min":0.46500000000000002},"min":{"1min":0.29299999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.53900000000000003,"5min":0.745,"15min":0.745},"last":0.33100000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.81299999999999994,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82499999999999996,"5min":1.02,"15min":1.02},"last":0.60699999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56799999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78700000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70699999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.54800000000000004}]},{"osd":6,"last update":"Tue Feb 13 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78700000000000003,"5min":1.054,"15min":1.054},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86799999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48899999999999999}]},{"osd":7,"last update":"Tue Feb 13 00:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.78500000000000003,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.62,"15min":0.623},"min":{"1min":0.41599999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81899999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56200000000000006}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297740,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447412,"statfs":{"total":95995035648,"available":95690149888,"internally_reserved":0,"allocated":659456,"data_stored":302879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:39:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34599999999999997,"15min":0.32800000000000001},"min":{"1min":0.254,"5min":0.192,"15min":0.192},"max":{"1min":0.52000000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.376,"15min":0.34399999999999997},"min":{"1min":0.23499999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.58899999999999997,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.313}]},{"osd":2,"last update":"Tue Feb 13 00:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40400000000000003,"15min":0.39800000000000002},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.65200000000000002,"5min":1.206,"15min":1.206},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.746,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.30199999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.64900000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45600000000000002,"15min":0.46999999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.37}]},{"osd":4,"last update":"Tue Feb 13 00:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.75600000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72099999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.71799999999999997}]},{"osd":5,"last update":"Tue Feb 13 00:39:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55800000000000005,"15min":0.57399999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.49199999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.41599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54200000000000004,"15min":0.53700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.44900000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:39:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.57299999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411382,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297732,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447420,"statfs":{"total":95995035648,"available":95690158080,"internally_reserved":0,"allocated":651264,"data_stored":303146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.36399999999999999,"15min":0.35699999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.49199999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36299999999999999,"15min":0.34599999999999997},"min":{"1min":0.22900000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.66800000000000004,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.25800000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.35699999999999998,"15min":0.34899999999999998},"min":{"1min":0.22900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.49099999999999999,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.308},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38400000000000001,"15min":0.374},"min":{"1min":0.251,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.51600000000000001,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.248}]},{"osd":3,"last update":"Tue Feb 13 00:39:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.28000000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.52800000000000002,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59099999999999997,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.28000000000000003}]},{"osd":4,"last update":"Tue Feb 13 00:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73699999999999999,"5min":0.873,"15min":0.873},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.58799999999999997,"15min":0.59199999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71699999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.41499999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:38:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.376}]},{"osd":6,"last update":"Tue Feb 13 00:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58099999999999996,"15min":0.58399999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79400000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.38700000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58299999999999996,"15min":0.59099999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.748,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70999999999999996,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.48799999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182448,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297756,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447396,"statfs":{"total":95995035648,"available":95690133504,"internally_reserved":0,"allocated":675840,"data_stored":303408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.36099999999999999,"15min":0.376},"min":{"1min":0.221,"5min":0.191,"15min":0.191},"max":{"1min":0.55300000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.39300000000000002,"15min":0.40799999999999997},"min":{"1min":0.27800000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46200000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.29199999999999998}]},{"osd":1,"last update":"Tue Feb 13 00:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38500000000000001,"15min":0.39800000000000002},"min":{"1min":0.28100000000000003,"5min":0.217,"15min":0.217},"max":{"1min":0.5,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.252},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.39100000000000001,"15min":0.39900000000000002},"min":{"1min":0.24399999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.56200000000000006,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.32600000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.41399999999999998,"15min":0.42099999999999999},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58899999999999997,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39200000000000002,"15min":0.40300000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.52900000000000003,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.309}]},{"osd":4,"last update":"Tue Feb 13 00:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53400000000000003,"15min":0.54000000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.64700000000000002,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.53600000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.75,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42399999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:38:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.50600000000000001,"15min":0.499},"min":{"1min":0.38200000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.52100000000000002,"15min":0.51600000000000001},"min":{"1min":0.41199999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.45600000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.50700000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:39:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.54400000000000004},"min":{"1min":0.372,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77800000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.498}]}]},{"osd":4,"up_from":31,"seq":133143986220,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":303919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67900000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.66500000000000004,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.45400000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.52100000000000002,"15min":0.53000000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.61499999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.52900000000000003}]},{"osd":2,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69899999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.374},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52000000000000002,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.62,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.39100000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.38500000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.441}]},{"osd":5,"last update":"Tue Feb 13 00:39:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.37,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.58299999999999996,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.39300000000000002,"15min":0.36799999999999999},"min":{"1min":0.315,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.55600000000000005}]},{"osd":6,"last update":"Tue Feb 13 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.32900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.41299999999999998,"15min":0.40400000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.503}]},{"osd":7,"last update":"Tue Feb 13 00:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46100000000000002,"15min":0.45900000000000002},"min":{"1min":0.34200000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.33000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.63800000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.54100000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724583,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":647168,"data_stored":303699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.48699999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.53100000000000003,"15min":0.51200000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.0369999999999999,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.48499999999999999}]},{"osd":1,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.5,"15min":0.496},"min":{"1min":0.34599999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.52400000000000002,"15min":0.502},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":4.0839999999999996,"5min":4.0839999999999996,"15min":4.0839999999999996},"last":0.52600000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.51400000000000001,"15min":0.50600000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.498}]},{"osd":3,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.51400000000000001,"15min":0.505},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.38300000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:38:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.40600000000000003,"15min":0.36199999999999999},"min":{"1min":0.246,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.1210000000000004,"5min":4.1210000000000004,"15min":4.1210000000000004},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.46700000000000003,"15min":0.40699999999999997},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.6950000000000003,"5min":5.6950000000000003,"15min":5.6950000000000003},"last":0.56499999999999995}]},{"osd":6,"last update":"Tue Feb 13 00:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.441,"15min":0.42699999999999999},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.873,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.46400000000000002,"15min":0.44700000000000001},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.327,"5min":1.377,"15min":1.377},"last":0.372}]},{"osd":7,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.755,"15min":0.80300000000000005},"min":{"1min":0.33200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.58499999999999996,"5min":7.5519999999999996,"15min":7.5519999999999996},"last":0.44},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.55800000000000005,"15min":0.57399999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.64800000000000002,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:39:55.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:39:55.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:39:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:39:55.562 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:56.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:54.185129+0000 mgr.y (mgr.14544) 91 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:56.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:54.614617+0000 mgr.y (mgr.14544) 92 : audit [DBG] from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:39:56.035 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:54.185129+0000 mgr.y (mgr.14544) 91 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:56.035 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:54.614617+0000 mgr.y (mgr.14544) 92 : audit [DBG] from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:39:56.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:54.185129+0000 mgr.y (mgr.14544) 91 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:39:56.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:54.614617+0000 mgr.y (mgr.14544) 92 : audit [DBG] from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:39:56.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:39:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:39:56] "GET /metrics HTTP/1.1" 200 172179 "" "Prometheus/2.33.4" 2024-02-13T00:39:57.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:55.262667+0000 mgr.y (mgr.14544) 93 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:56.117183+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:39:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:56.121041+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:39:57.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:55.262667+0000 mgr.y (mgr.14544) 93 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:57.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:56.117183+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:39:57.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:56.121041+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:39:57.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:55.262667+0000 mgr.y (mgr.14544) 93 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:39:57.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:56.117183+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:39:57.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:56.121041+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:39:57.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.534+0000 7f51e45f1700 1 -- 172.21.15.162:0/3729785641 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51dc0fcb30 msgr2=0x7f51dc0fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:57.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.534+0000 7f51e45f1700 1 --2- 172.21.15.162:0/3729785641 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51dc0fcb30 0x7f51dc0fcf10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f51d80047b0 tx=0x7f51d8009cb0).stop 2024-02-13T00:39:57.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 -- 172.21.15.162:0/3729785641 shutdown_connections 2024-02-13T00:39:57.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 --2- 172.21.15.162:0/3729785641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51dc0f73a0 0x7f51dc1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:57.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 --2- 172.21.15.162:0/3729785641 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51dc0fcb30 0x7f51dc0fcf10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 --2- 172.21.15.162:0/3729785641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51dc0fd490 0x7f51dc0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 -- 172.21.15.162:0/3729785641 >> 172.21.15.162:0/3729785641 conn(0x7f51dc06eb90 msgr2=0x7f51dc0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 -- 172.21.15.162:0/3729785641 shutdown_connections 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 -- 172.21.15.162:0/3729785641 wait complete. 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 librados: starting msgr at 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 librados: starting objecter 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 Processor -- start 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 -- start start 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 librados: setting wanted keys 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 librados: calling monclient init 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51dc0f73a0 0x7f51dc10e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51dc0fcb30 0x7f51dc10e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:57.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51dc0fd490 0x7f51dc1083c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc1088b0 con 0x7f51dc0fd490 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc108a20 con 0x7f51dc0f73a0 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e45f1700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc108be0 con 0x7f51dc0fcb30 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.535+0000 7f51e1b8c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51dc0fcb30 0x7f51dc10e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e1b8c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51dc0fcb30 0x7f51dc10e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:41934/0 (socket says 172.21.15.162:41934) 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e1b8c700 1 -- 172.21.15.162:0/3547930105 learned_addr learned my addr 172.21.15.162:0/3547930105 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e2b8e700 1 --2- 172.21.15.162:0/3547930105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51dc0fd490 0x7f51dc1083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e238d700 1 --2- 172.21.15.162:0/3547930105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51dc0f73a0 0x7f51dc10e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e1b8c700 1 -- 172.21.15.162:0/3547930105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51dc0f73a0 msgr2=0x7f51dc10e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e1b8c700 1 --2- 172.21.15.162:0/3547930105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51dc0f73a0 0x7f51dc10e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e1b8c700 1 -- 172.21.15.162:0/3547930105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51dc0fd490 msgr2=0x7f51dc1083c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e1b8c700 1 --2- 172.21.15.162:0/3547930105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51dc0fd490 0x7f51dc1083c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e1b8c700 1 -- 172.21.15.162:0/3547930105 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40097c0 con 0x7f51dc0fcb30 2024-02-13T00:39:57.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e2b8e700 1 --2- 172.21.15.162:0/3547930105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51dc0fd490 0x7f51dc1083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:39:57.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e1b8c700 1 --2- 172.21.15.162:0/3547930105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51dc0fcb30 0x7f51dc10e6c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f51cc00b530 tx=0x7f51cc0090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:57.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51d37fe700 10 client.?.objecter ms_handle_connect 0x7f51dc0fcb30 2024-02-13T00:39:57.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51d37fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:39:57.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51d37fe700 1 -- 172.21.15.162:0/3547930105 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc002320 con 0x7f51dc0fcb30 2024-02-13T00:39:57.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51d37fe700 1 -- 172.21.15.162:0/3547930105 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51cc002950 con 0x7f51dc0fcb30 2024-02-13T00:39:57.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e45f1700 1 -- 172.21.15.162:0/3547930105 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d8005040 con 0x7f51dc0fcb30 2024-02-13T00:39:57.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51d37fe700 1 -- 172.21.15.162:0/3547930105 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc012950 con 0x7f51dc0fcb30 2024-02-13T00:39:57.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e45f1700 10 client.24662.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:39:57.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e45f1700 1 -- 172.21.15.162:0/3547930105 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51dc109220 con 0x7f51dc0fcb30 2024-02-13T00:39:57.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.536+0000 7f51e45f1700 1 librados: init done 2024-02-13T00:39:57.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.537+0000 7f51e45f1700 1 -- 172.21.15.162:0/3547930105 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51dc0475e0 con 0x7f51dc0fcb30 2024-02-13T00:39:57.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.538+0000 7f51d37fe700 1 -- 172.21.15.162:0/3547930105 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f51cc002480 con 0x7f51dc0fcb30 2024-02-13T00:39:57.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.538+0000 7f51d37fe700 1 --2- 172.21.15.162:0/3547930105 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51c8062c90 0x7f51c8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:39:57.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.538+0000 7f51e238d700 1 --2- 172.21.15.162:0/3547930105 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51c8062c90 0x7f51c8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:39:57.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.538+0000 7f51d37fe700 1 -- 172.21.15.162:0/3547930105 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(86..86 src has 30..86) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f51cc07cb40 con 0x7f51dc0fcb30 2024-02-13T00:39:57.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.538+0000 7f51d37fe700 10 client.24662.objecter ms_dispatch 0x7f51dc001550 osd_map(86..86 src has 30..86) v4 2024-02-13T00:39:57.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.538+0000 7f51d37fe700 3 client.24662.objecter handle_osd_map got epochs [86,86] > 0 2024-02-13T00:39:57.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.538+0000 7f51d37fe700 3 client.24662.objecter handle_osd_map decoding full epoch 86 2024-02-13T00:39:57.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.538+0000 7f51d37fe700 20 client.24662.objecter dump_active .. 0 homeless 2024-02-13T00:39:57.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.538+0000 7f51d37fe700 10 client.24662.objecter ms_handle_connect 0x7f51c8062c90 2024-02-13T00:39:57.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.538+0000 7f51e238d700 1 --2- 172.21.15.162:0/3547930105 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51c8062c90 0x7f51c8065150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f51d8005010 tx=0x7f51d8007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:39:57.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.542+0000 7f51d37fe700 1 -- 172.21.15.162:0/3547930105 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f51cc0506f0 con 0x7f51dc0fcb30 2024-02-13T00:39:57.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.704+0000 7f51e45f1700 1 -- 172.21.15.162:0/3547930105 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51dc109950 con 0x7f51c8062c90 2024-02-13T00:39:57.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.714+0000 7f51d37fe700 1 -- 172.21.15.162:0/3547930105 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357062 (secure 0 0 0) 0x7f51d8019040 con 0x7f51c8062c90 2024-02-13T00:39:57.721 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:39:57.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 10 librados: watch_flush enter 2024-02-13T00:39:57.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 10 librados: watch_flush exit 2024-02-13T00:39:57.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 20 client.24662.objecter shutdown clearing up homeless session... 2024-02-13T00:39:57.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 10 client.24662.objecter successfully canceled tick 2024-02-13T00:39:57.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 -- 172.21.15.162:0/3547930105 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51c8062c90 msgr2=0x7f51c8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:57.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 --2- 172.21.15.162:0/3547930105 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51c8062c90 0x7f51c8065150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f51d8005010 tx=0x7f51d8007f90).stop 2024-02-13T00:39:57.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 -- 172.21.15.162:0/3547930105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51dc0fcb30 msgr2=0x7f51dc10e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:39:57.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 --2- 172.21.15.162:0/3547930105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51dc0fcb30 0x7f51dc10e6c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f51cc00b530 tx=0x7f51cc0090f0).stop 2024-02-13T00:39:57.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 -- 172.21.15.162:0/3547930105 shutdown_connections 2024-02-13T00:39:57.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 --2- 172.21.15.162:0/3547930105 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51c8062c90 0x7f51c8065150 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:57.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 --2- 172.21.15.162:0/3547930105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51dc0fd490 0x7f51dc1083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:57.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 --2- 172.21.15.162:0/3547930105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51dc0f73a0 0x7f51dc10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:57.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 --2- 172.21.15.162:0/3547930105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51dc0fcb30 0x7f51dc10e6c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:39:57.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 -- 172.21.15.162:0/3547930105 >> 172.21.15.162:0/3547930105 conn(0x7f51dc06eb90 msgr2=0x7f51dc060f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:39:57.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 -- 172.21.15.162:0/3547930105 shutdown_connections 2024-02-13T00:39:57.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 -- 172.21.15.162:0/3547930105 wait complete. 2024-02-13T00:39:57.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:39:57.717+0000 7f51e45f1700 1 librados: shutdown 2024-02-13T00:39:57.725 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:39:58.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:56.186169+0000 mgr.y (mgr.14544) 94 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:39:58.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:56.186169+0000 mgr.y (mgr.14544) 94 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:39:58.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:56.186169+0000 mgr.y (mgr.14544) 94 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:39:58.532 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-02-13T00:39:56.185587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1231,"num_read_kb":951,"num_write":620,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":730,"ondisk_log_size":730,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381944,"kb_used_data":5176,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579272,"statfs":{"total":767960285184,"available":765521174528,"internally_reserved":0,"allocated":5300224,"data_stored":2427549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.925436"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368327+0000","last_change":"2024-02-13T00:39:18.311781+0000","last_active":"2024-02-13T00:39:23.368327+0000","last_peered":"2024-02-13T00:39:23.368327+0000","last_clean":"2024-02-13T00:39:23.368327+0000","last_became_active":"2024-02-13T00:37:47.552890+0000","last_became_peered":"2024-02-13T00:37:47.552890+0000","last_unstale":"2024-02-13T00:39:23.368327+0000","last_undegraded":"2024-02-13T00:39:23.368327+0000","last_fullsized":"2024-02-13T00:39:23.368327+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.311669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.311669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.367866+0000","last_change":"2024-02-13T00:39:23.273409+0000","last_active":"2024-02-13T00:39:23.367866+0000","last_peered":"2024-02-13T00:39:23.367866+0000","last_clean":"2024-02-13T00:39:23.367866+0000","last_became_active":"2024-02-13T00:37:45.245133+0000","last_became_peered":"2024-02-13T00:37:45.245133+0000","last_unstale":"2024-02-13T00:39:23.367866+0000","last_undegraded":"2024-02-13T00:39:23.367866+0000","last_fullsized":"2024-02-13T00:39:23.367866+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.273346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:32.313966+0000","last_change":"2024-02-13T00:39:32.313930+0000","last_active":"2024-02-13T00:39:32.313966+0000","last_peered":"2024-02-13T00:39:32.313966+0000","last_clean":"2024-02-13T00:39:32.313966+0000","last_became_active":"2024-02-13T00:37:41.220583+0000","last_became_peered":"2024-02-13T00:37:41.220583+0000","last_unstale":"2024-02-13T00:39:32.313966+0000","last_undegraded":"2024-02-13T00:39:32.313966+0000","last_fullsized":"2024-02-13T00:39:32.313966+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.313841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"86'50","reported_seq":141,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:53.119049+0000","last_change":"2024-02-13T00:39:53.119049+0000","last_active":"2024-02-13T00:39:53.119049+0000","last_peered":"2024-02-13T00:39:53.119049+0000","last_clean":"2024-02-13T00:39:53.119049+0000","last_became_active":"2024-02-13T00:37:43.228728+0000","last_became_peered":"2024-02-13T00:37:43.228728+0000","last_unstale":"2024-02-13T00:39:53.119049+0000","last_undegraded":"2024-02-13T00:39:53.119049+0000","last_fullsized":"2024-02-13T00:39:53.119049+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:39:53.118989+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:37.210510+0000","last_change":"2024-02-13T00:39:37.210485+0000","last_active":"2024-02-13T00:39:37.210510+0000","last_peered":"2024-02-13T00:39:37.210510+0000","last_clean":"2024-02-13T00:39:37.210510+0000","last_became_active":"2024-02-13T00:37:47.259555+0000","last_became_peered":"2024-02-13T00:37:47.259555+0000","last_unstale":"2024-02-13T00:39:37.210510+0000","last_undegraded":"2024-02-13T00:39:37.210510+0000","last_fullsized":"2024-02-13T00:39:37.210510+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:37.210406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:37.210406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.332240+0000","last_change":"2024-02-13T00:39:24.332217+0000","last_active":"2024-02-13T00:39:24.332240+0000","last_peered":"2024-02-13T00:39:24.332240+0000","last_clean":"2024-02-13T00:39:24.332240+0000","last_became_active":"2024-02-13T00:37:45.249914+0000","last_became_peered":"2024-02-13T00:37:45.249914+0000","last_unstale":"2024-02-13T00:39:24.332240+0000","last_undegraded":"2024-02-13T00:39:24.332240+0000","last_fullsized":"2024-02-13T00:39:24.332240+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:24.332133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.332133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:40.280413+0000","last_change":"2024-02-13T00:39:40.280385+0000","last_active":"2024-02-13T00:39:40.280413+0000","last_peered":"2024-02-13T00:39:40.280413+0000","last_clean":"2024-02-13T00:39:40.280413+0000","last_became_active":"2024-02-13T00:37:41.216896+0000","last_became_peered":"2024-02-13T00:37:41.216896+0000","last_unstale":"2024-02-13T00:39:40.280413+0000","last_undegraded":"2024-02-13T00:39:40.280413+0000","last_fullsized":"2024-02-13T00:39:40.280413+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:39:40.280305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368028+0000","last_change":"2024-02-13T00:37:43.223162+0000","last_active":"2024-02-13T00:39:23.368028+0000","last_peered":"2024-02-13T00:39:23.368028+0000","last_clean":"2024-02-13T00:39:23.368028+0000","last_became_active":"2024-02-13T00:37:43.222849+0000","last_became_peered":"2024-02-13T00:37:43.222849+0000","last_unstale":"2024-02-13T00:39:23.368028+0000","last_undegraded":"2024-02-13T00:39:23.368028+0000","last_fullsized":"2024-02-13T00:39:23.368028+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313360+0000","last_change":"2024-02-13T00:38:53.600488+0000","last_active":"2024-02-13T00:39:23.313360+0000","last_peered":"2024-02-13T00:39:23.313360+0000","last_clean":"2024-02-13T00:39:23.313360+0000","last_became_active":"2024-02-13T00:37:47.260025+0000","last_became_peered":"2024-02-13T00:37:47.260025+0000","last_unstale":"2024-02-13T00:39:23.313360+0000","last_undegraded":"2024-02-13T00:39:23.313360+0000","last_fullsized":"2024-02-13T00:39:23.313360+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:38:53.600425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135669+0000","last_change":"2024-02-13T00:38:52.859190+0000","last_active":"2024-02-13T00:39:24.135669+0000","last_peered":"2024-02-13T00:39:24.135669+0000","last_clean":"2024-02-13T00:39:24.135669+0000","last_became_active":"2024-02-13T00:37:45.246184+0000","last_became_peered":"2024-02-13T00:37:45.246184+0000","last_unstale":"2024-02-13T00:39:24.135669+0000","last_undegraded":"2024-02-13T00:39:24.135669+0000","last_fullsized":"2024-02-13T00:39:24.135669+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.859089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.859089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135863+0000","last_change":"2024-02-13T00:39:21.165313+0000","last_active":"2024-02-13T00:39:24.135863+0000","last_peered":"2024-02-13T00:39:24.135863+0000","last_clean":"2024-02-13T00:39:24.135863+0000","last_became_active":"2024-02-13T00:37:41.221025+0000","last_became_peered":"2024-02-13T00:37:41.221025+0000","last_unstale":"2024-02-13T00:39:24.135863+0000","last_undegraded":"2024-02-13T00:39:24.135863+0000","last_fullsized":"2024-02-13T00:39:24.135863+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.165246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"85'14","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:42.156392+0000","last_change":"2024-02-13T00:39:42.156362+0000","last_active":"2024-02-13T00:39:42.156392+0000","last_peered":"2024-02-13T00:39:42.156392+0000","last_clean":"2024-02-13T00:39:42.156392+0000","last_became_active":"2024-02-13T00:37:43.230223+0000","last_became_peered":"2024-02-13T00:37:43.230223+0000","last_unstale":"2024-02-13T00:39:42.156392+0000","last_undegraded":"2024-02-13T00:39:42.156392+0000","last_fullsized":"2024-02-13T00:39:42.156392+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:39:42.156267+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:39:42.156267+0000","last_clean_scrub_stamp":"2024-02-13T00:39:42.156267+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040408+0000","last_change":"2024-02-13T00:39:22.338139+0000","last_active":"2024-02-13T00:39:24.040408+0000","last_peered":"2024-02-13T00:39:24.040408+0000","last_clean":"2024-02-13T00:39:24.040408+0000","last_became_active":"2024-02-13T00:37:47.259688+0000","last_became_peered":"2024-02-13T00:37:47.259688+0000","last_unstale":"2024-02-13T00:39:24.040408+0000","last_undegraded":"2024-02-13T00:39:24.040408+0000","last_fullsized":"2024-02-13T00:39:24.040408+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.338042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313713+0000","last_change":"2024-02-13T00:39:06.581693+0000","last_active":"2024-02-13T00:39:23.313713+0000","last_peered":"2024-02-13T00:39:23.313713+0000","last_clean":"2024-02-13T00:39:23.313713+0000","last_became_active":"2024-02-13T00:37:45.243961+0000","last_became_peered":"2024-02-13T00:37:45.243961+0000","last_unstale":"2024-02-13T00:39:23.313713+0000","last_undegraded":"2024-02-13T00:39:23.313713+0000","last_fullsized":"2024-02-13T00:39:23.313713+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:06.581608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:06.581608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135987+0000","last_change":"2024-02-13T00:38:45.896411+0000","last_active":"2024-02-13T00:39:24.135987+0000","last_peered":"2024-02-13T00:39:24.135987+0000","last_clean":"2024-02-13T00:39:24.135987+0000","last_became_active":"2024-02-13T00:37:41.222506+0000","last_became_peered":"2024-02-13T00:37:41.222506+0000","last_unstale":"2024-02-13T00:39:24.135987+0000","last_undegraded":"2024-02-13T00:39:24.135987+0000","last_fullsized":"2024-02-13T00:39:24.135987+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:45.896343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:45.896343+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":86,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.213185+0000","last_change":"2024-02-13T00:39:34.213153+0000","last_active":"2024-02-13T00:39:34.213185+0000","last_peered":"2024-02-13T00:39:34.213185+0000","last_clean":"2024-02-13T00:39:34.213185+0000","last_became_active":"2024-02-13T00:37:43.228966+0000","last_became_peered":"2024-02-13T00:37:43.228966+0000","last_unstale":"2024-02-13T00:39:34.213185+0000","last_undegraded":"2024-02-13T00:39:34.213185+0000","last_fullsized":"2024-02-13T00:39:34.213185+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:39:34.213074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.213074+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368823+0000","last_change":"2024-02-13T00:39:02.223894+0000","last_active":"2024-02-13T00:39:23.368823+0000","last_peered":"2024-02-13T00:39:23.368823+0000","last_clean":"2024-02-13T00:39:23.368823+0000","last_became_active":"2024-02-13T00:37:47.553853+0000","last_became_peered":"2024-02-13T00:37:47.553853+0000","last_unstale":"2024-02-13T00:39:23.368823+0000","last_undegraded":"2024-02-13T00:39:23.368823+0000","last_fullsized":"2024-02-13T00:39:23.368823+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:02.223823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.223823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368202+0000","last_change":"2024-02-13T00:39:21.301346+0000","last_active":"2024-02-13T00:39:23.368202+0000","last_peered":"2024-02-13T00:39:23.368202+0000","last_clean":"2024-02-13T00:39:23.368202+0000","last_became_active":"2024-02-13T00:37:45.242917+0000","last_became_peered":"2024-02-13T00:37:45.242917+0000","last_unstale":"2024-02-13T00:39:23.368202+0000","last_undegraded":"2024-02-13T00:39:23.368202+0000","last_fullsized":"2024-02-13T00:39:23.368202+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.301282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368528+0000","last_change":"2024-02-13T00:39:00.215830+0000","last_active":"2024-02-13T00:39:23.368528+0000","last_peered":"2024-02-13T00:39:23.368528+0000","last_clean":"2024-02-13T00:39:23.368528+0000","last_became_active":"2024-02-13T00:37:41.218333+0000","last_became_peered":"2024-02-13T00:37:41.218333+0000","last_unstale":"2024-02-13T00:39:23.368528+0000","last_undegraded":"2024-02-13T00:39:23.368528+0000","last_fullsized":"2024-02-13T00:39:23.368528+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.215755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.215755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:26.214174+0000","last_change":"2024-02-13T00:39:26.214150+0000","last_active":"2024-02-13T00:39:26.214174+0000","last_peered":"2024-02-13T00:39:26.214174+0000","last_clean":"2024-02-13T00:39:26.214174+0000","last_became_active":"2024-02-13T00:37:43.236026+0000","last_became_peered":"2024-02-13T00:37:43.236026+0000","last_unstale":"2024-02-13T00:39:26.214174+0000","last_undegraded":"2024-02-13T00:39:26.214174+0000","last_fullsized":"2024-02-13T00:39:26.214174+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:39:26.214084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:26.214084+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368515+0000","last_change":"2024-02-13T00:37:47.552844+0000","last_active":"2024-02-13T00:39:23.368515+0000","last_peered":"2024-02-13T00:39:23.368515+0000","last_clean":"2024-02-13T00:39:23.368515+0000","last_became_active":"2024-02-13T00:37:47.552547+0000","last_became_peered":"2024-02-13T00:37:47.552547+0000","last_unstale":"2024-02-13T00:39:23.368515+0000","last_undegraded":"2024-02-13T00:39:23.368515+0000","last_fullsized":"2024-02-13T00:39:23.368515+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:37:46.228444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:30.245605+0000","last_change":"2024-02-13T00:39:30.245586+0000","last_active":"2024-02-13T00:39:30.245605+0000","last_peered":"2024-02-13T00:39:30.245605+0000","last_clean":"2024-02-13T00:39:30.245605+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:39:30.245605+0000","last_undegraded":"2024-02-13T00:39:30.245605+0000","last_fullsized":"2024-02-13T00:39:30.245605+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:30.245508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.245508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138197+0000","last_change":"2024-02-13T00:39:10.364972+0000","last_active":"2024-02-13T00:39:24.138197+0000","last_peered":"2024-02-13T00:39:24.138197+0000","last_clean":"2024-02-13T00:39:24.138197+0000","last_became_active":"2024-02-13T00:37:41.218334+0000","last_became_peered":"2024-02-13T00:37:41.218334+0000","last_unstale":"2024-02-13T00:39:24.138197+0000","last_undegraded":"2024-02-13T00:39:24.138197+0000","last_fullsized":"2024-02-13T00:39:24.138197+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:10.364865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.364865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.320654+0000","last_change":"2024-02-13T00:39:08.582559+0000","last_active":"2024-02-13T00:39:23.320654+0000","last_peered":"2024-02-13T00:39:23.320654+0000","last_clean":"2024-02-13T00:39:23.320654+0000","last_became_active":"2024-02-13T00:37:43.232938+0000","last_became_peered":"2024-02-13T00:37:43.232938+0000","last_unstale":"2024-02-13T00:39:23.320654+0000","last_undegraded":"2024-02-13T00:39:23.320654+0000","last_fullsized":"2024-02-13T00:39:23.320654+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:39:08.582479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.582479+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319675+0000","last_change":"2024-02-13T00:39:00.452185+0000","last_active":"2024-02-13T00:39:23.319675+0000","last_peered":"2024-02-13T00:39:23.319675+0000","last_clean":"2024-02-13T00:39:23.319675+0000","last_became_active":"2024-02-13T00:37:47.260182+0000","last_became_peered":"2024-02-13T00:37:47.260182+0000","last_unstale":"2024-02-13T00:39:23.319675+0000","last_undegraded":"2024-02-13T00:39:23.319675+0000","last_fullsized":"2024-02-13T00:39:23.319675+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.452084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.452084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040900+0000","last_change":"2024-02-13T00:39:15.328037+0000","last_active":"2024-02-13T00:39:24.040900+0000","last_peered":"2024-02-13T00:39:24.040900+0000","last_clean":"2024-02-13T00:39:24.040900+0000","last_became_active":"2024-02-13T00:37:45.249064+0000","last_became_peered":"2024-02-13T00:37:45.249064+0000","last_unstale":"2024-02-13T00:39:24.040900+0000","last_undegraded":"2024-02-13T00:39:24.040900+0000","last_fullsized":"2024-02-13T00:39:24.040900+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.327971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.327971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:41.231368+0000","last_change":"2024-02-13T00:39:41.231336+0000","last_active":"2024-02-13T00:39:41.231368+0000","last_peered":"2024-02-13T00:39:41.231368+0000","last_clean":"2024-02-13T00:39:41.231368+0000","last_became_active":"2024-02-13T00:37:41.216574+0000","last_became_peered":"2024-02-13T00:37:41.216574+0000","last_unstale":"2024-02-13T00:39:41.231368+0000","last_undegraded":"2024-02-13T00:39:41.231368+0000","last_fullsized":"2024-02-13T00:39:41.231368+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:41.231244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:41.231244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138586+0000","last_change":"2024-02-13T00:38:58.341512+0000","last_active":"2024-02-13T00:39:24.138586+0000","last_peered":"2024-02-13T00:39:24.138586+0000","last_clean":"2024-02-13T00:39:24.138586+0000","last_became_active":"2024-02-13T00:37:43.230491+0000","last_became_peered":"2024-02-13T00:37:43.230491+0000","last_unstale":"2024-02-13T00:39:24.138586+0000","last_undegraded":"2024-02-13T00:39:24.138586+0000","last_fullsized":"2024-02-13T00:39:24.138586+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:38:58.341406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.341406+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.136647+0000","last_change":"2024-02-13T00:38:50.866411+0000","last_active":"2024-02-13T00:39:24.136647+0000","last_peered":"2024-02-13T00:39:24.136647+0000","last_clean":"2024-02-13T00:39:24.136647+0000","last_became_active":"2024-02-13T00:37:47.254586+0000","last_became_peered":"2024-02-13T00:37:47.254586+0000","last_unstale":"2024-02-13T00:39:24.136647+0000","last_undegraded":"2024-02-13T00:39:24.136647+0000","last_fullsized":"2024-02-13T00:39:24.136647+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:38:50.866323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:50.866323+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"64'8","reported_seq":77,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:49.236357+0000","last_change":"2024-02-13T00:39:02.452828+0000","last_active":"2024-02-13T00:39:49.236357+0000","last_peered":"2024-02-13T00:39:49.236357+0000","last_clean":"2024-02-13T00:39:49.236357+0000","last_became_active":"2024-02-13T00:37:45.249902+0000","last_became_peered":"2024-02-13T00:37:45.249902+0000","last_unstale":"2024-02-13T00:39:49.236357+0000","last_undegraded":"2024-02-13T00:39:49.236357+0000","last_fullsized":"2024-02-13T00:39:49.236357+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:02.452755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.452755+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313321+0000","last_change":"2024-02-13T00:38:49.606456+0000","last_active":"2024-02-13T00:39:23.313321+0000","last_peered":"2024-02-13T00:39:23.313321+0000","last_clean":"2024-02-13T00:39:23.313321+0000","last_became_active":"2024-02-13T00:37:41.214965+0000","last_became_peered":"2024-02-13T00:37:41.214965+0000","last_unstale":"2024-02-13T00:39:23.313321+0000","last_undegraded":"2024-02-13T00:39:23.313321+0000","last_fullsized":"2024-02-13T00:39:23.313321+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:49.606357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:49.606357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040877+0000","last_change":"2024-02-13T00:38:52.380430+0000","last_active":"2024-02-13T00:39:24.040877+0000","last_peered":"2024-02-13T00:39:24.040877+0000","last_clean":"2024-02-13T00:39:24.040877+0000","last_became_active":"2024-02-13T00:37:43.324221+0000","last_became_peered":"2024-02-13T00:37:43.324221+0000","last_unstale":"2024-02-13T00:39:24.040877+0000","last_undegraded":"2024-02-13T00:39:24.040877+0000","last_fullsized":"2024-02-13T00:39:24.040877+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.380369+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368642+0000","last_change":"2024-02-13T00:39:21.243606+0000","last_active":"2024-02-13T00:39:23.368642+0000","last_peered":"2024-02-13T00:39:23.368642+0000","last_clean":"2024-02-13T00:39:23.368642+0000","last_became_active":"2024-02-13T00:37:47.552917+0000","last_became_peered":"2024-02-13T00:37:47.552917+0000","last_unstale":"2024-02-13T00:39:23.368642+0000","last_undegraded":"2024-02-13T00:39:23.368642+0000","last_fullsized":"2024-02-13T00:39:23.368642+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.243527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135696+0000","last_change":"2024-02-13T00:39:13.027403+0000","last_active":"2024-02-13T00:39:24.135696+0000","last_peered":"2024-02-13T00:39:24.135696+0000","last_clean":"2024-02-13T00:39:24.135696+0000","last_became_active":"2024-02-13T00:37:45.248592+0000","last_became_peered":"2024-02-13T00:37:45.248592+0000","last_unstale":"2024-02-13T00:39:24.135696+0000","last_undegraded":"2024-02-13T00:39:24.135696+0000","last_fullsized":"2024-02-13T00:39:24.135696+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:13.027299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:13.027299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138660+0000","last_change":"2024-02-13T00:39:07.358630+0000","last_active":"2024-02-13T00:39:24.138660+0000","last_peered":"2024-02-13T00:39:24.138660+0000","last_clean":"2024-02-13T00:39:24.138660+0000","last_became_active":"2024-02-13T00:37:41.220344+0000","last_became_peered":"2024-02-13T00:37:41.220344+0000","last_unstale":"2024-02-13T00:39:24.138660+0000","last_undegraded":"2024-02-13T00:39:24.138660+0000","last_fullsized":"2024-02-13T00:39:24.138660+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.358534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.358534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:34.205013+0000","last_change":"2024-02-13T00:39:34.204969+0000","last_active":"2024-02-13T00:39:34.205013+0000","last_peered":"2024-02-13T00:39:34.205013+0000","last_clean":"2024-02-13T00:39:34.205013+0000","last_became_active":"2024-02-13T00:37:43.324196+0000","last_became_peered":"2024-02-13T00:37:43.324196+0000","last_unstale":"2024-02-13T00:39:34.205013+0000","last_undegraded":"2024-02-13T00:39:34.205013+0000","last_fullsized":"2024-02-13T00:39:34.205013+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:34.204888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.204888+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135581+0000","last_change":"2024-02-13T00:38:57.818305+0000","last_active":"2024-02-13T00:39:24.135581+0000","last_peered":"2024-02-13T00:39:24.135581+0000","last_clean":"2024-02-13T00:39:24.135581+0000","last_became_active":"2024-02-13T00:37:47.254667+0000","last_became_peered":"2024-02-13T00:37:47.254667+0000","last_unstale":"2024-02-13T00:39:24.135581+0000","last_undegraded":"2024-02-13T00:39:24.135581+0000","last_fullsized":"2024-02-13T00:39:24.135581+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:57.818202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.818202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138007+0000","last_change":"2024-02-13T00:38:52.493308+0000","last_active":"2024-02-13T00:39:24.138007+0000","last_peered":"2024-02-13T00:39:24.138007+0000","last_clean":"2024-02-13T00:39:24.138007+0000","last_became_active":"2024-02-13T00:37:45.249556+0000","last_became_peered":"2024-02-13T00:37:45.249556+0000","last_unstale":"2024-02-13T00:39:24.138007+0000","last_undegraded":"2024-02-13T00:39:24.138007+0000","last_fullsized":"2024-02-13T00:39:24.138007+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.493237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.493237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313839+0000","last_change":"2024-02-13T00:38:55.573704+0000","last_active":"2024-02-13T00:39:23.313839+0000","last_peered":"2024-02-13T00:39:23.313839+0000","last_clean":"2024-02-13T00:39:23.313839+0000","last_became_active":"2024-02-13T00:37:41.213033+0000","last_became_peered":"2024-02-13T00:37:41.213033+0000","last_unstale":"2024-02-13T00:39:23.313839+0000","last_undegraded":"2024-02-13T00:39:23.313839+0000","last_fullsized":"2024-02-13T00:39:23.313839+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:55.573637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:55.573637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368248+0000","last_change":"2024-02-13T00:39:23.231537+0000","last_active":"2024-02-13T00:39:23.368248+0000","last_peered":"2024-02-13T00:39:23.368248+0000","last_clean":"2024-02-13T00:39:23.368248+0000","last_became_active":"2024-02-13T00:37:43.234395+0000","last_became_peered":"2024-02-13T00:37:43.234395+0000","last_unstale":"2024-02-13T00:39:23.368248+0000","last_undegraded":"2024-02-13T00:39:23.368248+0000","last_fullsized":"2024-02-13T00:39:23.368248+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:39:23.231401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.231401+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368160+0000","last_change":"2024-02-13T00:39:01.197382+0000","last_active":"2024-02-13T00:39:23.368160+0000","last_peered":"2024-02-13T00:39:23.368160+0000","last_clean":"2024-02-13T00:39:23.368160+0000","last_became_active":"2024-02-13T00:37:47.254519+0000","last_became_peered":"2024-02-13T00:37:47.254519+0000","last_unstale":"2024-02-13T00:39:23.368160+0000","last_undegraded":"2024-02-13T00:39:23.368160+0000","last_fullsized":"2024-02-13T00:39:23.368160+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.197310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.197310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319556+0000","last_change":"2024-02-13T00:38:58.512117+0000","last_active":"2024-02-13T00:39:23.319556+0000","last_peered":"2024-02-13T00:39:23.319556+0000","last_clean":"2024-02-13T00:39:23.319556+0000","last_became_active":"2024-02-13T00:37:45.247114+0000","last_became_peered":"2024-02-13T00:37:45.247114+0000","last_unstale":"2024-02-13T00:39:23.319556+0000","last_undegraded":"2024-02-13T00:39:23.319556+0000","last_fullsized":"2024-02-13T00:39:23.319556+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.512053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:27.297205+0000","last_change":"2024-02-13T00:39:27.297182+0000","last_active":"2024-02-13T00:39:27.297205+0000","last_peered":"2024-02-13T00:39:27.297205+0000","last_clean":"2024-02-13T00:39:27.297205+0000","last_became_active":"2024-02-13T00:37:41.215531+0000","last_became_peered":"2024-02-13T00:37:41.215531+0000","last_unstale":"2024-02-13T00:39:27.297205+0000","last_undegraded":"2024-02-13T00:39:27.297205+0000","last_fullsized":"2024-02-13T00:39:27.297205+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:27.297107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:27.297107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":85,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368294+0000","last_change":"2024-02-13T00:39:06.398068+0000","last_active":"2024-02-13T00:39:23.368294+0000","last_peered":"2024-02-13T00:39:23.368294+0000","last_clean":"2024-02-13T00:39:23.368294+0000","last_became_active":"2024-02-13T00:39:05.965554+0000","last_became_peered":"2024-02-13T00:39:05.965554+0000","last_unstale":"2024-02-13T00:39:23.368294+0000","last_undegraded":"2024-02-13T00:39:23.368294+0000","last_fullsized":"2024-02-13T00:39:23.368294+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:39:03.189237+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:28.286655+0000","last_change":"2024-02-13T00:39:28.286635+0000","last_active":"2024-02-13T00:39:28.286655+0000","last_peered":"2024-02-13T00:39:28.286655+0000","last_clean":"2024-02-13T00:39:28.286655+0000","last_became_active":"2024-02-13T00:37:47.259792+0000","last_became_peered":"2024-02-13T00:37:47.259792+0000","last_unstale":"2024-02-13T00:39:28.286655+0000","last_undegraded":"2024-02-13T00:39:28.286655+0000","last_fullsized":"2024-02-13T00:39:28.286655+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.286574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.286574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:29.229804+0000","last_change":"2024-02-13T00:39:29.229760+0000","last_active":"2024-02-13T00:39:29.229804+0000","last_peered":"2024-02-13T00:39:29.229804+0000","last_clean":"2024-02-13T00:39:29.229804+0000","last_became_active":"2024-02-13T00:37:45.243594+0000","last_became_peered":"2024-02-13T00:37:45.243594+0000","last_unstale":"2024-02-13T00:39:29.229804+0000","last_undegraded":"2024-02-13T00:39:29.229804+0000","last_fullsized":"2024-02-13T00:39:29.229804+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.229676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138404+0000","last_change":"2024-02-13T00:39:00.395451+0000","last_active":"2024-02-13T00:39:24.138404+0000","last_peered":"2024-02-13T00:39:24.138404+0000","last_clean":"2024-02-13T00:39:24.138404+0000","last_became_active":"2024-02-13T00:37:41.216390+0000","last_became_peered":"2024-02-13T00:37:41.216390+0000","last_unstale":"2024-02-13T00:39:24.138404+0000","last_undegraded":"2024-02-13T00:39:24.138404+0000","last_fullsized":"2024-02-13T00:39:24.138404+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.395381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.395381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319472+0000","last_change":"2024-02-13T00:39:12.348547+0000","last_active":"2024-02-13T00:39:23.319472+0000","last_peered":"2024-02-13T00:39:23.319472+0000","last_clean":"2024-02-13T00:39:23.319472+0000","last_became_active":"2024-02-13T00:37:43.225347+0000","last_became_peered":"2024-02-13T00:37:43.225347+0000","last_unstale":"2024-02-13T00:39:23.319472+0000","last_undegraded":"2024-02-13T00:39:23.319472+0000","last_fullsized":"2024-02-13T00:39:23.319472+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:39:12.348480+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368585+0000","last_change":"2024-02-13T00:37:47.254258+0000","last_active":"2024-02-13T00:39:23.368585+0000","last_peered":"2024-02-13T00:39:23.368585+0000","last_clean":"2024-02-13T00:39:23.368585+0000","last_became_active":"2024-02-13T00:37:47.253945+0000","last_became_peered":"2024-02-13T00:37:47.253945+0000","last_unstale":"2024-02-13T00:39:23.368585+0000","last_undegraded":"2024-02-13T00:39:23.368585+0000","last_fullsized":"2024-02-13T00:39:23.368585+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:37:46.228444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"84'9","reported_seq":77,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:49.236381+0000","last_change":"2024-02-13T00:39:22.269192+0000","last_active":"2024-02-13T00:39:49.236381+0000","last_peered":"2024-02-13T00:39:49.236381+0000","last_clean":"2024-02-13T00:39:49.236381+0000","last_became_active":"2024-02-13T00:37:45.250598+0000","last_became_peered":"2024-02-13T00:37:45.250598+0000","last_unstale":"2024-02-13T00:39:49.236381+0000","last_undegraded":"2024-02-13T00:39:49.236381+0000","last_fullsized":"2024-02-13T00:39:49.236381+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.269104+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040525+0000","last_change":"2024-02-13T00:38:58.314612+0000","last_active":"2024-02-13T00:39:24.040525+0000","last_peered":"2024-02-13T00:39:24.040525+0000","last_clean":"2024-02-13T00:39:24.040525+0000","last_became_active":"2024-02-13T00:37:41.217953+0000","last_became_peered":"2024-02-13T00:37:41.217953+0000","last_unstale":"2024-02-13T00:39:24.040525+0000","last_undegraded":"2024-02-13T00:39:24.040525+0000","last_fullsized":"2024-02-13T00:39:24.040525+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.314542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.314542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313975+0000","last_change":"2024-02-13T00:39:10.549128+0000","last_active":"2024-02-13T00:39:23.313975+0000","last_peered":"2024-02-13T00:39:23.313975+0000","last_clean":"2024-02-13T00:39:23.313975+0000","last_became_active":"2024-02-13T00:37:43.322357+0000","last_became_peered":"2024-02-13T00:37:43.322357+0000","last_unstale":"2024-02-13T00:39:23.313975+0000","last_undegraded":"2024-02-13T00:39:23.313975+0000","last_fullsized":"2024-02-13T00:39:23.313975+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:39:10.549045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.549045+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040713+0000","last_change":"2024-02-13T00:39:21.344836+0000","last_active":"2024-02-13T00:39:24.040713+0000","last_peered":"2024-02-13T00:39:24.040713+0000","last_clean":"2024-02-13T00:39:24.040713+0000","last_became_active":"2024-02-13T00:37:47.252722+0000","last_became_peered":"2024-02-13T00:37:47.252722+0000","last_unstale":"2024-02-13T00:39:24.040713+0000","last_undegraded":"2024-02-13T00:39:24.040713+0000","last_fullsized":"2024-02-13T00:39:24.040713+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.344771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.344771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"64'8","reported_seq":73,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:49.236325+0000","last_change":"2024-02-13T00:39:11.518468+0000","last_active":"2024-02-13T00:39:49.236325+0000","last_peered":"2024-02-13T00:39:49.236325+0000","last_clean":"2024-02-13T00:39:49.236325+0000","last_became_active":"2024-02-13T00:37:45.246757+0000","last_became_peered":"2024-02-13T00:37:45.246757+0000","last_unstale":"2024-02-13T00:39:49.236325+0000","last_undegraded":"2024-02-13T00:39:49.236325+0000","last_fullsized":"2024-02-13T00:39:49.236325+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:11.518359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:11.518359+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313484+0000","last_change":"2024-02-13T00:39:01.561276+0000","last_active":"2024-02-13T00:39:23.313484+0000","last_peered":"2024-02-13T00:39:23.313484+0000","last_clean":"2024-02-13T00:39:23.313484+0000","last_became_active":"2024-02-13T00:37:41.219309+0000","last_became_peered":"2024-02-13T00:37:41.219309+0000","last_unstale":"2024-02-13T00:39:23.313484+0000","last_undegraded":"2024-02-13T00:39:23.313484+0000","last_fullsized":"2024-02-13T00:39:23.313484+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.561157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.561157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":92,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368503+0000","last_change":"2024-02-13T00:38:46.169426+0000","last_active":"2024-02-13T00:39:23.368503+0000","last_peered":"2024-02-13T00:39:23.368503+0000","last_clean":"2024-02-13T00:39:23.368503+0000","last_became_active":"2024-02-13T00:37:43.324313+0000","last_became_peered":"2024-02-13T00:37:43.324313+0000","last_unstale":"2024-02-13T00:39:23.368503+0000","last_undegraded":"2024-02-13T00:39:23.368503+0000","last_fullsized":"2024-02-13T00:39:23.368503+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:38:46.169344+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:25.166875+0000","last_change":"2024-02-13T00:39:25.166852+0000","last_active":"2024-02-13T00:39:25.166875+0000","last_peered":"2024-02-13T00:39:25.166875+0000","last_clean":"2024-02-13T00:39:25.166875+0000","last_became_active":"2024-02-13T00:37:47.553612+0000","last_became_peered":"2024-02-13T00:37:47.553612+0000","last_unstale":"2024-02-13T00:39:25.166875+0000","last_undegraded":"2024-02-13T00:39:25.166875+0000","last_fullsized":"2024-02-13T00:39:25.166875+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:25.166777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.166777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313787+0000","last_change":"2024-02-13T00:39:07.556868+0000","last_active":"2024-02-13T00:39:23.313787+0000","last_peered":"2024-02-13T00:39:23.313787+0000","last_clean":"2024-02-13T00:39:23.313787+0000","last_became_active":"2024-02-13T00:37:45.246907+0000","last_became_peered":"2024-02-13T00:37:45.246907+0000","last_unstale":"2024-02-13T00:39:23.313787+0000","last_undegraded":"2024-02-13T00:39:23.313787+0000","last_fullsized":"2024-02-13T00:39:23.313787+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.556772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.556772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138075+0000","last_change":"2024-02-13T00:39:01.439623+0000","last_active":"2024-02-13T00:39:24.138075+0000","last_peered":"2024-02-13T00:39:24.138075+0000","last_clean":"2024-02-13T00:39:24.138075+0000","last_became_active":"2024-02-13T00:37:41.212866+0000","last_became_peered":"2024-02-13T00:37:41.212866+0000","last_unstale":"2024-02-13T00:39:24.138075+0000","last_undegraded":"2024-02-13T00:39:24.138075+0000","last_fullsized":"2024-02-13T00:39:24.138075+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.439513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368814+0000","last_change":"2024-02-13T00:39:19.266801+0000","last_active":"2024-02-13T00:39:23.368814+0000","last_peered":"2024-02-13T00:39:23.368814+0000","last_clean":"2024-02-13T00:39:23.368814+0000","last_became_active":"2024-02-13T00:37:43.324510+0000","last_became_peered":"2024-02-13T00:37:43.324510+0000","last_unstale":"2024-02-13T00:39:23.368814+0000","last_undegraded":"2024-02-13T00:39:23.368814+0000","last_fullsized":"2024-02-13T00:39:23.368814+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:39:19.266729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:19.266729+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.320477+0000","last_change":"2024-02-13T00:39:22.615864+0000","last_active":"2024-02-13T00:39:23.320477+0000","last_peered":"2024-02-13T00:39:23.320477+0000","last_clean":"2024-02-13T00:39:23.320477+0000","last_became_active":"2024-02-13T00:37:47.254803+0000","last_became_peered":"2024-02-13T00:37:47.254803+0000","last_unstale":"2024-02-13T00:39:23.320477+0000","last_undegraded":"2024-02-13T00:39:23.320477+0000","last_fullsized":"2024-02-13T00:39:23.320477+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.615773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.615773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:46.155792+0000","last_change":"2024-02-13T00:39:46.155792+0000","last_active":"2024-02-13T00:39:46.155792+0000","last_peered":"2024-02-13T00:39:46.155792+0000","last_clean":"2024-02-13T00:39:46.155792+0000","last_became_active":"2024-02-13T00:37:45.250632+0000","last_became_peered":"2024-02-13T00:37:45.250632+0000","last_unstale":"2024-02-13T00:39:46.155792+0000","last_undegraded":"2024-02-13T00:39:46.155792+0000","last_fullsized":"2024-02-13T00:39:46.155792+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:46.155718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:46.155718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.318963+0000","last_change":"2024-02-13T00:39:01.569173+0000","last_active":"2024-02-13T00:39:23.318963+0000","last_peered":"2024-02-13T00:39:23.318963+0000","last_clean":"2024-02-13T00:39:23.318963+0000","last_became_active":"2024-02-13T00:37:41.214091+0000","last_became_peered":"2024-02-13T00:37:41.214091+0000","last_unstale":"2024-02-13T00:39:23.318963+0000","last_undegraded":"2024-02-13T00:39:23.318963+0000","last_fullsized":"2024-02-13T00:39:23.318963+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.569109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.569109+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":86,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040815+0000","last_change":"2024-02-13T00:38:56.331799+0000","last_active":"2024-02-13T00:39:24.040815+0000","last_peered":"2024-02-13T00:39:24.040815+0000","last_clean":"2024-02-13T00:39:24.040815+0000","last_became_active":"2024-02-13T00:37:43.322401+0000","last_became_peered":"2024-02-13T00:37:43.322401+0000","last_unstale":"2024-02-13T00:39:24.040815+0000","last_undegraded":"2024-02-13T00:39:24.040815+0000","last_fullsized":"2024-02-13T00:39:24.040815+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:38:56.331703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.331703+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:33.179336+0000","last_change":"2024-02-13T00:39:33.179309+0000","last_active":"2024-02-13T00:39:33.179336+0000","last_peered":"2024-02-13T00:39:33.179336+0000","last_clean":"2024-02-13T00:39:33.179336+0000","last_became_active":"2024-02-13T00:37:47.260242+0000","last_became_peered":"2024-02-13T00:37:47.260242+0000","last_unstale":"2024-02-13T00:39:33.179336+0000","last_undegraded":"2024-02-13T00:39:33.179336+0000","last_fullsized":"2024-02-13T00:39:33.179336+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.179225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.179225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":55,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368502+0000","last_change":"2024-02-13T00:38:48.379316+0000","last_active":"2024-02-13T00:39:23.368502+0000","last_peered":"2024-02-13T00:39:23.368502+0000","last_clean":"2024-02-13T00:39:23.368502+0000","last_became_active":"2024-02-13T00:37:41.219249+0000","last_became_peered":"2024-02-13T00:37:41.219249+0000","last_unstale":"2024-02-13T00:39:23.368502+0000","last_undegraded":"2024-02-13T00:39:23.368502+0000","last_fullsized":"2024-02-13T00:39:23.368502+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:38:48.379184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:48.379184+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":88,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368158+0000","last_change":"2024-02-13T00:37:43.229199+0000","last_active":"2024-02-13T00:39:23.368158+0000","last_peered":"2024-02-13T00:39:23.368158+0000","last_clean":"2024-02-13T00:39:23.368158+0000","last_became_active":"2024-02-13T00:37:43.228898+0000","last_became_peered":"2024-02-13T00:37:43.228898+0000","last_unstale":"2024-02-13T00:39:23.368158+0000","last_undegraded":"2024-02-13T00:39:23.368158+0000","last_fullsized":"2024-02-13T00:39:23.368158+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.318820+0000","last_change":"2024-02-13T00:39:21.654834+0000","last_active":"2024-02-13T00:39:23.318820+0000","last_peered":"2024-02-13T00:39:23.318820+0000","last_clean":"2024-02-13T00:39:23.318820+0000","last_became_active":"2024-02-13T00:37:45.244408+0000","last_became_peered":"2024-02-13T00:37:45.244408+0000","last_unstale":"2024-02-13T00:39:23.318820+0000","last_undegraded":"2024-02-13T00:39:23.318820+0000","last_fullsized":"2024-02-13T00:39:23.318820+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.654733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.654733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040159+0000","last_change":"2024-02-13T00:38:54.349240+0000","last_active":"2024-02-13T00:39:24.040159+0000","last_peered":"2024-02-13T00:39:24.040159+0000","last_clean":"2024-02-13T00:39:24.040159+0000","last_became_active":"2024-02-13T00:37:47.255928+0000","last_became_peered":"2024-02-13T00:37:47.255928+0000","last_unstale":"2024-02-13T00:39:24.040159+0000","last_undegraded":"2024-02-13T00:39:24.040159+0000","last_fullsized":"2024-02-13T00:39:24.040159+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:54.349171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:54.349171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319474+0000","last_change":"2024-02-13T00:38:49.618099+0000","last_active":"2024-02-13T00:39:23.319474+0000","last_peered":"2024-02-13T00:39:23.319474+0000","last_clean":"2024-02-13T00:39:23.319474+0000","last_became_active":"2024-02-13T00:37:41.217777+0000","last_became_peered":"2024-02-13T00:37:41.217777+0000","last_unstale":"2024-02-13T00:39:23.319474+0000","last_undegraded":"2024-02-13T00:39:23.319474+0000","last_fullsized":"2024-02-13T00:39:23.319474+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:38:49.617988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":28,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.320080+0000","last_change":"2024-02-13T00:39:05.974172+0000","last_active":"2024-02-13T00:39:23.320080+0000","last_peered":"2024-02-13T00:39:23.320080+0000","last_clean":"2024-02-13T00:39:23.320080+0000","last_became_active":"2024-02-13T00:39:05.965560+0000","last_became_peered":"2024-02-13T00:39:05.965560+0000","last_unstale":"2024-02-13T00:39:23.320080+0000","last_undegraded":"2024-02-13T00:39:23.320080+0000","last_fullsized":"2024-02-13T00:39:23.320080+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:39:01.456507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.456507+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"64'8","reported_seq":73,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:49.236288+0000","last_change":"2024-02-13T00:39:30.273237+0000","last_active":"2024-02-13T00:39:49.236288+0000","last_peered":"2024-02-13T00:39:49.236288+0000","last_clean":"2024-02-13T00:39:49.236288+0000","last_became_active":"2024-02-13T00:37:45.241274+0000","last_became_peered":"2024-02-13T00:37:45.241274+0000","last_unstale":"2024-02-13T00:39:49.236288+0000","last_undegraded":"2024-02-13T00:39:49.236288+0000","last_fullsized":"2024-02-13T00:39:49.236288+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:30.273165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.273165+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:36.157813+0000","last_change":"2024-02-13T00:39:36.157777+0000","last_active":"2024-02-13T00:39:36.157813+0000","last_peered":"2024-02-13T00:39:36.157813+0000","last_clean":"2024-02-13T00:39:36.157813+0000","last_became_active":"2024-02-13T00:37:47.252270+0000","last_became_peered":"2024-02-13T00:37:47.252270+0000","last_unstale":"2024-02-13T00:39:36.157813+0000","last_undegraded":"2024-02-13T00:39:36.157813+0000","last_fullsized":"2024-02-13T00:39:36.157813+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:36.157693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:36.157693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313530+0000","last_change":"2024-02-13T00:38:58.528669+0000","last_active":"2024-02-13T00:39:23.313530+0000","last_peered":"2024-02-13T00:39:23.313530+0000","last_clean":"2024-02-13T00:39:23.313530+0000","last_became_active":"2024-02-13T00:37:41.212924+0000","last_became_peered":"2024-02-13T00:37:41.212924+0000","last_unstale":"2024-02-13T00:39:23.313530+0000","last_undegraded":"2024-02-13T00:39:23.313530+0000","last_fullsized":"2024-02-13T00:39:23.313530+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.528580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.528580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"80'45","reported_seq":142,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:49.236294+0000","last_change":"2024-02-13T00:38:57.537373+0000","last_active":"2024-02-13T00:39:49.236294+0000","last_peered":"2024-02-13T00:39:49.236294+0000","last_clean":"2024-02-13T00:39:49.236294+0000","last_became_active":"2024-02-13T00:37:43.232014+0000","last_became_peered":"2024-02-13T00:37:43.232014+0000","last_unstale":"2024-02-13T00:39:49.236294+0000","last_undegraded":"2024-02-13T00:39:49.236294+0000","last_fullsized":"2024-02-13T00:39:49.236294+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'45","last_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_deep_scrub":"80'45","last_deep_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.537309+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040804+0000","last_change":"2024-02-13T00:38:47.393545+0000","last_active":"2024-02-13T00:39:24.040804+0000","last_peered":"2024-02-13T00:39:24.040804+0000","last_clean":"2024-02-13T00:39:24.040804+0000","last_became_active":"2024-02-13T00:37:45.247012+0000","last_became_peered":"2024-02-13T00:37:45.247012+0000","last_unstale":"2024-02-13T00:39:24.040804+0000","last_undegraded":"2024-02-13T00:39:24.040804+0000","last_fullsized":"2024-02-13T00:39:24.040804+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:47.393477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:47.393477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135417+0000","last_change":"2024-02-13T00:39:08.930351+0000","last_active":"2024-02-13T00:39:24.135417+0000","last_peered":"2024-02-13T00:39:24.135417+0000","last_clean":"2024-02-13T00:39:24.135417+0000","last_became_active":"2024-02-13T00:37:47.552693+0000","last_became_peered":"2024-02-13T00:37:47.552693+0000","last_unstale":"2024-02-13T00:39:24.135417+0000","last_undegraded":"2024-02-13T00:39:24.135417+0000","last_fullsized":"2024-02-13T00:39:24.135417+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:08.930286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.930286+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":175,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:53.543290+0000","last_change":"2024-02-13T00:39:10.287406+0000","last_active":"2024-02-13T00:39:53.543290+0000","last_peered":"2024-02-13T00:39:53.543290+0000","last_clean":"2024-02-13T00:39:53.543290+0000","last_became_active":"2024-02-13T00:37:41.215137+0000","last_became_peered":"2024-02-13T00:37:41.215137+0000","last_unstale":"2024-02-13T00:39:53.543290+0000","last_undegraded":"2024-02-13T00:39:53.543290+0000","last_fullsized":"2024-02-13T00:39:53.543290+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-13T00:39:10.287299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.287299+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":127,"num_read_kb":122,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.318856+0000","last_change":"2024-02-13T00:39:20.703787+0000","last_active":"2024-02-13T00:39:23.318856+0000","last_peered":"2024-02-13T00:39:23.318856+0000","last_clean":"2024-02-13T00:39:23.318856+0000","last_became_active":"2024-02-13T00:37:41.214442+0000","last_became_peered":"2024-02-13T00:37:41.214442+0000","last_unstale":"2024-02-13T00:39:23.318856+0000","last_undegraded":"2024-02-13T00:39:23.318856+0000","last_fullsized":"2024-02-13T00:39:23.318856+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:20.703700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:20.703700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":83,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:25.319209+0000","last_change":"2024-02-13T00:39:25.319173+0000","last_active":"2024-02-13T00:39:25.319209+0000","last_peered":"2024-02-13T00:39:25.319209+0000","last_clean":"2024-02-13T00:39:25.319209+0000","last_became_active":"2024-02-13T00:37:43.231120+0000","last_became_peered":"2024-02-13T00:37:43.231120+0000","last_unstale":"2024-02-13T00:39:25.319209+0000","last_undegraded":"2024-02-13T00:39:25.319209+0000","last_fullsized":"2024-02-13T00:39:25.319209+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:39:25.319066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.319066+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319718+0000","last_change":"2024-02-13T00:39:09.589876+0000","last_active":"2024-02-13T00:39:23.319718+0000","last_peered":"2024-02-13T00:39:23.319718+0000","last_clean":"2024-02-13T00:39:23.319718+0000","last_became_active":"2024-02-13T00:37:45.248383+0000","last_became_peered":"2024-02-13T00:37:45.248383+0000","last_unstale":"2024-02-13T00:39:23.319718+0000","last_undegraded":"2024-02-13T00:39:23.319718+0000","last_fullsized":"2024-02-13T00:39:23.319718+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:09.589813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:09.589813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.314207+0000","last_change":"2024-02-13T00:39:05.596529+0000","last_active":"2024-02-13T00:39:23.314207+0000","last_peered":"2024-02-13T00:39:23.314207+0000","last_clean":"2024-02-13T00:39:23.314207+0000","last_became_active":"2024-02-13T00:37:47.253996+0000","last_became_peered":"2024-02-13T00:37:47.253996+0000","last_unstale":"2024-02-13T00:39:23.314207+0000","last_undegraded":"2024-02-13T00:39:23.314207+0000","last_fullsized":"2024-02-13T00:39:23.314207+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:39:05.596462+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368285+0000","last_change":"2024-02-13T00:39:04.221630+0000","last_active":"2024-02-13T00:39:23.368285+0000","last_peered":"2024-02-13T00:39:23.368285+0000","last_clean":"2024-02-13T00:39:23.368285+0000","last_became_active":"2024-02-13T00:37:41.216367+0000","last_became_peered":"2024-02-13T00:37:41.216367+0000","last_unstale":"2024-02-13T00:39:23.368285+0000","last_undegraded":"2024-02-13T00:39:23.368285+0000","last_fullsized":"2024-02-13T00:39:23.368285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:39:04.221506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313651+0000","last_change":"2024-02-13T00:38:52.618768+0000","last_active":"2024-02-13T00:39:23.313651+0000","last_peered":"2024-02-13T00:39:23.313651+0000","last_clean":"2024-02-13T00:39:23.313651+0000","last_became_active":"2024-02-13T00:37:41.216101+0000","last_became_peered":"2024-02-13T00:37:41.216101+0000","last_unstale":"2024-02-13T00:39:23.313651+0000","last_undegraded":"2024-02-13T00:39:23.313651+0000","last_fullsized":"2024-02-13T00:39:23.313651+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.618696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.618696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:41.255887+0000","last_change":"2024-02-13T00:39:41.255860+0000","last_active":"2024-02-13T00:39:41.255887+0000","last_peered":"2024-02-13T00:39:41.255887+0000","last_clean":"2024-02-13T00:39:41.255887+0000","last_became_active":"2024-02-13T00:37:35.635369+0000","last_became_peered":"2024-02-13T00:37:35.635369+0000","last_unstale":"2024-02-13T00:39:41.255887+0000","last_undegraded":"2024-02-13T00:39:41.255887+0000","last_fullsized":"2024-02-13T00:39:41.255887+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:39:41.255785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'14","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138522+0000","last_change":"2024-02-13T00:39:15.415087+0000","last_active":"2024-02-13T00:39:24.138522+0000","last_peered":"2024-02-13T00:39:24.138522+0000","last_clean":"2024-02-13T00:39:24.138522+0000","last_became_active":"2024-02-13T00:37:43.324209+0000","last_became_peered":"2024-02-13T00:37:43.324209+0000","last_unstale":"2024-02-13T00:39:24.138522+0000","last_undegraded":"2024-02-13T00:39:24.138522+0000","last_fullsized":"2024-02-13T00:39:24.138522+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'14","last_scrub_stamp":"2024-02-13T00:39:15.415004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.415004+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:28.171840+0000","last_change":"2024-02-13T00:39:28.171808+0000","last_active":"2024-02-13T00:39:28.171840+0000","last_peered":"2024-02-13T00:39:28.171840+0000","last_clean":"2024-02-13T00:39:28.171840+0000","last_became_active":"2024-02-13T00:37:45.247708+0000","last_became_peered":"2024-02-13T00:37:45.247708+0000","last_unstale":"2024-02-13T00:39:28.171840+0000","last_undegraded":"2024-02-13T00:39:28.171840+0000","last_fullsized":"2024-02-13T00:39:28.171840+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.171740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.171740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:32.255351+0000","last_change":"2024-02-13T00:39:32.255327+0000","last_active":"2024-02-13T00:39:32.255351+0000","last_peered":"2024-02-13T00:39:32.255351+0000","last_clean":"2024-02-13T00:39:32.255351+0000","last_became_active":"2024-02-13T00:37:47.262024+0000","last_became_peered":"2024-02-13T00:37:47.262024+0000","last_unstale":"2024-02-13T00:39:32.255351+0000","last_undegraded":"2024-02-13T00:39:32.255351+0000","last_fullsized":"2024-02-13T00:39:32.255351+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.255260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040433+0000","last_change":"2024-02-13T00:38:55.353229+0000","last_active":"2024-02-13T00:39:24.040433+0000","last_peered":"2024-02-13T00:39:24.040433+0000","last_clean":"2024-02-13T00:39:24.040433+0000","last_became_active":"2024-02-13T00:37:41.217577+0000","last_became_peered":"2024-02-13T00:37:41.217577+0000","last_unstale":"2024-02-13T00:39:24.040433+0000","last_undegraded":"2024-02-13T00:39:24.040433+0000","last_fullsized":"2024-02-13T00:39:24.040433+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:55.353157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:55.353157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.318596+0000","last_change":"2024-02-13T00:39:05.968028+0000","last_active":"2024-02-13T00:39:23.318596+0000","last_peered":"2024-02-13T00:39:23.318596+0000","last_clean":"2024-02-13T00:39:23.318596+0000","last_became_active":"2024-02-13T00:39:05.967772+0000","last_became_peered":"2024-02-13T00:39:05.967772+0000","last_unstale":"2024-02-13T00:39:23.318596+0000","last_undegraded":"2024-02-13T00:39:23.318596+0000","last_fullsized":"2024-02-13T00:39:23.318596+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:44.893697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:44.893697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":81,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040735+0000","last_change":"2024-02-13T00:39:08.344202+0000","last_active":"2024-02-13T00:39:24.040735+0000","last_peered":"2024-02-13T00:39:24.040735+0000","last_clean":"2024-02-13T00:39:24.040735+0000","last_became_active":"2024-02-13T00:39:05.968155+0000","last_became_peered":"2024-02-13T00:39:05.968155+0000","last_unstale":"2024-02-13T00:39:24.040735+0000","last_undegraded":"2024-02-13T00:39:24.040735+0000","last_fullsized":"2024-02-13T00:39:24.040735+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:39:08.344129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.344129+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.314146+0000","last_change":"2024-02-13T00:38:48.618900+0000","last_active":"2024-02-13T00:39:23.314146+0000","last_peered":"2024-02-13T00:39:23.314146+0000","last_clean":"2024-02-13T00:39:23.314146+0000","last_became_active":"2024-02-13T00:37:45.245956+0000","last_became_peered":"2024-02-13T00:37:45.245956+0000","last_unstale":"2024-02-13T00:39:23.314146+0000","last_undegraded":"2024-02-13T00:39:23.314146+0000","last_fullsized":"2024-02-13T00:39:23.314146+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:48.618830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:48.618830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:33.304159+0000","last_change":"2024-02-13T00:39:33.304107+0000","last_active":"2024-02-13T00:39:33.304159+0000","last_peered":"2024-02-13T00:39:33.304159+0000","last_clean":"2024-02-13T00:39:33.304159+0000","last_became_active":"2024-02-13T00:37:47.552672+0000","last_became_peered":"2024-02-13T00:37:47.552672+0000","last_unstale":"2024-02-13T00:39:33.304159+0000","last_undegraded":"2024-02-13T00:39:33.304159+0000","last_fullsized":"2024-02-13T00:39:33.304159+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.303982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.303982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.136021+0000","last_change":"2024-02-13T00:38:46.927138+0000","last_active":"2024-02-13T00:39:24.136021+0000","last_peered":"2024-02-13T00:39:24.136021+0000","last_clean":"2024-02-13T00:39:24.136021+0000","last_became_active":"2024-02-13T00:37:41.221636+0000","last_became_peered":"2024-02-13T00:37:41.221636+0000","last_unstale":"2024-02-13T00:39:24.136021+0000","last_undegraded":"2024-02-13T00:39:24.136021+0000","last_fullsized":"2024-02-13T00:39:24.136021+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:46.927068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:46.927068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":100,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040454+0000","last_change":"2024-02-13T00:39:04.318990+0000","last_active":"2024-02-13T00:39:24.040454+0000","last_peered":"2024-02-13T00:39:24.040454+0000","last_clean":"2024-02-13T00:39:24.040454+0000","last_became_active":"2024-02-13T00:37:43.322450+0000","last_became_peered":"2024-02-13T00:37:43.322450+0000","last_unstale":"2024-02-13T00:39:24.040454+0000","last_undegraded":"2024-02-13T00:39:24.040454+0000","last_fullsized":"2024-02-13T00:39:24.040454+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_clean_scrub_stamp":"2024-02-13T00:39:04.318923+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138172+0000","last_change":"2024-02-13T00:39:20.451151+0000","last_active":"2024-02-13T00:39:24.138172+0000","last_peered":"2024-02-13T00:39:24.138172+0000","last_clean":"2024-02-13T00:39:24.138172+0000","last_became_active":"2024-02-13T00:37:45.245974+0000","last_became_peered":"2024-02-13T00:37:45.245974+0000","last_unstale":"2024-02-13T00:39:24.138172+0000","last_undegraded":"2024-02-13T00:39:24.138172+0000","last_fullsized":"2024-02-13T00:39:24.138172+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:20.451088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:20.451088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368220+0000","last_change":"2024-02-13T00:39:09.237388+0000","last_active":"2024-02-13T00:39:23.368220+0000","last_peered":"2024-02-13T00:39:23.368220+0000","last_clean":"2024-02-13T00:39:23.368220+0000","last_became_active":"2024-02-13T00:37:47.250291+0000","last_became_peered":"2024-02-13T00:37:47.250291+0000","last_unstale":"2024-02-13T00:39:23.368220+0000","last_undegraded":"2024-02-13T00:39:23.368220+0000","last_fullsized":"2024-02-13T00:39:23.368220+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:09.237312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:09.237312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319820+0000","last_change":"2024-02-13T00:39:21.329248+0000","last_active":"2024-02-13T00:39:23.319820+0000","last_peered":"2024-02-13T00:39:23.319820+0000","last_clean":"2024-02-13T00:39:23.319820+0000","last_became_active":"2024-02-13T00:37:41.214202+0000","last_became_peered":"2024-02-13T00:37:41.214202+0000","last_unstale":"2024-02-13T00:39:23.319820+0000","last_undegraded":"2024-02-13T00:39:23.319820+0000","last_fullsized":"2024-02-13T00:39:23.319820+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.329173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.329173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":102,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368161+0000","last_change":"2024-02-13T00:38:58.238608+0000","last_active":"2024-02-13T00:39:23.368161+0000","last_peered":"2024-02-13T00:39:23.368161+0000","last_clean":"2024-02-13T00:39:23.368161+0000","last_became_active":"2024-02-13T00:37:43.324275+0000","last_became_peered":"2024-02-13T00:37:43.324275+0000","last_unstale":"2024-02-13T00:39:23.368161+0000","last_undegraded":"2024-02-13T00:39:23.368161+0000","last_fullsized":"2024-02-13T00:39:23.368161+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:38:58.238526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.238526+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.367995+0000","last_change":"2024-02-13T00:38:52.388152+0000","last_active":"2024-02-13T00:39:23.367995+0000","last_peered":"2024-02-13T00:39:23.367995+0000","last_clean":"2024-02-13T00:39:23.367995+0000","last_became_active":"2024-02-13T00:37:45.244203+0000","last_became_peered":"2024-02-13T00:37:45.244203+0000","last_unstale":"2024-02-13T00:39:23.367995+0000","last_undegraded":"2024-02-13T00:39:23.367995+0000","last_fullsized":"2024-02-13T00:39:23.367995+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.388093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.388093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135371+0000","last_change":"2024-02-13T00:38:56.857630+0000","last_active":"2024-02-13T00:39:24.135371+0000","last_peered":"2024-02-13T00:39:24.135371+0000","last_clean":"2024-02-13T00:39:24.135371+0000","last_became_active":"2024-02-13T00:37:47.553554+0000","last_became_peered":"2024-02-13T00:37:47.553554+0000","last_unstale":"2024-02-13T00:39:24.135371+0000","last_undegraded":"2024-02-13T00:39:24.135371+0000","last_fullsized":"2024-02-13T00:39:24.135371+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:56.857534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.857534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138676+0000","last_change":"2024-02-13T00:39:14.367203+0000","last_active":"2024-02-13T00:39:24.138676+0000","last_peered":"2024-02-13T00:39:24.138676+0000","last_clean":"2024-02-13T00:39:24.138676+0000","last_became_active":"2024-02-13T00:37:41.218313+0000","last_became_peered":"2024-02-13T00:37:41.218313+0000","last_unstale":"2024-02-13T00:39:24.138676+0000","last_undegraded":"2024-02-13T00:39:24.138676+0000","last_fullsized":"2024-02-13T00:39:24.138676+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:14.367115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:14.367115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368081+0000","last_change":"2024-02-13T00:37:43.222098+0000","last_active":"2024-02-13T00:39:23.368081+0000","last_peered":"2024-02-13T00:39:23.368081+0000","last_clean":"2024-02-13T00:39:23.368081+0000","last_became_active":"2024-02-13T00:37:43.221753+0000","last_became_peered":"2024-02-13T00:37:43.221753+0000","last_unstale":"2024-02-13T00:39:23.368081+0000","last_undegraded":"2024-02-13T00:39:23.368081+0000","last_fullsized":"2024-02-13T00:39:23.368081+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:37:42.205050+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"64'8","reported_seq":75,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:49.236358+0000","last_change":"2024-02-13T00:39:16.304400+0000","last_active":"2024-02-13T00:39:49.236358+0000","last_peered":"2024-02-13T00:39:49.236358+0000","last_clean":"2024-02-13T00:39:49.236358+0000","last_became_active":"2024-02-13T00:37:45.245528+0000","last_became_peered":"2024-02-13T00:37:45.245528+0000","last_unstale":"2024-02-13T00:39:49.236358+0000","last_undegraded":"2024-02-13T00:39:49.236358+0000","last_fullsized":"2024-02-13T00:39:49.236358+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:16.304316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.304316+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368370+0000","last_change":"2024-02-13T00:38:59.188989+0000","last_active":"2024-02-13T00:39:23.368370+0000","last_peered":"2024-02-13T00:39:23.368370+0000","last_clean":"2024-02-13T00:39:23.368370+0000","last_became_active":"2024-02-13T00:37:47.552591+0000","last_became_peered":"2024-02-13T00:37:47.552591+0000","last_unstale":"2024-02-13T00:39:23.368370+0000","last_undegraded":"2024-02-13T00:39:23.368370+0000","last_fullsized":"2024-02-13T00:39:23.368370+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:59.188913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:59.188913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.314084+0000","last_change":"2024-02-13T00:38:57.494754+0000","last_active":"2024-02-13T00:39:23.314084+0000","last_peered":"2024-02-13T00:39:23.314084+0000","last_clean":"2024-02-13T00:39:23.314084+0000","last_became_active":"2024-02-13T00:37:45.245897+0000","last_became_peered":"2024-02-13T00:37:45.245897+0000","last_unstale":"2024-02-13T00:39:23.314084+0000","last_undegraded":"2024-02-13T00:39:23.314084+0000","last_fullsized":"2024-02-13T00:39:23.314084+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.494653+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:29.159204+0000","last_change":"2024-02-13T00:39:29.159172+0000","last_active":"2024-02-13T00:39:29.159204+0000","last_peered":"2024-02-13T00:39:29.159204+0000","last_clean":"2024-02-13T00:39:29.159204+0000","last_became_active":"2024-02-13T00:37:41.221627+0000","last_became_peered":"2024-02-13T00:37:41.221627+0000","last_unstale":"2024-02-13T00:39:29.159204+0000","last_undegraded":"2024-02-13T00:39:29.159204+0000","last_fullsized":"2024-02-13T00:39:29.159204+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.159080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.159080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:29.275226+0000","last_change":"2024-02-13T00:39:29.275198+0000","last_active":"2024-02-13T00:39:29.275226+0000","last_peered":"2024-02-13T00:39:29.275226+0000","last_clean":"2024-02-13T00:39:29.275226+0000","last_became_active":"2024-02-13T00:37:43.224302+0000","last_became_peered":"2024-02-13T00:37:43.224302+0000","last_unstale":"2024-02-13T00:39:29.275226+0000","last_undegraded":"2024-02-13T00:39:29.275226+0000","last_fullsized":"2024-02-13T00:39:29.275226+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:29.275134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.275134+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313776+0000","last_change":"2024-02-13T00:38:54.599127+0000","last_active":"2024-02-13T00:39:23.313776+0000","last_peered":"2024-02-13T00:39:23.313776+0000","last_clean":"2024-02-13T00:39:23.313776+0000","last_became_active":"2024-02-13T00:37:47.260477+0000","last_became_peered":"2024-02-13T00:37:47.260477+0000","last_unstale":"2024-02-13T00:39:23.313776+0000","last_undegraded":"2024-02-13T00:39:23.313776+0000","last_fullsized":"2024-02-13T00:39:23.313776+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:54.599057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:54.599057+0000","last_clean_scrub_stamp":"2024-02-13T00:38:54.599057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"64'8","reported_seq":68,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:49.236333+0000","last_change":"2024-02-13T00:37:45.245491+0000","last_active":"2024-02-13T00:39:49.236333+0000","last_peered":"2024-02-13T00:39:49.236333+0000","last_clean":"2024-02-13T00:39:49.236333+0000","last_became_active":"2024-02-13T00:37:45.245183+0000","last_became_peered":"2024-02-13T00:37:45.245183+0000","last_unstale":"2024-02-13T00:39:49.236333+0000","last_undegraded":"2024-02-13T00:39:49.236333+0000","last_fullsized":"2024-02-13T00:39:49.236333+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:37:44.215312+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":54,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:52.206001+0000","last_change":"2024-02-13T00:39:52.206001+0000","last_active":"2024-02-13T00:39:52.206001+0000","last_peered":"2024-02-13T00:39:52.206001+0000","last_clean":"2024-02-13T00:39:52.206001+0000","last_became_active":"2024-02-13T00:37:41.217979+0000","last_became_peered":"2024-02-13T00:37:41.217979+0000","last_unstale":"2024-02-13T00:39:52.206001+0000","last_undegraded":"2024-02-13T00:39:52.206001+0000","last_fullsized":"2024-02-13T00:39:52.206001+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:39:52.205926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:52.205926+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.370534+0000","last_change":"2024-02-13T00:39:23.370512+0000","last_active":"2024-02-13T00:39:23.370534+0000","last_peered":"2024-02-13T00:39:23.370534+0000","last_clean":"2024-02-13T00:39:23.370534+0000","last_became_active":"2024-02-13T00:37:43.232161+0000","last_became_peered":"2024-02-13T00:37:43.232161+0000","last_unstale":"2024-02-13T00:39:23.370534+0000","last_undegraded":"2024-02-13T00:39:23.370534+0000","last_fullsized":"2024-02-13T00:39:23.370534+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:39:23.370447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.370447+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368383+0000","last_change":"2024-02-13T00:37:47.259820+0000","last_active":"2024-02-13T00:39:23.368383+0000","last_peered":"2024-02-13T00:39:23.368383+0000","last_clean":"2024-02-13T00:39:23.368383+0000","last_became_active":"2024-02-13T00:37:47.259517+0000","last_became_peered":"2024-02-13T00:37:47.259517+0000","last_unstale":"2024-02-13T00:39:23.368383+0000","last_undegraded":"2024-02-13T00:39:23.368383+0000","last_fullsized":"2024-02-13T00:39:23.368383+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:37:46.228444+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"64'8","reported_seq":74,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:49.236287+0000","last_change":"2024-02-13T00:39:14.609925+0000","last_active":"2024-02-13T00:39:49.236287+0000","last_peered":"2024-02-13T00:39:49.236287+0000","last_clean":"2024-02-13T00:39:49.236287+0000","last_became_active":"2024-02-13T00:37:45.249684+0000","last_became_peered":"2024-02-13T00:37:45.249684+0000","last_unstale":"2024-02-13T00:39:49.236287+0000","last_undegraded":"2024-02-13T00:39:49.236287+0000","last_fullsized":"2024-02-13T00:39:49.236287+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:14.609850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:14.609850+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319382+0000","last_change":"2024-02-13T00:39:17.283996+0000","last_active":"2024-02-13T00:39:23.319382+0000","last_peered":"2024-02-13T00:39:23.319382+0000","last_clean":"2024-02-13T00:39:23.319382+0000","last_became_active":"2024-02-13T00:37:41.218346+0000","last_became_peered":"2024-02-13T00:37:41.218346+0000","last_unstale":"2024-02-13T00:39:23.319382+0000","last_undegraded":"2024-02-13T00:39:23.319382+0000","last_fullsized":"2024-02-13T00:39:23.319382+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.283902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.283902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:28.305261+0000","last_change":"2024-02-13T00:39:28.305239+0000","last_active":"2024-02-13T00:39:28.305261+0000","last_peered":"2024-02-13T00:39:28.305261+0000","last_clean":"2024-02-13T00:39:28.305261+0000","last_became_active":"2024-02-13T00:37:43.227980+0000","last_became_peered":"2024-02-13T00:37:43.227980+0000","last_unstale":"2024-02-13T00:39:28.305261+0000","last_undegraded":"2024-02-13T00:39:28.305261+0000","last_fullsized":"2024-02-13T00:39:28.305261+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.305179+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319487+0000","last_change":"2024-02-13T00:39:07.563502+0000","last_active":"2024-02-13T00:39:23.319487+0000","last_peered":"2024-02-13T00:39:23.319487+0000","last_clean":"2024-02-13T00:39:23.319487+0000","last_became_active":"2024-02-13T00:37:47.256807+0000","last_became_peered":"2024-02-13T00:37:47.256807+0000","last_unstale":"2024-02-13T00:39:23.319487+0000","last_undegraded":"2024-02-13T00:39:23.319487+0000","last_fullsized":"2024-02-13T00:39:23.319487+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.563440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.563440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:35.269627+0000","last_change":"2024-02-13T00:39:35.269607+0000","last_active":"2024-02-13T00:39:35.269627+0000","last_peered":"2024-02-13T00:39:35.269627+0000","last_clean":"2024-02-13T00:39:35.269627+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:39:35.269627+0000","last_undegraded":"2024-02-13T00:39:35.269627+0000","last_fullsized":"2024-02-13T00:39:35.269627+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:35.269538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:35.269538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138738+0000","last_change":"2024-02-13T00:39:24.138710+0000","last_active":"2024-02-13T00:39:24.138738+0000","last_peered":"2024-02-13T00:39:24.138738+0000","last_clean":"2024-02-13T00:39:24.138738+0000","last_became_active":"2024-02-13T00:37:41.220874+0000","last_became_peered":"2024-02-13T00:37:41.220874+0000","last_unstale":"2024-02-13T00:39:24.138738+0000","last_undegraded":"2024-02-13T00:39:24.138738+0000","last_fullsized":"2024-02-13T00:39:24.138738+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:39:24.138629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.138629+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.368495+0000","last_change":"2024-02-13T00:39:02.188162+0000","last_active":"2024-02-13T00:39:23.368495+0000","last_peered":"2024-02-13T00:39:23.368495+0000","last_clean":"2024-02-13T00:39:23.368495+0000","last_became_active":"2024-02-13T00:37:43.223574+0000","last_became_peered":"2024-02-13T00:37:43.223574+0000","last_unstale":"2024-02-13T00:39:23.368495+0000","last_undegraded":"2024-02-13T00:39:23.368495+0000","last_fullsized":"2024-02-13T00:39:23.368495+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:39:02.188096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.188096+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.135466+0000","last_change":"2024-02-13T00:39:16.086278+0000","last_active":"2024-02-13T00:39:24.135466+0000","last_peered":"2024-02-13T00:39:24.135466+0000","last_clean":"2024-02-13T00:39:24.135466+0000","last_became_active":"2024-02-13T00:37:47.254705+0000","last_became_peered":"2024-02-13T00:37:47.254705+0000","last_unstale":"2024-02-13T00:39:24.135466+0000","last_undegraded":"2024-02-13T00:39:24.135466+0000","last_fullsized":"2024-02-13T00:39:24.135466+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.086207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.086207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319190+0000","last_change":"2024-02-13T00:39:17.631975+0000","last_active":"2024-02-13T00:39:23.319190+0000","last_peered":"2024-02-13T00:39:23.319190+0000","last_clean":"2024-02-13T00:39:23.319190+0000","last_became_active":"2024-02-13T00:37:45.239867+0000","last_became_peered":"2024-02-13T00:37:45.239867+0000","last_unstale":"2024-02-13T00:39:23.319190+0000","last_undegraded":"2024-02-13T00:39:23.319190+0000","last_fullsized":"2024-02-13T00:39:23.319190+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.631877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.320609+0000","last_change":"2024-02-13T00:39:18.672595+0000","last_active":"2024-02-13T00:39:23.320609+0000","last_peered":"2024-02-13T00:39:23.320609+0000","last_clean":"2024-02-13T00:39:23.320609+0000","last_became_active":"2024-02-13T00:37:41.219457+0000","last_became_peered":"2024-02-13T00:37:41.219457+0000","last_unstale":"2024-02-13T00:39:23.320609+0000","last_undegraded":"2024-02-13T00:39:23.320609+0000","last_fullsized":"2024-02-13T00:39:23.320609+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.672516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.672516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'25","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.138107+0000","last_change":"2024-02-13T00:38:56.371071+0000","last_active":"2024-02-13T00:39:24.138107+0000","last_peered":"2024-02-13T00:39:24.138107+0000","last_clean":"2024-02-13T00:39:24.138107+0000","last_became_active":"2024-02-13T00:37:43.322957+0000","last_became_peered":"2024-02-13T00:37:43.322957+0000","last_unstale":"2024-02-13T00:39:24.138107+0000","last_undegraded":"2024-02-13T00:39:24.138107+0000","last_fullsized":"2024-02-13T00:39:24.138107+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'25","last_scrub_stamp":"2024-02-13T00:38:56.370998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.370998+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.040202+0000","last_change":"2024-02-13T00:39:18.359681+0000","last_active":"2024-02-13T00:39:24.040202+0000","last_peered":"2024-02-13T00:39:24.040202+0000","last_clean":"2024-02-13T00:39:24.040202+0000","last_became_active":"2024-02-13T00:37:47.249556+0000","last_became_peered":"2024-02-13T00:37:47.249556+0000","last_unstale":"2024-02-13T00:39:24.040202+0000","last_undegraded":"2024-02-13T00:39:24.040202+0000","last_fullsized":"2024-02-13T00:39:24.040202+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.359604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.320337+0000","last_change":"2024-02-13T00:39:15.603878+0000","last_active":"2024-02-13T00:39:23.320337+0000","last_peered":"2024-02-13T00:39:23.320337+0000","last_clean":"2024-02-13T00:39:23.320337+0000","last_became_active":"2024-02-13T00:37:45.240923+0000","last_became_peered":"2024-02-13T00:37:45.240923+0000","last_unstale":"2024-02-13T00:39:23.320337+0000","last_undegraded":"2024-02-13T00:39:23.320337+0000","last_fullsized":"2024-02-13T00:39:23.320337+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.603808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.603808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319903+0000","last_change":"2024-02-13T00:39:16.262449+0000","last_active":"2024-02-13T00:39:23.319903+0000","last_peered":"2024-02-13T00:39:23.319903+0000","last_clean":"2024-02-13T00:39:23.319903+0000","last_became_active":"2024-02-13T00:37:41.212429+0000","last_became_peered":"2024-02-13T00:37:41.212429+0000","last_unstale":"2024-02-13T00:39:23.319903+0000","last_undegraded":"2024-02-13T00:39:23.319903+0000","last_fullsized":"2024-02-13T00:39:23.319903+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.262380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.262380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:24.376199+0000","last_change":"2024-02-13T00:39:24.376175+0000","last_active":"2024-02-13T00:39:24.376199+0000","last_peered":"2024-02-13T00:39:24.376199+0000","last_clean":"2024-02-13T00:39:24.376199+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:39:24.376199+0000","last_undegraded":"2024-02-13T00:39:24.376199+0000","last_fullsized":"2024-02-13T00:39:24.376199+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:39:24.376100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.376100+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313416+0000","last_change":"2024-02-13T00:39:17.713034+0000","last_active":"2024-02-13T00:39:23.313416+0000","last_peered":"2024-02-13T00:39:23.313416+0000","last_clean":"2024-02-13T00:39:23.313416+0000","last_became_active":"2024-02-13T00:37:47.552384+0000","last_became_peered":"2024-02-13T00:37:47.552384+0000","last_unstale":"2024-02-13T00:39:23.313416+0000","last_undegraded":"2024-02-13T00:39:23.313416+0000","last_fullsized":"2024-02-13T00:39:23.313416+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.712965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.712965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.319980+0000","last_change":"2024-02-13T00:39:02.545556+0000","last_active":"2024-02-13T00:39:23.319980+0000","last_peered":"2024-02-13T00:39:23.319980+0000","last_clean":"2024-02-13T00:39:23.319980+0000","last_became_active":"2024-02-13T00:37:41.219093+0000","last_became_peered":"2024-02-13T00:37:41.219093+0000","last_unstale":"2024-02-13T00:39:23.319980+0000","last_undegraded":"2024-02-13T00:39:23.319980+0000","last_fullsized":"2024-02-13T00:39:23.319980+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:02.545495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.545495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":104,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-13T00:39:23.313402+0000","last_change":"2024-02-13T00:38:59.548948+0000","last_active":"2024-02-13T00:39:23.313402+0000","last_peered":"2024-02-13T00:39:23.313402+0000","last_clean":"2024-02-13T00:39:23.313402+0000","last_became_active":"2024-02-13T00:37:43.322374+0000","last_became_peered":"2024-02-13T00:37:43.322374+0000","last_unstale":"2024-02-13T00:39:23.313402+0000","last_undegraded":"2024-02-13T00:39:23.313402+0000","last_fullsized":"2024-02-13T00:39:23.313402+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_clean_scrub_stamp":"2024-02-13T00:38:59.548872+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"64'8","reported_seq":73,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-13T00:39:49.236372+0000","last_change":"2024-02-13T00:39:15.080406+0000","last_active":"2024-02-13T00:39:49.236372+0000","last_peered":"2024-02-13T00:39:49.236372+0000","last_clean":"2024-02-13T00:39:49.236372+0000","last_became_active":"2024-02-13T00:37:45.247905+0000","last_became_peered":"2024-02-13T00:37:45.247905+0000","last_unstale":"2024-02-13T00:39:49.236372+0000","last_undegraded":"2024-02-13T00:39:49.236372+0000","last_fullsized":"2024-02-13T00:39:49.236372+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:15.080315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.080315+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":654,"ondisk_log_size":654,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":127,"num_read_kb":122,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266717,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297744,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447408,"statfs":{"total":95995035648,"available":95690145792,"internally_reserved":0,"allocated":663552,"data_stored":304098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.498,"15min":0.501},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.63800000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52300000000000002,"15min":0.52800000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.63,"5min":1.028,"15min":1.028},"last":0.63}]},{"osd":1,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.49099999999999999,"15min":0.49399999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.503,"5min":0.59899999999999998,"15min":0.61499999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64200000000000002,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.58299999999999996}]},{"osd":2,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.628,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61099999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.57399999999999995}]},{"osd":3,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.55800000000000005,"15min":0.56799999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.64100000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.65700000000000003,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.498}]},{"osd":4,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.46100000000000002,"15min":0.46999999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.65400000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.45600000000000002,"15min":0.46400000000000002},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67000000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.40699999999999997}]},{"osd":5,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.63600000000000001,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46800000000000003,"15min":0.47199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65800000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.436}]},{"osd":6,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.49199999999999999,"15min":0.497},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68899999999999995,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.45900000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528355,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":302575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.46899999999999997,"15min":0.46800000000000003},"min":{"1min":0.35899999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.437}]},{"osd":1,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.504,"15min":0.50600000000000001},"min":{"1min":0.41499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61099999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.67800000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.56000000000000005}]},{"osd":2,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.497,"15min":0.5},"min":{"1min":0.34899999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.64100000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.501},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.626,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.38}]},{"osd":3,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.53200000000000003,"15min":0.54300000000000004},"min":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68999999999999995,"5min":1.181,"15min":1.181},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.42499999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.52800000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.39500000000000002},"min":{"1min":0.29999999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72199999999999998,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41699999999999998},"min":{"1min":0.31,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66700000000000004,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.621}]},{"osd":5,"last update":"Tue Feb 13 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.312,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61199999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.44700000000000001,"15min":0.442},"min":{"1min":0.34000000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.76300000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.54100000000000004}]},{"osd":7,"last update":"Tue Feb 13 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.499,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67700000000000005,"5min":1.173,"15min":1.173},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79000000000000004,"5min":1.262,"15min":1.262},"last":0.58699999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297772,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447380,"statfs":{"total":95995035648,"available":95690117120,"internally_reserved":0,"allocated":692224,"data_stored":303825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.32000000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.313},"min":{"1min":0.221,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.49099999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.33200000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.372},"min":{"1min":0.255,"5min":0.215,"15min":0.215},"max":{"1min":0.57999999999999996,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.27300000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.69499999999999995,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.28799999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65000000000000002,"5min":0.748,"15min":0.748},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.45000000000000001,"15min":0.46500000000000002},"min":{"1min":0.29299999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.53900000000000003,"5min":0.745,"15min":0.745},"last":0.35199999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.81299999999999994,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82499999999999996,"5min":1.02,"15min":1.02},"last":0.52400000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56799999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78700000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70699999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.42499999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78700000000000003,"5min":1.054,"15min":1.054},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86799999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.45200000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.78500000000000003,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.62,"15min":0.623},"min":{"1min":0.41599999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81899999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.41799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297740,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447412,"statfs":{"total":95995035648,"available":95690149888,"internally_reserved":0,"allocated":659456,"data_stored":302879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:39:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34599999999999997,"15min":0.32800000000000001},"min":{"1min":0.254,"5min":0.192,"15min":0.192},"max":{"1min":0.52000000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.376,"15min":0.34399999999999997},"min":{"1min":0.23499999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.58899999999999997,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.27800000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40400000000000003,"15min":0.39800000000000002},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.65200000000000002,"5min":1.206,"15min":1.206},"last":0.245},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.746,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.29199999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.64900000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45600000000000002,"15min":0.46999999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.311}]},{"osd":4,"last update":"Tue Feb 13 00:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.75600000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72099999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.60699999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:39:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55800000000000005,"15min":0.57399999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.64400000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.41599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54200000000000004,"15min":0.53700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.54500000000000004}]},{"osd":7,"last update":"Tue Feb 13 00:39:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.52200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411383,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297732,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447420,"statfs":{"total":95995035648,"available":95690158080,"internally_reserved":0,"allocated":651264,"data_stored":303146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.36399999999999999,"15min":0.35699999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.49199999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36299999999999999,"15min":0.34599999999999997},"min":{"1min":0.22900000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.66800000000000004,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.317}]},{"osd":1,"last update":"Tue Feb 13 00:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.35699999999999998,"15min":0.34899999999999998},"min":{"1min":0.22900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.49099999999999999,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38400000000000001,"15min":0.374},"min":{"1min":0.251,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.51600000000000001,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.28499999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:39:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.28000000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.52800000000000002,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59099999999999997,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.34799999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73699999999999999,"5min":0.873,"15min":0.873},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.58799999999999997,"15min":0.59199999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71699999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.65200000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:39:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54600000000000004,"15min":0.56000000000000005},"min":{"1min":0.38900000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65000000000000002,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.57499999999999996,"15min":0.59299999999999997},"min":{"1min":0.376,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.64400000000000002,"5min":0.872,"15min":0.872},"last":0.67700000000000005}]},{"osd":6,"last update":"Tue Feb 13 00:39:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58099999999999996,"15min":0.58399999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79400000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.51900000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58299999999999996,"15min":0.59099999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.748,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70999999999999996,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.60899999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182449,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297756,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447396,"statfs":{"total":95995035648,"available":95690133504,"internally_reserved":0,"allocated":675840,"data_stored":303408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.36099999999999999,"15min":0.376},"min":{"1min":0.221,"5min":0.191,"15min":0.191},"max":{"1min":0.55300000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.39300000000000002,"15min":0.40799999999999997},"min":{"1min":0.27800000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46200000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.442}]},{"osd":1,"last update":"Tue Feb 13 00:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38500000000000001,"15min":0.39800000000000002},"min":{"1min":0.28100000000000003,"5min":0.217,"15min":0.217},"max":{"1min":0.5,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.39100000000000001,"15min":0.39900000000000002},"min":{"1min":0.24399999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.56200000000000006,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.36899999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.41399999999999998,"15min":0.42099999999999999},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58899999999999997,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39200000000000002,"15min":0.40300000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.52900000000000003,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.434}]},{"osd":4,"last update":"Tue Feb 13 00:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53400000000000003,"15min":0.54000000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.64700000000000002,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.53600000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.75,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.59899999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:38:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.50600000000000001,"15min":0.499},"min":{"1min":0.38200000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.52100000000000002,"15min":0.51600000000000001},"min":{"1min":0.41199999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.51700000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.54000000000000004}]},{"osd":7,"last update":"Tue Feb 13 00:39:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.54400000000000004},"min":{"1min":0.372,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77800000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56899999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986221,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":303919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67900000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.66500000000000004,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53900000000000003}]},{"osd":1,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.52100000000000002,"15min":0.53000000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.61499999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.64200000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69899999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52000000000000002,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.62,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.624}]},{"osd":3,"last update":"Tue Feb 13 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.498,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.38500000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.61299999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:39:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.37,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.58299999999999996,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.39300000000000002,"15min":0.36799999999999999},"min":{"1min":0.315,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.52400000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.32900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.497},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.41299999999999998,"15min":0.40400000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.59799999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46100000000000002,"15min":0.45900000000000002},"min":{"1min":0.34200000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.33000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.63800000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.64900000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724584,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":647168,"data_stored":303699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49199999999999999,"15min":0.48599999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.84399999999999997,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.51200000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72199999999999998,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.51100000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.502,"15min":0.497},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68700000000000006,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52400000000000002,"15min":0.502},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":4.0839999999999996,"15min":4.0839999999999996},"last":0.54300000000000004}]},{"osd":2,"last update":"Tue Feb 13 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51100000000000001,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.745,"5min":1.099,"15min":1.099},"last":0.443},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51000000000000001,"15min":0.505},"min":{"1min":0.34100000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.90000000000000002,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.56999999999999995}]},{"osd":3,"last update":"Tue Feb 13 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85899999999999999,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51300000000000001,"15min":0.503},"min":{"1min":0.38200000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78100000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.60699999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.41799999999999998,"15min":0.36599999999999999},"min":{"1min":0.309,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66400000000000003,"5min":4.1210000000000004,"15min":4.1210000000000004},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.41499999999999998},"min":{"1min":0.379,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.74399999999999999,"5min":5.6950000000000003,"15min":5.6950000000000003},"last":0.59099999999999997}]},{"osd":6,"last update":"Tue Feb 13 00:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.441,"15min":0.42699999999999999},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.873,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.46400000000000002,"15min":0.44700000000000001},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.327,"5min":1.377,"15min":1.377},"last":0.42999999999999999}]},{"osd":7,"last update":"Tue Feb 13 00:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.755,"15min":0.80300000000000005},"min":{"1min":0.33200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.58499999999999996,"5min":7.5519999999999996,"15min":7.5519999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.55800000000000005,"15min":0.57399999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.64800000000000002,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.627}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:39:58.538 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-13T00:39:58.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump 2024-02-13T00:39:58.808 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:39:58.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:39:57.705819+0000 mgr.y (mgr.14544) 95 : audit [DBG] from='client.24662 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:39:58.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:57.975703+0000 osd.6 (osd.6) 29 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:39:58.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:57.978588+0000 osd.6 (osd.6) 30 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:39:58.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:39:57.705819+0000 mgr.y (mgr.14544) 95 : audit [DBG] from='client.24662 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:39:58.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:57.975703+0000 osd.6 (osd.6) 29 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:39:58.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:57.978588+0000 osd.6 (osd.6) 30 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:39:59.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:39:57.705819+0000 mgr.y (mgr.14544) 95 : audit [DBG] from='client.24662 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:39:59.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:57.975703+0000 osd.6 (osd.6) 29 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:39:59.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:57.978588+0000 osd.6 (osd.6) 30 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:39:59.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:39:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:39:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:39:59.944 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:39:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:39:58.186627+0000 mgr.y (mgr.14544) 96 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:39:59.944 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:39:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:39:58.186627+0000 mgr.y (mgr.14544) 96 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:40:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:39:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:39:58.186627+0000 mgr.y (mgr.14544) 96 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:40:00.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.425+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3164606466 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6f40fcb30 msgr2=0x7fe6f40fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:00.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.425+0000 7fe6fa1b3700 1 --2- 172.21.15.162:0/3164606466 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6f40fcb30 0x7fe6f40fcf10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fe6e8004870 tx=0x7fe6e8009cb0).stop 2024-02-13T00:40:00.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.425+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3164606466 shutdown_connections 2024-02-13T00:40:00.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.425+0000 7fe6fa1b3700 1 --2- 172.21.15.162:0/3164606466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6f40f73a0 0x7fe6f41039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:00.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.425+0000 7fe6fa1b3700 1 --2- 172.21.15.162:0/3164606466 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6f40fcb30 0x7fe6f40fcf10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:00.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.425+0000 7fe6fa1b3700 1 --2- 172.21.15.162:0/3164606466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6f40fd490 0x7fe6f40f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:00.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.425+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3164606466 >> 172.21.15.162:0/3164606466 conn(0x7fe6f406eb90 msgr2=0x7fe6f40f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.425+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3164606466 shutdown_connections 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.425+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3164606466 wait complete. 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.425+0000 7fe6fa1b3700 1 librados: starting msgr at 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.425+0000 7fe6fa1b3700 1 librados: starting objecter 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6fa1b3700 1 Processor -- start 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6fa1b3700 1 -- start start 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6fa1b3700 1 librados: setting wanted keys 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6fa1b3700 1 librados: calling monclient init 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6fa1b3700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6f40f73a0 0x7fe6f41091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6fa1b3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6f40fcb30 0x7fe6f41096d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6fa1b3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6f40fd490 0x7fe6f4109cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6fa1b3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f41143c0 con 0x7fe6f40fcb30 2024-02-13T00:40:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6fa1b3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f4114530 con 0x7fe6f40fd490 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6fa1b3700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f41146f0 con 0x7fe6f40f73a0 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f37fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6f40f73a0 0x7fe6f41091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f37fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6f40f73a0 0x7fe6f41091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:41960/0 (socket says 172.21.15.162:41960) 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f37fe700 1 -- 172.21.15.162:0/3279354026 learned_addr learned my addr 172.21.15.162:0/3279354026 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6f40fcb30 0x7fe6f41096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f3fff700 1 --2- 172.21.15.162:0/3279354026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6f40fd490 0x7fe6f4109cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f37fe700 1 -- 172.21.15.162:0/3279354026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6f40fd490 msgr2=0x7fe6f4109cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f37fe700 1 --2- 172.21.15.162:0/3279354026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6f40fd490 0x7fe6f4109cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f37fe700 1 -- 172.21.15.162:0/3279354026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6f40fcb30 msgr2=0x7fe6f41096d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f37fe700 1 --2- 172.21.15.162:0/3279354026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6f40fcb30 0x7fe6f41096d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f37fe700 1 -- 172.21.15.162:0/3279354026 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e4005040 con 0x7fe6f40f73a0 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f2ffd700 1 --2- 172.21.15.162:0/3279354026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6f40fcb30 0x7fe6f41096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:40:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f3fff700 1 --2- 172.21.15.162:0/3279354026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6f40fd490 0x7fe6f4109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f37fe700 1 --2- 172.21.15.162:0/3279354026 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6f40f73a0 0x7fe6f41091e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe6e8009eb0 tx=0x7fe6e800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f0ff9700 10 client.?.objecter ms_handle_connect 0x7fe6f40f73a0 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.426+0000 7fe6f0ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.427+0000 7fe6f0ff9700 1 -- 172.21.15.162:0/3279354026 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e800e070 con 0x7fe6f40f73a0 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.427+0000 7fe6f0ff9700 1 -- 172.21.15.162:0/3279354026 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6e8013da0 con 0x7fe6f40f73a0 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.427+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3279354026 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e8005040 con 0x7fe6f40f73a0 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.427+0000 7fe6f0ff9700 1 -- 172.21.15.162:0/3279354026 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8019070 con 0x7fe6f40f73a0 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.427+0000 7fe6fa1b3700 10 client.24668.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.427+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3279354026 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f4114cc0 con 0x7fe6f40f73a0 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.427+0000 7fe6fa1b3700 1 librados: init done 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.428+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3279354026 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f4105fe0 con 0x7fe6f40f73a0 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.428+0000 7fe6f0ff9700 1 -- 172.21.15.162:0/3279354026 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe6e80090d0 con 0x7fe6f40f73a0 2024-02-13T00:40:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.428+0000 7fe6f0ff9700 1 --2- 172.21.15.162:0/3279354026 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe6dc062c90 0x7fe6dc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:00.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.428+0000 7fe6f0ff9700 1 -- 172.21.15.162:0/3279354026 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(86..86 src has 30..86) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fe6e807d580 con 0x7fe6f40f73a0 2024-02-13T00:40:00.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.428+0000 7fe6f0ff9700 10 client.24668.objecter ms_dispatch 0x7fe6f4001550 osd_map(86..86 src has 30..86) v4 2024-02-13T00:40:00.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.428+0000 7fe6f0ff9700 3 client.24668.objecter handle_osd_map got epochs [86,86] > 0 2024-02-13T00:40:00.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.428+0000 7fe6f0ff9700 3 client.24668.objecter handle_osd_map decoding full epoch 86 2024-02-13T00:40:00.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.428+0000 7fe6f2ffd700 1 --2- 172.21.15.162:0/3279354026 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe6dc062c90 0x7fe6dc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:00.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.428+0000 7fe6f0ff9700 20 client.24668.objecter dump_active .. 0 homeless 2024-02-13T00:40:00.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.429+0000 7fe6f0ff9700 10 client.24668.objecter ms_handle_connect 0x7fe6dc062c90 2024-02-13T00:40:00.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.429+0000 7fe6f2ffd700 1 --2- 172.21.15.162:0/3279354026 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe6dc062c90 0x7fe6dc065150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe6f40617b0 tx=0x7fe6e400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:00.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.433+0000 7fe6f0ff9700 1 -- 172.21.15.162:0/3279354026 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe6e8051260 con 0x7fe6f40f73a0 2024-02-13T00:40:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.597+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3279354026 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fe6f405fc10 con 0x7fe6f40f73a0 2024-02-13T00:40:00.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.598+0000 7fe6f0ff9700 1 -- 172.21.15.162:0/3279354026 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v86) v1 ==== 56+0+4033 (secure 0 0 0) 0x7fe6e8023070 con 0x7fe6f40f73a0 2024-02-13T00:40:00.602 INFO:teuthology.orchestra.run.smithi162.stdout:epoch 86 2024-02-13T00:40:00.602 INFO:teuthology.orchestra.run.smithi162.stdout:fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:40:00.602 INFO:teuthology.orchestra.run.smithi162.stdout:created 2024-02-13T00:31:51.797017+0000 2024-02-13T00:40:00.602 INFO:teuthology.orchestra.run.smithi162.stdout:modified 2024-02-13T00:39:44.254452+0000 2024-02-13T00:40:00.602 INFO:teuthology.orchestra.run.smithi162.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-13T00:40:00.602 INFO:teuthology.orchestra.run.smithi162.stdout:crush_version 18 2024-02-13T00:40:00.602 INFO:teuthology.orchestra.run.smithi162.stdout:full_ratio 0.95 2024-02-13T00:40:00.602 INFO:teuthology.orchestra.run.smithi162.stdout:backfillfull_ratio 0.9 2024-02-13T00:40:00.602 INFO:teuthology.orchestra.run.smithi162.stdout:nearfull_ratio 0.85 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:require_min_compat_client luminous 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:min_compat_client luminous 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:require_osd_release pacific 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:stretch_mode_enabled false 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 58 flags hashpspool stripe_width 0 application rgw 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:pool 3 'iscsi' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 59 flags hashpspool stripe_width 0 application rbd 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 60 flags hashpspool stripe_width 0 application rgw 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 application rgw 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 64 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:max_osd 8 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) v2:172.21.15.162:6801/2103743338 v2:172.21.15.162:6802/2103743338 exists,up 9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1 up in weight 1 up_from 13 up_thru 62 down_at 0 last_clean_interval [0,0) v2:172.21.15.162:6805/150599780 v2:172.21.15.162:6806/150599780 exists,up 39e9d2ef-9aa4-45bb-8a47-b0e0ec222886 2024-02-13T00:40:00.603 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) v2:172.21.15.162:6809/3689719376 v2:172.21.15.162:6810/3689719376 exists,up 86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3 up in weight 1 up_from 25 up_thru 62 down_at 0 last_clean_interval [0,0) v2:172.21.15.162:6813/604971011 v2:172.21.15.162:6814/604971011 exists,up 2e6b9f0e-7a18-4cf8-9331-1a973fadc001 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4 up in weight 1 up_from 31 up_thru 81 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6800/3344263388 v2:172.21.15.181:6801/3344263388 exists,up c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5 up in weight 1 up_from 39 up_thru 62 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6804/2212296748 v2:172.21.15.181:6805/2212296748 exists,up 6aa2bd68-7f4b-4973-ac49-7fbc57c1a723 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6 up in weight 1 up_from 45 up_thru 60 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6808/1506640341 v2:172.21.15.181:6809/1506640341 exists,up 791ba814-5284-432e-addc-d6b7f5f3d474 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7 up in weight 1 up_from 53 up_thru 62 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6812/1964192772 v2:172.21.15.181:6813/1964192772 exists,up 1f369c65-e663-4eb2-87a5-f2900f15b35b 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:pg_upmap_items 2.0 [7,2] 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:pg_upmap_items 4.2 [1,2] 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:pg_upmap_items 4.6 [1,5] 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:pg_upmap_items 4.13 [1,2] 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:blocklist 172.21.15.162:6800/1909610488 expires 2024-02-14T00:38:04.142895+0000 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:blocklist 172.21.15.162:0/2370739667 expires 2024-02-14T00:38:04.142895+0000 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:blocklist 172.21.15.162:0/1502590187 expires 2024-02-14T00:32:47.469889+0000 2024-02-13T00:40:00.604 INFO:teuthology.orchestra.run.smithi162.stdout:blocklist 172.21.15.162:6800/4121673848 expires 2024-02-14T00:32:47.469889+0000 2024-02-13T00:40:00.605 INFO:teuthology.orchestra.run.smithi162.stdout:blocklist 172.21.15.162:0/2412382714 expires 2024-02-14T00:38:04.142895+0000 2024-02-13T00:40:00.605 INFO:teuthology.orchestra.run.smithi162.stdout:blocklist 172.21.15.162:0/3041562701 expires 2024-02-14T00:32:47.469889+0000 2024-02-13T00:40:00.605 INFO:teuthology.orchestra.run.smithi162.stdout:blocklist 172.21.15.162:6800/2775563314 expires 2024-02-14T00:32:24.380625+0000 2024-02-13T00:40:00.605 INFO:teuthology.orchestra.run.smithi162.stdout:blocklist 172.21.15.162:0/3135632190 expires 2024-02-14T00:32:24.380625+0000 2024-02-13T00:40:00.605 INFO:teuthology.orchestra.run.smithi162.stdout:blocklist 172.21.15.162:0/197307636 expires 2024-02-14T00:38:04.142895+0000 2024-02-13T00:40:00.605 INFO:teuthology.orchestra.run.smithi162.stdout:blocklist 172.21.15.162:0/2653247729 expires 2024-02-14T00:32:24.380625+0000 2024-02-13T00:40:00.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.599+0000 7fe6fa1b3700 10 librados: watch_flush enter 2024-02-13T00:40:00.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.599+0000 7fe6fa1b3700 10 librados: watch_flush exit 2024-02-13T00:40:00.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.599+0000 7fe6fa1b3700 20 client.24668.objecter shutdown clearing up homeless session... 2024-02-13T00:40:00.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.599+0000 7fe6fa1b3700 10 client.24668.objecter successfully canceled tick 2024-02-13T00:40:00.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.599+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3279354026 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe6dc062c90 msgr2=0x7fe6dc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:00.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.599+0000 7fe6fa1b3700 1 --2- 172.21.15.162:0/3279354026 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe6dc062c90 0x7fe6dc065150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe6f40617b0 tx=0x7fe6e400b040).stop 2024-02-13T00:40:00.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.600+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3279354026 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6f40f73a0 msgr2=0x7fe6f41091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:00.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.600+0000 7fe6fa1b3700 1 --2- 172.21.15.162:0/3279354026 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6f40f73a0 0x7fe6f41091e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe6e8009eb0 tx=0x7fe6e800b040).stop 2024-02-13T00:40:00.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.600+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3279354026 shutdown_connections 2024-02-13T00:40:00.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.600+0000 7fe6fa1b3700 1 --2- 172.21.15.162:0/3279354026 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe6dc062c90 0x7fe6dc065150 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:00.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.600+0000 7fe6fa1b3700 1 --2- 172.21.15.162:0/3279354026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6f40fd490 0x7fe6f4109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:00.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.600+0000 7fe6fa1b3700 1 --2- 172.21.15.162:0/3279354026 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6f40f73a0 0x7fe6f41091e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:00.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.600+0000 7fe6fa1b3700 1 --2- 172.21.15.162:0/3279354026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6f40fcb30 0x7fe6f41096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:00.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.600+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3279354026 >> 172.21.15.162:0/3279354026 conn(0x7fe6f406eb90 msgr2=0x7fe6f40fa840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:00.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.600+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3279354026 shutdown_connections 2024-02-13T00:40:00.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.600+0000 7fe6fa1b3700 1 -- 172.21.15.162:0/3279354026 wait complete. 2024-02-13T00:40:00.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:00.600+0000 7fe6fa1b3700 1 librados: shutdown 2024-02-13T00:40:00.955 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:00.000126+0000 mon.a (mon.0) 715 : cluster [INF] overall HEALTH_OK 2024-02-13T00:40:00.955 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:00.599792+0000 mon.c (mon.2) 80 : audit [DBG] from='client.? 172.21.15.162:0/3279354026' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-13T00:40:00.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:00.000126+0000 mon.a (mon.0) 715 : cluster [INF] overall HEALTH_OK 2024-02-13T00:40:00.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:00.599792+0000 mon.c (mon.2) 80 : audit [DBG] from='client.? 172.21.15.162:0/3279354026' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-13T00:40:01.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:00.000126+0000 mon.a (mon.0) 715 : cluster [INF] overall HEALTH_OK 2024-02-13T00:40:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:00.599792+0000 mon.c (mon.2) 80 : audit [DBG] from='client.? 172.21.15.162:0/3279354026' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-13T00:40:01.407 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) v2:172.21.15.162:6801/2103743338 v2:172.21.15.162:6802/2103743338 exists,up 9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99', 'osd.1 up in weight 1 up_from 13 up_thru 62 down_at 0 last_clean_interval [0,0) v2:172.21.15.162:6805/150599780 v2:172.21.15.162:6806/150599780 exists,up 39e9d2ef-9aa4-45bb-8a47-b0e0ec222886', 'osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) v2:172.21.15.162:6809/3689719376 v2:172.21.15.162:6810/3689719376 exists,up 86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e', 'osd.3 up in weight 1 up_from 25 up_thru 62 down_at 0 last_clean_interval [0,0) v2:172.21.15.162:6813/604971011 v2:172.21.15.162:6814/604971011 exists,up 2e6b9f0e-7a18-4cf8-9331-1a973fadc001', 'osd.4 up in weight 1 up_from 31 up_thru 81 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6800/3344263388 v2:172.21.15.181:6801/3344263388 exists,up c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3', 'osd.5 up in weight 1 up_from 39 up_thru 62 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6804/2212296748 v2:172.21.15.181:6805/2212296748 exists,up 6aa2bd68-7f4b-4973-ac49-7fbc57c1a723', 'osd.6 up in weight 1 up_from 45 up_thru 60 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6808/1506640341 v2:172.21.15.181:6809/1506640341 exists,up 791ba814-5284-432e-addc-d6b7f5f3d474', 'osd.7 up in weight 1 up_from 53 up_thru 62 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6812/1964192772 v2:172.21.15.181:6813/1964192772 exists,up 1f369c65-e663-4eb2-87a5-f2900f15b35b'] 2024-02-13T00:40:01.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon mon.a config show 2024-02-13T00:40:01.778 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:02.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:00.187206+0000 mgr.y (mgr.14544) 97 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:02.050 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:00.187206+0000 mgr.y (mgr.14544) 97 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:02.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:00.187206+0000 mgr.y (mgr.14544) 97 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:03.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:01.402226+0000 osd.7 (osd.7) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:03.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:01.402284+0000 osd.7 (osd.7) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:03.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:01.706143+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-13T00:40:03.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:02.086244+0000 osd.4 (osd.4) 33 : cluster [DBG] 5.e scrub starts 2024-02-13T00:40:03.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:02.089854+0000 osd.4 (osd.4) 34 : cluster [DBG] 5.e scrub ok 2024-02-13T00:40:03.069 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:01.402226+0000 osd.7 (osd.7) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:03.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:01.402284+0000 osd.7 (osd.7) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:03.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:01.706143+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-13T00:40:03.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:02.086244+0000 osd.4 (osd.4) 33 : cluster [DBG] 5.e scrub starts 2024-02-13T00:40:03.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:02.089854+0000 osd.4 (osd.4) 34 : cluster [DBG] 5.e scrub ok 2024-02-13T00:40:03.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:01.402226+0000 osd.7 (osd.7) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:03.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:01.402284+0000 osd.7 (osd.7) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:03.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:01.706143+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-13T00:40:03.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:02.086244+0000 osd.4 (osd.4) 33 : cluster [DBG] 5.e scrub starts 2024-02-13T00:40:03.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:02.089854+0000 osd.4 (osd.4) 34 : cluster [DBG] 5.e scrub ok 2024-02-13T00:40:03.706 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:40:03.706 INFO:teuthology.orchestra.run.smithi162.stdout: "name": "mon.a", 2024-02-13T00:40:03.706 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster": "ceph", 2024-02-13T00:40:03.706 INFO:teuthology.orchestra.run.smithi162.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-02-13T00:40:03.706 INFO:teuthology.orchestra.run.smithi162.stdout: "admin_socket_mode": "", 2024-02-13T00:40:03.706 INFO:teuthology.orchestra.run.smithi162.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-02-13T00:40:03.706 INFO:teuthology.orchestra.run.smithi162.stdout: "auth_client_required": "cephx, none", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "auth_cluster_required": "cephx", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "auth_debug": "true", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "auth_service_required": "cephx", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "auth_service_ticket_ttl": "240.000000", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "auth_supported": "", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_aio": "true", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_aio_max_queue_depth": "1024", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_aio_poll_ms": "250", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_aio_reap_max": "16", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_async_discard": "false", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_block_size": "4096", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_debug_aio": "false", 2024-02-13T00:40:03.707 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-02-13T00:40:03.708 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-02-13T00:40:03.708 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_debug_inflight_ios": "false", 2024-02-13T00:40:03.708 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_enable_discard": "false", 2024-02-13T00:40:03.708 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_flock_retry": "3", 2024-02-13T00:40:03.708 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_flock_retry_interval": "0.100000", 2024-02-13T00:40:03.708 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_inject_crash": "0", 2024-02-13T00:40:03.708 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_inject_crash_flush_delay": "2", 2024-02-13T00:40:03.708 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_ioring": "false", 2024-02-13T00:40:03.708 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_ioring_hipri": "false", 2024-02-13T00:40:03.708 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_ioring_sqthread_poll": "false", 2024-02-13T00:40:03.708 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-02-13T00:40:03.708 INFO:teuthology.orchestra.run.smithi162.stdout: "bdev_type": "", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_alloc_size": "1048576", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_allocator": "hybrid", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_buffered_io": "true", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_check_for_zeros": "false", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_check_volume_selector_often": "false", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_compact_log_sync": "false", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_log_compact_min_size": "16777216", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_log_replay_check_allocations": "true", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_max_log_runway": "4194304", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_max_prefetch": "1048576", 2024-02-13T00:40:03.709 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_min_flush_size": "524288", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_min_log_runway": "1048576", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_replay_recovery": "false", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_shared_alloc_size": "65536", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_sync_write": "false", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_allocator": "hybrid", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-02-13T00:40:03.710 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-02-13T00:40:03.711 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blobid_prealloc": "10240", 2024-02-13T00:40:03.711 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_block_create": "true", 2024-02-13T00:40:03.711 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_block_db_create": "false", 2024-02-13T00:40:03.711 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_block_db_path": "", 2024-02-13T00:40:03.711 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_block_db_size": "0", 2024-02-13T00:40:03.711 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_block_path": "", 2024-02-13T00:40:03.711 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_block_preallocate_file": "false", 2024-02-13T00:40:03.711 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_block_size": "107374182400", 2024-02-13T00:40:03.711 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_block_wal_create": "false", 2024-02-13T00:40:03.711 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_block_wal_path": "", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_block_wal_size": "100663296", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_bluefs": "true", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_bluefs_env_mirror": "false", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_autotune": "true", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_size": "0", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-02-13T00:40:03.712 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_type": "2q", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_clone_cow": "true", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_compression_algorithm": "snappy", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_compression_max_blob_size": "0", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_compression_min_blob_size": "0", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_compression_mode": "none", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_csum_type": "crc32c", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_enforce_settings": "default", 2024-02-13T00:40:03.713 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_freelist": "false", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_fsck_abort": "false", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_inject_bug21040": "false", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_inject_read_err": "false", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_legacy_omap": "false", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_prefill": "0.000000", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-02-13T00:40:03.714 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_small_allocations": "0", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_default_buffered_read": "true", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_default_buffered_write": "false", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_deferred_batch_ops": "0", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-02-13T00:40:03.715 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_on_mkfs": "true", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_on_mount": "false", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_on_umount": "false", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-02-13T00:40:03.716 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_ignore_data_csum": "false", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_kvbackend": "rocksdb", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_log_op_age": "5.000000", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_max_alloc_size": "0", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_max_blob_size": "0", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_max_defer_interval": "3.000000", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_max_deferred_txc": "32", 2024-02-13T00:40:03.717 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_min_alloc_size": "0", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_nid_prealloc": "1024", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_prefer_deferred_size": "0", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_retry_disk_reads": "3", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_rocksdb_cf": "true", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_rocksdb_options_annex": "", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_spdk_coremask": "0x1", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_spdk_io_sleep": "5", 2024-02-13T00:40:03.718 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_spdk_max_io_completion": "0", 2024-02-13T00:40:03.720 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_spdk_mem": "512", 2024-02-13T00:40:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_sync_submit_transaction": "false", 2024-02-13T00:40:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_throttle_bytes": "67108864", 2024-02-13T00:40:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_throttle_cost_per_io": "0", 2024-02-13T00:40:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-02-13T00:40:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-02-13T00:40:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-02-13T00:40:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-02-13T00:40:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_tracing": "false", 2024-02-13T00:40:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-02-13T00:40:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_volume_selection_reserved": "0", 2024-02-13T00:40:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-02-13T00:40:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-02-13T00:40:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-02-13T00:40:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-02-13T00:40:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-02-13T00:40:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-02-13T00:40:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-02-13T00:40:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "cephfs_mirror_action_update_interval": "2", 2024-02-13T00:40:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-02-13T00:40:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-02-13T00:40:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-02-13T00:40:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-02-13T00:40:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "cephfs_mirror_mount_timeout": "10", 2024-02-13T00:40:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-02-13T00:40:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-02-13T00:40:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-02-13T00:40:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "cephx_cluster_require_signatures": "false", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "cephx_cluster_require_version": "2", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "cephx_require_signatures": "false", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "cephx_require_version": "2", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "cephx_service_require_signatures": "false", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "cephx_service_require_version": "2", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "cephx_sign_messages": "true", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "chdir": "", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "client_acl_type": "", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "client_asio_thread_count": "2", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "client_cache_mid": "0.750000", 2024-02-13T00:40:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "client_cache_size": "16384", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_caps_release_delay": "5", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_check_pool_perm": "true", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_collect_and_send_global_metrics": "false", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_debug_force_sync_read": "false", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_debug_getattr_caps": "false", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_debug_inject_tick_delay": "0", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_die_on_failed_remount": "false", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_dirsize_rbytes": "true", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_force_lazyio": "false", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_fs": "", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_inject_fixed_oldest_tid": "false", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_inject_release_failure": "false", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_max_inline_size": "4096", 2024-02-13T00:40:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_max_retries_on_remount_failure": "5", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_mds_namespace": "", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_metadata": "", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_mount_gid": "-1", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_mount_timeout": "300", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_mount_uid": "-1", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_mountpoint": "/", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_notify_timeout": "10", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_oc": "true", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_oc_max_dirty": "104857600", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_oc_max_dirty_age": "5.000000", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_oc_max_objects": "1000", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_oc_size": "209715200", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_oc_target_dirty": "8388608", 2024-02-13T00:40:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_permissions": "true", 2024-02-13T00:40:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "client_quota": "true", 2024-02-13T00:40:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "client_quota_df": "true", 2024-02-13T00:40:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "client_readahead_max_bytes": "0", 2024-02-13T00:40:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "client_readahead_max_periods": "4", 2024-02-13T00:40:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "client_readahead_min": "131072", 2024-02-13T00:40:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "client_reconnect_stale": "false", 2024-02-13T00:40:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "client_shutdown_timeout": "30", 2024-02-13T00:40:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "client_snapdir": ".snap", 2024-02-13T00:40:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "client_tick_interval": "1", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client_trace": "", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client_try_dentry_invalidate": "false", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client_use_faked_inos": "false", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client_use_random_mds": "false", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "clog_to_graylog": "false", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "clog_to_graylog_port": "12201", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "clog_to_monitors": "default=true", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "clog_to_syslog": "false", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "clog_to_syslog_level": "info", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "-", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_network": "", 2024-02-13T00:40:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_network_interface": "", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "compressor_zlib_isal": "false", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "compressor_zlib_level": "5", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "compressor_zlib_winsize": "-15", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "compressor_zstd_level": "1", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "crimson_osd_obc_lru_size": "10", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_location": "", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_location_hook": "", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_location_hook_timeout": "10", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "daemonize": "false", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_allow_any_pool_priority": "false", 2024-02-13T00:40:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_asok": "1/5", 2024-02-13T00:40:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_asok_assert_abort": "false", 2024-02-13T00:40:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_asserts_on_shutdown": "true", 2024-02-13T00:40:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_auth": "1/5", 2024-02-13T00:40:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_bdev": "1/3", 2024-02-13T00:40:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_bluefs": "1/5", 2024-02-13T00:40:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_bluestore": "1/5", 2024-02-13T00:40:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_buffer": "0/1", 2024-02-13T00:40:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_cephfs_mirror": "0/5", 2024-02-13T00:40:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_cephsqlite": "0/5", 2024-02-13T00:40:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_civetweb": "1/10", 2024-02-13T00:40:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_client": "0/5", 2024-02-13T00:40:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_compressor": "1/5", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_context": "0/1", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_crush": "1/1", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_crypto": "1/5", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_deliberately_leak_memory": "false", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_disable_randomized_ping": "false", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_dpdk": "1/5", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_eventtrace": "1/5", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_filer": "0/1", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_filestore": "1/3", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_finisher": "1/1", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_fuse": "1/5", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_heartbeat_testing_span": "0", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_heartbeatmap": "1/5", 2024-02-13T00:40:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_immutable_obj_cache": "0/5", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_javaclient": "1/5", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_journal": "1/3", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_journaler": "0/5", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_kstore": "1/5", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_leveldb": "4/5", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_lockdep": "0/1", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_mds": "1/5", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_mds_balancer": "1/5", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_mds_locker": "1/5", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_mds_log": "1/5", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_mds_log_expire": "1/5", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_mds_migrator": "1/5", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_memdb": "4/5", 2024-02-13T00:40:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_mgr": "2/5", 2024-02-13T00:40:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_mgrc": "20/20", 2024-02-13T00:40:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_mon": "20/20", 2024-02-13T00:40:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_monc": "0/10", 2024-02-13T00:40:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_ms": "1/1", 2024-02-13T00:40:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_none": "0/5", 2024-02-13T00:40:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_objclass": "0/5", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_objectcacher": "0/5", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_objecter": "0/1", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_optracker": "0/5", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_osd": "1/5", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_paxos": "20/20", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_perfcounter": "1/5", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_prioritycache": "1/5", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_rados": "0/5", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_rbd": "0/5", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_rbd_mirror": "0/5", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_rbd_pwl": "0/5", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_rbd_replay": "0/5", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_refs": "0/0", 2024-02-13T00:40:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_reserver": "1/1", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_rgw": "1/5", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_rgw_sync": "1/5", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_rocksdb": "4/5", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_striper": "0/1", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_test": "0/5", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_throttle": "1/1", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_timer": "0/1", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "debug_tp": "0/5", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "device_failure_prediction_mode": "none", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "err_to_graylog": "false", 2024-02-13T00:40:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "err_to_stderr": "true", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "err_to_syslog": "false", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event_tracing": "false", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "fake_statfs_for_testing": "0", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "fatal_signal_handlers": "true", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "filer_max_purge_ops": "10", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "filer_max_truncate_ops": "128", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_apply_finisher_threads": "1", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_blackhole": "false", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_btrfs_clone_range": "true", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_btrfs_snap": "true", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_caller_concurrency": "10", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_collect_device_partition_information": "true", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_commit_timeout": "600.000000", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_debug_inject_read_err": "false", 2024-02-13T00:40:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_debug_omap_check": "false", 2024-02-13T00:40:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_debug_random_read_err": "0.000000", 2024-02-13T00:40:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_debug_verify_split": "false", 2024-02-13T00:40:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_dump_file": "", 2024-02-13T00:40:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-02-13T00:40:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-02-13T00:40:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_fadvise": "true", 2024-02-13T00:40:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_fail_eio": "true", 2024-02-13T00:40:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_fd_cache_shards": "16", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_fd_cache_size": "128", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_fiemap": "false", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_fiemap_threshold": "4096", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_index_retry_probability": "0.000000", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_inject_stall": "0", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_journal_parallel": "false", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_journal_trailing": "false", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_journal_writeahead": "false", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_kill_at": "0", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_inline_xattr_size": "0", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-02-13T00:40:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-02-13T00:40:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-02-13T00:40:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_inline_xattrs": "0", 2024-02-13T00:40:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-02-13T00:40:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_inline_xattrs_other": "2", 2024-02-13T00:40:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-02-13T00:40:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_sync_interval": "5.000000", 2024-02-13T00:40:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_xattr_value_size": "0", 2024-02-13T00:40:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-02-13T00:40:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-02-13T00:40:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-02-13T00:40:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_merge_threshold": "-10", 2024-02-13T00:40:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_min_sync_interval": "0.010000", 2024-02-13T00:40:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_odsync_write": "false", 2024-02-13T00:40:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_omap_backend": "rocksdb", 2024-02-13T00:40:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_omap_backend_path": "", 2024-02-13T00:40:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_omap_header_cache_size": "1024", 2024-02-13T00:40:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_ondisk_finisher_threads": "1", 2024-02-13T00:40:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-02-13T00:40:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_op_thread_timeout": "60", 2024-02-13T00:40:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_op_threads": "2", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_punch_hole": "false", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_queue_max_bytes": "104857600", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_queue_max_ops": "50", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-02-13T00:40:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_seek_data_hole": "false", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_sloppy_crc": "false", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_splice": "false", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_split_multiple": "2", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_split_rand_factor": "20", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_update_to": "1000", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_enable": "true", 2024-02-13T00:40:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_xfs_extsize": "false", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "filestore_zfs_snap": "false", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "fio_dir": "/tmp/fio", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_allow_other": "true", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_atomic_o_trunc": "true", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_big_writes": "true", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_debug": "false", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_default_permissions": "false", 2024-02-13T00:40:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_disable_pagecache": "false", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_max_write": "0", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_multithreaded": "true", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_require_active_mds": "true", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_set_user_groups": "true", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_splice_move": "true", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_splice_read": "true", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_splice_write": "true", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_syncfs_on_mksnap": "true", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "fuse_use_invalidate_cb": "true", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "gss_target_name": "ceph", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_file": "", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_inject_failure": "0", 2024-02-13T00:40:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_interval": "5", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "host": "", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_max_size": "1073741824", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-02-13T00:40:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "immutable_object_cache_watermark": "0.900000", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "inject_early_sigterm": "false", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_aio": "true", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_align_min_size": "65536", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_block_align": "true", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_block_size": "4096", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_dio": "true", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_discard": "false", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_force_aio": "false", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_ignore_corruption": "false", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_max_write_bytes": "10485760", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_max_write_entries": "100", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_replay_from": "0", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_throttle_high_multiple": "0.000000", 2024-02-13T00:40:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_throttle_max_multiple": "0.000000", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_write_header_frequency": "0", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "journal_zero_on_create": "false", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "journaler_prefetch_periods": "10", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "journaler_prezero_periods": "5", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "journaler_write_head_interval": "15", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "key": "", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "keyfile": "", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "kstore_backend": "rocksdb", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "kstore_default_stripe_size": "65536", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "kstore_fsck_on_mount": "false", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "kstore_fsck_on_mount_deep": "true", 2024-02-13T00:40:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "kstore_max_bytes": "67108864", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "kstore_max_ops": "512", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "kstore_nid_prealloc": "1024", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "kstore_onode_map_size": "1024", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "kstore_sync_submit_transaction": "false", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "kstore_sync_transaction": "false", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "leveldb_block_size": "65536", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "leveldb_bloom_size": "0", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "leveldb_cache_size": "536870912", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "leveldb_compact_on_mount": "false", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "leveldb_compression": "false", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "leveldb_log": "", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "leveldb_log_to_ceph_log": "true", 2024-02-13T00:40:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "leveldb_max_open_files": "0", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "leveldb_paranoid": "false", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "leveldb_write_buffer_size": "33554432", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "librados_thread_count": "2", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "lockdep": "false", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "lockdep_force_backtrace": "false", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "log_coarse_timestamps": "true", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "log_flush_on_exit": "false", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "log_graylog_host": "127.0.0.1", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "log_graylog_port": "12201", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "log_max_new": "1000", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "log_max_recent": "10000", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "log_stderr_prefix": "debug ", 2024-02-13T00:40:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "log_stop_at_utilization": "0.970000", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "log_to_file": "true", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "log_to_graylog": "false", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "log_to_stderr": "false", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "log_to_syslog": "false", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "max_rotating_auth_attempts": "10", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_action_on_write_error": "1", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_alternate_name_max": "8192", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_asio_thread_count": "2", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_export_pin": "true", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_fragment_dirs": "true", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_fragment_interval": "5", 2024-02-13T00:40:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_fragment_size_max": "100000", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_idle_threshold": "0.000000", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_interval": "10", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_max": "-1", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_max_until": "-1", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_merge_size": "50", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_midchunk": "0.300000", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_min_rebalance": "0.100000", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_min_start": "0.200000", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_minchunk": "0.001000", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_mode": "0", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_need_max": "1.200000", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_need_min": "0.800000", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_sample_interval": "3.000000", 2024-02-13T00:40:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_split_bits": "3", 2024-02-13T00:40:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_split_rd": "25000.000000", 2024-02-13T00:40:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_split_size": "10000", 2024-02-13T00:40:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_split_wr": "10000.000000", 2024-02-13T00:40:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_target_decay": "10.000000", 2024-02-13T00:40:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-02-13T00:40:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_beacon_grace": "15.000000", 2024-02-13T00:40:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_beacon_interval": "4.000000", 2024-02-13T00:40:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_beacon_mon_down_grace": "60", 2024-02-13T00:40:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache_memory_limit": "4294967296", 2024-02-13T00:40:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache_mid": "0.700000", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache_release_free_interval": "10", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache_reservation": "0.050000", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache_trim_interval": "1", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache_trim_threshold": "262144", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_client_delegate_inos_pct": "50", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_client_prealloc_inos": "1000", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_connect_bootstrapping": "false", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_damage_table_max_entries": "10000", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-02-13T00:40:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_debug_auth_pins": "false", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_debug_frag": "false", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_debug_scatterstat": "false", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_debug_subtrees": "false", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_decay_halflife": "5.000000", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_default_dir_hash": "2", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_defer_session_stale": "true", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_deny_all_reconnect": "false", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_dir_keys_per_op": "16384", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_dir_max_commit_size": "10", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_dir_max_entries": "0", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_dirstat_min_interval": "1.000000", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_dump_cache_after_rejoin": "false", 2024-02-13T00:40:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_dump_cache_on_map": "false", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_dump_cache_threshold_file": "0", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_early_reply": "true", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_enable_op_tracker": "true", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_enforce_unique_name": "true", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_export_ephemeral_distributed": "true", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_export_ephemeral_random": "true", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_forward_all_requests_to_auth": "false", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_health_cache_threshold": "1.500000", 2024-02-13T00:40:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_health_summarize_threshold": "10", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_heartbeat_grace": "15.000000", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_heartbeat_reset_grace": "1000", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_inject_health_dummy": "false", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_inject_migrator_session_race": "false", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_join_fs": "", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_journal_format": "1", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_kill_create_at": "0", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_kill_export_at": "0", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_kill_import_at": "0", 2024-02-13T00:40:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_kill_journal_at": "0", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_kill_journal_expire_at": "0", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_kill_journal_replay_at": "0", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_kill_link_at": "0", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_kill_mdstable_at": "0", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_kill_openc_at": "0", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_kill_rename_at": "0", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log_events_per_segment": "1024", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log_max_events": "-1", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log_max_segments": "128", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log_pause": "false", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log_segment_size": "0", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log_skip_corrupt_events": "false", 2024-02-13T00:40:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log_warn_factor": "2.000000", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_max_caps_per_client": "1048576", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_max_completed_flushes": "100000", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_max_completed_requests": "100000", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_max_export_size": "20971520", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_max_file_recover": "32", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_max_purge_files": "64", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_max_purge_ops": "8192", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_max_snaps_per_dir": "100", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_max_xattr_pairs_size": "65536", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_metrics_update_interval": "2", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_min_caps_per_client": "100", 2024-02-13T00:40:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_min_caps_working_set": "10000", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_numa_node": "-1", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_op_complaint_time": "30.000000", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_op_history_duration": "600", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_op_history_size": "20", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_op_log_threshold": "5", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_ping_grace": "15", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_ping_interval": "5", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_recall_max_caps": "30000", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_recall_max_decay_threshold": "131072", 2024-02-13T00:40:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_recall_warning_threshold": "262144", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_reconnect_timeout": "45.000000", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_replay_interval": "1.000000", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_root_ino_gid": "0", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_root_ino_uid": "0", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_session_blocklist_on_evict": "true", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_session_blocklist_on_timeout": "true", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-02-13T00:40:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_session_metadata_threshold": "16777216", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_shutdown_check": "0", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_skip_ino": "0", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sleep_rank_change": "0.000000", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_snap_max_uid": "4294967294", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_snap_min_uid": "0", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_snap_rstat": "false", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_standby_replay_damaged": "false", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_task_status_update_interval": "2.000000", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_thrash_exports": "0", 2024-02-13T00:40:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_thrash_fragments": "0", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_tick_interval": "5.000000", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_valgrind_exit": "false", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_verify_backtrace": "1", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_verify_scatter": "false", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_wipe_ino_prealloc": "false", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_wipe_sessions": "false", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool_debug": "false", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "memstore_debug_omit_block_device_write": "false", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "memstore_device_bytes": "1073741824", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "memstore_page_set": "false", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "memstore_page_size": "65536", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_client_bytes": "134217728", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_client_messages": "512", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-02-13T00:40:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_connect_retry_interval": "1.000000", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_disabled_modules": "", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_max_pg_num_change": "128", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_mds_bytes": "134217728", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_mds_messages": "128", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_mon_bytes": "134217728", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_mon_messages": "128", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_osd_bytes": "536870912", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_osd_messages": "8192", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_service_beacon_grace": "60.000000", 2024-02-13T00:40:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_standby_modules": "true", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_stats_period": "5", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_stats_threshold": "5", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_tick_period": "2", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_accept_timeout_factor": "2.000000", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_allow_pool_delete": "true", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_allow_pool_size_one": "false", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_auth_validate_all_caps": "true", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_client_bytes": "104857600", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_client_directed_command_retry": "2", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_client_hunt_interval": "3.000000", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-02-13T00:40:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_client_hunt_parallel": "3", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_client_log_interval": "1.000000", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_client_ping_interval": "10.000000", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_client_ping_timeout": "30.000000", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_clock_drift_allowed": "1.000000", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_cluster_log_file_level": "debug", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_cluster_log_to_file": "true", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_cluster_log_to_graylog": "false", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-02-13T00:40:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_cluster_log_to_stderr": "true", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_compact_on_bootstrap": "false", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_compact_on_start": "false", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_compact_on_trim": "true", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_con_tracker_persist_interval": "10", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_con_tracker_score_halflife": "43200", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_config_key_max_entry_size": "65536", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_cpu_threads": "4", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_crush_min_required_version": "hammer", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_daemon_bytes": "419430400", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-02-13T00:40:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_data_avail_crit": "5", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_data_avail_warn": "5", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_data_size_warn": "16106127360", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_debug_block_osdmap_trim": "false", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_debug_dump_json": "false", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_debug_dump_transactions": "false", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_debug_extra_checks": "false", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_debug_no_require_octopus": "false", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_debug_no_require_pacific": "false", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-02-13T00:40:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_delta_reset_interval": "10.000000", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_dns_srv_name": "ceph-mon", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_down_mkfs_grace": "60", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_election_default_strategy": "1", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_election_timeout": "5.000000", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_elector_ping_divisor": "2", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_elector_ping_timeout": "2.000000", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_enable_op_tracker": "true", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_fake_pool_delete": "false", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_force_quorum_join": "false", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_globalid_prealloc": "10000", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_health_detail_to_clog": "true", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_health_log_update_period": "5", 2024-02-13T00:40:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_health_max_detail": "50", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_health_to_clog": "true", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_health_to_clog_interval": "600", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_host": "[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0]", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_host_override": "", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_initial_members": "", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_keyvaluedb": "rocksdb", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_lease": "5.000000", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-02-13T00:40:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_log_max_summary": "50", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_max_log_entries_per_event": "4096", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_max_log_epochs": "500", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_max_mdsmap_epochs": "500", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_max_mgrmap_epochs": "500", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_max_osd": "10000", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_max_pg_per_osd": "10000", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_max_pool_pg_num": "65536", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_mds_force_trim_to": "0", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_mds_skip_sanity": "false", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_memory_autotune": "true", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_memory_target": "2147483648", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_mgr_beacon_grace": "30", 2024-02-13T00:40:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_mgr_digest_period": "5", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_mgr_inactive_grace": "60", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_mgr_mkfs_grace": "240", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_min_osdmap_epochs": "50", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_op_complaint_time": "30", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_op_history_duration": "600", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_op_history_size": "20", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_op_history_slow_op_size": "20", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_op_history_slow_op_threshold": "10", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_op_log_threshold": "5", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-02-13T00:40:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_auto_mark_in": "false", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_auto_mark_new_in": "true", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_cache_size": "500", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_cache_size_min": "134217728", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_crush_smoke_test": "true", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_destroyed_out_interval": "600", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_down_out_interval": "600", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_force_trim_to": "0", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_full_ratio": "0.950000", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-02-13T00:40:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_laggy_halflife": "3600", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_laggy_max_interval": "300", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_laggy_weight": "0.300000", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_max_creating_pgs": "1024", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_max_initial_pgs": "1024", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_min_down_reporters": "2", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_prime_pg_temp": "true", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_report_timeout": "900", 2024-02-13T00:40:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_warn_num_repaired": "10", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osd_warn_op_age": "32.000000", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osdmap_full_prune_interval": "2", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osdmap_full_prune_min": "15", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_pg_stuck_threshold": "60", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_pg_warn_min_objects": "10000", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_pg_warn_min_per_osd": "0", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_pool_quota_crit_threshold": "0", 2024-02-13T00:40:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_pool_quota_warn_threshold": "0", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_probe_timeout": "2.000000", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_reweight_max_change": "0.050000", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_reweight_max_osds": "4", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_scrub_interval": "86400", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_scrub_max_keys": "100", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_scrub_timeout": "300", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_session_timeout": "300", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_smart_report_timeout": "5", 2024-02-13T00:40:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_stat_smooth_intervals": "6", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_stretch_pool_min_size": "2", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_stretch_pool_size": "4", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_subscribe_interval": "86400.000000", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_sync_debug": "false", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_sync_max_payload_keys": "2000", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_sync_max_payload_size": "1048576", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_sync_provider_kill_at": "0", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_sync_requester_kill_at": "0", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_sync_timeout": "60.000000", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_target_pg_per_osd": "100", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_tick_interval": "5", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_timecheck_interval": "300.000000", 2024-02-13T00:40:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_older_version_delay": "604800", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_misplaced": "false", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_older_version": "true", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_pool_no_app": "false", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-02-13T00:40:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_on_too_few_osds": "false", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "monmap": "", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_max_op_threads": "5", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_op_threads": "3", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_buffer_size": "131072", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_cm": "false", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_device_name": "", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_dscp": "96", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-02-13T00:40:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_gid_idx": "0", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_local_gid": "", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_polling_us": "1000", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_port_num": "1", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_roce_ver": "1", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_send_buffers": "1024", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_sl": "3", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_support_srq": "true", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_rdma_type": "ib", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_async_reap_threshold": "5", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_bind_before_connect": "false", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_bind_ipv4": "true", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_bind_ipv6": "false", 2024-02-13T00:40:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_bind_msgr1": "false", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_bind_msgr2": "true", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_bind_port_max": "7300", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_bind_port_min": "6800", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_bind_prefer_ipv4": "false", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_bind_retry_count": "3", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_bind_retry_delay": "5", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_blackhole_client": "false", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_blackhole_mds": "false", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_blackhole_mgr": "false", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_blackhole_mon": "false", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_blackhole_osd": "false", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_client_mode": "crc secure", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_cluster_mode": "crc secure", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_cluster_type": "", 2024-02-13T00:40:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_connection_idle_timeout": "900", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_connection_ready_timeout": "10", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_crc_data": "true", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_crc_header": "true", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_die_on_bad_msg": "false", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_die_on_bug": "true", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_die_on_old_message": "true", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_die_on_skipped_message": "false", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_die_on_unhandled_msg": "false", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_coremask": "0xF", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-02-13T00:40:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_hugepages": "", 2024-02-13T00:40:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_hw_flow_control": "true", 2024-02-13T00:40:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-02-13T00:40:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_lro": "true", 2024-02-13T00:40:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_memory_channel": "4", 2024-02-13T00:40:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-02-13T00:40:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_pmd": "", 2024-02-13T00:40:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_port_id": "0", 2024-02-13T00:40:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-02-13T00:40:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dump_corrupt_message_level": "1", 2024-02-13T00:40:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_dump_on_send": "false", 2024-02-13T00:40:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_initial_backoff": "0.200000", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_inject_delay_max": "1.000000", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_inject_delay_probability": "0.000000", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_inject_delay_type": "", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_inject_internal_delays": "0.000000", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_inject_socket_failures": "0", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_learn_addr_from_peer": "true", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_max_accept_failures": "4", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_max_backoff": "15.000000", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_mon_client_mode": "secure crc", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_mon_cluster_mode": "secure crc", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_mon_service_mode": "secure crc", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_pq_min_cost": "65536", 2024-02-13T00:40:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_public_type": "", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_service_mode": "crc secure", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_tcp_listen_backlog": "512", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_tcp_nodelay": "true", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_tcp_rcvbuf": "0", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "ms_type": "async+posix", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "no_config_file": "false", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter_completion_locks_per_session": "32", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter_debug_inject_relock_delay": "false", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter_inflight_op_bytes": "104857600", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter_inflight_ops": "1024", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter_inject_no_watch_ping": "false", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter_tick_interval": "5.000000", 2024-02-13T00:40:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter_timeout": "10.000000", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "objectstore_blackhole": "false", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "openssl_engine_opts": "", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_agent_delay_time": "5.000000", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_agent_hist_halflife": "1000", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_agent_max_low_ops": "2", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_agent_max_ops": "4", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_agent_quantize_effort": "0.100000", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_agent_slop": "0.020000", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_allow_recovery_below_min_size": "true", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_asio_thread_count": "2", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_async_recovery_min_cost": "100", 2024-02-13T00:40:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_auto_mark_unfound_lost": "false", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_backfill_retry_interval": "30.000000", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_backfill_scan_max": "512", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_backfill_scan_min": "64", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_backoff_on_degraded": "false", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_backoff_on_peering": "false", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_backoff_on_unfound": "true", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_beacon_report_interval": "300", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bench_duration": "30", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bench_max_block_size": "67108864", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bench_small_size_max_iops": "100", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_blkin_trace_all": "false", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-02-13T00:40:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_check_for_log_corruption": "false", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_class_update_on_start": "true", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_client_message_cap": "256", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_client_message_size_cap": "524288000", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_client_op_priority": "63", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_client_watch_timeout": "30", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_command_max_records": "256", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_command_thread_suicide_timeout": "900", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_command_thread_timeout": "600", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_compact_on_start": "false", 2024-02-13T00:40:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_crush_chooseleaf_type": "1", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_crush_initial_weight": "-1.000000", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_crush_update_on_start": "true", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_crush_update_weight_set": "true", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_drop_ping_duration": "0", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_feed_pullee": "-1", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-02-13T00:40:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_misdirected_ops": "false", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_no_acting_change": "false", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_no_purge_strays": "false", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_op_order": "false", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_pg_log_writeout": "false", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_pretend_recovery_active": "false", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_shutdown": "false", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_verify_cached_snaps": "false", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_verify_missing_on_start": "false", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_verify_snaps": "false", 2024-02-13T00:40:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_deep_scrub_keys": "1024", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_deep_scrub_stride": "524288", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_default_data_pool_replay_window": "45", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_default_notify_timeout": "30", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_delete_sleep": "0.000000", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_discard_disconnected_ops": "true", 2024-02-13T00:40:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_enable_op_tracker": "true", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_fast_info": "true", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_fast_shutdown": "true", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_function_tracing": "false", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_heartbeat_grace": "20", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_heartbeat_interval": "6", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_heartbeat_min_peers": "10", 2024-02-13T00:40:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_heartbeat_min_size": "2000", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_heartbeat_stale": "600", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_hit_set_max_size": "100000", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_hit_set_min_size": "1000", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_journal_flush_on_shutdown": "true", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_journal_size": "5120", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_kill_backfill_at": "0", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_loop_before_reset_tphandle": "64", 2024-02-13T00:40:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_map_cache_size": "50", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_map_dedup": "true", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_map_message_max": "40", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_map_message_max_bytes": "10485760", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_map_share_max_epochs": "40", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_attr_name_len": "100", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_attr_size": "0", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_backfills": "1", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_markdown_count": "5", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_markdown_period": "600", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_object_name_len": "2048", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_object_namespace_len": "256", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_object_size": "134217728", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-02-13T00:40:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_omap_entries_per_request": "1024", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_pg_blocked_by": "16", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_pg_log_entries": "10000", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_pgls": "1024", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_push_cost": "8388608", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_push_objects": "10", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_scrubs": "1", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_trimming_pgs": "2", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_write_op_reply_len": "64", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_max_write_size": "90", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_cost_per_byte_usec": "0.000000", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_cost_per_byte_usec_hdd": "5.200000", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_cost_per_byte_usec_ssd": "0.011000", 2024-02-13T00:40:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_cost_per_io_usec": "0.000000", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_cost_per_io_usec_hdd": "25000.000000", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_cost_per_io_usec_ssd": "50.000000", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_profile": "high_client_ops", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-02-13T00:40:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_scheduler_client_lim": "999999", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_scheduler_client_res": "1", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mclock_skip_benchmark": "false", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_memory_base": "805306368", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_memory_cache_min": "134217728", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_memory_target": "4294967296", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_memory_target_autotune": "false", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_min_pg_log_entries": "250", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_min_recovery_priority": "0", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mon_heartbeat_interval": "30", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-02-13T00:40:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mon_report_interval": "5", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mon_report_max_in_flight": "2", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_num_cache_shards": "32", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_num_op_tracker_shard": "32", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_numa_auto_affinity": "true", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_numa_node": "-1", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_numa_prefer_iface": "true", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_objecter_finishers": "1", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_objectstore": "bluestore", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_objectstore_fuse": "false", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_objectstore_tracing": "false", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_complaint_time": "30.000000", 2024-02-13T00:40:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_history_duration": "600", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_history_size": "20", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_history_slow_op_size": "20", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_log_threshold": "5", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_num_shards": "0", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_num_shards_hdd": "5", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_num_shards_ssd": "8", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_num_threads_per_shard": "0", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_pq_min_cost": "65536", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_queue": "wpq", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_queue_cut_off": "high", 2024-02-13T00:40:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_thread_suicide_timeout": "150", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_op_thread_timeout": "15", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_open_classes_on_start": "true", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_os_flags": "0", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_peering_op_priority": "255", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pg_delete_cost": "1048576", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pg_delete_priority": "5", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pg_log_dups_tracked": "3000", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pg_log_trim_max": "10000", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pg_log_trim_min": "100", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pg_object_context_cache_count": "64", 2024-02-13T00:40:03.802 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pg_stat_report_interval_max": "500", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_crush_rule": "-1", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_ec_fast_read": "false", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_flag_bulk": "false", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_flag_nodelete": "false", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-02-13T00:40:03.803 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_flags": "0", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_min_size": "0", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_pg_num": "32", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_pgp_num": "0", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_size": "3", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_default_type": "replicated", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pool_use_gmt_hitset": "true", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_push_per_object_cost": "1000", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_read_ec_check_for_errors": "false", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recover_clone_overlap": "true", 2024-02-13T00:40:03.804 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recover_clone_overlap_limit": "10", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_cost": "20971520", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_delay_start": "0.000000", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_max_active": "0", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_max_active_hdd": "3", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_max_active_ssd": "10", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_max_chunk": "8388608", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_max_single_start": "1", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_op_priority": "3", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_op_warn_multiple": "16", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_priority": "5", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_retry_interval": "30.000000", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_sleep": "0.000000", 2024-02-13T00:40:03.805 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_repair_during_recovery": "false", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_requested_scrub_priority": "120", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_rollback_to_cluster_snap": "", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_auto_repair": "false", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_begin_hour": "0", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_begin_week_day": "0", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_chunk_max": "25", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_chunk_min": "5", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_cost": "52428800", 2024-02-13T00:40:03.806 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_during_recovery": "false", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_end_hour": "0", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_end_week_day": "0", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_invalid_stats": "true", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_load_threshold": "0.500000", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_max_interval": "604800.000000", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_max_preemptions": "5", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_min_interval": "86400.000000", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_priority": "5", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_scrub_sleep": "0.000000", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_shutdown_pgref_assert": "false", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_skip_data_digest": "false", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_smart_report_timeout": "5", 2024-02-13T00:40:03.807 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_snap_trim_cost": "1048576", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_snap_trim_priority": "5", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_snap_trim_sleep": "0.000000", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_target_transaction_size": "30", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-02-13T00:40:03.808 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_tier_default_cache_mode": "writeback", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_tracing": "false", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_use_stale_snap": "false", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "osdc_blkin_trace_all": "false", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "paxos_kill_at": "0", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "paxos_max_join_drift": "10", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "paxos_min": "500", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "paxos_min_wait": "0.050000", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "paxos_propose_interval": "1.000000", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "paxos_service_trim_max": "500", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "paxos_service_trim_max_multiplier": "20", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "paxos_service_trim_min": "10", 2024-02-13T00:40:03.809 INFO:teuthology.orchestra.run.smithi162.stdout: "paxos_stash_full_interval": "25", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "paxos_trim_max": "500", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "paxos_trim_min": "250", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "perf": "true", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "pid_file": "", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "-", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrv": "", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "public_bind_addr": "-", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "public_network": "", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "public_network_interface": "", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "qat_compressor_enabled": "false", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "rados_mon_op_timeout": "0", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "rados_osd_op_timeout": "0", 2024-02-13T00:40:03.810 INFO:teuthology.orchestra.run.smithi162.stdout: "rados_tracing": "false", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_atime_update_interval": "60", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_balance_parent_reads": "false", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_balance_snap_reads": "false", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_blkin_trace_all": "false", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_blocklist_expire_seconds": "0", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_blocklist_on_break_lock": "true", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_cache": "true", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_cache_block_writes_upfront": "false", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_cache_max_dirty": "25165824", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_cache_max_dirty_object": "0", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_cache_policy": "writearound", 2024-02-13T00:40:03.811 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_cache_size": "33554432", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_cache_target_dirty": "16777216", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_clone_copy_on_read": "false", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_compression_hint": "none", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_concurrent_management_ops": "10", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_default_clone_format": "auto", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_default_data_pool": "", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_default_features": "61", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_default_format": "2", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_default_map_options": "", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_default_order": "22", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_default_pool": "rbd", 2024-02-13T00:40:03.812 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_default_stripe_count": "0", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_default_stripe_unit": "0", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_disable_zero_copy_writes": "true", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_discard_granularity_bytes": "65536", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_enable_alloc_hint": "true", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_io_scheduler": "simple", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_journal_commit_age": "5.000000", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-02-13T00:40:03.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_journal_object_flush_interval": "0", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_journal_order": "24", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_journal_pool": "", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_journal_splay_width": "4", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_localize_parent_reads": "false", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_localize_snap_reads": "false", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-02-13T00:40:03.814 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_image_policy_type": "simple", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_memory_autotune": "true", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_memory_base": "805306368", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-02-13T00:40:03.815 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_memory_target": "4294967296", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirroring_delete_delay": "0", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirroring_replay_delay": "0", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_move_to_trash_on_remove": "false", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_mtime_update_interval": "60", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_non_blocking_aio": "true", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_op_thread_timeout": "60", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_op_threads": "1", 2024-02-13T00:40:03.816 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_parent_cache_enabled": "false", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_persistent_cache_mode": "disabled", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_persistent_cache_path": "/tmp", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_persistent_cache_size": "1073741824", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_plugins": "", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_bps_burst": "0", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_bps_limit": "0", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_iops_burst": "0", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_iops_limit": "0", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_read_bps_burst": "0", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_read_bps_limit": "0", 2024-02-13T00:40:03.817 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_read_iops_burst": "0", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_read_iops_limit": "0", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_schedule_tick_min": "50", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_write_bps_burst": "0", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_write_bps_limit": "0", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_write_iops_burst": "0", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_qos_write_iops_limit": "0", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_quiesce_notification_attempts": "10", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_read_from_replica_policy": "default", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_readahead_max_bytes": "524288", 2024-02-13T00:40:03.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_readahead_trigger_requests": "10", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_request_timed_out_seconds": "30", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_skip_partial_discard": "true", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_tracing": "false", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_validate_names": "true", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd_validate_pool": "true", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "restapi_base_url": "", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "restapi_log_level": "", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_acl_grants_max_num": "100", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_admin_entry": "admin", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_barbican_url": "", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_bucket_index_max_aio": "128", 2024-02-13T00:40:03.819 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_bucket_quota_ttl": "600", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_cache_enabled": "true", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_cache_expiry_interval": "900", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_cache_lru_size": "10000", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_content_length_compat": "false", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_copy_obj_progress": "true", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_cors_rules_max_num": "100", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_cross_domain_policy": "", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_default_encryption_key": "", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_kmip_addr": "", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_kmip_ca_path": "", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_kmip_client_cert": "", 2024-02-13T00:40:03.820 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_kmip_client_key": "", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_kmip_password": "", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_kmip_username": "", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_require_ssl": "true", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_suppress_logs": "true", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_vault_addr": "", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_vault_auth": "token", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_vault_namespace": "", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_vault_prefix": "", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-02-13T00:40:03.821 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_vault_token_file": "", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_curl_buffersize": "524288", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_curl_low_speed_limit": "1024", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_curl_low_speed_time": "300", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_data_log_changes_size": "1000", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_data_log_num_shards": "128", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_data_log_window": "30", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_data_notify_interval_msec": "200", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-02-13T00:40:03.822 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_default_data_log_backing": "fifo", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_default_region_info_oid": "default.region", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_defer_to_bucket_acls": "", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-02-13T00:40:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dmclock_data_res": "500.000000", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dns_name": "", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dns_s3website_name": "", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_dynamic_resharding": "true", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_enable_gc_threads": "true", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_enable_lc_threads": "true", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_enable_ops_log": "false", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_enable_quota_threads": "true", 2024-02-13T00:40:03.824 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_enable_static_website": "false", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_enable_usage_log": "false", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_enforce_swift_acls": "true", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_exit_timeout_secs": "120", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_expose_bucket": "false", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_extended_http_attrs": "", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_fcgi_socket_backlog": "1024", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_frontends": "beast port=7480", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_gc_max_concurrent_io": "10", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_gc_max_deferred": "50", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_gc_max_objs": "32", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_gc_max_queue_size": "134213632", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_gc_max_trim_chunk": "16", 2024-02-13T00:40:03.825 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_gc_obj_min_wait": "7200", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_gc_processor_max_time": "3600", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_gc_processor_period": "3600", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_get_obj_window_size": "16777216", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_healthcheck_disabling_path": "", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_host": "", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_ignore_get_invalid_range": "false", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_init_timeout": "300", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_admin_domain": "", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_admin_password": "", 2024-02-13T00:40:03.826 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_admin_password_path": "", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_admin_project": "", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_admin_tenant": "", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_admin_token": "", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_admin_token_path": "", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_admin_user": "", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_api_version": "2", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_barbican_domain": "", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_barbican_password": "", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_barbican_project": "", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_barbican_tenant": "", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_barbican_user": "", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_implicit_tenants": "false", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_token_cache_size": "10000", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_url": "", 2024-02-13T00:40:03.827 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_keystone_verify_ssl": "true", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_lc_debug_interval": "-1", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_lc_lock_max_time": "90", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_lc_max_objs": "32", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_lc_max_rules": "1000", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_lc_max_worker": "3", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_lc_max_wp_worker": "3", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_lc_thread_delay": "0", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_ldap_dnattr": "uid", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_ldap_searchfilter": "", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_ldap_uri": "ldaps://", 2024-02-13T00:40:03.828 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_log_http_headers": "", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_log_nonexistent_bucket": "false", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_log_object_name_utc": "false", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_max_attr_name_len": "0", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_max_attr_size": "0", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_max_attrs_num_in_req": "0", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_max_chunk_size": "4194304", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_max_concurrent_requests": "1024", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_max_dynamic_shards": "1999", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_max_listing_results": "1000", 2024-02-13T00:40:03.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_max_notify_retries": "10", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_max_objs_per_shard": "100000", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_max_put_param_size": "1048576", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_max_put_size": "5368709120", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_max_slo_entries": "1000", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_md_log_max_shards": "64", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_md_notify_interval_msec": "200", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_mp_lock_max_time": "600", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_multipart_min_part_size": "5242880", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_nfs_fhcache_size": "2017", 2024-02-13T00:40:03.830 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_nfs_lru_lanes": "5", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_nfs_max_gc": "300", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_nfs_run_gc_threads": "false", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_nfs_run_lc_threads": "false", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_nfs_run_quota_threads": "false", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_nfs_run_sync_thread": "false", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_num_async_rados_threads": "32", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_num_control_oids": "8", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_num_rados_handles": "1", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_numa_node": "-1", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_obj_stripe_size": "4194304", 2024-02-13T00:40:03.831 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_objexp_chunk_size": "100", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_objexp_gc_interval": "600", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_objexp_hints_num_shards": "127", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_op_thread_timeout": "600", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_op_tracing": "false", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_opa_token": "", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_opa_url": "", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_opa_verify_ssl": "true", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_ops_log_file_path": "", 2024-02-13T00:40:03.832 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_ops_log_rados": "true", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_ops_log_socket_path": "", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_period_push_interval": "2.000000", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_period_push_interval_max": "30.000000", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_period_root_pool": ".rgw.root", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_port": "", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_print_continue": "true", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_print_prohibited_content_length": "false", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_rados_tracing": "false", 2024-02-13T00:40:03.833 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_realm": "", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_region": "", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_region_root_pool": ".rgw.root", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_relaxed_region_enforcement": "false", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_request_uri": "", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_reshard_batch_size": "64", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_reshard_max_aio": "128", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_reshard_num_logs": "16", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_reshard_thread_interval": "600", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_resolve_cname": "false", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_rest_getusage_op_compat": "false", 2024-02-13T00:40:03.834 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_run_sync_thread": "true", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_s3_auth_use_keystone": "false", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_s3_auth_use_ldap": "false", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_s3_auth_use_rados": "true", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_s3_auth_use_sts": "false", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_s3_success_create_obj_status": "0", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_scheduler_type": "throttler", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_script_uri": "", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_service_provider_name": "", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_socket_path": "", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sts_client_id": "", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sts_client_secret": "", 2024-02-13T00:40:03.835 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sts_entry": "sts", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sts_key": "sts", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sts_max_session_duration": "43200", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sts_min_session_duration": "900", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sts_token_introspection_url": "", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_swift_account_in_url": "false", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_swift_auth_entry": "auth", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_swift_auth_url": "", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_swift_custom_header": "", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_swift_enforce_content_length": "false", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_swift_need_stats": "true", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_swift_tenant_name": "", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_swift_token_expiration": "86400", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_swift_url": "", 2024-02-13T00:40:03.836 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_swift_url_prefix": "swift", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_swift_versioning_enabled": "false", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sync_lease_period": "120", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sync_log_trim_interval": "1200", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sync_obj_etag_verify": "false", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sync_trace_history_size": "4096", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_thread_pool_size": "512", 2024-02-13T00:40:03.837 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_torrent_comment": "", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_torrent_createby": "", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_torrent_encoding": "", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_torrent_flag": "false", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_torrent_origin": "", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_torrent_sha_unit": "524288", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_torrent_tracker": "", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_trust_forwarded_https": "false", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_usage_log_tick_interval": "30", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_usage_max_shards": "32", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_usage_max_user_shards": "1", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_use_opa_authz": "false", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_user_default_quota_max_size": "-1", 2024-02-13T00:40:03.838 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_user_max_buckets": "1000", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_user_quota_sync_interval": "86400", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_user_unique_email": "true", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_verify_ssl": "true", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_website_routing_rules_max_num": "50", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_zone": "", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_zonegroup": "", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_block_size": "4096", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-02-13T00:40:03.839 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_cache_shard_bits": "4", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_cache_size": "536870912", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_cache_type": "binned_lru", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_cf_compact_on_deletion": "false", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_collect_compaction_stats": "false", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_collect_extended_stats": "false", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_collect_memory_stats": "false", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_index_type": "binary_search", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_log_to_ceph_log": "true", 2024-02-13T00:40:03.840 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_metadata_block_size": "4096", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_partition_filters": "false", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_perf": "false", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "rotating_keys_renewal_timeout": "10", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "run_dir": "/var/run/ceph", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "setgroup": "ceph", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "setuser": "ceph", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "setuser_match_path": "", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_misplaced_ratio": "0.050000", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "thp": "false", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "threadpool_default_timeout": "60", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "threadpool_empty_queue_max_wait": "2", 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout: "throttler_perf_counter": "true" 2024-02-13T00:40:03.841 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:40:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:02.165034+0000 osd.7 (osd.7) 43 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:40:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:02.168495+0000 osd.7 (osd.7) 44 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:40:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:02.187560+0000 mgr.y (mgr.14544) 98 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:04.047 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:02.165034+0000 osd.7 (osd.7) 43 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:40:04.047 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:02.168495+0000 osd.7 (osd.7) 44 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:40:04.047 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:02.187560+0000 mgr.y (mgr.14544) 98 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:04.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:02.165034+0000 osd.7 (osd.7) 43 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:40:04.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:02.168495+0000 osd.7 (osd.7) 44 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:40:04.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:02.187560+0000 mgr.y (mgr.14544) 98 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:04.517 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-02-13T00:40:04.792 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:04.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:03.350008+0000 osd.5 (osd.5) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:04.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:03.350080+0000 osd.5 (osd.5) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:04.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:03.700640+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:40:04.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:03.701601+0000 osd.5 (osd.5) 44 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:40:04.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:03.731088+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-13T00:40:04.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:04.181454+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:40:04.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:04.185120+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:40:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:03.350008+0000 osd.5 (osd.5) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:03.350080+0000 osd.5 (osd.5) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:03.700640+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:40:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:03.701601+0000 osd.5 (osd.5) 44 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:40:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:03.731088+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-13T00:40:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:04.181454+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:40:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:04.185120+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:40:05.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:03.350008+0000 osd.5 (osd.5) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:05.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:03.350080+0000 osd.5 (osd.5) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:03.700640+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:40:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:03.701601+0000 osd.5 (osd.5) 44 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:40:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:03.731088+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-13T00:40:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:04.181454+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:40:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:04.185120+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:40:05.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:40:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:40:06.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:04.187893+0000 mgr.y (mgr.14544) 99 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:06.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:05.003190+0000 osd.3 (osd.3) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:05.003242+0000 osd.3 (osd.3) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:05.133970+0000 osd.3 (osd.3) 57 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:40:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:05.136931+0000 osd.3 (osd.3) 58 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:40:06.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:04.187893+0000 mgr.y (mgr.14544) 99 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:06.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:05.003190+0000 osd.3 (osd.3) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:06.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:05.003242+0000 osd.3 (osd.3) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:06.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:05.133970+0000 osd.3 (osd.3) 57 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:40:06.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:05.136931+0000 osd.3 (osd.3) 58 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:40:06.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:04.187893+0000 mgr.y (mgr.14544) 99 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:06.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:05.003190+0000 osd.3 (osd.3) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:06.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:05.003242+0000 osd.3 (osd.3) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:06.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:05.133970+0000 osd.3 (osd.3) 57 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:40:06.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:05.136931+0000 osd.3 (osd.3) 58 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:40:06.477 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:40:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:40:06] "GET /metrics HTTP/1.1" 200 172182 "" "Prometheus/2.33.4" 2024-02-13T00:40:06.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.758+0000 7f0738c47700 1 -- 172.21.15.162:0/321792600 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340fdc50 msgr2=0x7f07340fe030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:06.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.758+0000 7f0738c47700 1 --2- 172.21.15.162:0/321792600 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340fdc50 0x7f07340fe030 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f07280047b0 tx=0x7f0728009cb0).stop 2024-02-13T00:40:06.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.758+0000 7f0738c47700 1 -- 172.21.15.162:0/321792600 shutdown_connections 2024-02-13T00:40:06.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.758+0000 7f0738c47700 1 --2- 172.21.15.162:0/321792600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07340f84c0 0x7f0734104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:06.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.758+0000 7f0738c47700 1 --2- 172.21.15.162:0/321792600 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340fdc50 0x7f07340fe030 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.758+0000 7f0738c47700 1 --2- 172.21.15.162:0/321792600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe5b0 0x7f07340f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.758+0000 7f0738c47700 1 -- 172.21.15.162:0/321792600 >> 172.21.15.162:0/321792600 conn(0x7f07340f38a0 msgr2=0x7f07340f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.758+0000 7f0738c47700 1 -- 172.21.15.162:0/321792600 shutdown_connections 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.758+0000 7f0738c47700 1 -- 172.21.15.162:0/321792600 wait complete. 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.758+0000 7f0738c47700 1 librados: starting msgr at 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.758+0000 7f0738c47700 1 librados: starting objecter 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0738c47700 1 Processor -- start 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0738c47700 1 -- start start 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0738c47700 1 librados: setting wanted keys 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0738c47700 1 librados: calling monclient init 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0738c47700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07340f84c0 0x7f073406b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0738c47700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340fdc50 0x7f073406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0738c47700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe5b0 0x7f073406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:06.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0738c47700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0734116670 con 0x7f07340fe5b0 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0738c47700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07341167e0 con 0x7f07340f84c0 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0738c47700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07341169a0 con 0x7f07340fdc50 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0731d9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340fdc50 0x7f073406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0732d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe5b0 0x7f073406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0731d9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340fdc50 0x7f073406b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:60274/0 (socket says 172.21.15.162:60274) 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0731d9b700 1 -- 172.21.15.162:0/3108207980 learned_addr learned my addr 172.21.15.162:0/3108207980 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f073259c700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07340f84c0 0x7f073406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0731d9b700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07340f84c0 msgr2=0x7f073406b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0731d9b700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07340f84c0 0x7f073406b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0731d9b700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe5b0 msgr2=0x7f073406bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0731d9b700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe5b0 0x7f073406bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0731d9b700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0724005040 con 0x7f07340fdc50 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f073259c700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07340f84c0 0x7f073406b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:40:06.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe5b0 0x7f073406bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f0731d9b700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340fdc50 0x7f073406b660 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f0724009400 tx=0x7f0724009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f07237fe700 10 client.?.objecter ms_handle_connect 0x7f07340fdc50 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f07237fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.759+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724007a40 con 0x7f07340fdc50 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.760+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0728005040 con 0x7f07340fdc50 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.760+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f072400f040 con 0x7f07340fdc50 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.760+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724014aa0 con 0x7f07340fdc50 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.760+0000 7f0738c47700 10 client.24674.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.760+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0734116fe0 con 0x7f07340fdc50 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.760+0000 7f0738c47700 1 librados: init done 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.760+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07341071b0 con 0x7f07340fdc50 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.761+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f072400e630 con 0x7f07340fdc50 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.761+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> v2:172.21.15.162:6800/2866534053 conn(0x7f071c062c90 0x7f071c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:06.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.761+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(89..89 src has 30..89) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f072407c670 con 0x7f07340fdc50 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.761+0000 7f07237fe700 10 client.24674.objecter ms_dispatch 0x7f0734001550 osd_map(89..89 src has 30..89) v4 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.761+0000 7f07237fe700 3 client.24674.objecter handle_osd_map got epochs [89,89] > 0 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.761+0000 7f07237fe700 3 client.24674.objecter handle_osd_map decoding full epoch 89 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.761+0000 7f073259c700 1 --2- 172.21.15.162:0/3108207980 >> v2:172.21.15.162:6800/2866534053 conn(0x7f071c062c90 0x7f071c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.761+0000 7f07237fe700 20 client.24674.objecter dump_active .. 0 homeless 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.761+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1267 (secure 0 0 0) 0x7f0724083090 con 0x7f07340fdc50 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f0738c47700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe9f0 0x7f07340fedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073406c650 con 0x7f07340fe9f0 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f07237fe700 10 client.24674.objecter ms_handle_connect 0x7f071c062c90 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f073259c700 1 --2- 172.21.15.162:0/3108207980 >> v2:172.21.15.162:6800/2866534053 conn(0x7f071c062c90 0x7f071c065150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0728009da0 tx=0x7f0728007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe9f0 0x7f07340fedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f0732d9d700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f073405fac0 con 0x7f07340fe9f0 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe9f0 0x7f07340fedd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f072c010fa0 tx=0x7f072c009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:06.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f07237fe700 10 client.24674.objecter ms_handle_connect 0x7f07340fe9f0 2024-02-13T00:40:06.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f07237fe700 10 client.24674.objecter resend_mon_ops 2024-02-13T00:40:06.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c00ed60 con 0x7f07340fe9f0 2024-02-13T00:40:06.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.0 v2:172.21.15.162:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+8167 (secure 0 0 0) 0x7f072c00f040 con 0x7f07340fe9f0 2024-02-13T00:40:06.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe9f0 msgr2=0x7f07340fedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:06.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe9f0 0x7f07340fedd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f072c010fa0 tx=0x7f072c009040).stop 2024-02-13T00:40:06.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe9f0 msgr2=0x7f07340fedd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-13T00:40:06.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.762+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340fe9f0 0x7f07340fedd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:06.799 INFO:teuthology.orchestra.run.smithi162.stdout:mon.a: {} 2024-02-13T00:40:06.799 INFO:teuthology.orchestra.run.smithi162.stdout:mon.b: {} 2024-02-13T00:40:06.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.769+0000 7f0738c47700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340ff2c0 0x7f073411b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:06.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.769+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07340ff6a0 con 0x7f07340ff2c0 2024-02-13T00:40:06.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.769+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340ff2c0 0x7f073411b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.769+0000 7f0732d9d700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f07340ff830 con 0x7f07340ff2c0 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.769+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340ff2c0 0x7f073411b1f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f072c017000 tx=0x7f072c018000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.769+0000 7f07237fe700 10 client.24674.objecter ms_handle_connect 0x7f07340ff2c0 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.769+0000 7f07237fe700 10 client.24674.objecter resend_mon_ops 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.769+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c014e10 con 0x7f07340ff2c0 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.771+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.0 v2:172.21.15.162:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '0' (not observed, change may require restart) ) v1 ==== 83+0+3 (secure 0 0 0) 0x7f072c00eb50 con 0x7f07340ff2c0 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.771+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340ff2c0 msgr2=0x7f073411b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.771+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340ff2c0 0x7f073411b1f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f072c017000 tx=0x7f072c018000).stop 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.771+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340ff2c0 msgr2=0x7f073411b1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.771+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07340ff2c0 0x7f073411b1f0 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0x7f072c017000 tx=0x7f072c018000).stop 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.771+0000 7f0738c47700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0734027410 0x7f0734027820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.771+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0734027d10 con 0x7f0734027410 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.771+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0734027410 0x7f0734027820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:06.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.771+0000 7f0732d9d700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0734027e80 con 0x7f0734027410 2024-02-13T00:40:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.772+0000 7f07237fe700 10 client.24674.objecter ms_handle_connect 0x7f0734027410 2024-02-13T00:40:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.772+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0734027410 0x7f0734027820 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f073406c8a0 tx=0x7f072c021000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.772+0000 7f07237fe700 10 client.24674.objecter resend_mon_ops 2024-02-13T00:40:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.772+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c02c480 con 0x7f0734027410 2024-02-13T00:40:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.772+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+8167 (secure 0 0 0) 0x7f072c014c00 con 0x7f0734027410 2024-02-13T00:40:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.772+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0734027410 msgr2=0x7f0734027820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.772+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0734027410 0x7f0734027820 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f073406c8a0 tx=0x7f072c021000).stop 2024-02-13T00:40:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.772+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0734027410 msgr2=0x7f0734027820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-13T00:40:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.772+0000 7f0732d9d700 1 -- 172.21.15.162:0/3108207980 reap_dead start 2024-02-13T00:40:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.772+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0734027410 0x7f0734027820 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.778+0000 7f0738c47700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f073406b230 0x7f073406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.778+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07340280f0 con 0x7f073406b230 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.779+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f073406b230 0x7f073406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.779+0000 7f0732d9d700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f0734028230 con 0x7f073406b230 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.779+0000 7f07237fe700 10 client.24674.objecter ms_handle_connect 0x7f073406b230 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.779+0000 7f07237fe700 10 client.24674.objecter resend_mon_ops 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.779+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f073406b230 0x7f073406bb50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f072c010fa0 tx=0x7f072c02c000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.779+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c02c7f0 con 0x7f073406b230 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '0' (not observed, change may require restart) ) v1 ==== 83+0+3 (secure 0 0 0) 0x7f072c016b40 con 0x7f073406b230 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f073406b230 msgr2=0x7f073406bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f073406b230 0x7f073406bb50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f072c010fa0 tx=0x7f072c02c000).stop 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f073406b230 msgr2=0x7f073406bb50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f073406b230 0x7f073406bb50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f0738c47700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340284f0 0x7f073411d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0734120340 con 0x7f07340284f0 2024-02-13T00:40:06.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340284f0 0x7f073411d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f0732d9d700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0734120480 con 0x7f07340284f0 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340284f0 0x7f073411d6f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f072c000f80 tx=0x7f072c032000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f07237fe700 10 client.24674.objecter ms_handle_connect 0x7f07340284f0 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f07237fe700 10 client.24674.objecter resend_mon_ops 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.781+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c0164e0 con 0x7f07340284f0 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.782+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.2 v2:172.21.15.162:3301/0 2 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+8167 (secure 0 0 0) 0x7f072c014740 con 0x7f07340284f0 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.782+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340284f0 msgr2=0x7f073411d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.782+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340284f0 0x7f073411d6f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f072c000f80 tx=0x7f072c032000).stop 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.782+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340284f0 msgr2=0x7f073411d6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.782+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340284f0 0x7f073411d6f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.788+0000 7f0738c47700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07341207a0 0x7f073411fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.788+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0734120140 con 0x7f07341207a0 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.788+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07341207a0 0x7f073411fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:06.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.788+0000 7f0732d9d700 1 -- 172.21.15.162:0/3108207980 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f0734122160 con 0x7f07341207a0 2024-02-13T00:40:06.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.788+0000 7f07237fe700 10 client.24674.objecter ms_handle_connect 0x7f07341207a0 2024-02-13T00:40:06.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.788+0000 7f07237fe700 10 client.24674.objecter resend_mon_ops 2024-02-13T00:40:06.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.788+0000 7f0732d9d700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07341207a0 0x7f073411fbf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f072c010fa0 tx=0x7f072c038040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:06.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.788+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c049430 con 0x7f07341207a0 2024-02-13T00:40:06.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 <== mon.2 v2:172.21.15.162:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '0' (not observed, change may require restart) ) v1 ==== 83+0+3 (secure 0 0 0) 0x7f072c049430 con 0x7f07341207a0 2024-02-13T00:40:06.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07341207a0 msgr2=0x7f073411fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:06.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07341207a0 0x7f073411fbf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f072c010fa0 tx=0x7f072c038040).stop 2024-02-13T00:40:06.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f07237fe700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07341207a0 msgr2=0x7f073411fbf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-13T00:40:06.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f07237fe700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07341207a0 0x7f073411fbf0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:06.804 INFO:teuthology.orchestra.run.smithi162.stdout:mon.c: {} 2024-02-13T00:40:06.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 10 librados: watch_flush enter 2024-02-13T00:40:06.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 10 librados: watch_flush exit 2024-02-13T00:40:06.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 20 client.24674.objecter shutdown clearing up homeless session... 2024-02-13T00:40:06.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 10 client.24674.objecter successfully canceled tick 2024-02-13T00:40:06.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 >> v2:172.21.15.162:6800/2866534053 conn(0x7f071c062c90 msgr2=0x7f071c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:06.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 1 --2- 172.21.15.162:0/3108207980 >> v2:172.21.15.162:6800/2866534053 conn(0x7f071c062c90 0x7f071c065150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0728009da0 tx=0x7f0728007f90).stop 2024-02-13T00:40:06.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340fdc50 msgr2=0x7f073406b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:06.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 1 --2- 172.21.15.162:0/3108207980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07340fdc50 0x7f073406b660 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f0724009400 tx=0x7f0724009cb0).stop 2024-02-13T00:40:06.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0732d9d700 1 -- 172.21.15.162:0/3108207980 reap_dead start 2024-02-13T00:40:06.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 shutdown_connections 2024-02-13T00:40:06.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 >> 172.21.15.162:0/3108207980 conn(0x7f07340f38a0 msgr2=0x7f07340f53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:06.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 shutdown_connections 2024-02-13T00:40:06.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 1 -- 172.21.15.162:0/3108207980 wait complete. 2024-02-13T00:40:06.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:06.790+0000 7f0738c47700 1 librados: shutdown 2024-02-13T00:40:06.806 INFO:teuthology.orchestra.run.smithi162.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-13T00:40:06.806 INFO:teuthology.orchestra.run.smithi162.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-13T00:40:06.806 INFO:teuthology.orchestra.run.smithi162.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-13T00:40:07.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:05.273842+0000 mgr.y (mgr.14544) 100 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:05.738778+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-13T00:40:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:06.165631+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:40:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:06.169083+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:40:07.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:05.273842+0000 mgr.y (mgr.14544) 100 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:07.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:05.738778+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-13T00:40:07.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:06.165631+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:40:07.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:06.169083+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:40:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:05.273842+0000 mgr.y (mgr.14544) 100 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:05.738778+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-13T00:40:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:06.165631+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:40:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:06.169083+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:40:07.600 INFO:teuthology.run_tasks:Running task workunit... 2024-02-13T00:40:07.612 INFO:tasks.workunit:Pulling workunits from ref 84e714279aa684636ac645daf0a1a85c2094cfb3 2024-02-13T00:40:07.612 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-13T00:40:07.613 DEBUG:teuthology.orchestra.run.smithi162:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-13T00:40:07.613 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-02-13T00:40:07.613 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:40:07.613 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:40:07.614 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-13T00:40:07.614 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-02-13T00:40:07.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-13T00:40:07.614 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-02-13T00:40:07.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:07.615 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-02-13T00:40:07.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:40:07.615 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-02-13T00:40:07.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-13T00:40:07.616 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-02-13T00:40:07.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:40:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T00:40:07.643 INFO:teuthology.orchestra.run.smithi162.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-02-13T00:40:07.643 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-02-13T00:40:07.656 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:07.681 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-02-13T00:40:07.681 DEBUG:teuthology.orchestra.run.smithi162:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-02-13T00:40:07.687 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:07.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:07.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:07.707 INFO:tasks.workunit:timeout=3h 2024-02-13T00:40:07.707 INFO:tasks.workunit:cleanup=True 2024-02-13T00:40:07.707 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ljflores/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 84e714279aa684636ac645daf0a1a85c2094cfb3 2024-02-13T00:40:07.773 INFO:tasks.workunit.client.0.smithi162.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-13T00:40:07.792 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:07.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:06.023105+0000 osd.1 (osd.1) 49 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:40:07.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:06.026314+0000 osd.1 (osd.1) 50 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:40:07.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:06.188313+0000 mgr.y (mgr.14544) 101 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:40:07.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:06.763196+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? 172.21.15.162:0/3108207980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T00:40:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:06.023105+0000 osd.1 (osd.1) 49 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:40:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:06.026314+0000 osd.1 (osd.1) 50 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:40:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:06.188313+0000 mgr.y (mgr.14544) 101 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:40:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:06.763196+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? 172.21.15.162:0/3108207980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T00:40:07.831 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:07.854 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:07.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:07.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:07.976 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:07.978 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:07.987 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:07.988 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:08.001 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:08.007 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:08.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:06.023105+0000 osd.1 (osd.1) 49 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:40:08.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:06.026314+0000 osd.1 (osd.1) 50 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:40:08.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:06.188313+0000 mgr.y (mgr.14544) 101 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:40:08.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:06.763196+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? 172.21.15.162:0/3108207980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T00:40:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:08.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:08.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:08.138 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:08.887 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:08.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:08.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:09.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:07.085019+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.1a scrub starts 2024-02-13T00:40:09.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:07.088047+0000 osd.7 (osd.7) 46 : cluster [DBG] 5.1a scrub ok 2024-02-13T00:40:09.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:07.267225+0000 osd.2 (osd.2) 33 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:40:09.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:07.270136+0000 osd.2 (osd.2) 34 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:40:09.020 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:40:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:40:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:40:09.023 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:09.045 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:09.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:09.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:09.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:07.085019+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.1a scrub starts 2024-02-13T00:40:09.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:07.088047+0000 osd.7 (osd.7) 46 : cluster [DBG] 5.1a scrub ok 2024-02-13T00:40:09.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:07.267225+0000 osd.2 (osd.2) 33 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:40:09.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:07.270136+0000 osd.2 (osd.2) 34 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:40:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:07.085019+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.1a scrub starts 2024-02-13T00:40:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:07.088047+0000 osd.7 (osd.7) 46 : cluster [DBG] 5.1a scrub ok 2024-02-13T00:40:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:07.267225+0000 osd.2 (osd.2) 33 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:40:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:07.270136+0000 osd.2 (osd.2) 34 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:40:09.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:09.178 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:09.198 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:09.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:09.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:09.330 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:09.780 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:09.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:09.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:09.917 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:09.939 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:09.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:09.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:10.015 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:08.188692+0000 mgr.y (mgr.14544) 102 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 789 B/s rd, 0 op/s 2024-02-13T00:40:10.015 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:08.576742+0000 osd.5 (osd.5) 45 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:40:10.015 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:08.577659+0000 osd.5 (osd.5) 46 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:40:10.015 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:09.096388+0000 osd.3 (osd.3) 59 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:40:10.015 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:09.099455+0000 osd.3 (osd.3) 60 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:40:10.016 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:08.188692+0000 mgr.y (mgr.14544) 102 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 789 B/s rd, 0 op/s 2024-02-13T00:40:10.016 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:08.576742+0000 osd.5 (osd.5) 45 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:40:10.016 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:08.577659+0000 osd.5 (osd.5) 46 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:40:10.016 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:09.096388+0000 osd.3 (osd.3) 59 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:40:10.016 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:09.099455+0000 osd.3 (osd.3) 60 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:40:10.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:10.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:08.188692+0000 mgr.y (mgr.14544) 102 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 789 B/s rd, 0 op/s 2024-02-13T00:40:10.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:08.576742+0000 osd.5 (osd.5) 45 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:40:10.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:08.577659+0000 osd.5 (osd.5) 46 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:40:10.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:09.096388+0000 osd.3 (osd.3) 59 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:40:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:09.099455+0000 osd.3 (osd.3) 60 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:40:10.069 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:10.122 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:10.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:10.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:10.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:10.261 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:10.282 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:10.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:10.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:10.417 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:10.438 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:10.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:10.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:10.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:10.570 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:10.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:08.943282+0000 osd.1 (osd.1) 51 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:40:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:08.946383+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:40:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:10.203601+0000 osd.4 (osd.4) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:10.203658+0000 osd.4 (osd.4) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:08.943282+0000 osd.1 (osd.1) 51 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:40:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:08.946383+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:40:10.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:10.203601+0000 osd.4 (osd.4) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:10.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:10.203658+0000 osd.4 (osd.4) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:11.023 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:11.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:11.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:08.943282+0000 osd.1 (osd.1) 51 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:40:11.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:08.946383+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:40:11.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:10.203601+0000 osd.4 (osd.4) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:11.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:10.203658+0000 osd.4 (osd.4) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:11.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:11.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:11.389 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:11.421 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:11.442 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:11.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:11.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:11.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:11.574 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:11.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.625+0000 7f7d84832700 1 -- 172.21.15.162:0/2285536762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d800fa440 msgr2=0x7f7d800f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:11.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.625+0000 7f7d84832700 1 --2- 172.21.15.162:0/2285536762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d800fa440 0x7f7d800f7e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f7d74004870 tx=0x7f7d74009cb0).stop 2024-02-13T00:40:11.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.625+0000 7f7d84832700 1 -- 172.21.15.162:0/2285536762 shutdown_connections 2024-02-13T00:40:11.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.625+0000 7f7d84832700 1 --2- 172.21.15.162:0/2285536762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d800f8500 0x7f7d80104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:11.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.625+0000 7f7d84832700 1 --2- 172.21.15.162:0/2285536762 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d800f9b70 0x7f7d800f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:11.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.625+0000 7f7d84832700 1 --2- 172.21.15.162:0/2285536762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d800fa440 0x7f7d800f7e20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:11.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.625+0000 7f7d84832700 1 -- 172.21.15.162:0/2285536762 >> 172.21.15.162:0/2285536762 conn(0x7f7d800f38c0 msgr2=0x7f7d800f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:11.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.626+0000 7f7d84832700 1 -- 172.21.15.162:0/2285536762 shutdown_connections 2024-02-13T00:40:11.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.626+0000 7f7d84832700 1 -- 172.21.15.162:0/2285536762 wait complete. 2024-02-13T00:40:11.631 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:11.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.626+0000 7f7d84832700 1 librados: starting msgr at 2024-02-13T00:40:11.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.626+0000 7f7d84832700 1 librados: starting objecter 2024-02-13T00:40:11.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.626+0000 7f7d84832700 1 Processor -- start 2024-02-13T00:40:11.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d84832700 1 -- start start 2024-02-13T00:40:11.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d84832700 1 librados: setting wanted keys 2024-02-13T00:40:11.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d84832700 1 librados: calling monclient init 2024-02-13T00:40:11.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d84832700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d800f8500 0x7f7d8006b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:11.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d84832700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d800f9b70 0x7f7d8006b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:11.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d84832700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d800fa440 0x7f7d8006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:11.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d84832700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80116780 con 0x7f7d800f8500 2024-02-13T00:40:11.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d84832700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d801168f0 con 0x7f7d800f9b70 2024-02-13T00:40:11.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d84832700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80116ab0 con 0x7f7d800fa440 2024-02-13T00:40:11.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d7ed9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d800fa440 0x7f7d8006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:11.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d7ed9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d800fa440 0x7f7d8006bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:60312/0 (socket says 172.21.15.162:60312) 2024-02-13T00:40:11.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d7ed9d700 1 -- 172.21.15.162:0/2518540935 learned_addr learned my addr 172.21.15.162:0/2518540935 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:11.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d7dd9b700 1 --2- 172.21.15.162:0/2518540935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d800f9b70 0x7f7d8006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:11.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d7e59c700 1 --2- 172.21.15.162:0/2518540935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d800f8500 0x7f7d8006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:11.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d7dd9b700 1 -- 172.21.15.162:0/2518540935 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d800fa440 msgr2=0x7f7d8006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:11.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d7dd9b700 1 --2- 172.21.15.162:0/2518540935 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d800fa440 0x7f7d8006bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:11.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d7dd9b700 1 -- 172.21.15.162:0/2518540935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d800f8500 msgr2=0x7f7d8006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:11.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d7dd9b700 1 --2- 172.21.15.162:0/2518540935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d800f8500 0x7f7d8006b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:11.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d7dd9b700 1 -- 172.21.15.162:0/2518540935 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d74005040 con 0x7f7d800f9b70 2024-02-13T00:40:11.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.627+0000 7f7d7ed9d700 1 --2- 172.21.15.162:0/2518540935 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d800fa440 0x7f7d8006bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:40:11.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.628+0000 7f7d7dd9b700 1 --2- 172.21.15.162:0/2518540935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d800f9b70 0x7f7d8006b660 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f7d740043d0 tx=0x7f7d74009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:11.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.628+0000 7f7d6f7fe700 10 client.?.objecter ms_handle_connect 0x7f7d800f9b70 2024-02-13T00:40:11.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.628+0000 7f7d6f7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:11.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.628+0000 7f7d6f7fe700 1 -- 172.21.15.162:0/2518540935 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7400e030 con 0x7f7d800f9b70 2024-02-13T00:40:11.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.628+0000 7f7d84832700 1 -- 172.21.15.162:0/2518540935 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d80116d30 con 0x7f7d800f9b70 2024-02-13T00:40:11.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.628+0000 7f7d6f7fe700 1 -- 172.21.15.162:0/2518540935 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d74013d60 con 0x7f7d800f9b70 2024-02-13T00:40:11.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.628+0000 7f7d6f7fe700 1 -- 172.21.15.162:0/2518540935 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d74018070 con 0x7f7d800f9b70 2024-02-13T00:40:11.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.628+0000 7f7d84832700 10 client.24616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:11.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.628+0000 7f7d84832700 1 -- 172.21.15.162:0/2518540935 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d80117130 con 0x7f7d800f9b70 2024-02-13T00:40:11.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.628+0000 7f7d84832700 1 librados: init done 2024-02-13T00:40:11.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.629+0000 7f7d84832700 1 -- 172.21.15.162:0/2518540935 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d800475e0 con 0x7f7d800f9b70 2024-02-13T00:40:11.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:11.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:11.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.630+0000 7f7d6f7fe700 1 -- 172.21.15.162:0/2518540935 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f7d74009070 con 0x7f7d800f9b70 2024-02-13T00:40:11.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.631+0000 7f7d6f7fe700 1 --2- 172.21.15.162:0/2518540935 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7d68062c90 0x7f7d68065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:11.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.631+0000 7f7d6f7fe700 1 -- 172.21.15.162:0/2518540935 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(90..90 src has 30..90) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f7d7407dd10 con 0x7f7d800f9b70 2024-02-13T00:40:11.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.631+0000 7f7d6f7fe700 10 client.24616.objecter ms_dispatch 0x7f7d80001550 osd_map(90..90 src has 30..90) v4 2024-02-13T00:40:11.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.631+0000 7f7d6f7fe700 3 client.24616.objecter handle_osd_map got epochs [90,90] > 0 2024-02-13T00:40:11.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.631+0000 7f7d6f7fe700 3 client.24616.objecter handle_osd_map decoding full epoch 90 2024-02-13T00:40:11.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.631+0000 7f7d7e59c700 1 --2- 172.21.15.162:0/2518540935 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7d68062c90 0x7f7d68065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:11.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.631+0000 7f7d6f7fe700 20 client.24616.objecter dump_active .. 0 homeless 2024-02-13T00:40:11.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.631+0000 7f7d7e59c700 1 --2- 172.21.15.162:0/2518540935 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7d68062c90 0x7f7d68065150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f7d70004e60 tx=0x7f7d7000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:11.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.631+0000 7f7d6f7fe700 10 client.24616.objecter ms_handle_connect 0x7f7d68062c90 2024-02-13T00:40:11.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.635+0000 7f7d6f7fe700 1 -- 172.21.15.162:0/2518540935 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7d74083020 con 0x7f7d800f9b70 2024-02-13T00:40:11.741 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:11.773 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:11.793 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:11.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:11.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:10.189303+0000 mgr.y (mgr.14544) 103 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:11.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:10.634381+0000 osd.5 (osd.5) 47 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:40:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:10.635474+0000 osd.5 (osd.5) 48 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:40:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:10.761119+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-13T00:40:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:10.189303+0000 mgr.y (mgr.14544) 103 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:10.634381+0000 osd.5 (osd.5) 47 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:40:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:10.635474+0000 osd.5 (osd.5) 48 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:40:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:10.761119+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-13T00:40:11.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:11.830+0000 7f7d84832700 1 -- 172.21.15.162:0/2518540935 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f7d8006c650 con 0x7f7d800f9b70 2024-02-13T00:40:11.898 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:11.926 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:12.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:10.189303+0000 mgr.y (mgr.14544) 103 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:12.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:10.634381+0000 osd.5 (osd.5) 47 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:40:12.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:10.635474+0000 osd.5 (osd.5) 48 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:40:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:10.761119+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-13T00:40:12.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.769+0000 7f7d6f7fe700 1 -- 172.21.15.162:0/2518540935 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v91) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7d74051940 con 0x7f7d800f9b70 2024-02-13T00:40:12.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.772+0000 7f7d84832700 10 librados: watch_flush enter 2024-02-13T00:40:12.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.772+0000 7f7d84832700 10 librados: watch_flush exit 2024-02-13T00:40:12.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.772+0000 7f7d84832700 20 client.24616.objecter shutdown clearing up homeless session... 2024-02-13T00:40:12.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.772+0000 7f7d84832700 10 client.24616.objecter successfully canceled tick 2024-02-13T00:40:12.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 -- 172.21.15.162:0/2518540935 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7d68062c90 msgr2=0x7f7d68065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:12.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 --2- 172.21.15.162:0/2518540935 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7d68062c90 0x7f7d68065150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f7d70004e60 tx=0x7f7d7000b040).stop 2024-02-13T00:40:12.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 -- 172.21.15.162:0/2518540935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d800f9b70 msgr2=0x7f7d8006b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:12.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 --2- 172.21.15.162:0/2518540935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d800f9b70 0x7f7d8006b660 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f7d740043d0 tx=0x7f7d74009cb0).stop 2024-02-13T00:40:12.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 -- 172.21.15.162:0/2518540935 shutdown_connections 2024-02-13T00:40:12.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 --2- 172.21.15.162:0/2518540935 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7d68062c90 0x7f7d68065150 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:12.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 --2- 172.21.15.162:0/2518540935 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d800fa440 0x7f7d8006bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:12.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 --2- 172.21.15.162:0/2518540935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d800f8500 0x7f7d8006b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:12.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 --2- 172.21.15.162:0/2518540935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d800f9b70 0x7f7d8006b660 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:12.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 -- 172.21.15.162:0/2518540935 >> 172.21.15.162:0/2518540935 conn(0x7f7d800f38c0 msgr2=0x7f7d800f43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:12.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 -- 172.21.15.162:0/2518540935 shutdown_connections 2024-02-13T00:40:12.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 -- 172.21.15.162:0/2518540935 wait complete. 2024-02-13T00:40:12.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:12.773+0000 7f7d84832700 1 librados: shutdown 2024-02-13T00:40:12.777 INFO:teuthology.orchestra.run.smithi162.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-13T00:40:13.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:11.831977+0000 mon.b (mon.1) 26 : audit [INF] from='client.? 172.21.15.162:0/2518540935' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:40:13.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:11.832636+0000 mon.a (mon.0) 720 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:40:13.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:11.831977+0000 mon.b (mon.1) 26 : audit [INF] from='client.? 172.21.15.162:0/2518540935' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:40:13.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:11.832636+0000 mon.a (mon.0) 720 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:40:13.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:11.831977+0000 mon.b (mon.1) 26 : audit [INF] from='client.? 172.21.15.162:0/2518540935' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:40:13.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:11.832636+0000 mon.a (mon.0) 720 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:40:13.204 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:13.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:13.309 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:13.338 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:13.411 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:13.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:13.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:13.546 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:13.567 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:13.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:13.671 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:13.700 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:13.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:12.044350+0000 osd.1 (osd.1) 53 : cluster [DBG] 4.4 scrub starts 2024-02-13T00:40:13.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:12.047860+0000 osd.1 (osd.1) 54 : cluster [DBG] 4.4 scrub ok 2024-02-13T00:40:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:12.189611+0000 mgr.y (mgr.14544) 104 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:12.769650+0000 mon.a (mon.0) 721 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:40:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:12.769724+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-13T00:40:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:13.138737+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.18 deep-scrub starts 2024-02-13T00:40:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:13.144433+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.18 deep-scrub ok 2024-02-13T00:40:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:13.523815+0000 mon.a (mon.0) 723 : audit [DBG] from='client.? 172.21.15.162:0/4030630483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:40:13.900 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:13.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:14.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:14.030 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:12.044350+0000 osd.1 (osd.1) 53 : cluster [DBG] 4.4 scrub starts 2024-02-13T00:40:14.030 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:12.047860+0000 osd.1 (osd.1) 54 : cluster [DBG] 4.4 scrub ok 2024-02-13T00:40:14.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:12.189611+0000 mgr.y (mgr.14544) 104 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:14.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:12.769650+0000 mon.a (mon.0) 721 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:40:14.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:12.769724+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-13T00:40:14.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:13.138737+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.18 deep-scrub starts 2024-02-13T00:40:14.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:13.144433+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.18 deep-scrub ok 2024-02-13T00:40:14.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:13.523815+0000 mon.a (mon.0) 723 : audit [DBG] from='client.? 172.21.15.162:0/4030630483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:40:14.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:12.044350+0000 osd.1 (osd.1) 53 : cluster [DBG] 4.4 scrub starts 2024-02-13T00:40:14.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:12.047860+0000 osd.1 (osd.1) 54 : cluster [DBG] 4.4 scrub ok 2024-02-13T00:40:14.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:12.189611+0000 mgr.y (mgr.14544) 104 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:14.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:12.769650+0000 mon.a (mon.0) 721 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:40:14.032 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:12.769724+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-13T00:40:14.032 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:13.138737+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.18 deep-scrub starts 2024-02-13T00:40:14.032 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:13.144433+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.18 deep-scrub ok 2024-02-13T00:40:14.032 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:13.523815+0000 mon.a (mon.0) 723 : audit [DBG] from='client.? 172.21.15.162:0/4030630483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:40:14.032 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:14.381 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:14.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:14.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:14.508 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:40:14.508 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:40:14.515 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:15.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:13.675898+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:40:15.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:13.678803+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:40:15.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:14.012320+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.c scrub starts 2024-02-13T00:40:15.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:14.015543+0000 osd.3 (osd.3) 62 : cluster [DBG] 6.c scrub ok 2024-02-13T00:40:15.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:13.675898+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:40:15.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:13.678803+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:40:15.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:14.012320+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.c scrub starts 2024-02-13T00:40:15.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:14.015543+0000 osd.3 (osd.3) 62 : cluster [DBG] 6.c scrub ok 2024-02-13T00:40:15.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:13.675898+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:40:15.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:13.678803+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:40:15.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:14.012320+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.c scrub starts 2024-02-13T00:40:15.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:14.015543+0000 osd.3 (osd.3) 62 : cluster [DBG] 6.c scrub ok 2024-02-13T00:40:15.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-13T00:40:15.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:40:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:40:15.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.595+0000 7f61a17b7700 1 -- 172.21.15.162:0/901643312 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f619c0f61e0 msgr2=0x7f619c0f6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:15.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.595+0000 7f61a17b7700 1 --2- 172.21.15.162:0/901643312 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f619c0f61e0 0x7f619c0f6660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f61900048a0 tx=0x7f6190009cb0).stop 2024-02-13T00:40:15.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.595+0000 7f61a17b7700 1 -- 172.21.15.162:0/901643312 shutdown_connections 2024-02-13T00:40:15.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.595+0000 7f61a17b7700 1 --2- 172.21.15.162:0/901643312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f619c0f6c30 0x7f619c106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:40:15.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.595+0000 7f61a17b7700 1 --2- 172.21.15.162:0/901643312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f619c0f8a20 0x7f619c0f8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:15.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.595+0000 7f61a17b7700 1 --2- 172.21.15.162:0/901643312 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f619c0f61e0 0x7f619c0f6660 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:15.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.595+0000 7f61a17b7700 1 -- 172.21.15.162:0/901643312 >> 172.21.15.162:0/901643312 conn(0x7f619c070080 msgr2=0x7f619c0f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:15.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.595+0000 7f61a17b7700 1 -- 172.21.15.162:0/901643312 shutdown_connections 2024-02-13T00:40:15.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.595+0000 7f61a17b7700 1 -- 172.21.15.162:0/901643312 wait complete. 2024-02-13T00:40:15.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f61a17b7700 1 librados: starting msgr at 2024-02-13T00:40:15.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f61a17b7700 1 librados: starting objecter 2024-02-13T00:40:15.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f61a17b7700 1 Processor -- start 2024-02-13T00:40:15.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f61a17b7700 1 -- start start 2024-02-13T00:40:15.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f61a17b7700 1 librados: setting wanted keys 2024-02-13T00:40:15.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f61a17b7700 1 librados: calling monclient init 2024-02-13T00:40:15.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f61a17b7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f619c0f61e0 0x7f619c110560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:15.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f61a17b7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f619c0f6c30 0x7f619c110a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:15.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f61a17b7700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f619c0f8a20 0x7f619c10a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:15.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f61a17b7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c10ac30 con 0x7f619c0f61e0 2024-02-13T00:40:15.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f61a17b7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c10ada0 con 0x7f619c0f6c30 2024-02-13T00:40:15.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f61a17b7700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c10af60 con 0x7f619c0f8a20 2024-02-13T00:40:15.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f619b7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f619c0f8a20 0x7f619c10a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f619affd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f619c0f61e0 0x7f619c110560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f619affd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f619c0f61e0 0x7f619c110560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:56192/0 (socket says 172.21.15.162:56192) 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f619affd700 1 -- 172.21.15.162:0/3774301780 learned_addr learned my addr 172.21.15.162:0/3774301780 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.596+0000 7f619b7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f619c0f8a20 0x7f619c10a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:57036/0 (socket says 172.21.15.162:57036) 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f619a7fc700 1 --2- 172.21.15.162:0/3774301780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f619c0f6c30 0x7f619c110a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f619b7fe700 1 -- 172.21.15.162:0/3774301780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f619c0f6c30 msgr2=0x7f619c110a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f619b7fe700 1 --2- 172.21.15.162:0/3774301780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f619c0f6c30 0x7f619c110a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f619b7fe700 1 -- 172.21.15.162:0/3774301780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f619c0f61e0 msgr2=0x7f619c110560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f619b7fe700 1 --2- 172.21.15.162:0/3774301780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f619c0f61e0 0x7f619c110560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f619b7fe700 1 -- 172.21.15.162:0/3774301780 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c005040 con 0x7f619c0f8a20 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f619affd700 1 --2- 172.21.15.162:0/3774301780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f619c0f61e0 0x7f619c110560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f619b7fe700 1 --2- 172.21.15.162:0/3774301780 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f619c0f8a20 0x7f619c10a700 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f619400bad0 tx=0x7f6194009780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f6183fff700 10 client.?.objecter ms_handle_connect 0x7f619c0f8a20 2024-02-13T00:40:15.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f6183fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:15.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f6183fff700 1 -- 172.21.15.162:0/3774301780 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619400c760 con 0x7f619c0f8a20 2024-02-13T00:40:15.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f6183fff700 1 -- 172.21.15.162:0/3774301780 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f619400cd90 con 0x7f619c0f8a20 2024-02-13T00:40:15.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f61a17b7700 1 -- 172.21.15.162:0/3774301780 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6190005040 con 0x7f619c0f8a20 2024-02-13T00:40:15.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f6183fff700 1 -- 172.21.15.162:0/3774301780 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194014bc0 con 0x7f619c0f8a20 2024-02-13T00:40:15.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f61a17b7700 10 client.24698.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:15.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f61a17b7700 1 -- 172.21.15.162:0/3774301780 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c10b570 con 0x7f619c0f8a20 2024-02-13T00:40:15.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.597+0000 7f61a17b7700 1 librados: init done 2024-02-13T00:40:15.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.599+0000 7f61a17b7700 1 -- 172.21.15.162:0/3774301780 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619c05fb80 con 0x7f619c0f8a20 2024-02-13T00:40:15.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.599+0000 7f6183fff700 1 -- 172.21.15.162:0/3774301780 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6194012020 con 0x7f619c0f8a20 2024-02-13T00:40:15.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.599+0000 7f6183fff700 1 --2- 172.21.15.162:0/3774301780 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6184062c90 0x7f6184065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:15.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.600+0000 7f6183fff700 1 -- 172.21.15.162:0/3774301780 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(91..91 src has 41..91) v4 ==== 6204+0+0 (secure 0 0 0) 0x7f619407c770 con 0x7f619c0f8a20 2024-02-13T00:40:15.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.600+0000 7f6183fff700 10 client.24698.objecter ms_dispatch 0x7f619c001550 osd_map(91..91 src has 41..91) v4 2024-02-13T00:40:15.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.600+0000 7f6183fff700 3 client.24698.objecter handle_osd_map got epochs [91,91] > 0 2024-02-13T00:40:15.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.600+0000 7f6183fff700 3 client.24698.objecter handle_osd_map decoding full epoch 91 2024-02-13T00:40:15.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.600+0000 7f619affd700 1 --2- 172.21.15.162:0/3774301780 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6184062c90 0x7f6184065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:15.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.600+0000 7f6183fff700 20 client.24698.objecter dump_active .. 0 homeless 2024-02-13T00:40:15.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.600+0000 7f6183fff700 10 client.24698.objecter ms_handle_connect 0x7f6184062c90 2024-02-13T00:40:15.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.600+0000 7f619affd700 1 --2- 172.21.15.162:0/3774301780 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6184062c90 0x7f6184065150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f619c0f8490 tx=0x7f618c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:15.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.604+0000 7f6183fff700 1 -- 172.21.15.162:0/3774301780 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6194050460 con 0x7f619c0f8a20 2024-02-13T00:40:15.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:15.810+0000 7f61a17b7700 1 -- 172.21.15.162:0/3774301780 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f619c10b570 con 0x7f619c0f8a20 2024-02-13T00:40:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:14.189997+0000 mgr.y (mgr.14544) 105 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:14.694084+0000 osd.5 (osd.5) 51 : cluster [DBG] 5.1b scrub starts 2024-02-13T00:40:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:14.696987+0000 osd.5 (osd.5) 52 : cluster [DBG] 5.1b scrub ok 2024-02-13T00:40:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:15.065611+0000 osd.6 (osd.6) 31 : cluster [DBG] 4.a deep-scrub starts 2024-02-13T00:40:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:15.074028+0000 osd.6 (osd.6) 32 : cluster [DBG] 4.a deep-scrub ok 2024-02-13T00:40:16.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:15.103994+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:40:16.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:15.106884+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:40:16.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 -- 172.21.15.162:0/2861368607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7e80fc2b0 msgr2=0x7fb7e80fc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 --2- 172.21.15.162:0/2861368607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7e80fc2b0 0x7fb7e80fc690 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fb7d80047b0 tx=0x7fb7d8009cb0).stop 2024-02-13T00:40:16.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 -- 172.21.15.162:0/2861368607 shutdown_connections 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 --2- 172.21.15.162:0/2861368607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7e80f6b20 0x7fb7e8103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 --2- 172.21.15.162:0/2861368607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7e80fc2b0 0x7fb7e80fc690 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 --2- 172.21.15.162:0/2861368607 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7e80fcc10 0x7fb7e80f6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 -- 172.21.15.162:0/2861368607 >> 172.21.15.162:0/2861368607 conn(0x7fb7e8070080 msgr2=0x7fb7e80f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 -- 172.21.15.162:0/2861368607 shutdown_connections 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 -- 172.21.15.162:0/2861368607 wait complete. 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 librados: starting msgr at 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 librados: starting objecter 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 Processor -- start 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 -- start start 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 librados: setting wanted keys 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.519+0000 7fb7ee966700 1 librados: calling monclient init 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7ee966700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7e80f6b20 0x7fb7e8109280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7ee966700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7e80fc2b0 0x7fb7e8109770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7ee966700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7e80fcc10 0x7fb7e8109d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7ee966700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8114470 con 0x7fb7e80fc2b0 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7ee966700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e81145e0 con 0x7fb7e80f6b20 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7ee966700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e81147a0 con 0x7fb7e80fcc10 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e77fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7e80fc2b0 0x7fb7e8109770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e77fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7e80fc2b0 0x7fb7e8109770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:48902/0 (socket says 172.21.15.162:48902) 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e77fe700 1 -- 172.21.15.162:0/4030630483 learned_addr learned my addr 172.21.15.162:0/4030630483 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e7fff700 1 --2- 172.21.15.162:0/4030630483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7e80f6b20 0x7fb7e8109280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7ecf03700 1 --2- 172.21.15.162:0/4030630483 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7e80fcc10 0x7fb7e8109d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e77fe700 1 -- 172.21.15.162:0/4030630483 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7e80fcc10 msgr2=0x7fb7e8109d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e77fe700 1 --2- 172.21.15.162:0/4030630483 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7e80fcc10 0x7fb7e8109d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e77fe700 1 -- 172.21.15.162:0/4030630483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7e80f6b20 msgr2=0x7fb7e8109280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e77fe700 1 --2- 172.21.15.162:0/4030630483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7e80f6b20 0x7fb7e8109280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e77fe700 1 -- 172.21.15.162:0/4030630483 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc005040 con 0x7fb7e80fc2b0 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e77fe700 1 --2- 172.21.15.162:0/4030630483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7e80fc2b0 0x7fb7e8109770 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb7dc009400 tx=0x7fb7dc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e57fa700 10 client.?.objecter ms_handle_connect 0x7fb7e80fc2b0 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e57fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc007a40 con 0x7fb7e80fc2b0 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7dc00f040 con 0x7fb7e80fc2b0 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d8005040 con 0x7fb7e80fc2b0 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.520+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc014a70 con 0x7fb7e80fc2b0 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.521+0000 7fb7ee966700 10 client.14793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.521+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e8114eb0 con 0x7fb7e80fc2b0 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.521+0000 7fb7ee966700 1 librados: init done 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.521+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb7e805fb80 con 0x7fb7e80fc2b0 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.522+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fb7dc010530 con 0x7fb7e80fc2b0 2024-02-13T00:40:16.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7e57fa700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb7d0062c90 0x7fb7d0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:16.064 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:40:16.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(91..91 src has 30..91) v4 ==== 6204+0+0 (secure 0 0 0) 0x7fb7dc07c640 con 0x7fb7e80fc2b0 2024-02-13T00:40:16.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 osd_map(91..91 src has 30..91) v4 2024-02-13T00:40:16.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7e57fa700 3 client.14793.objecter handle_osd_map got epochs [91,91] > 0 2024-02-13T00:40:16.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7e57fa700 3 client.14793.objecter handle_osd_map decoding full epoch 91 2024-02-13T00:40:16.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7e7fff700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb7d0062c90 0x7fb7d0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7e57fa700 20 client.14793.objecter dump_active .. 0 homeless 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v91) v1 ==== 54+0+16 (secure 0 0 0) 0x7fb7dc083070 con 0x7fb7e80fc2b0 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ee966700 10 client.14793.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 1 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb7e80fd560 0x7fb7e80fd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e80475e0 osd=0 3 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e80475e0 osd=0 3 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e80475e0 osd=0 3 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 1 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e80475e0 osd=0 3 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 0 1 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e80475e0 osd=0 4 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ee966700 10 client.14793.objecter _send_command 1 2024-02-13T00:40:16.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7e81060a0 con 0x7fb7e80fd560 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7e7fff700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb7d0062c90 0x7fb7d0065150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb7d8005010 tx=0x7fb7d8009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7e57fa700 10 client.14793.objecter ms_handle_connect 0x7fb7d0062c90 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.523+0000 7fb7ecf03700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb7e80fd560 0x7fb7e80fd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.524+0000 7fb7e57fa700 10 client.14793.objecter ms_handle_connect 0x7fb7e80fd560 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.524+0000 7fb7ecf03700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb7e80fd560 0x7fb7e80fd940 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.524+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb7e81060a0 con 0x7fb7e80fd560 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.524+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.524+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 1 = system:0 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.524+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e80475e0 osd=0 4 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.524+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 0 1 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.539+0000 7fb7ee966700 10 client.14793.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.539+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 2 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.539+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e80475e0 osd=0 3 2024-02-13T00:40:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.539+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e80475e0 osd=0 3 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.539+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e80475e0 osd=0 3 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.539+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 2 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.539+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e80475e0 osd=0 3 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.539+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 0 2 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.539+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e80475e0 osd=0 4 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.539+0000 7fb7ee966700 10 client.14793.objecter _send_command 2 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.539+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb7e8027410 con 0x7fb7e80fd560 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb7e8027410 con 0x7fb7e80fd560 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e80475e0 osd=0 4 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 0 2 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7ee966700 10 client.14793.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 3 2024-02-13T00:40:16.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6805/150599780 conn(0x7fb7e8059990 0x7fb7e8059dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:16.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e8028760 osd=1 3 2024-02-13T00:40:16.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8028760 osd=1 3 2024-02-13T00:40:16.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e8028760 osd=1 3 2024-02-13T00:40:16.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 3 2024-02-13T00:40:16.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e8028760 osd=1 3 2024-02-13T00:40:16.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 1 3 2024-02-13T00:40:16.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8028760 osd=1 4 2024-02-13T00:40:16.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7ee966700 10 client.14793.objecter _send_command 3 2024-02-13T00:40:16.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7e811b7c0 con 0x7fb7e8059990 2024-02-13T00:40:16.068 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:40:16.069 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:40:16.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7e7fff700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6805/150599780 conn(0x7fb7e8059990 0x7fb7e8059dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:16.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7e57fa700 10 client.14793.objecter ms_handle_connect 0x7fb7e8059990 2024-02-13T00:40:16.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.541+0000 7fb7e7fff700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6805/150599780 conn(0x7fb7e8059990 0x7fb7e8059dd0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:16.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.542+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb7e811b7c0 con 0x7fb7e8059990 2024-02-13T00:40:16.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.542+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:40:16.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.542+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 3 = system:0 2024-02-13T00:40:16.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.542+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e8028760 osd=1 4 2024-02-13T00:40:16.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.542+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 1 3 2024-02-13T00:40:16.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.557+0000 7fb7ee966700 10 client.14793.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:16.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.557+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 4 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.557+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e8028760 osd=1 3 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.557+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8028760 osd=1 3 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.557+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e8028760 osd=1 3 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.557+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 4 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.557+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e8028760 osd=1 3 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.557+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 1 4 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.557+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8028760 osd=1 4 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.557+0000 7fb7ee966700 10 client.14793.objecter _send_command 4 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.557+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb7e8059400 con 0x7fb7e8059990 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb7e8059400 con 0x7fb7e8059990 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e8028760 osd=1 4 2024-02-13T00:40:16.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 1 4 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7ee966700 10 client.14793.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 5 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb7e80282c0 0x7fb7e811a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e8059810 osd=2 3 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8059810 osd=2 3 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e8059810 osd=2 3 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 5 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e8059810 osd=2 3 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 2 5 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8059810 osd=2 4 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7ee966700 10 client.14793.objecter _send_command 5 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7e811a6d0 con 0x7fb7e80282c0 2024-02-13T00:40:16.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7e77fe700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb7e80282c0 0x7fb7e811a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7e57fa700 10 client.14793.objecter ms_handle_connect 0x7fb7e80282c0 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.559+0000 7fb7e77fe700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb7e80282c0 0x7fb7e811a1e0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.560+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb7e811a6d0 con 0x7fb7e80282c0 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.560+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.560+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 5 = system:0 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.560+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e8059810 osd=2 4 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.560+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 2 5 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.576+0000 7fb7ee966700 10 client.14793.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.576+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 6 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.576+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e8059810 osd=2 3 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.576+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8059810 osd=2 3 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.576+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e8059810 osd=2 3 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.576+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 6 2024-02-13T00:40:16.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.576+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e8059810 osd=2 3 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.576+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 2 6 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.576+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8059810 osd=2 4 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.576+0000 7fb7ee966700 10 client.14793.objecter _send_command 6 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.576+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb7e8027880 con 0x7fb7e80282c0 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb7e8027880 con 0x7fb7e80282c0 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e8059810 osd=2 4 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 2 6 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7ee966700 10 client.14793.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 7 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6813/604971011 conn(0x7fb7e811d940 0x7fb7e811fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e811b0c0 osd=3 3 2024-02-13T00:40:16.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811b0c0 osd=3 3 2024-02-13T00:40:16.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e811b0c0 osd=3 3 2024-02-13T00:40:16.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 7 2024-02-13T00:40:16.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e811b0c0 osd=3 3 2024-02-13T00:40:16.074 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:40:16.074 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:40:16.074 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:40:16.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 3 7 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811b0c0 osd=3 4 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7ee966700 10 client.14793.objecter _send_command 7 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.577+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7e8122a20 con 0x7fb7e811d940 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.578+0000 7fb7ecf03700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6813/604971011 conn(0x7fb7e811d940 0x7fb7e811fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.578+0000 7fb7e57fa700 10 client.14793.objecter ms_handle_connect 0x7fb7e811d940 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.578+0000 7fb7ecf03700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6813/604971011 conn(0x7fb7e811d940 0x7fb7e811fd30 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.578+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb7e8122a20 con 0x7fb7e811d940 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.578+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.578+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 7 = system:0 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.578+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e811b0c0 osd=3 4 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.578+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 3 7 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.594+0000 7fb7ee966700 10 client.14793.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:16.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.594+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 8 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.594+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e811b0c0 osd=3 3 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.594+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811b0c0 osd=3 3 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.594+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e811b0c0 osd=3 3 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.594+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 8 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.594+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e811b0c0 osd=3 3 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.594+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 3 8 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.594+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811b0c0 osd=3 4 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.594+0000 7fb7ee966700 10 client.14793.objecter _send_command 8 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.594+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb7e811ab30 con 0x7fb7e811d940 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb7e811ab30 con 0x7fb7e811d940 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e811b0c0 osd=3 4 2024-02-13T00:40:16.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 3 8 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7ee966700 10 client.14793.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 9 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb7e8119d40 0x7fb7e8121440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e811a8f0 osd=4 3 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811a8f0 osd=4 3 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e811a8f0 osd=4 3 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 9 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e811a8f0 osd=4 3 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 4 9 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811a8f0 osd=4 4 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7ee966700 10 client.14793.objecter _send_command 9 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.595+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7e811aa90 con 0x7fb7e8119d40 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.596+0000 7fb7e7fff700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb7e8119d40 0x7fb7e8121440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.596+0000 7fb7e7fff700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb7e8119d40 0x7fb7e8121440 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.596+0000 7fb7e57fa700 10 client.14793.objecter ms_handle_connect 0x7fb7e8119d40 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.597+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb7e811aa90 con 0x7fb7e8119d40 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.597+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.597+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 9 = system:0 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.597+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e811a8f0 osd=4 4 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.597+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 4 9 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.612+0000 7fb7ee966700 10 client.14793.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.612+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 10 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.612+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e811a8f0 osd=4 3 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.612+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811a8f0 osd=4 3 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.612+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e811a8f0 osd=4 3 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.612+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 10 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.612+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e811a8f0 osd=4 3 2024-02-13T00:40:16.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.612+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 4 10 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.612+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811a8f0 osd=4 4 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.612+0000 7fb7ee966700 10 client.14793.objecter _send_command 10 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.612+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb7e811aa90 con 0x7fb7e8119d40 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb7e811aa90 con 0x7fb7e8119d40 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e811a8f0 osd=4 4 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 4 10 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7ee966700 10 client.14793.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 11 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6804/2212296748 conn(0x7fb7e8121e70 0x7fb7e81222d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e811aa90 osd=5 3 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811aa90 osd=5 3 2024-02-13T00:40:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e811aa90 osd=5 3 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 11 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e811aa90 osd=5 3 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 5 11 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811aa90 osd=5 4 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7ee966700 10 client.14793.objecter _send_command 11 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7e81293c0 con 0x7fb7e8121e70 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7e77fe700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6804/2212296748 conn(0x7fb7e8121e70 0x7fb7e81222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7e57fa700 10 client.14793.objecter ms_handle_connect 0x7fb7e8121e70 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.614+0000 7fb7e77fe700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6804/2212296748 conn(0x7fb7e8121e70 0x7fb7e81222d0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.615+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb7e81293c0 con 0x7fb7e8121e70 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.615+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.615+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 11 = system:0 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.615+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e811aa90 osd=5 4 2024-02-13T00:40:16.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.615+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 5 11 2024-02-13T00:40:16.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.629+0000 7fb7ee966700 10 client.14793.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:16.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.629+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 12 2024-02-13T00:40:16.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.629+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e811aa90 osd=5 3 2024-02-13T00:40:16.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.629+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811aa90 osd=5 3 2024-02-13T00:40:16.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.629+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e811aa90 osd=5 3 2024-02-13T00:40:16.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.629+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 12 2024-02-13T00:40:16.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.629+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e811aa90 osd=5 3 2024-02-13T00:40:16.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.629+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 5 12 2024-02-13T00:40:16.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.629+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811aa90 osd=5 4 2024-02-13T00:40:16.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.629+0000 7fb7ee966700 10 client.14793.objecter _send_command 12 2024-02-13T00:40:16.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.629+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb7e8126c60 con 0x7fb7e8121e70 2024-02-13T00:40:16.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb7e8126c60 con 0x7fb7e8121e70 2024-02-13T00:40:16.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:16.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e811aa90 osd=5 4 2024-02-13T00:40:16.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 5 12 2024-02-13T00:40:16.082 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:40:16.082 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:40:16.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7ee966700 10 client.14793.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:16.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 13 2024-02-13T00:40:16.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb7e8127340 0x7fb7e8129730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:16.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e8126c60 osd=6 3 2024-02-13T00:40:16.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8126c60 osd=6 3 2024-02-13T00:40:16.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e8126c60 osd=6 3 2024-02-13T00:40:16.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 13 2024-02-13T00:40:16.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e8126c60 osd=6 3 2024-02-13T00:40:16.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 6 13 2024-02-13T00:40:16.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8126c60 osd=6 4 2024-02-13T00:40:16.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7ee966700 10 client.14793.objecter _send_command 13 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.631+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7e812c490 con 0x7fb7e8127340 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.632+0000 7fb7ecf03700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb7e8127340 0x7fb7e8129730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.632+0000 7fb7e57fa700 10 client.14793.objecter ms_handle_connect 0x7fb7e8127340 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.632+0000 7fb7ecf03700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb7e8127340 0x7fb7e8129730 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.633+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb7e812c490 con 0x7fb7e8127340 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.633+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.633+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 13 = system:0 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.633+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e8126c60 osd=6 4 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.633+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 6 13 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.649+0000 7fb7ee966700 10 client.14793.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.649+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 14 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.649+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e8126c60 osd=6 3 2024-02-13T00:40:16.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.649+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8126c60 osd=6 3 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.649+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e8126c60 osd=6 3 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.649+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 14 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.649+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e8126c60 osd=6 3 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.649+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 6 14 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.649+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8126c60 osd=6 4 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.649+0000 7fb7ee966700 10 client.14793.objecter _send_command 14 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.649+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb7e812c490 con 0x7fb7e8127340 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.651+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb7e812c490 con 0x7fb7e8127340 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.651+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.651+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.651+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e8126c60 osd=6 4 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.651+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 6 14 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.651+0000 7fb7ee966700 10 client.14793.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.651+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 15 2024-02-13T00:40:16.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.651+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb7e812a1d0 0x7fb7e812a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.652+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e812c490 osd=7 3 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.652+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e812c490 osd=7 3 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.652+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e812c490 osd=7 3 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.652+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 15 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.652+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e812c490 osd=7 3 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.652+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 7 15 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.652+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e812c490 osd=7 4 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.652+0000 7fb7ee966700 10 client.14793.objecter _send_command 15 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.652+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7e812c2a0 con 0x7fb7e812a1d0 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.652+0000 7fb7e7fff700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb7e812a1d0 0x7fb7e812a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.652+0000 7fb7e57fa700 10 client.14793.objecter ms_handle_connect 0x7fb7e812a1d0 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.652+0000 7fb7e7fff700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb7e812a1d0 0x7fb7e812a650 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.653+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb7d8019040 con 0x7fb7e812a1d0 2024-02-13T00:40:16.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.653+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.653+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 15 = system:0 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.653+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e812c490 osd=7 4 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.653+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 7 15 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.668+0000 7fb7ee966700 10 client.14793.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.668+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign -1 16 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.668+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e812c490 osd=7 3 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.668+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e812c490 osd=7 3 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.668+0000 7fb7ee966700 20 client.14793.objecter _get_session s=0x7fb7e812c490 osd=7 3 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.668+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_remove -1 16 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.668+0000 7fb7ee966700 20 client.14793.objecter get_session s=0x7fb7e812c490 osd=7 3 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.668+0000 7fb7ee966700 15 client.14793.objecter _session_command_op_assign 7 16 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.668+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e812c490 osd=7 4 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.668+0000 7fb7ee966700 10 client.14793.objecter _send_command 16 2024-02-13T00:40:16.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.668+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb7e812e6f0 con 0x7fb7e812a1d0 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7e57fa700 1 -- 172.21.15.162:0/4030630483 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb7e812e6f0 con 0x7fb7e812a1d0 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7e57fa700 10 client.14793.objecter ms_dispatch 0x7fb7e8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7e57fa700 10 client.14793.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7e57fa700 20 client.14793.objecter put_session s=0x7fb7e812c490 osd=7 4 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7e57fa700 15 client.14793.objecter _session_command_op_remove 7 16 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 10 librados: watch_flush enter 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 10 librados: watch_flush exit 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 10 client.14793.objecter close_session for osd.0 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb7e80fd560 msgr2=0x7fb7e80fd940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb7e80fd560 0x7fb7e80fd940 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e80475e0 osd=0 1 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 10 client.14793.objecter close_session for osd.1 2024-02-13T00:40:16.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6805/150599780 conn(0x7fb7e8059990 msgr2=0x7fb7e8059dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6805/150599780 conn(0x7fb7e8059990 0x7fb7e8059dd0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8028760 osd=1 1 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 10 client.14793.objecter close_session for osd.2 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb7e80282c0 msgr2=0x7fb7e811a1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb7e80282c0 0x7fb7e811a1e0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8059810 osd=2 1 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 10 client.14793.objecter close_session for osd.3 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6813/604971011 conn(0x7fb7e811d940 msgr2=0x7fb7e811fd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ecf03700 1 -- 172.21.15.162:0/4030630483 reap_dead start 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6813/604971011 conn(0x7fb7e811d940 0x7fb7e811fd30 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811b0c0 osd=3 1 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 10 client.14793.objecter close_session for osd.4 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb7e8119d40 msgr2=0x7fb7e8121440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb7e8119d40 0x7fb7e8121440 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811a8f0 osd=4 1 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 10 client.14793.objecter close_session for osd.5 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6804/2212296748 conn(0x7fb7e8121e70 msgr2=0x7fb7e81222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6804/2212296748 conn(0x7fb7e8121e70 0x7fb7e81222d0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e811aa90 osd=5 1 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 10 client.14793.objecter close_session for osd.6 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb7e8127340 msgr2=0x7fb7e8129730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb7e8127340 0x7fb7e8129730 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e8126c60 osd=6 1 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 10 client.14793.objecter close_session for osd.7 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb7e812a1d0 msgr2=0x7fb7e812a650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb7e812a1d0 0x7fb7e812a650 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 20 client.14793.objecter put_session s=0x7fb7e812c490 osd=7 1 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 20 client.14793.objecter shutdown clearing up homeless session... 2024-02-13T00:40:16.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 10 client.14793.objecter successfully canceled tick 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb7d0062c90 msgr2=0x7fb7d0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ecf03700 1 -- 172.21.15.162:0/4030630483 reap_dead start 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.670+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb7d0062c90 0x7fb7d0065150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb7d8005010 tx=0x7fb7d8009cb0).stop 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.671+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7e80fc2b0 msgr2=0x7fb7e8109770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.671+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7e80fc2b0 0x7fb7e8109770 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb7dc009400 tx=0x7fb7dc009cb0).stop 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.671+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 shutdown_connections 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.671+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7e80fc2b0 0x7fb7e8109770 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.671+0000 7fb7ee966700 1 --2- 172.21.15.162:0/4030630483 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb7d0062c90 0x7fb7d0065150 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.671+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 >> 172.21.15.162:0/4030630483 conn(0x7fb7e8070080 msgr2=0x7fb7e80fa040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.671+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 shutdown_connections 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.671+0000 7fb7ee966700 1 -- 172.21.15.162:0/4030630483 wait complete. 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:13.671+0000 7fb7ee966700 1 librados: shutdown 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.092 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.092 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.092 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.092 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.092 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.092 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:40:16.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:14.189997+0000 mgr.y (mgr.14544) 105 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:16.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:14.694084+0000 osd.5 (osd.5) 51 : cluster [DBG] 5.1b scrub starts 2024-02-13T00:40:16.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:14.696987+0000 osd.5 (osd.5) 52 : cluster [DBG] 5.1b scrub ok 2024-02-13T00:40:16.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:15.065611+0000 osd.6 (osd.6) 31 : cluster [DBG] 4.a deep-scrub starts 2024-02-13T00:40:16.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:15.074028+0000 osd.6 (osd.6) 32 : cluster [DBG] 4.a deep-scrub ok 2024-02-13T00:40:16.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:15.103994+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:40:16.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:15.106884+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:40:16.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:14.189997+0000 mgr.y (mgr.14544) 105 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:16.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:14.694084+0000 osd.5 (osd.5) 51 : cluster [DBG] 5.1b scrub starts 2024-02-13T00:40:16.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:14.696987+0000 osd.5 (osd.5) 52 : cluster [DBG] 5.1b scrub ok 2024-02-13T00:40:16.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:15.065611+0000 osd.6 (osd.6) 31 : cluster [DBG] 4.a deep-scrub starts 2024-02-13T00:40:16.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:15.074028+0000 osd.6 (osd.6) 32 : cluster [DBG] 4.a deep-scrub ok 2024-02-13T00:40:16.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:15.103994+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:40:16.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:15.106884+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:40:16.266 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:16.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:16.370 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:16.402 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:16.423 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:16.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:16.473 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:40:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:40:16] "GET /metrics HTTP/1.1" 200 172182 "" "Prometheus/2.33.4" 2024-02-13T00:40:16.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:16.557 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:16.576 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:16.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:16.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:16.680 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:16.709 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:16.727 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:16.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:16.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:16.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.793+0000 7f6183fff700 1 -- 172.21.15.162:0/3774301780 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v92) v1 ==== 87+0+0 (secure 0 0 0) 0x7f619c10b570 con 0x7f619c0f8a20 2024-02-13T00:40:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 10 librados: watch_flush enter 2024-02-13T00:40:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 10 librados: watch_flush exit 2024-02-13T00:40:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 20 client.24698.objecter shutdown clearing up homeless session... 2024-02-13T00:40:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 10 client.24698.objecter successfully canceled tick 2024-02-13T00:40:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 -- 172.21.15.162:0/3774301780 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6184062c90 msgr2=0x7f6184065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 --2- 172.21.15.162:0/3774301780 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6184062c90 0x7f6184065150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f619c0f8490 tx=0x7f618c00b040).stop 2024-02-13T00:40:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 -- 172.21.15.162:0/3774301780 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f619c0f8a20 msgr2=0x7f619c10a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 --2- 172.21.15.162:0/3774301780 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f619c0f8a20 0x7f619c10a700 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f619400bad0 tx=0x7f6194009780).stop 2024-02-13T00:40:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 -- 172.21.15.162:0/3774301780 shutdown_connections 2024-02-13T00:40:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 --2- 172.21.15.162:0/3774301780 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6184062c90 0x7f6184065150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 --2- 172.21.15.162:0/3774301780 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f619c0f8a20 0x7f619c10a700 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 --2- 172.21.15.162:0/3774301780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f619c0f61e0 0x7f619c110560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 --2- 172.21.15.162:0/3774301780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f619c0f6c30 0x7f619c110a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:16.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 -- 172.21.15.162:0/3774301780 >> 172.21.15.162:0/3774301780 conn(0x7f619c070080 msgr2=0x7f619c06c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:16.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 -- 172.21.15.162:0/3774301780 shutdown_connections 2024-02-13T00:40:16.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 -- 172.21.15.162:0/3774301780 wait complete. 2024-02-13T00:40:16.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:16.796+0000 7f61a17b7700 1 librados: shutdown 2024-02-13T00:40:16.800 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:40:16.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:16.859 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:16.877 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:16.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:16.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:16.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:17.009 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:15.277588+0000 mgr.y (mgr.14544) 106 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:15.811455+0000 mon.c (mon.2) 82 : audit [INF] from='client.? 172.21.15.162:0/3774301780' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:40:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:15.811747+0000 mon.a (mon.0) 724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:40:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:16.005925+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:40:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:16.009513+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:40:17.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:15.277588+0000 mgr.y (mgr.14544) 106 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:17.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:15.811455+0000 mon.c (mon.2) 82 : audit [INF] from='client.? 172.21.15.162:0/3774301780' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:40:17.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:15.811747+0000 mon.a (mon.0) 724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:40:17.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:16.005925+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:40:17.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:16.009513+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:40:17.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:15.277588+0000 mgr.y (mgr.14544) 106 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:15.811455+0000 mon.c (mon.2) 82 : audit [INF] from='client.? 172.21.15.162:0/3774301780' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:40:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:15.811747+0000 mon.a (mon.0) 724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:40:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:16.005925+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:40:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:16.009513+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:40:17.658 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:17.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:17.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:17.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:17.793 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:17.916 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:17.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:16.190723+0000 mgr.y (mgr.14544) 107 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:17.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:16.352045+0000 osd.2 (osd.2) 35 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:40:17.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:16.354158+0000 osd.2 (osd.2) 36 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:40:17.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:16.779209+0000 mon.a (mon.0) 725 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:40:17.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:16.791492+0000 mon.a (mon.0) 726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:40:17.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:16.791558+0000 mon.a (mon.0) 727 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-13T00:40:17.920 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:16.190723+0000 mgr.y (mgr.14544) 107 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:17.920 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:16.352045+0000 osd.2 (osd.2) 35 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:40:17.920 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:16.354158+0000 osd.2 (osd.2) 36 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:40:17.920 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:16.779209+0000 mon.a (mon.0) 725 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:40:17.920 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:16.791492+0000 mon.a (mon.0) 726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:40:17.920 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:16.791558+0000 mon.a (mon.0) 727 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-13T00:40:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:17.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:18.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:18.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:16.190723+0000 mgr.y (mgr.14544) 107 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:18.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:16.352045+0000 osd.2 (osd.2) 35 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:40:18.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:16.354158+0000 osd.2 (osd.2) 36 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:40:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:16.779209+0000 mon.a (mon.0) 725 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:40:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:16.791492+0000 mon.a (mon.0) 726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:40:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:16.791558+0000 mon.a (mon.0) 727 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-13T00:40:18.051 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:18.578 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:18.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:18.681 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:18.708 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:18.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:40:19.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:17.288470+0000 osd.7 (osd.7) 47 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:40:19.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:40:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:40:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:40:19.055 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:17.288470+0000 osd.7 (osd.7) 47 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:40:19.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:17.288470+0000 osd.7 (osd.7) 47 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:40:19.059 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:19.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:19.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:19.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:19.198 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:19.219 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:19.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:19.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:19.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:19.351 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:19.370 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:19.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:19.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:19.503 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:19.702 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:19.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:19.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:19.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:19.835 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:19.962 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:19.965 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:18.191124+0000 mgr.y (mgr.14544) 108 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 688 B/s rd, 0 op/s 2024-02-13T00:40:19.965 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:19.320176+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:40:19.965 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:18.191124+0000 mgr.y (mgr.14544) 108 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 688 B/s rd, 0 op/s 2024-02-13T00:40:19.965 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:19.320176+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:40:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:19.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:19.968 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:40:19.968 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:40:19.972 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:20.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:20.095 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:20.250 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:20.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:20.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:18.191124+0000 mgr.y (mgr.14544) 108 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 688 B/s rd, 0 op/s 2024-02-13T00:40:20.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:19.320176+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:40:20.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:20.386 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:20.407 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:20.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:20.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:20.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:40:20.538 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:20.738 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:20.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:20.759 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-13T00:40:20.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:20.870 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:20.911 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:20.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:21.019 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:21.051 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:21.072 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:21.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:21.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:21.203 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:21.300 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:21.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:21.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:21.436 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:21.457 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:21.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:21.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:21.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:21.589 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:21.662 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:21.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:21.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:21.691 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:21.695 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:21.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:21.795 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:21.817 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:21.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:21.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:21.952 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:20.191684+0000 mgr.y (mgr.14544) 109 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:21.952 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:20.191684+0000 mgr.y (mgr.14544) 109 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:21.954 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:22.200 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:40:22.201 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:40:22.201 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-13T00:40:22.201 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-02-13T00:40:22.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 4 0 2024-02-13T00:40:22.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:20.191684+0000 mgr.y (mgr.14544) 109 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:22.602 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:22.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:22.708 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:22.738 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:22.851 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:21.408061+0000 osd.2 (osd.2) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:22.852 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:21.408116+0000 osd.2 (osd.2) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:22.852 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:21.801179+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-13T00:40:22.852 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:21.408061+0000 osd.2 (osd.2) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:22.852 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:21.408116+0000 osd.2 (osd.2) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:22.852 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:21.801179+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-13T00:40:23.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:21.408061+0000 osd.2 (osd.2) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:23.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:21.408116+0000 osd.2 (osd.2) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:23.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:21.801179+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-13T00:40:24.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:22.191970+0000 mgr.y (mgr.14544) 110 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:24.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:22.191970+0000 mgr.y (mgr.14544) 110 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:24.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:22.191970+0000 mgr.y (mgr.14544) 110 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:25.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:40:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:40:26.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:24.192339+0000 mgr.y (mgr.14544) 111 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:24.192339+0000 mgr.y (mgr.14544) 111 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:26.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:24.192339+0000 mgr.y (mgr.14544) 111 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:26.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:40:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:40:26] "GET /metrics HTTP/1.1" 200 172263 "" "Prometheus/2.33.4" 2024-02-13T00:40:27.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:25.288449+0000 mgr.y (mgr.14544) 112 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:27.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:25.288449+0000 mgr.y (mgr.14544) 112 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:27.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:25.288449+0000 mgr.y (mgr.14544) 112 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:28.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:26.193071+0000 mgr.y (mgr.14544) 113 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:40:28.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:26.193071+0000 mgr.y (mgr.14544) 113 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:40:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:28.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:28.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:28.271 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:28.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:28.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:28.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:26.193071+0000 mgr.y (mgr.14544) 113 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:40:28.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:28.409 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:28.433 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:28.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:28.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:28.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:28.575 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:28.638 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:28.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:28.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:28.774 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:40:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:40:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:40:28.776 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:28.798 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:28.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:28.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:28.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:28.930 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:28.950 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:28.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:28.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:29.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:29.081 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:29.100 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:29.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:29.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:29.238 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:29.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:29.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:29.410 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:29.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.494+0000 7f953e1da700 1 -- 172.21.15.162:0/3246170719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95380fcc10 msgr2=0x7f95380f6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.494+0000 7f953e1da700 1 --2- 172.21.15.162:0/3246170719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95380fcc10 0x7f95380f6440 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f952c0048b0 tx=0x7f952c00adc0).stop 2024-02-13T00:40:29.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.495+0000 7f953e1da700 1 -- 172.21.15.162:0/3246170719 shutdown_connections 2024-02-13T00:40:29.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.495+0000 7f953e1da700 1 --2- 172.21.15.162:0/3246170719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95380f6b20 0x7f9538103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.495+0000 7f953e1da700 1 --2- 172.21.15.162:0/3246170719 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95380fc2b0 0x7f95380fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.495+0000 7f953e1da700 1 --2- 172.21.15.162:0/3246170719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95380fcc10 0x7f95380f6440 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.495+0000 7f953e1da700 1 -- 172.21.15.162:0/3246170719 >> 172.21.15.162:0/3246170719 conn(0x7f9538070080 msgr2=0x7f95380f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.495+0000 7f953e1da700 1 -- 172.21.15.162:0/3246170719 shutdown_connections 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.495+0000 7f953e1da700 1 -- 172.21.15.162:0/3246170719 wait complete. 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.495+0000 7f953e1da700 1 librados: starting msgr at 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.495+0000 7f953e1da700 1 librados: starting objecter 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.495+0000 7f953e1da700 1 Processor -- start 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f953e1da700 1 -- start start 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f953e1da700 1 librados: setting wanted keys 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f953e1da700 1 librados: calling monclient init 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f953e1da700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95380f6b20 0x7f953810e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f953e1da700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95380fc2b0 0x7f953810e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f953e1da700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95380fcc10 0x7f9538108360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f953e1da700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9538108850 con 0x7f95380fcc10 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f953e1da700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95381089c0 con 0x7f95380fc2b0 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f953e1da700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9538108b80 con 0x7f95380f6b20 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f95377fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95380f6b20 0x7f953810e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f95377fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95380f6b20 0x7f953810e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:44108/0 (socket says 172.21.15.162:44108) 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f95377fe700 1 -- 172.21.15.162:0/1590942684 learned_addr learned my addr 172.21.15.162:0/1590942684 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f9537fff700 1 --2- 172.21.15.162:0/1590942684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95380fcc10 0x7f9538108360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f9536ffd700 1 --2- 172.21.15.162:0/1590942684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95380fc2b0 0x7f953810e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f9537fff700 1 -- 172.21.15.162:0/1590942684 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95380f6b20 msgr2=0x7f953810e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f9537fff700 1 --2- 172.21.15.162:0/1590942684 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95380f6b20 0x7f953810e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f9537fff700 1 -- 172.21.15.162:0/1590942684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95380fc2b0 msgr2=0x7f953810e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.496+0000 7f9537fff700 1 --2- 172.21.15.162:0/1590942684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95380fc2b0 0x7f953810e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.497+0000 7f9537fff700 1 -- 172.21.15.162:0/1590942684 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c005040 con 0x7f95380fcc10 2024-02-13T00:40:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.497+0000 7f9537fff700 1 --2- 172.21.15.162:0/1590942684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95380fcc10 0x7f9538108360 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f953000d8d0 tx=0x7f9530004db0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.497+0000 7f9534ff9700 10 client.?.objecter ms_handle_connect 0x7f95380fcc10 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.497+0000 7f9534ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.497+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9530004020 con 0x7f95380fcc10 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.497+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9538108e60 con 0x7f95380fcc10 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.497+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f953000f040 con 0x7f95380fcc10 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.497+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9530003430 con 0x7f95380fcc10 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.497+0000 7f953e1da700 10 client.14805.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.497+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9538114ff0 con 0x7f95380fcc10 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.497+0000 7f953e1da700 1 librados: init done 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.498+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f9530004180 con 0x7f95380fcc10 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.499+0000 7f9534ff9700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9520062c90 0x7f9520065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.499+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(93..93 src has 41..93) v4 ==== 6204+0+0 (secure 0 0 0) 0x7f953007c5b0 con 0x7f95380fcc10 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.499+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 osd_map(93..93 src has 41..93) v4 2024-02-13T00:40:29.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.499+0000 7f9534ff9700 3 client.14805.objecter handle_osd_map got epochs [93,93] > 0 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.499+0000 7f9534ff9700 3 client.14805.objecter handle_osd_map decoding full epoch 93 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.499+0000 7f9534ff9700 20 client.14805.objecter dump_active .. 0 homeless 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.499+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f953805fb80 con 0x7f95380fcc10 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v93) v1 ==== 54+0+16 (secure 0 0 0) 0x7f9530050140 con 0x7f95380fcc10 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f953e1da700 10 client.14805.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 1 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6801/2103743338 conn(0x7f95380ff810 0x7f95380ffbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f95380475e0 osd=0 3 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f95380475e0 osd=0 3 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f95380475e0 osd=0 3 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 1 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f95380475e0 osd=0 3 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 0 1 2024-02-13T00:40:29.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f95380475e0 osd=0 4 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f953e1da700 10 client.14805.objecter _send_command 1 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9538104fa0 con 0x7f95380ff810 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f9536ffd700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6801/2103743338 conn(0x7f95380ff810 0x7f95380ffbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f9534ff9700 10 client.14805.objecter ms_handle_connect 0x7f95380ff810 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.500+0000 7f9536ffd700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6801/2103743338 conn(0x7f95380ff810 0x7f95380ffbf0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.501+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9538104fa0 con 0x7f95380ff810 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.501+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.501+0000 7f9534ff9700 10 client.14805.objecter _finish_command 1 = system:0 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.501+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f95380475e0 osd=0 4 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.501+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 0 1 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.502+0000 7f95377fe700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9520062c90 0x7f9520065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.502+0000 7f9534ff9700 10 client.14805.objecter ms_handle_connect 0x7f9520062c90 2024-02-13T00:40:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.502+0000 7f95377fe700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9520062c90 0x7f9520065150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9528004970 tx=0x7f952800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:29.519 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:29.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:29.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:29.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.530+0000 7f953e1da700 10 client.14805.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:29.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.530+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 2 2024-02-13T00:40:29.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.530+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f95380475e0 osd=0 3 2024-02-13T00:40:29.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.530+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f95380475e0 osd=0 3 2024-02-13T00:40:29.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.530+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f95380475e0 osd=0 3 2024-02-13T00:40:29.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.530+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 2 2024-02-13T00:40:29.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.530+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f95380475e0 osd=0 3 2024-02-13T00:40:29.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.530+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 0 2 2024-02-13T00:40:29.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.530+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f95380475e0 osd=0 4 2024-02-13T00:40:29.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.530+0000 7f953e1da700 10 client.14805.objecter _send_command 2 2024-02-13T00:40:29.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.530+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9538027410 con 0x7f95380ff810 2024-02-13T00:40:29.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.531+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9538027410 con 0x7f95380ff810 2024-02-13T00:40:29.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.531+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:29.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.531+0000 7f9534ff9700 10 client.14805.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.531+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f95380475e0 osd=0 4 2024-02-13T00:40:29.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.531+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 0 2 2024-02-13T00:40:29.535 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:40:29.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f953e1da700 10 client.14805.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:29.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 3 2024-02-13T00:40:29.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6805/150599780 conn(0x7f9538059990 0x7f9538059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:29.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f9538028760 osd=1 3 2024-02-13T00:40:29.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f9538028760 osd=1 3 2024-02-13T00:40:29.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f9538028760 osd=1 3 2024-02-13T00:40:29.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 3 2024-02-13T00:40:29.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f9538028760 osd=1 3 2024-02-13T00:40:29.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 1 3 2024-02-13T00:40:29.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f9538028760 osd=1 4 2024-02-13T00:40:29.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f953e1da700 10 client.14805.objecter _send_command 3 2024-02-13T00:40:29.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f953811b960 con 0x7f9538059990 2024-02-13T00:40:29.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f95377fe700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6805/150599780 conn(0x7f9538059990 0x7f9538059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:29.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f9534ff9700 10 client.14805.objecter ms_handle_connect 0x7f9538059990 2024-02-13T00:40:29.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.532+0000 7f95377fe700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6805/150599780 conn(0x7f9538059990 0x7f9538059d70 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:29.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.533+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f953811b960 con 0x7f9538059990 2024-02-13T00:40:29.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.533+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:40:29.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.533+0000 7f9534ff9700 10 client.14805.objecter _finish_command 3 = system:0 2024-02-13T00:40:29.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.533+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f9538028760 osd=1 4 2024-02-13T00:40:29.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.533+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 1 3 2024-02-13T00:40:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.562+0000 7f953e1da700 10 client.14805.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.562+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 4 2024-02-13T00:40:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.562+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f9538028760 osd=1 3 2024-02-13T00:40:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.562+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f9538028760 osd=1 3 2024-02-13T00:40:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.562+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f9538028760 osd=1 3 2024-02-13T00:40:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.562+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 4 2024-02-13T00:40:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.562+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f9538028760 osd=1 3 2024-02-13T00:40:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.562+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 1 4 2024-02-13T00:40:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.562+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f9538028760 osd=1 4 2024-02-13T00:40:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.562+0000 7f953e1da700 10 client.14805.objecter _send_command 4 2024-02-13T00:40:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.562+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9538059400 con 0x7f9538059990 2024-02-13T00:40:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.564+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9538059400 con 0x7f9538059990 2024-02-13T00:40:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.564+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.564+0000 7f9534ff9700 10 client.14805.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.564+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f9538028760 osd=1 4 2024-02-13T00:40:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.564+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 1 4 2024-02-13T00:40:29.568 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:40:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.564+0000 7f953e1da700 10 client.14805.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 5 2024-02-13T00:40:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6809/3689719376 conn(0x7f95380282c0 0x7f9538119200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f9538059810 osd=2 3 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f9538059810 osd=2 3 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f9538059810 osd=2 3 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 5 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f9538059810 osd=2 3 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 2 5 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f9538059810 osd=2 4 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f953e1da700 10 client.14805.objecter _send_command 5 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95380ff100 con 0x7f95380282c0 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f9536ffd700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6809/3689719376 conn(0x7f95380282c0 0x7f9538119200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f9534ff9700 10 client.14805.objecter ms_handle_connect 0x7f95380282c0 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.565+0000 7f9536ffd700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6809/3689719376 conn(0x7f95380282c0 0x7f9538119200 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.566+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f95380ff100 con 0x7f95380282c0 2024-02-13T00:40:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.566+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:40:29.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.566+0000 7f9534ff9700 10 client.14805.objecter _finish_command 5 = system:0 2024-02-13T00:40:29.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.566+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f9538059810 osd=2 4 2024-02-13T00:40:29.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.566+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 2 5 2024-02-13T00:40:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.594+0000 7f953e1da700 10 client.14805.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.594+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 6 2024-02-13T00:40:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.594+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f9538059810 osd=2 3 2024-02-13T00:40:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.594+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f9538059810 osd=2 3 2024-02-13T00:40:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.594+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f9538059810 osd=2 3 2024-02-13T00:40:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.594+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 6 2024-02-13T00:40:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.594+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f9538059810 osd=2 3 2024-02-13T00:40:29.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.594+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 2 6 2024-02-13T00:40:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.594+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f9538059810 osd=2 4 2024-02-13T00:40:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.594+0000 7f953e1da700 10 client.14805.objecter _send_command 6 2024-02-13T00:40:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.594+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9538027820 con 0x7f95380282c0 2024-02-13T00:40:29.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9538027820 con 0x7f95380282c0 2024-02-13T00:40:29.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:29.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f9534ff9700 10 client.14805.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f9538059810 osd=2 4 2024-02-13T00:40:29.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 2 6 2024-02-13T00:40:29.600 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:40:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f953e1da700 10 client.14805.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 7 2024-02-13T00:40:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6813/604971011 conn(0x7f953811a870 0x7f953811acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953811c2a0 osd=3 3 2024-02-13T00:40:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811c2a0 osd=3 3 2024-02-13T00:40:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953811c2a0 osd=3 3 2024-02-13T00:40:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 7 2024-02-13T00:40:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f953811c2a0 osd=3 3 2024-02-13T00:40:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 3 7 2024-02-13T00:40:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811c2a0 osd=3 4 2024-02-13T00:40:29.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f953e1da700 10 client.14805.objecter _send_command 7 2024-02-13T00:40:29.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f953811c920 con 0x7f953811a870 2024-02-13T00:40:29.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.597+0000 7f9537fff700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6813/604971011 conn(0x7f953811a870 0x7f953811acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:29.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.598+0000 7f9534ff9700 10 client.14805.objecter ms_handle_connect 0x7f953811a870 2024-02-13T00:40:29.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.598+0000 7f9537fff700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6813/604971011 conn(0x7f953811a870 0x7f953811acb0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:29.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.598+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f953811c920 con 0x7f953811a870 2024-02-13T00:40:29.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.599+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:40:29.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.599+0000 7f9534ff9700 10 client.14805.objecter _finish_command 7 = system:0 2024-02-13T00:40:29.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.599+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f953811c2a0 osd=3 4 2024-02-13T00:40:29.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.599+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 3 7 2024-02-13T00:40:29.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:29.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.626+0000 7f953e1da700 10 client.14805.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:29.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.626+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 8 2024-02-13T00:40:29.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.626+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953811c2a0 osd=3 3 2024-02-13T00:40:29.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.626+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811c2a0 osd=3 3 2024-02-13T00:40:29.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.626+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953811c2a0 osd=3 3 2024-02-13T00:40:29.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.626+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 8 2024-02-13T00:40:29.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.626+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f953811c2a0 osd=3 3 2024-02-13T00:40:29.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.626+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 3 8 2024-02-13T00:40:29.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.626+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811c2a0 osd=3 4 2024-02-13T00:40:29.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.626+0000 7f953e1da700 10 client.14805.objecter _send_command 8 2024-02-13T00:40:29.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.626+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f953811b3e0 con 0x7f953811a870 2024-02-13T00:40:29.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.628+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f953811b3e0 con 0x7f953811a870 2024-02-13T00:40:29.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.628+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:29.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.628+0000 7f9534ff9700 10 client.14805.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.628+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f953811c2a0 osd=3 4 2024-02-13T00:40:29.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.628+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 3 8 2024-02-13T00:40:29.632 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:40:29.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f953e1da700 10 client.14805.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:29.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 9 2024-02-13T00:40:29.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9538123580 0x7f9538125970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:29.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953811b7f0 osd=4 3 2024-02-13T00:40:29.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811b7f0 osd=4 3 2024-02-13T00:40:29.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953811b7f0 osd=4 3 2024-02-13T00:40:29.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 9 2024-02-13T00:40:29.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f953811b7f0 osd=4 3 2024-02-13T00:40:29.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 4 9 2024-02-13T00:40:29.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811b7f0 osd=4 4 2024-02-13T00:40:29.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f953e1da700 10 client.14805.objecter _send_command 9 2024-02-13T00:40:29.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f953811b2d0 con 0x7f9538123580 2024-02-13T00:40:29.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.629+0000 7f95377fe700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9538123580 0x7f9538125970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.630+0000 7f9534ff9700 10 client.14805.objecter ms_handle_connect 0x7f9538123580 2024-02-13T00:40:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.630+0000 7f95377fe700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9538123580 0x7f9538125970 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.631+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f953811b2d0 con 0x7f9538123580 2024-02-13T00:40:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.631+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:40:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.631+0000 7f9534ff9700 10 client.14805.objecter _finish_command 9 = system:0 2024-02-13T00:40:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.631+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f953811b7f0 osd=4 4 2024-02-13T00:40:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.631+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 4 9 2024-02-13T00:40:29.660 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:29.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.659+0000 7f953e1da700 10 client.14805.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:29.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.659+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 10 2024-02-13T00:40:29.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.659+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953811b7f0 osd=4 3 2024-02-13T00:40:29.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.659+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811b7f0 osd=4 3 2024-02-13T00:40:29.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.659+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953811b7f0 osd=4 3 2024-02-13T00:40:29.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.659+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 10 2024-02-13T00:40:29.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.659+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f953811b7f0 osd=4 3 2024-02-13T00:40:29.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.659+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 4 10 2024-02-13T00:40:29.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.659+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811b7f0 osd=4 4 2024-02-13T00:40:29.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.659+0000 7f953e1da700 10 client.14805.objecter _send_command 10 2024-02-13T00:40:29.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.659+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f953811b2d0 con 0x7f9538123580 2024-02-13T00:40:29.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f953811b2d0 con 0x7f9538123580 2024-02-13T00:40:29.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:29.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f9534ff9700 10 client.14805.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f953811b7f0 osd=4 4 2024-02-13T00:40:29.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 4 10 2024-02-13T00:40:29.665 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f953e1da700 10 client.14805.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 11 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6804/2212296748 conn(0x7f95381263b0 0x7f95381267f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953811b2d0 osd=5 3 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811b2d0 osd=5 3 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953811b2d0 osd=5 3 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 11 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f953811b2d0 osd=5 3 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 5 11 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811b2d0 osd=5 4 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f953e1da700 10 client.14805.objecter _send_command 11 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9538128440 con 0x7f95381263b0 2024-02-13T00:40:29.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.662+0000 7f9536ffd700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6804/2212296748 conn(0x7f95381263b0 0x7f95381267f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:29.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.663+0000 7f9534ff9700 10 client.14805.objecter ms_handle_connect 0x7f95381263b0 2024-02-13T00:40:29.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.663+0000 7f9536ffd700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6804/2212296748 conn(0x7f95381263b0 0x7f95381267f0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:29.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.664+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f952c007c00 con 0x7f95381263b0 2024-02-13T00:40:29.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.664+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:40:29.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.664+0000 7f9534ff9700 10 client.14805.objecter _finish_command 11 = system:0 2024-02-13T00:40:29.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.664+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f953811b2d0 osd=5 4 2024-02-13T00:40:29.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.664+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 5 11 2024-02-13T00:40:29.696 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:29.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:29.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.697+0000 7f953e1da700 10 client.14805.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:29.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.697+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 12 2024-02-13T00:40:29.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.697+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953811b2d0 osd=5 3 2024-02-13T00:40:29.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.697+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811b2d0 osd=5 3 2024-02-13T00:40:29.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.697+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953811b2d0 osd=5 3 2024-02-13T00:40:29.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.697+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 12 2024-02-13T00:40:29.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.697+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f953811b2d0 osd=5 3 2024-02-13T00:40:29.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.697+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 5 12 2024-02-13T00:40:29.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.697+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811b2d0 osd=5 4 2024-02-13T00:40:29.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.697+0000 7f953e1da700 10 client.14805.objecter _send_command 12 2024-02-13T00:40:29.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.697+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f953812a600 con 0x7f95381263b0 2024-02-13T00:40:29.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f953812a600 con 0x7f95381263b0 2024-02-13T00:40:29.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:29.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f9534ff9700 10 client.14805.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f953811b2d0 osd=5 4 2024-02-13T00:40:29.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 5 12 2024-02-13T00:40:29.703 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:40:29.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f953e1da700 10 client.14805.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:29.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 13 2024-02-13T00:40:29.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6808/1506640341 conn(0x7f953812acc0 0x7f953812b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:29.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953812a600 osd=6 3 2024-02-13T00:40:29.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953812a600 osd=6 3 2024-02-13T00:40:29.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953812a600 osd=6 3 2024-02-13T00:40:29.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 13 2024-02-13T00:40:29.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f953812a600 osd=6 3 2024-02-13T00:40:29.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 6 13 2024-02-13T00:40:29.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953812a600 osd=6 4 2024-02-13T00:40:29.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f953e1da700 10 client.14805.objecter _send_command 13 2024-02-13T00:40:29.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.699+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f953812e9c0 con 0x7f953812acc0 2024-02-13T00:40:29.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.700+0000 7f9537fff700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6808/1506640341 conn(0x7f953812acc0 0x7f953812b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:29.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.700+0000 7f9534ff9700 10 client.14805.objecter ms_handle_connect 0x7f953812acc0 2024-02-13T00:40:29.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.700+0000 7f9537fff700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6808/1506640341 conn(0x7f953812acc0 0x7f953812b120 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:29.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.701+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f953812e9c0 con 0x7f953812acc0 2024-02-13T00:40:29.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.701+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:40:29.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.701+0000 7f9534ff9700 10 client.14805.objecter _finish_command 13 = system:0 2024-02-13T00:40:29.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.701+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f953812a600 osd=6 4 2024-02-13T00:40:29.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.701+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 6 13 2024-02-13T00:40:29.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.729+0000 7f953e1da700 10 client.14805.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:29.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.729+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 14 2024-02-13T00:40:29.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.729+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953812a600 osd=6 3 2024-02-13T00:40:29.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.729+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953812a600 osd=6 3 2024-02-13T00:40:29.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.729+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953812a600 osd=6 3 2024-02-13T00:40:29.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.729+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 14 2024-02-13T00:40:29.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.729+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f953812a600 osd=6 3 2024-02-13T00:40:29.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.729+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 6 14 2024-02-13T00:40:29.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.729+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953812a600 osd=6 4 2024-02-13T00:40:29.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.729+0000 7f953e1da700 10 client.14805.objecter _send_command 14 2024-02-13T00:40:29.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.729+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f953812d840 con 0x7f953812acc0 2024-02-13T00:40:29.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f953812d840 con 0x7f953812acc0 2024-02-13T00:40:29.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:29.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f9534ff9700 10 client.14805.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f953812a600 osd=6 4 2024-02-13T00:40:29.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 6 14 2024-02-13T00:40:29.735 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:40:29.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f953e1da700 10 client.14805.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:29.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 15 2024-02-13T00:40:29.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6812/1964192772 conn(0x7f953812def0 0x7f953812e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:29.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953812d840 osd=7 3 2024-02-13T00:40:29.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953812d840 osd=7 3 2024-02-13T00:40:29.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953812d840 osd=7 3 2024-02-13T00:40:29.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 15 2024-02-13T00:40:29.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f953812d840 osd=7 3 2024-02-13T00:40:29.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 7 15 2024-02-13T00:40:29.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953812d840 osd=7 4 2024-02-13T00:40:29.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f953e1da700 10 client.14805.objecter _send_command 15 2024-02-13T00:40:29.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.731+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f953812e940 con 0x7f953812def0 2024-02-13T00:40:29.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.732+0000 7f95377fe700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6812/1964192772 conn(0x7f953812def0 0x7f953812e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:29.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.732+0000 7f9534ff9700 10 client.14805.objecter ms_handle_connect 0x7f953812def0 2024-02-13T00:40:29.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.732+0000 7f95377fe700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6812/1964192772 conn(0x7f953812def0 0x7f953812e330 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:29.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.733+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f953812e940 con 0x7f953812def0 2024-02-13T00:40:29.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.733+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:40:29.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.733+0000 7f9534ff9700 10 client.14805.objecter _finish_command 15 = system:0 2024-02-13T00:40:29.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.733+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f953812d840 osd=7 4 2024-02-13T00:40:29.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.733+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 7 15 2024-02-13T00:40:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.761+0000 7f953e1da700 10 client.14805.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.761+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign -1 16 2024-02-13T00:40:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.761+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953812d840 osd=7 3 2024-02-13T00:40:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.761+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953812d840 osd=7 3 2024-02-13T00:40:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.761+0000 7f953e1da700 20 client.14805.objecter _get_session s=0x7f953812d840 osd=7 3 2024-02-13T00:40:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.761+0000 7f953e1da700 15 client.14805.objecter _session_command_op_remove -1 16 2024-02-13T00:40:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.761+0000 7f953e1da700 20 client.14805.objecter get_session s=0x7f953812d840 osd=7 3 2024-02-13T00:40:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.761+0000 7f953e1da700 15 client.14805.objecter _session_command_op_assign 7 16 2024-02-13T00:40:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.761+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953812d840 osd=7 4 2024-02-13T00:40:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.761+0000 7f953e1da700 10 client.14805.objecter _send_command 16 2024-02-13T00:40:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.761+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9538126da0 con 0x7f953812def0 2024-02-13T00:40:29.766 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:40:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f9534ff9700 1 -- 172.21.15.162:0/1590942684 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9538126da0 con 0x7f953812def0 2024-02-13T00:40:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f9534ff9700 10 client.14805.objecter ms_dispatch 0x7f9538001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f9534ff9700 10 client.14805.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f9534ff9700 20 client.14805.objecter put_session s=0x7f953812d840 osd=7 4 2024-02-13T00:40:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f9534ff9700 15 client.14805.objecter _session_command_op_remove 7 16 2024-02-13T00:40:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 10 librados: watch_flush enter 2024-02-13T00:40:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 10 librados: watch_flush exit 2024-02-13T00:40:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 10 client.14805.objecter close_session for osd.0 2024-02-13T00:40:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6801/2103743338 conn(0x7f95380ff810 msgr2=0x7f95380ffbf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6801/2103743338 conn(0x7f95380ff810 0x7f95380ffbf0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f95380475e0 osd=0 1 2024-02-13T00:40:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 10 client.14805.objecter close_session for osd.1 2024-02-13T00:40:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6805/150599780 conn(0x7f9538059990 msgr2=0x7f9538059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6805/150599780 conn(0x7f9538059990 0x7f9538059d70 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f9538028760 osd=1 1 2024-02-13T00:40:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 10 client.14805.objecter close_session for osd.2 2024-02-13T00:40:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6809/3689719376 conn(0x7f95380282c0 msgr2=0x7f9538119200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6809/3689719376 conn(0x7f95380282c0 0x7f9538119200 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f9538059810 osd=2 1 2024-02-13T00:40:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 10 client.14805.objecter close_session for osd.3 2024-02-13T00:40:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6813/604971011 conn(0x7f953811a870 msgr2=0x7f953811acb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6813/604971011 conn(0x7f953811a870 0x7f953811acb0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811c2a0 osd=3 1 2024-02-13T00:40:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 10 client.14805.objecter close_session for osd.4 2024-02-13T00:40:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9538123580 msgr2=0x7f9538125970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9538123580 0x7f9538125970 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811b7f0 osd=4 1 2024-02-13T00:40:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 10 client.14805.objecter close_session for osd.5 2024-02-13T00:40:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6804/2212296748 conn(0x7f95381263b0 msgr2=0x7f95381267f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6804/2212296748 conn(0x7f95381263b0 0x7f95381267f0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953811b2d0 osd=5 1 2024-02-13T00:40:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 10 client.14805.objecter close_session for osd.6 2024-02-13T00:40:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6808/1506640341 conn(0x7f953812acc0 msgr2=0x7f953812b120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6808/1506640341 conn(0x7f953812acc0 0x7f953812b120 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953812a600 osd=6 1 2024-02-13T00:40:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 10 client.14805.objecter close_session for osd.7 2024-02-13T00:40:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6812/1964192772 conn(0x7f953812def0 msgr2=0x7f953812e330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.181:6812/1964192772 conn(0x7f953812def0 0x7f953812e330 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 20 client.14805.objecter put_session s=0x7f953812d840 osd=7 1 2024-02-13T00:40:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 20 client.14805.objecter shutdown clearing up homeless session... 2024-02-13T00:40:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f953e1da700 10 client.14805.objecter successfully canceled tick 2024-02-13T00:40:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f9537fff700 1 -- 172.21.15.162:0/1590942684 reap_dead start 2024-02-13T00:40:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.763+0000 7f9537fff700 1 -- 172.21.15.162:0/1590942684 reap_dead start 2024-02-13T00:40:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f9537fff700 1 -- 172.21.15.162:0/1590942684 reap_dead start 2024-02-13T00:40:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f9537fff700 1 -- 172.21.15.162:0/1590942684 reap_dead start 2024-02-13T00:40:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f9537fff700 1 -- 172.21.15.162:0/1590942684 reap_dead start 2024-02-13T00:40:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9520062c90 msgr2=0x7f9520065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9520062c90 0x7f9520065150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9528004970 tx=0x7f952800b040).stop 2024-02-13T00:40:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95380fcc10 msgr2=0x7f9538108360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95380fcc10 0x7f9538108360 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f953000d8d0 tx=0x7f9530004db0).stop 2024-02-13T00:40:29.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 shutdown_connections 2024-02-13T00:40:29.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95380fcc10 0x7f9538108360 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f953e1da700 1 --2- 172.21.15.162:0/1590942684 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9520062c90 0x7f9520065150 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:29.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 >> 172.21.15.162:0/1590942684 conn(0x7f9538070080 msgr2=0x7f953806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:29.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 shutdown_connections 2024-02-13T00:40:29.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f953e1da700 1 -- 172.21.15.162:0/1590942684 wait complete. 2024-02-13T00:40:29.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:29.764+0000 7f953e1da700 1 librados: shutdown 2024-02-13T00:40:29.774 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.774 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.774 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.774 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.774 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.774 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.775 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.775 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:40:29.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:29.834 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:29.855 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:29.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:29.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:29.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:29.988 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:30.008 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:30.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:30.021 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:30.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:28.193464+0000 mgr.y (mgr.14544) 114 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:30.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:29.500694+0000 mon.a (mon.0) 730 : audit [DBG] from='client.? 172.21.15.162:0/1590942684' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:40:30.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:28.193464+0000 mgr.y (mgr.14544) 114 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:30.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:29.500694+0000 mon.a (mon.0) 730 : audit [DBG] from='client.? 172.21.15.162:0/1590942684' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:40:30.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:30.150 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:30.265 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:30.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:30.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:28.193464+0000 mgr.y (mgr.14544) 114 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:30.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:29.500694+0000 mon.a (mon.0) 730 : audit [DBG] from='client.? 172.21.15.162:0/1590942684' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:40:30.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.303+0000 7fe024643700 1 -- 172.21.15.162:0/328365351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c0f8d10 msgr2=0x7fe01c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:30.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.303+0000 7fe024643700 1 --2- 172.21.15.162:0/328365351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c0f8d10 0x7fe01c0f90f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fe018004870 tx=0x7fe018009cb0).stop 2024-02-13T00:40:30.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.304+0000 7fe024643700 1 -- 172.21.15.162:0/328365351 shutdown_connections 2024-02-13T00:40:30.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.304+0000 7fe024643700 1 --2- 172.21.15.162:0/328365351 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe01c102270 0x7fe01c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:30.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.304+0000 7fe024643700 1 --2- 172.21.15.162:0/328365351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c0f8d10 0x7fe01c0f90f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:30.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.304+0000 7fe024643700 1 --2- 172.21.15.162:0/328365351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe01c0f95e0 0x7fe01c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:30.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.304+0000 7fe024643700 1 -- 172.21.15.162:0/328365351 >> 172.21.15.162:0/328365351 conn(0x7fe01c06eb90 msgr2=0x7fe01c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:30.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.304+0000 7fe024643700 1 -- 172.21.15.162:0/328365351 shutdown_connections 2024-02-13T00:40:30.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.304+0000 7fe024643700 1 -- 172.21.15.162:0/328365351 wait complete. 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.304+0000 7fe024643700 1 librados: starting msgr at 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.304+0000 7fe024643700 1 librados: starting objecter 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe024643700 1 Processor -- start 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe024643700 1 -- start start 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe024643700 1 librados: setting wanted keys 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe024643700 1 librados: calling monclient init 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe024643700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe01c0f8d10 0x7fe01c0f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe024643700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe01c0f95e0 0x7fe01c0f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe024643700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c102270 0x7fe01c0f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe024643700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0f8a20 con 0x7fe01c102270 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe024643700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0f8b90 con 0x7fe01c0f95e0 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe024643700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c10c6e0 con 0x7fe01c0f8d10 2024-02-13T00:40:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe0223df700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe01c0f8d10 0x7fe01c0f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe0223df700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe01c0f8d10 0x7fe01c0f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:44124/0 (socket says 172.21.15.162:44124) 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe0223df700 1 -- 172.21.15.162:0/3234025212 learned_addr learned my addr 172.21.15.162:0/3234025212 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe022be0700 1 --2- 172.21.15.162:0/3234025212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c102270 0x7fe01c0f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.305+0000 7fe021bde700 1 --2- 172.21.15.162:0/3234025212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe01c0f95e0 0x7fe01c0f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe0223df700 1 -- 172.21.15.162:0/3234025212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe01c0f95e0 msgr2=0x7fe01c0f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe0223df700 1 --2- 172.21.15.162:0/3234025212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe01c0f95e0 0x7fe01c0f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe0223df700 1 -- 172.21.15.162:0/3234025212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c102270 msgr2=0x7fe01c0f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe0223df700 1 --2- 172.21.15.162:0/3234025212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c102270 0x7fe01c0f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe0223df700 1 -- 172.21.15.162:0/3234025212 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c005040 con 0x7fe01c0f8d10 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe021bde700 1 --2- 172.21.15.162:0/3234025212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe01c0f95e0 0x7fe01c0f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe022be0700 1 --2- 172.21.15.162:0/3234025212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c102270 0x7fe01c0f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe0223df700 1 --2- 172.21.15.162:0/3234025212 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe01c0f8d10 0x7fe01c0f6cc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fe0180043d0 tx=0x7fe018009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe0137fe700 10 client.?.objecter ms_handle_connect 0x7fe01c0f8d10 2024-02-13T00:40:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe0137fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:30.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe0137fe700 1 -- 172.21.15.162:0/3234025212 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01800e030 con 0x7fe01c0f8d10 2024-02-13T00:40:30.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe0137fe700 1 -- 172.21.15.162:0/3234025212 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe018013da0 con 0x7fe01c0f8d10 2024-02-13T00:40:30.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe024643700 1 -- 172.21.15.162:0/3234025212 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe018005040 con 0x7fe01c0f8d10 2024-02-13T00:40:30.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.306+0000 7fe0137fe700 1 -- 172.21.15.162:0/3234025212 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018018070 con 0x7fe01c0f8d10 2024-02-13T00:40:30.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.307+0000 7fe024643700 10 client.24710.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:30.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.307+0000 7fe024643700 1 -- 172.21.15.162:0/3234025212 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c10cb80 con 0x7fe01c0f8d10 2024-02-13T00:40:30.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.307+0000 7fe024643700 1 librados: init done 2024-02-13T00:40:30.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.308+0000 7fe024643700 1 -- 172.21.15.162:0/3234025212 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe000004f50 con 0x7fe01c0f8d10 2024-02-13T00:40:30.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.309+0000 7fe0137fe700 1 -- 172.21.15.162:0/3234025212 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe01801d420 con 0x7fe01c0f8d10 2024-02-13T00:40:30.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.309+0000 7fe0137fe700 1 --2- 172.21.15.162:0/3234025212 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe008062ce0 0x7fe0080651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:30.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.309+0000 7fe0137fe700 1 -- 172.21.15.162:0/3234025212 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(93..93 src has 41..93) v4 ==== 6204+0+0 (secure 0 0 0) 0x7fe01807da80 con 0x7fe01c0f8d10 2024-02-13T00:40:30.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.309+0000 7fe0137fe700 10 client.24710.objecter ms_dispatch 0x7fe01c001550 osd_map(93..93 src has 41..93) v4 2024-02-13T00:40:30.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.309+0000 7fe0137fe700 3 client.24710.objecter handle_osd_map got epochs [93,93] > 0 2024-02-13T00:40:30.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.309+0000 7fe0137fe700 3 client.24710.objecter handle_osd_map decoding full epoch 93 2024-02-13T00:40:30.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.310+0000 7fe0137fe700 20 client.24710.objecter dump_active .. 0 homeless 2024-02-13T00:40:30.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.314+0000 7fe021bde700 1 --2- 172.21.15.162:0/3234025212 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe008062ce0 0x7fe0080651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:30.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.314+0000 7fe0137fe700 10 client.24710.objecter ms_handle_connect 0x7fe008062ce0 2024-02-13T00:40:30.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.314+0000 7fe021bde700 1 --2- 172.21.15.162:0/3234025212 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe008062ce0 0x7fe0080651a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fe01c0f8460 tx=0x7fe00c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:30.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.318+0000 7fe0137fe700 1 -- 172.21.15.162:0/3234025212 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe018051770 con 0x7fe01c0f8d10 2024-02-13T00:40:30.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:30.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:30.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:30.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:30.576 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:30.597 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:30.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:30.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:30.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.671+0000 7fe024643700 1 -- 172.21.15.162:0/3234025212 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fe000005200 con 0x7fe01c0f8d10 2024-02-13T00:40:30.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:30.713 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:40:30.713 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:40:30.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:40:30.713 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:40:30.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.713 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24329.0:89 4.b 4:d467b91b:::bucket.sync-target-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected e59)", 2024-02-13T00:40:30.713 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:43.226556+0000", 2024-02-13T00:40:30.713 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 167.47941258399999, 2024-02-13T00:40:30.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076504859999999997, 2024-02-13T00:40:30.713 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.713 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24329", 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 89 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226556+0000", 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226556+0000", 2024-02-13T00:40:30.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:40:30.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226554+0000", 2024-02-13T00:40:30.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4489999999999999e-06 2024-02-13T00:40:30.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226563+0000", 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2398000000000001e-05 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226575+0000", 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8539999999999997e-06 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226583+0000", 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023794300000000001 2024-02-13T00:40:30.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226821+0000", 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2050000000000001e-06 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226826+0000", 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072467690000000001 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.234073+0000", 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133807 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.234207+0000", 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076504859999999997 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24359.0:51 4.b 4:d467b91b:::bucket.sync-target-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected e59)", 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:43.226698+0000", 2024-02-13T00:40:30.718 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 167.47927059200001, 2024-02-13T00:40:30.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007781166, 2024-02-13T00:40:30.719 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.719 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:40:30.719 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.719 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24359", 2024-02-13T00:40:30.719 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/396906212", 2024-02-13T00:40:30.719 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 51 2024-02-13T00:40:30.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.719 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226698+0000", 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226698+0000", 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226697+0000", 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6469999999999998e-06 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226705+0000", 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.371e-06 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226713+0000", 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0090000000000001e-06 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226720+0000", 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119912 2024-02-13T00:40:30.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226840+0000", 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6649999999999999e-06 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.226843+0000", 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0075311859999999996 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.234374+0000", 2024-02-13T00:40:30.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105478 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:43.234480+0000", 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007781166 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24359.0:64 5.3 5:c609908c:::notify.5:head [watch watch cookie 94085721333424] snapc 0=[] ondisk+write+known_if_redirected e61)", 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:45.260601+0000", 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 165.44536790000001, 2024-02-13T00:40:30.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002036019, 2024-02-13T00:40:30.724 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.724 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.724 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.724 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24359", 2024-02-13T00:40:30.724 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/396906212", 2024-02-13T00:40:30.724 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 64 2024-02-13T00:40:30.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.724 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.260601+0000", 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.260601+0000", 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.260600+0000", 2024-02-13T00:40:30.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7039999999999997e-06 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.260606+0000", 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7459999999999995e-06 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.260616+0000", 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3810000000000004e-06 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.260623+0000", 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036213400000000001 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.260985+0000", 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8701000000000004e-05 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.261063+0000", 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000530139 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.261594+0000", 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8368999999999995e-05 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.261662+0000", 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017237400000000001 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.261834+0000", 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035719000000000001 2024-02-13T00:40:30.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.262192+0000", 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032328599999999999 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.262515+0000", 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3912999999999997e-05 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.262569+0000", 2024-02-13T00:40:30.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6804e-05 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.262596+0000", 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1548000000000003e-05 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:45.262637+0000", 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002036019 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:37 4.b 4:d5e0c857:lc::lc.5:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.303562+0000", 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.402407227, 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026877789999999999, 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 37 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.303562+0000", 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.303562+0000", 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.303560+0000", 2024-02-13T00:40:30.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0110000000000006e-06 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.303569+0000", 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9106999999999998e-05 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.303588+0000", 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7889999999999997e-06 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.303595+0000", 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6660999999999998e-05 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.303652+0000", 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012374400000000001 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.303775+0000", 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021257200000000001 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.303988+0000", 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049003499999999999 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.304478+0000", 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051870099999999999 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.304997+0000", 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011484710000000001 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.306145+0000", 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5900999999999999e-05 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.306201+0000", 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8469e-05 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.306249+0000", 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026877789999999999 2024-02-13T00:40:30.736 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.737 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.737 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.737 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:59 4.b 4:d5e0c857:lc::lc.5:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.737 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.309800+0000", 2024-02-13T00:40:30.737 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.39616904900001, 2024-02-13T00:40:30.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020189800000000001, 2024-02-13T00:40:30.737 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.737 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 59 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.309800+0000", 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.309800+0000", 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.309798+0000", 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.977e-06 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.309806+0000", 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1999e-05 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.309818+0000", 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8399999999999997e-06 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.309825+0000", 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2818000000000002e-05 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.309868+0000", 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8579999999999997e-05 2024-02-13T00:40:30.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.309937+0000", 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017823000000000001 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310115+0000", 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001572957 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.311688+0000", 2024-02-13T00:40:30.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7708000000000001e-05 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.311706+0000", 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0892e-05 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.311716+0000", 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.143e-05 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.311778+0000", 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1004999999999998e-05 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.311819+0000", 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020189800000000001 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:119 4.3 4:c3a682d4:reshard::reshard.0000000004:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.743 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.330373+0000", 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.37559632899999, 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026596459999999999, 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 119 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330373+0000", 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330373+0000", 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330370+0000", 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0334e-05 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330381+0000", 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7215e-05 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330398+0000", 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7430000000000002e-06 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330405+0000", 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3701e-05 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330459+0000", 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038366799999999999 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330843+0000", 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019968699999999999 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,7", 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.331043+0000", 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00065125000000000003 2024-02-13T00:40:30.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.331694+0000", 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012246679999999999 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.332918+0000", 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4876999999999998e-05 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.332953+0000", 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.35e-05 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.332997+0000", 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5404999999999997e-05 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.333032+0000", 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026596459999999999 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.749 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:125 4.3 4:c623b173:gc::gc.16:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.334132+0000", 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.37183668700001, 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017920099999999999, 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 125 2024-02-13T00:40:30.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.334132+0000", 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.334132+0000", 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.334130+0000", 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4740000000000005e-06 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.334138+0000", 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4399000000000001e-05 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.334153+0000", 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6833999999999998e-05 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.334169+0000", 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8263999999999995e-05 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.334248+0000", 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7686000000000001e-05 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.334325+0000", 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012936100000000001 2024-02-13T00:40:30.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,7", 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.334455+0000", 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033753199999999998 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.334792+0000", 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010086190000000001 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.335801+0000", 2024-02-13T00:40:30.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9635999999999999e-05 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.335831+0000", 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2562999999999998e-05 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.335883+0000", 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1117e-05 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.335924+0000", 2024-02-13T00:40:30.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017920099999999999 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:152 4.16 4:6849fd06:gc::gc.17:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.340896+0000", 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.36507323699999, 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021618539999999999, 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.756 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 152 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340896+0000", 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340896+0000", 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:40:30.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340894+0000", 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2169999999999999e-06 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340901+0000", 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2289999999999999e-05 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340913+0000", 2024-02-13T00:40:30.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6069999999999996e-06 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340920+0000", 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1147000000000002e-05 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340991+0000", 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9488999999999997e-05 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.341070+0000", 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000191344 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.341262+0000", 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000789524 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.342051+0000", 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00046829899999999998 2024-02-13T00:40:30.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.342519+0000", 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000441858 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.342961+0000", 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8108e-05 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.343019+0000", 2024-02-13T00:40:30.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8183999999999999e-05 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.343058+0000", 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021618539999999999 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:153 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.340933+0000", 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.36503546, 2024-02-13T00:40:30.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027709359999999999, 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 153 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340933+0000", 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340933+0000", 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340933+0000", 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3209999999999998e-06 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340937+0000", 2024-02-13T00:40:30.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6690000000000006e-06 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340945+0000", 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8230000000000002e-06 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340950+0000", 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00053245600000000005 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.341482+0000", 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113229 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.341595+0000", 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022506699999999999 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.341820+0000", 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00066690200000000003 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.342487+0000", 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00046670400000000002 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.342954+0000", 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00066240799999999996 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.343616+0000", 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9496000000000002e-05 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.343666+0000", 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8593999999999997e-05 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.343704+0000", 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027709359999999999 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.768 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:170 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.344283+0000", 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.36168586599999, 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018374489999999999, 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 170 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.344283+0000", 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.344283+0000", 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.344282+0000", 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0090000000000001e-06 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.344289+0000", 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2969000000000001e-05 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.344302+0000", 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2750000000000001e-06 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.344308+0000", 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9984000000000002e-05 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.344348+0000", 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9119e-05 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.344417+0000", 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000177799 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.344595+0000", 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049767799999999999 2024-02-13T00:40:30.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.345093+0000", 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020250800000000001 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.345295+0000", 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00074693800000000005 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.346042+0000", 2024-02-13T00:40:30.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4978000000000002e-05 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.346087+0000", 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.328e-05 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.346121+0000", 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018374489999999999 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:177 4.16 4:6a988a9b:reshard::reshard.0000000007:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.346328+0000", 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.35964060399999, 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001851159, 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 177 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.346328+0000", 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.346328+0000", 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.346328+0000", 2024-02-13T00:40:30.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4780000000000002e-06 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.346332+0000", 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0119999999999999e-06 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.346339+0000", 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6169999999999996e-06 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.346349+0000", 2024-02-13T00:40:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6582000000000003e-05 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.346385+0000", 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9672000000000006e-05 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.346465+0000", 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115345 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-13T00:40:30.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.346580+0000", 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049348999999999997 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.347074+0000", 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0009885810000000001 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.348062+0000", 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0561e-05 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.348103+0000", 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2645999999999998e-05 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.348145+0000", 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4004000000000002e-05 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.348179+0000", 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001851159 2024-02-13T00:40:30.780 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:187 4.16 4:6a988a9b:reshard::reshard.0000000007:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.349242+0000", 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.356726624, 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027256120000000001, 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 187 2024-02-13T00:40:30.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.349242+0000", 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.349242+0000", 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.349241+0000", 2024-02-13T00:40:30.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3739999999999998e-06 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.349250+0000", 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1977999999999999e-05 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.349261+0000", 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.058e-06 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.349269+0000", 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2908999999999998e-05 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.349321+0000", 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3629999999999999e-05 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.349385+0000", 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119004 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.349504+0000", 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069601299999999999 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.350200+0000", 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028028500000000002 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.350480+0000", 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001401799 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.351882+0000", 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.901e-05 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.351931+0000", 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6727000000000003e-05 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.351968+0000", 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027256120000000001 2024-02-13T00:40:30.786 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:432 4.1e 4:7e82ab18:lc::lc.13:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.404705+0000", 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.30126389899999, 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017370759999999999, 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 432 2024-02-13T00:40:30.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.404705+0000", 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.404705+0000", 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.404703+0000", 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3239999999999997e-06 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.404711+0000", 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0125e-05 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.404721+0000", 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2280000000000003e-06 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.404727+0000", 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9144999999999997e-05 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.404786+0000", 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012840099999999999 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.404915+0000", 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018877999999999999 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.405103+0000", 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051977800000000004 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.405623+0000", 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0605000000000003e-05 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.405654+0000", 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00074141099999999998 2024-02-13T00:40:30.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.406395+0000", 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7532e-05 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.406423+0000", 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.946e-05 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.406442+0000", 2024-02-13T00:40:30.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017370759999999999 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:484 4.16 4:6d4431ed:lc::lc.7:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.416929+0000", 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.28904022099999, 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002340876, 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.793 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 484 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.416929+0000", 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.416929+0000", 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.416928+0000", 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2870000000000001e-06 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.416934+0000", 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2619000000000001e-05 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.416947+0000", 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2040000000000002e-06 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.416953+0000", 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7920999999999998e-05 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.416991+0000", 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2236000000000005e-05 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.417053+0000", 2024-02-13T00:40:30.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015025600000000001 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.417203+0000", 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041320199999999999 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.417617+0000", 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00061621799999999997 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.418233+0000", 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00095108400000000004 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.419184+0000", 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5008e-05 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.419229+0000", 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0754000000000003e-05 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.419270+0000", 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002340876 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:511 4.1e 4:7e82ab18:lc::lc.13:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.425716+0000", 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.28025301, 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017652670000000001, 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.799 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 511 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.425716+0000", 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.425716+0000", 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.425715+0000", 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6340000000000003e-06 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.425721+0000", 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1374e-05 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.425733+0000", 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2750000000000003e-06 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.425738+0000", 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0125999999999999e-05 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.425808+0000", 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5140000000000002e-05 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.425883+0000", 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017960299999999999 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.426063+0000", 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076240699999999999 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.426825+0000", 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029580699999999998 2024-02-13T00:40:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.427121+0000", 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028197399999999999 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.427403+0000", 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3602e-05 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.427447+0000", 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4437999999999998e-05 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.427481+0000", 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017652670000000001 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:526 4.1e 4:7e82ab18:lc::lc.13:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.429194+0000", 2024-02-13T00:40:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.276775405, 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001962213, 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 526 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.429194+0000", 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.429194+0000", 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.429193+0000", 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9589999999999995e-06 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.429200+0000", 2024-02-13T00:40:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2670999999999999e-05 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.429213+0000", 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7720000000000002e-06 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.429218+0000", 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3170000000000007e-05 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.429281+0000", 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4978e-05 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.429346+0000", 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016773299999999999 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.429514+0000", 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00047893399999999998 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.429993+0000", 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051092300000000004 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.430504+0000", 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00057476700000000001 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.431078+0000", 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4789000000000001e-05 2024-02-13T00:40:30.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.431123+0000", 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2521000000000003e-05 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.431156+0000", 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001962213 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:570 4.b 4:d5e0c857:lc::lc.5:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.442124+0000", 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.26384487999999, 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020717019999999999, 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 570 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.812 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.442124+0000", 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.442124+0000", 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.442124+0000", 2024-02-13T00:40:30.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1500000000000002e-06 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.442126+0000", 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0869999999999998e-06 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.442129+0000", 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1260000000000001e-06 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.442131+0000", 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049585000000000005 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.442627+0000", 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7174000000000003e-05 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.442704+0000", 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014070099999999999 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.442845+0000", 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030113600000000003 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.443146+0000", 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030108899999999999 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.443447+0000", 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00066687199999999999 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.444114+0000", 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6705000000000001e-05 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.444161+0000", 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5204999999999999e-05 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.444196+0000", 2024-02-13T00:40:30.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020717019999999999 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24377.0:646 4.19 e64/58 4:9dc32cef:lc::lc.19:head v 64'20, mlcod=64'18)", 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.462889+0000", 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.243080128, 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029707510000000002, 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.462889+0000", 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.462889+0000", 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.462888+0000", 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.499e-06 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.462897+0000", 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1875999999999999e-05 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.462909+0000", 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8420000000000004e-06 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.462914+0000", 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7095000000000002e-05 2024-02-13T00:40:30.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.462961+0000", 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2486999999999999e-05 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.462973+0000", 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002855364 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.465829+0000", 2024-02-13T00:40:30.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0803999999999998e-05 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.465860+0000", 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029707510000000002 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:695 4.16 4:6d4431ed:lc::lc.7:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.474569+0000", 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.23139952899999, 2024-02-13T00:40:30.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036234549999999998, 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 695 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.474569+0000", 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.474569+0000", 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.474569+0000", 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.485e-06 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.474574+0000", 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4790000000000002e-06 2024-02-13T00:40:30.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.474583+0000", 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3510000000000001e-06 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.474589+0000", 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6637e-05 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.474645+0000", 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5196000000000003e-05 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.474721+0000", 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020450500000000001 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.474925+0000", 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032755600000000001 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.475253+0000", 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00044540600000000002 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.475698+0000", 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024195240000000002 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.478118+0000", 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1365e-05 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.478169+0000", 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3833999999999999e-05 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.478193+0000", 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036234549999999998 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.828 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:926 4.1e 4:7e82ab18:lc::lc.13:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:53.318613+0000", 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 157.387355453, 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023181199999999999, 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 926 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.318613+0000", 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.318613+0000", 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.318611+0000", 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1494e-05 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.318622+0000", 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0837000000000001e-05 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.318643+0000", 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8170000000000005e-06 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.318652+0000", 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6484999999999998e-05 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.318719+0000", 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130357 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.318849+0000", 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023321600000000001 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.319082+0000", 2024-02-13T00:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00066910699999999995 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.319751+0000", 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010648980000000001 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.320816+0000", 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5122e-05 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.320831+0000", 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7518000000000001e-05 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.320889+0000", 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2778999999999998e-05 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.320932+0000", 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023181199999999999 2024-02-13T00:40:30.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.835 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.835 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.835 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:30.835 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:30.835 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:40:30.835 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:30.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.839+0000 7fe0137fe700 1 -- 172.21.15.162:0/3234025212 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v94) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe018051360 con 0x7fe01c0f8d10 2024-02-13T00:40:30.844 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.841+0000 7fe024643700 10 librados: watch_flush enter 2024-02-13T00:40:30.844 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.841+0000 7fe024643700 10 librados: watch_flush exit 2024-02-13T00:40:30.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.841+0000 7fe024643700 20 client.24710.objecter shutdown clearing up homeless session... 2024-02-13T00:40:30.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.841+0000 7fe024643700 10 client.24710.objecter successfully canceled tick 2024-02-13T00:40:30.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 -- 172.21.15.162:0/3234025212 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe008062ce0 msgr2=0x7fe0080651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:30.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 --2- 172.21.15.162:0/3234025212 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe008062ce0 0x7fe0080651a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fe01c0f8460 tx=0x7fe00c00b040).stop 2024-02-13T00:40:30.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 -- 172.21.15.162:0/3234025212 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe01c0f8d10 msgr2=0x7fe01c0f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:30.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 --2- 172.21.15.162:0/3234025212 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe01c0f8d10 0x7fe01c0f6cc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fe0180043d0 tx=0x7fe018009cb0).stop 2024-02-13T00:40:30.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 -- 172.21.15.162:0/3234025212 shutdown_connections 2024-02-13T00:40:30.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 --2- 172.21.15.162:0/3234025212 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe008062ce0 0x7fe0080651a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:30.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 --2- 172.21.15.162:0/3234025212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe01c102270 0x7fe01c0f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:30.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 --2- 172.21.15.162:0/3234025212 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe01c0f8d10 0x7fe01c0f6cc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:30.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 --2- 172.21.15.162:0/3234025212 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe01c0f95e0 0x7fe01c0f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:30.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 -- 172.21.15.162:0/3234025212 >> 172.21.15.162:0/3234025212 conn(0x7fe01c06eb90 msgr2=0x7fe01c102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:30.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 -- 172.21.15.162:0/3234025212 shutdown_connections 2024-02-13T00:40:30.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 -- 172.21.15.162:0/3234025212 wait complete. 2024-02-13T00:40:30.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:30.842+0000 7fe024643700 1 librados: shutdown 2024-02-13T00:40:30.852 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:40:31.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:30.684992+0000 mon.c (mon.2) 83 : audit [INF] from='client.? 172.21.15.162:0/3234025212' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:40:31.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:30.685515+0000 mon.a (mon.0) 731 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:40:31.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:30.684992+0000 mon.c (mon.2) 83 : audit [INF] from='client.? 172.21.15.162:0/3234025212' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:40:31.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:30.685515+0000 mon.a (mon.0) 731 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:40:31.190 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:31.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:31.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:30.684992+0000 mon.c (mon.2) 83 : audit [INF] from='client.? 172.21.15.162:0/3234025212' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:40:31.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:30.685515+0000 mon.a (mon.0) 731 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:40:31.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:31.328 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:31.348 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:31.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:31.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:31.479 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:31.498 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:31.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:31.602 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:31.628 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:31.647 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:31.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:31.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:31.782 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:32.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:30.194193+0000 mgr.y (mgr.14544) 115 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:32.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:30.825542+0000 mon.a (mon.0) 732 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:40:32.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:30.835847+0000 mon.a (mon.0) 733 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:40:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:30.835918+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-13T00:40:32.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:30.194193+0000 mgr.y (mgr.14544) 115 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:32.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:30.825542+0000 mon.a (mon.0) 732 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:40:32.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:30.835847+0000 mon.a (mon.0) 733 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:40:32.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:30.835918+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-13T00:40:32.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:30.194193+0000 mgr.y (mgr.14544) 115 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:32.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:30.825542+0000 mon.a (mon.0) 732 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:40:32.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:30.835847+0000 mon.a (mon.0) 733 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:40:32.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:30.835918+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-13T00:40:32.542 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:32.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:32.647 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:32.751 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:32.909 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:32.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:33.034 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:33.069 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:33.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:33.101 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:33.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:33.202 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:33.233 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:33.256 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:33.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:33.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:33.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:33.392 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:34.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:32.194583+0000 mgr.y (mgr.14544) 116 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:34.096 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:32.194583+0000 mgr.y (mgr.14544) 116 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:34.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-13T00:40:34.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.885+0000 7f5300a51700 1 -- 172.21.15.162:0/768192114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52fc0fd490 msgr2=0x7f52fc0f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:34.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.885+0000 7f5300a51700 1 --2- 172.21.15.162:0/768192114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52fc0fd490 0x7f52fc0f6cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f52f0004870 tx=0x7f52f0009cb0).stop 2024-02-13T00:40:34.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.885+0000 7f5300a51700 1 -- 172.21.15.162:0/768192114 shutdown_connections 2024-02-13T00:40:34.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.885+0000 7f5300a51700 1 --2- 172.21.15.162:0/768192114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52fc0f73a0 0x7f52fc1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:34.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.885+0000 7f5300a51700 1 --2- 172.21.15.162:0/768192114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52fc0fcb30 0x7f52fc0fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:34.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.885+0000 7f5300a51700 1 --2- 172.21.15.162:0/768192114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52fc0fd490 0x7f52fc0f6cc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:34.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.885+0000 7f5300a51700 1 -- 172.21.15.162:0/768192114 >> 172.21.15.162:0/768192114 conn(0x7f52fc06eb90 msgr2=0x7f52fc0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:34.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.885+0000 7f5300a51700 1 -- 172.21.15.162:0/768192114 shutdown_connections 2024-02-13T00:40:34.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.885+0000 7f5300a51700 1 -- 172.21.15.162:0/768192114 wait complete. 2024-02-13T00:40:34.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.886+0000 7f5300a51700 1 librados: starting msgr at 2024-02-13T00:40:34.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.886+0000 7f5300a51700 1 librados: starting objecter 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.886+0000 7f5300a51700 1 Processor -- start 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.886+0000 7f5300a51700 1 -- start start 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.886+0000 7f5300a51700 1 librados: setting wanted keys 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.886+0000 7f5300a51700 1 librados: calling monclient init 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.886+0000 7f5300a51700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52fc0f73a0 0x7f52fc107140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.886+0000 7f5300a51700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52fc0fcb30 0x7f52fc107630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f5300a51700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52fc0fd490 0x7f52fc107b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f5300a51700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc1167c0 con 0x7f52fc0fd490 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f5300a51700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc116900 con 0x7f52fc0f73a0 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f5300a51700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc116a70 con 0x7f52fc0fcb30 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52f9d9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52fc0fcb30 0x7f52fc107630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52f9d9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52fc0fcb30 0x7f52fc107630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:44144/0 (socket says 172.21.15.162:44144) 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52f9d9b700 1 -- 172.21.15.162:0/1062994006 learned_addr learned my addr 172.21.15.162:0/1062994006 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52fad9d700 1 --2- 172.21.15.162:0/1062994006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52fc0fd490 0x7f52fc107b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:34.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52fa59c700 1 --2- 172.21.15.162:0/1062994006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52fc0f73a0 0x7f52fc107140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52f9d9b700 1 -- 172.21.15.162:0/1062994006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52fc0f73a0 msgr2=0x7f52fc107140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52f9d9b700 1 --2- 172.21.15.162:0/1062994006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52fc0f73a0 0x7f52fc107140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52f9d9b700 1 -- 172.21.15.162:0/1062994006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52fc0fd490 msgr2=0x7f52fc107b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52f9d9b700 1 --2- 172.21.15.162:0/1062994006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52fc0fd490 0x7f52fc107b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52f9d9b700 1 -- 172.21.15.162:0/1062994006 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f0005040 con 0x7f52fc0fcb30 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52f9d9b700 1 --2- 172.21.15.162:0/1062994006 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52fc0fcb30 0x7f52fc107630 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f52f0009eb0 tx=0x7f52f000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52eb7fe700 10 client.?.objecter ms_handle_connect 0x7f52fc0fcb30 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52eb7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52eb7fe700 1 -- 172.21.15.162:0/1062994006 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f000e070 con 0x7f52fc0fcb30 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f5300a51700 1 -- 172.21.15.162:0/1062994006 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52fc116cf0 con 0x7f52fc0fcb30 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52eb7fe700 1 -- 172.21.15.162:0/1062994006 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52f0013da0 con 0x7f52fc0fcb30 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.887+0000 7f52eb7fe700 1 -- 172.21.15.162:0/1062994006 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f0019070 con 0x7f52fc0fcb30 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.888+0000 7f5300a51700 10 client.24716.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:34.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.888+0000 7f5300a51700 1 -- 172.21.15.162:0/1062994006 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52fc117150 con 0x7f52fc0fcb30 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.888+0000 7f5300a51700 1 librados: init done 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.888+0000 7f5300a51700 1 -- 172.21.15.162:0/1062994006 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52fc0475e0 con 0x7f52fc0fcb30 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.890+0000 7f52eb7fe700 1 -- 172.21.15.162:0/1062994006 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f52f001e420 con 0x7f52fc0fcb30 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.890+0000 7f52eb7fe700 1 --2- 172.21.15.162:0/1062994006 >> v2:172.21.15.162:6800/2866534053 conn(0x7f52e4062c90 0x7f52e4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.890+0000 7f52eb7fe700 1 -- 172.21.15.162:0/1062994006 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(94..94 src has 41..94) v4 ==== 6204+0+0 (secure 0 0 0) 0x7f52f007d990 con 0x7f52fc0fcb30 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.890+0000 7f52eb7fe700 10 client.24716.objecter ms_dispatch 0x7f52fc001550 osd_map(94..94 src has 41..94) v4 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.890+0000 7f52eb7fe700 3 client.24716.objecter handle_osd_map got epochs [94,94] > 0 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.890+0000 7f52eb7fe700 3 client.24716.objecter handle_osd_map decoding full epoch 94 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.891+0000 7f52eb7fe700 20 client.24716.objecter dump_active .. 0 homeless 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.891+0000 7f52fa59c700 1 --2- 172.21.15.162:0/1062994006 >> v2:172.21.15.162:6800/2866534053 conn(0x7f52e4062c90 0x7f52e4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.891+0000 7f52fa59c700 1 --2- 172.21.15.162:0/1062994006 >> v2:172.21.15.162:6800/2866534053 conn(0x7f52e4062c90 0x7f52e4065150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f52ec0048b0 tx=0x7f52ec00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.891+0000 7f52eb7fe700 10 client.24716.objecter ms_handle_connect 0x7f52e4062c90 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:33.896+0000 7f52eb7fe700 1 -- 172.21.15.162:0/1062994006 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f52f00515a0 con 0x7f52fc0fcb30 2024-02-13T00:40:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.131+0000 7f5300a51700 1 -- 172.21.15.162:0/1062994006 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} v 0) v1 -- 0x7f52fc05fb80 con 0x7f52fc0fcb30 2024-02-13T00:40:34.247 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:34.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:34.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:32.194583+0000 mgr.y (mgr.14544) 116 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:34.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:34.387 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:34.408 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:34.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:34.461 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:34.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:34.543 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:34.563 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:34.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:34.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:34.699 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:34.738 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:34.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:34.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:34.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.849+0000 7f52eb7fe700 1 -- 172.21.15.162:0/1062994006 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]=0 set osd.4 primary-affinity to 0 (802) v95) v1 ==== 129+0+0 (secure 0 0 0) 0x7f52f0051190 con 0x7f52fc0fcb30 2024-02-13T00:40:34.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 10 librados: watch_flush enter 2024-02-13T00:40:34.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 10 librados: watch_flush exit 2024-02-13T00:40:34.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 20 client.24716.objecter shutdown clearing up homeless session... 2024-02-13T00:40:34.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 10 client.24716.objecter successfully canceled tick 2024-02-13T00:40:34.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 1 -- 172.21.15.162:0/1062994006 >> v2:172.21.15.162:6800/2866534053 conn(0x7f52e4062c90 msgr2=0x7f52e4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:34.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 1 --2- 172.21.15.162:0/1062994006 >> v2:172.21.15.162:6800/2866534053 conn(0x7f52e4062c90 0x7f52e4065150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f52ec0048b0 tx=0x7f52ec00b040).stop 2024-02-13T00:40:34.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 1 -- 172.21.15.162:0/1062994006 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52fc0fcb30 msgr2=0x7f52fc107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:34.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 1 --2- 172.21.15.162:0/1062994006 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52fc0fcb30 0x7f52fc107630 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f52f0009eb0 tx=0x7f52f000b040).stop 2024-02-13T00:40:34.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 1 -- 172.21.15.162:0/1062994006 shutdown_connections 2024-02-13T00:40:34.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 1 --2- 172.21.15.162:0/1062994006 >> v2:172.21.15.162:6800/2866534053 conn(0x7f52e4062c90 0x7f52e4065150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:34.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 1 --2- 172.21.15.162:0/1062994006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52fc0fd490 0x7f52fc107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:34.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 1 --2- 172.21.15.162:0/1062994006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52fc0f73a0 0x7f52fc107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:34.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 1 --2- 172.21.15.162:0/1062994006 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52fc0fcb30 0x7f52fc107630 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0x7f52f0009eb0 tx=0x7f52f000b040).stop 2024-02-13T00:40:34.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.853+0000 7f5300a51700 1 -- 172.21.15.162:0/1062994006 >> 172.21.15.162:0/1062994006 conn(0x7f52fc06eb90 msgr2=0x7f52fc0fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:34.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.861+0000 7f5300a51700 1 -- 172.21.15.162:0/1062994006 shutdown_connections 2024-02-13T00:40:34.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.865+0000 7f5300a51700 1 -- 172.21.15.162:0/1062994006 wait complete. 2024-02-13T00:40:34.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:34.865+0000 7f5300a51700 1 librados: shutdown 2024-02-13T00:40:34.870 INFO:teuthology.orchestra.run.smithi162.stderr:set osd.4 primary-affinity to 0 (802) 2024-02-13T00:40:34.879 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:34.900 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:34.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:34.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:34.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:40:34.964 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:34.133019+0000 mon.c (mon.2) 84 : audit [INF] from='client.? 172.21.15.162:0/1062994006' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-02-13T00:40:34.965 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:34.133347+0000 mon.a (mon.0) 735 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-02-13T00:40:34.965 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:34.133019+0000 mon.c (mon.2) 84 : audit [INF] from='client.? 172.21.15.162:0/1062994006' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-02-13T00:40:34.965 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:34.133347+0000 mon.a (mon.0) 735 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-02-13T00:40:35.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:35.033 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:35.053 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:35.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:35.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:35.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:35.183 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:34.133019+0000 mon.c (mon.2) 84 : audit [INF] from='client.? 172.21.15.162:0/1062994006' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-02-13T00:40:35.183 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:34.133347+0000 mon.a (mon.0) 735 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-02-13T00:40:35.185 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:35.204 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:35.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:35.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:35.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:35.335 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:35.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:40:35.354 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:35.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:35.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:35.458 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:35.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:40:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:40:35.484 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:35.502 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:35.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:35.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:35.642 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:36.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:34.194972+0000 mgr.y (mgr.14544) 117 : cluster [DBG] pgmap v104: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:36.029 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:34.838985+0000 mon.a (mon.0) 736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-02-13T00:40:36.029 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:34.839055+0000 mon.a (mon.0) 737 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-13T00:40:36.029 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:35.847456+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-13T00:40:36.030 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:34.194972+0000 mgr.y (mgr.14544) 117 : cluster [DBG] pgmap v104: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:36.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:34.838985+0000 mon.a (mon.0) 736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-02-13T00:40:36.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:34.839055+0000 mon.a (mon.0) 737 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-13T00:40:36.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:35.847456+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-13T00:40:36.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:34.194972+0000 mgr.y (mgr.14544) 117 : cluster [DBG] pgmap v104: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:40:36.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:34.838985+0000 mon.a (mon.0) 736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-02-13T00:40:36.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:34.839055+0000 mon.a (mon.0) 737 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-13T00:40:36.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:35.847456+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-13T00:40:36.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:40:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:40:36] "GET /metrics HTTP/1.1" 200 172261 "" "Prometheus/2.33.4" 2024-02-13T00:40:36.593 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:36.600 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:36.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:36.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:36.630 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:36.706 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:36.744 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:36.877 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:36.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:36.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:37.016 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:37.123 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:37.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:37.131 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:35.299227+0000 mgr.y (mgr.14544) 118 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:37.132 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:35.299227+0000 mgr.y (mgr.14544) 118 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:37.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:37.259 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:37.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:35.299227+0000 mgr.y (mgr.14544) 118 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:37.422 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:40:37.422 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:40:37.423 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-13T00:40:37.423 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:40:37.423 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:37.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:37.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:37.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:37.559 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:37.581 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:37.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:37.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:37.721 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:37.753 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:37.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:37.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:37.892 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:37.914 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:37.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:37.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:38.017 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:38.046 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:38.065 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:38.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:38.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:36.195503+0000 mgr.y (mgr.14544) 119 : cluster [DBG] pgmap v107: 132 pgs: 19 peering, 113 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:40:38.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:36.871430+0000 mon.a (mon.0) 739 : cluster [WRN] Health check failed: Reduced data availability: 7 pgs inactive, 17 pgs peering (PG_AVAILABILITY) 2024-02-13T00:40:38.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:36.195503+0000 mgr.y (mgr.14544) 119 : cluster [DBG] pgmap v107: 132 pgs: 19 peering, 113 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:40:38.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:36.871430+0000 mon.a (mon.0) 739 : cluster [WRN] Health check failed: Reduced data availability: 7 pgs inactive, 17 pgs peering (PG_AVAILABILITY) 2024-02-13T00:40:38.169 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:38.194 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:36.195503+0000 mgr.y (mgr.14544) 119 : cluster [DBG] pgmap v107: 132 pgs: 19 peering, 113 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:40:38.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:36.871430+0000 mon.a (mon.0) 739 : cluster [WRN] Health check failed: Reduced data availability: 7 pgs inactive, 17 pgs peering (PG_AVAILABILITY) 2024-02-13T00:40:38.196 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:38.215 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:38.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:38.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:38.319 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:38.351 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:38.377 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:38.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:38.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:38.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:38.508 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:38.527 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:38.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:38.630 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:38.658 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:38.677 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:38.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:38.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:38.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:38.794 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:40:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:40:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:40:38.813 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:39.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.434+0000 7feeb994e700 1 -- 172.21.15.162:0/2686480457 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feeb40f6930 msgr2=0x7feeb4104ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:39.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.434+0000 7feeb994e700 1 --2- 172.21.15.162:0/2686480457 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feeb40f6930 0x7feeb4104ad0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7feeac009c30 tx=0x7feeac004c50).stop 2024-02-13T00:40:39.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.435+0000 7feeb994e700 1 -- 172.21.15.162:0/2686480457 shutdown_connections 2024-02-13T00:40:39.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.435+0000 7feeb994e700 1 --2- 172.21.15.162:0/2686480457 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feeb40f6930 0x7feeb4104ad0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:39.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.435+0000 7feeb994e700 1 --2- 172.21.15.162:0/2686480457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feeb40fdcd0 0x7feeb40fe0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:39.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.435+0000 7feeb994e700 1 --2- 172.21.15.162:0/2686480457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feeb40fe630 0x7feeb40f6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:39.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.435+0000 7feeb994e700 1 -- 172.21.15.162:0/2686480457 >> 172.21.15.162:0/2686480457 conn(0x7feeb4070060 msgr2=0x7feeb40f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:39.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.435+0000 7feeb994e700 1 -- 172.21.15.162:0/2686480457 shutdown_connections 2024-02-13T00:40:39.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.435+0000 7feeb994e700 1 -- 172.21.15.162:0/2686480457 wait complete. 2024-02-13T00:40:39.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.435+0000 7feeb994e700 1 librados: starting msgr at 2024-02-13T00:40:39.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.435+0000 7feeb994e700 1 librados: starting objecter 2024-02-13T00:40:39.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.436+0000 7feeb994e700 1 Processor -- start 2024-02-13T00:40:39.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.436+0000 7feeb994e700 1 -- start start 2024-02-13T00:40:39.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.436+0000 7feeb994e700 1 librados: setting wanted keys 2024-02-13T00:40:39.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.436+0000 7feeb994e700 1 librados: calling monclient init 2024-02-13T00:40:39.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb994e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feeb40f6930 0x7feeb410e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:39.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb994e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feeb40fdcd0 0x7feeb410e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:39.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb994e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feeb40fe630 0x7feeb4108200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:39.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb994e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb41086f0 con 0x7feeb40f6930 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb994e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb4108860 con 0x7feeb40fe630 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb994e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb41089d0 con 0x7feeb40fdcd0 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feeb40f6930 0x7feeb410e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feeb40f6930 0x7feeb410e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:47558/0 (socket says 172.21.15.162:47558) 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb2ffd700 1 -- 172.21.15.162:0/4231587686 learned_addr learned my addr 172.21.15.162:0/4231587686 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb27fc700 1 --2- 172.21.15.162:0/4231587686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feeb40fdcd0 0x7feeb410e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb2ffd700 1 -- 172.21.15.162:0/4231587686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feeb40fdcd0 msgr2=0x7feeb410e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb2ffd700 1 --2- 172.21.15.162:0/4231587686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feeb40fdcd0 0x7feeb410e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb2ffd700 1 -- 172.21.15.162:0/4231587686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feeb40fe630 msgr2=0x7feeb4108200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb2ffd700 1 --2- 172.21.15.162:0/4231587686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feeb40fe630 0x7feeb4108200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb2ffd700 1 -- 172.21.15.162:0/4231587686 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac0097c0 con 0x7feeb40f6930 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb27fc700 1 --2- 172.21.15.162:0/4231587686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feeb40fdcd0 0x7feeb410e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:40:39.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7feeb2ffd700 1 --2- 172.21.15.162:0/4231587686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feeb40f6930 0x7feeb410e020 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7feea400c810 tx=0x7feea4007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:39.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7fee9bfff700 10 client.?.objecter ms_handle_connect 0x7feeb40f6930 2024-02-13T00:40:39.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.437+0000 7fee9bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:39.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.438+0000 7fee9bfff700 1 -- 172.21.15.162:0/4231587686 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4010510 con 0x7feeb40f6930 2024-02-13T00:40:39.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.438+0000 7fee9bfff700 1 -- 172.21.15.162:0/4231587686 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feea4010b40 con 0x7feeb40f6930 2024-02-13T00:40:39.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.438+0000 7feeb994e700 1 -- 172.21.15.162:0/4231587686 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb4108cb0 con 0x7feeb40f6930 2024-02-13T00:40:39.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.438+0000 7fee9bfff700 1 -- 172.21.15.162:0/4231587686 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4012ba0 con 0x7feeb40f6930 2024-02-13T00:40:39.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.438+0000 7feeb994e700 10 client.14823.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:39.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.438+0000 7feeb994e700 1 -- 172.21.15.162:0/4231587686 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb4114e90 con 0x7feeb40f6930 2024-02-13T00:40:39.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.438+0000 7feeb994e700 1 librados: init done 2024-02-13T00:40:39.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.439+0000 7feeb994e700 1 -- 172.21.15.162:0/4231587686 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb405f960 con 0x7feeb40f6930 2024-02-13T00:40:39.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.440+0000 7fee9bfff700 1 -- 172.21.15.162:0/4231587686 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7feea4010670 con 0x7feeb40f6930 2024-02-13T00:40:39.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.440+0000 7fee9bfff700 1 --2- 172.21.15.162:0/4231587686 >> v2:172.21.15.162:6800/2866534053 conn(0x7fee9c062ce0 0x7fee9c0651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:39.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.440+0000 7fee9bfff700 1 -- 172.21.15.162:0/4231587686 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(96..96 src has 41..96) v4 ==== 6204+0+0 (secure 0 0 0) 0x7feea407c980 con 0x7feeb40f6930 2024-02-13T00:40:39.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.440+0000 7fee9bfff700 10 client.14823.objecter ms_dispatch 0x7feeb4001550 osd_map(96..96 src has 41..96) v4 2024-02-13T00:40:39.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.440+0000 7fee9bfff700 3 client.14823.objecter handle_osd_map got epochs [96,96] > 0 2024-02-13T00:40:39.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.440+0000 7fee9bfff700 3 client.14823.objecter handle_osd_map decoding full epoch 96 2024-02-13T00:40:39.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.440+0000 7fee9bfff700 20 client.14823.objecter dump_active .. 0 homeless 2024-02-13T00:40:39.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.444+0000 7feeb27fc700 1 --2- 172.21.15.162:0/4231587686 >> v2:172.21.15.162:6800/2866534053 conn(0x7fee9c062ce0 0x7fee9c0651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:39.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.445+0000 7feeb27fc700 1 --2- 172.21.15.162:0/4231587686 >> v2:172.21.15.162:6800/2866534053 conn(0x7fee9c062ce0 0x7fee9c0651a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7feea8009f20 tx=0x7feea800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:39.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.445+0000 7fee9bfff700 10 client.14823.objecter ms_handle_connect 0x7fee9c062ce0 2024-02-13T00:40:39.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.448+0000 7fee9bfff700 1 -- 172.21.15.162:0/4231587686 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7feea40506e0 con 0x7feeb40f6930 2024-02-13T00:40:39.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:39.649+0000 7feeb994e700 1 -- 172.21.15.162:0/4231587686 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7feeb40475e0 con 0x7feeb40f6930 2024-02-13T00:40:40.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-13T00:40:40.143 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:38.196061+0000 mgr.y (mgr.14544) 120 : cluster [DBG] pgmap v108: 132 pgs: 19 peering, 113 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:40:40.143 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:39.650745+0000 mon.a (mon.0) 740 : audit [INF] from='client.? 172.21.15.162:0/4231587686' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:40:40.143 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:38.196061+0000 mgr.y (mgr.14544) 120 : cluster [DBG] pgmap v108: 132 pgs: 19 peering, 113 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:40:40.143 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:39.650745+0000 mon.a (mon.0) 740 : audit [INF] from='client.? 172.21.15.162:0/4231587686' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:40:40.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.252+0000 7fee9bfff700 1 -- 172.21.15.162:0/4231587686 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v97) v1 ==== 91+0+0 (secure 0 0 0) 0x7feea40502d0 con 0x7feeb40f6930 2024-02-13T00:40:40.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.258+0000 7feeb994e700 10 librados: watch_flush enter 2024-02-13T00:40:40.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.259+0000 7feeb994e700 10 librados: watch_flush exit 2024-02-13T00:40:40.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.259+0000 7feeb994e700 20 client.14823.objecter shutdown clearing up homeless session... 2024-02-13T00:40:40.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.259+0000 7feeb994e700 10 client.14823.objecter successfully canceled tick 2024-02-13T00:40:40.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.259+0000 7feeb994e700 1 -- 172.21.15.162:0/4231587686 >> v2:172.21.15.162:6800/2866534053 conn(0x7fee9c062ce0 msgr2=0x7fee9c0651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:40.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.259+0000 7feeb994e700 1 --2- 172.21.15.162:0/4231587686 >> v2:172.21.15.162:6800/2866534053 conn(0x7fee9c062ce0 0x7fee9c0651a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7feea8009f20 tx=0x7feea800b040).stop 2024-02-13T00:40:40.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.259+0000 7feeb994e700 1 -- 172.21.15.162:0/4231587686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feeb40f6930 msgr2=0x7feeb410e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:40.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.259+0000 7feeb994e700 1 --2- 172.21.15.162:0/4231587686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feeb40f6930 0x7feeb410e020 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7feea400c810 tx=0x7feea4007f90).stop 2024-02-13T00:40:40.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.262+0000 7feeb994e700 1 -- 172.21.15.162:0/4231587686 shutdown_connections 2024-02-13T00:40:40.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.262+0000 7feeb994e700 1 --2- 172.21.15.162:0/4231587686 >> v2:172.21.15.162:6800/2866534053 conn(0x7fee9c062ce0 0x7fee9c0651a0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0x7feea8009f20 tx=0x7feea800b040).stop 2024-02-13T00:40:40.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.262+0000 7feeb994e700 1 --2- 172.21.15.162:0/4231587686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feeb40fe630 0x7feeb4108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:40:40.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.262+0000 7feeb994e700 1 --2- 172.21.15.162:0/4231587686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feeb40f6930 0x7feeb410e020 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:40.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.262+0000 7feeb994e700 1 --2- 172.21.15.162:0/4231587686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feeb40fdcd0 0x7feeb410e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:40.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.262+0000 7feeb994e700 1 -- 172.21.15.162:0/4231587686 >> 172.21.15.162:0/4231587686 conn(0x7feeb4070060 msgr2=0x7feeb406d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:40.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.266+0000 7feeb994e700 1 -- 172.21.15.162:0/4231587686 shutdown_connections 2024-02-13T00:40:40.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.266+0000 7feeb994e700 1 -- 172.21.15.162:0/4231587686 wait complete. 2024-02-13T00:40:40.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:40.266+0000 7feeb994e700 1 librados: shutdown 2024-02-13T00:40:40.279 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:40:40.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:38.196061+0000 mgr.y (mgr.14544) 120 : cluster [DBG] pgmap v108: 132 pgs: 19 peering, 113 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:40:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:39.650745+0000 mon.a (mon.0) 740 : audit [INF] from='client.? 172.21.15.162:0/4231587686' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:40:40.670 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:40.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:40.778 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:40.811 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:40.896 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:40.247875+0000 mon.a (mon.0) 741 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:40:40.896 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:40.252507+0000 mon.a (mon.0) 742 : audit [INF] from='client.? 172.21.15.162:0/4231587686' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:40:40.896 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:40.252573+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-13T00:40:40.896 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:40.747961+0000 osd.1 (osd.1) 57 : cluster [DBG] 6.e scrub starts 2024-02-13T00:40:40.896 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:40.750263+0000 osd.1 (osd.1) 58 : cluster [DBG] 6.e scrub ok 2024-02-13T00:40:40.913 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:40.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:41.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:41.050 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:41.051 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:40:41.051 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:40:41.052 INFO:tasks.thrashosds.thrasher:fixing pg num pool iscsi 2024-02-13T00:40:41.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:40:41.082 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:41.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:41.187 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:41.228 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:40.247875+0000 mon.a (mon.0) 741 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:40:41.228 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:40.252507+0000 mon.a (mon.0) 742 : audit [INF] from='client.? 172.21.15.162:0/4231587686' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:40:41.229 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:40.252573+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-13T00:40:41.229 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:40.747961+0000 osd.1 (osd.1) 57 : cluster [DBG] 6.e scrub starts 2024-02-13T00:40:41.229 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:40.750263+0000 osd.1 (osd.1) 58 : cluster [DBG] 6.e scrub ok 2024-02-13T00:40:41.229 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:41.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:40.247875+0000 mon.a (mon.0) 741 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:40:41.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:40.252507+0000 mon.a (mon.0) 742 : audit [INF] from='client.? 172.21.15.162:0/4231587686' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:40:41.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:40.252573+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-13T00:40:41.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:40.747961+0000 osd.1 (osd.1) 57 : cluster [DBG] 6.e scrub starts 2024-02-13T00:40:41.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:40.750263+0000 osd.1 (osd.1) 58 : cluster [DBG] 6.e scrub ok 2024-02-13T00:40:41.299 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:41.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:41.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:41.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.243+0000 7f910283f700 1 -- 172.21.15.162:0/1160044815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90fc104010 msgr2=0x7f90fc1043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.243+0000 7f910283f700 1 --2- 172.21.15.162:0/1160044815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90fc104010 0x7f90fc1043f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f90f00048a0 tx=0x7f90f0009cb0).stop 2024-02-13T00:40:41.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.244+0000 7f910283f700 1 -- 172.21.15.162:0/1160044815 shutdown_connections 2024-02-13T00:40:41.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.244+0000 7f910283f700 1 --2- 172.21.15.162:0/1160044815 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90fc069ab0 0x7f90fc069f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:40:41.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.244+0000 7f910283f700 1 --2- 172.21.15.162:0/1160044815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90fc104010 0x7f90fc1043f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:41.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.244+0000 7f910283f700 1 --2- 172.21.15.162:0/1160044815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90fc1048e0 0x7f90fc0695c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:40:41.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.244+0000 7f910283f700 1 -- 172.21.15.162:0/1160044815 >> 172.21.15.162:0/1160044815 conn(0x7f90fc063df0 msgr2=0x7f90fc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:41.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.244+0000 7f910283f700 1 -- 172.21.15.162:0/1160044815 shutdown_connections 2024-02-13T00:40:41.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.244+0000 7f910283f700 1 -- 172.21.15.162:0/1160044815 wait complete. 2024-02-13T00:40:41.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.245+0000 7f910283f700 1 librados: starting msgr at 2024-02-13T00:40:41.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.245+0000 7f910283f700 1 librados: starting objecter 2024-02-13T00:40:41.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.245+0000 7f910283f700 1 Processor -- start 2024-02-13T00:40:41.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.245+0000 7f910283f700 1 -- start start 2024-02-13T00:40:41.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.245+0000 7f910283f700 1 librados: setting wanted keys 2024-02-13T00:40:41.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.245+0000 7f910283f700 1 librados: calling monclient init 2024-02-13T00:40:41.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.246+0000 7f910283f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90fc069ab0 0x7f90fc10c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:41.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.246+0000 7f910283f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90fc104010 0x7f90fc10c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:41.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.246+0000 7f910283f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90fc1048e0 0x7f90fc12b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:41.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.246+0000 7f910283f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc12bd50 con 0x7f90fc1048e0 2024-02-13T00:40:41.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.246+0000 7f910283f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc12be90 con 0x7f90fc104010 2024-02-13T00:40:41.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.246+0000 7f910283f700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc12c050 con 0x7f90fc069ab0 2024-02-13T00:40:41.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.246+0000 7f9100ddc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90fc1048e0 0x7f90fc12b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:41.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.246+0000 7f9100ddc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90fc1048e0 0x7f90fc12b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:47586/0 (socket says 172.21.15.162:47586) 2024-02-13T00:40:41.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.246+0000 7f9100ddc700 1 -- 172.21.15.162:0/1380186660 learned_addr learned my addr 172.21.15.162:0/1380186660 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:41.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.246+0000 7f90fb7fe700 1 --2- 172.21.15.162:0/1380186660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90fc104010 0x7f90fc10c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:41.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f90fb7fe700 1 -- 172.21.15.162:0/1380186660 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90fc069ab0 msgr2=0x7f90fc10c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f90fb7fe700 1 --2- 172.21.15.162:0/1380186660 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90fc069ab0 0x7f90fc10c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:40:41.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f90fb7fe700 1 -- 172.21.15.162:0/1380186660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90fc1048e0 msgr2=0x7f90fc12b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.435 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:40:41.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f90fb7fe700 1 --2- 172.21.15.162:0/1380186660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90fc1048e0 0x7f90fc12b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:41.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f90fb7fe700 1 -- 172.21.15.162:0/1380186660 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f0005040 con 0x7f90fc104010 2024-02-13T00:40:41.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f90fb7fe700 1 --2- 172.21.15.162:0/1380186660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90fc104010 0x7f90fc10c720 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f90f400bad0 tx=0x7f90f4009780).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:41.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f90f97fa700 10 client.?.objecter ms_handle_connect 0x7f90fc104010 2024-02-13T00:40:41.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f90f97fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:41.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f400c840 con 0x7f90fc104010 2024-02-13T00:40:41.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90f400ce70 con 0x7f90fc104010 2024-02-13T00:40:41.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4015c80 con 0x7f90fc104010 2024-02-13T00:40:41.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc12c330 con 0x7f90fc104010 2024-02-13T00:40:41.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f910283f700 10 client.24643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:41.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc12c7d0 con 0x7f90fc104010 2024-02-13T00:40:41.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.247+0000 7f910283f700 1 librados: init done 2024-02-13T00:40:41.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.248+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f90fc0603a0 con 0x7f90fc104010 2024-02-13T00:40:41.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.249+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f90f4012020 con 0x7f90fc104010 2024-02-13T00:40:41.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.250+0000 7f90f97fa700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6800/2866534053 conn(0x7f90e4062ce0 0x7f90e40651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:41.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.250+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(97..97 src has 41..97) v4 ==== 6204+0+0 (secure 0 0 0) 0x7f90f407d950 con 0x7f90fc104010 2024-02-13T00:40:41.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.250+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 osd_map(97..97 src has 41..97) v4 2024-02-13T00:40:41.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.250+0000 7f90f97fa700 3 client.24643.objecter handle_osd_map got epochs [97,97] > 0 2024-02-13T00:40:41.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.250+0000 7f90f97fa700 3 client.24643.objecter handle_osd_map decoding full epoch 97 2024-02-13T00:40:41.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.250+0000 7f90f97fa700 20 client.24643.objecter dump_active .. 0 homeless 2024-02-13T00:40:41.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.250+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v97) v1 ==== 54+0+16 (secure 0 0 0) 0x7f90f40515e0 con 0x7f90fc104010 2024-02-13T00:40:41.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f910283f700 10 client.24643.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:41.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 1 2024-02-13T00:40:41.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6801/2103743338 conn(0x7f90fc059990 0x7f90fc059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:41.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc0475e0 osd=0 3 2024-02-13T00:40:41.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc0475e0 osd=0 3 2024-02-13T00:40:41.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc0475e0 osd=0 3 2024-02-13T00:40:41.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 1 2024-02-13T00:40:41.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc0475e0 osd=0 3 2024-02-13T00:40:41.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 0 1 2024-02-13T00:40:41.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc0475e0 osd=0 4 2024-02-13T00:40:41.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f910283f700 10 client.24643.objecter _send_command 1 2024-02-13T00:40:41.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90fc116060 con 0x7f90fc059990 2024-02-13T00:40:41.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f9100ddc700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6801/2103743338 conn(0x7f90fc059990 0x7f90fc059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:41.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f90f97fa700 10 client.24643.objecter ms_handle_connect 0x7f90fc059990 2024-02-13T00:40:41.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.255+0000 7f9100ddc700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6801/2103743338 conn(0x7f90fc059990 0x7f90fc059d70 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:41.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.256+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f90fc116060 con 0x7f90fc059990 2024-02-13T00:40:41.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.256+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:40:41.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.256+0000 7f90f97fa700 10 client.24643.objecter _finish_command 1 = system:0 2024-02-13T00:40:41.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.256+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc0475e0 osd=0 4 2024-02-13T00:40:41.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.256+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 0 1 2024-02-13T00:40:41.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.259+0000 7f90fbfff700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6800/2866534053 conn(0x7f90e4062ce0 0x7f90e40651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:41.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.260+0000 7f90fbfff700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6800/2866534053 conn(0x7f90e4062ce0 0x7f90e40651a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f90f00043d0 tx=0x7f90f0009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:41.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.261+0000 7f90f97fa700 10 client.24643.objecter ms_handle_connect 0x7f90e4062ce0 2024-02-13T00:40:41.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.280+0000 7f910283f700 10 client.24643.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:41.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.280+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 2 2024-02-13T00:40:41.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.280+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc0475e0 osd=0 3 2024-02-13T00:40:41.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.280+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc0475e0 osd=0 3 2024-02-13T00:40:41.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.280+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc0475e0 osd=0 3 2024-02-13T00:40:41.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.280+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 2 2024-02-13T00:40:41.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.280+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc0475e0 osd=0 3 2024-02-13T00:40:41.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.280+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 0 2 2024-02-13T00:40:41.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.280+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc0475e0 osd=0 4 2024-02-13T00:40:41.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.280+0000 7f910283f700 10 client.24643.objecter _send_command 2 2024-02-13T00:40:41.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.280+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f90fc027410 con 0x7f90fc059990 2024-02-13T00:40:41.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.281+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f90fc027410 con 0x7f90fc059990 2024-02-13T00:40:41.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.282+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:41.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.282+0000 7f90f97fa700 10 client.24643.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.282+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc0475e0 osd=0 4 2024-02-13T00:40:41.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.282+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 0 2 2024-02-13T00:40:41.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.284+0000 7f910283f700 10 client.24643.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:41.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.284+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 3 2024-02-13T00:40:41.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.284+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6805/150599780 conn(0x7f90fc102640 0x7f90fc102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:41.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.284+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc028790 osd=1 3 2024-02-13T00:40:41.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.284+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc028790 osd=1 3 2024-02-13T00:40:41.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.284+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc028790 osd=1 3 2024-02-13T00:40:41.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.284+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 3 2024-02-13T00:40:41.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.284+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc028790 osd=1 3 2024-02-13T00:40:41.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.284+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 1 3 2024-02-13T00:40:41.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.284+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc028790 osd=1 4 2024-02-13T00:40:41.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.284+0000 7f910283f700 10 client.24643.objecter _send_command 3 2024-02-13T00:40:41.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.284+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90fc1331d0 con 0x7f90fc102640 2024-02-13T00:40:41.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.286+0000 7f90fbfff700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6805/150599780 conn(0x7f90fc102640 0x7f90fc102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:41.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.286+0000 7f90f97fa700 10 client.24643.objecter ms_handle_connect 0x7f90fc102640 2024-02-13T00:40:41.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.286+0000 7f90fbfff700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6805/150599780 conn(0x7f90fc102640 0x7f90fc102a50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:41.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.293+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f90fc1331d0 con 0x7f90fc102640 2024-02-13T00:40:41.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.293+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:40:41.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.293+0000 7f90f97fa700 10 client.24643.objecter _finish_command 3 = system:0 2024-02-13T00:40:41.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.293+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc028790 osd=1 4 2024-02-13T00:40:41.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.293+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 1 3 2024-02-13T00:40:41.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.317+0000 7f910283f700 10 client.24643.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:41.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.317+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 4 2024-02-13T00:40:41.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.317+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc028790 osd=1 3 2024-02-13T00:40:41.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.317+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc028790 osd=1 3 2024-02-13T00:40:41.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.317+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc028790 osd=1 3 2024-02-13T00:40:41.446 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:40:41.446 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:40:41.446 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:40:41.446 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:40:41.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.317+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 4 2024-02-13T00:40:41.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.317+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc028790 osd=1 3 2024-02-13T00:40:41.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.317+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 1 4 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.317+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc028790 osd=1 4 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.317+0000 7f910283f700 10 client.24643.objecter _send_command 4 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.317+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f90fc1020b0 con 0x7f90fc102640 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.328+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f90fc1020b0 con 0x7f90fc102640 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.328+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.328+0000 7f90f97fa700 10 client.24643.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.328+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc028790 osd=1 4 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.328+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 1 4 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.329+0000 7f910283f700 10 client.24643.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.329+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 5 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.329+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6809/3689719376 conn(0x7f90fc0282f0 0x7f90fc131bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.329+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc1024c0 osd=2 3 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.329+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc1024c0 osd=2 3 2024-02-13T00:40:41.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.329+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc1024c0 osd=2 3 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.329+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 5 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.329+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc1024c0 osd=2 3 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.329+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 2 5 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.329+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc1024c0 osd=2 4 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.329+0000 7f910283f700 10 client.24643.objecter _send_command 5 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.329+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90fc1320e0 con 0x7f90fc0282f0 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.350+0000 7f90fb7fe700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6809/3689719376 conn(0x7f90fc0282f0 0x7f90fc131bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.350+0000 7f90fb7fe700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6809/3689719376 conn(0x7f90fc0282f0 0x7f90fc131bf0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.350+0000 7f90f97fa700 10 client.24643.objecter ms_handle_connect 0x7f90fc0282f0 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.351+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f90fc1320e0 con 0x7f90fc0282f0 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.351+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.351+0000 7f90f97fa700 10 client.24643.objecter _finish_command 5 = system:0 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.351+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc1024c0 osd=2 4 2024-02-13T00:40:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.351+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 2 5 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.372+0000 7f910283f700 10 client.24643.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.372+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 6 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.372+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc1024c0 osd=2 3 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.372+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc1024c0 osd=2 3 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.372+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc1024c0 osd=2 3 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.372+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 6 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.372+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc1024c0 osd=2 3 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.372+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 2 6 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.372+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc1024c0 osd=2 4 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.372+0000 7f910283f700 10 client.24643.objecter _send_command 6 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.372+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f90fc027850 con 0x7f90fc0282f0 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.374+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f90fc027850 con 0x7f90fc0282f0 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.374+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.374+0000 7f90f97fa700 10 client.24643.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.374+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc1024c0 osd=2 4 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.374+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 2 6 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f910283f700 10 client.24643.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 7 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6813/604971011 conn(0x7f90fc135350 0x7f90fc137740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc027c00 osd=3 3 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc027c00 osd=3 3 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc027c00 osd=3 3 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 7 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc027c00 osd=3 3 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 3 7 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc027c00 osd=3 4 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f910283f700 10 client.24643.objecter _send_command 7 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90fc13a390 con 0x7f90fc135350 2024-02-13T00:40:41.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f9100ddc700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6813/604971011 conn(0x7f90fc135350 0x7f90fc137740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f9100ddc700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6813/604971011 conn(0x7f90fc135350 0x7f90fc137740 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.375+0000 7f90f97fa700 10 client.24643.objecter ms_handle_connect 0x7f90fc135350 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.376+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f90fc13a390 con 0x7f90fc135350 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.376+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.376+0000 7f90f97fa700 10 client.24643.objecter _finish_command 7 = system:0 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.376+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc027c00 osd=3 4 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.376+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 3 7 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.392+0000 7f910283f700 10 client.24643.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.392+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 8 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.392+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc027c00 osd=3 3 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.392+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc027c00 osd=3 3 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.392+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc027c00 osd=3 3 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.392+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 8 2024-02-13T00:40:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.392+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc027c00 osd=3 3 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.392+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 3 8 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.392+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc027c00 osd=3 4 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.392+0000 7f910283f700 10 client.24643.objecter _send_command 8 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.392+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f90fc132540 con 0x7f90fc135350 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f90fc132540 con 0x7f90fc135350 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f90f97fa700 10 client.24643.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc027c00 osd=3 4 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 3 8 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f910283f700 10 client.24643.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 9 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6800/3344263388 conn(0x7f90fc1326e0 0x7f90fc138db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc132540 osd=4 3 2024-02-13T00:40:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc132540 osd=4 3 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc132540 osd=4 3 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 9 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc132540 osd=4 3 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 4 9 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc132540 osd=4 4 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f910283f700 10 client.24643.objecter _send_command 9 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.395+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90fc131920 con 0x7f90fc1326e0 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.396+0000 7f90fbfff700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6800/3344263388 conn(0x7f90fc1326e0 0x7f90fc138db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.396+0000 7f90f97fa700 10 client.24643.objecter ms_handle_connect 0x7f90fc1326e0 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.396+0000 7f90fbfff700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6800/3344263388 conn(0x7f90fc1326e0 0x7f90fc138db0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.397+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f90fc131920 con 0x7f90fc1326e0 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.397+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.397+0000 7f90f97fa700 10 client.24643.objecter _finish_command 9 = system:0 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.397+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc132540 osd=4 4 2024-02-13T00:40:41.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.397+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 4 9 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.412+0000 7f910283f700 10 client.24643.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.412+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 10 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.412+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc132540 osd=4 3 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.412+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc132540 osd=4 3 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.412+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc132540 osd=4 3 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.412+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 10 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.412+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc132540 osd=4 3 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.412+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 4 10 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.412+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc132540 osd=4 4 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.412+0000 7f910283f700 10 client.24643.objecter _send_command 10 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.412+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f90fc131920 con 0x7f90fc1326e0 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f90fc131920 con 0x7f90fc1326e0 2024-02-13T00:40:41.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f90f97fa700 10 client.24643.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc132540 osd=4 4 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 4 10 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f910283f700 10 client.24643.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 11 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6804/2212296748 conn(0x7f90fc1397e0 0x7f90fc139c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc131920 osd=5 3 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc131920 osd=5 3 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc131920 osd=5 3 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 11 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc131920 osd=5 3 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 5 11 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc131920 osd=5 4 2024-02-13T00:40:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f910283f700 10 client.24643.objecter _send_command 11 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90fc140c80 con 0x7f90fc1397e0 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.414+0000 7f90fb7fe700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6804/2212296748 conn(0x7f90fc1397e0 0x7f90fc139c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.415+0000 7f90f97fa700 10 client.24643.objecter ms_handle_connect 0x7f90fc1397e0 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.415+0000 7f90fb7fe700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6804/2212296748 conn(0x7f90fc1397e0 0x7f90fc139c20 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.415+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f90fc140c80 con 0x7f90fc1397e0 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.415+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.415+0000 7f90f97fa700 10 client.24643.objecter _finish_command 11 = system:0 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.415+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc131920 osd=5 4 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.415+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 5 11 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.430+0000 7f910283f700 10 client.24643.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.430+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 12 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.430+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc131920 osd=5 3 2024-02-13T00:40:41.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.430+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc131920 osd=5 3 2024-02-13T00:40:41.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.430+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc131920 osd=5 3 2024-02-13T00:40:41.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.430+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 12 2024-02-13T00:40:41.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.430+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc131920 osd=5 3 2024-02-13T00:40:41.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.430+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 5 12 2024-02-13T00:40:41.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.430+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc131920 osd=5 4 2024-02-13T00:40:41.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.430+0000 7f910283f700 10 client.24643.objecter _send_command 12 2024-02-13T00:40:41.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.430+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f90fc13e520 con 0x7f90fc1397e0 2024-02-13T00:40:41.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f90fc13e520 con 0x7f90fc1397e0 2024-02-13T00:40:41.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:41.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f90f97fa700 10 client.24643.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc131920 osd=5 4 2024-02-13T00:40:41.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 5 12 2024-02-13T00:40:41.460 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:40:41.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f910283f700 10 client.24643.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:41.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 13 2024-02-13T00:40:41.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6808/1506640341 conn(0x7f90fc13ec30 0x7f90fc141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:41.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc13e520 osd=6 3 2024-02-13T00:40:41.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc13e520 osd=6 3 2024-02-13T00:40:41.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc13e520 osd=6 3 2024-02-13T00:40:41.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 13 2024-02-13T00:40:41.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc13e520 osd=6 3 2024-02-13T00:40:41.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 6 13 2024-02-13T00:40:41.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc13e520 osd=6 4 2024-02-13T00:40:41.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f910283f700 10 client.24643.objecter _send_command 13 2024-02-13T00:40:41.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90fc143dd0 con 0x7f90fc13ec30 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.432+0000 7f9100ddc700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6808/1506640341 conn(0x7f90fc13ec30 0x7f90fc141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.433+0000 7f90f97fa700 10 client.24643.objecter ms_handle_connect 0x7f90fc13ec30 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.433+0000 7f9100ddc700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6808/1506640341 conn(0x7f90fc13ec30 0x7f90fc141020 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.434+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f90fc143dd0 con 0x7f90fc13ec30 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.434+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.434+0000 7f90f97fa700 10 client.24643.objecter _finish_command 13 = system:0 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.434+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc13e520 osd=6 4 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.434+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 6 13 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.448+0000 7f910283f700 10 client.24643.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.448+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 14 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.448+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc13e520 osd=6 3 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.448+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc13e520 osd=6 3 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.448+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc13e520 osd=6 3 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.448+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 14 2024-02-13T00:40:41.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.448+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc13e520 osd=6 3 2024-02-13T00:40:41.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.448+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 6 14 2024-02-13T00:40:41.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.448+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc13e520 osd=6 4 2024-02-13T00:40:41.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.448+0000 7f910283f700 10 client.24643.objecter _send_command 14 2024-02-13T00:40:41.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.448+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f90fc143dd0 con 0x7f90fc13ec30 2024-02-13T00:40:41.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.450+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f90fc143dd0 con 0x7f90fc13ec30 2024-02-13T00:40:41.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.450+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:41.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.450+0000 7f90f97fa700 10 client.24643.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.450+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc13e520 osd=6 4 2024-02-13T00:40:41.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.450+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 6 14 2024-02-13T00:40:41.463 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:40:41.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f910283f700 10 client.24643.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:41.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 15 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6812/1964192772 conn(0x7f90fc143110 0x7f90fc143590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc143dd0 osd=7 3 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc143dd0 osd=7 3 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc143dd0 osd=7 3 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 15 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc143dd0 osd=7 3 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 7 15 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc143dd0 osd=7 4 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f910283f700 10 client.24643.objecter _send_command 15 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90fc1471b0 con 0x7f90fc143110 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f90fbfff700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6812/1964192772 conn(0x7f90fc143110 0x7f90fc143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f90f97fa700 10 client.24643.objecter ms_handle_connect 0x7f90fc143110 2024-02-13T00:40:41.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.451+0000 7f90fbfff700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6812/1964192772 conn(0x7f90fc143110 0x7f90fc143590 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:41.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.452+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f90fc1471b0 con 0x7f90fc143110 2024-02-13T00:40:41.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.452+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:40:41.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.452+0000 7f90f97fa700 10 client.24643.objecter _finish_command 15 = system:0 2024-02-13T00:40:41.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.452+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc143dd0 osd=7 4 2024-02-13T00:40:41.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.452+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 7 15 2024-02-13T00:40:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:41.467 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:41.467 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:41.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:41.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.468+0000 7f910283f700 10 client.24643.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:40:41.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.468+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign -1 16 2024-02-13T00:40:41.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.468+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc143dd0 osd=7 3 2024-02-13T00:40:41.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.468+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc143dd0 osd=7 3 2024-02-13T00:40:41.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.468+0000 7f910283f700 20 client.24643.objecter _get_session s=0x7f90fc143dd0 osd=7 3 2024-02-13T00:40:41.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.468+0000 7f910283f700 15 client.24643.objecter _session_command_op_remove -1 16 2024-02-13T00:40:41.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.468+0000 7f910283f700 20 client.24643.objecter get_session s=0x7f90fc143dd0 osd=7 3 2024-02-13T00:40:41.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.468+0000 7f910283f700 15 client.24643.objecter _session_command_op_assign 7 16 2024-02-13T00:40:41.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.468+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc143dd0 osd=7 4 2024-02-13T00:40:41.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.468+0000 7f910283f700 10 client.24643.objecter _send_command 16 2024-02-13T00:40:41.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.468+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f90fc1415b0 con 0x7f90fc143110 2024-02-13T00:40:41.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f90f97fa700 1 -- 172.21.15.162:0/1380186660 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f90fc1415b0 con 0x7f90fc143110 2024-02-13T00:40:41.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f90f97fa700 10 client.24643.objecter ms_dispatch 0x7f90fc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:40:41.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f90f97fa700 10 client.24643.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f90f97fa700 20 client.24643.objecter put_session s=0x7f90fc143dd0 osd=7 4 2024-02-13T00:40:41.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f90f97fa700 15 client.24643.objecter _session_command_op_remove 7 16 2024-02-13T00:40:41.473 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:40:41.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 10 librados: watch_flush enter 2024-02-13T00:40:41.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 10 librados: watch_flush exit 2024-02-13T00:40:41.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 10 client.24643.objecter close_session for osd.0 2024-02-13T00:40:41.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6801/2103743338 conn(0x7f90fc059990 msgr2=0x7f90fc059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6801/2103743338 conn(0x7f90fc059990 0x7f90fc059d70 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:41.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc0475e0 osd=0 1 2024-02-13T00:40:41.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 10 client.24643.objecter close_session for osd.1 2024-02-13T00:40:41.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6805/150599780 conn(0x7f90fc102640 msgr2=0x7f90fc102a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6805/150599780 conn(0x7f90fc102640 0x7f90fc102a50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:41.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc028790 osd=1 1 2024-02-13T00:40:41.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 10 client.24643.objecter close_session for osd.2 2024-02-13T00:40:41.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6809/3689719376 conn(0x7f90fc0282f0 msgr2=0x7f90fc131bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6809/3689719376 conn(0x7f90fc0282f0 0x7f90fc131bf0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc1024c0 osd=2 1 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 10 client.24643.objecter close_session for osd.3 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6813/604971011 conn(0x7f90fc135350 msgr2=0x7f90fc137740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6813/604971011 conn(0x7f90fc135350 0x7f90fc137740 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc027c00 osd=3 1 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 10 client.24643.objecter close_session for osd.4 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6800/3344263388 conn(0x7f90fc1326e0 msgr2=0x7f90fc138db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.470+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6800/3344263388 conn(0x7f90fc1326e0 0x7f90fc138db0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc132540 osd=4 1 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 10 client.24643.objecter close_session for osd.5 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6804/2212296748 conn(0x7f90fc1397e0 msgr2=0x7f90fc139c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6804/2212296748 conn(0x7f90fc1397e0 0x7f90fc139c20 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:41.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc131920 osd=5 1 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 10 client.24643.objecter close_session for osd.6 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6808/1506640341 conn(0x7f90fc13ec30 msgr2=0x7f90fc141020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6808/1506640341 conn(0x7f90fc13ec30 0x7f90fc141020 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc13e520 osd=6 1 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 10 client.24643.objecter close_session for osd.7 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6812/1964192772 conn(0x7f90fc143110 msgr2=0x7f90fc143590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.181:6812/1964192772 conn(0x7f90fc143110 0x7f90fc143590 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 20 client.24643.objecter put_session s=0x7f90fc143dd0 osd=7 1 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 20 client.24643.objecter shutdown clearing up homeless session... 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 10 client.24643.objecter successfully canceled tick 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f9100ddc700 1 -- 172.21.15.162:0/1380186660 reap_dead start 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f9100ddc700 1 -- 172.21.15.162:0/1380186660 reap_dead start 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f9100ddc700 1 -- 172.21.15.162:0/1380186660 reap_dead start 2024-02-13T00:40:41.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f9100ddc700 1 -- 172.21.15.162:0/1380186660 reap_dead start 2024-02-13T00:40:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f9100ddc700 1 -- 172.21.15.162:0/1380186660 reap_dead start 2024-02-13T00:40:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6800/2866534053 conn(0x7f90e4062ce0 msgr2=0x7f90e40651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6800/2866534053 conn(0x7f90e4062ce0 0x7f90e40651a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f90f00043d0 tx=0x7f90f0009cb0).stop 2024-02-13T00:40:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90fc104010 msgr2=0x7f90fc10c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90fc104010 0x7f90fc10c720 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f90f400bad0 tx=0x7f90f4009780).stop 2024-02-13T00:40:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 shutdown_connections 2024-02-13T00:40:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90fc104010 0x7f90fc10c720 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 --2- 172.21.15.162:0/1380186660 >> v2:172.21.15.162:6800/2866534053 conn(0x7f90e4062ce0 0x7f90e40651a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 >> 172.21.15.162:0/1380186660 conn(0x7f90fc063df0 msgr2=0x7f90fc105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 shutdown_connections 2024-02-13T00:40:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 -- 172.21.15.162:0/1380186660 wait complete. 2024-02-13T00:40:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:41.471+0000 7f910283f700 1 librados: shutdown 2024-02-13T00:40:41.479 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.479 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.479 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.479 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.479 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.479 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.479 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.479 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:40:41.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:41.595 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:41.848 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:41.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:41.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:41.986 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:42.137 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:42.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:40.196845+0000 mgr.y (mgr.14544) 121 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:40:42.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:40.887850+0000 mon.a (mon.0) 744 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 17 pgs peering) 2024-02-13T00:40:42.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:41.250513+0000 mon.b (mon.1) 27 : audit [DBG] from='client.? 172.21.15.162:0/1380186660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:40:42.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:41.383740+0000 osd.2 (osd.2) 39 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:40:42.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:41.387151+0000 osd.2 (osd.2) 40 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:40:42.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:40.196845+0000 mgr.y (mgr.14544) 121 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:40:42.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:40.887850+0000 mon.a (mon.0) 744 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 17 pgs peering) 2024-02-13T00:40:42.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:41.250513+0000 mon.b (mon.1) 27 : audit [DBG] from='client.? 172.21.15.162:0/1380186660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:40:42.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:41.383740+0000 osd.2 (osd.2) 39 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:40:42.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:41.387151+0000 osd.2 (osd.2) 40 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:40:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:42.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:42.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:42.284 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:42.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:40.196845+0000 mgr.y (mgr.14544) 121 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:40:42.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:40.887850+0000 mon.a (mon.0) 744 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 17 pgs peering) 2024-02-13T00:40:42.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:41.250513+0000 mon.b (mon.1) 27 : audit [DBG] from='client.? 172.21.15.162:0/1380186660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:40:42.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:41.383740+0000 osd.2 (osd.2) 39 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:40:42.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:41.387151+0000 osd.2 (osd.2) 40 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:40:42.492 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:42.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:42.519 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:42.560 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:42.602 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:42.634 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:42.655 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:42.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:42.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:42.759 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:42.811 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:43.011 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:41.217389+0000 osd.7 (osd.7) 48 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:40:43.011 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:41.220379+0000 osd.7 (osd.7) 49 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:40:43.012 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:42.229359+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:40:43.012 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:42.230971+0000 osd.6 (osd.6) 34 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:40:43.012 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:42.410123+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:40:43.012 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:42.413039+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:40:43.012 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:41.217389+0000 osd.7 (osd.7) 48 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:40:43.012 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:41.220379+0000 osd.7 (osd.7) 49 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:40:43.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:42.229359+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:40:43.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:42.230971+0000 osd.6 (osd.6) 34 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:40:43.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:42.410123+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:40:43.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:42.413039+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:40:43.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:41.217389+0000 osd.7 (osd.7) 48 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:40:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:41.220379+0000 osd.7 (osd.7) 49 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:40:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:42.229359+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:40:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:42.230971+0000 osd.6 (osd.6) 34 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:40:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:42.410123+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:40:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:42.413039+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:40:43.568 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:43.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:43.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:43.676 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:43.708 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:43.729 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:43.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:43.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:43.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:43.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:40:43.863 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:43.883 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:43.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:43.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:43.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:44.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:42.197223+0000 mgr.y (mgr.14544) 122 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:40:44.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:42.541353+0000 osd.6 (osd.6) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:44.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:42.541392+0000 osd.6 (osd.6) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:44.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:42.906696+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-13T00:40:44.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:43.008620+0000 osd.3 (osd.3) 63 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:40:44.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:43.011683+0000 osd.3 (osd.3) 64 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:40:44.015 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:44.033 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:44.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:44.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:44.137 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:44.163 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:44.182 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:44.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:44.285 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:44.312 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:44.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:42.197223+0000 mgr.y (mgr.14544) 122 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:40:44.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:42.541353+0000 osd.6 (osd.6) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:44.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:42.541392+0000 osd.6 (osd.6) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:44.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:42.906696+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-13T00:40:44.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:43.008620+0000 osd.3 (osd.3) 63 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:40:44.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:43.011683+0000 osd.3 (osd.3) 64 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:40:44.331 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:44.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:42.197223+0000 mgr.y (mgr.14544) 122 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:40:44.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:42.541353+0000 osd.6 (osd.6) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:44.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:42.541392+0000 osd.6 (osd.6) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:44.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:42.906696+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-13T00:40:44.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:43.008620+0000 osd.3 (osd.3) 63 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:40:44.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:43.011683+0000 osd.3 (osd.3) 64 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:40:44.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:44.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:44.433 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:44.459 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:44.477 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:44.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:44.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:44.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:40:44.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:45.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:45.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:45.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:45.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:45.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:43.194030+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:40:45.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:43.197152+0000 osd.7 (osd.7) 51 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:40:45.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:43.626892+0000 osd.5 (osd.5) 53 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:40:45.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:43.630531+0000 osd.5 (osd.5) 54 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:40:45.212 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:44.270023+0000 osd.6 (osd.6) 37 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:40:45.212 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:44.273507+0000 osd.6 (osd.6) 38 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:40:45.212 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:44.366782+0000 osd.2 (osd.2) 43 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:40:45.212 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:44.369061+0000 osd.2 (osd.2) 44 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:40:45.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:43.194030+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:40:45.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:43.197152+0000 osd.7 (osd.7) 51 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:40:45.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:43.626892+0000 osd.5 (osd.5) 53 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:40:45.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:43.630531+0000 osd.5 (osd.5) 54 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:40:45.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:44.270023+0000 osd.6 (osd.6) 37 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:40:45.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:44.273507+0000 osd.6 (osd.6) 38 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:40:45.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:44.366782+0000 osd.2 (osd.2) 43 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:40:45.213 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:44.369061+0000 osd.2 (osd.2) 44 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:40:45.215 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:45.272 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:45.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:45.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:45.276 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:45.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:43.194030+0000 osd.7 (osd.7) 50 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:40:45.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:43.197152+0000 osd.7 (osd.7) 51 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:40:45.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:43.626892+0000 osd.5 (osd.5) 53 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:40:45.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:43.630531+0000 osd.5 (osd.5) 54 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:40:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:44.270023+0000 osd.6 (osd.6) 37 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:40:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:44.273507+0000 osd.6 (osd.6) 38 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:40:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:44.366782+0000 osd.2 (osd.2) 43 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:40:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:44.369061+0000 osd.2 (osd.2) 44 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:40:45.328 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:40:45.328 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:40:45.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:45.408 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:45.429 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:45.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:45.533 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:45.560 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:40:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:40:45.562 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:45.582 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:45.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:45.686 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:45.713 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:45.732 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:45.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:45.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:45.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:45.867 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:45.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.976+0000 7ffb397ed700 1 -- 172.21.15.162:0/2541483306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340fcb10 msgr2=0x7ffb340fcef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:45.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.976+0000 7ffb397ed700 1 --2- 172.21.15.162:0/2541483306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340fcb10 0x7ffb340fcef0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7ffb24004870 tx=0x7ffb24009cb0).stop 2024-02-13T00:40:45.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.976+0000 7ffb397ed700 1 -- 172.21.15.162:0/2541483306 shutdown_connections 2024-02-13T00:40:45.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.976+0000 7ffb397ed700 1 --2- 172.21.15.162:0/2541483306 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340f7380 0x7ffb34103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:45.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.976+0000 7ffb397ed700 1 --2- 172.21.15.162:0/2541483306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340fcb10 0x7ffb340fcef0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:45.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.976+0000 7ffb397ed700 1 --2- 172.21.15.162:0/2541483306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb340fd470 0x7ffb340f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:45.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.976+0000 7ffb397ed700 1 -- 172.21.15.162:0/2541483306 >> 172.21.15.162:0/2541483306 conn(0x7ffb340f2740 msgr2=0x7ffb340f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:45.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.976+0000 7ffb397ed700 1 -- 172.21.15.162:0/2541483306 shutdown_connections 2024-02-13T00:40:45.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.976+0000 7ffb397ed700 1 -- 172.21.15.162:0/2541483306 wait complete. 2024-02-13T00:40:45.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.976+0000 7ffb397ed700 1 librados: starting msgr at 2024-02-13T00:40:45.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.976+0000 7ffb397ed700 1 librados: starting objecter 2024-02-13T00:40:45.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.976+0000 7ffb397ed700 1 Processor -- start 2024-02-13T00:40:45.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb397ed700 1 -- start start 2024-02-13T00:40:45.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb397ed700 1 librados: setting wanted keys 2024-02-13T00:40:45.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb397ed700 1 librados: calling monclient init 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb397ed700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb340f7380 0x7ffb341091c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb397ed700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340fcb10 0x7ffb341096b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb397ed700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340fd470 0x7ffb34109ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb397ed700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb341143f0 con 0x7ffb340fcb10 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb397ed700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb34114560 con 0x7ffb340f7380 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb397ed700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb34114720 con 0x7ffb340fd470 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb32ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb340f7380 0x7ffb341091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb32ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb340f7380 0x7ffb341091c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:41726/0 (socket says 172.21.15.162:41726) 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb32ffd700 1 -- 172.21.15.162:0/1513876580 learned_addr learned my addr 172.21.15.162:0/1513876580 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb337fe700 1 --2- 172.21.15.162:0/1513876580 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340fd470 0x7ffb34109ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb32ffd700 1 -- 172.21.15.162:0/1513876580 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340fd470 msgr2=0x7ffb34109ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb32ffd700 1 --2- 172.21.15.162:0/1513876580 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340fd470 0x7ffb34109ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:45.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb32ffd700 1 -- 172.21.15.162:0/1513876580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340fcb10 msgr2=0x7ffb341096b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb32ffd700 1 --2- 172.21.15.162:0/1513876580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340fcb10 0x7ffb341096b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.977+0000 7ffb32ffd700 1 -- 172.21.15.162:0/1513876580 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb24005040 con 0x7ffb340f7380 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.978+0000 7ffb337fe700 1 --2- 172.21.15.162:0/1513876580 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340fd470 0x7ffb34109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.978+0000 7ffb32ffd700 1 --2- 172.21.15.162:0/1513876580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb340f7380 0x7ffb341091c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7ffb240043a0 tx=0x7ffb2400b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.978+0000 7ffb1bfff700 10 client.?.objecter ms_handle_connect 0x7ffb340f7380 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.978+0000 7ffb1bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.978+0000 7ffb1bfff700 1 -- 172.21.15.162:0/1513876580 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2400e070 con 0x7ffb340f7380 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.978+0000 7ffb1bfff700 1 -- 172.21.15.162:0/1513876580 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb24009710 con 0x7ffb340f7380 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.978+0000 7ffb1bfff700 1 -- 172.21.15.162:0/1513876580 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb24019070 con 0x7ffb340f7380 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.978+0000 7ffb397ed700 1 -- 172.21.15.162:0/1513876580 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb341149a0 con 0x7ffb340f7380 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.978+0000 7ffb397ed700 10 client.24649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.978+0000 7ffb397ed700 1 -- 172.21.15.162:0/1513876580 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb34114e30 con 0x7ffb340f7380 2024-02-13T00:40:45.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.978+0000 7ffb397ed700 1 librados: init done 2024-02-13T00:40:45.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.979+0000 7ffb1bfff700 1 -- 172.21.15.162:0/1513876580 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ffb240090d0 con 0x7ffb340f7380 2024-02-13T00:40:45.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.979+0000 7ffb397ed700 1 -- 172.21.15.162:0/1513876580 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb340607b0 con 0x7ffb340f7380 2024-02-13T00:40:45.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.980+0000 7ffb1bfff700 1 --2- 172.21.15.162:0/1513876580 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffb1c062c90 0x7ffb1c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:45.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.980+0000 7ffb1bfff700 1 -- 172.21.15.162:0/1513876580 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(98..98 src has 41..98) v4 ==== 6204+0+0 (secure 0 0 0) 0x7ffb2407d520 con 0x7ffb340f7380 2024-02-13T00:40:45.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.980+0000 7ffb1bfff700 10 client.24649.objecter ms_dispatch 0x7ffb34001550 osd_map(98..98 src has 41..98) v4 2024-02-13T00:40:45.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.980+0000 7ffb1bfff700 3 client.24649.objecter handle_osd_map got epochs [98,98] > 0 2024-02-13T00:40:45.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.980+0000 7ffb1bfff700 3 client.24649.objecter handle_osd_map decoding full epoch 98 2024-02-13T00:40:45.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.980+0000 7ffb327fc700 1 --2- 172.21.15.162:0/1513876580 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffb1c062c90 0x7ffb1c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:45.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.980+0000 7ffb1bfff700 20 client.24649.objecter dump_active .. 0 homeless 2024-02-13T00:40:45.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.980+0000 7ffb1bfff700 10 client.24649.objecter ms_handle_connect 0x7ffb1c062c90 2024-02-13T00:40:45.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.980+0000 7ffb327fc700 1 --2- 172.21.15.162:0/1513876580 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffb1c062c90 0x7ffb1c065150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ffb34061950 tx=0x7ffb2800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:45.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:45.984+0000 7ffb1bfff700 1 -- 172.21.15.162:0/1513876580 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ffb24051200 con 0x7ffb340f7380 2024-02-13T00:40:46.076 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:46.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:46.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:44.197600+0000 mgr.y (mgr.14544) 123 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:46.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:44.240045+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:40:46.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:44.242942+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:40:46.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:45.304455+0000 mgr.y (mgr.14544) 124 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:46.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:44.197600+0000 mgr.y (mgr.14544) 123 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:46.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:44.240045+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:40:46.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:44.242942+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:40:46.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:45.304455+0000 mgr.y (mgr.14544) 124 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:46.166 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:40:46] "GET /metrics HTTP/1.1" 200 172261 "" "Prometheus/2.33.4" 2024-02-13T00:40:46.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:46.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:44.197600+0000 mgr.y (mgr.14544) 123 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:46.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:44.240045+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:40:46.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:44.242942+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:40:46.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:45.304455+0000 mgr.y (mgr.14544) 124 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:40:46.222 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:46.243 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:46.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:46.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:46.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.254+0000 7ffb397ed700 1 -- 172.21.15.162:0/1513876580 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb34061700 con 0x7ffb1c062c90 2024-02-13T00:40:46.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.273+0000 7ffb1bfff700 1 -- 172.21.15.162:0/1513876580 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357020 (secure 0 0 0) 0x7ffb34061700 con 0x7ffb1c062c90 2024-02-13T00:40:46.281 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:40:46.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.276+0000 7ffb397ed700 10 librados: watch_flush enter 2024-02-13T00:40:46.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.276+0000 7ffb397ed700 10 librados: watch_flush exit 2024-02-13T00:40:46.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.276+0000 7ffb397ed700 20 client.24649.objecter shutdown clearing up homeless session... 2024-02-13T00:40:46.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.276+0000 7ffb397ed700 10 client.24649.objecter successfully canceled tick 2024-02-13T00:40:46.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.276+0000 7ffb397ed700 1 -- 172.21.15.162:0/1513876580 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffb1c062c90 msgr2=0x7ffb1c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:46.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.276+0000 7ffb397ed700 1 --2- 172.21.15.162:0/1513876580 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffb1c062c90 0x7ffb1c065150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ffb34061950 tx=0x7ffb2800b040).stop 2024-02-13T00:40:46.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.276+0000 7ffb397ed700 1 -- 172.21.15.162:0/1513876580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb340f7380 msgr2=0x7ffb341091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:46.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.276+0000 7ffb397ed700 1 --2- 172.21.15.162:0/1513876580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb340f7380 0x7ffb341091c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7ffb240043a0 tx=0x7ffb2400b040).stop 2024-02-13T00:40:46.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.277+0000 7ffb397ed700 1 -- 172.21.15.162:0/1513876580 shutdown_connections 2024-02-13T00:40:46.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.277+0000 7ffb397ed700 1 --2- 172.21.15.162:0/1513876580 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffb1c062c90 0x7ffb1c065150 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:46.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.277+0000 7ffb397ed700 1 --2- 172.21.15.162:0/1513876580 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340fd470 0x7ffb34109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:46.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.277+0000 7ffb397ed700 1 --2- 172.21.15.162:0/1513876580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb340f7380 0x7ffb341091c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:46.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.277+0000 7ffb397ed700 1 --2- 172.21.15.162:0/1513876580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340fcb10 0x7ffb341096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:40:46.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.277+0000 7ffb397ed700 1 -- 172.21.15.162:0/1513876580 >> 172.21.15.162:0/1513876580 conn(0x7ffb340f2740 msgr2=0x7ffb340f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:46.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.277+0000 7ffb397ed700 1 -- 172.21.15.162:0/1513876580 shutdown_connections 2024-02-13T00:40:46.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.277+0000 7ffb397ed700 1 -- 172.21.15.162:0/1513876580 wait complete. 2024-02-13T00:40:46.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:46.277+0000 7ffb397ed700 1 librados: shutdown 2024-02-13T00:40:46.285 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:40:46.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:46.382 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:46.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-13T00:40:46.733 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:46.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:46.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:46.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:46.874 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:47.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:45.197304+0000 osd.7 (osd.7) 54 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:40:47.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:45.198226+0000 osd.7 (osd.7) 55 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:40:47.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:46.198324+0000 mgr.y (mgr.14544) 125 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:47.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:46.255126+0000 mgr.y (mgr.14544) 126 : audit [DBG] from='client.24649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:40:47.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:46.273522+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:40:47.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:46.274419+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:40:47.072 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:45.197304+0000 osd.7 (osd.7) 54 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:40:47.073 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:45.198226+0000 osd.7 (osd.7) 55 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:40:47.073 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:46.198324+0000 mgr.y (mgr.14544) 125 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:47.073 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:46.255126+0000 mgr.y (mgr.14544) 126 : audit [DBG] from='client.24649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:40:47.073 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:46.273522+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:40:47.073 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:46.274419+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:40:47.073 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:47.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:47.179 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:47.208 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:45.197304+0000 osd.7 (osd.7) 54 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:40:47.208 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:45.198226+0000 osd.7 (osd.7) 55 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:40:47.209 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:46.198324+0000 mgr.y (mgr.14544) 125 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:47.209 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:46.255126+0000 mgr.y (mgr.14544) 126 : audit [DBG] from='client.24649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:40:47.209 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:46.273522+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:40:47.209 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:46.274419+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:40:47.211 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:47.232 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:47.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:47.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:47.336 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:47.365 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:47.384 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:47.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:47.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:47.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:47.499 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-02-13T00:40:46.197758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1261,"num_read_kb":981,"num_write":620,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":746,"ondisk_log_size":746,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2386488,"kb_used_data":5624,"kb_used_omap":0,"kb_used_meta":2380800,"kb_avail":747574728,"statfs":{"total":767960285184,"available":765516521472,"internally_reserved":0,"allocated":5758976,"data_stored":2710285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2437939200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002080"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.260745+0000","last_change":"2024-02-13T00:39:18.311781+0000","last_active":"2024-02-13T00:40:40.260745+0000","last_peered":"2024-02-13T00:40:40.260745+0000","last_clean":"2024-02-13T00:40:40.260745+0000","last_became_active":"2024-02-13T00:37:47.552890+0000","last_became_peered":"2024-02-13T00:37:47.552890+0000","last_unstale":"2024-02-13T00:40:40.260745+0000","last_undegraded":"2024-02-13T00:40:40.260745+0000","last_fullsized":"2024-02-13T00:40:40.260745+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.311669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.311669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.654157+0000","last_change":"2024-02-13T00:40:35.847474+0000","last_active":"2024-02-13T00:40:40.654157+0000","last_peered":"2024-02-13T00:40:40.654157+0000","last_clean":"2024-02-13T00:40:40.654157+0000","last_became_active":"2024-02-13T00:40:35.846778+0000","last_became_peered":"2024-02-13T00:40:35.846778+0000","last_unstale":"2024-02-13T00:40:40.654157+0000","last_undegraded":"2024-02-13T00:40:40.654157+0000","last_fullsized":"2024-02-13T00:40:40.654157+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.273346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.256017+0000","last_change":"2024-02-13T00:39:32.313930+0000","last_active":"2024-02-13T00:40:40.256017+0000","last_peered":"2024-02-13T00:40:40.256017+0000","last_clean":"2024-02-13T00:40:40.256017+0000","last_became_active":"2024-02-13T00:37:41.220583+0000","last_became_peered":"2024-02-13T00:37:41.220583+0000","last_unstale":"2024-02-13T00:40:40.256017+0000","last_undegraded":"2024-02-13T00:40:40.256017+0000","last_fullsized":"2024-02-13T00:40:40.256017+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.313841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"86'50","reported_seq":153,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.260422+0000","last_change":"2024-02-13T00:39:53.119049+0000","last_active":"2024-02-13T00:40:40.260422+0000","last_peered":"2024-02-13T00:40:40.260422+0000","last_clean":"2024-02-13T00:40:40.260422+0000","last_became_active":"2024-02-13T00:37:43.228728+0000","last_became_peered":"2024-02-13T00:37:43.228728+0000","last_unstale":"2024-02-13T00:40:40.260422+0000","last_undegraded":"2024-02-13T00:40:40.260422+0000","last_fullsized":"2024-02-13T00:40:40.260422+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:39:53.118989+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.250883+0000","last_change":"2024-02-13T00:40:35.846021+0000","last_active":"2024-02-13T00:40:40.250883+0000","last_peered":"2024-02-13T00:40:40.250883+0000","last_clean":"2024-02-13T00:40:40.250883+0000","last_became_active":"2024-02-13T00:40:35.845664+0000","last_became_peered":"2024-02-13T00:40:35.845664+0000","last_unstale":"2024-02-13T00:40:40.250883+0000","last_undegraded":"2024-02-13T00:40:40.250883+0000","last_fullsized":"2024-02-13T00:40:40.250883+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:37.210406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:37.210406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":51,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.660615+0000","last_change":"2024-02-13T00:39:24.332217+0000","last_active":"2024-02-13T00:40:40.660615+0000","last_peered":"2024-02-13T00:40:40.660615+0000","last_clean":"2024-02-13T00:40:40.660615+0000","last_became_active":"2024-02-13T00:37:45.249914+0000","last_became_peered":"2024-02-13T00:37:45.249914+0000","last_unstale":"2024-02-13T00:40:40.660615+0000","last_undegraded":"2024-02-13T00:40:40.660615+0000","last_fullsized":"2024-02-13T00:40:40.660615+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:24.332133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.332133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.261488+0000","last_change":"2024-02-13T00:39:40.280385+0000","last_active":"2024-02-13T00:40:40.261488+0000","last_peered":"2024-02-13T00:40:40.261488+0000","last_clean":"2024-02-13T00:40:40.261488+0000","last_became_active":"2024-02-13T00:37:41.216896+0000","last_became_peered":"2024-02-13T00:37:41.216896+0000","last_unstale":"2024-02-13T00:40:40.261488+0000","last_undegraded":"2024-02-13T00:40:40.261488+0000","last_fullsized":"2024-02-13T00:40:40.261488+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:39:40.280305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"90'20","reported_seq":85,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.654317+0000","last_change":"2024-02-13T00:40:35.847653+0000","last_active":"2024-02-13T00:40:40.654317+0000","last_peered":"2024-02-13T00:40:40.654317+0000","last_clean":"2024-02-13T00:40:40.654317+0000","last_became_active":"2024-02-13T00:40:35.847203+0000","last_became_peered":"2024-02-13T00:40:35.847203+0000","last_unstale":"2024-02-13T00:40:40.654317+0000","last_undegraded":"2024-02-13T00:40:40.654317+0000","last_fullsized":"2024-02-13T00:40:40.654317+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:40:13.144451+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":53,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251016+0000","last_change":"2024-02-13T00:40:13.678873+0000","last_active":"2024-02-13T00:40:40.251016+0000","last_peered":"2024-02-13T00:40:40.251016+0000","last_clean":"2024-02-13T00:40:40.251016+0000","last_became_active":"2024-02-13T00:37:47.260025+0000","last_became_peered":"2024-02-13T00:37:47.260025+0000","last_unstale":"2024-02-13T00:40:40.251016+0000","last_undegraded":"2024-02-13T00:40:40.251016+0000","last_fullsized":"2024-02-13T00:40:40.251016+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:13.678807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:40:13.678807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.654650+0000","last_change":"2024-02-13T00:40:07.088136+0000","last_active":"2024-02-13T00:40:40.654650+0000","last_peered":"2024-02-13T00:40:40.654650+0000","last_clean":"2024-02-13T00:40:40.654650+0000","last_became_active":"2024-02-13T00:37:45.246184+0000","last_became_peered":"2024-02-13T00:37:45.246184+0000","last_unstale":"2024-02-13T00:40:40.654650+0000","last_undegraded":"2024-02-13T00:40:40.654650+0000","last_fullsized":"2024-02-13T00:40:40.654650+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:07.088053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:07.088053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655958+0000","last_change":"2024-02-13T00:39:21.165313+0000","last_active":"2024-02-13T00:40:40.655958+0000","last_peered":"2024-02-13T00:40:40.655958+0000","last_clean":"2024-02-13T00:40:40.655958+0000","last_became_active":"2024-02-13T00:37:41.221025+0000","last_became_peered":"2024-02-13T00:37:41.221025+0000","last_unstale":"2024-02-13T00:40:40.655958+0000","last_undegraded":"2024-02-13T00:40:40.655958+0000","last_fullsized":"2024-02-13T00:40:40.655958+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.165246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"85'14","reported_seq":74,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.260890+0000","last_change":"2024-02-13T00:40:35.857071+0000","last_active":"2024-02-13T00:40:40.260890+0000","last_peered":"2024-02-13T00:40:40.260890+0000","last_clean":"2024-02-13T00:40:40.260890+0000","last_became_active":"2024-02-13T00:40:35.856715+0000","last_became_peered":"2024-02-13T00:40:35.856715+0000","last_unstale":"2024-02-13T00:40:40.260890+0000","last_undegraded":"2024-02-13T00:40:40.260890+0000","last_fullsized":"2024-02-13T00:40:40.260890+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:39:42.156267+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:39:42.156267+0000","last_clean_scrub_stamp":"2024-02-13T00:39:42.156267+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.254660+0000","last_change":"2024-02-13T00:39:22.338139+0000","last_active":"2024-02-13T00:40:40.254660+0000","last_peered":"2024-02-13T00:40:40.254660+0000","last_clean":"2024-02-13T00:40:40.254660+0000","last_became_active":"2024-02-13T00:37:47.259688+0000","last_became_peered":"2024-02-13T00:37:47.259688+0000","last_unstale":"2024-02-13T00:40:40.254660+0000","last_undegraded":"2024-02-13T00:40:40.254660+0000","last_fullsized":"2024-02-13T00:40:40.254660+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.338042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251304+0000","last_change":"2024-02-13T00:40:14.697069+0000","last_active":"2024-02-13T00:40:40.251304+0000","last_peered":"2024-02-13T00:40:40.251304+0000","last_clean":"2024-02-13T00:40:40.251304+0000","last_became_active":"2024-02-13T00:37:45.243961+0000","last_became_peered":"2024-02-13T00:37:45.243961+0000","last_unstale":"2024-02-13T00:40:40.251304+0000","last_undegraded":"2024-02-13T00:40:40.251304+0000","last_fullsized":"2024-02-13T00:40:40.251304+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:14.696991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:14.696991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":59,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:43.197233+0000","last_change":"2024-02-13T00:40:43.197233+0000","last_active":"2024-02-13T00:40:43.197233+0000","last_peered":"2024-02-13T00:40:43.197233+0000","last_clean":"2024-02-13T00:40:43.197233+0000","last_became_active":"2024-02-13T00:37:41.222506+0000","last_became_peered":"2024-02-13T00:37:41.222506+0000","last_unstale":"2024-02-13T00:40:43.197233+0000","last_undegraded":"2024-02-13T00:40:43.197233+0000","last_fullsized":"2024-02-13T00:40:43.197233+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:43.197157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:43.197157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":73,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.260650+0000","last_change":"2024-02-13T00:40:35.857612+0000","last_active":"2024-02-13T00:40:40.260650+0000","last_peered":"2024-02-13T00:40:40.260650+0000","last_clean":"2024-02-13T00:40:40.260650+0000","last_became_active":"2024-02-13T00:40:35.857274+0000","last_became_peered":"2024-02-13T00:40:35.857274+0000","last_unstale":"2024-02-13T00:40:40.260650+0000","last_undegraded":"2024-02-13T00:40:40.260650+0000","last_fullsized":"2024-02-13T00:40:40.260650+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:39:34.213074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.213074+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.264328+0000","last_change":"2024-02-13T00:39:02.223894+0000","last_active":"2024-02-13T00:40:40.264328+0000","last_peered":"2024-02-13T00:40:40.264328+0000","last_clean":"2024-02-13T00:40:40.264328+0000","last_became_active":"2024-02-13T00:37:47.553853+0000","last_became_peered":"2024-02-13T00:37:47.553853+0000","last_unstale":"2024-02-13T00:40:40.264328+0000","last_undegraded":"2024-02-13T00:40:40.264328+0000","last_fullsized":"2024-02-13T00:40:40.264328+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:02.223823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.223823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.264666+0000","last_change":"2024-02-13T00:40:35.858850+0000","last_active":"2024-02-13T00:40:40.264666+0000","last_peered":"2024-02-13T00:40:40.264666+0000","last_clean":"2024-02-13T00:40:40.264666+0000","last_became_active":"2024-02-13T00:40:35.858538+0000","last_became_peered":"2024-02-13T00:40:35.858538+0000","last_unstale":"2024-02-13T00:40:40.264666+0000","last_undegraded":"2024-02-13T00:40:40.264666+0000","last_fullsized":"2024-02-13T00:40:40.264666+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.301282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.260967+0000","last_change":"2024-02-13T00:39:00.215830+0000","last_active":"2024-02-13T00:40:40.260967+0000","last_peered":"2024-02-13T00:40:40.260967+0000","last_clean":"2024-02-13T00:40:40.260967+0000","last_became_active":"2024-02-13T00:37:41.218333+0000","last_became_peered":"2024-02-13T00:37:41.218333+0000","last_unstale":"2024-02-13T00:40:40.260967+0000","last_undegraded":"2024-02-13T00:40:40.260967+0000","last_fullsized":"2024-02-13T00:40:40.260967+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.215755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.215755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":90,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.264276+0000","last_change":"2024-02-13T00:39:26.214150+0000","last_active":"2024-02-13T00:40:40.264276+0000","last_peered":"2024-02-13T00:40:40.264276+0000","last_clean":"2024-02-13T00:40:40.264276+0000","last_became_active":"2024-02-13T00:37:43.236026+0000","last_became_peered":"2024-02-13T00:37:43.236026+0000","last_unstale":"2024-02-13T00:40:40.264276+0000","last_undegraded":"2024-02-13T00:40:40.264276+0000","last_fullsized":"2024-02-13T00:40:40.264276+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:39:26.214084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:26.214084+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655390+0000","last_change":"2024-02-13T00:40:35.846828+0000","last_active":"2024-02-13T00:40:40.655390+0000","last_peered":"2024-02-13T00:40:40.655390+0000","last_clean":"2024-02-13T00:40:40.655390+0000","last_became_active":"2024-02-13T00:40:35.846506+0000","last_became_peered":"2024-02-13T00:40:35.846506+0000","last_unstale":"2024-02-13T00:40:40.655390+0000","last_undegraded":"2024-02-13T00:40:40.655390+0000","last_fullsized":"2024-02-13T00:40:40.655390+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:15.106889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:15.106889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":51,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.663201+0000","last_change":"2024-02-13T00:39:30.245586+0000","last_active":"2024-02-13T00:40:40.663201+0000","last_peered":"2024-02-13T00:40:40.663201+0000","last_clean":"2024-02-13T00:40:40.663201+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:40:40.663201+0000","last_undegraded":"2024-02-13T00:40:40.663201+0000","last_fullsized":"2024-02-13T00:40:40.663201+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:30.245508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.245508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655312+0000","last_change":"2024-02-13T00:39:10.364972+0000","last_active":"2024-02-13T00:40:40.655312+0000","last_peered":"2024-02-13T00:40:40.655312+0000","last_clean":"2024-02-13T00:40:40.655312+0000","last_became_active":"2024-02-13T00:37:41.218334+0000","last_became_peered":"2024-02-13T00:37:41.218334+0000","last_unstale":"2024-02-13T00:40:40.655312+0000","last_undegraded":"2024-02-13T00:40:40.655312+0000","last_fullsized":"2024-02-13T00:40:40.655312+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:10.364865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.364865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":83,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:44.369174+0000","last_change":"2024-02-13T00:40:44.369174+0000","last_active":"2024-02-13T00:40:44.369174+0000","last_peered":"2024-02-13T00:40:44.369174+0000","last_clean":"2024-02-13T00:40:44.369174+0000","last_became_active":"2024-02-13T00:37:43.232938+0000","last_became_peered":"2024-02-13T00:37:43.232938+0000","last_unstale":"2024-02-13T00:40:44.369174+0000","last_undegraded":"2024-02-13T00:40:44.369174+0000","last_fullsized":"2024-02-13T00:40:44.369174+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:40:44.369070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.369070+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.661926+0000","last_change":"2024-02-13T00:40:06.026382+0000","last_active":"2024-02-13T00:40:40.661926+0000","last_peered":"2024-02-13T00:40:40.661926+0000","last_clean":"2024-02-13T00:40:40.661926+0000","last_became_active":"2024-02-13T00:37:47.260182+0000","last_became_peered":"2024-02-13T00:37:47.260182+0000","last_unstale":"2024-02-13T00:40:40.661926+0000","last_undegraded":"2024-02-13T00:40:40.661926+0000","last_fullsized":"2024-02-13T00:40:40.661926+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:06.026318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:06.026318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.256640+0000","last_change":"2024-02-13T00:39:15.328037+0000","last_active":"2024-02-13T00:40:40.256640+0000","last_peered":"2024-02-13T00:40:40.256640+0000","last_clean":"2024-02-13T00:40:40.256640+0000","last_became_active":"2024-02-13T00:37:45.249064+0000","last_became_peered":"2024-02-13T00:37:45.249064+0000","last_unstale":"2024-02-13T00:40:40.256640+0000","last_undegraded":"2024-02-13T00:40:40.256640+0000","last_fullsized":"2024-02-13T00:40:40.256640+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.327971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.327971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.260040+0000","last_change":"2024-02-13T00:39:41.231336+0000","last_active":"2024-02-13T00:40:40.260040+0000","last_peered":"2024-02-13T00:40:40.260040+0000","last_clean":"2024-02-13T00:40:40.260040+0000","last_became_active":"2024-02-13T00:37:41.216574+0000","last_became_peered":"2024-02-13T00:37:41.216574+0000","last_unstale":"2024-02-13T00:40:40.260040+0000","last_undegraded":"2024-02-13T00:40:40.260040+0000","last_fullsized":"2024-02-13T00:40:40.260040+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:41.231244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:41.231244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":92,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:42.231108+0000","last_change":"2024-02-13T00:40:42.231057+0000","last_active":"2024-02-13T00:40:42.231108+0000","last_peered":"2024-02-13T00:40:42.231108+0000","last_clean":"2024-02-13T00:40:42.231108+0000","last_became_active":"2024-02-13T00:37:43.230491+0000","last_became_peered":"2024-02-13T00:37:43.230491+0000","last_unstale":"2024-02-13T00:40:42.231108+0000","last_undegraded":"2024-02-13T00:40:42.231108+0000","last_fullsized":"2024-02-13T00:40:42.231108+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:40:42.230976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:42.230976+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655841+0000","last_change":"2024-02-13T00:40:02.168591+0000","last_active":"2024-02-13T00:40:40.655841+0000","last_peered":"2024-02-13T00:40:40.655841+0000","last_clean":"2024-02-13T00:40:40.655841+0000","last_became_active":"2024-02-13T00:37:47.254586+0000","last_became_peered":"2024-02-13T00:37:47.254586+0000","last_unstale":"2024-02-13T00:40:40.655841+0000","last_undegraded":"2024-02-13T00:40:40.655841+0000","last_fullsized":"2024-02-13T00:40:40.655841+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:40:02.168499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:02.168499+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"64'8","reported_seq":99,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:43.626954+0000","last_change":"2024-02-13T00:39:02.452828+0000","last_active":"2024-02-13T00:40:43.626954+0000","last_peered":"2024-02-13T00:40:43.626954+0000","last_clean":"2024-02-13T00:40:43.626954+0000","last_became_active":"2024-02-13T00:37:45.249902+0000","last_became_peered":"2024-02-13T00:37:45.249902+0000","last_unstale":"2024-02-13T00:40:43.626954+0000","last_undegraded":"2024-02-13T00:40:43.626954+0000","last_fullsized":"2024-02-13T00:40:43.626954+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:02.452755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.452755+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.250911+0000","last_change":"2024-02-13T00:38:49.606456+0000","last_active":"2024-02-13T00:40:40.250911+0000","last_peered":"2024-02-13T00:40:40.250911+0000","last_clean":"2024-02-13T00:40:40.250911+0000","last_became_active":"2024-02-13T00:37:41.214965+0000","last_became_peered":"2024-02-13T00:37:41.214965+0000","last_unstale":"2024-02-13T00:40:40.250911+0000","last_undegraded":"2024-02-13T00:40:40.250911+0000","last_fullsized":"2024-02-13T00:40:40.250911+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:49.606357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:49.606357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":93,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.256559+0000","last_change":"2024-02-13T00:40:06.169154+0000","last_active":"2024-02-13T00:40:40.256559+0000","last_peered":"2024-02-13T00:40:40.256559+0000","last_clean":"2024-02-13T00:40:40.256559+0000","last_became_active":"2024-02-13T00:37:43.324221+0000","last_became_peered":"2024-02-13T00:37:43.324221+0000","last_unstale":"2024-02-13T00:40:40.256559+0000","last_undegraded":"2024-02-13T00:40:40.256559+0000","last_fullsized":"2024-02-13T00:40:40.256559+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:40:06.169087+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:40:06.169087+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.261196+0000","last_change":"2024-02-13T00:39:21.243606+0000","last_active":"2024-02-13T00:40:40.261196+0000","last_peered":"2024-02-13T00:40:40.261196+0000","last_clean":"2024-02-13T00:40:40.261196+0000","last_became_active":"2024-02-13T00:37:47.552917+0000","last_became_peered":"2024-02-13T00:37:47.552917+0000","last_unstale":"2024-02-13T00:40:40.261196+0000","last_undegraded":"2024-02-13T00:40:40.261196+0000","last_fullsized":"2024-02-13T00:40:40.261196+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.243527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.654521+0000","last_change":"2024-02-13T00:39:13.027403+0000","last_active":"2024-02-13T00:40:40.654521+0000","last_peered":"2024-02-13T00:40:40.654521+0000","last_clean":"2024-02-13T00:40:40.654521+0000","last_became_active":"2024-02-13T00:37:45.248592+0000","last_became_peered":"2024-02-13T00:37:45.248592+0000","last_unstale":"2024-02-13T00:40:40.654521+0000","last_undegraded":"2024-02-13T00:40:40.654521+0000","last_fullsized":"2024-02-13T00:40:40.654521+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:13.027299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:13.027299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655102+0000","last_change":"2024-02-13T00:39:07.358630+0000","last_active":"2024-02-13T00:40:40.655102+0000","last_peered":"2024-02-13T00:40:40.655102+0000","last_clean":"2024-02-13T00:40:40.655102+0000","last_became_active":"2024-02-13T00:37:41.220344+0000","last_became_peered":"2024-02-13T00:37:41.220344+0000","last_unstale":"2024-02-13T00:40:40.655102+0000","last_undegraded":"2024-02-13T00:40:40.655102+0000","last_fullsized":"2024-02-13T00:40:40.655102+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.358534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.358534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":86,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.260342+0000","last_change":"2024-02-13T00:39:34.204969+0000","last_active":"2024-02-13T00:40:40.260342+0000","last_peered":"2024-02-13T00:40:40.260342+0000","last_clean":"2024-02-13T00:40:40.260342+0000","last_became_active":"2024-02-13T00:37:43.324196+0000","last_became_peered":"2024-02-13T00:37:43.324196+0000","last_unstale":"2024-02-13T00:40:40.260342+0000","last_undegraded":"2024-02-13T00:40:40.260342+0000","last_fullsized":"2024-02-13T00:40:40.260342+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:34.204888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.204888+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":56,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:41.220566+0000","last_change":"2024-02-13T00:40:41.220513+0000","last_active":"2024-02-13T00:40:41.220566+0000","last_peered":"2024-02-13T00:40:41.220566+0000","last_clean":"2024-02-13T00:40:41.220566+0000","last_became_active":"2024-02-13T00:37:47.254667+0000","last_became_peered":"2024-02-13T00:37:47.254667+0000","last_unstale":"2024-02-13T00:40:41.220566+0000","last_undegraded":"2024-02-13T00:40:41.220566+0000","last_fullsized":"2024-02-13T00:40:41.220566+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:41.220387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:41.220387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":56,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.654359+0000","last_change":"2024-02-13T00:39:57.978687+0000","last_active":"2024-02-13T00:40:40.654359+0000","last_peered":"2024-02-13T00:40:40.654359+0000","last_clean":"2024-02-13T00:40:40.654359+0000","last_became_active":"2024-02-13T00:37:45.249556+0000","last_became_peered":"2024-02-13T00:37:45.249556+0000","last_unstale":"2024-02-13T00:40:40.654359+0000","last_undegraded":"2024-02-13T00:40:40.654359+0000","last_fullsized":"2024-02-13T00:40:40.654359+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:57.978593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:57.978593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251580+0000","last_change":"2024-02-13T00:40:10.635555+0000","last_active":"2024-02-13T00:40:40.251580+0000","last_peered":"2024-02-13T00:40:40.251580+0000","last_clean":"2024-02-13T00:40:40.251580+0000","last_became_active":"2024-02-13T00:37:41.213033+0000","last_became_peered":"2024-02-13T00:37:41.213033+0000","last_unstale":"2024-02-13T00:40:40.251580+0000","last_undegraded":"2024-02-13T00:40:40.251580+0000","last_fullsized":"2024-02-13T00:40:40.251580+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:10.635478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:10.635478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":58,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.259733+0000","last_change":"2024-02-13T00:39:23.231537+0000","last_active":"2024-02-13T00:40:40.259733+0000","last_peered":"2024-02-13T00:40:40.259733+0000","last_clean":"2024-02-13T00:40:40.259733+0000","last_became_active":"2024-02-13T00:37:43.234395+0000","last_became_peered":"2024-02-13T00:37:43.234395+0000","last_unstale":"2024-02-13T00:40:40.259733+0000","last_undegraded":"2024-02-13T00:40:40.259733+0000","last_fullsized":"2024-02-13T00:40:40.259733+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:39:23.231401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.231401+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.259913+0000","last_change":"2024-02-13T00:40:05.137000+0000","last_active":"2024-02-13T00:40:40.259913+0000","last_peered":"2024-02-13T00:40:40.259913+0000","last_clean":"2024-02-13T00:40:40.259913+0000","last_became_active":"2024-02-13T00:37:47.254519+0000","last_became_peered":"2024-02-13T00:37:47.254519+0000","last_unstale":"2024-02-13T00:40:40.259913+0000","last_undegraded":"2024-02-13T00:40:40.259913+0000","last_fullsized":"2024-02-13T00:40:40.259913+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:05.136935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:05.136935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":51,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.661334+0000","last_change":"2024-02-13T00:38:58.512117+0000","last_active":"2024-02-13T00:40:40.661334+0000","last_peered":"2024-02-13T00:40:40.661334+0000","last_clean":"2024-02-13T00:40:40.661334+0000","last_became_active":"2024-02-13T00:37:45.247114+0000","last_became_peered":"2024-02-13T00:37:45.247114+0000","last_unstale":"2024-02-13T00:40:40.661334+0000","last_undegraded":"2024-02-13T00:40:40.661334+0000","last_fullsized":"2024-02-13T00:40:40.661334+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.512053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.662893+0000","last_change":"2024-02-13T00:39:27.297182+0000","last_active":"2024-02-13T00:40:40.662893+0000","last_peered":"2024-02-13T00:40:40.662893+0000","last_clean":"2024-02-13T00:40:40.662893+0000","last_became_active":"2024-02-13T00:37:41.215531+0000","last_became_peered":"2024-02-13T00:37:41.215531+0000","last_unstale":"2024-02-13T00:40:40.662893+0000","last_undegraded":"2024-02-13T00:40:40.662893+0000","last_fullsized":"2024-02-13T00:40:40.662893+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:27.297107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:27.297107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":68,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:44.273570+0000","last_change":"2024-02-13T00:40:44.273570+0000","last_active":"2024-02-13T00:40:44.273570+0000","last_peered":"2024-02-13T00:40:44.273570+0000","last_clean":"2024-02-13T00:40:44.273570+0000","last_became_active":"2024-02-13T00:40:35.852956+0000","last_became_peered":"2024-02-13T00:40:35.852956+0000","last_unstale":"2024-02-13T00:40:44.273570+0000","last_undegraded":"2024-02-13T00:40:44.273570+0000","last_fullsized":"2024-02-13T00:40:44.273570+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:40:44.273511+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.273511+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.254239+0000","last_change":"2024-02-13T00:39:28.286635+0000","last_active":"2024-02-13T00:40:40.254239+0000","last_peered":"2024-02-13T00:40:40.254239+0000","last_clean":"2024-02-13T00:40:40.254239+0000","last_became_active":"2024-02-13T00:37:47.259792+0000","last_became_peered":"2024-02-13T00:37:47.259792+0000","last_unstale":"2024-02-13T00:40:40.254239+0000","last_undegraded":"2024-02-13T00:40:40.254239+0000","last_fullsized":"2024-02-13T00:40:40.254239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.286574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.286574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.261046+0000","last_change":"2024-02-13T00:39:29.229760+0000","last_active":"2024-02-13T00:40:40.261046+0000","last_peered":"2024-02-13T00:40:40.261046+0000","last_clean":"2024-02-13T00:40:40.261046+0000","last_became_active":"2024-02-13T00:37:45.243594+0000","last_became_peered":"2024-02-13T00:37:45.243594+0000","last_unstale":"2024-02-13T00:40:40.261046+0000","last_undegraded":"2024-02-13T00:40:40.261046+0000","last_fullsized":"2024-02-13T00:40:40.261046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.229676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655101+0000","last_change":"2024-02-13T00:39:00.395451+0000","last_active":"2024-02-13T00:40:40.655101+0000","last_peered":"2024-02-13T00:40:40.655101+0000","last_clean":"2024-02-13T00:40:40.655101+0000","last_became_active":"2024-02-13T00:37:41.216390+0000","last_became_peered":"2024-02-13T00:37:41.216390+0000","last_unstale":"2024-02-13T00:40:40.655101+0000","last_undegraded":"2024-02-13T00:40:40.655101+0000","last_fullsized":"2024-02-13T00:40:40.655101+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:00.395381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:00.395381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":78,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.661259+0000","last_change":"2024-02-13T00:40:16.009588+0000","last_active":"2024-02-13T00:40:40.661259+0000","last_peered":"2024-02-13T00:40:40.661259+0000","last_clean":"2024-02-13T00:40:40.661259+0000","last_became_active":"2024-02-13T00:37:43.225347+0000","last_became_peered":"2024-02-13T00:37:43.225347+0000","last_unstale":"2024-02-13T00:40:40.661259+0000","last_undegraded":"2024-02-13T00:40:40.661259+0000","last_fullsized":"2024-02-13T00:40:40.661259+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:40:16.009518+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:40:16.009518+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":19,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:41.387292+0000","last_change":"2024-02-13T00:40:41.387258+0000","last_active":"2024-02-13T00:40:41.387292+0000","last_peered":"2024-02-13T00:40:41.387292+0000","last_clean":"2024-02-13T00:40:41.387292+0000","last_became_active":"2024-02-13T00:40:35.850176+0000","last_became_peered":"2024-02-13T00:40:35.850176+0000","last_unstale":"2024-02-13T00:40:41.387292+0000","last_undegraded":"2024-02-13T00:40:41.387292+0000","last_fullsized":"2024-02-13T00:40:41.387292+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:41.387157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:41.387157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"84'9","reported_seq":99,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:43.069598+0000","last_change":"2024-02-13T00:39:22.269192+0000","last_active":"2024-02-13T00:40:43.069598+0000","last_peered":"2024-02-13T00:40:43.069598+0000","last_clean":"2024-02-13T00:40:43.069598+0000","last_became_active":"2024-02-13T00:37:45.250598+0000","last_became_peered":"2024-02-13T00:37:45.250598+0000","last_unstale":"2024-02-13T00:40:43.069598+0000","last_undegraded":"2024-02-13T00:40:43.069598+0000","last_fullsized":"2024-02-13T00:40:43.069598+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.269104+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.257030+0000","last_change":"2024-02-13T00:38:58.314612+0000","last_active":"2024-02-13T00:40:40.257030+0000","last_peered":"2024-02-13T00:40:40.257030+0000","last_clean":"2024-02-13T00:40:40.257030+0000","last_became_active":"2024-02-13T00:37:41.217953+0000","last_became_peered":"2024-02-13T00:37:41.217953+0000","last_unstale":"2024-02-13T00:40:40.257030+0000","last_undegraded":"2024-02-13T00:40:40.257030+0000","last_fullsized":"2024-02-13T00:40:40.257030+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.314542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.314542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":84,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251551+0000","last_change":"2024-02-13T00:39:10.549128+0000","last_active":"2024-02-13T00:40:40.251551+0000","last_peered":"2024-02-13T00:40:40.251551+0000","last_clean":"2024-02-13T00:40:40.251551+0000","last_became_active":"2024-02-13T00:37:43.322357+0000","last_became_peered":"2024-02-13T00:37:43.322357+0000","last_unstale":"2024-02-13T00:40:40.251551+0000","last_undegraded":"2024-02-13T00:40:40.251551+0000","last_fullsized":"2024-02-13T00:40:40.251551+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:39:10.549045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.549045+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.256200+0000","last_change":"2024-02-13T00:39:21.344836+0000","last_active":"2024-02-13T00:40:40.256200+0000","last_peered":"2024-02-13T00:40:40.256200+0000","last_clean":"2024-02-13T00:40:40.256200+0000","last_became_active":"2024-02-13T00:37:47.252722+0000","last_became_peered":"2024-02-13T00:37:47.252722+0000","last_unstale":"2024-02-13T00:40:40.256200+0000","last_undegraded":"2024-02-13T00:40:40.256200+0000","last_fullsized":"2024-02-13T00:40:40.256200+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.344771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.344771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"64'8","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:42.903808+0000","last_change":"2024-02-13T00:39:11.518468+0000","last_active":"2024-02-13T00:40:42.903808+0000","last_peered":"2024-02-13T00:40:42.903808+0000","last_clean":"2024-02-13T00:40:42.903808+0000","last_became_active":"2024-02-13T00:37:45.246757+0000","last_became_peered":"2024-02-13T00:37:45.246757+0000","last_unstale":"2024-02-13T00:40:42.903808+0000","last_undegraded":"2024-02-13T00:40:42.903808+0000","last_fullsized":"2024-02-13T00:40:42.903808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:11.518359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:11.518359+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251024+0000","last_change":"2024-02-13T00:39:01.561276+0000","last_active":"2024-02-13T00:40:40.251024+0000","last_peered":"2024-02-13T00:40:40.251024+0000","last_clean":"2024-02-13T00:40:40.251024+0000","last_became_active":"2024-02-13T00:37:41.219309+0000","last_became_peered":"2024-02-13T00:37:41.219309+0000","last_unstale":"2024-02-13T00:40:40.251024+0000","last_undegraded":"2024-02-13T00:40:40.251024+0000","last_fullsized":"2024-02-13T00:40:40.251024+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.561157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.561157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":109,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.264545+0000","last_change":"2024-02-13T00:40:04.185200+0000","last_active":"2024-02-13T00:40:40.264545+0000","last_peered":"2024-02-13T00:40:40.264545+0000","last_clean":"2024-02-13T00:40:40.264545+0000","last_became_active":"2024-02-13T00:37:43.324313+0000","last_became_peered":"2024-02-13T00:37:43.324313+0000","last_unstale":"2024-02-13T00:40:40.264545+0000","last_undegraded":"2024-02-13T00:40:40.264545+0000","last_fullsized":"2024-02-13T00:40:40.264545+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:40:04.185125+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:40:04.185125+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655016+0000","last_change":"2024-02-13T00:39:25.166852+0000","last_active":"2024-02-13T00:40:40.655016+0000","last_peered":"2024-02-13T00:40:40.655016+0000","last_clean":"2024-02-13T00:40:40.655016+0000","last_became_active":"2024-02-13T00:37:47.553612+0000","last_became_peered":"2024-02-13T00:37:47.553612+0000","last_unstale":"2024-02-13T00:40:40.655016+0000","last_undegraded":"2024-02-13T00:40:40.655016+0000","last_fullsized":"2024-02-13T00:40:40.655016+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:25.166777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.166777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251570+0000","last_change":"2024-02-13T00:39:07.556868+0000","last_active":"2024-02-13T00:40:40.251570+0000","last_peered":"2024-02-13T00:40:40.251570+0000","last_clean":"2024-02-13T00:40:40.251570+0000","last_became_active":"2024-02-13T00:37:45.246907+0000","last_became_peered":"2024-02-13T00:37:45.246907+0000","last_unstale":"2024-02-13T00:40:40.251570+0000","last_undegraded":"2024-02-13T00:40:40.251570+0000","last_fullsized":"2024-02-13T00:40:40.251570+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.556772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.556772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.654445+0000","last_change":"2024-02-13T00:39:01.439623+0000","last_active":"2024-02-13T00:40:40.654445+0000","last_peered":"2024-02-13T00:40:40.654445+0000","last_clean":"2024-02-13T00:40:40.654445+0000","last_became_active":"2024-02-13T00:37:41.212866+0000","last_became_peered":"2024-02-13T00:37:41.212866+0000","last_unstale":"2024-02-13T00:40:40.654445+0000","last_undegraded":"2024-02-13T00:40:40.654445+0000","last_fullsized":"2024-02-13T00:40:40.654445+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:39:01.439513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":76,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.261817+0000","last_change":"2024-02-13T00:39:19.266801+0000","last_active":"2024-02-13T00:40:40.261817+0000","last_peered":"2024-02-13T00:40:40.261817+0000","last_clean":"2024-02-13T00:40:40.261817+0000","last_became_active":"2024-02-13T00:37:43.324510+0000","last_became_peered":"2024-02-13T00:37:43.324510+0000","last_unstale":"2024-02-13T00:40:40.261817+0000","last_undegraded":"2024-02-13T00:40:40.261817+0000","last_fullsized":"2024-02-13T00:40:40.261817+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:39:19.266729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:19.266729+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":49,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.657836+0000","last_change":"2024-02-13T00:39:22.615864+0000","last_active":"2024-02-13T00:40:40.657836+0000","last_peered":"2024-02-13T00:40:40.657836+0000","last_clean":"2024-02-13T00:40:40.657836+0000","last_became_active":"2024-02-13T00:37:47.254803+0000","last_became_peered":"2024-02-13T00:37:47.254803+0000","last_unstale":"2024-02-13T00:40:40.657836+0000","last_undegraded":"2024-02-13T00:40:40.657836+0000","last_fullsized":"2024-02-13T00:40:40.657836+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.615773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.615773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.263956+0000","last_change":"2024-02-13T00:39:46.155792+0000","last_active":"2024-02-13T00:40:40.263956+0000","last_peered":"2024-02-13T00:40:40.263956+0000","last_clean":"2024-02-13T00:40:40.263956+0000","last_became_active":"2024-02-13T00:37:45.250632+0000","last_became_peered":"2024-02-13T00:37:45.250632+0000","last_unstale":"2024-02-13T00:40:40.263956+0000","last_undegraded":"2024-02-13T00:40:40.263956+0000","last_fullsized":"2024-02-13T00:40:40.263956+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:46.155718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:46.155718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":60,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.657659+0000","last_change":"2024-02-13T00:40:07.270252+0000","last_active":"2024-02-13T00:40:40.657659+0000","last_peered":"2024-02-13T00:40:40.657659+0000","last_clean":"2024-02-13T00:40:40.657659+0000","last_became_active":"2024-02-13T00:37:41.214091+0000","last_became_peered":"2024-02-13T00:37:41.214091+0000","last_unstale":"2024-02-13T00:40:40.657659+0000","last_undegraded":"2024-02-13T00:40:40.657659+0000","last_fullsized":"2024-02-13T00:40:40.657659+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:07.270157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:07.270157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":98,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.256363+0000","last_change":"2024-02-13T00:38:56.331799+0000","last_active":"2024-02-13T00:40:40.256363+0000","last_peered":"2024-02-13T00:40:40.256363+0000","last_clean":"2024-02-13T00:40:40.256363+0000","last_became_active":"2024-02-13T00:37:43.322401+0000","last_became_peered":"2024-02-13T00:37:43.322401+0000","last_unstale":"2024-02-13T00:40:40.256363+0000","last_undegraded":"2024-02-13T00:40:40.256363+0000","last_fullsized":"2024-02-13T00:40:40.256363+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:38:56.331703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:56.331703+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.261122+0000","last_change":"2024-02-13T00:39:33.179309+0000","last_active":"2024-02-13T00:40:40.261122+0000","last_peered":"2024-02-13T00:40:40.261122+0000","last_clean":"2024-02-13T00:40:40.261122+0000","last_became_active":"2024-02-13T00:37:47.260242+0000","last_became_peered":"2024-02-13T00:37:47.260242+0000","last_unstale":"2024-02-13T00:40:40.261122+0000","last_undegraded":"2024-02-13T00:40:40.261122+0000","last_fullsized":"2024-02-13T00:40:40.261122+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.179225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.179225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":23,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.256099+0000","last_change":"2024-02-13T00:40:35.852695+0000","last_active":"2024-02-13T00:40:40.256099+0000","last_peered":"2024-02-13T00:40:40.256099+0000","last_clean":"2024-02-13T00:40:40.256099+0000","last_became_active":"2024-02-13T00:40:35.852351+0000","last_became_peered":"2024-02-13T00:40:35.852351+0000","last_unstale":"2024-02-13T00:40:40.256099+0000","last_undegraded":"2024-02-13T00:40:40.256099+0000","last_fullsized":"2024-02-13T00:40:40.256099+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:38:48.379184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:48.379184+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":80,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.660639+0000","last_change":"2024-02-13T00:40:35.856095+0000","last_active":"2024-02-13T00:40:40.660639+0000","last_peered":"2024-02-13T00:40:40.660639+0000","last_clean":"2024-02-13T00:40:40.660639+0000","last_became_active":"2024-02-13T00:40:35.855722+0000","last_became_peered":"2024-02-13T00:40:35.855722+0000","last_unstale":"2024-02-13T00:40:40.660639+0000","last_undegraded":"2024-02-13T00:40:40.660639+0000","last_fullsized":"2024-02-13T00:40:40.660639+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'24","last_scrub_stamp":"2024-02-13T00:39:56.121046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:56.121046+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":36,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":51,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.657402+0000","last_change":"2024-02-13T00:39:21.654834+0000","last_active":"2024-02-13T00:40:40.657402+0000","last_peered":"2024-02-13T00:40:40.657402+0000","last_clean":"2024-02-13T00:40:40.657402+0000","last_became_active":"2024-02-13T00:37:45.244408+0000","last_became_peered":"2024-02-13T00:37:45.244408+0000","last_unstale":"2024-02-13T00:40:40.657402+0000","last_undegraded":"2024-02-13T00:40:40.657402+0000","last_fullsized":"2024-02-13T00:40:40.657402+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.654733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.654733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.254327+0000","last_change":"2024-02-13T00:38:54.349240+0000","last_active":"2024-02-13T00:40:40.254327+0000","last_peered":"2024-02-13T00:40:40.254327+0000","last_clean":"2024-02-13T00:40:40.254327+0000","last_became_active":"2024-02-13T00:37:47.255928+0000","last_became_peered":"2024-02-13T00:37:47.255928+0000","last_unstale":"2024-02-13T00:40:40.254327+0000","last_undegraded":"2024-02-13T00:40:40.254327+0000","last_fullsized":"2024-02-13T00:40:40.254327+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:54.349171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:54.349171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":60,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.662149+0000","last_change":"2024-02-13T00:40:08.946462+0000","last_active":"2024-02-13T00:40:40.662149+0000","last_peered":"2024-02-13T00:40:40.662149+0000","last_clean":"2024-02-13T00:40:40.662149+0000","last_became_active":"2024-02-13T00:37:41.217777+0000","last_became_peered":"2024-02-13T00:37:41.217777+0000","last_unstale":"2024-02-13T00:40:40.662149+0000","last_undegraded":"2024-02-13T00:40:40.662149+0000","last_fullsized":"2024-02-13T00:40:40.662149+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:08.946388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:40:08.946388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.657509+0000","last_change":"2024-02-13T00:40:16.354241+0000","last_active":"2024-02-13T00:40:40.657509+0000","last_peered":"2024-02-13T00:40:40.657509+0000","last_clean":"2024-02-13T00:40:40.657509+0000","last_became_active":"2024-02-13T00:39:05.965560+0000","last_became_peered":"2024-02-13T00:39:05.965560+0000","last_unstale":"2024-02-13T00:40:40.657509+0000","last_undegraded":"2024-02-13T00:40:40.657509+0000","last_fullsized":"2024-02-13T00:40:40.657509+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:40:16.354164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:16.354164+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"64'8","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:42.908926+0000","last_change":"2024-02-13T00:39:30.273237+0000","last_active":"2024-02-13T00:40:42.908926+0000","last_peered":"2024-02-13T00:40:42.908926+0000","last_clean":"2024-02-13T00:40:42.908926+0000","last_became_active":"2024-02-13T00:37:45.241274+0000","last_became_peered":"2024-02-13T00:37:45.241274+0000","last_unstale":"2024-02-13T00:40:42.908926+0000","last_undegraded":"2024-02-13T00:40:42.908926+0000","last_fullsized":"2024-02-13T00:40:42.908926+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:30.273165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.273165+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.261562+0000","last_change":"2024-02-13T00:39:36.157777+0000","last_active":"2024-02-13T00:40:40.261562+0000","last_peered":"2024-02-13T00:40:40.261562+0000","last_clean":"2024-02-13T00:40:40.261562+0000","last_became_active":"2024-02-13T00:37:47.252270+0000","last_became_peered":"2024-02-13T00:37:47.252270+0000","last_unstale":"2024-02-13T00:40:40.261562+0000","last_undegraded":"2024-02-13T00:40:40.261562+0000","last_fullsized":"2024-02-13T00:40:40.261562+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:36.157693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:36.157693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251193+0000","last_change":"2024-02-13T00:38:58.528669+0000","last_active":"2024-02-13T00:40:40.251193+0000","last_peered":"2024-02-13T00:40:40.251193+0000","last_clean":"2024-02-13T00:40:40.251193+0000","last_became_active":"2024-02-13T00:37:41.212924+0000","last_became_peered":"2024-02-13T00:37:41.212924+0000","last_unstale":"2024-02-13T00:40:40.251193+0000","last_undegraded":"2024-02-13T00:40:40.251193+0000","last_fullsized":"2024-02-13T00:40:40.251193+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:58.528580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.528580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"80'45","reported_seq":169,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:43.728014+0000","last_change":"2024-02-13T00:40:12.047980+0000","last_active":"2024-02-13T00:40:43.728014+0000","last_peered":"2024-02-13T00:40:43.728014+0000","last_clean":"2024-02-13T00:40:43.728014+0000","last_became_active":"2024-02-13T00:37:43.232014+0000","last_became_peered":"2024-02-13T00:37:43.232014+0000","last_unstale":"2024-02-13T00:40:43.728014+0000","last_undegraded":"2024-02-13T00:40:43.728014+0000","last_fullsized":"2024-02-13T00:40:43.728014+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'45","last_scrub_stamp":"2024-02-13T00:40:12.047869+0000","last_deep_scrub":"80'45","last_deep_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_clean_scrub_stamp":"2024-02-13T00:40:12.047869+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.255808+0000","last_change":"2024-02-13T00:38:47.393545+0000","last_active":"2024-02-13T00:40:40.255808+0000","last_peered":"2024-02-13T00:40:40.255808+0000","last_clean":"2024-02-13T00:40:40.255808+0000","last_became_active":"2024-02-13T00:37:45.247012+0000","last_became_peered":"2024-02-13T00:37:45.247012+0000","last_unstale":"2024-02-13T00:40:40.255808+0000","last_undegraded":"2024-02-13T00:40:40.255808+0000","last_fullsized":"2024-02-13T00:40:40.255808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:47.393477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:38:47.393477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655223+0000","last_change":"2024-02-13T00:39:08.930351+0000","last_active":"2024-02-13T00:40:40.655223+0000","last_peered":"2024-02-13T00:40:40.655223+0000","last_clean":"2024-02-13T00:40:40.655223+0000","last_became_active":"2024-02-13T00:37:47.552693+0000","last_became_peered":"2024-02-13T00:37:47.552693+0000","last_unstale":"2024-02-13T00:40:40.655223+0000","last_undegraded":"2024-02-13T00:40:40.655223+0000","last_fullsized":"2024-02-13T00:40:40.655223+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:08.930286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.930286+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":237,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:43.727661+0000","last_change":"2024-02-13T00:39:10.287406+0000","last_active":"2024-02-13T00:40:43.727661+0000","last_peered":"2024-02-13T00:40:43.727661+0000","last_clean":"2024-02-13T00:40:43.727661+0000","last_became_active":"2024-02-13T00:37:41.215137+0000","last_became_peered":"2024-02-13T00:37:41.215137+0000","last_unstale":"2024-02-13T00:40:43.727661+0000","last_undegraded":"2024-02-13T00:40:43.727661+0000","last_fullsized":"2024-02-13T00:40:43.727661+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-13T00:39:10.287299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.287299+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":177,"num_read_kb":172,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.658634+0000","last_change":"2024-02-13T00:39:20.703787+0000","last_active":"2024-02-13T00:40:40.658634+0000","last_peered":"2024-02-13T00:40:40.658634+0000","last_clean":"2024-02-13T00:40:40.658634+0000","last_became_active":"2024-02-13T00:37:41.214442+0000","last_became_peered":"2024-02-13T00:37:41.214442+0000","last_unstale":"2024-02-13T00:40:40.658634+0000","last_undegraded":"2024-02-13T00:40:40.658634+0000","last_fullsized":"2024-02-13T00:40:40.658634+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:20.703700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:20.703700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":95,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.663720+0000","last_change":"2024-02-13T00:39:25.319173+0000","last_active":"2024-02-13T00:40:40.663720+0000","last_peered":"2024-02-13T00:40:40.663720+0000","last_clean":"2024-02-13T00:40:40.663720+0000","last_became_active":"2024-02-13T00:37:43.231120+0000","last_became_peered":"2024-02-13T00:37:43.231120+0000","last_unstale":"2024-02-13T00:40:40.663720+0000","last_undegraded":"2024-02-13T00:40:40.663720+0000","last_fullsized":"2024-02-13T00:40:40.663720+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:39:25.319066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.319066+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":51,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.657929+0000","last_change":"2024-02-13T00:39:09.589876+0000","last_active":"2024-02-13T00:40:40.657929+0000","last_peered":"2024-02-13T00:40:40.657929+0000","last_clean":"2024-02-13T00:40:40.657929+0000","last_became_active":"2024-02-13T00:37:45.248383+0000","last_became_peered":"2024-02-13T00:37:45.248383+0000","last_unstale":"2024-02-13T00:40:40.657929+0000","last_undegraded":"2024-02-13T00:40:40.657929+0000","last_fullsized":"2024-02-13T00:40:40.657929+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:09.589813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:09.589813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":53,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251812+0000","last_change":"2024-02-13T00:40:08.577747+0000","last_active":"2024-02-13T00:40:40.251812+0000","last_peered":"2024-02-13T00:40:40.251812+0000","last_clean":"2024-02-13T00:40:40.251812+0000","last_became_active":"2024-02-13T00:37:47.253996+0000","last_became_peered":"2024-02-13T00:37:47.253996+0000","last_unstale":"2024-02-13T00:40:40.251812+0000","last_undegraded":"2024-02-13T00:40:40.251812+0000","last_fullsized":"2024-02-13T00:40:40.251812+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:08.577665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:40:08.577665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.259531+0000","last_change":"2024-02-13T00:40:09.099541+0000","last_active":"2024-02-13T00:40:40.259531+0000","last_peered":"2024-02-13T00:40:40.259531+0000","last_clean":"2024-02-13T00:40:40.259531+0000","last_became_active":"2024-02-13T00:37:41.216367+0000","last_became_peered":"2024-02-13T00:37:41.216367+0000","last_unstale":"2024-02-13T00:40:40.259531+0000","last_undegraded":"2024-02-13T00:40:40.259531+0000","last_fullsized":"2024-02-13T00:40:40.259531+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:09.099460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:40:09.099460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251232+0000","last_change":"2024-02-13T00:38:52.618768+0000","last_active":"2024-02-13T00:40:40.251232+0000","last_peered":"2024-02-13T00:40:40.251232+0000","last_clean":"2024-02-13T00:40:40.251232+0000","last_became_active":"2024-02-13T00:37:41.216101+0000","last_became_peered":"2024-02-13T00:37:41.216101+0000","last_unstale":"2024-02-13T00:40:40.251232+0000","last_undegraded":"2024-02-13T00:40:40.251232+0000","last_fullsized":"2024-02-13T00:40:40.251232+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.618696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.618696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":62,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.654212+0000","last_change":"2024-02-13T00:39:41.255860+0000","last_active":"2024-02-13T00:40:40.654212+0000","last_peered":"2024-02-13T00:40:40.654212+0000","last_clean":"2024-02-13T00:40:40.654212+0000","last_became_active":"2024-02-13T00:37:35.635369+0000","last_became_peered":"2024-02-13T00:37:35.635369+0000","last_unstale":"2024-02-13T00:40:40.654212+0000","last_undegraded":"2024-02-13T00:40:40.654212+0000","last_fullsized":"2024-02-13T00:40:40.654212+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:39:41.255785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'14","reported_seq":75,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655516+0000","last_change":"2024-02-13T00:39:15.415087+0000","last_active":"2024-02-13T00:40:40.655516+0000","last_peered":"2024-02-13T00:40:40.655516+0000","last_clean":"2024-02-13T00:40:40.655516+0000","last_became_active":"2024-02-13T00:37:43.324209+0000","last_became_peered":"2024-02-13T00:37:43.324209+0000","last_unstale":"2024-02-13T00:40:40.655516+0000","last_undegraded":"2024-02-13T00:40:40.655516+0000","last_fullsized":"2024-02-13T00:40:40.655516+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'14","last_scrub_stamp":"2024-02-13T00:39:15.415004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.415004+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655608+0000","last_change":"2024-02-13T00:39:28.171808+0000","last_active":"2024-02-13T00:40:40.655608+0000","last_peered":"2024-02-13T00:40:40.655608+0000","last_clean":"2024-02-13T00:40:40.655608+0000","last_became_active":"2024-02-13T00:37:45.247708+0000","last_became_peered":"2024-02-13T00:37:45.247708+0000","last_unstale":"2024-02-13T00:40:40.655608+0000","last_undegraded":"2024-02-13T00:40:40.655608+0000","last_fullsized":"2024-02-13T00:40:40.655608+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.171740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.171740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":49,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.661887+0000","last_change":"2024-02-13T00:39:32.255327+0000","last_active":"2024-02-13T00:40:40.661887+0000","last_peered":"2024-02-13T00:40:40.661887+0000","last_clean":"2024-02-13T00:40:40.661887+0000","last_became_active":"2024-02-13T00:37:47.262024+0000","last_became_peered":"2024-02-13T00:37:47.262024+0000","last_unstale":"2024-02-13T00:40:40.661887+0000","last_undegraded":"2024-02-13T00:40:40.661887+0000","last_fullsized":"2024-02-13T00:40:40.661887+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.255260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.255040+0000","last_change":"2024-02-13T00:38:55.353229+0000","last_active":"2024-02-13T00:40:40.255040+0000","last_peered":"2024-02-13T00:40:40.255040+0000","last_clean":"2024-02-13T00:40:40.255040+0000","last_became_active":"2024-02-13T00:37:41.217577+0000","last_became_peered":"2024-02-13T00:37:41.217577+0000","last_unstale":"2024-02-13T00:40:40.255040+0000","last_undegraded":"2024-02-13T00:40:40.255040+0000","last_fullsized":"2024-02-13T00:40:40.255040+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:55.353157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:55.353157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:42.413152+0000","last_change":"2024-02-13T00:40:42.413113+0000","last_active":"2024-02-13T00:40:42.413152+0000","last_peered":"2024-02-13T00:40:42.413152+0000","last_clean":"2024-02-13T00:40:42.413152+0000","last_became_active":"2024-02-13T00:39:05.967772+0000","last_became_peered":"2024-02-13T00:39:05.967772+0000","last_unstale":"2024-02-13T00:40:42.413152+0000","last_undegraded":"2024-02-13T00:40:42.413152+0000","last_fullsized":"2024-02-13T00:40:42.413152+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:42.413043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:42.413043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":93,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.256282+0000","last_change":"2024-02-13T00:39:08.344202+0000","last_active":"2024-02-13T00:40:40.256282+0000","last_peered":"2024-02-13T00:40:40.256282+0000","last_clean":"2024-02-13T00:40:40.256282+0000","last_became_active":"2024-02-13T00:39:05.968155+0000","last_became_peered":"2024-02-13T00:39:05.968155+0000","last_unstale":"2024-02-13T00:40:40.256282+0000","last_undegraded":"2024-02-13T00:40:40.256282+0000","last_fullsized":"2024-02-13T00:40:40.256282+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:39:08.344129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.344129+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251777+0000","last_change":"2024-02-13T00:40:03.701680+0000","last_active":"2024-02-13T00:40:40.251777+0000","last_peered":"2024-02-13T00:40:40.251777+0000","last_clean":"2024-02-13T00:40:40.251777+0000","last_became_active":"2024-02-13T00:37:45.245956+0000","last_became_peered":"2024-02-13T00:37:45.245956+0000","last_unstale":"2024-02-13T00:40:40.251777+0000","last_undegraded":"2024-02-13T00:40:40.251777+0000","last_fullsized":"2024-02-13T00:40:40.251777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:03.701604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:03.701604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":49,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.660499+0000","last_change":"2024-02-13T00:39:33.304107+0000","last_active":"2024-02-13T00:40:40.660499+0000","last_peered":"2024-02-13T00:40:40.660499+0000","last_clean":"2024-02-13T00:40:40.660499+0000","last_became_active":"2024-02-13T00:37:47.552672+0000","last_became_peered":"2024-02-13T00:37:47.552672+0000","last_unstale":"2024-02-13T00:40:40.660499+0000","last_undegraded":"2024-02-13T00:40:40.660499+0000","last_fullsized":"2024-02-13T00:40:40.660499+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.303982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.303982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655324+0000","last_change":"2024-02-13T00:38:46.927138+0000","last_active":"2024-02-13T00:40:40.655324+0000","last_peered":"2024-02-13T00:40:40.655324+0000","last_clean":"2024-02-13T00:40:40.655324+0000","last_became_active":"2024-02-13T00:37:41.221636+0000","last_became_peered":"2024-02-13T00:37:41.221636+0000","last_unstale":"2024-02-13T00:40:40.655324+0000","last_undegraded":"2024-02-13T00:40:40.655324+0000","last_fullsized":"2024-02-13T00:40:40.655324+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:46.927068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:46.927068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":112,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.256917+0000","last_change":"2024-02-13T00:39:04.318990+0000","last_active":"2024-02-13T00:40:40.256917+0000","last_peered":"2024-02-13T00:40:40.256917+0000","last_clean":"2024-02-13T00:40:40.256917+0000","last_became_active":"2024-02-13T00:37:43.322450+0000","last_became_peered":"2024-02-13T00:37:43.322450+0000","last_unstale":"2024-02-13T00:40:40.256917+0000","last_undegraded":"2024-02-13T00:40:40.256917+0000","last_fullsized":"2024-02-13T00:40:40.256917+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_clean_scrub_stamp":"2024-02-13T00:39:04.318923+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":51,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655226+0000","last_change":"2024-02-13T00:39:20.451151+0000","last_active":"2024-02-13T00:40:40.655226+0000","last_peered":"2024-02-13T00:40:40.655226+0000","last_clean":"2024-02-13T00:40:40.655226+0000","last_became_active":"2024-02-13T00:37:45.245974+0000","last_became_peered":"2024-02-13T00:37:45.245974+0000","last_unstale":"2024-02-13T00:40:40.655226+0000","last_undegraded":"2024-02-13T00:40:40.655226+0000","last_fullsized":"2024-02-13T00:40:40.655226+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:20.451088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:20.451088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.259446+0000","last_change":"2024-02-13T00:40:35.858228+0000","last_active":"2024-02-13T00:40:40.259446+0000","last_peered":"2024-02-13T00:40:40.259446+0000","last_clean":"2024-02-13T00:40:40.259446+0000","last_became_active":"2024-02-13T00:40:35.857936+0000","last_became_peered":"2024-02-13T00:40:35.857936+0000","last_unstale":"2024-02-13T00:40:40.259446+0000","last_undegraded":"2024-02-13T00:40:40.259446+0000","last_fullsized":"2024-02-13T00:40:40.259446+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:09.237312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:09.237312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.663465+0000","last_change":"2024-02-13T00:39:21.329248+0000","last_active":"2024-02-13T00:40:40.663465+0000","last_peered":"2024-02-13T00:40:40.663465+0000","last_clean":"2024-02-13T00:40:40.663465+0000","last_became_active":"2024-02-13T00:37:41.214202+0000","last_became_peered":"2024-02-13T00:37:41.214202+0000","last_unstale":"2024-02-13T00:40:40.663465+0000","last_undegraded":"2024-02-13T00:40:40.663465+0000","last_fullsized":"2024-02-13T00:40:40.663465+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.329173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.329173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":114,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.259598+0000","last_change":"2024-02-13T00:38:58.238608+0000","last_active":"2024-02-13T00:40:40.259598+0000","last_peered":"2024-02-13T00:40:40.259598+0000","last_clean":"2024-02-13T00:40:40.259598+0000","last_became_active":"2024-02-13T00:37:43.324275+0000","last_became_peered":"2024-02-13T00:37:43.324275+0000","last_unstale":"2024-02-13T00:40:40.259598+0000","last_undegraded":"2024-02-13T00:40:40.259598+0000","last_fullsized":"2024-02-13T00:40:40.259598+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:38:58.238526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:38:58.238526+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":19,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:43.011847+0000","last_change":"2024-02-13T00:40:43.011812+0000","last_active":"2024-02-13T00:40:43.011847+0000","last_peered":"2024-02-13T00:40:43.011847+0000","last_clean":"2024-02-13T00:40:43.011847+0000","last_became_active":"2024-02-13T00:40:35.856681+0000","last_became_peered":"2024-02-13T00:40:35.856681+0000","last_unstale":"2024-02-13T00:40:43.011847+0000","last_undegraded":"2024-02-13T00:40:43.011847+0000","last_fullsized":"2024-02-13T00:40:43.011847+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:43.011693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:43.011693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":53,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:44.243015+0000","last_change":"2024-02-13T00:40:44.243015+0000","last_active":"2024-02-13T00:40:44.243015+0000","last_peered":"2024-02-13T00:40:44.243015+0000","last_clean":"2024-02-13T00:40:44.243015+0000","last_became_active":"2024-02-13T00:37:47.553554+0000","last_became_peered":"2024-02-13T00:37:47.553554+0000","last_unstale":"2024-02-13T00:40:44.243015+0000","last_undegraded":"2024-02-13T00:40:44.243015+0000","last_fullsized":"2024-02-13T00:40:44.243015+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:44.242946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.242946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655012+0000","last_change":"2024-02-13T00:39:14.367203+0000","last_active":"2024-02-13T00:40:40.655012+0000","last_peered":"2024-02-13T00:40:40.655012+0000","last_clean":"2024-02-13T00:40:40.655012+0000","last_became_active":"2024-02-13T00:37:41.218313+0000","last_became_peered":"2024-02-13T00:37:41.218313+0000","last_unstale":"2024-02-13T00:40:40.655012+0000","last_undegraded":"2024-02-13T00:40:40.655012+0000","last_fullsized":"2024-02-13T00:40:40.655012+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:14.367115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:14.367115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":68,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:43.630615+0000","last_change":"2024-02-13T00:40:43.630615+0000","last_active":"2024-02-13T00:40:43.630615+0000","last_peered":"2024-02-13T00:40:43.630615+0000","last_clean":"2024-02-13T00:40:43.630615+0000","last_became_active":"2024-02-13T00:40:35.844184+0000","last_became_peered":"2024-02-13T00:40:35.844184+0000","last_unstale":"2024-02-13T00:40:43.630615+0000","last_undegraded":"2024-02-13T00:40:43.630615+0000","last_fullsized":"2024-02-13T00:40:43.630615+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2024-02-13T00:40:43.630535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:43.630535+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"64'8","reported_seq":97,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:43.068624+0000","last_change":"2024-02-13T00:39:16.304400+0000","last_active":"2024-02-13T00:40:43.068624+0000","last_peered":"2024-02-13T00:40:43.068624+0000","last_clean":"2024-02-13T00:40:43.068624+0000","last_became_active":"2024-02-13T00:37:45.245528+0000","last_became_peered":"2024-02-13T00:37:45.245528+0000","last_unstale":"2024-02-13T00:40:43.068624+0000","last_undegraded":"2024-02-13T00:40:43.068624+0000","last_fullsized":"2024-02-13T00:40:43.068624+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:16.304316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.304316+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":53,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.264467+0000","last_change":"2024-02-13T00:40:14.015620+0000","last_active":"2024-02-13T00:40:40.264467+0000","last_peered":"2024-02-13T00:40:40.264467+0000","last_clean":"2024-02-13T00:40:40.264467+0000","last_became_active":"2024-02-13T00:37:47.552591+0000","last_became_peered":"2024-02-13T00:37:47.552591+0000","last_unstale":"2024-02-13T00:40:40.264467+0000","last_undegraded":"2024-02-13T00:40:40.264467+0000","last_fullsized":"2024-02-13T00:40:40.264467+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:14.015547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:14.015547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251685+0000","last_change":"2024-02-13T00:38:57.494754+0000","last_active":"2024-02-13T00:40:40.251685+0000","last_peered":"2024-02-13T00:40:40.251685+0000","last_clean":"2024-02-13T00:40:40.251685+0000","last_became_active":"2024-02-13T00:37:45.245897+0000","last_became_peered":"2024-02-13T00:37:45.245897+0000","last_unstale":"2024-02-13T00:40:40.251685+0000","last_undegraded":"2024-02-13T00:40:40.251685+0000","last_fullsized":"2024-02-13T00:40:40.251685+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:38:57.494653+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.654304+0000","last_change":"2024-02-13T00:39:29.159172+0000","last_active":"2024-02-13T00:40:40.654304+0000","last_peered":"2024-02-13T00:40:40.654304+0000","last_clean":"2024-02-13T00:40:40.654304+0000","last_became_active":"2024-02-13T00:37:41.221627+0000","last_became_peered":"2024-02-13T00:37:41.221627+0000","last_unstale":"2024-02-13T00:40:40.654304+0000","last_undegraded":"2024-02-13T00:40:40.654304+0000","last_fullsized":"2024-02-13T00:40:40.654304+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.159080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.159080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":61,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655472+0000","last_change":"2024-02-13T00:40:35.848232+0000","last_active":"2024-02-13T00:40:40.655472+0000","last_peered":"2024-02-13T00:40:40.655472+0000","last_clean":"2024-02-13T00:40:40.655472+0000","last_became_active":"2024-02-13T00:40:35.847905+0000","last_became_peered":"2024-02-13T00:40:35.847905+0000","last_unstale":"2024-02-13T00:40:40.655472+0000","last_undegraded":"2024-02-13T00:40:40.655472+0000","last_fullsized":"2024-02-13T00:40:40.655472+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:29.275134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.275134+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251301+0000","last_change":"2024-02-13T00:38:54.599127+0000","last_active":"2024-02-13T00:40:40.251301+0000","last_peered":"2024-02-13T00:40:40.251301+0000","last_clean":"2024-02-13T00:40:40.251301+0000","last_became_active":"2024-02-13T00:37:47.260477+0000","last_became_peered":"2024-02-13T00:37:47.260477+0000","last_unstale":"2024-02-13T00:40:40.251301+0000","last_undegraded":"2024-02-13T00:40:40.251301+0000","last_fullsized":"2024-02-13T00:40:40.251301+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:54.599057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:54.599057+0000","last_clean_scrub_stamp":"2024-02-13T00:38:54.599057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"96'9","reported_seq":36,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:42.904087+0000","last_change":"2024-02-13T00:40:35.847666+0000","last_active":"2024-02-13T00:40:42.904087+0000","last_peered":"2024-02-13T00:40:42.904087+0000","last_clean":"2024-02-13T00:40:42.904087+0000","last_became_active":"2024-02-13T00:40:35.847293+0000","last_became_peered":"2024-02-13T00:40:35.847293+0000","last_unstale":"2024-02-13T00:40:42.904087+0000","last_undegraded":"2024-02-13T00:40:42.904087+0000","last_fullsized":"2024-02-13T00:40:42.904087+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:40:02.089864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:02.089864+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":66,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.660741+0000","last_change":"2024-02-13T00:39:52.206001+0000","last_active":"2024-02-13T00:40:40.660741+0000","last_peered":"2024-02-13T00:40:40.660741+0000","last_clean":"2024-02-13T00:40:40.660741+0000","last_became_active":"2024-02-13T00:37:41.217979+0000","last_became_peered":"2024-02-13T00:37:41.217979+0000","last_unstale":"2024-02-13T00:40:40.660741+0000","last_undegraded":"2024-02-13T00:40:40.660741+0000","last_fullsized":"2024-02-13T00:40:40.660741+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:39:52.205926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:52.205926+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":78,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.663855+0000","last_change":"2024-02-13T00:39:23.370512+0000","last_active":"2024-02-13T00:40:40.663855+0000","last_peered":"2024-02-13T00:40:40.663855+0000","last_clean":"2024-02-13T00:40:40.663855+0000","last_became_active":"2024-02-13T00:37:43.232161+0000","last_became_peered":"2024-02-13T00:37:43.232161+0000","last_unstale":"2024-02-13T00:40:40.663855+0000","last_undegraded":"2024-02-13T00:40:40.663855+0000","last_fullsized":"2024-02-13T00:40:40.663855+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:39:23.370447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.370447+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":28,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.750420+0000","last_change":"2024-02-13T00:40:40.750384+0000","last_active":"2024-02-13T00:40:40.750420+0000","last_peered":"2024-02-13T00:40:40.750420+0000","last_clean":"2024-02-13T00:40:40.750420+0000","last_became_active":"2024-02-13T00:40:35.851908+0000","last_became_peered":"2024-02-13T00:40:35.851908+0000","last_unstale":"2024-02-13T00:40:40.750420+0000","last_undegraded":"2024-02-13T00:40:40.750420+0000","last_fullsized":"2024-02-13T00:40:40.750420+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:40:40.750272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:40.750272+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"64'8","reported_seq":96,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:42.908114+0000","last_change":"2024-02-13T00:39:14.609925+0000","last_active":"2024-02-13T00:40:42.908114+0000","last_peered":"2024-02-13T00:40:42.908114+0000","last_clean":"2024-02-13T00:40:42.908114+0000","last_became_active":"2024-02-13T00:37:45.249684+0000","last_became_peered":"2024-02-13T00:37:45.249684+0000","last_unstale":"2024-02-13T00:40:42.908114+0000","last_undegraded":"2024-02-13T00:40:42.908114+0000","last_fullsized":"2024-02-13T00:40:42.908114+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:14.609850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:14.609850+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.661180+0000","last_change":"2024-02-13T00:39:17.283996+0000","last_active":"2024-02-13T00:40:40.661180+0000","last_peered":"2024-02-13T00:40:40.661180+0000","last_clean":"2024-02-13T00:40:40.661180+0000","last_became_active":"2024-02-13T00:37:41.218346+0000","last_became_peered":"2024-02-13T00:37:41.218346+0000","last_unstale":"2024-02-13T00:40:40.661180+0000","last_undegraded":"2024-02-13T00:40:40.661180+0000","last_fullsized":"2024-02-13T00:40:40.661180+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.283902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.283902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":64,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.264742+0000","last_change":"2024-02-13T00:40:35.856263+0000","last_active":"2024-02-13T00:40:40.264742+0000","last_peered":"2024-02-13T00:40:40.264742+0000","last_clean":"2024-02-13T00:40:40.264742+0000","last_became_active":"2024-02-13T00:40:35.855827+0000","last_became_peered":"2024-02-13T00:40:35.855827+0000","last_unstale":"2024-02-13T00:40:40.264742+0000","last_undegraded":"2024-02-13T00:40:40.264742+0000","last_fullsized":"2024-02-13T00:40:40.264742+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.305179+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":49,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.657581+0000","last_change":"2024-02-13T00:39:07.563502+0000","last_active":"2024-02-13T00:40:40.657581+0000","last_peered":"2024-02-13T00:40:40.657581+0000","last_clean":"2024-02-13T00:40:40.657581+0000","last_became_active":"2024-02-13T00:37:47.256807+0000","last_became_peered":"2024-02-13T00:37:47.256807+0000","last_unstale":"2024-02-13T00:40:40.657581+0000","last_undegraded":"2024-02-13T00:40:40.657581+0000","last_fullsized":"2024-02-13T00:40:40.657581+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.563440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.563440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":51,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.661966+0000","last_change":"2024-02-13T00:39:35.269607+0000","last_active":"2024-02-13T00:40:40.661966+0000","last_peered":"2024-02-13T00:40:40.661966+0000","last_clean":"2024-02-13T00:40:40.661966+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:40:40.661966+0000","last_undegraded":"2024-02-13T00:40:40.661966+0000","last_fullsized":"2024-02-13T00:40:40.661966+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:35.269538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:35.269538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":61,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655028+0000","last_change":"2024-02-13T00:39:24.138710+0000","last_active":"2024-02-13T00:40:40.655028+0000","last_peered":"2024-02-13T00:40:40.655028+0000","last_clean":"2024-02-13T00:40:40.655028+0000","last_became_active":"2024-02-13T00:37:41.220874+0000","last_became_peered":"2024-02-13T00:37:41.220874+0000","last_unstale":"2024-02-13T00:40:40.655028+0000","last_undegraded":"2024-02-13T00:40:40.655028+0000","last_fullsized":"2024-02-13T00:40:40.655028+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:39:24.138629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.138629+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":54,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.657591+0000","last_change":"2024-02-13T00:40:35.850976+0000","last_active":"2024-02-13T00:40:40.657591+0000","last_peered":"2024-02-13T00:40:40.657591+0000","last_clean":"2024-02-13T00:40:40.657591+0000","last_became_active":"2024-02-13T00:40:35.850675+0000","last_became_peered":"2024-02-13T00:40:35.850675+0000","last_unstale":"2024-02-13T00:40:40.657591+0000","last_undegraded":"2024-02-13T00:40:40.657591+0000","last_fullsized":"2024-02-13T00:40:40.657591+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:39:02.188096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.188096+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.654110+0000","last_change":"2024-02-13T00:39:16.086278+0000","last_active":"2024-02-13T00:40:40.654110+0000","last_peered":"2024-02-13T00:40:40.654110+0000","last_clean":"2024-02-13T00:40:40.654110+0000","last_became_active":"2024-02-13T00:37:47.254705+0000","last_became_peered":"2024-02-13T00:37:47.254705+0000","last_unstale":"2024-02-13T00:40:40.654110+0000","last_undegraded":"2024-02-13T00:40:40.654110+0000","last_fullsized":"2024-02-13T00:40:40.654110+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.086207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.086207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":51,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.657587+0000","last_change":"2024-02-13T00:39:17.631975+0000","last_active":"2024-02-13T00:40:40.657587+0000","last_peered":"2024-02-13T00:40:40.657587+0000","last_clean":"2024-02-13T00:40:40.657587+0000","last_became_active":"2024-02-13T00:37:45.239867+0000","last_became_peered":"2024-02-13T00:37:45.239867+0000","last_unstale":"2024-02-13T00:40:40.657587+0000","last_undegraded":"2024-02-13T00:40:40.657587+0000","last_fullsized":"2024-02-13T00:40:40.657587+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.631877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.658081+0000","last_change":"2024-02-13T00:39:18.672595+0000","last_active":"2024-02-13T00:40:40.658081+0000","last_peered":"2024-02-13T00:40:40.658081+0000","last_clean":"2024-02-13T00:40:40.658081+0000","last_became_active":"2024-02-13T00:37:41.219457+0000","last_became_peered":"2024-02-13T00:37:41.219457+0000","last_unstale":"2024-02-13T00:40:40.658081+0000","last_undegraded":"2024-02-13T00:40:40.658081+0000","last_fullsized":"2024-02-13T00:40:40.658081+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.672516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.672516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"91'35","reported_seq":114,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.655143+0000","last_change":"2024-02-13T00:40:15.074106+0000","last_active":"2024-02-13T00:40:40.655143+0000","last_peered":"2024-02-13T00:40:40.655143+0000","last_clean":"2024-02-13T00:40:40.655143+0000","last_became_active":"2024-02-13T00:37:43.322957+0000","last_became_peered":"2024-02-13T00:37:43.322957+0000","last_unstale":"2024-02-13T00:40:40.655143+0000","last_undegraded":"2024-02-13T00:40:40.655143+0000","last_fullsized":"2024-02-13T00:40:40.655143+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:40:15.074034+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.254813+0000","last_change":"2024-02-13T00:39:18.359681+0000","last_active":"2024-02-13T00:40:40.254813+0000","last_peered":"2024-02-13T00:40:40.254813+0000","last_clean":"2024-02-13T00:40:40.254813+0000","last_became_active":"2024-02-13T00:37:47.249556+0000","last_became_peered":"2024-02-13T00:37:47.249556+0000","last_unstale":"2024-02-13T00:40:40.254813+0000","last_undegraded":"2024-02-13T00:40:40.254813+0000","last_fullsized":"2024-02-13T00:40:40.254813+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.359604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":51,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.657778+0000","last_change":"2024-02-13T00:39:15.603878+0000","last_active":"2024-02-13T00:40:40.657778+0000","last_peered":"2024-02-13T00:40:40.657778+0000","last_clean":"2024-02-13T00:40:40.657778+0000","last_became_active":"2024-02-13T00:37:45.240923+0000","last_became_peered":"2024-02-13T00:37:45.240923+0000","last_unstale":"2024-02-13T00:40:40.657778+0000","last_undegraded":"2024-02-13T00:40:40.657778+0000","last_fullsized":"2024-02-13T00:40:40.657778+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.603808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.603808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.663071+0000","last_change":"2024-02-13T00:39:16.262449+0000","last_active":"2024-02-13T00:40:40.663071+0000","last_peered":"2024-02-13T00:40:40.663071+0000","last_clean":"2024-02-13T00:40:40.663071+0000","last_became_active":"2024-02-13T00:37:41.212429+0000","last_became_peered":"2024-02-13T00:37:41.212429+0000","last_unstale":"2024-02-13T00:40:40.663071+0000","last_undegraded":"2024-02-13T00:40:40.663071+0000","last_fullsized":"2024-02-13T00:40:40.663071+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.262380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.262380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":83,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.259201+0000","last_change":"2024-02-13T00:39:24.376175+0000","last_active":"2024-02-13T00:40:40.259201+0000","last_peered":"2024-02-13T00:40:40.259201+0000","last_clean":"2024-02-13T00:40:40.259201+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:40:40.259201+0000","last_undegraded":"2024-02-13T00:40:40.259201+0000","last_fullsized":"2024-02-13T00:40:40.259201+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:39:24.376100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.376100+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251015+0000","last_change":"2024-02-13T00:39:17.713034+0000","last_active":"2024-02-13T00:40:40.251015+0000","last_peered":"2024-02-13T00:40:40.251015+0000","last_clean":"2024-02-13T00:40:40.251015+0000","last_became_active":"2024-02-13T00:37:47.552384+0000","last_became_peered":"2024-02-13T00:37:47.552384+0000","last_unstale":"2024-02-13T00:40:40.251015+0000","last_undegraded":"2024-02-13T00:40:40.251015+0000","last_fullsized":"2024-02-13T00:40:40.251015+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.712965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.712965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.658166+0000","last_change":"2024-02-13T00:39:02.545556+0000","last_active":"2024-02-13T00:40:40.658166+0000","last_peered":"2024-02-13T00:40:40.658166+0000","last_clean":"2024-02-13T00:40:40.658166+0000","last_became_active":"2024-02-13T00:37:41.219093+0000","last_became_peered":"2024-02-13T00:37:41.219093+0000","last_unstale":"2024-02-13T00:40:40.658166+0000","last_undegraded":"2024-02-13T00:40:40.658166+0000","last_fullsized":"2024-02-13T00:40:40.658166+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:02.545495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:02.545495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":116,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-13T00:40:40.251058+0000","last_change":"2024-02-13T00:38:59.548948+0000","last_active":"2024-02-13T00:40:40.251058+0000","last_peered":"2024-02-13T00:40:40.251058+0000","last_clean":"2024-02-13T00:40:40.251058+0000","last_became_active":"2024-02-13T00:37:43.322374+0000","last_became_peered":"2024-02-13T00:37:43.322374+0000","last_unstale":"2024-02-13T00:40:40.251058+0000","last_undegraded":"2024-02-13T00:40:40.251058+0000","last_fullsized":"2024-02-13T00:40:40.251058+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_clean_scrub_stamp":"2024-02-13T00:38:59.548872+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"64'8","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-13T00:40:43.192649+0000","last_change":"2024-02-13T00:39:15.080406+0000","last_active":"2024-02-13T00:40:43.192649+0000","last_peered":"2024-02-13T00:40:43.192649+0000","last_clean":"2024-02-13T00:40:43.192649+0000","last_became_active":"2024-02-13T00:37:45.247905+0000","last_became_peered":"2024-02-13T00:37:45.247905+0000","last_unstale":"2024-02-13T00:40:43.192649+0000","last_undegraded":"2024-02-13T00:40:43.192649+0000","last_fullsized":"2024-02-13T00:40:43.192649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:15.080315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.080315+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1073,"num_read_kb":798,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":669,"ondisk_log_size":669,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":177,"num_read_kb":172,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266727,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297800,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447352,"statfs":{"total":95995035648,"available":95690088448,"internally_reserved":0,"allocated":720896,"data_stored":339440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.502,"15min":0.502},"min":{"1min":0.26400000000000001,"5min":0.25,"15min":0.25},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.9570000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":1.8819999999999999}]},{"osd":1,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.87},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.57199999999999995,"15min":0.60599999999999998},"min":{"1min":0.379,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61699999999999999,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.432}]},{"osd":2,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.1619999999999999,"5min":3.1619999999999999,"15min":3.1619999999999999},"last":3.1619999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":2.0099999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.097,"5min":3.097,"15min":3.097},"last":3.097},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.57999999999999996,"15min":0.54800000000000004},"min":{"1min":0.40699999999999997,"5min":0.318,"15min":0.318},"max":{"1min":3.7690000000000001,"5min":3.7690000000000001,"15min":3.7690000000000001},"last":3.1120000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.46300000000000002,"15min":0.47099999999999997},"min":{"1min":0.29199999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66000000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.45400000000000001,"15min":0.46400000000000002},"min":{"1min":0.32800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.38700000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.47099999999999997,"15min":0.47499999999999998},"min":{"1min":0.315,"5min":0.255,"15min":0.255},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.48099999999999998,"15min":0.47599999999999998},"min":{"1min":0.34200000000000003,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68600000000000005,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.54500000000000004}]},{"osd":6,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.502,"15min":0.5},"min":{"1min":0.38400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70199999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.52600000000000002},"min":{"1min":0.35899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.49399999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528365,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297776,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447376,"statfs":{"total":95995035648,"available":95690113024,"internally_reserved":0,"allocated":696320,"data_stored":337917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52700000000000002,"15min":0.54200000000000004},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.63300000000000001,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.38},{"interface":"front","average":{"1min":0.496,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.34499999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.622,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":1.244}]},{"osd":1,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.502,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66800000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34399999999999997,"5min":0.315,"15min":0.315},"max":{"1min":0.624,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":1.1839999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.498,"15min":0.5},"min":{"1min":0.379,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.65400000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.34399999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":1.032}]},{"osd":3,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.39500000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71599999999999997,"5min":1.181,"15min":1.181},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.372,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.65700000000000003,"5min":1.488,"15min":1.488},"last":0.71299999999999997}]},{"osd":4,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.41299999999999998,"15min":0.40000000000000002},"min":{"1min":0.32600000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64800000000000002,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.442,"15min":0.42499999999999999},"min":{"1min":0.41099999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76600000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.435}]},{"osd":5,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.41299999999999998,"15min":0.39100000000000001},"min":{"1min":0.35999999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.70699999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.47099999999999997,"15min":0.45000000000000001},"min":{"1min":0.38200000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.76200000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.495}]},{"osd":7,"last update":"Tue Feb 13 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.41699999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70299999999999996,"5min":1.173,"15min":1.173},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.42299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.749,"5min":1.262,"15min":1.262},"last":0.55200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301924,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93443228,"statfs":{"total":95995035648,"available":95685865472,"internally_reserved":0,"allocated":749568,"data_stored":339167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34699999999999998,"15min":0.32200000000000001},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.43099999999999999,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.34599999999999997,"15min":0.314},"min":{"1min":0.215,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.42299999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.33400000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.372},"min":{"1min":0.255,"5min":0.215,"15min":0.215},"max":{"1min":0.57999999999999996,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.27300000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.69499999999999995,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.44}]},{"osd":3,"last update":"Tue Feb 13 00:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.41699999999999998,"15min":0.44},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.46899999999999997,"5min":0.748,"15min":0.748},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.42999999999999999,"15min":0.45800000000000002},"min":{"1min":0.30099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.504,"5min":0.745,"15min":0.745},"last":0.59799999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76500000000000001,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.41699999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81899999999999995,"5min":1.02,"15min":1.02},"last":0.52500000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56799999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78700000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70699999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.67700000000000005}]},{"osd":6,"last update":"Tue Feb 13 00:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.58399999999999996,"15min":0.59499999999999997},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.75,"5min":1.054,"15min":1.054},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.42099999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88300000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.45300000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.59499999999999997},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.78100000000000003,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.41799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82899999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.51500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297796,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447356,"statfs":{"total":95995035648,"available":95690092544,"internally_reserved":0,"allocated":716800,"data_stored":338221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:40:19 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.34599999999999997,"15min":0.32800000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.69699999999999995,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.38200000000000001,"15min":0.34599999999999997},"min":{"1min":0.23200000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.60499999999999998,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.40400000000000003}]},{"osd":2,"last update":"Tue Feb 13 00:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40400000000000003,"15min":0.39800000000000002},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.65200000000000002,"5min":1.206,"15min":1.206},"last":1.494},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.746,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.99199999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.38600000000000001,"15min":0.39600000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66600000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.437,"15min":0.46300000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.71499999999999997,"5min":0.874,"15min":0.874},"last":1.6339999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:40:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.38900000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53200000000000003,"15min":0.51700000000000002},"min":{"1min":0.41699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.874}]},{"osd":5,"last update":"Tue Feb 13 00:39:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55800000000000005,"15min":0.57399999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.97099999999999997}]},{"osd":6,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52300000000000002,"15min":0.50900000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.84299999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.78500000000000003}]},{"osd":7,"last update":"Tue Feb 13 00:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.55600000000000005,"15min":0.55100000000000005},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.439,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.90400000000000003,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.65800000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297788,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447364,"statfs":{"total":95995035648,"available":95690100736,"internally_reserved":0,"allocated":708608,"data_stored":338488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.36399999999999999,"15min":0.35699999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.49199999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36299999999999999,"15min":0.34599999999999997},"min":{"1min":0.22900000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.66800000000000004,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.76100000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.35699999999999998,"15min":0.34899999999999998},"min":{"1min":0.22900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.49099999999999999,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38400000000000001,"15min":0.374},"min":{"1min":0.251,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.51600000000000001,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.83999999999999997}]},{"osd":3,"last update":"Tue Feb 13 00:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.51500000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.43099999999999999,"15min":0.441},"min":{"1min":0.28000000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56399999999999995,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.85199999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56100000000000005,"15min":0.56699999999999995},"min":{"1min":0.379,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76600000000000001,"5min":0.873,"15min":0.873},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.74099999999999999,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.89300000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:39:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54600000000000004,"15min":0.56000000000000005},"min":{"1min":0.38900000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65000000000000002,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.57499999999999996,"15min":0.59299999999999997},"min":{"1min":0.376,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.64400000000000002,"5min":0.872,"15min":0.872},"last":0.86499999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:40:10 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69999999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.878},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.38700000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68200000000000005,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.78200000000000003}]},{"osd":7,"last update":"Tue Feb 13 00:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.58799999999999997},"min":{"1min":0.379,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68300000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.433,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89400000000000002,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.91900000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297812,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447340,"statfs":{"total":95995035648,"available":95690076160,"internally_reserved":0,"allocated":733184,"data_stored":338750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35099999999999998,"15min":0.373},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.191},"max":{"1min":0.48699999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38100000000000001,"15min":0.40400000000000003},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.57799999999999996}]},{"osd":1,"last update":"Tue Feb 13 00:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.38400000000000001,"15min":0.39800000000000002},"min":{"1min":0.24099999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.378,"15min":0.39500000000000002},"min":{"1min":0.21299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.47999999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.69299999999999995}]},{"osd":2,"last update":"Tue Feb 13 00:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40799999999999997,"15min":0.41899999999999998},"min":{"1min":0.29299999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38600000000000001,"15min":0.40100000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.623,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.51700000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.39900000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72499999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.629},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53300000000000003,"15min":0.53800000000000003},"min":{"1min":0.42399999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.60999999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51500000000000001,"15min":0.502},"min":{"1min":0.40200000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71799999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.621,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.77100000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53000000000000003,"15min":0.51400000000000001},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74399999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.45200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.69899999999999995,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.70199999999999996}]},{"osd":7,"last update":"Tue Feb 13 00:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.55300000000000005,"15min":0.54700000000000004},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.435,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81899999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.80800000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986231,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297808,"kb_used_data":712,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447344,"statfs":{"total":95995035648,"available":95690080256,"internally_reserved":0,"allocated":729088,"data_stored":339261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.40200000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65700000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.52500000000000002,"15min":0.54100000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70299999999999996,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.29299999999999998}]},{"osd":1,"last update":"Tue Feb 13 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72499999999999998,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.40200000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.70999999999999996,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.67800000000000005}]},{"osd":2,"last update":"Tue Feb 13 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.374,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51900000000000002,"15min":0.52600000000000002},"min":{"1min":0.36699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.66700000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.39900000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.66900000000000004,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52700000000000002,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69599999999999995,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.48899999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:39:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.37,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.58299999999999996,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.317},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.39300000000000002,"15min":0.36799999999999999},"min":{"1min":0.315,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.47599999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:40:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.42299999999999999,"15min":0.40200000000000002},"min":{"1min":0.33400000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.41199999999999998},"min":{"1min":0.36399999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.38100000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.48099999999999998,"15min":0.46600000000000003},"min":{"1min":0.42099999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.439},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.47099999999999997,"15min":0.45700000000000002},"min":{"1min":0.40500000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.45000000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724594,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297784,"kb_used_data":688,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447368,"statfs":{"total":95995035648,"available":95690104832,"internally_reserved":0,"allocated":704512,"data_stored":339041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49199999999999999,"15min":0.48599999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.84399999999999997,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.51200000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72199999999999998,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.53200000000000003}]},{"osd":1,"last update":"Tue Feb 13 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.502,"15min":0.497},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68700000000000006,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52400000000000002,"15min":0.502},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":4.0839999999999996,"15min":4.0839999999999996},"last":0.56799999999999995}]},{"osd":2,"last update":"Tue Feb 13 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51100000000000001,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.745,"5min":1.099,"15min":1.099},"last":0.62},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51000000000000001,"15min":0.505},"min":{"1min":0.34100000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.90000000000000002,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Feb 13 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85899999999999999,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51300000000000001,"15min":0.503},"min":{"1min":0.38200000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78100000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.64700000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.41799999999999998,"15min":0.36599999999999999},"min":{"1min":0.309,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66400000000000003,"5min":4.1210000000000004,"15min":4.1210000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.41499999999999998},"min":{"1min":0.379,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.74399999999999999,"5min":5.6950000000000003,"15min":5.6950000000000003},"last":0.71699999999999997}]},{"osd":6,"last update":"Tue Feb 13 00:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.45600000000000002,"15min":0.432},"min":{"1min":0.33600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64800000000000002,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.47499999999999998,"15min":0.45100000000000001},"min":{"1min":0.34799999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73499999999999999,"5min":1.377,"15min":1.377},"last":0.67200000000000004}]},{"osd":7,"last update":"Tue Feb 13 00:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.69599999999999995,"15min":0.78400000000000003},"min":{"1min":0.377,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69499999999999995,"5min":7.5519999999999996,"15min":7.5519999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55900000000000005,"15min":0.57499999999999996},"min":{"1min":0.435,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.70199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:40:47.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool set iscsi pgp_num 3 2024-02-13T00:40:47.519 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:47.909 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:47.953 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:47.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:48.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:48.089 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:48.111 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:48.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:48.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:48.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:48.245 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:47.031010+0000 osd.3 (osd.3) 65 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:40:48.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:47.034608+0000 osd.3 (osd.3) 66 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:40:48.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:47.031010+0000 osd.3 (osd.3) 65 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:40:48.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:47.034608+0000 osd.3 (osd.3) 66 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:40:48.255 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:48.286 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:48.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:48.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:47.031010+0000 osd.3 (osd.3) 65 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:40:48.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:47.034608+0000 osd.3 (osd.3) 66 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:40:48.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:48.428 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:48.448 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:48.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:48.552 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:48.582 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:48.602 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:48.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:48.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:40:48.742 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:48.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:40:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:40:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:40:49.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:48.185413+0000 osd.0 (osd.0) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:49.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:48.185477+0000 osd.0 (osd.0) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:49.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:48.198685+0000 mgr.y (mgr.14544) 127 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:49.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:48.185413+0000 osd.0 (osd.0) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:49.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:48.185477+0000 osd.0 (osd.0) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:49.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:48.198685+0000 mgr.y (mgr.14544) 127 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:49.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:48.185413+0000 osd.0 (osd.0) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:48.185477+0000 osd.0 (osd.0) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:48.198685+0000 mgr.y (mgr.14544) 127 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:40:50.041 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:48.517593+0000 osd.5 (osd.5) 55 : cluster [DBG] 5.f scrub starts 2024-02-13T00:40:50.041 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:48.518488+0000 osd.5 (osd.5) 56 : cluster [DBG] 5.f scrub ok 2024-02-13T00:40:50.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:48.806144+0000 osd.0 (osd.0) 47 : cluster [DBG] 2.f scrub starts 2024-02-13T00:40:50.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:48.809590+0000 osd.0 (osd.0) 48 : cluster [DBG] 2.f scrub ok 2024-02-13T00:40:50.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:48.927116+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-13T00:40:50.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:49.107742+0000 osd.3 (osd.3) 67 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:40:50.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:49.110719+0000 osd.3 (osd.3) 68 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:40:50.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:49.663262+0000 osd.1 (osd.1) 59 : cluster [DBG] 5.12 scrub starts 2024-02-13T00:40:50.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:49.666383+0000 osd.1 (osd.1) 60 : cluster [DBG] 5.12 scrub ok 2024-02-13T00:40:50.042 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:48.517593+0000 osd.5 (osd.5) 55 : cluster [DBG] 5.f scrub starts 2024-02-13T00:40:50.042 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:48.518488+0000 osd.5 (osd.5) 56 : cluster [DBG] 5.f scrub ok 2024-02-13T00:40:50.042 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:48.806144+0000 osd.0 (osd.0) 47 : cluster [DBG] 2.f scrub starts 2024-02-13T00:40:50.042 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:48.809590+0000 osd.0 (osd.0) 48 : cluster [DBG] 2.f scrub ok 2024-02-13T00:40:50.042 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:48.927116+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-13T00:40:50.043 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:49.107742+0000 osd.3 (osd.3) 67 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:40:50.043 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:49.110719+0000 osd.3 (osd.3) 68 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:40:50.043 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:49.663262+0000 osd.1 (osd.1) 59 : cluster [DBG] 5.12 scrub starts 2024-02-13T00:40:50.043 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:49.666383+0000 osd.1 (osd.1) 60 : cluster [DBG] 5.12 scrub ok 2024-02-13T00:40:50.097 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:50.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:48.517593+0000 osd.5 (osd.5) 55 : cluster [DBG] 5.f scrub starts 2024-02-13T00:40:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:48.518488+0000 osd.5 (osd.5) 56 : cluster [DBG] 5.f scrub ok 2024-02-13T00:40:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:48.806144+0000 osd.0 (osd.0) 47 : cluster [DBG] 2.f scrub starts 2024-02-13T00:40:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:48.809590+0000 osd.0 (osd.0) 48 : cluster [DBG] 2.f scrub ok 2024-02-13T00:40:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:48.927116+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-13T00:40:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:49.107742+0000 osd.3 (osd.3) 67 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:40:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:49.110719+0000 osd.3 (osd.3) 68 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:40:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:49.663262+0000 osd.1 (osd.1) 59 : cluster [DBG] 5.12 scrub starts 2024-02-13T00:40:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:49.666383+0000 osd.1 (osd.1) 60 : cluster [DBG] 5.12 scrub ok 2024-02-13T00:40:51.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:49.764369+0000 osd.0 (osd.0) 49 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:40:51.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:49.767844+0000 osd.0 (osd.0) 50 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:40:51.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:50.199187+0000 mgr.y (mgr.14544) 128 : cluster [DBG] pgmap v117: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:51.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:50.400329+0000 osd.1 (osd.1) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:51.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:50.400424+0000 osd.1 (osd.1) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:51.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:50.432983+0000 osd.2 (osd.2) 45 : cluster [DBG] 2.e scrub starts 2024-02-13T00:40:51.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:50.435953+0000 osd.2 (osd.2) 46 : cluster [DBG] 2.e scrub ok 2024-02-13T00:40:51.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:49.764369+0000 osd.0 (osd.0) 49 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:40:51.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:49.767844+0000 osd.0 (osd.0) 50 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:40:51.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:50.199187+0000 mgr.y (mgr.14544) 128 : cluster [DBG] pgmap v117: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:51.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:50.400329+0000 osd.1 (osd.1) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:51.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:50.400424+0000 osd.1 (osd.1) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:51.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:50.432983+0000 osd.2 (osd.2) 45 : cluster [DBG] 2.e scrub starts 2024-02-13T00:40:51.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:50.435953+0000 osd.2 (osd.2) 46 : cluster [DBG] 2.e scrub ok 2024-02-13T00:40:51.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.102+0000 7fc9ed062700 1 -- 172.21.15.162:0/3732534609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9e80f8d10 msgr2=0x7fc9e80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:51.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.102+0000 7fc9ed062700 1 --2- 172.21.15.162:0/3732534609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9e80f8d10 0x7fc9e80f90f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc9d8004870 tx=0x7fc9d8009cb0).stop 2024-02-13T00:40:51.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.102+0000 7fc9ed062700 1 -- 172.21.15.162:0/3732534609 shutdown_connections 2024-02-13T00:40:51.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.102+0000 7fc9ed062700 1 --2- 172.21.15.162:0/3732534609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e8102270 0x7fc9e8104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:51.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.102+0000 7fc9ed062700 1 --2- 172.21.15.162:0/3732534609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9e80f8d10 0x7fc9e80f90f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:51.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.102+0000 7fc9ed062700 1 --2- 172.21.15.162:0/3732534609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9e80f95e0 0x7fc9e8101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:51.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.102+0000 7fc9ed062700 1 -- 172.21.15.162:0/3732534609 >> 172.21.15.162:0/3732534609 conn(0x7fc9e806eb90 msgr2=0x7fc9e80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:51.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.102+0000 7fc9ed062700 1 -- 172.21.15.162:0/3732534609 shutdown_connections 2024-02-13T00:40:51.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.102+0000 7fc9ed062700 1 -- 172.21.15.162:0/3732534609 wait complete. 2024-02-13T00:40:51.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.102+0000 7fc9ed062700 1 librados: starting msgr at 2024-02-13T00:40:51.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.102+0000 7fc9ed062700 1 librados: starting objecter 2024-02-13T00:40:51.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.103+0000 7fc9ed062700 1 Processor -- start 2024-02-13T00:40:51.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.103+0000 7fc9ed062700 1 -- start start 2024-02-13T00:40:51.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.103+0000 7fc9ed062700 1 librados: setting wanted keys 2024-02-13T00:40:51.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.103+0000 7fc9ed062700 1 librados: calling monclient init 2024-02-13T00:40:51.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.103+0000 7fc9ed062700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9e80f8d10 0x7fc9e810e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:51.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.103+0000 7fc9ed062700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9e80f95e0 0x7fc9e810e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:51.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.103+0000 7fc9ed062700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e8102270 0x7fc9e8108330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:51.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.103+0000 7fc9ed062700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e8108820 con 0x7fc9e8102270 2024-02-13T00:40:51.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.103+0000 7fc9ed062700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e8108990 con 0x7fc9e80f95e0 2024-02-13T00:40:51.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.103+0000 7fc9ed062700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e8108b50 con 0x7fc9e80f8d10 2024-02-13T00:40:51.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9e759e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e8102270 0x7fc9e8108330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:51.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9e759e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e8102270 0x7fc9e8108330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:36914/0 (socket says 172.21.15.162:36914) 2024-02-13T00:40:51.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9e759e700 1 -- 172.21.15.162:0/3753527011 learned_addr learned my addr 172.21.15.162:0/3753527011 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:51.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9e6d9d700 1 --2- 172.21.15.162:0/3753527011 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9e80f8d10 0x7fc9e810e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:51.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9e659c700 1 --2- 172.21.15.162:0/3753527011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9e80f95e0 0x7fc9e810e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:51.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9e759e700 1 -- 172.21.15.162:0/3753527011 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9e80f8d10 msgr2=0x7fc9e810e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:51.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9e759e700 1 --2- 172.21.15.162:0/3753527011 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9e80f8d10 0x7fc9e810e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:51.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9e759e700 1 -- 172.21.15.162:0/3753527011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9e80f95e0 msgr2=0x7fc9e810e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:51.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9e759e700 1 --2- 172.21.15.162:0/3753527011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9e80f95e0 0x7fc9e810e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:51.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9e759e700 1 -- 172.21.15.162:0/3753527011 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e00097c0 con 0x7fc9e8102270 2024-02-13T00:40:51.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9e759e700 1 --2- 172.21.15.162:0/3753527011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e8102270 0x7fc9e8108330 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc9e0001d70 tx=0x7fc9e0004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:51.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9cffff700 10 client.?.objecter ms_handle_connect 0x7fc9e8102270 2024-02-13T00:40:51.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9cffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:51.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9cffff700 1 -- 172.21.15.162:0/3753527011 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e00099f0 con 0x7fc9e8102270 2024-02-13T00:40:51.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9cffff700 1 -- 172.21.15.162:0/3753527011 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9e00095a0 con 0x7fc9e8102270 2024-02-13T00:40:51.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9ed062700 1 -- 172.21.15.162:0/3753527011 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d8005040 con 0x7fc9e8102270 2024-02-13T00:40:51.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9cffff700 1 -- 172.21.15.162:0/3753527011 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0007dc0 con 0x7fc9e8102270 2024-02-13T00:40:51.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9ed062700 10 client.14838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:51.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9ed062700 1 -- 172.21.15.162:0/3753527011 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e8109190 con 0x7fc9e8102270 2024-02-13T00:40:51.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.104+0000 7fc9ed062700 1 librados: init done 2024-02-13T00:40:51.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.105+0000 7fc9ed062700 1 -- 172.21.15.162:0/3753527011 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e80475e0 con 0x7fc9e8102270 2024-02-13T00:40:51.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.106+0000 7fc9cffff700 1 -- 172.21.15.162:0/3753527011 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc9e00090d0 con 0x7fc9e8102270 2024-02-13T00:40:51.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.107+0000 7fc9cffff700 1 --2- 172.21.15.162:0/3753527011 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc9d0062c90 0x7fc9d0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:51.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.107+0000 7fc9cffff700 1 -- 172.21.15.162:0/3753527011 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(100..100 src has 41..100) v4 ==== 6204+0+0 (secure 0 0 0) 0x7fc9e007d2d0 con 0x7fc9e8102270 2024-02-13T00:40:51.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.107+0000 7fc9cffff700 10 client.14838.objecter ms_dispatch 0x7fc9e8001550 osd_map(100..100 src has 41..100) v4 2024-02-13T00:40:51.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.107+0000 7fc9cffff700 3 client.14838.objecter handle_osd_map got epochs [100,100] > 0 2024-02-13T00:40:51.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.107+0000 7fc9cffff700 3 client.14838.objecter handle_osd_map decoding full epoch 100 2024-02-13T00:40:51.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.107+0000 7fc9e6d9d700 1 --2- 172.21.15.162:0/3753527011 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc9d0062c90 0x7fc9d0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:51.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.107+0000 7fc9cffff700 20 client.14838.objecter dump_active .. 0 homeless 2024-02-13T00:40:51.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.107+0000 7fc9cffff700 10 client.14838.objecter ms_handle_connect 0x7fc9d0062c90 2024-02-13T00:40:51.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.107+0000 7fc9e6d9d700 1 --2- 172.21.15.162:0/3753527011 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc9d0062c90 0x7fc9d0065150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fc9d80043d0 tx=0x7fc9d800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:51.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.111+0000 7fc9cffff700 1 -- 172.21.15.162:0/3753527011 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc9e0050f20 con 0x7fc9e8102270 2024-02-13T00:40:51.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:49.764369+0000 osd.0 (osd.0) 49 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:40:51.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:49.767844+0000 osd.0 (osd.0) 50 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:40:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:50.199187+0000 mgr.y (mgr.14544) 128 : cluster [DBG] pgmap v117: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:40:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:50.400329+0000 osd.1 (osd.1) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:40:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:50.400424+0000 osd.1 (osd.1) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:40:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:50.432983+0000 osd.2 (osd.2) 45 : cluster [DBG] 2.e scrub starts 2024-02-13T00:40:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:50.435953+0000 osd.2 (osd.2) 46 : cluster [DBG] 2.e scrub ok 2024-02-13T00:40:51.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.298+0000 7fc9ed062700 1 -- 172.21.15.162:0/3753527011 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc9e8107170 con 0x7fc9e8102270 2024-02-13T00:40:51.520 INFO:tasks.workunit.client.0.smithi162.stderr:Updating files: 92% (11259/12167) Updating files: 93% (11316/12167) Updating files: 94% (11437/12167) Updating files: 95% (11559/12167) Updating files: 96% (11681/12167) Updating files: 97% (11802/12167) Updating files: 98% (11924/12167) Updating files: 99% (12046/12167) Updating files: 100% (12167/12167) Updating files: 100% (12167/12167), done. 2024-02-13T00:40:51.595 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:40:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:51.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:40:51.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:51.730 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:51.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.938+0000 7fc9cffff700 1 -- 172.21.15.162:0/3753527011 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v101) v1 ==== 101+0+0 (secure 0 0 0) 0x7fc9e0050b10 con 0x7fc9e8102270 2024-02-13T00:40:51.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 10 librados: watch_flush enter 2024-02-13T00:40:51.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 10 librados: watch_flush exit 2024-02-13T00:40:51.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 20 client.14838.objecter shutdown clearing up homeless session... 2024-02-13T00:40:51.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 10 client.14838.objecter successfully canceled tick 2024-02-13T00:40:51.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 1 -- 172.21.15.162:0/3753527011 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc9d0062c90 msgr2=0x7fc9d0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:51.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 1 --2- 172.21.15.162:0/3753527011 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc9d0062c90 0x7fc9d0065150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fc9d80043d0 tx=0x7fc9d800b040).stop 2024-02-13T00:40:51.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 1 -- 172.21.15.162:0/3753527011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e8102270 msgr2=0x7fc9e8108330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:51.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 1 --2- 172.21.15.162:0/3753527011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e8102270 0x7fc9e8108330 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc9e0001d70 tx=0x7fc9e0004c50).stop 2024-02-13T00:40:51.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 1 -- 172.21.15.162:0/3753527011 shutdown_connections 2024-02-13T00:40:51.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 1 --2- 172.21.15.162:0/3753527011 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc9d0062c90 0x7fc9d0065150 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:51.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 1 --2- 172.21.15.162:0/3753527011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e8102270 0x7fc9e8108330 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:51.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 1 --2- 172.21.15.162:0/3753527011 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9e80f8d10 0x7fc9e810e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:51.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 1 --2- 172.21.15.162:0/3753527011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9e80f95e0 0x7fc9e810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:51.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 1 -- 172.21.15.162:0/3753527011 >> 172.21.15.162:0/3753527011 conn(0x7fc9e806eb90 msgr2=0x7fc9e8102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:51.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 1 -- 172.21.15.162:0/3753527011 shutdown_connections 2024-02-13T00:40:51.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.940+0000 7fc9ed062700 1 -- 172.21.15.162:0/3753527011 wait complete. 2024-02-13T00:40:51.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:51.941+0000 7fc9ed062700 1 librados: shutdown 2024-02-13T00:40:51.947 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:40:52.278 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:50.933203+0000 mon.a (mon.0) 747 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-13T00:40:52.278 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:51.051228+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:40:52.278 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:51.054718+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:40:52.278 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:51.300836+0000 mon.a (mon.0) 748 : audit [INF] from='client.? 172.21.15.162:0/3753527011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:40:52.278 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:51.681469+0000 mon.c (mon.2) 85 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:40:52.278 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:50.933203+0000 mon.a (mon.0) 747 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-13T00:40:52.278 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:51.051228+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:40:52.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:51.054718+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:40:52.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:51.300836+0000 mon.a (mon.0) 748 : audit [INF] from='client.? 172.21.15.162:0/3753527011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:40:52.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:51.681469+0000 mon.c (mon.2) 85 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:40:52.283 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:52.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:52.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:52.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:50.933203+0000 mon.a (mon.0) 747 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-13T00:40:52.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:51.051228+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:40:52.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:51.054718+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:40:52.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:51.300836+0000 mon.a (mon.0) 748 : audit [INF] from='client.? 172.21.15.162:0/3753527011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:40:52.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:51.681469+0000 mon.c (mon.2) 85 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:40:52.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:52.414 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:52.528 INFO:tasks.workunit.client.0.smithi162.stderr:Note: switching to '84e714279aa684636ac645daf0a1a85c2094cfb3'. 2024-02-13T00:40:52.528 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-02-13T00:40:52.528 INFO:tasks.workunit.client.0.smithi162.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-13T00:40:52.529 INFO:tasks.workunit.client.0.smithi162.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-13T00:40:52.529 INFO:tasks.workunit.client.0.smithi162.stderr:state without impacting any branches by switching back to a branch. 2024-02-13T00:40:52.529 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-02-13T00:40:52.529 INFO:tasks.workunit.client.0.smithi162.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-13T00:40:52.529 INFO:tasks.workunit.client.0.smithi162.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-13T00:40:52.529 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-02-13T00:40:52.529 INFO:tasks.workunit.client.0.smithi162.stderr: git switch -c 2024-02-13T00:40:52.530 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-02-13T00:40:52.530 INFO:tasks.workunit.client.0.smithi162.stderr:Or undo this operation with: 2024-02-13T00:40:52.530 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-02-13T00:40:52.530 INFO:tasks.workunit.client.0.smithi162.stderr: git switch - 2024-02-13T00:40:52.530 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-02-13T00:40:52.530 INFO:tasks.workunit.client.0.smithi162.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-13T00:40:52.530 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-02-13T00:40:52.530 INFO:tasks.workunit.client.0.smithi162.stderr:HEAD is now at 84e714279aa qa/suites/orch: whitelist warnings that are expected in test environments 2024-02-13T00:40:52.534 DEBUG:teuthology.orchestra.run.smithi162:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-13T00:40:52.619 INFO:tasks.workunit.client.0.smithi162.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-13T00:40:52.621 INFO:tasks.workunit.client.0.smithi162.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-13T00:40:52.621 INFO:tasks.workunit.client.0.smithi162.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-13T00:40:53.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:51.196902+0000 osd.7 (osd.7) 56 : cluster [DBG] 6.5 scrub starts 2024-02-13T00:40:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:51.198050+0000 osd.7 (osd.7) 57 : cluster [DBG] 6.5 scrub ok 2024-02-13T00:40:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:51.931261+0000 mon.a (mon.0) 749 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:40:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:51.931286+0000 mon.a (mon.0) 750 : cluster [INF] Cluster is now healthy 2024-02-13T00:40:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:51.940009+0000 mon.a (mon.0) 751 : audit [INF] from='client.? 172.21.15.162:0/3753527011' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:40:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:51.940064+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-13T00:40:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:52.063734+0000 osd.3 (osd.3) 71 : cluster [DBG] 5.1c scrub starts 2024-02-13T00:40:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:52.067092+0000 osd.3 (osd.3) 72 : cluster [DBG] 5.1c scrub ok 2024-02-13T00:40:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:52.199539+0000 mgr.y (mgr.14544) 129 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:40:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:52.426293+0000 mon.c (mon.2) 86 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:40:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:52.427140+0000 mon.c (mon.2) 87 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:40:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:52.434337+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:40:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:51.196902+0000 osd.7 (osd.7) 56 : cluster [DBG] 6.5 scrub starts 2024-02-13T00:40:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:51.198050+0000 osd.7 (osd.7) 57 : cluster [DBG] 6.5 scrub ok 2024-02-13T00:40:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:51.931261+0000 mon.a (mon.0) 749 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:40:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:51.931286+0000 mon.a (mon.0) 750 : cluster [INF] Cluster is now healthy 2024-02-13T00:40:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:51.940009+0000 mon.a (mon.0) 751 : audit [INF] from='client.? 172.21.15.162:0/3753527011' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:40:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:51.940064+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-13T00:40:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:52.063734+0000 osd.3 (osd.3) 71 : cluster [DBG] 5.1c scrub starts 2024-02-13T00:40:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:52.067092+0000 osd.3 (osd.3) 72 : cluster [DBG] 5.1c scrub ok 2024-02-13T00:40:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:52.199539+0000 mgr.y (mgr.14544) 129 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:40:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:52.426293+0000 mon.c (mon.2) 86 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:40:53.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:52.427140+0000 mon.c (mon.2) 87 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:40:53.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:52.434337+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:40:53.239 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:51.196902+0000 osd.7 (osd.7) 56 : cluster [DBG] 6.5 scrub starts 2024-02-13T00:40:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:51.198050+0000 osd.7 (osd.7) 57 : cluster [DBG] 6.5 scrub ok 2024-02-13T00:40:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:51.931261+0000 mon.a (mon.0) 749 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:40:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:51.931286+0000 mon.a (mon.0) 750 : cluster [INF] Cluster is now healthy 2024-02-13T00:40:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:51.940009+0000 mon.a (mon.0) 751 : audit [INF] from='client.? 172.21.15.162:0/3753527011' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:40:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:51.940064+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-13T00:40:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:52.063734+0000 osd.3 (osd.3) 71 : cluster [DBG] 5.1c scrub starts 2024-02-13T00:40:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:52.067092+0000 osd.3 (osd.3) 72 : cluster [DBG] 5.1c scrub ok 2024-02-13T00:40:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:52.199539+0000 mgr.y (mgr.14544) 129 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:40:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:52.426293+0000 mon.c (mon.2) 86 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:40:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:52.427140+0000 mon.c (mon.2) 87 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:40:53.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:52.434337+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:40:53.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.451+0000 7fb1e5189700 1 -- 172.21.15.162:0/4281291206 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1e00f8500 msgr2=0x7fb1e0104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:53.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.451+0000 7fb1e5189700 1 --2- 172.21.15.162:0/4281291206 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1e00f8500 0x7fb1e0104bd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fb1d8009be0 tx=0x7fb1d8004d10).stop 2024-02-13T00:40:53.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.451+0000 7fb1e5189700 1 -- 172.21.15.162:0/4281291206 shutdown_connections 2024-02-13T00:40:53.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.451+0000 7fb1e5189700 1 --2- 172.21.15.162:0/4281291206 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1e00f8500 0x7fb1e0104bd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.451+0000 7fb1e5189700 1 --2- 172.21.15.162:0/4281291206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1e00f9b70 0x7fb1e00f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.451+0000 7fb1e5189700 1 --2- 172.21.15.162:0/4281291206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1e00fa440 0x7fb1e00f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.451+0000 7fb1e5189700 1 -- 172.21.15.162:0/4281291206 >> 172.21.15.162:0/4281291206 conn(0x7fb1e00f38c0 msgr2=0x7fb1e00f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.451+0000 7fb1e5189700 1 -- 172.21.15.162:0/4281291206 shutdown_connections 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.451+0000 7fb1e5189700 1 -- 172.21.15.162:0/4281291206 wait complete. 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1e5189700 1 librados: starting msgr at 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1e5189700 1 librados: starting objecter 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1e5189700 1 Processor -- start 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1e5189700 1 -- start start 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1e5189700 1 librados: setting wanted keys 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1e5189700 1 librados: calling monclient init 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1e5189700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1e00f8500 0x7fb1e006b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1e5189700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1e00f9b70 0x7fb1e006b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1e5189700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1e00fa440 0x7fb1e006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1e5189700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e0116590 con 0x7fb1e00f9b70 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1e5189700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e0116700 con 0x7fb1e00fa440 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1e5189700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e01168c0 con 0x7fb1e00f8500 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1de59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1e00f9b70 0x7fb1e006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1ded9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1e00f8500 0x7fb1e006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1de59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1e00f9b70 0x7fb1e006b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:36932/0 (socket says 172.21.15.162:36932) 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1ded9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1e00f8500 0x7fb1e006b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:46946/0 (socket says 172.21.15.162:46946) 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.452+0000 7fb1de59c700 1 -- 172.21.15.162:0/3980525024 learned_addr learned my addr 172.21.15.162:0/3980525024 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1df59e700 1 --2- 172.21.15.162:0/3980525024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1e00fa440 0x7fb1e006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1ded9d700 1 -- 172.21.15.162:0/3980525024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1e00fa440 msgr2=0x7fb1e006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1ded9d700 1 --2- 172.21.15.162:0/3980525024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1e00fa440 0x7fb1e006bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1ded9d700 1 -- 172.21.15.162:0/3980525024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1e00f9b70 msgr2=0x7fb1e006b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:53.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1ded9d700 1 --2- 172.21.15.162:0/3980525024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1e00f9b70 0x7fb1e006b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1ded9d700 1 -- 172.21.15.162:0/3980525024 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d4005040 con 0x7fb1e00f8500 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1de59c700 1 --2- 172.21.15.162:0/3980525024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1e00f9b70 0x7fb1e006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1ded9d700 1 --2- 172.21.15.162:0/3980525024 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1e00f8500 0x7fb1e006b170 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fb1d4004920 tx=0x7fb1d4009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1c7fff700 10 client.?.objecter ms_handle_connect 0x7fb1e00f8500 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1c7fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1c7fff700 1 -- 172.21.15.162:0/3980525024 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d400e030 con 0x7fb1e00f8500 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1e5189700 1 -- 172.21.15.162:0/3980525024 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d80097c0 con 0x7fb1e00f8500 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1c7fff700 1 -- 172.21.15.162:0/3980525024 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb1d400f040 con 0x7fb1e00f8500 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1c7fff700 1 -- 172.21.15.162:0/3980525024 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4014850 con 0x7fb1e00f8500 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1e5189700 10 client.24737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1e5189700 1 -- 172.21.15.162:0/3980525024 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e0116ed0 con 0x7fb1e00f8500 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.453+0000 7fb1e5189700 1 librados: init done 2024-02-13T00:40:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.454+0000 7fb1e5189700 1 -- 172.21.15.162:0/3980525024 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e00475e0 con 0x7fb1e00f8500 2024-02-13T00:40:53.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.454+0000 7fb1c7fff700 1 -- 172.21.15.162:0/3980525024 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fb1d4010500 con 0x7fb1e00f8500 2024-02-13T00:40:53.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.455+0000 7fb1c7fff700 1 --2- 172.21.15.162:0/3980525024 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb1c8062c90 0x7fb1c8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:53.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.455+0000 7fb1c7fff700 1 -- 172.21.15.162:0/3980525024 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(101..101 src has 41..101) v4 ==== 6204+0+0 (secure 0 0 0) 0x7fb1d407cb40 con 0x7fb1e00f8500 2024-02-13T00:40:53.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.455+0000 7fb1c7fff700 10 client.24737.objecter ms_dispatch 0x7fb1e0001550 osd_map(101..101 src has 41..101) v4 2024-02-13T00:40:53.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.455+0000 7fb1c7fff700 3 client.24737.objecter handle_osd_map got epochs [101,101] > 0 2024-02-13T00:40:53.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.455+0000 7fb1c7fff700 3 client.24737.objecter handle_osd_map decoding full epoch 101 2024-02-13T00:40:53.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.455+0000 7fb1de59c700 1 --2- 172.21.15.162:0/3980525024 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb1c8062c90 0x7fb1c8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:53.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.455+0000 7fb1c7fff700 20 client.24737.objecter dump_active .. 0 homeless 2024-02-13T00:40:53.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.455+0000 7fb1c7fff700 10 client.24737.objecter ms_handle_connect 0x7fb1c8062c90 2024-02-13T00:40:53.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.455+0000 7fb1de59c700 1 --2- 172.21.15.162:0/3980525024 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb1c8062c90 0x7fb1c8065150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fb1d0004fd0 tx=0x7fb1d000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:53.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.459+0000 7fb1c7fff700 1 -- 172.21.15.162:0/3980525024 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb1d4050790 con 0x7fb1e00f8500 2024-02-13T00:40:53.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.634+0000 7fb1e5189700 1 -- 172.21.15.162:0/3980525024 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"} v 0) v1 -- 0x7fb1e006c720 con 0x7fb1e00f8500 2024-02-13T00:40:53.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.947+0000 7fb1c7fff700 1 -- 172.21.15.162:0/3980525024 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]=0 set pool 3 pgp_num to 3 v102) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb1d4050380 con 0x7fb1e00f8500 2024-02-13T00:40:53.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.949+0000 7fb1e5189700 10 librados: watch_flush enter 2024-02-13T00:40:53.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.949+0000 7fb1e5189700 10 librados: watch_flush exit 2024-02-13T00:40:53.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.949+0000 7fb1e5189700 20 client.24737.objecter shutdown clearing up homeless session... 2024-02-13T00:40:53.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.949+0000 7fb1e5189700 10 client.24737.objecter successfully canceled tick 2024-02-13T00:40:53.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.949+0000 7fb1e5189700 1 -- 172.21.15.162:0/3980525024 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb1c8062c90 msgr2=0x7fb1c8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:53.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.949+0000 7fb1e5189700 1 --2- 172.21.15.162:0/3980525024 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb1c8062c90 0x7fb1c8065150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fb1d0004fd0 tx=0x7fb1d000b040).stop 2024-02-13T00:40:53.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.949+0000 7fb1e5189700 1 -- 172.21.15.162:0/3980525024 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1e00f8500 msgr2=0x7fb1e006b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:53.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.949+0000 7fb1e5189700 1 --2- 172.21.15.162:0/3980525024 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1e00f8500 0x7fb1e006b170 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fb1d4004920 tx=0x7fb1d4009cb0).stop 2024-02-13T00:40:53.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.949+0000 7fb1e5189700 1 -- 172.21.15.162:0/3980525024 shutdown_connections 2024-02-13T00:40:53.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.949+0000 7fb1e5189700 1 --2- 172.21.15.162:0/3980525024 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb1c8062c90 0x7fb1c8065150 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:53.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.949+0000 7fb1e5189700 1 --2- 172.21.15.162:0/3980525024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1e00fa440 0x7fb1e006bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:53.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.950+0000 7fb1e5189700 1 --2- 172.21.15.162:0/3980525024 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1e00f8500 0x7fb1e006b170 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:53.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.950+0000 7fb1e5189700 1 --2- 172.21.15.162:0/3980525024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1e00f9b70 0x7fb1e006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:53.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.950+0000 7fb1e5189700 1 -- 172.21.15.162:0/3980525024 >> 172.21.15.162:0/3980525024 conn(0x7fb1e00f38c0 msgr2=0x7fb1e00f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:53.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.950+0000 7fb1e5189700 1 -- 172.21.15.162:0/3980525024 shutdown_connections 2024-02-13T00:40:53.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.950+0000 7fb1e5189700 1 -- 172.21.15.162:0/3980525024 wait complete. 2024-02-13T00:40:53.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:53.950+0000 7fb1e5189700 1 librados: shutdown 2024-02-13T00:40:53.956 INFO:teuthology.orchestra.run.smithi162.stderr:set pool 3 pgp_num to 3 2024-02-13T00:40:54.106 INFO:tasks.workunit.client.0.smithi162.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-13T00:40:54.243 INFO:tasks.workunit.client.0.smithi162.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-13T00:40:54.285 INFO:tasks.workunit.client.0.smithi162.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-13T00:40:54.287 INFO:tasks.workunit.client.0.smithi162.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-13T00:40:54.287 INFO:tasks.workunit.client.0.smithi162.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-13T00:40:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:52.500897+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:40:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:52.503800+0000 osd.5 (osd.5) 58 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:40:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:52.798401+0000 osd.0 (osd.0) 51 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:40:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:52.801451+0000 osd.0 (osd.0) 52 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:40:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:53.101039+0000 osd.3 (osd.3) 73 : cluster [DBG] 4.c scrub starts 2024-02-13T00:40:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:53.104719+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.c scrub ok 2024-02-13T00:40:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:53.361276+0000 osd.2 (osd.2) 47 : cluster [DBG] 5.a scrub starts 2024-02-13T00:40:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:53.362222+0000 osd.2 (osd.2) 48 : cluster [DBG] 5.a scrub ok 2024-02-13T00:40:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:53.636933+0000 mon.c (mon.2) 88 : audit [INF] from='client.? 172.21.15.162:0/3980525024' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-13T00:40:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:53.637244+0000 mon.a (mon.0) 754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-13T00:40:54.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:52.500897+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:40:54.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:52.503800+0000 osd.5 (osd.5) 58 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:40:54.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:52.798401+0000 osd.0 (osd.0) 51 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:40:54.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:52.801451+0000 osd.0 (osd.0) 52 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:40:54.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:53.101039+0000 osd.3 (osd.3) 73 : cluster [DBG] 4.c scrub starts 2024-02-13T00:40:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:53.104719+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.c scrub ok 2024-02-13T00:40:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:53.361276+0000 osd.2 (osd.2) 47 : cluster [DBG] 5.a scrub starts 2024-02-13T00:40:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:53.362222+0000 osd.2 (osd.2) 48 : cluster [DBG] 5.a scrub ok 2024-02-13T00:40:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:53.636933+0000 mon.c (mon.2) 88 : audit [INF] from='client.? 172.21.15.162:0/3980525024' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-13T00:40:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:53.637244+0000 mon.a (mon.0) 754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-13T00:40:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:52.500897+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:40:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:52.503800+0000 osd.5 (osd.5) 58 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:40:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:52.798401+0000 osd.0 (osd.0) 51 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:40:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:52.801451+0000 osd.0 (osd.0) 52 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:40:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:53.101039+0000 osd.3 (osd.3) 73 : cluster [DBG] 4.c scrub starts 2024-02-13T00:40:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:53.104719+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.c scrub ok 2024-02-13T00:40:54.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:53.361276+0000 osd.2 (osd.2) 47 : cluster [DBG] 5.a scrub starts 2024-02-13T00:40:54.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:53.362222+0000 osd.2 (osd.2) 48 : cluster [DBG] 5.a scrub ok 2024-02-13T00:40:54.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:53.636933+0000 mon.c (mon.2) 88 : audit [INF] from='client.? 172.21.15.162:0/3980525024' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-13T00:40:54.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:53.637244+0000 mon.a (mon.0) 754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-13T00:40:54.335 INFO:tasks.workunit.client.0.smithi162.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-13T00:40:54.338 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-13T00:40:54.339 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-13T00:40:54.357 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-02-13T00:40:54.358 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-02-13T00:40:54.358 DEBUG:teuthology.orchestra.run.smithi162:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=84e714279aa684636ac645daf0a1a85c2094cfb3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-02-13T00:40:54.457 INFO:tasks.workunit.client.0.smithi162.stderr:+ parallel=1 2024-02-13T00:40:54.457 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' '' = --serial ']' 2024-02-13T00:40:54.457 INFO:tasks.workunit.client.0.smithi162.stderr:+ color= 2024-02-13T00:40:54.457 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -t 1 ']' 2024-02-13T00:40:54.457 INFO:tasks.workunit.client.0.smithi162.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-02-13T00:40:54.457 INFO:tasks.workunit.client.0.smithi162.stderr:+ declare -A pids 2024-02-13T00:40:54.457 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.458 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.458 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_aio 2024-02-13T00:40:54.458 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_aio' 2024-02-13T00:40:54.458 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_aio 2024-02-13T00:40:54.458 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.458 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_aio 2024-02-13T00:40:54.459 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=83972 2024-02-13T00:40:54.459 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_aio on pid 83972' 2024-02-13T00:40:54.459 INFO:tasks.workunit.client.0.smithi162.stdout:test api_aio on pid 83972 2024-02-13T00:40:54.460 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=83972 2024-02-13T00:40:54.460 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.460 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.461 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-02-13T00:40:54.461 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_aio_pp 2024-02-13T00:40:54.461 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_aio_pp' 2024-02-13T00:40:54.461 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_aio_pp 2024-02-13T00:40:54.461 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.461 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_aio 2024-02-13T00:40:54.461 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_aio.log 2024-02-13T00:40:54.461 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_aio: /' 2024-02-13T00:40:54.461 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_aio_pp 2024-02-13T00:40:54.461 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=83980 2024-02-13T00:40:54.461 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_aio_pp on pid 83980' 2024-02-13T00:40:54.462 INFO:tasks.workunit.client.0.smithi162.stdout:test api_aio_pp on pid 83980 2024-02-13T00:40:54.463 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=83980 2024-02-13T00:40:54.463 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.463 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.463 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-02-13T00:40:54.463 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_io 2024-02-13T00:40:54.463 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_io' 2024-02-13T00:40:54.463 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_io 2024-02-13T00:40:54.463 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.463 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_aio_pp 2024-02-13T00:40:54.463 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-02-13T00:40:54.464 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_aio_pp: /' 2024-02-13T00:40:54.464 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_io 2024-02-13T00:40:54.464 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=83988 2024-02-13T00:40:54.464 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_io on pid 83988' 2024-02-13T00:40:54.464 INFO:tasks.workunit.client.0.smithi162.stdout:test api_io on pid 83988 2024-02-13T00:40:54.465 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=83988 2024-02-13T00:40:54.465 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-02-13T00:40:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_io_pp 2024-02-13T00:40:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_io_pp' 2024-02-13T00:40:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_io_pp 2024-02-13T00:40:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_io 2024-02-13T00:40:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_io.log 2024-02-13T00:40:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_io: /' 2024-02-13T00:40:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_io_pp 2024-02-13T00:40:54.466 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=83996 2024-02-13T00:40:54.467 INFO:tasks.workunit.client.0.smithi162.stdout:test api_io_pp on pid 83996 2024-02-13T00:40:54.468 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_io_pp on pid 83996' 2024-02-13T00:40:54.468 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=83996 2024-02-13T00:40:54.468 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.468 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.468 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-02-13T00:40:54.468 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_asio 2024-02-13T00:40:54.468 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_asio' 2024-02-13T00:40:54.468 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_asio 2024-02-13T00:40:54.469 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.469 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_io_pp 2024-02-13T00:40:54.469 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-02-13T00:40:54.469 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_io_pp: /' 2024-02-13T00:40:54.469 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_asio 2024-02-13T00:40:54.469 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84004 2024-02-13T00:40:54.469 INFO:tasks.workunit.client.0.smithi162.stdout:test api_asio on pid 84004 2024-02-13T00:40:54.470 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_asio on pid 84004' 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84004 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_list 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_list' 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_list 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_asio 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_asio.log 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_asio: /' 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_list 2024-02-13T00:40:54.471 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84012 2024-02-13T00:40:54.472 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_list on pid 84012' 2024-02-13T00:40:54.472 INFO:tasks.workunit.client.0.smithi162.stdout:test api_list on pid 84012 2024-02-13T00:40:54.473 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84012 2024-02-13T00:40:54.473 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.473 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.473 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-02-13T00:40:54.474 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_lock 2024-02-13T00:40:54.474 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_lock' 2024-02-13T00:40:54.474 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_lock 2024-02-13T00:40:54.474 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.474 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_list 2024-02-13T00:40:54.474 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_list.log 2024-02-13T00:40:54.474 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_list: /' 2024-02-13T00:40:54.474 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_lock 2024-02-13T00:40:54.474 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84020 2024-02-13T00:40:54.474 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_lock on pid 84020' 2024-02-13T00:40:54.474 INFO:tasks.workunit.client.0.smithi162.stdout:test api_lock on pid 84020 2024-02-13T00:40:54.476 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84020 2024-02-13T00:40:54.476 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.476 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.476 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-02-13T00:40:54.476 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_lock_pp 2024-02-13T00:40:54.476 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_lock_pp' 2024-02-13T00:40:54.476 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_lock_pp 2024-02-13T00:40:54.477 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.477 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_lock 2024-02-13T00:40:54.477 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_lock.log 2024-02-13T00:40:54.477 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_lock: /' 2024-02-13T00:40:54.477 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_lock_pp 2024-02-13T00:40:54.477 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84028 2024-02-13T00:40:54.477 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_lock_pp on pid 84028' 2024-02-13T00:40:54.477 INFO:tasks.workunit.client.0.smithi162.stdout:test api_lock_pp on pid 84028 2024-02-13T00:40:54.479 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84028 2024-02-13T00:40:54.479 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.479 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.479 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-02-13T00:40:54.479 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_misc 2024-02-13T00:40:54.479 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_misc' 2024-02-13T00:40:54.479 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_misc 2024-02-13T00:40:54.479 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.479 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_lock_pp 2024-02-13T00:40:54.479 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-02-13T00:40:54.480 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_lock_pp: /' 2024-02-13T00:40:54.480 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_misc 2024-02-13T00:40:54.480 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84036 2024-02-13T00:40:54.480 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_misc on pid 84036' 2024-02-13T00:40:54.480 INFO:tasks.workunit.client.0.smithi162.stdout:test api_misc on pid 84036 2024-02-13T00:40:54.481 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84036 2024-02-13T00:40:54.481 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.482 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.482 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-02-13T00:40:54.482 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_misc_pp 2024-02-13T00:40:54.482 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_misc_pp' 2024-02-13T00:40:54.482 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_misc_pp 2024-02-13T00:40:54.482 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.482 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_misc 2024-02-13T00:40:54.482 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_misc.log 2024-02-13T00:40:54.482 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_misc: /' 2024-02-13T00:40:54.482 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_misc_pp 2024-02-13T00:40:54.482 INFO:tasks.workunit.client.0.smithi162.stdout:test api_misc_pp on pid 84044 2024-02-13T00:40:54.483 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84044 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_misc_pp on pid 84044' 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84044 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_tier_pp 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_tier_pp' 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_tier_pp 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_misc_pp 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_misc_pp: /' 2024-02-13T00:40:54.484 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_tier_pp 2024-02-13T00:40:54.485 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84052 2024-02-13T00:40:54.485 INFO:tasks.workunit.client.0.smithi162.stdout:test api_tier_pp on pid 84052 2024-02-13T00:40:54.486 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_tier_pp on pid 84052' 2024-02-13T00:40:54.486 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84052 2024-02-13T00:40:54.486 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.486 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.486 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-02-13T00:40:54.487 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_pool 2024-02-13T00:40:54.487 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_pool' 2024-02-13T00:40:54.487 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_pool 2024-02-13T00:40:54.487 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.487 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_tier_pp 2024-02-13T00:40:54.487 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-02-13T00:40:54.487 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_tier_pp: /' 2024-02-13T00:40:54.487 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_pool 2024-02-13T00:40:54.487 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84060 2024-02-13T00:40:54.487 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_pool on pid 84060' 2024-02-13T00:40:54.487 INFO:tasks.workunit.client.0.smithi162.stdout:test api_pool on pid 84060 2024-02-13T00:40:54.489 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84060 2024-02-13T00:40:54.489 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.489 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.489 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-02-13T00:40:54.489 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_snapshots 2024-02-13T00:40:54.489 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_snapshots' 2024-02-13T00:40:54.489 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_snapshots 2024-02-13T00:40:54.489 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.489 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_pool 2024-02-13T00:40:54.489 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_pool.log 2024-02-13T00:40:54.490 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_pool: /' 2024-02-13T00:40:54.490 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_snapshots 2024-02-13T00:40:54.490 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84068 2024-02-13T00:40:54.490 INFO:tasks.workunit.client.0.smithi162.stdout:test api_snapshots on pid 84068 2024-02-13T00:40:54.491 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_snapshots on pid 84068' 2024-02-13T00:40:54.491 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84068 2024-02-13T00:40:54.491 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.491 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.492 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-02-13T00:40:54.492 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_snapshots_pp 2024-02-13T00:40:54.492 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_snapshots_pp' 2024-02-13T00:40:54.492 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_snapshots_pp 2024-02-13T00:40:54.492 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.492 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_snapshots 2024-02-13T00:40:54.492 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-02-13T00:40:54.492 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_snapshots: /' 2024-02-13T00:40:54.492 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_snapshots_pp 2024-02-13T00:40:54.492 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84076 2024-02-13T00:40:54.493 INFO:tasks.workunit.client.0.smithi162.stdout:test api_snapshots_pp on pid 84076 2024-02-13T00:40:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_snapshots_pp on pid 84076' 2024-02-13T00:40:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84076 2024-02-13T00:40:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-02-13T00:40:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_stat 2024-02-13T00:40:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_stat' 2024-02-13T00:40:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_stat 2024-02-13T00:40:54.494 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_snapshots_pp 2024-02-13T00:40:54.495 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-02-13T00:40:54.495 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-02-13T00:40:54.495 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_stat 2024-02-13T00:40:54.495 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84084 2024-02-13T00:40:54.495 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_stat on pid 84084' 2024-02-13T00:40:54.495 INFO:tasks.workunit.client.0.smithi162.stdout:test api_stat on pid 84084 2024-02-13T00:40:54.496 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84084 2024-02-13T00:40:54.497 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.497 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.497 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-02-13T00:40:54.497 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_stat_pp 2024-02-13T00:40:54.497 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_stat_pp' 2024-02-13T00:40:54.497 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_stat_pp 2024-02-13T00:40:54.497 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.497 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_stat 2024-02-13T00:40:54.497 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_stat.log 2024-02-13T00:40:54.497 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_stat: /' 2024-02-13T00:40:54.497 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_stat_pp 2024-02-13T00:40:54.499 INFO:tasks.workunit.client.0.smithi162.stdout:test api_stat_pp on pid 84092 2024-02-13T00:40:54.500 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84092 2024-02-13T00:40:54.500 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_stat_pp on pid 84092' 2024-02-13T00:40:54.500 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84092 2024-02-13T00:40:54.500 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.500 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.500 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-02-13T00:40:54.501 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_watch_notify 2024-02-13T00:40:54.501 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_watch_notify' 2024-02-13T00:40:54.501 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_watch_notify 2024-02-13T00:40:54.501 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.501 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_stat_pp 2024-02-13T00:40:54.501 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-02-13T00:40:54.501 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_stat_pp: /' 2024-02-13T00:40:54.501 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_watch_notify 2024-02-13T00:40:54.501 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84100 2024-02-13T00:40:54.501 INFO:tasks.workunit.client.0.smithi162.stdout:test api_watch_notify on pid 84100 2024-02-13T00:40:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_watch_notify on pid 84100' 2024-02-13T00:40:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84100 2024-02-13T00:40:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-02-13T00:40:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_watch_notify_pp 2024-02-13T00:40:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_watch_notify_pp' 2024-02-13T00:40:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_watch_notify_pp 2024-02-13T00:40:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_watch_notify 2024-02-13T00:40:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-02-13T00:40:54.504 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_watch_notify: /' 2024-02-13T00:40:54.504 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_watch_notify_pp 2024-02-13T00:40:54.504 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84108 2024-02-13T00:40:54.504 INFO:tasks.workunit.client.0.smithi162.stdout:test api_watch_notify_pp on pid 84108 2024-02-13T00:40:54.505 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_watch_notify_pp on pid 84108' 2024-02-13T00:40:54.505 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84108 2024-02-13T00:40:54.505 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.505 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.505 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-02-13T00:40:54.506 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_cmd 2024-02-13T00:40:54.506 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_cmd' 2024-02-13T00:40:54.506 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_cmd 2024-02-13T00:40:54.506 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.506 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-02-13T00:40:54.506 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-02-13T00:40:54.506 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-02-13T00:40:54.506 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_cmd 2024-02-13T00:40:54.506 INFO:tasks.workunit.client.0.smithi162.stdout:test api_cmd on pid 84116 2024-02-13T00:40:54.507 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84116 2024-02-13T00:40:54.507 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_cmd on pid 84116' 2024-02-13T00:40:54.507 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84116 2024-02-13T00:40:54.507 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-02-13T00:40:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_cmd_pp 2024-02-13T00:40:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_cmd_pp' 2024-02-13T00:40:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_cmd_pp 2024-02-13T00:40:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_cmd 2024-02-13T00:40:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_cmd.log 2024-02-13T00:40:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_cmd: /' 2024-02-13T00:40:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_cmd_pp 2024-02-13T00:40:54.508 INFO:tasks.workunit.client.0.smithi162.stdout:test api_cmd_pp on pid 84124 2024-02-13T00:40:54.510 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84124 2024-02-13T00:40:54.510 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_cmd_pp on pid 84124' 2024-02-13T00:40:54.510 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84124 2024-02-13T00:40:54.510 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.510 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.510 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-02-13T00:40:54.510 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_service 2024-02-13T00:40:54.510 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_service' 2024-02-13T00:40:54.510 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_service 2024-02-13T00:40:54.510 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.510 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_cmd_pp 2024-02-13T00:40:54.510 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-02-13T00:40:54.511 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-02-13T00:40:54.511 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_service 2024-02-13T00:40:54.511 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84132 2024-02-13T00:40:54.511 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_service on pid 84132' 2024-02-13T00:40:54.511 INFO:tasks.workunit.client.0.smithi162.stdout:test api_service on pid 84132 2024-02-13T00:40:54.512 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84132 2024-02-13T00:40:54.512 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.513 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.513 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-02-13T00:40:54.513 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_service_pp 2024-02-13T00:40:54.513 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_service_pp' 2024-02-13T00:40:54.513 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_service_pp 2024-02-13T00:40:54.513 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.513 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_service 2024-02-13T00:40:54.513 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_service.log 2024-02-13T00:40:54.513 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_service: /' 2024-02-13T00:40:54.513 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_service_pp 2024-02-13T00:40:54.513 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84140 2024-02-13T00:40:54.513 INFO:tasks.workunit.client.0.smithi162.stdout:test api_service_pp on pid 84140 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_service_pp on pid 84140' 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84140 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_c_write_operations 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_c_write_operations' 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_c_write_operations 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_service_pp 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_service_pp: /' 2024-02-13T00:40:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_c_write_operations 2024-02-13T00:40:54.516 INFO:tasks.workunit.client.0.smithi162.stdout:test api_c_write_operations on pid 84148 2024-02-13T00:40:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84148 2024-02-13T00:40:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_c_write_operations on pid 84148' 2024-02-13T00:40:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84148 2024-02-13T00:40:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-02-13T00:40:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_c_read_operations 2024-02-13T00:40:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_c_read_operations' 2024-02-13T00:40:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_c_read_operations 2024-02-13T00:40:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_c_write_operations 2024-02-13T00:40:54.518 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-02-13T00:40:54.518 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-02-13T00:40:54.518 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_c_read_operations 2024-02-13T00:40:54.518 INFO:tasks.workunit.client.0.smithi162.stdout:test api_c_read_operations on pid 84156 2024-02-13T00:40:54.519 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84156 2024-02-13T00:40:54.519 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_c_read_operations on pid 84156' 2024-02-13T00:40:54.519 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84156 2024-02-13T00:40:54.519 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.519 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.519 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-02-13T00:40:54.520 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s list_parallel 2024-02-13T00:40:54.520 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' list_parallel' 2024-02-13T00:40:54.520 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo list_parallel 2024-02-13T00:40:54.520 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.520 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_c_read_operations 2024-02-13T00:40:54.520 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-02-13T00:40:54.520 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-02-13T00:40:54.520 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=list_parallel 2024-02-13T00:40:54.520 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84164 2024-02-13T00:40:54.520 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test list_parallel on pid 84164' 2024-02-13T00:40:54.520 INFO:tasks.workunit.client.0.smithi162.stdout:test list_parallel on pid 84164 2024-02-13T00:40:54.521 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84164 2024-02-13T00:40:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-02-13T00:40:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s open_pools_parallel 2024-02-13T00:40:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' open_pools_parallel' 2024-02-13T00:40:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo open_pools_parallel 2024-02-13T00:40:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_list_parallel 2024-02-13T00:40:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_list_parallel.log 2024-02-13T00:40:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ list_parallel: /' 2024-02-13T00:40:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=open_pools_parallel 2024-02-13T00:40:54.523 INFO:tasks.workunit.client.0.smithi162.stdout:test open_pools_parallel on pid 84172 2024-02-13T00:40:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84172 2024-02-13T00:40:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test open_pools_parallel on pid 84172' 2024-02-13T00:40:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84172 2024-02-13T00:40:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-13T00:40:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-02-13T00:40:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s delete_pools_parallel 2024-02-13T00:40:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' delete_pools_parallel' 2024-02-13T00:40:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo delete_pools_parallel 2024-02-13T00:40:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-02-13T00:40:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_open_pools_parallel 2024-02-13T00:40:54.525 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-02-13T00:40:54.525 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-02-13T00:40:54.525 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=delete_pools_parallel 2024-02-13T00:40:54.525 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84180 2024-02-13T00:40:54.525 INFO:tasks.workunit.client.0.smithi162.stdout:test delete_pools_parallel on pid 84180 2024-02-13T00:40:54.526 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test delete_pools_parallel on pid 84180' 2024-02-13T00:40:54.526 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=84180 2024-02-13T00:40:54.526 INFO:tasks.workunit.client.0.smithi162.stderr:+ ret=0 2024-02-13T00:40:54.526 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-02-13T00:40:54.526 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-02-13T00:40:54.526 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:40:54.527 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84068 2024-02-13T00:40:54.527 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84068 2024-02-13T00:40:54.527 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_delete_pools_parallel 2024-02-13T00:40:54.527 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-02-13T00:40:54.527 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-02-13T00:40:54.970 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:54.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:54.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:55.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.067+0000 7fa60a0a4700 1 -- 172.21.15.162:0/2827685475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6040fd470 msgr2=0x7fa6040f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.067+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/2827685475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6040fd470 0x7fa6040f6ca0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fa5f8004870 tx=0x7fa5f8009cb0).stop 2024-02-13T00:40:55.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.067+0000 7fa60a0a4700 1 -- 172.21.15.162:0/2827685475 shutdown_connections 2024-02-13T00:40:55.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.067+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/2827685475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6040f7380 0x7fa604103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.067+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/2827685475 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6040fcb10 0x7fa6040fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.067+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/2827685475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6040fd470 0x7fa6040f6ca0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.067+0000 7fa60a0a4700 1 -- 172.21.15.162:0/2827685475 >> 172.21.15.162:0/2827685475 conn(0x7fa6040f2720 msgr2=0x7fa6040f4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.068+0000 7fa60a0a4700 1 -- 172.21.15.162:0/2827685475 shutdown_connections 2024-02-13T00:40:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.068+0000 7fa60a0a4700 1 -- 172.21.15.162:0/2827685475 wait complete. 2024-02-13T00:40:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.068+0000 7fa60a0a4700 1 librados: starting msgr at 2024-02-13T00:40:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.068+0000 7fa60a0a4700 1 librados: starting objecter 2024-02-13T00:40:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.068+0000 7fa60a0a4700 1 Processor -- start 2024-02-13T00:40:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.068+0000 7fa60a0a4700 1 -- start start 2024-02-13T00:40:55.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.068+0000 7fa60a0a4700 1 librados: setting wanted keys 2024-02-13T00:40:55.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.068+0000 7fa60a0a4700 1 librados: calling monclient init 2024-02-13T00:40:55.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.068+0000 7fa60a0a4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6040f7380 0x7fa6041035d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:55.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.068+0000 7fa60a0a4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6040fcb10 0x7fa60406a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:55.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.068+0000 7fa60a0a4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6040fd470 0x7fa604101860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:55.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa60a0a4700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa604103bd0 con 0x7fa6040f7380 2024-02-13T00:40:55.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa60a0a4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa604101d50 con 0x7fa6040fcb10 2024-02-13T00:40:55.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa60a0a4700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa604101f10 con 0x7fa6040fd470 2024-02-13T00:40:55.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa602ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6040fcb10 0x7fa60406a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:55.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa602ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6040fcb10 0x7fa60406a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:45408/0 (socket says 172.21.15.162:45408) 2024-02-13T00:40:55.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa602ffd700 1 -- 172.21.15.162:0/293721234 learned_addr learned my addr 172.21.15.162:0/293721234 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:40:55.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:40:55.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa602ffd700 1 -- 172.21.15.162:0/293721234 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6040fd470 msgr2=0x7fa604101860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa602ffd700 1 --2- 172.21.15.162:0/293721234 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6040fd470 0x7fa604101860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:40:55.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa602ffd700 1 -- 172.21.15.162:0/293721234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6040f7380 msgr2=0x7fa6041035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa602ffd700 1 --2- 172.21.15.162:0/293721234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6040f7380 0x7fa6041035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:40:55.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa602ffd700 1 -- 172.21.15.162:0/293721234 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc0048f0 con 0x7fa6040fcb10 2024-02-13T00:40:55.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa602ffd700 1 --2- 172.21.15.162:0/293721234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6040fcb10 0x7fa60406a800 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fa5f8009eb0 tx=0x7fa5f800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:55.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa600ff9700 10 client.?.objecter ms_handle_connect 0x7fa6040fcb10 2024-02-13T00:40:55.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa600ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:40:55.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f800e030 con 0x7fa6040fcb10 2024-02-13T00:40:55.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5f8014e80 con 0x7fa6040fcb10 2024-02-13T00:40:55.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.069+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8019070 con 0x7fa6040fcb10 2024-02-13T00:40:55.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.071+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f8005040 con 0x7fa6040fcb10 2024-02-13T00:40:55.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.071+0000 7fa60a0a4700 10 client.24808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:40:55.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.071+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6041023d0 con 0x7fa6040fcb10 2024-02-13T00:40:55.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.071+0000 7fa60a0a4700 1 librados: init done 2024-02-13T00:40:55.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.071+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa6040607d0 con 0x7fa6040fcb10 2024-02-13T00:40:55.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.073+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fa5f80092e0 con 0x7fa6040fcb10 2024-02-13T00:40:55.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.073+0000 7fa600ff9700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa5ec062c90 0x7fa5ec065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:55.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.073+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(102..102 src has 52..102) v4 ==== 6204+0+0 (secure 0 0 0) 0x7fa5f807e400 con 0x7fa6040fcb10 2024-02-13T00:40:55.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.073+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 osd_map(102..102 src has 52..102) v4 2024-02-13T00:40:55.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.073+0000 7fa600ff9700 3 client.24808.objecter handle_osd_map got epochs [102,102] > 0 2024-02-13T00:40:55.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.073+0000 7fa600ff9700 3 client.24808.objecter handle_osd_map decoding full epoch 102 2024-02-13T00:40:55.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.073+0000 7fa6037fe700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa5ec062c90 0x7fa5ec065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:55.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.073+0000 7fa600ff9700 20 client.24808.objecter dump_active .. 0 homeless 2024-02-13T00:40:55.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.074+0000 7fa600ff9700 10 client.24808.objecter ms_handle_connect 0x7fa5ec062c90 2024-02-13T00:40:55.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.074+0000 7fa6037fe700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa5ec062c90 0x7fa5ec065150 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fa5f4004b90 tx=0x7fa5f4009b60).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:55.108 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:55.129 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:40:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:55.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:40:55.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:55.261 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:55.281 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:55.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v102) v1 ==== 54+0+16 (secure 0 0 0) 0x7fa5f8052050 con 0x7fa6040fcb10 2024-02-13T00:40:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 1 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa6040f9c70 0x7fa6040fa050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa6040475e0 osd=0 3 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6040475e0 osd=0 3 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa6040475e0 osd=0 3 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 1 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa6040475e0 osd=0 3 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 0 1 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6040475e0 osd=0 4 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa60a0a4700 10 client.24808.objecter _send_command 1 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa604104e40 con 0x7fa6040f9c70 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.286+0000 7fa603fff700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa6040f9c70 0x7fa6040fa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.287+0000 7fa600ff9700 10 client.24808.objecter ms_handle_connect 0x7fa6040f9c70 2024-02-13T00:40:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.287+0000 7fa603fff700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa6040f9c70 0x7fa6040fa050 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:55.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.289+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa604104e40 con 0x7fa6040f9c70 2024-02-13T00:40:55.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.289+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:40:55.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.289+0000 7fa600ff9700 10 client.24808.objecter _finish_command 1 = system:0 2024-02-13T00:40:55.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.289+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa6040475e0 osd=0 4 2024-02-13T00:40:55.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.289+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 0 1 2024-02-13T00:40:55.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.314+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:55.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.314+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 2 2024-02-13T00:40:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.314+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa6040475e0 osd=0 3 2024-02-13T00:40:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.314+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6040475e0 osd=0 3 2024-02-13T00:40:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.314+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa6040475e0 osd=0 3 2024-02-13T00:40:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.314+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 2 2024-02-13T00:40:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.314+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa6040475e0 osd=0 3 2024-02-13T00:40:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.314+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 0 2 2024-02-13T00:40:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.314+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6040475e0 osd=0 4 2024-02-13T00:40:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.314+0000 7fa60a0a4700 10 client.24808.objecter _send_command 2 2024-02-13T00:40:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.314+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa6040fa540 con 0x7fa6040f9c70 2024-02-13T00:40:55.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa6040fa540 con 0x7fa6040f9c70 2024-02-13T00:40:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa600ff9700 10 client.24808.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa6040475e0 osd=0 4 2024-02-13T00:40:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 0 2 2024-02-13T00:40:55.322 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:40:55.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:55.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 3 2024-02-13T00:40:55.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6805/150599780 conn(0x7fa604028590 0x7fa604059990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:55.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa6040fa710 osd=1 3 2024-02-13T00:40:55.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6040fa710 osd=1 3 2024-02-13T00:40:55.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa6040fa710 osd=1 3 2024-02-13T00:40:55.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 3 2024-02-13T00:40:55.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa6040fa710 osd=1 3 2024-02-13T00:40:55.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 1 3 2024-02-13T00:40:55.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6040fa710 osd=1 4 2024-02-13T00:40:55.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa60a0a4700 10 client.24808.objecter _send_command 3 2024-02-13T00:40:55.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa604059e80 con 0x7fa604028590 2024-02-13T00:40:55.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa6037fe700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6805/150599780 conn(0x7fa604028590 0x7fa604059990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:55.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa600ff9700 10 client.24808.objecter ms_handle_connect 0x7fa604028590 2024-02-13T00:40:55.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.316+0000 7fa6037fe700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6805/150599780 conn(0x7fa604028590 0x7fa604059990 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:55.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.317+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa604059e80 con 0x7fa604028590 2024-02-13T00:40:55.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.317+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:40:55.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.317+0000 7fa600ff9700 10 client.24808.objecter _finish_command 3 = system:0 2024-02-13T00:40:55.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.317+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa6040fa710 osd=1 4 2024-02-13T00:40:55.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.317+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 1 3 2024-02-13T00:40:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.332+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.332+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 4 2024-02-13T00:40:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.332+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa6040fa710 osd=1 3 2024-02-13T00:40:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.332+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6040fa710 osd=1 3 2024-02-13T00:40:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.332+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa6040fa710 osd=1 3 2024-02-13T00:40:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.332+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 4 2024-02-13T00:40:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.332+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa6040fa710 osd=1 3 2024-02-13T00:40:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.332+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 1 4 2024-02-13T00:40:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.332+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6040fa710 osd=1 4 2024-02-13T00:40:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.332+0000 7fa60a0a4700 10 client.24808.objecter _send_command 4 2024-02-13T00:40:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.332+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa60410c710 con 0x7fa604028590 2024-02-13T00:40:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa60410c710 con 0x7fa604028590 2024-02-13T00:40:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa600ff9700 10 client.24808.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa6040fa710 osd=1 4 2024-02-13T00:40:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 1 4 2024-02-13T00:40:55.338 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:40:55.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:55.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 5 2024-02-13T00:40:55.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa604059570 0x7fa60410fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:55.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa604059400 osd=2 3 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604059400 osd=2 3 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa604059400 osd=2 3 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 5 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa604059400 osd=2 3 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 2 5 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604059400 osd=2 4 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa60a0a4700 10 client.24808.objecter _send_command 5 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6040281c0 con 0x7fa604059570 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.334+0000 7fa602ffd700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa604059570 0x7fa60410fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.335+0000 7fa600ff9700 10 client.24808.objecter ms_handle_connect 0x7fa604059570 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.335+0000 7fa602ffd700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa604059570 0x7fa60410fa40 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.335+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa6040281c0 con 0x7fa604059570 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.335+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:40:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.335+0000 7fa600ff9700 10 client.24808.objecter _finish_command 5 = system:0 2024-02-13T00:40:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.335+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa604059400 osd=2 4 2024-02-13T00:40:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.335+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 2 5 2024-02-13T00:40:55.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.350+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:55.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.350+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 6 2024-02-13T00:40:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.350+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa604059400 osd=2 3 2024-02-13T00:40:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.350+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604059400 osd=2 3 2024-02-13T00:40:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.350+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa604059400 osd=2 3 2024-02-13T00:40:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.350+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 6 2024-02-13T00:40:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.350+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa604059400 osd=2 3 2024-02-13T00:40:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.350+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 2 6 2024-02-13T00:40:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.350+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604059400 osd=2 4 2024-02-13T00:40:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.350+0000 7fa60a0a4700 10 client.24808.objecter _send_command 6 2024-02-13T00:40:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.350+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa604027650 con 0x7fa604059570 2024-02-13T00:40:55.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa604027650 con 0x7fa604059570 2024-02-13T00:40:55.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:55.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa600ff9700 10 client.24808.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa604059400 osd=2 4 2024-02-13T00:40:55.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 2 6 2024-02-13T00:40:55.355 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:40:55.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:55.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 7 2024-02-13T00:40:55.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6813/604971011 conn(0x7fa604134f30 0x7fa604137320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:55.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa604027a60 osd=3 3 2024-02-13T00:40:55.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604027a60 osd=3 3 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa604027a60 osd=3 3 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 7 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa604027a60 osd=3 3 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 3 7 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604027a60 osd=3 4 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa60a0a4700 10 client.24808.objecter _send_command 7 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa604110ca0 con 0x7fa604134f30 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.351+0000 7fa603fff700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6813/604971011 conn(0x7fa604134f30 0x7fa604137320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.352+0000 7fa600ff9700 10 client.24808.objecter ms_handle_connect 0x7fa604134f30 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.352+0000 7fa603fff700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6813/604971011 conn(0x7fa604134f30 0x7fa604137320 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.352+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa604110ca0 con 0x7fa604134f30 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.352+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.352+0000 7fa600ff9700 10 client.24808.objecter _finish_command 7 = system:0 2024-02-13T00:40:55.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.352+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa604027a60 osd=3 4 2024-02-13T00:40:55.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.352+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 3 7 2024-02-13T00:40:55.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.366+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:55.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.366+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 8 2024-02-13T00:40:55.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.366+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa604027a60 osd=3 3 2024-02-13T00:40:55.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.366+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604027a60 osd=3 3 2024-02-13T00:40:55.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.366+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa604027a60 osd=3 3 2024-02-13T00:40:55.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.366+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 8 2024-02-13T00:40:55.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.366+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa604027a60 osd=3 3 2024-02-13T00:40:55.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.366+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 3 8 2024-02-13T00:40:55.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.366+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604027a60 osd=3 4 2024-02-13T00:40:55.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.366+0000 7fa60a0a4700 10 client.24808.objecter _send_command 8 2024-02-13T00:40:55.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.366+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa604110170 con 0x7fa604134f30 2024-02-13T00:40:55.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa604110170 con 0x7fa604134f30 2024-02-13T00:40:55.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:55.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa600ff9700 10 client.24808.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa604027a60 osd=3 4 2024-02-13T00:40:55.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 3 8 2024-02-13T00:40:55.372 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:40:55.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:55.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 9 2024-02-13T00:40:55.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa60410d590 0x7fa60413a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa6041103b0 osd=4 3 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6041103b0 osd=4 3 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa6041103b0 osd=4 3 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 9 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa6041103b0 osd=4 3 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 4 9 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6041103b0 osd=4 4 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa60a0a4700 10 client.24808.objecter _send_command 9 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa60413d5f0 con 0x7fa60410d590 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.368+0000 7fa6037fe700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa60410d590 0x7fa60413a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.369+0000 7fa600ff9700 10 client.24808.objecter ms_handle_connect 0x7fa60410d590 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.369+0000 7fa6037fe700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa60410d590 0x7fa60413a9a0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.369+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa60413d5f0 con 0x7fa60410d590 2024-02-13T00:40:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.369+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:40:55.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.369+0000 7fa600ff9700 10 client.24808.objecter _finish_command 9 = system:0 2024-02-13T00:40:55.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.369+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa6041103b0 osd=4 4 2024-02-13T00:40:55.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.369+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 4 9 2024-02-13T00:40:55.384 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:55.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.384+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:55.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.384+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 10 2024-02-13T00:40:55.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.384+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa6041103b0 osd=4 3 2024-02-13T00:40:55.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.384+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6041103b0 osd=4 3 2024-02-13T00:40:55.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.384+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa6041103b0 osd=4 3 2024-02-13T00:40:55.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.384+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 10 2024-02-13T00:40:55.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.384+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa6041103b0 osd=4 3 2024-02-13T00:40:55.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.384+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 4 10 2024-02-13T00:40:55.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.384+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6041103b0 osd=4 4 2024-02-13T00:40:55.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.384+0000 7fa60a0a4700 10 client.24808.objecter _send_command 10 2024-02-13T00:40:55.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.384+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa60413ae90 con 0x7fa60410d590 2024-02-13T00:40:55.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa60413ae90 con 0x7fa60410d590 2024-02-13T00:40:55.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:55.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa600ff9700 10 client.24808.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa6041103b0 osd=4 4 2024-02-13T00:40:55.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 4 10 2024-02-13T00:40:55.390 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:40:55.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:55.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 11 2024-02-13T00:40:55.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6804/2212296748 conn(0x7fa60413b570 0x7fa60413d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:55.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa60413ae90 osd=5 3 2024-02-13T00:40:55.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa60413ae90 osd=5 3 2024-02-13T00:40:55.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa60413ae90 osd=5 3 2024-02-13T00:40:55.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 11 2024-02-13T00:40:55.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa60413ae90 osd=5 3 2024-02-13T00:40:55.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 5 11 2024-02-13T00:40:55.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa60413ae90 osd=5 4 2024-02-13T00:40:55.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa60a0a4700 10 client.24808.objecter _send_command 11 2024-02-13T00:40:55.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.385+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa604140730 con 0x7fa60413b570 2024-02-13T00:40:55.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.386+0000 7fa602ffd700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6804/2212296748 conn(0x7fa60413b570 0x7fa60413d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:55.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.386+0000 7fa600ff9700 10 client.24808.objecter ms_handle_connect 0x7fa60413b570 2024-02-13T00:40:55.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.386+0000 7fa602ffd700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6804/2212296748 conn(0x7fa60413b570 0x7fa60413d960 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:55.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.387+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa604140730 con 0x7fa60413b570 2024-02-13T00:40:55.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.387+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:40:55.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.387+0000 7fa600ff9700 10 client.24808.objecter _finish_command 11 = system:0 2024-02-13T00:40:55.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.387+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa60413ae90 osd=5 4 2024-02-13T00:40:55.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.387+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 5 11 2024-02-13T00:40:55.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.401+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:55.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.401+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 12 2024-02-13T00:40:55.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.401+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa60413ae90 osd=5 3 2024-02-13T00:40:55.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.401+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa60413ae90 osd=5 3 2024-02-13T00:40:55.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.401+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa60413ae90 osd=5 3 2024-02-13T00:40:55.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.401+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 12 2024-02-13T00:40:55.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.401+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa60413ae90 osd=5 3 2024-02-13T00:40:55.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.401+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 5 12 2024-02-13T00:40:55.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.401+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa60413ae90 osd=5 4 2024-02-13T00:40:55.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.401+0000 7fa60a0a4700 10 client.24808.objecter _send_command 12 2024-02-13T00:40:55.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.401+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa604140730 con 0x7fa60413b570 2024-02-13T00:40:55.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.402+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa604140730 con 0x7fa60413b570 2024-02-13T00:40:55.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.402+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:55.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.402+0000 7fa600ff9700 10 client.24808.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.402+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa60413ae90 osd=5 4 2024-02-13T00:40:55.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.402+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 5 12 2024-02-13T00:40:55.407 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:40:55.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:55.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 13 2024-02-13T00:40:55.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa60413fb00 0x7fa60413ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:55.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa604140730 osd=6 3 2024-02-13T00:40:55.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604140730 osd=6 3 2024-02-13T00:40:55.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa604140730 osd=6 3 2024-02-13T00:40:55.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 13 2024-02-13T00:40:55.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa604140730 osd=6 3 2024-02-13T00:40:55.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 6 13 2024-02-13T00:40:55.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604140730 osd=6 4 2024-02-13T00:40:55.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa60a0a4700 10 client.24808.objecter _send_command 13 2024-02-13T00:40:55.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa604143ae0 con 0x7fa60413fb00 2024-02-13T00:40:55.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa603fff700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa60413fb00 0x7fa60413ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:55.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa600ff9700 10 client.24808.objecter ms_handle_connect 0x7fa60413fb00 2024-02-13T00:40:55.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.403+0000 7fa603fff700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa60413fb00 0x7fa60413ff60 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:55.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.404+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa604143ae0 con 0x7fa60413fb00 2024-02-13T00:40:55.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.404+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:40:55.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.404+0000 7fa600ff9700 10 client.24808.objecter _finish_command 13 = system:0 2024-02-13T00:40:55.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.404+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa604140730 osd=6 4 2024-02-13T00:40:55.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.404+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 6 13 2024-02-13T00:40:55.410 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:40:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:40:55.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:53.535220+0000 osd.5 (osd.5) 59 : cluster [DBG] 5.16 scrub starts 2024-02-13T00:40:55.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:53.536167+0000 osd.5 (osd.5) 60 : cluster [DBG] 5.16 scrub ok 2024-02-13T00:40:55.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:53.948093+0000 mon.a (mon.0) 755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2024-02-13T00:40:55.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:53.948153+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-13T00:40:55.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:54.056399+0000 osd.3 (osd.3) 75 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:40:55.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:54.059270+0000 osd.3 (osd.3) 76 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:40:55.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:54.313708+0000 osd.6 (osd.6) 41 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:40:55.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:54.316586+0000 osd.6 (osd.6) 42 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:40:55.411 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:55.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.418+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:55.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.418+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 14 2024-02-13T00:40:55.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.418+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa604140730 osd=6 3 2024-02-13T00:40:55.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.418+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604140730 osd=6 3 2024-02-13T00:40:55.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.418+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa604140730 osd=6 3 2024-02-13T00:40:55.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.418+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 14 2024-02-13T00:40:55.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.418+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa604140730 osd=6 3 2024-02-13T00:40:55.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.418+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 6 14 2024-02-13T00:40:55.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.418+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604140730 osd=6 4 2024-02-13T00:40:55.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.418+0000 7fa60a0a4700 10 client.24808.objecter _send_command 14 2024-02-13T00:40:55.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.418+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa60413df10 con 0x7fa60413fb00 2024-02-13T00:40:55.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa60413df10 con 0x7fa60413fb00 2024-02-13T00:40:55.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:55.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa600ff9700 10 client.24808.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa604140730 osd=6 4 2024-02-13T00:40:55.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 6 14 2024-02-13T00:40:55.428 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:40:55.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:40:55.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 15 2024-02-13T00:40:55.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa60413e620 0x7fa60413ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:40:55.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa60413df10 osd=7 3 2024-02-13T00:40:55.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa60413df10 osd=7 3 2024-02-13T00:40:55.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa60413df10 osd=7 3 2024-02-13T00:40:55.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 15 2024-02-13T00:40:55.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa60413df10 osd=7 3 2024-02-13T00:40:55.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 7 15 2024-02-13T00:40:55.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa60413df10 osd=7 4 2024-02-13T00:40:55.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa60a0a4700 10 client.24808.objecter _send_command 15 2024-02-13T00:40:55.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.421+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa60413f0f0 con 0x7fa60413e620 2024-02-13T00:40:55.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.422+0000 7fa6037fe700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa60413e620 0x7fa60413ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:40:55.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.422+0000 7fa600ff9700 10 client.24808.objecter ms_handle_connect 0x7fa60413e620 2024-02-13T00:40:55.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.422+0000 7fa6037fe700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa60413e620 0x7fa60413ea40 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:40:55.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.423+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa60413f0f0 con 0x7fa60413e620 2024-02-13T00:40:55.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.423+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:40:55.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.423+0000 7fa600ff9700 10 client.24808.objecter _finish_command 15 = system:0 2024-02-13T00:40:55.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.423+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa60413df10 osd=7 4 2024-02-13T00:40:55.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.423+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 7 15 2024-02-13T00:40:55.430 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:55.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:55.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.437+0000 7fa60a0a4700 10 client.24808.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:40:55.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.437+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign -1 16 2024-02-13T00:40:55.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.437+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa60413df10 osd=7 3 2024-02-13T00:40:55.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.437+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa60413df10 osd=7 3 2024-02-13T00:40:55.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.437+0000 7fa60a0a4700 20 client.24808.objecter _get_session s=0x7fa60413df10 osd=7 3 2024-02-13T00:40:55.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.437+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_remove -1 16 2024-02-13T00:40:55.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.437+0000 7fa60a0a4700 20 client.24808.objecter get_session s=0x7fa60413df10 osd=7 3 2024-02-13T00:40:55.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.437+0000 7fa60a0a4700 15 client.24808.objecter _session_command_op_assign 7 16 2024-02-13T00:40:55.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.437+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa60413df10 osd=7 4 2024-02-13T00:40:55.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.437+0000 7fa60a0a4700 10 client.24808.objecter _send_command 16 2024-02-13T00:40:55.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.437+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa60413f0b0 con 0x7fa60413e620 2024-02-13T00:40:55.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa600ff9700 1 -- 172.21.15.162:0/293721234 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa60413f0b0 con 0x7fa60413e620 2024-02-13T00:40:55.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa600ff9700 10 client.24808.objecter ms_dispatch 0x7fa604001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:40:55.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa600ff9700 10 client.24808.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa600ff9700 20 client.24808.objecter put_session s=0x7fa60413df10 osd=7 4 2024-02-13T00:40:55.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa600ff9700 15 client.24808.objecter _session_command_op_remove 7 16 2024-02-13T00:40:55.443 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:40:55.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 10 librados: watch_flush enter 2024-02-13T00:40:55.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 10 librados: watch_flush exit 2024-02-13T00:40:55.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 10 client.24808.objecter close_session for osd.0 2024-02-13T00:40:55.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa6040f9c70 msgr2=0x7fa6040fa050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa6040f9c70 0x7fa6040fa050 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6040475e0 osd=0 1 2024-02-13T00:40:55.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 10 client.24808.objecter close_session for osd.1 2024-02-13T00:40:55.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6805/150599780 conn(0x7fa604028590 msgr2=0x7fa604059990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6805/150599780 conn(0x7fa604028590 0x7fa604059990 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6040fa710 osd=1 1 2024-02-13T00:40:55.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 10 client.24808.objecter close_session for osd.2 2024-02-13T00:40:55.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa604059570 msgr2=0x7fa60410fa40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa604059570 0x7fa60410fa40 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604059400 osd=2 1 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 10 client.24808.objecter close_session for osd.3 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa603fff700 1 -- 172.21.15.162:0/293721234 reap_dead start 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6813/604971011 conn(0x7fa604134f30 msgr2=0x7fa604137320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6813/604971011 conn(0x7fa604134f30 0x7fa604137320 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604027a60 osd=3 1 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 10 client.24808.objecter close_session for osd.4 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa60410d590 msgr2=0x7fa60413a9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa60410d590 0x7fa60413a9a0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa6041103b0 osd=4 1 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 10 client.24808.objecter close_session for osd.5 2024-02-13T00:40:55.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6804/2212296748 conn(0x7fa60413b570 msgr2=0x7fa60413d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6804/2212296748 conn(0x7fa60413b570 0x7fa60413d960 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa60413ae90 osd=5 1 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 10 client.24808.objecter close_session for osd.6 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa60413fb00 msgr2=0x7fa60413ff60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa60413fb00 0x7fa60413ff60 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa604140730 osd=6 1 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 10 client.24808.objecter close_session for osd.7 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa60413e620 msgr2=0x7fa60413ea40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa60413e620 0x7fa60413ea40 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 20 client.24808.objecter put_session s=0x7fa60413df10 osd=7 1 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 20 client.24808.objecter shutdown clearing up homeless session... 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 10 client.24808.objecter successfully canceled tick 2024-02-13T00:40:55.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa603fff700 1 -- 172.21.15.162:0/293721234 reap_dead start 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa5ec062c90 msgr2=0x7fa5ec065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa5ec062c90 0x7fa5ec065150 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fa5f4004b90 tx=0x7fa5f4009b60).stop 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6040fcb10 msgr2=0x7fa60406a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6040fcb10 0x7fa60406a800 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fa5f8009eb0 tx=0x7fa5f800b040).stop 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 shutdown_connections 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6040fcb10 0x7fa60406a800 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 --2- 172.21.15.162:0/293721234 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa5ec062c90 0x7fa5ec065150 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 >> 172.21.15.162:0/293721234 conn(0x7fa6040f2720 msgr2=0x7fa604107610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 shutdown_connections 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.439+0000 7fa60a0a4700 1 -- 172.21.15.162:0/293721234 wait complete. 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:40:55.440+0000 7fa60a0a4700 1 librados: shutdown 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.448 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.449 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.449 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.449 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.449 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.449 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.449 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:40:55.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:55.560 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:53.535220+0000 osd.5 (osd.5) 59 : cluster [DBG] 5.16 scrub starts 2024-02-13T00:40:55.560 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:53.536167+0000 osd.5 (osd.5) 60 : cluster [DBG] 5.16 scrub ok 2024-02-13T00:40:55.560 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:53.948093+0000 mon.a (mon.0) 755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2024-02-13T00:40:55.560 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:53.948153+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-13T00:40:55.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:54.056399+0000 osd.3 (osd.3) 75 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:40:55.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:54.059270+0000 osd.3 (osd.3) 76 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:40:55.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:54.313708+0000 osd.6 (osd.6) 41 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:40:55.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:54.316586+0000 osd.6 (osd.6) 42 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:40:55.561 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:53.535220+0000 osd.5 (osd.5) 59 : cluster [DBG] 5.16 scrub starts 2024-02-13T00:40:55.561 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:53.536167+0000 osd.5 (osd.5) 60 : cluster [DBG] 5.16 scrub ok 2024-02-13T00:40:55.562 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:53.948093+0000 mon.a (mon.0) 755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2024-02-13T00:40:55.562 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:53.948153+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-13T00:40:55.562 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:54.056399+0000 osd.3 (osd.3) 75 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:40:55.562 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:54.059270+0000 osd.3 (osd.3) 76 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:40:55.562 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:54.313708+0000 osd.6 (osd.6) 41 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:40:55.562 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:54.316586+0000 osd.6 (osd.6) 42 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:40:55.563 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:56.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:40:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:40:56] "GET /metrics HTTP/1.1" 200 172271 "" "Prometheus/2.33.4" 2024-02-13T00:40:56.612 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:56.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:56.715 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:56.747 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:56.769 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:56.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:56.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:56.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:40:56.901 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:56.920 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:40:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:56.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:40:57.023 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:40:57.051 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:57.070 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:40:57.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:57.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:40:57.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:57.202 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:57.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:57.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:57.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:40:57.436 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:57.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:54.199886+0000 mgr.y (mgr.14544) 130 : cluster [DBG] pgmap v122: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-02-13T00:40:57.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:54.801060+0000 mon.b (mon.1) 28 : audit [DBG] from='client.? 172.21.15.162:0/3066705702' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-13T00:40:57.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:55.040408+0000 mon.c (mon.2) 89 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:40:57.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:55.074049+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? 172.21.15.162:0/293721234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:40:57.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:55.289834+0000 mon.a (mon.0) 757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:40:57.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:54.199886+0000 mgr.y (mgr.14544) 130 : cluster [DBG] pgmap v122: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-02-13T00:40:57.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:54.801060+0000 mon.b (mon.1) 28 : audit [DBG] from='client.? 172.21.15.162:0/3066705702' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-13T00:40:57.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:55.040408+0000 mon.c (mon.2) 89 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:40:57.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:55.074049+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? 172.21.15.162:0/293721234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:40:57.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:55.289834+0000 mon.a (mon.0) 757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:40:57.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:40:57.646 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:40:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:40:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:40:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:40:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:34 4.4 4:227f63b5:gc::gc.14:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.302677+0000", 2024-02-13T00:40:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.33888908399999, 2024-02-13T00:40:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078023559999999999, 2024-02-13T00:40:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 34 2024-02-13T00:40:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.302677+0000", 2024-02-13T00:40:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.302677+0000", 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.302675+0000", 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9920000000000001e-06 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.302682+0000", 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1123e-05 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.302693+0000", 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6250000000000001e-06 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.302700+0000", 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8032999999999998e-05 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.302758+0000", 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117644 2024-02-13T00:40:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.302876+0000", 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000223799 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.303099+0000", 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050102100000000004 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.303600+0000", 2024-02-13T00:40:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00065706599999999999 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.304258+0000", 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061412660000000003 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310399+0000", 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2265000000000003e-05 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310441+0000", 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8077999999999997e-05 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310479+0000", 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078023559999999999 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:62 4.f 4:f1983ba0:::obj_delete_at_hint.0000000001:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.310368+0000", 2024-02-13T00:40:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.33119742900001, 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026770940000000001, 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 62 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310368+0000", 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310368+0000", 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310367+0000", 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1590000000000001e-06 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310372+0000", 2024-02-13T00:40:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1999999999999994e-06 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310381+0000", 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3000000000000001e-06 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310386+0000", 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000154473 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310540+0000", 2024-02-13T00:40:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1191000000000003e-05 2024-02-13T00:40:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310612+0000", 2024-02-13T00:40:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000201249 2024-02-13T00:40:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,4", 2024-02-13T00:40:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.310813+0000", 2024-02-13T00:40:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00046535899999999999 2024-02-13T00:40:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.311278+0000", 2024-02-13T00:40:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035753500000000003 2024-02-13T00:40:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.311636+0000", 2024-02-13T00:40:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013606619999999999 2024-02-13T00:40:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.312996+0000", 2024-02-13T00:40:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9006e-05 2024-02-13T00:40:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.313025+0000", 2024-02-13T00:40:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0080000000000001e-05 2024-02-13T00:40:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.313046+0000", 2024-02-13T00:40:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026770940000000001 2024-02-13T00:40:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:77 4.4 4:227f63b5:gc::gc.14:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.314311+0000", 2024-02-13T00:40:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.32725525999999, 2024-02-13T00:40:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052519749999999999, 2024-02-13T00:40:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 77 2024-02-13T00:40:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.314311+0000", 2024-02-13T00:40:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.314311+0000", 2024-02-13T00:40:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.314310+0000", 2024-02-13T00:40:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6029999999999998e-06 2024-02-13T00:40:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.314314+0000", 2024-02-13T00:40:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8859999999999993e-06 2024-02-13T00:40:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.314323+0000", 2024-02-13T00:40:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5210000000000004e-06 2024-02-13T00:40:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.314328+0000", 2024-02-13T00:40:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9712000000000002e-05 2024-02-13T00:40:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.314367+0000", 2024-02-13T00:40:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3357999999999997e-05 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.314451+0000", 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018688799999999999 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.314638+0000", 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050679999999999996 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.315144+0000", 2024-02-13T00:40:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069258299999999998 2024-02-13T00:40:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.315837+0000", 2024-02-13T00:40:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036480639999999999 2024-02-13T00:40:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.319485+0000", 2024-02-13T00:40:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2246000000000002e-05 2024-02-13T00:40:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.319527+0000", 2024-02-13T00:40:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5243999999999997e-05 2024-02-13T00:40:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.319563+0000", 2024-02-13T00:40:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052519749999999999 2024-02-13T00:40:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:96 4.4 4:26625e58:lc::lc.26:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.320612+0000", 2024-02-13T00:40:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.32095390999999, 2024-02-13T00:40:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002642035, 2024-02-13T00:40:57.670 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.670 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.670 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.670 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.670 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.670 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 96 2024-02-13T00:40:57.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.670 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320612+0000", 2024-02-13T00:40:57.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320612+0000", 2024-02-13T00:40:57.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:57.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320611+0000", 2024-02-13T00:40:57.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4790000000000004e-06 2024-02-13T00:40:57.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320616+0000", 2024-02-13T00:40:57.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4579999999999993e-06 2024-02-13T00:40:57.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320625+0000", 2024-02-13T00:40:57.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5119e-05 2024-02-13T00:40:57.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320640+0000", 2024-02-13T00:40:57.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076051899999999997 2024-02-13T00:40:57.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.321400+0000", 2024-02-13T00:40:57.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1709000000000002e-05 2024-02-13T00:40:57.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.321462+0000", 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013057300000000001 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.321593+0000", 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00062982399999999997 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.322222+0000", 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032895199999999998 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.322551+0000", 2024-02-13T00:40:57.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00062084599999999998 2024-02-13T00:40:57.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.323172+0000", 2024-02-13T00:40:57.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6307999999999998e-05 2024-02-13T00:40:57.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.323219+0000", 2024-02-13T00:40:57.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5432999999999998e-05 2024-02-13T00:40:57.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.323254+0000", 2024-02-13T00:40:57.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002642035 2024-02-13T00:40:57.676 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.676 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.676 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.676 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:97 4.7 4:e07a6548:gc::gc.15:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.677 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.320653+0000", 2024-02-13T00:40:57.677 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.320913179, 2024-02-13T00:40:57.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028186470000000001, 2024-02-13T00:40:57.677 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.677 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.677 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.677 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.677 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.677 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 97 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320653+0000", 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320653+0000", 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320652+0000", 2024-02-13T00:40:57.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3800000000000004e-06 2024-02-13T00:40:57.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320656+0000", 2024-02-13T00:40:57.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4459999999999998e-06 2024-02-13T00:40:57.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320663+0000", 2024-02-13T00:40:57.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5800000000000002e-06 2024-02-13T00:40:57.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320667+0000", 2024-02-13T00:40:57.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1532000000000007e-05 2024-02-13T00:40:57.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320729+0000", 2024-02-13T00:40:57.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106333 2024-02-13T00:40:57.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.320835+0000", 2024-02-13T00:40:57.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020555000000000001 2024-02-13T00:40:57.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-13T00:40:57.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.321041+0000", 2024-02-13T00:40:57.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00065579199999999999 2024-02-13T00:40:57.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.321697+0000", 2024-02-13T00:40:57.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012150399999999999 2024-02-13T00:40:57.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.321818+0000", 2024-02-13T00:40:57.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00157647 2024-02-13T00:40:57.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.323395+0000", 2024-02-13T00:40:57.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1702999999999997e-05 2024-02-13T00:40:57.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.323436+0000", 2024-02-13T00:40:57.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5111999999999999e-05 2024-02-13T00:40:57.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.323471+0000", 2024-02-13T00:40:57.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028186470000000001 2024-02-13T00:40:57.684 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.684 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.684 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.685 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:106 4.2 4:46cb1ec1:reshard::reshard.0000000003:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.685 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.325226+0000", 2024-02-13T00:40:57.685 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.316339626, 2024-02-13T00:40:57.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048545580000000001, 2024-02-13T00:40:57.685 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.685 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.685 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.685 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.685 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.685 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 106 2024-02-13T00:40:57.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.686 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.325226+0000", 2024-02-13T00:40:57.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.325226+0000", 2024-02-13T00:40:57.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:40:57.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.325225+0000", 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0849999999999999e-06 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.325232+0000", 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4433999999999999e-05 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.325247+0000", 2024-02-13T00:40:57.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8689999999999998e-06 2024-02-13T00:40:57.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.325253+0000", 2024-02-13T00:40:57.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025559480000000002 2024-02-13T00:40:57.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.327809+0000", 2024-02-13T00:40:57.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4678000000000005e-05 2024-02-13T00:40:57.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.327894+0000", 2024-02-13T00:40:57.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000199877 2024-02-13T00:40:57.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:40:57.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.328094+0000", 2024-02-13T00:40:57.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076560400000000002 2024-02-13T00:40:57.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.328859+0000", 2024-02-13T00:40:57.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1919999999999999e-06 2024-02-13T00:40:57.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.328867+0000", 2024-02-13T00:40:57.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011355079999999999 2024-02-13T00:40:57.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330002+0000", 2024-02-13T00:40:57.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4999000000000001e-05 2024-02-13T00:40:57.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330047+0000", 2024-02-13T00:40:57.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3609000000000003e-05 2024-02-13T00:40:57.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330081+0000", 2024-02-13T00:40:57.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048545580000000001 2024-02-13T00:40:57.692 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.692 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.692 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.693 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:120 4.4 4:26625e58:lc::lc.26:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.693 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.330920+0000", 2024-02-13T00:40:57.693 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.310646225, 2024-02-13T00:40:57.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029032569999999998, 2024-02-13T00:40:57.693 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.693 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.693 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.693 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.694 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.694 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 120 2024-02-13T00:40:57.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.694 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330920+0000", 2024-02-13T00:40:57.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330920+0000", 2024-02-13T00:40:57.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:57.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330919+0000", 2024-02-13T00:40:57.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.519e-06 2024-02-13T00:40:57.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330924+0000", 2024-02-13T00:40:57.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3828999999999999e-05 2024-02-13T00:40:57.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330938+0000", 2024-02-13T00:40:57.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3839999999999999e-06 2024-02-13T00:40:57.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330945+0000", 2024-02-13T00:40:57.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5617999999999997e-05 2024-02-13T00:40:57.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330981+0000", 2024-02-13T00:40:57.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0856999999999998e-05 2024-02-13T00:40:57.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.331032+0000", 2024-02-13T00:40:57.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000125599 2024-02-13T00:40:57.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:40:57.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.331158+0000", 2024-02-13T00:40:57.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00048313400000000003 2024-02-13T00:40:57.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.331641+0000", 2024-02-13T00:40:57.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001766093 2024-02-13T00:40:57.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.333407+0000", 2024-02-13T00:40:57.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032840700000000002 2024-02-13T00:40:57.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.333735+0000", 2024-02-13T00:40:57.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2593999999999998e-05 2024-02-13T00:40:57.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.333788+0000", 2024-02-13T00:40:57.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5151000000000003e-05 2024-02-13T00:40:57.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.333823+0000", 2024-02-13T00:40:57.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029032569999999998 2024-02-13T00:40:57.701 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.701 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.701 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.702 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:121 4.7 4:e07a6548:gc::gc.15:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.702 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.330957+0000", 2024-02-13T00:40:57.702 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.31060916300001, 2024-02-13T00:40:57.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023945619999999998, 2024-02-13T00:40:57.702 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.702 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.702 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.702 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.702 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.703 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 121 2024-02-13T00:40:57.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.703 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330957+0000", 2024-02-13T00:40:57.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330957+0000", 2024-02-13T00:40:57.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:40:57.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330956+0000", 2024-02-13T00:40:57.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6780000000000002e-06 2024-02-13T00:40:57.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330960+0000", 2024-02-13T00:40:57.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0429999999999998e-06 2024-02-13T00:40:57.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330965+0000", 2024-02-13T00:40:57.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3150000000000002e-06 2024-02-13T00:40:57.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.330969+0000", 2024-02-13T00:40:57.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0108999999999998e-05 2024-02-13T00:40:57.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.331009+0000", 2024-02-13T00:40:57.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0952999999999996e-05 2024-02-13T00:40:57.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.331080+0000", 2024-02-13T00:40:57.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017450199999999999 2024-02-13T00:40:57.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-13T00:40:57.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.331255+0000", 2024-02-13T00:40:57.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040140299999999999 2024-02-13T00:40:57.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.331656+0000", 2024-02-13T00:40:57.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040183500000000002 2024-02-13T00:40:57.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.332058+0000", 2024-02-13T00:40:57.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012038890000000001 2024-02-13T00:40:57.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.333262+0000", 2024-02-13T00:40:57.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0024000000000001e-05 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.333312+0000", 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9379999999999999e-05 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.333351+0000", 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023945619999999998 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.709 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.710 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:151 4.4 4:25ca39bb:::obj_delete_at_hint.0000000005:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.710 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.340755+0000", 2024-02-13T00:40:57.710 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.300810821, 2024-02-13T00:40:57.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024476070000000001, 2024-02-13T00:40:57.710 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.710 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.710 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.710 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.711 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.711 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 151 2024-02-13T00:40:57.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.711 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340755+0000", 2024-02-13T00:40:57.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340755+0000", 2024-02-13T00:40:57.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:40:57.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340753+0000", 2024-02-13T00:40:57.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4009999999999999e-05 2024-02-13T00:40:57.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340778+0000", 2024-02-13T00:40:57.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5111e-05 2024-02-13T00:40:57.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340793+0000", 2024-02-13T00:40:57.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8589999999999999e-06 2024-02-13T00:40:57.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340799+0000", 2024-02-13T00:40:57.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3728999999999999e-05 2024-02-13T00:40:57.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340863+0000", 2024-02-13T00:40:57.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9655999999999999e-05 2024-02-13T00:40:57.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.340933+0000", 2024-02-13T00:40:57.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018593400000000001 2024-02-13T00:40:57.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:40:57.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.341119+0000", 2024-02-13T00:40:57.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013877130000000001 2024-02-13T00:40:57.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.342507+0000", 2024-02-13T00:40:57.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0489999999999998e-06 2024-02-13T00:40:57.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.342514+0000", 2024-02-13T00:40:57.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000607874 2024-02-13T00:40:57.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.343121+0000", 2024-02-13T00:40:57.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7203999999999997e-05 2024-02-13T00:40:57.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.343169+0000", 2024-02-13T00:40:57.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4014999999999999e-05 2024-02-13T00:40:57.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.343203+0000", 2024-02-13T00:40:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024476070000000001 2024-02-13T00:40:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:205 4.4 4:24c8e1ae:gc::gc.19:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.718 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.354593+0000", 2024-02-13T00:40:57.719 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.28697305200001, 2024-02-13T00:40:57.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025302879999999999, 2024-02-13T00:40:57.719 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.719 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.719 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.719 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.719 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.719 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 205 2024-02-13T00:40:57.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.719 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.354593+0000", 2024-02-13T00:40:57.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.354593+0000", 2024-02-13T00:40:57.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:57.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.354592+0000", 2024-02-13T00:40:57.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5480000000000003e-06 2024-02-13T00:40:57.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.354597+0000", 2024-02-13T00:40:57.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4450000000000004e-06 2024-02-13T00:40:57.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.354602+0000", 2024-02-13T00:40:57.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.495e-06 2024-02-13T00:40:57.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.354605+0000", 2024-02-13T00:40:57.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2295000000000002e-05 2024-02-13T00:40:57.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.354638+0000", 2024-02-13T00:40:57.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4454000000000003e-05 2024-02-13T00:40:57.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.354702+0000", 2024-02-13T00:40:57.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017062800000000001 2024-02-13T00:40:57.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:40:57.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.354873+0000", 2024-02-13T00:40:57.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00061640900000000005 2024-02-13T00:40:57.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.355489+0000", 2024-02-13T00:40:57.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9780000000000002e-06 2024-02-13T00:40:57.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.355495+0000", 2024-02-13T00:40:57.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00157942 2024-02-13T00:40:57.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.357075+0000", 2024-02-13T00:40:57.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8527999999999999e-05 2024-02-13T00:40:57.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.357103+0000", 2024-02-13T00:40:57.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9888999999999999e-05 2024-02-13T00:40:57.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.357123+0000", 2024-02-13T00:40:57.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025302879999999999 2024-02-13T00:40:57.726 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.726 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.726 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.726 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:249 4.4 4:264a8785:::obj_delete_at_hint.0000000011:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.726 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.364435+0000", 2024-02-13T00:40:57.727 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.27713043, 2024-02-13T00:40:57.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002573653, 2024-02-13T00:40:57.727 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.727 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.727 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 249 2024-02-13T00:40:57.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.727 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.364435+0000", 2024-02-13T00:40:57.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.364435+0000", 2024-02-13T00:40:57.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:57.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.364434+0000", 2024-02-13T00:40:57.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2979999999999997e-06 2024-02-13T00:40:57.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.364441+0000", 2024-02-13T00:40:57.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0163000000000001e-05 2024-02-13T00:40:57.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.364451+0000", 2024-02-13T00:40:57.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1799999999999999e-06 2024-02-13T00:40:57.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.364458+0000", 2024-02-13T00:40:57.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.318e-05 2024-02-13T00:40:57.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.364541+0000", 2024-02-13T00:40:57.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106242 2024-02-13T00:40:57.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.364647+0000", 2024-02-13T00:40:57.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019701000000000001 2024-02-13T00:40:57.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:40:57.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.364845+0000", 2024-02-13T00:40:57.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001321455 2024-02-13T00:40:57.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.366166+0000", 2024-02-13T00:40:57.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5148e-05 2024-02-13T00:40:57.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.366181+0000", 2024-02-13T00:40:57.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075143900000000001 2024-02-13T00:40:57.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.366933+0000", 2024-02-13T00:40:57.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3795000000000003e-05 2024-02-13T00:40:57.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.366976+0000", 2024-02-13T00:40:57.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2743000000000001e-05 2024-02-13T00:40:57.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.367009+0000", 2024-02-13T00:40:57.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002573653 2024-02-13T00:40:57.734 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.734 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.734 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.734 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24377.0:491 4.1d e64/58 4:bbf2d10c:::obj_delete_at_hint.0000000025:head v 64'10, mlcod=64'8)", 2024-02-13T00:40:57.734 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.420007+0000", 2024-02-13T00:40:57.734 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.221558767, 2024-02-13T00:40:57.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024247259999999999, 2024-02-13T00:40:57.735 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.735 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.735 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420007+0000", 2024-02-13T00:40:57.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420007+0000", 2024-02-13T00:40:57.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:40:57.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420007+0000", 2024-02-13T00:40:57.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0209999999999999e-06 2024-02-13T00:40:57.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420012+0000", 2024-02-13T00:40:57.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0719999999999996e-06 2024-02-13T00:40:57.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420018+0000", 2024-02-13T00:40:57.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2429999999999999e-06 2024-02-13T00:40:57.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420022+0000", 2024-02-13T00:40:57.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4260000000000001e-05 2024-02-13T00:40:57.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420056+0000", 2024-02-13T00:40:57.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9295000000000001e-05 2024-02-13T00:40:57.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420076+0000", 2024-02-13T00:40:57.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023269269999999999 2024-02-13T00:40:57.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.422402+0000", 2024-02-13T00:40:57.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9369999999999998e-05 2024-02-13T00:40:57.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.422432+0000", 2024-02-13T00:40:57.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024247259999999999 2024-02-13T00:40:57.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.739 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.740 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.740 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24377.0:492 4.9 e64/58 4:9413d7df:lc::lc.30:head v 64'15, mlcod=64'13)", 2024-02-13T00:40:57.740 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.420120+0000", 2024-02-13T00:40:57.740 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.22144587099999, 2024-02-13T00:40:57.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023111659999999999, 2024-02-13T00:40:57.740 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.740 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.741 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420120+0000", 2024-02-13T00:40:57.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420120+0000", 2024-02-13T00:40:57.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:40:57.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420120+0000", 2024-02-13T00:40:57.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.476e-06 2024-02-13T00:40:57.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420124+0000", 2024-02-13T00:40:57.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.789e-06 2024-02-13T00:40:57.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420130+0000", 2024-02-13T00:40:57.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.819e-06 2024-02-13T00:40:57.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420133+0000", 2024-02-13T00:40:57.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111149 2024-02-13T00:40:57.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420244+0000", 2024-02-13T00:40:57.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2019999999999999e-05 2024-02-13T00:40:57.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.420266+0000", 2024-02-13T00:40:57.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021403170000000001 2024-02-13T00:40:57.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.422406+0000", 2024-02-13T00:40:57.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4939e-05 2024-02-13T00:40:57.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.422431+0000", 2024-02-13T00:40:57.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023111659999999999 2024-02-13T00:40:57.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.745 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.746 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:501 4.f 4:f24cb4a8:gc::gc.31:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.746 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.421647+0000", 2024-02-13T00:40:57.746 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.219919049, 2024-02-13T00:40:57.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033041569999999998, 2024-02-13T00:40:57.746 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.746 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.746 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.746 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.746 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.746 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 501 2024-02-13T00:40:57.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.747 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421647+0000", 2024-02-13T00:40:57.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421647+0000", 2024-02-13T00:40:57.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:57.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421646+0000", 2024-02-13T00:40:57.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3370000000000006e-06 2024-02-13T00:40:57.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421655+0000", 2024-02-13T00:40:57.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4049999999999999e-06 2024-02-13T00:40:57.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421661+0000", 2024-02-13T00:40:57.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1700000000000001e-06 2024-02-13T00:40:57.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421664+0000", 2024-02-13T00:40:57.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0837999999999998e-05 2024-02-13T00:40:57.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421695+0000", 2024-02-13T00:40:57.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3269999999999997e-05 2024-02-13T00:40:57.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421738+0000", 2024-02-13T00:40:57.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104559 2024-02-13T00:40:57.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,4", 2024-02-13T00:40:57.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421843+0000", 2024-02-13T00:40:57.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0004305 2024-02-13T00:40:57.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.422273+0000", 2024-02-13T00:40:57.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000346623 2024-02-13T00:40:57.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.422620+0000", 2024-02-13T00:40:57.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022591830000000001 2024-02-13T00:40:57.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.424879+0000", 2024-02-13T00:40:57.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9755999999999997e-05 2024-02-13T00:40:57.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.424919+0000", 2024-02-13T00:40:57.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2252999999999998e-05 2024-02-13T00:40:57.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.424951+0000", 2024-02-13T00:40:57.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033041569999999998 2024-02-13T00:40:57.753 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.753 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.753 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.754 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24377.0:502 4.2 4:4455f95b:lc::lc.27:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:40:57.754 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.421793+0000", 2024-02-13T00:40:57.754 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.21977280199999, 2024-02-13T00:40:57.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031272330000000001, 2024-02-13T00:40:57.754 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.754 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.754 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.754 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24377", 2024-02-13T00:40:57.754 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:40:57.755 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 502 2024-02-13T00:40:57.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.755 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421793+0000", 2024-02-13T00:40:57.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421793+0000", 2024-02-13T00:40:57.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:40:57.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421792+0000", 2024-02-13T00:40:57.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4719999999999997e-06 2024-02-13T00:40:57.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421797+0000", 2024-02-13T00:40:57.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7610000000000002e-06 2024-02-13T00:40:57.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421804+0000", 2024-02-13T00:40:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.067e-06 2024-02-13T00:40:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421807+0000", 2024-02-13T00:40:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8371999999999998e-05 2024-02-13T00:40:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421845+0000", 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4745000000000003e-05 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.421920+0000", 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018692000000000001 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.422107+0000", 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00043931100000000001 2024-02-13T00:40:57.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.422546+0000", 2024-02-13T00:40:57.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002288296 2024-02-13T00:40:57.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.424834+0000", 2024-02-13T00:40:57.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5189999999999998e-06 2024-02-13T00:40:57.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.424841+0000", 2024-02-13T00:40:57.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4790999999999999e-05 2024-02-13T00:40:57.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.424886+0000", 2024-02-13T00:40:57.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4666000000000003e-05 2024-02-13T00:40:57.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.424920+0000", 2024-02-13T00:40:57.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031272330000000001 2024-02-13T00:40:57.761 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.761 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.761 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.763 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24377.0:528 4.9 e64/58 4:9195808b:::obj_delete_at_hint.0000000027:head v 64'17, mlcod=64'15)", 2024-02-13T00:40:57.763 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.430753+0000", 2024-02-13T00:40:57.763 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.210812363, 2024-02-13T00:40:57.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031940459999999999, 2024-02-13T00:40:57.763 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.430753+0000", 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.430753+0000", 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.430752+0000", 2024-02-13T00:40:57.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5801e-05 2024-02-13T00:40:57.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.430768+0000", 2024-02-13T00:40:57.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6860000000000001e-05 2024-02-13T00:40:57.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.430785+0000", 2024-02-13T00:40:57.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6599999999999995e-06 2024-02-13T00:40:57.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.430793+0000", 2024-02-13T00:40:57.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9765999999999997e-05 2024-02-13T00:40:57.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.430852+0000", 2024-02-13T00:40:57.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0386000000000001e-05 2024-02-13T00:40:57.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.430873+0000", 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030406019999999999 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.433913+0000", 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4069999999999997e-05 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.433948+0000", 2024-02-13T00:40:57.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031940459999999999 2024-02-13T00:40:57.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.768 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.768 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24377.0:529 4.e e64/58 4:71a6e5bc:lc::lc.2:head v 64'4, mlcod=64'2)", 2024-02-13T00:40:57.768 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:48.431288+0000", 2024-02-13T00:40:57.768 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.210277773, 2024-02-13T00:40:57.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028634580000000001, 2024-02-13T00:40:57.768 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.768 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.769 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.431288+0000", 2024-02-13T00:40:57.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.431288+0000", 2024-02-13T00:40:57.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:57.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.431287+0000", 2024-02-13T00:40:57.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2370000000000003e-06 2024-02-13T00:40:57.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.431295+0000", 2024-02-13T00:40:57.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5470000000000002e-06 2024-02-13T00:40:57.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.431304+0000", 2024-02-13T00:40:57.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0386e-05 2024-02-13T00:40:57.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.431315+0000", 2024-02-13T00:40:57.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4976999999999999e-05 2024-02-13T00:40:57.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.431369+0000", 2024-02-13T00:40:57.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9099000000000001e-05 2024-02-13T00:40:57.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.431389+0000", 2024-02-13T00:40:57.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027329730000000001 2024-02-13T00:40:57.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.434122+0000", 2024-02-13T00:40:57.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9986000000000001e-05 2024-02-13T00:40:57.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:48.434152+0000", 2024-02-13T00:40:57.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028634580000000001 2024-02-13T00:40:57.773 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.773 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.773 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.774 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24403.0:1 3.0 3:353ec8f9:::gateway.conf:head [create] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-13T00:40:57.774 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:37:53.316961+0000", 2024-02-13T00:40:57.774 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 184.32460496300001, 2024-02-13T00:40:57.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041429509999999998, 2024-02-13T00:40:57.774 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.774 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:40:57.774 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:40:57.774 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24403", 2024-02-13T00:40:57.774 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.181:0/295858269", 2024-02-13T00:40:57.775 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:40:57.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.775 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.316961+0000", 2024-02-13T00:40:57.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.316961+0000", 2024-02-13T00:40:57.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:40:57.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.316959+0000", 2024-02-13T00:40:57.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4780000000000002e-06 2024-02-13T00:40:57.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.316967+0000", 2024-02-13T00:40:57.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6560000000000004e-06 2024-02-13T00:40:57.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.316976+0000", 2024-02-13T00:40:57.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0600000000000008e-06 2024-02-13T00:40:57.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.316984+0000", 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4240000000000002e-05 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.317039+0000", 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.878e-06 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.317046+0000", 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021227970000000001 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.319168+0000", 2024-02-13T00:40:57.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120693 2024-02-13T00:40:57.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.319289+0000", 2024-02-13T00:40:57.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012392699999999999 2024-02-13T00:40:57.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,6", 2024-02-13T00:40:57.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.319413+0000", 2024-02-13T00:40:57.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011017220000000001 2024-02-13T00:40:57.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.320515+0000", 2024-02-13T00:40:57.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020539700000000001 2024-02-13T00:40:57.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:40:57.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.320720+0000", 2024-02-13T00:40:57.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029719600000000001 2024-02-13T00:40:57.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:40:57.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.321017+0000", 2024-02-13T00:40:57.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0025000000000003e-05 2024-02-13T00:40:57.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:40:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.321067+0000", 2024-02-13T00:40:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6498999999999997e-05 2024-02-13T00:40:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:37:53.321104+0000", 2024-02-13T00:40:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041429509999999998 2024-02-13T00:40:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.783 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(osd.1.0:99 4.4 e80/58 ondisk, result = 0)", 2024-02-13T00:40:57.783 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:38:57.533049+0000", 2024-02-13T00:40:57.783 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 120.108516495, 2024-02-13T00:40:57.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024019610000000002, 2024-02-13T00:40:57.783 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.783 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:40:57.783 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.533049+0000", 2024-02-13T00:40:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.533049+0000", 2024-02-13T00:40:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:40:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.533049+0000", 2024-02-13T00:40:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0030000000000002e-06 2024-02-13T00:40:57.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.533054+0000", 2024-02-13T00:40:57.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0189999999999995e-06 2024-02-13T00:40:57.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.533063+0000", 2024-02-13T00:40:57.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2100000000000003e-06 2024-02-13T00:40:57.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.533067+0000", 2024-02-13T00:40:57.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023726559999999999 2024-02-13T00:40:57.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.535440+0000", 2024-02-13T00:40:57.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7450000000000003e-06 2024-02-13T00:40:57.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.535444+0000", 2024-02-13T00:40:57.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0330000000000003e-06 2024-02-13T00:40:57.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.535451+0000", 2024-02-13T00:40:57.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024019610000000002 2024-02-13T00:40:57.787 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.788 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.788 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.788 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(osd.1.0:100 4.4 e80/58 ondisk, result = 0)", 2024-02-13T00:40:57.788 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:38:57.533200+0000", 2024-02-13T00:40:57.788 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 120.10836632900001, 2024-02-13T00:40:57.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023142839999999998, 2024-02-13T00:40:57.788 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:40:57.789 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:40:57.789 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:40:57.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:40:57.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.533200+0000", 2024-02-13T00:40:57.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:40:57.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:40:57.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.533200+0000", 2024-02-13T00:40:57.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:40:57.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:40:57.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.533199+0000", 2024-02-13T00:40:57.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7759999999999997e-06 2024-02-13T00:40:57.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:40:57.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.533204+0000", 2024-02-13T00:40:57.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0845999999999999e-05 2024-02-13T00:40:57.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:40:57.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.533214+0000", 2024-02-13T00:40:57.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4269999999999999e-06 2024-02-13T00:40:57.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:40:57.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.533220+0000", 2024-02-13T00:40:57.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002256901 2024-02-13T00:40:57.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:40:57.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.535477+0000", 2024-02-13T00:40:57.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3000000000000004e-06 2024-02-13T00:40:57.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:40:57.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.535484+0000", 2024-02-13T00:40:57.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9751999999999999e-05 2024-02-13T00:40:57.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:40:57.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:40:57.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:40:57.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:38:57.535514+0000", 2024-02-13T00:40:57.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023142839999999998 2024-02-13T00:40:57.793 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.793 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.793 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.793 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:40:57.793 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:40:57.793 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:40:57.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:54.199886+0000 mgr.y (mgr.14544) 130 : cluster [DBG] pgmap v122: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-02-13T00:40:57.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:54.801060+0000 mon.b (mon.1) 28 : audit [DBG] from='client.? 172.21.15.162:0/3066705702' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-13T00:40:57.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:55.040408+0000 mon.c (mon.2) 89 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:40:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:55.074049+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? 172.21.15.162:0/293721234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:40:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:55.289834+0000 mon.a (mon.0) 757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:40:57.986 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:40:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:57.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:40:58.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:58.241 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:58.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:58.256 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:58.346 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:58.376 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:58.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-13T00:40:58.823 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:58.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:58.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:40:58.953 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:59.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:40:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:40:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:40:59.058 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:40:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:59.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:40:59.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:40:59.198 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:59.621 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:40:59.648 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:40:59.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:55.302253+0000 mon.b (mon.1) 30 : audit [INF] from='client.? 172.21.15.162:0/579994674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-83977-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:40:59.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:55.305797+0000 mon.b (mon.1) 31 : audit [INF] from='client.? 172.21.15.162:0/2984140146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:40:59.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:55.417381+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.d scrub starts 2024-02-13T00:40:59.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:55.418525+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.d scrub ok 2024-02-13T00:40:59.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:55.543973+0000 mon.c (mon.2) 90 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:40:59.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:56.432419+0000 osd.2 (osd.2) 51 : cluster [DBG] 4.d scrub starts 2024-02-13T00:40:59.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:56.434502+0000 osd.2 (osd.2) 52 : cluster [DBG] 4.d scrub ok 2024-02-13T00:40:59.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:57.202131+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-13T00:40:59.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:57.455634+0000 mon.a (mon.0) 759 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:40:59.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:55.302253+0000 mon.b (mon.1) 30 : audit [INF] from='client.? 172.21.15.162:0/579994674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-83977-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:40:59.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:55.305797+0000 mon.b (mon.1) 31 : audit [INF] from='client.? 172.21.15.162:0/2984140146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:40:59.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:55.417381+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.d scrub starts 2024-02-13T00:40:59.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:55.418525+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.d scrub ok 2024-02-13T00:40:59.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:55.543973+0000 mon.c (mon.2) 90 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:40:59.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:56.432419+0000 osd.2 (osd.2) 51 : cluster [DBG] 4.d scrub starts 2024-02-13T00:40:59.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:56.434502+0000 osd.2 (osd.2) 52 : cluster [DBG] 4.d scrub ok 2024-02-13T00:40:59.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:57.202131+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-13T00:40:59.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:40:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:57.455634+0000 mon.a (mon.0) 759 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:40:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:59.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:40:59.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:59.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:55.302253+0000 mon.b (mon.1) 30 : audit [INF] from='client.? 172.21.15.162:0/579994674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-83977-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:40:59.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:55.305797+0000 mon.b (mon.1) 31 : audit [INF] from='client.? 172.21.15.162:0/2984140146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:40:59.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:55.417381+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.d scrub starts 2024-02-13T00:40:59.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:55.418525+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.d scrub ok 2024-02-13T00:40:59.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:55.543973+0000 mon.c (mon.2) 90 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:40:59.784 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:56.432419+0000 osd.2 (osd.2) 51 : cluster [DBG] 4.d scrub starts 2024-02-13T00:40:59.784 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:56.434502+0000 osd.2 (osd.2) 52 : cluster [DBG] 4.d scrub ok 2024-02-13T00:40:59.784 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:57.202131+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-13T00:40:59.784 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:40:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:57.455634+0000 mon.a (mon.0) 759 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:40:59.785 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:59.807 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:59.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:59.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:59.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:40:59.945 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:40:59.962 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:40:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:40:59.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:40:59.989 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:41:00.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:41:00.094 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:00.113 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:41:00.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:00.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:41:00.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:41:00.243 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:00.261 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:41:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:00.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:41:00.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:00.390 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:00.445 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:00.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:00.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:00.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:00.575 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:00.593 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:00.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:00.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:00.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:00.721 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:55.314513+0000 mgr.y (mgr.14544) 131 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:00.721 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:57.458042+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.1 deep-scrub starts 2024-02-13T00:41:00.721 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:57.464980+0000 mon.c (mon.2) 91 : audit [INF] from='client.? 172.21.15.162:0/4011549658' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-84105-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:57.467636+0000 mon.b (mon.1) 32 : audit [INF] from='client.? 172.21.15.162:0/832124837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-84025-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:57.469012+0000 mon.b (mon.1) 33 : audit [INF] from='client.? 172.21.15.162:0/3590368593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-84057-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:57.469221+0000 mon.c (mon.2) 92 : audit [INF] from='client.? 172.21.15.162:0/2472368574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-84161-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:57.469736+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.1 deep-scrub ok 2024-02-13T00:41:00.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:57.470403+0000 mon.b (mon.1) 34 : audit [INF] from='client.? 172.21.15.162:0/3785255474' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-83985-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:57.473023+0000 mon.b (mon.1) 35 : audit [INF] from='client.? 172.21.15.162:0/1679583445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:57.473224+0000 mon.b (mon.1) 36 : audit [INF] from='client.? 172.21.15.162:0/1035949604' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-84097-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:57.949550+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:41:00.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:57.950480+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:41:00.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:59.158932+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-13T00:41:00.722 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:59.160120+0000 mon.c (mon.2) 93 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:55.314513+0000 mgr.y (mgr.14544) 131 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:57.458042+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.1 deep-scrub starts 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:57.464980+0000 mon.c (mon.2) 91 : audit [INF] from='client.? 172.21.15.162:0/4011549658' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-84105-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:57.467636+0000 mon.b (mon.1) 32 : audit [INF] from='client.? 172.21.15.162:0/832124837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-84025-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:57.469012+0000 mon.b (mon.1) 33 : audit [INF] from='client.? 172.21.15.162:0/3590368593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-84057-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:57.469221+0000 mon.c (mon.2) 92 : audit [INF] from='client.? 172.21.15.162:0/2472368574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-84161-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:57.469736+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.1 deep-scrub ok 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:57.470403+0000 mon.b (mon.1) 34 : audit [INF] from='client.? 172.21.15.162:0/3785255474' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-83985-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:57.473023+0000 mon.b (mon.1) 35 : audit [INF] from='client.? 172.21.15.162:0/1679583445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:57.473224+0000 mon.b (mon.1) 36 : audit [INF] from='client.? 172.21.15.162:0/1035949604' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-84097-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:57.949550+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:57.950480+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:41:00.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:59.158932+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-13T00:41:00.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:59.160120+0000 mon.c (mon.2) 93 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:00.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:59.433684+0000 mon.b (mon.1) 37 : audit [INF] from='client.? 172.21.15.162:0/3029464538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-84073-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:59.434073+0000 mon.b (mon.1) 38 : audit [INF] from='client.? 172.21.15.162:0/530375479' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-84081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:59.434323+0000 mon.b (mon.1) 39 : audit [INF] from='client.? 172.21.15.162:0/90739862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-84089-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:40:59.444607+0000 mon.b (mon.1) 40 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-84017-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:59.928716+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-13T00:41:00.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:59.433684+0000 mon.b (mon.1) 37 : audit [INF] from='client.? 172.21.15.162:0/3029464538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-84073-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.725 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:59.434073+0000 mon.b (mon.1) 38 : audit [INF] from='client.? 172.21.15.162:0/530375479' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-84081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.725 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:59.434323+0000 mon.b (mon.1) 39 : audit [INF] from='client.? 172.21.15.162:0/90739862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-84089-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.725 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:40:59.444607+0000 mon.b (mon.1) 40 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-84017-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.725 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:59.928716+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-13T00:41:00.725 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:55.314513+0000 mgr.y (mgr.14544) 131 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:57.458042+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.1 deep-scrub starts 2024-02-13T00:41:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:57.464980+0000 mon.c (mon.2) 91 : audit [INF] from='client.? 172.21.15.162:0/4011549658' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-84105-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:57.467636+0000 mon.b (mon.1) 32 : audit [INF] from='client.? 172.21.15.162:0/832124837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-84025-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:57.469012+0000 mon.b (mon.1) 33 : audit [INF] from='client.? 172.21.15.162:0/3590368593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-84057-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:57.469221+0000 mon.c (mon.2) 92 : audit [INF] from='client.? 172.21.15.162:0/2472368574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-84161-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:57.469736+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.1 deep-scrub ok 2024-02-13T00:41:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:57.470403+0000 mon.b (mon.1) 34 : audit [INF] from='client.? 172.21.15.162:0/3785255474' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-83985-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:57.473023+0000 mon.b (mon.1) 35 : audit [INF] from='client.? 172.21.15.162:0/1679583445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:57.473224+0000 mon.b (mon.1) 36 : audit [INF] from='client.? 172.21.15.162:0/1035949604' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-84097-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:57.949550+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:41:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:57.950480+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:41:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:59.158932+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-13T00:41:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:59.160120+0000 mon.c (mon.2) 93 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:59.433684+0000 mon.b (mon.1) 37 : audit [INF] from='client.? 172.21.15.162:0/3029464538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-84073-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:59.434073+0000 mon.b (mon.1) 38 : audit [INF] from='client.? 172.21.15.162:0/530375479' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-84081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:59.434323+0000 mon.b (mon.1) 39 : audit [INF] from='client.? 172.21.15.162:0/90739862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-84089-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:40:59.444607+0000 mon.b (mon.1) 40 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-84017-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:59.928716+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-13T00:41:00.848 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:00.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:00.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:00.980 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:01.306 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:01.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:01.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:01.437 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:01.541 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:01.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:01.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:01.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:55.355460+0000 osd.6 (osd.6) 43 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:41:01.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:55.358360+0000 osd.6 (osd.6) 44 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:41:01.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:56.200253+0000 mgr.y (mgr.14544) 132 : cluster [DBG] pgmap v124: 196 pgs: 64 unknown, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-02-13T00:41:01.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:57.313492+0000 osd.6 (osd.6) 45 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:41:01.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:57.316396+0000 osd.6 (osd.6) 46 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:41:01.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:58.175977+0000 osd.7 (osd.7) 58 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:41:01.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:58.179442+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:41:01.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:58.200751+0000 mgr.y (mgr.14544) 133 : cluster [DBG] pgmap v126: 484 pgs: 352 unknown, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:41:01.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:58.331790+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:41:01.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:58.334695+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:41:01.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:58.540292+0000 osd.5 (osd.5) 61 : cluster [DBG] 6.d deep-scrub starts 2024-02-13T00:41:01.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:58.541139+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.d deep-scrub ok 2024-02-13T00:41:01.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:59.163476+0000 osd.7 (osd.7) 60 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:59.164683+0000 osd.7 (osd.7) 61 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:59.464747+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:59.468517+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:59.515879+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:59.518771+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:59.878363+0000 osd.3 (osd.3) 79 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:40:59.881252+0000 osd.3 (osd.3) 80 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:00.125109+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.b scrub starts 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:00.128588+0000 osd.7 (osd.7) 63 : cluster [DBG] 2.b scrub ok 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:00.202603+0000 mgr.y (mgr.14544) 134 : cluster [DBG] pgmap v128: 740 pgs: 569 creating+peering, 171 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.453347+0000 mon.a (mon.0) 762 : audit [INF] from='client.? 172.21.15.162:0/79393202' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-84033-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.453600+0000 mon.a (mon.0) 763 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.453821+0000 mon.a (mon.0) 764 : audit [INF] from='client.? 172.21.15.162:0/1224070377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-83993-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.454041+0000 mon.a (mon.0) 765 : audit [INF] from='client.? 172.21.15.162:0/3501757973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-84001-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.455106+0000 mon.a (mon.0) 766 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-84105-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:00.466149+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-13T00:41:01.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.482924+0000 mon.b (mon.1) 41 : audit [INF] from='client.? 172.21.15.162:0/4028053558' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84129-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.483300+0000 mon.a (mon.0) 768 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-84161-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.483949+0000 mon.a (mon.0) 769 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-83977-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.484306+0000 mon.a (mon.0) 770 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.487036+0000 mon.a (mon.0) 771 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-84025-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.487185+0000 mon.a (mon.0) 772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.487561+0000 mon.a (mon.0) 773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-84057-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.487859+0000 mon.a (mon.0) 774 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-83985-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.499068+0000 mon.a (mon.0) 775 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.520898+0000 mon.b (mon.1) 42 : audit [DBG] from='client.? 172.21.15.162:0/2606010176' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.761866+0000 mon.a (mon.0) 776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-84097-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.763705+0000 mon.a (mon.0) 777 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-84073-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.764104+0000 mon.a (mon.0) 778 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-84081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.764498+0000 mon.a (mon.0) 779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-84089-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.765085+0000 mon.a (mon.0) 780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-84017-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:00.766947+0000 mon.a (mon.0) 781 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84129-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:00.878361+0000 osd.0 (osd.0) 53 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:41:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:00.881408+0000 osd.0 (osd.0) 54 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:41:01.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:55.355460+0000 osd.6 (osd.6) 43 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:41:01.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:55.358360+0000 osd.6 (osd.6) 44 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:41:01.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:56.200253+0000 mgr.y (mgr.14544) 132 : cluster [DBG] pgmap v124: 196 pgs: 64 unknown, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-02-13T00:41:01.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:57.313492+0000 osd.6 (osd.6) 45 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:41:01.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:57.316396+0000 osd.6 (osd.6) 46 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:41:01.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:58.175977+0000 osd.7 (osd.7) 58 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:41:01.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:58.179442+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:41:01.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:58.200751+0000 mgr.y (mgr.14544) 133 : cluster [DBG] pgmap v126: 484 pgs: 352 unknown, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:41:01.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:58.331790+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:41:01.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:58.334695+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:58.540292+0000 osd.5 (osd.5) 61 : cluster [DBG] 6.d deep-scrub starts 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:58.541139+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.d deep-scrub ok 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:59.163476+0000 osd.7 (osd.7) 60 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:59.164683+0000 osd.7 (osd.7) 61 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:59.464747+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:59.468517+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:59.515879+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:59.518771+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:59.878363+0000 osd.3 (osd.3) 79 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:40:59.881252+0000 osd.3 (osd.3) 80 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:00.125109+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.b scrub starts 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:00.128588+0000 osd.7 (osd.7) 63 : cluster [DBG] 2.b scrub ok 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:00.202603+0000 mgr.y (mgr.14544) 134 : cluster [DBG] pgmap v128: 740 pgs: 569 creating+peering, 171 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:41:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.453347+0000 mon.a (mon.0) 762 : audit [INF] from='client.? 172.21.15.162:0/79393202' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-84033-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.453600+0000 mon.a (mon.0) 763 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.453821+0000 mon.a (mon.0) 764 : audit [INF] from='client.? 172.21.15.162:0/1224070377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-83993-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.454041+0000 mon.a (mon.0) 765 : audit [INF] from='client.? 172.21.15.162:0/3501757973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-84001-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.455106+0000 mon.a (mon.0) 766 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-84105-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:00.466149+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.482924+0000 mon.b (mon.1) 41 : audit [INF] from='client.? 172.21.15.162:0/4028053558' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84129-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.483300+0000 mon.a (mon.0) 768 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-84161-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.483949+0000 mon.a (mon.0) 769 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-83977-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.484306+0000 mon.a (mon.0) 770 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.487036+0000 mon.a (mon.0) 771 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-84025-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.487185+0000 mon.a (mon.0) 772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.487561+0000 mon.a (mon.0) 773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-84057-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.487859+0000 mon.a (mon.0) 774 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-83985-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.499068+0000 mon.a (mon.0) 775 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.520898+0000 mon.b (mon.1) 42 : audit [DBG] from='client.? 172.21.15.162:0/2606010176' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-13T00:41:01.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.761866+0000 mon.a (mon.0) 776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-84097-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.763705+0000 mon.a (mon.0) 777 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-84073-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.764104+0000 mon.a (mon.0) 778 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-84081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.764498+0000 mon.a (mon.0) 779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-84089-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.765085+0000 mon.a (mon.0) 780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-84017-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:00.766947+0000 mon.a (mon.0) 781 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84129-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:00.878361+0000 osd.0 (osd.0) 53 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:41:01.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:00.881408+0000 osd.0 (osd.0) 54 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:41:01.648 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:01.685 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:01.724 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:01.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:01.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:55.355460+0000 osd.6 (osd.6) 43 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:41:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:55.358360+0000 osd.6 (osd.6) 44 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:41:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:56.200253+0000 mgr.y (mgr.14544) 132 : cluster [DBG] pgmap v124: 196 pgs: 64 unknown, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-02-13T00:41:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:57.313492+0000 osd.6 (osd.6) 45 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:41:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:57.316396+0000 osd.6 (osd.6) 46 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:41:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:58.175977+0000 osd.7 (osd.7) 58 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:41:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:58.179442+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:41:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:58.200751+0000 mgr.y (mgr.14544) 133 : cluster [DBG] pgmap v126: 484 pgs: 352 unknown, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:41:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:58.331790+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:41:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:58.334695+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:41:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:58.540292+0000 osd.5 (osd.5) 61 : cluster [DBG] 6.d deep-scrub starts 2024-02-13T00:41:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:58.541139+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.d deep-scrub ok 2024-02-13T00:41:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:59.163476+0000 osd.7 (osd.7) 60 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:59.164683+0000 osd.7 (osd.7) 61 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:59.464747+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:59.468517+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:59.515879+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:59.518771+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:59.878363+0000 osd.3 (osd.3) 79 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:40:59.881252+0000 osd.3 (osd.3) 80 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:00.125109+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.b scrub starts 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:00.128588+0000 osd.7 (osd.7) 63 : cluster [DBG] 2.b scrub ok 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:00.202603+0000 mgr.y (mgr.14544) 134 : cluster [DBG] pgmap v128: 740 pgs: 569 creating+peering, 171 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.453347+0000 mon.a (mon.0) 762 : audit [INF] from='client.? 172.21.15.162:0/79393202' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-84033-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.453600+0000 mon.a (mon.0) 763 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.453821+0000 mon.a (mon.0) 764 : audit [INF] from='client.? 172.21.15.162:0/1224070377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-83993-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.454041+0000 mon.a (mon.0) 765 : audit [INF] from='client.? 172.21.15.162:0/3501757973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-84001-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.455106+0000 mon.a (mon.0) 766 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-84105-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:00.466149+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.482924+0000 mon.b (mon.1) 41 : audit [INF] from='client.? 172.21.15.162:0/4028053558' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84129-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.483300+0000 mon.a (mon.0) 768 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-84161-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.483949+0000 mon.a (mon.0) 769 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-83977-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.484306+0000 mon.a (mon.0) 770 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.487036+0000 mon.a (mon.0) 771 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-84025-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.487185+0000 mon.a (mon.0) 772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.487561+0000 mon.a (mon.0) 773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-84057-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.487859+0000 mon.a (mon.0) 774 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-83985-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.499068+0000 mon.a (mon.0) 775 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.520898+0000 mon.b (mon.1) 42 : audit [DBG] from='client.? 172.21.15.162:0/2606010176' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.761866+0000 mon.a (mon.0) 776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-84097-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.763705+0000 mon.a (mon.0) 777 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-84073-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.764104+0000 mon.a (mon.0) 778 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-84081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.764498+0000 mon.a (mon.0) 779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-84089-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.765085+0000 mon.a (mon.0) 780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-84017-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:00.766947+0000 mon.a (mon.0) 781 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84129-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:01.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:00.878361+0000 osd.0 (osd.0) 53 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:41:01.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:00.881408+0000 osd.0 (osd.0) 54 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:41:01.806 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-02-13T00:41:01.806 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [----------] Global test environment set-up. 2024-02-13T00:41:01.806 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [----------] 12 tests from AsioRados 2024-02-13T00:41:01.806 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-02-13T00:41:01.806 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2024-02-13T00:41:01.806 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-02-13T00:41:01.806 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (2 ms) 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (4 ms) 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (4 ms) 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (2 ms) 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (2 ms) 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-02-13T00:41:01.807 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-02-13T00:41:01.808 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-02-13T00:41:01.808 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (4 ms) 2024-02-13T00:41:01.808 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-02-13T00:41:01.808 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (7 ms) 2024-02-13T00:41:01.808 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-02-13T00:41:01.808 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (3 ms) 2024-02-13T00:41:01.808 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [----------] 12 tests from AsioRados (31 ms total) 2024-02-13T00:41:01.808 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: 2024-02-13T00:41:01.808 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [----------] Global test environment tear-down 2024-02-13T00:41:01.808 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (6763 ms total) 2024-02-13T00:41:01.808 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ PASSED ] 12 tests. 2024-02-13T00:41:01.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:01.865 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:01.889 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:41:01.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:01.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:41:01.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:02.032 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:02.056 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:41:02.056 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:41:02.056 INFO:tasks.thrashosds.thrasher:Growing pool default.rgw.meta 2024-02-13T00:41:02.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:41:02.102 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:02.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:02.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:02.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.141+0000 7f603664a700 1 -- 172.21.15.162:0/1221374088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60300f5d30 msgr2=0x7f60300f6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:02.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.141+0000 7f603664a700 1 --2- 172.21.15.162:0/1221374088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60300f5d30 0x7f60300f6110 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f60200047b0 tx=0x7f6020009cb0).stop 2024-02-13T00:41:02.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.142+0000 7f603664a700 1 -- 172.21.15.162:0/1221374088 shutdown_connections 2024-02-13T00:41:02.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.142+0000 7f603664a700 1 --2- 172.21.15.162:0/1221374088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6030108a00 0x7f603010adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:02.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.142+0000 7f603664a700 1 --2- 172.21.15.162:0/1221374088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60300f5d30 0x7f60300f6110 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:02.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.142+0000 7f603664a700 1 --2- 172.21.15.162:0/1221374088 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60300f6600 0x7f6030108510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:02.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.142+0000 7f603664a700 1 -- 172.21.15.162:0/1221374088 >> 172.21.15.162:0/1221374088 conn(0x7f603006fc50 msgr2=0x7f60300f4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:02.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.142+0000 7f603664a700 1 -- 172.21.15.162:0/1221374088 shutdown_connections 2024-02-13T00:41:02.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.142+0000 7f603664a700 1 -- 172.21.15.162:0/1221374088 wait complete. 2024-02-13T00:41:02.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.142+0000 7f603664a700 1 librados: starting msgr at 2024-02-13T00:41:02.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.142+0000 7f603664a700 1 librados: starting objecter 2024-02-13T00:41:02.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f603664a700 1 Processor -- start 2024-02-13T00:41:02.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f603664a700 1 -- start start 2024-02-13T00:41:02.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f603664a700 1 librados: setting wanted keys 2024-02-13T00:41:02.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f603664a700 1 librados: calling monclient init 2024-02-13T00:41:02.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f603664a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60300f5d30 0x7f603006a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:02.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f603664a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60300f6600 0x7f603006a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:02.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f603664a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6030108a00 0x7f6030063610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:02.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f603664a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6030063cd0 con 0x7f60300f6600 2024-02-13T00:41:02.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f603664a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6030063e40 con 0x7f6030108a00 2024-02-13T00:41:02.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f603664a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6030064000 con 0x7f60300f5d30 2024-02-13T00:41:02.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f602f7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60300f6600 0x7f603006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:02.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f602f7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60300f6600 0x7f603006a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35282/0 (socket says 172.21.15.162:35282) 2024-02-13T00:41:02.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f602f7fe700 1 -- 172.21.15.162:0/1514900161 learned_addr learned my addr 172.21.15.162:0/1514900161 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:02.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.143+0000 7f6034be7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6030108a00 0x7f6030063610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:02.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.144+0000 7f602ffff700 1 --2- 172.21.15.162:0/1514900161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60300f5d30 0x7f603006a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:02.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.144+0000 7f602f7fe700 1 -- 172.21.15.162:0/1514900161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60300f5d30 msgr2=0x7f603006a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:02.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.144+0000 7f602f7fe700 1 --2- 172.21.15.162:0/1514900161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60300f5d30 0x7f603006a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:02.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.144+0000 7f602f7fe700 1 -- 172.21.15.162:0/1514900161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6030108a00 msgr2=0x7f6030063610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:02.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.144+0000 7f602f7fe700 1 --2- 172.21.15.162:0/1514900161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6030108a00 0x7f6030063610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:02.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.144+0000 7f602f7fe700 1 -- 172.21.15.162:0/1514900161 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6020005040 con 0x7f60300f6600 2024-02-13T00:41:02.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.144+0000 7f602ffff700 1 --2- 172.21.15.162:0/1514900161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60300f5d30 0x7f603006a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:41:02.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.144+0000 7f602f7fe700 1 --2- 172.21.15.162:0/1514900161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60300f6600 0x7f603006a960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f602800ebf0 tx=0x7f602800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:02.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.144+0000 7f602d7fa700 10 client.?.objecter ms_handle_connect 0x7f60300f6600 2024-02-13T00:41:02.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.144+0000 7f602d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:02.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.145+0000 7f602d7fa700 1 -- 172.21.15.162:0/1514900161 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028009860 con 0x7f60300f6600 2024-02-13T00:41:02.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.145+0000 7f602d7fa700 1 -- 172.21.15.162:0/1514900161 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f602800f040 con 0x7f60300f6600 2024-02-13T00:41:02.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.146+0000 7f603664a700 1 -- 172.21.15.162:0/1514900161 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60300642e0 con 0x7f60300f6600 2024-02-13T00:41:02.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.146+0000 7f602d7fa700 1 -- 172.21.15.162:0/1514900161 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602800c9b0 con 0x7f60300f6600 2024-02-13T00:41:02.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.146+0000 7f603664a700 10 client.15063.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:02.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.146+0000 7f603664a700 1 -- 172.21.15.162:0/1514900161 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6030068290 con 0x7f60300f6600 2024-02-13T00:41:02.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.146+0000 7f603664a700 1 librados: init done 2024-02-13T00:41:02.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.148+0000 7f602d7fa700 1 -- 172.21.15.162:0/1514900161 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f60280092e0 con 0x7f60300f6600 2024-02-13T00:41:02.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.148+0000 7f603664a700 1 -- 172.21.15.162:0/1514900161 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f601c004f50 con 0x7f60300f6600 2024-02-13T00:41:02.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.148+0000 7f602d7fa700 1 --2- 172.21.15.162:0/1514900161 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6018062ce0 0x7f60180651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:02.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.148+0000 7f602d7fa700 1 -- 172.21.15.162:0/1514900161 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(107..107 src has 52..107) v4 ==== 16015+0+0 (secure 0 0 0) 0x7f602807ed80 con 0x7f60300f6600 2024-02-13T00:41:02.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.148+0000 7f602d7fa700 10 client.15063.objecter ms_dispatch 0x7f6030001550 osd_map(107..107 src has 52..107) v4 2024-02-13T00:41:02.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.148+0000 7f602d7fa700 3 client.15063.objecter handle_osd_map got epochs [107,107] > 0 2024-02-13T00:41:02.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.148+0000 7f602d7fa700 3 client.15063.objecter handle_osd_map decoding full epoch 107 2024-02-13T00:41:02.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.148+0000 7f602ffff700 1 --2- 172.21.15.162:0/1514900161 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6018062ce0 0x7f60180651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:02.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.149+0000 7f602d7fa700 20 client.15063.objecter dump_active .. 0 homeless 2024-02-13T00:41:02.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.149+0000 7f602d7fa700 10 client.15063.objecter ms_handle_connect 0x7f6018062ce0 2024-02-13T00:41:02.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.149+0000 7f602ffff700 1 --2- 172.21.15.162:0/1514900161 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6018062ce0 0x7f60180651a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f6020005010 tx=0x7f6020009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:02.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.157+0000 7f602d7fa700 1 -- 172.21.15.162:0/1514900161 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f602804af60 con 0x7f60300f6600 2024-02-13T00:41:02.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:02.308 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-02-13T00:41:02.309 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] Global test environment set-up. 2024-02-13T00:41:02.309 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] 7 tests from LibRadosList 2024-02-13T00:41:02.309 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-02-13T00:41:02.309 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.ListObjects (341 ms) 2024-02-13T00:41:02.309 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-02-13T00:41:02.309 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (51 ms) 2024-02-13T00:41:02.309 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-02-13T00:41:02.309 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset foo1,foo2,foo3 2024-02-13T00:41:02.309 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo1 2024-02-13T00:41:02.310 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo2 2024-02-13T00:41:02.310 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo3 2024-02-13T00:41:02.310 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset foo1,foo4,foo5 2024-02-13T00:41:02.310 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo4 2024-02-13T00:41:02.310 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo5 2024-02-13T00:41:02.310 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo1 2024-02-13T00:41:02.310 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset foo6,foo7 2024-02-13T00:41:02.310 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo7 2024-02-13T00:41:02.310 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo6 2024-02-13T00:41:02.310 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-02-13T00:41:02.310 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns1:foo4 2024-02-13T00:41:02.311 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns1:foo5 2024-02-13T00:41:02.311 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns2:foo7 2024-02-13T00:41:02.311 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns2:foo6 2024-02-13T00:41:02.311 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns1:foo1 2024-02-13T00:41:02.311 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: :foo1 2024-02-13T00:41:02.311 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: :foo2 2024-02-13T00:41:02.311 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: :foo3 2024-02-13T00:41:02.311 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (136 ms) 2024-02-13T00:41:02.311 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-02-13T00:41:02.311 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 1 0 2024-02-13T00:41:02.311 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 10 0 2024-02-13T00:41:02.312 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 13 0 2024-02-13T00:41:02.312 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 7 0 2024-02-13T00:41:02.312 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 14 0 2024-02-13T00:41:02.312 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 0 0 2024-02-13T00:41:02.312 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 15 0 2024-02-13T00:41:02.312 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 11 0 2024-02-13T00:41:02.312 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 5 0 2024-02-13T00:41:02.312 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 8 0 2024-02-13T00:41:02.312 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 6 0 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 3 0 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 4 0 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 12 0 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 9 0 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 2 0 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (117 ms) 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: x cursor=MIN 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=1 cursor=18:02547ec2:::1:head 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=10 cursor=18:52ea6a34:::10:head 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=13 cursor=18:566253c9:::13:head 2024-02-13T00:41:02.313 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=7 cursor=18:5c6b0b28:::7:head 2024-02-13T00:41:02.314 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=14 cursor=18:62a1935d:::14:head 2024-02-13T00:41:02.314 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=0 cursor=18:6cac518f:::0:head 2024-02-13T00:41:02.314 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=15 cursor=18:863748b0:::15:head 2024-02-13T00:41:02.314 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=11 cursor=18:89d3ae78:::11:head 2024-02-13T00:41:02.314 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=5 cursor=18:b29083e3:::5:head 2024-02-13T00:41:02.314 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=8 cursor=18:bd63b0f1:::8:head 2024-02-13T00:41:02.314 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=6 cursor=18:c4fdafeb:::6:head 2024-02-13T00:41:02.314 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=3 cursor=18:cfc208b3:::3:head 2024-02-13T00:41:02.314 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=4 cursor=18:d83876eb:::4:head 2024-02-13T00:41:02.314 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=12 cursor=18:de5d7c5f:::12:head 2024-02-13T00:41:02.314 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=9 cursor=18:e960b815:::9:head 2024-02-13T00:41:02.314 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=2 cursor=18:f905c69b:::2:head 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: FIRST> seek to MIN oid=1 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=1 cursor=18:02547ec2:::1:head 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:02547ec2:::1:head 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:02547ec2:::1:head -> 1 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=10 cursor=18:52ea6a34:::10:head 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:52ea6a34:::10:head 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:52ea6a34:::10:head -> 10 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=13 cursor=18:566253c9:::13:head 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:566253c9:::13:head 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:566253c9:::13:head -> 13 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=7 cursor=18:5c6b0b28:::7:head 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:5c6b0b28:::7:head 2024-02-13T00:41:02.315 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:5c6b0b28:::7:head -> 7 2024-02-13T00:41:02.377 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:02.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:02.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:02.445 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:02.453 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:41:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:41:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24911.0:16 16.10 16.43766690 (undecoded) ondisk+write+known_if_redirected e107)", 2024-02-13T00:41:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.450503+0000", 2024-02-13T00:41:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.2334999999999999e-05, 2024-02-13T00:41:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2076000000000001e-05, 2024-02-13T00:41:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-02-13T00:41:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24911", 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1860840531", 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 16 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.450503+0000", 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.450503+0000", 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:41:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.450501+0000", 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4679999999999997e-06 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.450508+0000", 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4406999999999999e-05 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.450522+0000", 2024-02-13T00:41:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5689999999999999e-06 2024-02-13T00:41:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.450530+0000", 2024-02-13T00:41:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7123000000000001e-05 2024-02-13T00:41:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:41:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-02-13T00:41:02.456 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:41:02.460 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=14 cursor=18:62a1935d:::14:head 2024-02-13T00:41:02.460 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:62a1935d:::14:head 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:62a1935d:::14:head -> 14 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=0 cursor=18:6cac518f:::0:head 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:6cac518f:::0:head 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:6cac518f:::0:head -> 0 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=15 cursor=18:863748b0:::15:head 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:863748b0:::15:head 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:863748b0:::15:head -> 15 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=11 cursor=18:89d3ae78:::11:head 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:89d3ae78:::11:head 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:89d3ae78:::11:head -> 11 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=5 cursor=18:b29083e3:::5:head 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:b29083e3:::5:head 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:b29083e3:::5:head -> 5 2024-02-13T00:41:02.461 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=8 cursor=18:bd63b0f1:::8:head 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:bd63b0f1:::8:head 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:bd63b0f1:::8:head -> 8 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=6 cursor=18:c4fdafeb:::6:head 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:c4fdafeb:::6:head 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:c4fdafeb:::6:head -> 6 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=3 cursor=18:cfc208b3:::3:head 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:cfc208b3:::3:head 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:cfc208b3:::3:head -> 3 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=4 cursor=18:d83876eb:::4:head 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:d83876eb:::4:head 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:d83876eb:::4:head -> 4 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=12 cursor=18:de5d7c5f:::12:head 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:de5d7c5f:::12:head 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:de5d7c5f:::12:head -> 12 2024-02-13T00:41:02.462 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=9 cursor=18:e960b815:::9:head 2024-02-13T00:41:02.463 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:e960b815:::9:head 2024-02-13T00:41:02.463 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:e960b815:::9:head -> 9 2024-02-13T00:41:02.463 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=2 cursor=18:f905c69b:::2:head 2024-02-13T00:41:02.463 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:f905c69b:::2:head 2024-02-13T00:41:02.463 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:f905c69b:::2:head -> 2 2024-02-13T00:41:02.463 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:566253c9:::13:head 2024-02-13T00:41:02.463 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:566253c9:::13:head expected=18:566253c9:::13:head 2024-02-13T00:41:02.463 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:566253c9:::13:head -> 13 2024-02-13T00:41:02.463 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=13 expected=13 2024-02-13T00:41:02.463 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:f905c69b:::2:head 2024-02-13T00:41:02.463 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:f905c69b:::2:head expected=18:f905c69b:::2:head 2024-02-13T00:41:02.463 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:f905c69b:::2:head -> 2 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=2 expected=2 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:e960b815:::9:head 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:e960b815:::9:head expected=18:e960b815:::9:head 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:e960b815:::9:head -> 9 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=9 expected=9 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:de5d7c5f:::12:head 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:de5d7c5f:::12:head expected=18:de5d7c5f:::12:head 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:de5d7c5f:::12:head -> 12 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=12 expected=12 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:d83876eb:::4:head 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:d83876eb:::4:head expected=18:d83876eb:::4:head 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:d83876eb:::4:head -> 4 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=4 expected=4 2024-02-13T00:41:02.464 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:cfc208b3:::3:head 2024-02-13T00:41:02.465 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:cfc208b3:::3:head expected=18:cfc208b3:::3:head 2024-02-13T00:41:02.465 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:cfc208b3:::3:head -> 3 2024-02-13T00:41:02.465 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=3 expected=3 2024-02-13T00:41:02.465 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:c4fdafeb:::6:head 2024-02-13T00:41:02.465 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:c4fdafeb:::6:head expected=18:c4fdafeb:::6:head 2024-02-13T00:41:02.465 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:c4fdafeb:::6:head -> 6 2024-02-13T00:41:02.465 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=6 expected=6 2024-02-13T00:41:02.465 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:bd63b0f1:::8:head 2024-02-13T00:41:02.465 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:bd63b0f1:::8:head expected=18:bd63b0f1:::8:head 2024-02-13T00:41:02.465 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:bd63b0f1:::8:head -> 8 2024-02-13T00:41:02.465 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:02.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:02.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:02.499+0000 7f603664a700 1 -- 172.21.15.162:0/1514900161 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f601c005780 con 0x7f60300f6600 2024-02-13T00:41:02.544 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:02.811 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry= api_cmd_pp: Running main() from gmock_main.cc 2024-02-13T00:41:02.811 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-02-13T00:41:02.811 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-02-13T00:41:02.811 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-02-13T00:41:02.811 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-02-13T00:41:02.811 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (356 ms) 2024-02-13T00:41:02.811 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-02-13T00:41:02.811 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (22 ms) 2024-02-13T00:41:02.811 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-02-13T00:41:02.811 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (7727 ms) 2024-02-13T00:41:02.811 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (8105 ms total) 2024-02-13T00:41:02.811 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: 2024-02-13T00:41:02.812 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-02-13T00:41:02.812 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (8105 ms total) 2024-02-13T00:41:02.812 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-02-13T00:41:02.826 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-02-13T00:41:02.826 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [==========] Running 16 tests from 1 test suite. 2024-02-13T00:41:02.826 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (201 ms) 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (88 ms) 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (10 ms) 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (35 ms) 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (11 ms) 2024-02-13T00:41:02.827 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (8 ms) 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (4 ms) 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (12 ms) 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (10 ms) 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (10 ms) 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (13 ms) 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (27 ms) 2024-02-13T00:41:02.828 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-02-13T00:41:02.829 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (23 ms) 2024-02-13T00:41:02.829 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-02-13T00:41:02.829 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (7 ms) 2024-02-13T00:41:02.829 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-02-13T00:41:02.829 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (3 ms) 2024-02-13T00:41:02.829 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest (462 ms total) 2024-02-13T00:41:02.829 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: 2024-02-13T00:41:02.829 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-02-13T00:41:02.829 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [==========] 16 tests from 1 test suite ran. (8143 ms total) 2024-02-13T00:41:02.829 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ PASSED ] 16 tests. 2024-02-13T00:41:02.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:01.409132+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:41:02.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:01.412038+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:41:02.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.470777+0000 mon.a (mon.0) 782 : audit [INF] from='client.? 172.21.15.162:0/79393202' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-84033-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.470870+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.470937+0000 mon.a (mon.0) 784 : audit [INF] from='client.? 172.21.15.162:0/1224070377' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-83993-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.471004+0000 mon.a (mon.0) 785 : audit [INF] from='client.? 172.21.15.162:0/3501757973' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-84001-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.471080+0000 mon.a (mon.0) 786 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-84105-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.471177+0000 mon.a (mon.0) 787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-84161-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.471246+0000 mon.a (mon.0) 788 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-83977-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.471367+0000 mon.a (mon.0) 789 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.471459+0000 mon.a (mon.0) 790 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-84025-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.471539+0000 mon.a (mon.0) 791 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:02.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.471655+0000 mon.a (mon.0) 792 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-84057-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.471728+0000 mon.a (mon.0) 793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-83985-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.471816+0000 mon.a (mon.0) 794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.471933+0000 mon.a (mon.0) 795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-84097-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.472045+0000 mon.a (mon.0) 796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-84073-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.472125+0000 mon.a (mon.0) 797 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-84081-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.472195+0000 mon.a (mon.0) 798 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-84089-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.472286+0000 mon.a (mon.0) 799 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-84017-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.472408+0000 mon.a (mon.0) 800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84129-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:01.472515+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-13T00:41:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.476103+0000 mon.c (mon.2) 94 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-84113-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.486866+0000 mon.a (mon.0) 802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-84113-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:01.487571+0000 mon.a (mon.0) 803 : audit [INF] from='client.? 172.21.15.162:0/1336352966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84121-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:02.934 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:02.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:02.963 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[84192]: starting. 2024-02-13T00:41:02.963 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[84192]: creating pool ceph_test_rados_list_parallel.smithi162-84169 2024-02-13T00:41:02.963 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[84192]: created object 0... 2024-02-13T00:41:02.963 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[84192]: created object 25... 2024-02-13T00:41:02.963 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[84192]: created object 49... 2024-02-13T00:41:02.964 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[84192]: finishing. 2024-02-13T00:41:02.964 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[84192]: shutting down. 2024-02-13T00:41:02.964 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_2_[84193]: starting. 2024-02-13T00:41:02.964 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_2_[84193]: listing objects. 2024-02-13T00:41:02.964 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_2_[84193]: listed object 0... 2024-02-13T00:41:02.964 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_2_[84193]: listed object 25... 2024-02-13T00:41:02.964 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_2_[84193]: saw 50 objects 2024-02-13T00:41:02.964 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_2_[84193]: shutting down. 2024-02-13T00:41:02.964 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:02.964 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[86126]: starting. 2024-02-13T00:41:02.964 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[86126]: creating pool ceph_test_rados_list_parallel.smithi162-84169 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[86126]: created object 0... 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[86126]: created object 25... 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[86126]: created object 49... 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[86126]: finishing. 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[86126]: shutting down. 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_4_[86127]: starting. 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_4_[86127]: listing objects. 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_4_[86127]: listed object 0... 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_4_[86127]: listed object 25... 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_4_[86127]: saw 41 objects 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_4_[86127]: shutting down. 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:02.965 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_5_[86128]: starting. 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_5_[86128]: removed 25 objects... 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_5_[86128]: removed half of the objects 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_5_[86128]: removed 50 objects... 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_5_[86128]: removed 50 objects 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_5_[86128]: shutting down. 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[86258]: starting. 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[86258]: creating pool ceph_test_rados_list_parallel.smithi162-84169 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[86258]: created object 0... 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[86258]: created object 25... 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[86258]: created object 49... 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[86258]: finishing. 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[86258]: shutting down. 2024-02-13T00:41:02.966 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[86259]: starting. 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[86259]: listing objects. 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[86259]: listed object 0... 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[86259]: listed object 25... 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[86259]: listed object 50... 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[86259]: saw 51 objects 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[86259]: shutting down. 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_8_[86260]: starting. 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_8_[86260]: added 25 objects... 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_8_[86260]: added half of the objects 2024-02-13T00:41:02.967 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_8_[86260]: added 50 objects... 2024-02-13T00:41:02.968 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_8_[86260]: added 50 objects 2024-02-13T00:41:02.968 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_8_[86260]: shutting down. 2024-02-13T00:41:02.968 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:02.968 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.038 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:03.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:01.409132+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:41:03.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:01.412038+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:41:03.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.470777+0000 mon.a (mon.0) 782 : audit [INF] from='client.? 172.21.15.162:0/79393202' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-84033-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.470870+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.470937+0000 mon.a (mon.0) 784 : audit [INF] from='client.? 172.21.15.162:0/1224070377' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-83993-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.471004+0000 mon.a (mon.0) 785 : audit [INF] from='client.? 172.21.15.162:0/3501757973' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-84001-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.471080+0000 mon.a (mon.0) 786 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-84105-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.471177+0000 mon.a (mon.0) 787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-84161-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.471246+0000 mon.a (mon.0) 788 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-83977-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.471367+0000 mon.a (mon.0) 789 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.471459+0000 mon.a (mon.0) 790 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-84025-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.471539+0000 mon.a (mon.0) 791 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:03.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.471655+0000 mon.a (mon.0) 792 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-84057-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.471728+0000 mon.a (mon.0) 793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-83985-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.471816+0000 mon.a (mon.0) 794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.471933+0000 mon.a (mon.0) 795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-84097-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.472045+0000 mon.a (mon.0) 796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-84073-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.472125+0000 mon.a (mon.0) 797 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-84081-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.472195+0000 mon.a (mon.0) 798 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-84089-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.472286+0000 mon.a (mon.0) 799 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-84017-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.472408+0000 mon.a (mon.0) 800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84129-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:01.472515+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-13T00:41:03.097 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.476103+0000 mon.c (mon.2) 94 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-84113-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:03.097 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.486866+0000 mon.a (mon.0) 802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-84113-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:03.097 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:01.487571+0000 mon.a (mon.0) 803 : audit [INF] from='client.? 172.21.15.162:0/1336352966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84121-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:03.097 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:03.248 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.248 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[86472]: starting. 2024-02-13T00:41:03.248 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[86472]: creating pool ceph_test_rados_list_parallel.smithi162-84169 2024-02-13T00:41:03.248 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[86472]: created object 0... 2024-02-13T00:41:03.248 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[86472]: created object 25... 2024-02-13T00:41:03.249 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[86472]: created object 49... 2024-02-13T00:41:03.249 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[86472]: finishing. 2024-02-13T00:41:03.249 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[86472]: shutting down. 2024-02-13T00:41:03.249 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.249 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.249 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.249 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[86473]: starting. 2024-02-13T00:41:03.250 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[86473]: listing objects. 2024-02-13T00:41:03.250 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[86473]: listed object 0... 2024-02-13T00:41:03.250 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[86473]: listed object 25... 2024-02-13T00:41:03.250 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[86473]: listed object 50... 2024-02-13T00:41:03.250 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[86473]: listed object 75... 2024-02-13T00:41:03.250 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[86473]: saw 96 objects 2024-02-13T00:41:03.250 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[86473]: shutting down. 2024-02-13T00:41:03.250 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.251 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.251 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.251 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_11_[86474]: starting. 2024-02-13T00:41:03.251 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_11_[86474]: added 25 objects... 2024-02-13T00:41:03.251 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_11_[86474]: added half of the objects 2024-02-13T00:41:03.251 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_11_[86474]: added 50 objects... 2024-02-13T00:41:03.251 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_11_[86474]: added 50 objects 2024-02-13T00:41:03.252 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_11_[86474]: shutting down. 2024-02-13T00:41:03.252 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.252 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.252 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.252 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_13_[86476]: starting. 2024-02-13T00:41:03.252 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_13_[86476]: removed 25 objects... 2024-02-13T00:41:03.252 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_13_[86476]: removed half of the objects 2024-02-13T00:41:03.253 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_13_[86476]: removed 50 objects... 2024-02-13T00:41:03.253 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_13_[86476]: removed 50 objects 2024-02-13T00:41:03.253 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_13_[86476]: shutting down. 2024-02-13T00:41:03.253 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.253 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.253 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.253 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_12_[86475]: starting. 2024-02-13T00:41:03.253 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_12_[86475]: added 25 objects... 2024-02-13T00:41:03.254 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_12_[86475]: added half of the objects 2024-02-13T00:41:03.254 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_12_[86475]: added 50 objects... 2024-02-13T00:41:03.254 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_12_[86475]: added 50 objects 2024-02-13T00:41:03.254 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_12_[86475]: shutting down. 2024-02-13T00:41:03.254 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.254 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.254 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.255 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.255 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[86783]: starting. 2024-02-13T00:41:03.255 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[86783]: creating pool ceph_test_rados_list_parallel.smithi162-84169 2024-02-13T00:41:03.255 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[86783]: created object 0... 2024-02-13T00:41:03.255 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[86783]: created object 25... 2024-02-13T00:41:03.255 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[86783]: created object 49... 2024-02-13T00:41:03.255 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[86783]: finishing. 2024-02-13T00:41:03.256 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[86783]: shutting down. 2024-02-13T00:41:03.256 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.256 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.256 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.256 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:03.256 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[86784]: starting. 2024-02-13T00:41:03.256 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[86784]: listing objects. 2024-02-13T00:41:03.256 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[86784]: listed object 0... 2024-02-13T00:41:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:01.409132+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:41:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:01.412038+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:41:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.470777+0000 mon.a (mon.0) 782 : audit [INF] from='client.? 172.21.15.162:0/79393202' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-84033-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.470870+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.470937+0000 mon.a (mon.0) 784 : audit [INF] from='client.? 172.21.15.162:0/1224070377' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-83993-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.471004+0000 mon.a (mon.0) 785 : audit [INF] from='client.? 172.21.15.162:0/3501757973' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-84001-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.471080+0000 mon.a (mon.0) 786 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-84105-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.471177+0000 mon.a (mon.0) 787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-84161-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.471246+0000 mon.a (mon.0) 788 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-83977-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.471367+0000 mon.a (mon.0) 789 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.471459+0000 mon.a (mon.0) 790 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-84025-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.471539+0000 mon.a (mon.0) 791 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.471655+0000 mon.a (mon.0) 792 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-84057-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.471728+0000 mon.a (mon.0) 793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-83985-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.471816+0000 mon.a (mon.0) 794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.471933+0000 mon.a (mon.0) 795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-84097-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.472045+0000 mon.a (mon.0) 796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-84073-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.472125+0000 mon.a (mon.0) 797 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-84081-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.472195+0000 mon.a (mon.0) 798 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-84089-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.472286+0000 mon.a (mon.0) 799 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-84017-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.472408+0000 mon.a (mon.0) 800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84129-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:03.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:01.472515+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-13T00:41:03.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.476103+0000 mon.c (mon.2) 94 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-84113-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:03.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.486866+0000 mon.a (mon.0) 802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-84113-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:03.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:01.487571+0000 mon.a (mon.0) 803 : audit [INF] from='client.? 172.21.15.162:0/1336352966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84121-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:03.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-13T00:41:03.553 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:03.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:03.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:03.996+0000 7f602d7fa700 1 -- 172.21.15.162:0/1514900161 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v109) v1 ==== 87+0+0 (secure 0 0 0) 0x7f6028019080 con 0x7f60300f6600 2024-02-13T00:41:04.003 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[86784]: listed object 25... 2024-02-13T00:41:04.003 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[86784]: listed object 50... 2024-02-13T00:41:04.003 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[86784]: listed object 75... 2024-02-13T00:41:04.003 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[86784]: listed object 100... 2024-02-13T00:41:04.003 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[86784]: listed object 125... 2024-02-13T00:41:04.003 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[86784]: saw 150 objects 2024-02-13T00:41:04.003 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[86784]: shutting down. 2024-02-13T00:41:04.003 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:04.003 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_16_[86785]: starting. 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_16_[86785]: added 25 objects... 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_16_[86785]: added half of the objects 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_16_[86785]: added 50 objects... 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_16_[86785]: added 50 objects 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_16_[86785]: shutting down. 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-02-13T00:41:04.004 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******* SUCCESS ********** 2024-02-13T00:41:04.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.001+0000 7f603664a700 10 librados: watch_flush enter 2024-02-13T00:41:04.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.002+0000 7f603664a700 10 librados: watch_flush exit 2024-02-13T00:41:04.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.002+0000 7f603664a700 20 client.15063.objecter shutdown clearing up homeless session... 2024-02-13T00:41:04.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.002+0000 7f603664a700 10 client.15063.objecter successfully canceled tick 2024-02-13T00:41:04.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.002+0000 7f603664a700 1 -- 172.21.15.162:0/1514900161 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6018062ce0 msgr2=0x7f60180651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:04.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.002+0000 7f603664a700 1 --2- 172.21.15.162:0/1514900161 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6018062ce0 0x7f60180651a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f6020005010 tx=0x7f6020009cb0).stop 2024-02-13T00:41:04.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.002+0000 7f603664a700 1 -- 172.21.15.162:0/1514900161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60300f6600 msgr2=0x7f603006a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:04.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.002+0000 7f603664a700 1 --2- 172.21.15.162:0/1514900161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60300f6600 0x7f603006a960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f602800ebf0 tx=0x7f602800b040).stop 2024-02-13T00:41:04.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.006+0000 7f603664a700 1 -- 172.21.15.162:0/1514900161 shutdown_connections 2024-02-13T00:41:04.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.006+0000 7f603664a700 1 --2- 172.21.15.162:0/1514900161 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6018062ce0 0x7f60180651a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:04.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.006+0000 7f603664a700 1 --2- 172.21.15.162:0/1514900161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6030108a00 0x7f6030063610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:04.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.006+0000 7f603664a700 1 --2- 172.21.15.162:0/1514900161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60300f5d30 0x7f603006a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:04.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.006+0000 7f603664a700 1 --2- 172.21.15.162:0/1514900161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60300f6600 0x7f603006a960 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:04.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.006+0000 7f603664a700 1 -- 172.21.15.162:0/1514900161 >> 172.21.15.162:0/1514900161 conn(0x7f603006fc50 msgr2=0x7f603006c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:04.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.008+0000 7f603664a700 1 -- 172.21.15.162:0/1514900161 shutdown_connections 2024-02-13T00:41:04.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.010+0000 7f603664a700 1 -- 172.21.15.162:0/1514900161 wait complete. 2024-02-13T00:41:04.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:04.016+0000 7f603664a700 1 librados: shutdown 2024-02-13T00:41:04.032 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:41:04.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:02.203475+0000 mgr.y (mgr.14544) 135 : cluster [DBG] pgmap v131: 932 pgs: 192 unknown, 569 creating+peering, 171 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:41:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:02.322103+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.1f scrub starts 2024-02-13T00:41:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:02.323096+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.1f scrub ok 2024-02-13T00:41:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:02.399662+0000 osd.2 (osd.2) 59 : cluster [DBG] 6.f scrub starts 2024-02-13T00:41:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:02.402662+0000 osd.2 (osd.2) 60 : cluster [DBG] 6.f scrub ok 2024-02-13T00:41:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:02.780519+0000 mon.a (mon.0) 804 : audit [INF] from='client.? 172.21.15.162:0/1336352966' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84121-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:02.781034+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-13T00:41:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:02.806061+0000 mon.a (mon.0) 806 : audit [INF] from='client.? 172.21.15.162:0/1514900161' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:41:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:02.812017+0000 osd.3 (osd.3) 81 : cluster [DBG] 6.2 scrub starts 2024-02-13T00:41:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:02.817234+0000 osd.3 (osd.3) 82 : cluster [DBG] 6.2 scrub ok 2024-02-13T00:41:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:02.822504+0000 mon.a (mon.0) 807 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:02.837551+0000 mon.a (mon.0) 808 : audit [DBG] from='client.? 172.21.15.162:0/2239408765' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-13T00:41:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:02.868325+0000 mon.b (mon.1) 43 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:04.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:02.869402+0000 mon.b (mon.1) 44 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:04.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:02.872490+0000 mon.b (mon.1) 45 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:04.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:02.875006+0000 mon.b (mon.1) 46 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:04.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:02.902592+0000 mon.b (mon.1) 47 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:04.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:03.904010+0000 mon.b (mon.1) 48 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:04.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:02.203475+0000 mgr.y (mgr.14544) 135 : cluster [DBG] pgmap v131: 932 pgs: 192 unknown, 569 creating+peering, 171 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:41:04.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:02.322103+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.1f scrub starts 2024-02-13T00:41:04.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:02.323096+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.1f scrub ok 2024-02-13T00:41:04.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:02.399662+0000 osd.2 (osd.2) 59 : cluster [DBG] 6.f scrub starts 2024-02-13T00:41:04.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:02.402662+0000 osd.2 (osd.2) 60 : cluster [DBG] 6.f scrub ok 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:02.780519+0000 mon.a (mon.0) 804 : audit [INF] from='client.? 172.21.15.162:0/1336352966' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84121-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:02.781034+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:02.806061+0000 mon.a (mon.0) 806 : audit [INF] from='client.? 172.21.15.162:0/1514900161' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:02.812017+0000 osd.3 (osd.3) 81 : cluster [DBG] 6.2 scrub starts 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:02.817234+0000 osd.3 (osd.3) 82 : cluster [DBG] 6.2 scrub ok 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:02.822504+0000 mon.a (mon.0) 807 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:02.837551+0000 mon.a (mon.0) 808 : audit [DBG] from='client.? 172.21.15.162:0/2239408765' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:02.868325+0000 mon.b (mon.1) 43 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:02.869402+0000 mon.b (mon.1) 44 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:02.872490+0000 mon.b (mon.1) 45 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:02.875006+0000 mon.b (mon.1) 46 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:02.902592+0000 mon.b (mon.1) 47 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:04.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:03.904010+0000 mon.b (mon.1) 48 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:04.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-02-13T00:41:04.864 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [----------] Global test environment set-up. 2024-02-13T00:41:04.864 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [----------] 4 tests from LibRadosService 2024-02-13T00:41:04.864 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-02-13T00:41:04.864 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5304 ms) 2024-02-13T00:41:04.864 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-02-13T00:41:04.864 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ OK ] LibRadosService.RegisterLate (24 ms) 2024-02-13T00:41:04.864 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-02-13T00:41:04.864 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: cluster: 2024-02-13T00:41:04.864 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: id: d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:41:04.864 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: health: HEALTH_OK 2024-02-13T00:41:04.865 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:04.865 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: services: 2024-02-13T00:41:04.865 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-02-13T00:41:04.865 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-02-13T00:41:04.865 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-02-13T00:41:04.865 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-02-13T00:41:04.865 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-02-13T00:41:04.865 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:04.865 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: data: 2024-02-13T00:41:04.865 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: pools: 8 pools, 196 pgs 2024-02-13T00:41:04.866 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: objects: 224 objects, 5.7 KiB 2024-02-13T00:41:04.866 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:41:04.866 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: pgs: 32.653% pgs unknown 2024-02-13T00:41:04.866 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 132 active+clean 2024-02-13T00:41:04.866 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 64 unknown 2024-02-13T00:41:04.866 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:04.866 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: io: 2024-02-13T00:41:04.866 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: client: 972 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-13T00:41:04.866 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:04.866 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:04.866 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: cluster: 2024-02-13T00:41:04.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: id: d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:41:04.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: health: HEALTH_OK 2024-02-13T00:41:04.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:04.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: services: 2024-02-13T00:41:04.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-02-13T00:41:04.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-02-13T00:41:04.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-02-13T00:41:04.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-02-13T00:41:04.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:04.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: data: 2024-02-13T00:41:04.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: pools: 25 pools, 740 pgs 2024-02-13T00:41:04.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: objects: 224 objects, 5.7 KiB 2024-02-13T00:41:04.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:41:04.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: pgs: 76.892% pgs not active 2024-02-13T00:41:04.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 569 creating+peering 2024-02-13T00:41:04.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 171 active+clean 2024-02-13T00:41:04.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:04.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: io: 2024-02-13T00:41:04.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: client: 1.7 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-02-13T00:41:04.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:04.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:04.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: cluster: 2024-02-13T00:41:04.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: id: d4ee01fc-ca06-11ee-95b9-87774f69a715 2024-02-13T00:41:04.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: health: HEALTH_WARN 2024-02-13T00:41:04.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: noscrub flag(s) set 2024-02-13T00:41:04.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:04.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: services: 2024-02-13T00:41:04.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-02-13T00:41:04.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: mgr: y(active, since 3m), standbys: x 2024-02-13T00:41:04.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-02-13T00:41:04.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: flags noscrub 2024-02-13T00:41:04.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-02-13T00:41:04.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-02-13T00:41:04.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-02-13T00:41:04.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:04.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: data: 2024-02-13T00:41:04.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: pools: 31 pools, 932 pgs 2024-02-13T00:41:04.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: objects: 224 objects, 5.7 KiB 2024-02-13T00:41:04.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:41:04.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: pgs: 20.601% pgs unknown 2024-02-13T00:41:05.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:02.203475+0000 mgr.y (mgr.14544) 135 : cluster [DBG] pgmap v131: 932 pgs: 192 unknown, 569 creating+peering, 171 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:41:05.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:02.322103+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.1f scrub starts 2024-02-13T00:41:05.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:02.323096+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.1f scrub ok 2024-02-13T00:41:05.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:02.399662+0000 osd.2 (osd.2) 59 : cluster [DBG] 6.f scrub starts 2024-02-13T00:41:05.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:02.402662+0000 osd.2 (osd.2) 60 : cluster [DBG] 6.f scrub ok 2024-02-13T00:41:05.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:02.780519+0000 mon.a (mon.0) 804 : audit [INF] from='client.? 172.21.15.162:0/1336352966' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84121-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:02.781034+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-13T00:41:05.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:02.806061+0000 mon.a (mon.0) 806 : audit [INF] from='client.? 172.21.15.162:0/1514900161' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:41:05.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:02.812017+0000 osd.3 (osd.3) 81 : cluster [DBG] 6.2 scrub starts 2024-02-13T00:41:05.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:02.817234+0000 osd.3 (osd.3) 82 : cluster [DBG] 6.2 scrub ok 2024-02-13T00:41:05.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:02.822504+0000 mon.a (mon.0) 807 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:02.837551+0000 mon.a (mon.0) 808 : audit [DBG] from='client.? 172.21.15.162:0/2239408765' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-13T00:41:05.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:02.868325+0000 mon.b (mon.1) 43 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:02.869402+0000 mon.b (mon.1) 44 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:02.872490+0000 mon.b (mon.1) 45 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:02.875006+0000 mon.b (mon.1) 46 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:02.902592+0000 mon.b (mon.1) 47 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:05.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:03.904010+0000 mon.b (mon.1) 48 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:05.558 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: api_cmd: Running main() from gmock_main.cc 2024-02-13T00:41:05.558 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-02-13T00:41:05.558 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [----------] Global test environment set-up. 2024-02-13T00:41:05.558 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-02-13T00:41:05.558 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-02-13T00:41:05.558 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (81 ms) 2024-02-13T00:41:05.558 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-02-13T00:41:05.558 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (4622 ms) 2024-02-13T00:41:05.558 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-02-13T00:41:05.558 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (5120 ms) 2024-02-13T00:41:05.558 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:02.780519+0000 mon.a [INF] from='client.? 172.21.15.162:0/1336352966' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84121-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:02.806061+0000 mon.a [INF] from='client.? 172.21.15.162:0/1514900161' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:02.822504+0000 mon.a [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:02.868325+0000 mon.b [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:02.869402+0000 mon.b [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:02.872490+0000 mon.b [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:02.875006+0000 mon.b [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:03.976567+0000 mon.a [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:03.977609+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:03.977936+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:03.978182+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:03.978426+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:03.986596+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-84113-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]': finished 2024-02-13T00:41:05.559 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:03.986719+0000 mon.a [INF] from='client.? 172.21.15.162:0/1514900161' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:03.986809+0000 mon.a [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.005979+0000 mon.b [INF] from='client.? 172.21.15.162:0/508499360' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-83977-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.006410+0000 mon.b [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.006588+0000 mon.b [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.007018+0000 mon.b [INF] from='client.? 172.21.15.162:0/2253596229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.283028+0000 mon.b [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.533804+0000 mon.b [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.542318+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-83977-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.542733+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.543163+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.543571+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.546851+0000 mon.b [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.560 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.546980+0000 mon.b [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.584356+0000 mon.c [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.584524+0000 client.admin [INF] onexx 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.807852+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.808349+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.808520+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.808713+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.808820+0000 mon.b [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-84089-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.809074+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.809118+0000 mon.b [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-83993-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.809318+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-84089-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.809563+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-83993-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.984855+0000 mon.a [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:05.561 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.985485+0000 mon.a [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.985928+0000 mon.a [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.986303+0000 mon.a [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.986696+0000 mon.a [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.987068+0000 mon.a [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.987447+0000 mon.a [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.987852+0000 mon.a [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.997118+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-83977-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.997251+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.997391+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.997450+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.997526+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-84089-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.997597+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-83993-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.562 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:04.998361+0000 mon.a [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:05.563 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.002161+0000 mon.b [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-84001-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.563 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.002384+0000 mon.b [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-84089-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.563 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.004378+0000 mon.b [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-83993-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.563 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.005564+0000 mon.a [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:05.563 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.005720+0000 mon.a [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.563 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.006507+0000 mon.b [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-84097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.563 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.007119+0000 mon.b [INF] from='client.? 172.21.15.162:0/1085499831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.563 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.007283+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-84001-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.657 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:05.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:05.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:05.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:41:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:02.115165+0000 osd.7 (osd.7) 64 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:41:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:02.115852+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:41:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:03.410165+0000 osd.2 (osd.2) 61 : cluster [DBG] 5.b scrub starts 2024-02-13T00:41:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:03.411253+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.b scrub ok 2024-02-13T00:41:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:03.605466+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:41:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:03.608371+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:41:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:03.976567+0000 mon.a (mon.0) 809 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:03.977609+0000 mon.a (mon.0) 810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:03.977936+0000 mon.a (mon.0) 811 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:03.978182+0000 mon.a (mon.0) 812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:03.978426+0000 mon.a (mon.0) 813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:03.986596+0000 mon.a (mon.0) 814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-84113-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]': finished 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:03.986719+0000 mon.a (mon.0) 815 : audit [INF] from='client.? 172.21.15.162:0/1514900161' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:03.986809+0000 mon.a (mon.0) 816 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.005979+0000 mon.b (mon.1) 49 : audit [INF] from='client.? 172.21.15.162:0/508499360' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-83977-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.006410+0000 mon.b (mon.1) 50 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.006588+0000 mon.b (mon.1) 51 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.007018+0000 mon.b (mon.1) 52 : audit [INF] from='client.? 172.21.15.162:0/2253596229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:04.204105+0000 mgr.y (mgr.14544) 136 : cluster [DBG] pgmap v134: 652 pgs: 264 unknown, 256 creating+peering, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:41:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.283028+0000 mon.b (mon.1) 53 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:04.533167+0000 mon.a (mon.0) 817 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.533804+0000 mon.b (mon.1) 54 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.542318+0000 mon.a (mon.0) 818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-83977-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.542733+0000 mon.a (mon.0) 819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.543163+0000 mon.a (mon.0) 820 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.543571+0000 mon.a (mon.0) 821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.546851+0000 mon.b (mon.1) 55 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.546980+0000 mon.b (mon.1) 56 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.584356+0000 mon.c (mon.2) 95 : audit [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:04.584524+0000 client.admin (client.?) 0 : cluster [INF] onexx 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.807852+0000 mon.a (mon.0) 822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.808349+0000 mon.a (mon.0) 823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.808520+0000 mon.a (mon.0) 824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.808713+0000 mon.a (mon.0) 825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.808820+0000 mon.b (mon.1) 57 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-84089-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.809074+0000 mon.a (mon.0) 826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.809118+0000 mon.b (mon.1) 58 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-83993-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.809318+0000 mon.a (mon.0) 827 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-84089-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.809563+0000 mon.a (mon.0) 828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-83993-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.860931+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? 172.21.15.162:0/1581932932' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.906073+0000 mon.b (mon.1) 59 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.984855+0000 mon.a (mon.0) 829 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.985485+0000 mon.a (mon.0) 830 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.985928+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.986303+0000 mon.a (mon.0) 832 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.986696+0000 mon.a (mon.0) 833 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:05.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.987068+0000 mon.a (mon.0) 834 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.987447+0000 mon.a (mon.0) 835 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.987852+0000 mon.a (mon.0) 836 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.997118+0000 mon.a (mon.0) 837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-83977-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.997251+0000 mon.a (mon.0) 838 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.997391+0000 mon.a (mon.0) 839 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.997450+0000 mon.a (mon.0) 840 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.997526+0000 mon.a (mon.0) 841 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-84089-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.997597+0000 mon.a (mon.0) 842 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-83993-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:04.997663+0000 mon.a (mon.0) 843 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.998361+0000 mon.a (mon.0) 844 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.002161+0000 mon.b (mon.1) 60 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-84001-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.002384+0000 mon.b (mon.1) 61 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-84089-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.004378+0000 mon.b (mon.1) 62 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-83993-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.005564+0000 mon.a (mon.0) 845 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.005720+0000 mon.a (mon.0) 846 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.006507+0000 mon.b (mon.1) 63 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-84097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.007119+0000 mon.b (mon.1) 64 : audit [INF] from='client.? 172.21.15.162:0/1085499831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.007283+0000 mon.a (mon.0) 847 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-84001-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.007885+0000 mon.a (mon.0) 848 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-84089-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.008358+0000 mon.a (mon.0) 849 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-83993-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.012144+0000 mon.a (mon.0) 850 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-84097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.012218+0000 mon.b (mon.1) 65 : audit [INF] from='client.? 172.21.15.162:0/3769162824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.013127+0000 mon.a (mon.0) 851 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.013411+0000 mon.a (mon.0) 852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.056609+0000 mon.a (mon.0) 853 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.059269+0000 mon.a (mon.0) 854 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.060343+0000 mon.a (mon.0) 855 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-84025-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.801 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.065940+0000 mon.b (mon.1) 66 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:05.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.069037+0000 mon.a (mon.0) 856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:05.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.071945+0000 mon.b (mon.1) 67 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:05.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.078083+0000 mon.a (mon.0) 857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:05.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.089811+0000 mon.b (mon.1) 68 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.090841+0000 mon.a (mon.0) 858 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.802 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:02.115165+0000 osd.7 (osd.7) 64 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:02.115852+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:03.410165+0000 osd.2 (osd.2) 61 : cluster [DBG] 5.b scrub starts 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:03.411253+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.b scrub ok 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:03.605466+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:03.608371+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:03.976567+0000 mon.a (mon.0) 809 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:03.977609+0000 mon.a (mon.0) 810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:03.977936+0000 mon.a (mon.0) 811 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:03.978182+0000 mon.a (mon.0) 812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:03.978426+0000 mon.a (mon.0) 813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:03.986596+0000 mon.a (mon.0) 814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-84113-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]': finished 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:03.986719+0000 mon.a (mon.0) 815 : audit [INF] from='client.? 172.21.15.162:0/1514900161' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:41:05.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:03.986809+0000 mon.a (mon.0) 816 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.005979+0000 mon.b (mon.1) 49 : audit [INF] from='client.? 172.21.15.162:0/508499360' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-83977-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:02.115165+0000 osd.7 (osd.7) 64 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:41:05.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:02.115852+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:41:05.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:03.410165+0000 osd.2 (osd.2) 61 : cluster [DBG] 5.b scrub starts 2024-02-13T00:41:05.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:03.411253+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.b scrub ok 2024-02-13T00:41:05.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:03.605466+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:41:05.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:03.608371+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:41:05.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:03.976567+0000 mon.a (mon.0) 809 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:05.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:03.977609+0000 mon.a (mon.0) 810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:03.977936+0000 mon.a (mon.0) 811 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:03.978182+0000 mon.a (mon.0) 812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:03.978426+0000 mon.a (mon.0) 813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:03.986596+0000 mon.a (mon.0) 814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-84113-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]': finished 2024-02-13T00:41:05.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:03.986719+0000 mon.a (mon.0) 815 : audit [INF] from='client.? 172.21.15.162:0/1514900161' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:41:05.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:03.986809+0000 mon.a (mon.0) 816 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.005979+0000 mon.b (mon.1) 49 : audit [INF] from='client.? 172.21.15.162:0/508499360' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-83977-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.006410+0000 mon.b (mon.1) 50 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.006588+0000 mon.b (mon.1) 51 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.007018+0000 mon.b (mon.1) 52 : audit [INF] from='client.? 172.21.15.162:0/2253596229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:04.204105+0000 mgr.y (mgr.14544) 136 : cluster [DBG] pgmap v134: 652 pgs: 264 unknown, 256 creating+peering, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:41:05.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.283028+0000 mon.b (mon.1) 53 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:04.533167+0000 mon.a (mon.0) 817 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-13T00:41:05.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.533804+0000 mon.b (mon.1) 54 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.542318+0000 mon.a (mon.0) 818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-83977-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.542733+0000 mon.a (mon.0) 819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.543163+0000 mon.a (mon.0) 820 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.543571+0000 mon.a (mon.0) 821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.807 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.546851+0000 mon.b (mon.1) 55 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.807 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.546980+0000 mon.b (mon.1) 56 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.807 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.006410+0000 mon.b (mon.1) 50 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.807 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.006588+0000 mon.b (mon.1) 51 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.807 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.007018+0000 mon.b (mon.1) 52 : audit [INF] from='client.? 172.21.15.162:0/2253596229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.807 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:04.204105+0000 mgr.y (mgr.14544) 136 : cluster [DBG] pgmap v134: 652 pgs: 264 unknown, 256 creating+peering, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-13T00:41:05.807 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.283028+0000 mon.b (mon.1) 53 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.807 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:04.533167+0000 mon.a (mon.0) 817 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-13T00:41:05.807 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.533804+0000 mon.b (mon.1) 54 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.807 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.542318+0000 mon.a (mon.0) 818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-83977-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.808 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.542733+0000 mon.a (mon.0) 819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.808 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.543163+0000 mon.a (mon.0) 820 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.808 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.543571+0000 mon.a (mon.0) 821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.808 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.546851+0000 mon.b (mon.1) 55 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.808 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.546980+0000 mon.b (mon.1) 56 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.584356+0000 mon.c (mon.2) 95 : audit [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-13T00:41:05.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:04.584524+0000 client.admin (client.?) 0 : cluster [INF] onexx 2024-02-13T00:41:05.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.807852+0000 mon.a (mon.0) 822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-13T00:41:05.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.808349+0000 mon.a (mon.0) 823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.808520+0000 mon.a (mon.0) 824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.808713+0000 mon.a (mon.0) 825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.808820+0000 mon.b (mon.1) 57 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-84089-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.809074+0000 mon.a (mon.0) 826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.809118+0000 mon.b (mon.1) 58 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-83993-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.809318+0000 mon.a (mon.0) 827 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-84089-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.809563+0000 mon.a (mon.0) 828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-83993-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.860931+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? 172.21.15.162:0/1581932932' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-13T00:41:05.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.906073+0000 mon.b (mon.1) 59 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:05.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.984855+0000 mon.a (mon.0) 829 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:05.809 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.584356+0000 mon.c (mon.2) 95 : audit [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:04.584524+0000 client.admin (client.?) 0 : cluster [INF] onexx 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.807852+0000 mon.a (mon.0) 822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.808349+0000 mon.a (mon.0) 823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.808520+0000 mon.a (mon.0) 824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.808713+0000 mon.a (mon.0) 825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.808820+0000 mon.b (mon.1) 57 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-84089-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.809074+0000 mon.a (mon.0) 826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.809118+0000 mon.b (mon.1) 58 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-83993-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.809318+0000 mon.a (mon.0) 827 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-84089-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.809563+0000 mon.a (mon.0) 828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-83993-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.860931+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? 172.21.15.162:0/1581932932' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.906073+0000 mon.b (mon.1) 59 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:05.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.984855+0000 mon.a (mon.0) 829 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:05.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.985485+0000 mon.a (mon.0) 830 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:05.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.985928+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:05.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.986303+0000 mon.a (mon.0) 832 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:05.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.986696+0000 mon.a (mon.0) 833 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:05.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.987068+0000 mon.a (mon.0) 834 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:05.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.987447+0000 mon.a (mon.0) 835 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:05.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.987852+0000 mon.a (mon.0) 836 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:05.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.997118+0000 mon.a (mon.0) 837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-83977-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.997251+0000 mon.a (mon.0) 838 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.997391+0000 mon.a (mon.0) 839 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.997450+0000 mon.a (mon.0) 840 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.997526+0000 mon.a (mon.0) 841 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-84089-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.997597+0000 mon.a (mon.0) 842 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-83993-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:04.997663+0000 mon.a (mon.0) 843 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.998361+0000 mon.a (mon.0) 844 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.002161+0000 mon.b (mon.1) 60 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-84001-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.002384+0000 mon.b (mon.1) 61 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-84089-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.004378+0000 mon.b (mon.1) 62 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-83993-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.005564+0000 mon.a (mon.0) 845 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.005720+0000 mon.a (mon.0) 846 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.006507+0000 mon.b (mon.1) 63 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-84097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.813 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.007119+0000 mon.b (mon.1) 64 : audit [INF] from='client.? 172.21.15.162:0/1085499831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.813 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.007283+0000 mon.a (mon.0) 847 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-84001-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.813 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.007885+0000 mon.a (mon.0) 848 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-84089-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.985485+0000 mon.a (mon.0) 830 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:05.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.985928+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:05.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.986303+0000 mon.a (mon.0) 832 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:05.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.986696+0000 mon.a (mon.0) 833 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:05.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.987068+0000 mon.a (mon.0) 834 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:05.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.987447+0000 mon.a (mon.0) 835 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:05.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.987852+0000 mon.a (mon.0) 836 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:05.814 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.997118+0000 mon.a (mon.0) 837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-83977-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.814 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.997251+0000 mon.a (mon.0) 838 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:05.814 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.997391+0000 mon.a (mon.0) 839 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.814 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.997450+0000 mon.a (mon.0) 840 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.814 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.997526+0000 mon.a (mon.0) 841 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-84089-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.814 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.997597+0000 mon.a (mon.0) 842 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-83993-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:05.814 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.008358+0000 mon.a (mon.0) 849 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-83993-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.814 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.012144+0000 mon.a (mon.0) 850 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-84097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.814 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.012218+0000 mon.b (mon.1) 65 : audit [INF] from='client.? 172.21.15.162:0/3769162824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.814 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.013127+0000 mon.a (mon.0) 851 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.013411+0000 mon.a (mon.0) 852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.056609+0000 mon.a (mon.0) 853 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.059269+0000 mon.a (mon.0) 854 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.060343+0000 mon.a (mon.0) 855 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-84025-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.065940+0000 mon.b (mon.1) 66 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.069037+0000 mon.a (mon.0) 856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.071945+0000 mon.b (mon.1) 67 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.078083+0000 mon.a (mon.0) 857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:05.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:04.997663+0000 mon.a (mon.0) 843 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-13T00:41:05.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.998361+0000 mon.a (mon.0) 844 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:05.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.002161+0000 mon.b (mon.1) 60 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-84001-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.002384+0000 mon.b (mon.1) 61 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-84089-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.004378+0000 mon.b (mon.1) 62 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-83993-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.005564+0000 mon.a (mon.0) 845 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:05.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.005720+0000 mon.a (mon.0) 846 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.006507+0000 mon.b (mon.1) 63 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-84097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.007119+0000 mon.b (mon.1) 64 : audit [INF] from='client.? 172.21.15.162:0/1085499831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.007283+0000 mon.a (mon.0) 847 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-84001-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:05.817 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.007885+0000 mon.a (mon.0) 848 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-84089-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:05.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.089811+0000 mon.b (mon.1) 68 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.090841+0000 mon.a (mon.0) 858 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.817 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.008358+0000 mon.a (mon.0) 849 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-83993-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:05.817 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.012144+0000 mon.a (mon.0) 850 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-84097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:05.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.012218+0000 mon.b (mon.1) 65 : audit [INF] from='client.? 172.21.15.162:0/3769162824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.013127+0000 mon.a (mon.0) 851 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.013411+0000 mon.a (mon.0) 852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:05.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.056609+0000 mon.a (mon.0) 853 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:05.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.059269+0000 mon.a (mon.0) 854 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:05.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.060343+0000 mon.a (mon.0) 855 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-84025-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.065940+0000 mon.b (mon.1) 66 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:05.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.069037+0000 mon.a (mon.0) 856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:05.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.071945+0000 mon.b (mon.1) 67 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:05.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.078083+0000 mon.a (mon.0) 857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:05.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.089811+0000 mon.b (mon.1) 68 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.090841+0000 mon.a (mon.0) 858 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:05.819 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:06.053 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:06.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:06.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:06.191 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:06.213 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:06.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:06.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:06.340 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:41:06] "GET /metrics HTTP/1.1" 200 214531 "" "Prometheus/2.33.4" 2024-02-13T00:41:06.373 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:06.392 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:06.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:06.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:06.522 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:06.620 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:06.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:06.725 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:06.752 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:06.985 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.007885+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-84089-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-84089-6 delete_pools_parallel: process_1_[84229]: starting. 2024-02-13T00:41:06.985 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[84229]: creating pool ceph_test_rados_delete_pools_parallel.smithi162-84181 2024-02-13T00:41:06.985 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[84229]: created object 0... 2024-02-13T00:41:06.985 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[84229]: created object 25... 2024-02-13T00:41:06.985 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[84229]: created object 49... 2024-02-13T00:41:06.985 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[84229]: finishing. 2024-02-13T00:41:06.985 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[84229]: shutting down. 2024-02-13T00:41:06.985 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_2_[84230]: starting. 2024-02-13T00:41:06.985 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_2_[84230]: deleting pool ceph_test_rados_delete_pools_parallel.smithi162-84181 2024-02-13T00:41:06.985 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_2_[84230]: shutting down. 2024-02-13T00:41:06.985 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: ******************************* 2024-02-13T00:41:06.985 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[86553]: starting. 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[86553]: creating pool ceph_test_rados_delete_pools_parallel.smithi162-84181 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[86553]: created object 0... 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[86553]: created object 25... 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[86553]: created object 49... 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[86553]: finishing. 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[86553]: shutting down. 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: ******************************* 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_5_[86555]: starting. 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_5_[86555]: listing objects. 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_5_[86555]: listed object 0... 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_5_[86555]: listed object 25... 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_5_[86555]: saw 50 objects 2024-02-13T00:41:06.986 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_5_[86555]: shutting down. 2024-02-13T00:41:06.987 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: ******************************* 2024-02-13T00:41:06.987 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_4_[86554]: starting. 2024-02-13T00:41:06.987 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_4_[86554]: deleting pool ceph_test_rados_delete_pools_parallel.smithi162-84181 2024-02-13T00:41:06.987 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_4_[86554]: shutting down. 2024-02-13T00:41:06.987 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: ******************************* 2024-02-13T00:41:06.987 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: ******************************* 2024-02-13T00:41:06.987 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-02-13T00:41:06.987 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[84190]: starting. 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[84190]: creating pool ceph_test_rados_open_pools_parallel.smithi162-84177 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[84190]: created object 0... 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[84190]: created object 25... 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[84190]: created object 49... 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[84190]: finishing. 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[84190]: shutting down. 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_2_[84191]: starting. 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_2_[84191]: rados_pool_create. 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_2_[84191]: rados_ioctx_create. 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_2_[84191]: shutting down. 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: ******************************* 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[86572]: starting. 2024-02-13T00:41:06.988 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[86572]: creating pool ceph_test_rados_open_pools_parallel.smithi162-84177 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[86572]: created object 0... 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[86572]: created object 25... 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[86572]: created object 49... 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[86572]: finishing. 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[86572]: shutting down. 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: ******************************* 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_4_[86573]: starting. 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_4_[86573]: rados_pool_create. 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_4_[86573]: rados_ioctx_create. 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_4_[86573]: shutting down. 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: ******************************* 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: ******************************* 2024-02-13T00:41:06.989 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-02-13T00:41:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:07.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:07.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:07.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.985037+0000 mgr.y (mgr.14544) 137 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:07.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.985596+0000 mgr.y (mgr.14544) 138 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:07.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.986031+0000 mgr.y (mgr.14544) 139 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:07.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.986397+0000 mgr.y (mgr.14544) 140 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.986785+0000 mgr.y (mgr.14544) 141 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.987156+0000 mgr.y (mgr.14544) 142 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.987540+0000 mgr.y (mgr.14544) 143 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.987947+0000 mgr.y (mgr.14544) 144 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:04.998545+0000 mgr.y (mgr.14544) 145 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.006110+0000 mgr.y (mgr.14544) 146 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.317664+0000 mgr.y (mgr.14544) 147 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.552918+0000 mon.a (mon.0) 859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.558945+0000 mon.c (mon.2) 97 : audit [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:05.562741+0000 client.admin (client.?) 0 : cluster [INF] twoxx 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.562933+0000 mon.a (mon.0) 860 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.907967+0000 mon.b (mon.1) 69 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:05.998359+0000 mon.a (mon.0) 861 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.985037+0000 mgr.y (mgr.14544) 137 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.985596+0000 mgr.y (mgr.14544) 138 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.986031+0000 mgr.y (mgr.14544) 139 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.986397+0000 mgr.y (mgr.14544) 140 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.986785+0000 mgr.y (mgr.14544) 141 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.987156+0000 mgr.y (mgr.14544) 142 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.987540+0000 mgr.y (mgr.14544) 143 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.987947+0000 mgr.y (mgr.14544) 144 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:04.998545+0000 mgr.y (mgr.14544) 145 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.006110+0000 mgr.y (mgr.14544) 146 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.317664+0000 mgr.y (mgr.14544) 147 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.552918+0000 mon.a (mon.0) 859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-13T00:41:07.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.558945+0000 mon.c (mon.2) 97 : audit [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-13T00:41:07.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:05.562741+0000 client.admin (client.?) 0 : cluster [INF] twoxx 2024-02-13T00:41:07.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.562933+0000 mon.a (mon.0) 860 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-13T00:41:07.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.907967+0000 mon.b (mon.1) 69 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:07.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:05.998359+0000 mon.a (mon.0) 861 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:07.137 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:07.165 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:07.264 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:07.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:07.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:07.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.985037+0000 mgr.y (mgr.14544) 137 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.985596+0000 mgr.y (mgr.14544) 138 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.986031+0000 mgr.y (mgr.14544) 139 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.986397+0000 mgr.y (mgr.14544) 140 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.986785+0000 mgr.y (mgr.14544) 141 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.987156+0000 mgr.y (mgr.14544) 142 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.987540+0000 mgr.y (mgr.14544) 143 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.987947+0000 mgr.y (mgr.14544) 144 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:04.998545+0000 mgr.y (mgr.14544) 145 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.006110+0000 mgr.y (mgr.14544) 146 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.317664+0000 mgr.y (mgr.14544) 147 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.552918+0000 mon.a (mon.0) 859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.558945+0000 mon.c (mon.2) 97 : audit [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-13T00:41:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:05.562741+0000 client.admin (client.?) 0 : cluster [INF] twoxx 2024-02-13T00:41:07.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.562933+0000 mon.a (mon.0) 860 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-13T00:41:07.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.907967+0000 mon.b (mon.1) 69 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:07.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:05.998359+0000 mon.a (mon.0) 861 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:07.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:07.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.280+0000 7f87c4703700 1 -- 172.21.15.162:0/1960338923 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87bc0fcb30 msgr2=0x7f87bc0fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.280+0000 7f87c4703700 1 --2- 172.21.15.162:0/1960338923 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87bc0fcb30 0x7f87bc0fcf10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f87b8004870 tx=0x7f87b8009cb0).stop 2024-02-13T00:41:07.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.280+0000 7f87c4703700 1 -- 172.21.15.162:0/1960338923 shutdown_connections 2024-02-13T00:41:07.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.280+0000 7f87c4703700 1 --2- 172.21.15.162:0/1960338923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87bc0f73a0 0x7f87bc1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.280+0000 7f87c4703700 1 --2- 172.21.15.162:0/1960338923 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87bc0fcb30 0x7f87bc0fcf10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.280+0000 7f87c4703700 1 --2- 172.21.15.162:0/1960338923 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87bc0fd490 0x7f87bc0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.280+0000 7f87c4703700 1 -- 172.21.15.162:0/1960338923 >> 172.21.15.162:0/1960338923 conn(0x7f87bc06eb90 msgr2=0x7f87bc0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:07.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.280+0000 7f87c4703700 1 -- 172.21.15.162:0/1960338923 shutdown_connections 2024-02-13T00:41:07.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.280+0000 7f87c4703700 1 -- 172.21.15.162:0/1960338923 wait complete. 2024-02-13T00:41:07.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.281+0000 7f87c4703700 1 librados: starting msgr at 2024-02-13T00:41:07.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.281+0000 7f87c4703700 1 librados: starting objecter 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.281+0000 7f87c4703700 1 Processor -- start 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.281+0000 7f87c4703700 1 -- start start 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.281+0000 7f87c4703700 1 librados: setting wanted keys 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.281+0000 7f87c4703700 1 librados: calling monclient init 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c4703700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87bc0f73a0 0x7f87bc110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c4703700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87bc0fcb30 0x7f87bc110760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c4703700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87bc0fd490 0x7f87bc10a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c4703700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc10a970 con 0x7f87bc0f73a0 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c4703700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc10aae0 con 0x7f87bc0fcb30 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c4703700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc10aca0 con 0x7f87bc0fd490 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c2ca0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87bc0fd490 0x7f87bc10a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c2ca0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87bc0fd490 0x7f87bc10a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:34446/0 (socket says 172.21.15.162:34446) 2024-02-13T00:41:07.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c2ca0700 1 -- 172.21.15.162:0/2176837160 learned_addr learned my addr 172.21.15.162:0/2176837160 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c1c9e700 1 --2- 172.21.15.162:0/2176837160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87bc0fcb30 0x7f87bc110760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c2ca0700 1 -- 172.21.15.162:0/2176837160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87bc0fcb30 msgr2=0x7f87bc110760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c2ca0700 1 --2- 172.21.15.162:0/2176837160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87bc0fcb30 0x7f87bc110760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c2ca0700 1 -- 172.21.15.162:0/2176837160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87bc0f73a0 msgr2=0x7f87bc110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c2ca0700 1 --2- 172.21.15.162:0/2176837160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87bc0f73a0 0x7f87bc110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c2ca0700 1 -- 172.21.15.162:0/2176837160 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b8005040 con 0x7f87bc0fd490 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c2ca0700 1 --2- 172.21.15.162:0/2176837160 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87bc0fd490 0x7f87bc10a480 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f87b400ec20 tx=0x7f87b400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87b37fe700 10 client.?.objecter ms_handle_connect 0x7f87bc0fd490 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87b37fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4009950 con 0x7f87bc0fd490 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87bc10af20 con 0x7f87bc0fd490 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87b400f040 con 0x7f87bc0fd490 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b400c6e0 con 0x7f87bc0fd490 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c4703700 10 client.25040.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:07.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87bc1170a0 con 0x7f87bc0fd490 2024-02-13T00:41:07.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.282+0000 7f87c4703700 1 librados: init done 2024-02-13T00:41:07.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.283+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f87bc10b960 con 0x7f87bc0fd490 2024-02-13T00:41:07.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.284+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f87b40092e0 con 0x7f87bc0fd490 2024-02-13T00:41:07.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.284+0000 7f87b37fe700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6800/2866534053 conn(0x7f87a8062c90 0x7f87a8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.284+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(112..112 src has 52..112) v4 ==== 15264+0+0 (secure 0 0 0) 0x7f87b407f2a0 con 0x7f87bc0fd490 2024-02-13T00:41:07.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.284+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 osd_map(112..112 src has 52..112) v4 2024-02-13T00:41:07.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.284+0000 7f87b37fe700 3 client.25040.objecter handle_osd_map got epochs [112,112] > 0 2024-02-13T00:41:07.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.284+0000 7f87b37fe700 3 client.25040.objecter handle_osd_map decoding full epoch 112 2024-02-13T00:41:07.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c249f700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6800/2866534053 conn(0x7f87a8062c90 0x7f87a8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87b37fe700 20 client.25040.objecter dump_active .. 0 homeless 2024-02-13T00:41:07.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v112) v1 ==== 54+0+16 (secure 0 0 0) 0x7f87b407f720 con 0x7f87bc0fd490 2024-02-13T00:41:07.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87b37fe700 10 client.25040.objecter ms_handle_connect 0x7f87a8062c90 2024-02-13T00:41:07.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c249f700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6800/2866534053 conn(0x7f87a8062c90 0x7f87a8065150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f87b8005010 tx=0x7f87b800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:07.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c4703700 10 client.25040.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:07.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 1 2024-02-13T00:41:07.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6801/2103743338 conn(0x7f87bc059990 0x7f87bc059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc106020 osd=0 3 2024-02-13T00:41:07.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc106020 osd=0 3 2024-02-13T00:41:07.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc106020 osd=0 3 2024-02-13T00:41:07.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 1 2024-02-13T00:41:07.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc106020 osd=0 3 2024-02-13T00:41:07.406 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:41:07.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 0 1 2024-02-13T00:41:07.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc106020 osd=0 4 2024-02-13T00:41:07.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c4703700 10 client.25040.objecter _send_command 1 2024-02-13T00:41:07.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87bc05fb80 con 0x7f87bc059990 2024-02-13T00:41:07.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c1c9e700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6801/2103743338 conn(0x7f87bc059990 0x7f87bc059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87b37fe700 10 client.25040.objecter ms_handle_connect 0x7f87bc059990 2024-02-13T00:41:07.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.285+0000 7f87c1c9e700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6801/2103743338 conn(0x7f87bc059990 0x7f87bc059d70 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:07.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.286+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f87bc05fb80 con 0x7f87bc059990 2024-02-13T00:41:07.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.286+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:41:07.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.286+0000 7f87b37fe700 10 client.25040.objecter _finish_command 1 = system:0 2024-02-13T00:41:07.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.286+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc106020 osd=0 4 2024-02-13T00:41:07.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.286+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 0 1 2024-02-13T00:41:07.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.301+0000 7f87c4703700 10 client.25040.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:07.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.301+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 2 2024-02-13T00:41:07.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.301+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc106020 osd=0 3 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.301+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc106020 osd=0 3 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.301+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc106020 osd=0 3 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.301+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 2 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.301+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc106020 osd=0 3 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.301+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 0 2 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.301+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc106020 osd=0 4 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.301+0000 7f87c4703700 10 client.25040.objecter _send_command 2 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.301+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f87bc028590 con 0x7f87bc059990 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f87bc028590 con 0x7f87bc059990 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87b37fe700 10 client.25040.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc106020 osd=0 4 2024-02-13T00:41:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 0 2 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c4703700 10 client.25040.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 3 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6805/150599780 conn(0x7f87bc059570 0x7f87bc11b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc059400 osd=1 3 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc059400 osd=1 3 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc059400 osd=1 3 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 3 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc059400 osd=1 3 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 1 3 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc059400 osd=1 4 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c4703700 10 client.25040.objecter _send_command 3 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87bc0fa460 con 0x7f87bc059570 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c249f700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6805/150599780 conn(0x7f87bc059570 0x7f87bc11b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87b37fe700 10 client.25040.objecter ms_handle_connect 0x7f87bc059570 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.303+0000 7f87c249f700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6805/150599780 conn(0x7f87bc059570 0x7f87bc11b2b0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.304+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f87bc0fa460 con 0x7f87bc059570 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.304+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.304+0000 7f87b37fe700 10 client.25040.objecter _finish_command 3 = system:0 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.304+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc059400 osd=1 4 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.304+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 1 3 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.320+0000 7f87c4703700 10 client.25040.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.320+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 4 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.320+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc059400 osd=1 3 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.320+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc059400 osd=1 3 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.320+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc059400 osd=1 3 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.320+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 4 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.320+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc059400 osd=1 3 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.320+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 1 4 2024-02-13T00:41:07.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.320+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc059400 osd=1 4 2024-02-13T00:41:07.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.320+0000 7f87c4703700 10 client.25040.objecter _send_command 4 2024-02-13T00:41:07.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.320+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f87bc11b7a0 con 0x7f87bc059570 2024-02-13T00:41:07.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.321+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f87bc11b7a0 con 0x7f87bc059570 2024-02-13T00:41:07.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.321+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:07.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.321+0000 7f87b37fe700 10 client.25040.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.321+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc059400 osd=1 4 2024-02-13T00:41:07.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.321+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 1 4 2024-02-13T00:41:07.411 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:41:07.411 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:41:07.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c4703700 10 client.25040.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:07.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 5 2024-02-13T00:41:07.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6809/3689719376 conn(0x7f87bc0f9b30 0x7f87bc11eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc0f99c0 osd=2 3 2024-02-13T00:41:07.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc0f99c0 osd=2 3 2024-02-13T00:41:07.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc0f99c0 osd=2 3 2024-02-13T00:41:07.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 5 2024-02-13T00:41:07.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc0f99c0 osd=2 3 2024-02-13T00:41:07.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 2 5 2024-02-13T00:41:07.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc0f99c0 osd=2 4 2024-02-13T00:41:07.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c4703700 10 client.25040.objecter _send_command 5 2024-02-13T00:41:07.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87bc0280f0 con 0x7f87bc0f9b30 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c1c9e700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6809/3689719376 conn(0x7f87bc0f9b30 0x7f87bc11eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87b37fe700 10 client.25040.objecter ms_handle_connect 0x7f87bc0f9b30 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.322+0000 7f87c1c9e700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6809/3689719376 conn(0x7f87bc0f9b30 0x7f87bc11eb00 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.323+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f87bc0280f0 con 0x7f87bc0f9b30 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.323+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.323+0000 7f87b37fe700 10 client.25040.objecter _finish_command 5 = system:0 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.323+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc0f99c0 osd=2 4 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.323+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 2 5 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.337+0000 7f87c4703700 10 client.25040.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.337+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 6 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.337+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc0f99c0 osd=2 3 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.337+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc0f99c0 osd=2 3 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.337+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc0f99c0 osd=2 3 2024-02-13T00:41:07.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.337+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 6 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.337+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc0f99c0 osd=2 3 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.337+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 2 6 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.337+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc0f99c0 osd=2 4 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.337+0000 7f87c4703700 10 client.25040.objecter _send_command 6 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.337+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f87bc11eff0 con 0x7f87bc0f9b30 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f87bc11eff0 con 0x7f87bc0f9b30 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87b37fe700 10 client.25040.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc0f99c0 osd=2 4 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 2 6 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c4703700 10 client.25040.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 7 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6813/604971011 conn(0x7f87bc0277c0 0x7f87bc122380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc027650 osd=3 3 2024-02-13T00:41:07.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc027650 osd=3 3 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc027650 osd=3 3 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 7 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc027650 osd=3 3 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 3 7 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc027650 osd=3 4 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c4703700 10 client.25040.objecter _send_command 7 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87bc11c650 con 0x7f87bc0277c0 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c2ca0700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6813/604971011 conn(0x7f87bc0277c0 0x7f87bc122380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87b37fe700 10 client.25040.objecter ms_handle_connect 0x7f87bc0277c0 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.339+0000 7f87c2ca0700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6813/604971011 conn(0x7f87bc0277c0 0x7f87bc122380 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.340+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f87bc11c650 con 0x7f87bc0277c0 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.340+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.340+0000 7f87b37fe700 10 client.25040.objecter _finish_command 7 = system:0 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.340+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc027650 osd=3 4 2024-02-13T00:41:07.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.340+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 3 7 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.355+0000 7f87c4703700 10 client.25040.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.355+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 8 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.355+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc027650 osd=3 3 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.355+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc027650 osd=3 3 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.355+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc027650 osd=3 3 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.355+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 8 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.355+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc027650 osd=3 3 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.355+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 3 8 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.355+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc027650 osd=3 4 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.355+0000 7f87c4703700 10 client.25040.objecter _send_command 8 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.355+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f87bc122870 con 0x7f87bc0277c0 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.357+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f87bc122870 con 0x7f87bc0277c0 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.357+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:07.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.357+0000 7f87b37fe700 10 client.25040.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.357+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc027650 osd=3 4 2024-02-13T00:41:07.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.357+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 3 8 2024-02-13T00:41:07.417 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:41:07.417 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:41:07.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c4703700 10 client.25040.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:07.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 9 2024-02-13T00:41:07.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6800/3344263388 conn(0x7f87bc11f9c0 0x7f87bc11fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc122870 osd=4 3 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc122870 osd=4 3 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc122870 osd=4 3 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 9 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc122870 osd=4 3 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 4 9 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc122870 osd=4 4 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c4703700 10 client.25040.objecter _send_command 9 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87bc124e00 con 0x7f87bc11f9c0 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c249f700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6800/3344263388 conn(0x7f87bc11f9c0 0x7f87bc11fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87b37fe700 10 client.25040.objecter ms_handle_connect 0x7f87bc11f9c0 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.358+0000 7f87c249f700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6800/3344263388 conn(0x7f87bc11f9c0 0x7f87bc11fe20 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.359+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f87bc124e00 con 0x7f87bc11f9c0 2024-02-13T00:41:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.359+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.359+0000 7f87b37fe700 10 client.25040.objecter _finish_command 9 = system:0 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.359+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc122870 osd=4 4 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.359+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 4 9 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.383+0000 7f87c4703700 10 client.25040.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.383+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 10 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.383+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc122870 osd=4 3 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.383+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc122870 osd=4 3 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.383+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc122870 osd=4 3 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.383+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 10 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.383+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc122870 osd=4 3 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.383+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 4 10 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.383+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc122870 osd=4 4 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.383+0000 7f87c4703700 10 client.25040.objecter _send_command 10 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.383+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f87bc125020 con 0x7f87bc11f9c0 2024-02-13T00:41:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f87b8007d90 con 0x7f87bc11f9c0 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87b37fe700 10 client.25040.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc122870 osd=4 4 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 4 10 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87c4703700 10 client.25040.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 11 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6804/2212296748 conn(0x7f87bc124300 0x7f87bc124780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc123c80 osd=5 3 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc123c80 osd=5 3 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc123c80 osd=5 3 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 11 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc123c80 osd=5 3 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 5 11 2024-02-13T00:41:07.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc123c80 osd=5 4 2024-02-13T00:41:07.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87c4703700 10 client.25040.objecter _send_command 11 2024-02-13T00:41:07.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.385+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87bc124c70 con 0x7f87bc124300 2024-02-13T00:41:07.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.386+0000 7f87c1c9e700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6804/2212296748 conn(0x7f87bc124300 0x7f87bc124780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.386+0000 7f87b37fe700 10 client.25040.objecter ms_handle_connect 0x7f87bc124300 2024-02-13T00:41:07.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.386+0000 7f87c1c9e700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6804/2212296748 conn(0x7f87bc124300 0x7f87bc124780 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:07.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.387+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f87bc124c70 con 0x7f87bc124300 2024-02-13T00:41:07.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.387+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:41:07.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.387+0000 7f87b37fe700 10 client.25040.objecter _finish_command 11 = system:0 2024-02-13T00:41:07.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.387+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc123c80 osd=5 4 2024-02-13T00:41:07.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.387+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 5 11 2024-02-13T00:41:07.421 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:07.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.410+0000 7f87c4703700 10 client.25040.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:07.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.410+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 12 2024-02-13T00:41:07.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.410+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc123c80 osd=5 3 2024-02-13T00:41:07.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.410+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc123c80 osd=5 3 2024-02-13T00:41:07.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.410+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc123c80 osd=5 3 2024-02-13T00:41:07.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.410+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 12 2024-02-13T00:41:07.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.410+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc123c80 osd=5 3 2024-02-13T00:41:07.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.410+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 5 12 2024-02-13T00:41:07.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.410+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc123c80 osd=5 4 2024-02-13T00:41:07.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.410+0000 7f87c4703700 10 client.25040.objecter _send_command 12 2024-02-13T00:41:07.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.410+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f87bc124c70 con 0x7f87bc124300 2024-02-13T00:41:07.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f87bc124c70 con 0x7f87bc124300 2024-02-13T00:41:07.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:07.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87b37fe700 10 client.25040.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc123c80 osd=5 4 2024-02-13T00:41:07.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 5 12 2024-02-13T00:41:07.423 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:41:07.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87c4703700 10 client.25040.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:07.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 13 2024-02-13T00:41:07.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6808/1506640341 conn(0x7f87bc12ad70 0x7f87bc12b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc124c70 osd=6 3 2024-02-13T00:41:07.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc124c70 osd=6 3 2024-02-13T00:41:07.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc124c70 osd=6 3 2024-02-13T00:41:07.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 13 2024-02-13T00:41:07.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc124c70 osd=6 3 2024-02-13T00:41:07.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 6 13 2024-02-13T00:41:07.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc124c70 osd=6 4 2024-02-13T00:41:07.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87c4703700 10 client.25040.objecter _send_command 13 2024-02-13T00:41:07.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.413+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87bc12b7d0 con 0x7f87bc12ad70 2024-02-13T00:41:07.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.414+0000 7f87c2ca0700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6808/1506640341 conn(0x7f87bc12ad70 0x7f87bc12b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.414+0000 7f87b37fe700 10 client.25040.objecter ms_handle_connect 0x7f87bc12ad70 2024-02-13T00:41:07.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.414+0000 7f87c2ca0700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6808/1506640341 conn(0x7f87bc12ad70 0x7f87bc12b1b0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:07.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.415+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f87b407f130 con 0x7f87bc12ad70 2024-02-13T00:41:07.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.415+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:41:07.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.415+0000 7f87b37fe700 10 client.25040.objecter _finish_command 13 = system:0 2024-02-13T00:41:07.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.415+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc124c70 osd=6 4 2024-02-13T00:41:07.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.415+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 6 13 2024-02-13T00:41:07.426 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:07.426 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:07.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.438+0000 7f87c4703700 10 client.25040.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.438+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 14 2024-02-13T00:41:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.438+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc124c70 osd=6 3 2024-02-13T00:41:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.438+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc124c70 osd=6 3 2024-02-13T00:41:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.438+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc124c70 osd=6 3 2024-02-13T00:41:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.438+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 14 2024-02-13T00:41:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.438+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc124c70 osd=6 3 2024-02-13T00:41:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.438+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 6 14 2024-02-13T00:41:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.438+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc124c70 osd=6 4 2024-02-13T00:41:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.438+0000 7f87c4703700 10 client.25040.objecter _send_command 14 2024-02-13T00:41:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.438+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f87bc12da10 con 0x7f87bc12ad70 2024-02-13T00:41:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f87bc12da10 con 0x7f87bc12ad70 2024-02-13T00:41:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87b37fe700 10 client.25040.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc124c70 osd=6 4 2024-02-13T00:41:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 6 14 2024-02-13T00:41:07.443 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:41:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87c4703700 10 client.25040.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 15 2024-02-13T00:41:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6812/1964192772 conn(0x7f87bc12e160 0x7f87bc12e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc12da10 osd=7 3 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc12da10 osd=7 3 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc12da10 osd=7 3 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 15 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc12da10 osd=7 3 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 7 15 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc12da10 osd=7 4 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87c4703700 10 client.25040.objecter _send_command 15 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.440+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87bc131e70 con 0x7f87bc12e160 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.441+0000 7f87c249f700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6812/1964192772 conn(0x7f87bc12e160 0x7f87bc12e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.441+0000 7f87b37fe700 10 client.25040.objecter ms_handle_connect 0x7f87bc12e160 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.441+0000 7f87c249f700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6812/1964192772 conn(0x7f87bc12e160 0x7f87bc12e5c0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.442+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f87bc131e70 con 0x7f87bc12e160 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.442+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:41:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.442+0000 7f87b37fe700 10 client.25040.objecter _finish_command 15 = system:0 2024-02-13T00:41:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.442+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc12da10 osd=7 4 2024-02-13T00:41:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.442+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 7 15 2024-02-13T00:41:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.456+0000 7f87c4703700 10 client.25040.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.456+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign -1 16 2024-02-13T00:41:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.456+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc12da10 osd=7 3 2024-02-13T00:41:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.456+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc12da10 osd=7 3 2024-02-13T00:41:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.456+0000 7f87c4703700 20 client.25040.objecter _get_session s=0x7f87bc12da10 osd=7 3 2024-02-13T00:41:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.456+0000 7f87c4703700 15 client.25040.objecter _session_command_op_remove -1 16 2024-02-13T00:41:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.456+0000 7f87c4703700 20 client.25040.objecter get_session s=0x7f87bc12da10 osd=7 3 2024-02-13T00:41:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.456+0000 7f87c4703700 15 client.25040.objecter _session_command_op_assign 7 16 2024-02-13T00:41:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.456+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc12da10 osd=7 4 2024-02-13T00:41:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.456+0000 7f87c4703700 10 client.25040.objecter _send_command 16 2024-02-13T00:41:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.456+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f87bc131e70 con 0x7f87bc12e160 2024-02-13T00:41:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87b37fe700 1 -- 172.21.15.162:0/2176837160 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f87bc131e70 con 0x7f87bc12e160 2024-02-13T00:41:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87b37fe700 10 client.25040.objecter ms_dispatch 0x7f87bc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87b37fe700 10 client.25040.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87b37fe700 20 client.25040.objecter put_session s=0x7f87bc12da10 osd=7 4 2024-02-13T00:41:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87b37fe700 15 client.25040.objecter _session_command_op_remove 7 16 2024-02-13T00:41:07.461 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:41:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 10 librados: watch_flush enter 2024-02-13T00:41:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 10 librados: watch_flush exit 2024-02-13T00:41:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 10 client.25040.objecter close_session for osd.0 2024-02-13T00:41:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6801/2103743338 conn(0x7f87bc059990 msgr2=0x7f87bc059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6801/2103743338 conn(0x7f87bc059990 0x7f87bc059d70 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc106020 osd=0 1 2024-02-13T00:41:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 10 client.25040.objecter close_session for osd.1 2024-02-13T00:41:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6805/150599780 conn(0x7f87bc059570 msgr2=0x7f87bc11b2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6805/150599780 conn(0x7f87bc059570 0x7f87bc11b2b0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc059400 osd=1 1 2024-02-13T00:41:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 10 client.25040.objecter close_session for osd.2 2024-02-13T00:41:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6809/3689719376 conn(0x7f87bc0f9b30 msgr2=0x7f87bc11eb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6809/3689719376 conn(0x7f87bc0f9b30 0x7f87bc11eb00 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc0f99c0 osd=2 1 2024-02-13T00:41:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 10 client.25040.objecter close_session for osd.3 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6813/604971011 conn(0x7f87bc0277c0 msgr2=0x7f87bc122380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6813/604971011 conn(0x7f87bc0277c0 0x7f87bc122380 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc027650 osd=3 1 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 10 client.25040.objecter close_session for osd.4 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6800/3344263388 conn(0x7f87bc11f9c0 msgr2=0x7f87bc11fe20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6800/3344263388 conn(0x7f87bc11f9c0 0x7f87bc11fe20 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc122870 osd=4 1 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 10 client.25040.objecter close_session for osd.5 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6804/2212296748 conn(0x7f87bc124300 msgr2=0x7f87bc124780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6804/2212296748 conn(0x7f87bc124300 0x7f87bc124780 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c2ca0700 1 -- 172.21.15.162:0/2176837160 reap_dead start 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c2ca0700 1 -- 172.21.15.162:0/2176837160 reap_dead start 2024-02-13T00:41:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc123c80 osd=5 1 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 10 client.25040.objecter close_session for osd.6 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6808/1506640341 conn(0x7f87bc12ad70 msgr2=0x7f87bc12b1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6808/1506640341 conn(0x7f87bc12ad70 0x7f87bc12b1b0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc124c70 osd=6 1 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 10 client.25040.objecter close_session for osd.7 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6812/1964192772 conn(0x7f87bc12e160 msgr2=0x7f87bc12e5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6812/1964192772 conn(0x7f87bc12e160 0x7f87bc12e5c0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c2ca0700 1 -- 172.21.15.162:0/2176837160 reap_dead start 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 20 client.25040.objecter put_session s=0x7f87bc12da10 osd=7 1 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 20 client.25040.objecter shutdown clearing up homeless session... 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 10 client.25040.objecter successfully canceled tick 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6800/2866534053 conn(0x7f87a8062c90 msgr2=0x7f87a8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6800/2866534053 conn(0x7f87a8062c90 0x7f87a8065150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f87b8005010 tx=0x7f87b800b040).stop 2024-02-13T00:41:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87bc0fd490 msgr2=0x7f87bc10a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87bc0fd490 0x7f87bc10a480 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f87b400ec20 tx=0x7f87b400b040).stop 2024-02-13T00:41:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 shutdown_connections 2024-02-13T00:41:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.458+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.181:6812/1964192772 conn(0x7f87bc12e160 0x7f87bc12e5c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.459+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87bc0fd490 0x7f87bc10a480 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.459+0000 7f87c4703700 1 --2- 172.21.15.162:0/2176837160 >> v2:172.21.15.162:6800/2866534053 conn(0x7f87a8062c90 0x7f87a8065150 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.459+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 >> 172.21.15.162:0/2176837160 conn(0x7f87bc06eb90 msgr2=0x7f87bc06ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.459+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 shutdown_connections 2024-02-13T00:41:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.459+0000 7f87c4703700 1 -- 172.21.15.162:0/2176837160 wait complete. 2024-02-13T00:41:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.459+0000 7f87c4703700 1 librados: shutdown 2024-02-13T00:41:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:41:07.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.511+0000 7fb42ff4b700 1 -- 172.21.15.162:0/4027711365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4280fdc50 msgr2=0x7fb4280fe030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.511+0000 7fb42ff4b700 1 --2- 172.21.15.162:0/4027711365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4280fdc50 0x7fb4280fe030 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fb424004870 tx=0x7fb424009cb0).stop 2024-02-13T00:41:07.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.511+0000 7fb42ff4b700 1 -- 172.21.15.162:0/4027711365 shutdown_connections 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.511+0000 7fb42ff4b700 1 --2- 172.21.15.162:0/4027711365 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4280f84c0 0x7fb428104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.511+0000 7fb42ff4b700 1 --2- 172.21.15.162:0/4027711365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4280fdc50 0x7fb4280fe030 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.511+0000 7fb42ff4b700 1 --2- 172.21.15.162:0/4027711365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4280fe5b0 0x7fb4280f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.512+0000 7fb42ff4b700 1 -- 172.21.15.162:0/4027711365 >> 172.21.15.162:0/4027711365 conn(0x7fb4280f38a0 msgr2=0x7fb4280f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.512+0000 7fb42ff4b700 1 -- 172.21.15.162:0/4027711365 shutdown_connections 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.512+0000 7fb42ff4b700 1 -- 172.21.15.162:0/4027711365 wait complete. 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42ff4b700 1 librados: starting msgr at 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42ff4b700 1 librados: starting objecter 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42ff4b700 1 Processor -- start 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42ff4b700 1 -- start start 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42ff4b700 1 librados: setting wanted keys 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42ff4b700 1 librados: calling monclient init 2024-02-13T00:41:07.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42ff4b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4280f84c0 0x7fb4280febd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42ff4b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4280fdc50 0x7fb4280ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42ff4b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4280fe5b0 0x7fb4280ff5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42ff4b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb428102ca0 con 0x7fb4280fe5b0 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42ff4b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42811ac40 con 0x7fb4280f84c0 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42ff4b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42811ae00 con 0x7fb4280fdc50 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42e4e8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4280fe5b0 0x7fb4280ff5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42e4e8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4280fe5b0 0x7fb4280ff5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:53180/0 (socket says 172.21.15.162:53180) 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42e4e8700 1 -- 172.21.15.162:0/3419133734 learned_addr learned my addr 172.21.15.162:0/3419133734 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.513+0000 7fb42dce7700 1 --2- 172.21.15.162:0/3419133734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4280f84c0 0x7fb4280febd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb42dce7700 1 -- 172.21.15.162:0/3419133734 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4280fdc50 msgr2=0x7fb4280ff0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb42dce7700 1 --2- 172.21.15.162:0/3419133734 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4280fdc50 0x7fb4280ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb42dce7700 1 -- 172.21.15.162:0/3419133734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4280fe5b0 msgr2=0x7fb4280ff5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb42dce7700 1 --2- 172.21.15.162:0/3419133734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4280fe5b0 0x7fb4280ff5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb42dce7700 1 -- 172.21.15.162:0/3419133734 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4200097c0 con 0x7fb4280f84c0 2024-02-13T00:41:07.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb42dce7700 1 --2- 172.21.15.162:0/3419133734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4280f84c0 0x7fb4280febd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb424011000 tx=0x7fb424009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:07.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb41effd700 10 client.?.objecter ms_handle_connect 0x7fb4280f84c0 2024-02-13T00:41:07.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb41effd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:07.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb41effd700 1 -- 172.21.15.162:0/3419133734 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42400e030 con 0x7fb4280f84c0 2024-02-13T00:41:07.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb42ff4b700 1 -- 172.21.15.162:0/3419133734 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb424005040 con 0x7fb4280f84c0 2024-02-13T00:41:07.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb41effd700 1 -- 172.21.15.162:0/3419133734 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb424009710 con 0x7fb4280f84c0 2024-02-13T00:41:07.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb41effd700 1 -- 172.21.15.162:0/3419133734 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb424018070 con 0x7fb4280f84c0 2024-02-13T00:41:07.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb42ff4b700 10 client.25111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:07.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb42ff4b700 1 -- 172.21.15.162:0/3419133734 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42811b380 con 0x7fb4280f84c0 2024-02-13T00:41:07.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.514+0000 7fb42ff4b700 1 librados: init done 2024-02-13T00:41:07.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.516+0000 7fb42ff4b700 1 -- 172.21.15.162:0/3419133734 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40c004f50 con 0x7fb4280f84c0 2024-02-13T00:41:07.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.517+0000 7fb41effd700 1 -- 172.21.15.162:0/3419133734 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fb4240090d0 con 0x7fb4280f84c0 2024-02-13T00:41:07.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.517+0000 7fb41effd700 1 --2- 172.21.15.162:0/3419133734 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb414062c90 0x7fb414065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:07.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.517+0000 7fb41effd700 1 -- 172.21.15.162:0/3419133734 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(112..112 src has 52..112) v4 ==== 15264+0+0 (secure 0 0 0) 0x7fb42407fd30 con 0x7fb4280f84c0 2024-02-13T00:41:07.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.517+0000 7fb41effd700 10 client.25111.objecter ms_dispatch 0x7fb428001550 osd_map(112..112 src has 52..112) v4 2024-02-13T00:41:07.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.517+0000 7fb41effd700 3 client.25111.objecter handle_osd_map got epochs [112,112] > 0 2024-02-13T00:41:07.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.517+0000 7fb41effd700 3 client.25111.objecter handle_osd_map decoding full epoch 112 2024-02-13T00:41:07.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.517+0000 7fb42d4e6700 1 --2- 172.21.15.162:0/3419133734 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb414062c90 0x7fb414065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:07.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.517+0000 7fb41effd700 20 client.25111.objecter dump_active .. 0 homeless 2024-02-13T00:41:07.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.518+0000 7fb41effd700 10 client.25111.objecter ms_handle_connect 0x7fb414062c90 2024-02-13T00:41:07.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.518+0000 7fb42d4e6700 1 --2- 172.21.15.162:0/3419133734 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb414062c90 0x7fb414065150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb428100370 tx=0x7fb41800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:07.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.521+0000 7fb41effd700 1 -- 172.21.15.162:0/3419133734 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb424084070 con 0x7fb4280f84c0 2024-02-13T00:41:07.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:07.559 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:07.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.698+0000 7fb42ff4b700 1 -- 172.21.15.162:0/3419133734 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb40c000bf0 con 0x7fb414062c90 2024-02-13T00:41:07.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.735+0000 7fb41effd700 1 -- 172.21.15.162:0/3419133734 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+1315955 (secure 0 0 0) 0x7fb40c000bf0 con 0x7fb414062c90 2024-02-13T00:41:07.755 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:41:07.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 10 librados: watch_flush enter 2024-02-13T00:41:07.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 10 librados: watch_flush exit 2024-02-13T00:41:07.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 20 client.25111.objecter shutdown clearing up homeless session... 2024-02-13T00:41:07.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 10 client.25111.objecter successfully canceled tick 2024-02-13T00:41:07.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 -- 172.21.15.162:0/3419133734 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb414062c90 msgr2=0x7fb414065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 --2- 172.21.15.162:0/3419133734 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb414062c90 0x7fb414065150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb428100370 tx=0x7fb41800b040).stop 2024-02-13T00:41:07.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 -- 172.21.15.162:0/3419133734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4280f84c0 msgr2=0x7fb4280febd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:07.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 --2- 172.21.15.162:0/3419133734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4280f84c0 0x7fb4280febd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb424011000 tx=0x7fb424009cb0).stop 2024-02-13T00:41:07.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 -- 172.21.15.162:0/3419133734 shutdown_connections 2024-02-13T00:41:07.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 --2- 172.21.15.162:0/3419133734 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb414062c90 0x7fb414065150 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 --2- 172.21.15.162:0/3419133734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4280fe5b0 0x7fb4280ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 --2- 172.21.15.162:0/3419133734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4280f84c0 0x7fb4280febd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:07.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 --2- 172.21.15.162:0/3419133734 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4280fdc50 0x7fb4280ff0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:07.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 -- 172.21.15.162:0/3419133734 >> 172.21.15.162:0/3419133734 conn(0x7fb4280f38a0 msgr2=0x7fb4280f52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:07.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 -- 172.21.15.162:0/3419133734 shutdown_connections 2024-02-13T00:41:07.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 -- 172.21.15.162:0/3419133734 wait complete. 2024-02-13T00:41:07.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:07.742+0000 7fb42ff4b700 1 librados: shutdown 2024-02-13T00:41:07.782 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:41:08.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:06.204652+0000 mgr.y (mgr.14544) 148 : cluster [DBG] pgmap v137: 484 pgs: 96 creating+peering, 388 active+clean; 144 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-02-13T00:41:08.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:06.724459+0000 osd.3 (osd.3) 83 : cluster [DBG] 20.0 deep-scrub starts 2024-02-13T00:41:08.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:06.727492+0000 osd.3 (osd.3) 84 : cluster [DBG] 20.0 deep-scrub ok 2024-02-13T00:41:08.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:06.909552+0000 mon.b (mon.1) 70 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:08.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:06.973575+0000 mon.a (mon.0) 862 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:08.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:06.973666+0000 mon.a (mon.0) 863 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:08.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:06.973760+0000 mon.a (mon.0) 864 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-84025-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:08.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:06.973820+0000 mon.a (mon.0) 865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:08.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:06.973964+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-13T00:41:08.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:06.974341+0000 mon.b (mon.1) 71 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-84033-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:08.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:06.975012+0000 mon.c (mon.2) 98 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:08.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:06.976469+0000 mon.a (mon.0) 867 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-84025-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:08.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:06.981650+0000 mon.a (mon.0) 868 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-13T00:41:08.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:07.236845+0000 mon.a (mon.0) 869 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]: dispatch 2024-02-13T00:41:08.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:07.242909+0000 mon.a (mon.0) 870 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-84001-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-84001-22"}]': finished 2024-02-13T00:41:08.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:07.243008+0000 mon.a (mon.0) 871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-84089-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-84089-6"}]': finished 2024-02-13T00:41:08.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:07.243082+0000 mon.a (mon.0) 872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-83993-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-83993-15"}]': finished 2024-02-13T00:41:08.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:07.243157+0000 mon.a (mon.0) 873 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-84097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-84097-7"}]': finished 2024-02-13T00:41:08.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:07.284811+0000 mon.c (mon.2) 99 : audit [DBG] from='client.? 172.21.15.162:0/2176837160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:41:08.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:07.508250+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-13T00:41:08.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:07.508845+0000 mon.c (mon.2) 100 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:08.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:07.514451+0000 mon.a (mon.0) 875 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:08.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:41:08.293 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:06.204652+0000 mgr.y (mgr.14544) 148 : cluster [DBG] pgmap v137: 484 pgs: 96 creating+peering, 388 active+clean; 144 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-02-13T00:41:08.294 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:06.724459+0000 osd.3 (osd.3) 83 : cluster [DBG] 20.0 deep-scrub starts 2024-02-13T00:41:08.294 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:06.727492+0000 osd.3 (osd.3) 84 : cluster [DBG] 20.0 deep-scrub ok 2024-02-13T00:41:08.294 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:06.909552+0000 mon.b (mon.1) 70 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:08.294 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:06.973575+0000 mon.a (mon.0) 862 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:08.294 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:06.973666+0000 mon.a (mon.0) 863 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:08.294 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:06.973760+0000 mon.a (mon.0) 864 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-84025-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:08.294 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:06.973820+0000 mon.a (mon.0) 865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:08.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:06.973964+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-13T00:41:08.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:06.974341+0000 mon.b (mon.1) 71 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-84033-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:08.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:06.975012+0000 mon.c (mon.2) 98 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:08.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:06.976469+0000 mon.a (mon.0) 867 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-84025-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:08.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:06.981650+0000 mon.a (mon.0) 868 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-13T00:41:08.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:07.236845+0000 mon.a (mon.0) 869 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]: dispatch 2024-02-13T00:41:08.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:07.242909+0000 mon.a (mon.0) 870 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-84001-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-84001-22"}]': finished 2024-02-13T00:41:08.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:07.243008+0000 mon.a (mon.0) 871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-84089-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-84089-6"}]': finished 2024-02-13T00:41:08.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:07.243082+0000 mon.a (mon.0) 872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-83993-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-83993-15"}]': finished 2024-02-13T00:41:08.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:07.243157+0000 mon.a (mon.0) 873 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-84097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-84097-7"}]': finished 2024-02-13T00:41:08.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:07.284811+0000 mon.c (mon.2) 99 : audit [DBG] from='client.? 172.21.15.162:0/2176837160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:41:08.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:07.508250+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-13T00:41:08.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:07.508845+0000 mon.c (mon.2) 100 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:08.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:07.514451+0000 mon.a (mon.0) 875 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:08.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:06.204652+0000 mgr.y (mgr.14544) 148 : cluster [DBG] pgmap v137: 484 pgs: 96 creating+peering, 388 active+clean; 144 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-02-13T00:41:08.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:06.724459+0000 osd.3 (osd.3) 83 : cluster [DBG] 20.0 deep-scrub starts 2024-02-13T00:41:08.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:06.727492+0000 osd.3 (osd.3) 84 : cluster [DBG] 20.0 deep-scrub ok 2024-02-13T00:41:08.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:06.909552+0000 mon.b (mon.1) 70 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:08.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:06.973575+0000 mon.a (mon.0) 862 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:08.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:06.973666+0000 mon.a (mon.0) 863 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:08.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:06.973760+0000 mon.a (mon.0) 864 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-84025-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:08.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:06.973820+0000 mon.a (mon.0) 865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:08.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:06.973964+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-13T00:41:08.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:06.974341+0000 mon.b (mon.1) 71 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-84033-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:08.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:06.975012+0000 mon.c (mon.2) 98 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:08.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:06.976469+0000 mon.a (mon.0) 867 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-84025-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:08.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:06.981650+0000 mon.a (mon.0) 868 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-13T00:41:08.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:07.236845+0000 mon.a (mon.0) 869 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]: dispatch 2024-02-13T00:41:08.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:07.242909+0000 mon.a (mon.0) 870 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-84001-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-84001-22"}]': finished 2024-02-13T00:41:08.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:07.243008+0000 mon.a (mon.0) 871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-84089-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-84089-6"}]': finished 2024-02-13T00:41:08.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:07.243082+0000 mon.a (mon.0) 872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-83993-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-83993-15"}]': finished 2024-02-13T00:41:08.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:07.243157+0000 mon.a (mon.0) 873 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-84097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-84097-7"}]': finished 2024-02-13T00:41:08.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:07.284811+0000 mon.c (mon.2) 99 : audit [DBG] from='client.? 172.21.15.162:0/2176837160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:41:08.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:07.508250+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-13T00:41:08.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:07.508845+0000 mon.c (mon.2) 100 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:08.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:07.514451+0000 mon.a (mon.0) 875 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:08.601 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-02-13T00:41:07.240014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":151008235,"num_objects":282,"num_object_clones":0,"num_object_copies":846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":2301,"num_read_kb":50168,"num_write":1787,"num_write_kb":153097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1946,"ondisk_log_size":1946,"up":1452,"acting":1452,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1976,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2651308,"kb_used_data":228652,"kb_used_omap":0,"kb_used_meta":2421760,"kb_avail":747309908,"statfs":{"total":767960285184,"available":765245345792,"internally_reserved":0,"allocated":234139648,"data_stored":230052467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2479882240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":151002439,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":1025,"num_read_kb":49172,"num_write":1167,"num_write_kb":153056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.754056"},"pg_stats":[{"pgid":"45.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.025553+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.040215+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.041560+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.058899+0000","last_change":"2024-02-13T00:41:04.056407+0000","last_active":"2024-02-13T00:41:04.058899+0000","last_peered":"2024-02-13T00:41:04.058899+0000","last_clean":"2024-02-13T00:41:04.058899+0000","last_became_active":"2024-02-13T00:41:04.056131+0000","last_became_peered":"2024-02-13T00:41:04.056131+0000","last_unstale":"2024-02-13T00:41:04.058899+0000","last_undegraded":"2024-02-13T00:41:04.058899+0000","last_fullsized":"2024-02-13T00:41:04.058899+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.999643+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.018166+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.006664+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.042777+0000","last_change":"2024-02-13T00:41:04.033131+0000","last_active":"2024-02-13T00:41:04.042777+0000","last_peered":"2024-02-13T00:41:04.042777+0000","last_clean":"2024-02-13T00:41:04.042777+0000","last_became_active":"2024-02-13T00:41:04.032927+0000","last_became_peered":"2024-02-13T00:41:04.032927+0000","last_unstale":"2024-02-13T00:41:04.042777+0000","last_undegraded":"2024-02-13T00:41:04.042777+0000","last_fullsized":"2024-02-13T00:41:04.042777+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"45.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.000098+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.006543+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.995067+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.052258+0000","last_change":"2024-02-13T00:41:04.045017+0000","last_active":"2024-02-13T00:41:04.052258+0000","last_peered":"2024-02-13T00:41:04.052258+0000","last_clean":"2024-02-13T00:41:04.052258+0000","last_became_active":"2024-02-13T00:41:04.044755+0000","last_became_peered":"2024-02-13T00:41:04.044755+0000","last_unstale":"2024-02-13T00:41:04.052258+0000","last_undegraded":"2024-02-13T00:41:04.052258+0000","last_fullsized":"2024-02-13T00:41:04.052258+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.020551+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.039452+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.017671+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.062291+0000","last_change":"2024-02-13T00:41:04.055128+0000","last_active":"2024-02-13T00:41:04.062291+0000","last_peered":"2024-02-13T00:41:04.062291+0000","last_clean":"2024-02-13T00:41:04.062291+0000","last_became_active":"2024-02-13T00:41:04.054793+0000","last_became_peered":"2024-02-13T00:41:04.054793+0000","last_unstale":"2024-02-13T00:41:04.062291+0000","last_undegraded":"2024-02-13T00:41:04.062291+0000","last_fullsized":"2024-02-13T00:41:04.062291+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.039677+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.005082+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.026031+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.038771+0000","last_change":"2024-02-13T00:41:04.037910+0000","last_active":"2024-02-13T00:41:04.038771+0000","last_peered":"2024-02-13T00:41:04.038771+0000","last_clean":"2024-02-13T00:41:04.038771+0000","last_became_active":"2024-02-13T00:41:04.037588+0000","last_became_peered":"2024-02-13T00:41:04.037588+0000","last_unstale":"2024-02-13T00:41:04.038771+0000","last_undegraded":"2024-02-13T00:41:04.038771+0000","last_fullsized":"2024-02-13T00:41:04.038771+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.005734+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.014899+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.020787+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.046736+0000","last_change":"2024-02-13T00:41:04.030292+0000","last_active":"2024-02-13T00:41:04.046736+0000","last_peered":"2024-02-13T00:41:04.046736+0000","last_clean":"2024-02-13T00:41:04.046736+0000","last_became_active":"2024-02-13T00:41:04.030024+0000","last_became_peered":"2024-02-13T00:41:04.030024+0000","last_unstale":"2024-02-13T00:41:04.046736+0000","last_undegraded":"2024-02-13T00:41:04.046736+0000","last_fullsized":"2024-02-13T00:41:04.046736+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.005850+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.018148+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.996888+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.047058+0000","last_change":"2024-02-13T00:41:04.046857+0000","last_active":"2024-02-13T00:41:04.047058+0000","last_peered":"2024-02-13T00:41:04.047058+0000","last_clean":"2024-02-13T00:41:04.047058+0000","last_became_active":"2024-02-13T00:41:04.046595+0000","last_became_peered":"2024-02-13T00:41:04.046595+0000","last_unstale":"2024-02-13T00:41:04.047058+0000","last_undegraded":"2024-02-13T00:41:04.047058+0000","last_fullsized":"2024-02-13T00:41:04.047058+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"45.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.002920+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.010496+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.997284+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.032458+0000","last_change":"2024-02-13T00:41:04.031034+0000","last_active":"2024-02-13T00:41:04.032458+0000","last_peered":"2024-02-13T00:41:04.032458+0000","last_clean":"2024-02-13T00:41:04.032458+0000","last_became_active":"2024-02-13T00:41:04.030752+0000","last_became_peered":"2024-02-13T00:41:04.030752+0000","last_unstale":"2024-02-13T00:41:04.032458+0000","last_undegraded":"2024-02-13T00:41:04.032458+0000","last_fullsized":"2024-02-13T00:41:04.032458+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.000353+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.997012+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.015729+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.044285+0000","last_change":"2024-02-13T00:41:04.036439+0000","last_active":"2024-02-13T00:41:04.044285+0000","last_peered":"2024-02-13T00:41:04.044285+0000","last_clean":"2024-02-13T00:41:04.044285+0000","last_became_active":"2024-02-13T00:41:04.031999+0000","last_became_peered":"2024-02-13T00:41:04.031999+0000","last_unstale":"2024-02-13T00:41:04.044285+0000","last_undegraded":"2024-02-13T00:41:04.044285+0000","last_fullsized":"2024-02-13T00:41:04.044285+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.024346+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.041890+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.035512+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.062229+0000","last_change":"2024-02-13T00:41:04.051970+0000","last_active":"2024-02-13T00:41:04.062229+0000","last_peered":"2024-02-13T00:41:04.062229+0000","last_clean":"2024-02-13T00:41:04.062229+0000","last_became_active":"2024-02-13T00:41:04.051581+0000","last_became_peered":"2024-02-13T00:41:04.051581+0000","last_unstale":"2024-02-13T00:41:04.062229+0000","last_undegraded":"2024-02-13T00:41:04.062229+0000","last_fullsized":"2024-02-13T00:41:04.062229+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.011645+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.996957+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.008317+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.057845+0000","last_change":"2024-02-13T00:41:04.049180+0000","last_active":"2024-02-13T00:41:04.057845+0000","last_peered":"2024-02-13T00:41:04.057845+0000","last_clean":"2024-02-13T00:41:04.057845+0000","last_became_active":"2024-02-13T00:41:04.048960+0000","last_became_peered":"2024-02-13T00:41:04.048960+0000","last_unstale":"2024-02-13T00:41:04.057845+0000","last_undegraded":"2024-02-13T00:41:04.057845+0000","last_fullsized":"2024-02-13T00:41:04.057845+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.021130+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.004809+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.017315+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.053922+0000","last_change":"2024-02-13T00:41:04.053689+0000","last_active":"2024-02-13T00:41:04.053922+0000","last_peered":"2024-02-13T00:41:04.053922+0000","last_clean":"2024-02-13T00:41:04.053922+0000","last_became_active":"2024-02-13T00:41:04.053468+0000","last_became_peered":"2024-02-13T00:41:04.053468+0000","last_unstale":"2024-02-13T00:41:04.053922+0000","last_undegraded":"2024-02-13T00:41:04.053922+0000","last_fullsized":"2024-02-13T00:41:04.053922+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.008963+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.997376+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.010909+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.063311+0000","last_change":"2024-02-13T00:41:04.054158+0000","last_active":"2024-02-13T00:41:04.063311+0000","last_peered":"2024-02-13T00:41:04.063311+0000","last_clean":"2024-02-13T00:41:04.063311+0000","last_became_active":"2024-02-13T00:41:04.053910+0000","last_became_peered":"2024-02-13T00:41:04.053910+0000","last_unstale":"2024-02-13T00:41:04.063311+0000","last_undegraded":"2024-02-13T00:41:04.063311+0000","last_fullsized":"2024-02-13T00:41:04.063311+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.000208+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.011275+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.020439+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.035217+0000","last_change":"2024-02-13T00:41:04.033459+0000","last_active":"2024-02-13T00:41:04.035217+0000","last_peered":"2024-02-13T00:41:04.035217+0000","last_clean":"2024-02-13T00:41:04.035217+0000","last_became_active":"2024-02-13T00:41:04.033252+0000","last_became_peered":"2024-02-13T00:41:04.033252+0000","last_unstale":"2024-02-13T00:41:04.035217+0000","last_undegraded":"2024-02-13T00:41:04.035217+0000","last_fullsized":"2024-02-13T00:41:04.035217+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"45.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.021402+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.010868+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.018258+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.027294+0000","last_change":"2024-02-13T00:41:04.025698+0000","last_active":"2024-02-13T00:41:04.027294+0000","last_peered":"2024-02-13T00:41:04.027294+0000","last_clean":"2024-02-13T00:41:04.027294+0000","last_became_active":"2024-02-13T00:41:04.025425+0000","last_became_peered":"2024-02-13T00:41:04.025425+0000","last_unstale":"2024-02-13T00:41:04.027294+0000","last_undegraded":"2024-02-13T00:41:04.027294+0000","last_fullsized":"2024-02-13T00:41:04.027294+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.000066+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.018793+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.996932+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.062044+0000","last_change":"2024-02-13T00:41:04.053357+0000","last_active":"2024-02-13T00:41:04.062044+0000","last_peered":"2024-02-13T00:41:04.062044+0000","last_clean":"2024-02-13T00:41:04.062044+0000","last_became_active":"2024-02-13T00:41:04.053121+0000","last_became_peered":"2024-02-13T00:41:04.053121+0000","last_unstale":"2024-02-13T00:41:04.062044+0000","last_undegraded":"2024-02-13T00:41:04.062044+0000","last_fullsized":"2024-02-13T00:41:04.062044+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"44.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"43.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.020010+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.013527+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.041250+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.033278+0000","last_change":"2024-02-13T00:41:04.031545+0000","last_active":"2024-02-13T00:41:04.033278+0000","last_peered":"2024-02-13T00:41:04.033278+0000","last_clean":"2024-02-13T00:41:04.033278+0000","last_became_active":"2024-02-13T00:41:04.030586+0000","last_became_peered":"2024-02-13T00:41:04.030586+0000","last_unstale":"2024-02-13T00:41:04.033278+0000","last_undegraded":"2024-02-13T00:41:04.033278+0000","last_fullsized":"2024-02-13T00:41:04.033278+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"44.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"43.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.016275+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.016441+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.001471+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.062210+0000","last_change":"2024-02-13T00:41:04.051881+0000","last_active":"2024-02-13T00:41:04.062210+0000","last_peered":"2024-02-13T00:41:04.062210+0000","last_clean":"2024-02-13T00:41:04.062210+0000","last_became_active":"2024-02-13T00:41:04.051389+0000","last_became_peered":"2024-02-13T00:41:04.051389+0000","last_unstale":"2024-02-13T00:41:04.062210+0000","last_undegraded":"2024-02-13T00:41:04.062210+0000","last_fullsized":"2024-02-13T00:41:04.062210+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"44.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"43.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.020229+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.032638+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.007996+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.050402+0000","last_change":"2024-02-13T00:41:04.050170+0000","last_active":"2024-02-13T00:41:04.050402+0000","last_peered":"2024-02-13T00:41:04.050402+0000","last_clean":"2024-02-13T00:41:04.050402+0000","last_became_active":"2024-02-13T00:41:04.049853+0000","last_became_peered":"2024-02-13T00:41:04.049853+0000","last_unstale":"2024-02-13T00:41:04.050402+0000","last_undegraded":"2024-02-13T00:41:04.050402+0000","last_fullsized":"2024-02-13T00:41:04.050402+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"44.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"43.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.018374+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.008629+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.011740+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.044149+0000","last_change":"2024-02-13T00:41:04.041987+0000","last_active":"2024-02-13T00:41:04.044149+0000","last_peered":"2024-02-13T00:41:04.044149+0000","last_clean":"2024-02-13T00:41:04.044149+0000","last_became_active":"2024-02-13T00:41:04.041712+0000","last_became_peered":"2024-02-13T00:41:04.041712+0000","last_unstale":"2024-02-13T00:41:04.044149+0000","last_undegraded":"2024-02-13T00:41:04.044149+0000","last_fullsized":"2024-02-13T00:41:04.044149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"45.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"44.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"43.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.996234+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.004827+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.996831+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.033142+0000","last_change":"2024-02-13T00:41:04.031580+0000","last_active":"2024-02-13T00:41:04.033142+0000","last_peered":"2024-02-13T00:41:04.033142+0000","last_clean":"2024-02-13T00:41:04.033142+0000","last_became_active":"2024-02-13T00:41:04.031311+0000","last_became_peered":"2024-02-13T00:41:04.031311+0000","last_unstale":"2024-02-13T00:41:04.033142+0000","last_undegraded":"2024-02-13T00:41:04.033142+0000","last_fullsized":"2024-02-13T00:41:04.033142+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"45.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"44.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"43.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.032973+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.026382+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.007826+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.046716+0000","last_change":"2024-02-13T00:41:04.029791+0000","last_active":"2024-02-13T00:41:04.046716+0000","last_peered":"2024-02-13T00:41:04.046716+0000","last_clean":"2024-02-13T00:41:04.046716+0000","last_became_active":"2024-02-13T00:41:04.021709+0000","last_became_peered":"2024-02-13T00:41:04.021709+0000","last_unstale":"2024-02-13T00:41:04.046716+0000","last_undegraded":"2024-02-13T00:41:04.046716+0000","last_fullsized":"2024-02-13T00:41:04.046716+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"44.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"43.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.000960+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.041477+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.010439+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.050411+0000","last_change":"2024-02-13T00:41:04.050157+0000","last_active":"2024-02-13T00:41:04.050411+0000","last_peered":"2024-02-13T00:41:04.050411+0000","last_clean":"2024-02-13T00:41:04.050411+0000","last_became_active":"2024-02-13T00:41:04.049853+0000","last_became_peered":"2024-02-13T00:41:04.049853+0000","last_unstale":"2024-02-13T00:41:04.050411+0000","last_undegraded":"2024-02-13T00:41:04.050411+0000","last_fullsized":"2024-02-13T00:41:04.050411+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"44.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"43.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.009461+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.997198+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.015962+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.027112+0000","last_change":"2024-02-13T00:41:04.020729+0000","last_active":"2024-02-13T00:41:04.027112+0000","last_peered":"2024-02-13T00:41:04.027112+0000","last_clean":"2024-02-13T00:41:04.027112+0000","last_became_active":"2024-02-13T00:41:04.020444+0000","last_became_peered":"2024-02-13T00:41:04.020444+0000","last_unstale":"2024-02-13T00:41:04.027112+0000","last_undegraded":"2024-02-13T00:41:04.027112+0000","last_fullsized":"2024-02-13T00:41:04.027112+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"45.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.025224+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.003621+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.995832+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.056339+0000","last_change":"2024-02-13T00:41:04.055275+0000","last_active":"2024-02-13T00:41:04.056339+0000","last_peered":"2024-02-13T00:41:04.056339+0000","last_clean":"2024-02-13T00:41:04.056339+0000","last_became_active":"2024-02-13T00:41:04.055025+0000","last_became_peered":"2024-02-13T00:41:04.055025+0000","last_unstale":"2024-02-13T00:41:04.056339+0000","last_undegraded":"2024-02-13T00:41:04.056339+0000","last_fullsized":"2024-02-13T00:41:04.056339+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"45.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.029495+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.016130+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.001099+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.033133+0000","last_change":"2024-02-13T00:41:04.032732+0000","last_active":"2024-02-13T00:41:04.033133+0000","last_peered":"2024-02-13T00:41:04.033133+0000","last_clean":"2024-02-13T00:41:04.033133+0000","last_became_active":"2024-02-13T00:41:04.032529+0000","last_became_peered":"2024-02-13T00:41:04.032529+0000","last_unstale":"2024-02-13T00:41:04.033133+0000","last_undegraded":"2024-02-13T00:41:04.033133+0000","last_fullsized":"2024-02-13T00:41:04.033133+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"45.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.999748+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.032326+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.010643+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.032372+0000","last_change":"2024-02-13T00:41:04.030610+0000","last_active":"2024-02-13T00:41:04.032372+0000","last_peered":"2024-02-13T00:41:04.032372+0000","last_clean":"2024-02-13T00:41:04.032372+0000","last_became_active":"2024-02-13T00:41:04.030313+0000","last_became_peered":"2024-02-13T00:41:04.030313+0000","last_unstale":"2024-02-13T00:41:04.032372+0000","last_undegraded":"2024-02-13T00:41:04.032372+0000","last_fullsized":"2024-02-13T00:41:04.032372+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.011176+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.014713+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.032000+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.047300+0000","last_change":"2024-02-13T00:41:04.040932+0000","last_active":"2024-02-13T00:41:04.047300+0000","last_peered":"2024-02-13T00:41:04.047300+0000","last_clean":"2024-02-13T00:41:04.047300+0000","last_became_active":"2024-02-13T00:41:04.037109+0000","last_became_peered":"2024-02-13T00:41:04.037109+0000","last_unstale":"2024-02-13T00:41:04.047300+0000","last_undegraded":"2024-02-13T00:41:04.047300+0000","last_fullsized":"2024-02-13T00:41:04.047300+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.015241+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.997070+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.996465+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.052032+0000","last_change":"2024-02-13T00:41:04.031617+0000","last_active":"2024-02-13T00:41:04.052032+0000","last_peered":"2024-02-13T00:41:04.052032+0000","last_clean":"2024-02-13T00:41:04.052032+0000","last_became_active":"2024-02-13T00:41:04.031190+0000","last_became_peered":"2024-02-13T00:41:04.031190+0000","last_unstale":"2024-02-13T00:41:04.052032+0000","last_undegraded":"2024-02-13T00:41:04.052032+0000","last_fullsized":"2024-02-13T00:41:04.052032+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"45.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.033294+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.004686+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.020100+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.035195+0000","last_change":"2024-02-13T00:41:04.030017+0000","last_active":"2024-02-13T00:41:04.035195+0000","last_peered":"2024-02-13T00:41:04.035195+0000","last_clean":"2024-02-13T00:41:04.035195+0000","last_became_active":"2024-02-13T00:41:04.029868+0000","last_became_peered":"2024-02-13T00:41:04.029868+0000","last_unstale":"2024-02-13T00:41:04.035195+0000","last_undegraded":"2024-02-13T00:41:04.035195+0000","last_fullsized":"2024-02-13T00:41:04.035195+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"45.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.028063+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.010521+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.018423+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.040483+0000","last_change":"2024-02-13T00:41:04.027098+0000","last_active":"2024-02-13T00:41:04.040483+0000","last_peered":"2024-02-13T00:41:04.040483+0000","last_clean":"2024-02-13T00:41:04.040483+0000","last_became_active":"2024-02-13T00:41:04.026851+0000","last_became_peered":"2024-02-13T00:41:04.026851+0000","last_unstale":"2024-02-13T00:41:04.040483+0000","last_undegraded":"2024-02-13T00:41:04.040483+0000","last_fullsized":"2024-02-13T00:41:04.040483+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"45.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:07.234414+0000","last_change":"2024-02-13T00:41:07.234414+0000","last_active":"2024-02-13T00:41:07.234414+0000","last_peered":"2024-02-13T00:41:07.234414+0000","last_clean":"2024-02-13T00:41:07.234414+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:07.234414+0000","last_undegraded":"2024-02-13T00:41:07.234414+0000","last_fullsized":"2024-02-13T00:41:07.234414+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.999622+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:05.010766+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:04.983035+0000","last_change":"2024-02-13T00:41:04.996577+0000","last_active":"2024-02-13T00:41:04.983035+0000","last_peered":"2024-02-13T00:41:04.983035+0000","last_clean":"2024-02-13T00:41:04.983035+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:04.983035+0000","last_undegraded":"2024-02-13T00:41:04.983035+0000","last_fullsized":"2024-02-13T00:41:04.983035+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:04.983035+0000","last_clean_scrub_stamp":"2024-02-13T00:41:04.983035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":10,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.045347+0000","last_change":"2024-02-13T00:41:04.032098+0000","last_active":"2024-02-13T00:41:04.045347+0000","last_peered":"2024-02-13T00:41:04.045347+0000","last_clean":"2024-02-13T00:41:04.045347+0000","last_became_active":"2024-02-13T00:41:04.031643+0000","last_became_peered":"2024-02-13T00:41:04.031643+0000","last_unstale":"2024-02-13T00:41:04.045347+0000","last_undegraded":"2024-02-13T00:41:04.045347+0000","last_fullsized":"2024-02-13T00:41:04.045347+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:02.465076+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.465076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.999334+0000","last_change":"2024-02-13T00:41:01.591014+0000","last_active":"2024-02-13T00:41:03.999334+0000","last_peered":"2024-02-13T00:41:03.999334+0000","last_clean":"2024-02-13T00:41:03.999334+0000","last_became_active":"2024-02-13T00:41:01.590358+0000","last_became_peered":"2024-02-13T00:41:01.590358+0000","last_unstale":"2024-02-13T00:41:03.999334+0000","last_undegraded":"2024-02-13T00:41:03.999334+0000","last_fullsized":"2024-02-13T00:41:03.999334+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.021269+0000","last_change":"2024-02-13T00:41:04.021269+0000","last_active":"2024-02-13T00:41:04.021269+0000","last_peered":"2024-02-13T00:41:04.021269+0000","last_clean":"2024-02-13T00:41:04.021269+0000","last_became_active":"2024-02-13T00:41:01.696882+0000","last_became_peered":"2024-02-13T00:41:01.696882+0000","last_unstale":"2024-02-13T00:41:04.021269+0000","last_undegraded":"2024-02-13T00:41:04.021269+0000","last_fullsized":"2024-02-13T00:41:04.021269+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.7","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.006143+0000","last_change":"2024-02-13T00:41:01.623125+0000","last_active":"2024-02-13T00:41:04.006143+0000","last_peered":"2024-02-13T00:41:04.006143+0000","last_clean":"2024-02-13T00:41:04.006143+0000","last_became_active":"2024-02-13T00:41:01.621982+0000","last_became_peered":"2024-02-13T00:41:01.621982+0000","last_unstale":"2024-02-13T00:41:04.006143+0000","last_undegraded":"2024-02-13T00:41:04.006143+0000","last_fullsized":"2024-02-13T00:41:04.006143+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.015048+0000","last_change":"2024-02-13T00:41:01.587011+0000","last_active":"2024-02-13T00:41:04.015048+0000","last_peered":"2024-02-13T00:41:04.015048+0000","last_clean":"2024-02-13T00:41:04.015048+0000","last_became_active":"2024-02-13T00:41:01.585826+0000","last_became_peered":"2024-02-13T00:41:01.585826+0000","last_unstale":"2024-02-13T00:41:04.015048+0000","last_undegraded":"2024-02-13T00:41:04.015048+0000","last_fullsized":"2024-02-13T00:41:04.015048+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"64'8","reported_seq":116,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.249827+0000","last_change":"2024-02-13T00:40:58.179518+0000","last_active":"2024-02-13T00:41:04.249827+0000","last_peered":"2024-02-13T00:41:04.249827+0000","last_clean":"2024-02-13T00:41:04.249827+0000","last_became_active":"2024-02-13T00:37:45.247905+0000","last_became_peered":"2024-02-13T00:37:45.247905+0000","last_unstale":"2024-02-13T00:41:04.249827+0000","last_undegraded":"2024-02-13T00:41:04.249827+0000","last_fullsized":"2024-02-13T00:41:04.249827+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:40:58.179446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.179446+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":128,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993971+0000","last_change":"2024-02-13T00:38:59.548948+0000","last_active":"2024-02-13T00:41:03.993971+0000","last_peered":"2024-02-13T00:41:03.993971+0000","last_clean":"2024-02-13T00:41:03.993971+0000","last_became_active":"2024-02-13T00:37:43.322374+0000","last_became_peered":"2024-02-13T00:37:43.322374+0000","last_unstale":"2024-02-13T00:41:03.993971+0000","last_undegraded":"2024-02-13T00:41:03.993971+0000","last_fullsized":"2024-02-13T00:41:03.993971+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_clean_scrub_stamp":"2024-02-13T00:38:59.548872+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.022807+0000","last_change":"2024-02-13T00:40:50.436030+0000","last_active":"2024-02-13T00:41:04.022807+0000","last_peered":"2024-02-13T00:41:04.022807+0000","last_clean":"2024-02-13T00:41:04.022807+0000","last_became_active":"2024-02-13T00:37:41.219093+0000","last_became_peered":"2024-02-13T00:37:41.219093+0000","last_unstale":"2024-02-13T00:41:04.022807+0000","last_undegraded":"2024-02-13T00:41:04.022807+0000","last_fullsized":"2024-02-13T00:41:04.022807+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:50.435958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:50.435958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":60,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.989782+0000","last_change":"2024-02-13T00:39:17.713034+0000","last_active":"2024-02-13T00:41:03.989782+0000","last_peered":"2024-02-13T00:41:03.989782+0000","last_clean":"2024-02-13T00:41:03.989782+0000","last_became_active":"2024-02-13T00:37:47.552384+0000","last_became_peered":"2024-02-13T00:37:47.552384+0000","last_unstale":"2024-02-13T00:41:03.989782+0000","last_undegraded":"2024-02-13T00:41:03.989782+0000","last_fullsized":"2024-02-13T00:41:03.989782+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.712965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.712965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.015123+0000","last_change":"2024-02-13T00:41:01.627210+0000","last_active":"2024-02-13T00:41:04.015123+0000","last_peered":"2024-02-13T00:41:04.015123+0000","last_clean":"2024-02-13T00:41:04.015123+0000","last_became_active":"2024-02-13T00:41:01.624745+0000","last_became_peered":"2024-02-13T00:41:01.624745+0000","last_unstale":"2024-02-13T00:41:04.015123+0000","last_undegraded":"2024-02-13T00:41:04.015123+0000","last_fullsized":"2024-02-13T00:41:04.015123+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993886+0000","last_change":"2024-02-13T00:41:01.738760+0000","last_active":"2024-02-13T00:41:03.993886+0000","last_peered":"2024-02-13T00:41:03.993886+0000","last_clean":"2024-02-13T00:41:03.993886+0000","last_became_active":"2024-02-13T00:41:01.738337+0000","last_became_peered":"2024-02-13T00:41:01.738337+0000","last_unstale":"2024-02-13T00:41:03.993886+0000","last_undegraded":"2024-02-13T00:41:03.993886+0000","last_fullsized":"2024-02-13T00:41:03.993886+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.031530+0000","last_change":"2024-02-13T00:41:04.031530+0000","last_active":"2024-02-13T00:41:04.031530+0000","last_peered":"2024-02-13T00:41:04.031530+0000","last_clean":"2024-02-13T00:41:04.031530+0000","last_became_active":"2024-02-13T00:41:01.706431+0000","last_became_peered":"2024-02-13T00:41:01.706431+0000","last_unstale":"2024-02-13T00:41:04.031530+0000","last_undegraded":"2024-02-13T00:41:04.031530+0000","last_fullsized":"2024-02-13T00:41:04.031530+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.005462+0000","last_change":"2024-02-13T00:41:01.568682+0000","last_active":"2024-02-13T00:41:04.005462+0000","last_peered":"2024-02-13T00:41:04.005462+0000","last_clean":"2024-02-13T00:41:04.005462+0000","last_became_active":"2024-02-13T00:41:01.568236+0000","last_became_peered":"2024-02-13T00:41:01.568236+0000","last_unstale":"2024-02-13T00:41:04.005462+0000","last_undegraded":"2024-02-13T00:41:04.005462+0000","last_fullsized":"2024-02-13T00:41:04.005462+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.014395+0000","last_change":"2024-02-13T00:41:04.014395+0000","last_active":"2024-02-13T00:41:04.014395+0000","last_peered":"2024-02-13T00:41:04.014395+0000","last_clean":"2024-02-13T00:41:04.014395+0000","last_became_active":"2024-02-13T00:41:01.737535+0000","last_became_peered":"2024-02-13T00:41:01.737535+0000","last_unstale":"2024-02-13T00:41:04.014395+0000","last_undegraded":"2024-02-13T00:41:04.014395+0000","last_fullsized":"2024-02-13T00:41:04.014395+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995512+0000","last_change":"2024-02-13T00:41:01.568271+0000","last_active":"2024-02-13T00:41:03.995512+0000","last_peered":"2024-02-13T00:41:03.995512+0000","last_clean":"2024-02-13T00:41:03.995512+0000","last_became_active":"2024-02-13T00:41:01.567613+0000","last_became_peered":"2024-02-13T00:41:01.567613+0000","last_unstale":"2024-02-13T00:41:03.995512+0000","last_undegraded":"2024-02-13T00:41:03.995512+0000","last_fullsized":"2024-02-13T00:41:03.995512+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.009966+0000","last_change":"2024-02-13T00:41:00.542560+0000","last_active":"2024-02-13T00:41:04.009966+0000","last_peered":"2024-02-13T00:41:04.009966+0000","last_clean":"2024-02-13T00:41:04.009966+0000","last_became_active":"2024-02-13T00:41:00.541013+0000","last_became_peered":"2024-02-13T00:41:00.541013+0000","last_unstale":"2024-02-13T00:41:04.009966+0000","last_undegraded":"2024-02-13T00:41:04.009966+0000","last_fullsized":"2024-02-13T00:41:04.009966+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.011308+0000","last_change":"2024-02-13T00:39:24.376175+0000","last_active":"2024-02-13T00:41:04.011308+0000","last_peered":"2024-02-13T00:41:04.011308+0000","last_clean":"2024-02-13T00:41:04.011308+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:41:04.011308+0000","last_undegraded":"2024-02-13T00:41:04.011308+0000","last_fullsized":"2024-02-13T00:41:04.011308+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:39:24.376100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.376100+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.026294+0000","last_change":"2024-02-13T00:39:16.262449+0000","last_active":"2024-02-13T00:41:04.026294+0000","last_peered":"2024-02-13T00:41:04.026294+0000","last_clean":"2024-02-13T00:41:04.026294+0000","last_became_active":"2024-02-13T00:37:41.212429+0000","last_became_peered":"2024-02-13T00:37:41.212429+0000","last_unstale":"2024-02-13T00:41:04.026294+0000","last_undegraded":"2024-02-13T00:41:04.026294+0000","last_fullsized":"2024-02-13T00:41:04.026294+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.262380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.262380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.001751+0000","last_change":"2024-02-13T00:40:53.362331+0000","last_active":"2024-02-13T00:41:04.001751+0000","last_peered":"2024-02-13T00:41:04.001751+0000","last_clean":"2024-02-13T00:41:04.001751+0000","last_became_active":"2024-02-13T00:37:45.240923+0000","last_became_peered":"2024-02-13T00:37:45.240923+0000","last_unstale":"2024-02-13T00:41:04.001751+0000","last_undegraded":"2024-02-13T00:41:04.001751+0000","last_fullsized":"2024-02-13T00:41:04.001751+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:53.362232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:53.362232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":60,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.006378+0000","last_change":"2024-02-13T00:39:18.359681+0000","last_active":"2024-02-13T00:41:04.006378+0000","last_peered":"2024-02-13T00:41:04.006378+0000","last_clean":"2024-02-13T00:41:04.006378+0000","last_became_active":"2024-02-13T00:37:47.249556+0000","last_became_peered":"2024-02-13T00:37:47.249556+0000","last_unstale":"2024-02-13T00:41:04.006378+0000","last_undegraded":"2024-02-13T00:41:04.006378+0000","last_fullsized":"2024-02-13T00:41:04.006378+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.359604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1d","version":"108'7","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.013960+0000","last_change":"2024-02-13T00:41:01.697227+0000","last_active":"2024-02-13T00:41:04.013960+0000","last_peered":"2024-02-13T00:41:04.013960+0000","last_clean":"2024-02-13T00:41:04.013960+0000","last_became_active":"2024-02-13T00:41:01.696134+0000","last_became_peered":"2024-02-13T00:41:01.696134+0000","last_unstale":"2024-02-13T00:41:04.013960+0000","last_undegraded":"2024-02-13T00:41:04.013960+0000","last_fullsized":"2024-02-13T00:41:04.013960+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.1b","version":"109'5","reported_seq":19,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.983378+0000","last_change":"2024-02-13T00:41:01.594685+0000","last_active":"2024-02-13T00:41:04.983378+0000","last_peered":"2024-02-13T00:41:04.983378+0000","last_clean":"2024-02-13T00:41:04.983378+0000","last_became_active":"2024-02-13T00:41:01.594405+0000","last_became_peered":"2024-02-13T00:41:01.594405+0000","last_unstale":"2024-02-13T00:41:04.983378+0000","last_undegraded":"2024-02-13T00:41:04.983378+0000","last_fullsized":"2024-02-13T00:41:04.983378+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50332680,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":51903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.008236+0000","last_change":"2024-02-13T00:41:04.008236+0000","last_active":"2024-02-13T00:41:04.008236+0000","last_peered":"2024-02-13T00:41:04.008236+0000","last_clean":"2024-02-13T00:41:04.008236+0000","last_became_active":"2024-02-13T00:41:01.638325+0000","last_became_peered":"2024-02-13T00:41:01.638325+0000","last_unstale":"2024-02-13T00:41:04.008236+0000","last_undegraded":"2024-02-13T00:41:04.008236+0000","last_fullsized":"2024-02-13T00:41:04.008236+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.015039+0000","last_change":"2024-02-13T00:41:01.656850+0000","last_active":"2024-02-13T00:41:04.015039+0000","last_peered":"2024-02-13T00:41:04.015039+0000","last_clean":"2024-02-13T00:41:04.015039+0000","last_became_active":"2024-02-13T00:41:01.654410+0000","last_became_peered":"2024-02-13T00:41:01.654410+0000","last_unstale":"2024-02-13T00:41:04.015039+0000","last_undegraded":"2024-02-13T00:41:04.015039+0000","last_fullsized":"2024-02-13T00:41:04.015039+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.036277+0000","last_change":"2024-02-13T00:41:04.036277+0000","last_active":"2024-02-13T00:41:04.036277+0000","last_peered":"2024-02-13T00:41:04.036277+0000","last_clean":"2024-02-13T00:41:04.036277+0000","last_became_active":"2024-02-13T00:41:01.658241+0000","last_became_peered":"2024-02-13T00:41:01.658241+0000","last_unstale":"2024-02-13T00:41:04.036277+0000","last_undegraded":"2024-02-13T00:41:04.036277+0000","last_fullsized":"2024-02-13T00:41:04.036277+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.5","version":"107'3","reported_seq":19,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.002577+0000","last_change":"2024-02-13T00:41:01.583500+0000","last_active":"2024-02-13T00:41:04.002577+0000","last_peered":"2024-02-13T00:41:04.002577+0000","last_clean":"2024-02-13T00:41:04.002577+0000","last_became_active":"2024-02-13T00:41:01.576322+0000","last_became_peered":"2024-02-13T00:41:01.576322+0000","last_unstale":"2024-02-13T00:41:04.002577+0000","last_undegraded":"2024-02-13T00:41:04.002577+0000","last_fullsized":"2024-02-13T00:41:04.002577+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.032972+0000","last_change":"2024-02-13T00:41:01.665930+0000","last_active":"2024-02-13T00:41:04.032972+0000","last_peered":"2024-02-13T00:41:04.032972+0000","last_clean":"2024-02-13T00:41:04.032972+0000","last_became_active":"2024-02-13T00:41:01.665238+0000","last_became_peered":"2024-02-13T00:41:01.665238+0000","last_unstale":"2024-02-13T00:41:04.032972+0000","last_undegraded":"2024-02-13T00:41:04.032972+0000","last_fullsized":"2024-02-13T00:41:04.032972+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"91'35","reported_seq":126,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.015393+0000","last_change":"2024-02-13T00:40:15.074106+0000","last_active":"2024-02-13T00:41:04.015393+0000","last_peered":"2024-02-13T00:41:04.015393+0000","last_clean":"2024-02-13T00:41:04.015393+0000","last_became_active":"2024-02-13T00:37:43.322957+0000","last_became_peered":"2024-02-13T00:37:43.322957+0000","last_unstale":"2024-02-13T00:41:04.015393+0000","last_undegraded":"2024-02-13T00:41:04.015393+0000","last_fullsized":"2024-02-13T00:41:04.015393+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:40:15.074034+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003130+0000","last_change":"2024-02-13T00:39:18.672595+0000","last_active":"2024-02-13T00:41:04.003130+0000","last_peered":"2024-02-13T00:41:04.003130+0000","last_clean":"2024-02-13T00:41:04.003130+0000","last_became_active":"2024-02-13T00:37:41.219457+0000","last_became_peered":"2024-02-13T00:37:41.219457+0000","last_unstale":"2024-02-13T00:41:04.003130+0000","last_undegraded":"2024-02-13T00:41:04.003130+0000","last_fullsized":"2024-02-13T00:41:04.003130+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.672516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.672516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003970+0000","last_change":"2024-02-13T00:41:03.411330+0000","last_active":"2024-02-13T00:41:04.003970+0000","last_peered":"2024-02-13T00:41:04.003970+0000","last_clean":"2024-02-13T00:41:04.003970+0000","last_became_active":"2024-02-13T00:37:45.239867+0000","last_became_peered":"2024-02-13T00:37:45.239867+0000","last_unstale":"2024-02-13T00:41:04.003970+0000","last_undegraded":"2024-02-13T00:41:04.003970+0000","last_fullsized":"2024-02-13T00:41:04.003970+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:03.411257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_clean_scrub_stamp":"2024-02-13T00:41:03.411257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":60,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993459+0000","last_change":"2024-02-13T00:39:16.086278+0000","last_active":"2024-02-13T00:41:03.993459+0000","last_peered":"2024-02-13T00:41:03.993459+0000","last_clean":"2024-02-13T00:41:03.993459+0000","last_became_active":"2024-02-13T00:37:47.254705+0000","last_became_peered":"2024-02-13T00:37:47.254705+0000","last_unstale":"2024-02-13T00:41:03.993459+0000","last_undegraded":"2024-02-13T00:41:03.993459+0000","last_fullsized":"2024-02-13T00:41:03.993459+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.086207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.086207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.024954+0000","last_change":"2024-02-13T00:41:01.733877+0000","last_active":"2024-02-13T00:41:04.024954+0000","last_peered":"2024-02-13T00:41:04.024954+0000","last_clean":"2024-02-13T00:41:04.024954+0000","last_became_active":"2024-02-13T00:41:01.733241+0000","last_became_peered":"2024-02-13T00:41:01.733241+0000","last_unstale":"2024-02-13T00:41:04.024954+0000","last_undegraded":"2024-02-13T00:41:04.024954+0000","last_fullsized":"2024-02-13T00:41:04.024954+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.990605+0000","last_change":"2024-02-13T00:41:01.576784+0000","last_active":"2024-02-13T00:41:03.990605+0000","last_peered":"2024-02-13T00:41:03.990605+0000","last_clean":"2024-02-13T00:41:03.990605+0000","last_became_active":"2024-02-13T00:41:01.576488+0000","last_became_peered":"2024-02-13T00:41:01.576488+0000","last_unstale":"2024-02-13T00:41:03.990605+0000","last_undegraded":"2024-02-13T00:41:03.990605+0000","last_fullsized":"2024-02-13T00:41:03.990605+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.033376+0000","last_change":"2024-02-13T00:41:04.033376+0000","last_active":"2024-02-13T00:41:04.033376+0000","last_peered":"2024-02-13T00:41:04.033376+0000","last_clean":"2024-02-13T00:41:04.033376+0000","last_became_active":"2024-02-13T00:41:01.592518+0000","last_became_peered":"2024-02-13T00:41:01.592518+0000","last_unstale":"2024-02-13T00:41:04.033376+0000","last_undegraded":"2024-02-13T00:41:04.033376+0000","last_fullsized":"2024-02-13T00:41:04.033376+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.15","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.026058+0000","last_change":"2024-02-13T00:41:01.735253+0000","last_active":"2024-02-13T00:41:04.026058+0000","last_peered":"2024-02-13T00:41:04.026058+0000","last_clean":"2024-02-13T00:41:04.026058+0000","last_became_active":"2024-02-13T00:41:01.727925+0000","last_became_peered":"2024-02-13T00:41:01.727925+0000","last_unstale":"2024-02-13T00:41:04.026058+0000","last_undegraded":"2024-02-13T00:41:04.026058+0000","last_fullsized":"2024-02-13T00:41:04.026058+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.997122+0000","last_change":"2024-02-13T00:41:03.997122+0000","last_active":"2024-02-13T00:41:03.997122+0000","last_peered":"2024-02-13T00:41:03.997122+0000","last_clean":"2024-02-13T00:41:03.997122+0000","last_became_active":"2024-02-13T00:41:01.671425+0000","last_became_peered":"2024-02-13T00:41:01.671425+0000","last_unstale":"2024-02-13T00:41:03.997122+0000","last_undegraded":"2024-02-13T00:41:03.997122+0000","last_fullsized":"2024-02-13T00:41:03.997122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.2","version":"107'3","reported_seq":19,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.001680+0000","last_change":"2024-02-13T00:41:00.543973+0000","last_active":"2024-02-13T00:41:04.001680+0000","last_peered":"2024-02-13T00:41:04.001680+0000","last_clean":"2024-02-13T00:41:04.001680+0000","last_became_active":"2024-02-13T00:41:00.543742+0000","last_became_peered":"2024-02-13T00:41:00.543742+0000","last_unstale":"2024-02-13T00:41:04.001680+0000","last_undegraded":"2024-02-13T00:41:04.001680+0000","last_fullsized":"2024-02-13T00:41:04.001680+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003784+0000","last_change":"2024-02-13T00:41:00.549374+0000","last_active":"2024-02-13T00:41:04.003784+0000","last_peered":"2024-02-13T00:41:04.003784+0000","last_clean":"2024-02-13T00:41:04.003784+0000","last_became_active":"2024-02-13T00:41:00.549164+0000","last_became_peered":"2024-02-13T00:41:00.549164+0000","last_unstale":"2024-02-13T00:41:04.003784+0000","last_undegraded":"2024-02-13T00:41:04.003784+0000","last_fullsized":"2024-02-13T00:41:04.003784+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.010034+0000","last_change":"2024-02-13T00:40:56.434571+0000","last_active":"2024-02-13T00:41:04.010034+0000","last_peered":"2024-02-13T00:41:04.010034+0000","last_clean":"2024-02-13T00:41:04.010034+0000","last_became_active":"2024-02-13T00:40:35.850675+0000","last_became_peered":"2024-02-13T00:40:35.850675+0000","last_unstale":"2024-02-13T00:41:04.010034+0000","last_undegraded":"2024-02-13T00:41:04.010034+0000","last_fullsized":"2024-02-13T00:41:04.010034+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:40:56.434506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:56.434506+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":78,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.006747+0000","last_change":"2024-02-13T00:41:00.128667+0000","last_active":"2024-02-13T00:41:04.006747+0000","last_peered":"2024-02-13T00:41:04.006747+0000","last_clean":"2024-02-13T00:41:04.006747+0000","last_became_active":"2024-02-13T00:37:41.220874+0000","last_became_peered":"2024-02-13T00:37:41.220874+0000","last_unstale":"2024-02-13T00:41:04.006747+0000","last_undegraded":"2024-02-13T00:41:04.006747+0000","last_fullsized":"2024-02-13T00:41:04.006747+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:41:00.128595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.128595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":63,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.022868+0000","last_change":"2024-02-13T00:39:35.269607+0000","last_active":"2024-02-13T00:41:04.022868+0000","last_peered":"2024-02-13T00:41:04.022868+0000","last_clean":"2024-02-13T00:41:04.022868+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:41:04.022868+0000","last_undegraded":"2024-02-13T00:41:04.022868+0000","last_fullsized":"2024-02-13T00:41:04.022868+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:35.269538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:35.269538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.030127+0000","last_change":"2024-02-13T00:41:02.402785+0000","last_active":"2024-02-13T00:41:04.030127+0000","last_peered":"2024-02-13T00:41:04.030127+0000","last_clean":"2024-02-13T00:41:04.030127+0000","last_became_active":"2024-02-13T00:37:47.256807+0000","last_became_peered":"2024-02-13T00:37:47.256807+0000","last_unstale":"2024-02-13T00:41:04.030127+0000","last_undegraded":"2024-02-13T00:41:04.030127+0000","last_fullsized":"2024-02-13T00:41:04.030127+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.402668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.402668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.1b","version":"108'10","reported_seq":23,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.035252+0000","last_change":"2024-02-13T00:41:01.676174+0000","last_active":"2024-02-13T00:41:04.035252+0000","last_peered":"2024-02-13T00:41:04.035252+0000","last_clean":"2024-02-13T00:41:04.035252+0000","last_became_active":"2024-02-13T00:41:01.675424+0000","last_became_peered":"2024-02-13T00:41:01.675424+0000","last_unstale":"2024-02-13T00:41:04.035252+0000","last_undegraded":"2024-02-13T00:41:04.035252+0000","last_fullsized":"2024-02-13T00:41:04.035252+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"107'1","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.014139+0000","last_change":"2024-02-13T00:41:01.597218+0000","last_active":"2024-02-13T00:41:04.014139+0000","last_peered":"2024-02-13T00:41:04.014139+0000","last_clean":"2024-02-13T00:41:04.014139+0000","last_became_active":"2024-02-13T00:41:01.596590+0000","last_became_peered":"2024-02-13T00:41:01.596590+0000","last_unstale":"2024-02-13T00:41:04.014139+0000","last_undegraded":"2024-02-13T00:41:04.014139+0000","last_fullsized":"2024-02-13T00:41:04.014139+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003943+0000","last_change":"2024-02-13T00:41:04.003943+0000","last_active":"2024-02-13T00:41:04.003943+0000","last_peered":"2024-02-13T00:41:04.003943+0000","last_clean":"2024-02-13T00:41:04.003943+0000","last_became_active":"2024-02-13T00:41:01.738560+0000","last_became_peered":"2024-02-13T00:41:01.738560+0000","last_unstale":"2024-02-13T00:41:04.003943+0000","last_undegraded":"2024-02-13T00:41:04.003943+0000","last_fullsized":"2024-02-13T00:41:04.003943+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.991266+0000","last_change":"2024-02-13T00:41:01.536591+0000","last_active":"2024-02-13T00:41:03.991266+0000","last_peered":"2024-02-13T00:41:03.991266+0000","last_clean":"2024-02-13T00:41:03.991266+0000","last_became_active":"2024-02-13T00:41:01.536188+0000","last_became_peered":"2024-02-13T00:41:01.536188+0000","last_unstale":"2024-02-13T00:41:03.991266+0000","last_undegraded":"2024-02-13T00:41:03.991266+0000","last_fullsized":"2024-02-13T00:41:03.991266+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.008650+0000","last_change":"2024-02-13T00:41:04.008650+0000","last_active":"2024-02-13T00:41:04.008650+0000","last_peered":"2024-02-13T00:41:04.008650+0000","last_clean":"2024-02-13T00:41:04.008650+0000","last_became_active":"2024-02-13T00:41:01.580959+0000","last_became_peered":"2024-02-13T00:41:01.580959+0000","last_unstale":"2024-02-13T00:41:04.008650+0000","last_undegraded":"2024-02-13T00:41:04.008650+0000","last_fullsized":"2024-02-13T00:41:04.008650+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.011768+0000","last_change":"2024-02-13T00:41:01.642688+0000","last_active":"2024-02-13T00:41:04.011768+0000","last_peered":"2024-02-13T00:41:04.011768+0000","last_clean":"2024-02-13T00:41:04.011768+0000","last_became_active":"2024-02-13T00:41:01.642404+0000","last_became_peered":"2024-02-13T00:41:01.642404+0000","last_unstale":"2024-02-13T00:41:04.011768+0000","last_undegraded":"2024-02-13T00:41:04.011768+0000","last_fullsized":"2024-02-13T00:41:04.011768+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.023561+0000","last_change":"2024-02-13T00:41:01.732486+0000","last_active":"2024-02-13T00:41:04.023561+0000","last_peered":"2024-02-13T00:41:04.023561+0000","last_clean":"2024-02-13T00:41:04.023561+0000","last_became_active":"2024-02-13T00:41:01.728943+0000","last_became_peered":"2024-02-13T00:41:01.728943+0000","last_unstale":"2024-02-13T00:41:04.023561+0000","last_undegraded":"2024-02-13T00:41:04.023561+0000","last_fullsized":"2024-02-13T00:41:04.023561+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":81,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.012219+0000","last_change":"2024-02-13T00:40:53.104803+0000","last_active":"2024-02-13T00:41:04.012219+0000","last_peered":"2024-02-13T00:41:04.012219+0000","last_clean":"2024-02-13T00:41:04.012219+0000","last_became_active":"2024-02-13T00:40:35.855827+0000","last_became_peered":"2024-02-13T00:40:35.855827+0000","last_unstale":"2024-02-13T00:41:04.012219+0000","last_undegraded":"2024-02-13T00:41:04.012219+0000","last_fullsized":"2024-02-13T00:41:04.012219+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:40:53.104724+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:40:53.104724+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.023601+0000","last_change":"2024-02-13T00:39:17.283996+0000","last_active":"2024-02-13T00:41:04.023601+0000","last_peered":"2024-02-13T00:41:04.023601+0000","last_clean":"2024-02-13T00:41:04.023601+0000","last_became_active":"2024-02-13T00:37:41.218346+0000","last_became_peered":"2024-02-13T00:37:41.218346+0000","last_unstale":"2024-02-13T00:41:04.023601+0000","last_undegraded":"2024-02-13T00:41:04.023601+0000","last_fullsized":"2024-02-13T00:41:04.023601+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.283902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.283902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"64'8","reported_seq":117,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.249734+0000","last_change":"2024-02-13T00:40:55.418661+0000","last_active":"2024-02-13T00:41:04.249734+0000","last_peered":"2024-02-13T00:41:04.249734+0000","last_clean":"2024-02-13T00:41:04.249734+0000","last_became_active":"2024-02-13T00:37:45.249684+0000","last_became_peered":"2024-02-13T00:37:45.249684+0000","last_unstale":"2024-02-13T00:41:04.249734+0000","last_undegraded":"2024-02-13T00:41:04.249734+0000","last_fullsized":"2024-02-13T00:41:04.249734+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:40:55.418534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:55.418534+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":40,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.015227+0000","last_change":"2024-02-13T00:40:40.750384+0000","last_active":"2024-02-13T00:41:04.015227+0000","last_peered":"2024-02-13T00:41:04.015227+0000","last_clean":"2024-02-13T00:41:04.015227+0000","last_became_active":"2024-02-13T00:40:35.851908+0000","last_became_peered":"2024-02-13T00:40:35.851908+0000","last_unstale":"2024-02-13T00:41:04.015227+0000","last_undegraded":"2024-02-13T00:41:04.015227+0000","last_fullsized":"2024-02-13T00:41:04.015227+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:40:40.750272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:40.750272+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1a","version":"108'5","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.992404+0000","last_change":"2024-02-13T00:41:01.743545+0000","last_active":"2024-02-13T00:41:03.992404+0000","last_peered":"2024-02-13T00:41:03.992404+0000","last_clean":"2024-02-13T00:41:03.992404+0000","last_became_active":"2024-02-13T00:41:01.739001+0000","last_became_peered":"2024-02-13T00:41:01.739001+0000","last_unstale":"2024-02-13T00:41:03.992404+0000","last_undegraded":"2024-02-13T00:41:03.992404+0000","last_fullsized":"2024-02-13T00:41:03.992404+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1c","version":"109'4","reported_seq":21,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.436683+0000","last_change":"2024-02-13T00:41:01.706984+0000","last_active":"2024-02-13T00:41:04.436683+0000","last_peered":"2024-02-13T00:41:04.436683+0000","last_clean":"2024-02-13T00:41:04.436683+0000","last_became_active":"2024-02-13T00:41:01.706080+0000","last_became_peered":"2024-02-13T00:41:01.706080+0000","last_unstale":"2024-02-13T00:41:04.436683+0000","last_undegraded":"2024-02-13T00:41:04.436683+0000","last_fullsized":"2024-02-13T00:41:04.436683+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.012280+0000","last_change":"2024-02-13T00:41:04.012280+0000","last_active":"2024-02-13T00:41:04.012280+0000","last_peered":"2024-02-13T00:41:04.012280+0000","last_clean":"2024-02-13T00:41:04.012280+0000","last_became_active":"2024-02-13T00:41:01.599055+0000","last_became_peered":"2024-02-13T00:41:01.599055+0000","last_unstale":"2024-02-13T00:41:04.012280+0000","last_undegraded":"2024-02-13T00:41:04.012280+0000","last_fullsized":"2024-02-13T00:41:04.012280+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996027+0000","last_change":"2024-02-13T00:41:01.632475+0000","last_active":"2024-02-13T00:41:03.996027+0000","last_peered":"2024-02-13T00:41:03.996027+0000","last_clean":"2024-02-13T00:41:03.996027+0000","last_became_active":"2024-02-13T00:41:01.632199+0000","last_became_peered":"2024-02-13T00:41:01.632199+0000","last_unstale":"2024-02-13T00:41:03.996027+0000","last_undegraded":"2024-02-13T00:41:03.996027+0000","last_fullsized":"2024-02-13T00:41:03.996027+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.041126+0000","last_change":"2024-02-13T00:41:04.041126+0000","last_active":"2024-02-13T00:41:04.041126+0000","last_peered":"2024-02-13T00:41:04.041126+0000","last_clean":"2024-02-13T00:41:04.041126+0000","last_became_active":"2024-02-13T00:41:01.736278+0000","last_became_peered":"2024-02-13T00:41:01.736278+0000","last_unstale":"2024-02-13T00:41:04.041126+0000","last_undegraded":"2024-02-13T00:41:04.041126+0000","last_fullsized":"2024-02-13T00:41:04.041126+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.016052+0000","last_change":"2024-02-13T00:41:01.693212+0000","last_active":"2024-02-13T00:41:04.016052+0000","last_peered":"2024-02-13T00:41:04.016052+0000","last_clean":"2024-02-13T00:41:04.016052+0000","last_became_active":"2024-02-13T00:41:01.692967+0000","last_became_peered":"2024-02-13T00:41:01.692967+0000","last_unstale":"2024-02-13T00:41:04.016052+0000","last_undegraded":"2024-02-13T00:41:04.016052+0000","last_fullsized":"2024-02-13T00:41:04.016052+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.007872+0000","last_change":"2024-02-13T00:41:01.604018+0000","last_active":"2024-02-13T00:41:04.007872+0000","last_peered":"2024-02-13T00:41:04.007872+0000","last_clean":"2024-02-13T00:41:04.007872+0000","last_became_active":"2024-02-13T00:41:01.602124+0000","last_became_peered":"2024-02-13T00:41:01.602124+0000","last_unstale":"2024-02-13T00:41:04.007872+0000","last_undegraded":"2024-02-13T00:41:04.007872+0000","last_fullsized":"2024-02-13T00:41:04.007872+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":90,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.020517+0000","last_change":"2024-02-13T00:39:23.370512+0000","last_active":"2024-02-13T00:41:04.020517+0000","last_peered":"2024-02-13T00:41:04.020517+0000","last_clean":"2024-02-13T00:41:04.020517+0000","last_became_active":"2024-02-13T00:37:43.232161+0000","last_became_peered":"2024-02-13T00:37:43.232161+0000","last_unstale":"2024-02-13T00:41:04.020517+0000","last_undegraded":"2024-02-13T00:41:04.020517+0000","last_fullsized":"2024-02-13T00:41:04.020517+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:39:23.370447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.370447+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":78,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.027967+0000","last_change":"2024-02-13T00:39:52.206001+0000","last_active":"2024-02-13T00:41:04.027967+0000","last_peered":"2024-02-13T00:41:04.027967+0000","last_clean":"2024-02-13T00:41:04.027967+0000","last_became_active":"2024-02-13T00:37:41.217979+0000","last_became_peered":"2024-02-13T00:37:41.217979+0000","last_unstale":"2024-02-13T00:41:04.027967+0000","last_undegraded":"2024-02-13T00:41:04.027967+0000","last_fullsized":"2024-02-13T00:41:04.027967+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:39:52.205926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:52.205926+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"96'9","reported_seq":52,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.249813+0000","last_change":"2024-02-13T00:40:35.847666+0000","last_active":"2024-02-13T00:41:04.249813+0000","last_peered":"2024-02-13T00:41:04.249813+0000","last_clean":"2024-02-13T00:41:04.249813+0000","last_became_active":"2024-02-13T00:40:35.847293+0000","last_became_peered":"2024-02-13T00:40:35.847293+0000","last_unstale":"2024-02-13T00:41:04.249813+0000","last_undegraded":"2024-02-13T00:41:04.249813+0000","last_fullsized":"2024-02-13T00:41:04.249813+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:40:02.089864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:02.089864+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.004118+0000","last_change":"2024-02-13T00:40:58.541216+0000","last_active":"2024-02-13T00:41:04.004118+0000","last_peered":"2024-02-13T00:41:04.004118+0000","last_clean":"2024-02-13T00:41:04.004118+0000","last_became_active":"2024-02-13T00:37:47.260477+0000","last_became_peered":"2024-02-13T00:37:47.260477+0000","last_unstale":"2024-02-13T00:41:04.004118+0000","last_undegraded":"2024-02-13T00:41:04.004118+0000","last_fullsized":"2024-02-13T00:41:04.004118+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.541144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.008014+0000","last_change":"2024-02-13T00:41:01.596916+0000","last_active":"2024-02-13T00:41:04.008014+0000","last_peered":"2024-02-13T00:41:04.008014+0000","last_clean":"2024-02-13T00:41:04.008014+0000","last_became_active":"2024-02-13T00:41:01.594299+0000","last_became_peered":"2024-02-13T00:41:01.594299+0000","last_unstale":"2024-02-13T00:41:04.008014+0000","last_undegraded":"2024-02-13T00:41:04.008014+0000","last_fullsized":"2024-02-13T00:41:04.008014+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"107'5","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995944+0000","last_change":"2024-02-13T00:41:01.634587+0000","last_active":"2024-02-13T00:41:03.995944+0000","last_peered":"2024-02-13T00:41:03.995944+0000","last_clean":"2024-02-13T00:41:03.995944+0000","last_became_active":"2024-02-13T00:41:01.634317+0000","last_became_peered":"2024-02-13T00:41:01.634317+0000","last_unstale":"2024-02-13T00:41:03.995944+0000","last_undegraded":"2024-02-13T00:41:03.995944+0000","last_fullsized":"2024-02-13T00:41:03.995944+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":64,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.024189+0000","last_change":"2024-02-13T00:41:04.024189+0000","last_active":"2024-02-13T00:41:04.024189+0000","last_peered":"2024-02-13T00:41:04.024189+0000","last_clean":"2024-02-13T00:41:04.024189+0000","last_became_active":"2024-02-13T00:41:01.556974+0000","last_became_peered":"2024-02-13T00:41:01.556974+0000","last_unstale":"2024-02-13T00:41:04.024189+0000","last_undegraded":"2024-02-13T00:41:04.024189+0000","last_fullsized":"2024-02-13T00:41:04.024189+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.011826+0000","last_change":"2024-02-13T00:41:01.747617+0000","last_active":"2024-02-13T00:41:04.011826+0000","last_peered":"2024-02-13T00:41:04.011826+0000","last_clean":"2024-02-13T00:41:04.011826+0000","last_became_active":"2024-02-13T00:41:01.746476+0000","last_became_peered":"2024-02-13T00:41:01.746476+0000","last_unstale":"2024-02-13T00:41:04.011826+0000","last_undegraded":"2024-02-13T00:41:04.011826+0000","last_fullsized":"2024-02-13T00:41:04.011826+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.022232+0000","last_change":"2024-02-13T00:41:04.022232+0000","last_active":"2024-02-13T00:41:04.022232+0000","last_peered":"2024-02-13T00:41:04.022232+0000","last_clean":"2024-02-13T00:41:04.022232+0000","last_became_active":"2024-02-13T00:41:01.598804+0000","last_became_peered":"2024-02-13T00:41:01.598804+0000","last_unstale":"2024-02-13T00:41:04.022232+0000","last_undegraded":"2024-02-13T00:41:04.022232+0000","last_fullsized":"2024-02-13T00:41:04.022232+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.1","version":"107'5","reported_seq":22,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.007734+0000","last_change":"2024-02-13T00:41:01.655263+0000","last_active":"2024-02-13T00:41:04.007734+0000","last_peered":"2024-02-13T00:41:04.007734+0000","last_clean":"2024-02-13T00:41:04.007734+0000","last_became_active":"2024-02-13T00:41:01.654651+0000","last_became_peered":"2024-02-13T00:41:01.654651+0000","last_unstale":"2024-02-13T00:41:04.007734+0000","last_undegraded":"2024-02-13T00:41:04.007734+0000","last_fullsized":"2024-02-13T00:41:04.007734+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993532+0000","last_change":"2024-02-13T00:41:01.636851+0000","last_active":"2024-02-13T00:41:03.993532+0000","last_peered":"2024-02-13T00:41:03.993532+0000","last_clean":"2024-02-13T00:41:03.993532+0000","last_became_active":"2024-02-13T00:41:01.636533+0000","last_became_peered":"2024-02-13T00:41:01.636533+0000","last_unstale":"2024-02-13T00:41:03.993532+0000","last_undegraded":"2024-02-13T00:41:03.993532+0000","last_fullsized":"2024-02-13T00:41:03.993532+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003123+0000","last_change":"2024-02-13T00:40:35.848232+0000","last_active":"2024-02-13T00:41:04.003123+0000","last_peered":"2024-02-13T00:41:04.003123+0000","last_clean":"2024-02-13T00:41:04.003123+0000","last_became_active":"2024-02-13T00:40:35.847905+0000","last_became_peered":"2024-02-13T00:40:35.847905+0000","last_unstale":"2024-02-13T00:41:04.003123+0000","last_undegraded":"2024-02-13T00:41:04.003123+0000","last_fullsized":"2024-02-13T00:41:04.003123+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:29.275134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.275134+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993700+0000","last_change":"2024-02-13T00:39:29.159172+0000","last_active":"2024-02-13T00:41:03.993700+0000","last_peered":"2024-02-13T00:41:03.993700+0000","last_clean":"2024-02-13T00:41:03.993700+0000","last_became_active":"2024-02-13T00:37:41.221627+0000","last_became_peered":"2024-02-13T00:37:41.221627+0000","last_unstale":"2024-02-13T00:41:03.993700+0000","last_undegraded":"2024-02-13T00:41:03.993700+0000","last_fullsized":"2024-02-13T00:41:03.993700+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.159080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.159080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.006500+0000","last_change":"2024-02-13T00:40:48.518561+0000","last_active":"2024-02-13T00:41:04.006500+0000","last_peered":"2024-02-13T00:41:04.006500+0000","last_clean":"2024-02-13T00:41:04.006500+0000","last_became_active":"2024-02-13T00:37:45.245897+0000","last_became_peered":"2024-02-13T00:37:45.245897+0000","last_unstale":"2024-02-13T00:41:04.006500+0000","last_undegraded":"2024-02-13T00:41:04.006500+0000","last_fullsized":"2024-02-13T00:41:04.006500+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:48.518493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:40:48.518493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.990725+0000","last_change":"2024-02-13T00:40:14.015620+0000","last_active":"2024-02-13T00:41:03.990725+0000","last_peered":"2024-02-13T00:41:03.990725+0000","last_clean":"2024-02-13T00:41:03.990725+0000","last_became_active":"2024-02-13T00:37:47.552591+0000","last_became_peered":"2024-02-13T00:37:47.552591+0000","last_unstale":"2024-02-13T00:41:03.990725+0000","last_undegraded":"2024-02-13T00:41:03.990725+0000","last_fullsized":"2024-02-13T00:41:03.990725+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:14.015547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:14.015547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.18","version":"107'2","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.016859+0000","last_change":"2024-02-13T00:41:01.696324+0000","last_active":"2024-02-13T00:41:04.016859+0000","last_peered":"2024-02-13T00:41:04.016859+0000","last_clean":"2024-02-13T00:41:04.016859+0000","last_became_active":"2024-02-13T00:41:01.696158+0000","last_became_peered":"2024-02-13T00:41:01.696158+0000","last_unstale":"2024-02-13T00:41:04.016859+0000","last_undegraded":"2024-02-13T00:41:04.016859+0000","last_fullsized":"2024-02-13T00:41:04.016859+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1e","version":"109'4","reported_seq":19,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.627882+0000","last_change":"2024-02-13T00:41:01.660822+0000","last_active":"2024-02-13T00:41:04.627882+0000","last_peered":"2024-02-13T00:41:04.627882+0000","last_clean":"2024-02-13T00:41:04.627882+0000","last_became_active":"2024-02-13T00:41:01.660434+0000","last_became_peered":"2024-02-13T00:41:01.660434+0000","last_unstale":"2024-02-13T00:41:04.627882+0000","last_undegraded":"2024-02-13T00:41:04.627882+0000","last_fullsized":"2024-02-13T00:41:04.627882+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2808,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.016203+0000","last_change":"2024-02-13T00:41:04.016203+0000","last_active":"2024-02-13T00:41:04.016203+0000","last_peered":"2024-02-13T00:41:04.016203+0000","last_clean":"2024-02-13T00:41:04.016203+0000","last_became_active":"2024-02-13T00:41:01.685416+0000","last_became_peered":"2024-02-13T00:41:01.685416+0000","last_unstale":"2024-02-13T00:41:04.016203+0000","last_undegraded":"2024-02-13T00:41:04.016203+0000","last_fullsized":"2024-02-13T00:41:04.016203+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.992513+0000","last_change":"2024-02-13T00:41:01.583063+0000","last_active":"2024-02-13T00:41:03.992513+0000","last_peered":"2024-02-13T00:41:03.992513+0000","last_clean":"2024-02-13T00:41:03.992513+0000","last_became_active":"2024-02-13T00:41:01.582361+0000","last_became_peered":"2024-02-13T00:41:01.582361+0000","last_unstale":"2024-02-13T00:41:03.992513+0000","last_undegraded":"2024-02-13T00:41:03.992513+0000","last_fullsized":"2024-02-13T00:41:03.992513+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.008993+0000","last_change":"2024-02-13T00:41:04.008993+0000","last_active":"2024-02-13T00:41:04.008993+0000","last_peered":"2024-02-13T00:41:04.008993+0000","last_clean":"2024-02-13T00:41:04.008993+0000","last_became_active":"2024-02-13T00:41:01.585759+0000","last_became_peered":"2024-02-13T00:41:01.585759+0000","last_unstale":"2024-02-13T00:41:04.008993+0000","last_undegraded":"2024-02-13T00:41:04.008993+0000","last_fullsized":"2024-02-13T00:41:04.008993+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.e","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.014140+0000","last_change":"2024-02-13T00:41:01.695205+0000","last_active":"2024-02-13T00:41:04.014140+0000","last_peered":"2024-02-13T00:41:04.014140+0000","last_clean":"2024-02-13T00:41:04.014140+0000","last_became_active":"2024-02-13T00:41:01.693552+0000","last_became_peered":"2024-02-13T00:41:01.693552+0000","last_unstale":"2024-02-13T00:41:04.014140+0000","last_undegraded":"2024-02-13T00:41:04.014140+0000","last_fullsized":"2024-02-13T00:41:04.014140+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.013754+0000","last_change":"2024-02-13T00:41:01.661806+0000","last_active":"2024-02-13T00:41:04.013754+0000","last_peered":"2024-02-13T00:41:04.013754+0000","last_clean":"2024-02-13T00:41:04.013754+0000","last_became_active":"2024-02-13T00:41:01.660096+0000","last_became_peered":"2024-02-13T00:41:01.660096+0000","last_unstale":"2024-02-13T00:41:04.013754+0000","last_undegraded":"2024-02-13T00:41:04.013754+0000","last_fullsized":"2024-02-13T00:41:04.013754+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"64'8","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.249949+0000","last_change":"2024-02-13T00:39:16.304400+0000","last_active":"2024-02-13T00:41:04.249949+0000","last_peered":"2024-02-13T00:41:04.249949+0000","last_clean":"2024-02-13T00:41:04.249949+0000","last_became_active":"2024-02-13T00:37:45.245528+0000","last_became_peered":"2024-02-13T00:37:45.245528+0000","last_unstale":"2024-02-13T00:41:04.249949+0000","last_undegraded":"2024-02-13T00:41:04.249949+0000","last_fullsized":"2024-02-13T00:41:04.249949+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:16.304316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.304316+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":80,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.988729+0000","last_change":"2024-02-13T00:40:43.630615+0000","last_active":"2024-02-13T00:41:03.988729+0000","last_peered":"2024-02-13T00:41:03.988729+0000","last_clean":"2024-02-13T00:41:03.988729+0000","last_became_active":"2024-02-13T00:40:35.844184+0000","last_became_peered":"2024-02-13T00:40:35.844184+0000","last_unstale":"2024-02-13T00:41:03.988729+0000","last_undegraded":"2024-02-13T00:41:03.988729+0000","last_fullsized":"2024-02-13T00:41:03.988729+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2024-02-13T00:40:43.630535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:43.630535+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.013826+0000","last_change":"2024-02-13T00:40:46.274497+0000","last_active":"2024-02-13T00:41:04.013826+0000","last_peered":"2024-02-13T00:41:04.013826+0000","last_clean":"2024-02-13T00:41:04.013826+0000","last_became_active":"2024-02-13T00:37:41.218313+0000","last_became_peered":"2024-02-13T00:37:41.218313+0000","last_unstale":"2024-02-13T00:41:04.013826+0000","last_undegraded":"2024-02-13T00:41:04.013826+0000","last_fullsized":"2024-02-13T00:41:04.013826+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:46.274423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:46.274423+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.010650+0000","last_change":"2024-02-13T00:40:44.243015+0000","last_active":"2024-02-13T00:41:04.010650+0000","last_peered":"2024-02-13T00:41:04.010650+0000","last_clean":"2024-02-13T00:41:04.010650+0000","last_became_active":"2024-02-13T00:37:47.553554+0000","last_became_peered":"2024-02-13T00:37:47.553554+0000","last_unstale":"2024-02-13T00:41:04.010650+0000","last_undegraded":"2024-02-13T00:41:04.010650+0000","last_fullsized":"2024-02-13T00:41:04.010650+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:44.242946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.242946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.17","version":"108'5","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.006954+0000","last_change":"2024-02-13T00:41:01.623530+0000","last_active":"2024-02-13T00:41:04.006954+0000","last_peered":"2024-02-13T00:41:04.006954+0000","last_clean":"2024-02-13T00:41:04.006954+0000","last_became_active":"2024-02-13T00:41:01.622857+0000","last_became_peered":"2024-02-13T00:41:01.622857+0000","last_unstale":"2024-02-13T00:41:04.006954+0000","last_undegraded":"2024-02-13T00:41:04.006954+0000","last_fullsized":"2024-02-13T00:41:04.006954+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"107'6","reported_seq":19,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.032743+0000","last_change":"2024-02-13T00:41:01.734315+0000","last_active":"2024-02-13T00:41:04.032743+0000","last_peered":"2024-02-13T00:41:04.032743+0000","last_clean":"2024-02-13T00:41:04.032743+0000","last_became_active":"2024-02-13T00:41:01.728024+0000","last_became_peered":"2024-02-13T00:41:01.728024+0000","last_unstale":"2024-02-13T00:41:04.032743+0000","last_undegraded":"2024-02-13T00:41:04.032743+0000","last_fullsized":"2024-02-13T00:41:04.032743+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.040249+0000","last_change":"2024-02-13T00:41:04.040249+0000","last_active":"2024-02-13T00:41:04.040249+0000","last_peered":"2024-02-13T00:41:04.040249+0000","last_clean":"2024-02-13T00:41:04.040249+0000","last_became_active":"2024-02-13T00:41:01.734997+0000","last_became_peered":"2024-02-13T00:41:01.734997+0000","last_unstale":"2024-02-13T00:41:04.040249+0000","last_undegraded":"2024-02-13T00:41:04.040249+0000","last_fullsized":"2024-02-13T00:41:04.040249+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.988701+0000","last_change":"2024-02-13T00:41:01.682441+0000","last_active":"2024-02-13T00:41:03.988701+0000","last_peered":"2024-02-13T00:41:03.988701+0000","last_clean":"2024-02-13T00:41:03.988701+0000","last_became_active":"2024-02-13T00:41:01.682162+0000","last_became_peered":"2024-02-13T00:41:01.682162+0000","last_unstale":"2024-02-13T00:41:03.988701+0000","last_undegraded":"2024-02-13T00:41:03.988701+0000","last_fullsized":"2024-02-13T00:41:03.988701+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.006271+0000","last_change":"2024-02-13T00:41:04.006271+0000","last_active":"2024-02-13T00:41:04.006271+0000","last_peered":"2024-02-13T00:41:04.006271+0000","last_clean":"2024-02-13T00:41:04.006271+0000","last_became_active":"2024-02-13T00:41:01.619499+0000","last_became_peered":"2024-02-13T00:41:01.619499+0000","last_unstale":"2024-02-13T00:41:04.006271+0000","last_undegraded":"2024-02-13T00:41:04.006271+0000","last_fullsized":"2024-02-13T00:41:04.006271+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.f","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.994904+0000","last_change":"2024-02-13T00:41:01.682282+0000","last_active":"2024-02-13T00:41:03.994904+0000","last_peered":"2024-02-13T00:41:03.994904+0000","last_clean":"2024-02-13T00:41:03.994904+0000","last_became_active":"2024-02-13T00:41:01.682015+0000","last_became_peered":"2024-02-13T00:41:01.682015+0000","last_unstale":"2024-02-13T00:41:03.994904+0000","last_undegraded":"2024-02-13T00:41:03.994904+0000","last_fullsized":"2024-02-13T00:41:03.994904+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003321+0000","last_change":"2024-02-13T00:41:01.615477+0000","last_active":"2024-02-13T00:41:04.003321+0000","last_peered":"2024-02-13T00:41:04.003321+0000","last_clean":"2024-02-13T00:41:04.003321+0000","last_became_active":"2024-02-13T00:41:01.614868+0000","last_became_peered":"2024-02-13T00:41:01.614868+0000","last_unstale":"2024-02-13T00:41:04.003321+0000","last_undegraded":"2024-02-13T00:41:04.003321+0000","last_fullsized":"2024-02-13T00:41:04.003321+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":31,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.988431+0000","last_change":"2024-02-13T00:40:43.011812+0000","last_active":"2024-02-13T00:41:03.988431+0000","last_peered":"2024-02-13T00:41:03.988431+0000","last_clean":"2024-02-13T00:41:03.988431+0000","last_became_active":"2024-02-13T00:40:35.856681+0000","last_became_peered":"2024-02-13T00:40:35.856681+0000","last_unstale":"2024-02-13T00:41:03.988431+0000","last_undegraded":"2024-02-13T00:41:03.988431+0000","last_fullsized":"2024-02-13T00:41:03.988431+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:43.011693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:43.011693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":131,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.988589+0000","last_change":"2024-02-13T00:40:47.034719+0000","last_active":"2024-02-13T00:41:03.988589+0000","last_peered":"2024-02-13T00:41:03.988589+0000","last_clean":"2024-02-13T00:41:03.988589+0000","last_became_active":"2024-02-13T00:37:43.324275+0000","last_became_peered":"2024-02-13T00:37:43.324275+0000","last_unstale":"2024-02-13T00:41:03.988589+0000","last_undegraded":"2024-02-13T00:41:03.988589+0000","last_fullsized":"2024-02-13T00:41:03.988589+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:40:47.034615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:47.034615+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.014716+0000","last_change":"2024-02-13T00:39:21.329248+0000","last_active":"2024-02-13T00:41:04.014716+0000","last_peered":"2024-02-13T00:41:04.014716+0000","last_clean":"2024-02-13T00:41:04.014716+0000","last_became_active":"2024-02-13T00:37:41.214202+0000","last_became_peered":"2024-02-13T00:37:41.214202+0000","last_unstale":"2024-02-13T00:41:04.014716+0000","last_undegraded":"2024-02-13T00:41:04.014716+0000","last_fullsized":"2024-02-13T00:41:04.014716+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.329173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.329173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":31,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.991955+0000","last_change":"2024-02-13T00:41:02.817359+0000","last_active":"2024-02-13T00:41:03.991955+0000","last_peered":"2024-02-13T00:41:03.991955+0000","last_clean":"2024-02-13T00:41:03.991955+0000","last_became_active":"2024-02-13T00:40:35.857936+0000","last_became_peered":"2024-02-13T00:40:35.857936+0000","last_unstale":"2024-02-13T00:41:03.991955+0000","last_undegraded":"2024-02-13T00:41:03.991955+0000","last_fullsized":"2024-02-13T00:41:03.991955+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.16","version":"108'5","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.007899+0000","last_change":"2024-02-13T00:41:01.722389+0000","last_active":"2024-02-13T00:41:04.007899+0000","last_peered":"2024-02-13T00:41:04.007899+0000","last_clean":"2024-02-13T00:41:04.007899+0000","last_became_active":"2024-02-13T00:41:01.721562+0000","last_became_peered":"2024-02-13T00:41:01.721562+0000","last_unstale":"2024-02-13T00:41:04.007899+0000","last_undegraded":"2024-02-13T00:41:04.007899+0000","last_fullsized":"2024-02-13T00:41:04.007899+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.10","version":"107'4","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.994544+0000","last_change":"2024-02-13T00:41:01.578798+0000","last_active":"2024-02-13T00:41:03.994544+0000","last_peered":"2024-02-13T00:41:03.994544+0000","last_clean":"2024-02-13T00:41:03.994544+0000","last_became_active":"2024-02-13T00:41:01.578398+0000","last_became_peered":"2024-02-13T00:41:01.578398+0000","last_unstale":"2024-02-13T00:41:03.994544+0000","last_undegraded":"2024-02-13T00:41:03.994544+0000","last_fullsized":"2024-02-13T00:41:03.994544+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.016668+0000","last_change":"2024-02-13T00:41:04.016668+0000","last_active":"2024-02-13T00:41:04.016668+0000","last_peered":"2024-02-13T00:41:04.016668+0000","last_clean":"2024-02-13T00:41:04.016668+0000","last_became_active":"2024-02-13T00:41:01.647045+0000","last_became_peered":"2024-02-13T00:41:01.647045+0000","last_unstale":"2024-02-13T00:41:04.016668+0000","last_undegraded":"2024-02-13T00:41:04.016668+0000","last_fullsized":"2024-02-13T00:41:04.016668+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.1b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.011428+0000","last_change":"2024-02-13T00:41:01.568508+0000","last_active":"2024-02-13T00:41:04.011428+0000","last_peered":"2024-02-13T00:41:04.011428+0000","last_clean":"2024-02-13T00:41:04.011428+0000","last_became_active":"2024-02-13T00:41:01.567871+0000","last_became_peered":"2024-02-13T00:41:01.567871+0000","last_unstale":"2024-02-13T00:41:04.011428+0000","last_undegraded":"2024-02-13T00:41:04.011428+0000","last_fullsized":"2024-02-13T00:41:04.011428+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.038691+0000","last_change":"2024-02-13T00:41:04.038691+0000","last_active":"2024-02-13T00:41:04.038691+0000","last_peered":"2024-02-13T00:41:04.038691+0000","last_clean":"2024-02-13T00:41:04.038691+0000","last_became_active":"2024-02-13T00:41:01.683057+0000","last_became_peered":"2024-02-13T00:41:01.683057+0000","last_unstale":"2024-02-13T00:41:04.038691+0000","last_undegraded":"2024-02-13T00:41:04.038691+0000","last_fullsized":"2024-02-13T00:41:04.038691+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.022597+0000","last_change":"2024-02-13T00:41:01.717047+0000","last_active":"2024-02-13T00:41:04.022597+0000","last_peered":"2024-02-13T00:41:04.022597+0000","last_clean":"2024-02-13T00:41:04.022597+0000","last_became_active":"2024-02-13T00:41:01.715524+0000","last_became_peered":"2024-02-13T00:41:01.715524+0000","last_unstale":"2024-02-13T00:41:04.022597+0000","last_undegraded":"2024-02-13T00:41:04.022597+0000","last_fullsized":"2024-02-13T00:41:04.022597+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995365+0000","last_change":"2024-02-13T00:41:01.587561+0000","last_active":"2024-02-13T00:41:03.995365+0000","last_peered":"2024-02-13T00:41:03.995365+0000","last_clean":"2024-02-13T00:41:03.995365+0000","last_became_active":"2024-02-13T00:41:01.585929+0000","last_became_peered":"2024-02-13T00:41:01.585929+0000","last_unstale":"2024-02-13T00:41:03.995365+0000","last_undegraded":"2024-02-13T00:41:03.995365+0000","last_fullsized":"2024-02-13T00:41:03.995365+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.016982+0000","last_change":"2024-02-13T00:40:55.358468+0000","last_active":"2024-02-13T00:41:04.016982+0000","last_peered":"2024-02-13T00:41:04.016982+0000","last_clean":"2024-02-13T00:41:04.016982+0000","last_became_active":"2024-02-13T00:37:45.245974+0000","last_became_peered":"2024-02-13T00:37:45.245974+0000","last_unstale":"2024-02-13T00:41:04.016982+0000","last_undegraded":"2024-02-13T00:41:04.016982+0000","last_fullsized":"2024-02-13T00:41:04.016982+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:55.358367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:55.358367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":124,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.011147+0000","last_change":"2024-02-13T00:39:04.318990+0000","last_active":"2024-02-13T00:41:04.011147+0000","last_peered":"2024-02-13T00:41:04.011147+0000","last_clean":"2024-02-13T00:41:04.011147+0000","last_became_active":"2024-02-13T00:37:43.322450+0000","last_became_peered":"2024-02-13T00:37:43.322450+0000","last_unstale":"2024-02-13T00:41:04.011147+0000","last_undegraded":"2024-02-13T00:41:04.011147+0000","last_fullsized":"2024-02-13T00:41:04.011147+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_clean_scrub_stamp":"2024-02-13T00:39:04.318923+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003077+0000","last_change":"2024-02-13T00:40:45.198328+0000","last_active":"2024-02-13T00:41:04.003077+0000","last_peered":"2024-02-13T00:41:04.003077+0000","last_clean":"2024-02-13T00:41:04.003077+0000","last_became_active":"2024-02-13T00:37:41.221636+0000","last_became_peered":"2024-02-13T00:37:41.221636+0000","last_unstale":"2024-02-13T00:41:04.003077+0000","last_undegraded":"2024-02-13T00:41:04.003077+0000","last_fullsized":"2024-02-13T00:41:04.003077+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:45.198231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:45.198231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":61,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.028058+0000","last_change":"2024-02-13T00:39:33.304107+0000","last_active":"2024-02-13T00:41:04.028058+0000","last_peered":"2024-02-13T00:41:04.028058+0000","last_clean":"2024-02-13T00:41:04.028058+0000","last_became_active":"2024-02-13T00:37:47.552672+0000","last_became_peered":"2024-02-13T00:37:47.552672+0000","last_unstale":"2024-02-13T00:41:04.028058+0000","last_undegraded":"2024-02-13T00:41:04.028058+0000","last_fullsized":"2024-02-13T00:41:04.028058+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.303982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.303982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.004739+0000","last_change":"2024-02-13T00:41:01.672342+0000","last_active":"2024-02-13T00:41:04.004739+0000","last_peered":"2024-02-13T00:41:04.004739+0000","last_clean":"2024-02-13T00:41:04.004739+0000","last_became_active":"2024-02-13T00:41:01.671979+0000","last_became_peered":"2024-02-13T00:41:01.671979+0000","last_unstale":"2024-02-13T00:41:04.004739+0000","last_undegraded":"2024-02-13T00:41:04.004739+0000","last_fullsized":"2024-02-13T00:41:04.004739+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.013403+0000","last_change":"2024-02-13T00:41:01.578655+0000","last_active":"2024-02-13T00:41:04.013403+0000","last_peered":"2024-02-13T00:41:04.013403+0000","last_clean":"2024-02-13T00:41:04.013403+0000","last_became_active":"2024-02-13T00:41:01.578380+0000","last_became_peered":"2024-02-13T00:41:01.578380+0000","last_unstale":"2024-02-13T00:41:04.013403+0000","last_undegraded":"2024-02-13T00:41:04.013403+0000","last_fullsized":"2024-02-13T00:41:04.013403+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.036941+0000","last_change":"2024-02-13T00:41:04.036941+0000","last_active":"2024-02-13T00:41:04.036941+0000","last_peered":"2024-02-13T00:41:04.036941+0000","last_clean":"2024-02-13T00:41:04.036941+0000","last_became_active":"2024-02-13T00:41:01.601930+0000","last_became_peered":"2024-02-13T00:41:01.601930+0000","last_unstale":"2024-02-13T00:41:04.036941+0000","last_undegraded":"2024-02-13T00:41:04.036941+0000","last_fullsized":"2024-02-13T00:41:04.036941+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.022962+0000","last_change":"2024-02-13T00:41:01.590710+0000","last_active":"2024-02-13T00:41:04.022962+0000","last_peered":"2024-02-13T00:41:04.022962+0000","last_clean":"2024-02-13T00:41:04.022962+0000","last_became_active":"2024-02-13T00:41:01.590438+0000","last_became_peered":"2024-02-13T00:41:01.590438+0000","last_unstale":"2024-02-13T00:41:04.022962+0000","last_undegraded":"2024-02-13T00:41:04.022962+0000","last_fullsized":"2024-02-13T00:41:04.022962+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.037236+0000","last_change":"2024-02-13T00:41:04.037236+0000","last_active":"2024-02-13T00:41:04.037236+0000","last_peered":"2024-02-13T00:41:04.037236+0000","last_clean":"2024-02-13T00:41:04.037236+0000","last_became_active":"2024-02-13T00:41:01.682334+0000","last_became_peered":"2024-02-13T00:41:01.682334+0000","last_unstale":"2024-02-13T00:41:04.037236+0000","last_undegraded":"2024-02-13T00:41:04.037236+0000","last_fullsized":"2024-02-13T00:41:04.037236+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.988349+0000","last_change":"2024-02-13T00:41:00.546832+0000","last_active":"2024-02-13T00:41:03.988349+0000","last_peered":"2024-02-13T00:41:03.988349+0000","last_clean":"2024-02-13T00:41:03.988349+0000","last_became_active":"2024-02-13T00:41:00.546400+0000","last_became_peered":"2024-02-13T00:41:00.546400+0000","last_unstale":"2024-02-13T00:41:03.988349+0000","last_undegraded":"2024-02-13T00:41:03.988349+0000","last_fullsized":"2024-02-13T00:41:03.988349+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996233+0000","last_change":"2024-02-13T00:41:01.628842+0000","last_active":"2024-02-13T00:41:03.996233+0000","last_peered":"2024-02-13T00:41:03.996233+0000","last_clean":"2024-02-13T00:41:03.996233+0000","last_became_active":"2024-02-13T00:41:01.627528+0000","last_became_peered":"2024-02-13T00:41:01.627528+0000","last_unstale":"2024-02-13T00:41:03.996233+0000","last_undegraded":"2024-02-13T00:41:03.996233+0000","last_fullsized":"2024-02-13T00:41:03.996233+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.006706+0000","last_change":"2024-02-13T00:40:03.701680+0000","last_active":"2024-02-13T00:41:04.006706+0000","last_peered":"2024-02-13T00:41:04.006706+0000","last_clean":"2024-02-13T00:41:04.006706+0000","last_became_active":"2024-02-13T00:37:45.245956+0000","last_became_peered":"2024-02-13T00:37:45.245956+0000","last_unstale":"2024-02-13T00:41:04.006706+0000","last_undegraded":"2024-02-13T00:41:04.006706+0000","last_fullsized":"2024-02-13T00:41:04.006706+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:03.701604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:03.701604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":105,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.010437+0000","last_change":"2024-02-13T00:39:08.344202+0000","last_active":"2024-02-13T00:41:04.010437+0000","last_peered":"2024-02-13T00:41:04.010437+0000","last_clean":"2024-02-13T00:41:04.010437+0000","last_became_active":"2024-02-13T00:39:05.968155+0000","last_became_peered":"2024-02-13T00:39:05.968155+0000","last_unstale":"2024-02-13T00:41:04.010437+0000","last_undegraded":"2024-02-13T00:41:04.010437+0000","last_fullsized":"2024-02-13T00:41:04.010437+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:39:08.344129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.344129+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":43,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993700+0000","last_change":"2024-02-13T00:40:42.413113+0000","last_active":"2024-02-13T00:41:03.993700+0000","last_peered":"2024-02-13T00:41:03.993700+0000","last_clean":"2024-02-13T00:41:03.993700+0000","last_became_active":"2024-02-13T00:39:05.967772+0000","last_became_peered":"2024-02-13T00:39:05.967772+0000","last_unstale":"2024-02-13T00:41:03.993700+0000","last_undegraded":"2024-02-13T00:41:03.993700+0000","last_fullsized":"2024-02-13T00:41:03.993700+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:42.413043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:42.413043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.005333+0000","last_change":"2024-02-13T00:38:55.353229+0000","last_active":"2024-02-13T00:41:04.005333+0000","last_peered":"2024-02-13T00:41:04.005333+0000","last_clean":"2024-02-13T00:41:04.005333+0000","last_became_active":"2024-02-13T00:37:41.217577+0000","last_became_peered":"2024-02-13T00:37:41.217577+0000","last_unstale":"2024-02-13T00:41:04.005333+0000","last_undegraded":"2024-02-13T00:41:04.005333+0000","last_fullsized":"2024-02-13T00:41:04.005333+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:55.353157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:55.353157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":61,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.023032+0000","last_change":"2024-02-13T00:39:32.255327+0000","last_active":"2024-02-13T00:41:04.023032+0000","last_peered":"2024-02-13T00:41:04.023032+0000","last_clean":"2024-02-13T00:41:04.023032+0000","last_became_active":"2024-02-13T00:37:47.262024+0000","last_became_peered":"2024-02-13T00:37:47.262024+0000","last_unstale":"2024-02-13T00:41:04.023032+0000","last_undegraded":"2024-02-13T00:41:04.023032+0000","last_fullsized":"2024-02-13T00:41:04.023032+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.255260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995707+0000","last_change":"2024-02-13T00:41:01.662195+0000","last_active":"2024-02-13T00:41:03.995707+0000","last_peered":"2024-02-13T00:41:03.995707+0000","last_clean":"2024-02-13T00:41:03.995707+0000","last_became_active":"2024-02-13T00:41:01.660270+0000","last_became_peered":"2024-02-13T00:41:01.660270+0000","last_unstale":"2024-02-13T00:41:03.995707+0000","last_undegraded":"2024-02-13T00:41:03.995707+0000","last_fullsized":"2024-02-13T00:41:03.995707+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.16","version":"107'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.997516+0000","last_change":"2024-02-13T00:41:01.538909+0000","last_active":"2024-02-13T00:41:03.997516+0000","last_peered":"2024-02-13T00:41:03.997516+0000","last_clean":"2024-02-13T00:41:03.997516+0000","last_became_active":"2024-02-13T00:41:01.538309+0000","last_became_peered":"2024-02-13T00:41:01.538309+0000","last_unstale":"2024-02-13T00:41:03.997516+0000","last_undegraded":"2024-02-13T00:41:03.997516+0000","last_fullsized":"2024-02-13T00:41:03.997516+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.029951+0000","last_change":"2024-02-13T00:41:04.029951+0000","last_active":"2024-02-13T00:41:04.029951+0000","last_peered":"2024-02-13T00:41:04.029951+0000","last_clean":"2024-02-13T00:41:04.029951+0000","last_became_active":"2024-02-13T00:41:01.741357+0000","last_became_peered":"2024-02-13T00:41:01.741357+0000","last_unstale":"2024-02-13T00:41:04.029951+0000","last_undegraded":"2024-02-13T00:41:04.029951+0000","last_fullsized":"2024-02-13T00:41:04.029951+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.1d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.004816+0000","last_change":"2024-02-13T00:41:01.620285+0000","last_active":"2024-02-13T00:41:04.004816+0000","last_peered":"2024-02-13T00:41:04.004816+0000","last_clean":"2024-02-13T00:41:04.004816+0000","last_became_active":"2024-02-13T00:41:01.619449+0000","last_became_peered":"2024-02-13T00:41:01.619449+0000","last_unstale":"2024-02-13T00:41:04.004816+0000","last_undegraded":"2024-02-13T00:41:04.004816+0000","last_fullsized":"2024-02-13T00:41:04.004816+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.001817+0000","last_change":"2024-02-13T00:41:04.001817+0000","last_active":"2024-02-13T00:41:04.001817+0000","last_peered":"2024-02-13T00:41:04.001817+0000","last_clean":"2024-02-13T00:41:04.001817+0000","last_became_active":"2024-02-13T00:41:01.564467+0000","last_became_peered":"2024-02-13T00:41:01.564467+0000","last_unstale":"2024-02-13T00:41:04.001817+0000","last_undegraded":"2024-02-13T00:41:04.001817+0000","last_fullsized":"2024-02-13T00:41:04.001817+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.a","version":"109'3","reported_seq":21,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.842863+0000","last_change":"2024-02-13T00:41:00.551812+0000","last_active":"2024-02-13T00:41:04.842863+0000","last_peered":"2024-02-13T00:41:04.842863+0000","last_clean":"2024-02-13T00:41:04.842863+0000","last_became_active":"2024-02-13T00:41:00.550181+0000","last_became_peered":"2024-02-13T00:41:00.550181+0000","last_unstale":"2024-02-13T00:41:04.842863+0000","last_undegraded":"2024-02-13T00:41:04.842863+0000","last_fullsized":"2024-02-13T00:41:04.842863+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.001254+0000","last_change":"2024-02-13T00:41:01.636316+0000","last_active":"2024-02-13T00:41:04.001254+0000","last_peered":"2024-02-13T00:41:04.001254+0000","last_clean":"2024-02-13T00:41:04.001254+0000","last_became_active":"2024-02-13T00:41:01.632614+0000","last_became_peered":"2024-02-13T00:41:01.632614+0000","last_unstale":"2024-02-13T00:41:04.001254+0000","last_undegraded":"2024-02-13T00:41:04.001254+0000","last_fullsized":"2024-02-13T00:41:04.001254+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":62,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996133+0000","last_change":"2024-02-13T00:39:28.171808+0000","last_active":"2024-02-13T00:41:03.996133+0000","last_peered":"2024-02-13T00:41:03.996133+0000","last_clean":"2024-02-13T00:41:03.996133+0000","last_became_active":"2024-02-13T00:37:45.247708+0000","last_became_peered":"2024-02-13T00:37:45.247708+0000","last_unstale":"2024-02-13T00:41:03.996133+0000","last_undegraded":"2024-02-13T00:41:03.996133+0000","last_fullsized":"2024-02-13T00:41:03.996133+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.171740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.171740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'14","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996115+0000","last_change":"2024-02-13T00:39:15.415087+0000","last_active":"2024-02-13T00:41:03.996115+0000","last_peered":"2024-02-13T00:41:03.996115+0000","last_clean":"2024-02-13T00:41:03.996115+0000","last_became_active":"2024-02-13T00:37:43.324209+0000","last_became_peered":"2024-02-13T00:37:43.324209+0000","last_unstale":"2024-02-13T00:41:03.996115+0000","last_undegraded":"2024-02-13T00:41:03.996115+0000","last_fullsized":"2024-02-13T00:41:03.996115+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'14","last_scrub_stamp":"2024-02-13T00:39:15.415004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.415004+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993577+0000","last_change":"2024-02-13T00:39:41.255860+0000","last_active":"2024-02-13T00:41:03.993577+0000","last_peered":"2024-02-13T00:41:03.993577+0000","last_clean":"2024-02-13T00:41:03.993577+0000","last_became_active":"2024-02-13T00:37:35.635369+0000","last_became_peered":"2024-02-13T00:37:35.635369+0000","last_unstale":"2024-02-13T00:41:03.993577+0000","last_undegraded":"2024-02-13T00:41:03.993577+0000","last_fullsized":"2024-02-13T00:41:03.993577+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:39:41.255785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.999762+0000","last_change":"2024-02-13T00:38:52.618768+0000","last_active":"2024-02-13T00:41:03.999762+0000","last_peered":"2024-02-13T00:41:03.999762+0000","last_clean":"2024-02-13T00:41:03.999762+0000","last_became_active":"2024-02-13T00:37:41.216101+0000","last_became_peered":"2024-02-13T00:37:41.216101+0000","last_unstale":"2024-02-13T00:41:03.999762+0000","last_undegraded":"2024-02-13T00:41:03.999762+0000","last_fullsized":"2024-02-13T00:41:03.999762+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:52.618696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:52.618696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.992232+0000","last_change":"2024-02-13T00:40:09.099541+0000","last_active":"2024-02-13T00:41:03.992232+0000","last_peered":"2024-02-13T00:41:03.992232+0000","last_clean":"2024-02-13T00:41:03.992232+0000","last_became_active":"2024-02-13T00:37:41.216367+0000","last_became_peered":"2024-02-13T00:37:41.216367+0000","last_unstale":"2024-02-13T00:41:03.992232+0000","last_undegraded":"2024-02-13T00:41:03.992232+0000","last_fullsized":"2024-02-13T00:41:03.992232+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:09.099460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:40:09.099460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.006785+0000","last_change":"2024-02-13T00:40:08.577747+0000","last_active":"2024-02-13T00:41:04.006785+0000","last_peered":"2024-02-13T00:41:04.006785+0000","last_clean":"2024-02-13T00:41:04.006785+0000","last_became_active":"2024-02-13T00:37:47.253996+0000","last_became_peered":"2024-02-13T00:37:47.253996+0000","last_unstale":"2024-02-13T00:41:04.006785+0000","last_undegraded":"2024-02-13T00:41:04.006785+0000","last_fullsized":"2024-02-13T00:41:04.006785+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:08.577665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:40:08.577665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.13","version":"108'9","reported_seq":22,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003979+0000","last_change":"2024-02-13T00:41:01.560685+0000","last_active":"2024-02-13T00:41:04.003979+0000","last_peered":"2024-02-13T00:41:04.003979+0000","last_clean":"2024-02-13T00:41:04.003979+0000","last_became_active":"2024-02-13T00:41:01.560473+0000","last_became_peered":"2024-02-13T00:41:01.560473+0000","last_unstale":"2024-02-13T00:41:04.003979+0000","last_undegraded":"2024-02-13T00:41:04.003979+0000","last_fullsized":"2024-02-13T00:41:04.003979+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.15","version":"107'1","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996205+0000","last_change":"2024-02-13T00:41:01.656670+0000","last_active":"2024-02-13T00:41:03.996205+0000","last_peered":"2024-02-13T00:41:03.996205+0000","last_clean":"2024-02-13T00:41:03.996205+0000","last_became_active":"2024-02-13T00:41:01.655358+0000","last_became_peered":"2024-02-13T00:41:01.655358+0000","last_unstale":"2024-02-13T00:41:03.996205+0000","last_undegraded":"2024-02-13T00:41:03.996205+0000","last_fullsized":"2024-02-13T00:41:03.996205+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.021904+0000","last_change":"2024-02-13T00:41:04.021904+0000","last_active":"2024-02-13T00:41:04.021904+0000","last_peered":"2024-02-13T00:41:04.021904+0000","last_clean":"2024-02-13T00:41:04.021904+0000","last_became_active":"2024-02-13T00:41:01.594063+0000","last_became_peered":"2024-02-13T00:41:01.594063+0000","last_unstale":"2024-02-13T00:41:04.021904+0000","last_undegraded":"2024-02-13T00:41:04.021904+0000","last_fullsized":"2024-02-13T00:41:04.021904+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.019883+0000","last_change":"2024-02-13T00:41:01.594433+0000","last_active":"2024-02-13T00:41:04.019883+0000","last_peered":"2024-02-13T00:41:04.019883+0000","last_clean":"2024-02-13T00:41:04.019883+0000","last_became_active":"2024-02-13T00:41:01.593970+0000","last_became_peered":"2024-02-13T00:41:01.593970+0000","last_unstale":"2024-02-13T00:41:04.019883+0000","last_undegraded":"2024-02-13T00:41:04.019883+0000","last_fullsized":"2024-02-13T00:41:04.019883+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.027784+0000","last_change":"2024-02-13T00:41:04.027784+0000","last_active":"2024-02-13T00:41:04.027784+0000","last_peered":"2024-02-13T00:41:04.027784+0000","last_clean":"2024-02-13T00:41:04.027784+0000","last_became_active":"2024-02-13T00:41:01.654785+0000","last_became_peered":"2024-02-13T00:41:01.654785+0000","last_unstale":"2024-02-13T00:41:04.027784+0000","last_undegraded":"2024-02-13T00:41:04.027784+0000","last_fullsized":"2024-02-13T00:41:04.027784+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.987801+0000","last_change":"2024-02-13T00:41:00.547375+0000","last_active":"2024-02-13T00:41:03.987801+0000","last_peered":"2024-02-13T00:41:03.987801+0000","last_clean":"2024-02-13T00:41:03.987801+0000","last_became_active":"2024-02-13T00:41:00.547208+0000","last_became_peered":"2024-02-13T00:41:00.547208+0000","last_unstale":"2024-02-13T00:41:03.987801+0000","last_undegraded":"2024-02-13T00:41:03.987801+0000","last_fullsized":"2024-02-13T00:41:03.987801+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.988434+0000","last_change":"2024-02-13T00:41:01.611691+0000","last_active":"2024-02-13T00:41:03.988434+0000","last_peered":"2024-02-13T00:41:03.988434+0000","last_clean":"2024-02-13T00:41:03.988434+0000","last_became_active":"2024-02-13T00:41:01.611189+0000","last_became_peered":"2024-02-13T00:41:01.611189+0000","last_unstale":"2024-02-13T00:41:03.988434+0000","last_undegraded":"2024-02-13T00:41:03.988434+0000","last_fullsized":"2024-02-13T00:41:03.988434+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.023104+0000","last_change":"2024-02-13T00:41:01.412120+0000","last_active":"2024-02-13T00:41:04.023104+0000","last_peered":"2024-02-13T00:41:04.023104+0000","last_clean":"2024-02-13T00:41:04.023104+0000","last_became_active":"2024-02-13T00:37:45.248383+0000","last_became_peered":"2024-02-13T00:37:45.248383+0000","last_unstale":"2024-02-13T00:41:04.023104+0000","last_undegraded":"2024-02-13T00:41:04.023104+0000","last_fullsized":"2024-02-13T00:41:04.023104+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:01.412042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:01.412042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":107,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.020795+0000","last_change":"2024-02-13T00:39:25.319173+0000","last_active":"2024-02-13T00:41:04.020795+0000","last_peered":"2024-02-13T00:41:04.020795+0000","last_clean":"2024-02-13T00:41:04.020795+0000","last_became_active":"2024-02-13T00:37:43.231120+0000","last_became_peered":"2024-02-13T00:37:43.231120+0000","last_unstale":"2024-02-13T00:41:04.020795+0000","last_undegraded":"2024-02-13T00:41:04.020795+0000","last_fullsized":"2024-02-13T00:41:04.020795+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:39:25.319066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.319066+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.997301+0000","last_change":"2024-02-13T00:40:57.469853+0000","last_active":"2024-02-13T00:41:03.997301+0000","last_peered":"2024-02-13T00:41:03.997301+0000","last_clean":"2024-02-13T00:41:03.997301+0000","last_became_active":"2024-02-13T00:37:41.214442+0000","last_became_peered":"2024-02-13T00:37:41.214442+0000","last_unstale":"2024-02-13T00:41:03.997301+0000","last_undegraded":"2024-02-13T00:41:03.997301+0000","last_fullsized":"2024-02-13T00:41:03.997301+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.469743+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":269,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.712892+0000","last_change":"2024-02-13T00:39:10.287406+0000","last_active":"2024-02-13T00:41:04.712892+0000","last_peered":"2024-02-13T00:41:04.712892+0000","last_clean":"2024-02-13T00:41:04.712892+0000","last_became_active":"2024-02-13T00:37:41.215137+0000","last_became_peered":"2024-02-13T00:37:41.215137+0000","last_unstale":"2024-02-13T00:41:04.712892+0000","last_undegraded":"2024-02-13T00:41:04.712892+0000","last_fullsized":"2024-02-13T00:41:04.712892+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-13T00:39:10.287299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.287299+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":192,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.002999+0000","last_change":"2024-02-13T00:40:51.198184+0000","last_active":"2024-02-13T00:41:04.002999+0000","last_peered":"2024-02-13T00:41:04.002999+0000","last_clean":"2024-02-13T00:41:04.002999+0000","last_became_active":"2024-02-13T00:37:47.552693+0000","last_became_peered":"2024-02-13T00:37:47.552693+0000","last_unstale":"2024-02-13T00:41:04.002999+0000","last_undegraded":"2024-02-13T00:41:04.002999+0000","last_fullsized":"2024-02-13T00:41:04.002999+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.11","version":"108'7","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.032475+0000","last_change":"2024-02-13T00:41:01.721381+0000","last_active":"2024-02-13T00:41:04.032475+0000","last_peered":"2024-02-13T00:41:04.032475+0000","last_clean":"2024-02-13T00:41:04.032475+0000","last_became_active":"2024-02-13T00:41:01.718821+0000","last_became_peered":"2024-02-13T00:41:01.718821+0000","last_unstale":"2024-02-13T00:41:04.032475+0000","last_undegraded":"2024-02-13T00:41:04.032475+0000","last_fullsized":"2024-02-13T00:41:04.032475+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.17","version":"109'1","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.421906+0000","last_change":"2024-02-13T00:41:01.738284+0000","last_active":"2024-02-13T00:41:04.421906+0000","last_peered":"2024-02-13T00:41:04.421906+0000","last_clean":"2024-02-13T00:41:04.421906+0000","last_became_active":"2024-02-13T00:41:01.738000+0000","last_became_peered":"2024-02-13T00:41:01.738000+0000","last_unstale":"2024-02-13T00:41:04.421906+0000","last_undegraded":"2024-02-13T00:41:04.421906+0000","last_fullsized":"2024-02-13T00:41:04.421906+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1b","version":"107'1","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.014810+0000","last_change":"2024-02-13T00:41:04.014810+0000","last_active":"2024-02-13T00:41:04.014810+0000","last_peered":"2024-02-13T00:41:04.014810+0000","last_clean":"2024-02-13T00:41:04.014810+0000","last_became_active":"2024-02-13T00:41:01.588823+0000","last_became_peered":"2024-02-13T00:41:01.588823+0000","last_unstale":"2024-02-13T00:41:04.014810+0000","last_undegraded":"2024-02-13T00:41:04.014810+0000","last_fullsized":"2024-02-13T00:41:04.014810+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.991550+0000","last_change":"2024-02-13T00:41:01.548873+0000","last_active":"2024-02-13T00:41:03.991550+0000","last_peered":"2024-02-13T00:41:03.991550+0000","last_clean":"2024-02-13T00:41:03.991550+0000","last_became_active":"2024-02-13T00:41:01.547408+0000","last_became_peered":"2024-02-13T00:41:01.547408+0000","last_unstale":"2024-02-13T00:41:03.991550+0000","last_undegraded":"2024-02-13T00:41:03.991550+0000","last_fullsized":"2024-02-13T00:41:03.991550+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.016628+0000","last_change":"2024-02-13T00:41:04.016628+0000","last_active":"2024-02-13T00:41:04.016628+0000","last_peered":"2024-02-13T00:41:04.016628+0000","last_clean":"2024-02-13T00:41:04.016628+0000","last_became_active":"2024-02-13T00:41:01.595626+0000","last_became_peered":"2024-02-13T00:41:01.595626+0000","last_unstale":"2024-02-13T00:41:04.016628+0000","last_undegraded":"2024-02-13T00:41:04.016628+0000","last_fullsized":"2024-02-13T00:41:04.016628+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.b","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.005360+0000","last_change":"2024-02-13T00:41:00.551111+0000","last_active":"2024-02-13T00:41:04.005360+0000","last_peered":"2024-02-13T00:41:04.005360+0000","last_clean":"2024-02-13T00:41:04.005360+0000","last_became_active":"2024-02-13T00:41:00.543532+0000","last_became_peered":"2024-02-13T00:41:00.543532+0000","last_unstale":"2024-02-13T00:41:04.005360+0000","last_undegraded":"2024-02-13T00:41:04.005360+0000","last_fullsized":"2024-02-13T00:41:04.005360+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.015258+0000","last_change":"2024-02-13T00:41:01.558014+0000","last_active":"2024-02-13T00:41:04.015258+0000","last_peered":"2024-02-13T00:41:04.015258+0000","last_clean":"2024-02-13T00:41:04.015258+0000","last_became_active":"2024-02-13T00:41:01.557527+0000","last_became_peered":"2024-02-13T00:41:01.557527+0000","last_unstale":"2024-02-13T00:41:04.015258+0000","last_undegraded":"2024-02-13T00:41:04.015258+0000","last_fullsized":"2024-02-13T00:41:04.015258+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.013246+0000","last_change":"2024-02-13T00:41:00.881481+0000","last_active":"2024-02-13T00:41:04.013246+0000","last_peered":"2024-02-13T00:41:04.013246+0000","last_clean":"2024-02-13T00:41:04.013246+0000","last_became_active":"2024-02-13T00:37:45.247012+0000","last_became_peered":"2024-02-13T00:37:45.247012+0000","last_unstale":"2024-02-13T00:41:04.013246+0000","last_undegraded":"2024-02-13T00:41:04.013246+0000","last_fullsized":"2024-02-13T00:41:04.013246+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.881411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.881411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"80'45","reported_seq":185,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.249802+0000","last_change":"2024-02-13T00:40:12.047980+0000","last_active":"2024-02-13T00:41:04.249802+0000","last_peered":"2024-02-13T00:41:04.249802+0000","last_clean":"2024-02-13T00:41:04.249802+0000","last_became_active":"2024-02-13T00:37:43.232014+0000","last_became_peered":"2024-02-13T00:37:43.232014+0000","last_unstale":"2024-02-13T00:41:04.249802+0000","last_undegraded":"2024-02-13T00:41:04.249802+0000","last_fullsized":"2024-02-13T00:41:04.249802+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'45","last_scrub_stamp":"2024-02-13T00:40:12.047869+0000","last_deep_scrub":"80'45","last_deep_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_clean_scrub_stamp":"2024-02-13T00:40:12.047869+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.990075+0000","last_change":"2024-02-13T00:40:59.518865+0000","last_active":"2024-02-13T00:41:03.990075+0000","last_peered":"2024-02-13T00:41:03.990075+0000","last_clean":"2024-02-13T00:41:03.990075+0000","last_became_active":"2024-02-13T00:37:41.212924+0000","last_became_peered":"2024-02-13T00:37:41.212924+0000","last_unstale":"2024-02-13T00:41:03.990075+0000","last_undegraded":"2024-02-13T00:41:03.990075+0000","last_fullsized":"2024-02-13T00:41:03.990075+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.518778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.518778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":60,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.000551+0000","last_change":"2024-02-13T00:39:36.157777+0000","last_active":"2024-02-13T00:41:04.000551+0000","last_peered":"2024-02-13T00:41:04.000551+0000","last_clean":"2024-02-13T00:41:04.000551+0000","last_became_active":"2024-02-13T00:37:47.252270+0000","last_became_peered":"2024-02-13T00:37:47.252270+0000","last_unstale":"2024-02-13T00:41:04.000551+0000","last_undegraded":"2024-02-13T00:41:04.000551+0000","last_fullsized":"2024-02-13T00:41:04.000551+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:36.157693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:36.157693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.002750+0000","last_change":"2024-02-13T00:41:01.606464+0000","last_active":"2024-02-13T00:41:04.002750+0000","last_peered":"2024-02-13T00:41:04.002750+0000","last_clean":"2024-02-13T00:41:04.002750+0000","last_became_active":"2024-02-13T00:41:01.605731+0000","last_became_peered":"2024-02-13T00:41:01.605731+0000","last_unstale":"2024-02-13T00:41:04.002750+0000","last_undegraded":"2024-02-13T00:41:04.002750+0000","last_fullsized":"2024-02-13T00:41:04.002750+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"109'1001","reported_seq":2014,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.228772+0000","last_change":"2024-02-13T00:41:01.666080+0000","last_active":"2024-02-13T00:41:04.228772+0000","last_peered":"2024-02-13T00:41:04.228772+0000","last_clean":"2024-02-13T00:41:04.228772+0000","last_became_active":"2024-02-13T00:41:01.665589+0000","last_became_peered":"2024-02-13T00:41:01.665589+0000","last_unstale":"2024-02-13T00:41:04.228772+0000","last_undegraded":"2024-02-13T00:41:04.228772+0000","last_fullsized":"2024-02-13T00:41:04.228772+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1001,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1000,"num_read_kb":0,"num_write":1000,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.013241+0000","last_change":"2024-02-13T00:41:04.013241+0000","last_active":"2024-02-13T00:41:04.013241+0000","last_peered":"2024-02-13T00:41:04.013241+0000","last_clean":"2024-02-13T00:41:04.013241+0000","last_became_active":"2024-02-13T00:41:01.631455+0000","last_became_peered":"2024-02-13T00:41:01.631455+0000","last_unstale":"2024-02-13T00:41:04.013241+0000","last_undegraded":"2024-02-13T00:41:04.013241+0000","last_fullsized":"2024-02-13T00:41:04.013241+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.1a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.017530+0000","last_change":"2024-02-13T00:41:01.644491+0000","last_active":"2024-02-13T00:41:04.017530+0000","last_peered":"2024-02-13T00:41:04.017530+0000","last_clean":"2024-02-13T00:41:04.017530+0000","last_became_active":"2024-02-13T00:41:01.643774+0000","last_became_peered":"2024-02-13T00:41:01.643774+0000","last_unstale":"2024-02-13T00:41:04.017530+0000","last_undegraded":"2024-02-13T00:41:04.017530+0000","last_fullsized":"2024-02-13T00:41:04.017530+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1f","version":"107'1","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.027269+0000","last_change":"2024-02-13T00:41:04.027269+0000","last_active":"2024-02-13T00:41:04.027269+0000","last_peered":"2024-02-13T00:41:04.027269+0000","last_clean":"2024-02-13T00:41:04.027269+0000","last_became_active":"2024-02-13T00:41:01.645145+0000","last_became_peered":"2024-02-13T00:41:01.645145+0000","last_unstale":"2024-02-13T00:41:04.027269+0000","last_undegraded":"2024-02-13T00:41:04.027269+0000","last_fullsized":"2024-02-13T00:41:04.027269+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.d","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.009856+0000","last_change":"2024-02-13T00:41:00.522881+0000","last_active":"2024-02-13T00:41:04.009856+0000","last_peered":"2024-02-13T00:41:04.009856+0000","last_clean":"2024-02-13T00:41:04.009856+0000","last_became_active":"2024-02-13T00:41:00.522631+0000","last_became_peered":"2024-02-13T00:41:00.522631+0000","last_unstale":"2024-02-13T00:41:04.009856+0000","last_undegraded":"2024-02-13T00:41:04.009856+0000","last_fullsized":"2024-02-13T00:41:04.009856+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995731+0000","last_change":"2024-02-13T00:41:01.598530+0000","last_active":"2024-02-13T00:41:03.995731+0000","last_peered":"2024-02-13T00:41:03.995731+0000","last_clean":"2024-02-13T00:41:03.995731+0000","last_became_active":"2024-02-13T00:41:01.597430+0000","last_became_peered":"2024-02-13T00:41:01.597430+0000","last_unstale":"2024-02-13T00:41:03.995731+0000","last_undegraded":"2024-02-13T00:41:03.995731+0000","last_fullsized":"2024-02-13T00:41:03.995731+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"64'8","reported_seq":111,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.249732+0000","last_change":"2024-02-13T00:39:30.273237+0000","last_active":"2024-02-13T00:41:04.249732+0000","last_peered":"2024-02-13T00:41:04.249732+0000","last_clean":"2024-02-13T00:41:04.249732+0000","last_became_active":"2024-02-13T00:37:45.241274+0000","last_became_peered":"2024-02-13T00:37:45.241274+0000","last_unstale":"2024-02-13T00:41:04.249732+0000","last_undegraded":"2024-02-13T00:41:04.249732+0000","last_fullsized":"2024-02-13T00:41:04.249732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:30.273165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.273165+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.002083+0000","last_change":"2024-02-13T00:40:16.354241+0000","last_active":"2024-02-13T00:41:04.002083+0000","last_peered":"2024-02-13T00:41:04.002083+0000","last_clean":"2024-02-13T00:41:04.002083+0000","last_became_active":"2024-02-13T00:39:05.965560+0000","last_became_peered":"2024-02-13T00:39:05.965560+0000","last_unstale":"2024-02-13T00:41:04.002083+0000","last_undegraded":"2024-02-13T00:41:04.002083+0000","last_fullsized":"2024-02-13T00:41:04.002083+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:40:16.354164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:16.354164+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.022556+0000","last_change":"2024-02-13T00:40:08.946462+0000","last_active":"2024-02-13T00:41:04.022556+0000","last_peered":"2024-02-13T00:41:04.022556+0000","last_clean":"2024-02-13T00:41:04.022556+0000","last_became_active":"2024-02-13T00:37:41.217777+0000","last_became_peered":"2024-02-13T00:37:41.217777+0000","last_unstale":"2024-02-13T00:41:04.022556+0000","last_undegraded":"2024-02-13T00:41:04.022556+0000","last_fullsized":"2024-02-13T00:41:04.022556+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:08.946388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:40:08.946388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":60,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995822+0000","last_change":"2024-02-13T00:38:54.349240+0000","last_active":"2024-02-13T00:41:03.995822+0000","last_peered":"2024-02-13T00:41:03.995822+0000","last_clean":"2024-02-13T00:41:03.995822+0000","last_became_active":"2024-02-13T00:37:47.255928+0000","last_became_peered":"2024-02-13T00:37:47.255928+0000","last_unstale":"2024-02-13T00:41:03.995822+0000","last_undegraded":"2024-02-13T00:41:03.995822+0000","last_fullsized":"2024-02-13T00:41:03.995822+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:54.349171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:54.349171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"107'2","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.004944+0000","last_change":"2024-02-13T00:41:01.616697+0000","last_active":"2024-02-13T00:41:04.004944+0000","last_peered":"2024-02-13T00:41:04.004944+0000","last_clean":"2024-02-13T00:41:04.004944+0000","last_became_active":"2024-02-13T00:41:01.616419+0000","last_became_peered":"2024-02-13T00:41:01.616419+0000","last_unstale":"2024-02-13T00:41:04.004944+0000","last_undegraded":"2024-02-13T00:41:04.004944+0000","last_fullsized":"2024-02-13T00:41:04.004944+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.017453+0000","last_change":"2024-02-13T00:41:01.646177+0000","last_active":"2024-02-13T00:41:04.017453+0000","last_peered":"2024-02-13T00:41:04.017453+0000","last_clean":"2024-02-13T00:41:04.017453+0000","last_became_active":"2024-02-13T00:41:01.644151+0000","last_became_peered":"2024-02-13T00:41:01.644151+0000","last_unstale":"2024-02-13T00:41:04.017453+0000","last_undegraded":"2024-02-13T00:41:04.017453+0000","last_fullsized":"2024-02-13T00:41:04.017453+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.019687+0000","last_change":"2024-02-13T00:41:04.019687+0000","last_active":"2024-02-13T00:41:04.019687+0000","last_peered":"2024-02-13T00:41:04.019687+0000","last_clean":"2024-02-13T00:41:04.019687+0000","last_became_active":"2024-02-13T00:41:01.650747+0000","last_became_peered":"2024-02-13T00:41:01.650747+0000","last_unstale":"2024-02-13T00:41:04.019687+0000","last_undegraded":"2024-02-13T00:41:04.019687+0000","last_fullsized":"2024-02-13T00:41:04.019687+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.027829+0000","last_change":"2024-02-13T00:41:01.721182+0000","last_active":"2024-02-13T00:41:04.027829+0000","last_peered":"2024-02-13T00:41:04.027829+0000","last_clean":"2024-02-13T00:41:04.027829+0000","last_became_active":"2024-02-13T00:41:01.718793+0000","last_became_peered":"2024-02-13T00:41:01.718793+0000","last_unstale":"2024-02-13T00:41:04.027829+0000","last_undegraded":"2024-02-13T00:41:04.027829+0000","last_fullsized":"2024-02-13T00:41:04.027829+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.002627+0000","last_change":"2024-02-13T00:41:04.002627+0000","last_active":"2024-02-13T00:41:04.002627+0000","last_peered":"2024-02-13T00:41:04.002627+0000","last_clean":"2024-02-13T00:41:04.002627+0000","last_became_active":"2024-02-13T00:41:01.653085+0000","last_became_peered":"2024-02-13T00:41:01.653085+0000","last_unstale":"2024-02-13T00:41:04.002627+0000","last_undegraded":"2024-02-13T00:41:04.002627+0000","last_fullsized":"2024-02-13T00:41:04.002627+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.6","version":"109'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.845944+0000","last_change":"2024-02-13T00:41:01.747074+0000","last_active":"2024-02-13T00:41:04.845944+0000","last_peered":"2024-02-13T00:41:04.845944+0000","last_clean":"2024-02-13T00:41:04.845944+0000","last_became_active":"2024-02-13T00:41:01.739293+0000","last_became_peered":"2024-02-13T00:41:01.739293+0000","last_unstale":"2024-02-13T00:41:04.845944+0000","last_undegraded":"2024-02-13T00:41:04.845944+0000","last_fullsized":"2024-02-13T00:41:04.845944+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.013516+0000","last_change":"2024-02-13T00:41:01.662077+0000","last_active":"2024-02-13T00:41:04.013516+0000","last_peered":"2024-02-13T00:41:04.013516+0000","last_clean":"2024-02-13T00:41:04.013516+0000","last_became_active":"2024-02-13T00:41:01.660820+0000","last_became_peered":"2024-02-13T00:41:01.660820+0000","last_unstale":"2024-02-13T00:41:04.013516+0000","last_undegraded":"2024-02-13T00:41:04.013516+0000","last_fullsized":"2024-02-13T00:41:04.013516+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":63,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993555+0000","last_change":"2024-02-13T00:39:21.654834+0000","last_active":"2024-02-13T00:41:03.993555+0000","last_peered":"2024-02-13T00:41:03.993555+0000","last_clean":"2024-02-13T00:41:03.993555+0000","last_became_active":"2024-02-13T00:37:45.244408+0000","last_became_peered":"2024-02-13T00:37:45.244408+0000","last_unstale":"2024-02-13T00:41:03.993555+0000","last_undegraded":"2024-02-13T00:41:03.993555+0000","last_fullsized":"2024-02-13T00:41:03.993555+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.654733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.654733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":92,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.027740+0000","last_change":"2024-02-13T00:40:35.856095+0000","last_active":"2024-02-13T00:41:04.027740+0000","last_peered":"2024-02-13T00:41:04.027740+0000","last_clean":"2024-02-13T00:41:04.027740+0000","last_became_active":"2024-02-13T00:40:35.855722+0000","last_became_peered":"2024-02-13T00:40:35.855722+0000","last_unstale":"2024-02-13T00:41:04.027740+0000","last_undegraded":"2024-02-13T00:41:04.027740+0000","last_fullsized":"2024-02-13T00:41:04.027740+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'24","last_scrub_stamp":"2024-02-13T00:39:56.121046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:56.121046+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":36,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":40,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.019526+0000","last_change":"2024-02-13T00:40:48.809660+0000","last_active":"2024-02-13T00:41:04.019526+0000","last_peered":"2024-02-13T00:41:04.019526+0000","last_clean":"2024-02-13T00:41:04.019526+0000","last_became_active":"2024-02-13T00:40:35.852351+0000","last_became_peered":"2024-02-13T00:40:35.852351+0000","last_unstale":"2024-02-13T00:41:04.019526+0000","last_undegraded":"2024-02-13T00:41:04.019526+0000","last_fullsized":"2024-02-13T00:41:04.019526+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:40:48.809594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:48.809594+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":60,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996753+0000","last_change":"2024-02-13T00:39:33.179309+0000","last_active":"2024-02-13T00:41:03.996753+0000","last_peered":"2024-02-13T00:41:03.996753+0000","last_clean":"2024-02-13T00:41:03.996753+0000","last_became_active":"2024-02-13T00:37:47.260242+0000","last_became_peered":"2024-02-13T00:37:47.260242+0000","last_unstale":"2024-02-13T00:41:03.996753+0000","last_undegraded":"2024-02-13T00:41:03.996753+0000","last_fullsized":"2024-02-13T00:41:03.996753+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.179225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.179225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"108'5","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995629+0000","last_change":"2024-02-13T00:41:01.638321+0000","last_active":"2024-02-13T00:41:03.995629+0000","last_peered":"2024-02-13T00:41:03.995629+0000","last_clean":"2024-02-13T00:41:03.995629+0000","last_became_active":"2024-02-13T00:41:01.637897+0000","last_became_peered":"2024-02-13T00:41:01.637897+0000","last_unstale":"2024-02-13T00:41:03.995629+0000","last_undegraded":"2024-02-13T00:41:03.995629+0000","last_fullsized":"2024-02-13T00:41:03.995629+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.19","version":"109'1","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.546131+0000","last_change":"2024-02-13T00:41:01.593406+0000","last_active":"2024-02-13T00:41:04.546131+0000","last_peered":"2024-02-13T00:41:04.546131+0000","last_clean":"2024-02-13T00:41:04.546131+0000","last_became_active":"2024-02-13T00:41:01.593112+0000","last_became_peered":"2024-02-13T00:41:01.593112+0000","last_unstale":"2024-02-13T00:41:04.546131+0000","last_undegraded":"2024-02-13T00:41:04.546131+0000","last_fullsized":"2024-02-13T00:41:04.546131+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.014989+0000","last_change":"2024-02-13T00:41:04.014989+0000","last_active":"2024-02-13T00:41:04.014989+0000","last_peered":"2024-02-13T00:41:04.014989+0000","last_clean":"2024-02-13T00:41:04.014989+0000","last_became_active":"2024-02-13T00:41:01.559612+0000","last_became_peered":"2024-02-13T00:41:01.559612+0000","last_unstale":"2024-02-13T00:41:04.014989+0000","last_undegraded":"2024-02-13T00:41:04.014989+0000","last_fullsized":"2024-02-13T00:41:04.014989+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.010563+0000","last_change":"2024-02-13T00:41:01.594534+0000","last_active":"2024-02-13T00:41:04.010563+0000","last_peered":"2024-02-13T00:41:04.010563+0000","last_clean":"2024-02-13T00:41:04.010563+0000","last_became_active":"2024-02-13T00:41:01.590676+0000","last_became_peered":"2024-02-13T00:41:01.590676+0000","last_unstale":"2024-02-13T00:41:04.010563+0000","last_undegraded":"2024-02-13T00:41:04.010563+0000","last_fullsized":"2024-02-13T00:41:04.010563+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.037098+0000","last_change":"2024-02-13T00:41:04.037098+0000","last_active":"2024-02-13T00:41:04.037098+0000","last_peered":"2024-02-13T00:41:04.037098+0000","last_clean":"2024-02-13T00:41:04.037098+0000","last_became_active":"2024-02-13T00:41:01.658864+0000","last_became_peered":"2024-02-13T00:41:01.658864+0000","last_unstale":"2024-02-13T00:41:04.037098+0000","last_undegraded":"2024-02-13T00:41:04.037098+0000","last_fullsized":"2024-02-13T00:41:04.037098+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.004821+0000","last_change":"2024-02-13T00:41:01.632066+0000","last_active":"2024-02-13T00:41:04.004821+0000","last_peered":"2024-02-13T00:41:04.004821+0000","last_clean":"2024-02-13T00:41:04.004821+0000","last_became_active":"2024-02-13T00:41:01.631787+0000","last_became_peered":"2024-02-13T00:41:01.631787+0000","last_unstale":"2024-02-13T00:41:04.004821+0000","last_undegraded":"2024-02-13T00:41:04.004821+0000","last_fullsized":"2024-02-13T00:41:04.004821+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993463+0000","last_change":"2024-02-13T00:41:00.530334+0000","last_active":"2024-02-13T00:41:03.993463+0000","last_peered":"2024-02-13T00:41:03.993463+0000","last_clean":"2024-02-13T00:41:03.993463+0000","last_became_active":"2024-02-13T00:41:00.529675+0000","last_became_peered":"2024-02-13T00:41:00.529675+0000","last_unstale":"2024-02-13T00:41:03.993463+0000","last_undegraded":"2024-02-13T00:41:03.993463+0000","last_fullsized":"2024-02-13T00:41:03.993463+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":115,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.010547+0000","last_change":"2024-02-13T00:40:49.767975+0000","last_active":"2024-02-13T00:41:04.010547+0000","last_peered":"2024-02-13T00:41:04.010547+0000","last_clean":"2024-02-13T00:41:04.010547+0000","last_became_active":"2024-02-13T00:37:43.322401+0000","last_became_peered":"2024-02-13T00:37:43.322401+0000","last_unstale":"2024-02-13T00:41:04.010547+0000","last_undegraded":"2024-02-13T00:41:04.010547+0000","last_fullsized":"2024-02-13T00:41:04.010547+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:40:49.767853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:49.767853+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993419+0000","last_change":"2024-02-13T00:40:07.270252+0000","last_active":"2024-02-13T00:41:03.993419+0000","last_peered":"2024-02-13T00:41:03.993419+0000","last_clean":"2024-02-13T00:41:03.993419+0000","last_became_active":"2024-02-13T00:37:41.214091+0000","last_became_peered":"2024-02-13T00:37:41.214091+0000","last_unstale":"2024-02-13T00:41:03.993419+0000","last_undegraded":"2024-02-13T00:41:03.993419+0000","last_fullsized":"2024-02-13T00:41:03.993419+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:07.270157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:07.270157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":62,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.997255+0000","last_change":"2024-02-13T00:39:46.155792+0000","last_active":"2024-02-13T00:41:03.997255+0000","last_peered":"2024-02-13T00:41:03.997255+0000","last_clean":"2024-02-13T00:41:03.997255+0000","last_became_active":"2024-02-13T00:37:45.250632+0000","last_became_peered":"2024-02-13T00:37:45.250632+0000","last_unstale":"2024-02-13T00:41:03.997255+0000","last_undegraded":"2024-02-13T00:41:03.997255+0000","last_fullsized":"2024-02-13T00:41:03.997255+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:46.155718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:46.155718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.002888+0000","last_change":"2024-02-13T00:40:59.468608+0000","last_active":"2024-02-13T00:41:04.002888+0000","last_peered":"2024-02-13T00:41:04.002888+0000","last_clean":"2024-02-13T00:41:04.002888+0000","last_became_active":"2024-02-13T00:37:47.254803+0000","last_became_peered":"2024-02-13T00:37:47.254803+0000","last_unstale":"2024-02-13T00:41:04.002888+0000","last_undegraded":"2024-02-13T00:41:04.002888+0000","last_fullsized":"2024-02-13T00:41:04.002888+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.468525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.468525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.0","version":"108'5","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996028+0000","last_change":"2024-02-13T00:41:01.580498+0000","last_active":"2024-02-13T00:41:03.996028+0000","last_peered":"2024-02-13T00:41:03.996028+0000","last_clean":"2024-02-13T00:41:03.996028+0000","last_became_active":"2024-02-13T00:41:01.579665+0000","last_became_peered":"2024-02-13T00:41:01.579665+0000","last_unstale":"2024-02-13T00:41:03.996028+0000","last_undegraded":"2024-02-13T00:41:03.996028+0000","last_fullsized":"2024-02-13T00:41:03.996028+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.6","version":"109'2","reported_seq":23,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.497782+0000","last_change":"2024-02-13T00:41:01.708359+0000","last_active":"2024-02-13T00:41:04.497782+0000","last_peered":"2024-02-13T00:41:04.497782+0000","last_clean":"2024-02-13T00:41:04.497782+0000","last_became_active":"2024-02-13T00:41:01.707941+0000","last_became_peered":"2024-02-13T00:41:01.707941+0000","last_unstale":"2024-02-13T00:41:04.497782+0000","last_undegraded":"2024-02-13T00:41:04.497782+0000","last_fullsized":"2024-02-13T00:41:04.497782+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.015707+0000","last_change":"2024-02-13T00:41:04.015707+0000","last_active":"2024-02-13T00:41:04.015707+0000","last_peered":"2024-02-13T00:41:04.015707+0000","last_clean":"2024-02-13T00:41:04.015707+0000","last_became_active":"2024-02-13T00:41:01.566972+0000","last_became_peered":"2024-02-13T00:41:01.566972+0000","last_unstale":"2024-02-13T00:41:04.015707+0000","last_undegraded":"2024-02-13T00:41:04.015707+0000","last_fullsized":"2024-02-13T00:41:04.015707+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.016043+0000","last_change":"2024-02-13T00:41:01.652447+0000","last_active":"2024-02-13T00:41:04.016043+0000","last_peered":"2024-02-13T00:41:04.016043+0000","last_clean":"2024-02-13T00:41:04.016043+0000","last_became_active":"2024-02-13T00:41:01.645165+0000","last_became_peered":"2024-02-13T00:41:01.645165+0000","last_unstale":"2024-02-13T00:41:04.016043+0000","last_undegraded":"2024-02-13T00:41:04.016043+0000","last_fullsized":"2024-02-13T00:41:04.016043+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.016257+0000","last_change":"2024-02-13T00:41:04.016257+0000","last_active":"2024-02-13T00:41:04.016257+0000","last_peered":"2024-02-13T00:41:04.016257+0000","last_clean":"2024-02-13T00:41:04.016257+0000","last_became_active":"2024-02-13T00:41:01.625550+0000","last_became_peered":"2024-02-13T00:41:01.625550+0000","last_unstale":"2024-02-13T00:41:04.016257+0000","last_undegraded":"2024-02-13T00:41:04.016257+0000","last_fullsized":"2024-02-13T00:41:04.016257+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.18","version":"107'3","reported_seq":19,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993803+0000","last_change":"2024-02-13T00:41:00.526228+0000","last_active":"2024-02-13T00:41:03.993803+0000","last_peered":"2024-02-13T00:41:03.993803+0000","last_clean":"2024-02-13T00:41:03.993803+0000","last_became_active":"2024-02-13T00:41:00.523270+0000","last_became_peered":"2024-02-13T00:41:00.523270+0000","last_unstale":"2024-02-13T00:41:03.993803+0000","last_undegraded":"2024-02-13T00:41:03.993803+0000","last_fullsized":"2024-02-13T00:41:03.993803+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"107'5","reported_seq":22,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.027612+0000","last_change":"2024-02-13T00:41:01.679305+0000","last_active":"2024-02-13T00:41:04.027612+0000","last_peered":"2024-02-13T00:41:04.027612+0000","last_clean":"2024-02-13T00:41:04.027612+0000","last_became_active":"2024-02-13T00:41:01.676434+0000","last_became_peered":"2024-02-13T00:41:01.676434+0000","last_unstale":"2024-02-13T00:41:04.027612+0000","last_undegraded":"2024-02-13T00:41:04.027612+0000","last_fullsized":"2024-02-13T00:41:04.027612+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":88,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.997188+0000","last_change":"2024-02-13T00:39:19.266801+0000","last_active":"2024-02-13T00:41:03.997188+0000","last_peered":"2024-02-13T00:41:03.997188+0000","last_clean":"2024-02-13T00:41:03.997188+0000","last_became_active":"2024-02-13T00:37:43.324510+0000","last_became_peered":"2024-02-13T00:37:43.324510+0000","last_unstale":"2024-02-13T00:41:03.997188+0000","last_undegraded":"2024-02-13T00:41:03.997188+0000","last_fullsized":"2024-02-13T00:41:03.997188+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:39:19.266729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:19.266729+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.008007+0000","last_change":"2024-02-13T00:40:54.316667+0000","last_active":"2024-02-13T00:41:04.008007+0000","last_peered":"2024-02-13T00:41:04.008007+0000","last_clean":"2024-02-13T00:41:04.008007+0000","last_became_active":"2024-02-13T00:37:41.212866+0000","last_became_peered":"2024-02-13T00:37:41.212866+0000","last_unstale":"2024-02-13T00:41:04.008007+0000","last_undegraded":"2024-02-13T00:41:04.008007+0000","last_fullsized":"2024-02-13T00:41:04.008007+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:54.316591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:40:54.316591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.015370+0000","last_change":"2024-02-13T00:40:53.536240+0000","last_active":"2024-02-13T00:41:04.015370+0000","last_peered":"2024-02-13T00:41:04.015370+0000","last_clean":"2024-02-13T00:41:04.015370+0000","last_became_active":"2024-02-13T00:37:45.246907+0000","last_became_peered":"2024-02-13T00:37:45.246907+0000","last_unstale":"2024-02-13T00:41:04.015370+0000","last_undegraded":"2024-02-13T00:41:04.015370+0000","last_fullsized":"2024-02-13T00:41:04.015370+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:53.536170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:53.536170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.002490+0000","last_change":"2024-02-13T00:41:02.115927+0000","last_active":"2024-02-13T00:41:04.002490+0000","last_peered":"2024-02-13T00:41:04.002490+0000","last_clean":"2024-02-13T00:41:04.002490+0000","last_became_active":"2024-02-13T00:37:47.553612+0000","last_became_peered":"2024-02-13T00:37:47.553612+0000","last_unstale":"2024-02-13T00:41:04.002490+0000","last_undegraded":"2024-02-13T00:41:04.002490+0000","last_fullsized":"2024-02-13T00:41:04.002490+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.115856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.115856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1","version":"108'5","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.990695+0000","last_change":"2024-02-13T00:41:01.722177+0000","last_active":"2024-02-13T00:41:03.990695+0000","last_peered":"2024-02-13T00:41:03.990695+0000","last_clean":"2024-02-13T00:41:03.990695+0000","last_became_active":"2024-02-13T00:41:01.721284+0000","last_became_peered":"2024-02-13T00:41:01.721284+0000","last_unstale":"2024-02-13T00:41:03.990695+0000","last_undegraded":"2024-02-13T00:41:03.990695+0000","last_fullsized":"2024-02-13T00:41:03.990695+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995342+0000","last_change":"2024-02-13T00:41:01.560226+0000","last_active":"2024-02-13T00:41:03.995342+0000","last_peered":"2024-02-13T00:41:03.995342+0000","last_clean":"2024-02-13T00:41:03.995342+0000","last_became_active":"2024-02-13T00:41:01.560020+0000","last_became_peered":"2024-02-13T00:41:01.560020+0000","last_unstale":"2024-02-13T00:41:03.995342+0000","last_undegraded":"2024-02-13T00:41:03.995342+0000","last_fullsized":"2024-02-13T00:41:03.995342+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.043148+0000","last_change":"2024-02-13T00:41:04.043148+0000","last_active":"2024-02-13T00:41:04.043148+0000","last_peered":"2024-02-13T00:41:04.043148+0000","last_clean":"2024-02-13T00:41:04.043148+0000","last_became_active":"2024-02-13T00:41:01.713434+0000","last_became_peered":"2024-02-13T00:41:01.713434+0000","last_unstale":"2024-02-13T00:41:04.043148+0000","last_undegraded":"2024-02-13T00:41:04.043148+0000","last_fullsized":"2024-02-13T00:41:04.043148+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.990899+0000","last_change":"2024-02-13T00:41:01.653337+0000","last_active":"2024-02-13T00:41:03.990899+0000","last_peered":"2024-02-13T00:41:03.990899+0000","last_clean":"2024-02-13T00:41:03.990899+0000","last_became_active":"2024-02-13T00:41:01.645866+0000","last_became_peered":"2024-02-13T00:41:01.645866+0000","last_unstale":"2024-02-13T00:41:03.990899+0000","last_undegraded":"2024-02-13T00:41:03.990899+0000","last_fullsized":"2024-02-13T00:41:03.990899+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.008320+0000","last_change":"2024-02-13T00:41:04.008320+0000","last_active":"2024-02-13T00:41:04.008320+0000","last_peered":"2024-02-13T00:41:04.008320+0000","last_clean":"2024-02-13T00:41:04.008320+0000","last_became_active":"2024-02-13T00:41:01.635032+0000","last_became_peered":"2024-02-13T00:41:01.635032+0000","last_unstale":"2024-02-13T00:41:04.008320+0000","last_undegraded":"2024-02-13T00:41:04.008320+0000","last_fullsized":"2024-02-13T00:41:04.008320+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.1b","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003560+0000","last_change":"2024-02-13T00:41:01.595117+0000","last_active":"2024-02-13T00:41:04.003560+0000","last_peered":"2024-02-13T00:41:04.003560+0000","last_clean":"2024-02-13T00:41:04.003560+0000","last_became_active":"2024-02-13T00:41:01.593744+0000","last_became_peered":"2024-02-13T00:41:01.593744+0000","last_unstale":"2024-02-13T00:41:04.003560+0000","last_undegraded":"2024-02-13T00:41:04.003560+0000","last_fullsized":"2024-02-13T00:41:04.003560+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996535+0000","last_change":"2024-02-13T00:41:01.546109+0000","last_active":"2024-02-13T00:41:03.996535+0000","last_peered":"2024-02-13T00:41:03.996535+0000","last_clean":"2024-02-13T00:41:03.996535+0000","last_became_active":"2024-02-13T00:41:01.545746+0000","last_became_peered":"2024-02-13T00:41:01.545746+0000","last_unstale":"2024-02-13T00:41:03.996535+0000","last_undegraded":"2024-02-13T00:41:03.996535+0000","last_fullsized":"2024-02-13T00:41:03.996535+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":121,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.990823+0000","last_change":"2024-02-13T00:40:04.185200+0000","last_active":"2024-02-13T00:41:03.990823+0000","last_peered":"2024-02-13T00:41:03.990823+0000","last_clean":"2024-02-13T00:41:03.990823+0000","last_became_active":"2024-02-13T00:37:43.324313+0000","last_became_peered":"2024-02-13T00:37:43.324313+0000","last_unstale":"2024-02-13T00:41:03.990823+0000","last_undegraded":"2024-02-13T00:41:03.990823+0000","last_fullsized":"2024-02-13T00:41:03.990823+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:40:04.185125+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:40:04.185125+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.989857+0000","last_change":"2024-02-13T00:40:52.503871+0000","last_active":"2024-02-13T00:41:03.989857+0000","last_peered":"2024-02-13T00:41:03.989857+0000","last_clean":"2024-02-13T00:41:03.989857+0000","last_became_active":"2024-02-13T00:37:41.219309+0000","last_became_peered":"2024-02-13T00:37:41.219309+0000","last_unstale":"2024-02-13T00:41:03.989857+0000","last_undegraded":"2024-02-13T00:41:03.989857+0000","last_fullsized":"2024-02-13T00:41:03.989857+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:52.503804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:52.503804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"64'8","reported_seq":111,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.249796+0000","last_change":"2024-02-13T00:39:11.518468+0000","last_active":"2024-02-13T00:41:04.249796+0000","last_peered":"2024-02-13T00:41:04.249796+0000","last_clean":"2024-02-13T00:41:04.249796+0000","last_became_active":"2024-02-13T00:37:45.246757+0000","last_became_peered":"2024-02-13T00:37:45.246757+0000","last_unstale":"2024-02-13T00:41:04.249796+0000","last_undegraded":"2024-02-13T00:41:04.249796+0000","last_fullsized":"2024-02-13T00:41:04.249796+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:11.518359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:11.518359+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":60,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.010354+0000","last_change":"2024-02-13T00:39:21.344836+0000","last_active":"2024-02-13T00:41:04.010354+0000","last_peered":"2024-02-13T00:41:04.010354+0000","last_clean":"2024-02-13T00:41:04.010354+0000","last_became_active":"2024-02-13T00:37:47.252722+0000","last_became_peered":"2024-02-13T00:37:47.252722+0000","last_unstale":"2024-02-13T00:41:04.010354+0000","last_undegraded":"2024-02-13T00:41:04.010354+0000","last_fullsized":"2024-02-13T00:41:04.010354+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.344771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.344771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993274+0000","last_change":"2024-02-13T00:41:01.582310+0000","last_active":"2024-02-13T00:41:03.993274+0000","last_peered":"2024-02-13T00:41:03.993274+0000","last_clean":"2024-02-13T00:41:03.993274+0000","last_became_active":"2024-02-13T00:41:01.581659+0000","last_became_peered":"2024-02-13T00:41:01.581659+0000","last_unstale":"2024-02-13T00:41:03.993274+0000","last_undegraded":"2024-02-13T00:41:03.993274+0000","last_fullsized":"2024-02-13T00:41:03.993274+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.4","version":"107'1","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.022745+0000","last_change":"2024-02-13T00:41:01.717562+0000","last_active":"2024-02-13T00:41:04.022745+0000","last_peered":"2024-02-13T00:41:04.022745+0000","last_clean":"2024-02-13T00:41:04.022745+0000","last_became_active":"2024-02-13T00:41:01.716123+0000","last_became_peered":"2024-02-13T00:41:01.716123+0000","last_unstale":"2024-02-13T00:41:04.022745+0000","last_undegraded":"2024-02-13T00:41:04.022745+0000","last_fullsized":"2024-02-13T00:41:04.022745+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995414+0000","last_change":"2024-02-13T00:41:03.995414+0000","last_active":"2024-02-13T00:41:03.995414+0000","last_peered":"2024-02-13T00:41:03.995414+0000","last_clean":"2024-02-13T00:41:03.995414+0000","last_became_active":"2024-02-13T00:41:01.559746+0000","last_became_peered":"2024-02-13T00:41:01.559746+0000","last_unstale":"2024-02-13T00:41:03.995414+0000","last_undegraded":"2024-02-13T00:41:03.995414+0000","last_fullsized":"2024-02-13T00:41:03.995414+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.002842+0000","last_change":"2024-02-13T00:41:01.559598+0000","last_active":"2024-02-13T00:41:04.002842+0000","last_peered":"2024-02-13T00:41:04.002842+0000","last_clean":"2024-02-13T00:41:04.002842+0000","last_became_active":"2024-02-13T00:41:01.557578+0000","last_became_peered":"2024-02-13T00:41:01.557578+0000","last_unstale":"2024-02-13T00:41:04.002842+0000","last_undegraded":"2024-02-13T00:41:04.002842+0000","last_fullsized":"2024-02-13T00:41:04.002842+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.001257+0000","last_change":"2024-02-13T00:41:04.001257+0000","last_active":"2024-02-13T00:41:04.001257+0000","last_peered":"2024-02-13T00:41:04.001257+0000","last_clean":"2024-02-13T00:41:04.001257+0000","last_became_active":"2024-02-13T00:41:01.579257+0000","last_became_peered":"2024-02-13T00:41:01.579257+0000","last_unstale":"2024-02-13T00:41:04.001257+0000","last_undegraded":"2024-02-13T00:41:04.001257+0000","last_fullsized":"2024-02-13T00:41:04.001257+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.1a","version":"107'3","reported_seq":19,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.989997+0000","last_change":"2024-02-13T00:41:01.656619+0000","last_active":"2024-02-13T00:41:03.989997+0000","last_peered":"2024-02-13T00:41:03.989997+0000","last_clean":"2024-02-13T00:41:03.989997+0000","last_became_active":"2024-02-13T00:41:01.656303+0000","last_became_peered":"2024-02-13T00:41:01.656303+0000","last_unstale":"2024-02-13T00:41:03.989997+0000","last_undegraded":"2024-02-13T00:41:03.989997+0000","last_fullsized":"2024-02-13T00:41:03.989997+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996419+0000","last_change":"2024-02-13T00:41:00.540367+0000","last_active":"2024-02-13T00:41:03.996419+0000","last_peered":"2024-02-13T00:41:03.996419+0000","last_clean":"2024-02-13T00:41:03.996419+0000","last_became_active":"2024-02-13T00:41:00.540138+0000","last_became_peered":"2024-02-13T00:41:00.540138+0000","last_unstale":"2024-02-13T00:41:03.996419+0000","last_undegraded":"2024-02-13T00:41:03.996419+0000","last_fullsized":"2024-02-13T00:41:03.996419+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":96,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003691+0000","last_change":"2024-02-13T00:39:10.549128+0000","last_active":"2024-02-13T00:41:04.003691+0000","last_peered":"2024-02-13T00:41:04.003691+0000","last_clean":"2024-02-13T00:41:04.003691+0000","last_became_active":"2024-02-13T00:37:43.322357+0000","last_became_peered":"2024-02-13T00:37:43.322357+0000","last_unstale":"2024-02-13T00:41:04.003691+0000","last_undegraded":"2024-02-13T00:41:04.003691+0000","last_fullsized":"2024-02-13T00:41:04.003691+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:39:10.549045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.549045+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.011226+0000","last_change":"2024-02-13T00:40:52.801522+0000","last_active":"2024-02-13T00:41:04.011226+0000","last_peered":"2024-02-13T00:41:04.011226+0000","last_clean":"2024-02-13T00:41:04.011226+0000","last_became_active":"2024-02-13T00:37:41.217953+0000","last_became_peered":"2024-02-13T00:37:41.217953+0000","last_unstale":"2024-02-13T00:41:04.011226+0000","last_undegraded":"2024-02-13T00:41:04.011226+0000","last_fullsized":"2024-02-13T00:41:04.011226+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:52.801455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:52.801455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"84'9","reported_seq":115,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.250058+0000","last_change":"2024-02-13T00:39:22.269192+0000","last_active":"2024-02-13T00:41:04.250058+0000","last_peered":"2024-02-13T00:41:04.250058+0000","last_clean":"2024-02-13T00:41:04.250058+0000","last_became_active":"2024-02-13T00:37:45.250598+0000","last_became_peered":"2024-02-13T00:37:45.250598+0000","last_unstale":"2024-02-13T00:41:04.250058+0000","last_undegraded":"2024-02-13T00:41:04.250058+0000","last_fullsized":"2024-02-13T00:41:04.250058+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.269104+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":31,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.030208+0000","last_change":"2024-02-13T00:40:41.387258+0000","last_active":"2024-02-13T00:41:04.030208+0000","last_peered":"2024-02-13T00:41:04.030208+0000","last_clean":"2024-02-13T00:41:04.030208+0000","last_became_active":"2024-02-13T00:40:35.850176+0000","last_became_peered":"2024-02-13T00:40:35.850176+0000","last_unstale":"2024-02-13T00:41:04.030208+0000","last_undegraded":"2024-02-13T00:41:04.030208+0000","last_fullsized":"2024-02-13T00:41:04.030208+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:41.387157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:41.387157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.3","version":"108'5","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.035168+0000","last_change":"2024-02-13T00:41:01.705740+0000","last_active":"2024-02-13T00:41:04.035168+0000","last_peered":"2024-02-13T00:41:04.035168+0000","last_clean":"2024-02-13T00:41:04.035168+0000","last_became_active":"2024-02-13T00:41:01.705297+0000","last_became_peered":"2024-02-13T00:41:01.705297+0000","last_unstale":"2024-02-13T00:41:04.035168+0000","last_undegraded":"2024-02-13T00:41:04.035168+0000","last_fullsized":"2024-02-13T00:41:04.035168+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.5","version":"109'1","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.566986+0000","last_change":"2024-02-13T00:41:01.734234+0000","last_active":"2024-02-13T00:41:04.566986+0000","last_peered":"2024-02-13T00:41:04.566986+0000","last_clean":"2024-02-13T00:41:04.566986+0000","last_became_active":"2024-02-13T00:41:01.722886+0000","last_became_peered":"2024-02-13T00:41:01.722886+0000","last_unstale":"2024-02-13T00:41:04.566986+0000","last_undegraded":"2024-02-13T00:41:04.566986+0000","last_fullsized":"2024-02-13T00:41:04.566986+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.9","version":"109'1","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.054448+0000","last_change":"2024-02-13T00:41:04.037790+0000","last_active":"2024-02-13T00:41:04.054448+0000","last_peered":"2024-02-13T00:41:04.054448+0000","last_clean":"2024-02-13T00:41:04.054448+0000","last_became_active":"2024-02-13T00:41:01.719134+0000","last_became_peered":"2024-02-13T00:41:01.719134+0000","last_unstale":"2024-02-13T00:41:04.054448+0000","last_undegraded":"2024-02-13T00:41:04.054448+0000","last_fullsized":"2024-02-13T00:41:04.054448+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.991851+0000","last_change":"2024-02-13T00:41:01.672359+0000","last_active":"2024-02-13T00:41:03.991851+0000","last_peered":"2024-02-13T00:41:03.991851+0000","last_clean":"2024-02-13T00:41:03.991851+0000","last_became_active":"2024-02-13T00:41:01.672068+0000","last_became_peered":"2024-02-13T00:41:01.672068+0000","last_unstale":"2024-02-13T00:41:03.991851+0000","last_undegraded":"2024-02-13T00:41:03.991851+0000","last_fullsized":"2024-02-13T00:41:03.991851+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.019679+0000","last_change":"2024-02-13T00:41:04.019679+0000","last_active":"2024-02-13T00:41:04.019679+0000","last_peered":"2024-02-13T00:41:04.019679+0000","last_clean":"2024-02-13T00:41:04.019679+0000","last_became_active":"2024-02-13T00:41:01.577436+0000","last_became_peered":"2024-02-13T00:41:01.577436+0000","last_unstale":"2024-02-13T00:41:04.019679+0000","last_undegraded":"2024-02-13T00:41:04.019679+0000","last_fullsized":"2024-02-13T00:41:04.019679+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.1d","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.001792+0000","last_change":"2024-02-13T00:41:01.635993+0000","last_active":"2024-02-13T00:41:04.001792+0000","last_peered":"2024-02-13T00:41:04.001792+0000","last_clean":"2024-02-13T00:41:04.001792+0000","last_became_active":"2024-02-13T00:41:01.634524+0000","last_became_peered":"2024-02-13T00:41:01.634524+0000","last_unstale":"2024-02-13T00:41:04.001792+0000","last_undegraded":"2024-02-13T00:41:04.001792+0000","last_fullsized":"2024-02-13T00:41:04.001792+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.004443+0000","last_change":"2024-02-13T00:41:01.648502+0000","last_active":"2024-02-13T00:41:04.004443+0000","last_peered":"2024-02-13T00:41:04.004443+0000","last_clean":"2024-02-13T00:41:04.004443+0000","last_became_active":"2024-02-13T00:41:01.646327+0000","last_became_peered":"2024-02-13T00:41:01.646327+0000","last_unstale":"2024-02-13T00:41:04.004443+0000","last_undegraded":"2024-02-13T00:41:04.004443+0000","last_fullsized":"2024-02-13T00:41:04.004443+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":90,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.023643+0000","last_change":"2024-02-13T00:40:16.009588+0000","last_active":"2024-02-13T00:41:04.023643+0000","last_peered":"2024-02-13T00:41:04.023643+0000","last_clean":"2024-02-13T00:41:04.023643+0000","last_became_active":"2024-02-13T00:37:43.225347+0000","last_became_peered":"2024-02-13T00:37:43.225347+0000","last_unstale":"2024-02-13T00:41:04.023643+0000","last_undegraded":"2024-02-13T00:41:04.023643+0000","last_fullsized":"2024-02-13T00:41:04.023643+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:40:16.009518+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:40:16.009518+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.015706+0000","last_change":"2024-02-13T00:40:58.334820+0000","last_active":"2024-02-13T00:41:04.015706+0000","last_peered":"2024-02-13T00:41:04.015706+0000","last_clean":"2024-02-13T00:41:04.015706+0000","last_became_active":"2024-02-13T00:37:41.216390+0000","last_became_peered":"2024-02-13T00:37:41.216390+0000","last_unstale":"2024-02-13T00:41:04.015706+0000","last_undegraded":"2024-02-13T00:41:04.015706+0000","last_fullsized":"2024-02-13T00:41:04.015706+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:58.334705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.334705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":62,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996674+0000","last_change":"2024-02-13T00:39:29.229760+0000","last_active":"2024-02-13T00:41:03.996674+0000","last_peered":"2024-02-13T00:41:03.996674+0000","last_clean":"2024-02-13T00:41:03.996674+0000","last_became_active":"2024-02-13T00:37:45.243594+0000","last_became_peered":"2024-02-13T00:37:45.243594+0000","last_unstale":"2024-02-13T00:41:03.996674+0000","last_undegraded":"2024-02-13T00:41:03.996674+0000","last_fullsized":"2024-02-13T00:41:03.996674+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.229676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":60,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995732+0000","last_change":"2024-02-13T00:39:28.286635+0000","last_active":"2024-02-13T00:41:03.995732+0000","last_peered":"2024-02-13T00:41:03.995732+0000","last_clean":"2024-02-13T00:41:03.995732+0000","last_became_active":"2024-02-13T00:37:47.259792+0000","last_became_peered":"2024-02-13T00:37:47.259792+0000","last_unstale":"2024-02-13T00:41:03.995732+0000","last_undegraded":"2024-02-13T00:41:03.995732+0000","last_fullsized":"2024-02-13T00:41:03.995732+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.286574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.286574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.033952+0000","last_change":"2024-02-13T00:41:01.745030+0000","last_active":"2024-02-13T00:41:04.033952+0000","last_peered":"2024-02-13T00:41:04.033952+0000","last_clean":"2024-02-13T00:41:04.033952+0000","last_became_active":"2024-02-13T00:41:01.743858+0000","last_became_peered":"2024-02-13T00:41:01.743858+0000","last_unstale":"2024-02-13T00:41:04.033952+0000","last_undegraded":"2024-02-13T00:41:04.033952+0000","last_fullsized":"2024-02-13T00:41:04.033952+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.2","version":"107'2","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.992623+0000","last_change":"2024-02-13T00:41:01.631898+0000","last_active":"2024-02-13T00:41:03.992623+0000","last_peered":"2024-02-13T00:41:03.992623+0000","last_clean":"2024-02-13T00:41:03.992623+0000","last_became_active":"2024-02-13T00:41:01.629104+0000","last_became_peered":"2024-02-13T00:41:01.629104+0000","last_unstale":"2024-02-13T00:41:03.992623+0000","last_undegraded":"2024-02-13T00:41:03.992623+0000","last_fullsized":"2024-02-13T00:41:03.992623+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.028805+0000","last_change":"2024-02-13T00:41:04.028805+0000","last_active":"2024-02-13T00:41:04.028805+0000","last_peered":"2024-02-13T00:41:04.028805+0000","last_clean":"2024-02-13T00:41:04.028805+0000","last_became_active":"2024-02-13T00:41:01.722042+0000","last_became_peered":"2024-02-13T00:41:01.722042+0000","last_unstale":"2024-02-13T00:41:04.028805+0000","last_undegraded":"2024-02-13T00:41:04.028805+0000","last_fullsized":"2024-02-13T00:41:04.028805+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.012037+0000","last_change":"2024-02-13T00:41:01.578270+0000","last_active":"2024-02-13T00:41:04.012037+0000","last_peered":"2024-02-13T00:41:04.012037+0000","last_clean":"2024-02-13T00:41:04.012037+0000","last_became_active":"2024-02-13T00:41:01.577649+0000","last_became_peered":"2024-02-13T00:41:01.577649+0000","last_unstale":"2024-02-13T00:41:04.012037+0000","last_undegraded":"2024-02-13T00:41:04.012037+0000","last_fullsized":"2024-02-13T00:41:04.012037+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.021680+0000","last_change":"2024-02-13T00:41:04.021680+0000","last_active":"2024-02-13T00:41:04.021680+0000","last_peered":"2024-02-13T00:41:04.021680+0000","last_clean":"2024-02-13T00:41:04.021680+0000","last_became_active":"2024-02-13T00:41:01.666359+0000","last_became_peered":"2024-02-13T00:41:01.666359+0000","last_unstale":"2024-02-13T00:41:04.021680+0000","last_undegraded":"2024-02-13T00:41:04.021680+0000","last_fullsized":"2024-02-13T00:41:04.021680+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.1c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.994136+0000","last_change":"2024-02-13T00:41:01.614399+0000","last_active":"2024-02-13T00:41:03.994136+0000","last_peered":"2024-02-13T00:41:03.994136+0000","last_clean":"2024-02-13T00:41:03.994136+0000","last_became_active":"2024-02-13T00:41:01.614197+0000","last_became_peered":"2024-02-13T00:41:01.614197+0000","last_unstale":"2024-02-13T00:41:03.994136+0000","last_undegraded":"2024-02-13T00:41:03.994136+0000","last_fullsized":"2024-02-13T00:41:03.994136+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995230+0000","last_change":"2024-02-13T00:41:01.635770+0000","last_active":"2024-02-13T00:41:03.995230+0000","last_peered":"2024-02-13T00:41:03.995230+0000","last_clean":"2024-02-13T00:41:03.995230+0000","last_became_active":"2024-02-13T00:41:01.635459+0000","last_became_peered":"2024-02-13T00:41:01.635459+0000","last_unstale":"2024-02-13T00:41:03.995230+0000","last_undegraded":"2024-02-13T00:41:03.995230+0000","last_fullsized":"2024-02-13T00:41:03.995230+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":80,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.011952+0000","last_change":"2024-02-13T00:40:44.273570+0000","last_active":"2024-02-13T00:41:04.011952+0000","last_peered":"2024-02-13T00:41:04.011952+0000","last_clean":"2024-02-13T00:41:04.011952+0000","last_became_active":"2024-02-13T00:40:35.852956+0000","last_became_peered":"2024-02-13T00:40:35.852956+0000","last_unstale":"2024-02-13T00:41:04.011952+0000","last_undegraded":"2024-02-13T00:41:04.011952+0000","last_fullsized":"2024-02-13T00:41:04.011952+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:40:44.273511+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.273511+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.025865+0000","last_change":"2024-02-13T00:41:03.608458+0000","last_active":"2024-02-13T00:41:04.025865+0000","last_peered":"2024-02-13T00:41:04.025865+0000","last_clean":"2024-02-13T00:41:04.025865+0000","last_became_active":"2024-02-13T00:37:41.215531+0000","last_became_peered":"2024-02-13T00:37:41.215531+0000","last_unstale":"2024-02-13T00:41:04.025865+0000","last_undegraded":"2024-02-13T00:41:04.025865+0000","last_fullsized":"2024-02-13T00:41:04.025865+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:03.608379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:03.608379+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.023714+0000","last_change":"2024-02-13T00:40:49.666455+0000","last_active":"2024-02-13T00:41:04.023714+0000","last_peered":"2024-02-13T00:41:04.023714+0000","last_clean":"2024-02-13T00:41:04.023714+0000","last_became_active":"2024-02-13T00:37:45.247114+0000","last_became_peered":"2024-02-13T00:37:45.247114+0000","last_unstale":"2024-02-13T00:41:04.023714+0000","last_undegraded":"2024-02-13T00:41:04.023714+0000","last_fullsized":"2024-02-13T00:41:04.023714+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:49.666387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:40:49.666387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.987836+0000","last_change":"2024-02-13T00:40:05.137000+0000","last_active":"2024-02-13T00:41:03.987836+0000","last_peered":"2024-02-13T00:41:03.987836+0000","last_clean":"2024-02-13T00:41:03.987836+0000","last_became_active":"2024-02-13T00:37:47.254519+0000","last_became_peered":"2024-02-13T00:37:47.254519+0000","last_unstale":"2024-02-13T00:41:03.987836+0000","last_undegraded":"2024-02-13T00:41:03.987836+0000","last_fullsized":"2024-02-13T00:41:03.987836+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:05.136935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:05.136935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.025926+0000","last_change":"2024-02-13T00:41:01.722213+0000","last_active":"2024-02-13T00:41:04.025926+0000","last_peered":"2024-02-13T00:41:04.025926+0000","last_clean":"2024-02-13T00:41:04.025926+0000","last_became_active":"2024-02-13T00:41:01.721963+0000","last_became_peered":"2024-02-13T00:41:01.721963+0000","last_unstale":"2024-02-13T00:41:04.025926+0000","last_undegraded":"2024-02-13T00:41:04.025926+0000","last_fullsized":"2024-02-13T00:41:04.025926+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.012072+0000","last_change":"2024-02-13T00:41:01.599057+0000","last_active":"2024-02-13T00:41:04.012072+0000","last_peered":"2024-02-13T00:41:04.012072+0000","last_clean":"2024-02-13T00:41:04.012072+0000","last_became_active":"2024-02-13T00:41:01.598445+0000","last_became_peered":"2024-02-13T00:41:01.598445+0000","last_unstale":"2024-02-13T00:41:04.012072+0000","last_undegraded":"2024-02-13T00:41:04.012072+0000","last_fullsized":"2024-02-13T00:41:04.012072+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.037200+0000","last_change":"2024-02-13T00:41:04.037200+0000","last_active":"2024-02-13T00:41:04.037200+0000","last_peered":"2024-02-13T00:41:04.037200+0000","last_clean":"2024-02-13T00:41:04.037200+0000","last_became_active":"2024-02-13T00:41:01.523707+0000","last_became_peered":"2024-02-13T00:41:01.523707+0000","last_unstale":"2024-02-13T00:41:04.037200+0000","last_undegraded":"2024-02-13T00:41:04.037200+0000","last_fullsized":"2024-02-13T00:41:04.037200+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995628+0000","last_change":"2024-02-13T00:41:01.662653+0000","last_active":"2024-02-13T00:41:03.995628+0000","last_peered":"2024-02-13T00:41:03.995628+0000","last_clean":"2024-02-13T00:41:03.995628+0000","last_became_active":"2024-02-13T00:41:01.661372+0000","last_became_peered":"2024-02-13T00:41:01.661372+0000","last_unstale":"2024-02-13T00:41:03.995628+0000","last_undegraded":"2024-02-13T00:41:03.995628+0000","last_fullsized":"2024-02-13T00:41:03.995628+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.d","version":"109'1","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.047873+0000","last_change":"2024-02-13T00:41:04.012000+0000","last_active":"2024-02-13T00:41:04.047873+0000","last_peered":"2024-02-13T00:41:04.047873+0000","last_clean":"2024-02-13T00:41:04.047873+0000","last_became_active":"2024-02-13T00:41:01.654976+0000","last_became_peered":"2024-02-13T00:41:01.654976+0000","last_unstale":"2024-02-13T00:41:04.047873+0000","last_undegraded":"2024-02-13T00:41:04.047873+0000","last_fullsized":"2024-02-13T00:41:04.047873+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.1f","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.030386+0000","last_change":"2024-02-13T00:41:00.532602+0000","last_active":"2024-02-13T00:41:04.030386+0000","last_peered":"2024-02-13T00:41:04.030386+0000","last_clean":"2024-02-13T00:41:04.030386+0000","last_became_active":"2024-02-13T00:41:00.532332+0000","last_became_peered":"2024-02-13T00:41:00.532332+0000","last_unstale":"2024-02-13T00:41:04.030386+0000","last_undegraded":"2024-02-13T00:41:04.030386+0000","last_fullsized":"2024-02-13T00:41:04.030386+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.010144+0000","last_change":"2024-02-13T00:41:01.621505+0000","last_active":"2024-02-13T00:41:04.010144+0000","last_peered":"2024-02-13T00:41:04.010144+0000","last_clean":"2024-02-13T00:41:04.010144+0000","last_became_active":"2024-02-13T00:41:01.621080+0000","last_became_peered":"2024-02-13T00:41:01.621080+0000","last_unstale":"2024-02-13T00:41:04.010144+0000","last_undegraded":"2024-02-13T00:41:04.010144+0000","last_fullsized":"2024-02-13T00:41:04.010144+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":75,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.987878+0000","last_change":"2024-02-13T00:40:51.054856+0000","last_active":"2024-02-13T00:41:03.987878+0000","last_peered":"2024-02-13T00:41:03.987878+0000","last_clean":"2024-02-13T00:41:03.987878+0000","last_became_active":"2024-02-13T00:37:43.234395+0000","last_became_peered":"2024-02-13T00:37:43.234395+0000","last_unstale":"2024-02-13T00:41:03.987878+0000","last_undegraded":"2024-02-13T00:41:03.987878+0000","last_fullsized":"2024-02-13T00:41:03.987878+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:40:51.054728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.054728+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.015552+0000","last_change":"2024-02-13T00:40:10.635555+0000","last_active":"2024-02-13T00:41:04.015552+0000","last_peered":"2024-02-13T00:41:04.015552+0000","last_clean":"2024-02-13T00:41:04.015552+0000","last_became_active":"2024-02-13T00:37:41.213033+0000","last_became_peered":"2024-02-13T00:37:41.213033+0000","last_unstale":"2024-02-13T00:41:04.015552+0000","last_undegraded":"2024-02-13T00:41:04.015552+0000","last_fullsized":"2024-02-13T00:41:04.015552+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:10.635478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:10.635478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.007865+0000","last_change":"2024-02-13T00:39:57.978687+0000","last_active":"2024-02-13T00:41:04.007865+0000","last_peered":"2024-02-13T00:41:04.007865+0000","last_clean":"2024-02-13T00:41:04.007865+0000","last_became_active":"2024-02-13T00:37:45.249556+0000","last_became_peered":"2024-02-13T00:37:45.249556+0000","last_unstale":"2024-02-13T00:41:04.007865+0000","last_undegraded":"2024-02-13T00:41:04.007865+0000","last_fullsized":"2024-02-13T00:41:04.007865+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:57.978593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:57.978593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.992870+0000","last_change":"2024-02-13T00:40:41.220513+0000","last_active":"2024-02-13T00:41:03.992870+0000","last_peered":"2024-02-13T00:41:03.992870+0000","last_clean":"2024-02-13T00:41:03.992870+0000","last_became_active":"2024-02-13T00:37:47.254667+0000","last_became_peered":"2024-02-13T00:37:47.254667+0000","last_unstale":"2024-02-13T00:41:03.992870+0000","last_undegraded":"2024-02-13T00:41:03.992870+0000","last_fullsized":"2024-02-13T00:41:03.992870+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:41.220387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:41.220387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.6","version":"108'9","reported_seq":22,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.997662+0000","last_change":"2024-02-13T00:41:01.631258+0000","last_active":"2024-02-13T00:41:03.997662+0000","last_peered":"2024-02-13T00:41:03.997662+0000","last_clean":"2024-02-13T00:41:03.997662+0000","last_became_active":"2024-02-13T00:41:01.630983+0000","last_became_peered":"2024-02-13T00:41:01.630983+0000","last_unstale":"2024-02-13T00:41:03.997662+0000","last_undegraded":"2024-02-13T00:41:03.997662+0000","last_fullsized":"2024-02-13T00:41:03.997662+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.987955+0000","last_change":"2024-02-13T00:41:01.651099+0000","last_active":"2024-02-13T00:41:03.987955+0000","last_peered":"2024-02-13T00:41:03.987955+0000","last_clean":"2024-02-13T00:41:03.987955+0000","last_became_active":"2024-02-13T00:41:01.649656+0000","last_became_peered":"2024-02-13T00:41:01.649656+0000","last_unstale":"2024-02-13T00:41:03.987955+0000","last_undegraded":"2024-02-13T00:41:03.987955+0000","last_fullsized":"2024-02-13T00:41:03.987955+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.052405+0000","last_change":"2024-02-13T00:41:04.052405+0000","last_active":"2024-02-13T00:41:04.052405+0000","last_peered":"2024-02-13T00:41:04.052405+0000","last_clean":"2024-02-13T00:41:04.052405+0000","last_became_active":"2024-02-13T00:41:01.738950+0000","last_became_peered":"2024-02-13T00:41:01.738950+0000","last_unstale":"2024-02-13T00:41:04.052405+0000","last_undegraded":"2024-02-13T00:41:04.052405+0000","last_fullsized":"2024-02-13T00:41:04.052405+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.989893+0000","last_change":"2024-02-13T00:41:01.623348+0000","last_active":"2024-02-13T00:41:03.989893+0000","last_peered":"2024-02-13T00:41:03.989893+0000","last_clean":"2024-02-13T00:41:03.989893+0000","last_became_active":"2024-02-13T00:41:01.623088+0000","last_became_peered":"2024-02-13T00:41:01.623088+0000","last_unstale":"2024-02-13T00:41:03.989893+0000","last_undegraded":"2024-02-13T00:41:03.989893+0000","last_fullsized":"2024-02-13T00:41:03.989893+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.002785+0000","last_change":"2024-02-13T00:41:04.002785+0000","last_active":"2024-02-13T00:41:04.002785+0000","last_peered":"2024-02-13T00:41:04.002785+0000","last_clean":"2024-02-13T00:41:04.002785+0000","last_became_active":"2024-02-13T00:41:01.652355+0000","last_became_peered":"2024-02-13T00:41:01.652355+0000","last_unstale":"2024-02-13T00:41:04.002785+0000","last_undegraded":"2024-02-13T00:41:04.002785+0000","last_fullsized":"2024-02-13T00:41:04.002785+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.1e","version":"107'4","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995262+0000","last_change":"2024-02-13T00:41:01.576002+0000","last_active":"2024-02-13T00:41:03.995262+0000","last_peered":"2024-02-13T00:41:03.995262+0000","last_clean":"2024-02-13T00:41:03.995262+0000","last_became_active":"2024-02-13T00:41:01.575669+0000","last_became_peered":"2024-02-13T00:41:01.575669+0000","last_unstale":"2024-02-13T00:41:03.995262+0000","last_undegraded":"2024-02-13T00:41:03.995262+0000","last_fullsized":"2024-02-13T00:41:03.995262+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.994989+0000","last_change":"2024-02-13T00:41:01.682709+0000","last_active":"2024-02-13T00:41:03.994989+0000","last_peered":"2024-02-13T00:41:03.994989+0000","last_clean":"2024-02-13T00:41:03.994989+0000","last_became_active":"2024-02-13T00:41:01.682376+0000","last_became_peered":"2024-02-13T00:41:01.682376+0000","last_unstale":"2024-02-13T00:41:03.994989+0000","last_undegraded":"2024-02-13T00:41:03.994989+0000","last_fullsized":"2024-02-13T00:41:03.994989+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":98,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.988467+0000","last_change":"2024-02-13T00:39:34.204969+0000","last_active":"2024-02-13T00:41:03.988467+0000","last_peered":"2024-02-13T00:41:03.988467+0000","last_clean":"2024-02-13T00:41:03.988467+0000","last_became_active":"2024-02-13T00:37:43.324196+0000","last_became_peered":"2024-02-13T00:37:43.324196+0000","last_unstale":"2024-02-13T00:41:03.988467+0000","last_undegraded":"2024-02-13T00:41:03.988467+0000","last_fullsized":"2024-02-13T00:41:03.988467+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:34.204888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.204888+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.013682+0000","last_change":"2024-02-13T00:39:07.358630+0000","last_active":"2024-02-13T00:41:04.013682+0000","last_peered":"2024-02-13T00:41:04.013682+0000","last_clean":"2024-02-13T00:41:04.013682+0000","last_became_active":"2024-02-13T00:37:41.220344+0000","last_became_peered":"2024-02-13T00:37:41.220344+0000","last_unstale":"2024-02-13T00:41:04.013682+0000","last_undegraded":"2024-02-13T00:41:04.013682+0000","last_fullsized":"2024-02-13T00:41:04.013682+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.358534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.358534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993967+0000","last_change":"2024-02-13T00:40:59.164776+0000","last_active":"2024-02-13T00:41:03.993967+0000","last_peered":"2024-02-13T00:41:03.993967+0000","last_clean":"2024-02-13T00:41:03.993967+0000","last_became_active":"2024-02-13T00:37:45.248592+0000","last_became_peered":"2024-02-13T00:37:45.248592+0000","last_unstale":"2024-02-13T00:41:03.993967+0000","last_undegraded":"2024-02-13T00:41:03.993967+0000","last_fullsized":"2024-02-13T00:41:03.993967+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.164690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.164690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.000301+0000","last_change":"2024-02-13T00:40:59.881327+0000","last_active":"2024-02-13T00:41:04.000301+0000","last_peered":"2024-02-13T00:41:04.000301+0000","last_clean":"2024-02-13T00:41:04.000301+0000","last_became_active":"2024-02-13T00:37:47.552917+0000","last_became_peered":"2024-02-13T00:37:47.552917+0000","last_unstale":"2024-02-13T00:41:04.000301+0000","last_undegraded":"2024-02-13T00:41:04.000301+0000","last_fullsized":"2024-02-13T00:41:04.000301+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.881255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.881255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.030306+0000","last_change":"2024-02-13T00:41:01.595241+0000","last_active":"2024-02-13T00:41:04.030306+0000","last_peered":"2024-02-13T00:41:04.030306+0000","last_clean":"2024-02-13T00:41:04.030306+0000","last_became_active":"2024-02-13T00:41:01.581338+0000","last_became_peered":"2024-02-13T00:41:01.581338+0000","last_unstale":"2024-02-13T00:41:04.030306+0000","last_undegraded":"2024-02-13T00:41:04.030306+0000","last_fullsized":"2024-02-13T00:41:04.030306+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"107'1","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.008308+0000","last_change":"2024-02-13T00:41:01.591267+0000","last_active":"2024-02-13T00:41:04.008308+0000","last_peered":"2024-02-13T00:41:04.008308+0000","last_clean":"2024-02-13T00:41:04.008308+0000","last_became_active":"2024-02-13T00:41:01.590927+0000","last_became_peered":"2024-02-13T00:41:01.590927+0000","last_unstale":"2024-02-13T00:41:04.008308+0000","last_undegraded":"2024-02-13T00:41:04.008308+0000","last_fullsized":"2024-02-13T00:41:04.008308+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.023529+0000","last_change":"2024-02-13T00:41:04.023529+0000","last_active":"2024-02-13T00:41:04.023529+0000","last_peered":"2024-02-13T00:41:04.023529+0000","last_clean":"2024-02-13T00:41:04.023529+0000","last_became_active":"2024-02-13T00:41:01.576991+0000","last_became_peered":"2024-02-13T00:41:01.576991+0000","last_unstale":"2024-02-13T00:41:04.023529+0000","last_undegraded":"2024-02-13T00:41:04.023529+0000","last_fullsized":"2024-02-13T00:41:04.023529+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.997252+0000","last_change":"2024-02-13T00:41:01.569377+0000","last_active":"2024-02-13T00:41:03.997252+0000","last_peered":"2024-02-13T00:41:03.997252+0000","last_clean":"2024-02-13T00:41:03.997252+0000","last_became_active":"2024-02-13T00:41:01.565724+0000","last_became_peered":"2024-02-13T00:41:01.565724+0000","last_unstale":"2024-02-13T00:41:03.997252+0000","last_undegraded":"2024-02-13T00:41:03.997252+0000","last_fullsized":"2024-02-13T00:41:03.997252+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.016366+0000","last_change":"2024-02-13T00:41:04.016366+0000","last_active":"2024-02-13T00:41:04.016366+0000","last_peered":"2024-02-13T00:41:04.016366+0000","last_clean":"2024-02-13T00:41:04.016366+0000","last_became_active":"2024-02-13T00:41:01.681510+0000","last_became_peered":"2024-02-13T00:41:01.681510+0000","last_unstale":"2024-02-13T00:41:04.016366+0000","last_undegraded":"2024-02-13T00:41:04.016366+0000","last_fullsized":"2024-02-13T00:41:04.016366+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996610+0000","last_change":"2024-02-13T00:41:00.530571+0000","last_active":"2024-02-13T00:41:03.996610+0000","last_peered":"2024-02-13T00:41:03.996610+0000","last_clean":"2024-02-13T00:41:03.996610+0000","last_became_active":"2024-02-13T00:41:00.530300+0000","last_became_peered":"2024-02-13T00:41:00.530300+0000","last_unstale":"2024-02-13T00:41:03.996610+0000","last_undegraded":"2024-02-13T00:41:03.996610+0000","last_fullsized":"2024-02-13T00:41:03.996610+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993748+0000","last_change":"2024-02-13T00:41:01.602950+0000","last_active":"2024-02-13T00:41:03.993748+0000","last_peered":"2024-02-13T00:41:03.993748+0000","last_clean":"2024-02-13T00:41:03.993748+0000","last_became_active":"2024-02-13T00:41:01.600179+0000","last_became_peered":"2024-02-13T00:41:01.600179+0000","last_unstale":"2024-02-13T00:41:03.993748+0000","last_undegraded":"2024-02-13T00:41:03.993748+0000","last_fullsized":"2024-02-13T00:41:03.993748+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":105,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.010837+0000","last_change":"2024-02-13T00:40:06.169154+0000","last_active":"2024-02-13T00:41:04.010837+0000","last_peered":"2024-02-13T00:41:04.010837+0000","last_clean":"2024-02-13T00:41:04.010837+0000","last_became_active":"2024-02-13T00:37:43.324221+0000","last_became_peered":"2024-02-13T00:37:43.324221+0000","last_unstale":"2024-02-13T00:41:04.010837+0000","last_undegraded":"2024-02-13T00:41:04.010837+0000","last_fullsized":"2024-02-13T00:41:04.010837+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:40:06.169087+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:40:06.169087+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.994217+0000","last_change":"2024-02-13T00:38:49.606456+0000","last_active":"2024-02-13T00:41:03.994217+0000","last_peered":"2024-02-13T00:41:03.994217+0000","last_clean":"2024-02-13T00:41:03.994217+0000","last_became_active":"2024-02-13T00:37:41.214965+0000","last_became_peered":"2024-02-13T00:37:41.214965+0000","last_unstale":"2024-02-13T00:41:03.994217+0000","last_undegraded":"2024-02-13T00:41:03.994217+0000","last_fullsized":"2024-02-13T00:41:03.994217+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:49.606357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:38:49.606357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"64'8","reported_seq":120,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.249811+0000","last_change":"2024-02-13T00:41:02.323171+0000","last_active":"2024-02-13T00:41:04.249811+0000","last_peered":"2024-02-13T00:41:04.249811+0000","last_clean":"2024-02-13T00:41:04.249811+0000","last_became_active":"2024-02-13T00:37:45.249902+0000","last_became_peered":"2024-02-13T00:37:45.249902+0000","last_unstale":"2024-02-13T00:41:04.249811+0000","last_undegraded":"2024-02-13T00:41:04.249811+0000","last_fullsized":"2024-02-13T00:41:04.249811+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:02.323101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.323101+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996364+0000","last_change":"2024-02-13T00:40:02.168591+0000","last_active":"2024-02-13T00:41:03.996364+0000","last_peered":"2024-02-13T00:41:03.996364+0000","last_clean":"2024-02-13T00:41:03.996364+0000","last_became_active":"2024-02-13T00:37:47.254586+0000","last_became_peered":"2024-02-13T00:37:47.254586+0000","last_unstale":"2024-02-13T00:41:03.996364+0000","last_undegraded":"2024-02-13T00:41:03.996364+0000","last_fullsized":"2024-02-13T00:41:03.996364+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:40:02.168499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:02.168499+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"107'2","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.998255+0000","last_change":"2024-02-13T00:41:01.629145+0000","last_active":"2024-02-13T00:41:03.998255+0000","last_peered":"2024-02-13T00:41:03.998255+0000","last_clean":"2024-02-13T00:41:03.998255+0000","last_became_active":"2024-02-13T00:41:01.628231+0000","last_became_peered":"2024-02-13T00:41:01.628231+0000","last_unstale":"2024-02-13T00:41:03.998255+0000","last_undegraded":"2024-02-13T00:41:03.998255+0000","last_fullsized":"2024-02-13T00:41:03.998255+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.000235+0000","last_change":"2024-02-13T00:41:01.741477+0000","last_active":"2024-02-13T00:41:04.000235+0000","last_peered":"2024-02-13T00:41:04.000235+0000","last_clean":"2024-02-13T00:41:04.000235+0000","last_became_active":"2024-02-13T00:41:01.740950+0000","last_became_peered":"2024-02-13T00:41:01.740950+0000","last_unstale":"2024-02-13T00:41:04.000235+0000","last_undegraded":"2024-02-13T00:41:04.000235+0000","last_fullsized":"2024-02-13T00:41:04.000235+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.992437+0000","last_change":"2024-02-13T00:41:03.992437+0000","last_active":"2024-02-13T00:41:03.992437+0000","last_peered":"2024-02-13T00:41:03.992437+0000","last_clean":"2024-02-13T00:41:03.992437+0000","last_became_active":"2024-02-13T00:41:01.581724+0000","last_became_peered":"2024-02-13T00:41:01.581724+0000","last_unstale":"2024-02-13T00:41:03.992437+0000","last_undegraded":"2024-02-13T00:41:03.992437+0000","last_fullsized":"2024-02-13T00:41:03.992437+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.7","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.020980+0000","last_change":"2024-02-13T00:41:01.707939+0000","last_active":"2024-02-13T00:41:04.020980+0000","last_peered":"2024-02-13T00:41:04.020980+0000","last_clean":"2024-02-13T00:41:04.020980+0000","last_became_active":"2024-02-13T00:41:01.707651+0000","last_became_peered":"2024-02-13T00:41:01.707651+0000","last_unstale":"2024-02-13T00:41:04.020980+0000","last_undegraded":"2024-02-13T00:41:04.020980+0000","last_fullsized":"2024-02-13T00:41:04.020980+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996170+0000","last_change":"2024-02-13T00:41:03.996170+0000","last_active":"2024-02-13T00:41:03.996170+0000","last_peered":"2024-02-13T00:41:03.996170+0000","last_clean":"2024-02-13T00:41:03.996170+0000","last_became_active":"2024-02-13T00:41:01.682621+0000","last_became_peered":"2024-02-13T00:41:01.682621+0000","last_unstale":"2024-02-13T00:41:03.996170+0000","last_undegraded":"2024-02-13T00:41:03.996170+0000","last_fullsized":"2024-02-13T00:41:03.996170+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.10","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996149+0000","last_change":"2024-02-13T00:41:01.630758+0000","last_active":"2024-02-13T00:41:03.996149+0000","last_peered":"2024-02-13T00:41:03.996149+0000","last_clean":"2024-02-13T00:41:03.996149+0000","last_became_active":"2024-02-13T00:41:01.630491+0000","last_became_peered":"2024-02-13T00:41:01.630491+0000","last_unstale":"2024-02-13T00:41:03.996149+0000","last_undegraded":"2024-02-13T00:41:03.996149+0000","last_fullsized":"2024-02-13T00:41:03.996149+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.988355+0000","last_change":"2024-02-13T00:41:01.572705+0000","last_active":"2024-02-13T00:41:03.988355+0000","last_peered":"2024-02-13T00:41:03.988355+0000","last_clean":"2024-02-13T00:41:03.988355+0000","last_became_active":"2024-02-13T00:41:01.572216+0000","last_became_peered":"2024-02-13T00:41:01.572216+0000","last_unstale":"2024-02-13T00:41:03.988355+0000","last_undegraded":"2024-02-13T00:41:03.988355+0000","last_fullsized":"2024-02-13T00:41:03.988355+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":104,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.014208+0000","last_change":"2024-02-13T00:40:42.231057+0000","last_active":"2024-02-13T00:41:04.014208+0000","last_peered":"2024-02-13T00:41:04.014208+0000","last_clean":"2024-02-13T00:41:04.014208+0000","last_became_active":"2024-02-13T00:37:43.230491+0000","last_became_peered":"2024-02-13T00:37:43.230491+0000","last_unstale":"2024-02-13T00:41:04.014208+0000","last_undegraded":"2024-02-13T00:41:04.014208+0000","last_fullsized":"2024-02-13T00:41:04.014208+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:40:42.230976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:42.230976+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.987970+0000","last_change":"2024-02-13T00:39:41.231336+0000","last_active":"2024-02-13T00:41:03.987970+0000","last_peered":"2024-02-13T00:41:03.987970+0000","last_clean":"2024-02-13T00:41:03.987970+0000","last_became_active":"2024-02-13T00:37:41.216574+0000","last_became_peered":"2024-02-13T00:37:41.216574+0000","last_unstale":"2024-02-13T00:41:03.987970+0000","last_undegraded":"2024-02-13T00:41:03.987970+0000","last_fullsized":"2024-02-13T00:41:03.987970+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:41.231244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:41.231244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":62,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.010918+0000","last_change":"2024-02-13T00:39:15.328037+0000","last_active":"2024-02-13T00:41:04.010918+0000","last_peered":"2024-02-13T00:41:04.010918+0000","last_clean":"2024-02-13T00:41:04.010918+0000","last_became_active":"2024-02-13T00:37:45.249064+0000","last_became_peered":"2024-02-13T00:37:45.249064+0000","last_unstale":"2024-02-13T00:41:04.010918+0000","last_undegraded":"2024-02-13T00:41:04.010918+0000","last_fullsized":"2024-02-13T00:41:04.010918+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.327971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.327971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.025664+0000","last_change":"2024-02-13T00:40:06.026382+0000","last_active":"2024-02-13T00:41:04.025664+0000","last_peered":"2024-02-13T00:41:04.025664+0000","last_clean":"2024-02-13T00:41:04.025664+0000","last_became_active":"2024-02-13T00:37:47.260182+0000","last_became_peered":"2024-02-13T00:37:47.260182+0000","last_unstale":"2024-02-13T00:41:04.025664+0000","last_undegraded":"2024-02-13T00:41:04.025664+0000","last_fullsized":"2024-02-13T00:41:04.025664+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:06.026318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:06.026318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.988050+0000","last_change":"2024-02-13T00:41:01.682551+0000","last_active":"2024-02-13T00:41:03.988050+0000","last_peered":"2024-02-13T00:41:03.988050+0000","last_clean":"2024-02-13T00:41:03.988050+0000","last_became_active":"2024-02-13T00:41:01.682261+0000","last_became_peered":"2024-02-13T00:41:01.682261+0000","last_unstale":"2024-02-13T00:41:03.988050+0000","last_undegraded":"2024-02-13T00:41:03.988050+0000","last_fullsized":"2024-02-13T00:41:03.988050+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.006637+0000","last_change":"2024-02-13T00:41:01.645310+0000","last_active":"2024-02-13T00:41:04.006637+0000","last_peered":"2024-02-13T00:41:04.006637+0000","last_clean":"2024-02-13T00:41:04.006637+0000","last_became_active":"2024-02-13T00:41:01.641767+0000","last_became_peered":"2024-02-13T00:41:01.641767+0000","last_unstale":"2024-02-13T00:41:04.006637+0000","last_undegraded":"2024-02-13T00:41:04.006637+0000","last_fullsized":"2024-02-13T00:41:04.006637+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.998330+0000","last_change":"2024-02-13T00:41:03.998330+0000","last_active":"2024-02-13T00:41:03.998330+0000","last_peered":"2024-02-13T00:41:03.998330+0000","last_clean":"2024-02-13T00:41:03.998330+0000","last_became_active":"2024-02-13T00:41:01.603393+0000","last_became_peered":"2024-02-13T00:41:01.603393+0000","last_unstale":"2024-02-13T00:41:03.998330+0000","last_undegraded":"2024-02-13T00:41:03.998330+0000","last_fullsized":"2024-02-13T00:41:03.998330+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.005178+0000","last_change":"2024-02-13T00:41:01.705201+0000","last_active":"2024-02-13T00:41:04.005178+0000","last_peered":"2024-02-13T00:41:04.005178+0000","last_clean":"2024-02-13T00:41:04.005178+0000","last_became_active":"2024-02-13T00:41:01.704705+0000","last_became_peered":"2024-02-13T00:41:01.704705+0000","last_unstale":"2024-02-13T00:41:04.005178+0000","last_undegraded":"2024-02-13T00:41:04.005178+0000","last_fullsized":"2024-02-13T00:41:04.005178+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.007716+0000","last_change":"2024-02-13T00:41:04.007716+0000","last_active":"2024-02-13T00:41:04.007716+0000","last_peered":"2024-02-13T00:41:04.007716+0000","last_clean":"2024-02-13T00:41:04.007716+0000","last_became_active":"2024-02-13T00:41:01.643345+0000","last_became_peered":"2024-02-13T00:41:01.643345+0000","last_unstale":"2024-02-13T00:41:04.007716+0000","last_undegraded":"2024-02-13T00:41:04.007716+0000","last_fullsized":"2024-02-13T00:41:04.007716+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.13","version":"107'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.999909+0000","last_change":"2024-02-13T00:41:01.583525+0000","last_active":"2024-02-13T00:41:03.999909+0000","last_peered":"2024-02-13T00:41:03.999909+0000","last_clean":"2024-02-13T00:41:03.999909+0000","last_became_active":"2024-02-13T00:41:01.583065+0000","last_became_peered":"2024-02-13T00:41:01.583065+0000","last_unstale":"2024-02-13T00:41:03.999909+0000","last_undegraded":"2024-02-13T00:41:03.999909+0000","last_fullsized":"2024-02-13T00:41:03.999909+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.015469+0000","last_change":"2024-02-13T00:41:01.615117+0000","last_active":"2024-02-13T00:41:04.015469+0000","last_peered":"2024-02-13T00:41:04.015469+0000","last_clean":"2024-02-13T00:41:04.015469+0000","last_became_active":"2024-02-13T00:41:01.614748+0000","last_became_peered":"2024-02-13T00:41:01.614748+0000","last_unstale":"2024-02-13T00:41:04.015469+0000","last_undegraded":"2024-02-13T00:41:04.015469+0000","last_fullsized":"2024-02-13T00:41:04.015469+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003210+0000","last_change":"2024-02-13T00:40:44.369174+0000","last_active":"2024-02-13T00:41:04.003210+0000","last_peered":"2024-02-13T00:41:04.003210+0000","last_clean":"2024-02-13T00:41:04.003210+0000","last_became_active":"2024-02-13T00:37:43.232938+0000","last_became_peered":"2024-02-13T00:37:43.232938+0000","last_unstale":"2024-02-13T00:41:04.003210+0000","last_undegraded":"2024-02-13T00:41:04.003210+0000","last_fullsized":"2024-02-13T00:41:04.003210+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:40:44.369070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.369070+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.016827+0000","last_change":"2024-02-13T00:40:57.316472+0000","last_active":"2024-02-13T00:41:04.016827+0000","last_peered":"2024-02-13T00:41:04.016827+0000","last_clean":"2024-02-13T00:41:04.016827+0000","last_became_active":"2024-02-13T00:37:41.218334+0000","last_became_peered":"2024-02-13T00:37:41.218334+0000","last_unstale":"2024-02-13T00:41:04.016827+0000","last_undegraded":"2024-02-13T00:41:04.016827+0000","last_fullsized":"2024-02-13T00:41:04.016827+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.316400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.316400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":63,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.026562+0000","last_change":"2024-02-13T00:39:30.245586+0000","last_active":"2024-02-13T00:41:04.026562+0000","last_peered":"2024-02-13T00:41:04.026562+0000","last_clean":"2024-02-13T00:41:04.026562+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:41:04.026562+0000","last_undegraded":"2024-02-13T00:41:04.026562+0000","last_fullsized":"2024-02-13T00:41:04.026562+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:30.245508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.245508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":26,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003037+0000","last_change":"2024-02-13T00:40:35.846828+0000","last_active":"2024-02-13T00:41:04.003037+0000","last_peered":"2024-02-13T00:41:04.003037+0000","last_clean":"2024-02-13T00:41:04.003037+0000","last_became_active":"2024-02-13T00:40:35.846506+0000","last_became_peered":"2024-02-13T00:40:35.846506+0000","last_unstale":"2024-02-13T00:41:04.003037+0000","last_undegraded":"2024-02-13T00:41:04.003037+0000","last_fullsized":"2024-02-13T00:41:04.003037+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:15.106889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:15.106889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.a","version":"108'10","reported_seq":23,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.004164+0000","last_change":"2024-02-13T00:41:01.724237+0000","last_active":"2024-02-13T00:41:04.004164+0000","last_peered":"2024-02-13T00:41:04.004164+0000","last_clean":"2024-02-13T00:41:04.004164+0000","last_became_active":"2024-02-13T00:41:01.723568+0000","last_became_peered":"2024-02-13T00:41:01.723568+0000","last_unstale":"2024-02-13T00:41:04.004164+0000","last_undegraded":"2024-02-13T00:41:04.004164+0000","last_fullsized":"2024-02-13T00:41:04.004164+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.991406+0000","last_change":"2024-02-13T00:41:01.549028+0000","last_active":"2024-02-13T00:41:03.991406+0000","last_peered":"2024-02-13T00:41:03.991406+0000","last_clean":"2024-02-13T00:41:03.991406+0000","last_became_active":"2024-02-13T00:41:01.547783+0000","last_became_peered":"2024-02-13T00:41:01.547783+0000","last_unstale":"2024-02-13T00:41:03.991406+0000","last_undegraded":"2024-02-13T00:41:03.991406+0000","last_fullsized":"2024-02-13T00:41:03.991406+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.031820+0000","last_change":"2024-02-13T00:41:04.031820+0000","last_active":"2024-02-13T00:41:04.031820+0000","last_peered":"2024-02-13T00:41:04.031820+0000","last_clean":"2024-02-13T00:41:04.031820+0000","last_became_active":"2024-02-13T00:41:01.712962+0000","last_became_peered":"2024-02-13T00:41:01.712962+0000","last_unstale":"2024-02-13T00:41:04.031820+0000","last_undegraded":"2024-02-13T00:41:04.031820+0000","last_fullsized":"2024-02-13T00:41:04.031820+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.008902+0000","last_change":"2024-02-13T00:41:01.600797+0000","last_active":"2024-02-13T00:41:04.008902+0000","last_peered":"2024-02-13T00:41:04.008902+0000","last_clean":"2024-02-13T00:41:04.008902+0000","last_became_active":"2024-02-13T00:41:01.583043+0000","last_became_peered":"2024-02-13T00:41:01.583043+0000","last_unstale":"2024-02-13T00:41:04.008902+0000","last_undegraded":"2024-02-13T00:41:04.008902+0000","last_fullsized":"2024-02-13T00:41:04.008902+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.011767+0000","last_change":"2024-02-13T00:41:04.011767+0000","last_active":"2024-02-13T00:41:04.011767+0000","last_peered":"2024-02-13T00:41:04.011767+0000","last_clean":"2024-02-13T00:41:04.011767+0000","last_became_active":"2024-02-13T00:41:01.638421+0000","last_became_peered":"2024-02-13T00:41:01.638421+0000","last_unstale":"2024-02-13T00:41:04.011767+0000","last_undegraded":"2024-02-13T00:41:04.011767+0000","last_fullsized":"2024-02-13T00:41:04.011767+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.993500+0000","last_change":"2024-02-13T00:41:01.559358+0000","last_active":"2024-02-13T00:41:03.993500+0000","last_peered":"2024-02-13T00:41:03.993500+0000","last_clean":"2024-02-13T00:41:03.993500+0000","last_became_active":"2024-02-13T00:41:01.558224+0000","last_became_peered":"2024-02-13T00:41:01.558224+0000","last_unstale":"2024-02-13T00:41:03.993500+0000","last_undegraded":"2024-02-13T00:41:03.993500+0000","last_fullsized":"2024-02-13T00:41:03.993500+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.017750+0000","last_change":"2024-02-13T00:41:01.710918+0000","last_active":"2024-02-13T00:41:04.017750+0000","last_peered":"2024-02-13T00:41:04.017750+0000","last_clean":"2024-02-13T00:41:04.017750+0000","last_became_active":"2024-02-13T00:41:01.710754+0000","last_became_peered":"2024-02-13T00:41:01.710754+0000","last_unstale":"2024-02-13T00:41:04.017750+0000","last_undegraded":"2024-02-13T00:41:04.017750+0000","last_fullsized":"2024-02-13T00:41:04.017750+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":102,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.998650+0000","last_change":"2024-02-13T00:39:26.214150+0000","last_active":"2024-02-13T00:41:03.998650+0000","last_peered":"2024-02-13T00:41:03.998650+0000","last_clean":"2024-02-13T00:41:03.998650+0000","last_became_active":"2024-02-13T00:37:43.236026+0000","last_became_peered":"2024-02-13T00:37:43.236026+0000","last_unstale":"2024-02-13T00:41:03.998650+0000","last_undegraded":"2024-02-13T00:41:03.998650+0000","last_fullsized":"2024-02-13T00:41:03.998650+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:39:26.214084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:26.214084+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996597+0000","last_change":"2024-02-13T00:40:49.110833+0000","last_active":"2024-02-13T00:41:03.996597+0000","last_peered":"2024-02-13T00:41:03.996597+0000","last_clean":"2024-02-13T00:41:03.996597+0000","last_became_active":"2024-02-13T00:37:41.218333+0000","last_became_peered":"2024-02-13T00:37:41.218333+0000","last_unstale":"2024-02-13T00:41:03.996597+0000","last_undegraded":"2024-02-13T00:41:03.996597+0000","last_fullsized":"2024-02-13T00:41:03.996597+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:49.110725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:49.110725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":31,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.991029+0000","last_change":"2024-02-13T00:40:52.067206+0000","last_active":"2024-02-13T00:41:03.991029+0000","last_peered":"2024-02-13T00:41:03.991029+0000","last_clean":"2024-02-13T00:41:03.991029+0000","last_became_active":"2024-02-13T00:40:35.858538+0000","last_became_peered":"2024-02-13T00:40:35.858538+0000","last_unstale":"2024-02-13T00:41:03.991029+0000","last_undegraded":"2024-02-13T00:41:03.991029+0000","last_fullsized":"2024-02-13T00:41:03.991029+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:52.067101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:40:52.067101+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996977+0000","last_change":"2024-02-13T00:40:57.950554+0000","last_active":"2024-02-13T00:41:03.996977+0000","last_peered":"2024-02-13T00:41:03.996977+0000","last_clean":"2024-02-13T00:41:03.996977+0000","last_became_active":"2024-02-13T00:37:47.553853+0000","last_became_peered":"2024-02-13T00:37:47.553853+0000","last_unstale":"2024-02-13T00:41:03.996977+0000","last_undegraded":"2024-02-13T00:41:03.996977+0000","last_fullsized":"2024-02-13T00:41:03.996977+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.950483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.950483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"107'2","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.998880+0000","last_change":"2024-02-13T00:41:01.611411+0000","last_active":"2024-02-13T00:41:03.998880+0000","last_peered":"2024-02-13T00:41:03.998880+0000","last_clean":"2024-02-13T00:41:03.998880+0000","last_became_active":"2024-02-13T00:41:01.611115+0000","last_became_peered":"2024-02-13T00:41:01.611115+0000","last_unstale":"2024-02-13T00:41:03.998880+0000","last_undegraded":"2024-02-13T00:41:03.998880+0000","last_fullsized":"2024-02-13T00:41:03.998880+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.d","version":"107'1","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.010103+0000","last_change":"2024-02-13T00:41:01.638974+0000","last_active":"2024-02-13T00:41:04.010103+0000","last_peered":"2024-02-13T00:41:04.010103+0000","last_clean":"2024-02-13T00:41:04.010103+0000","last_became_active":"2024-02-13T00:41:01.638148+0000","last_became_peered":"2024-02-13T00:41:01.638148+0000","last_unstale":"2024-02-13T00:41:04.010103+0000","last_undegraded":"2024-02-13T00:41:04.010103+0000","last_fullsized":"2024-02-13T00:41:04.010103+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.016738+0000","last_change":"2024-02-13T00:41:04.016738+0000","last_active":"2024-02-13T00:41:04.016738+0000","last_peered":"2024-02-13T00:41:04.016738+0000","last_clean":"2024-02-13T00:41:04.016738+0000","last_became_active":"2024-02-13T00:41:01.579740+0000","last_became_peered":"2024-02-13T00:41:01.579740+0000","last_unstale":"2024-02-13T00:41:04.016738+0000","last_undegraded":"2024-02-13T00:41:04.016738+0000","last_fullsized":"2024-02-13T00:41:04.016738+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.009641+0000","last_change":"2024-02-13T00:41:01.627093+0000","last_active":"2024-02-13T00:41:04.009641+0000","last_peered":"2024-02-13T00:41:04.009641+0000","last_clean":"2024-02-13T00:41:04.009641+0000","last_became_active":"2024-02-13T00:41:01.626019+0000","last_became_peered":"2024-02-13T00:41:01.626019+0000","last_unstale":"2024-02-13T00:41:04.009641+0000","last_undegraded":"2024-02-13T00:41:04.009641+0000","last_fullsized":"2024-02-13T00:41:04.009641+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.027028+0000","last_change":"2024-02-13T00:41:04.027028+0000","last_active":"2024-02-13T00:41:04.027028+0000","last_peered":"2024-02-13T00:41:04.027028+0000","last_clean":"2024-02-13T00:41:04.027028+0000","last_became_active":"2024-02-13T00:41:01.593274+0000","last_became_peered":"2024-02-13T00:41:01.593274+0000","last_unstale":"2024-02-13T00:41:04.027028+0000","last_undegraded":"2024-02-13T00:41:04.027028+0000","last_fullsized":"2024-02-13T00:41:04.027028+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.15","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.009794+0000","last_change":"2024-02-13T00:41:00.527862+0000","last_active":"2024-02-13T00:41:04.009794+0000","last_peered":"2024-02-13T00:41:04.009794+0000","last_clean":"2024-02-13T00:41:04.009794+0000","last_became_active":"2024-02-13T00:41:00.527574+0000","last_became_peered":"2024-02-13T00:41:00.527574+0000","last_unstale":"2024-02-13T00:41:04.009794+0000","last_undegraded":"2024-02-13T00:41:04.009794+0000","last_fullsized":"2024-02-13T00:41:04.009794+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.022468+0000","last_change":"2024-02-13T00:41:01.717177+0000","last_active":"2024-02-13T00:41:04.022468+0000","last_peered":"2024-02-13T00:41:04.022468+0000","last_clean":"2024-02-13T00:41:04.022468+0000","last_became_active":"2024-02-13T00:41:01.715728+0000","last_became_peered":"2024-02-13T00:41:01.715728+0000","last_unstale":"2024-02-13T00:41:04.022468+0000","last_undegraded":"2024-02-13T00:41:04.022468+0000","last_fullsized":"2024-02-13T00:41:04.022468+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":85,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003799+0000","last_change":"2024-02-13T00:40:35.857612+0000","last_active":"2024-02-13T00:41:04.003799+0000","last_peered":"2024-02-13T00:41:04.003799+0000","last_clean":"2024-02-13T00:41:04.003799+0000","last_became_active":"2024-02-13T00:40:35.857274+0000","last_became_peered":"2024-02-13T00:40:35.857274+0000","last_unstale":"2024-02-13T00:41:04.003799+0000","last_undegraded":"2024-02-13T00:41:04.003799+0000","last_fullsized":"2024-02-13T00:41:04.003799+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:39:34.213074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.213074+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995869+0000","last_change":"2024-02-13T00:40:43.197233+0000","last_active":"2024-02-13T00:41:03.995869+0000","last_peered":"2024-02-13T00:41:03.995869+0000","last_clean":"2024-02-13T00:41:03.995869+0000","last_became_active":"2024-02-13T00:37:41.222506+0000","last_became_peered":"2024-02-13T00:37:41.222506+0000","last_unstale":"2024-02-13T00:41:03.995869+0000","last_undegraded":"2024-02-13T00:41:03.995869+0000","last_fullsized":"2024-02-13T00:41:03.995869+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:43.197157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:43.197157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.999685+0000","last_change":"2024-02-13T00:40:14.697069+0000","last_active":"2024-02-13T00:41:03.999685+0000","last_peered":"2024-02-13T00:41:03.999685+0000","last_clean":"2024-02-13T00:41:03.999685+0000","last_became_active":"2024-02-13T00:37:45.243961+0000","last_became_peered":"2024-02-13T00:37:45.243961+0000","last_unstale":"2024-02-13T00:41:03.999685+0000","last_undegraded":"2024-02-13T00:41:03.999685+0000","last_fullsized":"2024-02-13T00:41:03.999685+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:14.696991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:14.696991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":60,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996355+0000","last_change":"2024-02-13T00:39:22.338139+0000","last_active":"2024-02-13T00:41:03.996355+0000","last_peered":"2024-02-13T00:41:03.996355+0000","last_clean":"2024-02-13T00:41:03.996355+0000","last_became_active":"2024-02-13T00:37:47.259688+0000","last_became_peered":"2024-02-13T00:37:47.259688+0000","last_unstale":"2024-02-13T00:41:03.996355+0000","last_undegraded":"2024-02-13T00:41:03.996355+0000","last_fullsized":"2024-02-13T00:41:03.996355+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.338042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.999926+0000","last_change":"2024-02-13T00:41:01.620536+0000","last_active":"2024-02-13T00:41:03.999926+0000","last_peered":"2024-02-13T00:41:03.999926+0000","last_clean":"2024-02-13T00:41:03.999926+0000","last_became_active":"2024-02-13T00:41:01.620258+0000","last_became_peered":"2024-02-13T00:41:01.620258+0000","last_unstale":"2024-02-13T00:41:03.999926+0000","last_undegraded":"2024-02-13T00:41:03.999926+0000","last_fullsized":"2024-02-13T00:41:03.999926+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.a","version":"107'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.992533+0000","last_change":"2024-02-13T00:41:01.632973+0000","last_active":"2024-02-13T00:41:03.992533+0000","last_peered":"2024-02-13T00:41:03.992533+0000","last_clean":"2024-02-13T00:41:03.992533+0000","last_became_active":"2024-02-13T00:41:01.629461+0000","last_became_peered":"2024-02-13T00:41:01.629461+0000","last_unstale":"2024-02-13T00:41:03.992533+0000","last_undegraded":"2024-02-13T00:41:03.992533+0000","last_fullsized":"2024-02-13T00:41:03.992533+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.022378+0000","last_change":"2024-02-13T00:41:04.022378+0000","last_active":"2024-02-13T00:41:04.022378+0000","last_peered":"2024-02-13T00:41:04.022378+0000","last_clean":"2024-02-13T00:41:04.022378+0000","last_became_active":"2024-02-13T00:41:01.746104+0000","last_became_peered":"2024-02-13T00:41:01.746104+0000","last_unstale":"2024-02-13T00:41:04.022378+0000","last_undegraded":"2024-02-13T00:41:04.022378+0000","last_fullsized":"2024-02-13T00:41:04.022378+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.000202+0000","last_change":"2024-02-13T00:41:01.575407+0000","last_active":"2024-02-13T00:41:04.000202+0000","last_peered":"2024-02-13T00:41:04.000202+0000","last_clean":"2024-02-13T00:41:04.000202+0000","last_became_active":"2024-02-13T00:41:01.575117+0000","last_became_peered":"2024-02-13T00:41:01.575117+0000","last_unstale":"2024-02-13T00:41:04.000202+0000","last_undegraded":"2024-02-13T00:41:04.000202+0000","last_fullsized":"2024-02-13T00:41:04.000202+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.033272+0000","last_change":"2024-02-13T00:41:04.033272+0000","last_active":"2024-02-13T00:41:04.033272+0000","last_peered":"2024-02-13T00:41:04.033272+0000","last_clean":"2024-02-13T00:41:04.033272+0000","last_became_active":"2024-02-13T00:41:01.638985+0000","last_became_peered":"2024-02-13T00:41:01.638985+0000","last_unstale":"2024-02-13T00:41:04.033272+0000","last_undegraded":"2024-02-13T00:41:04.033272+0000","last_fullsized":"2024-02-13T00:41:04.033272+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.14","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.004736+0000","last_change":"2024-02-13T00:41:01.648682+0000","last_active":"2024-02-13T00:41:04.004736+0000","last_peered":"2024-02-13T00:41:04.004736+0000","last_clean":"2024-02-13T00:41:04.004736+0000","last_became_active":"2024-02-13T00:41:01.646687+0000","last_became_peered":"2024-02-13T00:41:01.646687+0000","last_unstale":"2024-02-13T00:41:04.004736+0000","last_undegraded":"2024-02-13T00:41:04.004736+0000","last_fullsized":"2024-02-13T00:41:04.004736+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.988890+0000","last_change":"2024-02-13T00:41:00.547242+0000","last_active":"2024-02-13T00:41:03.988890+0000","last_peered":"2024-02-13T00:41:03.988890+0000","last_clean":"2024-02-13T00:41:03.988890+0000","last_became_active":"2024-02-13T00:41:00.546784+0000","last_became_peered":"2024-02-13T00:41:00.546784+0000","last_unstale":"2024-02-13T00:41:03.988890+0000","last_undegraded":"2024-02-13T00:41:03.988890+0000","last_fullsized":"2024-02-13T00:41:03.988890+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"85'14","reported_seq":86,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.996513+0000","last_change":"2024-02-13T00:40:35.857071+0000","last_active":"2024-02-13T00:41:03.996513+0000","last_peered":"2024-02-13T00:41:03.996513+0000","last_clean":"2024-02-13T00:41:03.996513+0000","last_became_active":"2024-02-13T00:40:35.856715+0000","last_became_peered":"2024-02-13T00:40:35.856715+0000","last_unstale":"2024-02-13T00:41:03.996513+0000","last_undegraded":"2024-02-13T00:41:03.996513+0000","last_fullsized":"2024-02-13T00:41:03.996513+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:39:42.156267+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:39:42.156267+0000","last_clean_scrub_stamp":"2024-02-13T00:39:42.156267+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.002761+0000","last_change":"2024-02-13T00:39:21.165313+0000","last_active":"2024-02-13T00:41:04.002761+0000","last_peered":"2024-02-13T00:41:04.002761+0000","last_clean":"2024-02-13T00:41:04.002761+0000","last_became_active":"2024-02-13T00:37:41.221025+0000","last_became_peered":"2024-02-13T00:37:41.221025+0000","last_unstale":"2024-02-13T00:41:04.002761+0000","last_undegraded":"2024-02-13T00:41:04.002761+0000","last_fullsized":"2024-02-13T00:41:04.002761+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.165246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.992954+0000","last_change":"2024-02-13T00:40:07.088136+0000","last_active":"2024-02-13T00:41:03.992954+0000","last_peered":"2024-02-13T00:41:03.992954+0000","last_clean":"2024-02-13T00:41:03.992954+0000","last_became_active":"2024-02-13T00:37:45.246184+0000","last_became_peered":"2024-02-13T00:37:45.246184+0000","last_unstale":"2024-02-13T00:41:03.992954+0000","last_undegraded":"2024-02-13T00:41:03.992954+0000","last_fullsized":"2024-02-13T00:41:03.992954+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:07.088053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:07.088053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.989598+0000","last_change":"2024-02-13T00:40:13.678873+0000","last_active":"2024-02-13T00:41:03.989598+0000","last_peered":"2024-02-13T00:41:03.989598+0000","last_clean":"2024-02-13T00:41:03.989598+0000","last_became_active":"2024-02-13T00:37:47.260025+0000","last_became_peered":"2024-02-13T00:37:47.260025+0000","last_unstale":"2024-02-13T00:41:03.989598+0000","last_undegraded":"2024-02-13T00:41:03.989598+0000","last_fullsized":"2024-02-13T00:41:03.989598+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:13.678807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:40:13.678807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.d","version":"108'5","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.988974+0000","last_change":"2024-02-13T00:41:01.688873+0000","last_active":"2024-02-13T00:41:03.988974+0000","last_peered":"2024-02-13T00:41:03.988974+0000","last_clean":"2024-02-13T00:41:03.988974+0000","last_became_active":"2024-02-13T00:41:01.688580+0000","last_became_peered":"2024-02-13T00:41:01.688580+0000","last_unstale":"2024-02-13T00:41:03.988974+0000","last_undegraded":"2024-02-13T00:41:03.988974+0000","last_fullsized":"2024-02-13T00:41:03.988974+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.b","version":"109'4","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.862536+0000","last_change":"2024-02-13T00:41:01.599162+0000","last_active":"2024-02-13T00:41:04.862536+0000","last_peered":"2024-02-13T00:41:04.862536+0000","last_clean":"2024-02-13T00:41:04.862536+0000","last_became_active":"2024-02-13T00:41:01.598565+0000","last_became_peered":"2024-02-13T00:41:01.598565+0000","last_unstale":"2024-02-13T00:41:04.862536+0000","last_undegraded":"2024-02-13T00:41:04.862536+0000","last_fullsized":"2024-02-13T00:41:04.862536+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":27325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.007458+0000","last_change":"2024-02-13T00:41:04.007458+0000","last_active":"2024-02-13T00:41:04.007458+0000","last_peered":"2024-02-13T00:41:04.007458+0000","last_clean":"2024-02-13T00:41:04.007458+0000","last_became_active":"2024-02-13T00:41:01.564113+0000","last_became_peered":"2024-02-13T00:41:01.564113+0000","last_unstale":"2024-02-13T00:41:04.007458+0000","last_undegraded":"2024-02-13T00:41:04.007458+0000","last_fullsized":"2024-02-13T00:41:04.007458+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995322+0000","last_change":"2024-02-13T00:41:01.662500+0000","last_active":"2024-02-13T00:41:03.995322+0000","last_peered":"2024-02-13T00:41:03.995322+0000","last_clean":"2024-02-13T00:41:03.995322+0000","last_became_active":"2024-02-13T00:41:01.660978+0000","last_became_peered":"2024-02-13T00:41:01.660978+0000","last_unstale":"2024-02-13T00:41:03.995322+0000","last_undegraded":"2024-02-13T00:41:03.995322+0000","last_fullsized":"2024-02-13T00:41:03.995322+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.029245+0000","last_change":"2024-02-13T00:41:04.029245+0000","last_active":"2024-02-13T00:41:04.029245+0000","last_peered":"2024-02-13T00:41:04.029245+0000","last_clean":"2024-02-13T00:41:04.029245+0000","last_became_active":"2024-02-13T00:41:01.683430+0000","last_became_peered":"2024-02-13T00:41:01.683430+0000","last_unstale":"2024-02-13T00:41:04.029245+0000","last_undegraded":"2024-02-13T00:41:04.029245+0000","last_fullsized":"2024-02-13T00:41:04.029245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.17","version":"107'4","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.994825+0000","last_change":"2024-02-13T00:41:01.578161+0000","last_active":"2024-02-13T00:41:03.994825+0000","last_peered":"2024-02-13T00:41:03.994825+0000","last_clean":"2024-02-13T00:41:03.994825+0000","last_became_active":"2024-02-13T00:41:01.576899+0000","last_became_peered":"2024-02-13T00:41:01.576899+0000","last_unstale":"2024-02-13T00:41:03.994825+0000","last_undegraded":"2024-02-13T00:41:03.994825+0000","last_fullsized":"2024-02-13T00:41:03.994825+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.003501+0000","last_change":"2024-02-13T00:41:01.636289+0000","last_active":"2024-02-13T00:41:04.003501+0000","last_peered":"2024-02-13T00:41:04.003501+0000","last_clean":"2024-02-13T00:41:04.003501+0000","last_became_active":"2024-02-13T00:41:01.636013+0000","last_became_peered":"2024-02-13T00:41:01.636013+0000","last_unstale":"2024-02-13T00:41:04.003501+0000","last_undegraded":"2024-02-13T00:41:04.003501+0000","last_fullsized":"2024-02-13T00:41:04.003501+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"90'20","reported_seq":97,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995164+0000","last_change":"2024-02-13T00:40:35.847653+0000","last_active":"2024-02-13T00:41:03.995164+0000","last_peered":"2024-02-13T00:41:03.995164+0000","last_clean":"2024-02-13T00:41:03.995164+0000","last_became_active":"2024-02-13T00:40:35.847203+0000","last_became_peered":"2024-02-13T00:40:35.847203+0000","last_unstale":"2024-02-13T00:41:03.995164+0000","last_undegraded":"2024-02-13T00:41:03.995164+0000","last_fullsized":"2024-02-13T00:41:03.995164+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:40:13.144451+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.000472+0000","last_change":"2024-02-13T00:39:40.280385+0000","last_active":"2024-02-13T00:41:04.000472+0000","last_peered":"2024-02-13T00:41:04.000472+0000","last_clean":"2024-02-13T00:41:04.000472+0000","last_became_active":"2024-02-13T00:37:41.216896+0000","last_became_peered":"2024-02-13T00:37:41.216896+0000","last_unstale":"2024-02-13T00:41:04.000472+0000","last_undegraded":"2024-02-13T00:41:04.000472+0000","last_fullsized":"2024-02-13T00:41:04.000472+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:39:40.280305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":63,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.027890+0000","last_change":"2024-02-13T00:39:24.332217+0000","last_active":"2024-02-13T00:41:04.027890+0000","last_peered":"2024-02-13T00:41:04.027890+0000","last_clean":"2024-02-13T00:41:04.027890+0000","last_became_active":"2024-02-13T00:37:45.249914+0000","last_became_peered":"2024-02-13T00:37:45.249914+0000","last_unstale":"2024-02-13T00:41:04.027890+0000","last_undegraded":"2024-02-13T00:41:04.027890+0000","last_fullsized":"2024-02-13T00:41:04.027890+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:24.332133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.332133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":26,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.989702+0000","last_change":"2024-02-13T00:40:35.846021+0000","last_active":"2024-02-13T00:41:03.989702+0000","last_peered":"2024-02-13T00:41:03.989702+0000","last_clean":"2024-02-13T00:41:03.989702+0000","last_became_active":"2024-02-13T00:40:35.845664+0000","last_became_peered":"2024-02-13T00:40:35.845664+0000","last_unstale":"2024-02-13T00:41:03.989702+0000","last_undegraded":"2024-02-13T00:41:03.989702+0000","last_fullsized":"2024-02-13T00:41:03.989702+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:37.210406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:37.210406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.010636+0000","last_change":"2024-02-13T00:41:01.747460+0000","last_active":"2024-02-13T00:41:04.010636+0000","last_peered":"2024-02-13T00:41:04.010636+0000","last_clean":"2024-02-13T00:41:04.010636+0000","last_became_active":"2024-02-13T00:41:01.743088+0000","last_became_peered":"2024-02-13T00:41:01.743088+0000","last_unstale":"2024-02-13T00:41:04.010636+0000","last_undegraded":"2024-02-13T00:41:04.010636+0000","last_fullsized":"2024-02-13T00:41:04.010636+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995248+0000","last_change":"2024-02-13T00:41:01.662030+0000","last_active":"2024-02-13T00:41:03.995248+0000","last_peered":"2024-02-13T00:41:03.995248+0000","last_clean":"2024-02-13T00:41:03.995248+0000","last_became_active":"2024-02-13T00:41:01.659916+0000","last_became_peered":"2024-02-13T00:41:01.659916+0000","last_unstale":"2024-02-13T00:41:03.995248+0000","last_undegraded":"2024-02-13T00:41:03.995248+0000","last_fullsized":"2024-02-13T00:41:03.995248+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.041345+0000","last_change":"2024-02-13T00:41:04.041345+0000","last_active":"2024-02-13T00:41:04.041345+0000","last_peered":"2024-02-13T00:41:04.041345+0000","last_clean":"2024-02-13T00:41:04.041345+0000","last_became_active":"2024-02-13T00:41:01.721752+0000","last_became_peered":"2024-02-13T00:41:01.721752+0000","last_unstale":"2024-02-13T00:41:04.041345+0000","last_undegraded":"2024-02-13T00:41:04.041345+0000","last_fullsized":"2024-02-13T00:41:04.041345+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"28.1","version":"107'7","reported_seq":25,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.032832+0000","last_change":"2024-02-13T00:41:01.732312+0000","last_active":"2024-02-13T00:41:04.032832+0000","last_peered":"2024-02-13T00:41:04.032832+0000","last_clean":"2024-02-13T00:41:04.032832+0000","last_became_active":"2024-02-13T00:41:01.728806+0000","last_became_peered":"2024-02-13T00:41:01.728806+0000","last_unstale":"2024-02-13T00:41:04.032832+0000","last_undegraded":"2024-02-13T00:41:04.032832+0000","last_fullsized":"2024-02-13T00:41:04.032832+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.041474+0000","last_change":"2024-02-13T00:41:04.041474+0000","last_active":"2024-02-13T00:41:04.041474+0000","last_peered":"2024-02-13T00:41:04.041474+0000","last_clean":"2024-02-13T00:41:04.041474+0000","last_became_active":"2024-02-13T00:41:01.741335+0000","last_became_peered":"2024-02-13T00:41:01.741335+0000","last_unstale":"2024-02-13T00:41:04.041474+0000","last_undegraded":"2024-02-13T00:41:04.041474+0000","last_fullsized":"2024-02-13T00:41:04.041474+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.014970+0000","last_change":"2024-02-13T00:41:01.652393+0000","last_active":"2024-02-13T00:41:04.014970+0000","last_peered":"2024-02-13T00:41:04.014970+0000","last_clean":"2024-02-13T00:41:04.014970+0000","last_became_active":"2024-02-13T00:41:01.651333+0000","last_became_peered":"2024-02-13T00:41:01.651333+0000","last_unstale":"2024-02-13T00:41:04.014970+0000","last_undegraded":"2024-02-13T00:41:04.014970+0000","last_fullsized":"2024-02-13T00:41:04.014970+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.997922+0000","last_change":"2024-02-13T00:41:00.551894+0000","last_active":"2024-02-13T00:41:03.997922+0000","last_peered":"2024-02-13T00:41:03.997922+0000","last_clean":"2024-02-13T00:41:03.997922+0000","last_became_active":"2024-02-13T00:41:00.547009+0000","last_became_peered":"2024-02-13T00:41:00.547009+0000","last_unstale":"2024-02-13T00:41:03.997922+0000","last_undegraded":"2024-02-13T00:41:03.997922+0000","last_fullsized":"2024-02-13T00:41:03.997922+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.447090+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.447090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"86'50","reported_seq":165,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.988515+0000","last_change":"2024-02-13T00:39:53.119049+0000","last_active":"2024-02-13T00:41:03.988515+0000","last_peered":"2024-02-13T00:41:03.988515+0000","last_clean":"2024-02-13T00:41:03.988515+0000","last_became_active":"2024-02-13T00:37:43.228728+0000","last_became_peered":"2024-02-13T00:37:43.228728+0000","last_unstale":"2024-02-13T00:41:03.988515+0000","last_undegraded":"2024-02-13T00:41:03.988515+0000","last_fullsized":"2024-02-13T00:41:03.988515+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:39:53.118989+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.019593+0000","last_change":"2024-02-13T00:39:32.313930+0000","last_active":"2024-02-13T00:41:04.019593+0000","last_peered":"2024-02-13T00:41:04.019593+0000","last_clean":"2024-02-13T00:41:04.019593+0000","last_became_active":"2024-02-13T00:37:41.220583+0000","last_became_peered":"2024-02-13T00:37:41.220583+0000","last_unstale":"2024-02-13T00:41:04.019593+0000","last_undegraded":"2024-02-13T00:41:04.019593+0000","last_fullsized":"2024-02-13T00:41:04.019593+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.313841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":26,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.995002+0000","last_change":"2024-02-13T00:40:35.847474+0000","last_active":"2024-02-13T00:41:03.995002+0000","last_peered":"2024-02-13T00:41:03.995002+0000","last_clean":"2024-02-13T00:41:03.995002+0000","last_became_active":"2024-02-13T00:40:35.846778+0000","last_became_peered":"2024-02-13T00:40:35.846778+0000","last_unstale":"2024-02-13T00:41:03.995002+0000","last_undegraded":"2024-02-13T00:41:03.995002+0000","last_fullsized":"2024-02-13T00:41:03.995002+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.273346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:03.989121+0000","last_change":"2024-02-13T00:40:54.059354+0000","last_active":"2024-02-13T00:41:03.989121+0000","last_peered":"2024-02-13T00:41:03.989121+0000","last_clean":"2024-02-13T00:41:03.989121+0000","last_became_active":"2024-02-13T00:37:47.552890+0000","last_became_peered":"2024-02-13T00:37:47.552890+0000","last_unstale":"2024-02-13T00:41:03.989121+0000","last_undegraded":"2024-02-13T00:41:03.989121+0000","last_fullsized":"2024-02-13T00:41:03.989121+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:54.059274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:54.059274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.006866+0000","last_change":"2024-02-13T00:41:01.573580+0000","last_active":"2024-02-13T00:41:04.006866+0000","last_peered":"2024-02-13T00:41:04.006866+0000","last_clean":"2024-02-13T00:41:04.006866+0000","last_became_active":"2024-02-13T00:41:01.573417+0000","last_became_peered":"2024-02-13T00:41:01.573417+0000","last_unstale":"2024-02-13T00:41:04.006866+0000","last_undegraded":"2024-02-13T00:41:04.006866+0000","last_fullsized":"2024-02-13T00:41:04.006866+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"107'5","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.008681+0000","last_change":"2024-02-13T00:41:01.530688+0000","last_active":"2024-02-13T00:41:04.008681+0000","last_peered":"2024-02-13T00:41:04.008681+0000","last_clean":"2024-02-13T00:41:04.008681+0000","last_became_active":"2024-02-13T00:41:01.530359+0000","last_became_peered":"2024-02-13T00:41:01.530359+0000","last_unstale":"2024-02-13T00:41:04.008681+0000","last_undegraded":"2024-02-13T00:41:04.008681+0000","last_fullsized":"2024-02-13T00:41:04.008681+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-13T00:41:04.024059+0000","last_change":"2024-02-13T00:41:04.024059+0000","last_active":"2024-02-13T00:41:04.024059+0000","last_peered":"2024-02-13T00:41:04.024059+0000","last_clean":"2024-02-13T00:41:04.024059+0000","last_became_active":"2024-02-13T00:41:01.684915+0000","last_became_peered":"2024-02-13T00:41:01.684915+0000","last_unstale":"2024-02-13T00:41:04.024059+0000","last_undegraded":"2024-02-13T00:41:04.024059+0000","last_fullsized":"2024-02-13T00:41:04.024059+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"}]}],"pool_stats":[{"poolid":45,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":44,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":150998978,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":1011,"num_read_kb":49161,"num_write":1043,"num_write_kb":152978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":453267456,"data_stored":452996934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1053,"ondisk_log_size":1053,"up":96,"acting":96,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":43,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":41,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1073,"num_read_kb":798,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":669,"ondisk_log_size":669,"up":96,"acting":96,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":192,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":38,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":39,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":896,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266731,"num_pgs":252,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":400584,"kb_used_data":99300,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93344568,"statfs":{"total":95995035648,"available":95584837632,"internally_reserved":0,"allocated":101683200,"data_stored":101126334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.502,"15min":0.502},"min":{"1min":0.26400000000000001,"5min":0.25,"15min":0.25},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":4.585},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":0.26700000000000002}]},{"osd":1,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.57199999999999995,"15min":0.60599999999999998},"min":{"1min":0.379,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61699999999999999,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.32800000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.1619999999999999,"5min":3.1619999999999999,"15min":3.1619999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":0.47999999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.097,"5min":3.097,"15min":3.097},"last":0.317},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.57999999999999996,"15min":0.54800000000000004},"min":{"1min":0.40699999999999997,"5min":0.318,"15min":0.318},"max":{"1min":3.7690000000000001,"5min":3.7690000000000001,"15min":3.7690000000000001},"last":0.436}]},{"osd":4,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.46300000000000002,"15min":0.47099999999999997},"min":{"1min":0.29199999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66000000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.45400000000000001,"15min":0.46400000000000002},"min":{"1min":0.32800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.34599999999999997}]},{"osd":5,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.47099999999999997,"15min":0.47499999999999998},"min":{"1min":0.315,"5min":0.255,"15min":0.255},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.48099999999999998,"15min":0.47599999999999998},"min":{"1min":0.34200000000000003,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68600000000000005,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.375}]},{"osd":6,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.502,"15min":0.5},"min":{"1min":0.38400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70199999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.52600000000000002},"min":{"1min":0.35899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.38500000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528369,"num_pgs":232,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302184,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93442968,"statfs":{"total":95995035648,"available":95685599232,"internally_reserved":0,"allocated":917504,"data_stored":437713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52700000000000002,"15min":0.54200000000000004},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.63300000000000001,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.34499999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.622,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.44500000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.502,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66800000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":3.3780000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34399999999999997,"5min":0.315,"15min":0.315},"max":{"1min":0.624,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":5.0540000000000003}]},{"osd":2,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.498,"15min":0.5},"min":{"1min":0.379,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.65400000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":8.0280000000000005},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.34399999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":1.101}]},{"osd":3,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.39500000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71599999999999997,"5min":1.181,"15min":1.181},"last":1.343},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.372,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.65700000000000003,"5min":1.488,"15min":1.488},"last":3.3599999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.41299999999999998,"15min":0.40000000000000002},"min":{"1min":0.32600000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64800000000000002,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.442,"15min":0.42499999999999999},"min":{"1min":0.41099999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76600000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.435}]},{"osd":5,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.41299999999999998,"15min":0.39100000000000001},"min":{"1min":0.35999999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.70699999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.47099999999999997,"15min":0.45000000000000001},"min":{"1min":0.38200000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.76200000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47599999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.41699999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70299999999999996,"5min":1.173,"15min":1.173},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.42299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.749,"5min":1.262,"15min":1.262},"last":0.502}]}]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":248,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":351412,"kb_used_data":50124,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93393740,"statfs":{"total":95995035648,"available":95635189760,"internally_reserved":0,"allocated":51326976,"data_stored":50770785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34699999999999998,"15min":0.32200000000000001},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.43099999999999999,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":3.8919999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.34599999999999997,"15min":0.314},"min":{"1min":0.215,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.42299999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":4.1790000000000003}]},{"osd":2,"last update":"Tue Feb 13 00:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.374,"15min":0.372},"min":{"1min":0.218,"5min":0.215,"15min":0.215},"max":{"1min":0.56799999999999995,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.38700000000000001},"min":{"1min":0.25700000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":3.621}]},{"osd":3,"last update":"Tue Feb 13 00:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.41699999999999998,"15min":0.44},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.46899999999999997,"5min":0.748,"15min":0.748},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.42999999999999999,"15min":0.45800000000000002},"min":{"1min":0.30099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.504,"5min":0.745,"15min":0.745},"last":6.0540000000000003}]},{"osd":4,"last update":"Tue Feb 13 00:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76500000000000001,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":3.8679999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.41699999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81899999999999995,"5min":1.02,"15min":1.02},"last":3.9359999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58599999999999997,"15min":0.58499999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.95799999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":3.8140000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.58399999999999996,"15min":0.59499999999999997},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.75,"5min":1.054,"15min":1.054},"last":3.786},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.42099999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88300000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.35099999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.59499999999999997},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.78100000000000003,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":3.9049999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.41799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82899999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.32200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":232,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":330940,"kb_used_data":25524,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93414212,"statfs":{"total":95995035648,"available":95656153088,"internally_reserved":0,"allocated":26136576,"data_stored":25626322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:40:19 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.34599999999999997,"15min":0.32800000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.69699999999999995,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.38200000000000001,"15min":0.34599999999999997},"min":{"1min":0.23200000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.60499999999999998,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":1.4370000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.42299999999999999,"15min":0.40400000000000003},"min":{"1min":0.245,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.40000000000000002,"15min":0.38400000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.25700000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.29599999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.45900000000000002,"15min":0.47099999999999997},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.38200000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:40:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.38900000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53200000000000003,"15min":0.51700000000000002},"min":{"1min":0.41699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.59899999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:40:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56000000000000005,"15min":0.57399999999999995},"min":{"1min":0.34100000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.55300000000000005,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.51800000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52300000000000002,"15min":0.50900000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.84299999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.57399999999999995}]},{"osd":7,"last update":"Tue Feb 13 00:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.55600000000000005,"15min":0.55100000000000005},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.629},{"interface":"front","average":{"1min":0.622,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.439,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.90400000000000003,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.65900000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":343,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306380,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93438772,"statfs":{"total":95995035648,"available":95681302528,"internally_reserved":0,"allocated":929792,"data_stored":439240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35899999999999999,"15min":0.35499999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.438},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.374,"15min":0.34999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.76100000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.53100000000000003}]},{"osd":1,"last update":"Tue Feb 13 00:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.36799999999999999,"15min":0.35199999999999998},"min":{"1min":0.26000000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38700000000000001,"15min":0.375},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.222}]},{"osd":3,"last update":"Tue Feb 13 00:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.51500000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.43099999999999999,"15min":0.441},"min":{"1min":0.28000000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56399999999999995,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.32100000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:40:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56100000000000005,"15min":0.56699999999999995},"min":{"1min":0.379,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76600000000000001,"5min":0.873,"15min":0.873},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.74099999999999999,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.41199999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:40:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.56000000000000005},"min":{"1min":0.42399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.90500000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57399999999999995,"15min":0.59199999999999997},"min":{"1min":0.46899999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.86499999999999999,"5min":0.872,"15min":0.872},"last":0.38700000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:40:10 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69999999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.38700000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68200000000000005,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.504}]},{"osd":7,"last update":"Tue Feb 13 00:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.58799999999999997},"min":{"1min":0.379,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68300000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.433,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89400000000000002,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.56899999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":267,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355404,"kb_used_data":50100,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93389748,"statfs":{"total":95995035648,"available":95631101952,"internally_reserved":0,"allocated":51302400,"data_stored":50773820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35099999999999998,"15min":0.373},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.191},"max":{"1min":0.48699999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":1.657},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38100000000000001,"15min":0.40400000000000003},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":1.1879999999999999}]},{"osd":1,"last update":"Tue Feb 13 00:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.38400000000000001,"15min":0.39800000000000002},"min":{"1min":0.24099999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.378,"15min":0.39500000000000002},"min":{"1min":0.21299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.47999999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":5.7039999999999997}]},{"osd":2,"last update":"Tue Feb 13 00:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40799999999999997,"15min":0.41899999999999998},"min":{"1min":0.29299999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":5.6760000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38600000000000001,"15min":0.40100000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.623,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":1.6299999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.39900000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72499999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":1.526},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53300000000000003,"15min":0.53800000000000003},"min":{"1min":0.42399999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":1.101}]},{"osd":5,"last update":"Tue Feb 13 00:40:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.54300000000000004,"15min":0.51100000000000001},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":5.6020000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.54000000000000004,"15min":0.52200000000000002},"min":{"1min":0.46100000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81399999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":1.4910000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53000000000000003,"15min":0.51400000000000001},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74399999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":5.6470000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.45200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.69899999999999995,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":1.569}]},{"osd":7,"last update":"Tue Feb 13 00:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.55300000000000005,"15min":0.54700000000000004},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":1.5980000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.435,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81899999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":1.1359999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986235,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302148,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93443004,"statfs":{"total":95995035648,"available":95685636096,"internally_reserved":0,"allocated":942080,"data_stored":439818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.40200000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65700000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.313},{"interface":"front","average":{"1min":0.505,"5min":0.52500000000000002,"15min":0.54100000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70299999999999996,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.47899999999999998}]},{"osd":1,"last update":"Tue Feb 13 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72499999999999998,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.40200000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.70999999999999996,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.33700000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.374,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51900000000000002,"15min":0.52600000000000002},"min":{"1min":0.36699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.66700000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.439}]},{"osd":3,"last update":"Tue Feb 13 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.66900000000000004,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.373},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52700000000000002,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69599999999999995,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.46100000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:40:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.39200000000000002,"15min":0.35799999999999998},"min":{"1min":0.317,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.41699999999999998,"15min":0.376},"min":{"1min":0.36199999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60799999999999998,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.45300000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:40:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.42299999999999999,"15min":0.40200000000000002},"min":{"1min":0.33400000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.41199999999999998},"min":{"1min":0.36399999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.42699999999999999}]},{"osd":7,"last update":"Tue Feb 13 00:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.48099999999999998,"15min":0.46600000000000003},"min":{"1min":0.42099999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.47099999999999997,"15min":0.45700000000000002},"min":{"1min":0.40500000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.51100000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724598,"num_pgs":309,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302256,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93442896,"statfs":{"total":95995035648,"available":95685525504,"internally_reserved":0,"allocated":901120,"data_stored":438435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.49199999999999999,"15min":0.48599999999999999},"min":{"1min":0.318,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70599999999999996,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":72.444999999999993},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53400000000000003,"15min":0.51300000000000001},"min":{"1min":0.35399999999999998,"5min":0.309,"15min":0.309},"max":{"1min":1.0640000000000001,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":23.949000000000002}]},{"osd":1,"last update":"Tue Feb 13 00:40:56 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.504,"15min":0.498},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.66100000000000003,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":23.937000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52900000000000003,"15min":0.503},"min":{"1min":0.30499999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.69399999999999995,"5min":4.0839999999999996,"15min":4.0839999999999996},"last":42.826999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:40:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51600000000000001,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.94199999999999995,"5min":1.099,"15min":1.099},"last":32.790999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.51700000000000002,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.70999999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":31.422999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:40:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.58299999999999996,"15min":0.52800000000000002},"min":{"1min":0.36799999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.5640000000000001,"5min":5.5640000000000001,"15min":5.5640000000000001},"last":61.573},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.52800000000000002,"15min":0.50800000000000001},"min":{"1min":0.41399999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77600000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":10.135}]},{"osd":4,"last update":"Tue Feb 13 00:40:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.442,"15min":0.374},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63800000000000001,"5min":4.1210000000000004,"15min":4.1210000000000004},"last":7.8570000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.42899999999999999},"min":{"1min":0.39300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.71699999999999997,"5min":5.6950000000000003,"15min":5.6950000000000003},"last":8.0630000000000006}]},{"osd":6,"last update":"Tue Feb 13 00:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.45600000000000002,"15min":0.432},"min":{"1min":0.33600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64800000000000002,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":8.1020000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.47499999999999998,"15min":0.45100000000000001},"min":{"1min":0.34799999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73499999999999999,"5min":1.377,"15min":1.377},"last":7.7389999999999999}]},{"osd":7,"last update":"Tue Feb 13 00:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.69599999999999995,"15min":0.78400000000000003},"min":{"1min":0.377,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69499999999999995,"5min":7.5519999999999996,"15min":7.5519999999999996},"last":7.915},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55900000000000005,"15min":0.57499999999999996},"min":{"1min":0.435,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":8.0169999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25168657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50380800,"data_stored":50331798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75538432,"data_stored":75498616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50352128,"data_stored":50334487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":100704256,"data_stored":100664377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25182208,"data_stored":25165858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25165863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":100728832,"data_stored":100667278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:41:08.985 INFO:tasks.workunit.client.0.smithi162.stdout:"}]: dispatch 2024-02-13T00:41:08.985 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.008358+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-83993-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:08.985 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.012144+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-84097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:08.985 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.012218+0000 mon.b [INF] from='client.? 172.21.15.162:0/3769162824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:08.985 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.013127+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:08.985 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.013411+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:08.985 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.056609+0000 mon.a [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:08.985 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.059269+0000 mon.a [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:08.985 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.060343+0000 mon.a [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-84025-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:08.985 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.065940+0000 mon.b [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:08.986 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.069037+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:08.986 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.071945+0000 mon.b [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:08.986 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.078083+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:08.986 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.089811+0000 mon.b [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:08.986 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:05.090841+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:08.986 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:06.973575+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:09.009 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:09.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:09.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:09.022 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:41:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:41:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:41:09.023 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:06.973666+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:09.023 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:06.973760+0000 mon.a [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-84025-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:09.024 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:06.973820+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:09.024 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:06.974341+0000 mon.b [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-84033-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:09.024 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:06.975012+0000 mon.c [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:09.024 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:06.976469+0000 mon.a [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-84025-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:09.024 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:06.981650+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-13T00:41:09.024 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:07.236845+0000 mon.a [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]: dispatch 2024-02-13T00:41:09.024 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:07.242909+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-84001-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-84001-22"}]': finished 2024-02-13T00:41:09.024 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:07.243008+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-84089-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-84089-6"}]': finished 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:07.243082+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-83993-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-83993-15"}]': finished 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:07.243157+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-84097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-84097-7"}]': finished 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:07.508845+0000 mon.c [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:07.514451+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.009706+0000 mon.a [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.013751+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.014097+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-84033-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.277866+0000 mon.a [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-84025-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-84025-10"}]': finished 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.277950+0000 mon.a [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]': finished 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.278039+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]': finished 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.278197+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.294779+0000 mon.c [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:09.025 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.315692+0000 mon.c [INF] from='client.? 172.21.15.162:0/3325301458' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.026 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.318400+0000 mon.c [INF] from='client.? 172.21.15.162:0/1781481349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-83977-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.026 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.526552+0000 mon.b [INF] from='client.? 172.21.15.162:0/3769162824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.026 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.526821+0000 mon.a [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-4", "overlaypool": "test-rados-api-smithi162-84057-4-cache"}]: dispatch 2024-02-13T00:41:09.026 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.527451+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:09.026 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.527896+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.026 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.528201+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:09.141 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:09.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:06.960168+0000 osd.0 (osd.0) 55 : cluster [DBG] 20.3 deep-scrub starts 2024-02-13T00:41:09.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:06.963407+0000 osd.0 (osd.0) 56 : cluster [DBG] 20.3 deep-scrub ok 2024-02-13T00:41:09.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:07.146096+0000 osd.7 (osd.7) 66 : cluster [DBG] 20.2 deep-scrub starts 2024-02-13T00:41:09.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:07.150109+0000 osd.7 (osd.7) 67 : cluster [DBG] 20.2 deep-scrub ok 2024-02-13T00:41:09.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:07.338058+0000 osd.6 (osd.6) 51 : cluster [DBG] 20.1 deep-scrub starts 2024-02-13T00:41:09.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:07.341671+0000 osd.6 (osd.6) 52 : cluster [DBG] 20.1 deep-scrub ok 2024-02-13T00:41:09.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:07.699206+0000 mgr.y (mgr.14544) 149 : audit [DBG] from='client.25111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:41:09.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:07.911202+0000 mon.b (mon.1) 72 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:08.009706+0000 mon.a (mon.0) 876 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.013751+0000 mon.a (mon.0) 877 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.014097+0000 mon.a (mon.0) 878 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-84033-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:08.205166+0000 mgr.y (mgr.14544) 150 : cluster [DBG] pgmap v139: 548 pgs: 64 unknown, 96 creating+peering, 388 active+clean; 144 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 35 MiB/s wr, 519 op/s 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.277866+0000 mon.a (mon.0) 879 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-84025-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-84025-10"}]': finished 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.277950+0000 mon.a (mon.0) 880 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]': finished 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.278039+0000 mon.a (mon.0) 881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]': finished 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.278197+0000 mon.a (mon.0) 882 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.294779+0000 mon.c (mon.2) 101 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.315692+0000 mon.c (mon.2) 102 : audit [INF] from='client.? 172.21.15.162:0/3325301458' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.318400+0000 mon.c (mon.2) 103 : audit [INF] from='client.? 172.21.15.162:0/1781481349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-83977-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:08.525735+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.526552+0000 mon.b (mon.1) 73 : audit [INF] from='client.? 172.21.15.162:0/3769162824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.298 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.526821+0000 mon.a (mon.0) 884 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-4", "overlaypool": "test-rados-api-smithi162-84057-4-cache"}]: dispatch 2024-02-13T00:41:09.299 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.527451+0000 mon.a (mon.0) 885 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:09.299 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.527896+0000 mon.a (mon.0) 886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.299 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.528201+0000 mon.a (mon.0) 887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.299 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.528558+0000 mon.a (mon.0) 888 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-83977-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.299 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.912759+0000 mon.b (mon.1) 74 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:09.299 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.982877+0000 mon.c (mon.2) 104 : audit [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-13T00:41:09.299 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:08.983006+0000 client.admin (client.?) 0 : cluster [INF] threexx 2024-02-13T00:41:09.299 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:08.983148+0000 mon.a (mon.0) 889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-13T00:41:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:06.960168+0000 osd.0 (osd.0) 55 : cluster [DBG] 20.3 deep-scrub starts 2024-02-13T00:41:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:06.963407+0000 osd.0 (osd.0) 56 : cluster [DBG] 20.3 deep-scrub ok 2024-02-13T00:41:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:07.146096+0000 osd.7 (osd.7) 66 : cluster [DBG] 20.2 deep-scrub starts 2024-02-13T00:41:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:07.150109+0000 osd.7 (osd.7) 67 : cluster [DBG] 20.2 deep-scrub ok 2024-02-13T00:41:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:07.338058+0000 osd.6 (osd.6) 51 : cluster [DBG] 20.1 deep-scrub starts 2024-02-13T00:41:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:07.341671+0000 osd.6 (osd.6) 52 : cluster [DBG] 20.1 deep-scrub ok 2024-02-13T00:41:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:07.699206+0000 mgr.y (mgr.14544) 149 : audit [DBG] from='client.25111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:41:09.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:07.911202+0000 mon.b (mon.1) 72 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:09.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:08.009706+0000 mon.a (mon.0) 876 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:09.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.013751+0000 mon.a (mon.0) 877 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.014097+0000 mon.a (mon.0) 878 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-84033-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:09.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:08.205166+0000 mgr.y (mgr.14544) 150 : cluster [DBG] pgmap v139: 548 pgs: 64 unknown, 96 creating+peering, 388 active+clean; 144 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 35 MiB/s wr, 519 op/s 2024-02-13T00:41:09.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.277866+0000 mon.a (mon.0) 879 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-84025-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-84025-10"}]': finished 2024-02-13T00:41:09.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.277950+0000 mon.a (mon.0) 880 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]': finished 2024-02-13T00:41:09.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.278039+0000 mon.a (mon.0) 881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]': finished 2024-02-13T00:41:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:06.960168+0000 osd.0 (osd.0) 55 : cluster [DBG] 20.3 deep-scrub starts 2024-02-13T00:41:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:06.963407+0000 osd.0 (osd.0) 56 : cluster [DBG] 20.3 deep-scrub ok 2024-02-13T00:41:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:07.146096+0000 osd.7 (osd.7) 66 : cluster [DBG] 20.2 deep-scrub starts 2024-02-13T00:41:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:07.150109+0000 osd.7 (osd.7) 67 : cluster [DBG] 20.2 deep-scrub ok 2024-02-13T00:41:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:07.338058+0000 osd.6 (osd.6) 51 : cluster [DBG] 20.1 deep-scrub starts 2024-02-13T00:41:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:07.341671+0000 osd.6 (osd.6) 52 : cluster [DBG] 20.1 deep-scrub ok 2024-02-13T00:41:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:07.699206+0000 mgr.y (mgr.14544) 149 : audit [DBG] from='client.25111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:41:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:07.911202+0000 mon.b (mon.1) 72 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:08.009706+0000 mon.a (mon.0) 876 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.013751+0000 mon.a (mon.0) 877 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.014097+0000 mon.a (mon.0) 878 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-84033-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:09.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:08.205166+0000 mgr.y (mgr.14544) 150 : cluster [DBG] pgmap v139: 548 pgs: 64 unknown, 96 creating+peering, 388 active+clean; 144 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 35 MiB/s wr, 519 op/s 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.277866+0000 mon.a (mon.0) 879 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-84025-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-84025-10"}]': finished 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.277950+0000 mon.a (mon.0) 880 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]': finished 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.278039+0000 mon.a (mon.0) 881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-84113-1"}]': finished 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.278197+0000 mon.a (mon.0) 882 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.294779+0000 mon.c (mon.2) 101 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.315692+0000 mon.c (mon.2) 102 : audit [INF] from='client.? 172.21.15.162:0/3325301458' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.318400+0000 mon.c (mon.2) 103 : audit [INF] from='client.? 172.21.15.162:0/1781481349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-83977-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:08.525735+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.526552+0000 mon.b (mon.1) 73 : audit [INF] from='client.? 172.21.15.162:0/3769162824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.526821+0000 mon.a (mon.0) 884 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-4", "overlaypool": "test-rados-api-smithi162-84057-4-cache"}]: dispatch 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.527451+0000 mon.a (mon.0) 885 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.527896+0000 mon.a (mon.0) 886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.528201+0000 mon.a (mon.0) 887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.528558+0000 mon.a (mon.0) 888 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-83977-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.912759+0000 mon.b (mon.1) 74 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:09.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.982877+0000 mon.c (mon.2) 104 : audit [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-13T00:41:09.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:08.983006+0000 client.admin (client.?) 0 : cluster [INF] threexx 2024-02-13T00:41:09.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:08.983148+0000 mon.a (mon.0) 889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-13T00:41:09.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.278197+0000 mon.a (mon.0) 882 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:09.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.294779+0000 mon.c (mon.2) 101 : audit [INF] from='client.? 172.21.15.162:0/2725867940' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:09.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.315692+0000 mon.c (mon.2) 102 : audit [INF] from='client.? 172.21.15.162:0/3325301458' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.318400+0000 mon.c (mon.2) 103 : audit [INF] from='client.? 172.21.15.162:0/1781481349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-83977-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:08.525735+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-13T00:41:09.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.526552+0000 mon.b (mon.1) 73 : audit [INF] from='client.? 172.21.15.162:0/3769162824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.526821+0000 mon.a (mon.0) 884 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-4", "overlaypool": "test-rados-api-smithi162-84057-4-cache"}]: dispatch 2024-02-13T00:41:09.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.527451+0000 mon.a (mon.0) 885 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]: dispatch 2024-02-13T00:41:09.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.527896+0000 mon.a (mon.0) 886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.528201+0000 mon.a (mon.0) 887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.528558+0000 mon.a (mon.0) 888 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-83977-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:09.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.912759+0000 mon.b (mon.1) 74 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:09.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.982877+0000 mon.c (mon.2) 104 : audit [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-13T00:41:09.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:08.983006+0000 client.admin (client.?) 0 : cluster [INF] threexx 2024-02-13T00:41:09.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:08.983148+0000 mon.a (mon.0) 889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-13T00:41:09.405 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.528558+0000 mon.a [INF] from='client.? ' ent api_io_pp: Running main() from gmock_main.cc 2024-02-13T00:41:09.405 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-02-13T00:41:09.405 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [----------] Global test environment set-up. 2024-02-13T00:41:09.405 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-02-13T00:41:09.405 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: seed 84001 2024-02-13T00:41:09.405 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-02-13T00:41:09.405 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-02-13T00:41:09.405 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-02-13T00:41:09.405 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (255 ms) 2024-02-13T00:41:09.405 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (20 ms) 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (6 ms) 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (6 ms) 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (4 ms) 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (5 ms) 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (6 ms) 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (7 ms) 2024-02-13T00:41:09.406 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-02-13T00:41:09.407 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (5 ms) 2024-02-13T00:41:09.407 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-02-13T00:41:09.407 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (5 ms) 2024-02-13T00:41:09.407 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-02-13T00:41:09.407 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (5 ms) 2024-02-13T00:41:09.407 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-02-13T00:41:09.407 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (4 ms) 2024-02-13T00:41:09.407 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-02-13T00:41:09.407 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (15 ms) 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (15 ms) 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (20 ms) 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (9 ms) 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (8 ms) 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (4 ms) 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (9 ms) 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (408 ms total) 2024-02-13T00:41:09.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: 2024-02-13T00:41:09.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-02-13T00:41:09.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-02-13T00:41:09.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (2109 ms) 2024-02-13T00:41:09.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-02-13T00:41:09.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (13 ms) 2024-02-13T00:41:09.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-02-13T00:41:09.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (5 ms) 2024-02-13T00:41:09.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-02-13T00:41:09.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2024-02-13T00:41:09.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-02-13T00:41:09.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (7 ms) 2024-02-13T00:41:10.043 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.022854+0000 mon.a (mon.0) 890 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-13T00:41:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.024120+0000 mon.c (mon.2) 105 : audit [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-13T00:41:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:09.024250+0000 client.admin (client.?) 0 : cluster [INF] fourxx 2024-02-13T00:41:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.024390+0000 mon.a (mon.0) 891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-13T00:41:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.279810+0000 mon.a (mon.0) 892 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-84033-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-84033-10"}]': finished 2024-02-13T00:41:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.279997+0000 mon.a (mon.0) 893 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-4", "overlaypool": "test-rados-api-smithi162-84057-4-cache"}]': finished 2024-02-13T00:41:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.280059+0000 mon.a (mon.0) 894 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]': finished 2024-02-13T00:41:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.280113+0000 mon.a (mon.0) 895 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.280169+0000 mon.a (mon.0) 896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.280201+0000 mon.a (mon.0) 897 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-83977-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:09.280255+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.282560+0000 mon.a (mon.0) 899 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-4-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.293859+0000 mon.b (mon.1) 75 : audit [INF] from='client.? 172.21.15.162:0/3769162824' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-83985-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.297893+0000 mon.a (mon.0) 900 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-83985-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:09.368498+0000 osd.6 (osd.6) 53 : cluster [DBG] 20.8 deep-scrub starts 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:09.372187+0000 osd.6 (osd.6) 54 : cluster [DBG] 20.8 deep-scrub ok 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.373850+0000 mon.c (mon.2) 106 : audit [DBG] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.374595+0000 mon.c (mon.2) 107 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:09.588356+0000 mon.a (mon.0) 901 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.594065+0000 mon.a (mon.0) 902 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-4-cache", "mode": "writeback"}]': finished 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.594175+0000 mon.a (mon.0) 903 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-83985-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:09.594419+0000 mon.a (mon.0) 904 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-13T00:41:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.606791+0000 mon.c (mon.2) 108 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:10.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.856036+0000 mon.a (mon.0) 905 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-4"}]: dispatch 2024-02-13T00:41:10.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.857049+0000 mon.a (mon.0) 906 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:10.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.859556+0000 mon.a (mon.0) 907 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:10.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:09.913844+0000 mon.b (mon.1) 76 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:10.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.022854+0000 mon.a (mon.0) 890 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-13T00:41:10.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.024120+0000 mon.c (mon.2) 105 : audit [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-13T00:41:10.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:09.024250+0000 client.admin (client.?) 0 : cluster [INF] fourxx 2024-02-13T00:41:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.024390+0000 mon.a (mon.0) 891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-13T00:41:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.279810+0000 mon.a (mon.0) 892 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-84033-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-84033-10"}]': finished 2024-02-13T00:41:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.279997+0000 mon.a (mon.0) 893 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-4", "overlaypool": "test-rados-api-smithi162-84057-4-cache"}]': finished 2024-02-13T00:41:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.280059+0000 mon.a (mon.0) 894 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]': finished 2024-02-13T00:41:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.280113+0000 mon.a (mon.0) 895 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.280169+0000 mon.a (mon.0) 896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.280201+0000 mon.a (mon.0) 897 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-83977-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:09.280255+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-13T00:41:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.282560+0000 mon.a (mon.0) 899 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-4-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:41:10.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.293859+0000 mon.b (mon.1) 75 : audit [INF] from='client.? 172.21.15.162:0/3769162824' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-83985-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-13T00:41:10.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.022854+0000 mon.a (mon.0) 890 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-13T00:41:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.024120+0000 mon.c (mon.2) 105 : audit [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-13T00:41:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:09.024250+0000 client.admin (client.?) 0 : cluster [INF] fourxx 2024-02-13T00:41:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.024390+0000 mon.a (mon.0) 891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-13T00:41:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.279810+0000 mon.a (mon.0) 892 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-84033-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-84033-10"}]': finished 2024-02-13T00:41:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.279997+0000 mon.a (mon.0) 893 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-4", "overlaypool": "test-rados-api-smithi162-84057-4-cache"}]': finished 2024-02-13T00:41:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.280059+0000 mon.a (mon.0) 894 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-84113-1"}]': finished 2024-02-13T00:41:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.280113+0000 mon.a (mon.0) 895 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-83985-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.280169+0000 mon.a (mon.0) 896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.280201+0000 mon.a (mon.0) 897 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-83977-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:09.280255+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-13T00:41:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.282560+0000 mon.a (mon.0) 899 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-4-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:41:10.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.293859+0000 mon.b (mon.1) 75 : audit [INF] from='client.? 172.21.15.162:0/3769162824' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-83985-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-13T00:41:10.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.297893+0000 mon.a (mon.0) 900 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-83985-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-13T00:41:10.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:09.368498+0000 osd.6 (osd.6) 53 : cluster [DBG] 20.8 deep-scrub starts 2024-02-13T00:41:10.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:09.372187+0000 osd.6 (osd.6) 54 : cluster [DBG] 20.8 deep-scrub ok 2024-02-13T00:41:10.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.373850+0000 mon.c (mon.2) 106 : audit [DBG] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-13T00:41:10.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.374595+0000 mon.c (mon.2) 107 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:10.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:09.588356+0000 mon.a (mon.0) 901 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:10.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.594065+0000 mon.a (mon.0) 902 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-4-cache", "mode": "writeback"}]': finished 2024-02-13T00:41:10.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.594175+0000 mon.a (mon.0) 903 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-83985-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-13T00:41:10.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:09.594419+0000 mon.a (mon.0) 904 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-13T00:41:10.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.606791+0000 mon.c (mon.2) 108 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:10.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.856036+0000 mon.a (mon.0) 905 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-4"}]: dispatch 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.857049+0000 mon.a (mon.0) 906 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.859556+0000 mon.a (mon.0) 907 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:09.913844+0000 mon.b (mon.1) 76 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.297893+0000 mon.a (mon.0) 900 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-83985-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:09.368498+0000 osd.6 (osd.6) 53 : cluster [DBG] 20.8 deep-scrub starts 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:09.372187+0000 osd.6 (osd.6) 54 : cluster [DBG] 20.8 deep-scrub ok 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.373850+0000 mon.c (mon.2) 106 : audit [DBG] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.374595+0000 mon.c (mon.2) 107 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:09.588356+0000 mon.a (mon.0) 901 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.594065+0000 mon.a (mon.0) 902 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-4-cache", "mode": "writeback"}]': finished 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.594175+0000 mon.a (mon.0) 903 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-83985-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:09.594419+0000 mon.a (mon.0) 904 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-13T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.606791+0000 mon.c (mon.2) 108 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:10.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.856036+0000 mon.a (mon.0) 905 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-4"}]: dispatch 2024-02-13T00:41:10.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.857049+0000 mon.a (mon.0) 906 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:10.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.859556+0000 mon.a (mon.0) 907 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:10.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:09.913844+0000 mon.b (mon.1) 76 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:10.992 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:10.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:11.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:11.129 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:11.149 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:11.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:11.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:11.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:11.281 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:11.301 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:11.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:11.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:11.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:11.432 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:11.451 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:41:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:11.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:41:11.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:41:11.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:41:11.581 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:11.601 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:41:11.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:11.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:41:11.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:11.733 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.023910+0000 mon.a (mon.0) 908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-13T00:41:11.733 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.600642+0000 mon.a (mon.0) 909 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-4"}]': finished 2024-02-13T00:41:11.733 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.600757+0000 mon.a (mon.0) 910 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:11.733 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.600956+0000 mon.a (mon.0) 911 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:11.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:10.601095+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-13T00:41:11.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.606017+0000 mon.a (mon.0) 913 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]: dispatch 2024-02-13T00:41:11.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.607253+0000 mon.a (mon.0) 914 : audit [INF] from='client.? 172.21.15.162:0/477439904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:11.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.607527+0000 mon.c (mon.2) 109 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app2"}]: dispatch 2024-02-13T00:41:11.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.608219+0000 mon.b (mon.1) 77 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:11.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.608441+0000 mon.b (mon.1) 78 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:11.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.608897+0000 mon.b (mon.1) 79 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:11.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.609647+0000 mon.b (mon.1) 80 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:11.734 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.023910+0000 mon.a (mon.0) 908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-13T00:41:11.734 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.600642+0000 mon.a (mon.0) 909 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-4"}]': finished 2024-02-13T00:41:11.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.600757+0000 mon.a (mon.0) 910 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:11.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.600956+0000 mon.a (mon.0) 911 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:11.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:10.601095+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-13T00:41:11.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.606017+0000 mon.a (mon.0) 913 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]: dispatch 2024-02-13T00:41:11.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.607253+0000 mon.a (mon.0) 914 : audit [INF] from='client.? 172.21.15.162:0/477439904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:11.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.607527+0000 mon.c (mon.2) 109 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app2"}]: dispatch 2024-02-13T00:41:11.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.608219+0000 mon.b (mon.1) 77 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:11.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.608441+0000 mon.b (mon.1) 78 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:11.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.608897+0000 mon.b (mon.1) 79 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:11.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.609647+0000 mon.b (mon.1) 80 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:11.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.611365+0000 mon.c (mon.2) 110 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:11.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.612102+0000 mon.a (mon.0) 915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:11.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.612398+0000 mon.a (mon.0) 916 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:11.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.612665+0000 mon.a (mon.0) 917 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:11.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.612937+0000 mon.a (mon.0) 918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:11.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.613231+0000 mon.a (mon.0) 919 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:11.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.683718+0000 mon.c (mon.2) 111 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pg_num": 4}]: dispatch 2024-02-13T00:41:11.736 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.611365+0000 mon.c (mon.2) 110 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:11.736 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.612102+0000 mon.a (mon.0) 915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:11.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.612398+0000 mon.a (mon.0) 916 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:11.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.612665+0000 mon.a (mon.0) 917 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:11.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.612937+0000 mon.a (mon.0) 918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:11.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.613231+0000 mon.a (mon.0) 919 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:11.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.683718+0000 mon.c (mon.2) 111 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pg_num": 4}]: dispatch 2024-02-13T00:41:11.737 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.023910+0000 mon.a (mon.0) 908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-13T00:41:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.600642+0000 mon.a (mon.0) 909 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-4"}]': finished 2024-02-13T00:41:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.600757+0000 mon.a (mon.0) 910 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.600956+0000 mon.a (mon.0) 911 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:10.601095+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-13T00:41:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.606017+0000 mon.a (mon.0) 913 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]: dispatch 2024-02-13T00:41:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.607253+0000 mon.a (mon.0) 914 : audit [INF] from='client.? 172.21.15.162:0/477439904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.607527+0000 mon.c (mon.2) 109 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app2"}]: dispatch 2024-02-13T00:41:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.608219+0000 mon.b (mon.1) 77 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.608441+0000 mon.b (mon.1) 78 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.608897+0000 mon.b (mon.1) 79 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:11.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.609647+0000 mon.b (mon.1) 80 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:11.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.611365+0000 mon.c (mon.2) 110 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:11.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.612102+0000 mon.a (mon.0) 915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:11.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.612398+0000 mon.a (mon.0) 916 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:11.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.612665+0000 mon.a (mon.0) 917 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:11.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.612937+0000 mon.a (mon.0) 918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:11.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.613231+0000 mon.a (mon.0) 919 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:11.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.683718+0000 mon.c (mon.2) 111 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pg_num": 4}]: dispatch 2024-02-13T00:41:12.032 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-83977-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.032 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.982877+0000 mon.c [INF] from='client.? 172.21.15.162:0/1853566404' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-13T00:41:12.033 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.983006+0000 client.admin [INF] threexx 2024-02-13T00:41:12.033 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-02-13T00:41:08.983148+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-13T00:41:12.033 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7483 ms) 2024-02-13T00:41:12.033 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (17306 ms total) 2024-02-13T00:41:12.033 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: 2024-02-13T00:41:12.033 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [----------] Global test environment tear-down 2024-02-13T00:41:12.033 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (17306 ms total) 2024-02-13T00:41:12.033 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ PASSED ] 4 tests. 2024-02-13T00:41:12.284 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:12.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:12.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:12.418 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:12.438 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:12.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:12.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:12.542 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:12.571 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:12.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:10.206747+0000 mgr.y (mgr.14544) 151 : cluster [DBG] pgmap v143: 660 pgs: 32 creating+peering, 628 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 32 KiB/s wr, 36 op/s 2024-02-13T00:41:12.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:10.915118+0000 mon.b (mon.1) 81 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.149333+0000 mon.a (mon.0) 920 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pg_num": 4}]: dispatch 2024-02-13T00:41:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:11.180102+0000 osd.7 (osd.7) 68 : cluster [DBG] 20.7 deep-scrub starts 2024-02-13T00:41:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:11.180989+0000 osd.7 (osd.7) 69 : cluster [DBG] 20.7 deep-scrub ok 2024-02-13T00:41:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:11.303815+0000 osd.4 (osd.4) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:11.303907+0000 osd.4 (osd.4) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:11.595788+0000 mon.a (mon.0) 921 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:41:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.602702+0000 mon.a (mon.0) 922 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]': finished 2024-02-13T00:41:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.602775+0000 mon.a (mon.0) 923 : audit [INF] from='client.? 172.21.15.162:0/477439904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.602827+0000 mon.a (mon.0) 924 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]': finished 2024-02-13T00:41:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.602878+0000 mon.a (mon.0) 925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.602950+0000 mon.a (mon.0) 926 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]': finished 2024-02-13T00:41:12.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.603020+0000 mon.a (mon.0) 927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]': finished 2024-02-13T00:41:12.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:10.206747+0000 mgr.y (mgr.14544) 151 : cluster [DBG] pgmap v143: 660 pgs: 32 creating+peering, 628 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 32 KiB/s wr, 36 op/s 2024-02-13T00:41:12.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:10.915118+0000 mon.b (mon.1) 81 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:12.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.149333+0000 mon.a (mon.0) 920 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pg_num": 4}]: dispatch 2024-02-13T00:41:12.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:11.180102+0000 osd.7 (osd.7) 68 : cluster [DBG] 20.7 deep-scrub starts 2024-02-13T00:41:12.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:11.180989+0000 osd.7 (osd.7) 69 : cluster [DBG] 20.7 deep-scrub ok 2024-02-13T00:41:12.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:11.303815+0000 osd.4 (osd.4) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:12.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:11.303907+0000 osd.4 (osd.4) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:12.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:11.595788+0000 mon.a (mon.0) 921 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:41:12.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.602702+0000 mon.a (mon.0) 922 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]': finished 2024-02-13T00:41:12.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.602775+0000 mon.a (mon.0) 923 : audit [INF] from='client.? 172.21.15.162:0/477439904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:12.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.602827+0000 mon.a (mon.0) 924 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]': finished 2024-02-13T00:41:12.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.602878+0000 mon.a (mon.0) 925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:12.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.602950+0000 mon.a (mon.0) 926 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]': finished 2024-02-13T00:41:12.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.603020+0000 mon.a (mon.0) 927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]': finished 2024-02-13T00:41:12.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.603087+0000 mon.a (mon.0) 928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]': finished 2024-02-13T00:41:12.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.603276+0000 mon.a (mon.0) 929 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pg_num": 4}]': finished 2024-02-13T00:41:12.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:11.603402+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-13T00:41:12.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.608278+0000 mon.c (mon.2) 112 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-13T00:41:12.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.608729+0000 mon.b (mon.1) 82 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:12.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.609913+0000 mon.c (mon.2) 113 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-13T00:41:12.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.610187+0000 mon.a (mon.0) 931 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:12.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.610312+0000 mon.c (mon.2) 114 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:12.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.610576+0000 mon.a (mon.0) 932 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-13T00:41:12.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.611336+0000 mon.a (mon.0) 933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:12.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.612426+0000 mon.b (mon.1) 83 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:12.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.612631+0000 mon.b (mon.1) 84 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:12.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.617724+0000 mon.b (mon.1) 85 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:12.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.603087+0000 mon.a (mon.0) 928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]': finished 2024-02-13T00:41:12.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.603276+0000 mon.a (mon.0) 929 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pg_num": 4}]': finished 2024-02-13T00:41:12.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:11.603402+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-13T00:41:12.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.608278+0000 mon.c (mon.2) 112 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-13T00:41:12.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.608729+0000 mon.b (mon.1) 82 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:12.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.609913+0000 mon.c (mon.2) 113 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-13T00:41:12.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.610187+0000 mon.a (mon.0) 931 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:12.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.610312+0000 mon.c (mon.2) 114 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:12.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.610576+0000 mon.a (mon.0) 932 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-13T00:41:12.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.611336+0000 mon.a (mon.0) 933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:12.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.612426+0000 mon.b (mon.1) 83 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:12.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.612631+0000 mon.b (mon.1) 84 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:12.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.617724+0000 mon.b (mon.1) 85 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.617907+0000 mon.b (mon.1) 86 : audit [INF] from='client.? 172.21.15.162:0/1199330434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-83977-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.622545+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.162:0/2365182943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.623073+0000 mon.a (mon.0) 934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.623459+0000 mon.a (mon.0) 935 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.623672+0000 mon.a (mon.0) 936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.639092+0000 mon.a (mon.0) 937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.639400+0000 mon.a (mon.0) 938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-83977-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.639670+0000 mon.a (mon.0) 939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.664829+0000 mon.b (mon.1) 88 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.665396+0000 mon.a (mon.0) 940 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.665722+0000 mon.b (mon.1) 89 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.666469+0000 mon.a (mon.0) 941 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:12.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.667382+0000 mon.b (mon.1) 90 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:12.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.668020+0000 mon.a (mon.0) 942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:12.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:11.916231+0000 mon.b (mon.1) 91 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:12.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.617907+0000 mon.b (mon.1) 86 : audit [INF] from='client.? 172.21.15.162:0/1199330434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-83977-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.622545+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.162:0/2365182943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.623073+0000 mon.a (mon.0) 934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:12.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.623459+0000 mon.a (mon.0) 935 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:12.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.623672+0000 mon.a (mon.0) 936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:12.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.639092+0000 mon.a (mon.0) 937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:12.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.639400+0000 mon.a (mon.0) 938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-83977-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.639670+0000 mon.a (mon.0) 939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.664829+0000 mon.b (mon.1) 88 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:12.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.665396+0000 mon.a (mon.0) 940 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:12.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.665722+0000 mon.b (mon.1) 89 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:12.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.666469+0000 mon.a (mon.0) 941 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:12.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.667382+0000 mon.b (mon.1) 90 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:12.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.668020+0000 mon.a (mon.0) 942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:12.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:11.916231+0000 mon.b (mon.1) 91 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:12.591 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:12.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:12.629 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: Running main() from gmock_main.cc 2024-02-13T00:41:12.629 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [==========] Running 23 tests from 2 test suites. 2024-02-13T00:41:12.629 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [----------] Global test environment set-up. 2024-02-13T00:41:12.629 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [----------] 13 tests from LibRadosIo 2024-02-13T00:41:12.629 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-02-13T00:41:12.629 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (264 ms) 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: no timeout :/ 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: no timeout :/ 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: no timeout :/ 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: no timeout :/ 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: no timeout :/ 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (51 ms) 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (12 ms) 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.Checksum (3 ms) 2024-02-13T00:41:12.630 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (7 ms) 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (9 ms) 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (5 ms) 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.TruncTest (8 ms) 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (8 ms) 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (6 ms) 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.RmXattr (26 ms) 2024-02-13T00:41:12.631 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.XattrIter (5 ms) 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [----------] 13 tests from LibRadosIo (404 ms total) 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (2105 ms) 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (14 ms) 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (4 ms) 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (7 ms) 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-02-13T00:41:12.632 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (10 ms) 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (4 ms) 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9 ms) 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [----------] 10 tests from LibRadosIoEC (2164 ms total) 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [----------] Global test environment tear-down 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [==========] 23 tests from 2 test suites ran. (17945 ms total) 2024-02-13T00:41:12.633 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ PASSED ] 23 tests. 2024-02-13T00:41:12.638 INFO:tasks.workunit.client.0.smithi162.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (4 ms) 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (3 ms) 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (4 ms) 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (3 ms) 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-02-13T00:41:12.639 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (4 ms) 2024-02-13T00:41:12.640 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-02-13T00:41:12.640 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (8 ms) 2024-02-13T00:41:12.640 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-02-13T00:41:12.640 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (3 ms) 2024-02-13T00:41:12.640 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-02-13T00:41:12.640 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (8 ms) 2024-02-13T00:41:12.640 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-02-13T00:41:12.640 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (4 ms) 2024-02-13T00:41:12.640 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-02-13T00:41:12.640 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (7 ms) 2024-02-13T00:41:12.641 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (2194 ms total) 2024-02-13T00:41:12.641 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: 2024-02-13T00:41:12.641 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [----------] Global test environment tear-down 2024-02-13T00:41:12.641 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (17933 ms total) 2024-02-13T00:41:12.641 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-02-13T00:41:12.644 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-02-13T00:41:12.644 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-02-13T00:41:12.644 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-02-13T00:41:12.644 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-02-13T00:41:12.644 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: seed 84097 2024-02-13T00:41:12.644 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-02-13T00:41:12.644 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (125 ms) 2024-02-13T00:41:12.644 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-02-13T00:41:12.644 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (139 ms) 2024-02-13T00:41:12.644 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-02-13T00:41:12.644 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (19 ms) 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (19 ms) 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (303 ms total) 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1062 ms) 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (11 ms) 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-02-13T00:41:12.645 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (1036 ms) 2024-02-13T00:41:12.646 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (2110 ms total) 2024-02-13T00:41:12.646 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: 2024-02-13T00:41:12.646 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-02-13T00:41:12.646 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (17960 ms total) 2024-02-13T00:41:12.646 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-02-13T00:41:12.646 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: Running main() from gmock_main.cc 2024-02-13T00:41:12.646 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [==========] Running 8 tests from 2 test suites. 2024-02-13T00:41:12.646 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [----------] Global test environment set-up. 2024-02-13T00:41:12.646 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [----------] 4 tests from LibRadosStat 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStat.Stat (267 ms) 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStat.StatNS (39 ms) 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (11 ms) 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [----------] 4 tests from LibRadosStat (317 ms total) 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (2088 ms) 2024-02-13T00:41:12.647 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-02-13T00:41:12.648 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (31 ms) 2024-02-13T00:41:12.648 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-02-13T00:41:12.648 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-02-13T00:41:12.648 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-02-13T00:41:12.648 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-02-13T00:41:12.648 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (2121 ms total) 2024-02-13T00:41:12.648 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: 2024-02-13T00:41:12.648 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [----------] Global test environment tear-down 2024-02-13T00:41:12.648 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [==========] 8 tests from 2 test suites ran. (17961 ms total) 2024-02-13T00:41:12.648 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ PASSED ] 8 tests. 2024-02-13T00:41:12.695 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:12.720 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:12.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:10.206747+0000 mgr.y (mgr.14544) 151 : cluster [DBG] pgmap v143: 660 pgs: 32 creating+peering, 628 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 32 KiB/s wr, 36 op/s 2024-02-13T00:41:12.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:10.915118+0000 mon.b (mon.1) 81 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:12.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.149333+0000 mon.a (mon.0) 920 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pg_num": 4}]: dispatch 2024-02-13T00:41:12.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:11.180102+0000 osd.7 (osd.7) 68 : cluster [DBG] 20.7 deep-scrub starts 2024-02-13T00:41:12.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:11.180989+0000 osd.7 (osd.7) 69 : cluster [DBG] 20.7 deep-scrub ok 2024-02-13T00:41:12.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:11.303815+0000 osd.4 (osd.4) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:12.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:11.303907+0000 osd.4 (osd.4) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:12.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:11.595788+0000 mon.a (mon.0) 921 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:41:12.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.602702+0000 mon.a (mon.0) 922 : audit [INF] from='client.? 172.21.15.162:0/181103790' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-4", "tierpool": "test-rados-api-smithi162-84057-4-cache"}]': finished 2024-02-13T00:41:12.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.602775+0000 mon.a (mon.0) 923 : audit [INF] from='client.? 172.21.15.162:0/477439904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:12.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.602827+0000 mon.a (mon.0) 924 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-84001-22"}]': finished 2024-02-13T00:41:12.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.602878+0000 mon.a (mon.0) 925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-84041-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:12.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.602950+0000 mon.a (mon.0) 926 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-83993-15"}]': finished 2024-02-13T00:41:12.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.603020+0000 mon.a (mon.0) 927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-84089-6"}]': finished 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.603087+0000 mon.a (mon.0) 928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-84097-7"}]': finished 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.603276+0000 mon.a (mon.0) 929 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pg_num": 4}]': finished 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:11.603402+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.608278+0000 mon.c (mon.2) 112 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.608729+0000 mon.b (mon.1) 82 : audit [INF] from='client.? 172.21.15.162:0/2697770486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.609913+0000 mon.c (mon.2) 113 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.610187+0000 mon.a (mon.0) 931 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.610312+0000 mon.c (mon.2) 114 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.610576+0000 mon.a (mon.0) 932 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.611336+0000 mon.a (mon.0) 933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.612426+0000 mon.b (mon.1) 83 : audit [INF] from='client.? 172.21.15.162:0/3269067102' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.612631+0000 mon.b (mon.1) 84 : audit [INF] from='client.? 172.21.15.162:0/279109448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.617724+0000 mon.b (mon.1) 85 : audit [INF] from='client.? 172.21.15.162:0/2874107663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:12.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.617907+0000 mon.b (mon.1) 86 : audit [INF] from='client.? 172.21.15.162:0/1199330434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-83977-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.622545+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.162:0/2365182943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.623073+0000 mon.a (mon.0) 934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.623459+0000 mon.a (mon.0) 935 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.623672+0000 mon.a (mon.0) 936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.639092+0000 mon.a (mon.0) 937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.639400+0000 mon.a (mon.0) 938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-83977-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.639670+0000 mon.a (mon.0) 939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.664829+0000 mon.b (mon.1) 88 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.665396+0000 mon.a (mon.0) 940 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.665722+0000 mon.b (mon.1) 89 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.666469+0000 mon.a (mon.0) 941 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.667382+0000 mon.b (mon.1) 90 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.668020+0000 mon.a (mon.0) 942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:12.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:11.916231+0000 mon.b (mon.1) 91 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:13.625 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:41:13.625 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:41:13.625 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-13T00:41:13.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-13T00:41:13.893 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: Running main() from gmock_main.cc 2024-02-13T00:41:13.893 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-02-13T00:41:13.893 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [----------] Global test environment set-up. 2024-02-13T00:41:13.893 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-02-13T00:41:13.893 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-02-13T00:41:13.893 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (299 ms) 2024-02-13T00:41:13.893 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-02-13T00:41:13.893 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.LockShared (13 ms) 2024-02-13T00:41:13.893 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-02-13T00:41:13.893 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1063 ms) 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1018 ms) 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (3 ms) 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.Unlock (4 ms) 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (4 ms) 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (2 ms) 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [----------] 8 tests from LibRadosLock (2406 ms total) 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-02-13T00:41:13.894 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (844 ms) 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (2 ms) 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1006 ms) 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1006 ms) 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (5 ms) 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (3 ms) 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (5 ms) 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-02-13T00:41:13.895 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (4 ms) 2024-02-13T00:41:13.896 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (2875 ms total) 2024-02-13T00:41:13.896 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: 2024-02-13T00:41:13.896 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [----------] Global test environment tear-down 2024-02-13T00:41:13.896 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (19210 ms total) 2024-02-13T00:41:13.896 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ PASSED ] 16 tests. 2024-02-13T00:41:14.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:12.207392+0000 mgr.y (mgr.14544) 152 : cluster [DBG] pgmap v146: 624 pgs: 100 unknown, 32 creating+peering, 492 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.3 KiB/s wr, 4 op/s 2024-02-13T00:41:14.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.606044+0000 mon.a (mon.0) 943 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-84025-10"}]': finished 2024-02-13T00:41:14.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.606107+0000 mon.a (mon.0) 944 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-13T00:41:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.606191+0000 mon.a (mon.0) 945 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.606252+0000 mon.a (mon.0) 946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]': finished 2024-02-13T00:41:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.606306+0000 mon.a (mon.0) 947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]': finished 2024-02-13T00:41:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.606360+0000 mon.a (mon.0) 948 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]': finished 2024-02-13T00:41:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.606473+0000 mon.a (mon.0) 949 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]': finished 2024-02-13T00:41:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.606539+0000 mon.a (mon.0) 950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-83977-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.606623+0000 mon.a (mon.0) 951 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.606760+0000 mon.a (mon.0) 952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:12.606851+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-13T00:41:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:12.207392+0000 mgr.y (mgr.14544) 152 : cluster [DBG] pgmap v146: 624 pgs: 100 unknown, 32 creating+peering, 492 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.3 KiB/s wr, 4 op/s 2024-02-13T00:41:14.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.606044+0000 mon.a (mon.0) 943 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-84025-10"}]': finished 2024-02-13T00:41:14.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.606107+0000 mon.a (mon.0) 944 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-13T00:41:14.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.606191+0000 mon.a (mon.0) 945 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:14.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.606252+0000 mon.a (mon.0) 946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]': finished 2024-02-13T00:41:14.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.606306+0000 mon.a (mon.0) 947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]': finished 2024-02-13T00:41:14.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.606360+0000 mon.a (mon.0) 948 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]': finished 2024-02-13T00:41:14.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.606473+0000 mon.a (mon.0) 949 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]': finished 2024-02-13T00:41:14.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.606539+0000 mon.a (mon.0) 950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-83977-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:14.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.606623+0000 mon.a (mon.0) 951 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:14.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.606760+0000 mon.a (mon.0) 952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:14.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:12.606851+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-13T00:41:14.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.612394+0000 mon.a (mon.0) 954 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:14.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.614827+0000 mon.c (mon.2) 115 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]: dispatch 2024-02-13T00:41:14.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.616685+0000 mon.b (mon.1) 92 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-84105-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:14.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.621141+0000 mon.a (mon.0) 955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-84105-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:14.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.621773+0000 mon.a (mon.0) 956 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]: dispatch 2024-02-13T00:41:14.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.621980+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:14.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.622058+0000 mon.c (mon.2) 116 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-13T00:41:14.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.622426+0000 mon.a (mon.0) 957 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-13T00:41:14.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.627378+0000 mon.a (mon.0) 958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:14.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:12.917379+0000 mon.b (mon.1) 94 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:14.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.612394+0000 mon.a (mon.0) 954 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:14.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.614827+0000 mon.c (mon.2) 115 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]: dispatch 2024-02-13T00:41:14.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.616685+0000 mon.b (mon.1) 92 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-84105-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:14.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.621141+0000 mon.a (mon.0) 955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-84105-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:14.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.621773+0000 mon.a (mon.0) 956 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]: dispatch 2024-02-13T00:41:14.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.621980+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:14.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.622058+0000 mon.c (mon.2) 116 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-13T00:41:14.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.622426+0000 mon.a (mon.0) 957 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-13T00:41:14.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.627378+0000 mon.a (mon.0) 958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:14.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:12.917379+0000 mon.b (mon.1) 94 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:12.207392+0000 mgr.y (mgr.14544) 152 : cluster [DBG] pgmap v146: 624 pgs: 100 unknown, 32 creating+peering, 492 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.3 KiB/s wr, 4 op/s 2024-02-13T00:41:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.606044+0000 mon.a (mon.0) 943 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-84025-10"}]': finished 2024-02-13T00:41:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.606107+0000 mon.a (mon.0) 944 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-13T00:41:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.606191+0000 mon.a (mon.0) 945 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.606252+0000 mon.a (mon.0) 946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-84089-6"}]': finished 2024-02-13T00:41:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.606306+0000 mon.a (mon.0) 947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-84097-7"}]': finished 2024-02-13T00:41:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.606360+0000 mon.a (mon.0) 948 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-84001-22"}]': finished 2024-02-13T00:41:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.606473+0000 mon.a (mon.0) 949 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-83993-15"}]': finished 2024-02-13T00:41:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.606539+0000 mon.a (mon.0) 950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-83977-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.606623+0000 mon.a (mon.0) 951 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.606760+0000 mon.a (mon.0) 952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:12.606851+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.612394+0000 mon.a (mon.0) 954 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-84025-10"}]: dispatch 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.614827+0000 mon.c (mon.2) 115 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]: dispatch 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.616685+0000 mon.b (mon.1) 92 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-84105-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.621141+0000 mon.a (mon.0) 955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-84105-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.621773+0000 mon.a (mon.0) 956 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]: dispatch 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.621980+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.622058+0000 mon.c (mon.2) 116 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.622426+0000 mon.a (mon.0) 957 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.627378+0000 mon.a (mon.0) 958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:14.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:12.917379+0000 mon.b (mon.1) 94 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:14.663 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: seed 84033 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (255 ms) 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (8 ms) 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1052 ms) 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1037 ms) 2024-02-13T00:41:14.664 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (2 ms) 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4 ms) 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (4 ms) 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2365 ms total) 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (343 ms) 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-02-13T00:41:14.665 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (2 ms) 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1039 ms) 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1007 ms) 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (4 ms) 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (4 ms) 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (3 ms) 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (2405 ms total) 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: 2024-02-13T00:41:14.666 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-02-13T00:41:14.667 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (19969 ms total) 2024-02-13T00:41:14.667 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-02-13T00:41:14.773 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:41:14.773 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:41:15.076 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:15.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:15.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:15.211 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:15.229 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:15.232 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.609986+0000 mon.a (mon.0) 959 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-84025-10"}]': finished 2024-02-13T00:41:15.232 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.880472+0000 mon.a (mon.0) 960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]': finished 2024-02-13T00:41:15.232 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.880579+0000 mon.a (mon.0) 961 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-13T00:41:15.232 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.880665+0000 mon.a (mon.0) 962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]': finished 2024-02-13T00:41:15.232 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:13.880878+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-13T00:41:15.232 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.881537+0000 mon.c (mon.2) 117 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "mode": "writeback"}]: dispatch 2024-02-13T00:41:15.232 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.881656+0000 mon.b (mon.1) 95 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:15.232 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.881976+0000 mon.c (mon.2) 118 : audit [INF] from='client.? 172.21.15.162:0/2280980606' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.232 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.883435+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "mode": "writeback"}]: dispatch 2024-02-13T00:41:15.232 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.883872+0000 mon.a (mon.0) 965 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.884084+0000 mon.a (mon.0) 966 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.884133+0000 mon.c (mon.2) 119 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1"}]: dispatch 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.891456+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1"}]: dispatch 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:13.918388+0000 mon.b (mon.1) 96 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:14.207896+0000 mgr.y (mgr.14544) 153 : cluster [DBG] pgmap v149: 456 pgs: 36 unknown, 32 creating+peering, 388 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:14.616784+0000 mon.a (mon.0) 968 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:14.617049+0000 mon.a (mon.0) 969 : cluster [WRN] pool 'PoolQuotaPP_smithi162-83985-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:14.618450+0000 mon.a (mon.0) 970 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:14.618470+0000 mon.a (mon.0) 971 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:14.629643+0000 mon.a (mon.0) 972 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-84105-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]': finished 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:14.629833+0000 mon.a (mon.0) 973 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "mode": "writeback"}]': finished 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:14.629925+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:14.629995+0000 mon.a (mon.0) 975 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]': finished 2024-02-13T00:41:15.233 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:14.630103+0000 mon.a (mon.0) 976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1"}]': finished 2024-02-13T00:41:15.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:14.630236+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-13T00:41:15.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:14.644602+0000 mon.c (mon.2) 120 : audit [INF] from='client.? 172.21.15.162:0/2011180771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-84073-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:14.647854+0000 mon.a (mon.0) 978 : audit [INF] from='client.? 172.21.15.162:0/857244630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-83977-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:14.648027+0000 mon.c (mon.2) 121 : audit [INF] from='client.? 172.21.15.162:0/3165306026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:14.658588+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-84073-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:14.658874+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:14.724575+0000 osd.1 (osd.1) 65 : cluster [DBG] 20.4 deep-scrub starts 2024-02-13T00:41:15.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:14.728243+0000 osd.1 (osd.1) 66 : cluster [DBG] 20.4 deep-scrub ok 2024-02-13T00:41:15.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:14.919236+0000 mon.b (mon.1) 97 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:15.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:15.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.609986+0000 mon.a (mon.0) 959 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-84025-10"}]': finished 2024-02-13T00:41:15.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.880472+0000 mon.a (mon.0) 960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]': finished 2024-02-13T00:41:15.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.880579+0000 mon.a (mon.0) 961 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-13T00:41:15.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.880665+0000 mon.a (mon.0) 962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]': finished 2024-02-13T00:41:15.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:13.880878+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-13T00:41:15.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.881537+0000 mon.c (mon.2) 117 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "mode": "writeback"}]: dispatch 2024-02-13T00:41:15.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.881656+0000 mon.b (mon.1) 95 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:15.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.881976+0000 mon.c (mon.2) 118 : audit [INF] from='client.? 172.21.15.162:0/2280980606' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.883435+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "mode": "writeback"}]: dispatch 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.883872+0000 mon.a (mon.0) 965 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.884084+0000 mon.a (mon.0) 966 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.884133+0000 mon.c (mon.2) 119 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1"}]: dispatch 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.891456+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1"}]: dispatch 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:13.918388+0000 mon.b (mon.1) 96 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:14.207896+0000 mgr.y (mgr.14544) 153 : cluster [DBG] pgmap v149: 456 pgs: 36 unknown, 32 creating+peering, 388 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:14.616784+0000 mon.a (mon.0) 968 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:14.617049+0000 mon.a (mon.0) 969 : cluster [WRN] pool 'PoolQuotaPP_smithi162-83985-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:14.618450+0000 mon.a (mon.0) 970 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:14.618470+0000 mon.a (mon.0) 971 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:14.629643+0000 mon.a (mon.0) 972 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-84105-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]': finished 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:14.629833+0000 mon.a (mon.0) 973 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "mode": "writeback"}]': finished 2024-02-13T00:41:15.282 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:14.629925+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:15.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:14.629995+0000 mon.a (mon.0) 975 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]': finished 2024-02-13T00:41:15.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:14.630103+0000 mon.a (mon.0) 976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1"}]': finished 2024-02-13T00:41:15.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:14.630236+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-13T00:41:15.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:14.644602+0000 mon.c (mon.2) 120 : audit [INF] from='client.? 172.21.15.162:0/2011180771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-84073-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:14.647854+0000 mon.a (mon.0) 978 : audit [INF] from='client.? 172.21.15.162:0/857244630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-83977-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:14.648027+0000 mon.c (mon.2) 121 : audit [INF] from='client.? 172.21.15.162:0/3165306026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:14.658588+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-84073-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:14.658874+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:14.724575+0000 osd.1 (osd.1) 65 : cluster [DBG] 20.4 deep-scrub starts 2024-02-13T00:41:15.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:14.728243+0000 osd.1 (osd.1) 66 : cluster [DBG] 20.4 deep-scrub ok 2024-02-13T00:41:15.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:14.919236+0000 mon.b (mon.1) 97 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:15.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.609986+0000 mon.a (mon.0) 959 : audit [INF] from='client.? 172.21.15.162:0/1973880417' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-84025-10"}]': finished 2024-02-13T00:41:15.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.880472+0000 mon.a (mon.0) 960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]': finished 2024-02-13T00:41:15.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.880579+0000 mon.a (mon.0) 961 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-13T00:41:15.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.880665+0000 mon.a (mon.0) 962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-84033-10"}]': finished 2024-02-13T00:41:15.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:13.880878+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-13T00:41:15.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.881537+0000 mon.c (mon.2) 117 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "mode": "writeback"}]: dispatch 2024-02-13T00:41:15.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.881656+0000 mon.b (mon.1) 95 : audit [INF] from='client.? 172.21.15.162:0/3161060159' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:15.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.881976+0000 mon.c (mon.2) 118 : audit [INF] from='client.? 172.21.15.162:0/2280980606' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.883435+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "mode": "writeback"}]: dispatch 2024-02-13T00:41:15.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.883872+0000 mon.a (mon.0) 965 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.884084+0000 mon.a (mon.0) 966 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]: dispatch 2024-02-13T00:41:15.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.884133+0000 mon.c (mon.2) 119 : audit [INF] from='client.? 172.21.15.162:0/2792481616' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1"}]: dispatch 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.891456+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1"}]: dispatch 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:13.918388+0000 mon.b (mon.1) 96 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:14.207896+0000 mgr.y (mgr.14544) 153 : cluster [DBG] pgmap v149: 456 pgs: 36 unknown, 32 creating+peering, 388 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:14.616784+0000 mon.a (mon.0) 968 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:14.617049+0000 mon.a (mon.0) 969 : cluster [WRN] pool 'PoolQuotaPP_smithi162-83985-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:14.618450+0000 mon.a (mon.0) 970 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:14.618470+0000 mon.a (mon.0) 971 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:14.629643+0000 mon.a (mon.0) 972 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-84105-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]': finished 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:14.629833+0000 mon.a (mon.0) 973 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "mode": "writeback"}]': finished 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:14.629925+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:14.629995+0000 mon.a (mon.0) 975 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-84033-10"}]': finished 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:14.630103+0000 mon.a (mon.0) 976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-84041-1","app":"app1","key":"key1"}]': finished 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:14.630236+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-13T00:41:15.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:14.644602+0000 mon.c (mon.2) 120 : audit [INF] from='client.? 172.21.15.162:0/2011180771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-84073-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.286 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:14.647854+0000 mon.a (mon.0) 978 : audit [INF] from='client.? 172.21.15.162:0/857244630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-83977-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.286 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:14.648027+0000 mon.c (mon.2) 121 : audit [INF] from='client.? 172.21.15.162:0/3165306026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.286 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:14.658588+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-84073-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.286 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:14.658874+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:15.286 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:14.724575+0000 osd.1 (osd.1) 65 : cluster [DBG] 20.4 deep-scrub starts 2024-02-13T00:41:15.286 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:14.728243+0000 osd.1 (osd.1) 66 : cluster [DBG] 20.4 deep-scrub ok 2024-02-13T00:41:15.286 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:14.919236+0000 mon.b (mon.1) 97 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:15.336 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:41:15.361 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:15.379 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:41:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:15.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:41:15.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:15.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:41:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:41:15.509 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:15.528 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:15.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:15.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:15.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:15.671 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:15.695 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:15.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:15.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:15.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:15.831 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:16.033 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:16.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:16.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:16.170 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:16.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:41:16] "GET /metrics HTTP/1.1" 200 214531 "" "Prometheus/2.33.4" 2024-02-13T00:41:16.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-13T00:41:16.544 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:16.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:16.566 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:16.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:16.711 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:16.729 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:16.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:16.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:16.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:16.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:15.328547+0000 mgr.y (mgr.14544) 154 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:16.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:15.652239+0000 mon.a (mon.0) 981 : audit [INF] from='client.? 172.21.15.162:0/857244630' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-83977-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:16.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:15.652358+0000 mon.a (mon.0) 982 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-84073-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:16.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:15.652418+0000 mon.a (mon.0) 983 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:16.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:15.652554+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-13T00:41:16.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:15.667212+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:16.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:15.680821+0000 mon.a (mon.0) 985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:16.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:15.924104+0000 mon.b (mon.1) 99 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:16.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:15.328547+0000 mgr.y (mgr.14544) 154 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:16.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:15.652239+0000 mon.a (mon.0) 981 : audit [INF] from='client.? 172.21.15.162:0/857244630' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-83977-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:16.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:15.652358+0000 mon.a (mon.0) 982 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-84073-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:16.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:15.652418+0000 mon.a (mon.0) 983 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:16.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:15.652554+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-13T00:41:16.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:15.667212+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:16.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:15.680821+0000 mon.a (mon.0) 985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:16.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:15.924104+0000 mon.b (mon.1) 99 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:16.860 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:16.878 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:41:16.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:16.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:41:16.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:15.328547+0000 mgr.y (mgr.14544) 154 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:15.652239+0000 mon.a (mon.0) 981 : audit [INF] from='client.? 172.21.15.162:0/857244630' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-83977-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:15.652358+0000 mon.a (mon.0) 982 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-84073-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:15.652418+0000 mon.a (mon.0) 983 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:15.652554+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-13T00:41:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:15.667212+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:15.680821+0000 mon.a (mon.0) 985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:15.924104+0000 mon.b (mon.1) 99 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:17.059 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:17.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:17.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:17.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:17.234 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:17.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.528+0000 7f8f70e3a700 1 -- 172.21.15.162:0/129338931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f6c0f73d0 msgr2=0x7f8f6c0fbb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:17.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.528+0000 7f8f70e3a700 1 --2- 172.21.15.162:0/129338931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f6c0f73d0 0x7f8f6c0fbb50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f8f600047b0 tx=0x7f8f60009cb0).stop 2024-02-13T00:41:17.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.528+0000 7f8f70e3a700 1 -- 172.21.15.162:0/129338931 shutdown_connections 2024-02-13T00:41:17.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.528+0000 7f8f70e3a700 1 --2- 172.21.15.162:0/129338931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f6c0fc040 0x7f8f6c0fc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:17.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.528+0000 7f8f70e3a700 1 --2- 172.21.15.162:0/129338931 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f6c0f6b00 0x7f8f6c0f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:17.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.528+0000 7f8f70e3a700 1 --2- 172.21.15.162:0/129338931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f6c0f73d0 0x7f8f6c0fbb50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:17.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.528+0000 7f8f70e3a700 1 -- 172.21.15.162:0/129338931 >> 172.21.15.162:0/129338931 conn(0x7f8f6c06eb90 msgr2=0x7f8f6c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:17.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.528+0000 7f8f70e3a700 1 -- 172.21.15.162:0/129338931 shutdown_connections 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.528+0000 7f8f70e3a700 1 -- 172.21.15.162:0/129338931 wait complete. 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.529+0000 7f8f70e3a700 1 librados: starting msgr at 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.529+0000 7f8f70e3a700 1 librados: starting objecter 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.529+0000 7f8f70e3a700 1 Processor -- start 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.529+0000 7f8f70e3a700 1 -- start start 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.529+0000 7f8f70e3a700 1 librados: setting wanted keys 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.529+0000 7f8f70e3a700 1 librados: calling monclient init 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.529+0000 7f8f70e3a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f6c0f6b00 0x7f8f6c06a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.529+0000 7f8f70e3a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f6c0f73d0 0x7f8f6c06a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.529+0000 7f8f70e3a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f6c0fc040 0x7f8f6c063340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.529+0000 7f8f70e3a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c063830 con 0x7f8f6c0f6b00 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.529+0000 7f8f70e3a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0639a0 con 0x7f8f6c0f73d0 2024-02-13T00:41:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.529+0000 7f8f70e3a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c063b60 con 0x7f8f6c0fc040 2024-02-13T00:41:17.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f6ad9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f6c0fc040 0x7f8f6c063340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:17.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f6ad9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f6c0fc040 0x7f8f6c063340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:35322/0 (socket says 172.21.15.162:35322) 2024-02-13T00:41:17.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f6ad9d700 1 -- 172.21.15.162:0/2355755726 learned_addr learned my addr 172.21.15.162:0/2355755726 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:17.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f69d9b700 1 --2- 172.21.15.162:0/2355755726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f6c0f73d0 0x7f8f6c06a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:17.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f6a59c700 1 --2- 172.21.15.162:0/2355755726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f6c0f6b00 0x7f8f6c06a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:17.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f6ad9d700 1 -- 172.21.15.162:0/2355755726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f6c0f73d0 msgr2=0x7f8f6c06a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:17.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f6ad9d700 1 --2- 172.21.15.162:0/2355755726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f6c0f73d0 0x7f8f6c06a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:17.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f6ad9d700 1 -- 172.21.15.162:0/2355755726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f6c0f6b00 msgr2=0x7f8f6c06a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:17.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f6ad9d700 1 --2- 172.21.15.162:0/2355755726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f6c0f6b00 0x7f8f6c06a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:17.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f6ad9d700 1 -- 172.21.15.162:0/2355755726 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f640097c0 con 0x7f8f6c0fc040 2024-02-13T00:41:17.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f69d9b700 1 --2- 172.21.15.162:0/2355755726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f6c0f73d0 0x7f8f6c06a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:41:17.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f6a59c700 1 --2- 172.21.15.162:0/2355755726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f6c0f6b00 0x7f8f6c06a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:41:17.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f6ad9d700 1 --2- 172.21.15.162:0/2355755726 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f6c0fc040 0x7f8f6c063340 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f8f64001e80 tx=0x7f8f64004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:17.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f5b7fe700 10 client.?.objecter ms_handle_connect 0x7f8f6c0fc040 2024-02-13T00:41:17.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.530+0000 7f8f5b7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:17.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.531+0000 7f8f5b7fe700 1 -- 172.21.15.162:0/2355755726 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f64009ac0 con 0x7f8f6c0fc040 2024-02-13T00:41:17.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.531+0000 7f8f5b7fe700 1 -- 172.21.15.162:0/2355755726 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f640095a0 con 0x7f8f6c0fc040 2024-02-13T00:41:17.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.531+0000 7f8f70e3a700 1 -- 172.21.15.162:0/2355755726 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f60005040 con 0x7f8f6c0fc040 2024-02-13T00:41:17.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.531+0000 7f8f70e3a700 10 client.26174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:17.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.531+0000 7f8f70e3a700 1 -- 172.21.15.162:0/2355755726 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c067dd0 con 0x7f8f6c0fc040 2024-02-13T00:41:17.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.531+0000 7f8f70e3a700 1 librados: init done 2024-02-13T00:41:17.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.532+0000 7f8f70e3a700 1 -- 172.21.15.162:0/2355755726 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6c0475e0 con 0x7f8f6c0fc040 2024-02-13T00:41:17.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.532+0000 7f8f5b7fe700 1 -- 172.21.15.162:0/2355755726 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f64007dc0 con 0x7f8f6c0fc040 2024-02-13T00:41:17.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.536+0000 7f8f5b7fe700 1 -- 172.21.15.162:0/2355755726 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f8f6400fbe0 con 0x7f8f6c0fc040 2024-02-13T00:41:17.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.536+0000 7f8f5b7fe700 1 --2- 172.21.15.162:0/2355755726 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8f54062c90 0x7f8f54065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:17.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.536+0000 7f8f5b7fe700 1 -- 172.21.15.162:0/2355755726 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(122..122 src has 52..122) v4 ==== 11494+0+0 (secure 0 0 0) 0x7f8f6407f1e0 con 0x7f8f6c0fc040 2024-02-13T00:41:17.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.536+0000 7f8f5b7fe700 10 client.26174.objecter ms_dispatch 0x7f8f6c001550 osd_map(122..122 src has 52..122) v4 2024-02-13T00:41:17.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.536+0000 7f8f5b7fe700 3 client.26174.objecter handle_osd_map got epochs [122,122] > 0 2024-02-13T00:41:17.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.536+0000 7f8f5b7fe700 3 client.26174.objecter handle_osd_map decoding full epoch 122 2024-02-13T00:41:17.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.536+0000 7f8f5b7fe700 10 client.26174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:17.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.536+0000 7f8f5b7fe700 1 -- 172.21.15.162:0/2355755726 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=123}) v3 -- 0x7f8f54076640 con 0x7f8f6c0fc040 2024-02-13T00:41:17.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.536+0000 7f8f5b7fe700 20 client.26174.objecter dump_active .. 0 homeless 2024-02-13T00:41:17.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.537+0000 7f8f6a59c700 1 --2- 172.21.15.162:0/2355755726 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8f54062c90 0x7f8f54065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:17.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.538+0000 7f8f6a59c700 1 --2- 172.21.15.162:0/2355755726 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8f54062c90 0x7f8f54065150 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f8f5c009990 tx=0x7f8f5c008040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:17.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.538+0000 7f8f5b7fe700 10 client.26174.objecter ms_handle_connect 0x7f8f54062c90 2024-02-13T00:41:17.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.541+0000 7f8f5b7fe700 1 -- 172.21.15.162:0/2355755726 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8f64051940 con 0x7f8f6c0fc040 2024-02-13T00:41:17.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.782+0000 7f8f5b7fe700 1 -- 172.21.15.162:0/2355755726 <== mon.2 v2:172.21.15.162:3301/0 7 ==== osd_map(123..123 src has 52..123) v4 ==== 2529+0+0 (secure 0 0 0) 0x7f8f6401c300 con 0x7f8f6c0fc040 2024-02-13T00:41:17.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.782+0000 7f8f5b7fe700 10 client.26174.objecter ms_dispatch 0x7f8f6c001550 osd_map(123..123 src has 52..123) v4 2024-02-13T00:41:17.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.782+0000 7f8f5b7fe700 3 client.26174.objecter handle_osd_map got epochs [123,123] > 122 2024-02-13T00:41:17.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.782+0000 7f8f5b7fe700 3 client.26174.objecter handle_osd_map decoding incremental epoch 123 2024-02-13T00:41:17.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.782+0000 7f8f5b7fe700 10 client.26174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:17.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.782+0000 7f8f5b7fe700 1 -- 172.21.15.162:0/2355755726 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=124}) v3 -- 0x7f8f5406b540 con 0x7f8f6c0fc040 2024-02-13T00:41:17.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.782+0000 7f8f5b7fe700 20 client.26174.objecter dump_active .. 0 homeless 2024-02-13T00:41:17.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1118 ms) 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (6 ms) 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1124 ms total) 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-02-13T00:41:17.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (70 ms) 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4029 ms) 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (6 ms) 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (7 ms) 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94779274301856 notify_id 515396075520 notifier_gid 25049 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (37 ms) 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94779274301856 notify_id 515396075521 notifier_gid 25049 2024-02-13T00:41:17.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (19 ms) 2024-02-13T00:41:17.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-02-13T00:41:17.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94779274301856 notify_id 515396075522 notifier_gid 25049 2024-02-13T00:41:17.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (11 ms) 2024-02-13T00:41:17.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-02-13T00:41:17.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94779274301856 notify_id 515396075523 notifier_gid 25049 2024-02-13T00:41:17.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (15 ms) 2024-02-13T00:41:17.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-02-13T00:41:17.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94779274301856 notify_id 515396075524 notifier_gid 25049 2024-02-13T00:41:17.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (6 ms) 2024-02-13T00:41:17.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-02-13T00:41:17.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94779274301856 notify_id 515396075521 notifier_gid 25049 2024-02-13T00:41:17.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (22 ms) 2024-02-13T00:41:17.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-02-13T00:41:17.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: trying... 2024-02-13T00:41:17.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94779274301856 notify_id 515396075525 notifier_gid 25049 2024-02-13T00:41:17.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: timed out 2024-02-13T00:41:17.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: flushing 2024-02-13T00:41:17.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:17.894+0000 7f8f70e3a700 1 -- 172.21.15.162:0/2355755726 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f8f6c0641f0 con 0x7f8f6c0fc040 2024-02-13T00:41:18.105 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:16.209281+0000 mgr.y (mgr.14544) 155 : cluster [DBG] pgmap v152: 528 pgs: 104 creating+peering, 32 unknown, 392 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 5.0 KiB/s wr, 2 op/s 2024-02-13T00:41:18.105 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:16.220087+0000 osd.7 (osd.7) 70 : cluster [DBG] 20.9 deep-scrub starts 2024-02-13T00:41:18.105 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:16.226993+0000 osd.7 (osd.7) 71 : cluster [DBG] 20.9 deep-scrub ok 2024-02-13T00:41:18.105 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:16.657693+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:18.105 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:16.657899+0000 mon.a (mon.0) 987 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-13T00:41:18.105 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:16.670118+0000 mon.b (mon.1) 100 : audit [INF] from='client.? 172.21.15.162:0/4148338735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:18.105 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:16.677540+0000 mon.b (mon.1) 101 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:18.105 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:16.688053+0000 mon.a (mon.0) 988 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:18.105 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:16.688326+0000 mon.a (mon.0) 989 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:18.105 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:16.927034+0000 mon.b (mon.1) 102 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:18.106 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:16.209281+0000 mgr.y (mgr.14544) 155 : cluster [DBG] pgmap v152: 528 pgs: 104 creating+peering, 32 unknown, 392 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 5.0 KiB/s wr, 2 op/s 2024-02-13T00:41:18.106 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:16.220087+0000 osd.7 (osd.7) 70 : cluster [DBG] 20.9 deep-scrub starts 2024-02-13T00:41:18.106 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:16.226993+0000 osd.7 (osd.7) 71 : cluster [DBG] 20.9 deep-scrub ok 2024-02-13T00:41:18.106 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:16.657693+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:18.106 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:16.657899+0000 mon.a (mon.0) 987 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-13T00:41:18.106 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:16.670118+0000 mon.b (mon.1) 100 : audit [INF] from='client.? 172.21.15.162:0/4148338735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:18.106 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:16.677540+0000 mon.b (mon.1) 101 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:18.106 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:16.688053+0000 mon.a (mon.0) 988 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:18.107 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:16.688326+0000 mon.a (mon.0) 989 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:18.107 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:16.927034+0000 mon.b (mon.1) 102 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:18.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:16.209281+0000 mgr.y (mgr.14544) 155 : cluster [DBG] pgmap v152: 528 pgs: 104 creating+peering, 32 unknown, 392 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 5.0 KiB/s wr, 2 op/s 2024-02-13T00:41:18.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:16.220087+0000 osd.7 (osd.7) 70 : cluster [DBG] 20.9 deep-scrub starts 2024-02-13T00:41:18.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:16.226993+0000 osd.7 (osd.7) 71 : cluster [DBG] 20.9 deep-scrub ok 2024-02-13T00:41:18.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:16.657693+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:18.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:16.657899+0000 mon.a (mon.0) 987 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-13T00:41:18.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:16.670118+0000 mon.b (mon.1) 100 : audit [INF] from='client.? 172.21.15.162:0/4148338735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:18.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:16.677540+0000 mon.b (mon.1) 101 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:18.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:16.688053+0000 mon.a (mon.0) 988 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:18.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:16.688326+0000 mon.a (mon.0) 989 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:18.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:16.927034+0000 mon.b (mon.1) 102 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:18.322 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:18.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:18.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:18.397 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:18.416 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:18.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:18.515 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:18.608 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:18.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:18.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:18.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.786+0000 7f8f5b7fe700 1 -- 172.21.15.162:0/2355755726 <== mon.2 v2:172.21.15.162:3301/0 8 ==== osd_map(124..124 src has 52..124) v4 ==== 3085+0+0 (secure 0 0 0) 0x7f8f64051b70 con 0x7f8f6c0fc040 2024-02-13T00:41:18.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.786+0000 7f8f5b7fe700 10 client.26174.objecter ms_dispatch 0x7f8f6c001550 osd_map(124..124 src has 52..124) v4 2024-02-13T00:41:18.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.786+0000 7f8f5b7fe700 3 client.26174.objecter handle_osd_map got epochs [124,124] > 123 2024-02-13T00:41:18.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.786+0000 7f8f5b7fe700 3 client.26174.objecter handle_osd_map decoding incremental epoch 124 2024-02-13T00:41:18.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.786+0000 7f8f5b7fe700 10 client.26174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:18.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.787+0000 7f8f5b7fe700 1 -- 172.21.15.162:0/2355755726 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=125}) v3 -- 0x7f8f54071590 con 0x7f8f6c0fc040 2024-02-13T00:41:18.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.787+0000 7f8f5b7fe700 20 client.26174.objecter dump_active .. 0 homeless 2024-02-13T00:41:18.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.789+0000 7f8f5b7fe700 1 -- 172.21.15.162:0/2355755726 <== mon.2 v2:172.21.15.162:3301/0 9 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v124) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8f6405c090 con 0x7f8f6c0fc040 2024-02-13T00:41:18.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.792+0000 7f8f70e3a700 10 librados: watch_flush enter 2024-02-13T00:41:18.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.792+0000 7f8f70e3a700 10 librados: watch_flush exit 2024-02-13T00:41:18.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.792+0000 7f8f70e3a700 20 client.26174.objecter shutdown clearing up homeless session... 2024-02-13T00:41:18.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.792+0000 7f8f70e3a700 10 client.26174.objecter successfully canceled tick 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.793+0000 7f8f70e3a700 1 -- 172.21.15.162:0/2355755726 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8f54062c90 msgr2=0x7f8f54065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.793+0000 7f8f70e3a700 1 --2- 172.21.15.162:0/2355755726 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8f54062c90 0x7f8f54065150 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f8f5c009990 tx=0x7f8f5c008040).stop 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.793+0000 7f8f70e3a700 1 -- 172.21.15.162:0/2355755726 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f6c0fc040 msgr2=0x7f8f6c063340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.793+0000 7f8f70e3a700 1 --2- 172.21.15.162:0/2355755726 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f6c0fc040 0x7f8f6c063340 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f8f64001e80 tx=0x7f8f64004d10).stop 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.793+0000 7f8f70e3a700 1 -- 172.21.15.162:0/2355755726 shutdown_connections 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.793+0000 7f8f70e3a700 1 --2- 172.21.15.162:0/2355755726 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8f54062c90 0x7f8f54065150 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.793+0000 7f8f70e3a700 1 --2- 172.21.15.162:0/2355755726 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f6c0fc040 0x7f8f6c063340 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.793+0000 7f8f70e3a700 1 --2- 172.21.15.162:0/2355755726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f6c0f6b00 0x7f8f6c06a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.793+0000 7f8f70e3a700 1 --2- 172.21.15.162:0/2355755726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f6c0f73d0 0x7f8f6c06a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.793+0000 7f8f70e3a700 1 -- 172.21.15.162:0/2355755726 >> 172.21.15.162:0/2355755726 conn(0x7f8f6c06eb90 msgr2=0x7f8f6c0f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.794+0000 7f8f70e3a700 1 -- 172.21.15.162:0/2355755726 shutdown_connections 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.794+0000 7f8f70e3a700 1 -- 172.21.15.162:0/2355755726 wait complete. 2024-02-13T00:41:18.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:18.794+0000 7f8f70e3a700 1 librados: shutdown 2024-02-13T00:41:18.803 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:41:18.827 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:18.890 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:41:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:41:18.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:17.771798+0000 mon.a (mon.0) 990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:18.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:17.771879+0000 mon.a (mon.0) 991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]': finished 2024-02-13T00:41:18.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:17.772037+0000 mon.a (mon.0) 992 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-13T00:41:18.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:17.784041+0000 mon.b (mon.1) 103 : audit [INF] from='client.? 172.21.15.162:0/25871415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-83977-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:18.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:17.785043+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.162:0/1595084992' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:18.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:17.785158+0000 mon.b (mon.1) 105 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:18.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:17.785332+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:18.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:17.812152+0000 mon.a (mon.0) 993 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-83977-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:18.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:17.812377+0000 mon.a (mon.0) 994 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:18.895 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:17.812544+0000 mon.a (mon.0) 995 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:18.895 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:17.820216+0000 mon.a (mon.0) 996 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:18.903 INFO:tasks.workunit.client.0.smithi162.stdout: api_ api_watch_notify: Running main() from gmock_main.cc 2024-02-13T00:41:18.903 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-02-13T00:41:18.903 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-02-13T00:41:18.903 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-02-13T00:41:18.903 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-02-13T00:41:18.903 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify_test_cb 2024-02-13T00:41:18.904 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (231 ms) 2024-02-13T00:41:18.904 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-02-13T00:41:18.904 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94276611772496 err -107 2024-02-13T00:41:18.904 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-02-13T00:41:18.904 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (29 ms) 2024-02-13T00:41:18.904 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-02-13T00:41:18.904 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94276611772496 err -107 2024-02-13T00:41:18.904 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-02-13T00:41:18.904 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (5 ms) 2024-02-13T00:41:18.904 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-02-13T00:41:18.904 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24773 notify_id 459561500672 cookie 94276611772496 2024-02-13T00:41:18.905 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (8 ms) 2024-02-13T00:41:18.905 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-02-13T00:41:18.905 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24773 notify_id 459561500672 cookie 94276612587744 2024-02-13T00:41:18.905 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (12 ms) 2024-02-13T00:41:18.905 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-02-13T00:41:18.905 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24773 notify_id 459561500673 cookie 94276612587744 2024-02-13T00:41:18.905 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (7 ms) 2024-02-13T00:41:18.905 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-02-13T00:41:18.905 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24773 notify_id 459561500673 cookie 94276612587744 2024-02-13T00:41:18.905 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24773 notify_id 459561500673 cookie 94276612599280 2024-02-13T00:41:18.905 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (21 ms) 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24773 notify_id 459561500673 cookie 94276612612544 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24773 notify_id 463856467970 cookie 94276612612544 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3025 ms) 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94276612612544 err -107 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24773 notify_id 493921239041 cookie 94276612612544 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5008 ms) 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94276612612544 err -107 2024-02-13T00:41:18.906 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2024-02-13T00:41:18.907 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9352 ms total) 2024-02-13T00:41:18.907 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: 2024-02-13T00:41:18.907 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-02-13T00:41:18.907 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-02-13T00:41:18.907 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify_test_cb 2024-02-13T00:41:18.907 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1123 ms) 2024-02-13T00:41:18.907 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1123 ms total) 2024-02-13T00:41:18.907 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: 2024-02-13T00:41:18.908 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-02-13T00:41:18.908 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (24209 ms total) 2024-02-13T00:41:18.908 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-02-13T00:41:19.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:17.905045+0000 mon.c (mon.2) 122 : audit [INF] from='client.? 172.21.15.162:0/2355755726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:41:19.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:17.928667+0000 mon.b (mon.1) 107 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:19.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:18.211346+0000 mon.a (mon.0) 997 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:41:19.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:18.768257+0000 mon.a (mon.0) 998 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:19.266 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:18.785128+0000 mon.a (mon.0) 999 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-83977-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:19.266 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:18.785221+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:19.266 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:18.785276+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]': finished 2024-02-13T00:41:19.266 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:18.785345+0000 mon.a (mon.0) 1002 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:19.266 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:18.785496+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:41:19.266 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:18.785552+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-13T00:41:19.267 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:18.788836+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? 172.21.15.162:0/238911824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-84081-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.269 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:17.771798+0000 mon.a (mon.0) 990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:19.269 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:17.771879+0000 mon.a (mon.0) 991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]': finished 2024-02-13T00:41:19.269 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:17.772037+0000 mon.a (mon.0) 992 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-13T00:41:19.269 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:17.784041+0000 mon.b (mon.1) 103 : audit [INF] from='client.? 172.21.15.162:0/25871415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-83977-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.269 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:17.785043+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.162:0/1595084992' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.269 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:17.785158+0000 mon.b (mon.1) 105 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:19.269 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:17.785332+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.269 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:17.812152+0000 mon.a (mon.0) 993 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-83977-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.269 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:17.812377+0000 mon.a (mon.0) 994 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.270 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:17.812544+0000 mon.a (mon.0) 995 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:19.270 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:17.820216+0000 mon.a (mon.0) 996 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.270 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:17.905045+0000 mon.c (mon.2) 122 : audit [INF] from='client.? 172.21.15.162:0/2355755726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:41:19.270 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:17.928667+0000 mon.b (mon.1) 107 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:19.270 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:18.211346+0000 mon.a (mon.0) 997 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:41:19.270 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:18.768257+0000 mon.a (mon.0) 998 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:19.270 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:18.785128+0000 mon.a (mon.0) 999 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-83977-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:19.270 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:18.785221+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:19.270 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:18.785276+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]': finished 2024-02-13T00:41:19.270 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:18.785345+0000 mon.a (mon.0) 1002 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:19.271 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:18.785496+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:41:19.271 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:18.785552+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-13T00:41:19.271 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:18.788836+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? 172.21.15.162:0/238911824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-84081-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:17.771798+0000 mon.a (mon.0) 990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:17.771879+0000 mon.a (mon.0) 991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-84105-12"}]': finished 2024-02-13T00:41:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:17.772037+0000 mon.a (mon.0) 992 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-13T00:41:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:17.784041+0000 mon.b (mon.1) 103 : audit [INF] from='client.? 172.21.15.162:0/25871415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-83977-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:17.785043+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.162:0/1595084992' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:17.785158+0000 mon.b (mon.1) 105 : audit [INF] from='client.? 172.21.15.162:0/1363641633' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:17.785332+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:17.812152+0000 mon.a (mon.0) 993 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-83977-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:17.812377+0000 mon.a (mon.0) 994 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:17.812544+0000 mon.a (mon.0) 995 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]: dispatch 2024-02-13T00:41:19.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:17.820216+0000 mon.a (mon.0) 996 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:17.905045+0000 mon.c (mon.2) 122 : audit [INF] from='client.? 172.21.15.162:0/2355755726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:41:19.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:17.928667+0000 mon.b (mon.1) 107 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:19.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:18.211346+0000 mon.a (mon.0) 997 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:41:19.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:18.768257+0000 mon.a (mon.0) 998 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:19.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:18.785128+0000 mon.a (mon.0) 999 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-83977-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:19.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:18.785221+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:19.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:18.785276+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-84105-12"}]': finished 2024-02-13T00:41:19.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:18.785345+0000 mon.a (mon.0) 1002 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:19.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:18.785496+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:41:19.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:18.785552+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-13T00:41:19.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:18.788836+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? 172.21.15.162:0/238911824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-84081-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:19.797 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:19.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:19.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:19.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:20.036 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:20.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:18.215302+0000 mgr.y (mgr.14544) 156 : cluster [DBG] pgmap v155: 552 pgs: 32 creating+peering, 160 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1022 B/s rd, 5.7 KiB/s wr, 4 op/s 2024-02-13T00:41:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:18.930294+0000 mon.b (mon.1) 108 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:19.108877+0000 mon.b (mon.1) 109 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:19.470112+0000 mon.a (mon.0) 1006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:19.477004+0000 mon.a (mon.0) 1007 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:41:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:19.627245+0000 mon.a (mon.0) 1008 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:19.634230+0000 mon.a (mon.0) 1009 : audit [INF] from='client.? 172.21.15.162:0/238911824' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-84081-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:19.634356+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:19.634409+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-13T00:41:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:19.639713+0000 mon.c (mon.2) 123 : audit [INF] from='client.? 172.21.15.162:0/798911275' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:19.647545+0000 mon.b (mon.1) 110 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-7"}]: dispatch 2024-02-13T00:41:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:19.650369+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:20.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:18.215302+0000 mgr.y (mgr.14544) 156 : cluster [DBG] pgmap v155: 552 pgs: 32 creating+peering, 160 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1022 B/s rd, 5.7 KiB/s wr, 4 op/s 2024-02-13T00:41:20.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:18.930294+0000 mon.b (mon.1) 108 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:20.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:19.108877+0000 mon.b (mon.1) 109 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:20.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:19.470112+0000 mon.a (mon.0) 1006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:20.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:19.477004+0000 mon.a (mon.0) 1007 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:41:20.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:19.627245+0000 mon.a (mon.0) 1008 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:20.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:19.634230+0000 mon.a (mon.0) 1009 : audit [INF] from='client.? 172.21.15.162:0/238911824' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-84081-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:20.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:19.634356+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:20.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:19.634409+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-13T00:41:20.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:19.639713+0000 mon.c (mon.2) 123 : audit [INF] from='client.? 172.21.15.162:0/798911275' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:20.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:19.647545+0000 mon.b (mon.1) 110 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-7"}]: dispatch 2024-02-13T00:41:20.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:19.650369+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:20.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:18.215302+0000 mgr.y (mgr.14544) 156 : cluster [DBG] pgmap v155: 552 pgs: 32 creating+peering, 160 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1022 B/s rd, 5.7 KiB/s wr, 4 op/s 2024-02-13T00:41:20.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:18.930294+0000 mon.b (mon.1) 108 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:20.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:19.108877+0000 mon.b (mon.1) 109 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:20.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:19.470112+0000 mon.a (mon.0) 1006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:20.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:19.477004+0000 mon.a (mon.0) 1007 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:41:20.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:19.627245+0000 mon.a (mon.0) 1008 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:20.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:19.634230+0000 mon.a (mon.0) 1009 : audit [INF] from='client.? 172.21.15.162:0/238911824' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-84081-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:20.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:19.634356+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:20.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:19.634409+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-13T00:41:20.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:19.639713+0000 mon.c (mon.2) 123 : audit [INF] from='client.? 172.21.15.162:0/798911275' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:20.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:19.647545+0000 mon.b (mon.1) 110 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-7"}]: dispatch 2024-02-13T00:41:20.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:19.650369+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.277+0000 7fc2644d6700 1 -- 172.21.15.162:0/276599754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc25c104c10 msgr2=0x7fc25c0febb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:21.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.277+0000 7fc2644d6700 1 --2- 172.21.15.162:0/276599754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc25c104c10 0x7fc25c0febb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7fc24c0047b0 tx=0x7fc24c009cb0).stop 2024-02-13T00:41:21.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.277+0000 7fc2644d6700 1 -- 172.21.15.162:0/276599754 shutdown_connections 2024-02-13T00:41:21.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.277+0000 7fc2644d6700 1 --2- 172.21.15.162:0/276599754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c0ff240 0x7fc25c0ff6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:21.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.277+0000 7fc2644d6700 1 --2- 172.21.15.162:0/276599754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc25c104340 0x7fc25c104720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:21.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.277+0000 7fc2644d6700 1 --2- 172.21.15.162:0/276599754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc25c104c10 0x7fc25c0febb0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:21.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.277+0000 7fc2644d6700 1 -- 172.21.15.162:0/276599754 >> 172.21.15.162:0/276599754 conn(0x7fc25c06dae0 msgr2=0x7fc25c06ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:21.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.277+0000 7fc2644d6700 1 -- 172.21.15.162:0/276599754 shutdown_connections 2024-02-13T00:41:21.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.277+0000 7fc2644d6700 1 -- 172.21.15.162:0/276599754 wait complete. 2024-02-13T00:41:21.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.277+0000 7fc2644d6700 1 librados: starting msgr at 2024-02-13T00:41:21.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.277+0000 7fc2644d6700 1 librados: starting objecter 2024-02-13T00:41:21.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc2644d6700 1 Processor -- start 2024-02-13T00:41:21.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc2644d6700 1 -- start start 2024-02-13T00:41:21.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc2644d6700 1 librados: setting wanted keys 2024-02-13T00:41:21.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc2644d6700 1 librados: calling monclient init 2024-02-13T00:41:21.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc2644d6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc25c0ff240 0x7fc25c075dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:21.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc2644d6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c104340 0x7fc25c0762f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:21.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc2644d6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc25c104c10 0x7fc25c072110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:21.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc2644d6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c072600 con 0x7fc25c104340 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc2644d6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c072770 con 0x7fc25c104c10 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc2644d6700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c072930 con 0x7fc25c0ff240 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc261a71700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c104340 0x7fc25c0762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc261a71700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c104340 0x7fc25c0762f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:56122/0 (socket says 172.21.15.162:56122) 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.278+0000 7fc261a71700 1 -- 172.21.15.162:0/944439755 learned_addr learned my addr 172.21.15.162:0/944439755 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc261a71700 1 -- 172.21.15.162:0/944439755 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc25c0ff240 msgr2=0x7fc25c075dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc261a71700 1 --2- 172.21.15.162:0/944439755 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc25c0ff240 0x7fc25c075dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc261a71700 1 -- 172.21.15.162:0/944439755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc25c104c10 msgr2=0x7fc25c072110 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc261a71700 1 --2- 172.21.15.162:0/944439755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc25c104c10 0x7fc25c072110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc261a71700 1 -- 172.21.15.162:0/944439755 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2540097c0 con 0x7fc25c104340 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc261a71700 1 --2- 172.21.15.162:0/944439755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c104340 0x7fc25c0762f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7fc24c009eb0 tx=0x7fc24c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc2537fe700 10 client.?.objecter ms_handle_connect 0x7fc25c104340 2024-02-13T00:41:21.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc2537fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:21.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c018070 con 0x7fc25c104340 2024-02-13T00:41:21.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc24c005040 con 0x7fc25c104340 2024-02-13T00:41:21.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc2644d6700 10 client.17616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:21.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25c072dc0 con 0x7fc25c104340 2024-02-13T00:41:21.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc2644d6700 1 librados: init done 2024-02-13T00:41:21.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc24c013e70 con 0x7fc25c104340 2024-02-13T00:41:21.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.279+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c01dcf0 con 0x7fc25c104340 2024-02-13T00:41:21.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.280+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc240004f30 con 0x7fc25c104340 2024-02-13T00:41:21.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.281+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc24c01d420 con 0x7fc25c104340 2024-02-13T00:41:21.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.281+0000 7fc2537fe700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc248062c90 0x7fc248065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:21.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.281+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(126..126 src has 52..126) v4 ==== 11859+0+0 (secure 0 0 0) 0x7fc24c07fa40 con 0x7fc25c104340 2024-02-13T00:41:21.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.281+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 osd_map(126..126 src has 52..126) v4 2024-02-13T00:41:21.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.281+0000 7fc2537fe700 3 client.17616.objecter handle_osd_map got epochs [126,126] > 0 2024-02-13T00:41:21.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.281+0000 7fc2537fe700 3 client.17616.objecter handle_osd_map decoding full epoch 126 2024-02-13T00:41:21.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.281+0000 7fc2537fe700 20 client.17616.objecter dump_active .. 0 homeless 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.281+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v126) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc24c0520c0 con 0x7fc25c104340 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.281+0000 7fc262272700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc248062c90 0x7fc248065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2644d6700 10 client.17616.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 1 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc240004560 0x7fc240004980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc2400051d0 osd=0 3 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc2400051d0 osd=0 3 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc2400051d0 osd=0 3 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 1 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc2400051d0 osd=0 3 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 0 1 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc2400051d0 osd=0 4 2024-02-13T00:41:21.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2644d6700 10 client.17616.objecter _send_command 1 2024-02-13T00:41:21.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc240008960 con 0x7fc240004560 2024-02-13T00:41:21.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc262272700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc248062c90 0x7fc248065150 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fc258004b90 tx=0x7fc258009b60).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:21.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2537fe700 10 client.17616.objecter ms_handle_connect 0x7fc248062c90 2024-02-13T00:41:21.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc262a73700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc240004560 0x7fc240004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:21.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc262a73700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc240004560 0x7fc240004980 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:21.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.282+0000 7fc2537fe700 10 client.17616.objecter ms_handle_connect 0x7fc240004560 2024-02-13T00:41:21.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.283+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc240008960 con 0x7fc240004560 2024-02-13T00:41:21.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.283+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:41:21.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.283+0000 7fc2537fe700 10 client.17616.objecter _finish_command 1 = system:0 2024-02-13T00:41:21.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.283+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc2400051d0 osd=0 4 2024-02-13T00:41:21.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.283+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 0 1 2024-02-13T00:41:21.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.308+0000 7fc2644d6700 10 client.17616.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:21.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.308+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 2 2024-02-13T00:41:21.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.308+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc2400051d0 osd=0 3 2024-02-13T00:41:21.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.308+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc2400051d0 osd=0 3 2024-02-13T00:41:21.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.308+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc2400051d0 osd=0 3 2024-02-13T00:41:21.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.308+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 2 2024-02-13T00:41:21.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.308+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc2400051d0 osd=0 3 2024-02-13T00:41:21.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.308+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 0 2 2024-02-13T00:41:21.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.308+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc2400051d0 osd=0 4 2024-02-13T00:41:21.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.308+0000 7fc2644d6700 10 client.17616.objecter _send_command 2 2024-02-13T00:41:21.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.308+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc240003fd0 con 0x7fc240004560 2024-02-13T00:41:21.314 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:41:21.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.310+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc240003fd0 con 0x7fc240004560 2024-02-13T00:41:21.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.310+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:21.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.310+0000 7fc2537fe700 10 client.17616.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.310+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc2400051d0 osd=0 4 2024-02-13T00:41:21.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.310+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 0 2 2024-02-13T00:41:21.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.311+0000 7fc2644d6700 10 client.17616.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:21.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.311+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 3 2024-02-13T00:41:21.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.311+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6805/150599780 conn(0x7fc2400084c0 0x7fc240002780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:21.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.311+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240004380 osd=1 3 2024-02-13T00:41:21.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.311+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240004380 osd=1 3 2024-02-13T00:41:21.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.311+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240004380 osd=1 3 2024-02-13T00:41:21.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.311+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 3 2024-02-13T00:41:21.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.311+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc240004380 osd=1 3 2024-02-13T00:41:21.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.311+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 1 3 2024-02-13T00:41:21.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.311+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240004380 osd=1 4 2024-02-13T00:41:21.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.311+0000 7fc2644d6700 10 client.17616.objecter _send_command 3 2024-02-13T00:41:21.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.311+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc240002e40 con 0x7fc2400084c0 2024-02-13T00:41:21.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.313+0000 7fc262272700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6805/150599780 conn(0x7fc2400084c0 0x7fc240002780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:21.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.316+0000 7fc262272700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6805/150599780 conn(0x7fc2400084c0 0x7fc240002780 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:21.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.316+0000 7fc2537fe700 10 client.17616.objecter ms_handle_connect 0x7fc2400084c0 2024-02-13T00:41:21.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.319+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc240002e40 con 0x7fc2400084c0 2024-02-13T00:41:21.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.319+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:41:21.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.319+0000 7fc2537fe700 10 client.17616.objecter _finish_command 3 = system:0 2024-02-13T00:41:21.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.319+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc240004380 osd=1 4 2024-02-13T00:41:21.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.319+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 1 3 2024-02-13T00:41:21.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.345+0000 7fc2644d6700 10 client.17616.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:21.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.345+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 4 2024-02-13T00:41:21.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.345+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240004380 osd=1 3 2024-02-13T00:41:21.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.345+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240004380 osd=1 3 2024-02-13T00:41:21.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.345+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240004380 osd=1 3 2024-02-13T00:41:21.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.345+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 4 2024-02-13T00:41:21.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.345+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc240004380 osd=1 3 2024-02-13T00:41:21.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.345+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 1 4 2024-02-13T00:41:21.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.345+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240004380 osd=1 4 2024-02-13T00:41:21.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.345+0000 7fc2644d6700 10 client.17616.objecter _send_command 4 2024-02-13T00:41:21.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.345+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc240007ae0 con 0x7fc2400084c0 2024-02-13T00:41:21.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.346+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc240007ae0 con 0x7fc2400084c0 2024-02-13T00:41:21.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.346+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:21.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.346+0000 7fc2537fe700 10 client.17616.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.346+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc240004380 osd=1 4 2024-02-13T00:41:21.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.346+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 1 4 2024-02-13T00:41:21.350 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:41:21.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.348+0000 7fc2644d6700 10 client.17616.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:21.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.348+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 5 2024-02-13T00:41:21.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.348+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc24000aff0 0x7fc24000d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:21.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.348+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240007d80 osd=2 3 2024-02-13T00:41:21.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.348+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240007d80 osd=2 3 2024-02-13T00:41:21.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.348+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240007d80 osd=2 3 2024-02-13T00:41:21.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.348+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 5 2024-02-13T00:41:21.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.348+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc240007d80 osd=2 3 2024-02-13T00:41:21.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.348+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 2 5 2024-02-13T00:41:21.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.348+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240007d80 osd=2 4 2024-02-13T00:41:21.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.348+0000 7fc2644d6700 10 client.17616.objecter _send_command 5 2024-02-13T00:41:21.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.348+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc240003bb0 con 0x7fc24000aff0 2024-02-13T00:41:21.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.355+0000 7fc261a71700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc24000aff0 0x7fc24000d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:21.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.397+0000 7fc261a71700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc24000aff0 0x7fc24000d3e0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:21.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.397+0000 7fc2537fe700 10 client.17616.objecter ms_handle_connect 0x7fc24000aff0 2024-02-13T00:41:21.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.398+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc240003bb0 con 0x7fc24000aff0 2024-02-13T00:41:21.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.398+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:41:21.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.398+0000 7fc2537fe700 10 client.17616.objecter _finish_command 5 = system:0 2024-02-13T00:41:21.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.398+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc240007d80 osd=2 4 2024-02-13T00:41:21.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.398+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 2 5 2024-02-13T00:41:21.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.425+0000 7fc2644d6700 10 client.17616.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:21.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.425+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 6 2024-02-13T00:41:21.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.425+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240007d80 osd=2 3 2024-02-13T00:41:21.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.425+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240007d80 osd=2 3 2024-02-13T00:41:21.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.425+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240007d80 osd=2 3 2024-02-13T00:41:21.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.425+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 6 2024-02-13T00:41:21.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.425+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc240007d80 osd=2 3 2024-02-13T00:41:21.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.425+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 2 6 2024-02-13T00:41:21.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.425+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240007d80 osd=2 4 2024-02-13T00:41:21.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.425+0000 7fc2644d6700 10 client.17616.objecter _send_command 6 2024-02-13T00:41:21.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.425+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc240003550 con 0x7fc24000aff0 2024-02-13T00:41:21.432 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:41:21.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.428+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc240003550 con 0x7fc24000aff0 2024-02-13T00:41:21.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.428+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:21.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.428+0000 7fc2537fe700 10 client.17616.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.428+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc240007d80 osd=2 4 2024-02-13T00:41:21.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.428+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 2 6 2024-02-13T00:41:21.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.429+0000 7fc2644d6700 10 client.17616.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:21.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.429+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 7 2024-02-13T00:41:21.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.429+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6813/604971011 conn(0x7fc24000ea50 0x7fc24000eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:21.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.429+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240001b90 osd=3 3 2024-02-13T00:41:21.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.429+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240001b90 osd=3 3 2024-02-13T00:41:21.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.429+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240001b90 osd=3 3 2024-02-13T00:41:21.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.429+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 7 2024-02-13T00:41:21.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.429+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc240001b90 osd=3 3 2024-02-13T00:41:21.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.429+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 3 7 2024-02-13T00:41:21.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.429+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240001b90 osd=3 4 2024-02-13T00:41:21.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.429+0000 7fc2644d6700 10 client.17616.objecter _send_command 7 2024-02-13T00:41:21.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.429+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc2400022f0 con 0x7fc24000ea50 2024-02-13T00:41:21.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.445+0000 7fc262a73700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6813/604971011 conn(0x7fc24000ea50 0x7fc24000eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:21.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.445+0000 7fc262a73700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6813/604971011 conn(0x7fc24000ea50 0x7fc24000eeb0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:21.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.445+0000 7fc2537fe700 10 client.17616.objecter ms_handle_connect 0x7fc24000ea50 2024-02-13T00:41:21.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.446+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc2400022f0 con 0x7fc24000ea50 2024-02-13T00:41:21.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.446+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:41:21.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.446+0000 7fc2537fe700 10 client.17616.objecter _finish_command 7 = system:0 2024-02-13T00:41:21.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.446+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc240001b90 osd=3 4 2024-02-13T00:41:21.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.446+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 3 7 2024-02-13T00:41:21.485 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:41:21.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.478+0000 7fc2644d6700 10 client.17616.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:21.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.478+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 8 2024-02-13T00:41:21.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.478+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240001b90 osd=3 3 2024-02-13T00:41:21.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.478+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240001b90 osd=3 3 2024-02-13T00:41:21.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.478+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240001b90 osd=3 3 2024-02-13T00:41:21.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.478+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 8 2024-02-13T00:41:21.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.478+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc240001b90 osd=3 3 2024-02-13T00:41:21.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.478+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 3 8 2024-02-13T00:41:21.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.478+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240001b90 osd=3 4 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.478+0000 7fc2644d6700 10 client.17616.objecter _send_command 8 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.478+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc24000fb70 con 0x7fc24000ea50 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.479+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc24000fb70 con 0x7fc24000ea50 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.479+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.479+0000 7fc2537fe700 10 client.17616.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.479+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc240001b90 osd=3 4 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.479+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 3 8 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.480+0000 7fc2644d6700 10 client.17616.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.480+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 9 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.480+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc2400120c0 0x7fc2400144b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.480+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc24000f3a0 osd=4 3 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.480+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc24000f3a0 osd=4 3 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.480+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc24000f3a0 osd=4 3 2024-02-13T00:41:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.480+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 9 2024-02-13T00:41:21.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.480+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc24000f3a0 osd=4 3 2024-02-13T00:41:21.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.480+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 4 9 2024-02-13T00:41:21.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.480+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc24000f3a0 osd=4 4 2024-02-13T00:41:21.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.480+0000 7fc2644d6700 10 client.17616.objecter _send_command 9 2024-02-13T00:41:21.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.480+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc240017100 con 0x7fc2400120c0 2024-02-13T00:41:21.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.481+0000 7fc262272700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc2400120c0 0x7fc2400144b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:21.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.481+0000 7fc262272700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc2400120c0 0x7fc2400144b0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:21.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.481+0000 7fc2537fe700 10 client.17616.objecter ms_handle_connect 0x7fc2400120c0 2024-02-13T00:41:21.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.482+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc240017100 con 0x7fc2400120c0 2024-02-13T00:41:21.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.482+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:41:21.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.482+0000 7fc2537fe700 10 client.17616.objecter _finish_command 9 = system:0 2024-02-13T00:41:21.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.482+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc24000f3a0 osd=4 4 2024-02-13T00:41:21.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.482+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 4 9 2024-02-13T00:41:21.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.502+0000 7fc2644d6700 10 client.17616.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:21.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.502+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 10 2024-02-13T00:41:21.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.502+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc24000f3a0 osd=4 3 2024-02-13T00:41:21.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.502+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc24000f3a0 osd=4 3 2024-02-13T00:41:21.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.502+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc24000f3a0 osd=4 3 2024-02-13T00:41:21.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.502+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 10 2024-02-13T00:41:21.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.502+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc24000f3a0 osd=4 3 2024-02-13T00:41:21.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.502+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 4 10 2024-02-13T00:41:21.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.502+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc24000f3a0 osd=4 4 2024-02-13T00:41:21.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.502+0000 7fc2644d6700 10 client.17616.objecter _send_command 10 2024-02-13T00:41:21.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.502+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc2400149a0 con 0x7fc2400120c0 2024-02-13T00:41:21.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.503+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc2400149a0 con 0x7fc2400120c0 2024-02-13T00:41:21.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.503+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:21.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.503+0000 7fc2537fe700 10 client.17616.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.504+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc24000f3a0 osd=4 4 2024-02-13T00:41:21.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.504+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 4 10 2024-02-13T00:41:21.525 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:41:21.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.523+0000 7fc2644d6700 10 client.17616.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:21.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.523+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 11 2024-02-13T00:41:21.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.523+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc2400150b0 0x7fc2400174a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:21.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.523+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc2400149a0 osd=5 3 2024-02-13T00:41:21.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.523+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc2400149a0 osd=5 3 2024-02-13T00:41:21.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.523+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc2400149a0 osd=5 3 2024-02-13T00:41:21.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.523+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 11 2024-02-13T00:41:21.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.523+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc2400149a0 osd=5 3 2024-02-13T00:41:21.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.523+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 5 11 2024-02-13T00:41:21.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.523+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc2400149a0 osd=5 4 2024-02-13T00:41:21.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.523+0000 7fc2644d6700 10 client.17616.objecter _send_command 11 2024-02-13T00:41:21.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.523+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc24001a270 con 0x7fc2400150b0 2024-02-13T00:41:21.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.532+0000 7fc261a71700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc2400150b0 0x7fc2400174a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:21.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.532+0000 7fc261a71700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc2400150b0 0x7fc2400174a0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:21.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.532+0000 7fc2537fe700 10 client.17616.objecter ms_handle_connect 0x7fc2400150b0 2024-02-13T00:41:21.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.533+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc24001a270 con 0x7fc2400150b0 2024-02-13T00:41:21.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.533+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:41:21.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.533+0000 7fc2537fe700 10 client.17616.objecter _finish_command 11 = system:0 2024-02-13T00:41:21.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.533+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc2400149a0 osd=5 4 2024-02-13T00:41:21.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.533+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 5 11 2024-02-13T00:41:21.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.556+0000 7fc2644d6700 10 client.17616.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:21.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.556+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 12 2024-02-13T00:41:21.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.556+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc2400149a0 osd=5 3 2024-02-13T00:41:21.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.556+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc2400149a0 osd=5 3 2024-02-13T00:41:21.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.556+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc2400149a0 osd=5 3 2024-02-13T00:41:21.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.556+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 12 2024-02-13T00:41:21.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.556+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc2400149a0 osd=5 3 2024-02-13T00:41:21.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.556+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 5 12 2024-02-13T00:41:21.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.556+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc2400149a0 osd=5 4 2024-02-13T00:41:21.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.556+0000 7fc2644d6700 10 client.17616.objecter _send_command 12 2024-02-13T00:41:21.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.556+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc24001a270 con 0x7fc2400150b0 2024-02-13T00:41:21.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.558+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc24001a270 con 0x7fc2400150b0 2024-02-13T00:41:21.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.558+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:21.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.558+0000 7fc2537fe700 10 client.17616.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.558+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc2400149a0 osd=5 4 2024-02-13T00:41:21.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.558+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 5 12 2024-02-13T00:41:21.564 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:41:21.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc2644d6700 10 client.17616.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:21.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 13 2024-02-13T00:41:21.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc240019600 0x7fc240019a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:21.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc24001a270 osd=6 3 2024-02-13T00:41:21.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc24001a270 osd=6 3 2024-02-13T00:41:21.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc24001a270 osd=6 3 2024-02-13T00:41:21.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 13 2024-02-13T00:41:21.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc24001a270 osd=6 3 2024-02-13T00:41:21.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 6 13 2024-02-13T00:41:21.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc24001a270 osd=6 4 2024-02-13T00:41:21.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc2644d6700 10 client.17616.objecter _send_command 13 2024-02-13T00:41:21.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc24001d620 con 0x7fc240019600 2024-02-13T00:41:21.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.561+0000 7fc262a73700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc240019600 0x7fc240019a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:21.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.562+0000 7fc262a73700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc240019600 0x7fc240019a60 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:21.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.562+0000 7fc2537fe700 10 client.17616.objecter ms_handle_connect 0x7fc240019600 2024-02-13T00:41:21.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.565+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc24001d620 con 0x7fc240019600 2024-02-13T00:41:21.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.565+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:41:21.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.565+0000 7fc2537fe700 10 client.17616.objecter _finish_command 13 = system:0 2024-02-13T00:41:21.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.565+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc24001a270 osd=6 4 2024-02-13T00:41:21.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.565+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 6 13 2024-02-13T00:41:21.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.590+0000 7fc2644d6700 10 client.17616.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:21.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.590+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 14 2024-02-13T00:41:21.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.590+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc24001a270 osd=6 3 2024-02-13T00:41:21.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.590+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc24001a270 osd=6 3 2024-02-13T00:41:21.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.590+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc24001a270 osd=6 3 2024-02-13T00:41:21.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.590+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 14 2024-02-13T00:41:21.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.590+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc24001a270 osd=6 3 2024-02-13T00:41:21.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.590+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 6 14 2024-02-13T00:41:21.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.590+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc24001a270 osd=6 4 2024-02-13T00:41:21.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.590+0000 7fc2644d6700 10 client.17616.objecter _send_command 14 2024-02-13T00:41:21.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.590+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc240017a50 con 0x7fc240019600 2024-02-13T00:41:21.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.592+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc240017a50 con 0x7fc240019600 2024-02-13T00:41:21.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.592+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:21.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.592+0000 7fc2537fe700 10 client.17616.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.592+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc24001a270 osd=6 4 2024-02-13T00:41:21.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.592+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 6 14 2024-02-13T00:41:21.595 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:41:21.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc2644d6700 10 client.17616.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:21.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 15 2024-02-13T00:41:21.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc240017f60 0x7fc240018380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:21.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240018d70 osd=7 3 2024-02-13T00:41:21.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240018d70 osd=7 3 2024-02-13T00:41:21.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240018d70 osd=7 3 2024-02-13T00:41:21.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 15 2024-02-13T00:41:21.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc240018d70 osd=7 3 2024-02-13T00:41:21.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 7 15 2024-02-13T00:41:21.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240018d70 osd=7 4 2024-02-13T00:41:21.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc2644d6700 10 client.17616.objecter _send_command 15 2024-02-13T00:41:21.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc2400188e0 con 0x7fc240017f60 2024-02-13T00:41:21.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc262272700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc240017f60 0x7fc240018380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:21.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.593+0000 7fc262272700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc240017f60 0x7fc240018380 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:21.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.598+0000 7fc2537fe700 10 client.17616.objecter ms_handle_connect 0x7fc240017f60 2024-02-13T00:41:21.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.598+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc2400188e0 con 0x7fc240017f60 2024-02-13T00:41:21.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.598+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:41:21.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.598+0000 7fc2537fe700 10 client.17616.objecter _finish_command 15 = system:0 2024-02-13T00:41:21.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.598+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc240018d70 osd=7 4 2024-02-13T00:41:21.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.598+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 7 15 2024-02-13T00:41:21.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.618+0000 7fc2644d6700 10 client.17616.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:21.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.618+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign -1 16 2024-02-13T00:41:21.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.618+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240018d70 osd=7 3 2024-02-13T00:41:21.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.618+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240018d70 osd=7 3 2024-02-13T00:41:21.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.618+0000 7fc2644d6700 20 client.17616.objecter _get_session s=0x7fc240018d70 osd=7 3 2024-02-13T00:41:21.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.618+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_remove -1 16 2024-02-13T00:41:21.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.618+0000 7fc2644d6700 20 client.17616.objecter get_session s=0x7fc240018d70 osd=7 3 2024-02-13T00:41:21.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.618+0000 7fc2644d6700 15 client.17616.objecter _session_command_op_assign 7 16 2024-02-13T00:41:21.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.618+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240018d70 osd=7 4 2024-02-13T00:41:21.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.618+0000 7fc2644d6700 10 client.17616.objecter _send_command 16 2024-02-13T00:41:21.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.618+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc2400188a0 con 0x7fc240017f60 2024-02-13T00:41:21.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2537fe700 1 -- 172.21.15.162:0/944439755 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc2400188a0 con 0x7fc240017f60 2024-02-13T00:41:21.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2537fe700 10 client.17616.objecter ms_dispatch 0x7fc25c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:21.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2537fe700 10 client.17616.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2537fe700 20 client.17616.objecter put_session s=0x7fc240018d70 osd=7 4 2024-02-13T00:41:21.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2537fe700 15 client.17616.objecter _session_command_op_remove 7 16 2024-02-13T00:41:21.640 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:41:21.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 10 librados: watch_flush enter 2024-02-13T00:41:21.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 10 librados: watch_flush exit 2024-02-13T00:41:21.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 10 client.17616.objecter close_session for osd.0 2024-02-13T00:41:21.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc240004560 msgr2=0x7fc240004980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:21.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc240004560 0x7fc240004980 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:21.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc2400051d0 osd=0 1 2024-02-13T00:41:21.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 10 client.17616.objecter close_session for osd.1 2024-02-13T00:41:21.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6805/150599780 conn(0x7fc2400084c0 msgr2=0x7fc240002780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:21.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6805/150599780 conn(0x7fc2400084c0 0x7fc240002780 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:21.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240004380 osd=1 1 2024-02-13T00:41:21.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 10 client.17616.objecter close_session for osd.2 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc24000aff0 msgr2=0x7fc24000d3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc24000aff0 0x7fc24000d3e0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240007d80 osd=2 1 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 10 client.17616.objecter close_session for osd.3 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6813/604971011 conn(0x7fc24000ea50 msgr2=0x7fc24000eeb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6813/604971011 conn(0x7fc24000ea50 0x7fc24000eeb0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240001b90 osd=3 1 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 10 client.17616.objecter close_session for osd.4 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc2400120c0 msgr2=0x7fc2400144b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc2400120c0 0x7fc2400144b0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc24000f3a0 osd=4 1 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 10 client.17616.objecter close_session for osd.5 2024-02-13T00:41:21.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc2400150b0 msgr2=0x7fc2400174a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc2400150b0 0x7fc2400174a0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc2400149a0 osd=5 1 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 10 client.17616.objecter close_session for osd.6 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc240019600 msgr2=0x7fc240019a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc240019600 0x7fc240019a60 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc24001a270 osd=6 1 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 10 client.17616.objecter close_session for osd.7 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc240017f60 msgr2=0x7fc240018380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc240017f60 0x7fc240018380 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 20 client.17616.objecter put_session s=0x7fc240018d70 osd=7 1 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 20 client.17616.objecter shutdown clearing up homeless session... 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.623+0000 7fc2644d6700 10 client.17616.objecter successfully canceled tick 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.624+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc248062c90 msgr2=0x7fc248065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:21.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.624+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc248062c90 0x7fc248065150 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fc258004b90 tx=0x7fc258009b60).stop 2024-02-13T00:41:21.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.624+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c104340 msgr2=0x7fc25c0762f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:21.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.624+0000 7fc2644d6700 1 --2- 172.21.15.162:0/944439755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c104340 0x7fc25c0762f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7fc24c009eb0 tx=0x7fc24c009cb0).stop 2024-02-13T00:41:21.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.628+0000 7fc262a73700 1 -- 172.21.15.162:0/944439755 reap_dead start 2024-02-13T00:41:21.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.628+0000 7fc262a73700 1 -- 172.21.15.162:0/944439755 reap_dead start 2024-02-13T00:41:21.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.628+0000 7fc262a73700 1 -- 172.21.15.162:0/944439755 reap_dead start 2024-02-13T00:41:21.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.628+0000 7fc262a73700 1 -- 172.21.15.162:0/944439755 reap_dead start 2024-02-13T00:41:21.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.628+0000 7fc262a73700 1 -- 172.21.15.162:0/944439755 reap_dead start 2024-02-13T00:41:21.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.628+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 shutdown_connections 2024-02-13T00:41:21.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.628+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 >> 172.21.15.162:0/944439755 conn(0x7fc25c06dae0 msgr2=0x7fc25c06f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:21.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.637+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 shutdown_connections 2024-02-13T00:41:21.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.646+0000 7fc2644d6700 1 -- 172.21.15.162:0/944439755 wait complete. 2024-02-13T00:41:21.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:21.646+0000 7fc2644d6700 1 librados: shutdown 2024-02-13T00:41:21.658 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.658 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.658 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.658 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.658 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.658 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.658 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.658 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:41:21.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:19.696131+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-7"}]: dispatch 2024-02-13T00:41:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:19.933018+0000 mon.b (mon.1) 111 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:20.222793+0000 mgr.y (mgr.14544) 157 : cluster [DBG] pgmap v158: 520 pgs: 32 creating+peering, 1 active+clean+snaptrim, 487 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:41:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:20.633877+0000 mon.a (mon.0) 1014 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-13T00:41:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:20.666601+0000 mon.a (mon.0) 1015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:20.666704+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-7"}]': finished 2024-02-13T00:41:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:20.666886+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-13T00:41:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:20.677867+0000 mon.b (mon.1) 112 : audit [INF] from='client.? 172.21.15.162:0/677726405' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:20.681661+0000 mon.b (mon.1) 113 : audit [INF] from='client.? 172.21.15.162:0/3559302368' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-83977-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:20.721934+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:20.722224+0000 mon.a (mon.0) 1019 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-83977-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:20.937802+0000 mon.b (mon.1) 114 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:21.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:20.961608+0000 mon.b (mon.1) 115 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:21.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:20.962438+0000 mon.a (mon.0) 1020 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:21.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:21.281924+0000 mon.a (mon.0) 1021 : audit [DBG] from='client.? 172.21.15.162:0/944439755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:41:21.851 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:19.696131+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-7"}]: dispatch 2024-02-13T00:41:21.851 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:19.933018+0000 mon.b (mon.1) 111 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:21.851 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:20.222793+0000 mgr.y (mgr.14544) 157 : cluster [DBG] pgmap v158: 520 pgs: 32 creating+peering, 1 active+clean+snaptrim, 487 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:41:21.851 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:20.633877+0000 mon.a (mon.0) 1014 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-13T00:41:21.851 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:20.666601+0000 mon.a (mon.0) 1015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:21.851 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:20.666704+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-7"}]': finished 2024-02-13T00:41:21.851 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:20.666886+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-13T00:41:21.851 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:20.677867+0000 mon.b (mon.1) 112 : audit [INF] from='client.? 172.21.15.162:0/677726405' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.851 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:20.681661+0000 mon.b (mon.1) 113 : audit [INF] from='client.? 172.21.15.162:0/3559302368' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-83977-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.851 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:20.721934+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.851 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:20.722224+0000 mon.a (mon.0) 1019 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-83977-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.852 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:20.937802+0000 mon.b (mon.1) 114 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:21.852 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:20.961608+0000 mon.b (mon.1) 115 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:21.852 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:20.962438+0000 mon.a (mon.0) 1020 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:21.852 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:21.281924+0000 mon.a (mon.0) 1021 : audit [DBG] from='client.? 172.21.15.162:0/944439755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:41:21.852 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:19.696131+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-7"}]: dispatch 2024-02-13T00:41:21.852 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:19.933018+0000 mon.b (mon.1) 111 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:21.852 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:20.222793+0000 mgr.y (mgr.14544) 157 : cluster [DBG] pgmap v158: 520 pgs: 32 creating+peering, 1 active+clean+snaptrim, 487 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:41:21.852 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:20.633877+0000 mon.a (mon.0) 1014 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-13T00:41:21.852 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:20.666601+0000 mon.a (mon.0) 1015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:21.852 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:20.666704+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-7"}]': finished 2024-02-13T00:41:21.853 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:20.666886+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-13T00:41:21.853 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:20.677867+0000 mon.b (mon.1) 112 : audit [INF] from='client.? 172.21.15.162:0/677726405' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.853 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:20.681661+0000 mon.b (mon.1) 113 : audit [INF] from='client.? 172.21.15.162:0/3559302368' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-83977-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.853 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:20.721934+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.853 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:20.722224+0000 mon.a (mon.0) 1019 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-83977-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:21.853 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:20.937802+0000 mon.b (mon.1) 114 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:21.853 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:20.961608+0000 mon.b (mon.1) 115 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:21.853 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:20.962438+0000 mon.a (mon.0) 1020 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:21.853 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:21.281924+0000 mon.a (mon.0) 1021 : audit [DBG] from='client.? 172.21.15.162:0/944439755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:41:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:22.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:22.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:22.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:22.293 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:22.313 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:41:22.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:22.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:41:22.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:22.579 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:22.779 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-02-13T00:41:22.779 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-02-13T00:41:22.779 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-02-13T00:41:22.779 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-02-13T00:41:22.779 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (8103 ms) 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (4183 ms) 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3638 ms) 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3277 ms) 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-02-13T00:41:22.780 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2790 ms) 2024-02-13T00:41:22.781 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-02-13T00:41:22.781 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (2969 ms) 2024-02-13T00:41:22.781 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-02-13T00:41:22.781 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (3124 ms) 2024-02-13T00:41:22.781 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (28084 ms total) 2024-02-13T00:41:22.781 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: 2024-02-13T00:41:22.781 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-02-13T00:41:22.781 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (28084 ms total) 2024-02-13T00:41:22.781 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-02-13T00:41:22.817 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:21.740028+0000 osd.1 (osd.1) 67 : cluster [DBG] 20.5 deep-scrub starts 2024-02-13T00:41:22.817 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:21.743454+0000 mon.a (mon.0) 1022 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:22.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:21.743527+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-83977-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:22.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:21.743674+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:41:22.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:21.743718+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-13T00:41:22.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:21.752133+0000 mon.b (mon.1) 116 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7"}]: dispatch 2024-02-13T00:41:22.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:21.753519+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7"}]: dispatch 2024-02-13T00:41:22.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:21.767366+0000 osd.1 (osd.1) 68 : cluster [DBG] 20.5 deep-scrub ok 2024-02-13T00:41:22.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:21.950153+0000 mon.b (mon.1) 117 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:22.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:21.740028+0000 osd.1 (osd.1) 67 : cluster [DBG] 20.5 deep-scrub starts 2024-02-13T00:41:22.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:21.743454+0000 mon.a (mon.0) 1022 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:22.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:21.743527+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-83977-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:22.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:21.743674+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:41:22.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:21.743718+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-13T00:41:22.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:21.752133+0000 mon.b (mon.1) 116 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7"}]: dispatch 2024-02-13T00:41:22.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:21.753519+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7"}]: dispatch 2024-02-13T00:41:22.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:21.767366+0000 osd.1 (osd.1) 68 : cluster [DBG] 20.5 deep-scrub ok 2024-02-13T00:41:22.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:21.950153+0000 mon.b (mon.1) 117 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:22.837 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:22.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:22.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:22.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:23.004 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:23.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:21.740028+0000 osd.1 (osd.1) 67 : cluster [DBG] 20.5 deep-scrub starts 2024-02-13T00:41:23.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:21.743454+0000 mon.a (mon.0) 1022 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84153-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:21.743527+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-83977-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:21.743674+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:41:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:21.743718+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-13T00:41:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:21.752133+0000 mon.b (mon.1) 116 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7"}]: dispatch 2024-02-13T00:41:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:21.753519+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7"}]: dispatch 2024-02-13T00:41:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:21.767366+0000 osd.1 (osd.1) 68 : cluster [DBG] 20.5 deep-scrub ok 2024-02-13T00:41:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:21.950153+0000 mon.b (mon.1) 117 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:23.484 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:23.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:23.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:41:23.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:23.712 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:23.854 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:23.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:23.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:23.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:23.988 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:24.000 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:22.232710+0000 mgr.y (mgr.14544) 158 : cluster [DBG] pgmap v161: 488 pgs: 64 unknown, 1 active+clean+snaptrim, 423 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:41:24.000 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:22.756531+0000 mon.a (mon.0) 1027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7"}]': finished 2024-02-13T00:41:24.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:22.756700+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-13T00:41:24.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:22.783553+0000 mon.c (mon.2) 124 : audit [INF] from='client.? 172.21.15.162:0/3388257250' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:24.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:22.788036+0000 mon.a (mon.0) 1029 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:24.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:22.957223+0000 mon.b (mon.1) 118 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:24.003 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:22.232710+0000 mgr.y (mgr.14544) 158 : cluster [DBG] pgmap v161: 488 pgs: 64 unknown, 1 active+clean+snaptrim, 423 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:41:24.003 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:22.756531+0000 mon.a (mon.0) 1027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7"}]': finished 2024-02-13T00:41:24.003 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:22.756700+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-13T00:41:24.004 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:22.783553+0000 mon.c (mon.2) 124 : audit [INF] from='client.? 172.21.15.162:0/3388257250' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:24.004 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:22.788036+0000 mon.a (mon.0) 1029 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:24.004 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:22.957223+0000 mon.b (mon.1) 118 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:24.007 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:41:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:24.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:41:24.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:24.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:22.232710+0000 mgr.y (mgr.14544) 158 : cluster [DBG] pgmap v161: 488 pgs: 64 unknown, 1 active+clean+snaptrim, 423 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:41:24.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:22.756531+0000 mon.a (mon.0) 1027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-7"}]': finished 2024-02-13T00:41:24.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:22.756700+0000 mon.a (mon.0) 1028 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-13T00:41:24.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:22.783553+0000 mon.c (mon.2) 124 : audit [INF] from='client.? 172.21.15.162:0/3388257250' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:24.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:22.788036+0000 mon.a (mon.0) 1029 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:24.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:22.957223+0000 mon.b (mon.1) 118 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:24.330 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:24.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:24.428 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:41:24.428 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:41:24.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:41:24.428 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:41:24.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.428 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14913.0:2 22.16 22:69efe4bc:test-rados-api-smithi162-84001-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.531523+0000", 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.887596330000001, 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.084235272999999999, 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14913", 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3501757973", 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531523+0000", 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531523+0000", 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531521+0000", 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1880000000000008e-06 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531530+0000", 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0994e-05 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531541+0000", 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4500000000000004e-06 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531550+0000", 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0084330910000000002 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.539983+0000", 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1860000000000003e-06 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.539990+0000", 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043643534999999997 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.583634+0000", 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011933 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.583753+0000", 2024-02-13T00:41:24.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000156346 2024-02-13T00:41:24.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,7", 2024-02-13T00:41:24.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.583909+0000", 2024-02-13T00:41:24.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026647634 2024-02-13T00:41:24.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.610557+0000", 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043537769999999996 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.614911+0000", 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00078932899999999996 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615700+0000", 2024-02-13T00:41:24.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3908000000000001e-05 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615734+0000", 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4465e-05 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615758+0000", 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.084235272999999999 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24745.0:2 24.1b 24:de1a0751:test-rados-api-smithi162-84081-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.540986+0000", 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.878133957999999, 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12964152300000001, 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24745", 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/530375479", 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540986+0000", 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540986+0000", 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540984+0000", 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.49e-05 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540999+0000", 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0326000000000001e-05 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.541009+0000", 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.926e-06 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.541017+0000", 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004928562 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.545945+0000", 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3000000000000001e-06 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.545951+0000", 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043217637000000003 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589168+0000", 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011348799999999999 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589282+0000", 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0266999999999995e-05 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589372+0000", 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020246106999999999 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.609618+0000", 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005197513 2024-02-13T00:41:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.614816+0000", 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055733195999999999 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670549+0000", 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8872000000000002e-05 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670598+0000", 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9230999999999998e-05 2024-02-13T00:41:24.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670627+0000", 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12964152300000001 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24773.0:2 11.2 11:42a49006:test-rados-api-smithi162-84105-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.602097+0000", 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.817022267999999, 2024-02-13T00:41:24.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101585882, 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24773", 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4011549658", 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602097+0000", 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602097+0000", 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602095+0000", 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7520000000000002e-05 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602113+0000", 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1664999999999999e-05 2024-02-13T00:41:24.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602124+0000", 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0750000000000004e-06 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602133+0000", 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044129360000000001 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.606546+0000", 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126408 2024-02-13T00:41:24.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.606673+0000", 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002542182 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,3", 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.609215+0000", 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059528159999999997 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615168+0000", 2024-02-13T00:41:24.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055524639000000001 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670692+0000", 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032890434000000003 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703583+0000", 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5461000000000003e-05 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703638+0000", 2024-02-13T00:41:24.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4892000000000003e-05 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703683+0000", 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101585882 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14862.0:3 12.f e107/104 12:f4b50b61:::1.obj:head v 107'1, mlcod=0'0)", 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.626072+0000", 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.793047544, 2024-02-13T00:41:24.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014565409, 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.626072+0000", 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.626072+0000", 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:41:24.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.626072+0000", 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2354999999999999e-05 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.626084+0000", 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5319999999999997e-06 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.626090+0000", 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7090000000000002e-06 2024-02-13T00:41:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.626097+0000", 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015487140000000001 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.627646+0000", 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8757e-05 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.627665+0000", 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012947594 2024-02-13T00:41:24.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.640612+0000", 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5103e-05 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.640637+0000", 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014565409 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24911.0:20 16.1 16:83282cb0:::1.obj2:head [write 0~90] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.474569+0000", 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.944550929999998, 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.158591963, 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24911", 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1860840531", 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 20 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474569+0000", 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474569+0000", 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474567+0000", 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9480000000000003e-06 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474577+0000", 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1072e-05 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474588+0000", 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8759999999999999e-06 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474595+0000", 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028252126999999998 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.502847+0000", 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013890300000000001 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.502986+0000", 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015514899999999999 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.503141+0000", 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016316878999999999 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.519458+0000", 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058076720999999998 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.577535+0000", 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055542411999999999 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.633077+0000", 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3886e-05 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.633121+0000", 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9544e-05 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.633161+0000", 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.158591963 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.458 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14886.0:441 20.14 e107/105 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head v 107'198, mlcod=107'196)", 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.92418357, 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12436195799999999, 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:41:24.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989999999999997e-06 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494942+0000", 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037274475000000001 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.532217+0000", 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2009999999999999e-06 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.532225+0000", 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055477902000000003 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.587703+0000", 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6648999999999998e-05 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.587729+0000", 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031536963000000001 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.619266+0000", 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1473999999999997e-05 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.619298+0000", 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12436195799999999 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.462 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24911.0:22 16.a 16:526897e8:::42.obj2:head [write 0~23] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.672693+0000", 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.746426515, 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012701689, 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24911", 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1860840531", 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 22 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.672693+0000", 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.672693+0000", 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.672692+0000", 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0790000000000007e-06 2024-02-13T00:41:24.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.672701+0000", 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4192000000000001e-05 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.672715+0000", 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1380000000000004e-06 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.672721+0000", 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4718000000000002e-05 2024-02-13T00:41:24.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.672756+0000", 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013242500000000001 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.672888+0000", 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016134799999999999 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.673050+0000", 2024-02-13T00:41:24.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042199300000000001 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.673472+0000", 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00081930700000000002 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.674291+0000", 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011010417999999999 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.685301+0000", 2024-02-13T00:41:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3502999999999999e-05 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.685355+0000", 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9918e-05 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.685395+0000", 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012701689 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14886.0:1809 20.14 e108/105 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head v 108'882, mlcod=108'880)", 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:03.997813+0000", 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.421306465000001, 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020061802, 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.997813+0000", 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.997813+0000", 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.997812+0000", 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2629999999999992e-06 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.997822+0000", 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8209999999999997e-06 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.997831+0000", 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4880000000000001e-06 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.997839+0000", 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001451241 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.999290+0000", 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7339999999999999e-05 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.999317+0000", 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018523022 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.017840+0000", 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4415999999999997e-05 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.017875+0000", 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020061802 2024-02-13T00:41:24.472 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2064 20.19 20:9c8ffe76:test-rados-api-smithi162-84049-16::big.copy2:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.437684+0000", 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.981435994999998, 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088117519000000005, 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:41:24.473 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2064 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437684+0000", 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437684+0000", 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437682+0000", 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1730000000000001e-06 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437691+0000", 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4664999999999999e-05 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437706+0000", 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6229999999999997e-06 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437713+0000", 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6378999999999994e-05 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437800+0000", 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2805000000000003e-05 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437893+0000", 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.060281989000000001 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.498175+0000", 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034643450000000002 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.501639+0000", 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023806853999999999 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525446+0000", 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028838300000000002 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525734+0000", 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3445000000000002e-05 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525778+0000", 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3501000000000001e-05 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525801+0000", 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088117519000000005 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.478 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2065 20.19 20:9c8ffe76:test-rados-api-smithi162-84049-16::big.copy2:head [read 0~25165836] snapc 0=[] ondisk+read+known_if_redirected e109)", 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.526020+0000", 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.893099724999999, 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020150886999999999, 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2065 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.479 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.526020+0000", 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.526020+0000", 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.526018+0000", 2024-02-13T00:41:24.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6340000000000003e-06 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.526025+0000", 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0429e-05 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.526036+0000", 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1570000000000003e-06 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.526043+0000", 2024-02-13T00:41:24.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6650000000000003e-05 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.526079+0000", 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6159999999999998e-05 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.526136+0000", 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020035163000000002 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.546171+0000", 2024-02-13T00:41:24.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020150886999999999 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2069 20.1b 20:de4afda3:test-rados-api-smithi162-84049-17::big:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.630167+0000", 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.788952441999999, 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11190932200000001, 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:41:24.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2069 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.630167+0000", 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.630167+0000", 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.630164+0000", 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011350433 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641515+0000", 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3941e-05 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641539+0000", 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6942000000000001e-05 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641566+0000", 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0275e-05 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641646+0000", 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100653 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641747+0000", 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137825 2024-02-13T00:41:24.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,7", 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641884+0000", 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036914244999999998 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.678799+0000", 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.062888029999999998 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.741687+0000", 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032975399999999997 2024-02-13T00:41:24.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742016+0000", 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4996000000000002e-05 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742051+0000", 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4997000000000001e-05 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742076+0000", 2024-02-13T00:41:24.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11190932200000001 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2074 20.1b 20:de8a7420:test-rados-api-smithi162-84049-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.862849+0000", 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.556270612999999, 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101931788, 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:41:24.489 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2074 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.862849+0000", 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.862849+0000", 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:41:24.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.862846+0000", 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070632639999999997 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869909+0000", 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1322e-05 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869931+0000", 2024-02-13T00:41:24.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0179e-05 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869941+0000", 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1819e-05 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869993+0000", 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010545 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.870098+0000", 2024-02-13T00:41:24.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000146911 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,7", 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.870245+0000", 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029969216 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.900214+0000", 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064237111 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964452+0000", 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025591999999999998 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964707+0000", 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8297999999999998e-05 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964746+0000", 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4975999999999999e-05 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964781+0000", 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101931788 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2076 20.1b 20:de8a7420:test-rados-api-smithi162-84049-17::big3:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.966742+0000", 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.452377772999998, 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016689025999999999, 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:24.495 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2076 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.966742+0000", 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.966742+0000", 2024-02-13T00:41:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.966741+0000", 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012772339999999999 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.968018+0000", 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.203e-05 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.968030+0000", 2024-02-13T00:41:24.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1800000000000001e-06 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.968036+0000", 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1835999999999998e-05 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.968068+0000", 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8362999999999997e-05 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.968126+0000", 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023000350000000002 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,7", 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.970426+0000", 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0073046129999999997 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.977731+0000", 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055612910000000003 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.983292+0000", 2024-02-13T00:41:24.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8108000000000002e-05 2024-02-13T00:41:24.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.983330+0000", 2024-02-13T00:41:24.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6663999999999999e-05 2024-02-13T00:41:24.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.983377+0000", 2024-02-13T00:41:24.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3684999999999998e-05 2024-02-13T00:41:24.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.983431+0000", 2024-02-13T00:41:24.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016689025999999999 2024-02-13T00:41:24.502 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.503 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.503 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.503 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27739.0:2 68.6 68:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-02-13T00:41:24.503 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:21.788780+0000", 2024-02-13T00:41:24.503 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.630339947, 2024-02-13T00:41:24.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061004435000000003, 2024-02-13T00:41:24.503 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.504 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.504 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:24.504 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27739", 2024-02-13T00:41:24.504 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3559302368", 2024-02-13T00:41:24.504 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:41:24.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.504 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788780+0000", 2024-02-13T00:41:24.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788780+0000", 2024-02-13T00:41:24.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:41:24.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788777+0000", 2024-02-13T00:41:24.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2724999999999999e-05 2024-02-13T00:41:24.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788789+0000", 2024-02-13T00:41:24.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2472e-05 2024-02-13T00:41:24.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788802+0000", 2024-02-13T00:41:24.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3860000000000007e-06 2024-02-13T00:41:24.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788810+0000", 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00058719999999999996 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.789398+0000", 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3919999999999997e-06 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.789405+0000", 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047759248999999997 2024-02-13T00:41:24.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.837164+0000", 2024-02-13T00:41:24.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118143 2024-02-13T00:41:24.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.837282+0000", 2024-02-13T00:41:24.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015534899999999999 2024-02-13T00:41:24.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,7", 2024-02-13T00:41:24.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.837438+0000", 2024-02-13T00:41:24.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005545024 2024-02-13T00:41:24.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.842983+0000", 2024-02-13T00:41:24.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059063199999999996 2024-02-13T00:41:24.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:24.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.848889+0000", 2024-02-13T00:41:24.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00079681100000000003 2024-02-13T00:41:24.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:24.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.849686+0000", 2024-02-13T00:41:24.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3878999999999997e-05 2024-02-13T00:41:24.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.849740+0000", 2024-02-13T00:41:24.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4291e-05 2024-02-13T00:41:24.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.849784+0000", 2024-02-13T00:41:24.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061004435000000003 2024-02-13T00:41:24.511 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.511 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.511 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.512 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16275.0:5 66.6 e127/124 66:64b50987:test-rados-api-smithi162-84081-8::foo:head v 127'3, mlcod=0'0)", 2024-02-13T00:41:24.512 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:21.788919+0000", 2024-02-13T00:41:24.512 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.630200297, 2024-02-13T00:41:24.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023292693, 2024-02-13T00:41:24.512 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.512 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.512 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788919+0000", 2024-02-13T00:41:24.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788919+0000", 2024-02-13T00:41:24.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:41:24.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788919+0000", 2024-02-13T00:41:24.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9450000000000005e-06 2024-02-13T00:41:24.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788929+0000", 2024-02-13T00:41:24.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8590000000000005e-06 2024-02-13T00:41:24.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788937+0000", 2024-02-13T00:41:24.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5400000000000003e-06 2024-02-13T00:41:24.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788942+0000", 2024-02-13T00:41:24.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3648999999999999e-05 2024-02-13T00:41:24.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.789016+0000", 2024-02-13T00:41:24.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4042000000000001e-05 2024-02-13T00:41:24.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.789040+0000", 2024-02-13T00:41:24.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023135939000000001 2024-02-13T00:41:24.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.812176+0000", 2024-02-13T00:41:24.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6118999999999998e-05 2024-02-13T00:41:24.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.812212+0000", 2024-02-13T00:41:24.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023292693 2024-02-13T00:41:24.517 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.517 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.520 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.520 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25082.0:14 60.a e127/120 60:51f054f9:test-rados-api-smithi162-84073-8::foo:head v 127'3, mlcod=0'0)", 2024-02-13T00:41:24.520 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:21.788947+0000", 2024-02-13T00:41:24.520 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.6301722889999999, 2024-02-13T00:41:24.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023279877000000001, 2024-02-13T00:41:24.520 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.520 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.520 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788947+0000", 2024-02-13T00:41:24.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788947+0000", 2024-02-13T00:41:24.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:41:24.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788947+0000", 2024-02-13T00:41:24.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6019999999999999e-06 2024-02-13T00:41:24.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788952+0000", 2024-02-13T00:41:24.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1390000000000003e-06 2024-02-13T00:41:24.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788958+0000", 2024-02-13T00:41:24.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6770000000000001e-06 2024-02-13T00:41:24.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788961+0000", 2024-02-13T00:41:24.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019408839999999999 2024-02-13T00:41:24.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.790902+0000", 2024-02-13T00:41:24.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3784999999999999e-05 2024-02-13T00:41:24.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.790916+0000", 2024-02-13T00:41:24.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021288626000000001 2024-02-13T00:41:24.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.812205+0000", 2024-02-13T00:41:24.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2575999999999999e-05 2024-02-13T00:41:24.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.812227+0000", 2024-02-13T00:41:24.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023279877000000001 2024-02-13T00:41:24.525 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.525 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.525 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.525 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.5.0:281 66.6 e128/124 66:64b50987:test-rados-api-smithi162-84081-8::foo:3 v 128'5, mlcod=126'1)", 2024-02-13T00:41:24.525 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:22.771607+0000", 2024-02-13T00:41:24.525 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.647512547, 2024-02-13T00:41:24.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033789000999999999, 2024-02-13T00:41:24.526 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.526 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.526 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771607+0000", 2024-02-13T00:41:24.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771607+0000", 2024-02-13T00:41:24.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:41:24.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771606+0000", 2024-02-13T00:41:24.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4549999999999998e-06 2024-02-13T00:41:24.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771614+0000", 2024-02-13T00:41:24.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7230000000000004e-06 2024-02-13T00:41:24.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771622+0000", 2024-02-13T00:41:24.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2320000000000004e-06 2024-02-13T00:41:24.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771629+0000", 2024-02-13T00:41:24.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012838736 2024-02-13T00:41:24.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784468+0000", 2024-02-13T00:41:24.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1084e-05 2024-02-13T00:41:24.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784489+0000", 2024-02-13T00:41:24.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020873023000000001 2024-02-13T00:41:24.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.805362+0000", 2024-02-13T00:41:24.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4317999999999999e-05 2024-02-13T00:41:24.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.805396+0000", 2024-02-13T00:41:24.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033789000999999999 2024-02-13T00:41:24.530 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.530 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.530 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.530 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.5.0:282 60.a e128/120 60:51f054f9:test-rados-api-smithi162-84073-8::foo:7 v 128'6, mlcod=127'3)", 2024-02-13T00:41:24.531 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:22.784439+0000", 2024-02-13T00:41:24.531 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.63468072, 2024-02-13T00:41:24.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031021368000000001, 2024-02-13T00:41:24.531 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.531 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.531 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784439+0000", 2024-02-13T00:41:24.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784439+0000", 2024-02-13T00:41:24.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:41:24.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784438+0000", 2024-02-13T00:41:24.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1600999999999999e-05 2024-02-13T00:41:24.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784460+0000", 2024-02-13T00:41:24.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3000000000000007e-06 2024-02-13T00:41:24.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784469+0000", 2024-02-13T00:41:24.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2879999999999998e-06 2024-02-13T00:41:24.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784475+0000", 2024-02-13T00:41:24.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027856051 2024-02-13T00:41:24.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.812331+0000", 2024-02-13T00:41:24.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5820999999999999e-05 2024-02-13T00:41:24.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.812357+0000", 2024-02-13T00:41:24.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030688690000000001 2024-02-13T00:41:24.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.815426+0000", 2024-02-13T00:41:24.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4136e-05 2024-02-13T00:41:24.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.815460+0000", 2024-02-13T00:41:24.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031021368000000001 2024-02-13T00:41:24.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.536 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.536 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16275.0:9 66.6 e129/124 66:64b50987:test-rados-api-smithi162-84081-8::foo:head v 129'6, mlcod=127'3)", 2024-02-13T00:41:24.536 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:23.831556+0000", 2024-02-13T00:41:24.536 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.58756360200000002, 2024-02-13T00:41:24.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054147344, 2024-02-13T00:41:24.537 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.537 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.537 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831556+0000", 2024-02-13T00:41:24.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831556+0000", 2024-02-13T00:41:24.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:41:24.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831554+0000", 2024-02-13T00:41:24.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.209e-05 2024-02-13T00:41:24.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831576+0000", 2024-02-13T00:41:24.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4198e-05 2024-02-13T00:41:24.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831590+0000", 2024-02-13T00:41:24.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1470000000000004e-06 2024-02-13T00:41:24.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831597+0000", 2024-02-13T00:41:24.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051779300000000001 2024-02-13T00:41:24.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832115+0000", 2024-02-13T00:41:24.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0034000000000002e-05 2024-02-13T00:41:24.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832135+0000", 2024-02-13T00:41:24.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053545497999999997 2024-02-13T00:41:24.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.885681+0000", 2024-02-13T00:41:24.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2527999999999998e-05 2024-02-13T00:41:24.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.885703+0000", 2024-02-13T00:41:24.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054147344 2024-02-13T00:41:24.541 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.541 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.541 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.541 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25082.0:19 60.a e129/120 60:51f054f9:test-rados-api-smithi162-84073-8::foo:head v 129'7, mlcod=127'4)", 2024-02-13T00:41:24.542 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:41:24.542 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.58751522300000003, 2024-02-13T00:41:24.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048672976, 2024-02-13T00:41:24.542 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:24.542 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:24.542 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:24.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:24.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:41:24.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:24.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:24.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:41:24.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:41:24.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:24.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:41:24.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0720000000000001e-06 2024-02-13T00:41:24.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:24.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831608+0000", 2024-02-13T00:41:24.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9790000000000001e-06 2024-02-13T00:41:24.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:24.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831613+0000", 2024-02-13T00:41:24.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.641e-06 2024-02-13T00:41:24.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:24.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831617+0000", 2024-02-13T00:41:24.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013029420000000001 2024-02-13T00:41:24.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:24.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832920+0000", 2024-02-13T00:41:24.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8600000000000001e-05 2024-02-13T00:41:24.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:24.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832938+0000", 2024-02-13T00:41:24.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047309385000000002 2024-02-13T00:41:24.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:24.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.880247+0000", 2024-02-13T00:41:24.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9788e-05 2024-02-13T00:41:24.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:24.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.880277+0000", 2024-02-13T00:41:24.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048672976 2024-02-13T00:41:24.546 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.547 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.547 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.547 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.547 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:24.547 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:41:24.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:24.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:24.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.501+0000 7f8439480700 1 -- 172.21.15.162:0/1532793762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84340fc620 msgr2=0x7f84340fa000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:24.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.501+0000 7f8439480700 1 --2- 172.21.15.162:0/1532793762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84340fc620 0x7f84340fa000 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7f8428004870 tx=0x7f8428009cb0).stop 2024-02-13T00:41:24.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.501+0000 7f8439480700 1 -- 172.21.15.162:0/1532793762 shutdown_connections 2024-02-13T00:41:24.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.501+0000 7f8439480700 1 --2- 172.21.15.162:0/1532793762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84340fa6e0 0x7f8434106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:24.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.501+0000 7f8439480700 1 --2- 172.21.15.162:0/1532793762 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84340fbd50 0x7f84340fc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:24.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.501+0000 7f8439480700 1 --2- 172.21.15.162:0/1532793762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84340fc620 0x7f84340fa000 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:24.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.501+0000 7f8439480700 1 -- 172.21.15.162:0/1532793762 >> 172.21.15.162:0/1532793762 conn(0x7f84340f5aa0 msgr2=0x7f84340f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:24.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.502+0000 7f8439480700 1 -- 172.21.15.162:0/1532793762 shutdown_connections 2024-02-13T00:41:24.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.502+0000 7f8439480700 1 -- 172.21.15.162:0/1532793762 wait complete. 2024-02-13T00:41:24.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.502+0000 7f8439480700 1 librados: starting msgr at 2024-02-13T00:41:24.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.502+0000 7f8439480700 1 librados: starting objecter 2024-02-13T00:41:24.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.502+0000 7f8439480700 1 Processor -- start 2024-02-13T00:41:24.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.502+0000 7f8439480700 1 -- start start 2024-02-13T00:41:24.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.502+0000 7f8439480700 1 librados: setting wanted keys 2024-02-13T00:41:24.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.502+0000 7f8439480700 1 librados: calling monclient init 2024-02-13T00:41:24.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.502+0000 7f8439480700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84340fa6e0 0x7f8434110600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:24.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.502+0000 7f8439480700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84340fbd50 0x7f8434110b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:24.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.503+0000 7f8439480700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84340fc620 0x7f843410a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:24.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.503+0000 7f8439480700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843410adb0 con 0x7f84340fc620 2024-02-13T00:41:24.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.503+0000 7f8439480700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843410af20 con 0x7f84340fa6e0 2024-02-13T00:41:24.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.503+0000 7f8439480700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843410b0e0 con 0x7f84340fbd50 2024-02-13T00:41:24.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.503+0000 7f8432ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84340fa6e0 0x7f8434110600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:24.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.503+0000 7f8432ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84340fa6e0 0x7f8434110600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:42158/0 (socket says 172.21.15.162:42158) 2024-02-13T00:41:24.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.503+0000 7f8432ffd700 1 -- 172.21.15.162:0/1275818284 learned_addr learned my addr 172.21.15.162:0/1275818284 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:24.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.503+0000 7f8432ffd700 1 -- 172.21.15.162:0/1275818284 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84340fbd50 msgr2=0x7f8434110b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:24.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.503+0000 7f8432ffd700 1 --2- 172.21.15.162:0/1275818284 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84340fbd50 0x7f8434110b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:24.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.503+0000 7f8432ffd700 1 -- 172.21.15.162:0/1275818284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84340fc620 msgr2=0x7f843410a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:24.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.503+0000 7f8432ffd700 1 --2- 172.21.15.162:0/1275818284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84340fc620 0x7f843410a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:24.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.503+0000 7f8432ffd700 1 -- 172.21.15.162:0/1275818284 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8428005040 con 0x7f84340fa6e0 2024-02-13T00:41:24.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.504+0000 7f8432ffd700 1 --2- 172.21.15.162:0/1275818284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84340fa6e0 0x7f8434110600 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7f842400b530 tx=0x7f84240090f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:24.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.504+0000 7f841bfff700 10 client.?.objecter ms_handle_connect 0x7f84340fa6e0 2024-02-13T00:41:24.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.504+0000 7f841bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:24.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.506+0000 7f841bfff700 1 -- 172.21.15.162:0/1275818284 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842400d4a0 con 0x7f84340fa6e0 2024-02-13T00:41:24.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.506+0000 7f841bfff700 1 -- 172.21.15.162:0/1275818284 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f842400dad0 con 0x7f84340fa6e0 2024-02-13T00:41:24.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.506+0000 7f841bfff700 1 -- 172.21.15.162:0/1275818284 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424012a50 con 0x7f84340fa6e0 2024-02-13T00:41:24.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.506+0000 7f8439480700 1 -- 172.21.15.162:0/1275818284 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f843410b3c0 con 0x7f84340fa6e0 2024-02-13T00:41:24.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.506+0000 7f8439480700 10 client.30808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:24.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.506+0000 7f8439480700 1 -- 172.21.15.162:0/1275818284 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84341175c0 con 0x7f84340fa6e0 2024-02-13T00:41:24.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.506+0000 7f8439480700 1 librados: init done 2024-02-13T00:41:24.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.509+0000 7f841bfff700 1 -- 172.21.15.162:0/1275818284 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f842400d600 con 0x7f84340fa6e0 2024-02-13T00:41:24.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.510+0000 7f841bfff700 1 --2- 172.21.15.162:0/1275818284 >> v2:172.21.15.162:6800/2866534053 conn(0x7f841c062c90 0x7f841c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:24.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.510+0000 7f841bfff700 1 -- 172.21.15.162:0/1275818284 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(129..129 src has 52..129) v4 ==== 10809+0+0 (secure 0 0 0) 0x7f842407dd30 con 0x7f84340fa6e0 2024-02-13T00:41:24.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.510+0000 7f841bfff700 10 client.30808.objecter ms_dispatch 0x7f8434001550 osd_map(129..129 src has 52..129) v4 2024-02-13T00:41:24.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.510+0000 7f841bfff700 3 client.30808.objecter handle_osd_map got epochs [129,129] > 0 2024-02-13T00:41:24.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.510+0000 7f841bfff700 3 client.30808.objecter handle_osd_map decoding full epoch 129 2024-02-13T00:41:24.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.510+0000 7f841bfff700 20 client.30808.objecter dump_active .. 0 homeless 2024-02-13T00:41:24.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.511+0000 7f84327fc700 1 --2- 172.21.15.162:0/1275818284 >> v2:172.21.15.162:6800/2866534053 conn(0x7f841c062c90 0x7f841c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:24.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.512+0000 7f84327fc700 1 --2- 172.21.15.162:0/1275818284 >> v2:172.21.15.162:6800/2866534053 conn(0x7f841c062c90 0x7f841c065150 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f8428005010 tx=0x7f8428009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:24.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.512+0000 7f841bfff700 10 client.30808.objecter ms_handle_connect 0x7f841c062c90 2024-02-13T00:41:24.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.512+0000 7f8439480700 1 -- 172.21.15.162:0/1275818284 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84340475e0 con 0x7f84340fa6e0 2024-02-13T00:41:24.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.520+0000 7f841bfff700 1 -- 172.21.15.162:0/1275818284 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f84240507c0 con 0x7f84340fa6e0 2024-02-13T00:41:24.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:24.822 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:24.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.937+0000 7f8439480700 1 -- 172.21.15.162:0/1275818284 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f8434109510 con 0x7f84340fa6e0 2024-02-13T00:41:24.955 INFO:tasks.workunit.client.0.smithi162.stdout: 61.051% pgs not active 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 569 creating+peering 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 192 unknown 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 171 active+clean 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: io: 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: client: 1.1 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4806 ms) 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ RUN ] LibRadosService.Status 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ OK ] LibRadosService.Status (20056 ms) 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [----------] 4 tests from LibRadosService (30190 ms total) 2024-02-13T00:41:24.956 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-02-13T00:41:24.957 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [----------] Global test environment tear-down 2024-02-13T00:41:24.957 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [==========] 4 tests from 1 test suite ran. (30190 ms total) 2024-02-13T00:41:24.957 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ PASSED ] 4 tests. 2024-02-13T00:41:24.964 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:24.983 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:41:24.983 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:41:24.983 INFO:teuthology.orchestra.run.smithi162.stdout: "epoch": 130, 2024-02-13T00:41:24.983 INFO:teuthology.orchestra.run.smithi162.stdout: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:41:24.983 INFO:teuthology.orchestra.run.smithi162.stdout: "created": "2024-02-13T00:31:51.797017+0000", 2024-02-13T00:41:24.983 INFO:teuthology.orchestra.run.smithi162.stdout: "modified": "2024-02-13T00:41:24.817969+0000", 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "last_up_change": "2024-02-13T00:37:33.855829+0000", 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "last_in_change": "2024-02-13T00:37:16.202456+0000", 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_num": 5806080, 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_set": [ 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "nodeep-scrub", 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "noscrub", 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "pglog_hardlimit", 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "purged_snapdirs", 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_deletes", 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "sortbitwise" 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:41:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_version": 27, 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "full_ratio": 0.94999998807907104, 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_snapshot": "", 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_max": 74, 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "max_osd": 8, 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "require_min_compat_client": "luminous", 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "min_compat_client": "luminous", 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "require_osd_release": "pacific", 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "pools": [ 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 1, 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "device_health_metrics", 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:35:34.545390+0000", 2024-02-13T00:41:24.985 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 1, 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 1, 2024-02-13T00:41:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 1, 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 1, 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 1, 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "71", 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_max": 32, 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_min": 1 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_devicehealth": {} 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 2, 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": ".rgw.root", 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.459980+0000", 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "58", 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:41:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 3, 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "iscsi", 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.740219+0000", 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 3, 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 3, 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 3, 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 3, 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 3, 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "102", 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd": {} 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 4, 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.log", 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:41.489292+0000", 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "60", 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 5, 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.control", 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:43.466304+0000", 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "62", 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 6, 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.meta", 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:45.542442+0000", 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:41:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "64", 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:41:25.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.966+0000 7f841bfff700 1 -- 172.21.15.162:0/1275818284 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v130) v1 ==== 81+0+65999 (secure 0 0 0) 0x7f84240161b0 con 0x7f84340fa6e0 2024-02-13T00:41:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_bias": 4 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 18, 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "LibRadosList_smithi162-84017-1", 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:40:57.711006+0000", 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "108", 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 20, 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "LibRadosMiscPP_smithi162-84049-1", 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:40:58.193277+0000", 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "108", 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 28, 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:00.190777+0000", 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "108", 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 45, 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "LibRadosMisc_smithi162-84041-1", 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:07.234352+0000", 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "120", 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "app1": { 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "key2": "value2" 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "app2": {}, 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 51, 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", 2024-02-13T00:41:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:09.587046+0000", 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "119", 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "119", 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "119", 2024-02-13T00:41:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "119", 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 118, 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [ 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: 55 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": 55, 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": 55, 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 55, 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:11.386730+0000", 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 9, 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,incomplete_clones", 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 4, 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 4, 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 4, 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 4, 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 4, 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "120", 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "119", 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "119", 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "119", 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 118, 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": 51, 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "writeback", 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:41:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 61, 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-6", 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:14.963279+0000", 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.971+0000 7f8439480700 10 librados: watch_flush enter 2024-02-13T00:41:25.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.971+0000 7f8439480700 10 librados: watch_flush exit 2024-02-13T00:41:25.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.971+0000 7f8439480700 20 client.30808.objecter shutdown clearing up homeless session... 2024-02-13T00:41:25.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.971+0000 7f8439480700 10 client.30808.objecter successfully canceled tick 2024-02-13T00:41:25.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.972+0000 7f8439480700 1 -- 172.21.15.162:0/1275818284 >> v2:172.21.15.162:6800/2866534053 conn(0x7f841c062c90 msgr2=0x7f841c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:25.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.972+0000 7f8439480700 1 --2- 172.21.15.162:0/1275818284 >> v2:172.21.15.162:6800/2866534053 conn(0x7f841c062c90 0x7f841c065150 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f8428005010 tx=0x7f8428009cb0).stop 2024-02-13T00:41:25.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.972+0000 7f8439480700 1 -- 172.21.15.162:0/1275818284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84340fa6e0 msgr2=0x7f8434110600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:25.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.972+0000 7f8439480700 1 --2- 172.21.15.162:0/1275818284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84340fa6e0 0x7f8434110600 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7f842400b530 tx=0x7f84240090f0).stop 2024-02-13T00:41:25.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.973+0000 7f8439480700 1 -- 172.21.15.162:0/1275818284 shutdown_connections 2024-02-13T00:41:25.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.973+0000 7f8439480700 1 --2- 172.21.15.162:0/1275818284 >> v2:172.21.15.162:6800/2866534053 conn(0x7f841c062c90 0x7f841c065150 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:25.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.973+0000 7f8439480700 1 --2- 172.21.15.162:0/1275818284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84340fc620 0x7f843410a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:25.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.973+0000 7f8439480700 1 --2- 172.21.15.162:0/1275818284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84340fa6e0 0x7f8434110600 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:25.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.973+0000 7f8439480700 1 --2- 172.21.15.162:0/1275818284 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84340fbd50 0x7f8434110b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:25.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.973+0000 7f8439480700 1 -- 172.21.15.162:0/1275818284 >> 172.21.15.162:0/1275818284 conn(0x7f84340f5aa0 msgr2=0x7f84340f6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:25.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.975+0000 7f8439480700 1 -- 172.21.15.162:0/1275818284 shutdown_connections 2024-02-13T00:41:25.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.975+0000 7f8439480700 1 -- 172.21.15.162:0/1275818284 wait complete. 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "128", 2024-02-13T00:41:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "127", 2024-02-13T00:41:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "127", 2024-02-13T00:41:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "127", 2024-02-13T00:41:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 125, 2024-02-13T00:41:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 66, 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "LibRadosSnapshotsSelfManagedPP_smithi162-84081-7", 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:18.205544+0000", 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8193, 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "130", 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 6, 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 130, 2024-02-13T00:41:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 72, 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "RemoveTest_smithi162-83977-8", 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:23.215997+0000", 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 32769, 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,creating", 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "130", 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 73, 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-9", 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:24.346387+0000", 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 32769, 2024-02-13T00:41:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,creating", 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "130", 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 74, 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84065-7", 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:24.806992+0000", 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 32769, 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,creating", 2024-02-13T00:41:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:41:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "130", 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:41:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:41:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:41:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99", 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:41:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 8, 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 129, 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6801", 2024-02-13T00:41:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6802", 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6804", 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6803", 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6801/2103743338", 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6802/2103743338", 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6804/2103743338", 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6803/2103743338", 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886", 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:41:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 13, 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 129, 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6805", 2024-02-13T00:41:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6806", 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:41:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6808", 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6807", 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6805/150599780", 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6806/150599780", 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6808/150599780", 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6807/150599780", 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e", 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:41:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 18, 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 129, 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6809", 2024-02-13T00:41:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6810", 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:41:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6812", 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6811", 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:41:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6809/3689719376", 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6810/3689719376", 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6812/3689719376", 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6811/3689719376", 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:41:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001", 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 25, 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 129, 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6813", 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6814", 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6816", 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6815", 2024-02-13T00:41:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6813/604971011", 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6814/604971011", 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6816/604971011", 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6815/604971011", 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3", 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 31, 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 81, 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6800", 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6801", 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6803", 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6802", 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6800/3344263388", 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6801/3344263388", 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6803/3344263388", 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6802/3344263388", 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723", 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 39, 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 129, 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6804", 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6805", 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6807", 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6806", 2024-02-13T00:41:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6804/2212296748", 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6805/2212296748", 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2212296748", 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2212296748", 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474", 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 45, 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 129, 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6808", 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6809", 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6811", 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6810", 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6808/1506640341", 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6809/1506640341", 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1506640341", 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1506640341", 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b", 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 53, 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 129, 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:41:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6812", 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6813", 2024-02-13T00:41:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6815", 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:41:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6814", 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6812/1964192772", 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6813/1964192772", 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6815/1964192772", 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6814/1964192772", 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_xinfo": [ 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:40:48.185479+0000", 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:40:50.400426+0000", 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:41:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:40:21.408117+0000", 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:40:05.003245+0000", 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:41:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:41:11.303912+0000", 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:41:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:40:03.350082+0000", 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:41:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:40:42.541395+0000", 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:41:23.730592+0000", 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap": [], 2024-02-13T00:41:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap_items": [ 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "2.0", 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 7, 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 2 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.2", 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:41:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 2 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.6", 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 5 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.13", 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 2 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_temp": [], 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_temp": [], 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "blocklist": { 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/1909610488": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2370739667": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:41:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/1502590187": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/4121673848": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2412382714": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3041562701": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/2775563314": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3135632190": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/197307636": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2653247729": "2024-02-14T00:32:24.380625+0000" 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "range_blocklist": {}, 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profiles": { 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "default": { 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:41:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van" 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps_queue": [ 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 24, 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:41:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 9, 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 25, 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 60, 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 66, 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:41:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "new_removed_snaps": [], 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "new_purged_snaps": [], 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_node_flags": {}, 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "device_class_flags": {}, 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode": { 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_enabled": false, 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_bucket_count": 0, 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "degraded_stretch_mode": 0, 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "recovering_stretch_mode": 0, 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_bucket": 0 2024-02-13T00:41:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:25.123 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:41:25.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:24.985+0000 7f8439480700 1 librados: shutdown 2024-02-13T00:41:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:25.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:25.124 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:23.702847+0000 osd.1 (osd.1) 69 : cluster [DBG] 20.6 deep-scrub starts 2024-02-13T00:41:25.124 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:23.730552+0000 osd.7 (osd.7) 72 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:25.124 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:23.730590+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:25.124 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:23.761291+0000 osd.1 (osd.1) 70 : cluster [DBG] 20.6 deep-scrub ok 2024-02-13T00:41:25.124 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:23.769372+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:25.124 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:23.769518+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-13T00:41:25.125 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:23.775453+0000 mon.b (mon.1) 119 : audit [INF] from='client.? 172.21.15.162:0/3524561227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-83977-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:25.125 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:23.867624+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-83977-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:25.125 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:23.959302+0000 mon.b (mon.1) 120 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:25.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:23.702847+0000 osd.1 (osd.1) 69 : cluster [DBG] 20.6 deep-scrub starts 2024-02-13T00:41:25.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:23.730552+0000 osd.7 (osd.7) 72 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:25.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:23.730590+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:25.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:23.761291+0000 osd.1 (osd.1) 70 : cluster [DBG] 20.6 deep-scrub ok 2024-02-13T00:41:25.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:23.769372+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:25.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:23.769518+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-13T00:41:25.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:23.775453+0000 mon.b (mon.1) 119 : audit [INF] from='client.? 172.21.15.162:0/3524561227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-83977-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:25.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:23.867624+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-83977-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:25.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:23.959302+0000 mon.b (mon.1) 120 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:25.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:25.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:23.702847+0000 osd.1 (osd.1) 69 : cluster [DBG] 20.6 deep-scrub starts 2024-02-13T00:41:25.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:23.730552+0000 osd.7 (osd.7) 72 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:25.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:23.730590+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:25.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:23.761291+0000 osd.1 (osd.1) 70 : cluster [DBG] 20.6 deep-scrub ok 2024-02-13T00:41:25.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:23.769372+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:25.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:23.769518+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-13T00:41:25.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:23.775453+0000 mon.b (mon.1) 119 : audit [INF] from='client.? 172.21.15.162:0/3524561227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-83977-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:25.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:23.867624+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-83977-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:25.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:23.959302+0000 mon.b (mon.1) 120 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:25.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:41:25.397 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:25.437 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:25.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:25.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:25.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-13T00:41:25.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:25.568 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:25.586 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:25.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:25.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:25.650 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-02-13T00:41:25.650 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [----------] Global test environment set-up. 2024-02-13T00:41:25.650 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-02-13T00:41:25.650 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-02-13T00:41:25.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5334 ms) 2024-02-13T00:41:25.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-02-13T00:41:25.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (25 ms) 2024-02-13T00:41:25.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-02-13T00:41:25.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20419 ms) 2024-02-13T00:41:25.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-02-13T00:41:25.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: attempt 0 of 20 2024-02-13T00:41:25.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5134 ms) 2024-02-13T00:41:25.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30912 ms total) 2024-02-13T00:41:25.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: 2024-02-13T00:41:25.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [----------] Global test environment tear-down 2024-02-13T00:41:25.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30912 ms total) 2024-02-13T00:41:25.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-02-13T00:41:25.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:25.742 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:25.789 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:25.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:41:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:41:25.877 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:25.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:25.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:26.096 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:26.351 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:41:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:41:26] "GET /metrics HTTP/1.1" 200 209569 "" "Prometheus/2.33.4" 2024-02-13T00:41:26.353 INFO:tasks.thrashosds.thrasher:j is {'epoch': 130, 'fsid': 'd4ee01fc-ca06-11ee-95b9-87774f69a715', 'created': '2024-02-13T00:31:51.797017+0000', 'modified': '2024-02-13T00:41:24.817969+0000', 'last_up_change': '2024-02-13T00:37:33.855829+0000', 'last_in_change': '2024-02-13T00:37:16.202456+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 74, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-13T00:35:34.545390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-13T00:37:39.459980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-13T00:37:39.740219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-13T00:37:41.489292+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-13T00:37:43.466304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-13T00:37:45.542442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 18, 'pool_name': 'LibRadosList_smithi162-84017-1', 'create_time': '2024-02-13T00:40:57.711006+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 20, 'pool_name': 'LibRadosMiscPP_smithi162-84049-1', 'create_time': '2024-02-13T00:40:58.193277+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 28, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-13T00:41:00.190777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 45, 'pool_name': 'LibRadosMisc_smithi162-84041-1', 'create_time': '2024-02-13T00:41:07.234352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'app1': {'key2': 'value2'}, 'app2': {}, 'rados': {}}}, {'pool': 51, 'pool_name': 'LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4', 'create_time': '2024-02-13T00:41:09.587046+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '119', 'last_force_op_resend_prenautilus': '119', 'last_force_op_resend_preluminous': '119', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 118, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [55], 'tier_of': -1, 'read_tier': 55, 'write_tier': 55, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 55, 'pool_name': 'LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6', 'create_time': '2024-02-13T00:41:11.386730+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '119', 'last_force_op_resend_prenautilus': '119', 'last_force_op_resend_preluminous': '119', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 118, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 51, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 61, 'pool_name': 'test-rados-api-smithi162-84057-6', 'create_time': '2024-02-13T00:41:14.963279+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '127', 'last_force_op_resend_prenautilus': '127', 'last_force_op_resend_preluminous': '127', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 125, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 66, 'pool_name': 'LibRadosSnapshotsSelfManagedPP_smithi162-84081-7', 'create_time': '2024-02-13T00:41:18.205544+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 130, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 72, 'pool_name': 'RemoveTest_smithi162-83977-8', 'create_time': '2024-02-13T00:41:23.215997+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 73, 'pool_name': 'test-rados-api-smithi162-84057-9', 'create_time': '2024-02-13T00:41:24.346387+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 74, 'pool_name': 'test-rados-api-smithi162-84065-7', 'create_time': '2024-02-13T00:41:24.806992+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6801', 'nonce': 2103743338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 2103743338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 2103743338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6803', 'nonce': 2103743338}]}, 'public_addr': '172.21.15.162:6801/2103743338', 'cluster_addr': '172.21.15.162:6802/2103743338', 'heartbeat_back_addr': '172.21.15.162:6804/2103743338', 'heartbeat_front_addr': '172.21.15.162:6803/2103743338', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '39e9d2ef-9aa4-45bb-8a47-b0e0ec222886', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6805', 'nonce': 150599780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 150599780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 150599780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6807', 'nonce': 150599780}]}, 'public_addr': '172.21.15.162:6805/150599780', 'cluster_addr': '172.21.15.162:6806/150599780', 'heartbeat_back_addr': '172.21.15.162:6808/150599780', 'heartbeat_front_addr': '172.21.15.162:6807/150599780', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6809', 'nonce': 3689719376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3689719376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3689719376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6811', 'nonce': 3689719376}]}, 'public_addr': '172.21.15.162:6809/3689719376', 'cluster_addr': '172.21.15.162:6810/3689719376', 'heartbeat_back_addr': '172.21.15.162:6812/3689719376', 'heartbeat_front_addr': '172.21.15.162:6811/3689719376', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e6b9f0e-7a18-4cf8-9331-1a973fadc001', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6813', 'nonce': 604971011}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 604971011}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 604971011}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6815', 'nonce': 604971011}]}, 'public_addr': '172.21.15.162:6813/604971011', 'cluster_addr': '172.21.15.162:6814/604971011', 'heartbeat_back_addr': '172.21.15.162:6816/604971011', 'heartbeat_front_addr': '172.21.15.162:6815/604971011', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 3344263388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 3344263388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 3344263388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 3344263388}]}, 'public_addr': '172.21.15.181:6800/3344263388', 'cluster_addr': '172.21.15.181:6801/3344263388', 'heartbeat_back_addr': '172.21.15.181:6803/3344263388', 'heartbeat_front_addr': '172.21.15.181:6802/3344263388', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6aa2bd68-7f4b-4973-ac49-7fbc57c1a723', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2212296748}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2212296748}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2212296748}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2212296748}]}, 'public_addr': '172.21.15.181:6804/2212296748', 'cluster_addr': '172.21.15.181:6805/2212296748', 'heartbeat_back_addr': '172.21.15.181:6807/2212296748', 'heartbeat_front_addr': '172.21.15.181:6806/2212296748', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '791ba814-5284-432e-addc-d6b7f5f3d474', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1506640341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1506640341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1506640341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1506640341}]}, 'public_addr': '172.21.15.181:6808/1506640341', 'cluster_addr': '172.21.15.181:6809/1506640341', 'heartbeat_back_addr': '172.21.15.181:6811/1506640341', 'heartbeat_front_addr': '172.21.15.181:6810/1506640341', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f369c65-e663-4eb2-87a5-f2900f15b35b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 1964192772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 1964192772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 1964192772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 1964192772}]}, 'public_addr': '172.21.15.181:6812/1964192772', 'cluster_addr': '172.21.15.181:6813/1964192772', 'heartbeat_back_addr': '172.21.15.181:6815/1964192772', 'heartbeat_front_addr': '172.21.15.181:6814/1964192772', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:40:48.185479+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:40:50.400426+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:40:21.408117+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:40:05.003245+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:41:11.303912+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:40:03.350082+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:40:42.541395+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:41:23.730592+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.162:6800/1909610488': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2370739667': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/1502590187': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/4121673848': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:0/2412382714': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/3041562701': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/2775563314': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/3135632190': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/197307636': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2653247729': '2024-02-14T00:32:24.380625+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 24, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 60, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 66, 'snaps': [{'begin': 2, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-13T00:41:26.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:41:26.741 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:26.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:26.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:27.142 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:27.183 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:27.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:27.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:41:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:24.234958+0000 mgr.y (mgr.14544) 159 : cluster [DBG] pgmap v164: 456 pgs: 64 unknown, 1 active+clean+snaptrim, 391 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:24.867570+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-83977-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:24.867801+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-13T00:41:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:24.896981+0000 mon.b (mon.1) 121 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:24.919094+0000 mon.c (mon.2) 125 : audit [INF] from='client.? 172.21.15.162:0/2465425618' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:24.939252+0000 mon.b (mon.1) 122 : audit [DBG] from='client.? 172.21.15.162:0/1275818284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:41:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:24.966174+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:24.966812+0000 mon.b (mon.1) 123 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.000370+0000 mon.a (mon.0) 1036 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.338803+0000 mgr.y (mgr.14544) 160 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:27.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.410369+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:27.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.427844+0000 mon.a (mon.0) 1038 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:27.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.440104+0000 mon.a (mon.0) 1039 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:27.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.626052+0000 mon.b (mon.1) 124 : audit [DBG] from='client.? 172.21.15.162:0/20277865' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-13T00:41:27.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.626439+0000 mgr.y (mgr.14544) 161 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-13T00:41:27.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.925492+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:27.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.925565+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:27.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.925719+0000 mon.a (mon.0) 1042 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:27.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:25.925777+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-13T00:41:27.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.930715+0000 mon.b (mon.1) 125 : audit [INF] from='client.? 172.21.15.162:0/3145686446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.937670+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.950956+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-84073-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:27.319 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:27.338 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:41:27.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:24.234958+0000 mgr.y (mgr.14544) 159 : cluster [DBG] pgmap v164: 456 pgs: 64 unknown, 1 active+clean+snaptrim, 391 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:27.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:24.867570+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-83977-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:27.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:24.867801+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-13T00:41:27.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:24.896981+0000 mon.b (mon.1) 121 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:24.919094+0000 mon.c (mon.2) 125 : audit [INF] from='client.? 172.21.15.162:0/2465425618' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:24.939252+0000 mon.b (mon.1) 122 : audit [DBG] from='client.? 172.21.15.162:0/1275818284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:41:27.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:24.966174+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:24.966812+0000 mon.b (mon.1) 123 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:27.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.000370+0000 mon.a (mon.0) 1036 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.338803+0000 mgr.y (mgr.14544) 160 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:27.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.410369+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:27.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.427844+0000 mon.a (mon.0) 1038 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:27.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.440104+0000 mon.a (mon.0) 1039 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:27.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.626052+0000 mon.b (mon.1) 124 : audit [DBG] from='client.? 172.21.15.162:0/20277865' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-13T00:41:27.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.626439+0000 mgr.y (mgr.14544) 161 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-13T00:41:27.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.925492+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:27.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.925565+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:27.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.925719+0000 mon.a (mon.0) 1042 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:27.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:25.925777+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-13T00:41:27.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.930715+0000 mon.b (mon.1) 125 : audit [INF] from='client.? 172.21.15.162:0/3145686446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.937670+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.950956+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-84073-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:27.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:24.234958+0000 mgr.y (mgr.14544) 159 : cluster [DBG] pgmap v164: 456 pgs: 64 unknown, 1 active+clean+snaptrim, 391 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:27.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:24.867570+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-83977-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:27.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:24.867801+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-13T00:41:27.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:24.896981+0000 mon.b (mon.1) 121 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:24.919094+0000 mon.c (mon.2) 125 : audit [INF] from='client.? 172.21.15.162:0/2465425618' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:24.939252+0000 mon.b (mon.1) 122 : audit [DBG] from='client.? 172.21.15.162:0/1275818284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:24.966174+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:24.966812+0000 mon.b (mon.1) 123 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.000370+0000 mon.a (mon.0) 1036 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.338803+0000 mgr.y (mgr.14544) 160 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.410369+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.427844+0000 mon.a (mon.0) 1038 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.440104+0000 mon.a (mon.0) 1039 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.626052+0000 mon.b (mon.1) 124 : audit [DBG] from='client.? 172.21.15.162:0/20277865' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.626439+0000 mgr.y (mgr.14544) 161 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.925492+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.925565+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:27.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.925719+0000 mon.a (mon.0) 1042 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:27.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:25.925777+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-13T00:41:27.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.930715+0000 mon.b (mon.1) 125 : audit [INF] from='client.? 172.21.15.162:0/3145686446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.937670+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:27.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.950956+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-84073-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:27.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:27.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:41:27.446 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:27.473 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:27.492 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:27.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:27.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:41:27.622 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:27.641 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:41:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:27.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:41:27.663 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:27.669 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:27.744 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:27.788 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:27.902 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:27.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:28.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:28.039 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:28.057 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:41:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:28.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:41:28.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:28.260 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:28.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:25.171914+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:25.171978+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:25.974874+0000 mon.b (mon.1) 126 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:26.092121+0000 mon.b (mon.1) 127 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:26.235531+0000 mgr.y (mgr.14544) 162 : cluster [DBG] pgmap v167: 456 pgs: 64 creating+peering, 32 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-13T00:41:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:26.982471+0000 mon.b (mon.1) 128 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:27.021906+0000 mon.c (mon.2) 126 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4"}]: dispatch 2024-02-13T00:41:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:27.045340+0000 mon.a (mon.0) 1046 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:27.050585+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4"}]: dispatch 2024-02-13T00:41:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:27.074022+0000 mon.a (mon.0) 1048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:27.362628+0000 osd.3 (osd.3) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:28.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:27.362683+0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:28.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:27.526176+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-13T00:41:28.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:27.531364+0000 mon.b (mon.1) 129 : audit [INF] from='client.? 172.21.15.162:0/4131992107' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-83977-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:28.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:27.547783+0000 mon.a (mon.0) 1050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-83977-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:28.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:27.986356+0000 mon.b (mon.1) 130 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:28.424 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:25.171914+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:25.171978+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:25.974874+0000 mon.b (mon.1) 126 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:26.092121+0000 mon.b (mon.1) 127 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:26.235531+0000 mgr.y (mgr.14544) 162 : cluster [DBG] pgmap v167: 456 pgs: 64 creating+peering, 32 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:26.982471+0000 mon.b (mon.1) 128 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:27.021906+0000 mon.c (mon.2) 126 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4"}]: dispatch 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:27.045340+0000 mon.a (mon.0) 1046 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:27.050585+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4"}]: dispatch 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:27.074022+0000 mon.a (mon.0) 1048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:27.362628+0000 osd.3 (osd.3) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:27.362683+0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:27.526176+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-13T00:41:28.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:27.531364+0000 mon.b (mon.1) 129 : audit [INF] from='client.? 172.21.15.162:0/4131992107' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-83977-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:28.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:27.547783+0000 mon.a (mon.0) 1050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-83977-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:28.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:27.986356+0000 mon.b (mon.1) 130 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:28.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:25.171914+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:28.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:25.171978+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:28.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:25.974874+0000 mon.b (mon.1) 126 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:28.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:26.092121+0000 mon.b (mon.1) 127 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:28.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:26.235531+0000 mgr.y (mgr.14544) 162 : cluster [DBG] pgmap v167: 456 pgs: 64 creating+peering, 32 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-13T00:41:28.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:26.982471+0000 mon.b (mon.1) 128 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:28.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:27.021906+0000 mon.c (mon.2) 126 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4"}]: dispatch 2024-02-13T00:41:28.427 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:27.045340+0000 mon.a (mon.0) 1046 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:28.427 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:27.050585+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4"}]: dispatch 2024-02-13T00:41:28.427 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:27.074022+0000 mon.a (mon.0) 1048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-83985-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:28.427 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:27.362628+0000 osd.3 (osd.3) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:28.427 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:27.362683+0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:28.427 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:27.526176+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-13T00:41:28.427 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:27.531364+0000 mon.b (mon.1) 129 : audit [INF] from='client.? 172.21.15.162:0/4131992107' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-83977-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:28.427 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:27.547783+0000 mon.a (mon.0) 1050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-83977-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:28.427 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:27.986356+0000 mon.b (mon.1) 130 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:29.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:41:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:41:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:41:29.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:28.186825+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-84073-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]': finished 2024-02-13T00:41:29.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:28.186899+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:29.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:28.186961+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4"}]': finished 2024-02-13T00:41:29.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:28.187155+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-83977-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:29.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:28.187212+0000 mon.a (mon.0) 1055 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-13T00:41:29.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:28.190967+0000 mon.b (mon.1) 131 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-9"}]: dispatch 2024-02-13T00:41:29.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:28.193585+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-9"}]: dispatch 2024-02-13T00:41:29.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:28.206738+0000 mon.c (mon.2) 127 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]: dispatch 2024-02-13T00:41:29.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:28.224548+0000 mon.a (mon.0) 1057 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]: dispatch 2024-02-13T00:41:29.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:28.247404+0000 mgr.y (mgr.14544) 163 : cluster [DBG] pgmap v170: 464 pgs: 64 creating+peering, 40 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-13T00:41:29.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:28.988894+0000 mon.b (mon.1) 132 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:29.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:28.186825+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-84073-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]': finished 2024-02-13T00:41:29.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:28.186899+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:29.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:28.186961+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4"}]': finished 2024-02-13T00:41:29.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:28.187155+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-83977-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:29.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:28.187212+0000 mon.a (mon.0) 1055 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-13T00:41:29.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:28.190967+0000 mon.b (mon.1) 131 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-9"}]: dispatch 2024-02-13T00:41:29.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:28.193585+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-9"}]: dispatch 2024-02-13T00:41:29.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:28.206738+0000 mon.c (mon.2) 127 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]: dispatch 2024-02-13T00:41:29.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:28.224548+0000 mon.a (mon.0) 1057 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]: dispatch 2024-02-13T00:41:29.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:28.247404+0000 mgr.y (mgr.14544) 163 : cluster [DBG] pgmap v170: 464 pgs: 64 creating+peering, 40 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-13T00:41:29.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:28.988894+0000 mon.b (mon.1) 132 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:29.489 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:29.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:29.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:28.186825+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-84073-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]': finished 2024-02-13T00:41:29.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:28.186899+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:28.186961+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4"}]': finished 2024-02-13T00:41:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:28.187155+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-83977-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:28.187212+0000 mon.a (mon.0) 1055 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-13T00:41:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:28.190967+0000 mon.b (mon.1) 131 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-9"}]: dispatch 2024-02-13T00:41:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:28.193585+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-9"}]: dispatch 2024-02-13T00:41:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:28.206738+0000 mon.c (mon.2) 127 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]: dispatch 2024-02-13T00:41:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:28.224548+0000 mon.a (mon.0) 1057 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]: dispatch 2024-02-13T00:41:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:28.247404+0000 mgr.y (mgr.14544) 163 : cluster [DBG] pgmap v170: 464 pgs: 64 creating+peering, 40 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-13T00:41:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:28.988894+0000 mon.b (mon.1) 132 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:29.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:41:29.621 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:29.640 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:41:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:29.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:41:29.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:41:29.770 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:29.789 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:41:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:29.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:41:29.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:29.950 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:30.099 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:30.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:30.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:30.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:30.307 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:30.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:29.197511+0000 mon.a (mon.0) 1058 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:41:30.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:29.224495+0000 mon.a (mon.0) 1059 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-9"}]': finished 2024-02-13T00:41:30.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:29.224995+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]': finished 2024-02-13T00:41:30.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:29.225186+0000 mon.a (mon.0) 1061 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-13T00:41:30.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:29.228811+0000 mon.b (mon.1) 133 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-9", "mode": "writeback"}]: dispatch 2024-02-13T00:41:30.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:29.239235+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-9", "mode": "writeback"}]: dispatch 2024-02-13T00:41:30.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:29.240868+0000 mon.b (mon.1) 134 : audit [INF] from='client.? 172.21.15.162:0/2482738790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-83985-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:30.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:29.241508+0000 mon.a (mon.0) 1063 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-83985-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:30.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:29.265809+0000 mon.c (mon.2) 128 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-13T00:41:30.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:29.276736+0000 mon.a (mon.0) 1064 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-13T00:41:30.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:29.990151+0000 mon.b (mon.1) 135 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:30.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:29.197511+0000 mon.a (mon.0) 1058 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:41:30.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:29.224495+0000 mon.a (mon.0) 1059 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-9"}]': finished 2024-02-13T00:41:30.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:29.224995+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]': finished 2024-02-13T00:41:30.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:29.225186+0000 mon.a (mon.0) 1061 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-13T00:41:30.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:29.228811+0000 mon.b (mon.1) 133 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-9", "mode": "writeback"}]: dispatch 2024-02-13T00:41:30.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:29.239235+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-9", "mode": "writeback"}]: dispatch 2024-02-13T00:41:30.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:29.240868+0000 mon.b (mon.1) 134 : audit [INF] from='client.? 172.21.15.162:0/2482738790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-83985-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:30.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:29.241508+0000 mon.a (mon.0) 1063 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-83985-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:30.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:29.265809+0000 mon.c (mon.2) 128 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-13T00:41:30.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:29.276736+0000 mon.a (mon.0) 1064 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-13T00:41:30.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:29.990151+0000 mon.b (mon.1) 135 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:30.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:29.197511+0000 mon.a (mon.0) 1058 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:41:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:29.224495+0000 mon.a (mon.0) 1059 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-9"}]': finished 2024-02-13T00:41:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:29.224995+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6"}]': finished 2024-02-13T00:41:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:29.225186+0000 mon.a (mon.0) 1061 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-13T00:41:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:29.228811+0000 mon.b (mon.1) 133 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-9", "mode": "writeback"}]: dispatch 2024-02-13T00:41:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:29.239235+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-9", "mode": "writeback"}]: dispatch 2024-02-13T00:41:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:29.240868+0000 mon.b (mon.1) 134 : audit [INF] from='client.? 172.21.15.162:0/2482738790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-83985-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:29.241508+0000 mon.a (mon.0) 1063 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-83985-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:29.265809+0000 mon.c (mon.2) 128 : audit [INF] from='client.? 172.21.15.162:0/3953551643' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-13T00:41:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:29.276736+0000 mon.a (mon.0) 1064 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-13T00:41:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:29.990151+0000 mon.b (mon.1) 135 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:31.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.300+0000 7f00ec6f5700 1 -- 172.21.15.162:0/1894948222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00e40f73a0 msgr2=0x7f00e41039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:31.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.300+0000 7f00ec6f5700 1 --2- 172.21.15.162:0/1894948222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00e40f73a0 0x7f00e41039a0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7f00dc009be0 tx=0x7f00dc004d10).stop 2024-02-13T00:41:31.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.300+0000 7f00ec6f5700 1 -- 172.21.15.162:0/1894948222 shutdown_connections 2024-02-13T00:41:31.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.300+0000 7f00ec6f5700 1 --2- 172.21.15.162:0/1894948222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00e40f73a0 0x7f00e41039a0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:31.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.300+0000 7f00ec6f5700 1 --2- 172.21.15.162:0/1894948222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00e40fcb30 0x7f00e40fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:31.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.300+0000 7f00ec6f5700 1 --2- 172.21.15.162:0/1894948222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00e40fd490 0x7f00e40f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.300+0000 7f00ec6f5700 1 -- 172.21.15.162:0/1894948222 >> 172.21.15.162:0/1894948222 conn(0x7f00e406eb90 msgr2=0x7f00e40f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.300+0000 7f00ec6f5700 1 -- 172.21.15.162:0/1894948222 shutdown_connections 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.300+0000 7f00ec6f5700 1 -- 172.21.15.162:0/1894948222 wait complete. 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.301+0000 7f00ec6f5700 1 librados: starting msgr at 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.301+0000 7f00ec6f5700 1 librados: starting objecter 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.301+0000 7f00ec6f5700 1 Processor -- start 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.301+0000 7f00ec6f5700 1 -- start start 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.301+0000 7f00ec6f5700 1 librados: setting wanted keys 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.301+0000 7f00ec6f5700 1 librados: calling monclient init 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.301+0000 7f00ec6f5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00e40f73a0 0x7f00e4062a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.301+0000 7f00ec6f5700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00e40fcb30 0x7f00e4062f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.301+0000 7f00ec6f5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00e40fd490 0x7f00e406a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.302+0000 7f00ec6f5700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e40ff3c0 con 0x7f00e40fd490 2024-02-13T00:41:31.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.302+0000 7f00ec6f5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e40ff530 con 0x7f00e40f73a0 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.302+0000 7f00ec6f5700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00e40ff6f0 con 0x7f00e40fcb30 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.302+0000 7f00ea491700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00e40f73a0 0x7f00e4062a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.302+0000 7f00ea491700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00e40f73a0 0x7f00e4062a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:56186/0 (socket says 172.21.15.162:56186) 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.302+0000 7f00ea491700 1 -- 172.21.15.162:0/2729799896 learned_addr learned my addr 172.21.15.162:0/2729799896 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.305+0000 7f00e9c90700 1 --2- 172.21.15.162:0/2729799896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00e40fcb30 0x7f00e4062f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.305+0000 7f00ea491700 1 -- 172.21.15.162:0/2729799896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00e40fcb30 msgr2=0x7f00e4062f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.305+0000 7f00ea491700 1 --2- 172.21.15.162:0/2729799896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00e40fcb30 0x7f00e4062f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.305+0000 7f00ea491700 1 -- 172.21.15.162:0/2729799896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00e40fd490 msgr2=0x7f00e406a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.305+0000 7f00ea491700 1 --2- 172.21.15.162:0/2729799896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00e40fd490 0x7f00e406a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.305+0000 7f00ea491700 1 -- 172.21.15.162:0/2729799896 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00dc0097c0 con 0x7f00e40f73a0 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.305+0000 7f00e9c90700 1 --2- 172.21.15.162:0/2729799896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00e40fcb30 0x7f00e4062f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.305+0000 7f00ea491700 1 --2- 172.21.15.162:0/2729799896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00e40f73a0 0x7f00e4062a40 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 rx=0x7f00e000b530 tx=0x7f00e00090f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.305+0000 7f00db7fe700 10 client.?.objecter ms_handle_connect 0x7f00e40f73a0 2024-02-13T00:41:31.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.305+0000 7f00db7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.306+0000 7f00db7fe700 1 -- 172.21.15.162:0/2729799896 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e000d4a0 con 0x7f00e40f73a0 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.306+0000 7f00db7fe700 1 -- 172.21.15.162:0/2729799896 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00e000dad0 con 0x7f00e40f73a0 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.306+0000 7f00db7fe700 1 -- 172.21.15.162:0/2729799896 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e0012a50 con 0x7f00e40f73a0 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.306+0000 7f00ec6f5700 1 -- 172.21.15.162:0/2729799896 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e40ff9d0 con 0x7f00e40f73a0 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.306+0000 7f00ec6f5700 10 client.34567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.306+0000 7f00ec6f5700 1 -- 172.21.15.162:0/2729799896 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00e40ffe70 con 0x7f00e40f73a0 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.306+0000 7f00ec6f5700 1 librados: init done 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.311+0000 7f00ec6f5700 1 -- 172.21.15.162:0/2729799896 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c8004f50 con 0x7f00e40f73a0 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.319+0000 7f00db7fe700 1 -- 172.21.15.162:0/2729799896 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f00e000dc40 con 0x7f00e40f73a0 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.320+0000 7f00db7fe700 1 --2- 172.21.15.162:0/2729799896 >> v2:172.21.15.162:6800/2866534053 conn(0x7f00d0062d20 0x7f00d00651e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.320+0000 7f00db7fe700 1 -- 172.21.15.162:0/2729799896 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(135..135 src has 52..135) v4 ==== 11589+0+0 (secure 0 0 0) 0x7f00e007dde0 con 0x7f00e40f73a0 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.320+0000 7f00db7fe700 10 client.34567.objecter ms_dispatch 0x7f00e4001550 osd_map(135..135 src has 52..135) v4 2024-02-13T00:41:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.320+0000 7f00db7fe700 3 client.34567.objecter handle_osd_map got epochs [135,135] > 0 2024-02-13T00:41:31.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.320+0000 7f00db7fe700 3 client.34567.objecter handle_osd_map decoding full epoch 135 2024-02-13T00:41:31.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.320+0000 7f00db7fe700 20 client.34567.objecter dump_active .. 0 homeless 2024-02-13T00:41:31.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.320+0000 7f00db7fe700 1 -- 172.21.15.162:0/2729799896 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f00e0082400 con 0x7f00e40f73a0 2024-02-13T00:41:31.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.326+0000 7f00e9c90700 1 --2- 172.21.15.162:0/2729799896 >> v2:172.21.15.162:6800/2866534053 conn(0x7f00d0062d20 0x7f00d00651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:31.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.326+0000 7f00e9c90700 1 --2- 172.21.15.162:0/2729799896 >> v2:172.21.15.162:6800/2866534053 conn(0x7f00d0062d20 0x7f00d00651e0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0x7f00e4100b60 tx=0x7f00d4008040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:31.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.326+0000 7f00db7fe700 10 client.34567.objecter ms_handle_connect 0x7f00d0062d20 2024-02-13T00:41:31.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:30.718+0000 7f00ec6f5700 1 -- 172.21.15.162:0/2729799896 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f00c8005200 con 0x7f00e40f73a0 2024-02-13T00:41:31.208 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:31.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:31.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.303+0000 7f00db7fe700 1 -- 172.21.15.162:0/2729799896 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v136) v1 ==== 91+0+0 (secure 0 0 0) 0x7f00e00504e0 con 0x7f00e40f73a0 2024-02-13T00:41:31.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.309+0000 7f00ec6f5700 10 librados: watch_flush enter 2024-02-13T00:41:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.309+0000 7f00ec6f5700 10 librados: watch_flush exit 2024-02-13T00:41:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.309+0000 7f00ec6f5700 20 client.34567.objecter shutdown clearing up homeless session... 2024-02-13T00:41:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.309+0000 7f00ec6f5700 10 client.34567.objecter successfully canceled tick 2024-02-13T00:41:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.310+0000 7f00ec6f5700 1 -- 172.21.15.162:0/2729799896 >> v2:172.21.15.162:6800/2866534053 conn(0x7f00d0062d20 msgr2=0x7f00d00651e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.310+0000 7f00ec6f5700 1 --2- 172.21.15.162:0/2729799896 >> v2:172.21.15.162:6800/2866534053 conn(0x7f00d0062d20 0x7f00d00651e0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0x7f00e4100b60 tx=0x7f00d4008040).stop 2024-02-13T00:41:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.310+0000 7f00ec6f5700 1 -- 172.21.15.162:0/2729799896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00e40f73a0 msgr2=0x7f00e4062a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.310+0000 7f00ec6f5700 1 --2- 172.21.15.162:0/2729799896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00e40f73a0 0x7f00e4062a40 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 rx=0x7f00e000b530 tx=0x7f00e00090f0).stop 2024-02-13T00:41:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.316+0000 7f00ec6f5700 1 -- 172.21.15.162:0/2729799896 shutdown_connections 2024-02-13T00:41:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.316+0000 7f00ec6f5700 1 --2- 172.21.15.162:0/2729799896 >> v2:172.21.15.162:6800/2866534053 conn(0x7f00d0062d20 0x7f00d00651e0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.317+0000 7f00ec6f5700 1 --2- 172.21.15.162:0/2729799896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00e40fd490 0x7f00e406a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.317+0000 7f00ec6f5700 1 --2- 172.21.15.162:0/2729799896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00e40f73a0 0x7f00e4062a40 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.317+0000 7f00ec6f5700 1 --2- 172.21.15.162:0/2729799896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00e40fcb30 0x7f00e4062f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.317+0000 7f00ec6f5700 1 -- 172.21.15.162:0/2729799896 >> 172.21.15.162:0/2729799896 conn(0x7f00e406eb90 msgr2=0x7f00e406fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.318+0000 7f00ec6f5700 1 -- 172.21.15.162:0/2729799896 shutdown_connections 2024-02-13T00:41:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.318+0000 7f00ec6f5700 1 -- 172.21.15.162:0/2729799896 wait complete. 2024-02-13T00:41:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:31.318+0000 7f00ec6f5700 1 librados: shutdown 2024-02-13T00:41:31.331 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:41:31.339 INFO:tasks.workunit.client.0.smithi162.stdout:watch_notify_pp: flushed 2024-02-13T00:41:31.339 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3022 ms) 2024-02-13T00:41:31.339 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-02-13T00:41:31.339 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: trying... 2024-02-13T00:41:31.339 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94779274301856 notify_id 528280977414 notifier_gid 25049 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: timed out 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: flushing 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: flushed 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3050 ms) 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: List watches 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify2 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94779274301856 notify_id 541165879303 notifier_gid 25049 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify2 done 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: watch_check 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: unwatch2 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: flushing 2024-02-13T00:41:31.340 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: done 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3079 ms) 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: List watches 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify2 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94779274301856 notify_id 554050781189 notifier_gid 25049 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify2 done 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: watch_check 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: unwatch2 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: flushing 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: done 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3035 ms) 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16408 ms total) 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: 2024-02-13T00:41:31.341 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-02-13T00:41:31.342 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (36645 ms total) 2024-02-13T00:41:31.342 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-02-13T00:41:31.423 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:31.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:30.239720+0000 mon.a (mon.0) 1065 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:31.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:30.249186+0000 mgr.y (mgr.14544) 164 : cluster [DBG] pgmap v172: 432 pgs: 32 creating+peering, 400 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.1 KiB/s wr, 4 op/s 2024-02-13T00:41:31.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:30.266757+0000 mon.a (mon.0) 1066 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-9", "mode": "writeback"}]': finished 2024-02-13T00:41:31.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:30.266899+0000 mon.a (mon.0) 1067 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-83985-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:31.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:30.267004+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-13T00:41:31.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:30.267160+0000 mon.a (mon.0) 1069 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-02-13T00:41:31.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:30.286602+0000 mon.b (mon.1) 136 : audit [INF] from='client.? 172.21.15.162:0/3791899583' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:31.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:30.301738+0000 mon.c (mon.2) 129 : audit [INF] from='client.? 172.21.15.162:0/4055124953' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-83977-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:31.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:30.310523+0000 mon.a (mon.0) 1070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:30.318975+0000 mon.a (mon.0) 1071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-83977-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:30.477680+0000 mon.b (mon.1) 137 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:30.480192+0000 mon.a (mon.0) 1072 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:30.719716+0000 mon.b (mon.1) 138 : audit [INF] from='client.? 172.21.15.162:0/2729799896' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:30.725212+0000 mon.a (mon.0) 1073 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:30.996749+0000 mon.b (mon.1) 139 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:31.263429+0000 mon.a (mon.0) 1074 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:31.265436+0000 mon.a (mon.0) 1075 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:31.295783+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:31.295854+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-83977-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:31.295930+0000 mon.a (mon.0) 1078 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:31.295967+0000 mon.a (mon.0) 1079 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:41:31.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:31.296011+0000 mon.a (mon.0) 1080 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-02-13T00:41:31.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:31.297811+0000 mon.b (mon.1) 140 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9"}]: dispatch 2024-02-13T00:41:31.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:31.299841+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9"}]: dispatch 2024-02-13T00:41:31.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:30.239720+0000 mon.a (mon.0) 1065 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:31.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:30.249186+0000 mgr.y (mgr.14544) 164 : cluster [DBG] pgmap v172: 432 pgs: 32 creating+peering, 400 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.1 KiB/s wr, 4 op/s 2024-02-13T00:41:31.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:30.266757+0000 mon.a (mon.0) 1066 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-9", "mode": "writeback"}]': finished 2024-02-13T00:41:31.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:30.266899+0000 mon.a (mon.0) 1067 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-83985-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:31.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:30.267004+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-13T00:41:31.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:30.267160+0000 mon.a (mon.0) 1069 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:30.286602+0000 mon.b (mon.1) 136 : audit [INF] from='client.? 172.21.15.162:0/3791899583' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:30.301738+0000 mon.c (mon.2) 129 : audit [INF] from='client.? 172.21.15.162:0/4055124953' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-83977-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:30.310523+0000 mon.a (mon.0) 1070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:30.318975+0000 mon.a (mon.0) 1071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-83977-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:30.477680+0000 mon.b (mon.1) 137 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:30.480192+0000 mon.a (mon.0) 1072 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:30.719716+0000 mon.b (mon.1) 138 : audit [INF] from='client.? 172.21.15.162:0/2729799896' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:30.725212+0000 mon.a (mon.0) 1073 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:30.996749+0000 mon.b (mon.1) 139 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:31.263429+0000 mon.a (mon.0) 1074 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:31.265436+0000 mon.a (mon.0) 1075 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:31.295783+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:31.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:31.295854+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-83977-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:31.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:31.295930+0000 mon.a (mon.0) 1078 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:41:31.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:31.295967+0000 mon.a (mon.0) 1079 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:41:31.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:31.296011+0000 mon.a (mon.0) 1080 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-02-13T00:41:31.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:31.297811+0000 mon.b (mon.1) 140 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9"}]: dispatch 2024-02-13T00:41:31.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:31.299841+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9"}]: dispatch 2024-02-13T00:41:31.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:30.239720+0000 mon.a (mon.0) 1065 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:30.249186+0000 mgr.y (mgr.14544) 164 : cluster [DBG] pgmap v172: 432 pgs: 32 creating+peering, 400 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.1 KiB/s wr, 4 op/s 2024-02-13T00:41:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:30.266757+0000 mon.a (mon.0) 1066 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-9", "mode": "writeback"}]': finished 2024-02-13T00:41:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:30.266899+0000 mon.a (mon.0) 1067 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-83985-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:30.267004+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-84113-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-13T00:41:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:30.267160+0000 mon.a (mon.0) 1069 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-02-13T00:41:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:30.286602+0000 mon.b (mon.1) 136 : audit [INF] from='client.? 172.21.15.162:0/3791899583' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:30.301738+0000 mon.c (mon.2) 129 : audit [INF] from='client.? 172.21.15.162:0/4055124953' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-83977-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:30.310523+0000 mon.a (mon.0) 1070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:30.318975+0000 mon.a (mon.0) 1071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-83977-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:31.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:30.477680+0000 mon.b (mon.1) 137 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:31.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:30.480192+0000 mon.a (mon.0) 1072 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:31.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:30.719716+0000 mon.b (mon.1) 138 : audit [INF] from='client.? 172.21.15.162:0/2729799896' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:41:31.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:30.725212+0000 mon.a (mon.0) 1073 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:41:31.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:30.996749+0000 mon.b (mon.1) 139 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:31.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:31.263429+0000 mon.a (mon.0) 1074 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:31.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:31.265436+0000 mon.a (mon.0) 1075 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:31.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:31.295783+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:31.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:31.295854+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-83977-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:31.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:31.295930+0000 mon.a (mon.0) 1078 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:41:31.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:31.295967+0000 mon.a (mon.0) 1079 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:41:31.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:31.296011+0000 mon.a (mon.0) 1080 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-02-13T00:41:31.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:31.297811+0000 mon.b (mon.1) 140 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9"}]: dispatch 2024-02-13T00:41:31.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:31.299841+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9"}]: dispatch 2024-02-13T00:41:32.889 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:32.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:33.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:33.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:31.411796+0000 osd.5 (osd.5) 67 : cluster [DBG] 4.8 scrub starts 2024-02-13T00:41:33.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:31.413572+0000 osd.5 (osd.5) 68 : cluster [DBG] 4.8 scrub ok 2024-02-13T00:41:33.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:32.004109+0000 mon.b (mon.1) 141 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:32.289433+0000 mon.a (mon.0) 1082 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:41:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:32.296920+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9"}]': finished 2024-02-13T00:41:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:32.297073+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-02-13T00:41:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:32.305283+0000 mon.a (mon.0) 1085 : audit [INF] from='client.? 172.21.15.162:0/2435237253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-83985-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:31.411796+0000 osd.5 (osd.5) 67 : cluster [DBG] 4.8 scrub starts 2024-02-13T00:41:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:31.413572+0000 osd.5 (osd.5) 68 : cluster [DBG] 4.8 scrub ok 2024-02-13T00:41:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:32.004109+0000 mon.b (mon.1) 141 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:32.289433+0000 mon.a (mon.0) 1082 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:41:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:32.296920+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9"}]': finished 2024-02-13T00:41:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:32.297073+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-02-13T00:41:33.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:32.305283+0000 mon.a (mon.0) 1085 : audit [INF] from='client.? 172.21.15.162:0/2435237253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-83985-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:33.213 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:33.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:31.411796+0000 osd.5 (osd.5) 67 : cluster [DBG] 4.8 scrub starts 2024-02-13T00:41:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:31.413572+0000 osd.5 (osd.5) 68 : cluster [DBG] 4.8 scrub ok 2024-02-13T00:41:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:32.004109+0000 mon.b (mon.1) 141 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:32.289433+0000 mon.a (mon.0) 1082 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:41:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:32.296920+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-9"}]': finished 2024-02-13T00:41:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:32.297073+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-02-13T00:41:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:32.305283+0000 mon.a (mon.0) 1085 : audit [INF] from='client.? 172.21.15.162:0/2435237253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-83985-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:33.420 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:33.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:33.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:33.657 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:33.822 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:41:33.822 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:41:33.822 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-13T00:41:33.822 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:41:34.156 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:34.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:34.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:31.642288+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:41:34.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:31.648210+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:41:34.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:32.254212+0000 mgr.y (mgr.14544) 165 : cluster [DBG] pgmap v175: 428 pgs: 64 unknown, 364 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:34.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:32.398582+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:41:34.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:32.399470+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:41:34.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:32.424049+0000 osd.5 (osd.5) 69 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:41:34.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:32.424840+0000 osd.5 (osd.5) 70 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:41:34.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:32.711604+0000 osd.3 (osd.3) 89 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:41:34.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:32.715542+0000 osd.3 (osd.3) 90 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:41:34.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:33.008460+0000 mon.b (mon.1) 142 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:34.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:33.310798+0000 mon.a (mon.0) 1086 : audit [INF] from='client.? 172.21.15.162:0/2435237253' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-83985-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:34.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:33.315582+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-02-13T00:41:34.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:33.325452+0000 mon.b (mon.1) 143 : audit [INF] from='client.? 172.21.15.162:0/3777562399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-83977-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:34.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:33.339522+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-83977-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:34.166 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:31.642288+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:41:34.166 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:31.648210+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:41:34.166 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:32.254212+0000 mgr.y (mgr.14544) 165 : cluster [DBG] pgmap v175: 428 pgs: 64 unknown, 364 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:34.166 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:32.398582+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:41:34.166 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:32.399470+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:41:34.166 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:32.424049+0000 osd.5 (osd.5) 69 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:41:34.166 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:32.424840+0000 osd.5 (osd.5) 70 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:41:34.166 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:32.711604+0000 osd.3 (osd.3) 89 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:41:34.166 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:32.715542+0000 osd.3 (osd.3) 90 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:41:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:33.008460+0000 mon.b (mon.1) 142 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:33.310798+0000 mon.a (mon.0) 1086 : audit [INF] from='client.? 172.21.15.162:0/2435237253' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-83985-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:33.315582+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-02-13T00:41:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:33.325452+0000 mon.b (mon.1) 143 : audit [INF] from='client.? 172.21.15.162:0/3777562399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-83977-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:33.339522+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-83977-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:34.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:34.286 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:31.642288+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:41:34.286 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:31.648210+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:41:34.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:32.254212+0000 mgr.y (mgr.14544) 165 : cluster [DBG] pgmap v175: 428 pgs: 64 unknown, 364 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:34.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:32.398582+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:41:34.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:32.399470+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:41:34.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:32.424049+0000 osd.5 (osd.5) 69 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:41:34.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:32.424840+0000 osd.5 (osd.5) 70 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:41:34.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:32.711604+0000 osd.3 (osd.3) 89 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:41:34.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:32.715542+0000 osd.3 (osd.3) 90 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:41:34.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:33.008460+0000 mon.b (mon.1) 142 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:34.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:33.310798+0000 mon.a (mon.0) 1086 : audit [INF] from='client.? 172.21.15.162:0/2435237253' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-83985-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:34.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:33.315582+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-02-13T00:41:34.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:33.325452+0000 mon.b (mon.1) 143 : audit [INF] from='client.? 172.21.15.162:0/3777562399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-83977-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:34.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:33.339522+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-83977-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:34.288 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:34.309 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:41:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:34.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:41:34.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.356+0000 7f9c3eccc700 1 -- 172.21.15.162:0/732513728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c38104210 msgr2=0x7f9c381045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:34.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.356+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/732513728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c38104210 0x7f9c381045f0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7f9c2c008da0 tx=0x7f9c2c00bec0).stop 2024-02-13T00:41:34.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.367+0000 7f9c3eccc700 1 -- 172.21.15.162:0/732513728 shutdown_connections 2024-02-13T00:41:34.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.367+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/732513728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c38069dd0 0x7f9c3806a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:34.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.367+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/732513728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c38104210 0x7f9c381045f0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:34.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.367+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/732513728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c38104b70 0x7f9c38069740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:34.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.367+0000 7f9c3eccc700 1 -- 172.21.15.162:0/732513728 >> 172.21.15.162:0/732513728 conn(0x7f9c38063df0 msgr2=0x7f9c380676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:34.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.370+0000 7f9c3eccc700 1 -- 172.21.15.162:0/732513728 shutdown_connections 2024-02-13T00:41:34.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.370+0000 7f9c3eccc700 1 -- 172.21.15.162:0/732513728 wait complete. 2024-02-13T00:41:34.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.371+0000 7f9c3eccc700 1 librados: starting msgr at 2024-02-13T00:41:34.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.371+0000 7f9c3eccc700 1 librados: starting objecter 2024-02-13T00:41:34.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.371+0000 7f9c3eccc700 1 Processor -- start 2024-02-13T00:41:34.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.371+0000 7f9c3eccc700 1 -- start start 2024-02-13T00:41:34.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.371+0000 7f9c3eccc700 1 librados: setting wanted keys 2024-02-13T00:41:34.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.371+0000 7f9c3eccc700 1 librados: calling monclient init 2024-02-13T00:41:34.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.371+0000 7f9c3eccc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c38069dd0 0x7f9c380734e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:34.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.371+0000 7f9c3eccc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c38104210 0x7f9c380739f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:34.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.371+0000 7f9c3eccc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c38104b70 0x7f9c3812b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:34.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.371+0000 7f9c3eccc700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3812bbd0 con 0x7f9c38069dd0 2024-02-13T00:41:34.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.371+0000 7f9c3eccc700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3812bd40 con 0x7f9c38104210 2024-02-13T00:41:34.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.372+0000 7f9c3eccc700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3812bf00 con 0x7f9c38104b70 2024-02-13T00:41:34.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.372+0000 7f9c3d269700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c38104b70 0x7f9c3812b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:34.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.372+0000 7f9c3d269700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c38104b70 0x7f9c3812b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:50958/0 (socket says 172.21.15.162:50958) 2024-02-13T00:41:34.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.372+0000 7f9c3d269700 1 -- 172.21.15.162:0/313675181 learned_addr learned my addr 172.21.15.162:0/313675181 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:34.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.372+0000 7f9c3ca68700 1 --2- 172.21.15.162:0/313675181 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c38069dd0 0x7f9c380734e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:34.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.373+0000 7f9c3d269700 1 -- 172.21.15.162:0/313675181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c38104210 msgr2=0x7f9c380739f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-13T00:41:34.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.373+0000 7f9c3d269700 1 --2- 172.21.15.162:0/313675181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c38104210 0x7f9c380739f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:34.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.373+0000 7f9c3d269700 1 -- 172.21.15.162:0/313675181 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c38069dd0 msgr2=0x7f9c380734e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:34.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.373+0000 7f9c3d269700 1 --2- 172.21.15.162:0/313675181 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c38069dd0 0x7f9c380734e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:34.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.373+0000 7f9c3d269700 1 -- 172.21.15.162:0/313675181 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c005040 con 0x7f9c38104b70 2024-02-13T00:41:34.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.373+0000 7f9c3d269700 1 --2- 172.21.15.162:0/313675181 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c38104b70 0x7f9c3812b6e0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f9c3000d860 tx=0x7f9c30004a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:34.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.385+0000 7f9c35ffb700 10 client.?.objecter ms_handle_connect 0x7f9c38104b70 2024-02-13T00:41:34.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.385+0000 7f9c35ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:34.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.385+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3000b5f0 con 0x7f9c38104b70 2024-02-13T00:41:34.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.385+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c3000bc20 con 0x7f9c38104b70 2024-02-13T00:41:34.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.385+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30012cf0 con 0x7f9c38104b70 2024-02-13T00:41:34.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.385+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c3812c1e0 con 0x7f9c38104b70 2024-02-13T00:41:34.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.386+0000 7f9c3eccc700 10 client.32525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:34.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.386+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3812c680 con 0x7f9c38104b70 2024-02-13T00:41:34.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.386+0000 7f9c3eccc700 1 librados: init done 2024-02-13T00:41:34.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.388+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9c1c004f30 con 0x7f9c38104b70 2024-02-13T00:41:34.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:34.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.429+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f9c30012460 con 0x7f9c38104b70 2024-02-13T00:41:34.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.430+0000 7f9c35ffb700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9c28062d20 0x7f9c280651e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:34.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.430+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(139..139 src has 52..139) v4 ==== 10472+0+0 (secure 0 0 0) 0x7f9c3007dfe0 con 0x7f9c38104b70 2024-02-13T00:41:34.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.430+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 osd_map(139..139 src has 52..139) v4 2024-02-13T00:41:34.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.430+0000 7f9c35ffb700 3 client.32525.objecter handle_osd_map got epochs [139,139] > 0 2024-02-13T00:41:34.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.430+0000 7f9c35ffb700 3 client.32525.objecter handle_osd_map decoding full epoch 139 2024-02-13T00:41:34.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.430+0000 7f9c35ffb700 20 client.32525.objecter dump_active .. 0 homeless 2024-02-13T00:41:34.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.430+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v139) v1 ==== 54+0+16 (secure 0 0 0) 0x7f9c3007e440 con 0x7f9c38104b70 2024-02-13T00:41:34.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.431+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:34.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.431+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 1 2024-02-13T00:41:34.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.431+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9c1c004560 0x7f9c1c004980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:34.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.431+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c0051d0 osd=0 3 2024-02-13T00:41:34.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.431+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c0051d0 osd=0 3 2024-02-13T00:41:34.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.431+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c0051d0 osd=0 3 2024-02-13T00:41:34.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.431+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 1 2024-02-13T00:41:34.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.431+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c0051d0 osd=0 3 2024-02-13T00:41:34.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.431+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 0 1 2024-02-13T00:41:34.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.431+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c0051d0 osd=0 4 2024-02-13T00:41:34.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.431+0000 7f9c3eccc700 10 client.32525.objecter _send_command 1 2024-02-13T00:41:34.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.431+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1c008960 con 0x7f9c1c004560 2024-02-13T00:41:34.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.436+0000 7f9c3ca68700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9c28062d20 0x7f9c280651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:34.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.454+0000 7f9c3ca68700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9c28062d20 0x7f9c280651e0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7f9c2c00b040 tx=0x7f9c2c00d040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:34.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.455+0000 7f9c35ffb700 10 client.32525.objecter ms_handle_connect 0x7f9c28062d20 2024-02-13T00:41:34.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.485+0000 7f9c37fff700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9c1c004560 0x7f9c1c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:34.525 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:34.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.534+0000 7f9c37fff700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9c1c004560 0x7f9c1c004980 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:34.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.534+0000 7f9c35ffb700 10 client.32525.objecter ms_handle_connect 0x7f9c1c004560 2024-02-13T00:41:34.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.566+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9c1c008960 con 0x7f9c1c004560 2024-02-13T00:41:34.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.566+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:41:34.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.566+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 1 = system:0 2024-02-13T00:41:34.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.566+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c0051d0 osd=0 4 2024-02-13T00:41:34.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.566+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 0 1 2024-02-13T00:41:34.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.612+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:34.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.612+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 2 2024-02-13T00:41:34.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.612+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c0051d0 osd=0 3 2024-02-13T00:41:34.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.612+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c0051d0 osd=0 3 2024-02-13T00:41:34.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.612+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c0051d0 osd=0 3 2024-02-13T00:41:34.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.612+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 2 2024-02-13T00:41:34.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.612+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c0051d0 osd=0 3 2024-02-13T00:41:34.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.612+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 0 2 2024-02-13T00:41:34.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.612+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c0051d0 osd=0 4 2024-02-13T00:41:34.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.612+0000 7f9c3eccc700 10 client.32525.objecter _send_command 2 2024-02-13T00:41:34.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.612+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9c1c003fd0 con 0x7f9c1c004560 2024-02-13T00:41:34.624 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:41:34.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.618+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9c1c003fd0 con 0x7f9c1c004560 2024-02-13T00:41:34.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.618+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:34.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.618+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:34.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.618+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c0051d0 osd=0 4 2024-02-13T00:41:34.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.618+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 0 2 2024-02-13T00:41:34.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.621+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:34.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.621+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 3 2024-02-13T00:41:34.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.621+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6805/150599780 conn(0x7f9c1c0084c0 0x7f9c1c002780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:34.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.621+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c004380 osd=1 3 2024-02-13T00:41:34.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.621+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c004380 osd=1 3 2024-02-13T00:41:34.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.621+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c004380 osd=1 3 2024-02-13T00:41:34.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.621+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 3 2024-02-13T00:41:34.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.621+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c004380 osd=1 3 2024-02-13T00:41:34.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.621+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 1 3 2024-02-13T00:41:34.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.621+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c004380 osd=1 4 2024-02-13T00:41:34.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.621+0000 7f9c3eccc700 10 client.32525.objecter _send_command 3 2024-02-13T00:41:34.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.621+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1c002e40 con 0x7f9c1c0084c0 2024-02-13T00:41:34.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.628+0000 7f9c3ca68700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6805/150599780 conn(0x7f9c1c0084c0 0x7f9c1c002780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:34.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.636+0000 7f9c3ca68700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6805/150599780 conn(0x7f9c1c0084c0 0x7f9c1c002780 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:34.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.636+0000 7f9c35ffb700 10 client.32525.objecter ms_handle_connect 0x7f9c1c0084c0 2024-02-13T00:41:34.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.639+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9c1c002e40 con 0x7f9c1c0084c0 2024-02-13T00:41:34.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.639+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:41:34.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.639+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 3 = system:0 2024-02-13T00:41:34.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.639+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c004380 osd=1 4 2024-02-13T00:41:34.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.639+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 1 3 2024-02-13T00:41:34.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.668+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:34.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.668+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 4 2024-02-13T00:41:34.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.668+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c004380 osd=1 3 2024-02-13T00:41:34.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.668+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c004380 osd=1 3 2024-02-13T00:41:34.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.668+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c004380 osd=1 3 2024-02-13T00:41:34.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.668+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 4 2024-02-13T00:41:34.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.668+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c004380 osd=1 3 2024-02-13T00:41:34.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.668+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 1 4 2024-02-13T00:41:34.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.668+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c004380 osd=1 4 2024-02-13T00:41:34.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.668+0000 7f9c3eccc700 10 client.32525.objecter _send_command 4 2024-02-13T00:41:34.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.668+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9c1c007ae0 con 0x7f9c1c0084c0 2024-02-13T00:41:34.696 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:41:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.692+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9c1c007ae0 con 0x7f9c1c0084c0 2024-02-13T00:41:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.692+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.692+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.692+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c004380 osd=1 4 2024-02-13T00:41:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.692+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 1 4 2024-02-13T00:41:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 5 2024-02-13T00:41:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6809/3689719376 conn(0x7f9c1c003950 0x7f9c1c00d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c0037e0 osd=2 3 2024-02-13T00:41:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c0037e0 osd=2 3 2024-02-13T00:41:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c0037e0 osd=2 3 2024-02-13T00:41:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 5 2024-02-13T00:41:34.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c0037e0 osd=2 3 2024-02-13T00:41:34.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 2 5 2024-02-13T00:41:34.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c0037e0 osd=2 4 2024-02-13T00:41:34.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c3eccc700 10 client.32525.objecter _send_command 5 2024-02-13T00:41:34.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1c00fe20 con 0x7f9c1c003950 2024-02-13T00:41:34.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.693+0000 7f9c37fff700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6809/3689719376 conn(0x7f9c1c003950 0x7f9c1c00d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:34.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.694+0000 7f9c37fff700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6809/3689719376 conn(0x7f9c1c003950 0x7f9c1c00d000 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:34.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.694+0000 7f9c35ffb700 10 client.32525.objecter ms_handle_connect 0x7f9c1c003950 2024-02-13T00:41:34.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.695+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9c1c00fe20 con 0x7f9c1c003950 2024-02-13T00:41:34.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.695+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:41:34.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.695+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 5 = system:0 2024-02-13T00:41:34.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.695+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c0037e0 osd=2 4 2024-02-13T00:41:34.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.695+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 2 5 2024-02-13T00:41:34.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.727+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:34.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.727+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 6 2024-02-13T00:41:34.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.727+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c0037e0 osd=2 3 2024-02-13T00:41:34.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.727+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c0037e0 osd=2 3 2024-02-13T00:41:34.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.727+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c0037e0 osd=2 3 2024-02-13T00:41:34.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.727+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 6 2024-02-13T00:41:34.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.727+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c0037e0 osd=2 3 2024-02-13T00:41:34.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.727+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 2 6 2024-02-13T00:41:34.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.727+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c0037e0 osd=2 4 2024-02-13T00:41:34.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.727+0000 7f9c3eccc700 10 client.32525.objecter _send_command 6 2024-02-13T00:41:34.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.727+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9c1c0032b0 con 0x7f9c1c003950 2024-02-13T00:41:34.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.735+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9c1c0032b0 con 0x7f9c1c003950 2024-02-13T00:41:34.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.735+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:34.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.735+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:34.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.735+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c0037e0 osd=2 4 2024-02-13T00:41:34.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.735+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 2 6 2024-02-13T00:41:34.743 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:41:34.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.737+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:34.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.737+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 7 2024-02-13T00:41:34.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.737+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6813/604971011 conn(0x7f9c1c00e6f0 0x7f9c1c00eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:34.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.737+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c001b90 osd=3 3 2024-02-13T00:41:34.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.737+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c001b90 osd=3 3 2024-02-13T00:41:34.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.737+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c001b90 osd=3 3 2024-02-13T00:41:34.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.737+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 7 2024-02-13T00:41:34.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.737+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c001b90 osd=3 3 2024-02-13T00:41:34.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.737+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 3 7 2024-02-13T00:41:34.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.737+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c001b90 osd=3 4 2024-02-13T00:41:34.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.737+0000 7f9c3eccc700 10 client.32525.objecter _send_command 7 2024-02-13T00:41:34.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.737+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1c002300 con 0x7f9c1c00e6f0 2024-02-13T00:41:34.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.743+0000 7f9c3d269700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6813/604971011 conn(0x7f9c1c00e6f0 0x7f9c1c00eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:34.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.743+0000 7f9c3d269700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6813/604971011 conn(0x7f9c1c00e6f0 0x7f9c1c00eba0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:34.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.743+0000 7f9c35ffb700 10 client.32525.objecter ms_handle_connect 0x7f9c1c00e6f0 2024-02-13T00:41:34.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.744+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9c1c002300 con 0x7f9c1c00e6f0 2024-02-13T00:41:34.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.744+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:41:34.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.744+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 7 = system:0 2024-02-13T00:41:34.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.744+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c001b90 osd=3 4 2024-02-13T00:41:34.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.744+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 3 7 2024-02-13T00:41:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.770+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.770+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 8 2024-02-13T00:41:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.770+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c001b90 osd=3 3 2024-02-13T00:41:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.770+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c001b90 osd=3 3 2024-02-13T00:41:34.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.770+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c001b90 osd=3 3 2024-02-13T00:41:34.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.770+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 8 2024-02-13T00:41:34.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.770+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c001b90 osd=3 3 2024-02-13T00:41:34.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.770+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 3 8 2024-02-13T00:41:34.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.770+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c001b90 osd=3 4 2024-02-13T00:41:34.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.770+0000 7f9c3eccc700 10 client.32525.objecter _send_command 8 2024-02-13T00:41:34.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.770+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9c1c00f860 con 0x7f9c1c00e6f0 2024-02-13T00:41:34.805 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:41:34.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.802+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9c1c00f860 con 0x7f9c1c00e6f0 2024-02-13T00:41:34.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.802+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:34.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.802+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:34.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.802+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c001b90 osd=3 4 2024-02-13T00:41:34.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.802+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 3 8 2024-02-13T00:41:34.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 9 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9c1c012050 0x7f9c1c014440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c00f090 osd=4 3 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c00f090 osd=4 3 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c00f090 osd=4 3 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 9 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c00f090 osd=4 3 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 4 9 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c00f090 osd=4 4 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3eccc700 10 client.32525.objecter _send_command 9 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1c017090 con 0x7f9c1c012050 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.803+0000 7f9c3ca68700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9c1c012050 0x7f9c1c014440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.818+0000 7f9c3ca68700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9c1c012050 0x7f9c1c014440 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:34.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.818+0000 7f9c35ffb700 10 client.32525.objecter ms_handle_connect 0x7f9c1c012050 2024-02-13T00:41:34.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.819+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9c1c017090 con 0x7f9c1c012050 2024-02-13T00:41:34.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.819+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:41:34.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.819+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 9 = system:0 2024-02-13T00:41:34.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.819+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c00f090 osd=4 4 2024-02-13T00:41:34.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.819+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 4 9 2024-02-13T00:41:34.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.850+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:34.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.850+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 10 2024-02-13T00:41:34.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.850+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c00f090 osd=4 3 2024-02-13T00:41:34.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.850+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c00f090 osd=4 3 2024-02-13T00:41:34.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.850+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c00f090 osd=4 3 2024-02-13T00:41:34.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.850+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 10 2024-02-13T00:41:34.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.850+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c00f090 osd=4 3 2024-02-13T00:41:34.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.850+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 4 10 2024-02-13T00:41:34.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.850+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c00f090 osd=4 4 2024-02-13T00:41:34.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.850+0000 7f9c3eccc700 10 client.32525.objecter _send_command 10 2024-02-13T00:41:34.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.850+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9c1c014930 con 0x7f9c1c012050 2024-02-13T00:41:34.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.868+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9c1c014930 con 0x7f9c1c012050 2024-02-13T00:41:34.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.868+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:34.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.868+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:34.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.868+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c00f090 osd=4 4 2024-02-13T00:41:34.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.868+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 4 10 2024-02-13T00:41:34.880 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:41:34.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.877+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:34.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.877+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 11 2024-02-13T00:41:34.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.877+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6804/2212296748 conn(0x7f9c1c015010 0x7f9c1c017400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:34.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.877+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c014930 osd=5 3 2024-02-13T00:41:34.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.877+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c014930 osd=5 3 2024-02-13T00:41:34.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.877+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c014930 osd=5 3 2024-02-13T00:41:34.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.877+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 11 2024-02-13T00:41:34.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.877+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c014930 osd=5 3 2024-02-13T00:41:34.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.877+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 5 11 2024-02-13T00:41:34.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.877+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c014930 osd=5 4 2024-02-13T00:41:34.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.877+0000 7f9c3eccc700 10 client.32525.objecter _send_command 11 2024-02-13T00:41:34.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.877+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1c01a1b0 con 0x7f9c1c015010 2024-02-13T00:41:34.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.884+0000 7f9c37fff700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6804/2212296748 conn(0x7f9c1c015010 0x7f9c1c017400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.884+0000 7f9c37fff700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6804/2212296748 conn(0x7f9c1c015010 0x7f9c1c017400 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.884+0000 7f9c35ffb700 10 client.32525.objecter ms_handle_connect 0x7f9c1c015010 2024-02-13T00:41:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.885+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9c1c01a1b0 con 0x7f9c1c015010 2024-02-13T00:41:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.885+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:41:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.885+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 11 = system:0 2024-02-13T00:41:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.885+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c014930 osd=5 4 2024-02-13T00:41:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.885+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 5 11 2024-02-13T00:41:34.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-13T00:41:34.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.908+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:34.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.908+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 12 2024-02-13T00:41:34.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.908+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c014930 osd=5 3 2024-02-13T00:41:34.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.908+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c014930 osd=5 3 2024-02-13T00:41:34.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.908+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c014930 osd=5 3 2024-02-13T00:41:34.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.908+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 12 2024-02-13T00:41:34.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.908+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c014930 osd=5 3 2024-02-13T00:41:34.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.908+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 5 12 2024-02-13T00:41:34.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.908+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c014930 osd=5 4 2024-02-13T00:41:34.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.908+0000 7f9c3eccc700 10 client.32525.objecter _send_command 12 2024-02-13T00:41:34.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.908+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9c1c01a1b0 con 0x7f9c1c015010 2024-02-13T00:41:34.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.910+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9c1c01a1b0 con 0x7f9c1c015010 2024-02-13T00:41:34.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.910+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:34.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.910+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:34.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.910+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c014930 osd=5 4 2024-02-13T00:41:34.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.910+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 5 12 2024-02-13T00:41:34.917 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:41:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 13 2024-02-13T00:41:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9c1c0194b0 0x7f9c1c019930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c01a1b0 osd=6 3 2024-02-13T00:41:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c01a1b0 osd=6 3 2024-02-13T00:41:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c01a1b0 osd=6 3 2024-02-13T00:41:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 13 2024-02-13T00:41:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c01a1b0 osd=6 3 2024-02-13T00:41:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 6 13 2024-02-13T00:41:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c01a1b0 osd=6 4 2024-02-13T00:41:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3eccc700 10 client.32525.objecter _send_command 13 2024-02-13T00:41:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1c019fa0 con 0x7f9c1c0194b0 2024-02-13T00:41:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.914+0000 7f9c3d269700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9c1c0194b0 0x7f9c1c019930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.915+0000 7f9c3d269700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9c1c0194b0 0x7f9c1c019930 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.915+0000 7f9c35ffb700 10 client.32525.objecter ms_handle_connect 0x7f9c1c0194b0 2024-02-13T00:41:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.915+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9c1c019fa0 con 0x7f9c1c0194b0 2024-02-13T00:41:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.915+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:41:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.915+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 13 = system:0 2024-02-13T00:41:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.915+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c01a1b0 osd=6 4 2024-02-13T00:41:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.915+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 6 13 2024-02-13T00:41:34.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.941+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:34.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.941+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 14 2024-02-13T00:41:34.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.941+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c01a1b0 osd=6 3 2024-02-13T00:41:34.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.941+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c01a1b0 osd=6 3 2024-02-13T00:41:34.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.941+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c01a1b0 osd=6 3 2024-02-13T00:41:34.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.941+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 14 2024-02-13T00:41:34.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.941+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c01a1b0 osd=6 3 2024-02-13T00:41:34.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.941+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 6 14 2024-02-13T00:41:34.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.941+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c01a1b0 osd=6 4 2024-02-13T00:41:34.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.941+0000 7f9c3eccc700 10 client.32525.objecter _send_command 14 2024-02-13T00:41:34.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.941+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9c1c019fa0 con 0x7f9c1c0194b0 2024-02-13T00:41:34.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9c1c019fa0 con 0x7f9c1c0194b0 2024-02-13T00:41:34.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:34.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:34.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c01a1b0 osd=6 4 2024-02-13T00:41:34.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 6 14 2024-02-13T00:41:34.947 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:41:34.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:34.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 15 2024-02-13T00:41:34.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6812/1964192772 conn(0x7f9c1c01c920 0x7f9c1c01cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:34.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c019fa0 osd=7 3 2024-02-13T00:41:34.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c019fa0 osd=7 3 2024-02-13T00:41:34.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c019fa0 osd=7 3 2024-02-13T00:41:34.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 15 2024-02-13T00:41:34.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c019fa0 osd=7 3 2024-02-13T00:41:34.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 7 15 2024-02-13T00:41:34.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c019fa0 osd=7 4 2024-02-13T00:41:34.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c3eccc700 10 client.32525.objecter _send_command 15 2024-02-13T00:41:34.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.943+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c1c01d3a0 con 0x7f9c1c01c920 2024-02-13T00:41:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.944+0000 7f9c3ca68700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6812/1964192772 conn(0x7f9c1c01c920 0x7f9c1c01cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.944+0000 7f9c3ca68700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6812/1964192772 conn(0x7f9c1c01c920 0x7f9c1c01cd80 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.944+0000 7f9c35ffb700 10 client.32525.objecter ms_handle_connect 0x7f9c1c01c920 2024-02-13T00:41:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.945+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9c1c01d3a0 con 0x7f9c1c01c920 2024-02-13T00:41:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.945+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:41:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.945+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 15 = system:0 2024-02-13T00:41:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.945+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c019fa0 osd=7 4 2024-02-13T00:41:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.945+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 7 15 2024-02-13T00:41:34.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.971+0000 7f9c3eccc700 10 client.32525.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:34.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.971+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign -1 16 2024-02-13T00:41:34.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.971+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c019fa0 osd=7 3 2024-02-13T00:41:34.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.971+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c019fa0 osd=7 3 2024-02-13T00:41:34.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.971+0000 7f9c3eccc700 20 client.32525.objecter _get_session s=0x7f9c1c019fa0 osd=7 3 2024-02-13T00:41:34.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.971+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_remove -1 16 2024-02-13T00:41:34.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.971+0000 7f9c3eccc700 20 client.32525.objecter get_session s=0x7f9c1c019fa0 osd=7 3 2024-02-13T00:41:34.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.971+0000 7f9c3eccc700 15 client.32525.objecter _session_command_op_assign 7 16 2024-02-13T00:41:34.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.971+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c019fa0 osd=7 4 2024-02-13T00:41:34.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.971+0000 7f9c3eccc700 10 client.32525.objecter _send_command 16 2024-02-13T00:41:34.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.971+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9c1c01d3a0 con 0x7f9c1c01c920 2024-02-13T00:41:34.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.988+0000 7f9c35ffb700 1 -- 172.21.15.162:0/313675181 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9c1c01d3a0 con 0x7f9c1c01c920 2024-02-13T00:41:34.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.988+0000 7f9c35ffb700 10 client.32525.objecter ms_dispatch 0x7f9c38001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:34.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.988+0000 7f9c35ffb700 10 client.32525.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:34.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.988+0000 7f9c35ffb700 20 client.32525.objecter put_session s=0x7f9c1c019fa0 osd=7 4 2024-02-13T00:41:34.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.988+0000 7f9c35ffb700 15 client.32525.objecter _session_command_op_remove 7 16 2024-02-13T00:41:34.995 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:41:34.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.991+0000 7f9c3eccc700 10 librados: watch_flush enter 2024-02-13T00:41:35.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.993+0000 7f9c3eccc700 10 librados: watch_flush exit 2024-02-13T00:41:35.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.993+0000 7f9c3eccc700 10 client.32525.objecter close_session for osd.0 2024-02-13T00:41:35.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.993+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9c1c004560 msgr2=0x7f9c1c004980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:35.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.993+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9c1c004560 0x7f9c1c004980 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:35.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.993+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c0051d0 osd=0 1 2024-02-13T00:41:35.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 10 client.32525.objecter close_session for osd.1 2024-02-13T00:41:35.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6805/150599780 conn(0x7f9c1c0084c0 msgr2=0x7f9c1c002780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:35.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6805/150599780 conn(0x7f9c1c0084c0 0x7f9c1c002780 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:35.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c004380 osd=1 1 2024-02-13T00:41:35.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 10 client.32525.objecter close_session for osd.2 2024-02-13T00:41:35.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6809/3689719376 conn(0x7f9c1c003950 msgr2=0x7f9c1c00d000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:35.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6809/3689719376 conn(0x7f9c1c003950 0x7f9c1c00d000 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:35.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c0037e0 osd=2 1 2024-02-13T00:41:35.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 10 client.32525.objecter close_session for osd.3 2024-02-13T00:41:35.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6813/604971011 conn(0x7f9c1c00e6f0 msgr2=0x7f9c1c00eba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:35.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6813/604971011 conn(0x7f9c1c00e6f0 0x7f9c1c00eba0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:35.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c001b90 osd=3 1 2024-02-13T00:41:35.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 10 client.32525.objecter close_session for osd.4 2024-02-13T00:41:35.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9c1c012050 msgr2=0x7f9c1c014440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:35.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9c1c012050 0x7f9c1c014440 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:35.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c00f090 osd=4 1 2024-02-13T00:41:35.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 10 client.32525.objecter close_session for osd.5 2024-02-13T00:41:35.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6804/2212296748 conn(0x7f9c1c015010 msgr2=0x7f9c1c017400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:35.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6804/2212296748 conn(0x7f9c1c015010 0x7f9c1c017400 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:35.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c014930 osd=5 1 2024-02-13T00:41:35.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 10 client.32525.objecter close_session for osd.6 2024-02-13T00:41:35.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9c1c0194b0 msgr2=0x7f9c1c019930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:35.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9c1c0194b0 0x7f9c1c019930 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:35.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c01a1b0 osd=6 1 2024-02-13T00:41:35.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 10 client.32525.objecter close_session for osd.7 2024-02-13T00:41:35.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6812/1964192772 conn(0x7f9c1c01c920 msgr2=0x7f9c1c01cd80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:35.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.181:6812/1964192772 conn(0x7f9c1c01c920 0x7f9c1c01cd80 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:35.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 20 client.32525.objecter put_session s=0x7f9c1c019fa0 osd=7 1 2024-02-13T00:41:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 20 client.32525.objecter shutdown clearing up homeless session... 2024-02-13T00:41:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 10 client.32525.objecter successfully canceled tick 2024-02-13T00:41:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3d269700 1 -- 172.21.15.162:0/313675181 reap_dead start 2024-02-13T00:41:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3d269700 1 -- 172.21.15.162:0/313675181 reap_dead start 2024-02-13T00:41:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3d269700 1 -- 172.21.15.162:0/313675181 reap_dead start 2024-02-13T00:41:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3d269700 1 -- 172.21.15.162:0/313675181 reap_dead start 2024-02-13T00:41:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3d269700 1 -- 172.21.15.162:0/313675181 reap_dead start 2024-02-13T00:41:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9c28062d20 msgr2=0x7f9c280651e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9c28062d20 0x7f9c280651e0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7f9c2c00b040 tx=0x7f9c2c00d040).stop 2024-02-13T00:41:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c38104b70 msgr2=0x7f9c3812b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c38104b70 0x7f9c3812b6e0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f9c3000d860 tx=0x7f9c30004a00).stop 2024-02-13T00:41:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 shutdown_connections 2024-02-13T00:41:35.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c38104b70 0x7f9c3812b6e0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:35.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 --2- 172.21.15.162:0/313675181 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9c28062d20 0x7f9c280651e0 secure :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 rx=0x7f9c2c00b040 tx=0x7f9c2c00d040).stop 2024-02-13T00:41:35.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:34.994+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 >> 172.21.15.162:0/313675181 conn(0x7f9c38063df0 msgr2=0x7f9c38064e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:35.038 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:35.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:35.012+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 shutdown_connections 2024-02-13T00:41:35.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:35.012+0000 7f9c3eccc700 1 -- 172.21.15.162:0/313675181 wait complete. 2024-02-13T00:41:35.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:35.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:35.042+0000 7f9c3eccc700 1 librados: shutdown 2024-02-13T00:41:35.046 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:41:35.047 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:41:35.047 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:41:35.047 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:41:35.047 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:41:35.047 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:41:35.047 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:41:35.047 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:41:35.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:35.168 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:35.189 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:35.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:35.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:35.421 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:35.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:41:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:41:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:33.001661+0000 osd.2 (osd.2) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:33.001719+0000 osd.2 (osd.2) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:34.015966+0000 mon.b (mon.1) 144 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:34.326738+0000 mon.a (mon.0) 1089 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-83977-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:34.326972+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-02-13T00:41:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:34.329032+0000 mon.b (mon.1) 145 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:34.333698+0000 mon.a (mon.0) 1091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:34.413937+0000 mon.c (mon.2) 130 : audit [DBG] from='client.? 172.21.15.162:0/313675181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:41:35.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:34.417938+0000 mon.a (mon.0) 1092 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:41:35.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:33.001661+0000 osd.2 (osd.2) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:35.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:33.001719+0000 osd.2 (osd.2) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:35.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:34.015966+0000 mon.b (mon.1) 144 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:35.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:34.326738+0000 mon.a (mon.0) 1089 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-83977-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:35.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:34.326972+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-02-13T00:41:35.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:34.329032+0000 mon.b (mon.1) 145 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:35.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:34.333698+0000 mon.a (mon.0) 1091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:35.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:34.413937+0000 mon.c (mon.2) 130 : audit [DBG] from='client.? 172.21.15.162:0/313675181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:41:35.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:34.417938+0000 mon.a (mon.0) 1092 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:41:35.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:33.001661+0000 osd.2 (osd.2) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:41:35.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:33.001719+0000 osd.2 (osd.2) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:41:35.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:34.015966+0000 mon.b (mon.1) 144 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:35.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:34.326738+0000 mon.a (mon.0) 1089 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-83977-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:35.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:34.326972+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-02-13T00:41:35.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:34.329032+0000 mon.b (mon.1) 145 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:35.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:34.333698+0000 mon.a (mon.0) 1091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:35.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:34.413937+0000 mon.c (mon.2) 130 : audit [DBG] from='client.? 172.21.15.162:0/313675181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:41:35.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:34.417938+0000 mon.a (mon.0) 1092 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:41:36.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.111+0000 7f74bae90700 1 -- 172.21.15.162:0/3637868269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b4069ab0 msgr2=0x7f74b4069f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:36.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.111+0000 7f74bae90700 1 --2- 172.21.15.162:0/3637868269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b4069ab0 0x7f74b4069f10 secure :-1 s=READY pgs=3477 cs=0 l=1 rev1=1 rx=0x7f74ac009c10 tx=0x7f74ac004d10).stop 2024-02-13T00:41:36.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.112+0000 7f74bae90700 1 -- 172.21.15.162:0/3637868269 shutdown_connections 2024-02-13T00:41:36.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.112+0000 7f74bae90700 1 --2- 172.21.15.162:0/3637868269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b4069ab0 0x7f74b4069f10 unknown :-1 s=CLOSED pgs=3477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:36.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.112+0000 7f74bae90700 1 --2- 172.21.15.162:0/3637868269 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f74b4104010 0x7f74b41043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:36.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.112+0000 7f74bae90700 1 --2- 172.21.15.162:0/3637868269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f74b41048e0 0x7f74b40695c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:36.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.112+0000 7f74bae90700 1 -- 172.21.15.162:0/3637868269 >> 172.21.15.162:0/3637868269 conn(0x7f74b4063df0 msgr2=0x7f74b40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:36.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.112+0000 7f74bae90700 1 -- 172.21.15.162:0/3637868269 shutdown_connections 2024-02-13T00:41:36.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.112+0000 7f74bae90700 1 -- 172.21.15.162:0/3637868269 wait complete. 2024-02-13T00:41:36.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.113+0000 7f74bae90700 1 librados: starting msgr at 2024-02-13T00:41:36.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.113+0000 7f74bae90700 1 librados: starting objecter 2024-02-13T00:41:36.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.113+0000 7f74bae90700 1 Processor -- start 2024-02-13T00:41:36.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.113+0000 7f74bae90700 1 -- start start 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.113+0000 7f74bae90700 1 librados: setting wanted keys 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.113+0000 7f74bae90700 1 librados: calling monclient init 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.113+0000 7f74bae90700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f74b4069ab0 0x7f74b411ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.113+0000 7f74bae90700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f74b4104010 0x7f74b411d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.113+0000 7f74bae90700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b41048e0 0x7f74b4116fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.113+0000 7f74bae90700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b4117490 con 0x7f74b4104010 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.113+0000 7f74bae90700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b4117600 con 0x7f74b41048e0 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.113+0000 7f74bae90700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b41177c0 con 0x7f74b4069ab0 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.114+0000 7f74b942d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b41048e0 0x7f74b4116fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.114+0000 7f74b942d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b41048e0 0x7f74b4116fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:35904/0 (socket says 172.21.15.162:35904) 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.114+0000 7f74b942d700 1 -- 172.21.15.162:0/2650030991 learned_addr learned my addr 172.21.15.162:0/2650030991 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.114+0000 7f74b942d700 1 -- 172.21.15.162:0/2650030991 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f74b4069ab0 msgr2=0x7f74b411ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.114+0000 7f74b942d700 1 --2- 172.21.15.162:0/2650030991 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f74b4069ab0 0x7f74b411ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:36.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.114+0000 7f74b942d700 1 -- 172.21.15.162:0/2650030991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f74b4104010 msgr2=0x7f74b411d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:36.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.114+0000 7f74b942d700 1 --2- 172.21.15.162:0/2650030991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f74b4104010 0x7f74b411d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:36.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.114+0000 7f74b942d700 1 -- 172.21.15.162:0/2650030991 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac0097c0 con 0x7f74b41048e0 2024-02-13T00:41:36.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.115+0000 7f74b942d700 1 --2- 172.21.15.162:0/2650030991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b41048e0 0x7f74b4116fa0 secure :-1 s=READY pgs=3478 cs=0 l=1 rev1=1 rx=0x7f74ac00b770 tx=0x7f74ac004d10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:36.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.115+0000 7f74b1ffb700 10 client.?.objecter ms_handle_connect 0x7f74b41048e0 2024-02-13T00:41:36.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.115+0000 7f74b1ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:36.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.116+0000 7f74b1ffb700 1 -- 172.21.15.162:0/2650030991 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac00e030 con 0x7f74b41048e0 2024-02-13T00:41:36.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.116+0000 7f74b1ffb700 1 -- 172.21.15.162:0/2650030991 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74ac00be50 con 0x7f74b41048e0 2024-02-13T00:41:36.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.116+0000 7f74b1ffb700 1 -- 172.21.15.162:0/2650030991 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac018070 con 0x7f74b41048e0 2024-02-13T00:41:36.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.116+0000 7f74bae90700 1 -- 172.21.15.162:0/2650030991 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b4117a40 con 0x7f74b41048e0 2024-02-13T00:41:36.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.116+0000 7f74bae90700 10 client.38281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:36.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.116+0000 7f74bae90700 1 -- 172.21.15.162:0/2650030991 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b4123c30 con 0x7f74b41048e0 2024-02-13T00:41:36.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.116+0000 7f74bae90700 1 librados: init done 2024-02-13T00:41:36.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.118+0000 7f749b7fe700 1 -- 172.21.15.162:0/2650030991 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7494004f20 con 0x7f74b41048e0 2024-02-13T00:41:36.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.120+0000 7f74b1ffb700 1 -- 172.21.15.162:0/2650030991 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f74ac01d8e0 con 0x7f74b41048e0 2024-02-13T00:41:36.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.120+0000 7f74b1ffb700 1 --2- 172.21.15.162:0/2650030991 >> v2:172.21.15.162:6800/2866534053 conn(0x7f749c062ce0 0x7f749c0651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:36.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.120+0000 7f74b1ffb700 1 -- 172.21.15.162:0/2650030991 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(140..140 src has 52..140) v4 ==== 10823+0+0 (secure 0 0 0) 0x7f74ac07f6f0 con 0x7f74b41048e0 2024-02-13T00:41:36.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.120+0000 7f74b1ffb700 10 client.38281.objecter ms_dispatch 0x7f74b4001550 osd_map(140..140 src has 52..140) v4 2024-02-13T00:41:36.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.120+0000 7f74b1ffb700 3 client.38281.objecter handle_osd_map got epochs [140,140] > 0 2024-02-13T00:41:36.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.120+0000 7f74b8c2c700 1 --2- 172.21.15.162:0/2650030991 >> v2:172.21.15.162:6800/2866534053 conn(0x7f749c062ce0 0x7f749c0651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:36.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.120+0000 7f74b1ffb700 3 client.38281.objecter handle_osd_map decoding full epoch 140 2024-02-13T00:41:36.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.121+0000 7f74b1ffb700 20 client.38281.objecter dump_active .. 0 homeless 2024-02-13T00:41:36.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.123+0000 7f74b8c2c700 1 --2- 172.21.15.162:0/2650030991 >> v2:172.21.15.162:6800/2866534053 conn(0x7f749c062ce0 0x7f749c0651a0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7f74a4004b90 tx=0x7f74a4009b60).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:36.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.123+0000 7f74b1ffb700 10 client.38281.objecter ms_handle_connect 0x7f749c062ce0 2024-02-13T00:41:36.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.126+0000 7f74b1ffb700 1 -- 172.21.15.162:0/2650030991 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f74ac0521a0 con 0x7f74b41048e0 2024-02-13T00:41:36.216 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:36.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:36.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:36.349 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:36.367 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:36.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:36.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:36.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.468+0000 7f749b7fe700 1 -- 172.21.15.162:0/2650030991 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7494000bc0 con 0x7f749c062ce0 2024-02-13T00:41:36.570 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:41:36.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.564+0000 7f74b1ffb700 1 -- 172.21.15.162:0/2650030991 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+1109695 (secure 0 0 0) 0x7f7494000bc0 con 0x7f749c062ce0 2024-02-13T00:41:36.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:34.268011+0000 mgr.y (mgr.14544) 166 : cluster [DBG] pgmap v178: 428 pgs: 96 unknown, 332 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:36.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:34.326270+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:41:36.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:34.352023+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:41:36.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:34.468784+0000 osd.5 (osd.5) 71 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:41:36.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:34.472309+0000 osd.5 (osd.5) 72 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:41:36.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:34.686025+0000 osd.1 (osd.1) 71 : cluster [DBG] 5.19 scrub starts 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:34.693207+0000 osd.1 (osd.1) 72 : cluster [DBG] 5.19 scrub ok 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.024985+0000 mon.b (mon.1) 146 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.352958+0000 mgr.y (mgr.14544) 167 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.379650+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:35.379902+0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.391784+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? 172.21.15.162:0/4080261016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-83985-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.414724+0000 mon.c (mon.2) 131 : audit [INF] from='client.? 172.21.15.162:0/4066446528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.425936+0000 mon.a (mon.0) 1096 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:35.778166+0000 osd.1 (osd.1) 73 : cluster [DBG] 3.0 scrub starts 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:35.787171+0000 osd.1 (osd.1) 74 : cluster [DBG] 3.0 scrub ok 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.962930+0000 mon.a (mon.0) 1097 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.974561+0000 mgr.y (mgr.14544) 168 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:36.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.988530+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:36.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.989306+0000 mgr.y (mgr.14544) 169 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:36.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.991388+0000 mon.a (mon.0) 1099 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:36.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.991598+0000 mgr.y (mgr.14544) 170 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:36.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.992036+0000 mon.a (mon.0) 1100 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:36.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.992378+0000 mgr.y (mgr.14544) 171 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:36.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.993030+0000 mon.a (mon.0) 1101 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:36.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.993203+0000 mgr.y (mgr.14544) 172 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:36.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.993610+0000 mon.a (mon.0) 1102 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:36.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.993783+0000 mgr.y (mgr.14544) 173 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:36.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.994166+0000 mon.a (mon.0) 1103 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:36.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.994323+0000 mgr.y (mgr.14544) 174 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:36.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.994693+0000 mon.a (mon.0) 1104 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:36.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.994874+0000 mgr.y (mgr.14544) 175 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:36.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.995406+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:36.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.995503+0000 mgr.y (mgr.14544) 176 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:36.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.996840+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:36.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:35.997014+0000 mgr.y (mgr.14544) 177 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:36.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:36.026583+0000 mon.b (mon.1) 147 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:36.599 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:41:36] "GET /metrics HTTP/1.1" 200 197973 "" "Prometheus/2.33.4" 2024-02-13T00:41:36.601 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:34.268011+0000 mgr.y (mgr.14544) 166 : cluster [DBG] pgmap v178: 428 pgs: 96 unknown, 332 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:36.601 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:34.326270+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:41:36.601 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:34.352023+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:41:36.601 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:34.468784+0000 osd.5 (osd.5) 71 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:41:36.601 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:34.472309+0000 osd.5 (osd.5) 72 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:41:36.601 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:34.686025+0000 osd.1 (osd.1) 71 : cluster [DBG] 5.19 scrub starts 2024-02-13T00:41:36.601 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:34.693207+0000 osd.1 (osd.1) 72 : cluster [DBG] 5.19 scrub ok 2024-02-13T00:41:36.601 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.024985+0000 mon.b (mon.1) 146 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:36.602 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.352958+0000 mgr.y (mgr.14544) 167 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:36.602 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.379650+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:36.602 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:35.379902+0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-02-13T00:41:36.602 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.391784+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? 172.21.15.162:0/4080261016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-83985-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:36.602 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.414724+0000 mon.c (mon.2) 131 : audit [INF] from='client.? 172.21.15.162:0/4066446528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:36.602 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.425936+0000 mon.a (mon.0) 1096 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:36.602 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:35.778166+0000 osd.1 (osd.1) 73 : cluster [DBG] 3.0 scrub starts 2024-02-13T00:41:36.602 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:35.787171+0000 osd.1 (osd.1) 74 : cluster [DBG] 3.0 scrub ok 2024-02-13T00:41:36.602 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.962930+0000 mon.a (mon.0) 1097 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:36.604 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.974561+0000 mgr.y (mgr.14544) 168 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:36.604 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.988530+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:36.604 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.989306+0000 mgr.y (mgr.14544) 169 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.991388+0000 mon.a (mon.0) 1099 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.991598+0000 mgr.y (mgr.14544) 170 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.992036+0000 mon.a (mon.0) 1100 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.992378+0000 mgr.y (mgr.14544) 171 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.993030+0000 mon.a (mon.0) 1101 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.993203+0000 mgr.y (mgr.14544) 172 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.993610+0000 mon.a (mon.0) 1102 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.993783+0000 mgr.y (mgr.14544) 173 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.994166+0000 mon.a (mon.0) 1103 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.994323+0000 mgr.y (mgr.14544) 174 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.994693+0000 mon.a (mon.0) 1104 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.994874+0000 mgr.y (mgr.14544) 175 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.995406+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:36.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.995503+0000 mgr.y (mgr.14544) 176 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:36.606 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.996840+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:36.606 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:35.997014+0000 mgr.y (mgr.14544) 177 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:36.606 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:36.026583+0000 mon.b (mon.1) 147 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:36.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.577+0000 7f749b7fe700 10 librados: watch_flush enter 2024-02-13T00:41:36.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.577+0000 7f749b7fe700 10 librados: watch_flush exit 2024-02-13T00:41:36.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.577+0000 7f749b7fe700 20 client.38281.objecter shutdown clearing up homeless session... 2024-02-13T00:41:36.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.577+0000 7f749b7fe700 10 client.38281.objecter successfully canceled tick 2024-02-13T00:41:36.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.577+0000 7f749b7fe700 1 -- 172.21.15.162:0/2650030991 >> v2:172.21.15.162:6800/2866534053 conn(0x7f749c062ce0 msgr2=0x7f749c0651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:36.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.577+0000 7f749b7fe700 1 --2- 172.21.15.162:0/2650030991 >> v2:172.21.15.162:6800/2866534053 conn(0x7f749c062ce0 0x7f749c0651a0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7f74a4004b90 tx=0x7f74a4009b60).stop 2024-02-13T00:41:36.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.577+0000 7f749b7fe700 1 -- 172.21.15.162:0/2650030991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b41048e0 msgr2=0x7f74b4116fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:36.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.577+0000 7f749b7fe700 1 --2- 172.21.15.162:0/2650030991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b41048e0 0x7f74b4116fa0 secure :-1 s=READY pgs=3478 cs=0 l=1 rev1=1 rx=0x7f74ac00b770 tx=0x7f74ac004d10).stop 2024-02-13T00:41:36.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.578+0000 7f749b7fe700 1 -- 172.21.15.162:0/2650030991 shutdown_connections 2024-02-13T00:41:36.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.578+0000 7f749b7fe700 1 --2- 172.21.15.162:0/2650030991 >> v2:172.21.15.162:6800/2866534053 conn(0x7f749c062ce0 0x7f749c0651a0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:36.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.578+0000 7f749b7fe700 1 --2- 172.21.15.162:0/2650030991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74b41048e0 0x7f74b4116fa0 unknown :-1 s=CLOSED pgs=3478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:36.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.578+0000 7f749b7fe700 1 --2- 172.21.15.162:0/2650030991 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f74b4069ab0 0x7f74b411ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:36.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.578+0000 7f749b7fe700 1 --2- 172.21.15.162:0/2650030991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f74b4104010 0x7f74b411d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:36.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.578+0000 7f749b7fe700 1 -- 172.21.15.162:0/2650030991 >> 172.21.15.162:0/2650030991 conn(0x7f74b4063df0 msgr2=0x7f74b4111d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:36.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.579+0000 7f749b7fe700 1 -- 172.21.15.162:0/2650030991 shutdown_connections 2024-02-13T00:41:36.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.579+0000 7f749b7fe700 1 -- 172.21.15.162:0/2650030991 wait complete. 2024-02-13T00:41:36.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:36.579+0000 7f749b7fe700 1 librados: shutdown 2024-02-13T00:41:36.613 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:41:36.637 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:36.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:34.268011+0000 mgr.y (mgr.14544) 166 : cluster [DBG] pgmap v178: 428 pgs: 96 unknown, 332 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:36.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:34.326270+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:34.352023+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:34.468784+0000 osd.5 (osd.5) 71 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:34.472309+0000 osd.5 (osd.5) 72 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:34.686025+0000 osd.1 (osd.1) 71 : cluster [DBG] 5.19 scrub starts 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:34.693207+0000 osd.1 (osd.1) 72 : cluster [DBG] 5.19 scrub ok 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.024985+0000 mon.b (mon.1) 146 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.352958+0000 mgr.y (mgr.14544) 167 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.379650+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:35.379902+0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.391784+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? 172.21.15.162:0/4080261016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-83985-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.414724+0000 mon.c (mon.2) 131 : audit [INF] from='client.? 172.21.15.162:0/4066446528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.425936+0000 mon.a (mon.0) 1096 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:36.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:35.778166+0000 osd.1 (osd.1) 73 : cluster [DBG] 3.0 scrub starts 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:35.787171+0000 osd.1 (osd.1) 74 : cluster [DBG] 3.0 scrub ok 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.962930+0000 mon.a (mon.0) 1097 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.974561+0000 mgr.y (mgr.14544) 168 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.988530+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.989306+0000 mgr.y (mgr.14544) 169 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.991388+0000 mon.a (mon.0) 1099 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.991598+0000 mgr.y (mgr.14544) 170 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.992036+0000 mon.a (mon.0) 1100 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.992378+0000 mgr.y (mgr.14544) 171 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.993030+0000 mon.a (mon.0) 1101 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.993203+0000 mgr.y (mgr.14544) 172 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.993610+0000 mon.a (mon.0) 1102 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.993783+0000 mgr.y (mgr.14544) 173 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.994166+0000 mon.a (mon.0) 1103 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:36.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.994323+0000 mgr.y (mgr.14544) 174 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-02-13T00:41:36.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.994693+0000 mon.a (mon.0) 1104 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:36.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.994874+0000 mgr.y (mgr.14544) 175 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-02-13T00:41:36.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.995406+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:36.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.995503+0000 mgr.y (mgr.14544) 176 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-02-13T00:41:36.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.996840+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:36.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:35.997014+0000 mgr.y (mgr.14544) 177 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-02-13T00:41:36.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:36.026583+0000 mon.b (mon.1) 147 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:36.870 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:36.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:36.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:37.023 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:37.137 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:37.147 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:37.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:37.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:37.309 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:37.639 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-02-13T00:41:36.419612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":151012466,"num_objects":282,"num_object_clones":1,"num_object_copies":846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":2343,"num_read_kb":50206,"num_write":1804,"num_write_kb":153119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1956,"ondisk_log_size":1956,"up":1092,"acting":1092,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1263,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2931224,"kb_used_data":456152,"kb_used_omap":0,"kb_used_meta":2475008,"kb_avail":747029992,"statfs":{"total":767960285184,"available":764958711808,"internally_reserved":0,"allocated":467099648,"data_stored":460443589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2534408192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3312,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":13,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.117643"},"pg_stats":[{"pgid":"88.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.384737+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.1d","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.420423+0000","last_change":"2024-02-13T00:41:34.421107+0000","last_active":"2024-02-13T00:41:34.420423+0000","last_peered":"2024-02-13T00:41:34.420423+0000","last_clean":"2024-02-13T00:41:34.420423+0000","last_became_active":"2024-02-13T00:41:19.768742+0000","last_became_peered":"2024-02-13T00:41:19.768742+0000","last_unstale":"2024-02-13T00:41:34.420423+0000","last_undegraded":"2024-02-13T00:41:34.420423+0000","last_fullsized":"2024-02-13T00:41:34.420423+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.342790+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.1c","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.541641+0000","last_change":"2024-02-13T00:41:34.541805+0000","last_active":"2024-02-13T00:41:34.541641+0000","last_peered":"2024-02-13T00:41:34.541641+0000","last_clean":"2024-02-13T00:41:34.541641+0000","last_became_active":"2024-02-13T00:41:19.770203+0000","last_became_peered":"2024-02-13T00:41:19.770203+0000","last_unstale":"2024-02-13T00:41:34.541641+0000","last_undegraded":"2024-02-13T00:41:34.541641+0000","last_fullsized":"2024-02-13T00:41:34.541641+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.334114+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.1f","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.358756+0000","last_change":"2024-02-13T00:41:34.377143+0000","last_active":"2024-02-13T00:41:34.358756+0000","last_peered":"2024-02-13T00:41:34.358756+0000","last_clean":"2024-02-13T00:41:34.358756+0000","last_became_active":"2024-02-13T00:41:19.737160+0000","last_became_peered":"2024-02-13T00:41:19.737160+0000","last_unstale":"2024-02-13T00:41:34.358756+0000","last_undegraded":"2024-02-13T00:41:34.358756+0000","last_fullsized":"2024-02-13T00:41:34.358756+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.338546+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.1e","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.358811+0000","last_change":"2024-02-13T00:41:34.389548+0000","last_active":"2024-02-13T00:41:34.358811+0000","last_peered":"2024-02-13T00:41:34.358811+0000","last_clean":"2024-02-13T00:41:34.358811+0000","last_became_active":"2024-02-13T00:41:19.791471+0000","last_became_peered":"2024-02-13T00:41:19.791471+0000","last_unstale":"2024-02-13T00:41:34.358811+0000","last_undegraded":"2024-02-13T00:41:34.358811+0000","last_fullsized":"2024-02-13T00:41:34.358811+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.418824+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.19","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.348386+0000","last_change":"2024-02-13T00:41:34.348821+0000","last_active":"2024-02-13T00:41:34.348386+0000","last_peered":"2024-02-13T00:41:34.348386+0000","last_clean":"2024-02-13T00:41:34.348386+0000","last_became_active":"2024-02-13T00:41:19.706396+0000","last_became_peered":"2024-02-13T00:41:19.706396+0000","last_unstale":"2024-02-13T00:41:34.348386+0000","last_undegraded":"2024-02-13T00:41:34.348386+0000","last_fullsized":"2024-02-13T00:41:34.348386+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.325546+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.18","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.479463+0000","last_change":"2024-02-13T00:41:34.482116+0000","last_active":"2024-02-13T00:41:34.479463+0000","last_peered":"2024-02-13T00:41:34.479463+0000","last_clean":"2024-02-13T00:41:34.479463+0000","last_became_active":"2024-02-13T00:41:19.716585+0000","last_became_peered":"2024-02-13T00:41:19.716585+0000","last_unstale":"2024-02-13T00:41:34.479463+0000","last_undegraded":"2024-02-13T00:41:34.479463+0000","last_fullsized":"2024-02-13T00:41:34.479463+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.414492+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.1b","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.456862+0000","last_change":"2024-02-13T00:41:34.458499+0000","last_active":"2024-02-13T00:41:34.456862+0000","last_peered":"2024-02-13T00:41:34.456862+0000","last_clean":"2024-02-13T00:41:34.456862+0000","last_became_active":"2024-02-13T00:41:19.795594+0000","last_became_peered":"2024-02-13T00:41:19.795594+0000","last_unstale":"2024-02-13T00:41:34.456862+0000","last_undegraded":"2024-02-13T00:41:34.456862+0000","last_fullsized":"2024-02-13T00:41:34.456862+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.330284+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1a","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.339725+0000","last_change":"2024-02-13T00:41:34.349326+0000","last_active":"2024-02-13T00:41:34.339725+0000","last_peered":"2024-02-13T00:41:34.339725+0000","last_clean":"2024-02-13T00:41:34.339725+0000","last_became_active":"2024-02-13T00:41:19.749743+0000","last_became_peered":"2024-02-13T00:41:19.749743+0000","last_unstale":"2024-02-13T00:41:34.339725+0000","last_undegraded":"2024-02-13T00:41:34.339725+0000","last_fullsized":"2024-02-13T00:41:34.339725+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.412264+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.15","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.357004+0000","last_change":"2024-02-13T00:41:34.357207+0000","last_active":"2024-02-13T00:41:34.357004+0000","last_peered":"2024-02-13T00:41:34.357004+0000","last_clean":"2024-02-13T00:41:34.357004+0000","last_became_active":"2024-02-13T00:41:19.749716+0000","last_became_peered":"2024-02-13T00:41:19.749716+0000","last_unstale":"2024-02-13T00:41:34.357004+0000","last_undegraded":"2024-02-13T00:41:34.357004+0000","last_fullsized":"2024-02-13T00:41:34.357004+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.333948+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.14","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.332274+0000","last_change":"2024-02-13T00:41:34.332895+0000","last_active":"2024-02-13T00:41:34.332274+0000","last_peered":"2024-02-13T00:41:34.332274+0000","last_clean":"2024-02-13T00:41:34.332274+0000","last_became_active":"2024-02-13T00:41:19.765401+0000","last_became_peered":"2024-02-13T00:41:19.765401+0000","last_unstale":"2024-02-13T00:41:34.332274+0000","last_undegraded":"2024-02-13T00:41:34.332274+0000","last_fullsized":"2024-02-13T00:41:34.332274+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.352601+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.17","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.472692+0000","last_change":"2024-02-13T00:41:34.473240+0000","last_active":"2024-02-13T00:41:34.472692+0000","last_peered":"2024-02-13T00:41:34.472692+0000","last_clean":"2024-02-13T00:41:34.472692+0000","last_became_active":"2024-02-13T00:41:19.712784+0000","last_became_peered":"2024-02-13T00:41:19.712784+0000","last_unstale":"2024-02-13T00:41:34.472692+0000","last_undegraded":"2024-02-13T00:41:34.472692+0000","last_fullsized":"2024-02-13T00:41:34.472692+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.417195+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.16","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.377444+0000","last_change":"2024-02-13T00:41:34.389731+0000","last_active":"2024-02-13T00:41:34.377444+0000","last_peered":"2024-02-13T00:41:34.377444+0000","last_clean":"2024-02-13T00:41:34.377444+0000","last_became_active":"2024-02-13T00:41:19.791851+0000","last_became_peered":"2024-02-13T00:41:19.791851+0000","last_unstale":"2024-02-13T00:41:34.377444+0000","last_undegraded":"2024-02-13T00:41:34.377444+0000","last_fullsized":"2024-02-13T00:41:34.377444+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.338386+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.11","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.454112+0000","last_change":"2024-02-13T00:41:34.460918+0000","last_active":"2024-02-13T00:41:34.454112+0000","last_peered":"2024-02-13T00:41:34.454112+0000","last_clean":"2024-02-13T00:41:34.454112+0000","last_became_active":"2024-02-13T00:41:19.769476+0000","last_became_peered":"2024-02-13T00:41:19.769476+0000","last_unstale":"2024-02-13T00:41:34.454112+0000","last_undegraded":"2024-02-13T00:41:34.454112+0000","last_fullsized":"2024-02-13T00:41:34.454112+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.404053+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.10","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.454050+0000","last_change":"2024-02-13T00:41:34.460534+0000","last_active":"2024-02-13T00:41:34.454050+0000","last_peered":"2024-02-13T00:41:34.454050+0000","last_clean":"2024-02-13T00:41:34.454050+0000","last_became_active":"2024-02-13T00:41:19.767935+0000","last_became_peered":"2024-02-13T00:41:19.767935+0000","last_unstale":"2024-02-13T00:41:34.454050+0000","last_undegraded":"2024-02-13T00:41:34.454050+0000","last_fullsized":"2024-02-13T00:41:34.454050+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.325467+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.13","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.335392+0000","last_change":"2024-02-13T00:41:34.335783+0000","last_active":"2024-02-13T00:41:34.335392+0000","last_peered":"2024-02-13T00:41:34.335392+0000","last_clean":"2024-02-13T00:41:34.335392+0000","last_became_active":"2024-02-13T00:41:19.748009+0000","last_became_peered":"2024-02-13T00:41:19.748009+0000","last_unstale":"2024-02-13T00:41:34.335392+0000","last_undegraded":"2024-02-13T00:41:34.335392+0000","last_fullsized":"2024-02-13T00:41:34.335392+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.416043+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.12","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.348442+0000","last_change":"2024-02-13T00:41:34.348612+0000","last_active":"2024-02-13T00:41:34.348442+0000","last_peered":"2024-02-13T00:41:34.348442+0000","last_clean":"2024-02-13T00:41:34.348442+0000","last_became_active":"2024-02-13T00:41:19.750699+0000","last_became_peered":"2024-02-13T00:41:19.750699+0000","last_unstale":"2024-02-13T00:41:34.348442+0000","last_undegraded":"2024-02-13T00:41:34.348442+0000","last_fullsized":"2024-02-13T00:41:34.348442+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.420281+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"78.1","version":"0'0","reported_seq":22,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.462836+0000","last_change":"2024-02-13T00:41:33.462836+0000","last_active":"2024-02-13T00:41:33.462836+0000","last_peered":"2024-02-13T00:41:33.462836+0000","last_clean":"2024-02-13T00:41:33.462836+0000","last_became_active":"2024-02-13T00:41:29.259018+0000","last_became_peered":"2024-02-13T00:41:29.259018+0000","last_unstale":"2024-02-13T00:41:33.462836+0000","last_undegraded":"2024-02-13T00:41:33.462836+0000","last_fullsized":"2024-02-13T00:41:33.462836+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_clean_scrub_stamp":"2024-02-13T00:41:28.078905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"66.d","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.329953+0000","last_change":"2024-02-13T00:41:34.330354+0000","last_active":"2024-02-13T00:41:34.329953+0000","last_peered":"2024-02-13T00:41:34.329953+0000","last_clean":"2024-02-13T00:41:34.329953+0000","last_became_active":"2024-02-13T00:41:19.747287+0000","last_became_peered":"2024-02-13T00:41:19.747287+0000","last_unstale":"2024-02-13T00:41:34.329953+0000","last_undegraded":"2024-02-13T00:41:34.329953+0000","last_fullsized":"2024-02-13T00:41:34.329953+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.418434+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"78.0","version":"139'4","reported_seq":26,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.509523+0000","last_change":"2024-02-13T00:41:33.490335+0000","last_active":"2024-02-13T00:41:34.509523+0000","last_peered":"2024-02-13T00:41:34.509523+0000","last_clean":"2024-02-13T00:41:34.509523+0000","last_became_active":"2024-02-13T00:41:29.279554+0000","last_became_peered":"2024-02-13T00:41:29.279554+0000","last_unstale":"2024-02-13T00:41:34.509523+0000","last_undegraded":"2024-02-13T00:41:34.509523+0000","last_fullsized":"2024-02-13T00:41:34.509523+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_clean_scrub_stamp":"2024-02-13T00:41:28.078905+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"66.c","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.353756+0000","last_change":"2024-02-13T00:41:34.354163+0000","last_active":"2024-02-13T00:41:34.353756+0000","last_peered":"2024-02-13T00:41:34.353756+0000","last_clean":"2024-02-13T00:41:34.353756+0000","last_became_active":"2024-02-13T00:41:19.792123+0000","last_became_peered":"2024-02-13T00:41:19.792123+0000","last_unstale":"2024-02-13T00:41:34.353756+0000","last_undegraded":"2024-02-13T00:41:34.353756+0000","last_fullsized":"2024-02-13T00:41:34.353756+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.417861+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"78.3","version":"0'0","reported_seq":22,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.344743+0000","last_change":"2024-02-13T00:41:33.344743+0000","last_active":"2024-02-13T00:41:33.344743+0000","last_peered":"2024-02-13T00:41:33.344743+0000","last_clean":"2024-02-13T00:41:33.344743+0000","last_became_active":"2024-02-13T00:41:29.351740+0000","last_became_peered":"2024-02-13T00:41:29.351740+0000","last_unstale":"2024-02-13T00:41:33.344743+0000","last_undegraded":"2024-02-13T00:41:33.344743+0000","last_fullsized":"2024-02-13T00:41:33.344743+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_clean_scrub_stamp":"2024-02-13T00:41:28.078905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"66.f","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.421956+0000","last_change":"2024-02-13T00:41:34.430466+0000","last_active":"2024-02-13T00:41:34.421956+0000","last_peered":"2024-02-13T00:41:34.421956+0000","last_clean":"2024-02-13T00:41:34.421956+0000","last_became_active":"2024-02-13T00:41:19.773026+0000","last_became_peered":"2024-02-13T00:41:19.773026+0000","last_unstale":"2024-02-13T00:41:34.421956+0000","last_undegraded":"2024-02-13T00:41:34.421956+0000","last_fullsized":"2024-02-13T00:41:34.421956+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.388260+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"78.2","version":"0'0","reported_seq":22,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.334122+0000","last_change":"2024-02-13T00:41:33.334122+0000","last_active":"2024-02-13T00:41:33.334122+0000","last_peered":"2024-02-13T00:41:33.334122+0000","last_clean":"2024-02-13T00:41:33.334122+0000","last_became_active":"2024-02-13T00:41:29.276357+0000","last_became_peered":"2024-02-13T00:41:29.276357+0000","last_unstale":"2024-02-13T00:41:33.334122+0000","last_undegraded":"2024-02-13T00:41:33.334122+0000","last_fullsized":"2024-02-13T00:41:33.334122+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_clean_scrub_stamp":"2024-02-13T00:41:28.078905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"66.e","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.443612+0000","last_change":"2024-02-13T00:41:34.446027+0000","last_active":"2024-02-13T00:41:34.443612+0000","last_peered":"2024-02-13T00:41:34.443612+0000","last_clean":"2024-02-13T00:41:34.443612+0000","last_became_active":"2024-02-13T00:41:19.779615+0000","last_became_peered":"2024-02-13T00:41:19.779615+0000","last_unstale":"2024-02-13T00:41:34.443612+0000","last_undegraded":"2024-02-13T00:41:34.443612+0000","last_fullsized":"2024-02-13T00:41:34.443612+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.325595+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"78.5","version":"0'0","reported_seq":22,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.328074+0000","last_change":"2024-02-13T00:41:33.328074+0000","last_active":"2024-02-13T00:41:33.328074+0000","last_peered":"2024-02-13T00:41:33.328074+0000","last_clean":"2024-02-13T00:41:33.328074+0000","last_became_active":"2024-02-13T00:41:29.249274+0000","last_became_peered":"2024-02-13T00:41:29.249274+0000","last_unstale":"2024-02-13T00:41:33.328074+0000","last_undegraded":"2024-02-13T00:41:33.328074+0000","last_fullsized":"2024-02-13T00:41:33.328074+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_clean_scrub_stamp":"2024-02-13T00:41:28.078905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"66.9","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.330874+0000","last_change":"2024-02-13T00:41:34.341690+0000","last_active":"2024-02-13T00:41:34.330874+0000","last_peered":"2024-02-13T00:41:34.330874+0000","last_clean":"2024-02-13T00:41:34.330874+0000","last_became_active":"2024-02-13T00:41:19.669776+0000","last_became_peered":"2024-02-13T00:41:19.669776+0000","last_unstale":"2024-02-13T00:41:34.330874+0000","last_undegraded":"2024-02-13T00:41:34.330874+0000","last_fullsized":"2024-02-13T00:41:34.330874+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.405213+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"78.4","version":"136'2","reported_seq":24,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.337501+0000","last_change":"2024-02-13T00:41:33.337501+0000","last_active":"2024-02-13T00:41:33.337501+0000","last_peered":"2024-02-13T00:41:33.337501+0000","last_clean":"2024-02-13T00:41:33.337501+0000","last_became_active":"2024-02-13T00:41:29.291247+0000","last_became_peered":"2024-02-13T00:41:29.291247+0000","last_unstale":"2024-02-13T00:41:33.337501+0000","last_undegraded":"2024-02-13T00:41:33.337501+0000","last_fullsized":"2024-02-13T00:41:33.337501+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_clean_scrub_stamp":"2024-02-13T00:41:28.078905+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"66.8","version":"138'20","reported_seq":67,"reported_epoch":138,"state":"active+clean+snaptrim","last_fresh":"2024-02-13T00:41:34.446384+0000","last_change":"2024-02-13T00:41:34.350546+0000","last_active":"2024-02-13T00:41:34.446384+0000","last_peered":"2024-02-13T00:41:34.446384+0000","last_clean":"2024-02-13T00:41:34.446384+0000","last_became_active":"2024-02-13T00:41:19.706474+0000","last_became_peered":"2024-02-13T00:41:19.706474+0000","last_unstale":"2024-02-13T00:41:34.446384+0000","last_undegraded":"2024-02-13T00:41:34.446384+0000","last_fullsized":"2024-02-13T00:41:34.446384+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":1280,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"88.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.340312+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"78.7","version":"0'0","reported_seq":22,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.335011+0000","last_change":"2024-02-13T00:41:33.335011+0000","last_active":"2024-02-13T00:41:33.335011+0000","last_peered":"2024-02-13T00:41:33.335011+0000","last_clean":"2024-02-13T00:41:33.335011+0000","last_became_active":"2024-02-13T00:41:29.350965+0000","last_became_peered":"2024-02-13T00:41:29.350965+0000","last_unstale":"2024-02-13T00:41:33.335011+0000","last_undegraded":"2024-02-13T00:41:33.335011+0000","last_fullsized":"2024-02-13T00:41:33.335011+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_clean_scrub_stamp":"2024-02-13T00:41:28.078905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"66.b","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.414987+0000","last_change":"2024-02-13T00:41:34.420281+0000","last_active":"2024-02-13T00:41:34.414987+0000","last_peered":"2024-02-13T00:41:34.414987+0000","last_clean":"2024-02-13T00:41:34.414987+0000","last_became_active":"2024-02-13T00:41:19.765065+0000","last_became_peered":"2024-02-13T00:41:19.765065+0000","last_unstale":"2024-02-13T00:41:34.414987+0000","last_undegraded":"2024-02-13T00:41:34.414987+0000","last_fullsized":"2024-02-13T00:41:34.414987+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.405745+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"78.6","version":"0'0","reported_seq":22,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.334693+0000","last_change":"2024-02-13T00:41:33.334693+0000","last_active":"2024-02-13T00:41:33.334693+0000","last_peered":"2024-02-13T00:41:33.334693+0000","last_clean":"2024-02-13T00:41:33.334693+0000","last_became_active":"2024-02-13T00:41:29.349863+0000","last_became_peered":"2024-02-13T00:41:29.349863+0000","last_unstale":"2024-02-13T00:41:33.334693+0000","last_undegraded":"2024-02-13T00:41:33.334693+0000","last_fullsized":"2024-02-13T00:41:33.334693+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:28.078905+0000","last_clean_scrub_stamp":"2024-02-13T00:41:28.078905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"66.a","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.432671+0000","last_change":"2024-02-13T00:41:34.433488+0000","last_active":"2024-02-13T00:41:34.432671+0000","last_peered":"2024-02-13T00:41:34.432671+0000","last_clean":"2024-02-13T00:41:34.432671+0000","last_became_active":"2024-02-13T00:41:19.745521+0000","last_became_peered":"2024-02-13T00:41:19.745521+0000","last_unstale":"2024-02-13T00:41:34.432671+0000","last_undegraded":"2024-02-13T00:41:34.432671+0000","last_fullsized":"2024-02-13T00:41:34.432671+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.413103+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.5","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.330000+0000","last_change":"2024-02-13T00:41:34.330487+0000","last_active":"2024-02-13T00:41:34.330000+0000","last_peered":"2024-02-13T00:41:34.330000+0000","last_clean":"2024-02-13T00:41:34.330000+0000","last_became_active":"2024-02-13T00:41:19.794804+0000","last_became_peered":"2024-02-13T00:41:19.794804+0000","last_unstale":"2024-02-13T00:41:34.330000+0000","last_undegraded":"2024-02-13T00:41:34.330000+0000","last_fullsized":"2024-02-13T00:41:34.330000+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.325095+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.4","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.353805+0000","last_change":"2024-02-13T00:41:34.354307+0000","last_active":"2024-02-13T00:41:34.353805+0000","last_peered":"2024-02-13T00:41:34.353805+0000","last_clean":"2024-02-13T00:41:34.353805+0000","last_became_active":"2024-02-13T00:41:19.740855+0000","last_became_peered":"2024-02-13T00:41:19.740855+0000","last_unstale":"2024-02-13T00:41:34.353805+0000","last_undegraded":"2024-02-13T00:41:34.353805+0000","last_fullsized":"2024-02-13T00:41:34.353805+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.466876+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.7","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.357061+0000","last_change":"2024-02-13T00:41:34.357802+0000","last_active":"2024-02-13T00:41:34.357061+0000","last_peered":"2024-02-13T00:41:34.357061+0000","last_clean":"2024-02-13T00:41:34.357061+0000","last_became_active":"2024-02-13T00:41:19.792032+0000","last_became_peered":"2024-02-13T00:41:19.792032+0000","last_unstale":"2024-02-13T00:41:34.357061+0000","last_undegraded":"2024-02-13T00:41:34.357061+0000","last_fullsized":"2024-02-13T00:41:34.357061+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.325091+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.6","version":"132'15","reported_seq":72,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.351214+0000","last_change":"2024-02-13T00:41:34.351482+0000","last_active":"2024-02-13T00:41:34.351214+0000","last_peered":"2024-02-13T00:41:34.351214+0000","last_clean":"2024-02-13T00:41:34.351214+0000","last_became_active":"2024-02-13T00:41:19.737000+0000","last_became_peered":"2024-02-13T00:41:19.737000+0000","last_unstale":"2024-02-13T00:41:34.351214+0000","last_undegraded":"2024-02-13T00:41:34.351214+0000","last_fullsized":"2024-02-13T00:41:34.351214+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":384,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.389193+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"66.1","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.338632+0000","last_change":"2024-02-13T00:41:34.339271+0000","last_active":"2024-02-13T00:41:34.338632+0000","last_peered":"2024-02-13T00:41:34.338632+0000","last_clean":"2024-02-13T00:41:34.338632+0000","last_became_active":"2024-02-13T00:41:19.792351+0000","last_became_peered":"2024-02-13T00:41:19.792351+0000","last_unstale":"2024-02-13T00:41:34.338632+0000","last_undegraded":"2024-02-13T00:41:34.338632+0000","last_fullsized":"2024-02-13T00:41:34.338632+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.326191+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.0","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.340471+0000","last_change":"2024-02-13T00:41:34.344174+0000","last_active":"2024-02-13T00:41:34.340471+0000","last_peered":"2024-02-13T00:41:34.340471+0000","last_clean":"2024-02-13T00:41:34.340471+0000","last_became_active":"2024-02-13T00:41:19.742888+0000","last_became_peered":"2024-02-13T00:41:19.742888+0000","last_unstale":"2024-02-13T00:41:34.340471+0000","last_undegraded":"2024-02-13T00:41:34.340471+0000","last_fullsized":"2024-02-13T00:41:34.340471+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.415510+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.3","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.349115+0000","last_change":"2024-02-13T00:41:34.349558+0000","last_active":"2024-02-13T00:41:34.349115+0000","last_peered":"2024-02-13T00:41:34.349115+0000","last_clean":"2024-02-13T00:41:34.349115+0000","last_became_active":"2024-02-13T00:41:19.706557+0000","last_became_peered":"2024-02-13T00:41:19.706557+0000","last_unstale":"2024-02-13T00:41:34.349115+0000","last_undegraded":"2024-02-13T00:41:34.349115+0000","last_fullsized":"2024-02-13T00:41:34.349115+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"88.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:36.375667+0000","last_change":"2024-02-13T00:41:36.375667+0000","last_active":"2024-02-13T00:41:36.375667+0000","last_peered":"2024-02-13T00:41:36.375667+0000","last_clean":"2024-02-13T00:41:36.375667+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:36.375667+0000","last_undegraded":"2024-02-13T00:41:36.375667+0000","last_fullsized":"2024-02-13T00:41:36.375667+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:36.375667+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.375667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"87.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"86.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:41:35.325474+0000","last_change":"2024-02-13T00:41:35.325474+0000","last_active":"2024-02-13T00:41:35.325474+0000","last_peered":"2024-02-13T00:41:35.325474+0000","last_clean":"2024-02-13T00:41:35.325474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:35.325474+0000","last_undegraded":"2024-02-13T00:41:35.325474+0000","last_fullsized":"2024-02-13T00:41:35.325474+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:35.325474+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.325474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"85.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:41:34.312128+0000","last_change":"2024-02-13T00:41:34.420875+0000","last_active":"2024-02-13T00:41:34.312128+0000","last_peered":"2024-02-13T00:41:34.312128+0000","last_clean":"2024-02-13T00:41:34.312128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:41:34.312128+0000","last_undegraded":"2024-02-13T00:41:34.312128+0000","last_fullsized":"2024-02-13T00:41:34.312128+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:34.312128+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.312128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.2","version":"0'0","reported_seq":52,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.421895+0000","last_change":"2024-02-13T00:41:34.428825+0000","last_active":"2024-02-13T00:41:34.421895+0000","last_peered":"2024-02-13T00:41:34.421895+0000","last_clean":"2024-02-13T00:41:34.421895+0000","last_became_active":"2024-02-13T00:41:19.770148+0000","last_became_peered":"2024-02-13T00:41:19.770148+0000","last_unstale":"2024-02-13T00:41:34.421895+0000","last_undegraded":"2024-02-13T00:41:34.421895+0000","last_fullsized":"2024-02-13T00:41:34.421895+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:18.764513+0000","last_clean_scrub_stamp":"2024-02-13T00:41:18.764513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"a","length":"3"}]},{"pgid":"61.2","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.450220+0000","last_change":"2024-02-13T00:41:16.777309+0000","last_active":"2024-02-13T00:41:33.450220+0000","last_peered":"2024-02-13T00:41:33.450220+0000","last_clean":"2024-02-13T00:41:33.450220+0000","last_became_active":"2024-02-13T00:41:16.777030+0000","last_became_peered":"2024-02-13T00:41:16.777030+0000","last_unstale":"2024-02-13T00:41:33.450220+0000","last_undegraded":"2024-02-13T00:41:33.450220+0000","last_fullsized":"2024-02-13T00:41:33.450220+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.12","version":"114'3","reported_seq":39,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.321553+0000","last_change":"2024-02-13T00:41:09.356060+0000","last_active":"2024-02-13T00:41:33.321553+0000","last_peered":"2024-02-13T00:41:33.321553+0000","last_clean":"2024-02-13T00:41:33.321553+0000","last_became_active":"2024-02-13T00:41:09.355064+0000","last_became_peered":"2024-02-13T00:41:09.355064+0000","last_unstale":"2024-02-13T00:41:33.321553+0000","last_undegraded":"2024-02-13T00:41:33.321553+0000","last_fullsized":"2024-02-13T00:41:33.321553+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.3","version":"124'1","reported_seq":28,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.426181+0000","last_change":"2024-02-13T00:41:16.803893+0000","last_active":"2024-02-13T00:41:33.426181+0000","last_peered":"2024-02-13T00:41:33.426181+0000","last_clean":"2024-02-13T00:41:33.426181+0000","last_became_active":"2024-02-13T00:41:16.803266+0000","last_became_peered":"2024-02-13T00:41:16.803266+0000","last_unstale":"2024-02-13T00:41:33.426181+0000","last_undegraded":"2024-02-13T00:41:33.426181+0000","last_fullsized":"2024-02-13T00:41:33.426181+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.13","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.417180+0000","last_change":"2024-02-13T00:41:08.325300+0000","last_active":"2024-02-13T00:41:33.417180+0000","last_peered":"2024-02-13T00:41:33.417180+0000","last_clean":"2024-02-13T00:41:33.417180+0000","last_became_active":"2024-02-13T00:41:08.324812+0000","last_became_peered":"2024-02-13T00:41:08.324812+0000","last_unstale":"2024-02-13T00:41:33.417180+0000","last_undegraded":"2024-02-13T00:41:33.417180+0000","last_fullsized":"2024-02-13T00:41:33.417180+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.0","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.322241+0000","last_change":"2024-02-13T00:41:16.768299+0000","last_active":"2024-02-13T00:41:33.322241+0000","last_peered":"2024-02-13T00:41:33.322241+0000","last_clean":"2024-02-13T00:41:33.322241+0000","last_became_active":"2024-02-13T00:41:16.768067+0000","last_became_peered":"2024-02-13T00:41:16.768067+0000","last_unstale":"2024-02-13T00:41:33.322241+0000","last_undegraded":"2024-02-13T00:41:33.322241+0000","last_fullsized":"2024-02-13T00:41:33.322241+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"45.10","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.319229+0000","last_change":"2024-02-13T00:41:09.334192+0000","last_active":"2024-02-13T00:41:33.319229+0000","last_peered":"2024-02-13T00:41:33.319229+0000","last_clean":"2024-02-13T00:41:33.319229+0000","last_became_active":"2024-02-13T00:41:09.333680+0000","last_became_peered":"2024-02-13T00:41:09.333680+0000","last_unstale":"2024-02-13T00:41:33.319229+0000","last_undegraded":"2024-02-13T00:41:33.319229+0000","last_fullsized":"2024-02-13T00:41:33.319229+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.414276+0000","last_change":"2024-02-13T00:41:16.734847+0000","last_active":"2024-02-13T00:41:33.414276+0000","last_peered":"2024-02-13T00:41:33.414276+0000","last_clean":"2024-02-13T00:41:33.414276+0000","last_became_active":"2024-02-13T00:41:16.734552+0000","last_became_peered":"2024-02-13T00:41:16.734552+0000","last_unstale":"2024-02-13T00:41:33.414276+0000","last_undegraded":"2024-02-13T00:41:33.414276+0000","last_fullsized":"2024-02-13T00:41:33.414276+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.11","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.310318+0000","last_change":"2024-02-13T00:41:08.322159+0000","last_active":"2024-02-13T00:41:33.310318+0000","last_peered":"2024-02-13T00:41:33.310318+0000","last_clean":"2024-02-13T00:41:33.310318+0000","last_became_active":"2024-02-13T00:41:08.321905+0000","last_became_peered":"2024-02-13T00:41:08.321905+0000","last_unstale":"2024-02-13T00:41:33.310318+0000","last_undegraded":"2024-02-13T00:41:33.310318+0000","last_fullsized":"2024-02-13T00:41:33.310318+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.6","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.458325+0000","last_change":"2024-02-13T00:41:16.791810+0000","last_active":"2024-02-13T00:41:33.458325+0000","last_peered":"2024-02-13T00:41:33.458325+0000","last_clean":"2024-02-13T00:41:33.458325+0000","last_became_active":"2024-02-13T00:41:16.791271+0000","last_became_peered":"2024-02-13T00:41:16.791271+0000","last_unstale":"2024-02-13T00:41:33.458325+0000","last_undegraded":"2024-02-13T00:41:33.458325+0000","last_fullsized":"2024-02-13T00:41:33.458325+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"45.16","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.458334+0000","last_change":"2024-02-13T00:41:08.317745+0000","last_active":"2024-02-13T00:41:33.458334+0000","last_peered":"2024-02-13T00:41:33.458334+0000","last_clean":"2024-02-13T00:41:33.458334+0000","last_became_active":"2024-02-13T00:41:08.317194+0000","last_became_peered":"2024-02-13T00:41:08.317194+0000","last_unstale":"2024-02-13T00:41:33.458334+0000","last_undegraded":"2024-02-13T00:41:33.458334+0000","last_fullsized":"2024-02-13T00:41:33.458334+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.7","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.332479+0000","last_change":"2024-02-13T00:41:16.776299+0000","last_active":"2024-02-13T00:41:33.332479+0000","last_peered":"2024-02-13T00:41:33.332479+0000","last_clean":"2024-02-13T00:41:33.332479+0000","last_became_active":"2024-02-13T00:41:16.776064+0000","last_became_peered":"2024-02-13T00:41:16.776064+0000","last_unstale":"2024-02-13T00:41:33.332479+0000","last_undegraded":"2024-02-13T00:41:33.332479+0000","last_fullsized":"2024-02-13T00:41:33.332479+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"45.17","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333211+0000","last_change":"2024-02-13T00:41:09.367691+0000","last_active":"2024-02-13T00:41:33.333211+0000","last_peered":"2024-02-13T00:41:33.333211+0000","last_clean":"2024-02-13T00:41:33.333211+0000","last_became_active":"2024-02-13T00:41:09.367392+0000","last_became_peered":"2024-02-13T00:41:09.367392+0000","last_unstale":"2024-02-13T00:41:33.333211+0000","last_undegraded":"2024-02-13T00:41:33.333211+0000","last_fullsized":"2024-02-13T00:41:33.333211+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.4","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.324858+0000","last_change":"2024-02-13T00:41:16.791256+0000","last_active":"2024-02-13T00:41:33.324858+0000","last_peered":"2024-02-13T00:41:33.324858+0000","last_clean":"2024-02-13T00:41:33.324858+0000","last_became_active":"2024-02-13T00:41:16.790979+0000","last_became_peered":"2024-02-13T00:41:16.790979+0000","last_unstale":"2024-02-13T00:41:33.324858+0000","last_undegraded":"2024-02-13T00:41:33.324858+0000","last_fullsized":"2024-02-13T00:41:33.324858+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"45.14","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.325263+0000","last_change":"2024-02-13T00:41:09.320159+0000","last_active":"2024-02-13T00:41:33.325263+0000","last_peered":"2024-02-13T00:41:33.325263+0000","last_clean":"2024-02-13T00:41:33.325263+0000","last_became_active":"2024-02-13T00:41:09.319697+0000","last_became_peered":"2024-02-13T00:41:09.319697+0000","last_unstale":"2024-02-13T00:41:33.325263+0000","last_undegraded":"2024-02-13T00:41:33.325263+0000","last_fullsized":"2024-02-13T00:41:33.325263+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.5","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.431627+0000","last_change":"2024-02-13T00:41:16.742685+0000","last_active":"2024-02-13T00:41:33.431627+0000","last_peered":"2024-02-13T00:41:33.431627+0000","last_clean":"2024-02-13T00:41:33.431627+0000","last_became_active":"2024-02-13T00:41:16.733735+0000","last_became_peered":"2024-02-13T00:41:16.733735+0000","last_unstale":"2024-02-13T00:41:33.431627+0000","last_undegraded":"2024-02-13T00:41:33.431627+0000","last_fullsized":"2024-02-13T00:41:33.431627+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.15","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330323+0000","last_change":"2024-02-13T00:41:09.369472+0000","last_active":"2024-02-13T00:41:33.330323+0000","last_peered":"2024-02-13T00:41:33.330323+0000","last_clean":"2024-02-13T00:41:33.330323+0000","last_became_active":"2024-02-13T00:41:09.369094+0000","last_became_peered":"2024-02-13T00:41:09.369094+0000","last_unstale":"2024-02-13T00:41:33.330323+0000","last_undegraded":"2024-02-13T00:41:33.330323+0000","last_fullsized":"2024-02-13T00:41:33.330323+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.a","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.450293+0000","last_change":"2024-02-13T00:41:16.735342+0000","last_active":"2024-02-13T00:41:33.450293+0000","last_peered":"2024-02-13T00:41:33.450293+0000","last_clean":"2024-02-13T00:41:33.450293+0000","last_became_active":"2024-02-13T00:41:16.735087+0000","last_became_peered":"2024-02-13T00:41:16.735087+0000","last_unstale":"2024-02-13T00:41:33.450293+0000","last_undegraded":"2024-02-13T00:41:33.450293+0000","last_fullsized":"2024-02-13T00:41:33.450293+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.1a","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.424897+0000","last_change":"2024-02-13T00:41:08.336972+0000","last_active":"2024-02-13T00:41:33.424897+0000","last_peered":"2024-02-13T00:41:33.424897+0000","last_clean":"2024-02-13T00:41:33.424897+0000","last_became_active":"2024-02-13T00:41:08.336664+0000","last_became_peered":"2024-02-13T00:41:08.336664+0000","last_unstale":"2024-02-13T00:41:33.424897+0000","last_undegraded":"2024-02-13T00:41:33.424897+0000","last_fullsized":"2024-02-13T00:41:33.424897+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.b","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.426104+0000","last_change":"2024-02-13T00:41:16.804014+0000","last_active":"2024-02-13T00:41:33.426104+0000","last_peered":"2024-02-13T00:41:33.426104+0000","last_clean":"2024-02-13T00:41:33.426104+0000","last_became_active":"2024-02-13T00:41:16.803624+0000","last_became_peered":"2024-02-13T00:41:16.803624+0000","last_unstale":"2024-02-13T00:41:33.426104+0000","last_undegraded":"2024-02-13T00:41:33.426104+0000","last_fullsized":"2024-02-13T00:41:33.426104+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.1b","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.417059+0000","last_change":"2024-02-13T00:41:08.309199+0000","last_active":"2024-02-13T00:41:33.417059+0000","last_peered":"2024-02-13T00:41:33.417059+0000","last_clean":"2024-02-13T00:41:33.417059+0000","last_became_active":"2024-02-13T00:41:08.307466+0000","last_became_peered":"2024-02-13T00:41:08.307466+0000","last_unstale":"2024-02-13T00:41:33.417059+0000","last_undegraded":"2024-02-13T00:41:33.417059+0000","last_fullsized":"2024-02-13T00:41:33.417059+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.8","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.326392+0000","last_change":"2024-02-13T00:41:16.749716+0000","last_active":"2024-02-13T00:41:33.326392+0000","last_peered":"2024-02-13T00:41:33.326392+0000","last_clean":"2024-02-13T00:41:33.326392+0000","last_became_active":"2024-02-13T00:41:16.749465+0000","last_became_peered":"2024-02-13T00:41:16.749465+0000","last_unstale":"2024-02-13T00:41:33.326392+0000","last_undegraded":"2024-02-13T00:41:33.326392+0000","last_fullsized":"2024-02-13T00:41:33.326392+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"45.18","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.459215+0000","last_change":"2024-02-13T00:41:08.322109+0000","last_active":"2024-02-13T00:41:33.459215+0000","last_peered":"2024-02-13T00:41:33.459215+0000","last_clean":"2024-02-13T00:41:33.459215+0000","last_became_active":"2024-02-13T00:41:08.321821+0000","last_became_peered":"2024-02-13T00:41:08.321821+0000","last_unstale":"2024-02-13T00:41:33.459215+0000","last_undegraded":"2024-02-13T00:41:33.459215+0000","last_fullsized":"2024-02-13T00:41:33.459215+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.9","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.414202+0000","last_change":"2024-02-13T00:41:16.768748+0000","last_active":"2024-02-13T00:41:33.414202+0000","last_peered":"2024-02-13T00:41:33.414202+0000","last_clean":"2024-02-13T00:41:33.414202+0000","last_became_active":"2024-02-13T00:41:16.768452+0000","last_became_peered":"2024-02-13T00:41:16.768452+0000","last_unstale":"2024-02-13T00:41:33.414202+0000","last_undegraded":"2024-02-13T00:41:33.414202+0000","last_fullsized":"2024-02-13T00:41:33.414202+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.19","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.316755+0000","last_change":"2024-02-13T00:41:09.335853+0000","last_active":"2024-02-13T00:41:33.316755+0000","last_peered":"2024-02-13T00:41:33.316755+0000","last_clean":"2024-02-13T00:41:33.316755+0000","last_became_active":"2024-02-13T00:41:09.335381+0000","last_became_peered":"2024-02-13T00:41:09.335381+0000","last_unstale":"2024-02-13T00:41:33.316755+0000","last_undegraded":"2024-02-13T00:41:33.316755+0000","last_fullsized":"2024-02-13T00:41:33.316755+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.e","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330828+0000","last_change":"2024-02-13T00:41:16.696344+0000","last_active":"2024-02-13T00:41:33.330828+0000","last_peered":"2024-02-13T00:41:33.330828+0000","last_clean":"2024-02-13T00:41:33.330828+0000","last_became_active":"2024-02-13T00:41:16.696032+0000","last_became_peered":"2024-02-13T00:41:16.696032+0000","last_unstale":"2024-02-13T00:41:33.330828+0000","last_undegraded":"2024-02-13T00:41:33.330828+0000","last_fullsized":"2024-02-13T00:41:33.330828+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.1e","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.458260+0000","last_change":"2024-02-13T00:41:08.328958+0000","last_active":"2024-02-13T00:41:33.458260+0000","last_peered":"2024-02-13T00:41:33.458260+0000","last_clean":"2024-02-13T00:41:33.458260+0000","last_became_active":"2024-02-13T00:41:08.328534+0000","last_became_peered":"2024-02-13T00:41:08.328534+0000","last_unstale":"2024-02-13T00:41:33.458260+0000","last_undegraded":"2024-02-13T00:41:33.458260+0000","last_fullsized":"2024-02-13T00:41:33.458260+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.f","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.341354+0000","last_change":"2024-02-13T00:41:16.697669+0000","last_active":"2024-02-13T00:41:33.341354+0000","last_peered":"2024-02-13T00:41:33.341354+0000","last_clean":"2024-02-13T00:41:33.341354+0000","last_became_active":"2024-02-13T00:41:16.697369+0000","last_became_peered":"2024-02-13T00:41:16.697369+0000","last_unstale":"2024-02-13T00:41:33.341354+0000","last_undegraded":"2024-02-13T00:41:33.341354+0000","last_fullsized":"2024-02-13T00:41:33.341354+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.1f","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.334847+0000","last_change":"2024-02-13T00:41:08.304780+0000","last_active":"2024-02-13T00:41:33.334847+0000","last_peered":"2024-02-13T00:41:33.334847+0000","last_clean":"2024-02-13T00:41:33.334847+0000","last_became_active":"2024-02-13T00:41:08.304461+0000","last_became_peered":"2024-02-13T00:41:08.304461+0000","last_unstale":"2024-02-13T00:41:33.334847+0000","last_undegraded":"2024-02-13T00:41:33.334847+0000","last_fullsized":"2024-02-13T00:41:33.334847+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.c","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.325012+0000","last_change":"2024-02-13T00:41:16.703254+0000","last_active":"2024-02-13T00:41:33.325012+0000","last_peered":"2024-02-13T00:41:33.325012+0000","last_clean":"2024-02-13T00:41:33.325012+0000","last_became_active":"2024-02-13T00:41:16.702990+0000","last_became_peered":"2024-02-13T00:41:16.702990+0000","last_unstale":"2024-02-13T00:41:33.325012+0000","last_undegraded":"2024-02-13T00:41:33.325012+0000","last_fullsized":"2024-02-13T00:41:33.325012+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"45.1c","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.427413+0000","last_change":"2024-02-13T00:41:08.317554+0000","last_active":"2024-02-13T00:41:33.427413+0000","last_peered":"2024-02-13T00:41:33.427413+0000","last_clean":"2024-02-13T00:41:33.427413+0000","last_became_active":"2024-02-13T00:41:08.317271+0000","last_became_peered":"2024-02-13T00:41:08.317271+0000","last_unstale":"2024-02-13T00:41:33.427413+0000","last_undegraded":"2024-02-13T00:41:33.427413+0000","last_fullsized":"2024-02-13T00:41:33.427413+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.d","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.317161+0000","last_change":"2024-02-13T00:41:16.780518+0000","last_active":"2024-02-13T00:41:33.317161+0000","last_peered":"2024-02-13T00:41:33.317161+0000","last_clean":"2024-02-13T00:41:33.317161+0000","last_became_active":"2024-02-13T00:41:16.780278+0000","last_became_peered":"2024-02-13T00:41:16.780278+0000","last_unstale":"2024-02-13T00:41:33.317161+0000","last_undegraded":"2024-02-13T00:41:33.317161+0000","last_fullsized":"2024-02-13T00:41:33.317161+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"45.1d","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.339248+0000","last_change":"2024-02-13T00:41:09.345739+0000","last_active":"2024-02-13T00:41:33.339248+0000","last_peered":"2024-02-13T00:41:33.339248+0000","last_clean":"2024-02-13T00:41:33.339248+0000","last_became_active":"2024-02-13T00:41:09.345433+0000","last_became_peered":"2024-02-13T00:41:09.345433+0000","last_unstale":"2024-02-13T00:41:33.339248+0000","last_undegraded":"2024-02-13T00:41:33.339248+0000","last_fullsized":"2024-02-13T00:41:33.339248+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.12","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.322662+0000","last_change":"2024-02-13T00:41:16.779591+0000","last_active":"2024-02-13T00:41:33.322662+0000","last_peered":"2024-02-13T00:41:33.322662+0000","last_clean":"2024-02-13T00:41:33.322662+0000","last_became_active":"2024-02-13T00:41:16.779347+0000","last_became_peered":"2024-02-13T00:41:16.779347+0000","last_unstale":"2024-02-13T00:41:33.322662+0000","last_undegraded":"2024-02-13T00:41:33.322662+0000","last_fullsized":"2024-02-13T00:41:33.322662+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"45.2","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.329836+0000","last_change":"2024-02-13T00:41:09.341966+0000","last_active":"2024-02-13T00:41:33.329836+0000","last_peered":"2024-02-13T00:41:33.329836+0000","last_clean":"2024-02-13T00:41:33.329836+0000","last_became_active":"2024-02-13T00:41:09.341220+0000","last_became_peered":"2024-02-13T00:41:09.341220+0000","last_unstale":"2024-02-13T00:41:33.329836+0000","last_undegraded":"2024-02-13T00:41:33.329836+0000","last_fullsized":"2024-02-13T00:41:33.329836+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.13","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.340395+0000","last_change":"2024-02-13T00:41:16.696135+0000","last_active":"2024-02-13T00:41:33.340395+0000","last_peered":"2024-02-13T00:41:33.340395+0000","last_clean":"2024-02-13T00:41:33.340395+0000","last_became_active":"2024-02-13T00:41:16.695842+0000","last_became_peered":"2024-02-13T00:41:16.695842+0000","last_unstale":"2024-02-13T00:41:33.340395+0000","last_undegraded":"2024-02-13T00:41:33.340395+0000","last_fullsized":"2024-02-13T00:41:33.340395+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.3","version":"113'1","reported_seq":37,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.446669+0000","last_change":"2024-02-13T00:41:08.315245+0000","last_active":"2024-02-13T00:41:33.446669+0000","last_peered":"2024-02-13T00:41:33.446669+0000","last_clean":"2024-02-13T00:41:33.446669+0000","last_became_active":"2024-02-13T00:41:08.315006+0000","last_became_peered":"2024-02-13T00:41:08.315006+0000","last_unstale":"2024-02-13T00:41:33.446669+0000","last_undegraded":"2024-02-13T00:41:33.446669+0000","last_fullsized":"2024-02-13T00:41:33.446669+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.10","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.329762+0000","last_change":"2024-02-13T00:41:16.802641+0000","last_active":"2024-02-13T00:41:33.329762+0000","last_peered":"2024-02-13T00:41:33.329762+0000","last_clean":"2024-02-13T00:41:33.329762+0000","last_became_active":"2024-02-13T00:41:16.802269+0000","last_became_peered":"2024-02-13T00:41:16.802269+0000","last_unstale":"2024-02-13T00:41:33.329762+0000","last_undegraded":"2024-02-13T00:41:33.329762+0000","last_fullsized":"2024-02-13T00:41:33.329762+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.0","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.443821+0000","last_change":"2024-02-13T00:41:08.327898+0000","last_active":"2024-02-13T00:41:33.443821+0000","last_peered":"2024-02-13T00:41:33.443821+0000","last_clean":"2024-02-13T00:41:33.443821+0000","last_became_active":"2024-02-13T00:41:08.327674+0000","last_became_peered":"2024-02-13T00:41:08.327674+0000","last_unstale":"2024-02-13T00:41:33.443821+0000","last_undegraded":"2024-02-13T00:41:33.443821+0000","last_fullsized":"2024-02-13T00:41:33.443821+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.11","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.419572+0000","last_change":"2024-02-13T00:41:16.792703+0000","last_active":"2024-02-13T00:41:33.419572+0000","last_peered":"2024-02-13T00:41:33.419572+0000","last_clean":"2024-02-13T00:41:33.419572+0000","last_became_active":"2024-02-13T00:41:16.792442+0000","last_became_peered":"2024-02-13T00:41:16.792442+0000","last_unstale":"2024-02-13T00:41:33.419572+0000","last_undegraded":"2024-02-13T00:41:33.419572+0000","last_fullsized":"2024-02-13T00:41:33.419572+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"45.1","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.314384+0000","last_change":"2024-02-13T00:41:09.320847+0000","last_active":"2024-02-13T00:41:33.314384+0000","last_peered":"2024-02-13T00:41:33.314384+0000","last_clean":"2024-02-13T00:41:33.314384+0000","last_became_active":"2024-02-13T00:41:09.314193+0000","last_became_peered":"2024-02-13T00:41:09.314193+0000","last_unstale":"2024-02-13T00:41:33.314384+0000","last_undegraded":"2024-02-13T00:41:33.314384+0000","last_fullsized":"2024-02-13T00:41:33.314384+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.16","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.316608+0000","last_change":"2024-02-13T00:41:16.694545+0000","last_active":"2024-02-13T00:41:33.316608+0000","last_peered":"2024-02-13T00:41:33.316608+0000","last_clean":"2024-02-13T00:41:33.316608+0000","last_became_active":"2024-02-13T00:41:16.694271+0000","last_became_peered":"2024-02-13T00:41:16.694271+0000","last_unstale":"2024-02-13T00:41:33.316608+0000","last_undegraded":"2024-02-13T00:41:33.316608+0000","last_fullsized":"2024-02-13T00:41:33.316608+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"45.6","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.458405+0000","last_change":"2024-02-13T00:41:08.317886+0000","last_active":"2024-02-13T00:41:33.458405+0000","last_peered":"2024-02-13T00:41:33.458405+0000","last_clean":"2024-02-13T00:41:33.458405+0000","last_became_active":"2024-02-13T00:41:08.317571+0000","last_became_peered":"2024-02-13T00:41:08.317571+0000","last_unstale":"2024-02-13T00:41:33.458405+0000","last_undegraded":"2024-02-13T00:41:33.458405+0000","last_fullsized":"2024-02-13T00:41:33.458405+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.17","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.321984+0000","last_change":"2024-02-13T00:41:16.793550+0000","last_active":"2024-02-13T00:41:33.321984+0000","last_peered":"2024-02-13T00:41:33.321984+0000","last_clean":"2024-02-13T00:41:33.321984+0000","last_became_active":"2024-02-13T00:41:16.793277+0000","last_became_peered":"2024-02-13T00:41:16.793277+0000","last_unstale":"2024-02-13T00:41:33.321984+0000","last_undegraded":"2024-02-13T00:41:33.321984+0000","last_fullsized":"2024-02-13T00:41:33.321984+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"45.7","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333131+0000","last_change":"2024-02-13T00:41:09.362076+0000","last_active":"2024-02-13T00:41:33.333131+0000","last_peered":"2024-02-13T00:41:33.333131+0000","last_clean":"2024-02-13T00:41:33.333131+0000","last_became_active":"2024-02-13T00:41:09.359737+0000","last_became_peered":"2024-02-13T00:41:09.359737+0000","last_unstale":"2024-02-13T00:41:33.333131+0000","last_undegraded":"2024-02-13T00:41:33.333131+0000","last_fullsized":"2024-02-13T00:41:33.333131+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.14","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.320131+0000","last_change":"2024-02-13T00:41:16.784578+0000","last_active":"2024-02-13T00:41:33.320131+0000","last_peered":"2024-02-13T00:41:33.320131+0000","last_clean":"2024-02-13T00:41:33.320131+0000","last_became_active":"2024-02-13T00:41:16.784290+0000","last_became_peered":"2024-02-13T00:41:16.784290+0000","last_unstale":"2024-02-13T00:41:33.320131+0000","last_undegraded":"2024-02-13T00:41:33.320131+0000","last_fullsized":"2024-02-13T00:41:33.320131+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"45.4","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.325026+0000","last_change":"2024-02-13T00:41:08.305883+0000","last_active":"2024-02-13T00:41:33.325026+0000","last_peered":"2024-02-13T00:41:33.325026+0000","last_clean":"2024-02-13T00:41:33.325026+0000","last_became_active":"2024-02-13T00:41:08.305638+0000","last_became_peered":"2024-02-13T00:41:08.305638+0000","last_unstale":"2024-02-13T00:41:33.325026+0000","last_undegraded":"2024-02-13T00:41:33.325026+0000","last_fullsized":"2024-02-13T00:41:33.325026+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.15","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.339172+0000","last_change":"2024-02-13T00:41:16.752916+0000","last_active":"2024-02-13T00:41:33.339172+0000","last_peered":"2024-02-13T00:41:33.339172+0000","last_clean":"2024-02-13T00:41:33.339172+0000","last_became_active":"2024-02-13T00:41:16.752633+0000","last_became_peered":"2024-02-13T00:41:16.752633+0000","last_unstale":"2024-02-13T00:41:33.339172+0000","last_undegraded":"2024-02-13T00:41:33.339172+0000","last_fullsized":"2024-02-13T00:41:33.339172+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.5","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.329898+0000","last_change":"2024-02-13T00:41:08.322229+0000","last_active":"2024-02-13T00:41:33.329898+0000","last_peered":"2024-02-13T00:41:33.329898+0000","last_clean":"2024-02-13T00:41:33.329898+0000","last_became_active":"2024-02-13T00:41:08.321981+0000","last_became_peered":"2024-02-13T00:41:08.321981+0000","last_unstale":"2024-02-13T00:41:33.329898+0000","last_undegraded":"2024-02-13T00:41:33.329898+0000","last_fullsized":"2024-02-13T00:41:33.329898+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.1a","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.329756+0000","last_change":"2024-02-13T00:41:16.753032+0000","last_active":"2024-02-13T00:41:33.329756+0000","last_peered":"2024-02-13T00:41:33.329756+0000","last_clean":"2024-02-13T00:41:33.329756+0000","last_became_active":"2024-02-13T00:41:16.752735+0000","last_became_peered":"2024-02-13T00:41:16.752735+0000","last_unstale":"2024-02-13T00:41:33.329756+0000","last_undegraded":"2024-02-13T00:41:33.329756+0000","last_fullsized":"2024-02-13T00:41:33.329756+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.a","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.321625+0000","last_change":"2024-02-13T00:41:09.334489+0000","last_active":"2024-02-13T00:41:33.321625+0000","last_peered":"2024-02-13T00:41:33.321625+0000","last_clean":"2024-02-13T00:41:33.321625+0000","last_became_active":"2024-02-13T00:41:09.334205+0000","last_became_peered":"2024-02-13T00:41:09.334205+0000","last_unstale":"2024-02-13T00:41:33.321625+0000","last_undegraded":"2024-02-13T00:41:33.321625+0000","last_fullsized":"2024-02-13T00:41:33.321625+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1b","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.446586+0000","last_change":"2024-02-13T00:41:16.748663+0000","last_active":"2024-02-13T00:41:33.446586+0000","last_peered":"2024-02-13T00:41:33.446586+0000","last_clean":"2024-02-13T00:41:33.446586+0000","last_became_active":"2024-02-13T00:41:16.748358+0000","last_became_peered":"2024-02-13T00:41:16.748358+0000","last_unstale":"2024-02-13T00:41:33.446586+0000","last_undegraded":"2024-02-13T00:41:33.446586+0000","last_fullsized":"2024-02-13T00:41:33.446586+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.b","version":"114'1","reported_seq":37,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.426362+0000","last_change":"2024-02-13T00:41:08.316708+0000","last_active":"2024-02-13T00:41:33.426362+0000","last_peered":"2024-02-13T00:41:33.426362+0000","last_clean":"2024-02-13T00:41:33.426362+0000","last_became_active":"2024-02-13T00:41:08.316413+0000","last_became_peered":"2024-02-13T00:41:08.316413+0000","last_unstale":"2024-02-13T00:41:33.426362+0000","last_undegraded":"2024-02-13T00:41:33.426362+0000","last_fullsized":"2024-02-13T00:41:33.426362+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.18","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.443736+0000","last_change":"2024-02-13T00:41:16.754735+0000","last_active":"2024-02-13T00:41:33.443736+0000","last_peered":"2024-02-13T00:41:33.443736+0000","last_clean":"2024-02-13T00:41:33.443736+0000","last_became_active":"2024-02-13T00:41:16.747555+0000","last_became_peered":"2024-02-13T00:41:16.747555+0000","last_unstale":"2024-02-13T00:41:33.443736+0000","last_undegraded":"2024-02-13T00:41:33.443736+0000","last_fullsized":"2024-02-13T00:41:33.443736+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.8","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.356914+0000","last_change":"2024-02-13T00:41:08.334178+0000","last_active":"2024-02-13T00:41:33.356914+0000","last_peered":"2024-02-13T00:41:33.356914+0000","last_clean":"2024-02-13T00:41:33.356914+0000","last_became_active":"2024-02-13T00:41:08.333987+0000","last_became_peered":"2024-02-13T00:41:08.333987+0000","last_unstale":"2024-02-13T00:41:33.356914+0000","last_undegraded":"2024-02-13T00:41:33.356914+0000","last_fullsized":"2024-02-13T00:41:33.356914+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.19","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.316628+0000","last_change":"2024-02-13T00:41:16.749291+0000","last_active":"2024-02-13T00:41:33.316628+0000","last_peered":"2024-02-13T00:41:33.316628+0000","last_clean":"2024-02-13T00:41:33.316628+0000","last_became_active":"2024-02-13T00:41:16.749013+0000","last_became_peered":"2024-02-13T00:41:16.749013+0000","last_unstale":"2024-02-13T00:41:33.316628+0000","last_undegraded":"2024-02-13T00:41:33.316628+0000","last_fullsized":"2024-02-13T00:41:33.316628+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"45.9","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.321001+0000","last_change":"2024-02-13T00:41:09.344767+0000","last_active":"2024-02-13T00:41:33.321001+0000","last_peered":"2024-02-13T00:41:33.321001+0000","last_clean":"2024-02-13T00:41:33.321001+0000","last_became_active":"2024-02-13T00:41:09.344486+0000","last_became_peered":"2024-02-13T00:41:09.344486+0000","last_unstale":"2024-02-13T00:41:33.321001+0000","last_undegraded":"2024-02-13T00:41:33.321001+0000","last_fullsized":"2024-02-13T00:41:33.321001+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1e","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.340382+0000","last_change":"2024-02-13T00:41:16.802612+0000","last_active":"2024-02-13T00:41:33.340382+0000","last_peered":"2024-02-13T00:41:33.340382+0000","last_clean":"2024-02-13T00:41:33.340382+0000","last_became_active":"2024-02-13T00:41:16.802249+0000","last_became_peered":"2024-02-13T00:41:16.802249+0000","last_unstale":"2024-02-13T00:41:33.340382+0000","last_undegraded":"2024-02-13T00:41:33.340382+0000","last_fullsized":"2024-02-13T00:41:33.340382+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.e","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.340472+0000","last_change":"2024-02-13T00:41:09.340564+0000","last_active":"2024-02-13T00:41:33.340472+0000","last_peered":"2024-02-13T00:41:33.340472+0000","last_clean":"2024-02-13T00:41:33.340472+0000","last_became_active":"2024-02-13T00:41:09.340131+0000","last_became_peered":"2024-02-13T00:41:09.340131+0000","last_unstale":"2024-02-13T00:41:33.340472+0000","last_undegraded":"2024-02-13T00:41:33.340472+0000","last_fullsized":"2024-02-13T00:41:33.340472+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1f","version":"131'1","reported_seq":29,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.334700+0000","last_change":"2024-02-13T00:41:16.807574+0000","last_active":"2024-02-13T00:41:33.334700+0000","last_peered":"2024-02-13T00:41:33.334700+0000","last_clean":"2024-02-13T00:41:33.334700+0000","last_became_active":"2024-02-13T00:41:16.807294+0000","last_became_peered":"2024-02-13T00:41:16.807294+0000","last_unstale":"2024-02-13T00:41:33.334700+0000","last_undegraded":"2024-02-13T00:41:33.334700+0000","last_fullsized":"2024-02-13T00:41:33.334700+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"45.f","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.322123+0000","last_change":"2024-02-13T00:41:08.322110+0000","last_active":"2024-02-13T00:41:33.322123+0000","last_peered":"2024-02-13T00:41:33.322123+0000","last_clean":"2024-02-13T00:41:33.322123+0000","last_became_active":"2024-02-13T00:41:08.321786+0000","last_became_peered":"2024-02-13T00:41:08.321786+0000","last_unstale":"2024-02-13T00:41:33.322123+0000","last_undegraded":"2024-02-13T00:41:33.322123+0000","last_fullsized":"2024-02-13T00:41:33.322123+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.1c","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.324766+0000","last_change":"2024-02-13T00:41:16.692253+0000","last_active":"2024-02-13T00:41:33.324766+0000","last_peered":"2024-02-13T00:41:33.324766+0000","last_clean":"2024-02-13T00:41:33.324766+0000","last_became_active":"2024-02-13T00:41:16.691756+0000","last_became_peered":"2024-02-13T00:41:16.691756+0000","last_unstale":"2024-02-13T00:41:33.324766+0000","last_undegraded":"2024-02-13T00:41:33.324766+0000","last_fullsized":"2024-02-13T00:41:33.324766+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"45.c","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.328337+0000","last_change":"2024-02-13T00:41:09.358548+0000","last_active":"2024-02-13T00:41:33.328337+0000","last_peered":"2024-02-13T00:41:33.328337+0000","last_clean":"2024-02-13T00:41:33.328337+0000","last_became_active":"2024-02-13T00:41:09.358314+0000","last_became_peered":"2024-02-13T00:41:09.358314+0000","last_unstale":"2024-02-13T00:41:33.328337+0000","last_undegraded":"2024-02-13T00:41:33.328337+0000","last_fullsized":"2024-02-13T00:41:33.328337+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1d","version":"0'0","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.431562+0000","last_change":"2024-02-13T00:41:16.802863+0000","last_active":"2024-02-13T00:41:33.431562+0000","last_peered":"2024-02-13T00:41:33.431562+0000","last_clean":"2024-02-13T00:41:33.431562+0000","last_became_active":"2024-02-13T00:41:16.802530+0000","last_became_peered":"2024-02-13T00:41:16.802530+0000","last_unstale":"2024-02-13T00:41:33.431562+0000","last_undegraded":"2024-02-13T00:41:33.431562+0000","last_fullsized":"2024-02-13T00:41:33.431562+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.d","version":"0'0","reported_seq":35,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.455036+0000","last_change":"2024-02-13T00:41:08.320380+0000","last_active":"2024-02-13T00:41:33.455036+0000","last_peered":"2024-02-13T00:41:33.455036+0000","last_clean":"2024-02-13T00:41:33.455036+0000","last_became_active":"2024-02-13T00:41:08.320119+0000","last_became_peered":"2024-02-13T00:41:08.320119+0000","last_unstale":"2024-02-13T00:41:33.455036+0000","last_undegraded":"2024-02-13T00:41:33.455036+0000","last_fullsized":"2024-02-13T00:41:33.455036+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:07.234414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.234414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.340338+0000","last_change":"2024-02-13T00:41:01.591014+0000","last_active":"2024-02-13T00:41:33.340338+0000","last_peered":"2024-02-13T00:41:33.340338+0000","last_clean":"2024-02-13T00:41:33.340338+0000","last_became_active":"2024-02-13T00:41:01.590358+0000","last_became_peered":"2024-02-13T00:41:01.590358+0000","last_unstale":"2024-02-13T00:41:33.340338+0000","last_undegraded":"2024-02-13T00:41:33.340338+0000","last_fullsized":"2024-02-13T00:41:33.340338+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"64'8","reported_seq":151,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.272003+0000","last_change":"2024-02-13T00:40:58.179518+0000","last_active":"2024-02-13T00:41:34.272003+0000","last_peered":"2024-02-13T00:41:34.272003+0000","last_clean":"2024-02-13T00:41:34.272003+0000","last_became_active":"2024-02-13T00:37:45.247905+0000","last_became_peered":"2024-02-13T00:37:45.247905+0000","last_unstale":"2024-02-13T00:41:34.272003+0000","last_undegraded":"2024-02-13T00:41:34.272003+0000","last_fullsized":"2024-02-13T00:41:34.272003+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:40:58.179446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.179446+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":162,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.326738+0000","last_change":"2024-02-13T00:41:31.413681+0000","last_active":"2024-02-13T00:41:33.326738+0000","last_peered":"2024-02-13T00:41:33.326738+0000","last_clean":"2024-02-13T00:41:33.326738+0000","last_became_active":"2024-02-13T00:37:43.322374+0000","last_became_peered":"2024-02-13T00:37:43.322374+0000","last_unstale":"2024-02-13T00:41:33.326738+0000","last_undegraded":"2024-02-13T00:41:33.326738+0000","last_fullsized":"2024-02-13T00:41:33.326738+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:41:31.413581+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_clean_scrub_stamp":"2024-02-13T00:41:31.413581+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.458439+0000","last_change":"2024-02-13T00:40:50.436030+0000","last_active":"2024-02-13T00:41:33.458439+0000","last_peered":"2024-02-13T00:41:33.458439+0000","last_clean":"2024-02-13T00:41:33.458439+0000","last_became_active":"2024-02-13T00:37:41.219093+0000","last_became_peered":"2024-02-13T00:37:41.219093+0000","last_unstale":"2024-02-13T00:41:33.458439+0000","last_undegraded":"2024-02-13T00:41:33.458439+0000","last_fullsized":"2024-02-13T00:41:33.458439+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:50.435958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:50.435958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.316814+0000","last_change":"2024-02-13T00:39:17.713034+0000","last_active":"2024-02-13T00:41:33.316814+0000","last_peered":"2024-02-13T00:41:33.316814+0000","last_clean":"2024-02-13T00:41:33.316814+0000","last_became_active":"2024-02-13T00:37:47.552384+0000","last_became_peered":"2024-02-13T00:37:47.552384+0000","last_unstale":"2024-02-13T00:41:33.316814+0000","last_undegraded":"2024-02-13T00:41:33.316814+0000","last_fullsized":"2024-02-13T00:41:33.316814+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.712965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.712965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.316748+0000","last_change":"2024-02-13T00:41:01.627210+0000","last_active":"2024-02-13T00:41:33.316748+0000","last_peered":"2024-02-13T00:41:33.316748+0000","last_clean":"2024-02-13T00:41:33.316748+0000","last_became_active":"2024-02-13T00:41:01.624745+0000","last_became_peered":"2024-02-13T00:41:01.624745+0000","last_unstale":"2024-02-13T00:41:33.316748+0000","last_undegraded":"2024-02-13T00:41:33.316748+0000","last_fullsized":"2024-02-13T00:41:33.316748+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330306+0000","last_change":"2024-02-13T00:41:01.738760+0000","last_active":"2024-02-13T00:41:33.330306+0000","last_peered":"2024-02-13T00:41:33.330306+0000","last_clean":"2024-02-13T00:41:33.330306+0000","last_became_active":"2024-02-13T00:41:01.738337+0000","last_became_peered":"2024-02-13T00:41:01.738337+0000","last_unstale":"2024-02-13T00:41:33.330306+0000","last_undegraded":"2024-02-13T00:41:33.330306+0000","last_fullsized":"2024-02-13T00:41:33.330306+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.463943+0000","last_change":"2024-02-13T00:41:01.568682+0000","last_active":"2024-02-13T00:41:33.463943+0000","last_peered":"2024-02-13T00:41:33.463943+0000","last_clean":"2024-02-13T00:41:33.463943+0000","last_became_active":"2024-02-13T00:41:01.568236+0000","last_became_peered":"2024-02-13T00:41:01.568236+0000","last_unstale":"2024-02-13T00:41:33.463943+0000","last_undegraded":"2024-02-13T00:41:33.463943+0000","last_fullsized":"2024-02-13T00:41:33.463943+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":124,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.329062+0000","last_change":"2024-02-13T00:39:24.376175+0000","last_active":"2024-02-13T00:41:33.329062+0000","last_peered":"2024-02-13T00:41:33.329062+0000","last_clean":"2024-02-13T00:41:33.329062+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:41:33.329062+0000","last_undegraded":"2024-02-13T00:41:33.329062+0000","last_fullsized":"2024-02-13T00:41:33.329062+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:39:24.376100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.376100+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":96,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.445460+0000","last_change":"2024-02-13T00:39:16.262449+0000","last_active":"2024-02-13T00:41:33.445460+0000","last_peered":"2024-02-13T00:41:33.445460+0000","last_clean":"2024-02-13T00:41:33.445460+0000","last_became_active":"2024-02-13T00:37:41.212429+0000","last_became_peered":"2024-02-13T00:37:41.212429+0000","last_unstale":"2024-02-13T00:41:33.445460+0000","last_undegraded":"2024-02-13T00:41:33.445460+0000","last_fullsized":"2024-02-13T00:41:33.445460+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.262380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.262380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":97,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.322853+0000","last_change":"2024-02-13T00:40:53.362331+0000","last_active":"2024-02-13T00:41:33.322853+0000","last_peered":"2024-02-13T00:41:33.322853+0000","last_clean":"2024-02-13T00:41:33.322853+0000","last_became_active":"2024-02-13T00:37:45.240923+0000","last_became_peered":"2024-02-13T00:37:45.240923+0000","last_unstale":"2024-02-13T00:41:33.322853+0000","last_undegraded":"2024-02-13T00:41:33.322853+0000","last_fullsized":"2024-02-13T00:41:33.322853+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:53.362232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:53.362232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.311167+0000","last_change":"2024-02-13T00:39:18.359681+0000","last_active":"2024-02-13T00:41:33.311167+0000","last_peered":"2024-02-13T00:41:33.311167+0000","last_clean":"2024-02-13T00:41:33.311167+0000","last_became_active":"2024-02-13T00:37:47.249556+0000","last_became_peered":"2024-02-13T00:37:47.249556+0000","last_unstale":"2024-02-13T00:41:33.311167+0000","last_undegraded":"2024-02-13T00:41:33.311167+0000","last_fullsized":"2024-02-13T00:41:33.311167+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.359604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1d","version":"108'7","reported_seq":49,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330462+0000","last_change":"2024-02-13T00:41:01.697227+0000","last_active":"2024-02-13T00:41:33.330462+0000","last_peered":"2024-02-13T00:41:33.330462+0000","last_clean":"2024-02-13T00:41:33.330462+0000","last_became_active":"2024-02-13T00:41:01.696134+0000","last_became_peered":"2024-02-13T00:41:01.696134+0000","last_unstale":"2024-02-13T00:41:33.330462+0000","last_undegraded":"2024-02-13T00:41:33.330462+0000","last_fullsized":"2024-02-13T00:41:33.330462+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.1b","version":"109'5","reported_seq":54,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:35.159923+0000","last_change":"2024-02-13T00:41:01.594685+0000","last_active":"2024-02-13T00:41:35.159923+0000","last_peered":"2024-02-13T00:41:35.159923+0000","last_clean":"2024-02-13T00:41:35.159923+0000","last_became_active":"2024-02-13T00:41:01.594405+0000","last_became_peered":"2024-02-13T00:41:01.594405+0000","last_unstale":"2024-02-13T00:41:35.159923+0000","last_undegraded":"2024-02-13T00:41:35.159923+0000","last_fullsized":"2024-02-13T00:41:35.159923+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50332680,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":51903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.334035+0000","last_change":"2024-02-13T00:41:01.656850+0000","last_active":"2024-02-13T00:41:33.334035+0000","last_peered":"2024-02-13T00:41:33.334035+0000","last_clean":"2024-02-13T00:41:33.334035+0000","last_became_active":"2024-02-13T00:41:01.654410+0000","last_became_peered":"2024-02-13T00:41:01.654410+0000","last_unstale":"2024-02-13T00:41:33.334035+0000","last_undegraded":"2024-02-13T00:41:33.334035+0000","last_fullsized":"2024-02-13T00:41:33.334035+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"91'35","reported_seq":155,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.322164+0000","last_change":"2024-02-13T00:40:15.074106+0000","last_active":"2024-02-13T00:41:33.322164+0000","last_peered":"2024-02-13T00:41:33.322164+0000","last_clean":"2024-02-13T00:41:33.322164+0000","last_became_active":"2024-02-13T00:37:43.322957+0000","last_became_peered":"2024-02-13T00:37:43.322957+0000","last_unstale":"2024-02-13T00:41:33.322164+0000","last_undegraded":"2024-02-13T00:41:33.322164+0000","last_fullsized":"2024-02-13T00:41:33.322164+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:40:15.074034+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":96,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.335668+0000","last_change":"2024-02-13T00:39:18.672595+0000","last_active":"2024-02-13T00:41:33.335668+0000","last_peered":"2024-02-13T00:41:33.335668+0000","last_clean":"2024-02-13T00:41:33.335668+0000","last_became_active":"2024-02-13T00:37:41.219457+0000","last_became_peered":"2024-02-13T00:37:41.219457+0000","last_unstale":"2024-02-13T00:41:33.335668+0000","last_undegraded":"2024-02-13T00:41:33.335668+0000","last_fullsized":"2024-02-13T00:41:33.335668+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:18.672516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:18.672516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":97,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.417304+0000","last_change":"2024-02-13T00:41:03.411330+0000","last_active":"2024-02-13T00:41:33.417304+0000","last_peered":"2024-02-13T00:41:33.417304+0000","last_clean":"2024-02-13T00:41:33.417304+0000","last_became_active":"2024-02-13T00:37:45.239867+0000","last_became_peered":"2024-02-13T00:37:45.239867+0000","last_unstale":"2024-02-13T00:41:33.417304+0000","last_undegraded":"2024-02-13T00:41:33.417304+0000","last_fullsized":"2024-02-13T00:41:33.417304+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:03.411257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_clean_scrub_stamp":"2024-02-13T00:41:03.411257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":91,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.483197+0000","last_change":"2024-02-13T00:41:33.483197+0000","last_active":"2024-02-13T00:41:33.483197+0000","last_peered":"2024-02-13T00:41:33.483197+0000","last_clean":"2024-02-13T00:41:33.483197+0000","last_became_active":"2024-02-13T00:37:47.254705+0000","last_became_peered":"2024-02-13T00:37:47.254705+0000","last_unstale":"2024-02-13T00:41:33.483197+0000","last_undegraded":"2024-02-13T00:41:33.483197+0000","last_fullsized":"2024-02-13T00:41:33.483197+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.086207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.086207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.428587+0000","last_change":"2024-02-13T00:41:01.733877+0000","last_active":"2024-02-13T00:41:33.428587+0000","last_peered":"2024-02-13T00:41:33.428587+0000","last_clean":"2024-02-13T00:41:33.428587+0000","last_became_active":"2024-02-13T00:41:01.733241+0000","last_became_peered":"2024-02-13T00:41:01.733241+0000","last_unstale":"2024-02-13T00:41:33.428587+0000","last_undegraded":"2024-02-13T00:41:33.428587+0000","last_fullsized":"2024-02-13T00:41:33.428587+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.317354+0000","last_change":"2024-02-13T00:41:01.576784+0000","last_active":"2024-02-13T00:41:33.317354+0000","last_peered":"2024-02-13T00:41:33.317354+0000","last_clean":"2024-02-13T00:41:33.317354+0000","last_became_active":"2024-02-13T00:41:01.576488+0000","last_became_peered":"2024-02-13T00:41:01.576488+0000","last_unstale":"2024-02-13T00:41:33.317354+0000","last_undegraded":"2024-02-13T00:41:33.317354+0000","last_fullsized":"2024-02-13T00:41:33.317354+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.445117+0000","last_change":"2024-02-13T00:41:01.735253+0000","last_active":"2024-02-13T00:41:33.445117+0000","last_peered":"2024-02-13T00:41:33.445117+0000","last_clean":"2024-02-13T00:41:33.445117+0000","last_became_active":"2024-02-13T00:41:01.727925+0000","last_became_peered":"2024-02-13T00:41:01.727925+0000","last_unstale":"2024-02-13T00:41:33.445117+0000","last_undegraded":"2024-02-13T00:41:33.445117+0000","last_fullsized":"2024-02-13T00:41:33.445117+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.334951+0000","last_change":"2024-02-13T00:40:56.434571+0000","last_active":"2024-02-13T00:41:33.334951+0000","last_peered":"2024-02-13T00:41:33.334951+0000","last_clean":"2024-02-13T00:41:33.334951+0000","last_became_active":"2024-02-13T00:40:35.850675+0000","last_became_peered":"2024-02-13T00:40:35.850675+0000","last_unstale":"2024-02-13T00:41:33.334951+0000","last_undegraded":"2024-02-13T00:41:33.334951+0000","last_fullsized":"2024-02-13T00:41:33.334951+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:40:56.434506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:56.434506+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":107,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.340860+0000","last_change":"2024-02-13T00:41:00.128667+0000","last_active":"2024-02-13T00:41:33.340860+0000","last_peered":"2024-02-13T00:41:33.340860+0000","last_clean":"2024-02-13T00:41:33.340860+0000","last_became_active":"2024-02-13T00:37:41.220874+0000","last_became_peered":"2024-02-13T00:37:41.220874+0000","last_unstale":"2024-02-13T00:41:33.340860+0000","last_undegraded":"2024-02-13T00:41:33.340860+0000","last_fullsized":"2024-02-13T00:41:33.340860+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:41:00.128595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.128595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":92,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.428362+0000","last_change":"2024-02-13T00:39:35.269607+0000","last_active":"2024-02-13T00:41:33.428362+0000","last_peered":"2024-02-13T00:41:33.428362+0000","last_clean":"2024-02-13T00:41:33.428362+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:41:33.428362+0000","last_undegraded":"2024-02-13T00:41:33.428362+0000","last_fullsized":"2024-02-13T00:41:33.428362+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:35.269538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:35.269538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":95,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.335037+0000","last_change":"2024-02-13T00:41:02.402785+0000","last_active":"2024-02-13T00:41:33.335037+0000","last_peered":"2024-02-13T00:41:33.335037+0000","last_clean":"2024-02-13T00:41:33.335037+0000","last_became_active":"2024-02-13T00:37:47.256807+0000","last_became_peered":"2024-02-13T00:37:47.256807+0000","last_unstale":"2024-02-13T00:41:33.335037+0000","last_undegraded":"2024-02-13T00:41:33.335037+0000","last_fullsized":"2024-02-13T00:41:33.335037+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.402668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.402668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.1b","version":"108'10","reported_seq":52,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.426755+0000","last_change":"2024-02-13T00:41:01.676174+0000","last_active":"2024-02-13T00:41:33.426755+0000","last_peered":"2024-02-13T00:41:33.426755+0000","last_clean":"2024-02-13T00:41:33.426755+0000","last_became_active":"2024-02-13T00:41:01.675424+0000","last_became_peered":"2024-02-13T00:41:01.675424+0000","last_unstale":"2024-02-13T00:41:33.426755+0000","last_undegraded":"2024-02-13T00:41:33.426755+0000","last_fullsized":"2024-02-13T00:41:33.426755+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"107'1","reported_seq":43,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330604+0000","last_change":"2024-02-13T00:41:01.597218+0000","last_active":"2024-02-13T00:41:33.330604+0000","last_peered":"2024-02-13T00:41:33.330604+0000","last_clean":"2024-02-13T00:41:33.330604+0000","last_became_active":"2024-02-13T00:41:01.596590+0000","last_became_peered":"2024-02-13T00:41:01.596590+0000","last_unstale":"2024-02-13T00:41:33.330604+0000","last_undegraded":"2024-02-13T00:41:33.330604+0000","last_fullsized":"2024-02-13T00:41:33.330604+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.453693+0000","last_change":"2024-02-13T00:41:01.536591+0000","last_active":"2024-02-13T00:41:33.453693+0000","last_peered":"2024-02-13T00:41:33.453693+0000","last_clean":"2024-02-13T00:41:33.453693+0000","last_became_active":"2024-02-13T00:41:01.536188+0000","last_became_peered":"2024-02-13T00:41:01.536188+0000","last_unstale":"2024-02-13T00:41:33.453693+0000","last_undegraded":"2024-02-13T00:41:33.453693+0000","last_fullsized":"2024-02-13T00:41:33.453693+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":110,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.453617+0000","last_change":"2024-02-13T00:40:53.104803+0000","last_active":"2024-02-13T00:41:33.453617+0000","last_peered":"2024-02-13T00:41:33.453617+0000","last_clean":"2024-02-13T00:41:33.453617+0000","last_became_active":"2024-02-13T00:40:35.855827+0000","last_became_peered":"2024-02-13T00:40:35.855827+0000","last_unstale":"2024-02-13T00:41:33.453617+0000","last_undegraded":"2024-02-13T00:41:33.453617+0000","last_fullsized":"2024-02-13T00:41:33.453617+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:40:53.104724+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:40:53.104724+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":96,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.425247+0000","last_change":"2024-02-13T00:39:17.283996+0000","last_active":"2024-02-13T00:41:33.425247+0000","last_peered":"2024-02-13T00:41:33.425247+0000","last_clean":"2024-02-13T00:41:33.425247+0000","last_became_active":"2024-02-13T00:37:41.218346+0000","last_became_peered":"2024-02-13T00:37:41.218346+0000","last_unstale":"2024-02-13T00:41:33.425247+0000","last_undegraded":"2024-02-13T00:41:33.425247+0000","last_fullsized":"2024-02-13T00:41:33.425247+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.283902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.283902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"64'8","reported_seq":152,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.260466+0000","last_change":"2024-02-13T00:40:55.418661+0000","last_active":"2024-02-13T00:41:34.260466+0000","last_peered":"2024-02-13T00:41:34.260466+0000","last_clean":"2024-02-13T00:41:34.260466+0000","last_became_active":"2024-02-13T00:37:45.249684+0000","last_became_peered":"2024-02-13T00:37:45.249684+0000","last_unstale":"2024-02-13T00:41:34.260466+0000","last_undegraded":"2024-02-13T00:41:34.260466+0000","last_fullsized":"2024-02-13T00:41:34.260466+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:40:55.418534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:55.418534+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":69,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.440174+0000","last_change":"2024-02-13T00:40:40.750384+0000","last_active":"2024-02-13T00:41:33.440174+0000","last_peered":"2024-02-13T00:41:33.440174+0000","last_clean":"2024-02-13T00:41:33.440174+0000","last_became_active":"2024-02-13T00:40:35.851908+0000","last_became_peered":"2024-02-13T00:40:35.851908+0000","last_unstale":"2024-02-13T00:41:33.440174+0000","last_undegraded":"2024-02-13T00:41:33.440174+0000","last_fullsized":"2024-02-13T00:41:33.440174+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:40:40.750272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:40.750272+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1a","version":"108'5","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.450499+0000","last_change":"2024-02-13T00:41:01.743545+0000","last_active":"2024-02-13T00:41:33.450499+0000","last_peered":"2024-02-13T00:41:33.450499+0000","last_clean":"2024-02-13T00:41:33.450499+0000","last_became_active":"2024-02-13T00:41:01.739001+0000","last_became_peered":"2024-02-13T00:41:01.739001+0000","last_unstale":"2024-02-13T00:41:33.450499+0000","last_undegraded":"2024-02-13T00:41:33.450499+0000","last_fullsized":"2024-02-13T00:41:33.450499+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1c","version":"139'5","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.349626+0000","last_change":"2024-02-13T00:41:01.706984+0000","last_active":"2024-02-13T00:41:34.349626+0000","last_peered":"2024-02-13T00:41:34.349626+0000","last_clean":"2024-02-13T00:41:34.349626+0000","last_became_active":"2024-02-13T00:41:01.706080+0000","last_became_peered":"2024-02-13T00:41:01.706080+0000","last_unstale":"2024-02-13T00:41:34.349626+0000","last_undegraded":"2024-02-13T00:41:34.349626+0000","last_fullsized":"2024-02-13T00:41:34.349626+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2812,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.342188+0000","last_change":"2024-02-13T00:41:01.632475+0000","last_active":"2024-02-13T00:41:33.342188+0000","last_peered":"2024-02-13T00:41:33.342188+0000","last_clean":"2024-02-13T00:41:33.342188+0000","last_became_active":"2024-02-13T00:41:01.632199+0000","last_became_peered":"2024-02-13T00:41:01.632199+0000","last_unstale":"2024-02-13T00:41:33.342188+0000","last_undegraded":"2024-02-13T00:41:33.342188+0000","last_fullsized":"2024-02-13T00:41:33.342188+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":119,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.441491+0000","last_change":"2024-02-13T00:39:23.370512+0000","last_active":"2024-02-13T00:41:33.441491+0000","last_peered":"2024-02-13T00:41:33.441491+0000","last_clean":"2024-02-13T00:41:33.441491+0000","last_became_active":"2024-02-13T00:37:43.232161+0000","last_became_peered":"2024-02-13T00:37:43.232161+0000","last_unstale":"2024-02-13T00:41:33.441491+0000","last_undegraded":"2024-02-13T00:41:33.441491+0000","last_fullsized":"2024-02-13T00:41:33.441491+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:39:23.370447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.370447+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":107,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.414671+0000","last_change":"2024-02-13T00:39:52.206001+0000","last_active":"2024-02-13T00:41:33.414671+0000","last_peered":"2024-02-13T00:41:33.414671+0000","last_clean":"2024-02-13T00:41:33.414671+0000","last_became_active":"2024-02-13T00:37:41.217979+0000","last_became_peered":"2024-02-13T00:37:41.217979+0000","last_unstale":"2024-02-13T00:41:33.414671+0000","last_undegraded":"2024-02-13T00:41:33.414671+0000","last_fullsized":"2024-02-13T00:41:33.414671+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:39:52.205926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:52.205926+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"96'9","reported_seq":87,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.259644+0000","last_change":"2024-02-13T00:40:35.847666+0000","last_active":"2024-02-13T00:41:34.259644+0000","last_peered":"2024-02-13T00:41:34.259644+0000","last_clean":"2024-02-13T00:41:34.259644+0000","last_became_active":"2024-02-13T00:40:35.847293+0000","last_became_peered":"2024-02-13T00:40:35.847293+0000","last_unstale":"2024-02-13T00:41:34.259644+0000","last_undegraded":"2024-02-13T00:41:34.259644+0000","last_fullsized":"2024-02-13T00:41:34.259644+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:40:02.089864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:02.089864+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.327538+0000","last_change":"2024-02-13T00:40:58.541216+0000","last_active":"2024-02-13T00:41:33.327538+0000","last_peered":"2024-02-13T00:41:33.327538+0000","last_clean":"2024-02-13T00:41:33.327538+0000","last_became_active":"2024-02-13T00:37:47.260477+0000","last_became_peered":"2024-02-13T00:37:47.260477+0000","last_unstale":"2024-02-13T00:41:33.327538+0000","last_undegraded":"2024-02-13T00:41:33.327538+0000","last_fullsized":"2024-02-13T00:41:33.327538+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.541144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.331403+0000","last_change":"2024-02-13T00:41:01.596916+0000","last_active":"2024-02-13T00:41:33.331403+0000","last_peered":"2024-02-13T00:41:33.331403+0000","last_clean":"2024-02-13T00:41:33.331403+0000","last_became_active":"2024-02-13T00:41:01.594299+0000","last_became_peered":"2024-02-13T00:41:01.594299+0000","last_unstale":"2024-02-13T00:41:33.331403+0000","last_undegraded":"2024-02-13T00:41:33.331403+0000","last_fullsized":"2024-02-13T00:41:33.331403+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"107'5","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.342111+0000","last_change":"2024-02-13T00:41:01.634587+0000","last_active":"2024-02-13T00:41:33.342111+0000","last_peered":"2024-02-13T00:41:33.342111+0000","last_clean":"2024-02-13T00:41:33.342111+0000","last_became_active":"2024-02-13T00:41:01.634317+0000","last_became_peered":"2024-02-13T00:41:01.634317+0000","last_unstale":"2024-02-13T00:41:33.342111+0000","last_undegraded":"2024-02-13T00:41:33.342111+0000","last_fullsized":"2024-02-13T00:41:33.342111+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":64,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333748+0000","last_change":"2024-02-13T00:41:01.747617+0000","last_active":"2024-02-13T00:41:33.333748+0000","last_peered":"2024-02-13T00:41:33.333748+0000","last_clean":"2024-02-13T00:41:33.333748+0000","last_became_active":"2024-02-13T00:41:01.746476+0000","last_became_peered":"2024-02-13T00:41:01.746476+0000","last_unstale":"2024-02-13T00:41:33.333748+0000","last_undegraded":"2024-02-13T00:41:33.333748+0000","last_fullsized":"2024-02-13T00:41:33.333748+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330464+0000","last_change":"2024-02-13T00:40:35.848232+0000","last_active":"2024-02-13T00:41:33.330464+0000","last_peered":"2024-02-13T00:41:33.330464+0000","last_clean":"2024-02-13T00:41:33.330464+0000","last_became_active":"2024-02-13T00:40:35.847905+0000","last_became_peered":"2024-02-13T00:40:35.847905+0000","last_unstale":"2024-02-13T00:41:33.330464+0000","last_undegraded":"2024-02-13T00:41:33.330464+0000","last_fullsized":"2024-02-13T00:41:33.330464+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:29.275134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.275134+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330124+0000","last_change":"2024-02-13T00:41:32.399538+0000","last_active":"2024-02-13T00:41:33.330124+0000","last_peered":"2024-02-13T00:41:33.330124+0000","last_clean":"2024-02-13T00:41:33.330124+0000","last_became_active":"2024-02-13T00:37:41.221627+0000","last_became_peered":"2024-02-13T00:37:41.221627+0000","last_unstale":"2024-02-13T00:41:33.330124+0000","last_undegraded":"2024-02-13T00:41:33.330124+0000","last_fullsized":"2024-02-13T00:41:33.330124+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:32.399475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.399475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":96,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.332797+0000","last_change":"2024-02-13T00:40:48.518561+0000","last_active":"2024-02-13T00:41:33.332797+0000","last_peered":"2024-02-13T00:41:33.332797+0000","last_clean":"2024-02-13T00:41:33.332797+0000","last_became_active":"2024-02-13T00:37:45.245897+0000","last_became_peered":"2024-02-13T00:37:45.245897+0000","last_unstale":"2024-02-13T00:41:33.332797+0000","last_undegraded":"2024-02-13T00:41:33.332797+0000","last_fullsized":"2024-02-13T00:41:33.332797+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:48.518493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:40:48.518493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.453257+0000","last_change":"2024-02-13T00:40:14.015620+0000","last_active":"2024-02-13T00:41:33.453257+0000","last_peered":"2024-02-13T00:41:33.453257+0000","last_clean":"2024-02-13T00:41:33.453257+0000","last_became_active":"2024-02-13T00:37:47.552591+0000","last_became_peered":"2024-02-13T00:37:47.552591+0000","last_unstale":"2024-02-13T00:41:33.453257+0000","last_undegraded":"2024-02-13T00:41:33.453257+0000","last_fullsized":"2024-02-13T00:41:33.453257+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:14.015547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:14.015547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.18","version":"107'2","reported_seq":44,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.357693+0000","last_change":"2024-02-13T00:41:01.696324+0000","last_active":"2024-02-13T00:41:33.357693+0000","last_peered":"2024-02-13T00:41:33.357693+0000","last_clean":"2024-02-13T00:41:33.357693+0000","last_became_active":"2024-02-13T00:41:01.696158+0000","last_became_peered":"2024-02-13T00:41:01.696158+0000","last_unstale":"2024-02-13T00:41:33.357693+0000","last_undegraded":"2024-02-13T00:41:33.357693+0000","last_fullsized":"2024-02-13T00:41:33.357693+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1e","version":"109'4","reported_seq":49,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.534937+0000","last_change":"2024-02-13T00:41:01.660822+0000","last_active":"2024-02-13T00:41:34.534937+0000","last_peered":"2024-02-13T00:41:34.534937+0000","last_clean":"2024-02-13T00:41:34.534937+0000","last_became_active":"2024-02-13T00:41:01.660434+0000","last_became_peered":"2024-02-13T00:41:01.660434+0000","last_unstale":"2024-02-13T00:41:34.534937+0000","last_undegraded":"2024-02-13T00:41:34.534937+0000","last_fullsized":"2024-02-13T00:41:34.534937+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2808,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.323318+0000","last_change":"2024-02-13T00:41:01.583063+0000","last_active":"2024-02-13T00:41:33.323318+0000","last_peered":"2024-02-13T00:41:33.323318+0000","last_clean":"2024-02-13T00:41:33.323318+0000","last_became_active":"2024-02-13T00:41:01.582361+0000","last_became_peered":"2024-02-13T00:41:01.582361+0000","last_unstale":"2024-02-13T00:41:33.323318+0000","last_undegraded":"2024-02-13T00:41:33.323318+0000","last_fullsized":"2024-02-13T00:41:33.323318+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"64'8","reported_seq":148,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.260074+0000","last_change":"2024-02-13T00:39:16.304400+0000","last_active":"2024-02-13T00:41:34.260074+0000","last_peered":"2024-02-13T00:41:34.260074+0000","last_clean":"2024-02-13T00:41:34.260074+0000","last_became_active":"2024-02-13T00:37:45.245528+0000","last_became_peered":"2024-02-13T00:37:45.245528+0000","last_unstale":"2024-02-13T00:41:34.260074+0000","last_undegraded":"2024-02-13T00:41:34.260074+0000","last_fullsized":"2024-02-13T00:41:34.260074+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:16.304316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.304316+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":109,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.323232+0000","last_change":"2024-02-13T00:40:43.630615+0000","last_active":"2024-02-13T00:41:33.323232+0000","last_peered":"2024-02-13T00:41:33.323232+0000","last_clean":"2024-02-13T00:41:33.323232+0000","last_became_active":"2024-02-13T00:40:35.844184+0000","last_became_peered":"2024-02-13T00:40:35.844184+0000","last_unstale":"2024-02-13T00:41:33.323232+0000","last_undegraded":"2024-02-13T00:41:33.323232+0000","last_fullsized":"2024-02-13T00:41:33.323232+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2024-02-13T00:40:43.630535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:43.630535+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333503+0000","last_change":"2024-02-13T00:40:46.274497+0000","last_active":"2024-02-13T00:41:33.333503+0000","last_peered":"2024-02-13T00:41:33.333503+0000","last_clean":"2024-02-13T00:41:33.333503+0000","last_became_active":"2024-02-13T00:37:41.218313+0000","last_became_peered":"2024-02-13T00:37:41.218313+0000","last_unstale":"2024-02-13T00:41:33.333503+0000","last_undegraded":"2024-02-13T00:41:33.333503+0000","last_fullsized":"2024-02-13T00:41:33.333503+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:46.274423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:46.274423+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.344198+0000","last_change":"2024-02-13T00:40:44.243015+0000","last_active":"2024-02-13T00:41:33.344198+0000","last_peered":"2024-02-13T00:41:33.344198+0000","last_clean":"2024-02-13T00:41:33.344198+0000","last_became_active":"2024-02-13T00:37:47.553554+0000","last_became_peered":"2024-02-13T00:37:47.553554+0000","last_unstale":"2024-02-13T00:41:33.344198+0000","last_undegraded":"2024-02-13T00:41:33.344198+0000","last_fullsized":"2024-02-13T00:41:33.344198+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:44.242946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.242946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.17","version":"108'5","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333259+0000","last_change":"2024-02-13T00:41:01.623530+0000","last_active":"2024-02-13T00:41:33.333259+0000","last_peered":"2024-02-13T00:41:33.333259+0000","last_clean":"2024-02-13T00:41:33.333259+0000","last_became_active":"2024-02-13T00:41:01.622857+0000","last_became_peered":"2024-02-13T00:41:01.622857+0000","last_unstale":"2024-02-13T00:41:33.333259+0000","last_undegraded":"2024-02-13T00:41:33.333259+0000","last_fullsized":"2024-02-13T00:41:33.333259+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"107'6","reported_seq":48,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.429474+0000","last_change":"2024-02-13T00:41:01.734315+0000","last_active":"2024-02-13T00:41:33.429474+0000","last_peered":"2024-02-13T00:41:33.429474+0000","last_clean":"2024-02-13T00:41:33.429474+0000","last_became_active":"2024-02-13T00:41:01.728024+0000","last_became_peered":"2024-02-13T00:41:01.728024+0000","last_unstale":"2024-02-13T00:41:33.429474+0000","last_undegraded":"2024-02-13T00:41:33.429474+0000","last_fullsized":"2024-02-13T00:41:33.429474+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.444247+0000","last_change":"2024-02-13T00:41:01.682441+0000","last_active":"2024-02-13T00:41:33.444247+0000","last_peered":"2024-02-13T00:41:33.444247+0000","last_clean":"2024-02-13T00:41:33.444247+0000","last_became_active":"2024-02-13T00:41:01.682162+0000","last_became_peered":"2024-02-13T00:41:01.682162+0000","last_unstale":"2024-02-13T00:41:33.444247+0000","last_undegraded":"2024-02-13T00:41:33.444247+0000","last_fullsized":"2024-02-13T00:41:33.444247+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.446532+0000","last_change":"2024-02-13T00:40:43.011812+0000","last_active":"2024-02-13T00:41:33.446532+0000","last_peered":"2024-02-13T00:41:33.446532+0000","last_clean":"2024-02-13T00:41:33.446532+0000","last_became_active":"2024-02-13T00:40:35.856681+0000","last_became_peered":"2024-02-13T00:40:35.856681+0000","last_unstale":"2024-02-13T00:41:33.446532+0000","last_undegraded":"2024-02-13T00:41:33.446532+0000","last_fullsized":"2024-02-13T00:41:33.446532+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:43.011693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:43.011693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":160,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.444169+0000","last_change":"2024-02-13T00:40:47.034719+0000","last_active":"2024-02-13T00:41:33.444169+0000","last_peered":"2024-02-13T00:41:33.444169+0000","last_clean":"2024-02-13T00:41:33.444169+0000","last_became_active":"2024-02-13T00:37:43.324275+0000","last_became_peered":"2024-02-13T00:37:43.324275+0000","last_unstale":"2024-02-13T00:41:33.444169+0000","last_undegraded":"2024-02-13T00:41:33.444169+0000","last_fullsized":"2024-02-13T00:41:33.444169+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:40:47.034615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:47.034615+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":96,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.439547+0000","last_change":"2024-02-13T00:39:21.329248+0000","last_active":"2024-02-13T00:41:33.439547+0000","last_peered":"2024-02-13T00:41:33.439547+0000","last_clean":"2024-02-13T00:41:33.439547+0000","last_became_active":"2024-02-13T00:37:41.214202+0000","last_became_peered":"2024-02-13T00:37:41.214202+0000","last_unstale":"2024-02-13T00:41:33.439547+0000","last_undegraded":"2024-02-13T00:41:33.439547+0000","last_fullsized":"2024-02-13T00:41:33.439547+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.329173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.329173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.450393+0000","last_change":"2024-02-13T00:41:02.817359+0000","last_active":"2024-02-13T00:41:33.450393+0000","last_peered":"2024-02-13T00:41:33.450393+0000","last_clean":"2024-02-13T00:41:33.450393+0000","last_became_active":"2024-02-13T00:40:35.857936+0000","last_became_peered":"2024-02-13T00:40:35.857936+0000","last_unstale":"2024-02-13T00:41:33.450393+0000","last_undegraded":"2024-02-13T00:41:33.450393+0000","last_fullsized":"2024-02-13T00:41:33.450393+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.16","version":"108'5","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.459004+0000","last_change":"2024-02-13T00:41:01.722389+0000","last_active":"2024-02-13T00:41:33.459004+0000","last_peered":"2024-02-13T00:41:33.459004+0000","last_clean":"2024-02-13T00:41:33.459004+0000","last_became_active":"2024-02-13T00:41:01.721562+0000","last_became_peered":"2024-02-13T00:41:01.721562+0000","last_unstale":"2024-02-13T00:41:33.459004+0000","last_undegraded":"2024-02-13T00:41:33.459004+0000","last_fullsized":"2024-02-13T00:41:33.459004+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.10","version":"107'4","reported_seq":46,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.463381+0000","last_change":"2024-02-13T00:41:01.578798+0000","last_active":"2024-02-13T00:41:33.463381+0000","last_peered":"2024-02-13T00:41:33.463381+0000","last_clean":"2024-02-13T00:41:33.463381+0000","last_became_active":"2024-02-13T00:41:01.578398+0000","last_became_peered":"2024-02-13T00:41:01.578398+0000","last_unstale":"2024-02-13T00:41:33.463381+0000","last_undegraded":"2024-02-13T00:41:33.463381+0000","last_fullsized":"2024-02-13T00:41:33.463381+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.464008+0000","last_change":"2024-02-13T00:41:01.568508+0000","last_active":"2024-02-13T00:41:33.464008+0000","last_peered":"2024-02-13T00:41:33.464008+0000","last_clean":"2024-02-13T00:41:33.464008+0000","last_became_active":"2024-02-13T00:41:01.567871+0000","last_became_peered":"2024-02-13T00:41:01.567871+0000","last_unstale":"2024-02-13T00:41:33.464008+0000","last_undegraded":"2024-02-13T00:41:33.464008+0000","last_fullsized":"2024-02-13T00:41:33.464008+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":97,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.332638+0000","last_change":"2024-02-13T00:40:55.358468+0000","last_active":"2024-02-13T00:41:33.332638+0000","last_peered":"2024-02-13T00:41:33.332638+0000","last_clean":"2024-02-13T00:41:33.332638+0000","last_became_active":"2024-02-13T00:37:45.245974+0000","last_became_peered":"2024-02-13T00:37:45.245974+0000","last_unstale":"2024-02-13T00:41:33.332638+0000","last_undegraded":"2024-02-13T00:41:33.332638+0000","last_fullsized":"2024-02-13T00:41:33.332638+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:55.358367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:55.358367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":153,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.328909+0000","last_change":"2024-02-13T00:39:04.318990+0000","last_active":"2024-02-13T00:41:33.328909+0000","last_peered":"2024-02-13T00:41:33.328909+0000","last_clean":"2024-02-13T00:41:33.328909+0000","last_became_active":"2024-02-13T00:37:43.322450+0000","last_became_peered":"2024-02-13T00:37:43.322450+0000","last_unstale":"2024-02-13T00:41:33.328909+0000","last_undegraded":"2024-02-13T00:41:33.328909+0000","last_fullsized":"2024-02-13T00:41:33.328909+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_clean_scrub_stamp":"2024-02-13T00:39:04.318923+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.339698+0000","last_change":"2024-02-13T00:40:45.198328+0000","last_active":"2024-02-13T00:41:33.339698+0000","last_peered":"2024-02-13T00:41:33.339698+0000","last_clean":"2024-02-13T00:41:33.339698+0000","last_became_active":"2024-02-13T00:37:41.221636+0000","last_became_peered":"2024-02-13T00:37:41.221636+0000","last_unstale":"2024-02-13T00:41:33.339698+0000","last_undegraded":"2024-02-13T00:41:33.339698+0000","last_fullsized":"2024-02-13T00:41:33.339698+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:45.198231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:45.198231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":90,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.414752+0000","last_change":"2024-02-13T00:39:33.304107+0000","last_active":"2024-02-13T00:41:33.414752+0000","last_peered":"2024-02-13T00:41:33.414752+0000","last_clean":"2024-02-13T00:41:33.414752+0000","last_became_active":"2024-02-13T00:37:47.552672+0000","last_became_peered":"2024-02-13T00:37:47.552672+0000","last_unstale":"2024-02-13T00:41:33.414752+0000","last_undegraded":"2024-02-13T00:41:33.414752+0000","last_fullsized":"2024-02-13T00:41:33.414752+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.303982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.303982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.327846+0000","last_change":"2024-02-13T00:41:01.672342+0000","last_active":"2024-02-13T00:41:33.327846+0000","last_peered":"2024-02-13T00:41:33.327846+0000","last_clean":"2024-02-13T00:41:33.327846+0000","last_became_active":"2024-02-13T00:41:01.671979+0000","last_became_peered":"2024-02-13T00:41:01.671979+0000","last_unstale":"2024-02-13T00:41:33.327846+0000","last_undegraded":"2024-02-13T00:41:33.327846+0000","last_fullsized":"2024-02-13T00:41:33.327846+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.331856+0000","last_change":"2024-02-13T00:41:01.578655+0000","last_active":"2024-02-13T00:41:33.331856+0000","last_peered":"2024-02-13T00:41:33.331856+0000","last_clean":"2024-02-13T00:41:33.331856+0000","last_became_active":"2024-02-13T00:41:01.578380+0000","last_became_peered":"2024-02-13T00:41:01.578380+0000","last_unstale":"2024-02-13T00:41:33.331856+0000","last_undegraded":"2024-02-13T00:41:33.331856+0000","last_fullsized":"2024-02-13T00:41:33.331856+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.458601+0000","last_change":"2024-02-13T00:41:01.590710+0000","last_active":"2024-02-13T00:41:33.458601+0000","last_peered":"2024-02-13T00:41:33.458601+0000","last_clean":"2024-02-13T00:41:33.458601+0000","last_became_active":"2024-02-13T00:41:01.590438+0000","last_became_peered":"2024-02-13T00:41:01.590438+0000","last_unstale":"2024-02-13T00:41:33.458601+0000","last_undegraded":"2024-02-13T00:41:33.458601+0000","last_fullsized":"2024-02-13T00:41:33.458601+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":96,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333067+0000","last_change":"2024-02-13T00:40:03.701680+0000","last_active":"2024-02-13T00:41:33.333067+0000","last_peered":"2024-02-13T00:41:33.333067+0000","last_clean":"2024-02-13T00:41:33.333067+0000","last_became_active":"2024-02-13T00:37:45.245956+0000","last_became_peered":"2024-02-13T00:37:45.245956+0000","last_unstale":"2024-02-13T00:41:33.333067+0000","last_undegraded":"2024-02-13T00:41:33.333067+0000","last_fullsized":"2024-02-13T00:41:33.333067+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:03.701604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:03.701604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":134,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333386+0000","last_change":"2024-02-13T00:39:08.344202+0000","last_active":"2024-02-13T00:41:33.333386+0000","last_peered":"2024-02-13T00:41:33.333386+0000","last_clean":"2024-02-13T00:41:33.333386+0000","last_became_active":"2024-02-13T00:39:05.968155+0000","last_became_peered":"2024-02-13T00:39:05.968155+0000","last_unstale":"2024-02-13T00:41:33.333386+0000","last_undegraded":"2024-02-13T00:41:33.333386+0000","last_fullsized":"2024-02-13T00:41:33.333386+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:39:08.344129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:08.344129+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":72,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.463273+0000","last_change":"2024-02-13T00:40:42.413113+0000","last_active":"2024-02-13T00:41:33.463273+0000","last_peered":"2024-02-13T00:41:33.463273+0000","last_clean":"2024-02-13T00:41:33.463273+0000","last_became_active":"2024-02-13T00:39:05.967772+0000","last_became_peered":"2024-02-13T00:39:05.967772+0000","last_unstale":"2024-02-13T00:41:33.463273+0000","last_undegraded":"2024-02-13T00:41:33.463273+0000","last_fullsized":"2024-02-13T00:41:33.463273+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:42.413043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:42.413043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.352156+0000","last_change":"2024-02-13T00:41:34.352119+0000","last_active":"2024-02-13T00:41:34.352156+0000","last_peered":"2024-02-13T00:41:34.352156+0000","last_clean":"2024-02-13T00:41:34.352156+0000","last_became_active":"2024-02-13T00:37:41.217577+0000","last_became_peered":"2024-02-13T00:37:41.217577+0000","last_unstale":"2024-02-13T00:41:34.352156+0000","last_undegraded":"2024-02-13T00:41:34.352156+0000","last_fullsized":"2024-02-13T00:41:34.352156+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.352030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.352030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":90,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.428444+0000","last_change":"2024-02-13T00:39:32.255327+0000","last_active":"2024-02-13T00:41:33.428444+0000","last_peered":"2024-02-13T00:41:33.428444+0000","last_clean":"2024-02-13T00:41:33.428444+0000","last_became_active":"2024-02-13T00:37:47.262024+0000","last_became_peered":"2024-02-13T00:37:47.262024+0000","last_unstale":"2024-02-13T00:41:33.428444+0000","last_undegraded":"2024-02-13T00:41:33.428444+0000","last_fullsized":"2024-02-13T00:41:33.428444+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.255260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.320017+0000","last_change":"2024-02-13T00:41:01.662195+0000","last_active":"2024-02-13T00:41:33.320017+0000","last_peered":"2024-02-13T00:41:33.320017+0000","last_clean":"2024-02-13T00:41:33.320017+0000","last_became_active":"2024-02-13T00:41:01.660270+0000","last_became_peered":"2024-02-13T00:41:01.660270+0000","last_unstale":"2024-02-13T00:41:33.320017+0000","last_undegraded":"2024-02-13T00:41:33.320017+0000","last_fullsized":"2024-02-13T00:41:33.320017+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.16","version":"107'1","reported_seq":44,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.343992+0000","last_change":"2024-02-13T00:41:01.538909+0000","last_active":"2024-02-13T00:41:33.343992+0000","last_peered":"2024-02-13T00:41:33.343992+0000","last_clean":"2024-02-13T00:41:33.343992+0000","last_became_active":"2024-02-13T00:41:01.538309+0000","last_became_peered":"2024-02-13T00:41:01.538309+0000","last_unstale":"2024-02-13T00:41:33.343992+0000","last_undegraded":"2024-02-13T00:41:33.343992+0000","last_fullsized":"2024-02-13T00:41:33.343992+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.327861+0000","last_change":"2024-02-13T00:41:01.620285+0000","last_active":"2024-02-13T00:41:33.327861+0000","last_peered":"2024-02-13T00:41:33.327861+0000","last_clean":"2024-02-13T00:41:33.327861+0000","last_became_active":"2024-02-13T00:41:01.619449+0000","last_became_peered":"2024-02-13T00:41:01.619449+0000","last_unstale":"2024-02-13T00:41:33.327861+0000","last_undegraded":"2024-02-13T00:41:33.327861+0000","last_fullsized":"2024-02-13T00:41:33.327861+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":91,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.337044+0000","last_change":"2024-02-13T00:39:28.171808+0000","last_active":"2024-02-13T00:41:33.337044+0000","last_peered":"2024-02-13T00:41:33.337044+0000","last_clean":"2024-02-13T00:41:33.337044+0000","last_became_active":"2024-02-13T00:37:45.247708+0000","last_became_peered":"2024-02-13T00:37:45.247708+0000","last_unstale":"2024-02-13T00:41:33.337044+0000","last_undegraded":"2024-02-13T00:41:33.337044+0000","last_fullsized":"2024-02-13T00:41:33.337044+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.171740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.171740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"64'14","reported_seq":116,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330563+0000","last_change":"2024-02-13T00:39:15.415087+0000","last_active":"2024-02-13T00:41:33.330563+0000","last_peered":"2024-02-13T00:41:33.330563+0000","last_clean":"2024-02-13T00:41:33.330563+0000","last_became_active":"2024-02-13T00:37:43.324209+0000","last_became_peered":"2024-02-13T00:37:43.324209+0000","last_unstale":"2024-02-13T00:41:33.330563+0000","last_undegraded":"2024-02-13T00:41:33.330563+0000","last_fullsized":"2024-02-13T00:41:33.330563+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'14","last_scrub_stamp":"2024-02-13T00:39:15.415004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.415004+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":103,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.329989+0000","last_change":"2024-02-13T00:39:41.255860+0000","last_active":"2024-02-13T00:41:33.329989+0000","last_peered":"2024-02-13T00:41:33.329989+0000","last_clean":"2024-02-13T00:41:33.329989+0000","last_became_active":"2024-02-13T00:37:35.635369+0000","last_became_peered":"2024-02-13T00:37:35.635369+0000","last_unstale":"2024-02-13T00:41:33.329989+0000","last_undegraded":"2024-02-13T00:41:33.329989+0000","last_fullsized":"2024-02-13T00:41:33.329989+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:39:41.255785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":102,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.472385+0000","last_change":"2024-02-13T00:41:34.472385+0000","last_active":"2024-02-13T00:41:34.472385+0000","last_peered":"2024-02-13T00:41:34.472385+0000","last_clean":"2024-02-13T00:41:34.472385+0000","last_became_active":"2024-02-13T00:37:41.216101+0000","last_became_peered":"2024-02-13T00:37:41.216101+0000","last_unstale":"2024-02-13T00:41:34.472385+0000","last_undegraded":"2024-02-13T00:41:34.472385+0000","last_fullsized":"2024-02-13T00:41:34.472385+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.472312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.472312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.450432+0000","last_change":"2024-02-13T00:40:09.099541+0000","last_active":"2024-02-13T00:41:33.450432+0000","last_peered":"2024-02-13T00:41:33.450432+0000","last_clean":"2024-02-13T00:41:33.450432+0000","last_became_active":"2024-02-13T00:37:41.216367+0000","last_became_peered":"2024-02-13T00:37:41.216367+0000","last_unstale":"2024-02-13T00:41:33.450432+0000","last_undegraded":"2024-02-13T00:41:33.450432+0000","last_fullsized":"2024-02-13T00:41:33.450432+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:09.099460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:40:09.099460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333135+0000","last_change":"2024-02-13T00:40:08.577747+0000","last_active":"2024-02-13T00:41:33.333135+0000","last_peered":"2024-02-13T00:41:33.333135+0000","last_clean":"2024-02-13T00:41:33.333135+0000","last_became_active":"2024-02-13T00:37:47.253996+0000","last_became_peered":"2024-02-13T00:37:47.253996+0000","last_unstale":"2024-02-13T00:41:33.333135+0000","last_undegraded":"2024-02-13T00:41:33.333135+0000","last_fullsized":"2024-02-13T00:41:33.333135+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:08.577665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:40:08.577665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.13","version":"108'9","reported_seq":51,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.340818+0000","last_change":"2024-02-13T00:41:01.560685+0000","last_active":"2024-02-13T00:41:33.340818+0000","last_peered":"2024-02-13T00:41:33.340818+0000","last_clean":"2024-02-13T00:41:33.340818+0000","last_became_active":"2024-02-13T00:41:01.560473+0000","last_became_peered":"2024-02-13T00:41:01.560473+0000","last_unstale":"2024-02-13T00:41:33.340818+0000","last_undegraded":"2024-02-13T00:41:33.340818+0000","last_fullsized":"2024-02-13T00:41:33.340818+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.15","version":"107'1","reported_seq":43,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330681+0000","last_change":"2024-02-13T00:41:01.656670+0000","last_active":"2024-02-13T00:41:33.330681+0000","last_peered":"2024-02-13T00:41:33.330681+0000","last_clean":"2024-02-13T00:41:33.330681+0000","last_became_active":"2024-02-13T00:41:01.655358+0000","last_became_peered":"2024-02-13T00:41:01.655358+0000","last_unstale":"2024-02-13T00:41:33.330681+0000","last_undegraded":"2024-02-13T00:41:33.330681+0000","last_fullsized":"2024-02-13T00:41:33.330681+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330254+0000","last_change":"2024-02-13T00:41:01.594433+0000","last_active":"2024-02-13T00:41:33.330254+0000","last_peered":"2024-02-13T00:41:33.330254+0000","last_clean":"2024-02-13T00:41:33.330254+0000","last_became_active":"2024-02-13T00:41:01.593970+0000","last_became_peered":"2024-02-13T00:41:01.593970+0000","last_unstale":"2024-02-13T00:41:33.330254+0000","last_undegraded":"2024-02-13T00:41:33.330254+0000","last_fullsized":"2024-02-13T00:41:33.330254+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":97,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.458705+0000","last_change":"2024-02-13T00:41:01.412120+0000","last_active":"2024-02-13T00:41:33.458705+0000","last_peered":"2024-02-13T00:41:33.458705+0000","last_clean":"2024-02-13T00:41:33.458705+0000","last_became_active":"2024-02-13T00:37:45.248383+0000","last_became_peered":"2024-02-13T00:37:45.248383+0000","last_unstale":"2024-02-13T00:41:33.458705+0000","last_undegraded":"2024-02-13T00:41:33.458705+0000","last_fullsized":"2024-02-13T00:41:33.458705+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:01.412042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:01.412042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":136,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.441735+0000","last_change":"2024-02-13T00:39:25.319173+0000","last_active":"2024-02-13T00:41:33.441735+0000","last_peered":"2024-02-13T00:41:33.441735+0000","last_clean":"2024-02-13T00:41:33.441735+0000","last_became_active":"2024-02-13T00:37:43.231120+0000","last_became_peered":"2024-02-13T00:37:43.231120+0000","last_unstale":"2024-02-13T00:41:33.441735+0000","last_undegraded":"2024-02-13T00:41:33.441735+0000","last_fullsized":"2024-02-13T00:41:33.441735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:39:25.319066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.319066+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.419938+0000","last_change":"2024-02-13T00:40:57.469853+0000","last_active":"2024-02-13T00:41:33.419938+0000","last_peered":"2024-02-13T00:41:33.419938+0000","last_clean":"2024-02-13T00:41:33.419938+0000","last_became_active":"2024-02-13T00:37:41.214442+0000","last_became_peered":"2024-02-13T00:37:41.214442+0000","last_unstale":"2024-02-13T00:41:33.419938+0000","last_undegraded":"2024-02-13T00:41:33.419938+0000","last_fullsized":"2024-02-13T00:41:33.419938+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.469743+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":328,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.330768+0000","last_change":"2024-02-13T00:39:10.287406+0000","last_active":"2024-02-13T00:41:34.330768+0000","last_peered":"2024-02-13T00:41:34.330768+0000","last_clean":"2024-02-13T00:41:34.330768+0000","last_became_active":"2024-02-13T00:37:41.215137+0000","last_became_peered":"2024-02-13T00:37:41.215137+0000","last_unstale":"2024-02-13T00:41:34.330768+0000","last_undegraded":"2024-02-13T00:41:34.330768+0000","last_fullsized":"2024-02-13T00:41:34.330768+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-13T00:39:10.287299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.287299+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":227,"num_read_kb":222,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":96,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.339625+0000","last_change":"2024-02-13T00:40:51.198184+0000","last_active":"2024-02-13T00:41:33.339625+0000","last_peered":"2024-02-13T00:41:33.339625+0000","last_clean":"2024-02-13T00:41:33.339625+0000","last_became_active":"2024-02-13T00:37:47.552693+0000","last_became_peered":"2024-02-13T00:37:47.552693+0000","last_unstale":"2024-02-13T00:41:33.339625+0000","last_undegraded":"2024-02-13T00:41:33.339625+0000","last_fullsized":"2024-02-13T00:41:33.339625+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.11","version":"108'7","reported_seq":49,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.429396+0000","last_change":"2024-02-13T00:41:01.721381+0000","last_active":"2024-02-13T00:41:33.429396+0000","last_peered":"2024-02-13T00:41:33.429396+0000","last_clean":"2024-02-13T00:41:33.429396+0000","last_became_active":"2024-02-13T00:41:01.718821+0000","last_became_peered":"2024-02-13T00:41:01.718821+0000","last_unstale":"2024-02-13T00:41:33.429396+0000","last_undegraded":"2024-02-13T00:41:33.429396+0000","last_fullsized":"2024-02-13T00:41:33.429396+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.17","version":"109'1","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.457405+0000","last_change":"2024-02-13T00:41:01.738284+0000","last_active":"2024-02-13T00:41:33.457405+0000","last_peered":"2024-02-13T00:41:33.457405+0000","last_clean":"2024-02-13T00:41:33.457405+0000","last_became_active":"2024-02-13T00:41:01.738000+0000","last_became_peered":"2024-02-13T00:41:01.738000+0000","last_unstale":"2024-02-13T00:41:33.457405+0000","last_undegraded":"2024-02-13T00:41:33.457405+0000","last_fullsized":"2024-02-13T00:41:33.457405+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.454019+0000","last_change":"2024-02-13T00:41:01.548873+0000","last_active":"2024-02-13T00:41:33.454019+0000","last_peered":"2024-02-13T00:41:33.454019+0000","last_clean":"2024-02-13T00:41:33.454019+0000","last_became_active":"2024-02-13T00:41:01.547408+0000","last_became_peered":"2024-02-13T00:41:01.547408+0000","last_unstale":"2024-02-13T00:41:33.454019+0000","last_undegraded":"2024-02-13T00:41:33.454019+0000","last_fullsized":"2024-02-13T00:41:33.454019+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":96,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330375+0000","last_change":"2024-02-13T00:41:00.881481+0000","last_active":"2024-02-13T00:41:33.330375+0000","last_peered":"2024-02-13T00:41:33.330375+0000","last_clean":"2024-02-13T00:41:33.330375+0000","last_became_active":"2024-02-13T00:37:45.247012+0000","last_became_peered":"2024-02-13T00:37:45.247012+0000","last_unstale":"2024-02-13T00:41:33.330375+0000","last_undegraded":"2024-02-13T00:41:33.330375+0000","last_fullsized":"2024-02-13T00:41:33.330375+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.881411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.881411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"80'45","reported_seq":220,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.279182+0000","last_change":"2024-02-13T00:40:12.047980+0000","last_active":"2024-02-13T00:41:34.279182+0000","last_peered":"2024-02-13T00:41:34.279182+0000","last_clean":"2024-02-13T00:41:34.279182+0000","last_became_active":"2024-02-13T00:37:43.232014+0000","last_became_peered":"2024-02-13T00:37:43.232014+0000","last_unstale":"2024-02-13T00:41:34.279182+0000","last_undegraded":"2024-02-13T00:41:34.279182+0000","last_fullsized":"2024-02-13T00:41:34.279182+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'45","last_scrub_stamp":"2024-02-13T00:40:12.047869+0000","last_deep_scrub":"80'45","last_deep_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_clean_scrub_stamp":"2024-02-13T00:40:12.047869+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.316969+0000","last_change":"2024-02-13T00:40:59.518865+0000","last_active":"2024-02-13T00:41:33.316969+0000","last_peered":"2024-02-13T00:41:33.316969+0000","last_clean":"2024-02-13T00:41:33.316969+0000","last_became_active":"2024-02-13T00:37:41.212924+0000","last_became_peered":"2024-02-13T00:37:41.212924+0000","last_unstale":"2024-02-13T00:41:33.316969+0000","last_undegraded":"2024-02-13T00:41:33.316969+0000","last_fullsized":"2024-02-13T00:41:33.316969+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.518778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.518778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.458896+0000","last_change":"2024-02-13T00:39:36.157777+0000","last_active":"2024-02-13T00:41:33.458896+0000","last_peered":"2024-02-13T00:41:33.458896+0000","last_clean":"2024-02-13T00:41:33.458896+0000","last_became_active":"2024-02-13T00:37:47.252270+0000","last_became_peered":"2024-02-13T00:37:47.252270+0000","last_unstale":"2024-02-13T00:41:33.458896+0000","last_undegraded":"2024-02-13T00:41:33.458896+0000","last_fullsized":"2024-02-13T00:41:33.458896+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:36.157693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:36.157693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.323536+0000","last_change":"2024-02-13T00:41:01.606464+0000","last_active":"2024-02-13T00:41:33.323536+0000","last_peered":"2024-02-13T00:41:33.323536+0000","last_clean":"2024-02-13T00:41:33.323536+0000","last_became_active":"2024-02-13T00:41:01.605731+0000","last_became_peered":"2024-02-13T00:41:01.605731+0000","last_unstale":"2024-02-13T00:41:33.323536+0000","last_undegraded":"2024-02-13T00:41:33.323536+0000","last_fullsized":"2024-02-13T00:41:33.323536+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"109'1001","reported_seq":2043,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.320317+0000","last_change":"2024-02-13T00:41:01.666080+0000","last_active":"2024-02-13T00:41:33.320317+0000","last_peered":"2024-02-13T00:41:33.320317+0000","last_clean":"2024-02-13T00:41:33.320317+0000","last_became_active":"2024-02-13T00:41:01.665589+0000","last_became_peered":"2024-02-13T00:41:01.665589+0000","last_unstale":"2024-02-13T00:41:33.320317+0000","last_undegraded":"2024-02-13T00:41:33.320317+0000","last_fullsized":"2024-02-13T00:41:33.320317+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1001,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1000,"num_read_kb":0,"num_write":1000,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333173+0000","last_change":"2024-02-13T00:41:01.644491+0000","last_active":"2024-02-13T00:41:33.333173+0000","last_peered":"2024-02-13T00:41:33.333173+0000","last_clean":"2024-02-13T00:41:33.333173+0000","last_became_active":"2024-02-13T00:41:01.643774+0000","last_became_peered":"2024-02-13T00:41:01.643774+0000","last_unstale":"2024-02-13T00:41:33.333173+0000","last_undegraded":"2024-02-13T00:41:33.333173+0000","last_fullsized":"2024-02-13T00:41:33.333173+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"64'8","reported_seq":146,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.278683+0000","last_change":"2024-02-13T00:39:30.273237+0000","last_active":"2024-02-13T00:41:34.278683+0000","last_peered":"2024-02-13T00:41:34.278683+0000","last_clean":"2024-02-13T00:41:34.278683+0000","last_became_active":"2024-02-13T00:37:45.241274+0000","last_became_peered":"2024-02-13T00:37:45.241274+0000","last_unstale":"2024-02-13T00:41:34.278683+0000","last_undegraded":"2024-02-13T00:41:34.278683+0000","last_fullsized":"2024-02-13T00:41:34.278683+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:30.273165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.273165+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":86,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.323138+0000","last_change":"2024-02-13T00:40:16.354241+0000","last_active":"2024-02-13T00:41:33.323138+0000","last_peered":"2024-02-13T00:41:33.323138+0000","last_clean":"2024-02-13T00:41:33.323138+0000","last_became_active":"2024-02-13T00:39:05.965560+0000","last_became_peered":"2024-02-13T00:39:05.965560+0000","last_unstale":"2024-02-13T00:41:33.323138+0000","last_undegraded":"2024-02-13T00:41:33.323138+0000","last_fullsized":"2024-02-13T00:41:33.323138+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:40:16.354164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:16.354164+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.428062+0000","last_change":"2024-02-13T00:40:08.946462+0000","last_active":"2024-02-13T00:41:33.428062+0000","last_peered":"2024-02-13T00:41:33.428062+0000","last_clean":"2024-02-13T00:41:33.428062+0000","last_became_active":"2024-02-13T00:37:41.217777+0000","last_became_peered":"2024-02-13T00:37:41.217777+0000","last_unstale":"2024-02-13T00:41:33.428062+0000","last_undegraded":"2024-02-13T00:41:33.428062+0000","last_fullsized":"2024-02-13T00:41:33.428062+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:08.946388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:40:08.946388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.336208+0000","last_change":"2024-02-13T00:38:54.349240+0000","last_active":"2024-02-13T00:41:33.336208+0000","last_peered":"2024-02-13T00:41:33.336208+0000","last_clean":"2024-02-13T00:41:33.336208+0000","last_became_active":"2024-02-13T00:37:47.255928+0000","last_became_peered":"2024-02-13T00:37:47.255928+0000","last_unstale":"2024-02-13T00:41:33.336208+0000","last_undegraded":"2024-02-13T00:41:33.336208+0000","last_fullsized":"2024-02-13T00:41:33.336208+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:38:54.349171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:38:54.349171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"107'2","reported_seq":44,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.329369+0000","last_change":"2024-02-13T00:41:01.616697+0000","last_active":"2024-02-13T00:41:33.329369+0000","last_peered":"2024-02-13T00:41:33.329369+0000","last_clean":"2024-02-13T00:41:33.329369+0000","last_became_active":"2024-02-13T00:41:01.616419+0000","last_became_peered":"2024-02-13T00:41:01.616419+0000","last_unstale":"2024-02-13T00:41:33.329369+0000","last_undegraded":"2024-02-13T00:41:33.329369+0000","last_fullsized":"2024-02-13T00:41:33.329369+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333097+0000","last_change":"2024-02-13T00:41:01.646177+0000","last_active":"2024-02-13T00:41:33.333097+0000","last_peered":"2024-02-13T00:41:33.333097+0000","last_clean":"2024-02-13T00:41:33.333097+0000","last_became_active":"2024-02-13T00:41:01.644151+0000","last_became_peered":"2024-02-13T00:41:01.644151+0000","last_unstale":"2024-02-13T00:41:33.333097+0000","last_undegraded":"2024-02-13T00:41:33.333097+0000","last_fullsized":"2024-02-13T00:41:33.333097+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.414529+0000","last_change":"2024-02-13T00:41:01.721182+0000","last_active":"2024-02-13T00:41:33.414529+0000","last_peered":"2024-02-13T00:41:33.414529+0000","last_clean":"2024-02-13T00:41:33.414529+0000","last_became_active":"2024-02-13T00:41:01.718793+0000","last_became_peered":"2024-02-13T00:41:01.718793+0000","last_unstale":"2024-02-13T00:41:33.414529+0000","last_undegraded":"2024-02-13T00:41:33.414529+0000","last_fullsized":"2024-02-13T00:41:33.414529+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":92,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.459692+0000","last_change":"2024-02-13T00:39:21.654834+0000","last_active":"2024-02-13T00:41:33.459692+0000","last_peered":"2024-02-13T00:41:33.459692+0000","last_clean":"2024-02-13T00:41:33.459692+0000","last_became_active":"2024-02-13T00:37:45.244408+0000","last_became_peered":"2024-02-13T00:37:45.244408+0000","last_unstale":"2024-02-13T00:41:33.459692+0000","last_undegraded":"2024-02-13T00:41:33.459692+0000","last_fullsized":"2024-02-13T00:41:33.459692+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.654733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.654733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":121,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.414455+0000","last_change":"2024-02-13T00:40:35.856095+0000","last_active":"2024-02-13T00:41:33.414455+0000","last_peered":"2024-02-13T00:41:33.414455+0000","last_clean":"2024-02-13T00:41:33.414455+0000","last_became_active":"2024-02-13T00:40:35.855722+0000","last_became_peered":"2024-02-13T00:40:35.855722+0000","last_unstale":"2024-02-13T00:41:33.414455+0000","last_undegraded":"2024-02-13T00:41:33.414455+0000","last_fullsized":"2024-02-13T00:41:33.414455+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'24","last_scrub_stamp":"2024-02-13T00:39:56.121046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:56.121046+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":36,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":69,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.322207+0000","last_change":"2024-02-13T00:40:48.809660+0000","last_active":"2024-02-13T00:41:33.322207+0000","last_peered":"2024-02-13T00:41:33.322207+0000","last_clean":"2024-02-13T00:41:33.322207+0000","last_became_active":"2024-02-13T00:40:35.852351+0000","last_became_peered":"2024-02-13T00:40:35.852351+0000","last_unstale":"2024-02-13T00:41:33.322207+0000","last_undegraded":"2024-02-13T00:41:33.322207+0000","last_fullsized":"2024-02-13T00:41:33.322207+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:40:48.809594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:48.809594+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.451170+0000","last_change":"2024-02-13T00:39:33.179309+0000","last_active":"2024-02-13T00:41:33.451170+0000","last_peered":"2024-02-13T00:41:33.451170+0000","last_clean":"2024-02-13T00:41:33.451170+0000","last_became_active":"2024-02-13T00:37:47.260242+0000","last_became_peered":"2024-02-13T00:37:47.260242+0000","last_unstale":"2024-02-13T00:41:33.451170+0000","last_undegraded":"2024-02-13T00:41:33.451170+0000","last_fullsized":"2024-02-13T00:41:33.451170+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.179225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.179225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"108'5","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.341852+0000","last_change":"2024-02-13T00:41:01.638321+0000","last_active":"2024-02-13T00:41:33.341852+0000","last_peered":"2024-02-13T00:41:33.341852+0000","last_clean":"2024-02-13T00:41:33.341852+0000","last_became_active":"2024-02-13T00:41:01.637897+0000","last_became_peered":"2024-02-13T00:41:01.637897+0000","last_unstale":"2024-02-13T00:41:33.341852+0000","last_undegraded":"2024-02-13T00:41:33.341852+0000","last_fullsized":"2024-02-13T00:41:33.341852+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.19","version":"109'1","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.420097+0000","last_change":"2024-02-13T00:41:01.593406+0000","last_active":"2024-02-13T00:41:33.420097+0000","last_peered":"2024-02-13T00:41:33.420097+0000","last_clean":"2024-02-13T00:41:33.420097+0000","last_became_active":"2024-02-13T00:41:01.593112+0000","last_became_peered":"2024-02-13T00:41:01.593112+0000","last_unstale":"2024-02-13T00:41:33.420097+0000","last_undegraded":"2024-02-13T00:41:33.420097+0000","last_fullsized":"2024-02-13T00:41:33.420097+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333490+0000","last_change":"2024-02-13T00:41:01.594534+0000","last_active":"2024-02-13T00:41:33.333490+0000","last_peered":"2024-02-13T00:41:33.333490+0000","last_clean":"2024-02-13T00:41:33.333490+0000","last_became_active":"2024-02-13T00:41:01.590676+0000","last_became_peered":"2024-02-13T00:41:01.590676+0000","last_unstale":"2024-02-13T00:41:33.333490+0000","last_undegraded":"2024-02-13T00:41:33.333490+0000","last_fullsized":"2024-02-13T00:41:33.333490+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":144,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333448+0000","last_change":"2024-02-13T00:40:49.767975+0000","last_active":"2024-02-13T00:41:33.333448+0000","last_peered":"2024-02-13T00:41:33.333448+0000","last_clean":"2024-02-13T00:41:33.333448+0000","last_became_active":"2024-02-13T00:37:43.322401+0000","last_became_peered":"2024-02-13T00:37:43.322401+0000","last_unstale":"2024-02-13T00:41:33.333448+0000","last_undegraded":"2024-02-13T00:41:33.333448+0000","last_fullsized":"2024-02-13T00:41:33.333448+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:40:49.767853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:49.767853+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.459279+0000","last_change":"2024-02-13T00:40:07.270252+0000","last_active":"2024-02-13T00:41:33.459279+0000","last_peered":"2024-02-13T00:41:33.459279+0000","last_clean":"2024-02-13T00:41:33.459279+0000","last_became_active":"2024-02-13T00:37:41.214091+0000","last_became_peered":"2024-02-13T00:37:41.214091+0000","last_unstale":"2024-02-13T00:41:33.459279+0000","last_undegraded":"2024-02-13T00:41:33.459279+0000","last_fullsized":"2024-02-13T00:41:33.459279+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:07.270157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:07.270157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":91,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.457221+0000","last_change":"2024-02-13T00:39:46.155792+0000","last_active":"2024-02-13T00:41:33.457221+0000","last_peered":"2024-02-13T00:41:33.457221+0000","last_clean":"2024-02-13T00:41:33.457221+0000","last_became_active":"2024-02-13T00:37:45.250632+0000","last_became_peered":"2024-02-13T00:37:45.250632+0000","last_unstale":"2024-02-13T00:41:33.457221+0000","last_undegraded":"2024-02-13T00:41:33.457221+0000","last_fullsized":"2024-02-13T00:41:33.457221+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:46.155718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:46.155718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":95,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.325249+0000","last_change":"2024-02-13T00:40:59.468608+0000","last_active":"2024-02-13T00:41:33.325249+0000","last_peered":"2024-02-13T00:41:33.325249+0000","last_clean":"2024-02-13T00:41:33.325249+0000","last_became_active":"2024-02-13T00:37:47.254803+0000","last_became_peered":"2024-02-13T00:37:47.254803+0000","last_unstale":"2024-02-13T00:41:33.325249+0000","last_undegraded":"2024-02-13T00:41:33.325249+0000","last_fullsized":"2024-02-13T00:41:33.325249+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.468525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.468525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.0","version":"108'5","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.336399+0000","last_change":"2024-02-13T00:41:01.580498+0000","last_active":"2024-02-13T00:41:33.336399+0000","last_peered":"2024-02-13T00:41:33.336399+0000","last_clean":"2024-02-13T00:41:33.336399+0000","last_became_active":"2024-02-13T00:41:01.579665+0000","last_became_peered":"2024-02-13T00:41:01.579665+0000","last_unstale":"2024-02-13T00:41:33.336399+0000","last_undegraded":"2024-02-13T00:41:33.336399+0000","last_fullsized":"2024-02-13T00:41:33.336399+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.6","version":"109'2","reported_seq":57,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.440273+0000","last_change":"2024-02-13T00:41:23.761454+0000","last_active":"2024-02-13T00:41:33.440273+0000","last_peered":"2024-02-13T00:41:33.440273+0000","last_clean":"2024-02-13T00:41:33.440273+0000","last_became_active":"2024-02-13T00:41:01.707941+0000","last_became_peered":"2024-02-13T00:41:01.707941+0000","last_unstale":"2024-02-13T00:41:33.440273+0000","last_undegraded":"2024-02-13T00:41:33.440273+0000","last_fullsized":"2024-02-13T00:41:33.440273+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-02-13T00:41:23.761299+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-02-13T00:41:23.761299+0000","last_clean_scrub_stamp":"2024-02-13T00:41:23.761299+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333347+0000","last_change":"2024-02-13T00:41:01.652447+0000","last_active":"2024-02-13T00:41:33.333347+0000","last_peered":"2024-02-13T00:41:33.333347+0000","last_clean":"2024-02-13T00:41:33.333347+0000","last_became_active":"2024-02-13T00:41:01.645165+0000","last_became_peered":"2024-02-13T00:41:01.645165+0000","last_unstale":"2024-02-13T00:41:33.333347+0000","last_undegraded":"2024-02-13T00:41:33.333347+0000","last_fullsized":"2024-02-13T00:41:33.333347+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":117,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.457146+0000","last_change":"2024-02-13T00:39:19.266801+0000","last_active":"2024-02-13T00:41:33.457146+0000","last_peered":"2024-02-13T00:41:33.457146+0000","last_clean":"2024-02-13T00:41:33.457146+0000","last_became_active":"2024-02-13T00:37:43.324510+0000","last_became_peered":"2024-02-13T00:37:43.324510+0000","last_unstale":"2024-02-13T00:41:33.457146+0000","last_undegraded":"2024-02-13T00:41:33.457146+0000","last_fullsized":"2024-02-13T00:41:33.457146+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:39:19.266729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:19.266729+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.321401+0000","last_change":"2024-02-13T00:40:54.316667+0000","last_active":"2024-02-13T00:41:33.321401+0000","last_peered":"2024-02-13T00:41:33.321401+0000","last_clean":"2024-02-13T00:41:33.321401+0000","last_became_active":"2024-02-13T00:37:41.212866+0000","last_became_peered":"2024-02-13T00:37:41.212866+0000","last_unstale":"2024-02-13T00:41:33.321401+0000","last_undegraded":"2024-02-13T00:41:33.321401+0000","last_fullsized":"2024-02-13T00:41:33.321401+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:54.316591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:40:54.316591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":96,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.316853+0000","last_change":"2024-02-13T00:40:53.536240+0000","last_active":"2024-02-13T00:41:33.316853+0000","last_peered":"2024-02-13T00:41:33.316853+0000","last_clean":"2024-02-13T00:41:33.316853+0000","last_became_active":"2024-02-13T00:37:45.246907+0000","last_became_peered":"2024-02-13T00:37:45.246907+0000","last_unstale":"2024-02-13T00:41:33.316853+0000","last_undegraded":"2024-02-13T00:41:33.316853+0000","last_fullsized":"2024-02-13T00:41:33.316853+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:53.536170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:53.536170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.339435+0000","last_change":"2024-02-13T00:41:02.115927+0000","last_active":"2024-02-13T00:41:33.339435+0000","last_peered":"2024-02-13T00:41:33.339435+0000","last_clean":"2024-02-13T00:41:33.339435+0000","last_became_active":"2024-02-13T00:37:47.553612+0000","last_became_peered":"2024-02-13T00:37:47.553612+0000","last_unstale":"2024-02-13T00:41:33.339435+0000","last_undegraded":"2024-02-13T00:41:33.339435+0000","last_fullsized":"2024-02-13T00:41:33.339435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.115856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.115856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1","version":"108'5","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.449807+0000","last_change":"2024-02-13T00:41:01.722177+0000","last_active":"2024-02-13T00:41:33.449807+0000","last_peered":"2024-02-13T00:41:33.449807+0000","last_clean":"2024-02-13T00:41:33.449807+0000","last_became_active":"2024-02-13T00:41:01.721284+0000","last_became_peered":"2024-02-13T00:41:01.721284+0000","last_unstale":"2024-02-13T00:41:33.449807+0000","last_undegraded":"2024-02-13T00:41:33.449807+0000","last_fullsized":"2024-02-13T00:41:33.449807+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.341648+0000","last_change":"2024-02-13T00:41:11.181071+0000","last_active":"2024-02-13T00:41:33.341648+0000","last_peered":"2024-02-13T00:41:33.341648+0000","last_clean":"2024-02-13T00:41:33.341648+0000","last_became_active":"2024-02-13T00:41:01.560020+0000","last_became_peered":"2024-02-13T00:41:01.560020+0000","last_unstale":"2024-02-13T00:41:33.341648+0000","last_undegraded":"2024-02-13T00:41:33.341648+0000","last_fullsized":"2024-02-13T00:41:33.341648+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:11.180993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:11.180993+0000","last_clean_scrub_stamp":"2024-02-13T00:41:11.180993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.453415+0000","last_change":"2024-02-13T00:41:01.653337+0000","last_active":"2024-02-13T00:41:33.453415+0000","last_peered":"2024-02-13T00:41:33.453415+0000","last_clean":"2024-02-13T00:41:33.453415+0000","last_became_active":"2024-02-13T00:41:01.645866+0000","last_became_peered":"2024-02-13T00:41:01.645866+0000","last_unstale":"2024-02-13T00:41:33.453415+0000","last_undegraded":"2024-02-13T00:41:33.453415+0000","last_fullsized":"2024-02-13T00:41:33.453415+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":150,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.453337+0000","last_change":"2024-02-13T00:40:04.185200+0000","last_active":"2024-02-13T00:41:33.453337+0000","last_peered":"2024-02-13T00:41:33.453337+0000","last_clean":"2024-02-13T00:41:33.453337+0000","last_became_active":"2024-02-13T00:37:43.324313+0000","last_became_peered":"2024-02-13T00:37:43.324313+0000","last_unstale":"2024-02-13T00:41:33.453337+0000","last_undegraded":"2024-02-13T00:41:33.453337+0000","last_fullsized":"2024-02-13T00:41:33.453337+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:40:04.185125+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:40:04.185125+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.316886+0000","last_change":"2024-02-13T00:40:52.503871+0000","last_active":"2024-02-13T00:41:33.316886+0000","last_peered":"2024-02-13T00:41:33.316886+0000","last_clean":"2024-02-13T00:41:33.316886+0000","last_became_active":"2024-02-13T00:37:41.219309+0000","last_became_peered":"2024-02-13T00:37:41.219309+0000","last_unstale":"2024-02-13T00:41:33.316886+0000","last_undegraded":"2024-02-13T00:41:33.316886+0000","last_fullsized":"2024-02-13T00:41:33.316886+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:52.503804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:52.503804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"64'8","reported_seq":146,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.259653+0000","last_change":"2024-02-13T00:39:11.518468+0000","last_active":"2024-02-13T00:41:34.259653+0000","last_peered":"2024-02-13T00:41:34.259653+0000","last_clean":"2024-02-13T00:41:34.259653+0000","last_became_active":"2024-02-13T00:37:45.246757+0000","last_became_peered":"2024-02-13T00:37:45.246757+0000","last_unstale":"2024-02-13T00:41:34.259653+0000","last_undegraded":"2024-02-13T00:41:34.259653+0000","last_fullsized":"2024-02-13T00:41:34.259653+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:11.518359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:11.518359+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.331003+0000","last_change":"2024-02-13T00:39:21.344836+0000","last_active":"2024-02-13T00:41:33.331003+0000","last_peered":"2024-02-13T00:41:33.331003+0000","last_clean":"2024-02-13T00:41:33.331003+0000","last_became_active":"2024-02-13T00:37:47.252722+0000","last_became_peered":"2024-02-13T00:37:47.252722+0000","last_unstale":"2024-02-13T00:41:33.331003+0000","last_undegraded":"2024-02-13T00:41:33.331003+0000","last_fullsized":"2024-02-13T00:41:33.331003+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.344771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.344771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330810+0000","last_change":"2024-02-13T00:41:01.582310+0000","last_active":"2024-02-13T00:41:33.330810+0000","last_peered":"2024-02-13T00:41:33.330810+0000","last_clean":"2024-02-13T00:41:33.330810+0000","last_became_active":"2024-02-13T00:41:01.581659+0000","last_became_peered":"2024-02-13T00:41:01.581659+0000","last_unstale":"2024-02-13T00:41:33.330810+0000","last_undegraded":"2024-02-13T00:41:33.330810+0000","last_fullsized":"2024-02-13T00:41:33.330810+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.4","version":"107'1","reported_seq":48,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.428291+0000","last_change":"2024-02-13T00:41:14.728312+0000","last_active":"2024-02-13T00:41:33.428291+0000","last_peered":"2024-02-13T00:41:33.428291+0000","last_clean":"2024-02-13T00:41:33.428291+0000","last_became_active":"2024-02-13T00:41:01.716123+0000","last_became_peered":"2024-02-13T00:41:01.716123+0000","last_unstale":"2024-02-13T00:41:33.428291+0000","last_undegraded":"2024-02-13T00:41:33.428291+0000","last_fullsized":"2024-02-13T00:41:33.428291+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'1","last_scrub_stamp":"2024-02-13T00:41:14.728247+0000","last_deep_scrub":"107'1","last_deep_scrub_stamp":"2024-02-13T00:41:14.728247+0000","last_clean_scrub_stamp":"2024-02-13T00:41:14.728247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.339542+0000","last_change":"2024-02-13T00:41:01.559598+0000","last_active":"2024-02-13T00:41:33.339542+0000","last_peered":"2024-02-13T00:41:33.339542+0000","last_clean":"2024-02-13T00:41:33.339542+0000","last_became_active":"2024-02-13T00:41:01.557578+0000","last_became_peered":"2024-02-13T00:41:01.557578+0000","last_unstale":"2024-02-13T00:41:33.339542+0000","last_undegraded":"2024-02-13T00:41:33.339542+0000","last_fullsized":"2024-02-13T00:41:33.339542+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":125,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.317244+0000","last_change":"2024-02-13T00:39:10.549128+0000","last_active":"2024-02-13T00:41:33.317244+0000","last_peered":"2024-02-13T00:41:33.317244+0000","last_clean":"2024-02-13T00:41:33.317244+0000","last_became_active":"2024-02-13T00:37:43.322357+0000","last_became_peered":"2024-02-13T00:37:43.322357+0000","last_unstale":"2024-02-13T00:41:33.317244+0000","last_undegraded":"2024-02-13T00:41:33.317244+0000","last_fullsized":"2024-02-13T00:41:33.317244+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:39:10.549045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:10.549045+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.328987+0000","last_change":"2024-02-13T00:40:52.801522+0000","last_active":"2024-02-13T00:41:33.328987+0000","last_peered":"2024-02-13T00:41:33.328987+0000","last_clean":"2024-02-13T00:41:33.328987+0000","last_became_active":"2024-02-13T00:37:41.217953+0000","last_became_peered":"2024-02-13T00:37:41.217953+0000","last_unstale":"2024-02-13T00:41:33.328987+0000","last_undegraded":"2024-02-13T00:41:33.328987+0000","last_fullsized":"2024-02-13T00:41:33.328987+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:52.801455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:52.801455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"84'9","reported_seq":155,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.259569+0000","last_change":"2024-02-13T00:41:32.715662+0000","last_active":"2024-02-13T00:41:34.259569+0000","last_peered":"2024-02-13T00:41:34.259569+0000","last_clean":"2024-02-13T00:41:34.259569+0000","last_became_active":"2024-02-13T00:37:45.250598+0000","last_became_peered":"2024-02-13T00:37:45.250598+0000","last_unstale":"2024-02-13T00:41:34.259569+0000","last_undegraded":"2024-02-13T00:41:34.259569+0000","last_fullsized":"2024-02-13T00:41:34.259569+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-02-13T00:41:32.715550+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.715550+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.335115+0000","last_change":"2024-02-13T00:40:41.387258+0000","last_active":"2024-02-13T00:41:33.335115+0000","last_peered":"2024-02-13T00:41:33.335115+0000","last_clean":"2024-02-13T00:41:33.335115+0000","last_became_active":"2024-02-13T00:40:35.850176+0000","last_became_peered":"2024-02-13T00:40:35.850176+0000","last_unstale":"2024-02-13T00:41:33.335115+0000","last_undegraded":"2024-02-13T00:41:33.335115+0000","last_fullsized":"2024-02-13T00:41:33.335115+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:41.387157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:41.387157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.3","version":"108'5","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.426667+0000","last_change":"2024-02-13T00:41:01.705740+0000","last_active":"2024-02-13T00:41:33.426667+0000","last_peered":"2024-02-13T00:41:33.426667+0000","last_clean":"2024-02-13T00:41:33.426667+0000","last_became_active":"2024-02-13T00:41:01.705297+0000","last_became_peered":"2024-02-13T00:41:01.705297+0000","last_unstale":"2024-02-13T00:41:33.426667+0000","last_undegraded":"2024-02-13T00:41:33.426667+0000","last_fullsized":"2024-02-13T00:41:33.426667+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.5","version":"126'2","reported_seq":52,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.445741+0000","last_change":"2024-02-13T00:41:21.767473+0000","last_active":"2024-02-13T00:41:33.445741+0000","last_peered":"2024-02-13T00:41:33.445741+0000","last_clean":"2024-02-13T00:41:33.445741+0000","last_became_active":"2024-02-13T00:41:01.722886+0000","last_became_peered":"2024-02-13T00:41:01.722886+0000","last_unstale":"2024-02-13T00:41:33.445741+0000","last_undegraded":"2024-02-13T00:41:33.445741+0000","last_fullsized":"2024-02-13T00:41:33.445741+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-02-13T00:41:21.767375+0000","last_deep_scrub":"126'2","last_deep_scrub_stamp":"2024-02-13T00:41:21.767375+0000","last_clean_scrub_stamp":"2024-02-13T00:41:21.767375+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.317435+0000","last_change":"2024-02-13T00:41:01.672359+0000","last_active":"2024-02-13T00:41:33.317435+0000","last_peered":"2024-02-13T00:41:33.317435+0000","last_clean":"2024-02-13T00:41:33.317435+0000","last_became_active":"2024-02-13T00:41:01.672068+0000","last_became_peered":"2024-02-13T00:41:01.672068+0000","last_unstale":"2024-02-13T00:41:33.317435+0000","last_undegraded":"2024-02-13T00:41:33.317435+0000","last_fullsized":"2024-02-13T00:41:33.317435+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":119,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.425332+0000","last_change":"2024-02-13T00:40:16.009588+0000","last_active":"2024-02-13T00:41:33.425332+0000","last_peered":"2024-02-13T00:41:33.425332+0000","last_clean":"2024-02-13T00:41:33.425332+0000","last_became_active":"2024-02-13T00:37:43.225347+0000","last_became_peered":"2024-02-13T00:37:43.225347+0000","last_unstale":"2024-02-13T00:41:33.425332+0000","last_undegraded":"2024-02-13T00:41:33.425332+0000","last_fullsized":"2024-02-13T00:41:33.425332+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:40:16.009518+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:40:16.009518+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.328889+0000","last_change":"2024-02-13T00:40:58.334820+0000","last_active":"2024-02-13T00:41:33.328889+0000","last_peered":"2024-02-13T00:41:33.328889+0000","last_clean":"2024-02-13T00:41:33.328889+0000","last_became_active":"2024-02-13T00:37:41.216390+0000","last_became_peered":"2024-02-13T00:37:41.216390+0000","last_unstale":"2024-02-13T00:41:33.328889+0000","last_undegraded":"2024-02-13T00:41:33.328889+0000","last_fullsized":"2024-02-13T00:41:33.328889+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:58.334705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.334705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":91,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.451098+0000","last_change":"2024-02-13T00:39:29.229760+0000","last_active":"2024-02-13T00:41:33.451098+0000","last_peered":"2024-02-13T00:41:33.451098+0000","last_clean":"2024-02-13T00:41:33.451098+0000","last_became_active":"2024-02-13T00:37:45.243594+0000","last_became_peered":"2024-02-13T00:37:45.243594+0000","last_unstale":"2024-02-13T00:41:33.451098+0000","last_undegraded":"2024-02-13T00:41:33.451098+0000","last_fullsized":"2024-02-13T00:41:33.451098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.229676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.336127+0000","last_change":"2024-02-13T00:39:28.286635+0000","last_active":"2024-02-13T00:41:33.336127+0000","last_peered":"2024-02-13T00:41:33.336127+0000","last_clean":"2024-02-13T00:41:33.336127+0000","last_became_active":"2024-02-13T00:37:47.259792+0000","last_became_peered":"2024-02-13T00:37:47.259792+0000","last_unstale":"2024-02-13T00:41:33.336127+0000","last_undegraded":"2024-02-13T00:41:33.336127+0000","last_fullsized":"2024-02-13T00:41:33.336127+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.286574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.286574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.320592+0000","last_change":"2024-02-13T00:41:01.745030+0000","last_active":"2024-02-13T00:41:33.320592+0000","last_peered":"2024-02-13T00:41:33.320592+0000","last_clean":"2024-02-13T00:41:33.320592+0000","last_became_active":"2024-02-13T00:41:01.743858+0000","last_became_peered":"2024-02-13T00:41:01.743858+0000","last_unstale":"2024-02-13T00:41:33.320592+0000","last_undegraded":"2024-02-13T00:41:33.320592+0000","last_fullsized":"2024-02-13T00:41:33.320592+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.2","version":"107'2","reported_seq":49,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330374+0000","last_change":"2024-02-13T00:41:07.150195+0000","last_active":"2024-02-13T00:41:33.330374+0000","last_peered":"2024-02-13T00:41:33.330374+0000","last_clean":"2024-02-13T00:41:33.330374+0000","last_became_active":"2024-02-13T00:41:01.629104+0000","last_became_peered":"2024-02-13T00:41:01.629104+0000","last_unstale":"2024-02-13T00:41:33.330374+0000","last_undegraded":"2024-02-13T00:41:33.330374+0000","last_fullsized":"2024-02-13T00:41:33.330374+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-02-13T00:41:07.150117+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-02-13T00:41:07.150117+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.150117+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.331469+0000","last_change":"2024-02-13T00:41:01.578270+0000","last_active":"2024-02-13T00:41:33.331469+0000","last_peered":"2024-02-13T00:41:33.331469+0000","last_clean":"2024-02-13T00:41:33.331469+0000","last_became_active":"2024-02-13T00:41:01.577649+0000","last_became_peered":"2024-02-13T00:41:01.577649+0000","last_unstale":"2024-02-13T00:41:33.331469+0000","last_undegraded":"2024-02-13T00:41:33.331469+0000","last_fullsized":"2024-02-13T00:41:33.331469+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":109,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.331388+0000","last_change":"2024-02-13T00:40:44.273570+0000","last_active":"2024-02-13T00:41:33.331388+0000","last_peered":"2024-02-13T00:41:33.331388+0000","last_clean":"2024-02-13T00:41:33.331388+0000","last_became_active":"2024-02-13T00:40:35.852956+0000","last_became_peered":"2024-02-13T00:40:35.852956+0000","last_unstale":"2024-02-13T00:41:33.331388+0000","last_undegraded":"2024-02-13T00:41:33.331388+0000","last_fullsized":"2024-02-13T00:41:33.331388+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:40:44.273511+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.273511+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.444970+0000","last_change":"2024-02-13T00:41:03.608458+0000","last_active":"2024-02-13T00:41:33.444970+0000","last_peered":"2024-02-13T00:41:33.444970+0000","last_clean":"2024-02-13T00:41:33.444970+0000","last_became_active":"2024-02-13T00:37:41.215531+0000","last_became_peered":"2024-02-13T00:37:41.215531+0000","last_unstale":"2024-02-13T00:41:33.444970+0000","last_undegraded":"2024-02-13T00:41:33.444970+0000","last_fullsized":"2024-02-13T00:41:33.444970+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:03.608379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:03.608379+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":97,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.425464+0000","last_change":"2024-02-13T00:40:49.666455+0000","last_active":"2024-02-13T00:41:33.425464+0000","last_peered":"2024-02-13T00:41:33.425464+0000","last_clean":"2024-02-13T00:41:33.425464+0000","last_became_active":"2024-02-13T00:37:45.247114+0000","last_became_peered":"2024-02-13T00:37:45.247114+0000","last_unstale":"2024-02-13T00:41:33.425464+0000","last_undegraded":"2024-02-13T00:41:33.425464+0000","last_fullsized":"2024-02-13T00:41:33.425464+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:49.666387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:40:49.666387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.445966+0000","last_change":"2024-02-13T00:40:05.137000+0000","last_active":"2024-02-13T00:41:33.445966+0000","last_peered":"2024-02-13T00:41:33.445966+0000","last_clean":"2024-02-13T00:41:33.445966+0000","last_became_active":"2024-02-13T00:37:47.254519+0000","last_became_peered":"2024-02-13T00:37:47.254519+0000","last_unstale":"2024-02-13T00:41:33.445966+0000","last_undegraded":"2024-02-13T00:41:33.445966+0000","last_fullsized":"2024-02-13T00:41:33.445966+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:05.136935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:05.136935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.445046+0000","last_change":"2024-02-13T00:41:01.722213+0000","last_active":"2024-02-13T00:41:33.445046+0000","last_peered":"2024-02-13T00:41:33.445046+0000","last_clean":"2024-02-13T00:41:33.445046+0000","last_became_active":"2024-02-13T00:41:01.721963+0000","last_became_peered":"2024-02-13T00:41:01.721963+0000","last_unstale":"2024-02-13T00:41:33.445046+0000","last_undegraded":"2024-02-13T00:41:33.445046+0000","last_fullsized":"2024-02-13T00:41:33.445046+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330164+0000","last_change":"2024-02-13T00:41:06.963516+0000","last_active":"2024-02-13T00:41:33.330164+0000","last_peered":"2024-02-13T00:41:33.330164+0000","last_clean":"2024-02-13T00:41:33.330164+0000","last_became_active":"2024-02-13T00:41:01.598445+0000","last_became_peered":"2024-02-13T00:41:01.598445+0000","last_unstale":"2024-02-13T00:41:33.330164+0000","last_undegraded":"2024-02-13T00:41:33.330164+0000","last_fullsized":"2024-02-13T00:41:33.330164+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:06.963414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:06.963414+0000","last_clean_scrub_stamp":"2024-02-13T00:41:06.963414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.319917+0000","last_change":"2024-02-13T00:41:01.662653+0000","last_active":"2024-02-13T00:41:33.319917+0000","last_peered":"2024-02-13T00:41:33.319917+0000","last_clean":"2024-02-13T00:41:33.319917+0000","last_became_active":"2024-02-13T00:41:01.661372+0000","last_became_peered":"2024-02-13T00:41:01.661372+0000","last_unstale":"2024-02-13T00:41:33.319917+0000","last_undegraded":"2024-02-13T00:41:33.319917+0000","last_fullsized":"2024-02-13T00:41:33.319917+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.443904+0000","last_change":"2024-02-13T00:40:51.054856+0000","last_active":"2024-02-13T00:41:33.443904+0000","last_peered":"2024-02-13T00:41:33.443904+0000","last_clean":"2024-02-13T00:41:33.443904+0000","last_became_active":"2024-02-13T00:37:43.234395+0000","last_became_peered":"2024-02-13T00:37:43.234395+0000","last_unstale":"2024-02-13T00:41:33.443904+0000","last_undegraded":"2024-02-13T00:41:33.443904+0000","last_fullsized":"2024-02-13T00:41:33.443904+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:40:51.054728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.054728+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.316965+0000","last_change":"2024-02-13T00:40:10.635555+0000","last_active":"2024-02-13T00:41:33.316965+0000","last_peered":"2024-02-13T00:41:33.316965+0000","last_clean":"2024-02-13T00:41:33.316965+0000","last_became_active":"2024-02-13T00:37:41.213033+0000","last_became_peered":"2024-02-13T00:37:41.213033+0000","last_unstale":"2024-02-13T00:41:33.316965+0000","last_undegraded":"2024-02-13T00:41:33.316965+0000","last_fullsized":"2024-02-13T00:41:33.316965+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:10.635478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:10.635478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":97,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.321258+0000","last_change":"2024-02-13T00:39:57.978687+0000","last_active":"2024-02-13T00:41:33.321258+0000","last_peered":"2024-02-13T00:41:33.321258+0000","last_clean":"2024-02-13T00:41:33.321258+0000","last_became_active":"2024-02-13T00:37:45.249556+0000","last_became_peered":"2024-02-13T00:37:45.249556+0000","last_unstale":"2024-02-13T00:41:33.321258+0000","last_undegraded":"2024-02-13T00:41:33.321258+0000","last_fullsized":"2024-02-13T00:41:33.321258+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:57.978593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:57.978593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":97,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330446+0000","last_change":"2024-02-13T00:40:41.220513+0000","last_active":"2024-02-13T00:41:33.330446+0000","last_peered":"2024-02-13T00:41:33.330446+0000","last_clean":"2024-02-13T00:41:33.330446+0000","last_became_active":"2024-02-13T00:37:47.254667+0000","last_became_peered":"2024-02-13T00:37:47.254667+0000","last_unstale":"2024-02-13T00:41:33.330446+0000","last_undegraded":"2024-02-13T00:41:33.330446+0000","last_fullsized":"2024-02-13T00:41:33.330446+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:41.220387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:41.220387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.6","version":"108'9","reported_seq":51,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.344043+0000","last_change":"2024-02-13T00:41:01.631258+0000","last_active":"2024-02-13T00:41:33.344043+0000","last_peered":"2024-02-13T00:41:33.344043+0000","last_clean":"2024-02-13T00:41:33.344043+0000","last_became_active":"2024-02-13T00:41:01.630983+0000","last_became_peered":"2024-02-13T00:41:01.630983+0000","last_unstale":"2024-02-13T00:41:33.344043+0000","last_undegraded":"2024-02-13T00:41:33.344043+0000","last_fullsized":"2024-02-13T00:41:33.344043+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.443981+0000","last_change":"2024-02-13T00:41:06.727598+0000","last_active":"2024-02-13T00:41:33.443981+0000","last_peered":"2024-02-13T00:41:33.443981+0000","last_clean":"2024-02-13T00:41:33.443981+0000","last_became_active":"2024-02-13T00:41:01.649656+0000","last_became_peered":"2024-02-13T00:41:01.649656+0000","last_unstale":"2024-02-13T00:41:33.443981+0000","last_undegraded":"2024-02-13T00:41:33.443981+0000","last_fullsized":"2024-02-13T00:41:33.443981+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:06.727500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:06.727500+0000","last_clean_scrub_stamp":"2024-02-13T00:41:06.727500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.323487+0000","last_change":"2024-02-13T00:41:01.623348+0000","last_active":"2024-02-13T00:41:33.323487+0000","last_peered":"2024-02-13T00:41:33.323487+0000","last_clean":"2024-02-13T00:41:33.323487+0000","last_became_active":"2024-02-13T00:41:01.623088+0000","last_became_peered":"2024-02-13T00:41:01.623088+0000","last_unstale":"2024-02-13T00:41:33.323487+0000","last_undegraded":"2024-02-13T00:41:33.323487+0000","last_fullsized":"2024-02-13T00:41:33.323487+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":127,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.446632+0000","last_change":"2024-02-13T00:39:34.204969+0000","last_active":"2024-02-13T00:41:33.446632+0000","last_peered":"2024-02-13T00:41:33.446632+0000","last_clean":"2024-02-13T00:41:33.446632+0000","last_became_active":"2024-02-13T00:37:43.324196+0000","last_became_peered":"2024-02-13T00:37:43.324196+0000","last_unstale":"2024-02-13T00:41:33.446632+0000","last_undegraded":"2024-02-13T00:41:33.446632+0000","last_fullsized":"2024-02-13T00:41:33.446632+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:34.204888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:34.204888+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":98,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333426+0000","last_change":"2024-02-13T00:41:32.424389+0000","last_active":"2024-02-13T00:41:33.333426+0000","last_peered":"2024-02-13T00:41:33.333426+0000","last_clean":"2024-02-13T00:41:33.333426+0000","last_became_active":"2024-02-13T00:37:41.220344+0000","last_became_peered":"2024-02-13T00:37:41.220344+0000","last_unstale":"2024-02-13T00:41:33.333426+0000","last_undegraded":"2024-02-13T00:41:33.333426+0000","last_fullsized":"2024-02-13T00:41:33.333426+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:07.358534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:07.358534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":96,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330379+0000","last_change":"2024-02-13T00:40:59.164776+0000","last_active":"2024-02-13T00:41:33.330379+0000","last_peered":"2024-02-13T00:41:33.330379+0000","last_clean":"2024-02-13T00:41:33.330379+0000","last_became_active":"2024-02-13T00:37:45.248592+0000","last_became_peered":"2024-02-13T00:37:45.248592+0000","last_unstale":"2024-02-13T00:41:33.330379+0000","last_undegraded":"2024-02-13T00:41:33.330379+0000","last_fullsized":"2024-02-13T00:41:33.330379+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.164690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.164690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.451245+0000","last_change":"2024-02-13T00:40:59.881327+0000","last_active":"2024-02-13T00:41:33.451245+0000","last_peered":"2024-02-13T00:41:33.451245+0000","last_clean":"2024-02-13T00:41:33.451245+0000","last_became_active":"2024-02-13T00:37:47.552917+0000","last_became_peered":"2024-02-13T00:37:47.552917+0000","last_unstale":"2024-02-13T00:41:33.451245+0000","last_undegraded":"2024-02-13T00:41:33.451245+0000","last_fullsized":"2024-02-13T00:41:33.451245+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.881255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.881255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.335300+0000","last_change":"2024-02-13T00:41:01.595241+0000","last_active":"2024-02-13T00:41:33.335300+0000","last_peered":"2024-02-13T00:41:33.335300+0000","last_clean":"2024-02-13T00:41:33.335300+0000","last_became_active":"2024-02-13T00:41:01.581338+0000","last_became_peered":"2024-02-13T00:41:01.581338+0000","last_unstale":"2024-02-13T00:41:33.335300+0000","last_undegraded":"2024-02-13T00:41:33.335300+0000","last_fullsized":"2024-02-13T00:41:33.335300+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"107'1","reported_seq":48,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.321901+0000","last_change":"2024-02-13T00:41:07.341748+0000","last_active":"2024-02-13T00:41:33.321901+0000","last_peered":"2024-02-13T00:41:33.321901+0000","last_clean":"2024-02-13T00:41:33.321901+0000","last_became_active":"2024-02-13T00:41:01.590927+0000","last_became_peered":"2024-02-13T00:41:01.590927+0000","last_unstale":"2024-02-13T00:41:33.321901+0000","last_undegraded":"2024-02-13T00:41:33.321901+0000","last_fullsized":"2024-02-13T00:41:33.321901+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'1","last_scrub_stamp":"2024-02-13T00:41:07.341678+0000","last_deep_scrub":"107'1","last_deep_scrub_stamp":"2024-02-13T00:41:07.341678+0000","last_clean_scrub_stamp":"2024-02-13T00:41:07.341678+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.340900+0000","last_change":"2024-02-13T00:41:01.569377+0000","last_active":"2024-02-13T00:41:33.340900+0000","last_peered":"2024-02-13T00:41:33.340900+0000","last_clean":"2024-02-13T00:41:33.340900+0000","last_became_active":"2024-02-13T00:41:01.565724+0000","last_became_peered":"2024-02-13T00:41:01.565724+0000","last_unstale":"2024-02-13T00:41:33.340900+0000","last_undegraded":"2024-02-13T00:41:33.340900+0000","last_fullsized":"2024-02-13T00:41:33.340900+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":134,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333637+0000","last_change":"2024-02-13T00:40:06.169154+0000","last_active":"2024-02-13T00:41:33.333637+0000","last_peered":"2024-02-13T00:41:33.333637+0000","last_clean":"2024-02-13T00:41:33.333637+0000","last_became_active":"2024-02-13T00:37:43.324221+0000","last_became_peered":"2024-02-13T00:37:43.324221+0000","last_unstale":"2024-02-13T00:41:33.333637+0000","last_undegraded":"2024-02-13T00:41:33.333637+0000","last_fullsized":"2024-02-13T00:41:33.333637+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:40:06.169087+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:40:06.169087+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.326817+0000","last_change":"2024-02-13T00:41:32.424906+0000","last_active":"2024-02-13T00:41:33.326817+0000","last_peered":"2024-02-13T00:41:33.326817+0000","last_clean":"2024-02-13T00:41:33.326817+0000","last_became_active":"2024-02-13T00:37:41.214965+0000","last_became_peered":"2024-02-13T00:37:41.214965+0000","last_unstale":"2024-02-13T00:41:33.326817+0000","last_undegraded":"2024-02-13T00:41:33.326817+0000","last_fullsized":"2024-02-13T00:41:33.326817+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:32.424844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.424844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"64'8","reported_seq":155,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.272048+0000","last_change":"2024-02-13T00:41:02.323171+0000","last_active":"2024-02-13T00:41:34.272048+0000","last_peered":"2024-02-13T00:41:34.272048+0000","last_clean":"2024-02-13T00:41:34.272048+0000","last_became_active":"2024-02-13T00:37:45.249902+0000","last_became_peered":"2024-02-13T00:37:45.249902+0000","last_unstale":"2024-02-13T00:41:34.272048+0000","last_undegraded":"2024-02-13T00:41:34.272048+0000","last_fullsized":"2024-02-13T00:41:34.272048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:02.323101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.323101+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":95,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.337185+0000","last_change":"2024-02-13T00:40:02.168591+0000","last_active":"2024-02-13T00:41:33.337185+0000","last_peered":"2024-02-13T00:41:33.337185+0000","last_clean":"2024-02-13T00:41:33.337185+0000","last_became_active":"2024-02-13T00:37:47.254586+0000","last_became_peered":"2024-02-13T00:37:47.254586+0000","last_unstale":"2024-02-13T00:41:33.337185+0000","last_undegraded":"2024-02-13T00:41:33.337185+0000","last_fullsized":"2024-02-13T00:41:33.337185+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:40:02.168499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:02.168499+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"107'2","reported_seq":44,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.340032+0000","last_change":"2024-02-13T00:41:01.629145+0000","last_active":"2024-02-13T00:41:33.340032+0000","last_peered":"2024-02-13T00:41:33.340032+0000","last_clean":"2024-02-13T00:41:33.340032+0000","last_became_active":"2024-02-13T00:41:01.628231+0000","last_became_peered":"2024-02-13T00:41:01.628231+0000","last_unstale":"2024-02-13T00:41:33.340032+0000","last_undegraded":"2024-02-13T00:41:33.340032+0000","last_fullsized":"2024-02-13T00:41:33.340032+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.458643+0000","last_change":"2024-02-13T00:41:01.741477+0000","last_active":"2024-02-13T00:41:33.458643+0000","last_peered":"2024-02-13T00:41:33.458643+0000","last_clean":"2024-02-13T00:41:33.458643+0000","last_became_active":"2024-02-13T00:41:01.740950+0000","last_became_peered":"2024-02-13T00:41:01.740950+0000","last_unstale":"2024-02-13T00:41:33.458643+0000","last_undegraded":"2024-02-13T00:41:33.458643+0000","last_fullsized":"2024-02-13T00:41:33.458643+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.441969+0000","last_change":"2024-02-13T00:41:01.707939+0000","last_active":"2024-02-13T00:41:33.441969+0000","last_peered":"2024-02-13T00:41:33.441969+0000","last_clean":"2024-02-13T00:41:33.441969+0000","last_became_active":"2024-02-13T00:41:01.707651+0000","last_became_peered":"2024-02-13T00:41:01.707651+0000","last_unstale":"2024-02-13T00:41:33.441969+0000","last_undegraded":"2024-02-13T00:41:33.441969+0000","last_fullsized":"2024-02-13T00:41:33.441969+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":133,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333737+0000","last_change":"2024-02-13T00:40:42.231057+0000","last_active":"2024-02-13T00:41:33.333737+0000","last_peered":"2024-02-13T00:41:33.333737+0000","last_clean":"2024-02-13T00:41:33.333737+0000","last_became_active":"2024-02-13T00:37:43.230491+0000","last_became_peered":"2024-02-13T00:37:43.230491+0000","last_unstale":"2024-02-13T00:41:33.333737+0000","last_undegraded":"2024-02-13T00:41:33.333737+0000","last_fullsized":"2024-02-13T00:41:33.333737+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:40:42.230976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:42.230976+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":95,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.446088+0000","last_change":"2024-02-13T00:39:41.231336+0000","last_active":"2024-02-13T00:41:33.446088+0000","last_peered":"2024-02-13T00:41:33.446088+0000","last_clean":"2024-02-13T00:41:33.446088+0000","last_became_active":"2024-02-13T00:37:41.216574+0000","last_became_peered":"2024-02-13T00:37:41.216574+0000","last_unstale":"2024-02-13T00:41:33.446088+0000","last_undegraded":"2024-02-13T00:41:33.446088+0000","last_fullsized":"2024-02-13T00:41:33.446088+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:41.231244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:41.231244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":91,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333675+0000","last_change":"2024-02-13T00:39:15.328037+0000","last_active":"2024-02-13T00:41:33.333675+0000","last_peered":"2024-02-13T00:41:33.333675+0000","last_clean":"2024-02-13T00:41:33.333675+0000","last_became_active":"2024-02-13T00:37:45.249064+0000","last_became_peered":"2024-02-13T00:37:45.249064+0000","last_unstale":"2024-02-13T00:41:33.333675+0000","last_undegraded":"2024-02-13T00:41:33.333675+0000","last_fullsized":"2024-02-13T00:41:33.333675+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.327971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.327971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":95,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.446720+0000","last_change":"2024-02-13T00:40:06.026382+0000","last_active":"2024-02-13T00:41:33.446720+0000","last_peered":"2024-02-13T00:41:33.446720+0000","last_clean":"2024-02-13T00:41:33.446720+0000","last_became_active":"2024-02-13T00:37:47.260182+0000","last_became_peered":"2024-02-13T00:37:47.260182+0000","last_unstale":"2024-02-13T00:41:33.446720+0000","last_undegraded":"2024-02-13T00:41:33.446720+0000","last_fullsized":"2024-02-13T00:41:33.446720+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:06.026318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:06.026318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.446153+0000","last_change":"2024-02-13T00:41:01.682551+0000","last_active":"2024-02-13T00:41:33.446153+0000","last_peered":"2024-02-13T00:41:33.446153+0000","last_clean":"2024-02-13T00:41:33.446153+0000","last_became_active":"2024-02-13T00:41:01.682261+0000","last_became_peered":"2024-02-13T00:41:01.682261+0000","last_unstale":"2024-02-13T00:41:33.446153+0000","last_undegraded":"2024-02-13T00:41:33.446153+0000","last_fullsized":"2024-02-13T00:41:33.446153+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333014+0000","last_change":"2024-02-13T00:41:01.645310+0000","last_active":"2024-02-13T00:41:33.333014+0000","last_peered":"2024-02-13T00:41:33.333014+0000","last_clean":"2024-02-13T00:41:33.333014+0000","last_became_active":"2024-02-13T00:41:01.641767+0000","last_became_peered":"2024-02-13T00:41:01.641767+0000","last_unstale":"2024-02-13T00:41:33.333014+0000","last_undegraded":"2024-02-13T00:41:33.333014+0000","last_fullsized":"2024-02-13T00:41:33.333014+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.329447+0000","last_change":"2024-02-13T00:41:01.705201+0000","last_active":"2024-02-13T00:41:33.329447+0000","last_peered":"2024-02-13T00:41:33.329447+0000","last_clean":"2024-02-13T00:41:33.329447+0000","last_became_active":"2024-02-13T00:41:01.704705+0000","last_became_peered":"2024-02-13T00:41:01.704705+0000","last_unstale":"2024-02-13T00:41:33.329447+0000","last_undegraded":"2024-02-13T00:41:33.329447+0000","last_fullsized":"2024-02-13T00:41:33.329447+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":124,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.335757+0000","last_change":"2024-02-13T00:40:44.369174+0000","last_active":"2024-02-13T00:41:33.335757+0000","last_peered":"2024-02-13T00:41:33.335757+0000","last_clean":"2024-02-13T00:41:33.335757+0000","last_became_active":"2024-02-13T00:37:43.232938+0000","last_became_peered":"2024-02-13T00:37:43.232938+0000","last_unstale":"2024-02-13T00:41:33.335757+0000","last_undegraded":"2024-02-13T00:41:33.335757+0000","last_fullsized":"2024-02-13T00:41:33.335757+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:40:44.369070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.369070+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.322222+0000","last_change":"2024-02-13T00:40:57.316472+0000","last_active":"2024-02-13T00:41:33.322222+0000","last_peered":"2024-02-13T00:41:33.322222+0000","last_clean":"2024-02-13T00:41:33.322222+0000","last_became_active":"2024-02-13T00:37:41.218334+0000","last_became_peered":"2024-02-13T00:37:41.218334+0000","last_unstale":"2024-02-13T00:41:33.322222+0000","last_undegraded":"2024-02-13T00:41:33.322222+0000","last_fullsized":"2024-02-13T00:41:33.322222+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.316400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.316400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":92,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.445656+0000","last_change":"2024-02-13T00:39:30.245586+0000","last_active":"2024-02-13T00:41:33.445656+0000","last_peered":"2024-02-13T00:41:33.445656+0000","last_clean":"2024-02-13T00:41:33.445656+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:41:33.445656+0000","last_undegraded":"2024-02-13T00:41:33.445656+0000","last_fullsized":"2024-02-13T00:41:33.445656+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:30.245508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.245508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":55,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330369+0000","last_change":"2024-02-13T00:40:35.846828+0000","last_active":"2024-02-13T00:41:33.330369+0000","last_peered":"2024-02-13T00:41:33.330369+0000","last_clean":"2024-02-13T00:41:33.330369+0000","last_became_active":"2024-02-13T00:40:35.846506+0000","last_became_peered":"2024-02-13T00:40:35.846506+0000","last_unstale":"2024-02-13T00:41:33.330369+0000","last_undegraded":"2024-02-13T00:41:33.330369+0000","last_fullsized":"2024-02-13T00:41:33.330369+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:15.106889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:15.106889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.a","version":"108'10","reported_seq":52,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.450720+0000","last_change":"2024-02-13T00:41:01.724237+0000","last_active":"2024-02-13T00:41:33.450720+0000","last_peered":"2024-02-13T00:41:33.450720+0000","last_clean":"2024-02-13T00:41:33.450720+0000","last_became_active":"2024-02-13T00:41:01.723568+0000","last_became_peered":"2024-02-13T00:41:01.723568+0000","last_unstale":"2024-02-13T00:41:33.450720+0000","last_undegraded":"2024-02-13T00:41:33.450720+0000","last_fullsized":"2024-02-13T00:41:33.450720+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.453832+0000","last_change":"2024-02-13T00:41:01.549028+0000","last_active":"2024-02-13T00:41:33.453832+0000","last_peered":"2024-02-13T00:41:33.453832+0000","last_clean":"2024-02-13T00:41:33.453832+0000","last_became_active":"2024-02-13T00:41:01.547783+0000","last_became_peered":"2024-02-13T00:41:01.547783+0000","last_unstale":"2024-02-13T00:41:33.453832+0000","last_undegraded":"2024-02-13T00:41:33.453832+0000","last_fullsized":"2024-02-13T00:41:33.453832+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.340075+0000","last_change":"2024-02-13T00:41:01.600797+0000","last_active":"2024-02-13T00:41:33.340075+0000","last_peered":"2024-02-13T00:41:33.340075+0000","last_clean":"2024-02-13T00:41:33.340075+0000","last_became_active":"2024-02-13T00:41:01.583043+0000","last_became_peered":"2024-02-13T00:41:01.583043+0000","last_unstale":"2024-02-13T00:41:33.340075+0000","last_undegraded":"2024-02-13T00:41:33.340075+0000","last_fullsized":"2024-02-13T00:41:33.340075+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":131,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.455202+0000","last_change":"2024-02-13T00:39:26.214150+0000","last_active":"2024-02-13T00:41:33.455202+0000","last_peered":"2024-02-13T00:41:33.455202+0000","last_clean":"2024-02-13T00:41:33.455202+0000","last_became_active":"2024-02-13T00:37:43.236026+0000","last_became_peered":"2024-02-13T00:37:43.236026+0000","last_unstale":"2024-02-13T00:41:33.455202+0000","last_undegraded":"2024-02-13T00:41:33.455202+0000","last_fullsized":"2024-02-13T00:41:33.455202+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:39:26.214084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:26.214084+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.451029+0000","last_change":"2024-02-13T00:40:49.110833+0000","last_active":"2024-02-13T00:41:33.451029+0000","last_peered":"2024-02-13T00:41:33.451029+0000","last_clean":"2024-02-13T00:41:33.451029+0000","last_became_active":"2024-02-13T00:37:41.218333+0000","last_became_peered":"2024-02-13T00:37:41.218333+0000","last_unstale":"2024-02-13T00:41:33.451029+0000","last_undegraded":"2024-02-13T00:41:33.451029+0000","last_fullsized":"2024-02-13T00:41:33.451029+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:49.110725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:49.110725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.453537+0000","last_change":"2024-02-13T00:40:52.067206+0000","last_active":"2024-02-13T00:41:33.453537+0000","last_peered":"2024-02-13T00:41:33.453537+0000","last_clean":"2024-02-13T00:41:33.453537+0000","last_became_active":"2024-02-13T00:40:35.858538+0000","last_became_peered":"2024-02-13T00:40:35.858538+0000","last_unstale":"2024-02-13T00:41:33.453537+0000","last_undegraded":"2024-02-13T00:41:33.453537+0000","last_fullsized":"2024-02-13T00:41:33.453537+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:52.067101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:40:52.067101+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.457062+0000","last_change":"2024-02-13T00:40:57.950554+0000","last_active":"2024-02-13T00:41:33.457062+0000","last_peered":"2024-02-13T00:41:33.457062+0000","last_clean":"2024-02-13T00:41:33.457062+0000","last_became_active":"2024-02-13T00:37:47.553853+0000","last_became_peered":"2024-02-13T00:37:47.553853+0000","last_unstale":"2024-02-13T00:41:33.457062+0000","last_undegraded":"2024-02-13T00:41:33.457062+0000","last_fullsized":"2024-02-13T00:41:33.457062+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.950483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.950483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"107'2","reported_seq":44,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.325274+0000","last_change":"2024-02-13T00:41:01.611411+0000","last_active":"2024-02-13T00:41:33.325274+0000","last_peered":"2024-02-13T00:41:33.325274+0000","last_clean":"2024-02-13T00:41:33.325274+0000","last_became_active":"2024-02-13T00:41:01.611115+0000","last_became_peered":"2024-02-13T00:41:01.611115+0000","last_unstale":"2024-02-13T00:41:33.325274+0000","last_undegraded":"2024-02-13T00:41:33.325274+0000","last_fullsized":"2024-02-13T00:41:33.325274+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.d","version":"107'1","reported_seq":43,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.335037+0000","last_change":"2024-02-13T00:41:01.638974+0000","last_active":"2024-02-13T00:41:33.335037+0000","last_peered":"2024-02-13T00:41:33.335037+0000","last_clean":"2024-02-13T00:41:33.335037+0000","last_became_active":"2024-02-13T00:41:01.638148+0000","last_became_peered":"2024-02-13T00:41:01.638148+0000","last_unstale":"2024-02-13T00:41:33.335037+0000","last_undegraded":"2024-02-13T00:41:33.335037+0000","last_fullsized":"2024-02-13T00:41:33.335037+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333932+0000","last_change":"2024-02-13T00:41:01.627093+0000","last_active":"2024-02-13T00:41:33.333932+0000","last_peered":"2024-02-13T00:41:33.333932+0000","last_clean":"2024-02-13T00:41:33.333932+0000","last_became_active":"2024-02-13T00:41:01.626019+0000","last_became_peered":"2024-02-13T00:41:01.626019+0000","last_unstale":"2024-02-13T00:41:33.333932+0000","last_undegraded":"2024-02-13T00:41:33.333932+0000","last_fullsized":"2024-02-13T00:41:33.333932+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":119,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.450596+0000","last_change":"2024-02-13T00:41:31.648350+0000","last_active":"2024-02-13T00:41:33.450596+0000","last_peered":"2024-02-13T00:41:33.450596+0000","last_clean":"2024-02-13T00:41:33.450596+0000","last_became_active":"2024-02-13T00:40:35.857274+0000","last_became_peered":"2024-02-13T00:40:35.857274+0000","last_unstale":"2024-02-13T00:41:33.450596+0000","last_undegraded":"2024-02-13T00:41:33.450596+0000","last_fullsized":"2024-02-13T00:41:33.450596+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:41:31.648217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:31.648217+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.336762+0000","last_change":"2024-02-13T00:40:43.197233+0000","last_active":"2024-02-13T00:41:33.336762+0000","last_peered":"2024-02-13T00:41:33.336762+0000","last_clean":"2024-02-13T00:41:33.336762+0000","last_became_active":"2024-02-13T00:37:41.222506+0000","last_became_peered":"2024-02-13T00:37:41.222506+0000","last_unstale":"2024-02-13T00:41:33.336762+0000","last_undegraded":"2024-02-13T00:41:33.336762+0000","last_fullsized":"2024-02-13T00:41:33.336762+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:43.197157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:43.197157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":96,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.325423+0000","last_change":"2024-02-13T00:40:14.697069+0000","last_active":"2024-02-13T00:41:33.325423+0000","last_peered":"2024-02-13T00:41:33.325423+0000","last_clean":"2024-02-13T00:41:33.325423+0000","last_became_active":"2024-02-13T00:37:45.243961+0000","last_became_peered":"2024-02-13T00:37:45.243961+0000","last_unstale":"2024-02-13T00:41:33.325423+0000","last_undegraded":"2024-02-13T00:41:33.325423+0000","last_fullsized":"2024-02-13T00:41:33.325423+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:14.696991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:14.696991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.339957+0000","last_change":"2024-02-13T00:39:22.338139+0000","last_active":"2024-02-13T00:41:33.339957+0000","last_peered":"2024-02-13T00:41:33.339957+0000","last_clean":"2024-02-13T00:41:33.339957+0000","last_became_active":"2024-02-13T00:37:47.259688+0000","last_became_peered":"2024-02-13T00:37:47.259688+0000","last_unstale":"2024-02-13T00:41:33.339957+0000","last_undegraded":"2024-02-13T00:41:33.339957+0000","last_fullsized":"2024-02-13T00:41:33.339957+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.338042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.325337+0000","last_change":"2024-02-13T00:41:01.620536+0000","last_active":"2024-02-13T00:41:33.325337+0000","last_peered":"2024-02-13T00:41:33.325337+0000","last_clean":"2024-02-13T00:41:33.325337+0000","last_became_active":"2024-02-13T00:41:01.620258+0000","last_became_peered":"2024-02-13T00:41:01.620258+0000","last_unstale":"2024-02-13T00:41:33.325337+0000","last_undegraded":"2024-02-13T00:41:33.325337+0000","last_fullsized":"2024-02-13T00:41:33.325337+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.a","version":"107'2","reported_seq":45,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330302+0000","last_change":"2024-02-13T00:41:01.632973+0000","last_active":"2024-02-13T00:41:33.330302+0000","last_peered":"2024-02-13T00:41:33.330302+0000","last_clean":"2024-02-13T00:41:33.330302+0000","last_became_active":"2024-02-13T00:41:01.629461+0000","last_became_peered":"2024-02-13T00:41:01.629461+0000","last_unstale":"2024-02-13T00:41:33.330302+0000","last_undegraded":"2024-02-13T00:41:33.330302+0000","last_fullsized":"2024-02-13T00:41:33.330302+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.325798+0000","last_change":"2024-02-13T00:41:01.575407+0000","last_active":"2024-02-13T00:41:33.325798+0000","last_peered":"2024-02-13T00:41:33.325798+0000","last_clean":"2024-02-13T00:41:33.325798+0000","last_became_active":"2024-02-13T00:41:01.575117+0000","last_became_peered":"2024-02-13T00:41:01.575117+0000","last_unstale":"2024-02-13T00:41:33.325798+0000","last_undegraded":"2024-02-13T00:41:33.325798+0000","last_fullsized":"2024-02-13T00:41:33.325798+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"85'14","reported_seq":115,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.450958+0000","last_change":"2024-02-13T00:40:35.857071+0000","last_active":"2024-02-13T00:41:33.450958+0000","last_peered":"2024-02-13T00:41:33.450958+0000","last_clean":"2024-02-13T00:41:33.450958+0000","last_became_active":"2024-02-13T00:40:35.856715+0000","last_became_peered":"2024-02-13T00:40:35.856715+0000","last_unstale":"2024-02-13T00:41:33.450958+0000","last_undegraded":"2024-02-13T00:41:33.450958+0000","last_fullsized":"2024-02-13T00:41:33.450958+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:39:42.156267+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:39:42.156267+0000","last_clean_scrub_stamp":"2024-02-13T00:39:42.156267+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":95,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.339500+0000","last_change":"2024-02-13T00:39:21.165313+0000","last_active":"2024-02-13T00:41:33.339500+0000","last_peered":"2024-02-13T00:41:33.339500+0000","last_clean":"2024-02-13T00:41:33.339500+0000","last_became_active":"2024-02-13T00:37:41.221025+0000","last_became_peered":"2024-02-13T00:37:41.221025+0000","last_unstale":"2024-02-13T00:41:33.339500+0000","last_undegraded":"2024-02-13T00:41:33.339500+0000","last_fullsized":"2024-02-13T00:41:33.339500+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.165246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":96,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330536+0000","last_change":"2024-02-13T00:40:07.088136+0000","last_active":"2024-02-13T00:41:33.330536+0000","last_peered":"2024-02-13T00:41:33.330536+0000","last_clean":"2024-02-13T00:41:33.330536+0000","last_became_active":"2024-02-13T00:37:45.246184+0000","last_became_peered":"2024-02-13T00:37:45.246184+0000","last_unstale":"2024-02-13T00:41:33.330536+0000","last_undegraded":"2024-02-13T00:41:33.330536+0000","last_fullsized":"2024-02-13T00:41:33.330536+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:07.088053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:07.088053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.323409+0000","last_change":"2024-02-13T00:40:13.678873+0000","last_active":"2024-02-13T00:41:33.323409+0000","last_peered":"2024-02-13T00:41:33.323409+0000","last_clean":"2024-02-13T00:41:33.323409+0000","last_became_active":"2024-02-13T00:37:47.260025+0000","last_became_peered":"2024-02-13T00:37:47.260025+0000","last_unstale":"2024-02-13T00:41:33.323409+0000","last_undegraded":"2024-02-13T00:41:33.323409+0000","last_fullsized":"2024-02-13T00:41:33.323409+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:13.678807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:40:13.678807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.d","version":"108'5","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.455117+0000","last_change":"2024-02-13T00:41:01.688873+0000","last_active":"2024-02-13T00:41:33.455117+0000","last_peered":"2024-02-13T00:41:33.455117+0000","last_clean":"2024-02-13T00:41:33.455117+0000","last_became_active":"2024-02-13T00:41:01.688580+0000","last_became_peered":"2024-02-13T00:41:01.688580+0000","last_unstale":"2024-02-13T00:41:33.455117+0000","last_undegraded":"2024-02-13T00:41:33.455117+0000","last_fullsized":"2024-02-13T00:41:33.455117+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.b","version":"109'4","reported_seq":46,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.330814+0000","last_change":"2024-02-13T00:41:01.599162+0000","last_active":"2024-02-13T00:41:33.330814+0000","last_peered":"2024-02-13T00:41:33.330814+0000","last_clean":"2024-02-13T00:41:33.330814+0000","last_became_active":"2024-02-13T00:41:01.598565+0000","last_became_peered":"2024-02-13T00:41:01.598565+0000","last_unstale":"2024-02-13T00:41:33.330814+0000","last_undegraded":"2024-02-13T00:41:33.330814+0000","last_fullsized":"2024-02-13T00:41:33.330814+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":27325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":41,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.319792+0000","last_change":"2024-02-13T00:41:01.662500+0000","last_active":"2024-02-13T00:41:33.319792+0000","last_peered":"2024-02-13T00:41:33.319792+0000","last_clean":"2024-02-13T00:41:33.319792+0000","last_became_active":"2024-02-13T00:41:01.660978+0000","last_became_peered":"2024-02-13T00:41:01.660978+0000","last_unstale":"2024-02-13T00:41:33.319792+0000","last_undegraded":"2024-02-13T00:41:33.319792+0000","last_fullsized":"2024-02-13T00:41:33.319792+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"90'20","reported_seq":126,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.319687+0000","last_change":"2024-02-13T00:40:35.847653+0000","last_active":"2024-02-13T00:41:33.319687+0000","last_peered":"2024-02-13T00:41:33.319687+0000","last_clean":"2024-02-13T00:41:33.319687+0000","last_became_active":"2024-02-13T00:40:35.847203+0000","last_became_peered":"2024-02-13T00:40:35.847203+0000","last_unstale":"2024-02-13T00:41:33.319687+0000","last_undegraded":"2024-02-13T00:41:33.319687+0000","last_fullsized":"2024-02-13T00:41:33.319687+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:40:13.144451+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":95,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.458827+0000","last_change":"2024-02-13T00:39:40.280385+0000","last_active":"2024-02-13T00:41:33.458827+0000","last_peered":"2024-02-13T00:41:33.458827+0000","last_clean":"2024-02-13T00:41:33.458827+0000","last_became_active":"2024-02-13T00:37:41.216896+0000","last_became_peered":"2024-02-13T00:37:41.216896+0000","last_unstale":"2024-02-13T00:41:33.458827+0000","last_undegraded":"2024-02-13T00:41:33.458827+0000","last_fullsized":"2024-02-13T00:41:33.458827+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:39:40.280305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":97,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-13T00:41:34.693324+0000","last_change":"2024-02-13T00:41:34.693324+0000","last_active":"2024-02-13T00:41:34.693324+0000","last_peered":"2024-02-13T00:41:34.693324+0000","last_clean":"2024-02-13T00:41:34.693324+0000","last_became_active":"2024-02-13T00:37:45.249914+0000","last_became_peered":"2024-02-13T00:37:45.249914+0000","last_unstale":"2024-02-13T00:41:34.693324+0000","last_undegraded":"2024-02-13T00:41:34.693324+0000","last_fullsized":"2024-02-13T00:41:34.693324+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.693214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.693214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":55,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.315998+0000","last_change":"2024-02-13T00:40:35.846021+0000","last_active":"2024-02-13T00:41:33.315998+0000","last_peered":"2024-02-13T00:41:33.315998+0000","last_clean":"2024-02-13T00:41:33.315998+0000","last_became_active":"2024-02-13T00:40:35.845664+0000","last_became_peered":"2024-02-13T00:40:35.845664+0000","last_unstale":"2024-02-13T00:41:33.315998+0000","last_undegraded":"2024-02-13T00:41:33.315998+0000","last_fullsized":"2024-02-13T00:41:33.315998+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:37.210406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:37.210406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333562+0000","last_change":"2024-02-13T00:41:01.747460+0000","last_active":"2024-02-13T00:41:33.333562+0000","last_peered":"2024-02-13T00:41:33.333562+0000","last_clean":"2024-02-13T00:41:33.333562+0000","last_became_active":"2024-02-13T00:41:01.743088+0000","last_became_peered":"2024-02-13T00:41:01.743088+0000","last_unstale":"2024-02-13T00:41:33.333562+0000","last_undegraded":"2024-02-13T00:41:33.333562+0000","last_fullsized":"2024-02-13T00:41:33.333562+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":47,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.319770+0000","last_change":"2024-02-13T00:41:09.372268+0000","last_active":"2024-02-13T00:41:33.319770+0000","last_peered":"2024-02-13T00:41:33.319770+0000","last_clean":"2024-02-13T00:41:33.319770+0000","last_became_active":"2024-02-13T00:41:01.659916+0000","last_became_peered":"2024-02-13T00:41:01.659916+0000","last_unstale":"2024-02-13T00:41:33.319770+0000","last_undegraded":"2024-02-13T00:41:33.319770+0000","last_fullsized":"2024-02-13T00:41:33.319770+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:09.372195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:09.372195+0000","last_clean_scrub_stamp":"2024-02-13T00:41:09.372195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":54,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.429548+0000","last_change":"2024-02-13T00:41:01.732312+0000","last_active":"2024-02-13T00:41:33.429548+0000","last_peered":"2024-02-13T00:41:33.429548+0000","last_clean":"2024-02-13T00:41:33.429548+0000","last_became_active":"2024-02-13T00:41:01.728806+0000","last_became_peered":"2024-02-13T00:41:01.728806+0000","last_unstale":"2024-02-13T00:41:33.429548+0000","last_undegraded":"2024-02-13T00:41:33.429548+0000","last_fullsized":"2024-02-13T00:41:33.429548+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"86'50","reported_seq":194,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.446710+0000","last_change":"2024-02-13T00:39:53.119049+0000","last_active":"2024-02-13T00:41:33.446710+0000","last_peered":"2024-02-13T00:41:33.446710+0000","last_clean":"2024-02-13T00:41:33.446710+0000","last_became_active":"2024-02-13T00:37:43.228728+0000","last_became_peered":"2024-02-13T00:37:43.228728+0000","last_unstale":"2024-02-13T00:41:33.446710+0000","last_undegraded":"2024-02-13T00:41:33.446710+0000","last_fullsized":"2024-02-13T00:41:33.446710+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:39:53.118989+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":95,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.322285+0000","last_change":"2024-02-13T00:39:32.313930+0000","last_active":"2024-02-13T00:41:33.322285+0000","last_peered":"2024-02-13T00:41:33.322285+0000","last_clean":"2024-02-13T00:41:33.322285+0000","last_became_active":"2024-02-13T00:37:41.220583+0000","last_became_peered":"2024-02-13T00:37:41.220583+0000","last_unstale":"2024-02-13T00:41:33.322285+0000","last_undegraded":"2024-02-13T00:41:33.322285+0000","last_fullsized":"2024-02-13T00:41:33.322285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.313841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":55,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.319603+0000","last_change":"2024-02-13T00:40:35.847474+0000","last_active":"2024-02-13T00:41:33.319603+0000","last_peered":"2024-02-13T00:41:33.319603+0000","last_clean":"2024-02-13T00:41:33.319603+0000","last_became_active":"2024-02-13T00:40:35.846778+0000","last_became_peered":"2024-02-13T00:40:35.846778+0000","last_unstale":"2024-02-13T00:41:33.319603+0000","last_undegraded":"2024-02-13T00:41:33.319603+0000","last_fullsized":"2024-02-13T00:41:33.319603+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.273346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.451481+0000","last_change":"2024-02-13T00:40:54.059354+0000","last_active":"2024-02-13T00:41:33.451481+0000","last_peered":"2024-02-13T00:41:33.451481+0000","last_clean":"2024-02-13T00:41:33.451481+0000","last_became_active":"2024-02-13T00:37:47.552890+0000","last_became_peered":"2024-02-13T00:37:47.552890+0000","last_unstale":"2024-02-13T00:41:33.451481+0000","last_undegraded":"2024-02-13T00:41:33.451481+0000","last_fullsized":"2024-02-13T00:41:33.451481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:54.059274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:54.059274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":42,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.333148+0000","last_change":"2024-02-13T00:41:01.573580+0000","last_active":"2024-02-13T00:41:33.333148+0000","last_peered":"2024-02-13T00:41:33.333148+0000","last_clean":"2024-02-13T00:41:33.333148+0000","last_became_active":"2024-02-13T00:41:01.573417+0000","last_became_peered":"2024-02-13T00:41:01.573417+0000","last_unstale":"2024-02-13T00:41:33.333148+0000","last_undegraded":"2024-02-13T00:41:33.333148+0000","last_fullsized":"2024-02-13T00:41:33.333148+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"107'5","reported_seq":52,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-13T00:41:33.331801+0000","last_change":"2024-02-13T00:41:16.227063+0000","last_active":"2024-02-13T00:41:33.331801+0000","last_peered":"2024-02-13T00:41:33.331801+0000","last_clean":"2024-02-13T00:41:33.331801+0000","last_became_active":"2024-02-13T00:41:01.530359+0000","last_became_peered":"2024-02-13T00:41:01.530359+0000","last_unstale":"2024-02-13T00:41:33.331801+0000","last_undegraded":"2024-02-13T00:41:33.331801+0000","last_fullsized":"2024-02-13T00:41:33.331801+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'5","last_scrub_stamp":"2024-02-13T00:41:16.226998+0000","last_deep_scrub":"107'5","last_deep_scrub_stamp":"2024-02-13T00:41:16.226998+0000","last_clean_scrub_stamp":"2024-02-13T00:41:16.226998+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":88,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":87,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":86,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":85,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":66,"num_pg":32,"stat_sum":{"num_bytes":1664,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":4,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":4992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":96,"acting":96,"num_store_stats":8},{"poolid":45,"num_pg":32,"stat_sum":{"num_bytes":651,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":12,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":36,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":151001778,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":1011,"num_read_kb":49161,"num_write":1044,"num_write_kb":152981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":537174016,"data_stored":536891422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1055,"ondisk_log_size":1055,"up":96,"acting":96,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":78,"num_pg":8,"stat_sum":{"num_bytes":512,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":24,"acting":24,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1073,"num_read_kb":798,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":669,"ondisk_log_size":669,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":227,"num_read_kb":222,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266737,"num_pgs":189,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":409440,"kb_used_data":100056,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93335712,"statfs":{"total":95995035648,"available":95575769088,"internally_reserved":0,"allocated":102457344,"data_stored":101601650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.502,"15min":0.502},"min":{"1min":0.26400000000000001,"5min":0.25,"15min":0.25},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":3.6240000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":13.347},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.57199999999999995,"15min":0.60599999999999998},"min":{"1min":0.379,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61699999999999999,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":7.4089999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.1619999999999999,"5min":3.1619999999999999,"15min":3.1619999999999999},"last":14.359},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":7.6769999999999996}]},{"osd":3,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.097,"5min":3.097,"15min":3.097},"last":12.997},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.57999999999999996,"15min":0.54800000000000004},"min":{"1min":0.40699999999999997,"5min":0.318,"15min":0.318},"max":{"1min":3.7690000000000001,"5min":3.7690000000000001,"15min":3.7690000000000001},"last":7.6040000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.46300000000000002,"15min":0.47099999999999997},"min":{"1min":0.29199999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66000000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.45400000000000001,"15min":0.46400000000000002},"min":{"1min":0.32800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.318}]},{"osd":5,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.47099999999999997,"15min":0.47499999999999998},"min":{"1min":0.315,"5min":0.255,"15min":0.255},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.48099999999999998,"15min":0.47599999999999998},"min":{"1min":0.34200000000000003,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68600000000000005,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.34799999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.502,"15min":0.5},"min":{"1min":0.38400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70199999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.372},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.52600000000000002},"min":{"1min":0.35899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.35799999999999998}]}]},{"osd":6,"up_from":45,"seq":193273528375,"num_pgs":155,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":335636,"kb_used_data":26252,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93409516,"statfs":{"total":95995035648,"available":95651344384,"internally_reserved":0,"allocated":26882048,"data_stored":26098329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51200000000000001,"15min":0.53700000000000003},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.67100000000000004,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.49199999999999999,"15min":0.47599999999999998},"min":{"1min":0.35799999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.311}]},{"osd":1,"last update":"Tue Feb 13 00:41:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6269999999999998,"5min":1.1259999999999999,"15min":0.71299999999999997},"min":{"1min":0.311,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":54.002000000000002,"5min":54.002000000000002,"15min":54.002000000000002},"last":2.0609999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":0.64700000000000002,"15min":0.56299999999999994},"min":{"1min":0.33100000000000002,"5min":0.315,"15min":0.315},"max":{"1min":5.891,"5min":5.891,"15min":5.891},"last":0.36599999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:41:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":0.63500000000000001,"15min":0.54600000000000004},"min":{"1min":0.39100000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":8.0280000000000005,"5min":8.0280000000000005,"15min":8.0280000000000005},"last":1.008},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.54800000000000004,"15min":0.50900000000000001},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":4.1680000000000001,"5min":4.1680000000000001,"15min":4.1680000000000001},"last":6.0510000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.59599999999999997,"15min":0.56399999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.52,"5min":3.52,"15min":3.52},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.7430000000000001,"5min":0.78700000000000003,"15min":0.624},"min":{"1min":0.35899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":19.081,"5min":19.081,"15min":19.081},"last":1.194}]},{"osd":4,"last update":"Tue Feb 13 00:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42499999999999999,"15min":0.40400000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.42899999999999999},"min":{"1min":0.29699999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69799999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.35599999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.432,"15min":0.39700000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.48699999999999999,"15min":0.45500000000000002},"min":{"1min":0.34599999999999997,"5min":0.312,"15min":0.312},"max":{"1min":0.70099999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.32600000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.41699999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70299999999999996,"5min":1.173,"15min":1.173},"last":0.374},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.42299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.749,"5min":1.262,"15min":1.262},"last":0.38100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":164,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360308,"kb_used_data":50924,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93384844,"statfs":{"total":95995035648,"available":95626080256,"internally_reserved":0,"allocated":52146176,"data_stored":51258044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:41:20 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":0.82399999999999995,"15min":0.48099999999999998},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":41.954000000000001,"5min":41.954000000000001,"15min":41.954000000000001},"last":61.856000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.443,"15min":0.34699999999999998},"min":{"1min":0.23400000000000001,"5min":0.215,"15min":0.21199999999999999},"max":{"1min":4.1790000000000003,"5min":4.1790000000000003,"15min":4.1790000000000003},"last":61.908999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.374,"15min":0.372},"min":{"1min":0.218,"5min":0.215,"15min":0.215},"max":{"1min":0.56799999999999995,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":28.623000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.38700000000000001},"min":{"1min":0.25700000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":29.693000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:41:18 2024","interfaces":[{"interface":"back","average":{"1min":2.6869999999999998,"5min":0.86399999999999999,"15min":0.58899999999999997},"min":{"1min":0.254,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":32.200000000000003,"5min":32.200000000000003,"15min":32.200000000000003},"last":51.984000000000002},{"interface":"front","average":{"1min":1.3480000000000001,"5min":0.60499999999999998,"15min":0.51700000000000002},"min":{"1min":0.26000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":12.355,"5min":12.355,"15min":12.355},"last":43.237000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:41:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1739999999999999,"5min":1.0820000000000001,"15min":0.74099999999999999},"min":{"1min":0.435,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":16.547000000000001,"5min":16.547000000000001,"15min":16.547000000000001},"last":14.616},{"interface":"front","average":{"1min":1.9159999999999999,"5min":0.84999999999999998,"15min":0.65700000000000003},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":14.731999999999999,"5min":14.731999999999999,"15min":14.731999999999999},"last":14.731999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":26.561},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58599999999999997,"15min":0.58499999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.95799999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":14.648999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:41:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":0.73699999999999999,"15min":0.64600000000000002},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":12.403,"5min":12.403,"15min":12.403},"last":14.772},{"interface":"front","average":{"1min":2.5760000000000001,"5min":0.997,"15min":0.74099999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":26.396999999999998,"5min":26.396999999999998,"15min":26.396999999999998},"last":26.600000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.59499999999999997},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.78100000000000003,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":14.794},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.41799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82899999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":26.693999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":154,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":335648,"kb_used_data":26264,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93409504,"statfs":{"total":95995035648,"available":95651332096,"internally_reserved":0,"allocated":26894336,"data_stored":26081507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.38400000000000001,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.189,"15min":0.189},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.022},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.46400000000000002,"15min":0.374},"min":{"1min":0.187,"5min":0.187,"15min":0.17599999999999999},"max":{"1min":4.7409999999999997,"5min":4.7409999999999997,"15min":4.7409999999999997},"last":1.6579999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.42299999999999999,"15min":0.40400000000000003},"min":{"1min":0.245,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.40000000000000002,"15min":0.38400000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.48499999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.29599999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.749},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.45900000000000002,"15min":0.47099999999999997},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.94899999999999995}]},{"osd":4,"last update":"Tue Feb 13 00:41:33 2024","interfaces":[{"interface":"back","average":{"1min":2.9740000000000002,"5min":1.024,"15min":0.68400000000000005},"min":{"1min":0.35499999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":24.381,"5min":24.381,"15min":24.381},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.61299999999999999,"15min":0.54400000000000004},"min":{"1min":0.36099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.7069999999999999,"5min":3.7069999999999999,"15min":3.7069999999999999},"last":0.78900000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:40:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56000000000000005,"15min":0.57399999999999995},"min":{"1min":0.34100000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.55300000000000005,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":1.002}]},{"osd":6,"last update":"Tue Feb 13 00:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.55300000000000005,"15min":0.51900000000000002},"min":{"1min":0.41199999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.60099999999999998,"15min":0.55700000000000005},"min":{"1min":0.42299999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":4.681,"5min":4.681,"15min":4.681},"last":0.89000000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.55600000000000005,"15min":0.55100000000000005},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.622,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.439,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.90400000000000003,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.93000000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":156,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":384820,"kb_used_data":75436,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93360332,"statfs":{"total":95995035648,"available":95600979968,"internally_reserved":0,"allocated":77246464,"data_stored":76435628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35899999999999999,"15min":0.35499999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":2.694},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.374,"15min":0.34999999999999998},"min":{"1min":0.23200000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.76100000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.624}]},{"osd":1,"last update":"Tue Feb 13 00:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.36799999999999999,"15min":0.35199999999999998},"min":{"1min":0.26000000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":1.1279999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38700000000000001,"15min":0.375},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":2.6560000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:41:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41299999999999998,"15min":0.41899999999999998},"min":{"1min":0.29899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.74399999999999999,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.42899999999999999,"15min":0.441},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.85199999999999998,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":2.7850000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:41:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.59999999999999998,"15min":0.57999999999999996},"min":{"1min":0.377,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.286,"5min":2.286,"15min":2.286},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.65000000000000002,"15min":0.61299999999999999},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.8959999999999999,"5min":3.8959999999999999,"15min":3.8959999999999999},"last":2.7189999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:40:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.56000000000000005},"min":{"1min":0.42399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.90500000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57399999999999995,"15min":0.59199999999999997},"min":{"1min":0.46899999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.86499999999999999,"5min":0.872,"15min":0.872},"last":2.7629999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:41:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55700000000000005,"15min":0.57599999999999996},"min":{"1min":0.35899999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.878,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":2.742},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78200000000000003,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.65900000000000003}]},{"osd":7,"last update":"Tue Feb 13 00:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.58799999999999997},"min":{"1min":0.379,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68300000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.433,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89400000000000002,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":1.0660000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182469,"num_pgs":153,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360260,"kb_used_data":50876,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93384892,"statfs":{"total":95995035648,"available":95626129408,"internally_reserved":0,"allocated":52097024,"data_stored":51268224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.377},"min":{"1min":0.218,"5min":0.191,"15min":0.191},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":7.2039999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.38600000000000001,"15min":0.40600000000000003},"min":{"1min":0.23300000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":1.0509999999999999}]},{"osd":1,"last update":"Tue Feb 13 00:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.39100000000000001,"15min":0.40000000000000002},"min":{"1min":0.27700000000000002,"5min":0.217,"15min":0.217},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.44700000000000001,"15min":0.41799999999999998},"min":{"1min":0.255,"5min":0.183,"15min":0.183},"max":{"1min":5.7039999999999997,"5min":5.7039999999999997,"15min":5.7039999999999997},"last":1.091}]},{"osd":2,"last update":"Tue Feb 13 00:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.47299999999999998,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":5.6760000000000002},"last":4.9119999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.40600000000000003,"15min":0.40799999999999997},"min":{"1min":0.28499999999999998,"5min":0.247,"15min":0.247},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":7.1769999999999996}]},{"osd":4,"last update":"Tue Feb 13 00:41:34 2024","interfaces":[{"interface":"back","average":{"1min":4.0640000000000001,"5min":1.2410000000000001,"15min":0.77600000000000002},"min":{"1min":0.44800000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":35.156999999999996,"5min":35.156999999999996,"15min":35.156999999999996},"last":7.1210000000000004},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.629,"15min":0.56999999999999995},"min":{"1min":0.436,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.9420000000000002,"5min":3.9420000000000002,"15min":3.9420000000000002},"last":1.0289999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:40:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.54300000000000004,"15min":0.51100000000000001},"min":{"1min":0.38300000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":1.1160000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.54000000000000004,"15min":0.52200000000000002},"min":{"1min":0.46100000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81399999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":7.1500000000000004}]},{"osd":6,"last update":"Tue Feb 13 00:41:18 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":0.75900000000000001,"15min":0.59099999999999997},"min":{"1min":0.46500000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":14.438000000000001,"5min":14.438000000000001,"15min":14.438000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":2.5179999999999998,"5min":0.94099999999999995,"15min":0.67600000000000005},"min":{"1min":0.42599999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":35.280000000000001,"5min":35.280000000000001,"15min":35.280000000000001},"last":7.0899999999999999}]},{"osd":7,"last update":"Tue Feb 13 00:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.55300000000000005,"15min":0.54700000000000004},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":7.0300000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.435,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81899999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.99299999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986241,"num_pgs":146,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":409460,"kb_used_data":100076,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93335692,"statfs":{"total":95995035648,"available":95575748608,"internally_reserved":0,"allocated":102477824,"data_stored":101607921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.40200000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65700000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":9.3399999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.52500000000000002,"15min":0.54100000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70299999999999996,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":1.7430000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72499999999999998,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":7.7759999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.40200000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.70999999999999996,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":1.8939999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.374,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71199999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51900000000000002,"15min":0.52600000000000002},"min":{"1min":0.36699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.66700000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":1.7250000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.66900000000000004,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.5389999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52700000000000002,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69599999999999995,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":3.093}]},{"osd":5,"last update":"Tue Feb 13 00:40:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.39200000000000002,"15min":0.35799999999999998},"min":{"1min":0.317,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.41699999999999998,"15min":0.376},"min":{"1min":0.36199999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60799999999999998,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.35499999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:41:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.439,"15min":0.40699999999999997},"min":{"1min":0.34100000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.45700000000000002,"15min":0.41899999999999998},"min":{"1min":0.35799999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.40000000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.48099999999999998,"15min":0.46600000000000003},"min":{"1min":0.42099999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.47099999999999997,"15min":0.45700000000000002},"min":{"1min":0.40500000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.39200000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724604,"num_pgs":146,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":335652,"kb_used_data":26268,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93409500,"statfs":{"total":95995035648,"available":95651328000,"internally_reserved":0,"allocated":26898432,"data_stored":26092286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:40:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.49199999999999999,"15min":0.48599999999999999},"min":{"1min":0.318,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70599999999999996,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":11.134},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53400000000000003,"15min":0.51300000000000001},"min":{"1min":0.35399999999999998,"5min":0.309,"15min":0.309},"max":{"1min":1.0640000000000001,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.5}]},{"osd":1,"last update":"Tue Feb 13 00:40:56 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.504,"15min":0.498},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.66100000000000003,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":1.851},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52900000000000003,"15min":0.503},"min":{"1min":0.30499999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.69399999999999995,"5min":4.0839999999999996,"15min":4.0839999999999996},"last":1.9359999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:40:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51600000000000001,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.94199999999999995,"5min":1.099,"15min":1.099},"last":30.303000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.51700000000000002,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.70999999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":30.228000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:40:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.58299999999999996,"15min":0.52800000000000002},"min":{"1min":0.36799999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.5640000000000001,"5min":5.5640000000000001,"15min":5.5640000000000001},"last":54.843000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.52800000000000002,"15min":0.50800000000000001},"min":{"1min":0.41399999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77600000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":54.848999999999997}]},{"osd":4,"last update":"Tue Feb 13 00:40:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.442,"15min":0.374},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63800000000000001,"5min":4.1210000000000004,"15min":4.1210000000000004},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.42899999999999999},"min":{"1min":0.39300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.71699999999999997,"5min":5.6950000000000003,"15min":5.6950000000000003},"last":0.317}]},{"osd":6,"last update":"Tue Feb 13 00:41:15 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.53800000000000003,"15min":0.46000000000000002},"min":{"1min":0.30399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":8.1020000000000003,"5min":8.1020000000000003,"15min":8.1020000000000003},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.55800000000000005,"15min":0.47799999999999998},"min":{"1min":0.307,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.7389999999999999,"5min":7.7389999999999999,"15min":7.7389999999999999},"last":0.26100000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.69599999999999995,"15min":0.78400000000000003},"min":{"1min":0.377,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69499999999999995,"5min":7.5519999999999996,"15min":7.5519999999999996},"last":0.254},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55900000000000005,"15min":0.57499999999999996},"min":{"1min":0.435,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.27000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25168657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50384896,"data_stored":50334598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":100708352,"data_stored":100664444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58740736,"data_stored":58723095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":125878272,"data_stored":125833005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25182208,"data_stored":25165858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25165863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":125898752,"data_stored":125835902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":11,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":11,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":78,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":78,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":78,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":78,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":78,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":78,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":78,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":85,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":85,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":85,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":85,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":85,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":85,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":85,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:41:37.656 INFO:tasks.thrashosds.thrasher:Setting 20.c to [5, 6, 4] 2024-02-13T00:41:37.656 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '20.c', '5', '6', '4'] 2024-02-13T00:41:37.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pg-upmap 20.c 5 6 4 2024-02-13T00:41:37.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:37.657 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:37.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:37.758 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:37.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:36.268456+0000 mgr.y (mgr.14544) 178 : cluster [DBG] pgmap v181: 428 pgs: 32 creating+peering, 1 active+clean+snaptrim, 64 unknown, 331 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2024-02-13T00:41:37.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:36.394588+0000 mon.a (mon.0) 1107 : audit [INF] from='client.? 172.21.15.162:0/4080261016' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-83985-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:37.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:36.394731+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:37.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:36.394877+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-02-13T00:41:37.779 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:36.408218+0000 mon.b (mon.1) 148 : audit [INF] from='client.? 172.21.15.162:0/4108102799' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-83977-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:37.779 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:36.416676+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-83977-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:37.779 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:36.465495+0000 osd.5 (osd.5) 73 : cluster [DBG] 4.15 scrub starts 2024-02-13T00:41:37.779 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:36.470302+0000 mgr.y (mgr.14544) 179 : audit [DBG] from='client.38281 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:41:37.779 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:36.477030+0000 osd.5 (osd.5) 74 : cluster [DBG] 4.15 scrub ok 2024-02-13T00:41:37.779 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:36.521626+0000 mon.b (mon.1) 149 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:37.779 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:36.912235+0000 mon.a (mon.0) 1111 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:37.779 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:37.029873+0000 mon.b (mon.1) 150 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:37.779 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:36.268456+0000 mgr.y (mgr.14544) 178 : cluster [DBG] pgmap v181: 428 pgs: 32 creating+peering, 1 active+clean+snaptrim, 64 unknown, 331 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2024-02-13T00:41:37.779 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:36.394588+0000 mon.a (mon.0) 1107 : audit [INF] from='client.? 172.21.15.162:0/4080261016' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-83985-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:37.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:36.394731+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:37.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:36.394877+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-02-13T00:41:37.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:36.408218+0000 mon.b (mon.1) 148 : audit [INF] from='client.? 172.21.15.162:0/4108102799' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-83977-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:37.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:36.416676+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-83977-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:37.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:36.465495+0000 osd.5 (osd.5) 73 : cluster [DBG] 4.15 scrub starts 2024-02-13T00:41:37.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:36.470302+0000 mgr.y (mgr.14544) 179 : audit [DBG] from='client.38281 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:41:37.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:36.477030+0000 osd.5 (osd.5) 74 : cluster [DBG] 4.15 scrub ok 2024-02-13T00:41:37.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:36.521626+0000 mon.b (mon.1) 149 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:37.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:36.912235+0000 mon.a (mon.0) 1111 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:37.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:37.029873+0000 mon.b (mon.1) 150 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:37.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:36.268456+0000 mgr.y (mgr.14544) 178 : cluster [DBG] pgmap v181: 428 pgs: 32 creating+peering, 1 active+clean+snaptrim, 64 unknown, 331 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2024-02-13T00:41:37.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:36.394588+0000 mon.a (mon.0) 1107 : audit [INF] from='client.? 172.21.15.162:0/4080261016' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-83985-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:37.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:36.394731+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84065-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:37.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:36.394877+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-02-13T00:41:37.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:36.408218+0000 mon.b (mon.1) 148 : audit [INF] from='client.? 172.21.15.162:0/4108102799' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-83977-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:37.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:36.416676+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-83977-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:37.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:36.465495+0000 osd.5 (osd.5) 73 : cluster [DBG] 4.15 scrub starts 2024-02-13T00:41:37.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:36.470302+0000 mgr.y (mgr.14544) 179 : audit [DBG] from='client.38281 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:41:37.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:36.477030+0000 osd.5 (osd.5) 74 : cluster [DBG] 4.15 scrub ok 2024-02-13T00:41:37.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:36.521626+0000 mon.b (mon.1) 149 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:37.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:36.912235+0000 mon.a (mon.0) 1111 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:37.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:37.029873+0000 mon.b (mon.1) 150 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:37.784 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:37.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:41:37.802 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:37.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:37.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:37.932 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:37.950 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:37.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:38.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:38.113 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:38.251 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:38.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:38.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:38.355 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:38.381 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:38.400 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:38.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:38.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:38.540 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:38.559 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:41:38.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:38.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:41:38.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:38.688 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:41:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:41:38.689 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:36.756045+0000 osd.3 (osd.3) 91 : cluster [DBG] 20.0 deep-scrub starts 2024-02-13T00:41:38.689 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:36.766610+0000 osd.3 (osd.3) 92 : cluster [DBG] 20.0 deep-scrub ok 2024-02-13T00:41:38.689 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:37.449995+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:41:38.689 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:37.452874+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:41:38.689 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:37.477442+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-83977-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:38.689 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:37.477679+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:38.690 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:37.477741+0000 mon.a (mon.0) 1114 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-02-13T00:41:38.690 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:37.491894+0000 mon.c (mon.2) 132 : audit [INF] from='client.? 172.21.15.162:0/4066446528' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache", "force_nonempty":""}]: dispatch 2024-02-13T00:41:38.690 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:37.493174+0000 mon.b (mon.1) 151 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-11"}]: dispatch 2024-02-13T00:41:38.690 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:37.517273+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache", "force_nonempty":""}]: dispatch 2024-02-13T00:41:38.690 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:37.517496+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-11"}]: dispatch 2024-02-13T00:41:38.690 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:37.795973+0000 osd.1 (osd.1) 75 : cluster [DBG] 20.4 deep-scrub starts 2024-02-13T00:41:38.690 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:37.806137+0000 osd.1 (osd.1) 76 : cluster [DBG] 20.4 deep-scrub ok 2024-02-13T00:41:38.690 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:38.030775+0000 mon.b (mon.1) 152 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:38.691 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:38.710 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:38.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:38.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:38.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:36.756045+0000 osd.3 (osd.3) 91 : cluster [DBG] 20.0 deep-scrub starts 2024-02-13T00:41:38.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:36.766610+0000 osd.3 (osd.3) 92 : cluster [DBG] 20.0 deep-scrub ok 2024-02-13T00:41:38.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:37.449995+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:41:38.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:37.452874+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:41:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:37.477442+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-83977-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:37.477679+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:37.477741+0000 mon.a (mon.0) 1114 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-02-13T00:41:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:37.491894+0000 mon.c (mon.2) 132 : audit [INF] from='client.? 172.21.15.162:0/4066446528' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache", "force_nonempty":""}]: dispatch 2024-02-13T00:41:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:37.493174+0000 mon.b (mon.1) 151 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-11"}]: dispatch 2024-02-13T00:41:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:37.517273+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache", "force_nonempty":""}]: dispatch 2024-02-13T00:41:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:37.517496+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-11"}]: dispatch 2024-02-13T00:41:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:37.795973+0000 osd.1 (osd.1) 75 : cluster [DBG] 20.4 deep-scrub starts 2024-02-13T00:41:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:37.806137+0000 osd.1 (osd.1) 76 : cluster [DBG] 20.4 deep-scrub ok 2024-02-13T00:41:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:38.030775+0000 mon.b (mon.1) 152 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:38.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:36.756045+0000 osd.3 (osd.3) 91 : cluster [DBG] 20.0 deep-scrub starts 2024-02-13T00:41:38.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:36.766610+0000 osd.3 (osd.3) 92 : cluster [DBG] 20.0 deep-scrub ok 2024-02-13T00:41:38.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:37.449995+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:41:38.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:37.452874+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:41:38.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:37.477442+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-83977-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:38.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:37.477679+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:38.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:37.477741+0000 mon.a (mon.0) 1114 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-02-13T00:41:38.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:37.491894+0000 mon.c (mon.2) 132 : audit [INF] from='client.? 172.21.15.162:0/4066446528' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache", "force_nonempty":""}]: dispatch 2024-02-13T00:41:38.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:37.493174+0000 mon.b (mon.1) 151 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-11"}]: dispatch 2024-02-13T00:41:38.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:37.517273+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache", "force_nonempty":""}]: dispatch 2024-02-13T00:41:38.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:37.517496+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-11"}]: dispatch 2024-02-13T00:41:38.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:37.795973+0000 osd.1 (osd.1) 75 : cluster [DBG] 20.4 deep-scrub starts 2024-02-13T00:41:38.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:37.806137+0000 osd.1 (osd.1) 76 : cluster [DBG] 20.4 deep-scrub ok 2024-02-13T00:41:38.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:38.030775+0000 mon.b (mon.1) 152 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:38.840 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:38.858 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:38.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:38.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:38.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:41:38.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:39.044 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:39.327 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:39.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:39.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:39.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:39.398 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:39.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:39.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:38.271625+0000 mgr.y (mgr.14544) 180 : cluster [DBG] pgmap v184: 460 pgs: 32 creating+peering, 1 active+clean+snaptrim, 96 unknown, 331 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2024-02-13T00:41:39.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:38.456650+0000 osd.7 (osd.7) 76 : cluster [DBG] 20.2 deep-scrub starts 2024-02-13T00:41:39.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:38.468267+0000 osd.7 (osd.7) 77 : cluster [DBG] 20.2 deep-scrub ok 2024-02-13T00:41:39.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:38.590103+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache", "force_nonempty":""}]': finished 2024-02-13T00:41:39.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:38.590194+0000 mon.a (mon.0) 1118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-11"}]': finished 2024-02-13T00:41:39.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:38.590302+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-02-13T00:41:39.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:38.591947+0000 mon.b (mon.1) 153 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-11", "mode": "writeback"}]: dispatch 2024-02-13T00:41:39.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:38.593757+0000 mon.a (mon.0) 1120 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-11", "mode": "writeback"}]: dispatch 2024-02-13T00:41:39.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:38.596510+0000 mon.b (mon.1) 154 : audit [INF] from='client.? 172.21.15.162:0/3096836775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:39.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:38.598336+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:39.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:38.611088+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:39.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:38.620922+0000 mon.c (mon.2) 133 : audit [INF] from='client.? 172.21.15.162:0/4066446528' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84065-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:39.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:38.626200+0000 mon.a (mon.0) 1123 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84065-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:39.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:39.046785+0000 mon.b (mon.1) 155 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:39.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:39.350860+0000 osd.0 (osd.0) 59 : cluster [DBG] 20.3 deep-scrub starts 2024-02-13T00:41:39.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:39.359387+0000 osd.0 (osd.0) 60 : cluster [DBG] 20.3 deep-scrub ok 2024-02-13T00:41:39.616 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:39.942 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:39.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:39.948 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:38.271625+0000 mgr.y (mgr.14544) 180 : cluster [DBG] pgmap v184: 460 pgs: 32 creating+peering, 1 active+clean+snaptrim, 96 unknown, 331 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2024-02-13T00:41:39.948 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:38.456650+0000 osd.7 (osd.7) 76 : cluster [DBG] 20.2 deep-scrub starts 2024-02-13T00:41:39.948 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:38.468267+0000 osd.7 (osd.7) 77 : cluster [DBG] 20.2 deep-scrub ok 2024-02-13T00:41:39.948 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:38.590103+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache", "force_nonempty":""}]': finished 2024-02-13T00:41:39.949 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:38.590194+0000 mon.a (mon.0) 1118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-11"}]': finished 2024-02-13T00:41:39.949 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:38.590302+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-02-13T00:41:39.949 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:38.591947+0000 mon.b (mon.1) 153 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-11", "mode": "writeback"}]: dispatch 2024-02-13T00:41:39.949 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:38.593757+0000 mon.a (mon.0) 1120 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-11", "mode": "writeback"}]: dispatch 2024-02-13T00:41:39.949 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:38.596510+0000 mon.b (mon.1) 154 : audit [INF] from='client.? 172.21.15.162:0/3096836775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:39.949 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:38.598336+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:39.949 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:38.611088+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:39.949 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:38.620922+0000 mon.c (mon.2) 133 : audit [INF] from='client.? 172.21.15.162:0/4066446528' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84065-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:39.949 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:38.626200+0000 mon.a (mon.0) 1123 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84065-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:39.949 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:39.046785+0000 mon.b (mon.1) 155 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:39.949 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:39.350860+0000 osd.0 (osd.0) 59 : cluster [DBG] 20.3 deep-scrub starts 2024-02-13T00:41:39.949 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:39.359387+0000 osd.0 (osd.0) 60 : cluster [DBG] 20.3 deep-scrub ok 2024-02-13T00:41:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:38.271625+0000 mgr.y (mgr.14544) 180 : cluster [DBG] pgmap v184: 460 pgs: 32 creating+peering, 1 active+clean+snaptrim, 96 unknown, 331 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2024-02-13T00:41:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:38.456650+0000 osd.7 (osd.7) 76 : cluster [DBG] 20.2 deep-scrub starts 2024-02-13T00:41:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:38.468267+0000 osd.7 (osd.7) 77 : cluster [DBG] 20.2 deep-scrub ok 2024-02-13T00:41:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:38.590103+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache", "force_nonempty":""}]': finished 2024-02-13T00:41:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:38.590194+0000 mon.a (mon.0) 1118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-11"}]': finished 2024-02-13T00:41:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:38.590302+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-02-13T00:41:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:38.591947+0000 mon.b (mon.1) 153 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-11", "mode": "writeback"}]: dispatch 2024-02-13T00:41:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:38.593757+0000 mon.a (mon.0) 1120 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-11", "mode": "writeback"}]: dispatch 2024-02-13T00:41:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:38.596510+0000 mon.b (mon.1) 154 : audit [INF] from='client.? 172.21.15.162:0/3096836775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:40.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:38.598336+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:40.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:38.611088+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:40.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:38.620922+0000 mon.c (mon.2) 133 : audit [INF] from='client.? 172.21.15.162:0/4066446528' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84065-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:40.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:38.626200+0000 mon.a (mon.0) 1123 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84065-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:40.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:39.046785+0000 mon.b (mon.1) 155 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:40.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:39.350860+0000 osd.0 (osd.0) 59 : cluster [DBG] 20.3 deep-scrub starts 2024-02-13T00:41:40.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:39.359387+0000 osd.0 (osd.0) 60 : cluster [DBG] 20.3 deep-scrub ok 2024-02-13T00:41:40.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:40.247 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:40.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:40.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:40.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:40.581 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:41.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:39.520027+0000 mon.a (mon.0) 1124 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:41.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:39.546509+0000 mon.a (mon.0) 1125 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-11", "mode": "writeback"}]': finished 2024-02-13T00:41:41.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:39.546631+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:41.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:39.546703+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]': finished 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:39.546742+0000 mon.a (mon.0) 1128 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84065-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:39.546866+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:39.551166+0000 mon.b (mon.1) 156 : audit [INF] from='client.? 172.21.15.162:0/1389954781' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-83977-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:39.566058+0000 mon.a (mon.0) 1130 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:39.585108+0000 mon.c (mon.2) 134 : audit [INF] from='client.? 172.21.15.162:0/4066446528' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache"}]: dispatch 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:39.591992+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-83977-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:39.592497+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache"}]: dispatch 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:40.055877+0000 mon.b (mon.1) 157 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:40.317659+0000 mon.a (mon.0) 1133 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-84073-9"}]': finished 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:40.318809+0000 mon.a (mon.0) 1134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-83977-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:40.318926+0000 mon.a (mon.0) 1135 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache"}]': finished 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:40.319087+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-02-13T00:41:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:40.593663+0000 mon.b (mon.1) 158 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:39.520027+0000 mon.a (mon.0) 1124 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:39.546509+0000 mon.a (mon.0) 1125 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-11", "mode": "writeback"}]': finished 2024-02-13T00:41:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:39.546631+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:39.546703+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]': finished 2024-02-13T00:41:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:39.546742+0000 mon.a (mon.0) 1128 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84065-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:39.546866+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-02-13T00:41:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:39.551166+0000 mon.b (mon.1) 156 : audit [INF] from='client.? 172.21.15.162:0/1389954781' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-83977-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:39.566058+0000 mon.a (mon.0) 1130 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:39.585108+0000 mon.c (mon.2) 134 : audit [INF] from='client.? 172.21.15.162:0/4066446528' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache"}]: dispatch 2024-02-13T00:41:41.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:39.591992+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-83977-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:41.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:39.592497+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache"}]: dispatch 2024-02-13T00:41:41.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:40.055877+0000 mon.b (mon.1) 157 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:41.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:40.317659+0000 mon.a (mon.0) 1133 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-84073-9"}]': finished 2024-02-13T00:41:41.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:40.318809+0000 mon.a (mon.0) 1134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-83977-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:41.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:40.318926+0000 mon.a (mon.0) 1135 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache"}]': finished 2024-02-13T00:41:41.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:40.319087+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-02-13T00:41:41.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:40.593663+0000 mon.b (mon.1) 158 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:41.105 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:41.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:41.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:41.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:41.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:39.520027+0000 mon.a (mon.0) 1124 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:41.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:39.546509+0000 mon.a (mon.0) 1125 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-11", "mode": "writeback"}]': finished 2024-02-13T00:41:41.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:39.546631+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:41.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:39.546703+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-84073-9"}]': finished 2024-02-13T00:41:41.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:39.546742+0000 mon.a (mon.0) 1128 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84065-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:41.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:39.546866+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-02-13T00:41:41.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:39.551166+0000 mon.b (mon.1) 156 : audit [INF] from='client.? 172.21.15.162:0/1389954781' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-83977-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:41.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:39.566058+0000 mon.a (mon.0) 1130 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-84073-9"}]: dispatch 2024-02-13T00:41:41.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:39.585108+0000 mon.c (mon.2) 134 : audit [INF] from='client.? 172.21.15.162:0/4066446528' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache"}]: dispatch 2024-02-13T00:41:41.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:39.591992+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-83977-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:41.253 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:39.592497+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache"}]: dispatch 2024-02-13T00:41:41.253 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:40.055877+0000 mon.b (mon.1) 157 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:41.253 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:40.317659+0000 mon.a (mon.0) 1133 : audit [INF] from='client.? 172.21.15.162:0/3520430640' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-84073-9"}]': finished 2024-02-13T00:41:41.253 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:40.318809+0000 mon.a (mon.0) 1134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-83977-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:41.253 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:40.318926+0000 mon.a (mon.0) 1135 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84065-10", "tierpool":"test-rados-api-smithi162-84065-10-cache"}]': finished 2024-02-13T00:41:41.253 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:40.319087+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-02-13T00:41:41.253 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:40.593663+0000 mon.b (mon.1) 158 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:41.253 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:41.273 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:41.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:41.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:41.585 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:41.630 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:41:41.630 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:41:41.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:41.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:41.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:41.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:41.837 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:41.870 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:41.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:41.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:41:42.001 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:42.020 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:41:42.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:42.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:41:42.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:42.155 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:40.290499+0000 mgr.y (mgr.14544) 181 : cluster [DBG] pgmap v187: 484 pgs: 32 creating+peering, 452 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-02-13T00:41:42.155 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:40.762863+0000 mon.a (mon.0) 1137 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:42.155 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:40.767860+0000 mon.a (mon.0) 1138 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:42.155 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:40.768590+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:42.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:40.803913+0000 mon.a (mon.0) 1140 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:42.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:40.860394+0000 mon.a (mon.0) 1141 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:42.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:41.063595+0000 mon.b (mon.1) 159 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:42.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:41.493339+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:41:42.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:41.493412+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:42.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:41.493458+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-02-13T00:41:42.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:41.501735+0000 mon.b (mon.1) 160 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11"}]: dispatch 2024-02-13T00:41:42.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:41.503787+0000 mon.a (mon.0) 1145 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-84073-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:42.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:41.506481+0000 mon.c (mon.2) 135 : audit [INF] from='client.? 172.21.15.162:0/3839764873' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-83985-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:42.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:41.507373+0000 mon.a (mon.0) 1146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11"}]: dispatch 2024-02-13T00:41:42.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:41.507820+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-83985-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:42.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:40.290499+0000 mgr.y (mgr.14544) 181 : cluster [DBG] pgmap v187: 484 pgs: 32 creating+peering, 452 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-02-13T00:41:42.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:40.762863+0000 mon.a (mon.0) 1137 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:42.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:40.767860+0000 mon.a (mon.0) 1138 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:42.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:40.768590+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:42.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:40.803913+0000 mon.a (mon.0) 1140 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:42.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:40.860394+0000 mon.a (mon.0) 1141 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:42.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:41.063595+0000 mon.b (mon.1) 159 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:42.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:41.493339+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:41:42.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:41.493412+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:42.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:41.493458+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-02-13T00:41:42.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:41.501735+0000 mon.b (mon.1) 160 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11"}]: dispatch 2024-02-13T00:41:42.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:41.503787+0000 mon.a (mon.0) 1145 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-84073-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:42.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:41.506481+0000 mon.c (mon.2) 135 : audit [INF] from='client.? 172.21.15.162:0/3839764873' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-83985-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:42.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:41.507373+0000 mon.a (mon.0) 1146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11"}]: dispatch 2024-02-13T00:41:42.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:41.507820+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-83985-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:42.291 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:40.290499+0000 mgr.y (mgr.14544) 181 : cluster [DBG] pgmap v187: 484 pgs: 32 creating+peering, 452 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-02-13T00:41:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:40.762863+0000 mon.a (mon.0) 1137 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:40.767860+0000 mon.a (mon.0) 1138 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:40.768590+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:41:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:40.803913+0000 mon.a (mon.0) 1140 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:40.860394+0000 mon.a (mon.0) 1141 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:41:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:41.063595+0000 mon.b (mon.1) 159 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:41.493339+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:41:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:41.493412+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:41:42.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:41.493458+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-02-13T00:41:42.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:41.501735+0000 mon.b (mon.1) 160 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11"}]: dispatch 2024-02-13T00:41:42.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:41.503787+0000 mon.a (mon.0) 1145 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-84073-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:42.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:41.506481+0000 mon.c (mon.2) 135 : audit [INF] from='client.? 172.21.15.162:0/3839764873' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-83985-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:42.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:41.507373+0000 mon.a (mon.0) 1146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11"}]: dispatch 2024-02-13T00:41:42.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:41.507820+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-83985-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:42.529 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:42.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:42.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:42.642 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:42.739 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:42.969 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: Running main() from gmock_main.cc 2024-02-13T00:41:42.969 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-02-13T00:41:42.969 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [----------] Global test environment set-up. 2024-02-13T00:41:42.969 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-02-13T00:41:42.969 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolList (8103 ms) 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (6519 ms) 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3321 ms) 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3055 ms) 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3139 ms) 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (4998 ms) 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5422 ms) 2024-02-13T00:41:42.970 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-02-13T00:41:42.971 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5145 ms) 2024-02-13T00:41:42.971 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-02-13T00:41:42.971 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8579 ms) 2024-02-13T00:41:42.971 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [----------] 9 tests from LibRadosPools (48282 ms total) 2024-02-13T00:41:42.971 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: 2024-02-13T00:41:42.971 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [----------] Global test environment tear-down 2024-02-13T00:41:42.971 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (48282 ms total) 2024-02-13T00:41:42.971 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ PASSED ] 9 tests. 2024-02-13T00:41:43.217 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:43.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:43.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:43.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:41.474929+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.9 scrub starts 2024-02-13T00:41:43.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:41.476168+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.9 scrub ok 2024-02-13T00:41:43.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:41.681703+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:41:43.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:41.686052+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:41:43.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:41.901047+0000 osd.1 (osd.1) 77 : cluster [DBG] 20.5 deep-scrub starts 2024-02-13T00:41:43.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:41.904946+0000 osd.1 (osd.1) 78 : cluster [DBG] 20.5 deep-scrub ok 2024-02-13T00:41:43.244 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:42.066263+0000 mon.b (mon.1) 161 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:43.244 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:42.487759+0000 mon.a (mon.0) 1148 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:41:43.244 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:42.500618+0000 osd.6 (osd.6) 55 : cluster [DBG] 20.1 deep-scrub starts 2024-02-13T00:41:43.244 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:42.509858+0000 osd.6 (osd.6) 56 : cluster [DBG] 20.1 deep-scrub ok 2024-02-13T00:41:43.244 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:42.942360+0000 mon.a (mon.0) 1149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11"}]': finished 2024-02-13T00:41:43.244 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:42.942474+0000 mon.a (mon.0) 1150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-83985-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:43.244 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:42.942694+0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-02-13T00:41:43.244 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:42.950806+0000 mon.b (mon.1) 162 : audit [INF] from='client.? 172.21.15.162:0/1013715296' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-83977-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:43.244 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:42.974222+0000 mon.a (mon.0) 1152 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-83977-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:43.245 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:41.474929+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.9 scrub starts 2024-02-13T00:41:43.245 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:41.476168+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.9 scrub ok 2024-02-13T00:41:43.245 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:41.681703+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:41:43.245 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:41.686052+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:41:43.245 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:41.901047+0000 osd.1 (osd.1) 77 : cluster [DBG] 20.5 deep-scrub starts 2024-02-13T00:41:43.245 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:41.904946+0000 osd.1 (osd.1) 78 : cluster [DBG] 20.5 deep-scrub ok 2024-02-13T00:41:43.245 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:42.066263+0000 mon.b (mon.1) 161 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:43.245 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:42.487759+0000 mon.a (mon.0) 1148 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:41:43.245 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:42.500618+0000 osd.6 (osd.6) 55 : cluster [DBG] 20.1 deep-scrub starts 2024-02-13T00:41:43.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:42.509858+0000 osd.6 (osd.6) 56 : cluster [DBG] 20.1 deep-scrub ok 2024-02-13T00:41:43.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:42.942360+0000 mon.a (mon.0) 1149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11"}]': finished 2024-02-13T00:41:43.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:42.942474+0000 mon.a (mon.0) 1150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-83985-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:43.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:42.942694+0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-02-13T00:41:43.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:42.950806+0000 mon.b (mon.1) 162 : audit [INF] from='client.? 172.21.15.162:0/1013715296' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-83977-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:43.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:42.974222+0000 mon.a (mon.0) 1152 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-83977-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:43.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:41.474929+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.9 scrub starts 2024-02-13T00:41:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:41.476168+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.9 scrub ok 2024-02-13T00:41:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:41.681703+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:41:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:41.686052+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:41:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:41.901047+0000 osd.1 (osd.1) 77 : cluster [DBG] 20.5 deep-scrub starts 2024-02-13T00:41:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:41.904946+0000 osd.1 (osd.1) 78 : cluster [DBG] 20.5 deep-scrub ok 2024-02-13T00:41:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:42.066263+0000 mon.b (mon.1) 161 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:42.487759+0000 mon.a (mon.0) 1148 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:41:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:42.500618+0000 osd.6 (osd.6) 55 : cluster [DBG] 20.1 deep-scrub starts 2024-02-13T00:41:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:42.509858+0000 osd.6 (osd.6) 56 : cluster [DBG] 20.1 deep-scrub ok 2024-02-13T00:41:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:42.942360+0000 mon.a (mon.0) 1149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-11"}]': finished 2024-02-13T00:41:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:42.942474+0000 mon.a (mon.0) 1150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-83985-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:42.942694+0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-02-13T00:41:43.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:42.950806+0000 mon.b (mon.1) 162 : audit [INF] from='client.? 172.21.15.162:0/1013715296' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-83977-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:43.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:42.974222+0000 mon.a (mon.0) 1152 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-83977-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:43.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:43.389 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:43.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.717+0000 7f9733a84700 1 -- 172.21.15.162:0/3657696361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f972c1048e0 msgr2=0x7f972c0695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:43.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.717+0000 7f9733a84700 1 --2- 172.21.15.162:0/3657696361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f972c1048e0 0x7f972c0695c0 secure :-1 s=READY pgs=4731 cs=0 l=1 rev1=1 rx=0x7f971c0048a0 tx=0x7f971c009cb0).stop 2024-02-13T00:41:43.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.717+0000 7f9733a84700 1 -- 172.21.15.162:0/3657696361 shutdown_connections 2024-02-13T00:41:43.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.717+0000 7f9733a84700 1 --2- 172.21.15.162:0/3657696361 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f972c069ab0 0x7f972c069f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:43.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.717+0000 7f9733a84700 1 --2- 172.21.15.162:0/3657696361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f972c104010 0x7f972c1043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.717+0000 7f9733a84700 1 --2- 172.21.15.162:0/3657696361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f972c1048e0 0x7f972c0695c0 unknown :-1 s=CLOSED pgs=4731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.717+0000 7f9733a84700 1 -- 172.21.15.162:0/3657696361 >> 172.21.15.162:0/3657696361 conn(0x7f972c063df0 msgr2=0x7f972c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.718+0000 7f9733a84700 1 -- 172.21.15.162:0/3657696361 shutdown_connections 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.718+0000 7f9733a84700 1 -- 172.21.15.162:0/3657696361 wait complete. 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9733a84700 1 librados: starting msgr at 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9733a84700 1 librados: starting objecter 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9733a84700 1 Processor -- start 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9733a84700 1 -- start start 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9733a84700 1 librados: setting wanted keys 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9733a84700 1 librados: calling monclient init 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9733a84700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f972c069ab0 0x7f972c10cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9733a84700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f972c104010 0x7f972c10d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9733a84700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f972c1048e0 0x7f972c111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9733a84700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c10dba0 con 0x7f972c104010 2024-02-13T00:41:43.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9733a84700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c111d10 con 0x7f972c1048e0 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9733a84700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972c111ed0 con 0x7f972c069ab0 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9732021700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f972c1048e0 0x7f972c111820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9732021700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f972c1048e0 0x7f972c111820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:55204/0 (socket says 172.21.15.162:55204) 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.719+0000 7f9732021700 1 -- 172.21.15.162:0/2315394547 learned_addr learned my addr 172.21.15.162:0/2315394547 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.746+0000 7f9732021700 1 -- 172.21.15.162:0/2315394547 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f972c069ab0 msgr2=0x7f972c10cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.746+0000 7f9732021700 1 --2- 172.21.15.162:0/2315394547 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f972c069ab0 0x7f972c10cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.746+0000 7f9732021700 1 -- 172.21.15.162:0/2315394547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f972c104010 msgr2=0x7f972c10d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.746+0000 7f9732021700 1 --2- 172.21.15.162:0/2315394547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f972c104010 0x7f972c10d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.746+0000 7f9732021700 1 -- 172.21.15.162:0/2315394547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c005040 con 0x7f972c1048e0 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.752+0000 7f9732021700 1 --2- 172.21.15.162:0/2315394547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f972c1048e0 0x7f972c111820 secure :-1 s=READY pgs=4750 cs=0 l=1 rev1=1 rx=0x7f972400bc30 tx=0x7f9724009840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.752+0000 7f9722f78700 10 client.?.objecter ms_handle_connect 0x7f972c1048e0 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.752+0000 7f9722f78700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.753+0000 7f9722f78700 1 -- 172.21.15.162:0/2315394547 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972400c7a0 con 0x7f972c1048e0 2024-02-13T00:41:43.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.753+0000 7f9722f78700 1 -- 172.21.15.162:0/2315394547 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9724010070 con 0x7f972c1048e0 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.753+0000 7f9722f78700 1 -- 172.21.15.162:0/2315394547 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97240078f0 con 0x7f972c1048e0 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.754+0000 7f9733a84700 1 -- 172.21.15.162:0/2315394547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972c1121b0 con 0x7f972c1048e0 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.754+0000 7f9733a84700 10 client.43273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.754+0000 7f9733a84700 1 -- 172.21.15.162:0/2315394547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972c112620 con 0x7f972c1048e0 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.754+0000 7f9733a84700 1 librados: init done 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.756+0000 7f9733a84700 1 -- 172.21.15.162:0/2315394547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972c0603a0 con 0x7f972c1048e0 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.759+0000 7f9722f78700 1 -- 172.21.15.162:0/2315394547 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f972400cb10 con 0x7f972c1048e0 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.760+0000 7f9722f78700 1 --2- 172.21.15.162:0/2315394547 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9718062d20 0x7f97180651e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.760+0000 7f9722f78700 1 -- 172.21.15.162:0/2315394547 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(148..148 src has 52..148) v4 ==== 10215+0+0 (secure 0 0 0) 0x7f972407dea0 con 0x7f972c1048e0 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.760+0000 7f9722f78700 10 client.43273.objecter ms_dispatch 0x7f972c001550 osd_map(148..148 src has 52..148) v4 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.760+0000 7f9722f78700 3 client.43273.objecter handle_osd_map got epochs [148,148] > 0 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.760+0000 7f9722f78700 3 client.43273.objecter handle_osd_map decoding full epoch 148 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.760+0000 7f9722f78700 20 client.43273.objecter dump_active .. 0 homeless 2024-02-13T00:41:43.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.774+0000 7f9722f78700 1 -- 172.21.15.162:0/2315394547 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f972404af60 con 0x7f972c1048e0 2024-02-13T00:41:43.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.779+0000 7f9731820700 1 --2- 172.21.15.162:0/2315394547 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9718062d20 0x7f97180651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:43.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.790+0000 7f9731820700 1 --2- 172.21.15.162:0/2315394547 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9718062d20 0x7f97180651e0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 rx=0x7f9728006fd0 tx=0x7f9728006b90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:43.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:43.790+0000 7f9722f78700 10 client.43273.objecter ms_handle_connect 0x7f9718062d20 2024-02-13T00:41:44.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.128+0000 7f9733a84700 1 -- 172.21.15.162:0/2315394547 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f972c0475e0 con 0x7f972c1048e0 2024-02-13T00:41:44.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-13T00:41:44.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.410+0000 7fd37f22d700 1 -- 172.21.15.162:0/1016693059 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd3780fa440 msgr2=0x7fd3780f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.410+0000 7fd37f22d700 1 --2- 172.21.15.162:0/1016693059 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd3780fa440 0x7fd3780f7e20 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 rx=0x7fd364004870 tx=0x7fd364009cb0).stop 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.410+0000 7fd37f22d700 1 -- 172.21.15.162:0/1016693059 shutdown_connections 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.410+0000 7fd37f22d700 1 --2- 172.21.15.162:0/1016693059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3780f8500 0x7fd378104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.410+0000 7fd37f22d700 1 --2- 172.21.15.162:0/1016693059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3780f9b70 0x7fd3780f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.410+0000 7fd37f22d700 1 --2- 172.21.15.162:0/1016693059 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd3780fa440 0x7fd3780f7e20 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.410+0000 7fd37f22d700 1 -- 172.21.15.162:0/1016693059 >> 172.21.15.162:0/1016693059 conn(0x7fd3780f38c0 msgr2=0x7fd3780f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.410+0000 7fd37f22d700 1 -- 172.21.15.162:0/1016693059 shutdown_connections 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.410+0000 7fd37f22d700 1 -- 172.21.15.162:0/1016693059 wait complete. 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37f22d700 1 librados: starting msgr at 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37f22d700 1 librados: starting objecter 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37f22d700 1 Processor -- start 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37f22d700 1 -- start start 2024-02-13T00:41:44.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37f22d700 1 librados: setting wanted keys 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37f22d700 1 librados: calling monclient init 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37f22d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3780f8500 0x7fd37806b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37f22d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3780f9b70 0x7fd37806b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37f22d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd3780fa440 0x7fd37806bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37f22d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd378116730 con 0x7fd3780f9b70 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37f22d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3781168a0 con 0x7fd3780f8500 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37f22d700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd378116a60 con 0x7fd3780fa440 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd377fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3780f9b70 0x7fd37806b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd37d7ca700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd3780fa440 0x7fd37806bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd377fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3780f9b70 0x7fd37806b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:54410/0 (socket says 172.21.15.162:54410) 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.411+0000 7fd377fff700 1 -- 172.21.15.162:0/1853483917 learned_addr learned my addr 172.21.15.162:0/1853483917 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd37cfc9700 1 --2- 172.21.15.162:0/1853483917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3780f8500 0x7fd37806b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:44.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd37d7ca700 1 -- 172.21.15.162:0/1853483917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3780f8500 msgr2=0x7fd37806b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd37d7ca700 1 --2- 172.21.15.162:0/1853483917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3780f8500 0x7fd37806b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd37d7ca700 1 -- 172.21.15.162:0/1853483917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3780f9b70 msgr2=0x7fd37806b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd37d7ca700 1 --2- 172.21.15.162:0/1853483917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3780f9b70 0x7fd37806b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd37d7ca700 1 -- 172.21.15.162:0/1853483917 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36c005040 con 0x7fd3780fa440 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd377fff700 1 --2- 172.21.15.162:0/1853483917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3780f9b70 0x7fd37806b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd37d7ca700 1 --2- 172.21.15.162:0/1853483917 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd3780fa440 0x7fd37806bb50 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 rx=0x7fd37000eb70 tx=0x7fd37000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd375ffb700 10 client.?.objecter ms_handle_connect 0x7fd3780fa440 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd375ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd375ffb700 1 -- 172.21.15.162:0/1853483917 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd370009870 con 0x7fd3780fa440 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd375ffb700 1 -- 172.21.15.162:0/1853483917 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd37000f040 con 0x7fd3780fa440 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd37f22d700 1 -- 172.21.15.162:0/1853483917 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd364005040 con 0x7fd3780fa440 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd375ffb700 1 -- 172.21.15.162:0/1853483917 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd370007da0 con 0x7fd3780fa440 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd37f22d700 10 client.36464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:44.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd37f22d700 1 -- 172.21.15.162:0/1853483917 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd378117070 con 0x7fd3780fa440 2024-02-13T00:41:44.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.412+0000 7fd37f22d700 1 librados: init done 2024-02-13T00:41:44.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.413+0000 7fd37f22d700 1 -- 172.21.15.162:0/1853483917 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3780475e0 con 0x7fd3780fa440 2024-02-13T00:41:44.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.414+0000 7fd375ffb700 1 -- 172.21.15.162:0/1853483917 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fd3700040b0 con 0x7fd3780fa440 2024-02-13T00:41:44.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.414+0000 7fd375ffb700 1 --2- 172.21.15.162:0/1853483917 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd368062c90 0x7fd368065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:44.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.414+0000 7fd375ffb700 1 -- 172.21.15.162:0/1853483917 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(148..148 src has 52..148) v4 ==== 10215+0+0 (secure 0 0 0) 0x7fd37007d760 con 0x7fd3780fa440 2024-02-13T00:41:44.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.414+0000 7fd375ffb700 10 client.36464.objecter ms_dispatch 0x7fd378001550 osd_map(148..148 src has 52..148) v4 2024-02-13T00:41:44.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.414+0000 7fd375ffb700 3 client.36464.objecter handle_osd_map got epochs [148,148] > 0 2024-02-13T00:41:44.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.414+0000 7fd375ffb700 3 client.36464.objecter handle_osd_map decoding full epoch 148 2024-02-13T00:41:44.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.414+0000 7fd37cfc9700 1 --2- 172.21.15.162:0/1853483917 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd368062c90 0x7fd368065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:44.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.414+0000 7fd375ffb700 20 client.36464.objecter dump_active .. 0 homeless 2024-02-13T00:41:44.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.415+0000 7fd375ffb700 10 client.36464.objecter ms_handle_connect 0x7fd368062c90 2024-02-13T00:41:44.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.415+0000 7fd37cfc9700 1 --2- 172.21.15.162:0/1853483917 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd368062c90 0x7fd368065150 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7fd36c009350 tx=0x7fd36c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:44.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.419+0000 7fd375ffb700 1 -- 172.21.15.162:0/1853483917 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd370049f60 con 0x7fd3780fa440 2024-02-13T00:41:44.484 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:44.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:42.305821+0000 mgr.y (mgr.14544) 182 : cluster [DBG] pgmap v190: 420 pgs: 32 unknown, 388 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-02-13T00:41:44.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:42.691968+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:41:44.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:42.695513+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:41:44.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:43.067719+0000 mon.b (mon.1) 163 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:44.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:43.500679+0000 osd.6 (osd.6) 57 : cluster [DBG] 20.8 deep-scrub starts 2024-02-13T00:41:44.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:43.503577+0000 osd.6 (osd.6) 58 : cluster [DBG] 20.8 deep-scrub ok 2024-02-13T00:41:44.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:43.545153+0000 mon.a (mon.0) 1153 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-84073-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]': finished 2024-02-13T00:41:44.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:43.545281+0000 mon.a (mon.0) 1154 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-83977-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:44.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:43.545363+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-02-13T00:41:44.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.553+0000 7f9722f78700 1 -- 172.21.15.162:0/2315394547 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v149) v1 ==== 101+0+0 (secure 0 0 0) 0x7f97240506f0 con 0x7f972c1048e0 2024-02-13T00:41:44.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.559+0000 7f9733a84700 10 librados: watch_flush enter 2024-02-13T00:41:44.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.559+0000 7f9733a84700 10 librados: watch_flush exit 2024-02-13T00:41:44.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.559+0000 7f9733a84700 20 client.43273.objecter shutdown clearing up homeless session... 2024-02-13T00:41:44.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.559+0000 7f9733a84700 10 client.43273.objecter successfully canceled tick 2024-02-13T00:41:44.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.565+0000 7f9733a84700 1 -- 172.21.15.162:0/2315394547 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9718062d20 msgr2=0x7f97180651e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:44.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.565+0000 7f9733a84700 1 --2- 172.21.15.162:0/2315394547 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9718062d20 0x7f97180651e0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 rx=0x7f9728006fd0 tx=0x7f9728006b90).stop 2024-02-13T00:41:44.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.565+0000 7f9733a84700 1 -- 172.21.15.162:0/2315394547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f972c1048e0 msgr2=0x7f972c111820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:44.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.565+0000 7f9733a84700 1 --2- 172.21.15.162:0/2315394547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f972c1048e0 0x7f972c111820 secure :-1 s=READY pgs=4750 cs=0 l=1 rev1=1 rx=0x7f972400bc30 tx=0x7f9724009840).stop 2024-02-13T00:41:44.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.565+0000 7f9733a84700 1 -- 172.21.15.162:0/2315394547 shutdown_connections 2024-02-13T00:41:44.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.565+0000 7f9733a84700 1 --2- 172.21.15.162:0/2315394547 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9718062d20 0x7f97180651e0 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:44.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.565+0000 7f9733a84700 1 --2- 172.21.15.162:0/2315394547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f972c1048e0 0x7f972c111820 unknown :-1 s=CLOSED pgs=4750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:44.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.565+0000 7f9733a84700 1 --2- 172.21.15.162:0/2315394547 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f972c069ab0 0x7f972c10cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:44.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.565+0000 7f9733a84700 1 --2- 172.21.15.162:0/2315394547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f972c104010 0x7f972c10d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:44.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.565+0000 7f9733a84700 1 -- 172.21.15.162:0/2315394547 >> 172.21.15.162:0/2315394547 conn(0x7f972c063df0 msgr2=0x7f972c105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:44.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-02-13T00:41:44.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] Global test environment set-up. 2024-02-13T00:41:44.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: 2024-02-13T00:40:54.805+0000 7f1eec592700 -1 failed for service _ceph-mon._tcp 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: 2024-02-13T00:40:54.805+0000 7f1eec592700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (82 ms) 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2008 ms) 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2090 ms total) 2024-02-13T00:41:44.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: started 0x556246127110 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: started 0x5562461305f0 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: started 2 aios 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: waiting 0x556246127110 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: waiting 0x5562461305f0 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: done. 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (9231 ms) 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (9232 ms total) 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-02-13T00:41:44.579 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.Exec (18 ms) 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (1037 ms) 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (8 ms) 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5285 ms) 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-02-13T00:41:44.580 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-02-13T00:41:44.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (29702 ms) 2024-02-13T00:41:44.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 8 tests from LibRadosMisc (36050 ms total) 2024-02-13T00:41:44.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: 2024-02-13T00:41:44.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] Global test environment tear-down 2024-02-13T00:41:44.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (49833 ms total) 2024-02-13T00:41:44.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ PASSED ] 12 tests. 2024-02-13T00:41:44.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.571+0000 7f9733a84700 1 -- 172.21.15.162:0/2315394547 shutdown_connections 2024-02-13T00:41:44.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.571+0000 7f9733a84700 1 -- 172.21.15.162:0/2315394547 wait complete. 2024-02-13T00:41:44.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.573+0000 7f9733a84700 1 librados: shutdown 2024-02-13T00:41:44.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:42.305821+0000 mgr.y (mgr.14544) 182 : cluster [DBG] pgmap v190: 420 pgs: 32 unknown, 388 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-02-13T00:41:44.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:42.691968+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:41:44.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:42.695513+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:41:44.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:43.067719+0000 mon.b (mon.1) 163 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:44.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:43.500679+0000 osd.6 (osd.6) 57 : cluster [DBG] 20.8 deep-scrub starts 2024-02-13T00:41:44.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:43.503577+0000 osd.6 (osd.6) 58 : cluster [DBG] 20.8 deep-scrub ok 2024-02-13T00:41:44.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:43.545153+0000 mon.a (mon.0) 1153 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-84073-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]': finished 2024-02-13T00:41:44.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:43.545281+0000 mon.a (mon.0) 1154 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-83977-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:44.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:43.545363+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-02-13T00:41:44.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:42.305821+0000 mgr.y (mgr.14544) 182 : cluster [DBG] pgmap v190: 420 pgs: 32 unknown, 388 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-02-13T00:41:44.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:42.691968+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:41:44.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:42.695513+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:41:44.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:43.067719+0000 mon.b (mon.1) 163 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:44.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:43.500679+0000 osd.6 (osd.6) 57 : cluster [DBG] 20.8 deep-scrub starts 2024-02-13T00:41:44.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:43.503577+0000 osd.6 (osd.6) 58 : cluster [DBG] 20.8 deep-scrub ok 2024-02-13T00:41:44.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:43.545153+0000 mon.a (mon.0) 1153 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-84073-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]': finished 2024-02-13T00:41:44.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:43.545281+0000 mon.a (mon.0) 1154 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-83977-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:44.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:43.545363+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-02-13T00:41:44.605 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:41:44.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:44.656+0000 7fd37f22d700 1 -- 172.21.15.162:0/1853483917 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "20.c", "id": [5, 6, 4]} v 0) v1 -- 0x7fd37806c650 con 0x7fd3780fa440 2024-02-13T00:41:45.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:43.407040+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:41:45.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:43.426974+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:41:45.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:43.702978+0000 osd.3 (osd.3) 97 : cluster [DBG] 5.13 scrub starts 2024-02-13T00:41:45.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:43.716884+0000 osd.3 (osd.3) 98 : cluster [DBG] 5.13 scrub ok 2024-02-13T00:41:45.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:44.069296+0000 mon.b (mon.1) 164 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:45.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:44.130645+0000 mon.b (mon.1) 165 : audit [INF] from='client.? 172.21.15.162:0/2315394547' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:41:45.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:44.265677+0000 mon.a (mon.0) 1156 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:41:45.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:44.308105+0000 mgr.y (mgr.14544) 183 : cluster [DBG] pgmap v193: 364 pgs: 40 unknown, 324 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:45.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:44.456847+0000 osd.6 (osd.6) 59 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:41:45.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:44.457669+0000 osd.6 (osd.6) 60 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:41:45.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:44.542813+0000 mon.a (mon.0) 1157 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:41:45.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:44.547843+0000 mon.a (mon.0) 1158 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:41:45.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:44.547927+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-02-13T00:41:45.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:44.550385+0000 mon.c (mon.2) 136 : audit [INF] from='client.? 172.21.15.162:0/3985387687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-83985-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:45.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:44.551060+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-83985-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:45.426 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:44.558072+0000 mon.b (mon.1) 166 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:45.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:43.407040+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:41:45.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:43.426974+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:43.702978+0000 osd.3 (osd.3) 97 : cluster [DBG] 5.13 scrub starts 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:43.716884+0000 osd.3 (osd.3) 98 : cluster [DBG] 5.13 scrub ok 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:44.069296+0000 mon.b (mon.1) 164 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:44.130645+0000 mon.b (mon.1) 165 : audit [INF] from='client.? 172.21.15.162:0/2315394547' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:44.265677+0000 mon.a (mon.0) 1156 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:44.308105+0000 mgr.y (mgr.14544) 183 : cluster [DBG] pgmap v193: 364 pgs: 40 unknown, 324 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:44.456847+0000 osd.6 (osd.6) 59 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:44.457669+0000 osd.6 (osd.6) 60 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:44.542813+0000 mon.a (mon.0) 1157 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:44.547843+0000 mon.a (mon.0) 1158 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:44.547927+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:44.550385+0000 mon.c (mon.2) 136 : audit [INF] from='client.? 172.21.15.162:0/3985387687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-83985-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:44.551060+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-83985-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:45.427 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:44.558072+0000 mon.b (mon.1) 166 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:45.428 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:44.563549+0000 mon.a (mon.0) 1161 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:45.428 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:44.658695+0000 mon.c (mon.2) 137 : audit [INF] from='client.? 172.21.15.162:0/1853483917' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "20.c", "id": [5, 6, 4]}]: dispatch 2024-02-13T00:41:45.428 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:44.870332+0000 mon.a (mon.0) 1162 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "20.c", "id": [5, 6, 4]}]: dispatch 2024-02-13T00:41:45.428 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:45.073294+0000 mon.b (mon.1) 167 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:45.428 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:44.563549+0000 mon.a (mon.0) 1161 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:45.428 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:44.658695+0000 mon.c (mon.2) 137 : audit [INF] from='client.? 172.21.15.162:0/1853483917' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "20.c", "id": [5, 6, 4]}]: dispatch 2024-02-13T00:41:45.428 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:44.870332+0000 mon.a (mon.0) 1162 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "20.c", "id": [5, 6, 4]}]: dispatch 2024-02-13T00:41:45.428 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:45.073294+0000 mon.b (mon.1) 167 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:45.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:41:45.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:43.407040+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:43.426974+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:43.702978+0000 osd.3 (osd.3) 97 : cluster [DBG] 5.13 scrub starts 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:43.716884+0000 osd.3 (osd.3) 98 : cluster [DBG] 5.13 scrub ok 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:44.069296+0000 mon.b (mon.1) 164 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:44.130645+0000 mon.b (mon.1) 165 : audit [INF] from='client.? 172.21.15.162:0/2315394547' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:44.265677+0000 mon.a (mon.0) 1156 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:44.308105+0000 mgr.y (mgr.14544) 183 : cluster [DBG] pgmap v193: 364 pgs: 40 unknown, 324 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:44.456847+0000 osd.6 (osd.6) 59 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:44.457669+0000 osd.6 (osd.6) 60 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:44.542813+0000 mon.a (mon.0) 1157 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:44.547843+0000 mon.a (mon.0) 1158 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:44.547927+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-02-13T00:41:45.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:44.550385+0000 mon.c (mon.2) 136 : audit [INF] from='client.? 172.21.15.162:0/3985387687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-83985-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:45.547 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:44.551060+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-83985-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:45.547 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:44.558072+0000 mon.b (mon.1) 166 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:45.547 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:44.563549+0000 mon.a (mon.0) 1161 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:45.547 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:44.658695+0000 mon.c (mon.2) 137 : audit [INF] from='client.? 172.21.15.162:0/1853483917' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "20.c", "id": [5, 6, 4]}]: dispatch 2024-02-13T00:41:45.547 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:44.870332+0000 mon.a (mon.0) 1162 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "20.c", "id": [5, 6, 4]}]: dispatch 2024-02-13T00:41:45.547 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:45.073294+0000 mon.b (mon.1) 167 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:45.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.552+0000 7fd375ffb700 1 -- 172.21.15.162:0/1853483917 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "20.c", "id": [5, 6, 4]}]=0 set 20.c pg_upmap mapping to [5,6,4] v150) v1 ==== 129+0+0 (secure 0 0 0) 0x7fd3700503d0 con 0x7fd3780fa440 2024-02-13T00:41:45.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.555+0000 7fd37f22d700 10 librados: watch_flush enter 2024-02-13T00:41:45.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.555+0000 7fd37f22d700 10 librados: watch_flush exit 2024-02-13T00:41:45.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.555+0000 7fd37f22d700 20 client.36464.objecter shutdown clearing up homeless session... 2024-02-13T00:41:45.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.555+0000 7fd37f22d700 10 client.36464.objecter successfully canceled tick 2024-02-13T00:41:45.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.556+0000 7fd37f22d700 1 -- 172.21.15.162:0/1853483917 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd368062c90 msgr2=0x7fd368065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:45.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.556+0000 7fd37f22d700 1 --2- 172.21.15.162:0/1853483917 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd368062c90 0x7fd368065150 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7fd36c009350 tx=0x7fd36c00b040).stop 2024-02-13T00:41:45.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.556+0000 7fd37f22d700 1 -- 172.21.15.162:0/1853483917 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd3780fa440 msgr2=0x7fd37806bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:45.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.556+0000 7fd37f22d700 1 --2- 172.21.15.162:0/1853483917 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd3780fa440 0x7fd37806bb50 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 rx=0x7fd37000eb70 tx=0x7fd37000b040).stop 2024-02-13T00:41:45.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.558+0000 7fd37f22d700 1 -- 172.21.15.162:0/1853483917 shutdown_connections 2024-02-13T00:41:45.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.558+0000 7fd37f22d700 1 --2- 172.21.15.162:0/1853483917 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd368062c90 0x7fd368065150 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:45.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.558+0000 7fd37f22d700 1 --2- 172.21.15.162:0/1853483917 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd3780fa440 0x7fd37806bb50 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:45.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.558+0000 7fd37f22d700 1 --2- 172.21.15.162:0/1853483917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3780f8500 0x7fd37806b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:45.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.558+0000 7fd37f22d700 1 --2- 172.21.15.162:0/1853483917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3780f9b70 0x7fd37806b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:45.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.558+0000 7fd37f22d700 1 -- 172.21.15.162:0/1853483917 >> 172.21.15.162:0/1853483917 conn(0x7fd3780f38c0 msgr2=0x7fd3780f4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:45.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.558+0000 7fd37f22d700 1 -- 172.21.15.162:0/1853483917 shutdown_connections 2024-02-13T00:41:45.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.558+0000 7fd37f22d700 1 -- 172.21.15.162:0/1853483917 wait complete. 2024-02-13T00:41:45.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:45.558+0000 7fd37f22d700 1 librados: shutdown 2024-02-13T00:41:45.566 INFO:teuthology.orchestra.run.smithi162.stderr:set 20.c pg_upmap mapping to [5,6,4] 2024-02-13T00:41:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:44.694705+0000 osd.3 (osd.3) 99 : cluster [DBG] 5.0 deep-scrub starts 2024-02-13T00:41:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:44.698902+0000 osd.3 (osd.3) 100 : cluster [DBG] 5.0 deep-scrub ok 2024-02-13T00:41:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:45.364463+0000 mgr.y (mgr.14544) 184 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:45.413329+0000 osd.5 (osd.5) 77 : cluster [DBG] 6.a scrub starts 2024-02-13T00:41:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:45.416235+0000 osd.5 (osd.5) 78 : cluster [DBG] 6.a scrub ok 2024-02-13T00:41:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:45.549497+0000 mon.a (mon.0) 1163 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-83985-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:45.549641+0000 mon.a (mon.0) 1164 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:45.549822+0000 mon.a (mon.0) 1165 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "20.c", "id": [5, 6, 4]}]': finished 2024-02-13T00:41:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:45.549885+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-02-13T00:41:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:45.561487+0000 mon.c (mon.2) 138 : audit [INF] from='client.? 172.21.15.162:0/4234726551' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi162-83977-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:45.561997+0000 mon.a (mon.0) 1167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi162-83977-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:45.600946+0000 osd.7 (osd.7) 78 : cluster [DBG] 20.7 deep-scrub starts 2024-02-13T00:41:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:45.601838+0000 osd.7 (osd.7) 79 : cluster [DBG] 20.7 deep-scrub ok 2024-02-13T00:41:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:46.074234+0000 mon.b (mon.1) 168 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:46.591 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:41:46] "GET /metrics HTTP/1.1" 200 197973 "" "Prometheus/2.33.4" 2024-02-13T00:41:46.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:44.694705+0000 osd.3 (osd.3) 99 : cluster [DBG] 5.0 deep-scrub starts 2024-02-13T00:41:46.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:44.698902+0000 osd.3 (osd.3) 100 : cluster [DBG] 5.0 deep-scrub ok 2024-02-13T00:41:46.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:45.364463+0000 mgr.y (mgr.14544) 184 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:46.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:45.413329+0000 osd.5 (osd.5) 77 : cluster [DBG] 6.a scrub starts 2024-02-13T00:41:46.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:45.416235+0000 osd.5 (osd.5) 78 : cluster [DBG] 6.a scrub ok 2024-02-13T00:41:46.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:45.549497+0000 mon.a (mon.0) 1163 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-83985-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:46.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:45.549641+0000 mon.a (mon.0) 1164 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:46.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:45.549822+0000 mon.a (mon.0) 1165 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "20.c", "id": [5, 6, 4]}]': finished 2024-02-13T00:41:46.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:45.549885+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-02-13T00:41:46.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:45.561487+0000 mon.c (mon.2) 138 : audit [INF] from='client.? 172.21.15.162:0/4234726551' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi162-83977-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:46.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:45.561997+0000 mon.a (mon.0) 1167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi162-83977-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:46.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:45.600946+0000 osd.7 (osd.7) 78 : cluster [DBG] 20.7 deep-scrub starts 2024-02-13T00:41:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:45.601838+0000 osd.7 (osd.7) 79 : cluster [DBG] 20.7 deep-scrub ok 2024-02-13T00:41:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:46.074234+0000 mon.b (mon.1) 168 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:44.694705+0000 osd.3 (osd.3) 99 : cluster [DBG] 5.0 deep-scrub starts 2024-02-13T00:41:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:44.698902+0000 osd.3 (osd.3) 100 : cluster [DBG] 5.0 deep-scrub ok 2024-02-13T00:41:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:45.364463+0000 mgr.y (mgr.14544) 184 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:45.413329+0000 osd.5 (osd.5) 77 : cluster [DBG] 6.a scrub starts 2024-02-13T00:41:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:45.416235+0000 osd.5 (osd.5) 78 : cluster [DBG] 6.a scrub ok 2024-02-13T00:41:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:45.549497+0000 mon.a (mon.0) 1163 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-83985-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:45.549641+0000 mon.a (mon.0) 1164 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:45.549822+0000 mon.a (mon.0) 1165 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "20.c", "id": [5, 6, 4]}]': finished 2024-02-13T00:41:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:45.549885+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-02-13T00:41:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:45.561487+0000 mon.c (mon.2) 138 : audit [INF] from='client.? 172.21.15.162:0/4234726551' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi162-83977-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:45.561997+0000 mon.a (mon.0) 1167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi162-83977-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:45.600946+0000 osd.7 (osd.7) 78 : cluster [DBG] 20.7 deep-scrub starts 2024-02-13T00:41:46.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:45.601838+0000 osd.7 (osd.7) 79 : cluster [DBG] 20.7 deep-scrub ok 2024-02-13T00:41:46.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:46.074234+0000 mon.b (mon.1) 168 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:47.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:45.657720+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.b deep-scrub starts 2024-02-13T00:41:47.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:45.660644+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.b deep-scrub ok 2024-02-13T00:41:47.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:46.308560+0000 mgr.y (mgr.14544) 185 : cluster [DBG] pgmap v196: 396 pgs: 32 unknown, 64 creating+peering, 300 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:47.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:46.415965+0000 osd.6 (osd.6) 61 : cluster [DBG] 5.18 scrub starts 2024-02-13T00:41:47.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:46.418856+0000 osd.6 (osd.6) 62 : cluster [DBG] 5.18 scrub ok 2024-02-13T00:41:47.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:46.551064+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi162-83977-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:47.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:46.551212+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-02-13T00:41:47.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:47.075299+0000 mon.b (mon.1) 169 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:47.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:45.657720+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.b deep-scrub starts 2024-02-13T00:41:47.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:45.660644+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.b deep-scrub ok 2024-02-13T00:41:47.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:46.308560+0000 mgr.y (mgr.14544) 185 : cluster [DBG] pgmap v196: 396 pgs: 32 unknown, 64 creating+peering, 300 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:47.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:46.415965+0000 osd.6 (osd.6) 61 : cluster [DBG] 5.18 scrub starts 2024-02-13T00:41:47.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:46.418856+0000 osd.6 (osd.6) 62 : cluster [DBG] 5.18 scrub ok 2024-02-13T00:41:47.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:46.551064+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi162-83977-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:47.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:46.551212+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-02-13T00:41:47.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:47.075299+0000 mon.b (mon.1) 169 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:47.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:45.657720+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.b deep-scrub starts 2024-02-13T00:41:47.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:45.660644+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.b deep-scrub ok 2024-02-13T00:41:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:46.308560+0000 mgr.y (mgr.14544) 185 : cluster [DBG] pgmap v196: 396 pgs: 32 unknown, 64 creating+peering, 300 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:46.415965+0000 osd.6 (osd.6) 61 : cluster [DBG] 5.18 scrub starts 2024-02-13T00:41:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:46.418856+0000 osd.6 (osd.6) 62 : cluster [DBG] 5.18 scrub ok 2024-02-13T00:41:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:46.551064+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi162-83977-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:46.551212+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-02-13T00:41:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:47.075299+0000 mon.b (mon.1) 169 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:48.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:46.695679+0000 osd.3 (osd.3) 103 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:41:48.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:46.698592+0000 osd.3 (osd.3) 104 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:41:48.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:47.308594+0000 mon.a (mon.0) 1170 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:47.445003+0000 osd.2 (osd.2) 67 : cluster [DBG] 2.c scrub starts 2024-02-13T00:41:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:47.446006+0000 osd.2 (osd.2) 68 : cluster [DBG] 2.c scrub ok 2024-02-13T00:41:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:47.484897+0000 osd.5 (osd.5) 79 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:41:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:47.487851+0000 osd.5 (osd.5) 80 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:41:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:47.554856+0000 mon.a (mon.0) 1171 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-02-13T00:41:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:47.566918+0000 mon.c (mon.2) 139 : audit [INF] from='client.? 172.21.15.162:0/3530813235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-83985-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:47.568494+0000 mon.a (mon.0) 1172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-83985-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:48.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:48.076407+0000 mon.b (mon.1) 170 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:48.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:46.695679+0000 osd.3 (osd.3) 103 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:41:48.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:46.698592+0000 osd.3 (osd.3) 104 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:41:48.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:47.308594+0000 mon.a (mon.0) 1170 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:48.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:47.445003+0000 osd.2 (osd.2) 67 : cluster [DBG] 2.c scrub starts 2024-02-13T00:41:48.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:47.446006+0000 osd.2 (osd.2) 68 : cluster [DBG] 2.c scrub ok 2024-02-13T00:41:48.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:47.484897+0000 osd.5 (osd.5) 79 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:41:48.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:47.487851+0000 osd.5 (osd.5) 80 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:41:48.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:47.554856+0000 mon.a (mon.0) 1171 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-02-13T00:41:48.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:47.566918+0000 mon.c (mon.2) 139 : audit [INF] from='client.? 172.21.15.162:0/3530813235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-83985-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:48.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:47.568494+0000 mon.a (mon.0) 1172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-83985-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:48.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:48.076407+0000 mon.b (mon.1) 170 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:48.794 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:41:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:41:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:41:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:46.695679+0000 osd.3 (osd.3) 103 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:41:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:46.698592+0000 osd.3 (osd.3) 104 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:41:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:47.308594+0000 mon.a (mon.0) 1170 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:47.445003+0000 osd.2 (osd.2) 67 : cluster [DBG] 2.c scrub starts 2024-02-13T00:41:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:47.446006+0000 osd.2 (osd.2) 68 : cluster [DBG] 2.c scrub ok 2024-02-13T00:41:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:47.484897+0000 osd.5 (osd.5) 79 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:41:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:47.487851+0000 osd.5 (osd.5) 80 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:41:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:47.554856+0000 mon.a (mon.0) 1171 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-02-13T00:41:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:47.566918+0000 mon.c (mon.2) 139 : audit [INF] from='client.? 172.21.15.162:0/3530813235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-83985-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:47.568494+0000 mon.a (mon.0) 1172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-83985-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:48.076407+0000 mon.b (mon.1) 170 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:49.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:47.731274+0000 osd.3 (osd.3) 105 : cluster [DBG] 4.1b deep-scrub starts 2024-02-13T00:41:49.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:47.734895+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.1b deep-scrub ok 2024-02-13T00:41:49.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:48.309271+0000 mgr.y (mgr.14544) 186 : cluster [DBG] pgmap v199: 364 pgs: 32 unknown, 32 creating+peering, 300 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T00:41:49.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:48.559974+0000 mon.a (mon.0) 1173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-83985-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:49.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:48.560179+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-02-13T00:41:49.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:48.564225+0000 mon.b (mon.1) 171 : audit [INF] from='client.? 172.21.15.162:0/1384123226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-83977-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:49.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:48.572977+0000 mon.a (mon.0) 1175 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-83977-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:49.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:49.077448+0000 mon.b (mon.1) 172 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:47.731274+0000 osd.3 (osd.3) 105 : cluster [DBG] 4.1b deep-scrub starts 2024-02-13T00:41:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:47.734895+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.1b deep-scrub ok 2024-02-13T00:41:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:48.309271+0000 mgr.y (mgr.14544) 186 : cluster [DBG] pgmap v199: 364 pgs: 32 unknown, 32 creating+peering, 300 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T00:41:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:48.559974+0000 mon.a (mon.0) 1173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-83985-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:49.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:48.560179+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-02-13T00:41:49.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:48.564225+0000 mon.b (mon.1) 171 : audit [INF] from='client.? 172.21.15.162:0/1384123226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-83977-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:49.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:48.572977+0000 mon.a (mon.0) 1175 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-83977-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:49.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:49.077448+0000 mon.b (mon.1) 172 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:49.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:47.731274+0000 osd.3 (osd.3) 105 : cluster [DBG] 4.1b deep-scrub starts 2024-02-13T00:41:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:47.734895+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.1b deep-scrub ok 2024-02-13T00:41:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:48.309271+0000 mgr.y (mgr.14544) 186 : cluster [DBG] pgmap v199: 364 pgs: 32 unknown, 32 creating+peering, 300 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T00:41:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:48.559974+0000 mon.a (mon.0) 1173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-83985-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:48.560179+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-02-13T00:41:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:48.564225+0000 mon.b (mon.1) 171 : audit [INF] from='client.? 172.21.15.162:0/1384123226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-83977-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:48.572977+0000 mon.a (mon.0) 1175 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-83977-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:49.077448+0000 mon.b (mon.1) 172 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:50.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:48.644151+0000 osd.0 (osd.0) 63 : cluster [DBG] 6.0 scrub starts 2024-02-13T00:41:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:48.645011+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.0 scrub ok 2024-02-13T00:41:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:49.014417+0000 osd.1 (osd.1) 79 : cluster [DBG] 20.6 deep-scrub starts 2024-02-13T00:41:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:49.051358+0000 osd.1 (osd.1) 80 : cluster [DBG] 20.6 deep-scrub ok 2024-02-13T00:41:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:49.332667+0000 mon.a (mon.0) 1176 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:41:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:49.465523+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:41:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:49.468405+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:41:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:49.563962+0000 mon.a (mon.0) 1177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-83977-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:49.564122+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-02-13T00:41:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:49.710305+0000 mon.b (mon.1) 173 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:50.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:49.854624+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:50.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:50.078405+0000 mon.b (mon.1) 174 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:50.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:50.305578+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:50.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:50.305657+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-02-13T00:41:50.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:50.319415+0000 mon.b (mon.1) 175 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-13"}]: dispatch 2024-02-13T00:41:50.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:50.327804+0000 mon.a (mon.0) 1182 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-13"}]: dispatch 2024-02-13T00:41:50.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:50.329964+0000 mon.c (mon.2) 140 : audit [INF] from='client.? 172.21.15.162:0/2910215930' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-83985-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:50.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:50.330786+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-83985-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:50.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:48.644151+0000 osd.0 (osd.0) 63 : cluster [DBG] 6.0 scrub starts 2024-02-13T00:41:50.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:48.645011+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.0 scrub ok 2024-02-13T00:41:50.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:49.014417+0000 osd.1 (osd.1) 79 : cluster [DBG] 20.6 deep-scrub starts 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:49.051358+0000 osd.1 (osd.1) 80 : cluster [DBG] 20.6 deep-scrub ok 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:49.332667+0000 mon.a (mon.0) 1176 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:49.465523+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:49.468405+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:49.563962+0000 mon.a (mon.0) 1177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-83977-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:49.564122+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:49.710305+0000 mon.b (mon.1) 173 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:49.854624+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:50.078405+0000 mon.b (mon.1) 174 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:50.305578+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:50.305657+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:50.319415+0000 mon.b (mon.1) 175 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-13"}]: dispatch 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:50.327804+0000 mon.a (mon.0) 1182 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-13"}]: dispatch 2024-02-13T00:41:50.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:50.329964+0000 mon.c (mon.2) 140 : audit [INF] from='client.? 172.21.15.162:0/2910215930' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-83985-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:50.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:50.330786+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-83985-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:50.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:48.644151+0000 osd.0 (osd.0) 63 : cluster [DBG] 6.0 scrub starts 2024-02-13T00:41:50.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:48.645011+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.0 scrub ok 2024-02-13T00:41:50.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:49.014417+0000 osd.1 (osd.1) 79 : cluster [DBG] 20.6 deep-scrub starts 2024-02-13T00:41:50.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:49.051358+0000 osd.1 (osd.1) 80 : cluster [DBG] 20.6 deep-scrub ok 2024-02-13T00:41:50.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:49.332667+0000 mon.a (mon.0) 1176 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:41:50.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:49.465523+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:41:50.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:49.468405+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:41:50.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:49.563962+0000 mon.a (mon.0) 1177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-83977-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:50.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:49.564122+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-02-13T00:41:50.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:49.710305+0000 mon.b (mon.1) 173 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:50.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:49.854624+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:41:50.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:50.078405+0000 mon.b (mon.1) 174 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:50.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:50.305578+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:41:50.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:50.305657+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-02-13T00:41:50.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:50.319415+0000 mon.b (mon.1) 175 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-13"}]: dispatch 2024-02-13T00:41:50.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:50.327804+0000 mon.a (mon.0) 1182 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-13"}]: dispatch 2024-02-13T00:41:50.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:50.329964+0000 mon.c (mon.2) 140 : audit [INF] from='client.? 172.21.15.162:0/2910215930' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-83985-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:50.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:50.330786+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-83985-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:51.216 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:51.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:50.309849+0000 mgr.y (mgr.14544) 187 : cluster [DBG] pgmap v203: 364 pgs: 32 unknown, 332 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 139 KiB/s wr, 135 op/s 2024-02-13T00:41:51.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:51.079678+0000 mon.b (mon.1) 176 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:51.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:51.307154+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-13"}]': finished 2024-02-13T00:41:51.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:51.307240+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-83985-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:51.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:51.307329+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-02-13T00:41:51.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:51.312324+0000 mon.b (mon.1) 177 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-13", "mode": "writeback"}]: dispatch 2024-02-13T00:41:51.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:51.316173+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-13", "mode": "writeback"}]: dispatch 2024-02-13T00:41:51.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:51.316427+0000 mon.b (mon.1) 178 : audit [INF] from='client.? 172.21.15.162:0/62312231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-83977-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:51.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:51.317037+0000 mon.a (mon.0) 1188 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-83977-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:51.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:50.309849+0000 mgr.y (mgr.14544) 187 : cluster [DBG] pgmap v203: 364 pgs: 32 unknown, 332 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 139 KiB/s wr, 135 op/s 2024-02-13T00:41:51.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:51.079678+0000 mon.b (mon.1) 176 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:51.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:51.307154+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-13"}]': finished 2024-02-13T00:41:51.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:51.307240+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-83985-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:51.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:51.307329+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-02-13T00:41:51.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:51.312324+0000 mon.b (mon.1) 177 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-13", "mode": "writeback"}]: dispatch 2024-02-13T00:41:51.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:51.316173+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-13", "mode": "writeback"}]: dispatch 2024-02-13T00:41:51.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:51.316427+0000 mon.b (mon.1) 178 : audit [INF] from='client.? 172.21.15.162:0/62312231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-83977-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:51.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:51.317037+0000 mon.a (mon.0) 1188 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-83977-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:50.309849+0000 mgr.y (mgr.14544) 187 : cluster [DBG] pgmap v203: 364 pgs: 32 unknown, 332 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 139 KiB/s wr, 135 op/s 2024-02-13T00:41:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:51.079678+0000 mon.b (mon.1) 176 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:51.307154+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-13"}]': finished 2024-02-13T00:41:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:51.307240+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-83985-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:51.307329+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-02-13T00:41:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:51.312324+0000 mon.b (mon.1) 177 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-13", "mode": "writeback"}]: dispatch 2024-02-13T00:41:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:51.316173+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-13", "mode": "writeback"}]: dispatch 2024-02-13T00:41:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:51.316427+0000 mon.b (mon.1) 178 : audit [INF] from='client.? 172.21.15.162:0/62312231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-83977-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:51.317037+0000 mon.a (mon.0) 1188 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-83977-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:51.967 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:51.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:52.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:52.104 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:52.126 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:52.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:52.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:52.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:52.263 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:52.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:52.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:52.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:52.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:52.480 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:50.671598+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:41:52.480 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:50.675149+0000 osd.3 (osd.3) 108 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:41:52.480 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:51.412899+0000 osd.5 (osd.5) 81 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:41:52.480 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:51.414088+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:41:52.480 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:52.080634+0000 mon.b (mon.1) 179 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:52.480 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:52.303928+0000 mon.a (mon.0) 1189 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:52.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:52.310015+0000 mon.a (mon.0) 1190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-13", "mode": "writeback"}]': finished 2024-02-13T00:41:52.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:52.310152+0000 mon.a (mon.0) 1191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-83977-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:52.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:52.310304+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-02-13T00:41:52.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:52.316097+0000 mon.a (mon.0) 1193 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:52.482 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:52.486 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:50.671598+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:41:52.486 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:50.675149+0000 osd.3 (osd.3) 108 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:41:52.486 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:51.412899+0000 osd.5 (osd.5) 81 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:41:52.486 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:51.414088+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:41:52.486 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:52.080634+0000 mon.b (mon.1) 179 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:52.486 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:52.303928+0000 mon.a (mon.0) 1189 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:52.486 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:52.310015+0000 mon.a (mon.0) 1190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-13", "mode": "writeback"}]': finished 2024-02-13T00:41:52.486 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:52.310152+0000 mon.a (mon.0) 1191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-83977-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:52.486 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:52.310304+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-02-13T00:41:52.486 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:52.316097+0000 mon.a (mon.0) 1193 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:52.487 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:50.671598+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:41:52.487 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:50.675149+0000 osd.3 (osd.3) 108 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:41:52.487 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:51.412899+0000 osd.5 (osd.5) 81 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:41:52.487 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:51.414088+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:41:52.487 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:52.080634+0000 mon.b (mon.1) 179 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:52.487 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:52.303928+0000 mon.a (mon.0) 1189 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:41:52.487 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:52.310015+0000 mon.a (mon.0) 1190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-13", "mode": "writeback"}]': finished 2024-02-13T00:41:52.487 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:52.310152+0000 mon.a (mon.0) 1191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-83977-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:52.487 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:52.310304+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-02-13T00:41:52.487 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:52.316097+0000 mon.a (mon.0) 1193 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:52.501 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:41:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:52.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:41:52.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:52.633 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:52.656 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:52.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:52.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:52.790 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:52.864 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:52.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:52.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:52.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:52.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:41:52.996 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:53.005 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:41:53.005 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:41:53.005 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:41:53.005 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:41:53.005 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:41:53.005 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:41:53.005 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:41:53.005 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:41:53.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.474+0000 7fc3bdba7700 1 -- 172.21.15.162:0/2609014045 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80f6b20 msgr2=0x7fc3b8103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.474+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/2609014045 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80f6b20 0x7fc3b8103120 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7fc3b0009c30 tx=0x7fc3b0004c50).stop 2024-02-13T00:41:53.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.475+0000 7fc3bdba7700 1 -- 172.21.15.162:0/2609014045 shutdown_connections 2024-02-13T00:41:53.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.475+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/2609014045 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80f6b20 0x7fc3b8103120 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.475+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/2609014045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3b80fc2b0 0x7fc3b80fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.475+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/2609014045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fcc10 0x7fc3b80f6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.475+0000 7fc3bdba7700 1 -- 172.21.15.162:0/2609014045 >> 172.21.15.162:0/2609014045 conn(0x7fc3b8070080 msgr2=0x7fc3b80f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.475+0000 7fc3bdba7700 1 -- 172.21.15.162:0/2609014045 shutdown_connections 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.475+0000 7fc3bdba7700 1 -- 172.21.15.162:0/2609014045 wait complete. 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.475+0000 7fc3bdba7700 1 librados: starting msgr at 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.475+0000 7fc3bdba7700 1 librados: starting objecter 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.475+0000 7fc3bdba7700 1 Processor -- start 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3bdba7700 1 -- start start 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3bdba7700 1 librados: setting wanted keys 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3bdba7700 1 librados: calling monclient init 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3bdba7700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80f6b20 0x7fc3b810df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3bdba7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3b80fc2b0 0x7fc3b810e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:53.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3bdba7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fcc10 0x7fc3b81080d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:53.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3bdba7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b81085c0 con 0x7fc3b80fcc10 2024-02-13T00:41:53.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3bdba7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b8108730 con 0x7fc3b80fc2b0 2024-02-13T00:41:53.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3bdba7700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b81088f0 con 0x7fc3b80f6b20 2024-02-13T00:41:53.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3b6ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3b80fc2b0 0x7fc3b810e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:53.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3b77fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80f6b20 0x7fc3b810df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:53.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3b7fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fcc10 0x7fc3b81080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:53.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3b7fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fcc10 0x7fc3b81080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:54538/0 (socket says 172.21.15.162:54538) 2024-02-13T00:41:53.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3b77fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80f6b20 0x7fc3b810df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:38240/0 (socket says 172.21.15.162:38240) 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.476+0000 7fc3b7fff700 1 -- 172.21.15.162:0/3805197952 learned_addr learned my addr 172.21.15.162:0/3805197952 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3b77fe700 1 -- 172.21.15.162:0/3805197952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3b80fc2b0 msgr2=0x7fc3b810e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3b77fe700 1 --2- 172.21.15.162:0/3805197952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3b80fc2b0 0x7fc3b810e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3b77fe700 1 -- 172.21.15.162:0/3805197952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fcc10 msgr2=0x7fc3b81080d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3b77fe700 1 --2- 172.21.15.162:0/3805197952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fcc10 0x7fc3b81080d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3b77fe700 1 -- 172.21.15.162:0/3805197952 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a8005040 con 0x7fc3b80f6b20 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3b77fe700 1 --2- 172.21.15.162:0/3805197952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80f6b20 0x7fc3b810df30 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7fc3a800a470 tx=0x7fc3a800adc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3b4ff9700 10 client.?.objecter ms_handle_connect 0x7fc3b80f6b20 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3b4ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a80079e0 con 0x7fc3b80f6b20 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3a8010040 con 0x7fc3b80f6b20 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8015780 con 0x7fc3b80f6b20 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b00097c0 con 0x7fc3b80f6b20 2024-02-13T00:41:53.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3bdba7700 10 client.36500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b8108ee0 con 0x7fc3b80f6b20 2024-02-13T00:41:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.477+0000 7fc3bdba7700 1 librados: init done 2024-02-13T00:41:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.478+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc3b805fb80 con 0x7fc3b80f6b20 2024-02-13T00:41:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.480+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc3a8015bb0 con 0x7fc3b80f6b20 2024-02-13T00:41:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.480+0000 7fc3b4ff9700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc3a0062c90 0x7fc3a0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.480+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(157..157 src has 52..157) v4 ==== 9703+0+0 (secure 0 0 0) 0x7fc3a807d4c0 con 0x7fc3b80f6b20 2024-02-13T00:41:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.480+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 osd_map(157..157 src has 52..157) v4 2024-02-13T00:41:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.480+0000 7fc3b4ff9700 3 client.36500.objecter handle_osd_map got epochs [157,157] > 0 2024-02-13T00:41:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.480+0000 7fc3b4ff9700 3 client.36500.objecter handle_osd_map decoding full epoch 157 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.480+0000 7fc3b6ffd700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc3a0062c90 0x7fc3a0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.480+0000 7fc3b4ff9700 20 client.36500.objecter dump_active .. 0 homeless 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.480+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v157) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc3a807d8e0 con 0x7fc3b80f6b20 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3b4ff9700 10 client.36500.objecter ms_handle_connect 0x7fc3a0062c90 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3b6ffd700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc3a0062c90 0x7fc3a0065150 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 rx=0x7fc3b8109870 tx=0x7fc3ac009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 1 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc3b80fd580 0x7fc3b80fd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8108b10 osd=0 3 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8108b10 osd=0 3 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8108b10 osd=0 3 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 1 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b8108b10 osd=0 3 2024-02-13T00:41:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 0 1 2024-02-13T00:41:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8108b10 osd=0 4 2024-02-13T00:41:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3bdba7700 10 client.36500.objecter _send_command 1 2024-02-13T00:41:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b80475e0 con 0x7fc3b80fd580 2024-02-13T00:41:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.481+0000 7fc3b7fff700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc3b80fd580 0x7fc3b80fd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.482+0000 7fc3b7fff700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc3b80fd580 0x7fc3b80fd960 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.482+0000 7fc3b4ff9700 10 client.36500.objecter ms_handle_connect 0x7fc3b80fd580 2024-02-13T00:41:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.483+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b80475e0 con 0x7fc3b80fd580 2024-02-13T00:41:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.483+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:41:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.483+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 1 = system:0 2024-02-13T00:41:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.483+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b8108b10 osd=0 4 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.483+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 0 1 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.501+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.501+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 2 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.501+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8108b10 osd=0 3 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.501+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8108b10 osd=0 3 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.501+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8108b10 osd=0 3 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.501+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 2 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.501+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b8108b10 osd=0 3 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.501+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 0 2 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.501+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8108b10 osd=0 4 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.501+0000 7fc3bdba7700 10 client.36500.objecter _send_command 2 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.501+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b8105040 con 0x7fc3b80fd580 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.503+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b8105040 con 0x7fc3b80fd580 2024-02-13T00:41:53.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.503+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.503+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.503+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b8108b10 osd=0 4 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.503+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 0 2 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 3 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6805/150599780 conn(0x7fc3b8059990 0x7fc3b8059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8028590 osd=1 3 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8028590 osd=1 3 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8028590 osd=1 3 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 3 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b8028590 osd=1 3 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 1 3 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8028590 osd=1 4 2024-02-13T00:41:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3bdba7700 10 client.36500.objecter _send_command 3 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b8028700 con 0x7fc3b8059990 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3b77fe700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6805/150599780 conn(0x7fc3b8059990 0x7fc3b8059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3b4ff9700 10 client.36500.objecter ms_handle_connect 0x7fc3b8059990 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.504+0000 7fc3b77fe700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6805/150599780 conn(0x7fc3b8059990 0x7fc3b8059d70 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.505+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b8028700 con 0x7fc3b8059990 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.505+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.505+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 3 = system:0 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.505+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b8028590 osd=1 4 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.505+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 1 3 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.524+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.524+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 4 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.524+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8028590 osd=1 3 2024-02-13T00:41:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.524+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8028590 osd=1 3 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.524+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8028590 osd=1 3 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.524+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 4 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.524+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b8028590 osd=1 3 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.524+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 1 4 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.524+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8028590 osd=1 4 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.524+0000 7fc3bdba7700 10 client.36500.objecter _send_command 4 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.524+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b8059400 con 0x7fc3b8059990 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b8059400 con 0x7fc3b8059990 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b8028590 osd=1 4 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 1 4 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 5 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc3b80280f0 0x7fc3b811e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8059810 osd=2 3 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8059810 osd=2 3 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8059810 osd=2 3 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 5 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b8059810 osd=2 3 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 2 5 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8059810 osd=2 4 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3bdba7700 10 client.36500.objecter _send_command 5 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.525+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b8120f60 con 0x7fc3b80280f0 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.526+0000 7fc3b6ffd700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc3b80280f0 0x7fc3b811e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.526+0000 7fc3b4ff9700 10 client.36500.objecter ms_handle_connect 0x7fc3b80280f0 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.526+0000 7fc3b6ffd700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc3b80280f0 0x7fc3b811e310 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.527+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b8120f60 con 0x7fc3b80280f0 2024-02-13T00:41:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.527+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:41:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.527+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 5 = system:0 2024-02-13T00:41:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.527+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b8059810 osd=2 4 2024-02-13T00:41:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.527+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 2 5 2024-02-13T00:41:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.544+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.544+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 6 2024-02-13T00:41:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.544+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8059810 osd=2 3 2024-02-13T00:41:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.544+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8059810 osd=2 3 2024-02-13T00:41:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.544+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8059810 osd=2 3 2024-02-13T00:41:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.544+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 6 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.544+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b8059810 osd=2 3 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.544+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 2 6 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.544+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8059810 osd=2 4 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.544+0000 7fc3bdba7700 10 client.36500.objecter _send_command 6 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.544+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b8027650 con 0x7fc3b80280f0 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.546+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b8027650 con 0x7fc3b80280f0 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.546+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.546+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.546+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b8059810 osd=2 4 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.546+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 2 6 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 7 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6813/604971011 conn(0x7fc3b811be60 0x7fc3b811f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:53.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8027a60 osd=3 3 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8027a60 osd=3 3 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8027a60 osd=3 3 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 7 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b8027a60 osd=3 3 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 3 7 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8027a60 osd=3 4 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3bdba7700 10 client.36500.objecter _send_command 7 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b811ff60 con 0x7fc3b811be60 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3b7fff700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6813/604971011 conn(0x7fc3b811be60 0x7fc3b811f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3b4ff9700 10 client.36500.objecter ms_handle_connect 0x7fc3b811be60 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.547+0000 7fc3b7fff700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6813/604971011 conn(0x7fc3b811be60 0x7fc3b811f980 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.548+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b811ff60 con 0x7fc3b811be60 2024-02-13T00:41:53.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.548+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.548+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 7 = system:0 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.548+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b8027a60 osd=3 4 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.548+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 3 7 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.566+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.566+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 8 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.566+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8027a60 osd=3 3 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.566+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8027a60 osd=3 3 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.566+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8027a60 osd=3 3 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.566+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 8 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.566+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b8027a60 osd=3 3 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.566+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 3 8 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.566+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8027a60 osd=3 4 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.566+0000 7fc3bdba7700 10 client.36500.objecter _send_command 8 2024-02-13T00:41:53.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.566+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b811b3c0 con 0x7fc3b811be60 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.568+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b811b3c0 con 0x7fc3b811be60 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.568+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.568+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.568+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b8027a60 osd=3 4 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.568+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 3 8 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 9 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc3b811b530 0x7fc3b81250f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b811b3c0 osd=4 3 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b811b3c0 osd=4 3 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b811b3c0 osd=4 3 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 9 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b811b3c0 osd=4 3 2024-02-13T00:41:53.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 4 9 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b811b3c0 osd=4 4 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3bdba7700 10 client.36500.objecter _send_command 9 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b8127d80 con 0x7fc3b811b530 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3b77fe700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc3b811b530 0x7fc3b81250f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3b4ff9700 10 client.36500.objecter ms_handle_connect 0x7fc3b811b530 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.569+0000 7fc3b77fe700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc3b811b530 0x7fc3b81250f0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.570+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b8127d80 con 0x7fc3b811b530 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.570+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.570+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 9 = system:0 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.570+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b811b3c0 osd=4 4 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.570+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 4 9 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.588+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.588+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 10 2024-02-13T00:41:53.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.588+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b811b3c0 osd=4 3 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.588+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b811b3c0 osd=4 3 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.588+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b811b3c0 osd=4 3 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.588+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 10 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.588+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b811b3c0 osd=4 3 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.588+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 4 10 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.588+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b811b3c0 osd=4 4 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.588+0000 7fc3bdba7700 10 client.36500.objecter _send_command 10 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.588+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b8127d80 con 0x7fc3b811b530 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.589+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b8127d80 con 0x7fc3b811b530 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.589+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.589+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.589+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b811b3c0 osd=4 4 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.589+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 4 10 2024-02-13T00:41:53.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 11 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc3b8125af0 0x7fc3b8125f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8127d80 osd=5 3 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8127d80 osd=5 3 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8127d80 osd=5 3 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 11 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b8127d80 osd=5 3 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 5 11 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8127d80 osd=5 4 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3bdba7700 10 client.36500.objecter _send_command 11 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b8127bb0 con 0x7fc3b8125af0 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3b6ffd700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc3b8125af0 0x7fc3b8125f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3b4ff9700 10 client.36500.objecter ms_handle_connect 0x7fc3b8125af0 2024-02-13T00:41:53.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.590+0000 7fc3b6ffd700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc3b8125af0 0x7fc3b8125f30 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.592+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b8127bb0 con 0x7fc3b8125af0 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.592+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.592+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 11 = system:0 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.592+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b8127d80 osd=5 4 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.592+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 5 11 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.609+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.609+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 12 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.609+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8127d80 osd=5 3 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.609+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8127d80 osd=5 3 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.609+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8127d80 osd=5 3 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.609+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 12 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.609+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b8127d80 osd=5 3 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.609+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 5 12 2024-02-13T00:41:53.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.609+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8127d80 osd=5 4 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.609+0000 7fc3bdba7700 10 client.36500.objecter _send_command 12 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.609+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b8127bb0 con 0x7fc3b8125af0 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b8127bb0 con 0x7fc3b8125af0 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b8127d80 osd=5 4 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 5 12 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 13 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc3b812a4c0 0x7fc3b812a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8127bb0 osd=6 3 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8127bb0 osd=6 3 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8127bb0 osd=6 3 2024-02-13T00:41:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 13 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b8127bb0 osd=6 3 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 6 13 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8127bb0 osd=6 4 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3bdba7700 10 client.36500.objecter _send_command 13 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b812af20 con 0x7fc3b812a4c0 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3b7fff700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc3b812a4c0 0x7fc3b812a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3b4ff9700 10 client.36500.objecter ms_handle_connect 0x7fc3b812a4c0 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.612+0000 7fc3b7fff700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc3b812a4c0 0x7fc3b812a900 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.613+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b812af20 con 0x7fc3b812a4c0 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.613+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.613+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 13 = system:0 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.613+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b8127bb0 osd=6 4 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.613+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 6 13 2024-02-13T00:41:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.629+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.629+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 14 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.629+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8127bb0 osd=6 3 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.629+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8127bb0 osd=6 3 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.629+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b8127bb0 osd=6 3 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.629+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 14 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.629+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b8127bb0 osd=6 3 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.629+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 6 14 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.629+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8127bb0 osd=6 4 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.629+0000 7fc3bdba7700 10 client.36500.objecter _send_command 14 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.629+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b812af20 con 0x7fc3b812a4c0 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b812af20 con 0x7fc3b812a4c0 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b8127bb0 osd=6 4 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 6 14 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 15 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc3b812d800 0x7fc3b812dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b812af20 osd=7 3 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b812af20 osd=7 3 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b812af20 osd=7 3 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 15 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b812af20 osd=7 3 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 7 15 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b812af20 osd=7 4 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3bdba7700 10 client.36500.objecter _send_command 15 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b8131560 con 0x7fc3b812d800 2024-02-13T00:41:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3b77fe700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc3b812d800 0x7fc3b812dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3b4ff9700 10 client.36500.objecter ms_handle_connect 0x7fc3b812d800 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.631+0000 7fc3b77fe700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc3b812d800 0x7fc3b812dc20 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.632+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b8131560 con 0x7fc3b812d800 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.632+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.632+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 15 = system:0 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.632+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b812af20 osd=7 4 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.632+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 7 15 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.649+0000 7fc3bdba7700 10 client.36500.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.649+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign -1 16 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.649+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b812af20 osd=7 3 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.649+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b812af20 osd=7 3 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.649+0000 7fc3bdba7700 20 client.36500.objecter _get_session s=0x7fc3b812af20 osd=7 3 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.649+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_remove -1 16 2024-02-13T00:41:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.649+0000 7fc3bdba7700 20 client.36500.objecter get_session s=0x7fc3b812af20 osd=7 3 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.649+0000 7fc3bdba7700 15 client.36500.objecter _session_command_op_assign 7 16 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.649+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b812af20 osd=7 4 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.649+0000 7fc3bdba7700 10 client.36500.objecter _send_command 16 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.649+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b8126510 con 0x7fc3b812d800 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3b4ff9700 1 -- 172.21.15.162:0/3805197952 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b8126510 con 0x7fc3b812d800 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3b4ff9700 10 client.36500.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3b4ff9700 10 client.36500.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3b4ff9700 20 client.36500.objecter put_session s=0x7fc3b812af20 osd=7 4 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3b4ff9700 15 client.36500.objecter _session_command_op_remove 7 16 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 10 librados: watch_flush enter 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 10 librados: watch_flush exit 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 10 client.36500.objecter close_session for osd.0 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc3b80fd580 msgr2=0x7fc3b80fd960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc3b80fd580 0x7fc3b80fd960 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8108b10 osd=0 1 2024-02-13T00:41:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 10 client.36500.objecter close_session for osd.1 2024-02-13T00:41:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6805/150599780 conn(0x7fc3b8059990 msgr2=0x7fc3b8059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6805/150599780 conn(0x7fc3b8059990 0x7fc3b8059d70 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8028590 osd=1 1 2024-02-13T00:41:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 10 client.36500.objecter close_session for osd.2 2024-02-13T00:41:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc3b80280f0 msgr2=0x7fc3b811e310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc3b80280f0 0x7fc3b811e310 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8059810 osd=2 1 2024-02-13T00:41:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 10 client.36500.objecter close_session for osd.3 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6813/604971011 conn(0x7fc3b811be60 msgr2=0x7fc3b811f980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3b7fff700 1 -- 172.21.15.162:0/3805197952 reap_dead start 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6813/604971011 conn(0x7fc3b811be60 0x7fc3b811f980 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8027a60 osd=3 1 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 10 client.36500.objecter close_session for osd.4 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc3b811b530 msgr2=0x7fc3b81250f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc3b811b530 0x7fc3b81250f0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b811b3c0 osd=4 1 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 10 client.36500.objecter close_session for osd.5 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc3b8125af0 msgr2=0x7fc3b8125f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc3b8125af0 0x7fc3b8125f30 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8127d80 osd=5 1 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 10 client.36500.objecter close_session for osd.6 2024-02-13T00:41:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc3b812a4c0 msgr2=0x7fc3b812a900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc3b812a4c0 0x7fc3b812a900 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b8127bb0 osd=6 1 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 10 client.36500.objecter close_session for osd.7 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc3b812d800 msgr2=0x7fc3b812dc20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc3b812d800 0x7fc3b812dc20 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 20 client.36500.objecter put_session s=0x7fc3b812af20 osd=7 1 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 20 client.36500.objecter shutdown clearing up homeless session... 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.651+0000 7fc3bdba7700 10 client.36500.objecter successfully canceled tick 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.652+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc3a0062c90 msgr2=0x7fc3a0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.652+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc3a0062c90 0x7fc3a0065150 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 rx=0x7fc3b8109870 tx=0x7fc3ac009040).stop 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.652+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80f6b20 msgr2=0x7fc3b810df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.652+0000 7fc3bdba7700 1 --2- 172.21.15.162:0/3805197952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80f6b20 0x7fc3b810df30 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7fc3a800a470 tx=0x7fc3a800adc0).stop 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.652+0000 7fc3b7fff700 1 -- 172.21.15.162:0/3805197952 reap_dead start 2024-02-13T00:41:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.652+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 shutdown_connections 2024-02-13T00:41:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.652+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 >> 172.21.15.162:0/3805197952 conn(0x7fc3b8070080 msgr2=0x7fc3b806d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.652+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 shutdown_connections 2024-02-13T00:41:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.652+0000 7fc3bdba7700 1 -- 172.21.15.162:0/3805197952 wait complete. 2024-02-13T00:41:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:52.652+0000 7fc3bdba7700 1 librados: shutdown 2024-02-13T00:41:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:41:53.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:52.313999+0000 mgr.y (mgr.14544) 188 : cluster [DBG] pgmap v206: 356 pgs: 32 unknown, 324 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 131 KiB/s wr, 134 op/s 2024-02-13T00:41:53.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:52.409773+0000 osd.6 (osd.6) 63 : cluster [DBG] 4.5 deep-scrub starts 2024-02-13T00:41:53.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:52.415899+0000 osd.6 (osd.6) 64 : cluster [DBG] 4.5 deep-scrub ok 2024-02-13T00:41:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:52.444033+0000 mon.c (mon.2) 141 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:41:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:52.479883+0000 mon.c (mon.2) 142 : audit [DBG] from='client.? 172.21.15.162:0/3805197952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:41:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:52.514794+0000 osd.2 (osd.2) 71 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:41:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:52.517384+0000 osd.2 (osd.2) 72 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:41:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:52.594559+0000 osd.7 (osd.7) 80 : cluster [DBG] 20.9 deep-scrub starts 2024-02-13T00:41:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:52.604433+0000 osd.7 (osd.7) 81 : cluster [DBG] 20.9 deep-scrub ok 2024-02-13T00:41:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.050315+0000 mon.c (mon.2) 143 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:41:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.051000+0000 mon.c (mon.2) 144 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:41:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.056492+0000 mon.a (mon.0) 1194 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:41:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.071064+0000 mon.b (mon.1) 180 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-13T00:41:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.071771+0000 mon.b (mon.1) 181 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-13T00:41:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.072191+0000 mon.b (mon.1) 182 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-13T00:41:53.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.072620+0000 mon.b (mon.1) 183 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-13T00:41:53.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.073052+0000 mon.b (mon.1) 184 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-13T00:41:53.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:52.313999+0000 mgr.y (mgr.14544) 188 : cluster [DBG] pgmap v206: 356 pgs: 32 unknown, 324 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 131 KiB/s wr, 134 op/s 2024-02-13T00:41:53.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:52.409773+0000 osd.6 (osd.6) 63 : cluster [DBG] 4.5 deep-scrub starts 2024-02-13T00:41:53.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:52.415899+0000 osd.6 (osd.6) 64 : cluster [DBG] 4.5 deep-scrub ok 2024-02-13T00:41:53.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:52.444033+0000 mon.c (mon.2) 141 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:41:53.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:52.479883+0000 mon.c (mon.2) 142 : audit [DBG] from='client.? 172.21.15.162:0/3805197952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:41:53.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:52.514794+0000 osd.2 (osd.2) 71 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:41:53.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:52.517384+0000 osd.2 (osd.2) 72 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:41:53.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:52.594559+0000 osd.7 (osd.7) 80 : cluster [DBG] 20.9 deep-scrub starts 2024-02-13T00:41:53.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:52.604433+0000 osd.7 (osd.7) 81 : cluster [DBG] 20.9 deep-scrub ok 2024-02-13T00:41:53.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.050315+0000 mon.c (mon.2) 143 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:41:53.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.051000+0000 mon.c (mon.2) 144 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:41:53.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.056492+0000 mon.a (mon.0) 1194 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:41:53.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.071064+0000 mon.b (mon.1) 180 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-13T00:41:53.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.071771+0000 mon.b (mon.1) 181 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-13T00:41:53.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.072191+0000 mon.b (mon.1) 182 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-13T00:41:53.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.072620+0000 mon.b (mon.1) 183 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-13T00:41:53.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.073052+0000 mon.b (mon.1) 184 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-13T00:41:53.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.073486+0000 mon.b (mon.1) 185 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-13T00:41:53.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.073912+0000 mon.b (mon.1) 186 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-13T00:41:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.074306+0000 mon.b (mon.1) 187 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-13T00:41:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.074710+0000 mon.b (mon.1) 188 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-13T00:41:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.075095+0000 mon.b (mon.1) 189 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-13T00:41:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.081761+0000 mon.b (mon.1) 190 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:53.306508+0000 mon.a (mon.0) 1195 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.310417+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]': finished 2024-02-13T00:41:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:53.310508+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-02-13T00:41:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.315821+0000 mon.b (mon.1) 191 : audit [INF] from='client.? 172.21.15.162:0/891139537' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-83985-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.320206+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.325683+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-83985-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:53.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.073486+0000 mon.b (mon.1) 185 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-13T00:41:53.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.073912+0000 mon.b (mon.1) 186 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-13T00:41:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.074306+0000 mon.b (mon.1) 187 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-13T00:41:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.074710+0000 mon.b (mon.1) 188 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-13T00:41:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.075095+0000 mon.b (mon.1) 189 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-13T00:41:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.081761+0000 mon.b (mon.1) 190 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:53.306508+0000 mon.a (mon.0) 1195 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.310417+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]': finished 2024-02-13T00:41:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:53.310508+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-02-13T00:41:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.315821+0000 mon.b (mon.1) 191 : audit [INF] from='client.? 172.21.15.162:0/891139537' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-83985-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.320206+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.325683+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-83985-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:52.313999+0000 mgr.y (mgr.14544) 188 : cluster [DBG] pgmap v206: 356 pgs: 32 unknown, 324 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 131 KiB/s wr, 134 op/s 2024-02-13T00:41:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:52.409773+0000 osd.6 (osd.6) 63 : cluster [DBG] 4.5 deep-scrub starts 2024-02-13T00:41:53.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:52.415899+0000 osd.6 (osd.6) 64 : cluster [DBG] 4.5 deep-scrub ok 2024-02-13T00:41:53.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:52.444033+0000 mon.c (mon.2) 141 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:41:53.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:52.479883+0000 mon.c (mon.2) 142 : audit [DBG] from='client.? 172.21.15.162:0/3805197952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:41:53.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:52.514794+0000 osd.2 (osd.2) 71 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:41:53.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:52.517384+0000 osd.2 (osd.2) 72 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:41:53.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:52.594559+0000 osd.7 (osd.7) 80 : cluster [DBG] 20.9 deep-scrub starts 2024-02-13T00:41:53.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:52.604433+0000 osd.7 (osd.7) 81 : cluster [DBG] 20.9 deep-scrub ok 2024-02-13T00:41:53.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.050315+0000 mon.c (mon.2) 143 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:41:53.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.051000+0000 mon.c (mon.2) 144 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:41:53.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.056492+0000 mon.a (mon.0) 1194 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:41:53.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.071064+0000 mon.b (mon.1) 180 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-13T00:41:53.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.071771+0000 mon.b (mon.1) 181 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-13T00:41:53.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.072191+0000 mon.b (mon.1) 182 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-13T00:41:53.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.072620+0000 mon.b (mon.1) 183 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-13T00:41:53.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.073052+0000 mon.b (mon.1) 184 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-13T00:41:53.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.073486+0000 mon.b (mon.1) 185 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-13T00:41:53.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.073912+0000 mon.b (mon.1) 186 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-13T00:41:53.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.074306+0000 mon.b (mon.1) 187 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-13T00:41:53.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.074710+0000 mon.b (mon.1) 188 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-13T00:41:53.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.075095+0000 mon.b (mon.1) 189 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-13T00:41:53.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.081761+0000 mon.b (mon.1) 190 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:53.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:53.306508+0000 mon.a (mon.0) 1195 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:41:53.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.310417+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]': finished 2024-02-13T00:41:53.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:53.310508+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-02-13T00:41:53.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.315821+0000 mon.b (mon.1) 191 : audit [INF] from='client.? 172.21.15.162:0/891139537' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-83985-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:53.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.320206+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]: dispatch 2024-02-13T00:41:53.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.325683+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-83985-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:54.322 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:54.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: Running main() from gmock_main.cc 2024-02-13T00:41:54.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [==========] Running 12 tests from 4 test suites. 2024-02-13T00:41:54.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] Global test environment set-up. 2024-02-13T00:41:54.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-02-13T00:41:54.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-02-13T00:41:54.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2518 ms) 2024-02-13T00:41:54.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-02-13T00:41:54.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2970 ms) 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (3634 ms) 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2011 ms) 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (11133 ms total) 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4135 ms) 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4086 ms) 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged (8221 ms total) 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: 2024-02-13T00:41:54.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3096 ms) 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2012 ms) 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2041 ms) 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2097 ms) 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9246 ms total) 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (3962 ms) 2024-02-13T00:41:54.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-02-13T00:41:54.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (3750 ms) 2024-02-13T00:41:54.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (7712 ms total) 2024-02-13T00:41:54.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: 2024-02-13T00:41:54.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] Global test environment tear-down 2024-02-13T00:41:54.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [==========] 12 tests from 4 test suites ran. (59632 ms total) 2024-02-13T00:41:54.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ PASSED ] 12 tests. 2024-02-13T00:41:54.348 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:41:54.348 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84148 2024-02-13T00:41:54.348 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84148 2024-02-13T00:41:54.349 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:41:54.349 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84004 2024-02-13T00:41:54.349 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84004 2024-02-13T00:41:54.349 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:41:54.349 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84108 2024-02-13T00:41:54.349 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84108 2024-02-13T00:41:54.349 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:41:54.349 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84076 2024-02-13T00:41:54.349 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84076 2024-02-13T00:41:54.350 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:54.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:54.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:54.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:54.491 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:54.510 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:54.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:54.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:52.736355+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:41:54.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:52.739827+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:41:54.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.071505+0000 mgr.y (mgr.14544) 189 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-13T00:41:54.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.072036+0000 mgr.y (mgr.14544) 190 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-13T00:41:54.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.072462+0000 mgr.y (mgr.14544) 191 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-13T00:41:54.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.072895+0000 mgr.y (mgr.14544) 192 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-13T00:41:54.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.073315+0000 mgr.y (mgr.14544) 193 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-13T00:41:54.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.073750+0000 mgr.y (mgr.14544) 194 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-13T00:41:54.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.074182+0000 mgr.y (mgr.14544) 195 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-13T00:41:54.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.074565+0000 mgr.y (mgr.14544) 196 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-13T00:41:54.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.074973+0000 mgr.y (mgr.14544) 197 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-13T00:41:54.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:53.075355+0000 mgr.y (mgr.14544) 198 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-13T00:41:54.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:53.158417+0000 osd.1 (osd.1) 81 : cluster [DBG] 95.8 scrub starts 2024-02-13T00:41:54.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:53.162164+0000 osd.1 (osd.1) 82 : cluster [DBG] 95.8 scrub ok 2024-02-13T00:41:54.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:53.407743+0000 osd.5 (osd.5) 83 : cluster [DBG] 95.3 scrub starts 2024-02-13T00:41:54.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:53.411300+0000 osd.5 (osd.5) 84 : cluster [DBG] 95.3 scrub ok 2024-02-13T00:41:54.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:53.413354+0000 osd.6 (osd.6) 65 : cluster [DBG] 95.0 scrub starts 2024-02-13T00:41:54.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:53.416818+0000 osd.6 (osd.6) 66 : cluster [DBG] 95.0 scrub ok 2024-02-13T00:41:54.582 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:53.623009+0000 osd.7 (osd.7) 82 : cluster [DBG] 95.2 scrub starts 2024-02-13T00:41:54.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:52.736355+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:41:54.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:52.739827+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:41:54.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.071505+0000 mgr.y (mgr.14544) 189 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-13T00:41:54.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.072036+0000 mgr.y (mgr.14544) 190 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-13T00:41:54.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.072462+0000 mgr.y (mgr.14544) 191 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-13T00:41:54.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.072895+0000 mgr.y (mgr.14544) 192 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-13T00:41:54.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.073315+0000 mgr.y (mgr.14544) 193 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-13T00:41:54.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.073750+0000 mgr.y (mgr.14544) 194 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-13T00:41:54.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.074182+0000 mgr.y (mgr.14544) 195 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-13T00:41:54.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.074565+0000 mgr.y (mgr.14544) 196 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-13T00:41:54.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.074973+0000 mgr.y (mgr.14544) 197 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-13T00:41:54.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:53.075355+0000 mgr.y (mgr.14544) 198 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-13T00:41:54.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:53.158417+0000 osd.1 (osd.1) 81 : cluster [DBG] 95.8 scrub starts 2024-02-13T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:53.162164+0000 osd.1 (osd.1) 82 : cluster [DBG] 95.8 scrub ok 2024-02-13T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:53.407743+0000 osd.5 (osd.5) 83 : cluster [DBG] 95.3 scrub starts 2024-02-13T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:53.411300+0000 osd.5 (osd.5) 84 : cluster [DBG] 95.3 scrub ok 2024-02-13T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:53.413354+0000 osd.6 (osd.6) 65 : cluster [DBG] 95.0 scrub starts 2024-02-13T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:53.416818+0000 osd.6 (osd.6) 66 : cluster [DBG] 95.0 scrub ok 2024-02-13T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:53.623009+0000 osd.7 (osd.7) 82 : cluster [DBG] 95.2 scrub starts 2024-02-13T00:41:54.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:53.626708+0000 osd.7 (osd.7) 83 : cluster [DBG] 95.2 scrub ok 2024-02-13T00:41:54.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:54.082807+0000 mon.b (mon.1) 192 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:54.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:54.318833+0000 mon.a (mon.0) 1200 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]': finished 2024-02-13T00:41:54.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:54.318901+0000 mon.a (mon.0) 1201 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-83985-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:54.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:54.319004+0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-02-13T00:41:54.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:54.336488+0000 mon.a (mon.0) 1203 : audit [INF] from='client.? 172.21.15.162:0/3065351095' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-83977-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:54.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:53.626708+0000 osd.7 (osd.7) 83 : cluster [DBG] 95.2 scrub ok 2024-02-13T00:41:54.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:54.082807+0000 mon.b (mon.1) 192 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:54.318833+0000 mon.a (mon.0) 1200 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]': finished 2024-02-13T00:41:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:54.318901+0000 mon.a (mon.0) 1201 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-83985-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:54.319004+0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-02-13T00:41:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:54.336488+0000 mon.a (mon.0) 1203 : audit [INF] from='client.? 172.21.15.162:0/3065351095' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-83977-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:54.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:41:54.644 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:52.736355+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:41:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:52.739827+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:41:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.071505+0000 mgr.y (mgr.14544) 189 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-13T00:41:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.072036+0000 mgr.y (mgr.14544) 190 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-13T00:41:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.072462+0000 mgr.y (mgr.14544) 191 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-13T00:41:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.072895+0000 mgr.y (mgr.14544) 192 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-13T00:41:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.073315+0000 mgr.y (mgr.14544) 193 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-13T00:41:54.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.073750+0000 mgr.y (mgr.14544) 194 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-13T00:41:54.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.074182+0000 mgr.y (mgr.14544) 195 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-13T00:41:54.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.074565+0000 mgr.y (mgr.14544) 196 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-13T00:41:54.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.074973+0000 mgr.y (mgr.14544) 197 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-13T00:41:54.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:53.075355+0000 mgr.y (mgr.14544) 198 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-13T00:41:54.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:53.158417+0000 osd.1 (osd.1) 81 : cluster [DBG] 95.8 scrub starts 2024-02-13T00:41:54.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:53.162164+0000 osd.1 (osd.1) 82 : cluster [DBG] 95.8 scrub ok 2024-02-13T00:41:54.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:53.407743+0000 osd.5 (osd.5) 83 : cluster [DBG] 95.3 scrub starts 2024-02-13T00:41:54.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:53.411300+0000 osd.5 (osd.5) 84 : cluster [DBG] 95.3 scrub ok 2024-02-13T00:41:54.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:53.413354+0000 osd.6 (osd.6) 65 : cluster [DBG] 95.0 scrub starts 2024-02-13T00:41:54.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:53.416818+0000 osd.6 (osd.6) 66 : cluster [DBG] 95.0 scrub ok 2024-02-13T00:41:54.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:53.623009+0000 osd.7 (osd.7) 82 : cluster [DBG] 95.2 scrub starts 2024-02-13T00:41:54.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:53.626708+0000 osd.7 (osd.7) 83 : cluster [DBG] 95.2 scrub ok 2024-02-13T00:41:54.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:54.082807+0000 mon.b (mon.1) 192 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:54.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:54.318833+0000 mon.a (mon.0) 1200 : audit [INF] from='client.? 172.21.15.162:0/1722283609' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-84073-14"}]': finished 2024-02-13T00:41:54.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:54.318901+0000 mon.a (mon.0) 1201 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-83985-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:54.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:54.319004+0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-02-13T00:41:54.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:54.336488+0000 mon.a (mon.0) 1203 : audit [INF] from='client.? 172.21.15.162:0/3065351095' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-83977-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:54.843 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:41:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:54.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:41:54.882 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:41:54.882 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:41:54.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:41:54.882 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:41:54.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24739.0:2 16.1a 16:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:40:57.475809+0000", 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.401413810999998, 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.081608487, 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24739", 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2737601038", 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:57.475809+0000", 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:57.475809+0000", 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:57.475807+0000", 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6663999999999998e-05 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:57.475823+0000", 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3191e-05 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:57.475837+0000", 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3080000000000008e-06 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:57.475845+0000", 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7042351469999999 2024-02-13T00:41:54.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.180080+0000", 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8939999999999993e-06 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.180088+0000", 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3713813859999999 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.551469+0000", 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8527999999999999e-05 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.551528+0000", 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3727999999999996e-05 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.551602+0000", 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000807076 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.552409+0000", 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048816069999999996 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557290+0000", 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4697000000000007e-05 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557365+0000", 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1501999999999998e-05 2024-02-13T00:41:54.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557397+0000", 2024-02-13T00:41:54.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0664000000000001e-05 2024-02-13T00:41:54.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557417+0000", 2024-02-13T00:41:54.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.081608487 2024-02-13T00:41:54.889 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.889 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14862.0:2 12.1a 12:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:40:59.418169+0000", 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 55.459053505, 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1389136820000001, 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14862", 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4283548805", 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:41:54.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418169+0000", 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418169+0000", 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418167+0000", 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0829999999999993e-06 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418176+0000", 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2118e-05 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418188+0000", 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.859e-06 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418198+0000", 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6501e-05 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418254+0000", 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4610000000000008e-06 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418263+0000", 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.132368241 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.550631+0000", 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117495 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.550748+0000", 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128987 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.550877+0000", 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014619559999999999 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:54.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.552339+0000", 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003784141 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.556123+0000", 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087638699999999996 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557000+0000", 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3489999999999999e-05 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557043+0000", 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9342999999999999e-05 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557083+0000", 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1389136820000001 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24739.0:4 16.16 16:681f27cd:::2.obj:head [write 0~105] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:41:54.896 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.697879+0000", 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 53.179343328999998, 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059591969000000002, 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24739", 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2737601038", 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697879+0000", 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697879+0000", 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697877+0000", 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5740000000000001e-06 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697884+0000", 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7660000000000004e-06 2024-02-13T00:41:54.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697893+0000", 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5230000000000001e-06 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697899+0000", 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5326e-05 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697935+0000", 2024-02-13T00:41:54.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7998999999999997e-05 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.698003+0000", 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.038e-05 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.698083+0000", 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053553109999999998 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703438+0000", 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046529604000000002 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.749968+0000", 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007411946 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.757380+0000", 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8148e-05 2024-02-13T00:41:54.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.757428+0000", 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3118000000000003e-05 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.757471+0000", 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059591969000000002 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.902 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14862.0:5 12.12 12:4a6d8757:::3.obj:head [write 0~115] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.736730+0000", 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 53.140492055000003, 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020875984, 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14862", 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4283548805", 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736730+0000", 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736730+0000", 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736729+0000", 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9850000000000003e-06 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736735+0000", 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5379999999999997e-06 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736743+0000", 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0959999999999997e-06 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736749+0000", 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010354899999999999 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736852+0000", 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118468 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736971+0000", 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015531799999999999 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5", 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.737126+0000", 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043792969999999999 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.741505+0000", 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0087964570000000006 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.750302+0000", 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007230144 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.757532+0000", 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9459000000000003e-05 2024-02-13T00:41:54.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.757572+0000", 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4918e-05 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.757606+0000", 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020875984 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24748.0:871 18.9 18:90000000::::head [pgnls start_epoch 108] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108)", 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.656832+0000", 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.220390932999997, 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012913225, 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24748", 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2831164779", 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 871 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.656832+0000", 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.656832+0000", 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.656830+0000", 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0709999999999997e-06 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.656837+0000", 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026563569999999998 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.659494+0000", 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9329999999999996e-06 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.659503+0000", 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010096864000000001 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.669600+0000", 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9172999999999999e-05 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.669640+0000", 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105303 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.669745+0000", 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012913225 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.912 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "pg_update_log_missing(19.19 epoch 108/105 rep_tid 223 entries 108'2 (0'0) error 19:9bb10e29:test-rados-api-smithi162-84033-4::foo:head by client.14880.0:7 0.000000 -17 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.681756+0000", 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.195466320000001, 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012686235000000001, 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.681756+0000", 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.681756+0000", 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.681755+0000", 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7737e-05 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.681773+0000", 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4688999999999999e-05 2024-02-13T00:41:54.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.681788+0000", 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1689999999999992e-06 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.681796+0000", 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034450150000000001 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.685241+0000", 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092015529999999995 2024-02-13T00:41:54.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.694442+0000", 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012686235000000001 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24926.0:3 16.16 16:681f27cd:::2.obj:head [write 0~105] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.787450+0000", 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.089772089999997, 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0364658, 2024-02-13T00:41:54.916 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.917 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.917 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.917 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24926", 2024-02-13T00:41:54.917 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2404423687", 2024-02-13T00:41:54.917 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3 2024-02-13T00:41:54.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.917 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787450+0000", 2024-02-13T00:41:54.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787450+0000", 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787448+0000", 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6799999999999999e-06 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787457+0000", 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0567999999999999e-05 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787468+0000", 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6150000000000002e-06 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787474+0000", 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0272999999999998e-05 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787525+0000", 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8916000000000001e-05 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787604+0000", 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012292999999999999 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787726+0000", 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010954789999999999 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.788822+0000", 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0079658809999999993 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.796788+0000", 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027043886 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.823832+0000", 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6207e-05 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.823878+0000", 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8303999999999998e-05 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.823916+0000", 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0364658 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14886.0:1809 20.14 e108/105 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head v 108'882, mlcod=108'880)", 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.004814+0000", 2024-02-13T00:41:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.872408210000003, 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02470756, 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.004814+0000", 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.004814+0000", 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:41:54.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.004814+0000", 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3890000000000004e-06 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.004819+0000", 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1920000000000002e-06 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.004826+0000", 2024-02-13T00:41:54.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3960000000000003e-06 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.004832+0000", 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011069244000000001 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.015901+0000", 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1639000000000001e-05 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.015923+0000", 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013572752 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.029496+0000", 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6227000000000001e-05 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.029522+0000", 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02470756 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24770.0:1 33.7s0 33:f1dcbfa7:test-rados-api-smithi162-84113-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.018769+0000", 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.858453193999999, 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0820363020000001, 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24770", 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2725867940", 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.018769+0000", 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.018769+0000", 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999909 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.018760+0000", 2024-02-13T00:41:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7418000000000002e-05 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.018778+0000", 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5858e-05 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.018793+0000", 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2320000000000001e-06 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.018802+0000", 2024-02-13T00:41:54.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.516974135 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.535776+0000", 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2260000000000001e-06 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.535779+0000", 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.52173712800000005 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.057516+0000", 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012353 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.057640+0000", 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000412645 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.058052+0000", 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014702148999999999 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.072754+0000", 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027966924000000001 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.100721+0000", 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4263000000000003e-05 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.100806+0000", 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0820363020000001 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.932 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24772.0:5 25.d e109/105 25:b3257de0:test-rados-api-smithi162-84073-3::foo:head v 109'1, mlcod=0'0)", 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.031452+0000", 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.845770584, 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014190201, 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.031452+0000", 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.031452+0000", 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.031451+0000", 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6210000000000001e-06 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.031457+0000", 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2319999999999998e-06 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.031463+0000", 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5459999999999997e-06 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.031470+0000", 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044856189999999997 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.035955+0000", 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8810000000000001e-05 2024-02-13T00:41:54.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.035974+0000", 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0096325679999999993 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.045607+0000", 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5287000000000003e-05 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.045642+0000", 2024-02-13T00:41:54.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014190201 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14886.0:2059 20.6 e109/105 20:661c4854:test-rados-api-smithi162-84049-16::big:head v 109'1, mlcod=0'0)", 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.270780+0000", 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.606442788999999, 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059378140000000003, 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.270780+0000", 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.270780+0000", 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.270777+0000", 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016126680000000001 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.286904+0000", 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3373000000000001e-05 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.286927+0000", 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4949999999999997e-06 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.286937+0000", 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1576999999999997e-05 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.286998+0000", 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1797000000000002e-05 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.287020+0000", 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043100139000000003 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.330120+0000", 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7494999999999999e-05 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.330158+0000", 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059378140000000003 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2061 20.17 20:e88892cc:test-rados-api-smithi162-84049-16::big.copy:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.334388+0000", 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.542834310000003, 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.071394319999999997, 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.941 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2061 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334388+0000", 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334388+0000", 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334386+0000", 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1179999999999998e-06 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334395+0000", 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3471e-05 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334408+0000", 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2110000000000001e-06 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334416+0000", 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5129999999999997e-05 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334451+0000", 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7012e-05 2024-02-13T00:41:54.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334538+0000", 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055863379999999997 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.390401+0000", 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010490476 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:54.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.400892+0000", 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000425387 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.401317+0000", 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004380353 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405697+0000", 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5701999999999998e-05 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405733+0000", 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9394999999999997e-05 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405783+0000", 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.071394319999999997 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2062 20.17 20:e88892cc:test-rados-api-smithi162-84049-16::big.copy:head [read 0~25165836] snapc 0=[] ondisk+read+known_if_redirected e109)", 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.405952+0000", 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.471270988999997, 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015988108000000001, 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2062 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405952+0000", 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405952+0000", 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405950+0000", 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7400000000000001e-06 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405956+0000", 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5199999999999997e-06 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405964+0000", 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6949999999999999e-06 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405969+0000", 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1183e-05 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405990+0000", 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7864e-05 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.406048+0000", 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015891445000000001 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.421940+0000", 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015988108000000001 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.951 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2067 20.1e 20:78e1aec5:test-rados-api-smithi162-84049-17::small:head [writefull 0~2800] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.582089+0000", 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.29513309, 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044450968, 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:41:54.952 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2067 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582089+0000", 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582089+0000", 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999962 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582086+0000", 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2827999999999999e-05 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582109+0000", 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8418000000000001e-05 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582137+0000", 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4623999999999999e-05 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582152+0000", 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7082999999999995e-05 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582219+0000", 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012813000000000001 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582347+0000", 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000140655 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582487+0000", 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0084849650000000006 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.590972+0000", 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034854675000000002 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.625827+0000", 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00063645100000000001 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.626464+0000", 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4910000000000003e-05 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.626498+0000", 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1952000000000001e-05 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.626540+0000", 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044450968 2024-02-13T00:41:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24773.0:48 11.a 11:55b4a790:test-rados-api-smithi162-84105-9::foo:head [watch unwatch cookie 94276612612544] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.827722+0000", 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.049500426000002, 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015201684, 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24773", 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4011549658", 2024-02-13T00:41:54.958 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.827722+0000", 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.827722+0000", 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.827721+0000", 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4689999999999996e-06 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.827726+0000", 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7650000000000005e-06 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.827736+0000", 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.806e-06 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.827742+0000", 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6584000000000001e-05 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.827779+0000", 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6042999999999999e-05 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.827825+0000", 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106463 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.827931+0000", 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030161700000000001 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.828233+0000", 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014556173 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.842789+0000", 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3541999999999997e-05 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.842832+0000", 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9286999999999999e-05 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.842862+0000", 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2028999999999999e-05 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.842924+0000", 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015201684 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.963 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25009.0:2 37.6 37:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e110)", 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:05.024150+0000", 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.853072926000003, 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.087036812000000005, 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25009", 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/508499360", 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:41:54.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.024150+0000", 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.024150+0000", 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.024148+0000", 2024-02-13T00:41:54.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.06e-05 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.024158+0000", 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033910860000000002 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.027549+0000", 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0696999999999999e-05 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.027560+0000", 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014665497 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.042225+0000", 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0219999999999999e-06 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.042232+0000", 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059524986000000002 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.101757+0000", 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116793 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.101874+0000", 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014743800000000001 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,2", 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.102022+0000", 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051785600000000002 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.102539+0000", 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0082938850000000008 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.110833+0000", 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028565099999999999 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.111119+0000", 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8586e-05 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.111148+0000", 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8788999999999997e-05 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.111186+0000", 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.087036812000000005 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "pg_update_log_missing(69.15 epoch 127/126 rep_tid 352 entries 127'3 (0'0) error 69:ad551702:::test:head by client.27745.0:6 0.000000 -4096 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 127'1)", 2024-02-13T00:41:54.970 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:21.888669+0000", 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 32.988553633000002, 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019065159000000002, 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888669+0000", 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888669+0000", 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:41:54.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888668+0000", 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2411e-05 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888680+0000", 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8651000000000002e-05 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888699+0000", 2024-02-13T00:41:54.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8210000000000008e-06 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888708+0000", 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8349999999999994e-05 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888786+0000", 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018947997000000001 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.907734+0000", 2024-02-13T00:41:54.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019065159000000002 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.31189.0:2 75.1f 75:fd8eaea7:nspace::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:27.094388+0000", 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.782834496, 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.114165541, 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.31189", 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3145686446", 2024-02-13T00:41:54.974 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094388+0000", 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094388+0000", 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094386+0000", 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8520000000000004e-06 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094394+0000", 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3560999999999999e-05 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094407+0000", 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9789999999999993e-06 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094415+0000", 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0484e-05 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094466+0000", 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6290000000000004e-06 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094473+0000", 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.083557333999999997 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.178030+0000", 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6599999999999995e-06 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.178038+0000", 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017103554 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.195141+0000", 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121271 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.195262+0000", 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014633300000000001 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,6", 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.195409+0000", 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013076450000000001 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.196716+0000", 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6741000000000001e-05 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.196733+0000", 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011723007000000001 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.208456+0000", 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5758999999999999e-05 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.208512+0000", 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1690000000000002e-05 2024-02-13T00:41:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.208554+0000", 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.114165541 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDECSubOpWrite(78.4s1 134/133 ECSubWrite(tid=163, reqid=client.19047.0:1, at_version=0'0, trim_to=0'0, roll_forward_to=134'1))", 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:29.299708+0000", 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 25.577514428000001, 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024244241, 2024-02-13T00:41:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.299708+0000", 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.299708+0000", 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.299707+0000", 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0579999999999999e-05 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.299727+0000", 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8301999999999999e-05 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.299745+0000", 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7370000000000006e-06 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.299753+0000", 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050936760000000001 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.304847+0000", 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.222e-05 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.304859+0000", 2024-02-13T00:41:54.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019080481 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.323940+0000", 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2816000000000001e-05 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.323952+0000", 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024244241 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.985 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.33202.0:2 79.7 e135/134 79:e732be06:::oid_WaitForSafePP_:head v 135'1, mlcod=0'0)", 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:30.322331+0000", 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 24.554891266999999, 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013058551999999999, 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.322331+0000", 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:41:54.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.322331+0000", 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.322330+0000", 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9770000000000006e-06 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.322339+0000", 2024-02-13T00:41:54.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4824000000000001e-05 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.322354+0000", 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8490000000000006e-06 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.322362+0000", 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5455999999999999e-05 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:41:54.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.322417+0000", 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1838e-05 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.322439+0000", 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012893251999999999 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.335333+0000", 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7247000000000003e-05 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:41:54.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.335390+0000", 2024-02-13T00:41:54.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013058551999999999 2024-02-13T00:41:54.990 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.990 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.990 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.990 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:41:54.990 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:41:54.990 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:41:54.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:55.020 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:55.040 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:55.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:55.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:55.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:55.172 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:55.191 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:41:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:55.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:41:55.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:55.329 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:55.355 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:41:55.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:55.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:41:55.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:55.489 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:55.507 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:55.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:55.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:55.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:55.636 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:41:55.636 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:53.715878+0000 osd.0 (osd.0) 67 : cluster [DBG] 95.4 scrub starts 2024-02-13T00:41:55.636 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:53.719435+0000 osd.0 (osd.0) 68 : cluster [DBG] 95.4 scrub ok 2024-02-13T00:41:55.637 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:54.314520+0000 mgr.y (mgr.14544) 199 : cluster [DBG] pgmap v208: 356 pgs: 32 unknown, 324 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:55.637 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:54.421543+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:41:55.637 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:54.422481+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:41:55.637 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:54.454703+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:41:55.637 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:54.457598+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:41:55.637 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:54.668098+0000 osd.7 (osd.7) 84 : cluster [DBG] 95.7 scrub starts 2024-02-13T00:41:55.637 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:54.670648+0000 osd.7 (osd.7) 85 : cluster [DBG] 95.7 scrub ok 2024-02-13T00:41:55.637 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:55.083836+0000 mon.b (mon.1) 193 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:55.637 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:55.317025+0000 mon.a (mon.0) 1204 : audit [INF] from='client.? 172.21.15.162:0/3065351095' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-83977-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:55.637 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:55.317187+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-02-13T00:41:55.638 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:55.649 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:53.715878+0000 osd.0 (osd.0) 67 : cluster [DBG] 95.4 scrub starts 2024-02-13T00:41:55.649 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:53.719435+0000 osd.0 (osd.0) 68 : cluster [DBG] 95.4 scrub ok 2024-02-13T00:41:55.649 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:54.314520+0000 mgr.y (mgr.14544) 199 : cluster [DBG] pgmap v208: 356 pgs: 32 unknown, 324 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:55.649 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:54.421543+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:41:55.650 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:54.422481+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:41:55.650 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:54.454703+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:41:55.650 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:54.457598+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:41:55.650 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:54.668098+0000 osd.7 (osd.7) 84 : cluster [DBG] 95.7 scrub starts 2024-02-13T00:41:55.650 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:54.670648+0000 osd.7 (osd.7) 85 : cluster [DBG] 95.7 scrub ok 2024-02-13T00:41:55.650 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:55.083836+0000 mon.b (mon.1) 193 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:55.650 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:55.317025+0000 mon.a (mon.0) 1204 : audit [INF] from='client.? 172.21.15.162:0/3065351095' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-83977-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:55.650 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:55.317187+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-02-13T00:41:55.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:53.715878+0000 osd.0 (osd.0) 67 : cluster [DBG] 95.4 scrub starts 2024-02-13T00:41:55.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:53.719435+0000 osd.0 (osd.0) 68 : cluster [DBG] 95.4 scrub ok 2024-02-13T00:41:55.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:54.314520+0000 mgr.y (mgr.14544) 199 : cluster [DBG] pgmap v208: 356 pgs: 32 unknown, 324 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:41:55.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:54.421543+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:41:55.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:54.422481+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:41:55.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:54.454703+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:41:55.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:54.457598+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:41:55.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:54.668098+0000 osd.7 (osd.7) 84 : cluster [DBG] 95.7 scrub starts 2024-02-13T00:41:55.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:54.670648+0000 osd.7 (osd.7) 85 : cluster [DBG] 95.7 scrub ok 2024-02-13T00:41:55.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:55.083836+0000 mon.b (mon.1) 193 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:55.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:55.317025+0000 mon.a (mon.0) 1204 : audit [INF] from='client.? 172.21.15.162:0/3065351095' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-83977-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:55.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:55.317187+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-02-13T00:41:55.656 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:55.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:55.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:41:55.787 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:55.805 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:41:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:55.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:41:55.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:55.937 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:55.976 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:55.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:55.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-13T00:41:56.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:41:56.106 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:56.124 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:41:56.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:41:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:56.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:41:56.213 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:41:56.213 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:41:56.213 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:41:56.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd out 6 2024-02-13T00:41:56.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:56.240 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:41:56] "GET /metrics HTTP/1.1" 200 198554 "" "Prometheus/2.33.4" 2024-02-13T00:41:56.261 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:41:56.266 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:56.379 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:56.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:56.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:56.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:56.508 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:55.235895+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.6 scrub starts 2024-02-13T00:41:56.508 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:55.236723+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.6 scrub ok 2024-02-13T00:41:56.508 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:55.351090+0000 osd.6 (osd.6) 67 : cluster [DBG] 95.1 scrub starts 2024-02-13T00:41:56.508 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:55.360022+0000 osd.6 (osd.6) 68 : cluster [DBG] 95.1 scrub ok 2024-02-13T00:41:56.508 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:55.375213+0000 mgr.y (mgr.14544) 200 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:56.508 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:55.417790+0000 osd.2 (osd.2) 75 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:41:56.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:55.420996+0000 osd.2 (osd.2) 76 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:41:56.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:55.625823+0000 osd.7 (osd.7) 86 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:41:56.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:55.626866+0000 osd.7 (osd.7) 87 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:41:56.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:56.085196+0000 mon.b (mon.1) 194 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:56.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:56.322769+0000 mon.a (mon.0) 1206 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-02-13T00:41:56.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:56.324465+0000 mon.a (mon.0) 1207 : audit [INF] from='client.? 172.21.15.162:0/3729576308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi162-83985-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:56.509 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:55.235895+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.6 scrub starts 2024-02-13T00:41:56.509 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:55.236723+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.6 scrub ok 2024-02-13T00:41:56.509 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:55.351090+0000 osd.6 (osd.6) 67 : cluster [DBG] 95.1 scrub starts 2024-02-13T00:41:56.509 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:55.360022+0000 osd.6 (osd.6) 68 : cluster [DBG] 95.1 scrub ok 2024-02-13T00:41:56.509 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:55.375213+0000 mgr.y (mgr.14544) 200 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:56.509 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:55.417790+0000 osd.2 (osd.2) 75 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:41:56.510 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:55.420996+0000 osd.2 (osd.2) 76 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:41:56.510 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:55.625823+0000 osd.7 (osd.7) 86 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:41:56.510 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:55.626866+0000 osd.7 (osd.7) 87 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:41:56.510 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:56.085196+0000 mon.b (mon.1) 194 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:56.510 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:56.322769+0000 mon.a (mon.0) 1206 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-02-13T00:41:56.510 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:56.324465+0000 mon.a (mon.0) 1207 : audit [INF] from='client.? 172.21.15.162:0/3729576308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi162-83985-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:56.510 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:56.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:56.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:56.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:55.235895+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.6 scrub starts 2024-02-13T00:41:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:55.236723+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.6 scrub ok 2024-02-13T00:41:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:55.351090+0000 osd.6 (osd.6) 67 : cluster [DBG] 95.1 scrub starts 2024-02-13T00:41:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:55.360022+0000 osd.6 (osd.6) 68 : cluster [DBG] 95.1 scrub ok 2024-02-13T00:41:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:55.375213+0000 mgr.y (mgr.14544) 200 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:41:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:55.417790+0000 osd.2 (osd.2) 75 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:41:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:55.420996+0000 osd.2 (osd.2) 76 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:41:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:55.625823+0000 osd.7 (osd.7) 86 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:41:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:55.626866+0000 osd.7 (osd.7) 87 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:41:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:56.085196+0000 mon.b (mon.1) 194 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:56.322769+0000 mon.a (mon.0) 1206 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-02-13T00:41:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:56.324465+0000 mon.a (mon.0) 1207 : audit [INF] from='client.? 172.21.15.162:0/3729576308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi162-83985-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:56.820 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:56.858 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:56.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 -- 172.21.15.162:0/4186303806 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f591c0fe5d0 msgr2=0x7f591c0f6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 --2- 172.21.15.162:0/4186303806 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f591c0fe5d0 0x7f591c0f6150 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7f5910004870 tx=0x7f5910009cb0).stop 2024-02-13T00:41:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 -- 172.21.15.162:0/4186303806 shutdown_connections 2024-02-13T00:41:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 --2- 172.21.15.162:0/4186303806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f591c0f6830 0x7f591c104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 --2- 172.21.15.162:0/4186303806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f591c0fdc70 0x7f591c0fe050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 --2- 172.21.15.162:0/4186303806 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f591c0fe5d0 0x7f591c0f6150 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 -- 172.21.15.162:0/4186303806 >> 172.21.15.162:0/4186303806 conn(0x7f591c070060 msgr2=0x7f591c0f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 -- 172.21.15.162:0/4186303806 shutdown_connections 2024-02-13T00:41:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 -- 172.21.15.162:0/4186303806 wait complete. 2024-02-13T00:41:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 librados: starting msgr at 2024-02-13T00:41:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 librados: starting objecter 2024-02-13T00:41:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 Processor -- start 2024-02-13T00:41:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 -- start start 2024-02-13T00:41:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 librados: setting wanted keys 2024-02-13T00:41:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.954+0000 7f5921be6700 1 librados: calling monclient init 2024-02-13T00:41:56.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f5921be6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f591c0f6830 0x7f591c10e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:56.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f5921be6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f591c0fdc70 0x7f591c10e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:56.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f5921be6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f591c0fe5d0 0x7f591c1083c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:56.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f5921be6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c1088b0 con 0x7f591c0fe5d0 2024-02-13T00:41:56.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f5921be6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c108a20 con 0x7f591c0fdc70 2024-02-13T00:41:56.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f5921be6700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c108be0 con 0x7f591c0f6830 2024-02-13T00:41:56.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f591b7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f591c0f6830 0x7f591c10e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:56.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f591bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f591c0fe5d0 0x7f591c1083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:56.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f591b7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f591c0f6830 0x7f591c10e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:47166/0 (socket says 172.21.15.162:47166) 2024-02-13T00:41:56.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f591b7fe700 1 -- 172.21.15.162:0/202038053 learned_addr learned my addr 172.21.15.162:0/202038053 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:56.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f591affd700 1 --2- 172.21.15.162:0/202038053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f591c0fdc70 0x7f591c10e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:56.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f591b7fe700 1 -- 172.21.15.162:0/202038053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f591c0fdc70 msgr2=0x7f591c10e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f591b7fe700 1 --2- 172.21.15.162:0/202038053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f591c0fdc70 0x7f591c10e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f591b7fe700 1 -- 172.21.15.162:0/202038053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f591c0fe5d0 msgr2=0x7f591c1083c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f591b7fe700 1 --2- 172.21.15.162:0/202038053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f591c0fe5d0 0x7f591c1083c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f591b7fe700 1 -- 172.21.15.162:0/202038053 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5910005040 con 0x7f591c0f6830 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f591bfff700 1 --2- 172.21.15.162:0/202038053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f591c0fe5d0 0x7f591c1083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f591b7fe700 1 --2- 172.21.15.162:0/202038053 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f591c0f6830 0x7f591c10e220 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 rx=0x7f590c00b530 tx=0x7f590c0090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f5918ff9700 10 client.?.objecter ms_handle_connect 0x7f591c0f6830 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.955+0000 7f5918ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.956+0000 7f5918ff9700 1 -- 172.21.15.162:0/202038053 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c002320 con 0x7f591c0f6830 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.956+0000 7f5921be6700 1 -- 172.21.15.162:0/202038053 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591c108e60 con 0x7f591c0f6830 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.956+0000 7f5918ff9700 1 -- 172.21.15.162:0/202038053 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f590c002950 con 0x7f591c0f6830 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.956+0000 7f5918ff9700 1 -- 172.21.15.162:0/202038053 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c012880 con 0x7f591c0f6830 2024-02-13T00:41:56.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.956+0000 7f5921be6700 10 client.36518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:56.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.956+0000 7f5921be6700 1 -- 172.21.15.162:0/202038053 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591c0f7260 con 0x7f591c0f6830 2024-02-13T00:41:56.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.956+0000 7f5921be6700 1 librados: init done 2024-02-13T00:41:56.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.957+0000 7f5921be6700 1 -- 172.21.15.162:0/202038053 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591c05f960 con 0x7f591c0f6830 2024-02-13T00:41:56.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:41:56.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.958+0000 7f5918ff9700 1 -- 172.21.15.162:0/202038053 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f590c002480 con 0x7f591c0f6830 2024-02-13T00:41:56.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.958+0000 7f5918ff9700 1 --2- 172.21.15.162:0/202038053 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5904062c90 0x7f5904065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:56.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.958+0000 7f5918ff9700 1 -- 172.21.15.162:0/202038053 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(161..161 src has 52..161) v4 ==== 9253+0+0 (secure 0 0 0) 0x7f590c07dc80 con 0x7f591c0f6830 2024-02-13T00:41:56.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.958+0000 7f5918ff9700 10 client.36518.objecter ms_dispatch 0x7f591c001550 osd_map(161..161 src has 52..161) v4 2024-02-13T00:41:56.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.958+0000 7f5918ff9700 3 client.36518.objecter handle_osd_map got epochs [161,161] > 0 2024-02-13T00:41:56.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.958+0000 7f5918ff9700 3 client.36518.objecter handle_osd_map decoding full epoch 161 2024-02-13T00:41:56.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.958+0000 7f591affd700 1 --2- 172.21.15.162:0/202038053 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5904062c90 0x7f5904065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:56.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.959+0000 7f5918ff9700 20 client.36518.objecter dump_active .. 0 homeless 2024-02-13T00:41:56.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.959+0000 7f5918ff9700 10 client.36518.objecter ms_handle_connect 0x7f5904062c90 2024-02-13T00:41:56.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.959+0000 7f591affd700 1 --2- 172.21.15.162:0/202038053 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5904062c90 0x7f5904065150 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 rx=0x7f5910009a00 tx=0x7f591000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:56.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:56.963+0000 7f5918ff9700 1 -- 172.21.15.162:0/202038053 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f590c050cc0 con 0x7f591c0f6830 2024-02-13T00:41:56.986 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:41:56.995 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:57.018 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:41:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:57.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:41:57.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:57.149 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:57.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.166+0000 7f5921be6700 1 -- 172.21.15.162:0/202038053 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f591c0475e0 con 0x7f591c0f6830 2024-02-13T00:41:57.271 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:57.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:57.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.329+0000 7f5918ff9700 1 -- 172.21.15.162:0/202038053 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v162) v1 ==== 87+0+0 (secure 0 0 0) 0x7f590c0508b0 con 0x7f591c0f6830 2024-02-13T00:41:57.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 10 librados: watch_flush enter 2024-02-13T00:41:57.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 10 librados: watch_flush exit 2024-02-13T00:41:57.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 20 client.36518.objecter shutdown clearing up homeless session... 2024-02-13T00:41:57.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 10 client.36518.objecter successfully canceled tick 2024-02-13T00:41:57.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 -- 172.21.15.162:0/202038053 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5904062c90 msgr2=0x7f5904065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:57.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 --2- 172.21.15.162:0/202038053 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5904062c90 0x7f5904065150 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 rx=0x7f5910009a00 tx=0x7f591000b040).stop 2024-02-13T00:41:57.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 -- 172.21.15.162:0/202038053 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f591c0f6830 msgr2=0x7f591c10e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:57.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 --2- 172.21.15.162:0/202038053 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f591c0f6830 0x7f591c10e220 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 rx=0x7f590c00b530 tx=0x7f590c0090f0).stop 2024-02-13T00:41:57.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 -- 172.21.15.162:0/202038053 shutdown_connections 2024-02-13T00:41:57.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 --2- 172.21.15.162:0/202038053 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5904062c90 0x7f5904065150 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:57.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 --2- 172.21.15.162:0/202038053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f591c0fe5d0 0x7f591c1083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:57.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 --2- 172.21.15.162:0/202038053 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f591c0f6830 0x7f591c10e220 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:57.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 --2- 172.21.15.162:0/202038053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f591c0fdc70 0x7f591c10e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:57.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 -- 172.21.15.162:0/202038053 >> 172.21.15.162:0/202038053 conn(0x7f591c070060 msgr2=0x7f591c0fbb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:57.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 -- 172.21.15.162:0/202038053 shutdown_connections 2024-02-13T00:41:57.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 -- 172.21.15.162:0/202038053 wait complete. 2024-02-13T00:41:57.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:57.338+0000 7f5921be6700 1 librados: shutdown 2024-02-13T00:41:57.354 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:41:57.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:41:57.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:56.201701+0000 osd.1 (osd.1) 85 : cluster [DBG] 5.1d scrub starts 2024-02-13T00:41:57.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:56.204667+0000 osd.1 (osd.1) 86 : cluster [DBG] 5.1d scrub ok 2024-02-13T00:41:57.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:56.314951+0000 mgr.y (mgr.14544) 201 : cluster [DBG] pgmap v211: 356 pgs: 32 creating+peering, 324 active+clean; 216 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-13T00:41:57.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:56.342276+0000 osd.6 (osd.6) 69 : cluster [DBG] 95.6 scrub starts 2024-02-13T00:41:57.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:56.346974+0000 osd.6 (osd.6) 70 : cluster [DBG] 95.6 scrub ok 2024-02-13T00:41:57.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:56.421613+0000 osd.2 (osd.2) 77 : cluster [DBG] 5.d scrub starts 2024-02-13T00:41:57.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:56.422989+0000 osd.2 (osd.2) 78 : cluster [DBG] 5.d scrub ok 2024-02-13T00:41:57.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:56.662865+0000 osd.7 (osd.7) 88 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:41:57.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:56.665997+0000 osd.7 (osd.7) 89 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:41:57.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:57.086204+0000 mon.b (mon.1) 195 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:57.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:57.167485+0000 mon.c (mon.2) 145 : audit [INF] from='client.? 172.21.15.162:0/202038053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:41:57.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:57.167798+0000 mon.a (mon.0) 1208 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:41:57.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:57.317812+0000 mon.a (mon.0) 1209 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:57.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:57.325112+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? 172.21.15.162:0/3729576308' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi162-83985-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:57.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:57.325330+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:41:57.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:57.325385+0000 mon.a (mon.0) 1212 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-02-13T00:41:57.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:57.334979+0000 mon.b (mon.1) 196 : audit [INF] from='client.? 172.21.15.162:0/615754399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-83977-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:57.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:57.338541+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-83977-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:57.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:56.201701+0000 osd.1 (osd.1) 85 : cluster [DBG] 5.1d scrub starts 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:56.204667+0000 osd.1 (osd.1) 86 : cluster [DBG] 5.1d scrub ok 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:56.314951+0000 mgr.y (mgr.14544) 201 : cluster [DBG] pgmap v211: 356 pgs: 32 creating+peering, 324 active+clean; 216 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:56.342276+0000 osd.6 (osd.6) 69 : cluster [DBG] 95.6 scrub starts 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:56.346974+0000 osd.6 (osd.6) 70 : cluster [DBG] 95.6 scrub ok 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:56.421613+0000 osd.2 (osd.2) 77 : cluster [DBG] 5.d scrub starts 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:56.422989+0000 osd.2 (osd.2) 78 : cluster [DBG] 5.d scrub ok 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:56.662865+0000 osd.7 (osd.7) 88 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:56.665997+0000 osd.7 (osd.7) 89 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:57.086204+0000 mon.b (mon.1) 195 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:57.167485+0000 mon.c (mon.2) 145 : audit [INF] from='client.? 172.21.15.162:0/202038053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:57.167798+0000 mon.a (mon.0) 1208 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:57.317812+0000 mon.a (mon.0) 1209 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:57.325112+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? 172.21.15.162:0/3729576308' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi162-83985-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:57.325330+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:41:57.402 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:57.325385+0000 mon.a (mon.0) 1212 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-02-13T00:41:57.403 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:57.334979+0000 mon.b (mon.1) 196 : audit [INF] from='client.? 172.21.15.162:0/615754399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-83977-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:57.403 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:57.338541+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-83977-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:57.403 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:57.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:56.201701+0000 osd.1 (osd.1) 85 : cluster [DBG] 5.1d scrub starts 2024-02-13T00:41:57.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:56.204667+0000 osd.1 (osd.1) 86 : cluster [DBG] 5.1d scrub ok 2024-02-13T00:41:57.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:56.314951+0000 mgr.y (mgr.14544) 201 : cluster [DBG] pgmap v211: 356 pgs: 32 creating+peering, 324 active+clean; 216 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-13T00:41:57.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:56.342276+0000 osd.6 (osd.6) 69 : cluster [DBG] 95.6 scrub starts 2024-02-13T00:41:57.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:56.346974+0000 osd.6 (osd.6) 70 : cluster [DBG] 95.6 scrub ok 2024-02-13T00:41:57.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:56.421613+0000 osd.2 (osd.2) 77 : cluster [DBG] 5.d scrub starts 2024-02-13T00:41:57.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:56.422989+0000 osd.2 (osd.2) 78 : cluster [DBG] 5.d scrub ok 2024-02-13T00:41:57.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:56.662865+0000 osd.7 (osd.7) 88 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:41:57.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:56.665997+0000 osd.7 (osd.7) 89 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:41:57.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:57.086204+0000 mon.b (mon.1) 195 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:57.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:57.167485+0000 mon.c (mon.2) 145 : audit [INF] from='client.? 172.21.15.162:0/202038053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:41:57.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:57.167798+0000 mon.a (mon.0) 1208 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:41:57.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:57.317812+0000 mon.a (mon.0) 1209 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:41:57.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:57.325112+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? 172.21.15.162:0/3729576308' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi162-83985-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:57.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:57.325330+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:41:57.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:57.325385+0000 mon.a (mon.0) 1212 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-02-13T00:41:57.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:57.334979+0000 mon.b (mon.1) 196 : audit [INF] from='client.? 172.21.15.162:0/615754399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-83977-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:57.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:57.338541+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-83977-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:41:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:57.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:41:57.706 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:57.734 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:58.212 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:58.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:58.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:58.289 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:41:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:41:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:41:58.289 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:41:58.316 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:58.361 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:58.611 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:41:58.611 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:41:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:41:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:41:58.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:58.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:41:58.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:41:58.743 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:58.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:56.575782+0000 osd.3 (osd.3) 109 : cluster [DBG] 95.5 scrub starts 2024-02-13T00:41:58.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:56.579244+0000 osd.3 (osd.3) 110 : cluster [DBG] 95.5 scrub ok 2024-02-13T00:41:58.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:58.087239+0000 mon.b (mon.1) 197 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:58.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:58.323552+0000 mon.a (mon.0) 1214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-83977-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:58.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:58.323707+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-02-13T00:41:58.985 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:56.575782+0000 osd.3 (osd.3) 109 : cluster [DBG] 95.5 scrub starts 2024-02-13T00:41:58.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:56.579244+0000 osd.3 (osd.3) 110 : cluster [DBG] 95.5 scrub ok 2024-02-13T00:41:58.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:58.087239+0000 mon.b (mon.1) 197 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:58.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:58.323552+0000 mon.a (mon.0) 1214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-83977-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:58.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:58.323707+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-02-13T00:41:59.024 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:41:59.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:56.575782+0000 osd.3 (osd.3) 109 : cluster [DBG] 95.5 scrub starts 2024-02-13T00:41:59.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:56.579244+0000 osd.3 (osd.3) 110 : cluster [DBG] 95.5 scrub ok 2024-02-13T00:41:59.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:58.087239+0000 mon.b (mon.1) 197 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:59.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:58.323552+0000 mon.a (mon.0) 1214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-83977-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:41:59.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:58.323707+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-02-13T00:41:59.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:59.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:41:59.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-13T00:41:59.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:59.155 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:59.175 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:59.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:59.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:59.278 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:41:59.308 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:59.322 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:41:59.327 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:41:59.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:59.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:41:59.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:41:59.469 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:59.667 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:41:59.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:41:59.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:41:59.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:41:59.798 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:41:59.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.853+0000 7fc56f852700 1 -- 172.21.15.162:0/2588799415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc568102270 msgr2=0x7fc568104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:59.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.853+0000 7fc56f852700 1 --2- 172.21.15.162:0/2588799415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc568102270 0x7fc568104660 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 rx=0x7fc560009c30 tx=0x7fc560004c50).stop 2024-02-13T00:41:59.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.853+0000 7fc56f852700 1 -- 172.21.15.162:0/2588799415 shutdown_connections 2024-02-13T00:41:59.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.853+0000 7fc56f852700 1 --2- 172.21.15.162:0/2588799415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc568102270 0x7fc568104660 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:59.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.853+0000 7fc56f852700 1 --2- 172.21.15.162:0/2588799415 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc5680f8d10 0x7fc5680f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:59.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.853+0000 7fc56f852700 1 --2- 172.21.15.162:0/2588799415 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5680f95e0 0x7fc568101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:59.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.853+0000 7fc56f852700 1 -- 172.21.15.162:0/2588799415 >> 172.21.15.162:0/2588799415 conn(0x7fc56806eb90 msgr2=0x7fc5680f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:41:59.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.854+0000 7fc56f852700 1 -- 172.21.15.162:0/2588799415 shutdown_connections 2024-02-13T00:41:59.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.854+0000 7fc56f852700 1 -- 172.21.15.162:0/2588799415 wait complete. 2024-02-13T00:41:59.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.854+0000 7fc56f852700 1 librados: starting msgr at 2024-02-13T00:41:59.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.854+0000 7fc56f852700 1 librados: starting objecter 2024-02-13T00:41:59.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.854+0000 7fc56f852700 1 Processor -- start 2024-02-13T00:41:59.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.854+0000 7fc56f852700 1 -- start start 2024-02-13T00:41:59.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.854+0000 7fc56f852700 1 librados: setting wanted keys 2024-02-13T00:41:59.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.854+0000 7fc56f852700 1 librados: calling monclient init 2024-02-13T00:41:59.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56f852700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc5680f8d10 0x7fc5680f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:59.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56f852700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5680f95e0 0x7fc5680f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:59.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56f852700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc568102270 0x7fc5680f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:59.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56f852700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc568116610 con 0x7fc5680f95e0 2024-02-13T00:41:59.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56f852700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc568116780 con 0x7fc568102270 2024-02-13T00:41:59.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56f852700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc568116940 con 0x7fc5680f8d10 2024-02-13T00:41:59.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56d5ee700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc5680f8d10 0x7fc5680f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:59.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56d5ee700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc5680f8d10 0x7fc5680f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:47264/0 (socket says 172.21.15.162:47264) 2024-02-13T00:41:59.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56d5ee700 1 -- 172.21.15.162:0/3201713500 learned_addr learned my addr 172.21.15.162:0/3201713500 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:41:59.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56ddef700 1 --2- 172.21.15.162:0/3201713500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc568102270 0x7fc5680f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:59.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56d5ee700 1 -- 172.21.15.162:0/3201713500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc568102270 msgr2=0x7fc5680f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:59.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56d5ee700 1 --2- 172.21.15.162:0/3201713500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc568102270 0x7fc5680f76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:41:59.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56d5ee700 1 -- 172.21.15.162:0/3201713500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5680f95e0 msgr2=0x7fc5680f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:41:59.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56d5ee700 1 --2- 172.21.15.162:0/3201713500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5680f95e0 0x7fc5680f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:41:59.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56d5ee700 1 -- 172.21.15.162:0/3201713500 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc564005040 con 0x7fc5680f8d10 2024-02-13T00:41:59.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56d5ee700 1 --2- 172.21.15.162:0/3201713500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc5680f8d10 0x7fc5680f6cc0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7fc564009400 tx=0x7fc564009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:59.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc55e7fc700 10 client.?.objecter ms_handle_connect 0x7fc5680f8d10 2024-02-13T00:41:59.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc55e7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:41:59.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc564007a40 con 0x7fc5680f8d10 2024-02-13T00:41:59.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc56400f040 con 0x7fc5680f8d10 2024-02-13T00:41:59.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.855+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5600097c0 con 0x7fc5680f8d10 2024-02-13T00:41:59.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.856+0000 7fc56f852700 10 client.36542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:41:59.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.856+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc568116f50 con 0x7fc5680f8d10 2024-02-13T00:41:59.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.856+0000 7fc56f852700 1 librados: init done 2024-02-13T00:41:59.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.856+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc564014aa0 con 0x7fc5680f8d10 2024-02-13T00:41:59.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.857+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc54c004f30 con 0x7fc5680f8d10 2024-02-13T00:41:59.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.857+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc5640133c0 con 0x7fc5680f8d10 2024-02-13T00:41:59.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.857+0000 7fc55e7fc700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc554062c90 0x7fc554065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:59.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.857+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(164..164 src has 52..164) v4 ==== 8861+0+0 (secure 0 0 0) 0x7fc56407d100 con 0x7fc5680f8d10 2024-02-13T00:41:59.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.857+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 osd_map(164..164 src has 52..164) v4 2024-02-13T00:41:59.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.857+0000 7fc55e7fc700 3 client.36542.objecter handle_osd_map got epochs [164,164] > 0 2024-02-13T00:41:59.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.857+0000 7fc55e7fc700 3 client.36542.objecter handle_osd_map decoding full epoch 164 2024-02-13T00:41:59.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.857+0000 7fc56cded700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc554062c90 0x7fc554065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:59.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.857+0000 7fc55e7fc700 20 client.36542.objecter dump_active .. 0 homeless 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.857+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v164) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc564084070 con 0x7fc5680f8d10 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc55e7fc700 10 client.36542.objecter ms_handle_connect 0x7fc554062c90 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56cded700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc554062c90 0x7fc554065150 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 rx=0x7fc5680f8460 tx=0x7fc558009b60).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56f852700 10 client.36542.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 1 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc54c004560 0x7fc54c0049a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0051d0 osd=0 3 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0051d0 osd=0 3 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0051d0 osd=0 3 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 1 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c0051d0 osd=0 3 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 0 1 2024-02-13T00:41:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0051d0 osd=0 4 2024-02-13T00:41:59.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56f852700 10 client.36542.objecter _send_command 1 2024-02-13T00:41:59.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc54c008960 con 0x7fc54c004560 2024-02-13T00:41:59.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56ddef700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc54c004560 0x7fc54c0049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:59.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc55e7fc700 10 client.36542.objecter ms_handle_connect 0x7fc54c004560 2024-02-13T00:41:59.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.858+0000 7fc56ddef700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc54c004560 0x7fc54c0049a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:59.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.859+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc54c008960 con 0x7fc54c004560 2024-02-13T00:41:59.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.859+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:41:59.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.859+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 1 = system:0 2024-02-13T00:41:59.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.859+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c0051d0 osd=0 4 2024-02-13T00:41:59.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.859+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 0 1 2024-02-13T00:41:59.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.873+0000 7fc56f852700 10 client.36542.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:59.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.873+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 2 2024-02-13T00:41:59.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.873+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0051d0 osd=0 3 2024-02-13T00:41:59.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.873+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0051d0 osd=0 3 2024-02-13T00:41:59.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.873+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0051d0 osd=0 3 2024-02-13T00:41:59.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.873+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 2 2024-02-13T00:41:59.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.873+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c0051d0 osd=0 3 2024-02-13T00:41:59.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.873+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 0 2 2024-02-13T00:41:59.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.873+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0051d0 osd=0 4 2024-02-13T00:41:59.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.873+0000 7fc56f852700 10 client.36542.objecter _send_command 2 2024-02-13T00:41:59.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.873+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc54c003fa0 con 0x7fc54c004560 2024-02-13T00:41:59.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc54c003fa0 con 0x7fc54c004560 2024-02-13T00:41:59.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:59.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:59.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c0051d0 osd=0 4 2024-02-13T00:41:59.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 0 2 2024-02-13T00:41:59.880 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:41:59.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56f852700 10 client.36542.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:59.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 3 2024-02-13T00:41:59.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6805/150599780 conn(0x7fc54c0084c0 0x7fc54c002740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:59.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0043a0 osd=1 3 2024-02-13T00:41:59.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0043a0 osd=1 3 2024-02-13T00:41:59.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0043a0 osd=1 3 2024-02-13T00:41:59.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 3 2024-02-13T00:41:59.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c0043a0 osd=1 3 2024-02-13T00:41:59.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 1 3 2024-02-13T00:41:59.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0043a0 osd=1 4 2024-02-13T00:41:59.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56f852700 10 client.36542.objecter _send_command 3 2024-02-13T00:41:59.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc54c002de0 con 0x7fc54c0084c0 2024-02-13T00:41:59.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.875+0000 7fc56d5ee700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6805/150599780 conn(0x7fc54c0084c0 0x7fc54c002740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:59.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.876+0000 7fc55e7fc700 10 client.36542.objecter ms_handle_connect 0x7fc54c0084c0 2024-02-13T00:41:59.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.876+0000 7fc56d5ee700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6805/150599780 conn(0x7fc54c0084c0 0x7fc54c002740 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:59.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.876+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc54c002de0 con 0x7fc54c0084c0 2024-02-13T00:41:59.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.876+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:41:59.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.876+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 3 = system:0 2024-02-13T00:41:59.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.876+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c0043a0 osd=1 4 2024-02-13T00:41:59.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.876+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 1 3 2024-02-13T00:41:59.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.891+0000 7fc56f852700 10 client.36542.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:59.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.891+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 4 2024-02-13T00:41:59.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.891+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0043a0 osd=1 3 2024-02-13T00:41:59.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.891+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0043a0 osd=1 3 2024-02-13T00:41:59.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.891+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0043a0 osd=1 3 2024-02-13T00:41:59.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.891+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 4 2024-02-13T00:41:59.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.891+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c0043a0 osd=1 3 2024-02-13T00:41:59.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.891+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 1 4 2024-02-13T00:41:59.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.891+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0043a0 osd=1 4 2024-02-13T00:41:59.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.891+0000 7fc56f852700 10 client.36542.objecter _send_command 4 2024-02-13T00:41:59.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.891+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc54c007ae0 con 0x7fc54c0084c0 2024-02-13T00:41:59.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.892+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc54c007ae0 con 0x7fc54c0084c0 2024-02-13T00:41:59.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.892+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:59.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.892+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:59.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.892+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c0043a0 osd=1 4 2024-02-13T00:41:59.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.892+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 1 4 2024-02-13T00:41:59.897 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:41:59.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.892+0000 7fc56f852700 10 client.36542.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:59.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.892+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 5 2024-02-13T00:41:59.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.892+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc54c003940 0x7fc54c00d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:59.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.892+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0037d0 osd=2 3 2024-02-13T00:41:59.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0037d0 osd=2 3 2024-02-13T00:41:59.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0037d0 osd=2 3 2024-02-13T00:41:59.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 5 2024-02-13T00:41:59.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c0037d0 osd=2 3 2024-02-13T00:41:59.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 2 5 2024-02-13T00:41:59.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0037d0 osd=2 4 2024-02-13T00:41:59.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc56f852700 10 client.36542.objecter _send_command 5 2024-02-13T00:41:59.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc54c00fdd0 con 0x7fc54c003940 2024-02-13T00:41:59.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc56cded700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc54c003940 0x7fc54c00d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:59.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc55e7fc700 10 client.36542.objecter ms_handle_connect 0x7fc54c003940 2024-02-13T00:41:59.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc56cded700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc54c003940 0x7fc54c00d000 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:59.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc54c00fdd0 con 0x7fc54c003940 2024-02-13T00:41:59.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:41:59.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 5 = system:0 2024-02-13T00:41:59.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c0037d0 osd=2 4 2024-02-13T00:41:59.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.893+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 2 5 2024-02-13T00:41:59.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.908+0000 7fc56f852700 10 client.36542.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:59.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.908+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 6 2024-02-13T00:41:59.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.908+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0037d0 osd=2 3 2024-02-13T00:41:59.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.908+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0037d0 osd=2 3 2024-02-13T00:41:59.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.908+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0037d0 osd=2 3 2024-02-13T00:41:59.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.908+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 6 2024-02-13T00:41:59.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.908+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c0037d0 osd=2 3 2024-02-13T00:41:59.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.908+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 2 6 2024-02-13T00:41:59.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.908+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0037d0 osd=2 4 2024-02-13T00:41:59.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.908+0000 7fc56f852700 10 client.36542.objecter _send_command 6 2024-02-13T00:41:59.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.908+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc54c0032a0 con 0x7fc54c003940 2024-02-13T00:41:59.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc54c0032a0 con 0x7fc54c003940 2024-02-13T00:41:59.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:59.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:59.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c0037d0 osd=2 4 2024-02-13T00:41:59.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 2 6 2024-02-13T00:41:59.912 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:41:59.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56f852700 10 client.36542.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:59.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 7 2024-02-13T00:41:59.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6813/604971011 conn(0x7fc54c00e710 0x7fc54c00eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:59.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c001bb0 osd=3 3 2024-02-13T00:41:59.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c001bb0 osd=3 3 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c001bb0 osd=3 3 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 7 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c001bb0 osd=3 3 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 3 7 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c001bb0 osd=3 4 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56f852700 10 client.36542.objecter _send_command 7 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc54c002380 con 0x7fc54c00e710 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.909+0000 7fc56ddef700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6813/604971011 conn(0x7fc54c00e710 0x7fc54c00eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.910+0000 7fc55e7fc700 10 client.36542.objecter ms_handle_connect 0x7fc54c00e710 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.910+0000 7fc56ddef700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6813/604971011 conn(0x7fc54c00e710 0x7fc54c00eb90 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.910+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc54c002380 con 0x7fc54c00e710 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.910+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.910+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 7 = system:0 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.910+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c001bb0 osd=3 4 2024-02-13T00:41:59.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.910+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 3 7 2024-02-13T00:41:59.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:58.315433+0000 mgr.y (mgr.14544) 202 : cluster [DBG] pgmap v214: 388 pgs: 64 unknown, 324 active+clean; 216 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-13T00:41:59.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:59.088551+0000 mon.b (mon.1) 198 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:41:59.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:59.326590+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-02-13T00:41:59.920 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:41:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:59.343745+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? 172.21.15.162:0/3737706718' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-83985-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:41:59.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.925+0000 7fc56f852700 10 client.36542.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:59.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.925+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 8 2024-02-13T00:41:59.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.925+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c001bb0 osd=3 3 2024-02-13T00:41:59.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.925+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c001bb0 osd=3 3 2024-02-13T00:41:59.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.925+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c001bb0 osd=3 3 2024-02-13T00:41:59.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.925+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 8 2024-02-13T00:41:59.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.925+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c001bb0 osd=3 3 2024-02-13T00:41:59.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.925+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 3 8 2024-02-13T00:41:59.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.925+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c001bb0 osd=3 4 2024-02-13T00:41:59.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.925+0000 7fc56f852700 10 client.36542.objecter _send_command 8 2024-02-13T00:41:59.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.925+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc54c00f8d0 con 0x7fc54c00e710 2024-02-13T00:41:59.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc54c00f8d0 con 0x7fc54c00e710 2024-02-13T00:41:59.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:59.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:59.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c001bb0 osd=3 4 2024-02-13T00:41:59.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 3 8 2024-02-13T00:41:59.930 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:41:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56f852700 10 client.36542.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 9 2024-02-13T00:41:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc54c012000 0x7fc54c0143f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c00f100 osd=4 3 2024-02-13T00:41:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c00f100 osd=4 3 2024-02-13T00:41:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c00f100 osd=4 3 2024-02-13T00:41:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 9 2024-02-13T00:41:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c00f100 osd=4 3 2024-02-13T00:41:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 4 9 2024-02-13T00:41:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c00f100 osd=4 4 2024-02-13T00:41:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56f852700 10 client.36542.objecter _send_command 9 2024-02-13T00:41:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc54c017040 con 0x7fc54c012000 2024-02-13T00:41:59.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.926+0000 7fc56d5ee700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc54c012000 0x7fc54c0143f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:59.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.927+0000 7fc55e7fc700 10 client.36542.objecter ms_handle_connect 0x7fc54c012000 2024-02-13T00:41:59.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.927+0000 7fc56d5ee700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc54c012000 0x7fc54c0143f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:59.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.927+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc54c017040 con 0x7fc54c012000 2024-02-13T00:41:59.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.927+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:41:59.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.927+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 9 = system:0 2024-02-13T00:41:59.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.927+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c00f100 osd=4 4 2024-02-13T00:41:59.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.927+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 4 9 2024-02-13T00:41:59.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.942+0000 7fc56f852700 10 client.36542.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:59.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.942+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 10 2024-02-13T00:41:59.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.942+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c00f100 osd=4 3 2024-02-13T00:41:59.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.942+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c00f100 osd=4 3 2024-02-13T00:41:59.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.942+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c00f100 osd=4 3 2024-02-13T00:41:59.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.942+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 10 2024-02-13T00:41:59.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.942+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c00f100 osd=4 3 2024-02-13T00:41:59.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.942+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 4 10 2024-02-13T00:41:59.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.942+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c00f100 osd=4 4 2024-02-13T00:41:59.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.942+0000 7fc56f852700 10 client.36542.objecter _send_command 10 2024-02-13T00:41:59.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.942+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc54c0148e0 con 0x7fc54c012000 2024-02-13T00:41:59.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc54c0148e0 con 0x7fc54c012000 2024-02-13T00:41:59.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:59.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:59.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c00f100 osd=4 4 2024-02-13T00:41:59.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 4 10 2024-02-13T00:41:59.947 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:41:59.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56f852700 10 client.36542.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:59.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 11 2024-02-13T00:41:59.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc54c014fc0 0x7fc54c0173b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:59.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0148e0 osd=5 3 2024-02-13T00:41:59.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0148e0 osd=5 3 2024-02-13T00:41:59.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0148e0 osd=5 3 2024-02-13T00:41:59.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 11 2024-02-13T00:41:59.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c0148e0 osd=5 3 2024-02-13T00:41:59.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 5 11 2024-02-13T00:41:59.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0148e0 osd=5 4 2024-02-13T00:41:59.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56f852700 10 client.36542.objecter _send_command 11 2024-02-13T00:41:59.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc54c01a170 con 0x7fc54c014fc0 2024-02-13T00:41:59.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56cded700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc54c014fc0 0x7fc54c0173b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:59.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc55e7fc700 10 client.36542.objecter ms_handle_connect 0x7fc54c014fc0 2024-02-13T00:41:59.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.944+0000 7fc56cded700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc54c014fc0 0x7fc54c0173b0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:59.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.945+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc54c01a170 con 0x7fc54c014fc0 2024-02-13T00:41:59.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.945+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:41:59.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.945+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 11 = system:0 2024-02-13T00:41:59.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.945+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c0148e0 osd=5 4 2024-02-13T00:41:59.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.945+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 5 11 2024-02-13T00:41:59.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.959+0000 7fc56f852700 10 client.36542.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:59.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.959+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 12 2024-02-13T00:41:59.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.959+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0148e0 osd=5 3 2024-02-13T00:41:59.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.959+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0148e0 osd=5 3 2024-02-13T00:41:59.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.959+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c0148e0 osd=5 3 2024-02-13T00:41:59.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.959+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 12 2024-02-13T00:41:59.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.959+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c0148e0 osd=5 3 2024-02-13T00:41:59.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.959+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 5 12 2024-02-13T00:41:59.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.959+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0148e0 osd=5 4 2024-02-13T00:41:59.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.959+0000 7fc56f852700 10 client.36542.objecter _send_command 12 2024-02-13T00:41:59.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.959+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc54c01a130 con 0x7fc54c014fc0 2024-02-13T00:41:59.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc54c01a130 con 0x7fc54c014fc0 2024-02-13T00:41:59.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:59.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:59.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c0148e0 osd=5 4 2024-02-13T00:41:59.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 5 12 2024-02-13T00:41:59.965 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:41:59.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56f852700 10 client.36542.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:59.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 13 2024-02-13T00:41:59.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc54c0194c0 0x7fc54c019940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:59.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c01a130 osd=6 3 2024-02-13T00:41:59.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c01a130 osd=6 3 2024-02-13T00:41:59.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c01a130 osd=6 3 2024-02-13T00:41:59.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 13 2024-02-13T00:41:59.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c01a130 osd=6 3 2024-02-13T00:41:59.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 6 13 2024-02-13T00:41:59.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c01a130 osd=6 4 2024-02-13T00:41:59.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56f852700 10 client.36542.objecter _send_command 13 2024-02-13T00:41:59.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc54c01d4d0 con 0x7fc54c0194c0 2024-02-13T00:41:59.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56ddef700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc54c0194c0 0x7fc54c019940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:59.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc55e7fc700 10 client.36542.objecter ms_handle_connect 0x7fc54c0194c0 2024-02-13T00:41:59.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.962+0000 7fc56ddef700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc54c0194c0 0x7fc54c019940 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:59.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.963+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc54c01d4d0 con 0x7fc54c0194c0 2024-02-13T00:41:59.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.963+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:41:59.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.963+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 13 = system:0 2024-02-13T00:41:59.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.963+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c01a130 osd=6 4 2024-02-13T00:41:59.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.963+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 6 13 2024-02-13T00:41:59.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.977+0000 7fc56f852700 10 client.36542.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:59.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.977+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 14 2024-02-13T00:41:59.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.977+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c01a130 osd=6 3 2024-02-13T00:41:59.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.977+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c01a130 osd=6 3 2024-02-13T00:41:59.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.977+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c01a130 osd=6 3 2024-02-13T00:41:59.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.977+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 14 2024-02-13T00:41:59.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.977+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c01a130 osd=6 3 2024-02-13T00:41:59.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.977+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 6 14 2024-02-13T00:41:59.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.977+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c01a130 osd=6 4 2024-02-13T00:41:59.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.977+0000 7fc56f852700 10 client.36542.objecter _send_command 14 2024-02-13T00:41:59.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.977+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc54c017940 con 0x7fc54c0194c0 2024-02-13T00:41:59.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.979+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc54c017940 con 0x7fc54c0194c0 2024-02-13T00:41:59.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.979+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:41:59.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.979+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:41:59.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.979+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c01a130 osd=6 4 2024-02-13T00:41:59.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.979+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 6 14 2024-02-13T00:41:59.983 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:41:59.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56f852700 10 client.36542.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:41:59.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 15 2024-02-13T00:41:59.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc54c018020 0x7fc54c018480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:41:59.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c017940 osd=7 3 2024-02-13T00:41:59.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c017940 osd=7 3 2024-02-13T00:41:59.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c017940 osd=7 3 2024-02-13T00:41:59.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 15 2024-02-13T00:41:59.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c017940 osd=7 3 2024-02-13T00:41:59.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 7 15 2024-02-13T00:41:59.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c017940 osd=7 4 2024-02-13T00:41:59.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56f852700 10 client.36542.objecter _send_command 15 2024-02-13T00:41:59.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc54c018df0 con 0x7fc54c018020 2024-02-13T00:41:59.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56d5ee700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc54c018020 0x7fc54c018480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:41:59.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc55e7fc700 10 client.36542.objecter ms_handle_connect 0x7fc54c018020 2024-02-13T00:41:59.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.980+0000 7fc56d5ee700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc54c018020 0x7fc54c018480 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:41:59.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.981+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc54c018df0 con 0x7fc54c018020 2024-02-13T00:41:59.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.981+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:41:59.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.981+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 15 = system:0 2024-02-13T00:41:59.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.981+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c017940 osd=7 4 2024-02-13T00:41:59.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.981+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 7 15 2024-02-13T00:41:59.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.995+0000 7fc56f852700 10 client.36542.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:41:59.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.995+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign -1 16 2024-02-13T00:41:59.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.995+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c017940 osd=7 3 2024-02-13T00:41:59.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.995+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c017940 osd=7 3 2024-02-13T00:41:59.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.995+0000 7fc56f852700 20 client.36542.objecter _get_session s=0x7fc54c017940 osd=7 3 2024-02-13T00:41:59.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.995+0000 7fc56f852700 15 client.36542.objecter _session_command_op_remove -1 16 2024-02-13T00:41:59.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.995+0000 7fc56f852700 20 client.36542.objecter get_session s=0x7fc54c017940 osd=7 3 2024-02-13T00:41:59.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.995+0000 7fc56f852700 15 client.36542.objecter _session_command_op_assign 7 16 2024-02-13T00:41:59.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.995+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c017940 osd=7 4 2024-02-13T00:41:59.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.995+0000 7fc56f852700 10 client.36542.objecter _send_command 16 2024-02-13T00:42:00.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.995+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc54c018df0 con 0x7fc54c018020 2024-02-13T00:42:00.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc55e7fc700 1 -- 172.21.15.162:0/3201713500 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc54c018df0 con 0x7fc54c018020 2024-02-13T00:42:00.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc55e7fc700 10 client.36542.objecter ms_dispatch 0x7fc568001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:00.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc55e7fc700 10 client.36542.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:00.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc55e7fc700 20 client.36542.objecter put_session s=0x7fc54c017940 osd=7 4 2024-02-13T00:42:00.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc55e7fc700 15 client.36542.objecter _session_command_op_remove 7 16 2024-02-13T00:42:00.001 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:42:00.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 10 librados: watch_flush enter 2024-02-13T00:42:00.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 10 librados: watch_flush exit 2024-02-13T00:42:00.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 10 client.36542.objecter close_session for osd.0 2024-02-13T00:42:00.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc54c004560 msgr2=0x7fc54c0049a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:00.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc54c004560 0x7fc54c0049a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:00.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0051d0 osd=0 1 2024-02-13T00:42:00.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 10 client.36542.objecter close_session for osd.1 2024-02-13T00:42:00.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6805/150599780 conn(0x7fc54c0084c0 msgr2=0x7fc54c002740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:00.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6805/150599780 conn(0x7fc54c0084c0 0x7fc54c002740 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:00.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0043a0 osd=1 1 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 10 client.36542.objecter close_session for osd.2 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc54c003940 msgr2=0x7fc54c00d000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc54c003940 0x7fc54c00d000 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0037d0 osd=2 1 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 10 client.36542.objecter close_session for osd.3 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6813/604971011 conn(0x7fc54c00e710 msgr2=0x7fc54c00eb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6813/604971011 conn(0x7fc54c00e710 0x7fc54c00eb90 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56ddef700 1 -- 172.21.15.162:0/3201713500 reap_dead start 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c001bb0 osd=3 1 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 10 client.36542.objecter close_session for osd.4 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc54c012000 msgr2=0x7fc54c0143f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc54c012000 0x7fc54c0143f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:00.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c00f100 osd=4 1 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 10 client.36542.objecter close_session for osd.5 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc54c014fc0 msgr2=0x7fc54c0173b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc54c014fc0 0x7fc54c0173b0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c0148e0 osd=5 1 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 10 client.36542.objecter close_session for osd.6 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc54c0194c0 msgr2=0x7fc54c019940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc54c0194c0 0x7fc54c019940 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c01a130 osd=6 1 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 10 client.36542.objecter close_session for osd.7 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc54c018020 msgr2=0x7fc54c018480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc54c018020 0x7fc54c018480 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 20 client.36542.objecter put_session s=0x7fc54c017940 osd=7 1 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 20 client.36542.objecter shutdown clearing up homeless session... 2024-02-13T00:42:00.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 10 client.36542.objecter successfully canceled tick 2024-02-13T00:42:00.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56ddef700 1 -- 172.21.15.162:0/3201713500 reap_dead start 2024-02-13T00:42:00.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc554062c90 msgr2=0x7fc554065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:00.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc554062c90 0x7fc554065150 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 rx=0x7fc5680f8460 tx=0x7fc558009b60).stop 2024-02-13T00:42:00.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc5680f8d10 msgr2=0x7fc5680f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:00.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc5680f8d10 0x7fc5680f6cc0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7fc564009400 tx=0x7fc564009cb0).stop 2024-02-13T00:42:00.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 shutdown_connections 2024-02-13T00:42:00.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc5680f8d10 0x7fc5680f6cc0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:00.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 --2- 172.21.15.162:0/3201713500 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc554062c90 0x7fc554065150 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:00.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 >> 172.21.15.162:0/3201713500 conn(0x7fc56806eb90 msgr2=0x7fc568102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:00.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.997+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 shutdown_connections 2024-02-13T00:42:00.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.998+0000 7fc56f852700 1 -- 172.21.15.162:0/3201713500 wait complete. 2024-02-13T00:42:00.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:41:59.998+0000 7fc56f852700 1 librados: shutdown 2024-02-13T00:42:00.007 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:42:00.007 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:42:00.007 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:42:00.007 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:42:00.007 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:42:00.007 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:42:00.007 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:42:00.007 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:42:00.209 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:58.315433+0000 mgr.y (mgr.14544) 202 : cluster [DBG] pgmap v214: 388 pgs: 64 unknown, 324 active+clean; 216 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-13T00:42:00.210 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:59.088551+0000 mon.b (mon.1) 198 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:00.210 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:59.326590+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-02-13T00:42:00.210 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:41:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:59.343745+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? 172.21.15.162:0/3737706718' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-83985-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:00.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:58.315433+0000 mgr.y (mgr.14544) 202 : cluster [DBG] pgmap v214: 388 pgs: 64 unknown, 324 active+clean; 216 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-13T00:42:00.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:59.088551+0000 mon.b (mon.1) 198 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:00.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:59.326590+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-02-13T00:42:00.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:41:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:59.343745+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? 172.21.15.162:0/3737706718' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-83985-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.022 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:58.621268+0000 osd.0 (osd.0) 69 : cluster [DBG] 95.9 scrub starts 2024-02-13T00:42:01.022 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:41:58.622996+0000 osd.0 (osd.0) 70 : cluster [DBG] 95.9 scrub ok 2024-02-13T00:42:01.023 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:41:59.858294+0000 mon.c (mon.2) 146 : audit [DBG] from='client.? 172.21.15.162:0/3201713500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:01.023 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:00.089747+0000 mon.b (mon.1) 199 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:01.023 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:00.307897+0000 mon.a (mon.0) 1218 : audit [INF] from='client.? 172.21.15.162:0/3737706718' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-83985-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:01.023 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:00.308064+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-02-13T00:42:01.023 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:00.317168+0000 mon.b (mon.1) 200 : audit [INF] from='client.? 172.21.15.162:0/3535769977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84081-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.023 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:00.326829+0000 mon.a (mon.0) 1220 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84081-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.023 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:00.332424+0000 mon.c (mon.2) 147 : audit [INF] from='client.? 172.21.15.162:0/2884571582' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83977-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.023 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:00.338134+0000 mon.a (mon.0) 1221 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83977-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.048 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:01.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:01.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:01.137 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:58.621268+0000 osd.0 (osd.0) 69 : cluster [DBG] 95.9 scrub starts 2024-02-13T00:42:01.137 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:41:58.622996+0000 osd.0 (osd.0) 70 : cluster [DBG] 95.9 scrub ok 2024-02-13T00:42:01.137 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:41:59.858294+0000 mon.c (mon.2) 146 : audit [DBG] from='client.? 172.21.15.162:0/3201713500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:01.137 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:00.089747+0000 mon.b (mon.1) 199 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:01.137 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:00.307897+0000 mon.a (mon.0) 1218 : audit [INF] from='client.? 172.21.15.162:0/3737706718' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-83985-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:01.137 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:00.308064+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-02-13T00:42:01.137 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:00.317168+0000 mon.b (mon.1) 200 : audit [INF] from='client.? 172.21.15.162:0/3535769977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84081-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.137 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:00.326829+0000 mon.a (mon.0) 1220 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84081-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.137 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:00.332424+0000 mon.c (mon.2) 147 : audit [INF] from='client.? 172.21.15.162:0/2884571582' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83977-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:00.338134+0000 mon.a (mon.0) 1221 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83977-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:58.621268+0000 osd.0 (osd.0) 69 : cluster [DBG] 95.9 scrub starts 2024-02-13T00:42:01.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:41:58.622996+0000 osd.0 (osd.0) 70 : cluster [DBG] 95.9 scrub ok 2024-02-13T00:42:01.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:41:59.858294+0000 mon.c (mon.2) 146 : audit [DBG] from='client.? 172.21.15.162:0/3201713500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:01.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:00.089747+0000 mon.b (mon.1) 199 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:01.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:00.307897+0000 mon.a (mon.0) 1218 : audit [INF] from='client.? 172.21.15.162:0/3737706718' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-83985-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:01.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:00.308064+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-02-13T00:42:01.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:00.317168+0000 mon.b (mon.1) 200 : audit [INF] from='client.? 172.21.15.162:0/3535769977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84081-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:00.326829+0000 mon.a (mon.0) 1220 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84081-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:00.332424+0000 mon.c (mon.2) 147 : audit [INF] from='client.? 172.21.15.162:0/2884571582' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83977-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:00.338134+0000 mon.a (mon.0) 1221 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83977-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:01.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:01.182 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:01.309 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:42:01.309 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:42:01.560 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:01.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:01.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:01.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:01.695 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.898+0000 7fcdbc43e700 1 -- 172.21.15.162:0/1959264417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdb40f95e0 msgr2=0x7fcdb4101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.898+0000 7fcdbc43e700 1 --2- 172.21.15.162:0/1959264417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdb40f95e0 0x7fcdb4101d80 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 rx=0x7fcdb0004870 tx=0x7fcdb0009cb0).stop 2024-02-13T00:42:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.898+0000 7fcdbc43e700 1 -- 172.21.15.162:0/1959264417 shutdown_connections 2024-02-13T00:42:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.898+0000 7fcdbc43e700 1 --2- 172.21.15.162:0/1959264417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb4102270 0x7fcdb4104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.898+0000 7fcdbc43e700 1 --2- 172.21.15.162:0/1959264417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb40f8d10 0x7fcdb40f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.898+0000 7fcdbc43e700 1 --2- 172.21.15.162:0/1959264417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdb40f95e0 0x7fcdb4101d80 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.898+0000 7fcdbc43e700 1 -- 172.21.15.162:0/1959264417 >> 172.21.15.162:0/1959264417 conn(0x7fcdb406eb90 msgr2=0x7fcdb40f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:01.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.898+0000 7fcdbc43e700 1 -- 172.21.15.162:0/1959264417 shutdown_connections 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.898+0000 7fcdbc43e700 1 -- 172.21.15.162:0/1959264417 wait complete. 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdbc43e700 1 librados: starting msgr at 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdbc43e700 1 librados: starting objecter 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdbc43e700 1 Processor -- start 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdbc43e700 1 -- start start 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdbc43e700 1 librados: setting wanted keys 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdbc43e700 1 librados: calling monclient init 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdbc43e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdb40f8d10 0x7fcdb40f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdbc43e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb40f95e0 0x7fcdb40f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdbc43e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb4102270 0x7fcdb40f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdbc43e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb41166f0 con 0x7fcdb4102270 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdbc43e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb4116860 con 0x7fcdb40f95e0 2024-02-13T00:42:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdbc43e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb4116a20 con 0x7fcdb40f8d10 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdba9db700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb4102270 0x7fcdb40f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdba1da700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdb40f8d10 0x7fcdb40f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdba9db700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb4102270 0x7fcdb40f76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34314/0 (socket says 172.21.15.162:34314) 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdba1da700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdb40f8d10 0x7fcdb40f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:47312/0 (socket says 172.21.15.162:47312) 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdba9db700 1 -- 172.21.15.162:0/3256649763 learned_addr learned my addr 172.21.15.162:0/3256649763 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdb99d9700 1 --2- 172.21.15.162:0/3256649763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb40f95e0 0x7fcdb40f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdba9db700 1 -- 172.21.15.162:0/3256649763 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdb40f8d10 msgr2=0x7fcdb40f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.899+0000 7fcdba9db700 1 --2- 172.21.15.162:0/3256649763 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdb40f8d10 0x7fcdb40f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdba9db700 1 -- 172.21.15.162:0/3256649763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb40f95e0 msgr2=0x7fcdb40f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdba9db700 1 --2- 172.21.15.162:0/3256649763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb40f95e0 0x7fcdb40f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdba9db700 1 -- 172.21.15.162:0/3256649763 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac0097c0 con 0x7fcdb4102270 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdba1da700 1 --2- 172.21.15.162:0/3256649763 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdb40f8d10 0x7fcdb40f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:42:01.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdba9db700 1 --2- 172.21.15.162:0/3256649763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb4102270 0x7fcdb40f76c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7fcdac001d70 tx=0x7fcdac004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdab7fe700 10 client.?.objecter ms_handle_connect 0x7fcdb4102270 2024-02-13T00:42:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdab7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdab7fe700 1 -- 172.21.15.162:0/3256649763 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac0099f0 con 0x7fcdb4102270 2024-02-13T00:42:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdbc43e700 1 -- 172.21.15.162:0/3256649763 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb0005040 con 0x7fcdb4102270 2024-02-13T00:42:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdab7fe700 1 -- 172.21.15.162:0/3256649763 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdac0090d0 con 0x7fcdb4102270 2024-02-13T00:42:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdab7fe700 1 -- 172.21.15.162:0/3256649763 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac007cb0 con 0x7fcdb4102270 2024-02-13T00:42:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdbc43e700 10 client.26457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdbc43e700 1 -- 172.21.15.162:0/3256649763 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb41170b0 con 0x7fcdb4102270 2024-02-13T00:42:01.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.900+0000 7fcdbc43e700 1 librados: init done 2024-02-13T00:42:01.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.901+0000 7fcdbc43e700 1 -- 172.21.15.162:0/3256649763 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb40475e0 con 0x7fcdb4102270 2024-02-13T00:42:01.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.901+0000 7fcdab7fe700 1 -- 172.21.15.162:0/3256649763 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fcdac00fb30 con 0x7fcdb4102270 2024-02-13T00:42:01.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.902+0000 7fcdab7fe700 1 --2- 172.21.15.162:0/3256649763 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcda0062c90 0x7fcda0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:01.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.902+0000 7fcdab7fe700 1 -- 172.21.15.162:0/3256649763 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(166..166 src has 52..166) v4 ==== 9251+0+0 (secure 0 0 0) 0x7fcdac07de00 con 0x7fcdb4102270 2024-02-13T00:42:01.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.902+0000 7fcdab7fe700 10 client.26457.objecter ms_dispatch 0x7fcdb4001550 osd_map(166..166 src has 52..166) v4 2024-02-13T00:42:01.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.902+0000 7fcdab7fe700 3 client.26457.objecter handle_osd_map got epochs [166,166] > 0 2024-02-13T00:42:01.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.902+0000 7fcdab7fe700 3 client.26457.objecter handle_osd_map decoding full epoch 166 2024-02-13T00:42:01.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.902+0000 7fcdba1da700 1 --2- 172.21.15.162:0/3256649763 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcda0062c90 0x7fcda0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:01.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.902+0000 7fcdab7fe700 20 client.26457.objecter dump_active .. 0 homeless 2024-02-13T00:42:01.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.902+0000 7fcdab7fe700 10 client.26457.objecter ms_handle_connect 0x7fcda0062c90 2024-02-13T00:42:01.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.902+0000 7fcdba1da700 1 --2- 172.21.15.162:0/3256649763 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcda0062c90 0x7fcda0065150 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 rx=0x7fcda40041d0 tx=0x7fcda4009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:01.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:01.906+0000 7fcdab7fe700 1 -- 172.21.15.162:0/3256649763 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fcdac050dc0 con 0x7fcdb4102270 2024-02-13T00:42:02.018 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:00.319252+0000 mgr.y (mgr.14544) 203 : cluster [DBG] pgmap v218: 388 pgs: 64 unknown, 32 creating+peering, 292 active+clean; 216 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:42:02.018 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:01.090762+0000 mon.b (mon.1) 201 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:02.018 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:01.313826+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84081-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:02.018 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:01.313948+0000 mon.a (mon.0) 1223 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83977-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:02.018 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:01.314022+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-02-13T00:42:02.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-13T00:42:02.044 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:02.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:02.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.081+0000 7fcdbc43e700 1 -- 172.21.15.162:0/3256649763 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd out", "ids": ["6"]} v 0) v1 -- 0x7fcdb41071b0 con 0x7fcdb4102270 2024-02-13T00:42:02.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:02.174 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:02.193 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:02.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:02.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:42:02.255 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:00.319252+0000 mgr.y (mgr.14544) 203 : cluster [DBG] pgmap v218: 388 pgs: 64 unknown, 32 creating+peering, 292 active+clean; 216 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:42:02.255 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:01.090762+0000 mon.b (mon.1) 201 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:02.255 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:01.313826+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84081-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:02.255 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:01.313948+0000 mon.a (mon.0) 1223 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83977-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:02.255 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:01.314022+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-02-13T00:42:02.255 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:00.319252+0000 mgr.y (mgr.14544) 203 : cluster [DBG] pgmap v218: 388 pgs: 64 unknown, 32 creating+peering, 292 active+clean; 216 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:42:02.255 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:01.090762+0000 mon.b (mon.1) 201 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:02.255 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:01.313826+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84081-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:02.256 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:01.313948+0000 mon.a (mon.0) 1223 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83977-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:02.256 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:01.314022+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-02-13T00:42:02.256 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:42:02.081+0000 7f5f6486f700 -1 mon.a@0(leader).osd e166 definitely_dead 0 2024-02-13T00:42:02.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:02.311 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:42:02.324 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:02.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.337+0000 7fcdab7fe700 1 -- 172.21.15.162:0/3256649763 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["6"]}]=0 marked out osd.6. v167) v1 ==== 87+0+0 (secure 0 0 0) 0x7fcdb41071b0 con 0x7fcdb4102270 2024-02-13T00:42:02.355 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:02.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.343+0000 7fcdbc43e700 10 librados: watch_flush enter 2024-02-13T00:42:02.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.348+0000 7fcdbc43e700 10 librados: watch_flush exit 2024-02-13T00:42:02.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.348+0000 7fcdbc43e700 20 client.26457.objecter shutdown clearing up homeless session... 2024-02-13T00:42:02.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.348+0000 7fcdbc43e700 10 client.26457.objecter successfully canceled tick 2024-02-13T00:42:02.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.349+0000 7fcdbc43e700 1 -- 172.21.15.162:0/3256649763 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcda0062c90 msgr2=0x7fcda0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:02.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.349+0000 7fcdbc43e700 1 --2- 172.21.15.162:0/3256649763 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcda0062c90 0x7fcda0065150 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 rx=0x7fcda40041d0 tx=0x7fcda4009040).stop 2024-02-13T00:42:02.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.349+0000 7fcdbc43e700 1 -- 172.21.15.162:0/3256649763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb4102270 msgr2=0x7fcdb40f76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:02.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.349+0000 7fcdbc43e700 1 --2- 172.21.15.162:0/3256649763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb4102270 0x7fcdb40f76c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7fcdac001d70 tx=0x7fcdac004c50).stop 2024-02-13T00:42:02.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.349+0000 7fcdbc43e700 1 -- 172.21.15.162:0/3256649763 shutdown_connections 2024-02-13T00:42:02.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.349+0000 7fcdbc43e700 1 --2- 172.21.15.162:0/3256649763 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcda0062c90 0x7fcda0065150 secure :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 rx=0x7fcda40041d0 tx=0x7fcda4009040).stop 2024-02-13T00:42:02.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.349+0000 7fcdbc43e700 1 --2- 172.21.15.162:0/3256649763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdb4102270 0x7fcdb40f76c0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:02.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.349+0000 7fcdbc43e700 1 --2- 172.21.15.162:0/3256649763 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdb40f8d10 0x7fcdb40f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:02.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.349+0000 7fcdbc43e700 1 --2- 172.21.15.162:0/3256649763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb40f95e0 0x7fcdb40f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:02.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.349+0000 7fcdbc43e700 1 -- 172.21.15.162:0/3256649763 >> 172.21.15.162:0/3256649763 conn(0x7fcdb406eb90 msgr2=0x7fcdb4102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:02.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:02.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.361+0000 7fcdbc43e700 1 -- 172.21.15.162:0/3256649763 shutdown_connections 2024-02-13T00:42:02.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.362+0000 7fcdbc43e700 1 -- 172.21.15.162:0/3256649763 wait complete. 2024-02-13T00:42:02.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:02.371+0000 7fcdbc43e700 1 librados: shutdown 2024-02-13T00:42:02.385 INFO:teuthology.orchestra.run.smithi162.stderr:marked out osd.6. 2024-02-13T00:42:02.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:02.492 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:02.590 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:02.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:02.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:02.721 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:03.024 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:02.082117+0000 mon.a (mon.0) 1225 : audit [INF] from='client.? 172.21.15.162:0/3256649763' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-13T00:42:03.024 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:02.082224+0000 mon.a (mon.0) 1226 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-02-13T00:42:03.024 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:02.092184+0000 mon.b (mon.1) 202 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:03.024 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:02.337830+0000 mon.a (mon.0) 1227 : audit [INF] from='client.? 172.21.15.162:0/3256649763' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-13T00:42:03.024 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:02.337875+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e167: 8 total, 8 up, 7 in 2024-02-13T00:42:03.025 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:02.356704+0000 mon.b (mon.1) 203 : audit [INF] from='client.? 172.21.15.162:0/2253654154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:03.025 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:02.357909+0000 mon.a (mon.0) 1229 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:03.025 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:02.082117+0000 mon.a (mon.0) 1225 : audit [INF] from='client.? 172.21.15.162:0/3256649763' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-13T00:42:03.025 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:02.082224+0000 mon.a (mon.0) 1226 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-02-13T00:42:03.025 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:02.092184+0000 mon.b (mon.1) 202 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:03.025 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:02.337830+0000 mon.a (mon.0) 1227 : audit [INF] from='client.? 172.21.15.162:0/3256649763' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-13T00:42:03.025 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:02.337875+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e167: 8 total, 8 up, 7 in 2024-02-13T00:42:03.025 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:02.356704+0000 mon.b (mon.1) 203 : audit [INF] from='client.? 172.21.15.162:0/2253654154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:03.025 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:02.357909+0000 mon.a (mon.0) 1229 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:03.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:03.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:03.171 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:03.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:03.204 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:02.082117+0000 mon.a (mon.0) 1225 : audit [INF] from='client.? 172.21.15.162:0/3256649763' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-13T00:42:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:02.082224+0000 mon.a (mon.0) 1226 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-02-13T00:42:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:02.092184+0000 mon.b (mon.1) 202 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:02.337830+0000 mon.a (mon.0) 1227 : audit [INF] from='client.? 172.21.15.162:0/3256649763' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-13T00:42:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:02.337875+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e167: 8 total, 8 up, 7 in 2024-02-13T00:42:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:02.356704+0000 mon.b (mon.1) 203 : audit [INF] from='client.? 172.21.15.162:0/2253654154' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:02.357909+0000 mon.a (mon.0) 1229 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:03.406 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:03.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:03.517 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:03.553 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:03.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:42:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:02.319729+0000 mgr.y (mgr.14544) 204 : cluster [DBG] pgmap v220: 356 pgs: 64 unknown, 292 active+clean; 216 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:42:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:03.093177+0000 mon.b (mon.1) 204 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:03.342048+0000 mon.a (mon.0) 1230 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:03.342196+0000 mon.a (mon.0) 1231 : cluster [DBG] osdmap e168: 8 total, 8 up, 7 in 2024-02-13T00:42:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:03.349763+0000 mon.b (mon.1) 205 : audit [INF] from='client.? 172.21.15.162:0/2076913139' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-83977-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:03.350963+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-83977-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:04.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:02.319729+0000 mgr.y (mgr.14544) 204 : cluster [DBG] pgmap v220: 356 pgs: 64 unknown, 292 active+clean; 216 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:42:04.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:03.093177+0000 mon.b (mon.1) 204 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:04.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:03.342048+0000 mon.a (mon.0) 1230 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:04.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:03.342196+0000 mon.a (mon.0) 1231 : cluster [DBG] osdmap e168: 8 total, 8 up, 7 in 2024-02-13T00:42:04.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:03.349763+0000 mon.b (mon.1) 205 : audit [INF] from='client.? 172.21.15.162:0/2076913139' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-83977-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:04.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:03.350963+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-83977-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:04.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:02.319729+0000 mgr.y (mgr.14544) 204 : cluster [DBG] pgmap v220: 356 pgs: 64 unknown, 292 active+clean; 216 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-13T00:42:04.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:03.093177+0000 mon.b (mon.1) 204 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:04.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:03.342048+0000 mon.a (mon.0) 1230 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:04.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:03.342196+0000 mon.a (mon.0) 1231 : cluster [DBG] osdmap e168: 8 total, 8 up, 7 in 2024-02-13T00:42:04.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:03.349763+0000 mon.b (mon.1) 205 : audit [INF] from='client.? 172.21.15.162:0/2076913139' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-83977-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:04.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:03.350963+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-83977-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:04.329 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-rgw-foo-a[66295]: debug 2024-02-13T00:42:04.267+0000 7fa24b3c3700 -1 RGWWatcher::handle_error cookie 94514175194752 err (110) Connection timed out 2024-02-13T00:42:04.405 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:04.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:04.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:04.592 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:04.659 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:42:04.659 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:42:04.781 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:04.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:04.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:04.917 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:05.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:04.094236+0000 mon.b (mon.1) 206 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:04.331238+0000 mon.a (mon.0) 1233 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:04.342875+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-83977-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:04.343032+0000 mon.a (mon.0) 1235 : cluster [DBG] osdmap e169: 8 total, 8 up, 7 in 2024-02-13T00:42:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:04.747149+0000 mon.c (mon.2) 148 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:04.747539+0000 mon.a (mon.0) 1236 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:04.748103+0000 mon.c (mon.2) 149 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:04.748345+0000 mon.a (mon.0) 1237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:04.748792+0000 mon.c (mon.2) 150 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:04.749128+0000 mon.a (mon.0) 1238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:05.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:04.094236+0000 mon.b (mon.1) 206 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:04.331238+0000 mon.a (mon.0) 1233 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:04.342875+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-83977-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:04.343032+0000 mon.a (mon.0) 1235 : cluster [DBG] osdmap e169: 8 total, 8 up, 7 in 2024-02-13T00:42:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:04.747149+0000 mon.c (mon.2) 148 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:04.747539+0000 mon.a (mon.0) 1236 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:04.748103+0000 mon.c (mon.2) 149 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:04.748345+0000 mon.a (mon.0) 1237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:04.748792+0000 mon.c (mon.2) 150 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:05.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:04.749128+0000 mon.a (mon.0) 1238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:04.094236+0000 mon.b (mon.1) 206 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:04.331238+0000 mon.a (mon.0) 1233 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:04.342875+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-83977-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:04.343032+0000 mon.a (mon.0) 1235 : cluster [DBG] osdmap e169: 8 total, 8 up, 7 in 2024-02-13T00:42:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:04.747149+0000 mon.c (mon.2) 148 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:04.747539+0000 mon.a (mon.0) 1236 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:04.748103+0000 mon.c (mon.2) 149 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:04.748345+0000 mon.a (mon.0) 1237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:04.748792+0000 mon.c (mon.2) 150 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:04.749128+0000 mon.a (mon.0) 1238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:05.122 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:05.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.156+0000 7fd6e5772700 1 -- 172.21.15.162:0/3286144557 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6e00fe590 msgr2=0x7fd6e00f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:05.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.156+0000 7fd6e5772700 1 --2- 172.21.15.162:0/3286144557 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6e00fe590 0x7fd6e00f7dc0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7fd6d4004870 tx=0x7fd6d4009cb0).stop 2024-02-13T00:42:05.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.156+0000 7fd6e5772700 1 -- 172.21.15.162:0/3286144557 shutdown_connections 2024-02-13T00:42:05.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.156+0000 7fd6e5772700 1 --2- 172.21.15.162:0/3286144557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6e00f84a0 0x7fd6e0104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:05.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.156+0000 7fd6e5772700 1 --2- 172.21.15.162:0/3286144557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6e00fdc30 0x7fd6e00fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:42:05.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.156+0000 7fd6e5772700 1 --2- 172.21.15.162:0/3286144557 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6e00fe590 0x7fd6e00f7dc0 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:05.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.156+0000 7fd6e5772700 1 -- 172.21.15.162:0/3286144557 >> 172.21.15.162:0/3286144557 conn(0x7fd6e00f3880 msgr2=0x7fd6e00f5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:05.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.156+0000 7fd6e5772700 1 -- 172.21.15.162:0/3286144557 shutdown_connections 2024-02-13T00:42:05.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.156+0000 7fd6e5772700 1 -- 172.21.15.162:0/3286144557 wait complete. 2024-02-13T00:42:05.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.157+0000 7fd6e5772700 1 librados: starting msgr at 2024-02-13T00:42:05.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.157+0000 7fd6e5772700 1 librados: starting objecter 2024-02-13T00:42:05.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.157+0000 7fd6e5772700 1 Processor -- start 2024-02-13T00:42:05.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.157+0000 7fd6e5772700 1 -- start start 2024-02-13T00:42:05.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.157+0000 7fd6e5772700 1 librados: setting wanted keys 2024-02-13T00:42:05.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.157+0000 7fd6e5772700 1 librados: calling monclient init 2024-02-13T00:42:05.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.157+0000 7fd6e5772700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6e00f84a0 0x7fd6e010e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:05.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.157+0000 7fd6e5772700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6e00fdc30 0x7fd6e010e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:05.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.157+0000 7fd6e5772700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6e00fe590 0x7fd6e0108340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:05.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.157+0000 7fd6e5772700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e0108830 con 0x7fd6e00fdc30 2024-02-13T00:42:05.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.157+0000 7fd6e5772700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e01089a0 con 0x7fd6e00fe590 2024-02-13T00:42:05.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.157+0000 7fd6e5772700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e0108b60 con 0x7fd6e00f84a0 2024-02-13T00:42:05.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6deffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6e00f84a0 0x7fd6e010e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:05.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6deffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6e00f84a0 0x7fd6e010e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:33388/0 (socket says 172.21.15.162:33388) 2024-02-13T00:42:05.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6deffd700 1 -- 172.21.15.162:0/3707839158 learned_addr learned my addr 172.21.15.162:0/3707839158 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:05.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6df7fe700 1 --2- 172.21.15.162:0/3707839158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6e00fe590 0x7fd6e0108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:05.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6de7fc700 1 --2- 172.21.15.162:0/3707839158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6e00fdc30 0x7fd6e010e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:05.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6df7fe700 1 -- 172.21.15.162:0/3707839158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6e00f84a0 msgr2=0x7fd6e010e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6df7fe700 1 --2- 172.21.15.162:0/3707839158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6e00f84a0 0x7fd6e010e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6df7fe700 1 -- 172.21.15.162:0/3707839158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6e00fdc30 msgr2=0x7fd6e010e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6df7fe700 1 --2- 172.21.15.162:0/3707839158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6e00fdc30 0x7fd6e010e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6df7fe700 1 -- 172.21.15.162:0/3707839158 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d4005040 con 0x7fd6e00fe590 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6df7fe700 1 --2- 172.21.15.162:0/3707839158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6e00fe590 0x7fd6e0108340 secure :-1 s=READY pgs=4831 cs=0 l=1 rev1=1 rx=0x7fd6d800eb70 tx=0x7fd6d800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6c7fff700 10 client.?.objecter ms_handle_connect 0x7fd6e00fe590 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6c7fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6c7fff700 1 -- 172.21.15.162:0/3707839158 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8009870 con 0x7fd6e00fe590 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6e5772700 1 -- 172.21.15.162:0/3707839158 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e0108de0 con 0x7fd6e00fe590 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6c7fff700 1 -- 172.21.15.162:0/3707839158 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6d800f040 con 0x7fd6e00fe590 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6e5772700 10 client.43726.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6c7fff700 1 -- 172.21.15.162:0/3707839158 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8007e40 con 0x7fd6e00fe590 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6e5772700 1 -- 172.21.15.162:0/3707839158 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e0114fd0 con 0x7fd6e00fe590 2024-02-13T00:42:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.158+0000 7fd6e5772700 1 librados: init done 2024-02-13T00:42:05.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.159+0000 7fd6e5772700 1 -- 172.21.15.162:0/3707839158 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e005fae0 con 0x7fd6e00fe590 2024-02-13T00:42:05.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.160+0000 7fd6c7fff700 1 -- 172.21.15.162:0/3707839158 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fd6d80092e0 con 0x7fd6e00fe590 2024-02-13T00:42:05.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.160+0000 7fd6c7fff700 1 --2- 172.21.15.162:0/3707839158 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd6c8062c90 0x7fd6c8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:05.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.160+0000 7fd6c7fff700 1 -- 172.21.15.162:0/3707839158 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(169..169 src has 52..169) v4 ==== 8894+0+0 (secure 0 0 0) 0x7fd6d807d290 con 0x7fd6e00fe590 2024-02-13T00:42:05.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.160+0000 7fd6c7fff700 10 client.43726.objecter ms_dispatch 0x7fd6e0001550 osd_map(169..169 src has 52..169) v4 2024-02-13T00:42:05.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.160+0000 7fd6c7fff700 3 client.43726.objecter handle_osd_map got epochs [169,169] > 0 2024-02-13T00:42:05.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.160+0000 7fd6c7fff700 3 client.43726.objecter handle_osd_map decoding full epoch 169 2024-02-13T00:42:05.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.160+0000 7fd6deffd700 1 --2- 172.21.15.162:0/3707839158 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd6c8062c90 0x7fd6c8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:05.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.160+0000 7fd6c7fff700 20 client.43726.objecter dump_active .. 0 homeless 2024-02-13T00:42:05.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.161+0000 7fd6c7fff700 10 client.43726.objecter ms_handle_connect 0x7fd6c8062c90 2024-02-13T00:42:05.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.161+0000 7fd6deffd700 1 --2- 172.21.15.162:0/3707839158 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd6c8062c90 0x7fd6c8065150 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 rx=0x7fd6d0009b20 tx=0x7fd6d00090f0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:05.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.164+0000 7fd6c7fff700 1 -- 172.21.15.162:0/3707839158 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd6d8049f60 con 0x7fd6e00fe590 2024-02-13T00:42:05.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:05.393+0000 7fd6e5772700 1 -- 172.21.15.162:0/3707839158 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fd6e0107120 con 0x7fd6e00fe590 2024-02-13T00:42:05.489 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-13T00:42:05.567 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:05.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:05.671 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:42:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:05.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:42:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:42:05.826 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:42:05.963 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:42:05.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:04.320227+0000 mgr.y (mgr.14544) 205 : cluster [DBG] pgmap v223: 388 pgs: 96 unknown, 292 active+clean; 216 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-02-13T00:42:05.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:05.095023+0000 mon.b (mon.1) 207 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:05.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:05.396073+0000 mon.a (mon.0) 1239 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:05.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:05.396150+0000 mon.a (mon.0) 1240 : cluster [DBG] osdmap e170: 8 total, 8 up, 7 in 2024-02-13T00:42:05.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:05.396774+0000 mon.b (mon.1) 208 : audit [INF] from='client.? 172.21.15.162:0/3707839158' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:05.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:05.398385+0000 mon.a (mon.0) 1241 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:05.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:05.407762+0000 mon.b (mon.1) 209 : audit [INF] from='client.? 172.21.15.162:0/1630653510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-83985-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:05.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:05.435942+0000 mon.c (mon.2) 151 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-84081-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.967 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:05.436283+0000 mon.a (mon.0) 1242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-83985-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:05.967 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:05.437359+0000 mon.a (mon.0) 1243 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-84081-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.967 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:04.320227+0000 mgr.y (mgr.14544) 205 : cluster [DBG] pgmap v223: 388 pgs: 96 unknown, 292 active+clean; 216 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-02-13T00:42:05.967 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:05.095023+0000 mon.b (mon.1) 207 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:05.967 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:05.396073+0000 mon.a (mon.0) 1239 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:05.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:05.396150+0000 mon.a (mon.0) 1240 : cluster [DBG] osdmap e170: 8 total, 8 up, 7 in 2024-02-13T00:42:05.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:05.396774+0000 mon.b (mon.1) 208 : audit [INF] from='client.? 172.21.15.162:0/3707839158' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:05.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:05.398385+0000 mon.a (mon.0) 1241 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:05.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:05.407762+0000 mon.b (mon.1) 209 : audit [INF] from='client.? 172.21.15.162:0/1630653510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-83985-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:05.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:05.435942+0000 mon.c (mon.2) 151 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-84081-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:05.436283+0000 mon.a (mon.0) 1242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-83985-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:05.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:05.437359+0000 mon.a (mon.0) 1243 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-84081-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:05.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:42:06.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:06.102 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:06.126 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:04.320227+0000 mgr.y (mgr.14544) 205 : cluster [DBG] pgmap v223: 388 pgs: 96 unknown, 292 active+clean; 216 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-02-13T00:42:06.126 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:05.095023+0000 mon.b (mon.1) 207 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:06.126 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:05.396073+0000 mon.a (mon.0) 1239 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:06.126 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:05.396150+0000 mon.a (mon.0) 1240 : cluster [DBG] osdmap e170: 8 total, 8 up, 7 in 2024-02-13T00:42:06.126 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:05.396774+0000 mon.b (mon.1) 208 : audit [INF] from='client.? 172.21.15.162:0/3707839158' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:06.126 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:05.398385+0000 mon.a (mon.0) 1241 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:06.126 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:05.407762+0000 mon.b (mon.1) 209 : audit [INF] from='client.? 172.21.15.162:0/1630653510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-83985-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:06.126 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:05.435942+0000 mon.c (mon.2) 151 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-84081-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:06.126 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:05.436283+0000 mon.a (mon.0) 1242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-83985-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:06.126 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:05.437359+0000 mon.a (mon.0) 1243 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-84081-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:06.132 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:06.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:06.237 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:06.262 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:06.320 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:06.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:06.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:42:06] "GET /metrics HTTP/1.1" 200 193841 "" "Prometheus/2.33.4" 2024-02-13T00:42:06.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.401+0000 7fd6c7fff700 1 -- 172.21.15.162:0/3707839158 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v171) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd6d80503c0 con 0x7fd6e00fe590 2024-02-13T00:42:06.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.404+0000 7fd6e5772700 10 librados: watch_flush enter 2024-02-13T00:42:06.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.408+0000 7fd6e5772700 10 librados: watch_flush exit 2024-02-13T00:42:06.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.408+0000 7fd6e5772700 20 client.43726.objecter shutdown clearing up homeless session... 2024-02-13T00:42:06.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.408+0000 7fd6e5772700 10 client.43726.objecter successfully canceled tick 2024-02-13T00:42:06.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.410+0000 7fd6e5772700 1 -- 172.21.15.162:0/3707839158 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd6c8062c90 msgr2=0x7fd6c8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:06.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.410+0000 7fd6e5772700 1 --2- 172.21.15.162:0/3707839158 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd6c8062c90 0x7fd6c8065150 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 rx=0x7fd6d0009b20 tx=0x7fd6d00090f0).stop 2024-02-13T00:42:06.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.410+0000 7fd6e5772700 1 -- 172.21.15.162:0/3707839158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6e00fe590 msgr2=0x7fd6e0108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:06.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.410+0000 7fd6e5772700 1 --2- 172.21.15.162:0/3707839158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6e00fe590 0x7fd6e0108340 secure :-1 s=READY pgs=4831 cs=0 l=1 rev1=1 rx=0x7fd6d800eb70 tx=0x7fd6d800b040).stop 2024-02-13T00:42:06.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.414+0000 7fd6e5772700 1 -- 172.21.15.162:0/3707839158 shutdown_connections 2024-02-13T00:42:06.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.414+0000 7fd6e5772700 1 --2- 172.21.15.162:0/3707839158 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd6c8062c90 0x7fd6c8065150 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:06.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.414+0000 7fd6e5772700 1 --2- 172.21.15.162:0/3707839158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6e00fe590 0x7fd6e0108340 unknown :-1 s=CLOSED pgs=4831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:06.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.414+0000 7fd6e5772700 1 --2- 172.21.15.162:0/3707839158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6e00f84a0 0x7fd6e010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:06.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.414+0000 7fd6e5772700 1 --2- 172.21.15.162:0/3707839158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6e00fdc30 0x7fd6e010e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:06.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.414+0000 7fd6e5772700 1 -- 172.21.15.162:0/3707839158 >> 172.21.15.162:0/3707839158 conn(0x7fd6e00f3880 msgr2=0x7fd6e00fc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:06.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.418+0000 7fd6e5772700 1 -- 172.21.15.162:0/3707839158 shutdown_connections 2024-02-13T00:42:06.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.418+0000 7fd6e5772700 1 -- 172.21.15.162:0/3707839158 wait complete. 2024-02-13T00:42:06.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:06.418+0000 7fd6e5772700 1 librados: shutdown 2024-02-13T00:42:06.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:42:06.429 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:42:06.495 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:06.667 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:42:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:06.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:42:06.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:06.805 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:06.960 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:06.962 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:05.385475+0000 mgr.y (mgr.14544) 206 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:06.962 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:06.005577+0000 mon.a (mon.0) 1244 : audit [DBG] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-13T00:42:06.962 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:06.006087+0000 mon.a (mon.0) 1245 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:06.962 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:06.095804+0000 mon.b (mon.1) 210 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:06.962 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:06.394409+0000 mon.a (mon.0) 1246 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:06.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:06.401358+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:42:06.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:06.401510+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-83985-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:06.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:06.676459+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:06.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:06.676559+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e171: 8 total, 8 up, 7 in 2024-02-13T00:42:06.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:06.678242+0000 mon.b (mon.1) 211 : audit [INF] from='client.? 172.21.15.162:0/981068687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-83977-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:06.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:06.681429+0000 mon.a (mon.0) 1251 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-83977-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:06.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:06.683244+0000 mon.a (mon.0) 1252 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app2"}]: dispatch 2024-02-13T00:42:06.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:06.683962+0000 mon.a (mon.0) 1253 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:06.963 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:05.385475+0000 mgr.y (mgr.14544) 206 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:06.963 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:06.005577+0000 mon.a (mon.0) 1244 : audit [DBG] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-13T00:42:06.964 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:06.006087+0000 mon.a (mon.0) 1245 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:06.964 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:06.095804+0000 mon.b (mon.1) 210 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:06.964 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:06.394409+0000 mon.a (mon.0) 1246 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:06.964 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:06.401358+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:42:06.964 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:06.401510+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-83985-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:06.964 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:06.676459+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:06.964 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:06.676559+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e171: 8 total, 8 up, 7 in 2024-02-13T00:42:06.964 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:06.678242+0000 mon.b (mon.1) 211 : audit [INF] from='client.? 172.21.15.162:0/981068687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-83977-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:06.964 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:06.681429+0000 mon.a (mon.0) 1251 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-83977-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:06.964 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:06.683244+0000 mon.a (mon.0) 1252 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app2"}]: dispatch 2024-02-13T00:42:06.964 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:06.683962+0000 mon.a (mon.0) 1253 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:06.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:07.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:07.092 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:07.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:05.385475+0000 mgr.y (mgr.14544) 206 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:06.005577+0000 mon.a (mon.0) 1244 : audit [DBG] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-13T00:42:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:06.006087+0000 mon.a (mon.0) 1245 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:06.095804+0000 mon.b (mon.1) 210 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:06.394409+0000 mon.a (mon.0) 1246 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:06.401358+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:42:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:06.401510+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-83985-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:06.676459+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:06.676559+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e171: 8 total, 8 up, 7 in 2024-02-13T00:42:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:06.678242+0000 mon.b (mon.1) 211 : audit [INF] from='client.? 172.21.15.162:0/981068687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-83977-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:06.681429+0000 mon.a (mon.0) 1251 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-83977-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:06.683244+0000 mon.a (mon.0) 1252 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app2"}]: dispatch 2024-02-13T00:42:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:06.683962+0000 mon.a (mon.0) 1253 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:07.340 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:07.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:07.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:07.441 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:42:07.442 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:42:07.442 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:42:07.442 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:42:07.442 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:42:07.442 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:42:07.442 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:42:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.244+0000 7fd4296c4700 1 -- 172.21.15.162:0/1953851492 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4240fd410 msgr2=0x7fd4240f6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.244+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1953851492 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4240fd410 0x7fd4240f6150 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7fd418004870 tx=0x7fd418009cb0).stop 2024-02-13T00:42:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.244+0000 7fd4296c4700 1 -- 172.21.15.162:0/1953851492 shutdown_connections 2024-02-13T00:42:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.244+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1953851492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd4240f6830 0x7fd4240fa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:42:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.244+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1953851492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4240fcab0 0x7fd4240fce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.244+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1953851492 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4240fd410 0x7fd4240f6150 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.244+0000 7fd4296c4700 1 -- 172.21.15.162:0/1953851492 >> 172.21.15.162:0/1953851492 conn(0x7fd4240700a0 msgr2=0x7fd4240f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.244+0000 7fd4296c4700 1 -- 172.21.15.162:0/1953851492 shutdown_connections 2024-02-13T00:42:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.244+0000 7fd4296c4700 1 -- 172.21.15.162:0/1953851492 wait complete. 2024-02-13T00:42:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.244+0000 7fd4296c4700 1 librados: starting msgr at 2024-02-13T00:42:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.244+0000 7fd4296c4700 1 librados: starting objecter 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd4296c4700 1 Processor -- start 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd4296c4700 1 -- start start 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd4296c4700 1 librados: setting wanted keys 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd4296c4700 1 librados: calling monclient init 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd4296c4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4240f6830 0x7fd4241148f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd4296c4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd4240fcab0 0x7fd424114de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd4296c4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4240fd410 0x7fd42410ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd4296c4700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42410ef80 con 0x7fd4240fcab0 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd4296c4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42410f0c0 con 0x7fd4240fd410 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd4296c4700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42410f280 con 0x7fd4240f6830 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd422ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4240f6830 0x7fd4241148f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd422ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4240f6830 0x7fd4241148f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:33398/0 (socket says 172.21.15.162:33398) 2024-02-13T00:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd422ffd700 1 -- 172.21.15.162:0/1816868755 learned_addr learned my addr 172.21.15.162:0/1816868755 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.245+0000 7fd4237fe700 1 --2- 172.21.15.162:0/1816868755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4240fd410 0x7fd42410ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd422ffd700 1 -- 172.21.15.162:0/1816868755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4240fd410 msgr2=0x7fd42410ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd422ffd700 1 --2- 172.21.15.162:0/1816868755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4240fd410 0x7fd42410ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd422ffd700 1 -- 172.21.15.162:0/1816868755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd4240fcab0 msgr2=0x7fd424114de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd422ffd700 1 --2- 172.21.15.162:0/1816868755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd4240fcab0 0x7fd424114de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd422ffd700 1 -- 172.21.15.162:0/1816868755 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd418005040 con 0x7fd4240f6830 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd4237fe700 1 --2- 172.21.15.162:0/1816868755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4240fd410 0x7fd42410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd422ffd700 1 --2- 172.21.15.162:0/1816868755 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4240f6830 0x7fd4241148f0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 rx=0x7fd41400e970 tx=0x7fd41400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd40bfff700 10 client.?.objecter ms_handle_connect 0x7fd4240f6830 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd40bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41400ee60 con 0x7fd4240f6830 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd41400f040 con 0x7fd4240f6830 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414007ea0 con 0x7fd4240f6830 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd42410f560 con 0x7fd4240f6830 2024-02-13T00:42:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd4296c4700 10 client.36596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4240fd880 con 0x7fd4240f6830 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.246+0000 7fd4296c4700 1 librados: init done 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.247+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd42405f960 con 0x7fd4240f6830 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.249+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fd41400cb10 con 0x7fd4240f6830 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.249+0000 7fd40bfff700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd40c062c90 0x7fd40c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.249+0000 7fd4227fc700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd40c062c90 0x7fd40c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.249+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(171..171 src has 52..171) v4 ==== 9695+0+0 (secure 0 0 0) 0x7fd41407d910 con 0x7fd4240f6830 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.249+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 osd_map(171..171 src has 52..171) v4 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.249+0000 7fd40bfff700 3 client.36596.objecter handle_osd_map got epochs [171,171] > 0 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.249+0000 7fd40bfff700 3 client.36596.objecter handle_osd_map decoding full epoch 171 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd40bfff700 20 client.36596.objecter dump_active .. 0 homeless 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v171) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd41407e0a0 con 0x7fd4240f6830 2024-02-13T00:42:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4227fc700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd40c062c90 0x7fd40c065150 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 rx=0x7fd418004840 tx=0x7fd41800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd40bfff700 10 client.36596.objecter ms_handle_connect 0x7fd40c062c90 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4296c4700 10 client.36596.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 1 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd424028590 0x7fd424059990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd42410ff70 osd=0 3 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd42410ff70 osd=0 3 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd42410ff70 osd=0 3 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 1 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd42410ff70 osd=0 3 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 0 1 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd42410ff70 osd=0 4 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4296c4700 10 client.36596.objecter _send_command 1 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd42410d7b0 con 0x7fd424028590 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.250+0000 7fd4237fe700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd424028590 0x7fd424059990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.252+0000 7fd40bfff700 10 client.36596.objecter ms_handle_connect 0x7fd424028590 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.252+0000 7fd4237fe700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd424028590 0x7fd424059990 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.253+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd42410d7b0 con 0x7fd424028590 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.253+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.253+0000 7fd40bfff700 10 client.36596.objecter _finish_command 1 = system:0 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.253+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd42410ff70 osd=0 4 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.253+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 0 1 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.268+0000 7fd4296c4700 10 client.36596.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.268+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 2 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.268+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd42410ff70 osd=0 3 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.268+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd42410ff70 osd=0 3 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.268+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd42410ff70 osd=0 3 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.268+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 2 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.268+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd42410ff70 osd=0 3 2024-02-13T00:42:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.268+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 0 2 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.268+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd42410ff70 osd=0 4 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.268+0000 7fd4296c4700 10 client.36596.objecter _send_command 2 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.268+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd42405a0c0 con 0x7fd424028590 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.269+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd42405a0c0 con 0x7fd424028590 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.269+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.269+0000 7fd40bfff700 10 client.36596.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.269+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd42410ff70 osd=0 4 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.269+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 0 2 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd4296c4700 10 client.36596.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 3 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6805/150599780 conn(0x7fd424059570 0x7fd424102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424059400 osd=1 3 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424059400 osd=1 3 2024-02-13T00:42:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424059400 osd=1 3 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 3 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd424059400 osd=1 3 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 1 3 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424059400 osd=1 4 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd4296c4700 10 client.36596.objecter _send_command 3 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4240280f0 con 0x7fd424059570 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd422ffd700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6805/150599780 conn(0x7fd424059570 0x7fd424102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd422ffd700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6805/150599780 conn(0x7fd424059570 0x7fd424102d00 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.270+0000 7fd40bfff700 10 client.36596.objecter ms_handle_connect 0x7fd424059570 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.271+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd4240280f0 con 0x7fd424059570 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.271+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.271+0000 7fd40bfff700 10 client.36596.objecter _finish_command 3 = system:0 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.271+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd424059400 osd=1 4 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.271+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 1 3 2024-02-13T00:42:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.286+0000 7fd4296c4700 10 client.36596.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.286+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 4 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.286+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424059400 osd=1 3 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.286+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424059400 osd=1 3 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.286+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424059400 osd=1 3 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.286+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 4 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.286+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd424059400 osd=1 3 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.286+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 1 4 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.286+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424059400 osd=1 4 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.286+0000 7fd4296c4700 10 client.36596.objecter _send_command 4 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.286+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4241031f0 con 0x7fd424059570 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd4241031f0 con 0x7fd424059570 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd40bfff700 10 client.36596.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd424059400 osd=1 4 2024-02-13T00:42:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 1 4 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4296c4700 10 client.36596.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 5 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd4240277c0 0x7fd424106550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424027650 osd=2 3 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424027650 osd=2 3 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424027650 osd=2 3 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 5 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd424027650 osd=2 3 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 2 5 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424027650 osd=2 4 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4296c4700 10 client.36596.objecter _send_command 5 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd424100850 con 0x7fd4240277c0 2024-02-13T00:42:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4227fc700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd4240277c0 0x7fd424106550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd4227fc700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd4240277c0 0x7fd424106550 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.288+0000 7fd40bfff700 10 client.36596.objecter ms_handle_connect 0x7fd4240277c0 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.289+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd424100850 con 0x7fd4240277c0 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.289+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.289+0000 7fd40bfff700 10 client.36596.objecter _finish_command 5 = system:0 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.289+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd424027650 osd=2 4 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.289+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 2 5 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.305+0000 7fd4296c4700 10 client.36596.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.305+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 6 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.305+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424027650 osd=2 3 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.305+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424027650 osd=2 3 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.305+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424027650 osd=2 3 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.305+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 6 2024-02-13T00:42:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.305+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd424027650 osd=2 3 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.305+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 2 6 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.305+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424027650 osd=2 4 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.305+0000 7fd4296c4700 10 client.36596.objecter _send_command 6 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.305+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd424106a40 con 0x7fd4240277c0 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd424106a40 con 0x7fd4240277c0 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd40bfff700 10 client.36596.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd424027650 osd=2 4 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 2 6 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4296c4700 10 client.36596.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 7 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6813/604971011 conn(0x7fd42412cc10 0x7fd42412f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424106e20 osd=3 3 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424106e20 osd=3 3 2024-02-13T00:42:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424106e20 osd=3 3 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 7 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd424106e20 osd=3 3 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 3 7 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424106e20 osd=3 4 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4296c4700 10 client.36596.objecter _send_command 7 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd424107010 con 0x7fd42412cc10 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.307+0000 7fd4237fe700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6813/604971011 conn(0x7fd42412cc10 0x7fd42412f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.308+0000 7fd40bfff700 10 client.36596.objecter ms_handle_connect 0x7fd42412cc10 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.308+0000 7fd4237fe700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6813/604971011 conn(0x7fd42412cc10 0x7fd42412f000 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.309+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd424107010 con 0x7fd42412cc10 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.309+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.309+0000 7fd40bfff700 10 client.36596.objecter _finish_command 7 = system:0 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.309+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd424106e20 osd=3 4 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.309+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 3 7 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.323+0000 7fd4296c4700 10 client.36596.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.323+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 8 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.323+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424106e20 osd=3 3 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.323+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424106e20 osd=3 3 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.323+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424106e20 osd=3 3 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.323+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 8 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.323+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd424106e20 osd=3 3 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.323+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 3 8 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.323+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424106e20 osd=3 4 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.323+0000 7fd4296c4700 10 client.36596.objecter _send_command 8 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.323+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4240ffdb0 con 0x7fd42412cc10 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd4240ffdb0 con 0x7fd42412cc10 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd40bfff700 10 client.36596.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd424106e20 osd=3 4 2024-02-13T00:42:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 3 8 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd4296c4700 10 client.36596.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 9 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd424104040 0x7fd424130670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd4240fff80 osd=4 3 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd4240fff80 osd=4 3 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd4240fff80 osd=4 3 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 9 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd4240fff80 osd=4 3 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 4 9 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd4240fff80 osd=4 4 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd4296c4700 10 client.36596.objecter _send_command 9 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4241000f0 con 0x7fd424104040 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.325+0000 7fd422ffd700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd424104040 0x7fd424130670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.326+0000 7fd40bfff700 10 client.36596.objecter ms_handle_connect 0x7fd424104040 2024-02-13T00:42:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.326+0000 7fd422ffd700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd424104040 0x7fd424130670 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.326+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd4241000f0 con 0x7fd424104040 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.326+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.326+0000 7fd40bfff700 10 client.36596.objecter _finish_command 9 = system:0 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.326+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd4240fff80 osd=4 4 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.326+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 4 9 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.354+0000 7fd4296c4700 10 client.36596.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.354+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 10 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.354+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd4240fff80 osd=4 3 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.354+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd4240fff80 osd=4 3 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.354+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd4240fff80 osd=4 3 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.354+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 10 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.354+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd4240fff80 osd=4 3 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.354+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 4 10 2024-02-13T00:42:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.354+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd4240fff80 osd=4 4 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.354+0000 7fd4296c4700 10 client.36596.objecter _send_command 10 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.354+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd4241000f0 con 0x7fd424104040 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd4241000f0 con 0x7fd424104040 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd40bfff700 10 client.36596.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd4240fff80 osd=4 4 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 4 10 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4296c4700 10 client.36596.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 11 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6804/2212296748 conn(0x7fd4241310b0 0x7fd4241314f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd4241000f0 osd=5 3 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd4241000f0 osd=5 3 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd4241000f0 osd=5 3 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 11 2024-02-13T00:42:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd4241000f0 osd=5 3 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 5 11 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd4241000f0 osd=5 4 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4296c4700 10 client.36596.objecter _send_command 11 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd424138400 con 0x7fd4241310b0 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.356+0000 7fd4227fc700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6804/2212296748 conn(0x7fd4241310b0 0x7fd4241314f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.357+0000 7fd40bfff700 10 client.36596.objecter ms_handle_connect 0x7fd4241310b0 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.357+0000 7fd4227fc700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6804/2212296748 conn(0x7fd4241310b0 0x7fd4241314f0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.358+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd424138400 con 0x7fd4241310b0 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.358+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.358+0000 7fd40bfff700 10 client.36596.objecter _finish_command 11 = system:0 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.358+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd4241000f0 osd=5 4 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.358+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 5 11 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.384+0000 7fd4296c4700 10 client.36596.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.384+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 12 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.384+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd4241000f0 osd=5 3 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.384+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd4241000f0 osd=5 3 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.384+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd4241000f0 osd=5 3 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.384+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 12 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.384+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd4241000f0 osd=5 3 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.384+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 5 12 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.384+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd4241000f0 osd=5 4 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.384+0000 7fd4296c4700 10 client.36596.objecter _send_command 12 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.384+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd424135ca0 con 0x7fd4241310b0 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.386+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd424135ca0 con 0x7fd4241310b0 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.386+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.386+0000 7fd40bfff700 10 client.36596.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.386+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd4241000f0 osd=5 4 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.386+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 5 12 2024-02-13T00:42:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4296c4700 10 client.36596.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 13 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd424136360 0x7fd424138790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424135ca0 osd=6 3 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424135ca0 osd=6 3 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424135ca0 osd=6 3 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 13 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd424135ca0 osd=6 3 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 6 13 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424135ca0 osd=6 4 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4296c4700 10 client.36596.objecter _send_command 13 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd42413b560 con 0x7fd424136360 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4237fe700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd424136360 0x7fd424138790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd4237fe700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd424136360 0x7fd424138790 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.391+0000 7fd40bfff700 10 client.36596.objecter ms_handle_connect 0x7fd424136360 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.392+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd42413b560 con 0x7fd424136360 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.392+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.392+0000 7fd40bfff700 10 client.36596.objecter _finish_command 13 = system:0 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.392+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd424135ca0 osd=6 4 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.392+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 6 13 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.420+0000 7fd4296c4700 10 client.36596.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.420+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 14 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.420+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424135ca0 osd=6 3 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.420+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424135ca0 osd=6 3 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.420+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd424135ca0 osd=6 3 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.420+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 14 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.420+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd424135ca0 osd=6 3 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.420+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 6 14 2024-02-13T00:42:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.420+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424135ca0 osd=6 4 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.420+0000 7fd4296c4700 10 client.36596.objecter _send_command 14 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.420+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd42413b560 con 0x7fd424136360 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.429+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd42413b560 con 0x7fd424136360 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.429+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.429+0000 7fd40bfff700 10 client.36596.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.429+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd424135ca0 osd=6 4 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.429+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 6 14 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.436+0000 7fd4296c4700 10 client.36596.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.436+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 15 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.436+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd42413a940 0x7fd42413ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.436+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd42413b560 osd=7 3 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.436+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd42413b560 osd=7 3 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.436+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd42413b560 osd=7 3 2024-02-13T00:42:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.436+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 15 2024-02-13T00:42:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.436+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd42413b560 osd=7 3 2024-02-13T00:42:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.436+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 7 15 2024-02-13T00:42:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.436+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd42413b560 osd=7 4 2024-02-13T00:42:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.436+0000 7fd4296c4700 10 client.36596.objecter _send_command 15 2024-02-13T00:42:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.436+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd42413e910 con 0x7fd42413a940 2024-02-13T00:42:07.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.439+0000 7fd422ffd700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd42413a940 0x7fd42413ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.446+0000 7fd422ffd700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd42413a940 0x7fd42413ada0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.446+0000 7fd40bfff700 10 client.36596.objecter ms_handle_connect 0x7fd42413a940 2024-02-13T00:42:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.448+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd42413e910 con 0x7fd42413a940 2024-02-13T00:42:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.448+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:42:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.448+0000 7fd40bfff700 10 client.36596.objecter _finish_command 15 = system:0 2024-02-13T00:42:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.448+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd42413b560 osd=7 4 2024-02-13T00:42:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.448+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 7 15 2024-02-13T00:42:07.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.473+0000 7fd4296c4700 10 client.36596.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:07.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.473+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign -1 16 2024-02-13T00:42:07.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.473+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd42413b560 osd=7 3 2024-02-13T00:42:07.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.473+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd42413b560 osd=7 3 2024-02-13T00:42:07.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.473+0000 7fd4296c4700 20 client.36596.objecter _get_session s=0x7fd42413b560 osd=7 3 2024-02-13T00:42:07.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.473+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_remove -1 16 2024-02-13T00:42:07.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.473+0000 7fd4296c4700 20 client.36596.objecter get_session s=0x7fd42413b560 osd=7 3 2024-02-13T00:42:07.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.473+0000 7fd4296c4700 15 client.36596.objecter _session_command_op_assign 7 16 2024-02-13T00:42:07.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.473+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd42413b560 osd=7 4 2024-02-13T00:42:07.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.473+0000 7fd4296c4700 10 client.36596.objecter _send_command 16 2024-02-13T00:42:07.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.473+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd424138d40 con 0x7fd42413a940 2024-02-13T00:42:07.478 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:42:07.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.475+0000 7fd40bfff700 1 -- 172.21.15.162:0/1816868755 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd424138d40 con 0x7fd42413a940 2024-02-13T00:42:07.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.475+0000 7fd40bfff700 10 client.36596.objecter ms_dispatch 0x7fd424001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:07.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.475+0000 7fd40bfff700 10 client.36596.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.475+0000 7fd40bfff700 20 client.36596.objecter put_session s=0x7fd42413b560 osd=7 4 2024-02-13T00:42:07.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.475+0000 7fd40bfff700 15 client.36596.objecter _session_command_op_remove 7 16 2024-02-13T00:42:07.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.475+0000 7fd4296c4700 10 librados: watch_flush enter 2024-02-13T00:42:07.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.475+0000 7fd4296c4700 10 librados: watch_flush exit 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.475+0000 7fd4296c4700 10 client.36596.objecter close_session for osd.0 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.475+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd424028590 msgr2=0x7fd424059990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.475+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd424028590 0x7fd424059990 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd42410ff70 osd=0 1 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 10 client.36596.objecter close_session for osd.1 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6805/150599780 conn(0x7fd424059570 msgr2=0x7fd424102d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6805/150599780 conn(0x7fd424059570 0x7fd424102d00 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424059400 osd=1 1 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 10 client.36596.objecter close_session for osd.2 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd4240277c0 msgr2=0x7fd424106550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd4240277c0 0x7fd424106550 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424027650 osd=2 1 2024-02-13T00:42:07.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 10 client.36596.objecter close_session for osd.3 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6813/604971011 conn(0x7fd42412cc10 msgr2=0x7fd42412f000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6813/604971011 conn(0x7fd42412cc10 0x7fd42412f000 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424106e20 osd=3 1 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 10 client.36596.objecter close_session for osd.4 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd424104040 msgr2=0x7fd424130670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd424104040 0x7fd424130670 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd4240fff80 osd=4 1 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 10 client.36596.objecter close_session for osd.5 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6804/2212296748 conn(0x7fd4241310b0 msgr2=0x7fd4241314f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6804/2212296748 conn(0x7fd4241310b0 0x7fd4241314f0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd4241000f0 osd=5 1 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 10 client.36596.objecter close_session for osd.6 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd424136360 msgr2=0x7fd424138790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd424136360 0x7fd424138790 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd424135ca0 osd=6 1 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 10 client.36596.objecter close_session for osd.7 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd42413a940 msgr2=0x7fd42413ada0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd42413a940 0x7fd42413ada0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 20 client.36596.objecter put_session s=0x7fd42413b560 osd=7 1 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 20 client.36596.objecter shutdown clearing up homeless session... 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 10 client.36596.objecter successfully canceled tick 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4237fe700 1 -- 172.21.15.162:0/1816868755 reap_dead start 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4237fe700 1 -- 172.21.15.162:0/1816868755 reap_dead start 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4237fe700 1 -- 172.21.15.162:0/1816868755 reap_dead start 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4237fe700 1 -- 172.21.15.162:0/1816868755 reap_dead start 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4237fe700 1 -- 172.21.15.162:0/1816868755 reap_dead start 2024-02-13T00:42:07.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd40c062c90 msgr2=0x7fd40c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd40c062c90 0x7fd40c065150 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 rx=0x7fd418004840 tx=0x7fd41800b040).stop 2024-02-13T00:42:07.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4240f6830 msgr2=0x7fd4241148f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:07.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4240f6830 0x7fd4241148f0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 rx=0x7fd41400e970 tx=0x7fd41400b040).stop 2024-02-13T00:42:07.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 shutdown_connections 2024-02-13T00:42:07.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4240f6830 0x7fd4241148f0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:07.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 --2- 172.21.15.162:0/1816868755 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd40c062c90 0x7fd40c065150 secure :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 rx=0x7fd418004840 tx=0x7fd41800b040).stop 2024-02-13T00:42:07.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.476+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 >> 172.21.15.162:0/1816868755 conn(0x7fd4240700a0 msgr2=0x7fd42406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:07.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.482+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 shutdown_connections 2024-02-13T00:42:07.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.482+0000 7fd4296c4700 1 -- 172.21.15.162:0/1816868755 wait complete. 2024-02-13T00:42:07.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:07.482+0000 7fd4296c4700 1 librados: shutdown 2024-02-13T00:42:07.492 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.492 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.492 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.492 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.492 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.492 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.492 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.492 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:42:07.533 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:07.890 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:07.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:07.906 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:42:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:42:07.906 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:42:07.906 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:42:07.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:08.021 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:08.043 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:42:08.044 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:42:08.044 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-13T00:42:08.044 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-13T00:42:08.265 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:06.320782+0000 mgr.y (mgr.14544) 207 : cluster [DBG] pgmap v226: 324 pgs: 2 active+recovery_wait, 32 unknown, 4 active+recovering, 31 active+recovery_wait+degraded, 255 active+clean; 216 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 208/3000 objects degraded (6.933%); 33/3000 objects misplaced (1.100%); 6.0 MiB/s, 249 keys/s, 26 objects/s recovering 2024-02-13T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:06.872553+0000 mon.a (mon.0) 1254 : cluster [WRN] Health check failed: Degraded data redundancy: 208/3000 objects degraded (6.933%), 31 pgs degraded (PG_DEGRADED) 2024-02-13T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:07.096752+0000 mon.b (mon.1) 212 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:07.248618+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? 172.21.15.162:0/1816868755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:07.400968+0000 mon.a (mon.0) 1255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-84081-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]': finished 2024-02-13T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:07.401126+0000 mon.a (mon.0) 1256 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-83977-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:07.401226+0000 mon.a (mon.0) 1257 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:07.401271+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e172: 8 total, 8 up, 7 in 2024-02-13T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:07.402228+0000 mon.a (mon.0) 1259 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-13T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:07.402947+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-13T00:42:08.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:06.320782+0000 mgr.y (mgr.14544) 207 : cluster [DBG] pgmap v226: 324 pgs: 2 active+recovery_wait, 32 unknown, 4 active+recovering, 31 active+recovery_wait+degraded, 255 active+clean; 216 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 208/3000 objects degraded (6.933%); 33/3000 objects misplaced (1.100%); 6.0 MiB/s, 249 keys/s, 26 objects/s recovering 2024-02-13T00:42:08.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:06.872553+0000 mon.a (mon.0) 1254 : cluster [WRN] Health check failed: Degraded data redundancy: 208/3000 objects degraded (6.933%), 31 pgs degraded (PG_DEGRADED) 2024-02-13T00:42:08.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:07.096752+0000 mon.b (mon.1) 212 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:07.248618+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? 172.21.15.162:0/1816868755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:07.400968+0000 mon.a (mon.0) 1255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-84081-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]': finished 2024-02-13T00:42:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:07.401126+0000 mon.a (mon.0) 1256 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-83977-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:07.401226+0000 mon.a (mon.0) 1257 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:07.401271+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e172: 8 total, 8 up, 7 in 2024-02-13T00:42:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:07.402228+0000 mon.a (mon.0) 1259 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-13T00:42:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:07.402947+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-13T00:42:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:06.320782+0000 mgr.y (mgr.14544) 207 : cluster [DBG] pgmap v226: 324 pgs: 2 active+recovery_wait, 32 unknown, 4 active+recovering, 31 active+recovery_wait+degraded, 255 active+clean; 216 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 208/3000 objects degraded (6.933%); 33/3000 objects misplaced (1.100%); 6.0 MiB/s, 249 keys/s, 26 objects/s recovering 2024-02-13T00:42:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:06.872553+0000 mon.a (mon.0) 1254 : cluster [WRN] Health check failed: Degraded data redundancy: 208/3000 objects degraded (6.933%), 31 pgs degraded (PG_DEGRADED) 2024-02-13T00:42:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:07.096752+0000 mon.b (mon.1) 212 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:07.248618+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? 172.21.15.162:0/1816868755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:07.400968+0000 mon.a (mon.0) 1255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-84081-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]': finished 2024-02-13T00:42:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:07.401126+0000 mon.a (mon.0) 1256 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-83977-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:07.401226+0000 mon.a (mon.0) 1257 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-84049-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:07.401271+0000 mon.a (mon.0) 1258 : cluster [DBG] osdmap e172: 8 total, 8 up, 7 in 2024-02-13T00:42:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:07.402228+0000 mon.a (mon.0) 1259 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-13T00:42:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:07.402947+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-13T00:42:08.373 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:08.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:08.458 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:08.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:08.507 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:08.705 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:08.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:08.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:08.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-13T00:42:08.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:08.834 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:42:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:42:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:42:08.836 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:08.860 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:08.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:08.959 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:08.097715+0000 mon.b (mon.1) 213 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:08.959 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:08.402515+0000 mon.a (mon.0) 1261 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-13T00:42:08.960 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:08.402728+0000 mon.a (mon.0) 1262 : cluster [DBG] osdmap e173: 8 total, 8 up, 7 in 2024-02-13T00:42:08.960 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:08.408713+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-13T00:42:08.960 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:08.416011+0000 mon.b (mon.1) 214 : audit [INF] from='client.? 172.21.15.162:0/2780403279' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-83985-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:08.960 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:08.419311+0000 mon.a (mon.0) 1264 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-83985-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:08.960 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:08.097715+0000 mon.b (mon.1) 213 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:08.960 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:08.402515+0000 mon.a (mon.0) 1261 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-13T00:42:08.960 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:08.402728+0000 mon.a (mon.0) 1262 : cluster [DBG] osdmap e173: 8 total, 8 up, 7 in 2024-02-13T00:42:08.960 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:08.408713+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-13T00:42:08.960 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:08.416011+0000 mon.b (mon.1) 214 : audit [INF] from='client.? 172.21.15.162:0/2780403279' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-83985-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:08.960 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:08.419311+0000 mon.a (mon.0) 1264 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-83985-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:08.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:08.992 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:09.004 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:42:09.014 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:09.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:09.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:09.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:08.097715+0000 mon.b (mon.1) 213 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:09.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:08.402515+0000 mon.a (mon.0) 1261 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-13T00:42:09.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:08.402728+0000 mon.a (mon.0) 1262 : cluster [DBG] osdmap e173: 8 total, 8 up, 7 in 2024-02-13T00:42:09.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:08.408713+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-13T00:42:09.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:08.416011+0000 mon.b (mon.1) 214 : audit [INF] from='client.? 172.21.15.162:0/2780403279' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-83985-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:09.144 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:08.419311+0000 mon.a (mon.0) 1264 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-83985-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:09.146 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:09.294 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:09.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:09.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:09.437 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:09.510 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:09.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:09.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:09.614 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:09.641 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:09.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:42:09.788 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:09.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:09.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:09.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:09.919 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:09.942 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:09.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:09.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:10.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:08.321280+0000 mgr.y (mgr.14544) 208 : cluster [DBG] pgmap v229: 332 pgs: 2 active+recovery_wait, 40 unknown, 4 active+recovering, 31 active+recovery_wait+degraded, 255 active+clean; 216 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 208/3000 objects degraded (6.933%); 33/3000 objects misplaced (1.100%); 6.0 MiB/s, 249 keys/s, 26 objects/s recovering 2024-02-13T00:42:10.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:09.098626+0000 mon.b (mon.1) 215 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:10.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:09.403661+0000 mon.a (mon.0) 1265 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-13T00:42:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:09.403848+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-83985-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:09.403990+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e174: 8 total, 8 up, 7 in 2024-02-13T00:42:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:09.407989+0000 mon.a (mon.0) 1268 : audit [INF] from='client.? 172.21.15.162:0/2413197893' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-83977-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:09.409393+0000 mon.a (mon.0) 1269 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key1"}]: dispatch 2024-02-13T00:42:10.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:42:10.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:08.321280+0000 mgr.y (mgr.14544) 208 : cluster [DBG] pgmap v229: 332 pgs: 2 active+recovery_wait, 40 unknown, 4 active+recovering, 31 active+recovery_wait+degraded, 255 active+clean; 216 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 208/3000 objects degraded (6.933%); 33/3000 objects misplaced (1.100%); 6.0 MiB/s, 249 keys/s, 26 objects/s recovering 2024-02-13T00:42:10.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:09.098626+0000 mon.b (mon.1) 215 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:10.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:09.403661+0000 mon.a (mon.0) 1265 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-13T00:42:10.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:09.403848+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-83985-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:10.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:09.403990+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e174: 8 total, 8 up, 7 in 2024-02-13T00:42:10.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:09.407989+0000 mon.a (mon.0) 1268 : audit [INF] from='client.? 172.21.15.162:0/2413197893' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-83977-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:09.409393+0000 mon.a (mon.0) 1269 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key1"}]: dispatch 2024-02-13T00:42:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:08.321280+0000 mgr.y (mgr.14544) 208 : cluster [DBG] pgmap v229: 332 pgs: 2 active+recovery_wait, 40 unknown, 4 active+recovering, 31 active+recovery_wait+degraded, 255 active+clean; 216 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 208/3000 objects degraded (6.933%); 33/3000 objects misplaced (1.100%); 6.0 MiB/s, 249 keys/s, 26 objects/s recovering 2024-02-13T00:42:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:09.098626+0000 mon.b (mon.1) 215 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:09.403661+0000 mon.a (mon.0) 1265 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-13T00:42:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:09.403848+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-83985-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:09.403990+0000 mon.a (mon.0) 1267 : cluster [DBG] osdmap e174: 8 total, 8 up, 7 in 2024-02-13T00:42:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:09.407989+0000 mon.a (mon.0) 1268 : audit [INF] from='client.? 172.21.15.162:0/2413197893' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-83977-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:09.409393+0000 mon.a (mon.0) 1269 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key1"}]: dispatch 2024-02-13T00:42:10.083 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:10.103 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:42:10.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:10.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:42:10.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:10.233 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:10.330 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:10.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:10.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.362+0000 7f66ff54e700 1 -- 172.21.15.162:0/896247647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66f80fe5b0 msgr2=0x7f66f80f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:10.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.362+0000 7f66ff54e700 1 --2- 172.21.15.162:0/896247647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66f80fe5b0 0x7f66f80f7de0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 rx=0x7f66e8004870 tx=0x7f66e8009cb0).stop 2024-02-13T00:42:10.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 -- 172.21.15.162:0/896247647 shutdown_connections 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 --2- 172.21.15.162:0/896247647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66f80f84c0 0x7f66f8104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 --2- 172.21.15.162:0/896247647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66f80fdc50 0x7f66f80fe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 --2- 172.21.15.162:0/896247647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66f80fe5b0 0x7f66f80f7de0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 -- 172.21.15.162:0/896247647 >> 172.21.15.162:0/896247647 conn(0x7f66f80f38a0 msgr2=0x7f66f80f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 -- 172.21.15.162:0/896247647 shutdown_connections 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 -- 172.21.15.162:0/896247647 wait complete. 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 librados: starting msgr at 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 librados: starting objecter 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 Processor -- start 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 -- start start 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 librados: setting wanted keys 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 librados: calling monclient init 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66f80f84c0 0x7f66f806b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:10.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66f80fdc50 0x7f66f806b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66f80fe5b0 0x7f66f806bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f8116600 con 0x7f66f80fdc50 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f8116770 con 0x7f66f80f84c0 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.363+0000 7f66ff54e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f8116930 con 0x7f66f80fe5b0 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66fcae9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66f80fdc50 0x7f66f806b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66fcae9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66f80fdc50 0x7f66f806b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:58802/0 (socket says 172.21.15.162:58802) 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66fcae9700 1 -- 172.21.15.162:0/3268303014 learned_addr learned my addr 172.21.15.162:0/3268303014 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66fdaeb700 1 --2- 172.21.15.162:0/3268303014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66f80fe5b0 0x7f66f806bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66fd2ea700 1 --2- 172.21.15.162:0/3268303014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66f80f84c0 0x7f66f806b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66fdaeb700 1 -- 172.21.15.162:0/3268303014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66f80f84c0 msgr2=0x7f66f806b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66fdaeb700 1 --2- 172.21.15.162:0/3268303014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66f80f84c0 0x7f66f806b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66fdaeb700 1 -- 172.21.15.162:0/3268303014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66f80fdc50 msgr2=0x7f66f806b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66fdaeb700 1 --2- 172.21.15.162:0/3268303014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66f80fdc50 0x7f66f806b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66fdaeb700 1 -- 172.21.15.162:0/3268303014 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e8005040 con 0x7f66f80fe5b0 2024-02-13T00:42:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66fdaeb700 1 --2- 172.21.15.162:0/3268303014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66f80fe5b0 0x7f66f806bb50 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f66f000d8d0 tx=0x7f66f0004db0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:10.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66ee7fc700 10 client.?.objecter ms_handle_connect 0x7f66f80fe5b0 2024-02-13T00:42:10.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66ee7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:10.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66ee7fc700 1 -- 172.21.15.162:0/3268303014 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0004020 con 0x7f66f80fe5b0 2024-02-13T00:42:10.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66ee7fc700 1 -- 172.21.15.162:0/3268303014 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f66f000f040 con 0x7f66f80fe5b0 2024-02-13T00:42:10.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66ff54e700 1 -- 172.21.15.162:0/3268303014 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f8116c10 con 0x7f66f80fe5b0 2024-02-13T00:42:10.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66ee7fc700 1 -- 172.21.15.162:0/3268303014 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0003430 con 0x7f66f80fe5b0 2024-02-13T00:42:10.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66ff54e700 10 client.36608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:10.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66ff54e700 1 -- 172.21.15.162:0/3268303014 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f8117130 con 0x7f66f80fe5b0 2024-02-13T00:42:10.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.364+0000 7f66ff54e700 1 librados: init done 2024-02-13T00:42:10.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.366+0000 7f66ff54e700 1 -- 172.21.15.162:0/3268303014 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f8107140 con 0x7f66f80fe5b0 2024-02-13T00:42:10.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.366+0000 7f66ee7fc700 1 -- 172.21.15.162:0/3268303014 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f66f0004180 con 0x7f66f80fe5b0 2024-02-13T00:42:10.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.367+0000 7f66ee7fc700 1 --2- 172.21.15.162:0/3268303014 >> v2:172.21.15.162:6800/2866534053 conn(0x7f66e4062c90 0x7f66e4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:10.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.367+0000 7f66ee7fc700 1 -- 172.21.15.162:0/3268303014 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(175..175 src has 52..175) v4 ==== 9836+0+0 (secure 0 0 0) 0x7f66f007db10 con 0x7f66f80fe5b0 2024-02-13T00:42:10.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.367+0000 7f66ee7fc700 10 client.36608.objecter ms_dispatch 0x7f66f8001550 osd_map(175..175 src has 52..175) v4 2024-02-13T00:42:10.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.367+0000 7f66ee7fc700 3 client.36608.objecter handle_osd_map got epochs [175,175] > 0 2024-02-13T00:42:10.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.367+0000 7f66ee7fc700 3 client.36608.objecter handle_osd_map decoding full epoch 175 2024-02-13T00:42:10.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.367+0000 7f66fd2ea700 1 --2- 172.21.15.162:0/3268303014 >> v2:172.21.15.162:6800/2866534053 conn(0x7f66e4062c90 0x7f66e4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:10.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.367+0000 7f66ee7fc700 20 client.36608.objecter dump_active .. 0 homeless 2024-02-13T00:42:10.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.367+0000 7f66ee7fc700 10 client.36608.objecter ms_handle_connect 0x7f66e4062c90 2024-02-13T00:42:10.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.367+0000 7f66fd2ea700 1 --2- 172.21.15.162:0/3268303014 >> v2:172.21.15.162:6800/2866534053 conn(0x7f66e4062c90 0x7f66e4065150 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 rx=0x7f66f40048b0 tx=0x7f66f400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:10.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.373+0000 7f66ee7fc700 1 -- 172.21.15.162:0/3268303014 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f66f00508f0 con 0x7f66f80fe5b0 2024-02-13T00:42:10.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:42:10.436 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:10.463 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:10.464 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:10.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.565+0000 7f66ff54e700 1 -- 172.21.15.162:0/3268303014 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f66f806c650 con 0x7f66f80fe5b0 2024-02-13T00:42:10.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.567+0000 7f66ee7fc700 1 -- 172.21.15.162:0/3268303014 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v175) v1 ==== 81+0+57546 (secure 0 0 0) 0x7f66f0021070 con 0x7f66f80fe5b0 2024-02-13T00:42:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:42:10.572 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:42:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "epoch": 175, 2024-02-13T00:42:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:42:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "created": "2024-02-13T00:31:51.797017+0000", 2024-02-13T00:42:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "modified": "2024-02-13T00:42:10.306086+0000", 2024-02-13T00:42:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "last_up_change": "2024-02-13T00:37:33.855829+0000", 2024-02-13T00:42:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "last_in_change": "2024-02-13T00:42:02.331570+0000", 2024-02-13T00:42:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-13T00:42:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_num": 5806080, 2024-02-13T00:42:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_set": [ 2024-02-13T00:42:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "nodeep-scrub", 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "noscrub", 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "pglog_hardlimit", 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "purged_snapdirs", 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_deletes", 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "sortbitwise" 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_version": 32, 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "full_ratio": 0.94999998807907104, 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_snapshot": "", 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_max": 115, 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "max_osd": 8, 2024-02-13T00:42:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "require_min_compat_client": "luminous", 2024-02-13T00:42:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "min_compat_client": "luminous", 2024-02-13T00:42:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "require_osd_release": "pacific", 2024-02-13T00:42:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "pools": [ 2024-02-13T00:42:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 1, 2024-02-13T00:42:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "device_health_metrics", 2024-02-13T00:42:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:35:34.545390+0000", 2024-02-13T00:42:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:42:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:42:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:42:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:42:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 1, 2024-02-13T00:42:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 1, 2024-02-13T00:42:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 1, 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 1, 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 1, 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "71", 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:42:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:42:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_max": 32, 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_min": 1 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:42:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_devicehealth": {} 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 2, 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": ".rgw.root", 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.459980+0000", 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:42:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "58", 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 3, 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "iscsi", 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.740219+0000", 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:42:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 3, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 3, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 3, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 3, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 3, 2024-02-13T00:42:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "102", 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:42:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:42:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:42:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:42:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd": {} 2024-02-13T00:42:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 4, 2024-02-13T00:42:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.log", 2024-02-13T00:42:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:41.489292+0000", 2024-02-13T00:42:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:42:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:42:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:42:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "60", 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:42:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:42:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 5, 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.control", 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:43.466304+0000", 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "62", 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:42:10.601 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 6, 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.meta", 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:45.542442+0000", 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:42:10.602 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:42:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "64", 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.604 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.605 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.606 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_bias": 4 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.607 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 18, 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "LibRadosList_smithi162-84017-1", 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:40:57.711006+0000", 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.608 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:42:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:42:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:42:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:42:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:42:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.609 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "108", 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:42:10.610 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:42:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:42:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:42:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:42:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:42:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.611 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.612 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 20, 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "LibRadosMiscPP_smithi162-84049-1", 2024-02-13T00:42:10.613 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:40:58.193277+0000", 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.614 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "175", 2024-02-13T00:42:10.615 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:42:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:42:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:42:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:42:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:42:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:42:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:42:10.616 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.617 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:42:10.618 INFO:teuthology.orchestra.run.smithi162.stdout: "app1": { 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "key2": "value2" 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "app2": {}, 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 28, 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:00.190777+0000", 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:42:10.619 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:42:10.620 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "108", 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:42:10.621 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.622 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.623 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 61, 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-6", 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:14.963279+0000", 2024-02-13T00:42:10.624 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8193, 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:42:10.625 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "156", 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "156", 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "156", 2024-02-13T00:42:10.626 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "156", 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 7, 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 155, 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [ 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: 95 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": 95, 2024-02-13T00:42:10.627 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": 95, 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.628 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:42:10.629 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 95, 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-13", 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:43.959940+0000", 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8201, 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.630 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.631 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "157", 2024-02-13T00:42:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "156", 2024-02-13T00:42:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "156", 2024-02-13T00:42:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "156", 2024-02-13T00:42:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:42:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 7, 2024-02-13T00:42:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 155, 2024-02-13T00:42:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": 61, 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "writeback", 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:42:10.634 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 113, 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "LibRadosSnapshotsECPP_smithi162-84081-15", 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:42:06.676273+0000", 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 16385, 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,pool_snaps", 2024-02-13T00:42:10.635 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 3, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 1, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 8, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 8, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 8, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 8, 2024-02-13T00:42:10.636 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 8, 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "175", 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.637 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "pool", 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 2, 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 175, 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.638 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15", 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.639 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 8192, 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 115, 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "ExecuteClass_smithi162-83977-23", 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:42:08.724014+0000", 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 32769, 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,creating", 2024-02-13T00:42:10.640 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:42:10.641 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "175", 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:42:10.642 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:42:10.643 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:42:10.644 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:42:10.645 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99", 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 8, 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 174, 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:42:10.646 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6801", 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6802", 2024-02-13T00:42:10.647 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6804", 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.648 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6803", 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6801/2103743338", 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6802/2103743338", 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6804/2103743338", 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6803/2103743338", 2024-02-13T00:42:10.649 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886", 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:42:10.650 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 13, 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 174, 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6805", 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.651 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6806", 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.652 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6808", 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6807", 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.653 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6805/150599780", 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6806/150599780", 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6808/150599780", 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6807/150599780", 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e", 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:42:10.654 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 18, 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 174, 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6809", 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.655 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6810", 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6812", 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6811", 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.657 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6809/3689719376", 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6810/3689719376", 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6812/3689719376", 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6811/3689719376", 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001", 2024-02-13T00:42:10.658 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 25, 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 173, 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.659 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6813", 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6814", 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6816", 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.661 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6815", 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6813/604971011", 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6814/604971011", 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6816/604971011", 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6815/604971011", 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3", 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 31, 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 81, 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:42:10.663 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6800", 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.664 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6801", 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6803", 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.665 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6802", 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6800/3344263388", 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6801/3344263388", 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6803/3344263388", 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6802/3344263388", 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:42:10.666 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723", 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 39, 2024-02-13T00:42:10.667 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 174, 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6804", 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:42:10.668 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6805", 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6807", 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:42:10.669 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6806", 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6804/2212296748", 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6805/2212296748", 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2212296748", 2024-02-13T00:42:10.670 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2212296748", 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474", 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:42:10.671 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 45, 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 165, 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6808", 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6809", 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.673 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6811", 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6810", 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6808/1506640341", 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6809/1506640341", 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1506640341", 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1506640341", 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b", 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:42:10.675 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:42:10.676 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:42:10.676 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:42:10.676 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 53, 2024-02-13T00:42:10.676 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 174, 2024-02-13T00:42:10.676 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:42:10.676 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:42:10.676 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:42:10.676 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.676 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.676 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6812", 2024-02-13T00:42:10.676 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6813", 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:42:10.677 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6815", 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6814", 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.678 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6812/1964192772", 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6813/1964192772", 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6815/1964192772", 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6814/1964192772", 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_xinfo": [ 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.679 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:40:48.185479+0000", 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:42:10.680 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:40:50.400426+0000", 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:41:33.001722+0000", 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:42:10.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:41:27.362685+0000", 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:42:10.682 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:41:11.303912+0000", 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:42:10.683 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:41:25.171981+0000", 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:40:42.541395+0000", 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:42:10.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:41:23.730592+0000", 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap": [ 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.685 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "20.c", 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: 5, 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: 6, 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: 4 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap_items": [ 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "2.0", 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 7, 2024-02-13T00:42:10.686 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 2 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.2", 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 2 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.687 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.6", 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 5 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.13", 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:42:10.688 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 2 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_temp": [], 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_temp": [], 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: "blocklist": { 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/1909610488": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2370739667": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/1502590187": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/4121673848": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2412382714": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3041562701": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:42:10.689 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/2775563314": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3135632190": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/197307636": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2653247729": "2024-02-14T00:32:24.380625+0000" 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: "range_blocklist": {}, 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profiles": { 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: "default": { 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:42:10.690 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van" 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15": { 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-device-class": "", 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-root": "default", 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van", 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: "w": "8" 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.692 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps_queue": [ 2024-02-13T00:42:10.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.692 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 24, 2024-02-13T00:42:10.692 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:42:10.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.692 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:42:10.692 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.692 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:42:10.692 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 9, 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 25, 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 60, 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:42:10.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 78, 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:42:10.695 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.696 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 85, 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 94, 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:42:10.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 107, 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.698 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 113, 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.699 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: "new_removed_snaps": [ 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 113, 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: "new_purged_snaps": [], 2024-02-13T00:42:10.700 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_node_flags": {}, 2024-02-13T00:42:10.701 INFO:teuthology.orchestra.run.smithi162.stdout: "device_class_flags": {}, 2024-02-13T00:42:10.701 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode": { 2024-02-13T00:42:10.701 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_enabled": false, 2024-02-13T00:42:10.701 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_bucket_count": 0, 2024-02-13T00:42:10.701 INFO:teuthology.orchestra.run.smithi162.stdout: "degraded_stretch_mode": 0, 2024-02-13T00:42:10.701 INFO:teuthology.orchestra.run.smithi162.stdout: "recovering_stretch_mode": 0, 2024-02-13T00:42:10.701 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_bucket": 0 2024-02-13T00:42:10.701 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:10.701 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:42:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:10.703 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:10.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.569+0000 7f66ff54e700 10 librados: watch_flush enter 2024-02-13T00:42:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.569+0000 7f66ff54e700 10 librados: watch_flush exit 2024-02-13T00:42:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.569+0000 7f66ff54e700 20 client.36608.objecter shutdown clearing up homeless session... 2024-02-13T00:42:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.569+0000 7f66ff54e700 10 client.36608.objecter successfully canceled tick 2024-02-13T00:42:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 -- 172.21.15.162:0/3268303014 >> v2:172.21.15.162:6800/2866534053 conn(0x7f66e4062c90 msgr2=0x7f66e4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 --2- 172.21.15.162:0/3268303014 >> v2:172.21.15.162:6800/2866534053 conn(0x7f66e4062c90 0x7f66e4065150 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 rx=0x7f66f40048b0 tx=0x7f66f400b040).stop 2024-02-13T00:42:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 -- 172.21.15.162:0/3268303014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66f80fe5b0 msgr2=0x7f66f806bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 --2- 172.21.15.162:0/3268303014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66f80fe5b0 0x7f66f806bb50 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f66f000d8d0 tx=0x7f66f0004db0).stop 2024-02-13T00:42:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 -- 172.21.15.162:0/3268303014 shutdown_connections 2024-02-13T00:42:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 --2- 172.21.15.162:0/3268303014 >> v2:172.21.15.162:6800/2866534053 conn(0x7f66e4062c90 0x7f66e4065150 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 --2- 172.21.15.162:0/3268303014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66f80fe5b0 0x7f66f806bb50 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 --2- 172.21.15.162:0/3268303014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66f80f84c0 0x7f66f806b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 --2- 172.21.15.162:0/3268303014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66f80fdc50 0x7f66f806b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 -- 172.21.15.162:0/3268303014 >> 172.21.15.162:0/3268303014 conn(0x7f66f80f38a0 msgr2=0x7f66f80fb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 -- 172.21.15.162:0/3268303014 shutdown_connections 2024-02-13T00:42:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 -- 172.21.15.162:0/3268303014 wait complete. 2024-02-13T00:42:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:10.570+0000 7f66ff54e700 1 librados: shutdown 2024-02-13T00:42:10.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:10.833 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:10.896 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:42:10.896 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:42:10.896 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-02-13T00:42:10.896 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-02-13T00:42:10.896 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:42:11.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:10.100046+0000 mon.b (mon.1) 216 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:11.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:10.311280+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? 172.21.15.162:0/2413197893' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-83977-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:11.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:10.311376+0000 mon.a (mon.0) 1271 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key1"}]': finished 2024-02-13T00:42:11.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:10.311526+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e175: 8 total, 8 up, 7 in 2024-02-13T00:42:11.083 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:10.566630+0000 mon.c (mon.2) 153 : audit [DBG] from='client.? 172.21.15.162:0/3268303014' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:42:11.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:10.100046+0000 mon.b (mon.1) 216 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:11.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:10.311280+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? 172.21.15.162:0/2413197893' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-83977-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:11.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:10.311376+0000 mon.a (mon.0) 1271 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key1"}]': finished 2024-02-13T00:42:11.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:10.311526+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e175: 8 total, 8 up, 7 in 2024-02-13T00:42:11.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:10.566630+0000 mon.c (mon.2) 153 : audit [DBG] from='client.? 172.21.15.162:0/3268303014' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:42:11.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:10.100046+0000 mon.b (mon.1) 216 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:11.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:10.311280+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? 172.21.15.162:0/2413197893' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-83977-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:11.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:10.311376+0000 mon.a (mon.0) 1271 : audit [INF] from='client.? 172.21.15.162:0/3870171401' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-84049-1","app":"app1","key":"key1"}]': finished 2024-02-13T00:42:11.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:10.311526+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e175: 8 total, 8 up, 7 in 2024-02-13T00:42:11.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:10.566630+0000 mon.c (mon.2) 153 : audit [DBG] from='client.? 172.21.15.162:0/3268303014' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:42:11.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-02-13T00:42:11.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: seed 84049 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (3 ms) 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (263 ms) 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (12 ms) 2024-02-13T00:42:11.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (9 ms) 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (21 ms) 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (5 ms) 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (8 ms) 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (4 ms) 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (15 ms) 2024-02-13T00:42:11.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (3 ms) 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (10 ms) 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (13 ms) 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: osd_max_attr_size = 0 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2354 ms) 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (343 ms) 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-02-13T00:42:11.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: done waiting, doing copies 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: waiting for final deep scrubs... 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: done waiting 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (61426 ms) 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (4 ms) 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (4 ms) 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4314 ms) 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-02-13T00:42:11.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-02-13T00:42:11.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-02-13T00:42:11.384 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:11.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:11.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:11.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:11.520 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:11.768 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:11.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:11.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-13T00:42:11.851 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:11.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:11.902 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:11.921 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:11.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:11.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:12.025 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:10.330854+0000 mgr.y (mgr.14544) 209 : cluster [DBG] pgmap v233: 332 pgs: 32 creating+peering, 300 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.3 KiB/s wr, 3 op/s; 6.1 MiB/s, 254 keys/s, 61 objects/s recovering 2024-02-13T00:42:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:10.699255+0000 osd.6 (osd.6) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:10.699429+0000 osd.6 (osd.6) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:11.101065+0000 mon.b (mon.1) 217 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:11.319709+0000 mon.a (mon.0) 1273 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:42:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:11.319744+0000 mon.a (mon.0) 1274 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 208/3000 objects degraded (6.933%), 31 pgs degraded) 2024-02-13T00:42:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:11.328079+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e176: 8 total, 8 up, 7 in 2024-02-13T00:42:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:11.333517+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? 172.21.15.162:0/2631190006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi162-83985-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:11.359891+0000 mon.b (mon.1) 218 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:11.366387+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:11.367852+0000 mon.b (mon.1) 219 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:11.368387+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:11.370956+0000 mon.b (mon.1) 220 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84049-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:11.371700+0000 mon.a (mon.0) 1279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84049-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:12.053 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:12.073 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:12.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:12.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:10.330854+0000 mgr.y (mgr.14544) 209 : cluster [DBG] pgmap v233: 332 pgs: 32 creating+peering, 300 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.3 KiB/s wr, 3 op/s; 6.1 MiB/s, 254 keys/s, 61 objects/s recovering 2024-02-13T00:42:12.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:10.699255+0000 osd.6 (osd.6) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:12.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:10.699429+0000 osd.6 (osd.6) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:11.101065+0000 mon.b (mon.1) 217 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:11.319709+0000 mon.a (mon.0) 1273 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:42:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:11.319744+0000 mon.a (mon.0) 1274 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 208/3000 objects degraded (6.933%), 31 pgs degraded) 2024-02-13T00:42:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:11.328079+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e176: 8 total, 8 up, 7 in 2024-02-13T00:42:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:11.333517+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? 172.21.15.162:0/2631190006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi162-83985-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:11.359891+0000 mon.b (mon.1) 218 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:11.366387+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:11.367852+0000 mon.b (mon.1) 219 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:11.368387+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:11.370956+0000 mon.b (mon.1) 220 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84049-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:11.371700+0000 mon.a (mon.0) 1279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84049-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:12.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:10.330854+0000 mgr.y (mgr.14544) 209 : cluster [DBG] pgmap v233: 332 pgs: 32 creating+peering, 300 active+clean; 216 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.3 KiB/s wr, 3 op/s; 6.1 MiB/s, 254 keys/s, 61 objects/s recovering 2024-02-13T00:42:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:10.699255+0000 osd.6 (osd.6) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:10.699429+0000 osd.6 (osd.6) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:11.101065+0000 mon.b (mon.1) 217 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:11.319709+0000 mon.a (mon.0) 1273 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:42:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:11.319744+0000 mon.a (mon.0) 1274 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 208/3000 objects degraded (6.933%), 31 pgs degraded) 2024-02-13T00:42:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:11.328079+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e176: 8 total, 8 up, 7 in 2024-02-13T00:42:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:11.333517+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? 172.21.15.162:0/2631190006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi162-83985-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:11.359891+0000 mon.b (mon.1) 218 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:11.366387+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:11.367852+0000 mon.b (mon.1) 219 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:12.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:11.368387+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:12.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:11.370956+0000 mon.b (mon.1) 220 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84049-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:12.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:11.371700+0000 mon.a (mon.0) 1279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84049-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:12.093 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:42:12.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:42:12.205 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:12.277 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:42:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:12.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:42:12.288 INFO:tasks.thrashosds.thrasher:j is {'epoch': 175, 'fsid': 'd4ee01fc-ca06-11ee-95b9-87774f69a715', 'created': '2024-02-13T00:31:51.797017+0000', 'modified': '2024-02-13T00:42:10.306086+0000', 'last_up_change': '2024-02-13T00:37:33.855829+0000', 'last_in_change': '2024-02-13T00:42:02.331570+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 32, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 115, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-13T00:35:34.545390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-13T00:37:39.459980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-13T00:37:39.740219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-13T00:37:41.489292+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-13T00:37:43.466304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-13T00:37:45.542442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 18, 'pool_name': 'LibRadosList_smithi162-84017-1', 'create_time': '2024-02-13T00:40:57.711006+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 20, 'pool_name': 'LibRadosMiscPP_smithi162-84049-1', 'create_time': '2024-02-13T00:40:58.193277+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '175', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'app1': {'key2': 'value2'}, 'app2': {}, 'rados': {}}}, {'pool': 28, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-13T00:41:00.190777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 61, 'pool_name': 'test-rados-api-smithi162-84057-6', 'create_time': '2024-02-13T00:41:14.963279+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '156', 'last_force_op_resend': '156', 'last_force_op_resend_prenautilus': '156', 'last_force_op_resend_preluminous': '156', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 155, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [95], 'tier_of': -1, 'read_tier': 95, 'write_tier': 95, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 95, 'pool_name': 'test-rados-api-smithi162-84057-13', 'create_time': '2024-02-13T00:41:43.959940+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '157', 'last_force_op_resend': '156', 'last_force_op_resend_prenautilus': '156', 'last_force_op_resend_preluminous': '156', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 155, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 61, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 113, 'pool_name': 'LibRadosSnapshotsECPP_smithi162-84081-15', 'create_time': '2024-02-13T00:42:06.676273+0000', 'flags': 16385, 'flags_names': 'hashpspool,pool_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '175', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'pool', 'snap_seq': 2, 'snap_epoch': 175, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-LibRadosSnapshotsECPP_smithi162-84081-15', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 115, 'pool_name': 'ExecuteClass_smithi162-83977-23', 'create_time': '2024-02-13T00:42:08.724014+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '175', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 174, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6801', 'nonce': 2103743338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 2103743338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 2103743338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6803', 'nonce': 2103743338}]}, 'public_addr': '172.21.15.162:6801/2103743338', 'cluster_addr': '172.21.15.162:6802/2103743338', 'heartbeat_back_addr': '172.21.15.162:6804/2103743338', 'heartbeat_front_addr': '172.21.15.162:6803/2103743338', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '39e9d2ef-9aa4-45bb-8a47-b0e0ec222886', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 174, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6805', 'nonce': 150599780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 150599780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 150599780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6807', 'nonce': 150599780}]}, 'public_addr': '172.21.15.162:6805/150599780', 'cluster_addr': '172.21.15.162:6806/150599780', 'heartbeat_back_addr': '172.21.15.162:6808/150599780', 'heartbeat_front_addr': '172.21.15.162:6807/150599780', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 174, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6809', 'nonce': 3689719376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3689719376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3689719376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6811', 'nonce': 3689719376}]}, 'public_addr': '172.21.15.162:6809/3689719376', 'cluster_addr': '172.21.15.162:6810/3689719376', 'heartbeat_back_addr': '172.21.15.162:6812/3689719376', 'heartbeat_front_addr': '172.21.15.162:6811/3689719376', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e6b9f0e-7a18-4cf8-9331-1a973fadc001', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6813', 'nonce': 604971011}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 604971011}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 604971011}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6815', 'nonce': 604971011}]}, 'public_addr': '172.21.15.162:6813/604971011', 'cluster_addr': '172.21.15.162:6814/604971011', 'heartbeat_back_addr': '172.21.15.162:6816/604971011', 'heartbeat_front_addr': '172.21.15.162:6815/604971011', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 3344263388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 3344263388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 3344263388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 3344263388}]}, 'public_addr': '172.21.15.181:6800/3344263388', 'cluster_addr': '172.21.15.181:6801/3344263388', 'heartbeat_back_addr': '172.21.15.181:6803/3344263388', 'heartbeat_front_addr': '172.21.15.181:6802/3344263388', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6aa2bd68-7f4b-4973-ac49-7fbc57c1a723', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 174, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2212296748}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2212296748}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2212296748}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2212296748}]}, 'public_addr': '172.21.15.181:6804/2212296748', 'cluster_addr': '172.21.15.181:6805/2212296748', 'heartbeat_back_addr': '172.21.15.181:6807/2212296748', 'heartbeat_front_addr': '172.21.15.181:6806/2212296748', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '791ba814-5284-432e-addc-d6b7f5f3d474', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 165, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1506640341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1506640341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1506640341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1506640341}]}, 'public_addr': '172.21.15.181:6808/1506640341', 'cluster_addr': '172.21.15.181:6809/1506640341', 'heartbeat_back_addr': '172.21.15.181:6811/1506640341', 'heartbeat_front_addr': '172.21.15.181:6810/1506640341', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f369c65-e663-4eb2-87a5-f2900f15b35b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 174, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 1964192772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 1964192772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 1964192772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 1964192772}]}, 'public_addr': '172.21.15.181:6812/1964192772', 'cluster_addr': '172.21.15.181:6813/1964192772', 'heartbeat_back_addr': '172.21.15.181:6815/1964192772', 'heartbeat_front_addr': '172.21.15.181:6814/1964192772', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:40:48.185479+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:40:50.400426+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:41:33.001722+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:41:27.362685+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:41:11.303912+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:41:25.171981+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:40:42.541395+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:41:23.730592+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '20.c', 'osds': [5, 6, 4]}], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.162:6800/1909610488': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2370739667': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/1502590187': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/4121673848': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:0/2412382714': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/3041562701': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/2775563314': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/3135632190': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/197307636': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2653247729': '2024-02-14T00:32:24.380625+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-LibRadosSnapshotsECPP_smithi162-84081-15': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 24, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 60, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 78, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 94, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 113, 'snaps': [{'begin': 1, 'length': 1}]}], 'new_removed_snaps': [{'pool': 113, 'snaps': [{'begin': 1, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-13T00:42:12.289 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:42:12.381 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:12.408 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:12.532 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:12.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:12.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:12.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:12.664 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:13.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.970+0000 7f24a94dc700 1 -- 172.21.15.162:0/810374635 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a40fc300 msgr2=0x7f24a40fc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:13.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.970+0000 7f24a94dc700 1 --2- 172.21.15.162:0/810374635 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a40fc300 0x7f24a40fc6e0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 rx=0x7f2498004870 tx=0x7f2498009cb0).stop 2024-02-13T00:42:13.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.970+0000 7f24a94dc700 1 -- 172.21.15.162:0/810374635 shutdown_connections 2024-02-13T00:42:13.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.970+0000 7f24a94dc700 1 --2- 172.21.15.162:0/810374635 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24a40f6b70 0x7f24a4103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:13.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.970+0000 7f24a94dc700 1 --2- 172.21.15.162:0/810374635 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a40fc300 0x7f24a40fc6e0 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:13.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.970+0000 7f24a94dc700 1 --2- 172.21.15.162:0/810374635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a40fcc60 0x7f24a40f6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:13.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.970+0000 7f24a94dc700 1 -- 172.21.15.162:0/810374635 >> 172.21.15.162:0/810374635 conn(0x7f24a4070100 msgr2=0x7f24a40f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:13.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.970+0000 7f24a94dc700 1 -- 172.21.15.162:0/810374635 shutdown_connections 2024-02-13T00:42:13.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.970+0000 7f24a94dc700 1 -- 172.21.15.162:0/810374635 wait complete. 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.970+0000 7f24a94dc700 1 librados: starting msgr at 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.970+0000 7f24a94dc700 1 librados: starting objecter 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a94dc700 1 Processor -- start 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a94dc700 1 -- start start 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a94dc700 1 librados: setting wanted keys 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a94dc700 1 librados: calling monclient init 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a94dc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a40f6b70 0x7f24a41092a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a94dc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24a40fc300 0x7f24a4109790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a94dc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a40fcc60 0x7f24a4109d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a94dc700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a4114480 con 0x7f24a40f6b70 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a94dc700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a41145f0 con 0x7f24a40fc300 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a94dc700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a41147b0 con 0x7f24a40fcc60 2024-02-13T00:42:13.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a40f6b70 0x7f24a41092a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a40f6b70 0x7f24a41092a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:58862/0 (socket says 172.21.15.162:58862) 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a2ffd700 1 -- 172.21.15.162:0/3530947426 learned_addr learned my addr 172.21.15.162:0/3530947426 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a37fe700 1 --2- 172.21.15.162:0/3530947426 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a40fcc60 0x7f24a4109d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a27fc700 1 --2- 172.21.15.162:0/3530947426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24a40fc300 0x7f24a4109790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a2ffd700 1 -- 172.21.15.162:0/3530947426 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a40fcc60 msgr2=0x7f24a4109d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a2ffd700 1 --2- 172.21.15.162:0/3530947426 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a40fcc60 0x7f24a4109d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a2ffd700 1 -- 172.21.15.162:0/3530947426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24a40fc300 msgr2=0x7f24a4109790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.971+0000 7f24a2ffd700 1 --2- 172.21.15.162:0/3530947426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24a40fc300 0x7f24a4109790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.972+0000 7f24a2ffd700 1 -- 172.21.15.162:0/3530947426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2494005040 con 0x7f24a40f6b70 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.972+0000 7f24a2ffd700 1 --2- 172.21.15.162:0/3530947426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a40f6b70 0x7f24a41092a0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 rx=0x7f2498009eb0 tx=0x7f249800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.972+0000 7f248bfff700 10 client.?.objecter ms_handle_connect 0x7f24a40f6b70 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.972+0000 7f248bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.972+0000 7f248bfff700 1 -- 172.21.15.162:0/3530947426 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249800e070 con 0x7f24a40f6b70 2024-02-13T00:42:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.972+0000 7f248bfff700 1 -- 172.21.15.162:0/3530947426 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2498013da0 con 0x7f24a40f6b70 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.972+0000 7f24a94dc700 1 -- 172.21.15.162:0/3530947426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2498005040 con 0x7f24a40f6b70 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.972+0000 7f248bfff700 1 -- 172.21.15.162:0/3530947426 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498019070 con 0x7f24a40f6b70 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.972+0000 7f24a94dc700 10 client.26517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.972+0000 7f24a94dc700 1 -- 172.21.15.162:0/3530947426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a4114d80 con 0x7f24a40f6b70 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.972+0000 7f24a94dc700 1 librados: init done 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.973+0000 7f24a94dc700 1 -- 172.21.15.162:0/3530947426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a405f980 con 0x7f24a40f6b70 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.974+0000 7f248bfff700 1 -- 172.21.15.162:0/3530947426 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f24980090d0 con 0x7f24a40f6b70 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.974+0000 7f248bfff700 1 --2- 172.21.15.162:0/3530947426 >> v2:172.21.15.162:6800/2866534053 conn(0x7f248c062c90 0x7f248c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.974+0000 7f248bfff700 1 -- 172.21.15.162:0/3530947426 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(177..177 src has 52..177) v4 ==== 10053+0+0 (secure 0 0 0) 0x7f249807e430 con 0x7f24a40f6b70 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.974+0000 7f248bfff700 10 client.26517.objecter ms_dispatch 0x7f24a4001550 osd_map(177..177 src has 52..177) v4 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.974+0000 7f248bfff700 3 client.26517.objecter handle_osd_map got epochs [177,177] > 0 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.974+0000 7f248bfff700 3 client.26517.objecter handle_osd_map decoding full epoch 177 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.974+0000 7f24a27fc700 1 --2- 172.21.15.162:0/3530947426 >> v2:172.21.15.162:6800/2866534053 conn(0x7f248c062c90 0x7f248c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:13.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.974+0000 7f248bfff700 20 client.26517.objecter dump_active .. 0 homeless 2024-02-13T00:42:13.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.974+0000 7f248bfff700 10 client.26517.objecter ms_handle_connect 0x7f248c062c90 2024-02-13T00:42:13.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.974+0000 7f24a27fc700 1 --2- 172.21.15.162:0/3530947426 >> v2:172.21.15.162:6800/2866534053 conn(0x7f248c062c90 0x7f248c065150 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 rx=0x7f24a40f8890 tx=0x7f249400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:13.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:12.979+0000 7f248bfff700 1 -- 172.21.15.162:0/3530947426 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2498051130 con 0x7f24a40f6b70 2024-02-13T00:42:13.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.157+0000 7f24a94dc700 1 -- 172.21.15.162:0/3530947426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f24a40475e0 con 0x7f24a40f6b70 2024-02-13T00:42:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:12.102125+0000 mon.b (mon.1) 221 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:12.329995+0000 mon.a (mon.0) 1280 : audit [INF] from='client.? 172.21.15.162:0/2631190006' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi162-83985-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:12.330131+0000 mon.a (mon.0) 1281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84049-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:12.330225+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e177: 8 total, 8 up, 7 in 2024-02-13T00:42:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:12.334677+0000 mon.c (mon.2) 154 : audit [INF] from='client.? 172.21.15.162:0/2221600674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-83977-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:12.336468+0000 mon.b (mon.1) 222 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84049-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:12.337245+0000 mon.a (mon.0) 1283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84049-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:12.350379+0000 mon.a (mon.0) 1284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-83977-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:13.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:12.102125+0000 mon.b (mon.1) 221 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:13.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:12.329995+0000 mon.a (mon.0) 1280 : audit [INF] from='client.? 172.21.15.162:0/2631190006' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi162-83985-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:13.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:12.330131+0000 mon.a (mon.0) 1281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84049-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:13.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:12.330225+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e177: 8 total, 8 up, 7 in 2024-02-13T00:42:13.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:12.334677+0000 mon.c (mon.2) 154 : audit [INF] from='client.? 172.21.15.162:0/2221600674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-83977-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:13.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:12.336468+0000 mon.b (mon.1) 222 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84049-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:13.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:12.337245+0000 mon.a (mon.0) 1283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84049-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:13.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:12.350379+0000 mon.a (mon.0) 1284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-83977-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:13.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:12.102125+0000 mon.b (mon.1) 221 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:13.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:12.329995+0000 mon.a (mon.0) 1280 : audit [INF] from='client.? 172.21.15.162:0/2631190006' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi162-83985-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:13.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:12.330131+0000 mon.a (mon.0) 1281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84049-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:13.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:12.330225+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e177: 8 total, 8 up, 7 in 2024-02-13T00:42:13.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:12.334677+0000 mon.c (mon.2) 154 : audit [INF] from='client.? 172.21.15.162:0/2221600674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-83977-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:13.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:12.336468+0000 mon.b (mon.1) 222 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84049-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:13.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:12.337245+0000 mon.a (mon.0) 1283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84049-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:13.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:12.350379+0000 mon.a (mon.0) 1284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-83977-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:13.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.577+0000 7f248bfff700 1 -- 172.21.15.162:0/3530947426 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v178) v1 ==== 91+0+0 (secure 0 0 0) 0x7f2498023030 con 0x7f24a40f6b70 2024-02-13T00:42:13.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.580+0000 7f24a94dc700 10 librados: watch_flush enter 2024-02-13T00:42:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.592+0000 7f24a94dc700 10 librados: watch_flush exit 2024-02-13T00:42:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.592+0000 7f24a94dc700 20 client.26517.objecter shutdown clearing up homeless session... 2024-02-13T00:42:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.592+0000 7f24a94dc700 10 client.26517.objecter successfully canceled tick 2024-02-13T00:42:13.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 -- 172.21.15.162:0/3530947426 >> v2:172.21.15.162:6800/2866534053 conn(0x7f248c062c90 msgr2=0x7f248c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:13.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 --2- 172.21.15.162:0/3530947426 >> v2:172.21.15.162:6800/2866534053 conn(0x7f248c062c90 0x7f248c065150 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 rx=0x7f24a40f8890 tx=0x7f249400b040).stop 2024-02-13T00:42:13.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 -- 172.21.15.162:0/3530947426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a40f6b70 msgr2=0x7f24a41092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:13.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 --2- 172.21.15.162:0/3530947426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a40f6b70 0x7f24a41092a0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 rx=0x7f2498009eb0 tx=0x7f249800b040).stop 2024-02-13T00:42:13.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 -- 172.21.15.162:0/3530947426 shutdown_connections 2024-02-13T00:42:13.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 --2- 172.21.15.162:0/3530947426 >> v2:172.21.15.162:6800/2866534053 conn(0x7f248c062c90 0x7f248c065150 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:13.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 --2- 172.21.15.162:0/3530947426 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a40fcc60 0x7f24a4109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:13.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 --2- 172.21.15.162:0/3530947426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a40f6b70 0x7f24a41092a0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:13.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 --2- 172.21.15.162:0/3530947426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24a40fc300 0x7f24a4109790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:13.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 -- 172.21.15.162:0/3530947426 >> 172.21.15.162:0/3530947426 conn(0x7f24a4070100 msgr2=0x7f24a406dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:13.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 -- 172.21.15.162:0/3530947426 shutdown_connections 2024-02-13T00:42:13.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 -- 172.21.15.162:0/3530947426 wait complete. 2024-02-13T00:42:13.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:13.600+0000 7f24a94dc700 1 librados: shutdown 2024-02-13T00:42:13.605 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:42:13.816 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:13.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:13.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:13.921 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:13.950 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:14.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:12.331356+0000 mgr.y (mgr.14544) 210 : cluster [DBG] pgmap v235: 300 pgs: 32 unknown, 268 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 0 B/s, 58 objects/s recovering 2024-02-13T00:42:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:13.103358+0000 mon.b (mon.1) 223 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:13.158376+0000 mon.a (mon.0) 1285 : audit [INF] from='client.? 172.21.15.162:0/3530947426' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:42:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:13.328682+0000 mon.a (mon.0) 1286 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:13.577686+0000 mon.a (mon.0) 1287 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-83977-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:13.577833+0000 mon.a (mon.0) 1288 : audit [INF] from='client.? 172.21.15.162:0/3530947426' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:42:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:13.577892+0000 mon.a (mon.0) 1289 : cluster [DBG] osdmap e178: 8 total, 8 up, 7 in 2024-02-13T00:42:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:12.331356+0000 mgr.y (mgr.14544) 210 : cluster [DBG] pgmap v235: 300 pgs: 32 unknown, 268 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 0 B/s, 58 objects/s recovering 2024-02-13T00:42:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:13.103358+0000 mon.b (mon.1) 223 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:13.158376+0000 mon.a (mon.0) 1285 : audit [INF] from='client.? 172.21.15.162:0/3530947426' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:42:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:13.328682+0000 mon.a (mon.0) 1286 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:13.577686+0000 mon.a (mon.0) 1287 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-83977-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:13.577833+0000 mon.a (mon.0) 1288 : audit [INF] from='client.? 172.21.15.162:0/3530947426' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:42:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:13.577892+0000 mon.a (mon.0) 1289 : cluster [DBG] osdmap e178: 8 total, 8 up, 7 in 2024-02-13T00:42:14.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:12.331356+0000 mgr.y (mgr.14544) 210 : cluster [DBG] pgmap v235: 300 pgs: 32 unknown, 268 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 0 B/s, 58 objects/s recovering 2024-02-13T00:42:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:13.103358+0000 mon.b (mon.1) 223 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:13.158376+0000 mon.a (mon.0) 1285 : audit [INF] from='client.? 172.21.15.162:0/3530947426' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:42:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:13.328682+0000 mon.a (mon.0) 1286 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:13.577686+0000 mon.a (mon.0) 1287 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-83977-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:13.577833+0000 mon.a (mon.0) 1288 : audit [INF] from='client.? 172.21.15.162:0/3530947426' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:42:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:13.577892+0000 mon.a (mon.0) 1289 : cluster [DBG] osdmap e178: 8 total, 8 up, 7 in 2024-02-13T00:42:14.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.855+0000 7efd21ed1700 1 -- 172.21.15.162:0/1073720188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1c0fc2b0 msgr2=0x7efd1c0fc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:14.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.855+0000 7efd21ed1700 1 --2- 172.21.15.162:0/1073720188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1c0fc2b0 0x7efd1c0fc690 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7efd0c004970 tx=0x7efd0c00adc0).stop 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.855+0000 7efd21ed1700 1 -- 172.21.15.162:0/1073720188 shutdown_connections 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.855+0000 7efd21ed1700 1 --2- 172.21.15.162:0/1073720188 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd1c0f6b20 0x7efd1c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.855+0000 7efd21ed1700 1 --2- 172.21.15.162:0/1073720188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1c0fc2b0 0x7efd1c0fc690 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.855+0000 7efd21ed1700 1 --2- 172.21.15.162:0/1073720188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd1c0fcc10 0x7efd1c0f6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.855+0000 7efd21ed1700 1 -- 172.21.15.162:0/1073720188 >> 172.21.15.162:0/1073720188 conn(0x7efd1c070080 msgr2=0x7efd1c0f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.855+0000 7efd21ed1700 1 -- 172.21.15.162:0/1073720188 shutdown_connections 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.855+0000 7efd21ed1700 1 -- 172.21.15.162:0/1073720188 wait complete. 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.855+0000 7efd21ed1700 1 librados: starting msgr at 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.855+0000 7efd21ed1700 1 librados: starting objecter 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.855+0000 7efd21ed1700 1 Processor -- start 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd21ed1700 1 -- start start 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd21ed1700 1 librados: setting wanted keys 2024-02-13T00:42:14.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd21ed1700 1 librados: calling monclient init 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd21ed1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd1c0f6b20 0x7efd1c1060d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd21ed1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1c0fc2b0 0x7efd1c1065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd21ed1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd1c0fcc10 0x7efd1c106ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd21ed1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c116610 con 0x7efd1c0f6b20 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd21ed1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c116780 con 0x7efd1c0fcc10 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd21ed1700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c116940 con 0x7efd1c0fc2b0 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd1b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd1c0f6b20 0x7efd1c1060d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd1affd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1c0fc2b0 0x7efd1c1065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd1b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd1c0f6b20 0x7efd1c1060d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:37018/0 (socket says 172.21.15.162:37018) 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd1affd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1c0fc2b0 0x7efd1c1065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:53506/0 (socket says 172.21.15.162:53506) 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd1b7fe700 1 -- 172.21.15.162:0/3866767111 learned_addr learned my addr 172.21.15.162:0/3866767111 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd1bfff700 1 --2- 172.21.15.162:0/3866767111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd1c0fcc10 0x7efd1c106ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd1b7fe700 1 -- 172.21.15.162:0/3866767111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1c0fc2b0 msgr2=0x7efd1c1065c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd1b7fe700 1 --2- 172.21.15.162:0/3866767111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1c0fc2b0 0x7efd1c1065c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd1b7fe700 1 -- 172.21.15.162:0/3866767111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd1c0fcc10 msgr2=0x7efd1c106ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd1b7fe700 1 --2- 172.21.15.162:0/3866767111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd1c0fcc10 0x7efd1c106ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd1b7fe700 1 -- 172.21.15.162:0/3866767111 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd10005040 con 0x7efd1c0f6b20 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd1b7fe700 1 --2- 172.21.15.162:0/3866767111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd1c0f6b20 0x7efd1c1060d0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 rx=0x7efd0c015fa0 tx=0x7efd0c00adc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd18ff9700 10 client.?.objecter ms_handle_connect 0x7efd1c0f6b20 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd18ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.856+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0c00f040 con 0x7efd1c0f6b20 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.857+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0c005040 con 0x7efd1c0f6b20 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.857+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd0c00a7b0 con 0x7efd1c0f6b20 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.857+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0c01dcb0 con 0x7efd1c0f6b20 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.857+0000 7efd21ed1700 10 client.26535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.857+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1c116e50 con 0x7efd1c0f6b20 2024-02-13T00:42:14.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.857+0000 7efd21ed1700 1 librados: init done 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.857+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7efd1c05fb80 con 0x7efd1c0f6b20 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.858+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7efd0c00a2e0 con 0x7efd1c0f6b20 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd18ff9700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6800/2866534053 conn(0x7efd04062c90 0x7efd04065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd1affd700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6800/2866534053 conn(0x7efd04062c90 0x7efd04065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(179..179 src has 52..179) v4 ==== 10212+0+0 (secure 0 0 0) 0x7efd0c07e560 con 0x7efd1c0f6b20 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 osd_map(179..179 src has 52..179) v4 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd18ff9700 3 client.26535.objecter handle_osd_map got epochs [179,179] > 0 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd18ff9700 3 client.26535.objecter handle_osd_map decoding full epoch 179 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd18ff9700 20 client.26535.objecter dump_active .. 0 homeless 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v179) v1 ==== 54+0+16 (secure 0 0 0) 0x7efd0c085070 con 0x7efd1c0f6b20 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd18ff9700 10 client.26535.objecter ms_handle_connect 0x7efd04062c90 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd1affd700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6800/2866534053 conn(0x7efd04062c90 0x7efd04065150 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 rx=0x7efd1c107870 tx=0x7efd1000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd21ed1700 10 client.26535.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 1 2024-02-13T00:42:14.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6801/2103743338 conn(0x7efd1c0fd560 0x7efd1c0fd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c104f40 osd=0 3 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c104f40 osd=0 3 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c104f40 osd=0 3 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 1 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c104f40 osd=0 3 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 0 1 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c104f40 osd=0 4 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd21ed1700 10 client.26535.objecter _send_command 1 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd1c0fde30 con 0x7efd1c0fd560 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.859+0000 7efd1bfff700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6801/2103743338 conn(0x7efd1c0fd560 0x7efd1c0fd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.860+0000 7efd18ff9700 10 client.26535.objecter ms_handle_connect 0x7efd1c0fd560 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.860+0000 7efd1bfff700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6801/2103743338 conn(0x7efd1c0fd560 0x7efd1c0fd940 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.860+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd1c0fde30 con 0x7efd1c0fd560 2024-02-13T00:42:14.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.860+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:42:14.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.860+0000 7efd18ff9700 10 client.26535.objecter _finish_command 1 = system:0 2024-02-13T00:42:14.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.860+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c104f40 osd=0 4 2024-02-13T00:42:14.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.860+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 0 1 2024-02-13T00:42:14.875 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:14.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.877+0000 7efd21ed1700 10 client.26535.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:14.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.877+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 2 2024-02-13T00:42:14.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.877+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c104f40 osd=0 3 2024-02-13T00:42:14.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.877+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c104f40 osd=0 3 2024-02-13T00:42:14.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.877+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c104f40 osd=0 3 2024-02-13T00:42:14.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.877+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 2 2024-02-13T00:42:14.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.877+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c104f40 osd=0 3 2024-02-13T00:42:14.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.877+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 0 2 2024-02-13T00:42:14.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.877+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c104f40 osd=0 4 2024-02-13T00:42:14.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.877+0000 7efd21ed1700 10 client.26535.objecter _send_command 2 2024-02-13T00:42:14.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.877+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd1c0fdfd0 con 0x7efd1c0fd560 2024-02-13T00:42:14.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.879+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd1c0fdfd0 con 0x7efd1c0fd560 2024-02-13T00:42:14.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.879+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:14.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.879+0000 7efd18ff9700 10 client.26535.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:14.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.879+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c104f40 osd=0 4 2024-02-13T00:42:14.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.879+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 0 2 2024-02-13T00:42:14.883 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.879+0000 7efd21ed1700 10 client.26535.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.879+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 3 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.879+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6805/150599780 conn(0x7efd1c028590 0x7efd1c11b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.880+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c0fbb60 osd=1 3 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.880+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c0fbb60 osd=1 3 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.880+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c0fbb60 osd=1 3 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.880+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 3 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.880+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c0fbb60 osd=1 3 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.880+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 1 3 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.880+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c0fbb60 osd=1 4 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.880+0000 7efd21ed1700 10 client.26535.objecter _send_command 3 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.880+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd1c0fbcd0 con 0x7efd1c028590 2024-02-13T00:42:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.880+0000 7efd1b7fe700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6805/150599780 conn(0x7efd1c028590 0x7efd1c11b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:14.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.880+0000 7efd18ff9700 10 client.26535.objecter ms_handle_connect 0x7efd1c028590 2024-02-13T00:42:14.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.880+0000 7efd1b7fe700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6805/150599780 conn(0x7efd1c028590 0x7efd1c11b060 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:14.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.881+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd1c0fbcd0 con 0x7efd1c028590 2024-02-13T00:42:14.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.881+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:42:14.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.881+0000 7efd18ff9700 10 client.26535.objecter _finish_command 3 = system:0 2024-02-13T00:42:14.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.881+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c0fbb60 osd=1 4 2024-02-13T00:42:14.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.881+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 1 3 2024-02-13T00:42:14.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.906+0000 7efd21ed1700 10 client.26535.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:14.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.906+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 4 2024-02-13T00:42:14.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.906+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c0fbb60 osd=1 3 2024-02-13T00:42:14.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.906+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c0fbb60 osd=1 3 2024-02-13T00:42:14.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.906+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c0fbb60 osd=1 3 2024-02-13T00:42:14.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.906+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 4 2024-02-13T00:42:14.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.906+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c0fbb60 osd=1 3 2024-02-13T00:42:14.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.906+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 1 4 2024-02-13T00:42:14.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.906+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c0fbb60 osd=1 4 2024-02-13T00:42:14.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.906+0000 7efd21ed1700 10 client.26535.objecter _send_command 4 2024-02-13T00:42:14.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.906+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd1c05a0e0 con 0x7efd1c028590 2024-02-13T00:42:14.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd1c05a0e0 con 0x7efd1c028590 2024-02-13T00:42:14.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:14.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd18ff9700 10 client.26535.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:14.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c0fbb60 osd=1 4 2024-02-13T00:42:14.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 1 4 2024-02-13T00:42:14.912 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:42:14.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd21ed1700 10 client.26535.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:14.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 5 2024-02-13T00:42:14.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6809/3689719376 conn(0x7efd1c059570 0x7efd1c11e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:14.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c059400 osd=2 3 2024-02-13T00:42:14.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c059400 osd=2 3 2024-02-13T00:42:14.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c059400 osd=2 3 2024-02-13T00:42:14.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 5 2024-02-13T00:42:14.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c059400 osd=2 3 2024-02-13T00:42:14.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 2 5 2024-02-13T00:42:14.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c059400 osd=2 4 2024-02-13T00:42:14.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd21ed1700 10 client.26535.objecter _send_command 5 2024-02-13T00:42:14.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.908+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd1c028200 con 0x7efd1c059570 2024-02-13T00:42:14.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.909+0000 7efd1affd700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6809/3689719376 conn(0x7efd1c059570 0x7efd1c11e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:14.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.909+0000 7efd18ff9700 10 client.26535.objecter ms_handle_connect 0x7efd1c059570 2024-02-13T00:42:14.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.909+0000 7efd1affd700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6809/3689719376 conn(0x7efd1c059570 0x7efd1c11e6e0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:14.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.910+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd1c028200 con 0x7efd1c059570 2024-02-13T00:42:14.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.910+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:42:14.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.910+0000 7efd18ff9700 10 client.26535.objecter _finish_command 5 = system:0 2024-02-13T00:42:14.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.910+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c059400 osd=2 4 2024-02-13T00:42:14.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.910+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 2 5 2024-02-13T00:42:14.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.924+0000 7efd21ed1700 10 client.26535.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:14.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.924+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 6 2024-02-13T00:42:14.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.924+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c059400 osd=2 3 2024-02-13T00:42:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.924+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c059400 osd=2 3 2024-02-13T00:42:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.924+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c059400 osd=2 3 2024-02-13T00:42:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.924+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 6 2024-02-13T00:42:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.924+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c059400 osd=2 3 2024-02-13T00:42:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.924+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 2 6 2024-02-13T00:42:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.924+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c059400 osd=2 4 2024-02-13T00:42:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.924+0000 7efd21ed1700 10 client.26535.objecter _send_command 6 2024-02-13T00:42:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.924+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd1c027650 con 0x7efd1c059570 2024-02-13T00:42:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd1c027650 con 0x7efd1c059570 2024-02-13T00:42:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd18ff9700 10 client.26535.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c059400 osd=2 4 2024-02-13T00:42:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 2 6 2024-02-13T00:42:14.929 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:42:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd21ed1700 10 client.26535.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 7 2024-02-13T00:42:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6813/604971011 conn(0x7efd1c11c230 0x7efd1c121d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c027a60 osd=3 3 2024-02-13T00:42:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c027a60 osd=3 3 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c027a60 osd=3 3 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 7 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c027a60 osd=3 3 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 3 7 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c027a60 osd=3 4 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd21ed1700 10 client.26535.objecter _send_command 7 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd1c124ac0 con 0x7efd1c11c230 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd1bfff700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6813/604971011 conn(0x7efd1c11c230 0x7efd1c121d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd18ff9700 10 client.26535.objecter ms_handle_connect 0x7efd1c11c230 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.926+0000 7efd1bfff700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6813/604971011 conn(0x7efd1c11c230 0x7efd1c121d60 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.927+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd1c124ac0 con 0x7efd1c11c230 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.927+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:42:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.927+0000 7efd18ff9700 10 client.26535.objecter _finish_command 7 = system:0 2024-02-13T00:42:14.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.927+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c027a60 osd=3 4 2024-02-13T00:42:14.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.927+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 3 7 2024-02-13T00:42:14.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.942+0000 7efd21ed1700 10 client.26535.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:14.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.942+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 8 2024-02-13T00:42:14.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.942+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c027a60 osd=3 3 2024-02-13T00:42:14.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.942+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c027a60 osd=3 3 2024-02-13T00:42:14.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.942+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c027a60 osd=3 3 2024-02-13T00:42:14.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.942+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 8 2024-02-13T00:42:14.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.942+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c027a60 osd=3 3 2024-02-13T00:42:14.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.942+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 3 8 2024-02-13T00:42:14.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.942+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c027a60 osd=3 4 2024-02-13T00:42:14.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.942+0000 7efd21ed1700 10 client.26535.objecter _send_command 8 2024-02-13T00:42:14.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.942+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd1c11b790 con 0x7efd1c11c230 2024-02-13T00:42:14.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd1c11b790 con 0x7efd1c11c230 2024-02-13T00:42:14.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:14.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd18ff9700 10 client.26535.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:14.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c027a60 osd=3 4 2024-02-13T00:42:14.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 3 8 2024-02-13T00:42:14.946 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:42:14.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd21ed1700 10 client.26535.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:14.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 9 2024-02-13T00:42:14.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6800/3344263388 conn(0x7efd1c11f8b0 0x7efd1c123470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:14.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c11b790 osd=4 3 2024-02-13T00:42:14.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c11b790 osd=4 3 2024-02-13T00:42:14.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c11b790 osd=4 3 2024-02-13T00:42:14.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 9 2024-02-13T00:42:14.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c11b790 osd=4 3 2024-02-13T00:42:14.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 4 9 2024-02-13T00:42:14.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c11b790 osd=4 4 2024-02-13T00:42:14.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd21ed1700 10 client.26535.objecter _send_command 9 2024-02-13T00:42:14.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.943+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd1c123960 con 0x7efd1c11f8b0 2024-02-13T00:42:14.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.944+0000 7efd1b7fe700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6800/3344263388 conn(0x7efd1c11f8b0 0x7efd1c123470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:14.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.944+0000 7efd18ff9700 10 client.26535.objecter ms_handle_connect 0x7efd1c11f8b0 2024-02-13T00:42:14.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.944+0000 7efd1b7fe700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6800/3344263388 conn(0x7efd1c11f8b0 0x7efd1c123470 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:14.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.945+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd1c123960 con 0x7efd1c11f8b0 2024-02-13T00:42:14.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.945+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:42:14.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.945+0000 7efd18ff9700 10 client.26535.objecter _finish_command 9 = system:0 2024-02-13T00:42:14.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.945+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c11b790 osd=4 4 2024-02-13T00:42:14.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.945+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 4 9 2024-02-13T00:42:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.960+0000 7efd21ed1700 10 client.26535.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.960+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 10 2024-02-13T00:42:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.960+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c11b790 osd=4 3 2024-02-13T00:42:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.960+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c11b790 osd=4 3 2024-02-13T00:42:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.960+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c11b790 osd=4 3 2024-02-13T00:42:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.960+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 10 2024-02-13T00:42:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.960+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c11b790 osd=4 3 2024-02-13T00:42:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.960+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 4 10 2024-02-13T00:42:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.960+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c11b790 osd=4 4 2024-02-13T00:42:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.960+0000 7efd21ed1700 10 client.26535.objecter _send_command 10 2024-02-13T00:42:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.960+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd1c123960 con 0x7efd1c11f8b0 2024-02-13T00:42:14.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd1c123960 con 0x7efd1c11f8b0 2024-02-13T00:42:14.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:14.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd18ff9700 10 client.26535.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:14.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c11b790 osd=4 4 2024-02-13T00:42:14.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 4 10 2024-02-13T00:42:14.965 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:42:14.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd21ed1700 10 client.26535.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:14.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 11 2024-02-13T00:42:14.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6804/2212296748 conn(0x7efd1c1240b0 0x7efd1c124510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:14.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c123960 osd=5 3 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c123960 osd=5 3 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c123960 osd=5 3 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 11 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c123960 osd=5 3 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 5 11 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c123960 osd=5 4 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd21ed1700 10 client.26535.objecter _send_command 11 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd1c12b590 con 0x7efd1c1240b0 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd1affd700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6804/2212296748 conn(0x7efd1c1240b0 0x7efd1c124510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd18ff9700 10 client.26535.objecter ms_handle_connect 0x7efd1c1240b0 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.962+0000 7efd1affd700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6804/2212296748 conn(0x7efd1c1240b0 0x7efd1c124510 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.963+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd1c12b590 con 0x7efd1c1240b0 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.963+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.963+0000 7efd18ff9700 10 client.26535.objecter _finish_command 11 = system:0 2024-02-13T00:42:14.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.963+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c123960 osd=5 4 2024-02-13T00:42:14.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.963+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 5 11 2024-02-13T00:42:14.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.978+0000 7efd21ed1700 10 client.26535.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:14.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.978+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 12 2024-02-13T00:42:14.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.978+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c123960 osd=5 3 2024-02-13T00:42:14.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.978+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c123960 osd=5 3 2024-02-13T00:42:14.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.978+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c123960 osd=5 3 2024-02-13T00:42:14.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.978+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 12 2024-02-13T00:42:14.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.978+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c123960 osd=5 3 2024-02-13T00:42:14.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.978+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 5 12 2024-02-13T00:42:14.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.978+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c123960 osd=5 4 2024-02-13T00:42:14.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.978+0000 7efd21ed1700 10 client.26535.objecter _send_command 12 2024-02-13T00:42:14.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.978+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd1c12b590 con 0x7efd1c1240b0 2024-02-13T00:42:14.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd1c12b590 con 0x7efd1c1240b0 2024-02-13T00:42:14.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:14.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd18ff9700 10 client.26535.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:14.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c123960 osd=5 4 2024-02-13T00:42:14.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 5 12 2024-02-13T00:42:14.983 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:42:14.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd21ed1700 10 client.26535.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:14.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 13 2024-02-13T00:42:14.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6808/1506640341 conn(0x7efd1c1292f0 0x7efd1c129750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:14.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c12b590 osd=6 3 2024-02-13T00:42:14.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c12b590 osd=6 3 2024-02-13T00:42:14.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c12b590 osd=6 3 2024-02-13T00:42:14.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 13 2024-02-13T00:42:14.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c12b590 osd=6 3 2024-02-13T00:42:14.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 6 13 2024-02-13T00:42:14.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c12b590 osd=6 4 2024-02-13T00:42:14.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd21ed1700 10 client.26535.objecter _send_command 13 2024-02-13T00:42:14.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd1c12e9c0 con 0x7efd1c1292f0 2024-02-13T00:42:14.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd1bfff700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6808/1506640341 conn(0x7efd1c1292f0 0x7efd1c129750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:14.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd18ff9700 10 client.26535.objecter ms_handle_connect 0x7efd1c1292f0 2024-02-13T00:42:14.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.980+0000 7efd1bfff700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6808/1506640341 conn(0x7efd1c1292f0 0x7efd1c129750 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:14.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.981+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd1c12e9c0 con 0x7efd1c1292f0 2024-02-13T00:42:14.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.981+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:42:14.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.981+0000 7efd18ff9700 10 client.26535.objecter _finish_command 13 = system:0 2024-02-13T00:42:14.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.981+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c12b590 osd=6 4 2024-02-13T00:42:14.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.981+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 6 13 2024-02-13T00:42:14.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.995+0000 7efd21ed1700 10 client.26535.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:14.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.995+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 14 2024-02-13T00:42:14.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.995+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c12b590 osd=6 3 2024-02-13T00:42:14.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.995+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c12b590 osd=6 3 2024-02-13T00:42:14.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.995+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c12b590 osd=6 3 2024-02-13T00:42:14.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.995+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 14 2024-02-13T00:42:14.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.995+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c12b590 osd=6 3 2024-02-13T00:42:14.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.995+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 6 14 2024-02-13T00:42:14.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.995+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c12b590 osd=6 4 2024-02-13T00:42:14.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.995+0000 7efd21ed1700 10 client.26535.objecter _send_command 14 2024-02-13T00:42:14.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.995+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd1c129d30 con 0x7efd1c1292f0 2024-02-13T00:42:15.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd1c129d30 con 0x7efd1c1292f0 2024-02-13T00:42:15.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:15.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd18ff9700 10 client.26535.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:15.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c12b590 osd=6 4 2024-02-13T00:42:15.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 6 14 2024-02-13T00:42:15.001 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:42:15.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd21ed1700 10 client.26535.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:15.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 15 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6812/1964192772 conn(0x7efd1c12a3f0 0x7efd1c12a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c129d30 osd=7 3 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c129d30 osd=7 3 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c129d30 osd=7 3 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 15 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c129d30 osd=7 3 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 7 15 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c129d30 osd=7 4 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd21ed1700 10 client.26535.objecter _send_command 15 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.997+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd1c12ae80 con 0x7efd1c12a3f0 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.998+0000 7efd1b7fe700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6812/1964192772 conn(0x7efd1c12a3f0 0x7efd1c12a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.998+0000 7efd18ff9700 10 client.26535.objecter ms_handle_connect 0x7efd1c12a3f0 2024-02-13T00:42:15.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.998+0000 7efd1b7fe700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6812/1964192772 conn(0x7efd1c12a3f0 0x7efd1c12a810 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:15.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.999+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd1c12ae80 con 0x7efd1c12a3f0 2024-02-13T00:42:15.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.999+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:42:15.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.999+0000 7efd18ff9700 10 client.26535.objecter _finish_command 15 = system:0 2024-02-13T00:42:15.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.999+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c129d30 osd=7 4 2024-02-13T00:42:15.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:14.999+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 7 15 2024-02-13T00:42:15.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.014+0000 7efd21ed1700 10 client.26535.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:15.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.014+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign -1 16 2024-02-13T00:42:15.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.014+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c129d30 osd=7 3 2024-02-13T00:42:15.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.014+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c129d30 osd=7 3 2024-02-13T00:42:15.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.014+0000 7efd21ed1700 20 client.26535.objecter _get_session s=0x7efd1c129d30 osd=7 3 2024-02-13T00:42:15.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.014+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_remove -1 16 2024-02-13T00:42:15.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.014+0000 7efd21ed1700 20 client.26535.objecter get_session s=0x7efd1c129d30 osd=7 3 2024-02-13T00:42:15.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.014+0000 7efd21ed1700 15 client.26535.objecter _session_command_op_assign 7 16 2024-02-13T00:42:15.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.014+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c129d30 osd=7 4 2024-02-13T00:42:15.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.014+0000 7efd21ed1700 10 client.26535.objecter _send_command 16 2024-02-13T00:42:15.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.014+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd1c12ae80 con 0x7efd1c12a3f0 2024-02-13T00:42:15.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.015+0000 7efd18ff9700 1 -- 172.21.15.162:0/3866767111 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd1c12ae80 con 0x7efd1c12a3f0 2024-02-13T00:42:15.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.015+0000 7efd18ff9700 10 client.26535.objecter ms_dispatch 0x7efd1c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:15.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.015+0000 7efd18ff9700 10 client.26535.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:15.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.015+0000 7efd18ff9700 20 client.26535.objecter put_session s=0x7efd1c129d30 osd=7 4 2024-02-13T00:42:15.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.015+0000 7efd18ff9700 15 client.26535.objecter _session_command_op_remove 7 16 2024-02-13T00:42:15.019 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:42:15.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 10 librados: watch_flush enter 2024-02-13T00:42:15.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 10 librados: watch_flush exit 2024-02-13T00:42:15.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 10 client.26535.objecter close_session for osd.0 2024-02-13T00:42:15.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6801/2103743338 conn(0x7efd1c0fd560 msgr2=0x7efd1c0fd940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:15.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6801/2103743338 conn(0x7efd1c0fd560 0x7efd1c0fd940 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:15.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c104f40 osd=0 1 2024-02-13T00:42:15.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 10 client.26535.objecter close_session for osd.1 2024-02-13T00:42:15.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6805/150599780 conn(0x7efd1c028590 msgr2=0x7efd1c11b060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:15.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6805/150599780 conn(0x7efd1c028590 0x7efd1c11b060 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:15.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c0fbb60 osd=1 1 2024-02-13T00:42:15.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 10 client.26535.objecter close_session for osd.2 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6809/3689719376 conn(0x7efd1c059570 msgr2=0x7efd1c11e6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6809/3689719376 conn(0x7efd1c059570 0x7efd1c11e6e0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c059400 osd=2 1 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 10 client.26535.objecter close_session for osd.3 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6813/604971011 conn(0x7efd1c11c230 msgr2=0x7efd1c121d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6813/604971011 conn(0x7efd1c11c230 0x7efd1c121d60 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd1bfff700 1 -- 172.21.15.162:0/3866767111 reap_dead start 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c027a60 osd=3 1 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 10 client.26535.objecter close_session for osd.4 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6800/3344263388 conn(0x7efd1c11f8b0 msgr2=0x7efd1c123470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6800/3344263388 conn(0x7efd1c11f8b0 0x7efd1c123470 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c11b790 osd=4 1 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 10 client.26535.objecter close_session for osd.5 2024-02-13T00:42:15.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6804/2212296748 conn(0x7efd1c1240b0 msgr2=0x7efd1c124510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6804/2212296748 conn(0x7efd1c1240b0 0x7efd1c124510 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c123960 osd=5 1 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 10 client.26535.objecter close_session for osd.6 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6808/1506640341 conn(0x7efd1c1292f0 msgr2=0x7efd1c129750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6808/1506640341 conn(0x7efd1c1292f0 0x7efd1c129750 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c12b590 osd=6 1 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 10 client.26535.objecter close_session for osd.7 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6812/1964192772 conn(0x7efd1c12a3f0 msgr2=0x7efd1c12a810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.181:6812/1964192772 conn(0x7efd1c12a3f0 0x7efd1c12a810 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 20 client.26535.objecter put_session s=0x7efd1c129d30 osd=7 1 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 20 client.26535.objecter shutdown clearing up homeless session... 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 10 client.26535.objecter successfully canceled tick 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6800/2866534053 conn(0x7efd04062c90 msgr2=0x7efd04065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> v2:172.21.15.162:6800/2866534053 conn(0x7efd04062c90 0x7efd04065150 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 rx=0x7efd1c107870 tx=0x7efd1000b040).stop 2024-02-13T00:42:15.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd1bfff700 1 -- 172.21.15.162:0/3866767111 reap_dead start 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd1c0f6b20 msgr2=0x7efd1c1060d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 --2- 172.21.15.162:0/3866767111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd1c0f6b20 0x7efd1c1060d0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 rx=0x7efd0c015fa0 tx=0x7efd0c00adc0).stop 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd1bfff700 1 -- 172.21.15.162:0/3866767111 reap_dead start 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 shutdown_connections 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 >> 172.21.15.162:0/3866767111 conn(0x7efd1c070080 msgr2=0x7efd1c0f9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 shutdown_connections 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 -- 172.21.15.162:0/3866767111 wait complete. 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:15.016+0000 7efd21ed1700 1 librados: shutdown 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:42:15.024 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:42:15.025 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:42:15.025 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:42:15.025 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:42:15.060 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:42:15.060 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:42:15.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:14.104543+0000 mon.b (mon.1) 224 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:15.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:14.339805+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84049-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84049-24"}]': finished 2024-02-13T00:42:15.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:14.339958+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e179: 8 total, 8 up, 7 in 2024-02-13T00:42:15.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:14.357381+0000 mon.c (mon.2) 155 : audit [INF] from='client.? 172.21.15.162:0/752574467' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-83985-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:15.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:14.603759+0000 mon.a (mon.0) 1292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-83985-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:15.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:14.855346+0000 mon.a (mon.0) 1293 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:15.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:14.859459+0000 mon.a (mon.0) 1294 : audit [DBG] from='client.? 172.21.15.162:0/3866767111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:15.169 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:14.104543+0000 mon.b (mon.1) 224 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:15.169 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:14.339805+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84049-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84049-24"}]': finished 2024-02-13T00:42:15.169 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:14.339958+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e179: 8 total, 8 up, 7 in 2024-02-13T00:42:15.169 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:14.357381+0000 mon.c (mon.2) 155 : audit [INF] from='client.? 172.21.15.162:0/752574467' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-83985-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:15.170 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:14.603759+0000 mon.a (mon.0) 1292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-83985-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:15.170 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:14.855346+0000 mon.a (mon.0) 1293 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:15.170 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:14.859459+0000 mon.a (mon.0) 1294 : audit [DBG] from='client.? 172.21.15.162:0/3866767111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:14.104543+0000 mon.b (mon.1) 224 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:14.339805+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84049-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84049-24"}]': finished 2024-02-13T00:42:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:14.339958+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e179: 8 total, 8 up, 7 in 2024-02-13T00:42:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:14.357381+0000 mon.c (mon.2) 155 : audit [INF] from='client.? 172.21.15.162:0/752574467' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-83985-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:14.603759+0000 mon.a (mon.0) 1292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-83985-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:14.855346+0000 mon.a (mon.0) 1293 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:14.859459+0000 mon.a (mon.0) 1294 : audit [DBG] from='client.? 172.21.15.162:0/3866767111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:15.328 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:15.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:15.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:15.432 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:15.464 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:15.483 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:15.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:15.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:15.587 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:15.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:42:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:42:15.616 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:15.961 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-13T00:42:15.964 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:15.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:16.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:16.096 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:16.164 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:14.331835+0000 mgr.y (mgr.14544) 211 : cluster [DBG] pgmap v238: 300 pgs: 32 unknown, 268 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-13T00:42:16.164 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:15.106188+0000 mon.b (mon.1) 225 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:16.164 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:15.342041+0000 mon.a (mon.0) 1295 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-83985-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:16.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:15.342245+0000 mon.a (mon.0) 1296 : cluster [DBG] osdmap e180: 8 total, 8 up, 7 in 2024-02-13T00:42:16.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:15.354161+0000 mon.b (mon.1) 226 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:16.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:15.355268+0000 mon.c (mon.2) 156 : audit [INF] from='client.? 172.21.15.162:0/1712454008' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-83977-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:16.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:15.356010+0000 mon.a (mon.0) 1297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-83977-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:16.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:15.357526+0000 mon.a (mon.0) 1298 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:16.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:15.457908+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.17 scrub starts 2024-02-13T00:42:16.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:15.458746+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.17 scrub ok 2024-02-13T00:42:16.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:14.331835+0000 mgr.y (mgr.14544) 211 : cluster [DBG] pgmap v238: 300 pgs: 32 unknown, 268 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-13T00:42:16.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:15.106188+0000 mon.b (mon.1) 225 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:16.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:15.342041+0000 mon.a (mon.0) 1295 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-83985-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:16.165 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:15.342245+0000 mon.a (mon.0) 1296 : cluster [DBG] osdmap e180: 8 total, 8 up, 7 in 2024-02-13T00:42:16.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:15.354161+0000 mon.b (mon.1) 226 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:16.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:15.355268+0000 mon.c (mon.2) 156 : audit [INF] from='client.? 172.21.15.162:0/1712454008' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-83977-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:16.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:15.356010+0000 mon.a (mon.0) 1297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-83977-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:16.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:15.357526+0000 mon.a (mon.0) 1298 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:16.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:15.457908+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.17 scrub starts 2024-02-13T00:42:16.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:15.458746+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.17 scrub ok 2024-02-13T00:42:16.219 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:16.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:16.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:16.226 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:42:16.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:14.331835+0000 mgr.y (mgr.14544) 211 : cluster [DBG] pgmap v238: 300 pgs: 32 unknown, 268 active+clean; 15 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-02-13T00:42:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:15.106188+0000 mon.b (mon.1) 225 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:15.342041+0000 mon.a (mon.0) 1295 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-83985-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:15.342245+0000 mon.a (mon.0) 1296 : cluster [DBG] osdmap e180: 8 total, 8 up, 7 in 2024-02-13T00:42:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:15.354161+0000 mon.b (mon.1) 226 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:15.355268+0000 mon.c (mon.2) 156 : audit [INF] from='client.? 172.21.15.162:0/1712454008' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-83977-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:15.356010+0000 mon.a (mon.0) 1297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-83977-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:15.357526+0000 mon.a (mon.0) 1298 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:15.457908+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.17 scrub starts 2024-02-13T00:42:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:15.458746+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.17 scrub ok 2024-02-13T00:42:16.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:16.355 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:16.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:42:16.432 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:42:16] "GET /metrics HTTP/1.1" 200 193841 "" "Prometheus/2.33.4" 2024-02-13T00:42:17.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:15.395604+0000 mgr.y (mgr.14544) 212 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:16.107161+0000 mon.b (mon.1) 227 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:16.344396+0000 mon.a (mon.0) 1299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-83977-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:16.344475+0000 mon.a (mon.0) 1300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:16.344588+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e181: 8 total, 8 up, 7 in 2024-02-13T00:42:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:16.357981+0000 mon.b (mon.1) 228 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:16.358843+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:17.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:15.395604+0000 mgr.y (mgr.14544) 212 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:17.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:16.107161+0000 mon.b (mon.1) 227 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:17.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:16.344396+0000 mon.a (mon.0) 1299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-83977-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:17.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:16.344475+0000 mon.a (mon.0) 1300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:17.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:16.344588+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e181: 8 total, 8 up, 7 in 2024-02-13T00:42:17.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:16.357981+0000 mon.b (mon.1) 228 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:17.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:16.358843+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:15.395604+0000 mgr.y (mgr.14544) 212 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:16.107161+0000 mon.b (mon.1) 227 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:16.344396+0000 mon.a (mon.0) 1299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-83977-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:16.344475+0000 mon.a (mon.0) 1300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:17.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:16.344588+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e181: 8 total, 8 up, 7 in 2024-02-13T00:42:17.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:16.357981+0000 mon.b (mon.1) 228 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:17.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:16.358843+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:17.416 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:17.418 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:17.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.429+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3097226591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a640fcb30 msgr2=0x7f1a640fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.429+0000 7f1a6baf8700 1 --2- 172.21.15.162:0/3097226591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a640fcb30 0x7f1a640fcf10 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 rx=0x7f1a600047b0 tx=0x7f1a60009cb0).stop 2024-02-13T00:42:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.429+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3097226591 shutdown_connections 2024-02-13T00:42:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.429+0000 7f1a6baf8700 1 --2- 172.21.15.162:0/3097226591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a640f73a0 0x7f1a641039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.429+0000 7f1a6baf8700 1 --2- 172.21.15.162:0/3097226591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a640fcb30 0x7f1a640fcf10 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.429+0000 7f1a6baf8700 1 --2- 172.21.15.162:0/3097226591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a640fd490 0x7f1a640f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.429+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3097226591 >> 172.21.15.162:0/3097226591 conn(0x7f1a6406eb90 msgr2=0x7f1a640f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.429+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3097226591 shutdown_connections 2024-02-13T00:42:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.429+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3097226591 wait complete. 2024-02-13T00:42:17.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6baf8700 1 librados: starting msgr at 2024-02-13T00:42:17.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6baf8700 1 librados: starting objecter 2024-02-13T00:42:17.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6baf8700 1 Processor -- start 2024-02-13T00:42:17.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6baf8700 1 -- start start 2024-02-13T00:42:17.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6baf8700 1 librados: setting wanted keys 2024-02-13T00:42:17.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6baf8700 1 librados: calling monclient init 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6baf8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a640f73a0 0x7f1a641068f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6baf8700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a640fcb30 0x7f1a64106de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6baf8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a640fd490 0x7f1a6410bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6baf8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6410c260 con 0x7f1a640f73a0 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6baf8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6410c3d0 con 0x7f1a640fd490 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6baf8700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6410c590 con 0x7f1a640fcb30 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a69093700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a640fcb30 0x7f1a64106de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a69093700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a640fcb30 0x7f1a64106de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:53560/0 (socket says 172.21.15.162:53560) 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a69093700 1 -- 172.21.15.162:0/3538035531 learned_addr learned my addr 172.21.15.162:0/3538035531 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a69894700 1 --2- 172.21.15.162:0/3538035531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a640f73a0 0x7f1a641068f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.430+0000 7f1a6a095700 1 --2- 172.21.15.162:0/3538035531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a640fd490 0x7f1a6410bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a69894700 1 -- 172.21.15.162:0/3538035531 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a640fcb30 msgr2=0x7f1a64106de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:17.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a69894700 1 --2- 172.21.15.162:0/3538035531 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a640fcb30 0x7f1a64106de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a69894700 1 -- 172.21.15.162:0/3538035531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a640fd490 msgr2=0x7f1a6410bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a69894700 1 --2- 172.21.15.162:0/3538035531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a640fd490 0x7f1a6410bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a69894700 1 -- 172.21.15.162:0/3538035531 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a60005040 con 0x7f1a640f73a0 2024-02-13T00:42:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a6a095700 1 --2- 172.21.15.162:0/3538035531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a640fd490 0x7f1a6410bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:42:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a69894700 1 --2- 172.21.15.162:0/3538035531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a640f73a0 0x7f1a641068f0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 rx=0x7f1a60004650 tx=0x7f1a60007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a5affd700 10 client.?.objecter ms_handle_connect 0x7f1a640f73a0 2024-02-13T00:42:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a5affd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a5affd700 1 -- 172.21.15.162:0/3538035531 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a60018070 con 0x7f1a640f73a0 2024-02-13T00:42:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a5affd700 1 -- 172.21.15.162:0/3538035531 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a600092e0 con 0x7f1a640f73a0 2024-02-13T00:42:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3538035531 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6410c810 con 0x7f1a640f73a0 2024-02-13T00:42:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a5affd700 1 -- 172.21.15.162:0/3538035531 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6001dbd0 con 0x7f1a640f73a0 2024-02-13T00:42:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a6baf8700 10 client.26550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3538035531 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a64112e10 con 0x7f1a640f73a0 2024-02-13T00:42:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.431+0000 7f1a6baf8700 1 librados: init done 2024-02-13T00:42:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.433+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3538035531 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a640475e0 con 0x7f1a640f73a0 2024-02-13T00:42:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.435+0000 7f1a5affd700 1 -- 172.21.15.162:0/3538035531 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1a6001d420 con 0x7f1a640f73a0 2024-02-13T00:42:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.436+0000 7f1a5affd700 1 --2- 172.21.15.162:0/3538035531 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1a50062ce0 0x7f1a500651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.436+0000 7f1a5affd700 1 -- 172.21.15.162:0/3538035531 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(182..182 src has 52..182) v4 ==== 10167+0+0 (secure 0 0 0) 0x7f1a6007f370 con 0x7f1a640f73a0 2024-02-13T00:42:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.436+0000 7f1a5affd700 10 client.26550.objecter ms_dispatch 0x7f1a64001550 osd_map(182..182 src has 52..182) v4 2024-02-13T00:42:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.436+0000 7f1a5affd700 3 client.26550.objecter handle_osd_map got epochs [182,182] > 0 2024-02-13T00:42:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.436+0000 7f1a5affd700 3 client.26550.objecter handle_osd_map decoding full epoch 182 2024-02-13T00:42:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.436+0000 7f1a5affd700 20 client.26550.objecter dump_active .. 0 homeless 2024-02-13T00:42:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.436+0000 7f1a69093700 1 --2- 172.21.15.162:0/3538035531 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1a50062ce0 0x7f1a500651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.437+0000 7f1a69093700 1 --2- 172.21.15.162:0/3538035531 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1a50062ce0 0x7f1a500651a0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 rx=0x7f1a64107a00 tx=0x7f1a5400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.437+0000 7f1a5affd700 10 client.26550.objecter ms_handle_connect 0x7f1a50062ce0 2024-02-13T00:42:17.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.440+0000 7f1a5affd700 1 -- 172.21.15.162:0/3538035531 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1a60052080 con 0x7f1a640f73a0 2024-02-13T00:42:17.523 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:17.550 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:17.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.617+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3538035531 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1a6405fc80 con 0x7f1a50062ce0 2024-02-13T00:42:17.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.638+0000 7f1a5affd700 1 -- 172.21.15.162:0/3538035531 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+810859 (secure 0 0 0) 0x7f1a6405fc80 con 0x7f1a50062ce0 2024-02-13T00:42:17.652 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:42:17.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.643+0000 7f1a6baf8700 10 librados: watch_flush enter 2024-02-13T00:42:17.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.643+0000 7f1a6baf8700 10 librados: watch_flush exit 2024-02-13T00:42:17.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.643+0000 7f1a6baf8700 20 client.26550.objecter shutdown clearing up homeless session... 2024-02-13T00:42:17.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.643+0000 7f1a6baf8700 10 client.26550.objecter successfully canceled tick 2024-02-13T00:42:17.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.643+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3538035531 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1a50062ce0 msgr2=0x7f1a500651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.643+0000 7f1a6baf8700 1 --2- 172.21.15.162:0/3538035531 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1a50062ce0 0x7f1a500651a0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 rx=0x7f1a64107a00 tx=0x7f1a5400b040).stop 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.644+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3538035531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a640f73a0 msgr2=0x7f1a641068f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.644+0000 7f1a6baf8700 1 --2- 172.21.15.162:0/3538035531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a640f73a0 0x7f1a641068f0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 rx=0x7f1a60004650 tx=0x7f1a60007f90).stop 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.644+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3538035531 shutdown_connections 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.644+0000 7f1a6baf8700 1 --2- 172.21.15.162:0/3538035531 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1a50062ce0 0x7f1a500651a0 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.644+0000 7f1a6baf8700 1 --2- 172.21.15.162:0/3538035531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a640fd490 0x7f1a6410bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.644+0000 7f1a6baf8700 1 --2- 172.21.15.162:0/3538035531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a640f73a0 0x7f1a641068f0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.644+0000 7f1a6baf8700 1 --2- 172.21.15.162:0/3538035531 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1a640fcb30 0x7f1a64106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.644+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3538035531 >> 172.21.15.162:0/3538035531 conn(0x7f1a6406eb90 msgr2=0x7f1a64060ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.644+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3538035531 shutdown_connections 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.644+0000 7f1a6baf8700 1 -- 172.21.15.162:0/3538035531 wait complete. 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:17.644+0000 7f1a6baf8700 1 librados: shutdown 2024-02-13T00:42:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:42:17.698 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:17.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:17.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:17.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:17.830 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:17.928 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:17.930 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:16.332271+0000 mgr.y (mgr.14544) 213 : cluster [DBG] pgmap v241: 372 pgs: 64 unknown, 40 creating+peering, 1 active+clean+snaptrim, 267 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:16.656720+0000 osd.0 (osd.0) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:16.657198+0000 osd.0 (osd.0) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:17.108143+0000 mon.b (mon.1) 229 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:17.346731+0000 mon.a (mon.0) 1303 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:17.346859+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e182: 8 total, 8 up, 7 in 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:17.354293+0000 mon.c (mon.2) 157 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:17.362116+0000 mon.a (mon.0) 1305 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:17.363035+0000 mon.b (mon.1) 230 : audit [INF] from='client.? 172.21.15.162:0/2360399912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83985-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:17.364824+0000 mon.a (mon.0) 1306 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83985-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:17.393537+0000 mon.a (mon.0) 1307 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:17.394427+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:17.394727+0000 mon.a (mon.0) 1309 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-83977-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:17.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:17.444844+0000 osd.3 (osd.3) 113 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:42:17.932 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:17.454222+0000 osd.3 (osd.3) 114 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:42:17.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:16.332271+0000 mgr.y (mgr.14544) 213 : cluster [DBG] pgmap v241: 372 pgs: 64 unknown, 40 creating+peering, 1 active+clean+snaptrim, 267 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-13T00:42:17.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:16.656720+0000 osd.0 (osd.0) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:17.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:16.657198+0000 osd.0 (osd.0) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:17.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:17.108143+0000 mon.b (mon.1) 229 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:17.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:17.346731+0000 mon.a (mon.0) 1303 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:17.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:17.346859+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e182: 8 total, 8 up, 7 in 2024-02-13T00:42:17.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:17.354293+0000 mon.c (mon.2) 157 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:17.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:17.362116+0000 mon.a (mon.0) 1305 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:17.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:17.363035+0000 mon.b (mon.1) 230 : audit [INF] from='client.? 172.21.15.162:0/2360399912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83985-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:17.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:17.364824+0000 mon.a (mon.0) 1306 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83985-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:17.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:17.393537+0000 mon.a (mon.0) 1307 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:17.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:17.394427+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:17.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:17.394727+0000 mon.a (mon.0) 1309 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-83977-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:17.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:17.444844+0000 osd.3 (osd.3) 113 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:42:17.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:17.454222+0000 osd.3 (osd.3) 114 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:42:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:17.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:17.947 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:42:17.947 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:42:18.034 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:18.064 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:18.187 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:18.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:16.332271+0000 mgr.y (mgr.14544) 213 : cluster [DBG] pgmap v241: 372 pgs: 64 unknown, 40 creating+peering, 1 active+clean+snaptrim, 267 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-13T00:42:18.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:16.656720+0000 osd.0 (osd.0) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:18.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:16.657198+0000 osd.0 (osd.0) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:18.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:17.108143+0000 mon.b (mon.1) 229 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:18.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:17.346731+0000 mon.a (mon.0) 1303 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84049-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:18.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:17.346859+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e182: 8 total, 8 up, 7 in 2024-02-13T00:42:18.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:17.354293+0000 mon.c (mon.2) 157 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:18.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:17.362116+0000 mon.a (mon.0) 1305 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:18.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:17.363035+0000 mon.b (mon.1) 230 : audit [INF] from='client.? 172.21.15.162:0/2360399912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83985-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:18.189 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:17.364824+0000 mon.a (mon.0) 1306 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83985-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:18.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:17.393537+0000 mon.a (mon.0) 1307 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:18.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:17.394427+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:18.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:17.394727+0000 mon.a (mon.0) 1309 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-83977-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:18.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:17.444844+0000 osd.3 (osd.3) 113 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:42:18.190 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:17.454222+0000 osd.3 (osd.3) 114 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:42:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:18.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:18.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:18.318 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:18.523 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-02-13T00:42:17.347732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":15357,"num_objects":967,"num_object_clones":541,"num_object_copies":2901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":746,"num_whiteouts":0,"num_read":1644,"num_read_kb":1282,"num_write":1237,"num_write_kb":615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":353,"num_bytes_recovered":4677,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2008,"ondisk_log_size":2008,"up":804,"acting":804,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":932,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2131152,"kb_used_data":25996,"kb_used_omap":0,"kb_used_meta":2105088,"kb_avail":654084912,"statfs":{"total":671965249536,"available":669782949888,"internally_reserved":0,"allocated":26619904,"data_stored":9896581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2155610112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-128,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.007790"},"pg_stats":[{"pgid":"122.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"118.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:42:14.334995+0000","last_change":"2024-02-13T00:42:14.350652+0000","last_active":"2024-02-13T00:42:14.334995+0000","last_peered":"2024-02-13T00:42:14.334995+0000","last_clean":"2024-02-13T00:42:14.334995+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:14.334995+0000","last_undegraded":"2024-02-13T00:42:14.334995+0000","last_fullsized":"2024-02-13T00:42:14.334995+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_clean_scrub_stamp":"2024-02-13T00:42:14.334995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"122.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"118.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:42:14.334995+0000","last_change":"2024-02-13T00:42:14.351279+0000","last_active":"2024-02-13T00:42:14.334995+0000","last_peered":"2024-02-13T00:42:14.334995+0000","last_clean":"2024-02-13T00:42:14.334995+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:14.334995+0000","last_undegraded":"2024-02-13T00:42:14.334995+0000","last_fullsized":"2024-02-13T00:42:14.334995+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_clean_scrub_stamp":"2024-02-13T00:42:14.334995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"122.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"118.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:42:14.334995+0000","last_change":"2024-02-13T00:42:14.345132+0000","last_active":"2024-02-13T00:42:14.334995+0000","last_peered":"2024-02-13T00:42:14.334995+0000","last_clean":"2024-02-13T00:42:14.334995+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:14.334995+0000","last_undegraded":"2024-02-13T00:42:14.334995+0000","last_fullsized":"2024-02-13T00:42:14.334995+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_clean_scrub_stamp":"2024-02-13T00:42:14.334995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"122.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"118.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:42:14.334995+0000","last_change":"2024-02-13T00:42:14.343291+0000","last_active":"2024-02-13T00:42:14.334995+0000","last_peered":"2024-02-13T00:42:14.334995+0000","last_clean":"2024-02-13T00:42:14.334995+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:14.334995+0000","last_undegraded":"2024-02-13T00:42:14.334995+0000","last_fullsized":"2024-02-13T00:42:14.334995+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_clean_scrub_stamp":"2024-02-13T00:42:14.334995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"122.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"118.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:42:14.334995+0000","last_change":"2024-02-13T00:42:14.347913+0000","last_active":"2024-02-13T00:42:14.334995+0000","last_peered":"2024-02-13T00:42:14.334995+0000","last_clean":"2024-02-13T00:42:14.334995+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:14.334995+0000","last_undegraded":"2024-02-13T00:42:14.334995+0000","last_fullsized":"2024-02-13T00:42:14.334995+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_clean_scrub_stamp":"2024-02-13T00:42:14.334995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"122.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"118.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:42:14.334995+0000","last_change":"2024-02-13T00:42:14.350899+0000","last_active":"2024-02-13T00:42:14.334995+0000","last_peered":"2024-02-13T00:42:14.334995+0000","last_clean":"2024-02-13T00:42:14.334995+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:14.334995+0000","last_undegraded":"2024-02-13T00:42:14.334995+0000","last_fullsized":"2024-02-13T00:42:14.334995+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_clean_scrub_stamp":"2024-02-13T00:42:14.334995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"122.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"118.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:42:14.334995+0000","last_change":"2024-02-13T00:42:14.353321+0000","last_active":"2024-02-13T00:42:14.334995+0000","last_peered":"2024-02-13T00:42:14.334995+0000","last_clean":"2024-02-13T00:42:14.334995+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:14.334995+0000","last_undegraded":"2024-02-13T00:42:14.334995+0000","last_fullsized":"2024-02-13T00:42:14.334995+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_clean_scrub_stamp":"2024-02-13T00:42:14.334995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"122.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"118.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:42:14.334995+0000","last_change":"2024-02-13T00:42:14.353286+0000","last_active":"2024-02-13T00:42:14.334995+0000","last_peered":"2024-02-13T00:42:14.334995+0000","last_clean":"2024-02-13T00:42:14.334995+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:14.334995+0000","last_undegraded":"2024-02-13T00:42:14.334995+0000","last_fullsized":"2024-02-13T00:42:14.334995+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:14.334995+0000","last_clean_scrub_stamp":"2024-02-13T00:42:14.334995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"122.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"122.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:17.340813+0000","last_change":"2024-02-13T00:42:17.340813+0000","last_active":"2024-02-13T00:42:17.340813+0000","last_peered":"2024-02-13T00:42:17.340813+0000","last_clean":"2024-02-13T00:42:17.340813+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:17.340813+0000","last_undegraded":"2024-02-13T00:42:17.340813+0000","last_fullsized":"2024-02-13T00:42:17.340813+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:17.340813+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.340813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"120.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:42:15.336792+0000","last_change":"2024-02-13T00:42:15.336792+0000","last_active":"2024-02-13T00:42:15.336792+0000","last_peered":"2024-02-13T00:42:15.336792+0000","last_clean":"2024-02-13T00:42:15.336792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:42:15.336792+0000","last_undegraded":"2024-02-13T00:42:15.336792+0000","last_fullsized":"2024-02-13T00:42:15.336792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:15.336792+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.336792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"95.0","version":"157'6","reported_seq":61,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.344360+0000","last_change":"2024-02-13T00:42:06.692515+0000","last_active":"2024-02-13T00:42:13.344360+0000","last_peered":"2024-02-13T00:42:13.344360+0000","last_clean":"2024-02-13T00:42:13.344360+0000","last_became_active":"2024-02-13T00:42:03.403338+0000","last_became_peered":"2024-02-13T00:42:03.403338+0000","last_unstale":"2024-02-13T00:42:13.344360+0000","last_undegraded":"2024-02-13T00:42:13.344360+0000","last_fullsized":"2024-02-13T00:42:13.344360+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'6","last_scrub_stamp":"2024-02-13T00:41:53.416822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:53.416822+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":60,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.1","version":"157'4","reported_seq":53,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.581829+0000","last_change":"2024-02-13T00:42:03.796899+0000","last_active":"2024-02-13T00:42:13.581829+0000","last_peered":"2024-02-13T00:42:13.581829+0000","last_clean":"2024-02-13T00:42:13.581829+0000","last_became_active":"2024-02-13T00:42:03.396206+0000","last_became_peered":"2024-02-13T00:42:03.396206+0000","last_unstale":"2024-02-13T00:42:13.581829+0000","last_undegraded":"2024-02-13T00:42:13.581829+0000","last_fullsized":"2024-02-13T00:42:13.581829+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'4","last_scrub_stamp":"2024-02-13T00:41:55.360030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.360030+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.2","version":"157'5","reported_seq":72,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583024+0000","last_change":"2024-02-13T00:42:07.453099+0000","last_active":"2024-02-13T00:42:13.583024+0000","last_peered":"2024-02-13T00:42:13.583024+0000","last_clean":"2024-02-13T00:42:13.583024+0000","last_became_active":"2024-02-13T00:42:03.427902+0000","last_became_peered":"2024-02-13T00:42:03.427902+0000","last_unstale":"2024-02-13T00:42:13.583024+0000","last_undegraded":"2024-02-13T00:42:13.583024+0000","last_fullsized":"2024-02-13T00:42:13.583024+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'5","last_scrub_stamp":"2024-02-13T00:41:53.626722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:53.626722+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"95.3","version":"157'10","reported_seq":93,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346426+0000","last_change":"2024-02-13T00:42:04.280389+0000","last_active":"2024-02-13T00:42:13.346426+0000","last_peered":"2024-02-13T00:42:13.346426+0000","last_clean":"2024-02-13T00:42:13.346426+0000","last_became_active":"2024-02-13T00:42:03.395519+0000","last_became_peered":"2024-02-13T00:42:03.395519+0000","last_unstale":"2024-02-13T00:42:13.346426+0000","last_undegraded":"2024-02-13T00:42:13.346426+0000","last_fullsized":"2024-02-13T00:42:13.346426+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'10","last_scrub_stamp":"2024-02-13T00:41:53.411320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:53.411320+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.4","version":"157'6","reported_seq":54,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346910+0000","last_change":"2024-02-13T00:41:53.719522+0000","last_active":"2024-02-13T00:42:13.346910+0000","last_peered":"2024-02-13T00:42:13.346910+0000","last_clean":"2024-02-13T00:42:13.346910+0000","last_became_active":"2024-02-13T00:41:45.587390+0000","last_became_peered":"2024-02-13T00:41:45.587390+0000","last_unstale":"2024-02-13T00:42:13.346910+0000","last_undegraded":"2024-02-13T00:42:13.346910+0000","last_fullsized":"2024-02-13T00:42:13.346910+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'6","last_scrub_stamp":"2024-02-13T00:41:53.719439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:53.719439+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"95.5","version":"157'6","reported_seq":53,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584037+0000","last_change":"2024-02-13T00:41:56.579323+0000","last_active":"2024-02-13T00:42:13.584037+0000","last_peered":"2024-02-13T00:42:13.584037+0000","last_clean":"2024-02-13T00:42:13.584037+0000","last_became_active":"2024-02-13T00:41:45.577948+0000","last_became_peered":"2024-02-13T00:41:45.577948+0000","last_unstale":"2024-02-13T00:42:13.584037+0000","last_undegraded":"2024-02-13T00:42:13.584037+0000","last_fullsized":"2024-02-13T00:42:13.584037+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'6","last_scrub_stamp":"2024-02-13T00:41:56.579248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.579248+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.6","version":"157'3","reported_seq":49,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345500+0000","last_change":"2024-02-13T00:42:03.795273+0000","last_active":"2024-02-13T00:42:13.345500+0000","last_peered":"2024-02-13T00:42:13.345500+0000","last_clean":"2024-02-13T00:42:13.345500+0000","last_became_active":"2024-02-13T00:42:03.400945+0000","last_became_peered":"2024-02-13T00:42:03.400945+0000","last_unstale":"2024-02-13T00:42:13.345500+0000","last_undegraded":"2024-02-13T00:42:13.345500+0000","last_fullsized":"2024-02-13T00:42:13.345500+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'3","last_scrub_stamp":"2024-02-13T00:41:56.346980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.346980+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":30,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"95.7","version":"157'15","reported_seq":71,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585870+0000","last_change":"2024-02-13T00:41:54.670725+0000","last_active":"2024-02-13T00:42:13.585870+0000","last_peered":"2024-02-13T00:42:13.585870+0000","last_clean":"2024-02-13T00:42:13.585870+0000","last_became_active":"2024-02-13T00:41:45.587309+0000","last_became_peered":"2024-02-13T00:41:45.587309+0000","last_unstale":"2024-02-13T00:42:13.585870+0000","last_undegraded":"2024-02-13T00:42:13.585870+0000","last_fullsized":"2024-02-13T00:42:13.585870+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'15","last_scrub_stamp":"2024-02-13T00:41:54.670653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:54.670653+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"95.8","version":"157'9","reported_seq":87,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586397+0000","last_change":"2024-02-13T00:42:08.850651+0000","last_active":"2024-02-13T00:42:13.586397+0000","last_peered":"2024-02-13T00:42:13.586397+0000","last_clean":"2024-02-13T00:42:13.586397+0000","last_became_active":"2024-02-13T00:42:03.428410+0000","last_became_peered":"2024-02-13T00:42:03.428410+0000","last_unstale":"2024-02-13T00:42:13.586397+0000","last_undegraded":"2024-02-13T00:42:13.586397+0000","last_fullsized":"2024-02-13T00:42:13.586397+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'9","last_scrub_stamp":"2024-02-13T00:41:53.162169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:53.162169+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":93,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.9","version":"157'2","reported_seq":46,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347096+0000","last_change":"2024-02-13T00:41:58.623065+0000","last_active":"2024-02-13T00:42:13.347096+0000","last_peered":"2024-02-13T00:42:13.347096+0000","last_clean":"2024-02-13T00:42:13.347096+0000","last_became_active":"2024-02-13T00:41:45.588805+0000","last_became_peered":"2024-02-13T00:41:45.588805+0000","last_unstale":"2024-02-13T00:42:13.347096+0000","last_undegraded":"2024-02-13T00:42:13.347096+0000","last_fullsized":"2024-02-13T00:42:13.347096+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'2","last_scrub_stamp":"2024-02-13T00:41:58.623000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:58.623000+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"95.a","version":"157'14","reported_seq":155,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.349142+0000","last_change":"2024-02-13T00:42:06.115588+0000","last_active":"2024-02-13T00:42:13.349142+0000","last_peered":"2024-02-13T00:42:13.349142+0000","last_clean":"2024-02-13T00:42:13.349142+0000","last_became_active":"2024-02-13T00:42:03.397712+0000","last_became_peered":"2024-02-13T00:42:03.397712+0000","last_unstale":"2024-02-13T00:42:13.349142+0000","last_undegraded":"2024-02-13T00:42:13.349142+0000","last_fullsized":"2024-02-13T00:42:13.349142+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":142,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":64,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.b","version":"0'0","reported_seq":38,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346321+0000","last_change":"2024-02-13T00:41:45.586208+0000","last_active":"2024-02-13T00:42:13.346321+0000","last_peered":"2024-02-13T00:42:13.346321+0000","last_clean":"2024-02-13T00:42:13.346321+0000","last_became_active":"2024-02-13T00:41:45.585923+0000","last_became_peered":"2024-02-13T00:41:45.585923+0000","last_unstale":"2024-02-13T00:42:13.346321+0000","last_undegraded":"2024-02-13T00:42:13.346321+0000","last_fullsized":"2024-02-13T00:42:13.346321+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.c","version":"157'15","reported_seq":65,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346716+0000","last_change":"2024-02-13T00:41:45.583404+0000","last_active":"2024-02-13T00:42:13.346716+0000","last_peered":"2024-02-13T00:42:13.346716+0000","last_clean":"2024-02-13T00:42:13.346716+0000","last_became_active":"2024-02-13T00:41:45.583098+0000","last_became_peered":"2024-02-13T00:41:45.583098+0000","last_unstale":"2024-02-13T00:42:13.346716+0000","last_undegraded":"2024-02-13T00:42:13.346716+0000","last_fullsized":"2024-02-13T00:42:13.346716+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":153,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.d","version":"157'10","reported_seq":86,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584412+0000","last_change":"2024-02-13T00:42:07.543694+0000","last_active":"2024-02-13T00:42:13.584412+0000","last_peered":"2024-02-13T00:42:13.584412+0000","last_clean":"2024-02-13T00:42:13.584412+0000","last_became_active":"2024-02-13T00:42:03.419557+0000","last_became_peered":"2024-02-13T00:42:03.419557+0000","last_unstale":"2024-02-13T00:42:13.584412+0000","last_undegraded":"2024-02-13T00:42:13.584412+0000","last_fullsized":"2024-02-13T00:42:13.584412+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.e","version":"157'9","reported_seq":53,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586458+0000","last_change":"2024-02-13T00:41:45.583958+0000","last_active":"2024-02-13T00:42:13.586458+0000","last_peered":"2024-02-13T00:42:13.586458+0000","last_clean":"2024-02-13T00:42:13.586458+0000","last_became_active":"2024-02-13T00:41:45.583635+0000","last_became_peered":"2024-02-13T00:41:45.583635+0000","last_unstale":"2024-02-13T00:42:13.586458+0000","last_undegraded":"2024-02-13T00:42:13.586458+0000","last_fullsized":"2024-02-13T00:42:13.586458+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"95.f","version":"157'5","reported_seq":67,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347541+0000","last_change":"2024-02-13T00:42:06.935024+0000","last_active":"2024-02-13T00:42:13.347541+0000","last_peered":"2024-02-13T00:42:13.347541+0000","last_clean":"2024-02-13T00:42:13.347541+0000","last_became_active":"2024-02-13T00:42:03.413594+0000","last_became_peered":"2024-02-13T00:42:03.413594+0000","last_unstale":"2024-02-13T00:42:13.347541+0000","last_undegraded":"2024-02-13T00:42:13.347541+0000","last_fullsized":"2024-02-13T00:42:13.347541+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"95.10","version":"157'12","reported_seq":84,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.580430+0000","last_change":"2024-02-13T00:42:03.452546+0000","last_active":"2024-02-13T00:42:13.580430+0000","last_peered":"2024-02-13T00:42:13.580430+0000","last_clean":"2024-02-13T00:42:13.580430+0000","last_became_active":"2024-02-13T00:42:03.400686+0000","last_became_peered":"2024-02-13T00:42:03.400686+0000","last_unstale":"2024-02-13T00:42:13.580430+0000","last_undegraded":"2024-02-13T00:42:13.580430+0000","last_fullsized":"2024-02-13T00:42:13.580430+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":120,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"95.11","version":"157'6","reported_seq":49,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.581841+0000","last_change":"2024-02-13T00:41:45.584306+0000","last_active":"2024-02-13T00:42:13.581841+0000","last_peered":"2024-02-13T00:42:13.581841+0000","last_clean":"2024-02-13T00:42:13.581841+0000","last_became_active":"2024-02-13T00:41:45.582708+0000","last_became_peered":"2024-02-13T00:41:45.582708+0000","last_unstale":"2024-02-13T00:42:13.581841+0000","last_undegraded":"2024-02-13T00:42:13.581841+0000","last_fullsized":"2024-02-13T00:42:13.581841+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.12","version":"157'4","reported_seq":45,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585324+0000","last_change":"2024-02-13T00:41:45.581540+0000","last_active":"2024-02-13T00:42:13.585324+0000","last_peered":"2024-02-13T00:42:13.585324+0000","last_clean":"2024-02-13T00:42:13.585324+0000","last_became_active":"2024-02-13T00:41:45.581239+0000","last_became_peered":"2024-02-13T00:41:45.581239+0000","last_unstale":"2024-02-13T00:42:13.585324+0000","last_undegraded":"2024-02-13T00:42:13.585324+0000","last_fullsized":"2024-02-13T00:42:13.585324+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.13","version":"157'8","reported_seq":52,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.344536+0000","last_change":"2024-02-13T00:41:45.589749+0000","last_active":"2024-02-13T00:42:13.344536+0000","last_peered":"2024-02-13T00:42:13.344536+0000","last_clean":"2024-02-13T00:42:13.344536+0000","last_became_active":"2024-02-13T00:41:45.589398+0000","last_became_peered":"2024-02-13T00:41:45.589398+0000","last_unstale":"2024-02-13T00:42:13.344536+0000","last_undegraded":"2024-02-13T00:42:13.344536+0000","last_fullsized":"2024-02-13T00:42:13.344536+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"95.14","version":"157'2","reported_seq":56,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347338+0000","last_change":"2024-02-13T00:42:07.021971+0000","last_active":"2024-02-13T00:42:13.347338+0000","last_peered":"2024-02-13T00:42:13.347338+0000","last_clean":"2024-02-13T00:42:13.347338+0000","last_became_active":"2024-02-13T00:42:03.404220+0000","last_became_peered":"2024-02-13T00:42:03.404220+0000","last_unstale":"2024-02-13T00:42:13.347338+0000","last_undegraded":"2024-02-13T00:42:13.347338+0000","last_fullsized":"2024-02-13T00:42:13.347338+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.15","version":"157'10","reported_seq":85,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582923+0000","last_change":"2024-02-13T00:42:07.824810+0000","last_active":"2024-02-13T00:42:13.582923+0000","last_peered":"2024-02-13T00:42:13.582923+0000","last_clean":"2024-02-13T00:42:13.582923+0000","last_became_active":"2024-02-13T00:42:03.413088+0000","last_became_peered":"2024-02-13T00:42:03.413088+0000","last_unstale":"2024-02-13T00:42:13.582923+0000","last_undegraded":"2024-02-13T00:42:13.582923+0000","last_fullsized":"2024-02-13T00:42:13.582923+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":107,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.16","version":"157'6","reported_seq":49,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.341585+0000","last_change":"2024-02-13T00:41:45.564229+0000","last_active":"2024-02-13T00:42:13.341585+0000","last_peered":"2024-02-13T00:42:13.341585+0000","last_clean":"2024-02-13T00:42:13.341585+0000","last_became_active":"2024-02-13T00:41:45.563989+0000","last_became_peered":"2024-02-13T00:41:45.563989+0000","last_unstale":"2024-02-13T00:42:13.341585+0000","last_undegraded":"2024-02-13T00:42:13.341585+0000","last_fullsized":"2024-02-13T00:42:13.341585+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"95.17","version":"157'10","reported_seq":55,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.348519+0000","last_change":"2024-02-13T00:41:45.583444+0000","last_active":"2024-02-13T00:42:13.348519+0000","last_peered":"2024-02-13T00:42:13.348519+0000","last_clean":"2024-02-13T00:42:13.348519+0000","last_became_active":"2024-02-13T00:41:45.583153+0000","last_became_peered":"2024-02-13T00:41:45.583153+0000","last_unstale":"2024-02-13T00:42:13.348519+0000","last_undegraded":"2024-02-13T00:42:13.348519+0000","last_fullsized":"2024-02-13T00:42:13.348519+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.18","version":"157'7","reported_seq":65,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.581413+0000","last_change":"2024-02-13T00:42:06.439094+0000","last_active":"2024-02-13T00:42:13.581413+0000","last_peered":"2024-02-13T00:42:13.581413+0000","last_clean":"2024-02-13T00:42:13.581413+0000","last_became_active":"2024-02-13T00:42:03.398953+0000","last_became_peered":"2024-02-13T00:42:03.398953+0000","last_unstale":"2024-02-13T00:42:13.581413+0000","last_undegraded":"2024-02-13T00:42:13.581413+0000","last_fullsized":"2024-02-13T00:42:13.581413+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":71,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.19","version":"157'3","reported_seq":60,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.581736+0000","last_change":"2024-02-13T00:42:06.965034+0000","last_active":"2024-02-13T00:42:13.581736+0000","last_peered":"2024-02-13T00:42:13.581736+0000","last_clean":"2024-02-13T00:42:13.581736+0000","last_became_active":"2024-02-13T00:42:03.425499+0000","last_became_peered":"2024-02-13T00:42:03.425499+0000","last_unstale":"2024-02-13T00:42:13.581736+0000","last_undegraded":"2024-02-13T00:42:13.581736+0000","last_fullsized":"2024-02-13T00:42:13.581736+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":30,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.1a","version":"157'4","reported_seq":46,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584148+0000","last_change":"2024-02-13T00:41:45.588294+0000","last_active":"2024-02-13T00:42:13.584148+0000","last_peered":"2024-02-13T00:42:13.584148+0000","last_clean":"2024-02-13T00:42:13.584148+0000","last_became_active":"2024-02-13T00:41:45.588053+0000","last_became_peered":"2024-02-13T00:41:45.588053+0000","last_unstale":"2024-02-13T00:42:13.584148+0000","last_undegraded":"2024-02-13T00:42:13.584148+0000","last_fullsized":"2024-02-13T00:42:13.584148+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.1b","version":"157'2","reported_seq":45,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586250+0000","last_change":"2024-02-13T00:42:07.478136+0000","last_active":"2024-02-13T00:42:13.586250+0000","last_peered":"2024-02-13T00:42:13.586250+0000","last_clean":"2024-02-13T00:42:13.586250+0000","last_became_active":"2024-02-13T00:42:03.428980+0000","last_became_peered":"2024-02-13T00:42:03.428980+0000","last_unstale":"2024-02-13T00:42:13.586250+0000","last_undegraded":"2024-02-13T00:42:13.586250+0000","last_fullsized":"2024-02-13T00:42:13.586250+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.1c","version":"157'6","reported_seq":61,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582886+0000","last_change":"2024-02-13T00:42:04.796712+0000","last_active":"2024-02-13T00:42:13.582886+0000","last_peered":"2024-02-13T00:42:13.582886+0000","last_clean":"2024-02-13T00:42:13.582886+0000","last_became_active":"2024-02-13T00:42:03.405382+0000","last_became_peered":"2024-02-13T00:42:03.405382+0000","last_unstale":"2024-02-13T00:42:13.582886+0000","last_undegraded":"2024-02-13T00:42:13.582886+0000","last_fullsized":"2024-02-13T00:42:13.582886+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":63,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.1d","version":"157'9","reported_seq":54,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583947+0000","last_change":"2024-02-13T00:41:45.575272+0000","last_active":"2024-02-13T00:42:13.583947+0000","last_peered":"2024-02-13T00:42:13.583947+0000","last_clean":"2024-02-13T00:42:13.583947+0000","last_became_active":"2024-02-13T00:41:45.575007+0000","last_became_peered":"2024-02-13T00:41:45.575007+0000","last_unstale":"2024-02-13T00:42:13.583947+0000","last_undegraded":"2024-02-13T00:42:13.583947+0000","last_fullsized":"2024-02-13T00:42:13.583947+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.1e","version":"157'6","reported_seq":50,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.587348+0000","last_change":"2024-02-13T00:41:45.578695+0000","last_active":"2024-02-13T00:42:13.587348+0000","last_peered":"2024-02-13T00:42:13.587348+0000","last_clean":"2024-02-13T00:42:13.587348+0000","last_became_active":"2024-02-13T00:41:45.577095+0000","last_became_peered":"2024-02-13T00:41:45.577095+0000","last_unstale":"2024-02-13T00:42:13.587348+0000","last_undegraded":"2024-02-13T00:42:13.587348+0000","last_fullsized":"2024-02-13T00:42:13.587348+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.1f","version":"157'7","reported_seq":50,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.348481+0000","last_change":"2024-02-13T00:41:45.579789+0000","last_active":"2024-02-13T00:42:13.348481+0000","last_peered":"2024-02-13T00:42:13.348481+0000","last_clean":"2024-02-13T00:42:13.348481+0000","last_became_active":"2024-02-13T00:41:45.579553+0000","last_became_peered":"2024-02-13T00:41:45.579553+0000","last_unstale":"2024-02-13T00:42:13.348481+0000","last_undegraded":"2024-02-13T00:42:13.348481+0000","last_fullsized":"2024-02-13T00:42:13.348481+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:44.541836+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.541836+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.2","version":"154'18","reported_seq":119,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585981+0000","last_change":"2024-02-13T00:42:09.361982+0000","last_active":"2024-02-13T00:42:13.585981+0000","last_peered":"2024-02-13T00:42:13.585981+0000","last_clean":"2024-02-13T00:42:13.585981+0000","last_became_active":"2024-02-13T00:42:03.433088+0000","last_became_peered":"2024-02-13T00:42:03.433088+0000","last_unstale":"2024-02-13T00:42:13.585981+0000","last_undegraded":"2024-02-13T00:42:13.585981+0000","last_fullsized":"2024-02-13T00:42:13.585981+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.3","version":"154'55","reported_seq":117,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584974+0000","last_change":"2024-02-13T00:41:40.457027+0000","last_active":"2024-02-13T00:42:13.584974+0000","last_peered":"2024-02-13T00:42:13.584974+0000","last_clean":"2024-02-13T00:42:13.584974+0000","last_became_active":"2024-02-13T00:41:16.803266+0000","last_became_peered":"2024-02-13T00:41:16.803266+0000","last_unstale":"2024-02-13T00:42:13.584974+0000","last_undegraded":"2024-02-13T00:42:13.584974+0000","last_fullsized":"2024-02-13T00:42:13.584974+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":316,"num_objects":31,"num_object_clones":24,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.0","version":"154'18","reported_seq":89,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.349742+0000","last_change":"2024-02-13T00:41:40.439197+0000","last_active":"2024-02-13T00:42:13.349742+0000","last_peered":"2024-02-13T00:42:13.349742+0000","last_clean":"2024-02-13T00:42:13.349742+0000","last_became_active":"2024-02-13T00:41:16.768067+0000","last_became_peered":"2024-02-13T00:41:16.768067+0000","last_unstale":"2024-02-13T00:42:13.349742+0000","last_undegraded":"2024-02-13T00:42:13.349742+0000","last_fullsized":"2024-02-13T00:42:13.349742+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.1","version":"154'18","reported_seq":86,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582079+0000","last_change":"2024-02-13T00:41:40.459485+0000","last_active":"2024-02-13T00:42:13.582079+0000","last_peered":"2024-02-13T00:42:13.582079+0000","last_clean":"2024-02-13T00:42:13.582079+0000","last_became_active":"2024-02-13T00:41:16.734552+0000","last_became_peered":"2024-02-13T00:41:16.734552+0000","last_unstale":"2024-02-13T00:42:13.582079+0000","last_undegraded":"2024-02-13T00:42:13.582079+0000","last_fullsized":"2024-02-13T00:42:13.582079+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.6","version":"154'9","reported_seq":80,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342438+0000","last_change":"2024-02-13T00:41:40.421568+0000","last_active":"2024-02-13T00:42:13.342438+0000","last_peered":"2024-02-13T00:42:13.342438+0000","last_clean":"2024-02-13T00:42:13.342438+0000","last_became_active":"2024-02-13T00:41:16.791271+0000","last_became_peered":"2024-02-13T00:41:16.791271+0000","last_unstale":"2024-02-13T00:42:13.342438+0000","last_undegraded":"2024-02-13T00:42:13.342438+0000","last_fullsized":"2024-02-13T00:42:13.342438+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.7","version":"154'45","reported_seq":116,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.348581+0000","last_change":"2024-02-13T00:41:40.343578+0000","last_active":"2024-02-13T00:42:13.348581+0000","last_peered":"2024-02-13T00:42:13.348581+0000","last_clean":"2024-02-13T00:42:13.348581+0000","last_became_active":"2024-02-13T00:41:16.776064+0000","last_became_peered":"2024-02-13T00:41:16.776064+0000","last_unstale":"2024-02-13T00:42:13.348581+0000","last_undegraded":"2024-02-13T00:42:13.348581+0000","last_fullsized":"2024-02-13T00:42:13.348581+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.4","version":"154'32","reported_seq":101,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.344075+0000","last_change":"2024-02-13T00:41:42.454078+0000","last_active":"2024-02-13T00:42:13.344075+0000","last_peered":"2024-02-13T00:42:13.344075+0000","last_clean":"2024-02-13T00:42:13.344075+0000","last_became_active":"2024-02-13T00:41:16.790979+0000","last_became_peered":"2024-02-13T00:41:16.790979+0000","last_unstale":"2024-02-13T00:42:13.344075+0000","last_undegraded":"2024-02-13T00:42:13.344075+0000","last_fullsized":"2024-02-13T00:42:13.344075+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":161,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.5","version":"154'32","reported_seq":146,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583174+0000","last_change":"2024-02-13T00:42:05.796628+0000","last_active":"2024-02-13T00:42:13.583174+0000","last_peered":"2024-02-13T00:42:13.583174+0000","last_clean":"2024-02-13T00:42:13.583174+0000","last_became_active":"2024-02-13T00:42:03.412443+0000","last_became_peered":"2024-02-13T00:42:03.412443+0000","last_unstale":"2024-02-13T00:42:13.583174+0000","last_undegraded":"2024-02-13T00:42:13.583174+0000","last_fullsized":"2024-02-13T00:42:13.583174+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.a","version":"154'45","reported_seq":114,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586561+0000","last_change":"2024-02-13T00:41:40.475242+0000","last_active":"2024-02-13T00:42:13.586561+0000","last_peered":"2024-02-13T00:42:13.586561+0000","last_clean":"2024-02-13T00:42:13.586561+0000","last_became_active":"2024-02-13T00:41:16.735087+0000","last_became_peered":"2024-02-13T00:41:16.735087+0000","last_unstale":"2024-02-13T00:42:13.586561+0000","last_undegraded":"2024-02-13T00:42:13.586561+0000","last_fullsized":"2024-02-13T00:42:13.586561+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.b","version":"154'9","reported_seq":97,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584882+0000","last_change":"2024-02-13T00:42:05.227753+0000","last_active":"2024-02-13T00:42:13.584882+0000","last_peered":"2024-02-13T00:42:13.584882+0000","last_clean":"2024-02-13T00:42:13.584882+0000","last_became_active":"2024-02-13T00:42:03.404523+0000","last_became_peered":"2024-02-13T00:42:03.404523+0000","last_unstale":"2024-02-13T00:42:13.584882+0000","last_undegraded":"2024-02-13T00:42:13.584882+0000","last_fullsized":"2024-02-13T00:42:13.584882+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.8","version":"154'27","reported_seq":98,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.344461+0000","last_change":"2024-02-13T00:41:40.323413+0000","last_active":"2024-02-13T00:42:13.344461+0000","last_peered":"2024-02-13T00:42:13.344461+0000","last_clean":"2024-02-13T00:42:13.344461+0000","last_became_active":"2024-02-13T00:41:16.749465+0000","last_became_peered":"2024-02-13T00:41:16.749465+0000","last_unstale":"2024-02-13T00:42:13.344461+0000","last_undegraded":"2024-02-13T00:42:13.344461+0000","last_fullsized":"2024-02-13T00:42:13.344461+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.9","version":"154'18","reported_seq":84,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582004+0000","last_change":"2024-02-13T00:41:40.459296+0000","last_active":"2024-02-13T00:42:13.582004+0000","last_peered":"2024-02-13T00:42:13.582004+0000","last_clean":"2024-02-13T00:42:13.582004+0000","last_became_active":"2024-02-13T00:41:16.768452+0000","last_became_peered":"2024-02-13T00:41:16.768452+0000","last_unstale":"2024-02-13T00:42:13.582004+0000","last_undegraded":"2024-02-13T00:42:13.582004+0000","last_fullsized":"2024-02-13T00:42:13.582004+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.e","version":"154'54","reported_seq":158,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347312+0000","last_change":"2024-02-13T00:42:05.834144+0000","last_active":"2024-02-13T00:42:13.347312+0000","last_peered":"2024-02-13T00:42:13.347312+0000","last_clean":"2024-02-13T00:42:13.347312+0000","last_became_active":"2024-02-13T00:42:03.399160+0000","last_became_peered":"2024-02-13T00:42:03.399160+0000","last_unstale":"2024-02-13T00:42:13.347312+0000","last_undegraded":"2024-02-13T00:42:13.347312+0000","last_fullsized":"2024-02-13T00:42:13.347312+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":312,"num_objects":30,"num_object_clones":24,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.f","version":"154'27","reported_seq":93,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586989+0000","last_change":"2024-02-13T00:41:40.334158+0000","last_active":"2024-02-13T00:42:13.586989+0000","last_peered":"2024-02-13T00:42:13.586989+0000","last_clean":"2024-02-13T00:42:13.586989+0000","last_became_active":"2024-02-13T00:41:16.697369+0000","last_became_peered":"2024-02-13T00:41:16.697369+0000","last_unstale":"2024-02-13T00:42:13.586989+0000","last_undegraded":"2024-02-13T00:42:13.586989+0000","last_fullsized":"2024-02-13T00:42:13.586989+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.c","version":"154'54","reported_seq":120,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346864+0000","last_change":"2024-02-13T00:41:40.347034+0000","last_active":"2024-02-13T00:42:13.346864+0000","last_peered":"2024-02-13T00:42:13.346864+0000","last_clean":"2024-02-13T00:42:13.346864+0000","last_became_active":"2024-02-13T00:41:16.702990+0000","last_became_peered":"2024-02-13T00:41:16.702990+0000","last_unstale":"2024-02-13T00:42:13.346864+0000","last_undegraded":"2024-02-13T00:42:13.346864+0000","last_fullsized":"2024-02-13T00:42:13.346864+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":312,"num_objects":30,"num_object_clones":24,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.d","version":"154'45","reported_seq":111,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345261+0000","last_change":"2024-02-13T00:41:40.337820+0000","last_active":"2024-02-13T00:42:13.345261+0000","last_peered":"2024-02-13T00:42:13.345261+0000","last_clean":"2024-02-13T00:42:13.345261+0000","last_became_active":"2024-02-13T00:41:16.780278+0000","last_became_peered":"2024-02-13T00:41:16.780278+0000","last_unstale":"2024-02-13T00:42:13.345261+0000","last_undegraded":"2024-02-13T00:42:13.345261+0000","last_fullsized":"2024-02-13T00:42:13.345261+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.12","version":"154'9","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.341201+0000","last_change":"2024-02-13T00:41:40.441960+0000","last_active":"2024-02-13T00:42:13.341201+0000","last_peered":"2024-02-13T00:42:13.341201+0000","last_clean":"2024-02-13T00:42:13.341201+0000","last_became_active":"2024-02-13T00:41:16.779347+0000","last_became_peered":"2024-02-13T00:41:16.779347+0000","last_unstale":"2024-02-13T00:42:13.341201+0000","last_undegraded":"2024-02-13T00:42:13.341201+0000","last_fullsized":"2024-02-13T00:42:13.341201+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.13","version":"154'36","reported_seq":153,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585266+0000","last_change":"2024-02-13T00:42:06.849462+0000","last_active":"2024-02-13T00:42:13.585266+0000","last_peered":"2024-02-13T00:42:13.585266+0000","last_clean":"2024-02-13T00:42:13.585266+0000","last_became_active":"2024-02-13T00:42:03.409062+0000","last_became_peered":"2024-02-13T00:42:03.409062+0000","last_unstale":"2024-02-13T00:42:13.585266+0000","last_undegraded":"2024-02-13T00:42:13.585266+0000","last_fullsized":"2024-02-13T00:42:13.585266+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.10","version":"154'36","reported_seq":107,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.580663+0000","last_change":"2024-02-13T00:41:40.325448+0000","last_active":"2024-02-13T00:42:13.580663+0000","last_peered":"2024-02-13T00:42:13.580663+0000","last_clean":"2024-02-13T00:42:13.580663+0000","last_became_active":"2024-02-13T00:41:16.802269+0000","last_became_peered":"2024-02-13T00:41:16.802269+0000","last_unstale":"2024-02-13T00:42:13.580663+0000","last_undegraded":"2024-02-13T00:42:13.580663+0000","last_fullsized":"2024-02-13T00:42:13.580663+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.11","version":"154'18","reported_seq":120,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342191+0000","last_change":"2024-02-13T00:42:04.408957+0000","last_active":"2024-02-13T00:42:13.342191+0000","last_peered":"2024-02-13T00:42:13.342191+0000","last_clean":"2024-02-13T00:42:13.342191+0000","last_became_active":"2024-02-13T00:42:03.391427+0000","last_became_peered":"2024-02-13T00:42:03.391427+0000","last_unstale":"2024-02-13T00:42:13.342191+0000","last_undegraded":"2024-02-13T00:42:13.342191+0000","last_fullsized":"2024-02-13T00:42:13.342191+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.16","version":"154'36","reported_seq":99,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347398+0000","last_change":"2024-02-13T00:41:40.350516+0000","last_active":"2024-02-13T00:42:13.347398+0000","last_peered":"2024-02-13T00:42:13.347398+0000","last_clean":"2024-02-13T00:42:13.347398+0000","last_became_active":"2024-02-13T00:41:16.694271+0000","last_became_peered":"2024-02-13T00:41:16.694271+0000","last_unstale":"2024-02-13T00:42:13.347398+0000","last_undegraded":"2024-02-13T00:42:13.347398+0000","last_fullsized":"2024-02-13T00:42:13.347398+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.17","version":"154'36","reported_seq":105,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.351929+0000","last_change":"2024-02-13T00:41:40.463812+0000","last_active":"2024-02-13T00:42:13.351929+0000","last_peered":"2024-02-13T00:42:13.351929+0000","last_clean":"2024-02-13T00:42:13.351929+0000","last_became_active":"2024-02-13T00:41:16.793277+0000","last_became_peered":"2024-02-13T00:41:16.793277+0000","last_unstale":"2024-02-13T00:42:13.351929+0000","last_undegraded":"2024-02-13T00:42:13.351929+0000","last_fullsized":"2024-02-13T00:42:13.351929+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.14","version":"154'9","reported_seq":79,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347424+0000","last_change":"2024-02-13T00:41:40.439969+0000","last_active":"2024-02-13T00:42:13.347424+0000","last_peered":"2024-02-13T00:42:13.347424+0000","last_clean":"2024-02-13T00:42:13.347424+0000","last_became_active":"2024-02-13T00:41:16.784290+0000","last_became_peered":"2024-02-13T00:41:16.784290+0000","last_unstale":"2024-02-13T00:42:13.347424+0000","last_undegraded":"2024-02-13T00:42:13.347424+0000","last_fullsized":"2024-02-13T00:42:13.347424+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.15","version":"154'46","reported_seq":117,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586403+0000","last_change":"2024-02-13T00:41:42.340164+0000","last_active":"2024-02-13T00:42:13.586403+0000","last_peered":"2024-02-13T00:42:13.586403+0000","last_clean":"2024-02-13T00:42:13.586403+0000","last_became_active":"2024-02-13T00:41:16.752633+0000","last_became_peered":"2024-02-13T00:41:16.752633+0000","last_unstale":"2024-02-13T00:42:13.586403+0000","last_undegraded":"2024-02-13T00:42:13.586403+0000","last_fullsized":"2024-02-13T00:42:13.586403+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":218,"num_objects":22,"num_object_clones":16,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1a","version":"154'27","reported_seq":134,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583110+0000","last_change":"2024-02-13T00:42:09.261606+0000","last_active":"2024-02-13T00:42:13.583110+0000","last_peered":"2024-02-13T00:42:13.583110+0000","last_clean":"2024-02-13T00:42:13.583110+0000","last_became_active":"2024-02-13T00:42:03.433054+0000","last_became_peered":"2024-02-13T00:42:03.433054+0000","last_unstale":"2024-02-13T00:42:13.583110+0000","last_undegraded":"2024-02-13T00:42:13.583110+0000","last_fullsized":"2024-02-13T00:42:13.583110+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1b","version":"154'18","reported_seq":115,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586344+0000","last_change":"2024-02-13T00:42:04.955132+0000","last_active":"2024-02-13T00:42:13.586344+0000","last_peered":"2024-02-13T00:42:13.586344+0000","last_clean":"2024-02-13T00:42:13.586344+0000","last_became_active":"2024-02-13T00:42:03.418968+0000","last_became_peered":"2024-02-13T00:42:03.418968+0000","last_unstale":"2024-02-13T00:42:13.586344+0000","last_undegraded":"2024-02-13T00:42:13.586344+0000","last_fullsized":"2024-02-13T00:42:13.586344+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.18","version":"154'18","reported_seq":90,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.581997+0000","last_change":"2024-02-13T00:41:40.493263+0000","last_active":"2024-02-13T00:42:13.581997+0000","last_peered":"2024-02-13T00:42:13.581997+0000","last_clean":"2024-02-13T00:42:13.581997+0000","last_became_active":"2024-02-13T00:41:16.747555+0000","last_became_peered":"2024-02-13T00:41:16.747555+0000","last_unstale":"2024-02-13T00:42:13.581997+0000","last_undegraded":"2024-02-13T00:42:13.581997+0000","last_fullsized":"2024-02-13T00:42:13.581997+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.19","version":"154'9","reported_seq":80,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342499+0000","last_change":"2024-02-13T00:41:40.321369+0000","last_active":"2024-02-13T00:42:13.342499+0000","last_peered":"2024-02-13T00:42:13.342499+0000","last_clean":"2024-02-13T00:42:13.342499+0000","last_became_active":"2024-02-13T00:41:16.749013+0000","last_became_peered":"2024-02-13T00:41:16.749013+0000","last_unstale":"2024-02-13T00:42:13.342499+0000","last_undegraded":"2024-02-13T00:42:13.342499+0000","last_fullsized":"2024-02-13T00:42:13.342499+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1e","version":"154'27","reported_seq":136,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585924+0000","last_change":"2024-02-13T00:42:06.815264+0000","last_active":"2024-02-13T00:42:13.585924+0000","last_peered":"2024-02-13T00:42:13.585924+0000","last_clean":"2024-02-13T00:42:13.585924+0000","last_became_active":"2024-02-13T00:42:03.414355+0000","last_became_peered":"2024-02-13T00:42:03.414355+0000","last_unstale":"2024-02-13T00:42:13.585924+0000","last_undegraded":"2024-02-13T00:42:13.585924+0000","last_fullsized":"2024-02-13T00:42:13.585924+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1f","version":"154'28","reported_seq":98,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347575+0000","last_change":"2024-02-13T00:41:40.413596+0000","last_active":"2024-02-13T00:42:13.347575+0000","last_peered":"2024-02-13T00:42:13.347575+0000","last_clean":"2024-02-13T00:42:13.347575+0000","last_became_active":"2024-02-13T00:41:16.807294+0000","last_became_peered":"2024-02-13T00:41:16.807294+0000","last_unstale":"2024-02-13T00:42:13.347575+0000","last_undegraded":"2024-02-13T00:42:13.347575+0000","last_fullsized":"2024-02-13T00:42:13.347575+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":164,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.1c","version":"154'27","reported_seq":135,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346801+0000","last_change":"2024-02-13T00:42:06.688943+0000","last_active":"2024-02-13T00:42:13.346801+0000","last_peered":"2024-02-13T00:42:13.346801+0000","last_clean":"2024-02-13T00:42:13.346801+0000","last_became_active":"2024-02-13T00:42:03.399127+0000","last_became_peered":"2024-02-13T00:42:03.399127+0000","last_unstale":"2024-02-13T00:42:13.346801+0000","last_undegraded":"2024-02-13T00:42:13.346801+0000","last_fullsized":"2024-02-13T00:42:13.346801+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1d","version":"154'36","reported_seq":103,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583080+0000","last_change":"2024-02-13T00:41:40.481431+0000","last_active":"2024-02-13T00:42:13.583080+0000","last_peered":"2024-02-13T00:42:13.583080+0000","last_clean":"2024-02-13T00:42:13.583080+0000","last_became_active":"2024-02-13T00:41:16.802530+0000","last_became_peered":"2024-02-13T00:41:16.802530+0000","last_unstale":"2024-02-13T00:42:13.583080+0000","last_undegraded":"2024-02-13T00:42:13.583080+0000","last_fullsized":"2024-02-13T00:42:13.583080+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":90,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.351379+0000","last_change":"2024-02-13T00:42:03.378511+0000","last_active":"2024-02-13T00:42:13.351379+0000","last_peered":"2024-02-13T00:42:13.351379+0000","last_clean":"2024-02-13T00:42:13.351379+0000","last_became_active":"2024-02-13T00:42:03.378168+0000","last_became_peered":"2024-02-13T00:42:03.378168+0000","last_unstale":"2024-02-13T00:42:13.351379+0000","last_undegraded":"2024-02-13T00:42:13.351379+0000","last_fullsized":"2024-02-13T00:42:13.351379+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"168'9","reported_seq":212,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:14.270515+0000","last_change":"2024-02-13T00:42:07.022553+0000","last_active":"2024-02-13T00:42:14.270515+0000","last_peered":"2024-02-13T00:42:14.270515+0000","last_clean":"2024-02-13T00:42:14.270515+0000","last_became_active":"2024-02-13T00:42:03.423814+0000","last_became_peered":"2024-02-13T00:42:03.423814+0000","last_unstale":"2024-02-13T00:42:14.270515+0000","last_undegraded":"2024-02-13T00:42:14.270515+0000","last_fullsized":"2024-02-13T00:42:14.270515+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:40:58.179446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.179446+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":231,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345459+0000","last_change":"2024-02-13T00:42:06.700478+0000","last_active":"2024-02-13T00:42:13.345459+0000","last_peered":"2024-02-13T00:42:13.345459+0000","last_clean":"2024-02-13T00:42:13.345459+0000","last_became_active":"2024-02-13T00:42:03.403540+0000","last_became_peered":"2024-02-13T00:42:03.403540+0000","last_unstale":"2024-02-13T00:42:13.345459+0000","last_undegraded":"2024-02-13T00:42:13.345459+0000","last_fullsized":"2024-02-13T00:42:13.345459+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:41:31.413581+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_clean_scrub_stamp":"2024-02-13T00:41:31.413581+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":141,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342932+0000","last_change":"2024-02-13T00:40:50.436030+0000","last_active":"2024-02-13T00:42:13.342932+0000","last_peered":"2024-02-13T00:42:13.342932+0000","last_clean":"2024-02-13T00:42:13.342932+0000","last_became_active":"2024-02-13T00:37:41.219093+0000","last_became_peered":"2024-02-13T00:37:41.219093+0000","last_unstale":"2024-02-13T00:42:13.342932+0000","last_undegraded":"2024-02-13T00:42:13.342932+0000","last_fullsized":"2024-02-13T00:42:13.342932+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:50.435958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:50.435958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":145,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.348894+0000","last_change":"2024-02-13T00:42:03.396737+0000","last_active":"2024-02-13T00:42:13.348894+0000","last_peered":"2024-02-13T00:42:13.348894+0000","last_clean":"2024-02-13T00:42:13.348894+0000","last_became_active":"2024-02-13T00:42:03.395729+0000","last_became_peered":"2024-02-13T00:42:03.395729+0000","last_unstale":"2024-02-13T00:42:13.348894+0000","last_undegraded":"2024-02-13T00:42:13.348894+0000","last_fullsized":"2024-02-13T00:42:13.348894+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:45.416238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:45.416238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":82,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347478+0000","last_change":"2024-02-13T00:41:01.627210+0000","last_active":"2024-02-13T00:42:13.347478+0000","last_peered":"2024-02-13T00:42:13.347478+0000","last_clean":"2024-02-13T00:42:13.347478+0000","last_became_active":"2024-02-13T00:41:01.624745+0000","last_became_peered":"2024-02-13T00:41:01.624745+0000","last_unstale":"2024-02-13T00:42:13.347478+0000","last_undegraded":"2024-02-13T00:42:13.347478+0000","last_fullsized":"2024-02-13T00:42:13.347478+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":90,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342569+0000","last_change":"2024-02-13T00:42:03.375612+0000","last_active":"2024-02-13T00:42:13.342569+0000","last_peered":"2024-02-13T00:42:13.342569+0000","last_clean":"2024-02-13T00:42:13.342569+0000","last_became_active":"2024-02-13T00:42:03.375294+0000","last_became_peered":"2024-02-13T00:42:03.375294+0000","last_unstale":"2024-02-13T00:42:13.342569+0000","last_undegraded":"2024-02-13T00:42:13.342569+0000","last_fullsized":"2024-02-13T00:42:13.342569+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":164,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345063+0000","last_change":"2024-02-13T00:39:24.376175+0000","last_active":"2024-02-13T00:42:13.345063+0000","last_peered":"2024-02-13T00:42:13.345063+0000","last_clean":"2024-02-13T00:42:13.345063+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:42:13.345063+0000","last_undegraded":"2024-02-13T00:42:13.345063+0000","last_fullsized":"2024-02-13T00:42:13.345063+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:39:24.376100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:24.376100+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":136,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583916+0000","last_change":"2024-02-13T00:39:16.262449+0000","last_active":"2024-02-13T00:42:13.583916+0000","last_peered":"2024-02-13T00:42:13.583916+0000","last_clean":"2024-02-13T00:42:13.583916+0000","last_became_active":"2024-02-13T00:37:41.212429+0000","last_became_peered":"2024-02-13T00:37:41.212429+0000","last_unstale":"2024-02-13T00:42:13.583916+0000","last_undegraded":"2024-02-13T00:42:13.583916+0000","last_fullsized":"2024-02-13T00:42:13.583916+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:16.262380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:16.262380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":137,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342049+0000","last_change":"2024-02-13T00:40:53.362331+0000","last_active":"2024-02-13T00:42:13.342049+0000","last_peered":"2024-02-13T00:42:13.342049+0000","last_clean":"2024-02-13T00:42:13.342049+0000","last_became_active":"2024-02-13T00:37:45.240923+0000","last_became_peered":"2024-02-13T00:37:45.240923+0000","last_unstale":"2024-02-13T00:42:13.342049+0000","last_undegraded":"2024-02-13T00:42:13.342049+0000","last_fullsized":"2024-02-13T00:42:13.342049+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:53.362232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:53.362232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":134,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.349513+0000","last_change":"2024-02-13T00:41:41.476236+0000","last_active":"2024-02-13T00:42:13.349513+0000","last_peered":"2024-02-13T00:42:13.349513+0000","last_clean":"2024-02-13T00:42:13.349513+0000","last_became_active":"2024-02-13T00:37:47.249556+0000","last_became_peered":"2024-02-13T00:37:47.249556+0000","last_unstale":"2024-02-13T00:42:13.349513+0000","last_undegraded":"2024-02-13T00:42:13.349513+0000","last_fullsized":"2024-02-13T00:42:13.349513+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1d","version":"108'7","reported_seq":89,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346811+0000","last_change":"2024-02-13T00:41:01.697227+0000","last_active":"2024-02-13T00:42:13.346811+0000","last_peered":"2024-02-13T00:42:13.346811+0000","last_clean":"2024-02-13T00:42:13.346811+0000","last_became_active":"2024-02-13T00:41:01.696134+0000","last_became_peered":"2024-02-13T00:41:01.696134+0000","last_unstale":"2024-02-13T00:42:13.346811+0000","last_undegraded":"2024-02-13T00:42:13.346811+0000","last_fullsized":"2024-02-13T00:42:13.346811+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346203+0000","last_change":"2024-02-13T00:41:01.656850+0000","last_active":"2024-02-13T00:42:13.346203+0000","last_peered":"2024-02-13T00:42:13.346203+0000","last_clean":"2024-02-13T00:42:13.346203+0000","last_became_active":"2024-02-13T00:41:01.654410+0000","last_became_peered":"2024-02-13T00:41:01.654410+0000","last_unstale":"2024-02-13T00:42:13.346203+0000","last_undegraded":"2024-02-13T00:42:13.346203+0000","last_fullsized":"2024-02-13T00:42:13.346203+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"91'35","reported_seq":138,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584467+0000","last_change":"2024-02-13T00:42:06.389716+0000","last_active":"2024-02-13T00:42:13.584467+0000","last_peered":"2024-02-13T00:42:13.584467+0000","last_clean":"2024-02-13T00:42:13.584467+0000","last_became_active":"2024-02-13T00:42:03.423529+0000","last_became_peered":"2024-02-13T00:42:03.423529+0000","last_unstale":"2024-02-13T00:42:13.584467+0000","last_undegraded":"2024-02-13T00:42:13.584467+0000","last_fullsized":"2024-02-13T00:42:13.584467+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:40:15.074034+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":141,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345048+0000","last_change":"2024-02-13T00:41:47.446081+0000","last_active":"2024-02-13T00:42:13.345048+0000","last_peered":"2024-02-13T00:42:13.345048+0000","last_clean":"2024-02-13T00:42:13.345048+0000","last_became_active":"2024-02-13T00:37:41.219457+0000","last_became_peered":"2024-02-13T00:37:41.219457+0000","last_unstale":"2024-02-13T00:42:13.345048+0000","last_undegraded":"2024-02-13T00:42:13.345048+0000","last_fullsized":"2024-02-13T00:42:13.345048+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:47.446012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:47.446012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":137,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342046+0000","last_change":"2024-02-13T00:41:03.411330+0000","last_active":"2024-02-13T00:42:13.342046+0000","last_peered":"2024-02-13T00:42:13.342046+0000","last_clean":"2024-02-13T00:42:13.342046+0000","last_became_active":"2024-02-13T00:37:45.239867+0000","last_became_peered":"2024-02-13T00:37:45.239867+0000","last_unstale":"2024-02-13T00:42:13.342046+0000","last_undegraded":"2024-02-13T00:42:13.342046+0000","last_fullsized":"2024-02-13T00:42:13.342046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:03.411257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_clean_scrub_stamp":"2024-02-13T00:41:03.411257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":136,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.581192+0000","last_change":"2024-02-13T00:41:55.626977+0000","last_active":"2024-02-13T00:42:13.581192+0000","last_peered":"2024-02-13T00:42:13.581192+0000","last_clean":"2024-02-13T00:42:13.581192+0000","last_became_active":"2024-02-13T00:37:47.254705+0000","last_became_peered":"2024-02-13T00:37:47.254705+0000","last_unstale":"2024-02-13T00:42:13.581192+0000","last_undegraded":"2024-02-13T00:42:13.581192+0000","last_fullsized":"2024-02-13T00:42:13.581192+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":82,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583636+0000","last_change":"2024-02-13T00:41:01.733877+0000","last_active":"2024-02-13T00:42:13.583636+0000","last_peered":"2024-02-13T00:42:13.583636+0000","last_clean":"2024-02-13T00:42:13.583636+0000","last_became_active":"2024-02-13T00:41:01.733241+0000","last_became_peered":"2024-02-13T00:41:01.733241+0000","last_unstale":"2024-02-13T00:42:13.583636+0000","last_undegraded":"2024-02-13T00:42:13.583636+0000","last_fullsized":"2024-02-13T00:42:13.583636+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":90,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583718+0000","last_change":"2024-02-13T00:42:03.414876+0000","last_active":"2024-02-13T00:42:13.583718+0000","last_peered":"2024-02-13T00:42:13.583718+0000","last_clean":"2024-02-13T00:42:13.583718+0000","last_became_active":"2024-02-13T00:42:03.414095+0000","last_became_peered":"2024-02-13T00:42:03.414095+0000","last_unstale":"2024-02-13T00:42:13.583718+0000","last_undegraded":"2024-02-13T00:42:13.583718+0000","last_fullsized":"2024-02-13T00:42:13.583718+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":140,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.344483+0000","last_change":"2024-02-13T00:40:56.434571+0000","last_active":"2024-02-13T00:42:13.344483+0000","last_peered":"2024-02-13T00:42:13.344483+0000","last_clean":"2024-02-13T00:42:13.344483+0000","last_became_active":"2024-02-13T00:40:35.850675+0000","last_became_peered":"2024-02-13T00:40:35.850675+0000","last_unstale":"2024-02-13T00:42:13.344483+0000","last_undegraded":"2024-02-13T00:42:13.344483+0000","last_fullsized":"2024-02-13T00:42:13.344483+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:40:56.434506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:56.434506+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":147,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585482+0000","last_change":"2024-02-13T00:41:00.128667+0000","last_active":"2024-02-13T00:42:13.585482+0000","last_peered":"2024-02-13T00:42:13.585482+0000","last_clean":"2024-02-13T00:42:13.585482+0000","last_became_active":"2024-02-13T00:37:41.220874+0000","last_became_peered":"2024-02-13T00:37:41.220874+0000","last_unstale":"2024-02-13T00:42:13.585482+0000","last_undegraded":"2024-02-13T00:42:13.585482+0000","last_fullsized":"2024-02-13T00:42:13.585482+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:41:00.128595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.128595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":132,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583410+0000","last_change":"2024-02-13T00:39:35.269607+0000","last_active":"2024-02-13T00:42:13.583410+0000","last_peered":"2024-02-13T00:42:13.583410+0000","last_clean":"2024-02-13T00:42:13.583410+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:42:13.583410+0000","last_undegraded":"2024-02-13T00:42:13.583410+0000","last_fullsized":"2024-02-13T00:42:13.583410+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:35.269538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:35.269538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":135,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347714+0000","last_change":"2024-02-13T00:41:02.402785+0000","last_active":"2024-02-13T00:42:13.347714+0000","last_peered":"2024-02-13T00:42:13.347714+0000","last_clean":"2024-02-13T00:42:13.347714+0000","last_became_active":"2024-02-13T00:37:47.256807+0000","last_became_peered":"2024-02-13T00:37:47.256807+0000","last_unstale":"2024-02-13T00:42:13.347714+0000","last_undegraded":"2024-02-13T00:42:13.347714+0000","last_fullsized":"2024-02-13T00:42:13.347714+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.402668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.402668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.1b","version":"108'10","reported_seq":92,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585470+0000","last_change":"2024-02-13T00:41:01.676174+0000","last_active":"2024-02-13T00:42:13.585470+0000","last_peered":"2024-02-13T00:42:13.585470+0000","last_clean":"2024-02-13T00:42:13.585470+0000","last_became_active":"2024-02-13T00:41:01.675424+0000","last_became_peered":"2024-02-13T00:41:01.675424+0000","last_unstale":"2024-02-13T00:42:13.585470+0000","last_undegraded":"2024-02-13T00:42:13.585470+0000","last_fullsized":"2024-02-13T00:42:13.585470+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.579865+0000","last_change":"2024-02-13T00:41:01.536591+0000","last_active":"2024-02-13T00:42:13.579865+0000","last_peered":"2024-02-13T00:42:13.579865+0000","last_clean":"2024-02-13T00:42:13.579865+0000","last_became_active":"2024-02-13T00:41:01.536188+0000","last_became_peered":"2024-02-13T00:41:01.536188+0000","last_unstale":"2024-02-13T00:42:13.579865+0000","last_undegraded":"2024-02-13T00:42:13.579865+0000","last_fullsized":"2024-02-13T00:42:13.579865+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":171,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584747+0000","last_change":"2024-02-13T00:42:07.304841+0000","last_active":"2024-02-13T00:42:13.584747+0000","last_peered":"2024-02-13T00:42:13.584747+0000","last_clean":"2024-02-13T00:42:13.584747+0000","last_became_active":"2024-02-13T00:42:03.421112+0000","last_became_peered":"2024-02-13T00:42:03.421112+0000","last_unstale":"2024-02-13T00:42:13.584747+0000","last_undegraded":"2024-02-13T00:42:13.584747+0000","last_fullsized":"2024-02-13T00:42:13.584747+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:40:53.104724+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:40:53.104724+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":136,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584608+0000","last_change":"2024-02-13T00:39:17.283996+0000","last_active":"2024-02-13T00:42:13.584608+0000","last_peered":"2024-02-13T00:42:13.584608+0000","last_clean":"2024-02-13T00:42:13.584608+0000","last_became_active":"2024-02-13T00:37:41.218346+0000","last_became_peered":"2024-02-13T00:37:41.218346+0000","last_unstale":"2024-02-13T00:42:13.584608+0000","last_undegraded":"2024-02-13T00:42:13.584608+0000","last_fullsized":"2024-02-13T00:42:13.584608+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:17.283902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:17.283902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"64'8","reported_seq":205,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:14.270416+0000","last_change":"2024-02-13T00:41:56.423094+0000","last_active":"2024-02-13T00:42:14.270416+0000","last_peered":"2024-02-13T00:42:14.270416+0000","last_clean":"2024-02-13T00:42:14.270416+0000","last_became_active":"2024-02-13T00:37:45.249684+0000","last_became_peered":"2024-02-13T00:37:45.249684+0000","last_unstale":"2024-02-13T00:42:14.270416+0000","last_undegraded":"2024-02-13T00:42:14.270416+0000","last_fullsized":"2024-02-13T00:42:14.270416+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:56.422996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.422996+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":109,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585932+0000","last_change":"2024-02-13T00:40:40.750384+0000","last_active":"2024-02-13T00:42:13.585932+0000","last_peered":"2024-02-13T00:42:13.585932+0000","last_clean":"2024-02-13T00:42:13.585932+0000","last_became_active":"2024-02-13T00:40:35.851908+0000","last_became_peered":"2024-02-13T00:40:35.851908+0000","last_unstale":"2024-02-13T00:42:13.585932+0000","last_undegraded":"2024-02-13T00:42:13.585932+0000","last_fullsized":"2024-02-13T00:42:13.585932+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:40:40.750272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:40.750272+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1a","version":"108'5","reported_seq":87,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586932+0000","last_change":"2024-02-13T00:41:01.743545+0000","last_active":"2024-02-13T00:42:13.586932+0000","last_peered":"2024-02-13T00:42:13.586932+0000","last_clean":"2024-02-13T00:42:13.586932+0000","last_became_active":"2024-02-13T00:41:01.739001+0000","last_became_peered":"2024-02-13T00:41:01.739001+0000","last_unstale":"2024-02-13T00:42:13.586932+0000","last_undegraded":"2024-02-13T00:42:13.586932+0000","last_fullsized":"2024-02-13T00:42:13.586932+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.587555+0000","last_change":"2024-02-13T00:41:01.632475+0000","last_active":"2024-02-13T00:42:13.587555+0000","last_peered":"2024-02-13T00:42:13.587555+0000","last_clean":"2024-02-13T00:42:13.587555+0000","last_became_active":"2024-02-13T00:41:01.632199+0000","last_became_peered":"2024-02-13T00:41:01.632199+0000","last_unstale":"2024-02-13T00:42:13.587555+0000","last_undegraded":"2024-02-13T00:42:13.587555+0000","last_fullsized":"2024-02-13T00:42:13.587555+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":159,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.588882+0000","last_change":"2024-02-13T00:39:23.370512+0000","last_active":"2024-02-13T00:42:13.588882+0000","last_peered":"2024-02-13T00:42:13.588882+0000","last_clean":"2024-02-13T00:42:13.588882+0000","last_became_active":"2024-02-13T00:37:43.232161+0000","last_became_peered":"2024-02-13T00:37:43.232161+0000","last_unstale":"2024-02-13T00:42:13.588882+0000","last_undegraded":"2024-02-13T00:42:13.588882+0000","last_fullsized":"2024-02-13T00:42:13.588882+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:39:23.370447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:23.370447+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":147,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584128+0000","last_change":"2024-02-13T00:39:52.206001+0000","last_active":"2024-02-13T00:42:13.584128+0000","last_peered":"2024-02-13T00:42:13.584128+0000","last_clean":"2024-02-13T00:42:13.584128+0000","last_became_active":"2024-02-13T00:37:41.217979+0000","last_became_peered":"2024-02-13T00:37:41.217979+0000","last_unstale":"2024-02-13T00:42:13.584128+0000","last_undegraded":"2024-02-13T00:42:13.584128+0000","last_fullsized":"2024-02-13T00:42:13.584128+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:39:52.205926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:52.205926+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"96'9","reported_seq":135,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:14.270507+0000","last_change":"2024-02-13T00:40:35.847666+0000","last_active":"2024-02-13T00:42:14.270507+0000","last_peered":"2024-02-13T00:42:14.270507+0000","last_clean":"2024-02-13T00:42:14.270507+0000","last_became_active":"2024-02-13T00:40:35.847293+0000","last_became_peered":"2024-02-13T00:40:35.847293+0000","last_unstale":"2024-02-13T00:42:14.270507+0000","last_undegraded":"2024-02-13T00:42:14.270507+0000","last_fullsized":"2024-02-13T00:42:14.270507+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:40:02.089864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:02.089864+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":134,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345690+0000","last_change":"2024-02-13T00:40:58.541216+0000","last_active":"2024-02-13T00:42:13.345690+0000","last_peered":"2024-02-13T00:42:13.345690+0000","last_clean":"2024-02-13T00:42:13.345690+0000","last_became_active":"2024-02-13T00:37:47.260477+0000","last_became_peered":"2024-02-13T00:37:47.260477+0000","last_unstale":"2024-02-13T00:42:13.345690+0000","last_undegraded":"2024-02-13T00:42:13.345690+0000","last_fullsized":"2024-02-13T00:42:13.345690+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.541144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":82,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582793+0000","last_change":"2024-02-13T00:41:01.596916+0000","last_active":"2024-02-13T00:42:13.582793+0000","last_peered":"2024-02-13T00:42:13.582793+0000","last_clean":"2024-02-13T00:42:13.582793+0000","last_became_active":"2024-02-13T00:41:01.594299+0000","last_became_peered":"2024-02-13T00:41:01.594299+0000","last_unstale":"2024-02-13T00:42:13.582793+0000","last_undegraded":"2024-02-13T00:42:13.582793+0000","last_fullsized":"2024-02-13T00:42:13.582793+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.351435+0000","last_change":"2024-02-13T00:41:01.747617+0000","last_active":"2024-02-13T00:42:13.351435+0000","last_peered":"2024-02-13T00:42:13.351435+0000","last_clean":"2024-02-13T00:42:13.351435+0000","last_became_active":"2024-02-13T00:41:01.746476+0000","last_became_peered":"2024-02-13T00:41:01.746476+0000","last_unstale":"2024-02-13T00:42:13.351435+0000","last_undegraded":"2024-02-13T00:42:13.351435+0000","last_fullsized":"2024-02-13T00:42:13.351435+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":84,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585280+0000","last_change":"2024-02-13T00:42:06.684352+0000","last_active":"2024-02-13T00:42:13.585280+0000","last_peered":"2024-02-13T00:42:13.585280+0000","last_clean":"2024-02-13T00:42:13.585280+0000","last_became_active":"2024-02-13T00:42:03.424601+0000","last_became_peered":"2024-02-13T00:42:03.424601+0000","last_unstale":"2024-02-13T00:42:13.585280+0000","last_undegraded":"2024-02-13T00:42:13.585280+0000","last_fullsized":"2024-02-13T00:42:13.585280+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:39:29.275134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:29.275134+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":140,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.581387+0000","last_change":"2024-02-13T00:41:32.399538+0000","last_active":"2024-02-13T00:42:13.581387+0000","last_peered":"2024-02-13T00:42:13.581387+0000","last_clean":"2024-02-13T00:42:13.581387+0000","last_became_active":"2024-02-13T00:37:41.221627+0000","last_became_peered":"2024-02-13T00:37:41.221627+0000","last_unstale":"2024-02-13T00:42:13.581387+0000","last_undegraded":"2024-02-13T00:42:13.581387+0000","last_fullsized":"2024-02-13T00:42:13.581387+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:32.399475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.399475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":145,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.349197+0000","last_change":"2024-02-13T00:42:03.396094+0000","last_active":"2024-02-13T00:42:13.349197+0000","last_peered":"2024-02-13T00:42:13.349197+0000","last_clean":"2024-02-13T00:42:13.349197+0000","last_became_active":"2024-02-13T00:42:03.395752+0000","last_became_peered":"2024-02-13T00:42:03.395752+0000","last_unstale":"2024-02-13T00:42:13.349197+0000","last_undegraded":"2024-02-13T00:42:13.349197+0000","last_fullsized":"2024-02-13T00:42:13.349197+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:48.518493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:40:48.518493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":143,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584342+0000","last_change":"2024-02-13T00:42:03.428677+0000","last_active":"2024-02-13T00:42:13.584342+0000","last_peered":"2024-02-13T00:42:13.584342+0000","last_clean":"2024-02-13T00:42:13.584342+0000","last_became_active":"2024-02-13T00:42:03.428380+0000","last_became_peered":"2024-02-13T00:42:03.428380+0000","last_unstale":"2024-02-13T00:42:13.584342+0000","last_undegraded":"2024-02-13T00:42:13.584342+0000","last_fullsized":"2024-02-13T00:42:13.584342+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:14.015547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:14.015547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.18","version":"107'2","reported_seq":97,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586990+0000","last_change":"2024-02-13T00:42:08.249193+0000","last_active":"2024-02-13T00:42:13.586990+0000","last_peered":"2024-02-13T00:42:13.586990+0000","last_clean":"2024-02-13T00:42:13.586990+0000","last_became_active":"2024-02-13T00:42:03.430784+0000","last_became_peered":"2024-02-13T00:42:03.430784+0000","last_unstale":"2024-02-13T00:42:13.586990+0000","last_undegraded":"2024-02-13T00:42:13.586990+0000","last_fullsized":"2024-02-13T00:42:13.586990+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342842+0000","last_change":"2024-02-13T00:41:01.583063+0000","last_active":"2024-02-13T00:42:13.342842+0000","last_peered":"2024-02-13T00:42:13.342842+0000","last_clean":"2024-02-13T00:42:13.342842+0000","last_became_active":"2024-02-13T00:41:01.582361+0000","last_became_peered":"2024-02-13T00:41:01.582361+0000","last_unstale":"2024-02-13T00:42:13.342842+0000","last_undegraded":"2024-02-13T00:42:13.342842+0000","last_fullsized":"2024-02-13T00:42:13.342842+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"149'9","reported_seq":202,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:14.270534+0000","last_change":"2024-02-13T00:41:44.698967+0000","last_active":"2024-02-13T00:42:14.270534+0000","last_peered":"2024-02-13T00:42:14.270534+0000","last_clean":"2024-02-13T00:42:14.270534+0000","last_became_active":"2024-02-13T00:37:45.245528+0000","last_became_peered":"2024-02-13T00:37:45.245528+0000","last_unstale":"2024-02-13T00:42:14.270534+0000","last_undegraded":"2024-02-13T00:42:14.270534+0000","last_fullsized":"2024-02-13T00:42:14.270534+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'9","last_scrub_stamp":"2024-02-13T00:41:44.698906+0000","last_deep_scrub":"149'9","last_deep_scrub_stamp":"2024-02-13T00:41:44.698906+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.698906+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":176,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342764+0000","last_change":"2024-02-13T00:42:04.162915+0000","last_active":"2024-02-13T00:42:13.342764+0000","last_peered":"2024-02-13T00:42:13.342764+0000","last_clean":"2024-02-13T00:42:13.342764+0000","last_became_active":"2024-02-13T00:42:03.393132+0000","last_became_peered":"2024-02-13T00:42:03.393132+0000","last_unstale":"2024-02-13T00:42:13.342764+0000","last_undegraded":"2024-02-13T00:42:13.342764+0000","last_fullsized":"2024-02-13T00:42:13.342764+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2024-02-13T00:40:43.630535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:43.630535+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.587167+0000","last_change":"2024-02-13T00:42:03.433157+0000","last_active":"2024-02-13T00:42:13.587167+0000","last_peered":"2024-02-13T00:42:13.587167+0000","last_clean":"2024-02-13T00:42:13.587167+0000","last_became_active":"2024-02-13T00:42:03.432782+0000","last_became_peered":"2024-02-13T00:42:03.432782+0000","last_unstale":"2024-02-13T00:42:13.587167+0000","last_undegraded":"2024-02-13T00:42:13.587167+0000","last_fullsized":"2024-02-13T00:42:13.587167+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:46.274423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:46.274423+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":143,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585647+0000","last_change":"2024-02-13T00:42:03.423590+0000","last_active":"2024-02-13T00:42:13.585647+0000","last_peered":"2024-02-13T00:42:13.585647+0000","last_clean":"2024-02-13T00:42:13.585647+0000","last_became_active":"2024-02-13T00:42:03.423211+0000","last_became_peered":"2024-02-13T00:42:03.423211+0000","last_unstale":"2024-02-13T00:42:13.585647+0000","last_undegraded":"2024-02-13T00:42:13.585647+0000","last_fullsized":"2024-02-13T00:42:13.585647+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:44.242946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.242946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.17","version":"108'5","reported_seq":87,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.350141+0000","last_change":"2024-02-13T00:41:01.623530+0000","last_active":"2024-02-13T00:42:13.350141+0000","last_peered":"2024-02-13T00:42:13.350141+0000","last_clean":"2024-02-13T00:42:13.350141+0000","last_became_active":"2024-02-13T00:41:01.622857+0000","last_became_peered":"2024-02-13T00:41:01.622857+0000","last_unstale":"2024-02-13T00:42:13.350141+0000","last_undegraded":"2024-02-13T00:42:13.350141+0000","last_fullsized":"2024-02-13T00:42:13.350141+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582391+0000","last_change":"2024-02-13T00:41:01.682441+0000","last_active":"2024-02-13T00:42:13.582391+0000","last_peered":"2024-02-13T00:42:13.582391+0000","last_clean":"2024-02-13T00:42:13.582391+0000","last_became_active":"2024-02-13T00:41:01.682162+0000","last_became_peered":"2024-02-13T00:41:01.682162+0000","last_unstale":"2024-02-13T00:42:13.582391+0000","last_undegraded":"2024-02-13T00:42:13.582391+0000","last_fullsized":"2024-02-13T00:42:13.582391+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":100,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582502+0000","last_change":"2024-02-13T00:40:43.011812+0000","last_active":"2024-02-13T00:42:13.582502+0000","last_peered":"2024-02-13T00:42:13.582502+0000","last_clean":"2024-02-13T00:42:13.582502+0000","last_became_active":"2024-02-13T00:40:35.856681+0000","last_became_peered":"2024-02-13T00:40:35.856681+0000","last_unstale":"2024-02-13T00:42:13.582502+0000","last_undegraded":"2024-02-13T00:42:13.582502+0000","last_fullsized":"2024-02-13T00:42:13.582502+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:43.011693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:43.011693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":200,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582310+0000","last_change":"2024-02-13T00:40:47.034719+0000","last_active":"2024-02-13T00:42:13.582310+0000","last_peered":"2024-02-13T00:42:13.582310+0000","last_clean":"2024-02-13T00:42:13.582310+0000","last_became_active":"2024-02-13T00:37:43.324275+0000","last_became_peered":"2024-02-13T00:37:43.324275+0000","last_unstale":"2024-02-13T00:42:13.582310+0000","last_undegraded":"2024-02-13T00:42:13.582310+0000","last_fullsized":"2024-02-13T00:42:13.582310+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:40:47.034615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:47.034615+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":150,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585082+0000","last_change":"2024-02-13T00:42:03.402563+0000","last_active":"2024-02-13T00:42:13.585082+0000","last_peered":"2024-02-13T00:42:13.585082+0000","last_clean":"2024-02-13T00:42:13.585082+0000","last_became_active":"2024-02-13T00:42:03.402206+0000","last_became_peered":"2024-02-13T00:42:03.402206+0000","last_unstale":"2024-02-13T00:42:13.585082+0000","last_undegraded":"2024-02-13T00:42:13.585082+0000","last_fullsized":"2024-02-13T00:42:13.585082+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.236727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.236727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":100,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586705+0000","last_change":"2024-02-13T00:41:02.817359+0000","last_active":"2024-02-13T00:42:13.586705+0000","last_peered":"2024-02-13T00:42:13.586705+0000","last_clean":"2024-02-13T00:42:13.586705+0000","last_became_active":"2024-02-13T00:40:35.857936+0000","last_became_peered":"2024-02-13T00:40:35.857936+0000","last_unstale":"2024-02-13T00:42:13.586705+0000","last_undegraded":"2024-02-13T00:42:13.586705+0000","last_fullsized":"2024-02-13T00:42:13.586705+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.16","version":"108'5","reported_seq":100,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.588286+0000","last_change":"2024-02-13T00:42:06.795504+0000","last_active":"2024-02-13T00:42:13.588286+0000","last_peered":"2024-02-13T00:42:13.588286+0000","last_clean":"2024-02-13T00:42:13.588286+0000","last_became_active":"2024-02-13T00:42:03.415083+0000","last_became_peered":"2024-02-13T00:42:03.415083+0000","last_unstale":"2024-02-13T00:42:13.588286+0000","last_undegraded":"2024-02-13T00:42:13.588286+0000","last_fullsized":"2024-02-13T00:42:13.588286+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":90,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342661+0000","last_change":"2024-02-13T00:42:03.375192+0000","last_active":"2024-02-13T00:42:13.342661+0000","last_peered":"2024-02-13T00:42:13.342661+0000","last_clean":"2024-02-13T00:42:13.342661+0000","last_became_active":"2024-02-13T00:42:03.374865+0000","last_became_peered":"2024-02-13T00:42:03.374865+0000","last_unstale":"2024-02-13T00:42:13.342661+0000","last_undegraded":"2024-02-13T00:42:13.342661+0000","last_fullsized":"2024-02-13T00:42:13.342661+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.344935+0000","last_change":"2024-02-13T00:42:03.387052+0000","last_active":"2024-02-13T00:42:13.344935+0000","last_peered":"2024-02-13T00:42:13.344935+0000","last_clean":"2024-02-13T00:42:13.344935+0000","last_became_active":"2024-02-13T00:42:03.386776+0000","last_became_peered":"2024-02-13T00:42:03.386776+0000","last_unstale":"2024-02-13T00:42:13.344935+0000","last_undegraded":"2024-02-13T00:42:13.344935+0000","last_fullsized":"2024-02-13T00:42:13.344935+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:55.358367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:55.358367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":193,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.344839+0000","last_change":"2024-02-13T00:39:04.318990+0000","last_active":"2024-02-13T00:42:13.344839+0000","last_peered":"2024-02-13T00:42:13.344839+0000","last_clean":"2024-02-13T00:42:13.344839+0000","last_became_active":"2024-02-13T00:37:43.322450+0000","last_became_peered":"2024-02-13T00:37:43.322450+0000","last_unstale":"2024-02-13T00:42:13.344839+0000","last_undegraded":"2024-02-13T00:42:13.344839+0000","last_fullsized":"2024-02-13T00:42:13.344839+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:39:04.318923+0000","last_clean_scrub_stamp":"2024-02-13T00:39:04.318923+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":140,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.587847+0000","last_change":"2024-02-13T00:40:45.198328+0000","last_active":"2024-02-13T00:42:13.587847+0000","last_peered":"2024-02-13T00:42:13.587847+0000","last_clean":"2024-02-13T00:42:13.587847+0000","last_became_active":"2024-02-13T00:37:41.221636+0000","last_became_peered":"2024-02-13T00:37:41.221636+0000","last_unstale":"2024-02-13T00:42:13.587847+0000","last_undegraded":"2024-02-13T00:42:13.587847+0000","last_fullsized":"2024-02-13T00:42:13.587847+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:45.198231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:45.198231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":139,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584276+0000","last_change":"2024-02-13T00:42:03.422683+0000","last_active":"2024-02-13T00:42:13.584276+0000","last_peered":"2024-02-13T00:42:13.584276+0000","last_clean":"2024-02-13T00:42:13.584276+0000","last_became_active":"2024-02-13T00:42:03.422365+0000","last_became_peered":"2024-02-13T00:42:03.422365+0000","last_unstale":"2024-02-13T00:42:13.584276+0000","last_undegraded":"2024-02-13T00:42:13.584276+0000","last_fullsized":"2024-02-13T00:42:13.584276+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:33.303982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:33.303982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":82,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345833+0000","last_change":"2024-02-13T00:41:01.672342+0000","last_active":"2024-02-13T00:42:13.345833+0000","last_peered":"2024-02-13T00:42:13.345833+0000","last_clean":"2024-02-13T00:42:13.345833+0000","last_became_active":"2024-02-13T00:41:01.671979+0000","last_became_peered":"2024-02-13T00:41:01.671979+0000","last_unstale":"2024-02-13T00:42:13.345833+0000","last_undegraded":"2024-02-13T00:42:13.345833+0000","last_fullsized":"2024-02-13T00:42:13.345833+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345186+0000","last_change":"2024-02-13T00:41:01.590710+0000","last_active":"2024-02-13T00:42:13.345186+0000","last_peered":"2024-02-13T00:42:13.345186+0000","last_clean":"2024-02-13T00:42:13.345186+0000","last_became_active":"2024-02-13T00:41:01.590438+0000","last_became_peered":"2024-02-13T00:41:01.590438+0000","last_unstale":"2024-02-13T00:42:13.345186+0000","last_undegraded":"2024-02-13T00:42:13.345186+0000","last_fullsized":"2024-02-13T00:42:13.345186+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":141,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.349370+0000","last_change":"2024-02-13T00:41:54.422549+0000","last_active":"2024-02-13T00:42:13.349370+0000","last_peered":"2024-02-13T00:42:13.349370+0000","last_clean":"2024-02-13T00:42:13.349370+0000","last_became_active":"2024-02-13T00:37:45.245956+0000","last_became_peered":"2024-02-13T00:37:45.245956+0000","last_unstale":"2024-02-13T00:42:13.349370+0000","last_undegraded":"2024-02-13T00:42:13.349370+0000","last_fullsized":"2024-02-13T00:42:13.349370+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:54.422484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:54.422484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":179,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.350222+0000","last_change":"2024-02-13T00:41:52.739904+0000","last_active":"2024-02-13T00:42:13.350222+0000","last_peered":"2024-02-13T00:42:13.350222+0000","last_clean":"2024-02-13T00:42:13.350222+0000","last_became_active":"2024-02-13T00:39:05.968155+0000","last_became_peered":"2024-02-13T00:39:05.968155+0000","last_unstale":"2024-02-13T00:42:13.350222+0000","last_undegraded":"2024-02-13T00:42:13.350222+0000","last_fullsized":"2024-02-13T00:42:13.350222+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:41:52.739833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.739833+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":117,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.341960+0000","last_change":"2024-02-13T00:41:54.457692+0000","last_active":"2024-02-13T00:42:13.341960+0000","last_peered":"2024-02-13T00:42:13.341960+0000","last_clean":"2024-02-13T00:42:13.341960+0000","last_became_active":"2024-02-13T00:39:05.967772+0000","last_became_peered":"2024-02-13T00:39:05.967772+0000","last_unstale":"2024-02-13T00:42:13.341960+0000","last_undegraded":"2024-02-13T00:42:13.341960+0000","last_fullsized":"2024-02-13T00:42:13.341960+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:54.457605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:54.457605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":140,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347455+0000","last_change":"2024-02-13T00:41:34.352119+0000","last_active":"2024-02-13T00:42:13.347455+0000","last_peered":"2024-02-13T00:42:13.347455+0000","last_clean":"2024-02-13T00:42:13.347455+0000","last_became_active":"2024-02-13T00:37:41.217577+0000","last_became_peered":"2024-02-13T00:37:41.217577+0000","last_unstale":"2024-02-13T00:42:13.347455+0000","last_undegraded":"2024-02-13T00:42:13.347455+0000","last_fullsized":"2024-02-13T00:42:13.347455+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.352030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.352030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":130,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583494+0000","last_change":"2024-02-13T00:39:32.255327+0000","last_active":"2024-02-13T00:42:13.583494+0000","last_peered":"2024-02-13T00:42:13.583494+0000","last_clean":"2024-02-13T00:42:13.583494+0000","last_became_active":"2024-02-13T00:37:47.262024+0000","last_became_peered":"2024-02-13T00:37:47.262024+0000","last_unstale":"2024-02-13T00:42:13.583494+0000","last_undegraded":"2024-02-13T00:42:13.583494+0000","last_fullsized":"2024-02-13T00:42:13.583494+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.255260+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.255260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":24,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582615+0000","last_change":"2024-02-13T00:42:03.436077+0000","last_active":"2024-02-13T00:42:13.582615+0000","last_peered":"2024-02-13T00:42:13.582615+0000","last_clean":"2024-02-13T00:42:13.582615+0000","last_became_active":"2024-02-13T00:42:03.434658+0000","last_became_peered":"2024-02-13T00:42:03.434658+0000","last_unstale":"2024-02-13T00:42:13.582615+0000","last_undegraded":"2024-02-13T00:42:13.582615+0000","last_fullsized":"2024-02-13T00:42:13.582615+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":90,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345915+0000","last_change":"2024-02-13T00:42:03.399569+0000","last_active":"2024-02-13T00:42:13.345915+0000","last_peered":"2024-02-13T00:42:13.345915+0000","last_clean":"2024-02-13T00:42:13.345915+0000","last_became_active":"2024-02-13T00:42:03.398892+0000","last_became_peered":"2024-02-13T00:42:03.398892+0000","last_unstale":"2024-02-13T00:42:13.345915+0000","last_undegraded":"2024-02-13T00:42:13.345915+0000","last_fullsized":"2024-02-13T00:42:13.345915+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":136,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584451+0000","last_change":"2024-02-13T00:41:56.666094+0000","last_active":"2024-02-13T00:42:13.584451+0000","last_peered":"2024-02-13T00:42:13.584451+0000","last_clean":"2024-02-13T00:42:13.584451+0000","last_became_active":"2024-02-13T00:37:45.247708+0000","last_became_peered":"2024-02-13T00:37:45.247708+0000","last_unstale":"2024-02-13T00:42:13.584451+0000","last_undegraded":"2024-02-13T00:42:13.584451+0000","last_fullsized":"2024-02-13T00:42:13.584451+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:56.666003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.666003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"157'21","reported_seq":174,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.587919+0000","last_change":"2024-02-13T00:42:06.705359+0000","last_active":"2024-02-13T00:42:13.587919+0000","last_peered":"2024-02-13T00:42:13.587919+0000","last_clean":"2024-02-13T00:42:13.587919+0000","last_became_active":"2024-02-13T00:42:03.412812+0000","last_became_peered":"2024-02-13T00:42:03.412812+0000","last_unstale":"2024-02-13T00:42:13.587919+0000","last_undegraded":"2024-02-13T00:42:13.587919+0000","last_fullsized":"2024-02-13T00:42:13.587919+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.415907+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":152,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.581263+0000","last_change":"2024-02-13T00:42:03.409142+0000","last_active":"2024-02-13T00:42:13.581263+0000","last_peered":"2024-02-13T00:42:13.581263+0000","last_clean":"2024-02-13T00:42:13.581263+0000","last_became_active":"2024-02-13T00:42:03.408774+0000","last_became_peered":"2024-02-13T00:42:03.408774+0000","last_unstale":"2024-02-13T00:42:13.581263+0000","last_undegraded":"2024-02-13T00:42:13.581263+0000","last_fullsized":"2024-02-13T00:42:13.581263+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:39:41.255785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":142,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346609+0000","last_change":"2024-02-13T00:41:34.472385+0000","last_active":"2024-02-13T00:42:13.346609+0000","last_peered":"2024-02-13T00:42:13.346609+0000","last_clean":"2024-02-13T00:42:13.346609+0000","last_became_active":"2024-02-13T00:37:41.216101+0000","last_became_peered":"2024-02-13T00:37:41.216101+0000","last_unstale":"2024-02-13T00:42:13.346609+0000","last_undegraded":"2024-02-13T00:42:13.346609+0000","last_fullsized":"2024-02-13T00:42:13.346609+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.472312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.472312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":149,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586790+0000","last_change":"2024-02-13T00:42:03.417355+0000","last_active":"2024-02-13T00:42:13.586790+0000","last_peered":"2024-02-13T00:42:13.586790+0000","last_clean":"2024-02-13T00:42:13.586790+0000","last_became_active":"2024-02-13T00:42:03.416523+0000","last_became_peered":"2024-02-13T00:42:03.416523+0000","last_unstale":"2024-02-13T00:42:13.586790+0000","last_undegraded":"2024-02-13T00:42:13.586790+0000","last_fullsized":"2024-02-13T00:42:13.586790+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:09.099460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:40:09.099460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":139,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.349804+0000","last_change":"2024-02-13T00:41:51.414166+0000","last_active":"2024-02-13T00:42:13.349804+0000","last_peered":"2024-02-13T00:42:13.349804+0000","last_clean":"2024-02-13T00:42:13.349804+0000","last_became_active":"2024-02-13T00:37:47.253996+0000","last_became_peered":"2024-02-13T00:37:47.253996+0000","last_unstale":"2024-02-13T00:42:13.349804+0000","last_undegraded":"2024-02-13T00:42:13.349804+0000","last_fullsized":"2024-02-13T00:42:13.349804+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.13","version":"108'9","reported_seq":91,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.579292+0000","last_change":"2024-02-13T00:41:01.560685+0000","last_active":"2024-02-13T00:42:13.579292+0000","last_peered":"2024-02-13T00:42:13.579292+0000","last_clean":"2024-02-13T00:42:13.579292+0000","last_became_active":"2024-02-13T00:41:01.560473+0000","last_became_peered":"2024-02-13T00:41:01.560473+0000","last_unstale":"2024-02-13T00:42:13.579292+0000","last_undegraded":"2024-02-13T00:42:13.579292+0000","last_fullsized":"2024-02-13T00:42:13.579292+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.352698+0000","last_change":"2024-02-13T00:41:01.594433+0000","last_active":"2024-02-13T00:42:13.352698+0000","last_peered":"2024-02-13T00:42:13.352698+0000","last_clean":"2024-02-13T00:42:13.352698+0000","last_became_active":"2024-02-13T00:41:01.593970+0000","last_became_peered":"2024-02-13T00:41:01.593970+0000","last_unstale":"2024-02-13T00:42:13.352698+0000","last_undegraded":"2024-02-13T00:42:13.352698+0000","last_fullsized":"2024-02-13T00:42:13.352698+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":137,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345297+0000","last_change":"2024-02-13T00:41:01.412120+0000","last_active":"2024-02-13T00:42:13.345297+0000","last_peered":"2024-02-13T00:42:13.345297+0000","last_clean":"2024-02-13T00:42:13.345297+0000","last_became_active":"2024-02-13T00:37:45.248383+0000","last_became_peered":"2024-02-13T00:37:45.248383+0000","last_unstale":"2024-02-13T00:42:13.345297+0000","last_undegraded":"2024-02-13T00:42:13.345297+0000","last_fullsized":"2024-02-13T00:42:13.345297+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:01.412042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:01.412042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":176,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.589073+0000","last_change":"2024-02-13T00:39:25.319173+0000","last_active":"2024-02-13T00:42:13.589073+0000","last_peered":"2024-02-13T00:42:13.589073+0000","last_clean":"2024-02-13T00:42:13.589073+0000","last_became_active":"2024-02-13T00:37:43.231120+0000","last_became_peered":"2024-02-13T00:37:43.231120+0000","last_unstale":"2024-02-13T00:42:13.589073+0000","last_undegraded":"2024-02-13T00:42:13.589073+0000","last_fullsized":"2024-02-13T00:42:13.589073+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:39:25.319066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:25.319066+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":141,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.343759+0000","last_change":"2024-02-13T00:40:57.469853+0000","last_active":"2024-02-13T00:42:13.343759+0000","last_peered":"2024-02-13T00:42:13.343759+0000","last_clean":"2024-02-13T00:42:13.343759+0000","last_became_active":"2024-02-13T00:37:41.214442+0000","last_became_peered":"2024-02-13T00:37:41.214442+0000","last_unstale":"2024-02-13T00:42:13.343759+0000","last_undegraded":"2024-02-13T00:42:13.343759+0000","last_fullsized":"2024-02-13T00:42:13.343759+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.469743+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":425,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-02-13T00:42:14.364628+0000","last_change":"2024-02-13T00:42:08.427157+0000","last_active":"2024-02-13T00:42:14.364628+0000","last_peered":"2024-02-13T00:42:14.364628+0000","last_clean":"2024-02-13T00:42:14.364628+0000","last_became_active":"2024-02-13T00:42:03.428394+0000","last_became_peered":"2024-02-13T00:42:03.428394+0000","last_unstale":"2024-02-13T00:42:14.364628+0000","last_undegraded":"2024-02-13T00:42:14.364628+0000","last_fullsized":"2024-02-13T00:42:14.364628+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-13T00:41:35.787179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.787179+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":265,"num_read_kb":260,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":145,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.587768+0000","last_change":"2024-02-13T00:42:03.442573+0000","last_active":"2024-02-13T00:42:13.587768+0000","last_peered":"2024-02-13T00:42:13.587768+0000","last_clean":"2024-02-13T00:42:13.587768+0000","last_became_active":"2024-02-13T00:42:03.442318+0000","last_became_peered":"2024-02-13T00:42:03.442318+0000","last_unstale":"2024-02-13T00:42:13.587768+0000","last_undegraded":"2024-02-13T00:42:13.587768+0000","last_fullsized":"2024-02-13T00:42:13.587768+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.11","version":"108'7","reported_seq":89,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584421+0000","last_change":"2024-02-13T00:41:01.721381+0000","last_active":"2024-02-13T00:42:13.584421+0000","last_peered":"2024-02-13T00:42:13.584421+0000","last_clean":"2024-02-13T00:42:13.584421+0000","last_became_active":"2024-02-13T00:41:01.718821+0000","last_became_peered":"2024-02-13T00:41:01.718821+0000","last_unstale":"2024-02-13T00:42:13.584421+0000","last_undegraded":"2024-02-13T00:42:13.584421+0000","last_fullsized":"2024-02-13T00:42:13.584421+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.579755+0000","last_change":"2024-02-13T00:41:01.548873+0000","last_active":"2024-02-13T00:42:13.579755+0000","last_peered":"2024-02-13T00:42:13.579755+0000","last_clean":"2024-02-13T00:42:13.579755+0000","last_became_active":"2024-02-13T00:41:01.547408+0000","last_became_peered":"2024-02-13T00:41:01.547408+0000","last_unstale":"2024-02-13T00:42:13.579755+0000","last_undegraded":"2024-02-13T00:42:13.579755+0000","last_fullsized":"2024-02-13T00:42:13.579755+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":136,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346722+0000","last_change":"2024-02-13T00:41:00.881481+0000","last_active":"2024-02-13T00:42:13.346722+0000","last_peered":"2024-02-13T00:42:13.346722+0000","last_clean":"2024-02-13T00:42:13.346722+0000","last_became_active":"2024-02-13T00:37:45.247012+0000","last_became_peered":"2024-02-13T00:37:45.247012+0000","last_unstale":"2024-02-13T00:42:13.346722+0000","last_undegraded":"2024-02-13T00:42:13.346722+0000","last_fullsized":"2024-02-13T00:42:13.346722+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.881411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.881411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"80'45","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:14.270415+0000","last_change":"2024-02-13T00:40:12.047980+0000","last_active":"2024-02-13T00:42:14.270415+0000","last_peered":"2024-02-13T00:42:14.270415+0000","last_clean":"2024-02-13T00:42:14.270415+0000","last_became_active":"2024-02-13T00:37:43.232014+0000","last_became_peered":"2024-02-13T00:37:43.232014+0000","last_unstale":"2024-02-13T00:42:14.270415+0000","last_undegraded":"2024-02-13T00:42:14.270415+0000","last_fullsized":"2024-02-13T00:42:14.270415+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'45","last_scrub_stamp":"2024-02-13T00:40:12.047869+0000","last_deep_scrub":"80'45","last_deep_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_clean_scrub_stamp":"2024-02-13T00:40:12.047869+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":149,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.348991+0000","last_change":"2024-02-13T00:42:03.394450+0000","last_active":"2024-02-13T00:42:13.348991+0000","last_peered":"2024-02-13T00:42:13.348991+0000","last_clean":"2024-02-13T00:42:13.348991+0000","last_became_active":"2024-02-13T00:42:03.394072+0000","last_became_peered":"2024-02-13T00:42:03.394072+0000","last_unstale":"2024-02-13T00:42:13.348991+0000","last_undegraded":"2024-02-13T00:42:13.348991+0000","last_fullsized":"2024-02-13T00:42:13.348991+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.518778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.518778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":129,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.588064+0000","last_change":"2024-02-13T00:39:36.157777+0000","last_active":"2024-02-13T00:42:13.588064+0000","last_peered":"2024-02-13T00:42:13.588064+0000","last_clean":"2024-02-13T00:42:13.588064+0000","last_became_active":"2024-02-13T00:37:47.252270+0000","last_became_peered":"2024-02-13T00:37:47.252270+0000","last_unstale":"2024-02-13T00:42:13.588064+0000","last_undegraded":"2024-02-13T00:42:13.588064+0000","last_fullsized":"2024-02-13T00:42:13.588064+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:36.157693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:36.157693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":82,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342636+0000","last_change":"2024-02-13T00:41:01.606464+0000","last_active":"2024-02-13T00:42:13.342636+0000","last_peered":"2024-02-13T00:42:13.342636+0000","last_clean":"2024-02-13T00:42:13.342636+0000","last_became_active":"2024-02-13T00:41:01.605731+0000","last_became_peered":"2024-02-13T00:41:01.605731+0000","last_unstale":"2024-02-13T00:42:13.342636+0000","last_undegraded":"2024-02-13T00:42:13.342636+0000","last_fullsized":"2024-02-13T00:42:13.342636+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342315+0000","last_change":"2024-02-13T00:42:03.392457+0000","last_active":"2024-02-13T00:42:13.342315+0000","last_peered":"2024-02-13T00:42:13.342315+0000","last_clean":"2024-02-13T00:42:13.342315+0000","last_became_active":"2024-02-13T00:42:03.392221+0000","last_became_peered":"2024-02-13T00:42:03.392221+0000","last_unstale":"2024-02-13T00:42:13.342315+0000","last_undegraded":"2024-02-13T00:42:13.342315+0000","last_fullsized":"2024-02-13T00:42:13.342315+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"168'10","reported_seq":208,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:14.270416+0000","last_change":"2024-02-13T00:42:03.532514+0000","last_active":"2024-02-13T00:42:14.270416+0000","last_peered":"2024-02-13T00:42:14.270416+0000","last_clean":"2024-02-13T00:42:14.270416+0000","last_became_active":"2024-02-13T00:42:03.405246+0000","last_became_peered":"2024-02-13T00:42:03.405246+0000","last_unstale":"2024-02-13T00:42:14.270416+0000","last_undegraded":"2024-02-13T00:42:14.270416+0000","last_fullsized":"2024-02-13T00:42:14.270416+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:39:30.273165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:30.273165+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":131,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342553+0000","last_change":"2024-02-13T00:41:52.517499+0000","last_active":"2024-02-13T00:42:13.342553+0000","last_peered":"2024-02-13T00:42:13.342553+0000","last_clean":"2024-02-13T00:42:13.342553+0000","last_became_active":"2024-02-13T00:39:05.965560+0000","last_became_peered":"2024-02-13T00:39:05.965560+0000","last_unstale":"2024-02-13T00:42:13.342553+0000","last_undegraded":"2024-02-13T00:42:13.342553+0000","last_fullsized":"2024-02-13T00:42:13.342553+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:52.517392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.517392+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":141,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.581378+0000","last_change":"2024-02-13T00:40:08.946462+0000","last_active":"2024-02-13T00:42:13.581378+0000","last_peered":"2024-02-13T00:42:13.581378+0000","last_clean":"2024-02-13T00:42:13.581378+0000","last_became_active":"2024-02-13T00:37:41.217777+0000","last_became_peered":"2024-02-13T00:37:41.217777+0000","last_unstale":"2024-02-13T00:42:13.581378+0000","last_undegraded":"2024-02-13T00:42:13.581378+0000","last_fullsized":"2024-02-13T00:42:13.581378+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:08.946388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:40:08.946388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":134,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.350082+0000","last_change":"2024-02-13T00:41:48.645080+0000","last_active":"2024-02-13T00:42:13.350082+0000","last_peered":"2024-02-13T00:42:13.350082+0000","last_clean":"2024-02-13T00:42:13.350082+0000","last_became_active":"2024-02-13T00:37:47.255928+0000","last_became_peered":"2024-02-13T00:37:47.255928+0000","last_unstale":"2024-02-13T00:42:13.350082+0000","last_undegraded":"2024-02-13T00:42:13.350082+0000","last_fullsized":"2024-02-13T00:42:13.350082+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"107'2","reported_seq":37,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584222+0000","last_change":"2024-02-13T00:42:04.368431+0000","last_active":"2024-02-13T00:42:13.584222+0000","last_peered":"2024-02-13T00:42:13.584222+0000","last_clean":"2024-02-13T00:42:13.584222+0000","last_became_active":"2024-02-13T00:42:03.411926+0000","last_became_peered":"2024-02-13T00:42:03.411926+0000","last_unstale":"2024-02-13T00:42:13.584222+0000","last_undegraded":"2024-02-13T00:42:13.584222+0000","last_fullsized":"2024-02-13T00:42:13.584222+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583969+0000","last_change":"2024-02-13T00:41:01.721182+0000","last_active":"2024-02-13T00:42:13.583969+0000","last_peered":"2024-02-13T00:42:13.583969+0000","last_clean":"2024-02-13T00:42:13.583969+0000","last_became_active":"2024-02-13T00:41:01.718793+0000","last_became_peered":"2024-02-13T00:41:01.718793+0000","last_unstale":"2024-02-13T00:42:13.583969+0000","last_undegraded":"2024-02-13T00:42:13.583969+0000","last_fullsized":"2024-02-13T00:42:13.583969+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":139,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.341866+0000","last_change":"2024-02-13T00:41:43.427096+0000","last_active":"2024-02-13T00:42:13.341866+0000","last_peered":"2024-02-13T00:42:13.341866+0000","last_clean":"2024-02-13T00:42:13.341866+0000","last_became_active":"2024-02-13T00:37:45.244408+0000","last_became_peered":"2024-02-13T00:37:45.244408+0000","last_unstale":"2024-02-13T00:42:13.341866+0000","last_undegraded":"2024-02-13T00:42:13.341866+0000","last_fullsized":"2024-02-13T00:42:13.341866+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:43.426982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:43.426982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":161,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583857+0000","last_change":"2024-02-13T00:40:35.856095+0000","last_active":"2024-02-13T00:42:13.583857+0000","last_peered":"2024-02-13T00:42:13.583857+0000","last_clean":"2024-02-13T00:42:13.583857+0000","last_became_active":"2024-02-13T00:40:35.855722+0000","last_became_peered":"2024-02-13T00:40:35.855722+0000","last_unstale":"2024-02-13T00:42:13.583857+0000","last_undegraded":"2024-02-13T00:42:13.583857+0000","last_fullsized":"2024-02-13T00:42:13.583857+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'24","last_scrub_stamp":"2024-02-13T00:39:56.121046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:56.121046+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":36,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":109,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.352081+0000","last_change":"2024-02-13T00:40:48.809660+0000","last_active":"2024-02-13T00:42:13.352081+0000","last_peered":"2024-02-13T00:42:13.352081+0000","last_clean":"2024-02-13T00:42:13.352081+0000","last_became_active":"2024-02-13T00:40:35.852351+0000","last_became_peered":"2024-02-13T00:40:35.852351+0000","last_unstale":"2024-02-13T00:42:13.352081+0000","last_undegraded":"2024-02-13T00:42:13.352081+0000","last_fullsized":"2024-02-13T00:42:13.352081+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:40:48.809594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:48.809594+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":134,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586780+0000","last_change":"2024-02-13T00:41:45.660743+0000","last_active":"2024-02-13T00:42:13.586780+0000","last_peered":"2024-02-13T00:42:13.586780+0000","last_clean":"2024-02-13T00:42:13.586780+0000","last_became_active":"2024-02-13T00:37:47.260242+0000","last_became_peered":"2024-02-13T00:37:47.260242+0000","last_unstale":"2024-02-13T00:42:13.586780+0000","last_undegraded":"2024-02-13T00:42:13.586780+0000","last_fullsized":"2024-02-13T00:42:13.586780+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:41:45.660653+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"108'5","reported_seq":87,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.587401+0000","last_change":"2024-02-13T00:41:01.638321+0000","last_active":"2024-02-13T00:42:13.587401+0000","last_peered":"2024-02-13T00:42:13.587401+0000","last_clean":"2024-02-13T00:42:13.587401+0000","last_became_active":"2024-02-13T00:41:01.637897+0000","last_became_peered":"2024-02-13T00:41:01.637897+0000","last_unstale":"2024-02-13T00:42:13.587401+0000","last_undegraded":"2024-02-13T00:42:13.587401+0000","last_fullsized":"2024-02-13T00:42:13.587401+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.350565+0000","last_change":"2024-02-13T00:41:01.594534+0000","last_active":"2024-02-13T00:42:13.350565+0000","last_peered":"2024-02-13T00:42:13.350565+0000","last_clean":"2024-02-13T00:42:13.350565+0000","last_became_active":"2024-02-13T00:41:01.590676+0000","last_became_peered":"2024-02-13T00:41:01.590676+0000","last_unstale":"2024-02-13T00:42:13.350565+0000","last_undegraded":"2024-02-13T00:42:13.350565+0000","last_fullsized":"2024-02-13T00:42:13.350565+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":184,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.350300+0000","last_change":"2024-02-13T00:40:49.767975+0000","last_active":"2024-02-13T00:42:13.350300+0000","last_peered":"2024-02-13T00:42:13.350300+0000","last_clean":"2024-02-13T00:42:13.350300+0000","last_became_active":"2024-02-13T00:37:43.322401+0000","last_became_peered":"2024-02-13T00:37:43.322401+0000","last_unstale":"2024-02-13T00:42:13.350300+0000","last_undegraded":"2024-02-13T00:42:13.350300+0000","last_fullsized":"2024-02-13T00:42:13.350300+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:40:49.767853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:49.767853+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":146,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.341738+0000","last_change":"2024-02-13T00:41:49.468480+0000","last_active":"2024-02-13T00:42:13.341738+0000","last_peered":"2024-02-13T00:42:13.341738+0000","last_clean":"2024-02-13T00:42:13.341738+0000","last_became_active":"2024-02-13T00:37:41.214091+0000","last_became_peered":"2024-02-13T00:37:41.214091+0000","last_unstale":"2024-02-13T00:42:13.341738+0000","last_undegraded":"2024-02-13T00:42:13.341738+0000","last_fullsized":"2024-02-13T00:42:13.341738+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:49.468410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:49.468410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":131,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.588451+0000","last_change":"2024-02-13T00:39:46.155792+0000","last_active":"2024-02-13T00:42:13.588451+0000","last_peered":"2024-02-13T00:42:13.588451+0000","last_clean":"2024-02-13T00:42:13.588451+0000","last_became_active":"2024-02-13T00:37:45.250632+0000","last_became_peered":"2024-02-13T00:37:45.250632+0000","last_unstale":"2024-02-13T00:42:13.588451+0000","last_undegraded":"2024-02-13T00:42:13.588451+0000","last_fullsized":"2024-02-13T00:42:13.588451+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:46.155718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:46.155718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":135,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.344597+0000","last_change":"2024-02-13T00:40:59.468608+0000","last_active":"2024-02-13T00:42:13.344597+0000","last_peered":"2024-02-13T00:42:13.344597+0000","last_clean":"2024-02-13T00:42:13.344597+0000","last_became_active":"2024-02-13T00:37:47.254803+0000","last_became_peered":"2024-02-13T00:37:47.254803+0000","last_unstale":"2024-02-13T00:42:13.344597+0000","last_undegraded":"2024-02-13T00:42:13.344597+0000","last_fullsized":"2024-02-13T00:42:13.344597+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.468525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.468525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.0","version":"108'5","reported_seq":87,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.354667+0000","last_change":"2024-02-13T00:41:01.580498+0000","last_active":"2024-02-13T00:42:13.354667+0000","last_peered":"2024-02-13T00:42:13.354667+0000","last_clean":"2024-02-13T00:42:13.354667+0000","last_became_active":"2024-02-13T00:41:01.579665+0000","last_became_peered":"2024-02-13T00:41:01.579665+0000","last_unstale":"2024-02-13T00:42:13.354667+0000","last_undegraded":"2024-02-13T00:42:13.354667+0000","last_fullsized":"2024-02-13T00:42:13.354667+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347909+0000","last_change":"2024-02-13T00:42:03.399201+0000","last_active":"2024-02-13T00:42:13.347909+0000","last_peered":"2024-02-13T00:42:13.347909+0000","last_clean":"2024-02-13T00:42:13.347909+0000","last_became_active":"2024-02-13T00:42:03.393040+0000","last_became_peered":"2024-02-13T00:42:03.393040+0000","last_unstale":"2024-02-13T00:42:13.347909+0000","last_undegraded":"2024-02-13T00:42:13.347909+0000","last_fullsized":"2024-02-13T00:42:13.347909+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":162,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.588377+0000","last_change":"2024-02-13T00:41:42.695643+0000","last_active":"2024-02-13T00:42:13.588377+0000","last_peered":"2024-02-13T00:42:13.588377+0000","last_clean":"2024-02-13T00:42:13.588377+0000","last_became_active":"2024-02-13T00:37:43.324510+0000","last_became_peered":"2024-02-13T00:37:43.324510+0000","last_unstale":"2024-02-13T00:42:13.588377+0000","last_undegraded":"2024-02-13T00:42:13.588377+0000","last_fullsized":"2024-02-13T00:42:13.588377+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:41:42.695521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:42.695521+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583773+0000","last_change":"2024-02-13T00:42:03.411393+0000","last_active":"2024-02-13T00:42:13.583773+0000","last_peered":"2024-02-13T00:42:13.583773+0000","last_clean":"2024-02-13T00:42:13.583773+0000","last_became_active":"2024-02-13T00:42:03.402924+0000","last_became_peered":"2024-02-13T00:42:03.402924+0000","last_unstale":"2024-02-13T00:42:13.583773+0000","last_undegraded":"2024-02-13T00:42:13.583773+0000","last_fullsized":"2024-02-13T00:42:13.583773+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:54.316591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:40:54.316591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":136,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347673+0000","last_change":"2024-02-13T00:40:53.536240+0000","last_active":"2024-02-13T00:42:13.347673+0000","last_peered":"2024-02-13T00:42:13.347673+0000","last_clean":"2024-02-13T00:42:13.347673+0000","last_became_active":"2024-02-13T00:37:45.246907+0000","last_became_peered":"2024-02-13T00:37:45.246907+0000","last_unstale":"2024-02-13T00:42:13.347673+0000","last_undegraded":"2024-02-13T00:42:13.347673+0000","last_fullsized":"2024-02-13T00:42:13.347673+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:53.536170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:53.536170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":143,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586541+0000","last_change":"2024-02-13T00:42:03.407084+0000","last_active":"2024-02-13T00:42:13.586541+0000","last_peered":"2024-02-13T00:42:13.586541+0000","last_clean":"2024-02-13T00:42:13.586541+0000","last_became_active":"2024-02-13T00:42:03.406738+0000","last_became_peered":"2024-02-13T00:42:03.406738+0000","last_unstale":"2024-02-13T00:42:13.586541+0000","last_undegraded":"2024-02-13T00:42:13.586541+0000","last_fullsized":"2024-02-13T00:42:13.586541+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.115856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.115856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1","version":"108'5","reported_seq":100,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582790+0000","last_change":"2024-02-13T00:42:03.968757+0000","last_active":"2024-02-13T00:42:13.582790+0000","last_peered":"2024-02-13T00:42:13.582790+0000","last_clean":"2024-02-13T00:42:13.582790+0000","last_became_active":"2024-02-13T00:42:03.396535+0000","last_became_peered":"2024-02-13T00:42:03.396535+0000","last_unstale":"2024-02-13T00:42:13.582790+0000","last_undegraded":"2024-02-13T00:42:13.582790+0000","last_fullsized":"2024-02-13T00:42:13.582790+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584504+0000","last_change":"2024-02-13T00:41:01.653337+0000","last_active":"2024-02-13T00:42:13.584504+0000","last_peered":"2024-02-13T00:42:13.584504+0000","last_clean":"2024-02-13T00:42:13.584504+0000","last_became_active":"2024-02-13T00:41:01.645866+0000","last_became_peered":"2024-02-13T00:41:01.645866+0000","last_unstale":"2024-02-13T00:42:13.584504+0000","last_undegraded":"2024-02-13T00:42:13.584504+0000","last_fullsized":"2024-02-13T00:42:13.584504+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":190,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584426+0000","last_change":"2024-02-13T00:40:04.185200+0000","last_active":"2024-02-13T00:42:13.584426+0000","last_peered":"2024-02-13T00:42:13.584426+0000","last_clean":"2024-02-13T00:42:13.584426+0000","last_became_active":"2024-02-13T00:37:43.324313+0000","last_became_peered":"2024-02-13T00:37:43.324313+0000","last_unstale":"2024-02-13T00:42:13.584426+0000","last_undegraded":"2024-02-13T00:42:13.584426+0000","last_fullsized":"2024-02-13T00:42:13.584426+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:40:04.185125+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:40:04.185125+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":140,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.348950+0000","last_change":"2024-02-13T00:40:52.503871+0000","last_active":"2024-02-13T00:42:13.348950+0000","last_peered":"2024-02-13T00:42:13.348950+0000","last_clean":"2024-02-13T00:42:13.348950+0000","last_became_active":"2024-02-13T00:37:41.219309+0000","last_became_peered":"2024-02-13T00:37:41.219309+0000","last_unstale":"2024-02-13T00:42:13.348950+0000","last_undegraded":"2024-02-13T00:42:13.348950+0000","last_fullsized":"2024-02-13T00:42:13.348950+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:52.503804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:52.503804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"64'8","reported_seq":199,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:14.270515+0000","last_change":"2024-02-13T00:41:37.453007+0000","last_active":"2024-02-13T00:42:14.270515+0000","last_peered":"2024-02-13T00:42:14.270515+0000","last_clean":"2024-02-13T00:42:14.270515+0000","last_became_active":"2024-02-13T00:37:45.246757+0000","last_became_peered":"2024-02-13T00:37:45.246757+0000","last_unstale":"2024-02-13T00:42:14.270515+0000","last_undegraded":"2024-02-13T00:42:14.270515+0000","last_fullsized":"2024-02-13T00:42:14.270515+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:37.452882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:37.452882+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":129,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.350137+0000","last_change":"2024-02-13T00:39:21.344836+0000","last_active":"2024-02-13T00:42:13.350137+0000","last_peered":"2024-02-13T00:42:13.350137+0000","last_clean":"2024-02-13T00:42:13.350137+0000","last_became_active":"2024-02-13T00:37:47.252722+0000","last_became_peered":"2024-02-13T00:37:47.252722+0000","last_unstale":"2024-02-13T00:42:13.350137+0000","last_undegraded":"2024-02-13T00:42:13.350137+0000","last_fullsized":"2024-02-13T00:42:13.350137+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.344771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.344771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":82,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583450+0000","last_change":"2024-02-13T00:41:01.582310+0000","last_active":"2024-02-13T00:42:13.583450+0000","last_peered":"2024-02-13T00:42:13.583450+0000","last_clean":"2024-02-13T00:42:13.583450+0000","last_became_active":"2024-02-13T00:41:01.581659+0000","last_became_peered":"2024-02-13T00:41:01.581659+0000","last_unstale":"2024-02-13T00:42:13.583450+0000","last_undegraded":"2024-02-13T00:42:13.583450+0000","last_fullsized":"2024-02-13T00:42:13.583450+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.579358+0000","last_change":"2024-02-13T00:41:01.559598+0000","last_active":"2024-02-13T00:42:13.579358+0000","last_peered":"2024-02-13T00:42:13.579358+0000","last_clean":"2024-02-13T00:42:13.579358+0000","last_became_active":"2024-02-13T00:41:01.557578+0000","last_became_peered":"2024-02-13T00:41:01.557578+0000","last_unstale":"2024-02-13T00:42:13.579358+0000","last_undegraded":"2024-02-13T00:42:13.579358+0000","last_fullsized":"2024-02-13T00:42:13.579358+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":170,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345357+0000","last_change":"2024-02-13T00:41:36.477098+0000","last_active":"2024-02-13T00:42:13.345357+0000","last_peered":"2024-02-13T00:42:13.345357+0000","last_clean":"2024-02-13T00:42:13.345357+0000","last_became_active":"2024-02-13T00:37:43.322357+0000","last_became_peered":"2024-02-13T00:37:43.322357+0000","last_unstale":"2024-02-13T00:42:13.345357+0000","last_undegraded":"2024-02-13T00:42:13.345357+0000","last_fullsized":"2024-02-13T00:42:13.345357+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:36.477034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.477034+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":140,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.344921+0000","last_change":"2024-02-13T00:40:52.801522+0000","last_active":"2024-02-13T00:42:13.344921+0000","last_peered":"2024-02-13T00:42:13.344921+0000","last_clean":"2024-02-13T00:42:13.344921+0000","last_became_active":"2024-02-13T00:37:41.217953+0000","last_became_peered":"2024-02-13T00:37:41.217953+0000","last_unstale":"2024-02-13T00:42:13.344921+0000","last_undegraded":"2024-02-13T00:42:13.344921+0000","last_fullsized":"2024-02-13T00:42:13.344921+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:52.801455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:52.801455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"84'9","reported_seq":203,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:14.270473+0000","last_change":"2024-02-13T00:41:32.715662+0000","last_active":"2024-02-13T00:42:14.270473+0000","last_peered":"2024-02-13T00:42:14.270473+0000","last_clean":"2024-02-13T00:42:14.270473+0000","last_became_active":"2024-02-13T00:37:45.250598+0000","last_became_peered":"2024-02-13T00:37:45.250598+0000","last_unstale":"2024-02-13T00:42:14.270473+0000","last_undegraded":"2024-02-13T00:42:14.270473+0000","last_fullsized":"2024-02-13T00:42:14.270473+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-02-13T00:41:32.715550+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.715550+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":105,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347740+0000","last_change":"2024-02-13T00:41:55.421064+0000","last_active":"2024-02-13T00:42:13.347740+0000","last_peered":"2024-02-13T00:42:13.347740+0000","last_clean":"2024-02-13T00:42:13.347740+0000","last_became_active":"2024-02-13T00:40:35.850176+0000","last_became_peered":"2024-02-13T00:40:35.850176+0000","last_unstale":"2024-02-13T00:42:13.347740+0000","last_undegraded":"2024-02-13T00:42:13.347740+0000","last_fullsized":"2024-02-13T00:42:13.347740+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.421001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.421001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.3","version":"108'5","reported_seq":100,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585380+0000","last_change":"2024-02-13T00:42:05.399708+0000","last_active":"2024-02-13T00:42:13.585380+0000","last_peered":"2024-02-13T00:42:13.585380+0000","last_clean":"2024-02-13T00:42:13.585380+0000","last_became_active":"2024-02-13T00:42:03.420690+0000","last_became_peered":"2024-02-13T00:42:03.420690+0000","last_unstale":"2024-02-13T00:42:13.585380+0000","last_undegraded":"2024-02-13T00:42:13.585380+0000","last_fullsized":"2024-02-13T00:42:13.585380+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":90,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.349371+0000","last_change":"2024-02-13T00:42:03.415812+0000","last_active":"2024-02-13T00:42:13.349371+0000","last_peered":"2024-02-13T00:42:13.349371+0000","last_clean":"2024-02-13T00:42:13.349371+0000","last_became_active":"2024-02-13T00:42:03.415302+0000","last_became_peered":"2024-02-13T00:42:03.415302+0000","last_unstale":"2024-02-13T00:42:13.349371+0000","last_undegraded":"2024-02-13T00:42:13.349371+0000","last_fullsized":"2024-02-13T00:42:13.349371+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":178,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584748+0000","last_change":"2024-02-13T00:42:07.212237+0000","last_active":"2024-02-13T00:42:13.584748+0000","last_peered":"2024-02-13T00:42:13.584748+0000","last_clean":"2024-02-13T00:42:13.584748+0000","last_became_active":"2024-02-13T00:42:03.424278+0000","last_became_peered":"2024-02-13T00:42:03.424278+0000","last_unstale":"2024-02-13T00:42:13.584748+0000","last_undegraded":"2024-02-13T00:42:13.584748+0000","last_fullsized":"2024-02-13T00:42:13.584748+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:40:16.009518+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:40:16.009518+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584587+0000","last_change":"2024-02-13T00:42:03.432596+0000","last_active":"2024-02-13T00:42:13.584587+0000","last_peered":"2024-02-13T00:42:13.584587+0000","last_clean":"2024-02-13T00:42:13.584587+0000","last_became_active":"2024-02-13T00:42:03.431748+0000","last_became_peered":"2024-02-13T00:42:03.431748+0000","last_unstale":"2024-02-13T00:42:13.584587+0000","last_undegraded":"2024-02-13T00:42:13.584587+0000","last_fullsized":"2024-02-13T00:42:13.584587+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:58.334705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.334705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":136,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586692+0000","last_change":"2024-02-13T00:41:43.717024+0000","last_active":"2024-02-13T00:42:13.586692+0000","last_peered":"2024-02-13T00:42:13.586692+0000","last_clean":"2024-02-13T00:42:13.586692+0000","last_became_active":"2024-02-13T00:37:45.243594+0000","last_became_peered":"2024-02-13T00:37:45.243594+0000","last_unstale":"2024-02-13T00:42:13.586692+0000","last_undegraded":"2024-02-13T00:42:13.586692+0000","last_fullsized":"2024-02-13T00:42:13.586692+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:43.716891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_clean_scrub_stamp":"2024-02-13T00:41:43.716891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":129,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.349992+0000","last_change":"2024-02-13T00:39:28.286635+0000","last_active":"2024-02-13T00:42:13.349992+0000","last_peered":"2024-02-13T00:42:13.349992+0000","last_clean":"2024-02-13T00:42:13.349992+0000","last_became_active":"2024-02-13T00:37:47.259792+0000","last_became_peered":"2024-02-13T00:37:47.259792+0000","last_unstale":"2024-02-13T00:42:13.349992+0000","last_undegraded":"2024-02-13T00:42:13.349992+0000","last_fullsized":"2024-02-13T00:42:13.349992+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:28.286574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:39:28.286574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":82,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.349578+0000","last_change":"2024-02-13T00:41:01.745030+0000","last_active":"2024-02-13T00:42:13.349578+0000","last_peered":"2024-02-13T00:42:13.349578+0000","last_clean":"2024-02-13T00:42:13.349578+0000","last_became_active":"2024-02-13T00:41:01.743858+0000","last_became_peered":"2024-02-13T00:41:01.743858+0000","last_unstale":"2024-02-13T00:42:13.349578+0000","last_undegraded":"2024-02-13T00:42:13.349578+0000","last_fullsized":"2024-02-13T00:42:13.349578+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585760+0000","last_change":"2024-02-13T00:42:03.414725+0000","last_active":"2024-02-13T00:42:13.585760+0000","last_peered":"2024-02-13T00:42:13.585760+0000","last_clean":"2024-02-13T00:42:13.585760+0000","last_became_active":"2024-02-13T00:42:03.414406+0000","last_became_peered":"2024-02-13T00:42:03.414406+0000","last_unstale":"2024-02-13T00:42:13.585760+0000","last_undegraded":"2024-02-13T00:42:13.585760+0000","last_fullsized":"2024-02-13T00:42:13.585760+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":95,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342471+0000","last_change":"2024-02-13T00:42:03.430410+0000","last_active":"2024-02-13T00:42:13.342471+0000","last_peered":"2024-02-13T00:42:13.342471+0000","last_clean":"2024-02-13T00:42:13.342471+0000","last_became_active":"2024-02-13T00:42:03.384628+0000","last_became_peered":"2024-02-13T00:42:03.384628+0000","last_unstale":"2024-02-13T00:42:13.342471+0000","last_undegraded":"2024-02-13T00:42:13.342471+0000","last_fullsized":"2024-02-13T00:42:13.342471+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:40:44.273511+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.273511+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":141,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583562+0000","last_change":"2024-02-13T00:41:03.608458+0000","last_active":"2024-02-13T00:42:13.583562+0000","last_peered":"2024-02-13T00:42:13.583562+0000","last_clean":"2024-02-13T00:42:13.583562+0000","last_became_active":"2024-02-13T00:37:41.215531+0000","last_became_peered":"2024-02-13T00:37:41.215531+0000","last_unstale":"2024-02-13T00:42:13.583562+0000","last_undegraded":"2024-02-13T00:42:13.583562+0000","last_fullsized":"2024-02-13T00:42:13.583562+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:03.608379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:03.608379+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":137,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.585273+0000","last_change":"2024-02-13T00:40:49.666455+0000","last_active":"2024-02-13T00:42:13.585273+0000","last_peered":"2024-02-13T00:42:13.585273+0000","last_clean":"2024-02-13T00:42:13.585273+0000","last_became_active":"2024-02-13T00:37:45.247114+0000","last_became_peered":"2024-02-13T00:37:45.247114+0000","last_unstale":"2024-02-13T00:42:13.585273+0000","last_undegraded":"2024-02-13T00:42:13.585273+0000","last_fullsized":"2024-02-13T00:42:13.585273+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:49.666387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:40:49.666387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":134,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.581973+0000","last_change":"2024-02-13T00:40:05.137000+0000","last_active":"2024-02-13T00:42:13.581973+0000","last_peered":"2024-02-13T00:42:13.581973+0000","last_clean":"2024-02-13T00:42:13.581973+0000","last_became_active":"2024-02-13T00:37:47.254519+0000","last_became_peered":"2024-02-13T00:37:47.254519+0000","last_unstale":"2024-02-13T00:42:13.581973+0000","last_undegraded":"2024-02-13T00:42:13.581973+0000","last_fullsized":"2024-02-13T00:42:13.581973+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:05.136935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:05.136935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":91,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583642+0000","last_change":"2024-02-13T00:42:03.414688+0000","last_active":"2024-02-13T00:42:13.583642+0000","last_peered":"2024-02-13T00:42:13.583642+0000","last_clean":"2024-02-13T00:42:13.583642+0000","last_became_active":"2024-02-13T00:42:03.413667+0000","last_became_peered":"2024-02-13T00:42:03.413667+0000","last_unstale":"2024-02-13T00:42:13.583642+0000","last_undegraded":"2024-02-13T00:42:13.583642+0000","last_fullsized":"2024-02-13T00:42:13.583642+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582166+0000","last_change":"2024-02-13T00:42:03.435906+0000","last_active":"2024-02-13T00:42:13.582166+0000","last_peered":"2024-02-13T00:42:13.582166+0000","last_clean":"2024-02-13T00:42:13.582166+0000","last_became_active":"2024-02-13T00:42:03.433274+0000","last_became_peered":"2024-02-13T00:42:03.433274+0000","last_unstale":"2024-02-13T00:42:13.582166+0000","last_undegraded":"2024-02-13T00:42:13.582166+0000","last_fullsized":"2024-02-13T00:42:13.582166+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":159,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582085+0000","last_change":"2024-02-13T00:42:06.873341+0000","last_active":"2024-02-13T00:42:13.582085+0000","last_peered":"2024-02-13T00:42:13.582085+0000","last_clean":"2024-02-13T00:42:13.582085+0000","last_became_active":"2024-02-13T00:42:03.420647+0000","last_became_peered":"2024-02-13T00:42:03.420647+0000","last_unstale":"2024-02-13T00:42:13.582085+0000","last_undegraded":"2024-02-13T00:42:13.582085+0000","last_fullsized":"2024-02-13T00:42:13.582085+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:40:51.054728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.054728+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":149,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.348357+0000","last_change":"2024-02-13T00:42:03.399638+0000","last_active":"2024-02-13T00:42:13.348357+0000","last_peered":"2024-02-13T00:42:13.348357+0000","last_clean":"2024-02-13T00:42:13.348357+0000","last_became_active":"2024-02-13T00:42:03.399329+0000","last_became_peered":"2024-02-13T00:42:03.399329+0000","last_unstale":"2024-02-13T00:42:13.348357+0000","last_undegraded":"2024-02-13T00:42:13.348357+0000","last_fullsized":"2024-02-13T00:42:13.348357+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:10.635478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:10.635478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583058+0000","last_change":"2024-02-13T00:42:03.409256+0000","last_active":"2024-02-13T00:42:13.583058+0000","last_peered":"2024-02-13T00:42:13.583058+0000","last_clean":"2024-02-13T00:42:13.583058+0000","last_became_active":"2024-02-13T00:42:03.408952+0000","last_became_peered":"2024-02-13T00:42:03.408952+0000","last_unstale":"2024-02-13T00:42:13.583058+0000","last_undegraded":"2024-02-13T00:42:13.583058+0000","last_fullsized":"2024-02-13T00:42:13.583058+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:57.978593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:57.978593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":137,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583194+0000","last_change":"2024-02-13T00:40:41.220513+0000","last_active":"2024-02-13T00:42:13.583194+0000","last_peered":"2024-02-13T00:42:13.583194+0000","last_clean":"2024-02-13T00:42:13.583194+0000","last_became_active":"2024-02-13T00:37:47.254667+0000","last_became_peered":"2024-02-13T00:37:47.254667+0000","last_unstale":"2024-02-13T00:42:13.583194+0000","last_undegraded":"2024-02-13T00:42:13.583194+0000","last_fullsized":"2024-02-13T00:42:13.583194+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:41.220387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:41.220387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.6","version":"108'9","reported_seq":91,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586539+0000","last_change":"2024-02-13T00:41:01.631258+0000","last_active":"2024-02-13T00:42:13.586539+0000","last_peered":"2024-02-13T00:42:13.586539+0000","last_clean":"2024-02-13T00:42:13.586539+0000","last_became_active":"2024-02-13T00:41:01.630983+0000","last_became_peered":"2024-02-13T00:41:01.630983+0000","last_unstale":"2024-02-13T00:42:13.586539+0000","last_undegraded":"2024-02-13T00:42:13.586539+0000","last_fullsized":"2024-02-13T00:42:13.586539+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":90,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.343128+0000","last_change":"2024-02-13T00:42:03.402279+0000","last_active":"2024-02-13T00:42:13.343128+0000","last_peered":"2024-02-13T00:42:13.343128+0000","last_clean":"2024-02-13T00:42:13.343128+0000","last_became_active":"2024-02-13T00:42:03.401648+0000","last_became_peered":"2024-02-13T00:42:03.401648+0000","last_unstale":"2024-02-13T00:42:13.343128+0000","last_undegraded":"2024-02-13T00:42:13.343128+0000","last_fullsized":"2024-02-13T00:42:13.343128+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":195,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582595+0000","last_change":"2024-02-13T00:42:08.062022+0000","last_active":"2024-02-13T00:42:13.582595+0000","last_peered":"2024-02-13T00:42:13.582595+0000","last_clean":"2024-02-13T00:42:13.582595+0000","last_became_active":"2024-02-13T00:42:03.432551+0000","last_became_peered":"2024-02-13T00:42:03.432551+0000","last_unstale":"2024-02-13T00:42:13.582595+0000","last_undegraded":"2024-02-13T00:42:13.582595+0000","last_fullsized":"2024-02-13T00:42:13.582595+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:41:50.675154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:50.675154+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.349047+0000","last_change":"2024-02-13T00:42:03.397136+0000","last_active":"2024-02-13T00:42:13.349047+0000","last_peered":"2024-02-13T00:42:13.349047+0000","last_clean":"2024-02-13T00:42:13.349047+0000","last_became_active":"2024-02-13T00:42:03.396688+0000","last_became_peered":"2024-02-13T00:42:03.396688+0000","last_unstale":"2024-02-13T00:42:13.349047+0000","last_undegraded":"2024-02-13T00:42:13.349047+0000","last_fullsized":"2024-02-13T00:42:13.349047+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.457672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.457672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":145,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582018+0000","last_change":"2024-02-13T00:42:03.409230+0000","last_active":"2024-02-13T00:42:13.582018+0000","last_peered":"2024-02-13T00:42:13.582018+0000","last_clean":"2024-02-13T00:42:13.582018+0000","last_became_active":"2024-02-13T00:42:03.408858+0000","last_became_peered":"2024-02-13T00:42:03.408858+0000","last_unstale":"2024-02-13T00:42:13.582018+0000","last_undegraded":"2024-02-13T00:42:13.582018+0000","last_fullsized":"2024-02-13T00:42:13.582018+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.164690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.164690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":143,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586859+0000","last_change":"2024-02-13T00:42:03.408944+0000","last_active":"2024-02-13T00:42:13.586859+0000","last_peered":"2024-02-13T00:42:13.586859+0000","last_clean":"2024-02-13T00:42:13.586859+0000","last_became_active":"2024-02-13T00:42:03.408629+0000","last_became_peered":"2024-02-13T00:42:03.408629+0000","last_unstale":"2024-02-13T00:42:13.586859+0000","last_undegraded":"2024-02-13T00:42:13.586859+0000","last_fullsized":"2024-02-13T00:42:13.586859+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.881255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.881255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":82,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.347831+0000","last_change":"2024-02-13T00:41:01.595241+0000","last_active":"2024-02-13T00:42:13.347831+0000","last_peered":"2024-02-13T00:42:13.347831+0000","last_clean":"2024-02-13T00:42:13.347831+0000","last_became_active":"2024-02-13T00:41:01.581338+0000","last_became_peered":"2024-02-13T00:41:01.581338+0000","last_unstale":"2024-02-13T00:42:13.347831+0000","last_undegraded":"2024-02-13T00:42:13.347831+0000","last_fullsized":"2024-02-13T00:42:13.347831+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.579391+0000","last_change":"2024-02-13T00:41:01.569377+0000","last_active":"2024-02-13T00:42:13.579391+0000","last_peered":"2024-02-13T00:42:13.579391+0000","last_clean":"2024-02-13T00:42:13.579391+0000","last_became_active":"2024-02-13T00:41:01.565724+0000","last_became_peered":"2024-02-13T00:41:01.565724+0000","last_unstale":"2024-02-13T00:42:13.579391+0000","last_undegraded":"2024-02-13T00:42:13.579391+0000","last_fullsized":"2024-02-13T00:42:13.579391+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":174,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.350853+0000","last_change":"2024-02-13T00:40:06.169154+0000","last_active":"2024-02-13T00:42:13.350853+0000","last_peered":"2024-02-13T00:42:13.350853+0000","last_clean":"2024-02-13T00:42:13.350853+0000","last_became_active":"2024-02-13T00:37:43.324221+0000","last_became_peered":"2024-02-13T00:37:43.324221+0000","last_unstale":"2024-02-13T00:42:13.350853+0000","last_undegraded":"2024-02-13T00:42:13.350853+0000","last_fullsized":"2024-02-13T00:42:13.350853+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:40:06.169087+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:40:06.169087+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":140,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345527+0000","last_change":"2024-02-13T00:41:32.424906+0000","last_active":"2024-02-13T00:42:13.345527+0000","last_peered":"2024-02-13T00:42:13.345527+0000","last_clean":"2024-02-13T00:42:13.345527+0000","last_became_active":"2024-02-13T00:37:41.214965+0000","last_became_peered":"2024-02-13T00:37:41.214965+0000","last_unstale":"2024-02-13T00:42:13.345527+0000","last_undegraded":"2024-02-13T00:42:13.345527+0000","last_fullsized":"2024-02-13T00:42:13.345527+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:32.424844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.424844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"168'9","reported_seq":52,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:14.270485+0000","last_change":"2024-02-13T00:42:06.873690+0000","last_active":"2024-02-13T00:42:14.270485+0000","last_peered":"2024-02-13T00:42:14.270485+0000","last_clean":"2024-02-13T00:42:14.270485+0000","last_became_active":"2024-02-13T00:42:03.415030+0000","last_became_peered":"2024-02-13T00:42:03.415030+0000","last_unstale":"2024-02-13T00:42:14.270485+0000","last_undegraded":"2024-02-13T00:42:14.270485+0000","last_fullsized":"2024-02-13T00:42:14.270485+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:02.323101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.323101+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":135,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584634+0000","last_change":"2024-02-13T00:40:02.168591+0000","last_active":"2024-02-13T00:42:13.584634+0000","last_peered":"2024-02-13T00:42:13.584634+0000","last_clean":"2024-02-13T00:42:13.584634+0000","last_became_active":"2024-02-13T00:37:47.254586+0000","last_became_peered":"2024-02-13T00:37:47.254586+0000","last_unstale":"2024-02-13T00:42:13.584634+0000","last_undegraded":"2024-02-13T00:42:13.584634+0000","last_fullsized":"2024-02-13T00:42:13.584634+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:40:02.168499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:02.168499+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"107'2","reported_seq":97,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.351312+0000","last_change":"2024-02-13T00:42:04.282151+0000","last_active":"2024-02-13T00:42:13.351312+0000","last_peered":"2024-02-13T00:42:13.351312+0000","last_clean":"2024-02-13T00:42:13.351312+0000","last_became_active":"2024-02-13T00:42:03.410938+0000","last_became_peered":"2024-02-13T00:42:03.410938+0000","last_unstale":"2024-02-13T00:42:13.351312+0000","last_undegraded":"2024-02-13T00:42:13.351312+0000","last_fullsized":"2024-02-13T00:42:13.351312+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":90,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.589215+0000","last_change":"2024-02-13T00:42:03.427616+0000","last_active":"2024-02-13T00:42:13.589215+0000","last_peered":"2024-02-13T00:42:13.589215+0000","last_clean":"2024-02-13T00:42:13.589215+0000","last_became_active":"2024-02-13T00:42:03.427079+0000","last_became_peered":"2024-02-13T00:42:03.427079+0000","last_unstale":"2024-02-13T00:42:13.589215+0000","last_undegraded":"2024-02-13T00:42:13.589215+0000","last_fullsized":"2024-02-13T00:42:13.589215+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":84,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.349285+0000","last_change":"2024-02-13T00:42:03.451350+0000","last_active":"2024-02-13T00:42:13.349285+0000","last_peered":"2024-02-13T00:42:13.349285+0000","last_clean":"2024-02-13T00:42:13.349285+0000","last_became_active":"2024-02-13T00:42:03.397231+0000","last_became_peered":"2024-02-13T00:42:03.397231+0000","last_unstale":"2024-02-13T00:42:13.349285+0000","last_undegraded":"2024-02-13T00:42:13.349285+0000","last_fullsized":"2024-02-13T00:42:13.349285+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:40:42.230976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:42.230976+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":149,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582144+0000","last_change":"2024-02-13T00:42:03.408124+0000","last_active":"2024-02-13T00:42:13.582144+0000","last_peered":"2024-02-13T00:42:13.582144+0000","last_clean":"2024-02-13T00:42:13.582144+0000","last_became_active":"2024-02-13T00:42:03.403904+0000","last_became_peered":"2024-02-13T00:42:03.403904+0000","last_unstale":"2024-02-13T00:42:13.582144+0000","last_undegraded":"2024-02-13T00:42:13.582144+0000","last_fullsized":"2024-02-13T00:42:13.582144+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:46.698599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:46.698599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":131,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.351299+0000","last_change":"2024-02-13T00:39:15.328037+0000","last_active":"2024-02-13T00:42:13.351299+0000","last_peered":"2024-02-13T00:42:13.351299+0000","last_clean":"2024-02-13T00:42:13.351299+0000","last_became_active":"2024-02-13T00:37:45.249064+0000","last_became_peered":"2024-02-13T00:37:45.249064+0000","last_unstale":"2024-02-13T00:42:13.351299+0000","last_undegraded":"2024-02-13T00:42:13.351299+0000","last_fullsized":"2024-02-13T00:42:13.351299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:15.327971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:39:15.327971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":135,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583429+0000","last_change":"2024-02-13T00:40:06.026382+0000","last_active":"2024-02-13T00:42:13.583429+0000","last_peered":"2024-02-13T00:42:13.583429+0000","last_clean":"2024-02-13T00:42:13.583429+0000","last_became_active":"2024-02-13T00:37:47.260182+0000","last_became_peered":"2024-02-13T00:37:47.260182+0000","last_unstale":"2024-02-13T00:42:13.583429+0000","last_undegraded":"2024-02-13T00:42:13.583429+0000","last_fullsized":"2024-02-13T00:42:13.583429+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:06.026318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:06.026318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":82,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582220+0000","last_change":"2024-02-13T00:41:01.682551+0000","last_active":"2024-02-13T00:42:13.582220+0000","last_peered":"2024-02-13T00:42:13.582220+0000","last_clean":"2024-02-13T00:42:13.582220+0000","last_became_active":"2024-02-13T00:41:01.682261+0000","last_became_peered":"2024-02-13T00:41:01.682261+0000","last_unstale":"2024-02-13T00:42:13.582220+0000","last_undegraded":"2024-02-13T00:42:13.582220+0000","last_fullsized":"2024-02-13T00:42:13.582220+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583200+0000","last_change":"2024-02-13T00:42:03.406045+0000","last_active":"2024-02-13T00:42:13.583200+0000","last_peered":"2024-02-13T00:42:13.583200+0000","last_clean":"2024-02-13T00:42:13.583200+0000","last_became_active":"2024-02-13T00:42:03.405677+0000","last_became_peered":"2024-02-13T00:42:03.405677+0000","last_unstale":"2024-02-13T00:42:13.583200+0000","last_undegraded":"2024-02-13T00:42:13.583200+0000","last_fullsized":"2024-02-13T00:42:13.583200+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":164,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345089+0000","last_change":"2024-02-13T00:40:44.369174+0000","last_active":"2024-02-13T00:42:13.345089+0000","last_peered":"2024-02-13T00:42:13.345089+0000","last_clean":"2024-02-13T00:42:13.345089+0000","last_became_active":"2024-02-13T00:37:43.232938+0000","last_became_peered":"2024-02-13T00:37:43.232938+0000","last_unstale":"2024-02-13T00:42:13.345089+0000","last_undegraded":"2024-02-13T00:42:13.345089+0000","last_fullsized":"2024-02-13T00:42:13.345089+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:40:44.369070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.369070+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583382+0000","last_change":"2024-02-13T00:42:03.414747+0000","last_active":"2024-02-13T00:42:13.583382+0000","last_peered":"2024-02-13T00:42:13.583382+0000","last_clean":"2024-02-13T00:42:13.583382+0000","last_became_active":"2024-02-13T00:42:03.414402+0000","last_became_peered":"2024-02-13T00:42:03.414402+0000","last_unstale":"2024-02-13T00:42:13.583382+0000","last_undegraded":"2024-02-13T00:42:13.583382+0000","last_fullsized":"2024-02-13T00:42:13.583382+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.316400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.316400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":137,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584057+0000","last_change":"2024-02-13T00:41:56.204734+0000","last_active":"2024-02-13T00:42:13.584057+0000","last_peered":"2024-02-13T00:42:13.584057+0000","last_clean":"2024-02-13T00:42:13.584057+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:42:13.584057+0000","last_undegraded":"2024-02-13T00:42:13.584057+0000","last_fullsized":"2024-02-13T00:42:13.584057+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:56.204671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.204671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":26,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.348473+0000","last_change":"2024-02-13T00:42:03.417141+0000","last_active":"2024-02-13T00:42:13.348473+0000","last_peered":"2024-02-13T00:42:13.348473+0000","last_clean":"2024-02-13T00:42:13.348473+0000","last_became_active":"2024-02-13T00:42:03.416740+0000","last_became_peered":"2024-02-13T00:42:03.416740+0000","last_unstale":"2024-02-13T00:42:13.348473+0000","last_undegraded":"2024-02-13T00:42:13.348473+0000","last_fullsized":"2024-02-13T00:42:13.348473+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:15.106889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:15.106889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.a","version":"108'10","reported_seq":107,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.590368+0000","last_change":"2024-02-13T00:42:06.672391+0000","last_active":"2024-02-13T00:42:13.590368+0000","last_peered":"2024-02-13T00:42:13.590368+0000","last_clean":"2024-02-13T00:42:13.590368+0000","last_became_active":"2024-02-13T00:42:03.419452+0000","last_became_peered":"2024-02-13T00:42:03.419452+0000","last_unstale":"2024-02-13T00:42:13.590368+0000","last_undegraded":"2024-02-13T00:42:13.590368+0000","last_fullsized":"2024-02-13T00:42:13.590368+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":90,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.588415+0000","last_change":"2024-02-13T00:42:03.442906+0000","last_active":"2024-02-13T00:42:13.588415+0000","last_peered":"2024-02-13T00:42:13.588415+0000","last_clean":"2024-02-13T00:42:13.588415+0000","last_became_active":"2024-02-13T00:42:03.442395+0000","last_became_peered":"2024-02-13T00:42:03.442395+0000","last_unstale":"2024-02-13T00:42:13.588415+0000","last_undegraded":"2024-02-13T00:42:13.588415+0000","last_fullsized":"2024-02-13T00:42:13.588415+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":178,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584245+0000","last_change":"2024-02-13T00:41:41.686152+0000","last_active":"2024-02-13T00:42:13.584245+0000","last_peered":"2024-02-13T00:42:13.584245+0000","last_clean":"2024-02-13T00:42:13.584245+0000","last_became_active":"2024-02-13T00:37:43.236026+0000","last_became_peered":"2024-02-13T00:37:43.236026+0000","last_unstale":"2024-02-13T00:42:13.584245+0000","last_undegraded":"2024-02-13T00:42:13.584245+0000","last_fullsized":"2024-02-13T00:42:13.584245+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:41:41.686058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.686058+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":149,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586616+0000","last_change":"2024-02-13T00:42:03.429962+0000","last_active":"2024-02-13T00:42:13.586616+0000","last_peered":"2024-02-13T00:42:13.586616+0000","last_clean":"2024-02-13T00:42:13.586616+0000","last_became_active":"2024-02-13T00:42:03.429539+0000","last_became_peered":"2024-02-13T00:42:03.429539+0000","last_unstale":"2024-02-13T00:42:13.586616+0000","last_undegraded":"2024-02-13T00:42:13.586616+0000","last_fullsized":"2024-02-13T00:42:13.586616+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:49.110725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:49.110725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":100,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584662+0000","last_change":"2024-02-13T00:40:52.067206+0000","last_active":"2024-02-13T00:42:13.584662+0000","last_peered":"2024-02-13T00:42:13.584662+0000","last_clean":"2024-02-13T00:42:13.584662+0000","last_became_active":"2024-02-13T00:40:35.858538+0000","last_became_peered":"2024-02-13T00:40:35.858538+0000","last_unstale":"2024-02-13T00:42:13.584662+0000","last_undegraded":"2024-02-13T00:42:13.584662+0000","last_fullsized":"2024-02-13T00:42:13.584662+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:52.067101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:40:52.067101+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":143,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.588294+0000","last_change":"2024-02-13T00:42:03.427267+0000","last_active":"2024-02-13T00:42:13.588294+0000","last_peered":"2024-02-13T00:42:13.588294+0000","last_clean":"2024-02-13T00:42:13.588294+0000","last_became_active":"2024-02-13T00:42:03.426960+0000","last_became_peered":"2024-02-13T00:42:03.426960+0000","last_unstale":"2024-02-13T00:42:13.588294+0000","last_undegraded":"2024-02-13T00:42:13.588294+0000","last_fullsized":"2024-02-13T00:42:13.588294+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.950483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.950483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"107'2","reported_seq":84,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346499+0000","last_change":"2024-02-13T00:41:01.611411+0000","last_active":"2024-02-13T00:42:13.346499+0000","last_peered":"2024-02-13T00:42:13.346499+0000","last_clean":"2024-02-13T00:42:13.346499+0000","last_became_active":"2024-02-13T00:41:01.611115+0000","last_became_peered":"2024-02-13T00:41:01.611115+0000","last_unstale":"2024-02-13T00:42:13.346499+0000","last_undegraded":"2024-02-13T00:42:13.346499+0000","last_fullsized":"2024-02-13T00:42:13.346499+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":90,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.345076+0000","last_change":"2024-02-13T00:42:03.382598+0000","last_active":"2024-02-13T00:42:13.345076+0000","last_peered":"2024-02-13T00:42:13.345076+0000","last_clean":"2024-02-13T00:42:13.345076+0000","last_became_active":"2024-02-13T00:42:03.382303+0000","last_became_peered":"2024-02-13T00:42:03.382303+0000","last_unstale":"2024-02-13T00:42:13.345076+0000","last_undegraded":"2024-02-13T00:42:13.345076+0000","last_fullsized":"2024-02-13T00:42:13.345076+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":159,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.588145+0000","last_change":"2024-02-13T00:41:31.648350+0000","last_active":"2024-02-13T00:42:13.588145+0000","last_peered":"2024-02-13T00:42:13.588145+0000","last_clean":"2024-02-13T00:42:13.588145+0000","last_became_active":"2024-02-13T00:40:35.857274+0000","last_became_peered":"2024-02-13T00:40:35.857274+0000","last_unstale":"2024-02-13T00:42:13.588145+0000","last_undegraded":"2024-02-13T00:42:13.588145+0000","last_fullsized":"2024-02-13T00:42:13.588145+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:41:31.648217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:31.648217+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":140,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584312+0000","last_change":"2024-02-13T00:40:43.197233+0000","last_active":"2024-02-13T00:42:13.584312+0000","last_peered":"2024-02-13T00:42:13.584312+0000","last_clean":"2024-02-13T00:42:13.584312+0000","last_became_active":"2024-02-13T00:37:41.222506+0000","last_became_peered":"2024-02-13T00:37:41.222506+0000","last_unstale":"2024-02-13T00:42:13.584312+0000","last_undegraded":"2024-02-13T00:42:13.584312+0000","last_fullsized":"2024-02-13T00:42:13.584312+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:43.197157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:43.197157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":136,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346600+0000","last_change":"2024-02-13T00:40:14.697069+0000","last_active":"2024-02-13T00:42:13.346600+0000","last_peered":"2024-02-13T00:42:13.346600+0000","last_clean":"2024-02-13T00:42:13.346600+0000","last_became_active":"2024-02-13T00:37:45.243961+0000","last_became_peered":"2024-02-13T00:37:45.243961+0000","last_unstale":"2024-02-13T00:42:13.346600+0000","last_undegraded":"2024-02-13T00:42:13.346600+0000","last_fullsized":"2024-02-13T00:42:13.346600+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:14.696991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:14.696991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":129,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.351176+0000","last_change":"2024-02-13T00:39:22.338139+0000","last_active":"2024-02-13T00:42:13.351176+0000","last_peered":"2024-02-13T00:42:13.351176+0000","last_clean":"2024-02-13T00:42:13.351176+0000","last_became_active":"2024-02-13T00:37:47.259688+0000","last_became_peered":"2024-02-13T00:37:47.259688+0000","last_unstale":"2024-02-13T00:42:13.351176+0000","last_undegraded":"2024-02-13T00:42:13.351176+0000","last_fullsized":"2024-02-13T00:42:13.351176+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:39:22.338042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":82,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346968+0000","last_change":"2024-02-13T00:41:01.620536+0000","last_active":"2024-02-13T00:42:13.346968+0000","last_peered":"2024-02-13T00:42:13.346968+0000","last_clean":"2024-02-13T00:42:13.346968+0000","last_became_active":"2024-02-13T00:41:01.620258+0000","last_became_peered":"2024-02-13T00:41:01.620258+0000","last_unstale":"2024-02-13T00:42:13.346968+0000","last_undegraded":"2024-02-13T00:42:13.346968+0000","last_fullsized":"2024-02-13T00:42:13.346968+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":81,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.346797+0000","last_change":"2024-02-13T00:41:01.575407+0000","last_active":"2024-02-13T00:42:13.346797+0000","last_peered":"2024-02-13T00:42:13.346797+0000","last_clean":"2024-02-13T00:42:13.346797+0000","last_became_active":"2024-02-13T00:41:01.575117+0000","last_became_peered":"2024-02-13T00:41:01.575117+0000","last_unstale":"2024-02-13T00:42:13.346797+0000","last_undegraded":"2024-02-13T00:42:13.346797+0000","last_fullsized":"2024-02-13T00:42:13.346797+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"85'14","reported_seq":160,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586537+0000","last_change":"2024-02-13T00:41:47.734999+0000","last_active":"2024-02-13T00:42:13.586537+0000","last_peered":"2024-02-13T00:42:13.586537+0000","last_clean":"2024-02-13T00:42:13.586537+0000","last_became_active":"2024-02-13T00:40:35.856715+0000","last_became_peered":"2024-02-13T00:40:35.856715+0000","last_unstale":"2024-02-13T00:42:13.586537+0000","last_undegraded":"2024-02-13T00:42:13.586537+0000","last_fullsized":"2024-02-13T00:42:13.586537+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:41:47.734904+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":144,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586614+0000","last_change":"2024-02-13T00:42:03.434362+0000","last_active":"2024-02-13T00:42:13.586614+0000","last_peered":"2024-02-13T00:42:13.586614+0000","last_clean":"2024-02-13T00:42:13.586614+0000","last_became_active":"2024-02-13T00:42:03.433827+0000","last_became_peered":"2024-02-13T00:42:03.433827+0000","last_unstale":"2024-02-13T00:42:13.586614+0000","last_undegraded":"2024-02-13T00:42:13.586614+0000","last_fullsized":"2024-02-13T00:42:13.586614+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:39:21.165246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":136,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.583287+0000","last_change":"2024-02-13T00:40:07.088136+0000","last_active":"2024-02-13T00:42:13.583287+0000","last_peered":"2024-02-13T00:42:13.583287+0000","last_clean":"2024-02-13T00:42:13.583287+0000","last_became_active":"2024-02-13T00:37:45.246184+0000","last_became_peered":"2024-02-13T00:37:45.246184+0000","last_unstale":"2024-02-13T00:42:13.583287+0000","last_undegraded":"2024-02-13T00:42:13.583287+0000","last_fullsized":"2024-02-13T00:42:13.583287+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:07.088053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:07.088053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":134,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.342923+0000","last_change":"2024-02-13T00:40:13.678873+0000","last_active":"2024-02-13T00:42:13.342923+0000","last_peered":"2024-02-13T00:42:13.342923+0000","last_clean":"2024-02-13T00:42:13.342923+0000","last_became_active":"2024-02-13T00:37:47.260025+0000","last_became_peered":"2024-02-13T00:37:47.260025+0000","last_unstale":"2024-02-13T00:42:13.342923+0000","last_undegraded":"2024-02-13T00:42:13.342923+0000","last_fullsized":"2024-02-13T00:42:13.342923+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:13.678807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:40:13.678807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.d","version":"108'5","reported_seq":87,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584183+0000","last_change":"2024-02-13T00:41:01.688873+0000","last_active":"2024-02-13T00:42:13.584183+0000","last_peered":"2024-02-13T00:42:13.584183+0000","last_clean":"2024-02-13T00:42:13.584183+0000","last_became_active":"2024-02-13T00:41:01.688580+0000","last_became_peered":"2024-02-13T00:41:01.688580+0000","last_unstale":"2024-02-13T00:42:13.584183+0000","last_undegraded":"2024-02-13T00:42:13.584183+0000","last_fullsized":"2024-02-13T00:42:13.584183+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582752+0000","last_change":"2024-02-13T00:42:03.421775+0000","last_active":"2024-02-13T00:42:13.582752+0000","last_peered":"2024-02-13T00:42:13.582752+0000","last_clean":"2024-02-13T00:42:13.582752+0000","last_became_active":"2024-02-13T00:42:03.421135+0000","last_became_peered":"2024-02-13T00:42:03.421135+0000","last_unstale":"2024-02-13T00:42:13.582752+0000","last_undegraded":"2024-02-13T00:42:13.582752+0000","last_fullsized":"2024-02-13T00:42:13.582752+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"90'20","reported_seq":106,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.587108+0000","last_change":"2024-02-13T00:42:09.194378+0000","last_active":"2024-02-13T00:42:13.587108+0000","last_peered":"2024-02-13T00:42:13.587108+0000","last_clean":"2024-02-13T00:42:13.587108+0000","last_became_active":"2024-02-13T00:42:03.431355+0000","last_became_peered":"2024-02-13T00:42:03.431355+0000","last_unstale":"2024-02-13T00:42:13.587108+0000","last_undegraded":"2024-02-13T00:42:13.587108+0000","last_fullsized":"2024-02-13T00:42:13.587108+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:40:13.144451+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":135,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.587977+0000","last_change":"2024-02-13T00:39:40.280385+0000","last_active":"2024-02-13T00:42:13.587977+0000","last_peered":"2024-02-13T00:42:13.587977+0000","last_clean":"2024-02-13T00:42:13.587977+0000","last_became_active":"2024-02-13T00:37:41.216896+0000","last_became_peered":"2024-02-13T00:37:41.216896+0000","last_unstale":"2024-02-13T00:42:13.587977+0000","last_undegraded":"2024-02-13T00:42:13.587977+0000","last_fullsized":"2024-02-13T00:42:13.587977+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:39:40.280305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":137,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584046+0000","last_change":"2024-02-13T00:41:34.693324+0000","last_active":"2024-02-13T00:42:13.584046+0000","last_peered":"2024-02-13T00:42:13.584046+0000","last_clean":"2024-02-13T00:42:13.584046+0000","last_became_active":"2024-02-13T00:37:45.249914+0000","last_became_peered":"2024-02-13T00:37:45.249914+0000","last_unstale":"2024-02-13T00:42:13.584046+0000","last_undegraded":"2024-02-13T00:42:13.584046+0000","last_fullsized":"2024-02-13T00:42:13.584046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.693214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.693214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":100,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.348858+0000","last_change":"2024-02-13T00:41:47.487953+0000","last_active":"2024-02-13T00:42:13.348858+0000","last_peered":"2024-02-13T00:42:13.348858+0000","last_clean":"2024-02-13T00:42:13.348858+0000","last_became_active":"2024-02-13T00:40:35.845664+0000","last_became_peered":"2024-02-13T00:40:35.845664+0000","last_unstale":"2024-02-13T00:42:13.348858+0000","last_undegraded":"2024-02-13T00:42:13.348858+0000","last_fullsized":"2024-02-13T00:42:13.348858+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:47.487860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:47.487860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":91,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.350689+0000","last_change":"2024-02-13T00:42:03.409916+0000","last_active":"2024-02-13T00:42:13.350689+0000","last_peered":"2024-02-13T00:42:13.350689+0000","last_clean":"2024-02-13T00:42:13.350689+0000","last_became_active":"2024-02-13T00:42:03.409603+0000","last_became_peered":"2024-02-13T00:42:03.409603+0000","last_unstale":"2024-02-13T00:42:13.350689+0000","last_undegraded":"2024-02-13T00:42:13.350689+0000","last_fullsized":"2024-02-13T00:42:13.350689+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":107,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.584507+0000","last_change":"2024-02-13T00:42:03.539776+0000","last_active":"2024-02-13T00:42:13.584507+0000","last_peered":"2024-02-13T00:42:13.584507+0000","last_clean":"2024-02-13T00:42:13.584507+0000","last_became_active":"2024-02-13T00:42:03.399849+0000","last_became_peered":"2024-02-13T00:42:03.399849+0000","last_unstale":"2024-02-13T00:42:13.584507+0000","last_undegraded":"2024-02-13T00:42:13.584507+0000","last_fullsized":"2024-02-13T00:42:13.584507+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"86'50","reported_seq":234,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.582685+0000","last_change":"2024-02-13T00:39:53.119049+0000","last_active":"2024-02-13T00:42:13.582685+0000","last_peered":"2024-02-13T00:42:13.582685+0000","last_clean":"2024-02-13T00:42:13.582685+0000","last_became_active":"2024-02-13T00:37:43.228728+0000","last_became_peered":"2024-02-13T00:37:43.228728+0000","last_unstale":"2024-02-13T00:42:13.582685+0000","last_undegraded":"2024-02-13T00:42:13.582685+0000","last_fullsized":"2024-02-13T00:42:13.582685+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:39:53.118989+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":135,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.352532+0000","last_change":"2024-02-13T00:39:32.313930+0000","last_active":"2024-02-13T00:42:13.352532+0000","last_peered":"2024-02-13T00:42:13.352532+0000","last_clean":"2024-02-13T00:42:13.352532+0000","last_became_active":"2024-02-13T00:37:41.220583+0000","last_became_peered":"2024-02-13T00:37:41.220583+0000","last_unstale":"2024-02-13T00:42:13.352532+0000","last_undegraded":"2024-02-13T00:42:13.352532+0000","last_fullsized":"2024-02-13T00:42:13.352532+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:32.313841+0000","last_clean_scrub_stamp":"2024-02-13T00:39:32.313841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":26,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586660+0000","last_change":"2024-02-13T00:42:03.404545+0000","last_active":"2024-02-13T00:42:13.586660+0000","last_peered":"2024-02-13T00:42:13.586660+0000","last_clean":"2024-02-13T00:42:13.586660+0000","last_became_active":"2024-02-13T00:42:03.404116+0000","last_became_peered":"2024-02-13T00:42:03.404116+0000","last_unstale":"2024-02-13T00:42:13.586660+0000","last_undegraded":"2024-02-13T00:42:13.586660+0000","last_fullsized":"2024-02-13T00:42:13.586660+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:46.418859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:41:46.418859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":143,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.586430+0000","last_change":"2024-02-13T00:42:03.429789+0000","last_active":"2024-02-13T00:42:13.586430+0000","last_peered":"2024-02-13T00:42:13.586430+0000","last_clean":"2024-02-13T00:42:13.586430+0000","last_became_active":"2024-02-13T00:42:03.429290+0000","last_became_peered":"2024-02-13T00:42:03.429290+0000","last_unstale":"2024-02-13T00:42:13.586430+0000","last_undegraded":"2024-02-13T00:42:13.586430+0000","last_fullsized":"2024-02-13T00:42:13.586430+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:54.059274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:54.059274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":82,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-13T00:42:13.350112+0000","last_change":"2024-02-13T00:41:01.573580+0000","last_active":"2024-02-13T00:42:13.350112+0000","last_peered":"2024-02-13T00:42:13.350112+0000","last_clean":"2024-02-13T00:42:13.350112+0000","last_became_active":"2024-02-13T00:41:01.573417+0000","last_became_peered":"2024-02-13T00:41:01.573417+0000","last_unstale":"2024-02-13T00:42:13.350112+0000","last_undegraded":"2024-02-13T00:42:13.350112+0000","last_fullsized":"2024-02-13T00:42:13.350112+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":122,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":120,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":118,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":5},{"poolid":95,"num_pg":32,"stat_sum":{"num_bytes":2281,"num_objects":221,"num_object_clones":141,"num_object_copies":663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":208,"num_read_kb":208,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":1101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3014656,"data_stored":7589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":5227,"num_objects":505,"num_object_clones":400,"num_object_copies":1515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":505,"num_whiteouts":0,"num_read":83,"num_read_kb":1,"num_write":510,"num_write_kb":509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6205440,"data_stored":15681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":922,"ondisk_log_size":922,"up":96,"acting":96,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":6656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1071,"num_read_kb":796,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":676,"ondisk_log_size":676,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":265,"num_read_kb":260,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266745,"num_pgs":158,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305068,"kb_used_data":3744,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440084,"statfs":{"total":95995035648,"available":95682646016,"internally_reserved":0,"allocated":3833856,"data_stored":1415302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":0.90800000000000003,"15min":0.63800000000000001},"min":{"1min":0.29799999999999999,"5min":0.25,"15min":0.25},"max":{"1min":10.682,"5min":10.682,"15min":10.682},"last":0.53200000000000003},{"interface":"front","average":{"1min":3.8380000000000001,"5min":1.1859999999999999,"15min":0.749},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":25.504000000000001,"5min":25.504000000000001,"15min":25.504000000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Tue Feb 13 00:41:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6950000000000001,"5min":0.73299999999999998,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":13.347,"5min":13.347,"15min":13.347},"last":0.47999999999999998},{"interface":"front","average":{"1min":2.0299999999999998,"5min":0.85299999999999998,"15min":0.69999999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":13.526999999999999,"5min":13.526999999999999,"15min":13.526999999999999},"last":0.51500000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":0.84799999999999998,"15min":0.64500000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":14.359,"5min":14.359,"15min":14.359},"last":0.47599999999999998},{"interface":"front","average":{"1min":2.5979999999999999,"5min":0.95999999999999996,"15min":0.70399999999999996},"min":{"1min":0.29599999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":21.497,"5min":21.497,"15min":21.497},"last":0.505}]},{"osd":3,"last update":"Tue Feb 13 00:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.4620000000000002,"5min":0.94699999999999995,"15min":0.69799999999999995},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":12.997,"5min":12.997,"15min":12.997},"last":0.499},{"interface":"front","average":{"1min":4.1580000000000004,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.35999999999999999,"5min":0.318,"15min":0.318},"max":{"1min":21.622,"5min":21.622,"15min":21.622},"last":0.441}]},{"osd":4,"last update":"Tue Feb 13 00:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.441,"15min":0.46300000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.61799999999999999,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.439,"15min":0.45900000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.71399999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.41899999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.45700000000000002,"15min":0.47099999999999997},"min":{"1min":0.26600000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46800000000000003,"15min":0.47199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72799999999999998,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.46600000000000003}]},{"osd":6,"last update":"Tue Feb 13 00:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67300000000000004,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51900000000000002},"min":{"1min":0.30199999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.75800000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.45200000000000001}]}]},{"osd":6,"up_from":0,"seq":193273528383,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":156,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300972,"kb_used_data":3748,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444180,"statfs":{"total":95995035648,"available":95686840320,"internally_reserved":0,"allocated":3837952,"data_stored":1419730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:41:20 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":0.82399999999999995,"15min":0.48099999999999998},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":41.954000000000001,"5min":41.954000000000001,"15min":41.954000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.443,"15min":0.34699999999999998},"min":{"1min":0.23400000000000001,"5min":0.215,"15min":0.21199999999999999},"max":{"1min":4.1790000000000003,"5min":4.1790000000000003,"15min":4.1790000000000003},"last":0.28000000000000003}]},{"osd":2,"last update":"Tue Feb 13 00:41:48 2024","interfaces":[{"interface":"back","average":{"1min":6.6529999999999996,"5min":1.6299999999999999,"15min":0.79000000000000004},"min":{"1min":0.23599999999999999,"5min":0.215,"15min":0.215},"max":{"1min":32.161000000000001,"5min":32.161000000000001,"15min":32.161000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":6.7670000000000003,"5min":1.677,"15min":0.81299999999999994},"min":{"1min":0.28599999999999998,"5min":0.251,"15min":0.251},"max":{"1min":29.693000000000001,"5min":29.693000000000001,"15min":29.693000000000001},"last":0.29299999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:41:18 2024","interfaces":[{"interface":"back","average":{"1min":2.6869999999999998,"5min":0.86399999999999999,"15min":0.58899999999999997},"min":{"1min":0.254,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":32.200000000000003,"5min":32.200000000000003,"15min":32.200000000000003},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.3480000000000001,"5min":0.60499999999999998,"15min":0.51700000000000002},"min":{"1min":0.26000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":12.355,"5min":12.355,"15min":12.355},"last":0.33100000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:41:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1739999999999999,"5min":1.0820000000000001,"15min":0.74099999999999999},"min":{"1min":0.435,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":16.547000000000001,"5min":16.547000000000001,"15min":16.547000000000001},"last":0.502},{"interface":"front","average":{"1min":1.9159999999999999,"5min":0.84999999999999998,"15min":0.65700000000000003},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":14.731999999999999,"5min":14.731999999999999,"15min":14.731999999999999},"last":0.38200000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:41:48 2024","interfaces":[{"interface":"back","average":{"1min":5.0700000000000003,"5min":1.466,"15min":0.87},"min":{"1min":0.34000000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":26.561,"5min":26.561,"15min":26.561},"last":0.495},{"interface":"front","average":{"1min":3.4809999999999999,"5min":1.165,"15min":0.77800000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":16.494,"5min":16.494,"15min":16.494},"last":0.45200000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:41:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":0.73699999999999999,"15min":0.64600000000000002},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":12.403,"5min":12.403,"15min":12.403},"last":0.46000000000000002},{"interface":"front","average":{"1min":2.5760000000000001,"5min":0.997,"15min":0.74099999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":26.396999999999998,"5min":26.396999999999998,"15min":26.396999999999998},"last":0.42399999999999999}]},{"osd":7,"last update":"Tue Feb 13 00:41:44 2024","interfaces":[{"interface":"back","average":{"1min":2.4420000000000002,"5min":0.96099999999999997,"15min":0.71799999999999997},"min":{"1min":0.45800000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":14.794,"5min":14.794,"15min":14.794},"last":0.41599999999999998},{"interface":"front","average":{"1min":5.1050000000000004,"5min":1.5089999999999999,"15min":0.91900000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":26.693999999999999,"5min":26.693999999999999,"15min":26.693999999999999},"last":0.47099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":130,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305008,"kb_used_data":3688,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440144,"statfs":{"total":95995035648,"available":95682707456,"internally_reserved":0,"allocated":3776512,"data_stored":1410146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.38400000000000001,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.189,"15min":0.189},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":9.9190000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.46400000000000002,"15min":0.374},"min":{"1min":0.187,"5min":0.187,"15min":0.17599999999999999},"max":{"1min":4.7409999999999997,"5min":4.7409999999999997,"15min":4.7409999999999997},"last":1.804}]},{"osd":2,"last update":"Tue Feb 13 00:41:49 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":0.56499999999999995,"15min":0.45100000000000001},"min":{"1min":0.29699999999999999,"5min":0.218,"15min":0.20999999999999999},"max":{"1min":5.9539999999999997,"5min":5.9539999999999997,"15min":5.9539999999999997},"last":4.1230000000000002},{"interface":"front","average":{"1min":3.754,"5min":1.0760000000000001,"15min":0.60899999999999999},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":24.459,"5min":24.459,"15min":24.459},"last":9.4160000000000004}]},{"osd":3,"last update":"Tue Feb 13 00:42:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":0.58399999999999996,"15min":0.46200000000000002},"min":{"1min":0.28100000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.718,"5min":8.718,"15min":8.718},"last":2.2530000000000001},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.55200000000000005,"15min":0.502},"min":{"1min":0.255,"5min":0.247,"15min":0.247},"max":{"1min":5.931,"5min":5.931,"15min":5.931},"last":5.2359999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:41:33 2024","interfaces":[{"interface":"back","average":{"1min":2.9740000000000002,"5min":1.024,"15min":0.68400000000000005},"min":{"1min":0.35499999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":24.381,"5min":24.381,"15min":24.381},"last":9.1509999999999998},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.61299999999999999,"15min":0.54400000000000004},"min":{"1min":0.36099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.7069999999999999,"5min":3.7069999999999999,"15min":3.7069999999999999},"last":4.2599999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:41:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.622,"15min":0.59499999999999997},"min":{"1min":0.34999999999999998,"5min":0.316,"15min":0.316},"max":{"1min":4.7270000000000003,"5min":4.7270000000000003,"15min":4.7270000000000003},"last":1.7729999999999999},{"interface":"front","average":{"1min":4.0810000000000004,"5min":1.256,"15min":0.79600000000000004},"min":{"1min":0.38400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":24.427,"5min":24.427,"15min":24.427},"last":9.1880000000000006}]},{"osd":6,"last update":"Tue Feb 13 00:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.55300000000000005,"15min":0.51900000000000002},"min":{"1min":0.41199999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":2.121},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.60099999999999998,"15min":0.55700000000000005},"min":{"1min":0.42299999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":4.681,"5min":4.681,"15min":4.681},"last":0.51600000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":0.72499999999999998,"15min":0.60699999999999998},"min":{"1min":0.38200000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":8.6679999999999993,"5min":8.6679999999999993,"15min":8.6679999999999993},"last":2.1640000000000001},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.65600000000000003,"15min":0.60599999999999998},"min":{"1min":0.45400000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":4.7110000000000003,"5min":4.7110000000000003,"15min":4.7110000000000003},"last":4.157}]}]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305020,"kb_used_data":3696,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440132,"statfs":{"total":95995035648,"available":95682695168,"internally_reserved":0,"allocated":3784704,"data_stored":1409950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:41:47 2024","interfaces":[{"interface":"back","average":{"1min":2.1619999999999999,"5min":0.71999999999999997,"15min":0.47599999999999998},"min":{"1min":0.26000000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.7769999999999992,"5min":9.7769999999999992,"15min":9.7769999999999992},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.7589999999999999,"5min":0.65900000000000003,"15min":0.44500000000000001},"min":{"1min":0.20599999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":9.8160000000000007,"5min":9.8160000000000007,"15min":9.8160000000000007},"last":0.23200000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:41:47 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":0.65700000000000003,"15min":0.44900000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":10.141,"5min":10.141,"15min":10.141},"last":0.224},{"interface":"front","average":{"1min":1.4650000000000001,"5min":0.60599999999999998,"15min":0.44800000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"last":0.29099999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:42:08 2024","interfaces":[{"interface":"back","average":{"1min":1.9910000000000001,"5min":0.72599999999999998,"15min":0.52300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":11.807,"5min":11.807,"15min":11.807},"last":0.30399999999999999},{"interface":"front","average":{"1min":2.306,"5min":0.80100000000000005,"15min":0.56499999999999995},"min":{"1min":0.26600000000000001,"5min":0.253,"15min":0.253},"max":{"1min":16.981000000000002,"5min":16.981000000000002,"15min":16.981000000000002},"last":0.32000000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:41:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.59999999999999998,"15min":0.57999999999999996},"min":{"1min":0.377,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.286,"5min":2.286,"15min":2.286},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.65000000000000002,"15min":0.61299999999999999},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.8959999999999999,"5min":3.8959999999999999,"15min":3.8959999999999999},"last":0.372}]},{"osd":5,"last update":"Tue Feb 13 00:41:55 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.628,"15min":0.58699999999999997},"min":{"1min":0.35099999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":6.2149999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.1759999999999999,"5min":0.68899999999999995,"15min":0.63100000000000001},"min":{"1min":0.38700000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.4640000000000004,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.436}]},{"osd":6,"last update":"Tue Feb 13 00:42:14 2024","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":0.749,"15min":0.64000000000000001},"min":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":9.0199999999999996,"5min":9.0199999999999996,"15min":9.0199999999999996},"last":0.35699999999999998},{"interface":"front","average":{"1min":2.0270000000000001,"5min":0.84999999999999998,"15min":0.66900000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":9.5670000000000002,"5min":9.5670000000000002,"15min":9.5670000000000002},"last":0.42199999999999999}]},{"osd":7,"last update":"Tue Feb 13 00:41:41 2024","interfaces":[{"interface":"back","average":{"1min":1.8089999999999999,"5min":0.81699999999999995,"15min":0.66900000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":9.6099999999999994,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.091,"5min":0.67700000000000005,"15min":0.61299999999999999},"min":{"1min":0.47699999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.1829999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"last":0.379}]}]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":133,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304720,"kb_used_data":3400,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440432,"statfs":{"total":95995035648,"available":95683002368,"internally_reserved":0,"allocated":3481600,"data_stored":1418542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:42:09 2024","interfaces":[{"interface":"back","average":{"1min":5.4299999999999997,"5min":1.3720000000000001,"15min":0.71299999999999997},"min":{"1min":0.20100000000000001,"5min":0.191,"15min":0.191},"max":{"1min":42.459000000000003,"5min":42.459000000000003,"15min":42.459000000000003},"last":0.314},{"interface":"front","average":{"1min":5.4359999999999999,"5min":1.3899999999999999,"15min":0.73999999999999999},"min":{"1min":0.23499999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":48.088999999999999,"5min":48.088999999999999,"15min":48.088999999999999},"last":0.29999999999999999}]},{"osd":1,"last update":"Tue Feb 13 00:42:09 2024","interfaces":[{"interface":"back","average":{"1min":3.6579999999999999,"5min":1.0409999999999999,"15min":0.61699999999999999},"min":{"1min":0.249,"5min":0.217,"15min":0.217},"max":{"1min":47.999000000000002,"5min":47.999000000000002,"15min":47.999000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":5.077,"5min":1.381,"15min":0.72899999999999998},"min":{"1min":0.20200000000000001,"5min":0.183,"15min":0.183},"max":{"1min":32.973999999999997,"5min":32.973999999999997,"15min":32.973999999999997},"last":0.33000000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:42:09 2024","interfaces":[{"interface":"back","average":{"1min":4.3730000000000002,"5min":1.2629999999999999,"15min":0.70399999999999996},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":29.661999999999999,"5min":29.661999999999999,"15min":29.661999999999999},"last":0.372},{"interface":"front","average":{"1min":3.948,"5min":1.1140000000000001,"15min":0.64400000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":35.256999999999998,"5min":35.256999999999998,"15min":35.256999999999998},"last":0.38900000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:41:34 2024","interfaces":[{"interface":"back","average":{"1min":4.0640000000000001,"5min":1.2410000000000001,"15min":0.77600000000000002},"min":{"1min":0.44800000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":35.156999999999996,"5min":35.156999999999996,"15min":35.156999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.629,"15min":0.56999999999999995},"min":{"1min":0.436,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.9420000000000002,"5min":3.9420000000000002,"15min":3.9420000000000002},"last":0.41299999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:41:57 2024","interfaces":[{"interface":"back","average":{"1min":4.6429999999999998,"5min":1.3720000000000001,"15min":0.78800000000000003},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":31.420000000000002,"5min":31.420000000000002,"15min":31.420000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":4.4450000000000003,"5min":1.3260000000000001,"15min":0.78400000000000003},"min":{"1min":0.45300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":35.209000000000003,"5min":35.209000000000003,"15min":35.209000000000003},"last":0.58199999999999996}]},{"osd":6,"last update":"Tue Feb 13 00:41:18 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":0.75900000000000001,"15min":0.59099999999999997},"min":{"1min":0.46500000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":14.438000000000001,"5min":14.438000000000001,"15min":14.438000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":2.5179999999999998,"5min":0.94099999999999995,"15min":0.67600000000000005},"min":{"1min":0.42599999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":35.280000000000001,"5min":35.280000000000001,"15min":35.280000000000001},"last":0.61399999999999999}]},{"osd":7,"last update":"Tue Feb 13 00:41:43 2024","interfaces":[{"interface":"back","average":{"1min":4.4790000000000001,"5min":1.3400000000000001,"15min":0.81000000000000005},"min":{"1min":0.47999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":35.234000000000002,"5min":35.234000000000002,"15min":35.234000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.0620000000000001,"5min":0.66300000000000003,"15min":0.58299999999999996},"min":{"1min":0.51900000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":3.9129999999999998,"5min":3.9129999999999998,"15min":3.9129999999999998},"last":0.626}]}]},{"osd":4,"up_from":31,"seq":133143986249,"num_pgs":122,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305152,"kb_used_data":3832,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440000,"statfs":{"total":95995035648,"available":95682560000,"internally_reserved":0,"allocated":3923968,"data_stored":1411475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.631,"5min":0.73399999999999999,"15min":0.58999999999999997},"min":{"1min":0.30399999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":9.3399999999999999,"5min":9.3399999999999999,"15min":9.3399999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.369,"5min":0.68899999999999995,"15min":0.59599999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":13.833,"5min":13.833,"15min":13.833},"last":0.45100000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:41:38 2024","interfaces":[{"interface":"back","average":{"1min":3.7570000000000001,"5min":1.165,"15min":0.745},"min":{"1min":0.34399999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":53.188000000000002,"5min":53.188000000000002,"15min":53.188000000000002},"last":0.308},{"interface":"front","average":{"1min":2.895,"5min":0.99199999999999999,"15min":0.68700000000000006},"min":{"1min":0.33700000000000002,"5min":0.308,"15min":0.308},"max":{"1min":44.793999999999997,"5min":44.793999999999997,"15min":44.793999999999997},"last":0.46000000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":0.68200000000000005,"15min":0.57999999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":8.6829999999999998,"5min":8.6829999999999998,"15min":8.6829999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.6890000000000001,"5min":0.751,"15min":0.60299999999999998},"min":{"1min":0.36199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":18.911000000000001,"5min":18.911000000000001,"15min":18.911000000000001},"last":0.47599999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:41:39 2024","interfaces":[{"interface":"back","average":{"1min":2.0510000000000002,"5min":0.84899999999999998,"15min":0.63900000000000001},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":23.5,"5min":23.5,"15min":23.5},"last":0.42799999999999999},{"interface":"front","average":{"1min":3.1720000000000002,"5min":1.0529999999999999,"15min":0.70999999999999996},"min":{"1min":0.38200000000000001,"5min":0.311,"15min":0.311},"max":{"1min":36.152999999999999,"5min":36.152999999999999,"15min":36.152999999999999},"last":0.496}]},{"osd":5,"last update":"Tue Feb 13 00:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.39600000000000002,"15min":0.35999999999999999},"min":{"1min":0.24399999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.54300000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.42499999999999999,"15min":0.379},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65500000000000003,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.437}]},{"osd":6,"last update":"Tue Feb 13 00:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.434,"15min":0.40600000000000003},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56399999999999995,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.379},{"interface":"front","average":{"1min":0.377,"5min":0.45200000000000001,"15min":0.41699999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.503,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.35599999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.48299999999999998,"15min":0.46700000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46600000000000003,"15min":0.45500000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.58699999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.60499999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724612,"num_pgs":117,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305212,"kb_used_data":3888,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93439940,"statfs":{"total":95995035648,"available":95682498560,"internally_reserved":0,"allocated":3981312,"data_stored":1411436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:41:56 2024","interfaces":[{"interface":"back","average":{"1min":6.3230000000000004,"5min":1.6599999999999999,"15min":0.876},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":72.444999999999993,"5min":72.444999999999993,"15min":72.444999999999993},"last":0.46600000000000003},{"interface":"front","average":{"1min":3.9860000000000002,"5min":1.2310000000000001,"15min":0.745},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":23.949000000000002},"last":0.48799999999999999}]},{"osd":1,"last update":"Tue Feb 13 00:41:56 2024","interfaces":[{"interface":"back","average":{"1min":3.1190000000000002,"5min":1.0289999999999999,"15min":0.67300000000000004},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":23.937000000000001,"5min":23.937000000000001,"15min":23.937000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":5.0839999999999996,"5min":1.4470000000000001,"15min":0.80900000000000005},"min":{"1min":0.28999999999999998,"5min":0.254,"15min":0.254},"max":{"1min":42.826999999999998,"5min":42.826999999999998,"15min":42.826999999999998},"last":0.45800000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:41:56 2024","interfaces":[{"interface":"back","average":{"1min":8.3810000000000002,"5min":2.0920000000000001,"15min":1.032},"min":{"1min":0.35099999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":54.683999999999997,"5min":54.683999999999997,"15min":54.683999999999997},"last":0.51000000000000001},{"interface":"front","average":{"1min":6.5190000000000001,"5min":1.72,"15min":0.90800000000000003},"min":{"1min":0.34100000000000003,"5min":0.316,"15min":0.316},"max":{"1min":31.422999999999998,"5min":31.422999999999998,"15min":31.422999999999998},"last":0.499}]},{"osd":3,"last update":"Tue Feb 13 00:41:56 2024","interfaces":[{"interface":"back","average":{"1min":10.406000000000001,"5min":2.5640000000000001,"15min":1.1879999999999999},"min":{"1min":0.33100000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":61.573,"5min":61.573,"15min":61.573},"last":0.55200000000000005},{"interface":"front","average":{"1min":7.2930000000000001,"5min":1.8859999999999999,"15min":0.96099999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":54.848999999999997,"5min":54.848999999999997,"15min":54.848999999999997},"last":0.434}]},{"osd":4,"last update":"Tue Feb 13 00:41:56 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.52200000000000002,"15min":0.40100000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":7.8570000000000002,"5min":7.8570000000000002,"15min":7.8570000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.623,"15min":0.45900000000000002},"min":{"1min":0.27900000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.0630000000000006,"5min":8.0630000000000006,"15min":8.0630000000000006},"last":0.44600000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:41:15 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.53800000000000003,"15min":0.46000000000000002},"min":{"1min":0.30399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":8.1020000000000003,"5min":8.1020000000000003,"15min":8.1020000000000003},"last":0.376},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.55800000000000005,"15min":0.47799999999999998},"min":{"1min":0.307,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.7389999999999999,"5min":7.7389999999999999,"15min":7.7389999999999999},"last":0.38800000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.68799999999999994,"15min":0.78100000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":7.915,"5min":7.915,"15min":7.915},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.61199999999999999,"15min":0.59199999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":8.0169999999999995,"5min":8.0169999999999995,"15min":8.0169999999999995},"last":0.47499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":1880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":745472,"data_stored":1884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":913408,"data_stored":2305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":1466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1056768,"data_stored":2667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1146880,"data_stored":2912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1019904,"data_stored":2567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":1322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":118,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:42:18.540 INFO:tasks.thrashosds.thrasher:Setting 95.16 to [4, 3, 7] 2024-02-13T00:42:18.540 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '95.16', '4', '3', '7'] 2024-02-13T00:42:18.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pg-upmap 95.16 4 3 7 2024-02-13T00:42:18.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-13T00:42:18.666 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:18.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:18.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:18.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:42:18.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:18.797 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:18.828 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:18.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:18.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:18.922 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:42:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:42:18.931 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:42:18.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:18.959 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:17.411961+0000 osd.2 (osd.2) 79 : cluster [DBG] 4.d scrub starts 2024-02-13T00:42:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:17.413821+0000 osd.2 (osd.2) 80 : cluster [DBG] 4.d scrub ok 2024-02-13T00:42:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:17.618892+0000 mgr.y (mgr.14544) 214 : audit [DBG] from='client.26550 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:42:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:18.109249+0000 mon.b (mon.1) 231 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:18.347629+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]': finished 2024-02-13T00:42:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:18.347696+0000 mon.a (mon.0) 1311 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83985-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:18.347787+0000 mon.a (mon.0) 1312 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-83977-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:18.347853+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e183: 8 total, 8 up, 7 in 2024-02-13T00:42:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:18.353649+0000 mon.c (mon.2) 158 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:18.354571+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:18.357512+0000 mon.a (mon.0) 1315 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-83977-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:18.461502+0000 osd.3 (osd.3) 115 : cluster [DBG] 2.1e scrub starts 2024-02-13T00:42:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:18.464398+0000 osd.3 (osd.3) 116 : cluster [DBG] 2.1e scrub ok 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:17.411961+0000 osd.2 (osd.2) 79 : cluster [DBG] 4.d scrub starts 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:17.413821+0000 osd.2 (osd.2) 80 : cluster [DBG] 4.d scrub ok 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:17.618892+0000 mgr.y (mgr.14544) 214 : audit [DBG] from='client.26550 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:18.109249+0000 mon.b (mon.1) 231 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:18.347629+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]': finished 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:18.347696+0000 mon.a (mon.0) 1311 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83985-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:18.347787+0000 mon.a (mon.0) 1312 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-83977-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:18.347853+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e183: 8 total, 8 up, 7 in 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:18.353649+0000 mon.c (mon.2) 158 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:18.354571+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:18.357512+0000 mon.a (mon.0) 1315 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-83977-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:18.461502+0000 osd.3 (osd.3) 115 : cluster [DBG] 2.1e scrub starts 2024-02-13T00:42:19.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:18.464398+0000 osd.3 (osd.3) 116 : cluster [DBG] 2.1e scrub ok 2024-02-13T00:42:19.222 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:17.411961+0000 osd.2 (osd.2) 79 : cluster [DBG] 4.d scrub starts 2024-02-13T00:42:19.222 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:17.413821+0000 osd.2 (osd.2) 80 : cluster [DBG] 4.d scrub ok 2024-02-13T00:42:19.222 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:17.618892+0000 mgr.y (mgr.14544) 214 : audit [DBG] from='client.26550 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:42:19.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:18.109249+0000 mon.b (mon.1) 231 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:19.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:18.347629+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-84081-15"}]': finished 2024-02-13T00:42:19.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:18.347696+0000 mon.a (mon.0) 1311 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-83985-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:19.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:18.347787+0000 mon.a (mon.0) 1312 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-83977-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:19.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:18.347853+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e183: 8 total, 8 up, 7 in 2024-02-13T00:42:19.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:18.353649+0000 mon.c (mon.2) 158 : audit [INF] from='client.? 172.21.15.162:0/1785390881' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:19.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:18.354571+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]: dispatch 2024-02-13T00:42:19.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:18.357512+0000 mon.a (mon.0) 1315 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-83977-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:19.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:18.461502+0000 osd.3 (osd.3) 115 : cluster [DBG] 2.1e scrub starts 2024-02-13T00:42:19.223 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:18.464398+0000 osd.3 (osd.3) 116 : cluster [DBG] 2.1e scrub ok 2024-02-13T00:42:19.310 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:19.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:19.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:19.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscP api_snapshots_pp: Running main() from gmock_main.cc 2024-02-13T00:42:19.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [==========] Running 20 tests from 5 test suites. 2024-02-13T00:42:19.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-02-13T00:42:19.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-02-13T00:42:19.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: seed 84081 2024-02-13T00:42:19.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-02-13T00:42:19.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2521 ms) 2024-02-13T00:42:19.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-02-13T00:42:19.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2969 ms) 2024-02-13T00:42:19.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-02-13T00:42:19.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (3633 ms) 2024-02-13T00:42:19.361 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-02-13T00:42:19.361 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2005 ms) 2024-02-13T00:42:19.361 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-02-13T00:42:19.361 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4055 ms) 2024-02-13T00:42:19.361 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (15183 ms total) 2024-02-13T00:42:19.361 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: 2024-02-13T00:42:19.361 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP 2024-02-13T00:42:19.361 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-02-13T00:42:19.361 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4241 ms) 2024-02-13T00:42:19.361 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4309 ms) 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6137 ms) 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4257 ms) 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (1875 ms) 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi162-84081-7 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: waiting for snaps to purge 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18493 ms) 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP (39312 ms total) 2024-02-13T00:42:19.362 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (7 ms) 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5117 ms) 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5124 ms total) 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2905 ms) 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2017 ms) 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-02-13T00:42:19.363 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2015 ms) 2024-02-13T00:42:19.364 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-02-13T00:42:19.364 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1995 ms) 2024-02-13T00:42:19.364 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8932 ms total) 2024-02-13T00:42:19.364 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: 2024-02-13T00:42:19.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:19.446 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:19.499 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:19.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:19.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:19.633 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:19.683 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:19.781 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:19.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:19.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:19.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.907+0000 7f14c1a9a700 1 -- 172.21.15.162:0/441897952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14bc0f8d10 msgr2=0x7f14bc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:19.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.907+0000 7f14c1a9a700 1 --2- 172.21.15.162:0/441897952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14bc0f8d10 0x7f14bc0f90f0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 rx=0x7f14ac004870 tx=0x7f14ac009cb0).stop 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.907+0000 7f14c1a9a700 1 -- 172.21.15.162:0/441897952 shutdown_connections 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.907+0000 7f14c1a9a700 1 --2- 172.21.15.162:0/441897952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14bc102270 0x7f14bc104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.907+0000 7f14c1a9a700 1 --2- 172.21.15.162:0/441897952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14bc0f8d10 0x7f14bc0f90f0 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.907+0000 7f14c1a9a700 1 --2- 172.21.15.162:0/441897952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14bc0f95e0 0x7f14bc101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.907+0000 7f14c1a9a700 1 -- 172.21.15.162:0/441897952 >> 172.21.15.162:0/441897952 conn(0x7f14bc06eb90 msgr2=0x7f14bc0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.907+0000 7f14c1a9a700 1 -- 172.21.15.162:0/441897952 shutdown_connections 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.907+0000 7f14c1a9a700 1 -- 172.21.15.162:0/441897952 wait complete. 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.908+0000 7f14c1a9a700 1 librados: starting msgr at 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.908+0000 7f14c1a9a700 1 librados: starting objecter 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.908+0000 7f14c1a9a700 1 Processor -- start 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.908+0000 7f14c1a9a700 1 -- start start 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.908+0000 7f14c1a9a700 1 librados: setting wanted keys 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.908+0000 7f14c1a9a700 1 librados: calling monclient init 2024-02-13T00:42:19.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.908+0000 7f14c1a9a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14bc0f8d10 0x7f14bc0f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.908+0000 7f14c1a9a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14bc0f95e0 0x7f14bc0f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.908+0000 7f14c1a9a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14bc102270 0x7f14bc0f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.908+0000 7f14c1a9a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc116640 con 0x7f14bc0f8d10 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.908+0000 7f14c1a9a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc1167b0 con 0x7f14bc102270 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.908+0000 7f14c1a9a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc116970 con 0x7f14bc0f95e0 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14baffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14bc0f95e0 0x7f14bc0f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14baffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14bc0f95e0 0x7f14bc0f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:53614/0 (socket says 172.21.15.162:53614) 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14baffd700 1 -- 172.21.15.162:0/3431329757 learned_addr learned my addr 172.21.15.162:0/3431329757 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14bb7fe700 1 --2- 172.21.15.162:0/3431329757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14bc0f8d10 0x7f14bc0f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14bbfff700 1 --2- 172.21.15.162:0/3431329757 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14bc102270 0x7f14bc0f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14baffd700 1 -- 172.21.15.162:0/3431329757 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14bc102270 msgr2=0x7f14bc0f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14baffd700 1 --2- 172.21.15.162:0/3431329757 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14bc102270 0x7f14bc0f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14baffd700 1 -- 172.21.15.162:0/3431329757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14bc0f8d10 msgr2=0x7f14bc0f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14baffd700 1 --2- 172.21.15.162:0/3431329757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14bc0f8d10 0x7f14bc0f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14baffd700 1 -- 172.21.15.162:0/3431329757 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b40097c0 con 0x7f14bc0f95e0 2024-02-13T00:42:19.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14bb7fe700 1 --2- 172.21.15.162:0/3431329757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14bc0f8d10 0x7f14bc0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:42:19.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14baffd700 1 --2- 172.21.15.162:0/3431329757 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14bc0f95e0 0x7f14bc0f71b0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 rx=0x7f14b00027f0 tx=0x7f14b0002b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:19.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14b8f74700 10 client.?.objecter ms_handle_connect 0x7f14bc0f95e0 2024-02-13T00:42:19.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14b8f74700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:19.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14b8f74700 1 -- 172.21.15.162:0/3431329757 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0007b40 con 0x7f14bc0f95e0 2024-02-13T00:42:19.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14b8f74700 1 -- 172.21.15.162:0/3431329757 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14b000f040 con 0x7f14bc0f95e0 2024-02-13T00:42:19.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14c1a9a700 1 -- 172.21.15.162:0/3431329757 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac005040 con 0x7f14bc0f95e0 2024-02-13T00:42:19.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14b8f74700 1 -- 172.21.15.162:0/3431329757 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b00146f0 con 0x7f14bc0f95e0 2024-02-13T00:42:19.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14c1a9a700 10 client.36671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:19.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14c1a9a700 1 -- 172.21.15.162:0/3431329757 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14bc117000 con 0x7f14bc0f95e0 2024-02-13T00:42:19.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.909+0000 7f14c1a9a700 1 librados: init done 2024-02-13T00:42:19.916 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:19.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.910+0000 7f14c1a9a700 1 -- 172.21.15.162:0/3431329757 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14bc0475e0 con 0x7f14bc0f95e0 2024-02-13T00:42:19.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.911+0000 7f14b8f74700 1 -- 172.21.15.162:0/3431329757 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f14b0009110 con 0x7f14bc0f95e0 2024-02-13T00:42:19.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.911+0000 7f14b8f74700 1 --2- 172.21.15.162:0/3431329757 >> v2:172.21.15.162:6800/2866534053 conn(0x7f14a4062c90 0x7f14a4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:19.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.911+0000 7f14b8f74700 1 -- 172.21.15.162:0/3431329757 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(184..184 src has 52..184) v4 ==== 8962+0+0 (secure 0 0 0) 0x7f14b007d270 con 0x7f14bc0f95e0 2024-02-13T00:42:19.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.911+0000 7f14bb7fe700 1 --2- 172.21.15.162:0/3431329757 >> v2:172.21.15.162:6800/2866534053 conn(0x7f14a4062c90 0x7f14a4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:19.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.911+0000 7f14b8f74700 10 client.36671.objecter ms_dispatch 0x7f14bc001550 osd_map(184..184 src has 52..184) v4 2024-02-13T00:42:19.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.911+0000 7f14b8f74700 3 client.36671.objecter handle_osd_map got epochs [184,184] > 0 2024-02-13T00:42:19.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.911+0000 7f14b8f74700 3 client.36671.objecter handle_osd_map decoding full epoch 184 2024-02-13T00:42:19.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.911+0000 7f14b8f74700 20 client.36671.objecter dump_active .. 0 homeless 2024-02-13T00:42:19.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.911+0000 7f14b8f74700 10 client.36671.objecter ms_handle_connect 0x7f14a4062c90 2024-02-13T00:42:19.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.911+0000 7f14bb7fe700 1 --2- 172.21.15.162:0/3431329757 >> v2:172.21.15.162:6800/2866534053 conn(0x7f14a4062c90 0x7f14a4065150 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 rx=0x7f14ac0043d0 tx=0x7f14ac00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:19.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:19.917+0000 7f14b8f74700 1 -- 172.21.15.162:0/3431329757 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f14b0050350 con 0x7f14bc0f95e0 2024-02-13T00:42:19.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:18.332908+0000 mgr.y (mgr.14544) 215 : cluster [DBG] pgmap v244: 332 pgs: 64 unknown, 8 creating+peering, 260 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:42:20.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.099+0000 7f14c1a9a700 1 -- 172.21.15.162:0/3431329757 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f14bc05fb80 con 0x7f14bc0f95e0 2024-02-13T00:42:20.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:18.332908+0000 mgr.y (mgr.14544) 215 : cluster [DBG] pgmap v244: 332 pgs: 64 unknown, 8 creating+peering, 260 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:42:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:18.912668+0000 mon.a (mon.0) 1316 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:42:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:19.110214+0000 mon.b (mon.1) 232 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:19.351185+0000 mon.a (mon.0) 1317 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]': finished 2024-02-13T00:42:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:19.373140+0000 mon.c (mon.2) 159 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:19.595620+0000 mon.a (mon.0) 1318 : cluster [DBG] osdmap e184: 8 total, 8 up, 7 in 2024-02-13T00:42:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:19.596762+0000 mon.b (mon.1) 233 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:19.596938+0000 mon.a (mon.0) 1319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:19.600531+0000 mon.c (mon.2) 160 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:19.601418+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:19.601933+0000 mon.a (mon.0) 1321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:19.601975+0000 mon.c (mon.2) 161 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:19.602215+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:20.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:19.872570+0000 mon.a (mon.0) 1323 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:20.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.320+0000 7f14b8f74700 1 -- 172.21.15.162:0/3431329757 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v185) v1 ==== 101+0+0 (secure 0 0 0) 0x7f14b001b300 con 0x7f14bc0f95e0 2024-02-13T00:42:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:18.912668+0000 mon.a (mon.0) 1316 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:42:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:19.110214+0000 mon.b (mon.1) 232 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:19.351185+0000 mon.a (mon.0) 1317 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]': finished 2024-02-13T00:42:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:19.373140+0000 mon.c (mon.2) 159 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:19.595620+0000 mon.a (mon.0) 1318 : cluster [DBG] osdmap e184: 8 total, 8 up, 7 in 2024-02-13T00:42:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:19.596762+0000 mon.b (mon.1) 233 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:19.596938+0000 mon.a (mon.0) 1319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:19.600531+0000 mon.c (mon.2) 160 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:19.601418+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:19.601933+0000 mon.a (mon.0) 1321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:19.601975+0000 mon.c (mon.2) 161 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:19.602215+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:20.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:19.872570+0000 mon.a (mon.0) 1323 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:20.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.326+0000 7f14c1a9a700 10 librados: watch_flush enter 2024-02-13T00:42:20.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.327+0000 7f14c1a9a700 10 librados: watch_flush exit 2024-02-13T00:42:20.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.327+0000 7f14c1a9a700 20 client.36671.objecter shutdown clearing up homeless session... 2024-02-13T00:42:20.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.327+0000 7f14c1a9a700 10 client.36671.objecter successfully canceled tick 2024-02-13T00:42:20.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.327+0000 7f14c1a9a700 1 -- 172.21.15.162:0/3431329757 >> v2:172.21.15.162:6800/2866534053 conn(0x7f14a4062c90 msgr2=0x7f14a4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:20.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.327+0000 7f14c1a9a700 1 --2- 172.21.15.162:0/3431329757 >> v2:172.21.15.162:6800/2866534053 conn(0x7f14a4062c90 0x7f14a4065150 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 rx=0x7f14ac0043d0 tx=0x7f14ac00b040).stop 2024-02-13T00:42:20.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.328+0000 7f14c1a9a700 1 -- 172.21.15.162:0/3431329757 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14bc0f95e0 msgr2=0x7f14bc0f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:20.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.328+0000 7f14c1a9a700 1 --2- 172.21.15.162:0/3431329757 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14bc0f95e0 0x7f14bc0f71b0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 rx=0x7f14b00027f0 tx=0x7f14b0002b00).stop 2024-02-13T00:42:20.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:18.332908+0000 mgr.y (mgr.14544) 215 : cluster [DBG] pgmap v244: 332 pgs: 64 unknown, 8 creating+peering, 260 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:42:20.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:18.912668+0000 mon.a (mon.0) 1316 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:42:20.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:19.110214+0000 mon.b (mon.1) 232 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:20.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:19.351185+0000 mon.a (mon.0) 1317 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-84081-15"}]': finished 2024-02-13T00:42:20.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:19.373140+0000 mon.c (mon.2) 159 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:20.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:19.595620+0000 mon.a (mon.0) 1318 : cluster [DBG] osdmap e184: 8 total, 8 up, 7 in 2024-02-13T00:42:20.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:19.596762+0000 mon.b (mon.1) 233 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:20.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:19.596938+0000 mon.a (mon.0) 1319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:20.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:19.600531+0000 mon.c (mon.2) 160 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:20.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:19.601418+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:20.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:19.601933+0000 mon.a (mon.0) 1321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:20.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:19.601975+0000 mon.c (mon.2) 161 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:20.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:19.602215+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:20.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:19.872570+0000 mon.a (mon.0) 1323 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:20.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.328+0000 7f14c1a9a700 1 -- 172.21.15.162:0/3431329757 shutdown_connections 2024-02-13T00:42:20.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.328+0000 7f14c1a9a700 1 --2- 172.21.15.162:0/3431329757 >> v2:172.21.15.162:6800/2866534053 conn(0x7f14a4062c90 0x7f14a4065150 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:20.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.328+0000 7f14c1a9a700 1 --2- 172.21.15.162:0/3431329757 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14bc102270 0x7f14bc0f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:20.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.328+0000 7f14c1a9a700 1 --2- 172.21.15.162:0/3431329757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14bc0f8d10 0x7f14bc0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:20.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.328+0000 7f14c1a9a700 1 --2- 172.21.15.162:0/3431329757 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14bc0f95e0 0x7f14bc0f71b0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:20.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.328+0000 7f14c1a9a700 1 -- 172.21.15.162:0/3431329757 >> 172.21.15.162:0/3431329757 conn(0x7f14bc06eb90 msgr2=0x7f14bc102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:20.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.329+0000 7f14c1a9a700 1 -- 172.21.15.162:0/3431329757 shutdown_connections 2024-02-13T00:42:20.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.329+0000 7f14c1a9a700 1 -- 172.21.15.162:0/3431329757 wait complete. 2024-02-13T00:42:20.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:20.329+0000 7f14c1a9a700 1 librados: shutdown 2024-02-13T00:42:20.335 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:42:20.860 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:20.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:20.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:20.910 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:20.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:20.994 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:21.068 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:21.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:21.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:21.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:21.187 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:42:21.187 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:42:21.199 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:19.405468+0000 osd.2 (osd.2) 81 : cluster [DBG] 28.1a scrub starts 2024-02-13T00:42:21.199 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:19.407004+0000 osd.2 (osd.2) 82 : cluster [DBG] 28.1a scrub ok 2024-02-13T00:42:21.199 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:19.603730+0000 osd.1 (osd.1) 87 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:21.199 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:19.604470+0000 osd.1 (osd.1) 88 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:21.199 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:19.649187+0000 osd.0 (osd.0) 73 : cluster [DBG] 4.b scrub starts 2024-02-13T00:42:21.199 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:19.652685+0000 osd.0 (osd.0) 74 : cluster [DBG] 4.b scrub ok 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:20.101099+0000 mon.c (mon.2) 162 : audit [INF] from='client.? 172.21.15.162:0/3431329757' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:20.101493+0000 mon.a (mon.0) 1324 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:20.112016+0000 mon.b (mon.1) 234 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:20.310502+0000 mon.a (mon.0) 1325 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:20.316066+0000 mon.a (mon.0) 1326 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-83977-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-83977-26"}]': finished 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:20.316147+0000 mon.a (mon.0) 1327 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]': finished 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:20.316203+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:20.316295+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:20.316338+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e185: 8 total, 8 up, 7 in 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:20.318430+0000 mon.b (mon.1) 235 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:20.318646+0000 mon.c (mon.2) 163 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:20.320431+0000 mon.a (mon.0) 1331 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:20.320750+0000 mon.a (mon.0) 1332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:21.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:20.325517+0000 mon.a (mon.0) 1333 : audit [INF] from='client.? 172.21.15.162:0/1305029591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-83985-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:21.201 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:21.221 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:21.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:21.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:21.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:21.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:19.405468+0000 osd.2 (osd.2) 81 : cluster [DBG] 28.1a scrub starts 2024-02-13T00:42:21.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:19.407004+0000 osd.2 (osd.2) 82 : cluster [DBG] 28.1a scrub ok 2024-02-13T00:42:21.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:19.603730+0000 osd.1 (osd.1) 87 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:19.604470+0000 osd.1 (osd.1) 88 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:19.649187+0000 osd.0 (osd.0) 73 : cluster [DBG] 4.b scrub starts 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:19.652685+0000 osd.0 (osd.0) 74 : cluster [DBG] 4.b scrub ok 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:20.101099+0000 mon.c (mon.2) 162 : audit [INF] from='client.? 172.21.15.162:0/3431329757' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:20.101493+0000 mon.a (mon.0) 1324 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:20.112016+0000 mon.b (mon.1) 234 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:20.310502+0000 mon.a (mon.0) 1325 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:20.316066+0000 mon.a (mon.0) 1326 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-83977-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-83977-26"}]': finished 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:20.316147+0000 mon.a (mon.0) 1327 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]': finished 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:20.316203+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:20.316295+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:20.316338+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e185: 8 total, 8 up, 7 in 2024-02-13T00:42:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:20.318430+0000 mon.b (mon.1) 235 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:21.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:19.405468+0000 osd.2 (osd.2) 81 : cluster [DBG] 28.1a scrub starts 2024-02-13T00:42:21.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:19.407004+0000 osd.2 (osd.2) 82 : cluster [DBG] 28.1a scrub ok 2024-02-13T00:42:21.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:19.603730+0000 osd.1 (osd.1) 87 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:21.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:19.604470+0000 osd.1 (osd.1) 88 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:21.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:19.649187+0000 osd.0 (osd.0) 73 : cluster [DBG] 4.b scrub starts 2024-02-13T00:42:21.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:19.652685+0000 osd.0 (osd.0) 74 : cluster [DBG] 4.b scrub ok 2024-02-13T00:42:21.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:20.101099+0000 mon.c (mon.2) 162 : audit [INF] from='client.? 172.21.15.162:0/3431329757' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:21.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:20.101493+0000 mon.a (mon.0) 1324 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:20.112016+0000 mon.b (mon.1) 234 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:20.310502+0000 mon.a (mon.0) 1325 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:42:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:20.316066+0000 mon.a (mon.0) 1326 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-83977-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-83977-26"}]': finished 2024-02-13T00:42:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:20.316147+0000 mon.a (mon.0) 1327 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84049-24"}]': finished 2024-02-13T00:42:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:20.316203+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:20.316295+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:42:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:20.316338+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e185: 8 total, 8 up, 7 in 2024-02-13T00:42:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:20.318430+0000 mon.b (mon.1) 235 : audit [INF] from='client.? 172.21.15.162:0/1409332149' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:21.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:20.318646+0000 mon.c (mon.2) 163 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:21.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:20.320431+0000 mon.a (mon.0) 1331 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:21.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:20.320750+0000 mon.a (mon.0) 1332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:21.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:20.325517+0000 mon.a (mon.0) 1333 : audit [INF] from='client.? 172.21.15.162:0/1305029591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-83985-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:21.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:20.318646+0000 mon.c (mon.2) 163 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:21.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:20.320431+0000 mon.a (mon.0) 1331 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]: dispatch 2024-02-13T00:42:21.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:20.320750+0000 mon.a (mon.0) 1332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:21.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:20.325517+0000 mon.a (mon.0) 1333 : audit [INF] from='client.? 172.21.15.162:0/1305029591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-83985-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:21.351 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:22.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-13T00:42:22.028 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:22.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:22.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:20.099546+0000 osd.1 (osd.1) 89 : cluster [DBG] 2.d scrub starts 2024-02-13T00:42:22.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:20.102466+0000 osd.1 (osd.1) 90 : cluster [DBG] 2.d scrub ok 2024-02-13T00:42:22.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:20.333329+0000 mgr.y (mgr.14544) 216 : cluster [DBG] pgmap v248: 300 pgs: 40 unknown, 260 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:42:22.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:20.683878+0000 osd.0 (osd.0) 75 : cluster [DBG] 4.3 deep-scrub starts 2024-02-13T00:42:22.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:20.688128+0000 osd.0 (osd.0) 76 : cluster [DBG] 4.3 deep-scrub ok 2024-02-13T00:42:22.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:21.112915+0000 mon.b (mon.1) 236 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:22.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:21.316250+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]': finished 2024-02-13T00:42:22.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:21.561338+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? 172.21.15.162:0/1305029591' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-83985-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:22.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:21.561470+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e186: 8 total, 8 up, 7 in 2024-02-13T00:42:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:20.099546+0000 osd.1 (osd.1) 89 : cluster [DBG] 2.d scrub starts 2024-02-13T00:42:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:20.102466+0000 osd.1 (osd.1) 90 : cluster [DBG] 2.d scrub ok 2024-02-13T00:42:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:20.333329+0000 mgr.y (mgr.14544) 216 : cluster [DBG] pgmap v248: 300 pgs: 40 unknown, 260 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:42:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:20.683878+0000 osd.0 (osd.0) 75 : cluster [DBG] 4.3 deep-scrub starts 2024-02-13T00:42:22.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:20.688128+0000 osd.0 (osd.0) 76 : cluster [DBG] 4.3 deep-scrub ok 2024-02-13T00:42:22.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:21.112915+0000 mon.b (mon.1) 236 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:22.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:21.316250+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]': finished 2024-02-13T00:42:22.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:21.561338+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? 172.21.15.162:0/1305029591' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-83985-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:22.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:21.561470+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e186: 8 total, 8 up, 7 in 2024-02-13T00:42:22.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:22.157 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:22.177 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:22.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:22.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:22.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:20.099546+0000 osd.1 (osd.1) 89 : cluster [DBG] 2.d scrub starts 2024-02-13T00:42:22.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:20.102466+0000 osd.1 (osd.1) 90 : cluster [DBG] 2.d scrub ok 2024-02-13T00:42:22.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:20.333329+0000 mgr.y (mgr.14544) 216 : cluster [DBG] pgmap v248: 300 pgs: 40 unknown, 260 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:42:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:20.683878+0000 osd.0 (osd.0) 75 : cluster [DBG] 4.3 deep-scrub starts 2024-02-13T00:42:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:20.688128+0000 osd.0 (osd.0) 76 : cluster [DBG] 4.3 deep-scrub ok 2024-02-13T00:42:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:21.112915+0000 mon.b (mon.1) 236 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:21.316250+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84049-24"}]': finished 2024-02-13T00:42:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:21.561338+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? 172.21.15.162:0/1305029591' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-83985-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:21.561470+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e186: 8 total, 8 up, 7 in 2024-02-13T00:42:22.295 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:42:22.308 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:22.338 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:22.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:22.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:22.469 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:22.567 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:22.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:22.670 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:22.697 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:22.796 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:22.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:22.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:22.926 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:23.174 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:23.175 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:21.068154+0000 osd.1 (osd.1) 91 : cluster [DBG] 4.f scrub starts 2024-02-13T00:42:23.175 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:21.069477+0000 osd.1 (osd.1) 92 : cluster [DBG] 4.f scrub ok 2024-02-13T00:42:23.175 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:21.629792+0000 osd.7 (osd.7) 90 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:42:23.175 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:21.630528+0000 osd.7 (osd.7) 91 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:42:23.176 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:21.647064+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:42:23.176 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:21.647995+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:42:23.176 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:22.113975+0000 mon.b (mon.1) 237 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:23.176 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:22.320602+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]': finished 2024-02-13T00:42:23.176 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:22.320757+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e187: 8 total, 8 up, 7 in 2024-02-13T00:42:23.176 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:22.321993+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:23.176 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:22.326021+0000 mon.c (mon.2) 164 : audit [INF] from='client.? 172.21.15.162:0/2785938865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-84049-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:23.176 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:22.327380+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-84049-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:23.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:23.278 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:23.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 -- 172.21.15.162:0/2243130433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b00fe5b0 msgr2=0x7f16b00f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 --2- 172.21.15.162:0/2243130433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b00fe5b0 0x7f16b00f7de0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 rx=0x7f169c0047b0 tx=0x7f169c009cb0).stop 2024-02-13T00:42:23.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 -- 172.21.15.162:0/2243130433 shutdown_connections 2024-02-13T00:42:23.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 --2- 172.21.15.162:0/2243130433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b00f84c0 0x7f16b0104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 --2- 172.21.15.162:0/2243130433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16b00fdc50 0x7f16b00fe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 --2- 172.21.15.162:0/2243130433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b00fe5b0 0x7f16b00f7de0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 -- 172.21.15.162:0/2243130433 >> 172.21.15.162:0/2243130433 conn(0x7f16b00f38a0 msgr2=0x7f16b00f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:23.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 -- 172.21.15.162:0/2243130433 shutdown_connections 2024-02-13T00:42:23.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 -- 172.21.15.162:0/2243130433 wait complete. 2024-02-13T00:42:23.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 librados: starting msgr at 2024-02-13T00:42:23.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 librados: starting objecter 2024-02-13T00:42:23.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 Processor -- start 2024-02-13T00:42:23.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 -- start start 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 librados: setting wanted keys 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.950+0000 7f16b6deb700 1 librados: calling monclient init 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b6deb700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16b00f84c0 0x7f16b010e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b6deb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b00fdc50 0x7f16b010e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b6deb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b00fe5b0 0x7f16b0108300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b6deb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b01087f0 con 0x7f16b00fe5b0 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b6deb700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b0108960 con 0x7f16b00fdc50 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b6deb700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b0108b20 con 0x7f16b00f84c0 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b4b87700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16b00f84c0 0x7f16b010e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b4b87700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16b00f84c0 0x7f16b010e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:53666/0 (socket says 172.21.15.162:53666) 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b5388700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b00fe5b0 0x7f16b0108300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b4b87700 1 -- 172.21.15.162:0/565243763 learned_addr learned my addr 172.21.15.162:0/565243763 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16affff700 1 --2- 172.21.15.162:0/565243763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b00fdc50 0x7f16b010e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:23.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b4b87700 1 -- 172.21.15.162:0/565243763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b00fdc50 msgr2=0x7f16b010e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b4b87700 1 --2- 172.21.15.162:0/565243763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b00fdc50 0x7f16b010e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b4b87700 1 -- 172.21.15.162:0/565243763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b00fe5b0 msgr2=0x7f16b0108300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b4b87700 1 --2- 172.21.15.162:0/565243763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b00fe5b0 0x7f16b0108300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b4b87700 1 -- 172.21.15.162:0/565243763 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a4005040 con 0x7f16b00f84c0 2024-02-13T00:42:23.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b4b87700 1 --2- 172.21.15.162:0/565243763 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16b00f84c0 0x7f16b010e160 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7f16a4009400 tx=0x7f16a4009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:23.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16adffb700 10 client.?.objecter ms_handle_connect 0x7f16b00f84c0 2024-02-13T00:42:23.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16adffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:23.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4007a40 con 0x7f16b00f84c0 2024-02-13T00:42:23.292 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:42:23.293 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:42:23.293 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:42:23.293 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:42:23.293 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:42:23.293 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:42:23.293 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:42:23.293 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:42:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16a400f040 con 0x7f16b00f84c0 2024-02-13T00:42:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169c005040 con 0x7f16b00f84c0 2024-02-13T00:42:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.951+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4014a70 con 0x7f16b00f84c0 2024-02-13T00:42:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.952+0000 7f16b6deb700 10 client.36683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.952+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b0109160 con 0x7f16b00f84c0 2024-02-13T00:42:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.952+0000 7f16b6deb700 1 librados: init done 2024-02-13T00:42:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.952+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f16b0107170 con 0x7f16b00f84c0 2024-02-13T00:42:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.953+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f16a4010530 con 0x7f16b00f84c0 2024-02-13T00:42:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.953+0000 7f16adffb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6800/2866534053 conn(0x7f16a0062c90 0x7f16a0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.953+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(187..187 src has 52..187) v4 ==== 9827+0+0 (secure 0 0 0) 0x7f16a407d490 con 0x7f16b00f84c0 2024-02-13T00:42:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.953+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 osd_map(187..187 src has 52..187) v4 2024-02-13T00:42:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.953+0000 7f16adffb700 3 client.36683.objecter handle_osd_map got epochs [187,187] > 0 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.953+0000 7f16adffb700 3 client.36683.objecter handle_osd_map decoding full epoch 187 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.953+0000 7f16affff700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6800/2866534053 conn(0x7f16a0062c90 0x7f16a0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.953+0000 7f16adffb700 20 client.36683.objecter dump_active .. 0 homeless 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.953+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v187) v1 ==== 54+0+16 (secure 0 0 0) 0x7f16a4084070 con 0x7f16b00f84c0 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16affff700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6800/2866534053 conn(0x7f16a0062c90 0x7f16a0065150 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 rx=0x7f16b0109aa0 tx=0x7f169c007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16adffb700 10 client.36683.objecter ms_handle_connect 0x7f16a0062c90 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b6deb700 10 client.36683.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 1 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6801/2103743338 conn(0x7f16b00feec0 0x7f16b00ff2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b01097e0 osd=0 3 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b01097e0 osd=0 3 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b01097e0 osd=0 3 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 1 2024-02-13T00:42:23.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b01097e0 osd=0 3 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 0 1 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b01097e0 osd=0 4 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b6deb700 10 client.36683.objecter _send_command 1 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b00ff790 con 0x7f16b00feec0 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b5388700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6801/2103743338 conn(0x7f16b00feec0 0x7f16b00ff2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16adffb700 10 client.36683.objecter ms_handle_connect 0x7f16b00feec0 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.954+0000 7f16b5388700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6801/2103743338 conn(0x7f16b00feec0 0x7f16b00ff2a0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.955+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f16b00ff790 con 0x7f16b00feec0 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.955+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.955+0000 7f16adffb700 10 client.36683.objecter _finish_command 1 = system:0 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.955+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b01097e0 osd=0 4 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.955+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 0 1 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.969+0000 7f16b6deb700 10 client.36683.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:23.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.969+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 2 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.969+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b01097e0 osd=0 3 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.969+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b01097e0 osd=0 3 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.969+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b01097e0 osd=0 3 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.969+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 2 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.969+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b01097e0 osd=0 3 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.969+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 0 2 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.969+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b01097e0 osd=0 4 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.969+0000 7f16b6deb700 10 client.36683.objecter _send_command 2 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.969+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b00ff950 con 0x7f16b00feec0 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f16b00ff950 con 0x7f16b00feec0 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16adffb700 10 client.36683.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b01097e0 osd=0 4 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 0 2 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b6deb700 10 client.36683.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:23.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 3 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6805/150599780 conn(0x7f16b0028590 0x7f16b0119010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b00f6000 osd=1 3 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b00f6000 osd=1 3 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b00f6000 osd=1 3 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 3 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b00f6000 osd=1 3 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 1 3 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b00f6000 osd=1 4 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b6deb700 10 client.36683.objecter _send_command 3 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b00f6170 con 0x7f16b0028590 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b4b87700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6805/150599780 conn(0x7f16b0028590 0x7f16b0119010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16adffb700 10 client.36683.objecter ms_handle_connect 0x7f16b0028590 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.971+0000 7f16b4b87700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6805/150599780 conn(0x7f16b0028590 0x7f16b0119010 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:23.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.972+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f16b00f6170 con 0x7f16b0028590 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.972+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.972+0000 7f16adffb700 10 client.36683.objecter _finish_command 3 = system:0 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.972+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b00f6000 osd=1 4 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.972+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 1 3 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.987+0000 7f16b6deb700 10 client.36683.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.987+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 4 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.987+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b00f6000 osd=1 3 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.987+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b00f6000 osd=1 3 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.987+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b00f6000 osd=1 3 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.987+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 4 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.987+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b00f6000 osd=1 3 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.987+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 1 4 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.987+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b00f6000 osd=1 4 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.987+0000 7f16b6deb700 10 client.36683.objecter _send_command 4 2024-02-13T00:42:23.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.987+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b005a0e0 con 0x7f16b0028590 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f16b005a0e0 con 0x7f16b0028590 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16adffb700 10 client.36683.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b00f6000 osd=1 4 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 1 4 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16b6deb700 10 client.36683.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 5 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6809/3689719376 conn(0x7f16b0059570 0x7f16b011c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0059400 osd=2 3 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0059400 osd=2 3 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0059400 osd=2 3 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 5 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b0059400 osd=2 3 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 2 5 2024-02-13T00:42:23.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0059400 osd=2 4 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16b6deb700 10 client.36683.objecter _send_command 5 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b0028180 con 0x7f16b0059570 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.988+0000 7f16affff700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6809/3689719376 conn(0x7f16b0059570 0x7f16b011c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.989+0000 7f16adffb700 10 client.36683.objecter ms_handle_connect 0x7f16b0059570 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.989+0000 7f16affff700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6809/3689719376 conn(0x7f16b0059570 0x7f16b011c690 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.989+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f16b0028180 con 0x7f16b0059570 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.989+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.989+0000 7f16adffb700 10 client.36683.objecter _finish_command 5 = system:0 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.989+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b0059400 osd=2 4 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:22.989+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 2 5 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.003+0000 7f16b6deb700 10 client.36683.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.003+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 6 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.003+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0059400 osd=2 3 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.003+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0059400 osd=2 3 2024-02-13T00:42:23.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.003+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0059400 osd=2 3 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.003+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 6 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.003+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b0059400 osd=2 3 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.003+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 2 6 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.003+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0059400 osd=2 4 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.003+0000 7f16b6deb700 10 client.36683.objecter _send_command 6 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.003+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b0027650 con 0x7f16b0059570 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.004+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f16b0027650 con 0x7f16b0059570 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.004+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.004+0000 7f16adffb700 10 client.36683.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.004+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b0059400 osd=2 4 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.004+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 2 6 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b6deb700 10 client.36683.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 7 2024-02-13T00:42:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6813/604971011 conn(0x7f16b011dd00 0x7f16b01200f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0119cd0 osd=3 3 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0119cd0 osd=3 3 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0119cd0 osd=3 3 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 7 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b0119cd0 osd=3 3 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 3 7 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0119cd0 osd=3 4 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b6deb700 10 client.36683.objecter _send_command 7 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b011a350 con 0x7f16b011dd00 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b5388700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6813/604971011 conn(0x7f16b011dd00 0x7f16b01200f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16adffb700 10 client.36683.objecter ms_handle_connect 0x7f16b011dd00 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.005+0000 7f16b5388700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6813/604971011 conn(0x7f16b011dd00 0x7f16b01200f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.006+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f16b011a350 con 0x7f16b011dd00 2024-02-13T00:42:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.006+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.006+0000 7f16adffb700 10 client.36683.objecter _finish_command 7 = system:0 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.006+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b0119cd0 osd=3 4 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.006+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 3 7 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.020+0000 7f16b6deb700 10 client.36683.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.020+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 8 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.020+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0119cd0 osd=3 3 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.020+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0119cd0 osd=3 3 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.020+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0119cd0 osd=3 3 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.020+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 8 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.020+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b0119cd0 osd=3 3 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.020+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 3 8 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.020+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0119cd0 osd=3 4 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.020+0000 7f16b6deb700 10 client.36683.objecter _send_command 8 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.020+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b0119740 con 0x7f16b011dd00 2024-02-13T00:42:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f16b0119740 con 0x7f16b011dd00 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16adffb700 10 client.36683.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b0119cd0 osd=3 4 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 3 8 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16b6deb700 10 client.36683.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 9 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6800/3344263388 conn(0x7f16b01198e0 0x7f16b0123770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0119740 osd=4 3 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0119740 osd=4 3 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0119740 osd=4 3 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 9 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b0119740 osd=4 3 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 4 9 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0119740 osd=4 4 2024-02-13T00:42:23.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16b6deb700 10 client.36683.objecter _send_command 9 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.021+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b011d930 con 0x7f16b01198e0 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.022+0000 7f16b4b87700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6800/3344263388 conn(0x7f16b01198e0 0x7f16b0123770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.022+0000 7f16adffb700 10 client.36683.objecter ms_handle_connect 0x7f16b01198e0 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.022+0000 7f16b4b87700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6800/3344263388 conn(0x7f16b01198e0 0x7f16b0123770 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.023+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f16b011d930 con 0x7f16b01198e0 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.023+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.023+0000 7f16adffb700 10 client.36683.objecter _finish_command 9 = system:0 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.023+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b0119740 osd=4 4 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.023+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 4 9 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.037+0000 7f16b6deb700 10 client.36683.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.037+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 10 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.037+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0119740 osd=4 3 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.037+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0119740 osd=4 3 2024-02-13T00:42:23.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.037+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0119740 osd=4 3 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.037+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 10 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.037+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b0119740 osd=4 3 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.037+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 4 10 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.037+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0119740 osd=4 4 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.037+0000 7f16b6deb700 10 client.36683.objecter _send_command 10 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.037+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b011d930 con 0x7f16b01198e0 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f16b011d930 con 0x7f16b01198e0 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16adffb700 10 client.36683.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b0119740 osd=4 4 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 4 10 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16b6deb700 10 client.36683.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 11 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6804/2212296748 conn(0x7f16b0124170 0x7f16b0126560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:23.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b011d930 osd=5 3 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b011d930 osd=5 3 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b011d930 osd=5 3 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 11 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b011d930 osd=5 3 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 5 11 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b011d930 osd=5 4 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16b6deb700 10 client.36683.objecter _send_command 11 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b0129290 con 0x7f16b0124170 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.039+0000 7f16affff700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6804/2212296748 conn(0x7f16b0124170 0x7f16b0126560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.040+0000 7f16affff700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6804/2212296748 conn(0x7f16b0124170 0x7f16b0126560 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.040+0000 7f16adffb700 10 client.36683.objecter ms_handle_connect 0x7f16b0124170 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.040+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f16b0129290 con 0x7f16b0124170 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.040+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:42:23.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.040+0000 7f16adffb700 10 client.36683.objecter _finish_command 11 = system:0 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.040+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b011d930 osd=5 4 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.040+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 5 11 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.055+0000 7f16b6deb700 10 client.36683.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.055+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 12 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.055+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b011d930 osd=5 3 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.055+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b011d930 osd=5 3 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.055+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b011d930 osd=5 3 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.055+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 12 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.055+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b011d930 osd=5 3 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.055+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 5 12 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.055+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b011d930 osd=5 4 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.055+0000 7f16b6deb700 10 client.36683.objecter _send_command 12 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.055+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b0129290 con 0x7f16b0124170 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f16b0129290 con 0x7f16b0124170 2024-02-13T00:42:23.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16adffb700 10 client.36683.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b011d930 osd=5 4 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 5 12 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16b6deb700 10 client.36683.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 13 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6808/1506640341 conn(0x7f16b0128590 0x7f16b0128a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0129290 osd=6 3 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0129290 osd=6 3 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0129290 osd=6 3 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 13 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b0129290 osd=6 3 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 6 13 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0129290 osd=6 4 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16b6deb700 10 client.36683.objecter _send_command 13 2024-02-13T00:42:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.056+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b0128ff0 con 0x7f16b0128590 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.057+0000 7f16b5388700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6808/1506640341 conn(0x7f16b0128590 0x7f16b0128a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.057+0000 7f16b5388700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6808/1506640341 conn(0x7f16b0128590 0x7f16b0128a10 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.057+0000 7f16adffb700 10 client.36683.objecter ms_handle_connect 0x7f16b0128590 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.058+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f16b0128ff0 con 0x7f16b0128590 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.058+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.058+0000 7f16adffb700 10 client.36683.objecter _finish_command 13 = system:0 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.058+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b0129290 osd=6 4 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.058+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 6 13 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.072+0000 7f16b6deb700 10 client.36683.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.072+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 14 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.072+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0129290 osd=6 3 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.072+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0129290 osd=6 3 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.072+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0129290 osd=6 3 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.072+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 14 2024-02-13T00:42:23.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.072+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b0129290 osd=6 3 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.072+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 6 14 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.072+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0129290 osd=6 4 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.072+0000 7f16b6deb700 10 client.36683.objecter _send_command 14 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.072+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b0128ff0 con 0x7f16b0128590 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f16b0128ff0 con 0x7f16b0128590 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16adffb700 10 client.36683.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b0129290 osd=6 4 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 6 14 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16b6deb700 10 client.36683.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 15 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6812/1964192772 conn(0x7f16b0126fe0 0x7f16b0127420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0128ff0 osd=7 3 2024-02-13T00:42:23.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0128ff0 osd=7 3 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0128ff0 osd=7 3 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 15 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b0128ff0 osd=7 3 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 7 15 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0128ff0 osd=7 4 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16b6deb700 10 client.36683.objecter _send_command 15 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.074+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16b0127e50 con 0x7f16b0126fe0 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.075+0000 7f16b4b87700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6812/1964192772 conn(0x7f16b0126fe0 0x7f16b0127420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.075+0000 7f16b4b87700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6812/1964192772 conn(0x7f16b0126fe0 0x7f16b0127420 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.075+0000 7f16adffb700 10 client.36683.objecter ms_handle_connect 0x7f16b0126fe0 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.076+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f16b0127e50 con 0x7f16b0126fe0 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.076+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.076+0000 7f16adffb700 10 client.36683.objecter _finish_command 15 = system:0 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.076+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b0128ff0 osd=7 4 2024-02-13T00:42:23.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.076+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 7 15 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.090+0000 7f16b6deb700 10 client.36683.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.090+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign -1 16 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.090+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0128ff0 osd=7 3 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.090+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0128ff0 osd=7 3 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.090+0000 7f16b6deb700 20 client.36683.objecter _get_session s=0x7f16b0128ff0 osd=7 3 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.090+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_remove -1 16 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.090+0000 7f16b6deb700 20 client.36683.objecter get_session s=0x7f16b0128ff0 osd=7 3 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.090+0000 7f16b6deb700 15 client.36683.objecter _session_command_op_assign 7 16 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.090+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0128ff0 osd=7 4 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.090+0000 7f16b6deb700 10 client.36683.objecter _send_command 16 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.090+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f16b0127e50 con 0x7f16b0126fe0 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16adffb700 1 -- 172.21.15.162:0/565243763 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f16b0127e50 con 0x7f16b0126fe0 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16adffb700 10 client.36683.objecter ms_dispatch 0x7f16b0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:23.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16adffb700 10 client.36683.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16adffb700 20 client.36683.objecter put_session s=0x7f16b0128ff0 osd=7 4 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16adffb700 15 client.36683.objecter _session_command_op_remove 7 16 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 10 librados: watch_flush enter 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 10 librados: watch_flush exit 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 10 client.36683.objecter close_session for osd.0 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6801/2103743338 conn(0x7f16b00feec0 msgr2=0x7f16b00ff2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6801/2103743338 conn(0x7f16b00feec0 0x7f16b00ff2a0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b01097e0 osd=0 1 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 10 client.36683.objecter close_session for osd.1 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6805/150599780 conn(0x7f16b0028590 msgr2=0x7f16b0119010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6805/150599780 conn(0x7f16b0028590 0x7f16b0119010 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b00f6000 osd=1 1 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 10 client.36683.objecter close_session for osd.2 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6809/3689719376 conn(0x7f16b0059570 msgr2=0x7f16b011c690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6809/3689719376 conn(0x7f16b0059570 0x7f16b011c690 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0059400 osd=2 1 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 10 client.36683.objecter close_session for osd.3 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b5388700 1 -- 172.21.15.162:0/565243763 reap_dead start 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6813/604971011 conn(0x7f16b011dd00 msgr2=0x7f16b01200f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6813/604971011 conn(0x7f16b011dd00 0x7f16b01200f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0119cd0 osd=3 1 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 10 client.36683.objecter close_session for osd.4 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6800/3344263388 conn(0x7f16b01198e0 msgr2=0x7f16b0123770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6800/3344263388 conn(0x7f16b01198e0 0x7f16b0123770 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0119740 osd=4 1 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 10 client.36683.objecter close_session for osd.5 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6804/2212296748 conn(0x7f16b0124170 msgr2=0x7f16b0126560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6804/2212296748 conn(0x7f16b0124170 0x7f16b0126560 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b011d930 osd=5 1 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 10 client.36683.objecter close_session for osd.6 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6808/1506640341 conn(0x7f16b0128590 msgr2=0x7f16b0128a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6808/1506640341 conn(0x7f16b0128590 0x7f16b0128a10 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0129290 osd=6 1 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 10 client.36683.objecter close_session for osd.7 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6812/1964192772 conn(0x7f16b0126fe0 msgr2=0x7f16b0127420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.181:6812/1964192772 conn(0x7f16b0126fe0 0x7f16b0127420 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 20 client.36683.objecter put_session s=0x7f16b0128ff0 osd=7 1 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 20 client.36683.objecter shutdown clearing up homeless session... 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b6deb700 10 client.36683.objecter successfully canceled tick 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.092+0000 7f16b5388700 1 -- 172.21.15.162:0/565243763 reap_dead start 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.093+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6800/2866534053 conn(0x7f16a0062c90 msgr2=0x7f16a0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.093+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6800/2866534053 conn(0x7f16a0062c90 0x7f16a0065150 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 rx=0x7f16b0109aa0 tx=0x7f169c007f90).stop 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.093+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16b00f84c0 msgr2=0x7f16b010e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:23.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.093+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16b00f84c0 0x7f16b010e160 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7f16a4009400 tx=0x7f16a4009cb0).stop 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.093+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 shutdown_connections 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.093+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16b00f84c0 0x7f16b010e160 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.093+0000 7f16b6deb700 1 --2- 172.21.15.162:0/565243763 >> v2:172.21.15.162:6800/2866534053 conn(0x7f16a0062c90 0x7f16a0065150 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.093+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 >> 172.21.15.162:0/565243763 conn(0x7f16b00f38a0 msgr2=0x7f16b00f5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.093+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 shutdown_connections 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.093+0000 7f16b6deb700 1 -- 172.21.15.162:0/565243763 wait complete. 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:23.093+0000 7f16b6deb700 1 librados: shutdown 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.318 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:42:23.319 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:23.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:21.068154+0000 osd.1 (osd.1) 91 : cluster [DBG] 4.f scrub starts 2024-02-13T00:42:23.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:21.069477+0000 osd.1 (osd.1) 92 : cluster [DBG] 4.f scrub ok 2024-02-13T00:42:23.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:21.629792+0000 osd.7 (osd.7) 90 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:42:23.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:21.630528+0000 osd.7 (osd.7) 91 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:42:23.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:21.647064+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:42:23.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:21.647995+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:42:23.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:22.113975+0000 mon.b (mon.1) 237 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:23.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:22.320602+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]': finished 2024-02-13T00:42:23.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:22.320757+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e187: 8 total, 8 up, 7 in 2024-02-13T00:42:23.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:22.321993+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:23.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:22.326021+0000 mon.c (mon.2) 164 : audit [INF] from='client.? 172.21.15.162:0/2785938865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-84049-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:23.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:22.327380+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-84049-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:23.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:21.068154+0000 osd.1 (osd.1) 91 : cluster [DBG] 4.f scrub starts 2024-02-13T00:42:23.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:21.069477+0000 osd.1 (osd.1) 92 : cluster [DBG] 4.f scrub ok 2024-02-13T00:42:23.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:21.629792+0000 osd.7 (osd.7) 90 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:42:23.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:21.630528+0000 osd.7 (osd.7) 91 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:42:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:21.647064+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:42:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:21.647995+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:42:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:22.113975+0000 mon.b (mon.1) 237 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:22.320602+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]': finished 2024-02-13T00:42:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:22.320757+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e187: 8 total, 8 up, 7 in 2024-02-13T00:42:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:22.321993+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:22.326021+0000 mon.c (mon.2) 164 : audit [INF] from='client.? 172.21.15.162:0/2785938865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-84049-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:22.327380+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-84049-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:24.032 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:22.333816+0000 mgr.y (mgr.14544) 217 : cluster [DBG] pgmap v251: 300 pgs: 40 unknown, 260 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:42:24.033 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:22.954258+0000 mon.c (mon.2) 165 : audit [DBG] from='client.? 172.21.15.162:0/565243763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:24.033 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:23.115208+0000 mon.b (mon.1) 238 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:24.033 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:23.327481+0000 mon.a (mon.0) 1341 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-83977-26"}]': finished 2024-02-13T00:42:24.033 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:23.327618+0000 mon.a (mon.0) 1342 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-84049-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:24.033 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:23.327789+0000 mon.a (mon.0) 1343 : cluster [DBG] osdmap e188: 8 total, 8 up, 7 in 2024-02-13T00:42:24.033 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:23.335142+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:24.033 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:23.339396+0000 mon.b (mon.1) 239 : audit [INF] from='client.? 172.21.15.162:0/2758753290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-83985-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:24.033 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:23.348096+0000 mon.a (mon.0) 1345 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-83985-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:24.055 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:24.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:24.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:24.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:24.188 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:24.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:22.333816+0000 mgr.y (mgr.14544) 217 : cluster [DBG] pgmap v251: 300 pgs: 40 unknown, 260 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:42:24.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:22.954258+0000 mon.c (mon.2) 165 : audit [DBG] from='client.? 172.21.15.162:0/565243763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:24.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:23.115208+0000 mon.b (mon.1) 238 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:24.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:23.327481+0000 mon.a (mon.0) 1341 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-83977-26"}]': finished 2024-02-13T00:42:24.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:23.327618+0000 mon.a (mon.0) 1342 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-84049-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:24.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:23.327789+0000 mon.a (mon.0) 1343 : cluster [DBG] osdmap e188: 8 total, 8 up, 7 in 2024-02-13T00:42:24.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:23.335142+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:24.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:23.339396+0000 mon.b (mon.1) 239 : audit [INF] from='client.? 172.21.15.162:0/2758753290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-83985-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:24.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:23.348096+0000 mon.a (mon.0) 1345 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-83985-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:24.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:22.333816+0000 mgr.y (mgr.14544) 217 : cluster [DBG] pgmap v251: 300 pgs: 40 unknown, 260 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:42:24.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:22.954258+0000 mon.c (mon.2) 165 : audit [DBG] from='client.? 172.21.15.162:0/565243763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:24.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:23.115208+0000 mon.b (mon.1) 238 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:24.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:23.327481+0000 mon.a (mon.0) 1341 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-83977-26"}]': finished 2024-02-13T00:42:24.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:23.327618+0000 mon.a (mon.0) 1342 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-84049-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:24.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:23.327789+0000 mon.a (mon.0) 1343 : cluster [DBG] osdmap e188: 8 total, 8 up, 7 in 2024-02-13T00:42:24.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:23.335142+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-83977-26"}]: dispatch 2024-02-13T00:42:24.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:23.339396+0000 mon.b (mon.1) 239 : audit [INF] from='client.? 172.21.15.162:0/2758753290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-83985-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:24.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:23.348096+0000 mon.a (mon.0) 1345 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-83985-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:24.337 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:24.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:24.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp api_aio: Running main() from gmock_main.cc 2024-02-13T00:42:24.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [==========] Running 41 tests from 2 test suites. 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [----------] Global test environment set-up. 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [----------] 25 tests from LibRadosAio 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.TooBig (8100 ms) 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (4177 ms) 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3647 ms) 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3262 ms) 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2803 ms) 2024-02-13T00:42:24.348 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-02-13T00:42:24.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (2958 ms) 2024-02-13T00:42:24.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-02-13T00:42:24.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3131 ms) 2024-02-13T00:42:24.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-02-13T00:42:24.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3152 ms) 2024-02-13T00:42:24.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-02-13T00:42:24.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3355 ms) 2024-02-13T00:42:24.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-02-13T00:42:24.349 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3045 ms) 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3057 ms) 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3220 ms) 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2899 ms) 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3073 ms) 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.Flush (2995 ms) 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2745 ms) 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3007 ms) 2024-02-13T00:42:24.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3023 ms) 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2991 ms) 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3028 ms) 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3043 ms) 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3008 ms) 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2922 ms) 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3011 ms) 2024-02-13T00:42:24.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-02-13T00:42:24.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3014 ms) 2024-02-13T00:42:24.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [----------] 25 tests from LibRadosAio (82668 ms total) 2024-02-13T00:42:24.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: 2024-02-13T00:42:24.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-02-13T00:42:24.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-02-13T00:42:24.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6976 ms) 2024-02-13T00:42:24.353 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:42:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:42:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:42:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:42:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24359.0:53 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected e61)", 2024-02-13T00:42:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:37:45.245417+0000", 2024-02-13T00:42:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 279.10171911700002, 2024-02-13T00:42:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0074320890000000002, 2024-02-13T00:42:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24359", 2024-02-13T00:42:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/396906212", 2024-02-13T00:42:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 53 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.245417+0000", 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.245417+0000", 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.245415+0000", 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3310000000000004e-06 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.245423+0000", 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1078e-05 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.245434+0000", 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8869999999999992e-06 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.245442+0000", 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00098138799999999992 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.246423+0000", 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2900000000000002e-06 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.246432+0000", 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046513309999999999 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.251083+0000", 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7970000000000001e-05 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.251151+0000", 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126136 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 4,7", 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.251277+0000", 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067628999999999996 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.251953+0000", 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000592532 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.252546+0000", 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025015600000000002 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.252796+0000", 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9139e-05 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.252825+0000", 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3811999999999998e-05 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.252849+0000", 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0074320890000000002 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:91 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected e61)", 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:37:45.246505+0000", 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 279.100631126, 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006252568, 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 91 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.246505+0000", 2024-02-13T00:42:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.246505+0000", 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.246504+0000", 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1570000000000003e-06 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.246511+0000", 2024-02-13T00:42:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0015e-05 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.246521+0000", 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3289999999999994e-06 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.246529+0000", 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9412000000000001e-05 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.246579+0000", 2024-02-13T00:42:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8170000000000004e-06 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.246584+0000", 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039436289999999997 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.250528+0000", 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013882900000000001 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.250667+0000", 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015314199999999999 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 4,7", 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.250820+0000", 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00079850600000000004 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.251618+0000", 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00075603599999999997 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.252374+0000", 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031766500000000002 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.252692+0000", 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1510999999999997e-05 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.252724+0000", 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3766000000000002e-05 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:45.252757+0000", 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006252568 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14862.0:3 12.f 12:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:00.557921+0000", 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 83.789214606000002, 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0933016659999999, 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14862", 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/4283548805", 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 3 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557921+0000", 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557921+0000", 2024-02-13T00:42:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557919+0000", 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9099999999999999e-06 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557926+0000", 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4139e-05 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557940+0000", 2024-02-13T00:42:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9110000000000004e-06 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557947+0000", 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4711e-05 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557982+0000", 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8269999999999997e-06 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557987+0000", 2024-02-13T00:42:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.060214974 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.618202+0000", 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016521299999999999 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.618367+0000", 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015337700000000001 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,5", 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.618520+0000", 2024-02-13T00:42:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083500600000000001 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.619355+0000", 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011477988999999999 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.630833+0000", 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020342135000000001 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.651176+0000", 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5749e-05 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.651201+0000", 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1682e-05 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.651223+0000", 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0933016659999999 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24727.0:2 9.13 e107/104 9:ce900539:test-rados-api-smithi162-84097-2::foo:head v 107'1, mlcod=0'0)", 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.584856+0000", 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 82.762280197999999, 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0050506450000000003, 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.584856+0000", 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.584856+0000", 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.584855+0000", 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5489999999999997e-06 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.584861+0000", 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7619999999999998e-06 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.584868+0000", 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0099999999999998e-06 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.584875+0000", 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033051199999999998 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.585205+0000", 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8519e-05 2024-02-13T00:42:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.585224+0000", 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046618149999999997 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.589886+0000", 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0820999999999999e-05 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.589906+0000", 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0050506450000000003 2024-02-13T00:42:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24814.0:4 27.1f 27:ff7b1f36:::obj1:head [stat] snapc 0=[] ondisk+read+known_if_redirected e107)", 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.629056+0000", 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 82.718079781, 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.033158676999999998, 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24814", 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/4028053558", 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 4 2024-02-13T00:42:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629056+0000", 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629056+0000", 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629054+0000", 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.032e-06 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629060+0000", 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4870000000000002e-06 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629069+0000", 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.793e-06 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629076+0000", 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012851900000000001 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629204+0000", 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5579999999999997e-06 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629211+0000", 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.032924434000000002 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.662135+0000", 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9778999999999996e-05 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.662215+0000", 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.033158676999999998 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14862.0:4 12.16 12:681f27cd:::2.obj:head [write 0~105] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.652086+0000", 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 82.695049557999994, 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014126975, 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14862", 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/4283548805", 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 4 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.652086+0000", 2024-02-13T00:42:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.652086+0000", 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.652085+0000", 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1689999999999992e-06 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.652093+0000", 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5660000000000007e-06 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.652102+0000", 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6590000000000001e-06 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.652108+0000", 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012632240000000001 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.653372+0000", 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5828999999999997e-05 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.653428+0000", 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2966000000000005e-05 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,4", 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.653511+0000", 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00084050800000000001 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:42:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.654351+0000", 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025650040000000001 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.656916+0000", 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0092078209999999997 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.666124+0000", 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0042e-05 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.666174+0000", 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9520000000000001e-05 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.666213+0000", 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014126975 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24911.0:18 16.12 16:48b3d3f4:::45.obj2:head [write 0~183] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:02.460070+0000", 2024-02-13T00:42:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 81.887066427999997, 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0079173890000000004, 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24911", 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/1860840531", 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 18 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.460070+0000", 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.460070+0000", 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.460067+0000", 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7382999999999999e-05 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.460084+0000", 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6567000000000002e-05 2024-02-13T00:42:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.460111+0000", 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8626000000000001e-05 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.460130+0000", 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000111493 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.460241+0000", 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6878999999999994e-05 2024-02-13T00:42:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.460318+0000", 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4905e-05 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.460413+0000", 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032821700000000001 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.460741+0000", 2024-02-13T00:42:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022888199999999999 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.460970+0000", 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0069544639999999996 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.467924+0000", 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0407999999999997e-05 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.467965+0000", 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2152000000000001e-05 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.467987+0000", 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0079173890000000004 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24911.0:20 16.1 e108/104 16:83282cb0:::1.obj2:head v 108'14, mlcod=107'12)", 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:02.519083+0000", 2024-02-13T00:42:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 81.828052567, 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019362523999999999, 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.519083+0000", 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.519083+0000", 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.519083+0000", 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.339e-06 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.519088+0000", 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4150000000000004e-06 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.519095+0000", 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8859999999999998e-06 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.519102+0000", 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015855470000000001 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.520687+0000", 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2031000000000001e-05 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.520699+0000", 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017723135000000001 2024-02-13T00:42:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.538422+0000", 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3485e-05 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.538446+0000", 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019362523999999999 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24926.0:4 16.12 16:4a6d8757:::3.obj:head [write 0~115] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:02.829836+0000", 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 81.517299965000007, 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020494032999999998, 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24926", 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2404423687", 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 4 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829836+0000", 2024-02-13T00:42:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829836+0000", 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829835+0000", 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.639e-06 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829839+0000", 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6390000000000003e-06 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829846+0000", 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3669999999999999e-06 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829850+0000", 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2035e-05 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829892+0000", 2024-02-13T00:42:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0727000000000003e-05 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829953+0000", 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0786999999999997e-05 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.830044+0000", 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031020700000000001 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:42:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.830354+0000", 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013243400000000001 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.830486+0000", 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019756763 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850243+0000", 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5442999999999998e-05 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850289+0000", 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1400999999999999e-05 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850330+0000", 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020494032999999998 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24926.0:5 16.d 16:b3c717c4:::4.obj:head [write 0~81] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:02.850670+0000", 2024-02-13T00:42:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 81.496465748000006, 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063625679999999999, 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24926", 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2404423687", 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 5 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850670+0000", 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850670+0000", 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850669+0000", 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6659999999999999e-06 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850674+0000", 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8729999999999996e-06 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850683+0000", 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3959999999999999e-06 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850687+0000", 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7623999999999998e-05 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850725+0000", 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3025999999999997e-05 2024-02-13T00:42:24.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850778+0000", 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1166000000000001e-05 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850869+0000", 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000344268 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.851213+0000", 2024-02-13T00:42:24.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057495530000000001 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.856963+0000", 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0726000000000001e-05 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.856984+0000", 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8555e-05 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.857012+0000", 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0686000000000002e-05 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.857033+0000", 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063625679999999999 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24926.0:6 16.d 16:b46e3eb9:::5.obj:head [write 0~255] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:02.857631+0000", 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 81.489505452000003, 2024-02-13T00:42:24.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084912259999999993, 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24926", 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2404423687", 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 6 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.857631+0000", 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.857631+0000", 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.857630+0000", 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.947e-06 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.857635+0000", 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0499999999999999e-05 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.857646+0000", 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7539999999999997e-06 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.857651+0000", 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1909000000000001e-05 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.857693+0000", 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1328000000000005e-05 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.857774+0000", 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014728899999999999 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.857921+0000", 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033964100000000001 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.858261+0000", 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000319964 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.858581+0000", 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0074900169999999999 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.866071+0000", 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9836999999999999e-05 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.866101+0000", 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1067999999999999e-05 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.866122+0000", 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084912259999999993 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24997.0:2 36.1a 36:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.022858+0000", 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.324278282999998, 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0671349779999999, 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24997", 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/1225415695", 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022858+0000", 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022858+0000", 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-02-13T00:42:24.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022855+0000", 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9489999999999996e-06 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022863+0000", 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4700000000000008e-06 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022873+0000", 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2189999999999998e-06 2024-02-13T00:42:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022880+0000", 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9550999999999999e-05 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022950+0000", 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0539999999999997e-06 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022957+0000", 2024-02-13T00:42:24.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0397542660000001 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.062711+0000", 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109751 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.062821+0000", 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015550399999999999 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.062976+0000", 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001836599 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.064813+0000", 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004613102 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.069426+0000", 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020460633999999998 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.089887+0000", 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5318e-05 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.089952+0000", 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0765e-05 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.089993+0000", 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0671349779999999 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.422 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24772.0:5 25.d 25:b3257de0:test-rados-api-smithi162-84073-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.030406+0000", 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.316730411999998, 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017494136, 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24772", 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3029464538", 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 5 2024-02-13T00:42:24.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030406+0000", 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030406+0000", 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030404+0000", 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0809999999999996e-06 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030411+0000", 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3829999999999998e-06 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030420+0000", 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8079999999999999e-06 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030427+0000", 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025939399999999998 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030687+0000", 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119636 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030806+0000", 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017253700000000001 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-02-13T00:42:24.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030979+0000", 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022362010000000002 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.033215+0000", 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0085149660000000005 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.041730+0000", 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061089999999999998 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.047839+0000", 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3524e-05 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.047873+0000", 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7160000000000001e-05 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.047900+0000", 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017494136 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.428 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2070 20.b e109/105 20:d3a22371:test-rados-api-smithi162-84049-17::big2:head v 109'1, mlcod=0'0)", 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.754351+0000", 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 79.592784874000003, 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088475559999999995, 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754351+0000", 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754351+0000", 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754350+0000", 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.045778653000000002 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800129+0000", 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1767999999999999e-05 2024-02-13T00:42:24.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800150+0000", 2024-02-13T00:42:24.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6830000000000008e-06 2024-02-13T00:42:24.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800158+0000", 2024-02-13T00:42:24.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7021999999999998e-05 2024-02-13T00:42:24.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800205+0000", 2024-02-13T00:42:24.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.98e-05 2024-02-13T00:42:24.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800225+0000", 2024-02-13T00:42:24.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.042577241000000002 2024-02-13T00:42:24.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842802+0000", 2024-02-13T00:42:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4448e-05 2024-02-13T00:42:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842827+0000", 2024-02-13T00:42:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088475559999999995 2024-02-13T00:42:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24773.0:48 11.a e109/104 11:55b4a790:test-rados-api-smithi162-84105-9::foo:head v 109'3, mlcod=107'1)", 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.827958+0000", 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 79.519177780999996, 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014482019, 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.827958+0000", 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.827958+0000", 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.827957+0000", 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4120000000000001e-06 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.827966+0000", 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.207e-05 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.827978+0000", 2024-02-13T00:42:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0089999999999998e-06 2024-02-13T00:42:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.827983+0000", 2024-02-13T00:42:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1709999999999999e-05 2024-02-13T00:42:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.828024+0000", 2024-02-13T00:42:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8414999999999999e-05 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.828043+0000", 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014314863000000001 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842358+0000", 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2528000000000007e-05 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842440+0000", 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014482019 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2073 20.b e109/105 20:d3a22371:test-rados-api-smithi162-84049-17::big2:head v 109'4, mlcod=109'2)", 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.849365+0000", 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 79.497770962999994, 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012837360000000001, 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.849365+0000", 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.849365+0000", 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.849363+0000", 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048158109999999997 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854179+0000", 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4929999999999999e-05 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854194+0000", 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1550000000000004e-06 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854200+0000", 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3585999999999999e-05 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854244+0000", 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3010000000000002e-05 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854267+0000", 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0079004929999999998 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.862167+0000", 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5088e-05 2024-02-13T00:42:24.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.862202+0000", 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012837360000000001 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25162.0:978 95.1c 95:39dc3238:test-rados-api-smithi162-84057-14::foo41:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e157)", 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:52.632782+0000", 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.714354182000001, 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058054279999999996, 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.442 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25162", 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 978 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.632782+0000", 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.632782+0000", 2024-02-13T00:42:24.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.632781+0000", 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8020000000000003e-06 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.632787+0000", 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1299e-05 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.632799+0000", 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7899999999999996e-06 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.632805+0000", 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4254e-05 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.632849+0000", 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017913200000000001 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-02-13T00:42:24.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.633028+0000", 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014072550000000001 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.634435+0000", 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000146897 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.634582+0000", 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010731200000000001 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.635655+0000", 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9870000000000006e-05 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.635735+0000", 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036065199999999999 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.636096+0000", 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9994000000000005e-05 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.636196+0000", 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002391546 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.638587+0000", 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058054279999999996 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25162.0:981 95.1 95:864dd171:test-rados-api-smithi162-84057-14::foo52:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e157)", 2024-02-13T00:42:24.448 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:52.650610+0000", 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.696525841, 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0072914479999999999, 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25162", 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 981 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.650610+0000", 2024-02-13T00:42:24.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.650610+0000", 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.650608+0000", 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0309000000000001e-05 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.650619+0000", 2024-02-13T00:42:24.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6422e-05 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.650635+0000", 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4269999999999996e-06 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.650641+0000", 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4418000000000006e-05 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.650706+0000", 2024-02-13T00:42:24.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027330799999999999 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.650979+0000", 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001746285 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.652725+0000", 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014355300000000001 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-02-13T00:42:24.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.652869+0000", 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002147628 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.655017+0000", 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2891000000000005e-05 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.655079+0000", 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00048108800000000001 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.655561+0000", 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9592999999999997e-05 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.655620+0000", 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022814440000000001 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.657902+0000", 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0072914479999999999 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25162.0:982 95.0 95:0760a695:test-rados-api-smithi162-84057-14::foo29:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e157)", 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:52.658242+0000", 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.688894406999999, 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053524899999999997, 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25162", 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 982 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.658242+0000", 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.658242+0000", 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.658240+0000", 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4579999999999998e-06 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.658246+0000", 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8349999999999999e-06 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.658256+0000", 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.938e-06 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.658261+0000", 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4759999999999999e-05 2024-02-13T00:42:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.658296+0000", 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012667200000000001 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.658423+0000", 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012512580000000001 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.659674+0000", 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014766000000000001 2024-02-13T00:42:24.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.659822+0000", 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00082623600000000005 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.660648+0000", 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6668e-05 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.660704+0000", 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032938299999999999 2024-02-13T00:42:24.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.661034+0000", 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9089000000000003e-05 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.661093+0000", 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002501114 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.663594+0000", 2024-02-13T00:42:24.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053524899999999997 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25162.0:1027 95.6 95:67343f0e:test-rados-api-smithi162-84057-14::foo48:5 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e157)", 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:52.828389+0000", 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.51874664, 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056239439999999996, 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25162", 2024-02-13T00:42:24.461 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1027 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.828389+0000", 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.828389+0000", 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.828387+0000", 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3182e-05 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.828400+0000", 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6296999999999999e-05 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.828427+0000", 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2123000000000001e-05 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.828439+0000", 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117268 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.828556+0000", 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036903400000000002 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.828925+0000", 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001200427 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.830125+0000", 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018713000000000001 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.830313+0000", 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00079410099999999996 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.831107+0000", 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9840999999999997e-05 2024-02-13T00:42:24.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.831186+0000", 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032007200000000002 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.831507+0000", 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7611000000000001e-05 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.831564+0000", 2024-02-13T00:42:24.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024491069999999998 2024-02-13T00:42:24.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:42:24.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:42:24.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:42:24.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:52.834013+0000", 2024-02-13T00:42:24.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056239439999999996 2024-02-13T00:42:24.467 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.467 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.467 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.467 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:42:24.467 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:42:24.467 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:42:24.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:42:24.468 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:24.496 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:24.599 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:24.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:24.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:24.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:24.734 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:24.983 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:24.985 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:23.495349+0000 osd.5 (osd.5) 87 : cluster [DBG] 5.e scrub starts 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:23.496796+0000 osd.5 (osd.5) 88 : cluster [DBG] 5.e scrub ok 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:23.560288+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:23.561157+0000 osd.7 (osd.7) 93 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:24.116311+0000 mon.b (mon.1) 240 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:24.331702+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-83977-26"}]': finished 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:24.331875+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-83985-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:24.332029+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e189: 8 total, 8 up, 7 in 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:24.381363+0000 mon.b (mon.1) 241 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:24.631198+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:24.631789+0000 mon.b (mon.1) 242 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:24.632772+0000 mon.a (mon.0) 1350 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:24.633093+0000 mon.b (mon.1) 243 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-83977-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:24.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:24.633557+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-83977-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:24.987 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:24.884108+0000 mon.a (mon.0) 1352 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:24.987 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:23.495349+0000 osd.5 (osd.5) 87 : cluster [DBG] 5.e scrub starts 2024-02-13T00:42:24.987 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:23.496796+0000 osd.5 (osd.5) 88 : cluster [DBG] 5.e scrub ok 2024-02-13T00:42:24.987 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:23.560288+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:42:24.987 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:23.561157+0000 osd.7 (osd.7) 93 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:42:24.987 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:24.116311+0000 mon.b (mon.1) 240 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:24.987 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:24.331702+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-83977-26"}]': finished 2024-02-13T00:42:24.987 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:24.331875+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-83985-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:24.987 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:24.332029+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e189: 8 total, 8 up, 7 in 2024-02-13T00:42:24.987 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:24.381363+0000 mon.b (mon.1) 241 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:24.987 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:24.631198+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:24.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:24.631789+0000 mon.b (mon.1) 242 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:24.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:24.632772+0000 mon.a (mon.0) 1350 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:24.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:24.633093+0000 mon.b (mon.1) 243 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-83977-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:24.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:24.633557+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-83977-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:24.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:24.884108+0000 mon.a (mon.0) 1352 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:24.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:25.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.001+0000 7f024d651700 1 -- 172.21.15.162:0/1848863445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02480fe590 msgr2=0x7f02480f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:25.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.001+0000 7f024d651700 1 --2- 172.21.15.162:0/1848863445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02480fe590 0x7f02480f7dc0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7f023c0047b0 tx=0x7f023c009cb0).stop 2024-02-13T00:42:25.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.001+0000 7f024d651700 1 -- 172.21.15.162:0/1848863445 shutdown_connections 2024-02-13T00:42:25.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.001+0000 7f024d651700 1 --2- 172.21.15.162:0/1848863445 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02480f84a0 0x7f0248104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:25.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.001+0000 7f024d651700 1 --2- 172.21.15.162:0/1848863445 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02480fdc30 0x7f02480fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:25.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.001+0000 7f024d651700 1 --2- 172.21.15.162:0/1848863445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02480fe590 0x7f02480f7dc0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:25.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.001+0000 7f024d651700 1 -- 172.21.15.162:0/1848863445 >> 172.21.15.162:0/1848863445 conn(0x7f02480f3880 msgr2=0x7f02480f5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:25.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.001+0000 7f024d651700 1 -- 172.21.15.162:0/1848863445 shutdown_connections 2024-02-13T00:42:25.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.001+0000 7f024d651700 1 -- 172.21.15.162:0/1848863445 wait complete. 2024-02-13T00:42:25.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f024d651700 1 librados: starting msgr at 2024-02-13T00:42:25.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f024d651700 1 librados: starting objecter 2024-02-13T00:42:25.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f024d651700 1 Processor -- start 2024-02-13T00:42:25.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f024d651700 1 -- start start 2024-02-13T00:42:25.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f024d651700 1 librados: setting wanted keys 2024-02-13T00:42:25.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f024d651700 1 librados: calling monclient init 2024-02-13T00:42:25.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f024d651700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02480f84a0 0x7f0248067ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:25.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f024d651700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02480fdc30 0x7f0248068520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:25.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f024d651700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02480fe590 0x7f0248063230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f024d651700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02480637b0 con 0x7f02480f84a0 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f024d651700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0248063920 con 0x7f02480fdc30 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f024d651700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0248063ae0 con 0x7f02480fe590 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f0246ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02480f84a0 0x7f0248067ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f0246ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02480f84a0 0x7f0248067ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44768/0 (socket says 172.21.15.162:44768) 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f0246ffd700 1 -- 172.21.15.162:0/3312780117 learned_addr learned my addr 172.21.15.162:0/3312780117 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f02477fe700 1 --2- 172.21.15.162:0/3312780117 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02480fe590 0x7f0248063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.002+0000 7f02467fc700 1 --2- 172.21.15.162:0/3312780117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02480fdc30 0x7f0248068520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f0246ffd700 1 -- 172.21.15.162:0/3312780117 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02480fe590 msgr2=0x7f0248063230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f0246ffd700 1 --2- 172.21.15.162:0/3312780117 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02480fe590 0x7f0248063230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f0246ffd700 1 -- 172.21.15.162:0/3312780117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02480fdc30 msgr2=0x7f0248068520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f0246ffd700 1 --2- 172.21.15.162:0/3312780117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02480fdc30 0x7f0248068520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f0246ffd700 1 -- 172.21.15.162:0/3312780117 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0238005040 con 0x7f02480f84a0 2024-02-13T00:42:25.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f0246ffd700 1 --2- 172.21.15.162:0/3312780117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02480f84a0 0x7f0248067ff0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7f0238009400 tx=0x7f0238009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:25.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f022ffff700 10 client.?.objecter ms_handle_connect 0x7f02480f84a0 2024-02-13T00:42:25.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f022ffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:25.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f022ffff700 1 -- 172.21.15.162:0/3312780117 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238007a40 con 0x7f02480f84a0 2024-02-13T00:42:25.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f024d651700 1 -- 172.21.15.162:0/3312780117 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f023c005040 con 0x7f02480f84a0 2024-02-13T00:42:25.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f024d651700 10 client.26598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:25.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f024d651700 1 -- 172.21.15.162:0/3312780117 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024806be10 con 0x7f02480f84a0 2024-02-13T00:42:25.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f024d651700 1 librados: init done 2024-02-13T00:42:25.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f022ffff700 1 -- 172.21.15.162:0/3312780117 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f023800f040 con 0x7f02480f84a0 2024-02-13T00:42:25.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.003+0000 7f022ffff700 1 -- 172.21.15.162:0/3312780117 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238014aa0 con 0x7f02480f84a0 2024-02-13T00:42:25.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.004+0000 7f024d651700 1 -- 172.21.15.162:0/3312780117 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0234004f50 con 0x7f02480f84a0 2024-02-13T00:42:25.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.005+0000 7f022ffff700 1 -- 172.21.15.162:0/3312780117 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f023800e630 con 0x7f02480f84a0 2024-02-13T00:42:25.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.005+0000 7f022ffff700 1 --2- 172.21.15.162:0/3312780117 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0230062c90 0x7f0230065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:25.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.005+0000 7f022ffff700 1 -- 172.21.15.162:0/3312780117 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(189..189 src has 52..189) v4 ==== 9553+0+0 (secure 0 0 0) 0x7f023807dc30 con 0x7f02480f84a0 2024-02-13T00:42:25.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.005+0000 7f022ffff700 10 client.26598.objecter ms_dispatch 0x7f0248001550 osd_map(189..189 src has 52..189) v4 2024-02-13T00:42:25.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.005+0000 7f022ffff700 3 client.26598.objecter handle_osd_map got epochs [189,189] > 0 2024-02-13T00:42:25.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.005+0000 7f022ffff700 3 client.26598.objecter handle_osd_map decoding full epoch 189 2024-02-13T00:42:25.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.005+0000 7f02467fc700 1 --2- 172.21.15.162:0/3312780117 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0230062c90 0x7f0230065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:25.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.006+0000 7f022ffff700 20 client.26598.objecter dump_active .. 0 homeless 2024-02-13T00:42:25.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.006+0000 7f022ffff700 10 client.26598.objecter ms_handle_connect 0x7f0230062c90 2024-02-13T00:42:25.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.006+0000 7f02467fc700 1 --2- 172.21.15.162:0/3312780117 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0230062c90 0x7f0230065150 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 rx=0x7f023c004e20 tx=0x7f023c007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:25.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.010+0000 7f022ffff700 1 -- 172.21.15.162:0/3312780117 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0238050ba0 con 0x7f02480f84a0 2024-02-13T00:42:25.025 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:25.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:25.119 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:25.157 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:25.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:25.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:25.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-13T00:42:25.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.198+0000 7f024d651700 1 -- 172.21.15.162:0/3312780117 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "95.16", "id": [4, 3, 7]} v 0) v1 -- 0x7f0234005780 con 0x7f02480f84a0 2024-02-13T00:42:25.242 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:23.495349+0000 osd.5 (osd.5) 87 : cluster [DBG] 5.e scrub starts 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:23.496796+0000 osd.5 (osd.5) 88 : cluster [DBG] 5.e scrub ok 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:23.560288+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:23.561157+0000 osd.7 (osd.7) 93 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:24.116311+0000 mon.b (mon.1) 240 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:24.331702+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? 172.21.15.162:0/1733891340' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-83977-26"}]': finished 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:24.331875+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-83985-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:24.332029+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e189: 8 total, 8 up, 7 in 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:24.381363+0000 mon.b (mon.1) 241 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:24.631198+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:24.631789+0000 mon.b (mon.1) 242 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:24.632772+0000 mon.a (mon.0) 1350 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:24.633093+0000 mon.b (mon.1) 243 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-83977-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:24.633557+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-83977-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:25.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:24.884108+0000 mon.a (mon.0) 1352 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:25.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:25.296 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:25.313 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:25.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:25.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:25.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.339+0000 7f022ffff700 1 -- 172.21.15.162:0/3312780117 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "95.16", "id": [4, 3, 7]}]=0 set 95.16 pg_upmap mapping to [4,3,7] v190) v1 ==== 131+0+0 (secure 0 0 0) 0x7f0238050790 con 0x7f02480f84a0 2024-02-13T00:42:25.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.351+0000 7f024d651700 10 librados: watch_flush enter 2024-02-13T00:42:25.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.352+0000 7f024d651700 10 librados: watch_flush exit 2024-02-13T00:42:25.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.352+0000 7f024d651700 20 client.26598.objecter shutdown clearing up homeless session... 2024-02-13T00:42:25.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.352+0000 7f024d651700 10 client.26598.objecter successfully canceled tick 2024-02-13T00:42:25.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.356+0000 7f024d651700 1 -- 172.21.15.162:0/3312780117 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0230062c90 msgr2=0x7f0230065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:25.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.356+0000 7f024d651700 1 --2- 172.21.15.162:0/3312780117 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0230062c90 0x7f0230065150 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 rx=0x7f023c004e20 tx=0x7f023c007f90).stop 2024-02-13T00:42:25.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.356+0000 7f024d651700 1 -- 172.21.15.162:0/3312780117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02480f84a0 msgr2=0x7f0248067ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:25.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.356+0000 7f024d651700 1 --2- 172.21.15.162:0/3312780117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02480f84a0 0x7f0248067ff0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7f0238009400 tx=0x7f0238009cb0).stop 2024-02-13T00:42:25.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.356+0000 7f024d651700 1 -- 172.21.15.162:0/3312780117 shutdown_connections 2024-02-13T00:42:25.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.356+0000 7f024d651700 1 --2- 172.21.15.162:0/3312780117 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0230062c90 0x7f0230065150 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:25.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.356+0000 7f024d651700 1 --2- 172.21.15.162:0/3312780117 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02480fe590 0x7f0248063230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:25.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.356+0000 7f024d651700 1 --2- 172.21.15.162:0/3312780117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02480f84a0 0x7f0248067ff0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:25.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.356+0000 7f024d651700 1 --2- 172.21.15.162:0/3312780117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02480fdc30 0x7f0248068520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:25.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.356+0000 7f024d651700 1 -- 172.21.15.162:0/3312780117 >> 172.21.15.162:0/3312780117 conn(0x7f02480f3880 msgr2=0x7f02480f53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:25.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.358+0000 7f024d651700 1 -- 172.21.15.162:0/3312780117 shutdown_connections 2024-02-13T00:42:25.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.358+0000 7f024d651700 1 -- 172.21.15.162:0/3312780117 wait complete. 2024-02-13T00:42:25.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:25.360+0000 7f024d651700 1 librados: shutdown 2024-02-13T00:42:25.367 INFO:teuthology.orchestra.run.smithi162.stderr:set 95.16 pg_upmap mapping to [4,3,7] 2024-02-13T00:42:25.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:25.449 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:25.469 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:25.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:25.504 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:42:25.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:42:25.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:25.601 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:25.852 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:25.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:25.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:25.956 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:25.982 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:25.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:42:26.001 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:26.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:26.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:26.028 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:24.056412+0000 osd.1 (osd.1) 93 : cluster [DBG] 2.a scrub starts 2024-02-13T00:42:26.028 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:24.059309+0000 osd.1 (osd.1) 94 : cluster [DBG] 2.a scrub ok 2024-02-13T00:42:26.028 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:24.334512+0000 mgr.y (mgr.14544) 218 : cluster [DBG] pgmap v254: 300 pgs: 40 unknown, 260 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:26.028 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:25.117639+0000 mon.b (mon.1) 244 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:26.028 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:25.199156+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? 172.21.15.162:0/3312780117' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "95.16", "id": [4, 3, 7]}]: dispatch 2024-02-13T00:42:26.028 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:25.331440+0000 mon.a (mon.0) 1354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-83977-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:26.028 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:25.331533+0000 mon.a (mon.0) 1355 : audit [INF] from='client.? 172.21.15.162:0/3312780117' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "95.16", "id": [4, 3, 7]}]': finished 2024-02-13T00:42:26.028 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:25.331588+0000 mon.a (mon.0) 1356 : cluster [DBG] osdmap e190: 8 total, 8 up, 7 in 2024-02-13T00:42:26.028 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:25.336862+0000 mon.b (mon.1) 245 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-83977-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:26.029 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:25.340970+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-83977-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:26.029 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:25.342395+0000 osd.4 (osd.4) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:26.029 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:25.345344+0000 osd.4 (osd.4) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:26.029 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:25.346513+0000 mon.a (mon.0) 1358 : audit [INF] from='client.? 172.21.15.162:0/3770585613' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-84049-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:26.029 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:25.454394+0000 osd.3 (osd.3) 117 : cluster [DBG] 6.6 scrub starts 2024-02-13T00:42:26.029 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:25.457286+0000 osd.3 (osd.3) 118 : cluster [DBG] 6.6 scrub ok 2024-02-13T00:42:26.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:26.132 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:26.155 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:24.056412+0000 osd.1 (osd.1) 93 : cluster [DBG] 2.a scrub starts 2024-02-13T00:42:26.155 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:24.059309+0000 osd.1 (osd.1) 94 : cluster [DBG] 2.a scrub ok 2024-02-13T00:42:26.155 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:24.334512+0000 mgr.y (mgr.14544) 218 : cluster [DBG] pgmap v254: 300 pgs: 40 unknown, 260 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:26.155 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:25.117639+0000 mon.b (mon.1) 244 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:26.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:25.199156+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? 172.21.15.162:0/3312780117' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "95.16", "id": [4, 3, 7]}]: dispatch 2024-02-13T00:42:26.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:25.331440+0000 mon.a (mon.0) 1354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-83977-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:26.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:25.331533+0000 mon.a (mon.0) 1355 : audit [INF] from='client.? 172.21.15.162:0/3312780117' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "95.16", "id": [4, 3, 7]}]': finished 2024-02-13T00:42:26.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:25.331588+0000 mon.a (mon.0) 1356 : cluster [DBG] osdmap e190: 8 total, 8 up, 7 in 2024-02-13T00:42:26.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:25.336862+0000 mon.b (mon.1) 245 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-83977-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:26.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:25.340970+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-83977-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:26.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:25.342395+0000 osd.4 (osd.4) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:26.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:25.345344+0000 osd.4 (osd.4) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:26.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:25.346513+0000 mon.a (mon.0) 1358 : audit [INF] from='client.? 172.21.15.162:0/3770585613' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-84049-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:26.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:25.454394+0000 osd.3 (osd.3) 117 : cluster [DBG] 6.6 scrub starts 2024-02-13T00:42:26.156 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:25.457286+0000 osd.3 (osd.3) 118 : cluster [DBG] 6.6 scrub ok 2024-02-13T00:42:26.156 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:24.056412+0000 osd.1 (osd.1) 93 : cluster [DBG] 2.a scrub starts 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:24.059309+0000 osd.1 (osd.1) 94 : cluster [DBG] 2.a scrub ok 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:24.334512+0000 mgr.y (mgr.14544) 218 : cluster [DBG] pgmap v254: 300 pgs: 40 unknown, 260 active+clean; 15 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:25.117639+0000 mon.b (mon.1) 244 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:25.199156+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? 172.21.15.162:0/3312780117' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "95.16", "id": [4, 3, 7]}]: dispatch 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:25.331440+0000 mon.a (mon.0) 1354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-83977-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:25.331533+0000 mon.a (mon.0) 1355 : audit [INF] from='client.? 172.21.15.162:0/3312780117' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "95.16", "id": [4, 3, 7]}]': finished 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:25.331588+0000 mon.a (mon.0) 1356 : cluster [DBG] osdmap e190: 8 total, 8 up, 7 in 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:25.336862+0000 mon.b (mon.1) 245 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-83977-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:25.340970+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-83977-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:25.342395+0000 osd.4 (osd.4) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:25.345344+0000 osd.4 (osd.4) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:25.346513+0000 mon.a (mon.0) 1358 : audit [INF] from='client.? 172.21.15.162:0/3770585613' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-84049-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:25.454394+0000 osd.3 (osd.3) 117 : cluster [DBG] 6.6 scrub starts 2024-02-13T00:42:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:25.457286+0000 osd.3 (osd.3) 118 : cluster [DBG] 6.6 scrub ok 2024-02-13T00:42:26.461 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:42:26] "GET /metrics HTTP/1.1" 200 191998 "" "Prometheus/2.33.4" 2024-02-13T00:42:26.880 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:26.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:26.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:27.010 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:27.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:25.406273+0000 mgr.y (mgr.14544) 219 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:27.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:26.118576+0000 mon.b (mon.1) 246 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:27.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:26.310684+0000 osd.2 (osd.2) 83 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:42:27.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:26.313576+0000 osd.2 (osd.2) 84 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:42:27.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:26.574181+0000 mon.b (mon.1) 247 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:27.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:26.595377+0000 mon.a (mon.0) 1359 : audit [INF] from='client.? 172.21.15.162:0/3770585613' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-84049-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:27.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:26.595534+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e191: 8 total, 8 up, 7 in 2024-02-13T00:42:27.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:26.596547+0000 mon.c (mon.2) 166 : audit [INF] from='client.? 172.21.15.162:0/2150629092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-83985-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:27.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:26.599908+0000 mon.a (mon.0) 1361 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-83985-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:27.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:26.600745+0000 mon.a (mon.0) 1362 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:27.158 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:27.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:27.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:27.181 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:25.406273+0000 mgr.y (mgr.14544) 219 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:27.181 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:26.118576+0000 mon.b (mon.1) 246 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:27.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:26.310684+0000 osd.2 (osd.2) 83 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:42:27.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:26.313576+0000 osd.2 (osd.2) 84 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:42:27.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:26.574181+0000 mon.b (mon.1) 247 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:27.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:26.595377+0000 mon.a (mon.0) 1359 : audit [INF] from='client.? 172.21.15.162:0/3770585613' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-84049-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:27.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:26.595534+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e191: 8 total, 8 up, 7 in 2024-02-13T00:42:27.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:26.596547+0000 mon.c (mon.2) 166 : audit [INF] from='client.? 172.21.15.162:0/2150629092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-83985-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:27.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:26.599908+0000 mon.a (mon.0) 1361 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-83985-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:27.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:26.600745+0000 mon.a (mon.0) 1362 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:27.182 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:25.406273+0000 mgr.y (mgr.14544) 219 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:27.182 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:26.118576+0000 mon.b (mon.1) 246 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:27.182 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:26.310684+0000 osd.2 (osd.2) 83 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:42:27.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:26.313576+0000 osd.2 (osd.2) 84 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:42:27.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:26.574181+0000 mon.b (mon.1) 247 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:27.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:26.595377+0000 mon.a (mon.0) 1359 : audit [INF] from='client.? 172.21.15.162:0/3770585613' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-84049-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:27.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:26.595534+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e191: 8 total, 8 up, 7 in 2024-02-13T00:42:27.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:26.596547+0000 mon.c (mon.2) 166 : audit [INF] from='client.? 172.21.15.162:0/2150629092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-83985-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:27.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:26.599908+0000 mon.a (mon.0) 1361 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-83985-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:27.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:26.600745+0000 mon.a (mon.0) 1362 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:27.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:27.292 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:27.428 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:42:27.428 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:42:27.428 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:42:27.429 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:42:27.442 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:27.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:27.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:27.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.442+0000 7fc1aa699700 1 -- 172.21.15.162:0/2654352231 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1a40fcb20 msgr2=0x7fc1a40fcf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.442+0000 7fc1aa699700 1 --2- 172.21.15.162:0/2654352231 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1a40fcb20 0x7fc1a40fcf00 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7fc194004870 tx=0x7fc194009cb0).stop 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 -- 172.21.15.162:0/2654352231 shutdown_connections 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 --2- 172.21.15.162:0/2654352231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1a40f7390 0x7fc1a4103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 --2- 172.21.15.162:0/2654352231 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1a40fcb20 0x7fc1a40fcf00 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 --2- 172.21.15.162:0/2654352231 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1a40fd480 0x7fc1a40f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 -- 172.21.15.162:0/2654352231 >> 172.21.15.162:0/2654352231 conn(0x7fc1a40f2710 msgr2=0x7fc1a40f4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 -- 172.21.15.162:0/2654352231 shutdown_connections 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 -- 172.21.15.162:0/2654352231 wait complete. 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 librados: starting msgr at 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 librados: starting objecter 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 Processor -- start 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 -- start start 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 librados: setting wanted keys 2024-02-13T00:42:27.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.443+0000 7fc1aa699700 1 librados: calling monclient init 2024-02-13T00:42:27.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1aa699700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1a40f7390 0x7fc1a40696e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:27.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1aa699700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1a40fcb20 0x7fc1a4069bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:27.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1aa699700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1a40fd480 0x7fc1a411ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:27.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1aa699700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a411ef00 con 0x7fc1a40fd480 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1aa699700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a411f040 con 0x7fc1a40f7390 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1aa699700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a411f180 con 0x7fc1a40fcb20 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a37fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1a40fcb20 0x7fc1a4069bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a8c36700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1a40fd480 0x7fc1a411ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a37fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1a40fcb20 0x7fc1a4069bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:56850/0 (socket says 172.21.15.162:56850) 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a37fe700 1 -- 172.21.15.162:0/3032680114 learned_addr learned my addr 172.21.15.162:0/3032680114 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a8c36700 1 --2- 172.21.15.162:0/3032680114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1a40fd480 0x7fc1a411ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44822/0 (socket says 172.21.15.162:44822) 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a3fff700 1 --2- 172.21.15.162:0/3032680114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1a40f7390 0x7fc1a40696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a37fe700 1 -- 172.21.15.162:0/3032680114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1a40f7390 msgr2=0x7fc1a40696e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a37fe700 1 --2- 172.21.15.162:0/3032680114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1a40f7390 0x7fc1a40696e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a37fe700 1 -- 172.21.15.162:0/3032680114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1a40fd480 msgr2=0x7fc1a411ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a37fe700 1 --2- 172.21.15.162:0/3032680114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1a40fd480 0x7fc1a411ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:27.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a37fe700 1 -- 172.21.15.162:0/3032680114 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194005040 con 0x7fc1a40fcb20 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a37fe700 1 --2- 172.21.15.162:0/3032680114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1a40fcb20 0x7fc1a4069bd0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fc19800e970 tx=0x7fc19800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a17fa700 10 client.?.objecter ms_handle_connect 0x7fc1a40fcb20 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a17fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.444+0000 7fc1a17fa700 1 -- 172.21.15.162:0/3032680114 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198004660 con 0x7fc1a40fcb20 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.445+0000 7fc1aa699700 1 -- 172.21.15.162:0/3032680114 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a411f460 con 0x7fc1a40fcb20 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.445+0000 7fc1a17fa700 1 -- 172.21.15.162:0/3032680114 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc19800f040 con 0x7fc1a40fcb20 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.445+0000 7fc1a17fa700 1 -- 172.21.15.162:0/3032680114 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198014790 con 0x7fc1a40fcb20 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.445+0000 7fc1aa699700 10 client.36713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.445+0000 7fc1aa699700 1 -- 172.21.15.162:0/3032680114 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a411f9a0 con 0x7fc1a40fcb20 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.445+0000 7fc1aa699700 1 librados: init done 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.446+0000 7fc1aa699700 1 -- 172.21.15.162:0/3032680114 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a40607d0 con 0x7fc1a40fcb20 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.446+0000 7fc1a17fa700 1 -- 172.21.15.162:0/3032680114 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc1980090d0 con 0x7fc1a40fcb20 2024-02-13T00:42:27.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.446+0000 7fc1a17fa700 1 --2- 172.21.15.162:0/3032680114 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc18c062c90 0x7fc18c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:27.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.446+0000 7fc1a17fa700 1 -- 172.21.15.162:0/3032680114 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(192..192 src has 52..192) v4 ==== 10380+0+0 (secure 0 0 0) 0x7fc19807d660 con 0x7fc1a40fcb20 2024-02-13T00:42:27.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.446+0000 7fc1a17fa700 10 client.36713.objecter ms_dispatch 0x7fc1a4001550 osd_map(192..192 src has 52..192) v4 2024-02-13T00:42:27.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.446+0000 7fc1a17fa700 3 client.36713.objecter handle_osd_map got epochs [192,192] > 0 2024-02-13T00:42:27.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.446+0000 7fc1a17fa700 3 client.36713.objecter handle_osd_map decoding full epoch 192 2024-02-13T00:42:27.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.446+0000 7fc1a3fff700 1 --2- 172.21.15.162:0/3032680114 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc18c062c90 0x7fc18c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:27.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.447+0000 7fc1a17fa700 20 client.36713.objecter dump_active .. 0 homeless 2024-02-13T00:42:27.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.447+0000 7fc1a3fff700 1 --2- 172.21.15.162:0/3032680114 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc18c062c90 0x7fc18c065150 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 rx=0x7fc1940043d0 tx=0x7fc19400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:27.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.447+0000 7fc1a17fa700 10 client.36713.objecter ms_handle_connect 0x7fc18c062c90 2024-02-13T00:42:27.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.451+0000 7fc1a17fa700 1 -- 172.21.15.162:0/3032680114 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc19804af60 con 0x7fc1a40fcb20 2024-02-13T00:42:27.456 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:27.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:42:27.575 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:27.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:27.646+0000 7fc1aa699700 1 -- 172.21.15.162:0/3032680114 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fc1a406aa00 con 0x7fc1a40fcb20 2024-02-13T00:42:27.697 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:42:27.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:27.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:42:27.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:42:27.829 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:28.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:26.335082+0000 mgr.y (mgr.14544) 220 : cluster [DBG] pgmap v257: 332 pgs: 12 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 64 unknown, 204 active+clean; 31 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-13T00:42:28.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:26.881134+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:26.881785+0000 osd.7 (osd.7) 95 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:27.119976+0000 mon.b (mon.1) 248 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:27.331414+0000 mon.a (mon.0) 1363 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:27.335910+0000 osd.2 (osd.2) 85 : cluster [DBG] 2.e scrub starts 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:27.338728+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-83977-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-83977-27"}]': finished 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:27.338929+0000 mon.a (mon.0) 1365 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-83985-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:27.338991+0000 mon.a (mon.0) 1366 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:27.339062+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e192: 8 total, 8 up, 7 in 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:27.340073+0000 osd.2 (osd.2) 86 : cluster [DBG] 2.e scrub ok 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:27.344185+0000 mon.b (mon.1) 249 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13"}]: dispatch 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:27.358212+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13"}]: dispatch 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:27.647854+0000 mon.c (mon.2) 167 : audit [INF] from='client.? 172.21.15.162:0/3032680114' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:42:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:27.648132+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:42:28.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:26.335082+0000 mgr.y (mgr.14544) 220 : cluster [DBG] pgmap v257: 332 pgs: 12 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 64 unknown, 204 active+clean; 31 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-13T00:42:28.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:26.881134+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:28.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:26.881785+0000 osd.7 (osd.7) 95 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:28.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:27.119976+0000 mon.b (mon.1) 248 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:27.331414+0000 mon.a (mon.0) 1363 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:42:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:27.335910+0000 osd.2 (osd.2) 85 : cluster [DBG] 2.e scrub starts 2024-02-13T00:42:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:27.338728+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-83977-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-83977-27"}]': finished 2024-02-13T00:42:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:27.338929+0000 mon.a (mon.0) 1365 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-83985-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:27.338991+0000 mon.a (mon.0) 1366 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:42:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:27.339062+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e192: 8 total, 8 up, 7 in 2024-02-13T00:42:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:27.340073+0000 osd.2 (osd.2) 86 : cluster [DBG] 2.e scrub ok 2024-02-13T00:42:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:27.344185+0000 mon.b (mon.1) 249 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13"}]: dispatch 2024-02-13T00:42:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:27.358212+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13"}]: dispatch 2024-02-13T00:42:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:27.647854+0000 mon.c (mon.2) 167 : audit [INF] from='client.? 172.21.15.162:0/3032680114' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:42:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:27.648132+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:26.335082+0000 mgr.y (mgr.14544) 220 : cluster [DBG] pgmap v257: 332 pgs: 12 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 64 unknown, 204 active+clean; 31 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:26.881134+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:26.881785+0000 osd.7 (osd.7) 95 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:27.119976+0000 mon.b (mon.1) 248 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:27.331414+0000 mon.a (mon.0) 1363 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:27.335910+0000 osd.2 (osd.2) 85 : cluster [DBG] 2.e scrub starts 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:27.338728+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-83977-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-83977-27"}]': finished 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:27.338929+0000 mon.a (mon.0) 1365 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-83985-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:27.338991+0000 mon.a (mon.0) 1366 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:27.339062+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e192: 8 total, 8 up, 7 in 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:27.340073+0000 osd.2 (osd.2) 86 : cluster [DBG] 2.e scrub ok 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:27.344185+0000 mon.b (mon.1) 249 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13"}]: dispatch 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:27.358212+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13"}]: dispatch 2024-02-13T00:42:28.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:27.647854+0000 mon.c (mon.2) 167 : audit [INF] from='client.? 172.21.15.162:0/3032680114' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:42:28.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:27.648132+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:42:28.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-13T00:42:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:42:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:28.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:42:28.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.347+0000 7fc1a17fa700 1 -- 172.21.15.162:0/3032680114 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v193) v1 ==== 87+0+0 (secure 0 0 0) 0x7fc198019070 con 0x7fc1a40fcb20 2024-02-13T00:42:28.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.351+0000 7fc1aa699700 10 librados: watch_flush enter 2024-02-13T00:42:28.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.351+0000 7fc1aa699700 10 librados: watch_flush exit 2024-02-13T00:42:28.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.351+0000 7fc1aa699700 20 client.36713.objecter shutdown clearing up homeless session... 2024-02-13T00:42:28.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.351+0000 7fc1aa699700 10 client.36713.objecter successfully canceled tick 2024-02-13T00:42:28.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.353+0000 7fc1aa699700 1 -- 172.21.15.162:0/3032680114 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc18c062c90 msgr2=0x7fc18c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:28.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.353+0000 7fc1aa699700 1 --2- 172.21.15.162:0/3032680114 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc18c062c90 0x7fc18c065150 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 rx=0x7fc1940043d0 tx=0x7fc19400b040).stop 2024-02-13T00:42:28.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.353+0000 7fc1aa699700 1 -- 172.21.15.162:0/3032680114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1a40fcb20 msgr2=0x7fc1a4069bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:28.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.353+0000 7fc1aa699700 1 --2- 172.21.15.162:0/3032680114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1a40fcb20 0x7fc1a4069bd0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fc19800e970 tx=0x7fc19800b040).stop 2024-02-13T00:42:28.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:28.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.357+0000 7fc1aa699700 1 -- 172.21.15.162:0/3032680114 shutdown_connections 2024-02-13T00:42:28.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.357+0000 7fc1aa699700 1 --2- 172.21.15.162:0/3032680114 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc18c062c90 0x7fc18c065150 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:28.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.357+0000 7fc1aa699700 1 --2- 172.21.15.162:0/3032680114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1a40fd480 0x7fc1a411ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:28.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.357+0000 7fc1aa699700 1 --2- 172.21.15.162:0/3032680114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1a40f7390 0x7fc1a40696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:28.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.357+0000 7fc1aa699700 1 --2- 172.21.15.162:0/3032680114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1a40fcb20 0x7fc1a4069bd0 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:28.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.357+0000 7fc1aa699700 1 -- 172.21.15.162:0/3032680114 >> 172.21.15.162:0/3032680114 conn(0x7fc1a40f2710 msgr2=0x7fc1a4061080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:28.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.364+0000 7fc1aa699700 1 -- 172.21.15.162:0/3032680114 shutdown_connections 2024-02-13T00:42:28.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.365+0000 7fc1aa699700 1 -- 172.21.15.162:0/3032680114 wait complete. 2024-02-13T00:42:28.370 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:28.365+0000 7fc1aa699700 1 librados: shutdown 2024-02-13T00:42:28.370 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:42:28.408 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:28.410 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:42:28.944 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:42:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:42:29.289 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:27.409957+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:42:29.289 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:27.410773+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:42:29.289 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:27.541443+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:42:29.289 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:27.544316+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:42:29.289 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:27.593287+0000 osd.0 (osd.0) 79 : cluster [DBG] 5.1e scrub starts 2024-02-13T00:42:29.290 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:27.596327+0000 osd.0 (osd.0) 80 : cluster [DBG] 5.1e scrub ok 2024-02-13T00:42:29.290 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:27.921465+0000 osd.5 (osd.5) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:29.290 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:27.921584+0000 osd.5 (osd.5) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:29.290 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:28.120952+0000 mon.b (mon.1) 250 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:29.290 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:28.336269+0000 mon.a (mon.0) 1370 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:29.290 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:28.336292+0000 mon.a (mon.0) 1371 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:42:29.290 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:28.343872+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13"}]': finished 2024-02-13T00:42:29.290 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:28.344030+0000 mon.a (mon.0) 1373 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:42:29.290 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:28.344084+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e193: 8 total, 8 up, 7 in 2024-02-13T00:42:29.290 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:28.355398+0000 mon.a (mon.0) 1375 : audit [INF] from='client.? 172.21.15.162:0/580184505' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi162-84049-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:27.409957+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:42:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:27.410773+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:42:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:27.541443+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:42:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:27.544316+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:42:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:27.593287+0000 osd.0 (osd.0) 79 : cluster [DBG] 5.1e scrub starts 2024-02-13T00:42:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:27.596327+0000 osd.0 (osd.0) 80 : cluster [DBG] 5.1e scrub ok 2024-02-13T00:42:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:27.921465+0000 osd.5 (osd.5) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:27.921584+0000 osd.5 (osd.5) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:29.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:28.120952+0000 mon.b (mon.1) 250 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:29.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:28.336269+0000 mon.a (mon.0) 1370 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:29.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:28.336292+0000 mon.a (mon.0) 1371 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:42:29.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:28.343872+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13"}]': finished 2024-02-13T00:42:29.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:28.344030+0000 mon.a (mon.0) 1373 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:42:29.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:28.344084+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e193: 8 total, 8 up, 7 in 2024-02-13T00:42:29.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:28.355398+0000 mon.a (mon.0) 1375 : audit [INF] from='client.? 172.21.15.162:0/580184505' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi162-84049-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:29.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:27.409957+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:42:29.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:27.410773+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:42:29.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:27.541443+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:42:29.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:27.544316+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:42:29.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:27.593287+0000 osd.0 (osd.0) 79 : cluster [DBG] 5.1e scrub starts 2024-02-13T00:42:29.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:27.596327+0000 osd.0 (osd.0) 80 : cluster [DBG] 5.1e scrub ok 2024-02-13T00:42:29.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:27.921465+0000 osd.5 (osd.5) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:29.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:27.921584+0000 osd.5 (osd.5) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:29.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:28.120952+0000 mon.b (mon.1) 250 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:29.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:28.336269+0000 mon.a (mon.0) 1370 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:29.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:28.336292+0000 mon.a (mon.0) 1371 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:42:29.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:28.343872+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-13"}]': finished 2024-02-13T00:42:29.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:28.344030+0000 mon.a (mon.0) 1373 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:42:29.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:28.344084+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e193: 8 total, 8 up, 7 in 2024-02-13T00:42:29.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:28.355398+0000 mon.a (mon.0) 1375 : audit [INF] from='client.? 172.21.15.162:0/580184505' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi162-84049-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:29.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.406+0000 7f5914109700 1 -- 172.21.15.162:0/878247037 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590c0f84b0 msgr2=0x7f590c104ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.406+0000 7f5914109700 1 --2- 172.21.15.162:0/878247037 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590c0f84b0 0x7f590c104ab0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7f5904009c30 tx=0x7f5904004c50).stop 2024-02-13T00:42:29.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.406+0000 7f5914109700 1 -- 172.21.15.162:0/878247037 shutdown_connections 2024-02-13T00:42:29.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.406+0000 7f5914109700 1 --2- 172.21.15.162:0/878247037 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590c0f84b0 0x7f590c104ab0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.406+0000 7f5914109700 1 --2- 172.21.15.162:0/878247037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f590c0fdc40 0x7f590c0fe020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.406+0000 7f5914109700 1 --2- 172.21.15.162:0/878247037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f590c0fe5a0 0x7f590c0f7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.406+0000 7f5914109700 1 -- 172.21.15.162:0/878247037 >> 172.21.15.162:0/878247037 conn(0x7f590c0f3850 msgr2=0x7f590c0f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.406+0000 7f5914109700 1 -- 172.21.15.162:0/878247037 shutdown_connections 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.406+0000 7f5914109700 1 -- 172.21.15.162:0/878247037 wait complete. 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.407+0000 7f5914109700 1 librados: starting msgr at 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.407+0000 7f5914109700 1 librados: starting objecter 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.407+0000 7f5914109700 1 Processor -- start 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.407+0000 7f5914109700 1 -- start start 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.407+0000 7f5914109700 1 librados: setting wanted keys 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.407+0000 7f5914109700 1 librados: calling monclient init 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5914109700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f590c0f84b0 0x7f590c10e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5914109700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f590c0fdc40 0x7f590c10e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5914109700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590c0fe5a0 0x7f590c108240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5914109700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c108730 con 0x7f590c0f84b0 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5914109700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c1088a0 con 0x7f590c0fdc40 2024-02-13T00:42:29.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5914109700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590c108a60 con 0x7f590c0fe5a0 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5911ea5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f590c0f84b0 0x7f590c10e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5911ea5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f590c0f84b0 0x7f590c10e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44854/0 (socket says 172.21.15.162:44854) 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5911ea5700 1 -- 172.21.15.162:0/1821082507 learned_addr learned my addr 172.21.15.162:0/1821082507 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f59126a6700 1 --2- 172.21.15.162:0/1821082507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590c0fe5a0 0x7f590c108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f59116a4700 1 --2- 172.21.15.162:0/1821082507 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f590c0fdc40 0x7f590c10e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f59126a6700 1 -- 172.21.15.162:0/1821082507 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f590c0fdc40 msgr2=0x7f590c10e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f59126a6700 1 --2- 172.21.15.162:0/1821082507 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f590c0fdc40 0x7f590c10e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f59126a6700 1 -- 172.21.15.162:0/1821082507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f590c0f84b0 msgr2=0x7f590c10e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f59126a6700 1 --2- 172.21.15.162:0/1821082507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f590c0f84b0 0x7f590c10e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f59126a6700 1 -- 172.21.15.162:0/1821082507 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59040097c0 con 0x7f590c0fe5a0 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5911ea5700 1 --2- 172.21.15.162:0/1821082507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f590c0f84b0 0x7f590c10e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f59126a6700 1 --2- 172.21.15.162:0/1821082507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590c0fe5a0 0x7f590c108240 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7f5904009ad0 tx=0x7f5904004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5902ffd700 10 client.?.objecter ms_handle_connect 0x7f590c0fe5a0 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5902ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:29.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590400e070 con 0x7f590c0fe5a0 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5904018040 con 0x7f590c0fe5a0 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.408+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904017070 con 0x7f590c0fe5a0 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.409+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590c108ce0 con 0x7f590c0fe5a0 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.409+0000 7f5914109700 10 client.36722.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.409+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590c06afb0 con 0x7f590c0fe5a0 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.409+0000 7f5914109700 1 librados: init done 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.410+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f590c1090f0 con 0x7f590c0fe5a0 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.410+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f5904007d30 con 0x7f590c0fe5a0 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5902ffd700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6800/2866534053 conn(0x7f58f8062c90 0x7f58f8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(194..194 src has 52..194) v4 ==== 9947+0+0 (secure 0 0 0) 0x7f590407e990 con 0x7f590c0fe5a0 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 osd_map(194..194 src has 52..194) v4 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5902ffd700 3 client.36722.objecter handle_osd_map got epochs [194,194] > 0 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5902ffd700 3 client.36722.objecter handle_osd_map decoding full epoch 194 2024-02-13T00:42:29.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5902ffd700 20 client.36722.objecter dump_active .. 0 homeless 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v194) v1 ==== 54+0+16 (secure 0 0 0) 0x7f5904083200 con 0x7f590c0fe5a0 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5911ea5700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6800/2866534053 conn(0x7f58f8062c90 0x7f58f8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5914109700 10 client.36722.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 1 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6801/2103743338 conn(0x7f590c0fef10 0x7f590c0ff2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c05fb00 osd=0 3 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c05fb00 osd=0 3 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c05fb00 osd=0 3 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 1 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c05fb00 osd=0 3 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 0 1 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c05fb00 osd=0 4 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5914109700 10 client.36722.objecter _send_command 1 2024-02-13T00:42:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590c1071a0 con 0x7f590c0fef10 2024-02-13T00:42:29.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f59116a4700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6801/2103743338 conn(0x7f590c0fef10 0x7f590c0ff2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:29.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5902ffd700 10 client.36722.objecter ms_handle_connect 0x7f58f8062c90 2024-02-13T00:42:29.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.411+0000 7f5911ea5700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6800/2866534053 conn(0x7f58f8062c90 0x7f58f8065150 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f5908007d30 tx=0x7f5908014040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:29.745 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:42:29.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.412+0000 7f5902ffd700 10 client.36722.objecter ms_handle_connect 0x7f590c0fef10 2024-02-13T00:42:29.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.412+0000 7f59116a4700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6801/2103743338 conn(0x7f590c0fef10 0x7f590c0ff2f0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.412+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f590c1071a0 con 0x7f590c0fef10 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.412+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.412+0000 7f5902ffd700 10 client.36722.objecter _finish_command 1 = system:0 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.412+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c05fb00 osd=0 4 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.412+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 0 1 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.436+0000 7f5914109700 10 client.36722.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.436+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 2 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.436+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c05fb00 osd=0 3 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.436+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c05fb00 osd=0 3 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.436+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c05fb00 osd=0 3 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.436+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 2 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.436+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c05fb00 osd=0 3 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.436+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 0 2 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.436+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c05fb00 osd=0 4 2024-02-13T00:42:29.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.436+0000 7f5914109700 10 client.36722.objecter _send_command 2 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.436+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f590c027410 con 0x7f590c0fef10 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f590c027410 con 0x7f590c0fef10 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5902ffd700 10 client.36722.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c05fb00 osd=0 4 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 0 2 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5914109700 10 client.36722.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 3 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6805/150599780 conn(0x7f590c059990 0x7f590c059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c028760 osd=1 3 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c028760 osd=1 3 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c028760 osd=1 3 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 3 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c028760 osd=1 3 2024-02-13T00:42:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 1 3 2024-02-13T00:42:29.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c028760 osd=1 4 2024-02-13T00:42:29.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5914109700 10 client.36722.objecter _send_command 3 2024-02-13T00:42:29.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590c06c7b0 con 0x7f590c059990 2024-02-13T00:42:29.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5911ea5700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6805/150599780 conn(0x7f590c059990 0x7f590c059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:29.748 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:42:29.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5902ffd700 10 client.36722.objecter ms_handle_connect 0x7f590c059990 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.438+0000 7f5911ea5700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6805/150599780 conn(0x7f590c059990 0x7f590c059d70 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.439+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f590c06c7b0 con 0x7f590c059990 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.439+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.439+0000 7f5902ffd700 10 client.36722.objecter _finish_command 3 = system:0 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.439+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c028760 osd=1 4 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.439+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 1 3 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.463+0000 7f5914109700 10 client.36722.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.463+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 4 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.463+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c028760 osd=1 3 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.463+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c028760 osd=1 3 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.463+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c028760 osd=1 3 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.463+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 4 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.463+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c028760 osd=1 3 2024-02-13T00:42:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.463+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 1 4 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.463+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c028760 osd=1 4 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.463+0000 7f5914109700 10 client.36722.objecter _send_command 4 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.463+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f590c059400 con 0x7f590c059990 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.464+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f590c059400 con 0x7f590c059990 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.464+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.464+0000 7f5902ffd700 10 client.36722.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.464+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c028760 osd=1 4 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.464+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 1 4 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5914109700 10 client.36722.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 5 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6809/3689719376 conn(0x7f590c06b850 0x7f590c06bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c06b1d0 osd=2 3 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c06b1d0 osd=2 3 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c06b1d0 osd=2 3 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 5 2024-02-13T00:42:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c06b1d0 osd=2 3 2024-02-13T00:42:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 2 5 2024-02-13T00:42:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c06b1d0 osd=2 4 2024-02-13T00:42:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5914109700 10 client.36722.objecter _send_command 5 2024-02-13T00:42:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590c06c140 con 0x7f590c06b850 2024-02-13T00:42:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f59116a4700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6809/3689719376 conn(0x7f590c06b850 0x7f590c06bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:29.751 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:42:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f5902ffd700 10 client.36722.objecter ms_handle_connect 0x7f590c06b850 2024-02-13T00:42:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.465+0000 7f59116a4700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6809/3689719376 conn(0x7f590c06b850 0x7f590c06bc50 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.466+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f590c06c140 con 0x7f590c06b850 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.466+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.466+0000 7f5902ffd700 10 client.36722.objecter _finish_command 5 = system:0 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.466+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c06b1d0 osd=2 4 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.466+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 2 5 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.489+0000 7f5914109700 10 client.36722.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.489+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 6 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.489+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c06b1d0 osd=2 3 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.489+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c06b1d0 osd=2 3 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.489+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c06b1d0 osd=2 3 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.489+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 6 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.489+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c06b1d0 osd=2 3 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.489+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 2 6 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.489+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c06b1d0 osd=2 4 2024-02-13T00:42:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.489+0000 7f5914109700 10 client.36722.objecter _send_command 6 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.489+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f590c06c5a0 con 0x7f590c06b850 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.491+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f590c06c5a0 con 0x7f590c06b850 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5902ffd700 10 client.36722.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c06b1d0 osd=2 4 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 2 6 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5914109700 10 client.36722.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 7 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6813/604971011 conn(0x7f590c06cd80 0x7f590c11e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c06cc10 osd=3 3 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c06cc10 osd=3 3 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c06cc10 osd=3 3 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 7 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c06cc10 osd=3 3 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 3 7 2024-02-13T00:42:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c06cc10 osd=3 4 2024-02-13T00:42:29.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5914109700 10 client.36722.objecter _send_command 7 2024-02-13T00:42:29.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590c0282c0 con 0x7f590c06cd80 2024-02-13T00:42:29.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f59126a6700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6813/604971011 conn(0x7f590c06cd80 0x7f590c11e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:29.754 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:42:29.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f5902ffd700 10 client.36722.objecter ms_handle_connect 0x7f590c06cd80 2024-02-13T00:42:29.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.492+0000 7f59126a6700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6813/604971011 conn(0x7f590c06cd80 0x7f590c11e090 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:29.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.493+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f590c0282c0 con 0x7f590c06cd80 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.493+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.493+0000 7f5902ffd700 10 client.36722.objecter _finish_command 7 = system:0 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.493+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c06cc10 osd=3 4 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.493+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 3 7 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.516+0000 7f5914109700 10 client.36722.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.516+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 8 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.516+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c06cc10 osd=3 3 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.516+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c06cc10 osd=3 3 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.516+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c06cc10 osd=3 3 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.516+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 8 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.516+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c06cc10 osd=3 3 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.516+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 3 8 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.516+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c06cc10 osd=3 4 2024-02-13T00:42:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.516+0000 7f5914109700 10 client.36722.objecter _send_command 8 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.516+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f590c11e580 con 0x7f590c06cd80 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f590c11e580 con 0x7f590c06cd80 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5902ffd700 10 client.36722.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c06cc10 osd=3 4 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 3 8 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5914109700 10 client.36722.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 9 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6800/3344263388 conn(0x7f590c11ec00 0x7f590c120ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c11e580 osd=4 3 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c11e580 osd=4 3 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c11e580 osd=4 3 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 9 2024-02-13T00:42:29.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c11e580 osd=4 3 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 4 9 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c11e580 osd=4 4 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5914109700 10 client.36722.objecter _send_command 9 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590c123c40 con 0x7f590c11ec00 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5911ea5700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6800/3344263388 conn(0x7f590c11ec00 0x7f590c120ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5902ffd700 10 client.36722.objecter ms_handle_connect 0x7f590c11ec00 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.518+0000 7f5911ea5700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6800/3344263388 conn(0x7f590c11ec00 0x7f590c120ff0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.519+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f590c123c40 con 0x7f590c11ec00 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.519+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.519+0000 7f5902ffd700 10 client.36722.objecter _finish_command 9 = system:0 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.519+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c11e580 osd=4 4 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.519+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 4 9 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.534+0000 7f5914109700 10 client.36722.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.534+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 10 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.534+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c11e580 osd=4 3 2024-02-13T00:42:29.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.534+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c11e580 osd=4 3 2024-02-13T00:42:29.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.534+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c11e580 osd=4 3 2024-02-13T00:42:29.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.534+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 10 2024-02-13T00:42:29.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.534+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c11e580 osd=4 3 2024-02-13T00:42:29.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.534+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 4 10 2024-02-13T00:42:29.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.534+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c11e580 osd=4 4 2024-02-13T00:42:29.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.534+0000 7f5914109700 10 client.36722.objecter _send_command 10 2024-02-13T00:42:29.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.534+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f590c1214e0 con 0x7f590c11ec00 2024-02-13T00:42:29.758 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:42:29.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.535+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f590c1214e0 con 0x7f590c11ec00 2024-02-13T00:42:29.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.535+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.535+0000 7f5902ffd700 10 client.36722.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.535+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c11e580 osd=4 4 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.535+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 4 10 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5914109700 10 client.36722.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 11 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6804/2212296748 conn(0x7f590c1231a0 0x7f590c1235e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c1214e0 osd=5 3 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c1214e0 osd=5 3 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c1214e0 osd=5 3 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 11 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c1214e0 osd=5 3 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 5 11 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c1214e0 osd=5 4 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5914109700 10 client.36722.objecter _send_command 11 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590c126fc0 con 0x7f590c1231a0 2024-02-13T00:42:29.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f59116a4700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6804/2212296748 conn(0x7f590c1231a0 0x7f590c1235e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:29.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f59116a4700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6804/2212296748 conn(0x7f590c1231a0 0x7f590c1235e0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:29.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.536+0000 7f5902ffd700 10 client.36722.objecter ms_handle_connect 0x7f590c1231a0 2024-02-13T00:42:29.760 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:42:29.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.537+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f590c126fc0 con 0x7f590c1231a0 2024-02-13T00:42:29.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.537+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.537+0000 7f5902ffd700 10 client.36722.objecter _finish_command 11 = system:0 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.537+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c1214e0 osd=5 4 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.537+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 5 11 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.551+0000 7f5914109700 10 client.36722.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.551+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 12 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.551+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c1214e0 osd=5 3 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.551+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c1214e0 osd=5 3 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.551+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c1214e0 osd=5 3 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.551+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 12 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.551+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c1214e0 osd=5 3 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.551+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 5 12 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.551+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c1214e0 osd=5 4 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.551+0000 7f5914109700 10 client.36722.objecter _send_command 12 2024-02-13T00:42:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.551+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f590c121680 con 0x7f590c1231a0 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f590c121680 con 0x7f590c1231a0 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5902ffd700 10 client.36722.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c1214e0 osd=5 4 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 5 12 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5914109700 10 client.36722.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 13 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6808/1506640341 conn(0x7f590c121d00 0x7f590c122160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c121680 osd=6 3 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c121680 osd=6 3 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c121680 osd=6 3 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 13 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c121680 osd=6 3 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 6 13 2024-02-13T00:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c121680 osd=6 4 2024-02-13T00:42:29.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5914109700 10 client.36722.objecter _send_command 13 2024-02-13T00:42:29.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.553+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590c122760 con 0x7f590c121d00 2024-02-13T00:42:29.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.554+0000 7f59126a6700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6808/1506640341 conn(0x7f590c121d00 0x7f590c122160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:29.763 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:42:29.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.554+0000 7f5902ffd700 10 client.36722.objecter ms_handle_connect 0x7f590c121d00 2024-02-13T00:42:29.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.554+0000 7f59126a6700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6808/1506640341 conn(0x7f590c121d00 0x7f590c122160 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:29.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.555+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f590c122760 con 0x7f590c121d00 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.555+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.555+0000 7f5902ffd700 10 client.36722.objecter _finish_command 13 = system:0 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.555+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c121680 osd=6 4 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.555+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 6 13 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.570+0000 7f5914109700 10 client.36722.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.570+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 14 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.570+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c121680 osd=6 3 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.570+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c121680 osd=6 3 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.570+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c121680 osd=6 3 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.570+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 14 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.570+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c121680 osd=6 3 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.570+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 6 14 2024-02-13T00:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.570+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c121680 osd=6 4 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.570+0000 7f5914109700 10 client.36722.objecter _send_command 14 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.570+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f590c122760 con 0x7f590c121d00 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f590c122760 con 0x7f590c121d00 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5902ffd700 10 client.36722.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c121680 osd=6 4 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 6 14 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5914109700 10 client.36722.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 15 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6812/1964192772 conn(0x7f590c128360 0x7f590c12a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c122760 osd=7 3 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c122760 osd=7 3 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c122760 osd=7 3 2024-02-13T00:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 15 2024-02-13T00:42:29.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c122760 osd=7 3 2024-02-13T00:42:29.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 7 15 2024-02-13T00:42:29.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c122760 osd=7 4 2024-02-13T00:42:29.766 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:42:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5914109700 10 client.36722.objecter _send_command 15 2024-02-13T00:42:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.572+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590c12d3a0 con 0x7f590c128360 2024-02-13T00:42:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.573+0000 7f5911ea5700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6812/1964192772 conn(0x7f590c128360 0x7f590c12a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.573+0000 7f5911ea5700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6812/1964192772 conn(0x7f590c128360 0x7f590c12a750 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.573+0000 7f5902ffd700 10 client.36722.objecter ms_handle_connect 0x7f590c128360 2024-02-13T00:42:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.574+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f590c12d3a0 con 0x7f590c128360 2024-02-13T00:42:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.574+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:42:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.574+0000 7f5902ffd700 10 client.36722.objecter _finish_command 15 = system:0 2024-02-13T00:42:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.574+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c122760 osd=7 4 2024-02-13T00:42:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.574+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 7 15 2024-02-13T00:42:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.589+0000 7f5914109700 10 client.36722.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:29.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.589+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign -1 16 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.589+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c122760 osd=7 3 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.589+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c122760 osd=7 3 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.589+0000 7f5914109700 20 client.36722.objecter _get_session s=0x7f590c122760 osd=7 3 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.589+0000 7f5914109700 15 client.36722.objecter _session_command_op_remove -1 16 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.589+0000 7f5914109700 20 client.36722.objecter get_session s=0x7f590c122760 osd=7 3 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.589+0000 7f5914109700 15 client.36722.objecter _session_command_op_assign 7 16 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.589+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c122760 osd=7 4 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.589+0000 7f5914109700 10 client.36722.objecter _send_command 16 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.589+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f590c12ac40 con 0x7f590c128360 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5902ffd700 1 -- 172.21.15.162:0/1821082507 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f590c12ac40 con 0x7f590c128360 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5902ffd700 10 client.36722.objecter ms_dispatch 0x7f590c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5902ffd700 10 client.36722.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5902ffd700 20 client.36722.objecter put_session s=0x7f590c122760 osd=7 4 2024-02-13T00:42:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5902ffd700 15 client.36722.objecter _session_command_op_remove 7 16 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 10 librados: watch_flush enter 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 10 librados: watch_flush exit 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 10 client.36722.objecter close_session for osd.0 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6801/2103743338 conn(0x7f590c0fef10 msgr2=0x7f590c0ff2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6801/2103743338 conn(0x7f590c0fef10 0x7f590c0ff2f0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c05fb00 osd=0 1 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 10 client.36722.objecter close_session for osd.1 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6805/150599780 conn(0x7f590c059990 msgr2=0x7f590c059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6805/150599780 conn(0x7f590c059990 0x7f590c059d70 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c028760 osd=1 1 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 10 client.36722.objecter close_session for osd.2 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6809/3689719376 conn(0x7f590c06b850 msgr2=0x7f590c06bc50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6809/3689719376 conn(0x7f590c06b850 0x7f590c06bc50 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c06b1d0 osd=2 1 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 10 client.36722.objecter close_session for osd.3 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6813/604971011 conn(0x7f590c06cd80 msgr2=0x7f590c11e090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6813/604971011 conn(0x7f590c06cd80 0x7f590c11e090 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f59126a6700 1 -- 172.21.15.162:0/1821082507 reap_dead start 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c06cc10 osd=3 1 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 10 client.36722.objecter close_session for osd.4 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6800/3344263388 conn(0x7f590c11ec00 msgr2=0x7f590c120ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6800/3344263388 conn(0x7f590c11ec00 0x7f590c120ff0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f59126a6700 1 -- 172.21.15.162:0/1821082507 reap_dead start 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c11e580 osd=4 1 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 10 client.36722.objecter close_session for osd.5 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6804/2212296748 conn(0x7f590c1231a0 msgr2=0x7f590c1235e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6804/2212296748 conn(0x7f590c1231a0 0x7f590c1235e0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c1214e0 osd=5 1 2024-02-13T00:42:29.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 10 client.36722.objecter close_session for osd.6 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6808/1506640341 conn(0x7f590c121d00 msgr2=0x7f590c122160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6808/1506640341 conn(0x7f590c121d00 0x7f590c122160 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c121680 osd=6 1 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 10 client.36722.objecter close_session for osd.7 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6812/1964192772 conn(0x7f590c128360 msgr2=0x7f590c12a750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.181:6812/1964192772 conn(0x7f590c128360 0x7f590c12a750 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 20 client.36722.objecter put_session s=0x7f590c122760 osd=7 1 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 20 client.36722.objecter shutdown clearing up homeless session... 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.591+0000 7f5914109700 10 client.36722.objecter successfully canceled tick 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.592+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6800/2866534053 conn(0x7f58f8062c90 msgr2=0x7f58f8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.592+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> v2:172.21.15.162:6800/2866534053 conn(0x7f58f8062c90 0x7f58f8065150 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f5908007d30 tx=0x7f5908014040).stop 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.592+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590c0fe5a0 msgr2=0x7f590c108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.592+0000 7f5914109700 1 --2- 172.21.15.162:0/1821082507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590c0fe5a0 0x7f590c108240 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7f5904009ad0 tx=0x7f5904004c50).stop 2024-02-13T00:42:29.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.592+0000 7f59126a6700 1 -- 172.21.15.162:0/1821082507 reap_dead start 2024-02-13T00:42:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.592+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 shutdown_connections 2024-02-13T00:42:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.592+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 >> 172.21.15.162:0/1821082507 conn(0x7f590c0f3850 msgr2=0x7f590c0f50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.592+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 shutdown_connections 2024-02-13T00:42:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.592+0000 7f5914109700 1 -- 172.21.15.162:0/1821082507 wait complete. 2024-02-13T00:42:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:29.592+0000 7f5914109700 1 librados: shutdown 2024-02-13T00:42:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.772 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.773 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:42:29.961 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:29.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:28.034472+0000 osd.1 (osd.1) 95 : cluster [DBG] 5.c scrub starts 2024-02-13T00:42:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:28.037361+0000 osd.1 (osd.1) 96 : cluster [DBG] 5.c scrub ok 2024-02-13T00:42:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:28.335463+0000 mgr.y (mgr.14544) 221 : cluster [DBG] pgmap v259: 308 pgs: 12 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 40 unknown, 204 active+clean; 31 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-13T00:42:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:29.122583+0000 mon.b (mon.1) 251 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:29.342617+0000 mon.a (mon.0) 1376 : audit [INF] from='client.? 172.21.15.162:0/580184505' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi162-84049-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:29.342820+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e194: 8 total, 8 up, 7 in 2024-02-13T00:42:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:29.353478+0000 mon.b (mon.1) 252 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:30.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:29.354458+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:30.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:29.356591+0000 mon.b (mon.1) 253 : audit [INF] from='client.? 172.21.15.162:0/1640859285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-83985-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:30.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:29.358081+0000 mon.a (mon.0) 1379 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-83985-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:30.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:29.412724+0000 mon.c (mon.2) 168 : audit [DBG] from='client.? 172.21.15.162:0/1821082507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:30.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:30.095 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:30.114 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:30.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:30.220 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:30.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:28.034472+0000 osd.1 (osd.1) 95 : cluster [DBG] 5.c scrub starts 2024-02-13T00:42:30.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:28.037361+0000 osd.1 (osd.1) 96 : cluster [DBG] 5.c scrub ok 2024-02-13T00:42:30.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:28.335463+0000 mgr.y (mgr.14544) 221 : cluster [DBG] pgmap v259: 308 pgs: 12 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 40 unknown, 204 active+clean; 31 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-13T00:42:30.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:29.122583+0000 mon.b (mon.1) 251 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:30.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:29.342617+0000 mon.a (mon.0) 1376 : audit [INF] from='client.? 172.21.15.162:0/580184505' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi162-84049-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:30.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:29.342820+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e194: 8 total, 8 up, 7 in 2024-02-13T00:42:30.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:29.353478+0000 mon.b (mon.1) 252 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:30.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:29.354458+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:30.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:29.356591+0000 mon.b (mon.1) 253 : audit [INF] from='client.? 172.21.15.162:0/1640859285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-83985-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:30.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:29.358081+0000 mon.a (mon.0) 1379 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-83985-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:30.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:29.412724+0000 mon.c (mon.2) 168 : audit [DBG] from='client.? 172.21.15.162:0/1821082507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:30.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:28.034472+0000 osd.1 (osd.1) 95 : cluster [DBG] 5.c scrub starts 2024-02-13T00:42:30.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:28.037361+0000 osd.1 (osd.1) 96 : cluster [DBG] 5.c scrub ok 2024-02-13T00:42:30.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:28.335463+0000 mgr.y (mgr.14544) 221 : cluster [DBG] pgmap v259: 308 pgs: 12 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 40 unknown, 204 active+clean; 31 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-13T00:42:30.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:29.122583+0000 mon.b (mon.1) 251 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:30.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:29.342617+0000 mon.a (mon.0) 1376 : audit [INF] from='client.? 172.21.15.162:0/580184505' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi162-84049-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:30.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:29.342820+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e194: 8 total, 8 up, 7 in 2024-02-13T00:42:30.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:29.353478+0000 mon.b (mon.1) 252 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:30.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:29.354458+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:30.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:29.356591+0000 mon.b (mon.1) 253 : audit [INF] from='client.? 172.21.15.162:0/1640859285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-83985-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:30.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:29.358081+0000 mon.a (mon.0) 1379 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-83985-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:30.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:29.412724+0000 mon.c (mon.2) 168 : audit [DBG] from='client.? 172.21.15.162:0/1821082507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:30.248 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:30.353 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:42:30.353 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:42:30.448 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:30.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:30.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:30.551 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:30.582 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:30.635 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:30.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:30.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:30.766 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:30.785 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:30.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:30.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:30.915 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:30.933 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:30.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:31.037 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:42:31.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.124151+0000 mon.b (mon.1) 254 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:31.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.319733+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]': finished 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.319832+0000 mon.a (mon.0) 1381 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-83985-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:30.319948+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e195: 8 total, 8 up, 7 in 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.327133+0000 mon.b (mon.1) 255 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.335534+0000 mon.a (mon.0) 1383 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.340725+0000 mon.b (mon.1) 256 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.353425+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.374626+0000 mon.c (mon.2) 169 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.374977+0000 mon.a (mon.0) 1385 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.375400+0000 mon.c (mon.2) 170 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.375622+0000 mon.a (mon.0) 1386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.376006+0000 mon.c (mon.2) 171 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:30.376242+0000 mon.a (mon.0) 1387 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:30.407525+0000 osd.3 (osd.3) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:30.407572+0000 osd.3 (osd.3) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:31.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.124151+0000 mon.b (mon.1) 254 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:31.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.319733+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]': finished 2024-02-13T00:42:31.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.319832+0000 mon.a (mon.0) 1381 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-83985-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:31.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:30.319948+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e195: 8 total, 8 up, 7 in 2024-02-13T00:42:31.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.327133+0000 mon.b (mon.1) 255 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:31.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.335534+0000 mon.a (mon.0) 1383 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:31.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.340725+0000 mon.b (mon.1) 256 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:31.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.353425+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:31.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.374626+0000 mon.c (mon.2) 169 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.374977+0000 mon.a (mon.0) 1385 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.375400+0000 mon.c (mon.2) 170 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.375622+0000 mon.a (mon.0) 1386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.376006+0000 mon.c (mon.2) 171 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:31.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.124151+0000 mon.b (mon.1) 254 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:31.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.319733+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-83977-27"}]': finished 2024-02-13T00:42:31.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.319832+0000 mon.a (mon.0) 1381 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-83985-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:31.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:30.319948+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e195: 8 total, 8 up, 7 in 2024-02-13T00:42:31.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.327133+0000 mon.b (mon.1) 255 : audit [INF] from='client.? 172.21.15.162:0/977122138' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:31.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.335534+0000 mon.a (mon.0) 1383 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]: dispatch 2024-02-13T00:42:31.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.340725+0000 mon.b (mon.1) 256 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:31.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.353425+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:31.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.374626+0000 mon.c (mon.2) 169 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.374977+0000 mon.a (mon.0) 1385 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.375400+0000 mon.c (mon.2) 170 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.375622+0000 mon.a (mon.0) 1386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.376006+0000 mon.c (mon.2) 171 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:31.065 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:30.376242+0000 mon.a (mon.0) 1387 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:31.065 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:30.407525+0000 osd.3 (osd.3) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:31.065 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:30.407572+0000 osd.3 (osd.3) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:31.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:30.376242+0000 mon.a (mon.0) 1387 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:31.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:30.407525+0000 osd.3 (osd.3) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:31.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:30.407572+0000 osd.3 (osd.3) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:31.065 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:31.083 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:42:31.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:31.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:42:31.165 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-13T00:42:31.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:31.235 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:31.259 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:31.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:31.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:31.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:31.395 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:31.416 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:31.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:31.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:31.462 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:42:31.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:31.546 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:31.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:42:31.744 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:31.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:31.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:31.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:31.855 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:31.874 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:31.949 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:42:31.949 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:42:31.949 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-02-13T00:42:31.949 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-13T00:42:31.949 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-13T00:42:31.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-02-13T00:42:31.975 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:30.341473+0000 mgr.y (mgr.14544) 222 : cluster [DBG] pgmap v263: 300 pgs: 32 creating+peering, 12 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 32 unknown, 196 active+clean; 44 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-02-13T00:42:31.975 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:31.125207+0000 mon.b (mon.1) 257 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:31.975 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:31.322698+0000 mon.a (mon.0) 1388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]': finished 2024-02-13T00:42:31.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:31.322778+0000 mon.a (mon.0) 1389 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:31.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:31.322853+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:31.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:31.322931+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e196: 8 total, 8 up, 7 in 2024-02-13T00:42:31.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:31.330683+0000 mon.c (mon.2) 172 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-84049-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:31.332054+0000 mon.a (mon.0) 1392 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-84049-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:31.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:31.368257+0000 mon.c (mon.2) 173 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:31.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:31.368722+0000 mon.a (mon.0) 1393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:31.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:31.369251+0000 mon.c (mon.2) 174 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:31.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:31.369744+0000 mon.a (mon.0) 1394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:31.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:31.370727+0000 mon.c (mon.2) 175 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-83977-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:31.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:31.371526+0000 mon.a (mon.0) 1395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-83977-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:30.341473+0000 mgr.y (mgr.14544) 222 : cluster [DBG] pgmap v263: 300 pgs: 32 creating+peering, 12 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 32 unknown, 196 active+clean; 44 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-02-13T00:42:32.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:31.125207+0000 mon.b (mon.1) 257 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:32.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:31.322698+0000 mon.a (mon.0) 1388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]': finished 2024-02-13T00:42:32.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:31.322778+0000 mon.a (mon.0) 1389 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:32.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:31.322853+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:32.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:31.322931+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e196: 8 total, 8 up, 7 in 2024-02-13T00:42:32.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:31.330683+0000 mon.c (mon.2) 172 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-84049-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:32.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:31.332054+0000 mon.a (mon.0) 1392 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-84049-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:32.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:31.368257+0000 mon.c (mon.2) 173 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:32.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:31.368722+0000 mon.a (mon.0) 1393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:32.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:31.369251+0000 mon.c (mon.2) 174 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:31.369744+0000 mon.a (mon.0) 1394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:31.370727+0000 mon.c (mon.2) 175 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-83977-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:31.371526+0000 mon.a (mon.0) 1395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-83977-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:30.341473+0000 mgr.y (mgr.14544) 222 : cluster [DBG] pgmap v263: 300 pgs: 32 creating+peering, 12 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 32 unknown, 196 active+clean; 44 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:31.125207+0000 mon.b (mon.1) 257 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:31.322698+0000 mon.a (mon.0) 1388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-83977-27"}]': finished 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:31.322778+0000 mon.a (mon.0) 1389 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:31.322853+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:31.322931+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e196: 8 total, 8 up, 7 in 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:31.330683+0000 mon.c (mon.2) 172 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-84049-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:31.332054+0000 mon.a (mon.0) 1392 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-84049-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:31.368257+0000 mon.c (mon.2) 173 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:32.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:31.368722+0000 mon.a (mon.0) 1393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:32.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:31.369251+0000 mon.c (mon.2) 174 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:32.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:31.369744+0000 mon.a (mon.0) 1394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:32.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:31.370727+0000 mon.c (mon.2) 175 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-83977-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:32.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:31.371526+0000 mon.a (mon.0) 1395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-83977-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:32.440 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:42:32.789 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:42:32.801 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:32.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:32.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:32.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:32.933 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:33.029 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:32.126243+0000 mon.b (mon.1) 258 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:33.029 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:32.565831+0000 mon.a (mon.0) 1396 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-83977-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:33.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:32.565966+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e197: 8 total, 8 up, 7 in 2024-02-13T00:42:33.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:32.566967+0000 mon.c (mon.2) 176 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-83977-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:33.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:32.569389+0000 mon.a (mon.0) 1398 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-83977-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:33.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:32.571410+0000 mon.a (mon.0) 1399 : audit [INF] from='client.? 172.21.15.162:0/3145559724' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-83985-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:33.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:32.572402+0000 mon.b (mon.1) 259 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:33.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:32.572873+0000 mon.a (mon.0) 1400 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:33.031 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:33.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:33.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:42:33.164 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:33.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:32.126243+0000 mon.b (mon.1) 258 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:33.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:32.565831+0000 mon.a (mon.0) 1396 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-83977-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:33.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:32.565966+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e197: 8 total, 8 up, 7 in 2024-02-13T00:42:33.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:32.566967+0000 mon.c (mon.2) 176 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-83977-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:33.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:32.569389+0000 mon.a (mon.0) 1398 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-83977-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:33.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:32.571410+0000 mon.a (mon.0) 1399 : audit [INF] from='client.? 172.21.15.162:0/3145559724' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-83985-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:33.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:32.572402+0000 mon.b (mon.1) 259 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:33.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:32.572873+0000 mon.a (mon.0) 1400 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:33.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:32.126243+0000 mon.b (mon.1) 258 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:33.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:32.565831+0000 mon.a (mon.0) 1396 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-83977-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:33.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:32.565966+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e197: 8 total, 8 up, 7 in 2024-02-13T00:42:33.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:32.566967+0000 mon.c (mon.2) 176 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-83977-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:33.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:32.569389+0000 mon.a (mon.0) 1398 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-83977-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:33.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:32.571410+0000 mon.a (mon.0) 1399 : audit [INF] from='client.? 172.21.15.162:0/3145559724' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-83985-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:33.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:32.572402+0000 mon.b (mon.1) 259 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:33.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:32.572873+0000 mon.a (mon.0) 1400 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:33.412 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:42:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:33.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:42:33.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:33.522 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:42:33.522 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:42:33.547 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:33.646 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:33.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:33.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:33.779 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:33.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.922+0000 7f1ffc49c700 1 -- 172.21.15.162:0/1814857049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ff40fd490 msgr2=0x7f1ff40f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.922+0000 7f1ffc49c700 1 --2- 172.21.15.162:0/1814857049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ff40fd490 0x7f1ff40f6cc0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 rx=0x7f1ff0004870 tx=0x7f1ff0009cb0).stop 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.922+0000 7f1ffc49c700 1 -- 172.21.15.162:0/1814857049 shutdown_connections 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.922+0000 7f1ffc49c700 1 --2- 172.21.15.162:0/1814857049 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ff40f73a0 0x7f1ff41039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.922+0000 7f1ffc49c700 1 --2- 172.21.15.162:0/1814857049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ff40fcb30 0x7f1ff40fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.922+0000 7f1ffc49c700 1 --2- 172.21.15.162:0/1814857049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ff40fd490 0x7f1ff40f6cc0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.922+0000 7f1ffc49c700 1 -- 172.21.15.162:0/1814857049 >> 172.21.15.162:0/1814857049 conn(0x7f1ff406eb90 msgr2=0x7f1ff40f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.922+0000 7f1ffc49c700 1 -- 172.21.15.162:0/1814857049 shutdown_connections 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.922+0000 7f1ffc49c700 1 -- 172.21.15.162:0/1814857049 wait complete. 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.922+0000 7f1ffc49c700 1 librados: starting msgr at 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.922+0000 7f1ffc49c700 1 librados: starting objecter 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ffc49c700 1 Processor -- start 2024-02-13T00:42:33.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ffc49c700 1 -- start start 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ffc49c700 1 librados: setting wanted keys 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ffc49c700 1 librados: calling monclient init 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ffc49c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ff40f73a0 0x7f1ff410e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ffc49c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ff40fcb30 0x7f1ff410e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ffc49c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ff40fd490 0x7f1ff4108300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ffc49c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff41087f0 con 0x7f1ff40f73a0 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ffc49c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff4108960 con 0x7f1ff40fd490 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ffc49c700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff4108b20 con 0x7f1ff40fcb30 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ff9a37700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ff40fcb30 0x7f1ff410e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ff9a37700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ff40fcb30 0x7f1ff410e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:56938/0 (socket says 172.21.15.162:56938) 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ff9a37700 1 -- 172.21.15.162:0/4253641647 learned_addr learned my addr 172.21.15.162:0/4253641647 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ffaa39700 1 --2- 172.21.15.162:0/4253641647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ff40fd490 0x7f1ff4108300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ff9a37700 1 -- 172.21.15.162:0/4253641647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ff40fd490 msgr2=0x7f1ff4108300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:33.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ff9a37700 1 --2- 172.21.15.162:0/4253641647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ff40fd490 0x7f1ff4108300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:33.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ff9a37700 1 -- 172.21.15.162:0/4253641647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ff40f73a0 msgr2=0x7f1ff410e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:33.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ff9a37700 1 --2- 172.21.15.162:0/4253641647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ff40f73a0 0x7f1ff410e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:42:33.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ff9a37700 1 -- 172.21.15.162:0/4253641647 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec0097c0 con 0x7f1ff40fcb30 2024-02-13T00:42:33.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1ff9a37700 1 --2- 172.21.15.162:0/4253641647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ff40fcb30 0x7f1ff410e650 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f1ff0009eb0 tx=0x7f1ff000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:33.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1feb7fe700 10 client.?.objecter ms_handle_connect 0x7f1ff40fcb30 2024-02-13T00:42:33.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.923+0000 7f1feb7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:33.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.924+0000 7f1feb7fe700 1 -- 172.21.15.162:0/4253641647 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff000e070 con 0x7f1ff40fcb30 2024-02-13T00:42:33.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.924+0000 7f1feb7fe700 1 -- 172.21.15.162:0/4253641647 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ff0013da0 con 0x7f1ff40fcb30 2024-02-13T00:42:33.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.924+0000 7f1ffc49c700 1 -- 172.21.15.162:0/4253641647 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff0005040 con 0x7f1ff40fcb30 2024-02-13T00:42:33.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.924+0000 7f1feb7fe700 1 -- 172.21.15.162:0/4253641647 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0019070 con 0x7f1ff40fcb30 2024-02-13T00:42:33.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.924+0000 7f1ffc49c700 10 client.36743.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:33.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.924+0000 7f1ffc49c700 1 -- 172.21.15.162:0/4253641647 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff4109010 con 0x7f1ff40fcb30 2024-02-13T00:42:33.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.924+0000 7f1ffc49c700 1 librados: init done 2024-02-13T00:42:33.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.925+0000 7f1feb7fe700 1 -- 172.21.15.162:0/4253641647 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1ff001e420 con 0x7f1ff40fcb30 2024-02-13T00:42:33.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.925+0000 7f1ffc49c700 1 -- 172.21.15.162:0/4253641647 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff40475e0 con 0x7f1ff40fcb30 2024-02-13T00:42:33.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.925+0000 7f1feb7fe700 1 --2- 172.21.15.162:0/4253641647 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fe0062c90 0x7f1fe0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:33.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.925+0000 7f1feb7fe700 1 -- 172.21.15.162:0/4253641647 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(198..198 src has 52..198) v4 ==== 10798+0+0 (secure 0 0 0) 0x7f1ff007e890 con 0x7f1ff40fcb30 2024-02-13T00:42:33.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.925+0000 7f1feb7fe700 10 client.36743.objecter ms_dispatch 0x7f1ff4001550 osd_map(198..198 src has 52..198) v4 2024-02-13T00:42:33.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.925+0000 7f1feb7fe700 3 client.36743.objecter handle_osd_map got epochs [198,198] > 0 2024-02-13T00:42:33.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.925+0000 7f1feb7fe700 3 client.36743.objecter handle_osd_map decoding full epoch 198 2024-02-13T00:42:33.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.925+0000 7f1feb7fe700 20 client.36743.objecter dump_active .. 0 homeless 2024-02-13T00:42:33.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.926+0000 7f1ffa238700 1 --2- 172.21.15.162:0/4253641647 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fe0062c90 0x7f1fe0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:33.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.926+0000 7f1ffa238700 1 --2- 172.21.15.162:0/4253641647 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fe0062c90 0x7f1fe0065150 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 rx=0x7f1fe4004950 tx=0x7f1fe4007bc0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:33.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.926+0000 7f1feb7fe700 10 client.36743.objecter ms_handle_connect 0x7f1fe0062c90 2024-02-13T00:42:33.933 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:33.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:33.931+0000 7f1feb7fe700 1 -- 172.21.15.162:0/4253641647 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1ff00512b0 con 0x7f1ff40fcb30 2024-02-13T00:42:33.935 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:33.965 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:34.114 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:34.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:34.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.123+0000 7f1ffc49c700 1 -- 172.21.15.162:0/4253641647 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f1ff4109810 con 0x7f1ff40fcb30 2024-02-13T00:42:34.218 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:34.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:32.341972+0000 mgr.y (mgr.14544) 223 : cluster [DBG] pgmap v266: 300 pgs: 12 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 64 unknown, 196 active+clean; 44 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-02-13T00:42:34.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:33.127278+0000 mon.b (mon.1) 260 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:34.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:33.326475+0000 mon.a (mon.0) 1401 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-84049-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-84049-36"}]': finished 2024-02-13T00:42:34.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:33.582725+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? 172.21.15.162:0/3145559724' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-83985-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:34.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:33.582847+0000 mon.a (mon.0) 1403 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:42:34.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:33.582909+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e198: 8 total, 8 up, 7 in 2024-02-13T00:42:34.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:33.583536+0000 mon.b (mon.1) 261 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-15"}]: dispatch 2024-02-13T00:42:34.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:33.588907+0000 mon.a (mon.0) 1405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-15"}]: dispatch 2024-02-13T00:42:34.222 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:32.341972+0000 mgr.y (mgr.14544) 223 : cluster [DBG] pgmap v266: 300 pgs: 12 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 64 unknown, 196 active+clean; 44 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-02-13T00:42:34.222 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:33.127278+0000 mon.b (mon.1) 260 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:34.222 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:33.326475+0000 mon.a (mon.0) 1401 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-84049-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-84049-36"}]': finished 2024-02-13T00:42:34.222 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:33.582725+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? 172.21.15.162:0/3145559724' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-83985-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:34.222 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:33.582847+0000 mon.a (mon.0) 1403 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:42:34.222 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:33.582909+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e198: 8 total, 8 up, 7 in 2024-02-13T00:42:34.222 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:33.583536+0000 mon.b (mon.1) 261 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-15"}]: dispatch 2024-02-13T00:42:34.222 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:33.588907+0000 mon.a (mon.0) 1405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-15"}]: dispatch 2024-02-13T00:42:34.222 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:32.341972+0000 mgr.y (mgr.14544) 223 : cluster [DBG] pgmap v266: 300 pgs: 12 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 64 unknown, 196 active+clean; 44 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-02-13T00:42:34.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:33.127278+0000 mon.b (mon.1) 260 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:34.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:33.326475+0000 mon.a (mon.0) 1401 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-84049-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-84049-36"}]': finished 2024-02-13T00:42:34.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:33.582725+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? 172.21.15.162:0/3145559724' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-83985-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:34.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:33.582847+0000 mon.a (mon.0) 1403 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:42:34.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:33.582909+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e198: 8 total, 8 up, 7 in 2024-02-13T00:42:34.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:33.583536+0000 mon.b (mon.1) 261 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-15"}]: dispatch 2024-02-13T00:42:34.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:33.588907+0000 mon.a (mon.0) 1405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-15"}]: dispatch 2024-02-13T00:42:34.244 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:34.264 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:34.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:34.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:34.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.337+0000 7f1feb7fe700 1 -- 172.21.15.162:0/4253641647 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v199) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1ff0023030 con 0x7f1ff40fcb30 2024-02-13T00:42:34.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.355+0000 7f1ffc49c700 10 librados: watch_flush enter 2024-02-13T00:42:34.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.356+0000 7f1ffc49c700 10 librados: watch_flush exit 2024-02-13T00:42:34.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.356+0000 7f1ffc49c700 20 client.36743.objecter shutdown clearing up homeless session... 2024-02-13T00:42:34.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.356+0000 7f1ffc49c700 10 client.36743.objecter successfully canceled tick 2024-02-13T00:42:34.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.356+0000 7f1ffc49c700 1 -- 172.21.15.162:0/4253641647 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fe0062c90 msgr2=0x7f1fe0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:34.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.356+0000 7f1ffc49c700 1 --2- 172.21.15.162:0/4253641647 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fe0062c90 0x7f1fe0065150 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 rx=0x7f1fe4004950 tx=0x7f1fe4007bc0).stop 2024-02-13T00:42:34.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.356+0000 7f1ffc49c700 1 -- 172.21.15.162:0/4253641647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ff40fcb30 msgr2=0x7f1ff410e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:34.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.356+0000 7f1ffc49c700 1 --2- 172.21.15.162:0/4253641647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ff40fcb30 0x7f1ff410e650 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f1ff0009eb0 tx=0x7f1ff000b040).stop 2024-02-13T00:42:34.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.357+0000 7f1ffc49c700 1 -- 172.21.15.162:0/4253641647 shutdown_connections 2024-02-13T00:42:34.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.357+0000 7f1ffc49c700 1 --2- 172.21.15.162:0/4253641647 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fe0062c90 0x7f1fe0065150 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:34.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.357+0000 7f1ffc49c700 1 --2- 172.21.15.162:0/4253641647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ff40fd490 0x7f1ff4108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:34.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.357+0000 7f1ffc49c700 1 --2- 172.21.15.162:0/4253641647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ff40f73a0 0x7f1ff410e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:42:34.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.357+0000 7f1ffc49c700 1 --2- 172.21.15.162:0/4253641647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ff40fcb30 0x7f1ff410e650 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:34.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.357+0000 7f1ffc49c700 1 -- 172.21.15.162:0/4253641647 >> 172.21.15.162:0/4253641647 conn(0x7f1ff406eb90 msgr2=0x7f1ff406f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:34.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.359+0000 7f1ffc49c700 1 -- 172.21.15.162:0/4253641647 shutdown_connections 2024-02-13T00:42:34.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.359+0000 7f1ffc49c700 1 -- 172.21.15.162:0/4253641647 wait complete. 2024-02-13T00:42:34.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:34.360+0000 7f1ffc49c700 1 librados: shutdown 2024-02-13T00:42:34.366 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:42:34.368 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:34.398 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:35.224 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:42:35.225 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-13T00:42:35.225 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:42:35.246 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:34.125605+0000 mon.c (mon.2) 177 : audit [INF] from='client.? 172.21.15.162:0/4253641647' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:35.246 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:34.125907+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:35.246 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:34.128270+0000 mon.b (mon.1) 262 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:35.246 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:34.321987+0000 mon.a (mon.0) 1407 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:35.246 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:34.330677+0000 mon.a (mon.0) 1408 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-83977-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-83977-28"}]': finished 2024-02-13T00:42:35.246 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:34.330884+0000 mon.a (mon.0) 1409 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-15"}]': finished 2024-02-13T00:42:35.246 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:34.330995+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:42:35.246 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:34.331051+0000 mon.a (mon.0) 1411 : cluster [DBG] osdmap e199: 8 total, 8 up, 7 in 2024-02-13T00:42:35.246 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:34.337336+0000 mon.b (mon.1) 263 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-15", "mode": "writeback"}]: dispatch 2024-02-13T00:42:35.247 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:34.338366+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-15", "mode": "writeback"}]: dispatch 2024-02-13T00:42:35.247 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:34.350482+0000 mon.a (mon.0) 1413 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:35.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:34.125605+0000 mon.c (mon.2) 177 : audit [INF] from='client.? 172.21.15.162:0/4253641647' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:35.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:34.125907+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:35.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:34.128270+0000 mon.b (mon.1) 262 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:35.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:34.321987+0000 mon.a (mon.0) 1407 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:35.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:34.330677+0000 mon.a (mon.0) 1408 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-83977-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-83977-28"}]': finished 2024-02-13T00:42:35.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:34.330884+0000 mon.a (mon.0) 1409 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-15"}]': finished 2024-02-13T00:42:35.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:34.330995+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:42:35.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:34.331051+0000 mon.a (mon.0) 1411 : cluster [DBG] osdmap e199: 8 total, 8 up, 7 in 2024-02-13T00:42:35.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:34.337336+0000 mon.b (mon.1) 263 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-15", "mode": "writeback"}]: dispatch 2024-02-13T00:42:35.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:34.338366+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-15", "mode": "writeback"}]: dispatch 2024-02-13T00:42:35.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:34.350482+0000 mon.a (mon.0) 1413 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:34.125605+0000 mon.c (mon.2) 177 : audit [INF] from='client.? 172.21.15.162:0/4253641647' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:34.125907+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:34.128270+0000 mon.b (mon.1) 262 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:34.321987+0000 mon.a (mon.0) 1407 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:34.330677+0000 mon.a (mon.0) 1408 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-83977-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-83977-28"}]': finished 2024-02-13T00:42:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:34.330884+0000 mon.a (mon.0) 1409 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-15"}]': finished 2024-02-13T00:42:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:34.330995+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:42:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:34.331051+0000 mon.a (mon.0) 1411 : cluster [DBG] osdmap e199: 8 total, 8 up, 7 in 2024-02-13T00:42:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:34.337336+0000 mon.b (mon.1) 263 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-15", "mode": "writeback"}]: dispatch 2024-02-13T00:42:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:34.338366+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-15", "mode": "writeback"}]: dispatch 2024-02-13T00:42:35.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:34.350482+0000 mon.a (mon.0) 1413 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:35.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:42:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:42:35.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-13T00:42:36.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.015+0000 7f1692c15700 1 -- 172.21.15.162:0/4034947159 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f168c102270 msgr2=0x7f168c104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.015+0000 7f1692c15700 1 --2- 172.21.15.162:0/4034947159 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f168c102270 0x7f168c104660 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 rx=0x7f1684009c30 tx=0x7f1684004c50).stop 2024-02-13T00:42:36.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.015+0000 7f1692c15700 1 -- 172.21.15.162:0/4034947159 shutdown_connections 2024-02-13T00:42:36.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.015+0000 7f1692c15700 1 --2- 172.21.15.162:0/4034947159 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f168c102270 0x7f168c104660 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.015+0000 7f1692c15700 1 --2- 172.21.15.162:0/4034947159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f168c0f8d10 0x7f168c0f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.015+0000 7f1692c15700 1 --2- 172.21.15.162:0/4034947159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f168c0f95e0 0x7f168c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.015+0000 7f1692c15700 1 -- 172.21.15.162:0/4034947159 >> 172.21.15.162:0/4034947159 conn(0x7f168c06eb90 msgr2=0x7f168c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:36.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.015+0000 7f1692c15700 1 -- 172.21.15.162:0/4034947159 shutdown_connections 2024-02-13T00:42:36.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.015+0000 7f1692c15700 1 -- 172.21.15.162:0/4034947159 wait complete. 2024-02-13T00:42:36.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.015+0000 7f1692c15700 1 librados: starting msgr at 2024-02-13T00:42:36.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.015+0000 7f1692c15700 1 librados: starting objecter 2024-02-13T00:42:36.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.015+0000 7f1692c15700 1 Processor -- start 2024-02-13T00:42:36.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f1692c15700 1 -- start start 2024-02-13T00:42:36.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f1692c15700 1 librados: setting wanted keys 2024-02-13T00:42:36.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f1692c15700 1 librados: calling monclient init 2024-02-13T00:42:36.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f1692c15700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f168c0f8d10 0x7f168c10e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f1692c15700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f168c0f95e0 0x7f168c10e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f1692c15700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f168c102270 0x7f168c108270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f1692c15700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c108760 con 0x7f168c0f95e0 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f1692c15700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c1088d0 con 0x7f168c102270 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f1692c15700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168c108a90 con 0x7f168c0f8d10 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f16909b1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f168c0f8d10 0x7f168c10e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f168bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f168c0f95e0 0x7f168c10e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f16909b1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f168c0f8d10 0x7f168c10e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:50052/0 (socket says 172.21.15.162:50052) 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f16909b1700 1 -- 172.21.15.162:0/3741717118 learned_addr learned my addr 172.21.15.162:0/3741717118 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f16911b2700 1 --2- 172.21.15.162:0/3741717118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f168c102270 0x7f168c108270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f16909b1700 1 -- 172.21.15.162:0/3741717118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f168c102270 msgr2=0x7f168c108270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f16909b1700 1 --2- 172.21.15.162:0/3741717118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f168c102270 0x7f168c108270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f16909b1700 1 -- 172.21.15.162:0/3741717118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f168c0f95e0 msgr2=0x7f168c10e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f16909b1700 1 --2- 172.21.15.162:0/3741717118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f168c0f95e0 0x7f168c10e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f16909b1700 1 -- 172.21.15.162:0/3741717118 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680005040 con 0x7f168c0f8d10 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f168bfff700 1 --2- 172.21.15.162:0/3741717118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f168c0f95e0 0x7f168c10e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f16909b1700 1 --2- 172.21.15.162:0/3741717118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f168c0f8d10 0x7f168c10e0d0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7f1680009400 tx=0x7f1680009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f1689ffb700 10 client.?.objecter ms_handle_connect 0x7f168c0f8d10 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.016+0000 7f1689ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.017+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680007a40 con 0x7f168c0f8d10 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.017+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f168000f040 con 0x7f168c0f8d10 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.017+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16840097c0 con 0x7f168c0f8d10 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.017+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680014a70 con 0x7f168c0f8d10 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.017+0000 7f1692c15700 10 client.36755.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.017+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168c109070 con 0x7f168c0f8d10 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.017+0000 7f1692c15700 1 librados: init done 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.018+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f168c107170 con 0x7f168c0f8d10 2024-02-13T00:42:36.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.018+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1680010530 con 0x7f168c0f8d10 2024-02-13T00:42:36.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.018+0000 7f1689ffb700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6800/2866534053 conn(0x7f167c062c90 0x7f167c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:36.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(200..200 src has 52..200) v4 ==== 10289+0+0 (secure 0 0 0) 0x7f168007d660 con 0x7f168c0f8d10 2024-02-13T00:42:36.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 osd_map(200..200 src has 52..200) v4 2024-02-13T00:42:36.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1689ffb700 3 client.36755.objecter handle_osd_map got epochs [200,200] > 0 2024-02-13T00:42:36.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1689ffb700 3 client.36755.objecter handle_osd_map decoding full epoch 200 2024-02-13T00:42:36.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f168bfff700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6800/2866534053 conn(0x7f167c062c90 0x7f167c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:36.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1689ffb700 20 client.36755.objecter dump_active .. 0 homeless 2024-02-13T00:42:36.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v200) v1 ==== 54+0+16 (secure 0 0 0) 0x7f1680084070 con 0x7f168c0f8d10 2024-02-13T00:42:36.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1689ffb700 10 client.36755.objecter ms_handle_connect 0x7f167c062c90 2024-02-13T00:42:36.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f168bfff700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6800/2866534053 conn(0x7f167c062c90 0x7f167c065150 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 rx=0x7f168c109a10 tx=0x7f167800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:36.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1692c15700 10 client.36755.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:36.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 1 2024-02-13T00:42:36.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6801/2103743338 conn(0x7f168c028590 0x7f168c0fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:36.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c109750 osd=0 3 2024-02-13T00:42:36.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c109750 osd=0 3 2024-02-13T00:42:36.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c109750 osd=0 3 2024-02-13T00:42:36.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 1 2024-02-13T00:42:36.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c109750 osd=0 3 2024-02-13T00:42:36.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 0 1 2024-02-13T00:42:36.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c109750 osd=0 4 2024-02-13T00:42:36.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1692c15700 10 client.36755.objecter _send_command 1 2024-02-13T00:42:36.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f168c0f8b70 con 0x7f168c028590 2024-02-13T00:42:36.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f16911b2700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6801/2103743338 conn(0x7f168c028590 0x7f168c0fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:36.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f16911b2700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6801/2103743338 conn(0x7f168c028590 0x7f168c0fbd00 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:36.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.019+0000 7f1689ffb700 10 client.36755.objecter ms_handle_connect 0x7f168c028590 2024-02-13T00:42:36.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.020+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f168c0f8b70 con 0x7f168c028590 2024-02-13T00:42:36.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.020+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:42:36.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.020+0000 7f1689ffb700 10 client.36755.objecter _finish_command 1 = system:0 2024-02-13T00:42:36.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.020+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c109750 osd=0 4 2024-02-13T00:42:36.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.020+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 0 1 2024-02-13T00:42:36.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.034+0000 7f1692c15700 10 client.36755.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:36.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.035+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 2 2024-02-13T00:42:36.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.035+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c109750 osd=0 3 2024-02-13T00:42:36.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.035+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c109750 osd=0 3 2024-02-13T00:42:36.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.035+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c109750 osd=0 3 2024-02-13T00:42:36.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.035+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 2 2024-02-13T00:42:36.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.035+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c109750 osd=0 3 2024-02-13T00:42:36.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.035+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 0 2 2024-02-13T00:42:36.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.035+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c109750 osd=0 4 2024-02-13T00:42:36.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.035+0000 7f1692c15700 10 client.36755.objecter _send_command 2 2024-02-13T00:42:36.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.035+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f168c05a0e0 con 0x7f168c028590 2024-02-13T00:42:36.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f168c05a0e0 con 0x7f168c028590 2024-02-13T00:42:36.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:36.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1689ffb700 10 client.36755.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c109750 osd=0 4 2024-02-13T00:42:36.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 0 2 2024-02-13T00:42:36.041 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:42:36.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1692c15700 10 client.36755.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 3 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6805/150599780 conn(0x7f168c027be0 0x7f168c028000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c0fd370 osd=1 3 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c0fd370 osd=1 3 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c0fd370 osd=1 3 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 3 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c0fd370 osd=1 3 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 1 3 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c0fd370 osd=1 4 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1692c15700 10 client.36755.objecter _send_command 3 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f168c0fd4e0 con 0x7f168c027be0 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.036+0000 7f16909b1700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6805/150599780 conn(0x7f168c027be0 0x7f168c028000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.037+0000 7f1689ffb700 10 client.36755.objecter ms_handle_connect 0x7f168c027be0 2024-02-13T00:42:36.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.037+0000 7f16909b1700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6805/150599780 conn(0x7f168c027be0 0x7f168c028000 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.037+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f168c0fd4e0 con 0x7f168c027be0 2024-02-13T00:42:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.037+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:42:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.037+0000 7f1689ffb700 10 client.36755.objecter _finish_command 3 = system:0 2024-02-13T00:42:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.037+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c0fd370 osd=1 4 2024-02-13T00:42:36.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.037+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 1 3 2024-02-13T00:42:36.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.052+0000 7f1692c15700 10 client.36755.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:36.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.052+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 4 2024-02-13T00:42:36.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.052+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c0fd370 osd=1 3 2024-02-13T00:42:36.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.052+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c0fd370 osd=1 3 2024-02-13T00:42:36.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.052+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c0fd370 osd=1 3 2024-02-13T00:42:36.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.052+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 4 2024-02-13T00:42:36.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.052+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c0fd370 osd=1 3 2024-02-13T00:42:36.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.052+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 1 4 2024-02-13T00:42:36.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.052+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c0fd370 osd=1 4 2024-02-13T00:42:36.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.052+0000 7f1692c15700 10 client.36755.objecter _send_command 4 2024-02-13T00:42:36.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.052+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f168c027650 con 0x7f168c027be0 2024-02-13T00:42:36.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f168c027650 con 0x7f168c027be0 2024-02-13T00:42:36.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:36.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1689ffb700 10 client.36755.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c0fd370 osd=1 4 2024-02-13T00:42:36.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 1 4 2024-02-13T00:42:36.058 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:42:36.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1692c15700 10 client.36755.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:36.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 5 2024-02-13T00:42:36.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6809/3689719376 conn(0x7f168c0fced0 0x7f168c11e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c027a60 osd=2 3 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c027a60 osd=2 3 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c027a60 osd=2 3 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 5 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c027a60 osd=2 3 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 2 5 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c027a60 osd=2 4 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1692c15700 10 client.36755.objecter _send_command 5 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.053+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f168c121100 con 0x7f168c0fced0 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.054+0000 7f168bfff700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6809/3689719376 conn(0x7f168c0fced0 0x7f168c11e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.054+0000 7f1689ffb700 10 client.36755.objecter ms_handle_connect 0x7f168c0fced0 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.054+0000 7f168bfff700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6809/3689719376 conn(0x7f168c0fced0 0x7f168c11e4b0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.054+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f168c121100 con 0x7f168c0fced0 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.054+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.054+0000 7f1689ffb700 10 client.36755.objecter _finish_command 5 = system:0 2024-02-13T00:42:36.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.054+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c027a60 osd=2 4 2024-02-13T00:42:36.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.054+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 2 5 2024-02-13T00:42:36.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.069+0000 7f1692c15700 10 client.36755.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:36.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.069+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 6 2024-02-13T00:42:36.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.069+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c027a60 osd=2 3 2024-02-13T00:42:36.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.069+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c027a60 osd=2 3 2024-02-13T00:42:36.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.069+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c027a60 osd=2 3 2024-02-13T00:42:36.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.069+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 6 2024-02-13T00:42:36.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.069+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c027a60 osd=2 3 2024-02-13T00:42:36.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.069+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 2 6 2024-02-13T00:42:36.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.069+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c027a60 osd=2 4 2024-02-13T00:42:36.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.069+0000 7f1692c15700 10 client.36755.objecter _send_command 6 2024-02-13T00:42:36.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.069+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f168c0fc430 con 0x7f168c0fced0 2024-02-13T00:42:36.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.070+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f168c0fc430 con 0x7f168c0fced0 2024-02-13T00:42:36.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.070+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:36.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.070+0000 7f1689ffb700 10 client.36755.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.070+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c027a60 osd=2 4 2024-02-13T00:42:36.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.070+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 2 6 2024-02-13T00:42:36.075 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:42:36.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1692c15700 10 client.36755.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:36.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 7 2024-02-13T00:42:36.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6813/604971011 conn(0x7f168c11c000 0x7f168c11fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:36.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c0fc840 osd=3 3 2024-02-13T00:42:36.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c0fc840 osd=3 3 2024-02-13T00:42:36.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c0fc840 osd=3 3 2024-02-13T00:42:36.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 7 2024-02-13T00:42:36.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c0fc840 osd=3 3 2024-02-13T00:42:36.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 3 7 2024-02-13T00:42:36.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c0fc840 osd=3 4 2024-02-13T00:42:36.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1692c15700 10 client.36755.objecter _send_command 7 2024-02-13T00:42:36.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f168c1200b0 con 0x7f168c11c000 2024-02-13T00:42:36.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f16911b2700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6813/604971011 conn(0x7f168c11c000 0x7f168c11fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:36.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f1689ffb700 10 client.36755.objecter ms_handle_connect 0x7f168c11c000 2024-02-13T00:42:36.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.071+0000 7f16911b2700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6813/604971011 conn(0x7f168c11c000 0x7f168c11fb20 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:36.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.072+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f168c1200b0 con 0x7f168c11c000 2024-02-13T00:42:36.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.072+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:42:36.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.072+0000 7f1689ffb700 10 client.36755.objecter _finish_command 7 = system:0 2024-02-13T00:42:36.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.072+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c0fc840 osd=3 4 2024-02-13T00:42:36.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.072+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 3 7 2024-02-13T00:42:36.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:34.344163+0000 mgr.y (mgr.14544) 224 : cluster [DBG] pgmap v269: 284 pgs: 12 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 48 unknown, 196 active+clean; 44 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:36.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:35.129972+0000 mon.b (mon.1) 264 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:36.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:35.325728+0000 mon.a (mon.0) 1414 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:42:36.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:35.329494+0000 mon.a (mon.0) 1415 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-15", "mode": "writeback"}]': finished 2024-02-13T00:42:36.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:35.329658+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e200: 8 total, 8 up, 7 in 2024-02-13T00:42:36.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:35.334401+0000 mon.c (mon.2) 178 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:36.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:35.335816+0000 mon.a (mon.0) 1417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:36.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:35.339950+0000 mon.c (mon.2) 179 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:36.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:35.341897+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:36.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:35.345725+0000 mon.b (mon.1) 265 : audit [INF] from='client.? 172.21.15.162:0/2535615870' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-83985-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:36.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:35.346300+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-83985-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:34.344163+0000 mgr.y (mgr.14544) 224 : cluster [DBG] pgmap v269: 284 pgs: 12 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 48 unknown, 196 active+clean; 44 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:35.129972+0000 mon.b (mon.1) 264 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:35.325728+0000 mon.a (mon.0) 1414 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:42:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:35.329494+0000 mon.a (mon.0) 1415 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-15", "mode": "writeback"}]': finished 2024-02-13T00:42:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:35.329658+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e200: 8 total, 8 up, 7 in 2024-02-13T00:42:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:35.334401+0000 mon.c (mon.2) 178 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:36.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:35.335816+0000 mon.a (mon.0) 1417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:36.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:35.339950+0000 mon.c (mon.2) 179 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:36.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:35.341897+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:36.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:35.345725+0000 mon.b (mon.1) 265 : audit [INF] from='client.? 172.21.15.162:0/2535615870' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-83985-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:36.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:35.346300+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-83985-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:36.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.086+0000 7f1692c15700 10 client.36755.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:36.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.086+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 8 2024-02-13T00:42:36.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.086+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c0fc840 osd=3 3 2024-02-13T00:42:36.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.086+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c0fc840 osd=3 3 2024-02-13T00:42:36.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.086+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c0fc840 osd=3 3 2024-02-13T00:42:36.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.086+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 8 2024-02-13T00:42:36.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.086+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c0fc840 osd=3 3 2024-02-13T00:42:36.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.086+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 3 8 2024-02-13T00:42:36.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.086+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c0fc840 osd=3 4 2024-02-13T00:42:36.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.086+0000 7f1692c15700 10 client.36755.objecter _send_command 8 2024-02-13T00:42:36.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.086+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f168c11b560 con 0x7f168c11c000 2024-02-13T00:42:36.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.087+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f168c11b560 con 0x7f168c11c000 2024-02-13T00:42:36.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.087+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:36.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.087+0000 7f1689ffb700 10 client.36755.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.087+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c0fc840 osd=3 4 2024-02-13T00:42:36.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.087+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 3 8 2024-02-13T00:42:36.092 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:42:36.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1692c15700 10 client.36755.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:36.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 9 2024-02-13T00:42:36.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6800/3344263388 conn(0x7f168c11b6d0 0x7f168c125290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:36.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c11b560 osd=4 3 2024-02-13T00:42:36.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c11b560 osd=4 3 2024-02-13T00:42:36.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c11b560 osd=4 3 2024-02-13T00:42:36.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 9 2024-02-13T00:42:36.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c11b560 osd=4 3 2024-02-13T00:42:36.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 4 9 2024-02-13T00:42:36.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c11b560 osd=4 4 2024-02-13T00:42:36.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1692c15700 10 client.36755.objecter _send_command 9 2024-02-13T00:42:36.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f168c127f10 con 0x7f168c11b6d0 2024-02-13T00:42:36.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f16909b1700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6800/3344263388 conn(0x7f168c11b6d0 0x7f168c125290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:36.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f1689ffb700 10 client.36755.objecter ms_handle_connect 0x7f168c11b6d0 2024-02-13T00:42:36.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.088+0000 7f16909b1700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6800/3344263388 conn(0x7f168c11b6d0 0x7f168c125290 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:36.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.089+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f168c127f10 con 0x7f168c11b6d0 2024-02-13T00:42:36.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.089+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:42:36.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.089+0000 7f1689ffb700 10 client.36755.objecter _finish_command 9 = system:0 2024-02-13T00:42:36.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.089+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c11b560 osd=4 4 2024-02-13T00:42:36.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.089+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 4 9 2024-02-13T00:42:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.104+0000 7f1692c15700 10 client.36755.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.104+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 10 2024-02-13T00:42:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.104+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c11b560 osd=4 3 2024-02-13T00:42:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.104+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c11b560 osd=4 3 2024-02-13T00:42:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.104+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c11b560 osd=4 3 2024-02-13T00:42:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.104+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 10 2024-02-13T00:42:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.104+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c11b560 osd=4 3 2024-02-13T00:42:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.104+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 4 10 2024-02-13T00:42:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.104+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c11b560 osd=4 4 2024-02-13T00:42:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.104+0000 7f1692c15700 10 client.36755.objecter _send_command 10 2024-02-13T00:42:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.104+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f168c127f10 con 0x7f168c11b6d0 2024-02-13T00:42:36.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.105+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f168c127f10 con 0x7f168c11b6d0 2024-02-13T00:42:36.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.105+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:36.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.105+0000 7f1689ffb700 10 client.36755.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.105+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c11b560 osd=4 4 2024-02-13T00:42:36.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.105+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 4 10 2024-02-13T00:42:36.110 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:42:36.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1692c15700 10 client.36755.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 11 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6804/2212296748 conn(0x7f168c125c90 0x7f168c1260d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c127f10 osd=5 3 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c127f10 osd=5 3 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c127f10 osd=5 3 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 11 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c127f10 osd=5 3 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 5 11 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c127f10 osd=5 4 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1692c15700 10 client.36755.objecter _send_command 11 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f168c127d20 con 0x7f168c125c90 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f168bfff700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6804/2212296748 conn(0x7f168c125c90 0x7f168c1260d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f1689ffb700 10 client.36755.objecter ms_handle_connect 0x7f168c125c90 2024-02-13T00:42:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.106+0000 7f168bfff700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6804/2212296748 conn(0x7f168c125c90 0x7f168c1260d0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:36.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.107+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f168c127d20 con 0x7f168c125c90 2024-02-13T00:42:36.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.107+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:42:36.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.107+0000 7f1689ffb700 10 client.36755.objecter _finish_command 11 = system:0 2024-02-13T00:42:36.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.107+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c127f10 osd=5 4 2024-02-13T00:42:36.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.107+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 5 11 2024-02-13T00:42:36.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.122+0000 7f1692c15700 10 client.36755.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:36.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.122+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 12 2024-02-13T00:42:36.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.122+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c127f10 osd=5 3 2024-02-13T00:42:36.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.122+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c127f10 osd=5 3 2024-02-13T00:42:36.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.122+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c127f10 osd=5 3 2024-02-13T00:42:36.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.122+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 12 2024-02-13T00:42:36.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.122+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c127f10 osd=5 3 2024-02-13T00:42:36.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.122+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 5 12 2024-02-13T00:42:36.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.122+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c127f10 osd=5 4 2024-02-13T00:42:36.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.122+0000 7f1692c15700 10 client.36755.objecter _send_command 12 2024-02-13T00:42:36.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.122+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f168c127d20 con 0x7f168c125c90 2024-02-13T00:42:36.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f168c127d20 con 0x7f168c125c90 2024-02-13T00:42:36.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:36.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1689ffb700 10 client.36755.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c127f10 osd=5 4 2024-02-13T00:42:36.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 5 12 2024-02-13T00:42:36.129 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:42:36.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1692c15700 10 client.36755.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:36.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 13 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6808/1506640341 conn(0x7f168c12a680 0x7f168c12aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c127d20 osd=6 3 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c127d20 osd=6 3 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c127d20 osd=6 3 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 13 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c127d20 osd=6 3 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 6 13 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c127d20 osd=6 4 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1692c15700 10 client.36755.objecter _send_command 13 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.124+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f168c12e500 con 0x7f168c12a680 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.125+0000 7f16911b2700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6808/1506640341 conn(0x7f168c12a680 0x7f168c12aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.125+0000 7f1689ffb700 10 client.36755.objecter ms_handle_connect 0x7f168c12a680 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.125+0000 7f16911b2700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6808/1506640341 conn(0x7f168c12a680 0x7f168c12aae0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:36.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.126+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f168c12e500 con 0x7f168c12a680 2024-02-13T00:42:36.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.126+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:42:36.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.126+0000 7f1689ffb700 10 client.36755.objecter _finish_command 13 = system:0 2024-02-13T00:42:36.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.126+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c127d20 osd=6 4 2024-02-13T00:42:36.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.126+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 6 13 2024-02-13T00:42:36.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.140+0000 7f1692c15700 10 client.36755.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:36.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.140+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 14 2024-02-13T00:42:36.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.140+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c127d20 osd=6 3 2024-02-13T00:42:36.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.140+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c127d20 osd=6 3 2024-02-13T00:42:36.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.140+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c127d20 osd=6 3 2024-02-13T00:42:36.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.140+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 14 2024-02-13T00:42:36.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.140+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c127d20 osd=6 3 2024-02-13T00:42:36.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.140+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 6 14 2024-02-13T00:42:36.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.140+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c127d20 osd=6 4 2024-02-13T00:42:36.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.140+0000 7f1692c15700 10 client.36755.objecter _send_command 14 2024-02-13T00:42:36.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.140+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f168c126680 con 0x7f168c12a680 2024-02-13T00:42:36.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f168c126680 con 0x7f168c12a680 2024-02-13T00:42:36.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:36.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1689ffb700 10 client.36755.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c127d20 osd=6 4 2024-02-13T00:42:36.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 6 14 2024-02-13T00:42:36.146 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:42:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1692c15700 10 client.36755.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 15 2024-02-13T00:42:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6812/1964192772 conn(0x7f168c126d60 0x7f168c1271c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c126680 osd=7 3 2024-02-13T00:42:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c126680 osd=7 3 2024-02-13T00:42:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c126680 osd=7 3 2024-02-13T00:42:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 15 2024-02-13T00:42:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c126680 osd=7 3 2024-02-13T00:42:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 7 15 2024-02-13T00:42:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c126680 osd=7 4 2024-02-13T00:42:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1692c15700 10 client.36755.objecter _send_command 15 2024-02-13T00:42:36.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f168c127770 con 0x7f168c126d60 2024-02-13T00:42:36.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f16909b1700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6812/1964192772 conn(0x7f168c126d60 0x7f168c1271c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:36.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f1689ffb700 10 client.36755.objecter ms_handle_connect 0x7f168c126d60 2024-02-13T00:42:36.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.142+0000 7f16909b1700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6812/1964192772 conn(0x7f168c126d60 0x7f168c1271c0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:36.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.143+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f168c127770 con 0x7f168c126d60 2024-02-13T00:42:36.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.143+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:42:36.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.143+0000 7f1689ffb700 10 client.36755.objecter _finish_command 15 = system:0 2024-02-13T00:42:36.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.143+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c126680 osd=7 4 2024-02-13T00:42:36.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.143+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 7 15 2024-02-13T00:42:36.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.157+0000 7f1692c15700 10 client.36755.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:36.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.157+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign -1 16 2024-02-13T00:42:36.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.157+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c126680 osd=7 3 2024-02-13T00:42:36.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.157+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c126680 osd=7 3 2024-02-13T00:42:36.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.157+0000 7f1692c15700 20 client.36755.objecter _get_session s=0x7f168c126680 osd=7 3 2024-02-13T00:42:36.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.157+0000 7f1692c15700 15 client.36755.objecter _session_command_op_remove -1 16 2024-02-13T00:42:36.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.157+0000 7f1692c15700 20 client.36755.objecter get_session s=0x7f168c126680 osd=7 3 2024-02-13T00:42:36.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.157+0000 7f1692c15700 15 client.36755.objecter _session_command_op_assign 7 16 2024-02-13T00:42:36.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.157+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c126680 osd=7 4 2024-02-13T00:42:36.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.157+0000 7f1692c15700 10 client.36755.objecter _send_command 16 2024-02-13T00:42:36.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.157+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f168c127770 con 0x7f168c126d60 2024-02-13T00:42:36.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.159+0000 7f1689ffb700 1 -- 172.21.15.162:0/3741717118 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f168c127770 con 0x7f168c126d60 2024-02-13T00:42:36.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.159+0000 7f1689ffb700 10 client.36755.objecter ms_dispatch 0x7f168c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:36.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.159+0000 7f1689ffb700 10 client.36755.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.159+0000 7f1689ffb700 20 client.36755.objecter put_session s=0x7f168c126680 osd=7 4 2024-02-13T00:42:36.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.159+0000 7f1689ffb700 15 client.36755.objecter _session_command_op_remove 7 16 2024-02-13T00:42:36.164 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:42:36.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.159+0000 7f1692c15700 10 librados: watch_flush enter 2024-02-13T00:42:36.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.159+0000 7f1692c15700 10 librados: watch_flush exit 2024-02-13T00:42:36.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.159+0000 7f1692c15700 10 client.36755.objecter close_session for osd.0 2024-02-13T00:42:36.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.159+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6801/2103743338 conn(0x7f168c028590 msgr2=0x7f168c0fbd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6801/2103743338 conn(0x7f168c028590 0x7f168c0fbd00 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c109750 osd=0 1 2024-02-13T00:42:36.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 10 client.36755.objecter close_session for osd.1 2024-02-13T00:42:36.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6805/150599780 conn(0x7f168c027be0 msgr2=0x7f168c028000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6805/150599780 conn(0x7f168c027be0 0x7f168c028000 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c0fd370 osd=1 1 2024-02-13T00:42:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 10 client.36755.objecter close_session for osd.2 2024-02-13T00:42:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6809/3689719376 conn(0x7f168c0fced0 msgr2=0x7f168c11e4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6809/3689719376 conn(0x7f168c0fced0 0x7f168c11e4b0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c027a60 osd=2 1 2024-02-13T00:42:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 10 client.36755.objecter close_session for osd.3 2024-02-13T00:42:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6813/604971011 conn(0x7f168c11c000 msgr2=0x7f168c11fb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6813/604971011 conn(0x7f168c11c000 0x7f168c11fb20 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c0fc840 osd=3 1 2024-02-13T00:42:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 10 client.36755.objecter close_session for osd.4 2024-02-13T00:42:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6800/3344263388 conn(0x7f168c11b6d0 msgr2=0x7f168c125290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.166 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6800/3344263388 conn(0x7f168c11b6d0 0x7f168c125290 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c11b560 osd=4 1 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 10 client.36755.objecter close_session for osd.5 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6804/2212296748 conn(0x7f168c125c90 msgr2=0x7f168c1260d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6804/2212296748 conn(0x7f168c125c90 0x7f168c1260d0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f16911b2700 1 -- 172.21.15.162:0/3741717118 reap_dead start 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c127f10 osd=5 1 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 10 client.36755.objecter close_session for osd.6 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6808/1506640341 conn(0x7f168c12a680 msgr2=0x7f168c12aae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6808/1506640341 conn(0x7f168c12a680 0x7f168c12aae0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f16911b2700 1 -- 172.21.15.162:0/3741717118 reap_dead start 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c127d20 osd=6 1 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 10 client.36755.objecter close_session for osd.7 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6812/1964192772 conn(0x7f168c126d60 msgr2=0x7f168c1271c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.181:6812/1964192772 conn(0x7f168c126d60 0x7f168c1271c0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 20 client.36755.objecter put_session s=0x7f168c126680 osd=7 1 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 20 client.36755.objecter shutdown clearing up homeless session... 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 10 client.36755.objecter successfully canceled tick 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f16911b2700 1 -- 172.21.15.162:0/3741717118 reap_dead start 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f16911b2700 1 -- 172.21.15.162:0/3741717118 reap_dead start 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6800/2866534053 conn(0x7f167c062c90 msgr2=0x7f167c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6800/2866534053 conn(0x7f167c062c90 0x7f167c065150 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 rx=0x7f168c109a10 tx=0x7f167800b040).stop 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f168c0f8d10 msgr2=0x7f168c10e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f168c0f8d10 0x7f168c10e0d0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7f1680009400 tx=0x7f1680009cb0).stop 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 shutdown_connections 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f168c0f8d10 0x7f168c10e0d0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 --2- 172.21.15.162:0/3741717118 >> v2:172.21.15.162:6800/2866534053 conn(0x7f167c062c90 0x7f167c065150 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 >> 172.21.15.162:0/3741717118 conn(0x7f168c06eb90 msgr2=0x7f168c102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 shutdown_connections 2024-02-13T00:42:36.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 -- 172.21.15.162:0/3741717118 wait complete. 2024-02-13T00:42:36.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:36.160+0000 7f1692c15700 1 librados: shutdown 2024-02-13T00:42:36.169 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.169 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.169 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.169 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.169 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.169 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.169 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.169 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:42:36.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:34.344163+0000 mgr.y (mgr.14544) 224 : cluster [DBG] pgmap v269: 284 pgs: 12 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 48 unknown, 196 active+clean; 44 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:36.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:35.129972+0000 mon.b (mon.1) 264 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:36.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:35.325728+0000 mon.a (mon.0) 1414 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:42:36.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:35.329494+0000 mon.a (mon.0) 1415 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-15", "mode": "writeback"}]': finished 2024-02-13T00:42:36.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:35.329658+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e200: 8 total, 8 up, 7 in 2024-02-13T00:42:36.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:35.334401+0000 mon.c (mon.2) 178 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:36.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:35.335816+0000 mon.a (mon.0) 1417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:36.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:35.339950+0000 mon.c (mon.2) 179 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:36.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:35.341897+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:36.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:35.345725+0000 mon.b (mon.1) 265 : audit [INF] from='client.? 172.21.15.162:0/2535615870' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-83985-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:36.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:35.346300+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-83985-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:36.464 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:42:36] "GET /metrics HTTP/1.1" 200 194964 "" "Prometheus/2.33.4" 2024-02-13T00:42:37.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:35.416374+0000 mgr.y (mgr.14544) 225 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:37.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.020460+0000 mon.c (mon.2) 180 : audit [DBG] from='client.? 172.21.15.162:0/3741717118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:37.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.131044+0000 mon.b (mon.1) 266 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:37.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.333973+0000 mon.a (mon.0) 1420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]': finished 2024-02-13T00:42:37.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.334071+0000 mon.a (mon.0) 1421 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]': finished 2024-02-13T00:42:37.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.334170+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-83985-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:37.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:36.334300+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e201: 8 total, 8 up, 7 in 2024-02-13T00:42:37.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.335865+0000 mon.c (mon.2) 181 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:37.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.336270+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:37.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.337920+0000 mon.c (mon.2) 182 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:37.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.340390+0000 mon.c (mon.2) 183 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:37.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.340876+0000 mon.a (mon.0) 1425 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:37.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.341146+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:37.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.373394+0000 mon.b (mon.1) 267 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:37.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:36.373834+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:37.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:35.416374+0000 mgr.y (mgr.14544) 225 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:37.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.020460+0000 mon.c (mon.2) 180 : audit [DBG] from='client.? 172.21.15.162:0/3741717118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:37.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.131044+0000 mon.b (mon.1) 266 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:37.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.333973+0000 mon.a (mon.0) 1420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]': finished 2024-02-13T00:42:37.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.334071+0000 mon.a (mon.0) 1421 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]': finished 2024-02-13T00:42:37.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.334170+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-83985-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:37.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:36.334300+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e201: 8 total, 8 up, 7 in 2024-02-13T00:42:37.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.335865+0000 mon.c (mon.2) 181 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:37.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.336270+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:37.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.337920+0000 mon.c (mon.2) 182 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:37.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.340390+0000 mon.c (mon.2) 183 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:37.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.340876+0000 mon.a (mon.0) 1425 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:37.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.341146+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:37.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.373394+0000 mon.b (mon.1) 267 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:37.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:36.373834+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:37.203 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:35.416374+0000 mgr.y (mgr.14544) 225 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.020460+0000 mon.c (mon.2) 180 : audit [DBG] from='client.? 172.21.15.162:0/3741717118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.131044+0000 mon.b (mon.1) 266 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.333973+0000 mon.a (mon.0) 1420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]': finished 2024-02-13T00:42:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.334071+0000 mon.a (mon.0) 1421 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-84049-36"}]': finished 2024-02-13T00:42:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.334170+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-83985-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:36.334300+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e201: 8 total, 8 up, 7 in 2024-02-13T00:42:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.335865+0000 mon.c (mon.2) 181 : audit [INF] from='client.? 172.21.15.162:0/371088046' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:37.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.336270+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]: dispatch 2024-02-13T00:42:37.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.337920+0000 mon.c (mon.2) 182 : audit [INF] from='client.? 172.21.15.162:0/695542472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:37.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.340390+0000 mon.c (mon.2) 183 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:37.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.340876+0000 mon.a (mon.0) 1425 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]: dispatch 2024-02-13T00:42:37.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.341146+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:37.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.373394+0000 mon.b (mon.1) 267 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:37.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:36.373834+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:37.355 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComple: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-02-13T00:42:37.355 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-02-13T00:42:37.355 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4035 ms) 2024-02-13T00:42:37.355 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-02-13T00:42:37.356 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3965 ms) 2024-02-13T00:42:37.356 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-02-13T00:42:37.356 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4025 ms) 2024-02-13T00:42:37.356 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12025 ms total) 2024-02-13T00:42:37.356 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: 2024-02-13T00:42:37.356 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-02-13T00:42:37.356 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [==========] 20 tests from 5 test suites ran. (102660 ms total) 2024-02-13T00:42:37.356 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-02-13T00:42:37.357 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:42:37.357 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84036 2024-02-13T00:42:37.357 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84036 2024-02-13T00:42:37.357 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:42:37.357 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84028 2024-02-13T00:42:37.357 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84028 2024-02-13T00:42:37.357 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:42:37.357 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84012 2024-02-13T00:42:37.357 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84012 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout:P (68813 ms total) 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (119 ms) 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (119 ms total) 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (37 ms) 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (2 ms) 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (39 ms total) 2024-02-13T00:42:37.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (3 ms) 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (4 ms) 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (7 ms total) 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (34 ms) 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (17 ms) 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (51 ms total) 2024-02-13T00:42:37.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-02-13T00:42:37.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-02-13T00:42:37.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-02-13T00:42:37.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1024 ms) 2024-02-13T00:42:37.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1024 ms total) 2024-02-13T00:42:37.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-02-13T00:42:37.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-02-13T00:42:37.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (102635 ms total) 2024-02-13T00:42:37.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-02-13T00:42:37.449 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:37.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:37.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:37.591 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:37.613 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:37.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:37.718 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:37.747 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:37.766 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:37.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:37.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:37.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:37.899 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:36.344926+0000 mgr.y (mgr.14544) 226 : cluster [DBG] pgmap v272: 292 pgs: 12 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 32 unknown, 228 active+clean; 12 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:37.132165+0000 mon.b (mon.1) 268 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:37.331000+0000 mon.a (mon.0) 1428 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:42:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:37.344196+0000 mon.a (mon.0) 1429 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]': finished 2024-02-13T00:42:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:37.344257+0000 mon.a (mon.0) 1430 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]': finished 2024-02-13T00:42:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:37.344303+0000 mon.a (mon.0) 1431 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]': finished 2024-02-13T00:42:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:37.344372+0000 mon.a (mon.0) 1432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:42:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:37.344416+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e202: 8 total, 8 up, 7 in 2024-02-13T00:42:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:37.345922+0000 mon.c (mon.2) 184 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:37.346267+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:37.350782+0000 mon.b (mon.1) 269 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15"}]: dispatch 2024-02-13T00:42:38.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:37.351518+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15"}]: dispatch 2024-02-13T00:42:38.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:36.344926+0000 mgr.y (mgr.14544) 226 : cluster [DBG] pgmap v272: 292 pgs: 12 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 32 unknown, 228 active+clean; 12 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:38.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:37.132165+0000 mon.b (mon.1) 268 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:38.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:37.331000+0000 mon.a (mon.0) 1428 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:42:38.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:37.344196+0000 mon.a (mon.0) 1429 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]': finished 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:37.344257+0000 mon.a (mon.0) 1430 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]': finished 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:37.344303+0000 mon.a (mon.0) 1431 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]': finished 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:37.344372+0000 mon.a (mon.0) 1432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:37.344416+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e202: 8 total, 8 up, 7 in 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:37.345922+0000 mon.c (mon.2) 184 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:37.346267+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:37.350782+0000 mon.b (mon.1) 269 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15"}]: dispatch 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:37.351518+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15"}]: dispatch 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:36.344926+0000 mgr.y (mgr.14544) 226 : cluster [DBG] pgmap v272: 292 pgs: 12 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 32 unknown, 228 active+clean; 12 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:37.132165+0000 mon.b (mon.1) 268 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:37.331000+0000 mon.a (mon.0) 1428 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:37.344196+0000 mon.a (mon.0) 1429 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-84081-20"}]': finished 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:37.344257+0000 mon.a (mon.0) 1430 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-84049-36"}]': finished 2024-02-13T00:42:38.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:37.344303+0000 mon.a (mon.0) 1431 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-83977-28"}]': finished 2024-02-13T00:42:38.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:37.344372+0000 mon.a (mon.0) 1432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:42:38.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:37.344416+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e202: 8 total, 8 up, 7 in 2024-02-13T00:42:38.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:37.345922+0000 mon.c (mon.2) 184 : audit [INF] from='client.? 172.21.15.162:0/3880682194' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:38.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:37.346267+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]: dispatch 2024-02-13T00:42:38.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:37.350782+0000 mon.b (mon.1) 269 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15"}]: dispatch 2024-02-13T00:42:38.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:37.351518+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15"}]: dispatch 2024-02-13T00:42:38.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:42:38.967 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:42:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:42:39.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:42:39.219 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:38.134090+0000 mon.b (mon.1) 270 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:39.219 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:38.340260+0000 mon.a (mon.0) 1436 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:42:39.219 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:38.346028+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]': finished 2024-02-13T00:42:39.219 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:38.346274+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15"}]': finished 2024-02-13T00:42:39.219 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:38.346387+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e203: 8 total, 8 up, 7 in 2024-02-13T00:42:39.219 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:38.351218+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? 172.21.15.162:0/2667602579' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-83985-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:39.219 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:38.375548+0000 mon.c (mon.2) 185 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:39.219 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:38.375915+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:39.219 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:38.376427+0000 mon.c (mon.2) 186 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:39.219 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:38.376715+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:39.219 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:38.377168+0000 mon.c (mon.2) 187 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-83977-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:39.219 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:38.377424+0000 mon.a (mon.0) 1443 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-83977-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:39.220 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:38.134090+0000 mon.b (mon.1) 270 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:39.220 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:38.340260+0000 mon.a (mon.0) 1436 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:42:39.220 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:38.346028+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]': finished 2024-02-13T00:42:39.220 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:38.346274+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15"}]': finished 2024-02-13T00:42:39.220 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:38.346387+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e203: 8 total, 8 up, 7 in 2024-02-13T00:42:39.220 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:38.351218+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? 172.21.15.162:0/2667602579' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-83985-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:39.220 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:38.375548+0000 mon.c (mon.2) 185 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:39.220 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:38.375915+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:39.220 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:38.376427+0000 mon.c (mon.2) 186 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:39.220 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:38.376715+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:39.220 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:38.377168+0000 mon.c (mon.2) 187 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-83977-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:39.220 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:38.377424+0000 mon.a (mon.0) 1443 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-83977-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:38.134090+0000 mon.b (mon.1) 270 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:38.340260+0000 mon.a (mon.0) 1436 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:42:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:38.346028+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-83977-28"}]': finished 2024-02-13T00:42:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:38.346274+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-15"}]': finished 2024-02-13T00:42:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:38.346387+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e203: 8 total, 8 up, 7 in 2024-02-13T00:42:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:38.351218+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? 172.21.15.162:0/2667602579' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-83985-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:38.375548+0000 mon.c (mon.2) 185 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:38.375915+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:38.376427+0000 mon.c (mon.2) 186 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:38.376715+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:38.377168+0000 mon.c (mon.2) 187 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-83977-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:38.377424+0000 mon.a (mon.0) 1443 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-83977-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:38.345913+0000 mgr.y (mgr.14544) 227 : cluster [DBG] pgmap v275: 292 pgs: 12 active+clean+snaptrim, 32 unknown, 20 active+clean+snaptrim_wait, 228 active+clean; 12 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:39.135353+0000 mon.b (mon.1) 271 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:39.351718+0000 mon.a (mon.0) 1444 : audit [INF] from='client.? 172.21.15.162:0/2667602579' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-83985-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:39.351852+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-83977-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:39.351926+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e204: 8 total, 8 up, 7 in 2024-02-13T00:42:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:39.362261+0000 mon.c (mon.2) 188 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-83977-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:39.362852+0000 mon.a (mon.0) 1447 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-83977-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:40.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:38.345913+0000 mgr.y (mgr.14544) 227 : cluster [DBG] pgmap v275: 292 pgs: 12 active+clean+snaptrim, 32 unknown, 20 active+clean+snaptrim_wait, 228 active+clean; 12 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:40.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:39.135353+0000 mon.b (mon.1) 271 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:40.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:39.351718+0000 mon.a (mon.0) 1444 : audit [INF] from='client.? 172.21.15.162:0/2667602579' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-83985-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:40.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:39.351852+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-83977-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:40.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:39.351926+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e204: 8 total, 8 up, 7 in 2024-02-13T00:42:40.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:39.362261+0000 mon.c (mon.2) 188 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-83977-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:40.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:39.362852+0000 mon.a (mon.0) 1447 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-83977-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:40.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:38.345913+0000 mgr.y (mgr.14544) 227 : cluster [DBG] pgmap v275: 292 pgs: 12 active+clean+snaptrim, 32 unknown, 20 active+clean+snaptrim_wait, 228 active+clean; 12 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:40.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:39.135353+0000 mon.b (mon.1) 271 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:40.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:39.351718+0000 mon.a (mon.0) 1444 : audit [INF] from='client.? 172.21.15.162:0/2667602579' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-83985-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:40.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:39.351852+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-83977-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:40.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:39.351926+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e204: 8 total, 8 up, 7 in 2024-02-13T00:42:40.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:39.362261+0000 mon.c (mon.2) 188 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-83977-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:40.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:39.362852+0000 mon.a (mon.0) 1447 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-83977-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:40.568 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-02-13T00:42:40.568 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [==========] Running 56 tests from 4 test suites. 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 31 tests from LibRadosAio 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (8097 ms) 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (18972 ms) 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6443 ms) 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3096 ms) 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3095 ms) 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-02-13T00:42:40.569 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3143 ms) 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2795 ms) 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3287 ms) 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2953 ms) 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3012 ms) 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2731 ms) 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3023 ms) 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3009 ms) 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-02-13T00:42:40.570 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2986 ms) 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3053 ms) 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3037 ms) 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2894 ms) 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3273 ms) 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2786 ms) 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3221 ms) 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2726 ms) 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3020 ms) 2024-02-13T00:42:40.571 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-02-13T00:42:40.572 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3014 ms) 2024-02-13T00:42:40.572 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-02-13T00:42:40.572 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2989 ms) 2024-02-13T00:42:40.572 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-02-13T00:42:40.572 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2996 ms) 2024-02-13T00:42:40.572 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-02-13T00:42:40.572 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3011 ms) 2024-02-13T00:42:40.572 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-02-13T00:42:41.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:40.136594+0000 mon.b (mon.1) 272 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:41.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:40.561684+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e205: 8 total, 8 up, 7 in 2024-02-13T00:42:41.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:40.564900+0000 mon.b (mon.1) 273 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:41.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:40.569199+0000 mon.a (mon.0) 1449 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:40.136594+0000 mon.b (mon.1) 272 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:40.561684+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e205: 8 total, 8 up, 7 in 2024-02-13T00:42:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:40.564900+0000 mon.b (mon.1) 273 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:40.569199+0000 mon.a (mon.0) 1449 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:40.136594+0000 mon.b (mon.1) 272 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:40.561684+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e205: 8 total, 8 up, 7 in 2024-02-13T00:42:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:40.564900+0000 mon.b (mon.1) 273 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:40.569199+0000 mon.a (mon.0) 1449 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:41.453 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:41.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:41.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:41.591 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:41.611 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:41.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:41.682 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:41.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:42:41.748 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:41.863 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:42:41.863 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:42:41.863 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-13T00:42:41.863 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.000000 2024-02-13T00:42:41.863 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 7 0 2024-02-13T00:42:41.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:40.346371+0000 mgr.y (mgr.14544) 228 : cluster [DBG] pgmap v278: 260 pgs: 11 active+clean+snaptrim, 32 unknown, 19 active+clean+snaptrim_wait, 198 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:42:41.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:41.137828+0000 mon.b (mon.1) 274 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:41.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:41.334052+0000 mon.a (mon.0) 1450 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-83977-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-83977-29"}]': finished 2024-02-13T00:42:41.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:41.334264+0000 mon.a (mon.0) 1451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:41.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:41.334357+0000 mon.a (mon.0) 1452 : cluster [DBG] osdmap e206: 8 total, 8 up, 7 in 2024-02-13T00:42:41.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:41.350160+0000 mon.b (mon.1) 275 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:41.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:41.350879+0000 mon.c (mon.2) 189 : audit [INF] from='client.? 172.21.15.162:0/3728040861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-83985-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:41.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:41.351436+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:41.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:41.352244+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-83985-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:41.996 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:42:42.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:42.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:42:42.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:42.134 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:42.156 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:42.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:42.260 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:42.287 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:40.346371+0000 mgr.y (mgr.14544) 228 : cluster [DBG] pgmap v278: 260 pgs: 11 active+clean+snaptrim, 32 unknown, 19 active+clean+snaptrim_wait, 198 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:42:42.287 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:41.137828+0000 mon.b (mon.1) 274 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:42.287 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:41.334052+0000 mon.a (mon.0) 1450 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-83977-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-83977-29"}]': finished 2024-02-13T00:42:42.287 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:41.334264+0000 mon.a (mon.0) 1451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:42.287 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:41.334357+0000 mon.a (mon.0) 1452 : cluster [DBG] osdmap e206: 8 total, 8 up, 7 in 2024-02-13T00:42:42.287 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:41.350160+0000 mon.b (mon.1) 275 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:42.287 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:41.350879+0000 mon.c (mon.2) 189 : audit [INF] from='client.? 172.21.15.162:0/3728040861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-83985-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:42.287 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:41.351436+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:42.287 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:41.352244+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-83985-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:42.289 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:42.290 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:42.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:40.346371+0000 mgr.y (mgr.14544) 228 : cluster [DBG] pgmap v278: 260 pgs: 11 active+clean+snaptrim, 32 unknown, 19 active+clean+snaptrim_wait, 198 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:42:42.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:41.137828+0000 mon.b (mon.1) 274 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:41.334052+0000 mon.a (mon.0) 1450 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-83977-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-83977-29"}]': finished 2024-02-13T00:42:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:41.334264+0000 mon.a (mon.0) 1451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:41.334357+0000 mon.a (mon.0) 1452 : cluster [DBG] osdmap e206: 8 total, 8 up, 7 in 2024-02-13T00:42:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:41.350160+0000 mon.b (mon.1) 275 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:41.350879+0000 mon.c (mon.2) 189 : audit [INF] from='client.? 172.21.15.162:0/3728040861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-83985-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:41.351436+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:41.352244+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-83985-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:42.309 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:42:42.309 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:42:42.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:42.309 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36767.0:1 140.6s0 140.7fc1f406 (undecoded) ondisk+write+known_if_redirected e206)", 2024-02-13T00:42:42.309 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:41.361654+0000", 2024-02-13T00:42:42.309 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.94509298799999997, 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94513575599999999, 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for peered", 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36767", 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1509178592", 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361654+0000", 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:42.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:42.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:42.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361654+0000", 2024-02-13T00:42:42.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:42:42.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:42.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:42.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:42.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361653+0000", 2024-02-13T00:42:42.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2099999999999998e-06 2024-02-13T00:42:42.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:42.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:42.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:42.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361659+0000", 2024-02-13T00:42:42.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1799999999999996e-06 2024-02-13T00:42:42.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:42.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:42.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:42.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361667+0000", 2024-02-13T00:42:42.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8100000000000003e-06 2024-02-13T00:42:42.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:42.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:42.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:42.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361673+0000", 2024-02-13T00:42:42.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0318000000000002e-05 2024-02-13T00:42:42.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:42.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:42.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:42.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361723+0000", 2024-02-13T00:42:42.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5279999999999999e-06 2024-02-13T00:42:42.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:42.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:42.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:42:42.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361727+0000", 2024-02-13T00:42:42.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2640999999999998e-05 2024-02-13T00:42:42.313 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:42.314 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:42.314 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:42.314 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:42.314 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:42:42.314 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-02-13T00:42:42.314 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:42:42.741 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:42.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:42.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:42.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:42.872 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:42.996 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:42.998 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:42.139134+0000 mon.b (mon.1) 276 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:42.998 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:42.333680+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:42:42.998 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:42.333782+0000 mon.a (mon.0) 1456 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-83985-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:42.998 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:42.333861+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e207: 8 total, 8 up, 7 in 2024-02-13T00:42:42.998 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:42.335429+0000 mon.b (mon.1) 277 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-17"}]: dispatch 2024-02-13T00:42:42.998 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:42.339575+0000 mon.a (mon.0) 1458 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-17"}]: dispatch 2024-02-13T00:42:42.998 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:42.139134+0000 mon.b (mon.1) 276 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:42.999 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:42.333680+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:42:42.999 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:42.333782+0000 mon.a (mon.0) 1456 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-83985-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:42.999 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:42.333861+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e207: 8 total, 8 up, 7 in 2024-02-13T00:42:42.999 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:42.335429+0000 mon.b (mon.1) 277 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-17"}]: dispatch 2024-02-13T00:42:42.999 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:42.339575+0000 mon.a (mon.0) 1458 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-17"}]: dispatch 2024-02-13T00:42:42.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:43.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:43.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:43.128 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:42.139134+0000 mon.b (mon.1) 276 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:42.333680+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:42:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:42.333782+0000 mon.a (mon.0) 1456 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-83985-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:42.333861+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e207: 8 total, 8 up, 7 in 2024-02-13T00:42:43.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:42.335429+0000 mon.b (mon.1) 277 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-17"}]: dispatch 2024-02-13T00:42:43.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:42.339575+0000 mon.a (mon.0) 1458 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-17"}]: dispatch 2024-02-13T00:42:43.576 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:43.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:43.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:43.706 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:43.953 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:43.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:43.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-13T00:42:44.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:44.090 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:44.111 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:44.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:44.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:44.242 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:42.346925+0000 mgr.y (mgr.14544) 229 : cluster [DBG] pgmap v281: 300 pgs: 11 active+clean+snaptrim, 72 unknown, 19 active+clean+snaptrim_wait, 198 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:42:44.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:43.140730+0000 mon.b (mon.1) 278 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:44.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:43.342358+0000 mon.a (mon.0) 1459 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-17"}]': finished 2024-02-13T00:42:44.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:43.342477+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e208: 8 total, 8 up, 7 in 2024-02-13T00:42:44.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:43.348784+0000 mon.b (mon.1) 279 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-17", "mode": "writeback"}]: dispatch 2024-02-13T00:42:44.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:43.351867+0000 mon.c (mon.2) 190 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:44.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:43.352535+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-17", "mode": "writeback"}]: dispatch 2024-02-13T00:42:44.243 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:43.352884+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:44.245 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:44.264 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:44.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:44.304 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:42.346925+0000 mgr.y (mgr.14544) 229 : cluster [DBG] pgmap v281: 300 pgs: 11 active+clean+snaptrim, 72 unknown, 19 active+clean+snaptrim_wait, 198 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:42:44.304 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:43.140730+0000 mon.b (mon.1) 278 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:44.304 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:43.342358+0000 mon.a (mon.0) 1459 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-17"}]': finished 2024-02-13T00:42:44.304 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:43.342477+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e208: 8 total, 8 up, 7 in 2024-02-13T00:42:44.304 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:43.348784+0000 mon.b (mon.1) 279 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-17", "mode": "writeback"}]: dispatch 2024-02-13T00:42:44.304 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:43.351867+0000 mon.c (mon.2) 190 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:44.305 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:43.352535+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-17", "mode": "writeback"}]: dispatch 2024-02-13T00:42:44.305 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:43.352884+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:44.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:42.346925+0000 mgr.y (mgr.14544) 229 : cluster [DBG] pgmap v281: 300 pgs: 11 active+clean+snaptrim, 72 unknown, 19 active+clean+snaptrim_wait, 198 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:42:44.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:43.140730+0000 mon.b (mon.1) 278 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:44.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:43.342358+0000 mon.a (mon.0) 1459 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-17"}]': finished 2024-02-13T00:42:44.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:43.342477+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e208: 8 total, 8 up, 7 in 2024-02-13T00:42:44.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:43.348784+0000 mon.b (mon.1) 279 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-17", "mode": "writeback"}]: dispatch 2024-02-13T00:42:44.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:43.351867+0000 mon.c (mon.2) 190 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:44.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:43.352535+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-17", "mode": "writeback"}]: dispatch 2024-02-13T00:42:44.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:43.352884+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:44.368 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:44.397 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:44.951 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:44.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:44.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:45.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:42:45.084 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:45.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.187+0000 7f915782c700 1 -- 172.21.15.162:0/1838367628 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f91500f95e0 msgr2=0x7f9150101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.187+0000 7f915782c700 1 --2- 172.21.15.162:0/1838367628 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f91500f95e0 0x7f9150101d80 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7f914c004870 tx=0x7f914c009cb0).stop 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.187+0000 7f915782c700 1 -- 172.21.15.162:0/1838367628 shutdown_connections 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.187+0000 7f915782c700 1 --2- 172.21.15.162:0/1838367628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9150102270 0x7f9150104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.187+0000 7f915782c700 1 --2- 172.21.15.162:0/1838367628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91500f8d10 0x7f91500f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.187+0000 7f915782c700 1 --2- 172.21.15.162:0/1838367628 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f91500f95e0 0x7f9150101d80 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.187+0000 7f915782c700 1 -- 172.21.15.162:0/1838367628 >> 172.21.15.162:0/1838367628 conn(0x7f915006eb90 msgr2=0x7f91500f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.187+0000 7f915782c700 1 -- 172.21.15.162:0/1838367628 shutdown_connections 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.187+0000 7f915782c700 1 -- 172.21.15.162:0/1838367628 wait complete. 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f915782c700 1 librados: starting msgr at 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f915782c700 1 librados: starting objecter 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f915782c700 1 Processor -- start 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f915782c700 1 -- start start 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f915782c700 1 librados: setting wanted keys 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f915782c700 1 librados: calling monclient init 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f915782c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91500f8d10 0x7f91500f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:45.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f915782c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91500f95e0 0x7f91500f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f915782c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9150102270 0x7f91500f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f915782c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91501166d0 con 0x7f91500f8d10 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f915782c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9150116840 con 0x7f91500f95e0 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f915782c700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9150116a00 con 0x7f9150102270 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f91555c8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91500f8d10 0x7f91500f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f9155dc9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9150102270 0x7f91500f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f91555c8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91500f8d10 0x7f91500f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:47278/0 (socket says 172.21.15.162:47278) 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f9155dc9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9150102270 0x7f91500f76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:33596/0 (socket says 172.21.15.162:33596) 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f91555c8700 1 -- 172.21.15.162:0/717747144 learned_addr learned my addr 172.21.15.162:0/717747144 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f9154dc7700 1 --2- 172.21.15.162:0/717747144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91500f95e0 0x7f91500f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f91555c8700 1 -- 172.21.15.162:0/717747144 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9150102270 msgr2=0x7f91500f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f91555c8700 1 --2- 172.21.15.162:0/717747144 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9150102270 0x7f91500f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f91555c8700 1 -- 172.21.15.162:0/717747144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91500f95e0 msgr2=0x7f91500f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f91555c8700 1 --2- 172.21.15.162:0/717747144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91500f95e0 0x7f91500f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f91555c8700 1 -- 172.21.15.162:0/717747144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9140005040 con 0x7f91500f8d10 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f9155dc9700 1 --2- 172.21.15.162:0/717747144 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9150102270 0x7f91500f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f91555c8700 1 --2- 172.21.15.162:0/717747144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91500f8d10 0x7f91500f6cc0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 rx=0x7f91400048f0 tx=0x7f9140009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f91467fc700 10 client.?.objecter ms_handle_connect 0x7f91500f8d10 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.188+0000 7f91467fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.189+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9140007b00 con 0x7f91500f8d10 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.189+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c005040 con 0x7f91500f8d10 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.189+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f914000f040 con 0x7f91500f8d10 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.189+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9140014990 con 0x7f91500f8d10 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.189+0000 7f915782c700 10 client.26682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.189+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9150117090 con 0x7f91500f8d10 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.189+0000 7f915782c700 1 librados: init done 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.189+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9150107170 con 0x7f91500f8d10 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.190+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f9140010500 con 0x7f91500f8d10 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.190+0000 7f91467fc700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6800/2866534053 conn(0x7f913c062c90 0x7f913c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:45.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(209..209 src has 52..209) v4 ==== 8810+0+0 (secure 0 0 0) 0x7f914007d2e0 con 0x7f91500f8d10 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 osd_map(209..209 src has 52..209) v4 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f91467fc700 3 client.26682.objecter handle_osd_map got epochs [209,209] > 0 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f91467fc700 3 client.26682.objecter handle_osd_map decoding full epoch 209 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f9154dc7700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6800/2866534053 conn(0x7f913c062c90 0x7f913c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f91467fc700 20 client.26682.objecter dump_active .. 0 homeless 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v209) v1 ==== 54+0+16 (secure 0 0 0) 0x7f914007d780 con 0x7f91500f8d10 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f915782c700 10 client.26682.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f91467fc700 10 client.26682.objecter ms_handle_connect 0x7f913c062c90 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 1 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f9154dc7700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6800/2866534053 conn(0x7f913c062c90 0x7f913c065150 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 rx=0x7f914c004ee0 tx=0x7f914c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9150028590 0x7f91500fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f91500f81a0 osd=0 3 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91500f81a0 osd=0 3 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f91500f81a0 osd=0 3 2024-02-13T00:42:45.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 1 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f91500f81a0 osd=0 3 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 0 1 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91500f81a0 osd=0 4 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f915782c700 10 client.26682.objecter _send_command 1 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91500fd370 con 0x7f9150028590 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f9155dc9700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9150028590 0x7f91500fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f91467fc700 10 client.26682.objecter ms_handle_connect 0x7f9150028590 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.191+0000 7f9155dc9700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9150028590 0x7f91500fbd00 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.192+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f91500fd370 con 0x7f9150028590 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.192+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.192+0000 7f91467fc700 10 client.26682.objecter _finish_command 1 = system:0 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.192+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f91500f81a0 osd=0 4 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.192+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 0 1 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.207+0000 7f915782c700 10 client.26682.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.207+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 2 2024-02-13T00:42:45.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.207+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f91500f81a0 osd=0 3 2024-02-13T00:42:45.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.207+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91500f81a0 osd=0 3 2024-02-13T00:42:45.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.207+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f91500f81a0 osd=0 3 2024-02-13T00:42:45.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.207+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 2 2024-02-13T00:42:45.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.207+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f91500f81a0 osd=0 3 2024-02-13T00:42:45.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.207+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 0 2 2024-02-13T00:42:45.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.207+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91500f81a0 osd=0 4 2024-02-13T00:42:45.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.207+0000 7f915782c700 10 client.26682.objecter _send_command 2 2024-02-13T00:42:45.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.207+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f91500fd4e0 con 0x7f9150028590 2024-02-13T00:42:45.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f91500fd4e0 con 0x7f9150028590 2024-02-13T00:42:45.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:45.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f91467fc700 10 client.26682.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f91500f81a0 osd=0 4 2024-02-13T00:42:45.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 0 2 2024-02-13T00:42:45.216 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f915782c700 10 client.26682.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 3 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6805/150599780 conn(0x7f91500fc870 0x7f91500fcc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f91500fc1f0 osd=1 3 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91500fc1f0 osd=1 3 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f91500fc1f0 osd=1 3 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 3 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f91500fc1f0 osd=1 3 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 1 3 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91500fc1f0 osd=1 4 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f915782c700 10 client.26682.objecter _send_command 3 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91500fd160 con 0x7f91500fc870 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.208+0000 7f91555c8700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6805/150599780 conn(0x7f91500fc870 0x7f91500fcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.209+0000 7f91467fc700 10 client.26682.objecter ms_handle_connect 0x7f91500fc870 2024-02-13T00:42:45.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.209+0000 7f91555c8700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6805/150599780 conn(0x7f91500fc870 0x7f91500fcc70 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:45.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.209+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f91500fd160 con 0x7f91500fc870 2024-02-13T00:42:45.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.209+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:42:45.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.209+0000 7f91467fc700 10 client.26682.objecter _finish_command 3 = system:0 2024-02-13T00:42:45.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.209+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f91500fc1f0 osd=1 4 2024-02-13T00:42:45.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.209+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 1 3 2024-02-13T00:42:45.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.224+0000 7f915782c700 10 client.26682.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:45.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.224+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 4 2024-02-13T00:42:45.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.224+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f91500fc1f0 osd=1 3 2024-02-13T00:42:45.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.224+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91500fc1f0 osd=1 3 2024-02-13T00:42:45.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.224+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f91500fc1f0 osd=1 3 2024-02-13T00:42:45.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.224+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 4 2024-02-13T00:42:45.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.224+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f91500fc1f0 osd=1 3 2024-02-13T00:42:45.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.224+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 1 4 2024-02-13T00:42:45.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.224+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91500fc1f0 osd=1 4 2024-02-13T00:42:45.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.224+0000 7f915782c700 10 client.26682.objecter _send_command 4 2024-02-13T00:42:45.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.224+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9150059400 con 0x7f91500fc870 2024-02-13T00:42:45.228 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:42:45.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9150059400 con 0x7f91500fc870 2024-02-13T00:42:45.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:45.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f91467fc700 10 client.26682.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f91500fc1f0 osd=1 4 2024-02-13T00:42:45.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 1 4 2024-02-13T00:42:45.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f915782c700 10 client.26682.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:45.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 5 2024-02-13T00:42:45.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6809/3689719376 conn(0x7f91500280f0 0x7f915011e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:45.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f9150059810 osd=2 3 2024-02-13T00:42:45.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150059810 osd=2 3 2024-02-13T00:42:45.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f9150059810 osd=2 3 2024-02-13T00:42:45.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 5 2024-02-13T00:42:45.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f9150059810 osd=2 3 2024-02-13T00:42:45.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 2 5 2024-02-13T00:42:45.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150059810 osd=2 4 2024-02-13T00:42:45.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f915782c700 10 client.26682.objecter _send_command 5 2024-02-13T00:42:45.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.225+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9150121110 con 0x7f91500280f0 2024-02-13T00:42:45.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.226+0000 7f9154dc7700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6809/3689719376 conn(0x7f91500280f0 0x7f915011e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:45.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.226+0000 7f91467fc700 10 client.26682.objecter ms_handle_connect 0x7f91500280f0 2024-02-13T00:42:45.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.226+0000 7f9154dc7700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6809/3689719376 conn(0x7f91500280f0 0x7f915011e430 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:45.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.226+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9150121110 con 0x7f91500280f0 2024-02-13T00:42:45.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.226+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:42:45.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.226+0000 7f91467fc700 10 client.26682.objecter _finish_command 5 = system:0 2024-02-13T00:42:45.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.226+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f9150059810 osd=2 4 2024-02-13T00:42:45.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.226+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 2 5 2024-02-13T00:42:45.234 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:42:45.236 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:44.141748+0000 mon.b (mon.1) 280 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:45.236 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:44.346687+0000 mon.a (mon.0) 1463 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:42:45.237 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:44.350915+0000 mon.a (mon.0) 1464 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-17", "mode": "writeback"}]': finished 2024-02-13T00:42:45.237 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:44.350997+0000 mon.a (mon.0) 1465 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]': finished 2024-02-13T00:42:45.237 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:44.351082+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e209: 8 total, 8 up, 7 in 2024-02-13T00:42:45.237 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:44.354382+0000 mon.c (mon.2) 191 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:45.237 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:44.355130+0000 mon.a (mon.0) 1467 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:45.237 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:44.357696+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? 172.21.15.162:0/3917516956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-83985-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:45.237 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:44.410046+0000 mon.b (mon.1) 281 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:45.237 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:44.410561+0000 mon.a (mon.0) 1469 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:45.237 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:44.141748+0000 mon.b (mon.1) 280 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:45.237 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:44.346687+0000 mon.a (mon.0) 1463 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:42:45.237 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:44.350915+0000 mon.a (mon.0) 1464 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-17", "mode": "writeback"}]': finished 2024-02-13T00:42:45.238 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:44.350997+0000 mon.a (mon.0) 1465 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]': finished 2024-02-13T00:42:45.238 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:44.351082+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e209: 8 total, 8 up, 7 in 2024-02-13T00:42:45.238 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:44.354382+0000 mon.c (mon.2) 191 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:45.238 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:44.355130+0000 mon.a (mon.0) 1467 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:45.238 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:44.357696+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? 172.21.15.162:0/3917516956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-83985-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:45.238 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:44.410046+0000 mon.b (mon.1) 281 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:45.238 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:44.410561+0000 mon.a (mon.0) 1469 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:45.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:42:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.244+0000 7f915782c700 10 client.26682.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.244+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 6 2024-02-13T00:42:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.244+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f9150059810 osd=2 3 2024-02-13T00:42:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.244+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150059810 osd=2 3 2024-02-13T00:42:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.244+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f9150059810 osd=2 3 2024-02-13T00:42:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.244+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 6 2024-02-13T00:42:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.244+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f9150059810 osd=2 3 2024-02-13T00:42:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.244+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 2 6 2024-02-13T00:42:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.244+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150059810 osd=2 4 2024-02-13T00:42:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.244+0000 7f915782c700 10 client.26682.objecter _send_command 6 2024-02-13T00:42:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.244+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f91500276b0 con 0x7f91500280f0 2024-02-13T00:42:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f91500276b0 con 0x7f91500280f0 2024-02-13T00:42:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f91467fc700 10 client.26682.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f9150059810 osd=2 4 2024-02-13T00:42:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 2 6 2024-02-13T00:42:45.250 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:42:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f915782c700 10 client.26682.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 7 2024-02-13T00:42:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6813/604971011 conn(0x7f915011faa0 0x7f915011fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f915011ba70 osd=3 3 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f915011ba70 osd=3 3 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f915011ba70 osd=3 3 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 7 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f915011ba70 osd=3 3 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 3 7 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f915011ba70 osd=3 4 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f915782c700 10 client.26682.objecter _send_command 7 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f915011c0f0 con 0x7f915011faa0 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f9155dc9700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6813/604971011 conn(0x7f915011faa0 0x7f915011fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f91467fc700 10 client.26682.objecter ms_handle_connect 0x7f915011faa0 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.246+0000 7f9155dc9700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6813/604971011 conn(0x7f915011faa0 0x7f915011fee0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.247+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f915011c0f0 con 0x7f915011faa0 2024-02-13T00:42:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.247+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:42:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.247+0000 7f91467fc700 10 client.26682.objecter _finish_command 7 = system:0 2024-02-13T00:42:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.247+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f915011ba70 osd=3 4 2024-02-13T00:42:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.247+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 3 7 2024-02-13T00:42:45.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.261+0000 7f915782c700 10 client.26682.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:45.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.261+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 8 2024-02-13T00:42:45.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.261+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f915011ba70 osd=3 3 2024-02-13T00:42:45.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.261+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f915011ba70 osd=3 3 2024-02-13T00:42:45.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.261+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f915011ba70 osd=3 3 2024-02-13T00:42:45.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.261+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 8 2024-02-13T00:42:45.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.261+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f915011ba70 osd=3 3 2024-02-13T00:42:45.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.261+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 3 8 2024-02-13T00:42:45.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.261+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f915011ba70 osd=3 4 2024-02-13T00:42:45.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.261+0000 7f915782c700 10 client.26682.objecter _send_command 8 2024-02-13T00:42:45.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.261+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9150120ba0 con 0x7f915011faa0 2024-02-13T00:42:45.266 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9150120ba0 con 0x7f915011faa0 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f91467fc700 10 client.26682.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f915011ba70 osd=3 4 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 3 8 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f915782c700 10 client.26682.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 9 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9150123340 0x7f9150125730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f91501203d0 osd=4 3 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91501203d0 osd=4 3 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f91501203d0 osd=4 3 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 9 2024-02-13T00:42:45.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f91501203d0 osd=4 3 2024-02-13T00:42:45.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 4 9 2024-02-13T00:42:45.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91501203d0 osd=4 4 2024-02-13T00:42:45.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f915782c700 10 client.26682.objecter _send_command 9 2024-02-13T00:42:45.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9150120ac0 con 0x7f9150123340 2024-02-13T00:42:45.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.263+0000 7f91555c8700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9150123340 0x7f9150125730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:45.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.264+0000 7f91467fc700 10 client.26682.objecter ms_handle_connect 0x7f9150123340 2024-02-13T00:42:45.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.264+0000 7f91555c8700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9150123340 0x7f9150125730 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:45.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.265+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9150120ac0 con 0x7f9150123340 2024-02-13T00:42:45.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.265+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:42:45.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.265+0000 7f91467fc700 10 client.26682.objecter _finish_command 9 = system:0 2024-02-13T00:42:45.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.265+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f91501203d0 osd=4 4 2024-02-13T00:42:45.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.265+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 4 9 2024-02-13T00:42:45.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.279+0000 7f915782c700 10 client.26682.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:45.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.279+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 10 2024-02-13T00:42:45.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.279+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f91501203d0 osd=4 3 2024-02-13T00:42:45.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.279+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91501203d0 osd=4 3 2024-02-13T00:42:45.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.279+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f91501203d0 osd=4 3 2024-02-13T00:42:45.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.279+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 10 2024-02-13T00:42:45.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.279+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f91501203d0 osd=4 3 2024-02-13T00:42:45.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.279+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 4 10 2024-02-13T00:42:45.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.279+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91501203d0 osd=4 4 2024-02-13T00:42:45.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.279+0000 7f915782c700 10 client.26682.objecter _send_command 10 2024-02-13T00:42:45.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.279+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9150120ac0 con 0x7f9150123340 2024-02-13T00:42:45.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.281+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9150120ac0 con 0x7f9150123340 2024-02-13T00:42:45.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.281+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:45.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.281+0000 7f91467fc700 10 client.26682.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.281+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f91501203d0 osd=4 4 2024-02-13T00:42:45.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.281+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 4 10 2024-02-13T00:42:45.285 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f915782c700 10 client.26682.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 11 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6804/2212296748 conn(0x7f9150126130 0x7f91501285a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f9150120ac0 osd=5 3 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150120ac0 osd=5 3 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f9150120ac0 osd=5 3 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 11 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f9150120ac0 osd=5 3 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 5 11 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150120ac0 osd=5 4 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f915782c700 10 client.26682.objecter _send_command 11 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f915012b3b0 con 0x7f9150126130 2024-02-13T00:42:45.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f9154dc7700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6804/2212296748 conn(0x7f9150126130 0x7f91501285a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:45.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f91467fc700 10 client.26682.objecter ms_handle_connect 0x7f9150126130 2024-02-13T00:42:45.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.282+0000 7f9154dc7700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6804/2212296748 conn(0x7f9150126130 0x7f91501285a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:45.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.283+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f915012b3b0 con 0x7f9150126130 2024-02-13T00:42:45.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.283+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:42:45.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.283+0000 7f91467fc700 10 client.26682.objecter _finish_command 11 = system:0 2024-02-13T00:42:45.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.283+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f9150120ac0 osd=5 4 2024-02-13T00:42:45.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.283+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 5 11 2024-02-13T00:42:45.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:44.141748+0000 mon.b (mon.1) 280 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:45.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:44.346687+0000 mon.a (mon.0) 1463 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:42:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:44.350915+0000 mon.a (mon.0) 1464 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-17", "mode": "writeback"}]': finished 2024-02-13T00:42:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:44.350997+0000 mon.a (mon.0) 1465 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-83977-29"}]': finished 2024-02-13T00:42:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:44.351082+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e209: 8 total, 8 up, 7 in 2024-02-13T00:42:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:44.354382+0000 mon.c (mon.2) 191 : audit [INF] from='client.? 172.21.15.162:0/1509178592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:44.355130+0000 mon.a (mon.0) 1467 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]: dispatch 2024-02-13T00:42:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:44.357696+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? 172.21.15.162:0/3917516956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-83985-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:44.410046+0000 mon.b (mon.1) 281 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:44.410561+0000 mon.a (mon.0) 1469 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:45.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.297+0000 7f915782c700 10 client.26682.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:45.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.297+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 12 2024-02-13T00:42:45.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.297+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f9150120ac0 osd=5 3 2024-02-13T00:42:45.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.297+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150120ac0 osd=5 3 2024-02-13T00:42:45.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.297+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f9150120ac0 osd=5 3 2024-02-13T00:42:45.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.297+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 12 2024-02-13T00:42:45.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.297+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f9150120ac0 osd=5 3 2024-02-13T00:42:45.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.297+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 5 12 2024-02-13T00:42:45.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.297+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150120ac0 osd=5 4 2024-02-13T00:42:45.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.297+0000 7f915782c700 10 client.26682.objecter _send_command 12 2024-02-13T00:42:45.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.297+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f915012b370 con 0x7f9150126130 2024-02-13T00:42:45.303 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:42:45.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f915012b370 con 0x7f9150126130 2024-02-13T00:42:45.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:45.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f91467fc700 10 client.26682.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f9150120ac0 osd=5 4 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 5 12 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f915782c700 10 client.26682.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 13 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6808/1506640341 conn(0x7f915012a670 0x7f915012aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f915012b370 osd=6 3 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f915012b370 osd=6 3 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f915012b370 osd=6 3 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 13 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f915012b370 osd=6 3 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 6 13 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f915012b370 osd=6 4 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f915782c700 10 client.26682.objecter _send_command 13 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.299+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f915012e710 con 0x7f915012a670 2024-02-13T00:42:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.300+0000 7f9155dc9700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6808/1506640341 conn(0x7f915012a670 0x7f915012aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:45.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.300+0000 7f91467fc700 10 client.26682.objecter ms_handle_connect 0x7f915012a670 2024-02-13T00:42:45.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.300+0000 7f9155dc9700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6808/1506640341 conn(0x7f915012a670 0x7f915012aaf0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:45.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.300+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f915012e710 con 0x7f915012a670 2024-02-13T00:42:45.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.300+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:42:45.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.300+0000 7f91467fc700 10 client.26682.objecter _finish_command 13 = system:0 2024-02-13T00:42:45.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.300+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f915012b370 osd=6 4 2024-02-13T00:42:45.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.300+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 6 13 2024-02-13T00:42:45.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.314+0000 7f915782c700 10 client.26682.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:45.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.314+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 14 2024-02-13T00:42:45.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.314+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f915012b370 osd=6 3 2024-02-13T00:42:45.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.314+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f915012b370 osd=6 3 2024-02-13T00:42:45.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.314+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f915012b370 osd=6 3 2024-02-13T00:42:45.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.314+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 14 2024-02-13T00:42:45.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.314+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f915012b370 osd=6 3 2024-02-13T00:42:45.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.314+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 6 14 2024-02-13T00:42:45.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.314+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f915012b370 osd=6 4 2024-02-13T00:42:45.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.314+0000 7f915782c700 10 client.26682.objecter _send_command 14 2024-02-13T00:42:45.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.314+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9150128b50 con 0x7f915012a670 2024-02-13T00:42:45.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.316+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9150128b50 con 0x7f915012a670 2024-02-13T00:42:45.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.316+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:45.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.316+0000 7f91467fc700 10 client.26682.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.316+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f915012b370 osd=6 4 2024-02-13T00:42:45.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.316+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 6 14 2024-02-13T00:42:45.320 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:42:45.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f915782c700 10 client.26682.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 15 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6812/1964192772 conn(0x7f91501291d0 0x7f9150129610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f9150128b50 osd=7 3 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150128b50 osd=7 3 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f9150128b50 osd=7 3 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 15 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f9150128b50 osd=7 3 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 7 15 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150128b50 osd=7 4 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f915782c700 10 client.26682.objecter _send_command 15 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9150129c30 con 0x7f91501291d0 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f91555c8700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6812/1964192772 conn(0x7f91501291d0 0x7f9150129610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f91467fc700 10 client.26682.objecter ms_handle_connect 0x7f91501291d0 2024-02-13T00:42:45.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.317+0000 7f91555c8700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6812/1964192772 conn(0x7f91501291d0 0x7f9150129610 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:45.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.318+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9150129c30 con 0x7f91501291d0 2024-02-13T00:42:45.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.318+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:42:45.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.318+0000 7f91467fc700 10 client.26682.objecter _finish_command 15 = system:0 2024-02-13T00:42:45.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.318+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f9150128b50 osd=7 4 2024-02-13T00:42:45.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.318+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 7 15 2024-02-13T00:42:45.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.333+0000 7f915782c700 10 client.26682.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:42:45.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.333+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign -1 16 2024-02-13T00:42:45.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.333+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f9150128b50 osd=7 3 2024-02-13T00:42:45.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.333+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150128b50 osd=7 3 2024-02-13T00:42:45.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.333+0000 7f915782c700 20 client.26682.objecter _get_session s=0x7f9150128b50 osd=7 3 2024-02-13T00:42:45.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.333+0000 7f915782c700 15 client.26682.objecter _session_command_op_remove -1 16 2024-02-13T00:42:45.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.333+0000 7f915782c700 20 client.26682.objecter get_session s=0x7f9150128b50 osd=7 3 2024-02-13T00:42:45.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.333+0000 7f915782c700 15 client.26682.objecter _session_command_op_assign 7 16 2024-02-13T00:42:45.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.333+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150128b50 osd=7 4 2024-02-13T00:42:45.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.333+0000 7f915782c700 10 client.26682.objecter _send_command 16 2024-02-13T00:42:45.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.333+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9150129c30 con 0x7f91501291d0 2024-02-13T00:42:45.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.335+0000 7f91467fc700 1 -- 172.21.15.162:0/717747144 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9150129c30 con 0x7f91501291d0 2024-02-13T00:42:45.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.335+0000 7f91467fc700 10 client.26682.objecter ms_dispatch 0x7f9150001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:42:45.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.335+0000 7f91467fc700 10 client.26682.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.335+0000 7f91467fc700 20 client.26682.objecter put_session s=0x7f9150128b50 osd=7 4 2024-02-13T00:42:45.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.335+0000 7f91467fc700 15 client.26682.objecter _session_command_op_remove 7 16 2024-02-13T00:42:45.339 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:42:45.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.335+0000 7f915782c700 10 librados: watch_flush enter 2024-02-13T00:42:45.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.335+0000 7f915782c700 10 librados: watch_flush exit 2024-02-13T00:42:45.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 10 client.26682.objecter close_session for osd.0 2024-02-13T00:42:45.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9150028590 msgr2=0x7f91500fbd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9150028590 0x7f91500fbd00 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91500f81a0 osd=0 1 2024-02-13T00:42:45.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 10 client.26682.objecter close_session for osd.1 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6805/150599780 conn(0x7f91500fc870 msgr2=0x7f91500fcc70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6805/150599780 conn(0x7f91500fc870 0x7f91500fcc70 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91500fc1f0 osd=1 1 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 10 client.26682.objecter close_session for osd.2 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6809/3689719376 conn(0x7f91500280f0 msgr2=0x7f915011e430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6809/3689719376 conn(0x7f91500280f0 0x7f915011e430 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150059810 osd=2 1 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 10 client.26682.objecter close_session for osd.3 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6813/604971011 conn(0x7f915011faa0 msgr2=0x7f915011fee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6813/604971011 conn(0x7f915011faa0 0x7f915011fee0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f9155dc9700 1 -- 172.21.15.162:0/717747144 reap_dead start 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f915011ba70 osd=3 1 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 10 client.26682.objecter close_session for osd.4 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9150123340 msgr2=0x7f9150125730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9150123340 0x7f9150125730 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f91501203d0 osd=4 1 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 10 client.26682.objecter close_session for osd.5 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6804/2212296748 conn(0x7f9150126130 msgr2=0x7f91501285a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6804/2212296748 conn(0x7f9150126130 0x7f91501285a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150120ac0 osd=5 1 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 10 client.26682.objecter close_session for osd.6 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6808/1506640341 conn(0x7f915012a670 msgr2=0x7f915012aaf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6808/1506640341 conn(0x7f915012a670 0x7f915012aaf0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f915012b370 osd=6 1 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 10 client.26682.objecter close_session for osd.7 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6812/1964192772 conn(0x7f91501291d0 msgr2=0x7f9150129610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.181:6812/1964192772 conn(0x7f91501291d0 0x7f9150129610 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 20 client.26682.objecter put_session s=0x7f9150128b50 osd=7 1 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 20 client.26682.objecter shutdown clearing up homeless session... 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 10 client.26682.objecter successfully canceled tick 2024-02-13T00:42:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f9155dc9700 1 -- 172.21.15.162:0/717747144 reap_dead start 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6800/2866534053 conn(0x7f913c062c90 msgr2=0x7f913c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6800/2866534053 conn(0x7f913c062c90 0x7f913c065150 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 rx=0x7f914c004ee0 tx=0x7f914c00b040).stop 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91500f8d10 msgr2=0x7f91500f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91500f8d10 0x7f91500f6cc0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 rx=0x7f91400048f0 tx=0x7f9140009cb0).stop 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 shutdown_connections 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91500f8d10 0x7f91500f6cc0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 --2- 172.21.15.162:0/717747144 >> v2:172.21.15.162:6800/2866534053 conn(0x7f913c062c90 0x7f913c065150 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 >> 172.21.15.162:0/717747144 conn(0x7f915006eb90 msgr2=0x7f9150102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 shutdown_connections 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 -- 172.21.15.162:0/717747144 wait complete. 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:45.336+0000 7f915782c700 1 librados: shutdown 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.344 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.344 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.344 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:42:45.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:45.380 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:45.404 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:45.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:45.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:45.535 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:45.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:42:45.921 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:46.277 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:42:46] "GET /metrics HTTP/1.1" 200 194964 "" "Prometheus/2.33.4" 2024-02-13T00:42:46.278 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:44.347304+0000 mgr.y (mgr.14544) 230 : cluster [DBG] pgmap v283: 260 pgs: 11 active+clean+snaptrim, 32 unknown, 19 active+clean+snaptrim_wait, 198 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:46.278 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.142841+0000 mon.b (mon.1) 282 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:46.278 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.190967+0000 mon.a (mon.0) 1470 : audit [DBG] from='client.? 172.21.15.162:0/717747144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:46.278 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.362456+0000 mon.a (mon.0) 1471 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]': finished 2024-02-13T00:42:46.278 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.362544+0000 mon.a (mon.0) 1472 : audit [INF] from='client.? 172.21.15.162:0/3917516956' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-83985-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:46.278 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.362674+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:42:46.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:45.362733+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e210: 8 total, 8 up, 7 in 2024-02-13T00:42:46.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.363320+0000 mon.b (mon.1) 283 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17"}]: dispatch 2024-02-13T00:42:46.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.363931+0000 mon.a (mon.0) 1475 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17"}]: dispatch 2024-02-13T00:42:46.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.393672+0000 mon.c (mon.2) 192 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:46.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.393998+0000 mon.a (mon.0) 1476 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:46.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.394409+0000 mon.c (mon.2) 193 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:46.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.394622+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:46.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:44.347304+0000 mgr.y (mgr.14544) 230 : cluster [DBG] pgmap v283: 260 pgs: 11 active+clean+snaptrim, 32 unknown, 19 active+clean+snaptrim_wait, 198 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:46.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.142841+0000 mon.b (mon.1) 282 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:46.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.190967+0000 mon.a (mon.0) 1470 : audit [DBG] from='client.? 172.21.15.162:0/717747144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:46.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.362456+0000 mon.a (mon.0) 1471 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]': finished 2024-02-13T00:42:46.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.362544+0000 mon.a (mon.0) 1472 : audit [INF] from='client.? 172.21.15.162:0/3917516956' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-83985-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:46.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.362674+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:42:46.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:45.362733+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e210: 8 total, 8 up, 7 in 2024-02-13T00:42:46.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.363320+0000 mon.b (mon.1) 283 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17"}]: dispatch 2024-02-13T00:42:46.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.363931+0000 mon.a (mon.0) 1475 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17"}]: dispatch 2024-02-13T00:42:46.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.393672+0000 mon.c (mon.2) 192 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:46.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.393998+0000 mon.a (mon.0) 1476 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:46.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.394409+0000 mon.c (mon.2) 193 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:46.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.394622+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:46.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.394963+0000 mon.c (mon.2) 194 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-83977-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:46.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.395187+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-83977-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:46.281 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.394963+0000 mon.c (mon.2) 194 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-83977-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:46.281 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.395187+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-83977-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:46.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:44.347304+0000 mgr.y (mgr.14544) 230 : cluster [DBG] pgmap v283: 260 pgs: 11 active+clean+snaptrim, 32 unknown, 19 active+clean+snaptrim_wait, 198 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.142841+0000 mon.b (mon.1) 282 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.190967+0000 mon.a (mon.0) 1470 : audit [DBG] from='client.? 172.21.15.162:0/717747144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.362456+0000 mon.a (mon.0) 1471 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-83977-29"}]': finished 2024-02-13T00:42:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.362544+0000 mon.a (mon.0) 1472 : audit [INF] from='client.? 172.21.15.162:0/3917516956' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-83985-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.362674+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:42:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:45.362733+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e210: 8 total, 8 up, 7 in 2024-02-13T00:42:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.363320+0000 mon.b (mon.1) 283 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17"}]: dispatch 2024-02-13T00:42:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.363931+0000 mon.a (mon.0) 1475 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17"}]: dispatch 2024-02-13T00:42:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.393672+0000 mon.c (mon.2) 192 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.393998+0000 mon.a (mon.0) 1476 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.394409+0000 mon.c (mon.2) 193 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:46.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.394622+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:46.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.394963+0000 mon.c (mon.2) 194 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-83977-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:46.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.395187+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-83977-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:46.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.511+0000 7f855baf9700 1 -- 172.21.15.162:0/2450972795 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85540fdc10 msgr2=0x7f85540fdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:46.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.511+0000 7f855baf9700 1 --2- 172.21.15.162:0/2450972795 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85540fdc10 0x7f85540fdff0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7f8550004870 tx=0x7f8550009cb0).stop 2024-02-13T00:42:46.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.511+0000 7f855baf9700 1 -- 172.21.15.162:0/2450972795 shutdown_connections 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.511+0000 7f855baf9700 1 --2- 172.21.15.162:0/2450972795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85540f8480 0x7f8554104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.511+0000 7f855baf9700 1 --2- 172.21.15.162:0/2450972795 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85540fdc10 0x7f85540fdff0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.511+0000 7f855baf9700 1 --2- 172.21.15.162:0/2450972795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85540fe570 0x7f85540f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.511+0000 7f855baf9700 1 -- 172.21.15.162:0/2450972795 >> 172.21.15.162:0/2450972795 conn(0x7f85540f3800 msgr2=0x7f85540f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.511+0000 7f855baf9700 1 -- 172.21.15.162:0/2450972795 shutdown_connections 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.511+0000 7f855baf9700 1 -- 172.21.15.162:0/2450972795 wait complete. 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855baf9700 1 librados: starting msgr at 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855baf9700 1 librados: starting objecter 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855baf9700 1 Processor -- start 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855baf9700 1 -- start start 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855baf9700 1 librados: setting wanted keys 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855baf9700 1 librados: calling monclient init 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855baf9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85540f8480 0x7f855410e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855baf9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85540fdc10 0x7f855410e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:46.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855baf9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85540fe570 0x7f85541081d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855baf9700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85541086c0 con 0x7f85540f8480 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855baf9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8554108830 con 0x7f85540fdc10 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855baf9700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85541089f0 con 0x7f85540fe570 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855a096700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85540fe570 0x7f85541081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f8559895700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85540f8480 0x7f855410e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855a096700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85540fe570 0x7f85541081d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:33644/0 (socket says 172.21.15.162:33644) 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f855a096700 1 -- 172.21.15.162:0/4034609847 learned_addr learned my addr 172.21.15.162:0/4034609847 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.512+0000 7f8559094700 1 --2- 172.21.15.162:0/4034609847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85540fdc10 0x7f855410e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f855a096700 1 -- 172.21.15.162:0/4034609847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85540fdc10 msgr2=0x7f855410e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f855a096700 1 --2- 172.21.15.162:0/4034609847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85540fdc10 0x7f855410e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f855a096700 1 -- 172.21.15.162:0/4034609847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85540f8480 msgr2=0x7f855410e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f855a096700 1 --2- 172.21.15.162:0/4034609847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85540f8480 0x7f855410e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f855a096700 1 -- 172.21.15.162:0/4034609847 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8550005040 con 0x7f85540fe570 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f855a096700 1 --2- 172.21.15.162:0/4034609847 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85540fe570 0x7f85541081d0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 rx=0x7f854c00d970 tx=0x7f854c004e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:46.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f854affd700 10 client.?.objecter ms_handle_connect 0x7f85540fe570 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f854affd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f854affd700 1 -- 172.21.15.162:0/4034609847 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c004020 con 0x7f85540fe570 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f855baf9700 1 -- 172.21.15.162:0/4034609847 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8554108cd0 con 0x7f85540fe570 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f854affd700 1 -- 172.21.15.162:0/4034609847 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f854c00f040 con 0x7f85540fe570 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f854affd700 1 -- 172.21.15.162:0/4034609847 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c0036c0 con 0x7f85540fe570 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f855baf9700 10 client.36803.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f855baf9700 1 -- 172.21.15.162:0/4034609847 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8554114e60 con 0x7f85540fe570 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.513+0000 7f855baf9700 1 librados: init done 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.514+0000 7f855baf9700 1 -- 172.21.15.162:0/4034609847 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855405fb20 con 0x7f85540fe570 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.514+0000 7f854affd700 1 -- 172.21.15.162:0/4034609847 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f854c003af0 con 0x7f85540fe570 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.515+0000 7f854affd700 1 --2- 172.21.15.162:0/4034609847 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8540062c90 0x7f8540065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.515+0000 7f854affd700 1 -- 172.21.15.162:0/4034609847 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(211..211 src has 52..211) v4 ==== 8613+0+0 (secure 0 0 0) 0x7f854c07ce60 con 0x7f85540fe570 2024-02-13T00:42:46.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.515+0000 7f854affd700 10 client.36803.objecter ms_dispatch 0x7f8554001550 osd_map(211..211 src has 52..211) v4 2024-02-13T00:42:46.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.515+0000 7f854affd700 3 client.36803.objecter handle_osd_map got epochs [211,211] > 0 2024-02-13T00:42:46.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.515+0000 7f854affd700 3 client.36803.objecter handle_osd_map decoding full epoch 211 2024-02-13T00:42:46.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.515+0000 7f8559895700 1 --2- 172.21.15.162:0/4034609847 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8540062c90 0x7f8540065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:46.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.515+0000 7f854affd700 20 client.36803.objecter dump_active .. 0 homeless 2024-02-13T00:42:46.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.515+0000 7f854affd700 10 client.36803.objecter ms_handle_connect 0x7f8540062c90 2024-02-13T00:42:46.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.515+0000 7f8559895700 1 --2- 172.21.15.162:0/4034609847 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8540062c90 0x7f8540065150 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 rx=0x7f8550004540 tx=0x7f8550009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:46.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.522+0000 7f854affd700 1 -- 172.21.15.162:0/4034609847 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f854c048f60 con 0x7f85540fe570 2024-02-13T00:42:46.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:46.701+0000 7f855baf9700 1 -- 172.21.15.162:0/4034609847 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f85541096b0 con 0x7f85540fe570 2024-02-13T00:42:46.938 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:47.098 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:45.426732+0000 mgr.y (mgr.14544) 231 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:47.098 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:46.144525+0000 mon.b (mon.1) 284 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:47.098 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:46.354823+0000 mon.a (mon.0) 1479 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:42:47.099 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:46.359177+0000 mon.a (mon.0) 1480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17"}]': finished 2024-02-13T00:42:47.099 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:46.359288+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-83977-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:47.099 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:46.359353+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e211: 8 total, 8 up, 7 in 2024-02-13T00:42:47.099 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:46.368695+0000 mon.c (mon.2) 195 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-83977-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:47.099 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:46.369232+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-83977-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:47.099 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:46.702022+0000 mon.c (mon.2) 196 : audit [INF] from='client.? 172.21.15.162:0/4034609847' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:42:47.099 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:46.702321+0000 mon.a (mon.0) 1484 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:42:47.099 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:45.426732+0000 mgr.y (mgr.14544) 231 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:47.099 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:46.144525+0000 mon.b (mon.1) 284 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:47.099 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:46.354823+0000 mon.a (mon.0) 1479 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:42:47.099 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:46.359177+0000 mon.a (mon.0) 1480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17"}]': finished 2024-02-13T00:42:47.099 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:46.359288+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-83977-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:47.100 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:46.359353+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e211: 8 total, 8 up, 7 in 2024-02-13T00:42:47.100 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:46.368695+0000 mon.c (mon.2) 195 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-83977-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:47.100 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:46.369232+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-83977-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:47.100 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:46.702022+0000 mon.c (mon.2) 196 : audit [INF] from='client.? 172.21.15.162:0/4034609847' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:42:47.100 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:46.702321+0000 mon.a (mon.0) 1484 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:42:47.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:45.426732+0000 mgr.y (mgr.14544) 231 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:46.144525+0000 mon.b (mon.1) 284 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:46.354823+0000 mon.a (mon.0) 1479 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:42:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:46.359177+0000 mon.a (mon.0) 1480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-17"}]': finished 2024-02-13T00:42:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:46.359288+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-83977-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:46.359353+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e211: 8 total, 8 up, 7 in 2024-02-13T00:42:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:46.368695+0000 mon.c (mon.2) 195 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-83977-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:46.369232+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-83977-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:46.702022+0000 mon.c (mon.2) 196 : audit [INF] from='client.? 172.21.15.162:0/4034609847' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:42:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:46.702321+0000 mon.a (mon.0) 1484 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:42:47.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.610+0000 7f854affd700 1 -- 172.21.15.162:0/4034609847 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v212) v1 ==== 91+0+0 (secure 0 0 0) 0x7f854c050100 con 0x7f85540fe570 2024-02-13T00:42:47.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 10 librados: watch_flush enter 2024-02-13T00:42:47.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 10 librados: watch_flush exit 2024-02-13T00:42:47.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 20 client.36803.objecter shutdown clearing up homeless session... 2024-02-13T00:42:47.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 10 client.36803.objecter successfully canceled tick 2024-02-13T00:42:47.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 -- 172.21.15.162:0/4034609847 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8540062c90 msgr2=0x7f8540065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:47.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 --2- 172.21.15.162:0/4034609847 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8540062c90 0x7f8540065150 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 rx=0x7f8550004540 tx=0x7f8550009cb0).stop 2024-02-13T00:42:47.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 -- 172.21.15.162:0/4034609847 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85540fe570 msgr2=0x7f85541081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:47.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 --2- 172.21.15.162:0/4034609847 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85540fe570 0x7f85541081d0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 rx=0x7f854c00d970 tx=0x7f854c004e60).stop 2024-02-13T00:42:47.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 -- 172.21.15.162:0/4034609847 shutdown_connections 2024-02-13T00:42:47.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 --2- 172.21.15.162:0/4034609847 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8540062c90 0x7f8540065150 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:47.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 --2- 172.21.15.162:0/4034609847 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85540fe570 0x7f85541081d0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:47.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 --2- 172.21.15.162:0/4034609847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85540f8480 0x7f855410e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:47.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 --2- 172.21.15.162:0/4034609847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85540fdc10 0x7f855410e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:47.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 -- 172.21.15.162:0/4034609847 >> 172.21.15.162:0/4034609847 conn(0x7f85540f3800 msgr2=0x7f85540f52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:47.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 -- 172.21.15.162:0/4034609847 shutdown_connections 2024-02-13T00:42:47.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 -- 172.21.15.162:0/4034609847 wait complete. 2024-02-13T00:42:47.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:47.613+0000 7f855baf9700 1 librados: shutdown 2024-02-13T00:42:47.618 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:42:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:46.347747+0000 mgr.y (mgr.14544) 232 : cluster [DBG] pgmap v286: 292 pgs: 32 creating+peering, 9 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 234 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 765 B/s wr, 3 op/s 2024-02-13T00:42:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:47.145568+0000 mon.b (mon.1) 285 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:47.357122+0000 mon.a (mon.0) 1485 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:47.610467+0000 mon.a (mon.0) 1486 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:42:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:47.610547+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e212: 8 total, 8 up, 7 in 2024-02-13T00:42:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:47.612628+0000 mon.a (mon.0) 1488 : audit [INF] from='client.? 172.21.15.162:0/3125404246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi162-83985-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:48.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:46.347747+0000 mgr.y (mgr.14544) 232 : cluster [DBG] pgmap v286: 292 pgs: 32 creating+peering, 9 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 234 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 765 B/s wr, 3 op/s 2024-02-13T00:42:48.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:47.145568+0000 mon.b (mon.1) 285 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:48.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:47.357122+0000 mon.a (mon.0) 1485 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:48.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:47.610467+0000 mon.a (mon.0) 1486 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:42:48.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:47.610547+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e212: 8 total, 8 up, 7 in 2024-02-13T00:42:48.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:47.612628+0000 mon.a (mon.0) 1488 : audit [INF] from='client.? 172.21.15.162:0/3125404246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi162-83985-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:48.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:46.347747+0000 mgr.y (mgr.14544) 232 : cluster [DBG] pgmap v286: 292 pgs: 32 creating+peering, 9 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 234 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 765 B/s wr, 3 op/s 2024-02-13T00:42:48.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:47.145568+0000 mon.b (mon.1) 285 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:48.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:47.357122+0000 mon.a (mon.0) 1485 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:42:48.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:47.610467+0000 mon.a (mon.0) 1486 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:42:48.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:47.610547+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e212: 8 total, 8 up, 7 in 2024-02-13T00:42:48.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:47.612628+0000 mon.a (mon.0) 1488 : audit [INF] from='client.? 172.21.15.162:0/3125404246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi162-83985-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:48.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:48.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:48.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:48.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:48.527 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:48.547 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:48.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:48.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:48.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:48.677 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:48.696 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:48.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:48.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:48.794 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:42:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:42:48.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:48.833 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:48.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:42:49.252 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:47.412526+0000 osd.5 (osd.5) 93 : cluster [DBG] 5.1b scrub starts 2024-02-13T00:42:49.252 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:47.415408+0000 osd.5 (osd.5) 94 : cluster [DBG] 5.1b scrub ok 2024-02-13T00:42:49.253 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:47.643432+0000 osd.7 (osd.7) 96 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:42:49.253 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:47.646316+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:42:49.253 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:47.779368+0000 osd.3 (osd.3) 123 : cluster [DBG] 6.c scrub starts 2024-02-13T00:42:49.253 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:47.782340+0000 osd.3 (osd.3) 124 : cluster [DBG] 6.c scrub ok 2024-02-13T00:42:49.253 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:48.146585+0000 mon.b (mon.1) 286 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:49.253 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:48.299399+0000 osd.2 (osd.2) 87 : cluster [DBG] 28.1b scrub starts 2024-02-13T00:42:49.253 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:48.300397+0000 osd.2 (osd.2) 88 : cluster [DBG] 28.1b scrub ok 2024-02-13T00:42:49.253 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:48.365386+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-83977-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-83977-30"}]': finished 2024-02-13T00:42:49.253 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:48.365586+0000 mon.a (mon.0) 1490 : audit [INF] from='client.? 172.21.15.162:0/3125404246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi162-83985-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:49.253 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:48.365705+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e213: 8 total, 8 up, 7 in 2024-02-13T00:42:49.253 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:48.374403+0000 mon.b (mon.1) 287 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:49.253 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:48.375426+0000 mon.a (mon.0) 1492 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:49.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:47.412526+0000 osd.5 (osd.5) 93 : cluster [DBG] 5.1b scrub starts 2024-02-13T00:42:49.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:47.415408+0000 osd.5 (osd.5) 94 : cluster [DBG] 5.1b scrub ok 2024-02-13T00:42:49.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:47.643432+0000 osd.7 (osd.7) 96 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:42:49.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:47.646316+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:42:49.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:47.779368+0000 osd.3 (osd.3) 123 : cluster [DBG] 6.c scrub starts 2024-02-13T00:42:49.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:47.782340+0000 osd.3 (osd.3) 124 : cluster [DBG] 6.c scrub ok 2024-02-13T00:42:49.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:48.146585+0000 mon.b (mon.1) 286 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:49.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:48.299399+0000 osd.2 (osd.2) 87 : cluster [DBG] 28.1b scrub starts 2024-02-13T00:42:49.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:48.300397+0000 osd.2 (osd.2) 88 : cluster [DBG] 28.1b scrub ok 2024-02-13T00:42:49.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:48.365386+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-83977-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-83977-30"}]': finished 2024-02-13T00:42:49.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:48.365586+0000 mon.a (mon.0) 1490 : audit [INF] from='client.? 172.21.15.162:0/3125404246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi162-83985-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:49.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:48.365705+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e213: 8 total, 8 up, 7 in 2024-02-13T00:42:49.255 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:48.374403+0000 mon.b (mon.1) 287 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:49.255 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:48.375426+0000 mon.a (mon.0) 1492 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:49.283 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:49.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:49.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:47.412526+0000 osd.5 (osd.5) 93 : cluster [DBG] 5.1b scrub starts 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:47.415408+0000 osd.5 (osd.5) 94 : cluster [DBG] 5.1b scrub ok 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:47.643432+0000 osd.7 (osd.7) 96 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:47.646316+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:47.779368+0000 osd.3 (osd.3) 123 : cluster [DBG] 6.c scrub starts 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:47.782340+0000 osd.3 (osd.3) 124 : cluster [DBG] 6.c scrub ok 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:48.146585+0000 mon.b (mon.1) 286 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:48.299399+0000 osd.2 (osd.2) 87 : cluster [DBG] 28.1b scrub starts 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:48.300397+0000 osd.2 (osd.2) 88 : cluster [DBG] 28.1b scrub ok 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:48.365386+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-83977-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-83977-30"}]': finished 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:48.365586+0000 mon.a (mon.0) 1490 : audit [INF] from='client.? 172.21.15.162:0/3125404246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi162-83985-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:48.365705+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e213: 8 total, 8 up, 7 in 2024-02-13T00:42:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:48.374403+0000 mon.b (mon.1) 287 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:49.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:48.375426+0000 mon.a (mon.0) 1492 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:49.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:49.426 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:49.445 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:49.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:49.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:49.575 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:49.594 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:49.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:49.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:49.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:49.727 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:50.222 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:42:50.222 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:42:50.245 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:48.348215+0000 mgr.y (mgr.14544) 233 : cluster [DBG] pgmap v289: 260 pgs: 9 active+clean+snaptrim, 32 unknown, 17 active+clean+snaptrim_wait, 202 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:42:50.245 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:48.626110+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.1a scrub starts 2024-02-13T00:42:50.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:48.629017+0000 osd.7 (osd.7) 99 : cluster [DBG] 5.1a scrub ok 2024-02-13T00:42:50.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:48.770637+0000 osd.3 (osd.3) 125 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:42:50.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:48.774164+0000 osd.3 (osd.3) 126 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:42:50.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:49.147709+0000 mon.b (mon.1) 288 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:50.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:49.367818+0000 mon.a (mon.0) 1493 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:50.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:49.367950+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e214: 8 total, 8 up, 7 in 2024-02-13T00:42:50.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:49.377789+0000 mon.b (mon.1) 289 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:50.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:49.382541+0000 mon.a (mon.0) 1495 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:50.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:48.348215+0000 mgr.y (mgr.14544) 233 : cluster [DBG] pgmap v289: 260 pgs: 9 active+clean+snaptrim, 32 unknown, 17 active+clean+snaptrim_wait, 202 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:42:50.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:48.626110+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.1a scrub starts 2024-02-13T00:42:50.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:48.629017+0000 osd.7 (osd.7) 99 : cluster [DBG] 5.1a scrub ok 2024-02-13T00:42:50.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:48.770637+0000 osd.3 (osd.3) 125 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:42:50.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:48.774164+0000 osd.3 (osd.3) 126 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:42:50.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:49.147709+0000 mon.b (mon.1) 288 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:50.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:49.367818+0000 mon.a (mon.0) 1493 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:50.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:49.367950+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e214: 8 total, 8 up, 7 in 2024-02-13T00:42:50.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:49.377789+0000 mon.b (mon.1) 289 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:50.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:49.382541+0000 mon.a (mon.0) 1495 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:50.251 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:50.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:48.348215+0000 mgr.y (mgr.14544) 233 : cluster [DBG] pgmap v289: 260 pgs: 9 active+clean+snaptrim, 32 unknown, 17 active+clean+snaptrim_wait, 202 active+clean; 11 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:42:50.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:48.626110+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.1a scrub starts 2024-02-13T00:42:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:48.629017+0000 osd.7 (osd.7) 99 : cluster [DBG] 5.1a scrub ok 2024-02-13T00:42:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:48.770637+0000 osd.3 (osd.3) 125 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:42:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:48.774164+0000 osd.3 (osd.3) 126 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:42:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:49.147709+0000 mon.b (mon.1) 288 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:49.367818+0000 mon.a (mon.0) 1493 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:49.367950+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e214: 8 total, 8 up, 7 in 2024-02-13T00:42:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:49.377789+0000 mon.b (mon.1) 289 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:49.382541+0000 mon.a (mon.0) 1495 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:50.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:42:50.977 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:42:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:50.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:42:51.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:51.113 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:51.134 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:51.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:51.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:51.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:51.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:50.149374+0000 mon.b (mon.1) 290 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:51.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:50.322167+0000 osd.2 (osd.2) 89 : cluster [DBG] 28.1e scrub starts 2024-02-13T00:42:51.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:50.325059+0000 osd.2 (osd.2) 90 : cluster [DBG] 28.1e scrub ok 2024-02-13T00:42:51.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:50.328125+0000 mon.a (mon.0) 1496 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:42:51.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:50.328268+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e215: 8 total, 8 up, 7 in 2024-02-13T00:42:51.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:50.331174+0000 mon.b (mon.1) 291 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-19"}]: dispatch 2024-02-13T00:42:51.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:50.333209+0000 mon.a (mon.0) 1498 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-19"}]: dispatch 2024-02-13T00:42:51.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:50.344995+0000 mon.c (mon.2) 197 : audit [INF] from='client.? 172.21.15.162:0/2345739317' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-83985-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:51.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:50.345816+0000 mon.c (mon.2) 198 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:51.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:50.347053+0000 mon.a (mon.0) 1499 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-83985-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:51.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:50.347499+0000 mon.a (mon.0) 1500 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:51.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:50.149374+0000 mon.b (mon.1) 290 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:51.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:50.322167+0000 osd.2 (osd.2) 89 : cluster [DBG] 28.1e scrub starts 2024-02-13T00:42:51.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:50.325059+0000 osd.2 (osd.2) 90 : cluster [DBG] 28.1e scrub ok 2024-02-13T00:42:51.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:50.328125+0000 mon.a (mon.0) 1496 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:42:51.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:50.328268+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e215: 8 total, 8 up, 7 in 2024-02-13T00:42:51.265 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:50.331174+0000 mon.b (mon.1) 291 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-19"}]: dispatch 2024-02-13T00:42:51.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:50.333209+0000 mon.a (mon.0) 1498 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-19"}]: dispatch 2024-02-13T00:42:51.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:50.344995+0000 mon.c (mon.2) 197 : audit [INF] from='client.? 172.21.15.162:0/2345739317' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-83985-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:51.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:50.345816+0000 mon.c (mon.2) 198 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:51.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:50.347053+0000 mon.a (mon.0) 1499 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-83985-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:51.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:50.347499+0000 mon.a (mon.0) 1500 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:51.266 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:51.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:50.149374+0000 mon.b (mon.1) 290 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:50.322167+0000 osd.2 (osd.2) 89 : cluster [DBG] 28.1e scrub starts 2024-02-13T00:42:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:50.325059+0000 osd.2 (osd.2) 90 : cluster [DBG] 28.1e scrub ok 2024-02-13T00:42:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:50.328125+0000 mon.a (mon.0) 1496 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:42:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:50.328268+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e215: 8 total, 8 up, 7 in 2024-02-13T00:42:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:50.331174+0000 mon.b (mon.1) 291 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-19"}]: dispatch 2024-02-13T00:42:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:50.333209+0000 mon.a (mon.0) 1498 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-19"}]: dispatch 2024-02-13T00:42:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:50.344995+0000 mon.c (mon.2) 197 : audit [INF] from='client.? 172.21.15.162:0/2345739317' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-83985-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:50.345816+0000 mon.c (mon.2) 198 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:50.347053+0000 mon.a (mon.0) 1499 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-83985-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:50.347499+0000 mon.a (mon.0) 1500 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:51.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.217+0000 7ff168035700 1 -- 172.21.15.162:0/1707185834 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1600f8500 msgr2=0x7ff160104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:51.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.217+0000 7ff168035700 1 --2- 172.21.15.162:0/1707185834 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1600f8500 0x7ff160104bd0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7ff158009be0 tx=0x7ff158004d10).stop 2024-02-13T00:42:51.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.217+0000 7ff168035700 1 -- 172.21.15.162:0/1707185834 shutdown_connections 2024-02-13T00:42:51.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.217+0000 7ff168035700 1 --2- 172.21.15.162:0/1707185834 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1600f8500 0x7ff160104bd0 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:51.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.217+0000 7ff168035700 1 --2- 172.21.15.162:0/1707185834 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1600f9b70 0x7ff1600f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:51.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.217+0000 7ff168035700 1 --2- 172.21.15.162:0/1707185834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1600fa440 0x7ff1600f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:51.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.217+0000 7ff168035700 1 -- 172.21.15.162:0/1707185834 >> 172.21.15.162:0/1707185834 conn(0x7ff1600f38c0 msgr2=0x7ff1600f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:51.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.217+0000 7ff168035700 1 -- 172.21.15.162:0/1707185834 shutdown_connections 2024-02-13T00:42:51.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.217+0000 7ff168035700 1 -- 172.21.15.162:0/1707185834 wait complete. 2024-02-13T00:42:51.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.217+0000 7ff168035700 1 librados: starting msgr at 2024-02-13T00:42:51.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.217+0000 7ff168035700 1 librados: starting objecter 2024-02-13T00:42:51.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.217+0000 7ff168035700 1 Processor -- start 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff168035700 1 -- start start 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff168035700 1 librados: setting wanted keys 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff168035700 1 librados: calling monclient init 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff168035700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1600f8500 0x7ff16006b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff168035700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1600f9b70 0x7ff16006b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff168035700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1600fa440 0x7ff16006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff168035700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1601165c0 con 0x7ff1600f8500 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff168035700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff160116730 con 0x7ff1600fa440 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff168035700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1601168f0 con 0x7ff1600f9b70 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff165dd1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1600f8500 0x7ff16006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff165dd1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1600f8500 0x7ff16006b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:47338/0 (socket says 172.21.15.162:47338) 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff1655d0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1600f9b70 0x7ff16006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff165dd1700 1 -- 172.21.15.162:0/3940005366 learned_addr learned my addr 172.21.15.162:0/3940005366 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff1665d2700 1 --2- 172.21.15.162:0/3940005366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1600fa440 0x7ff16006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff165dd1700 1 -- 172.21.15.162:0/3940005366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1600f9b70 msgr2=0x7ff16006b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff165dd1700 1 --2- 172.21.15.162:0/3940005366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1600f9b70 0x7ff16006b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff165dd1700 1 -- 172.21.15.162:0/3940005366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1600fa440 msgr2=0x7ff16006bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff165dd1700 1 --2- 172.21.15.162:0/3940005366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1600fa440 0x7ff16006bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff165dd1700 1 -- 172.21.15.162:0/3940005366 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c005040 con 0x7ff1600f8500 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff1655d0700 1 --2- 172.21.15.162:0/3940005366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1600f9b70 0x7ff16006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff165dd1700 1 --2- 172.21.15.162:0/3940005366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1600f8500 0x7ff16006b170 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 rx=0x7ff15000e990 tx=0x7ff15000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff156ffd700 10 client.?.objecter ms_handle_connect 0x7ff1600f8500 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff156ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff156ffd700 1 -- 172.21.15.162:0/3940005366 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150004680 con 0x7ff1600f8500 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff156ffd700 1 -- 172.21.15.162:0/3940005366 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff15000f040 con 0x7ff1600f8500 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff156ffd700 1 -- 172.21.15.162:0/3940005366 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1500146b0 con 0x7ff1600f8500 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff168035700 1 -- 172.21.15.162:0/3940005366 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1580097c0 con 0x7ff1600f8500 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.218+0000 7ff168035700 10 client.26706.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.219+0000 7ff168035700 1 -- 172.21.15.162:0/3940005366 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff160116f00 con 0x7ff1600f8500 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.219+0000 7ff168035700 1 librados: init done 2024-02-13T00:42:51.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.219+0000 7ff168035700 1 -- 172.21.15.162:0/3940005366 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1600475e0 con 0x7ff1600f8500 2024-02-13T00:42:51.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.220+0000 7ff156ffd700 1 -- 172.21.15.162:0/3940005366 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff150014ae0 con 0x7ff1600f8500 2024-02-13T00:42:51.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.220+0000 7ff156ffd700 1 --2- 172.21.15.162:0/3940005366 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff14c062c90 0x7ff14c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:51.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.220+0000 7ff156ffd700 1 -- 172.21.15.162:0/3940005366 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(215..215 src has 52..215) v4 ==== 9099+0+0 (secure 0 0 0) 0x7ff15007d160 con 0x7ff1600f8500 2024-02-13T00:42:51.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.220+0000 7ff156ffd700 10 client.26706.objecter ms_dispatch 0x7ff160001550 osd_map(215..215 src has 52..215) v4 2024-02-13T00:42:51.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.220+0000 7ff156ffd700 3 client.26706.objecter handle_osd_map got epochs [215,215] > 0 2024-02-13T00:42:51.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.220+0000 7ff156ffd700 3 client.26706.objecter handle_osd_map decoding full epoch 215 2024-02-13T00:42:51.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.220+0000 7ff1655d0700 1 --2- 172.21.15.162:0/3940005366 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff14c062c90 0x7ff14c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:51.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.221+0000 7ff156ffd700 20 client.26706.objecter dump_active .. 0 homeless 2024-02-13T00:42:51.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.221+0000 7ff156ffd700 10 client.26706.objecter ms_handle_connect 0x7ff14c062c90 2024-02-13T00:42:51.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.221+0000 7ff1655d0700 1 --2- 172.21.15.162:0/3940005366 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff14c062c90 0x7ff14c065150 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 rx=0x7ff15c009fd0 tx=0x7ff15c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:51.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.225+0000 7ff156ffd700 1 -- 172.21.15.162:0/3940005366 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff150049f60 con 0x7ff1600f8500 2024-02-13T00:42:51.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:51.450+0000 7ff168035700 1 -- 172.21.15.162:0/3940005366 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 0.0} v 0) v1 -- 0x7ff16006c650 con 0x7ff1600f8500 2024-02-13T00:42:52.000 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:50.348609+0000 mgr.y (mgr.14544) 234 : cluster [DBG] pgmap v293: 292 pgs: 9 active+clean+snaptrim, 32 unknown, 14 active+clean+snaptrim_wait, 237 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 op/s 2024-02-13T00:42:52.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:50.825172+0000 osd.3 (osd.3) 127 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:42:52.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:50.828261+0000 osd.3 (osd.3) 128 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:42:52.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:51.150289+0000 mon.b (mon.1) 292 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:52.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:51.326737+0000 mon.a (mon.0) 1501 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-19"}]': finished 2024-02-13T00:42:52.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:51.326872+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-83985-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:52.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:51.326950+0000 mon.a (mon.0) 1503 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]': finished 2024-02-13T00:42:52.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:51.327007+0000 mon.a (mon.0) 1504 : cluster [DBG] osdmap e216: 8 total, 8 up, 7 in 2024-02-13T00:42:52.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:51.329078+0000 mon.b (mon.1) 293 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-19", "mode": "writeback"}]: dispatch 2024-02-13T00:42:52.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:51.330039+0000 mon.a (mon.0) 1505 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-19", "mode": "writeback"}]: dispatch 2024-02-13T00:42:52.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:51.341894+0000 mon.c (mon.2) 199 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:52.001 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:51.342542+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:52.002 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:51.359411+0000 osd.2 (osd.2) 91 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:42:52.002 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:51.361481+0000 osd.2 (osd.2) 92 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:42:52.002 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:51.451463+0000 mon.a (mon.0) 1507 : audit [INF] from='client.? 172.21.15.162:0/3940005366' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-02-13T00:42:52.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-13T00:42:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:50.348609+0000 mgr.y (mgr.14544) 234 : cluster [DBG] pgmap v293: 292 pgs: 9 active+clean+snaptrim, 32 unknown, 14 active+clean+snaptrim_wait, 237 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 op/s 2024-02-13T00:42:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:50.825172+0000 osd.3 (osd.3) 127 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:42:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:50.828261+0000 osd.3 (osd.3) 128 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:42:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:51.150289+0000 mon.b (mon.1) 292 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:51.326737+0000 mon.a (mon.0) 1501 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-19"}]': finished 2024-02-13T00:42:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:51.326872+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-83985-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:51.326950+0000 mon.a (mon.0) 1503 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]': finished 2024-02-13T00:42:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:51.327007+0000 mon.a (mon.0) 1504 : cluster [DBG] osdmap e216: 8 total, 8 up, 7 in 2024-02-13T00:42:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:51.329078+0000 mon.b (mon.1) 293 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-19", "mode": "writeback"}]: dispatch 2024-02-13T00:42:52.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:51.330039+0000 mon.a (mon.0) 1505 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-19", "mode": "writeback"}]: dispatch 2024-02-13T00:42:52.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:51.341894+0000 mon.c (mon.2) 199 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:52.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:51.342542+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:52.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:51.359411+0000 osd.2 (osd.2) 91 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:42:52.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:51.361481+0000 osd.2 (osd.2) 92 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:42:52.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:51.451463+0000 mon.a (mon.0) 1507 : audit [INF] from='client.? 172.21.15.162:0/3940005366' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-02-13T00:42:52.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:50.348609+0000 mgr.y (mgr.14544) 234 : cluster [DBG] pgmap v293: 292 pgs: 9 active+clean+snaptrim, 32 unknown, 14 active+clean+snaptrim_wait, 237 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 op/s 2024-02-13T00:42:52.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:50.825172+0000 osd.3 (osd.3) 127 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:42:52.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:50.828261+0000 osd.3 (osd.3) 128 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:42:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:51.150289+0000 mon.b (mon.1) 292 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:51.326737+0000 mon.a (mon.0) 1501 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-19"}]': finished 2024-02-13T00:42:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:51.326872+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-83985-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:51.326950+0000 mon.a (mon.0) 1503 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-83977-30"}]': finished 2024-02-13T00:42:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:51.327007+0000 mon.a (mon.0) 1504 : cluster [DBG] osdmap e216: 8 total, 8 up, 7 in 2024-02-13T00:42:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:51.329078+0000 mon.b (mon.1) 293 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-19", "mode": "writeback"}]: dispatch 2024-02-13T00:42:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:51.330039+0000 mon.a (mon.0) 1505 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-19", "mode": "writeback"}]: dispatch 2024-02-13T00:42:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:51.341894+0000 mon.c (mon.2) 199 : audit [INF] from='client.? 172.21.15.162:0/1049303344' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:51.342542+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]: dispatch 2024-02-13T00:42:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:51.359411+0000 osd.2 (osd.2) 91 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:42:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:51.361481+0000 osd.2 (osd.2) 92 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:42:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:51.451463+0000 mon.a (mon.0) 1507 : audit [INF] from='client.? 172.21.15.162:0/3940005366' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-02-13T00:42:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.330+0000 7ff156ffd700 1 -- 172.21.15.162:0/3940005366 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]=0 set osd.7 primary-affinity to 0 (802) v217) v1 ==== 129+0+0 (secure 0 0 0) 0x7ff150050290 con 0x7ff1600f8500 2024-02-13T00:42:52.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.337+0000 7ff168035700 10 librados: watch_flush enter 2024-02-13T00:42:52.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.337+0000 7ff168035700 10 librados: watch_flush exit 2024-02-13T00:42:52.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.337+0000 7ff168035700 20 client.26706.objecter shutdown clearing up homeless session... 2024-02-13T00:42:52.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.337+0000 7ff168035700 10 client.26706.objecter successfully canceled tick 2024-02-13T00:42:52.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.337+0000 7ff168035700 1 -- 172.21.15.162:0/3940005366 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff14c062c90 msgr2=0x7ff14c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:52.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.337+0000 7ff168035700 1 --2- 172.21.15.162:0/3940005366 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff14c062c90 0x7ff14c065150 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 rx=0x7ff15c009fd0 tx=0x7ff15c00b040).stop 2024-02-13T00:42:52.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.338+0000 7ff168035700 1 -- 172.21.15.162:0/3940005366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1600f8500 msgr2=0x7ff16006b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:52.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.338+0000 7ff168035700 1 --2- 172.21.15.162:0/3940005366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1600f8500 0x7ff16006b170 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 rx=0x7ff15000e990 tx=0x7ff15000b040).stop 2024-02-13T00:42:52.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.338+0000 7ff168035700 1 -- 172.21.15.162:0/3940005366 shutdown_connections 2024-02-13T00:42:52.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.338+0000 7ff168035700 1 --2- 172.21.15.162:0/3940005366 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff14c062c90 0x7ff14c065150 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:52.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.339+0000 7ff168035700 1 --2- 172.21.15.162:0/3940005366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1600fa440 0x7ff16006bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:52.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.339+0000 7ff168035700 1 --2- 172.21.15.162:0/3940005366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1600f8500 0x7ff16006b170 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:52.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.339+0000 7ff168035700 1 --2- 172.21.15.162:0/3940005366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1600f9b70 0x7ff16006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:52.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.339+0000 7ff168035700 1 -- 172.21.15.162:0/3940005366 >> 172.21.15.162:0/3940005366 conn(0x7ff1600f38c0 msgr2=0x7ff1600f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:52.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.341+0000 7ff168035700 1 -- 172.21.15.162:0/3940005366 shutdown_connections 2024-02-13T00:42:52.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.342+0000 7ff168035700 1 -- 172.21.15.162:0/3940005366 wait complete. 2024-02-13T00:42:52.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:52.345+0000 7ff168035700 1 librados: shutdown 2024-02-13T00:42:52.372 INFO:teuthology.orchestra.run.smithi162.stderr:set osd.7 primary-affinity to 0 (802) 2024-02-13T00:42:52.815 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:52.817 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:52.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:52.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:52.948 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:53.002 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:52.151629+0000 mon.b (mon.1) 294 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:53.002 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:52.243927+0000 osd.1 (osd.1) 97 : cluster [DBG] 4.e scrub starts 2024-02-13T00:42:53.002 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:52.245746+0000 osd.1 (osd.1) 98 : cluster [DBG] 4.e scrub ok 2024-02-13T00:42:53.002 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:52.324889+0000 mon.a (mon.0) 1508 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:42:53.002 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:52.330966+0000 mon.a (mon.0) 1509 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-19", "mode": "writeback"}]': finished 2024-02-13T00:42:53.003 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:52.331051+0000 mon.a (mon.0) 1510 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]': finished 2024-02-13T00:42:53.003 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:52.331139+0000 mon.a (mon.0) 1511 : audit [INF] from='client.? 172.21.15.162:0/3940005366' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-02-13T00:42:53.003 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:52.331192+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e217: 8 total, 8 up, 7 in 2024-02-13T00:42:53.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:52.151629+0000 mon.b (mon.1) 294 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:52.243927+0000 osd.1 (osd.1) 97 : cluster [DBG] 4.e scrub starts 2024-02-13T00:42:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:52.245746+0000 osd.1 (osd.1) 98 : cluster [DBG] 4.e scrub ok 2024-02-13T00:42:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:52.324889+0000 mon.a (mon.0) 1508 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:42:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:52.330966+0000 mon.a (mon.0) 1509 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-19", "mode": "writeback"}]': finished 2024-02-13T00:42:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:52.331051+0000 mon.a (mon.0) 1510 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]': finished 2024-02-13T00:42:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:52.331139+0000 mon.a (mon.0) 1511 : audit [INF] from='client.? 172.21.15.162:0/3940005366' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-02-13T00:42:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:52.331192+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e217: 8 total, 8 up, 7 in 2024-02-13T00:42:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:52.376181+0000 mon.c (mon.2) 200 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:52.389015+0000 osd.0 (osd.0) 81 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:42:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:52.391902+0000 osd.0 (osd.0) 82 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:42:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:52.620108+0000 mon.a (mon.0) 1513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:53.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:52.620616+0000 mon.c (mon.2) 201 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:52.620865+0000 mon.a (mon.0) 1514 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:52.621233+0000 mon.c (mon.2) 202 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-83977-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:52.621467+0000 mon.a (mon.0) 1515 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-83977-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:53.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:52.151629+0000 mon.b (mon.1) 294 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:53.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:52.243927+0000 osd.1 (osd.1) 97 : cluster [DBG] 4.e scrub starts 2024-02-13T00:42:53.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:52.245746+0000 osd.1 (osd.1) 98 : cluster [DBG] 4.e scrub ok 2024-02-13T00:42:53.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:52.324889+0000 mon.a (mon.0) 1508 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:42:53.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:52.330966+0000 mon.a (mon.0) 1509 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-19", "mode": "writeback"}]': finished 2024-02-13T00:42:53.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:52.331051+0000 mon.a (mon.0) 1510 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-83977-30"}]': finished 2024-02-13T00:42:53.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:52.331139+0000 mon.a (mon.0) 1511 : audit [INF] from='client.? 172.21.15.162:0/3940005366' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-02-13T00:42:53.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:52.331192+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e217: 8 total, 8 up, 7 in 2024-02-13T00:42:53.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:52.376181+0000 mon.c (mon.2) 200 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:53.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:52.389015+0000 osd.0 (osd.0) 81 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:42:53.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:52.391902+0000 osd.0 (osd.0) 82 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:42:53.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:52.620108+0000 mon.a (mon.0) 1513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:53.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:52.620616+0000 mon.c (mon.2) 201 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:53.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:52.620865+0000 mon.a (mon.0) 1514 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:53.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:52.621233+0000 mon.c (mon.2) 202 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-83977-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:53.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:52.621467+0000 mon.a (mon.0) 1515 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-83977-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:53.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:52.376181+0000 mon.c (mon.2) 200 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:52.389015+0000 osd.0 (osd.0) 81 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:42:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:52.391902+0000 osd.0 (osd.0) 82 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:42:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:52.620108+0000 mon.a (mon.0) 1513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:52.620616+0000 mon.c (mon.2) 201 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:52.620865+0000 mon.a (mon.0) 1514 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:52.621233+0000 mon.c (mon.2) 202 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-83977-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:52.621467+0000 mon.a (mon.0) 1515 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-83977-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:52.349107+0000 mgr.y (mgr.14544) 235 : cluster [DBG] pgmap v296: 260 pgs: 9 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 237 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:52.444668+0000 osd.5 (osd.5) 95 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:52.447694+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:53.061950+0000 mon.c (mon.2) 203 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:53.152511+0000 mon.b (mon.1) 295 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:53.195404+0000 osd.1 (osd.1) 99 : cluster [DBG] 6.1 scrub starts 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:53.198287+0000 osd.1 (osd.1) 100 : cluster [DBG] 6.1 scrub ok 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:53.333860+0000 mon.a (mon.0) 1516 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-83977-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:53.333945+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e218: 8 total, 8 up, 7 in 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:53.335041+0000 mon.c (mon.2) 204 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-83977-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:53.336031+0000 mon.a (mon.0) 1518 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-83977-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:53.345905+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? 172.21.15.162:0/3835703849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-83985-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:53.396655+0000 mon.b (mon.1) 296 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:53.397189+0000 mon.a (mon.0) 1520 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:53.403799+0000 mon.a (mon.0) 1521 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:53.442388+0000 osd.2 (osd.2) 93 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:42:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:53.443273+0000 osd.2 (osd.2) 94 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:42:54.307 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:52.349107+0000 mgr.y (mgr.14544) 235 : cluster [DBG] pgmap v296: 260 pgs: 9 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 237 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:42:54.307 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:52.444668+0000 osd.5 (osd.5) 95 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:42:54.307 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:52.447694+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:42:54.307 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:53.061950+0000 mon.c (mon.2) 203 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:42:54.307 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:53.152511+0000 mon.b (mon.1) 295 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:54.307 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:53.195404+0000 osd.1 (osd.1) 99 : cluster [DBG] 6.1 scrub starts 2024-02-13T00:42:54.307 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:53.198287+0000 osd.1 (osd.1) 100 : cluster [DBG] 6.1 scrub ok 2024-02-13T00:42:54.307 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:53.333860+0000 mon.a (mon.0) 1516 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-83977-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:54.307 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:53.333945+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e218: 8 total, 8 up, 7 in 2024-02-13T00:42:54.308 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:53.335041+0000 mon.c (mon.2) 204 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-83977-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:54.308 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:53.336031+0000 mon.a (mon.0) 1518 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-83977-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:54.308 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:53.345905+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? 172.21.15.162:0/3835703849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-83985-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:54.308 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:53.396655+0000 mon.b (mon.1) 296 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:54.308 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:53.397189+0000 mon.a (mon.0) 1520 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:54.308 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:53.403799+0000 mon.a (mon.0) 1521 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:54.308 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:53.442388+0000 osd.2 (osd.2) 93 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:42:54.308 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:53.443273+0000 osd.2 (osd.2) 94 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:42:54.308 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:52.349107+0000 mgr.y (mgr.14544) 235 : cluster [DBG] pgmap v296: 260 pgs: 9 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 237 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:42:54.308 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:52.444668+0000 osd.5 (osd.5) 95 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:42:54.308 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:52.447694+0000 osd.5 (osd.5) 96 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:53.061950+0000 mon.c (mon.2) 203 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:53.152511+0000 mon.b (mon.1) 295 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:53.195404+0000 osd.1 (osd.1) 99 : cluster [DBG] 6.1 scrub starts 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:53.198287+0000 osd.1 (osd.1) 100 : cluster [DBG] 6.1 scrub ok 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:53.333860+0000 mon.a (mon.0) 1516 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-83977-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:53.333945+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e218: 8 total, 8 up, 7 in 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:53.335041+0000 mon.c (mon.2) 204 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-83977-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:53.336031+0000 mon.a (mon.0) 1518 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-83977-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:53.345905+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? 172.21.15.162:0/3835703849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-83985-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:53.396655+0000 mon.b (mon.1) 296 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:53.397189+0000 mon.a (mon.0) 1520 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:53.403799+0000 mon.a (mon.0) 1521 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:53.442388+0000 osd.2 (osd.2) 93 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:42:54.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:53.443273+0000 osd.2 (osd.2) 94 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:42:54.398 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:54.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:54.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.501+0000 7fd44b192700 1 -- 172.21.15.162:0/2470203444 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4440fcb30 msgr2=0x7fd4440fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.501+0000 7fd44b192700 1 --2- 172.21.15.162:0/2470203444 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4440fcb30 0x7fd4440fcf10 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 rx=0x7fd438004870 tx=0x7fd438009cb0).stop 2024-02-13T00:42:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.501+0000 7fd44b192700 1 -- 172.21.15.162:0/2470203444 shutdown_connections 2024-02-13T00:42:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.501+0000 7fd44b192700 1 --2- 172.21.15.162:0/2470203444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4440f73a0 0x7fd4441039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.501+0000 7fd44b192700 1 --2- 172.21.15.162:0/2470203444 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4440fcb30 0x7fd4440fcf10 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.501+0000 7fd44b192700 1 --2- 172.21.15.162:0/2470203444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd4440fd490 0x7fd4440f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.501+0000 7fd44b192700 1 -- 172.21.15.162:0/2470203444 >> 172.21.15.162:0/2470203444 conn(0x7fd44406eb90 msgr2=0x7fd4440f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.501+0000 7fd44b192700 1 -- 172.21.15.162:0/2470203444 shutdown_connections 2024-02-13T00:42:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.501+0000 7fd44b192700 1 -- 172.21.15.162:0/2470203444 wait complete. 2024-02-13T00:42:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44b192700 1 librados: starting msgr at 2024-02-13T00:42:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44b192700 1 librados: starting objecter 2024-02-13T00:42:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44b192700 1 Processor -- start 2024-02-13T00:42:54.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44b192700 1 -- start start 2024-02-13T00:42:54.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44b192700 1 librados: setting wanted keys 2024-02-13T00:42:54.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44b192700 1 librados: calling monclient init 2024-02-13T00:42:54.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44b192700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd4440f73a0 0x7fd44410e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:54.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44b192700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4440fcb30 0x7fd44410e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:54.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44b192700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4440fd490 0x7fd444108320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:54.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44b192700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd444108810 con 0x7fd4440f73a0 2024-02-13T00:42:54.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44b192700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd444108980 con 0x7fd4440fd490 2024-02-13T00:42:54.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44b192700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd444108b40 con 0x7fd4440fcb30 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd443fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4440fcb30 0x7fd44410e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd443fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4440fcb30 0x7fd44410e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:52646/0 (socket says 172.21.15.162:52646) 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd443fff700 1 -- 172.21.15.162:0/4083791199 learned_addr learned my addr 172.21.15.162:0/4083791199 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd448f2e700 1 --2- 172.21.15.162:0/4083791199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd4440f73a0 0x7fd44410e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.502+0000 7fd44972f700 1 --2- 172.21.15.162:0/4083791199 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4440fd490 0x7fd444108320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd443fff700 1 -- 172.21.15.162:0/4083791199 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4440fd490 msgr2=0x7fd444108320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd443fff700 1 --2- 172.21.15.162:0/4083791199 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4440fd490 0x7fd444108320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd443fff700 1 -- 172.21.15.162:0/4083791199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd4440f73a0 msgr2=0x7fd44410e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd443fff700 1 --2- 172.21.15.162:0/4083791199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd4440f73a0 0x7fd44410e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd443fff700 1 -- 172.21.15.162:0/4083791199 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd438005040 con 0x7fd4440fcb30 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd443fff700 1 --2- 172.21.15.162:0/4083791199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4440fcb30 0x7fd44410e670 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7fd43000e970 tx=0x7fd43000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd441ffb700 10 client.?.objecter ms_handle_connect 0x7fd4440fcb30 2024-02-13T00:42:54.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd441ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:54.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430004660 con 0x7fd4440fcb30 2024-02-13T00:42:54.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd43000f040 con 0x7fd4440fcb30 2024-02-13T00:42:54.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd444108e20 con 0x7fd4440fcb30 2024-02-13T00:42:54.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430014790 con 0x7fd4440fcb30 2024-02-13T00:42:54.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd44b192700 10 client.36830.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:54.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4441091a0 con 0x7fd4440fcb30 2024-02-13T00:42:54.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.503+0000 7fd44b192700 1 librados: init done 2024-02-13T00:42:54.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.504+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd444105fe0 con 0x7fd4440fcb30 2024-02-13T00:42:54.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.504+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fd4300090d0 con 0x7fd4440fcb30 2024-02-13T00:42:54.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd441ffb700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd434062c90 0x7fd434065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:54.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(219..219 src has 52..219) v4 ==== 9110+0+0 (secure 0 0 0) 0x7fd43007d170 con 0x7fd4440fcb30 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 osd_map(219..219 src has 52..219) v4 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd441ffb700 3 client.36830.objecter handle_osd_map got epochs [219,219] > 0 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd441ffb700 3 client.36830.objecter handle_osd_map decoding full epoch 219 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd448f2e700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd434062c90 0x7fd434065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd441ffb700 20 client.36830.objecter dump_active .. 0 homeless 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v219) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd430084070 con 0x7fd4440fcb30 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44b192700 10 client.36830.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 1 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd4440ffdf0 0x7fd4441001f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd441ffb700 10 client.36830.objecter ms_handle_connect 0x7fd434062c90 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444108fe0 osd=0 3 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd448f2e700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd434062c90 0x7fd434065150 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 rx=0x7fd4380043d0 tx=0x7fd43800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444108fe0 osd=0 3 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444108fe0 osd=0 3 2024-02-13T00:42:54.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 1 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd444108fe0 osd=0 3 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 0 1 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444108fe0 osd=0 4 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44b192700 10 client.36830.objecter _send_command 1 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd444109800 con 0x7fd4440ffdf0 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.505+0000 7fd44972f700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd4440ffdf0 0x7fd4441001f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.506+0000 7fd441ffb700 10 client.36830.objecter ms_handle_connect 0x7fd4440ffdf0 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.506+0000 7fd44972f700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd4440ffdf0 0x7fd4441001f0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.506+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd444109800 con 0x7fd4440ffdf0 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.506+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.506+0000 7fd441ffb700 10 client.36830.objecter _finish_command 1 = system:0 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.506+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd444108fe0 osd=0 4 2024-02-13T00:42:54.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.506+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 0 1 2024-02-13T00:42:54.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.521+0000 7fd44b192700 10 client.36830.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:54.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.521+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 2 2024-02-13T00:42:54.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.521+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444108fe0 osd=0 3 2024-02-13T00:42:54.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.521+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444108fe0 osd=0 3 2024-02-13T00:42:54.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.521+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444108fe0 osd=0 3 2024-02-13T00:42:54.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.521+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 2 2024-02-13T00:42:54.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.521+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd444108fe0 osd=0 3 2024-02-13T00:42:54.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.521+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 0 2 2024-02-13T00:42:54.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.521+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444108fe0 osd=0 4 2024-02-13T00:42:54.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.521+0000 7fd44b192700 10 client.36830.objecter _send_command 2 2024-02-13T00:42:54.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.521+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd444027410 con 0x7fd4440ffdf0 2024-02-13T00:42:54.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd444027410 con 0x7fd4440ffdf0 2024-02-13T00:42:54.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:54.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd441ffb700 10 client.36830.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd444108fe0 osd=0 4 2024-02-13T00:42:54.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 0 2 2024-02-13T00:42:54.526 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:42:54.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd44b192700 10 client.36830.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:54.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 3 2024-02-13T00:42:54.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6805/150599780 conn(0x7fd444059990 0x7fd444059db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:54.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444028760 osd=1 3 2024-02-13T00:42:54.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444028760 osd=1 3 2024-02-13T00:42:54.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444028760 osd=1 3 2024-02-13T00:42:54.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 3 2024-02-13T00:42:54.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd444028760 osd=1 3 2024-02-13T00:42:54.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 1 3 2024-02-13T00:42:54.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444028760 osd=1 4 2024-02-13T00:42:54.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd44b192700 10 client.36830.objecter _send_command 3 2024-02-13T00:42:54.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd44411b9b0 con 0x7fd444059990 2024-02-13T00:42:54.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.523+0000 7fd448f2e700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6805/150599780 conn(0x7fd444059990 0x7fd444059db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:54.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.524+0000 7fd441ffb700 10 client.36830.objecter ms_handle_connect 0x7fd444059990 2024-02-13T00:42:54.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.524+0000 7fd448f2e700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6805/150599780 conn(0x7fd444059990 0x7fd444059db0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:54.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.524+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd44411b9b0 con 0x7fd444059990 2024-02-13T00:42:54.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.524+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:42:54.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.524+0000 7fd441ffb700 10 client.36830.objecter _finish_command 3 = system:0 2024-02-13T00:42:54.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.524+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd444028760 osd=1 4 2024-02-13T00:42:54.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.524+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 1 3 2024-02-13T00:42:54.530 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:54.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.539+0000 7fd44b192700 10 client.36830.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:54.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.539+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 4 2024-02-13T00:42:54.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.539+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444028760 osd=1 3 2024-02-13T00:42:54.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.539+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444028760 osd=1 3 2024-02-13T00:42:54.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.539+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444028760 osd=1 3 2024-02-13T00:42:54.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.539+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 4 2024-02-13T00:42:54.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.539+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd444028760 osd=1 3 2024-02-13T00:42:54.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.539+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 1 4 2024-02-13T00:42:54.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.539+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444028760 osd=1 4 2024-02-13T00:42:54.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.539+0000 7fd44b192700 10 client.36830.objecter _send_command 4 2024-02-13T00:42:54.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.539+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd444059400 con 0x7fd444059990 2024-02-13T00:42:54.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd444059400 con 0x7fd444059990 2024-02-13T00:42:54.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:54.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd441ffb700 10 client.36830.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd444028760 osd=1 4 2024-02-13T00:42:54.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 1 4 2024-02-13T00:42:54.545 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:42:54.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd44b192700 10 client.36830.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 5 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd4440282c0 0x7fd44411a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444059810 osd=2 3 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444059810 osd=2 3 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444059810 osd=2 3 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 5 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd444059810 osd=2 3 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 2 5 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444059810 osd=2 4 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd44b192700 10 client.36830.objecter _send_command 5 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd44411a8c0 con 0x7fd4440282c0 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.541+0000 7fd443fff700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd4440282c0 0x7fd44411a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:54.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.542+0000 7fd441ffb700 10 client.36830.objecter ms_handle_connect 0x7fd4440282c0 2024-02-13T00:42:54.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.542+0000 7fd443fff700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd4440282c0 0x7fd44411a3d0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:54.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.542+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd44411a8c0 con 0x7fd4440282c0 2024-02-13T00:42:54.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.542+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:42:54.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.542+0000 7fd441ffb700 10 client.36830.objecter _finish_command 5 = system:0 2024-02-13T00:42:54.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.542+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd444059810 osd=2 4 2024-02-13T00:42:54.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.542+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 2 5 2024-02-13T00:42:54.549 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:54.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:54.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.557+0000 7fd44b192700 10 client.36830.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:54.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.557+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 6 2024-02-13T00:42:54.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.557+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444059810 osd=2 3 2024-02-13T00:42:54.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.557+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444059810 osd=2 3 2024-02-13T00:42:54.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.557+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444059810 osd=2 3 2024-02-13T00:42:54.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.557+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 6 2024-02-13T00:42:54.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.557+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd444059810 osd=2 3 2024-02-13T00:42:54.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.557+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 2 6 2024-02-13T00:42:54.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.557+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444059810 osd=2 4 2024-02-13T00:42:54.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.557+0000 7fd44b192700 10 client.36830.objecter _send_command 6 2024-02-13T00:42:54.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.557+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd444027880 con 0x7fd4440282c0 2024-02-13T00:42:54.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd444027880 con 0x7fd4440282c0 2024-02-13T00:42:54.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:54.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd441ffb700 10 client.36830.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd444059810 osd=2 4 2024-02-13T00:42:54.562 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 2 6 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44b192700 10 client.36830.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 7 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6813/604971011 conn(0x7fd44411db30 0x7fd44411ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44411b2b0 osd=3 3 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411b2b0 osd=3 3 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44411b2b0 osd=3 3 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 7 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd44411b2b0 osd=3 3 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 3 7 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411b2b0 osd=3 4 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44b192700 10 client.36830.objecter _send_command 7 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd444122b70 con 0x7fd44411db30 2024-02-13T00:42:54.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.558+0000 7fd44972f700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6813/604971011 conn(0x7fd44411db30 0x7fd44411ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:54.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.559+0000 7fd441ffb700 10 client.36830.objecter ms_handle_connect 0x7fd44411db30 2024-02-13T00:42:54.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.559+0000 7fd44972f700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6813/604971011 conn(0x7fd44411db30 0x7fd44411ff20 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:54.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.559+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd444122b70 con 0x7fd44411db30 2024-02-13T00:42:54.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.559+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:42:54.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.559+0000 7fd441ffb700 10 client.36830.objecter _finish_command 7 = system:0 2024-02-13T00:42:54.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.559+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd44411b2b0 osd=3 4 2024-02-13T00:42:54.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.559+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 3 7 2024-02-13T00:42:54.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.579+0000 7fd44b192700 10 client.36830.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:54.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.579+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 8 2024-02-13T00:42:54.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.579+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44411b2b0 osd=3 3 2024-02-13T00:42:54.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.579+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411b2b0 osd=3 3 2024-02-13T00:42:54.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.579+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44411b2b0 osd=3 3 2024-02-13T00:42:54.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.579+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 8 2024-02-13T00:42:54.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.579+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd44411b2b0 osd=3 3 2024-02-13T00:42:54.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.579+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 3 8 2024-02-13T00:42:54.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.579+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411b2b0 osd=3 4 2024-02-13T00:42:54.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.579+0000 7fd44b192700 10 client.36830.objecter _send_command 8 2024-02-13T00:42:54.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.579+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd44411ad20 con 0x7fd44411db30 2024-02-13T00:42:54.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd44411ad20 con 0x7fd44411db30 2024-02-13T00:42:54.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:54.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd441ffb700 10 client.36830.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd44411b2b0 osd=3 4 2024-02-13T00:42:54.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 3 8 2024-02-13T00:42:54.584 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:42:54.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd44b192700 10 client.36830.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:54.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 9 2024-02-13T00:42:54.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd444119f30 0x7fd444121590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:54.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44411aef0 osd=4 3 2024-02-13T00:42:54.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411aef0 osd=4 3 2024-02-13T00:42:54.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44411aef0 osd=4 3 2024-02-13T00:42:54.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 9 2024-02-13T00:42:54.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd44411aef0 osd=4 3 2024-02-13T00:42:54.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 4 9 2024-02-13T00:42:54.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411aef0 osd=4 4 2024-02-13T00:42:54.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd44b192700 10 client.36830.objecter _send_command 9 2024-02-13T00:42:54.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.580+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd44411b060 con 0x7fd444119f30 2024-02-13T00:42:54.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.581+0000 7fd448f2e700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd444119f30 0x7fd444121590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:54.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.581+0000 7fd441ffb700 10 client.36830.objecter ms_handle_connect 0x7fd444119f30 2024-02-13T00:42:54.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.581+0000 7fd448f2e700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd444119f30 0x7fd444121590 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:54.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.582+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd44411b060 con 0x7fd444119f30 2024-02-13T00:42:54.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.582+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:42:54.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.582+0000 7fd441ffb700 10 client.36830.objecter _finish_command 9 = system:0 2024-02-13T00:42:54.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.582+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd44411aef0 osd=4 4 2024-02-13T00:42:54.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.582+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 4 9 2024-02-13T00:42:54.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.605+0000 7fd44b192700 10 client.36830.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:54.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.605+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 10 2024-02-13T00:42:54.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.605+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44411aef0 osd=4 3 2024-02-13T00:42:54.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.605+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411aef0 osd=4 3 2024-02-13T00:42:54.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.605+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44411aef0 osd=4 3 2024-02-13T00:42:54.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.605+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 10 2024-02-13T00:42:54.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.605+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd44411aef0 osd=4 3 2024-02-13T00:42:54.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.605+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 4 10 2024-02-13T00:42:54.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.605+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411aef0 osd=4 4 2024-02-13T00:42:54.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.605+0000 7fd44b192700 10 client.36830.objecter _send_command 10 2024-02-13T00:42:54.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.605+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd44411b060 con 0x7fd444119f30 2024-02-13T00:42:54.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd44411b060 con 0x7fd444119f30 2024-02-13T00:42:54.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:54.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd441ffb700 10 client.36830.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd44411aef0 osd=4 4 2024-02-13T00:42:54.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 4 10 2024-02-13T00:42:54.611 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:42:54.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd44b192700 10 client.36830.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:54.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 11 2024-02-13T00:42:54.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6804/2212296748 conn(0x7fd444121ff0 0x7fd444122430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:54.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44411b060 osd=5 3 2024-02-13T00:42:54.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411b060 osd=5 3 2024-02-13T00:42:54.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44411b060 osd=5 3 2024-02-13T00:42:54.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 11 2024-02-13T00:42:54.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd44411b060 osd=5 3 2024-02-13T00:42:54.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 5 11 2024-02-13T00:42:54.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411b060 osd=5 4 2024-02-13T00:42:54.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd44b192700 10 client.36830.objecter _send_command 11 2024-02-13T00:42:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.607+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd444122920 con 0x7fd444121ff0 2024-02-13T00:42:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.608+0000 7fd443fff700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6804/2212296748 conn(0x7fd444121ff0 0x7fd444122430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.608+0000 7fd441ffb700 10 client.36830.objecter ms_handle_connect 0x7fd444121ff0 2024-02-13T00:42:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.608+0000 7fd443fff700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6804/2212296748 conn(0x7fd444121ff0 0x7fd444122430 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.609+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd444122920 con 0x7fd444121ff0 2024-02-13T00:42:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.609+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:42:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.609+0000 7fd441ffb700 10 client.36830.objecter _finish_command 11 = system:0 2024-02-13T00:42:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.609+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd44411b060 osd=5 4 2024-02-13T00:42:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.609+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 5 11 2024-02-13T00:42:54.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.623+0000 7fd44b192700 10 client.36830.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:54.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.623+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 12 2024-02-13T00:42:54.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.623+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44411b060 osd=5 3 2024-02-13T00:42:54.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.623+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411b060 osd=5 3 2024-02-13T00:42:54.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.623+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44411b060 osd=5 3 2024-02-13T00:42:54.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.623+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 12 2024-02-13T00:42:54.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.623+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd44411b060 osd=5 3 2024-02-13T00:42:54.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.623+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 5 12 2024-02-13T00:42:54.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.623+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411b060 osd=5 4 2024-02-13T00:42:54.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.623+0000 7fd44b192700 10 client.36830.objecter _send_command 12 2024-02-13T00:42:54.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.623+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd444122920 con 0x7fd444121ff0 2024-02-13T00:42:54.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd444122920 con 0x7fd444121ff0 2024-02-13T00:42:54.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:54.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd441ffb700 10 client.36830.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd44411b060 osd=5 4 2024-02-13T00:42:54.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 5 12 2024-02-13T00:42:54.628 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:42:54.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44b192700 10 client.36830.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:54.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 13 2024-02-13T00:42:54.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd444129610 0x7fd44412ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:54.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444122920 osd=6 3 2024-02-13T00:42:54.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444122920 osd=6 3 2024-02-13T00:42:54.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444122920 osd=6 3 2024-02-13T00:42:54.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 13 2024-02-13T00:42:54.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd444122920 osd=6 3 2024-02-13T00:42:54.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 6 13 2024-02-13T00:42:54.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444122920 osd=6 4 2024-02-13T00:42:54.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44b192700 10 client.36830.objecter _send_command 13 2024-02-13T00:42:54.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd44412e7a0 con 0x7fd444129610 2024-02-13T00:42:54.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.625+0000 7fd44972f700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd444129610 0x7fd44412ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:54.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.626+0000 7fd441ffb700 10 client.36830.objecter ms_handle_connect 0x7fd444129610 2024-02-13T00:42:54.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.626+0000 7fd44972f700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd444129610 0x7fd44412ba00 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:54.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.626+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd44412e7a0 con 0x7fd444129610 2024-02-13T00:42:54.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.626+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:42:54.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.626+0000 7fd441ffb700 10 client.36830.objecter _finish_command 13 = system:0 2024-02-13T00:42:54.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.626+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd444122920 osd=6 4 2024-02-13T00:42:54.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.626+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 6 13 2024-02-13T00:42:54.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.641+0000 7fd44b192700 10 client.36830.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:54.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.641+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 14 2024-02-13T00:42:54.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.641+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444122920 osd=6 3 2024-02-13T00:42:54.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.641+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444122920 osd=6 3 2024-02-13T00:42:54.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.641+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd444122920 osd=6 3 2024-02-13T00:42:54.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.641+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 14 2024-02-13T00:42:54.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.641+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd444122920 osd=6 3 2024-02-13T00:42:54.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.641+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 6 14 2024-02-13T00:42:54.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.641+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444122920 osd=6 4 2024-02-13T00:42:54.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.641+0000 7fd44b192700 10 client.36830.objecter _send_command 14 2024-02-13T00:42:54.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.641+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd44412e7a0 con 0x7fd444129610 2024-02-13T00:42:54.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.642+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd44412e7a0 con 0x7fd444129610 2024-02-13T00:42:54.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.642+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:54.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd441ffb700 10 client.36830.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd444122920 osd=6 4 2024-02-13T00:42:54.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 6 14 2024-02-13T00:42:54.646 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd44b192700 10 client.36830.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 15 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd44412dc40 0x7fd44412e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44412d5c0 osd=7 3 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44412d5c0 osd=7 3 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44412d5c0 osd=7 3 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 15 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd44412d5c0 osd=7 3 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 7 15 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44412d5c0 osd=7 4 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd44b192700 10 client.36830.objecter _send_command 15 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd444131ba0 con 0x7fd44412dc40 2024-02-13T00:42:54.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd448f2e700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd44412dc40 0x7fd44412e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:54.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd441ffb700 10 client.36830.objecter ms_handle_connect 0x7fd44412dc40 2024-02-13T00:42:54.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.643+0000 7fd448f2e700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd44412dc40 0x7fd44412e0a0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:54.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.644+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd444131ba0 con 0x7fd44412dc40 2024-02-13T00:42:54.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.644+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:42:54.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.644+0000 7fd441ffb700 10 client.36830.objecter _finish_command 15 = system:0 2024-02-13T00:42:54.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.644+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd44412d5c0 osd=7 4 2024-02-13T00:42:54.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.644+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 7 15 2024-02-13T00:42:54.652 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:54.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.658+0000 7fd44b192700 10 client.36830.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:42:54.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.658+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign -1 16 2024-02-13T00:42:54.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.658+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44412d5c0 osd=7 3 2024-02-13T00:42:54.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.658+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44412d5c0 osd=7 3 2024-02-13T00:42:54.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.658+0000 7fd44b192700 20 client.36830.objecter _get_session s=0x7fd44412d5c0 osd=7 3 2024-02-13T00:42:54.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.658+0000 7fd44b192700 15 client.36830.objecter _session_command_op_remove -1 16 2024-02-13T00:42:54.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.658+0000 7fd44b192700 20 client.36830.objecter get_session s=0x7fd44412d5c0 osd=7 3 2024-02-13T00:42:54.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.658+0000 7fd44b192700 15 client.36830.objecter _session_command_op_assign 7 16 2024-02-13T00:42:54.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.658+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44412d5c0 osd=7 4 2024-02-13T00:42:54.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.658+0000 7fd44b192700 10 client.36830.objecter _send_command 16 2024-02-13T00:42:54.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.658+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd44412bfb0 con 0x7fd44412dc40 2024-02-13T00:42:54.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd441ffb700 1 -- 172.21.15.162:0/4083791199 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd44412bfb0 con 0x7fd44412dc40 2024-02-13T00:42:54.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd441ffb700 10 client.36830.objecter ms_dispatch 0x7fd444001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:42:54.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd441ffb700 10 client.36830.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd441ffb700 20 client.36830.objecter put_session s=0x7fd44412d5c0 osd=7 4 2024-02-13T00:42:54.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd441ffb700 15 client.36830.objecter _session_command_op_remove 7 16 2024-02-13T00:42:54.664 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:42:54.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 10 librados: watch_flush enter 2024-02-13T00:42:54.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 10 librados: watch_flush exit 2024-02-13T00:42:54.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 10 client.36830.objecter close_session for osd.0 2024-02-13T00:42:54.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd4440ffdf0 msgr2=0x7fd4441001f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd4440ffdf0 0x7fd4441001f0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444108fe0 osd=0 1 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 10 client.36830.objecter close_session for osd.1 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6805/150599780 conn(0x7fd444059990 msgr2=0x7fd444059db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6805/150599780 conn(0x7fd444059990 0x7fd444059db0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444028760 osd=1 1 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 10 client.36830.objecter close_session for osd.2 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd4440282c0 msgr2=0x7fd44411a3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd4440282c0 0x7fd44411a3d0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444059810 osd=2 1 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 10 client.36830.objecter close_session for osd.3 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6813/604971011 conn(0x7fd44411db30 msgr2=0x7fd44411ff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6813/604971011 conn(0x7fd44411db30 0x7fd44411ff20 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411b2b0 osd=3 1 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 10 client.36830.objecter close_session for osd.4 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd444119f30 msgr2=0x7fd444121590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44972f700 1 -- 172.21.15.162:0/4083791199 reap_dead start 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd444119f30 0x7fd444121590 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44972f700 1 -- 172.21.15.162:0/4083791199 reap_dead start 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411aef0 osd=4 1 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 10 client.36830.objecter close_session for osd.5 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6804/2212296748 conn(0x7fd444121ff0 msgr2=0x7fd444122430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6804/2212296748 conn(0x7fd444121ff0 0x7fd444122430 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44411b060 osd=5 1 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 10 client.36830.objecter close_session for osd.6 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd444129610 msgr2=0x7fd44412ba00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd444129610 0x7fd44412ba00 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd444122920 osd=6 1 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 10 client.36830.objecter close_session for osd.7 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd44412dc40 msgr2=0x7fd44412e0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd44412dc40 0x7fd44412e0a0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 20 client.36830.objecter put_session s=0x7fd44412d5c0 osd=7 1 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 20 client.36830.objecter shutdown clearing up homeless session... 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.660+0000 7fd44b192700 10 client.36830.objecter successfully canceled tick 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.661+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd434062c90 msgr2=0x7fd434065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.661+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd434062c90 0x7fd434065150 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 rx=0x7fd4380043d0 tx=0x7fd43800b040).stop 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.661+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4440fcb30 msgr2=0x7fd44410e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.661+0000 7fd44b192700 1 --2- 172.21.15.162:0/4083791199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd4440fcb30 0x7fd44410e670 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7fd43000e970 tx=0x7fd43000b040).stop 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.661+0000 7fd44972f700 1 -- 172.21.15.162:0/4083791199 reap_dead start 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.661+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 shutdown_connections 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.661+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 >> 172.21.15.162:0/4083791199 conn(0x7fd44406eb90 msgr2=0x7fd444060ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.661+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 shutdown_connections 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.661+0000 7fd44b192700 1 -- 172.21.15.162:0/4083791199 wait complete. 2024-02-13T00:42:54.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:54.661+0000 7fd44b192700 1 librados: shutdown 2024-02-13T00:42:54.669 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.669 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.669 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.669 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.669 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.669 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.669 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.669 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:42:54.679 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:54.697 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:54.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:54.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:54.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:42:54.830 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:55.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:53.878128+0000 osd.3 (osd.3) 129 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:42:55.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:53.881589+0000 osd.3 (osd.3) 130 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:42:55.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:54.153483+0000 mon.b (mon.1) 297 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:55.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:54.173144+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:42:55.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:54.174946+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:42:55.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:54.505830+0000 mon.c (mon.2) 205 : audit [DBG] from='client.? 172.21.15.162:0/4083791199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:55.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:54.584011+0000 mon.a (mon.0) 1522 : audit [INF] from='client.? 172.21.15.162:0/3835703849' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-83985-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:55.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:54.584209+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:42:55.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:54.584281+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e219: 8 total, 8 up, 7 in 2024-02-13T00:42:55.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:54.584708+0000 mon.b (mon.1) 298 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19"}]: dispatch 2024-02-13T00:42:55.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:54.587551+0000 mon.a (mon.0) 1525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19"}]: dispatch 2024-02-13T00:42:55.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:53.878128+0000 osd.3 (osd.3) 129 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:42:55.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:53.881589+0000 osd.3 (osd.3) 130 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:42:55.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:54.153483+0000 mon.b (mon.1) 297 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:55.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:54.173144+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:42:55.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:54.174946+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:42:55.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:54.505830+0000 mon.c (mon.2) 205 : audit [DBG] from='client.? 172.21.15.162:0/4083791199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:55.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:54.584011+0000 mon.a (mon.0) 1522 : audit [INF] from='client.? 172.21.15.162:0/3835703849' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-83985-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:55.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:54.584209+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:42:55.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:54.584281+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e219: 8 total, 8 up, 7 in 2024-02-13T00:42:55.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:54.584708+0000 mon.b (mon.1) 298 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19"}]: dispatch 2024-02-13T00:42:55.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:54.587551+0000 mon.a (mon.0) 1525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19"}]: dispatch 2024-02-13T00:42:55.279 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:42:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:55.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:42:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:53.878128+0000 osd.3 (osd.3) 129 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:42:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:53.881589+0000 osd.3 (osd.3) 130 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:42:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:54.153483+0000 mon.b (mon.1) 297 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:54.173144+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:42:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:54.174946+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:42:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:54.505830+0000 mon.c (mon.2) 205 : audit [DBG] from='client.? 172.21.15.162:0/4083791199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:42:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:54.584011+0000 mon.a (mon.0) 1522 : audit [INF] from='client.? 172.21.15.162:0/3835703849' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-83985-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:54.584209+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:42:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:54.584281+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e219: 8 total, 8 up, 7 in 2024-02-13T00:42:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:54.584708+0000 mon.b (mon.1) 298 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19"}]: dispatch 2024-02-13T00:42:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:54.587551+0000 mon.a (mon.0) 1525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19"}]: dispatch 2024-02-13T00:42:55.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:42:55.410 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:55.430 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:42:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:55.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:42:55.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:42:55.563 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:55.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 -- 172.21.15.162:0/2376151533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c0fd490 msgr2=0x7fdd7c0f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:55.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 --2- 172.21.15.162:0/2376151533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c0fd490 0x7fdd7c0f6cc0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 rx=0x7fdd680047b0 tx=0x7fdd68009cb0).stop 2024-02-13T00:42:55.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 -- 172.21.15.162:0/2376151533 shutdown_connections 2024-02-13T00:42:55.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 --2- 172.21.15.162:0/2376151533 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdd7c0f73a0 0x7fdd7c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:55.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 --2- 172.21.15.162:0/2376151533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd7c0fcb30 0x7fdd7c0fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:55.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 --2- 172.21.15.162:0/2376151533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c0fd490 0x7fdd7c0f6cc0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:55.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 -- 172.21.15.162:0/2376151533 >> 172.21.15.162:0/2376151533 conn(0x7fdd7c06eb90 msgr2=0x7fdd7c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:55.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 -- 172.21.15.162:0/2376151533 shutdown_connections 2024-02-13T00:42:55.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 -- 172.21.15.162:0/2376151533 wait complete. 2024-02-13T00:42:55.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 librados: starting msgr at 2024-02-13T00:42:55.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 librados: starting objecter 2024-02-13T00:42:55.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 Processor -- start 2024-02-13T00:42:55.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 -- start start 2024-02-13T00:42:55.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 librados: setting wanted keys 2024-02-13T00:42:55.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.627+0000 7fdd82bd4700 1 librados: calling monclient init 2024-02-13T00:42:55.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd82bd4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdd7c0f73a0 0x7fdd7c10e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:55.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd82bd4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd7c0fcb30 0x7fdd7c10e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:55.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd82bd4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c0fd490 0x7fdd7c1083b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:55.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd82bd4700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c1088a0 con 0x7fdd7c0fd490 2024-02-13T00:42:55.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd82bd4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c108a10 con 0x7fdd7c0fcb30 2024-02-13T00:42:55.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd82bd4700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c108bd0 con 0x7fdd7c0f73a0 2024-02-13T00:42:55.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd80970700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdd7c0f73a0 0x7fdd7c10e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:55.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd80970700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdd7c0f73a0 0x7fdd7c10e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:52676/0 (socket says 172.21.15.162:52676) 2024-02-13T00:42:55.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd80970700 1 -- 172.21.15.162:0/259059345 learned_addr learned my addr 172.21.15.162:0/259059345 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:42:55.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd81171700 1 --2- 172.21.15.162:0/259059345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c0fd490 0x7fdd7c1083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:55.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd7bfff700 1 --2- 172.21.15.162:0/259059345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd7c0fcb30 0x7fdd7c10e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:55.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd80970700 1 -- 172.21.15.162:0/259059345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd7c0fcb30 msgr2=0x7fdd7c10e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:55.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd80970700 1 --2- 172.21.15.162:0/259059345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd7c0fcb30 0x7fdd7c10e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:55.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd80970700 1 -- 172.21.15.162:0/259059345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c0fd490 msgr2=0x7fdd7c1083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:55.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd80970700 1 --2- 172.21.15.162:0/259059345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c0fd490 0x7fdd7c1083b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:55.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd80970700 1 -- 172.21.15.162:0/259059345 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd740097c0 con 0x7fdd7c0f73a0 2024-02-13T00:42:55.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd81171700 1 --2- 172.21.15.162:0/259059345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c0fd490 0x7fdd7c1083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:42:55.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd7bfff700 1 --2- 172.21.15.162:0/259059345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd7c0fcb30 0x7fdd7c10e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:42:55.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd80970700 1 --2- 172.21.15.162:0/259059345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdd7c0f73a0 0x7fdd7c10e210 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7fdd7000e970 tx=0x7fdd7000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:55.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd79ffb700 10 client.?.objecter ms_handle_connect 0x7fdd7c0f73a0 2024-02-13T00:42:55.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd79ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:42:55.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd79ffb700 1 -- 172.21.15.162:0/259059345 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd70004660 con 0x7fdd7c0f73a0 2024-02-13T00:42:55.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd82bd4700 1 -- 172.21.15.162:0/259059345 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd68005040 con 0x7fdd7c0f73a0 2024-02-13T00:42:55.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd79ffb700 1 -- 172.21.15.162:0/259059345 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd7000f040 con 0x7fdd7c0f73a0 2024-02-13T00:42:55.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.628+0000 7fdd79ffb700 1 -- 172.21.15.162:0/259059345 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd70014790 con 0x7fdd7c0f73a0 2024-02-13T00:42:55.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.629+0000 7fdd82bd4700 10 client.36842.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:42:55.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.629+0000 7fdd82bd4700 1 -- 172.21.15.162:0/259059345 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7c115040 con 0x7fdd7c0f73a0 2024-02-13T00:42:55.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.629+0000 7fdd82bd4700 1 librados: init done 2024-02-13T00:42:55.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.630+0000 7fdd82bd4700 1 -- 172.21.15.162:0/259059345 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd60004f50 con 0x7fdd7c0f73a0 2024-02-13T00:42:55.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.630+0000 7fdd79ffb700 1 -- 172.21.15.162:0/259059345 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fdd700133c0 con 0x7fdd7c0f73a0 2024-02-13T00:42:55.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.630+0000 7fdd79ffb700 1 --2- 172.21.15.162:0/259059345 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdd6c062c90 0x7fdd6c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:42:55.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.630+0000 7fdd79ffb700 1 -- 172.21.15.162:0/259059345 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(220..220 src has 52..220) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fdd7007d0f0 con 0x7fdd7c0f73a0 2024-02-13T00:42:55.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.630+0000 7fdd79ffb700 10 client.36842.objecter ms_dispatch 0x7fdd7c001550 osd_map(220..220 src has 52..220) v4 2024-02-13T00:42:55.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.630+0000 7fdd79ffb700 3 client.36842.objecter handle_osd_map got epochs [220,220] > 0 2024-02-13T00:42:55.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.630+0000 7fdd79ffb700 3 client.36842.objecter handle_osd_map decoding full epoch 220 2024-02-13T00:42:55.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.631+0000 7fdd7bfff700 1 --2- 172.21.15.162:0/259059345 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdd6c062c90 0x7fdd6c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:42:55.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.631+0000 7fdd79ffb700 20 client.36842.objecter dump_active .. 0 homeless 2024-02-13T00:42:55.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.631+0000 7fdd79ffb700 10 client.36842.objecter ms_handle_connect 0x7fdd6c062c90 2024-02-13T00:42:55.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.631+0000 7fdd7bfff700 1 --2- 172.21.15.162:0/259059345 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdd6c062c90 0x7fdd6c065150 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 rx=0x7fdd7c109b50 tx=0x7fdd68007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:42:55.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.636+0000 7fdd79ffb700 1 -- 172.21.15.162:0/259059345 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fdd70050150 con 0x7fdd7c0f73a0 2024-02-13T00:42:55.642 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:42:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:55.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:42:55.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:55.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:42:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:42:55.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:55.831+0000 7fdd82bd4700 1 -- 172.21.15.162:0/259059345 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fdd60005200 con 0x7fdd7c0f73a0 2024-02-13T00:42:56.104 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:56.121 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:56.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:56.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:56.251 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:54.349525+0000 mgr.y (mgr.14544) 236 : cluster [DBG] pgmap v299: 292 pgs: 9 active+clean+snaptrim, 32 unknown, 14 active+clean+snaptrim_wait, 237 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:54.839103+0000 osd.3 (osd.3) 131 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:42:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:54.842051+0000 osd.3 (osd.3) 132 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:42:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:55.155222+0000 mon.b (mon.1) 299 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:55.323436+0000 mon.a (mon.0) 1526 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:42:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:55.329255+0000 mon.a (mon.0) 1527 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-83977-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-83977-31"}]': finished 2024-02-13T00:42:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:55.329346+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19"}]': finished 2024-02-13T00:42:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:55.329427+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e220: 8 total, 8 up, 7 in 2024-02-13T00:42:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:55.741217+0000 mon.c (mon.2) 206 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:42:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:55.741942+0000 mon.c (mon.2) 207 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:42:56.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:55.748447+0000 mon.a (mon.0) 1530 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:56.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:55.832589+0000 mon.c (mon.2) 208 : audit [INF] from='client.? 172.21.15.162:0/259059345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:56.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:55.832922+0000 mon.a (mon.0) 1531 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:56.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:42:56] "GET /metrics HTTP/1.1" 200 192850 "" "Prometheus/2.33.4" 2024-02-13T00:42:56.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:54.349525+0000 mgr.y (mgr.14544) 236 : cluster [DBG] pgmap v299: 292 pgs: 9 active+clean+snaptrim, 32 unknown, 14 active+clean+snaptrim_wait, 237 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:56.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:54.839103+0000 osd.3 (osd.3) 131 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:42:56.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:54.842051+0000 osd.3 (osd.3) 132 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:42:56.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:55.155222+0000 mon.b (mon.1) 299 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:56.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:55.323436+0000 mon.a (mon.0) 1526 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:42:56.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:55.329255+0000 mon.a (mon.0) 1527 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-83977-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-83977-31"}]': finished 2024-02-13T00:42:56.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:55.329346+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19"}]': finished 2024-02-13T00:42:56.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:55.329427+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e220: 8 total, 8 up, 7 in 2024-02-13T00:42:56.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:55.741217+0000 mon.c (mon.2) 206 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:42:56.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:55.741942+0000 mon.c (mon.2) 207 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:42:56.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:55.748447+0000 mon.a (mon.0) 1530 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:56.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:55.832589+0000 mon.c (mon.2) 208 : audit [INF] from='client.? 172.21.15.162:0/259059345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:56.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:55.832922+0000 mon.a (mon.0) 1531 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:56.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:54.349525+0000 mgr.y (mgr.14544) 236 : cluster [DBG] pgmap v299: 292 pgs: 9 active+clean+snaptrim, 32 unknown, 14 active+clean+snaptrim_wait, 237 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:42:56.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:54.839103+0000 osd.3 (osd.3) 131 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:42:56.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:54.842051+0000 osd.3 (osd.3) 132 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:42:56.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:55.155222+0000 mon.b (mon.1) 299 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:56.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:55.323436+0000 mon.a (mon.0) 1526 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:42:56.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:55.329255+0000 mon.a (mon.0) 1527 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-83977-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-83977-31"}]': finished 2024-02-13T00:42:56.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:55.329346+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-19"}]': finished 2024-02-13T00:42:56.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:55.329427+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e220: 8 total, 8 up, 7 in 2024-02-13T00:42:56.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:55.741217+0000 mon.c (mon.2) 206 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:42:56.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:55.741942+0000 mon.c (mon.2) 207 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:42:56.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:55.748447+0000 mon.a (mon.0) 1530 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:42:56.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:55.832589+0000 mon.c (mon.2) 208 : audit [INF] from='client.? 172.21.15.162:0/259059345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:56.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:55.832922+0000 mon.a (mon.0) 1531 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:42:56.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.333+0000 7fdd79ffb700 1 -- 172.21.15.162:0/259059345 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v221) v1 ==== 101+0+0 (secure 0 0 0) 0x7fdd70019030 con 0x7fdd7c0f73a0 2024-02-13T00:42:56.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.338+0000 7fdd82bd4700 10 librados: watch_flush enter 2024-02-13T00:42:56.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 10 librados: watch_flush exit 2024-02-13T00:42:56.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 20 client.36842.objecter shutdown clearing up homeless session... 2024-02-13T00:42:56.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 10 client.36842.objecter successfully canceled tick 2024-02-13T00:42:56.350 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:56.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:56.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 -- 172.21.15.162:0/259059345 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdd6c062c90 msgr2=0x7fdd6c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:56.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 --2- 172.21.15.162:0/259059345 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdd6c062c90 0x7fdd6c065150 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 rx=0x7fdd7c109b50 tx=0x7fdd68007f90).stop 2024-02-13T00:42:56.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 -- 172.21.15.162:0/259059345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdd7c0f73a0 msgr2=0x7fdd7c10e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:42:56.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 --2- 172.21.15.162:0/259059345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdd7c0f73a0 0x7fdd7c10e210 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7fdd7000e970 tx=0x7fdd7000b040).stop 2024-02-13T00:42:56.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 -- 172.21.15.162:0/259059345 shutdown_connections 2024-02-13T00:42:56.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 --2- 172.21.15.162:0/259059345 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdd6c062c90 0x7fdd6c065150 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:56.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 --2- 172.21.15.162:0/259059345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd7c0fd490 0x7fdd7c1083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:56.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 --2- 172.21.15.162:0/259059345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdd7c0f73a0 0x7fdd7c10e210 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:56.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 --2- 172.21.15.162:0/259059345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd7c0fcb30 0x7fdd7c10e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:42:56.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 -- 172.21.15.162:0/259059345 >> 172.21.15.162:0/259059345 conn(0x7fdd7c06eb90 msgr2=0x7fdd7c06f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:42:56.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 -- 172.21.15.162:0/259059345 shutdown_connections 2024-02-13T00:42:56.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 -- 172.21.15.162:0/259059345 wait complete. 2024-02-13T00:42:56.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:42:56.344+0000 7fdd82bd4700 1 librados: shutdown 2024-02-13T00:42:56.369 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:42:56.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:42:56.502 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:56.523 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:42:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:56.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:42:56.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:56.657 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:56.677 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:56.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:56.781 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:56.809 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:56.828 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:56.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:56.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:56.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:42:56.959 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:56.979 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:42:56.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:56.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:42:57.011 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:55.436373+0000 mgr.y (mgr.14544) 237 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:57.012 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:55.477654+0000 osd.5 (osd.5) 97 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:42:57.012 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:55.481123+0000 osd.5 (osd.5) 98 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:42:57.012 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:56.156144+0000 mon.b (mon.1) 300 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:57.012 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:56.325611+0000 mon.a (mon.0) 1532 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:42:57.012 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:56.330214+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:42:57.012 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:56.330280+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e221: 8 total, 8 up, 7 in 2024-02-13T00:42:57.012 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:56.335515+0000 mon.a (mon.0) 1535 : audit [INF] from='client.? 172.21.15.162:0/44862729' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:57.012 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:56.412494+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.10 deep-scrub starts 2024-02-13T00:42:57.012 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:56.413322+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.10 deep-scrub ok 2024-02-13T00:42:57.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:56.478521+0000 osd.2 (osd.2) 95 : cluster [DBG] 6.12 deep-scrub starts 2024-02-13T00:42:57.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:56.479446+0000 osd.2 (osd.2) 96 : cluster [DBG] 6.12 deep-scrub ok 2024-02-13T00:42:57.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:56.795137+0000 osd.2 (osd.2) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:57.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:56.795376+0000 osd.2 (osd.2) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:57.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:42:57.109 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:55.436373+0000 mgr.y (mgr.14544) 237 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:55.477654+0000 osd.5 (osd.5) 97 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:42:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:55.481123+0000 osd.5 (osd.5) 98 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:42:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:56.156144+0000 mon.b (mon.1) 300 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:56.325611+0000 mon.a (mon.0) 1532 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:42:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:56.330214+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:42:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:56.330280+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e221: 8 total, 8 up, 7 in 2024-02-13T00:42:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:56.335515+0000 mon.a (mon.0) 1535 : audit [INF] from='client.? 172.21.15.162:0/44862729' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:56.412494+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.10 deep-scrub starts 2024-02-13T00:42:57.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:56.413322+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.10 deep-scrub ok 2024-02-13T00:42:57.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:56.478521+0000 osd.2 (osd.2) 95 : cluster [DBG] 6.12 deep-scrub starts 2024-02-13T00:42:57.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:56.479446+0000 osd.2 (osd.2) 96 : cluster [DBG] 6.12 deep-scrub ok 2024-02-13T00:42:57.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:56.795137+0000 osd.2 (osd.2) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:57.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:56.795376+0000 osd.2 (osd.2) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:57.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:55.436373+0000 mgr.y (mgr.14544) 237 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:42:57.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:55.477654+0000 osd.5 (osd.5) 97 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:42:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:55.481123+0000 osd.5 (osd.5) 98 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:42:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:56.156144+0000 mon.b (mon.1) 300 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:56.325611+0000 mon.a (mon.0) 1532 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:42:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:56.330214+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:42:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:56.330280+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e221: 8 total, 8 up, 7 in 2024-02-13T00:42:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:56.335515+0000 mon.a (mon.0) 1535 : audit [INF] from='client.? 172.21.15.162:0/44862729' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:56.412494+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.10 deep-scrub starts 2024-02-13T00:42:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:56.413322+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.10 deep-scrub ok 2024-02-13T00:42:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:56.478521+0000 osd.2 (osd.2) 95 : cluster [DBG] 6.12 deep-scrub starts 2024-02-13T00:42:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:56.479446+0000 osd.2 (osd.2) 96 : cluster [DBG] 6.12 deep-scrub ok 2024-02-13T00:42:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:56.795137+0000 osd.2 (osd.2) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:42:57.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:56.795376+0000 osd.2 (osd.2) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:42:57.460 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:42:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:57.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:42:57.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:57.590 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:57.741 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:57.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:57.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:42:57.878 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:57.899 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:42:57.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:57.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:42:58.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:42:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:42:58.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:42:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:56.350292+0000 mgr.y (mgr.14544) 238 : cluster [DBG] pgmap v302: 268 pgs: 9 active+clean+snaptrim, 40 unknown, 12 active+clean+snaptrim_wait, 207 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s wr, 0 op/s 2024-02-13T00:42:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:56.498642+0000 osd.5 (osd.5) 99 : cluster [DBG] 2.16 deep-scrub starts 2024-02-13T00:42:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:56.499571+0000 osd.5 (osd.5) 100 : cluster [DBG] 2.16 deep-scrub ok 2024-02-13T00:42:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:57.157503+0000 mon.b (mon.1) 301 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:57.338825+0000 mon.a (mon.0) 1536 : audit [INF] from='client.? 172.21.15.162:0/44862729' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:57.339008+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e222: 8 total, 8 up, 7 in 2024-02-13T00:42:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:57.342672+0000 mon.c (mon.2) 209 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:57.347672+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:57.349123+0000 mon.b (mon.1) 302 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:58.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:57.349793+0000 mon.a (mon.0) 1539 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:58.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:57.516006+0000 osd.2 (osd.2) 99 : cluster [DBG] 18.13 scrub starts 2024-02-13T00:42:58.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:57.517531+0000 osd.2 (osd.2) 100 : cluster [DBG] 18.13 scrub ok 2024-02-13T00:42:58.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:56.350292+0000 mgr.y (mgr.14544) 238 : cluster [DBG] pgmap v302: 268 pgs: 9 active+clean+snaptrim, 40 unknown, 12 active+clean+snaptrim_wait, 207 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s wr, 0 op/s 2024-02-13T00:42:58.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:56.498642+0000 osd.5 (osd.5) 99 : cluster [DBG] 2.16 deep-scrub starts 2024-02-13T00:42:58.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:56.499571+0000 osd.5 (osd.5) 100 : cluster [DBG] 2.16 deep-scrub ok 2024-02-13T00:42:58.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:57.157503+0000 mon.b (mon.1) 301 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:58.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:57.338825+0000 mon.a (mon.0) 1536 : audit [INF] from='client.? 172.21.15.162:0/44862729' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:58.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:57.339008+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e222: 8 total, 8 up, 7 in 2024-02-13T00:42:58.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:57.342672+0000 mon.c (mon.2) 209 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:58.312 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:57.347672+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:58.312 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:57.349123+0000 mon.b (mon.1) 302 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:58.312 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:57.349793+0000 mon.a (mon.0) 1539 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:58.312 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:57.516006+0000 osd.2 (osd.2) 99 : cluster [DBG] 18.13 scrub starts 2024-02-13T00:42:58.312 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:57.517531+0000 osd.2 (osd.2) 100 : cluster [DBG] 18.13 scrub ok 2024-02-13T00:42:58.312 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:56.350292+0000 mgr.y (mgr.14544) 238 : cluster [DBG] pgmap v302: 268 pgs: 9 active+clean+snaptrim, 40 unknown, 12 active+clean+snaptrim_wait, 207 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s wr, 0 op/s 2024-02-13T00:42:58.312 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:56.498642+0000 osd.5 (osd.5) 99 : cluster [DBG] 2.16 deep-scrub starts 2024-02-13T00:42:58.312 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:56.499571+0000 osd.5 (osd.5) 100 : cluster [DBG] 2.16 deep-scrub ok 2024-02-13T00:42:58.313 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:57.157503+0000 mon.b (mon.1) 301 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:58.313 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:57.338825+0000 mon.a (mon.0) 1536 : audit [INF] from='client.? 172.21.15.162:0/44862729' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:58.313 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:57.339008+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e222: 8 total, 8 up, 7 in 2024-02-13T00:42:58.313 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:57.342672+0000 mon.c (mon.2) 209 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:58.313 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:57.347672+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:58.313 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:57.349123+0000 mon.b (mon.1) 302 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:58.313 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:57.349793+0000 mon.a (mon.0) 1539 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:58.313 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:57.516006+0000 osd.2 (osd.2) 99 : cluster [DBG] 18.13 scrub starts 2024-02-13T00:42:58.313 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:57.517531+0000 osd.2 (osd.2) 100 : cluster [DBG] 18.13 scrub ok 2024-02-13T00:42:58.439 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:42:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:42:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:42:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:42:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24739.0:3 16.f 16:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-13T00:42:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:00.558130+0000", 2024-02-13T00:42:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 117.583803782, 2024-02-13T00:42:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.081761433, 2024-02-13T00:42:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24739", 2024-02-13T00:42:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2737601038", 2024-02-13T00:42:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3 2024-02-13T00:42:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558130+0000", 2024-02-13T00:42:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558130+0000", 2024-02-13T00:42:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558128+0000", 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2910000000000002e-06 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558135+0000", 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0115999999999999e-05 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558145+0000", 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2560000000000002e-06 2024-02-13T00:42:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558153+0000", 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0846e-05 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558214+0000", 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8210000000000008e-06 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558223+0000", 2024-02-13T00:42:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.017115333 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.575338+0000", 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122046 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.575460+0000", 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001568549 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-02-13T00:42:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.577029+0000", 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012430968000000001 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589460+0000", 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015196783 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.604657+0000", 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034770550999999997 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.639427+0000", 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040860799999999997 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.639836+0000", 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5262e-05 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.639891+0000", 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.081761433 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24763.0:2 23.d 23:b36015c1:test-rados-api-smithi162-84089-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.495169+0000", 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 116.646764319, 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22330878300000001, 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24763", 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/90739862", 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495169+0000", 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495169+0000", 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495167+0000", 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1899999999999995e-06 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495175+0000", 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0474e-05 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495186+0000", 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9990000000000008e-06 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495194+0000", 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3149000000000002e-05 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495247+0000", 2024-02-13T00:42:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.533e-06 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495254+0000", 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031521518999999998 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.526775+0000", 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0169999999999996e-06 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.526783+0000", 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.170767906 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697551+0000", 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3193000000000003e-05 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697614+0000", 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8979000000000002e-05 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,4", 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697713+0000", 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012975179999999999 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.699011+0000", 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003049412 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.702060+0000", 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016330172 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.718390+0000", 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8174999999999999e-05 2024-02-13T00:42:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.718438+0000", 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9382999999999999e-05 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.718478+0000", 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22330878300000001 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24767.0:2 14.1f 14:feb32ed4:test-rados-api-smithi162-84161-3::testobj:head [write 0~8] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.597458+0000", 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 116.54447500800001, 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.074387914999999999, 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24767", 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2472368574", 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597458+0000", 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597458+0000", 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597456+0000", 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.696e-06 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597466+0000", 2024-02-13T00:42:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0883999999999999e-05 2024-02-13T00:42:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597477+0000", 2024-02-13T00:42:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7740000000000004e-06 2024-02-13T00:42:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597485+0000", 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029994400000000001 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597785+0000", 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113497 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597898+0000", 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015973699999999999 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.598058+0000", 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020561050000000001 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.618619+0000", 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033024080000000002 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.621921+0000", 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049844082999999997 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.671766+0000", 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4239000000000001e-05 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.671810+0000", 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6464999999999997e-05 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.671846+0000", 2024-02-13T00:42:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.074387914999999999 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24736.0:2 13.15 13:ad551702:::test:head [stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.664313+0000", 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 116.47762015399999, 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032550835, 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24736", 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1679583445", 2024-02-13T00:42:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.664313+0000", 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.664313+0000", 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.664311+0000", 2024-02-13T00:42:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6179999999999997e-06 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.664319+0000", 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7450000000000007e-06 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.664329+0000", 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5780000000000001e-06 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.664336+0000", 2024-02-13T00:42:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045557999999999996 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.668892+0000", 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6909999999999997e-06 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.668900+0000", 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027904875999999999 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.696805+0000", 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9225999999999998e-05 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.696864+0000", 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032550835 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:441 20.14 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head [setxattr a196 in=4b] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.493977+0000", 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 115.64795592599999, 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16342337300000001, 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 441 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.493977+0000", 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.493977+0000", 2024-02-13T00:42:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:42:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.493975+0000", 2024-02-13T00:42:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3498e-05 2024-02-13T00:42:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.493988+0000", 2024-02-13T00:42:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5116999999999999e-05 2024-02-13T00:42:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494003+0000", 2024-02-13T00:42:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3450000000000006e-06 2024-02-13T00:42:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494012+0000", 2024-02-13T00:42:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0686999999999998e-05 2024-02-13T00:42:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494062+0000", 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6800999999999998e-05 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494149+0000", 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147002 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494296+0000", 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001189708 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:42:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.495486+0000", 2024-02-13T00:42:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000325693 2024-02-13T00:42:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.495812+0000", 2024-02-13T00:42:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.161531377 2024-02-13T00:42:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.657343+0000", 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1778e-05 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.657375+0000", 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5942999999999999e-05 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.657401+0000", 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16342337300000001 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14880.0:9 19.f e108/105 19:f3112f20:test-rados-api-smithi162-84033-5::foo:head v 108'1, mlcod=0'0)", 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.832175+0000", 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 115.309758633, 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024228629000000002, 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.832175+0000", 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.832175+0000", 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.832174+0000", 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5426000000000001e-05 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.832189+0000", 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.117e-05 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.832200+0000", 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3759999999999996e-06 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.832207+0000", 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3139999999999998e-05 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.832240+0000", 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4718e-05 2024-02-13T00:42:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.832255+0000", 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024110612 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.856365+0000", 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8167000000000002e-05 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.856403+0000", 2024-02-13T00:42:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024228629000000002 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:1809 20.14 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head [setxattr a880 in=4b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:03.990416+0000", 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.151517205, 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055387081999999997, 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:42:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:42:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1809 2024-02-13T00:42:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990416+0000", 2024-02-13T00:42:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990416+0000", 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990415+0000", 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7190000000000004e-06 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990422+0000", 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5320000000000006e-06 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990430+0000", 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0110000000000003e-06 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990437+0000", 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043852149999999996 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.994823+0000", 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7717000000000001e-05 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.994900+0000", 2024-02-13T00:42:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013310199999999999 2024-02-13T00:42:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:42:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.995033+0000", 2024-02-13T00:42:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089106199999999997 2024-02-13T00:42:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:42:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.995924+0000", 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047824802 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.043749+0000", 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00194904 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.045698+0000", 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8034e-05 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.045756+0000", 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6875000000000001e-05 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.045803+0000", 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055387081999999997 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14886.0:2067 20.1e e109/105 20:78e1aec5:test-rados-api-smithi162-84049-17::small:head v 109'3, mlcod=109'1)", 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.582878+0000", 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 113.559055726, 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043476111999999997, 2024-02-13T00:42:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582878+0000", 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582878+0000", 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582876+0000", 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2574e-05 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582889+0000", 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5733999999999999e-05 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582905+0000", 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4379999999999998e-06 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582912+0000", 2024-02-13T00:42:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9762999999999997e-05 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582972+0000", 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3986999999999999e-05 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582996+0000", 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043311409000000002 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.626307+0000", 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6532000000000001e-05 2024-02-13T00:42:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.626354+0000", 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043476111999999997 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2070 20.b 20:d3a22371:test-rados-api-smithi162-84049-17::big2:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.742315+0000", 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 113.39961839, 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.100851074, 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2070 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742315+0000", 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742315+0000", 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742313+0000", 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011733019000000001 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754046+0000", 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3737e-05 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754070+0000", 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0747999999999999e-05 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754081+0000", 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1600000000000001e-05 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754132+0000", 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6428999999999994e-05 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754229+0000", 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106959 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,6", 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754336+0000", 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036365882000000002 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.790702+0000", 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.052175179000000002 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.842877+0000", 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000231101 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.843108+0000", 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3439000000000003e-05 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.843141+0000", 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4810000000000001e-05 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.843166+0000", 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.100851074 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16275.0:35 66.8 66:110c41bf:test-rados-api-smithi162-84081-10::foo:head [write 128~128] snapc b=[b,a] ondisk+write+known_if_redirected e135)", 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:30.279104+0000", 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 87.862828878000002, 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044119924999999997, 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16275", 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/238911824", 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 35 2024-02-13T00:42:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279104+0000", 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279104+0000", 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279103+0000", 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1609999999999997e-06 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279110+0000", 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9039999999999998e-06 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279119+0000", 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8399999999999997e-06 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279126+0000", 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026694828 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.305820+0000", 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2825999999999995e-05 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.305913+0000", 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018201799999999999 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.306095+0000", 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025887029999999999 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.308684+0000", 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013424475999999999 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.322109+0000", 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010389150000000001 2024-02-13T00:42:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.323147+0000", 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0759999999999997e-05 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.323178+0000", 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6211000000000002e-05 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.323224+0000", 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044119924999999997 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16275.0:42 66.8 66:110c41bf:test-rados-api-smithi162-84081-10::foo:head [write 128~128] snapc c=[c,b,a] ondisk+write+known_if_redirected e136)", 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:31.308000+0000", 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 86.833933521000006, 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075133532000000003, 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16275", 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/238911824", 2024-02-13T00:42:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 42 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308000+0000", 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308000+0000", 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.307997+0000", 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5245999999999999e-05 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308012+0000", 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8859e-05 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308031+0000", 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0020999999999999e-05 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308041+0000", 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014614509999999999 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.309503+0000", 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8609999999999998e-06 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.309511+0000", 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043170606 2024-02-13T00:42:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.352681+0000", 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010319699999999999 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.352784+0000", 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016862800000000001 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.352953+0000", 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002530109 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.355483+0000", 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0083680739999999997 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.363851+0000", 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019179723999999999 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383031+0000", 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6895000000000003e-05 2024-02-13T00:42:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383088+0000", 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5457000000000002e-05 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383133+0000", 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075133532000000003 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16275.0:43 66.8 66:110c41bf:test-rados-api-smithi162-84081-10::foo:head [write 512~128] snapc c=[c,b,a] ondisk+write+known_if_redirected e136)", 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:31.383796+0000", 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 86.758137202, 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.039042471000000002, 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16275", 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/238911824", 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 43 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383796+0000", 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383796+0000", 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383795+0000", 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7560000000000002e-06 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383801+0000", 2024-02-13T00:42:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6559999999999998e-06 2024-02-13T00:42:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383810+0000", 2024-02-13T00:42:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4289999999999998e-06 2024-02-13T00:42:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383817+0000", 2024-02-13T00:42:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7485000000000006e-05 2024-02-13T00:42:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383914+0000", 2024-02-13T00:42:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7400999999999998e-05 2024-02-13T00:42:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.384011+0000", 2024-02-13T00:42:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000166083 2024-02-13T00:42:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:42:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.384177+0000", 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00081086000000000005 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.384988+0000", 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076699100000000003 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.392658+0000", 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030073246000000001 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.422731+0000", 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6507000000000005e-05 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.422798+0000", 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0617000000000001e-05 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.422839+0000", 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.039042471000000002 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.35941.0:2 84.6 e139/138 84:602f83fe:::foo:head v 139'1, mlcod=0'0)", 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:34.483127+0000", 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.658806022999997, 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041127036999999998, 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483127+0000", 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483127+0000", 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483127+0000", 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3149999999999999e-06 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483132+0000", 2024-02-13T00:42:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0050000000000004e-06 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483137+0000", 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1290000000000001e-06 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483143+0000", 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022933839999999999 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.485437+0000", 2024-02-13T00:42:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6535e-05 2024-02-13T00:42:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.485453+0000", 2024-02-13T00:42:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038765455999999997 2024-02-13T00:42:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.524219+0000", 2024-02-13T00:42:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5552000000000001e-05 2024-02-13T00:42:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.524254+0000", 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041127036999999998 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.43732.0:2 112.6 112:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e172)", 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:07.426136+0000", 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.715797082999998, 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058639376999999999, 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.43732", 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/981068687", 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426136+0000", 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426136+0000", 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426134+0000", 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5360000000000004e-06 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426142+0000", 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0329999999999999e-05 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426152+0000", 2024-02-13T00:42:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5750000000000003e-06 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426160+0000", 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4138999999999998e-05 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426214+0000", 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6590000000000007e-06 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:42:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426222+0000", 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014917939 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.441140+0000", 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109622 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.441249+0000", 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015227799999999999 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.441401+0000", 2024-02-13T00:42:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028880260000000001 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.444289+0000", 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033112480000000001 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.447601+0000", 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037075416 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.484676+0000", 2024-02-13T00:42:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2821999999999997e-05 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.484729+0000", 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6717e-05 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.484776+0000", 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058639376999999999 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36677.0:2 126.13 126:cd6547c0:test-rados-api-smithi162-84049-28::foo:head [write 0~4096] snapc 0=[] ondisk+write+known_if_redirected e188)", 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:23.332783+0000", 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.809150396, 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034374348999999998, 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36677", 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2785938865", 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.332783+0000", 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.332783+0000", 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.332781+0000", 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0675e-05 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.332792+0000", 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1491000000000001e-05 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.332803+0000", 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5930000000000002e-06 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.332810+0000", 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0075805630000000002 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.340390+0000", 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.849e-06 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.340397+0000", 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023686730999999999 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.364084+0000", 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013252500000000001 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.364216+0000", 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015555899999999999 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.364372+0000", 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089011899999999998 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.365262+0000", 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001636141 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.366898+0000", 2024-02-13T00:42:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020851100000000001 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.367107+0000", 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9037999999999999e-05 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.367136+0000", 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1498999999999999e-05 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:23.367157+0000", 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034374348999999998 2024-02-13T00:42:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36659.0:10 125.1s1 125:939f3713:test-rados-api-smithi162-84081-22::foo:head [write 0~8192] snapc 6=[6] ondisk+write+known_if_redirected e192)", 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:27.342201+0000", 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.799732080999998, 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028875405, 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36659", 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/371088046", 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 10 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.342201+0000", 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.342201+0000", 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.342200+0000", 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2655e-05 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.342212+0000", 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5929999999999995e-06 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.342222+0000", 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7259999999999997e-06 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.342232+0000", 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036521629999999999 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.345884+0000", 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9699999999999999e-06 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.345892+0000", 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042535580000000002 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.350145+0000", 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129002 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.350274+0000", 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042259300000000002 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.350697+0000", 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013678648999999999 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.364376+0000", 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0066308019999999999 2024-02-13T00:42:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.371006+0000", 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0241999999999996e-05 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.371077+0000", 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028875405 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36704.0:2 129.c 129:311cbe94:::oid_ExecuteClassPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e192)", 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:27.357488+0000", 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.784445594000001, 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029546602000000002, 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36704", 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2150629092", 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.357488+0000", 2024-02-13T00:42:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.357488+0000", 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.357486+0000", 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3229999999999998e-06 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.357494+0000", 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1316e-05 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.357505+0000", 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4279999999999996e-06 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.357514+0000", 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00061996700000000002 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.358134+0000", 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3350000000000003e-06 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.358140+0000", 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026860390000000001 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.385001+0000", 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7750000000000006e-05 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.385078+0000", 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1675999999999994e-05 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,4", 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.385170+0000", 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016256689999999999 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.386796+0000", 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014129 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.386937+0000", 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1279999999999999e-05 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.386968+0000", 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0604e-05 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.386999+0000", 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5519999999999999e-05 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.387034+0000", 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029546602000000002 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36659.0:15 125.1s1 125:939f3713:test-rados-api-smithi162-84081-22::foo:head [write 24576~8192] snapc 7=[7,6] ondisk+write+known_if_redirected e193)", 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:28.354457+0000", 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.787476355999999, 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025691510000000001, 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36659", 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/371088046", 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 15 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.354457+0000", 2024-02-13T00:42:58.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.354457+0000", 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.354455+0000", 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8345000000000001e-05 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.354473+0000", 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1754e-05 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.354485+0000", 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9619999999999993e-06 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.354493+0000", 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024852499999999998 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.354742+0000", 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010662000000000001 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.354848+0000", 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000591451 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.355440+0000", 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033907529999999998 2024-02-13T00:42:58.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.358831+0000", 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021248106999999999 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.380079+0000", 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9887000000000004e-05 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.380149+0000", 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025691510000000001 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.543 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36767.0:1 140.6s0 140:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:41.361654+0000", 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.780279427, 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99434686900000002, 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36767", 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1509178592", 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.544 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361654+0000", 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361654+0000", 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361653+0000", 2024-02-13T00:42:58.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2099999999999998e-06 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361659+0000", 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1799999999999996e-06 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361667+0000", 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8100000000000003e-06 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361673+0000", 2024-02-13T00:42:58.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0318000000000002e-05 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361723+0000", 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5279999999999999e-06 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361727+0000", 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98580988800000002 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.347536+0000", 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149586 2024-02-13T00:42:58.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.347686+0000", 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038543399999999998 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.348071+0000", 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0075773510000000004 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.355649+0000", 2024-02-13T00:42:58.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027853600000000002 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.355927+0000", 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3433e-05 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.356001+0000", 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99434686900000002 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.549 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36824.0:1 148.6s1 148:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e220)", 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:55.339285+0000", 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.8026484410000001, 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0318101019999999, 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36824", 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4039137531", 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339285+0000", 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:42:58.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339285+0000", 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339283+0000", 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8839999999999994e-06 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339291+0000", 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1019000000000001e-05 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339302+0000", 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6029999999999994e-06 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339312+0000", 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061423129999999999 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.345454+0000", 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3289999999999998e-06 2024-02-13T00:42:58.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.345457+0000", 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.020285004 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.365742+0000", 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137575 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.365880+0000", 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040855500000000001 2024-02-13T00:42:58.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.366289+0000", 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051522999999999998 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.366804+0000", 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004224088 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.371028+0000", 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7168999999999993e-05 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:42:58.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:42:58.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:42:58.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.371095+0000", 2024-02-13T00:42:58.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0318101019999999 2024-02-13T00:42:58.555 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.555 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.555 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.555 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:42:58.555 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:42:58.555 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:42:59.009 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-13T00:42:59.011 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:42:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:42:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:42:59.025 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:42:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:57.545748+0000 osd.5 (osd.5) 101 : cluster [DBG] 5.f scrub starts 2024-02-13T00:42:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:57.546511+0000 osd.5 (osd.5) 102 : cluster [DBG] 5.f scrub ok 2024-02-13T00:42:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:57.799379+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.1c scrub starts 2024-02-13T00:42:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:57.802414+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.1c scrub ok 2024-02-13T00:42:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:58.158632+0000 mon.b (mon.1) 303 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:58.347916+0000 mon.a (mon.0) 1540 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]': finished 2024-02-13T00:42:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:58.348038+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:58.348147+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e223: 8 total, 8 up, 7 in 2024-02-13T00:42:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:58.355183+0000 mon.c (mon.2) 210 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:58.355728+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? 172.21.15.162:0/304213416' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:58.356665+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:58.375286+0000 mon.b (mon.1) 304 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:59.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:42:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:58.375848+0000 mon.a (mon.0) 1545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:59.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:57.545748+0000 osd.5 (osd.5) 101 : cluster [DBG] 5.f scrub starts 2024-02-13T00:42:59.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:57.546511+0000 osd.5 (osd.5) 102 : cluster [DBG] 5.f scrub ok 2024-02-13T00:42:59.303 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:57.799379+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.1c scrub starts 2024-02-13T00:42:59.303 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:57.802414+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.1c scrub ok 2024-02-13T00:42:59.303 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:58.158632+0000 mon.b (mon.1) 303 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:59.303 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:58.347916+0000 mon.a (mon.0) 1540 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]': finished 2024-02-13T00:42:59.303 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:58.348038+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:59.303 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:58.348147+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e223: 8 total, 8 up, 7 in 2024-02-13T00:42:59.303 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:58.355183+0000 mon.c (mon.2) 210 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:59.303 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:58.355728+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? 172.21.15.162:0/304213416' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:59.303 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:58.356665+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:59.303 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:58.375286+0000 mon.b (mon.1) 304 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:59.303 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:58.375848+0000 mon.a (mon.0) 1545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:59.304 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:57.545748+0000 osd.5 (osd.5) 101 : cluster [DBG] 5.f scrub starts 2024-02-13T00:42:59.304 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:57.546511+0000 osd.5 (osd.5) 102 : cluster [DBG] 5.f scrub ok 2024-02-13T00:42:59.304 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:57.799379+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.1c scrub starts 2024-02-13T00:42:59.304 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:57.802414+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.1c scrub ok 2024-02-13T00:42:59.304 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:58.158632+0000 mon.b (mon.1) 303 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:42:59.304 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:58.347916+0000 mon.a (mon.0) 1540 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-83977-31"}]': finished 2024-02-13T00:42:59.304 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:58.348038+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:42:59.304 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:58.348147+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e223: 8 total, 8 up, 7 in 2024-02-13T00:42:59.304 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:58.355183+0000 mon.c (mon.2) 210 : audit [INF] from='client.? 172.21.15.162:0/4039137531' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:59.304 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:58.355728+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? 172.21.15.162:0/304213416' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:42:59.304 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:58.356665+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]: dispatch 2024-02-13T00:42:59.304 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:58.375286+0000 mon.b (mon.1) 304 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:59.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:42:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:58.375848+0000 mon.a (mon.0) 1545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:42:59.328 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:42:59.328 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:42:59.328 INFO:tasks.thrashosds.thrasher:Growing pool default.rgw.meta 2024-02-13T00:42:59.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:42:59.384 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:42:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:42:59.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:42:59.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:42:59.518 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:00.289 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:58.350909+0000 mgr.y (mgr.14544) 239 : cluster [DBG] pgmap v305: 324 pgs: 9 active+clean+snaptrim, 96 unknown, 12 active+clean+snaptrim_wait, 207 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s wr, 0 op/s 2024-02-13T00:43:00.289 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:59.159555+0000 mon.b (mon.1) 305 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:00.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:59.352461+0000 mon.a (mon.0) 1546 : audit [INF] from='client.? 172.21.15.162:0/304213416' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:00.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:59.352541+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]': finished 2024-02-13T00:43:00.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:59.352633+0000 mon.a (mon.0) 1548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:00.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:59.352685+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e224: 8 total, 8 up, 7 in 2024-02-13T00:43:00.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:59.357021+0000 mon.b (mon.1) 306 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-21"}]: dispatch 2024-02-13T00:43:00.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:59.360940+0000 mon.a (mon.0) 1550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-21"}]: dispatch 2024-02-13T00:43:00.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:59.399666+0000 mon.b (mon.1) 307 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:00.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:59.407043+0000 mon.a (mon.0) 1551 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:00.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:59.407630+0000 mon.b (mon.1) 308 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:00.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:59.408791+0000 mon.a (mon.0) 1552 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:00.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:59.409162+0000 mon.b (mon.1) 309 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-83977-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:00.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:42:59.409807+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-83977-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:58.350909+0000 mgr.y (mgr.14544) 239 : cluster [DBG] pgmap v305: 324 pgs: 9 active+clean+snaptrim, 96 unknown, 12 active+clean+snaptrim_wait, 207 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s wr, 0 op/s 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:59.159555+0000 mon.b (mon.1) 305 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:59.352461+0000 mon.a (mon.0) 1546 : audit [INF] from='client.? 172.21.15.162:0/304213416' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:59.352541+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]': finished 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:59.352633+0000 mon.a (mon.0) 1548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:59.352685+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e224: 8 total, 8 up, 7 in 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:59.357021+0000 mon.b (mon.1) 306 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-21"}]: dispatch 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:59.360940+0000 mon.a (mon.0) 1550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-21"}]: dispatch 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:59.399666+0000 mon.b (mon.1) 307 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:59.407043+0000 mon.a (mon.0) 1551 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:59.407630+0000 mon.b (mon.1) 308 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:59.408791+0000 mon.a (mon.0) 1552 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:00.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:59.409162+0000 mon.b (mon.1) 309 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-83977-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:00.292 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:42:59.409807+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-83977-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:00.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:58.350909+0000 mgr.y (mgr.14544) 239 : cluster [DBG] pgmap v305: 324 pgs: 9 active+clean+snaptrim, 96 unknown, 12 active+clean+snaptrim_wait, 207 active+clean; 10 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s wr, 0 op/s 2024-02-13T00:43:00.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:59.159555+0000 mon.b (mon.1) 305 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:00.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:59.352461+0000 mon.a (mon.0) 1546 : audit [INF] from='client.? 172.21.15.162:0/304213416' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:59.352541+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-83977-31"}]': finished 2024-02-13T00:43:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:59.352633+0000 mon.a (mon.0) 1548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:59.352685+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e224: 8 total, 8 up, 7 in 2024-02-13T00:43:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:59.357021+0000 mon.b (mon.1) 306 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-21"}]: dispatch 2024-02-13T00:43:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:59.360940+0000 mon.a (mon.0) 1550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-21"}]: dispatch 2024-02-13T00:43:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:59.399666+0000 mon.b (mon.1) 307 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:59.407043+0000 mon.a (mon.0) 1551 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:59.407630+0000 mon.b (mon.1) 308 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:59.408791+0000 mon.a (mon.0) 1552 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:59.409162+0000 mon.b (mon.1) 309 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-83977-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:42:59.409807+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-83977-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:00.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:43:00.606 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:00.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:00.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:00.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:00.738 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:59.602233+0000 osd.5 (osd.5) 103 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:43:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:42:59.603998+0000 osd.5 (osd.5) 104 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:43:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:00.160732+0000 mon.b (mon.1) 310 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:00.330880+0000 mon.a (mon.0) 1554 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-21"}]': finished 2024-02-13T00:43:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:00.330987+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-83977-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:00.331095+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e225: 8 total, 8 up, 7 in 2024-02-13T00:43:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:00.333584+0000 mon.b (mon.1) 311 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-21", "mode": "writeback"}]: dispatch 2024-02-13T00:43:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:00.333746+0000 mon.b (mon.1) 312 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-83977-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:00.342704+0000 mon.a (mon.0) 1557 : audit [INF] from='client.? 172.21.15.162:0/410125232' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:01.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:00.347343+0000 mon.a (mon.0) 1558 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-21", "mode": "writeback"}]: dispatch 2024-02-13T00:43:01.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:00.347619+0000 mon.a (mon.0) 1559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-83977-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:59.602233+0000 osd.5 (osd.5) 103 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:43:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:42:59.603998+0000 osd.5 (osd.5) 104 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:43:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:00.160732+0000 mon.b (mon.1) 310 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:00.330880+0000 mon.a (mon.0) 1554 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-21"}]': finished 2024-02-13T00:43:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:00.330987+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-83977-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:00.331095+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e225: 8 total, 8 up, 7 in 2024-02-13T00:43:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:00.333584+0000 mon.b (mon.1) 311 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-21", "mode": "writeback"}]: dispatch 2024-02-13T00:43:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:00.333746+0000 mon.b (mon.1) 312 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-83977-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:00.342704+0000 mon.a (mon.0) 1557 : audit [INF] from='client.? 172.21.15.162:0/410125232' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:00.347343+0000 mon.a (mon.0) 1558 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-21", "mode": "writeback"}]: dispatch 2024-02-13T00:43:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:00.347619+0000 mon.a (mon.0) 1559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-83977-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:01.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:59.602233+0000 osd.5 (osd.5) 103 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:43:01.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:42:59.603998+0000 osd.5 (osd.5) 104 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:43:01.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:00.160732+0000 mon.b (mon.1) 310 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:01.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:00.330880+0000 mon.a (mon.0) 1554 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-21"}]': finished 2024-02-13T00:43:01.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:00.330987+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-83977-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:01.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:00.331095+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e225: 8 total, 8 up, 7 in 2024-02-13T00:43:01.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:00.333584+0000 mon.b (mon.1) 311 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-21", "mode": "writeback"}]: dispatch 2024-02-13T00:43:01.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:00.333746+0000 mon.b (mon.1) 312 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-83977-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:01.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:00.342704+0000 mon.a (mon.0) 1557 : audit [INF] from='client.? 172.21.15.162:0/410125232' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:01.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:00.347343+0000 mon.a (mon.0) 1558 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-21", "mode": "writeback"}]: dispatch 2024-02-13T00:43:01.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:00.347619+0000 mon.a (mon.0) 1559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-83977-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:02.171 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:00.351449+0000 mgr.y (mgr.14544) 240 : cluster [DBG] pgmap v308: 356 pgs: 6 active+clean+snaptrim, 32 unknown, 10 active+clean+snaptrim_wait, 308 active+clean; 9.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:43:02.172 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:00.633224+0000 osd.5 (osd.5) 105 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:43:02.172 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:00.636794+0000 osd.5 (osd.5) 106 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:43:02.172 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:00.773103+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.c scrub starts 2024-02-13T00:43:02.172 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:00.776555+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.c scrub ok 2024-02-13T00:43:02.172 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:01.161995+0000 mon.b (mon.1) 313 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:02.172 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:01.247031+0000 mon.a (mon.0) 1560 : audit [DBG] from='client.? 172.21.15.162:0/2469735035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:02.172 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:01.328798+0000 mon.a (mon.0) 1561 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:02.172 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:01.332837+0000 mon.a (mon.0) 1562 : audit [INF] from='client.? 172.21.15.162:0/410125232' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:02.172 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:01.332926+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-21", "mode": "writeback"}]': finished 2024-02-13T00:43:02.172 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:01.405507+0000 mon.b (mon.1) 314 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:02.173 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:01.586929+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e226: 8 total, 8 up, 7 in 2024-02-13T00:43:02.173 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:01.591965+0000 mon.a (mon.0) 1565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:02.173 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:00.351449+0000 mgr.y (mgr.14544) 240 : cluster [DBG] pgmap v308: 356 pgs: 6 active+clean+snaptrim, 32 unknown, 10 active+clean+snaptrim_wait, 308 active+clean; 9.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:43:02.173 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:00.633224+0000 osd.5 (osd.5) 105 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:43:02.173 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:00.636794+0000 osd.5 (osd.5) 106 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:43:02.173 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:00.773103+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.c scrub starts 2024-02-13T00:43:02.173 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:00.776555+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.c scrub ok 2024-02-13T00:43:02.173 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:01.161995+0000 mon.b (mon.1) 313 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:02.173 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:01.247031+0000 mon.a (mon.0) 1560 : audit [DBG] from='client.? 172.21.15.162:0/2469735035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:02.173 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:01.328798+0000 mon.a (mon.0) 1561 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:02.173 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:01.332837+0000 mon.a (mon.0) 1562 : audit [INF] from='client.? 172.21.15.162:0/410125232' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:02.173 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:01.332926+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-21", "mode": "writeback"}]': finished 2024-02-13T00:43:02.174 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:01.405507+0000 mon.b (mon.1) 314 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:02.174 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:01.586929+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e226: 8 total, 8 up, 7 in 2024-02-13T00:43:02.174 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:01.591965+0000 mon.a (mon.0) 1565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.242+0000 7f0eb0811700 1 -- 172.21.15.162:0/743396148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0eac0fa440 msgr2=0x7f0eac0f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.242+0000 7f0eb0811700 1 --2- 172.21.15.162:0/743396148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0eac0fa440 0x7f0eac0f7e20 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7f0e9c004870 tx=0x7f0e9c009cb0).stop 2024-02-13T00:43:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.243+0000 7f0eb0811700 1 -- 172.21.15.162:0/743396148 shutdown_connections 2024-02-13T00:43:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.243+0000 7f0eb0811700 1 --2- 172.21.15.162:0/743396148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac0f8500 0x7f0eac104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.243+0000 7f0eb0811700 1 --2- 172.21.15.162:0/743396148 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0eac0f9b70 0x7f0eac0f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.243+0000 7f0eb0811700 1 --2- 172.21.15.162:0/743396148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0eac0fa440 0x7f0eac0f7e20 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.243+0000 7f0eb0811700 1 -- 172.21.15.162:0/743396148 >> 172.21.15.162:0/743396148 conn(0x7f0eac0f38c0 msgr2=0x7f0eac0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.243+0000 7f0eb0811700 1 -- 172.21.15.162:0/743396148 shutdown_connections 2024-02-13T00:43:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.243+0000 7f0eb0811700 1 -- 172.21.15.162:0/743396148 wait complete. 2024-02-13T00:43:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.243+0000 7f0eb0811700 1 librados: starting msgr at 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.243+0000 7f0eb0811700 1 librados: starting objecter 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.243+0000 7f0eb0811700 1 Processor -- start 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eb0811700 1 -- start start 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eb0811700 1 librados: setting wanted keys 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eb0811700 1 librados: calling monclient init 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eb0811700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac0f8500 0x7f0eac06cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eb0811700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0eac0f9b70 0x7f0eac06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eb0811700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0eac0fa440 0x7f0eac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eb0811700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac06be10 con 0x7f0eac0f8500 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eb0811700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac06bf80 con 0x7f0eac0fa440 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eb0811700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac11ae70 con 0x7f0eac0f9b70 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eaa59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac0f8500 0x7f0eac06cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eaa59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac0f8500 0x7f0eac06cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41358/0 (socket says 172.21.15.162:41358) 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eaa59c700 1 -- 172.21.15.162:0/2469735035 learned_addr learned my addr 172.21.15.162:0/2469735035 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0ea9d9b700 1 --2- 172.21.15.162:0/2469735035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0eac0f9b70 0x7f0eac06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eaad9d700 1 --2- 172.21.15.162:0/2469735035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0eac0fa440 0x7f0eac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eaa59c700 1 -- 172.21.15.162:0/2469735035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0eac0f9b70 msgr2=0x7f0eac06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eaa59c700 1 --2- 172.21.15.162:0/2469735035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0eac0f9b70 0x7f0eac06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eaa59c700 1 -- 172.21.15.162:0/2469735035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0eac0fa440 msgr2=0x7f0eac06b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eaa59c700 1 --2- 172.21.15.162:0/2469735035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0eac0fa440 0x7f0eac06b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eaa59c700 1 -- 172.21.15.162:0/2469735035 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea40097c0 con 0x7f0eac0f8500 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0ea9d9b700 1 --2- 172.21.15.162:0/2469735035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0eac0f9b70 0x7f0eac06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0eaa59c700 1 --2- 172.21.15.162:0/2469735035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac0f8500 0x7f0eac06cb80 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7f0ea00028a0 tx=0x7f0ea0009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0e9b7fe700 10 client.?.objecter ms_handle_connect 0x7f0eac0f8500 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0e9b7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.244+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0007b40 con 0x7f0eac0f8500 2024-02-13T00:43:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.245+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ea000f040 con 0x7f0eac0f8500 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.245+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e9c005040 con 0x7f0eac0f8500 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.245+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea00147f0 con 0x7f0eac0f8500 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.245+0000 7f0eb0811700 10 client.26757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.245+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac11b480 con 0x7f0eac0f8500 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.245+0000 7f0eb0811700 1 librados: init done 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.245+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0eac0475e0 con 0x7f0eac0f8500 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.246+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0ea0014be0 con 0x7f0eac0f8500 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0e9b7fe700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0e94062c90 0x7f0e94065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(225..225 src has 52..225) v4 ==== 9763+0+0 (secure 0 0 0) 0x7f0ea007d610 con 0x7f0eac0f8500 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 osd_map(225..225 src has 52..225) v4 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0e9b7fe700 3 client.26757.objecter handle_osd_map got epochs [225,225] > 0 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0e9b7fe700 3 client.26757.objecter handle_osd_map decoding full epoch 225 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0ea9d9b700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0e94062c90 0x7f0e94065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0e9b7fe700 20 client.26757.objecter dump_active .. 0 homeless 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v225) v1 ==== 54+0+16 (secure 0 0 0) 0x7f0ea007dab0 con 0x7f0eac0f8500 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0e9b7fe700 10 client.26757.objecter ms_handle_connect 0x7f0e94062c90 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0ea9d9b700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0e94062c90 0x7f0e94065150 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 rx=0x7f0e9c0043d0 tx=0x7f0e9c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eb0811700 10 client.26757.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 1 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0eac028590 0x7f0eac0feb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac107250 osd=0 3 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac107250 osd=0 3 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac107250 osd=0 3 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 1 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac107250 osd=0 3 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 0 1 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac107250 osd=0 4 2024-02-13T00:43:02.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eb0811700 10 client.26757.objecter _send_command 1 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0eac1001d0 con 0x7f0eac028590 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.247+0000 7f0eaad9d700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0eac028590 0x7f0eac0feb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.248+0000 7f0eaad9d700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0eac028590 0x7f0eac0feb60 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.248+0000 7f0e9b7fe700 10 client.26757.objecter ms_handle_connect 0x7f0eac028590 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.248+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0eac1001d0 con 0x7f0eac028590 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.248+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.248+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 1 = system:0 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.248+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac107250 osd=0 4 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.248+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 0 1 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.263+0000 7f0eb0811700 10 client.26757.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.263+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 2 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.263+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac107250 osd=0 3 2024-02-13T00:43:02.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.263+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac107250 osd=0 3 2024-02-13T00:43:02.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.263+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac107250 osd=0 3 2024-02-13T00:43:02.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.263+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 2 2024-02-13T00:43:02.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.263+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac107250 osd=0 3 2024-02-13T00:43:02.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.263+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 0 2 2024-02-13T00:43:02.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.263+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac107250 osd=0 4 2024-02-13T00:43:02.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.263+0000 7f0eb0811700 10 client.26757.objecter _send_command 2 2024-02-13T00:43:02.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.263+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0eac100340 con 0x7f0eac028590 2024-02-13T00:43:02.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0eac100340 con 0x7f0eac028590 2024-02-13T00:43:02.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:02.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac107250 osd=0 4 2024-02-13T00:43:02.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 0 2 2024-02-13T00:43:02.282 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:43:02.282 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:43:02.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eb0811700 10 client.26757.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:02.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 3 2024-02-13T00:43:02.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6805/150599780 conn(0x7f0eac0ff6d0 0x7f0eac0ffab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:02.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac0ff050 osd=1 3 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac0ff050 osd=1 3 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac0ff050 osd=1 3 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 3 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac0ff050 osd=1 3 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 1 3 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac0ff050 osd=1 4 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eb0811700 10 client.26757.objecter _send_command 3 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0eac0fffa0 con 0x7f0eac0ff6d0 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eaa59c700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6805/150599780 conn(0x7f0eac0ff6d0 0x7f0eac0ffab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0e9b7fe700 10 client.26757.objecter ms_handle_connect 0x7f0eac0ff6d0 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.265+0000 7f0eaa59c700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6805/150599780 conn(0x7f0eac0ff6d0 0x7f0eac0ffab0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.266+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0ea005cac0 con 0x7f0eac0ff6d0 2024-02-13T00:43:02.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.266+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.266+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 3 = system:0 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.266+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac0ff050 osd=1 4 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.266+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 1 3 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.280+0000 7f0eb0811700 10 client.26757.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.280+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 4 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.280+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac0ff050 osd=1 3 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.280+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac0ff050 osd=1 3 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.280+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac0ff050 osd=1 3 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.280+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 4 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.280+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac0ff050 osd=1 3 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.280+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 1 4 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.280+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac0ff050 osd=1 4 2024-02-13T00:43:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.280+0000 7f0eb0811700 10 client.26757.objecter _send_command 4 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.280+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0eac059400 con 0x7f0eac0ff6d0 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0eac059400 con 0x7f0eac0ff6d0 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac0ff050 osd=1 4 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 1 4 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0eb0811700 10 client.26757.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 5 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0eac1208a0 0x7f0eac122c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac027be0 osd=2 3 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac027be0 osd=2 3 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac027be0 osd=2 3 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 5 2024-02-13T00:43:02.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac027be0 osd=2 3 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 2 5 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac027be0 osd=2 4 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0eb0811700 10 client.26757.objecter _send_command 5 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0eac028260 con 0x7f0eac1208a0 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0ea9d9b700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0eac1208a0 0x7f0eac122c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0e9b7fe700 10 client.26757.objecter ms_handle_connect 0x7f0eac1208a0 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.282+0000 7f0ea9d9b700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0eac1208a0 0x7f0eac122c90 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.283+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0eac028260 con 0x7f0eac1208a0 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.283+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.283+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 5 = system:0 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.283+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac027be0 osd=2 4 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.283+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 2 5 2024-02-13T00:43:02.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.297+0000 7f0eb0811700 10 client.26757.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.297+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 6 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.297+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac027be0 osd=2 3 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.297+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac027be0 osd=2 3 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.297+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac027be0 osd=2 3 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.297+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 6 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.297+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac027be0 osd=2 3 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.297+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 2 6 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.297+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac027be0 osd=2 4 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.297+0000 7f0eb0811700 10 client.26757.objecter _send_command 6 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.297+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0eac027650 con 0x7f0eac1208a0 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0eac027650 con 0x7f0eac1208a0 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac027be0 osd=2 4 2024-02-13T00:43:02.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 2 6 2024-02-13T00:43:02.291 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:43:02.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eb0811700 10 client.26757.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:02.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 7 2024-02-13T00:43:02.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6813/604971011 conn(0x7f0eac120400 0x7f0eac126310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:02.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac027a60 osd=3 3 2024-02-13T00:43:02.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac027a60 osd=3 3 2024-02-13T00:43:02.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac027a60 osd=3 3 2024-02-13T00:43:02.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 7 2024-02-13T00:43:02.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac027a60 osd=3 3 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 3 7 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac027a60 osd=3 4 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eb0811700 10 client.26757.objecter _send_command 7 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0eac128f60 con 0x7f0eac120400 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eaad9d700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6813/604971011 conn(0x7f0eac120400 0x7f0eac126310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0e9b7fe700 10 client.26757.objecter ms_handle_connect 0x7f0eac120400 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.299+0000 7f0eaad9d700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6813/604971011 conn(0x7f0eac120400 0x7f0eac126310 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.300+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0eac128f60 con 0x7f0eac120400 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.300+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.300+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 7 = system:0 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.300+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac027a60 osd=3 4 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.300+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 3 7 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.314+0000 7f0eb0811700 10 client.26757.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:02.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.314+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 8 2024-02-13T00:43:02.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.314+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac027a60 osd=3 3 2024-02-13T00:43:02.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.314+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac027a60 osd=3 3 2024-02-13T00:43:02.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.314+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac027a60 osd=3 3 2024-02-13T00:43:02.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.314+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 8 2024-02-13T00:43:02.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.314+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac027a60 osd=3 3 2024-02-13T00:43:02.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.314+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 3 8 2024-02-13T00:43:02.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.314+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac027a60 osd=3 4 2024-02-13T00:43:02.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.314+0000 7f0eb0811700 10 client.26757.objecter _send_command 8 2024-02-13T00:43:02.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.314+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0eac11f960 con 0x7f0eac120400 2024-02-13T00:43:02.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0eac11f960 con 0x7f0eac120400 2024-02-13T00:43:02.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:02.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.295 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:43:02.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac027a60 osd=3 4 2024-02-13T00:43:02.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 3 8 2024-02-13T00:43:02.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eb0811700 10 client.26757.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:02.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 9 2024-02-13T00:43:02.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0eac11fb00 0x7f0eac127980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:02.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac11f960 osd=4 3 2024-02-13T00:43:02.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac11f960 osd=4 3 2024-02-13T00:43:02.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac11f960 osd=4 3 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 9 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac11f960 osd=4 3 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 4 9 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac11f960 osd=4 4 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eb0811700 10 client.26757.objecter _send_command 9 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0eac123f90 con 0x7f0eac11fb00 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.316+0000 7f0eaa59c700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0eac11fb00 0x7f0eac127980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.317+0000 7f0e9b7fe700 10 client.26757.objecter ms_handle_connect 0x7f0eac11fb00 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.317+0000 7f0eaa59c700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0eac11fb00 0x7f0eac127980 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.317+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0eac123f90 con 0x7f0eac11fb00 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.317+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.317+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 9 = system:0 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.317+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac11f960 osd=4 4 2024-02-13T00:43:02.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.317+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 4 9 2024-02-13T00:43:02.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.336+0000 7f0eb0811700 10 client.26757.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:02.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.336+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 10 2024-02-13T00:43:02.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.336+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac11f960 osd=4 3 2024-02-13T00:43:02.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.336+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac11f960 osd=4 3 2024-02-13T00:43:02.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.336+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac11f960 osd=4 3 2024-02-13T00:43:02.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.336+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 10 2024-02-13T00:43:02.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.336+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac11f960 osd=4 3 2024-02-13T00:43:02.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.336+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 4 10 2024-02-13T00:43:02.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.336+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac11f960 osd=4 4 2024-02-13T00:43:02.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.336+0000 7f0eb0811700 10 client.26757.objecter _send_command 10 2024-02-13T00:43:02.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.336+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0eac123f90 con 0x7f0eac11fb00 2024-02-13T00:43:02.298 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:43:02.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0eac123f90 con 0x7f0eac11fb00 2024-02-13T00:43:02.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:02.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac11f960 osd=4 4 2024-02-13T00:43:02.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 4 10 2024-02-13T00:43:02.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0eb0811700 10 client.26757.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:02.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 11 2024-02-13T00:43:02.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0eac128380 0x7f0eac1287e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac123f90 osd=5 3 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac123f90 osd=5 3 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac123f90 osd=5 3 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 11 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac123f90 osd=5 3 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 5 11 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac123f90 osd=5 4 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0eb0811700 10 client.26757.objecter _send_command 11 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0eac128d40 con 0x7f0eac128380 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.341+0000 7f0ea9d9b700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0eac128380 0x7f0eac1287e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.342+0000 7f0e9b7fe700 10 client.26757.objecter ms_handle_connect 0x7f0eac128380 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.342+0000 7f0ea9d9b700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0eac128380 0x7f0eac1287e0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.342+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0eac128d40 con 0x7f0eac128380 2024-02-13T00:43:02.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.342+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:43:02.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.342+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 11 = system:0 2024-02-13T00:43:02.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.342+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac123f90 osd=5 4 2024-02-13T00:43:02.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.342+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 5 11 2024-02-13T00:43:02.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.364+0000 7f0eb0811700 10 client.26757.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:02.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.364+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 12 2024-02-13T00:43:02.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.364+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac123f90 osd=5 3 2024-02-13T00:43:02.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.364+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac123f90 osd=5 3 2024-02-13T00:43:02.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.364+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac123f90 osd=5 3 2024-02-13T00:43:02.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.364+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 12 2024-02-13T00:43:02.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.364+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac123f90 osd=5 3 2024-02-13T00:43:02.301 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:43:02.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.364+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 5 12 2024-02-13T00:43:02.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.364+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac123f90 osd=5 4 2024-02-13T00:43:02.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.364+0000 7f0eb0811700 10 client.26757.objecter _send_command 12 2024-02-13T00:43:02.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.364+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0eac128d40 con 0x7f0eac128380 2024-02-13T00:43:02.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0eac128d40 con 0x7f0eac128380 2024-02-13T00:43:02.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:02.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac123f90 osd=5 4 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 5 12 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0eb0811700 10 client.26757.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 13 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0eac12d600 0x7f0eac12fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac128d40 osd=6 3 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac128d40 osd=6 3 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac128d40 osd=6 3 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 13 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac128d40 osd=6 3 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 6 13 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac128d40 osd=6 4 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0eb0811700 10 client.26757.objecter _send_command 13 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.367+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0eac1327e0 con 0x7f0eac12d600 2024-02-13T00:43:02.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.368+0000 7f0eaad9d700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0eac12d600 0x7f0eac12fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.368+0000 7f0e9b7fe700 10 client.26757.objecter ms_handle_connect 0x7f0eac12d600 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.368+0000 7f0eaad9d700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0eac12d600 0x7f0eac12fa10 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.370+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0eac1327e0 con 0x7f0eac12d600 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.370+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.370+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 13 = system:0 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.370+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac128d40 osd=6 4 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.370+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 6 13 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.390+0000 7f0eb0811700 10 client.26757.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.391+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 14 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.391+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac128d40 osd=6 3 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.391+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac128d40 osd=6 3 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.391+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac128d40 osd=6 3 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.391+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 14 2024-02-13T00:43:02.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.391+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac128d40 osd=6 3 2024-02-13T00:43:02.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.391+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 6 14 2024-02-13T00:43:02.304 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:43:02.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.391+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac128d40 osd=6 4 2024-02-13T00:43:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.391+0000 7f0eb0811700 10 client.26757.objecter _send_command 14 2024-02-13T00:43:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.391+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0eac1327e0 con 0x7f0eac12d600 2024-02-13T00:43:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.392+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0eac1327e0 con 0x7f0eac12d600 2024-02-13T00:43:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.392+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.392+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.392+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac128d40 osd=6 4 2024-02-13T00:43:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.392+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 6 14 2024-02-13T00:43:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eb0811700 10 client.26757.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 15 2024-02-13T00:43:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0eac131b20 0x7f0eac131f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac1327e0 osd=7 3 2024-02-13T00:43:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac1327e0 osd=7 3 2024-02-13T00:43:02.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac1327e0 osd=7 3 2024-02-13T00:43:02.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 15 2024-02-13T00:43:02.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac1327e0 osd=7 3 2024-02-13T00:43:02.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 7 15 2024-02-13T00:43:02.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac1327e0 osd=7 4 2024-02-13T00:43:02.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eb0811700 10 client.26757.objecter _send_command 15 2024-02-13T00:43:02.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0eac1325f0 con 0x7f0eac131b20 2024-02-13T00:43:02.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eaa59c700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0eac131b20 0x7f0eac131f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:02.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0e9b7fe700 10 client.26757.objecter ms_handle_connect 0x7f0eac131b20 2024-02-13T00:43:02.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.393+0000 7f0eaa59c700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0eac131b20 0x7f0eac131f80 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:02.306 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:00.351449+0000 mgr.y (mgr.14544) 240 : cluster [DBG] pgmap v308: 356 pgs: 6 active+clean+snaptrim, 32 unknown, 10 active+clean+snaptrim_wait, 308 active+clean; 9.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:43:02.307 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:00.633224+0000 osd.5 (osd.5) 105 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:43:02.307 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:00.636794+0000 osd.5 (osd.5) 106 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:43:02.307 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:00.773103+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.c scrub starts 2024-02-13T00:43:02.307 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:00.776555+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.c scrub ok 2024-02-13T00:43:02.307 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:01.161995+0000 mon.b (mon.1) 313 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:02.307 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:01.247031+0000 mon.a (mon.0) 1560 : audit [DBG] from='client.? 172.21.15.162:0/2469735035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:02.307 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:01.328798+0000 mon.a (mon.0) 1561 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:02.307 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:01.332837+0000 mon.a (mon.0) 1562 : audit [INF] from='client.? 172.21.15.162:0/410125232' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-83985-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:02.307 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:01.332926+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-21", "mode": "writeback"}]': finished 2024-02-13T00:43:02.307 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:01.405507+0000 mon.b (mon.1) 314 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:02.307 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:01.586929+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e226: 8 total, 8 up, 7 in 2024-02-13T00:43:02.307 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:01.591965+0000 mon.a (mon.0) 1565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:02.308 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:43:02.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.394+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0ea0057040 con 0x7f0eac131b20 2024-02-13T00:43:02.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.394+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:43:02.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.394+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 15 = system:0 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.394+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac1327e0 osd=7 4 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.394+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 7 15 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.410+0000 7f0eb0811700 10 client.26757.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.410+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign -1 16 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.410+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac1327e0 osd=7 3 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.410+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac1327e0 osd=7 3 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.410+0000 7f0eb0811700 20 client.26757.objecter _get_session s=0x7f0eac1327e0 osd=7 3 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.410+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_remove -1 16 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.410+0000 7f0eb0811700 20 client.26757.objecter get_session s=0x7f0eac1327e0 osd=7 3 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.410+0000 7f0eb0811700 15 client.26757.objecter _session_command_op_assign 7 16 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.410+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac1327e0 osd=7 4 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.410+0000 7f0eb0811700 10 client.26757.objecter _send_command 16 2024-02-13T00:43:02.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.410+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0eac134a40 con 0x7f0eac131b20 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0e9b7fe700 1 -- 172.21.15.162:0/2469735035 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0eac134a40 con 0x7f0eac131b20 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0e9b7fe700 10 client.26757.objecter ms_dispatch 0x7f0eac001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0e9b7fe700 10 client.26757.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0e9b7fe700 20 client.26757.objecter put_session s=0x7f0eac1327e0 osd=7 4 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0e9b7fe700 15 client.26757.objecter _session_command_op_remove 7 16 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 10 librados: watch_flush enter 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 10 librados: watch_flush exit 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 10 client.26757.objecter close_session for osd.0 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0eac028590 msgr2=0x7f0eac0feb60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0eac028590 0x7f0eac0feb60 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac107250 osd=0 1 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 10 client.26757.objecter close_session for osd.1 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6805/150599780 conn(0x7f0eac0ff6d0 msgr2=0x7f0eac0ffab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6805/150599780 conn(0x7f0eac0ff6d0 0x7f0eac0ffab0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac0ff050 osd=1 1 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 10 client.26757.objecter close_session for osd.2 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0eac1208a0 msgr2=0x7f0eac122c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0eac1208a0 0x7f0eac122c90 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac027be0 osd=2 1 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 10 client.26757.objecter close_session for osd.3 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6813/604971011 conn(0x7f0eac120400 msgr2=0x7f0eac126310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6813/604971011 conn(0x7f0eac120400 0x7f0eac126310 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eaad9d700 1 -- 172.21.15.162:0/2469735035 reap_dead start 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac027a60 osd=3 1 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 10 client.26757.objecter close_session for osd.4 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0eac11fb00 msgr2=0x7f0eac127980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0eac11fb00 0x7f0eac127980 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eaad9d700 1 -- 172.21.15.162:0/2469735035 reap_dead start 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac11f960 osd=4 1 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 10 client.26757.objecter close_session for osd.5 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0eac128380 msgr2=0x7f0eac1287e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0eac128380 0x7f0eac1287e0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac123f90 osd=5 1 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 10 client.26757.objecter close_session for osd.6 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0eac12d600 msgr2=0x7f0eac12fa10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0eac12d600 0x7f0eac12fa10 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.412+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac128d40 osd=6 1 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 10 client.26757.objecter close_session for osd.7 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0eac131b20 msgr2=0x7f0eac131f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0eac131b20 0x7f0eac131f80 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:02.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 20 client.26757.objecter put_session s=0x7f0eac1327e0 osd=7 1 2024-02-13T00:43:02.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 20 client.26757.objecter shutdown clearing up homeless session... 2024-02-13T00:43:02.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 10 client.26757.objecter successfully canceled tick 2024-02-13T00:43:02.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0e94062c90 msgr2=0x7f0e94065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0e94062c90 0x7f0e94065150 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 rx=0x7f0e9c0043d0 tx=0x7f0e9c00b040).stop 2024-02-13T00:43:02.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac0f8500 msgr2=0x7f0eac06cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:02.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 1 --2- 172.21.15.162:0/2469735035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac0f8500 0x7f0eac06cb80 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7f0ea00028a0 tx=0x7f0ea0009f60).stop 2024-02-13T00:43:02.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eaad9d700 1 -- 172.21.15.162:0/2469735035 reap_dead start 2024-02-13T00:43:02.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 shutdown_connections 2024-02-13T00:43:02.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 >> 172.21.15.162:0/2469735035 conn(0x7f0eac0f38c0 msgr2=0x7f0eac0f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:02.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 shutdown_connections 2024-02-13T00:43:02.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 1 -- 172.21.15.162:0/2469735035 wait complete. 2024-02-13T00:43:02.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:01.413+0000 7f0eb0811700 1 librados: shutdown 2024-02-13T00:43:02.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:43:02.483 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:02.486 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:02.487 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:02.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:02.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:02.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:02.625 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:02.646 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:02.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:02.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:02.780 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:02.800 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:02.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:02.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:02.904 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:02.933 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:03.181 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:43:03.182 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:43:03.183 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:03.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:01.649334+0000 osd.5 (osd.5) 107 : cluster [DBG] 18.c scrub starts 2024-02-13T00:43:03.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:01.652238+0000 osd.5 (osd.5) 108 : cluster [DBG] 18.c scrub ok 2024-02-13T00:43:03.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:01.774785+0000 osd.3 (osd.3) 137 : cluster [DBG] 18.10 scrub starts 2024-02-13T00:43:03.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:01.777670+0000 osd.3 (osd.3) 138 : cluster [DBG] 18.10 scrub ok 2024-02-13T00:43:03.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:02.163454+0000 mon.b (mon.1) 315 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:03.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:02.335352+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-83977-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-83977-32"}]': finished 2024-02-13T00:43:03.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:02.335471+0000 mon.a (mon.0) 1567 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:03.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:02.335528+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e227: 8 total, 8 up, 7 in 2024-02-13T00:43:03.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:02.346733+0000 mon.b (mon.1) 316 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21"}]: dispatch 2024-02-13T00:43:03.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:02.349091+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21"}]: dispatch 2024-02-13T00:43:03.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:02.419225+0000 osd.0 (osd.0) 85 : cluster [DBG] 5.3 deep-scrub starts 2024-02-13T00:43:03.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:02.423353+0000 osd.0 (osd.0) 86 : cluster [DBG] 5.3 deep-scrub ok 2024-02-13T00:43:03.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:01.649334+0000 osd.5 (osd.5) 107 : cluster [DBG] 18.c scrub starts 2024-02-13T00:43:03.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:01.652238+0000 osd.5 (osd.5) 108 : cluster [DBG] 18.c scrub ok 2024-02-13T00:43:03.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:01.774785+0000 osd.3 (osd.3) 137 : cluster [DBG] 18.10 scrub starts 2024-02-13T00:43:03.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:01.777670+0000 osd.3 (osd.3) 138 : cluster [DBG] 18.10 scrub ok 2024-02-13T00:43:03.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:02.163454+0000 mon.b (mon.1) 315 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:03.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:02.335352+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-83977-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-83977-32"}]': finished 2024-02-13T00:43:03.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:02.335471+0000 mon.a (mon.0) 1567 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:03.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:02.335528+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e227: 8 total, 8 up, 7 in 2024-02-13T00:43:03.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:02.346733+0000 mon.b (mon.1) 316 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21"}]: dispatch 2024-02-13T00:43:03.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:02.349091+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21"}]: dispatch 2024-02-13T00:43:03.188 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:02.419225+0000 osd.0 (osd.0) 85 : cluster [DBG] 5.3 deep-scrub starts 2024-02-13T00:43:03.188 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:02.423353+0000 osd.0 (osd.0) 86 : cluster [DBG] 5.3 deep-scrub ok 2024-02-13T00:43:03.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:03.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:03.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:03.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:01.649334+0000 osd.5 (osd.5) 107 : cluster [DBG] 18.c scrub starts 2024-02-13T00:43:03.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:01.652238+0000 osd.5 (osd.5) 108 : cluster [DBG] 18.c scrub ok 2024-02-13T00:43:03.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:01.774785+0000 osd.3 (osd.3) 137 : cluster [DBG] 18.10 scrub starts 2024-02-13T00:43:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:01.777670+0000 osd.3 (osd.3) 138 : cluster [DBG] 18.10 scrub ok 2024-02-13T00:43:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:02.163454+0000 mon.b (mon.1) 315 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:02.335352+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-83977-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-83977-32"}]': finished 2024-02-13T00:43:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:02.335471+0000 mon.a (mon.0) 1567 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:02.335528+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e227: 8 total, 8 up, 7 in 2024-02-13T00:43:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:02.346733+0000 mon.b (mon.1) 316 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21"}]: dispatch 2024-02-13T00:43:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:02.349091+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21"}]: dispatch 2024-02-13T00:43:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:02.419225+0000 osd.0 (osd.0) 85 : cluster [DBG] 5.3 deep-scrub starts 2024-02-13T00:43:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:02.423353+0000 osd.0 (osd.0) 86 : cluster [DBG] 5.3 deep-scrub ok 2024-02-13T00:43:03.322 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:03.350 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:03.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:03.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:03.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:03.484 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:03.503 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:03.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:03.607 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:03.636 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:03.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-13T00:43:04.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:02.351972+0000 mgr.y (mgr.14544) 241 : cluster [DBG] pgmap v311: 332 pgs: 6 active+clean+snaptrim, 8 unknown, 10 active+clean+snaptrim_wait, 308 active+clean; 9.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:43:04.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:03.164520+0000 mon.b (mon.1) 317 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:04.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:03.332393+0000 mon.a (mon.0) 1570 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:04.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:03.341498+0000 mon.a (mon.0) 1571 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21"}]': finished 2024-02-13T00:43:04.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:03.341595+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e228: 8 total, 8 up, 7 in 2024-02-13T00:43:04.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:03.461489+0000 osd.0 (osd.0) 87 : cluster [DBG] 2.1f deep-scrub starts 2024-02-13T00:43:04.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:03.464394+0000 osd.0 (osd.0) 88 : cluster [DBG] 2.1f deep-scrub ok 2024-02-13T00:43:04.162 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:02.351972+0000 mgr.y (mgr.14544) 241 : cluster [DBG] pgmap v311: 332 pgs: 6 active+clean+snaptrim, 8 unknown, 10 active+clean+snaptrim_wait, 308 active+clean; 9.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:43:04.162 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:03.164520+0000 mon.b (mon.1) 317 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:04.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:03.332393+0000 mon.a (mon.0) 1570 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:04.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:03.341498+0000 mon.a (mon.0) 1571 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21"}]': finished 2024-02-13T00:43:04.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:03.341595+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e228: 8 total, 8 up, 7 in 2024-02-13T00:43:04.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:03.461489+0000 osd.0 (osd.0) 87 : cluster [DBG] 2.1f deep-scrub starts 2024-02-13T00:43:04.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:03.464394+0000 osd.0 (osd.0) 88 : cluster [DBG] 2.1f deep-scrub ok 2024-02-13T00:43:04.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:02.351972+0000 mgr.y (mgr.14544) 241 : cluster [DBG] pgmap v311: 332 pgs: 6 active+clean+snaptrim, 8 unknown, 10 active+clean+snaptrim_wait, 308 active+clean; 9.9 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:43:04.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:03.164520+0000 mon.b (mon.1) 317 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:04.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:03.332393+0000 mon.a (mon.0) 1570 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:04.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:03.341498+0000 mon.a (mon.0) 1571 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-21"}]': finished 2024-02-13T00:43:04.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:03.341595+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e228: 8 total, 8 up, 7 in 2024-02-13T00:43:04.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:03.461489+0000 osd.0 (osd.0) 87 : cluster [DBG] 2.1f deep-scrub starts 2024-02-13T00:43:04.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:03.464394+0000 osd.0 (osd.0) 88 : cluster [DBG] 2.1f deep-scrub ok 2024-02-13T00:43:04.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:04.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:04.789 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:04.816 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:04.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:43:05.014 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:05.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:05.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:05.150 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:05.172 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:05.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:05.276 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:05.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:03.663060+0000 osd.5 (osd.5) 109 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:43:05.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:03.665969+0000 osd.5 (osd.5) 110 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:43:05.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:04.165702+0000 mon.b (mon.1) 318 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:04.246775+0000 osd.1 (osd.1) 103 : cluster [DBG] 6.4 deep-scrub starts 2024-02-13T00:43:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:04.249699+0000 osd.1 (osd.1) 104 : cluster [DBG] 6.4 deep-scrub ok 2024-02-13T00:43:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:04.333526+0000 mon.a (mon.0) 1573 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:04.345258+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e229: 8 total, 8 up, 7 in 2024-02-13T00:43:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:04.349059+0000 mon.b (mon.1) 319 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:04.352821+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:04.554421+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:43:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:04.557467+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:43:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:04.967682+0000 mon.c (mon.2) 211 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]: dispatch 2024-02-13T00:43:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:04.968084+0000 mon.a (mon.0) 1576 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]: dispatch 2024-02-13T00:43:05.305 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:05.324 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:05.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:03.663060+0000 osd.5 (osd.5) 109 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:03.665969+0000 osd.5 (osd.5) 110 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:04.165702+0000 mon.b (mon.1) 318 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:04.246775+0000 osd.1 (osd.1) 103 : cluster [DBG] 6.4 deep-scrub starts 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:04.249699+0000 osd.1 (osd.1) 104 : cluster [DBG] 6.4 deep-scrub ok 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:04.333526+0000 mon.a (mon.0) 1573 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:04.345258+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e229: 8 total, 8 up, 7 in 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:04.349059+0000 mon.b (mon.1) 319 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:04.352821+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:04.554421+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:04.557467+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:04.967682+0000 mon.c (mon.2) 211 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]: dispatch 2024-02-13T00:43:05.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:04.968084+0000 mon.a (mon.0) 1576 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]: dispatch 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:03.663060+0000 osd.5 (osd.5) 109 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:03.665969+0000 osd.5 (osd.5) 110 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:04.165702+0000 mon.b (mon.1) 318 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:04.246775+0000 osd.1 (osd.1) 103 : cluster [DBG] 6.4 deep-scrub starts 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:04.249699+0000 osd.1 (osd.1) 104 : cluster [DBG] 6.4 deep-scrub ok 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:04.333526+0000 mon.a (mon.0) 1573 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:04.345258+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e229: 8 total, 8 up, 7 in 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:04.349059+0000 mon.b (mon.1) 319 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:04.352821+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:04.554421+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:04.557467+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:04.967682+0000 mon.c (mon.2) 211 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]: dispatch 2024-02-13T00:43:05.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:04.968084+0000 mon.a (mon.0) 1576 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]: dispatch 2024-02-13T00:43:05.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:05.456 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:05.529 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:05.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:05.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:05.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:05.661 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:05.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:43:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:43:06.010 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:06.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:06.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:06.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:06.141 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:04.356187+0000 mgr.y (mgr.14544) 242 : cluster [DBG] pgmap v314: 228 pgs: 6 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 212 active+clean; 9.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:43:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:05.167227+0000 mon.b (mon.1) 320 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:05.349167+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]': finished 2024-02-13T00:43:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:05.349298+0000 mon.a (mon.0) 1578 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]': finished 2024-02-13T00:43:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:05.349368+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e230: 8 total, 8 up, 7 in 2024-02-13T00:43:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:05.364435+0000 mon.c (mon.2) 212 : audit [INF] from='client.? 172.21.15.162:0/2406699123' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:05.368970+0000 mon.a (mon.0) 1580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:05.370172+0000 mon.b (mon.1) 321 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:05.370982+0000 mon.b (mon.1) 322 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:05.371040+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:06.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:05.371573+0000 mon.a (mon.0) 1582 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:06.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:05.601572+0000 osd.2 (osd.2) 103 : cluster [DBG] 5.1f scrub starts 2024-02-13T00:43:06.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:05.604456+0000 osd.2 (osd.2) 104 : cluster [DBG] 5.1f scrub ok 2024-02-13T00:43:06.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:43:06] "GET /metrics HTTP/1.1" 200 193238 "" "Prometheus/2.33.4" 2024-02-13T00:43:06.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:04.356187+0000 mgr.y (mgr.14544) 242 : cluster [DBG] pgmap v314: 228 pgs: 6 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 212 active+clean; 9.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:43:06.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:05.167227+0000 mon.b (mon.1) 320 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:06.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:05.349167+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]': finished 2024-02-13T00:43:06.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:05.349298+0000 mon.a (mon.0) 1578 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]': finished 2024-02-13T00:43:06.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:05.349368+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e230: 8 total, 8 up, 7 in 2024-02-13T00:43:06.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:05.364435+0000 mon.c (mon.2) 212 : audit [INF] from='client.? 172.21.15.162:0/2406699123' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:06.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:05.368970+0000 mon.a (mon.0) 1580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:06.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:05.370172+0000 mon.b (mon.1) 321 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:06.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:05.370982+0000 mon.b (mon.1) 322 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:06.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:05.371040+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:06.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:05.371573+0000 mon.a (mon.0) 1582 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:06.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:05.601572+0000 osd.2 (osd.2) 103 : cluster [DBG] 5.1f scrub starts 2024-02-13T00:43:06.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:05.604456+0000 osd.2 (osd.2) 104 : cluster [DBG] 5.1f scrub ok 2024-02-13T00:43:06.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:04.356187+0000 mgr.y (mgr.14544) 242 : cluster [DBG] pgmap v314: 228 pgs: 6 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 212 active+clean; 9.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:43:06.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:05.167227+0000 mon.b (mon.1) 320 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:06.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:05.349167+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-83977-32"}]': finished 2024-02-13T00:43:06.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:05.349298+0000 mon.a (mon.0) 1578 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]': finished 2024-02-13T00:43:06.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:05.349368+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e230: 8 total, 8 up, 7 in 2024-02-13T00:43:06.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:05.364435+0000 mon.c (mon.2) 212 : audit [INF] from='client.? 172.21.15.162:0/2406699123' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:06.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:05.368970+0000 mon.a (mon.0) 1580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:06.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:05.370172+0000 mon.b (mon.1) 321 : audit [INF] from='client.? 172.21.15.162:0/4173845195' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:06.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:05.370982+0000 mon.b (mon.1) 322 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:06.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:05.371040+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]: dispatch 2024-02-13T00:43:06.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:05.371573+0000 mon.a (mon.0) 1582 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:06.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:05.601572+0000 osd.2 (osd.2) 103 : cluster [DBG] 5.1f scrub starts 2024-02-13T00:43:06.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:05.604456+0000 osd.2 (osd.2) 104 : cluster [DBG] 5.1f scrub ok 2024-02-13T00:43:06.692 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:06.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:06.796 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:06.823 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:07.021 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:07.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:07.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:07.157 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:07.178 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:07.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:07.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.250+0000 7f737cc20700 1 -- 172.21.15.162:0/1397580667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73780f9b70 msgr2=0x7f73780f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:07.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.250+0000 7f737cc20700 1 --2- 172.21.15.162:0/1397580667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73780f9b70 0x7f73780f9f50 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 rx=0x7f736c0047b0 tx=0x7f736c009cb0).stop 2024-02-13T00:43:07.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.250+0000 7f737cc20700 1 -- 172.21.15.162:0/1397580667 shutdown_connections 2024-02-13T00:43:07.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.250+0000 7f737cc20700 1 --2- 172.21.15.162:0/1397580667 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780f8500 0x7f7378104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.250+0000 7f737cc20700 1 --2- 172.21.15.162:0/1397580667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73780f9b70 0x7f73780f9f50 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.250+0000 7f737cc20700 1 --2- 172.21.15.162:0/1397580667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73780fa440 0x7f73780f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.250+0000 7f737cc20700 1 -- 172.21.15.162:0/1397580667 >> 172.21.15.162:0/1397580667 conn(0x7f73780f38c0 msgr2=0x7f73780f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:07.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.250+0000 7f737cc20700 1 -- 172.21.15.162:0/1397580667 shutdown_connections 2024-02-13T00:43:07.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.250+0000 7f737cc20700 1 -- 172.21.15.162:0/1397580667 wait complete. 2024-02-13T00:43:07.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.250+0000 7f737cc20700 1 librados: starting msgr at 2024-02-13T00:43:07.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.250+0000 7f737cc20700 1 librados: starting objecter 2024-02-13T00:43:07.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f737cc20700 1 Processor -- start 2024-02-13T00:43:07.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f737cc20700 1 -- start start 2024-02-13T00:43:07.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f737cc20700 1 librados: setting wanted keys 2024-02-13T00:43:07.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f737cc20700 1 librados: calling monclient init 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f737cc20700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73780f8500 0x7f737810e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f737cc20700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73780f9b70 0x7f737810e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f737cc20700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780fa440 0x7f7378108390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f737cc20700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7378108880 con 0x7f73780f9b70 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f737cc20700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73781089f0 con 0x7f73780f8500 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f737cc20700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7378108bb0 con 0x7f73780fa440 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f7376d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780fa440 0x7f7378108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f7375d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73780f9b70 0x7f737810e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f7376d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780fa440 0x7f7378108390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48682/0 (socket says 172.21.15.162:48682) 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f7375d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73780f9b70 0x7f737810e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:58714/0 (socket says 172.21.15.162:58714) 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f7376d9d700 1 -- 172.21.15.162:0/3588174164 learned_addr learned my addr 172.21.15.162:0/3588174164 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.251+0000 7f737659c700 1 --2- 172.21.15.162:0/3588174164 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73780f8500 0x7f737810e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f7375d9b700 1 -- 172.21.15.162:0/3588174164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780fa440 msgr2=0x7f7378108390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:07.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f7375d9b700 1 --2- 172.21.15.162:0/3588174164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780fa440 0x7f7378108390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f7375d9b700 1 -- 172.21.15.162:0/3588174164 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73780f8500 msgr2=0x7f737810e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f7375d9b700 1 --2- 172.21.15.162:0/3588174164 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73780f8500 0x7f737810e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f7375d9b700 1 -- 172.21.15.162:0/3588174164 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73700097c0 con 0x7f73780f9b70 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f7376d9d700 1 --2- 172.21.15.162:0/3588174164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780fa440 0x7f7378108390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f7375d9b700 1 --2- 172.21.15.162:0/3588174164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73780f9b70 0x7f737810e6e0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 rx=0x7f7368002800 tx=0x7f7368009ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f73677fe700 10 client.?.objecter ms_handle_connect 0x7f73780f9b70 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f73677fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f73677fe700 1 -- 172.21.15.162:0/3588174164 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7368007b40 con 0x7f73780f9b70 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f737cc20700 1 -- 172.21.15.162:0/3588174164 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736c005040 con 0x7f73780f9b70 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f73677fe700 1 -- 172.21.15.162:0/3588174164 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f736800f040 con 0x7f73780f9b70 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f73677fe700 1 -- 172.21.15.162:0/3588174164 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7368014790 con 0x7f73780f9b70 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f737cc20700 10 client.26781.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:07.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f737cc20700 1 -- 172.21.15.162:0/3588174164 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7378115020 con 0x7f73780f9b70 2024-02-13T00:43:07.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.252+0000 7f737cc20700 1 librados: init done 2024-02-13T00:43:07.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.253+0000 7f737cc20700 1 -- 172.21.15.162:0/3588174164 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73780475e0 con 0x7f73780f9b70 2024-02-13T00:43:07.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.253+0000 7f73677fe700 1 -- 172.21.15.162:0/3588174164 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f7368007ca0 con 0x7f73780f9b70 2024-02-13T00:43:07.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.254+0000 7f73677fe700 1 --2- 172.21.15.162:0/3588174164 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7360062c90 0x7f7360065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:07.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.254+0000 7f73677fe700 1 -- 172.21.15.162:0/3588174164 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(231..231 src has 52..231) v4 ==== 8731+0+0 (secure 0 0 0) 0x7f736807d4d0 con 0x7f73780f9b70 2024-02-13T00:43:07.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.254+0000 7f73677fe700 10 client.26781.objecter ms_dispatch 0x7f7378001550 osd_map(231..231 src has 52..231) v4 2024-02-13T00:43:07.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.254+0000 7f73677fe700 3 client.26781.objecter handle_osd_map got epochs [231,231] > 0 2024-02-13T00:43:07.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.254+0000 7f73677fe700 3 client.26781.objecter handle_osd_map decoding full epoch 231 2024-02-13T00:43:07.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.254+0000 7f737659c700 1 --2- 172.21.15.162:0/3588174164 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7360062c90 0x7f7360065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:07.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.254+0000 7f73677fe700 20 client.26781.objecter dump_active .. 0 homeless 2024-02-13T00:43:07.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.254+0000 7f73677fe700 10 client.26781.objecter ms_handle_connect 0x7f7360062c90 2024-02-13T00:43:07.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.254+0000 7f737659c700 1 --2- 172.21.15.162:0/3588174164 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7360062c90 0x7f7360065150 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 rx=0x7f736c005010 tx=0x7f736c007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:07.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.258+0000 7f73677fe700 1 -- 172.21.15.162:0/3588174164 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7368050780 con 0x7f73780f9b70 2024-02-13T00:43:07.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:07.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:05.446370+0000 mgr.y (mgr.14544) 243 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:07.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:06.168448+0000 mon.b (mon.1) 323 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:06.351429+0000 mon.a (mon.0) 1583 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:06.351513+0000 mon.a (mon.0) 1584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]': finished 2024-02-13T00:43:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:06.351578+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:06.351644+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e231: 8 total, 8 up, 7 in 2024-02-13T00:43:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:06.373536+0000 mon.c (mon.2) 213 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:06.373837+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:06.374363+0000 mon.c (mon.2) 214 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:06.374600+0000 mon.a (mon.0) 1588 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:06.374962+0000 mon.c (mon.2) 215 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-83977-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:06.375189+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-83977-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:06.649872+0000 osd.2 (osd.2) 105 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:43:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:06.651070+0000 osd.2 (osd.2) 106 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:43:07.311 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:07.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:05.446370+0000 mgr.y (mgr.14544) 243 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:07.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:06.168448+0000 mon.b (mon.1) 323 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:07.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:06.351429+0000 mon.a (mon.0) 1583 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:06.351513+0000 mon.a (mon.0) 1584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]': finished 2024-02-13T00:43:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:06.351578+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:06.351644+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e231: 8 total, 8 up, 7 in 2024-02-13T00:43:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:06.373536+0000 mon.c (mon.2) 213 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:06.373837+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:06.374363+0000 mon.c (mon.2) 214 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:06.374600+0000 mon.a (mon.0) 1588 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:06.374962+0000 mon.c (mon.2) 215 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-83977-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:06.375189+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-83977-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:06.649872+0000 osd.2 (osd.2) 105 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:43:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:06.651070+0000 osd.2 (osd.2) 106 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:43:07.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:05.446370+0000 mgr.y (mgr.14544) 243 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:06.168448+0000 mon.b (mon.1) 323 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:06.351429+0000 mon.a (mon.0) 1583 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:06.351513+0000 mon.a (mon.0) 1584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-83977-32"}]': finished 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:06.351578+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:06.351644+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e231: 8 total, 8 up, 7 in 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:06.373536+0000 mon.c (mon.2) 213 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:06.373837+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:06.374363+0000 mon.c (mon.2) 214 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:06.374600+0000 mon.a (mon.0) 1588 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:06.374962+0000 mon.c (mon.2) 215 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-83977-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:06.375189+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-83977-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:06.649872+0000 osd.2 (osd.2) 105 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:43:07.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:06.651070+0000 osd.2 (osd.2) 106 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:43:07.332 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:07.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:07.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:07.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:07.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.458+0000 7f737cc20700 1 -- 172.21.15.162:0/3588174164 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f73781072c0 con 0x7f73780f9b70 2024-02-13T00:43:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:07.489 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:07.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:07.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:07.622 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:07.695 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:07.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:07.704 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:07.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.729+0000 7f2d54828700 1 -- 172.21.15.162:0/114573702 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d500fa440 msgr2=0x7f2d500f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:07.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.729+0000 7f2d54828700 1 --2- 172.21.15.162:0/114573702 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d500fa440 0x7f2d500f7e20 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 rx=0x7f2d44004870 tx=0x7f2d44009cb0).stop 2024-02-13T00:43:07.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.729+0000 7f2d54828700 1 -- 172.21.15.162:0/114573702 shutdown_connections 2024-02-13T00:43:07.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.729+0000 7f2d54828700 1 --2- 172.21.15.162:0/114573702 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d500f8500 0x7f2d50104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.729+0000 7f2d54828700 1 --2- 172.21.15.162:0/114573702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d500f9b70 0x7f2d500f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.729+0000 7f2d54828700 1 --2- 172.21.15.162:0/114573702 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d500fa440 0x7f2d500f7e20 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.729+0000 7f2d54828700 1 -- 172.21.15.162:0/114573702 >> 172.21.15.162:0/114573702 conn(0x7f2d500f38c0 msgr2=0x7f2d500f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:07.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.729+0000 7f2d54828700 1 -- 172.21.15.162:0/114573702 shutdown_connections 2024-02-13T00:43:07.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.730+0000 7f2d54828700 1 -- 172.21.15.162:0/114573702 wait complete. 2024-02-13T00:43:07.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.730+0000 7f2d54828700 1 librados: starting msgr at 2024-02-13T00:43:07.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.730+0000 7f2d54828700 1 librados: starting objecter 2024-02-13T00:43:07.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.730+0000 7f2d54828700 1 Processor -- start 2024-02-13T00:43:07.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d54828700 1 -- start start 2024-02-13T00:43:07.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d54828700 1 librados: setting wanted keys 2024-02-13T00:43:07.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d54828700 1 librados: calling monclient init 2024-02-13T00:43:07.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d54828700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d500f8500 0x7f2d5010c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:07.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d54828700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d500f9b70 0x7f2d5010c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:07.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d54828700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d500fa440 0x7f2d50106250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:07.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d54828700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d50106740 con 0x7f2d500fa440 2024-02-13T00:43:07.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d54828700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d501068b0 con 0x7f2d500f8500 2024-02-13T00:43:07.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d54828700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d50106a70 con 0x7f2d500f9b70 2024-02-13T00:43:07.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4dd9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d500f9b70 0x7f2d5010c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:07.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4ed9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d500fa440 0x7f2d50106250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:07.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4ed9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d500fa440 0x7f2d50106250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:58724/0 (socket says 172.21.15.162:58724) 2024-02-13T00:43:07.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4dd9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d500f9b70 0x7f2d5010c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48708/0 (socket says 172.21.15.162:48708) 2024-02-13T00:43:07.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4ed9d700 1 -- 172.21.15.162:0/508440629 learned_addr learned my addr 172.21.15.162:0/508440629 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:07.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4e59c700 1 --2- 172.21.15.162:0/508440629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d500f8500 0x7f2d5010c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:07.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4ed9d700 1 -- 172.21.15.162:0/508440629 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d500f9b70 msgr2=0x7f2d5010c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:07.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4ed9d700 1 --2- 172.21.15.162:0/508440629 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d500f9b70 0x7f2d5010c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4ed9d700 1 -- 172.21.15.162:0/508440629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d500f8500 msgr2=0x7f2d5010c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:07.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4ed9d700 1 --2- 172.21.15.162:0/508440629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d500f8500 0x7f2d5010c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4ed9d700 1 -- 172.21.15.162:0/508440629 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d44005040 con 0x7f2d500fa440 2024-02-13T00:43:07.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4dd9b700 1 --2- 172.21.15.162:0/508440629 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d500f9b70 0x7f2d5010c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:43:07.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d4ed9d700 1 --2- 172.21.15.162:0/508440629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d500fa440 0x7f2d50106250 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7f2d48009fc0 tx=0x7f2d48011040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:07.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d3f7fe700 10 client.?.objecter ms_handle_connect 0x7f2d500fa440 2024-02-13T00:43:07.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.731+0000 7f2d3f7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:07.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.732+0000 7f2d3f7fe700 1 -- 172.21.15.162:0/508440629 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48014040 con 0x7f2d500fa440 2024-02-13T00:43:07.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.732+0000 7f2d3f7fe700 1 -- 172.21.15.162:0/508440629 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d48009950 con 0x7f2d500fa440 2024-02-13T00:43:07.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.732+0000 7f2d3f7fe700 1 -- 172.21.15.162:0/508440629 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48012ee0 con 0x7f2d500fa440 2024-02-13T00:43:07.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.732+0000 7f2d54828700 1 -- 172.21.15.162:0/508440629 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d50106d50 con 0x7f2d500fa440 2024-02-13T00:43:07.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.732+0000 7f2d54828700 10 client.26787.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:07.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.732+0000 7f2d54828700 1 -- 172.21.15.162:0/508440629 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d50112e90 con 0x7f2d500fa440 2024-02-13T00:43:07.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.732+0000 7f2d54828700 1 librados: init done 2024-02-13T00:43:07.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.733+0000 7f2d54828700 1 -- 172.21.15.162:0/508440629 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d500475e0 con 0x7f2d500fa440 2024-02-13T00:43:07.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.734+0000 7f2d3f7fe700 1 -- 172.21.15.162:0/508440629 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f2d48007d00 con 0x7f2d500fa440 2024-02-13T00:43:07.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.735+0000 7f2d3f7fe700 1 --2- 172.21.15.162:0/508440629 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2d38062c90 0x7f2d38065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:07.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.735+0000 7f2d3f7fe700 1 -- 172.21.15.162:0/508440629 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(232..232 src has 52..232) v4 ==== 9385+0+0 (secure 0 0 0) 0x7f2d480824b0 con 0x7f2d500fa440 2024-02-13T00:43:07.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.735+0000 7f2d4e59c700 1 --2- 172.21.15.162:0/508440629 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2d38062c90 0x7f2d38065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:07.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.735+0000 7f2d3f7fe700 10 client.26787.objecter ms_dispatch 0x7f2d50001550 osd_map(232..232 src has 52..232) v4 2024-02-13T00:43:07.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.735+0000 7f2d3f7fe700 3 client.26787.objecter handle_osd_map got epochs [232,232] > 0 2024-02-13T00:43:07.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.735+0000 7f2d3f7fe700 3 client.26787.objecter handle_osd_map decoding full epoch 232 2024-02-13T00:43:07.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.735+0000 7f2d3f7fe700 20 client.26787.objecter dump_active .. 0 homeless 2024-02-13T00:43:07.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.735+0000 7f2d4e59c700 1 --2- 172.21.15.162:0/508440629 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2d38062c90 0x7f2d38065150 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 rx=0x7f2d400048b0 tx=0x7f2d4000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:07.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.735+0000 7f2d3f7fe700 10 client.26787.objecter ms_handle_connect 0x7f2d38062c90 2024-02-13T00:43:07.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.738+0000 7f2d3f7fe700 1 -- 172.21.15.162:0/508440629 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2d48055450 con 0x7f2d500fa440 2024-02-13T00:43:07.747 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:07.799 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:07.827 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:07.847 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:07.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:07.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.905+0000 7f2d54828700 1 -- 172.21.15.162:0/508440629 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2d501076e0 con 0x7f2d38062c90 2024-02-13T00:43:07.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.926+0000 7f2d3f7fe700 1 -- 172.21.15.162:0/508440629 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+788561 (secure 0 0 0) 0x7f2d501076e0 con 0x7f2d38062c90 2024-02-13T00:43:07.940 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:43:07.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.930+0000 7f2d54828700 10 librados: watch_flush enter 2024-02-13T00:43:07.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.930+0000 7f2d54828700 10 librados: watch_flush exit 2024-02-13T00:43:07.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.930+0000 7f2d54828700 20 client.26787.objecter shutdown clearing up homeless session... 2024-02-13T00:43:07.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.930+0000 7f2d54828700 10 client.26787.objecter successfully canceled tick 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 -- 172.21.15.162:0/508440629 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2d38062c90 msgr2=0x7f2d38065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 --2- 172.21.15.162:0/508440629 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2d38062c90 0x7f2d38065150 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 rx=0x7f2d400048b0 tx=0x7f2d4000b040).stop 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 -- 172.21.15.162:0/508440629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d500fa440 msgr2=0x7f2d50106250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 --2- 172.21.15.162:0/508440629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d500fa440 0x7f2d50106250 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7f2d48009fc0 tx=0x7f2d48011040).stop 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 -- 172.21.15.162:0/508440629 shutdown_connections 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 --2- 172.21.15.162:0/508440629 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2d38062c90 0x7f2d38065150 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 --2- 172.21.15.162:0/508440629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d500fa440 0x7f2d50106250 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 --2- 172.21.15.162:0/508440629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d500f8500 0x7f2d5010c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 --2- 172.21.15.162:0/508440629 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d500f9b70 0x7f2d5010c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 -- 172.21.15.162:0/508440629 >> 172.21.15.162:0/508440629 conn(0x7f2d500f38c0 msgr2=0x7f2d500f4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 -- 172.21.15.162:0/508440629 shutdown_connections 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 -- 172.21.15.162:0/508440629 wait complete. 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:07.931+0000 7f2d54828700 1 librados: shutdown 2024-02-13T00:43:07.950 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:43:07.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:07.977 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:08.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:08.042 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:08.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:08.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:08.172 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:08.225 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:08.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:06.356659+0000 mgr.y (mgr.14544) 244 : cluster [DBG] pgmap v317: 292 pgs: 4 active+clean+snaptrim, 64 unknown, 10 active+clean+snaptrim_wait, 214 active+clean; 9.6 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:08.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:06.800911+0000 osd.3 (osd.3) 139 : cluster [DBG] 18.1 scrub starts 2024-02-13T00:43:08.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:06.804417+0000 osd.3 (osd.3) 140 : cluster [DBG] 18.1 scrub ok 2024-02-13T00:43:08.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:07.169599+0000 mon.b (mon.1) 324 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:08.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:07.359259+0000 mon.a (mon.0) 1590 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-83977-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:08.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:07.359388+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e232: 8 total, 8 up, 7 in 2024-02-13T00:43:08.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:07.362159+0000 mon.c (mon.2) 216 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-83977-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:08.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:07.362583+0000 mon.a (mon.0) 1592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-83977-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:08.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:07.366440+0000 mon.a (mon.0) 1593 : audit [INF] from='client.? 172.21.15.162:0/3058490964' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:08.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:07.380801+0000 mon.b (mon.1) 325 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:08.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:07.381384+0000 mon.a (mon.0) 1594 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:08.228 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:07.389694+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:43:08.228 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:07.393189+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:43:08.228 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:07.459595+0000 mon.a (mon.0) 1595 : audit [INF] from='client.? 172.21.15.162:0/3588174164' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:43:08.228 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:07.602205+0000 osd.2 (osd.2) 107 : cluster [DBG] 28.6 scrub starts 2024-02-13T00:43:08.228 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:07.605107+0000 osd.2 (osd.2) 108 : cluster [DBG] 28.6 scrub ok 2024-02-13T00:43:08.228 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:06.356659+0000 mgr.y (mgr.14544) 244 : cluster [DBG] pgmap v317: 292 pgs: 4 active+clean+snaptrim, 64 unknown, 10 active+clean+snaptrim_wait, 214 active+clean; 9.6 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:06.800911+0000 osd.3 (osd.3) 139 : cluster [DBG] 18.1 scrub starts 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:06.804417+0000 osd.3 (osd.3) 140 : cluster [DBG] 18.1 scrub ok 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:07.169599+0000 mon.b (mon.1) 324 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:07.359259+0000 mon.a (mon.0) 1590 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-83977-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:07.359388+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e232: 8 total, 8 up, 7 in 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:07.362159+0000 mon.c (mon.2) 216 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-83977-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:07.362583+0000 mon.a (mon.0) 1592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-83977-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:07.366440+0000 mon.a (mon.0) 1593 : audit [INF] from='client.? 172.21.15.162:0/3058490964' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:07.380801+0000 mon.b (mon.1) 325 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:07.381384+0000 mon.a (mon.0) 1594 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:07.389694+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:07.393189+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:43:08.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:07.459595+0000 mon.a (mon.0) 1595 : audit [INF] from='client.? 172.21.15.162:0/3588174164' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:43:08.230 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:07.602205+0000 osd.2 (osd.2) 107 : cluster [DBG] 28.6 scrub starts 2024-02-13T00:43:08.230 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:07.605107+0000 osd.2 (osd.2) 108 : cluster [DBG] 28.6 scrub ok 2024-02-13T00:43:08.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:08.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:08.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:06.356659+0000 mgr.y (mgr.14544) 244 : cluster [DBG] pgmap v317: 292 pgs: 4 active+clean+snaptrim, 64 unknown, 10 active+clean+snaptrim_wait, 214 active+clean; 9.6 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:08.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:06.800911+0000 osd.3 (osd.3) 139 : cluster [DBG] 18.1 scrub starts 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:06.804417+0000 osd.3 (osd.3) 140 : cluster [DBG] 18.1 scrub ok 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:07.169599+0000 mon.b (mon.1) 324 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:07.359259+0000 mon.a (mon.0) 1590 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-83977-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:07.359388+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e232: 8 total, 8 up, 7 in 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:07.362159+0000 mon.c (mon.2) 216 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-83977-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:07.362583+0000 mon.a (mon.0) 1592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-83977-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:07.366440+0000 mon.a (mon.0) 1593 : audit [INF] from='client.? 172.21.15.162:0/3058490964' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:07.380801+0000 mon.b (mon.1) 325 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:07.381384+0000 mon.a (mon.0) 1594 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:07.389694+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:07.393189+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:07.459595+0000 mon.a (mon.0) 1595 : audit [INF] from='client.? 172.21.15.162:0/3588174164' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:07.602205+0000 osd.2 (osd.2) 107 : cluster [DBG] 28.6 scrub starts 2024-02-13T00:43:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:07.605107+0000 osd.2 (osd.2) 108 : cluster [DBG] 28.6 scrub ok 2024-02-13T00:43:08.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:08.360 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:08.379 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:08.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:08.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:08.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:08.511 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:08.530 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:08.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.599+0000 7f73677fe700 1 -- 172.21.15.162:0/3588174164 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v233) v1 ==== 87+0+0 (secure 0 0 0) 0x7f7368050370 con 0x7f73780f9b70 2024-02-13T00:43:08.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 10 librados: watch_flush enter 2024-02-13T00:43:08.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 10 librados: watch_flush exit 2024-02-13T00:43:08.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 20 client.26781.objecter shutdown clearing up homeless session... 2024-02-13T00:43:08.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 10 client.26781.objecter successfully canceled tick 2024-02-13T00:43:08.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 1 -- 172.21.15.162:0/3588174164 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7360062c90 msgr2=0x7f7360065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:08.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 1 --2- 172.21.15.162:0/3588174164 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7360062c90 0x7f7360065150 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 rx=0x7f736c005010 tx=0x7f736c007f90).stop 2024-02-13T00:43:08.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 1 -- 172.21.15.162:0/3588174164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73780f9b70 msgr2=0x7f737810e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:08.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 1 --2- 172.21.15.162:0/3588174164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73780f9b70 0x7f737810e6e0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 rx=0x7f7368002800 tx=0x7f7368009ec0).stop 2024-02-13T00:43:08.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 1 -- 172.21.15.162:0/3588174164 shutdown_connections 2024-02-13T00:43:08.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 1 --2- 172.21.15.162:0/3588174164 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7360062c90 0x7f7360065150 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:08.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 1 --2- 172.21.15.162:0/3588174164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780fa440 0x7f7378108390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:08.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 1 --2- 172.21.15.162:0/3588174164 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73780f8500 0x7f737810e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:08.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 1 --2- 172.21.15.162:0/3588174164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73780f9b70 0x7f737810e6e0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:08.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.602+0000 7f737cc20700 1 -- 172.21.15.162:0/3588174164 >> 172.21.15.162:0/3588174164 conn(0x7f73780f38c0 msgr2=0x7f73780fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:08.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.603+0000 7f737cc20700 1 -- 172.21.15.162:0/3588174164 shutdown_connections 2024-02-13T00:43:08.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.603+0000 7f737cc20700 1 -- 172.21.15.162:0/3588174164 wait complete. 2024-02-13T00:43:08.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:08.603+0000 7f737cc20700 1 librados: shutdown 2024-02-13T00:43:08.607 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:43:08.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:08.659 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:08.794 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:43:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:43:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:43:08.818 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-02-13T00:43:07.357089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9825,"num_objects":426,"num_object_clones":75,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":426,"num_whiteouts":0,"num_read":1483,"num_read_kb":1118,"num_write":1245,"num_write_kb":622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":3422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2451,"ondisk_log_size":2451,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":919,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2172256,"kb_used_data":26088,"kb_used_omap":0,"kb_used_meta":2146048,"kb_avail":654043808,"statfs":{"total":671965249536,"available":669740859392,"internally_reserved":0,"allocated":26714112,"data_stored":13826106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2197553152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-293,"num_objects":-15,"num_object_clones":-15,"num_object_copies":-45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017731"},"pg_stats":[{"pgid":"156.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:07.347323+0000","last_change":"2024-02-13T00:43:07.347323+0000","last_active":"2024-02-13T00:43:07.347323+0000","last_peered":"2024-02-13T00:43:07.347323+0000","last_clean":"2024-02-13T00:43:07.347323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:07.347323+0000","last_undegraded":"2024-02-13T00:43:07.347323+0000","last_fullsized":"2024-02-13T00:43:07.347323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:07.347323+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.347323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"154.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:43:05.339934+0000","last_change":"2024-02-13T00:43:05.339934+0000","last_active":"2024-02-13T00:43:05.339934+0000","last_peered":"2024-02-13T00:43:05.339934+0000","last_clean":"2024-02-13T00:43:05.339934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:43:05.339934+0000","last_undegraded":"2024-02-13T00:43:05.339934+0000","last_fullsized":"2024-02-13T00:43:05.339934+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:05.339934+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.339934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.2","version":"204'34","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.367258+0000","last_change":"2024-02-13T00:42:43.412343+0000","last_active":"2024-02-13T00:43:03.367258+0000","last_peered":"2024-02-13T00:43:03.367258+0000","last_clean":"2024-02-13T00:43:03.367258+0000","last_became_active":"2024-02-13T00:42:03.433088+0000","last_became_peered":"2024-02-13T00:42:03.433088+0000","last_unstale":"2024-02-13T00:43:03.367258+0000","last_undegraded":"2024-02-13T00:43:03.367258+0000","last_fullsized":"2024-02-13T00:43:03.367258+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.3","version":"228'103","reported_seq":218,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-02-13T00:43:03.441211+0000","last_change":"2024-02-13T00:42:59.429439+0000","last_active":"2024-02-13T00:43:03.441211+0000","last_peered":"2024-02-13T00:43:03.441211+0000","last_clean":"2024-02-13T00:43:03.441211+0000","last_became_active":"2024-02-13T00:41:16.803266+0000","last_became_peered":"2024-02-13T00:41:16.803266+0000","last_unstale":"2024-02-13T00:43:03.441211+0000","last_undegraded":"2024-02-13T00:43:03.441211+0000","last_fullsized":"2024-02-13T00:43:03.441211+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":103,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":70,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"2"}]},{"pgid":"61.0","version":"202'34","reported_seq":172,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.349257+0000","last_change":"2024-02-13T00:42:39.408531+0000","last_active":"2024-02-13T00:43:03.349257+0000","last_peered":"2024-02-13T00:43:03.349257+0000","last_clean":"2024-02-13T00:43:03.349257+0000","last_became_active":"2024-02-13T00:41:16.768067+0000","last_became_peered":"2024-02-13T00:41:16.768067+0000","last_unstale":"2024-02-13T00:43:03.349257+0000","last_undegraded":"2024-02-13T00:43:03.349257+0000","last_fullsized":"2024-02-13T00:43:03.349257+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.1","version":"214'34","reported_seq":169,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347549+0000","last_change":"2024-02-13T00:42:57.436704+0000","last_active":"2024-02-13T00:43:03.347549+0000","last_peered":"2024-02-13T00:43:03.347549+0000","last_clean":"2024-02-13T00:43:03.347549+0000","last_became_active":"2024-02-13T00:41:16.734552+0000","last_became_peered":"2024-02-13T00:41:16.734552+0000","last_unstale":"2024-02-13T00:43:03.347549+0000","last_undegraded":"2024-02-13T00:43:03.347549+0000","last_fullsized":"2024-02-13T00:43:03.347549+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.6","version":"210'17","reported_seq":159,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.355010+0000","last_change":"2024-02-13T00:42:49.425577+0000","last_active":"2024-02-13T00:43:03.355010+0000","last_peered":"2024-02-13T00:43:03.355010+0000","last_clean":"2024-02-13T00:43:03.355010+0000","last_became_active":"2024-02-13T00:41:16.791271+0000","last_became_peered":"2024-02-13T00:41:16.791271+0000","last_unstale":"2024-02-13T00:43:03.355010+0000","last_undegraded":"2024-02-13T00:43:03.355010+0000","last_fullsized":"2024-02-13T00:43:03.355010+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.7","version":"228'63","reported_seq":187,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-02-13T00:43:03.431222+0000","last_change":"2024-02-13T00:43:01.427840+0000","last_active":"2024-02-13T00:43:03.431222+0000","last_peered":"2024-02-13T00:43:03.431222+0000","last_clean":"2024-02-13T00:43:03.431222+0000","last_became_active":"2024-02-13T00:41:16.776064+0000","last_became_peered":"2024-02-13T00:41:16.776064+0000","last_unstale":"2024-02-13T00:43:03.431222+0000","last_undegraded":"2024-02-13T00:43:03.431222+0000","last_fullsized":"2024-02-13T00:43:03.431222+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":63,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":176,"num_objects":16,"num_object_clones":11,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.4","version":"214'56","reported_seq":192,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350007+0000","last_change":"2024-02-13T00:42:51.428675+0000","last_active":"2024-02-13T00:43:03.350007+0000","last_peered":"2024-02-13T00:43:03.350007+0000","last_clean":"2024-02-13T00:43:03.350007+0000","last_became_active":"2024-02-13T00:41:16.790979+0000","last_became_peered":"2024-02-13T00:41:16.790979+0000","last_unstale":"2024-02-13T00:43:03.350007+0000","last_undegraded":"2024-02-13T00:43:03.350007+0000","last_fullsized":"2024-02-13T00:43:03.350007+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.5","version":"224'56","reported_seq":234,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-13T00:43:03.349457+0000","last_change":"2024-02-13T00:43:01.439926+0000","last_active":"2024-02-13T00:43:03.349457+0000","last_peered":"2024-02-13T00:43:03.349457+0000","last_clean":"2024-02-13T00:43:03.349457+0000","last_became_active":"2024-02-13T00:42:03.412443+0000","last_became_peered":"2024-02-13T00:42:03.412443+0000","last_unstale":"2024-02-13T00:43:03.349457+0000","last_undegraded":"2024-02-13T00:43:03.349457+0000","last_fullsized":"2024-02-13T00:43:03.349457+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"3"}]},{"pgid":"61.a","version":"216'85","reported_seq":217,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.367482+0000","last_change":"2024-02-13T00:42:53.422636+0000","last_active":"2024-02-13T00:43:03.367482+0000","last_peered":"2024-02-13T00:43:03.367482+0000","last_clean":"2024-02-13T00:43:03.367482+0000","last_became_active":"2024-02-13T00:41:16.735087+0000","last_became_peered":"2024-02-13T00:41:16.735087+0000","last_unstale":"2024-02-13T00:43:03.367482+0000","last_undegraded":"2024-02-13T00:43:03.367482+0000","last_fullsized":"2024-02-13T00:43:03.367482+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.b","version":"222'17","reported_seq":173,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-13T00:43:03.343330+0000","last_change":"2024-02-13T00:42:59.429198+0000","last_active":"2024-02-13T00:43:03.343330+0000","last_peered":"2024-02-13T00:43:03.343330+0000","last_clean":"2024-02-13T00:43:03.343330+0000","last_became_active":"2024-02-13T00:42:03.404523+0000","last_became_peered":"2024-02-13T00:42:03.404523+0000","last_unstale":"2024-02-13T00:43:03.343330+0000","last_undegraded":"2024-02-13T00:43:03.343330+0000","last_fullsized":"2024-02-13T00:43:03.343330+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"3"}]},{"pgid":"61.8","version":"228'47","reported_seq":178,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-02-13T00:43:03.443651+0000","last_change":"2024-02-13T00:43:01.433853+0000","last_active":"2024-02-13T00:43:03.443651+0000","last_peered":"2024-02-13T00:43:03.443651+0000","last_clean":"2024-02-13T00:43:03.443651+0000","last_became_active":"2024-02-13T00:41:16.749465+0000","last_became_peered":"2024-02-13T00:41:16.749465+0000","last_unstale":"2024-02-13T00:43:03.443651+0000","last_undegraded":"2024-02-13T00:43:03.443651+0000","last_fullsized":"2024-02-13T00:43:03.443651+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"61.9","version":"216'34","reported_seq":167,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346719+0000","last_change":"2024-02-13T00:42:59.429378+0000","last_active":"2024-02-13T00:43:03.346719+0000","last_peered":"2024-02-13T00:43:03.346719+0000","last_clean":"2024-02-13T00:43:03.346719+0000","last_became_active":"2024-02-13T00:41:16.768452+0000","last_became_peered":"2024-02-13T00:41:16.768452+0000","last_unstale":"2024-02-13T00:43:03.346719+0000","last_undegraded":"2024-02-13T00:43:03.346719+0000","last_fullsized":"2024-02-13T00:43:03.346719+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.e","version":"222'78","reported_seq":236,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-13T00:43:03.346291+0000","last_change":"2024-02-13T00:42:59.425085+0000","last_active":"2024-02-13T00:43:03.346291+0000","last_peered":"2024-02-13T00:43:03.346291+0000","last_clean":"2024-02-13T00:43:03.346291+0000","last_became_active":"2024-02-13T00:42:03.399160+0000","last_became_peered":"2024-02-13T00:42:03.399160+0000","last_unstale":"2024-02-13T00:43:03.346291+0000","last_undegraded":"2024-02-13T00:43:03.346291+0000","last_fullsized":"2024-02-13T00:43:03.346291+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":198,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"61.f","version":"216'43","reported_seq":175,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-13T00:43:03.345094+0000","last_change":"2024-02-13T00:42:53.346727+0000","last_active":"2024-02-13T00:43:03.345094+0000","last_peered":"2024-02-13T00:43:03.345094+0000","last_clean":"2024-02-13T00:43:03.345094+0000","last_became_active":"2024-02-13T00:42:53.346223+0000","last_became_peered":"2024-02-13T00:42:53.346223+0000","last_unstale":"2024-02-13T00:43:03.345094+0000","last_undegraded":"2024-02-13T00:43:03.345094+0000","last_fullsized":"2024-02-13T00:43:03.345094+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"61.c","version":"220'80","reported_seq":202,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-13T00:43:03.340756+0000","last_change":"2024-02-13T00:42:57.426665+0000","last_active":"2024-02-13T00:43:03.340756+0000","last_peered":"2024-02-13T00:43:03.340756+0000","last_clean":"2024-02-13T00:43:03.340756+0000","last_became_active":"2024-02-13T00:41:16.702990+0000","last_became_peered":"2024-02-13T00:41:16.702990+0000","last_unstale":"2024-02-13T00:43:03.340756+0000","last_undegraded":"2024-02-13T00:43:03.340756+0000","last_fullsized":"2024-02-13T00:43:03.340756+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":80,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":206,"num_objects":19,"num_object_clones":12,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"61.d","version":"214'65","reported_seq":187,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-13T00:43:03.342580+0000","last_change":"2024-02-13T00:42:51.421016+0000","last_active":"2024-02-13T00:43:03.342580+0000","last_peered":"2024-02-13T00:43:03.342580+0000","last_clean":"2024-02-13T00:43:03.342580+0000","last_became_active":"2024-02-13T00:41:16.780278+0000","last_became_peered":"2024-02-13T00:41:16.780278+0000","last_unstale":"2024-02-13T00:43:03.342580+0000","last_undegraded":"2024-02-13T00:43:03.342580+0000","last_fullsized":"2024-02-13T00:43:03.342580+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":165,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"61.12","version":"208'17","reported_seq":160,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346016+0000","last_change":"2024-02-13T00:42:49.425322+0000","last_active":"2024-02-13T00:43:03.346016+0000","last_peered":"2024-02-13T00:43:03.346016+0000","last_clean":"2024-02-13T00:43:03.346016+0000","last_became_active":"2024-02-13T00:41:16.779347+0000","last_became_peered":"2024-02-13T00:41:16.779347+0000","last_unstale":"2024-02-13T00:43:03.346016+0000","last_undegraded":"2024-02-13T00:43:03.346016+0000","last_fullsized":"2024-02-13T00:43:03.346016+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.13","version":"220'68","reported_seq":251,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.360027+0000","last_change":"2024-02-13T00:42:57.363317+0000","last_active":"2024-02-13T00:43:03.360027+0000","last_peered":"2024-02-13T00:43:03.360027+0000","last_clean":"2024-02-13T00:43:03.360027+0000","last_became_active":"2024-02-13T00:42:53.351441+0000","last_became_peered":"2024-02-13T00:42:53.351441+0000","last_unstale":"2024-02-13T00:43:03.360027+0000","last_undegraded":"2024-02-13T00:43:03.360027+0000","last_fullsized":"2024-02-13T00:43:03.360027+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.10","version":"207'52","reported_seq":175,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-13T00:43:03.340030+0000","last_change":"2024-02-13T00:42:53.368882+0000","last_active":"2024-02-13T00:43:03.340030+0000","last_peered":"2024-02-13T00:43:03.340030+0000","last_clean":"2024-02-13T00:43:03.340030+0000","last_became_active":"2024-02-13T00:42:53.368394+0000","last_became_peered":"2024-02-13T00:42:53.368394+0000","last_unstale":"2024-02-13T00:43:03.340030+0000","last_undegraded":"2024-02-13T00:43:03.340030+0000","last_fullsized":"2024-02-13T00:43:03.340030+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":132,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"61.11","version":"208'34","reported_seq":203,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.349835+0000","last_change":"2024-02-13T00:42:49.425017+0000","last_active":"2024-02-13T00:43:03.349835+0000","last_peered":"2024-02-13T00:43:03.349835+0000","last_clean":"2024-02-13T00:43:03.349835+0000","last_became_active":"2024-02-13T00:42:03.391427+0000","last_became_peered":"2024-02-13T00:42:03.391427+0000","last_unstale":"2024-02-13T00:43:03.349835+0000","last_undegraded":"2024-02-13T00:43:03.349835+0000","last_fullsized":"2024-02-13T00:43:03.349835+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.16","version":"224'52","reported_seq":171,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-13T00:43:03.346409+0000","last_change":"2024-02-13T00:43:01.432310+0000","last_active":"2024-02-13T00:43:03.346409+0000","last_peered":"2024-02-13T00:43:03.346409+0000","last_clean":"2024-02-13T00:43:03.346409+0000","last_became_active":"2024-02-13T00:41:16.694271+0000","last_became_peered":"2024-02-13T00:41:16.694271+0000","last_unstale":"2024-02-13T00:43:03.346409+0000","last_undegraded":"2024-02-13T00:43:03.346409+0000","last_fullsized":"2024-02-13T00:43:03.346409+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":132,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"61.17","version":"206'68","reported_seq":200,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.360398+0000","last_change":"2024-02-13T00:42:43.424639+0000","last_active":"2024-02-13T00:43:03.360398+0000","last_peered":"2024-02-13T00:43:03.360398+0000","last_clean":"2024-02-13T00:43:03.360398+0000","last_became_active":"2024-02-13T00:41:16.793277+0000","last_became_peered":"2024-02-13T00:41:16.793277+0000","last_unstale":"2024-02-13T00:43:03.360398+0000","last_undegraded":"2024-02-13T00:43:03.360398+0000","last_fullsized":"2024-02-13T00:43:03.360398+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.14","version":"200'17","reported_seq":158,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.357811+0000","last_change":"2024-02-13T00:42:39.408199+0000","last_active":"2024-02-13T00:43:03.357811+0000","last_peered":"2024-02-13T00:43:03.357811+0000","last_clean":"2024-02-13T00:43:03.357811+0000","last_became_active":"2024-02-13T00:41:16.784290+0000","last_became_peered":"2024-02-13T00:41:16.784290+0000","last_unstale":"2024-02-13T00:43:03.357811+0000","last_undegraded":"2024-02-13T00:43:03.357811+0000","last_fullsized":"2024-02-13T00:43:03.357811+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.15","version":"224'78","reported_seq":211,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.359983+0000","last_change":"2024-02-13T00:43:01.382017+0000","last_active":"2024-02-13T00:43:03.359983+0000","last_peered":"2024-02-13T00:43:03.359983+0000","last_clean":"2024-02-13T00:43:03.359983+0000","last_became_active":"2024-02-13T00:42:53.357753+0000","last_became_peered":"2024-02-13T00:42:53.357753+0000","last_unstale":"2024-02-13T00:43:03.359983+0000","last_undegraded":"2024-02-13T00:43:03.359983+0000","last_fullsized":"2024-02-13T00:43:03.359983+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.1a","version":"220'51","reported_seq":235,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.367371+0000","last_change":"2024-02-13T00:42:57.427984+0000","last_active":"2024-02-13T00:43:03.367371+0000","last_peered":"2024-02-13T00:43:03.367371+0000","last_clean":"2024-02-13T00:43:03.367371+0000","last_became_active":"2024-02-13T00:42:53.372644+0000","last_became_peered":"2024-02-13T00:42:53.372644+0000","last_unstale":"2024-02-13T00:43:03.367371+0000","last_undegraded":"2024-02-13T00:43:03.367371+0000","last_fullsized":"2024-02-13T00:43:03.367371+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.1b","version":"226'35","reported_seq":199,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.358302+0000","last_change":"2024-02-13T00:42:43.412868+0000","last_active":"2024-02-13T00:43:03.358302+0000","last_peered":"2024-02-13T00:43:03.358302+0000","last_clean":"2024-02-13T00:43:03.358302+0000","last_became_active":"2024-02-13T00:42:03.418968+0000","last_became_peered":"2024-02-13T00:42:03.418968+0000","last_unstale":"2024-02-13T00:43:03.358302+0000","last_undegraded":"2024-02-13T00:43:03.358302+0000","last_fullsized":"2024-02-13T00:43:03.358302+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.18","version":"204'34","reported_seq":173,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.368689+0000","last_change":"2024-02-13T00:42:43.412643+0000","last_active":"2024-02-13T00:43:03.368689+0000","last_peered":"2024-02-13T00:43:03.368689+0000","last_clean":"2024-02-13T00:43:03.368689+0000","last_became_active":"2024-02-13T00:41:16.747555+0000","last_became_peered":"2024-02-13T00:41:16.747555+0000","last_unstale":"2024-02-13T00:43:03.368689+0000","last_undegraded":"2024-02-13T00:43:03.368689+0000","last_fullsized":"2024-02-13T00:43:03.368689+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.19","version":"224'15","reported_seq":150,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-13T00:43:03.341007+0000","last_change":"2024-02-13T00:43:01.427818+0000","last_active":"2024-02-13T00:43:03.341007+0000","last_peered":"2024-02-13T00:43:03.341007+0000","last_clean":"2024-02-13T00:43:03.341007+0000","last_became_active":"2024-02-13T00:41:16.749013+0000","last_became_peered":"2024-02-13T00:41:16.749013+0000","last_unstale":"2024-02-13T00:43:03.341007+0000","last_undegraded":"2024-02-13T00:43:03.341007+0000","last_fullsized":"2024-02-13T00:43:03.341007+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"61.1e","version":"224'52","reported_seq":230,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.354958+0000","last_change":"2024-02-13T00:43:01.395882+0000","last_active":"2024-02-13T00:43:03.354958+0000","last_peered":"2024-02-13T00:43:03.354958+0000","last_clean":"2024-02-13T00:43:03.354958+0000","last_became_active":"2024-02-13T00:42:53.378417+0000","last_became_peered":"2024-02-13T00:42:53.378417+0000","last_unstale":"2024-02-13T00:43:03.354958+0000","last_undegraded":"2024-02-13T00:43:03.354958+0000","last_fullsized":"2024-02-13T00:43:03.354958+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.1f","version":"218'57","reported_seq":196,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.352717+0000","last_change":"2024-02-13T00:42:55.429962+0000","last_active":"2024-02-13T00:43:03.352717+0000","last_peered":"2024-02-13T00:43:03.352717+0000","last_clean":"2024-02-13T00:43:03.352717+0000","last_became_active":"2024-02-13T00:41:16.807294+0000","last_became_peered":"2024-02-13T00:41:16.807294+0000","last_unstale":"2024-02-13T00:43:03.352717+0000","last_undegraded":"2024-02-13T00:43:03.352717+0000","last_fullsized":"2024-02-13T00:43:03.352717+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"61.1c","version":"216'39","reported_seq":205,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-13T00:43:03.340362+0000","last_change":"2024-02-13T00:42:53.420356+0000","last_active":"2024-02-13T00:43:03.340362+0000","last_peered":"2024-02-13T00:43:03.340362+0000","last_clean":"2024-02-13T00:43:03.340362+0000","last_became_active":"2024-02-13T00:42:03.399127+0000","last_became_peered":"2024-02-13T00:42:03.399127+0000","last_unstale":"2024-02-13T00:43:03.340362+0000","last_undegraded":"2024-02-13T00:43:03.340362+0000","last_fullsized":"2024-02-13T00:43:03.340362+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":99,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"61.1d","version":"228'68","reported_seq":192,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-02-13T00:43:03.443937+0000","last_change":"2024-02-13T00:43:01.439959+0000","last_active":"2024-02-13T00:43:03.443937+0000","last_peered":"2024-02-13T00:43:03.443937+0000","last_clean":"2024-02-13T00:43:03.443937+0000","last_became_active":"2024-02-13T00:41:16.802530+0000","last_became_peered":"2024-02-13T00:41:16.802530+0000","last_unstale":"2024-02-13T00:43:03.443937+0000","last_undegraded":"2024-02-13T00:43:03.443937+0000","last_fullsized":"2024-02-13T00:43:03.443937+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"2"}]},{"pgid":"28.10","version":"0'0","reported_seq":140,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350892+0000","last_change":"2024-02-13T00:42:03.378511+0000","last_active":"2024-02-13T00:43:03.350892+0000","last_peered":"2024-02-13T00:43:03.350892+0000","last_clean":"2024-02-13T00:43:03.350892+0000","last_became_active":"2024-02-13T00:42:03.378168+0000","last_became_peered":"2024-02-13T00:42:03.378168+0000","last_unstale":"2024-02-13T00:43:03.350892+0000","last_undegraded":"2024-02-13T00:43:03.350892+0000","last_fullsized":"2024-02-13T00:43:03.350892+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"218'10","reported_seq":211,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.279604+0000","last_change":"2024-02-13T00:42:53.371697+0000","last_active":"2024-02-13T00:43:04.279604+0000","last_peered":"2024-02-13T00:43:04.279604+0000","last_clean":"2024-02-13T00:43:04.279604+0000","last_became_active":"2024-02-13T00:42:53.371376+0000","last_became_peered":"2024-02-13T00:42:53.371376+0000","last_unstale":"2024-02-13T00:43:04.279604+0000","last_undegraded":"2024-02-13T00:43:04.279604+0000","last_fullsized":"2024-02-13T00:43:04.279604+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:40:58.179446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.179446+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":281,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.340225+0000","last_change":"2024-02-13T00:42:06.700478+0000","last_active":"2024-02-13T00:43:03.340225+0000","last_peered":"2024-02-13T00:43:03.340225+0000","last_clean":"2024-02-13T00:43:03.340225+0000","last_became_active":"2024-02-13T00:42:03.403540+0000","last_became_peered":"2024-02-13T00:42:03.403540+0000","last_unstale":"2024-02-13T00:43:03.340225+0000","last_undegraded":"2024-02-13T00:43:03.340225+0000","last_fullsized":"2024-02-13T00:43:03.340225+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:41:31.413581+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_clean_scrub_stamp":"2024-02-13T00:41:31.413581+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":196,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.355045+0000","last_change":"2024-02-13T00:42:27.340169+0000","last_active":"2024-02-13T00:43:03.355045+0000","last_peered":"2024-02-13T00:43:03.355045+0000","last_clean":"2024-02-13T00:43:03.355045+0000","last_became_active":"2024-02-13T00:37:41.219093+0000","last_became_peered":"2024-02-13T00:37:41.219093+0000","last_unstale":"2024-02-13T00:43:03.355045+0000","last_undegraded":"2024-02-13T00:43:03.355045+0000","last_fullsized":"2024-02-13T00:43:03.355045+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.340080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.340080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":195,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.339823+0000","last_change":"2024-02-13T00:42:03.396737+0000","last_active":"2024-02-13T00:43:03.339823+0000","last_peered":"2024-02-13T00:43:03.339823+0000","last_clean":"2024-02-13T00:43:03.339823+0000","last_became_active":"2024-02-13T00:42:03.395729+0000","last_became_peered":"2024-02-13T00:42:03.395729+0000","last_unstale":"2024-02-13T00:43:03.339823+0000","last_undegraded":"2024-02-13T00:43:03.339823+0000","last_fullsized":"2024-02-13T00:43:03.339823+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:45.416238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:45.416238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":132,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346491+0000","last_change":"2024-02-13T00:41:01.627210+0000","last_active":"2024-02-13T00:43:03.346491+0000","last_peered":"2024-02-13T00:43:03.346491+0000","last_clean":"2024-02-13T00:43:03.346491+0000","last_became_active":"2024-02-13T00:41:01.624745+0000","last_became_peered":"2024-02-13T00:41:01.624745+0000","last_unstale":"2024-02-13T00:43:03.346491+0000","last_undegraded":"2024-02-13T00:43:03.346491+0000","last_fullsized":"2024-02-13T00:43:03.346491+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":140,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347216+0000","last_change":"2024-02-13T00:42:03.375612+0000","last_active":"2024-02-13T00:43:03.347216+0000","last_peered":"2024-02-13T00:43:03.347216+0000","last_clean":"2024-02-13T00:43:03.347216+0000","last_became_active":"2024-02-13T00:42:03.375294+0000","last_became_peered":"2024-02-13T00:42:03.375294+0000","last_unstale":"2024-02-13T00:43:03.347216+0000","last_undegraded":"2024-02-13T00:43:03.347216+0000","last_fullsized":"2024-02-13T00:43:03.347216+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":219,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.360450+0000","last_change":"2024-02-13T00:42:19.652772+0000","last_active":"2024-02-13T00:43:03.360450+0000","last_peered":"2024-02-13T00:43:03.360450+0000","last_clean":"2024-02-13T00:43:03.360450+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:43:03.360450+0000","last_undegraded":"2024-02-13T00:43:03.360450+0000","last_fullsized":"2024-02-13T00:43:03.360450+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:42:19.652689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:19.652689+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":191,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350185+0000","last_change":"2024-02-13T00:42:20.102542+0000","last_active":"2024-02-13T00:43:03.350185+0000","last_peered":"2024-02-13T00:43:03.350185+0000","last_clean":"2024-02-13T00:43:03.350185+0000","last_became_active":"2024-02-13T00:37:41.212429+0000","last_became_peered":"2024-02-13T00:37:41.212429+0000","last_unstale":"2024-02-13T00:43:03.350185+0000","last_undegraded":"2024-02-13T00:43:03.350185+0000","last_fullsized":"2024-02-13T00:43:03.350185+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:20.102471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:20.102471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":187,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346094+0000","last_change":"2024-02-13T00:40:53.362331+0000","last_active":"2024-02-13T00:43:03.346094+0000","last_peered":"2024-02-13T00:43:03.346094+0000","last_clean":"2024-02-13T00:43:03.346094+0000","last_became_active":"2024-02-13T00:37:45.240923+0000","last_became_peered":"2024-02-13T00:37:45.240923+0000","last_unstale":"2024-02-13T00:43:03.346094+0000","last_undegraded":"2024-02-13T00:43:03.346094+0000","last_fullsized":"2024-02-13T00:43:03.346094+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:53.362232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:53.362232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":184,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.351562+0000","last_change":"2024-02-13T00:41:41.476236+0000","last_active":"2024-02-13T00:43:03.351562+0000","last_peered":"2024-02-13T00:43:03.351562+0000","last_clean":"2024-02-13T00:43:03.351562+0000","last_became_active":"2024-02-13T00:37:47.249556+0000","last_became_peered":"2024-02-13T00:37:47.249556+0000","last_unstale":"2024-02-13T00:43:03.351562+0000","last_undegraded":"2024-02-13T00:43:03.351562+0000","last_fullsized":"2024-02-13T00:43:03.351562+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1d","version":"108'7","reported_seq":139,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.352982+0000","last_change":"2024-02-13T00:41:01.697227+0000","last_active":"2024-02-13T00:43:03.352982+0000","last_peered":"2024-02-13T00:43:03.352982+0000","last_clean":"2024-02-13T00:43:03.352982+0000","last_became_active":"2024-02-13T00:41:01.696134+0000","last_became_peered":"2024-02-13T00:41:01.696134+0000","last_unstale":"2024-02-13T00:43:03.352982+0000","last_undegraded":"2024-02-13T00:43:03.352982+0000","last_fullsized":"2024-02-13T00:43:03.352982+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":131,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.353710+0000","last_change":"2024-02-13T00:41:01.656850+0000","last_active":"2024-02-13T00:43:03.353710+0000","last_peered":"2024-02-13T00:43:03.353710+0000","last_clean":"2024-02-13T00:43:03.353710+0000","last_became_active":"2024-02-13T00:41:01.654410+0000","last_became_peered":"2024-02-13T00:41:01.654410+0000","last_unstale":"2024-02-13T00:43:03.353710+0000","last_undegraded":"2024-02-13T00:43:03.353710+0000","last_fullsized":"2024-02-13T00:43:03.353710+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"91'35","reported_seq":188,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.348677+0000","last_change":"2024-02-13T00:42:06.389716+0000","last_active":"2024-02-13T00:43:03.348677+0000","last_peered":"2024-02-13T00:43:03.348677+0000","last_clean":"2024-02-13T00:43:03.348677+0000","last_became_active":"2024-02-13T00:42:03.423529+0000","last_became_peered":"2024-02-13T00:42:03.423529+0000","last_unstale":"2024-02-13T00:43:03.348677+0000","last_undegraded":"2024-02-13T00:43:03.348677+0000","last_fullsized":"2024-02-13T00:43:03.348677+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:40:15.074034+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":191,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350286+0000","last_change":"2024-02-13T00:41:47.446081+0000","last_active":"2024-02-13T00:43:03.350286+0000","last_peered":"2024-02-13T00:43:03.350286+0000","last_clean":"2024-02-13T00:43:03.350286+0000","last_became_active":"2024-02-13T00:37:41.219457+0000","last_became_peered":"2024-02-13T00:37:41.219457+0000","last_unstale":"2024-02-13T00:43:03.350286+0000","last_undegraded":"2024-02-13T00:43:03.350286+0000","last_fullsized":"2024-02-13T00:43:03.350286+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:47.446012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:47.446012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":187,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347026+0000","last_change":"2024-02-13T00:41:03.411330+0000","last_active":"2024-02-13T00:43:03.347026+0000","last_peered":"2024-02-13T00:43:03.347026+0000","last_clean":"2024-02-13T00:43:03.347026+0000","last_became_active":"2024-02-13T00:37:45.239867+0000","last_became_peered":"2024-02-13T00:37:45.239867+0000","last_unstale":"2024-02-13T00:43:03.347026+0000","last_undegraded":"2024-02-13T00:43:03.347026+0000","last_fullsized":"2024-02-13T00:43:03.347026+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:03.411257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_clean_scrub_stamp":"2024-02-13T00:41:03.411257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":23,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.348488+0000","last_change":"2024-02-13T00:42:53.374822+0000","last_active":"2024-02-13T00:43:03.348488+0000","last_peered":"2024-02-13T00:43:03.348488+0000","last_clean":"2024-02-13T00:43:03.348488+0000","last_became_active":"2024-02-13T00:42:53.374520+0000","last_became_peered":"2024-02-13T00:42:53.374520+0000","last_unstale":"2024-02-13T00:43:03.348488+0000","last_undegraded":"2024-02-13T00:43:03.348488+0000","last_fullsized":"2024-02-13T00:43:03.348488+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":132,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346038+0000","last_change":"2024-02-13T00:41:01.733877+0000","last_active":"2024-02-13T00:43:03.346038+0000","last_peered":"2024-02-13T00:43:03.346038+0000","last_clean":"2024-02-13T00:43:03.346038+0000","last_became_active":"2024-02-13T00:41:01.733241+0000","last_became_peered":"2024-02-13T00:41:01.733241+0000","last_unstale":"2024-02-13T00:43:03.346038+0000","last_undegraded":"2024-02-13T00:43:03.346038+0000","last_fullsized":"2024-02-13T00:43:03.346038+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":140,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.349993+0000","last_change":"2024-02-13T00:42:03.414876+0000","last_active":"2024-02-13T00:43:03.349993+0000","last_peered":"2024-02-13T00:43:03.349993+0000","last_clean":"2024-02-13T00:43:03.349993+0000","last_became_active":"2024-02-13T00:42:03.414095+0000","last_became_peered":"2024-02-13T00:42:03.414095+0000","last_unstale":"2024-02-13T00:43:03.349993+0000","last_undegraded":"2024-02-13T00:43:03.349993+0000","last_fullsized":"2024-02-13T00:43:03.349993+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":195,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.352337+0000","last_change":"2024-02-13T00:42:17.413898+0000","last_active":"2024-02-13T00:43:03.352337+0000","last_peered":"2024-02-13T00:43:03.352337+0000","last_clean":"2024-02-13T00:43:03.352337+0000","last_became_active":"2024-02-13T00:40:35.850675+0000","last_became_peered":"2024-02-13T00:40:35.850675+0000","last_unstale":"2024-02-13T00:43:03.352337+0000","last_undegraded":"2024-02-13T00:43:03.352337+0000","last_fullsized":"2024-02-13T00:43:03.352337+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:42:17.413826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.413826+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":31,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.340247+0000","last_change":"2024-02-13T00:42:53.345715+0000","last_active":"2024-02-13T00:43:03.340247+0000","last_peered":"2024-02-13T00:43:03.340247+0000","last_clean":"2024-02-13T00:43:03.340247+0000","last_became_active":"2024-02-13T00:42:53.345415+0000","last_became_peered":"2024-02-13T00:42:53.345415+0000","last_unstale":"2024-02-13T00:43:03.340247+0000","last_undegraded":"2024-02-13T00:43:03.340247+0000","last_fullsized":"2024-02-13T00:43:03.340247+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:41:00.128595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.128595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":189,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.345838+0000","last_change":"2024-02-13T00:42:28.037452+0000","last_active":"2024-02-13T00:43:03.345838+0000","last_peered":"2024-02-13T00:43:03.345838+0000","last_clean":"2024-02-13T00:43:03.345838+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:43:03.345838+0000","last_undegraded":"2024-02-13T00:43:03.345838+0000","last_fullsized":"2024-02-13T00:43:03.345838+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:28.037365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:28.037365+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":185,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.352950+0000","last_change":"2024-02-13T00:41:02.402785+0000","last_active":"2024-02-13T00:43:03.352950+0000","last_peered":"2024-02-13T00:43:03.352950+0000","last_clean":"2024-02-13T00:43:03.352950+0000","last_became_active":"2024-02-13T00:37:47.256807+0000","last_became_peered":"2024-02-13T00:37:47.256807+0000","last_unstale":"2024-02-13T00:43:03.352950+0000","last_undegraded":"2024-02-13T00:43:03.352950+0000","last_fullsized":"2024-02-13T00:43:03.352950+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.402668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.402668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.1b","version":"108'10","reported_seq":142,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350436+0000","last_change":"2024-02-13T00:41:01.676174+0000","last_active":"2024-02-13T00:43:03.350436+0000","last_peered":"2024-02-13T00:43:03.350436+0000","last_clean":"2024-02-13T00:43:03.350436+0000","last_became_active":"2024-02-13T00:41:01.675424+0000","last_became_peered":"2024-02-13T00:41:01.675424+0000","last_unstale":"2024-02-13T00:43:03.350436+0000","last_undegraded":"2024-02-13T00:43:03.350436+0000","last_fullsized":"2024-02-13T00:43:03.350436+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":131,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.369154+0000","last_change":"2024-02-13T00:41:01.536591+0000","last_active":"2024-02-13T00:43:03.369154+0000","last_peered":"2024-02-13T00:43:03.369154+0000","last_clean":"2024-02-13T00:43:03.369154+0000","last_became_active":"2024-02-13T00:41:01.536188+0000","last_became_peered":"2024-02-13T00:41:01.536188+0000","last_unstale":"2024-02-13T00:43:03.369154+0000","last_undegraded":"2024-02-13T00:43:03.369154+0000","last_fullsized":"2024-02-13T00:43:03.369154+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":226,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.369083+0000","last_change":"2024-02-13T00:43:00.776626+0000","last_active":"2024-02-13T00:43:03.369083+0000","last_peered":"2024-02-13T00:43:03.369083+0000","last_clean":"2024-02-13T00:43:03.369083+0000","last_became_active":"2024-02-13T00:42:03.421112+0000","last_became_peered":"2024-02-13T00:42:03.421112+0000","last_unstale":"2024-02-13T00:43:03.369083+0000","last_undegraded":"2024-02-13T00:43:03.369083+0000","last_fullsized":"2024-02-13T00:43:03.369083+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:43:00.776560+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.776560+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":191,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.348801+0000","last_change":"2024-02-13T00:42:24.059383+0000","last_active":"2024-02-13T00:43:03.348801+0000","last_peered":"2024-02-13T00:43:03.348801+0000","last_clean":"2024-02-13T00:43:03.348801+0000","last_became_active":"2024-02-13T00:37:41.218346+0000","last_became_peered":"2024-02-13T00:37:41.218346+0000","last_unstale":"2024-02-13T00:43:03.348801+0000","last_undegraded":"2024-02-13T00:43:03.348801+0000","last_fullsized":"2024-02-13T00:43:03.348801+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:24.059313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:24.059313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"64'8","reported_seq":265,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.279639+0000","last_change":"2024-02-13T00:41:56.423094+0000","last_active":"2024-02-13T00:43:04.279639+0000","last_peered":"2024-02-13T00:43:04.279639+0000","last_clean":"2024-02-13T00:43:04.279639+0000","last_became_active":"2024-02-13T00:37:45.249684+0000","last_became_peered":"2024-02-13T00:37:45.249684+0000","last_unstale":"2024-02-13T00:43:04.279639+0000","last_undegraded":"2024-02-13T00:43:04.279639+0000","last_fullsized":"2024-02-13T00:43:04.279639+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:56.422996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.422996+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":159,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.353079+0000","last_change":"2024-02-13T00:40:40.750384+0000","last_active":"2024-02-13T00:43:03.353079+0000","last_peered":"2024-02-13T00:43:03.353079+0000","last_clean":"2024-02-13T00:43:03.353079+0000","last_became_active":"2024-02-13T00:40:35.851908+0000","last_became_peered":"2024-02-13T00:40:35.851908+0000","last_unstale":"2024-02-13T00:43:03.353079+0000","last_undegraded":"2024-02-13T00:43:03.353079+0000","last_fullsized":"2024-02-13T00:43:03.353079+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:40:40.750272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:40.750272+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1a","version":"108'5","reported_seq":137,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.368935+0000","last_change":"2024-02-13T00:41:01.743545+0000","last_active":"2024-02-13T00:43:03.368935+0000","last_peered":"2024-02-13T00:43:03.368935+0000","last_clean":"2024-02-13T00:43:03.368935+0000","last_became_active":"2024-02-13T00:41:01.739001+0000","last_became_peered":"2024-02-13T00:41:01.739001+0000","last_unstale":"2024-02-13T00:43:03.368935+0000","last_undegraded":"2024-02-13T00:43:03.368935+0000","last_fullsized":"2024-02-13T00:43:03.368935+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":23,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.345961+0000","last_change":"2024-02-13T00:42:53.370114+0000","last_active":"2024-02-13T00:43:03.345961+0000","last_peered":"2024-02-13T00:43:03.345961+0000","last_clean":"2024-02-13T00:43:03.345961+0000","last_became_active":"2024-02-13T00:42:53.369851+0000","last_became_peered":"2024-02-13T00:42:53.369851+0000","last_unstale":"2024-02-13T00:43:03.345961+0000","last_undegraded":"2024-02-13T00:43:03.345961+0000","last_fullsized":"2024-02-13T00:43:03.345961+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":214,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.343507+0000","last_change":"2024-02-13T00:42:21.069550+0000","last_active":"2024-02-13T00:43:03.343507+0000","last_peered":"2024-02-13T00:43:03.343507+0000","last_clean":"2024-02-13T00:43:03.343507+0000","last_became_active":"2024-02-13T00:37:43.232161+0000","last_became_peered":"2024-02-13T00:37:43.232161+0000","last_unstale":"2024-02-13T00:43:03.343507+0000","last_undegraded":"2024-02-13T00:43:03.343507+0000","last_fullsized":"2024-02-13T00:43:03.343507+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:42:21.069482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.069482+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":197,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347178+0000","last_change":"2024-02-13T00:39:52.206001+0000","last_active":"2024-02-13T00:43:03.347178+0000","last_peered":"2024-02-13T00:43:03.347178+0000","last_clean":"2024-02-13T00:43:03.347178+0000","last_became_active":"2024-02-13T00:37:41.217979+0000","last_became_peered":"2024-02-13T00:37:41.217979+0000","last_unstale":"2024-02-13T00:43:03.347178+0000","last_undegraded":"2024-02-13T00:43:03.347178+0000","last_fullsized":"2024-02-13T00:43:03.347178+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:39:52.205926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:39:52.205926+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"96'9","reported_seq":204,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.279591+0000","last_change":"2024-02-13T00:42:23.496877+0000","last_active":"2024-02-13T00:43:04.279591+0000","last_peered":"2024-02-13T00:43:04.279591+0000","last_clean":"2024-02-13T00:43:04.279591+0000","last_became_active":"2024-02-13T00:40:35.847293+0000","last_became_peered":"2024-02-13T00:40:35.847293+0000","last_unstale":"2024-02-13T00:43:04.279591+0000","last_undegraded":"2024-02-13T00:43:04.279591+0000","last_fullsized":"2024-02-13T00:43:04.279591+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-02-13T00:42:23.496800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:23.496800+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":184,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.342804+0000","last_change":"2024-02-13T00:40:58.541216+0000","last_active":"2024-02-13T00:43:03.342804+0000","last_peered":"2024-02-13T00:43:03.342804+0000","last_clean":"2024-02-13T00:43:03.342804+0000","last_became_active":"2024-02-13T00:37:47.260477+0000","last_became_peered":"2024-02-13T00:37:47.260477+0000","last_unstale":"2024-02-13T00:43:03.342804+0000","last_undegraded":"2024-02-13T00:43:03.342804+0000","last_fullsized":"2024-02-13T00:43:03.342804+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.541144+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.356707+0000","last_change":"2024-02-13T00:42:53.351622+0000","last_active":"2024-02-13T00:43:03.356707+0000","last_peered":"2024-02-13T00:43:03.356707+0000","last_clean":"2024-02-13T00:43:03.356707+0000","last_became_active":"2024-02-13T00:42:53.350337+0000","last_became_peered":"2024-02-13T00:42:53.350337+0000","last_unstale":"2024-02-13T00:43:03.356707+0000","last_undegraded":"2024-02-13T00:43:03.356707+0000","last_fullsized":"2024-02-13T00:43:03.356707+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":131,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.356041+0000","last_change":"2024-02-13T00:41:01.747617+0000","last_active":"2024-02-13T00:43:03.356041+0000","last_peered":"2024-02-13T00:43:03.356041+0000","last_clean":"2024-02-13T00:43:03.356041+0000","last_became_active":"2024-02-13T00:41:01.746476+0000","last_became_peered":"2024-02-13T00:41:01.746476+0000","last_unstale":"2024-02-13T00:43:03.356041+0000","last_undegraded":"2024-02-13T00:43:03.356041+0000","last_fullsized":"2024-02-13T00:43:03.356041+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":139,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.343889+0000","last_change":"2024-02-13T00:42:52.245826+0000","last_active":"2024-02-13T00:43:03.343889+0000","last_peered":"2024-02-13T00:43:03.343889+0000","last_clean":"2024-02-13T00:43:03.343889+0000","last_became_active":"2024-02-13T00:42:03.424601+0000","last_became_peered":"2024-02-13T00:42:03.424601+0000","last_unstale":"2024-02-13T00:43:03.343889+0000","last_undegraded":"2024-02-13T00:43:03.343889+0000","last_fullsized":"2024-02-13T00:43:03.343889+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:42:52.245750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.245750+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350490+0000","last_change":"2024-02-13T00:42:53.376425+0000","last_active":"2024-02-13T00:43:03.350490+0000","last_peered":"2024-02-13T00:43:03.350490+0000","last_clean":"2024-02-13T00:43:03.350490+0000","last_became_active":"2024-02-13T00:42:53.375971+0000","last_became_peered":"2024-02-13T00:42:53.375971+0000","last_unstale":"2024-02-13T00:43:03.350490+0000","last_undegraded":"2024-02-13T00:43:03.350490+0000","last_fullsized":"2024-02-13T00:43:03.350490+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:32.399475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.399475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":200,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.345504+0000","last_change":"2024-02-13T00:42:57.546591+0000","last_active":"2024-02-13T00:43:03.345504+0000","last_peered":"2024-02-13T00:43:03.345504+0000","last_clean":"2024-02-13T00:43:03.345504+0000","last_became_active":"2024-02-13T00:42:03.395752+0000","last_became_peered":"2024-02-13T00:42:03.395752+0000","last_unstale":"2024-02-13T00:43:03.345504+0000","last_undegraded":"2024-02-13T00:43:03.345504+0000","last_fullsized":"2024-02-13T00:43:03.345504+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:57.546517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.546517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":198,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.368686+0000","last_change":"2024-02-13T00:42:47.782415+0000","last_active":"2024-02-13T00:43:03.368686+0000","last_peered":"2024-02-13T00:43:03.368686+0000","last_clean":"2024-02-13T00:43:03.368686+0000","last_became_active":"2024-02-13T00:42:03.428380+0000","last_became_peered":"2024-02-13T00:42:03.428380+0000","last_unstale":"2024-02-13T00:43:03.368686+0000","last_undegraded":"2024-02-13T00:43:03.368686+0000","last_fullsized":"2024-02-13T00:43:03.368686+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.782345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.782345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.18","version":"107'2","reported_seq":147,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.343034+0000","last_change":"2024-02-13T00:42:08.249193+0000","last_active":"2024-02-13T00:43:03.343034+0000","last_peered":"2024-02-13T00:43:03.343034+0000","last_clean":"2024-02-13T00:43:03.343034+0000","last_became_active":"2024-02-13T00:42:03.430784+0000","last_became_peered":"2024-02-13T00:42:03.430784+0000","last_unstale":"2024-02-13T00:43:03.343034+0000","last_undegraded":"2024-02-13T00:43:03.343034+0000","last_fullsized":"2024-02-13T00:43:03.343034+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":131,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.341114+0000","last_change":"2024-02-13T00:41:01.583063+0000","last_active":"2024-02-13T00:43:03.341114+0000","last_peered":"2024-02-13T00:43:03.341114+0000","last_clean":"2024-02-13T00:43:03.341114+0000","last_became_active":"2024-02-13T00:41:01.582361+0000","last_became_peered":"2024-02-13T00:41:01.582361+0000","last_unstale":"2024-02-13T00:43:03.341114+0000","last_undegraded":"2024-02-13T00:43:03.341114+0000","last_fullsized":"2024-02-13T00:43:03.341114+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"149'9","reported_seq":262,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.279830+0000","last_change":"2024-02-13T00:41:44.698967+0000","last_active":"2024-02-13T00:43:04.279830+0000","last_peered":"2024-02-13T00:43:04.279830+0000","last_clean":"2024-02-13T00:43:04.279830+0000","last_became_active":"2024-02-13T00:37:45.245528+0000","last_became_peered":"2024-02-13T00:37:45.245528+0000","last_unstale":"2024-02-13T00:43:04.279830+0000","last_undegraded":"2024-02-13T00:43:04.279830+0000","last_fullsized":"2024-02-13T00:43:04.279830+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'9","last_scrub_stamp":"2024-02-13T00:41:44.698906+0000","last_deep_scrub":"149'9","last_deep_scrub_stamp":"2024-02-13T00:41:44.698906+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.698906+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":231,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.342164+0000","last_change":"2024-02-13T00:43:00.636883+0000","last_active":"2024-02-13T00:43:03.342164+0000","last_peered":"2024-02-13T00:43:03.342164+0000","last_clean":"2024-02-13T00:43:03.342164+0000","last_became_active":"2024-02-13T00:42:03.393132+0000","last_became_peered":"2024-02-13T00:42:03.393132+0000","last_unstale":"2024-02-13T00:43:03.342164+0000","last_undegraded":"2024-02-13T00:43:03.342164+0000","last_fullsized":"2024-02-13T00:43:03.342164+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2024-02-13T00:43:00.636798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.636798+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":31,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.557594+0000","last_change":"2024-02-13T00:43:04.557594+0000","last_active":"2024-02-13T00:43:04.557594+0000","last_peered":"2024-02-13T00:43:04.557594+0000","last_clean":"2024-02-13T00:43:04.557594+0000","last_became_active":"2024-02-13T00:42:53.374514+0000","last_became_peered":"2024-02-13T00:42:53.374514+0000","last_unstale":"2024-02-13T00:43:04.557594+0000","last_undegraded":"2024-02-13T00:43:04.557594+0000","last_fullsized":"2024-02-13T00:43:04.557594+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.557475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.557475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":151,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347126+0000","last_change":"2024-02-13T00:42:53.443367+0000","last_active":"2024-02-13T00:43:03.347126+0000","last_peered":"2024-02-13T00:43:03.347126+0000","last_clean":"2024-02-13T00:43:03.347126+0000","last_became_active":"2024-02-13T00:42:53.372410+0000","last_became_peered":"2024-02-13T00:42:53.372410+0000","last_unstale":"2024-02-13T00:43:03.347126+0000","last_undegraded":"2024-02-13T00:43:03.347126+0000","last_fullsized":"2024-02-13T00:43:03.347126+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"108'5","reported_seq":137,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.345871+0000","last_change":"2024-02-13T00:41:01.623530+0000","last_active":"2024-02-13T00:43:03.345871+0000","last_peered":"2024-02-13T00:43:03.345871+0000","last_clean":"2024-02-13T00:43:03.345871+0000","last_became_active":"2024-02-13T00:41:01.622857+0000","last_became_peered":"2024-02-13T00:41:01.622857+0000","last_unstale":"2024-02-13T00:43:03.345871+0000","last_undegraded":"2024-02-13T00:43:03.345871+0000","last_fullsized":"2024-02-13T00:43:03.345871+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":131,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.369022+0000","last_change":"2024-02-13T00:41:01.682441+0000","last_active":"2024-02-13T00:43:03.369022+0000","last_peered":"2024-02-13T00:43:03.369022+0000","last_clean":"2024-02-13T00:43:03.369022+0000","last_became_active":"2024-02-13T00:41:01.682162+0000","last_became_peered":"2024-02-13T00:41:01.682162+0000","last_unstale":"2024-02-13T00:43:03.369022+0000","last_undegraded":"2024-02-13T00:43:03.369022+0000","last_fullsized":"2024-02-13T00:43:03.369022+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":155,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.356885+0000","last_change":"2024-02-13T00:42:54.842136+0000","last_active":"2024-02-13T00:43:03.356885+0000","last_peered":"2024-02-13T00:43:03.356885+0000","last_clean":"2024-02-13T00:43:03.356885+0000","last_became_active":"2024-02-13T00:40:35.856681+0000","last_became_peered":"2024-02-13T00:40:35.856681+0000","last_unstale":"2024-02-13T00:43:03.356885+0000","last_undegraded":"2024-02-13T00:43:03.356885+0000","last_fullsized":"2024-02-13T00:43:03.356885+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:54.842056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:54.842056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":255,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.368946+0000","last_change":"2024-02-13T00:42:53.881669+0000","last_active":"2024-02-13T00:43:03.368946+0000","last_peered":"2024-02-13T00:43:03.368946+0000","last_clean":"2024-02-13T00:43:03.368946+0000","last_became_active":"2024-02-13T00:37:43.324275+0000","last_became_peered":"2024-02-13T00:37:43.324275+0000","last_unstale":"2024-02-13T00:43:03.368946+0000","last_undegraded":"2024-02-13T00:43:03.368946+0000","last_fullsized":"2024-02-13T00:43:03.368946+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:42:53.881594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.881594+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":200,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.342823+0000","last_change":"2024-02-13T00:42:03.402563+0000","last_active":"2024-02-13T00:43:03.342823+0000","last_peered":"2024-02-13T00:43:03.342823+0000","last_clean":"2024-02-13T00:43:03.342823+0000","last_became_active":"2024-02-13T00:42:03.402206+0000","last_became_peered":"2024-02-13T00:42:03.402206+0000","last_unstale":"2024-02-13T00:43:03.342823+0000","last_undegraded":"2024-02-13T00:43:03.342823+0000","last_fullsized":"2024-02-13T00:43:03.342823+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.236727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.236727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":150,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.367627+0000","last_change":"2024-02-13T00:41:02.817359+0000","last_active":"2024-02-13T00:43:03.367627+0000","last_peered":"2024-02-13T00:43:03.367627+0000","last_clean":"2024-02-13T00:43:03.367627+0000","last_became_active":"2024-02-13T00:40:35.857936+0000","last_became_peered":"2024-02-13T00:40:35.857936+0000","last_unstale":"2024-02-13T00:43:03.367627+0000","last_undegraded":"2024-02-13T00:43:03.367627+0000","last_fullsized":"2024-02-13T00:43:03.367627+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.16","version":"108'5","reported_seq":150,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.364535+0000","last_change":"2024-02-13T00:42:06.795504+0000","last_active":"2024-02-13T00:43:03.364535+0000","last_peered":"2024-02-13T00:43:03.364535+0000","last_clean":"2024-02-13T00:43:03.364535+0000","last_became_active":"2024-02-13T00:42:03.415083+0000","last_became_peered":"2024-02-13T00:42:03.415083+0000","last_unstale":"2024-02-13T00:43:03.364535+0000","last_undegraded":"2024-02-13T00:43:03.364535+0000","last_fullsized":"2024-02-13T00:43:03.364535+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":145,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347308+0000","last_change":"2024-02-13T00:42:48.300467+0000","last_active":"2024-02-13T00:43:03.347308+0000","last_peered":"2024-02-13T00:43:03.347308+0000","last_clean":"2024-02-13T00:43:03.347308+0000","last_became_active":"2024-02-13T00:42:03.374865+0000","last_became_peered":"2024-02-13T00:42:03.374865+0000","last_unstale":"2024-02-13T00:43:03.347308+0000","last_undegraded":"2024-02-13T00:43:03.347308+0000","last_fullsized":"2024-02-13T00:43:03.347308+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:48.300400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.300400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":73,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.353460+0000","last_change":"2024-02-13T00:42:03.387052+0000","last_active":"2024-02-13T00:43:03.353460+0000","last_peered":"2024-02-13T00:43:03.353460+0000","last_clean":"2024-02-13T00:43:03.353460+0000","last_became_active":"2024-02-13T00:42:03.386776+0000","last_became_peered":"2024-02-13T00:42:03.386776+0000","last_unstale":"2024-02-13T00:43:03.353460+0000","last_undegraded":"2024-02-13T00:43:03.353460+0000","last_fullsized":"2024-02-13T00:43:03.353460+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:55.358367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:55.358367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.360228+0000","last_change":"2024-02-13T00:42:20.688201+0000","last_active":"2024-02-13T00:43:03.360228+0000","last_peered":"2024-02-13T00:43:03.360228+0000","last_clean":"2024-02-13T00:43:03.360228+0000","last_became_active":"2024-02-13T00:37:43.322450+0000","last_became_peered":"2024-02-13T00:37:43.322450+0000","last_unstale":"2024-02-13T00:43:03.360228+0000","last_undegraded":"2024-02-13T00:43:03.360228+0000","last_fullsized":"2024-02-13T00:43:03.360228+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:42:20.688132+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.352537+0000","last_change":"2024-02-13T00:42:53.361013+0000","last_active":"2024-02-13T00:43:03.352537+0000","last_peered":"2024-02-13T00:43:03.352537+0000","last_clean":"2024-02-13T00:43:03.352537+0000","last_became_active":"2024-02-13T00:42:53.360496+0000","last_became_peered":"2024-02-13T00:42:53.360496+0000","last_unstale":"2024-02-13T00:43:03.352537+0000","last_undegraded":"2024-02-13T00:43:03.352537+0000","last_fullsized":"2024-02-13T00:43:03.352537+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:45.198231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:45.198231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":194,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347261+0000","last_change":"2024-02-13T00:42:53.198357+0000","last_active":"2024-02-13T00:43:03.347261+0000","last_peered":"2024-02-13T00:43:03.347261+0000","last_clean":"2024-02-13T00:43:03.347261+0000","last_became_active":"2024-02-13T00:42:03.422365+0000","last_became_peered":"2024-02-13T00:42:03.422365+0000","last_unstale":"2024-02-13T00:43:03.347261+0000","last_undegraded":"2024-02-13T00:43:03.347261+0000","last_fullsized":"2024-02-13T00:43:03.347261+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":132,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.342969+0000","last_change":"2024-02-13T00:41:01.672342+0000","last_active":"2024-02-13T00:43:03.342969+0000","last_peered":"2024-02-13T00:43:03.342969+0000","last_clean":"2024-02-13T00:43:03.342969+0000","last_became_active":"2024-02-13T00:41:01.671979+0000","last_became_peered":"2024-02-13T00:41:01.671979+0000","last_unstale":"2024-02-13T00:43:03.342969+0000","last_undegraded":"2024-02-13T00:43:03.342969+0000","last_fullsized":"2024-02-13T00:43:03.342969+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":136,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.355205+0000","last_change":"2024-02-13T00:42:50.325127+0000","last_active":"2024-02-13T00:43:03.355205+0000","last_peered":"2024-02-13T00:43:03.355205+0000","last_clean":"2024-02-13T00:43:03.355205+0000","last_became_active":"2024-02-13T00:41:01.590438+0000","last_became_peered":"2024-02-13T00:41:01.590438+0000","last_unstale":"2024-02-13T00:43:03.355205+0000","last_undegraded":"2024-02-13T00:43:03.355205+0000","last_fullsized":"2024-02-13T00:43:03.355205+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:50.325063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:50.325063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":191,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.345623+0000","last_change":"2024-02-13T00:41:54.422549+0000","last_active":"2024-02-13T00:43:03.345623+0000","last_peered":"2024-02-13T00:43:03.345623+0000","last_clean":"2024-02-13T00:43:03.345623+0000","last_became_active":"2024-02-13T00:37:45.245956+0000","last_became_peered":"2024-02-13T00:37:45.245956+0000","last_unstale":"2024-02-13T00:43:03.345623+0000","last_undegraded":"2024-02-13T00:43:03.345623+0000","last_fullsized":"2024-02-13T00:43:03.345623+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:54.422484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:54.422484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":229,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.355507+0000","last_change":"2024-02-13T00:41:52.739904+0000","last_active":"2024-02-13T00:43:03.355507+0000","last_peered":"2024-02-13T00:43:03.355507+0000","last_clean":"2024-02-13T00:43:03.355507+0000","last_became_active":"2024-02-13T00:39:05.968155+0000","last_became_peered":"2024-02-13T00:39:05.968155+0000","last_unstale":"2024-02-13T00:43:03.355507+0000","last_undegraded":"2024-02-13T00:43:03.355507+0000","last_fullsized":"2024-02-13T00:43:03.355507+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:41:52.739833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.739833+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":167,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.348568+0000","last_change":"2024-02-13T00:41:54.457692+0000","last_active":"2024-02-13T00:43:03.348568+0000","last_peered":"2024-02-13T00:43:03.348568+0000","last_clean":"2024-02-13T00:43:03.348568+0000","last_became_active":"2024-02-13T00:39:05.967772+0000","last_became_peered":"2024-02-13T00:39:05.967772+0000","last_unstale":"2024-02-13T00:43:03.348568+0000","last_undegraded":"2024-02-13T00:43:03.348568+0000","last_fullsized":"2024-02-13T00:43:03.348568+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:54.457605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:54.457605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":190,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.351277+0000","last_change":"2024-02-13T00:41:34.352119+0000","last_active":"2024-02-13T00:43:03.351277+0000","last_peered":"2024-02-13T00:43:03.351277+0000","last_clean":"2024-02-13T00:43:03.351277+0000","last_became_active":"2024-02-13T00:37:41.217577+0000","last_became_peered":"2024-02-13T00:37:41.217577+0000","last_unstale":"2024-02-13T00:43:03.351277+0000","last_undegraded":"2024-02-13T00:43:03.351277+0000","last_fullsized":"2024-02-13T00:43:03.351277+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.352030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.352030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":185,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.249792+0000","last_change":"2024-02-13T00:43:04.249772+0000","last_active":"2024-02-13T00:43:04.249792+0000","last_peered":"2024-02-13T00:43:04.249792+0000","last_clean":"2024-02-13T00:43:04.249792+0000","last_became_active":"2024-02-13T00:37:47.262024+0000","last_became_peered":"2024-02-13T00:37:47.262024+0000","last_unstale":"2024-02-13T00:43:04.249792+0000","last_undegraded":"2024-02-13T00:43:04.249792+0000","last_fullsized":"2024-02-13T00:43:04.249792+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":79,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.369857+0000","last_change":"2024-02-13T00:43:01.777736+0000","last_active":"2024-02-13T00:43:03.369857+0000","last_peered":"2024-02-13T00:43:03.369857+0000","last_clean":"2024-02-13T00:43:03.369857+0000","last_became_active":"2024-02-13T00:42:03.434658+0000","last_became_peered":"2024-02-13T00:42:03.434658+0000","last_unstale":"2024-02-13T00:43:03.369857+0000","last_undegraded":"2024-02-13T00:43:03.369857+0000","last_fullsized":"2024-02-13T00:43:03.369857+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:01.777674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:43:01.777674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":140,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.343051+0000","last_change":"2024-02-13T00:42:03.399569+0000","last_active":"2024-02-13T00:43:03.343051+0000","last_peered":"2024-02-13T00:43:03.343051+0000","last_clean":"2024-02-13T00:43:03.343051+0000","last_became_active":"2024-02-13T00:42:03.398892+0000","last_became_peered":"2024-02-13T00:42:03.398892+0000","last_unstale":"2024-02-13T00:43:03.343051+0000","last_undegraded":"2024-02-13T00:43:03.343051+0000","last_fullsized":"2024-02-13T00:43:03.343051+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":23,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350125+0000","last_change":"2024-02-13T00:42:53.367878+0000","last_active":"2024-02-13T00:43:03.350125+0000","last_peered":"2024-02-13T00:43:03.350125+0000","last_clean":"2024-02-13T00:43:03.350125+0000","last_became_active":"2024-02-13T00:42:53.367560+0000","last_became_peered":"2024-02-13T00:42:53.367560+0000","last_unstale":"2024-02-13T00:43:03.350125+0000","last_undegraded":"2024-02-13T00:43:03.350125+0000","last_fullsized":"2024-02-13T00:43:03.350125+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:56.666003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.666003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"157'21","reported_seq":161,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.352136+0000","last_change":"2024-02-13T00:42:53.360831+0000","last_active":"2024-02-13T00:43:03.352136+0000","last_peered":"2024-02-13T00:43:03.352136+0000","last_clean":"2024-02-13T00:43:03.352136+0000","last_became_active":"2024-02-13T00:42:53.359960+0000","last_became_peered":"2024-02-13T00:42:53.359960+0000","last_unstale":"2024-02-13T00:43:03.352136+0000","last_undegraded":"2024-02-13T00:43:03.352136+0000","last_fullsized":"2024-02-13T00:43:03.352136+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.415907+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":155,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350151+0000","last_change":"2024-02-13T00:42:53.376525+0000","last_active":"2024-02-13T00:43:03.350151+0000","last_peered":"2024-02-13T00:43:03.350151+0000","last_clean":"2024-02-13T00:43:03.350151+0000","last_became_active":"2024-02-13T00:42:53.376259+0000","last_became_peered":"2024-02-13T00:42:53.376259+0000","last_unstale":"2024-02-13T00:43:03.350151+0000","last_undegraded":"2024-02-13T00:43:03.350151+0000","last_fullsized":"2024-02-13T00:43:03.350151+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:23.561163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:42:23.561163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.340335+0000","last_change":"2024-02-13T00:41:34.472385+0000","last_active":"2024-02-13T00:43:03.340335+0000","last_peered":"2024-02-13T00:43:03.340335+0000","last_clean":"2024-02-13T00:43:03.340335+0000","last_became_active":"2024-02-13T00:37:41.216101+0000","last_became_peered":"2024-02-13T00:37:41.216101+0000","last_unstale":"2024-02-13T00:43:03.340335+0000","last_undegraded":"2024-02-13T00:43:03.340335+0000","last_fullsized":"2024-02-13T00:43:03.340335+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.472312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.472312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":204,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.368795+0000","last_change":"2024-02-13T00:42:27.544386+0000","last_active":"2024-02-13T00:43:03.368795+0000","last_peered":"2024-02-13T00:43:03.368795+0000","last_clean":"2024-02-13T00:43:03.368795+0000","last_became_active":"2024-02-13T00:42:03.416523+0000","last_became_peered":"2024-02-13T00:42:03.416523+0000","last_unstale":"2024-02-13T00:43:03.368795+0000","last_undegraded":"2024-02-13T00:43:03.368795+0000","last_fullsized":"2024-02-13T00:43:03.368795+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.544320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.544320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":189,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.345709+0000","last_change":"2024-02-13T00:41:51.414166+0000","last_active":"2024-02-13T00:43:03.345709+0000","last_peered":"2024-02-13T00:43:03.345709+0000","last_clean":"2024-02-13T00:43:03.345709+0000","last_became_active":"2024-02-13T00:37:47.253996+0000","last_became_peered":"2024-02-13T00:37:47.253996+0000","last_unstale":"2024-02-13T00:43:03.345709+0000","last_undegraded":"2024-02-13T00:43:03.345709+0000","last_fullsized":"2024-02-13T00:43:03.345709+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.13","version":"108'9","reported_seq":46,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347082+0000","last_change":"2024-02-13T00:42:57.517602+0000","last_active":"2024-02-13T00:43:03.347082+0000","last_peered":"2024-02-13T00:43:03.347082+0000","last_clean":"2024-02-13T00:43:03.347082+0000","last_became_active":"2024-02-13T00:42:53.368593+0000","last_became_peered":"2024-02-13T00:42:53.368593+0000","last_unstale":"2024-02-13T00:43:03.347082+0000","last_undegraded":"2024-02-13T00:43:03.347082+0000","last_fullsized":"2024-02-13T00:43:03.347082+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-02-13T00:42:57.517535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.517535+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":131,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.360809+0000","last_change":"2024-02-13T00:41:01.594433+0000","last_active":"2024-02-13T00:43:03.360809+0000","last_peered":"2024-02-13T00:43:03.360809+0000","last_clean":"2024-02-13T00:43:03.360809+0000","last_became_active":"2024-02-13T00:41:01.593970+0000","last_became_peered":"2024-02-13T00:41:01.593970+0000","last_unstale":"2024-02-13T00:43:03.360809+0000","last_undegraded":"2024-02-13T00:43:03.360809+0000","last_fullsized":"2024-02-13T00:43:03.360809+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":192,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.355331+0000","last_change":"2024-02-13T00:42:26.313657+0000","last_active":"2024-02-13T00:43:03.355331+0000","last_peered":"2024-02-13T00:43:03.355331+0000","last_clean":"2024-02-13T00:43:03.355331+0000","last_became_active":"2024-02-13T00:37:45.248383+0000","last_became_peered":"2024-02-13T00:37:45.248383+0000","last_unstale":"2024-02-13T00:43:03.355331+0000","last_undegraded":"2024-02-13T00:43:03.355331+0000","last_fullsized":"2024-02-13T00:43:03.355331+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:26.313580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:26.313580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":231,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.343713+0000","last_change":"2024-02-13T00:42:54.175029+0000","last_active":"2024-02-13T00:43:03.343713+0000","last_peered":"2024-02-13T00:43:03.343713+0000","last_clean":"2024-02-13T00:43:03.343713+0000","last_became_active":"2024-02-13T00:37:43.231120+0000","last_became_peered":"2024-02-13T00:37:43.231120+0000","last_unstale":"2024-02-13T00:43:03.343713+0000","last_undegraded":"2024-02-13T00:43:03.343713+0000","last_fullsized":"2024-02-13T00:43:03.343713+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:42:54.174950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:54.174950+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":191,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350107+0000","last_change":"2024-02-13T00:40:57.469853+0000","last_active":"2024-02-13T00:43:03.350107+0000","last_peered":"2024-02-13T00:43:03.350107+0000","last_clean":"2024-02-13T00:43:03.350107+0000","last_became_active":"2024-02-13T00:37:41.214442+0000","last_became_peered":"2024-02-13T00:37:41.214442+0000","last_unstale":"2024-02-13T00:43:03.350107+0000","last_undegraded":"2024-02-13T00:43:03.350107+0000","last_fullsized":"2024-02-13T00:43:03.350107+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.469743+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":525,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.544102+0000","last_change":"2024-02-13T00:42:08.427157+0000","last_active":"2024-02-13T00:43:04.544102+0000","last_peered":"2024-02-13T00:43:04.544102+0000","last_clean":"2024-02-13T00:43:04.544102+0000","last_became_active":"2024-02-13T00:42:03.428394+0000","last_became_peered":"2024-02-13T00:42:03.428394+0000","last_unstale":"2024-02-13T00:43:04.544102+0000","last_undegraded":"2024-02-13T00:43:04.544102+0000","last_fullsized":"2024-02-13T00:43:04.544102+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-13T00:41:35.787179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.787179+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":315,"num_read_kb":310,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":148,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.360582+0000","last_change":"2024-02-13T00:42:53.371968+0000","last_active":"2024-02-13T00:43:03.360582+0000","last_peered":"2024-02-13T00:43:03.360582+0000","last_clean":"2024-02-13T00:43:03.360582+0000","last_became_active":"2024-02-13T00:42:53.371547+0000","last_became_peered":"2024-02-13T00:42:53.371547+0000","last_unstale":"2024-02-13T00:43:03.360582+0000","last_undegraded":"2024-02-13T00:43:03.360582+0000","last_fullsized":"2024-02-13T00:43:03.360582+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"108'7","reported_seq":139,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347392+0000","last_change":"2024-02-13T00:41:01.721381+0000","last_active":"2024-02-13T00:43:03.347392+0000","last_peered":"2024-02-13T00:43:03.347392+0000","last_clean":"2024-02-13T00:43:03.347392+0000","last_became_active":"2024-02-13T00:41:01.718821+0000","last_became_peered":"2024-02-13T00:41:01.718821+0000","last_unstale":"2024-02-13T00:43:03.347392+0000","last_undegraded":"2024-02-13T00:43:03.347392+0000","last_fullsized":"2024-02-13T00:43:03.347392+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":131,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.371980+0000","last_change":"2024-02-13T00:41:01.548873+0000","last_active":"2024-02-13T00:43:03.371980+0000","last_peered":"2024-02-13T00:43:03.371980+0000","last_clean":"2024-02-13T00:43:03.371980+0000","last_became_active":"2024-02-13T00:41:01.547408+0000","last_became_peered":"2024-02-13T00:41:01.547408+0000","last_unstale":"2024-02-13T00:43:03.371980+0000","last_undegraded":"2024-02-13T00:43:03.371980+0000","last_fullsized":"2024-02-13T00:43:03.371980+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":186,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.352607+0000","last_change":"2024-02-13T00:41:00.881481+0000","last_active":"2024-02-13T00:43:03.352607+0000","last_peered":"2024-02-13T00:43:03.352607+0000","last_clean":"2024-02-13T00:43:03.352607+0000","last_became_active":"2024-02-13T00:37:45.247012+0000","last_became_peered":"2024-02-13T00:37:45.247012+0000","last_unstale":"2024-02-13T00:43:03.352607+0000","last_undegraded":"2024-02-13T00:43:03.352607+0000","last_fullsized":"2024-02-13T00:43:03.352607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.881411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.881411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"80'45","reported_seq":328,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.279565+0000","last_change":"2024-02-13T00:40:12.047980+0000","last_active":"2024-02-13T00:43:04.279565+0000","last_peered":"2024-02-13T00:43:04.279565+0000","last_clean":"2024-02-13T00:43:04.279565+0000","last_became_active":"2024-02-13T00:37:43.232014+0000","last_became_peered":"2024-02-13T00:37:43.232014+0000","last_unstale":"2024-02-13T00:43:04.279565+0000","last_undegraded":"2024-02-13T00:43:04.279565+0000","last_fullsized":"2024-02-13T00:43:04.279565+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'45","last_scrub_stamp":"2024-02-13T00:40:12.047869+0000","last_deep_scrub":"80'45","last_deep_scrub_stamp":"2024-02-13T00:38:57.537309+0000","last_clean_scrub_stamp":"2024-02-13T00:40:12.047869+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":199,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.339929+0000","last_change":"2024-02-13T00:42:03.394450+0000","last_active":"2024-02-13T00:43:03.339929+0000","last_peered":"2024-02-13T00:43:03.339929+0000","last_clean":"2024-02-13T00:43:03.339929+0000","last_became_active":"2024-02-13T00:42:03.394072+0000","last_became_peered":"2024-02-13T00:42:03.394072+0000","last_unstale":"2024-02-13T00:43:03.339929+0000","last_undegraded":"2024-02-13T00:43:03.339929+0000","last_fullsized":"2024-02-13T00:43:03.339929+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.518778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.518778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":184,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.364369+0000","last_change":"2024-02-13T00:42:25.457362+0000","last_active":"2024-02-13T00:43:03.364369+0000","last_peered":"2024-02-13T00:43:03.364369+0000","last_clean":"2024-02-13T00:43:03.364369+0000","last_became_active":"2024-02-13T00:37:47.252270+0000","last_became_peered":"2024-02-13T00:37:47.252270+0000","last_unstale":"2024-02-13T00:43:03.364369+0000","last_undegraded":"2024-02-13T00:43:03.364369+0000","last_fullsized":"2024-02-13T00:43:03.364369+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":132,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346413+0000","last_change":"2024-02-13T00:41:01.606464+0000","last_active":"2024-02-13T00:43:03.346413+0000","last_peered":"2024-02-13T00:43:03.346413+0000","last_clean":"2024-02-13T00:43:03.346413+0000","last_became_active":"2024-02-13T00:41:01.605731+0000","last_became_peered":"2024-02-13T00:41:01.605731+0000","last_unstale":"2024-02-13T00:43:03.346413+0000","last_undegraded":"2024-02-13T00:43:03.346413+0000","last_fullsized":"2024-02-13T00:43:03.346413+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":78,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346219+0000","last_change":"2024-02-13T00:42:19.407107+0000","last_active":"2024-02-13T00:43:03.346219+0000","last_peered":"2024-02-13T00:43:03.346219+0000","last_clean":"2024-02-13T00:43:03.346219+0000","last_became_active":"2024-02-13T00:42:03.392221+0000","last_became_peered":"2024-02-13T00:42:03.392221+0000","last_unstale":"2024-02-13T00:43:03.346219+0000","last_undegraded":"2024-02-13T00:43:03.346219+0000","last_fullsized":"2024-02-13T00:43:03.346219+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:19.407011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:19.407011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"227'11","reported_seq":274,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.279566+0000","last_change":"2024-02-13T00:43:02.423419+0000","last_active":"2024-02-13T00:43:04.279566+0000","last_peered":"2024-02-13T00:43:04.279566+0000","last_clean":"2024-02-13T00:43:04.279566+0000","last_became_active":"2024-02-13T00:42:03.405246+0000","last_became_peered":"2024-02-13T00:42:03.405246+0000","last_unstale":"2024-02-13T00:43:04.279566+0000","last_undegraded":"2024-02-13T00:43:04.279566+0000","last_fullsized":"2024-02-13T00:43:04.279566+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'11","last_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:43:02.423358+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":181,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346264+0000","last_change":"2024-02-13T00:41:52.517499+0000","last_active":"2024-02-13T00:43:03.346264+0000","last_peered":"2024-02-13T00:43:03.346264+0000","last_clean":"2024-02-13T00:43:03.346264+0000","last_became_active":"2024-02-13T00:39:05.965560+0000","last_became_peered":"2024-02-13T00:39:05.965560+0000","last_unstale":"2024-02-13T00:43:03.346264+0000","last_undegraded":"2024-02-13T00:43:03.346264+0000","last_fullsized":"2024-02-13T00:43:03.346264+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:52.517392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.517392+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":191,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.345579+0000","last_change":"2024-02-13T00:40:08.946462+0000","last_active":"2024-02-13T00:43:03.345579+0000","last_peered":"2024-02-13T00:43:03.345579+0000","last_clean":"2024-02-13T00:43:03.345579+0000","last_became_active":"2024-02-13T00:37:41.217777+0000","last_became_peered":"2024-02-13T00:37:41.217777+0000","last_unstale":"2024-02-13T00:43:03.345579+0000","last_undegraded":"2024-02-13T00:43:03.345579+0000","last_fullsized":"2024-02-13T00:43:03.345579+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:08.946388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:40:08.946388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":184,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.349569+0000","last_change":"2024-02-13T00:41:48.645080+0000","last_active":"2024-02-13T00:43:03.349569+0000","last_peered":"2024-02-13T00:43:03.349569+0000","last_clean":"2024-02-13T00:43:03.349569+0000","last_became_active":"2024-02-13T00:37:47.255928+0000","last_became_peered":"2024-02-13T00:37:47.255928+0000","last_unstale":"2024-02-13T00:43:03.349569+0000","last_undegraded":"2024-02-13T00:43:03.349569+0000","last_fullsized":"2024-02-13T00:43:03.349569+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"107'2","reported_seq":33,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350500+0000","last_change":"2024-02-13T00:42:53.378296+0000","last_active":"2024-02-13T00:43:03.350500+0000","last_peered":"2024-02-13T00:43:03.350500+0000","last_clean":"2024-02-13T00:43:03.350500+0000","last_became_active":"2024-02-13T00:42:53.377959+0000","last_became_peered":"2024-02-13T00:42:53.377959+0000","last_unstale":"2024-02-13T00:43:03.350500+0000","last_undegraded":"2024-02-13T00:43:03.350500+0000","last_fullsized":"2024-02-13T00:43:03.350500+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":131,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347024+0000","last_change":"2024-02-13T00:41:01.721182+0000","last_active":"2024-02-13T00:43:03.347024+0000","last_peered":"2024-02-13T00:43:03.347024+0000","last_clean":"2024-02-13T00:43:03.347024+0000","last_became_active":"2024-02-13T00:41:01.718793+0000","last_became_peered":"2024-02-13T00:41:01.718793+0000","last_unstale":"2024-02-13T00:43:03.347024+0000","last_undegraded":"2024-02-13T00:43:03.347024+0000","last_fullsized":"2024-02-13T00:43:03.347024+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":189,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.349390+0000","last_change":"2024-02-13T00:41:43.427096+0000","last_active":"2024-02-13T00:43:03.349390+0000","last_peered":"2024-02-13T00:43:03.349390+0000","last_clean":"2024-02-13T00:43:03.349390+0000","last_became_active":"2024-02-13T00:37:45.244408+0000","last_became_peered":"2024-02-13T00:37:45.244408+0000","last_unstale":"2024-02-13T00:43:03.349390+0000","last_undegraded":"2024-02-13T00:43:03.349390+0000","last_fullsized":"2024-02-13T00:43:03.349390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:43.426982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:43.426982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":211,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346957+0000","last_change":"2024-02-13T00:40:35.856095+0000","last_active":"2024-02-13T00:43:03.346957+0000","last_peered":"2024-02-13T00:43:03.346957+0000","last_clean":"2024-02-13T00:43:03.346957+0000","last_became_active":"2024-02-13T00:40:35.855722+0000","last_became_peered":"2024-02-13T00:40:35.855722+0000","last_unstale":"2024-02-13T00:43:03.346957+0000","last_undegraded":"2024-02-13T00:43:03.346957+0000","last_fullsized":"2024-02-13T00:43:03.346957+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'24","last_scrub_stamp":"2024-02-13T00:39:56.121046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:39:56.121046+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":36,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":159,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.360554+0000","last_change":"2024-02-13T00:40:48.809660+0000","last_active":"2024-02-13T00:43:03.360554+0000","last_peered":"2024-02-13T00:43:03.360554+0000","last_clean":"2024-02-13T00:43:03.360554+0000","last_became_active":"2024-02-13T00:40:35.852351+0000","last_became_peered":"2024-02-13T00:40:35.852351+0000","last_unstale":"2024-02-13T00:43:03.360554+0000","last_undegraded":"2024-02-13T00:43:03.360554+0000","last_fullsized":"2024-02-13T00:43:03.360554+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:40:48.809594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:48.809594+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":184,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.358751+0000","last_change":"2024-02-13T00:41:45.660743+0000","last_active":"2024-02-13T00:43:03.358751+0000","last_peered":"2024-02-13T00:43:03.358751+0000","last_clean":"2024-02-13T00:43:03.358751+0000","last_became_active":"2024-02-13T00:37:47.260242+0000","last_became_peered":"2024-02-13T00:37:47.260242+0000","last_unstale":"2024-02-13T00:43:03.358751+0000","last_undegraded":"2024-02-13T00:43:03.358751+0000","last_fullsized":"2024-02-13T00:43:03.358751+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:41:45.660653+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"108'5","reported_seq":37,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.353248+0000","last_change":"2024-02-13T00:42:53.372706+0000","last_active":"2024-02-13T00:43:03.353248+0000","last_peered":"2024-02-13T00:43:03.353248+0000","last_clean":"2024-02-13T00:43:03.353248+0000","last_became_active":"2024-02-13T00:42:53.372355+0000","last_became_peered":"2024-02-13T00:42:53.372355+0000","last_unstale":"2024-02-13T00:43:03.353248+0000","last_undegraded":"2024-02-13T00:43:03.353248+0000","last_fullsized":"2024-02-13T00:43:03.353248+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":131,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.355653+0000","last_change":"2024-02-13T00:41:01.594534+0000","last_active":"2024-02-13T00:43:03.355653+0000","last_peered":"2024-02-13T00:43:03.355653+0000","last_clean":"2024-02-13T00:43:03.355653+0000","last_became_active":"2024-02-13T00:41:01.590676+0000","last_became_peered":"2024-02-13T00:41:01.590676+0000","last_unstale":"2024-02-13T00:43:03.355653+0000","last_undegraded":"2024-02-13T00:43:03.355653+0000","last_fullsized":"2024-02-13T00:43:03.355653+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":234,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.355584+0000","last_change":"2024-02-13T00:40:49.767975+0000","last_active":"2024-02-13T00:43:03.355584+0000","last_peered":"2024-02-13T00:43:03.355584+0000","last_clean":"2024-02-13T00:43:03.355584+0000","last_became_active":"2024-02-13T00:37:43.322401+0000","last_became_peered":"2024-02-13T00:37:43.322401+0000","last_unstale":"2024-02-13T00:43:03.355584+0000","last_undegraded":"2024-02-13T00:43:03.355584+0000","last_fullsized":"2024-02-13T00:43:03.355584+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:40:49.767853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:49.767853+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":196,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.345358+0000","last_change":"2024-02-13T00:41:49.468480+0000","last_active":"2024-02-13T00:43:03.345358+0000","last_peered":"2024-02-13T00:43:03.345358+0000","last_clean":"2024-02-13T00:43:03.345358+0000","last_became_active":"2024-02-13T00:37:41.214091+0000","last_became_peered":"2024-02-13T00:37:41.214091+0000","last_unstale":"2024-02-13T00:43:03.345358+0000","last_undegraded":"2024-02-13T00:43:03.345358+0000","last_fullsized":"2024-02-13T00:43:03.345358+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:49.468410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:49.468410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":186,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.362695+0000","last_change":"2024-02-13T00:42:15.458861+0000","last_active":"2024-02-13T00:43:03.362695+0000","last_peered":"2024-02-13T00:43:03.362695+0000","last_clean":"2024-02-13T00:43:03.362695+0000","last_became_active":"2024-02-13T00:37:45.250632+0000","last_became_peered":"2024-02-13T00:37:45.250632+0000","last_unstale":"2024-02-13T00:43:03.362695+0000","last_undegraded":"2024-02-13T00:43:03.362695+0000","last_fullsized":"2024-02-13T00:43:03.362695+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.458752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.458752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":185,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350176+0000","last_change":"2024-02-13T00:40:59.468608+0000","last_active":"2024-02-13T00:43:03.350176+0000","last_peered":"2024-02-13T00:43:03.350176+0000","last_clean":"2024-02-13T00:43:03.350176+0000","last_became_active":"2024-02-13T00:37:47.254803+0000","last_became_peered":"2024-02-13T00:37:47.254803+0000","last_unstale":"2024-02-13T00:43:03.350176+0000","last_undegraded":"2024-02-13T00:43:03.350176+0000","last_fullsized":"2024-02-13T00:43:03.350176+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.468525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.468525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.0","version":"108'5","reported_seq":137,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350308+0000","last_change":"2024-02-13T00:41:01.580498+0000","last_active":"2024-02-13T00:43:03.350308+0000","last_peered":"2024-02-13T00:43:03.350308+0000","last_clean":"2024-02-13T00:43:03.350308+0000","last_became_active":"2024-02-13T00:41:01.579665+0000","last_became_peered":"2024-02-13T00:41:01.579665+0000","last_unstale":"2024-02-13T00:43:03.350308+0000","last_undegraded":"2024-02-13T00:43:03.350308+0000","last_fullsized":"2024-02-13T00:43:03.350308+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":73,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.353325+0000","last_change":"2024-02-13T00:42:03.399201+0000","last_active":"2024-02-13T00:43:03.353325+0000","last_peered":"2024-02-13T00:43:03.353325+0000","last_clean":"2024-02-13T00:43:03.353325+0000","last_became_active":"2024-02-13T00:42:03.393040+0000","last_became_peered":"2024-02-13T00:42:03.393040+0000","last_unstale":"2024-02-13T00:43:03.353325+0000","last_undegraded":"2024-02-13T00:43:03.353325+0000","last_fullsized":"2024-02-13T00:43:03.353325+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":212,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.362622+0000","last_change":"2024-02-13T00:41:42.695643+0000","last_active":"2024-02-13T00:43:03.362622+0000","last_peered":"2024-02-13T00:43:03.362622+0000","last_clean":"2024-02-13T00:43:03.362622+0000","last_became_active":"2024-02-13T00:37:43.324510+0000","last_became_peered":"2024-02-13T00:37:43.324510+0000","last_unstale":"2024-02-13T00:43:03.362622+0000","last_undegraded":"2024-02-13T00:43:03.362622+0000","last_fullsized":"2024-02-13T00:43:03.362622+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:41:42.695521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:42.695521+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":73,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346892+0000","last_change":"2024-02-13T00:42:03.411393+0000","last_active":"2024-02-13T00:43:03.346892+0000","last_peered":"2024-02-13T00:43:03.346892+0000","last_clean":"2024-02-13T00:43:03.346892+0000","last_became_active":"2024-02-13T00:42:03.402924+0000","last_became_peered":"2024-02-13T00:42:03.402924+0000","last_unstale":"2024-02-13T00:43:03.346892+0000","last_undegraded":"2024-02-13T00:43:03.346892+0000","last_fullsized":"2024-02-13T00:43:03.346892+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:54.316591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:40:54.316591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":186,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346658+0000","last_change":"2024-02-13T00:40:53.536240+0000","last_active":"2024-02-13T00:43:03.346658+0000","last_peered":"2024-02-13T00:43:03.346658+0000","last_clean":"2024-02-13T00:43:03.346658+0000","last_became_active":"2024-02-13T00:37:45.246907+0000","last_became_peered":"2024-02-13T00:37:45.246907+0000","last_unstale":"2024-02-13T00:43:03.346658+0000","last_undegraded":"2024-02-13T00:43:03.346658+0000","last_fullsized":"2024-02-13T00:43:03.346658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:53.536170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:53.536170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":146,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.342428+0000","last_change":"2024-02-13T00:42:53.343902+0000","last_active":"2024-02-13T00:43:03.342428+0000","last_peered":"2024-02-13T00:43:03.342428+0000","last_clean":"2024-02-13T00:43:03.342428+0000","last_became_active":"2024-02-13T00:42:53.343569+0000","last_became_peered":"2024-02-13T00:42:53.343569+0000","last_unstale":"2024-02-13T00:43:03.342428+0000","last_undegraded":"2024-02-13T00:43:03.342428+0000","last_fullsized":"2024-02-13T00:43:03.342428+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.115856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.115856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1","version":"108'5","reported_seq":150,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.357132+0000","last_change":"2024-02-13T00:42:03.968757+0000","last_active":"2024-02-13T00:43:03.357132+0000","last_peered":"2024-02-13T00:43:03.357132+0000","last_clean":"2024-02-13T00:43:03.357132+0000","last_became_active":"2024-02-13T00:42:03.396535+0000","last_became_peered":"2024-02-13T00:42:03.396535+0000","last_unstale":"2024-02-13T00:43:03.357132+0000","last_undegraded":"2024-02-13T00:43:03.357132+0000","last_fullsized":"2024-02-13T00:43:03.357132+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":131,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.368852+0000","last_change":"2024-02-13T00:41:01.653337+0000","last_active":"2024-02-13T00:43:03.368852+0000","last_peered":"2024-02-13T00:43:03.368852+0000","last_clean":"2024-02-13T00:43:03.368852+0000","last_became_active":"2024-02-13T00:41:01.645866+0000","last_became_peered":"2024-02-13T00:41:01.645866+0000","last_unstale":"2024-02-13T00:43:03.368852+0000","last_undegraded":"2024-02-13T00:43:03.368852+0000","last_fullsized":"2024-02-13T00:43:03.368852+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":245,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.368776+0000","last_change":"2024-02-13T00:42:48.774248+0000","last_active":"2024-02-13T00:43:03.368776+0000","last_peered":"2024-02-13T00:43:03.368776+0000","last_clean":"2024-02-13T00:43:03.368776+0000","last_became_active":"2024-02-13T00:37:43.324313+0000","last_became_peered":"2024-02-13T00:37:43.324313+0000","last_unstale":"2024-02-13T00:43:03.368776+0000","last_undegraded":"2024-02-13T00:43:03.368776+0000","last_fullsized":"2024-02-13T00:43:03.368776+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:42:48.774168+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.774168+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":190,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.339882+0000","last_change":"2024-02-13T00:40:52.503871+0000","last_active":"2024-02-13T00:43:03.339882+0000","last_peered":"2024-02-13T00:43:03.339882+0000","last_clean":"2024-02-13T00:43:03.339882+0000","last_became_active":"2024-02-13T00:37:41.219309+0000","last_became_peered":"2024-02-13T00:37:41.219309+0000","last_unstale":"2024-02-13T00:43:03.339882+0000","last_undegraded":"2024-02-13T00:43:03.339882+0000","last_fullsized":"2024-02-13T00:43:03.339882+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:52.503804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:52.503804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"64'8","reported_seq":259,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.279594+0000","last_change":"2024-02-13T00:41:37.453007+0000","last_active":"2024-02-13T00:43:04.279594+0000","last_peered":"2024-02-13T00:43:04.279594+0000","last_clean":"2024-02-13T00:43:04.279594+0000","last_became_active":"2024-02-13T00:37:45.246757+0000","last_became_peered":"2024-02-13T00:37:45.246757+0000","last_unstale":"2024-02-13T00:43:04.279594+0000","last_undegraded":"2024-02-13T00:43:04.279594+0000","last_fullsized":"2024-02-13T00:43:04.279594+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:37.452882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:37.452882+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":184,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.355429+0000","last_change":"2024-02-13T00:42:52.391979+0000","last_active":"2024-02-13T00:43:03.355429+0000","last_peered":"2024-02-13T00:43:03.355429+0000","last_clean":"2024-02-13T00:43:03.355429+0000","last_became_active":"2024-02-13T00:37:47.252722+0000","last_became_peered":"2024-02-13T00:37:47.252722+0000","last_unstale":"2024-02-13T00:43:03.355429+0000","last_undegraded":"2024-02-13T00:43:03.355429+0000","last_fullsized":"2024-02-13T00:43:03.355429+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:52.391906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.391906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.353837+0000","last_change":"2024-02-13T00:42:53.367472+0000","last_active":"2024-02-13T00:43:03.353837+0000","last_peered":"2024-02-13T00:43:03.353837+0000","last_clean":"2024-02-13T00:43:03.353837+0000","last_became_active":"2024-02-13T00:42:53.367169+0000","last_became_peered":"2024-02-13T00:42:53.367169+0000","last_unstale":"2024-02-13T00:43:03.353837+0000","last_undegraded":"2024-02-13T00:43:03.353837+0000","last_fullsized":"2024-02-13T00:43:03.353837+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":23,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.360499+0000","last_change":"2024-02-13T00:42:53.371831+0000","last_active":"2024-02-13T00:43:03.360499+0000","last_peered":"2024-02-13T00:43:03.360499+0000","last_clean":"2024-02-13T00:43:03.360499+0000","last_became_active":"2024-02-13T00:42:53.371213+0000","last_became_peered":"2024-02-13T00:42:53.371213+0000","last_unstale":"2024-02-13T00:43:03.360499+0000","last_undegraded":"2024-02-13T00:43:03.360499+0000","last_fullsized":"2024-02-13T00:43:03.360499+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":220,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.342616+0000","last_change":"2024-02-13T00:41:36.477098+0000","last_active":"2024-02-13T00:43:03.342616+0000","last_peered":"2024-02-13T00:43:03.342616+0000","last_clean":"2024-02-13T00:43:03.342616+0000","last_became_active":"2024-02-13T00:37:43.322357+0000","last_became_peered":"2024-02-13T00:37:43.322357+0000","last_unstale":"2024-02-13T00:43:03.342616+0000","last_undegraded":"2024-02-13T00:43:03.342616+0000","last_fullsized":"2024-02-13T00:43:03.342616+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:36.477034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.477034+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":190,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.360308+0000","last_change":"2024-02-13T00:40:52.801522+0000","last_active":"2024-02-13T00:43:03.360308+0000","last_peered":"2024-02-13T00:43:03.360308+0000","last_clean":"2024-02-13T00:43:03.360308+0000","last_became_active":"2024-02-13T00:37:41.217953+0000","last_became_peered":"2024-02-13T00:37:41.217953+0000","last_unstale":"2024-02-13T00:43:03.360308+0000","last_undegraded":"2024-02-13T00:43:03.360308+0000","last_fullsized":"2024-02-13T00:43:03.360308+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:52.801455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:52.801455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"84'9","reported_seq":263,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.279631+0000","last_change":"2024-02-13T00:41:32.715662+0000","last_active":"2024-02-13T00:43:04.279631+0000","last_peered":"2024-02-13T00:43:04.279631+0000","last_clean":"2024-02-13T00:43:04.279631+0000","last_became_active":"2024-02-13T00:37:45.250598+0000","last_became_peered":"2024-02-13T00:37:45.250598+0000","last_unstale":"2024-02-13T00:43:04.279631+0000","last_undegraded":"2024-02-13T00:43:04.279631+0000","last_fullsized":"2024-02-13T00:43:04.279631+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-02-13T00:41:32.715550+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.715550+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":155,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.353032+0000","last_change":"2024-02-13T00:41:55.421064+0000","last_active":"2024-02-13T00:43:03.353032+0000","last_peered":"2024-02-13T00:43:03.353032+0000","last_clean":"2024-02-13T00:43:03.353032+0000","last_became_active":"2024-02-13T00:40:35.850176+0000","last_became_peered":"2024-02-13T00:40:35.850176+0000","last_unstale":"2024-02-13T00:43:03.353032+0000","last_undegraded":"2024-02-13T00:43:03.353032+0000","last_fullsized":"2024-02-13T00:43:03.353032+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.421001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.421001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.3","version":"108'5","reported_seq":150,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.343883+0000","last_change":"2024-02-13T00:42:05.399708+0000","last_active":"2024-02-13T00:43:03.343883+0000","last_peered":"2024-02-13T00:43:03.343883+0000","last_clean":"2024-02-13T00:43:03.343883+0000","last_became_active":"2024-02-13T00:42:03.420690+0000","last_became_peered":"2024-02-13T00:42:03.420690+0000","last_unstale":"2024-02-13T00:43:03.343883+0000","last_undegraded":"2024-02-13T00:43:03.343883+0000","last_fullsized":"2024-02-13T00:43:03.343883+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":140,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.340696+0000","last_change":"2024-02-13T00:42:03.415812+0000","last_active":"2024-02-13T00:43:03.340696+0000","last_peered":"2024-02-13T00:43:03.340696+0000","last_clean":"2024-02-13T00:43:03.340696+0000","last_became_active":"2024-02-13T00:42:03.415302+0000","last_became_peered":"2024-02-13T00:42:03.415302+0000","last_unstale":"2024-02-13T00:43:03.340696+0000","last_undegraded":"2024-02-13T00:43:03.340696+0000","last_fullsized":"2024-02-13T00:43:03.340696+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":228,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.348936+0000","last_change":"2024-02-13T00:42:07.212237+0000","last_active":"2024-02-13T00:43:03.348936+0000","last_peered":"2024-02-13T00:43:03.348936+0000","last_clean":"2024-02-13T00:43:03.348936+0000","last_became_active":"2024-02-13T00:42:03.424278+0000","last_became_peered":"2024-02-13T00:42:03.424278+0000","last_unstale":"2024-02-13T00:43:03.348936+0000","last_undegraded":"2024-02-13T00:43:03.348936+0000","last_fullsized":"2024-02-13T00:43:03.348936+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:40:16.009518+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:40:16.009518+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":73,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.368929+0000","last_change":"2024-02-13T00:42:03.432596+0000","last_active":"2024-02-13T00:43:03.368929+0000","last_peered":"2024-02-13T00:43:03.368929+0000","last_clean":"2024-02-13T00:43:03.368929+0000","last_became_active":"2024-02-13T00:42:03.431748+0000","last_became_peered":"2024-02-13T00:42:03.431748+0000","last_unstale":"2024-02-13T00:43:03.368929+0000","last_undegraded":"2024-02-13T00:43:03.368929+0000","last_fullsized":"2024-02-13T00:43:03.368929+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:58.334705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.334705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":186,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.358674+0000","last_change":"2024-02-13T00:41:43.717024+0000","last_active":"2024-02-13T00:43:03.358674+0000","last_peered":"2024-02-13T00:43:03.358674+0000","last_clean":"2024-02-13T00:43:03.358674+0000","last_became_active":"2024-02-13T00:37:45.243594+0000","last_became_peered":"2024-02-13T00:37:45.243594+0000","last_unstale":"2024-02-13T00:43:03.358674+0000","last_undegraded":"2024-02-13T00:43:03.358674+0000","last_fullsized":"2024-02-13T00:43:03.358674+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:43.716891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_clean_scrub_stamp":"2024-02-13T00:41:43.716891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":184,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.349488+0000","last_change":"2024-02-13T00:42:56.413422+0000","last_active":"2024-02-13T00:43:03.349488+0000","last_peered":"2024-02-13T00:43:03.349488+0000","last_clean":"2024-02-13T00:43:03.349488+0000","last_became_active":"2024-02-13T00:37:47.259792+0000","last_became_peered":"2024-02-13T00:37:47.259792+0000","last_unstale":"2024-02-13T00:43:03.349488+0000","last_undegraded":"2024-02-13T00:43:03.349488+0000","last_fullsized":"2024-02-13T00:43:03.349488+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.413326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":132,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.358134+0000","last_change":"2024-02-13T00:41:01.745030+0000","last_active":"2024-02-13T00:43:03.358134+0000","last_peered":"2024-02-13T00:43:03.358134+0000","last_clean":"2024-02-13T00:43:03.358134+0000","last_became_active":"2024-02-13T00:41:01.743858+0000","last_became_peered":"2024-02-13T00:41:01.743858+0000","last_unstale":"2024-02-13T00:43:03.358134+0000","last_undegraded":"2024-02-13T00:43:03.358134+0000","last_fullsized":"2024-02-13T00:43:03.358134+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":26,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.360673+0000","last_change":"2024-02-13T00:42:53.376007+0000","last_active":"2024-02-13T00:43:03.360673+0000","last_peered":"2024-02-13T00:43:03.360673+0000","last_clean":"2024-02-13T00:43:03.360673+0000","last_became_active":"2024-02-13T00:42:53.375683+0000","last_became_peered":"2024-02-13T00:42:53.375683+0000","last_unstale":"2024-02-13T00:43:03.360673+0000","last_undegraded":"2024-02-13T00:43:03.360673+0000","last_fullsized":"2024-02-13T00:43:03.360673+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":150,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347211+0000","last_change":"2024-02-13T00:42:51.361592+0000","last_active":"2024-02-13T00:43:03.347211+0000","last_peered":"2024-02-13T00:43:03.347211+0000","last_clean":"2024-02-13T00:43:03.347211+0000","last_became_active":"2024-02-13T00:42:03.384628+0000","last_became_peered":"2024-02-13T00:42:03.384628+0000","last_unstale":"2024-02-13T00:43:03.347211+0000","last_undegraded":"2024-02-13T00:43:03.347211+0000","last_fullsized":"2024-02-13T00:43:03.347211+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:42:51.361489+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:42:51.361489+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":191,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.349841+0000","last_change":"2024-02-13T00:41:03.608458+0000","last_active":"2024-02-13T00:43:03.349841+0000","last_peered":"2024-02-13T00:43:03.349841+0000","last_clean":"2024-02-13T00:43:03.349841+0000","last_became_active":"2024-02-13T00:37:41.215531+0000","last_became_peered":"2024-02-13T00:37:41.215531+0000","last_unstale":"2024-02-13T00:43:03.349841+0000","last_undegraded":"2024-02-13T00:43:03.349841+0000","last_fullsized":"2024-02-13T00:43:03.349841+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:03.608379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:03.608379+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":187,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.349013+0000","last_change":"2024-02-13T00:40:49.666455+0000","last_active":"2024-02-13T00:43:03.349013+0000","last_peered":"2024-02-13T00:43:03.349013+0000","last_clean":"2024-02-13T00:43:03.349013+0000","last_became_active":"2024-02-13T00:37:45.247114+0000","last_became_peered":"2024-02-13T00:37:45.247114+0000","last_unstale":"2024-02-13T00:43:03.349013+0000","last_undegraded":"2024-02-13T00:43:03.349013+0000","last_fullsized":"2024-02-13T00:43:03.349013+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:49.666387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:40:49.666387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":189,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.356362+0000","last_change":"2024-02-13T00:42:50.828326+0000","last_active":"2024-02-13T00:43:03.356362+0000","last_peered":"2024-02-13T00:43:03.356362+0000","last_clean":"2024-02-13T00:43:03.356362+0000","last_became_active":"2024-02-13T00:37:47.254519+0000","last_became_peered":"2024-02-13T00:37:47.254519+0000","last_unstale":"2024-02-13T00:43:03.356362+0000","last_undegraded":"2024-02-13T00:43:03.356362+0000","last_fullsized":"2024-02-13T00:43:03.356362+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:50.828264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:50.828264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":141,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.349916+0000","last_change":"2024-02-13T00:42:03.414688+0000","last_active":"2024-02-13T00:43:03.349916+0000","last_peered":"2024-02-13T00:43:03.349916+0000","last_clean":"2024-02-13T00:43:03.349916+0000","last_became_active":"2024-02-13T00:42:03.413667+0000","last_became_peered":"2024-02-13T00:42:03.413667+0000","last_unstale":"2024-02-13T00:43:03.349916+0000","last_undegraded":"2024-02-13T00:43:03.349916+0000","last_fullsized":"2024-02-13T00:43:03.349916+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":73,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.368867+0000","last_change":"2024-02-13T00:42:03.435906+0000","last_active":"2024-02-13T00:43:03.368867+0000","last_peered":"2024-02-13T00:43:03.368867+0000","last_clean":"2024-02-13T00:43:03.368867+0000","last_became_active":"2024-02-13T00:42:03.433274+0000","last_became_peered":"2024-02-13T00:42:03.433274+0000","last_unstale":"2024-02-13T00:43:03.368867+0000","last_undegraded":"2024-02-13T00:43:03.368867+0000","last_fullsized":"2024-02-13T00:43:03.368867+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":209,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.368786+0000","last_change":"2024-02-13T00:42:06.873341+0000","last_active":"2024-02-13T00:43:03.368786+0000","last_peered":"2024-02-13T00:43:03.368786+0000","last_clean":"2024-02-13T00:43:03.368786+0000","last_became_active":"2024-02-13T00:42:03.420647+0000","last_became_peered":"2024-02-13T00:42:03.420647+0000","last_unstale":"2024-02-13T00:43:03.368786+0000","last_undegraded":"2024-02-13T00:43:03.368786+0000","last_fullsized":"2024-02-13T00:43:03.368786+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:40:51.054728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.054728+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":204,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346775+0000","last_change":"2024-02-13T00:42:56.499654+0000","last_active":"2024-02-13T00:43:03.346775+0000","last_peered":"2024-02-13T00:43:03.346775+0000","last_clean":"2024-02-13T00:43:03.346775+0000","last_became_active":"2024-02-13T00:42:03.399329+0000","last_became_peered":"2024-02-13T00:42:03.399329+0000","last_unstale":"2024-02-13T00:43:03.346775+0000","last_undegraded":"2024-02-13T00:43:03.346775+0000","last_fullsized":"2024-02-13T00:43:03.346775+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.499576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":26,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350993+0000","last_change":"2024-02-13T00:42:53.351539+0000","last_active":"2024-02-13T00:43:03.350993+0000","last_peered":"2024-02-13T00:43:03.350993+0000","last_clean":"2024-02-13T00:43:03.350993+0000","last_became_active":"2024-02-13T00:42:53.351244+0000","last_became_peered":"2024-02-13T00:42:53.351244+0000","last_unstale":"2024-02-13T00:43:03.350993+0000","last_undegraded":"2024-02-13T00:43:03.350993+0000","last_fullsized":"2024-02-13T00:43:03.350993+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.646321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.646321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":28,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346171+0000","last_change":"2024-02-13T00:42:56.479516+0000","last_active":"2024-02-13T00:43:03.346171+0000","last_peered":"2024-02-13T00:43:03.346171+0000","last_clean":"2024-02-13T00:43:03.346171+0000","last_became_active":"2024-02-13T00:42:53.368896+0000","last_became_peered":"2024-02-13T00:42:53.368896+0000","last_unstale":"2024-02-13T00:43:03.346171+0000","last_undegraded":"2024-02-13T00:43:03.346171+0000","last_fullsized":"2024-02-13T00:43:03.346171+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.479450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"108'9","reported_seq":41,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.353228+0000","last_change":"2024-02-13T00:42:53.370757+0000","last_active":"2024-02-13T00:43:03.353228+0000","last_peered":"2024-02-13T00:43:03.353228+0000","last_clean":"2024-02-13T00:43:03.353228+0000","last_became_active":"2024-02-13T00:42:53.370432+0000","last_became_peered":"2024-02-13T00:42:53.370432+0000","last_unstale":"2024-02-13T00:43:03.353228+0000","last_undegraded":"2024-02-13T00:43:03.353228+0000","last_fullsized":"2024-02-13T00:43:03.353228+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":140,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.341449+0000","last_change":"2024-02-13T00:42:03.402279+0000","last_active":"2024-02-13T00:43:03.341449+0000","last_peered":"2024-02-13T00:43:03.341449+0000","last_clean":"2024-02-13T00:43:03.341449+0000","last_became_active":"2024-02-13T00:42:03.401648+0000","last_became_peered":"2024-02-13T00:42:03.401648+0000","last_unstale":"2024-02-13T00:43:03.341449+0000","last_undegraded":"2024-02-13T00:43:03.341449+0000","last_fullsized":"2024-02-13T00:43:03.341449+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":245,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.356966+0000","last_change":"2024-02-13T00:42:08.062022+0000","last_active":"2024-02-13T00:43:03.356966+0000","last_peered":"2024-02-13T00:43:03.356966+0000","last_clean":"2024-02-13T00:43:03.356966+0000","last_became_active":"2024-02-13T00:42:03.432551+0000","last_became_peered":"2024-02-13T00:42:03.432551+0000","last_unstale":"2024-02-13T00:43:03.356966+0000","last_undegraded":"2024-02-13T00:43:03.356966+0000","last_fullsized":"2024-02-13T00:43:03.356966+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:41:50.675154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:50.675154+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":73,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.345356+0000","last_change":"2024-02-13T00:42:03.397136+0000","last_active":"2024-02-13T00:43:03.345356+0000","last_peered":"2024-02-13T00:43:03.345356+0000","last_clean":"2024-02-13T00:43:03.345356+0000","last_became_active":"2024-02-13T00:42:03.396688+0000","last_became_peered":"2024-02-13T00:42:03.396688+0000","last_unstale":"2024-02-13T00:43:03.345356+0000","last_undegraded":"2024-02-13T00:43:03.345356+0000","last_fullsized":"2024-02-13T00:43:03.345356+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.457672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.457672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":148,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.349023+0000","last_change":"2024-02-13T00:42:53.361607+0000","last_active":"2024-02-13T00:43:03.349023+0000","last_peered":"2024-02-13T00:43:03.349023+0000","last_clean":"2024-02-13T00:43:03.349023+0000","last_became_active":"2024-02-13T00:42:53.361106+0000","last_became_peered":"2024-02-13T00:42:53.361106+0000","last_unstale":"2024-02-13T00:43:03.349023+0000","last_undegraded":"2024-02-13T00:43:03.349023+0000","last_fullsized":"2024-02-13T00:43:03.349023+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.164690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.164690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":193,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.358837+0000","last_change":"2024-02-13T00:42:03.408944+0000","last_active":"2024-02-13T00:43:03.358837+0000","last_peered":"2024-02-13T00:43:03.358837+0000","last_clean":"2024-02-13T00:43:03.358837+0000","last_became_active":"2024-02-13T00:42:03.408629+0000","last_became_peered":"2024-02-13T00:42:03.408629+0000","last_unstale":"2024-02-13T00:43:03.358837+0000","last_undegraded":"2024-02-13T00:43:03.358837+0000","last_fullsized":"2024-02-13T00:43:03.358837+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.881255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.881255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":132,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.353169+0000","last_change":"2024-02-13T00:41:01.595241+0000","last_active":"2024-02-13T00:43:03.353169+0000","last_peered":"2024-02-13T00:43:03.353169+0000","last_clean":"2024-02-13T00:43:03.353169+0000","last_became_active":"2024-02-13T00:41:01.581338+0000","last_became_peered":"2024-02-13T00:41:01.581338+0000","last_unstale":"2024-02-13T00:43:03.353169+0000","last_undegraded":"2024-02-13T00:43:03.353169+0000","last_fullsized":"2024-02-13T00:43:03.353169+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":23,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.355479+0000","last_change":"2024-02-13T00:42:53.374915+0000","last_active":"2024-02-13T00:43:03.355479+0000","last_peered":"2024-02-13T00:43:03.355479+0000","last_clean":"2024-02-13T00:43:03.355479+0000","last_became_active":"2024-02-13T00:42:53.374600+0000","last_became_peered":"2024-02-13T00:42:53.374600+0000","last_unstale":"2024-02-13T00:43:03.355479+0000","last_undegraded":"2024-02-13T00:43:03.355479+0000","last_fullsized":"2024-02-13T00:43:03.355479+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":226,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.412757+0000","last_change":"2024-02-13T00:43:04.416945+0000","last_active":"2024-02-13T00:43:04.412757+0000","last_peered":"2024-02-13T00:43:04.412757+0000","last_clean":"2024-02-13T00:43:04.412757+0000","last_became_active":"2024-02-13T00:37:43.324221+0000","last_became_peered":"2024-02-13T00:37:43.324221+0000","last_unstale":"2024-02-13T00:43:04.412757+0000","last_undegraded":"2024-02-13T00:43:04.412757+0000","last_fullsized":"2024-02-13T00:43:04.412757+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:40:06.169087+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:40:06.169087+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":190,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.340330+0000","last_change":"2024-02-13T00:41:32.424906+0000","last_active":"2024-02-13T00:43:03.340330+0000","last_peered":"2024-02-13T00:43:03.340330+0000","last_clean":"2024-02-13T00:43:03.340330+0000","last_became_active":"2024-02-13T00:37:41.214965+0000","last_became_peered":"2024-02-13T00:37:41.214965+0000","last_unstale":"2024-02-13T00:43:03.340330+0000","last_undegraded":"2024-02-13T00:43:03.340330+0000","last_fullsized":"2024-02-13T00:43:03.340330+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:32.424844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.424844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"218'10","reported_seq":51,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:04.279676+0000","last_change":"2024-02-13T00:42:53.368374+0000","last_active":"2024-02-13T00:43:04.279676+0000","last_peered":"2024-02-13T00:43:04.279676+0000","last_clean":"2024-02-13T00:43:04.279676+0000","last_became_active":"2024-02-13T00:42:53.368046+0000","last_became_peered":"2024-02-13T00:42:53.368046+0000","last_unstale":"2024-02-13T00:43:04.279676+0000","last_undegraded":"2024-02-13T00:43:04.279676+0000","last_fullsized":"2024-02-13T00:43:04.279676+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:02.323101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.323101+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":37,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.341202+0000","last_change":"2024-02-13T00:42:55.481193+0000","last_active":"2024-02-13T00:43:03.341202+0000","last_peered":"2024-02-13T00:43:03.341202+0000","last_clean":"2024-02-13T00:43:03.341202+0000","last_became_active":"2024-02-13T00:42:53.368344+0000","last_became_peered":"2024-02-13T00:42:53.368344+0000","last_unstale":"2024-02-13T00:43:03.341202+0000","last_undegraded":"2024-02-13T00:43:03.341202+0000","last_fullsized":"2024-02-13T00:43:03.341202+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:42:55.481128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:55.481128+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.8","version":"107'2","reported_seq":147,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350813+0000","last_change":"2024-02-13T00:42:04.282151+0000","last_active":"2024-02-13T00:43:03.350813+0000","last_peered":"2024-02-13T00:43:03.350813+0000","last_clean":"2024-02-13T00:43:03.350813+0000","last_became_active":"2024-02-13T00:42:03.410938+0000","last_became_peered":"2024-02-13T00:42:03.410938+0000","last_unstale":"2024-02-13T00:43:03.350813+0000","last_undegraded":"2024-02-13T00:43:03.350813+0000","last_fullsized":"2024-02-13T00:43:03.350813+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":140,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.343865+0000","last_change":"2024-02-13T00:42:03.427616+0000","last_active":"2024-02-13T00:43:03.343865+0000","last_peered":"2024-02-13T00:43:03.343865+0000","last_clean":"2024-02-13T00:43:03.343865+0000","last_became_active":"2024-02-13T00:42:03.427079+0000","last_became_peered":"2024-02-13T00:42:03.427079+0000","last_unstale":"2024-02-13T00:43:03.343865+0000","last_undegraded":"2024-02-13T00:43:03.343865+0000","last_fullsized":"2024-02-13T00:43:03.343865+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":139,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.345553+0000","last_change":"2024-02-13T00:42:59.604074+0000","last_active":"2024-02-13T00:43:03.345553+0000","last_peered":"2024-02-13T00:43:03.345553+0000","last_clean":"2024-02-13T00:43:03.345553+0000","last_became_active":"2024-02-13T00:42:03.397231+0000","last_became_peered":"2024-02-13T00:42:03.397231+0000","last_unstale":"2024-02-13T00:43:03.345553+0000","last_undegraded":"2024-02-13T00:43:03.345553+0000","last_fullsized":"2024-02-13T00:43:03.345553+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:42:59.604002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:59.604002+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":199,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.356501+0000","last_change":"2024-02-13T00:42:03.408124+0000","last_active":"2024-02-13T00:43:03.356501+0000","last_peered":"2024-02-13T00:43:03.356501+0000","last_clean":"2024-02-13T00:43:03.356501+0000","last_became_active":"2024-02-13T00:42:03.403904+0000","last_became_peered":"2024-02-13T00:42:03.403904+0000","last_unstale":"2024-02-13T00:43:03.356501+0000","last_undegraded":"2024-02-13T00:43:03.356501+0000","last_fullsized":"2024-02-13T00:43:03.356501+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:46.698599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:46.698599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":188,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.355973+0000","last_change":"2024-02-13T00:42:27.596444+0000","last_active":"2024-02-13T00:43:03.355973+0000","last_peered":"2024-02-13T00:43:03.355973+0000","last_clean":"2024-02-13T00:43:03.355973+0000","last_became_active":"2024-02-13T00:37:45.249064+0000","last_became_peered":"2024-02-13T00:37:45.249064+0000","last_unstale":"2024-02-13T00:43:03.355973+0000","last_undegraded":"2024-02-13T00:43:03.355973+0000","last_fullsized":"2024-02-13T00:43:03.355973+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.596334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.596334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":185,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.349705+0000","last_change":"2024-02-13T00:40:06.026382+0000","last_active":"2024-02-13T00:43:03.349705+0000","last_peered":"2024-02-13T00:43:03.349705+0000","last_clean":"2024-02-13T00:43:03.349705+0000","last_became_active":"2024-02-13T00:37:47.260182+0000","last_became_peered":"2024-02-13T00:37:47.260182+0000","last_unstale":"2024-02-13T00:43:03.349705+0000","last_undegraded":"2024-02-13T00:43:03.349705+0000","last_fullsized":"2024-02-13T00:43:03.349705+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:06.026318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:06.026318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":132,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.356577+0000","last_change":"2024-02-13T00:41:01.682551+0000","last_active":"2024-02-13T00:43:03.356577+0000","last_peered":"2024-02-13T00:43:03.356577+0000","last_clean":"2024-02-13T00:43:03.356577+0000","last_became_active":"2024-02-13T00:41:01.682261+0000","last_became_peered":"2024-02-13T00:41:01.682261+0000","last_unstale":"2024-02-13T00:43:03.356577+0000","last_undegraded":"2024-02-13T00:43:03.356577+0000","last_fullsized":"2024-02-13T00:43:03.356577+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":73,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.345653+0000","last_change":"2024-02-13T00:42:03.406045+0000","last_active":"2024-02-13T00:43:03.345653+0000","last_peered":"2024-02-13T00:43:03.345653+0000","last_clean":"2024-02-13T00:43:03.345653+0000","last_became_active":"2024-02-13T00:42:03.405677+0000","last_became_peered":"2024-02-13T00:42:03.405677+0000","last_unstale":"2024-02-13T00:43:03.345653+0000","last_undegraded":"2024-02-13T00:43:03.345653+0000","last_fullsized":"2024-02-13T00:43:03.345653+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":214,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350322+0000","last_change":"2024-02-13T00:40:44.369174+0000","last_active":"2024-02-13T00:43:03.350322+0000","last_peered":"2024-02-13T00:43:03.350322+0000","last_clean":"2024-02-13T00:43:03.350322+0000","last_became_active":"2024-02-13T00:37:43.232938+0000","last_became_peered":"2024-02-13T00:37:43.232938+0000","last_unstale":"2024-02-13T00:43:03.350322+0000","last_undegraded":"2024-02-13T00:43:03.350322+0000","last_fullsized":"2024-02-13T00:43:03.350322+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:40:44.369070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:40:44.369070+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":26,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346443+0000","last_change":"2024-02-13T00:42:53.369684+0000","last_active":"2024-02-13T00:43:03.346443+0000","last_peered":"2024-02-13T00:43:03.346443+0000","last_clean":"2024-02-13T00:43:03.346443+0000","last_became_active":"2024-02-13T00:42:53.369255+0000","last_became_peered":"2024-02-13T00:42:53.369255+0000","last_unstale":"2024-02-13T00:43:03.346443+0000","last_undegraded":"2024-02-13T00:43:03.346443+0000","last_fullsized":"2024-02-13T00:43:03.346443+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.316400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.316400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":187,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350347+0000","last_change":"2024-02-13T00:41:56.204734+0000","last_active":"2024-02-13T00:43:03.350347+0000","last_peered":"2024-02-13T00:43:03.350347+0000","last_clean":"2024-02-13T00:43:03.350347+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:43:03.350347+0000","last_undegraded":"2024-02-13T00:43:03.350347+0000","last_fullsized":"2024-02-13T00:43:03.350347+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:56.204671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.204671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":81,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.346830+0000","last_change":"2024-02-13T00:42:27.410841+0000","last_active":"2024-02-13T00:43:03.346830+0000","last_peered":"2024-02-13T00:43:03.346830+0000","last_clean":"2024-02-13T00:43:03.346830+0000","last_became_active":"2024-02-13T00:42:03.416740+0000","last_became_peered":"2024-02-13T00:42:03.416740+0000","last_unstale":"2024-02-13T00:43:03.346830+0000","last_undegraded":"2024-02-13T00:43:03.346830+0000","last_fullsized":"2024-02-13T00:43:03.346830+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.410777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.410777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.a","version":"108'10","reported_seq":157,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.369987+0000","last_change":"2024-02-13T00:42:06.672391+0000","last_active":"2024-02-13T00:43:03.369987+0000","last_peered":"2024-02-13T00:43:03.369987+0000","last_clean":"2024-02-13T00:43:03.369987+0000","last_became_active":"2024-02-13T00:42:03.419452+0000","last_became_peered":"2024-02-13T00:42:03.419452+0000","last_unstale":"2024-02-13T00:43:03.369987+0000","last_undegraded":"2024-02-13T00:43:03.369987+0000","last_fullsized":"2024-02-13T00:43:03.369987+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":93,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.352410+0000","last_change":"2024-02-13T00:42:53.374893+0000","last_active":"2024-02-13T00:43:03.352410+0000","last_peered":"2024-02-13T00:43:03.352410+0000","last_clean":"2024-02-13T00:43:03.352410+0000","last_became_active":"2024-02-13T00:42:53.374567+0000","last_became_peered":"2024-02-13T00:42:53.374567+0000","last_unstale":"2024-02-13T00:43:03.352410+0000","last_undegraded":"2024-02-13T00:43:03.352410+0000","last_fullsized":"2024-02-13T00:43:03.352410+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":228,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.360266+0000","last_change":"2024-02-13T00:41:41.686152+0000","last_active":"2024-02-13T00:43:03.360266+0000","last_peered":"2024-02-13T00:43:03.360266+0000","last_clean":"2024-02-13T00:43:03.360266+0000","last_became_active":"2024-02-13T00:37:43.236026+0000","last_became_peered":"2024-02-13T00:37:43.236026+0000","last_unstale":"2024-02-13T00:43:03.360266+0000","last_undegraded":"2024-02-13T00:43:03.360266+0000","last_fullsized":"2024-02-13T00:43:03.360266+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:41:41.686058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.686058+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":199,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.358601+0000","last_change":"2024-02-13T00:42:03.429962+0000","last_active":"2024-02-13T00:43:03.358601+0000","last_peered":"2024-02-13T00:43:03.358601+0000","last_clean":"2024-02-13T00:43:03.358601+0000","last_became_active":"2024-02-13T00:42:03.429539+0000","last_became_peered":"2024-02-13T00:42:03.429539+0000","last_unstale":"2024-02-13T00:43:03.358601+0000","last_undegraded":"2024-02-13T00:43:03.358601+0000","last_fullsized":"2024-02-13T00:43:03.358601+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:49.110725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:40:49.110725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":155,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.369003+0000","last_change":"2024-02-13T00:42:57.802494+0000","last_active":"2024-02-13T00:43:03.369003+0000","last_peered":"2024-02-13T00:43:03.369003+0000","last_clean":"2024-02-13T00:43:03.369003+0000","last_became_active":"2024-02-13T00:40:35.858538+0000","last_became_peered":"2024-02-13T00:40:35.858538+0000","last_unstale":"2024-02-13T00:43:03.369003+0000","last_undegraded":"2024-02-13T00:43:03.369003+0000","last_fullsized":"2024-02-13T00:43:03.369003+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:57.802417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.802417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":193,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.362539+0000","last_change":"2024-02-13T00:42:03.427267+0000","last_active":"2024-02-13T00:43:03.362539+0000","last_peered":"2024-02-13T00:43:03.362539+0000","last_clean":"2024-02-13T00:43:03.362539+0000","last_became_active":"2024-02-13T00:42:03.426960+0000","last_became_peered":"2024-02-13T00:42:03.426960+0000","last_unstale":"2024-02-13T00:43:03.362539+0000","last_undegraded":"2024-02-13T00:43:03.362539+0000","last_fullsized":"2024-02-13T00:43:03.362539+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.950483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.950483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"107'2","reported_seq":134,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.340185+0000","last_change":"2024-02-13T00:41:01.611411+0000","last_active":"2024-02-13T00:43:03.340185+0000","last_peered":"2024-02-13T00:43:03.340185+0000","last_clean":"2024-02-13T00:43:03.340185+0000","last_became_active":"2024-02-13T00:41:01.611115+0000","last_became_peered":"2024-02-13T00:41:01.611115+0000","last_unstale":"2024-02-13T00:43:03.340185+0000","last_undegraded":"2024-02-13T00:43:03.340185+0000","last_fullsized":"2024-02-13T00:43:03.340185+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":140,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.353585+0000","last_change":"2024-02-13T00:42:03.382598+0000","last_active":"2024-02-13T00:43:03.353585+0000","last_peered":"2024-02-13T00:43:03.353585+0000","last_clean":"2024-02-13T00:43:03.353585+0000","last_became_active":"2024-02-13T00:42:03.382303+0000","last_became_peered":"2024-02-13T00:42:03.382303+0000","last_unstale":"2024-02-13T00:43:03.353585+0000","last_undegraded":"2024-02-13T00:43:03.353585+0000","last_fullsized":"2024-02-13T00:43:03.353585+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":209,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.369127+0000","last_change":"2024-02-13T00:41:31.648350+0000","last_active":"2024-02-13T00:43:03.369127+0000","last_peered":"2024-02-13T00:43:03.369127+0000","last_clean":"2024-02-13T00:43:03.369127+0000","last_became_active":"2024-02-13T00:40:35.857274+0000","last_became_peered":"2024-02-13T00:40:35.857274+0000","last_unstale":"2024-02-13T00:43:03.369127+0000","last_undegraded":"2024-02-13T00:43:03.369127+0000","last_fullsized":"2024-02-13T00:43:03.369127+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:41:31.648217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:31.648217+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":28,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.666060+0000","last_change":"2024-02-13T00:43:03.666039+0000","last_active":"2024-02-13T00:43:03.666060+0000","last_peered":"2024-02-13T00:43:03.666060+0000","last_clean":"2024-02-13T00:43:03.666060+0000","last_became_active":"2024-02-13T00:42:53.368249+0000","last_became_peered":"2024-02-13T00:42:53.368249+0000","last_unstale":"2024-02-13T00:43:03.666060+0000","last_undegraded":"2024-02-13T00:43:03.666060+0000","last_fullsized":"2024-02-13T00:43:03.666060+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:03.665973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:03.665973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":191,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.340304+0000","last_change":"2024-02-13T00:42:47.415483+0000","last_active":"2024-02-13T00:43:03.340304+0000","last_peered":"2024-02-13T00:43:03.340304+0000","last_clean":"2024-02-13T00:43:03.340304+0000","last_became_active":"2024-02-13T00:37:45.243961+0000","last_became_peered":"2024-02-13T00:37:45.243961+0000","last_unstale":"2024-02-13T00:43:03.340304+0000","last_undegraded":"2024-02-13T00:43:03.340304+0000","last_fullsized":"2024-02-13T00:43:03.340304+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.415414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.415414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":184,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.350676+0000","last_change":"2024-02-13T00:42:21.648088+0000","last_active":"2024-02-13T00:43:03.350676+0000","last_peered":"2024-02-13T00:43:03.350676+0000","last_clean":"2024-02-13T00:43:03.350676+0000","last_became_active":"2024-02-13T00:37:47.259688+0000","last_became_peered":"2024-02-13T00:37:47.259688+0000","last_unstale":"2024-02-13T00:43:03.350676+0000","last_undegraded":"2024-02-13T00:43:03.350676+0000","last_fullsized":"2024-02-13T00:43:03.350676+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:21.648002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.648002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":137,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.340866+0000","last_change":"2024-02-13T00:43:01.652331+0000","last_active":"2024-02-13T00:43:03.340866+0000","last_peered":"2024-02-13T00:43:03.340866+0000","last_clean":"2024-02-13T00:43:03.340866+0000","last_became_active":"2024-02-13T00:41:01.620258+0000","last_became_peered":"2024-02-13T00:41:01.620258+0000","last_unstale":"2024-02-13T00:43:03.340866+0000","last_undegraded":"2024-02-13T00:43:03.340866+0000","last_fullsized":"2024-02-13T00:43:03.340866+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:01.652242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:43:01.652242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":131,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.340749+0000","last_change":"2024-02-13T00:41:01.575407+0000","last_active":"2024-02-13T00:43:03.340749+0000","last_peered":"2024-02-13T00:43:03.340749+0000","last_clean":"2024-02-13T00:43:03.340749+0000","last_became_active":"2024-02-13T00:41:01.575117+0000","last_became_peered":"2024-02-13T00:41:01.575117+0000","last_unstale":"2024-02-13T00:43:03.340749+0000","last_undegraded":"2024-02-13T00:43:03.340749+0000","last_fullsized":"2024-02-13T00:43:03.340749+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"85'14","reported_seq":210,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.358524+0000","last_change":"2024-02-13T00:41:47.734999+0000","last_active":"2024-02-13T00:43:03.358524+0000","last_peered":"2024-02-13T00:43:03.358524+0000","last_clean":"2024-02-13T00:43:03.358524+0000","last_became_active":"2024-02-13T00:40:35.856715+0000","last_became_peered":"2024-02-13T00:40:35.856715+0000","last_unstale":"2024-02-13T00:43:03.358524+0000","last_undegraded":"2024-02-13T00:43:03.358524+0000","last_fullsized":"2024-02-13T00:43:03.358524+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:41:47.734904+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":147,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.352416+0000","last_change":"2024-02-13T00:42:53.369328+0000","last_active":"2024-02-13T00:43:03.352416+0000","last_peered":"2024-02-13T00:43:03.352416+0000","last_clean":"2024-02-13T00:43:03.352416+0000","last_became_active":"2024-02-13T00:42:53.369040+0000","last_became_peered":"2024-02-13T00:42:53.369040+0000","last_unstale":"2024-02-13T00:43:03.352416+0000","last_undegraded":"2024-02-13T00:43:03.352416+0000","last_fullsized":"2024-02-13T00:43:03.352416+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:21.630532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.630532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":23,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.348628+0000","last_change":"2024-02-13T00:42:53.369811+0000","last_active":"2024-02-13T00:43:03.348628+0000","last_peered":"2024-02-13T00:43:03.348628+0000","last_clean":"2024-02-13T00:43:03.348628+0000","last_became_active":"2024-02-13T00:42:53.369493+0000","last_became_peered":"2024-02-13T00:42:53.369493+0000","last_unstale":"2024-02-13T00:43:03.348628+0000","last_undegraded":"2024-02-13T00:43:03.348628+0000","last_fullsized":"2024-02-13T00:43:03.348628+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:48.629024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.629024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":189,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.341211+0000","last_change":"2024-02-13T00:42:52.447787+0000","last_active":"2024-02-13T00:43:03.341211+0000","last_peered":"2024-02-13T00:43:03.341211+0000","last_clean":"2024-02-13T00:43:03.341211+0000","last_became_active":"2024-02-13T00:37:47.260025+0000","last_became_peered":"2024-02-13T00:37:47.260025+0000","last_unstale":"2024-02-13T00:43:03.341211+0000","last_undegraded":"2024-02-13T00:43:03.341211+0000","last_fullsized":"2024-02-13T00:43:03.341211+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:52.447698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.447698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.d","version":"108'5","reported_seq":137,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.360130+0000","last_change":"2024-02-13T00:41:01.688873+0000","last_active":"2024-02-13T00:43:03.360130+0000","last_peered":"2024-02-13T00:43:03.360130+0000","last_clean":"2024-02-13T00:43:03.360130+0000","last_became_active":"2024-02-13T00:41:01.688580+0000","last_became_peered":"2024-02-13T00:41:01.688580+0000","last_unstale":"2024-02-13T00:43:03.360130+0000","last_undegraded":"2024-02-13T00:43:03.360130+0000","last_fullsized":"2024-02-13T00:43:03.360130+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":73,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.370225+0000","last_change":"2024-02-13T00:42:03.421775+0000","last_active":"2024-02-13T00:43:03.370225+0000","last_peered":"2024-02-13T00:43:03.370225+0000","last_clean":"2024-02-13T00:43:03.370225+0000","last_became_active":"2024-02-13T00:42:03.421135+0000","last_became_peered":"2024-02-13T00:42:03.421135+0000","last_unstale":"2024-02-13T00:43:03.370225+0000","last_undegraded":"2024-02-13T00:43:03.370225+0000","last_fullsized":"2024-02-13T00:43:03.370225+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"90'20","reported_seq":156,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.343278+0000","last_change":"2024-02-13T00:42:09.194378+0000","last_active":"2024-02-13T00:43:03.343278+0000","last_peered":"2024-02-13T00:43:03.343278+0000","last_clean":"2024-02-13T00:43:03.343278+0000","last_became_active":"2024-02-13T00:42:03.431355+0000","last_became_peered":"2024-02-13T00:42:03.431355+0000","last_unstale":"2024-02-13T00:43:03.343278+0000","last_undegraded":"2024-02-13T00:43:03.343278+0000","last_fullsized":"2024-02-13T00:43:03.343278+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:40:13.144451+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":190,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.364324+0000","last_change":"2024-02-13T00:42:18.464467+0000","last_active":"2024-02-13T00:43:03.364324+0000","last_peered":"2024-02-13T00:43:03.364324+0000","last_clean":"2024-02-13T00:43:03.364324+0000","last_became_active":"2024-02-13T00:37:41.216896+0000","last_became_peered":"2024-02-13T00:37:41.216896+0000","last_unstale":"2024-02-13T00:43:03.364324+0000","last_undegraded":"2024-02-13T00:43:03.364324+0000","last_fullsized":"2024-02-13T00:43:03.364324+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:18.464402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:42:18.464402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":187,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347097+0000","last_change":"2024-02-13T00:41:34.693324+0000","last_active":"2024-02-13T00:43:03.347097+0000","last_peered":"2024-02-13T00:43:03.347097+0000","last_clean":"2024-02-13T00:43:03.347097+0000","last_became_active":"2024-02-13T00:37:45.249914+0000","last_became_peered":"2024-02-13T00:37:45.249914+0000","last_unstale":"2024-02-13T00:43:03.347097+0000","last_undegraded":"2024-02-13T00:43:03.347097+0000","last_fullsized":"2024-02-13T00:43:03.347097+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.693214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.693214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":150,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.339719+0000","last_change":"2024-02-13T00:41:47.487953+0000","last_active":"2024-02-13T00:43:03.339719+0000","last_peered":"2024-02-13T00:43:03.339719+0000","last_clean":"2024-02-13T00:43:03.339719+0000","last_became_active":"2024-02-13T00:40:35.845664+0000","last_became_peered":"2024-02-13T00:40:35.845664+0000","last_unstale":"2024-02-13T00:43:03.339719+0000","last_undegraded":"2024-02-13T00:43:03.339719+0000","last_fullsized":"2024-02-13T00:43:03.339719+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:47.487860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:47.487860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":141,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.355773+0000","last_change":"2024-02-13T00:42:03.409916+0000","last_active":"2024-02-13T00:43:03.355773+0000","last_peered":"2024-02-13T00:43:03.355773+0000","last_clean":"2024-02-13T00:43:03.355773+0000","last_became_active":"2024-02-13T00:42:03.409603+0000","last_became_peered":"2024-02-13T00:42:03.409603+0000","last_unstale":"2024-02-13T00:43:03.355773+0000","last_undegraded":"2024-02-13T00:43:03.355773+0000","last_fullsized":"2024-02-13T00:43:03.355773+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":157,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.347456+0000","last_change":"2024-02-13T00:42:03.539776+0000","last_active":"2024-02-13T00:43:03.347456+0000","last_peered":"2024-02-13T00:43:03.347456+0000","last_clean":"2024-02-13T00:43:03.347456+0000","last_became_active":"2024-02-13T00:42:03.399849+0000","last_became_peered":"2024-02-13T00:42:03.399849+0000","last_unstale":"2024-02-13T00:43:03.347456+0000","last_undegraded":"2024-02-13T00:43:03.347456+0000","last_fullsized":"2024-02-13T00:43:03.347456+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"86'50","reported_seq":289,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.357050+0000","last_change":"2024-02-13T00:42:17.454326+0000","last_active":"2024-02-13T00:43:03.357050+0000","last_peered":"2024-02-13T00:43:03.357050+0000","last_clean":"2024-02-13T00:43:03.357050+0000","last_became_active":"2024-02-13T00:37:43.228728+0000","last_became_peered":"2024-02-13T00:37:43.228728+0000","last_unstale":"2024-02-13T00:43:03.357050+0000","last_undegraded":"2024-02-13T00:43:03.357050+0000","last_fullsized":"2024-02-13T00:43:03.357050+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:42:17.454232+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.454232+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":190,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.464493+0000","last_change":"2024-02-13T00:43:03.464468+0000","last_active":"2024-02-13T00:43:03.464493+0000","last_peered":"2024-02-13T00:43:03.464493+0000","last_clean":"2024-02-13T00:43:03.464493+0000","last_became_active":"2024-02-13T00:37:41.220583+0000","last_became_peered":"2024-02-13T00:37:41.220583+0000","last_unstale":"2024-02-13T00:43:03.464493+0000","last_undegraded":"2024-02-13T00:43:03.464493+0000","last_fullsized":"2024-02-13T00:43:03.464493+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:43:03.464398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":76,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.342825+0000","last_change":"2024-02-13T00:42:03.404545+0000","last_active":"2024-02-13T00:43:03.342825+0000","last_peered":"2024-02-13T00:43:03.342825+0000","last_clean":"2024-02-13T00:43:03.342825+0000","last_became_active":"2024-02-13T00:42:03.404116+0000","last_became_peered":"2024-02-13T00:42:03.404116+0000","last_unstale":"2024-02-13T00:43:03.342825+0000","last_undegraded":"2024-02-13T00:43:03.342825+0000","last_fullsized":"2024-02-13T00:43:03.342825+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:46.418859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:41:46.418859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":193,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.358390+0000","last_change":"2024-02-13T00:42:03.429789+0000","last_active":"2024-02-13T00:43:03.358390+0000","last_peered":"2024-02-13T00:43:03.358390+0000","last_clean":"2024-02-13T00:43:03.358390+0000","last_became_active":"2024-02-13T00:42:03.429290+0000","last_became_peered":"2024-02-13T00:42:03.429290+0000","last_unstale":"2024-02-13T00:43:03.358390+0000","last_undegraded":"2024-02-13T00:43:03.358390+0000","last_fullsized":"2024-02-13T00:43:03.358390+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:54.059274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:54.059274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":132,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-13T00:43:03.345790+0000","last_change":"2024-02-13T00:41:01.573580+0000","last_active":"2024-02-13T00:43:03.345790+0000","last_peered":"2024-02-13T00:43:03.345790+0000","last_clean":"2024-02-13T00:43:03.345790+0000","last_became_active":"2024-02-13T00:41:01.573417+0000","last_became_peered":"2024-02-13T00:41:01.573417+0000","last_unstale":"2024-02-13T00:43:03.345790+0000","last_undegraded":"2024-02-13T00:43:03.345790+0000","last_fullsized":"2024-02-13T00:43:03.345790+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:59.413183+0000","last_clean_scrub_stamp":"2024-02-13T00:40:59.413183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":156,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":155,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":1976,"num_objects":185,"num_object_clones":75,"num_object_copies":555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":86,"num_read_kb":1,"num_write":518,"num_write_kb":516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2260992,"data_stored":5928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1583,"ondisk_log_size":1583,"up":96,"acting":96,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":6656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":154,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1071,"num_read_kb":796,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":676,"ondisk_log_size":676,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":315,"num_read_kb":310,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266755,"num_pgs":140,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313264,"kb_used_data":3748,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431888,"statfs":{"total":95995035648,"available":95674253312,"internally_reserved":0,"allocated":3837952,"data_stored":1977138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.90900000000000003,"15min":0.63800000000000001},"min":{"1min":0.34100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.73199999999999998,"5min":10.682,"15min":10.682},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.22,"15min":0.76000000000000001},"min":{"1min":0.36399999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.4160000000000004,"5min":25.504000000000001,"15min":25.504000000000001},"last":0.45300000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.72999999999999998,"15min":0.57399999999999995},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.64200000000000002,"5min":13.347,"15min":13.347},"last":0.441},{"interface":"front","average":{"1min":1.2,"5min":0.96799999999999997,"15min":0.73799999999999999},"min":{"1min":0.379,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":13.07,"5min":13.526999999999999,"15min":13.526999999999999},"last":0.499}]},{"osd":2,"last update":"Tue Feb 13 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.83199999999999996,"15min":0.64000000000000001},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.625,"5min":14.359,"15min":14.359},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.97899999999999998,"15min":0.70999999999999996},"min":{"1min":0.35499999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.4289999999999998,"5min":21.497,"15min":21.497},"last":0.47199999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.97999999999999998,"15min":0.70899999999999996},"min":{"1min":0.40000000000000002,"5min":0.313,"15min":0.313},"max":{"1min":4.524,"5min":12.997,"15min":12.997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":1.298,"15min":0.78700000000000003},"min":{"1min":0.36699999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.69999999999999996,"5min":21.622,"15min":21.622},"last":0.51400000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.42299999999999999,"15min":0.45700000000000002},"min":{"1min":0.27200000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.52100000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.42699999999999999,"15min":0.45500000000000002},"min":{"1min":0.28000000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.57099999999999995,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.47899999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.44500000000000001,"15min":0.46700000000000003},"min":{"1min":0.26800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.53200000000000003,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.46100000000000002,"15min":0.46899999999999997},"min":{"1min":0.311,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.57299999999999995,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.57099999999999995}]},{"osd":6,"last update":"Tue Feb 13 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47699999999999998,"15min":0.49199999999999999},"min":{"1min":0.309,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.61199999999999999,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.47799999999999998,"15min":0.51400000000000001},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.57899999999999996,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.41199999999999998}]}]},{"osd":6,"up_from":0,"seq":193273528393,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":151,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309076,"kb_used_data":3652,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436076,"statfs":{"total":95995035648,"available":95678541824,"internally_reserved":0,"allocated":3739648,"data_stored":1971405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:42:21 2024","interfaces":[{"interface":"back","average":{"1min":6.9429999999999996,"5min":2.137,"15min":0.92300000000000004},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":61.856000000000002,"5min":61.856000000000002,"15min":61.856000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":4.5880000000000001,"5min":1.2809999999999999,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":61.908999999999999,"5min":61.908999999999999,"15min":61.908999999999999},"last":0.27000000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":1.7150000000000001,"15min":0.82199999999999995},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":11.304,"5min":32.161000000000001,"15min":32.161000000000001},"last":0.312},{"interface":"front","average":{"1min":0.33400000000000002,"5min":1.657,"15min":0.81000000000000005},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.45400000000000001,"5min":29.693000000000001,"15min":29.693000000000001},"last":0.47599999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":10.718,"5min":2.9169999999999998,"15min":1.2729999999999999},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":131.23099999999999,"5min":131.23099999999999,"15min":131.23099999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":6.0789999999999997,"5min":1.726,"15min":0.89000000000000001},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":49.055,"5min":49.055,"15min":49.055},"last":0.32100000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.135,"15min":0.75900000000000001},"min":{"1min":0.29699999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":7.2089999999999996,"5min":16.547000000000001,"15min":16.547000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.94199999999999995,"15min":0.68799999999999994},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":10.050000000000001,"5min":14.731999999999999,"15min":14.731999999999999},"last":0.45800000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":1.4630000000000001,"15min":0.86899999999999999},"min":{"1min":0.40799999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79100000000000004,"5min":26.561,"15min":26.561},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.499,"5min":1.1479999999999999,"15min":0.77200000000000002},"min":{"1min":0.36399999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71999999999999997,"5min":16.494,"15min":16.494},"last":0.44700000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":2.0590000000000002,"5min":1.0289999999999999,"15min":0.74299999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":14.772,"5min":14.772,"15min":14.772},"last":0.41599999999999998},{"interface":"front","average":{"1min":3.3039999999999998,"5min":1.5349999999999999,"15min":0.92000000000000004},"min":{"1min":0.39700000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":26.600000000000001,"5min":26.600000000000001,"15min":26.600000000000001},"last":0.55700000000000005}]},{"osd":7,"last update":"Tue Feb 13 00:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.94899999999999995,"15min":0.71399999999999997},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79900000000000004,"5min":14.794,"15min":14.794},"last":0.439},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.4930000000000001,"15min":0.91400000000000003},"min":{"1min":0.36599999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71399999999999997,"5min":26.693999999999999,"15min":26.693999999999999},"last":0.54100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309056,"kb_used_data":3628,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436096,"statfs":{"total":95995035648,"available":95678562304,"internally_reserved":0,"allocated":3715072,"data_stored":1975858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":4.2510000000000003,"5min":1.1579999999999999,"15min":0.60299999999999998},"min":{"1min":0.20499999999999999,"5min":0.189,"15min":0.189},"max":{"1min":30.184000000000001,"5min":30.184000000000001,"15min":30.184000000000001},"last":0.25},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.50800000000000001,"15min":0.39500000000000002},"min":{"1min":0.215,"5min":0.187,"15min":0.17599999999999999},"max":{"1min":2.2879999999999998,"5min":4.7409999999999997,"15min":4.7409999999999997},"last":0.318}]},{"osd":2,"last update":"Tue Feb 13 00:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59099999999999997,"15min":0.46100000000000002},"min":{"1min":0.23100000000000001,"5min":0.218,"15min":0.20999999999999999},"max":{"1min":4.1230000000000002,"5min":5.9539999999999997,"15min":5.9539999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.161,"15min":0.64000000000000001},"min":{"1min":0.20899999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.4160000000000004,"5min":24.459,"15min":24.459},"last":0.30199999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:42:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":0.58399999999999996,"15min":0.46200000000000002},"min":{"1min":0.28100000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.718,"5min":8.718,"15min":8.718},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.55200000000000005,"15min":0.502},"min":{"1min":0.255,"5min":0.247,"15min":0.247},"max":{"1min":5.931,"5min":5.931,"15min":5.931},"last":0.34000000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:42:33 2024","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.22,"15min":0.75},"min":{"1min":0.29599999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":9.2400000000000002,"5min":24.381,"15min":24.381},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.746,"15min":0.58799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.6950000000000003,"5min":8.6950000000000003,"15min":8.6950000000000003},"last":0.54300000000000004}]},{"osd":5,"last update":"Tue Feb 13 00:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61699999999999999,"15min":0.59299999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.7729999999999999,"5min":4.7270000000000003,"15min":4.7270000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.3320000000000001,"15min":0.82199999999999995},"min":{"1min":0.307,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":9.1880000000000006,"5min":24.427,"15min":24.427},"last":0.55200000000000005}]},{"osd":6,"last update":"Tue Feb 13 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":0.70699999999999996,"15min":0.56999999999999995},"min":{"1min":0.32900000000000001,"5min":0.31,"15min":0.31},"max":{"1min":8.6129999999999995,"5min":8.6129999999999995,"15min":8.6129999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.621,"15min":0.56299999999999994},"min":{"1min":0.41399999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.5619999999999998,"5min":4.681,"15min":4.681},"last":0.52500000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.73199999999999998,"15min":0.60999999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":2.1640000000000001,"5min":8.6679999999999993,"15min":8.6679999999999993},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.67900000000000005,"15min":0.61299999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.157,"5min":4.7110000000000003,"15min":4.7110000000000003},"last":0.45200000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309128,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436024,"statfs":{"total":95995035648,"available":95678488576,"internally_reserved":0,"allocated":3792896,"data_stored":1975867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.80100000000000005,"15min":0.505},"min":{"1min":0.20599999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.0619999999999994,"5min":9.7769999999999992,"15min":9.7769999999999992},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.74199999999999999,"15min":0.47599999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2959999999999994,"5min":9.8160000000000007,"15min":9.8160000000000007},"last":0.251}]},{"osd":1,"last update":"Tue Feb 13 00:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.72599999999999998,"15min":0.47399999999999998},"min":{"1min":0.19,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.5940000000000003,"5min":10.141,"15min":10.141},"last":0.435},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.68899999999999995,"15min":0.47799999999999998},"min":{"1min":0.252,"5min":0.222,"15min":0.222},"max":{"1min":9.0809999999999995,"5min":9.0809999999999995,"15min":9.0809999999999995},"last":0.311}]},{"osd":3,"last update":"Tue Feb 13 00:42:08 2024","interfaces":[{"interface":"back","average":{"1min":1.9910000000000001,"5min":0.72599999999999998,"15min":0.52300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":11.807,"5min":11.807,"15min":11.807},"last":0.505},{"interface":"front","average":{"1min":2.306,"5min":0.80100000000000005,"15min":0.56499999999999995},"min":{"1min":0.26600000000000001,"5min":0.253,"15min":0.253},"max":{"1min":16.981000000000002,"5min":16.981000000000002,"15min":16.981000000000002},"last":0.39400000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:42:30 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.63900000000000001,"15min":0.59299999999999997},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.1420000000000003,"5min":6.1420000000000003,"15min":6.1420000000000003},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.77800000000000002,"15min":0.65600000000000003},"min":{"1min":0.372,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":8.9640000000000004,"5min":8.9640000000000004,"15min":8.9640000000000004},"last":0.51800000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.622,"15min":0.58499999999999996},"min":{"1min":0.34300000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.415,"5min":6.2149999999999999,"15min":6.2149999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.76900000000000002,"15min":0.65700000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":8.9969999999999999,"5min":8.9969999999999999,"15min":8.9969999999999999},"last":0.59599999999999997}]},{"osd":6,"last update":"Tue Feb 13 00:42:14 2024","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":0.749,"15min":0.64000000000000001},"min":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":9.0199999999999996,"5min":9.0199999999999996,"15min":9.0199999999999996},"last":0.624},{"interface":"front","average":{"1min":2.0270000000000001,"5min":0.84999999999999998,"15min":0.66900000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":9.5670000000000002,"5min":9.5670000000000002,"15min":9.5670000000000002},"last":0.40300000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.88800000000000001,"15min":0.69299999999999995},"min":{"1min":0.34699999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":9.343,"5min":9.6099999999999994,"15min":9.6099999999999994},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.66600000000000004,"15min":0.60999999999999999},"min":{"1min":0.36599999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71499999999999997,"5min":6.1829999999999998,"15min":6.1829999999999998},"last":0.55600000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182487,"num_pgs":153,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309056,"kb_used_data":3628,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436096,"statfs":{"total":95995035648,"available":95678562304,"internally_reserved":0,"allocated":3715072,"data_stored":1976836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:42:09 2024","interfaces":[{"interface":"back","average":{"1min":5.4299999999999997,"5min":1.3720000000000001,"15min":0.71299999999999997},"min":{"1min":0.20100000000000001,"5min":0.191,"15min":0.191},"max":{"1min":42.459000000000003,"5min":42.459000000000003,"15min":42.459000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":5.4359999999999999,"5min":1.3899999999999999,"15min":0.73999999999999999},"min":{"1min":0.23499999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":48.088999999999999,"5min":48.088999999999999,"15min":48.088999999999999},"last":0.25}]},{"osd":1,"last update":"Tue Feb 13 00:42:09 2024","interfaces":[{"interface":"back","average":{"1min":3.6579999999999999,"5min":1.0409999999999999,"15min":0.61699999999999999},"min":{"1min":0.249,"5min":0.217,"15min":0.217},"max":{"1min":47.999000000000002,"5min":47.999000000000002,"15min":47.999000000000002},"last":0.314},{"interface":"front","average":{"1min":5.077,"5min":1.381,"15min":0.72899999999999998},"min":{"1min":0.20200000000000001,"5min":0.183,"15min":0.183},"max":{"1min":32.973999999999997,"5min":32.973999999999997,"15min":32.973999999999997},"last":0.224}]},{"osd":2,"last update":"Tue Feb 13 00:42:09 2024","interfaces":[{"interface":"back","average":{"1min":4.3730000000000002,"5min":1.2629999999999999,"15min":0.70399999999999996},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":29.661999999999999,"5min":29.661999999999999,"15min":29.661999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":3.948,"5min":1.1140000000000001,"15min":0.64400000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":35.256999999999998,"5min":35.256999999999998,"15min":35.256999999999998},"last":0.29199999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.3129999999999999,"15min":0.80000000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":7.1210000000000004,"5min":35.156999999999996,"15min":35.156999999999996},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.71599999999999997,"15min":0.59899999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.56799999999999995}]},{"osd":5,"last update":"Tue Feb 13 00:43:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.488,"15min":0.82599999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":7.7210000000000001,"5min":31.420000000000002,"15min":31.420000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":1.341,"15min":0.78900000000000003},"min":{"1min":0.34699999999999998,"5min":0.33900000000000002,"15min":0.32300000000000001},"max":{"1min":2.4209999999999998,"5min":35.209000000000003,"15min":35.209000000000003},"last":0.69799999999999995}]},{"osd":6,"last update":"Tue Feb 13 00:42:21 2024","interfaces":[{"interface":"back","average":{"1min":3.5840000000000001,"5min":1.3740000000000001,"15min":0.79600000000000004},"min":{"1min":0.39300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":31.457999999999998,"5min":31.457999999999998,"15min":31.457999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":2.1549999999999998,"5min":1.2629999999999999,"15min":0.78300000000000003},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":13.603999999999999,"5min":35.280000000000001,"15min":35.280000000000001},"last":0.753}]},{"osd":7,"last update":"Tue Feb 13 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":1.359,"15min":0.81599999999999995},"min":{"1min":0.376,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.677,"5min":35.234000000000002,"15min":35.234000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.75,"15min":0.61199999999999999},"min":{"1min":0.42599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":4.8659999999999997,"5min":4.8659999999999997,"15min":4.8659999999999997},"last":0.68600000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986259,"num_pgs":123,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309220,"kb_used_data":3792,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435932,"statfs":{"total":95995035648,"available":95678394368,"internally_reserved":0,"allocated":3883008,"data_stored":1971501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.746,"15min":0.59399999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.8799999999999999,"5min":9.3399999999999999,"15min":9.3399999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.66800000000000004,"15min":0.58899999999999997},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.91400000000000003,"5min":13.833,"15min":13.833},"last":0.51300000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":1.139,"15min":0.73599999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.61599999999999999,"5min":53.188000000000002,"15min":53.188000000000002},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.97599999999999998,"15min":0.68200000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.0900000000000001,"5min":44.793999999999997,"15min":44.793999999999997},"last":0.498}]},{"osd":2,"last update":"Tue Feb 13 00:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.65900000000000003,"15min":0.57199999999999995},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60699999999999998,"5min":8.6829999999999998,"15min":8.6829999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.76300000000000001,"15min":0.60699999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":2.992,"5min":18.911000000000001,"15min":18.911000000000001},"last":0.42499999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.90000000000000002,"15min":0.65600000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.0300000000000002,"5min":23.5,"15min":23.5},"last":0.379},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.05,"15min":0.70899999999999996},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.01,"5min":36.152999999999999,"15min":36.152999999999999},"last":0.47999999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.41399999999999998,"15min":0.36599999999999999},"min":{"1min":0.26700000000000002,"5min":0.24399999999999999,"15min":0.20599999999999999},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.437,"15min":0.38300000000000001},"min":{"1min":0.29699999999999999,"5min":0.28899999999999998,"15min":0.23799999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.65300000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.434,"15min":0.40600000000000003},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56399999999999995,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.45200000000000001,"15min":0.41699999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.503,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.55300000000000005}]},{"osd":7,"last update":"Tue Feb 13 00:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.47399999999999998,"15min":0.46400000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.56799999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.63},{"interface":"front","average":{"1min":0.443,"5min":0.46500000000000002,"15min":0.45500000000000002},"min":{"1min":0.312,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.60499999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.46000000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724622,"num_pgs":125,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313456,"kb_used_data":3936,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431696,"statfs":{"total":95995035648,"available":95674056704,"internally_reserved":0,"allocated":4030464,"data_stored":1977501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":1.6779999999999999,"15min":0.88200000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.641,"5min":72.444999999999993,"15min":72.444999999999993},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":1.216,"15min":0.73999999999999999},"min":{"1min":0.23799999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.60299999999999998,"5min":23.949000000000002,"15min":23.949000000000002},"last":0.39800000000000002}]},{"osd":1,"last update":"Tue Feb 13 00:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":1.0489999999999999,"15min":0.67900000000000005},"min":{"1min":0.32000000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.6920000000000002,"5min":23.937000000000001,"15min":23.937000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":1.4390000000000001,"15min":0.80700000000000005},"min":{"1min":0.28999999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.57399999999999995,"5min":42.826999999999998,"15min":42.826999999999998},"last":0.44600000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":2.0830000000000002,"15min":1.0289999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68100000000000005,"5min":54.683999999999997,"15min":54.683999999999997},"last":0.433},{"interface":"front","average":{"1min":0.60099999999999998,"5min":1.74,"15min":0.91500000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.6720000000000002,"5min":31.422999999999998,"15min":31.422999999999998},"last":0.46400000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":2.5609999999999999,"15min":1.1870000000000001},"min":{"1min":0.33600000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.91000000000000003,"5min":61.573,"15min":61.573},"last":0.372},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.915,"15min":0.97099999999999997},"min":{"1min":0.33500000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.5870000000000002,"5min":54.848999999999997,"15min":54.848999999999997},"last":0.55900000000000005}]},{"osd":4,"last update":"Tue Feb 13 00:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.53100000000000003,"15min":0.40400000000000003},"min":{"1min":0.25900000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57499999999999996,"5min":7.8570000000000002,"15min":7.8570000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.63900000000000001,"15min":0.46400000000000002},"min":{"1min":0.35799999999999998,"5min":0.27900000000000003,"15min":0.27300000000000002},"max":{"1min":0.56299999999999994,"5min":8.0630000000000006,"15min":8.0630000000000006},"last":0.54400000000000004}]},{"osd":6,"last update":"Tue Feb 13 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.52400000000000002,"15min":0.45500000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.57899999999999996,"5min":8.1020000000000003,"15min":8.1020000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58099999999999996,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":3.7109999999999999,"5min":7.7389999999999999,"15min":7.7389999999999999},"last":0.58199999999999996}]},{"osd":7,"last update":"Tue Feb 13 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.60099999999999998,"15min":0.752},"min":{"1min":0.28899999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.50800000000000001,"5min":7.915,"15min":7.915},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.58399999999999996,"15min":0.58299999999999996},"min":{"1min":0.31900000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.57999999999999996,"5min":8.0169999999999995,"15min":8.0169999999999995},"last":0.57299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":1057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:43:08.831 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-13T00:43:08.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool set default.rgw.meta pg_num 42 2024-02-13T00:43:09.207 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:09.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:07.817395+0000 osd.3 (osd.3) 141 : cluster [DBG] 28.8 scrub starts 2024-02-13T00:43:09.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:07.820306+0000 osd.3 (osd.3) 142 : cluster [DBG] 28.8 scrub ok 2024-02-13T00:43:09.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:07.906688+0000 mgr.y (mgr.14544) 245 : audit [DBG] from='client.26787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:43:09.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:08.170499+0000 mon.b (mon.1) 326 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:09.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:08.356327+0000 mon.a (mon.0) 1596 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:43:09.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:08.600148+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? 172.21.15.162:0/3058490964' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:09.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:08.600243+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:09.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:08.600303+0000 mon.a (mon.0) 1599 : audit [INF] from='client.? 172.21.15.162:0/3588174164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:43:09.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:08.600350+0000 mon.a (mon.0) 1600 : cluster [DBG] osdmap e233: 8 total, 8 up, 7 in 2024-02-13T00:43:09.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:08.600731+0000 mon.b (mon.1) 327 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-23"}]: dispatch 2024-02-13T00:43:09.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:08.602880+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-23"}]: dispatch 2024-02-13T00:43:09.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:07.817395+0000 osd.3 (osd.3) 141 : cluster [DBG] 28.8 scrub starts 2024-02-13T00:43:09.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:07.820306+0000 osd.3 (osd.3) 142 : cluster [DBG] 28.8 scrub ok 2024-02-13T00:43:09.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:07.906688+0000 mgr.y (mgr.14544) 245 : audit [DBG] from='client.26787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:43:09.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:08.170499+0000 mon.b (mon.1) 326 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:09.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:08.356327+0000 mon.a (mon.0) 1596 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:43:09.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:08.600148+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? 172.21.15.162:0/3058490964' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:09.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:08.600243+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:09.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:08.600303+0000 mon.a (mon.0) 1599 : audit [INF] from='client.? 172.21.15.162:0/3588174164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:43:09.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:08.600350+0000 mon.a (mon.0) 1600 : cluster [DBG] osdmap e233: 8 total, 8 up, 7 in 2024-02-13T00:43:09.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:08.600731+0000 mon.b (mon.1) 327 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-23"}]: dispatch 2024-02-13T00:43:09.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:08.602880+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-23"}]: dispatch 2024-02-13T00:43:09.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:09.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:09.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:07.817395+0000 osd.3 (osd.3) 141 : cluster [DBG] 28.8 scrub starts 2024-02-13T00:43:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:07.820306+0000 osd.3 (osd.3) 142 : cluster [DBG] 28.8 scrub ok 2024-02-13T00:43:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:07.906688+0000 mgr.y (mgr.14544) 245 : audit [DBG] from='client.26787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:43:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:08.170499+0000 mon.b (mon.1) 326 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:08.356327+0000 mon.a (mon.0) 1596 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:43:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:08.600148+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? 172.21.15.162:0/3058490964' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:08.600243+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:08.600303+0000 mon.a (mon.0) 1599 : audit [INF] from='client.? 172.21.15.162:0/3588174164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:43:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:08.600350+0000 mon.a (mon.0) 1600 : cluster [DBG] osdmap e233: 8 total, 8 up, 7 in 2024-02-13T00:43:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:08.600731+0000 mon.b (mon.1) 327 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-23"}]: dispatch 2024-02-13T00:43:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:08.602880+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-23"}]: dispatch 2024-02-13T00:43:09.314 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:09.342 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:09.796 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:09.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:09.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:09.933 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:09.954 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:09.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:10.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:10.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:08.357056+0000 mgr.y (mgr.14544) 246 : cluster [DBG] pgmap v319: 324 pgs: 4 active+clean+snaptrim, 96 unknown, 10 active+clean+snaptrim_wait, 214 active+clean; 9.6 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:09.171429+0000 mon.b (mon.1) 328 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:09.370652+0000 mon.a (mon.0) 1602 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-83977-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-83977-33"}]': finished 2024-02-13T00:43:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:09.370742+0000 mon.a (mon.0) 1603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-23"}]': finished 2024-02-13T00:43:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:09.370856+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e234: 8 total, 8 up, 7 in 2024-02-13T00:43:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:09.371617+0000 mon.b (mon.1) 329 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-23", "mode": "writeback"}]: dispatch 2024-02-13T00:43:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:09.373539+0000 mon.c (mon.2) 217 : audit [INF] from='client.? 172.21.15.162:0/1733369977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:09.373627+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-23", "mode": "writeback"}]: dispatch 2024-02-13T00:43:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:09.378480+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:08.357056+0000 mgr.y (mgr.14544) 246 : cluster [DBG] pgmap v319: 324 pgs: 4 active+clean+snaptrim, 96 unknown, 10 active+clean+snaptrim_wait, 214 active+clean; 9.6 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:09.171429+0000 mon.b (mon.1) 328 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:09.370652+0000 mon.a (mon.0) 1602 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-83977-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-83977-33"}]': finished 2024-02-13T00:43:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:09.370742+0000 mon.a (mon.0) 1603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-23"}]': finished 2024-02-13T00:43:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:09.370856+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e234: 8 total, 8 up, 7 in 2024-02-13T00:43:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:09.371617+0000 mon.b (mon.1) 329 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-23", "mode": "writeback"}]: dispatch 2024-02-13T00:43:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:09.373539+0000 mon.c (mon.2) 217 : audit [INF] from='client.? 172.21.15.162:0/1733369977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:09.373627+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-23", "mode": "writeback"}]: dispatch 2024-02-13T00:43:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:09.378480+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:10.086 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:10.106 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:10.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:10.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:10.236 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:08.357056+0000 mgr.y (mgr.14544) 246 : cluster [DBG] pgmap v319: 324 pgs: 4 active+clean+snaptrim, 96 unknown, 10 active+clean+snaptrim_wait, 214 active+clean; 9.6 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:10.236 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:09.171429+0000 mon.b (mon.1) 328 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:10.236 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:09.370652+0000 mon.a (mon.0) 1602 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-83977-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-83977-33"}]': finished 2024-02-13T00:43:10.236 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:09.370742+0000 mon.a (mon.0) 1603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-23"}]': finished 2024-02-13T00:43:10.236 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:09.370856+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e234: 8 total, 8 up, 7 in 2024-02-13T00:43:10.236 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:09.371617+0000 mon.b (mon.1) 329 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-23", "mode": "writeback"}]: dispatch 2024-02-13T00:43:10.237 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:09.373539+0000 mon.c (mon.2) 217 : audit [INF] from='client.? 172.21.15.162:0/1733369977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:10.237 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:09.373627+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-23", "mode": "writeback"}]: dispatch 2024-02-13T00:43:10.237 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:09.378480+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:10.238 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:10.257 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:10.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:10.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:10.416 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:10.564 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:10.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:10.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:10.695 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:11.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:10.172597+0000 mon.b (mon.1) 330 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:10.330380+0000 mon.a (mon.0) 1607 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:10.336420+0000 mon.a (mon.0) 1608 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-23", "mode": "writeback"}]': finished 2024-02-13T00:43:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:10.336526+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:10.336659+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e235: 8 total, 8 up, 7 in 2024-02-13T00:43:11.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:10.172597+0000 mon.b (mon.1) 330 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:11.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:10.330380+0000 mon.a (mon.0) 1607 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:11.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:10.336420+0000 mon.a (mon.0) 1608 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-23", "mode": "writeback"}]': finished 2024-02-13T00:43:11.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:10.336526+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:11.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:10.336659+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e235: 8 total, 8 up, 7 in 2024-02-13T00:43:11.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:10.172597+0000 mon.b (mon.1) 330 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:11.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:10.330380+0000 mon.a (mon.0) 1607 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:11.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:10.336420+0000 mon.a (mon.0) 1608 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-23", "mode": "writeback"}]': finished 2024-02-13T00:43:11.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:10.336526+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:11.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:10.336659+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e235: 8 total, 8 up, 7 in 2024-02-13T00:43:11.344 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:11.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:11.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:11.476 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:10.358017+0000 mgr.y (mgr.14544) 247 : cluster [DBG] pgmap v323: 364 pgs: 40 creating+peering, 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 314 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s; 27 B/s, 1 objects/s recovering 2024-02-13T00:43:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:11.173872+0000 mon.b (mon.1) 331 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:11.350805+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e236: 8 total, 8 up, 7 in 2024-02-13T00:43:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:11.357843+0000 mon.c (mon.2) 218 : audit [INF] from='client.? 172.21.15.162:0/508408979' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:11.359076+0000 mon.c (mon.2) 219 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:11.369389+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:11.369629+0000 mon.a (mon.0) 1613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:11.394901+0000 mon.b (mon.1) 332 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:11.395431+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:12.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:10.358017+0000 mgr.y (mgr.14544) 247 : cluster [DBG] pgmap v323: 364 pgs: 40 creating+peering, 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 314 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s; 27 B/s, 1 objects/s recovering 2024-02-13T00:43:12.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:11.173872+0000 mon.b (mon.1) 331 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:11.350805+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e236: 8 total, 8 up, 7 in 2024-02-13T00:43:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:11.357843+0000 mon.c (mon.2) 218 : audit [INF] from='client.? 172.21.15.162:0/508408979' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:11.359076+0000 mon.c (mon.2) 219 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:11.369389+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:11.369629+0000 mon.a (mon.0) 1613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:11.394901+0000 mon.b (mon.1) 332 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:11.395431+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:12.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:10.358017+0000 mgr.y (mgr.14544) 247 : cluster [DBG] pgmap v323: 364 pgs: 40 creating+peering, 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 314 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s; 27 B/s, 1 objects/s recovering 2024-02-13T00:43:12.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:11.173872+0000 mon.b (mon.1) 331 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:12.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:11.350805+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e236: 8 total, 8 up, 7 in 2024-02-13T00:43:12.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:11.357843+0000 mon.c (mon.2) 218 : audit [INF] from='client.? 172.21.15.162:0/508408979' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:12.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:11.359076+0000 mon.c (mon.2) 219 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:12.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:11.369389+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:12.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:11.369629+0000 mon.a (mon.0) 1613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:12.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:11.394901+0000 mon.b (mon.1) 332 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:12.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:11.395431+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:12.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:43:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:12.175101+0000 mon.b (mon.1) 333 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:12.352679+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:12.352826+0000 mon.a (mon.0) 1616 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]': finished 2024-02-13T00:43:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:12.352943+0000 mon.a (mon.0) 1617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:12.353005+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e237: 8 total, 8 up, 7 in 2024-02-13T00:43:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:12.363125+0000 mon.c (mon.2) 220 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:12.364950+0000 mon.b (mon.1) 334 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23"}]: dispatch 2024-02-13T00:43:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:12.369198+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23"}]: dispatch 2024-02-13T00:43:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:12.369446+0000 mon.a (mon.0) 1620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:13.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:12.175101+0000 mon.b (mon.1) 333 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:13.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:12.352679+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:13.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:12.352826+0000 mon.a (mon.0) 1616 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]': finished 2024-02-13T00:43:13.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:12.352943+0000 mon.a (mon.0) 1617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:13.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:12.353005+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e237: 8 total, 8 up, 7 in 2024-02-13T00:43:13.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:12.363125+0000 mon.c (mon.2) 220 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:13.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:12.364950+0000 mon.b (mon.1) 334 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23"}]: dispatch 2024-02-13T00:43:13.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:12.369198+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23"}]: dispatch 2024-02-13T00:43:13.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:12.369446+0000 mon.a (mon.0) 1620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:13.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:12.175101+0000 mon.b (mon.1) 333 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:13.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:12.352679+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:13.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:12.352826+0000 mon.a (mon.0) 1616 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-83977-33"}]': finished 2024-02-13T00:43:13.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:12.352943+0000 mon.a (mon.0) 1617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:13.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:12.353005+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e237: 8 total, 8 up, 7 in 2024-02-13T00:43:13.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:12.363125+0000 mon.c (mon.2) 220 : audit [INF] from='client.? 172.21.15.162:0/1804765961' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:13.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:12.364950+0000 mon.b (mon.1) 334 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23"}]: dispatch 2024-02-13T00:43:13.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:12.369198+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23"}]: dispatch 2024-02-13T00:43:13.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:12.369446+0000 mon.a (mon.0) 1620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]: dispatch 2024-02-13T00:43:14.119 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:43:14.119 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:43:14.143 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:12.359232+0000 mgr.y (mgr.14544) 248 : cluster [DBG] pgmap v326: 388 pgs: 32 creating+peering, 2 active+clean+snaptrim, 32 unknown, 8 active+clean+snaptrim_wait, 314 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2 KiB/s wr, 4 op/s; 27 B/s, 1 objects/s recovering 2024-02-13T00:43:14.143 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:13.176347+0000 mon.b (mon.1) 335 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:14.143 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:13.350758+0000 mon.a (mon.0) 1621 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:14.143 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:13.359972+0000 mon.a (mon.0) 1622 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23"}]': finished 2024-02-13T00:43:14.143 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:13.360054+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]': finished 2024-02-13T00:43:14.143 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:13.360112+0000 mon.a (mon.0) 1624 : cluster [DBG] osdmap e238: 8 total, 8 up, 7 in 2024-02-13T00:43:14.143 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:13.366755+0000 mon.c (mon.2) 221 : audit [INF] from='client.? 172.21.15.162:0/2642188595' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:14.143 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:13.367423+0000 mon.a (mon.0) 1625 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:14.143 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:13.399075+0000 mon.a (mon.0) 1626 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:14.144 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:13.400077+0000 mon.a (mon.0) 1627 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:14.144 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:13.400467+0000 mon.a (mon.0) 1628 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi162-83977-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:14.144 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:12.359232+0000 mgr.y (mgr.14544) 248 : cluster [DBG] pgmap v326: 388 pgs: 32 creating+peering, 2 active+clean+snaptrim, 32 unknown, 8 active+clean+snaptrim_wait, 314 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2 KiB/s wr, 4 op/s; 27 B/s, 1 objects/s recovering 2024-02-13T00:43:14.144 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:13.176347+0000 mon.b (mon.1) 335 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:14.144 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:13.350758+0000 mon.a (mon.0) 1621 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:14.144 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:13.359972+0000 mon.a (mon.0) 1622 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23"}]': finished 2024-02-13T00:43:14.144 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:13.360054+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]': finished 2024-02-13T00:43:14.144 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:13.360112+0000 mon.a (mon.0) 1624 : cluster [DBG] osdmap e238: 8 total, 8 up, 7 in 2024-02-13T00:43:14.144 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:13.366755+0000 mon.c (mon.2) 221 : audit [INF] from='client.? 172.21.15.162:0/2642188595' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:14.144 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:13.367423+0000 mon.a (mon.0) 1625 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:14.144 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:13.399075+0000 mon.a (mon.0) 1626 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:14.144 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:13.400077+0000 mon.a (mon.0) 1627 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:14.145 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:13.400467+0000 mon.a (mon.0) 1628 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi162-83977-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:14.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:12.359232+0000 mgr.y (mgr.14544) 248 : cluster [DBG] pgmap v326: 388 pgs: 32 creating+peering, 2 active+clean+snaptrim, 32 unknown, 8 active+clean+snaptrim_wait, 314 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2 KiB/s wr, 4 op/s; 27 B/s, 1 objects/s recovering 2024-02-13T00:43:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:13.176347+0000 mon.b (mon.1) 335 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:13.350758+0000 mon.a (mon.0) 1621 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:13.359972+0000 mon.a (mon.0) 1622 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-23"}]': finished 2024-02-13T00:43:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:13.360054+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-83977-33"}]': finished 2024-02-13T00:43:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:13.360112+0000 mon.a (mon.0) 1624 : cluster [DBG] osdmap e238: 8 total, 8 up, 7 in 2024-02-13T00:43:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:13.366755+0000 mon.c (mon.2) 221 : audit [INF] from='client.? 172.21.15.162:0/2642188595' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:13.367423+0000 mon.a (mon.0) 1625 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:13.399075+0000 mon.a (mon.0) 1626 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:13.400077+0000 mon.a (mon.0) 1627 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:13.400467+0000 mon.a (mon.0) 1628 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi162-83977-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:14.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.273+0000 7f0c38155700 1 -- 172.21.15.162:0/4028922531 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c300f95e0 msgr2=0x7f0c30101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.273+0000 7f0c38155700 1 --2- 172.21.15.162:0/4028922531 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c300f95e0 0x7f0c30101d80 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 rx=0x7f0c200047b0 tx=0x7f0c20009cb0).stop 2024-02-13T00:43:14.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.273+0000 7f0c38155700 1 -- 172.21.15.162:0/4028922531 shutdown_connections 2024-02-13T00:43:14.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.273+0000 7f0c38155700 1 --2- 172.21.15.162:0/4028922531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c30102270 0x7f0c30104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:43:14.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.273+0000 7f0c38155700 1 --2- 172.21.15.162:0/4028922531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c300f8d10 0x7f0c300f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.273+0000 7f0c38155700 1 --2- 172.21.15.162:0/4028922531 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c300f95e0 0x7f0c30101d80 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.273+0000 7f0c38155700 1 -- 172.21.15.162:0/4028922531 >> 172.21.15.162:0/4028922531 conn(0x7f0c3006eb90 msgr2=0x7f0c300f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:14.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.273+0000 7f0c38155700 1 -- 172.21.15.162:0/4028922531 shutdown_connections 2024-02-13T00:43:14.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.273+0000 7f0c38155700 1 -- 172.21.15.162:0/4028922531 wait complete. 2024-02-13T00:43:14.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c38155700 1 librados: starting msgr at 2024-02-13T00:43:14.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c38155700 1 librados: starting objecter 2024-02-13T00:43:14.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c38155700 1 Processor -- start 2024-02-13T00:43:14.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c38155700 1 -- start start 2024-02-13T00:43:14.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c38155700 1 librados: setting wanted keys 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c38155700 1 librados: calling monclient init 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c38155700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c300f8d10 0x7f0c3010e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c38155700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c300f95e0 0x7f0c3010e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c38155700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c30102270 0x7f0c30108490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c38155700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30108980 con 0x7f0c300f95e0 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c38155700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30108af0 con 0x7f0c300f8d10 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c38155700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30108cb0 con 0x7f0c30102270 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c356f0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c300f95e0 0x7f0c3010e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c366f2700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c30102270 0x7f0c30108490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c366f2700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c30102270 0x7f0c30108490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48788/0 (socket says 172.21.15.162:48788) 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c356f0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c300f95e0 0x7f0c3010e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:58796/0 (socket says 172.21.15.162:58796) 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c366f2700 1 -- 172.21.15.162:0/3867626461 learned_addr learned my addr 172.21.15.162:0/3867626461 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:14.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c35ef1700 1 --2- 172.21.15.162:0/3867626461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c300f8d10 0x7f0c3010e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c366f2700 1 -- 172.21.15.162:0/3867626461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c300f8d10 msgr2=0x7f0c3010e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c366f2700 1 --2- 172.21.15.162:0/3867626461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c300f8d10 0x7f0c3010e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c366f2700 1 -- 172.21.15.162:0/3867626461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c300f95e0 msgr2=0x7f0c3010e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c366f2700 1 --2- 172.21.15.162:0/3867626461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c300f95e0 0x7f0c3010e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c366f2700 1 -- 172.21.15.162:0/3867626461 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c005040 con 0x7f0c30102270 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.274+0000 7f0c356f0700 1 --2- 172.21.15.162:0/3867626461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c300f95e0 0x7f0c3010e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.275+0000 7f0c366f2700 1 --2- 172.21.15.162:0/3867626461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c30102270 0x7f0c30108490 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7f0c2800bb60 tx=0x7f0c28009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.275+0000 7f0c26ffd700 10 client.?.objecter ms_handle_connect 0x7f0c30102270 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.275+0000 7f0c26ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.275+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2800c7e0 con 0x7f0c30102270 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.275+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c20005040 con 0x7f0c30102270 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.275+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c2800ce10 con 0x7f0c30102270 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.275+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28015c10 con 0x7f0c30102270 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.275+0000 7f0c38155700 10 client.36917.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.275+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c301092a0 con 0x7f0c30102270 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.275+0000 7f0c38155700 1 librados: init done 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.275+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0c300475e0 con 0x7f0c30102270 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.276+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0c28012020 con 0x7f0c30102270 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.276+0000 7f0c26ffd700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0c1c062c90 0x7f0c1c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.276+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(238..238 src has 52..238) v4 ==== 10282+0+0 (secure 0 0 0) 0x7f0c2807dbb0 con 0x7f0c30102270 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.276+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 osd_map(238..238 src has 52..238) v4 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.276+0000 7f0c26ffd700 3 client.36917.objecter handle_osd_map got epochs [238,238] > 0 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.276+0000 7f0c26ffd700 3 client.36917.objecter handle_osd_map decoding full epoch 238 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.276+0000 7f0c35ef1700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0c1c062c90 0x7f0c1c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c26ffd700 20 client.36917.objecter dump_active .. 0 homeless 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v238) v1 ==== 54+0+16 (secure 0 0 0) 0x7f0c28082220 con 0x7f0c30102270 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c26ffd700 10 client.36917.objecter ms_handle_connect 0x7f0c1c062c90 2024-02-13T00:43:14.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c35ef1700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0c1c062c90 0x7f0c1c065150 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 rx=0x7f0c2c004870 tx=0x7f0c2c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c38155700 10 client.36917.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 1 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0c30028590 0x7f0c300fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c301071b0 osd=0 3 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c301071b0 osd=0 3 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c301071b0 osd=0 3 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 1 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c301071b0 osd=0 3 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 0 1 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c301071b0 osd=0 4 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c38155700 10 client.36917.objecter _send_command 1 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c30109970 con 0x7f0c30028590 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c356f0700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0c30028590 0x7f0c300fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c356f0700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0c30028590 0x7f0c300fbd00 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.277+0000 7f0c26ffd700 10 client.36917.objecter ms_handle_connect 0x7f0c30028590 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.278+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0c30109970 con 0x7f0c30028590 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.278+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.278+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 1 = system:0 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.278+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c301071b0 osd=0 4 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.278+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 0 1 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.293+0000 7f0c38155700 10 client.36917.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.293+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 2 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.293+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c301071b0 osd=0 3 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.293+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c301071b0 osd=0 3 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.293+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c301071b0 osd=0 3 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.293+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 2 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.293+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c301071b0 osd=0 3 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.293+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 0 2 2024-02-13T00:43:14.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.293+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c301071b0 osd=0 4 2024-02-13T00:43:14.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.293+0000 7f0c38155700 10 client.36917.objecter _send_command 2 2024-02-13T00:43:14.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.293+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0c3005a0e0 con 0x7f0c30028590 2024-02-13T00:43:14.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.294+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0c3005a0e0 con 0x7f0c30028590 2024-02-13T00:43:14.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.294+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:14.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.294+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.294+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c301071b0 osd=0 4 2024-02-13T00:43:14.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.294+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 0 2 2024-02-13T00:43:14.304 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:43:14.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c38155700 10 client.36917.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 3 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6805/150599780 conn(0x7f0c30027be0 0x7f0c30028000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c300fd370 osd=1 3 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c300fd370 osd=1 3 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c300fd370 osd=1 3 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 3 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c300fd370 osd=1 3 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 1 3 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c300fd370 osd=1 4 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c38155700 10 client.36917.objecter _send_command 3 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c300fd4e0 con 0x7f0c30027be0 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c35ef1700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6805/150599780 conn(0x7f0c30027be0 0x7f0c30028000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c26ffd700 10 client.36917.objecter ms_handle_connect 0x7f0c30027be0 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.295+0000 7f0c35ef1700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6805/150599780 conn(0x7f0c30027be0 0x7f0c30028000 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.296+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0c300fd4e0 con 0x7f0c30027be0 2024-02-13T00:43:14.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.296+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:43:14.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.296+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 3 = system:0 2024-02-13T00:43:14.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.296+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c300fd370 osd=1 4 2024-02-13T00:43:14.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.296+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 1 3 2024-02-13T00:43:14.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.311+0000 7f0c38155700 10 client.36917.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:14.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.311+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 4 2024-02-13T00:43:14.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.311+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c300fd370 osd=1 3 2024-02-13T00:43:14.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.311+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c300fd370 osd=1 3 2024-02-13T00:43:14.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.311+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c300fd370 osd=1 3 2024-02-13T00:43:14.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.311+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 4 2024-02-13T00:43:14.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.311+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c300fd370 osd=1 3 2024-02-13T00:43:14.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.311+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 1 4 2024-02-13T00:43:14.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.311+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c300fd370 osd=1 4 2024-02-13T00:43:14.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.311+0000 7f0c38155700 10 client.36917.objecter _send_command 4 2024-02-13T00:43:14.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.311+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0c30027650 con 0x7f0c30027be0 2024-02-13T00:43:14.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0c30027650 con 0x7f0c30027be0 2024-02-13T00:43:14.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:14.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c300fd370 osd=1 4 2024-02-13T00:43:14.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 1 4 2024-02-13T00:43:14.316 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c38155700 10 client.36917.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 5 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0c300fced0 0x7f0c3011c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c30027a60 osd=2 3 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30027a60 osd=2 3 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c30027a60 osd=2 3 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 5 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c30027a60 osd=2 3 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 2 5 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30027a60 osd=2 4 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c38155700 10 client.36917.objecter _send_command 5 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.312+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c3011ef80 con 0x7f0c300fced0 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.313+0000 7f0c356f0700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0c300fced0 0x7f0c3011c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:14.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.313+0000 7f0c26ffd700 10 client.36917.objecter ms_handle_connect 0x7f0c300fced0 2024-02-13T00:43:14.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.313+0000 7f0c356f0700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0c300fced0 0x7f0c3011c330 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:14.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.313+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0c3011ef80 con 0x7f0c300fced0 2024-02-13T00:43:14.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.313+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:43:14.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.313+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 5 = system:0 2024-02-13T00:43:14.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.313+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c30027a60 osd=2 4 2024-02-13T00:43:14.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.313+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 2 5 2024-02-13T00:43:14.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.328+0000 7f0c38155700 10 client.36917.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:14.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.328+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 6 2024-02-13T00:43:14.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.328+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c30027a60 osd=2 3 2024-02-13T00:43:14.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.328+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30027a60 osd=2 3 2024-02-13T00:43:14.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.328+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c30027a60 osd=2 3 2024-02-13T00:43:14.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.328+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 6 2024-02-13T00:43:14.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.328+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c30027a60 osd=2 3 2024-02-13T00:43:14.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.328+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 2 6 2024-02-13T00:43:14.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.328+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30027a60 osd=2 4 2024-02-13T00:43:14.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.328+0000 7f0c38155700 10 client.36917.objecter _send_command 6 2024-02-13T00:43:14.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.328+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0c300fc430 con 0x7f0c300fced0 2024-02-13T00:43:14.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0c300fc430 con 0x7f0c300fced0 2024-02-13T00:43:14.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:14.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c30027a60 osd=2 4 2024-02-13T00:43:14.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 2 6 2024-02-13T00:43:14.333 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:43:14.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c38155700 10 client.36917.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:14.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 7 2024-02-13T00:43:14.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6813/604971011 conn(0x7f0c30119e80 0x7f0c3011d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c300fc840 osd=3 3 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c300fc840 osd=3 3 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c300fc840 osd=3 3 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 7 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c300fc840 osd=3 3 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 3 7 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c300fc840 osd=3 4 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c38155700 10 client.36917.objecter _send_command 7 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c3011df90 con 0x7f0c30119e80 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.329+0000 7f0c366f2700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6813/604971011 conn(0x7f0c30119e80 0x7f0c3011d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.330+0000 7f0c26ffd700 10 client.36917.objecter ms_handle_connect 0x7f0c30119e80 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.330+0000 7f0c366f2700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6813/604971011 conn(0x7f0c30119e80 0x7f0c3011d9a0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.330+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0c3011df90 con 0x7f0c30119e80 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.330+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:43:14.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.330+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 7 = system:0 2024-02-13T00:43:14.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.330+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c300fc840 osd=3 4 2024-02-13T00:43:14.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.330+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 3 7 2024-02-13T00:43:14.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.344+0000 7f0c38155700 10 client.36917.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:14.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.344+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 8 2024-02-13T00:43:14.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.344+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c300fc840 osd=3 3 2024-02-13T00:43:14.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.344+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c300fc840 osd=3 3 2024-02-13T00:43:14.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.344+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c300fc840 osd=3 3 2024-02-13T00:43:14.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.344+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 8 2024-02-13T00:43:14.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.344+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c300fc840 osd=3 3 2024-02-13T00:43:14.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.344+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 3 8 2024-02-13T00:43:14.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.344+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c300fc840 osd=3 4 2024-02-13T00:43:14.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.344+0000 7f0c38155700 10 client.36917.objecter _send_command 8 2024-02-13T00:43:14.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.344+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0c301193e0 con 0x7f0c30119e80 2024-02-13T00:43:14.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.345+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0c301193e0 con 0x7f0c30119e80 2024-02-13T00:43:14.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.345+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:14.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.345+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.345+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c300fc840 osd=3 4 2024-02-13T00:43:14.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.345+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 3 8 2024-02-13T00:43:14.349 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:43:14.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c38155700 10 client.36917.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:14.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 9 2024-02-13T00:43:14.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0c30119580 0x7f0c30123160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c301193e0 osd=4 3 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c301193e0 osd=4 3 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c301193e0 osd=4 3 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 9 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c301193e0 osd=4 3 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 4 9 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c301193e0 osd=4 4 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c38155700 10 client.36917.objecter _send_command 9 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c30125ee0 con 0x7f0c30119580 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c35ef1700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0c30119580 0x7f0c30123160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c26ffd700 10 client.36917.objecter ms_handle_connect 0x7f0c30119580 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.346+0000 7f0c35ef1700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0c30119580 0x7f0c30123160 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.347+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0c30125ee0 con 0x7f0c30119580 2024-02-13T00:43:14.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.347+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:43:14.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.347+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 9 = system:0 2024-02-13T00:43:14.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.347+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c301193e0 osd=4 4 2024-02-13T00:43:14.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.347+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 4 9 2024-02-13T00:43:14.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.368+0000 7f0c38155700 10 client.36917.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:14.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.368+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 10 2024-02-13T00:43:14.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.368+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c301193e0 osd=4 3 2024-02-13T00:43:14.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.368+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c301193e0 osd=4 3 2024-02-13T00:43:14.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.368+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c301193e0 osd=4 3 2024-02-13T00:43:14.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.368+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 10 2024-02-13T00:43:14.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.368+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c301193e0 osd=4 3 2024-02-13T00:43:14.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.368+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 4 10 2024-02-13T00:43:14.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.368+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c301193e0 osd=4 4 2024-02-13T00:43:14.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.368+0000 7f0c38155700 10 client.36917.objecter _send_command 10 2024-02-13T00:43:14.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.368+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0c30125ee0 con 0x7f0c30119580 2024-02-13T00:43:14.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.374+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0c30125ee0 con 0x7f0c30119580 2024-02-13T00:43:14.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.374+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:14.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.374+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.374+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c301193e0 osd=4 4 2024-02-13T00:43:14.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.374+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 4 10 2024-02-13T00:43:14.378 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:43:14.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.375+0000 7f0c38155700 10 client.36917.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:14.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 11 2024-02-13T00:43:14.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0c3012aa60 0x7f0c3012aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:14.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c30125ee0 osd=5 3 2024-02-13T00:43:14.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30125ee0 osd=5 3 2024-02-13T00:43:14.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c30125ee0 osd=5 3 2024-02-13T00:43:14.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 11 2024-02-13T00:43:14.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c30125ee0 osd=5 3 2024-02-13T00:43:14.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 5 11 2024-02-13T00:43:14.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30125ee0 osd=5 4 2024-02-13T00:43:14.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c38155700 10 client.36917.objecter _send_command 11 2024-02-13T00:43:14.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c3012b3b0 con 0x7f0c3012aa60 2024-02-13T00:43:14.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c356f0700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0c3012aa60 0x7f0c3012aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:14.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c356f0700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0c3012aa60 0x7f0c3012aec0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:14.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.376+0000 7f0c26ffd700 10 client.36917.objecter ms_handle_connect 0x7f0c3012aa60 2024-02-13T00:43:14.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.377+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0c3012b3b0 con 0x7f0c3012aa60 2024-02-13T00:43:14.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.377+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:43:14.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.377+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 11 = system:0 2024-02-13T00:43:14.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.377+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c30125ee0 osd=5 4 2024-02-13T00:43:14.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.377+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 5 11 2024-02-13T00:43:14.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.403+0000 7f0c38155700 10 client.36917.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:14.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.403+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 12 2024-02-13T00:43:14.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.403+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c30125ee0 osd=5 3 2024-02-13T00:43:14.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.403+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30125ee0 osd=5 3 2024-02-13T00:43:14.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.403+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c30125ee0 osd=5 3 2024-02-13T00:43:14.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.403+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 12 2024-02-13T00:43:14.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.403+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c30125ee0 osd=5 3 2024-02-13T00:43:14.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.403+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 5 12 2024-02-13T00:43:14.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.403+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30125ee0 osd=5 4 2024-02-13T00:43:14.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.403+0000 7f0c38155700 10 client.36917.objecter _send_command 12 2024-02-13T00:43:14.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.403+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0c3012b3b0 con 0x7f0c3012aa60 2024-02-13T00:43:14.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0c3012b3b0 con 0x7f0c3012aa60 2024-02-13T00:43:14.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:14.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c30125ee0 osd=5 4 2024-02-13T00:43:14.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 5 12 2024-02-13T00:43:14.410 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:43:14.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c38155700 10 client.36917.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 13 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0c301236f0 0x7f0c3012bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c3012b3b0 osd=6 3 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c3012b3b0 osd=6 3 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c3012b3b0 osd=6 3 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 13 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c3012b3b0 osd=6 3 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 6 13 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c3012b3b0 osd=6 4 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c38155700 10 client.36917.objecter _send_command 13 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.406+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c30125c90 con 0x7f0c301236f0 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.407+0000 7f0c366f2700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0c301236f0 0x7f0c3012bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.407+0000 7f0c26ffd700 10 client.36917.objecter ms_handle_connect 0x7f0c301236f0 2024-02-13T00:43:14.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.407+0000 7f0c366f2700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0c301236f0 0x7f0c3012bb70 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:14.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.408+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0c30125c90 con 0x7f0c301236f0 2024-02-13T00:43:14.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.408+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:43:14.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.408+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 13 = system:0 2024-02-13T00:43:14.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.408+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c3012b3b0 osd=6 4 2024-02-13T00:43:14.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.408+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 6 13 2024-02-13T00:43:14.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.423+0000 7f0c38155700 10 client.36917.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:14.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.423+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 14 2024-02-13T00:43:14.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.423+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c3012b3b0 osd=6 3 2024-02-13T00:43:14.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.423+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c3012b3b0 osd=6 3 2024-02-13T00:43:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.423+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c3012b3b0 osd=6 3 2024-02-13T00:43:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.423+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 14 2024-02-13T00:43:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.423+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c3012b3b0 osd=6 3 2024-02-13T00:43:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.423+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 6 14 2024-02-13T00:43:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.423+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c3012b3b0 osd=6 4 2024-02-13T00:43:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.423+0000 7f0c38155700 10 client.36917.objecter _send_command 14 2024-02-13T00:43:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.423+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0c30125c90 con 0x7f0c301236f0 2024-02-13T00:43:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0c30125c90 con 0x7f0c301236f0 2024-02-13T00:43:14.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:14.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c3012b3b0 osd=6 4 2024-02-13T00:43:14.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 6 14 2024-02-13T00:43:14.428 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:43:14.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c38155700 10 client.36917.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:14.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 15 2024-02-13T00:43:14.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0c3012c570 0x7f0c3012c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:14.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c30125c90 osd=7 3 2024-02-13T00:43:14.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30125c90 osd=7 3 2024-02-13T00:43:14.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c30125c90 osd=7 3 2024-02-13T00:43:14.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 15 2024-02-13T00:43:14.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c30125c90 osd=7 3 2024-02-13T00:43:14.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 7 15 2024-02-13T00:43:14.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30125c90 osd=7 4 2024-02-13T00:43:14.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c38155700 10 client.36917.objecter _send_command 15 2024-02-13T00:43:14.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c3012e5f0 con 0x7f0c3012c570 2024-02-13T00:43:14.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c35ef1700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0c3012c570 0x7f0c3012c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:14.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c26ffd700 10 client.36917.objecter ms_handle_connect 0x7f0c3012c570 2024-02-13T00:43:14.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.425+0000 7f0c35ef1700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0c3012c570 0x7f0c3012c9f0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:14.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.426+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0c3012e5f0 con 0x7f0c3012c570 2024-02-13T00:43:14.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.426+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:43:14.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.426+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 15 = system:0 2024-02-13T00:43:14.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.426+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c30125c90 osd=7 4 2024-02-13T00:43:14.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.426+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 7 15 2024-02-13T00:43:14.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.450+0000 7f0c38155700 10 client.36917.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:14.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.450+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign -1 16 2024-02-13T00:43:14.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.450+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c30125c90 osd=7 3 2024-02-13T00:43:14.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.450+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30125c90 osd=7 3 2024-02-13T00:43:14.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.450+0000 7f0c38155700 20 client.36917.objecter _get_session s=0x7f0c30125c90 osd=7 3 2024-02-13T00:43:14.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.450+0000 7f0c38155700 15 client.36917.objecter _session_command_op_remove -1 16 2024-02-13T00:43:14.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.450+0000 7f0c38155700 20 client.36917.objecter get_session s=0x7f0c30125c90 osd=7 3 2024-02-13T00:43:14.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.450+0000 7f0c38155700 15 client.36917.objecter _session_command_op_assign 7 16 2024-02-13T00:43:14.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.450+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30125c90 osd=7 4 2024-02-13T00:43:14.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.450+0000 7f0c38155700 10 client.36917.objecter _send_command 16 2024-02-13T00:43:14.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.450+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0c3012e5f0 con 0x7f0c3012c570 2024-02-13T00:43:14.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c26ffd700 1 -- 172.21.15.162:0/3867626461 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0c3012e5f0 con 0x7f0c3012c570 2024-02-13T00:43:14.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c26ffd700 10 client.36917.objecter ms_dispatch 0x7f0c30001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:14.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c26ffd700 10 client.36917.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c26ffd700 20 client.36917.objecter put_session s=0x7f0c30125c90 osd=7 4 2024-02-13T00:43:14.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c26ffd700 15 client.36917.objecter _session_command_op_remove 7 16 2024-02-13T00:43:14.458 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:43:14.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 10 librados: watch_flush enter 2024-02-13T00:43:14.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 10 librados: watch_flush exit 2024-02-13T00:43:14.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 10 client.36917.objecter close_session for osd.0 2024-02-13T00:43:14.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0c30028590 msgr2=0x7f0c300fbd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0c30028590 0x7f0c300fbd00 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c301071b0 osd=0 1 2024-02-13T00:43:14.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 10 client.36917.objecter close_session for osd.1 2024-02-13T00:43:14.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6805/150599780 conn(0x7f0c30027be0 msgr2=0x7f0c30028000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6805/150599780 conn(0x7f0c30027be0 0x7f0c30028000 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c300fd370 osd=1 1 2024-02-13T00:43:14.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 10 client.36917.objecter close_session for osd.2 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0c300fced0 msgr2=0x7f0c3011c330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0c300fced0 0x7f0c3011c330 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30027a60 osd=2 1 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 10 client.36917.objecter close_session for osd.3 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6813/604971011 conn(0x7f0c30119e80 msgr2=0x7f0c3011d9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6813/604971011 conn(0x7f0c30119e80 0x7f0c3011d9a0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c300fc840 osd=3 1 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 10 client.36917.objecter close_session for osd.4 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0c30119580 msgr2=0x7f0c30123160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0c30119580 0x7f0c30123160 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c366f2700 1 -- 172.21.15.162:0/3867626461 reap_dead start 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c301193e0 osd=4 1 2024-02-13T00:43:14.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 10 client.36917.objecter close_session for osd.5 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0c3012aa60 msgr2=0x7f0c3012aec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0c3012aa60 0x7f0c3012aec0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c366f2700 1 -- 172.21.15.162:0/3867626461 reap_dead start 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30125ee0 osd=5 1 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 10 client.36917.objecter close_session for osd.6 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0c301236f0 msgr2=0x7f0c3012bb70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0c301236f0 0x7f0c3012bb70 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c3012b3b0 osd=6 1 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 10 client.36917.objecter close_session for osd.7 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0c3012c570 msgr2=0x7f0c3012c9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0c3012c570 0x7f0c3012c9f0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 20 client.36917.objecter put_session s=0x7f0c30125c90 osd=7 1 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 20 client.36917.objecter shutdown clearing up homeless session... 2024-02-13T00:43:14.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c366f2700 1 -- 172.21.15.162:0/3867626461 reap_dead start 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 10 client.36917.objecter successfully canceled tick 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0c1c062c90 msgr2=0x7f0c1c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0c1c062c90 0x7f0c1c065150 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 rx=0x7f0c2c004870 tx=0x7f0c2c00b040).stop 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c30102270 msgr2=0x7f0c30108490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c30102270 0x7f0c30108490 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7f0c2800bb60 tx=0x7f0c28009810).stop 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 shutdown_connections 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c30102270 0x7f0c30108490 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 --2- 172.21.15.162:0/3867626461 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0c1c062c90 0x7f0c1c065150 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.455+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 >> 172.21.15.162:0/3867626461 conn(0x7f0c3006eb90 msgr2=0x7f0c30102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.456+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 shutdown_connections 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.456+0000 7f0c38155700 1 -- 172.21.15.162:0/3867626461 wait complete. 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:14.456+0000 7f0c38155700 1 librados: shutdown 2024-02-13T00:43:14.463 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:43:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:43:15.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:14.177409+0000 mon.b (mon.1) 336 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:15.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:14.277359+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? 172.21.15.162:0/3867626461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:15.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:14.362477+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:15.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:14.362606+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi162-83977-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:15.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:14.362686+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e239: 8 total, 8 up, 7 in 2024-02-13T00:43:15.166 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:14.369336+0000 mon.a (mon.0) 1632 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi162-83977-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:15.166 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:14.177409+0000 mon.b (mon.1) 336 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:15.166 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:14.277359+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? 172.21.15.162:0/3867626461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:15.166 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:14.362477+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:14.362606+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi162-83977-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:14.362686+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e239: 8 total, 8 up, 7 in 2024-02-13T00:43:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:14.369336+0000 mon.a (mon.0) 1632 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi162-83977-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:15.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:14.177409+0000 mon.b (mon.1) 336 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:14.277359+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? 172.21.15.162:0/3867626461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:14.362477+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-83985-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:14.362606+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi162-83977-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:14.362686+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e239: 8 total, 8 up, 7 in 2024-02-13T00:43:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:14.369336+0000 mon.a (mon.0) 1632 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi162-83977-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:15.706 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-13T00:43:15.757 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:15.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:15.774 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:15.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:43:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:43:15.817 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:15.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:15.894 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:15.915 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:15.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:15.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:16.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:16.046 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:16.066 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:16.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:16.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:16.200 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:14.359725+0000 mgr.y (mgr.14544) 249 : cluster [DBG] pgmap v328: 420 pgs: 32 creating+peering, 2 active+clean+snaptrim, 64 unknown, 8 active+clean+snaptrim_wait, 314 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:43:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:15.178420+0000 mon.b (mon.1) 337 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:15.581087+0000 mon.a (mon.0) 1633 : cluster [DBG] osdmap e240: 8 total, 8 up, 7 in 2024-02-13T00:43:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:15.585874+0000 mon.b (mon.1) 338 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:15.589413+0000 mon.a (mon.0) 1634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:16.310 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:43:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:43:16] "GET /metrics HTTP/1.1" 200 193238 "" "Prometheus/2.33.4" 2024-02-13T00:43:16.310 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:14.359725+0000 mgr.y (mgr.14544) 249 : cluster [DBG] pgmap v328: 420 pgs: 32 creating+peering, 2 active+clean+snaptrim, 64 unknown, 8 active+clean+snaptrim_wait, 314 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:43:16.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:15.178420+0000 mon.b (mon.1) 337 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:16.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:15.581087+0000 mon.a (mon.0) 1633 : cluster [DBG] osdmap e240: 8 total, 8 up, 7 in 2024-02-13T00:43:16.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:15.585874+0000 mon.b (mon.1) 338 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:16.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:15.589413+0000 mon.a (mon.0) 1634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:16.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:14.359725+0000 mgr.y (mgr.14544) 249 : cluster [DBG] pgmap v328: 420 pgs: 32 creating+peering, 2 active+clean+snaptrim, 64 unknown, 8 active+clean+snaptrim_wait, 314 active+clean; 9.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:43:16.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:15.178420+0000 mon.b (mon.1) 337 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:16.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:15.581087+0000 mon.a (mon.0) 1633 : cluster [DBG] osdmap e240: 8 total, 8 up, 7 in 2024-02-13T00:43:16.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:15.585874+0000 mon.b (mon.1) 338 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:16.311 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:15.589413+0000 mon.a (mon.0) 1634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:17.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:43:17.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:15.448920+0000 mgr.y (mgr.14544) 250 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:17.211 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:16.179754+0000 mon.b (mon.1) 339 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:17.212 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:16.340897+0000 mon.a (mon.0) 1635 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi162-83977-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi162-83977-34"}]': finished 2024-02-13T00:43:17.212 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:16.341068+0000 mon.a (mon.0) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:17.212 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:16.341162+0000 mon.a (mon.0) 1637 : cluster [DBG] osdmap e241: 8 total, 8 up, 7 in 2024-02-13T00:43:17.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:15.448920+0000 mgr.y (mgr.14544) 250 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:17.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:16.179754+0000 mon.b (mon.1) 339 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:17.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:16.340897+0000 mon.a (mon.0) 1635 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi162-83977-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi162-83977-34"}]': finished 2024-02-13T00:43:17.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:16.341068+0000 mon.a (mon.0) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:17.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:16.341162+0000 mon.a (mon.0) 1637 : cluster [DBG] osdmap e241: 8 total, 8 up, 7 in 2024-02-13T00:43:17.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:15.448920+0000 mgr.y (mgr.14544) 250 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:17.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:16.179754+0000 mon.b (mon.1) 339 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:17.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:16.340897+0000 mon.a (mon.0) 1635 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi162-83977-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi162-83977-34"}]': finished 2024-02-13T00:43:17.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:16.341068+0000 mon.a (mon.0) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:17.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:16.341162+0000 mon.a (mon.0) 1637 : cluster [DBG] osdmap e241: 8 total, 8 up, 7 in 2024-02-13T00:43:17.751 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:17.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:17.764 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:17.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:17.888 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:17.909 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:17.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:17.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:18.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:18.044 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:18.142 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:18.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:18.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:18.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:18.283 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:16.360344+0000 mgr.y (mgr.14544) 251 : cluster [DBG] pgmap v332: 364 pgs: 2 active+clean+snaptrim, 40 unknown, 8 active+clean+snaptrim_wait, 314 active+clean; 9.6 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:43:18.283 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:17.180984+0000 mon.b (mon.1) 340 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:18.283 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:17.352123+0000 mon.a (mon.0) 1638 : cluster [DBG] osdmap e242: 8 total, 8 up, 7 in 2024-02-13T00:43:18.283 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:17.366416+0000 mon.b (mon.1) 341 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:18.283 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:17.368009+0000 mon.a (mon.0) 1639 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:18.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:16.360344+0000 mgr.y (mgr.14544) 251 : cluster [DBG] pgmap v332: 364 pgs: 2 active+clean+snaptrim, 40 unknown, 8 active+clean+snaptrim_wait, 314 active+clean; 9.6 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:43:18.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:17.180984+0000 mon.b (mon.1) 340 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:18.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:17.352123+0000 mon.a (mon.0) 1638 : cluster [DBG] osdmap e242: 8 total, 8 up, 7 in 2024-02-13T00:43:18.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:17.366416+0000 mon.b (mon.1) 341 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:18.284 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:17.368009+0000 mon.a (mon.0) 1639 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:18.285 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:18.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:16.360344+0000 mgr.y (mgr.14544) 251 : cluster [DBG] pgmap v332: 364 pgs: 2 active+clean+snaptrim, 40 unknown, 8 active+clean+snaptrim_wait, 314 active+clean; 9.6 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:43:18.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:17.180984+0000 mon.b (mon.1) 340 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:18.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:17.352123+0000 mon.a (mon.0) 1638 : cluster [DBG] osdmap e242: 8 total, 8 up, 7 in 2024-02-13T00:43:18.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:17.366416+0000 mon.b (mon.1) 341 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:18.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:17.368009+0000 mon.a (mon.0) 1639 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:19.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:43:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:43:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:43:19.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.239+0000 7f570d2fc700 1 -- 172.21.15.162:0/2878608626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57080fd470 msgr2=0x7f57080f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:19.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.239+0000 7f570d2fc700 1 --2- 172.21.15.162:0/2878608626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57080fd470 0x7f57080f6ca0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 rx=0x7f56fc0047b0 tx=0x7f56fc009cb0).stop 2024-02-13T00:43:19.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.239+0000 7f570d2fc700 1 -- 172.21.15.162:0/2878608626 shutdown_connections 2024-02-13T00:43:19.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.239+0000 7f570d2fc700 1 --2- 172.21.15.162:0/2878608626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57080f7380 0x7f5708103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:43:19.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.239+0000 7f570d2fc700 1 --2- 172.21.15.162:0/2878608626 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57080fcb10 0x7f57080fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:19.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.239+0000 7f570d2fc700 1 --2- 172.21.15.162:0/2878608626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57080fd470 0x7f57080f6ca0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:19.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.239+0000 7f570d2fc700 1 -- 172.21.15.162:0/2878608626 >> 172.21.15.162:0/2878608626 conn(0x7f57080f2740 msgr2=0x7f57080f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:19.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.239+0000 7f570d2fc700 1 -- 172.21.15.162:0/2878608626 shutdown_connections 2024-02-13T00:43:19.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.239+0000 7f570d2fc700 1 -- 172.21.15.162:0/2878608626 wait complete. 2024-02-13T00:43:19.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.240+0000 7f570d2fc700 1 librados: starting msgr at 2024-02-13T00:43:19.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.240+0000 7f570d2fc700 1 librados: starting objecter 2024-02-13T00:43:19.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.240+0000 7f570d2fc700 1 Processor -- start 2024-02-13T00:43:19.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.240+0000 7f570d2fc700 1 -- start start 2024-02-13T00:43:19.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.240+0000 7f570d2fc700 1 librados: setting wanted keys 2024-02-13T00:43:19.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.240+0000 7f570d2fc700 1 librados: calling monclient init 2024-02-13T00:43:19.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.240+0000 7f570d2fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57080f7380 0x7f570810e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:19.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.240+0000 7f570d2fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57080fcb10 0x7f570810e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:19.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.240+0000 7f570d2fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57080fd470 0x7f5708108390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:19.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.240+0000 7f570d2fc700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5708108880 con 0x7f57080f7380 2024-02-13T00:43:19.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.240+0000 7f570d2fc700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57081089f0 con 0x7f57080fd470 2024-02-13T00:43:19.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.240+0000 7f570d2fc700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5708108bb0 con 0x7f57080fcb10 2024-02-13T00:43:19.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f57067fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57080fcb10 0x7f570810e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:19.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f57067fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57080fcb10 0x7f570810e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48540/0 (socket says 172.21.15.162:48540) 2024-02-13T00:43:19.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f5706ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57080f7380 0x7f570810e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:19.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f57067fc700 1 -- 172.21.15.162:0/499175595 learned_addr learned my addr 172.21.15.162:0/499175595 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:19.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f57067fc700 1 -- 172.21.15.162:0/499175595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57080fd470 msgr2=0x7f5708108390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:19.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f57067fc700 1 --2- 172.21.15.162:0/499175595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57080fd470 0x7f5708108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:43:19.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f57067fc700 1 -- 172.21.15.162:0/499175595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57080f7380 msgr2=0x7f570810e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:19.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f57067fc700 1 --2- 172.21.15.162:0/499175595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57080f7380 0x7f570810e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:19.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f57067fc700 1 -- 172.21.15.162:0/499175595 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f8005040 con 0x7f57080fcb10 2024-02-13T00:43:19.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f5706ffd700 1 --2- 172.21.15.162:0/499175595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57080f7380 0x7f570810e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:43:19.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f57067fc700 1 --2- 172.21.15.162:0/499175595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57080fcb10 0x7f570810e6e0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 rx=0x7f56fc0044e0 tx=0x7f56fc007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f56effff700 10 client.?.objecter ms_handle_connect 0x7f57080fcb10 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f56effff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f56effff700 1 -- 172.21.15.162:0/499175595 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc018070 con 0x7f57080fcb10 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f570d2fc700 1 -- 172.21.15.162:0/499175595 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc005040 con 0x7f57080fcb10 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f56effff700 1 -- 172.21.15.162:0/499175595 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56fc0093b0 con 0x7f57080fcb10 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f56effff700 1 -- 172.21.15.162:0/499175595 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc01de40 con 0x7f57080fcb10 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f570d2fc700 10 client.36923.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f570d2fc700 1 -- 172.21.15.162:0/499175595 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5708109050 con 0x7f57080fcb10 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.241+0000 7f570d2fc700 1 librados: init done 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.242+0000 7f570d2fc700 1 -- 172.21.15.162:0/499175595 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5708109240 con 0x7f57080fcb10 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.243+0000 7f56effff700 1 -- 172.21.15.162:0/499175595 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f56fc013ad0 con 0x7f57080fcb10 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.243+0000 7f56effff700 1 --2- 172.21.15.162:0/499175595 >> v2:172.21.15.162:6800/2866534053 conn(0x7f56f0062c90 0x7f56f0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.243+0000 7f56effff700 1 -- 172.21.15.162:0/499175595 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(243..243 src has 52..243) v4 ==== 9164+0+0 (secure 0 0 0) 0x7f56fc07eae0 con 0x7f57080fcb10 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.243+0000 7f56effff700 10 client.36923.objecter ms_dispatch 0x7f5708001550 osd_map(243..243 src has 52..243) v4 2024-02-13T00:43:19.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.243+0000 7f56effff700 3 client.36923.objecter handle_osd_map got epochs [243,243] > 0 2024-02-13T00:43:19.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.243+0000 7f56effff700 3 client.36923.objecter handle_osd_map decoding full epoch 243 2024-02-13T00:43:19.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.243+0000 7f5706ffd700 1 --2- 172.21.15.162:0/499175595 >> v2:172.21.15.162:6800/2866534053 conn(0x7f56f0062c90 0x7f56f0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:19.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.243+0000 7f56effff700 20 client.36923.objecter dump_active .. 0 homeless 2024-02-13T00:43:19.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.243+0000 7f56effff700 10 client.36923.objecter ms_handle_connect 0x7f56f0062c90 2024-02-13T00:43:19.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.243+0000 7f5706ffd700 1 --2- 172.21.15.162:0/499175595 >> v2:172.21.15.162:6800/2866534053 conn(0x7f56f0062c90 0x7f56f0065150 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 rx=0x7f56f8004c80 tx=0x7f56f800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:19.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.247+0000 7f56effff700 1 -- 172.21.15.162:0/499175595 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f56fc051b60 con 0x7f57080fcb10 2024-02-13T00:43:19.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:18.182119+0000 mon.b (mon.1) 342 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:19.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:18.350270+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:19.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:18.350372+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e243: 8 total, 8 up, 7 in 2024-02-13T00:43:19.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:18.351051+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:19.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:18.360583+0000 mon.b (mon.1) 343 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-25"}]: dispatch 2024-02-13T00:43:19.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:18.361277+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-25"}]: dispatch 2024-02-13T00:43:19.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:18.182119+0000 mon.b (mon.1) 342 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:19.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:18.350270+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:19.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:18.350372+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e243: 8 total, 8 up, 7 in 2024-02-13T00:43:19.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:18.351051+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:19.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:18.360583+0000 mon.b (mon.1) 343 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-25"}]: dispatch 2024-02-13T00:43:19.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:18.361277+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-25"}]: dispatch 2024-02-13T00:43:19.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.434+0000 7f570d2fc700 1 -- 172.21.15.162:0/499175595 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"} v 0) v1 -- 0x7f57080607b0 con 0x7f57080fcb10 2024-02-13T00:43:19.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.477+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/4154973282 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d640f6b00 msgr2=0x7f6d640f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:19.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.477+0000 7f6d6a2a3700 1 --2- 172.21.15.162:0/4154973282 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d640f6b00 0x7f6d640f6ee0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f6d540048a0 tx=0x7f6d54009cb0).stop 2024-02-13T00:43:19.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.477+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/4154973282 shutdown_connections 2024-02-13T00:43:19.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.477+0000 7f6d6a2a3700 1 --2- 172.21.15.162:0/4154973282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d64104660 0x7f6d64106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.477+0000 7f6d6a2a3700 1 --2- 172.21.15.162:0/4154973282 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d640f6b00 0x7f6d640f6ee0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.477+0000 7f6d6a2a3700 1 --2- 172.21.15.162:0/4154973282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d640f73d0 0x7f6d64104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.477+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/4154973282 >> 172.21.15.162:0/4154973282 conn(0x7f6d6406eb90 msgr2=0x7f6d640f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.477+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/4154973282 shutdown_connections 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.477+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/4154973282 wait complete. 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.478+0000 7f6d6a2a3700 1 librados: starting msgr at 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.478+0000 7f6d6a2a3700 1 librados: starting objecter 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.478+0000 7f6d6a2a3700 1 Processor -- start 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.478+0000 7f6d6a2a3700 1 -- start start 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.478+0000 7f6d6a2a3700 1 librados: setting wanted keys 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.478+0000 7f6d6a2a3700 1 librados: calling monclient init 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.478+0000 7f6d6a2a3700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d640f6b00 0x7f6d64110450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.478+0000 7f6d6a2a3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d640f73d0 0x7f6d64110940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.478+0000 7f6d6a2a3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d64104660 0x7f6d6410a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:19.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.478+0000 7f6d6a2a3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6410aae0 con 0x7f6d64104660 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d6a2a3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6410ac20 con 0x7f6d640f73d0 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d6a2a3700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6410ade0 con 0x7f6d640f6b00 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d63fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d640f6b00 0x7f6d64110450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d63fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d640f6b00 0x7f6d64110450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48580/0 (socket says 172.21.15.162:48580) 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d63fff700 1 -- 172.21.15.162:0/885884411 learned_addr learned my addr 172.21.15.162:0/885884411 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d637fe700 1 --2- 172.21.15.162:0/885884411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d640f73d0 0x7f6d64110940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d68840700 1 --2- 172.21.15.162:0/885884411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d64104660 0x7f6d6410a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d63fff700 1 -- 172.21.15.162:0/885884411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d640f73d0 msgr2=0x7f6d64110940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d63fff700 1 --2- 172.21.15.162:0/885884411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d640f73d0 0x7f6d64110940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d63fff700 1 -- 172.21.15.162:0/885884411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d64104660 msgr2=0x7f6d6410a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d63fff700 1 --2- 172.21.15.162:0/885884411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d64104660 0x7f6d6410a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d63fff700 1 -- 172.21.15.162:0/885884411 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d54005040 con 0x7f6d640f6b00 2024-02-13T00:43:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d63fff700 1 --2- 172.21.15.162:0/885884411 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d640f6b00 0x7f6d64110450 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7f6d54004710 tx=0x7f6d54007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d61775700 10 client.?.objecter ms_handle_connect 0x7f6d640f6b00 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d61775700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d61775700 1 -- 172.21.15.162:0/885884411 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5400e070 con 0x7f6d640f6b00 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d61775700 1 -- 172.21.15.162:0/885884411 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d540092e0 con 0x7f6d640f6b00 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/885884411 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6410b060 con 0x7f6d640f6b00 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d61775700 1 -- 172.21.15.162:0/885884411 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54018070 con 0x7f6d640f6b00 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d6a2a3700 10 client.36935.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/885884411 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d64117250 con 0x7f6d640f6b00 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.479+0000 7f6d6a2a3700 1 librados: init done 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.480+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/885884411 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d640475e0 con 0x7f6d640f6b00 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.481+0000 7f6d61775700 1 -- 172.21.15.162:0/885884411 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6d5401d420 con 0x7f6d640f6b00 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.481+0000 7f6d61775700 1 --2- 172.21.15.162:0/885884411 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d4c062c90 0x7f6d4c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.481+0000 7f6d61775700 1 -- 172.21.15.162:0/885884411 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(244..244 src has 52..244) v4 ==== 8503+0+0 (secure 0 0 0) 0x7f6d5407dc70 con 0x7f6d640f6b00 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.481+0000 7f6d61775700 10 client.36935.objecter ms_dispatch 0x7f6d64001550 osd_map(244..244 src has 52..244) v4 2024-02-13T00:43:19.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.481+0000 7f6d61775700 3 client.36935.objecter handle_osd_map got epochs [244,244] > 0 2024-02-13T00:43:19.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.481+0000 7f6d61775700 3 client.36935.objecter handle_osd_map decoding full epoch 244 2024-02-13T00:43:19.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.481+0000 7f6d637fe700 1 --2- 172.21.15.162:0/885884411 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d4c062c90 0x7f6d4c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:19.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.481+0000 7f6d61775700 20 client.36935.objecter dump_active .. 0 homeless 2024-02-13T00:43:19.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.481+0000 7f6d61775700 10 client.36935.objecter ms_handle_connect 0x7f6d4c062c90 2024-02-13T00:43:19.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.481+0000 7f6d637fe700 1 --2- 172.21.15.162:0/885884411 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d4c062c90 0x7f6d4c065150 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 rx=0x7f6d6410bd90 tx=0x7f6d5800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:19.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.485+0000 7f6d61775700 1 -- 172.21.15.162:0/885884411 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6d54051070 con 0x7f6d640f6b00 2024-02-13T00:43:19.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:18.182119+0000 mon.b (mon.1) 342 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:19.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:18.350270+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:19.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:18.350372+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e243: 8 total, 8 up, 7 in 2024-02-13T00:43:19.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:18.351051+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:19.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:18.360583+0000 mon.b (mon.1) 343 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-25"}]: dispatch 2024-02-13T00:43:19.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:18.361277+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-25"}]: dispatch 2024-02-13T00:43:19.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:19.677+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/885884411 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f6d6410bb60 con 0x7f6d640f6b00 2024-02-13T00:43:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:19.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:19.917 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:20.018 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:20.050 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:20.071 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:20.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:20.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:20.195 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:18.360868+0000 mgr.y (mgr.14544) 252 : cluster [DBG] pgmap v335: 292 pgs: 2 active+clean+snaptrim, 32 unknown, 8 active+clean+snaptrim_wait, 250 active+clean; 9.6 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 256 B/s wr, 2 op/s 2024-02-13T00:43:20.195 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:19.183919+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:20.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:19.332676+0000 mon.a (mon.0) 1644 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:20.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:19.354859+0000 mon.a (mon.0) 1645 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-83977-34"}]': finished 2024-02-13T00:43:20.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:19.354938+0000 mon.a (mon.0) 1646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-25"}]': finished 2024-02-13T00:43:20.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:19.354999+0000 mon.a (mon.0) 1647 : cluster [DBG] osdmap e244: 8 total, 8 up, 7 in 2024-02-13T00:43:20.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:19.355521+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:20.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:19.361567+0000 mon.b (mon.1) 345 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-25", "mode": "writeback"}]: dispatch 2024-02-13T00:43:20.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:19.362435+0000 mon.a (mon.0) 1649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-25", "mode": "writeback"}]: dispatch 2024-02-13T00:43:20.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:19.436224+0000 mon.c (mon.2) 223 : audit [INF] from='client.? 172.21.15.162:0/499175595' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]: dispatch 2024-02-13T00:43:20.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:19.626942+0000 mon.a (mon.0) 1650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]: dispatch 2024-02-13T00:43:20.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:19.679490+0000 mon.c (mon.2) 224 : audit [INF] from='client.? 172.21.15.162:0/885884411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:43:20.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:19.679859+0000 mon.a (mon.0) 1651 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:43:20.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:18.360868+0000 mgr.y (mgr.14544) 252 : cluster [DBG] pgmap v335: 292 pgs: 2 active+clean+snaptrim, 32 unknown, 8 active+clean+snaptrim_wait, 250 active+clean; 9.6 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 256 B/s wr, 2 op/s 2024-02-13T00:43:20.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:19.183919+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:20.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:19.332676+0000 mon.a (mon.0) 1644 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:20.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:19.354859+0000 mon.a (mon.0) 1645 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-83977-34"}]': finished 2024-02-13T00:43:20.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:19.354938+0000 mon.a (mon.0) 1646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-25"}]': finished 2024-02-13T00:43:20.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:19.354999+0000 mon.a (mon.0) 1647 : cluster [DBG] osdmap e244: 8 total, 8 up, 7 in 2024-02-13T00:43:20.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:19.355521+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:20.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:19.361567+0000 mon.b (mon.1) 345 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-25", "mode": "writeback"}]: dispatch 2024-02-13T00:43:20.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:19.362435+0000 mon.a (mon.0) 1649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-25", "mode": "writeback"}]: dispatch 2024-02-13T00:43:20.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:19.436224+0000 mon.c (mon.2) 223 : audit [INF] from='client.? 172.21.15.162:0/499175595' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]: dispatch 2024-02-13T00:43:20.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:19.626942+0000 mon.a (mon.0) 1650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]: dispatch 2024-02-13T00:43:20.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:19.679490+0000 mon.c (mon.2) 224 : audit [INF] from='client.? 172.21.15.162:0/885884411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:43:20.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:19.679859+0000 mon.a (mon.0) 1651 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:43:20.204 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:20.225 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:20.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:20.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:20.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.345+0000 7f6d61775700 1 -- 172.21.15.162:0/885884411 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v245) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6d54050c60 con 0x7f6d640f6b00 2024-02-13T00:43:20.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.346+0000 7f56effff700 1 -- 172.21.15.162:0/499175595 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]=0 set pool 6 pg_num to 42 v245) v1 ==== 141+0+0 (secure 0 0 0) 0x7f56fc051750 con 0x7f57080fcb10 2024-02-13T00:43:20.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 10 librados: watch_flush enter 2024-02-13T00:43:20.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 10 librados: watch_flush exit 2024-02-13T00:43:20.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 20 client.36935.objecter shutdown clearing up homeless session... 2024-02-13T00:43:20.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 10 client.36935.objecter successfully canceled tick 2024-02-13T00:43:20.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/885884411 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d4c062c90 msgr2=0x7f6d4c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:20.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 1 --2- 172.21.15.162:0/885884411 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d4c062c90 0x7f6d4c065150 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 rx=0x7f6d6410bd90 tx=0x7f6d5800b040).stop 2024-02-13T00:43:20.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/885884411 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d640f6b00 msgr2=0x7f6d64110450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:20.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 1 --2- 172.21.15.162:0/885884411 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d640f6b00 0x7f6d64110450 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7f6d54004710 tx=0x7f6d54007f90).stop 2024-02-13T00:43:20.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/885884411 shutdown_connections 2024-02-13T00:43:20.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 1 --2- 172.21.15.162:0/885884411 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d4c062c90 0x7f6d4c065150 unknown :-1 s=CLOSED pgs=2815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:20.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 1 --2- 172.21.15.162:0/885884411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d64104660 0x7f6d6410a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:20.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 1 --2- 172.21.15.162:0/885884411 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d640f6b00 0x7f6d64110450 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:20.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 1 --2- 172.21.15.162:0/885884411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d640f73d0 0x7f6d64110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:20.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.350+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/885884411 >> 172.21.15.162:0/885884411 conn(0x7f6d6406eb90 msgr2=0x7f6d6406fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:20.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.351+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/885884411 shutdown_connections 2024-02-13T00:43:20.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.351+0000 7f6d6a2a3700 1 -- 172.21.15.162:0/885884411 wait complete. 2024-02-13T00:43:20.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.351+0000 7f6d6a2a3700 1 librados: shutdown 2024-02-13T00:43:20.360 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:43:20.361 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:18.360868+0000 mgr.y (mgr.14544) 252 : cluster [DBG] pgmap v335: 292 pgs: 2 active+clean+snaptrim, 32 unknown, 8 active+clean+snaptrim_wait, 250 active+clean; 9.6 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 256 B/s wr, 2 op/s 2024-02-13T00:43:20.361 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:19.183919+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:20.361 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:19.332676+0000 mon.a (mon.0) 1644 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:20.361 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:19.354859+0000 mon.a (mon.0) 1645 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-83977-34"}]': finished 2024-02-13T00:43:20.361 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:19.354938+0000 mon.a (mon.0) 1646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-25"}]': finished 2024-02-13T00:43:20.361 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:19.354999+0000 mon.a (mon.0) 1647 : cluster [DBG] osdmap e244: 8 total, 8 up, 7 in 2024-02-13T00:43:20.362 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:19.355521+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-83977-34"}]: dispatch 2024-02-13T00:43:20.362 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:19.361567+0000 mon.b (mon.1) 345 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-25", "mode": "writeback"}]: dispatch 2024-02-13T00:43:20.362 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:19.362435+0000 mon.a (mon.0) 1649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-25", "mode": "writeback"}]: dispatch 2024-02-13T00:43:20.362 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:19.436224+0000 mon.c (mon.2) 223 : audit [INF] from='client.? 172.21.15.162:0/499175595' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]: dispatch 2024-02-13T00:43:20.362 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:19.626942+0000 mon.a (mon.0) 1650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]: dispatch 2024-02-13T00:43:20.362 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:19.679490+0000 mon.c (mon.2) 224 : audit [INF] from='client.? 172.21.15.162:0/885884411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:43:20.362 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:19.679859+0000 mon.a (mon.0) 1651 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:43:20.363 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:20.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.361+0000 7f570d2fc700 10 librados: watch_flush enter 2024-02-13T00:43:20.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 10 librados: watch_flush exit 2024-02-13T00:43:20.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 20 client.36923.objecter shutdown clearing up homeless session... 2024-02-13T00:43:20.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 10 client.36923.objecter successfully canceled tick 2024-02-13T00:43:20.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 1 -- 172.21.15.162:0/499175595 >> v2:172.21.15.162:6800/2866534053 conn(0x7f56f0062c90 msgr2=0x7f56f0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:20.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 1 --2- 172.21.15.162:0/499175595 >> v2:172.21.15.162:6800/2866534053 conn(0x7f56f0062c90 0x7f56f0065150 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 rx=0x7f56f8004c80 tx=0x7f56f800b040).stop 2024-02-13T00:43:20.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 1 -- 172.21.15.162:0/499175595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57080fcb10 msgr2=0x7f570810e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:20.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 1 --2- 172.21.15.162:0/499175595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57080fcb10 0x7f570810e6e0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 rx=0x7f56fc0044e0 tx=0x7f56fc007f90).stop 2024-02-13T00:43:20.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 1 -- 172.21.15.162:0/499175595 shutdown_connections 2024-02-13T00:43:20.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 1 --2- 172.21.15.162:0/499175595 >> v2:172.21.15.162:6800/2866534053 conn(0x7f56f0062c90 0x7f56f0065150 unknown :-1 s=CLOSED pgs=2813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:20.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 1 --2- 172.21.15.162:0/499175595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57080fd470 0x7f5708108390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:43:20.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 1 --2- 172.21.15.162:0/499175595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57080f7380 0x7f570810e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:20.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 1 --2- 172.21.15.162:0/499175595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57080fcb10 0x7f570810e6e0 secure :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 rx=0x7f56fc0044e0 tx=0x7f56fc007f90).stop 2024-02-13T00:43:20.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.363+0000 7f570d2fc700 1 -- 172.21.15.162:0/499175595 >> 172.21.15.162:0/499175595 conn(0x7f57080f2740 msgr2=0x7f5708061000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:20.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.364+0000 7f570d2fc700 1 -- 172.21.15.162:0/499175595 shutdown_connections 2024-02-13T00:43:20.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.364+0000 7f570d2fc700 1 -- 172.21.15.162:0/499175595 wait complete. 2024-02-13T00:43:20.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:20.364+0000 7f570d2fc700 1 librados: shutdown 2024-02-13T00:43:20.369 INFO:teuthology.orchestra.run.smithi162.stderr:set pool 6 pg_num to 42 2024-02-13T00:43:20.383 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:20.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:20.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:20.514 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:20.534 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:20.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:20.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:20.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:20.668 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:20.921 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:20.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:20.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:21.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.184958+0000 mon.b (mon.1) 346 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:20.334848+0000 mon.a (mon.0) 1652 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:20.334873+0000 mon.a (mon.0) 1653 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:43:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.340872+0000 mon.a (mon.0) 1654 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-83977-34"}]': finished 2024-02-13T00:43:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.340939+0000 mon.a (mon.0) 1655 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-25", "mode": "writeback"}]': finished 2024-02-13T00:43:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.341035+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]': finished 2024-02-13T00:43:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.341102+0000 mon.a (mon.0) 1657 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:43:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:20.341158+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e245: 8 total, 8 up, 7 in 2024-02-13T00:43:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.342676+0000 mon.a (mon.0) 1659 : audit [INF] from='client.? 172.21.15.162:0/1763366776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-83985-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:21.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.361845+0000 mon.c (mon.2) 225 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-13T00:43:21.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.363786+0000 mon.a (mon.0) 1660 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-13T00:43:21.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.374659+0000 mon.c (mon.2) 226 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:21.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.375058+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:21.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.375920+0000 mon.c (mon.2) 227 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:21.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.376220+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:21.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.376739+0000 mon.c (mon.2) 228 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-83977-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:21.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.377078+0000 mon.a (mon.0) 1663 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-83977-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:21.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.380142+0000 mon.b (mon.1) 347 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:21.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:20.380673+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.184958+0000 mon.b (mon.1) 346 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:20.334848+0000 mon.a (mon.0) 1652 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:20.334873+0000 mon.a (mon.0) 1653 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:43:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.340872+0000 mon.a (mon.0) 1654 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-83977-34"}]': finished 2024-02-13T00:43:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.340939+0000 mon.a (mon.0) 1655 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-25", "mode": "writeback"}]': finished 2024-02-13T00:43:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.341035+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]': finished 2024-02-13T00:43:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.341102+0000 mon.a (mon.0) 1657 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:43:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:20.341158+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e245: 8 total, 8 up, 7 in 2024-02-13T00:43:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.342676+0000 mon.a (mon.0) 1659 : audit [INF] from='client.? 172.21.15.162:0/1763366776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-83985-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.361845+0000 mon.c (mon.2) 225 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-13T00:43:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.363786+0000 mon.a (mon.0) 1660 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-13T00:43:21.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.374659+0000 mon.c (mon.2) 226 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:21.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.375058+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:21.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.375920+0000 mon.c (mon.2) 227 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:21.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.376220+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:21.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.376739+0000 mon.c (mon.2) 228 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-83977-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:21.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.377078+0000 mon.a (mon.0) 1663 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-83977-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:21.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.380142+0000 mon.b (mon.1) 347 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:21.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:20.380673+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:21.504 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:21.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:21.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.184958+0000 mon.b (mon.1) 346 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:21.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:20.334848+0000 mon.a (mon.0) 1652 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:21.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:20.334873+0000 mon.a (mon.0) 1653 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:43:21.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.340872+0000 mon.a (mon.0) 1654 : audit [INF] from='client.? 172.21.15.162:0/2722505542' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-83977-34"}]': finished 2024-02-13T00:43:21.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.340939+0000 mon.a (mon.0) 1655 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-25", "mode": "writeback"}]': finished 2024-02-13T00:43:21.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.341035+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]': finished 2024-02-13T00:43:21.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.341102+0000 mon.a (mon.0) 1657 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:43:21.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:20.341158+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e245: 8 total, 8 up, 7 in 2024-02-13T00:43:21.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.342676+0000 mon.a (mon.0) 1659 : audit [INF] from='client.? 172.21.15.162:0/1763366776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-83985-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:21.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.361845+0000 mon.c (mon.2) 225 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-13T00:43:21.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.363786+0000 mon.a (mon.0) 1660 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-13T00:43:21.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.374659+0000 mon.c (mon.2) 226 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:21.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.375058+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:21.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.375920+0000 mon.c (mon.2) 227 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:21.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.376220+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:21.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.376739+0000 mon.c (mon.2) 228 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-83977-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:21.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.377078+0000 mon.a (mon.0) 1663 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-83977-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:21.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.380142+0000 mon.b (mon.1) 347 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:21.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:20.380673+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:21.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:21.640 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:21.661 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:21.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:21.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:21.792 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:21.830 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:21.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:21.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:21.965 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:21.987 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:21.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:21.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:22.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:22.117 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:22.239 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:22.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:22.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:20.361393+0000 mgr.y (mgr.14544) 253 : cluster [DBG] pgmap v338: 292 pgs: 2 active+clean+snaptrim, 32 unknown, 8 active+clean+snaptrim_wait, 250 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:21.186135+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:21.341566+0000 mon.a (mon.0) 1665 : audit [INF] from='client.? 172.21.15.162:0/1763366776' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-83985-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:21.341722+0000 mon.a (mon.0) 1666 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]': finished 2024-02-13T00:43:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:21.341805+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-83977-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:21.341876+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:21.341977+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e246: 8 total, 8 up, 7 in 2024-02-13T00:43:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:21.349586+0000 mon.b (mon.1) 349 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25"}]: dispatch 2024-02-13T00:43:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:21.350547+0000 mon.c (mon.2) 229 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-83977-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:21.351887+0000 mon.a (mon.0) 1670 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25"}]: dispatch 2024-02-13T00:43:22.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:21.362690+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-83977-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:22.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:20.361393+0000 mgr.y (mgr.14544) 253 : cluster [DBG] pgmap v338: 292 pgs: 2 active+clean+snaptrim, 32 unknown, 8 active+clean+snaptrim_wait, 250 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:22.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:21.186135+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:22.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:21.341566+0000 mon.a (mon.0) 1665 : audit [INF] from='client.? 172.21.15.162:0/1763366776' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-83985-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:22.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:21.341722+0000 mon.a (mon.0) 1666 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]': finished 2024-02-13T00:43:22.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:21.341805+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-83977-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:22.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:21.341876+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:22.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:21.341977+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e246: 8 total, 8 up, 7 in 2024-02-13T00:43:22.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:21.349586+0000 mon.b (mon.1) 349 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25"}]: dispatch 2024-02-13T00:43:22.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:21.350547+0000 mon.c (mon.2) 229 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-83977-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:22.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:21.351887+0000 mon.a (mon.0) 1670 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25"}]: dispatch 2024-02-13T00:43:22.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:21.362690+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-83977-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:22.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:20.361393+0000 mgr.y (mgr.14544) 253 : cluster [DBG] pgmap v338: 292 pgs: 2 active+clean+snaptrim, 32 unknown, 8 active+clean+snaptrim_wait, 250 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:22.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:21.186135+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:22.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:21.341566+0000 mon.a (mon.0) 1665 : audit [INF] from='client.? 172.21.15.162:0/1763366776' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-83985-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:22.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:21.341722+0000 mon.a (mon.0) 1666 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]': finished 2024-02-13T00:43:22.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:21.341805+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-83977-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:22.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:21.341876+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:22.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:21.341977+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e246: 8 total, 8 up, 7 in 2024-02-13T00:43:22.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:21.349586+0000 mon.b (mon.1) 349 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25"}]: dispatch 2024-02-13T00:43:22.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:21.350547+0000 mon.c (mon.2) 229 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-83977-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:22.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:21.351887+0000 mon.a (mon.0) 1670 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25"}]: dispatch 2024-02-13T00:43:22.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:21.362690+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-83977-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:22.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:22.382 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:22.402 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:22.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:22.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:22.534 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:22.553 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:22.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:22.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:22.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:22.687 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:22.884 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:22.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:22.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:22.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:23.015 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:23.111 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:22.187179+0000 mon.b (mon.1) 350 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:23.111 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:22.338552+0000 mon.a (mon.0) 1672 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:23.111 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:22.345777+0000 mon.a (mon.0) 1673 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25"}]': finished 2024-02-13T00:43:23.111 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:22.368785+0000 mon.c (mon.2) 230 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-13T00:43:23.111 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:22.590602+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e247: 8 total, 8 up, 7 in 2024-02-13T00:43:23.111 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:22.592206+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-13T00:43:23.111 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:22.187179+0000 mon.b (mon.1) 350 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:23.111 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:22.338552+0000 mon.a (mon.0) 1672 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:23.111 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:22.345777+0000 mon.a (mon.0) 1673 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25"}]': finished 2024-02-13T00:43:23.112 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:22.368785+0000 mon.c (mon.2) 230 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-13T00:43:23.112 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:22.590602+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e247: 8 total, 8 up, 7 in 2024-02-13T00:43:23.112 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:22.592206+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-13T00:43:23.113 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:23.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:23.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:23.246 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:23.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:43:23.523 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:23.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:23.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:22.187179+0000 mon.b (mon.1) 350 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:23.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:22.338552+0000 mon.a (mon.0) 1672 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:23.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:22.345777+0000 mon.a (mon.0) 1673 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-25"}]': finished 2024-02-13T00:43:23.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:22.368785+0000 mon.c (mon.2) 230 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-13T00:43:23.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:22.590602+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e247: 8 total, 8 up, 7 in 2024-02-13T00:43:23.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:22.592206+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-13T00:43:23.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:23.658 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:23.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:23.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:23.781 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:43:23.781 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:43:23.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:23.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.629+0000 7f475fade700 1 -- 172.21.15.162:0/571461838 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47580fd490 msgr2=0x7f47580f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.629+0000 7f475fade700 1 --2- 172.21.15.162:0/571461838 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47580fd490 0x7f47580f6cc0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7f4748004870 tx=0x7f4748009cb0).stop 2024-02-13T00:43:23.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.629+0000 7f475fade700 1 -- 172.21.15.162:0/571461838 shutdown_connections 2024-02-13T00:43:23.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.629+0000 7f475fade700 1 --2- 172.21.15.162:0/571461838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47580f73a0 0x7f47581039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:23.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.629+0000 7f475fade700 1 --2- 172.21.15.162:0/571461838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47580fcb30 0x7f47580fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:23.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.629+0000 7f475fade700 1 --2- 172.21.15.162:0/571461838 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47580fd490 0x7f47580f6cc0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:23.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.629+0000 7f475fade700 1 -- 172.21.15.162:0/571461838 >> 172.21.15.162:0/571461838 conn(0x7f475806eb90 msgr2=0x7f47580f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:23.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.629+0000 7f475fade700 1 -- 172.21.15.162:0/571461838 shutdown_connections 2024-02-13T00:43:23.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.629+0000 7f475fade700 1 -- 172.21.15.162:0/571461838 wait complete. 2024-02-13T00:43:23.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475fade700 1 librados: starting msgr at 2024-02-13T00:43:23.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475fade700 1 librados: starting objecter 2024-02-13T00:43:23.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475fade700 1 Processor -- start 2024-02-13T00:43:23.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475fade700 1 -- start start 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475fade700 1 librados: setting wanted keys 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475fade700 1 librados: calling monclient init 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475fade700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47580f73a0 0x7f475810e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475fade700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47580fcb30 0x7f475810e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475fade700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47580fd490 0x7f47581081f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475fade700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47581086e0 con 0x7f47580f73a0 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475fade700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4758108850 con 0x7f47580fd490 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475fade700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4758108a10 con 0x7f47580fcb30 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475d079700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47580fcb30 0x7f475810e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475d079700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47580fcb30 0x7f475810e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48648/0 (socket says 172.21.15.162:48648) 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.630+0000 7f475d079700 1 -- 172.21.15.162:0/3678521902 learned_addr learned my addr 172.21.15.162:0/3678521902 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f475e07b700 1 --2- 172.21.15.162:0/3678521902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47580fd490 0x7f47581081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f475d079700 1 -- 172.21.15.162:0/3678521902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47580fd490 msgr2=0x7f47581081f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f475d079700 1 --2- 172.21.15.162:0/3678521902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47580fd490 0x7f47581081f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f475d079700 1 -- 172.21.15.162:0/3678521902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47580f73a0 msgr2=0x7f475810e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f475d079700 1 --2- 172.21.15.162:0/3678521902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47580f73a0 0x7f475810e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f475d079700 1 -- 172.21.15.162:0/3678521902 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4748005040 con 0x7f47580fcb30 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f475d079700 1 --2- 172.21.15.162:0/3678521902 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47580fcb30 0x7f475810e540 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7f4748009eb0 tx=0x7f474800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f474ef78700 10 client.?.objecter ms_handle_connect 0x7f47580fcb30 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f474ef78700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474800e070 con 0x7f47580fcb30 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4758108c90 con 0x7f47580fcb30 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4748013da0 con 0x7f47580fcb30 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748019070 con 0x7f47580fcb30 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f475fade700 10 client.36953.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4758114e80 con 0x7f47580fcb30 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.631+0000 7f475fade700 1 librados: init done 2024-02-13T00:43:23.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.632+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4758105fe0 con 0x7f47580fcb30 2024-02-13T00:43:23.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.633+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f474801e420 con 0x7f47580fcb30 2024-02-13T00:43:23.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f474ef78700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4744062c90 0x7f4744065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:23.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(248..248 src has 52..248) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f474807e6d0 con 0x7f47580fcb30 2024-02-13T00:43:23.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 osd_map(248..248 src has 52..248) v4 2024-02-13T00:43:23.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f474ef78700 3 client.36953.objecter handle_osd_map got epochs [248,248] > 0 2024-02-13T00:43:23.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f474ef78700 3 client.36953.objecter handle_osd_map decoding full epoch 248 2024-02-13T00:43:23.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475d87a700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4744062c90 0x7f4744065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:23.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f474ef78700 20 client.36953.objecter dump_active .. 0 homeless 2024-02-13T00:43:23.891 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:43:23.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v248) v1 ==== 54+0+16 (secure 0 0 0) 0x7f474807eb70 con 0x7f47580fcb30 2024-02-13T00:43:23.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f474ef78700 10 client.36953.objecter ms_handle_connect 0x7f4744062c90 2024-02-13T00:43:23.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475d87a700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4744062c90 0x7f4744065150 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f4754004a90 tx=0x7f4754007b40).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:23.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475fade700 10 client.36953.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:23.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 1 2024-02-13T00:43:23.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6801/2103743338 conn(0x7f47580ffdb0 0x7f4758100190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:23.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f47581096d0 osd=0 3 2024-02-13T00:43:23.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f47581096d0 osd=0 3 2024-02-13T00:43:23.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f47581096d0 osd=0 3 2024-02-13T00:43:23.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 1 2024-02-13T00:43:23.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f47581096d0 osd=0 3 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 0 1 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f47581096d0 osd=0 4 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475fade700 10 client.36953.objecter _send_command 1 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4758100680 con 0x7f47580ffdb0 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.634+0000 7f475e07b700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6801/2103743338 conn(0x7f47580ffdb0 0x7f4758100190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.635+0000 7f474ef78700 10 client.36953.objecter ms_handle_connect 0x7f47580ffdb0 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.635+0000 7f475e07b700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6801/2103743338 conn(0x7f47580ffdb0 0x7f4758100190 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.636+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4758100680 con 0x7f47580ffdb0 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.636+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.636+0000 7f474ef78700 10 client.36953.objecter _finish_command 1 = system:0 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.636+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f47581096d0 osd=0 4 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.636+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 0 1 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.653+0000 7f475fade700 10 client.36953.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.653+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 2 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.653+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f47581096d0 osd=0 3 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.653+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f47581096d0 osd=0 3 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.653+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f47581096d0 osd=0 3 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.653+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 2 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.653+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f47581096d0 osd=0 3 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.653+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 0 2 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.653+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f47581096d0 osd=0 4 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.653+0000 7f475fade700 10 client.36953.objecter _send_command 2 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.653+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47581007f0 con 0x7f47580ffdb0 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f47581007f0 con 0x7f47580ffdb0 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f474ef78700 10 client.36953.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f47581096d0 osd=0 4 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 0 2 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f475fade700 10 client.36953.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:23.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 3 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6805/150599780 conn(0x7f4758028590 0x7f4758117020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758061a30 osd=1 3 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758061a30 osd=1 3 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758061a30 osd=1 3 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 3 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f4758061a30 osd=1 3 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 1 3 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758061a30 osd=1 4 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f475fade700 10 client.36953.objecter _send_command 3 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.656+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4758061ba0 con 0x7f4758028590 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.657+0000 7f475d87a700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6805/150599780 conn(0x7f4758028590 0x7f4758117020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.657+0000 7f474ef78700 10 client.36953.objecter ms_handle_connect 0x7f4758028590 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.657+0000 7f475d87a700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6805/150599780 conn(0x7f4758028590 0x7f4758117020 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:23.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.658+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4758061ba0 con 0x7f4758028590 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.658+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.658+0000 7f474ef78700 10 client.36953.objecter _finish_command 3 = system:0 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.658+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f4758061a30 osd=1 4 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.658+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 1 3 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.674+0000 7f475fade700 10 client.36953.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.674+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 4 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.674+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758061a30 osd=1 3 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.674+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758061a30 osd=1 3 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.674+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758061a30 osd=1 3 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.674+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 4 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.674+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f4758061a30 osd=1 3 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.674+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 1 4 2024-02-13T00:43:23.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.674+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758061a30 osd=1 4 2024-02-13T00:43:23.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.674+0000 7f475fade700 10 client.36953.objecter _send_command 4 2024-02-13T00:43:23.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.674+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47580fa780 con 0x7f4758028590 2024-02-13T00:43:23.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f47580fa780 con 0x7f4758028590 2024-02-13T00:43:23.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:23.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f474ef78700 10 client.36953.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.897 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:43:23.897 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f4758061a30 osd=1 4 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 1 4 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475fade700 10 client.36953.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 5 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4758117b90 0x7f4758117f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758117510 osd=2 3 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758117510 osd=2 3 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758117510 osd=2 3 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 5 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f4758117510 osd=2 3 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 2 5 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758117510 osd=2 4 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475fade700 10 client.36953.objecter _send_command 5 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4758121180 con 0x7f4758117b90 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475d079700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4758117b90 0x7f4758117f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:23.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f474ef78700 10 client.36953.objecter ms_handle_connect 0x7f4758117b90 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.677+0000 7f475d079700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4758117b90 0x7f4758117f90 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.678+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4758121180 con 0x7f4758117b90 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.678+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.678+0000 7f474ef78700 10 client.36953.objecter _finish_command 5 = system:0 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.678+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f4758117510 osd=2 4 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.678+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 2 5 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.699+0000 7f475fade700 10 client.36953.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.699+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 6 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.699+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758117510 osd=2 3 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.699+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758117510 osd=2 3 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.699+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758117510 osd=2 3 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.699+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 6 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.699+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f4758117510 osd=2 3 2024-02-13T00:43:23.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.699+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 2 6 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.699+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758117510 osd=2 4 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.699+0000 7f475fade700 10 client.36953.objecter _send_command 6 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.699+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4758059400 con 0x7f4758117b90 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4758059400 con 0x7f4758117b90 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f474ef78700 10 client.36953.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f4758117510 osd=2 4 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 2 6 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475fade700 10 client.36953.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 7 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6813/604971011 conn(0x7f475811fba0 0x7f475811ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758027be0 osd=3 3 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758027be0 osd=3 3 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758027be0 osd=3 3 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 7 2024-02-13T00:43:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f4758027be0 osd=3 3 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 3 7 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758027be0 osd=3 4 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475fade700 10 client.36953.objecter _send_command 7 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4758028290 con 0x7f475811fba0 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.701+0000 7f475e07b700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6813/604971011 conn(0x7f475811fba0 0x7f475811ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.702+0000 7f474ef78700 10 client.36953.objecter ms_handle_connect 0x7f475811fba0 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.702+0000 7f475e07b700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6813/604971011 conn(0x7f475811fba0 0x7f475811ffe0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.702+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4758028290 con 0x7f475811fba0 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.702+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.702+0000 7f474ef78700 10 client.36953.objecter _finish_command 7 = system:0 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.702+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f4758027be0 osd=3 4 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.702+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 3 7 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.717+0000 7f475fade700 10 client.36953.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.717+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 8 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.717+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758027be0 osd=3 3 2024-02-13T00:43:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.717+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758027be0 osd=3 3 2024-02-13T00:43:23.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.717+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758027be0 osd=3 3 2024-02-13T00:43:23.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.717+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 8 2024-02-13T00:43:23.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.717+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f4758027be0 osd=3 3 2024-02-13T00:43:23.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.717+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 3 8 2024-02-13T00:43:23.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.717+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758027be0 osd=3 4 2024-02-13T00:43:23.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.717+0000 7f475fade700 10 client.36953.objecter _send_command 8 2024-02-13T00:43:23.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.717+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4758120ca0 con 0x7f475811fba0 2024-02-13T00:43:23.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4758120ca0 con 0x7f475811fba0 2024-02-13T00:43:23.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:23.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f474ef78700 10 client.36953.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f4758027be0 osd=3 4 2024-02-13T00:43:23.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 3 8 2024-02-13T00:43:23.903 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:43:23.903 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:43:23.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f475fade700 10 client.36953.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:23.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 9 2024-02-13T00:43:23.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4758123300 0x7f47581256f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:23.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f47581204d0 osd=4 3 2024-02-13T00:43:23.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f47581204d0 osd=4 3 2024-02-13T00:43:23.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f47581204d0 osd=4 3 2024-02-13T00:43:23.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 9 2024-02-13T00:43:23.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f47581204d0 osd=4 3 2024-02-13T00:43:23.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 4 9 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f47581204d0 osd=4 4 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f475fade700 10 client.36953.objecter _send_command 9 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.719+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4758120b50 con 0x7f4758123300 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.720+0000 7f475d87a700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4758123300 0x7f47581256f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.720+0000 7f474ef78700 10 client.36953.objecter ms_handle_connect 0x7f4758123300 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.720+0000 7f475d87a700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4758123300 0x7f47581256f0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.721+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4758120b50 con 0x7f4758123300 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.721+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.721+0000 7f474ef78700 10 client.36953.objecter _finish_command 9 = system:0 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.721+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f47581204d0 osd=4 4 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.721+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 4 9 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.737+0000 7f475fade700 10 client.36953.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.737+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 10 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.737+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f47581204d0 osd=4 3 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.737+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f47581204d0 osd=4 3 2024-02-13T00:43:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.737+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f47581204d0 osd=4 3 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.737+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 10 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.737+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f47581204d0 osd=4 3 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.737+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 4 10 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.737+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f47581204d0 osd=4 4 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.737+0000 7f475fade700 10 client.36953.objecter _send_command 10 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.737+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4758120b50 con 0x7f4758123300 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4758120b50 con 0x7f4758123300 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f474ef78700 10 client.36953.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f47581204d0 osd=4 4 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 4 10 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f475fade700 10 client.36953.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 11 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6804/2212296748 conn(0x7f47581260f0 0x7f4758126530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:23.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758120b50 osd=5 3 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758120b50 osd=5 3 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758120b50 osd=5 3 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 11 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f4758120b50 osd=5 3 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 5 11 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758120b50 osd=5 4 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f475fade700 10 client.36953.objecter _send_command 11 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.739+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4758128190 con 0x7f47581260f0 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.740+0000 7f475d079700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6804/2212296748 conn(0x7f47581260f0 0x7f4758126530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.740+0000 7f474ef78700 10 client.36953.objecter ms_handle_connect 0x7f47581260f0 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.740+0000 7f475d079700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6804/2212296748 conn(0x7f47581260f0 0x7f4758126530 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.741+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4758128190 con 0x7f47581260f0 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.741+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.741+0000 7f474ef78700 10 client.36953.objecter _finish_command 11 = system:0 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.741+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f4758120b50 osd=5 4 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.741+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 5 11 2024-02-13T00:43:23.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.758+0000 7f475fade700 10 client.36953.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.758+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 12 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.758+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758120b50 osd=5 3 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.758+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758120b50 osd=5 3 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.758+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758120b50 osd=5 3 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.758+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 12 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.758+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f4758120b50 osd=5 3 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.758+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 5 12 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.758+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758120b50 osd=5 4 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.758+0000 7f475fade700 10 client.36953.objecter _send_command 12 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.758+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4758128190 con 0x7f47581260f0 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.759+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4758128190 con 0x7f47581260f0 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.759+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.759+0000 7f474ef78700 10 client.36953.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.759+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f4758120b50 osd=5 4 2024-02-13T00:43:23.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.759+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 5 12 2024-02-13T00:43:23.908 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:43:23.908 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:43:23.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475fade700 10 client.36953.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:23.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 13 2024-02-13T00:43:23.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6808/1506640341 conn(0x7f475812a8d0 0x7f475812ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:23.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758128190 osd=6 3 2024-02-13T00:43:23.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758128190 osd=6 3 2024-02-13T00:43:23.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758128190 osd=6 3 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 13 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f4758128190 osd=6 3 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 6 13 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758128190 osd=6 4 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475fade700 10 client.36953.objecter _send_command 13 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f475812e710 con 0x7f475812a8d0 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475e07b700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6808/1506640341 conn(0x7f475812a8d0 0x7f475812ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f474ef78700 10 client.36953.objecter ms_handle_connect 0x7f475812a8d0 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.760+0000 7f475e07b700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6808/1506640341 conn(0x7f475812a8d0 0x7f475812ad10 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.761+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f475812e710 con 0x7f475812a8d0 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.761+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.761+0000 7f474ef78700 10 client.36953.objecter _finish_command 13 = system:0 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.761+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f4758128190 osd=6 4 2024-02-13T00:43:23.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.761+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 6 13 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.779+0000 7f475fade700 10 client.36953.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.779+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 14 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.779+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758128190 osd=6 3 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.779+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758128190 osd=6 3 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.779+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758128190 osd=6 3 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.779+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 14 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.779+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f4758128190 osd=6 3 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.779+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 6 14 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.779+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758128190 osd=6 4 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.779+0000 7f475fade700 10 client.36953.objecter _send_command 14 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.779+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4758126b40 con 0x7f475812a8d0 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4758126b40 con 0x7f475812a8d0 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f474ef78700 10 client.36953.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f4758128190 osd=6 4 2024-02-13T00:43:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 6 14 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f475fade700 10 client.36953.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 15 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4758127250 0x7f47581276b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758126b40 osd=7 3 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758126b40 osd=7 3 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758126b40 osd=7 3 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 15 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f4758126b40 osd=7 3 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 7 15 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758126b40 osd=7 4 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f475fade700 10 client.36953.objecter _send_command 15 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.781+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4758127d20 con 0x7f4758127250 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.782+0000 7f475d87a700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4758127250 0x7f47581276b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.782+0000 7f474ef78700 10 client.36953.objecter ms_handle_connect 0x7f4758127250 2024-02-13T00:43:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.782+0000 7f475d87a700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4758127250 0x7f47581276b0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:23.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.783+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4758127d20 con 0x7f4758127250 2024-02-13T00:43:23.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.783+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:43:23.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.783+0000 7f474ef78700 10 client.36953.objecter _finish_command 15 = system:0 2024-02-13T00:43:23.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.783+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f4758126b40 osd=7 4 2024-02-13T00:43:23.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.783+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 7 15 2024-02-13T00:43:23.912 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:43:23.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.797+0000 7f475fade700 10 client.36953.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:23.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.797+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign -1 16 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.797+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758126b40 osd=7 3 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.797+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758126b40 osd=7 3 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.797+0000 7f475fade700 20 client.36953.objecter _get_session s=0x7f4758126b40 osd=7 3 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.797+0000 7f475fade700 15 client.36953.objecter _session_command_op_remove -1 16 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.797+0000 7f475fade700 20 client.36953.objecter get_session s=0x7f4758126b40 osd=7 3 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.797+0000 7f475fade700 15 client.36953.objecter _session_command_op_assign 7 16 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.797+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758126b40 osd=7 4 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.797+0000 7f475fade700 10 client.36953.objecter _send_command 16 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.797+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4758127d20 con 0x7f4758127250 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.799+0000 7f474ef78700 1 -- 172.21.15.162:0/3678521902 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4758127d20 con 0x7f4758127250 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.799+0000 7f474ef78700 10 client.36953.objecter ms_dispatch 0x7f4758001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.799+0000 7f474ef78700 10 client.36953.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.799+0000 7f474ef78700 20 client.36953.objecter put_session s=0x7f4758126b40 osd=7 4 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.799+0000 7f474ef78700 15 client.36953.objecter _session_command_op_remove 7 16 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 10 librados: watch_flush enter 2024-02-13T00:43:23.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 10 librados: watch_flush exit 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 10 client.36953.objecter close_session for osd.0 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6801/2103743338 conn(0x7f47580ffdb0 msgr2=0x7f4758100190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6801/2103743338 conn(0x7f47580ffdb0 0x7f4758100190 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f47581096d0 osd=0 1 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 10 client.36953.objecter close_session for osd.1 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6805/150599780 conn(0x7f4758028590 msgr2=0x7f4758117020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6805/150599780 conn(0x7f4758028590 0x7f4758117020 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758061a30 osd=1 1 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 10 client.36953.objecter close_session for osd.2 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4758117b90 msgr2=0x7f4758117f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4758117b90 0x7f4758117f90 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758117510 osd=2 1 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 10 client.36953.objecter close_session for osd.3 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6813/604971011 conn(0x7f475811fba0 msgr2=0x7f475811ffe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6813/604971011 conn(0x7f475811fba0 0x7f475811ffe0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758027be0 osd=3 1 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 10 client.36953.objecter close_session for osd.4 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4758123300 msgr2=0x7f47581256f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475e07b700 1 -- 172.21.15.162:0/3678521902 reap_dead start 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4758123300 0x7f47581256f0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475e07b700 1 -- 172.21.15.162:0/3678521902 reap_dead start 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f47581204d0 osd=4 1 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 10 client.36953.objecter close_session for osd.5 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6804/2212296748 conn(0x7f47581260f0 msgr2=0x7f4758126530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6804/2212296748 conn(0x7f47581260f0 0x7f4758126530 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758120b50 osd=5 1 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 10 client.36953.objecter close_session for osd.6 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6808/1506640341 conn(0x7f475812a8d0 msgr2=0x7f475812ad10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6808/1506640341 conn(0x7f475812a8d0 0x7f475812ad10 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758128190 osd=6 1 2024-02-13T00:43:23.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 10 client.36953.objecter close_session for osd.7 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4758127250 msgr2=0x7f47581276b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4758127250 0x7f47581276b0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 20 client.36953.objecter put_session s=0x7f4758126b40 osd=7 1 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 20 client.36953.objecter shutdown clearing up homeless session... 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 10 client.36953.objecter successfully canceled tick 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4744062c90 msgr2=0x7f4744065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4744062c90 0x7f4744065150 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f4754004a90 tx=0x7f4754007b40).stop 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47580fcb30 msgr2=0x7f475810e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 --2- 172.21.15.162:0/3678521902 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47580fcb30 0x7f475810e540 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7f4748009eb0 tx=0x7f474800b040).stop 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475e07b700 1 -- 172.21.15.162:0/3678521902 reap_dead start 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 shutdown_connections 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 >> 172.21.15.162:0/3678521902 conn(0x7f475806eb90 msgr2=0x7f4758060ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:23.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 shutdown_connections 2024-02-13T00:43:23.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 -- 172.21.15.162:0/3678521902 wait complete. 2024-02-13T00:43:23.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:23.800+0000 7f475fade700 1 librados: shutdown 2024-02-13T00:43:23.917 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.917 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.917 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.917 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.917 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.917 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.917 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.917 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:43:23.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:23.918 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:23.918 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:23.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:24.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:24.048 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:24.066 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:24.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:24.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:24.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:24.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:22.361937+0000 mgr.y (mgr.14544) 254 : cluster [DBG] pgmap v341: 270 pgs: 2 active+clean+snaptrim, 10 unknown, 8 active+clean+snaptrim_wait, 250 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:24.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:23.188986+0000 mon.b (mon.1) 351 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:24.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:23.355038+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-83977-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-83977-35"}]': finished 2024-02-13T00:43:24.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:23.355164+0000 mon.a (mon.0) 1677 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]': finished 2024-02-13T00:43:24.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:23.355249+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e248: 8 total, 8 up, 7 in 2024-02-13T00:43:24.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:23.367214+0000 mon.c (mon.2) 231 : audit [INF] from='client.? 172.21.15.162:0/394404351' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-83985-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:24.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:23.367667+0000 mon.a (mon.0) 1679 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-83985-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:24.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:23.634679+0000 mon.c (mon.2) 232 : audit [DBG] from='client.? 172.21.15.162:0/3678521902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:24.295 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:22.361937+0000 mgr.y (mgr.14544) 254 : cluster [DBG] pgmap v341: 270 pgs: 2 active+clean+snaptrim, 10 unknown, 8 active+clean+snaptrim_wait, 250 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:24.296 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:23.188986+0000 mon.b (mon.1) 351 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:24.296 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:23.355038+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-83977-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-83977-35"}]': finished 2024-02-13T00:43:24.296 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:23.355164+0000 mon.a (mon.0) 1677 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]': finished 2024-02-13T00:43:24.296 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:23.355249+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e248: 8 total, 8 up, 7 in 2024-02-13T00:43:24.296 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:23.367214+0000 mon.c (mon.2) 231 : audit [INF] from='client.? 172.21.15.162:0/394404351' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-83985-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:24.296 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:23.367667+0000 mon.a (mon.0) 1679 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-83985-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:24.296 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:23.634679+0000 mon.c (mon.2) 232 : audit [DBG] from='client.? 172.21.15.162:0/3678521902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:24.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:22.361937+0000 mgr.y (mgr.14544) 254 : cluster [DBG] pgmap v341: 270 pgs: 2 active+clean+snaptrim, 10 unknown, 8 active+clean+snaptrim_wait, 250 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:24.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:23.188986+0000 mon.b (mon.1) 351 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:24.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:23.355038+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-83977-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-83977-35"}]': finished 2024-02-13T00:43:24.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:23.355164+0000 mon.a (mon.0) 1677 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]': finished 2024-02-13T00:43:24.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:23.355249+0000 mon.a (mon.0) 1678 : cluster [DBG] osdmap e248: 8 total, 8 up, 7 in 2024-02-13T00:43:24.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:23.367214+0000 mon.c (mon.2) 231 : audit [INF] from='client.? 172.21.15.162:0/394404351' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-83985-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:24.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:23.367667+0000 mon.a (mon.0) 1679 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-83985-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:24.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:23.634679+0000 mon.c (mon.2) 232 : audit [DBG] from='client.? 172.21.15.162:0/3678521902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:24.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:24.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:24.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:24.425 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:24.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-13T00:43:24.873 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:24.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:24.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:24.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:25.008 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:25.029 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:25.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:25.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:25.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:25.157 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:24.190192+0000 mon.b (mon.1) 352 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:25.158 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:24.354278+0000 mon.a (mon.0) 1680 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-83985-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:25.158 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:24.354376+0000 mon.a (mon.0) 1681 : cluster [DBG] osdmap e249: 8 total, 8 up, 7 in 2024-02-13T00:43:25.158 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:24.368172+0000 mon.b (mon.1) 353 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:25.158 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:24.376139+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:25.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:24.190192+0000 mon.b (mon.1) 352 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:25.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:24.354278+0000 mon.a (mon.0) 1680 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-83985-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:25.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:24.354376+0000 mon.a (mon.0) 1681 : cluster [DBG] osdmap e249: 8 total, 8 up, 7 in 2024-02-13T00:43:25.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:24.368172+0000 mon.b (mon.1) 353 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:25.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:24.376139+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:25.160 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:25.283 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:25.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:25.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:24.190192+0000 mon.b (mon.1) 352 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:25.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:24.354278+0000 mon.a (mon.0) 1680 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-83985-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:25.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:24.354376+0000 mon.a (mon.0) 1681 : cluster [DBG] osdmap e249: 8 total, 8 up, 7 in 2024-02-13T00:43:25.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:24.368172+0000 mon.b (mon.1) 353 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:25.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:24.376139+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:25.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:25.414 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:25.449 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:25.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:43:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:43:26.307 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:26.309 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:43:26] "GET /metrics HTTP/1.1" 200 191061 "" "Prometheus/2.33.4" 2024-02-13T00:43:26.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:24.362515+0000 mgr.y (mgr.14544) 255 : cluster [DBG] pgmap v344: 310 pgs: 2 active+clean+snaptrim, 82 unknown, 8 active+clean+snaptrim_wait, 218 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:43:26.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:25.191333+0000 mon.b (mon.1) 354 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:26.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:25.368684+0000 mon.a (mon.0) 1683 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:26.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:25.368875+0000 mon.a (mon.0) 1684 : cluster [DBG] osdmap e250: 8 total, 8 up, 7 in 2024-02-13T00:43:26.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:25.378866+0000 mon.c (mon.2) 233 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:26.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:25.381280+0000 mon.a (mon.0) 1685 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:26.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:25.443550+0000 mon.b (mon.1) 355 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:26.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:25.444245+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:26.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:25.445866+0000 mon.b (mon.1) 356 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:26.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:25.447798+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:26.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:25.453012+0000 mon.b (mon.1) 357 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:26.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:25.453812+0000 mon.a (mon.0) 1688 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:26.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:24.362515+0000 mgr.y (mgr.14544) 255 : cluster [DBG] pgmap v344: 310 pgs: 2 active+clean+snaptrim, 82 unknown, 8 active+clean+snaptrim_wait, 218 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:43:26.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:25.191333+0000 mon.b (mon.1) 354 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:26.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:25.368684+0000 mon.a (mon.0) 1683 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:26.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:25.368875+0000 mon.a (mon.0) 1684 : cluster [DBG] osdmap e250: 8 total, 8 up, 7 in 2024-02-13T00:43:26.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:25.378866+0000 mon.c (mon.2) 233 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:26.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:25.381280+0000 mon.a (mon.0) 1685 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:26.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:25.443550+0000 mon.b (mon.1) 355 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:26.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:25.444245+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:26.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:25.445866+0000 mon.b (mon.1) 356 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:26.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:25.447798+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:26.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:25.453012+0000 mon.b (mon.1) 357 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:26.311 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:25.453812+0000 mon.a (mon.0) 1688 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:26.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:26.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:26.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:24.362515+0000 mgr.y (mgr.14544) 255 : cluster [DBG] pgmap v344: 310 pgs: 2 active+clean+snaptrim, 82 unknown, 8 active+clean+snaptrim_wait, 218 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-13T00:43:26.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:25.191333+0000 mon.b (mon.1) 354 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:26.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:25.368684+0000 mon.a (mon.0) 1683 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:26.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:25.368875+0000 mon.a (mon.0) 1684 : cluster [DBG] osdmap e250: 8 total, 8 up, 7 in 2024-02-13T00:43:26.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:25.378866+0000 mon.c (mon.2) 233 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:26.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:25.381280+0000 mon.a (mon.0) 1685 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:26.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:25.443550+0000 mon.b (mon.1) 355 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:26.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:25.444245+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:26.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:25.445866+0000 mon.b (mon.1) 356 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:26.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:25.447798+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:26.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:25.453012+0000 mon.b (mon.1) 357 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:26.445 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:25.453812+0000 mon.a (mon.0) 1688 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:26.445 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:26.464 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:26.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:26.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:26.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:26.597 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:26.618 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:26.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:26.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:26.752 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:26.771 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:26.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:26.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:26.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:26.904 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:26.926 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:26.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:26.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:27.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:27.058 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:27.195 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:25.462817+0000 mgr.y (mgr.14544) 256 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:27.195 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:26.193069+0000 mon.b (mon.1) 358 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:27.195 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:26.367332+0000 mon.a (mon.0) 1689 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]': finished 2024-02-13T00:43:27.195 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:26.367801+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:27.195 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:26.367882+0000 mon.a (mon.0) 1691 : cluster [DBG] osdmap e251: 8 total, 8 up, 7 in 2024-02-13T00:43:27.195 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:26.369646+0000 mon.c (mon.2) 234 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:27.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:26.370254+0000 mon.a (mon.0) 1692 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:27.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:26.370956+0000 mon.b (mon.1) 359 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:27.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:26.371485+0000 mon.a (mon.0) 1693 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:27.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:26.382207+0000 mon.b (mon.1) 360 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:27.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:26.383008+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:27.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:25.462817+0000 mgr.y (mgr.14544) 256 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:27.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:26.193069+0000 mon.b (mon.1) 358 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:27.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:26.367332+0000 mon.a (mon.0) 1689 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]': finished 2024-02-13T00:43:27.197 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:26.367801+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:27.197 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:26.367882+0000 mon.a (mon.0) 1691 : cluster [DBG] osdmap e251: 8 total, 8 up, 7 in 2024-02-13T00:43:27.197 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:26.369646+0000 mon.c (mon.2) 234 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:27.197 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:26.370254+0000 mon.a (mon.0) 1692 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:27.197 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:26.370956+0000 mon.b (mon.1) 359 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:27.197 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:26.371485+0000 mon.a (mon.0) 1693 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:27.197 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:26.382207+0000 mon.b (mon.1) 360 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:27.197 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:26.383008+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:27.305 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:43:27.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:25.462817+0000 mgr.y (mgr.14544) 256 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:27.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:26.193069+0000 mon.b (mon.1) 358 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:27.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:26.367332+0000 mon.a (mon.0) 1689 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-83977-35"}]': finished 2024-02-13T00:43:27.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:26.367801+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:27.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:26.367882+0000 mon.a (mon.0) 1691 : cluster [DBG] osdmap e251: 8 total, 8 up, 7 in 2024-02-13T00:43:27.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:26.369646+0000 mon.c (mon.2) 234 : audit [INF] from='client.? 172.21.15.162:0/3484348857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:27.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:26.370254+0000 mon.a (mon.0) 1692 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]: dispatch 2024-02-13T00:43:27.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:26.370956+0000 mon.b (mon.1) 359 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:27.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:26.371485+0000 mon.a (mon.0) 1693 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:27.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:26.382207+0000 mon.b (mon.1) 360 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:27.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:26.383008+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:27.608 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:27.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:27.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:27.744 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:27.765 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:27.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:27.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:27.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:27.872 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:43:27.873 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:43:27.873 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 7] 2024-02-13T00:43:27.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd out 3 2024-02-13T00:43:27.900 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:28.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.205+0000 7f17fc319700 1 -- 172.21.15.162:0/2879265092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17f40f7940 msgr2=0x7f17f4103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:28.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.205+0000 7f17fc319700 1 --2- 172.21.15.162:0/2879265092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17f40f7940 0x7f17f4103a00 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 rx=0x7f17ec009be0 tx=0x7f17ec004d10).stop 2024-02-13T00:43:28.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.205+0000 7f17fc319700 1 -- 172.21.15.162:0/2879265092 shutdown_connections 2024-02-13T00:43:28.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.205+0000 7f17fc319700 1 --2- 172.21.15.162:0/2879265092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17f40f7940 0x7f17f4103a00 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:28.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.205+0000 7f17fc319700 1 --2- 172.21.15.162:0/2879265092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17f40fcbb0 0x7f17f40fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:28.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.205+0000 7f17fc319700 1 --2- 172.21.15.162:0/2879265092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f40fd510 0x7f17f40f7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:28.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.205+0000 7f17fc319700 1 -- 172.21.15.162:0/2879265092 >> 172.21.15.162:0/2879265092 conn(0x7f17f4070100 msgr2=0x7f17f40f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:28.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.205+0000 7f17fc319700 1 -- 172.21.15.162:0/2879265092 shutdown_connections 2024-02-13T00:43:28.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.205+0000 7f17fc319700 1 -- 172.21.15.162:0/2879265092 wait complete. 2024-02-13T00:43:28.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.205+0000 7f17fc319700 1 librados: starting msgr at 2024-02-13T00:43:28.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.205+0000 7f17fc319700 1 librados: starting objecter 2024-02-13T00:43:28.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fc319700 1 Processor -- start 2024-02-13T00:43:28.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fc319700 1 -- start start 2024-02-13T00:43:28.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fc319700 1 librados: setting wanted keys 2024-02-13T00:43:28.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fc319700 1 librados: calling monclient init 2024-02-13T00:43:28.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fc319700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17f40f7940 0x7f17f410e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fc319700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f40fcbb0 0x7f17f410e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fc319700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17f40fd510 0x7f17f4108220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fc319700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f4108710 con 0x7f17f40f7940 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fc319700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f4108880 con 0x7f17f40fcbb0 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fc319700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f4108a40 con 0x7f17f40fd510 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fa8b6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17f40fd510 0x7f17f4108220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fa8b6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17f40fd510 0x7f17f4108220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:53686/0 (socket says 172.21.15.162:53686) 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fa8b6700 1 -- 172.21.15.162:0/984209203 learned_addr learned my addr 172.21.15.162:0/984209203 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fa0b5700 1 --2- 172.21.15.162:0/984209203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17f40f7940 0x7f17f410e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17f98b4700 1 --2- 172.21.15.162:0/984209203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f40fcbb0 0x7f17f410e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fa8b6700 1 -- 172.21.15.162:0/984209203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f40fcbb0 msgr2=0x7f17f410e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fa8b6700 1 --2- 172.21.15.162:0/984209203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f40fcbb0 0x7f17f410e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:28.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fa8b6700 1 -- 172.21.15.162:0/984209203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17f40f7940 msgr2=0x7f17f410e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.206+0000 7f17fa8b6700 1 --2- 172.21.15.162:0/984209203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17f40f7940 0x7f17f410e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.207+0000 7f17fa8b6700 1 -- 172.21.15.162:0/984209203 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f0005040 con 0x7f17f40fd510 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.207+0000 7f17f98b4700 1 --2- 172.21.15.162:0/984209203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f40fcbb0 0x7f17f410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.207+0000 7f17fa8b6700 1 --2- 172.21.15.162:0/984209203 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17f40fd510 0x7f17f4108220 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7f17ec004ed0 tx=0x7f17ec004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.207+0000 7f17eb7fe700 10 client.?.objecter ms_handle_connect 0x7f17f40fd510 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.207+0000 7f17eb7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.207+0000 7f17eb7fe700 1 -- 172.21.15.162:0/984209203 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec00e040 con 0x7f17f40fd510 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.207+0000 7f17eb7fe700 1 -- 172.21.15.162:0/984209203 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17ec018040 con 0x7f17f40fd510 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.207+0000 7f17eb7fe700 1 -- 172.21.15.162:0/984209203 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec01c3f0 con 0x7f17f40fd510 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.207+0000 7f17fc319700 1 -- 172.21.15.162:0/984209203 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ec0097c0 con 0x7f17f40fd510 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.207+0000 7f17fc319700 10 client.36971.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.207+0000 7f17fc319700 1 -- 172.21.15.162:0/984209203 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f4108ee0 con 0x7f17f40fd510 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.207+0000 7f17fc319700 1 librados: init done 2024-02-13T00:43:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.208+0000 7f17fc319700 1 -- 172.21.15.162:0/984209203 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f41090d0 con 0x7f17f40fd510 2024-02-13T00:43:28.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.208+0000 7f17eb7fe700 1 -- 172.21.15.162:0/984209203 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f17ec013420 con 0x7f17f40fd510 2024-02-13T00:43:28.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.209+0000 7f17eb7fe700 1 --2- 172.21.15.162:0/984209203 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17e0062c90 0x7f17e0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:28.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.209+0000 7f17eb7fe700 1 -- 172.21.15.162:0/984209203 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(252..252 src has 52..252) v4 ==== 8800+0+0 (secure 0 0 0) 0x7f17ec07e4a0 con 0x7f17f40fd510 2024-02-13T00:43:28.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.209+0000 7f17eb7fe700 10 client.36971.objecter ms_dispatch 0x7f17f4001550 osd_map(252..252 src has 52..252) v4 2024-02-13T00:43:28.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.209+0000 7f17eb7fe700 3 client.36971.objecter handle_osd_map got epochs [252,252] > 0 2024-02-13T00:43:28.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.209+0000 7f17eb7fe700 3 client.36971.objecter handle_osd_map decoding full epoch 252 2024-02-13T00:43:28.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.209+0000 7f17eb7fe700 20 client.36971.objecter dump_active .. 0 homeless 2024-02-13T00:43:28.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.211+0000 7f17fa0b5700 1 --2- 172.21.15.162:0/984209203 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17e0062c90 0x7f17e0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:28.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.211+0000 7f17eb7fe700 10 client.36971.objecter ms_handle_connect 0x7f17e0062c90 2024-02-13T00:43:28.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.211+0000 7f17fa0b5700 1 --2- 172.21.15.162:0/984209203 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17e0062c90 0x7f17e0065150 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 rx=0x7f17f0009350 tx=0x7f17f000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:28.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.213+0000 7f17eb7fe700 1 -- 172.21.15.162:0/984209203 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f17ec051740 con 0x7f17f40fd510 2024-02-13T00:43:28.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:26.362926+0000 mgr.y (mgr.14544) 257 : cluster [DBG] pgmap v346: 270 pgs: 32 creating+peering, 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 228 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:28.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:27.194327+0000 mon.b (mon.1) 361 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:28.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:27.368439+0000 mon.a (mon.0) 1695 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]': finished 2024-02-13T00:43:28.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:27.392992+0000 mon.c (mon.2) 235 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:28.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:27.614028+0000 mon.a (mon.0) 1696 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:28.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:27.614130+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e252: 8 total, 8 up, 7 in 2024-02-13T00:43:28.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:27.614779+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:28.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:27.614905+0000 mon.b (mon.1) 362 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-27"}]: dispatch 2024-02-13T00:43:28.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:27.615333+0000 mon.c (mon.2) 236 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:28.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:27.615425+0000 mon.a (mon.0) 1699 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-27"}]: dispatch 2024-02-13T00:43:28.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:27.615900+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:28.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:27.616438+0000 mon.c (mon.2) 237 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:28.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:27.616758+0000 mon.a (mon.0) 1701 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:26.362926+0000 mgr.y (mgr.14544) 257 : cluster [DBG] pgmap v346: 270 pgs: 32 creating+peering, 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 228 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:27.194327+0000 mon.b (mon.1) 361 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:27.368439+0000 mon.a (mon.0) 1695 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]': finished 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:27.392992+0000 mon.c (mon.2) 235 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:27.614028+0000 mon.a (mon.0) 1696 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:27.614130+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e252: 8 total, 8 up, 7 in 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:27.614779+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:27.614905+0000 mon.b (mon.1) 362 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-27"}]: dispatch 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:27.615333+0000 mon.c (mon.2) 236 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:27.615425+0000 mon.a (mon.0) 1699 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-27"}]: dispatch 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:27.615900+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:27.616438+0000 mon.c (mon.2) 237 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:28.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:27.616758+0000 mon.a (mon.0) 1701 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:28.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:28.399+0000 7f17fc319700 1 -- 172.21.15.162:0/984209203 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f17f41060f0 con 0x7f17f40fd510 2024-02-13T00:43:28.448 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:28.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:28.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:26.362926+0000 mgr.y (mgr.14544) 257 : cluster [DBG] pgmap v346: 270 pgs: 32 creating+peering, 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 228 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:28.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:27.194327+0000 mon.b (mon.1) 361 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:28.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:27.368439+0000 mon.a (mon.0) 1695 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-83977-35"}]': finished 2024-02-13T00:43:28.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:27.392992+0000 mon.c (mon.2) 235 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:28.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:27.614028+0000 mon.a (mon.0) 1696 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:28.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:27.614130+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e252: 8 total, 8 up, 7 in 2024-02-13T00:43:28.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:27.614779+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:28.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:27.614905+0000 mon.b (mon.1) 362 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-27"}]: dispatch 2024-02-13T00:43:28.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:27.615333+0000 mon.c (mon.2) 236 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:28.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:27.615425+0000 mon.a (mon.0) 1699 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-27"}]: dispatch 2024-02-13T00:43:28.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:27.615900+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:28.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:27.616438+0000 mon.c (mon.2) 237 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:28.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:27.616758+0000 mon.a (mon.0) 1701 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:28.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:28.660 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:28.761 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:43:28.761 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:43:28.770 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:28.968 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:28.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:28.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:28.992 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:29.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:43:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:43:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:43:29.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:29.104 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:29.125 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:29.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:29.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:29.258 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:28.195602+0000 mon.b (mon.1) 363 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:28.370119+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-45"}]': finished 2024-02-13T00:43:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:28.370212+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-27"}]': finished 2024-02-13T00:43:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:28.370281+0000 mon.a (mon.0) 1704 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:28.370344+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e253: 8 total, 8 up, 7 in 2024-02-13T00:43:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:28.373488+0000 mon.c (mon.2) 238 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-83977-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:28.373761+0000 mon.b (mon.1) 364 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-27", "mode": "writeback"}]: dispatch 2024-02-13T00:43:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:28.373854+0000 mon.a (mon.0) 1706 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-83977-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:28.374259+0000 mon.a (mon.0) 1707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-27", "mode": "writeback"}]: dispatch 2024-02-13T00:43:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:28.401559+0000 mon.c (mon.2) 239 : audit [INF] from='client.? 172.21.15.162:0/984209203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:43:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:28.401855+0000 mon.a (mon.0) 1708 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:43:29.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:28.195602+0000 mon.b (mon.1) 363 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:29.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:28.370119+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-45"}]': finished 2024-02-13T00:43:29.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:28.370212+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-27"}]': finished 2024-02-13T00:43:29.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:28.370281+0000 mon.a (mon.0) 1704 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:29.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:28.370344+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e253: 8 total, 8 up, 7 in 2024-02-13T00:43:29.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:28.373488+0000 mon.c (mon.2) 238 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-83977-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:29.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:28.373761+0000 mon.b (mon.1) 364 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-27", "mode": "writeback"}]: dispatch 2024-02-13T00:43:29.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:28.373854+0000 mon.a (mon.0) 1706 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-83977-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:29.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:28.374259+0000 mon.a (mon.0) 1707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-27", "mode": "writeback"}]: dispatch 2024-02-13T00:43:29.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:28.401559+0000 mon.c (mon.2) 239 : audit [INF] from='client.? 172.21.15.162:0/984209203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:43:29.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:28.401855+0000 mon.a (mon.0) 1708 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:43:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:28.195602+0000 mon.b (mon.1) 363 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:28.370119+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-45"}]': finished 2024-02-13T00:43:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:28.370212+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-27"}]': finished 2024-02-13T00:43:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:28.370281+0000 mon.a (mon.0) 1704 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:28.370344+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e253: 8 total, 8 up, 7 in 2024-02-13T00:43:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:28.373488+0000 mon.c (mon.2) 238 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-83977-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:28.373761+0000 mon.b (mon.1) 364 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-27", "mode": "writeback"}]: dispatch 2024-02-13T00:43:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:28.373854+0000 mon.a (mon.0) 1706 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-83977-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:28.374259+0000 mon.a (mon.0) 1707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-27", "mode": "writeback"}]: dispatch 2024-02-13T00:43:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:28.401559+0000 mon.c (mon.2) 239 : audit [INF] from='client.? 172.21.15.162:0/984209203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:43:29.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:28.401855+0000 mon.a (mon.0) 1708 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:43:29.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.614+0000 7f17eb7fe700 1 -- 172.21.15.162:0/984209203 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v254) v1 ==== 91+0+0 (secure 0 0 0) 0x7f17ec021030 con 0x7f17f40fd510 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 10 librados: watch_flush enter 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 10 librados: watch_flush exit 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 20 client.36971.objecter shutdown clearing up homeless session... 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 10 client.36971.objecter successfully canceled tick 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 -- 172.21.15.162:0/984209203 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17e0062c90 msgr2=0x7f17e0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 --2- 172.21.15.162:0/984209203 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17e0062c90 0x7f17e0065150 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 rx=0x7f17f0009350 tx=0x7f17f000b040).stop 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 -- 172.21.15.162:0/984209203 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17f40fd510 msgr2=0x7f17f4108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 --2- 172.21.15.162:0/984209203 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17f40fd510 0x7f17f4108220 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7f17ec004ed0 tx=0x7f17ec004d10).stop 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 -- 172.21.15.162:0/984209203 shutdown_connections 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 --2- 172.21.15.162:0/984209203 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17e0062c90 0x7f17e0065150 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 --2- 172.21.15.162:0/984209203 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17f40fd510 0x7f17f4108220 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 --2- 172.21.15.162:0/984209203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17f40f7940 0x7f17f410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 --2- 172.21.15.162:0/984209203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f40fcbb0 0x7f17f410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:29.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 -- 172.21.15.162:0/984209203 >> 172.21.15.162:0/984209203 conn(0x7f17f4070100 msgr2=0x7f17f40fae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:29.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 -- 172.21.15.162:0/984209203 shutdown_connections 2024-02-13T00:43:29.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 -- 172.21.15.162:0/984209203 wait complete. 2024-02-13T00:43:29.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:29.616+0000 7f17fc319700 1 librados: shutdown 2024-02-13T00:43:29.622 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:43:30.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-13T00:43:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:30.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:30.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:28.363291+0000 mgr.y (mgr.14544) 258 : cluster [DBG] pgmap v349: 270 pgs: 32 creating+peering, 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 228 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:30.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:29.196646+0000 mon.b (mon.1) 365 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:30.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:29.368146+0000 mon.a (mon.0) 1709 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:30.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:29.368171+0000 mon.a (mon.0) 1710 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:43:30.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:29.615149+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-27", "mode": "writeback"}]': finished 2024-02-13T00:43:30.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:29.615314+0000 mon.a (mon.0) 1712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:43:30.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:29.615372+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e254: 8 total, 8 up, 7 in 2024-02-13T00:43:30.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:28.363291+0000 mgr.y (mgr.14544) 258 : cluster [DBG] pgmap v349: 270 pgs: 32 creating+peering, 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 228 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:30.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:29.196646+0000 mon.b (mon.1) 365 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:30.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:29.368146+0000 mon.a (mon.0) 1709 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:30.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:29.368171+0000 mon.a (mon.0) 1710 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:43:30.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:29.615149+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-27", "mode": "writeback"}]': finished 2024-02-13T00:43:30.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:29.615314+0000 mon.a (mon.0) 1712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:43:30.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:29.615372+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e254: 8 total, 8 up, 7 in 2024-02-13T00:43:30.418 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:30.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:28.363291+0000 mgr.y (mgr.14544) 258 : cluster [DBG] pgmap v349: 270 pgs: 32 creating+peering, 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 228 active+clean; 9.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:30.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:29.196646+0000 mon.b (mon.1) 365 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:30.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:29.368146+0000 mon.a (mon.0) 1709 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:30.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:29.368171+0000 mon.a (mon.0) 1710 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:43:30.444 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:29.615149+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-27", "mode": "writeback"}]': finished 2024-02-13T00:43:30.445 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:29.615314+0000 mon.a (mon.0) 1712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:43:30.445 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:29.615372+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e254: 8 total, 8 up, 7 in 2024-02-13T00:43:30.446 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:30.466 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:30.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:30.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:30.597 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:30.615 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:30.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:30.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:30.747 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:30.998 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:31.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:31.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:31.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:31.129 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:31.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:29.990844+0000 osd.3 (osd.3) 143 : cluster [DBG] 18.1a scrub starts 2024-02-13T00:43:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:29.994455+0000 osd.3 (osd.3) 144 : cluster [DBG] 18.1a scrub ok 2024-02-13T00:43:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:30.197902+0000 mon.b (mon.1) 366 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:30.256553+0000 osd.5 (osd.5) 111 : cluster [DBG] 18.f scrub starts 2024-02-13T00:43:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:30.257591+0000 osd.5 (osd.5) 112 : cluster [DBG] 18.f scrub ok 2024-02-13T00:43:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:30.343478+0000 mon.a (mon.0) 1714 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-83977-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-83977-36"}]': finished 2024-02-13T00:43:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:30.343597+0000 mon.a (mon.0) 1715 : cluster [DBG] osdmap e255: 8 total, 8 up, 7 in 2024-02-13T00:43:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:30.351927+0000 mon.b (mon.1) 367 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:30.352053+0000 osd.0 (osd.0) 91 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:43:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:30.365531+0000 osd.0 (osd.0) 92 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:43:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:30.367358+0000 mon.a (mon.0) 1716 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:30.388065+0000 mon.b (mon.1) 368 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:30.388640+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:30.409669+0000 osd.1 (osd.1) 105 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:30.413365+0000 osd.1 (osd.1) 106 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:30.576823+0000 osd.2 (osd.2) 109 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:30.578598+0000 osd.2 (osd.2) 110 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:29.990844+0000 osd.3 (osd.3) 143 : cluster [DBG] 18.1a scrub starts 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:29.994455+0000 osd.3 (osd.3) 144 : cluster [DBG] 18.1a scrub ok 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:30.197902+0000 mon.b (mon.1) 366 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:30.256553+0000 osd.5 (osd.5) 111 : cluster [DBG] 18.f scrub starts 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:30.257591+0000 osd.5 (osd.5) 112 : cluster [DBG] 18.f scrub ok 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:30.343478+0000 mon.a (mon.0) 1714 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-83977-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-83977-36"}]': finished 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:30.343597+0000 mon.a (mon.0) 1715 : cluster [DBG] osdmap e255: 8 total, 8 up, 7 in 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:30.351927+0000 mon.b (mon.1) 367 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:30.352053+0000 osd.0 (osd.0) 91 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:43:31.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:30.365531+0000 osd.0 (osd.0) 92 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:43:31.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:30.367358+0000 mon.a (mon.0) 1716 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:31.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:30.388065+0000 mon.b (mon.1) 368 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:31.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:30.388640+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:31.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:30.409669+0000 osd.1 (osd.1) 105 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:43:31.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:30.413365+0000 osd.1 (osd.1) 106 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:43:31.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:30.576823+0000 osd.2 (osd.2) 109 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:43:31.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:30.578598+0000 osd.2 (osd.2) 110 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:43:31.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:29.990844+0000 osd.3 (osd.3) 143 : cluster [DBG] 18.1a scrub starts 2024-02-13T00:43:31.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:29.994455+0000 osd.3 (osd.3) 144 : cluster [DBG] 18.1a scrub ok 2024-02-13T00:43:31.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:30.197902+0000 mon.b (mon.1) 366 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:31.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:30.256553+0000 osd.5 (osd.5) 111 : cluster [DBG] 18.f scrub starts 2024-02-13T00:43:31.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:30.257591+0000 osd.5 (osd.5) 112 : cluster [DBG] 18.f scrub ok 2024-02-13T00:43:31.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:30.343478+0000 mon.a (mon.0) 1714 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-83977-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-83977-36"}]': finished 2024-02-13T00:43:31.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:30.343597+0000 mon.a (mon.0) 1715 : cluster [DBG] osdmap e255: 8 total, 8 up, 7 in 2024-02-13T00:43:31.548 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:30.351927+0000 mon.b (mon.1) 367 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:31.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:30.352053+0000 osd.0 (osd.0) 91 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:43:31.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:30.365531+0000 osd.0 (osd.0) 92 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:43:31.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:30.367358+0000 mon.a (mon.0) 1716 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:31.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:30.388065+0000 mon.b (mon.1) 368 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:31.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:30.388640+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:31.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:30.409669+0000 osd.1 (osd.1) 105 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:43:31.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:30.413365+0000 osd.1 (osd.1) 106 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:43:31.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:30.576823+0000 osd.2 (osd.2) 109 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:43:31.549 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:30.578598+0000 osd.2 (osd.2) 110 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:43:32.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:30.363789+0000 mgr.y (mgr.14544) 259 : cluster [DBG] pgmap v353: 278 pgs: 2 active+clean+snaptrim, 8 unknown, 8 active+clean+snaptrim_wait, 260 active+clean; 9.2 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-02-13T00:43:32.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:31.199052+0000 mon.b (mon.1) 369 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:32.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:31.278010+0000 osd.5 (osd.5) 113 : cluster [DBG] 28.19 scrub starts 2024-02-13T00:43:32.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:31.281094+0000 osd.5 (osd.5) 114 : cluster [DBG] 28.19 scrub ok 2024-02-13T00:43:32.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:31.315217+0000 osd.0 (osd.0) 93 : cluster [DBG] 18.8 scrub starts 2024-02-13T00:43:32.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:31.316087+0000 osd.0 (osd.0) 94 : cluster [DBG] 18.8 scrub ok 2024-02-13T00:43:32.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:31.345313+0000 mon.a (mon.0) 1718 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]': finished 2024-02-13T00:43:32.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:31.345431+0000 mon.a (mon.0) 1719 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:32.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:31.345490+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e256: 8 total, 8 up, 7 in 2024-02-13T00:43:32.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:31.349511+0000 mon.b (mon.1) 370 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27"}]: dispatch 2024-02-13T00:43:32.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:31.349765+0000 mon.b (mon.1) 371 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:32.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:31.350171+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27"}]: dispatch 2024-02-13T00:43:32.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:31.350551+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:32.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:31.552986+0000 osd.2 (osd.2) 111 : cluster [DBG] 28.5 scrub starts 2024-02-13T00:43:32.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:31.554062+0000 osd.2 (osd.2) 112 : cluster [DBG] 28.5 scrub ok 2024-02-13T00:43:32.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:30.363789+0000 mgr.y (mgr.14544) 259 : cluster [DBG] pgmap v353: 278 pgs: 2 active+clean+snaptrim, 8 unknown, 8 active+clean+snaptrim_wait, 260 active+clean; 9.2 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-02-13T00:43:32.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:31.199052+0000 mon.b (mon.1) 369 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:31.278010+0000 osd.5 (osd.5) 113 : cluster [DBG] 28.19 scrub starts 2024-02-13T00:43:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:31.281094+0000 osd.5 (osd.5) 114 : cluster [DBG] 28.19 scrub ok 2024-02-13T00:43:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:31.315217+0000 osd.0 (osd.0) 93 : cluster [DBG] 18.8 scrub starts 2024-02-13T00:43:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:31.316087+0000 osd.0 (osd.0) 94 : cluster [DBG] 18.8 scrub ok 2024-02-13T00:43:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:31.345313+0000 mon.a (mon.0) 1718 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]': finished 2024-02-13T00:43:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:31.345431+0000 mon.a (mon.0) 1719 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:31.345490+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e256: 8 total, 8 up, 7 in 2024-02-13T00:43:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:31.349511+0000 mon.b (mon.1) 370 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27"}]: dispatch 2024-02-13T00:43:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:31.349765+0000 mon.b (mon.1) 371 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:31.350171+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27"}]: dispatch 2024-02-13T00:43:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:31.350551+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:32.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:31.552986+0000 osd.2 (osd.2) 111 : cluster [DBG] 28.5 scrub starts 2024-02-13T00:43:32.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:31.554062+0000 osd.2 (osd.2) 112 : cluster [DBG] 28.5 scrub ok 2024-02-13T00:43:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:30.363789+0000 mgr.y (mgr.14544) 259 : cluster [DBG] pgmap v353: 278 pgs: 2 active+clean+snaptrim, 8 unknown, 8 active+clean+snaptrim_wait, 260 active+clean; 9.2 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-02-13T00:43:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:31.199052+0000 mon.b (mon.1) 369 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:31.278010+0000 osd.5 (osd.5) 113 : cluster [DBG] 28.19 scrub starts 2024-02-13T00:43:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:31.281094+0000 osd.5 (osd.5) 114 : cluster [DBG] 28.19 scrub ok 2024-02-13T00:43:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:31.315217+0000 osd.0 (osd.0) 93 : cluster [DBG] 18.8 scrub starts 2024-02-13T00:43:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:31.316087+0000 osd.0 (osd.0) 94 : cluster [DBG] 18.8 scrub ok 2024-02-13T00:43:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:31.345313+0000 mon.a (mon.0) 1718 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-45"}]': finished 2024-02-13T00:43:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:31.345431+0000 mon.a (mon.0) 1719 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:31.345490+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e256: 8 total, 8 up, 7 in 2024-02-13T00:43:32.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:31.349511+0000 mon.b (mon.1) 370 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27"}]: dispatch 2024-02-13T00:43:32.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:31.349765+0000 mon.b (mon.1) 371 : audit [INF] from='client.? 172.21.15.162:0/2880420196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:32.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:31.350171+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27"}]: dispatch 2024-02-13T00:43:32.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:31.350551+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]: dispatch 2024-02-13T00:43:32.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:31.552986+0000 osd.2 (osd.2) 111 : cluster [DBG] 28.5 scrub starts 2024-02-13T00:43:32.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:31.554062+0000 osd.2 (osd.2) 112 : cluster [DBG] 28.5 scrub ok 2024-02-13T00:43:32.663 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:43:33.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.292+0000 7f845d452700 1 -- 172.21.15.162:0/1530863955 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84580fbf50 msgr2=0x7f84580fc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.292+0000 7f845d452700 1 --2- 172.21.15.162:0/1530863955 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84580fbf50 0x7f84580fc3d0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7f8450009be0 tx=0x7f8450004d10).stop 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.292+0000 7f845d452700 1 -- 172.21.15.162:0/1530863955 shutdown_connections 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.292+0000 7f845d452700 1 --2- 172.21.15.162:0/1530863955 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84580fbf50 0x7f84580fc3d0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.292+0000 7f845d452700 1 --2- 172.21.15.162:0/1530863955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84580f6a10 0x7f84580f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.292+0000 7f845d452700 1 --2- 172.21.15.162:0/1530863955 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84580f7370 0x7f84580fba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.292+0000 7f845d452700 1 -- 172.21.15.162:0/1530863955 >> 172.21.15.162:0/1530863955 conn(0x7f84580f2730 msgr2=0x7f84580f4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.293+0000 7f845d452700 1 -- 172.21.15.162:0/1530863955 shutdown_connections 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.293+0000 7f845d452700 1 -- 172.21.15.162:0/1530863955 wait complete. 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.293+0000 7f845d452700 1 librados: starting msgr at 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.293+0000 7f845d452700 1 librados: starting objecter 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.293+0000 7f845d452700 1 Processor -- start 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.293+0000 7f845d452700 1 -- start start 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.293+0000 7f845d452700 1 librados: setting wanted keys 2024-02-13T00:43:33.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.293+0000 7f845d452700 1 librados: calling monclient init 2024-02-13T00:43:33.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f845d452700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84580f6a10 0x7f84581145a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f845d452700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84580f7370 0x7f8458114ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f845d452700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84580fbf50 0x7f845810e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f845d452700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845810ec70 con 0x7f84580fbf50 2024-02-13T00:43:33.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f845d452700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845810ede0 con 0x7f84580f6a10 2024-02-13T00:43:33.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f845d452700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845810efa0 con 0x7f84580f7370 2024-02-13T00:43:33.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f84577fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84580fbf50 0x7f845810e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f84577fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84580fbf50 0x7f845810e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59772/0 (socket says 172.21.15.162:59772) 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f84577fe700 1 -- 172.21.15.162:0/3528056441 learned_addr learned my addr 172.21.15.162:0/3528056441 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f84567fc700 1 --2- 172.21.15.162:0/3528056441 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84580f7370 0x7f8458114ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f8456ffd700 1 --2- 172.21.15.162:0/3528056441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84580f6a10 0x7f84581145a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f84577fe700 1 -- 172.21.15.162:0/3528056441 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84580f7370 msgr2=0x7f8458114ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f84577fe700 1 --2- 172.21.15.162:0/3528056441 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84580f7370 0x7f8458114ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f84577fe700 1 -- 172.21.15.162:0/3528056441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84580f6a10 msgr2=0x7f84581145a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f84577fe700 1 --2- 172.21.15.162:0/3528056441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84580f6a10 0x7f84581145a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f84577fe700 1 -- 172.21.15.162:0/3528056441 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84500097c0 con 0x7f84580fbf50 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f84567fc700 1 --2- 172.21.15.162:0/3528056441 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84580f7370 0x7f8458114ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f84577fe700 1 --2- 172.21.15.162:0/3528056441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84580fbf50 0x7f845810e740 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7f8450004a30 tx=0x7f8450004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f843ffff700 10 client.?.objecter ms_handle_connect 0x7f84580fbf50 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f843ffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.294+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845000e040 con 0x7f84580fbf50 2024-02-13T00:43:33.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.295+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8450018040 con 0x7f84580fbf50 2024-02-13T00:43:33.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.295+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845001c3f0 con 0x7f84580fbf50 2024-02-13T00:43:33.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.295+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845810f220 con 0x7f84580fbf50 2024-02-13T00:43:33.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.295+0000 7f845d452700 10 client.26874.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:33.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.295+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845811b3a0 con 0x7f84580fbf50 2024-02-13T00:43:33.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.295+0000 7f845d452700 1 librados: init done 2024-02-13T00:43:33.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.295+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f84580607d0 con 0x7f84580fbf50 2024-02-13T00:43:33.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.296+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f8450007b20 con 0x7f84580fbf50 2024-02-13T00:43:33.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.296+0000 7f843ffff700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8440062c90 0x7f8440065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(257..257 src has 52..257) v4 ==== 8846+0+0 (secure 0 0 0) 0x7f845007e4d0 con 0x7f84580fbf50 2024-02-13T00:43:33.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 osd_map(257..257 src has 52..257) v4 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f843ffff700 3 client.26874.objecter handle_osd_map got epochs [257,257] > 0 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f843ffff700 3 client.26874.objecter handle_osd_map decoding full epoch 257 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f8456ffd700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8440062c90 0x7f8440065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f843ffff700 20 client.26874.objecter dump_active .. 0 homeless 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v257) v1 ==== 54+0+16 (secure 0 0 0) 0x7f8450085070 con 0x7f84580fbf50 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f843ffff700 10 client.26874.objecter ms_handle_connect 0x7f8440062c90 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f8456ffd700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8440062c90 0x7f8440065150 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 rx=0x7f84580f5fd0 tx=0x7f844800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f845d452700 10 client.26874.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 1 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6801/2103743338 conn(0x7f8458105d50 0x7f8458106130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458114270 osd=0 3 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458114270 osd=0 3 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458114270 osd=0 3 2024-02-13T00:43:33.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 1 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f8458114270 osd=0 3 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 0 1 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458114270 osd=0 4 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f845d452700 10 client.26874.objecter _send_command 1 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f845810d6c0 con 0x7f8458105d50 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f84567fc700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6801/2103743338 conn(0x7f8458105d50 0x7f8458106130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f843ffff700 10 client.26874.objecter ms_handle_connect 0x7f8458105d50 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.297+0000 7f84567fc700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6801/2103743338 conn(0x7f8458105d50 0x7f8458106130 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.298+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f845810d6c0 con 0x7f8458105d50 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.298+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.298+0000 7f843ffff700 10 client.26874.objecter _finish_command 1 = system:0 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.298+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f8458114270 osd=0 4 2024-02-13T00:43:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.298+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 0 1 2024-02-13T00:43:33.305 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:33.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.320+0000 7f845d452700 10 client.26874.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:33.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.320+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 2 2024-02-13T00:43:33.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.320+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458114270 osd=0 3 2024-02-13T00:43:33.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.320+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458114270 osd=0 3 2024-02-13T00:43:33.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.320+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458114270 osd=0 3 2024-02-13T00:43:33.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.320+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 2 2024-02-13T00:43:33.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.320+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f8458114270 osd=0 3 2024-02-13T00:43:33.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.320+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 0 2 2024-02-13T00:43:33.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.320+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458114270 osd=0 4 2024-02-13T00:43:33.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.320+0000 7f845d452700 10 client.26874.objecter _send_command 2 2024-02-13T00:43:33.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.320+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8458027410 con 0x7f8458105d50 2024-02-13T00:43:33.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.322+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8458027410 con 0x7f8458105d50 2024-02-13T00:43:33.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.322+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:33.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.322+0000 7f843ffff700 10 client.26874.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.322+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f8458114270 osd=0 4 2024-02-13T00:43:33.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.322+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 0 2 2024-02-13T00:43:33.326 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:43:33.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f845d452700 10 client.26874.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:33.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 3 2024-02-13T00:43:33.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6805/150599780 conn(0x7f8458059990 0x7f8458059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458028760 osd=1 3 2024-02-13T00:43:33.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458028760 osd=1 3 2024-02-13T00:43:33.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458028760 osd=1 3 2024-02-13T00:43:33.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 3 2024-02-13T00:43:33.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f8458028760 osd=1 3 2024-02-13T00:43:33.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 1 3 2024-02-13T00:43:33.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458028760 osd=1 4 2024-02-13T00:43:33.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f845d452700 10 client.26874.objecter _send_command 3 2024-02-13T00:43:33.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8458121cb0 con 0x7f8458059990 2024-02-13T00:43:33.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f8456ffd700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6805/150599780 conn(0x7f8458059990 0x7f8458059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f843ffff700 10 client.26874.objecter ms_handle_connect 0x7f8458059990 2024-02-13T00:43:33.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.323+0000 7f8456ffd700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6805/150599780 conn(0x7f8458059990 0x7f8458059d70 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:33.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.324+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8458121cb0 con 0x7f8458059990 2024-02-13T00:43:33.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.324+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:43:33.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.324+0000 7f843ffff700 10 client.26874.objecter _finish_command 3 = system:0 2024-02-13T00:43:33.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.324+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f8458028760 osd=1 4 2024-02-13T00:43:33.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.324+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 1 3 2024-02-13T00:43:33.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:31.950783+0000 osd.3 (osd.3) 145 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:43:33.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:31.953984+0000 osd.3 (osd.3) 146 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:43:33.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:32.200802+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:33.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:32.343026+0000 mon.a (mon.0) 1723 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:33.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:32.347776+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27"}]': finished 2024-02-13T00:43:33.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:32.347834+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]': finished 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:32.347894+0000 mon.a (mon.0) 1726 : cluster [DBG] osdmap e257: 8 total, 8 up, 7 in 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:32.351722+0000 mon.c (mon.2) 240 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:32.354055+0000 mon.a (mon.0) 1727 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:32.374360+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:32.375505+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:32.375902+0000 mon.a (mon.0) 1730 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:32.530012+0000 osd.2 (osd.2) 113 : cluster [DBG] 18.14 scrub starts 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:32.530962+0000 osd.2 (osd.2) 114 : cluster [DBG] 18.14 scrub ok 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:31.950783+0000 osd.3 (osd.3) 145 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:31.953984+0000 osd.3 (osd.3) 146 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:32.200802+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:32.343026+0000 mon.a (mon.0) 1723 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:32.347776+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27"}]': finished 2024-02-13T00:43:33.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:32.347834+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]': finished 2024-02-13T00:43:33.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:32.347894+0000 mon.a (mon.0) 1726 : cluster [DBG] osdmap e257: 8 total, 8 up, 7 in 2024-02-13T00:43:33.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:32.351722+0000 mon.c (mon.2) 240 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:33.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:32.354055+0000 mon.a (mon.0) 1727 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:33.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:32.374360+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:33.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:32.375505+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:33.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:32.375902+0000 mon.a (mon.0) 1730 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:33.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:32.530012+0000 osd.2 (osd.2) 113 : cluster [DBG] 18.14 scrub starts 2024-02-13T00:43:33.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:32.530962+0000 osd.2 (osd.2) 114 : cluster [DBG] 18.14 scrub ok 2024-02-13T00:43:33.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.340+0000 7f845d452700 10 client.26874.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:33.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.340+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 4 2024-02-13T00:43:33.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.340+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458028760 osd=1 3 2024-02-13T00:43:33.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.340+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458028760 osd=1 3 2024-02-13T00:43:33.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.340+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458028760 osd=1 3 2024-02-13T00:43:33.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.340+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 4 2024-02-13T00:43:33.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.340+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f8458028760 osd=1 3 2024-02-13T00:43:33.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.340+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 1 4 2024-02-13T00:43:33.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.340+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458028760 osd=1 4 2024-02-13T00:43:33.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.340+0000 7f845d452700 10 client.26874.objecter _send_command 4 2024-02-13T00:43:33.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.340+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8458059400 con 0x7f8458059990 2024-02-13T00:43:33.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8458059400 con 0x7f8458059990 2024-02-13T00:43:33.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:33.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f843ffff700 10 client.26874.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f8458028760 osd=1 4 2024-02-13T00:43:33.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 1 4 2024-02-13T00:43:33.345 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:43:33.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f845d452700 10 client.26874.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:33.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 5 2024-02-13T00:43:33.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6809/3689719376 conn(0x7f84580282c0 0x7f84581206d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458059810 osd=2 3 2024-02-13T00:43:33.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458059810 osd=2 3 2024-02-13T00:43:33.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458059810 osd=2 3 2024-02-13T00:43:33.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 5 2024-02-13T00:43:33.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f8458059810 osd=2 3 2024-02-13T00:43:33.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 2 5 2024-02-13T00:43:33.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458059810 osd=2 4 2024-02-13T00:43:33.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f845d452700 10 client.26874.objecter _send_command 5 2024-02-13T00:43:33.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8458120c50 con 0x7f84580282c0 2024-02-13T00:43:33.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f84567fc700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6809/3689719376 conn(0x7f84580282c0 0x7f84581206d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f843ffff700 10 client.26874.objecter ms_handle_connect 0x7f84580282c0 2024-02-13T00:43:33.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.342+0000 7f84567fc700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6809/3689719376 conn(0x7f84580282c0 0x7f84581206d0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:33.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.343+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8458120c50 con 0x7f84580282c0 2024-02-13T00:43:33.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.343+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:43:33.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.343+0000 7f843ffff700 10 client.26874.objecter _finish_command 5 = system:0 2024-02-13T00:43:33.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.343+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f8458059810 osd=2 4 2024-02-13T00:43:33.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.343+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 2 5 2024-02-13T00:43:33.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.365+0000 7f845d452700 10 client.26874.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:33.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.365+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 6 2024-02-13T00:43:33.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.365+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458059810 osd=2 3 2024-02-13T00:43:33.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.365+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458059810 osd=2 3 2024-02-13T00:43:33.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.365+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458059810 osd=2 3 2024-02-13T00:43:33.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.365+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 6 2024-02-13T00:43:33.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.365+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f8458059810 osd=2 3 2024-02-13T00:43:33.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.365+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 2 6 2024-02-13T00:43:33.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.365+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458059810 osd=2 4 2024-02-13T00:43:33.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.365+0000 7f845d452700 10 client.26874.objecter _send_command 6 2024-02-13T00:43:33.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.365+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8458027820 con 0x7f84580282c0 2024-02-13T00:43:33.372 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:43:33.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.368+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8458027820 con 0x7f84580282c0 2024-02-13T00:43:33.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.368+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:33.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.368+0000 7f843ffff700 10 client.26874.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.368+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f8458059810 osd=2 4 2024-02-13T00:43:33.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.368+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 2 6 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f845d452700 10 client.26874.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 7 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6813/604971011 conn(0x7f8458123e30 0x7f8458126220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458027bd0 osd=3 3 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458027bd0 osd=3 3 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458027bd0 osd=3 3 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 7 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f8458027bd0 osd=3 3 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 3 7 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458027bd0 osd=3 4 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f845d452700 10 client.26874.objecter _send_command 7 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8458128ea0 con 0x7f8458123e30 2024-02-13T00:43:33.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.369+0000 7f84577fe700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6813/604971011 conn(0x7f8458123e30 0x7f8458126220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.370+0000 7f84577fe700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6813/604971011 conn(0x7f8458123e30 0x7f8458126220 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:33.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.370+0000 7f843ffff700 10 client.26874.objecter ms_handle_connect 0x7f8458123e30 2024-02-13T00:43:33.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.371+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8458128ea0 con 0x7f8458123e30 2024-02-13T00:43:33.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.371+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:43:33.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.371+0000 7f843ffff700 10 client.26874.objecter _finish_command 7 = system:0 2024-02-13T00:43:33.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.371+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f8458027bd0 osd=3 4 2024-02-13T00:43:33.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.371+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 3 7 2024-02-13T00:43:33.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.393+0000 7f845d452700 10 client.26874.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:33.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.393+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 8 2024-02-13T00:43:33.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.393+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458027bd0 osd=3 3 2024-02-13T00:43:33.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.393+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458027bd0 osd=3 3 2024-02-13T00:43:33.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.393+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458027bd0 osd=3 3 2024-02-13T00:43:33.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.393+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 8 2024-02-13T00:43:33.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.393+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f8458027bd0 osd=3 3 2024-02-13T00:43:33.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.393+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 3 8 2024-02-13T00:43:33.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.393+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458027bd0 osd=3 4 2024-02-13T00:43:33.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.393+0000 7f845d452700 10 client.26874.objecter _send_command 8 2024-02-13T00:43:33.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.393+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f84581210b0 con 0x7f8458123e30 2024-02-13T00:43:33.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f84581210b0 con 0x7f8458123e30 2024-02-13T00:43:33.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:33.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f843ffff700 10 client.26874.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f8458027bd0 osd=3 4 2024-02-13T00:43:33.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 3 8 2024-02-13T00:43:33.398 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f845d452700 10 client.26874.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 9 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6800/3344263388 conn(0x7f8458121250 0x7f8458127890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f84581210b0 osd=4 3 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f84581210b0 osd=4 3 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f84581210b0 osd=4 3 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 9 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f84581210b0 osd=4 3 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 4 9 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f84581210b0 osd=4 4 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f845d452700 10 client.26874.objecter _send_command 9 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.394+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8458120410 con 0x7f8458121250 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.395+0000 7f8456ffd700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6800/3344263388 conn(0x7f8458121250 0x7f8458127890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.395+0000 7f843ffff700 10 client.26874.objecter ms_handle_connect 0x7f8458121250 2024-02-13T00:43:33.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.395+0000 7f8456ffd700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6800/3344263388 conn(0x7f8458121250 0x7f8458127890 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:33.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.396+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8458120410 con 0x7f8458121250 2024-02-13T00:43:33.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.396+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:43:33.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.396+0000 7f843ffff700 10 client.26874.objecter _finish_command 9 = system:0 2024-02-13T00:43:33.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.396+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f84581210b0 osd=4 4 2024-02-13T00:43:33.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.396+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 4 9 2024-02-13T00:43:33.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.411+0000 7f845d452700 10 client.26874.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:33.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.411+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 10 2024-02-13T00:43:33.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.411+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f84581210b0 osd=4 3 2024-02-13T00:43:33.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.411+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f84581210b0 osd=4 3 2024-02-13T00:43:33.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.411+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f84581210b0 osd=4 3 2024-02-13T00:43:33.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.411+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 10 2024-02-13T00:43:33.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.411+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f84581210b0 osd=4 3 2024-02-13T00:43:33.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.411+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 4 10 2024-02-13T00:43:33.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.411+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f84581210b0 osd=4 4 2024-02-13T00:43:33.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.411+0000 7f845d452700 10 client.26874.objecter _send_command 10 2024-02-13T00:43:33.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.411+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8458120410 con 0x7f8458121250 2024-02-13T00:43:33.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8458120410 con 0x7f8458121250 2024-02-13T00:43:33.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:33.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f843ffff700 10 client.26874.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f84581210b0 osd=4 4 2024-02-13T00:43:33.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 4 10 2024-02-13T00:43:33.416 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:43:33.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f845d452700 10 client.26874.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:33.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 11 2024-02-13T00:43:33.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6804/2212296748 conn(0x7f8458128290 0x7f84581286d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458120410 osd=5 3 2024-02-13T00:43:33.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458120410 osd=5 3 2024-02-13T00:43:33.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458120410 osd=5 3 2024-02-13T00:43:33.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 11 2024-02-13T00:43:33.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f8458120410 osd=5 3 2024-02-13T00:43:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 5 11 2024-02-13T00:43:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458120410 osd=5 4 2024-02-13T00:43:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f845d452700 10 client.26874.objecter _send_command 11 2024-02-13T00:43:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f845812f840 con 0x7f8458128290 2024-02-13T00:43:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f84567fc700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6804/2212296748 conn(0x7f8458128290 0x7f84581286d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f843ffff700 10 client.26874.objecter ms_handle_connect 0x7f8458128290 2024-02-13T00:43:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.413+0000 7f84567fc700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6804/2212296748 conn(0x7f8458128290 0x7f84581286d0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.414+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f845812f840 con 0x7f8458128290 2024-02-13T00:43:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.414+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:43:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.414+0000 7f843ffff700 10 client.26874.objecter _finish_command 11 = system:0 2024-02-13T00:43:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.414+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f8458120410 osd=5 4 2024-02-13T00:43:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.414+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 5 11 2024-02-13T00:43:33.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.428+0000 7f845d452700 10 client.26874.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:33.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.428+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 12 2024-02-13T00:43:33.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.428+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458120410 osd=5 3 2024-02-13T00:43:33.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.428+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458120410 osd=5 3 2024-02-13T00:43:33.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.428+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458120410 osd=5 3 2024-02-13T00:43:33.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.428+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 12 2024-02-13T00:43:33.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.428+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f8458120410 osd=5 3 2024-02-13T00:43:33.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.428+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 5 12 2024-02-13T00:43:33.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.428+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458120410 osd=5 4 2024-02-13T00:43:33.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.428+0000 7f845d452700 10 client.26874.objecter _send_command 12 2024-02-13T00:43:33.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.428+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f845812d0e0 con 0x7f8458128290 2024-02-13T00:43:33.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f845812d0e0 con 0x7f8458128290 2024-02-13T00:43:33.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:33.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f843ffff700 10 client.26874.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f8458120410 osd=5 4 2024-02-13T00:43:33.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 5 12 2024-02-13T00:43:33.434 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:43:33.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f845d452700 10 client.26874.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 13 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6808/1506640341 conn(0x7f845812d790 0x7f845812fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f845812d0e0 osd=6 3 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f845812d0e0 osd=6 3 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f845812d0e0 osd=6 3 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 13 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f845812d0e0 osd=6 3 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 6 13 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f845812d0e0 osd=6 4 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f845d452700 10 client.26874.objecter _send_command 13 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.430+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8458132940 con 0x7f845812d790 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.431+0000 7f84577fe700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6808/1506640341 conn(0x7f845812d790 0x7f845812fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.431+0000 7f843ffff700 10 client.26874.objecter ms_handle_connect 0x7f845812d790 2024-02-13T00:43:33.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.431+0000 7f84577fe700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6808/1506640341 conn(0x7f845812d790 0x7f845812fba0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:33.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.432+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8458132940 con 0x7f845812d790 2024-02-13T00:43:33.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.432+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:43:33.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.432+0000 7f843ffff700 10 client.26874.objecter _finish_command 13 = system:0 2024-02-13T00:43:33.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.432+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f845812d0e0 osd=6 4 2024-02-13T00:43:33.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.432+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 6 13 2024-02-13T00:43:33.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.449+0000 7f845d452700 10 client.26874.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:33.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.449+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 14 2024-02-13T00:43:33.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.449+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f845812d0e0 osd=6 3 2024-02-13T00:43:33.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.449+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f845812d0e0 osd=6 3 2024-02-13T00:43:33.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.449+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f845812d0e0 osd=6 3 2024-02-13T00:43:33.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.449+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 14 2024-02-13T00:43:33.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.449+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f845812d0e0 osd=6 3 2024-02-13T00:43:33.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.449+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 6 14 2024-02-13T00:43:33.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.449+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f845812d0e0 osd=6 4 2024-02-13T00:43:33.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.449+0000 7f845d452700 10 client.26874.objecter _send_command 14 2024-02-13T00:43:33.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.449+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8458132940 con 0x7f845812d790 2024-02-13T00:43:33.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.450+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8458132940 con 0x7f845812d790 2024-02-13T00:43:33.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.450+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:33.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.450+0000 7f843ffff700 10 client.26874.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.450+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f845812d0e0 osd=6 4 2024-02-13T00:43:33.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.450+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 6 14 2024-02-13T00:43:33.454 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:43:33.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f845d452700 10 client.26874.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 15 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6812/1964192772 conn(0x7f8458131c70 0x7f84581320f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458132940 osd=7 3 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458132940 osd=7 3 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458132940 osd=7 3 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 15 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f8458132940 osd=7 3 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 7 15 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458132940 osd=7 4 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f845d452700 10 client.26874.objecter _send_command 15 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8458132760 con 0x7f8458131c70 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f8456ffd700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6812/1964192772 conn(0x7f8458131c70 0x7f84581320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f843ffff700 10 client.26874.objecter ms_handle_connect 0x7f8458131c70 2024-02-13T00:43:33.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.451+0000 7f8456ffd700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6812/1964192772 conn(0x7f8458131c70 0x7f84581320f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:33.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.452+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8458132760 con 0x7f8458131c70 2024-02-13T00:43:33.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.452+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:43:33.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.452+0000 7f843ffff700 10 client.26874.objecter _finish_command 15 = system:0 2024-02-13T00:43:33.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.452+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f8458132940 osd=7 4 2024-02-13T00:43:33.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.452+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 7 15 2024-02-13T00:43:33.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.472+0000 7f845d452700 10 client.26874.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:33.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.472+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign -1 16 2024-02-13T00:43:33.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.472+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458132940 osd=7 3 2024-02-13T00:43:33.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.472+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458132940 osd=7 3 2024-02-13T00:43:33.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.472+0000 7f845d452700 20 client.26874.objecter _get_session s=0x7f8458132940 osd=7 3 2024-02-13T00:43:33.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.472+0000 7f845d452700 15 client.26874.objecter _session_command_op_remove -1 16 2024-02-13T00:43:33.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.472+0000 7f845d452700 20 client.26874.objecter get_session s=0x7f8458132940 osd=7 3 2024-02-13T00:43:33.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.472+0000 7f845d452700 15 client.26874.objecter _session_command_op_assign 7 16 2024-02-13T00:43:33.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.472+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458132940 osd=7 4 2024-02-13T00:43:33.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.472+0000 7f845d452700 10 client.26874.objecter _send_command 16 2024-02-13T00:43:33.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.472+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8458132760 con 0x7f8458131c70 2024-02-13T00:43:33.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f843ffff700 1 -- 172.21.15.162:0/3528056441 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8458132760 con 0x7f8458131c70 2024-02-13T00:43:33.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f843ffff700 10 client.26874.objecter ms_dispatch 0x7f8458001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:33.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f843ffff700 10 client.26874.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f843ffff700 20 client.26874.objecter put_session s=0x7f8458132940 osd=7 4 2024-02-13T00:43:33.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f843ffff700 15 client.26874.objecter _session_command_op_remove 7 16 2024-02-13T00:43:33.479 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:43:33.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 10 librados: watch_flush enter 2024-02-13T00:43:33.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 10 librados: watch_flush exit 2024-02-13T00:43:33.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 10 client.26874.objecter close_session for osd.0 2024-02-13T00:43:33.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6801/2103743338 conn(0x7f8458105d50 msgr2=0x7f8458106130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6801/2103743338 conn(0x7f8458105d50 0x7f8458106130 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458114270 osd=0 1 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 10 client.26874.objecter close_session for osd.1 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6805/150599780 conn(0x7f8458059990 msgr2=0x7f8458059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6805/150599780 conn(0x7f8458059990 0x7f8458059d70 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458028760 osd=1 1 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 10 client.26874.objecter close_session for osd.2 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6809/3689719376 conn(0x7f84580282c0 msgr2=0x7f84581206d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6809/3689719376 conn(0x7f84580282c0 0x7f84581206d0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458059810 osd=2 1 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 10 client.26874.objecter close_session for osd.3 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6813/604971011 conn(0x7f8458123e30 msgr2=0x7f8458126220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6813/604971011 conn(0x7f8458123e30 0x7f8458126220 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458027bd0 osd=3 1 2024-02-13T00:43:33.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 10 client.26874.objecter close_session for osd.4 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f84577fe700 1 -- 172.21.15.162:0/3528056441 reap_dead start 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6800/3344263388 conn(0x7f8458121250 msgr2=0x7f8458127890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6800/3344263388 conn(0x7f8458121250 0x7f8458127890 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f84577fe700 1 -- 172.21.15.162:0/3528056441 reap_dead start 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f84581210b0 osd=4 1 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 10 client.26874.objecter close_session for osd.5 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6804/2212296748 conn(0x7f8458128290 msgr2=0x7f84581286d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6804/2212296748 conn(0x7f8458128290 0x7f84581286d0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458120410 osd=5 1 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 10 client.26874.objecter close_session for osd.6 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6808/1506640341 conn(0x7f845812d790 msgr2=0x7f845812fba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.474+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6808/1506640341 conn(0x7f845812d790 0x7f845812fba0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f845812d0e0 osd=6 1 2024-02-13T00:43:33.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 10 client.26874.objecter close_session for osd.7 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6812/1964192772 conn(0x7f8458131c70 msgr2=0x7f84581320f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.181:6812/1964192772 conn(0x7f8458131c70 0x7f84581320f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 20 client.26874.objecter put_session s=0x7f8458132940 osd=7 1 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 20 client.26874.objecter shutdown clearing up homeless session... 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 10 client.26874.objecter successfully canceled tick 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8440062c90 msgr2=0x7f8440065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8440062c90 0x7f8440065150 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 rx=0x7f84580f5fd0 tx=0x7f844800b040).stop 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84580fbf50 msgr2=0x7f845810e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 1 --2- 172.21.15.162:0/3528056441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84580fbf50 0x7f845810e740 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7f8450004a30 tx=0x7f8450004d10).stop 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f84577fe700 1 -- 172.21.15.162:0/3528056441 reap_dead start 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 shutdown_connections 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 >> 172.21.15.162:0/3528056441 conn(0x7f84580f2730 msgr2=0x7f84580f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 shutdown_connections 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 1 -- 172.21.15.162:0/3528056441 wait complete. 2024-02-13T00:43:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.475+0000 7f845d452700 1 librados: shutdown 2024-02-13T00:43:33.483 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.483 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.483 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.483 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.483 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.483 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.483 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.483 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:43:33.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:31.950783+0000 osd.3 (osd.3) 145 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:43:33.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:31.953984+0000 osd.3 (osd.3) 146 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:43:33.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:32.200802+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:33.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:32.343026+0000 mon.a (mon.0) 1723 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:33.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:32.347776+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-27"}]': finished 2024-02-13T00:43:33.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:32.347834+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-45"}]': finished 2024-02-13T00:43:33.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:32.347894+0000 mon.a (mon.0) 1726 : cluster [DBG] osdmap e257: 8 total, 8 up, 7 in 2024-02-13T00:43:33.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:32.351722+0000 mon.c (mon.2) 240 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:33.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:32.354055+0000 mon.a (mon.0) 1727 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:33.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:32.374360+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:33.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:32.375505+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:33.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:32.375902+0000 mon.a (mon.0) 1730 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:33.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:32.530012+0000 osd.2 (osd.2) 113 : cluster [DBG] 18.14 scrub starts 2024-02-13T00:43:33.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:32.530962+0000 osd.2 (osd.2) 114 : cluster [DBG] 18.14 scrub ok 2024-02-13T00:43:33.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.934+0000 7f6914e26700 1 -- 172.21.15.162:0/2635446576 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69100f8d10 msgr2=0x7f69100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.934+0000 7f6914e26700 1 --2- 172.21.15.162:0/2635446576 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69100f8d10 0x7f69100f90f0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7f6908009be0 tx=0x7f6908004d10).stop 2024-02-13T00:43:33.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.934+0000 7f6914e26700 1 -- 172.21.15.162:0/2635446576 shutdown_connections 2024-02-13T00:43:33.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.934+0000 7f6914e26700 1 --2- 172.21.15.162:0/2635446576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6910102270 0x7f6910104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.934+0000 7f6914e26700 1 --2- 172.21.15.162:0/2635446576 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69100f8d10 0x7f69100f90f0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.934+0000 7f6914e26700 1 --2- 172.21.15.162:0/2635446576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69100f95e0 0x7f6910101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.934+0000 7f6914e26700 1 -- 172.21.15.162:0/2635446576 >> 172.21.15.162:0/2635446576 conn(0x7f691006eb90 msgr2=0x7f69100f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:33.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.934+0000 7f6914e26700 1 -- 172.21.15.162:0/2635446576 shutdown_connections 2024-02-13T00:43:33.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.934+0000 7f6914e26700 1 -- 172.21.15.162:0/2635446576 wait complete. 2024-02-13T00:43:33.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f6914e26700 1 librados: starting msgr at 2024-02-13T00:43:33.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f6914e26700 1 librados: starting objecter 2024-02-13T00:43:33.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f6914e26700 1 Processor -- start 2024-02-13T00:43:33.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f6914e26700 1 -- start start 2024-02-13T00:43:33.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f6914e26700 1 librados: setting wanted keys 2024-02-13T00:43:33.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f6914e26700 1 librados: calling monclient init 2024-02-13T00:43:33.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f6914e26700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69100f8d10 0x7f691010e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f6914e26700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69100f95e0 0x7f691010e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f6914e26700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6910102270 0x7f6910108410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f6914e26700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6910108900 con 0x7f69100f8d10 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f6914e26700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6910108a70 con 0x7f69100f95e0 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f6914e26700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6910108be0 con 0x7f6910102270 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f690e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69100f8d10 0x7f691010e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f690e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69100f8d10 0x7f691010e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59790/0 (socket says 172.21.15.162:59790) 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f690e59c700 1 -- 172.21.15.162:0/3977563402 learned_addr learned my addr 172.21.15.162:0/3977563402 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.935+0000 7f690ed9d700 1 --2- 172.21.15.162:0/3977563402 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6910102270 0x7f6910108410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f690dd9b700 1 --2- 172.21.15.162:0/3977563402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69100f95e0 0x7f691010e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f690e59c700 1 -- 172.21.15.162:0/3977563402 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6910102270 msgr2=0x7f6910108410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f690e59c700 1 --2- 172.21.15.162:0/3977563402 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6910102270 0x7f6910108410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f690e59c700 1 -- 172.21.15.162:0/3977563402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69100f95e0 msgr2=0x7f691010e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f690e59c700 1 --2- 172.21.15.162:0/3977563402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69100f95e0 0x7f691010e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f690e59c700 1 -- 172.21.15.162:0/3977563402 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69080097c0 con 0x7f69100f8d10 2024-02-13T00:43:33.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f690e59c700 1 --2- 172.21.15.162:0/3977563402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69100f8d10 0x7f691010e1e0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f6908004a30 tx=0x7f6908004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:33.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f69077fe700 10 client.?.objecter ms_handle_connect 0x7f69100f8d10 2024-02-13T00:43:33.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f69077fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:33.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f69077fe700 1 -- 172.21.15.162:0/3977563402 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690800e040 con 0x7f69100f8d10 2024-02-13T00:43:33.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f6914e26700 1 -- 172.21.15.162:0/3977563402 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6910108e60 con 0x7f69100f8d10 2024-02-13T00:43:33.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f69077fe700 1 -- 172.21.15.162:0/3977563402 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6908018040 con 0x7f69100f8d10 2024-02-13T00:43:33.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f6914e26700 10 client.26880.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:33.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f69077fe700 1 -- 172.21.15.162:0/3977563402 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6908013ce0 con 0x7f69100f8d10 2024-02-13T00:43:33.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f6914e26700 1 -- 172.21.15.162:0/3977563402 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69100f6b00 con 0x7f69100f8d10 2024-02-13T00:43:33.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.936+0000 7f6914e26700 1 librados: init done 2024-02-13T00:43:33.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.937+0000 7f6914e26700 1 -- 172.21.15.162:0/3977563402 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69100475e0 con 0x7f69100f8d10 2024-02-13T00:43:33.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.938+0000 7f69077fe700 1 -- 172.21.15.162:0/3977563402 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6908013420 con 0x7f69100f8d10 2024-02-13T00:43:33.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.938+0000 7f69077fe700 1 --2- 172.21.15.162:0/3977563402 >> v2:172.21.15.162:6800/2866534053 conn(0x7f68fc062c90 0x7f68fc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:33.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.938+0000 7f69077fe700 1 -- 172.21.15.162:0/3977563402 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(258..258 src has 52..258) v4 ==== 8456+0+0 (secure 0 0 0) 0x7f690807ea30 con 0x7f69100f8d10 2024-02-13T00:43:33.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.938+0000 7f69077fe700 10 client.26880.objecter ms_dispatch 0x7f6910001550 osd_map(258..258 src has 52..258) v4 2024-02-13T00:43:33.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.938+0000 7f69077fe700 3 client.26880.objecter handle_osd_map got epochs [258,258] > 0 2024-02-13T00:43:33.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.938+0000 7f69077fe700 3 client.26880.objecter handle_osd_map decoding full epoch 258 2024-02-13T00:43:33.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.938+0000 7f690dd9b700 1 --2- 172.21.15.162:0/3977563402 >> v2:172.21.15.162:6800/2866534053 conn(0x7f68fc062c90 0x7f68fc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:33.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.938+0000 7f69077fe700 20 client.26880.objecter dump_active .. 0 homeless 2024-02-13T00:43:33.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.938+0000 7f69077fe700 10 client.26880.objecter ms_handle_connect 0x7f68fc062c90 2024-02-13T00:43:33.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.938+0000 7f690dd9b700 1 --2- 172.21.15.162:0/3977563402 >> v2:172.21.15.162:6800/2866534053 conn(0x7f68fc062c90 0x7f68fc065150 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 rx=0x7f6910109b60 tx=0x7f68f8009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:33.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:33.942+0000 7f69077fe700 1 -- 172.21.15.162:0/3977563402 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6908051ea0 con 0x7f69100f8d10 2024-02-13T00:43:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.127+0000 7f6914e26700 1 -- 172.21.15.162:0/3977563402 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd out", "ids": ["3"]} v 0) v1 -- 0x7f6910107170 con 0x7f69100f8d10 2024-02-13T00:43:34.179 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:34.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:34.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:34.209 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:34.285 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:34.317 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:34.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:32.364221+0000 mgr.y (mgr.14544) 260 : cluster [DBG] pgmap v356: 270 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 260 active+clean; 9.2 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-02-13T00:43:34.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:33.202407+0000 mon.b (mon.1) 373 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:34.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:33.297497+0000 mon.a (mon.0) 1731 : audit [DBG] from='client.? 172.21.15.162:0/3528056441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:34.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:33.347902+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]': finished 2024-02-13T00:43:34.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:33.347969+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:34.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:33.348029+0000 mon.a (mon.0) 1734 : cluster [DBG] osdmap e258: 8 total, 8 up, 7 in 2024-02-13T00:43:34.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:33.348802+0000 mon.a (mon.0) 1735 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:34.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:33.349722+0000 mon.c (mon.2) 241 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:34.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:33.355603+0000 mon.a (mon.0) 1736 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:34.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:32.364221+0000 mgr.y (mgr.14544) 260 : cluster [DBG] pgmap v356: 270 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 260 active+clean; 9.2 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-02-13T00:43:34.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:33.202407+0000 mon.b (mon.1) 373 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:34.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:33.297497+0000 mon.a (mon.0) 1731 : audit [DBG] from='client.? 172.21.15.162:0/3528056441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:34.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:33.347902+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]': finished 2024-02-13T00:43:34.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:33.347969+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:34.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:33.348029+0000 mon.a (mon.0) 1734 : cluster [DBG] osdmap e258: 8 total, 8 up, 7 in 2024-02-13T00:43:34.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:33.348802+0000 mon.a (mon.0) 1735 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:34.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:33.349722+0000 mon.c (mon.2) 241 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:34.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:33.355603+0000 mon.a (mon.0) 1736 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:34.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:43:34.128+0000 7f5f6486f700 -1 mon.a@0(leader).osd e258 definitely_dead 0 2024-02-13T00:43:34.337 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:34.340 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:32.364221+0000 mgr.y (mgr.14544) 260 : cluster [DBG] pgmap v356: 270 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 260 active+clean; 9.2 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-02-13T00:43:34.340 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:33.202407+0000 mon.b (mon.1) 373 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:34.340 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:33.297497+0000 mon.a (mon.0) 1731 : audit [DBG] from='client.? 172.21.15.162:0/3528056441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:34.340 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:33.347902+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-83977-36"}]': finished 2024-02-13T00:43:34.340 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:33.347969+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-83985-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:34.340 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:33.348029+0000 mon.a (mon.0) 1734 : cluster [DBG] osdmap e258: 8 total, 8 up, 7 in 2024-02-13T00:43:34.341 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:33.348802+0000 mon.a (mon.0) 1735 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:34.341 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:33.349722+0000 mon.c (mon.2) 241 : audit [INF] from='client.? 172.21.15.162:0/2850878693' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:34.341 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:33.355603+0000 mon.a (mon.0) 1736 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]: dispatch 2024-02-13T00:43:34.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:34.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:34.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:34.473 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:34.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.601+0000 7f69077fe700 1 -- 172.21.15.162:0/3977563402 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["3"]}]=0 marked out osd.3. v259) v1 ==== 87+0+0 (secure 0 0 0) 0x7f690802b070 con 0x7f69100f8d10 2024-02-13T00:43:34.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 10 librados: watch_flush enter 2024-02-13T00:43:34.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 10 librados: watch_flush exit 2024-02-13T00:43:34.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 20 client.26880.objecter shutdown clearing up homeless session... 2024-02-13T00:43:34.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 10 client.26880.objecter successfully canceled tick 2024-02-13T00:43:34.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 -- 172.21.15.162:0/3977563402 >> v2:172.21.15.162:6800/2866534053 conn(0x7f68fc062c90 msgr2=0x7f68fc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:34.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 --2- 172.21.15.162:0/3977563402 >> v2:172.21.15.162:6800/2866534053 conn(0x7f68fc062c90 0x7f68fc065150 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 rx=0x7f6910109b60 tx=0x7f68f8009040).stop 2024-02-13T00:43:34.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 -- 172.21.15.162:0/3977563402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69100f8d10 msgr2=0x7f691010e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:34.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 --2- 172.21.15.162:0/3977563402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69100f8d10 0x7f691010e1e0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f6908004a30 tx=0x7f6908004d10).stop 2024-02-13T00:43:34.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 -- 172.21.15.162:0/3977563402 shutdown_connections 2024-02-13T00:43:34.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 --2- 172.21.15.162:0/3977563402 >> v2:172.21.15.162:6800/2866534053 conn(0x7f68fc062c90 0x7f68fc065150 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:34.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 --2- 172.21.15.162:0/3977563402 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6910102270 0x7f6910108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:34.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 --2- 172.21.15.162:0/3977563402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69100f8d10 0x7f691010e1e0 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:34.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 --2- 172.21.15.162:0/3977563402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69100f95e0 0x7f691010e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:34.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 -- 172.21.15.162:0/3977563402 >> 172.21.15.162:0/3977563402 conn(0x7f691006eb90 msgr2=0x7f6910102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:34.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 -- 172.21.15.162:0/3977563402 shutdown_connections 2024-02-13T00:43:34.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 -- 172.21.15.162:0/3977563402 wait complete. 2024-02-13T00:43:34.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:34.605+0000 7f6914e26700 1 librados: shutdown 2024-02-13T00:43:34.610 INFO:teuthology.orchestra.run.smithi162.stderr:marked out osd.3. 2024-02-13T00:43:35.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:33.926574+0000 osd.3 (osd.3) 147 : cluster [DBG] 18.d scrub starts 2024-02-13T00:43:35.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:33.930142+0000 osd.3 (osd.3) 148 : cluster [DBG] 18.d scrub ok 2024-02-13T00:43:35.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:34.128808+0000 mon.a (mon.0) 1737 : audit [INF] from='client.? 172.21.15.162:0/3977563402' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-13T00:43:35.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:34.128921+0000 mon.a (mon.0) 1738 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-02-13T00:43:35.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:34.203488+0000 mon.b (mon.1) 374 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:35.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:34.227372+0000 osd.5 (osd.5) 115 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:43:35.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:34.230326+0000 osd.5 (osd.5) 116 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:43:35.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:34.317164+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:43:35.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:34.320633+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:43:35.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:34.364684+0000 mgr.y (mgr.14544) 261 : cluster [DBG] pgmap v359: 270 pgs: 2 active+clean+snaptrim, 32 unknown, 8 active+clean+snaptrim_wait, 228 active+clean; 9.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:34.492355+0000 osd.6 (osd.6) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:34.492509+0000 osd.6 (osd.6) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:34.507092+0000 osd.2 (osd.2) 115 : cluster [DBG] 28.f scrub starts 2024-02-13T00:43:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:34.510198+0000 osd.2 (osd.2) 116 : cluster [DBG] 28.f scrub ok 2024-02-13T00:43:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:34.602448+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]': finished 2024-02-13T00:43:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:34.602554+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? 172.21.15.162:0/3977563402' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-13T00:43:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:34.602622+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e259: 8 total, 8 up, 6 in 2024-02-13T00:43:35.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:34.606591+0000 mon.b (mon.1) 375 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:35.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:33.926574+0000 osd.3 (osd.3) 147 : cluster [DBG] 18.d scrub starts 2024-02-13T00:43:35.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:33.930142+0000 osd.3 (osd.3) 148 : cluster [DBG] 18.d scrub ok 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:34.128808+0000 mon.a (mon.0) 1737 : audit [INF] from='client.? 172.21.15.162:0/3977563402' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:34.128921+0000 mon.a (mon.0) 1738 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:34.203488+0000 mon.b (mon.1) 374 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:34.227372+0000 osd.5 (osd.5) 115 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:34.230326+0000 osd.5 (osd.5) 116 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:34.317164+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:34.320633+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:34.364684+0000 mgr.y (mgr.14544) 261 : cluster [DBG] pgmap v359: 270 pgs: 2 active+clean+snaptrim, 32 unknown, 8 active+clean+snaptrim_wait, 228 active+clean; 9.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:34.492355+0000 osd.6 (osd.6) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:34.492509+0000 osd.6 (osd.6) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:34.507092+0000 osd.2 (osd.2) 115 : cluster [DBG] 28.f scrub starts 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:34.510198+0000 osd.2 (osd.2) 116 : cluster [DBG] 28.f scrub ok 2024-02-13T00:43:35.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:34.602448+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]': finished 2024-02-13T00:43:35.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:34.602554+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? 172.21.15.162:0/3977563402' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-13T00:43:35.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:34.602622+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e259: 8 total, 8 up, 6 in 2024-02-13T00:43:35.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:34.606591+0000 mon.b (mon.1) 375 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:35.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:34.607246+0000 mon.a (mon.0) 1742 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:35.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:34.621512+0000 mon.c (mon.2) 242 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:35.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:34.621821+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:35.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:34.622443+0000 mon.c (mon.2) 243 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:35.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:34.622693+0000 mon.a (mon.0) 1744 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:35.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:34.623069+0000 mon.c (mon.2) 244 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-83977-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:35.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:34.623293+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-83977-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:35.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:34.607246+0000 mon.a (mon.0) 1742 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:35.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:34.621512+0000 mon.c (mon.2) 242 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:35.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:34.621821+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:35.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:34.622443+0000 mon.c (mon.2) 243 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:35.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:34.622693+0000 mon.a (mon.0) 1744 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:35.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:34.623069+0000 mon.c (mon.2) 244 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-83977-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:35.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:34.623293+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-83977-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:35.476 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:33.926574+0000 osd.3 (osd.3) 147 : cluster [DBG] 18.d scrub starts 2024-02-13T00:43:35.476 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:33.930142+0000 osd.3 (osd.3) 148 : cluster [DBG] 18.d scrub ok 2024-02-13T00:43:35.476 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:34.128808+0000 mon.a (mon.0) 1737 : audit [INF] from='client.? 172.21.15.162:0/3977563402' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-13T00:43:35.476 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:34.128921+0000 mon.a (mon.0) 1738 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-02-13T00:43:35.476 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:34.203488+0000 mon.b (mon.1) 374 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:34.227372+0000 osd.5 (osd.5) 115 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:34.230326+0000 osd.5 (osd.5) 116 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:34.317164+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:34.320633+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:34.364684+0000 mgr.y (mgr.14544) 261 : cluster [DBG] pgmap v359: 270 pgs: 2 active+clean+snaptrim, 32 unknown, 8 active+clean+snaptrim_wait, 228 active+clean; 9.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:34.492355+0000 osd.6 (osd.6) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:34.492509+0000 osd.6 (osd.6) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:34.507092+0000 osd.2 (osd.2) 115 : cluster [DBG] 28.f scrub starts 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:34.510198+0000 osd.2 (osd.2) 116 : cluster [DBG] 28.f scrub ok 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:34.602448+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-83977-36"}]': finished 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:34.602554+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? 172.21.15.162:0/3977563402' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:34.602622+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e259: 8 total, 8 up, 6 in 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:34.606591+0000 mon.b (mon.1) 375 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:35.477 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:34.607246+0000 mon.a (mon.0) 1742 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:35.478 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:34.621512+0000 mon.c (mon.2) 242 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:35.478 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:34.621821+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:35.478 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:34.622443+0000 mon.c (mon.2) 243 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:35.478 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:34.622693+0000 mon.a (mon.0) 1744 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:35.478 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:34.623069+0000 mon.c (mon.2) 244 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-83977-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:35.478 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:34.623293+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-83977-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:35.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:43:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:43:36.223 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:36.224 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:35.204604+0000 mon.b (mon.1) 376 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:36.225 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:35.305845+0000 osd.0 (osd.0) 95 : cluster [DBG] 2.f scrub starts 2024-02-13T00:43:36.225 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:35.309304+0000 osd.0 (osd.0) 96 : cluster [DBG] 2.f scrub ok 2024-02-13T00:43:36.225 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:35.359843+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-46"}]': finished 2024-02-13T00:43:36.225 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:35.360012+0000 mon.a (mon.0) 1747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:36.225 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:35.360100+0000 mon.a (mon.0) 1748 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-83977-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:36.225 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:35.360167+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e260: 8 total, 8 up, 6 in 2024-02-13T00:43:36.225 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:35.377708+0000 mon.b (mon.1) 377 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:36.225 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:35.379892+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:36.225 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:35.383685+0000 mon.c (mon.2) 245 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-83977-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:36.225 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:35.385275+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-83977-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:36.225 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:35.473374+0000 mgr.y (mgr.14544) 262 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:36.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:35.204604+0000 mon.b (mon.1) 376 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:36.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:35.305845+0000 osd.0 (osd.0) 95 : cluster [DBG] 2.f scrub starts 2024-02-13T00:43:36.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:35.309304+0000 osd.0 (osd.0) 96 : cluster [DBG] 2.f scrub ok 2024-02-13T00:43:36.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:35.359843+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-46"}]': finished 2024-02-13T00:43:36.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:35.360012+0000 mon.a (mon.0) 1747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:36.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:35.360100+0000 mon.a (mon.0) 1748 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-83977-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:36.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:35.360167+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e260: 8 total, 8 up, 6 in 2024-02-13T00:43:36.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:35.377708+0000 mon.b (mon.1) 377 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:36.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:35.379892+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:36.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:35.383685+0000 mon.c (mon.2) 245 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-83977-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:36.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:35.385275+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-83977-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:36.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:35.473374+0000 mgr.y (mgr.14544) 262 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:36.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:36.228 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:43:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:43:36] "GET /metrics HTTP/1.1" 200 186206 "" "Prometheus/2.33.4" 2024-02-13T00:43:36.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:36.361 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:36.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:36.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:36.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:36.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:36.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:35.204604+0000 mon.b (mon.1) 376 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:36.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:35.305845+0000 osd.0 (osd.0) 95 : cluster [DBG] 2.f scrub starts 2024-02-13T00:43:36.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:35.309304+0000 osd.0 (osd.0) 96 : cluster [DBG] 2.f scrub ok 2024-02-13T00:43:36.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:35.359843+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-83985-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-83985-46"}]': finished 2024-02-13T00:43:36.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:35.360012+0000 mon.a (mon.0) 1747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:36.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:35.360100+0000 mon.a (mon.0) 1748 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-83977-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:36.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:35.360167+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e260: 8 total, 8 up, 6 in 2024-02-13T00:43:36.531 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:35.377708+0000 mon.b (mon.1) 377 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:36.532 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:35.379892+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:36.532 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:35.383685+0000 mon.c (mon.2) 245 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-83977-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:36.532 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:35.385275+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-83977-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:36.532 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:35.473374+0000 mgr.y (mgr.14544) 262 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:36.532 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:36.550 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:36.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:36.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:36.679 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:36.698 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:36.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:36.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:36.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:36.829 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:37.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:43:37.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:36.205941+0000 mon.b (mon.1) 378 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:37.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:36.363655+0000 mon.a (mon.0) 1752 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:37.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:36.366767+0000 mgr.y (mgr.14544) 263 : cluster [DBG] pgmap v362: 278 pgs: 32 creating+peering, 2 active+clean+snaptrim, 102 peering, 8 unknown, 6 active+clean+snaptrim_wait, 128 active+clean; 9.1 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:37.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:36.382444+0000 mon.b (mon.1) 379 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-29"}]: dispatch 2024-02-13T00:43:37.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:36.651621+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e261: 8 total, 8 up, 6 in 2024-02-13T00:43:37.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:36.653274+0000 mon.a (mon.0) 1754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-29"}]: dispatch 2024-02-13T00:43:37.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:36.205941+0000 mon.b (mon.1) 378 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:37.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:36.363655+0000 mon.a (mon.0) 1752 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:37.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:36.366767+0000 mgr.y (mgr.14544) 263 : cluster [DBG] pgmap v362: 278 pgs: 32 creating+peering, 2 active+clean+snaptrim, 102 peering, 8 unknown, 6 active+clean+snaptrim_wait, 128 active+clean; 9.1 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:37.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:36.382444+0000 mon.b (mon.1) 379 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-29"}]: dispatch 2024-02-13T00:43:37.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:36.651621+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e261: 8 total, 8 up, 6 in 2024-02-13T00:43:37.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:36.653274+0000 mon.a (mon.0) 1754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-29"}]: dispatch 2024-02-13T00:43:37.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:36.205941+0000 mon.b (mon.1) 378 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:37.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:36.363655+0000 mon.a (mon.0) 1752 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:37.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:36.366767+0000 mgr.y (mgr.14544) 263 : cluster [DBG] pgmap v362: 278 pgs: 32 creating+peering, 2 active+clean+snaptrim, 102 peering, 8 unknown, 6 active+clean+snaptrim_wait, 128 active+clean; 9.1 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:37.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:36.382444+0000 mon.b (mon.1) 379 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-29"}]: dispatch 2024-02-13T00:43:37.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:36.651621+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e261: 8 total, 8 up, 6 in 2024-02-13T00:43:37.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:36.653274+0000 mon.a (mon.0) 1754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-29"}]: dispatch 2024-02-13T00:43:37.776 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:38.011 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:38.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:38.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:38.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:38.149 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:38.278 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:38.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:37.206725+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:38.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:37.358302+0000 mon.a (mon.0) 1755 : cluster [WRN] Health check failed: Reduced data availability: 25 pgs inactive, 38 pgs peering (PG_AVAILABILITY) 2024-02-13T00:43:38.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:37.367467+0000 mon.a (mon.0) 1756 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-83977-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-83977-37"}]': finished 2024-02-13T00:43:38.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:37.367590+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-29"}]': finished 2024-02-13T00:43:38.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:37.367658+0000 mon.a (mon.0) 1758 : cluster [DBG] osdmap e262: 8 total, 8 up, 6 in 2024-02-13T00:43:38.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:37.371125+0000 mon.b (mon.1) 381 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-29", "mode": "writeback"}]: dispatch 2024-02-13T00:43:38.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:37.372529+0000 mon.a (mon.0) 1759 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-29", "mode": "writeback"}]: dispatch 2024-02-13T00:43:38.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:37.374068+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:38.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:37.206725+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:38.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:37.358302+0000 mon.a (mon.0) 1755 : cluster [WRN] Health check failed: Reduced data availability: 25 pgs inactive, 38 pgs peering (PG_AVAILABILITY) 2024-02-13T00:43:38.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:37.367467+0000 mon.a (mon.0) 1756 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-83977-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-83977-37"}]': finished 2024-02-13T00:43:38.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:37.367590+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-29"}]': finished 2024-02-13T00:43:38.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:37.367658+0000 mon.a (mon.0) 1758 : cluster [DBG] osdmap e262: 8 total, 8 up, 6 in 2024-02-13T00:43:38.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:37.371125+0000 mon.b (mon.1) 381 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-29", "mode": "writeback"}]: dispatch 2024-02-13T00:43:38.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:37.372529+0000 mon.a (mon.0) 1759 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-29", "mode": "writeback"}]: dispatch 2024-02-13T00:43:38.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:37.374068+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:38.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:38.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:38.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:37.206725+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:38.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:37.358302+0000 mon.a (mon.0) 1755 : cluster [WRN] Health check failed: Reduced data availability: 25 pgs inactive, 38 pgs peering (PG_AVAILABILITY) 2024-02-13T00:43:38.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:37.367467+0000 mon.a (mon.0) 1756 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-83977-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-83977-37"}]': finished 2024-02-13T00:43:38.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:37.367590+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-29"}]': finished 2024-02-13T00:43:38.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:37.367658+0000 mon.a (mon.0) 1758 : cluster [DBG] osdmap e262: 8 total, 8 up, 6 in 2024-02-13T00:43:38.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:37.371125+0000 mon.b (mon.1) 381 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-29", "mode": "writeback"}]: dispatch 2024-02-13T00:43:38.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:37.372529+0000 mon.a (mon.0) 1759 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-29", "mode": "writeback"}]: dispatch 2024-02-13T00:43:38.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:37.374068+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:38.412 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:38.434 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:38.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:38.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:38.569 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:38.794 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:43:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:43:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:43:38.918 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:38.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:39.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:39.047 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:39.066 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:39.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:39.170 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:39.195 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:38.207561+0000 mon.b (mon.1) 382 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:39.195 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:38.364802+0000 mon.a (mon.0) 1761 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:39.195 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:38.367474+0000 mgr.y (mgr.14544) 264 : cluster [DBG] pgmap v364: 278 pgs: 32 creating+peering, 2 active+clean+snaptrim, 102 peering, 8 unknown, 6 active+clean+snaptrim_wait, 128 active+clean; 9.1 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:39.195 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:38.369562+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-29", "mode": "writeback"}]': finished 2024-02-13T00:43:39.195 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:38.369631+0000 mon.a (mon.0) 1763 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-46"}]': finished 2024-02-13T00:43:39.195 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:38.369682+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e263: 8 total, 8 up, 6 in 2024-02-13T00:43:39.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:38.372988+0000 mon.a (mon.0) 1765 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:39.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:38.377642+0000 osd.1 (osd.1) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:39.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:38.377844+0000 osd.1 (osd.1) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:39.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:38.450983+0000 mon.b (mon.1) 383 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:39.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:38.451605+0000 mon.a (mon.0) 1766 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:39.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:38.725564+0000 mon.b (mon.1) 384 : audit [INF] from='client.? 172.21.15.162:0/1569800268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:43:39.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:38.726169+0000 mon.a (mon.0) 1767 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:43:39.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:38.870387+0000 osd.5 (osd.5) 117 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:39.196 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:38.870578+0000 osd.5 (osd.5) 118 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:39.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:38.207561+0000 mon.b (mon.1) 382 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:39.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:38.364802+0000 mon.a (mon.0) 1761 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:39.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:38.367474+0000 mgr.y (mgr.14544) 264 : cluster [DBG] pgmap v364: 278 pgs: 32 creating+peering, 2 active+clean+snaptrim, 102 peering, 8 unknown, 6 active+clean+snaptrim_wait, 128 active+clean; 9.1 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:39.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:38.369562+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-29", "mode": "writeback"}]': finished 2024-02-13T00:43:39.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:38.369631+0000 mon.a (mon.0) 1763 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-46"}]': finished 2024-02-13T00:43:39.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:38.369682+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e263: 8 total, 8 up, 6 in 2024-02-13T00:43:39.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:38.372988+0000 mon.a (mon.0) 1765 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:39.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:38.377642+0000 osd.1 (osd.1) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:39.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:38.377844+0000 osd.1 (osd.1) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:39.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:38.450983+0000 mon.b (mon.1) 383 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:39.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:38.451605+0000 mon.a (mon.0) 1766 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:39.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:38.725564+0000 mon.b (mon.1) 384 : audit [INF] from='client.? 172.21.15.162:0/1569800268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:43:39.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:38.726169+0000 mon.a (mon.0) 1767 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:43:39.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:38.870387+0000 osd.5 (osd.5) 117 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:39.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:38.870578+0000 osd.5 (osd.5) 118 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:39.198 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:39.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.524+0000 7ff0f2966700 1 -- 172.21.15.162:0/2798712361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0ec0fcbb0 msgr2=0x7ff0ec0fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:39.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.524+0000 7ff0f2966700 1 --2- 172.21.15.162:0/2798712361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0ec0fcbb0 0x7ff0ec0fcf90 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 rx=0x7ff0dc0047b0 tx=0x7ff0dc009cb0).stop 2024-02-13T00:43:39.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.524+0000 7ff0f2966700 1 -- 172.21.15.162:0/2798712361 shutdown_connections 2024-02-13T00:43:39.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.524+0000 7ff0f2966700 1 --2- 172.21.15.162:0/2798712361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0ec0f7990 0x7ff0ec1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:39.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.524+0000 7ff0f2966700 1 --2- 172.21.15.162:0/2798712361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0ec0fcbb0 0x7ff0ec0fcf90 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:39.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.524+0000 7ff0f2966700 1 --2- 172.21.15.162:0/2798712361 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff0ec0fd510 0x7ff0ec0f72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:39.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.524+0000 7ff0f2966700 1 -- 172.21.15.162:0/2798712361 >> 172.21.15.162:0/2798712361 conn(0x7ff0ec070080 msgr2=0x7ff0ec0f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:39.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.524+0000 7ff0f2966700 1 -- 172.21.15.162:0/2798712361 shutdown_connections 2024-02-13T00:43:39.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.524+0000 7ff0f2966700 1 -- 172.21.15.162:0/2798712361 wait complete. 2024-02-13T00:43:39.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.524+0000 7ff0f2966700 1 librados: starting msgr at 2024-02-13T00:43:39.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.524+0000 7ff0f2966700 1 librados: starting objecter 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0f2966700 1 Processor -- start 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0f2966700 1 -- start start 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0f2966700 1 librados: setting wanted keys 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0f2966700 1 librados: calling monclient init 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0f2966700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0ec0f7990 0x7ff0ec10d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0f2966700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0ec0fcbb0 0x7ff0ec10d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0f2966700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff0ec0fd510 0x7ff0ec107340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0f2966700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec107830 con 0x7ff0ec0f7990 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0f2966700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec1079a0 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0f2966700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec107ae0 con 0x7ff0ec0fd510 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0ebfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0ec0f7990 0x7ff0ec10d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0ebfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0ec0f7990 0x7ff0ec10d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45504/0 (socket says 172.21.15.162:45504) 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0ebfff700 1 -- 172.21.15.162:0/1569800268 learned_addr learned my addr 172.21.15.162:0/1569800268 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:39.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0f0f03700 1 --2- 172.21.15.162:0/1569800268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff0ec0fd510 0x7ff0ec107340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.525+0000 7ff0eb7fe700 1 --2- 172.21.15.162:0/1569800268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0ec0fcbb0 0x7ff0ec10d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0eb7fe700 1 -- 172.21.15.162:0/1569800268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff0ec0fd510 msgr2=0x7ff0ec107340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0eb7fe700 1 --2- 172.21.15.162:0/1569800268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff0ec0fd510 0x7ff0ec107340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0eb7fe700 1 -- 172.21.15.162:0/1569800268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0ec0f7990 msgr2=0x7ff0ec10d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0eb7fe700 1 --2- 172.21.15.162:0/1569800268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0ec0f7990 0x7ff0ec10d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0eb7fe700 1 -- 172.21.15.162:0/1569800268 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e0005040 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0ebfff700 1 --2- 172.21.15.162:0/1569800268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0ec0f7990 0x7ff0ec10d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0f0f03700 1 --2- 172.21.15.162:0/1569800268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff0ec0fd510 0x7ff0ec107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0eb7fe700 1 --2- 172.21.15.162:0/1569800268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0ec0fcbb0 0x7ff0ec10d690 secure :-1 s=READY pgs=4849 cs=0 l=1 rev1=1 rx=0x7ff0e00091f0 tx=0x7ff0e0009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0e97fa700 10 client.?.objecter ms_handle_connect 0x7ff0ec0fcbb0 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0e97fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0e97fa700 1 -- 172.21.15.162:0/1569800268 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e00079e0 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0f2966700 1 -- 172.21.15.162:0/1569800268 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0dc005040 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0e97fa700 1 -- 172.21.15.162:0/1569800268 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff0e000f040 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0e97fa700 1 -- 172.21.15.162:0/1569800268 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0014950 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0f2966700 10 client.44110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0f2966700 1 -- 172.21.15.162:0/1569800268 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ec108170 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.526+0000 7ff0f2966700 1 librados: init done 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.527+0000 7ff0f2966700 1 -- 172.21.15.162:0/1569800268 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ec05fb80 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.528+0000 7ff0e97fa700 1 -- 172.21.15.162:0/1569800268 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff0e00104f0 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.528+0000 7ff0e97fa700 1 --2- 172.21.15.162:0/1569800268 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff0d4062c90 0x7ff0d4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.528+0000 7ff0e97fa700 1 -- 172.21.15.162:0/1569800268 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(263..263 src has 52..263) v4 ==== 9326+0+0 (secure 0 0 0) 0x7ff0e007d190 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.528+0000 7ff0e97fa700 10 client.44110.objecter ms_dispatch 0x7ff0ec001550 osd_map(263..263 src has 52..263) v4 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.528+0000 7ff0e97fa700 3 client.44110.objecter handle_osd_map got epochs [263,263] > 0 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.528+0000 7ff0e97fa700 3 client.44110.objecter handle_osd_map decoding full epoch 263 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.529+0000 7ff0e97fa700 20 client.44110.objecter dump_active .. 0 homeless 2024-02-13T00:43:39.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.529+0000 7ff0ebfff700 1 --2- 172.21.15.162:0/1569800268 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff0d4062c90 0x7ff0d4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.529+0000 7ff0e97fa700 10 client.44110.objecter ms_handle_connect 0x7ff0d4062c90 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.529+0000 7ff0ebfff700 1 --2- 172.21.15.162:0/1569800268 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff0d4062c90 0x7ff0d4065150 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 rx=0x7ff0dc009f20 tx=0x7ff0dc009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.532+0000 7ff0e97fa700 1 -- 172.21.15.162:0/1569800268 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff0e0049f60 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:38.724+0000 7ff0f2966700 1 -- 172.21.15.162:0/1569800268 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7ff0ec0475e0 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.372+0000 7ff0e97fa700 1 -- 172.21.15.162:0/1569800268 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v264) v1 ==== 101+0+0 (secure 0 0 0) 0x7ff0e00501f0 con 0x7ff0ec0fcbb0 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.376+0000 7ff0f2966700 10 librados: watch_flush enter 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.377+0000 7ff0f2966700 10 librados: watch_flush exit 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.377+0000 7ff0f2966700 20 client.44110.objecter shutdown clearing up homeless session... 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.377+0000 7ff0f2966700 10 client.44110.objecter successfully canceled tick 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 -- 172.21.15.162:0/1569800268 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff0d4062c90 msgr2=0x7ff0d4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 --2- 172.21.15.162:0/1569800268 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff0d4062c90 0x7ff0d4065150 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 rx=0x7ff0dc009f20 tx=0x7ff0dc009cb0).stop 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 -- 172.21.15.162:0/1569800268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0ec0fcbb0 msgr2=0x7ff0ec10d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 --2- 172.21.15.162:0/1569800268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0ec0fcbb0 0x7ff0ec10d690 secure :-1 s=READY pgs=4849 cs=0 l=1 rev1=1 rx=0x7ff0e00091f0 tx=0x7ff0e0009cb0).stop 2024-02-13T00:43:39.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 -- 172.21.15.162:0/1569800268 shutdown_connections 2024-02-13T00:43:39.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 --2- 172.21.15.162:0/1569800268 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff0d4062c90 0x7ff0d4065150 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:39.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 --2- 172.21.15.162:0/1569800268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff0ec0fd510 0x7ff0ec107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:39.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 --2- 172.21.15.162:0/1569800268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0ec0f7990 0x7ff0ec10d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:39.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 --2- 172.21.15.162:0/1569800268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0ec0fcbb0 0x7ff0ec10d690 unknown :-1 s=CLOSED pgs=4849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:39.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 -- 172.21.15.162:0/1569800268 >> 172.21.15.162:0/1569800268 conn(0x7ff0ec070080 msgr2=0x7ff0ec0fa9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:39.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 -- 172.21.15.162:0/1569800268 shutdown_connections 2024-02-13T00:43:39.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 -- 172.21.15.162:0/1569800268 wait complete. 2024-02-13T00:43:39.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:39.380+0000 7ff0f2966700 1 librados: shutdown 2024-02-13T00:43:39.431 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:43:39.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:38.207561+0000 mon.b (mon.1) 382 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:39.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:38.364802+0000 mon.a (mon.0) 1761 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:39.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:38.367474+0000 mgr.y (mgr.14544) 264 : cluster [DBG] pgmap v364: 278 pgs: 32 creating+peering, 2 active+clean+snaptrim, 102 peering, 8 unknown, 6 active+clean+snaptrim_wait, 128 active+clean; 9.1 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:39.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:38.369562+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-29", "mode": "writeback"}]': finished 2024-02-13T00:43:39.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:38.369631+0000 mon.a (mon.0) 1763 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-83985-46"}]': finished 2024-02-13T00:43:39.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:38.369682+0000 mon.a (mon.0) 1764 : cluster [DBG] osdmap e263: 8 total, 8 up, 6 in 2024-02-13T00:43:39.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:38.372988+0000 mon.a (mon.0) 1765 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-46"}]: dispatch 2024-02-13T00:43:39.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:38.377642+0000 osd.1 (osd.1) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:39.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:38.377844+0000 osd.1 (osd.1) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:39.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:38.450983+0000 mon.b (mon.1) 383 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:39.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:38.451605+0000 mon.a (mon.0) 1766 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:39.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:38.725564+0000 mon.b (mon.1) 384 : audit [INF] from='client.? 172.21.15.162:0/1569800268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:43:39.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:38.726169+0000 mon.a (mon.0) 1767 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:43:39.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:38.870387+0000 osd.5 (osd.5) 117 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:39.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:38.870578+0000 osd.5 (osd.5) 118 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:39.651 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:39.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:39.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:39.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:39.782 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:39.800 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:39.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:39.904 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:39.931 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:38.595415+0000 osd.4 (osd.4) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:38.595606+0000 osd.4 (osd.4) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:39.208930+0000 mon.b (mon.1) 385 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:39.366710+0000 mon.a (mon.0) 1768 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:43:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:39.372255+0000 mon.a (mon.0) 1769 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-46"}]': finished 2024-02-13T00:43:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:39.372321+0000 mon.a (mon.0) 1770 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:39.372388+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:43:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:39.372443+0000 mon.a (mon.0) 1772 : cluster [DBG] osdmap e264: 8 total, 8 up, 6 in 2024-02-13T00:43:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:39.373572+0000 mon.b (mon.1) 386 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29"}]: dispatch 2024-02-13T00:43:40.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:39.374156+0000 mon.a (mon.0) 1773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29"}]: dispatch 2024-02-13T00:43:40.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:39.381529+0000 mon.c (mon.2) 246 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:40.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:39.388120+0000 mon.a (mon.0) 1774 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:40.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:39.400496+0000 mon.a (mon.0) 1775 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:40.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:39.401310+0000 mon.a (mon.0) 1776 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:40.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:39.401668+0000 mon.a (mon.0) 1777 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-83985-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:40.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:38.595415+0000 osd.4 (osd.4) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:40.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:38.595606+0000 osd.4 (osd.4) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:40.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:39.208930+0000 mon.b (mon.1) 385 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:40.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:39.366710+0000 mon.a (mon.0) 1768 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:43:40.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:39.372255+0000 mon.a (mon.0) 1769 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-46"}]': finished 2024-02-13T00:43:40.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:39.372321+0000 mon.a (mon.0) 1770 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:40.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:39.372388+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:43:40.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:39.372443+0000 mon.a (mon.0) 1772 : cluster [DBG] osdmap e264: 8 total, 8 up, 6 in 2024-02-13T00:43:40.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:39.373572+0000 mon.b (mon.1) 386 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29"}]: dispatch 2024-02-13T00:43:40.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:39.374156+0000 mon.a (mon.0) 1773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29"}]: dispatch 2024-02-13T00:43:40.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:39.381529+0000 mon.c (mon.2) 246 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:40.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:39.388120+0000 mon.a (mon.0) 1774 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:40.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:39.400496+0000 mon.a (mon.0) 1775 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:40.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:39.401310+0000 mon.a (mon.0) 1776 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:40.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:39.401668+0000 mon.a (mon.0) 1777 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-83985-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:38.595415+0000 osd.4 (osd.4) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:38.595606+0000 osd.4 (osd.4) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:39.208930+0000 mon.b (mon.1) 385 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:39.366710+0000 mon.a (mon.0) 1768 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:43:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:39.372255+0000 mon.a (mon.0) 1769 : audit [INF] from='client.? 172.21.15.162:0/1602322330' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-83985-46"}]': finished 2024-02-13T00:43:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:39.372321+0000 mon.a (mon.0) 1770 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:39.372388+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:43:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:39.372443+0000 mon.a (mon.0) 1772 : cluster [DBG] osdmap e264: 8 total, 8 up, 6 in 2024-02-13T00:43:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:39.373572+0000 mon.b (mon.1) 386 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29"}]: dispatch 2024-02-13T00:43:40.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:39.374156+0000 mon.a (mon.0) 1773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29"}]: dispatch 2024-02-13T00:43:40.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:39.381529+0000 mon.c (mon.2) 246 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:40.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:39.388120+0000 mon.a (mon.0) 1774 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:40.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:39.400496+0000 mon.a (mon.0) 1775 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:40.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:39.401310+0000 mon.a (mon.0) 1776 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:40.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:39.401668+0000 mon.a (mon.0) 1777 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-83985-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:40.682 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:40.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:40.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:40.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:40.818 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:40.839 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:40.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:40.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:40.974 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:40.987 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7] out_osds: [6, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:43:40.987 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:43:40.987 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-02-13T00:43:40.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.0 reset_purged_snaps_last 2024-02-13T00:43:40.994 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:40.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:41.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:41.125 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:41.199 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:41.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:41.206 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:43:41.206 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:43:41.206 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:43:41.206 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:43:41.206 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:43:41.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:41.328 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:40.210075+0000 mon.b (mon.1) 387 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:41.328 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:40.241465+0000 osd.1 (osd.1) 111 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:43:41.328 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:40.244926+0000 osd.1 (osd.1) 112 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:43:41.328 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:40.332848+0000 osd.0 (osd.0) 97 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:43:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:40.335887+0000 osd.0 (osd.0) 98 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:43:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:40.342350+0000 mon.a (mon.0) 1778 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:40.349678+0000 mon.a (mon.0) 1779 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29"}]': finished 2024-02-13T00:43:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:40.349755+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]': finished 2024-02-13T00:43:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:40.349828+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-83985-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:40.349885+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e265: 8 total, 8 up, 6 in 2024-02-13T00:43:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:40.353591+0000 mon.a (mon.0) 1783 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-83985-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:40.353759+0000 mon.c (mon.2) 247 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:40.356389+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:40.367886+0000 mgr.y (mgr.14544) 265 : cluster [DBG] pgmap v368: 270 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 260 active+clean; 9.0 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 10 op/s; 981 B/s, 70 objects/s recovering 2024-02-13T00:43:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:40.483505+0000 osd.0 (osd.0) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:41.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:40.483670+0000 osd.0 (osd.0) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:40.210075+0000 mon.b (mon.1) 387 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:40.241465+0000 osd.1 (osd.1) 111 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:40.244926+0000 osd.1 (osd.1) 112 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:40.332848+0000 osd.0 (osd.0) 97 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:40.335887+0000 osd.0 (osd.0) 98 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:40.342350+0000 mon.a (mon.0) 1778 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:40.349678+0000 mon.a (mon.0) 1779 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29"}]': finished 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:40.349755+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]': finished 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:40.349828+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-83985-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:40.349885+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e265: 8 total, 8 up, 6 in 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:40.353591+0000 mon.a (mon.0) 1783 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-83985-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:40.353759+0000 mon.c (mon.2) 247 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:41.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:40.356389+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:41.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:40.367886+0000 mgr.y (mgr.14544) 265 : cluster [DBG] pgmap v368: 270 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 260 active+clean; 9.0 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 10 op/s; 981 B/s, 70 objects/s recovering 2024-02-13T00:43:41.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:40.483505+0000 osd.0 (osd.0) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:41.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:40.483670+0000 osd.0 (osd.0) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:41.331 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:41.355 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:41.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:41.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:41.497 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:40.210075+0000 mon.b (mon.1) 387 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:40.241465+0000 osd.1 (osd.1) 111 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:40.244926+0000 osd.1 (osd.1) 112 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:40.332848+0000 osd.0 (osd.0) 97 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:40.335887+0000 osd.0 (osd.0) 98 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:40.342350+0000 mon.a (mon.0) 1778 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:40.349678+0000 mon.a (mon.0) 1779 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-29"}]': finished 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:40.349755+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-83977-37"}]': finished 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:40.349828+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-83985-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:40.349885+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e265: 8 total, 8 up, 6 in 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:40.353591+0000 mon.a (mon.0) 1783 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-83985-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:40.353759+0000 mon.c (mon.2) 247 : audit [INF] from='client.? 172.21.15.162:0/605281806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:40.356389+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]: dispatch 2024-02-13T00:43:41.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:40.367886+0000 mgr.y (mgr.14544) 265 : cluster [DBG] pgmap v368: 270 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 260 active+clean; 9.0 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 10 op/s; 981 B/s, 70 objects/s recovering 2024-02-13T00:43:41.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:40.483505+0000 osd.0 (osd.0) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:41.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:40.483670+0000 osd.0 (osd.0) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:41.499 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:41.520 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:41.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:41.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:41.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:41.653 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:41.672 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:41.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:41.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:41.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.462+0000 7ff126e79700 1 -- 172.21.15.162:0/2941075886 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1200fe570 msgr2=0x7ff1200f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.462+0000 7ff126e79700 1 --2- 172.21.15.162:0/2941075886 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1200fe570 0x7ff1200f7da0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 rx=0x7ff10c004870 tx=0x7ff10c009cb0).stop 2024-02-13T00:43:41.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.462+0000 7ff126e79700 1 -- 172.21.15.162:0/2941075886 shutdown_connections 2024-02-13T00:43:41.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.462+0000 7ff126e79700 1 --2- 172.21.15.162:0/2941075886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1200f8480 0x7ff120104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.462+0000 7ff126e79700 1 --2- 172.21.15.162:0/2941075886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1200fdc10 0x7ff1200fdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.462+0000 7ff126e79700 1 --2- 172.21.15.162:0/2941075886 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1200fe570 0x7ff1200f7da0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.462+0000 7ff126e79700 1 -- 172.21.15.162:0/2941075886 >> 172.21.15.162:0/2941075886 conn(0x7ff1200f3800 msgr2=0x7ff1200f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:41.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.462+0000 7ff126e79700 1 -- 172.21.15.162:0/2941075886 shutdown_connections 2024-02-13T00:43:41.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.462+0000 7ff126e79700 1 -- 172.21.15.162:0/2941075886 wait complete. 2024-02-13T00:43:41.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.463+0000 7ff126e79700 1 librados: starting msgr at 2024-02-13T00:43:41.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.463+0000 7ff126e79700 1 librados: starting objecter 2024-02-13T00:43:41.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.463+0000 7ff126e79700 1 Processor -- start 2024-02-13T00:43:41.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.463+0000 7ff126e79700 1 -- start start 2024-02-13T00:43:41.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.463+0000 7ff126e79700 1 librados: setting wanted keys 2024-02-13T00:43:41.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.463+0000 7ff126e79700 1 librados: calling monclient init 2024-02-13T00:43:41.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.463+0000 7ff126e79700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1200f8480 0x7ff12010e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:41.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.463+0000 7ff126e79700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1200fdc10 0x7ff12010e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:41.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.463+0000 7ff126e79700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1200fe570 0x7ff120108310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:41.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.463+0000 7ff126e79700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff120108800 con 0x7ff1200fdc10 2024-02-13T00:43:41.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.463+0000 7ff126e79700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff120108970 con 0x7ff1200fe570 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.463+0000 7ff126e79700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff120108b30 con 0x7ff1200f8480 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff124c15700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1200f8480 0x7ff12010e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff124c15700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1200f8480 0x7ff12010e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:40980/0 (socket says 172.21.15.162:40980) 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff124c15700 1 -- 172.21.15.162:0/2377249329 learned_addr learned my addr 172.21.15.162:0/2377249329 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff11ffff700 1 --2- 172.21.15.162:0/2377249329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1200fdc10 0x7ff12010e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff125416700 1 --2- 172.21.15.162:0/2377249329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1200fe570 0x7ff120108310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff124c15700 1 -- 172.21.15.162:0/2377249329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1200fe570 msgr2=0x7ff120108310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff124c15700 1 --2- 172.21.15.162:0/2377249329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1200fe570 0x7ff120108310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff124c15700 1 -- 172.21.15.162:0/2377249329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1200fdc10 msgr2=0x7ff12010e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff124c15700 1 --2- 172.21.15.162:0/2377249329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1200fdc10 0x7ff12010e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff124c15700 1 -- 172.21.15.162:0/2377249329 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10c005040 con 0x7ff1200f8480 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff124c15700 1 --2- 172.21.15.162:0/2377249329 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1200f8480 0x7ff12010e170 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7ff11400e970 tx=0x7ff11400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff11dffb700 10 client.?.objecter ms_handle_connect 0x7ff1200f8480 2024-02-13T00:43:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff11dffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114004660 con 0x7ff1200f8480 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff11400f040 con 0x7ff1200f8480 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff120108e10 con 0x7ff1200f8480 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114014790 con 0x7ff1200f8480 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff126e79700 10 client.37013.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff120109190 con 0x7ff1200f8480 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.464+0000 7ff126e79700 1 librados: init done 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.465+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff12005fb20 con 0x7ff1200f8480 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff1140090d0 con 0x7ff1200f8480 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff11dffb700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff110062c90 0x7ff110065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(266..266 src has 52..266) v4 ==== 8422+0+0 (secure 0 0 0) 0x7ff11407ce90 con 0x7ff1200f8480 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 osd_map(266..266 src has 52..266) v4 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff11dffb700 3 client.37013.objecter handle_osd_map got epochs [266,266] > 0 2024-02-13T00:43:41.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff11dffb700 3 client.37013.objecter handle_osd_map decoding full epoch 266 2024-02-13T00:43:41.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff11ffff700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff110062c90 0x7ff110065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:41.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff11dffb700 20 client.37013.objecter dump_active .. 0 homeless 2024-02-13T00:43:41.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v266) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff11407f560 con 0x7ff1200f8480 2024-02-13T00:43:41.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff11dffb700 10 client.37013.objecter ms_handle_connect 0x7ff110062c90 2024-02-13T00:43:41.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff11ffff700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff110062c90 0x7ff110065150 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 rx=0x7ff10c004ee0 tx=0x7ff10c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:41.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff126e79700 10 client.37013.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:41.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 1 2024-02-13T00:43:41.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff120028590 0x7ff1200fe950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:41.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1201071b0 osd=0 3 2024-02-13T00:43:41.712 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:43:41.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201071b0 osd=0 3 2024-02-13T00:43:41.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1201071b0 osd=0 3 2024-02-13T00:43:41.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 1 2024-02-13T00:43:41.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff1201071b0 osd=0 3 2024-02-13T00:43:41.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 0 1 2024-02-13T00:43:41.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201071b0 osd=0 4 2024-02-13T00:43:41.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff126e79700 10 client.37013.objecter _send_command 1 2024-02-13T00:43:41.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1200475e0 con 0x7ff120028590 2024-02-13T00:43:41.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.466+0000 7ff125416700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff120028590 0x7ff1200fe950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:41.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.467+0000 7ff11dffb700 10 client.37013.objecter ms_handle_connect 0x7ff120028590 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.467+0000 7ff125416700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff120028590 0x7ff1200fe950 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.467+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff1200475e0 con 0x7ff120028590 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.467+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.467+0000 7ff11dffb700 10 client.37013.objecter _finish_command 1 = system:0 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.467+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff1201071b0 osd=0 4 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.467+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 0 1 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.483+0000 7ff126e79700 10 client.37013.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.483+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 2 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.483+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1201071b0 osd=0 3 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.483+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201071b0 osd=0 3 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.483+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1201071b0 osd=0 3 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.483+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 2 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.483+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff1201071b0 osd=0 3 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.483+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 0 2 2024-02-13T00:43:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.483+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201071b0 osd=0 4 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.483+0000 7ff126e79700 10 client.37013.objecter _send_command 2 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.483+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff12005a0e0 con 0x7ff120028590 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff12005a0e0 con 0x7ff120028590 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff11dffb700 10 client.37013.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff1201071b0 osd=0 4 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 0 2 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff126e79700 10 client.37013.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 3 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6805/150599780 conn(0x7ff120027be0 0x7ff120028010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1200fffc0 osd=1 3 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1200fffc0 osd=1 3 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1200fffc0 osd=1 3 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 3 2024-02-13T00:43:41.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff1200fffc0 osd=1 3 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 1 3 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1200fffc0 osd=1 4 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff126e79700 10 client.37013.objecter _send_command 3 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.484+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff120100130 con 0x7ff120027be0 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.485+0000 7ff124c15700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6805/150599780 conn(0x7ff120027be0 0x7ff120028010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.485+0000 7ff11dffb700 10 client.37013.objecter ms_handle_connect 0x7ff120027be0 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.485+0000 7ff124c15700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6805/150599780 conn(0x7ff120027be0 0x7ff120028010 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.485+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff120100130 con 0x7ff120027be0 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.485+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.485+0000 7ff11dffb700 10 client.37013.objecter _finish_command 3 = system:0 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.485+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff1200fffc0 osd=1 4 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.485+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 1 3 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.500+0000 7ff126e79700 10 client.37013.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.500+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 4 2024-02-13T00:43:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.500+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1200fffc0 osd=1 3 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.500+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1200fffc0 osd=1 3 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.500+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1200fffc0 osd=1 3 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.500+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 4 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.500+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff1200fffc0 osd=1 3 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.500+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 1 4 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.500+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1200fffc0 osd=1 4 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.500+0000 7ff126e79700 10 client.37013.objecter _send_command 4 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.500+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff120027650 con 0x7ff120027be0 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff120027650 con 0x7ff120027be0 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff11dffb700 10 client.37013.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff1200fffc0 osd=1 4 2024-02-13T00:43:41.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 1 4 2024-02-13T00:43:41.718 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:43:41.718 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:43:41.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff126e79700 10 client.37013.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:41.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 5 2024-02-13T00:43:41.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff1200ffb20 0x7ff12011c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:41.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff120027a60 osd=2 3 2024-02-13T00:43:41.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120027a60 osd=2 3 2024-02-13T00:43:41.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff120027a60 osd=2 3 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 5 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff120027a60 osd=2 3 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 2 5 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120027a60 osd=2 4 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff126e79700 10 client.37013.objecter _send_command 5 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff12011f080 con 0x7ff1200ffb20 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff11ffff700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff1200ffb20 0x7ff12011c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff11dffb700 10 client.37013.objecter ms_handle_connect 0x7ff1200ffb20 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.502+0000 7ff11ffff700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff1200ffb20 0x7ff12011c370 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.503+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff12011f080 con 0x7ff1200ffb20 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.503+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.503+0000 7ff11dffb700 10 client.37013.objecter _finish_command 5 = system:0 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.503+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff120027a60 osd=2 4 2024-02-13T00:43:41.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.503+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 2 5 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.517+0000 7ff126e79700 10 client.37013.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.517+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 6 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.517+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff120027a60 osd=2 3 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.517+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120027a60 osd=2 3 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.517+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff120027a60 osd=2 3 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.517+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 6 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.517+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff120027a60 osd=2 3 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.517+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 2 6 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.517+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120027a60 osd=2 4 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.517+0000 7ff126e79700 10 client.37013.objecter _send_command 6 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.517+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff1200ff080 con 0x7ff1200ffb20 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.518+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff1200ff080 con 0x7ff1200ffb20 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.518+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:41.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.518+0000 7ff11dffb700 10 client.37013.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.518+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff120027a60 osd=2 4 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.518+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 2 6 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff126e79700 10 client.37013.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 7 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6813/604971011 conn(0x7ff120119ec0 0x7ff12011d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1200ff490 osd=3 3 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1200ff490 osd=3 3 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1200ff490 osd=3 3 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 7 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff1200ff490 osd=3 3 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 3 7 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1200ff490 osd=3 4 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff126e79700 10 client.37013.objecter _send_command 7 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff12011dfc0 con 0x7ff120119ec0 2024-02-13T00:43:41.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff125416700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6813/604971011 conn(0x7ff120119ec0 0x7ff12011d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:41.722 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:43:41.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff11dffb700 10 client.37013.objecter ms_handle_connect 0x7ff120119ec0 2024-02-13T00:43:41.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.519+0000 7ff125416700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6813/604971011 conn(0x7ff120119ec0 0x7ff12011d9e0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.520+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff12011dfc0 con 0x7ff120119ec0 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.520+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.520+0000 7ff11dffb700 10 client.37013.objecter _finish_command 7 = system:0 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.520+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff1200ff490 osd=3 4 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.520+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 3 7 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.534+0000 7ff126e79700 10 client.37013.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.534+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 8 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.534+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1200ff490 osd=3 3 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.534+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1200ff490 osd=3 3 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.534+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1200ff490 osd=3 3 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.534+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 8 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.534+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff1200ff490 osd=3 3 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.534+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 3 8 2024-02-13T00:43:41.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.534+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1200ff490 osd=3 4 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.534+0000 7ff126e79700 10 client.37013.objecter _send_command 8 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.534+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff120119420 con 0x7ff120119ec0 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff120119420 con 0x7ff120119ec0 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff11dffb700 10 client.37013.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff1200ff490 osd=3 4 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 3 8 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff126e79700 10 client.37013.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 9 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff120121260 0x7ff120123650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1201195c0 osd=4 3 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201195c0 osd=4 3 2024-02-13T00:43:41.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1201195c0 osd=4 3 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 9 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff1201195c0 osd=4 3 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 4 9 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201195c0 osd=4 4 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff126e79700 10 client.37013.objecter _send_command 9 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.535+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff120119730 con 0x7ff120121260 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.536+0000 7ff124c15700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff120121260 0x7ff120123650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.536+0000 7ff11dffb700 10 client.37013.objecter ms_handle_connect 0x7ff120121260 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.536+0000 7ff124c15700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff120121260 0x7ff120123650 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.537+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff120119730 con 0x7ff120121260 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.537+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.537+0000 7ff11dffb700 10 client.37013.objecter _finish_command 9 = system:0 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.537+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff1201195c0 osd=4 4 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.537+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 4 9 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.551+0000 7ff126e79700 10 client.37013.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:41.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.551+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 10 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.551+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1201195c0 osd=4 3 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.551+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201195c0 osd=4 3 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.551+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1201195c0 osd=4 3 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.551+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 10 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.551+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff1201195c0 osd=4 3 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.551+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 4 10 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.551+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201195c0 osd=4 4 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.551+0000 7ff126e79700 10 client.37013.objecter _send_command 10 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.551+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff120119730 con 0x7ff120121260 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff120119730 con 0x7ff120121260 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff11dffb700 10 client.37013.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff1201195c0 osd=4 4 2024-02-13T00:43:41.727 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:43:41.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 4 10 2024-02-13T00:43:41.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff126e79700 10 client.37013.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:41.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 11 2024-02-13T00:43:41.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff120124050 0x7ff120126440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:41.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff120119730 osd=5 3 2024-02-13T00:43:41.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120119730 osd=5 3 2024-02-13T00:43:41.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff120119730 osd=5 3 2024-02-13T00:43:41.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 11 2024-02-13T00:43:41.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff120119730 osd=5 3 2024-02-13T00:43:41.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 5 11 2024-02-13T00:43:41.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120119730 osd=5 4 2024-02-13T00:43:41.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff126e79700 10 client.37013.objecter _send_command 11 2024-02-13T00:43:41.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.553+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1201291c0 con 0x7ff120124050 2024-02-13T00:43:41.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.554+0000 7ff11ffff700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff120124050 0x7ff120126440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:41.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.554+0000 7ff11dffb700 10 client.37013.objecter ms_handle_connect 0x7ff120124050 2024-02-13T00:43:41.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.554+0000 7ff11ffff700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff120124050 0x7ff120126440 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:41.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.555+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff1201291c0 con 0x7ff120124050 2024-02-13T00:43:41.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.555+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:43:41.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.555+0000 7ff11dffb700 10 client.37013.objecter _finish_command 11 = system:0 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.555+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff120119730 osd=5 4 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.555+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 5 11 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.569+0000 7ff126e79700 10 client.37013.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.569+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 12 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.569+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff120119730 osd=5 3 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.569+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120119730 osd=5 3 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.569+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff120119730 osd=5 3 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.569+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 12 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.569+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff120119730 osd=5 3 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.569+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 5 12 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.569+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120119730 osd=5 4 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.569+0000 7ff126e79700 10 client.37013.objecter _send_command 12 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.569+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff1201291c0 con 0x7ff120124050 2024-02-13T00:43:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff1201291c0 con 0x7ff120124050 2024-02-13T00:43:41.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:41.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff11dffb700 10 client.37013.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff120119730 osd=5 4 2024-02-13T00:43:41.730 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:43:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 5 12 2024-02-13T00:43:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff126e79700 10 client.37013.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 13 2024-02-13T00:43:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff120126ee0 0x7ff120127340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1201291c0 osd=6 3 2024-02-13T00:43:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201291c0 osd=6 3 2024-02-13T00:43:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1201291c0 osd=6 3 2024-02-13T00:43:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 13 2024-02-13T00:43:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff1201291c0 osd=6 3 2024-02-13T00:43:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 6 13 2024-02-13T00:43:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201291c0 osd=6 4 2024-02-13T00:43:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff126e79700 10 client.37013.objecter _send_command 13 2024-02-13T00:43:41.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff120128f70 con 0x7ff120126ee0 2024-02-13T00:43:41.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff125416700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff120126ee0 0x7ff120127340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:41.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff11dffb700 10 client.37013.objecter ms_handle_connect 0x7ff120126ee0 2024-02-13T00:43:41.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.571+0000 7ff125416700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff120126ee0 0x7ff120127340 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:41.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.572+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff120128f70 con 0x7ff120126ee0 2024-02-13T00:43:41.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.572+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:43:41.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.572+0000 7ff11dffb700 10 client.37013.objecter _finish_command 13 = system:0 2024-02-13T00:43:41.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.572+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff1201291c0 osd=6 4 2024-02-13T00:43:41.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.572+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 6 13 2024-02-13T00:43:41.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.586+0000 7ff126e79700 10 client.37013.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:41.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.586+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 14 2024-02-13T00:43:41.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.586+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1201291c0 osd=6 3 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.586+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201291c0 osd=6 3 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.586+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff1201291c0 osd=6 3 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.586+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 14 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.586+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff1201291c0 osd=6 3 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.586+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 6 14 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.586+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201291c0 osd=6 4 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.586+0000 7ff126e79700 10 client.37013.objecter _send_command 14 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.586+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff120128f70 con 0x7ff120126ee0 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff120128f70 con 0x7ff120126ee0 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff11dffb700 10 client.37013.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff1201291c0 osd=6 4 2024-02-13T00:43:41.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 6 14 2024-02-13T00:43:41.734 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:43:41.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff126e79700 10 client.37013.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:41.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 15 2024-02-13T00:43:41.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff120127e90 0x7ff1201282d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:41.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff120128f70 osd=7 3 2024-02-13T00:43:41.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120128f70 osd=7 3 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff120128f70 osd=7 3 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 15 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff120128f70 osd=7 3 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 7 15 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120128f70 osd=7 4 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff126e79700 10 client.37013.objecter _send_command 15 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.588+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff120128d00 con 0x7ff120127e90 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.589+0000 7ff124c15700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff120127e90 0x7ff1201282d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.589+0000 7ff11dffb700 10 client.37013.objecter ms_handle_connect 0x7ff120127e90 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.589+0000 7ff124c15700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff120127e90 0x7ff1201282d0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.590+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff120128d00 con 0x7ff120127e90 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.590+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.590+0000 7ff11dffb700 10 client.37013.objecter _finish_command 15 = system:0 2024-02-13T00:43:41.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.590+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff120128f70 osd=7 4 2024-02-13T00:43:41.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.590+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 7 15 2024-02-13T00:43:41.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.609+0000 7ff126e79700 10 client.37013.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:43:41.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.609+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign -1 16 2024-02-13T00:43:41.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.609+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff120128f70 osd=7 3 2024-02-13T00:43:41.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.609+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120128f70 osd=7 3 2024-02-13T00:43:41.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.609+0000 7ff126e79700 20 client.37013.objecter _get_session s=0x7ff120128f70 osd=7 3 2024-02-13T00:43:41.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.609+0000 7ff126e79700 15 client.37013.objecter _session_command_op_remove -1 16 2024-02-13T00:43:41.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.609+0000 7ff126e79700 20 client.37013.objecter get_session s=0x7ff120128f70 osd=7 3 2024-02-13T00:43:41.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.609+0000 7ff126e79700 15 client.37013.objecter _session_command_op_assign 7 16 2024-02-13T00:43:41.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.609+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120128f70 osd=7 4 2024-02-13T00:43:41.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.609+0000 7ff126e79700 10 client.37013.objecter _send_command 16 2024-02-13T00:43:41.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.609+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff120128d00 con 0x7ff120127e90 2024-02-13T00:43:41.737 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:43:41.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff11dffb700 1 -- 172.21.15.162:0/2377249329 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff120128d00 con 0x7ff120127e90 2024-02-13T00:43:41.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff11dffb700 10 client.37013.objecter ms_dispatch 0x7ff120001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:43:41.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff11dffb700 10 client.37013.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff11dffb700 20 client.37013.objecter put_session s=0x7ff120128f70 osd=7 4 2024-02-13T00:43:41.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff11dffb700 15 client.37013.objecter _session_command_op_remove 7 16 2024-02-13T00:43:41.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 10 librados: watch_flush enter 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 10 librados: watch_flush exit 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 10 client.37013.objecter close_session for osd.0 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff120028590 msgr2=0x7ff1200fe950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff120028590 0x7ff1200fe950 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201071b0 osd=0 1 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 10 client.37013.objecter close_session for osd.1 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6805/150599780 conn(0x7ff120027be0 msgr2=0x7ff120028010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6805/150599780 conn(0x7ff120027be0 0x7ff120028010 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1200fffc0 osd=1 1 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 10 client.37013.objecter close_session for osd.2 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff1200ffb20 msgr2=0x7ff12011c370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff1200ffb20 0x7ff12011c370 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120027a60 osd=2 1 2024-02-13T00:43:41.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 10 client.37013.objecter close_session for osd.3 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6813/604971011 conn(0x7ff120119ec0 msgr2=0x7ff12011d9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6813/604971011 conn(0x7ff120119ec0 0x7ff12011d9e0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1200ff490 osd=3 1 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 10 client.37013.objecter close_session for osd.4 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff120121260 msgr2=0x7ff120123650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff120121260 0x7ff120123650 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff125416700 1 -- 172.21.15.162:0/2377249329 reap_dead start 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201195c0 osd=4 1 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 10 client.37013.objecter close_session for osd.5 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff120124050 msgr2=0x7ff120126440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff120124050 0x7ff120126440 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120119730 osd=5 1 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 10 client.37013.objecter close_session for osd.6 2024-02-13T00:43:41.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff120126ee0 msgr2=0x7ff120127340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff120126ee0 0x7ff120127340 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff1201291c0 osd=6 1 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 10 client.37013.objecter close_session for osd.7 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff120127e90 msgr2=0x7ff1201282d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff120127e90 0x7ff1201282d0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 20 client.37013.objecter put_session s=0x7ff120128f70 osd=7 1 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 20 client.37013.objecter shutdown clearing up homeless session... 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 10 client.37013.objecter successfully canceled tick 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff125416700 1 -- 172.21.15.162:0/2377249329 reap_dead start 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff125416700 1 -- 172.21.15.162:0/2377249329 reap_dead start 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff125416700 1 -- 172.21.15.162:0/2377249329 reap_dead start 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff125416700 1 -- 172.21.15.162:0/2377249329 reap_dead start 2024-02-13T00:43:41.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff110062c90 msgr2=0x7ff110065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff110062c90 0x7ff110065150 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 rx=0x7ff10c004ee0 tx=0x7ff10c00b040).stop 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1200f8480 msgr2=0x7ff12010e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1200f8480 0x7ff12010e170 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7ff11400e970 tx=0x7ff11400b040).stop 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 shutdown_connections 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1200f8480 0x7ff12010e170 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 --2- 172.21.15.162:0/2377249329 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff110062c90 0x7ff110065150 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 >> 172.21.15.162:0/2377249329 conn(0x7ff1200f3800 msgr2=0x7ff1200fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 shutdown_connections 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 -- 172.21.15.162:0/2377249329 wait complete. 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:41.611+0000 7ff126e79700 1 librados: shutdown 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.741 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.742 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.742 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.742 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.742 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:43:41.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:41.802 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:41.900 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:41.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:41.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:42.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:42.034 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:42.055 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:42.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:42.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:43:42.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:42.184 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:41.211167+0000 mon.b (mon.1) 388 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:42.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:41.220063+0000 osd.1 (osd.1) 113 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:43:42.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:41.223021+0000 osd.1 (osd.1) 114 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:43:42.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:41.466962+0000 mon.c (mon.2) 248 : audit [DBG] from='client.? 172.21.15.162:0/2377249329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:42.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:41.596693+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]': finished 2024-02-13T00:43:42.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:41.596795+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e266: 8 total, 8 up, 6 in 2024-02-13T00:43:42.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:41.597246+0000 mon.a (mon.0) 1787 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 25 pgs inactive, 38 pgs peering) 2024-02-13T00:43:42.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:41.619318+0000 mon.c (mon.2) 249 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:42.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:41.847983+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:42.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:41.848565+0000 mon.c (mon.2) 250 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:42.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:41.848855+0000 mon.a (mon.0) 1789 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:42.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:41.849247+0000 mon.c (mon.2) 251 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-83977-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:42.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:41.849489+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-83977-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:42.186 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:41.211167+0000 mon.b (mon.1) 388 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:42.186 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:41.220063+0000 osd.1 (osd.1) 113 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:43:42.186 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:41.223021+0000 osd.1 (osd.1) 114 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:43:42.186 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:41.466962+0000 mon.c (mon.2) 248 : audit [DBG] from='client.? 172.21.15.162:0/2377249329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:42.186 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:41.596693+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]': finished 2024-02-13T00:43:42.186 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:41.596795+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e266: 8 total, 8 up, 6 in 2024-02-13T00:43:42.186 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:41.597246+0000 mon.a (mon.0) 1787 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 25 pgs inactive, 38 pgs peering) 2024-02-13T00:43:42.186 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:41.619318+0000 mon.c (mon.2) 249 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:42.186 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:41.847983+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:42.186 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:41.848565+0000 mon.c (mon.2) 250 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:42.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:41.848855+0000 mon.a (mon.0) 1789 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:42.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:41.849247+0000 mon.c (mon.2) 251 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-83977-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:42.187 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:41.849489+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-83977-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:42.187 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:41.211167+0000 mon.b (mon.1) 388 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:41.220063+0000 osd.1 (osd.1) 113 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:43:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:41.223021+0000 osd.1 (osd.1) 114 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:43:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:41.466962+0000 mon.c (mon.2) 248 : audit [DBG] from='client.? 172.21.15.162:0/2377249329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:41.596693+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-83977-37"}]': finished 2024-02-13T00:43:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:41.596795+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e266: 8 total, 8 up, 6 in 2024-02-13T00:43:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:41.597246+0000 mon.a (mon.0) 1787 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 25 pgs inactive, 38 pgs peering) 2024-02-13T00:43:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:41.619318+0000 mon.c (mon.2) 249 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:41.847983+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:41.848565+0000 mon.c (mon.2) 250 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:41.848855+0000 mon.a (mon.0) 1789 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:42.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:41.849247+0000 mon.c (mon.2) 251 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-83977-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:42.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:41.849489+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-83977-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:42.435 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:42.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:42.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:42.567 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:42.865 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:42.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-13T00:43:43.115 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:43.118 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:42.212792+0000 mon.b (mon.1) 389 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:43.118 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:42.351086+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-83985-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-83985-47"}]': finished 2024-02-13T00:43:43.118 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:42.351219+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-83977-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:43.118 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:42.351295+0000 mon.a (mon.0) 1793 : cluster [DBG] osdmap e267: 8 total, 8 up, 6 in 2024-02-13T00:43:43.118 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:42.357370+0000 mon.b (mon.1) 390 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:43.118 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:42.358159+0000 mon.a (mon.0) 1794 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:43.118 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:42.361302+0000 mon.c (mon.2) 252 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-83977-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:43.118 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:42.361925+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-83977-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:43.118 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:42.368498+0000 mgr.y (mgr.14544) 266 : cluster [DBG] pgmap v371: 278 pgs: 2 active+clean+snaptrim, 40 unknown, 8 active+clean+snaptrim_wait, 228 active+clean; 9.0 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 6.3 KiB/s wr, 9 op/s; 982 B/s, 70 objects/s recovering 2024-02-13T00:43:43.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:42.212792+0000 mon.b (mon.1) 389 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:43.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:42.351086+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-83985-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-83985-47"}]': finished 2024-02-13T00:43:43.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:42.351219+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-83977-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:43.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:42.351295+0000 mon.a (mon.0) 1793 : cluster [DBG] osdmap e267: 8 total, 8 up, 6 in 2024-02-13T00:43:43.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:42.357370+0000 mon.b (mon.1) 390 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:43.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:42.358159+0000 mon.a (mon.0) 1794 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:43.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:42.361302+0000 mon.c (mon.2) 252 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-83977-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:43.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:42.361925+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-83977-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:43.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:42.368498+0000 mgr.y (mgr.14544) 266 : cluster [DBG] pgmap v371: 278 pgs: 2 active+clean+snaptrim, 40 unknown, 8 active+clean+snaptrim_wait, 228 active+clean; 9.0 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 6.3 KiB/s wr, 9 op/s; 982 B/s, 70 objects/s recovering 2024-02-13T00:43:43.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:43.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:43.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:43.253 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:43.274 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:43.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:43.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:43.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:42.212792+0000 mon.b (mon.1) 389 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:43.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:42.351086+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-83985-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-83985-47"}]': finished 2024-02-13T00:43:43.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:42.351219+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-83977-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:43.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:42.351295+0000 mon.a (mon.0) 1793 : cluster [DBG] osdmap e267: 8 total, 8 up, 6 in 2024-02-13T00:43:43.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:42.357370+0000 mon.b (mon.1) 390 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:43.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:42.358159+0000 mon.a (mon.0) 1794 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:43.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:42.361302+0000 mon.c (mon.2) 252 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-83977-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:43.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:42.361925+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-83977-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:43.405 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:42.368498+0000 mgr.y (mgr.14544) 266 : cluster [DBG] pgmap v371: 278 pgs: 2 active+clean+snaptrim, 40 unknown, 8 active+clean+snaptrim_wait, 228 active+clean; 9.0 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 6.3 KiB/s wr, 9 op/s; 982 B/s, 70 objects/s recovering 2024-02-13T00:43:43.406 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:43.429 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:43.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:43.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:43.560 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:43.578 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:43.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:43.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:43.668 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:43.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:43.711 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:44.141 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:44.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:44.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:44.147 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:44.246 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:44.274 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:44.293 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:44.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:44.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:44.381 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:43.048335+0000 osd.5 (osd.5) 119 : cluster [DBG] 28.18 scrub starts 2024-02-13T00:43:44.381 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:43.049225+0000 osd.5 (osd.5) 120 : cluster [DBG] 28.18 scrub ok 2024-02-13T00:43:44.381 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:43.213878+0000 mon.b (mon.1) 391 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:44.381 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:43.358201+0000 mon.a (mon.0) 1796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:44.381 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:43.366998+0000 mon.b (mon.1) 392 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:44.381 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:43.601511+0000 mon.a (mon.0) 1797 : cluster [DBG] osdmap e268: 8 total, 8 up, 6 in 2024-02-13T00:43:44.381 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:43.602146+0000 mon.a (mon.0) 1798 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:44.381 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:43.048335+0000 osd.5 (osd.5) 119 : cluster [DBG] 28.18 scrub starts 2024-02-13T00:43:44.382 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:43.049225+0000 osd.5 (osd.5) 120 : cluster [DBG] 28.18 scrub ok 2024-02-13T00:43:44.382 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:43.213878+0000 mon.b (mon.1) 391 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:44.382 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:43.358201+0000 mon.a (mon.0) 1796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:44.382 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:43.366998+0000 mon.b (mon.1) 392 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:44.382 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:43.601511+0000 mon.a (mon.0) 1797 : cluster [DBG] osdmap e268: 8 total, 8 up, 6 in 2024-02-13T00:43:44.382 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:43.602146+0000 mon.a (mon.0) 1798 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:44.396 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:44.420 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:43.048335+0000 osd.5 (osd.5) 119 : cluster [DBG] 28.18 scrub starts 2024-02-13T00:43:44.420 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:43.049225+0000 osd.5 (osd.5) 120 : cluster [DBG] 28.18 scrub ok 2024-02-13T00:43:44.420 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:43.213878+0000 mon.b (mon.1) 391 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:44.421 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:43.358201+0000 mon.a (mon.0) 1796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:44.421 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:43.366998+0000 mon.b (mon.1) 392 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:44.421 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:43.601511+0000 mon.a (mon.0) 1797 : cluster [DBG] osdmap e268: 8 total, 8 up, 6 in 2024-02-13T00:43:44.421 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:43.602146+0000 mon.a (mon.0) 1798 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:44.422 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:44.440 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:44.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:44.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:43:44.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:44.570 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:44.610 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:44.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:44.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:44.743 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:45.479 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:44.214913+0000 mon.b (mon.1) 393 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:45.479 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:44.237597+0000 osd.1 (osd.1) 115 : cluster [DBG] 4.18 scrub starts 2024-02-13T00:43:45.479 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:44.241349+0000 osd.1 (osd.1) 116 : cluster [DBG] 4.18 scrub ok 2024-02-13T00:43:45.479 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:44.330427+0000 osd.0 (osd.0) 101 : cluster [DBG] 18.16 scrub starts 2024-02-13T00:43:45.479 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:44.333897+0000 osd.0 (osd.0) 102 : cluster [DBG] 18.16 scrub ok 2024-02-13T00:43:45.479 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:44.359449+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-83977-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-83977-38"}]': finished 2024-02-13T00:43:45.479 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:44.359541+0000 mon.a (mon.0) 1800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:45.479 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:44.359632+0000 mon.a (mon.0) 1801 : cluster [DBG] osdmap e269: 8 total, 8 up, 6 in 2024-02-13T00:43:45.479 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:44.363241+0000 mon.b (mon.1) 394 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-31"}]: dispatch 2024-02-13T00:43:45.479 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:44.363922+0000 mon.a (mon.0) 1802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-31"}]: dispatch 2024-02-13T00:43:45.480 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:44.368992+0000 mgr.y (mgr.14544) 267 : cluster [DBG] pgmap v374: 278 pgs: 2 active+clean+snaptrim, 40 unknown, 8 active+clean+snaptrim_wait, 228 active+clean; 9.0 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:45.480 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:44.376118+0000 mon.a (mon.0) 1803 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:45.480 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:44.937454+0000 mon.a (mon.0) 1804 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:45.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:44.214913+0000 mon.b (mon.1) 393 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:45.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:44.237597+0000 osd.1 (osd.1) 115 : cluster [DBG] 4.18 scrub starts 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:44.241349+0000 osd.1 (osd.1) 116 : cluster [DBG] 4.18 scrub ok 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:44.330427+0000 osd.0 (osd.0) 101 : cluster [DBG] 18.16 scrub starts 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:44.333897+0000 osd.0 (osd.0) 102 : cluster [DBG] 18.16 scrub ok 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:44.359449+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-83977-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-83977-38"}]': finished 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:44.359541+0000 mon.a (mon.0) 1800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:44.359632+0000 mon.a (mon.0) 1801 : cluster [DBG] osdmap e269: 8 total, 8 up, 6 in 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:44.363241+0000 mon.b (mon.1) 394 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-31"}]: dispatch 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:44.363922+0000 mon.a (mon.0) 1802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-31"}]: dispatch 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:44.368992+0000 mgr.y (mgr.14544) 267 : cluster [DBG] pgmap v374: 278 pgs: 2 active+clean+snaptrim, 40 unknown, 8 active+clean+snaptrim_wait, 228 active+clean; 9.0 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:44.376118+0000 mon.a (mon.0) 1803 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:44.937454+0000 mon.a (mon.0) 1804 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:44.214913+0000 mon.b (mon.1) 393 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:44.237597+0000 osd.1 (osd.1) 115 : cluster [DBG] 4.18 scrub starts 2024-02-13T00:43:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:44.241349+0000 osd.1 (osd.1) 116 : cluster [DBG] 4.18 scrub ok 2024-02-13T00:43:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:44.330427+0000 osd.0 (osd.0) 101 : cluster [DBG] 18.16 scrub starts 2024-02-13T00:43:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:44.333897+0000 osd.0 (osd.0) 102 : cluster [DBG] 18.16 scrub ok 2024-02-13T00:43:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:44.359449+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-83977-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-83977-38"}]': finished 2024-02-13T00:43:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:44.359541+0000 mon.a (mon.0) 1800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:44.359632+0000 mon.a (mon.0) 1801 : cluster [DBG] osdmap e269: 8 total, 8 up, 6 in 2024-02-13T00:43:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:44.363241+0000 mon.b (mon.1) 394 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-31"}]: dispatch 2024-02-13T00:43:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:44.363922+0000 mon.a (mon.0) 1802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-31"}]: dispatch 2024-02-13T00:43:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:44.368992+0000 mgr.y (mgr.14544) 267 : cluster [DBG] pgmap v374: 278 pgs: 2 active+clean+snaptrim, 40 unknown, 8 active+clean+snaptrim_wait, 228 active+clean; 9.0 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:44.376118+0000 mon.a (mon.0) 1803 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:44.937454+0000 mon.a (mon.0) 1804 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:45.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:43:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:43:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:45.216618+0000 mon.b (mon.1) 395 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:45.348025+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-31"}]': finished 2024-02-13T00:43:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:45.348106+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-83985-47"}]': finished 2024-02-13T00:43:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:45.348155+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e270: 8 total, 8 up, 6 in 2024-02-13T00:43:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:45.349461+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:45.349815+0000 mon.b (mon.1) 396 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-31", "mode": "writeback"}]: dispatch 2024-02-13T00:43:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:45.350308+0000 mon.a (mon.0) 1809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-31", "mode": "writeback"}]: dispatch 2024-02-13T00:43:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:45.476245+0000 mgr.y (mgr.14544) 268 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:46.041230+0000 osd.5 (osd.5) 121 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:43:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:46.042286+0000 osd.5 (osd.5) 122 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:43:46.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:43:46] "GET /metrics HTTP/1.1" 200 186206 "" "Prometheus/2.33.4" 2024-02-13T00:43:46.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:45.216618+0000 mon.b (mon.1) 395 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:46.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:45.348025+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-31"}]': finished 2024-02-13T00:43:46.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:45.348106+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-83985-47"}]': finished 2024-02-13T00:43:46.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:45.348155+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e270: 8 total, 8 up, 6 in 2024-02-13T00:43:46.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:45.349461+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:46.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:45.349815+0000 mon.b (mon.1) 396 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-31", "mode": "writeback"}]: dispatch 2024-02-13T00:43:46.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:45.350308+0000 mon.a (mon.0) 1809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-31", "mode": "writeback"}]: dispatch 2024-02-13T00:43:46.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:45.476245+0000 mgr.y (mgr.14544) 268 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:46.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:46.041230+0000 osd.5 (osd.5) 121 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:43:46.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:46.042286+0000 osd.5 (osd.5) 122 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:43:46.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:45.216618+0000 mon.b (mon.1) 395 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:46.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:45.348025+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-31"}]': finished 2024-02-13T00:43:46.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:45.348106+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-83985-47"}]': finished 2024-02-13T00:43:46.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:45.348155+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e270: 8 total, 8 up, 6 in 2024-02-13T00:43:46.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:45.349461+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-83985-47"}]: dispatch 2024-02-13T00:43:46.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:45.349815+0000 mon.b (mon.1) 396 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-31", "mode": "writeback"}]: dispatch 2024-02-13T00:43:46.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:45.350308+0000 mon.a (mon.0) 1809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-31", "mode": "writeback"}]: dispatch 2024-02-13T00:43:46.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:45.476245+0000 mgr.y (mgr.14544) 268 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:46.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:46.041230+0000 osd.5 (osd.5) 121 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:43:46.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:46.042286+0000 osd.5 (osd.5) 122 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:43:46.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.528+0000 7f504ba7e700 1 -- 172.21.15.162:0/1620702897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50440fd490 msgr2=0x7f50440f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.528+0000 7f504ba7e700 1 --2- 172.21.15.162:0/1620702897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50440fd490 0x7f50440f6cc0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 rx=0x7f50400047b0 tx=0x7f5040009cb0).stop 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.528+0000 7f504ba7e700 1 -- 172.21.15.162:0/1620702897 shutdown_connections 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.528+0000 7f504ba7e700 1 --2- 172.21.15.162:0/1620702897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50440f73a0 0x7f50441039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.528+0000 7f504ba7e700 1 --2- 172.21.15.162:0/1620702897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50440fcb30 0x7f50440fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.528+0000 7f504ba7e700 1 --2- 172.21.15.162:0/1620702897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50440fd490 0x7f50440f6cc0 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.528+0000 7f504ba7e700 1 -- 172.21.15.162:0/1620702897 >> 172.21.15.162:0/1620702897 conn(0x7f504406eb90 msgr2=0x7f50440f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.528+0000 7f504ba7e700 1 -- 172.21.15.162:0/1620702897 shutdown_connections 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.528+0000 7f504ba7e700 1 -- 172.21.15.162:0/1620702897 wait complete. 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504ba7e700 1 librados: starting msgr at 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504ba7e700 1 librados: starting objecter 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504ba7e700 1 Processor -- start 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504ba7e700 1 -- start start 2024-02-13T00:43:46.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504ba7e700 1 librados: setting wanted keys 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504ba7e700 1 librados: calling monclient init 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504ba7e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50440f73a0 0x7f504410d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504ba7e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50440fcb30 0x7f504410d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504ba7e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50440fd490 0x7f5044107220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504ba7e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5044107710 con 0x7f50440f73a0 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504ba7e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5044107880 con 0x7f50440fcb30 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504ba7e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5044107a40 con 0x7f50440fd490 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504981a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50440f73a0 0x7f504410d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504a01b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50440fd490 0x7f5044107220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504981a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50440f73a0 0x7f504410d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35294/0 (socket says 172.21.15.162:35294) 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504981a700 1 -- 172.21.15.162:0/244932388 learned_addr learned my addr 172.21.15.162:0/244932388 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f5049019700 1 --2- 172.21.15.162:0/244932388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50440fcb30 0x7f504410d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:46.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504981a700 1 -- 172.21.15.162:0/244932388 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50440fd490 msgr2=0x7f5044107220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504981a700 1 --2- 172.21.15.162:0/244932388 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50440fd490 0x7f5044107220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504981a700 1 -- 172.21.15.162:0/244932388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50440fcb30 msgr2=0x7f504410d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504981a700 1 --2- 172.21.15.162:0/244932388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50440fcb30 0x7f504410d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.529+0000 7f504981a700 1 -- 172.21.15.162:0/244932388 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5034005040 con 0x7f50440f73a0 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.530+0000 7f5049019700 1 --2- 172.21.15.162:0/244932388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50440fcb30 0x7f504410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.530+0000 7f504a01b700 1 --2- 172.21.15.162:0/244932388 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50440fd490 0x7f5044107220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.530+0000 7f504981a700 1 --2- 172.21.15.162:0/244932388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50440f73a0 0x7f504410d080 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f5034009400 tx=0x7f5034009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.530+0000 7f503af78700 10 client.?.objecter ms_handle_connect 0x7f50440f73a0 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.530+0000 7f503af78700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.530+0000 7f503af78700 1 -- 172.21.15.162:0/244932388 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034007a40 con 0x7f50440f73a0 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.530+0000 7f504ba7e700 1 -- 172.21.15.162:0/244932388 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5040005040 con 0x7f50440f73a0 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.530+0000 7f503af78700 1 -- 172.21.15.162:0/244932388 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f503400f040 con 0x7f50440f73a0 2024-02-13T00:43:46.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.530+0000 7f503af78700 1 -- 172.21.15.162:0/244932388 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034014aa0 con 0x7f50440f73a0 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.530+0000 7f504ba7e700 10 client.26922.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.530+0000 7f504ba7e700 1 -- 172.21.15.162:0/244932388 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5044108080 con 0x7f50440f73a0 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.530+0000 7f504ba7e700 1 librados: init done 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.531+0000 7f504ba7e700 10 client.26922.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.531+0000 7f504ba7e700 15 client.26922.objecter _session_command_op_assign -1 1 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.531+0000 7f504ba7e700 20 client.26922.objecter _get_session osd=-1 returning homeless 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.531+0000 7f504ba7e700 10 client.26922.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.531+0000 7f504ba7e700 1 -- 172.21.15.162:0/244932388 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f50440475e0 con 0x7f50440f73a0 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.531+0000 7f503af78700 1 -- 172.21.15.162:0/244932388 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f503400e630 con 0x7f50440f73a0 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 1 --2- 172.21.15.162:0/244932388 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5030062c90 0x7f5030065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 1 -- 172.21.15.162:0/244932388 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(271..271 src has 52..271) v4 ==== 8813+0+0 (secure 0 0 0) 0x7f503407d620 con 0x7f50440f73a0 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 10 client.26922.objecter ms_dispatch 0x7f5044001550 osd_map(271..271 src has 52..271) v4 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 3 client.26922.objecter handle_osd_map got epochs [271,271] > 0 2024-02-13T00:43:46.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 3 client.26922.objecter handle_osd_map decoding full epoch 271 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f5049019700 1 --2- 172.21.15.162:0/244932388 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5030062c90 0x7f5030065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 10 client.26922.objecter checking command 1 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 1 --2- 172.21.15.162:0/244932388 >> v2:172.21.15.162:6801/2103743338 conn(0x7f5030070e20 0x7f50300732c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 20 client.26922.objecter _get_session s=0x7f5030070770 osd=0 3 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 20 client.26922.objecter put_session s=0x7f5030070770 osd=0 3 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 15 client.26922.objecter _session_command_op_remove -1 1 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 20 client.26922.objecter _get_session s=0x7f5030070770 osd=0 3 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 20 client.26922.objecter get_session s=0x7f5030070770 osd=0 3 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 15 client.26922.objecter _session_command_op_assign 0 1 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 20 client.26922.objecter put_session s=0x7f5030070770 osd=0 4 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 10 client.26922.objecter _send_command 1 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 1 -- 172.21.15.162:0/244932388 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5030073940 con 0x7f5030070e20 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 20 client.26922.objecter dump_active .. 0 homeless 2024-02-13T00:43:46.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 1 -- 172.21.15.162:0/244932388 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=271) v2 ==== 24+0+0 (secure 0 0 0) 0x7f503407d990 con 0x7f50440f73a0 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f504a01b700 1 --2- 172.21.15.162:0/244932388 >> v2:172.21.15.162:6801/2103743338 conn(0x7f5030070e20 0x7f50300732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 10 client.26922.objecter ms_handle_connect 0x7f5030062c90 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f5049019700 1 --2- 172.21.15.162:0/244932388 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5030062c90 0x7f5030065150 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 rx=0x7f5040005010 tx=0x7f5040007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f503af78700 10 client.26922.objecter ms_handle_connect 0x7f5030070e20 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.532+0000 7f504a01b700 1 --2- 172.21.15.162:0/244932388 >> v2:172.21.15.162:6801/2103743338 conn(0x7f5030070e20 0x7f50300732c0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.533+0000 7f503af78700 1 -- 172.21.15.162:0/244932388 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5030073940 con 0x7f5030070e20 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.533+0000 7f503af78700 10 client.26922.objecter ms_dispatch 0x7f5044001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.533+0000 7f503af78700 10 client.26922.objecter _finish_command 1 = system:0 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.533+0000 7f503af78700 20 client.26922.objecter put_session s=0x7f5030070770 osd=0 4 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.533+0000 7f503af78700 15 client.26922.objecter _session_command_op_remove 0 1 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.549+0000 7f504ba7e700 10 client.26922.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.549+0000 7f504ba7e700 15 client.26922.objecter _session_command_op_assign -1 2 2024-02-13T00:43:46.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.549+0000 7f504ba7e700 20 client.26922.objecter _get_session s=0x7f5030070770 osd=0 3 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.549+0000 7f504ba7e700 20 client.26922.objecter put_session s=0x7f5030070770 osd=0 3 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.549+0000 7f504ba7e700 20 client.26922.objecter _get_session s=0x7f5030070770 osd=0 3 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.549+0000 7f504ba7e700 15 client.26922.objecter _session_command_op_remove -1 2 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.549+0000 7f504ba7e700 20 client.26922.objecter get_session s=0x7f5030070770 osd=0 3 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.549+0000 7f504ba7e700 15 client.26922.objecter _session_command_op_assign 0 2 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.549+0000 7f504ba7e700 20 client.26922.objecter put_session s=0x7f5030070770 osd=0 4 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.549+0000 7f504ba7e700 10 client.26922.objecter _send_command 2 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.549+0000 7f504ba7e700 1 -- 172.21.15.162:0/244932388 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7f5044106050 con 0x7f5030070e20 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f503af78700 1 -- 172.21.15.162:0/244932388 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5044106050 con 0x7f5030070e20 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f503af78700 10 client.26922.objecter ms_dispatch 0x7f5044001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f503af78700 10 client.26922.objecter _finish_command 2 = system:0 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f503af78700 20 client.26922.objecter put_session s=0x7f5030070770 osd=0 4 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f503af78700 15 client.26922.objecter _session_command_op_remove 0 2 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 10 librados: watch_flush enter 2024-02-13T00:43:46.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 10 librados: watch_flush exit 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 10 client.26922.objecter close_session for osd.0 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 1 -- 172.21.15.162:0/244932388 >> v2:172.21.15.162:6801/2103743338 conn(0x7f5030070e20 msgr2=0x7f50300732c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 1 --2- 172.21.15.162:0/244932388 >> v2:172.21.15.162:6801/2103743338 conn(0x7f5030070e20 0x7f50300732c0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 20 client.26922.objecter put_session s=0x7f5030070770 osd=0 1 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 20 client.26922.objecter shutdown clearing up homeless session... 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 10 client.26922.objecter successfully canceled tick 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 1 -- 172.21.15.162:0/244932388 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5030062c90 msgr2=0x7f5030065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 1 --2- 172.21.15.162:0/244932388 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5030062c90 0x7f5030065150 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 rx=0x7f5040005010 tx=0x7f5040007f90).stop 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 1 -- 172.21.15.162:0/244932388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50440f73a0 msgr2=0x7f504410d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 1 --2- 172.21.15.162:0/244932388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50440f73a0 0x7f504410d080 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f5034009400 tx=0x7f5034009cb0).stop 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504a01b700 1 -- 172.21.15.162:0/244932388 reap_dead start 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 1 -- 172.21.15.162:0/244932388 shutdown_connections 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 1 -- 172.21.15.162:0/244932388 >> 172.21.15.162:0/244932388 conn(0x7f504406eb90 msgr2=0x7f50440fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:46.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 1 -- 172.21.15.162:0/244932388 shutdown_connections 2024-02-13T00:43:46.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 1 -- 172.21.15.162:0/244932388 wait complete. 2024-02-13T00:43:46.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:46.550+0000 7f504ba7e700 1 librados: shutdown 2024-02-13T00:43:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:46.217702+0000 mon.b (mon.1) 397 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:46.283695+0000 osd.0 (osd.0) 103 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:43:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:46.284736+0000 osd.0 (osd.0) 104 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:43:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:46.346732+0000 mon.a (mon.0) 1810 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:46.352796+0000 mon.a (mon.0) 1811 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-83985-47"}]': finished 2024-02-13T00:43:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:46.352877+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-31", "mode": "writeback"}]': finished 2024-02-13T00:43:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:46.352956+0000 mon.a (mon.0) 1813 : cluster [DBG] osdmap e271: 8 total, 8 up, 6 in 2024-02-13T00:43:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:46.358256+0000 mon.c (mon.2) 253 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:46.358698+0000 mon.a (mon.0) 1814 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:46.369414+0000 mgr.y (mgr.14544) 269 : cluster [DBG] pgmap v377: 270 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 260 active+clean; 8.9 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:46.378858+0000 mon.a (mon.0) 1815 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:47.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:46.379837+0000 mon.a (mon.0) 1816 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:47.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:46.380212+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-83985-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:47.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:46.461285+0000 mon.b (mon.1) 398 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:47.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:46.461820+0000 mon.a (mon.0) 1818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:47.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:46.217702+0000 mon.b (mon.1) 397 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:47.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:46.283695+0000 osd.0 (osd.0) 103 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:43:47.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:46.284736+0000 osd.0 (osd.0) 104 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:46.346732+0000 mon.a (mon.0) 1810 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:46.352796+0000 mon.a (mon.0) 1811 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-83985-47"}]': finished 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:46.352877+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-31", "mode": "writeback"}]': finished 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:46.352956+0000 mon.a (mon.0) 1813 : cluster [DBG] osdmap e271: 8 total, 8 up, 6 in 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:46.358256+0000 mon.c (mon.2) 253 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:46.358698+0000 mon.a (mon.0) 1814 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:46.369414+0000 mgr.y (mgr.14544) 269 : cluster [DBG] pgmap v377: 270 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 260 active+clean; 8.9 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:46.378858+0000 mon.a (mon.0) 1815 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:46.379837+0000 mon.a (mon.0) 1816 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:46.380212+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-83985-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:46.461285+0000 mon.b (mon.1) 398 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:46.461820+0000 mon.a (mon.0) 1818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:47.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:46.217702+0000 mon.b (mon.1) 397 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:46.283695+0000 osd.0 (osd.0) 103 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:46.284736+0000 osd.0 (osd.0) 104 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:46.346732+0000 mon.a (mon.0) 1810 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:46.352796+0000 mon.a (mon.0) 1811 : audit [INF] from='client.? 172.21.15.162:0/4271277698' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-83985-47"}]': finished 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:46.352877+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-31", "mode": "writeback"}]': finished 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:46.352956+0000 mon.a (mon.0) 1813 : cluster [DBG] osdmap e271: 8 total, 8 up, 6 in 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:46.358256+0000 mon.c (mon.2) 253 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:46.358698+0000 mon.a (mon.0) 1814 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:46.369414+0000 mgr.y (mgr.14544) 269 : cluster [DBG] pgmap v377: 270 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 260 active+clean; 8.9 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:46.378858+0000 mon.a (mon.0) 1815 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:46.379837+0000 mon.a (mon.0) 1816 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:46.380212+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-83985-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:46.461285+0000 mon.b (mon.1) 398 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:47.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:46.461820+0000 mon.a (mon.0) 1818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:47.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.1 reset_purged_snaps_last 2024-02-13T00:43:47.693 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:47.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:47.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:47.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:47.825 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:47.897 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:47.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:48.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:48.034 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:48.055 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:48.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:48.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:48.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:48.184 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:47.091084+0000 osd.5 (osd.5) 123 : cluster [DBG] 6.d scrub starts 2024-02-13T00:43:48.184 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:47.092245+0000 osd.5 (osd.5) 124 : cluster [DBG] 6.d scrub ok 2024-02-13T00:43:48.184 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:47.218882+0000 mon.b (mon.1) 399 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:48.185 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:47.363562+0000 mon.a (mon.0) 1819 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]': finished 2024-02-13T00:43:48.185 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:47.363646+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-83985-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:48.185 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:47.363707+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:48.185 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:47.363759+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e272: 8 total, 8 up, 6 in 2024-02-13T00:43:48.185 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:47.367121+0000 mon.b (mon.1) 400 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31"}]: dispatch 2024-02-13T00:43:48.185 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:47.367547+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-83985-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:48.185 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:47.367945+0000 mon.c (mon.2) 254 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:48.185 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:47.368233+0000 mon.a (mon.0) 1824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31"}]: dispatch 2024-02-13T00:43:48.185 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:47.368560+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:48.185 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:47.511230+0000 osd.2 (osd.2) 117 : cluster [DBG] 5.a scrub starts 2024-02-13T00:43:48.185 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:47.512177+0000 osd.2 (osd.2) 118 : cluster [DBG] 5.a scrub ok 2024-02-13T00:43:48.185 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:47.091084+0000 osd.5 (osd.5) 123 : cluster [DBG] 6.d scrub starts 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:47.092245+0000 osd.5 (osd.5) 124 : cluster [DBG] 6.d scrub ok 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:47.218882+0000 mon.b (mon.1) 399 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:47.363562+0000 mon.a (mon.0) 1819 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]': finished 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:47.363646+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-83985-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:47.363707+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:47.363759+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e272: 8 total, 8 up, 6 in 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:47.367121+0000 mon.b (mon.1) 400 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31"}]: dispatch 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:47.367547+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-83985-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:47.367945+0000 mon.c (mon.2) 254 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:47.368233+0000 mon.a (mon.0) 1824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31"}]: dispatch 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:47.368560+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:47.511230+0000 osd.2 (osd.2) 117 : cluster [DBG] 5.a scrub starts 2024-02-13T00:43:48.186 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:47.512177+0000 osd.2 (osd.2) 118 : cluster [DBG] 5.a scrub ok 2024-02-13T00:43:48.187 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:47.091084+0000 osd.5 (osd.5) 123 : cluster [DBG] 6.d scrub starts 2024-02-13T00:43:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:47.092245+0000 osd.5 (osd.5) 124 : cluster [DBG] 6.d scrub ok 2024-02-13T00:43:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:47.218882+0000 mon.b (mon.1) 399 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:47.363562+0000 mon.a (mon.0) 1819 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-83977-38"}]': finished 2024-02-13T00:43:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:47.363646+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-83985-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:47.363707+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:47.363759+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e272: 8 total, 8 up, 6 in 2024-02-13T00:43:48.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:47.367121+0000 mon.b (mon.1) 400 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31"}]: dispatch 2024-02-13T00:43:48.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:47.367547+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-83985-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:48.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:47.367945+0000 mon.c (mon.2) 254 : audit [INF] from='client.? 172.21.15.162:0/1920897534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:48.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:47.368233+0000 mon.a (mon.0) 1824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31"}]: dispatch 2024-02-13T00:43:48.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:47.368560+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]: dispatch 2024-02-13T00:43:48.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:47.511230+0000 osd.2 (osd.2) 117 : cluster [DBG] 5.a scrub starts 2024-02-13T00:43:48.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:47.512177+0000 osd.2 (osd.2) 118 : cluster [DBG] 5.a scrub ok 2024-02-13T00:43:48.535 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:48.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:48.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:48.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:48.667 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:48.816 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:48.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:48.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:48.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:48.950 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:43:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:43:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:43:48.952 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:48.973 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:48.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:49.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.044+0000 7f4c719a5700 1 -- 172.21.15.162:0/3196434258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c6c0fe5d0 msgr2=0x7f4c6c0f6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:49.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.044+0000 7f4c719a5700 1 --2- 172.21.15.162:0/3196434258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c6c0fe5d0 0x7f4c6c0f6150 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 rx=0x7f4c60004870 tx=0x7f4c60009cb0).stop 2024-02-13T00:43:49.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.044+0000 7f4c719a5700 1 -- 172.21.15.162:0/3196434258 shutdown_connections 2024-02-13T00:43:49.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.044+0000 7f4c719a5700 1 --2- 172.21.15.162:0/3196434258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c6c0f6830 0x7f4c6c104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:49.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.044+0000 7f4c719a5700 1 --2- 172.21.15.162:0/3196434258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c6c0fdc70 0x7f4c6c0fe050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:49.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.044+0000 7f4c719a5700 1 --2- 172.21.15.162:0/3196434258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c6c0fe5d0 0x7f4c6c0f6150 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:49.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.044+0000 7f4c719a5700 1 -- 172.21.15.162:0/3196434258 >> 172.21.15.162:0/3196434258 conn(0x7f4c6c070060 msgr2=0x7f4c6c0f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:49.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.044+0000 7f4c719a5700 1 -- 172.21.15.162:0/3196434258 shutdown_connections 2024-02-13T00:43:49.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.044+0000 7f4c719a5700 1 -- 172.21.15.162:0/3196434258 wait complete. 2024-02-13T00:43:49.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c719a5700 1 librados: starting msgr at 2024-02-13T00:43:49.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c719a5700 1 librados: starting objecter 2024-02-13T00:43:49.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c719a5700 1 Processor -- start 2024-02-13T00:43:49.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c719a5700 1 -- start start 2024-02-13T00:43:49.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c719a5700 1 librados: setting wanted keys 2024-02-13T00:43:49.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c719a5700 1 librados: calling monclient init 2024-02-13T00:43:49.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c719a5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c6c0f6830 0x7f4c6c0f7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:49.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c719a5700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c6c0fdc70 0x7f4c6c0f7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:49.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c719a5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c6c0fe5d0 0x7f4c6c0f7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:49.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c719a5700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c116670 con 0x7f4c6c0f6830 2024-02-13T00:43:49.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c719a5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c1167e0 con 0x7f4c6c0fe5d0 2024-02-13T00:43:49.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c719a5700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c1169a0 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c6affd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c6c0f6830 0x7f4c6c0f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:49.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c6a7fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c6c0fdc70 0x7f4c6c0f7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:49.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c6affd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c6c0f6830 0x7f4c6c0f7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35314/0 (socket says 172.21.15.162:35314) 2024-02-13T00:43:49.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c6affd700 1 -- 172.21.15.162:0/1301321952 learned_addr learned my addr 172.21.15.162:0/1301321952 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:49.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c6b7fe700 1 --2- 172.21.15.162:0/1301321952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c6c0fe5d0 0x7f4c6c0f7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:49.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c6a7fc700 1 -- 172.21.15.162:0/1301321952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c6c0fe5d0 msgr2=0x7f4c6c0f7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:49.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c6a7fc700 1 --2- 172.21.15.162:0/1301321952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c6c0fe5d0 0x7f4c6c0f7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:49.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c6a7fc700 1 -- 172.21.15.162:0/1301321952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c6c0f6830 msgr2=0x7f4c6c0f7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:49.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c6a7fc700 1 --2- 172.21.15.162:0/1301321952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c6c0f6830 0x7f4c6c0f7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:49.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.045+0000 7f4c6a7fc700 1 -- 172.21.15.162:0/1301321952 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60005040 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.046+0000 7f4c6affd700 1 --2- 172.21.15.162:0/1301321952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c6c0f6830 0x7f4c6c0f7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:43:49.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.046+0000 7f4c6a7fc700 1 --2- 172.21.15.162:0/1301321952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c6c0fdc70 0x7f4c6c0f7910 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f4c60004870 tx=0x7f4c6000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:49.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.046+0000 7f4c53fff700 10 client.?.objecter ms_handle_connect 0x7f4c6c0fdc70 2024-02-13T00:43:49.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.046+0000 7f4c53fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:49.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.046+0000 7f4c53fff700 1 -- 172.21.15.162:0/1301321952 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6000e070 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.046+0000 7f4c719a5700 1 -- 172.21.15.162:0/1301321952 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6c116c20 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.046+0000 7f4c53fff700 1 -- 172.21.15.162:0/1301321952 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c60013da0 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.046+0000 7f4c53fff700 1 -- 172.21.15.162:0/1301321952 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60019070 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.046+0000 7f4c719a5700 10 client.37043.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:49.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.046+0000 7f4c719a5700 1 -- 172.21.15.162:0/1301321952 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6c117020 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.046+0000 7f4c719a5700 1 librados: init done 2024-02-13T00:43:49.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.047+0000 7f4c719a5700 1 -- 172.21.15.162:0/1301321952 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6c05f960 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.047+0000 7f4c53fff700 1 -- 172.21.15.162:0/1301321952 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4c6001e420 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.048+0000 7f4c53fff700 1 --2- 172.21.15.162:0/1301321952 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4c54062c90 0x7f4c54065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:49.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.048+0000 7f4c53fff700 1 -- 172.21.15.162:0/1301321952 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(273..273 src has 52..273) v4 ==== 8803+0+0 (secure 0 0 0) 0x7f4c6007dd40 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.048+0000 7f4c53fff700 10 client.37043.objecter ms_dispatch 0x7f4c6c001550 osd_map(273..273 src has 52..273) v4 2024-02-13T00:43:49.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.048+0000 7f4c53fff700 3 client.37043.objecter handle_osd_map got epochs [273,273] > 0 2024-02-13T00:43:49.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.048+0000 7f4c53fff700 3 client.37043.objecter handle_osd_map decoding full epoch 273 2024-02-13T00:43:49.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.048+0000 7f4c6affd700 1 --2- 172.21.15.162:0/1301321952 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4c54062c90 0x7f4c54065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:49.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.048+0000 7f4c53fff700 20 client.37043.objecter dump_active .. 0 homeless 2024-02-13T00:43:49.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.048+0000 7f4c53fff700 10 client.37043.objecter ms_handle_connect 0x7f4c54062c90 2024-02-13T00:43:49.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.048+0000 7f4c6affd700 1 --2- 172.21.15.162:0/1301321952 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4c54062c90 0x7f4c54065150 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7f4c5c007d30 tx=0x7f4c5c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:49.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.052+0000 7f4c53fff700 1 -- 172.21.15.162:0/1301321952 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4c60051000 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:49.106 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:49.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:49.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.241+0000 7f4c719a5700 1 -- 172.21.15.162:0/1301321952 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f4c6c0475e0 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:49.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:48.207526+0000 osd.1 (osd.1) 117 : cluster [DBG] 4.4 deep-scrub starts 2024-02-13T00:43:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:48.212424+0000 osd.1 (osd.1) 118 : cluster [DBG] 4.4 deep-scrub ok 2024-02-13T00:43:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:48.220247+0000 mon.b (mon.1) 401 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:48.282674+0000 osd.0 (osd.0) 105 : cluster [DBG] 18.19 scrub starts 2024-02-13T00:43:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:48.283642+0000 osd.0 (osd.0) 106 : cluster [DBG] 18.19 scrub ok 2024-02-13T00:43:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:48.356239+0000 mon.a (mon.0) 1826 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:48.369834+0000 mgr.y (mgr.14544) 270 : cluster [DBG] pgmap v380: 270 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 260 active+clean; 8.9 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:49.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:48.601702+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31"}]': finished 2024-02-13T00:43:49.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:48.601806+0000 mon.a (mon.0) 1828 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]': finished 2024-02-13T00:43:49.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:48.601853+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e273: 8 total, 8 up, 6 in 2024-02-13T00:43:49.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:48.621387+0000 mon.c (mon.2) 255 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:49.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:48.621693+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:49.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:48.622321+0000 mon.c (mon.2) 256 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:49.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:48.622594+0000 mon.a (mon.0) 1831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:49.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:48.623079+0000 mon.c (mon.2) 257 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-83977-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:49.297 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:48.623337+0000 mon.a (mon.0) 1832 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-83977-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:49.318 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:49.339 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:49.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:49.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:49.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.373+0000 7f4c53fff700 1 -- 172.21.15.162:0/1301321952 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v274) v1 ==== 87+0+0 (secure 0 0 0) 0x7f4c60050bf0 con 0x7f4c6c0fdc70 2024-02-13T00:43:49.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.376+0000 7f4c719a5700 10 librados: watch_flush enter 2024-02-13T00:43:49.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.376+0000 7f4c719a5700 10 librados: watch_flush exit 2024-02-13T00:43:49.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.376+0000 7f4c719a5700 20 client.37043.objecter shutdown clearing up homeless session... 2024-02-13T00:43:49.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.376+0000 7f4c719a5700 10 client.37043.objecter successfully canceled tick 2024-02-13T00:43:49.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.376+0000 7f4c719a5700 1 -- 172.21.15.162:0/1301321952 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4c54062c90 msgr2=0x7f4c54065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:49.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.376+0000 7f4c719a5700 1 --2- 172.21.15.162:0/1301321952 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4c54062c90 0x7f4c54065150 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7f4c5c007d30 tx=0x7f4c5c00b040).stop 2024-02-13T00:43:49.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.376+0000 7f4c719a5700 1 -- 172.21.15.162:0/1301321952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c6c0fdc70 msgr2=0x7f4c6c0f7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:49.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.376+0000 7f4c719a5700 1 --2- 172.21.15.162:0/1301321952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c6c0fdc70 0x7f4c6c0f7910 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f4c60004870 tx=0x7f4c6000b040).stop 2024-02-13T00:43:49.382 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.377+0000 7f4c719a5700 1 -- 172.21.15.162:0/1301321952 shutdown_connections 2024-02-13T00:43:49.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.377+0000 7f4c719a5700 1 --2- 172.21.15.162:0/1301321952 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4c54062c90 0x7f4c54065150 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:49.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.377+0000 7f4c719a5700 1 --2- 172.21.15.162:0/1301321952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c6c0fe5d0 0x7f4c6c0f7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:49.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.377+0000 7f4c719a5700 1 --2- 172.21.15.162:0/1301321952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c6c0f6830 0x7f4c6c0f7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:49.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.377+0000 7f4c719a5700 1 --2- 172.21.15.162:0/1301321952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c6c0fdc70 0x7f4c6c0f7910 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:49.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.377+0000 7f4c719a5700 1 -- 172.21.15.162:0/1301321952 >> 172.21.15.162:0/1301321952 conn(0x7f4c6c070060 msgr2=0x7f4c6c0fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:49.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.377+0000 7f4c719a5700 1 -- 172.21.15.162:0/1301321952 shutdown_connections 2024-02-13T00:43:49.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.378+0000 7f4c719a5700 1 -- 172.21.15.162:0/1301321952 wait complete. 2024-02-13T00:43:49.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:49.378+0000 7f4c719a5700 1 librados: shutdown 2024-02-13T00:43:49.384 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:43:49.402 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:48.207526+0000 osd.1 (osd.1) 117 : cluster [DBG] 4.4 deep-scrub starts 2024-02-13T00:43:49.403 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:48.212424+0000 osd.1 (osd.1) 118 : cluster [DBG] 4.4 deep-scrub ok 2024-02-13T00:43:49.403 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:48.220247+0000 mon.b (mon.1) 401 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:49.403 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:48.282674+0000 osd.0 (osd.0) 105 : cluster [DBG] 18.19 scrub starts 2024-02-13T00:43:49.403 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:48.283642+0000 osd.0 (osd.0) 106 : cluster [DBG] 18.19 scrub ok 2024-02-13T00:43:49.403 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:48.356239+0000 mon.a (mon.0) 1826 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:49.403 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:48.369834+0000 mgr.y (mgr.14544) 270 : cluster [DBG] pgmap v380: 270 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 260 active+clean; 8.9 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:49.403 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:48.601702+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31"}]': finished 2024-02-13T00:43:49.404 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:48.601806+0000 mon.a (mon.0) 1828 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]': finished 2024-02-13T00:43:49.404 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:48.601853+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e273: 8 total, 8 up, 6 in 2024-02-13T00:43:49.404 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:48.621387+0000 mon.c (mon.2) 255 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:49.404 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:48.621693+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:49.404 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:48.622321+0000 mon.c (mon.2) 256 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:49.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:48.622594+0000 mon.a (mon.0) 1831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:49.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:48.623079+0000 mon.c (mon.2) 257 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-83977-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:49.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:48.623337+0000 mon.a (mon.0) 1832 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-83977-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:49.405 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:48.207526+0000 osd.1 (osd.1) 117 : cluster [DBG] 4.4 deep-scrub starts 2024-02-13T00:43:49.405 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:48.212424+0000 osd.1 (osd.1) 118 : cluster [DBG] 4.4 deep-scrub ok 2024-02-13T00:43:49.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:48.220247+0000 mon.b (mon.1) 401 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:49.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:48.282674+0000 osd.0 (osd.0) 105 : cluster [DBG] 18.19 scrub starts 2024-02-13T00:43:49.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:48.283642+0000 osd.0 (osd.0) 106 : cluster [DBG] 18.19 scrub ok 2024-02-13T00:43:49.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:48.356239+0000 mon.a (mon.0) 1826 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:43:49.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:48.369834+0000 mgr.y (mgr.14544) 270 : cluster [DBG] pgmap v380: 270 pgs: 2 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 260 active+clean; 8.9 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:43:49.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:48.601702+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-31"}]': finished 2024-02-13T00:43:49.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:48.601806+0000 mon.a (mon.0) 1828 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-83977-38"}]': finished 2024-02-13T00:43:49.407 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:48.601853+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e273: 8 total, 8 up, 6 in 2024-02-13T00:43:49.407 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:48.621387+0000 mon.c (mon.2) 255 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:49.407 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:48.621693+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:49.407 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:48.622321+0000 mon.c (mon.2) 256 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:49.407 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:48.622594+0000 mon.a (mon.0) 1831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:49.407 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:48.623079+0000 mon.c (mon.2) 257 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-83977-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:49.408 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:48.623337+0000 mon.a (mon.0) 1832 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-83977-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:49.444 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:49.472 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:49.492 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:49.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:49.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:49.541 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:49.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:43:49.624 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:49.642 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:43:49.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:49.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:43:49.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:49.774 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:49.900 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:49.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:50.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:50.030 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:50.048 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:50.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:50.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:50.177 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:50.196 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:50.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:50.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:50.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:49.092997+0000 osd.5 (osd.5) 125 : cluster [DBG] 2.14 deep-scrub starts 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:49.095884+0000 osd.5 (osd.5) 126 : cluster [DBG] 2.14 deep-scrub ok 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:49.221812+0000 mon.b (mon.1) 402 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:49.242075+0000 osd.1 (osd.1) 119 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:49.242528+0000 mon.c (mon.2) 258 : audit [INF] from='client.? 172.21.15.162:0/1301321952' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:49.242820+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:49.245233+0000 osd.1 (osd.1) 120 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:49.333844+0000 mon.a (mon.0) 1834 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:49.360039+0000 mon.a (mon.0) 1835 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:49.369677+0000 mon.a (mon.0) 1836 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-83985-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-83985-48"}]': finished 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:49.369745+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-83977-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:49.369861+0000 mon.a (mon.0) 1838 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:49.369909+0000 mon.a (mon.0) 1839 : cluster [DBG] osdmap e274: 8 total, 8 up, 6 in 2024-02-13T00:43:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:49.374561+0000 mon.c (mon.2) 259 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-83977-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:50.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:49.374998+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-83977-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:50.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:49.952901+0000 mon.a (mon.0) 1841 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:50.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:50.326 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:50.344 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:50.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:50.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:50.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:49.092997+0000 osd.5 (osd.5) 125 : cluster [DBG] 2.14 deep-scrub starts 2024-02-13T00:43:50.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:49.095884+0000 osd.5 (osd.5) 126 : cluster [DBG] 2.14 deep-scrub ok 2024-02-13T00:43:50.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:49.221812+0000 mon.b (mon.1) 402 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:50.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:49.242075+0000 osd.1 (osd.1) 119 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:43:50.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:49.242528+0000 mon.c (mon.2) 258 : audit [INF] from='client.? 172.21.15.162:0/1301321952' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:43:50.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:49.242820+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:43:50.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:49.245233+0000 osd.1 (osd.1) 120 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:43:50.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:49.333844+0000 mon.a (mon.0) 1834 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:50.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:49.360039+0000 mon.a (mon.0) 1835 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:43:50.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:49.369677+0000 mon.a (mon.0) 1836 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-83985-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-83985-48"}]': finished 2024-02-13T00:43:50.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:49.369745+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-83977-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:50.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:49.369861+0000 mon.a (mon.0) 1838 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:43:50.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:49.369909+0000 mon.a (mon.0) 1839 : cluster [DBG] osdmap e274: 8 total, 8 up, 6 in 2024-02-13T00:43:50.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:49.374561+0000 mon.c (mon.2) 259 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-83977-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:50.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:49.374998+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-83977-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:50.474 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:49.952901+0000 mon.a (mon.0) 1841 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:50.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:49.092997+0000 osd.5 (osd.5) 125 : cluster [DBG] 2.14 deep-scrub starts 2024-02-13T00:43:50.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:49.095884+0000 osd.5 (osd.5) 126 : cluster [DBG] 2.14 deep-scrub ok 2024-02-13T00:43:50.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:49.221812+0000 mon.b (mon.1) 402 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:50.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:49.242075+0000 osd.1 (osd.1) 119 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:43:50.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:49.242528+0000 mon.c (mon.2) 258 : audit [INF] from='client.? 172.21.15.162:0/1301321952' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:43:50.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:49.242820+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:43:50.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:49.245233+0000 osd.1 (osd.1) 120 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:43:50.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:49.333844+0000 mon.a (mon.0) 1834 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:50.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:49.360039+0000 mon.a (mon.0) 1835 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:43:50.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:49.369677+0000 mon.a (mon.0) 1836 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-83985-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-83985-48"}]': finished 2024-02-13T00:43:50.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:49.369745+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-83977-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:50.475 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:49.369861+0000 mon.a (mon.0) 1838 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:43:50.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:49.369909+0000 mon.a (mon.0) 1839 : cluster [DBG] osdmap e274: 8 total, 8 up, 6 in 2024-02-13T00:43:50.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:49.374561+0000 mon.c (mon.2) 259 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-83977-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:50.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:49.374998+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-83977-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:50.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:49.952901+0000 mon.a (mon.0) 1841 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:50.476 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:50.672 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:50.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:50.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:50.806 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:50.825 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:50.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:50.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:50.955 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:50.973 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:50.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:51.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:51.105 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:51.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:50.222774+0000 mon.b (mon.1) 403 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:51.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:50.370220+0000 mgr.y (mgr.14544) 271 : cluster [DBG] pgmap v383: 278 pgs: 8 creating+peering, 2 active+clean+snaptrim, 32 unknown, 6 active+clean+snaptrim_wait, 230 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:51.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:50.595958+0000 mon.a (mon.0) 1842 : cluster [DBG] osdmap e275: 8 total, 8 up, 6 in 2024-02-13T00:43:51.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:50.597921+0000 mon.b (mon.1) 404 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:51.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:50.602893+0000 mon.a (mon.0) 1843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:51.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:50.684361+0000 osd.7 (osd.7) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:51.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:50.684569+0000 osd.7 (osd.7) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:51.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:50.222774+0000 mon.b (mon.1) 403 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:51.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:50.370220+0000 mgr.y (mgr.14544) 271 : cluster [DBG] pgmap v383: 278 pgs: 8 creating+peering, 2 active+clean+snaptrim, 32 unknown, 6 active+clean+snaptrim_wait, 230 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:51.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:50.595958+0000 mon.a (mon.0) 1842 : cluster [DBG] osdmap e275: 8 total, 8 up, 6 in 2024-02-13T00:43:51.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:50.597921+0000 mon.b (mon.1) 404 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:51.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:50.602893+0000 mon.a (mon.0) 1843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:51.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:50.684361+0000 osd.7 (osd.7) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:51.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:50.684569+0000 osd.7 (osd.7) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:51.411 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:51.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:50.222774+0000 mon.b (mon.1) 403 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:51.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:50.370220+0000 mgr.y (mgr.14544) 271 : cluster [DBG] pgmap v383: 278 pgs: 8 creating+peering, 2 active+clean+snaptrim, 32 unknown, 6 active+clean+snaptrim_wait, 230 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:51.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:50.595958+0000 mon.a (mon.0) 1842 : cluster [DBG] osdmap e275: 8 total, 8 up, 6 in 2024-02-13T00:43:51.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:50.597921+0000 mon.b (mon.1) 404 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:51.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:50.602893+0000 mon.a (mon.0) 1843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:43:51.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:50.684361+0000 osd.7 (osd.7) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:51.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:50.684569+0000 osd.7 (osd.7) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:52.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:51.223749+0000 mon.b (mon.1) 405 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:52.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:51.361483+0000 mon.a (mon.0) 1844 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-83977-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-83977-39"}]': finished 2024-02-13T00:43:52.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:51.361665+0000 mon.a (mon.0) 1845 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:52.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:51.361730+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e276: 8 total, 8 up, 6 in 2024-02-13T00:43:52.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:51.366894+0000 mon.b (mon.1) 406 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:52.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:51.370242+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:52.182 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:51.371068+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:52.182 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:51.223749+0000 mon.b (mon.1) 405 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:52.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:51.361483+0000 mon.a (mon.0) 1844 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-83977-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-83977-39"}]': finished 2024-02-13T00:43:52.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:51.361665+0000 mon.a (mon.0) 1845 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:52.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:51.361730+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e276: 8 total, 8 up, 6 in 2024-02-13T00:43:52.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:51.366894+0000 mon.b (mon.1) 406 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:52.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:51.370242+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:52.183 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:51.371068+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:52.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:43:52.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:51.223749+0000 mon.b (mon.1) 405 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:52.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:51.361483+0000 mon.a (mon.0) 1844 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-83977-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-83977-39"}]': finished 2024-02-13T00:43:52.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:51.361665+0000 mon.a (mon.0) 1845 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:43:52.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:51.361730+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e276: 8 total, 8 up, 6 in 2024-02-13T00:43:52.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:51.366894+0000 mon.b (mon.1) 406 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:52.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:51.370242+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:52.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:51.371068+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:43:52.656 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:52.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:52.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:52.792 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:52.813 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:52.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:52.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:52.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.860+0000 7f4d3f5da700 1 -- 172.21.15.162:0/4029490737 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d380fcb30 msgr2=0x7f4d380fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:52.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.860+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/4029490737 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d380fcb30 0x7f4d380fcf10 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7f4d34004870 tx=0x7f4d34009cb0).stop 2024-02-13T00:43:52.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.860+0000 7f4d3f5da700 1 -- 172.21.15.162:0/4029490737 shutdown_connections 2024-02-13T00:43:52.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.860+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/4029490737 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d380f73a0 0x7f4d381039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.860+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/4029490737 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d380fcb30 0x7f4d380fcf10 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.860+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/4029490737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d380fd490 0x7f4d380f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.860+0000 7f4d3f5da700 1 -- 172.21.15.162:0/4029490737 >> 172.21.15.162:0/4029490737 conn(0x7f4d3806eb90 msgr2=0x7f4d380f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.860+0000 7f4d3f5da700 1 -- 172.21.15.162:0/4029490737 shutdown_connections 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.860+0000 7f4d3f5da700 1 -- 172.21.15.162:0/4029490737 wait complete. 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3f5da700 1 librados: starting msgr at 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3f5da700 1 librados: starting objecter 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3f5da700 1 Processor -- start 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3f5da700 1 -- start start 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3f5da700 1 librados: setting wanted keys 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3f5da700 1 librados: calling monclient init 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3f5da700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d380f73a0 0x7f4d38107030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:52.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3f5da700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d380fcb30 0x7f4d38107520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3f5da700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d380fd490 0x7f4d38107a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3f5da700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d381166b0 con 0x7f4d380f73a0 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3f5da700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d381167f0 con 0x7f4d380fd490 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3f5da700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d38116930 con 0x7f4d380fcb30 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3cb75700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d380fcb30 0x7f4d38107520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3cb75700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d380fcb30 0x7f4d38107520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:41344/0 (socket says 172.21.15.162:41344) 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3cb75700 1 -- 172.21.15.162:0/3415284138 learned_addr learned my addr 172.21.15.162:0/3415284138 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3d376700 1 --2- 172.21.15.162:0/3415284138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d380f73a0 0x7f4d38107030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3db77700 1 --2- 172.21.15.162:0/3415284138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d380fd490 0x7f4d38107a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3cb75700 1 -- 172.21.15.162:0/3415284138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d380fd490 msgr2=0x7f4d38107a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3cb75700 1 --2- 172.21.15.162:0/3415284138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d380fd490 0x7f4d38107a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3cb75700 1 -- 172.21.15.162:0/3415284138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d380f73a0 msgr2=0x7f4d38107030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:52.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3cb75700 1 --2- 172.21.15.162:0/3415284138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d380f73a0 0x7f4d38107030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:52.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3cb75700 1 -- 172.21.15.162:0/3415284138 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d28005040 con 0x7f4d380fcb30 2024-02-13T00:43:52.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d3cb75700 1 --2- 172.21.15.162:0/3415284138 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d380fcb30 0x7f4d38107520 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f4d280048f0 tx=0x7f4d28009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:52.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d2e7fc700 10 client.?.objecter ms_handle_connect 0x7f4d380fcb30 2024-02-13T00:43:52.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.861+0000 7f4d2e7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:52.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.862+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28007b00 con 0x7f4d380fcb30 2024-02-13T00:43:52.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.862+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d2800f040 con 0x7f4d380fcb30 2024-02-13T00:43:52.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.862+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d34005040 con 0x7f4d380fcb30 2024-02-13T00:43:52.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.862+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28014a00 con 0x7f4d380fcb30 2024-02-13T00:43:52.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.862+0000 7f4d3f5da700 10 client.37049.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:52.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.862+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d38116f90 con 0x7f4d380fcb30 2024-02-13T00:43:52.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.862+0000 7f4d3f5da700 1 librados: init done 2024-02-13T00:43:52.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.862+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4d380475e0 con 0x7f4d380fcb30 2024-02-13T00:43:52.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.863+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4d2800e580 con 0x7f4d380fcb30 2024-02-13T00:43:52.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.863+0000 7f4d2e7fc700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4d24062c90 0x7f4d24065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:52.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.863+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(277..277 src has 52..277) v4 ==== 9324+0+0 (secure 0 0 0) 0x7f4d2807d210 con 0x7f4d380fcb30 2024-02-13T00:43:52.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.863+0000 7f4d3d376700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4d24062c90 0x7f4d24065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:52.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.863+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 osd_map(277..277 src has 52..277) v4 2024-02-13T00:43:52.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.863+0000 7f4d2e7fc700 3 client.37049.objecter handle_osd_map got epochs [277,277] > 0 2024-02-13T00:43:52.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.863+0000 7f4d2e7fc700 3 client.37049.objecter handle_osd_map decoding full epoch 277 2024-02-13T00:43:52.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d2e7fc700 20 client.37049.objecter dump_active .. 0 homeless 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v277) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4d28084070 con 0x7f4d380fcb30 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3d376700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4d24062c90 0x7f4d24065150 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 rx=0x7f4d340043d0 tx=0x7f4d3400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d2e7fc700 10 client.37049.objecter ms_handle_connect 0x7f4d24062c90 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 1 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4d380fde00 0x7f4d380fe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38106020 osd=0 3 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38106020 osd=0 3 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38106020 osd=0 3 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 1 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d38106020 osd=0 3 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 0 1 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38106020 osd=0 4 2024-02-13T00:43:52.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3f5da700 10 client.37049.objecter _send_command 1 2024-02-13T00:43:52.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d380614a0 con 0x7f4d380fde00 2024-02-13T00:43:52.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3db77700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4d380fde00 0x7f4d380fe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:52.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d2e7fc700 10 client.37049.objecter ms_handle_connect 0x7f4d380fde00 2024-02-13T00:43:52.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.864+0000 7f4d3db77700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4d380fde00 0x7f4d380fe1e0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:52.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.865+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4d380614a0 con 0x7f4d380fde00 2024-02-13T00:43:52.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.865+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:43:52.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.865+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 1 = system:0 2024-02-13T00:43:52.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.865+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d38106020 osd=0 4 2024-02-13T00:43:52.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.865+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 0 1 2024-02-13T00:43:52.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.879+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:52.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.879+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 2 2024-02-13T00:43:52.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.879+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38106020 osd=0 3 2024-02-13T00:43:52.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.879+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38106020 osd=0 3 2024-02-13T00:43:52.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.879+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38106020 osd=0 3 2024-02-13T00:43:52.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.879+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 2 2024-02-13T00:43:52.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.879+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d38106020 osd=0 3 2024-02-13T00:43:52.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.879+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 0 2 2024-02-13T00:43:52.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.879+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38106020 osd=0 4 2024-02-13T00:43:52.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.879+0000 7f4d3f5da700 10 client.37049.objecter _send_command 2 2024-02-13T00:43:52.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.879+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4d38116b80 con 0x7f4d380fde00 2024-02-13T00:43:52.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4d38116b80 con 0x7f4d380fde00 2024-02-13T00:43:52.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:52.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:52.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d38106020 osd=0 4 2024-02-13T00:43:52.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 0 2 2024-02-13T00:43:52.885 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:43:52.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:52.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 3 2024-02-13T00:43:52.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6805/150599780 conn(0x7f4d38059990 0x7f4d38059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:52.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38028590 osd=1 3 2024-02-13T00:43:52.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38028590 osd=1 3 2024-02-13T00:43:52.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38028590 osd=1 3 2024-02-13T00:43:52.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 3 2024-02-13T00:43:52.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d38028590 osd=1 3 2024-02-13T00:43:52.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 1 3 2024-02-13T00:43:52.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38028590 osd=1 4 2024-02-13T00:43:52.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3f5da700 10 client.37049.objecter _send_command 3 2024-02-13T00:43:52.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d38028700 con 0x7f4d38059990 2024-02-13T00:43:52.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.881+0000 7f4d3d376700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6805/150599780 conn(0x7f4d38059990 0x7f4d38059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:52.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.882+0000 7f4d2e7fc700 10 client.37049.objecter ms_handle_connect 0x7f4d38059990 2024-02-13T00:43:52.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.882+0000 7f4d3d376700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6805/150599780 conn(0x7f4d38059990 0x7f4d38059d70 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:52.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.882+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4d38028700 con 0x7f4d38059990 2024-02-13T00:43:52.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.882+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:43:52.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.882+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 3 = system:0 2024-02-13T00:43:52.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.882+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d38028590 osd=1 4 2024-02-13T00:43:52.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.882+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 1 3 2024-02-13T00:43:52.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.897+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:52.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.897+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 4 2024-02-13T00:43:52.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.897+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38028590 osd=1 3 2024-02-13T00:43:52.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.897+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38028590 osd=1 3 2024-02-13T00:43:52.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.897+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38028590 osd=1 3 2024-02-13T00:43:52.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.897+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 4 2024-02-13T00:43:52.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.897+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d38028590 osd=1 3 2024-02-13T00:43:52.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.897+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 1 4 2024-02-13T00:43:52.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.897+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38028590 osd=1 4 2024-02-13T00:43:52.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.897+0000 7f4d3f5da700 10 client.37049.objecter _send_command 4 2024-02-13T00:43:52.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.897+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4d38059400 con 0x7f4d38059990 2024-02-13T00:43:52.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.898+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4d38059400 con 0x7f4d38059990 2024-02-13T00:43:52.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.898+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:52.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.898+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:52.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.898+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d38028590 osd=1 4 2024-02-13T00:43:52.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.898+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 1 4 2024-02-13T00:43:52.902 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:43:52.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:52.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 5 2024-02-13T00:43:52.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4d380280f0 0x7f4d3811e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:52.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38059810 osd=2 3 2024-02-13T00:43:52.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38059810 osd=2 3 2024-02-13T00:43:52.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38059810 osd=2 3 2024-02-13T00:43:52.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 5 2024-02-13T00:43:52.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d38059810 osd=2 3 2024-02-13T00:43:52.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 2 5 2024-02-13T00:43:52.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38059810 osd=2 4 2024-02-13T00:43:52.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3f5da700 10 client.37049.objecter _send_command 5 2024-02-13T00:43:52.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d381210d0 con 0x7f4d380280f0 2024-02-13T00:43:52.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3cb75700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4d380280f0 0x7f4d3811e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:52.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d2e7fc700 10 client.37049.objecter ms_handle_connect 0x7f4d380280f0 2024-02-13T00:43:52.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.899+0000 7f4d3cb75700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4d380280f0 0x7f4d3811e3f0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:52.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.900+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4d381210d0 con 0x7f4d380280f0 2024-02-13T00:43:52.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.900+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:43:52.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.900+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 5 = system:0 2024-02-13T00:43:52.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.900+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d38059810 osd=2 4 2024-02-13T00:43:52.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.900+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 2 5 2024-02-13T00:43:52.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:52.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.914+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:52.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.914+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 6 2024-02-13T00:43:52.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.914+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38059810 osd=2 3 2024-02-13T00:43:52.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.914+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38059810 osd=2 3 2024-02-13T00:43:52.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.914+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38059810 osd=2 3 2024-02-13T00:43:52.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.914+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 6 2024-02-13T00:43:52.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.914+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d38059810 osd=2 3 2024-02-13T00:43:52.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.914+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 2 6 2024-02-13T00:43:52.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.914+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38059810 osd=2 4 2024-02-13T00:43:52.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.914+0000 7f4d3f5da700 10 client.37049.objecter _send_command 6 2024-02-13T00:43:52.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.914+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4d38027650 con 0x7f4d380280f0 2024-02-13T00:43:52.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.915+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4d38027650 con 0x7f4d380280f0 2024-02-13T00:43:52.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.915+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:52.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.915+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:52.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.915+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d38059810 osd=2 4 2024-02-13T00:43:52.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.915+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 2 6 2024-02-13T00:43:52.919 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:43:52.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.915+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:52.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.915+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 7 2024-02-13T00:43:52.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6813/604971011 conn(0x7f4d3811fa60 0x7f4d3811fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:52.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d3811ba30 osd=3 3 2024-02-13T00:43:52.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3811ba30 osd=3 3 2024-02-13T00:43:52.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d3811ba30 osd=3 3 2024-02-13T00:43:52.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 7 2024-02-13T00:43:52.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d3811ba30 osd=3 3 2024-02-13T00:43:52.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 3 7 2024-02-13T00:43:52.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3811ba30 osd=3 4 2024-02-13T00:43:52.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d3f5da700 10 client.37049.objecter _send_command 7 2024-02-13T00:43:52.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d3811c0b0 con 0x7f4d3811fa60 2024-02-13T00:43:52.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d3db77700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6813/604971011 conn(0x7f4d3811fa60 0x7f4d3811fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:52.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d2e7fc700 10 client.37049.objecter ms_handle_connect 0x7f4d3811fa60 2024-02-13T00:43:52.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.916+0000 7f4d3db77700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6813/604971011 conn(0x7f4d3811fa60 0x7f4d3811fea0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:52.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.917+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4d3811c0b0 con 0x7f4d3811fa60 2024-02-13T00:43:52.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.917+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:43:52.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.917+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 7 = system:0 2024-02-13T00:43:52.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.917+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d3811ba30 osd=3 4 2024-02-13T00:43:52.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.917+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 3 7 2024-02-13T00:43:52.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.931+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:52.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.931+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 8 2024-02-13T00:43:52.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.931+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d3811ba30 osd=3 3 2024-02-13T00:43:52.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.931+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3811ba30 osd=3 3 2024-02-13T00:43:52.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.931+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d3811ba30 osd=3 3 2024-02-13T00:43:52.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.931+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 8 2024-02-13T00:43:52.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.931+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d3811ba30 osd=3 3 2024-02-13T00:43:52.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.931+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 3 8 2024-02-13T00:43:52.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.931+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3811ba30 osd=3 4 2024-02-13T00:43:52.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.931+0000 7f4d3f5da700 10 client.37049.objecter _send_command 8 2024-02-13T00:43:52.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.931+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4d38120b60 con 0x7f4d3811fa60 2024-02-13T00:43:52.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4d38120b60 con 0x7f4d3811fa60 2024-02-13T00:43:52.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:52.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:52.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d3811ba30 osd=3 4 2024-02-13T00:43:52.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 3 8 2024-02-13T00:43:52.936 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:43:52.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:52.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 9 2024-02-13T00:43:52.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4d381232a0 0x7f4d38125690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:52.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38120b60 osd=4 3 2024-02-13T00:43:52.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38120b60 osd=4 3 2024-02-13T00:43:52.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38120b60 osd=4 3 2024-02-13T00:43:52.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 9 2024-02-13T00:43:52.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d38120b60 osd=4 3 2024-02-13T00:43:52.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 4 9 2024-02-13T00:43:52.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38120b60 osd=4 4 2024-02-13T00:43:52.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3f5da700 10 client.37049.objecter _send_command 9 2024-02-13T00:43:52.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d38120d40 con 0x7f4d381232a0 2024-02-13T00:43:52.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.933+0000 7f4d3d376700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4d381232a0 0x7f4d38125690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:52.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.934+0000 7f4d2e7fc700 10 client.37049.objecter ms_handle_connect 0x7f4d381232a0 2024-02-13T00:43:52.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.934+0000 7f4d3d376700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4d381232a0 0x7f4d38125690 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:52.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.934+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4d38120d40 con 0x7f4d381232a0 2024-02-13T00:43:52.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.934+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:43:52.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.934+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 9 = system:0 2024-02-13T00:43:52.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.934+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d38120b60 osd=4 4 2024-02-13T00:43:52.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.934+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 4 9 2024-02-13T00:43:52.944 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:52.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.950+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:52.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.950+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 10 2024-02-13T00:43:52.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.950+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38120b60 osd=4 3 2024-02-13T00:43:52.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.950+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38120b60 osd=4 3 2024-02-13T00:43:52.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.950+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38120b60 osd=4 3 2024-02-13T00:43:52.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.950+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 10 2024-02-13T00:43:52.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.950+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d38120b60 osd=4 3 2024-02-13T00:43:52.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.950+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 4 10 2024-02-13T00:43:52.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.950+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38120b60 osd=4 4 2024-02-13T00:43:52.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.950+0000 7f4d3f5da700 10 client.37049.objecter _send_command 10 2024-02-13T00:43:52.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.950+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4d38120d40 con 0x7f4d381232a0 2024-02-13T00:43:52.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4d38120d40 con 0x7f4d381232a0 2024-02-13T00:43:52.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:52.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:52.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d38120b60 osd=4 4 2024-02-13T00:43:52.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 4 10 2024-02-13T00:43:52.955 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:43:52.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:52.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 11 2024-02-13T00:43:52.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6804/2212296748 conn(0x7f4d38126090 0x7f4d381284c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:52.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38120d40 osd=5 3 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38120d40 osd=5 3 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38120d40 osd=5 3 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 11 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d38120d40 osd=5 3 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 5 11 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38120d40 osd=5 4 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3f5da700 10 client.37049.objecter _send_command 11 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d3812b290 con 0x7f4d38126090 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.952+0000 7f4d3cb75700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6804/2212296748 conn(0x7f4d38126090 0x7f4d381284c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.953+0000 7f4d2e7fc700 10 client.37049.objecter ms_handle_connect 0x7f4d38126090 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.953+0000 7f4d3cb75700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6804/2212296748 conn(0x7f4d38126090 0x7f4d381284c0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.953+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4d3812b290 con 0x7f4d38126090 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.953+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.953+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 11 = system:0 2024-02-13T00:43:52.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.953+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d38120d40 osd=5 4 2024-02-13T00:43:52.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.953+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 5 11 2024-02-13T00:43:52.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.969+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:52.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.969+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 12 2024-02-13T00:43:52.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.969+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38120d40 osd=5 3 2024-02-13T00:43:52.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.969+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38120d40 osd=5 3 2024-02-13T00:43:52.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.969+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d38120d40 osd=5 3 2024-02-13T00:43:52.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.969+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 12 2024-02-13T00:43:52.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.969+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d38120d40 osd=5 3 2024-02-13T00:43:52.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.969+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 5 12 2024-02-13T00:43:52.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.969+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38120d40 osd=5 4 2024-02-13T00:43:52.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.969+0000 7f4d3f5da700 10 client.37049.objecter _send_command 12 2024-02-13T00:43:52.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.969+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4d3812b290 con 0x7f4d38126090 2024-02-13T00:43:52.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.970+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4d3812b290 con 0x7f4d38126090 2024-02-13T00:43:52.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:52.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:52.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d38120d40 osd=5 4 2024-02-13T00:43:52.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 5 12 2024-02-13T00:43:52.974 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:43:52.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:52.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 13 2024-02-13T00:43:52.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4d3812a590 0x7f4d3812aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:52.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d3812b290 osd=6 3 2024-02-13T00:43:52.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3812b290 osd=6 3 2024-02-13T00:43:52.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d3812b290 osd=6 3 2024-02-13T00:43:52.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 13 2024-02-13T00:43:52.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d3812b290 osd=6 3 2024-02-13T00:43:52.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 6 13 2024-02-13T00:43:52.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3812b290 osd=6 4 2024-02-13T00:43:52.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3f5da700 10 client.37049.objecter _send_command 13 2024-02-13T00:43:52.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d3812b080 con 0x7f4d3812a590 2024-02-13T00:43:52.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3db77700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4d3812a590 0x7f4d3812aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:52.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d2e7fc700 10 client.37049.objecter ms_handle_connect 0x7f4d3812a590 2024-02-13T00:43:52.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.971+0000 7f4d3db77700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4d3812a590 0x7f4d3812aa10 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:52.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.972+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4d3812b080 con 0x7f4d3812a590 2024-02-13T00:43:52.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.972+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:43:52.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.972+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 13 = system:0 2024-02-13T00:43:52.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.972+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d3812b290 osd=6 4 2024-02-13T00:43:52.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.972+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 6 13 2024-02-13T00:43:52.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.987+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:52.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.987+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 14 2024-02-13T00:43:52.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.987+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d3812b290 osd=6 3 2024-02-13T00:43:52.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.987+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3812b290 osd=6 3 2024-02-13T00:43:52.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.987+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d3812b290 osd=6 3 2024-02-13T00:43:52.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.987+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 14 2024-02-13T00:43:52.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.987+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d3812b290 osd=6 3 2024-02-13T00:43:52.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.987+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 6 14 2024-02-13T00:43:52.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.987+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3812b290 osd=6 4 2024-02-13T00:43:52.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.987+0000 7f4d3f5da700 10 client.37049.objecter _send_command 14 2024-02-13T00:43:52.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.987+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4d3812b080 con 0x7f4d3812a590 2024-02-13T00:43:52.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4d3812b080 con 0x7f4d3812a590 2024-02-13T00:43:52.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:52.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:52.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d3812b290 osd=6 4 2024-02-13T00:43:52.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 6 14 2024-02-13T00:43:52.992 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:43:52.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:52.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 15 2024-02-13T00:43:52.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4d3812da40 0x7f4d3812de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:52.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d3812b080 osd=7 3 2024-02-13T00:43:52.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3812b080 osd=7 3 2024-02-13T00:43:52.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d3812b080 osd=7 3 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 15 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d3812b080 osd=7 3 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 7 15 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3812b080 osd=7 4 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3f5da700 10 client.37049.objecter _send_command 15 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d3812e4f0 con 0x7f4d3812da40 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.989+0000 7f4d3d376700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4d3812da40 0x7f4d3812de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.990+0000 7f4d2e7fc700 10 client.37049.objecter ms_handle_connect 0x7f4d3812da40 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.990+0000 7f4d3d376700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4d3812da40 0x7f4d3812de80 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.990+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4d3812e4f0 con 0x7f4d3812da40 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.990+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.990+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 15 = system:0 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.990+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d3812b080 osd=7 4 2024-02-13T00:43:52.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:52.990+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 7 15 2024-02-13T00:43:52.997 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:53.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:53.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.007+0000 7f4d3f5da700 10 client.37049.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:43:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.007+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign -1 16 2024-02-13T00:43:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.007+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d3812b080 osd=7 3 2024-02-13T00:43:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.007+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3812b080 osd=7 3 2024-02-13T00:43:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.007+0000 7f4d3f5da700 20 client.37049.objecter _get_session s=0x7f4d3812b080 osd=7 3 2024-02-13T00:43:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.007+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_remove -1 16 2024-02-13T00:43:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.007+0000 7f4d3f5da700 20 client.37049.objecter get_session s=0x7f4d3812b080 osd=7 3 2024-02-13T00:43:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.007+0000 7f4d3f5da700 15 client.37049.objecter _session_command_op_assign 7 16 2024-02-13T00:43:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.007+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3812b080 osd=7 4 2024-02-13T00:43:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.007+0000 7f4d3f5da700 10 client.37049.objecter _send_command 16 2024-02-13T00:43:53.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.007+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4d3812e4f0 con 0x7f4d3812da40 2024-02-13T00:43:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d2e7fc700 1 -- 172.21.15.162:0/3415284138 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4d3812e4f0 con 0x7f4d3812da40 2024-02-13T00:43:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d2e7fc700 10 client.37049.objecter ms_dispatch 0x7f4d38001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:43:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d2e7fc700 10 client.37049.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:43:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d2e7fc700 20 client.37049.objecter put_session s=0x7f4d3812b080 osd=7 4 2024-02-13T00:43:53.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d2e7fc700 15 client.37049.objecter _session_command_op_remove 7 16 2024-02-13T00:43:53.012 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:43:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 10 librados: watch_flush enter 2024-02-13T00:43:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 10 librados: watch_flush exit 2024-02-13T00:43:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 10 client.37049.objecter close_session for osd.0 2024-02-13T00:43:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4d380fde00 msgr2=0x7f4d380fe1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4d380fde00 0x7f4d380fe1e0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38106020 osd=0 1 2024-02-13T00:43:53.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 10 client.37049.objecter close_session for osd.1 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6805/150599780 conn(0x7f4d38059990 msgr2=0x7f4d38059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6805/150599780 conn(0x7f4d38059990 0x7f4d38059d70 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38028590 osd=1 1 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 10 client.37049.objecter close_session for osd.2 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4d380280f0 msgr2=0x7f4d3811e3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4d380280f0 0x7f4d3811e3f0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38059810 osd=2 1 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 10 client.37049.objecter close_session for osd.3 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6813/604971011 conn(0x7f4d3811fa60 msgr2=0x7f4d3811fea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6813/604971011 conn(0x7f4d3811fa60 0x7f4d3811fea0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3db77700 1 -- 172.21.15.162:0/3415284138 reap_dead start 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3811ba30 osd=3 1 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 10 client.37049.objecter close_session for osd.4 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4d381232a0 msgr2=0x7f4d38125690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:53.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4d381232a0 0x7f4d38125690 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3db77700 1 -- 172.21.15.162:0/3415284138 reap_dead start 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38120b60 osd=4 1 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 10 client.37049.objecter close_session for osd.5 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6804/2212296748 conn(0x7f4d38126090 msgr2=0x7f4d381284c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6804/2212296748 conn(0x7f4d38126090 0x7f4d381284c0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d38120d40 osd=5 1 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 10 client.37049.objecter close_session for osd.6 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4d3812a590 msgr2=0x7f4d3812aa10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4d3812a590 0x7f4d3812aa10 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3812b290 osd=6 1 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 10 client.37049.objecter close_session for osd.7 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4d3812da40 msgr2=0x7f4d3812de80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:53.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4d3812da40 0x7f4d3812de80 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 20 client.37049.objecter put_session s=0x7f4d3812b080 osd=7 1 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 20 client.37049.objecter shutdown clearing up homeless session... 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 10 client.37049.objecter successfully canceled tick 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.009+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4d24062c90 msgr2=0x7f4d24065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.010+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4d24062c90 0x7f4d24065150 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 rx=0x7f4d340043d0 tx=0x7f4d3400b040).stop 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.010+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d380fcb30 msgr2=0x7f4d38107520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.010+0000 7f4d3f5da700 1 --2- 172.21.15.162:0/3415284138 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d380fcb30 0x7f4d38107520 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f4d280048f0 tx=0x7f4d28009cb0).stop 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.010+0000 7f4d3db77700 1 -- 172.21.15.162:0/3415284138 reap_dead start 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.010+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 shutdown_connections 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.010+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 >> 172.21.15.162:0/3415284138 conn(0x7f4d3806eb90 msgr2=0x7f4d380fa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.010+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 shutdown_connections 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.010+0000 7f4d3f5da700 1 -- 172.21.15.162:0/3415284138 wait complete. 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:53.010+0000 7f4d3f5da700 1 librados: shutdown 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:43:53.017 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:43:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:43:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:43:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:43:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:43:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:43:53.018 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:43:53.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:53.129 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:53.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:52.225131+0000 mon.b (mon.1) 407 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:53.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:52.361863+0000 mon.a (mon.0) 1849 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-83985-48"}]': finished 2024-02-13T00:43:53.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:52.361936+0000 mon.a (mon.0) 1850 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:53.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:52.362053+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e277: 8 total, 8 up, 6 in 2024-02-13T00:43:53.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:52.366546+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:53.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:52.369433+0000 mon.b (mon.1) 408 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:43:53.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:52.370766+0000 mgr.y (mgr.14544) 272 : cluster [DBG] pgmap v386: 278 pgs: 2 active+clean+snaptrim, 40 unknown, 6 active+clean+snaptrim_wait, 230 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:53.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:52.370900+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:43:53.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:52.864419+0000 mon.c (mon.2) 260 : audit [DBG] from='client.? 172.21.15.162:0/3415284138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:53.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:52.225131+0000 mon.b (mon.1) 407 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:53.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:52.361863+0000 mon.a (mon.0) 1849 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-83985-48"}]': finished 2024-02-13T00:43:53.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:52.361936+0000 mon.a (mon.0) 1850 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:53.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:52.362053+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e277: 8 total, 8 up, 6 in 2024-02-13T00:43:53.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:52.366546+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:53.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:52.369433+0000 mon.b (mon.1) 408 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:43:53.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:52.370766+0000 mgr.y (mgr.14544) 272 : cluster [DBG] pgmap v386: 278 pgs: 2 active+clean+snaptrim, 40 unknown, 6 active+clean+snaptrim_wait, 230 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:53.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:52.370900+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:43:53.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:52.864419+0000 mon.c (mon.2) 260 : audit [DBG] from='client.? 172.21.15.162:0/3415284138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:53.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:52.225131+0000 mon.b (mon.1) 407 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:53.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:52.361863+0000 mon.a (mon.0) 1849 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-83985-48"}]': finished 2024-02-13T00:43:53.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:52.361936+0000 mon.a (mon.0) 1850 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:43:53.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:52.362053+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e277: 8 total, 8 up, 6 in 2024-02-13T00:43:53.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:52.366546+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-83985-48"}]: dispatch 2024-02-13T00:43:53.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:52.369433+0000 mon.b (mon.1) 408 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:43:53.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:52.370766+0000 mgr.y (mgr.14544) 272 : cluster [DBG] pgmap v386: 278 pgs: 2 active+clean+snaptrim, 40 unknown, 6 active+clean+snaptrim_wait, 230 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:53.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:52.370900+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:43:53.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:52.864419+0000 mon.c (mon.2) 260 : audit [DBG] from='client.? 172.21.15.162:0/3415284138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:43:53.576 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:43:53.576 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:43:53.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:43:53.576 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:43:53.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.576 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14913.0:2 22.16 22:69efe4bc:test-rados-api-smithi162-84001-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:43:53.576 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.531523+0000", 2024-02-13T00:43:53.576 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 172.039300916, 2024-02-13T00:43:53.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.084235272999999999, 2024-02-13T00:43:53.576 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.576 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14913", 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3501757973", 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531523+0000", 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531523+0000", 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531521+0000", 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1880000000000008e-06 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531530+0000", 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0994e-05 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531541+0000", 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4500000000000004e-06 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531550+0000", 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0084330910000000002 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.539983+0000", 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1860000000000003e-06 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.539990+0000", 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043643534999999997 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.583634+0000", 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011933 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.583753+0000", 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000156346 2024-02-13T00:43:53.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,7", 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.583909+0000", 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026647634 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.610557+0000", 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043537769999999996 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.614911+0000", 2024-02-13T00:43:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00078932899999999996 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615700+0000", 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3908000000000001e-05 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615734+0000", 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4465e-05 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615758+0000", 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.084235272999999999 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24745.0:2 24.1b 24:de1a0751:test-rados-api-smithi162-84081-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.540986+0000", 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 172.029838544, 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12964152300000001, 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:43:53.583 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24745", 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/530375479", 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540986+0000", 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540986+0000", 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540984+0000", 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.49e-05 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540999+0000", 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0326000000000001e-05 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.541009+0000", 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.926e-06 2024-02-13T00:43:53.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.541017+0000", 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004928562 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.545945+0000", 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3000000000000001e-06 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.545951+0000", 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043217637000000003 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589168+0000", 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011348799999999999 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589282+0000", 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0266999999999995e-05 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589372+0000", 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020246106999999999 2024-02-13T00:43:53.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.609618+0000", 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005197513 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.614816+0000", 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055733195999999999 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670549+0000", 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8872000000000002e-05 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670598+0000", 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9230999999999998e-05 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670627+0000", 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12964152300000001 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.589 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24773.0:2 11.2 11:42a49006:test-rados-api-smithi162-84105-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.602097+0000", 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 171.96872685400001, 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101585882, 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24773", 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4011549658", 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602097+0000", 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602097+0000", 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602095+0000", 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7520000000000002e-05 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602113+0000", 2024-02-13T00:43:53.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1664999999999999e-05 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602124+0000", 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0750000000000004e-06 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602133+0000", 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044129360000000001 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.606546+0000", 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126408 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.606673+0000", 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002542182 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,3", 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.609215+0000", 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059528159999999997 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:43:53.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615168+0000", 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055524639000000001 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670692+0000", 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032890434000000003 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703583+0000", 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5461000000000003e-05 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703638+0000", 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4892000000000003e-05 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703683+0000", 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101585882 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24911.0:20 16.1 16:83282cb0:::1.obj2:head [write 0~90] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.474569+0000", 2024-02-13T00:43:53.595 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 171.09625551600001, 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.158591963, 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24911", 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1860840531", 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 20 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474569+0000", 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474569+0000", 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474567+0000", 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9480000000000003e-06 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474577+0000", 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1072e-05 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474588+0000", 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8759999999999999e-06 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474595+0000", 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028252126999999998 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.502847+0000", 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013890300000000001 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.502986+0000", 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015514899999999999 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.503141+0000", 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016316878999999999 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.519458+0000", 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058076720999999998 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.577535+0000", 2024-02-13T00:43:53.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055542411999999999 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.633077+0000", 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3886e-05 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.633121+0000", 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9544e-05 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.633161+0000", 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.158591963 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14886.0:441 20.14 e107/105 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head v 107'198, mlcod=107'196)", 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 171.07588815599999, 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12436195799999999, 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:43:53.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989999999999997e-06 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494942+0000", 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037274475000000001 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.532217+0000", 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2009999999999999e-06 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.532225+0000", 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055477902000000003 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.587703+0000", 2024-02-13T00:43:53.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6648999999999998e-05 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.587729+0000", 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031536963000000001 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.619266+0000", 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1473999999999997e-05 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.619298+0000", 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12436195799999999 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2064 20.19 20:9c8ffe76:test-rados-api-smithi162-84049-16::big.copy2:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.437684+0000", 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 169.13314058099999, 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088117519000000005, 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:43:53.605 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2064 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437684+0000", 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437684+0000", 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437682+0000", 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1730000000000001e-06 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437691+0000", 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4664999999999999e-05 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437706+0000", 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6229999999999997e-06 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437713+0000", 2024-02-13T00:43:53.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6378999999999994e-05 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437800+0000", 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2805000000000003e-05 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437893+0000", 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.060281989000000001 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.498175+0000", 2024-02-13T00:43:53.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034643450000000002 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.501639+0000", 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023806853999999999 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525446+0000", 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028838300000000002 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525734+0000", 2024-02-13T00:43:53.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3445000000000002e-05 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525778+0000", 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3501000000000001e-05 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525801+0000", 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088117519000000005 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2069 20.1b 20:de4afda3:test-rados-api-smithi162-84049-17::big:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:43:53.610 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.630167+0000", 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 168.940657028, 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11190932200000001, 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2069 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.630167+0000", 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.630167+0000", 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.630164+0000", 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011350433 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641515+0000", 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3941e-05 2024-02-13T00:43:53.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641539+0000", 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6942000000000001e-05 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641566+0000", 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0275e-05 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641646+0000", 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100653 2024-02-13T00:43:53.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641747+0000", 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137825 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,7", 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641884+0000", 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036914244999999998 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.678799+0000", 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.062888029999999998 2024-02-13T00:43:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.741687+0000", 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032975399999999997 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742016+0000", 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4996000000000002e-05 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742051+0000", 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4997000000000001e-05 2024-02-13T00:43:53.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742076+0000", 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11190932200000001 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2074 20.1b 20:de8a7420:test-rados-api-smithi162-84049-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.862849+0000", 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 168.707975199, 2024-02-13T00:43:53.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101931788, 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2074 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.862849+0000", 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.862849+0000", 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.862846+0000", 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070632639999999997 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869909+0000", 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1322e-05 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869931+0000", 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0179e-05 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869941+0000", 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1819e-05 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869993+0000", 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010545 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.870098+0000", 2024-02-13T00:43:53.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000146911 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,7", 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.870245+0000", 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029969216 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.900214+0000", 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064237111 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964452+0000", 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025591999999999998 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964707+0000", 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8297999999999998e-05 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964746+0000", 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4975999999999999e-05 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964781+0000", 2024-02-13T00:43:53.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101931788 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27739.0:2 68.6 68:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:21.788780+0000", 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 151.782044533, 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061004435000000003, 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27739", 2024-02-13T00:43:53.622 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3559302368", 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788780+0000", 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788780+0000", 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788777+0000", 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2724999999999999e-05 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788789+0000", 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2472e-05 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788802+0000", 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3860000000000007e-06 2024-02-13T00:43:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788810+0000", 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00058719999999999996 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.789398+0000", 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3919999999999997e-06 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.789405+0000", 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047759248999999997 2024-02-13T00:43:53.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.837164+0000", 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118143 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.837282+0000", 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015534899999999999 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,7", 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.837438+0000", 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005545024 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.842983+0000", 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059063199999999996 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.848889+0000", 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00079681100000000003 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.849686+0000", 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3878999999999997e-05 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.849740+0000", 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4291e-05 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.849784+0000", 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061004435000000003 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.5.0:281 66.6 e128/124 66:64b50987:test-rados-api-smithi162-84081-8::foo:3 v 128'5, mlcod=126'1)", 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:22.771607+0000", 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 150.79921713300001, 2024-02-13T00:43:53.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033789000999999999, 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771607+0000", 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771607+0000", 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771606+0000", 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4549999999999998e-06 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771614+0000", 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7230000000000004e-06 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771622+0000", 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2320000000000004e-06 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771629+0000", 2024-02-13T00:43:53.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012838736 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784468+0000", 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1084e-05 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784489+0000", 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020873023000000001 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.805362+0000", 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4317999999999999e-05 2024-02-13T00:43:53.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.805396+0000", 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033789000999999999 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.5.0:282 60.a e128/120 60:51f054f9:test-rados-api-smithi162-84073-8::foo:7 v 128'6, mlcod=127'3)", 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:22.784439+0000", 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 150.786385306, 2024-02-13T00:43:53.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031021368000000001, 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784439+0000", 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784439+0000", 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784438+0000", 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1600999999999999e-05 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784460+0000", 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3000000000000007e-06 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784469+0000", 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2879999999999998e-06 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784475+0000", 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027856051 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.812331+0000", 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5820999999999999e-05 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.812357+0000", 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030688690000000001 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.815426+0000", 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4136e-05 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.815460+0000", 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031021368000000001 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16275.0:9 66.6 e129/124 66:64b50987:test-rados-api-smithi162-84081-8::foo:head v 129'6, mlcod=127'3)", 2024-02-13T00:43:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:23.831556+0000", 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.73926818800001, 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054147344, 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831556+0000", 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831556+0000", 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831554+0000", 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.209e-05 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831576+0000", 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4198e-05 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831590+0000", 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1470000000000004e-06 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831597+0000", 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051779300000000001 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832115+0000", 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0034000000000002e-05 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832135+0000", 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053545497999999997 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.885681+0000", 2024-02-13T00:43:53.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2527999999999998e-05 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.885703+0000", 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054147344 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25082.0:19 60.a e129/120 60:51f054f9:test-rados-api-smithi162-84073-8::foo:head v 129'7, mlcod=127'4)", 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.73921980899999, 2024-02-13T00:43:53.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048672976, 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0720000000000001e-06 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831608+0000", 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9790000000000001e-06 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831613+0000", 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.641e-06 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831617+0000", 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013029420000000001 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832920+0000", 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8600000000000001e-05 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832938+0000", 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047309385000000002 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.880247+0000", 2024-02-13T00:43:53.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9788e-05 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.880277+0000", 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048672976 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16275.0:11 66.6 e130/124 66:629bc174:test-rados-api-smithi162-84081-9::foo:head v 130'7, mlcod=128'5)", 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:24.923953+0000", 2024-02-13T00:43:53.644 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 148.64687124599999, 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.077278500999999999, 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923953+0000", 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923953+0000", 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923951+0000", 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1398e-05 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923963+0000", 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5424e-05 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923978+0000", 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4609999999999997e-06 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923988+0000", 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075993709000000007 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.999981+0000", 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6506000000000001e-05 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.999998+0000", 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012042310000000001 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:25.001202+0000", 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9149000000000002e-05 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:25.001231+0000", 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.077278500999999999 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.648 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25162.0:109 61.1f 61:fec11214:test-rados-api-smithi162-84057-10::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:26.012720+0000", 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 147.55810401700001, 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075674577000000007, 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25162", 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 109 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.012720+0000", 2024-02-13T00:43:53.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.012720+0000", 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.012718+0000", 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6659999999999997e-06 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.012727+0000", 2024-02-13T00:43:53.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042113080000000004 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.016938+0000", 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0062e-05 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.016948+0000", 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029517039999999999 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.019900+0000", 2024-02-13T00:43:53.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010618700000000001 2024-02-13T00:43:53.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.020006+0000", 2024-02-13T00:43:53.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130426 2024-02-13T00:43:53.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-02-13T00:43:53.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.020136+0000", 2024-02-13T00:43:53.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061208299999999998 2024-02-13T00:43:53.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.026257+0000", 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030848077000000002 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.057105+0000", 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031192258 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.088298+0000", 2024-02-13T00:43:53.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5677000000000002e-05 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.088353+0000", 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1408000000000001e-05 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.088395+0000", 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075674577000000007 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.5.0:293 66.6 e132/124 66:629bc174:test-rados-api-smithi162-84081-9::foo:7 v 132'15, mlcod=131'12)", 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:27.094916+0000", 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 146.475908605, 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082889299, 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094916+0000", 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094916+0000", 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094915+0000", 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2129999999999999e-06 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094920+0000", 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1179999999999997e-06 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094927+0000", 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7989999999999996e-06 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094933+0000", 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6577999999999996e-05 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.095030+0000", 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1804e-05 2024-02-13T00:43:53.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.095052+0000", 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082724803 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.177777+0000", 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8288000000000001e-05 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.177805+0000", 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082889299 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.19047.0:7 78.0s1 78:1b3d1cfa:test-rados-api-smithi162-84073-12::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e138)", 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:33.486471+0000", 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 140.08435354700001, 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028919100999999999, 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.19047", 2024-02-13T00:43:53.659 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3520430640", 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:33.486471+0000", 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:33.486471+0000", 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:43:53.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:33.486469+0000", 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4660000000000002e-06 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:33.486476+0000", 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.19e-05 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:33.486488+0000", 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.413e-06 2024-02-13T00:43:53.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:33.486495+0000", 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040191159999999997 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:33.490515+0000", 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000142103 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:33.490657+0000", 2024-02-13T00:43:53.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042695599999999998 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:33.491084+0000", 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024012156999999999 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:33.515096+0000", 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0659000000000003e-05 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:33.515156+0000", 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000233274 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:33.515390+0000", 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028919100999999999 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.19047.0:9 78.0s1 78:1b3d1cfa:test-rados-api-smithi162-84073-12::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected e139)", 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:34.355949+0000", 2024-02-13T00:43:53.664 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 139.21487470100001, 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056363352999999998, 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.19047", 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3520430640", 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 9 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355949+0000", 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355949+0000", 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999919 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355941+0000", 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4331000000000001e-05 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355956+0000", 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0973000000000001e-05 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355977+0000", 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2830000000000001e-06 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355985+0000", 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052008540000000004 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.361186+0000", 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7719000000000005e-05 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.361274+0000", 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059102099999999995 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.361865+0000", 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050108890000000003 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.411974+0000", 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7391000000000006e-05 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.412041+0000", 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027185499999999999 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.412313+0000", 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056363352999999998 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDECSubOpWriteReply(78.0s1 139/133 ECSubWriteReply(tid=284, last_complete=139'3, committed=1, applied=1))", 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:34.413870+0000", 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 139.15695414800001, 2024-02-13T00:43:53.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033827619000000003, 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413870+0000", 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413870+0000", 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999967 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413867+0000", 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0979999999999996e-06 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413874+0000", 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3100000000000001e-06 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413882+0000", 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.49e-06 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413887+0000", 2024-02-13T00:43:53.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033771486000000003 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.447659+0000", 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8617e-05 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.447698+0000", 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033827619000000003 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.19047.0:10 78.0s1 78:1b3d1cfa:test-rados-api-smithi162-84073-12::foo:head [rollback 5] snapc 0=[] ondisk+write+known_if_redirected e139)", 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:34.415564+0000", 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 139.155260513, 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094033364999999994, 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.19047", 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3520430640", 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 10 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415564+0000", 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415564+0000", 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415561+0000", 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6379999999999999e-06 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:43:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415571+0000", 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4296e-05 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415585+0000", 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6030000000000005e-06 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415594+0000", 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032129234 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.447723+0000", 2024-02-13T00:43:53.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010490200000000001 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.447828+0000", 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044360579999999997 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.452264+0000", 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00171576 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.453980+0000", 2024-02-13T00:43:53.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055540957000000002 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.509521+0000", 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6073000000000002e-05 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.509597+0000", 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094033364999999994 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:43:53.677 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:43:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:53.679 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:53.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:53.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:53.807 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:53.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:53.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:53.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:53.940 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:53.968 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:54.447 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.226302+0000 mon.b (mon.1) 409 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:54.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.370339+0000 mon.a (mon.0) 1854 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-83985-48"}]': finished 2024-02-13T00:43:54.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.370422+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]': finished 2024-02-13T00:43:54.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:53.370516+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e278: 8 total, 8 up, 6 in 2024-02-13T00:43:54.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.371696+0000 mon.b (mon.1) 410 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-33", "mode": "writeback"}]: dispatch 2024-02-13T00:43:54.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.372709+0000 mon.c (mon.2) 261 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:54.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.372846+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-33", "mode": "writeback"}]: dispatch 2024-02-13T00:43:54.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.373124+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:54.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.388860+0000 mon.c (mon.2) 262 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:54.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.389196+0000 mon.a (mon.0) 1859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:54.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.389744+0000 mon.c (mon.2) 263 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:54.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.389969+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:54.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.390329+0000 mon.c (mon.2) 264 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-83985-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:54.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:53.390543+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-83985-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:54.449 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.226302+0000 mon.b (mon.1) 409 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:54.449 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.370339+0000 mon.a (mon.0) 1854 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-83985-48"}]': finished 2024-02-13T00:43:54.449 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.370422+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]': finished 2024-02-13T00:43:54.449 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:53.370516+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e278: 8 total, 8 up, 6 in 2024-02-13T00:43:54.449 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.371696+0000 mon.b (mon.1) 410 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-33", "mode": "writeback"}]: dispatch 2024-02-13T00:43:54.449 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.372709+0000 mon.c (mon.2) 261 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:54.449 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.372846+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-33", "mode": "writeback"}]: dispatch 2024-02-13T00:43:54.449 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.373124+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:54.449 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.388860+0000 mon.c (mon.2) 262 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:54.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.389196+0000 mon.a (mon.0) 1859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:54.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.389744+0000 mon.c (mon.2) 263 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:54.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.389969+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:54.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.390329+0000 mon.c (mon.2) 264 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-83985-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:54.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:53.390543+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-83985-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:54.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.226302+0000 mon.b (mon.1) 409 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:54.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.370339+0000 mon.a (mon.0) 1854 : audit [INF] from='client.? 172.21.15.162:0/1398491297' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-83985-48"}]': finished 2024-02-13T00:43:54.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.370422+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]': finished 2024-02-13T00:43:54.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:53.370516+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e278: 8 total, 8 up, 6 in 2024-02-13T00:43:54.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.371696+0000 mon.b (mon.1) 410 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-33", "mode": "writeback"}]: dispatch 2024-02-13T00:43:54.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.372709+0000 mon.c (mon.2) 261 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:54.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.372846+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-33", "mode": "writeback"}]: dispatch 2024-02-13T00:43:54.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.373124+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:54.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.388860+0000 mon.c (mon.2) 262 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:54.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.389196+0000 mon.a (mon.0) 1859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:54.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.389744+0000 mon.c (mon.2) 263 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:54.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.389969+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:54.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.390329+0000 mon.c (mon.2) 264 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-83985-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:54.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:53.390543+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-83985-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:54.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:54.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:54.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:55.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.322+0000 7f5469bbe700 1 -- 172.21.15.162:0/1857226360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54640fa440 msgr2=0x7f54640f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:55.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.322+0000 7f5469bbe700 1 --2- 172.21.15.162:0/1857226360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54640fa440 0x7f54640f7e20 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 rx=0x7f544c004870 tx=0x7f544c009cb0).stop 2024-02-13T00:43:55.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.322+0000 7f5469bbe700 1 -- 172.21.15.162:0/1857226360 shutdown_connections 2024-02-13T00:43:55.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.322+0000 7f5469bbe700 1 --2- 172.21.15.162:0/1857226360 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54640f8500 0x7f5464104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:55.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.322+0000 7f5469bbe700 1 --2- 172.21.15.162:0/1857226360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54640f9b70 0x7f54640f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:43:55.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.322+0000 7f5469bbe700 1 --2- 172.21.15.162:0/1857226360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54640fa440 0x7f54640f7e20 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:55.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.322+0000 7f5469bbe700 1 -- 172.21.15.162:0/1857226360 >> 172.21.15.162:0/1857226360 conn(0x7f54640f38c0 msgr2=0x7f54640f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:55.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.322+0000 7f5469bbe700 1 -- 172.21.15.162:0/1857226360 shutdown_connections 2024-02-13T00:43:55.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.322+0000 7f5469bbe700 1 -- 172.21.15.162:0/1857226360 wait complete. 2024-02-13T00:43:55.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5469bbe700 1 librados: starting msgr at 2024-02-13T00:43:55.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5469bbe700 1 librados: starting objecter 2024-02-13T00:43:55.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5469bbe700 1 Processor -- start 2024-02-13T00:43:55.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5469bbe700 1 -- start start 2024-02-13T00:43:55.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5469bbe700 1 librados: setting wanted keys 2024-02-13T00:43:55.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5469bbe700 1 librados: calling monclient init 2024-02-13T00:43:55.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5469bbe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54640f8500 0x7f546410e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:55.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5469bbe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54640f9b70 0x7f546410e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:55.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5469bbe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54640fa440 0x7f5464108490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:55.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5469bbe700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464108980 con 0x7f54640fa440 2024-02-13T00:43:55.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5469bbe700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464108af0 con 0x7f54640f9b70 2024-02-13T00:43:55.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5469bbe700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5464108cb0 con 0x7f54640f8500 2024-02-13T00:43:55.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f54637fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54640f8500 0x7f546410e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:55.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f5463fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54640fa440 0x7f5464108490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:55.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f54637fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54640f8500 0x7f546410e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:36270/0 (socket says 172.21.15.162:36270) 2024-02-13T00:43:55.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.323+0000 7f54637fe700 1 -- 172.21.15.162:0/4015993730 learned_addr learned my addr 172.21.15.162:0/4015993730 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f545bfff700 1 --2- 172.21.15.162:0/4015993730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54640f9b70 0x7f546410e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f5463fff700 1 -- 172.21.15.162:0/4015993730 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54640f8500 msgr2=0x7f546410e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f5463fff700 1 --2- 172.21.15.162:0/4015993730 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54640f8500 0x7f546410e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f5463fff700 1 -- 172.21.15.162:0/4015993730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54640f9b70 msgr2=0x7f546410e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f5463fff700 1 --2- 172.21.15.162:0/4015993730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54640f9b70 0x7f546410e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f5463fff700 1 -- 172.21.15.162:0/4015993730 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544c005040 con 0x7f54640fa440 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f545bfff700 1 --2- 172.21.15.162:0/4015993730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54640f9b70 0x7f546410e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f54637fe700 1 --2- 172.21.15.162:0/4015993730 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54640f8500 0x7f546410e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f5463fff700 1 --2- 172.21.15.162:0/4015993730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54640fa440 0x7f5464108490 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7f545c00eb70 tx=0x7f545c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f54617fa700 10 client.?.objecter ms_handle_connect 0x7f54640fa440 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f54617fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f54617fa700 1 -- 172.21.15.162:0/4015993730 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c009870 con 0x7f54640fa440 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f5469bbe700 1 -- 172.21.15.162:0/4015993730 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5464108f30 con 0x7f54640fa440 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f54617fa700 1 -- 172.21.15.162:0/4015993730 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f545c00f040 con 0x7f54640fa440 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f54617fa700 1 -- 172.21.15.162:0/4015993730 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c007e40 con 0x7f54640fa440 2024-02-13T00:43:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f5469bbe700 10 client.26952.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f5469bbe700 1 -- 172.21.15.162:0/4015993730 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54641150d0 con 0x7f54640fa440 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.324+0000 7f5469bbe700 1 librados: init done 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.325+0000 7f5469bbe700 10 client.26952.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.325+0000 7f5469bbe700 15 client.26952.objecter _session_command_op_assign -1 1 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.325+0000 7f5469bbe700 20 client.26952.objecter _get_session osd=-1 returning homeless 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.325+0000 7f5469bbe700 10 client.26952.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.325+0000 7f5469bbe700 1 -- 172.21.15.162:0/4015993730 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f54640475e0 con 0x7f54640fa440 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 1 -- 172.21.15.162:0/4015993730 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f545c004190 con 0x7f54640fa440 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 1 --2- 172.21.15.162:0/4015993730 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5450062c90 0x7f5450065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 1 -- 172.21.15.162:0/4015993730 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(279..279 src has 52..279) v4 ==== 8811+0+0 (secure 0 0 0) 0x7f545c07d920 con 0x7f54640fa440 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 10 client.26952.objecter ms_dispatch 0x7f5464001550 osd_map(279..279 src has 52..279) v4 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 3 client.26952.objecter handle_osd_map got epochs [279,279] > 0 2024-02-13T00:43:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 3 client.26952.objecter handle_osd_map decoding full epoch 279 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54637fe700 1 --2- 172.21.15.162:0/4015993730 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5450062c90 0x7f5450065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 10 client.26952.objecter checking command 1 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 1 --2- 172.21.15.162:0/4015993730 >> v2:172.21.15.162:6805/150599780 conn(0x7f5450070e20 0x7f54500732c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 20 client.26952.objecter _get_session s=0x7f5450070770 osd=1 3 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 20 client.26952.objecter put_session s=0x7f5450070770 osd=1 3 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 15 client.26952.objecter _session_command_op_remove -1 1 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 20 client.26952.objecter _get_session s=0x7f5450070770 osd=1 3 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 20 client.26952.objecter get_session s=0x7f5450070770 osd=1 3 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 15 client.26952.objecter _session_command_op_assign 1 1 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 20 client.26952.objecter put_session s=0x7f5450070770 osd=1 4 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 10 client.26952.objecter _send_command 1 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 1 -- 172.21.15.162:0/4015993730 --> v2:172.21.15.162:6805/150599780 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5450073900 con 0x7f5450070e20 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 20 client.26952.objecter dump_active .. 0 homeless 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 1 -- 172.21.15.162:0/4015993730 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=279) v2 ==== 24+0+0 (secure 0 0 0) 0x7f545c07dc90 con 0x7f54640fa440 2024-02-13T00:43:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f545bfff700 1 --2- 172.21.15.162:0/4015993730 >> v2:172.21.15.162:6805/150599780 conn(0x7f5450070e20 0x7f54500732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54637fe700 1 --2- 172.21.15.162:0/4015993730 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5450062c90 0x7f5450065150 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 rx=0x7f5454009f00 tx=0x7f54540090f0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.326+0000 7f54617fa700 10 client.26952.objecter ms_handle_connect 0x7f5450062c90 2024-02-13T00:43:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.327+0000 7f54617fa700 10 client.26952.objecter ms_handle_connect 0x7f5450070e20 2024-02-13T00:43:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.327+0000 7f545bfff700 1 --2- 172.21.15.162:0/4015993730 >> v2:172.21.15.162:6805/150599780 conn(0x7f5450070e20 0x7f54500732c0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.327+0000 7f54617fa700 1 -- 172.21.15.162:0/4015993730 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5450073900 con 0x7f5450070e20 2024-02-13T00:43:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.327+0000 7f54617fa700 10 client.26952.objecter ms_dispatch 0x7f5464001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:43:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.327+0000 7f54617fa700 10 client.26952.objecter _finish_command 1 = system:0 2024-02-13T00:43:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.327+0000 7f54617fa700 20 client.26952.objecter put_session s=0x7f5450070770 osd=1 4 2024-02-13T00:43:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.327+0000 7f54617fa700 15 client.26952.objecter _session_command_op_remove 1 1 2024-02-13T00:43:55.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.344+0000 7f5469bbe700 10 client.26952.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-02-13T00:43:55.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.344+0000 7f5469bbe700 15 client.26952.objecter _session_command_op_assign -1 2 2024-02-13T00:43:55.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.344+0000 7f5469bbe700 20 client.26952.objecter _get_session s=0x7f5450070770 osd=1 3 2024-02-13T00:43:55.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.344+0000 7f5469bbe700 20 client.26952.objecter put_session s=0x7f5450070770 osd=1 3 2024-02-13T00:43:55.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.344+0000 7f5469bbe700 20 client.26952.objecter _get_session s=0x7f5450070770 osd=1 3 2024-02-13T00:43:55.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.344+0000 7f5469bbe700 15 client.26952.objecter _session_command_op_remove -1 2 2024-02-13T00:43:55.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.344+0000 7f5469bbe700 20 client.26952.objecter get_session s=0x7f5450070770 osd=1 3 2024-02-13T00:43:55.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.344+0000 7f5469bbe700 15 client.26952.objecter _session_command_op_assign 1 2 2024-02-13T00:43:55.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.344+0000 7f5469bbe700 20 client.26952.objecter put_session s=0x7f5450070770 osd=1 4 2024-02-13T00:43:55.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.344+0000 7f5469bbe700 10 client.26952.objecter _send_command 2 2024-02-13T00:43:55.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.344+0000 7f5469bbe700 1 -- 172.21.15.162:0/4015993730 --> v2:172.21.15.162:6805/150599780 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7f5464107250 con 0x7f5450070e20 2024-02-13T00:43:55.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f54617fa700 1 -- 172.21.15.162:0/4015993730 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5464107250 con 0x7f5450070e20 2024-02-13T00:43:55.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f54617fa700 10 client.26952.objecter ms_dispatch 0x7f5464001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:43:55.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f54617fa700 10 client.26952.objecter _finish_command 2 = system:0 2024-02-13T00:43:55.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f54617fa700 20 client.26952.objecter put_session s=0x7f5450070770 osd=1 4 2024-02-13T00:43:55.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f54617fa700 15 client.26952.objecter _session_command_op_remove 1 2 2024-02-13T00:43:55.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 10 librados: watch_flush enter 2024-02-13T00:43:55.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 10 librados: watch_flush exit 2024-02-13T00:43:55.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 10 client.26952.objecter close_session for osd.1 2024-02-13T00:43:55.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 1 -- 172.21.15.162:0/4015993730 >> v2:172.21.15.162:6805/150599780 conn(0x7f5450070e20 msgr2=0x7f54500732c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:55.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 1 --2- 172.21.15.162:0/4015993730 >> v2:172.21.15.162:6805/150599780 conn(0x7f5450070e20 0x7f54500732c0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:55.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 20 client.26952.objecter put_session s=0x7f5450070770 osd=1 1 2024-02-13T00:43:55.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 20 client.26952.objecter shutdown clearing up homeless session... 2024-02-13T00:43:55.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 10 client.26952.objecter successfully canceled tick 2024-02-13T00:43:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 1 -- 172.21.15.162:0/4015993730 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5450062c90 msgr2=0x7f5450065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 1 --2- 172.21.15.162:0/4015993730 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5450062c90 0x7f5450065150 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 rx=0x7f5454009f00 tx=0x7f54540090f0).stop 2024-02-13T00:43:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 1 -- 172.21.15.162:0/4015993730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54640fa440 msgr2=0x7f5464108490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 1 --2- 172.21.15.162:0/4015993730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54640fa440 0x7f5464108490 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7f545c00eb70 tx=0x7f545c00b040).stop 2024-02-13T00:43:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5463fff700 1 -- 172.21.15.162:0/4015993730 reap_dead start 2024-02-13T00:43:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 1 -- 172.21.15.162:0/4015993730 shutdown_connections 2024-02-13T00:43:55.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 1 -- 172.21.15.162:0/4015993730 >> 172.21.15.162:0/4015993730 conn(0x7f54640f38c0 msgr2=0x7f54640f45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:55.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 1 -- 172.21.15.162:0/4015993730 shutdown_connections 2024-02-13T00:43:55.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 1 -- 172.21.15.162:0/4015993730 wait complete. 2024-02-13T00:43:55.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:55.345+0000 7f5469bbe700 1 librados: shutdown 2024-02-13T00:43:55.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:54.209733+0000 osd.3 (osd.3) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:55.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:54.209880+0000 osd.3 (osd.3) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:55.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:54.227592+0000 mon.b (mon.1) 411 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:55.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:54.367748+0000 mon.a (mon.0) 1862 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:55.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:54.371158+0000 mgr.y (mgr.14544) 273 : cluster [DBG] pgmap v389: 270 pgs: 2 active+clean+snaptrim, 32 unknown, 6 active+clean+snaptrim_wait, 230 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:55.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:54.372976+0000 mon.a (mon.0) 1863 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-33", "mode": "writeback"}]': finished 2024-02-13T00:43:55.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:54.373057+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]': finished 2024-02-13T00:43:55.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:54.373118+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-83985-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:55.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:54.373179+0000 mon.a (mon.0) 1866 : cluster [DBG] osdmap e279: 8 total, 8 up, 6 in 2024-02-13T00:43:55.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:54.376695+0000 mon.c (mon.2) 265 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:55.448 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:54.377859+0000 mon.a (mon.0) 1867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:55.448 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:54.377898+0000 mon.c (mon.2) 266 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-83985-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:55.448 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:54.378220+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-83985-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:55.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:54.209733+0000 osd.3 (osd.3) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:55.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:54.209880+0000 osd.3 (osd.3) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:55.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:54.227592+0000 mon.b (mon.1) 411 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:55.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:54.367748+0000 mon.a (mon.0) 1862 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:55.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:54.371158+0000 mgr.y (mgr.14544) 273 : cluster [DBG] pgmap v389: 270 pgs: 2 active+clean+snaptrim, 32 unknown, 6 active+clean+snaptrim_wait, 230 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:55.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:54.372976+0000 mon.a (mon.0) 1863 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-33", "mode": "writeback"}]': finished 2024-02-13T00:43:55.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:54.373057+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]': finished 2024-02-13T00:43:55.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:54.373118+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-83985-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:55.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:54.373179+0000 mon.a (mon.0) 1866 : cluster [DBG] osdmap e279: 8 total, 8 up, 6 in 2024-02-13T00:43:55.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:54.376695+0000 mon.c (mon.2) 265 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:55.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:54.377859+0000 mon.a (mon.0) 1867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:55.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:54.377898+0000 mon.c (mon.2) 266 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-83985-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:55.449 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:54.378220+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-83985-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:55.451 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:55.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:55.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:54.209733+0000 osd.3 (osd.3) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:55.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:54.209880+0000 osd.3 (osd.3) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:55.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:54.227592+0000 mon.b (mon.1) 411 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:55.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:54.367748+0000 mon.a (mon.0) 1862 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:43:55.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:54.371158+0000 mgr.y (mgr.14544) 273 : cluster [DBG] pgmap v389: 270 pgs: 2 active+clean+snaptrim, 32 unknown, 6 active+clean+snaptrim_wait, 230 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:43:55.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:54.372976+0000 mon.a (mon.0) 1863 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-33", "mode": "writeback"}]': finished 2024-02-13T00:43:55.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:54.373057+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-83977-39"}]': finished 2024-02-13T00:43:55.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:54.373118+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-83985-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:55.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:54.373179+0000 mon.a (mon.0) 1866 : cluster [DBG] osdmap e279: 8 total, 8 up, 6 in 2024-02-13T00:43:55.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:54.376695+0000 mon.c (mon.2) 265 : audit [INF] from='client.? 172.21.15.162:0/2250253721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:55.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:54.377859+0000 mon.a (mon.0) 1867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]: dispatch 2024-02-13T00:43:55.491 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:54.377898+0000 mon.c (mon.2) 266 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-83985-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:55.491 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:54.378220+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-83985-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:55.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:55.585 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:55.636 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:55.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:55.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:55.642 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:55.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:43:55.770 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:55.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:43:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:43:56.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.2 reset_purged_snaps_last 2024-02-13T00:43:56.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:55.228621+0000 mon.b (mon.1) 412 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:56.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:55.354663+0000 mon.a (mon.0) 1869 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]': finished 2024-02-13T00:43:56.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:55.378467+0000 mon.c (mon.2) 267 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:56.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:55.487085+0000 mgr.y (mgr.14544) 274 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:56.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:55.602404+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e280: 8 total, 8 up, 6 in 2024-02-13T00:43:56.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:55.603394+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:56.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:55.605923+0000 mon.c (mon.2) 268 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:56.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:55.606844+0000 mon.a (mon.0) 1872 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:56.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:55.607512+0000 mon.c (mon.2) 269 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-83977-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:56.461 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:55.617268+0000 mon.a (mon.0) 1873 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-83977-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:56.462 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:55.753697+0000 mon.c (mon.2) 270 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:43:56.462 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:55.995669+0000 mon.a (mon.0) 1874 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:56.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:55.228621+0000 mon.b (mon.1) 412 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:56.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:55.354663+0000 mon.a (mon.0) 1869 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]': finished 2024-02-13T00:43:56.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:55.378467+0000 mon.c (mon.2) 267 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:56.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:55.487085+0000 mgr.y (mgr.14544) 274 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:56.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:55.602404+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e280: 8 total, 8 up, 6 in 2024-02-13T00:43:56.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:55.603394+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:56.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:55.605923+0000 mon.c (mon.2) 268 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:56.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:55.606844+0000 mon.a (mon.0) 1872 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:56.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:55.607512+0000 mon.c (mon.2) 269 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-83977-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:56.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:55.617268+0000 mon.a (mon.0) 1873 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-83977-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:56.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:55.753697+0000 mon.c (mon.2) 270 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:43:56.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:55.995669+0000 mon.a (mon.0) 1874 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:56.463 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:43:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:43:56] "GET /metrics HTTP/1.1" 200 188730 "" "Prometheus/2.33.4" 2024-02-13T00:43:56.518 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:43:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:56.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:43:56.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:55.228621+0000 mon.b (mon.1) 412 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:55.354663+0000 mon.a (mon.0) 1869 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-83977-39"}]': finished 2024-02-13T00:43:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:55.378467+0000 mon.c (mon.2) 267 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:55.487085+0000 mgr.y (mgr.14544) 274 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:43:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:55.602404+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e280: 8 total, 8 up, 6 in 2024-02-13T00:43:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:55.603394+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:55.605923+0000 mon.c (mon.2) 268 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:55.606844+0000 mon.a (mon.0) 1872 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:55.607512+0000 mon.c (mon.2) 269 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-83977-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:55.617268+0000 mon.a (mon.0) 1873 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-83977-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:43:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:55.753697+0000 mon.c (mon.2) 270 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:43:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:55.995669+0000 mon.a (mon.0) 1874 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:56.622 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:43:56.649 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:56.998 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:43:57.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-13T00:43:57.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:57.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:43:57.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:57.132 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:57.151 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:57.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:57.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:57.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:57.284 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:57.304 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:57.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:56.229683+0000 mon.b (mon.1) 413 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:56.275491+0000 mon.c (mon.2) 271 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:43:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:56.276215+0000 mon.c (mon.2) 272 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:43:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:56.281901+0000 mon.a (mon.0) 1875 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:56.356242+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-83985-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-83985-49"}]': finished 2024-02-13T00:43:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:56.356338+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-83977-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:56.356430+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e281: 8 total, 8 up, 6 in 2024-02-13T00:43:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:56.362811+0000 mon.c (mon.2) 273 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-83977-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:56.363223+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-83977-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:56.371560+0000 mgr.y (mgr.14544) 275 : cluster [DBG] pgmap v392: 278 pgs: 8 unknown, 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:56.398031+0000 mon.b (mon.1) 414 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:57.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:56.398554+0000 mon.a (mon.0) 1880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:57.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:56.229683+0000 mon.b (mon.1) 413 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:57.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:56.275491+0000 mon.c (mon.2) 271 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:43:57.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:56.276215+0000 mon.c (mon.2) 272 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:43:57.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:56.281901+0000 mon.a (mon.0) 1875 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:57.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:56.356242+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-83985-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-83985-49"}]': finished 2024-02-13T00:43:57.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:56.356338+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-83977-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:57.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:56.356430+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e281: 8 total, 8 up, 6 in 2024-02-13T00:43:57.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:56.362811+0000 mon.c (mon.2) 273 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-83977-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:57.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:56.363223+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-83977-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:57.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:56.371560+0000 mgr.y (mgr.14544) 275 : cluster [DBG] pgmap v392: 278 pgs: 8 unknown, 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:57.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:56.398031+0000 mon.b (mon.1) 414 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:57.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:56.398554+0000 mon.a (mon.0) 1880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:57.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:43:57.432 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:56.229683+0000 mon.b (mon.1) 413 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:57.432 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:56.275491+0000 mon.c (mon.2) 271 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:43:57.433 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:56.276215+0000 mon.c (mon.2) 272 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:43:57.433 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:56.281901+0000 mon.a (mon.0) 1875 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:43:57.433 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:56.356242+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-83985-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-83985-49"}]': finished 2024-02-13T00:43:57.433 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:56.356338+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-83977-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:43:57.433 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:56.356430+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e281: 8 total, 8 up, 6 in 2024-02-13T00:43:57.433 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:56.362811+0000 mon.c (mon.2) 273 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-83977-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:57.433 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:56.363223+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-83977-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:43:57.433 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:56.371560+0000 mgr.y (mgr.14544) 275 : cluster [DBG] pgmap v392: 278 pgs: 8 unknown, 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:57.433 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:56.398031+0000 mon.b (mon.1) 414 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:57.433 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:56.398554+0000 mon.a (mon.0) 1880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:43:57.434 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:57.453 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:43:57.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:57.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:43:57.472 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:43:57.557 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:57.583 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:57.602 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:57.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:57.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:57.731 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:57.749 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:57.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:57.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:57.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:57.879 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:57.966 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:57.976 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:57.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:57.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:58.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:58.107 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:58.256 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:58.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:58.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:43:58.392 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:57.231231+0000 mon.b (mon.1) 415 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:58.393 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:57.607758+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:58.393 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:57.607858+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e282: 8 total, 8 up, 6 in 2024-02-13T00:43:58.393 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:57.610663+0000 mon.b (mon.1) 416 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:43:58.393 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:57.611214+0000 mon.a (mon.0) 1883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:43:58.393 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:57.231231+0000 mon.b (mon.1) 415 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:58.393 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:57.607758+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:58.393 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:57.607858+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e282: 8 total, 8 up, 6 in 2024-02-13T00:43:58.393 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:57.610663+0000 mon.b (mon.1) 416 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:43:58.393 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:57.611214+0000 mon.a (mon.0) 1883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:43:58.395 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:58.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:57.231231+0000 mon.b (mon.1) 415 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:58.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:57.607758+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:43:58.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:57.607858+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e282: 8 total, 8 up, 6 in 2024-02-13T00:43:58.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:57.610663+0000 mon.b (mon.1) 416 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:43:58.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:57.611214+0000 mon.a (mon.0) 1883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:43:58.592 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:43:58.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:58.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:43:58.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:43:58.725 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:58.745 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:43:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:58.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:43:58.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:58.876 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:59.006 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:59.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:59.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:59.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:43:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:43:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:43:59.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:43:59.140 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:59.159 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:43:59.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:59.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:43:59.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:59.292 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:59.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.332+0000 7efd0dfa8700 1 -- 172.21.15.162:0/2481621360 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd08102270 msgr2=0x7efd08104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:59.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.332+0000 7efd0dfa8700 1 --2- 172.21.15.162:0/2481621360 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd08102270 0x7efd08104660 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 rx=0x7efcf8004870 tx=0x7efcf8009cb0).stop 2024-02-13T00:43:59.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.332+0000 7efd0dfa8700 1 -- 172.21.15.162:0/2481621360 shutdown_connections 2024-02-13T00:43:59.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.332+0000 7efd0dfa8700 1 --2- 172.21.15.162:0/2481621360 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd08102270 0x7efd08104660 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:59.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.332+0000 7efd0dfa8700 1 --2- 172.21.15.162:0/2481621360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd080f8d10 0x7efd080f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:59.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.332+0000 7efd0dfa8700 1 --2- 172.21.15.162:0/2481621360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd080f95e0 0x7efd08101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:59.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.332+0000 7efd0dfa8700 1 -- 172.21.15.162:0/2481621360 >> 172.21.15.162:0/2481621360 conn(0x7efd0806eb90 msgr2=0x7efd080f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:43:59.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.332+0000 7efd0dfa8700 1 -- 172.21.15.162:0/2481621360 shutdown_connections 2024-02-13T00:43:59.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.332+0000 7efd0dfa8700 1 -- 172.21.15.162:0/2481621360 wait complete. 2024-02-13T00:43:59.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.333+0000 7efd0dfa8700 1 librados: starting msgr at 2024-02-13T00:43:59.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.333+0000 7efd0dfa8700 1 librados: starting objecter 2024-02-13T00:43:59.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.333+0000 7efd0dfa8700 1 Processor -- start 2024-02-13T00:43:59.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.333+0000 7efd0dfa8700 1 -- start start 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.333+0000 7efd0dfa8700 1 librados: setting wanted keys 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.333+0000 7efd0dfa8700 1 librados: calling monclient init 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.333+0000 7efd0dfa8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd080f8d10 0x7efd0810dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.333+0000 7efd0dfa8700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd080f95e0 0x7efd0810e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.333+0000 7efd0dfa8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd08102270 0x7efd08108180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.333+0000 7efd0dfa8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd08108670 con 0x7efd08102270 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd0dfa8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd081087e0 con 0x7efd080f8d10 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd0dfa8700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd08108950 con 0x7efd080f95e0 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd07fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd08102270 0x7efd08108180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd07fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd08102270 0x7efd08108180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59762/0 (socket says 172.21.15.162:59762) 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd07fff700 1 -- 172.21.15.162:0/4294192676 learned_addr learned my addr 172.21.15.162:0/4294192676 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd06ffd700 1 --2- 172.21.15.162:0/4294192676 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd080f95e0 0x7efd0810e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:59.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd077fe700 1 --2- 172.21.15.162:0/4294192676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd080f8d10 0x7efd0810dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:59.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd07fff700 1 -- 172.21.15.162:0/4294192676 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd080f95e0 msgr2=0x7efd0810e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:59.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd07fff700 1 --2- 172.21.15.162:0/4294192676 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd080f95e0 0x7efd0810e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:59.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd07fff700 1 -- 172.21.15.162:0/4294192676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd080f8d10 msgr2=0x7efd0810dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:43:59.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd07fff700 1 --2- 172.21.15.162:0/4294192676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd080f8d10 0x7efd0810dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:43:59.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd07fff700 1 -- 172.21.15.162:0/4294192676 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf0005040 con 0x7efd08102270 2024-02-13T00:43:59.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd06ffd700 1 --2- 172.21.15.162:0/4294192676 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd080f95e0 0x7efd0810e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:43:59.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd07fff700 1 --2- 172.21.15.162:0/4294192676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd08102270 0x7efd08108180 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 rx=0x7efcf80043d0 tx=0x7efcf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:59.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd04ff9700 10 client.?.objecter ms_handle_connect 0x7efd08102270 2024-02-13T00:43:59.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd04ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:43:59.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.334+0000 7efd04ff9700 1 -- 172.21.15.162:0/4294192676 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf800e070 con 0x7efd08102270 2024-02-13T00:43:59.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.335+0000 7efd0dfa8700 1 -- 172.21.15.162:0/4294192676 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf8005040 con 0x7efd08102270 2024-02-13T00:43:59.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.335+0000 7efd0dfa8700 10 client.26964.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:43:59.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.335+0000 7efd0dfa8700 1 -- 172.21.15.162:0/4294192676 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd08108e40 con 0x7efd08102270 2024-02-13T00:43:59.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.335+0000 7efd0dfa8700 1 librados: init done 2024-02-13T00:43:59.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.335+0000 7efd04ff9700 1 -- 172.21.15.162:0/4294192676 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcf801e420 con 0x7efd08102270 2024-02-13T00:43:59.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.335+0000 7efd04ff9700 1 -- 172.21.15.162:0/4294192676 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8019070 con 0x7efd08102270 2024-02-13T00:43:59.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.336+0000 7efd0dfa8700 1 -- 172.21.15.162:0/4294192676 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd080475e0 con 0x7efd08102270 2024-02-13T00:43:59.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.336+0000 7efd04ff9700 1 -- 172.21.15.162:0/4294192676 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7efcf80139d0 con 0x7efd08102270 2024-02-13T00:43:59.344 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:43:59.346 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:59.346 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:43:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:43:59.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:43:59.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.341+0000 7efd04ff9700 1 --2- 172.21.15.162:0/4294192676 >> v2:172.21.15.162:6800/2866534053 conn(0x7efcf4062c90 0x7efcf4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:43:59.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.341+0000 7efd04ff9700 1 -- 172.21.15.162:0/4294192676 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(283..283 src has 52..283) v4 ==== 9605+0+0 (secure 0 0 0) 0x7efcf807e260 con 0x7efd08102270 2024-02-13T00:43:59.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.341+0000 7efd04ff9700 10 client.26964.objecter ms_dispatch 0x7efd08001550 osd_map(283..283 src has 52..283) v4 2024-02-13T00:43:59.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.341+0000 7efd04ff9700 3 client.26964.objecter handle_osd_map got epochs [283,283] > 0 2024-02-13T00:43:59.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.341+0000 7efd04ff9700 3 client.26964.objecter handle_osd_map decoding full epoch 283 2024-02-13T00:43:59.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.341+0000 7efd04ff9700 20 client.26964.objecter dump_active .. 0 homeless 2024-02-13T00:43:59.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.342+0000 7efd077fe700 1 --2- 172.21.15.162:0/4294192676 >> v2:172.21.15.162:6800/2866534053 conn(0x7efcf4062c90 0x7efcf4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:43:59.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.343+0000 7efd077fe700 1 --2- 172.21.15.162:0/4294192676 >> v2:172.21.15.162:6800/2866534053 conn(0x7efcf4062c90 0x7efcf4065150 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 rx=0x7efd00004fd0 tx=0x7efd00009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:43:59.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.343+0000 7efd04ff9700 10 client.26964.objecter ms_handle_connect 0x7efcf4062c90 2024-02-13T00:43:59.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.345+0000 7efd04ff9700 1 -- 172.21.15.162:0/4294192676 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7efcf8051120 con 0x7efd08102270 2024-02-13T00:43:59.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:43:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:58.232360+0000 mon.b (mon.1) 417 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:58.363240+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-83977-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-83977-40"}]': finished 2024-02-13T00:43:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:58.363356+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]': finished 2024-02-13T00:43:59.475 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:58.363416+0000 mon.a (mon.0) 1886 : cluster [DBG] osdmap e283: 8 total, 8 up, 6 in 2024-02-13T00:43:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:58.368490+0000 mon.c (mon.2) 274 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:58.369547+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:58.372072+0000 mgr.y (mgr.14544) 276 : cluster [DBG] pgmap v395: 278 pgs: 8 unknown, 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:58.862343+0000 osd.2 (osd.2) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:59.476 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:58.862505+0000 osd.2 (osd.2) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:59.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:58.232360+0000 mon.b (mon.1) 417 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:59.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:58.363240+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-83977-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-83977-40"}]': finished 2024-02-13T00:43:59.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:58.363356+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]': finished 2024-02-13T00:43:59.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:58.363416+0000 mon.a (mon.0) 1886 : cluster [DBG] osdmap e283: 8 total, 8 up, 6 in 2024-02-13T00:43:59.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:58.368490+0000 mon.c (mon.2) 274 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:59.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:58.369547+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:59.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:58.372072+0000 mgr.y (mgr.14544) 276 : cluster [DBG] pgmap v395: 278 pgs: 8 unknown, 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:59.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:58.862343+0000 osd.2 (osd.2) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:59.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:43:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:58.862505+0000 osd.2 (osd.2) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:59.477 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:43:59.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:58.232360+0000 mon.b (mon.1) 417 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:43:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:58.363240+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-83977-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-83977-40"}]': finished 2024-02-13T00:43:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:58.363356+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-33"}]': finished 2024-02-13T00:43:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:58.363416+0000 mon.a (mon.0) 1886 : cluster [DBG] osdmap e283: 8 total, 8 up, 6 in 2024-02-13T00:43:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:58.368490+0000 mon.c (mon.2) 274 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:58.369547+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:43:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:58.372072+0000 mgr.y (mgr.14544) 276 : cluster [DBG] pgmap v395: 278 pgs: 8 unknown, 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:43:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:58.862343+0000 osd.2 (osd.2) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:43:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:43:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:58.862505+0000 osd.2 (osd.2) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:43:59.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:43:59.562+0000 7efd0dfa8700 1 -- 172.21.15.162:0/4294192676 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7efd081096f0 con 0x7efd08102270 2024-02-13T00:44:00.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.355+0000 7efd04ff9700 1 -- 172.21.15.162:0/4294192676 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v285) v1 ==== 97+0+0 (secure 0 0 0) 0x7efcf8024250 con 0x7efd08102270 2024-02-13T00:44:00.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.360+0000 7efd0dfa8700 10 librados: watch_flush enter 2024-02-13T00:44:00.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 10 librados: watch_flush exit 2024-02-13T00:44:00.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 20 client.26964.objecter shutdown clearing up homeless session... 2024-02-13T00:44:00.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 10 client.26964.objecter successfully canceled tick 2024-02-13T00:44:00.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 1 -- 172.21.15.162:0/4294192676 >> v2:172.21.15.162:6800/2866534053 conn(0x7efcf4062c90 msgr2=0x7efcf4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:00.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 1 --2- 172.21.15.162:0/4294192676 >> v2:172.21.15.162:6800/2866534053 conn(0x7efcf4062c90 0x7efcf4065150 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 rx=0x7efd00004fd0 tx=0x7efd00009040).stop 2024-02-13T00:44:00.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 1 -- 172.21.15.162:0/4294192676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd08102270 msgr2=0x7efd08108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:00.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 1 --2- 172.21.15.162:0/4294192676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd08102270 0x7efd08108180 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 rx=0x7efcf80043d0 tx=0x7efcf800b040).stop 2024-02-13T00:44:00.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 1 -- 172.21.15.162:0/4294192676 shutdown_connections 2024-02-13T00:44:00.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 1 --2- 172.21.15.162:0/4294192676 >> v2:172.21.15.162:6800/2866534053 conn(0x7efcf4062c90 0x7efcf4065150 secure :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 rx=0x7efd00004fd0 tx=0x7efd00009040).stop 2024-02-13T00:44:00.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 1 --2- 172.21.15.162:0/4294192676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd08102270 0x7efd08108180 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:00.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 1 --2- 172.21.15.162:0/4294192676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd080f8d10 0x7efd0810dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:00.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 1 --2- 172.21.15.162:0/4294192676 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd080f95e0 0x7efd0810e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:00.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.362+0000 7efd0dfa8700 1 -- 172.21.15.162:0/4294192676 >> 172.21.15.162:0/4294192676 conn(0x7efd0806eb90 msgr2=0x7efd080f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:00.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.365+0000 7efd0dfa8700 1 -- 172.21.15.162:0/4294192676 shutdown_connections 2024-02-13T00:44:00.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.365+0000 7efd0dfa8700 1 -- 172.21.15.162:0/4294192676 wait complete. 2024-02-13T00:44:00.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:00.365+0000 7efd0dfa8700 1 librados: shutdown 2024-02-13T00:44:00.370 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:44:00.394 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:59.233279+0000 mon.b (mon.1) 418 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:00.394 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:59.369225+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]': finished 2024-02-13T00:44:00.394 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:43:59.369330+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e284: 8 total, 8 up, 6 in 2024-02-13T00:44:00.394 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:59.372411+0000 mon.c (mon.2) 275 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:44:00.394 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:59.372928+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:44:00.394 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:59.405101+0000 mon.b (mon.1) 419 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:00.394 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:59.405624+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:00.394 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:43:59.563487+0000 mon.a (mon.0) 1892 : audit [INF] from='client.? 172.21.15.162:0/4294192676' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:00.394 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:59.233279+0000 mon.b (mon.1) 418 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:00.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:59.369225+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]': finished 2024-02-13T00:44:00.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:43:59.369330+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e284: 8 total, 8 up, 6 in 2024-02-13T00:44:00.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:59.372411+0000 mon.c (mon.2) 275 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:44:00.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:59.372928+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:44:00.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:59.405101+0000 mon.b (mon.1) 419 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:00.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:59.405624+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:00.395 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:43:59.563487+0000 mon.a (mon.0) 1892 : audit [INF] from='client.? 172.21.15.162:0/4294192676' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:00.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:59.233279+0000 mon.b (mon.1) 418 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:00.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:59.369225+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-83985-49"}]': finished 2024-02-13T00:44:00.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:43:59.369330+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e284: 8 total, 8 up, 6 in 2024-02-13T00:44:00.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:59.372411+0000 mon.c (mon.2) 275 : audit [INF] from='client.? 172.21.15.162:0/2766912156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:44:00.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:59.372928+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]: dispatch 2024-02-13T00:44:00.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:59.405101+0000 mon.b (mon.1) 419 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:00.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:59.405624+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:00.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:43:59.563487+0000 mon.a (mon.0) 1892 : audit [INF] from='client.? 172.21.15.162:0/4294192676' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:00.755 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:00.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:00.859 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:00.891 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:00.912 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:00.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:00.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:01.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:01.046 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:01.066 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:01.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:01.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:01.209 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:00.234501+0000 mon.b (mon.1) 420 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:00.352126+0000 mon.a (mon.0) 1893 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:00.356021+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]': finished 2024-02-13T00:44:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:00.356121+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:44:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:00.356182+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? 172.21.15.162:0/4294192676' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:44:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:00.356227+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e285: 8 total, 8 up, 6 in 2024-02-13T00:44:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:00.357605+0000 mon.c (mon.2) 276 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:44:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:00.360989+0000 mon.b (mon.1) 421 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:44:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:00.366345+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:44:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:00.368937+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:44:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:00.372585+0000 mgr.y (mgr.14544) 277 : cluster [DBG] pgmap v398: 270 pgs: 2 active+clean+snaptrim, 268 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-13T00:44:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:00.234501+0000 mon.b (mon.1) 420 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:00.352126+0000 mon.a (mon.0) 1893 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:00.356021+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]': finished 2024-02-13T00:44:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:00.356121+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:44:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:00.356182+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? 172.21.15.162:0/4294192676' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:44:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:00.356227+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e285: 8 total, 8 up, 6 in 2024-02-13T00:44:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:00.357605+0000 mon.c (mon.2) 276 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:44:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:00.360989+0000 mon.b (mon.1) 421 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:44:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:00.366345+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:44:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:00.368937+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:44:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:00.372585+0000 mgr.y (mgr.14544) 277 : cluster [DBG] pgmap v398: 270 pgs: 2 active+clean+snaptrim, 268 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-13T00:44:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:00.234501+0000 mon.b (mon.1) 420 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:00.352126+0000 mon.a (mon.0) 1893 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:00.356021+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-83985-49"}]': finished 2024-02-13T00:44:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:00.356121+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:44:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:00.356182+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? 172.21.15.162:0/4294192676' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:44:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:00.356227+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e285: 8 total, 8 up, 6 in 2024-02-13T00:44:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:00.357605+0000 mon.c (mon.2) 276 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:44:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:00.360989+0000 mon.b (mon.1) 421 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:44:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:00.366345+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33"}]: dispatch 2024-02-13T00:44:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:00.368937+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:44:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:00.372585+0000 mgr.y (mgr.14544) 277 : cluster [DBG] pgmap v398: 270 pgs: 2 active+clean+snaptrim, 268 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-13T00:44:01.557 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:01.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:01.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:01.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:01.688 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:01.786 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:01.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:01.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:01.921 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:02.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.110+0000 7fbdec871700 1 -- 172.21.15.162:0/1733377211 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbde80f84c0 msgr2=0x7fbde8104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.110+0000 7fbdec871700 1 --2- 172.21.15.162:0/1733377211 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbde80f84c0 0x7fbde8104ac0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 rx=0x7fbde0009c30 tx=0x7fbde0004c50).stop 2024-02-13T00:44:02.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.110+0000 7fbdec871700 1 -- 172.21.15.162:0/1733377211 shutdown_connections 2024-02-13T00:44:02.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.110+0000 7fbdec871700 1 --2- 172.21.15.162:0/1733377211 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbde80f84c0 0x7fbde8104ac0 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.110+0000 7fbdec871700 1 --2- 172.21.15.162:0/1733377211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbde80fdc50 0x7fbde80fe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.110+0000 7fbdec871700 1 --2- 172.21.15.162:0/1733377211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbde80fe5b0 0x7fbde80f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.110+0000 7fbdec871700 1 -- 172.21.15.162:0/1733377211 >> 172.21.15.162:0/1733377211 conn(0x7fbde80f38a0 msgr2=0x7fbde80f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:02.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.110+0000 7fbdec871700 1 -- 172.21.15.162:0/1733377211 shutdown_connections 2024-02-13T00:44:02.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.110+0000 7fbdec871700 1 -- 172.21.15.162:0/1733377211 wait complete. 2024-02-13T00:44:02.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbdec871700 1 librados: starting msgr at 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbdec871700 1 librados: starting objecter 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbdec871700 1 Processor -- start 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbdec871700 1 -- start start 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbdec871700 1 librados: setting wanted keys 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbdec871700 1 librados: calling monclient init 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbdec871700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbde80f84c0 0x7fbde810df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbdec871700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbde80fdc50 0x7fbde810e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbdec871700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbde80fe5b0 0x7fbde8108150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbdec871700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8108640 con 0x7fbde80f84c0 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbdec871700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde81087b0 con 0x7fbde80fdc50 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbdec871700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8108970 con 0x7fbde80fe5b0 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbde659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbde80f84c0 0x7fbde810df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:02.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbde6d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbde80fe5b0 0x7fbde8108150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbde659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbde80f84c0 0x7fbde810df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59802/0 (socket says 172.21.15.162:59802) 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbde659c700 1 -- 172.21.15.162:0/3155837089 learned_addr learned my addr 172.21.15.162:0/3155837089 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.111+0000 7fbde5d9b700 1 --2- 172.21.15.162:0/3155837089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbde80fdc50 0x7fbde810e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbde659c700 1 -- 172.21.15.162:0/3155837089 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbde80fe5b0 msgr2=0x7fbde8108150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbde659c700 1 --2- 172.21.15.162:0/3155837089 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbde80fe5b0 0x7fbde8108150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbde659c700 1 -- 172.21.15.162:0/3155837089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbde80fdc50 msgr2=0x7fbde810e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbde659c700 1 --2- 172.21.15.162:0/3155837089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbde80fdc50 0x7fbde810e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbde659c700 1 -- 172.21.15.162:0/3155837089 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8005040 con 0x7fbde80f84c0 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbde6d9d700 1 --2- 172.21.15.162:0/3155837089 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbde80fe5b0 0x7fbde8108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbde659c700 1 --2- 172.21.15.162:0/3155837089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbde80f84c0 0x7fbde810df60 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7fbdd00041d0 tx=0x7fbdd0009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbddf7fe700 10 client.?.objecter ms_handle_connect 0x7fbde80f84c0 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbddf7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0007b40 con 0x7fbde80f84c0 2024-02-13T00:44:02.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde00097c0 con 0x7fbde80f84c0 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdd000f040 con 0x7fbde80f84c0 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbdec871700 10 client.26976.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde8108fb0 con 0x7fbde80f84c0 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbdec871700 1 librados: init done 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.112+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd00147e0 con 0x7fbde80f84c0 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.113+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fbde81071b0 con 0x7fbde80f84c0 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.113+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fbdd0014b90 con 0x7fbde80f84c0 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbddf7fe700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbdd4062c90 0x7fbdd4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(286..286 src has 52..286) v4 ==== 8915+0+0 (secure 0 0 0) 0x7fbdd007d070 con 0x7fbde80f84c0 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 osd_map(286..286 src has 52..286) v4 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbddf7fe700 3 client.26976.objecter handle_osd_map got epochs [286,286] > 0 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbddf7fe700 3 client.26976.objecter handle_osd_map decoding full epoch 286 2024-02-13T00:44:02.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbde5d9b700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbdd4062c90 0x7fbdd4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbddf7fe700 20 client.26976.objecter dump_active .. 0 homeless 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v286) v1 ==== 54+0+16 (secure 0 0 0) 0x7fbdd0084070 con 0x7fbde80f84c0 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbddf7fe700 10 client.26976.objecter ms_handle_connect 0x7fbdd4062c90 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbde5d9b700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbdd4062c90 0x7fbdd4065150 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7fbde81098a0 tx=0x7fbdd800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbdec871700 10 client.26976.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 1 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6801/2103743338 conn(0x7fbde80fef00 0x7fbde80ff2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde81095e0 osd=0 3 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde81095e0 osd=0 3 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde81095e0 osd=0 3 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 1 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde81095e0 osd=0 3 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 0 1 2024-02-13T00:44:02.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde81095e0 osd=0 4 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbdec871700 10 client.26976.objecter _send_command 1 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde80ff7d0 con 0x7fbde80fef00 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.114+0000 7fbde6d9d700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6801/2103743338 conn(0x7fbde80fef00 0x7fbde80ff2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.115+0000 7fbddf7fe700 10 client.26976.objecter ms_handle_connect 0x7fbde80fef00 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.115+0000 7fbde6d9d700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6801/2103743338 conn(0x7fbde80fef00 0x7fbde80ff2e0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.115+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde80ff7d0 con 0x7fbde80fef00 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.115+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.115+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 1 = system:0 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.115+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde81095e0 osd=0 4 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.115+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 0 1 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.130+0000 7fbdec871700 10 client.26976.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.130+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 2 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.130+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde81095e0 osd=0 3 2024-02-13T00:44:02.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.130+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde81095e0 osd=0 3 2024-02-13T00:44:02.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.130+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde81095e0 osd=0 3 2024-02-13T00:44:02.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.130+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 2 2024-02-13T00:44:02.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.130+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde81095e0 osd=0 3 2024-02-13T00:44:02.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.130+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 0 2 2024-02-13T00:44:02.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.130+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde81095e0 osd=0 4 2024-02-13T00:44:02.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.130+0000 7fbdec871700 10 client.26976.objecter _send_command 2 2024-02-13T00:44:02.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.130+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbde80ff940 con 0x7fbde80fef00 2024-02-13T00:44:02.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fbde80ff940 con 0x7fbde80fef00 2024-02-13T00:44:02.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:02.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde81095e0 osd=0 4 2024-02-13T00:44:02.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 0 2 2024-02-13T00:44:02.254 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:44:02.254 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:44:02.254 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:44:02.254 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:44:02.254 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:44:02.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbdec871700 10 client.26976.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 3 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6805/150599780 conn(0x7fbde8028590 0x7fbde8118ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde80f5c90 osd=1 3 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde80f5c90 osd=1 3 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde80f5c90 osd=1 3 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 3 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde80f5c90 osd=1 3 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 1 3 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde80f5c90 osd=1 4 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbdec871700 10 client.26976.objecter _send_command 3 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde80f5e00 con 0x7fbde8028590 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbde659c700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6805/150599780 conn(0x7fbde8028590 0x7fbde8118ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbddf7fe700 10 client.26976.objecter ms_handle_connect 0x7fbde8028590 2024-02-13T00:44:02.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.132+0000 7fbde659c700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6805/150599780 conn(0x7fbde8028590 0x7fbde8118ef0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.133+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde80f5e00 con 0x7fbde8028590 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.133+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.133+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 3 = system:0 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.133+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde80f5c90 osd=1 4 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.133+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 1 3 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.148+0000 7fbdec871700 10 client.26976.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.148+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 4 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.148+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde80f5c90 osd=1 3 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.148+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde80f5c90 osd=1 3 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.148+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde80f5c90 osd=1 3 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.148+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 4 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.148+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde80f5c90 osd=1 3 2024-02-13T00:44:02.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.148+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 1 4 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.148+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde80f5c90 osd=1 4 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.148+0000 7fbdec871700 10 client.26976.objecter _send_command 4 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.148+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbde805a0e0 con 0x7fbde8028590 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fbde805a0e0 con 0x7fbde8028590 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde80f5c90 osd=1 4 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 1 4 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbdec871700 10 client.26976.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 5 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6809/3689719376 conn(0x7fbde8059570 0x7fbde806c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde8059400 osd=2 3 2024-02-13T00:44:02.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde8059400 osd=2 3 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde8059400 osd=2 3 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 5 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde8059400 osd=2 3 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 2 5 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde8059400 osd=2 4 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbdec871700 10 client.26976.objecter _send_command 5 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.150+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde80280f0 con 0x7fbde8059570 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.151+0000 7fbde5d9b700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6809/3689719376 conn(0x7fbde8059570 0x7fbde806c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.151+0000 7fbddf7fe700 10 client.26976.objecter ms_handle_connect 0x7fbde8059570 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.151+0000 7fbde5d9b700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6809/3689719376 conn(0x7fbde8059570 0x7fbde806c130 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.151+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde80280f0 con 0x7fbde8059570 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.151+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.151+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 5 = system:0 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.151+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde8059400 osd=2 4 2024-02-13T00:44:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.151+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 2 5 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.166+0000 7fbdec871700 10 client.26976.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.166+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 6 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.166+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde8059400 osd=2 3 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.166+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde8059400 osd=2 3 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.166+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde8059400 osd=2 3 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.166+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 6 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.166+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde8059400 osd=2 3 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.166+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 2 6 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.166+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde8059400 osd=2 4 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.166+0000 7fbdec871700 10 client.26976.objecter _send_command 6 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.166+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbde806c620 con 0x7fbde8059570 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fbde806c620 con 0x7fbde8059570 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde8059400 osd=2 4 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 2 6 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbdec871700 10 client.26976.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 7 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6813/604971011 conn(0x7fbde811f800 0x7fbde8121bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde806d040 osd=3 3 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806d040 osd=3 3 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde806d040 osd=3 3 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 7 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde806d040 osd=3 3 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 3 7 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806d040 osd=3 4 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbdec871700 10 client.26976.objecter _send_command 7 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde8124990 con 0x7fbde811f800 2024-02-13T00:44:02.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.168+0000 7fbde6d9d700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6813/604971011 conn(0x7fbde811f800 0x7fbde8121bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.169+0000 7fbddf7fe700 10 client.26976.objecter ms_handle_connect 0x7fbde811f800 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.169+0000 7fbde6d9d700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6813/604971011 conn(0x7fbde811f800 0x7fbde8121bf0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.169+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde8124990 con 0x7fbde811f800 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.169+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.169+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 7 = system:0 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.169+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde806d040 osd=3 4 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.169+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 3 7 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.184+0000 7fbdec871700 10 client.26976.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.184+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 8 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.184+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde806d040 osd=3 3 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.184+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806d040 osd=3 3 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.184+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde806d040 osd=3 3 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.184+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 8 2024-02-13T00:44:02.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.184+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde806d040 osd=3 3 2024-02-13T00:44:02.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.184+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 3 8 2024-02-13T00:44:02.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.184+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806d040 osd=3 4 2024-02-13T00:44:02.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.184+0000 7fbdec871700 10 client.26976.objecter _send_command 8 2024-02-13T00:44:02.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.184+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbde806cab0 con 0x7fbde811f800 2024-02-13T00:44:02.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fbde806cab0 con 0x7fbde811f800 2024-02-13T00:44:02.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:02.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde806d040 osd=3 4 2024-02-13T00:44:02.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 3 8 2024-02-13T00:44:02.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbdec871700 10 client.26976.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:02.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 9 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6800/3344263388 conn(0x7fbde806bc90 0x7fbde81221b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde806cab0 osd=4 3 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806cab0 osd=4 3 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde806cab0 osd=4 3 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 9 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde806cab0 osd=4 3 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 4 9 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806cab0 osd=4 4 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbdec871700 10 client.26976.objecter _send_command 9 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.185+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde806cd40 con 0x7fbde806bc90 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.186+0000 7fbde659c700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6800/3344263388 conn(0x7fbde806bc90 0x7fbde81221b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.186+0000 7fbddf7fe700 10 client.26976.objecter ms_handle_connect 0x7fbde806bc90 2024-02-13T00:44:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.186+0000 7fbde659c700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6800/3344263388 conn(0x7fbde806bc90 0x7fbde81221b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:02.264 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:44:02.264 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:44:02.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.187+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde806cd40 con 0x7fbde806bc90 2024-02-13T00:44:02.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.187+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:44:02.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.187+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 9 = system:0 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.187+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde806cab0 osd=4 4 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.187+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 4 9 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.202+0000 7fbdec871700 10 client.26976.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.202+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 10 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.202+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde806cab0 osd=4 3 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.202+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806cab0 osd=4 3 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.202+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde806cab0 osd=4 3 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.202+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 10 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.202+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde806cab0 osd=4 3 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.202+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 4 10 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.202+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806cab0 osd=4 4 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.202+0000 7fbdec871700 10 client.26976.objecter _send_command 10 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.202+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbde806cd40 con 0x7fbde806bc90 2024-02-13T00:44:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fbde806cd40 con 0x7fbde806bc90 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde806cab0 osd=4 4 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 4 10 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbdec871700 10 client.26976.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 11 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6804/2212296748 conn(0x7fbde8123d30 0x7fbde8124170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde806cd40 osd=5 3 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806cd40 osd=5 3 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde806cd40 osd=5 3 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 11 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde806cd40 osd=5 3 2024-02-13T00:44:02.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 5 11 2024-02-13T00:44:02.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806cd40 osd=5 4 2024-02-13T00:44:02.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbdec871700 10 client.26976.objecter _send_command 11 2024-02-13T00:44:02.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.204+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde8124780 con 0x7fbde8123d30 2024-02-13T00:44:02.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.205+0000 7fbde5d9b700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6804/2212296748 conn(0x7fbde8123d30 0x7fbde8124170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:02.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.205+0000 7fbde5d9b700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6804/2212296748 conn(0x7fbde8123d30 0x7fbde8124170 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:02.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.205+0000 7fbddf7fe700 10 client.26976.objecter ms_handle_connect 0x7fbde8123d30 2024-02-13T00:44:02.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.206+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde8124780 con 0x7fbde8123d30 2024-02-13T00:44:02.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.206+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:44:02.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.206+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 11 = system:0 2024-02-13T00:44:02.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.206+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde806cd40 osd=5 4 2024-02-13T00:44:02.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.206+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 5 11 2024-02-13T00:44:02.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.221+0000 7fbdec871700 10 client.26976.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.221+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 12 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.221+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde806cd40 osd=5 3 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.221+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806cd40 osd=5 3 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.221+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde806cd40 osd=5 3 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.221+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 12 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.221+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde806cd40 osd=5 3 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.221+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 5 12 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.221+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806cd40 osd=5 4 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.221+0000 7fbdec871700 10 client.26976.objecter _send_command 12 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.221+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbde8124780 con 0x7fbde8123d30 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fbde8124780 con 0x7fbde8123d30 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde806cd40 osd=5 4 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 5 12 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbdec871700 10 client.26976.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 13 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6808/1506640341 conn(0x7fbde81290e0 0x7fbde8129540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde8124780 osd=6 3 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde8124780 osd=6 3 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde8124780 osd=6 3 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 13 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde8124780 osd=6 3 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 6 13 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde8124780 osd=6 4 2024-02-13T00:44:02.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbdec871700 10 client.26976.objecter _send_command 13 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde812b170 con 0x7fbde81290e0 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.223+0000 7fbde6d9d700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6808/1506640341 conn(0x7fbde81290e0 0x7fbde8129540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.224+0000 7fbddf7fe700 10 client.26976.objecter ms_handle_connect 0x7fbde81290e0 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.224+0000 7fbde6d9d700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6808/1506640341 conn(0x7fbde81290e0 0x7fbde8129540 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.224+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde812b170 con 0x7fbde81290e0 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.224+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.224+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 13 = system:0 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.224+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde8124780 osd=6 4 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.224+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 6 13 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.238+0000 7fbdec871700 10 client.26976.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.239+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 14 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.239+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde8124780 osd=6 3 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.239+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde8124780 osd=6 3 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.239+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde8124780 osd=6 3 2024-02-13T00:44:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.239+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 14 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.239+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde8124780 osd=6 3 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.239+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 6 14 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.239+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde8124780 osd=6 4 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.239+0000 7fbdec871700 10 client.26976.objecter _send_command 14 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.239+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbde812b170 con 0x7fbde81290e0 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fbde812b170 con 0x7fbde81290e0 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde8124780 osd=6 4 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 6 14 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbdec871700 10 client.26976.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 15 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6812/1964192772 conn(0x7fbde812d900 0x7fbde812dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde812b170 osd=7 3 2024-02-13T00:44:02.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde812b170 osd=7 3 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde812b170 osd=7 3 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 15 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde812b170 osd=7 3 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 7 15 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde812b170 osd=7 4 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbdec871700 10 client.26976.objecter _send_command 15 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde812e3b0 con 0x7fbde812d900 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbde659c700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6812/1964192772 conn(0x7fbde812d900 0x7fbde812dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbddf7fe700 10 client.26976.objecter ms_handle_connect 0x7fbde812d900 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.241+0000 7fbde659c700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6812/1964192772 conn(0x7fbde812d900 0x7fbde812dd40 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.242+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde812e3b0 con 0x7fbde812d900 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.242+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.242+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 15 = system:0 2024-02-13T00:44:02.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.242+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde812b170 osd=7 4 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.242+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 7 15 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.257+0000 7fbdec871700 10 client.26976.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.257+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign -1 16 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.257+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde812b170 osd=7 3 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.257+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde812b170 osd=7 3 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.257+0000 7fbdec871700 20 client.26976.objecter _get_session s=0x7fbde812b170 osd=7 3 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.257+0000 7fbdec871700 15 client.26976.objecter _session_command_op_remove -1 16 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.257+0000 7fbdec871700 20 client.26976.objecter get_session s=0x7fbde812b170 osd=7 3 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.257+0000 7fbdec871700 15 client.26976.objecter _session_command_op_assign 7 16 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.257+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde812b170 osd=7 4 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.257+0000 7fbdec871700 10 client.26976.objecter _send_command 16 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.257+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbde812e3b0 con 0x7fbde812d900 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbddf7fe700 1 -- 172.21.15.162:0/3155837089 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fbde812e3b0 con 0x7fbde812d900 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbddf7fe700 10 client.26976.objecter ms_dispatch 0x7fbde8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbddf7fe700 10 client.26976.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbddf7fe700 20 client.26976.objecter put_session s=0x7fbde812b170 osd=7 4 2024-02-13T00:44:02.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbddf7fe700 15 client.26976.objecter _session_command_op_remove 7 16 2024-02-13T00:44:02.274 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbdec871700 10 librados: watch_flush enter 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbdec871700 10 librados: watch_flush exit 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbdec871700 10 client.26976.objecter close_session for osd.0 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6801/2103743338 conn(0x7fbde80fef00 msgr2=0x7fbde80ff2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6801/2103743338 conn(0x7fbde80fef00 0x7fbde80ff2e0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde81095e0 osd=0 1 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbdec871700 10 client.26976.objecter close_session for osd.1 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.259+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6805/150599780 conn(0x7fbde8028590 msgr2=0x7fbde8118ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6805/150599780 conn(0x7fbde8028590 0x7fbde8118ef0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde80f5c90 osd=1 1 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 10 client.26976.objecter close_session for osd.2 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6809/3689719376 conn(0x7fbde8059570 msgr2=0x7fbde806c130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6809/3689719376 conn(0x7fbde8059570 0x7fbde806c130 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde8059400 osd=2 1 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 10 client.26976.objecter close_session for osd.3 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6813/604971011 conn(0x7fbde811f800 msgr2=0x7fbde8121bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6813/604971011 conn(0x7fbde811f800 0x7fbde8121bf0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806d040 osd=3 1 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbde6d9d700 1 -- 172.21.15.162:0/3155837089 reap_dead start 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 10 client.26976.objecter close_session for osd.4 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6800/3344263388 conn(0x7fbde806bc90 msgr2=0x7fbde81221b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6800/3344263388 conn(0x7fbde806bc90 0x7fbde81221b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbde6d9d700 1 -- 172.21.15.162:0/3155837089 reap_dead start 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806cab0 osd=4 1 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 10 client.26976.objecter close_session for osd.5 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6804/2212296748 conn(0x7fbde8123d30 msgr2=0x7fbde8124170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6804/2212296748 conn(0x7fbde8123d30 0x7fbde8124170 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde806cd40 osd=5 1 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 10 client.26976.objecter close_session for osd.6 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6808/1506640341 conn(0x7fbde81290e0 msgr2=0x7fbde8129540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6808/1506640341 conn(0x7fbde81290e0 0x7fbde8129540 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde8124780 osd=6 1 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 10 client.26976.objecter close_session for osd.7 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6812/1964192772 conn(0x7fbde812d900 msgr2=0x7fbde812dd40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.181:6812/1964192772 conn(0x7fbde812d900 0x7fbde812dd40 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 20 client.26976.objecter put_session s=0x7fbde812b170 osd=7 1 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 20 client.26976.objecter shutdown clearing up homeless session... 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 10 client.26976.objecter successfully canceled tick 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbdd4062c90 msgr2=0x7fbdd4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbdd4062c90 0x7fbdd4065150 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7fbde81098a0 tx=0x7fbdd800b040).stop 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbde80f84c0 msgr2=0x7fbde810df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 --2- 172.21.15.162:0/3155837089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbde80f84c0 0x7fbde810df60 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7fbdd00041d0 tx=0x7fbdd0009f60).stop 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbde6d9d700 1 -- 172.21.15.162:0/3155837089 reap_dead start 2024-02-13T00:44:02.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 shutdown_connections 2024-02-13T00:44:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 >> 172.21.15.162:0/3155837089 conn(0x7fbde80f38a0 msgr2=0x7fbde80f5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 shutdown_connections 2024-02-13T00:44:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 -- 172.21.15.162:0/3155837089 wait complete. 2024-02-13T00:44:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:02.260+0000 7fbdec871700 1 librados: shutdown 2024-02-13T00:44:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.278 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:44:02.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:01.235435+0000 mon.b (mon.1) 422 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:02.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:01.364183+0000 mon.a (mon.0) 1900 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:44:02.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:01.375727+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33"}]': finished 2024-02-13T00:44:02.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:01.375820+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]': finished 2024-02-13T00:44:02.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:01.375902+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e286: 8 total, 8 up, 6 in 2024-02-13T00:44:02.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:01.381965+0000 mon.c (mon.2) 277 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:44:02.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:01.382733+0000 mon.a (mon.0) 1904 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:44:02.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:01.383835+0000 mon.c (mon.2) 278 : audit [INF] from='client.? 172.21.15.162:0/1422030667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:02.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:01.384294+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:02.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:02.114877+0000 mon.a (mon.0) 1906 : audit [DBG] from='client.? 172.21.15.162:0/3155837089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:02.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:01.235435+0000 mon.b (mon.1) 422 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:02.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:01.364183+0000 mon.a (mon.0) 1900 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:44:02.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:01.375727+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33"}]': finished 2024-02-13T00:44:02.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:01.375820+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]': finished 2024-02-13T00:44:02.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:01.375902+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e286: 8 total, 8 up, 6 in 2024-02-13T00:44:02.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:01.381965+0000 mon.c (mon.2) 277 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:44:02.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:01.382733+0000 mon.a (mon.0) 1904 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:44:02.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:01.383835+0000 mon.c (mon.2) 278 : audit [INF] from='client.? 172.21.15.162:0/1422030667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:02.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:01.384294+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:02.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:02.114877+0000 mon.a (mon.0) 1906 : audit [DBG] from='client.? 172.21.15.162:0/3155837089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:02.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:01.235435+0000 mon.b (mon.1) 422 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:02.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:01.364183+0000 mon.a (mon.0) 1900 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:44:02.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:01.375727+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-33"}]': finished 2024-02-13T00:44:02.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:01.375820+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-83977-40"}]': finished 2024-02-13T00:44:02.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:01.375902+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e286: 8 total, 8 up, 6 in 2024-02-13T00:44:02.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:01.381965+0000 mon.c (mon.2) 277 : audit [INF] from='client.? 172.21.15.162:0/2949282291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:44:02.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:01.382733+0000 mon.a (mon.0) 1904 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]: dispatch 2024-02-13T00:44:02.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:01.383835+0000 mon.c (mon.2) 278 : audit [INF] from='client.? 172.21.15.162:0/1422030667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:02.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:01.384294+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:02.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:02.114877+0000 mon.a (mon.0) 1906 : audit [DBG] from='client.? 172.21.15.162:0/3155837089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:02.870 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:02.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:02.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:03.004 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:03.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:02.236759+0000 mon.b (mon.1) 423 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:02.372982+0000 mgr.y (mgr.14544) 278 : cluster [DBG] pgmap v400: 302 pgs: 32 unknown, 2 active+clean+snaptrim, 268 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-13T00:44:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:02.379268+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]': finished 2024-02-13T00:44:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:02.379351+0000 mon.a (mon.0) 1908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-50","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:02.379420+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e287: 8 total, 8 up, 6 in 2024-02-13T00:44:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:02.413546+0000 mon.c (mon.2) 279 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:02.639143+0000 mon.a (mon.0) 1910 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:02.639664+0000 mon.c (mon.2) 280 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:02.639938+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:02.640322+0000 mon.c (mon.2) 281 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-83977-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:02.640538+0000 mon.a (mon.0) 1912 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-83977-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:02.236759+0000 mon.b (mon.1) 423 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:02.372982+0000 mgr.y (mgr.14544) 278 : cluster [DBG] pgmap v400: 302 pgs: 32 unknown, 2 active+clean+snaptrim, 268 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-13T00:44:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:02.379268+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]': finished 2024-02-13T00:44:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:02.379351+0000 mon.a (mon.0) 1908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-50","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:02.379420+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e287: 8 total, 8 up, 6 in 2024-02-13T00:44:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:02.413546+0000 mon.c (mon.2) 279 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:02.639143+0000 mon.a (mon.0) 1910 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:02.639664+0000 mon.c (mon.2) 280 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:02.639938+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:02.640322+0000 mon.c (mon.2) 281 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-83977-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:02.640538+0000 mon.a (mon.0) 1912 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-83977-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:03.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.086+0000 7f1fd765d700 1 -- 172.21.15.162:0/225291164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd00f8470 msgr2=0x7f1fd0104a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:03.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.086+0000 7f1fd765d700 1 --2- 172.21.15.162:0/225291164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd00f8470 0x7f1fd0104a70 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 rx=0x7f1fc8009be0 tx=0x7f1fc8004d10).stop 2024-02-13T00:44:03.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.086+0000 7f1fd765d700 1 -- 172.21.15.162:0/225291164 shutdown_connections 2024-02-13T00:44:03.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.086+0000 7f1fd765d700 1 --2- 172.21.15.162:0/225291164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd00f8470 0x7f1fd0104a70 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:03.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.086+0000 7f1fd765d700 1 --2- 172.21.15.162:0/225291164 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd00fdc00 0x7f1fd00fdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:44:03.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.086+0000 7f1fd765d700 1 --2- 172.21.15.162:0/225291164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd00fe560 0x7f1fd00f7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:03.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.086+0000 7f1fd765d700 1 -- 172.21.15.162:0/225291164 >> 172.21.15.162:0/225291164 conn(0x7f1fd00f3850 msgr2=0x7f1fd00f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:03.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.086+0000 7f1fd765d700 1 -- 172.21.15.162:0/225291164 shutdown_connections 2024-02-13T00:44:03.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.086+0000 7f1fd765d700 1 -- 172.21.15.162:0/225291164 wait complete. 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd765d700 1 librados: starting msgr at 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd765d700 1 librados: starting objecter 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd765d700 1 Processor -- start 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd765d700 1 -- start start 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd765d700 1 librados: setting wanted keys 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd765d700 1 librados: calling monclient init 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd765d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd00f8470 0x7f1fd010df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd765d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd00fdc00 0x7f1fd010e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd765d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd00fe560 0x7f1fd0108160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd765d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd0108650 con 0x7f1fd00fdc00 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd765d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd01087c0 con 0x7f1fd00f8470 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd765d700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd0108930 con 0x7f1fd00fe560 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd4bf8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd00fdc00 0x7f1fd010e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:03.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd4bf8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd00fdc00 0x7f1fd010e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59840/0 (socket says 172.21.15.162:59840) 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd4bf8700 1 -- 172.21.15.162:0/1849087634 learned_addr learned my addr 172.21.15.162:0/1849087634 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.087+0000 7f1fd5bfa700 1 --2- 172.21.15.162:0/1849087634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd00fe560 0x7f1fd0108160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fd4bf8700 1 -- 172.21.15.162:0/1849087634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd00fe560 msgr2=0x7f1fd0108160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fd4bf8700 1 --2- 172.21.15.162:0/1849087634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd00fe560 0x7f1fd0108160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fd4bf8700 1 -- 172.21.15.162:0/1849087634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd00f8470 msgr2=0x7f1fd010df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fd4bf8700 1 --2- 172.21.15.162:0/1849087634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd00f8470 0x7f1fd010df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fd4bf8700 1 -- 172.21.15.162:0/1849087634 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc0005040 con 0x7f1fd00fdc00 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fd5bfa700 1 --2- 172.21.15.162:0/1849087634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd00fe560 0x7f1fd0108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fd4bf8700 1 --2- 172.21.15.162:0/1849087634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd00fdc00 0x7f1fd010e470 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f1fc00048f0 tx=0x7f1fc0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fc67fc700 10 client.?.objecter ms_handle_connect 0x7f1fd00fdc00 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fc67fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fc67fc700 1 -- 172.21.15.162:0/1849087634 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0007b00 con 0x7f1fd00fdc00 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fd765d700 1 -- 172.21.15.162:0/1849087634 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc80097c0 con 0x7f1fd00fdc00 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fc67fc700 1 -- 172.21.15.162:0/1849087634 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1fc000f040 con 0x7f1fd00fdc00 2024-02-13T00:44:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fc67fc700 1 -- 172.21.15.162:0/1849087634 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0014a00 con 0x7f1fd00fdc00 2024-02-13T00:44:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fd765d700 10 client.26988.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fd765d700 1 -- 172.21.15.162:0/1849087634 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd0114df0 con 0x7f1fd00fdc00 2024-02-13T00:44:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.088+0000 7f1fd765d700 1 librados: init done 2024-02-13T00:44:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.089+0000 7f1fd765d700 10 client.26988.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.089+0000 7f1fd765d700 15 client.26988.objecter _session_command_op_assign -1 1 2024-02-13T00:44:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.089+0000 7f1fd765d700 20 client.26988.objecter _get_session osd=-1 returning homeless 2024-02-13T00:44:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.089+0000 7f1fd765d700 10 client.26988.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.089+0000 7f1fd765d700 1 -- 172.21.15.162:0/1849087634 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1fd0060380 con 0x7f1fd00fdc00 2024-02-13T00:44:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 1 -- 172.21.15.162:0/1849087634 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1fc000e580 con 0x7f1fd00fdc00 2024-02-13T00:44:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 1 --2- 172.21.15.162:0/1849087634 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fbc062c90 0x7f1fbc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 1 -- 172.21.15.162:0/1849087634 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(287..287 src has 52..287) v4 ==== 8436+0+0 (secure 0 0 0) 0x7f1fc007d280 con 0x7f1fd00fdc00 2024-02-13T00:44:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 10 client.26988.objecter ms_dispatch 0x7f1fd0001550 osd_map(287..287 src has 52..287) v4 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 3 client.26988.objecter handle_osd_map got epochs [287,287] > 0 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 3 client.26988.objecter handle_osd_map decoding full epoch 287 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fd53f9700 1 --2- 172.21.15.162:0/1849087634 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fbc062c90 0x7f1fbc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 10 client.26988.objecter checking command 1 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 1 --2- 172.21.15.162:0/1849087634 >> v2:172.21.15.162:6809/3689719376 conn(0x7f1fbc070820 0x7f1fbc072cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 20 client.26988.objecter _get_session s=0x7f1fbc070170 osd=2 3 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 20 client.26988.objecter put_session s=0x7f1fbc070170 osd=2 3 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 15 client.26988.objecter _session_command_op_remove -1 1 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 20 client.26988.objecter _get_session s=0x7f1fbc070170 osd=2 3 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 20 client.26988.objecter get_session s=0x7f1fbc070170 osd=2 3 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 15 client.26988.objecter _session_command_op_assign 2 1 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 20 client.26988.objecter put_session s=0x7f1fbc070170 osd=2 4 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 10 client.26988.objecter _send_command 1 2024-02-13T00:44:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 1 -- 172.21.15.162:0/1849087634 --> v2:172.21.15.162:6809/3689719376 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1fbc073300 con 0x7f1fbc070820 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 20 client.26988.objecter dump_active .. 0 homeless 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 1 -- 172.21.15.162:0/1849087634 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=287) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fc007d5f0 con 0x7f1fd00fdc00 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fd5bfa700 1 --2- 172.21.15.162:0/1849087634 >> v2:172.21.15.162:6809/3689719376 conn(0x7f1fbc070820 0x7f1fbc072cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fc67fc700 10 client.26988.objecter ms_handle_connect 0x7f1fbc062c90 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.090+0000 7f1fd53f9700 1 --2- 172.21.15.162:0/1849087634 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fbc062c90 0x7f1fbc065150 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 rx=0x7f1fcc009700 tx=0x7f1fcc006c30).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.091+0000 7f1fc67fc700 10 client.26988.objecter ms_handle_connect 0x7f1fbc070820 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.091+0000 7f1fd5bfa700 1 --2- 172.21.15.162:0/1849087634 >> v2:172.21.15.162:6809/3689719376 conn(0x7f1fbc070820 0x7f1fbc072cc0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.091+0000 7f1fc67fc700 1 -- 172.21.15.162:0/1849087634 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1fbc073300 con 0x7f1fbc070820 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.091+0000 7f1fc67fc700 10 client.26988.objecter ms_dispatch 0x7f1fd0001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.091+0000 7f1fc67fc700 10 client.26988.objecter _finish_command 1 = system:0 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.091+0000 7f1fc67fc700 20 client.26988.objecter put_session s=0x7f1fbc070170 osd=2 4 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.091+0000 7f1fc67fc700 15 client.26988.objecter _session_command_op_remove 2 1 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fd765d700 10 client.26988.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fd765d700 15 client.26988.objecter _session_command_op_assign -1 2 2024-02-13T00:44:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fd765d700 20 client.26988.objecter _get_session s=0x7f1fbc070170 osd=2 3 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fd765d700 20 client.26988.objecter put_session s=0x7f1fbc070170 osd=2 3 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fd765d700 20 client.26988.objecter _get_session s=0x7f1fbc070170 osd=2 3 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fd765d700 15 client.26988.objecter _session_command_op_remove -1 2 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fd765d700 20 client.26988.objecter get_session s=0x7f1fbc070170 osd=2 3 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fd765d700 15 client.26988.objecter _session_command_op_assign 2 2 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fd765d700 20 client.26988.objecter put_session s=0x7f1fbc070170 osd=2 4 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fd765d700 10 client.26988.objecter _send_command 2 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fd765d700 1 -- 172.21.15.162:0/1849087634 --> v2:172.21.15.162:6809/3689719376 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7f1fd00475e0 con 0x7f1fbc070820 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fc67fc700 1 -- 172.21.15.162:0/1849087634 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1fd00475e0 con 0x7f1fbc070820 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fc67fc700 10 client.26988.objecter ms_dispatch 0x7f1fd0001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fc67fc700 10 client.26988.objecter _finish_command 2 = system:0 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fc67fc700 20 client.26988.objecter put_session s=0x7f1fbc070170 osd=2 4 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.108+0000 7f1fc67fc700 15 client.26988.objecter _session_command_op_remove 2 2 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 10 librados: watch_flush enter 2024-02-13T00:44:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 10 librados: watch_flush exit 2024-02-13T00:44:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 10 client.26988.objecter close_session for osd.2 2024-02-13T00:44:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 1 -- 172.21.15.162:0/1849087634 >> v2:172.21.15.162:6809/3689719376 conn(0x7f1fbc070820 msgr2=0x7f1fbc072cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 1 --2- 172.21.15.162:0/1849087634 >> v2:172.21.15.162:6809/3689719376 conn(0x7f1fbc070820 0x7f1fbc072cc0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 20 client.26988.objecter put_session s=0x7f1fbc070170 osd=2 1 2024-02-13T00:44:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 20 client.26988.objecter shutdown clearing up homeless session... 2024-02-13T00:44:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 10 client.26988.objecter successfully canceled tick 2024-02-13T00:44:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 1 -- 172.21.15.162:0/1849087634 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fbc062c90 msgr2=0x7f1fbc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 1 --2- 172.21.15.162:0/1849087634 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fbc062c90 0x7f1fbc065150 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 rx=0x7f1fcc009700 tx=0x7f1fcc006c30).stop 2024-02-13T00:44:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 1 -- 172.21.15.162:0/1849087634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd00fdc00 msgr2=0x7f1fd010e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 1 --2- 172.21.15.162:0/1849087634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd00fdc00 0x7f1fd010e470 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f1fc00048f0 tx=0x7f1fc0009cb0).stop 2024-02-13T00:44:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd5bfa700 1 -- 172.21.15.162:0/1849087634 reap_dead start 2024-02-13T00:44:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 1 -- 172.21.15.162:0/1849087634 shutdown_connections 2024-02-13T00:44:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 1 -- 172.21.15.162:0/1849087634 >> 172.21.15.162:0/1849087634 conn(0x7f1fd00f3850 msgr2=0x7f1fd00f51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 1 -- 172.21.15.162:0/1849087634 shutdown_connections 2024-02-13T00:44:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 1 -- 172.21.15.162:0/1849087634 wait complete. 2024-02-13T00:44:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:03.109+0000 7f1fd765d700 1 librados: shutdown 2024-02-13T00:44:03.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:02.236759+0000 mon.b (mon.1) 423 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:03.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:02.372982+0000 mgr.y (mgr.14544) 278 : cluster [DBG] pgmap v400: 302 pgs: 32 unknown, 2 active+clean+snaptrim, 268 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-13T00:44:03.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:02.379268+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-83977-40"}]': finished 2024-02-13T00:44:03.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:02.379351+0000 mon.a (mon.0) 1908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-83985-50","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:03.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:02.379420+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e287: 8 total, 8 up, 6 in 2024-02-13T00:44:03.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:02.413546+0000 mon.c (mon.2) 279 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:03.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:02.639143+0000 mon.a (mon.0) 1910 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:03.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:02.639664+0000 mon.c (mon.2) 280 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:03.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:02.639938+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:03.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:02.640322+0000 mon.c (mon.2) 281 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-83977-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:03.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:02.640538+0000 mon.a (mon.0) 1912 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-83977-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:04.354 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:04.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:04.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:04.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:03.237943+0000 mon.b (mon.1) 424 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:04.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:03.378966+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-83977-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:04.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:03.379057+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e288: 8 total, 8 up, 6 in 2024-02-13T00:44:04.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:03.381601+0000 mon.c (mon.2) 282 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-83977-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:04.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:03.388535+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-83977-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:04.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:03.392105+0000 mon.b (mon.1) 425 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:04.372 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:03.393390+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:04.373 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:03.413375+0000 mon.c (mon.2) 283 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:04.373 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:03.413687+0000 mon.a (mon.0) 1917 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:04.373 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:03.414349+0000 mon.c (mon.2) 284 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:04.373 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:03.414561+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:04.373 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:03.414976+0000 mon.c (mon.2) 285 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:04.373 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:03.415220+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:04.373 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:03.237943+0000 mon.b (mon.1) 424 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:04.373 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:03.378966+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-83977-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:04.373 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:03.379057+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e288: 8 total, 8 up, 6 in 2024-02-13T00:44:04.373 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:03.381601+0000 mon.c (mon.2) 282 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-83977-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:04.373 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:03.388535+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-83977-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:04.374 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:03.392105+0000 mon.b (mon.1) 425 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:04.374 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:03.393390+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:04.374 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:03.413375+0000 mon.c (mon.2) 283 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:04.374 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:03.413687+0000 mon.a (mon.0) 1917 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:04.374 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:03.414349+0000 mon.c (mon.2) 284 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:04.374 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:03.414561+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:04.374 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:03.414976+0000 mon.c (mon.2) 285 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:04.374 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:03.415220+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:04.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:04.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:03.237943+0000 mon.b (mon.1) 424 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:04.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:03.378966+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-83977-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:04.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:03.379057+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e288: 8 total, 8 up, 6 in 2024-02-13T00:44:04.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:03.381601+0000 mon.c (mon.2) 282 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-83977-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:04.486 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:03.388535+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-83977-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:04.487 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:03.392105+0000 mon.b (mon.1) 425 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:04.487 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:03.393390+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:04.487 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:03.413375+0000 mon.c (mon.2) 283 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:04.487 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:03.413687+0000 mon.a (mon.0) 1917 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:04.487 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:03.414349+0000 mon.c (mon.2) 284 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:04.487 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:03.414561+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:04.487 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:03.414976+0000 mon.c (mon.2) 285 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:04.487 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:03.415220+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:04.488 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:04.507 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:04.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:04.611 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:04.653 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:04.712 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:44:04.712 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:44:04.727 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:04.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:04.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:04.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:04.860 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:04.880 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:04.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:04.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:04.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:05.012 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:05.031 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:05.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:05.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:05.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:05.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:05.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:05.319 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:05.338 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:05.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:05.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:05.466 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:04.239204+0000 mon.b (mon.1) 426 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:05.467 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:04.333722+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:05.467 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:04.373489+0000 mgr.y (mgr.14544) 279 : cluster [DBG] pgmap v403: 270 pgs: 32 unknown, 1 active+clean+snaptrim, 237 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:05.467 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:04.632680+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:05.467 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:04.632851+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:05.467 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:04.632955+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e289: 8 total, 8 up, 6 in 2024-02-13T00:44:05.467 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:04.635734+0000 mon.b (mon.1) 427 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:05.467 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:04.636481+0000 mon.c (mon.2) 286 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-83985-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:05.467 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:04.636616+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:05.467 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:04.637300+0000 mon.a (mon.0) 1925 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-83985-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:05.468 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:05.487 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:05.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:05.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:05.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:04.239204+0000 mon.b (mon.1) 426 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:05.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:04.333722+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:05.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:04.373489+0000 mgr.y (mgr.14544) 279 : cluster [DBG] pgmap v403: 270 pgs: 32 unknown, 1 active+clean+snaptrim, 237 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:05.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:04.632680+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:05.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:04.632851+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:05.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:04.632955+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e289: 8 total, 8 up, 6 in 2024-02-13T00:44:05.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:04.635734+0000 mon.b (mon.1) 427 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:05.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:04.636481+0000 mon.c (mon.2) 286 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-83985-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:05.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:04.636616+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:05.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:04.637300+0000 mon.a (mon.0) 1925 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-83985-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:05.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:04.239204+0000 mon.b (mon.1) 426 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:05.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:04.333722+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:05.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:04.373489+0000 mgr.y (mgr.14544) 279 : cluster [DBG] pgmap v403: 270 pgs: 32 unknown, 1 active+clean+snaptrim, 237 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:05.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:04.632680+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:05.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:04.632851+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:05.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:04.632955+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e289: 8 total, 8 up, 6 in 2024-02-13T00:44:05.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:04.635734+0000 mon.b (mon.1) 427 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:05.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:04.636481+0000 mon.c (mon.2) 286 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-83985-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:05.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:04.636616+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:05.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:04.637300+0000 mon.a (mon.0) 1925 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-83985-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:05.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:05.626 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:05.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:44:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:44:05.824 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:05.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:05.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.4 reset_purged_snaps_last 2024-02-13T00:44:05.927 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:05.954 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:05.973 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:05.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:05.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:06.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:06.102 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:06.120 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:06.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:06.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:06.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:44:06.164 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:05.240159+0000 mon.b (mon.1) 428 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:06.164 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:05.359145+0000 mon.a (mon.0) 1926 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-83977-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-83977-41"}]': finished 2024-02-13T00:44:06.164 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:05.359243+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:06.164 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:05.360254+0000 mon.b (mon.1) 429 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-35"}]: dispatch 2024-02-13T00:44:06.164 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:05.497522+0000 mgr.y (mgr.14544) 280 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:06.164 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:05.598912+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e290: 8 total, 8 up, 6 in 2024-02-13T00:44:06.164 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:05.599979+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-35"}]: dispatch 2024-02-13T00:44:06.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:05.240159+0000 mon.b (mon.1) 428 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:06.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:05.359145+0000 mon.a (mon.0) 1926 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-83977-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-83977-41"}]': finished 2024-02-13T00:44:06.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:05.359243+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:06.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:05.360254+0000 mon.b (mon.1) 429 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-35"}]: dispatch 2024-02-13T00:44:06.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:05.497522+0000 mgr.y (mgr.14544) 280 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:06.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:05.598912+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e290: 8 total, 8 up, 6 in 2024-02-13T00:44:06.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:05.599979+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-35"}]: dispatch 2024-02-13T00:44:06.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:06.247 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:05.240159+0000 mon.b (mon.1) 428 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:06.248 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:05.359145+0000 mon.a (mon.0) 1926 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-83977-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-83977-41"}]': finished 2024-02-13T00:44:06.248 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:05.359243+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:06.248 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:05.360254+0000 mon.b (mon.1) 429 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-35"}]: dispatch 2024-02-13T00:44:06.248 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:05.497522+0000 mgr.y (mgr.14544) 280 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:06.248 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:05.598912+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e290: 8 total, 8 up, 6 in 2024-02-13T00:44:06.248 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:05.599979+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-35"}]: dispatch 2024-02-13T00:44:06.249 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:06.268 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:06.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:06.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:06.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:06.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-13T00:44:06.398 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:06.417 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:06.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:06.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:06.444 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:06.488 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:06.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:44:06.546 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:06.564 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:44:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:06.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:44:06.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:44:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:44:06] "GET /metrics HTTP/1.1" 200 188859 "" "Prometheus/2.33.4" 2024-02-13T00:44:06.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:06.694 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:06.747 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:06.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:06.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:06.878 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:07.001 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:07.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:07.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:07.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:07.135 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:07.154 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:07.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:07.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:44:07.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:07.285 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:07.304 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:07.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:07.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:06.241137+0000 mon.b (mon.1) 430 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:07.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:06.362673+0000 mon.a (mon.0) 1930 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-83985-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]': finished 2024-02-13T00:44:07.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:06.362779+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-35"}]': finished 2024-02-13T00:44:07.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:06.362940+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e291: 8 total, 8 up, 6 in 2024-02-13T00:44:07.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:06.367999+0000 mon.b (mon.1) 431 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-35", "mode": "writeback"}]: dispatch 2024-02-13T00:44:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:06.368506+0000 mon.a (mon.0) 1933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-35", "mode": "writeback"}]: dispatch 2024-02-13T00:44:07.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:06.381111+0000 mgr.y (mgr.14544) 281 : cluster [DBG] pgmap v407: 286 pgs: 16 unknown, 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:07.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:06.241137+0000 mon.b (mon.1) 430 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:07.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:06.362673+0000 mon.a (mon.0) 1930 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-83985-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]': finished 2024-02-13T00:44:07.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:06.362779+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-35"}]': finished 2024-02-13T00:44:07.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:06.362940+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e291: 8 total, 8 up, 6 in 2024-02-13T00:44:07.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:06.367999+0000 mon.b (mon.1) 431 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-35", "mode": "writeback"}]: dispatch 2024-02-13T00:44:07.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:06.368506+0000 mon.a (mon.0) 1933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-35", "mode": "writeback"}]: dispatch 2024-02-13T00:44:07.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:06.381111+0000 mgr.y (mgr.14544) 281 : cluster [DBG] pgmap v407: 286 pgs: 16 unknown, 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:07.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:07.435 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:07.510 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:07.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:07.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:06.241137+0000 mon.b (mon.1) 430 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:07.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:06.362673+0000 mon.a (mon.0) 1930 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-83985-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]': finished 2024-02-13T00:44:07.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:06.362779+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-35"}]': finished 2024-02-13T00:44:07.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:06.362940+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e291: 8 total, 8 up, 6 in 2024-02-13T00:44:07.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:06.367999+0000 mon.b (mon.1) 431 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-35", "mode": "writeback"}]: dispatch 2024-02-13T00:44:07.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:06.368506+0000 mon.a (mon.0) 1933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-35", "mode": "writeback"}]: dispatch 2024-02-13T00:44:07.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:06.381111+0000 mgr.y (mgr.14544) 281 : cluster [DBG] pgmap v407: 286 pgs: 16 unknown, 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:07.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:07.641 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:08.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:07.242075+0000 mon.b (mon.1) 432 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:07.358903+0000 mon.a (mon.0) 1934 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:44:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:07.365504+0000 mon.a (mon.0) 1935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-35", "mode": "writeback"}]': finished 2024-02-13T00:44:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:07.365615+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e292: 8 total, 8 up, 6 in 2024-02-13T00:44:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:07.377723+0000 mon.c (mon.2) 287 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:07.378249+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:07.402277+0000 mon.b (mon.1) 433 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:07.402773+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:07.242075+0000 mon.b (mon.1) 432 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:07.358903+0000 mon.a (mon.0) 1934 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:44:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:07.365504+0000 mon.a (mon.0) 1935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-35", "mode": "writeback"}]': finished 2024-02-13T00:44:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:07.365615+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e292: 8 total, 8 up, 6 in 2024-02-13T00:44:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:07.377723+0000 mon.c (mon.2) 287 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:07.378249+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:07.402277+0000 mon.b (mon.1) 433 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:07.402773+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:08.390 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:08.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:08.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:08.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:07.242075+0000 mon.b (mon.1) 432 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:08.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:07.358903+0000 mon.a (mon.0) 1934 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:44:08.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:07.365504+0000 mon.a (mon.0) 1935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-35", "mode": "writeback"}]': finished 2024-02-13T00:44:08.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:07.365615+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e292: 8 total, 8 up, 6 in 2024-02-13T00:44:08.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:07.377723+0000 mon.c (mon.2) 287 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:08.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:07.378249+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:08.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:07.402277+0000 mon.b (mon.1) 433 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:08.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:07.402773+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:08.525 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:08.545 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:08.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:08.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:08.677 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:08.794 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:44:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:44:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:44:08.875 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:08.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:08.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:09.007 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:09.204 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:09.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:09.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:09.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:08.243106+0000 mon.b (mon.1) 434 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:09.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:08.367814+0000 mon.a (mon.0) 1939 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]': finished 2024-02-13T00:44:09.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:08.367910+0000 mon.a (mon.0) 1940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:44:09.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:08.367958+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e293: 8 total, 8 up, 6 in 2024-02-13T00:44:09.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:08.370624+0000 mon.c (mon.2) 288 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:09.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:08.370869+0000 mon.b (mon.1) 435 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35"}]: dispatch 2024-02-13T00:44:09.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:08.370932+0000 mon.c (mon.2) 289 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:09.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:08.371265+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:09.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:08.371544+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:09.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:08.371809+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35"}]: dispatch 2024-02-13T00:44:09.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:08.381570+0000 mgr.y (mgr.14544) 282 : cluster [DBG] pgmap v410: 270 pgs: 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:09.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:08.243106+0000 mon.b (mon.1) 434 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:09.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:08.367814+0000 mon.a (mon.0) 1939 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]': finished 2024-02-13T00:44:09.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:08.367910+0000 mon.a (mon.0) 1940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:44:09.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:08.367958+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e293: 8 total, 8 up, 6 in 2024-02-13T00:44:09.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:08.370624+0000 mon.c (mon.2) 288 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:08.370869+0000 mon.b (mon.1) 435 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35"}]: dispatch 2024-02-13T00:44:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:08.370932+0000 mon.c (mon.2) 289 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:08.371265+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:08.371544+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:08.371809+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35"}]: dispatch 2024-02-13T00:44:09.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:08.381570+0000 mgr.y (mgr.14544) 282 : cluster [DBG] pgmap v410: 270 pgs: 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:09.337 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:09.365 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:09.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:09.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:09.384 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAiote 2024-02-13T00:44:09.384 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6985 ms) 2024-02-13T00:44:09.385 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-02-13T00:44:09.385 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7026 ms) 2024-02-13T00:44:09.385 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-02-13T00:44:09.385 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7015 ms) 2024-02-13T00:44:09.385 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-02-13T00:44:09.385 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6979 ms) 2024-02-13T00:44:09.385 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-02-13T00:44:09.385 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7031 ms) 2024-02-13T00:44:09.385 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-02-13T00:44:09.385 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6977 ms) 2024-02-13T00:44:09.385 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-02-13T00:44:09.385 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7021 ms) 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6971 ms) 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7027 ms) 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7229 ms) 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6993 ms) 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7004 ms) 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6759 ms) 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7030 ms) 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-02-13T00:44:09.386 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6987 ms) 2024-02-13T00:44:09.387 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112010 ms total) 2024-02-13T00:44:09.387 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: 2024-02-13T00:44:09.387 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [----------] Global test environment tear-down 2024-02-13T00:44:09.387 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [==========] 41 tests from 2 test suites ran. (194678 ms total) 2024-02-13T00:44:09.387 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ PASSED ] 41 tests. 2024-02-13T00:44:09.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:09.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.473+0000 7f0451d2c700 1 -- 172.21.15.162:0/2057412284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f044c0f84e0 msgr2=0x7f044c104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.473+0000 7f0451d2c700 1 --2- 172.21.15.162:0/2057412284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f044c0f84e0 0x7f044c104ae0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 rx=0x7f0444009c30 tx=0x7f0444004c50).stop 2024-02-13T00:44:09.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.473+0000 7f0451d2c700 1 -- 172.21.15.162:0/2057412284 shutdown_connections 2024-02-13T00:44:09.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.473+0000 7f0451d2c700 1 --2- 172.21.15.162:0/2057412284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f044c0f84e0 0x7f044c104ae0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.473+0000 7f0451d2c700 1 --2- 172.21.15.162:0/2057412284 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f044c0fdc70 0x7f044c0fe050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.473+0000 7f0451d2c700 1 --2- 172.21.15.162:0/2057412284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f044c0fe5d0 0x7f044c0f7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.473+0000 7f0451d2c700 1 -- 172.21.15.162:0/2057412284 >> 172.21.15.162:0/2057412284 conn(0x7f044c0f38a0 msgr2=0x7f044c0f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:09.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.474+0000 7f0451d2c700 1 -- 172.21.15.162:0/2057412284 shutdown_connections 2024-02-13T00:44:09.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.474+0000 7f0451d2c700 1 -- 172.21.15.162:0/2057412284 wait complete. 2024-02-13T00:44:09.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.474+0000 7f0451d2c700 1 librados: starting msgr at 2024-02-13T00:44:09.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.474+0000 7f0451d2c700 1 librados: starting objecter 2024-02-13T00:44:09.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.474+0000 7f0451d2c700 1 Processor -- start 2024-02-13T00:44:09.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.474+0000 7f0451d2c700 1 -- start start 2024-02-13T00:44:09.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.474+0000 7f0451d2c700 1 librados: setting wanted keys 2024-02-13T00:44:09.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.474+0000 7f0451d2c700 1 librados: calling monclient init 2024-02-13T00:44:09.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0451d2c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f044c0f84e0 0x7f044c10dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:09.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0451d2c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f044c0fdc70 0x7f044c10e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0451d2c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f044c0fe5d0 0x7f044c108190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0451d2c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c108680 con 0x7f044c0fdc70 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0451d2c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c1087f0 con 0x7f044c0fe5d0 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0451d2c700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044c1089b0 con 0x7f044c0f84e0 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044affd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f044c0fdc70 0x7f044c10e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044b7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f044c0f84e0 0x7f044c10dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044b7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f044c0f84e0 0x7f044c10dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:38018/0 (socket says 172.21.15.162:38018) 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044affd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f044c0fdc70 0x7f044c10e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45834/0 (socket says 172.21.15.162:45834) 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044b7fe700 1 -- 172.21.15.162:0/871361626 learned_addr learned my addr 172.21.15.162:0/871361626 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044bfff700 1 --2- 172.21.15.162:0/871361626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f044c0fe5d0 0x7f044c108190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044b7fe700 1 -- 172.21.15.162:0/871361626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f044c0fe5d0 msgr2=0x7f044c108190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044b7fe700 1 --2- 172.21.15.162:0/871361626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f044c0fe5d0 0x7f044c108190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044b7fe700 1 -- 172.21.15.162:0/871361626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f044c0fdc70 msgr2=0x7f044c10e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044b7fe700 1 --2- 172.21.15.162:0/871361626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f044c0fdc70 0x7f044c10e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044b7fe700 1 -- 172.21.15.162:0/871361626 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c005040 con 0x7f044c0f84e0 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044affd700 1 --2- 172.21.15.162:0/871361626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f044c0fdc70 0x7f044c10e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044bfff700 1 --2- 172.21.15.162:0/871361626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f044c0fe5d0 0x7f044c108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f044b7fe700 1 --2- 172.21.15.162:0/871361626 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f044c0f84e0 0x7f044c10dff0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 rx=0x7f043c009400 tx=0x7f043c009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0448ff9700 10 client.?.objecter ms_handle_connect 0x7f044c0f84e0 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0448ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c007a40 con 0x7f044c0f84e0 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04440097c0 con 0x7f044c0f84e0 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f043c00f040 con 0x7f044c0f84e0 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c014aa0 con 0x7f044c0f84e0 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0451d2c700 10 client.37109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044c108ff0 con 0x7f044c0f84e0 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.475+0000 7f0451d2c700 1 librados: init done 2024-02-13T00:44:09.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.476+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f044c060770 con 0x7f044c0f84e0 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f043c00e630 con 0x7f044c0f84e0 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0448ff9700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0434062c90 0x7f0434065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(294..294 src has 52..294) v4 ==== 8506+0+0 (secure 0 0 0) 0x7f043c07d4d0 con 0x7f044c0f84e0 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 osd_map(294..294 src has 52..294) v4 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0448ff9700 3 client.37109.objecter handle_osd_map got epochs [294,294] > 0 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0448ff9700 3 client.37109.objecter handle_osd_map decoding full epoch 294 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f044affd700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0434062c90 0x7f0434065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0448ff9700 20 client.37109.objecter dump_active .. 0 homeless 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v294) v1 ==== 54+0+16 (secure 0 0 0) 0x7f043c07d970 con 0x7f044c0f84e0 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f044affd700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0434062c90 0x7f0434065150 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 rx=0x7f044c109930 tx=0x7f044000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0448ff9700 10 client.37109.objecter ms_handle_connect 0x7f0434062c90 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0451d2c700 10 client.37109.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 1 2024-02-13T00:44:09.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6801/2103743338 conn(0x7f044c0fef40 0x7f044c0ff320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c0475e0 osd=0 3 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c0475e0 osd=0 3 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c0475e0 osd=0 3 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 1 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c0475e0 osd=0 3 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 0 1 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c0475e0 osd=0 4 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0451d2c700 10 client.37109.objecter _send_command 1 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.477+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f044c1071d0 con 0x7f044c0fef40 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.478+0000 7f044bfff700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6801/2103743338 conn(0x7f044c0fef40 0x7f044c0ff320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.478+0000 7f0448ff9700 10 client.37109.objecter ms_handle_connect 0x7f044c0fef40 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.478+0000 7f044bfff700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6801/2103743338 conn(0x7f044c0fef40 0x7f044c0ff320 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.478+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f044c1071d0 con 0x7f044c0fef40 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.478+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:44:09.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.478+0000 7f0448ff9700 10 client.37109.objecter _finish_command 1 = system:0 2024-02-13T00:44:09.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.478+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c0475e0 osd=0 4 2024-02-13T00:44:09.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.478+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 0 1 2024-02-13T00:44:09.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.493+0000 7f0451d2c700 10 client.37109.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:09.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.493+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 2 2024-02-13T00:44:09.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.493+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c0475e0 osd=0 3 2024-02-13T00:44:09.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.493+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c0475e0 osd=0 3 2024-02-13T00:44:09.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.493+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c0475e0 osd=0 3 2024-02-13T00:44:09.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.493+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 2 2024-02-13T00:44:09.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.493+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c0475e0 osd=0 3 2024-02-13T00:44:09.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.493+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 0 2 2024-02-13T00:44:09.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.493+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c0475e0 osd=0 4 2024-02-13T00:44:09.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.493+0000 7f0451d2c700 10 client.37109.objecter _send_command 2 2024-02-13T00:44:09.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.493+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f044c027410 con 0x7f044c0fef40 2024-02-13T00:44:09.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:08.243106+0000 mon.b (mon.1) 434 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:09.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:08.367814+0000 mon.a (mon.0) 1939 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-83977-41"}]': finished 2024-02-13T00:44:09.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:08.367910+0000 mon.a (mon.0) 1940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:44:09.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:08.367958+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e293: 8 total, 8 up, 6 in 2024-02-13T00:44:09.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:08.370624+0000 mon.c (mon.2) 288 : audit [INF] from='client.? 172.21.15.162:0/2097984534' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:09.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:08.370869+0000 mon.b (mon.1) 435 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35"}]: dispatch 2024-02-13T00:44:09.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:08.370932+0000 mon.c (mon.2) 289 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:09.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:08.371265+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]: dispatch 2024-02-13T00:44:09.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:08.371544+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:09.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:08.371809+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35"}]: dispatch 2024-02-13T00:44:09.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:08.381570+0000 mgr.y (mgr.14544) 282 : cluster [DBG] pgmap v410: 270 pgs: 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:09.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f044c027410 con 0x7f044c0fef40 2024-02-13T00:44:09.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:09.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0448ff9700 10 client.37109.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c0475e0 osd=0 4 2024-02-13T00:44:09.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 0 2 2024-02-13T00:44:09.500 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:44:09.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0451d2c700 10 client.37109.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:09.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 3 2024-02-13T00:44:09.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6805/150599780 conn(0x7f044c059990 0x7f044c059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:09.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c028760 osd=1 3 2024-02-13T00:44:09.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c028760 osd=1 3 2024-02-13T00:44:09.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c028760 osd=1 3 2024-02-13T00:44:09.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 3 2024-02-13T00:44:09.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c028760 osd=1 3 2024-02-13T00:44:09.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 1 3 2024-02-13T00:44:09.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c028760 osd=1 4 2024-02-13T00:44:09.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0451d2c700 10 client.37109.objecter _send_command 3 2024-02-13T00:44:09.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f044c06cfc0 con 0x7f044c059990 2024-02-13T00:44:09.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f044b7fe700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6805/150599780 conn(0x7f044c059990 0x7f044c059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:09.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f0448ff9700 10 client.37109.objecter ms_handle_connect 0x7f044c059990 2024-02-13T00:44:09.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.495+0000 7f044b7fe700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6805/150599780 conn(0x7f044c059990 0x7f044c059d70 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:09.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.496+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f044c06cfc0 con 0x7f044c059990 2024-02-13T00:44:09.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.496+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:44:09.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.496+0000 7f0448ff9700 10 client.37109.objecter _finish_command 3 = system:0 2024-02-13T00:44:09.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.496+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c028760 osd=1 4 2024-02-13T00:44:09.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.496+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 1 3 2024-02-13T00:44:09.503 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:09.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.510+0000 7f0451d2c700 10 client.37109.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:09.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.510+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 4 2024-02-13T00:44:09.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.510+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c028760 osd=1 3 2024-02-13T00:44:09.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.510+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c028760 osd=1 3 2024-02-13T00:44:09.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.510+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c028760 osd=1 3 2024-02-13T00:44:09.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.510+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 4 2024-02-13T00:44:09.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.510+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c028760 osd=1 3 2024-02-13T00:44:09.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.510+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 1 4 2024-02-13T00:44:09.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.510+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c028760 osd=1 4 2024-02-13T00:44:09.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.510+0000 7f0451d2c700 10 client.37109.objecter _send_command 4 2024-02-13T00:44:09.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.510+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f044c059400 con 0x7f044c059990 2024-02-13T00:44:09.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f044c059400 con 0x7f044c059990 2024-02-13T00:44:09.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:09.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0448ff9700 10 client.37109.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c028760 osd=1 4 2024-02-13T00:44:09.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 1 4 2024-02-13T00:44:09.517 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0451d2c700 10 client.37109.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 5 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6809/3689719376 conn(0x7f044c0282c0 0x7f044c11e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c059810 osd=2 3 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c059810 osd=2 3 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c059810 osd=2 3 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 5 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c059810 osd=2 3 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 2 5 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c059810 osd=2 4 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0451d2c700 10 client.37109.objecter _send_command 5 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f044c1210b0 con 0x7f044c0282c0 2024-02-13T00:44:09.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f044affd700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6809/3689719376 conn(0x7f044c0282c0 0x7f044c11e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:09.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f044affd700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6809/3689719376 conn(0x7f044c0282c0 0x7f044c11e460 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:09.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.512+0000 7f0448ff9700 10 client.37109.objecter ms_handle_connect 0x7f044c0282c0 2024-02-13T00:44:09.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.513+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f044c1210b0 con 0x7f044c0282c0 2024-02-13T00:44:09.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.513+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:44:09.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.513+0000 7f0448ff9700 10 client.37109.objecter _finish_command 5 = system:0 2024-02-13T00:44:09.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.513+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c059810 osd=2 4 2024-02-13T00:44:09.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.513+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 2 5 2024-02-13T00:44:09.520 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:09.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:09.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.528+0000 7f0451d2c700 10 client.37109.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:09.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.528+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 6 2024-02-13T00:44:09.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.528+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c059810 osd=2 3 2024-02-13T00:44:09.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.528+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c059810 osd=2 3 2024-02-13T00:44:09.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.528+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c059810 osd=2 3 2024-02-13T00:44:09.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.528+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 6 2024-02-13T00:44:09.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.528+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c059810 osd=2 3 2024-02-13T00:44:09.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.528+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 2 6 2024-02-13T00:44:09.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.528+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c059810 osd=2 4 2024-02-13T00:44:09.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.528+0000 7f0451d2c700 10 client.37109.objecter _send_command 6 2024-02-13T00:44:09.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.528+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f044c027820 con 0x7f044c0282c0 2024-02-13T00:44:09.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f044c027820 con 0x7f044c0282c0 2024-02-13T00:44:09.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:09.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0448ff9700 10 client.37109.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c059810 osd=2 4 2024-02-13T00:44:09.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 2 6 2024-02-13T00:44:09.534 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:44:09.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0451d2c700 10 client.37109.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:09.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 7 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6813/604971011 conn(0x7f044c11fad0 0x7f044c11fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c11baa0 osd=3 3 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c11baa0 osd=3 3 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c11baa0 osd=3 3 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 7 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c11baa0 osd=3 3 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 3 7 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c11baa0 osd=3 4 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0451d2c700 10 client.37109.objecter _send_command 7 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f044c11c170 con 0x7f044c11fad0 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.529+0000 7f044bfff700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6813/604971011 conn(0x7f044c11fad0 0x7f044c11fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.530+0000 7f0448ff9700 10 client.37109.objecter ms_handle_connect 0x7f044c11fad0 2024-02-13T00:44:09.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.530+0000 7f044bfff700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6813/604971011 conn(0x7f044c11fad0 0x7f044c11fef0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:09.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.530+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f044c11c170 con 0x7f044c11fad0 2024-02-13T00:44:09.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.530+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:44:09.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.530+0000 7f0448ff9700 10 client.37109.objecter _finish_command 7 = system:0 2024-02-13T00:44:09.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.530+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c11baa0 osd=3 4 2024-02-13T00:44:09.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.530+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 3 7 2024-02-13T00:44:09.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.547+0000 7f0451d2c700 10 client.37109.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:09.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.547+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 8 2024-02-13T00:44:09.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.547+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c11baa0 osd=3 3 2024-02-13T00:44:09.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.547+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c11baa0 osd=3 3 2024-02-13T00:44:09.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.547+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c11baa0 osd=3 3 2024-02-13T00:44:09.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.547+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 8 2024-02-13T00:44:09.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.547+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c11baa0 osd=3 3 2024-02-13T00:44:09.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.547+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 3 8 2024-02-13T00:44:09.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.547+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c11baa0 osd=3 4 2024-02-13T00:44:09.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.547+0000 7f0451d2c700 10 client.37109.objecter _send_command 8 2024-02-13T00:44:09.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.547+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f044c120bb0 con 0x7f044c11fad0 2024-02-13T00:44:09.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f044c120bb0 con 0x7f044c11fad0 2024-02-13T00:44:09.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:09.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0448ff9700 10 client.37109.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c11baa0 osd=3 4 2024-02-13T00:44:09.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 3 8 2024-02-13T00:44:09.554 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:44:09.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0451d2c700 10 client.37109.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:09.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 9 2024-02-13T00:44:09.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6800/3344263388 conn(0x7f044c123230 0x7f044c125620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:09.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c120bb0 osd=4 3 2024-02-13T00:44:09.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c120bb0 osd=4 3 2024-02-13T00:44:09.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c120bb0 osd=4 3 2024-02-13T00:44:09.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 9 2024-02-13T00:44:09.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c120bb0 osd=4 3 2024-02-13T00:44:09.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 4 9 2024-02-13T00:44:09.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c120bb0 osd=4 4 2024-02-13T00:44:09.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0451d2c700 10 client.37109.objecter _send_command 9 2024-02-13T00:44:09.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f044c120d90 con 0x7f044c123230 2024-02-13T00:44:09.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.550+0000 7f044b7fe700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6800/3344263388 conn(0x7f044c123230 0x7f044c125620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:09.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.551+0000 7f044b7fe700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6800/3344263388 conn(0x7f044c123230 0x7f044c125620 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:09.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.551+0000 7f0448ff9700 10 client.37109.objecter ms_handle_connect 0x7f044c123230 2024-02-13T00:44:09.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.551+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f044c120d90 con 0x7f044c123230 2024-02-13T00:44:09.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.551+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:44:09.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.551+0000 7f0448ff9700 10 client.37109.objecter _finish_command 9 = system:0 2024-02-13T00:44:09.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.551+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c120bb0 osd=4 4 2024-02-13T00:44:09.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.551+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 4 9 2024-02-13T00:44:09.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.568+0000 7f0451d2c700 10 client.37109.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:09.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.568+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 10 2024-02-13T00:44:09.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.568+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c120bb0 osd=4 3 2024-02-13T00:44:09.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.568+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c120bb0 osd=4 3 2024-02-13T00:44:09.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.568+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c120bb0 osd=4 3 2024-02-13T00:44:09.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.568+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 10 2024-02-13T00:44:09.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.568+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c120bb0 osd=4 3 2024-02-13T00:44:09.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.568+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 4 10 2024-02-13T00:44:09.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.568+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c120bb0 osd=4 4 2024-02-13T00:44:09.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.568+0000 7f0451d2c700 10 client.37109.objecter _send_command 10 2024-02-13T00:44:09.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.568+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f044c120d90 con 0x7f044c123230 2024-02-13T00:44:09.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f044c120d90 con 0x7f044c123230 2024-02-13T00:44:09.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:09.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0448ff9700 10 client.37109.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c120bb0 osd=4 4 2024-02-13T00:44:09.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 4 10 2024-02-13T00:44:09.575 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:44:09.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0451d2c700 10 client.37109.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:09.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 11 2024-02-13T00:44:09.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6804/2212296748 conn(0x7f044c126020 0x7f044c128410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:09.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c120d90 osd=5 3 2024-02-13T00:44:09.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c120d90 osd=5 3 2024-02-13T00:44:09.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c120d90 osd=5 3 2024-02-13T00:44:09.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 11 2024-02-13T00:44:09.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c120d90 osd=5 3 2024-02-13T00:44:09.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 5 11 2024-02-13T00:44:09.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c120d90 osd=5 4 2024-02-13T00:44:09.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0451d2c700 10 client.37109.objecter _send_command 11 2024-02-13T00:44:09.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.570+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f044c12b1e0 con 0x7f044c126020 2024-02-13T00:44:09.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.571+0000 7f044affd700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6804/2212296748 conn(0x7f044c126020 0x7f044c128410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:09.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.571+0000 7f0448ff9700 10 client.37109.objecter ms_handle_connect 0x7f044c126020 2024-02-13T00:44:09.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.571+0000 7f044affd700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6804/2212296748 conn(0x7f044c126020 0x7f044c128410 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:09.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.572+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f044c12b1e0 con 0x7f044c126020 2024-02-13T00:44:09.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.572+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:44:09.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.572+0000 7f0448ff9700 10 client.37109.objecter _finish_command 11 = system:0 2024-02-13T00:44:09.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.572+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c120d90 osd=5 4 2024-02-13T00:44:09.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.572+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 5 11 2024-02-13T00:44:09.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.586+0000 7f0451d2c700 10 client.37109.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:09.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.586+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 12 2024-02-13T00:44:09.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.586+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c120d90 osd=5 3 2024-02-13T00:44:09.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.586+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c120d90 osd=5 3 2024-02-13T00:44:09.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.586+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c120d90 osd=5 3 2024-02-13T00:44:09.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.586+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 12 2024-02-13T00:44:09.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.586+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c120d90 osd=5 3 2024-02-13T00:44:09.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.586+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 5 12 2024-02-13T00:44:09.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.586+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c120d90 osd=5 4 2024-02-13T00:44:09.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.586+0000 7f0451d2c700 10 client.37109.objecter _send_command 12 2024-02-13T00:44:09.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.586+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f044c12b1a0 con 0x7f044c126020 2024-02-13T00:44:09.592 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f044c12b1a0 con 0x7f044c126020 2024-02-13T00:44:09.592 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:09.592 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0448ff9700 10 client.37109.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.592 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c120d90 osd=5 4 2024-02-13T00:44:09.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 5 12 2024-02-13T00:44:09.593 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:44:09.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0451d2c700 10 client.37109.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:09.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 13 2024-02-13T00:44:09.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6808/1506640341 conn(0x7f044c12a4c0 0x7f044c12a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:09.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c12b1a0 osd=6 3 2024-02-13T00:44:09.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c12b1a0 osd=6 3 2024-02-13T00:44:09.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c12b1a0 osd=6 3 2024-02-13T00:44:09.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 13 2024-02-13T00:44:09.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c12b1a0 osd=6 3 2024-02-13T00:44:09.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 6 13 2024-02-13T00:44:09.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c12b1a0 osd=6 4 2024-02-13T00:44:09.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0451d2c700 10 client.37109.objecter _send_command 13 2024-02-13T00:44:09.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f044c12afb0 con 0x7f044c12a4c0 2024-02-13T00:44:09.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.588+0000 7f044bfff700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6808/1506640341 conn(0x7f044c12a4c0 0x7f044c12a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:09.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.589+0000 7f0448ff9700 10 client.37109.objecter ms_handle_connect 0x7f044c12a4c0 2024-02-13T00:44:09.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.589+0000 7f044bfff700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6808/1506640341 conn(0x7f044c12a4c0 0x7f044c12a8e0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:09.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.589+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0444027b30 con 0x7f044c12a4c0 2024-02-13T00:44:09.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.589+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:44:09.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.589+0000 7f0448ff9700 10 client.37109.objecter _finish_command 13 = system:0 2024-02-13T00:44:09.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.589+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c12b1a0 osd=6 4 2024-02-13T00:44:09.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.589+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 6 13 2024-02-13T00:44:09.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.603+0000 7f0451d2c700 10 client.37109.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:09.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.603+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 14 2024-02-13T00:44:09.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.603+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c12b1a0 osd=6 3 2024-02-13T00:44:09.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.603+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c12b1a0 osd=6 3 2024-02-13T00:44:09.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.603+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c12b1a0 osd=6 3 2024-02-13T00:44:09.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.603+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 14 2024-02-13T00:44:09.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.603+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c12b1a0 osd=6 3 2024-02-13T00:44:09.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.603+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 6 14 2024-02-13T00:44:09.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.603+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c12b1a0 osd=6 4 2024-02-13T00:44:09.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.603+0000 7f0451d2c700 10 client.37109.objecter _send_command 14 2024-02-13T00:44:09.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.603+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f044c12d3c0 con 0x7f044c12a4c0 2024-02-13T00:44:09.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f044c12d3c0 con 0x7f044c12a4c0 2024-02-13T00:44:09.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:09.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0448ff9700 10 client.37109.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c12b1a0 osd=6 4 2024-02-13T00:44:09.610 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:44:09.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 6 14 2024-02-13T00:44:09.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0451d2c700 10 client.37109.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:09.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 15 2024-02-13T00:44:09.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6812/1964192772 conn(0x7f044c12dae0 0x7f044c12df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:09.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c12d3c0 osd=7 3 2024-02-13T00:44:09.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c12d3c0 osd=7 3 2024-02-13T00:44:09.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c12d3c0 osd=7 3 2024-02-13T00:44:09.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 15 2024-02-13T00:44:09.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c12d3c0 osd=7 3 2024-02-13T00:44:09.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 7 15 2024-02-13T00:44:09.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c12d3c0 osd=7 4 2024-02-13T00:44:09.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0451d2c700 10 client.37109.objecter _send_command 15 2024-02-13T00:44:09.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.605+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f044c131830 con 0x7f044c12dae0 2024-02-13T00:44:09.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.606+0000 7f044b7fe700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6812/1964192772 conn(0x7f044c12dae0 0x7f044c12df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:09.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.606+0000 7f0448ff9700 10 client.37109.objecter ms_handle_connect 0x7f044c12dae0 2024-02-13T00:44:09.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.606+0000 7f044b7fe700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6812/1964192772 conn(0x7f044c12dae0 0x7f044c12df40 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:09.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.607+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f044c131830 con 0x7f044c12dae0 2024-02-13T00:44:09.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.607+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:44:09.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.607+0000 7f0448ff9700 10 client.37109.objecter _finish_command 15 = system:0 2024-02-13T00:44:09.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.607+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c12d3c0 osd=7 4 2024-02-13T00:44:09.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.607+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 7 15 2024-02-13T00:44:09.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:44:09.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.621+0000 7f0451d2c700 10 client.37109.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:09.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.621+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign -1 16 2024-02-13T00:44:09.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.621+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c12d3c0 osd=7 3 2024-02-13T00:44:09.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.621+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c12d3c0 osd=7 3 2024-02-13T00:44:09.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.621+0000 7f0451d2c700 20 client.37109.objecter _get_session s=0x7f044c12d3c0 osd=7 3 2024-02-13T00:44:09.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.621+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_remove -1 16 2024-02-13T00:44:09.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.621+0000 7f0451d2c700 20 client.37109.objecter get_session s=0x7f044c12d3c0 osd=7 3 2024-02-13T00:44:09.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.621+0000 7f0451d2c700 15 client.37109.objecter _session_command_op_assign 7 16 2024-02-13T00:44:09.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.621+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c12d3c0 osd=7 4 2024-02-13T00:44:09.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.621+0000 7f0451d2c700 10 client.37109.objecter _send_command 16 2024-02-13T00:44:09.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.621+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f044c1317f0 con 0x7f044c12dae0 2024-02-13T00:44:09.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0448ff9700 1 -- 172.21.15.162:0/871361626 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f044c1317f0 con 0x7f044c12dae0 2024-02-13T00:44:09.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0448ff9700 10 client.37109.objecter ms_dispatch 0x7f044c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:09.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0448ff9700 10 client.37109.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0448ff9700 20 client.37109.objecter put_session s=0x7f044c12d3c0 osd=7 4 2024-02-13T00:44:09.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0448ff9700 15 client.37109.objecter _session_command_op_remove 7 16 2024-02-13T00:44:09.628 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:44:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 10 librados: watch_flush enter 2024-02-13T00:44:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 10 librados: watch_flush exit 2024-02-13T00:44:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 10 client.37109.objecter close_session for osd.0 2024-02-13T00:44:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6801/2103743338 conn(0x7f044c0fef40 msgr2=0x7f044c0ff320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6801/2103743338 conn(0x7f044c0fef40 0x7f044c0ff320 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c0475e0 osd=0 1 2024-02-13T00:44:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 10 client.37109.objecter close_session for osd.1 2024-02-13T00:44:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6805/150599780 conn(0x7f044c059990 msgr2=0x7f044c059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6805/150599780 conn(0x7f044c059990 0x7f044c059d70 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c028760 osd=1 1 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 10 client.37109.objecter close_session for osd.2 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6809/3689719376 conn(0x7f044c0282c0 msgr2=0x7f044c11e460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6809/3689719376 conn(0x7f044c0282c0 0x7f044c11e460 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c059810 osd=2 1 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 10 client.37109.objecter close_session for osd.3 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6813/604971011 conn(0x7f044c11fad0 msgr2=0x7f044c11fef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f044bfff700 1 -- 172.21.15.162:0/871361626 reap_dead start 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6813/604971011 conn(0x7f044c11fad0 0x7f044c11fef0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c11baa0 osd=3 1 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 10 client.37109.objecter close_session for osd.4 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6800/3344263388 conn(0x7f044c123230 msgr2=0x7f044c125620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6800/3344263388 conn(0x7f044c123230 0x7f044c125620 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c120bb0 osd=4 1 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 10 client.37109.objecter close_session for osd.5 2024-02-13T00:44:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6804/2212296748 conn(0x7f044c126020 msgr2=0x7f044c128410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6804/2212296748 conn(0x7f044c126020 0x7f044c128410 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c120d90 osd=5 1 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 10 client.37109.objecter close_session for osd.6 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6808/1506640341 conn(0x7f044c12a4c0 msgr2=0x7f044c12a8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6808/1506640341 conn(0x7f044c12a4c0 0x7f044c12a8e0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c12b1a0 osd=6 1 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 10 client.37109.objecter close_session for osd.7 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6812/1964192772 conn(0x7f044c12dae0 msgr2=0x7f044c12df40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.181:6812/1964192772 conn(0x7f044c12dae0 0x7f044c12df40 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 20 client.37109.objecter put_session s=0x7f044c12d3c0 osd=7 1 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 20 client.37109.objecter shutdown clearing up homeless session... 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 10 client.37109.objecter successfully canceled tick 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f044bfff700 1 -- 172.21.15.162:0/871361626 reap_dead start 2024-02-13T00:44:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0434062c90 msgr2=0x7f0434065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0434062c90 0x7f0434065150 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 rx=0x7f044c109930 tx=0x7f044000b040).stop 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f044c0f84e0 msgr2=0x7f044c10dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f044c0f84e0 0x7f044c10dff0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 rx=0x7f043c009400 tx=0x7f043c009cb0).stop 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 shutdown_connections 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f044c0f84e0 0x7f044c10dff0 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 --2- 172.21.15.162:0/871361626 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0434062c90 0x7f0434065150 unknown :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 >> 172.21.15.162:0/871361626 conn(0x7f044c0f38a0 msgr2=0x7f044c0f5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 shutdown_connections 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 -- 172.21.15.162:0/871361626 wait complete. 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:09.623+0000 7f0451d2c700 1 librados: shutdown 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.632 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.633 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.633 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.633 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.633 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.633 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:44:09.652 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:09.671 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:44:09.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:09.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:44:09.774 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:09.801 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:09.820 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:09.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:09.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:09.952 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:10.025 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:10.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:10.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:10.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:10.156 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:10.175 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:10.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:10.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.213+0000 7f936b7b6700 1 -- 172.21.15.162:0/3240120813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93640f8d10 msgr2=0x7f93640f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:10.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.213+0000 7f936b7b6700 1 --2- 172.21.15.162:0/3240120813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93640f8d10 0x7f93640f90f0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 rx=0x7f9358004870 tx=0x7f9358009cb0).stop 2024-02-13T00:44:10.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.213+0000 7f936b7b6700 1 -- 172.21.15.162:0/3240120813 shutdown_connections 2024-02-13T00:44:10.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.213+0000 7f936b7b6700 1 --2- 172.21.15.162:0/3240120813 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9364102270 0x7f9364104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:10.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.213+0000 7f936b7b6700 1 --2- 172.21.15.162:0/3240120813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93640f8d10 0x7f93640f90f0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:10.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.213+0000 7f936b7b6700 1 --2- 172.21.15.162:0/3240120813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f93640f95e0 0x7f9364101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:10.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.213+0000 7f936b7b6700 1 -- 172.21.15.162:0/3240120813 >> 172.21.15.162:0/3240120813 conn(0x7f936406eb90 msgr2=0x7f93640f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:10.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.213+0000 7f936b7b6700 1 -- 172.21.15.162:0/3240120813 shutdown_connections 2024-02-13T00:44:10.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.213+0000 7f936b7b6700 1 -- 172.21.15.162:0/3240120813 wait complete. 2024-02-13T00:44:10.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f936b7b6700 1 librados: starting msgr at 2024-02-13T00:44:10.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f936b7b6700 1 librados: starting objecter 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f936b7b6700 1 Processor -- start 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f936b7b6700 1 -- start start 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f936b7b6700 1 librados: setting wanted keys 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f936b7b6700 1 librados: calling monclient init 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f936b7b6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f93640f8d10 0x7f936410e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f936b7b6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93640f95e0 0x7f936410e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f936b7b6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9364102270 0x7f93641083c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f936b7b6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93641088b0 con 0x7f93640f95e0 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f936b7b6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9364108a20 con 0x7f9364102270 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f936b7b6700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9364108b90 con 0x7f93640f8d10 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f9369552700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f93640f8d10 0x7f936410e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f9368d51700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93640f95e0 0x7f936410e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f9369552700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f93640f8d10 0x7f936410e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:38026/0 (socket says 172.21.15.162:38026) 2024-02-13T00:44:10.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f9369552700 1 -- 172.21.15.162:0/3796778684 learned_addr learned my addr 172.21.15.162:0/3796778684 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:10.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f9369d53700 1 --2- 172.21.15.162:0/3796778684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9364102270 0x7f93641083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:10.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f9369552700 1 -- 172.21.15.162:0/3796778684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9364102270 msgr2=0x7f93641083c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:10.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f9369552700 1 --2- 172.21.15.162:0/3796778684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9364102270 0x7f93641083c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:10.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f9369552700 1 -- 172.21.15.162:0/3796778684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93640f95e0 msgr2=0x7f936410e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:10.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f9369552700 1 --2- 172.21.15.162:0/3796778684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93640f95e0 0x7f936410e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:10.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f9369552700 1 -- 172.21.15.162:0/3796778684 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c0097c0 con 0x7f93640f8d10 2024-02-13T00:44:10.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.214+0000 7f9368d51700 1 --2- 172.21.15.162:0/3796778684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93640f95e0 0x7f936410e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:10.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.215+0000 7f9369552700 1 --2- 172.21.15.162:0/3796778684 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f93640f8d10 0x7f936410e1e0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f93580043a0 tx=0x7f935800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:10.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.215+0000 7f93567fc700 10 client.?.objecter ms_handle_connect 0x7f93640f8d10 2024-02-13T00:44:10.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.215+0000 7f93567fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:10.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.215+0000 7f93567fc700 1 -- 172.21.15.162:0/3796778684 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935800e070 con 0x7f93640f8d10 2024-02-13T00:44:10.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.215+0000 7f936b7b6700 1 -- 172.21.15.162:0/3796778684 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9358005040 con 0x7f93640f8d10 2024-02-13T00:44:10.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.215+0000 7f93567fc700 1 -- 172.21.15.162:0/3796778684 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9358009710 con 0x7f93640f8d10 2024-02-13T00:44:10.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.215+0000 7f93567fc700 1 -- 172.21.15.162:0/3796778684 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358019070 con 0x7f93640f8d10 2024-02-13T00:44:10.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.215+0000 7f936b7b6700 10 client.37115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:10.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.215+0000 7f936b7b6700 1 -- 172.21.15.162:0/3796778684 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9364109080 con 0x7f93640f8d10 2024-02-13T00:44:10.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.215+0000 7f936b7b6700 1 librados: init done 2024-02-13T00:44:10.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.216+0000 7f936b7b6700 10 client.37115.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:10.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.216+0000 7f936b7b6700 15 client.37115.objecter _session_command_op_assign -1 1 2024-02-13T00:44:10.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.216+0000 7f936b7b6700 20 client.37115.objecter _get_session osd=-1 returning homeless 2024-02-13T00:44:10.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.216+0000 7f936b7b6700 10 client.37115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:10.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.216+0000 7f936b7b6700 1 -- 172.21.15.162:0/3796778684 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f93640475e0 con 0x7f93640f8d10 2024-02-13T00:44:10.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.216+0000 7f93567fc700 1 -- 172.21.15.162:0/3796778684 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f93580090d0 con 0x7f93640f8d10 2024-02-13T00:44:10.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.216+0000 7f93567fc700 1 --2- 172.21.15.162:0/3796778684 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9350062c90 0x7f9350065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:10.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 1 -- 172.21.15.162:0/3796778684 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(294..294 src has 52..294) v4 ==== 8506+0+0 (secure 0 0 0) 0x7f935807dd00 con 0x7f93640f8d10 2024-02-13T00:44:10.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 10 client.37115.objecter ms_dispatch 0x7f9364001550 osd_map(294..294 src has 52..294) v4 2024-02-13T00:44:10.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 3 client.37115.objecter handle_osd_map got epochs [294,294] > 0 2024-02-13T00:44:10.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 3 client.37115.objecter handle_osd_map decoding full epoch 294 2024-02-13T00:44:10.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f9368d51700 1 --2- 172.21.15.162:0/3796778684 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9350062c90 0x7f9350065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:10.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 10 client.37115.objecter checking command 1 2024-02-13T00:44:10.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 1 --2- 172.21.15.162:0/3796778684 >> v2:172.21.15.181:6800/3344263388 conn(0x7f93500708e0 0x7f9350072d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:10.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 20 client.37115.objecter _get_session s=0x7f9350070230 osd=4 3 2024-02-13T00:44:10.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 20 client.37115.objecter put_session s=0x7f9350070230 osd=4 3 2024-02-13T00:44:10.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 15 client.37115.objecter _session_command_op_remove -1 1 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 20 client.37115.objecter _get_session s=0x7f9350070230 osd=4 3 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 20 client.37115.objecter get_session s=0x7f9350070230 osd=4 3 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 15 client.37115.objecter _session_command_op_assign 4 1 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 20 client.37115.objecter put_session s=0x7f9350070230 osd=4 4 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 10 client.37115.objecter _send_command 1 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 1 -- 172.21.15.162:0/3796778684 --> v2:172.21.15.181:6800/3344263388 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9350073400 con 0x7f93500708e0 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 20 client.37115.objecter dump_active .. 0 homeless 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 1 -- 172.21.15.162:0/3796778684 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=294) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9358082030 con 0x7f93640f8d10 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f9369d53700 1 --2- 172.21.15.162:0/3796778684 >> v2:172.21.15.181:6800/3344263388 conn(0x7f93500708e0 0x7f9350072d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 10 client.37115.objecter ms_handle_connect 0x7f9350062c90 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f9368d51700 1 --2- 172.21.15.162:0/3796778684 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9350062c90 0x7f9350065150 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 rx=0x7f9364109b60 tx=0x7f936000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f93567fc700 10 client.37115.objecter ms_handle_connect 0x7f93500708e0 2024-02-13T00:44:10.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.217+0000 7f9369d53700 1 --2- 172.21.15.162:0/3796778684 >> v2:172.21.15.181:6800/3344263388 conn(0x7f93500708e0 0x7f9350072d80 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:10.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.218+0000 7f93567fc700 1 -- 172.21.15.162:0/3796778684 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9350073400 con 0x7f93500708e0 2024-02-13T00:44:10.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.218+0000 7f93567fc700 10 client.37115.objecter ms_dispatch 0x7f9364001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:44:10.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.218+0000 7f93567fc700 10 client.37115.objecter _finish_command 1 = system:0 2024-02-13T00:44:10.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.218+0000 7f93567fc700 20 client.37115.objecter put_session s=0x7f9350070230 osd=4 4 2024-02-13T00:44:10.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.218+0000 7f93567fc700 15 client.37115.objecter _session_command_op_remove 4 1 2024-02-13T00:44:10.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f936b7b6700 10 client.37115.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-02-13T00:44:10.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f936b7b6700 15 client.37115.objecter _session_command_op_assign -1 2 2024-02-13T00:44:10.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f936b7b6700 20 client.37115.objecter _get_session s=0x7f9350070230 osd=4 3 2024-02-13T00:44:10.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f936b7b6700 20 client.37115.objecter put_session s=0x7f9350070230 osd=4 3 2024-02-13T00:44:10.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f936b7b6700 20 client.37115.objecter _get_session s=0x7f9350070230 osd=4 3 2024-02-13T00:44:10.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f936b7b6700 15 client.37115.objecter _session_command_op_remove -1 2 2024-02-13T00:44:10.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f936b7b6700 20 client.37115.objecter get_session s=0x7f9350070230 osd=4 3 2024-02-13T00:44:10.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f936b7b6700 15 client.37115.objecter _session_command_op_assign 4 2 2024-02-13T00:44:10.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f936b7b6700 20 client.37115.objecter put_session s=0x7f9350070230 osd=4 4 2024-02-13T00:44:10.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f936b7b6700 10 client.37115.objecter _send_command 2 2024-02-13T00:44:10.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f936b7b6700 1 -- 172.21.15.162:0/3796778684 --> v2:172.21.15.181:6800/3344263388 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7f9364107170 con 0x7f93500708e0 2024-02-13T00:44:10.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f93567fc700 1 -- 172.21.15.162:0/3796778684 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9364107170 con 0x7f93500708e0 2024-02-13T00:44:10.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f93567fc700 10 client.37115.objecter ms_dispatch 0x7f9364001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:44:10.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f93567fc700 10 client.37115.objecter _finish_command 2 = system:0 2024-02-13T00:44:10.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f93567fc700 20 client.37115.objecter put_session s=0x7f9350070230 osd=4 4 2024-02-13T00:44:10.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.235+0000 7f93567fc700 15 client.37115.objecter _session_command_op_remove 4 2 2024-02-13T00:44:10.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 10 librados: watch_flush enter 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 10 librados: watch_flush exit 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 10 client.37115.objecter close_session for osd.4 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 1 -- 172.21.15.162:0/3796778684 >> v2:172.21.15.181:6800/3344263388 conn(0x7f93500708e0 msgr2=0x7f9350072d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 1 --2- 172.21.15.162:0/3796778684 >> v2:172.21.15.181:6800/3344263388 conn(0x7f93500708e0 0x7f9350072d80 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 20 client.37115.objecter put_session s=0x7f9350070230 osd=4 1 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 20 client.37115.objecter shutdown clearing up homeless session... 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 10 client.37115.objecter successfully canceled tick 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 1 -- 172.21.15.162:0/3796778684 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9350062c90 msgr2=0x7f9350065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 1 --2- 172.21.15.162:0/3796778684 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9350062c90 0x7f9350065150 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 rx=0x7f9364109b60 tx=0x7f936000b040).stop 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 1 -- 172.21.15.162:0/3796778684 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f93640f8d10 msgr2=0x7f936410e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 1 --2- 172.21.15.162:0/3796778684 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f93640f8d10 0x7f936410e1e0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f93580043a0 tx=0x7f935800b040).stop 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f9369d53700 1 -- 172.21.15.162:0/3796778684 reap_dead start 2024-02-13T00:44:10.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 1 -- 172.21.15.162:0/3796778684 shutdown_connections 2024-02-13T00:44:10.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 1 -- 172.21.15.162:0/3796778684 >> 172.21.15.162:0/3796778684 conn(0x7f936406eb90 msgr2=0x7f9364102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:10.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 1 -- 172.21.15.162:0/3796778684 shutdown_connections 2024-02-13T00:44:10.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 1 -- 172.21.15.162:0/3796778684 wait complete. 2024-02-13T00:44:10.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:10.236+0000 7f936b7b6700 1 librados: shutdown 2024-02-13T00:44:10.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:09.244109+0000 mon.b (mon.1) 436 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:10.264 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:09.365604+0000 mon.a (mon.0) 1945 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:44:10.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:09.375012+0000 mon.a (mon.0) 1946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]': finished 2024-02-13T00:44:10.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:09.375093+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]': finished 2024-02-13T00:44:10.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:09.375160+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35"}]': finished 2024-02-13T00:44:10.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:09.375218+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e294: 8 total, 8 up, 6 in 2024-02-13T00:44:10.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:09.377883+0000 mon.c (mon.2) 290 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:10.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:09.378360+0000 mon.a (mon.0) 1950 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:10.265 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:09.478468+0000 mon.c (mon.2) 291 : audit [DBG] from='client.? 172.21.15.162:0/871361626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:10.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:09.244109+0000 mon.b (mon.1) 436 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:10.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:09.365604+0000 mon.a (mon.0) 1945 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:44:10.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:09.375012+0000 mon.a (mon.0) 1946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]': finished 2024-02-13T00:44:10.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:09.375093+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]': finished 2024-02-13T00:44:10.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:09.375160+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35"}]': finished 2024-02-13T00:44:10.266 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:09.375218+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e294: 8 total, 8 up, 6 in 2024-02-13T00:44:10.267 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:09.377883+0000 mon.c (mon.2) 290 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:10.267 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:09.378360+0000 mon.a (mon.0) 1950 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:10.267 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:09.478468+0000 mon.c (mon.2) 291 : audit [DBG] from='client.? 172.21.15.162:0/871361626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:10.278 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:44:10.304 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:10.323 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:44:10.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:10.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:44:10.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:10.453 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:10.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:09.244109+0000 mon.b (mon.1) 436 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:09.365604+0000 mon.a (mon.0) 1945 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:44:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:09.375012+0000 mon.a (mon.0) 1946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-83977-41"}]': finished 2024-02-13T00:44:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:09.375093+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-83985-51"}]': finished 2024-02-13T00:44:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:09.375160+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-35"}]': finished 2024-02-13T00:44:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:09.375218+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e294: 8 total, 8 up, 6 in 2024-02-13T00:44:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:09.377883+0000 mon.c (mon.2) 290 : audit [INF] from='client.? 172.21.15.162:0/3808116253' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:09.378360+0000 mon.a (mon.0) 1950 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]: dispatch 2024-02-13T00:44:10.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:09.478468+0000 mon.c (mon.2) 291 : audit [DBG] from='client.? 172.21.15.162:0/871361626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:11.004 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:11.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:11.108 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:11.137 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:11.158 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:11.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:11.262 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:11.287 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.245040+0000 mon.b (mon.1) 437 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:11.287 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.245647+0000 mon.b (mon.1) 438 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]: dispatch 2024-02-13T00:44:11.287 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.246056+0000 mon.a (mon.0) 1951 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]: dispatch 2024-02-13T00:44:11.287 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.364086+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]': finished 2024-02-13T00:44:11.287 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.364188+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]': finished 2024-02-13T00:44:11.287 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:10.364245+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e295: 8 total, 8 up, 6 in 2024-02-13T00:44:11.287 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.367571+0000 mon.b (mon.1) 439 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:11.288 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.371301+0000 mon.b (mon.1) 440 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:11.288 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.374708+0000 mon.b (mon.1) 441 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-13T00:44:11.288 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.376233+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-13T00:44:11.288 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:10.381972+0000 mgr.y (mgr.14544) 283 : cluster [DBG] pgmap v413: 238 pgs: 238 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:44:11.288 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.382337+0000 mon.c (mon.2) 292 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-13T00:44:11.288 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.405736+0000 mon.c (mon.2) 293 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:11.288 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.245040+0000 mon.b (mon.1) 437 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:11.288 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.245647+0000 mon.b (mon.1) 438 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]: dispatch 2024-02-13T00:44:11.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.246056+0000 mon.a (mon.0) 1951 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]: dispatch 2024-02-13T00:44:11.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.364086+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]': finished 2024-02-13T00:44:11.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.364188+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]': finished 2024-02-13T00:44:11.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:10.364245+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e295: 8 total, 8 up, 6 in 2024-02-13T00:44:11.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.367571+0000 mon.b (mon.1) 439 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:11.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.371301+0000 mon.b (mon.1) 440 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:11.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.374708+0000 mon.b (mon.1) 441 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-13T00:44:11.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.376233+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-13T00:44:11.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:10.381972+0000 mgr.y (mgr.14544) 283 : cluster [DBG] pgmap v413: 238 pgs: 238 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:44:11.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.382337+0000 mon.c (mon.2) 292 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-13T00:44:11.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.405736+0000 mon.c (mon.2) 293 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:11.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.626730+0000 mon.a (mon.0) 1956 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-13T00:44:11.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.626931+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:11.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.627407+0000 mon.c (mon.2) 294 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:11.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.627628+0000 mon.a (mon.0) 1958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:11.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.628009+0000 mon.c (mon.2) 295 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:11.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:10.628229+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:11.290 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.626730+0000 mon.a (mon.0) 1956 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-13T00:44:11.290 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.626931+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:11.290 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.627407+0000 mon.c (mon.2) 294 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:11.290 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.627628+0000 mon.a (mon.0) 1958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:11.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.628009+0000 mon.c (mon.2) 295 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:11.291 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:10.628229+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:11.291 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:11.445 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:11.446 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:11.472 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.5 reset_purged_snaps_last 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.245040+0000 mon.b (mon.1) 437 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.245647+0000 mon.b (mon.1) 438 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]: dispatch 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.246056+0000 mon.a (mon.0) 1951 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]: dispatch 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.364086+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-83985-51"}]': finished 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.364188+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]': finished 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:10.364245+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e295: 8 total, 8 up, 6 in 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.367571+0000 mon.b (mon.1) 439 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.371301+0000 mon.b (mon.1) 440 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.374708+0000 mon.b (mon.1) 441 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.376233+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:10.381972+0000 mgr.y (mgr.14544) 283 : cluster [DBG] pgmap v413: 238 pgs: 238 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.382337+0000 mon.c (mon.2) 292 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.405736+0000 mon.c (mon.2) 293 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.626730+0000 mon.a (mon.0) 1956 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-13T00:44:11.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.626931+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:11.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.627407+0000 mon.c (mon.2) 294 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:11.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.627628+0000 mon.a (mon.0) 1958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:11.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.628009+0000 mon.c (mon.2) 295 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:11.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:10.628229+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:11.737 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:11.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:11.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:11.869 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:12.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:11.376517+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]': finished 2024-02-13T00:44:12.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:11.376630+0000 mon.a (mon.0) 1961 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-13T00:44:12.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:11.376700+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:12.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:11.376794+0000 mon.a (mon.0) 1963 : cluster [DBG] osdmap e296: 8 total, 8 up, 6 in 2024-02-13T00:44:12.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:11.379776+0000 mon.b (mon.1) 442 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:12.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:11.380106+0000 mon.b (mon.1) 443 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:12.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:11.384176+0000 mon.c (mon.2) 296 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-83985-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:12.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:11.386319+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:12.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:11.386603+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-83985-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:12.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:11.376517+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]': finished 2024-02-13T00:44:12.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:11.376630+0000 mon.a (mon.0) 1961 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-13T00:44:12.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:11.376700+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:12.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:11.376794+0000 mon.a (mon.0) 1963 : cluster [DBG] osdmap e296: 8 total, 8 up, 6 in 2024-02-13T00:44:12.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:11.379776+0000 mon.b (mon.1) 442 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:12.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:11.380106+0000 mon.b (mon.1) 443 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:12.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:11.384176+0000 mon.c (mon.2) 296 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-83985-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:12.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:11.386319+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:12.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:11.386603+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-83985-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:12.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:44:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:11.376517+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]': finished 2024-02-13T00:44:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:11.376630+0000 mon.a (mon.0) 1961 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-13T00:44:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:11.376700+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:11.376794+0000 mon.a (mon.0) 1963 : cluster [DBG] osdmap e296: 8 total, 8 up, 6 in 2024-02-13T00:44:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:11.379776+0000 mon.b (mon.1) 442 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:11.380106+0000 mon.b (mon.1) 443 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:11.384176+0000 mon.c (mon.2) 296 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-83985-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:11.386319+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:11.386603+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-83985-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:12.380654+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:12.382484+0000 mgr.y (mgr.14544) 284 : cluster [DBG] pgmap v416: 270 pgs: 32 unknown, 238 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:44:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:12.399056+0000 mon.b (mon.1) 444 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:12.630651+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e297: 8 total, 8 up, 6 in 2024-02-13T00:44:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:12.634375+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:13.382511+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-83985-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-83985-52"}]': finished 2024-02-13T00:44:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:13.382610+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:13.382672+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e298: 8 total, 8 up, 6 in 2024-02-13T00:44:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:13.388945+0000 mon.b (mon.1) 445 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-37"}]: dispatch 2024-02-13T00:44:14.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:13.389605+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-37"}]: dispatch 2024-02-13T00:44:14.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:12.380654+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:14.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:12.382484+0000 mgr.y (mgr.14544) 284 : cluster [DBG] pgmap v416: 270 pgs: 32 unknown, 238 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:44:14.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:12.399056+0000 mon.b (mon.1) 444 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:12.630651+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e297: 8 total, 8 up, 6 in 2024-02-13T00:44:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:12.634375+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:13.382511+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-83985-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-83985-52"}]': finished 2024-02-13T00:44:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:13.382610+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:13.382672+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e298: 8 total, 8 up, 6 in 2024-02-13T00:44:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:13.388945+0000 mon.b (mon.1) 445 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-37"}]: dispatch 2024-02-13T00:44:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:13.389605+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-37"}]: dispatch 2024-02-13T00:44:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:12.380654+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:12.382484+0000 mgr.y (mgr.14544) 284 : cluster [DBG] pgmap v416: 270 pgs: 32 unknown, 238 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:44:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:12.399056+0000 mon.b (mon.1) 444 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:12.630651+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e297: 8 total, 8 up, 6 in 2024-02-13T00:44:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:12.634375+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:13.382511+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-83985-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-83985-52"}]': finished 2024-02-13T00:44:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:13.382610+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:13.382672+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e298: 8 total, 8 up, 6 in 2024-02-13T00:44:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:13.388945+0000 mon.b (mon.1) 445 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-37"}]: dispatch 2024-02-13T00:44:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:13.389605+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-37"}]: dispatch 2024-02-13T00:44:14.120 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:14.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:14.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:14.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:14.260 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:14.281 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:14.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:14.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:14.420 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:14.440 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:14.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:14.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:14.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:14.788 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:14.807 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:14.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:14.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:14.943 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:15.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.002+0000 7f88cf4d8700 1 -- 172.21.15.162:0/1553211153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88c8104660 msgr2=0x7f88c8106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:15.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.002+0000 7f88cf4d8700 1 --2- 172.21.15.162:0/1553211153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88c8104660 0x7f88c8106a50 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 rx=0x7f88c0009c10 tx=0x7f88c0004d10).stop 2024-02-13T00:44:15.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 -- 172.21.15.162:0/1553211153 shutdown_connections 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 --2- 172.21.15.162:0/1553211153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88c8104660 0x7f88c8106a50 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 --2- 172.21.15.162:0/1553211153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88c80f6b00 0x7f88c80f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 --2- 172.21.15.162:0/1553211153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88c80f73d0 0x7f88c8104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 -- 172.21.15.162:0/1553211153 >> 172.21.15.162:0/1553211153 conn(0x7f88c806eb90 msgr2=0x7f88c80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 -- 172.21.15.162:0/1553211153 shutdown_connections 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 -- 172.21.15.162:0/1553211153 wait complete. 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 librados: starting msgr at 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 librados: starting objecter 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 Processor -- start 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 -- start start 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 librados: setting wanted keys 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 librados: calling monclient init 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88c80f6b00 0x7f88c8110140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88c80f73d0 0x7f88c8110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88c8104660 0x7f88c810a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c810a7d0 con 0x7f88c80f73d0 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c810a940 con 0x7f88c8104660 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cf4d8700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c810ab00 con 0x7f88c80f6b00 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.003+0000 7f88cd274700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88c80f6b00 0x7f88c8110140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cca73700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88c80f73d0 0x7f88c8110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cd274700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88c80f6b00 0x7f88c8110140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:32920/0 (socket says 172.21.15.162:32920) 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cd274700 1 -- 172.21.15.162:0/404314068 learned_addr learned my addr 172.21.15.162:0/404314068 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cda75700 1 --2- 172.21.15.162:0/404314068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88c8104660 0x7f88c810a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cca73700 1 -- 172.21.15.162:0/404314068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88c80f6b00 msgr2=0x7f88c8110140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cca73700 1 --2- 172.21.15.162:0/404314068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88c80f6b00 0x7f88c8110140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cca73700 1 -- 172.21.15.162:0/404314068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88c8104660 msgr2=0x7f88c810a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cca73700 1 --2- 172.21.15.162:0/404314068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88c8104660 0x7f88c810a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cca73700 1 -- 172.21.15.162:0/404314068 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c4005040 con 0x7f88c80f73d0 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cd274700 1 --2- 172.21.15.162:0/404314068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88c80f6b00 0x7f88c8110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:15.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cca73700 1 --2- 172.21.15.162:0/404314068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88c80f73d0 0x7f88c8110630 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 rx=0x7f88c4004920 tx=0x7f88c4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88be7fc700 10 client.?.objecter ms_handle_connect 0x7f88c80f73d0 2024-02-13T00:44:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88be7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88be7fc700 1 -- 172.21.15.162:0/404314068 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c4007a40 con 0x7f88c80f73d0 2024-02-13T00:44:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cf4d8700 1 -- 172.21.15.162:0/404314068 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c00097c0 con 0x7f88c80f73d0 2024-02-13T00:44:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88be7fc700 1 -- 172.21.15.162:0/404314068 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88c400f040 con 0x7f88c80f73d0 2024-02-13T00:44:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cf4d8700 10 client.27018.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cf4d8700 1 -- 172.21.15.162:0/404314068 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c8116f70 con 0x7f88c80f73d0 2024-02-13T00:44:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88be7fc700 1 -- 172.21.15.162:0/404314068 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c40149f0 con 0x7f88c80f73d0 2024-02-13T00:44:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.004+0000 7f88cf4d8700 1 librados: init done 2024-02-13T00:44:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.005+0000 7f88cf4d8700 1 -- 172.21.15.162:0/404314068 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c80475e0 con 0x7f88c80f73d0 2024-02-13T00:44:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.005+0000 7f88be7fc700 1 -- 172.21.15.162:0/404314068 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f88c4010530 con 0x7f88c80f73d0 2024-02-13T00:44:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.006+0000 7f88be7fc700 1 --2- 172.21.15.162:0/404314068 >> v2:172.21.15.162:6800/2866534053 conn(0x7f88b4062c90 0x7f88b4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:15.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.006+0000 7f88be7fc700 1 -- 172.21.15.162:0/404314068 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(299..299 src has 52..299) v4 ==== 9226+0+0 (secure 0 0 0) 0x7f88c407d1a0 con 0x7f88c80f73d0 2024-02-13T00:44:15.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.006+0000 7f88cd274700 1 --2- 172.21.15.162:0/404314068 >> v2:172.21.15.162:6800/2866534053 conn(0x7f88b4062c90 0x7f88b4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:15.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.006+0000 7f88be7fc700 10 client.27018.objecter ms_dispatch 0x7f88c8001550 osd_map(299..299 src has 52..299) v4 2024-02-13T00:44:15.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.006+0000 7f88be7fc700 3 client.27018.objecter handle_osd_map got epochs [299,299] > 0 2024-02-13T00:44:15.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.006+0000 7f88be7fc700 3 client.27018.objecter handle_osd_map decoding full epoch 299 2024-02-13T00:44:15.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.006+0000 7f88be7fc700 20 client.27018.objecter dump_active .. 0 homeless 2024-02-13T00:44:15.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.006+0000 7f88be7fc700 10 client.27018.objecter ms_handle_connect 0x7f88b4062c90 2024-02-13T00:44:15.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.006+0000 7f88cd274700 1 --2- 172.21.15.162:0/404314068 >> v2:172.21.15.162:6800/2866534053 conn(0x7f88b4062c90 0x7f88b4065150 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7f88b800f3d0 tx=0x7f88b8009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:15.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.010+0000 7f88be7fc700 1 -- 172.21.15.162:0/404314068 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f88c4049f60 con 0x7f88c80f73d0 2024-02-13T00:44:15.042 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:15.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:15.146 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:15.178 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:15.184 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:44:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:44:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:44:15.184 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:44:15.184 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:44:15.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.188+0000 7f88cf4d8700 1 -- 172.21.15.162:0/404314068 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f88c810b7c0 con 0x7f88c80f73d0 2024-02-13T00:44:15.199 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:15.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:15.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:15.333 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:15.353 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:15.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:15.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.365+0000 7f88be7fc700 1 -- 172.21.15.162:0/404314068 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v300) v1 ==== 91+0+0 (secure 0 0 0) 0x7f88c4050180 con 0x7f88c80f73d0 2024-02-13T00:44:15.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.369+0000 7f88cf4d8700 10 librados: watch_flush enter 2024-02-13T00:44:15.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 10 librados: watch_flush exit 2024-02-13T00:44:15.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 20 client.27018.objecter shutdown clearing up homeless session... 2024-02-13T00:44:15.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 10 client.27018.objecter successfully canceled tick 2024-02-13T00:44:15.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 1 -- 172.21.15.162:0/404314068 >> v2:172.21.15.162:6800/2866534053 conn(0x7f88b4062c90 msgr2=0x7f88b4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:15.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 1 --2- 172.21.15.162:0/404314068 >> v2:172.21.15.162:6800/2866534053 conn(0x7f88b4062c90 0x7f88b4065150 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7f88b800f3d0 tx=0x7f88b8009040).stop 2024-02-13T00:44:15.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 1 -- 172.21.15.162:0/404314068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88c80f73d0 msgr2=0x7f88c8110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:15.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 1 --2- 172.21.15.162:0/404314068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88c80f73d0 0x7f88c8110630 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 rx=0x7f88c4004920 tx=0x7f88c4009cb0).stop 2024-02-13T00:44:15.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 1 -- 172.21.15.162:0/404314068 shutdown_connections 2024-02-13T00:44:15.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 1 --2- 172.21.15.162:0/404314068 >> v2:172.21.15.162:6800/2866534053 conn(0x7f88b4062c90 0x7f88b4065150 secure :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 rx=0x7f88b800f3d0 tx=0x7f88b8009040).stop 2024-02-13T00:44:15.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 1 --2- 172.21.15.162:0/404314068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88c8104660 0x7f88c810a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:15.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 1 --2- 172.21.15.162:0/404314068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88c80f6b00 0x7f88c8110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:15.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 1 --2- 172.21.15.162:0/404314068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88c80f73d0 0x7f88c8110630 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:15.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 1 -- 172.21.15.162:0/404314068 >> 172.21.15.162:0/404314068 conn(0x7f88c806eb90 msgr2=0x7f88c806f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:15.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.370+0000 7f88cf4d8700 1 -- 172.21.15.162:0/404314068 shutdown_connections 2024-02-13T00:44:15.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.371+0000 7f88cf4d8700 1 -- 172.21.15.162:0/404314068 wait complete. 2024-02-13T00:44:15.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:15.371+0000 7f88cf4d8700 1 librados: shutdown 2024-02-13T00:44:15.376 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:44:15.457 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:15.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:14.382982+0000 mgr.y (mgr.14544) 285 : cluster [DBG] pgmap v418: 278 pgs: 40 unknown, 238 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:15.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:14.385452+0000 mon.a (mon.0) 1973 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-37"}]': finished 2024-02-13T00:44:15.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:14.385515+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e299: 8 total, 8 up, 6 in 2024-02-13T00:44:15.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:14.392307+0000 mon.b (mon.1) 446 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-37", "mode": "writeback"}]: dispatch 2024-02-13T00:44:15.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:14.393950+0000 mon.a (mon.0) 1975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-37", "mode": "writeback"}]: dispatch 2024-02-13T00:44:15.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:14.974613+0000 mon.a (mon.0) 1976 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:15.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:15.190243+0000 mon.a (mon.0) 1977 : audit [INF] from='client.? 172.21.15.162:0/404314068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:44:15.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:15.358976+0000 mon.a (mon.0) 1978 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:44:15.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:15.359000+0000 mon.a (mon.0) 1979 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:15.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:15.363712+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-37", "mode": "writeback"}]': finished 2024-02-13T00:44:15.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:15.363834+0000 mon.a (mon.0) 1981 : audit [INF] from='client.? 172.21.15.162:0/404314068' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:44:15.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:15.363890+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e300: 8 total, 8 up, 6 in 2024-02-13T00:44:15.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:15.365257+0000 mon.c (mon.2) 297 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:15.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:15.375999+0000 mon.a (mon.0) 1983 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:15.483 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:15.501 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:15.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:15.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:15.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:15.633 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:15.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:44:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:44:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:14.382982+0000 mgr.y (mgr.14544) 285 : cluster [DBG] pgmap v418: 278 pgs: 40 unknown, 238 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:14.385452+0000 mon.a (mon.0) 1973 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-37"}]': finished 2024-02-13T00:44:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:14.385515+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e299: 8 total, 8 up, 6 in 2024-02-13T00:44:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:14.392307+0000 mon.b (mon.1) 446 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-37", "mode": "writeback"}]: dispatch 2024-02-13T00:44:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:14.393950+0000 mon.a (mon.0) 1975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-37", "mode": "writeback"}]: dispatch 2024-02-13T00:44:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:14.974613+0000 mon.a (mon.0) 1976 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:15.190243+0000 mon.a (mon.0) 1977 : audit [INF] from='client.? 172.21.15.162:0/404314068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:44:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:15.358976+0000 mon.a (mon.0) 1978 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:44:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:15.359000+0000 mon.a (mon.0) 1979 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:15.363712+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-37", "mode": "writeback"}]': finished 2024-02-13T00:44:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:15.363834+0000 mon.a (mon.0) 1981 : audit [INF] from='client.? 172.21.15.162:0/404314068' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:44:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:15.363890+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e300: 8 total, 8 up, 6 in 2024-02-13T00:44:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:15.365257+0000 mon.c (mon.2) 297 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:15.375999+0000 mon.a (mon.0) 1983 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:14.382982+0000 mgr.y (mgr.14544) 285 : cluster [DBG] pgmap v418: 278 pgs: 40 unknown, 238 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:14.385452+0000 mon.a (mon.0) 1973 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-37"}]': finished 2024-02-13T00:44:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:14.385515+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e299: 8 total, 8 up, 6 in 2024-02-13T00:44:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:14.392307+0000 mon.b (mon.1) 446 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-37", "mode": "writeback"}]: dispatch 2024-02-13T00:44:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:14.393950+0000 mon.a (mon.0) 1975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-37", "mode": "writeback"}]: dispatch 2024-02-13T00:44:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:14.974613+0000 mon.a (mon.0) 1976 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:15.190243+0000 mon.a (mon.0) 1977 : audit [INF] from='client.? 172.21.15.162:0/404314068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:44:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:15.358976+0000 mon.a (mon.0) 1978 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:44:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:15.359000+0000 mon.a (mon.0) 1979 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:15.363712+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-37", "mode": "writeback"}]': finished 2024-02-13T00:44:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:15.363834+0000 mon.a (mon.0) 1981 : audit [INF] from='client.? 172.21.15.162:0/404314068' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:44:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:15.363890+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e300: 8 total, 8 up, 6 in 2024-02-13T00:44:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:15.365257+0000 mon.c (mon.2) 297 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:15.375999+0000 mon.a (mon.0) 1983 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:15.982 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:15.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:16.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:16.116 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:16.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-13T00:44:16.135 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:16.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:16.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:16.259 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:44:16] "GET /metrics HTTP/1.1" 200 188859 "" "Prometheus/2.33.4" 2024-02-13T00:44:16.267 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:15.468291+0000 mon.b (mon.1) 447 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:15.468804+0000 mon.a (mon.0) 1984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:15.507479+0000 mgr.y (mgr.14544) 286 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:16.376830+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]': finished 2024-02-13T00:44:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:16.376933+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:44:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:16.376992+0000 mon.a (mon.0) 1987 : cluster [DBG] osdmap e301: 8 total, 8 up, 6 in 2024-02-13T00:44:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:16.378690+0000 mon.c (mon.2) 298 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:16.378832+0000 mon.b (mon.1) 448 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37"}]: dispatch 2024-02-13T00:44:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:16.379249+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:16.379610+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37"}]: dispatch 2024-02-13T00:44:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:16.383878+0000 mon.c (mon.2) 299 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-13T00:44:16.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:16.384729+0000 mon.a (mon.0) 1990 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-13T00:44:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:15.468291+0000 mon.b (mon.1) 447 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:15.468804+0000 mon.a (mon.0) 1984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:15.507479+0000 mgr.y (mgr.14544) 286 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:16.376830+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]': finished 2024-02-13T00:44:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:16.376933+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:44:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:16.376992+0000 mon.a (mon.0) 1987 : cluster [DBG] osdmap e301: 8 total, 8 up, 6 in 2024-02-13T00:44:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:16.378690+0000 mon.c (mon.2) 298 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:16.378832+0000 mon.b (mon.1) 448 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37"}]: dispatch 2024-02-13T00:44:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:16.379249+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:16.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:16.379610+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37"}]: dispatch 2024-02-13T00:44:16.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:16.383878+0000 mon.c (mon.2) 299 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-13T00:44:16.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:16.384729+0000 mon.a (mon.0) 1990 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-13T00:44:16.711 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:16.716 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:16.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:16.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:15.468291+0000 mon.b (mon.1) 447 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:15.468804+0000 mon.a (mon.0) 1984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:15.507479+0000 mgr.y (mgr.14544) 286 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:16.376830+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-83985-52"}]': finished 2024-02-13T00:44:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:16.376933+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:44:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:16.376992+0000 mon.a (mon.0) 1987 : cluster [DBG] osdmap e301: 8 total, 8 up, 6 in 2024-02-13T00:44:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:16.378690+0000 mon.c (mon.2) 298 : audit [INF] from='client.? 172.21.15.162:0/695397089' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:16.378832+0000 mon.b (mon.1) 448 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37"}]: dispatch 2024-02-13T00:44:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:16.379249+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]: dispatch 2024-02-13T00:44:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:16.379610+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37"}]: dispatch 2024-02-13T00:44:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:16.383878+0000 mon.c (mon.2) 299 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-13T00:44:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:16.384729+0000 mon.a (mon.0) 1990 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-13T00:44:16.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:16.849 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:16.972 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:16.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:16.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:17.077 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:44:17.109 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:17.130 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:44:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:17.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:44:17.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:17.264 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:17.284 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:17.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:17.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:17.418 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:17.437 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:17.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:17.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:17.567 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:17.585 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:17.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:17.689 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:17.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:16.253348+0000 osd.0 (osd.0) 107 : cluster [DBG] 28.1f scrub starts 2024-02-13T00:44:17.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:16.256507+0000 osd.0 (osd.0) 108 : cluster [DBG] 28.1f scrub ok 2024-02-13T00:44:17.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:16.383563+0000 mgr.y (mgr.14544) 287 : cluster [DBG] pgmap v422: 270 pgs: 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 287 B/s, 3 objects/s recovering 2024-02-13T00:44:17.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:16.498845+0000 osd.2 (osd.2) 121 : cluster [DBG] 28.d scrub starts 2024-02-13T00:44:17.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:16.499776+0000 osd.2 (osd.2) 122 : cluster [DBG] 28.d scrub ok 2024-02-13T00:44:17.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:17.373005+0000 mon.a (mon.0) 1991 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:44:17.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:17.381342+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]': finished 2024-02-13T00:44:17.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:17.381389+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37"}]': finished 2024-02-13T00:44:17.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:17.381425+0000 mon.a (mon.0) 1994 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-13T00:44:17.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:17.381475+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e302: 8 total, 8 up, 6 in 2024-02-13T00:44:17.715 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:17.733 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:17.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:16.253348+0000 osd.0 (osd.0) 107 : cluster [DBG] 28.1f scrub starts 2024-02-13T00:44:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:16.256507+0000 osd.0 (osd.0) 108 : cluster [DBG] 28.1f scrub ok 2024-02-13T00:44:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:16.383563+0000 mgr.y (mgr.14544) 287 : cluster [DBG] pgmap v422: 270 pgs: 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 287 B/s, 3 objects/s recovering 2024-02-13T00:44:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:16.498845+0000 osd.2 (osd.2) 121 : cluster [DBG] 28.d scrub starts 2024-02-13T00:44:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:16.499776+0000 osd.2 (osd.2) 122 : cluster [DBG] 28.d scrub ok 2024-02-13T00:44:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:17.373005+0000 mon.a (mon.0) 1991 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:44:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:17.381342+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]': finished 2024-02-13T00:44:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:17.381389+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37"}]': finished 2024-02-13T00:44:17.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:17.381425+0000 mon.a (mon.0) 1994 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-13T00:44:17.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:17.381475+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e302: 8 total, 8 up, 6 in 2024-02-13T00:44:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:16.253348+0000 osd.0 (osd.0) 107 : cluster [DBG] 28.1f scrub starts 2024-02-13T00:44:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:16.256507+0000 osd.0 (osd.0) 108 : cluster [DBG] 28.1f scrub ok 2024-02-13T00:44:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:16.383563+0000 mgr.y (mgr.14544) 287 : cluster [DBG] pgmap v422: 270 pgs: 270 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 287 B/s, 3 objects/s recovering 2024-02-13T00:44:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:16.498845+0000 osd.2 (osd.2) 121 : cluster [DBG] 28.d scrub starts 2024-02-13T00:44:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:16.499776+0000 osd.2 (osd.2) 122 : cluster [DBG] 28.d scrub ok 2024-02-13T00:44:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:17.373005+0000 mon.a (mon.0) 1991 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:44:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:17.381342+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-83985-52"}]': finished 2024-02-13T00:44:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:17.381389+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-37"}]': finished 2024-02-13T00:44:17.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:17.381425+0000 mon.a (mon.0) 1994 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-13T00:44:17.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:17.381475+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e302: 8 total, 8 up, 6 in 2024-02-13T00:44:17.836 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:17.865 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:18.205 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:18.214 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:18.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:18.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:18.318 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:18.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:18.347 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:18.365 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:18.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:18.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:18.469 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:18.498 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:18.640 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:44:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:44:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:44:18.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:44:18.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:17.292509+0000 osd.0 (osd.0) 109 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:44:18.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:17.295455+0000 osd.0 (osd.0) 110 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:44:18.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:17.407716+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:18.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:17.408163+0000 mon.a (mon.0) 1997 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:18.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:17.408496+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi162-83985-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:18.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:18.383691+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi162-83985-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:18.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:18.383793+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e303: 8 total, 8 up, 6 in 2024-02-13T00:44:18.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:18.385163+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi162-83985-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:18.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:17.292509+0000 osd.0 (osd.0) 109 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:44:18.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:17.295455+0000 osd.0 (osd.0) 110 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:44:18.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:17.407716+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:18.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:17.408163+0000 mon.a (mon.0) 1997 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:18.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:17.408496+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi162-83985-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:18.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:18.383691+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi162-83985-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:18.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:18.383793+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e303: 8 total, 8 up, 6 in 2024-02-13T00:44:18.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:18.385163+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi162-83985-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:18.945 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:18.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:19.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:17.292509+0000 osd.0 (osd.0) 109 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:44:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:17.295455+0000 osd.0 (osd.0) 110 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:44:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:17.407716+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:17.408163+0000 mon.a (mon.0) 1997 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:17.408496+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi162-83985-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:18.383691+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi162-83985-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:18.383793+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e303: 8 total, 8 up, 6 in 2024-02-13T00:44:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:18.385163+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi162-83985-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:19.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:19.077 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:19.096 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:19.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:19.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:19.226 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:19.374 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:19.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:19.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:19.505 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:19.524 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:19.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:19.628 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:19.655 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:19.673 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:19.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:19.777 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:19.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:18.088196+0000 osd.5 (osd.5) 127 : cluster [DBG] 28.17 scrub starts 2024-02-13T00:44:19.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:18.089244+0000 osd.5 (osd.5) 128 : cluster [DBG] 28.17 scrub ok 2024-02-13T00:44:19.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:18.387158+0000 mgr.y (mgr.14544) 288 : cluster [DBG] pgmap v425: 238 pgs: 238 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 287 B/s, 3 objects/s recovering 2024-02-13T00:44:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:19.626565+0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e304: 8 total, 8 up, 6 in 2024-02-13T00:44:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:19.627710+0000 mon.b (mon.1) 449 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:19.628334+0000 mon.a (mon.0) 2003 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:19.806 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:20.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:18.088196+0000 osd.5 (osd.5) 127 : cluster [DBG] 28.17 scrub starts 2024-02-13T00:44:20.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:18.089244+0000 osd.5 (osd.5) 128 : cluster [DBG] 28.17 scrub ok 2024-02-13T00:44:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:18.387158+0000 mgr.y (mgr.14544) 288 : cluster [DBG] pgmap v425: 238 pgs: 238 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 287 B/s, 3 objects/s recovering 2024-02-13T00:44:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:19.626565+0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e304: 8 total, 8 up, 6 in 2024-02-13T00:44:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:19.627710+0000 mon.b (mon.1) 449 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:19.628334+0000 mon.a (mon.0) 2003 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:20.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:18.088196+0000 osd.5 (osd.5) 127 : cluster [DBG] 28.17 scrub starts 2024-02-13T00:44:20.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:18.089244+0000 osd.5 (osd.5) 128 : cluster [DBG] 28.17 scrub ok 2024-02-13T00:44:20.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:18.387158+0000 mgr.y (mgr.14544) 288 : cluster [DBG] pgmap v425: 238 pgs: 238 active+clean; 8.8 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 287 B/s, 3 objects/s recovering 2024-02-13T00:44:20.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:19.626565+0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e304: 8 total, 8 up, 6 in 2024-02-13T00:44:20.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:19.627710+0000 mon.b (mon.1) 449 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:20.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:19.628334+0000 mon.a (mon.0) 2003 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:20.856 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:20.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:20.961 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:44:20.993 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:21.014 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:44:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:21.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:44:21.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.095+0000 7f3b1e862700 1 -- 172.21.15.162:0/3790509584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b180fe580 msgr2=0x7f3b180f7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:21.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.095+0000 7f3b1e862700 1 --2- 172.21.15.162:0/3790509584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b180fe580 0x7f3b180f7db0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 rx=0x7f3b0c004870 tx=0x7f3b0c009cb0).stop 2024-02-13T00:44:21.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.095+0000 7f3b1e862700 1 -- 172.21.15.162:0/3790509584 shutdown_connections 2024-02-13T00:44:21.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.095+0000 7f3b1e862700 1 --2- 172.21.15.162:0/3790509584 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b180f8490 0x7f3b18104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:21.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.095+0000 7f3b1e862700 1 --2- 172.21.15.162:0/3790509584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b180fdc20 0x7f3b180fe000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:21.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.096+0000 7f3b1e862700 1 --2- 172.21.15.162:0/3790509584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b180fe580 0x7f3b180f7db0 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:21.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.096+0000 7f3b1e862700 1 -- 172.21.15.162:0/3790509584 >> 172.21.15.162:0/3790509584 conn(0x7f3b180f3850 msgr2=0x7f3b180f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:21.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.096+0000 7f3b1e862700 1 -- 172.21.15.162:0/3790509584 shutdown_connections 2024-02-13T00:44:21.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.096+0000 7f3b1e862700 1 -- 172.21.15.162:0/3790509584 wait complete. 2024-02-13T00:44:21.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.096+0000 7f3b1e862700 1 librados: starting msgr at 2024-02-13T00:44:21.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.096+0000 7f3b1e862700 1 librados: starting objecter 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.096+0000 7f3b1e862700 1 Processor -- start 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.096+0000 7f3b1e862700 1 -- start start 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.096+0000 7f3b1e862700 1 librados: setting wanted keys 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.096+0000 7f3b1e862700 1 librados: calling monclient init 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b1e862700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b180f8490 0x7f3b1810e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b1e862700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b180fdc20 0x7f3b1810e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b1e862700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b180fe580 0x7f3b18108250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b1e862700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b18108740 con 0x7f3b180fe580 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b1e862700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b181088b0 con 0x7f3b180f8490 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b1e862700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b18108a70 con 0x7f3b180fdc20 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b177fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b180fdc20 0x7f3b1810e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b177fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b180fdc20 0x7f3b1810e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:32962/0 (socket says 172.21.15.162:32962) 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b177fe700 1 -- 172.21.15.162:0/3244817528 learned_addr learned my addr 172.21.15.162:0/3244817528 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:21.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b1cdff700 1 --2- 172.21.15.162:0/3244817528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b180fe580 0x7f3b18108250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b17fff700 1 --2- 172.21.15.162:0/3244817528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b180f8490 0x7f3b1810e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b177fe700 1 -- 172.21.15.162:0/3244817528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b180f8490 msgr2=0x7f3b1810e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b177fe700 1 --2- 172.21.15.162:0/3244817528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b180f8490 0x7f3b1810e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b177fe700 1 -- 172.21.15.162:0/3244817528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b180fe580 msgr2=0x7f3b18108250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b177fe700 1 --2- 172.21.15.162:0/3244817528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b180fe580 0x7f3b18108250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b177fe700 1 -- 172.21.15.162:0/3244817528 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b100097c0 con 0x7f3b180fdc20 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b177fe700 1 --2- 172.21.15.162:0/3244817528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b180fdc20 0x7f3b1810e5a0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 rx=0x7f3b0c009eb0 tx=0x7f3b0c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b157fa700 10 client.?.objecter ms_handle_connect 0x7f3b180fdc20 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b157fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b157fa700 1 -- 172.21.15.162:0/3244817528 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c00e070 con 0x7f3b180fdc20 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b1e862700 1 -- 172.21.15.162:0/3244817528 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0c005040 con 0x7f3b180fdc20 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b157fa700 1 -- 172.21.15.162:0/3244817528 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b0c013da0 con 0x7f3b180fdc20 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b157fa700 1 -- 172.21.15.162:0/3244817528 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c019070 con 0x7f3b180fdc20 2024-02-13T00:44:21.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b1e862700 10 client.37139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:21.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b1e862700 1 -- 172.21.15.162:0/3244817528 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b18108fe0 con 0x7f3b180fdc20 2024-02-13T00:44:21.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.097+0000 7f3b1e862700 1 librados: init done 2024-02-13T00:44:21.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.098+0000 7f3b1e862700 10 client.37139.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:21.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.098+0000 7f3b1e862700 15 client.37139.objecter _session_command_op_assign -1 1 2024-02-13T00:44:21.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.098+0000 7f3b1e862700 20 client.37139.objecter _get_session osd=-1 returning homeless 2024-02-13T00:44:21.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.098+0000 7f3b1e862700 10 client.37139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:21.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.098+0000 7f3b1e862700 1 -- 172.21.15.162:0/3244817528 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3b180607d0 con 0x7f3b180fdc20 2024-02-13T00:44:21.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 1 -- 172.21.15.162:0/3244817528 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f3b0c0283f0 con 0x7f3b180fdc20 2024-02-13T00:44:21.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 1 --2- 172.21.15.162:0/3244817528 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3b00062c90 0x7f3b00065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:21.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b17fff700 1 --2- 172.21.15.162:0/3244817528 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3b00062c90 0x7f3b00065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 1 -- 172.21.15.162:0/3244817528 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(305..305 src has 52..305) v4 ==== 9170+0+0 (secure 0 0 0) 0x7f3b0c07e160 con 0x7f3b180fdc20 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 10 client.37139.objecter ms_dispatch 0x7f3b18001550 osd_map(305..305 src has 52..305) v4 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 3 client.37139.objecter handle_osd_map got epochs [305,305] > 0 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 3 client.37139.objecter handle_osd_map decoding full epoch 305 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 10 client.37139.objecter checking command 1 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 1 --2- 172.21.15.162:0/3244817528 >> v2:172.21.15.181:6804/2212296748 conn(0x7f3b000713f0 0x7f3b00073890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 20 client.37139.objecter _get_session s=0x7f3b00070d40 osd=5 3 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 20 client.37139.objecter put_session s=0x7f3b00070d40 osd=5 3 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 15 client.37139.objecter _session_command_op_remove -1 1 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 20 client.37139.objecter _get_session s=0x7f3b00070d40 osd=5 3 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 20 client.37139.objecter get_session s=0x7f3b00070d40 osd=5 3 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 15 client.37139.objecter _session_command_op_assign 5 1 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 20 client.37139.objecter put_session s=0x7f3b00070d40 osd=5 4 2024-02-13T00:44:21.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 10 client.37139.objecter _send_command 1 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 1 -- 172.21.15.162:0/3244817528 --> v2:172.21.15.181:6804/2212296748 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b00073f50 con 0x7f3b000713f0 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 20 client.37139.objecter dump_active .. 0 homeless 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 1 -- 172.21.15.162:0/3244817528 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=305) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b0c080740 con 0x7f3b180fdc20 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b157fa700 10 client.37139.objecter ms_handle_connect 0x7f3b00062c90 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.099+0000 7f3b17fff700 1 --2- 172.21.15.162:0/3244817528 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3b00062c90 0x7f3b00065150 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 rx=0x7f3b08004dc0 tx=0x7f3b0800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.100+0000 7f3b1cdff700 1 --2- 172.21.15.162:0/3244817528 >> v2:172.21.15.181:6804/2212296748 conn(0x7f3b000713f0 0x7f3b00073890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.100+0000 7f3b157fa700 10 client.37139.objecter ms_handle_connect 0x7f3b000713f0 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.100+0000 7f3b1cdff700 1 --2- 172.21.15.162:0/3244817528 >> v2:172.21.15.181:6804/2212296748 conn(0x7f3b000713f0 0x7f3b00073890 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.101+0000 7f3b157fa700 1 -- 172.21.15.162:0/3244817528 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3b00073f50 con 0x7f3b000713f0 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.101+0000 7f3b157fa700 10 client.37139.objecter ms_dispatch 0x7f3b18001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.101+0000 7f3b157fa700 10 client.37139.objecter _finish_command 1 = system:0 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.101+0000 7f3b157fa700 20 client.37139.objecter put_session s=0x7f3b00070d40 osd=5 4 2024-02-13T00:44:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.101+0000 7f3b157fa700 15 client.37139.objecter _session_command_op_remove 5 1 2024-02-13T00:44:21.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:21.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.117+0000 7f3b1e862700 10 client.37139.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-02-13T00:44:21.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.117+0000 7f3b1e862700 15 client.37139.objecter _session_command_op_assign -1 2 2024-02-13T00:44:21.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.117+0000 7f3b1e862700 20 client.37139.objecter _get_session s=0x7f3b00070d40 osd=5 3 2024-02-13T00:44:21.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.117+0000 7f3b1e862700 20 client.37139.objecter put_session s=0x7f3b00070d40 osd=5 3 2024-02-13T00:44:21.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.117+0000 7f3b1e862700 20 client.37139.objecter _get_session s=0x7f3b00070d40 osd=5 3 2024-02-13T00:44:21.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.117+0000 7f3b1e862700 15 client.37139.objecter _session_command_op_remove -1 2 2024-02-13T00:44:21.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.117+0000 7f3b1e862700 20 client.37139.objecter get_session s=0x7f3b00070d40 osd=5 3 2024-02-13T00:44:21.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.117+0000 7f3b1e862700 15 client.37139.objecter _session_command_op_assign 5 2 2024-02-13T00:44:21.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.117+0000 7f3b1e862700 20 client.37139.objecter put_session s=0x7f3b00070d40 osd=5 4 2024-02-13T00:44:21.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.117+0000 7f3b1e862700 10 client.37139.objecter _send_command 2 2024-02-13T00:44:21.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.117+0000 7f3b1e862700 1 -- 172.21.15.162:0/3244817528 --> v2:172.21.15.181:6804/2212296748 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7f3b180475e0 con 0x7f3b000713f0 2024-02-13T00:44:21.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b157fa700 1 -- 172.21.15.162:0/3244817528 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3b180475e0 con 0x7f3b000713f0 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b157fa700 10 client.37139.objecter ms_dispatch 0x7f3b18001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b157fa700 10 client.37139.objecter _finish_command 2 = system:0 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b157fa700 20 client.37139.objecter put_session s=0x7f3b00070d40 osd=5 4 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b157fa700 15 client.37139.objecter _session_command_op_remove 5 2 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 10 librados: watch_flush enter 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 10 librados: watch_flush exit 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 10 client.37139.objecter close_session for osd.5 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 1 -- 172.21.15.162:0/3244817528 >> v2:172.21.15.181:6804/2212296748 conn(0x7f3b000713f0 msgr2=0x7f3b00073890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 1 --2- 172.21.15.162:0/3244817528 >> v2:172.21.15.181:6804/2212296748 conn(0x7f3b000713f0 0x7f3b00073890 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 20 client.37139.objecter put_session s=0x7f3b00070d40 osd=5 1 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 20 client.37139.objecter shutdown clearing up homeless session... 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 10 client.37139.objecter successfully canceled tick 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 1 -- 172.21.15.162:0/3244817528 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3b00062c90 msgr2=0x7f3b00065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 1 --2- 172.21.15.162:0/3244817528 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3b00062c90 0x7f3b00065150 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 rx=0x7f3b08004dc0 tx=0x7f3b0800b040).stop 2024-02-13T00:44:21.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 1 -- 172.21.15.162:0/3244817528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b180fdc20 msgr2=0x7f3b1810e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:21.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 1 --2- 172.21.15.162:0/3244817528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b180fdc20 0x7f3b1810e5a0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 rx=0x7f3b0c009eb0 tx=0x7f3b0c00b040).stop 2024-02-13T00:44:21.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1cdff700 1 -- 172.21.15.162:0/3244817528 reap_dead start 2024-02-13T00:44:21.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 1 -- 172.21.15.162:0/3244817528 shutdown_connections 2024-02-13T00:44:21.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 1 -- 172.21.15.162:0/3244817528 >> 172.21.15.162:0/3244817528 conn(0x7f3b180f3850 msgr2=0x7f3b180fb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:21.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 1 -- 172.21.15.162:0/3244817528 shutdown_connections 2024-02-13T00:44:21.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 1 -- 172.21.15.162:0/3244817528 wait complete. 2024-02-13T00:44:21.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:21.118+0000 7f3b1e862700 1 librados: shutdown 2024-02-13T00:44:21.147 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:21.167 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:21.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:21.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:21.299 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:21.317 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:21.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:21.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:21.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:21.447 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:21.465 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:21.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:21.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:21.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:20.368133+0000 mon.a (mon.0) 2004 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi162-83985-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi162-83985-53"}]': finished 2024-02-13T00:44:21.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:20.368304+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:21.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:20.368389+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e305: 8 total, 8 up, 6 in 2024-02-13T00:44:21.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:20.384603+0000 mon.b (mon.1) 450 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:21.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:20.386074+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:21.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:20.387667+0000 mgr.y (mgr.14544) 289 : cluster [DBG] pgmap v428: 277 pgs: 8 unknown, 32 creating+peering, 1 peering, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-13T00:44:21.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:20.388443+0000 mon.c (mon.2) 300 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-13T00:44:21.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:20.390480+0000 mon.a (mon.0) 2008 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-13T00:44:21.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:20.478175+0000 osd.2 (osd.2) 123 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:44:21.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:20.481080+0000 osd.2 (osd.2) 124 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:44:21.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:20.368133+0000 mon.a (mon.0) 2004 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi162-83985-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi162-83985-53"}]': finished 2024-02-13T00:44:21.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:20.368304+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:21.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:20.368389+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e305: 8 total, 8 up, 6 in 2024-02-13T00:44:21.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:20.384603+0000 mon.b (mon.1) 450 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:21.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:20.386074+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:21.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:20.387667+0000 mgr.y (mgr.14544) 289 : cluster [DBG] pgmap v428: 277 pgs: 8 unknown, 32 creating+peering, 1 peering, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-13T00:44:21.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:20.388443+0000 mon.c (mon.2) 300 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-13T00:44:21.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:20.390480+0000 mon.a (mon.0) 2008 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-13T00:44:21.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:20.478175+0000 osd.2 (osd.2) 123 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:44:21.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:20.481080+0000 osd.2 (osd.2) 124 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:44:21.582 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:21.600 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:20.368133+0000 mon.a (mon.0) 2004 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi162-83985-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi162-83985-53"}]': finished 2024-02-13T00:44:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:20.368304+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:20.368389+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e305: 8 total, 8 up, 6 in 2024-02-13T00:44:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:20.384603+0000 mon.b (mon.1) 450 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:20.386074+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:20.387667+0000 mgr.y (mgr.14544) 289 : cluster [DBG] pgmap v428: 277 pgs: 8 unknown, 32 creating+peering, 1 peering, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-13T00:44:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:20.388443+0000 mon.c (mon.2) 300 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-13T00:44:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:20.390480+0000 mon.a (mon.0) 2008 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-13T00:44:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:20.478175+0000 osd.2 (osd.2) 123 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:44:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:20.481080+0000 osd.2 (osd.2) 124 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:44:22.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.7 reset_purged_snaps_last 2024-02-13T00:44:22.507 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:21.084291+0000 osd.5 (osd.5) 129 : cluster [DBG] 28.3 scrub starts 2024-02-13T00:44:22.507 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:21.087194+0000 osd.5 (osd.5) 130 : cluster [DBG] 28.3 scrub ok 2024-02-13T00:44:22.507 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:21.366655+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:22.508 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:21.366753+0000 mon.a (mon.0) 2010 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-13T00:44:22.508 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:21.366816+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e306: 8 total, 8 up, 6 in 2024-02-13T00:44:22.508 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:21.376699+0000 mon.b (mon.1) 451 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:44:22.508 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:21.377776+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:44:22.508 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:22.042788+0000 osd.5 (osd.5) 131 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:44:22.508 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:22.045908+0000 osd.5 (osd.5) 132 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:44:22.508 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:21.084291+0000 osd.5 (osd.5) 129 : cluster [DBG] 28.3 scrub starts 2024-02-13T00:44:22.508 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:21.087194+0000 osd.5 (osd.5) 130 : cluster [DBG] 28.3 scrub ok 2024-02-13T00:44:22.508 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:21.366655+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:22.508 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:21.366753+0000 mon.a (mon.0) 2010 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-13T00:44:22.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:21.366816+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e306: 8 total, 8 up, 6 in 2024-02-13T00:44:22.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:21.376699+0000 mon.b (mon.1) 451 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:44:22.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:21.377776+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:44:22.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:22.042788+0000 osd.5 (osd.5) 131 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:44:22.509 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:22.045908+0000 osd.5 (osd.5) 132 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:44:22.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:21.084291+0000 osd.5 (osd.5) 129 : cluster [DBG] 28.3 scrub starts 2024-02-13T00:44:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:21.087194+0000 osd.5 (osd.5) 130 : cluster [DBG] 28.3 scrub ok 2024-02-13T00:44:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:21.366655+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:21.366753+0000 mon.a (mon.0) 2010 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-13T00:44:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:21.366816+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e306: 8 total, 8 up, 6 in 2024-02-13T00:44:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:21.376699+0000 mon.b (mon.1) 451 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:44:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:21.377776+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:44:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:22.042788+0000 osd.5 (osd.5) 131 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:44:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:22.045908+0000 osd.5 (osd.5) 132 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:44:22.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 -- 172.21.15.162:0/2921634438 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6f00f8d10 msgr2=0x7fc6f00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:22.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2921634438 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6f00f8d10 0x7fc6f00f90f0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 rx=0x7fc6d8004870 tx=0x7fc6d8009cb0).stop 2024-02-13T00:44:22.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 -- 172.21.15.162:0/2921634438 shutdown_connections 2024-02-13T00:44:22.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2921634438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6f0102270 0x7fc6f0104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:22.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2921634438 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6f00f8d10 0x7fc6f00f90f0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:22.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2921634438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6f00f95e0 0x7fc6f0101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:22.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 -- 172.21.15.162:0/2921634438 >> 172.21.15.162:0/2921634438 conn(0x7fc6f006eb90 msgr2=0x7fc6f00f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:22.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 -- 172.21.15.162:0/2921634438 shutdown_connections 2024-02-13T00:44:22.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 -- 172.21.15.162:0/2921634438 wait complete. 2024-02-13T00:44:23.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 librados: starting msgr at 2024-02-13T00:44:23.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 librados: starting objecter 2024-02-13T00:44:23.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 Processor -- start 2024-02-13T00:44:23.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 -- start start 2024-02-13T00:44:23.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 librados: setting wanted keys 2024-02-13T00:44:23.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.995+0000 7fc6f676e700 1 librados: calling monclient init 2024-02-13T00:44:23.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6f676e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6f00f8d10 0x7fc6f010e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:23.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6f676e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6f00f95e0 0x7fc6f010e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:23.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6f676e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6f0102270 0x7fc6f01082f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6f676e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f01087e0 con 0x7fc6f0102270 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6f676e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f0108950 con 0x7fc6f00f95e0 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6f676e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f0108b10 con 0x7fc6f00f8d10 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6effff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6f00f8d10 0x7fc6f010e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6effff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6f00f8d10 0x7fc6f010e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:32978/0 (socket says 172.21.15.162:32978) 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6effff700 1 -- 172.21.15.162:0/2850704875 learned_addr learned my addr 172.21.15.162:0/2850704875 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6f4d0b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6f0102270 0x7fc6f01082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6ef7fe700 1 --2- 172.21.15.162:0/2850704875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6f00f95e0 0x7fc6f010e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6effff700 1 -- 172.21.15.162:0/2850704875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6f00f95e0 msgr2=0x7fc6f010e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6effff700 1 --2- 172.21.15.162:0/2850704875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6f00f95e0 0x7fc6f010e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6effff700 1 -- 172.21.15.162:0/2850704875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6f0102270 msgr2=0x7fc6f01082f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6effff700 1 --2- 172.21.15.162:0/2850704875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6f0102270 0x7fc6f01082f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:23.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6effff700 1 -- 172.21.15.162:0/2850704875 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d8005040 con 0x7fc6f00f8d10 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6f4d0b700 1 --2- 172.21.15.162:0/2850704875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6f0102270 0x7fc6f01082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6effff700 1 --2- 172.21.15.162:0/2850704875 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6f00f8d10 0x7fc6f010e100 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 rx=0x7fc6d8009eb0 tx=0x7fc6d800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6ed7fa700 10 client.?.objecter ms_handle_connect 0x7fc6f00f8d10 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6ed7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.996+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d800e070 con 0x7fc6f00f8d10 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.997+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6d8013da0 con 0x7fc6f00f8d10 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.997+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f0108d90 con 0x7fc6f00f8d10 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.997+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8019070 con 0x7fc6f00f8d10 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.997+0000 7fc6f676e700 10 client.37145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.997+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f0114f30 con 0x7fc6f00f8d10 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.997+0000 7fc6f676e700 1 librados: init done 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.997+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc6f00475e0 con 0x7fc6f00f8d10 2024-02-13T00:44:23.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.998+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc6d80090d0 con 0x7fc6f00f8d10 2024-02-13T00:44:23.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.998+0000 7fc6ed7fa700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc6dc062c90 0x7fc6dc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:23.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.998+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(307..307 src has 52..307) v4 ==== 8765+0+0 (secure 0 0 0) 0x7fc6d807dfd0 con 0x7fc6f00f8d10 2024-02-13T00:44:23.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.998+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 osd_map(307..307 src has 52..307) v4 2024-02-13T00:44:23.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.998+0000 7fc6ed7fa700 3 client.37145.objecter handle_osd_map got epochs [307,307] > 0 2024-02-13T00:44:23.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.998+0000 7fc6ed7fa700 3 client.37145.objecter handle_osd_map decoding full epoch 307 2024-02-13T00:44:23.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.998+0000 7fc6ef7fe700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc6dc062c90 0x7fc6dc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:23.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.998+0000 7fc6ed7fa700 20 client.37145.objecter dump_active .. 0 homeless 2024-02-13T00:44:23.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.998+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v307) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc6d8080600 con 0x7fc6f00f8d10 2024-02-13T00:44:23.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6ed7fa700 10 client.37145.objecter ms_handle_connect 0x7fc6dc062c90 2024-02-13T00:44:23.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6ef7fe700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc6dc062c90 0x7fc6dc065150 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7fc6f0109a40 tx=0x7fc6e0009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:23.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f676e700 10 client.37145.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:23.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 1 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc6f0028590 0x7fc6f00fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0107170 osd=0 3 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0107170 osd=0 3 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0107170 osd=0 3 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 1 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f0107170 osd=0 3 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 0 1 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0107170 osd=0 4 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f676e700 10 client.37145.objecter _send_command 1 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6f005fb80 con 0x7fc6f0028590 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f4d0b700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc6f0028590 0x7fc6f00fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6ed7fa700 10 client.37145.objecter ms_handle_connect 0x7fc6f0028590 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:22.999+0000 7fc6f4d0b700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc6f0028590 0x7fc6f00fbd00 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.000+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc6f005fb80 con 0x7fc6f0028590 2024-02-13T00:44:23.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.000+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:44:23.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.000+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 1 = system:0 2024-02-13T00:44:23.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.000+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f0107170 osd=0 4 2024-02-13T00:44:23.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.000+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 0 1 2024-02-13T00:44:23.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.014+0000 7fc6f676e700 10 client.37145.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:23.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.014+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 2 2024-02-13T00:44:23.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.014+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0107170 osd=0 3 2024-02-13T00:44:23.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.014+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0107170 osd=0 3 2024-02-13T00:44:23.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.014+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0107170 osd=0 3 2024-02-13T00:44:23.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.014+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 2 2024-02-13T00:44:23.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.014+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f0107170 osd=0 3 2024-02-13T00:44:23.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.014+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 0 2 2024-02-13T00:44:23.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.014+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0107170 osd=0 4 2024-02-13T00:44:23.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.014+0000 7fc6f676e700 10 client.37145.objecter _send_command 2 2024-02-13T00:44:23.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.014+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6f005a0e0 con 0x7fc6f0028590 2024-02-13T00:44:23.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc6f005a0e0 con 0x7fc6f0028590 2024-02-13T00:44:23.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:23.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f0107170 osd=0 4 2024-02-13T00:44:23.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 0 2 2024-02-13T00:44:23.020 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:44:23.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6f676e700 10 client.37145.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:23.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 3 2024-02-13T00:44:23.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6805/150599780 conn(0x7fc6f0027be0 0x7fc6f0027fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:23.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f00fd370 osd=1 3 2024-02-13T00:44:23.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f00fd370 osd=1 3 2024-02-13T00:44:23.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f00fd370 osd=1 3 2024-02-13T00:44:23.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 3 2024-02-13T00:44:23.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f00fd370 osd=1 3 2024-02-13T00:44:23.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 1 3 2024-02-13T00:44:23.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f00fd370 osd=1 4 2024-02-13T00:44:23.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6f676e700 10 client.37145.objecter _send_command 3 2024-02-13T00:44:23.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6f00fd4e0 con 0x7fc6f0027be0 2024-02-13T00:44:23.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.016+0000 7fc6effff700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6805/150599780 conn(0x7fc6f0027be0 0x7fc6f0027fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:23.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.017+0000 7fc6ed7fa700 10 client.37145.objecter ms_handle_connect 0x7fc6f0027be0 2024-02-13T00:44:23.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.017+0000 7fc6effff700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6805/150599780 conn(0x7fc6f0027be0 0x7fc6f0027fc0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:23.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.017+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc6f00fd4e0 con 0x7fc6f0027be0 2024-02-13T00:44:23.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.017+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:44:23.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.017+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 3 = system:0 2024-02-13T00:44:23.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.017+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f00fd370 osd=1 4 2024-02-13T00:44:23.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.017+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 1 3 2024-02-13T00:44:23.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.032+0000 7fc6f676e700 10 client.37145.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:23.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.032+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 4 2024-02-13T00:44:23.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.032+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f00fd370 osd=1 3 2024-02-13T00:44:23.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.032+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f00fd370 osd=1 3 2024-02-13T00:44:23.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.032+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f00fd370 osd=1 3 2024-02-13T00:44:23.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.032+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 4 2024-02-13T00:44:23.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.032+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f00fd370 osd=1 3 2024-02-13T00:44:23.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.032+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 1 4 2024-02-13T00:44:23.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.032+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f00fd370 osd=1 4 2024-02-13T00:44:23.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.032+0000 7fc6f676e700 10 client.37145.objecter _send_command 4 2024-02-13T00:44:23.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.032+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6f0027650 con 0x7fc6f0027be0 2024-02-13T00:44:23.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc6f0027650 con 0x7fc6f0027be0 2024-02-13T00:44:23.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:23.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f00fd370 osd=1 4 2024-02-13T00:44:23.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 1 4 2024-02-13T00:44:23.038 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:44:23.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6f676e700 10 client.37145.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:23.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 5 2024-02-13T00:44:23.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc6f00fced0 0x7fc6f011c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:23.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0027a60 osd=2 3 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0027a60 osd=2 3 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0027a60 osd=2 3 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 5 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f0027a60 osd=2 3 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 2 5 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0027a60 osd=2 4 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6f676e700 10 client.37145.objecter _send_command 5 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6f011eec0 con 0x7fc6f00fced0 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6ef7fe700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc6f00fced0 0x7fc6f011c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6ef7fe700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc6f00fced0 0x7fc6f011c270 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.034+0000 7fc6ed7fa700 10 client.37145.objecter ms_handle_connect 0x7fc6f00fced0 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.035+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc6f011eec0 con 0x7fc6f00fced0 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.035+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.035+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 5 = system:0 2024-02-13T00:44:23.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.035+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f0027a60 osd=2 4 2024-02-13T00:44:23.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.035+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 2 5 2024-02-13T00:44:23.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.049+0000 7fc6f676e700 10 client.37145.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:23.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.049+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 6 2024-02-13T00:44:23.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.049+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0027a60 osd=2 3 2024-02-13T00:44:23.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.049+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0027a60 osd=2 3 2024-02-13T00:44:23.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.049+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0027a60 osd=2 3 2024-02-13T00:44:23.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.049+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 6 2024-02-13T00:44:23.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.049+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f0027a60 osd=2 3 2024-02-13T00:44:23.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.049+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 2 6 2024-02-13T00:44:23.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.049+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0027a60 osd=2 4 2024-02-13T00:44:23.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.049+0000 7fc6f676e700 10 client.37145.objecter _send_command 6 2024-02-13T00:44:23.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.049+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6f00fc490 con 0x7fc6f00fced0 2024-02-13T00:44:23.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc6f00fc490 con 0x7fc6f00fced0 2024-02-13T00:44:23.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:23.054 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:44:23.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f0027a60 osd=2 4 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 2 6 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f676e700 10 client.37145.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 7 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6813/604971011 conn(0x7fc6f011d8e0 0x7fc6f011dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f01198b0 osd=3 3 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f01198b0 osd=3 3 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f01198b0 osd=3 3 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 7 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f01198b0 osd=3 3 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 3 7 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f01198b0 osd=3 4 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f676e700 10 client.37145.objecter _send_command 7 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6f011a050 con 0x7fc6f011d8e0 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f4d0b700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6813/604971011 conn(0x7fc6f011d8e0 0x7fc6f011dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:23.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6ed7fa700 10 client.37145.objecter ms_handle_connect 0x7fc6f011d8e0 2024-02-13T00:44:23.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.051+0000 7fc6f4d0b700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6813/604971011 conn(0x7fc6f011d8e0 0x7fc6f011dd40 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:23.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.052+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc6f011a050 con 0x7fc6f011d8e0 2024-02-13T00:44:23.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.052+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:44:23.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.052+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 7 = system:0 2024-02-13T00:44:23.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.052+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f01198b0 osd=3 4 2024-02-13T00:44:23.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.052+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 3 7 2024-02-13T00:44:23.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.066+0000 7fc6f676e700 10 client.37145.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:23.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.066+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 8 2024-02-13T00:44:23.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.066+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f01198b0 osd=3 3 2024-02-13T00:44:23.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.066+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f01198b0 osd=3 3 2024-02-13T00:44:23.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.066+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f01198b0 osd=3 3 2024-02-13T00:44:23.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.066+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 8 2024-02-13T00:44:23.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.066+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f01198b0 osd=3 3 2024-02-13T00:44:23.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.066+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 3 8 2024-02-13T00:44:23.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.066+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f01198b0 osd=3 4 2024-02-13T00:44:23.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.066+0000 7fc6f676e700 10 client.37145.objecter _send_command 8 2024-02-13T00:44:23.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.066+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6f011ea30 con 0x7fc6f011d8e0 2024-02-13T00:44:23.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc6f011ea30 con 0x7fc6f011d8e0 2024-02-13T00:44:23.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:23.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f01198b0 osd=3 4 2024-02-13T00:44:23.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 3 8 2024-02-13T00:44:23.074 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:44:23.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6f676e700 10 client.37145.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:23.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 9 2024-02-13T00:44:23.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc6f0121040 0x7fc6f0123430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f011ea30 osd=4 3 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f011ea30 osd=4 3 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f011ea30 osd=4 3 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 9 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f011ea30 osd=4 3 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 4 9 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f011ea30 osd=4 4 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6f676e700 10 client.37145.objecter _send_command 9 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6f0126180 con 0x7fc6f0121040 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6effff700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc6f0121040 0x7fc6f0123430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6ed7fa700 10 client.37145.objecter ms_handle_connect 0x7fc6f0121040 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.068+0000 7fc6effff700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc6f0121040 0x7fc6f0123430 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.069+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc6f0126180 con 0x7fc6f0121040 2024-02-13T00:44:23.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.069+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:44:23.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.069+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 9 = system:0 2024-02-13T00:44:23.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.069+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f011ea30 osd=4 4 2024-02-13T00:44:23.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.069+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 4 9 2024-02-13T00:44:23.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.084+0000 7fc6f676e700 10 client.37145.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:23.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.084+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 10 2024-02-13T00:44:23.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.084+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f011ea30 osd=4 3 2024-02-13T00:44:23.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.084+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f011ea30 osd=4 3 2024-02-13T00:44:23.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.084+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f011ea30 osd=4 3 2024-02-13T00:44:23.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.084+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 10 2024-02-13T00:44:23.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.084+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f011ea30 osd=4 3 2024-02-13T00:44:23.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.084+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 4 10 2024-02-13T00:44:23.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.084+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f011ea30 osd=4 4 2024-02-13T00:44:23.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.084+0000 7fc6f676e700 10 client.37145.objecter _send_command 10 2024-02-13T00:44:23.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.084+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6f0126180 con 0x7fc6f0121040 2024-02-13T00:44:23.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.086+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc6f0126180 con 0x7fc6f0121040 2024-02-13T00:44:23.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.086+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:23.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.086+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.086+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f011ea30 osd=4 4 2024-02-13T00:44:23.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.086+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 4 10 2024-02-13T00:44:23.090 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:44:23.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6f676e700 10 client.37145.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:23.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 11 2024-02-13T00:44:23.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc6f0123f70 0x7fc6f01243d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:23.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0126180 osd=5 3 2024-02-13T00:44:23.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0126180 osd=5 3 2024-02-13T00:44:23.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0126180 osd=5 3 2024-02-13T00:44:23.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 11 2024-02-13T00:44:23.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f0126180 osd=5 3 2024-02-13T00:44:23.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 5 11 2024-02-13T00:44:23.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0126180 osd=5 4 2024-02-13T00:44:23.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6f676e700 10 client.37145.objecter _send_command 11 2024-02-13T00:44:23.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6f0129530 con 0x7fc6f0123f70 2024-02-13T00:44:23.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6ef7fe700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc6f0123f70 0x7fc6f01243d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:23.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6ef7fe700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc6f0123f70 0x7fc6f01243d0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:23.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.087+0000 7fc6ed7fa700 10 client.37145.objecter ms_handle_connect 0x7fc6f0123f70 2024-02-13T00:44:23.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.088+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc6f0129530 con 0x7fc6f0123f70 2024-02-13T00:44:23.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.088+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:44:23.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.088+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 11 = system:0 2024-02-13T00:44:23.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.088+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f0126180 osd=5 4 2024-02-13T00:44:23.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.088+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 5 11 2024-02-13T00:44:23.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.102+0000 7fc6f676e700 10 client.37145.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:23.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.102+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 12 2024-02-13T00:44:23.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.102+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0126180 osd=5 3 2024-02-13T00:44:23.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.102+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0126180 osd=5 3 2024-02-13T00:44:23.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.102+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0126180 osd=5 3 2024-02-13T00:44:23.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.102+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 12 2024-02-13T00:44:23.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.102+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f0126180 osd=5 3 2024-02-13T00:44:23.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.102+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 5 12 2024-02-13T00:44:23.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.102+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0126180 osd=5 4 2024-02-13T00:44:23.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.102+0000 7fc6f676e700 10 client.37145.objecter _send_command 12 2024-02-13T00:44:23.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.102+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6f0124980 con 0x7fc6f0123f70 2024-02-13T00:44:23.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc6f0124980 con 0x7fc6f0123f70 2024-02-13T00:44:23.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:23.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f0126180 osd=5 4 2024-02-13T00:44:23.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 5 12 2024-02-13T00:44:23.108 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:44:23.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6f676e700 10 client.37145.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:23.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 13 2024-02-13T00:44:23.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc6f0125030 0x7fc6f0125490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:23.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0124980 osd=6 3 2024-02-13T00:44:23.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0124980 osd=6 3 2024-02-13T00:44:23.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0124980 osd=6 3 2024-02-13T00:44:23.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 13 2024-02-13T00:44:23.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f0124980 osd=6 3 2024-02-13T00:44:23.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 6 13 2024-02-13T00:44:23.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0124980 osd=6 4 2024-02-13T00:44:23.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6f676e700 10 client.37145.objecter _send_command 13 2024-02-13T00:44:23.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.104+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6f0125b30 con 0x7fc6f0125030 2024-02-13T00:44:23.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.105+0000 7fc6f4d0b700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc6f0125030 0x7fc6f0125490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:23.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.105+0000 7fc6f4d0b700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc6f0125030 0x7fc6f0125490 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:23.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.105+0000 7fc6ed7fa700 10 client.37145.objecter ms_handle_connect 0x7fc6f0125030 2024-02-13T00:44:23.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.106+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc6f0125b30 con 0x7fc6f0125030 2024-02-13T00:44:23.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.106+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:44:23.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.106+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 13 = system:0 2024-02-13T00:44:23.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.106+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f0124980 osd=6 4 2024-02-13T00:44:23.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.106+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 6 13 2024-02-13T00:44:23.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.120+0000 7fc6f676e700 10 client.37145.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:23.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.120+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 14 2024-02-13T00:44:23.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.120+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0124980 osd=6 3 2024-02-13T00:44:23.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.120+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0124980 osd=6 3 2024-02-13T00:44:23.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.120+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0124980 osd=6 3 2024-02-13T00:44:23.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.120+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 14 2024-02-13T00:44:23.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.120+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f0124980 osd=6 3 2024-02-13T00:44:23.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.120+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 6 14 2024-02-13T00:44:23.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.120+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0124980 osd=6 4 2024-02-13T00:44:23.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.120+0000 7fc6f676e700 10 client.37145.objecter _send_command 14 2024-02-13T00:44:23.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.120+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6f0125b30 con 0x7fc6f0125030 2024-02-13T00:44:23.125 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:44:23.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc6f0125b30 con 0x7fc6f0125030 2024-02-13T00:44:23.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:23.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f0124980 osd=6 4 2024-02-13T00:44:23.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 6 14 2024-02-13T00:44:23.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6f676e700 10 client.37145.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:23.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 15 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc6f012a8d0 0x7fc6f012cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0125b30 osd=7 3 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0125b30 osd=7 3 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0125b30 osd=7 3 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 15 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f0125b30 osd=7 3 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 7 15 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0125b30 osd=7 4 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6f676e700 10 client.37145.objecter _send_command 15 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.122+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6f012fb10 con 0x7fc6f012a8d0 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.123+0000 7fc6effff700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc6f012a8d0 0x7fc6f012cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.123+0000 7fc6ed7fa700 10 client.37145.objecter ms_handle_connect 0x7fc6f012a8d0 2024-02-13T00:44:23.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.123+0000 7fc6effff700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc6f012a8d0 0x7fc6f012cd40 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:23.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.124+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc6f012fb10 con 0x7fc6f012a8d0 2024-02-13T00:44:23.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.124+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:44:23.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.124+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 15 = system:0 2024-02-13T00:44:23.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.124+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f0125b30 osd=7 4 2024-02-13T00:44:23.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.124+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 7 15 2024-02-13T00:44:23.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.137+0000 7fc6f676e700 10 client.37145.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:23.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.137+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign -1 16 2024-02-13T00:44:23.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.137+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0125b30 osd=7 3 2024-02-13T00:44:23.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.137+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0125b30 osd=7 3 2024-02-13T00:44:23.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.137+0000 7fc6f676e700 20 client.37145.objecter _get_session s=0x7fc6f0125b30 osd=7 3 2024-02-13T00:44:23.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.137+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_remove -1 16 2024-02-13T00:44:23.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.137+0000 7fc6f676e700 20 client.37145.objecter get_session s=0x7fc6f0125b30 osd=7 3 2024-02-13T00:44:23.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.137+0000 7fc6f676e700 15 client.37145.objecter _session_command_op_assign 7 16 2024-02-13T00:44:23.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.137+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0125b30 osd=7 4 2024-02-13T00:44:23.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.137+0000 7fc6f676e700 10 client.37145.objecter _send_command 16 2024-02-13T00:44:23.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.137+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc6f012fb10 con 0x7fc6f012a8d0 2024-02-13T00:44:23.142 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:44:23.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.139+0000 7fc6ed7fa700 1 -- 172.21.15.162:0/2850704875 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc6f012fb10 con 0x7fc6f012a8d0 2024-02-13T00:44:23.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.139+0000 7fc6ed7fa700 10 client.37145.objecter ms_dispatch 0x7fc6f0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:23.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.139+0000 7fc6ed7fa700 10 client.37145.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.139+0000 7fc6ed7fa700 20 client.37145.objecter put_session s=0x7fc6f0125b30 osd=7 4 2024-02-13T00:44:23.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.139+0000 7fc6ed7fa700 15 client.37145.objecter _session_command_op_remove 7 16 2024-02-13T00:44:23.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 10 librados: watch_flush enter 2024-02-13T00:44:23.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 10 librados: watch_flush exit 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 10 client.37145.objecter close_session for osd.0 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc6f0028590 msgr2=0x7fc6f00fbd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc6f0028590 0x7fc6f00fbd00 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0107170 osd=0 1 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 10 client.37145.objecter close_session for osd.1 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6805/150599780 conn(0x7fc6f0027be0 msgr2=0x7fc6f0027fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6805/150599780 conn(0x7fc6f0027be0 0x7fc6f0027fc0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f00fd370 osd=1 1 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 10 client.37145.objecter close_session for osd.2 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc6f00fced0 msgr2=0x7fc6f011c270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc6f00fced0 0x7fc6f011c270 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0027a60 osd=2 1 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 10 client.37145.objecter close_session for osd.3 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6813/604971011 conn(0x7fc6f011d8e0 msgr2=0x7fc6f011dd40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:23.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6813/604971011 conn(0x7fc6f011d8e0 0x7fc6f011dd40 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f01198b0 osd=3 1 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 10 client.37145.objecter close_session for osd.4 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc6f0121040 msgr2=0x7fc6f0123430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc6f0121040 0x7fc6f0123430 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f011ea30 osd=4 1 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 10 client.37145.objecter close_session for osd.5 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc6f0123f70 msgr2=0x7fc6f01243d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6804/2212296748 conn(0x7fc6f0123f70 0x7fc6f01243d0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0126180 osd=5 1 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 10 client.37145.objecter close_session for osd.6 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc6f0125030 msgr2=0x7fc6f0125490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc6f0125030 0x7fc6f0125490 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0124980 osd=6 1 2024-02-13T00:44:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 10 client.37145.objecter close_session for osd.7 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc6f012a8d0 msgr2=0x7fc6f012cd40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc6f012a8d0 0x7fc6f012cd40 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f4d0b700 1 -- 172.21.15.162:0/2850704875 reap_dead start 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f4d0b700 1 -- 172.21.15.162:0/2850704875 reap_dead start 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 20 client.37145.objecter put_session s=0x7fc6f0125b30 osd=7 1 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 20 client.37145.objecter shutdown clearing up homeless session... 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 10 client.37145.objecter successfully canceled tick 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f4d0b700 1 -- 172.21.15.162:0/2850704875 reap_dead start 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f4d0b700 1 -- 172.21.15.162:0/2850704875 reap_dead start 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc6dc062c90 msgr2=0x7fc6dc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc6dc062c90 0x7fc6dc065150 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7fc6f0109a40 tx=0x7fc6e0009040).stop 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6f00f8d10 msgr2=0x7fc6f010e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6f00f8d10 0x7fc6f010e100 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 rx=0x7fc6d8009eb0 tx=0x7fc6d800b040).stop 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 shutdown_connections 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6f00f8d10 0x7fc6f010e100 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:23.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 --2- 172.21.15.162:0/2850704875 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc6dc062c90 0x7fc6dc065150 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:23.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 >> 172.21.15.162:0/2850704875 conn(0x7fc6f006eb90 msgr2=0x7fc6f0102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:23.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 shutdown_connections 2024-02-13T00:44:23.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 -- 172.21.15.162:0/2850704875 wait complete. 2024-02-13T00:44:23.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:23.140+0000 7fc6f676e700 1 librados: shutdown 2024-02-13T00:44:23.148 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.148 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.148 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.148 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.148 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.148 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.148 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.148 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:44:23.353 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:23.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:23.457 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:23.485 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:23.505 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:23.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:22.375823+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:22.375977+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e307: 8 total, 8 up, 6 in 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:22.377695+0000 mon.b (mon.1) 452 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:22.379385+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:22.384381+0000 mon.a (mon.0) 2016 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:22.388144+0000 mgr.y (mgr.14544) 290 : cluster [DBG] pgmap v431: 269 pgs: 32 creating+peering, 1 peering, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:22.487341+0000 osd.2 (osd.2) 125 : cluster [DBG] 28.14 scrub starts 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:22.488349+0000 osd.2 (osd.2) 126 : cluster [DBG] 28.14 scrub ok 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:22.999244+0000 mon.c (mon.2) 301 : audit [DBG] from='client.? 172.21.15.162:0/2850704875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:23.377854+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:23.377943+0000 mon.a (mon.0) 2018 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-83985-53"}]': finished 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:23.378006+0000 mon.a (mon.0) 2019 : cluster [DBG] osdmap e308: 8 total, 8 up, 6 in 2024-02-13T00:44:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:23.379324+0000 mon.a (mon.0) 2020 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:22.375823+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:22.375977+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e307: 8 total, 8 up, 6 in 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:22.377695+0000 mon.b (mon.1) 452 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:22.379385+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:22.384381+0000 mon.a (mon.0) 2016 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:22.388144+0000 mgr.y (mgr.14544) 290 : cluster [DBG] pgmap v431: 269 pgs: 32 creating+peering, 1 peering, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:22.487341+0000 osd.2 (osd.2) 125 : cluster [DBG] 28.14 scrub starts 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:22.488349+0000 osd.2 (osd.2) 126 : cluster [DBG] 28.14 scrub ok 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:22.999244+0000 mon.c (mon.2) 301 : audit [DBG] from='client.? 172.21.15.162:0/2850704875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:23.377854+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:23.377943+0000 mon.a (mon.0) 2018 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-83985-53"}]': finished 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:23.378006+0000 mon.a (mon.0) 2019 : cluster [DBG] osdmap e308: 8 total, 8 up, 6 in 2024-02-13T00:44:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:23.379324+0000 mon.a (mon.0) 2020 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:23.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:23.638 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:22.375823+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:44:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:22.375977+0000 mon.a (mon.0) 2014 : cluster [DBG] osdmap e307: 8 total, 8 up, 6 in 2024-02-13T00:44:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:22.377695+0000 mon.b (mon.1) 452 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:44:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:22.379385+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:44:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:22.384381+0000 mon.a (mon.0) 2016 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:22.388144+0000 mgr.y (mgr.14544) 290 : cluster [DBG] pgmap v431: 269 pgs: 32 creating+peering, 1 peering, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-13T00:44:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:22.487341+0000 osd.2 (osd.2) 125 : cluster [DBG] 28.14 scrub starts 2024-02-13T00:44:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:22.488349+0000 osd.2 (osd.2) 126 : cluster [DBG] 28.14 scrub ok 2024-02-13T00:44:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:22.999244+0000 mon.c (mon.2) 301 : audit [DBG] from='client.? 172.21.15.162:0/2850704875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:23.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:23.377854+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:44:23.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:23.377943+0000 mon.a (mon.0) 2018 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-83985-53"}]': finished 2024-02-13T00:44:23.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:23.378006+0000 mon.a (mon.0) 2019 : cluster [DBG] osdmap e308: 8 total, 8 up, 6 in 2024-02-13T00:44:23.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:23.379324+0000 mon.a (mon.0) 2020 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-83985-53"}]: dispatch 2024-02-13T00:44:24.396 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:23.380848+0000 mon.b (mon.1) 453 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:44:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:23.380848+0000 mon.b (mon.1) 453 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:44:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:23.383302+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:44:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:24.378495+0000 mon.a (mon.0) 2022 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-83985-53"}]': finished 2024-02-13T00:44:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:24.378572+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-13T00:44:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:24.378649+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e309: 8 total, 8 up, 6 in 2024-02-13T00:44:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:24.388902+0000 mon.c (mon.2) 302 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-13T00:44:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:24.389826+0000 mon.a (mon.0) 2025 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-13T00:44:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:23.380848+0000 mon.b (mon.1) 453 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:44:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:23.383302+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:44:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:24.378495+0000 mon.a (mon.0) 2022 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-83985-53"}]': finished 2024-02-13T00:44:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:24.378572+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-13T00:44:24.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:24.378649+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e309: 8 total, 8 up, 6 in 2024-02-13T00:44:24.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:24.388902+0000 mon.c (mon.2) 302 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-13T00:44:24.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:24.389826+0000 mon.a (mon.0) 2025 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-13T00:44:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:23.383302+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:44:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:24.378495+0000 mon.a (mon.0) 2022 : audit [INF] from='client.? 172.21.15.162:0/3998292204' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-83985-53"}]': finished 2024-02-13T00:44:24.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:24.378572+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-39","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-13T00:44:24.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:24.378649+0000 mon.a (mon.0) 2024 : cluster [DBG] osdmap e309: 8 total, 8 up, 6 in 2024-02-13T00:44:24.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:24.388902+0000 mon.c (mon.2) 302 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-13T00:44:24.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:24.389826+0000 mon.a (mon.0) 2025 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-13T00:44:24.988 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:24.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:24.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:25.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:25.102 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:44:25.102 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:44:25.122 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:25.404 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:24.192546+0000 osd.1 (osd.1) 121 : cluster [DBG] 4.a scrub starts 2024-02-13T00:44:25.404 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:24.196209+0000 osd.1 (osd.1) 122 : cluster [DBG] 4.a scrub ok 2024-02-13T00:44:25.404 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:24.388539+0000 mgr.y (mgr.14544) 291 : cluster [DBG] pgmap v434: 268 pgs: 32 creating+peering, 1 peering, 235 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:25.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:44:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:24.192546+0000 osd.1 (osd.1) 121 : cluster [DBG] 4.a scrub starts 2024-02-13T00:44:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:24.196209+0000 osd.1 (osd.1) 122 : cluster [DBG] 4.a scrub ok 2024-02-13T00:44:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:24.388539+0000 mgr.y (mgr.14544) 291 : cluster [DBG] pgmap v434: 268 pgs: 32 creating+peering, 1 peering, 235 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:24.400103+0000 mon.c (mon.2) 303 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:24.400421+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:24.400877+0000 mon.c (mon.2) 304 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:24.401068+0000 mon.a (mon.0) 2027 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:25.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:24.401434+0000 mon.c (mon.2) 305 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-83985-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:25.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:24.401695+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-83985-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:25.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:24.935362+0000 mon.b (mon.1) 454 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:25.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:25.386310+0000 mon.a (mon.0) 2029 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-13T00:44:25.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:25.386394+0000 mon.a (mon.0) 2030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-83985-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:25.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:25.386464+0000 mon.a (mon.0) 2031 : cluster [DBG] osdmap e310: 8 total, 8 up, 6 in 2024-02-13T00:44:25.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:24.400103+0000 mon.c (mon.2) 303 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:25.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:24.400421+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:25.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:24.400877+0000 mon.c (mon.2) 304 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:25.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:24.401068+0000 mon.a (mon.0) 2027 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:25.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:24.401434+0000 mon.c (mon.2) 305 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-83985-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:25.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:24.401695+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-83985-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:25.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:24.935362+0000 mon.b (mon.1) 454 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:25.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:25.386310+0000 mon.a (mon.0) 2029 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-13T00:44:25.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:25.386394+0000 mon.a (mon.0) 2030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-83985-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:25.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:25.386464+0000 mon.a (mon.0) 2031 : cluster [DBG] osdmap e310: 8 total, 8 up, 6 in 2024-02-13T00:44:25.801 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:24.192546+0000 osd.1 (osd.1) 121 : cluster [DBG] 4.a scrub starts 2024-02-13T00:44:25.801 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:24.196209+0000 osd.1 (osd.1) 122 : cluster [DBG] 4.a scrub ok 2024-02-13T00:44:25.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:24.388539+0000 mgr.y (mgr.14544) 291 : cluster [DBG] pgmap v434: 268 pgs: 32 creating+peering, 1 peering, 235 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:25.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:24.400103+0000 mon.c (mon.2) 303 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:25.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:24.400421+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:25.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:24.400877+0000 mon.c (mon.2) 304 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:25.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:24.401068+0000 mon.a (mon.0) 2027 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:25.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:24.401434+0000 mon.c (mon.2) 305 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-83985-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:25.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:24.401695+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-83985-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:25.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:24.935362+0000 mon.b (mon.1) 454 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:25.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:25.386310+0000 mon.a (mon.0) 2029 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-13T00:44:25.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:25.386394+0000 mon.a (mon.0) 2030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-83985-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:25.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:25.386464+0000 mon.a (mon.0) 2031 : cluster [DBG] osdmap e310: 8 total, 8 up, 6 in 2024-02-13T00:44:25.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-13T00:44:26.092 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:44:26.371 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:26.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:26.375 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:44:26] "GET /metrics HTTP/1.1" 200 186450 "" "Prometheus/2.33.4" 2024-02-13T00:44:26.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:26.503 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:26.532 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:26.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:26.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:26.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:44:26.636 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:26.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:26.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.653+0000 7fbfe4055700 1 -- 172.21.15.162:0/1228217803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfdc100a20 msgr2=0x7fbfdc071e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:26.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.653+0000 7fbfe4055700 1 --2- 172.21.15.162:0/1228217803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfdc100a20 0x7fbfdc071e30 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 rx=0x7fbfcc0047b0 tx=0x7fbfcc009cb0).stop 2024-02-13T00:44:26.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.653+0000 7fbfe4055700 1 -- 172.21.15.162:0/1228217803 shutdown_connections 2024-02-13T00:44:26.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.653+0000 7fbfe4055700 1 --2- 172.21.15.162:0/1228217803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfdc072510 0x7fbfdc10b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:26.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.653+0000 7fbfe4055700 1 --2- 172.21.15.162:0/1228217803 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbfdc100150 0x7fbfdc100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.653+0000 7fbfe4055700 1 --2- 172.21.15.162:0/1228217803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfdc100a20 0x7fbfdc071e30 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.653+0000 7fbfe4055700 1 -- 172.21.15.162:0/1228217803 >> 172.21.15.162:0/1228217803 conn(0x7fbfdc06d8d0 msgr2=0x7fbfdc06fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.653+0000 7fbfe4055700 1 -- 172.21.15.162:0/1228217803 shutdown_connections 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.653+0000 7fbfe4055700 1 -- 172.21.15.162:0/1228217803 wait complete. 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe4055700 1 librados: starting msgr at 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe4055700 1 librados: starting objecter 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe4055700 1 Processor -- start 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe4055700 1 -- start start 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe4055700 1 librados: setting wanted keys 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe4055700 1 librados: calling monclient init 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe4055700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfdc072510 0x7fbfdc1147e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe4055700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbfdc100150 0x7fbfdc114cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe4055700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfdc100a20 0x7fbfdc10e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:26.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe4055700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc10eee0 con 0x7fbfdc072510 2024-02-13T00:44:26.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe4055700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc10f020 con 0x7fbfdc100a20 2024-02-13T00:44:26.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe4055700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc10f1e0 con 0x7fbfdc100150 2024-02-13T00:44:26.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe25f2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfdc100a20 0x7fbfdc10e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:26.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe1df1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfdc072510 0x7fbfdc1147e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:26.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe25f2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfdc100a20 0x7fbfdc10e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:51538/0 (socket says 172.21.15.162:51538) 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.654+0000 7fbfe25f2700 1 -- 172.21.15.162:0/1110424101 learned_addr learned my addr 172.21.15.162:0/1110424101 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfe25f2700 1 -- 172.21.15.162:0/1110424101 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbfdc100150 msgr2=0x7fbfdc114cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfe25f2700 1 --2- 172.21.15.162:0/1110424101 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbfdc100150 0x7fbfdc114cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfe25f2700 1 -- 172.21.15.162:0/1110424101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfdc072510 msgr2=0x7fbfdc1147e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfe25f2700 1 --2- 172.21.15.162:0/1110424101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfdc072510 0x7fbfdc1147e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfe25f2700 1 -- 172.21.15.162:0/1110424101 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd8005040 con 0x7fbfdc100a20 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfe25f2700 1 --2- 172.21.15.162:0/1110424101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfdc100a20 0x7fbfdc10e9f0 secure :-1 s=READY pgs=4851 cs=0 l=1 rev1=1 rx=0x7fbfd400ec20 tx=0x7fbfd400b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfd2ffd700 10 client.?.objecter ms_handle_connect 0x7fbfdc100a20 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfd2ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfd2ffd700 1 -- 172.21.15.162:0/1110424101 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4009950 con 0x7fbfdc100a20 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfd2ffd700 1 -- 172.21.15.162:0/1110424101 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbfd400f040 con 0x7fbfdc100a20 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfe4055700 1 -- 172.21.15.162:0/1110424101 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfcc005040 con 0x7fbfdc100a20 2024-02-13T00:44:26.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfd2ffd700 1 -- 172.21.15.162:0/1110424101 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd400c6e0 con 0x7fbfdc100a20 2024-02-13T00:44:26.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfe4055700 10 client.44224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:26.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfe4055700 1 -- 172.21.15.162:0/1110424101 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc10f840 con 0x7fbfdc100a20 2024-02-13T00:44:26.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.655+0000 7fbfe4055700 1 librados: init done 2024-02-13T00:44:26.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.656+0000 7fbfd2ffd700 1 -- 172.21.15.162:0/1110424101 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fbfd400cb10 con 0x7fbfdc100a20 2024-02-13T00:44:26.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.657+0000 7fbfd2ffd700 1 --2- 172.21.15.162:0/1110424101 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbfc8062c90 0x7fbfc8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:26.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.657+0000 7fbfd2ffd700 1 -- 172.21.15.162:0/1110424101 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(311..311 src has 52..311) v4 ==== 8749+0+0 (secure 0 0 0) 0x7fbfd407d210 con 0x7fbfdc100a20 2024-02-13T00:44:26.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.657+0000 7fbfd2ffd700 10 client.44224.objecter ms_dispatch 0x7fbfdc001550 osd_map(311..311 src has 52..311) v4 2024-02-13T00:44:26.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.657+0000 7fbfd2ffd700 3 client.44224.objecter handle_osd_map got epochs [311,311] > 0 2024-02-13T00:44:26.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.657+0000 7fbfd2ffd700 3 client.44224.objecter handle_osd_map decoding full epoch 311 2024-02-13T00:44:26.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.657+0000 7fbfe1df1700 1 --2- 172.21.15.162:0/1110424101 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbfc8062c90 0x7fbfc8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:26.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.657+0000 7fbfd2ffd700 20 client.44224.objecter dump_active .. 0 homeless 2024-02-13T00:44:26.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.657+0000 7fbfd2ffd700 10 client.44224.objecter ms_handle_connect 0x7fbfc8062c90 2024-02-13T00:44:26.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.657+0000 7fbfe1df1700 1 --2- 172.21.15.162:0/1110424101 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbfc8062c90 0x7fbfc8065150 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 rx=0x7fbfd8004870 tx=0x7fbfd800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:26.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.658+0000 7fbfe4055700 1 -- 172.21.15.162:0/1110424101 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfdc0475e0 con 0x7fbfdc100a20 2024-02-13T00:44:26.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.664+0000 7fbfd2ffd700 1 -- 172.21.15.162:0/1110424101 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fbfd4049f60 con 0x7fbfdc100a20 2024-02-13T00:44:26.674 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:26.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:26.865+0000 7fbfe4055700 1 -- 172.21.15.162:0/1110424101 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fbfdc10fed0 con 0x7fbfdc100a20 2024-02-13T00:44:26.885 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:26.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:26.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:25.229510+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.e scrub starts 2024-02-13T00:44:26.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:25.230976+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.e scrub ok 2024-02-13T00:44:26.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:25.391251+0000 mon.c (mon.2) 306 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-83985-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:25.396788+0000 mon.a (mon.0) 2032 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-83985-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:25.409387+0000 mon.b (mon.1) 455 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:25.409844+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:25.410232+0000 mon.b (mon.1) 456 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39"}]: dispatch 2024-02-13T00:44:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:25.410577+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39"}]: dispatch 2024-02-13T00:44:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:25.499373+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.f scrub starts 2024-02-13T00:44:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:25.502307+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.f scrub ok 2024-02-13T00:44:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:25.518423+0000 mgr.y (mgr.14544) 292 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:25.936206+0000 mon.b (mon.1) 457 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:26.633234+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39"}]': finished 2024-02-13T00:44:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:26.633351+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e311: 8 total, 8 up, 6 in 2024-02-13T00:44:26.910 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:25.229510+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.e scrub starts 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:25.230976+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.e scrub ok 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:25.391251+0000 mon.c (mon.2) 306 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-83985-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:25.396788+0000 mon.a (mon.0) 2032 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-83985-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:25.409387+0000 mon.b (mon.1) 455 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:25.409844+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:25.410232+0000 mon.b (mon.1) 456 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39"}]: dispatch 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:25.410577+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39"}]: dispatch 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:25.499373+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.f scrub starts 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:25.502307+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.f scrub ok 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:25.518423+0000 mgr.y (mgr.14544) 292 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:25.936206+0000 mon.b (mon.1) 457 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:26.633234+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39"}]': finished 2024-02-13T00:44:26.911 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:26.633351+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e311: 8 total, 8 up, 6 in 2024-02-13T00:44:26.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:25.229510+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.e scrub starts 2024-02-13T00:44:26.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:25.230976+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.e scrub ok 2024-02-13T00:44:26.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:25.391251+0000 mon.c (mon.2) 306 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-83985-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:26.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:25.396788+0000 mon.a (mon.0) 2032 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-83985-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:26.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:25.409387+0000 mon.b (mon.1) 455 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:26.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:25.409844+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:26.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:25.410232+0000 mon.b (mon.1) 456 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39"}]: dispatch 2024-02-13T00:44:26.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:25.410577+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39"}]: dispatch 2024-02-13T00:44:26.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:25.499373+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.f scrub starts 2024-02-13T00:44:26.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:25.502307+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.f scrub ok 2024-02-13T00:44:26.981 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:25.518423+0000 mgr.y (mgr.14544) 292 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:26.981 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:25.936206+0000 mon.b (mon.1) 457 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:26.981 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:26.633234+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-39"}]': finished 2024-02-13T00:44:26.981 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:26.633351+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e311: 8 total, 8 up, 6 in 2024-02-13T00:44:26.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:27.015 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:27.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.401+0000 7fbfd2ffd700 1 -- 172.21.15.162:0/1110424101 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v312) v1 ==== 101+0+0 (secure 0 0 0) 0x7fbfd40504b0 con 0x7fbfdc100a20 2024-02-13T00:44:27.407 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [==========] Running 72 tests from 4 test suites. 2024-02-13T00:44:27.407 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-02-13T00:44:27.407 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: seed 84057 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (296 ms) 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (12106 ms) 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (4 ms) 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (12406 ms total) 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 43 tests from LibRadosTwoPoolsPP 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7087 ms) 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (9550 ms) 2024-02-13T00:44:27.408 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10219 ms) 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: my_snaps [4] 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: my_snaps [5,4] 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: my_snaps [6,5,4] 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: my_snaps [7,6,5,4] 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: promoting some heads 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: promoting from clones for snap 7 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: promoting from clones for snap 6 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: promoting from clones for snap 5 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: promoting from clones for snap 4 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: waiting for scrubs... 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: done waiting 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (45806 ms) 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10008 ms) 2024-02-13T00:44:27.409 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8252 ms) 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8725 ms) 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8012 ms) 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10021 ms) 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8997 ms) 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9982 ms) 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8247 ms) 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-02-13T00:44:27.410 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7779 ms) 2024-02-13T00:44:27.411 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-02-13T00:44:27.411 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13019 ms) 2024-02-13T00:44:27.411 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-02-13T00:44:27.411 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7978 ms) 2024-02-13T00:44:27.411 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-02-13T00:44:27.411 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8019 ms) 2024-02-13T00:44:27.411 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-02-13T00:44:27.411 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: hmm, no HitSet yet 2024-02-13T00:44:27.411 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: ok, hit_set contains 186:602f83fe:::foo:head 2024-02-13T00:44:27.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.407+0000 7fbfe4055700 10 librados: watch_flush enter 2024-02-13T00:44:27.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.407+0000 7fbfe4055700 10 librados: watch_flush exit 2024-02-13T00:44:27.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.407+0000 7fbfe4055700 20 client.44224.objecter shutdown clearing up homeless session... 2024-02-13T00:44:27.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.407+0000 7fbfe4055700 10 client.44224.objecter successfully canceled tick 2024-02-13T00:44:27.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.408+0000 7fbfe4055700 1 -- 172.21.15.162:0/1110424101 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbfc8062c90 msgr2=0x7fbfc8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:27.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.408+0000 7fbfe4055700 1 --2- 172.21.15.162:0/1110424101 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbfc8062c90 0x7fbfc8065150 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 rx=0x7fbfd8004870 tx=0x7fbfd800b040).stop 2024-02-13T00:44:27.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.408+0000 7fbfe4055700 1 -- 172.21.15.162:0/1110424101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfdc100a20 msgr2=0x7fbfdc10e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:27.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.408+0000 7fbfe4055700 1 --2- 172.21.15.162:0/1110424101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfdc100a20 0x7fbfdc10e9f0 secure :-1 s=READY pgs=4851 cs=0 l=1 rev1=1 rx=0x7fbfd400ec20 tx=0x7fbfd400b040).stop 2024-02-13T00:44:27.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.408+0000 7fbfe4055700 1 -- 172.21.15.162:0/1110424101 shutdown_connections 2024-02-13T00:44:27.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.408+0000 7fbfe4055700 1 --2- 172.21.15.162:0/1110424101 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbfc8062c90 0x7fbfc8065150 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:27.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.408+0000 7fbfe4055700 1 --2- 172.21.15.162:0/1110424101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfdc100a20 0x7fbfdc10e9f0 unknown :-1 s=CLOSED pgs=4851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:27.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.408+0000 7fbfe4055700 1 --2- 172.21.15.162:0/1110424101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbfdc072510 0x7fbfdc1147e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:27.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.408+0000 7fbfe4055700 1 --2- 172.21.15.162:0/1110424101 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbfdc100150 0x7fbfdc114cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:44:27.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.408+0000 7fbfe4055700 1 -- 172.21.15.162:0/1110424101 >> 172.21.15.162:0/1110424101 conn(0x7fbfdc06d8d0 msgr2=0x7fbfdc102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:27.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.408+0000 7fbfe4055700 1 -- 172.21.15.162:0/1110424101 shutdown_connections 2024-02-13T00:44:27.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.409+0000 7fbfe4055700 1 -- 172.21.15.162:0/1110424101 wait complete. 2024-02-13T00:44:27.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:27.409+0000 7fbfe4055700 1 librados: shutdown 2024-02-13T00:44:27.421 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:44:27.693 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:26.215774+0000 osd.1 (osd.1) 125 : cluster [DBG] 5.12 scrub starts 2024-02-13T00:44:27.694 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:26.218803+0000 osd.1 (osd.1) 126 : cluster [DBG] 5.12 scrub ok 2024-02-13T00:44:27.694 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:26.389440+0000 mgr.y (mgr.14544) 293 : cluster [DBG] pgmap v437: 268 pgs: 268 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-02-13T00:44:27.694 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:26.541536+0000 osd.2 (osd.2) 129 : cluster [DBG] 18.11 scrub starts 2024-02-13T00:44:27.694 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:26.544996+0000 osd.2 (osd.2) 130 : cluster [DBG] 18.11 scrub ok 2024-02-13T00:44:27.694 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:26.866915+0000 mon.b (mon.1) 458 : audit [INF] from='client.? 172.21.15.162:0/1110424101' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:27.694 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:26.867439+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:27.694 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:26.937350+0000 mon.b (mon.1) 459 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:27.694 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:27.384930+0000 mon.a (mon.0) 2038 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:44:27.694 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:27.392534+0000 mon.a (mon.0) 2039 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-83985-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-83985-54"}]': finished 2024-02-13T00:44:27.694 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:27.392654+0000 mon.a (mon.0) 2040 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:44:27.694 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:27.392701+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e312: 8 total, 8 up, 6 in 2024-02-13T00:44:27.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:26.215774+0000 osd.1 (osd.1) 125 : cluster [DBG] 5.12 scrub starts 2024-02-13T00:44:27.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:26.218803+0000 osd.1 (osd.1) 126 : cluster [DBG] 5.12 scrub ok 2024-02-13T00:44:27.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:26.389440+0000 mgr.y (mgr.14544) 293 : cluster [DBG] pgmap v437: 268 pgs: 268 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-02-13T00:44:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:26.541536+0000 osd.2 (osd.2) 129 : cluster [DBG] 18.11 scrub starts 2024-02-13T00:44:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:26.544996+0000 osd.2 (osd.2) 130 : cluster [DBG] 18.11 scrub ok 2024-02-13T00:44:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:26.866915+0000 mon.b (mon.1) 458 : audit [INF] from='client.? 172.21.15.162:0/1110424101' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:26.867439+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:26.937350+0000 mon.b (mon.1) 459 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:27.384930+0000 mon.a (mon.0) 2038 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:44:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:27.392534+0000 mon.a (mon.0) 2039 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-83985-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-83985-54"}]': finished 2024-02-13T00:44:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:27.392654+0000 mon.a (mon.0) 2040 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:44:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:27.392701+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e312: 8 total, 8 up, 6 in 2024-02-13T00:44:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:26.215774+0000 osd.1 (osd.1) 125 : cluster [DBG] 5.12 scrub starts 2024-02-13T00:44:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:26.218803+0000 osd.1 (osd.1) 126 : cluster [DBG] 5.12 scrub ok 2024-02-13T00:44:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:26.389440+0000 mgr.y (mgr.14544) 293 : cluster [DBG] pgmap v437: 268 pgs: 268 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-02-13T00:44:27.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:26.541536+0000 osd.2 (osd.2) 129 : cluster [DBG] 18.11 scrub starts 2024-02-13T00:44:27.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:26.544996+0000 osd.2 (osd.2) 130 : cluster [DBG] 18.11 scrub ok 2024-02-13T00:44:27.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:26.866915+0000 mon.b (mon.1) 458 : audit [INF] from='client.? 172.21.15.162:0/1110424101' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:27.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:26.867439+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:27.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:26.937350+0000 mon.b (mon.1) 459 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:27.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:27.384930+0000 mon.a (mon.0) 2038 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:44:27.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:27.392534+0000 mon.a (mon.0) 2039 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-83985-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-83985-54"}]': finished 2024-02-13T00:44:27.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:27.392654+0000 mon.a (mon.0) 2040 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:44:27.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:27.392701+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e312: 8 total, 8 up, 6 in 2024-02-13T00:44:28.004 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:44:28.004 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:44:28.064 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:28.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:28.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:28.197 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:28.794 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:44:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:44:28.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:27.070106+0000 osd.5 (osd.5) 133 : cluster [DBG] 28.a scrub starts 2024-02-13T00:44:28.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:27.073031+0000 osd.5 (osd.5) 134 : cluster [DBG] 28.a scrub ok 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:27.938271+0000 mon.b (mon.1) 460 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:27.938848+0000 mon.b (mon.1) 461 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]: dispatch 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:27.939245+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]: dispatch 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:28.390114+0000 mon.c (mon.2) 307 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:28.392027+0000 mon.a (mon.0) 2043 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:28.396726+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]': finished 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:28.396884+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e313: 8 total, 8 up, 6 in 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:28.397145+0000 mon.b (mon.1) 462 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:28.397429+0000 mon.b (mon.1) 463 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:28.398077+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:28.401160+0000 mon.b (mon.1) 464 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:28.401826+0000 mon.b (mon.1) 465 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-13T00:44:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:28.406216+0000 mon.a (mon.0) 2047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-13T00:44:28.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:27.070106+0000 osd.5 (osd.5) 133 : cluster [DBG] 28.a scrub starts 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:27.073031+0000 osd.5 (osd.5) 134 : cluster [DBG] 28.a scrub ok 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:27.938271+0000 mon.b (mon.1) 460 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:27.938848+0000 mon.b (mon.1) 461 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]: dispatch 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:27.939245+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]: dispatch 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:28.390114+0000 mon.c (mon.2) 307 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:28.392027+0000 mon.a (mon.0) 2043 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:28.396726+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]': finished 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:28.396884+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e313: 8 total, 8 up, 6 in 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:28.397145+0000 mon.b (mon.1) 462 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:28.397429+0000 mon.b (mon.1) 463 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:28.398077+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:28.401160+0000 mon.b (mon.1) 464 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:28.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:28.401826+0000 mon.b (mon.1) 465 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:28.406216+0000 mon.a (mon.0) 2047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:27.070106+0000 osd.5 (osd.5) 133 : cluster [DBG] 28.a scrub starts 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:27.073031+0000 osd.5 (osd.5) 134 : cluster [DBG] 28.a scrub ok 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:27.938271+0000 mon.b (mon.1) 460 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:27.938848+0000 mon.b (mon.1) 461 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]: dispatch 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:27.939245+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]: dispatch 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:28.390114+0000 mon.c (mon.2) 307 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:28.392027+0000 mon.a (mon.0) 2043 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:28.396726+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pg_num","val":"11"}]': finished 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:28.396884+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e313: 8 total, 8 up, 6 in 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:28.397145+0000 mon.b (mon.1) 462 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:28.397429+0000 mon.b (mon.1) 463 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:28.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:28.398077+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:28.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:28.401160+0000 mon.b (mon.1) 464 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:28.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:28.401826+0000 mon.b (mon.1) 465 : audit [INF] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-13T00:44:28.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:28.406216+0000 mon.a (mon.0) 2047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-13T00:44:29.088 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:44:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.839+0000 7fce46b33700 1 -- 172.21.15.162:0/89251275 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce400f8490 msgr2=0x7fce40104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.839+0000 7fce46b33700 1 --2- 172.21.15.162:0/89251275 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce400f8490 0x7fce40104a90 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 rx=0x7fce38009be0 tx=0x7fce38004d10).stop 2024-02-13T00:44:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.839+0000 7fce46b33700 1 -- 172.21.15.162:0/89251275 shutdown_connections 2024-02-13T00:44:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.839+0000 7fce46b33700 1 --2- 172.21.15.162:0/89251275 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce400f8490 0x7fce40104a90 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.839+0000 7fce46b33700 1 --2- 172.21.15.162:0/89251275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce400fdc20 0x7fce400fe000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.839+0000 7fce46b33700 1 --2- 172.21.15.162:0/89251275 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce400fe580 0x7fce400f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.839+0000 7fce46b33700 1 -- 172.21.15.162:0/89251275 >> 172.21.15.162:0/89251275 conn(0x7fce400f3850 msgr2=0x7fce400f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.839+0000 7fce46b33700 1 -- 172.21.15.162:0/89251275 shutdown_connections 2024-02-13T00:44:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.839+0000 7fce46b33700 1 -- 172.21.15.162:0/89251275 wait complete. 2024-02-13T00:44:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce46b33700 1 librados: starting msgr at 2024-02-13T00:44:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce46b33700 1 librados: starting objecter 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce46b33700 1 Processor -- start 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce46b33700 1 -- start start 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce46b33700 1 librados: setting wanted keys 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce46b33700 1 librados: calling monclient init 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce46b33700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce400f8490 0x7fce4006b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce46b33700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce400fdc20 0x7fce4006b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce46b33700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce400fe580 0x7fce4006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce46b33700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce401163e0 con 0x7fce400f8490 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce46b33700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce40116550 con 0x7fce400fdc20 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce46b33700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce40116710 con 0x7fce400fe580 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce448cf700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce400f8490 0x7fce4006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce450d0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce400fe580 0x7fce4006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce448cf700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce400f8490 0x7fce4006b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:57936/0 (socket says 172.21.15.162:57936) 2024-02-13T00:44:29.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.840+0000 7fce448cf700 1 -- 172.21.15.162:0/3335960410 learned_addr learned my addr 172.21.15.162:0/3335960410 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce3ffff700 1 --2- 172.21.15.162:0/3335960410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce400fdc20 0x7fce4006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce450d0700 1 -- 172.21.15.162:0/3335960410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce400fdc20 msgr2=0x7fce4006b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce450d0700 1 --2- 172.21.15.162:0/3335960410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce400fdc20 0x7fce4006b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce450d0700 1 -- 172.21.15.162:0/3335960410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce400f8490 msgr2=0x7fce4006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce450d0700 1 --2- 172.21.15.162:0/3335960410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce400f8490 0x7fce4006b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce450d0700 1 -- 172.21.15.162:0/3335960410 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c005040 con 0x7fce400fe580 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce448cf700 1 --2- 172.21.15.162:0/3335960410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce400f8490 0x7fce4006b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce450d0700 1 --2- 172.21.15.162:0/3335960410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce400fe580 0x7fce4006bb50 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7fce3800b6d0 tx=0x7fce38004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce3dffb700 10 client.?.objecter ms_handle_connect 0x7fce400fe580 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce3dffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce3dffb700 1 -- 172.21.15.162:0/3335960410 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3800e040 con 0x7fce400fe580 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce3dffb700 1 -- 172.21.15.162:0/3335960410 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce38018040 con 0x7fce400fe580 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce46b33700 1 -- 172.21.15.162:0/3335960410 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce380097c0 con 0x7fce400fe580 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce3dffb700 1 -- 172.21.15.162:0/3335960410 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38013e90 con 0x7fce400fe580 2024-02-13T00:44:29.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce46b33700 10 client.37160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce46b33700 1 -- 172.21.15.162:0/3335960410 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce40116c00 con 0x7fce400fe580 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.841+0000 7fce46b33700 1 librados: init done 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.842+0000 7fce46b33700 10 client.37160.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.842+0000 7fce46b33700 15 client.37160.objecter _session_command_op_assign -1 1 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.842+0000 7fce46b33700 20 client.37160.objecter _get_session osd=-1 returning homeless 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.842+0000 7fce46b33700 10 client.37160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.842+0000 7fce46b33700 1 -- 172.21.15.162:0/3335960410 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fce400607b0 con 0x7fce400fe580 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 1 -- 172.21.15.162:0/3335960410 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fce38013590 con 0x7fce400fe580 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 1 --2- 172.21.15.162:0/3335960410 >> v2:172.21.15.162:6800/2866534053 conn(0x7fce30062c90 0x7fce30065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 1 -- 172.21.15.162:0/3335960410 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(313..313 src has 52..313) v4 ==== 9141+0+0 (secure 0 0 0) 0x7fce3807e650 con 0x7fce400fe580 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 10 client.37160.objecter ms_dispatch 0x7fce40001550 osd_map(313..313 src has 52..313) v4 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 3 client.37160.objecter handle_osd_map got epochs [313,313] > 0 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 3 client.37160.objecter handle_osd_map decoding full epoch 313 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce448cf700 1 --2- 172.21.15.162:0/3335960410 >> v2:172.21.15.162:6800/2866534053 conn(0x7fce30062c90 0x7fce30065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:29.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 10 client.37160.objecter checking command 1 2024-02-13T00:44:29.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 1 --2- 172.21.15.162:0/3335960410 >> v2:172.21.15.181:6812/1964192772 conn(0x7fce30071250 0x7fce300736f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:29.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 20 client.37160.objecter _get_session s=0x7fce38013e90 osd=7 3 2024-02-13T00:44:29.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 20 client.37160.objecter put_session s=0x7fce38013e90 osd=7 3 2024-02-13T00:44:29.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 15 client.37160.objecter _session_command_op_remove -1 1 2024-02-13T00:44:29.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 20 client.37160.objecter _get_session s=0x7fce38013e90 osd=7 3 2024-02-13T00:44:29.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 20 client.37160.objecter get_session s=0x7fce38013e90 osd=7 3 2024-02-13T00:44:29.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 15 client.37160.objecter _session_command_op_assign 7 1 2024-02-13T00:44:29.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 20 client.37160.objecter put_session s=0x7fce38013e90 osd=7 4 2024-02-13T00:44:29.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 10 client.37160.objecter _send_command 1 2024-02-13T00:44:29.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 1 -- 172.21.15.162:0/3335960410 --> v2:172.21.15.181:6812/1964192772 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce30073d30 con 0x7fce30071250 2024-02-13T00:44:29.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 20 client.37160.objecter dump_active .. 0 homeless 2024-02-13T00:44:29.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 1 -- 172.21.15.162:0/3335960410 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=313) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce38080c70 con 0x7fce400fe580 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3dffb700 10 client.37160.objecter ms_handle_connect 0x7fce30062c90 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce448cf700 1 --2- 172.21.15.162:0/3335960410 >> v2:172.21.15.162:6800/2866534053 conn(0x7fce30062c90 0x7fce30065150 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 rx=0x7fce3400f3d0 tx=0x7fce34009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.843+0000 7fce3ffff700 1 --2- 172.21.15.162:0/3335960410 >> v2:172.21.15.181:6812/1964192772 conn(0x7fce30071250 0x7fce300736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.844+0000 7fce3dffb700 10 client.37160.objecter ms_handle_connect 0x7fce30071250 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.844+0000 7fce3ffff700 1 --2- 172.21.15.162:0/3335960410 >> v2:172.21.15.181:6812/1964192772 conn(0x7fce30071250 0x7fce300736f0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.844+0000 7fce3dffb700 1 -- 172.21.15.162:0/3335960410 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fce30073d30 con 0x7fce30071250 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.844+0000 7fce3dffb700 10 client.37160.objecter ms_dispatch 0x7fce40001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.844+0000 7fce3dffb700 10 client.37160.objecter _finish_command 1 = system:0 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.844+0000 7fce3dffb700 20 client.37160.objecter put_session s=0x7fce38013e90 osd=7 4 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.844+0000 7fce3dffb700 15 client.37160.objecter _session_command_op_remove 7 1 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.861+0000 7fce46b33700 10 client.37160.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.861+0000 7fce46b33700 15 client.37160.objecter _session_command_op_assign -1 2 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.861+0000 7fce46b33700 20 client.37160.objecter _get_session s=0x7fce38013e90 osd=7 3 2024-02-13T00:44:29.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.861+0000 7fce46b33700 20 client.37160.objecter put_session s=0x7fce38013e90 osd=7 3 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.861+0000 7fce46b33700 20 client.37160.objecter _get_session s=0x7fce38013e90 osd=7 3 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.861+0000 7fce46b33700 15 client.37160.objecter _session_command_op_remove -1 2 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.861+0000 7fce46b33700 20 client.37160.objecter get_session s=0x7fce38013e90 osd=7 3 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.861+0000 7fce46b33700 15 client.37160.objecter _session_command_op_assign 7 2 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.861+0000 7fce46b33700 20 client.37160.objecter put_session s=0x7fce38013e90 osd=7 4 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.861+0000 7fce46b33700 10 client.37160.objecter _send_command 2 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.861+0000 7fce46b33700 1 -- 172.21.15.162:0/3335960410 --> v2:172.21.15.181:6812/1964192772 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7fce400475e0 con 0x7fce30071250 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce3dffb700 1 -- 172.21.15.162:0/3335960410 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fce400475e0 con 0x7fce30071250 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce3dffb700 10 client.37160.objecter ms_dispatch 0x7fce40001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce3dffb700 10 client.37160.objecter _finish_command 2 = system:0 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce3dffb700 20 client.37160.objecter put_session s=0x7fce38013e90 osd=7 4 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce3dffb700 15 client.37160.objecter _session_command_op_remove 7 2 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 10 librados: watch_flush enter 2024-02-13T00:44:29.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 10 librados: watch_flush exit 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 10 client.37160.objecter close_session for osd.7 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 1 -- 172.21.15.162:0/3335960410 >> v2:172.21.15.181:6812/1964192772 conn(0x7fce30071250 msgr2=0x7fce300736f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 1 --2- 172.21.15.162:0/3335960410 >> v2:172.21.15.181:6812/1964192772 conn(0x7fce30071250 0x7fce300736f0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 20 client.37160.objecter put_session s=0x7fce38013e90 osd=7 1 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 20 client.37160.objecter shutdown clearing up homeless session... 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 10 client.37160.objecter successfully canceled tick 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 1 -- 172.21.15.162:0/3335960410 >> v2:172.21.15.162:6800/2866534053 conn(0x7fce30062c90 msgr2=0x7fce30065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 1 --2- 172.21.15.162:0/3335960410 >> v2:172.21.15.162:6800/2866534053 conn(0x7fce30062c90 0x7fce30065150 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 rx=0x7fce3400f3d0 tx=0x7fce34009040).stop 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 1 -- 172.21.15.162:0/3335960410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce400fe580 msgr2=0x7fce4006bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 1 --2- 172.21.15.162:0/3335960410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce400fe580 0x7fce4006bb50 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7fce3800b6d0 tx=0x7fce38004d10).stop 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce450d0700 1 -- 172.21.15.162:0/3335960410 reap_dead start 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 1 -- 172.21.15.162:0/3335960410 shutdown_connections 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 1 -- 172.21.15.162:0/3335960410 >> 172.21.15.162:0/3335960410 conn(0x7fce400f3850 msgr2=0x7fce400f5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 1 -- 172.21.15.162:0/3335960410 shutdown_connections 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 1 -- 172.21.15.162:0/3335960410 wait complete. 2024-02-13T00:44:29.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:28.862+0000 7fce46b33700 1 librados: shutdown 2024-02-13T00:44:29.446 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:29.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:29.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:29.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:29.579 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:29.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:28.098363+0000 osd.5 (osd.5) 135 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:44:29.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:28.101251+0000 osd.5 (osd.5) 136 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:44:29.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:28.389806+0000 mgr.y (mgr.14544) 294 : cluster [DBG] pgmap v439: 243 pgs: 8 unknown, 235 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-02-13T00:44:29.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:29.397029+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-13T00:44:29.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:29.397122+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:29.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:29.397233+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]': finished 2024-02-13T00:44:29.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:29.397355+0000 mon.a (mon.0) 2051 : cluster [DBG] osdmap e314: 8 total, 8 up, 6 in 2024-02-13T00:44:29.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:29.398471+0000 mon.c (mon.2) 308 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:29.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:29.399159+0000 mon.a (mon.0) 2052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:29.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:29.401139+0000 mon.b (mon.1) 466 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:29.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:29.402775+0000 mon.b (mon.1) 467 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi162-84057-6","var": "pg_num","format": "json"}]: dispatch 2024-02-13T00:44:29.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:29.404628+0000 mon.b (mon.1) 468 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:29.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:29.408206+0000 mon.a (mon.0) 2053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:29.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:28.098363+0000 osd.5 (osd.5) 135 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:44:29.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:28.101251+0000 osd.5 (osd.5) 136 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:44:29.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:28.389806+0000 mgr.y (mgr.14544) 294 : cluster [DBG] pgmap v439: 243 pgs: 8 unknown, 235 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-02-13T00:44:29.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:29.397029+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-13T00:44:29.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:29.397122+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:29.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:29.397233+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]': finished 2024-02-13T00:44:29.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:29.397355+0000 mon.a (mon.0) 2051 : cluster [DBG] osdmap e314: 8 total, 8 up, 6 in 2024-02-13T00:44:29.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:29.398471+0000 mon.c (mon.2) 308 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:29.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:29.399159+0000 mon.a (mon.0) 2052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:29.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:29.401139+0000 mon.b (mon.1) 466 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:29.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:29.402775+0000 mon.b (mon.1) 467 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi162-84057-6","var": "pg_num","format": "json"}]: dispatch 2024-02-13T00:44:29.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:29.404628+0000 mon.b (mon.1) 468 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:29.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:29.408206+0000 mon.a (mon.0) 2053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:30.028 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:30.029 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:30.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:28.098363+0000 osd.5 (osd.5) 135 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:44:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:28.101251+0000 osd.5 (osd.5) 136 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:44:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:28.389806+0000 mgr.y (mgr.14544) 294 : cluster [DBG] pgmap v439: 243 pgs: 8 unknown, 235 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-02-13T00:44:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:29.397029+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-13T00:44:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:29.397122+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:29.397233+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-84017-1","var":"pgp_num","val":"11"}]': finished 2024-02-13T00:44:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:29.397355+0000 mon.a (mon.0) 2051 : cluster [DBG] osdmap e314: 8 total, 8 up, 6 in 2024-02-13T00:44:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:29.398471+0000 mon.c (mon.2) 308 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:29.399159+0000 mon.a (mon.0) 2052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:29.401139+0000 mon.b (mon.1) 466 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:29.402775+0000 mon.b (mon.1) 467 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi162-84057-6","var": "pg_num","format": "json"}]: dispatch 2024-02-13T00:44:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:29.404628+0000 mon.b (mon.1) 468 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:30.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:29.408206+0000 mon.a (mon.0) 2053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:30.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:30.163 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:30.512 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:30.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:30.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:30.642 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:30.740 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:30.742 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:29.147734+0000 osd.1 (osd.1) 127 : cluster [DBG] 18.a scrub starts 2024-02-13T00:44:30.742 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:29.151233+0000 osd.1 (osd.1) 128 : cluster [DBG] 18.a scrub ok 2024-02-13T00:44:30.742 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:29.531729+0000 osd.2 (osd.2) 131 : cluster [DBG] 18.17 deep-scrub starts 2024-02-13T00:44:30.743 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:29.535203+0000 osd.2 (osd.2) 132 : cluster [DBG] 18.17 deep-scrub ok 2024-02-13T00:44:30.743 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:30.045129+0000 osd.5 (osd.5) 137 : cluster [DBG] 61.8 scrub starts 2024-02-13T00:44:30.743 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:30.046217+0000 osd.5 (osd.5) 138 : cluster [DBG] 61.8 scrub ok 2024-02-13T00:44:30.743 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:30.371483+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]': finished 2024-02-13T00:44:30.743 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:30.371577+0000 mon.a (mon.0) 2055 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:30.743 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:30.371635+0000 mon.a (mon.0) 2056 : cluster [DBG] osdmap e315: 8 total, 8 up, 6 in 2024-02-13T00:44:30.743 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:30.376391+0000 mon.b (mon.1) 469 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-13T00:44:30.743 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:30.376503+0000 mon.c (mon.2) 309 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:30.743 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:30.381129+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-13T00:44:30.743 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:30.383387+0000 mon.a (mon.0) 2058 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:30.743 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:30.401983+0000 mon.b (mon.1) 470 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:30.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:30.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:30.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:29.147734+0000 osd.1 (osd.1) 127 : cluster [DBG] 18.a scrub starts 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:29.151233+0000 osd.1 (osd.1) 128 : cluster [DBG] 18.a scrub ok 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:29.531729+0000 osd.2 (osd.2) 131 : cluster [DBG] 18.17 deep-scrub starts 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:29.535203+0000 osd.2 (osd.2) 132 : cluster [DBG] 18.17 deep-scrub ok 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:30.045129+0000 osd.5 (osd.5) 137 : cluster [DBG] 61.8 scrub starts 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:30.046217+0000 osd.5 (osd.5) 138 : cluster [DBG] 61.8 scrub ok 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:30.371483+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]': finished 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:30.371577+0000 mon.a (mon.0) 2055 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:30.371635+0000 mon.a (mon.0) 2056 : cluster [DBG] osdmap e315: 8 total, 8 up, 6 in 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:30.376391+0000 mon.b (mon.1) 469 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:30.376503+0000 mon.c (mon.2) 309 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:30.381129+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:30.383387+0000 mon.a (mon.0) 2058 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:30.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:30.401983+0000 mon.b (mon.1) 470 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:30.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:29.147734+0000 osd.1 (osd.1) 127 : cluster [DBG] 18.a scrub starts 2024-02-13T00:44:30.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:29.151233+0000 osd.1 (osd.1) 128 : cluster [DBG] 18.a scrub ok 2024-02-13T00:44:30.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:29.531729+0000 osd.2 (osd.2) 131 : cluster [DBG] 18.17 deep-scrub starts 2024-02-13T00:44:30.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:29.535203+0000 osd.2 (osd.2) 132 : cluster [DBG] 18.17 deep-scrub ok 2024-02-13T00:44:30.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:30.045129+0000 osd.5 (osd.5) 137 : cluster [DBG] 61.8 scrub starts 2024-02-13T00:44:30.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:30.046217+0000 osd.5 (osd.5) 138 : cluster [DBG] 61.8 scrub ok 2024-02-13T00:44:30.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:30.371483+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-83985-54"}]': finished 2024-02-13T00:44:30.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:30.371577+0000 mon.a (mon.0) 2055 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:30.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:30.371635+0000 mon.a (mon.0) 2056 : cluster [DBG] osdmap e315: 8 total, 8 up, 6 in 2024-02-13T00:44:30.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:30.376391+0000 mon.b (mon.1) 469 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-13T00:44:30.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:30.376503+0000 mon.c (mon.2) 309 : audit [INF] from='client.? 172.21.15.162:0/3895428577' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:30.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:30.381129+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-13T00:44:30.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:30.383387+0000 mon.a (mon.0) 2058 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]: dispatch 2024-02-13T00:44:30.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:30.401983+0000 mon.b (mon.1) 470 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:30.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:44:30.873 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:30.893 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:44:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:30.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:44:30.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:31.021 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:44:31.021 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:44:31.025 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:31.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:44:31.273 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:31.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:31.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:31.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:31.419 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:31.543 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:31.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:31.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:31.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:31.678 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:31.776 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:31.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:31.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-13T00:44:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.800+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3415020386 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0dd41021d0 msgr2=0x7f0dd41045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.800+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3415020386 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0dd41021d0 0x7f0dd41045c0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 rx=0x7f0dcc009be0 tx=0x7f0dcc004d10).stop 2024-02-13T00:44:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.800+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3415020386 shutdown_connections 2024-02-13T00:44:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.800+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3415020386 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0dd41021d0 0x7f0dd41045c0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.800+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3415020386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dd40f8c70 0x7f0dd40f9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.800+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3415020386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dd40f9540 0x7f0dd4101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.800+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3415020386 >> 172.21.15.162:0/3415020386 conn(0x7f0dd40f2740 msgr2=0x7f0dd40f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.801+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3415020386 shutdown_connections 2024-02-13T00:44:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.801+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3415020386 wait complete. 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.801+0000 7f0dd9d32700 1 librados: starting msgr at 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.801+0000 7f0dd9d32700 1 librados: starting objecter 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.801+0000 7f0dd9d32700 1 Processor -- start 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.801+0000 7f0dd9d32700 1 -- start start 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.801+0000 7f0dd9d32700 1 librados: setting wanted keys 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.801+0000 7f0dd9d32700 1 librados: calling monclient init 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd9d32700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dd40f8c70 0x7f0dd40f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd9d32700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dd40f9540 0x7f0dd40f7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd9d32700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0dd41021d0 0x7f0dd40f7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd9d32700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd41165b0 con 0x7f0dd40f8c70 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd9d32700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd4116720 con 0x7f0dd40f9540 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd9d32700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd41168e0 con 0x7f0dd41021d0 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0dd41021d0 0x7f0dd40f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0dd41021d0 0x7f0dd40f7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:47970/0 (socket says 172.21.15.162:47970) 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd3fff700 1 -- 172.21.15.162:0/3066150105 learned_addr learned my addr 172.21.15.162:0/3066150105 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd2ffd700 1 --2- 172.21.15.162:0/3066150105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dd40f9540 0x7f0dd40f7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd37fe700 1 --2- 172.21.15.162:0/3066150105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dd40f8c70 0x7f0dd40f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd3fff700 1 -- 172.21.15.162:0/3066150105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dd40f9540 msgr2=0x7f0dd40f7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd3fff700 1 --2- 172.21.15.162:0/3066150105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dd40f9540 0x7f0dd40f7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd3fff700 1 -- 172.21.15.162:0/3066150105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dd40f8c70 msgr2=0x7f0dd40f6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd3fff700 1 --2- 172.21.15.162:0/3066150105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0dd40f8c70 0x7f0dd40f6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd3fff700 1 -- 172.21.15.162:0/3066150105 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc005040 con 0x7f0dd41021d0 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd3fff700 1 --2- 172.21.15.162:0/3066150105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0dd41021d0 0x7f0dd40f7620 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 rx=0x7f0dcc004ed0 tx=0x7f0dcc004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd0ff9700 10 client.?.objecter ms_handle_connect 0x7f0dd41021d0 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd0ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc00e040 con 0x7f0dd41021d0 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dcc0097c0 con 0x7f0dd41021d0 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0dcc018040 con 0x7f0dd41021d0 2024-02-13T00:44:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc01c3f0 con 0x7f0dd41021d0 2024-02-13T00:44:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd9d32700 10 client.37172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd4116dd0 con 0x7f0dd41021d0 2024-02-13T00:44:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.802+0000 7f0dd9d32700 1 librados: init done 2024-02-13T00:44:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.803+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0dd40607d0 con 0x7f0dd41021d0 2024-02-13T00:44:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.804+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0dcc013590 con 0x7f0dd41021d0 2024-02-13T00:44:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.804+0000 7f0dd0ff9700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0dc0062c90 0x7f0dc0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.804+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(316..316 src has 52..316) v4 ==== 8370+0+0 (secure 0 0 0) 0x7f0dcc07e230 con 0x7f0dd41021d0 2024-02-13T00:44:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.804+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 osd_map(316..316 src has 52..316) v4 2024-02-13T00:44:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.804+0000 7f0dd0ff9700 3 client.37172.objecter handle_osd_map got epochs [316,316] > 0 2024-02-13T00:44:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.804+0000 7f0dd0ff9700 3 client.37172.objecter handle_osd_map decoding full epoch 316 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.804+0000 7f0dd37fe700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0dc0062c90 0x7f0dc0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.804+0000 7f0dd0ff9700 20 client.37172.objecter dump_active .. 0 homeless 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.804+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v316) v1 ==== 54+0+16 (secure 0 0 0) 0x7f0dcc085070 con 0x7f0dd41021d0 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd0ff9700 10 client.37172.objecter ms_handle_connect 0x7f0dc0062c90 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd37fe700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0dc0062c90 0x7f0dc0065150 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7f0dc4004de0 tx=0x7f0dc400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 1 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0dd40fc180 0x7f0dd40fc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd40475e0 osd=0 3 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd40475e0 osd=0 3 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd40475e0 osd=0 3 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 1 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd40475e0 osd=0 3 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 0 1 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd40475e0 osd=0 4 2024-02-13T00:44:31.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd9d32700 10 client.37172.objecter _send_command 1 2024-02-13T00:44:31.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0dd4107190 con 0x7f0dd40fc180 2024-02-13T00:44:31.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd2ffd700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0dd40fc180 0x7f0dd40fc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:31.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd0ff9700 10 client.37172.objecter ms_handle_connect 0x7f0dd40fc180 2024-02-13T00:44:31.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.805+0000 7f0dd2ffd700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0dd40fc180 0x7f0dd40fc560 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:31.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.806+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0dd4107190 con 0x7f0dd40fc180 2024-02-13T00:44:31.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.806+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:44:31.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.806+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 1 = system:0 2024-02-13T00:44:31.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.806+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd40475e0 osd=0 4 2024-02-13T00:44:31.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.806+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 0 1 2024-02-13T00:44:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.823+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.823+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 2 2024-02-13T00:44:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.823+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd40475e0 osd=0 3 2024-02-13T00:44:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.823+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd40475e0 osd=0 3 2024-02-13T00:44:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.823+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd40475e0 osd=0 3 2024-02-13T00:44:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.823+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 2 2024-02-13T00:44:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.823+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd40475e0 osd=0 3 2024-02-13T00:44:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.823+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 0 2 2024-02-13T00:44:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.823+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd40475e0 osd=0 4 2024-02-13T00:44:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.823+0000 7f0dd9d32700 10 client.37172.objecter _send_command 2 2024-02-13T00:44:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.823+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0dd4027410 con 0x7f0dd40fc180 2024-02-13T00:44:31.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.824+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0dd4027410 con 0x7f0dd40fc180 2024-02-13T00:44:31.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.824+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:31.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.824+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.824+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd40475e0 osd=0 4 2024-02-13T00:44:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.824+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 0 2 2024-02-13T00:44:31.828 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:44:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 3 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6805/150599780 conn(0x7f0dd4059990 0x7f0dd4059d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4028760 osd=1 3 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4028760 osd=1 3 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4028760 osd=1 3 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 3 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd4028760 osd=1 3 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 1 3 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4028760 osd=1 4 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd9d32700 10 client.37172.objecter _send_command 3 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0dd411d770 con 0x7f0dd4059990 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd37fe700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6805/150599780 conn(0x7f0dd4059990 0x7f0dd4059d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd0ff9700 10 client.37172.objecter ms_handle_connect 0x7f0dd4059990 2024-02-13T00:44:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.825+0000 7f0dd37fe700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6805/150599780 conn(0x7f0dd4059990 0x7f0dd4059d90 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:31.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.826+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0dd411d770 con 0x7f0dd4059990 2024-02-13T00:44:31.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.826+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:44:31.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.826+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 3 = system:0 2024-02-13T00:44:31.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.826+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd4028760 osd=1 4 2024-02-13T00:44:31.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.826+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 1 3 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:30.390357+0000 mgr.y (mgr.14544) 295 : cluster [DBG] pgmap v443: 267 pgs: 2 peering, 265 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:30.536967+0000 osd.2 (osd.2) 133 : cluster [DBG] 18.7 scrub starts 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:30.539822+0000 osd.2 (osd.2) 134 : cluster [DBG] 18.7 scrub ok 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:31.397436+0000 mon.a (mon.0) 2059 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_count","val": "8"}]': finished 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:31.397520+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]': finished 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:31.397593+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e316: 8 total, 8 up, 6 in 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:31.398489+0000 mon.b (mon.1) 471 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:31.398968+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:31.402903+0000 mon.b (mon.1) 472 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:31.419372+0000 mon.a (mon.0) 2063 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:31.420487+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:31.424566+0000 mon.a (mon.0) 2065 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi162-83985-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:31.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:30.390357+0000 mgr.y (mgr.14544) 295 : cluster [DBG] pgmap v443: 267 pgs: 2 peering, 265 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:31.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:30.536967+0000 osd.2 (osd.2) 133 : cluster [DBG] 18.7 scrub starts 2024-02-13T00:44:31.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:30.539822+0000 osd.2 (osd.2) 134 : cluster [DBG] 18.7 scrub ok 2024-02-13T00:44:31.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:31.397436+0000 mon.a (mon.0) 2059 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_count","val": "8"}]': finished 2024-02-13T00:44:31.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:31.397520+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]': finished 2024-02-13T00:44:31.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:31.397593+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e316: 8 total, 8 up, 6 in 2024-02-13T00:44:31.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:31.398489+0000 mon.b (mon.1) 471 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:44:31.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:31.398968+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:44:31.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:31.402903+0000 mon.b (mon.1) 472 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:31.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:31.419372+0000 mon.a (mon.0) 2063 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:31.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:31.420487+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:31.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:31.424566+0000 mon.a (mon.0) 2065 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi162-83985-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:31.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.844+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:31.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.844+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 4 2024-02-13T00:44:31.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.844+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4028760 osd=1 3 2024-02-13T00:44:31.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.844+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4028760 osd=1 3 2024-02-13T00:44:31.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.844+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4028760 osd=1 3 2024-02-13T00:44:31.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.844+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 4 2024-02-13T00:44:31.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.844+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd4028760 osd=1 3 2024-02-13T00:44:31.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.844+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 1 4 2024-02-13T00:44:31.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.844+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4028760 osd=1 4 2024-02-13T00:44:31.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.844+0000 7f0dd9d32700 10 client.37172.objecter _send_command 4 2024-02-13T00:44:31.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.844+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0dd4059400 con 0x7f0dd4059990 2024-02-13T00:44:31.848 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:31.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.846+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0dd4059400 con 0x7f0dd4059990 2024-02-13T00:44:31.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.846+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:31.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.846+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.846+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd4028760 osd=1 4 2024-02-13T00:44:31.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.846+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 1 4 2024-02-13T00:44:31.850 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 5 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0dd40282c0 0x7f0dd411c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4059810 osd=2 3 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4059810 osd=2 3 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4059810 osd=2 3 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 5 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd4059810 osd=2 3 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 2 5 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4059810 osd=2 4 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd9d32700 10 client.37172.objecter _send_command 5 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0dd411c680 con 0x7f0dd40282c0 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd2ffd700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0dd40282c0 0x7f0dd411c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd0ff9700 10 client.37172.objecter ms_handle_connect 0x7f0dd40282c0 2024-02-13T00:44:31.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.847+0000 7f0dd2ffd700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0dd40282c0 0x7f0dd411c190 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:31.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.848+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0dd411c680 con 0x7f0dd40282c0 2024-02-13T00:44:31.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.848+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:44:31.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.848+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 5 = system:0 2024-02-13T00:44:31.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.848+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd4059810 osd=2 4 2024-02-13T00:44:31.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.848+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 2 5 2024-02-13T00:44:31.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.866+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:31.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.866+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 6 2024-02-13T00:44:31.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.866+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4059810 osd=2 3 2024-02-13T00:44:31.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.866+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4059810 osd=2 3 2024-02-13T00:44:31.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.866+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4059810 osd=2 3 2024-02-13T00:44:31.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.866+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 6 2024-02-13T00:44:31.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.866+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd4059810 osd=2 3 2024-02-13T00:44:31.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.866+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 2 6 2024-02-13T00:44:31.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.866+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4059810 osd=2 4 2024-02-13T00:44:31.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.866+0000 7f0dd9d32700 10 client.37172.objecter _send_command 6 2024-02-13T00:44:31.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.866+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0dd4027820 con 0x7f0dd40282c0 2024-02-13T00:44:31.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0dd4027820 con 0x7f0dd40282c0 2024-02-13T00:44:31.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:31.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd4059810 osd=2 4 2024-02-13T00:44:31.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 2 6 2024-02-13T00:44:31.871 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:44:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 7 2024-02-13T00:44:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6813/604971011 conn(0x7f0dd411f8f0 0x7f0dd4121ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4027c00 osd=3 3 2024-02-13T00:44:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4027c00 osd=3 3 2024-02-13T00:44:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4027c00 osd=3 3 2024-02-13T00:44:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 7 2024-02-13T00:44:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd4027c00 osd=3 3 2024-02-13T00:44:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 3 7 2024-02-13T00:44:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4027c00 osd=3 4 2024-02-13T00:44:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd9d32700 10 client.37172.objecter _send_command 7 2024-02-13T00:44:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0dd411d580 con 0x7f0dd411f8f0 2024-02-13T00:44:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd3fff700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6813/604971011 conn(0x7f0dd411f8f0 0x7f0dd4121ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd3fff700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6813/604971011 conn(0x7f0dd411f8f0 0x7f0dd4121ce0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.868+0000 7f0dd0ff9700 10 client.37172.objecter ms_handle_connect 0x7f0dd411f8f0 2024-02-13T00:44:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.869+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0dd411d580 con 0x7f0dd411f8f0 2024-02-13T00:44:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.869+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:44:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.869+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 7 = system:0 2024-02-13T00:44:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.869+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd4027c00 osd=3 4 2024-02-13T00:44:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.869+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 3 7 2024-02-13T00:44:31.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:31.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.884+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:31.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.884+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 8 2024-02-13T00:44:31.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.884+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4027c00 osd=3 3 2024-02-13T00:44:31.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.884+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4027c00 osd=3 3 2024-02-13T00:44:31.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.884+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4027c00 osd=3 3 2024-02-13T00:44:31.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.884+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 8 2024-02-13T00:44:31.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.884+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd4027c00 osd=3 3 2024-02-13T00:44:31.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.884+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 3 8 2024-02-13T00:44:31.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.884+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4027c00 osd=3 4 2024-02-13T00:44:31.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.884+0000 7f0dd9d32700 10 client.37172.objecter _send_command 8 2024-02-13T00:44:31.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.884+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0dd411cae0 con 0x7f0dd411f8f0 2024-02-13T00:44:31.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0dd411cae0 con 0x7f0dd411f8f0 2024-02-13T00:44:31.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:31.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd4027c00 osd=3 4 2024-02-13T00:44:31.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 3 8 2024-02-13T00:44:31.890 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:44:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 9 2024-02-13T00:44:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0dd411bcf0 0x7f0dd4123350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd411c8a0 osd=4 3 2024-02-13T00:44:31.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd411c8a0 osd=4 3 2024-02-13T00:44:31.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd411c8a0 osd=4 3 2024-02-13T00:44:31.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 9 2024-02-13T00:44:31.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd411c8a0 osd=4 3 2024-02-13T00:44:31.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 4 9 2024-02-13T00:44:31.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd411c8a0 osd=4 4 2024-02-13T00:44:31.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd9d32700 10 client.37172.objecter _send_command 9 2024-02-13T00:44:31.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.886+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0dd411ca80 con 0x7f0dd411bcf0 2024-02-13T00:44:31.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.887+0000 7f0dd37fe700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0dd411bcf0 0x7f0dd4123350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:31.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.887+0000 7f0dd0ff9700 10 client.37172.objecter ms_handle_connect 0x7f0dd411bcf0 2024-02-13T00:44:31.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.887+0000 7f0dd37fe700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0dd411bcf0 0x7f0dd4123350 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:31.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.888+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0dd411ca80 con 0x7f0dd411bcf0 2024-02-13T00:44:31.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.888+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:44:31.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.888+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 9 = system:0 2024-02-13T00:44:31.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.888+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd411c8a0 osd=4 4 2024-02-13T00:44:31.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.888+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 4 9 2024-02-13T00:44:31.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.903+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:31.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.903+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 10 2024-02-13T00:44:31.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.903+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd411c8a0 osd=4 3 2024-02-13T00:44:31.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.903+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd411c8a0 osd=4 3 2024-02-13T00:44:31.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.903+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd411c8a0 osd=4 3 2024-02-13T00:44:31.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.903+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 10 2024-02-13T00:44:31.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.903+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd411c8a0 osd=4 3 2024-02-13T00:44:31.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.903+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 4 10 2024-02-13T00:44:31.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.903+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd411c8a0 osd=4 4 2024-02-13T00:44:31.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.903+0000 7f0dd9d32700 10 client.37172.objecter _send_command 10 2024-02-13T00:44:31.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.903+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0dd411ca80 con 0x7f0dd411bcf0 2024-02-13T00:44:31.907 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0dd411ca80 con 0x7f0dd411bcf0 2024-02-13T00:44:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd411c8a0 osd=4 4 2024-02-13T00:44:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 4 10 2024-02-13T00:44:31.909 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:44:31.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 11 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0dd4123dd0 0x7f0dd4124250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd411ca80 osd=5 3 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd411ca80 osd=5 3 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd411ca80 osd=5 3 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 11 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd411ca80 osd=5 3 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 5 11 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd411ca80 osd=5 4 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd9d32700 10 client.37172.objecter _send_command 11 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.905+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0dd412b0e0 con 0x7f0dd4123dd0 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.906+0000 7f0dd2ffd700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0dd4123dd0 0x7f0dd4124250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.906+0000 7f0dd0ff9700 10 client.37172.objecter ms_handle_connect 0x7f0dd4123dd0 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.906+0000 7f0dd2ffd700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0dd4123dd0 0x7f0dd4124250 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.907+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0dd412b0e0 con 0x7f0dd4123dd0 2024-02-13T00:44:31.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.907+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:44:31.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.907+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 11 = system:0 2024-02-13T00:44:31.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.907+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd411ca80 osd=5 4 2024-02-13T00:44:31.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.907+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 5 11 2024-02-13T00:44:31.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.922+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:31.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.922+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 12 2024-02-13T00:44:31.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.922+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd411ca80 osd=5 3 2024-02-13T00:44:31.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.922+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd411ca80 osd=5 3 2024-02-13T00:44:31.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.922+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd411ca80 osd=5 3 2024-02-13T00:44:31.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.922+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 12 2024-02-13T00:44:31.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.922+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd411ca80 osd=5 3 2024-02-13T00:44:31.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.922+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 5 12 2024-02-13T00:44:31.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.922+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd411ca80 osd=5 4 2024-02-13T00:44:31.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.922+0000 7f0dd9d32700 10 client.37172.objecter _send_command 12 2024-02-13T00:44:31.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.922+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0dd4128980 con 0x7f0dd4123dd0 2024-02-13T00:44:31.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0dd4128980 con 0x7f0dd4123dd0 2024-02-13T00:44:31.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:31.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd411ca80 osd=5 4 2024-02-13T00:44:31.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 5 12 2024-02-13T00:44:31.927 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 13 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0dd4129000 0x7f0dd4129460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4128980 osd=6 3 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4128980 osd=6 3 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4128980 osd=6 3 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 13 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd4128980 osd=6 3 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 6 13 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4128980 osd=6 4 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd9d32700 10 client.37172.objecter _send_command 13 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0dd412b060 con 0x7f0dd4129000 2024-02-13T00:44:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd3fff700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0dd4129000 0x7f0dd4129460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd0ff9700 10 client.37172.objecter ms_handle_connect 0x7f0dd4129000 2024-02-13T00:44:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.924+0000 7f0dd3fff700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0dd4129000 0x7f0dd4129460 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.925+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0dd412b060 con 0x7f0dd4129000 2024-02-13T00:44:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.925+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:44:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.925+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 13 = system:0 2024-02-13T00:44:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.925+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd4128980 osd=6 4 2024-02-13T00:44:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.925+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 6 13 2024-02-13T00:44:31.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.940+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:31.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.940+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 14 2024-02-13T00:44:31.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.940+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4128980 osd=6 3 2024-02-13T00:44:31.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.940+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4128980 osd=6 3 2024-02-13T00:44:31.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.940+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd4128980 osd=6 3 2024-02-13T00:44:31.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.940+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 14 2024-02-13T00:44:31.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.940+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd4128980 osd=6 3 2024-02-13T00:44:31.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.940+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 6 14 2024-02-13T00:44:31.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.940+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4128980 osd=6 4 2024-02-13T00:44:31.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.940+0000 7f0dd9d32700 10 client.37172.objecter _send_command 14 2024-02-13T00:44:31.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.940+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0dd412b060 con 0x7f0dd4129000 2024-02-13T00:44:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0dd412b060 con 0x7f0dd4129000 2024-02-13T00:44:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd4128980 osd=6 4 2024-02-13T00:44:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 6 14 2024-02-13T00:44:31.945 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:44:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 15 2024-02-13T00:44:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0dd412d830 0x7f0dd412dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd412b060 osd=7 3 2024-02-13T00:44:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd412b060 osd=7 3 2024-02-13T00:44:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd412b060 osd=7 3 2024-02-13T00:44:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 15 2024-02-13T00:44:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd412b060 osd=7 3 2024-02-13T00:44:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 7 15 2024-02-13T00:44:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd412b060 osd=7 4 2024-02-13T00:44:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd9d32700 10 client.37172.objecter _send_command 15 2024-02-13T00:44:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.941+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0dd412e270 con 0x7f0dd412d830 2024-02-13T00:44:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.942+0000 7f0dd37fe700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0dd412d830 0x7f0dd412dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.942+0000 7f0dd0ff9700 10 client.37172.objecter ms_handle_connect 0x7f0dd412d830 2024-02-13T00:44:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.942+0000 7f0dd37fe700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0dd412d830 0x7f0dd412dc90 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:31.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.943+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0dd412e270 con 0x7f0dd412d830 2024-02-13T00:44:31.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.943+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:44:31.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.943+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 15 = system:0 2024-02-13T00:44:31.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.943+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd412b060 osd=7 4 2024-02-13T00:44:31.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.943+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 7 15 2024-02-13T00:44:31.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.957+0000 7f0dd9d32700 10 client.37172.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:31.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.957+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign -1 16 2024-02-13T00:44:31.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.957+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd412b060 osd=7 3 2024-02-13T00:44:31.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.957+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd412b060 osd=7 3 2024-02-13T00:44:31.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.957+0000 7f0dd9d32700 20 client.37172.objecter _get_session s=0x7f0dd412b060 osd=7 3 2024-02-13T00:44:31.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.957+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_remove -1 16 2024-02-13T00:44:31.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.957+0000 7f0dd9d32700 20 client.37172.objecter get_session s=0x7f0dd412b060 osd=7 3 2024-02-13T00:44:31.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.957+0000 7f0dd9d32700 15 client.37172.objecter _session_command_op_assign 7 16 2024-02-13T00:44:31.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.957+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd412b060 osd=7 4 2024-02-13T00:44:31.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.957+0000 7f0dd9d32700 10 client.37172.objecter _send_command 16 2024-02-13T00:44:31.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.957+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0dd412e270 con 0x7f0dd412d830 2024-02-13T00:44:31.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd0ff9700 1 -- 172.21.15.162:0/3066150105 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0dd412e270 con 0x7f0dd412d830 2024-02-13T00:44:31.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd0ff9700 10 client.37172.objecter ms_dispatch 0x7f0dd4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:31.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd0ff9700 10 client.37172.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd0ff9700 20 client.37172.objecter put_session s=0x7f0dd412b060 osd=7 4 2024-02-13T00:44:31.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd0ff9700 15 client.37172.objecter _session_command_op_remove 7 16 2024-02-13T00:44:31.962 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:44:31.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 10 librados: watch_flush enter 2024-02-13T00:44:31.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 10 librados: watch_flush exit 2024-02-13T00:44:31.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 10 client.37172.objecter close_session for osd.0 2024-02-13T00:44:31.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0dd40fc180 msgr2=0x7f0dd40fc560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6801/2103743338 conn(0x7f0dd40fc180 0x7f0dd40fc560 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd40475e0 osd=0 1 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 10 client.37172.objecter close_session for osd.1 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6805/150599780 conn(0x7f0dd4059990 msgr2=0x7f0dd4059d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6805/150599780 conn(0x7f0dd4059990 0x7f0dd4059d90 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4028760 osd=1 1 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 10 client.37172.objecter close_session for osd.2 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0dd40282c0 msgr2=0x7f0dd411c190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6809/3689719376 conn(0x7f0dd40282c0 0x7f0dd411c190 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4059810 osd=2 1 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 10 client.37172.objecter close_session for osd.3 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6813/604971011 conn(0x7f0dd411f8f0 msgr2=0x7f0dd4121ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6813/604971011 conn(0x7f0dd411f8f0 0x7f0dd4121ce0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd3fff700 1 -- 172.21.15.162:0/3066150105 reap_dead start 2024-02-13T00:44:31.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4027c00 osd=3 1 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 10 client.37172.objecter close_session for osd.4 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0dd411bcf0 msgr2=0x7f0dd4123350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6800/3344263388 conn(0x7f0dd411bcf0 0x7f0dd4123350 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd411c8a0 osd=4 1 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 10 client.37172.objecter close_session for osd.5 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd3fff700 1 -- 172.21.15.162:0/3066150105 reap_dead start 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0dd4123dd0 msgr2=0x7f0dd4124250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6804/2212296748 conn(0x7f0dd4123dd0 0x7f0dd4124250 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd411ca80 osd=5 1 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 10 client.37172.objecter close_session for osd.6 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0dd4129000 msgr2=0x7f0dd4129460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6808/1506640341 conn(0x7f0dd4129000 0x7f0dd4129460 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd4128980 osd=6 1 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 10 client.37172.objecter close_session for osd.7 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0dd412d830 msgr2=0x7f0dd412dc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.181:6812/1964192772 conn(0x7f0dd412d830 0x7f0dd412dc90 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 20 client.37172.objecter put_session s=0x7f0dd412b060 osd=7 1 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 20 client.37172.objecter shutdown clearing up homeless session... 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 10 client.37172.objecter successfully canceled tick 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0dc0062c90 msgr2=0x7f0dc0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0dc0062c90 0x7f0dc0065150 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7f0dc4004de0 tx=0x7f0dc400b040).stop 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0dd41021d0 msgr2=0x7f0dd40f7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 --2- 172.21.15.162:0/3066150105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0dd41021d0 0x7f0dd40f7620 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 rx=0x7f0dcc004ed0 tx=0x7f0dcc004d10).stop 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd3fff700 1 -- 172.21.15.162:0/3066150105 reap_dead start 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 shutdown_connections 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.959+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 >> 172.21.15.162:0/3066150105 conn(0x7f0dd40f2740 msgr2=0x7f0dd40f3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.960+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 shutdown_connections 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.960+0000 7f0dd9d32700 1 -- 172.21.15.162:0/3066150105 wait complete. 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:31.960+0000 7f0dd9d32700 1 librados: shutdown 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.967 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.967 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.967 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:44:31.967 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:44:32.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:30.390357+0000 mgr.y (mgr.14544) 295 : cluster [DBG] pgmap v443: 267 pgs: 2 peering, 265 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:32.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:30.536967+0000 osd.2 (osd.2) 133 : cluster [DBG] 18.7 scrub starts 2024-02-13T00:44:32.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:30.539822+0000 osd.2 (osd.2) 134 : cluster [DBG] 18.7 scrub ok 2024-02-13T00:44:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:31.397436+0000 mon.a (mon.0) 2059 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_count","val": "8"}]': finished 2024-02-13T00:44:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:31.397520+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-83985-54"}]': finished 2024-02-13T00:44:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:31.397593+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e316: 8 total, 8 up, 6 in 2024-02-13T00:44:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:31.398489+0000 mon.b (mon.1) 471 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:44:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:31.398968+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:44:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:31.402903+0000 mon.b (mon.1) 472 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:31.419372+0000 mon.a (mon.0) 2063 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:31.420487+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:31.424566+0000 mon.a (mon.0) 2065 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi162-83985-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:32.096 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:44:32.106 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:32.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:32.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:32.238 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:32.788 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:32.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:31.078378+0000 osd.5 (osd.5) 139 : cluster [DBG] 61.d scrub starts 2024-02-13T00:44:32.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:31.080240+0000 osd.5 (osd.5) 140 : cluster [DBG] 61.d scrub ok 2024-02-13T00:44:32.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:31.805147+0000 mon.c (mon.2) 310 : audit [DBG] from='client.? 172.21.15.162:0/3066150105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:32.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:32.107201+0000 osd.0 (osd.0) 111 : cluster [DBG] 18.4 scrub starts 2024-02-13T00:44:32.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:32.110069+0000 osd.0 (osd.0) 112 : cluster [DBG] 18.4 scrub ok 2024-02-13T00:44:32.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:32.390940+0000 mon.a (mon.0) 2066 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:44:32.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:32.391001+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi162-83985-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:32.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:32.391049+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e317: 8 total, 8 up, 6 in 2024-02-13T00:44:32.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:32.391813+0000 mon.a (mon.0) 2069 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi162-83985-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:32.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:32.393297+0000 mon.b (mon.1) 473 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-13T00:44:32.791 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:32.394482+0000 mon.c (mon.2) 311 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-13T00:44:32.791 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:32.397506+0000 mon.a (mon.0) 2070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-13T00:44:32.791 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:32.398592+0000 mon.a (mon.0) 2071 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-13T00:44:32.791 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:32.408383+0000 mon.b (mon.1) 474 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:32.791 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:31.078378+0000 osd.5 (osd.5) 139 : cluster [DBG] 61.d scrub starts 2024-02-13T00:44:32.791 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:31.080240+0000 osd.5 (osd.5) 140 : cluster [DBG] 61.d scrub ok 2024-02-13T00:44:32.791 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:31.805147+0000 mon.c (mon.2) 310 : audit [DBG] from='client.? 172.21.15.162:0/3066150105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:32.791 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:32.107201+0000 osd.0 (osd.0) 111 : cluster [DBG] 18.4 scrub starts 2024-02-13T00:44:32.791 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:32.110069+0000 osd.0 (osd.0) 112 : cluster [DBG] 18.4 scrub ok 2024-02-13T00:44:32.791 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:32.390940+0000 mon.a (mon.0) 2066 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:44:32.791 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:32.391001+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi162-83985-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:32.791 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:32.391049+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e317: 8 total, 8 up, 6 in 2024-02-13T00:44:32.792 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:32.391813+0000 mon.a (mon.0) 2069 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi162-83985-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:32.792 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:32.393297+0000 mon.b (mon.1) 473 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-13T00:44:32.792 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:32.394482+0000 mon.c (mon.2) 311 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-13T00:44:32.792 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:32.397506+0000 mon.a (mon.0) 2070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-13T00:44:32.792 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:32.398592+0000 mon.a (mon.0) 2071 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-13T00:44:32.792 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:32.408383+0000 mon.b (mon.1) 474 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:32.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:32.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:32.921 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:32.940 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:32.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:33.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:31.078378+0000 osd.5 (osd.5) 139 : cluster [DBG] 61.d scrub starts 2024-02-13T00:44:33.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:31.080240+0000 osd.5 (osd.5) 140 : cluster [DBG] 61.d scrub ok 2024-02-13T00:44:33.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:31.805147+0000 mon.c (mon.2) 310 : audit [DBG] from='client.? 172.21.15.162:0/3066150105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:33.020 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:32.107201+0000 osd.0 (osd.0) 111 : cluster [DBG] 18.4 scrub starts 2024-02-13T00:44:33.021 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:32.110069+0000 osd.0 (osd.0) 112 : cluster [DBG] 18.4 scrub ok 2024-02-13T00:44:33.021 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:32.390940+0000 mon.a (mon.0) 2066 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:44:33.021 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:32.391001+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi162-83985-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:33.021 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:32.391049+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e317: 8 total, 8 up, 6 in 2024-02-13T00:44:33.021 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:32.391813+0000 mon.a (mon.0) 2069 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi162-83985-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:33.021 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:32.393297+0000 mon.b (mon.1) 473 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-13T00:44:33.021 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:32.394482+0000 mon.c (mon.2) 311 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-13T00:44:33.021 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:32.397506+0000 mon.a (mon.0) 2070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-13T00:44:33.021 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:32.398592+0000 mon.a (mon.0) 2071 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-13T00:44:33.021 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:32.408383+0000 mon.b (mon.1) 474 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:33.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:33.079 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:33.728 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:33.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:33.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:32.176610+0000 osd.1 (osd.1) 129 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:44:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:32.180109+0000 osd.1 (osd.1) 130 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:44:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:32.390754+0000 mgr.y (mgr.14544) 296 : cluster [DBG] pgmap v445: 266 pgs: 1 peering, 265 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 63 B/s, 0 objects/s recovering 2024-02-13T00:44:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:33.107865+0000 osd.0 (osd.0) 113 : cluster [DBG] 28.e scrub starts 2024-02-13T00:44:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:33.110738+0000 osd.0 (osd.0) 114 : cluster [DBG] 28.e scrub ok 2024-02-13T00:44:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:33.409314+0000 mon.b (mon.1) 475 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:33.633689+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-13T00:44:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:33.633773+0000 mon.a (mon.0) 2073 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-13T00:44:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:33.633832+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e318: 8 total, 8 up, 6 in 2024-02-13T00:44:33.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:33.849 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9012 ms) 2024-02-13T00:44:33.849 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-02-13T00:44:33.849 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg_num = 32 2024-02-13T00:44:33.849 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 0 ls 1707785074,0 2024-02-13T00:44:33.849 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 1 ls 1707785074,0 2024-02-13T00:44:33.849 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 2 ls 1707785074,0 2024-02-13T00:44:33.849 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 3 ls 1707785074,0 2024-02-13T00:44:33.849 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 4 ls 1707785074,0 2024-02-13T00:44:33.849 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 5 ls 1707785074,0 2024-02-13T00:44:33.849 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 6 ls 1707785074,0 2024-02-13T00:44:33.849 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 7 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 8 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 9 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 10 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 11 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 12 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 13 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 14 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 15 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 16 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 17 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 18 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 19 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 20 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 21 ls 1707785074,0 2024-02-13T00:44:33.850 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 22 ls 1707785074,0 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 23 ls 1707785074,0 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 24 ls 1707785074,0 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 25 ls 1707785074,0 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 26 ls 1707785074,0 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 27 ls 1707785074,0 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 28 ls 1707785074,0 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 29 ls 1707785074,0 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 30 ls 1707785074,0 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 31 ls 1707785074,0 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg_num = 32 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:6cac518f:::0:head 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:02547ec2:::1:head 2024-02-13T00:44:33.851 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:f905c69b:::2:head 2024-02-13T00:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:cfc208b3:::3:head 2024-02-13T00:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:d83876eb:::4:head 2024-02-13T00:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:b29083e3:::5:head 2024-02-13T00:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:c4fdafeb:::6:head 2024-02-13T00:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:5c6b0b28:::7:head 2024-02-13T00:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:bd63b0f1:::8:head 2024-02-13T00:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:e960b815:::9:head 2024-02-13T00:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:52ea6a34:::10:head 2024-02-13T00:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:89d3ae78:::11:head 2024-02-13T00:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:de5d7c5f:::12:head 2024-02-13T00:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:566253c9:::13:head 2024-02-13T00:44:33.852 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:62a1935d:::14:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:863748b0:::15:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:3958e169:::16:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:4d4dabf9:::17:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:8391935d:::18:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:28883081:::19:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:69259c59:::20:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:4bdb80b7:::21:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:a11c5d71:::22:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:271af37b:::23:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:95b121be:::24:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:58d1031b:::25:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:0a050783:::26:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:c709704c:::27:head 2024-02-13T00:44:33.853 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:cbe56eaf:::28:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:86b4b162:::29:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:70d89383:::30:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:dd450c7c:::31:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:6d5729b1:::32:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:c388f3fb:::33:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:56cfea31:::34:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:9dbc1bf7:::35:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:40b74ccd:::36:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:4d5aaf42:::37:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:920f362c:::38:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:6cc53222:::39:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:9cad833f:::40:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:1ea84d41:::41:head 2024-02-13T00:44:33.854 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:c4480ef6:::42:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:a694361e:::43:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:d1bd33e9:::44:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:ddc2cd5d:::45:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:2b782207:::46:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:7b187fca:::47:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:90ecdf6f:::48:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:a5ed95fe:::49:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:ea0eaa55:::50:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:f33ef17b:::51:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:a0d1b2f6:::52:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:60c5229e:::53:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:edcbc575:::54:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:102cf253:::55:head 2024-02-13T00:44:33.855 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:efb7fb0b:::56:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:50d0a326:::57:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:d4dc5daf:::58:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:3a130462:::59:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:ec87ed71:::60:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:d5bc9454:::61:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:3ddfe313:::62:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:7c2816b9:::63:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:47e00e4d:::64:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:c6410c18:::65:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:b48ed237:::66:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:cd63ad31:::67:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:b179e92b:::68:head 2024-02-13T00:44:33.856 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:0d9f741a:::69:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:6d3352ae:::70:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:c6d5c19e:::71:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:bc4729c3:::72:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:77e930b9:::73:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:0abeecfd:::74:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:b7c37e15:::75:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:b6378398:::76:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:02bd68de:::77:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:cc795d2d:::78:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:630d4fea:::79:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:e0d29ef5:::80:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:fd6f13d2:::81:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:606461d5:::82:head 2024-02-13T00:44:33.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:eadbdc43:::83:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:8761d0bb:::84:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:9ef0186f:::85:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:e0d41294:::86:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:961de695:::87:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:1423148f:::88:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:633a8fa2:::89:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:a8653809:::90:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:3dac8b33:::91:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:35aad435:::92:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:f6dcc343:::93:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:dbbdad87:::94:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:1cb48ce0:::95:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:03cd461c:::96:head 2024-02-13T00:44:33.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:17a4ea99:::97:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:9993c9a7:::98:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:6394211c:::99:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:94c7ae57:::100:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:6fdee5bb:::101:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:9a477fd1:::102:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:eb850916:::103:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:affc56b9:::104:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:b42dc814:::105:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:f319f8f0:::106:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:9a40b9de:::107:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:8b524f28:::108:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:e3de589f:::109:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:90f90a5b:::110:head 2024-02-13T00:44:33.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:a7b4f1d7:::111:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:af51766e:::112:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:b6f90bd1:::113:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:e0261208:::114:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:c9569ef7:::115:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:61bebe50:::116:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:fe93412b:::117:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:d3d38bee:::118:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:3100ba0c:::119:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:d0560ada:::120:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:f0ea8b35:::121:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:766f231a:::122:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:a07a2582:::123:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:bd7c6b3a:::124:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:fb2ddaff:::125:head 2024-02-13T00:44:33.860 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:4408e1fe:::126:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:ee1df7a7:::127:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:c3002909:::128:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:4f48ffa9:::129:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:edf38733:::130:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:c08425c0:::131:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:5f902d98:::132:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:41ea2c93:::133:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:813cee13:::134:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:0131818d:::135:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:26ba5a85:::136:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:381b8a5a:::137:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:28797e47:::138:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:bfca7f22:::139:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:36807075:::140:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:80b03975:::141:head 2024-02-13T00:44:33.861 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:5c15709b:::142:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:f39ea15e:::143:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:ea992956:::144:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:48887b1c:::145:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:9f24a9dd:::146:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:987f100b:::147:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:d2dd3581:::148:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:7fed1808:::149:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:c80b70e9:::150:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:85ed90f9:::151:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:36428b24:::152:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:d044c34a:::153:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:7c18bf58:::154:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:d1c21232:::155:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:a7a3c575:::156:head 2024-02-13T00:44:33.862 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:87da0633:::157:head 2024-02-13T00:44:33.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:d5ac3822:::158:head 2024-02-13T00:44:33.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:3f20522d:::159:head 2024-02-13T00:44:33.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:6ca26563:::160:head 2024-02-13T00:44:33.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:532ce135:::161:head 2024-02-13T00:44:33.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:c78863e6:::162:head 2024-02-13T00:44:33.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:7462ddf6:::163:head 2024-02-13T00:44:33.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:5d165639:::164:head 2024-02-13T00:44:33.863 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:f43765fc:::165:head 2024-02-13T00:44:33.863 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:33.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:32.176610+0000 osd.1 (osd.1) 129 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:44:33.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:32.180109+0000 osd.1 (osd.1) 130 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:44:33.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:32.390754+0000 mgr.y (mgr.14544) 296 : cluster [DBG] pgmap v445: 266 pgs: 1 peering, 265 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 63 B/s, 0 objects/s recovering 2024-02-13T00:44:33.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:33.107865+0000 osd.0 (osd.0) 113 : cluster [DBG] 28.e scrub starts 2024-02-13T00:44:33.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:33.110738+0000 osd.0 (osd.0) 114 : cluster [DBG] 28.e scrub ok 2024-02-13T00:44:33.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:33.409314+0000 mon.b (mon.1) 475 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:33.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:33.633689+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-13T00:44:33.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:33.633773+0000 mon.a (mon.0) 2073 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-13T00:44:33.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:33.633832+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e318: 8 total, 8 up, 6 in 2024-02-13T00:44:33.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:32.176610+0000 osd.1 (osd.1) 129 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:44:33.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:32.180109+0000 osd.1 (osd.1) 130 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:44:33.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:32.390754+0000 mgr.y (mgr.14544) 296 : cluster [DBG] pgmap v445: 266 pgs: 1 peering, 265 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 63 B/s, 0 objects/s recovering 2024-02-13T00:44:33.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:33.107865+0000 osd.0 (osd.0) 113 : cluster [DBG] 28.e scrub starts 2024-02-13T00:44:33.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:33.110738+0000 osd.0 (osd.0) 114 : cluster [DBG] 28.e scrub ok 2024-02-13T00:44:33.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:33.409314+0000 mon.b (mon.1) 475 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:33.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:33.633689+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-41","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-13T00:44:33.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:33.633773+0000 mon.a (mon.0) 2073 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-13T00:44:33.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:33.633832+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e318: 8 total, 8 up, 6 in 2024-02-13T00:44:33.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:33.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:34.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:34.105 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24377.0:528 4.9 4:9195808b:::obj_delete_at_hint.0000000027:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:37:48.430264+0000", 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 405.67054821400001, 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003660361, 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24377", 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:44:34.106 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 528 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.430264+0000", 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.430264+0000", 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.430263+0000", 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2140000000000004e-06 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.430268+0000", 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8870000000000004e-06 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.430278+0000", 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5430000000000004e-06 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.430282+0000", 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3047999999999999e-05 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.430316+0000", 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3547999999999994e-05 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.430389+0000", 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011972900000000001 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.430509+0000", 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025313400000000002 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.430762+0000", 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00070287799999999999 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.431465+0000", 2024-02-13T00:44:34.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024128930000000002 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.433878+0000", 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6709000000000001e-05 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.433904+0000", 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9792999999999998e-05 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.433924+0000", 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003660361 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24377.0:623 4.9 4:954be8d2:gc::gc.5:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:37:48.456190+0000", 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 405.64462201600003, 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0037092359999999999, 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.112 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24377", 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 623 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456190+0000", 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456190+0000", 2024-02-13T00:44:34.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:44:34.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456189+0000", 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6030000000000001e-06 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456196+0000", 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1809e-05 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456207+0000", 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4989999999999999e-06 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456214+0000", 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1881e-05 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456266+0000", 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3955000000000005e-05 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456350+0000", 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013941100000000001 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456489+0000", 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061411899999999999 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.457103+0000", 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001897262 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.459001+0000", 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00084291699999999999 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.459843+0000", 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5435000000000002e-05 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.459879+0000", 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0389000000000001e-05 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.459899+0000", 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0037092359999999999 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.119 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24377.0:944 4.e 4:71a6e5bc:lc::lc.2:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:37:53.331569+0000", 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 400.76924313699999, 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056003779999999996, 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24377", 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:44:34.120 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 944 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331569+0000", 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331569+0000", 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331566+0000", 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0927000000000001e-05 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331577+0000", 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6793e-05 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331594+0000", 2024-02-13T00:44:34.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1973999999999999e-05 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331606+0000", 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8511999999999999e-05 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331655+0000", 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6979999999999997e-06 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331659+0000", 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002240658 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.333900+0000", 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103423 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.334003+0000", 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0002767 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.334280+0000", 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027515080000000002 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.337032+0000", 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8433000000000002e-05 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.337080+0000", 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4619999999999999e-06 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.337088+0000", 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7343999999999999e-05 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.337135+0000", 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4400000000000003e-05 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.337169+0000", 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056003779999999996 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.126 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24748.0:2 18.6 e107/105 18:602f83fe:::foo:head v 107'1, mlcod=0'0)", 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.642263+0000", 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 212.45854858600001, 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057417939999999997, 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642263+0000", 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642263+0000", 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642263+0000", 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2369999999999998e-06 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642272+0000", 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9379999999999997e-06 2024-02-13T00:44:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642278+0000", 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2829999999999998e-06 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642285+0000", 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000456229 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642741+0000", 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3309e-05 2024-02-13T00:44:34.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642764+0000", 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0052050890000000004 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.647969+0000", 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6060999999999999e-05 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.648005+0000", 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057417939999999997 2024-02-13T00:44:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14862.0:4 12.16 e107/104 12:681f27cd:::2.obj:head v 107'1, mlcod=0'0)", 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.655596+0000", 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 212.44521630400001, 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0085331920000000002, 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.655596+0000", 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.655596+0000", 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.655595+0000", 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1340000000000004e-06 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.655603+0000", 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.273e-05 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.655616+0000", 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0099999999999998e-06 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.655623+0000", 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.007108006 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.662731+0000", 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2544999999999999e-05 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.662743+0000", 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001350221 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.664094+0000", 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5250000000000003e-05 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.664129+0000", 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0085331920000000002 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.2.0:191 20.19 e109/105 -22:9c8ffe76:::temp_20.19_0_14286_1:head v 0'0, mlcod=0'0)", 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.448877+0000", 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.65193470400001, 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057776421000000001, 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.135 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.448877+0000", 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.448877+0000", 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.448877+0000", 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.026996183 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475873+0000", 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2881e-05 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475896+0000", 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8559999999999996e-06 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475905+0000", 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6680000000000004e-05 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475981+0000", 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0239999999999999e-05 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.476002+0000", 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.030597243999999999 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.506599+0000", 2024-02-13T00:44:34.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4880000000000003e-05 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.506654+0000", 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057776421000000001 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.2.0:192 20.19 e109/105 -22:9c8ffe76:::temp_20.19_0_14286_1:head v 0'0, mlcod=0'0)", 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.475919+0000", 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.62489350800001, 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.046592573999999998, 2024-02-13T00:44:34.139 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475919+0000", 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475919+0000", 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475918+0000", 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014380096 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.490298+0000", 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8105e-05 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.490316+0000", 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7569999999999999e-06 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.490325+0000", 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4147000000000002e-05 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.490380+0000", 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8797999999999999e-05 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.490398+0000", 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.03208714 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.522486+0000", 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5590999999999999e-05 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.522511+0000", 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.046592573999999998 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.2.0:193 20.19 e109/105 -22:9c8ffe76:::temp_20.19_0_14286_1:head v 0'0, mlcod=0'0)", 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.491530+0000", 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.60928241100001, 2024-02-13T00:44:34.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.033860855000000002, 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.491530+0000", 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.491530+0000", 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.491528+0000", 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.01643648 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507965+0000", 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2159e-05 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507977+0000", 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1469999999999998e-06 2024-02-13T00:44:34.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507983+0000", 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2548000000000001e-05 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.508016+0000", 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7958999999999999e-05 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.508034+0000", 2024-02-13T00:44:34.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017333595 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.525367+0000", 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3408999999999999e-05 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.525391+0000", 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.033860855000000002 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2064 20.19 e109/105 20:9c8ffe76:test-rados-api-smithi162-84049-16::big.copy2:head v 109'1, mlcod=0'0)", 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.507993+0000", 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.59281912500001, 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017450688999999998, 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507993+0000", 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507993+0000", 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507992+0000", 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.891e-06 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507998+0000", 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4490000000000001e-06 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.508002+0000", 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7609999999999999e-06 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.508005+0000", 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003349201 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.511354+0000", 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1878999999999998e-05 2024-02-13T00:44:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.511376+0000", 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014054255999999999 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.525430+0000", 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3195e-05 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.525444+0000", 2024-02-13T00:44:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017450688999999998 2024-02-13T00:44:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2069 20.1b e109/105 20:de4afda3:test-rados-api-smithi162-84049-17::big:head v 109'2, mlcod=0'0)", 2024-02-13T00:44:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.641911+0000", 2024-02-13T00:44:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.45890102600001, 2024-02-13T00:44:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099432847000000005, 2024-02-13T00:44:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.641911+0000", 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.641911+0000", 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.641910+0000", 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.044861537 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.686771+0000", 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9842e-05 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.686801+0000", 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8167e-05 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.686829+0000", 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5619999999999999e-05 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.686885+0000", 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7118999999999999e-05 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.686912+0000", 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.054351272999999999 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.741263+0000", 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0814000000000002e-05 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.741344+0000", 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099432847000000005 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2070 20.b e109/105 20:d3a22371:test-rados-api-smithi162-84049-17::big2:head v 109'1, mlcod=0'0)", 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.754348+0000", 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.346464019, 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088186930999999996, 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754348+0000", 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754348+0000", 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754347+0000", 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.034034943999999998 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.788381+0000", 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3618e-05 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.788405+0000", 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4919999999999993e-06 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.788414+0000", 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6088000000000003e-05 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.788460+0000", 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9964999999999999e-05 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.788480+0000", 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.053965705000000003 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842445+0000", 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9630999999999996e-05 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842535+0000", 2024-02-13T00:44:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088186930999999996 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2073 20.b e109/105 20:d3a22371:test-rados-api-smithi162-84049-17::big2:head v 109'4, mlcod=109'2)", 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.849406+0000", 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.251406514, 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.01207979, 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.849406+0000", 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.849406+0000", 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.849404+0000", 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047755940000000002 2024-02-13T00:44:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854179+0000", 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4504999999999999e-05 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854194+0000", 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9820000000000002e-06 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854201+0000", 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9499000000000002e-05 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854240+0000", 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7659000000000002e-05 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854258+0000", 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071836679999999998 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.861442+0000", 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3723000000000001e-05 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.861485+0000", 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.01207979 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2074 20.1b e109/105 20:de8a7420:test-rados-api-smithi162-84049-17::big3:head v 109'3, mlcod=107'1)", 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.870270+0000", 2024-02-13T00:44:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.23054257300001, 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094008466999999998, 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.870270+0000", 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.870270+0000", 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999938 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.870263+0000", 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049442282999999997 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.919706+0000", 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8632e-05 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.919734+0000", 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8073e-05 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.919762+0000", 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5293000000000001e-05 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.919818+0000", 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1254e-05 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.919839+0000", 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.044407744999999998 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.964247+0000", 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1446999999999999e-05 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.964278+0000", 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094008466999999998 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2076 20.1b e109/105 20:de8a7420:test-rados-api-smithi162-84049-17::big3:head v 109'5, mlcod=109'3)", 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.970516+0000", 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 209.13029644900001, 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0122755, 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.970516+0000", 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.970516+0000", 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:44:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.970514+0000", 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048866400000000003 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.975401+0000", 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3018e-05 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.975414+0000", 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.849e-06 2024-02-13T00:44:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.975420+0000", 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3612000000000001e-05 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.975474+0000", 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3119e-05 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.975497+0000", 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0072508829999999996 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.982748+0000", 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3099000000000002e-05 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.982791+0000", 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0122755 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.5.0:282 60.a e128/120 60:51f054f9:test-rados-api-smithi162-84073-8::foo:7 v 128'6, mlcod=127'3)", 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:22.774235+0000", 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 191.326577431, 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.007417252, 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774235+0000", 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774235+0000", 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774234+0000", 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4309999999999996e-06 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774239+0000", 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3290000000000001e-06 2024-02-13T00:44:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774244+0000", 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8940000000000003e-06 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774248+0000", 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000423449 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774671+0000", 2024-02-13T00:44:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7167e-05 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774689+0000", 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0069276140000000003 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.781616+0000", 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5657000000000001e-05 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.781652+0000", 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.007417252 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.16275.0:11 66.6 e130/124 66:629bc174:test-rados-api-smithi162-84081-9::foo:head v 130'7, mlcod=128'5)", 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:24.903032+0000", 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 189.197780153, 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041090170000000004, 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903032+0000", 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903032+0000", 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903031+0000", 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3899999999999999e-06 2024-02-13T00:44:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903040+0000", 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0533999999999999e-05 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903051+0000", 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0879999999999997e-06 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903058+0000", 2024-02-13T00:44:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6895000000000002e-05 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903135+0000", 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2158999999999999e-05 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903157+0000", 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039450989999999997 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.907102+0000", 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9041000000000001e-05 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.907141+0000", 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041090170000000004 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.5.0:293 66.6 e132/124 66:629bc174:test-rados-api-smithi162-84081-9::foo:7 v 132'15, mlcod=131'12)", 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:27.091552+0000", 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 187.00926015600001, 2024-02-13T00:44:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084929270000000008, 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091552+0000", 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091552+0000", 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091550+0000", 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1979e-05 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091562+0000", 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2990000000000002e-06 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091572+0000", 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1720000000000001e-06 2024-02-13T00:44:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091579+0000", 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000886285 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.092465+0000", 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2421000000000001e-05 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.092488+0000", 2024-02-13T00:44:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075259120000000001 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.100013+0000", 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1429e-05 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.100045+0000", 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084929270000000008 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.2.0:365 61.4 e188/121 61:21c2c914:test-rados-api-smithi162-84057-14::foo8:7 v 188'36, mlcod=154'30)", 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:42:23.329308+0000", 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 130.77150454299999, 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0089267670000000004, 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329308+0000", 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329308+0000", 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329307+0000", 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3189999999999996e-06 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329312+0000", 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.648e-06 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329317+0000", 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3749999999999996e-06 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329322+0000", 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00029372900000000001 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329615+0000", 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7045999999999999e-05 2024-02-13T00:44:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329632+0000", 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0085733790000000008 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.338206+0000", 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8561e-05 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.338234+0000", 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0089267670000000004 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.5.0:512 61.19 e188/121 61:9a2cc200:test-rados-api-smithi162-84057-14::foo19:7 v 188'11, mlcod=153'7)", 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:42:23.330027+0000", 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 130.77078487700001, 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073812160000000003, 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330027+0000", 2024-02-13T00:44:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330027+0000", 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330026+0000", 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5390000000000002e-06 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330034+0000", 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0818999999999999e-05 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330045+0000", 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1320000000000002e-06 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330050+0000", 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000152163 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330202+0000", 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9652000000000001e-05 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330222+0000", 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071553379999999998 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.337377+0000", 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1399999999999998e-05 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.337408+0000", 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073812160000000003 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.25162.0:3128 184.0 e300/296 184:00adb854:test-rados-api-smithi162-84057-38::foo:head v 300'2, mlcod=0'0)", 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:15.383766+0000", 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 18.717045862999999, 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013839536, 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.383766+0000", 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.383766+0000", 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.383765+0000", 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002561158 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386326+0000", 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7059e-05 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386344+0000", 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.259e-06 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386352+0000", 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014103499999999999 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386493+0000", 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2059999999999999e-05 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386515+0000", 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011046402 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.397561+0000", 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4487000000000003e-05 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.397606+0000", 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013839536 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:34.199 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:44:34.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:34.201 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:34.201 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:34.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:34.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 -- 172.21.15.162:0/1496223601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2984102170 msgr2=0x7f2984104560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:34.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 --2- 172.21.15.162:0/1496223601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2984102170 0x7f2984104560 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 rx=0x7f297c009be0 tx=0x7f297c004d10).stop 2024-02-13T00:44:34.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 -- 172.21.15.162:0/1496223601 shutdown_connections 2024-02-13T00:44:34.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 --2- 172.21.15.162:0/1496223601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2984102170 0x7f2984104560 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:34.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 --2- 172.21.15.162:0/1496223601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29840f8c10 0x7f29840f8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:34.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 --2- 172.21.15.162:0/1496223601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29840f9570 0x7f2984101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:34.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 -- 172.21.15.162:0/1496223601 >> 172.21.15.162:0/1496223601 conn(0x7f29840f2740 msgr2=0x7f29840f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:34.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 -- 172.21.15.162:0/1496223601 shutdown_connections 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 -- 172.21.15.162:0/1496223601 wait complete. 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 librados: starting msgr at 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 librados: starting objecter 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 Processor -- start 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 -- start start 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 librados: setting wanted keys 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.220+0000 7f298c091700 1 librados: calling monclient init 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298c091700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29840f8c10 0x7f298410dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298c091700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29840f9570 0x7f298410e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298c091700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2984102170 0x7f29841081d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298c091700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29841086c0 con 0x7f2984102170 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298c091700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2984108830 con 0x7f29840f9570 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298c091700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29841089a0 con 0x7f29840f8c10 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f2989e2d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29840f8c10 0x7f298410dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:34.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298a62e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2984102170 0x7f29841081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f2989e2d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29840f8c10 0x7f298410dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:47996/0 (socket says 172.21.15.162:47996) 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f2989e2d700 1 -- 172.21.15.162:0/3812509340 learned_addr learned my addr 172.21.15.162:0/3812509340 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298962c700 1 --2- 172.21.15.162:0/3812509340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29840f9570 0x7f298410e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f2989e2d700 1 -- 172.21.15.162:0/3812509340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29840f9570 msgr2=0x7f298410e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f2989e2d700 1 --2- 172.21.15.162:0/3812509340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29840f9570 0x7f298410e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f2989e2d700 1 -- 172.21.15.162:0/3812509340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2984102170 msgr2=0x7f29841081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f2989e2d700 1 --2- 172.21.15.162:0/3812509340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2984102170 0x7f29841081d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f2989e2d700 1 -- 172.21.15.162:0/3812509340 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2980005040 con 0x7f29840f8c10 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298a62e700 1 --2- 172.21.15.162:0/3812509340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2984102170 0x7f29841081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f2989e2d700 1 --2- 172.21.15.162:0/3812509340 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29840f8c10 0x7f298410dff0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7f29800048f0 tx=0x7f2980009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f297affd700 10 client.?.objecter ms_handle_connect 0x7f29840f8c10 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f297affd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:34.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f297affd700 1 -- 172.21.15.162:0/3812509340 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2980007b00 con 0x7f29840f8c10 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f297affd700 1 -- 172.21.15.162:0/3812509340 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f298000f040 con 0x7f29840f8c10 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298c091700 1 -- 172.21.15.162:0/3812509340 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297c0097c0 con 0x7f29840f8c10 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f297affd700 1 -- 172.21.15.162:0/3812509340 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2980014990 con 0x7f29840f8c10 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298c091700 10 client.37178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298c091700 1 -- 172.21.15.162:0/3812509340 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2984108fe0 con 0x7f29840f8c10 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.221+0000 7f298c091700 1 librados: init done 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.222+0000 7f298c091700 1 -- 172.21.15.162:0/3812509340 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29840607b0 con 0x7f29840f8c10 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.223+0000 7f297affd700 1 -- 172.21.15.162:0/3812509340 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f2980010500 con 0x7f29840f8c10 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.223+0000 7f297affd700 1 --2- 172.21.15.162:0/3812509340 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2970062c90 0x7f2970065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.223+0000 7f297affd700 1 -- 172.21.15.162:0/3812509340 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(318..318 src has 52..318) v4 ==== 8767+0+0 (secure 0 0 0) 0x7f298007d000 con 0x7f29840f8c10 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.223+0000 7f297affd700 10 client.37178.objecter ms_dispatch 0x7f2984001550 osd_map(318..318 src has 52..318) v4 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.223+0000 7f297affd700 3 client.37178.objecter handle_osd_map got epochs [318,318] > 0 2024-02-13T00:44:34.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.223+0000 7f297affd700 3 client.37178.objecter handle_osd_map decoding full epoch 318 2024-02-13T00:44:34.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.223+0000 7f298962c700 1 --2- 172.21.15.162:0/3812509340 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2970062c90 0x7f2970065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:34.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.223+0000 7f297affd700 20 client.37178.objecter dump_active .. 0 homeless 2024-02-13T00:44:34.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.223+0000 7f297affd700 10 client.37178.objecter ms_handle_connect 0x7f2970062c90 2024-02-13T00:44:34.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.223+0000 7f298962c700 1 --2- 172.21.15.162:0/3812509340 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2970062c90 0x7f2970065150 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 rx=0x7f2984109970 tx=0x7f297400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:34.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.227+0000 7f297affd700 1 -- 172.21.15.162:0/3812509340 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2980049f60 con 0x7f29840f8c10 2024-02-13T00:44:34.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:34.330 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:34.350 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:34.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:34.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:34.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:34.445+0000 7f298c091700 1 -- 172.21.15.162:0/3812509340 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f29840475e0 con 0x7f29840f8c10 2024-02-13T00:44:34.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:34.483 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:34.504 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:34.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:34.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:34.635 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:34.653 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:34.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:34.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:34.784 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:33.101863+0000 osd.5 (osd.5) 141 : cluster [DBG] 28.9 scrub starts 2024-02-13T00:44:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:33.104899+0000 osd.5 (osd.5) 142 : cluster [DBG] 28.9 scrub ok 2024-02-13T00:44:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:33.846193+0000 mon.b (mon.1) 476 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi162-84057-41","var": "pg_num","format": "json"}]: dispatch 2024-02-13T00:44:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:33.947801+0000 mon.b (mon.1) 477 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:33.948309+0000 mon.a (mon.0) 2075 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:33.948665+0000 mon.b (mon.1) 478 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41"}]: dispatch 2024-02-13T00:44:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:33.949012+0000 mon.a (mon.0) 2076 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41"}]: dispatch 2024-02-13T00:44:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:34.335578+0000 mon.a (mon.0) 2077 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:34.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:34.399354+0000 mon.a (mon.0) 2078 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi162-83985-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi162-83985-55"}]': finished 2024-02-13T00:44:34.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:34.399431+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41"}]': finished 2024-02-13T00:44:34.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:34.399551+0000 mon.a (mon.0) 2080 : cluster [DBG] osdmap e319: 8 total, 8 up, 6 in 2024-02-13T00:44:34.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:34.411020+0000 mon.b (mon.1) 479 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:34.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:34.446678+0000 mon.c (mon.2) 312 : audit [INF] from='client.? 172.21.15.162:0/3812509340' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:44:34.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:34.447056+0000 mon.a (mon.0) 2081 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:44:34.803 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:34.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:34.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:34.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:34.928 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-13T00:44:34.936 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:34.955 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:34.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:34.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:35.031 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7] out_osds: [6, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:44:35.031 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:44:35.031 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-02-13T00:44:35.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd in 6 2024-02-13T00:44:35.058 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:33.101863+0000 osd.5 (osd.5) 141 : cluster [DBG] 28.9 scrub starts 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:33.104899+0000 osd.5 (osd.5) 142 : cluster [DBG] 28.9 scrub ok 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:33.846193+0000 mon.b (mon.1) 476 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi162-84057-41","var": "pg_num","format": "json"}]: dispatch 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:33.947801+0000 mon.b (mon.1) 477 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:33.948309+0000 mon.a (mon.0) 2075 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:33.948665+0000 mon.b (mon.1) 478 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41"}]: dispatch 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:33.949012+0000 mon.a (mon.0) 2076 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41"}]: dispatch 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:34.335578+0000 mon.a (mon.0) 2077 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:34.399354+0000 mon.a (mon.0) 2078 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi162-83985-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi162-83985-55"}]': finished 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:34.399431+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41"}]': finished 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:34.399551+0000 mon.a (mon.0) 2080 : cluster [DBG] osdmap e319: 8 total, 8 up, 6 in 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:34.411020+0000 mon.b (mon.1) 479 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:34.446678+0000 mon.c (mon.2) 312 : audit [INF] from='client.? 172.21.15.162:0/3812509340' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:44:35.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:34.447056+0000 mon.a (mon.0) 2081 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:44:35.060 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:33.101863+0000 osd.5 (osd.5) 141 : cluster [DBG] 28.9 scrub starts 2024-02-13T00:44:35.060 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:33.104899+0000 osd.5 (osd.5) 142 : cluster [DBG] 28.9 scrub ok 2024-02-13T00:44:35.060 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:33.846193+0000 mon.b (mon.1) 476 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi162-84057-41","var": "pg_num","format": "json"}]: dispatch 2024-02-13T00:44:35.060 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:33.947801+0000 mon.b (mon.1) 477 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:35.060 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:33.948309+0000 mon.a (mon.0) 2075 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:35.060 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:33.948665+0000 mon.b (mon.1) 478 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41"}]: dispatch 2024-02-13T00:44:35.060 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:33.949012+0000 mon.a (mon.0) 2076 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41"}]: dispatch 2024-02-13T00:44:35.060 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:34.335578+0000 mon.a (mon.0) 2077 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:35.060 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:34.399354+0000 mon.a (mon.0) 2078 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi162-83985-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi162-83985-55"}]': finished 2024-02-13T00:44:35.060 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:34.399431+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-41"}]': finished 2024-02-13T00:44:35.060 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:34.399551+0000 mon.a (mon.0) 2080 : cluster [DBG] osdmap e319: 8 total, 8 up, 6 in 2024-02-13T00:44:35.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:34.411020+0000 mon.b (mon.1) 479 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:35.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:34.446678+0000 mon.c (mon.2) 312 : audit [INF] from='client.? 172.21.15.162:0/3812509340' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:44:35.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:34.447056+0000 mon.a (mon.0) 2081 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:44:35.061 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:35.087 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:35.106 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:35.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:35.202 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:44:35.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:35.237 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:35.298 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:35.435 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:35.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:35.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.463+0000 7f297affd700 1 -- 172.21.15.162:0/3812509340 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v320) v1 ==== 87+0+0 (secure 0 0 0) 0x7f2980050210 con 0x7f29840f8c10 2024-02-13T00:44:35.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.466+0000 7f298c091700 10 librados: watch_flush enter 2024-02-13T00:44:35.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.468+0000 7f298c091700 10 librados: watch_flush exit 2024-02-13T00:44:35.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.468+0000 7f298c091700 20 client.37178.objecter shutdown clearing up homeless session... 2024-02-13T00:44:35.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.468+0000 7f298c091700 10 client.37178.objecter successfully canceled tick 2024-02-13T00:44:35.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.469+0000 7f298c091700 1 -- 172.21.15.162:0/3812509340 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2970062c90 msgr2=0x7f2970065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:35.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.469+0000 7f298c091700 1 --2- 172.21.15.162:0/3812509340 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2970062c90 0x7f2970065150 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 rx=0x7f2984109970 tx=0x7f297400b040).stop 2024-02-13T00:44:35.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.470+0000 7f298c091700 1 -- 172.21.15.162:0/3812509340 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29840f8c10 msgr2=0x7f298410dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:35.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.470+0000 7f298c091700 1 --2- 172.21.15.162:0/3812509340 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29840f8c10 0x7f298410dff0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7f29800048f0 tx=0x7f2980009cb0).stop 2024-02-13T00:44:35.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.472+0000 7f298c091700 1 -- 172.21.15.162:0/3812509340 shutdown_connections 2024-02-13T00:44:35.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.472+0000 7f298c091700 1 --2- 172.21.15.162:0/3812509340 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2970062c90 0x7f2970065150 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:35.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.472+0000 7f298c091700 1 --2- 172.21.15.162:0/3812509340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2984102170 0x7f29841081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:35.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.472+0000 7f298c091700 1 --2- 172.21.15.162:0/3812509340 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29840f8c10 0x7f298410dff0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:35.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.472+0000 7f298c091700 1 --2- 172.21.15.162:0/3812509340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29840f9570 0x7f298410e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:35.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.472+0000 7f298c091700 1 -- 172.21.15.162:0/3812509340 >> 172.21.15.162:0/3812509340 conn(0x7f29840f2740 msgr2=0x7f29840f3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:35.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.473+0000 7f298c091700 1 -- 172.21.15.162:0/3812509340 shutdown_connections 2024-02-13T00:44:35.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.473+0000 7f298c091700 1 -- 172.21.15.162:0/3812509340 wait complete. 2024-02-13T00:44:35.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:35.473+0000 7f298c091700 1 librados: shutdown 2024-02-13T00:44:35.479 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:44:35.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:35.570 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:35.708 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:44:35.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:44:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:44:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:34.105719+0000 osd.5 (osd.5) 143 : cluster [DBG] 5.16 scrub starts 2024-02-13T00:44:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:34.106631+0000 osd.5 (osd.5) 144 : cluster [DBG] 5.16 scrub ok 2024-02-13T00:44:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:34.168289+0000 osd.1 (osd.1) 131 : cluster [DBG] 28.11 deep-scrub starts 2024-02-13T00:44:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:34.171181+0000 osd.1 (osd.1) 132 : cluster [DBG] 28.11 deep-scrub ok 2024-02-13T00:44:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:34.391171+0000 mgr.y (mgr.14544) 297 : cluster [DBG] pgmap v448: 266 pgs: 1 peering, 265 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:34.627905+0000 osd.2 (osd.2) 135 : cluster [DBG] 61.12 scrub starts 2024-02-13T00:44:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:34.629424+0000 osd.2 (osd.2) 136 : cluster [DBG] 61.12 scrub ok 2024-02-13T00:44:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:35.396747+0000 mon.a (mon.0) 2082 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:35.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:35.411857+0000 mon.b (mon.1) 480 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:35.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:35.461082+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:44:35.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:35.461180+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e320: 8 total, 8 up, 6 in 2024-02-13T00:44:35.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:34.105719+0000 osd.5 (osd.5) 143 : cluster [DBG] 5.16 scrub starts 2024-02-13T00:44:35.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:34.106631+0000 osd.5 (osd.5) 144 : cluster [DBG] 5.16 scrub ok 2024-02-13T00:44:35.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:34.168289+0000 osd.1 (osd.1) 131 : cluster [DBG] 28.11 deep-scrub starts 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:34.171181+0000 osd.1 (osd.1) 132 : cluster [DBG] 28.11 deep-scrub ok 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:34.391171+0000 mgr.y (mgr.14544) 297 : cluster [DBG] pgmap v448: 266 pgs: 1 peering, 265 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:34.627905+0000 osd.2 (osd.2) 135 : cluster [DBG] 61.12 scrub starts 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:34.629424+0000 osd.2 (osd.2) 136 : cluster [DBG] 61.12 scrub ok 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:35.396747+0000 mon.a (mon.0) 2082 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:35.411857+0000 mon.b (mon.1) 480 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:35.461082+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:35.461180+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e320: 8 total, 8 up, 6 in 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:34.105719+0000 osd.5 (osd.5) 143 : cluster [DBG] 5.16 scrub starts 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:34.106631+0000 osd.5 (osd.5) 144 : cluster [DBG] 5.16 scrub ok 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:34.168289+0000 osd.1 (osd.1) 131 : cluster [DBG] 28.11 deep-scrub starts 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:34.171181+0000 osd.1 (osd.1) 132 : cluster [DBG] 28.11 deep-scrub ok 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:34.391171+0000 mgr.y (mgr.14544) 297 : cluster [DBG] pgmap v448: 266 pgs: 1 peering, 265 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:35.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:34.627905+0000 osd.2 (osd.2) 135 : cluster [DBG] 61.12 scrub starts 2024-02-13T00:44:35.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:34.629424+0000 osd.2 (osd.2) 136 : cluster [DBG] 61.12 scrub ok 2024-02-13T00:44:35.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:35.396747+0000 mon.a (mon.0) 2082 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:35.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:35.411857+0000 mon.b (mon.1) 480 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:35.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:35.461082+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:44:35.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:35.461180+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e320: 8 total, 8 up, 6 in 2024-02-13T00:44:36.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:44:36] "GET /metrics HTTP/1.1" 200 189014 "" "Prometheus/2.33.4" 2024-02-13T00:44:36.722 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:36.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:36.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:35.525803+0000 mgr.y (mgr.14544) 298 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:36.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:36.391874+0000 mon.c (mon.2) 313 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-13T00:44:36.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:36.392960+0000 mon.a (mon.0) 2085 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-13T00:44:36.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:36.412815+0000 mon.b (mon.1) 481 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:36.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:36.458843+0000 mon.a (mon.0) 2086 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-13T00:44:36.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:36.458930+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e321: 8 total, 8 up, 6 in 2024-02-13T00:44:36.765 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:36.460585+0000 mon.a (mon.0) 2088 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:36.765 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:36.470233+0000 mon.b (mon.1) 482 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:36.765 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:36.471443+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:36.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:36.853 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:37.044 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:44:37.044 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:44:37.044 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:44:37.044 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:44:37.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:35.525803+0000 mgr.y (mgr.14544) 298 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:37.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:36.391874+0000 mon.c (mon.2) 313 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-13T00:44:37.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:36.392960+0000 mon.a (mon.0) 2085 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-13T00:44:37.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:36.412815+0000 mon.b (mon.1) 481 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:37.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:36.458843+0000 mon.a (mon.0) 2086 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:36.458930+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e321: 8 total, 8 up, 6 in 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:36.460585+0000 mon.a (mon.0) 2088 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:36.470233+0000 mon.b (mon.1) 482 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:36.471443+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:35.525803+0000 mgr.y (mgr.14544) 298 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:36.391874+0000 mon.c (mon.2) 313 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:36.392960+0000 mon.a (mon.0) 2085 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:36.412815+0000 mon.b (mon.1) 481 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:36.458843+0000 mon.a (mon.0) 2086 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:36.458930+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e321: 8 total, 8 up, 6 in 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:36.460585+0000 mon.a (mon.0) 2088 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:36.470233+0000 mon.b (mon.1) 482 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:36.471443+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:37.204 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:37.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:37.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:37.338 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:37.488 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:37.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:37.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:37.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:44:37.632 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:37.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.725+0000 7f4fa9804700 1 -- 172.21.15.162:0/3999162467 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fa40fdd70 msgr2=0x7f4fa40fe150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.725+0000 7f4fa9804700 1 --2- 172.21.15.162:0/3999162467 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fa40fdd70 0x7f4fa40fe150 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 rx=0x7f4f94004870 tx=0x7f4f94009cb0).stop 2024-02-13T00:44:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.725+0000 7f4fa9804700 1 -- 172.21.15.162:0/3999162467 shutdown_connections 2024-02-13T00:44:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.725+0000 7f4fa9804700 1 --2- 172.21.15.162:0/3999162467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fa40f66c0 0x7f4fa4104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.725+0000 7f4fa9804700 1 --2- 172.21.15.162:0/3999162467 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fa40fdd70 0x7f4fa40fe150 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.725+0000 7f4fa9804700 1 --2- 172.21.15.162:0/3999162467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fa40f5d50 0x7f4fa40f61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.725+0000 7f4fa9804700 1 -- 172.21.15.162:0/3999162467 >> 172.21.15.162:0/3999162467 conn(0x7f4fa406fc50 msgr2=0x7f4fa40f4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.725+0000 7f4fa9804700 1 -- 172.21.15.162:0/3999162467 shutdown_connections 2024-02-13T00:44:37.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.725+0000 7f4fa9804700 1 -- 172.21.15.162:0/3999162467 wait complete. 2024-02-13T00:44:37.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.726+0000 7f4fa9804700 1 librados: starting msgr at 2024-02-13T00:44:37.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.726+0000 7f4fa9804700 1 librados: starting objecter 2024-02-13T00:44:37.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.726+0000 7f4fa9804700 1 Processor -- start 2024-02-13T00:44:37.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.726+0000 7f4fa9804700 1 -- start start 2024-02-13T00:44:37.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.726+0000 7f4fa9804700 1 librados: setting wanted keys 2024-02-13T00:44:37.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.726+0000 7f4fa9804700 1 librados: calling monclient init 2024-02-13T00:44:37.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa9804700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fa40f5d50 0x7f4fa40f78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:37.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa9804700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fa40f66c0 0x7f4fa40f7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:37.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa9804700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fa40fdd70 0x7f4fa40f8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:37.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa9804700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa4116700 con 0x7f4fa40f5d50 2024-02-13T00:44:37.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa9804700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa4116870 con 0x7f4fa40fdd70 2024-02-13T00:44:37.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa9804700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa4116a30 con 0x7f4fa40f66c0 2024-02-13T00:44:37.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4f9a7fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fa40f66c0 0x7f4fa40f7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:37.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fa40f5d50 0x7f4fa40f78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:37.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4f9a7fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fa40f66c0 0x7f4fa40f7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:46382/0 (socket says 172.21.15.162:46382) 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fa40f5d50 0x7f4fa40f78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34828/0 (socket says 172.21.15.162:34828) 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4f9a7fc700 1 -- 172.21.15.162:0/163553206 learned_addr learned my addr 172.21.15.162:0/163553206 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa37fe700 1 --2- 172.21.15.162:0/163553206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fa40fdd70 0x7f4fa40f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4f9a7fc700 1 -- 172.21.15.162:0/163553206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fa40fdd70 msgr2=0x7f4fa40f8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4f9a7fc700 1 --2- 172.21.15.162:0/163553206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fa40fdd70 0x7f4fa40f8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4f9a7fc700 1 -- 172.21.15.162:0/163553206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fa40f5d50 msgr2=0x7f4fa40f78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4f9a7fc700 1 --2- 172.21.15.162:0/163553206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fa40f5d50 0x7f4fa40f78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4f9a7fc700 1 -- 172.21.15.162:0/163553206 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9c0097c0 con 0x7f4fa40f66c0 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa2ffd700 1 --2- 172.21.15.162:0/163553206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fa40f5d50 0x7f4fa40f78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4f9a7fc700 1 --2- 172.21.15.162:0/163553206 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fa40f66c0 0x7f4fa40f7da0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 rx=0x7f4f8c0041d0 tx=0x7f4f8c009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa0ff9700 10 client.?.objecter ms_handle_connect 0x7f4fa40f66c0 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa0ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa0ff9700 1 -- 172.21.15.162:0/163553206 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c007b40 con 0x7f4fa40f66c0 2024-02-13T00:44:37.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa0ff9700 1 -- 172.21.15.162:0/163553206 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f8c00f040 con 0x7f4fa40f66c0 2024-02-13T00:44:37.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.727+0000 7f4fa9804700 1 -- 172.21.15.162:0/163553206 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f94005040 con 0x7f4fa40f66c0 2024-02-13T00:44:37.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.728+0000 7f4fa0ff9700 1 -- 172.21.15.162:0/163553206 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c014810 con 0x7f4fa40f66c0 2024-02-13T00:44:37.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.728+0000 7f4fa9804700 10 client.37184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:37.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.728+0000 7f4fa9804700 1 -- 172.21.15.162:0/163553206 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa4117070 con 0x7f4fa40f66c0 2024-02-13T00:44:37.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.728+0000 7f4fa9804700 1 librados: init done 2024-02-13T00:44:37.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.729+0000 7f4fa9804700 1 -- 172.21.15.162:0/163553206 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa40607f0 con 0x7f4fa40f66c0 2024-02-13T00:44:37.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.729+0000 7f4fa0ff9700 1 -- 172.21.15.162:0/163553206 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4f8c009110 con 0x7f4fa40f66c0 2024-02-13T00:44:37.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.729+0000 7f4fa0ff9700 1 --2- 172.21.15.162:0/163553206 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4f90062c90 0x7f4f90065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:37.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.729+0000 7f4fa2ffd700 1 --2- 172.21.15.162:0/163553206 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4f90062c90 0x7f4f90065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:37.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.729+0000 7f4fa0ff9700 1 -- 172.21.15.162:0/163553206 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(322..322 src has 52..322) v4 ==== 8470+0+0 (secure 0 0 0) 0x7f4f8c07d360 con 0x7f4fa40f66c0 2024-02-13T00:44:37.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.729+0000 7f4fa0ff9700 10 client.37184.objecter ms_dispatch 0x7f4fa4001550 osd_map(322..322 src has 52..322) v4 2024-02-13T00:44:37.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.729+0000 7f4fa0ff9700 3 client.37184.objecter handle_osd_map got epochs [322,322] > 0 2024-02-13T00:44:37.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.729+0000 7f4fa0ff9700 3 client.37184.objecter handle_osd_map decoding full epoch 322 2024-02-13T00:44:37.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.730+0000 7f4fa0ff9700 20 client.37184.objecter dump_active .. 0 homeless 2024-02-13T00:44:37.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.730+0000 7f4fa0ff9700 10 client.37184.objecter ms_handle_connect 0x7f4f90062c90 2024-02-13T00:44:37.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.730+0000 7f4fa2ffd700 1 --2- 172.21.15.162:0/163553206 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4f90062c90 0x7f4f90065150 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7f4f940043d0 tx=0x7f4f9400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:37.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.734+0000 7f4fa0ff9700 1 -- 172.21.15.162:0/163553206 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4f8c050690 con 0x7f4fa40f66c0 2024-02-13T00:44:37.758 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:37.830 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:44:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:37.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:44:37.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-13T00:44:37.912 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:36.391547+0000 mgr.y (mgr.14544) 299 : cluster [DBG] pgmap v451: 241 pgs: 8 creating+peering, 1 clean+premerge+peered, 232 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:37.912 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:37.414353+0000 mon.b (mon.1) 483 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:37.912 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:37.462449+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-83985-55"}]': finished 2024-02-13T00:44:37.912 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:37.462603+0000 mon.a (mon.0) 2091 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:37.912 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:37.462696+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e322: 8 total, 8 up, 6 in 2024-02-13T00:44:37.912 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:37.464494+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:37.912 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:37.474589+0000 mon.b (mon.1) 484 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:37.912 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:37.475350+0000 mon.a (mon.0) 2094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:37.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:37.916+0000 7f4fa9804700 1 -- 172.21.15.162:0/163553206 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd in", "ids": ["6"]} v 0) v1 -- 0x7f4fa41072f0 con 0x7f4fa40f66c0 2024-02-13T00:44:37.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:37.964 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:38.017 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:38.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:36.391547+0000 mgr.y (mgr.14544) 299 : cluster [DBG] pgmap v451: 241 pgs: 8 creating+peering, 1 clean+premerge+peered, 232 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:38.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:37.414353+0000 mon.b (mon.1) 483 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:38.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:37.462449+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-83985-55"}]': finished 2024-02-13T00:44:38.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:37.462603+0000 mon.a (mon.0) 2091 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:38.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:37.462696+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e322: 8 total, 8 up, 6 in 2024-02-13T00:44:38.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:37.464494+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:38.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:37.474589+0000 mon.b (mon.1) 484 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:38.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:37.475350+0000 mon.a (mon.0) 2094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:38.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:36.391547+0000 mgr.y (mgr.14544) 299 : cluster [DBG] pgmap v451: 241 pgs: 8 creating+peering, 1 clean+premerge+peered, 232 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:38.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:37.414353+0000 mon.b (mon.1) 483 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:38.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:37.462449+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-83985-55"}]': finished 2024-02-13T00:44:38.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:37.462603+0000 mon.a (mon.0) 2091 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:38.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:37.462696+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e322: 8 total, 8 up, 6 in 2024-02-13T00:44:38.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:37.464494+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-83985-55"}]: dispatch 2024-02-13T00:44:38.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:37.474589+0000 mon.b (mon.1) 484 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:38.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:37.475350+0000 mon.a (mon.0) 2094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:38.021 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:44:37.916+0000 7f5f6486f700 -1 mon.a@0(leader).osd e322 definitely_dead 0 2024-02-13T00:44:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:38.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:38.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:38.149 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:38.150 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:44:38.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.464+0000 7f4fa0ff9700 1 -- 172.21.15.162:0/163553206 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["6"]}]=0 marked in osd.6. v323) v1 ==== 85+0+0 (secure 0 0 0) 0x7f4f8c050280 con 0x7f4fa40f66c0 2024-02-13T00:44:38.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.475+0000 7f4fa9804700 10 librados: watch_flush enter 2024-02-13T00:44:38.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.483+0000 7f4fa9804700 10 librados: watch_flush exit 2024-02-13T00:44:38.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.483+0000 7f4fa9804700 20 client.37184.objecter shutdown clearing up homeless session... 2024-02-13T00:44:38.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.483+0000 7f4fa9804700 10 client.37184.objecter successfully canceled tick 2024-02-13T00:44:38.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.491+0000 7f4fa9804700 1 -- 172.21.15.162:0/163553206 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4f90062c90 msgr2=0x7f4f90065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:38.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.491+0000 7f4fa9804700 1 --2- 172.21.15.162:0/163553206 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4f90062c90 0x7f4f90065150 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7f4f940043d0 tx=0x7f4f9400b040).stop 2024-02-13T00:44:38.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.491+0000 7f4fa9804700 1 -- 172.21.15.162:0/163553206 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fa40f66c0 msgr2=0x7f4fa40f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:38.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.491+0000 7f4fa9804700 1 --2- 172.21.15.162:0/163553206 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fa40f66c0 0x7f4fa40f7da0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 rx=0x7f4f8c0041d0 tx=0x7f4f8c009f60).stop 2024-02-13T00:44:38.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.492+0000 7f4fa9804700 1 -- 172.21.15.162:0/163553206 shutdown_connections 2024-02-13T00:44:38.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.492+0000 7f4fa9804700 1 --2- 172.21.15.162:0/163553206 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4f90062c90 0x7f4f90065150 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:38.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.492+0000 7f4fa9804700 1 --2- 172.21.15.162:0/163553206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fa40fdd70 0x7f4fa40f8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:38.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.492+0000 7f4fa9804700 1 --2- 172.21.15.162:0/163553206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fa40f5d50 0x7f4fa40f78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:38.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.492+0000 7f4fa9804700 1 --2- 172.21.15.162:0/163553206 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fa40f66c0 0x7f4fa40f7da0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:38.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.492+0000 7f4fa9804700 1 -- 172.21.15.162:0/163553206 >> 172.21.15.162:0/163553206 conn(0x7f4fa406fc50 msgr2=0x7f4fa40fc130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:38.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.492+0000 7f4fa9804700 1 -- 172.21.15.162:0/163553206 shutdown_connections 2024-02-13T00:44:38.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.492+0000 7f4fa9804700 1 -- 172.21.15.162:0/163553206 wait complete. 2024-02-13T00:44:38.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:38.493+0000 7f4fa9804700 1 librados: shutdown 2024-02-13T00:44:38.509 INFO:teuthology.orchestra.run.smithi162.stderr:marked in osd.6. 2024-02-13T00:44:38.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:44:38.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:37.917311+0000 mon.c (mon.2) 314 : audit [DBG] from='client.? 172.21.15.162:0/163553206' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-13T00:44:38.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:37.917561+0000 mon.a (mon.0) 2095 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-13T00:44:38.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:38.415447+0000 mon.b (mon.1) 485 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:38.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:38.462683+0000 mon.a (mon.0) 2096 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-83985-55"}]': finished 2024-02-13T00:44:38.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:38.462769+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:38.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:38.462855+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-13T00:44:38.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:38.462909+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e323: 8 total, 8 up, 7 in 2024-02-13T00:44:38.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:38.477847+0000 mon.b (mon.1) 486 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-13T00:44:38.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:38.479314+0000 mon.a (mon.0) 2100 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-13T00:44:38.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:38.521784+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:38.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:38.522127+0000 mon.a (mon.0) 2102 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:38.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:38.522393+0000 mon.a (mon.0) 2103 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi162-83985-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:37.917311+0000 mon.c (mon.2) 314 : audit [DBG] from='client.? 172.21.15.162:0/163553206' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-13T00:44:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:37.917561+0000 mon.a (mon.0) 2095 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-13T00:44:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:38.415447+0000 mon.b (mon.1) 485 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:38.462683+0000 mon.a (mon.0) 2096 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-83985-55"}]': finished 2024-02-13T00:44:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:38.462769+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:38.462855+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-13T00:44:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:38.462909+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e323: 8 total, 8 up, 7 in 2024-02-13T00:44:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:38.477847+0000 mon.b (mon.1) 486 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-13T00:44:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:38.479314+0000 mon.a (mon.0) 2100 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-13T00:44:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:38.521784+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:38.522127+0000 mon.a (mon.0) 2102 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:38.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:38.522393+0000 mon.a (mon.0) 2103 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi162-83985-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:38.848 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:44:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:44:38.848 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:37.917311+0000 mon.c (mon.2) 314 : audit [DBG] from='client.? 172.21.15.162:0/163553206' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-13T00:44:38.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:37.917561+0000 mon.a (mon.0) 2095 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-13T00:44:38.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:38.415447+0000 mon.b (mon.1) 485 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:38.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:38.462683+0000 mon.a (mon.0) 2096 : audit [INF] from='client.? 172.21.15.162:0/4276929747' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-83985-55"}]': finished 2024-02-13T00:44:38.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:38.462769+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:38.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:38.462855+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-13T00:44:38.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:38.462909+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e323: 8 total, 8 up, 7 in 2024-02-13T00:44:38.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:38.477847+0000 mon.b (mon.1) 486 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-13T00:44:38.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:38.479314+0000 mon.a (mon.0) 2100 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-13T00:44:38.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:38.521784+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:38.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:38.522127+0000 mon.a (mon.0) 2102 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:38.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:38.522393+0000 mon.a (mon.0) 2103 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi162-83985-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:39.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.774+0000 7ff250ff1700 1 -- 172.21.15.162:0/893976296 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff24c102270 msgr2=0x7ff24c104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.774+0000 7ff250ff1700 1 --2- 172.21.15.162:0/893976296 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff24c102270 0x7ff24c104660 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 rx=0x7ff244009c30 tx=0x7ff244004c50).stop 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.774+0000 7ff250ff1700 1 -- 172.21.15.162:0/893976296 shutdown_connections 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.774+0000 7ff250ff1700 1 --2- 172.21.15.162:0/893976296 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff24c102270 0x7ff24c104660 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.774+0000 7ff250ff1700 1 --2- 172.21.15.162:0/893976296 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24c0f8d10 0x7ff24c0f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.774+0000 7ff250ff1700 1 --2- 172.21.15.162:0/893976296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff24c0f95e0 0x7ff24c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.774+0000 7ff250ff1700 1 -- 172.21.15.162:0/893976296 >> 172.21.15.162:0/893976296 conn(0x7ff24c06eb90 msgr2=0x7ff24c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.774+0000 7ff250ff1700 1 -- 172.21.15.162:0/893976296 shutdown_connections 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.774+0000 7ff250ff1700 1 -- 172.21.15.162:0/893976296 wait complete. 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.774+0000 7ff250ff1700 1 librados: starting msgr at 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.774+0000 7ff250ff1700 1 librados: starting objecter 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.774+0000 7ff250ff1700 1 Processor -- start 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff250ff1700 1 -- start start 2024-02-13T00:44:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff250ff1700 1 librados: setting wanted keys 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff250ff1700 1 librados: calling monclient init 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff250ff1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24c0f8d10 0x7ff24c10e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff250ff1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff24c0f95e0 0x7ff24c10e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff250ff1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff24c102270 0x7ff24c108290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff250ff1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c108780 con 0x7ff24c0f95e0 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff250ff1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c1088f0 con 0x7ff24c0f8d10 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff250ff1700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c108ab0 con 0x7ff24c102270 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff249d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff24c0f95e0 0x7ff24c10e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff24a59c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24c0f8d10 0x7ff24c10e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff249d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff24c0f95e0 0x7ff24c10e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34862/0 (socket says 172.21.15.162:34862) 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff24a59c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24c0f8d10 0x7ff24c10e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:43060/0 (socket says 172.21.15.162:43060) 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff24ad9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff24c102270 0x7ff24c108290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:39.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff249d9b700 1 -- 172.21.15.162:0/898071383 learned_addr learned my addr 172.21.15.162:0/898071383 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff249d9b700 1 -- 172.21.15.162:0/898071383 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff24c102270 msgr2=0x7ff24c108290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff249d9b700 1 --2- 172.21.15.162:0/898071383 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff24c102270 0x7ff24c108290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff249d9b700 1 -- 172.21.15.162:0/898071383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24c0f8d10 msgr2=0x7ff24c10e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff249d9b700 1 --2- 172.21.15.162:0/898071383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24c0f8d10 0x7ff24c10e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff249d9b700 1 -- 172.21.15.162:0/898071383 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff240005040 con 0x7ff24c0f95e0 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff24a59c700 1 --2- 172.21.15.162:0/898071383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff24c0f8d10 0x7ff24c10e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff24ad9d700 1 --2- 172.21.15.162:0/898071383 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff24c102270 0x7ff24c108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff249d9b700 1 --2- 172.21.15.162:0/898071383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff24c0f95e0 0x7ff24c10e5e0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7ff240009400 tx=0x7ff240009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff23b7fe700 10 client.?.objecter ms_handle_connect 0x7ff24c0f95e0 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff23b7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff240007a40 con 0x7ff24c0f95e0 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2440097c0 con 0x7ff24c0f95e0 2024-02-13T00:44:39.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.775+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff24000f040 con 0x7ff24c0f95e0 2024-02-13T00:44:39.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.776+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff240014a70 con 0x7ff24c0f95e0 2024-02-13T00:44:39.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.776+0000 7ff250ff1700 10 client.27087.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:39.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.776+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c1090f0 con 0x7ff24c0f95e0 2024-02-13T00:44:39.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.776+0000 7ff250ff1700 1 librados: init done 2024-02-13T00:44:39.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.776+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff24c0475e0 con 0x7ff24c0f95e0 2024-02-13T00:44:39.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.777+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff240010530 con 0x7ff24c0f95e0 2024-02-13T00:44:39.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.777+0000 7ff23b7fe700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff234062c90 0x7ff234065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:39.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.777+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(324..324 src has 52..324) v4 ==== 8814+0+0 (secure 0 0 0) 0x7ff24007d490 con 0x7ff24c0f95e0 2024-02-13T00:44:39.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.777+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 osd_map(324..324 src has 52..324) v4 2024-02-13T00:44:39.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.777+0000 7ff24a59c700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff234062c90 0x7ff234065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:39.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.777+0000 7ff23b7fe700 3 client.27087.objecter handle_osd_map got epochs [324,324] > 0 2024-02-13T00:44:39.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.777+0000 7ff23b7fe700 3 client.27087.objecter handle_osd_map decoding full epoch 324 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff23b7fe700 20 client.27087.objecter dump_active .. 0 homeless 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v324) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff24007d930 con 0x7ff24c0f95e0 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff24a59c700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff234062c90 0x7ff234065150 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 rx=0x7ff23c011510 tx=0x7ff23c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff23b7fe700 10 client.27087.objecter ms_handle_connect 0x7ff234062c90 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff250ff1700 10 client.27087.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 1 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff24c028590 0x7ff24c0fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c1071b0 osd=0 3 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c1071b0 osd=0 3 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c1071b0 osd=0 3 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 1 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c1071b0 osd=0 3 2024-02-13T00:44:39.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 0 1 2024-02-13T00:44:39.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c1071b0 osd=0 4 2024-02-13T00:44:39.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff250ff1700 10 client.27087.objecter _send_command 1 2024-02-13T00:44:39.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff24c109770 con 0x7ff24c028590 2024-02-13T00:44:39.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff24ad9d700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff24c028590 0x7ff24c0fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:39.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff23b7fe700 10 client.27087.objecter ms_handle_connect 0x7ff24c028590 2024-02-13T00:44:39.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.778+0000 7ff24ad9d700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff24c028590 0x7ff24c0fbd00 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:39.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.779+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff24c109770 con 0x7ff24c028590 2024-02-13T00:44:39.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.779+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:44:39.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.779+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 1 = system:0 2024-02-13T00:44:39.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.779+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c1071b0 osd=0 4 2024-02-13T00:44:39.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.779+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 0 1 2024-02-13T00:44:39.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.794+0000 7ff250ff1700 10 client.27087.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:39.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.794+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 2 2024-02-13T00:44:39.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.794+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c1071b0 osd=0 3 2024-02-13T00:44:39.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.794+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c1071b0 osd=0 3 2024-02-13T00:44:39.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.794+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c1071b0 osd=0 3 2024-02-13T00:44:39.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.794+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 2 2024-02-13T00:44:39.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.794+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c1071b0 osd=0 3 2024-02-13T00:44:39.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.794+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 0 2 2024-02-13T00:44:39.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.794+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c1071b0 osd=0 4 2024-02-13T00:44:39.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.794+0000 7ff250ff1700 10 client.27087.objecter _send_command 2 2024-02-13T00:44:39.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.794+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff24c05a0e0 con 0x7ff24c028590 2024-02-13T00:44:39.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff24c05a0e0 con 0x7ff24c028590 2024-02-13T00:44:39.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:39.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c1071b0 osd=0 4 2024-02-13T00:44:39.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 0 2 2024-02-13T00:44:39.799 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:44:39.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff250ff1700 10 client.27087.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:39.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 3 2024-02-13T00:44:39.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6805/150599780 conn(0x7ff24c027be0 0x7ff24c027fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:39.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c0fd370 osd=1 3 2024-02-13T00:44:39.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fd370 osd=1 3 2024-02-13T00:44:39.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c0fd370 osd=1 3 2024-02-13T00:44:39.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 3 2024-02-13T00:44:39.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c0fd370 osd=1 3 2024-02-13T00:44:39.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 1 3 2024-02-13T00:44:39.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fd370 osd=1 4 2024-02-13T00:44:39.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff250ff1700 10 client.27087.objecter _send_command 3 2024-02-13T00:44:39.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.795+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff24c0fd4e0 con 0x7ff24c027be0 2024-02-13T00:44:39.801 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:39.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.796+0000 7ff24a59c700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6805/150599780 conn(0x7ff24c027be0 0x7ff24c027fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:39.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.796+0000 7ff23b7fe700 10 client.27087.objecter ms_handle_connect 0x7ff24c027be0 2024-02-13T00:44:39.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.796+0000 7ff24a59c700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6805/150599780 conn(0x7ff24c027be0 0x7ff24c027fc0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:39.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.797+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff24c0fd4e0 con 0x7ff24c027be0 2024-02-13T00:44:39.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.797+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:44:39.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.797+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 3 = system:0 2024-02-13T00:44:39.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.797+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c0fd370 osd=1 4 2024-02-13T00:44:39.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.797+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 1 3 2024-02-13T00:44:39.802 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:38.392086+0000 mgr.y (mgr.14544) 300 : cluster [DBG] pgmap v454: 265 pgs: 32 unknown, 1 clean+premerge+peered, 232 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:39.802 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:39.416225+0000 mon.b (mon.1) 487 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:39.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:39.464329+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_count","val": "3"}]': finished 2024-02-13T00:44:39.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:39.464484+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi162-83985-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:39.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:39.464545+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e324: 8 total, 8 up, 7 in 2024-02-13T00:44:39.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:39.468040+0000 mon.b (mon.1) 488 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-13T00:44:39.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:39.468632+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-13T00:44:39.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:39.469506+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi162-83985-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:39.803 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:38.392086+0000 mgr.y (mgr.14544) 300 : cluster [DBG] pgmap v454: 265 pgs: 32 unknown, 1 clean+premerge+peered, 232 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:39.803 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:39.416225+0000 mon.b (mon.1) 487 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:39.803 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:39.464329+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_count","val": "3"}]': finished 2024-02-13T00:44:39.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:39.464484+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi162-83985-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:39.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:39.464545+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e324: 8 total, 8 up, 7 in 2024-02-13T00:44:39.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:39.468040+0000 mon.b (mon.1) 488 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-13T00:44:39.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:39.468632+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-13T00:44:39.804 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:39.469506+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi162-83985-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:39.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:39.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.819+0000 7ff250ff1700 10 client.27087.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:39.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.819+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 4 2024-02-13T00:44:39.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.819+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c0fd370 osd=1 3 2024-02-13T00:44:39.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.819+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fd370 osd=1 3 2024-02-13T00:44:39.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.819+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c0fd370 osd=1 3 2024-02-13T00:44:39.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.819+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 4 2024-02-13T00:44:39.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.819+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c0fd370 osd=1 3 2024-02-13T00:44:39.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.819+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 1 4 2024-02-13T00:44:39.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.819+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fd370 osd=1 4 2024-02-13T00:44:39.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.819+0000 7ff250ff1700 10 client.27087.objecter _send_command 4 2024-02-13T00:44:39.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.819+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff24c027650 con 0x7ff24c027be0 2024-02-13T00:44:39.825 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:44:39.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.821+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff24c027650 con 0x7ff24c027be0 2024-02-13T00:44:39.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.821+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.821+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.821+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c0fd370 osd=1 4 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.821+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 1 4 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff250ff1700 10 client.27087.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 5 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff24c11a120 0x7ff24c11c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c0fc9c0 osd=2 3 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fc9c0 osd=2 3 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c0fc9c0 osd=2 3 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 5 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c0fc9c0 osd=2 3 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 2 5 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fc9c0 osd=2 4 2024-02-13T00:44:39.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff250ff1700 10 client.27087.objecter _send_command 5 2024-02-13T00:44:39.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff24c0fd040 con 0x7ff24c11a120 2024-02-13T00:44:39.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff249d9b700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff24c11a120 0x7ff24c11c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:39.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff23b7fe700 10 client.27087.objecter ms_handle_connect 0x7ff24c11a120 2024-02-13T00:44:39.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.822+0000 7ff249d9b700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff24c11a120 0x7ff24c11c510 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:39.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.823+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff24c0fd040 con 0x7ff24c11a120 2024-02-13T00:44:39.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.823+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:44:39.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.823+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 5 = system:0 2024-02-13T00:44:39.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.823+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c0fc9c0 osd=2 4 2024-02-13T00:44:39.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.823+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 2 5 2024-02-13T00:44:39.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.845+0000 7ff250ff1700 10 client.27087.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:39.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.845+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 6 2024-02-13T00:44:39.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.845+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c0fc9c0 osd=2 3 2024-02-13T00:44:39.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.845+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fc9c0 osd=2 3 2024-02-13T00:44:39.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.845+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c0fc9c0 osd=2 3 2024-02-13T00:44:39.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.845+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 6 2024-02-13T00:44:39.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.845+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c0fc9c0 osd=2 3 2024-02-13T00:44:39.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.845+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 2 6 2024-02-13T00:44:39.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.845+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fc9c0 osd=2 4 2024-02-13T00:44:39.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.845+0000 7ff250ff1700 10 client.27087.objecter _send_command 6 2024-02-13T00:44:39.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.845+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff24c0fc430 con 0x7ff24c11a120 2024-02-13T00:44:39.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff24c0fc430 con 0x7ff24c11a120 2024-02-13T00:44:39.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:39.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c0fc9c0 osd=2 4 2024-02-13T00:44:39.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 2 6 2024-02-13T00:44:39.850 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:44:39.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff250ff1700 10 client.27087.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:39.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 7 2024-02-13T00:44:39.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6813/604971011 conn(0x7ff24c119c80 0x7ff24c11fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c0fc840 osd=3 3 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fc840 osd=3 3 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c0fc840 osd=3 3 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 7 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c0fc840 osd=3 3 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 3 7 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fc840 osd=3 4 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff250ff1700 10 client.27087.objecter _send_command 7 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff24c1227e0 con 0x7ff24c119c80 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff24ad9d700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6813/604971011 conn(0x7ff24c119c80 0x7ff24c11fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff23b7fe700 10 client.27087.objecter ms_handle_connect 0x7ff24c119c80 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.847+0000 7ff24ad9d700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6813/604971011 conn(0x7ff24c119c80 0x7ff24c11fb90 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:39.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.848+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff24c1227e0 con 0x7ff24c119c80 2024-02-13T00:44:39.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.848+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:44:39.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.848+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 7 = system:0 2024-02-13T00:44:39.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.848+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c0fc840 osd=3 4 2024-02-13T00:44:39.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.848+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 3 7 2024-02-13T00:44:39.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.862+0000 7ff250ff1700 10 client.27087.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:39.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.863+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 8 2024-02-13T00:44:39.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.863+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c0fc840 osd=3 3 2024-02-13T00:44:39.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.863+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fc840 osd=3 3 2024-02-13T00:44:39.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.863+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c0fc840 osd=3 3 2024-02-13T00:44:39.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.863+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 8 2024-02-13T00:44:39.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.863+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c0fc840 osd=3 3 2024-02-13T00:44:39.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.863+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 3 8 2024-02-13T00:44:39.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.863+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fc840 osd=3 4 2024-02-13T00:44:39.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.863+0000 7ff250ff1700 10 client.27087.objecter _send_command 8 2024-02-13T00:44:39.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.863+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff24c1191e0 con 0x7ff24c119c80 2024-02-13T00:44:39.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff24c1191e0 con 0x7ff24c119c80 2024-02-13T00:44:39.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:39.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c0fc840 osd=3 4 2024-02-13T00:44:39.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 3 8 2024-02-13T00:44:39.868 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:44:39.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff250ff1700 10 client.27087.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 9 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff24c119350 0x7ff24c122090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c1191e0 osd=4 3 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c1191e0 osd=4 3 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c1191e0 osd=4 3 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 9 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c1191e0 osd=4 3 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 4 9 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c1191e0 osd=4 4 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff250ff1700 10 client.27087.objecter _send_command 9 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff24c122670 con 0x7ff24c119350 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.864+0000 7ff24a59c700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff24c119350 0x7ff24c122090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:39.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.865+0000 7ff24a59c700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff24c119350 0x7ff24c122090 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:39.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.865+0000 7ff23b7fe700 10 client.27087.objecter ms_handle_connect 0x7ff24c119350 2024-02-13T00:44:39.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.866+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff24c122670 con 0x7ff24c119350 2024-02-13T00:44:39.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.866+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:44:39.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.866+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 9 = system:0 2024-02-13T00:44:39.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.866+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c1191e0 osd=4 4 2024-02-13T00:44:39.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.866+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 4 9 2024-02-13T00:44:39.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:38.392086+0000 mgr.y (mgr.14544) 300 : cluster [DBG] pgmap v454: 265 pgs: 32 unknown, 1 clean+premerge+peered, 232 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:44:39.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:39.416225+0000 mon.b (mon.1) 487 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:39.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:39.464329+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_count","val": "3"}]': finished 2024-02-13T00:44:39.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:39.464484+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi162-83985-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:39.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:39.464545+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e324: 8 total, 8 up, 7 in 2024-02-13T00:44:39.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:39.468040+0000 mon.b (mon.1) 488 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-13T00:44:39.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:39.468632+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-13T00:44:39.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:39.469506+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi162-83985-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:39.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.880+0000 7ff250ff1700 10 client.27087.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:39.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.880+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 10 2024-02-13T00:44:39.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.880+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c1191e0 osd=4 3 2024-02-13T00:44:39.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.880+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c1191e0 osd=4 3 2024-02-13T00:44:39.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.880+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c1191e0 osd=4 3 2024-02-13T00:44:39.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.880+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 10 2024-02-13T00:44:39.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.880+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c1191e0 osd=4 3 2024-02-13T00:44:39.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.880+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 4 10 2024-02-13T00:44:39.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.880+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c1191e0 osd=4 4 2024-02-13T00:44:39.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.880+0000 7ff250ff1700 10 client.27087.objecter _send_command 10 2024-02-13T00:44:39.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.880+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff24c122670 con 0x7ff24c119350 2024-02-13T00:44:39.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff24c122670 con 0x7ff24c119350 2024-02-13T00:44:39.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:39.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c1191e0 osd=4 4 2024-02-13T00:44:39.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 4 10 2024-02-13T00:44:39.886 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:44:39.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff250ff1700 10 client.27087.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:39.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 11 2024-02-13T00:44:39.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff24c0f8190 0x7ff24c0f85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:39.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c122670 osd=5 3 2024-02-13T00:44:39.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c122670 osd=5 3 2024-02-13T00:44:39.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c122670 osd=5 3 2024-02-13T00:44:39.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 11 2024-02-13T00:44:39.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c122670 osd=5 3 2024-02-13T00:44:39.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 5 11 2024-02-13T00:44:39.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c122670 osd=5 4 2024-02-13T00:44:39.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff250ff1700 10 client.27087.objecter _send_command 11 2024-02-13T00:44:39.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff24c11d810 con 0x7ff24c0f8190 2024-02-13T00:44:39.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.882+0000 7ff249d9b700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff24c0f8190 0x7ff24c0f85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:39.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.883+0000 7ff23b7fe700 10 client.27087.objecter ms_handle_connect 0x7ff24c0f8190 2024-02-13T00:44:39.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.883+0000 7ff249d9b700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff24c0f8190 0x7ff24c0f85b0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:39.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.884+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff24c11d810 con 0x7ff24c0f8190 2024-02-13T00:44:39.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.884+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:44:39.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.884+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 11 = system:0 2024-02-13T00:44:39.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.884+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c122670 osd=5 4 2024-02-13T00:44:39.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.884+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 5 11 2024-02-13T00:44:39.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.898+0000 7ff250ff1700 10 client.27087.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:39.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.898+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 12 2024-02-13T00:44:39.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.898+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c122670 osd=5 3 2024-02-13T00:44:39.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.898+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c122670 osd=5 3 2024-02-13T00:44:39.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.898+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c122670 osd=5 3 2024-02-13T00:44:39.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.898+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 12 2024-02-13T00:44:39.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.898+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c122670 osd=5 3 2024-02-13T00:44:39.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.898+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 5 12 2024-02-13T00:44:39.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.898+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c122670 osd=5 4 2024-02-13T00:44:39.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.898+0000 7ff250ff1700 10 client.27087.objecter _send_command 12 2024-02-13T00:44:39.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.898+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff24c11d810 con 0x7ff24c0f8190 2024-02-13T00:44:39.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff24c11d810 con 0x7ff24c0f8190 2024-02-13T00:44:39.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:39.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c122670 osd=5 4 2024-02-13T00:44:39.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 5 12 2024-02-13T00:44:39.904 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:44:39.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff250ff1700 10 client.27087.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:39.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 13 2024-02-13T00:44:39.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff24c129280 0x7ff24c12b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:39.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c11d810 osd=6 3 2024-02-13T00:44:39.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c11d810 osd=6 3 2024-02-13T00:44:39.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c11d810 osd=6 3 2024-02-13T00:44:39.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 13 2024-02-13T00:44:39.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c11d810 osd=6 3 2024-02-13T00:44:39.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 6 13 2024-02-13T00:44:39.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c11d810 osd=6 4 2024-02-13T00:44:39.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff250ff1700 10 client.27087.objecter _send_command 13 2024-02-13T00:44:39.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff24c12e2c0 con 0x7ff24c129280 2024-02-13T00:44:39.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.901+0000 7ff24ad9d700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff24c129280 0x7ff24c12b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:39.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.902+0000 7ff23b7fe700 10 client.27087.objecter ms_handle_connect 0x7ff24c129280 2024-02-13T00:44:39.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.902+0000 7ff24ad9d700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff24c129280 0x7ff24c12b670 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:39.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:39.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.903+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff24c12e2c0 con 0x7ff24c129280 2024-02-13T00:44:39.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.903+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:44:39.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.903+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 13 = system:0 2024-02-13T00:44:39.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.903+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c11d810 osd=6 4 2024-02-13T00:44:39.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.903+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 6 13 2024-02-13T00:44:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.917+0000 7ff250ff1700 10 client.27087.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.917+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 14 2024-02-13T00:44:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.917+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c11d810 osd=6 3 2024-02-13T00:44:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.917+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c11d810 osd=6 3 2024-02-13T00:44:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.917+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c11d810 osd=6 3 2024-02-13T00:44:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.917+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 14 2024-02-13T00:44:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.917+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c11d810 osd=6 3 2024-02-13T00:44:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.917+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 6 14 2024-02-13T00:44:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.917+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c11d810 osd=6 4 2024-02-13T00:44:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.917+0000 7ff250ff1700 10 client.27087.objecter _send_command 14 2024-02-13T00:44:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.917+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff24c12bb60 con 0x7ff24c129280 2024-02-13T00:44:39.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.919+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff24c12bb60 con 0x7ff24c129280 2024-02-13T00:44:39.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.919+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:39.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.919+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.919+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c11d810 osd=6 4 2024-02-13T00:44:39.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.919+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 6 14 2024-02-13T00:44:39.923 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:44:39.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff250ff1700 10 client.27087.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:39.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 15 2024-02-13T00:44:39.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff24c12c240 0x7ff24c12c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:39.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c12bb60 osd=7 3 2024-02-13T00:44:39.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c12bb60 osd=7 3 2024-02-13T00:44:39.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c12bb60 osd=7 3 2024-02-13T00:44:39.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 15 2024-02-13T00:44:39.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c12bb60 osd=7 3 2024-02-13T00:44:39.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 7 15 2024-02-13T00:44:39.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c12bb60 osd=7 4 2024-02-13T00:44:39.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff250ff1700 10 client.27087.objecter _send_command 15 2024-02-13T00:44:39.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff24c12e2c0 con 0x7ff24c12c240 2024-02-13T00:44:39.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff24a59c700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff24c12c240 0x7ff24c12c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:39.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff23b7fe700 10 client.27087.objecter ms_handle_connect 0x7ff24c12c240 2024-02-13T00:44:39.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.920+0000 7ff24a59c700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff24c12c240 0x7ff24c12c6a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:39.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.921+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff24c12e2c0 con 0x7ff24c12c240 2024-02-13T00:44:39.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.921+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:44:39.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.921+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 15 = system:0 2024-02-13T00:44:39.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.921+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c12bb60 osd=7 4 2024-02-13T00:44:39.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.921+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 7 15 2024-02-13T00:44:39.934 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:39.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.937+0000 7ff250ff1700 10 client.27087.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:39.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.937+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign -1 16 2024-02-13T00:44:39.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.937+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c12bb60 osd=7 3 2024-02-13T00:44:39.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.937+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c12bb60 osd=7 3 2024-02-13T00:44:39.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.937+0000 7ff250ff1700 20 client.27087.objecter _get_session s=0x7ff24c12bb60 osd=7 3 2024-02-13T00:44:39.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.937+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_remove -1 16 2024-02-13T00:44:39.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.937+0000 7ff250ff1700 20 client.27087.objecter get_session s=0x7ff24c12bb60 osd=7 3 2024-02-13T00:44:39.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.937+0000 7ff250ff1700 15 client.27087.objecter _session_command_op_assign 7 16 2024-02-13T00:44:39.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.937+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c12bb60 osd=7 4 2024-02-13T00:44:39.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.937+0000 7ff250ff1700 10 client.27087.objecter _send_command 16 2024-02-13T00:44:39.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.937+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff24c12e2c0 con 0x7ff24c12c240 2024-02-13T00:44:39.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff23b7fe700 1 -- 172.21.15.162:0/898071383 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff24c12e2c0 con 0x7ff24c12c240 2024-02-13T00:44:39.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff23b7fe700 10 client.27087.objecter ms_dispatch 0x7ff24c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:39.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff23b7fe700 10 client.27087.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff23b7fe700 20 client.27087.objecter put_session s=0x7ff24c12bb60 osd=7 4 2024-02-13T00:44:39.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff23b7fe700 15 client.27087.objecter _session_command_op_remove 7 16 2024-02-13T00:44:39.943 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:44:39.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 10 librados: watch_flush enter 2024-02-13T00:44:39.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 10 librados: watch_flush exit 2024-02-13T00:44:39.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 10 client.27087.objecter close_session for osd.0 2024-02-13T00:44:39.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff24c028590 msgr2=0x7ff24c0fbd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff24c028590 0x7ff24c0fbd00 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c1071b0 osd=0 1 2024-02-13T00:44:39.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 10 client.27087.objecter close_session for osd.1 2024-02-13T00:44:39.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6805/150599780 conn(0x7ff24c027be0 msgr2=0x7ff24c027fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6805/150599780 conn(0x7ff24c027be0 0x7ff24c027fc0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fd370 osd=1 1 2024-02-13T00:44:39.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 10 client.27087.objecter close_session for osd.2 2024-02-13T00:44:39.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff24c11a120 msgr2=0x7ff24c11c510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff24c11a120 0x7ff24c11c510 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fc9c0 osd=2 1 2024-02-13T00:44:39.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 10 client.27087.objecter close_session for osd.3 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6813/604971011 conn(0x7ff24c119c80 msgr2=0x7ff24c11fb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff24ad9d700 1 -- 172.21.15.162:0/898071383 reap_dead start 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6813/604971011 conn(0x7ff24c119c80 0x7ff24c11fb90 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c0fc840 osd=3 1 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 10 client.27087.objecter close_session for osd.4 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff24c119350 msgr2=0x7ff24c122090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff24c119350 0x7ff24c122090 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c1191e0 osd=4 1 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 10 client.27087.objecter close_session for osd.5 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff24c0f8190 msgr2=0x7ff24c0f85b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff24c0f8190 0x7ff24c0f85b0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c122670 osd=5 1 2024-02-13T00:44:39.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.939+0000 7ff250ff1700 10 client.27087.objecter close_session for osd.6 2024-02-13T00:44:39.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.940+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff24c129280 msgr2=0x7ff24c12b670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.940+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff24c129280 0x7ff24c12b670 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.940+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c11d810 osd=6 1 2024-02-13T00:44:39.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.940+0000 7ff250ff1700 10 client.27087.objecter close_session for osd.7 2024-02-13T00:44:39.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.940+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff24c12c240 msgr2=0x7ff24c12c6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.940+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff24c12c240 0x7ff24c12c6a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.940+0000 7ff250ff1700 20 client.27087.objecter put_session s=0x7ff24c12bb60 osd=7 1 2024-02-13T00:44:39.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.940+0000 7ff250ff1700 20 client.27087.objecter shutdown clearing up homeless session... 2024-02-13T00:44:39.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.940+0000 7ff250ff1700 10 client.27087.objecter successfully canceled tick 2024-02-13T00:44:39.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.940+0000 7ff24ad9d700 1 -- 172.21.15.162:0/898071383 reap_dead start 2024-02-13T00:44:39.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.942+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff234062c90 msgr2=0x7ff234065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.942+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff234062c90 0x7ff234065150 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 rx=0x7ff23c011510 tx=0x7ff23c00b040).stop 2024-02-13T00:44:39.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.942+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff24c0f95e0 msgr2=0x7ff24c10e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:39.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.942+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff24c0f95e0 0x7ff24c10e5e0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7ff240009400 tx=0x7ff240009cb0).stop 2024-02-13T00:44:39.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.942+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 shutdown_connections 2024-02-13T00:44:39.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.942+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff24c0f95e0 0x7ff24c10e5e0 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.942+0000 7ff250ff1700 1 --2- 172.21.15.162:0/898071383 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff234062c90 0x7ff234065150 unknown :-1 s=CLOSED pgs=2859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:39.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.942+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 >> 172.21.15.162:0/898071383 conn(0x7ff24c06eb90 msgr2=0x7ff24c102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:39.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.942+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 shutdown_connections 2024-02-13T00:44:39.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.942+0000 7ff250ff1700 1 -- 172.21.15.162:0/898071383 wait complete. 2024-02-13T00:44:39.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:39.942+0000 7ff250ff1700 1 librados: shutdown 2024-02-13T00:44:39.948 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.948 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.948 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.948 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.948 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.948 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.948 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:44:39.948 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:44:40.167 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:44:40.167 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:44:40.397 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:40.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:40.413 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-02-13T00:44:40.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:40.546 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:40.670 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:40.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:40.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:40.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:39.777962+0000 mon.a (mon.0) 2109 : audit [DBG] from='client.? 172.21.15.162:0/898071383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:40.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:40.372642+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_period","val": "3"}]': finished 2024-02-13T00:44:40.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:40.377692+0000 mon.b (mon.1) 489 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:44:40.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:40.400192+0000 mon.c (mon.2) 315 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-13T00:44:40.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:40.417370+0000 mon.b (mon.1) 490 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:40.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:40.625029+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e325: 8 total, 8 up, 7 in 2024-02-13T00:44:40.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:40.626284+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:44:40.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:40.631089+0000 mon.a (mon.0) 2113 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-13T00:44:40.801 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:40.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-13T00:44:41.034 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:39.777962+0000 mon.a (mon.0) 2109 : audit [DBG] from='client.? 172.21.15.162:0/898071383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:41.034 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:40.372642+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_period","val": "3"}]': finished 2024-02-13T00:44:41.034 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:40.377692+0000 mon.b (mon.1) 489 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:44:41.034 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:40.400192+0000 mon.c (mon.2) 315 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-13T00:44:41.034 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:40.417370+0000 mon.b (mon.1) 490 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:41.034 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:40.625029+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e325: 8 total, 8 up, 7 in 2024-02-13T00:44:41.034 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:40.626284+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:44:41.034 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:40.631089+0000 mon.a (mon.0) 2113 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-13T00:44:41.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:39.777962+0000 mon.a (mon.0) 2109 : audit [DBG] from='client.? 172.21.15.162:0/898071383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:41.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:40.372642+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_period","val": "3"}]': finished 2024-02-13T00:44:41.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:40.377692+0000 mon.b (mon.1) 489 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:44:41.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:40.400192+0000 mon.c (mon.2) 315 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-13T00:44:41.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:40.417370+0000 mon.b (mon.1) 490 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:41.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:40.625029+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e325: 8 total, 8 up, 7 in 2024-02-13T00:44:41.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:40.626284+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:44:41.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:40.631089+0000 mon.a (mon.0) 2113 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-13T00:44:41.035 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:41.050 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:41.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:41.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:41.198 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:41.219 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:41.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:41.254 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:44:41.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:41.352 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:41.373 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:41.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:41.476 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:41.510 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:41.540 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:41.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:41.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:41.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:44:41.681 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:41.788 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:44:41.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:41.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:44:41.792 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:40.376375+0000 osd.1 (osd.1) 133 : cluster [DBG] 18.a starting backfill to osd.6 from (0'0,0'0] MAX to 309'14 2024-02-13T00:44:41.792 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:40.376408+0000 osd.1 (osd.1) 134 : cluster [DBG] 18.b starting backfill to osd.6 from (0'0,0'0] MAX to 309'14 2024-02-13T00:44:41.792 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:40.377050+0000 osd.1 (osd.1) 135 : cluster [DBG] 18.d starting backfill to osd.6 from (0'0,0'0] MAX to 309'9 2024-02-13T00:44:41.792 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:40.392854+0000 mgr.y (mgr.14544) 301 : cluster [DBG] pgmap v458: 264 pgs: 5 remapped+peering, 227 active+clean, 25 active+recovery_wait+degraded, 5 active+recovering, 2 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 781 B/s rd, 2.0 KiB/s wr, 4 op/s; 177/1092 objects degraded (16.209%); 8/1092 objects misplaced (0.733%); 112 B/s, 10 objects/s recovering 2024-02-13T00:44:41.792 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:41.375744+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi162-83985-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi162-83985-56"}]': finished 2024-02-13T00:44:41.792 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:41.375832+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:44:41.793 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:41.375906+0000 mon.a (mon.0) 2116 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-13T00:44:41.793 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:41.375962+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e326: 8 total, 8 up, 7 in 2024-02-13T00:44:41.793 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:41.378461+0000 mon.b (mon.1) 491 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-13T00:44:41.793 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:41.381026+0000 mon.a (mon.0) 2118 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-13T00:44:41.793 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:41.418198+0000 mon.b (mon.1) 492 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:41.793 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:41.626643+0000 mon.a (mon.0) 2119 : cluster [WRN] Health check failed: Degraded data redundancy: 177/1092 objects degraded (16.209%), 25 pgs degraded (PG_DEGRADED) 2024-02-13T00:44:41.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:40.376375+0000 osd.1 (osd.1) 133 : cluster [DBG] 18.a starting backfill to osd.6 from (0'0,0'0] MAX to 309'14 2024-02-13T00:44:41.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:40.376408+0000 osd.1 (osd.1) 134 : cluster [DBG] 18.b starting backfill to osd.6 from (0'0,0'0] MAX to 309'14 2024-02-13T00:44:41.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:40.377050+0000 osd.1 (osd.1) 135 : cluster [DBG] 18.d starting backfill to osd.6 from (0'0,0'0] MAX to 309'9 2024-02-13T00:44:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:40.392854+0000 mgr.y (mgr.14544) 301 : cluster [DBG] pgmap v458: 264 pgs: 5 remapped+peering, 227 active+clean, 25 active+recovery_wait+degraded, 5 active+recovering, 2 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 781 B/s rd, 2.0 KiB/s wr, 4 op/s; 177/1092 objects degraded (16.209%); 8/1092 objects misplaced (0.733%); 112 B/s, 10 objects/s recovering 2024-02-13T00:44:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:41.375744+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi162-83985-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi162-83985-56"}]': finished 2024-02-13T00:44:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:41.375832+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:44:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:41.375906+0000 mon.a (mon.0) 2116 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-13T00:44:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:41.375962+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e326: 8 total, 8 up, 7 in 2024-02-13T00:44:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:41.378461+0000 mon.b (mon.1) 491 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-13T00:44:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:41.381026+0000 mon.a (mon.0) 2118 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-13T00:44:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:41.418198+0000 mon.b (mon.1) 492 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:41.626643+0000 mon.a (mon.0) 2119 : cluster [WRN] Health check failed: Degraded data redundancy: 177/1092 objects degraded (16.209%), 25 pgs degraded (PG_DEGRADED) 2024-02-13T00:44:41.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:40.376375+0000 osd.1 (osd.1) 133 : cluster [DBG] 18.a starting backfill to osd.6 from (0'0,0'0] MAX to 309'14 2024-02-13T00:44:41.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:40.376408+0000 osd.1 (osd.1) 134 : cluster [DBG] 18.b starting backfill to osd.6 from (0'0,0'0] MAX to 309'14 2024-02-13T00:44:41.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:40.377050+0000 osd.1 (osd.1) 135 : cluster [DBG] 18.d starting backfill to osd.6 from (0'0,0'0] MAX to 309'9 2024-02-13T00:44:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:40.392854+0000 mgr.y (mgr.14544) 301 : cluster [DBG] pgmap v458: 264 pgs: 5 remapped+peering, 227 active+clean, 25 active+recovery_wait+degraded, 5 active+recovering, 2 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 781 B/s rd, 2.0 KiB/s wr, 4 op/s; 177/1092 objects degraded (16.209%); 8/1092 objects misplaced (0.733%); 112 B/s, 10 objects/s recovering 2024-02-13T00:44:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:41.375744+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi162-83985-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi162-83985-56"}]': finished 2024-02-13T00:44:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:41.375832+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:44:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:41.375906+0000 mon.a (mon.0) 2116 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-13T00:44:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:41.375962+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e326: 8 total, 8 up, 7 in 2024-02-13T00:44:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:41.378461+0000 mon.b (mon.1) 491 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-13T00:44:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:41.381026+0000 mon.a (mon.0) 2118 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-13T00:44:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:41.418198+0000 mon.b (mon.1) 492 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:41.626643+0000 mon.a (mon.0) 2119 : cluster [WRN] Health check failed: Degraded data redundancy: 177/1092 objects degraded (16.209%), 25 pgs degraded (PG_DEGRADED) 2024-02-13T00:44:41.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:41.919 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:42.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.934+0000 7f6345a6b700 1 -- 172.21.15.162:0/2364655255 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63400fcb30 msgr2=0x7f63400fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.934+0000 7f6345a6b700 1 --2- 172.21.15.162:0/2364655255 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63400fcb30 0x7f63400fcf10 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7f6330004870 tx=0x7f6330009cb0).stop 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.934+0000 7f6345a6b700 1 -- 172.21.15.162:0/2364655255 shutdown_connections 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.934+0000 7f6345a6b700 1 --2- 172.21.15.162:0/2364655255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63400f73a0 0x7f63401039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.934+0000 7f6345a6b700 1 --2- 172.21.15.162:0/2364655255 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63400fcb30 0x7f63400fcf10 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.934+0000 7f6345a6b700 1 --2- 172.21.15.162:0/2364655255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63400fd490 0x7f63400f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.934+0000 7f6345a6b700 1 -- 172.21.15.162:0/2364655255 >> 172.21.15.162:0/2364655255 conn(0x7f634006eb90 msgr2=0x7f63400f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.935+0000 7f6345a6b700 1 -- 172.21.15.162:0/2364655255 shutdown_connections 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.935+0000 7f6345a6b700 1 -- 172.21.15.162:0/2364655255 wait complete. 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.935+0000 7f6345a6b700 1 librados: starting msgr at 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.935+0000 7f6345a6b700 1 librados: starting objecter 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.935+0000 7f6345a6b700 1 Processor -- start 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.935+0000 7f6345a6b700 1 -- start start 2024-02-13T00:44:42.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.935+0000 7f6345a6b700 1 librados: setting wanted keys 2024-02-13T00:44:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.935+0000 7f6345a6b700 1 librados: calling monclient init 2024-02-13T00:44:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f6345a6b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63400f73a0 0x7f63401070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f6345a6b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63400fcb30 0x7f63401075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f6345a6b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63400fd490 0x7f6340107ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f6345a6b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6340116750 con 0x7f63400fcb30 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f6345a6b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6340116890 con 0x7f63400fd490 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f6345a6b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6340116a00 con 0x7f63400f73a0 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63400f73a0 0x7f63401070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63400f73a0 0x7f63401070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:46450/0 (socket says 172.21.15.162:46450) 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633f7fe700 1 -- 172.21.15.162:0/3270177184 learned_addr learned my addr 172.21.15.162:0/3270177184 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633effd700 1 --2- 172.21.15.162:0/3270177184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63400fcb30 0x7f63401075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633ffff700 1 --2- 172.21.15.162:0/3270177184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63400fd490 0x7f6340107ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633f7fe700 1 -- 172.21.15.162:0/3270177184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63400fd490 msgr2=0x7f6340107ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633f7fe700 1 --2- 172.21.15.162:0/3270177184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63400fd490 0x7f6340107ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633f7fe700 1 -- 172.21.15.162:0/3270177184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63400fcb30 msgr2=0x7f63401075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633f7fe700 1 --2- 172.21.15.162:0/3270177184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63400fcb30 0x7f63401075c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633f7fe700 1 -- 172.21.15.162:0/3270177184 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6334005040 con 0x7f63400f73a0 2024-02-13T00:44:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633effd700 1 --2- 172.21.15.162:0/3270177184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63400fcb30 0x7f63401075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633f7fe700 1 --2- 172.21.15.162:0/3270177184 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63400f73a0 0x7f63401070d0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7f63300043a0 tx=0x7f6330009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633cf74700 10 client.?.objecter ms_handle_connect 0x7f63400f73a0 2024-02-13T00:44:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633cf74700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.936+0000 7f633cf74700 1 -- 172.21.15.162:0/3270177184 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633000e030 con 0x7f63400f73a0 2024-02-13T00:44:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.937+0000 7f633cf74700 1 -- 172.21.15.162:0/3270177184 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6330013da0 con 0x7f63400f73a0 2024-02-13T00:44:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.937+0000 7f6345a6b700 1 -- 172.21.15.162:0/3270177184 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6330005040 con 0x7f63400f73a0 2024-02-13T00:44:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.937+0000 7f633cf74700 1 -- 172.21.15.162:0/3270177184 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6330018070 con 0x7f63400f73a0 2024-02-13T00:44:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.937+0000 7f6345a6b700 10 client.37202.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.937+0000 7f6345a6b700 1 -- 172.21.15.162:0/3270177184 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6340116f80 con 0x7f63400f73a0 2024-02-13T00:44:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.937+0000 7f6345a6b700 1 librados: init done 2024-02-13T00:44:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.938+0000 7f6345a6b700 1 -- 172.21.15.162:0/3270177184 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63400475e0 con 0x7f63400f73a0 2024-02-13T00:44:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.938+0000 7f633cf74700 1 -- 172.21.15.162:0/3270177184 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f633001d420 con 0x7f63400f73a0 2024-02-13T00:44:42.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.939+0000 7f633cf74700 1 --2- 172.21.15.162:0/3270177184 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6328062c90 0x7f6328065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:42.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.939+0000 7f633cf74700 1 -- 172.21.15.162:0/3270177184 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(327..327 src has 52..327) v4 ==== 9345+0+0 (secure 0 0 0) 0x7f633007e860 con 0x7f63400f73a0 2024-02-13T00:44:42.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.939+0000 7f633cf74700 10 client.37202.objecter ms_dispatch 0x7f6340001550 osd_map(327..327 src has 52..327) v4 2024-02-13T00:44:42.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.939+0000 7f633cf74700 3 client.37202.objecter handle_osd_map got epochs [327,327] > 0 2024-02-13T00:44:42.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.939+0000 7f633cf74700 3 client.37202.objecter handle_osd_map decoding full epoch 327 2024-02-13T00:44:42.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.939+0000 7f633effd700 1 --2- 172.21.15.162:0/3270177184 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6328062c90 0x7f6328065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:42.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.939+0000 7f633cf74700 20 client.37202.objecter dump_active .. 0 homeless 2024-02-13T00:44:42.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.939+0000 7f633cf74700 10 client.37202.objecter ms_handle_connect 0x7f6328062c90 2024-02-13T00:44:42.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.939+0000 7f633effd700 1 --2- 172.21.15.162:0/3270177184 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6328062c90 0x7f6328065150 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 rx=0x7f63400617b0 tx=0x7f633400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:42.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:42.943+0000 7f633cf74700 1 -- 172.21.15.162:0/3270177184 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6330051820 con 0x7f63400f73a0 2024-02-13T00:44:42.968 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:42.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:43.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:44:43.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:43.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.135+0000 7f6345a6b700 1 -- 172.21.15.162:0/3270177184 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f634005fb80 con 0x7f63400f73a0 2024-02-13T00:44:43.141 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:43.165 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:43.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:43.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:43.302 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:43.312 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:44:43.312 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:44:43.324 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:43.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:43.352 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:43.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.396+0000 7f633cf74700 1 -- 172.21.15.162:0/3270177184 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v328) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6330051410 con 0x7f63400f73a0 2024-02-13T00:44:43.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.398+0000 7f6345a6b700 10 librados: watch_flush enter 2024-02-13T00:44:43.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.398+0000 7f6345a6b700 10 librados: watch_flush exit 2024-02-13T00:44:43.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.398+0000 7f6345a6b700 20 client.37202.objecter shutdown clearing up homeless session... 2024-02-13T00:44:43.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.398+0000 7f6345a6b700 10 client.37202.objecter successfully canceled tick 2024-02-13T00:44:43.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.398+0000 7f6345a6b700 1 -- 172.21.15.162:0/3270177184 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6328062c90 msgr2=0x7f6328065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:43.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.398+0000 7f6345a6b700 1 --2- 172.21.15.162:0/3270177184 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6328062c90 0x7f6328065150 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 rx=0x7f63400617b0 tx=0x7f633400b040).stop 2024-02-13T00:44:43.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.398+0000 7f6345a6b700 1 -- 172.21.15.162:0/3270177184 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63400f73a0 msgr2=0x7f63401070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:43.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.398+0000 7f6345a6b700 1 --2- 172.21.15.162:0/3270177184 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63400f73a0 0x7f63401070d0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7f63300043a0 tx=0x7f6330009cb0).stop 2024-02-13T00:44:43.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.399+0000 7f6345a6b700 1 -- 172.21.15.162:0/3270177184 shutdown_connections 2024-02-13T00:44:43.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.399+0000 7f6345a6b700 1 --2- 172.21.15.162:0/3270177184 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6328062c90 0x7f6328065150 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:43.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.399+0000 7f6345a6b700 1 --2- 172.21.15.162:0/3270177184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63400fd490 0x7f6340107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:43.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.399+0000 7f6345a6b700 1 --2- 172.21.15.162:0/3270177184 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63400f73a0 0x7f63401070d0 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:43.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.399+0000 7f6345a6b700 1 --2- 172.21.15.162:0/3270177184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63400fcb30 0x7f63401075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:43.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.399+0000 7f6345a6b700 1 -- 172.21.15.162:0/3270177184 >> 172.21.15.162:0/3270177184 conn(0x7f634006eb90 msgr2=0x7f63400fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:43.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.399+0000 7f6345a6b700 1 -- 172.21.15.162:0/3270177184 shutdown_connections 2024-02-13T00:44:43.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.399+0000 7f6345a6b700 1 -- 172.21.15.162:0/3270177184 wait complete. 2024-02-13T00:44:43.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:43.399+0000 7f6345a6b700 1 librados: shutdown 2024-02-13T00:44:43.403 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:44:43.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:43.454 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:43.610 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:43.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:43.660 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:42.376425+0000 mon.a (mon.0) 2120 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-13T00:44:43.660 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:42.376546+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e327: 8 total, 8 up, 7 in 2024-02-13T00:44:43.660 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:42.393376+0000 mgr.y (mgr.14544) 302 : cluster [DBG] pgmap v461: 272 pgs: 8 unknown, 5 remapped+peering, 227 active+clean, 25 active+recovery_wait+degraded, 5 active+recovering, 2 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 781 B/s rd, 3.1 KiB/s wr, 6 op/s; 177/1092 objects degraded (16.209%); 8/1092 objects misplaced (0.733%); 177 B/s, 10 objects/s recovering 2024-02-13T00:44:43.660 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:42.419071+0000 mon.b (mon.1) 493 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:43.660 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:43.136281+0000 mon.c (mon.2) 316 : audit [INF] from='client.? 172.21.15.162:0/3270177184' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:43.660 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:43.136683+0000 mon.a (mon.0) 2122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:43.661 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:42.376425+0000 mon.a (mon.0) 2120 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-13T00:44:43.661 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:42.376546+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e327: 8 total, 8 up, 7 in 2024-02-13T00:44:43.661 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:42.393376+0000 mgr.y (mgr.14544) 302 : cluster [DBG] pgmap v461: 272 pgs: 8 unknown, 5 remapped+peering, 227 active+clean, 25 active+recovery_wait+degraded, 5 active+recovering, 2 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 781 B/s rd, 3.1 KiB/s wr, 6 op/s; 177/1092 objects degraded (16.209%); 8/1092 objects misplaced (0.733%); 177 B/s, 10 objects/s recovering 2024-02-13T00:44:43.661 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:42.419071+0000 mon.b (mon.1) 493 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:43.661 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:43.136281+0000 mon.c (mon.2) 316 : audit [INF] from='client.? 172.21.15.162:0/3270177184' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:43.661 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:43.136683+0000 mon.a (mon.0) 2122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:43.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:43.743 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:42.376425+0000 mon.a (mon.0) 2120 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-43","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-13T00:44:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:42.376546+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e327: 8 total, 8 up, 7 in 2024-02-13T00:44:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:42.393376+0000 mgr.y (mgr.14544) 302 : cluster [DBG] pgmap v461: 272 pgs: 8 unknown, 5 remapped+peering, 227 active+clean, 25 active+recovery_wait+degraded, 5 active+recovering, 2 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 781 B/s rd, 3.1 KiB/s wr, 6 op/s; 177/1092 objects degraded (16.209%); 8/1092 objects misplaced (0.733%); 177 B/s, 10 objects/s recovering 2024-02-13T00:44:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:42.419071+0000 mon.b (mon.1) 493 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:43.136281+0000 mon.c (mon.2) 316 : audit [INF] from='client.? 172.21.15.162:0/3270177184' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:43.136683+0000 mon.a (mon.0) 2122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:43.872 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:43.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:43.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:44.002 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:44.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-13T00:44:44.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:43.374232+0000 mon.a (mon.0) 2123 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:44.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:43.387028+0000 mon.a (mon.0) 2124 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:44:44.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:43.387104+0000 mon.a (mon.0) 2125 : cluster [DBG] osdmap e328: 8 total, 8 up, 7 in 2024-02-13T00:44:44.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:43.388357+0000 mon.a (mon.0) 2126 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:44.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:43.420120+0000 mon.b (mon.1) 494 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:44.611 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:44:44.701 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:43.374232+0000 mon.a (mon.0) 2123 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:44.702 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:43.387028+0000 mon.a (mon.0) 2124 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:44:44.702 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:43.387104+0000 mon.a (mon.0) 2125 : cluster [DBG] osdmap e328: 8 total, 8 up, 7 in 2024-02-13T00:44:44.702 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:43.388357+0000 mon.a (mon.0) 2126 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:44.702 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:43.420120+0000 mon.b (mon.1) 494 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:44.702 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:43.374232+0000 mon.a (mon.0) 2123 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:44.702 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:43.387028+0000 mon.a (mon.0) 2124 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:44:44.702 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:43.387104+0000 mon.a (mon.0) 2125 : cluster [DBG] osdmap e328: 8 total, 8 up, 7 in 2024-02-13T00:44:44.702 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:43.388357+0000 mon.a (mon.0) 2126 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:44.702 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:43.420120+0000 mon.b (mon.1) 494 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:44.757 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:44.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:44.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:44.894 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:45.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:45.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:45.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:45.384 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:45.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:44.391449+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-83985-56"}]': finished 2024-02-13T00:44:45.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:44.391514+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-02-13T00:44:45.400 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:44.391449+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-83985-56"}]': finished 2024-02-13T00:44:45.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:44.391514+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-02-13T00:44:45.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:44.392490+0000 mon.a (mon.0) 2129 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:45.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:44.393974+0000 mgr.y (mgr.14544) 303 : cluster [DBG] pgmap v464: 263 pgs: 5 remapped+peering, 226 active+clean, 25 active+recovery_wait+degraded, 5 active+recovering, 2 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 177/1092 objects degraded (16.209%); 8/1092 objects misplaced (0.733%) 2024-02-13T00:44:45.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:44.395620+0000 mon.c (mon.2) 317 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-13T00:44:45.401 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:44.396972+0000 mon.a (mon.0) 2130 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-13T00:44:45.413 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:44:45.413 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:44:45.413 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 4, 5, 7, 6] 2024-02-13T00:44:45.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd out 2 2024-02-13T00:44:45.489 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:45.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:44.391449+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-83985-56"}]': finished 2024-02-13T00:44:45.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:44.391514+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-02-13T00:44:45.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:44.392490+0000 mon.a (mon.0) 2129 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:45.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:44.393974+0000 mgr.y (mgr.14544) 303 : cluster [DBG] pgmap v464: 263 pgs: 5 remapped+peering, 226 active+clean, 25 active+recovery_wait+degraded, 5 active+recovering, 2 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 177/1092 objects degraded (16.209%); 8/1092 objects misplaced (0.733%) 2024-02-13T00:44:45.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:44.395620+0000 mon.c (mon.2) 317 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-13T00:44:45.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:44.396972+0000 mon.a (mon.0) 2130 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-13T00:44:45.490 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:44.421057+0000 mon.b (mon.1) 495 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:45.491 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:45.375114+0000 mon.a (mon.0) 2131 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-83985-56"}]': finished 2024-02-13T00:44:45.491 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:45.375185+0000 mon.a (mon.0) 2132 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-13T00:44:45.491 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:45.375280+0000 mon.a (mon.0) 2133 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-02-13T00:44:45.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:45.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:45.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:44.421057+0000 mon.b (mon.1) 495 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:45.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:45.375114+0000 mon.a (mon.0) 2131 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-83985-56"}]': finished 2024-02-13T00:44:45.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:45.375185+0000 mon.a (mon.0) 2132 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-13T00:44:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:45.375280+0000 mon.a (mon.0) 2133 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-02-13T00:44:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:44.392490+0000 mon.a (mon.0) 2129 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-83985-56"}]: dispatch 2024-02-13T00:44:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:44.393974+0000 mgr.y (mgr.14544) 303 : cluster [DBG] pgmap v464: 263 pgs: 5 remapped+peering, 226 active+clean, 25 active+recovery_wait+degraded, 5 active+recovering, 2 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 177/1092 objects degraded (16.209%); 8/1092 objects misplaced (0.733%) 2024-02-13T00:44:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:44.395620+0000 mon.c (mon.2) 317 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-13T00:44:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:44.396972+0000 mon.a (mon.0) 2130 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-13T00:44:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:44.421057+0000 mon.b (mon.1) 495 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:45.375114+0000 mon.a (mon.0) 2131 : audit [INF] from='client.? 172.21.15.162:0/2155676044' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-83985-56"}]': finished 2024-02-13T00:44:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:45.375185+0000 mon.a (mon.0) 2132 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-13T00:44:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:45.375280+0000 mon.a (mon.0) 2133 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-02-13T00:44:45.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:44:45.617 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:45.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:44:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:44:46.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.281+0000 7fa780d3e700 1 -- 172.21.15.162:0/639886069 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa77c102270 msgr2=0x7fa77c104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.281+0000 7fa780d3e700 1 --2- 172.21.15.162:0/639886069 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa77c102270 0x7fa77c104660 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 rx=0x7fa774009be0 tx=0x7fa774004d10).stop 2024-02-13T00:44:46.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.281+0000 7fa780d3e700 1 -- 172.21.15.162:0/639886069 shutdown_connections 2024-02-13T00:44:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.281+0000 7fa780d3e700 1 --2- 172.21.15.162:0/639886069 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa77c102270 0x7fa77c104660 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.281+0000 7fa780d3e700 1 --2- 172.21.15.162:0/639886069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa77c0f8d10 0x7fa77c0f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.281+0000 7fa780d3e700 1 --2- 172.21.15.162:0/639886069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa77c0f95e0 0x7fa77c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.281+0000 7fa780d3e700 1 -- 172.21.15.162:0/639886069 >> 172.21.15.162:0/639886069 conn(0x7fa77c06eb90 msgr2=0x7fa77c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.282+0000 7fa780d3e700 1 -- 172.21.15.162:0/639886069 shutdown_connections 2024-02-13T00:44:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.282+0000 7fa780d3e700 1 -- 172.21.15.162:0/639886069 wait complete. 2024-02-13T00:44:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.282+0000 7fa780d3e700 1 librados: starting msgr at 2024-02-13T00:44:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.282+0000 7fa780d3e700 1 librados: starting objecter 2024-02-13T00:44:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.282+0000 7fa780d3e700 1 Processor -- start 2024-02-13T00:44:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.282+0000 7fa780d3e700 1 -- start start 2024-02-13T00:44:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.282+0000 7fa780d3e700 1 librados: setting wanted keys 2024-02-13T00:44:46.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.282+0000 7fa780d3e700 1 librados: calling monclient init 2024-02-13T00:44:46.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa780d3e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa77c0f8d10 0x7fa77c0f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:46.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa780d3e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa77c0f95e0 0x7fa77c0f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:46.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa780d3e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa77c102270 0x7fa77c0f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:46.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa780d3e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c116560 con 0x7fa77c102270 2024-02-13T00:44:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa780d3e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c1166d0 con 0x7fa77c0f95e0 2024-02-13T00:44:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa780d3e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c116890 con 0x7fa77c0f8d10 2024-02-13T00:44:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa77ad9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa77c102270 0x7fa77c0f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa779d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa77c0f95e0 0x7fa77c0f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa779d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa77c0f95e0 0x7fa77c0f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:37596/0 (socket says 172.21.15.162:37596) 2024-02-13T00:44:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa77ad9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa77c102270 0x7fa77c0f76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46616/0 (socket says 172.21.15.162:46616) 2024-02-13T00:44:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa779d9b700 1 -- 172.21.15.162:0/1672471096 learned_addr learned my addr 172.21.15.162:0/1672471096 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa77a59c700 1 --2- 172.21.15.162:0/1672471096 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa77c0f8d10 0x7fa77c0f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa77ad9d700 1 -- 172.21.15.162:0/1672471096 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa77c0f8d10 msgr2=0x7fa77c0f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa77ad9d700 1 --2- 172.21.15.162:0/1672471096 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa77c0f8d10 0x7fa77c0f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa77ad9d700 1 -- 172.21.15.162:0/1672471096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa77c0f95e0 msgr2=0x7fa77c0f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa77ad9d700 1 --2- 172.21.15.162:0/1672471096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa77c0f95e0 0x7fa77c0f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa77ad9d700 1 -- 172.21.15.162:0/1672471096 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c005040 con 0x7fa77c102270 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa77a59c700 1 --2- 172.21.15.162:0/1672471096 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa77c0f8d10 0x7fa77c0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa779d9b700 1 --2- 172.21.15.162:0/1672471096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa77c0f95e0 0x7fa77c0f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa77ad9d700 1 --2- 172.21.15.162:0/1672471096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa77c102270 0x7fa77c0f76a0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fa774009bb0 tx=0x7fa774004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa76b7fe700 10 client.?.objecter ms_handle_connect 0x7fa77c102270 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa76b7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77400e040 con 0x7fa77c102270 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7740097c0 con 0x7fa77c102270 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa774018040 con 0x7fa77c102270 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.283+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774013e90 con 0x7fa77c102270 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.284+0000 7fa780d3e700 10 client.27102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.284+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c116d80 con 0x7fa77c102270 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.284+0000 7fa780d3e700 1 librados: init done 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.284+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa77c107170 con 0x7fa77c102270 2024-02-13T00:44:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.285+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fa774013520 con 0x7fa77c102270 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.285+0000 7fa76b7fe700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa764062c90 0x7fa764065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.285+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(330..330 src has 52..330) v4 ==== 8429+0+0 (secure 0 0 0) 0x7fa77407e960 con 0x7fa77c102270 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.285+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 osd_map(330..330 src has 52..330) v4 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.285+0000 7fa76b7fe700 3 client.27102.objecter handle_osd_map got epochs [330,330] > 0 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.285+0000 7fa76b7fe700 3 client.27102.objecter handle_osd_map decoding full epoch 330 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.285+0000 7fa77a59c700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa764062c90 0x7fa764065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa76b7fe700 20 client.27102.objecter dump_active .. 0 homeless 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v330) v1 ==== 54+0+16 (secure 0 0 0) 0x7fa77407f0a0 con 0x7fa77c102270 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa76b7fe700 10 client.27102.objecter ms_handle_connect 0x7fa764062c90 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa77a59c700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa764062c90 0x7fa764065150 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 rx=0x7fa76c009760 tx=0x7fa76c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa780d3e700 10 client.27102.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 1 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa77c0fc210 0x7fa77c0fc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c0f8150 osd=0 3 2024-02-13T00:44:46.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c0f8150 osd=0 3 2024-02-13T00:44:46.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c0f8150 osd=0 3 2024-02-13T00:44:46.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 1 2024-02-13T00:44:46.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c0f8150 osd=0 3 2024-02-13T00:44:46.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 0 1 2024-02-13T00:44:46.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c0f8150 osd=0 4 2024-02-13T00:44:46.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa780d3e700 10 client.27102.objecter _send_command 1 2024-02-13T00:44:46.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa77c0fcb00 con 0x7fa77c0fc210 2024-02-13T00:44:46.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa779d9b700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa77c0fc210 0x7fa77c0fc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:46.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa76b7fe700 10 client.27102.objecter ms_handle_connect 0x7fa77c0fc210 2024-02-13T00:44:46.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.286+0000 7fa779d9b700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa77c0fc210 0x7fa77c0fc610 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:46.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.287+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa77c0fcb00 con 0x7fa77c0fc210 2024-02-13T00:44:46.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.287+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:44:46.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.287+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 1 = system:0 2024-02-13T00:44:46.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.287+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c0f8150 osd=0 4 2024-02-13T00:44:46.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.287+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 0 1 2024-02-13T00:44:46.299 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:44:46.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:46.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:44:46.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.303+0000 7fa780d3e700 10 client.27102.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.303+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 2 2024-02-13T00:44:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.303+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c0f8150 osd=0 3 2024-02-13T00:44:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.303+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c0f8150 osd=0 3 2024-02-13T00:44:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.303+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c0f8150 osd=0 3 2024-02-13T00:44:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.303+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 2 2024-02-13T00:44:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.303+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c0f8150 osd=0 3 2024-02-13T00:44:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.303+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 0 2 2024-02-13T00:44:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.303+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c0f8150 osd=0 4 2024-02-13T00:44:46.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.303+0000 7fa780d3e700 10 client.27102.objecter _send_command 2 2024-02-13T00:44:46.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.303+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa77c0fccc0 con 0x7fa77c0fc210 2024-02-13T00:44:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa77c0fccc0 con 0x7fa77c0fc210 2024-02-13T00:44:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c0f8150 osd=0 4 2024-02-13T00:44:46.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 0 2 2024-02-13T00:44:46.310 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:44:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa780d3e700 10 client.27102.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 3 2024-02-13T00:44:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6805/150599780 conn(0x7fa77c028590 0x7fa77c11af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c104300 osd=1 3 2024-02-13T00:44:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c104300 osd=1 3 2024-02-13T00:44:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c104300 osd=1 3 2024-02-13T00:44:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 3 2024-02-13T00:44:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c104300 osd=1 3 2024-02-13T00:44:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 1 3 2024-02-13T00:44:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c104300 osd=1 4 2024-02-13T00:44:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa780d3e700 10 client.27102.objecter _send_command 3 2024-02-13T00:44:46.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa77c104470 con 0x7fa77c028590 2024-02-13T00:44:46.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.306+0000 7fa77a59c700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6805/150599780 conn(0x7fa77c028590 0x7fa77c11af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:46.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.307+0000 7fa76b7fe700 10 client.27102.objecter ms_handle_connect 0x7fa77c028590 2024-02-13T00:44:46.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.307+0000 7fa77a59c700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6805/150599780 conn(0x7fa77c028590 0x7fa77c11af90 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:46.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.307+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa77c104470 con 0x7fa77c028590 2024-02-13T00:44:46.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.307+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:44:46.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.307+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 3 = system:0 2024-02-13T00:44:46.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.307+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c104300 osd=1 4 2024-02-13T00:44:46.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.308+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 1 3 2024-02-13T00:44:46.312 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:46.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.322+0000 7fa780d3e700 10 client.27102.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:46.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.322+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 4 2024-02-13T00:44:46.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.322+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c104300 osd=1 3 2024-02-13T00:44:46.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.322+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c104300 osd=1 3 2024-02-13T00:44:46.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.322+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c104300 osd=1 3 2024-02-13T00:44:46.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.322+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 4 2024-02-13T00:44:46.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.322+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c104300 osd=1 3 2024-02-13T00:44:46.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.322+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 1 4 2024-02-13T00:44:46.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.322+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c104300 osd=1 4 2024-02-13T00:44:46.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.322+0000 7fa780d3e700 10 client.27102.objecter _send_command 4 2024-02-13T00:44:46.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.322+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa77c05a0e0 con 0x7fa77c028590 2024-02-13T00:44:46.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.325+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa77c05a0e0 con 0x7fa77c028590 2024-02-13T00:44:46.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.325+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:46.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.325+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.325+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c104300 osd=1 4 2024-02-13T00:44:46.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.325+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 1 4 2024-02-13T00:44:46.329 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:44:46.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.325+0000 7fa780d3e700 10 client.27102.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:46.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.325+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 5 2024-02-13T00:44:46.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa77c059570 0x7fa77c11e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:46.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c059400 osd=2 3 2024-02-13T00:44:46.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c059400 osd=2 3 2024-02-13T00:44:46.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c059400 osd=2 3 2024-02-13T00:44:46.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 5 2024-02-13T00:44:46.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c059400 osd=2 3 2024-02-13T00:44:46.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 2 5 2024-02-13T00:44:46.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c059400 osd=2 4 2024-02-13T00:44:46.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa780d3e700 10 client.27102.objecter _send_command 5 2024-02-13T00:44:46.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa77c0280f0 con 0x7fa77c059570 2024-02-13T00:44:46.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa779d9b700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa77c059570 0x7fa77c11e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:46.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa76b7fe700 10 client.27102.objecter ms_handle_connect 0x7fa77c059570 2024-02-13T00:44:46.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.326+0000 7fa779d9b700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa77c059570 0x7fa77c11e610 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:46.333 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:44:46] "GET /metrics HTTP/1.1" 200 189014 "" "Prometheus/2.33.4" 2024-02-13T00:44:46.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.327+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa77c0280f0 con 0x7fa77c059570 2024-02-13T00:44:46.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.327+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:44:46.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.327+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 5 = system:0 2024-02-13T00:44:46.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.327+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c059400 osd=2 4 2024-02-13T00:44:46.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.327+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 2 5 2024-02-13T00:44:46.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.342+0000 7fa780d3e700 10 client.27102.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:46.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.342+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 6 2024-02-13T00:44:46.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.342+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c059400 osd=2 3 2024-02-13T00:44:46.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.342+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c059400 osd=2 3 2024-02-13T00:44:46.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.342+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c059400 osd=2 3 2024-02-13T00:44:46.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.342+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 6 2024-02-13T00:44:46.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.342+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c059400 osd=2 3 2024-02-13T00:44:46.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.342+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 2 6 2024-02-13T00:44:46.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.342+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c059400 osd=2 4 2024-02-13T00:44:46.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.342+0000 7fa780d3e700 10 client.27102.objecter _send_command 6 2024-02-13T00:44:46.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.342+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa77c11eb00 con 0x7fa77c059570 2024-02-13T00:44:46.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa77c11eb00 con 0x7fa77c059570 2024-02-13T00:44:46.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:46.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c059400 osd=2 4 2024-02-13T00:44:46.348 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:44:46.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 2 6 2024-02-13T00:44:46.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa780d3e700 10 client.27102.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:46.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 7 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6813/604971011 conn(0x7fa77c0277c0 0x7fa77c121e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c027650 osd=3 3 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c027650 osd=3 3 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c027650 osd=3 3 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 7 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c027650 osd=3 3 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 3 7 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c027650 osd=3 4 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa780d3e700 10 client.27102.objecter _send_command 7 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa77c11c250 con 0x7fa77c0277c0 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa77ad9d700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6813/604971011 conn(0x7fa77c0277c0 0x7fa77c121e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa76b7fe700 10 client.27102.objecter ms_handle_connect 0x7fa77c0277c0 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.344+0000 7fa77ad9d700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6813/604971011 conn(0x7fa77c0277c0 0x7fa77c121e60 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.345+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa77c11c250 con 0x7fa77c0277c0 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.345+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:44:46.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.345+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 7 = system:0 2024-02-13T00:44:46.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.345+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c027650 osd=3 4 2024-02-13T00:44:46.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.345+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 3 7 2024-02-13T00:44:46.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.359+0000 7fa780d3e700 10 client.27102.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:46.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.359+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 8 2024-02-13T00:44:46.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.359+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c027650 osd=3 3 2024-02-13T00:44:46.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.359+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c027650 osd=3 3 2024-02-13T00:44:46.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.359+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c027650 osd=3 3 2024-02-13T00:44:46.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.359+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 8 2024-02-13T00:44:46.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.359+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c027650 osd=3 3 2024-02-13T00:44:46.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.359+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 3 8 2024-02-13T00:44:46.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.359+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c027650 osd=3 4 2024-02-13T00:44:46.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.359+0000 7fa780d3e700 10 client.27102.objecter _send_command 8 2024-02-13T00:44:46.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.359+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa77c11c250 con 0x7fa77c0277c0 2024-02-13T00:44:46.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa77c11c250 con 0x7fa77c0277c0 2024-02-13T00:44:46.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:46.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c027650 osd=3 4 2024-02-13T00:44:46.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 3 8 2024-02-13T00:44:46.364 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:44:46.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa780d3e700 10 client.27102.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:46.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 9 2024-02-13T00:44:46.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa77c11f440 0x7fa77c11f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:46.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c11c250 osd=4 3 2024-02-13T00:44:46.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c11c250 osd=4 3 2024-02-13T00:44:46.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c11c250 osd=4 3 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 9 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c11c250 osd=4 3 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 4 9 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c11c250 osd=4 4 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa780d3e700 10 client.27102.objecter _send_command 9 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa77c127c40 con 0x7fa77c11f440 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa77a59c700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa77c11f440 0x7fa77c11f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa77a59c700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa77c11f440 0x7fa77c11f8c0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.361+0000 7fa76b7fe700 10 client.27102.objecter ms_handle_connect 0x7fa77c11f440 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.362+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa77c127c40 con 0x7fa77c11f440 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.362+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.362+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 9 = system:0 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.362+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c11c250 osd=4 4 2024-02-13T00:44:46.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.362+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 4 9 2024-02-13T00:44:46.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.380+0000 7fa780d3e700 10 client.27102.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:46.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.380+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 10 2024-02-13T00:44:46.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.380+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c11c250 osd=4 3 2024-02-13T00:44:46.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.380+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c11c250 osd=4 3 2024-02-13T00:44:46.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.380+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c11c250 osd=4 3 2024-02-13T00:44:46.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.380+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 10 2024-02-13T00:44:46.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.380+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c11c250 osd=4 3 2024-02-13T00:44:46.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.380+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 4 10 2024-02-13T00:44:46.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.380+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c11c250 osd=4 4 2024-02-13T00:44:46.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.380+0000 7fa780d3e700 10 client.27102.objecter _send_command 10 2024-02-13T00:44:46.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.380+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa77c1254e0 con 0x7fa77c11f440 2024-02-13T00:44:46.388 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:44:46.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.382+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa77c1254e0 con 0x7fa77c11f440 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.382+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.382+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.382+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c11c250 osd=4 4 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.382+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 4 10 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa780d3e700 10 client.27102.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 11 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6804/2212296748 conn(0x7fa77c1271a0 0x7fa77c127600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c1254e0 osd=5 3 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c1254e0 osd=5 3 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c1254e0 osd=5 3 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 11 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c1254e0 osd=5 3 2024-02-13T00:44:46.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 5 11 2024-02-13T00:44:46.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c1254e0 osd=5 4 2024-02-13T00:44:46.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa780d3e700 10 client.27102.objecter _send_command 11 2024-02-13T00:44:46.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa77c12afc0 con 0x7fa77c1271a0 2024-02-13T00:44:46.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.385+0000 7fa779d9b700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6804/2212296748 conn(0x7fa77c1271a0 0x7fa77c127600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:46.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.386+0000 7fa76b7fe700 10 client.27102.objecter ms_handle_connect 0x7fa77c1271a0 2024-02-13T00:44:46.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.386+0000 7fa779d9b700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6804/2212296748 conn(0x7fa77c1271a0 0x7fa77c127600 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:46.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.391+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa77c12afc0 con 0x7fa77c1271a0 2024-02-13T00:44:46.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.391+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:44:46.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.391+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 11 = system:0 2024-02-13T00:44:46.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.391+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c1254e0 osd=5 4 2024-02-13T00:44:46.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.391+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 5 11 2024-02-13T00:44:46.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:46.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.406+0000 7fa780d3e700 10 client.27102.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:46.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.406+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 12 2024-02-13T00:44:46.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.407+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c1254e0 osd=5 3 2024-02-13T00:44:46.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.407+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c1254e0 osd=5 3 2024-02-13T00:44:46.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.407+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c1254e0 osd=5 3 2024-02-13T00:44:46.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.407+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 12 2024-02-13T00:44:46.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.407+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c1254e0 osd=5 3 2024-02-13T00:44:46.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.407+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 5 12 2024-02-13T00:44:46.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.407+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c1254e0 osd=5 4 2024-02-13T00:44:46.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.407+0000 7fa780d3e700 10 client.27102.objecter _send_command 12 2024-02-13T00:44:46.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.407+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa77c125680 con 0x7fa77c1271a0 2024-02-13T00:44:46.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa77c125680 con 0x7fa77c1271a0 2024-02-13T00:44:46.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:46.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c1254e0 osd=5 4 2024-02-13T00:44:46.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 5 12 2024-02-13T00:44:46.413 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:44:46.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa780d3e700 10 client.27102.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:46.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 13 2024-02-13T00:44:46.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa77c125d60 0x7fa77c1261c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:46.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c125680 osd=6 3 2024-02-13T00:44:46.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c125680 osd=6 3 2024-02-13T00:44:46.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c125680 osd=6 3 2024-02-13T00:44:46.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 13 2024-02-13T00:44:46.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c125680 osd=6 3 2024-02-13T00:44:46.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 6 13 2024-02-13T00:44:46.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c125680 osd=6 4 2024-02-13T00:44:46.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa780d3e700 10 client.27102.objecter _send_command 13 2024-02-13T00:44:46.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa77c126800 con 0x7fa77c125d60 2024-02-13T00:44:46.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.409+0000 7fa77ad9d700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa77c125d60 0x7fa77c1261c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:46.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.410+0000 7fa76b7fe700 10 client.27102.objecter ms_handle_connect 0x7fa77c125d60 2024-02-13T00:44:46.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.410+0000 7fa77ad9d700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa77c125d60 0x7fa77c1261c0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:46.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.411+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa77c126800 con 0x7fa77c125d60 2024-02-13T00:44:46.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.411+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:44:46.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.411+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 13 = system:0 2024-02-13T00:44:46.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.411+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c125680 osd=6 4 2024-02-13T00:44:46.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.411+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 6 13 2024-02-13T00:44:46.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.426+0000 7fa780d3e700 10 client.27102.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:46.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.426+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 14 2024-02-13T00:44:46.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.426+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c125680 osd=6 3 2024-02-13T00:44:46.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.426+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c125680 osd=6 3 2024-02-13T00:44:46.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.426+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c125680 osd=6 3 2024-02-13T00:44:46.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.426+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 14 2024-02-13T00:44:46.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.426+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c125680 osd=6 3 2024-02-13T00:44:46.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.426+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 6 14 2024-02-13T00:44:46.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.426+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c125680 osd=6 4 2024-02-13T00:44:46.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.426+0000 7fa780d3e700 10 client.27102.objecter _send_command 14 2024-02-13T00:44:46.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.426+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa77c126800 con 0x7fa77c125d60 2024-02-13T00:44:46.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa77c126800 con 0x7fa77c125d60 2024-02-13T00:44:46.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:46.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c125680 osd=6 4 2024-02-13T00:44:46.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 6 14 2024-02-13T00:44:46.434 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:44:46.434 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa780d3e700 10 client.27102.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 15 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa77c12c360 0x7fa77c12c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c126800 osd=7 3 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c126800 osd=7 3 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c126800 osd=7 3 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 15 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c126800 osd=7 3 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 7 15 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c126800 osd=7 4 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa780d3e700 10 client.27102.objecter _send_command 15 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.430+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa77c12e420 con 0x7fa77c12c360 2024-02-13T00:44:46.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.431+0000 7fa77a59c700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa77c12c360 0x7fa77c12c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:46.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.431+0000 7fa76b7fe700 10 client.27102.objecter ms_handle_connect 0x7fa77c12c360 2024-02-13T00:44:46.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.431+0000 7fa77a59c700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa77c12c360 0x7fa77c12c7a0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:46.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.432+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa77c12e420 con 0x7fa77c12c360 2024-02-13T00:44:46.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.432+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:44:46.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.432+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 15 = system:0 2024-02-13T00:44:46.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.432+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c126800 osd=7 4 2024-02-13T00:44:46.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.432+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 7 15 2024-02-13T00:44:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.451+0000 7fa780d3e700 10 client.27102.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:44:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.451+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign -1 16 2024-02-13T00:44:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.451+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c126800 osd=7 3 2024-02-13T00:44:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.451+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c126800 osd=7 3 2024-02-13T00:44:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.451+0000 7fa780d3e700 20 client.27102.objecter _get_session s=0x7fa77c126800 osd=7 3 2024-02-13T00:44:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.451+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_remove -1 16 2024-02-13T00:44:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.451+0000 7fa780d3e700 20 client.27102.objecter get_session s=0x7fa77c126800 osd=7 3 2024-02-13T00:44:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.451+0000 7fa780d3e700 15 client.27102.objecter _session_command_op_assign 7 16 2024-02-13T00:44:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.451+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c126800 osd=7 4 2024-02-13T00:44:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.451+0000 7fa780d3e700 10 client.27102.objecter _send_command 16 2024-02-13T00:44:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.451+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa77c12e420 con 0x7fa77c12c360 2024-02-13T00:44:46.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa76b7fe700 1 -- 172.21.15.162:0/1672471096 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa77c12e420 con 0x7fa77c12c360 2024-02-13T00:44:46.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa76b7fe700 10 client.27102.objecter ms_dispatch 0x7fa77c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:44:46.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa76b7fe700 10 client.27102.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa76b7fe700 20 client.27102.objecter put_session s=0x7fa77c126800 osd=7 4 2024-02-13T00:44:46.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa76b7fe700 15 client.27102.objecter _session_command_op_remove 7 16 2024-02-13T00:44:46.457 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:44:46.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 10 librados: watch_flush enter 2024-02-13T00:44:46.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 10 librados: watch_flush exit 2024-02-13T00:44:46.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 10 client.27102.objecter close_session for osd.0 2024-02-13T00:44:46.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa77c0fc210 msgr2=0x7fa77c0fc610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa77c0fc210 0x7fa77c0fc610 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c0f8150 osd=0 1 2024-02-13T00:44:46.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 10 client.27102.objecter close_session for osd.1 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6805/150599780 conn(0x7fa77c028590 msgr2=0x7fa77c11af90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6805/150599780 conn(0x7fa77c028590 0x7fa77c11af90 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c104300 osd=1 1 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 10 client.27102.objecter close_session for osd.2 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa77c059570 msgr2=0x7fa77c11e610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa77c059570 0x7fa77c11e610 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c059400 osd=2 1 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 10 client.27102.objecter close_session for osd.3 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6813/604971011 conn(0x7fa77c0277c0 msgr2=0x7fa77c121e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6813/604971011 conn(0x7fa77c0277c0 0x7fa77c121e60 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c027650 osd=3 1 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 10 client.27102.objecter close_session for osd.4 2024-02-13T00:44:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa77c11f440 msgr2=0x7fa77c11f8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa77c11f440 0x7fa77c11f8c0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c11c250 osd=4 1 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 10 client.27102.objecter close_session for osd.5 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6804/2212296748 conn(0x7fa77c1271a0 msgr2=0x7fa77c127600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6804/2212296748 conn(0x7fa77c1271a0 0x7fa77c127600 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c1254e0 osd=5 1 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 10 client.27102.objecter close_session for osd.6 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa77c125d60 msgr2=0x7fa77c1261c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa77c125d60 0x7fa77c1261c0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c125680 osd=6 1 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 10 client.27102.objecter close_session for osd.7 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa77c12c360 msgr2=0x7fa77c12c7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa77c12c360 0x7fa77c12c7a0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa77ad9d700 1 -- 172.21.15.162:0/1672471096 reap_dead start 2024-02-13T00:44:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa77ad9d700 1 -- 172.21.15.162:0/1672471096 reap_dead start 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 20 client.27102.objecter put_session s=0x7fa77c126800 osd=7 1 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 20 client.27102.objecter shutdown clearing up homeless session... 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 10 client.27102.objecter successfully canceled tick 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa77ad9d700 1 -- 172.21.15.162:0/1672471096 reap_dead start 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa77ad9d700 1 -- 172.21.15.162:0/1672471096 reap_dead start 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa764062c90 msgr2=0x7fa764065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa764062c90 0x7fa764065150 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 rx=0x7fa76c009760 tx=0x7fa76c00b040).stop 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa77c102270 msgr2=0x7fa77c0f76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa77c102270 0x7fa77c0f76a0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fa774009bb0 tx=0x7fa774004d10).stop 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 shutdown_connections 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa77c102270 0x7fa77c0f76a0 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 --2- 172.21.15.162:0/1672471096 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa764062c90 0x7fa764065150 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.454+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 >> 172.21.15.162:0/1672471096 conn(0x7fa77c06eb90 msgr2=0x7fa77c0708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.455+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 shutdown_connections 2024-02-13T00:44:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.455+0000 7fa780d3e700 1 -- 172.21.15.162:0/1672471096 wait complete. 2024-02-13T00:44:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:46.455+0000 7fa780d3e700 1 librados: shutdown 2024-02-13T00:44:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:44:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:45.404343+0000 mon.c (mon.2) 318 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:45.404703+0000 mon.a (mon.0) 2134 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:45.405170+0000 mon.c (mon.2) 319 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:45.405354+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:45.405691+0000 mon.c (mon.2) 320 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-83985-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:45.406443+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-83985-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:46.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:45.421889+0000 mon.b (mon.1) 496 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:46.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:45.531385+0000 mgr.y (mgr.14544) 304 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:46.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:46.285963+0000 mon.a (mon.0) 2137 : audit [DBG] from='client.? 172.21.15.162:0/1672471096' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:46.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:46.377363+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-83985-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:46.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:46.377448+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-02-13T00:44:46.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:46.379991+0000 mon.c (mon.2) 321 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-83985-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:46.380567+0000 mon.a (mon.0) 2140 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-83985-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.590 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:44:46.590 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:44:46.590 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:44:46.591 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:44:46.650 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:45.404343+0000 mon.c (mon.2) 318 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.650 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:45.404703+0000 mon.a (mon.0) 2134 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.650 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:45.405170+0000 mon.c (mon.2) 319 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:45.405354+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:45.405691+0000 mon.c (mon.2) 320 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-83985-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:46.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:45.406443+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-83985-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:46.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:45.421889+0000 mon.b (mon.1) 496 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:46.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:45.531385+0000 mgr.y (mgr.14544) 304 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:46.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:46.285963+0000 mon.a (mon.0) 2137 : audit [DBG] from='client.? 172.21.15.162:0/1672471096' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:46.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:46.377363+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-83985-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:46.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:46.377448+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-02-13T00:44:46.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:46.379991+0000 mon.c (mon.2) 321 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-83985-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:46.380567+0000 mon.a (mon.0) 2140 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-83985-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:45.404343+0000 mon.c (mon.2) 318 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:45.404703+0000 mon.a (mon.0) 2134 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:45.405170+0000 mon.c (mon.2) 319 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:45.405354+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:45.405691+0000 mon.c (mon.2) 320 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-83985-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:46.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:45.406443+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-83985-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:46.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:45.421889+0000 mon.b (mon.1) 496 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:46.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:45.531385+0000 mgr.y (mgr.14544) 304 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:46.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:46.285963+0000 mon.a (mon.0) 2137 : audit [DBG] from='client.? 172.21.15.162:0/1672471096' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:46.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:46.377363+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-83985-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:46.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:46.377448+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-02-13T00:44:46.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:46.379991+0000 mon.c (mon.2) 321 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-83985-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:46.380567+0000 mon.a (mon.0) 2140 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-83985-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:46.724 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:44:47.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-13T00:44:47.670 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:44:47.891 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:46.394417+0000 mgr.y (mgr.14544) 305 : cluster [DBG] pgmap v467: 263 pgs: 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped, 249 active+clean, 5 active+recovery_wait+degraded, 3 active+recovering, 1 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s; 39/1095 objects degraded (3.562%); 11/1095 objects misplaced (1.005%); 865 B/s, 53 objects/s recovering 2024-02-13T00:44:47.892 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:46.422709+0000 mon.b (mon.1) 497 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:47.892 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:47.373296+0000 mon.a (mon.0) 2141 : cluster [WRN] Health check update: Degraded data redundancy: 39/1095 objects degraded (3.562%), 5 pgs degraded (PG_DEGRADED) 2024-02-13T00:44:47.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:46.394417+0000 mgr.y (mgr.14544) 305 : cluster [DBG] pgmap v467: 263 pgs: 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped, 249 active+clean, 5 active+recovery_wait+degraded, 3 active+recovering, 1 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s; 39/1095 objects degraded (3.562%); 11/1095 objects misplaced (1.005%); 865 B/s, 53 objects/s recovering 2024-02-13T00:44:47.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:46.422709+0000 mon.b (mon.1) 497 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:47.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:47.373296+0000 mon.a (mon.0) 2141 : cluster [WRN] Health check update: Degraded data redundancy: 39/1095 objects degraded (3.562%), 5 pgs degraded (PG_DEGRADED) 2024-02-13T00:44:47.915 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:46.394417+0000 mgr.y (mgr.14544) 305 : cluster [DBG] pgmap v467: 263 pgs: 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped, 249 active+clean, 5 active+recovery_wait+degraded, 3 active+recovering, 1 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s; 39/1095 objects degraded (3.562%); 11/1095 objects misplaced (1.005%); 865 B/s, 53 objects/s recovering 2024-02-13T00:44:47.915 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:46.422709+0000 mon.b (mon.1) 497 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:47.915 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:47.373296+0000 mon.a (mon.0) 2141 : cluster [WRN] Health check update: Degraded data redundancy: 39/1095 objects degraded (3.562%), 5 pgs degraded (PG_DEGRADED) 2024-02-13T00:44:48.394 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:48.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:48.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:48.401 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:48.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:48.528 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:48.627 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:48.628 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:47.050555+0000 osd.6 (osd.6) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:44:48.628 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:47.050699+0000 osd.6 (osd.6) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:44:48.628 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:47.423550+0000 mon.b (mon.1) 498 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:48.628 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:47.621531+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-02-13T00:44:48.629 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:48.384438+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-83985-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-83985-57"}]': finished 2024-02-13T00:44:48.629 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:48.384560+0000 mon.a (mon.0) 2144 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-02-13T00:44:48.629 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:48.396275+0000 mon.c (mon.2) 322 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-13T00:44:48.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:48.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:48.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:48.764 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:48.884 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:48.401410+0000 mon.a (mon.0) 2145 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-13T00:44:48.884 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:48.424501+0000 mon.b (mon.1) 499 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:48.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:47.050555+0000 osd.6 (osd.6) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:44:48.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:47.050699+0000 osd.6 (osd.6) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:44:48.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:47.423550+0000 mon.b (mon.1) 498 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:48.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:47.621531+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-02-13T00:44:48.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:48.384438+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-83985-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-83985-57"}]': finished 2024-02-13T00:44:48.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:48.384560+0000 mon.a (mon.0) 2144 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-02-13T00:44:48.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:48.396275+0000 mon.c (mon.2) 322 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-13T00:44:48.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:48.401410+0000 mon.a (mon.0) 2145 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-13T00:44:48.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:48.424501+0000 mon.b (mon.1) 499 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:48.902 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:44:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:44:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:44:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:47.050555+0000 osd.6 (osd.6) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:44:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:47.050699+0000 osd.6 (osd.6) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:44:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:47.423550+0000 mon.b (mon.1) 498 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:47.621531+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-02-13T00:44:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:48.384438+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-83985-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-83985-57"}]': finished 2024-02-13T00:44:48.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:48.384560+0000 mon.a (mon.0) 2144 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-02-13T00:44:48.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:48.396275+0000 mon.c (mon.2) 322 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-13T00:44:48.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:48.401410+0000 mon.a (mon.0) 2145 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-13T00:44:48.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:48.424501+0000 mon.b (mon.1) 499 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:49.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.097+0000 7ff24808f700 1 -- 172.21.15.162:0/1092492355 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff2400f95e0 msgr2=0x7ff240101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:49.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.097+0000 7ff24808f700 1 --2- 172.21.15.162:0/1092492355 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff2400f95e0 0x7ff240101d80 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7ff2300047b0 tx=0x7ff230009cb0).stop 2024-02-13T00:44:49.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.097+0000 7ff24808f700 1 -- 172.21.15.162:0/1092492355 shutdown_connections 2024-02-13T00:44:49.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.097+0000 7ff24808f700 1 --2- 172.21.15.162:0/1092492355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff240102270 0x7ff240104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:49.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.097+0000 7ff24808f700 1 --2- 172.21.15.162:0/1092492355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2400f8d10 0x7ff2400f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:49.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.097+0000 7ff24808f700 1 --2- 172.21.15.162:0/1092492355 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff2400f95e0 0x7ff240101d80 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:49.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.097+0000 7ff24808f700 1 -- 172.21.15.162:0/1092492355 >> 172.21.15.162:0/1092492355 conn(0x7ff24006eb90 msgr2=0x7ff2400f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:49.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.097+0000 7ff24808f700 1 -- 172.21.15.162:0/1092492355 shutdown_connections 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.097+0000 7ff24808f700 1 -- 172.21.15.162:0/1092492355 wait complete. 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24808f700 1 librados: starting msgr at 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24808f700 1 librados: starting objecter 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24808f700 1 Processor -- start 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24808f700 1 -- start start 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24808f700 1 librados: setting wanted keys 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24808f700 1 librados: calling monclient init 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24808f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2400f8d10 0x7ff24010e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24808f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff2400f95e0 0x7ff24010e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24808f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff240102270 0x7ff240108430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24808f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff240108920 con 0x7ff240102270 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24808f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff240108a90 con 0x7ff2400f8d10 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24808f700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff240108c00 con 0x7ff2400f95e0 2024-02-13T00:44:49.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24562a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff2400f95e0 0x7ff24010e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24662c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff240102270 0x7ff240108430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24562a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff2400f95e0 0x7ff24010e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:60754/0 (socket says 172.21.15.162:60754) 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24562a700 1 -- 172.21.15.162:0/2876194428 learned_addr learned my addr 172.21.15.162:0/2876194428 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff245e2b700 1 --2- 172.21.15.162:0/2876194428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2400f8d10 0x7ff24010e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.098+0000 7ff24562a700 1 -- 172.21.15.162:0/2876194428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2400f8d10 msgr2=0x7ff24010e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff24562a700 1 --2- 172.21.15.162:0/2876194428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2400f8d10 0x7ff24010e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff24562a700 1 -- 172.21.15.162:0/2876194428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff240102270 msgr2=0x7ff240108430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff24562a700 1 --2- 172.21.15.162:0/2876194428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff240102270 0x7ff240108430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff24562a700 1 -- 172.21.15.162:0/2876194428 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c005040 con 0x7ff2400f95e0 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff24662c700 1 --2- 172.21.15.162:0/2876194428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff240102270 0x7ff240108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff24562a700 1 --2- 172.21.15.162:0/2876194428 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff2400f95e0 0x7ff24010e740 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 rx=0x7ff230009b10 tx=0x7ff230009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff236ffd700 10 client.?.objecter ms_handle_connect 0x7ff2400f95e0 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff236ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:49.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff236ffd700 1 -- 172.21.15.162:0/2876194428 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff230018070 con 0x7ff2400f95e0 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff236ffd700 1 -- 172.21.15.162:0/2876194428 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff230013da0 con 0x7ff2400f95e0 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff24808f700 1 -- 172.21.15.162:0/2876194428 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff230005040 con 0x7ff2400f95e0 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff236ffd700 1 -- 172.21.15.162:0/2876194428 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23001dbc0 con 0x7ff2400f95e0 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff24808f700 10 client.37220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff24808f700 1 -- 172.21.15.162:0/2876194428 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2401090a0 con 0x7ff2400f95e0 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.099+0000 7ff24808f700 1 librados: init done 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.100+0000 7ff24808f700 1 -- 172.21.15.162:0/2876194428 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2400475e0 con 0x7ff2400f95e0 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.100+0000 7ff236ffd700 1 -- 172.21.15.162:0/2876194428 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff23001d420 con 0x7ff2400f95e0 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.101+0000 7ff236ffd700 1 --2- 172.21.15.162:0/2876194428 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff22c062c90 0x7ff22c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.101+0000 7ff245e2b700 1 --2- 172.21.15.162:0/2876194428 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff22c062c90 0x7ff22c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.101+0000 7ff236ffd700 1 -- 172.21.15.162:0/2876194428 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(333..333 src has 52..333) v4 ==== 9204+0+0 (secure 0 0 0) 0x7ff23007eb10 con 0x7ff2400f95e0 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.101+0000 7ff236ffd700 10 client.37220.objecter ms_dispatch 0x7ff240001550 osd_map(333..333 src has 52..333) v4 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.101+0000 7ff236ffd700 3 client.37220.objecter handle_osd_map got epochs [333,333] > 0 2024-02-13T00:44:49.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.101+0000 7ff236ffd700 3 client.37220.objecter handle_osd_map decoding full epoch 333 2024-02-13T00:44:49.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.101+0000 7ff236ffd700 20 client.37220.objecter dump_active .. 0 homeless 2024-02-13T00:44:49.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.101+0000 7ff236ffd700 10 client.37220.objecter ms_handle_connect 0x7ff22c062c90 2024-02-13T00:44:49.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.101+0000 7ff245e2b700 1 --2- 172.21.15.162:0/2876194428 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff22c062c90 0x7ff22c065150 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 rx=0x7ff23c0047e0 tx=0x7ff23c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:49.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.105+0000 7ff236ffd700 1 -- 172.21.15.162:0/2876194428 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff230051b60 con 0x7ff2400f95e0 2024-02-13T00:44:49.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.285+0000 7ff24808f700 1 -- 172.21.15.162:0/2876194428 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd out", "ids": ["2"]} v 0) v1 -- 0x7ff2401099a0 con 0x7ff2400f95e0 2024-02-13T00:44:49.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.395+0000 7ff236ffd700 1 -- 172.21.15.162:0/2876194428 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["2"]}]=0 marked out osd.2. v334) v1 ==== 87+0+0 (secure 0 0 0) 0x7ff230051750 con 0x7ff2400f95e0 2024-02-13T00:44:49.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.398+0000 7ff24808f700 10 librados: watch_flush enter 2024-02-13T00:44:49.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.398+0000 7ff24808f700 10 librados: watch_flush exit 2024-02-13T00:44:49.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.398+0000 7ff24808f700 20 client.37220.objecter shutdown clearing up homeless session... 2024-02-13T00:44:49.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.398+0000 7ff24808f700 10 client.37220.objecter successfully canceled tick 2024-02-13T00:44:49.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.398+0000 7ff24808f700 1 -- 172.21.15.162:0/2876194428 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff22c062c90 msgr2=0x7ff22c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:49.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.398+0000 7ff24808f700 1 --2- 172.21.15.162:0/2876194428 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff22c062c90 0x7ff22c065150 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 rx=0x7ff23c0047e0 tx=0x7ff23c00b040).stop 2024-02-13T00:44:49.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.398+0000 7ff24808f700 1 -- 172.21.15.162:0/2876194428 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff2400f95e0 msgr2=0x7ff24010e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:49.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.398+0000 7ff24808f700 1 --2- 172.21.15.162:0/2876194428 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff2400f95e0 0x7ff24010e740 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 rx=0x7ff230009b10 tx=0x7ff230009cb0).stop 2024-02-13T00:44:49.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.409+0000 7ff24808f700 1 -- 172.21.15.162:0/2876194428 shutdown_connections 2024-02-13T00:44:49.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.409+0000 7ff24808f700 1 --2- 172.21.15.162:0/2876194428 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff22c062c90 0x7ff22c065150 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:49.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.409+0000 7ff24808f700 1 --2- 172.21.15.162:0/2876194428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff240102270 0x7ff240108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:49.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.409+0000 7ff24808f700 1 --2- 172.21.15.162:0/2876194428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2400f8d10 0x7ff24010e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:49.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.409+0000 7ff24808f700 1 --2- 172.21.15.162:0/2876194428 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff2400f95e0 0x7ff24010e740 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:49.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.409+0000 7ff24808f700 1 -- 172.21.15.162:0/2876194428 >> 172.21.15.162:0/2876194428 conn(0x7ff24006eb90 msgr2=0x7ff240102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:49.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.413+0000 7ff24808f700 1 -- 172.21.15.162:0/2876194428 shutdown_connections 2024-02-13T00:44:49.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.414+0000 7ff24808f700 1 -- 172.21.15.162:0/2876194428 wait complete. 2024-02-13T00:44:49.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:49.414+0000 7ff24808f700 1 librados: shutdown 2024-02-13T00:44:49.418 INFO:teuthology.orchestra.run.smithi162.stderr:marked out osd.2. 2024-02-13T00:44:49.460 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:44:49.460 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:44:49.460 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-02-13T00:44:49.460 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-02-13T00:44:49.460 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:44:49.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:44:49.286+0000 7f5f6486f700 -1 mon.a@0(leader).osd e333 definitely_dead 0 2024-02-13T00:44:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:48.395002+0000 mgr.y (mgr.14544) 306 : cluster [DBG] pgmap v470: 270 pgs: 8 unknown, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped, 248 active+clean, 5 active+recovery_wait+degraded, 3 active+recovering, 1 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 39/1095 objects degraded (3.562%); 11/1095 objects misplaced (1.005%); 865 B/s, 52 objects/s recovering 2024-02-13T00:44:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:48.972113+0000 osd.5 (osd.5) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:44:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:48.972407+0000 osd.5 (osd.5) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:44:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:49.286345+0000 mon.c (mon.2) 323 : audit [INF] from='client.? 172.21.15.162:0/2876194428' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-13T00:44:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:49.286703+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-13T00:44:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:49.286799+0000 mon.a (mon.0) 2147 : cluster [INF] Client client.admin marked osd.2 out, while it was still marked up 2024-02-13T00:44:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:49.334241+0000 mon.a (mon.0) 2148 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:49.394509+0000 mon.a (mon.0) 2149 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-13T00:44:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:49.394599+0000 mon.a (mon.0) 2150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-13T00:44:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:49.394642+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e334: 8 total, 8 up, 6 in 2024-02-13T00:44:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:49.431419+0000 mon.b (mon.1) 500 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:49.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:44:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:48.395002+0000 mgr.y (mgr.14544) 306 : cluster [DBG] pgmap v470: 270 pgs: 8 unknown, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped, 248 active+clean, 5 active+recovery_wait+degraded, 3 active+recovering, 1 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 39/1095 objects degraded (3.562%); 11/1095 objects misplaced (1.005%); 865 B/s, 52 objects/s recovering 2024-02-13T00:44:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:48.972113+0000 osd.5 (osd.5) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:44:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:48.972407+0000 osd.5 (osd.5) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:44:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:49.286345+0000 mon.c (mon.2) 323 : audit [INF] from='client.? 172.21.15.162:0/2876194428' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-13T00:44:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:49.286703+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-13T00:44:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:49.286799+0000 mon.a (mon.0) 2147 : cluster [INF] Client client.admin marked osd.2 out, while it was still marked up 2024-02-13T00:44:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:49.334241+0000 mon.a (mon.0) 2148 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:49.394509+0000 mon.a (mon.0) 2149 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-13T00:44:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:49.394599+0000 mon.a (mon.0) 2150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-13T00:44:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:49.394642+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e334: 8 total, 8 up, 6 in 2024-02-13T00:44:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:49.431419+0000 mon.b (mon.1) 500 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:48.395002+0000 mgr.y (mgr.14544) 306 : cluster [DBG] pgmap v470: 270 pgs: 8 unknown, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped, 248 active+clean, 5 active+recovery_wait+degraded, 3 active+recovering, 1 active+recovery_wait; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 39/1095 objects degraded (3.562%); 11/1095 objects misplaced (1.005%); 865 B/s, 52 objects/s recovering 2024-02-13T00:44:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:48.972113+0000 osd.5 (osd.5) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:44:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:48.972407+0000 osd.5 (osd.5) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:44:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:49.286345+0000 mon.c (mon.2) 323 : audit [INF] from='client.? 172.21.15.162:0/2876194428' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-13T00:44:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:49.286703+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-13T00:44:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:49.286799+0000 mon.a (mon.0) 2147 : cluster [INF] Client client.admin marked osd.2 out, while it was still marked up 2024-02-13T00:44:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:49.334241+0000 mon.a (mon.0) 2148 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:49.394509+0000 mon.a (mon.0) 2149 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-13T00:44:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:49.394599+0000 mon.a (mon.0) 2150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-13T00:44:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:49.394642+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e334: 8 total, 8 up, 6 in 2024-02-13T00:44:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:49.431419+0000 mon.b (mon.1) 500 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:50.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-13T00:44:50.555 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:44:51.065 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:50.011369+0000 mon.a (mon.0) 2152 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:51.065 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:50.378227+0000 mon.a (mon.0) 2153 : cluster [DBG] osdmap e335: 8 total, 8 up, 6 in 2024-02-13T00:44:51.065 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:50.383681+0000 mon.c (mon.2) 324 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:51.065 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:50.393407+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:51.065 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:50.433604+0000 mon.b (mon.1) 501 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:51.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:50.011369+0000 mon.a (mon.0) 2152 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:51.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:50.378227+0000 mon.a (mon.0) 2153 : cluster [DBG] osdmap e335: 8 total, 8 up, 6 in 2024-02-13T00:44:51.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:50.383681+0000 mon.c (mon.2) 324 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:51.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:50.393407+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:51.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:50.433604+0000 mon.b (mon.1) 501 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:51.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:50.011369+0000 mon.a (mon.0) 2152 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:51.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:50.378227+0000 mon.a (mon.0) 2153 : cluster [DBG] osdmap e335: 8 total, 8 up, 6 in 2024-02-13T00:44:51.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:50.383681+0000 mon.c (mon.2) 324 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:51.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:50.393407+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:51.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:50.433604+0000 mon.b (mon.1) 501 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:51.416 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:51.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:51.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:51.549 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:51.757 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:51.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:51.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:51.892 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:52.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 -- 172.21.15.162:0/1478536226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e80f8d10 msgr2=0x7f58e80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:52.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 --2- 172.21.15.162:0/1478536226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e80f8d10 0x7f58e80f90f0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7f58e4004870 tx=0x7f58e4009cb0).stop 2024-02-13T00:44:52.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 -- 172.21.15.162:0/1478536226 shutdown_connections 2024-02-13T00:44:52.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 --2- 172.21.15.162:0/1478536226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58e8102270 0x7f58e8104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:52.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 --2- 172.21.15.162:0/1478536226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e80f8d10 0x7f58e80f90f0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 --2- 172.21.15.162:0/1478536226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58e80f95e0 0x7f58e8101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 -- 172.21.15.162:0/1478536226 >> 172.21.15.162:0/1478536226 conn(0x7f58e806eb90 msgr2=0x7f58e80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 -- 172.21.15.162:0/1478536226 shutdown_connections 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 -- 172.21.15.162:0/1478536226 wait complete. 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 librados: starting msgr at 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 librados: starting objecter 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 Processor -- start 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 -- start start 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 librados: setting wanted keys 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.020+0000 7f58efe20700 1 librados: calling monclient init 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58efe20700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58e80f8d10 0x7f58e810e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58efe20700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e80f95e0 0x7f58e810e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:52.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58efe20700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58e8102270 0x7f58e81082e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58efe20700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e81087d0 con 0x7f58e8102270 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58efe20700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e8108940 con 0x7f58e80f8d10 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58efe20700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e8108b00 con 0x7f58e80f95e0 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58edbbc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58e80f8d10 0x7f58e810e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58edbbc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58e80f8d10 0x7f58e810e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:37638/0 (socket says 172.21.15.162:37638) 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58edbbc700 1 -- 172.21.15.162:0/4059015823 learned_addr learned my addr 172.21.15.162:0/4059015823 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58ee3bd700 1 --2- 172.21.15.162:0/4059015823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58e8102270 0x7f58e81082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58edbbc700 1 -- 172.21.15.162:0/4059015823 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e80f95e0 msgr2=0x7f58e810e630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58edbbc700 1 --2- 172.21.15.162:0/4059015823 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e80f95e0 0x7f58e810e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58edbbc700 1 -- 172.21.15.162:0/4059015823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58e8102270 msgr2=0x7f58e81082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58edbbc700 1 --2- 172.21.15.162:0/4059015823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58e8102270 0x7f58e81082e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.021+0000 7f58edbbc700 1 -- 172.21.15.162:0/4059015823 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e4005040 con 0x7f58e80f8d10 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.022+0000 7f58edbbc700 1 --2- 172.21.15.162:0/4059015823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58e80f8d10 0x7f58e810e140 secure :-1 s=READY pgs=4853 cs=0 l=1 rev1=1 rx=0x7f58e40043a0 tx=0x7f58e4009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:52.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.022+0000 7f58deffd700 10 client.?.objecter ms_handle_connect 0x7f58e80f8d10 2024-02-13T00:44:52.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.022+0000 7f58deffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:52.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.022+0000 7f58deffd700 1 -- 172.21.15.162:0/4059015823 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e400e030 con 0x7f58e80f8d10 2024-02-13T00:44:52.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.022+0000 7f58efe20700 1 -- 172.21.15.162:0/4059015823 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e8108d80 con 0x7f58e80f8d10 2024-02-13T00:44:52.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.022+0000 7f58deffd700 1 -- 172.21.15.162:0/4059015823 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58e4013da0 con 0x7f58e80f8d10 2024-02-13T00:44:52.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.022+0000 7f58deffd700 1 -- 172.21.15.162:0/4059015823 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e4018070 con 0x7f58e80f8d10 2024-02-13T00:44:52.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.022+0000 7f58efe20700 10 client.44287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:52.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.022+0000 7f58efe20700 1 -- 172.21.15.162:0/4059015823 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e80f6b00 con 0x7f58e80f8d10 2024-02-13T00:44:52.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.022+0000 7f58efe20700 1 librados: init done 2024-02-13T00:44:52.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.023+0000 7f58efe20700 1 -- 172.21.15.162:0/4059015823 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e80475e0 con 0x7f58e80f8d10 2024-02-13T00:44:52.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.023+0000 7f58deffd700 1 -- 172.21.15.162:0/4059015823 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f58e40090d0 con 0x7f58e80f8d10 2024-02-13T00:44:52.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.024+0000 7f58deffd700 1 --2- 172.21.15.162:0/4059015823 >> v2:172.21.15.162:6800/2866534053 conn(0x7f58d4062c90 0x7f58d4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:52.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.024+0000 7f58deffd700 1 -- 172.21.15.162:0/4059015823 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(336..336 src has 52..336) v4 ==== 8544+0+0 (secure 0 0 0) 0x7f58e407e730 con 0x7f58e80f8d10 2024-02-13T00:44:52.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.024+0000 7f58deffd700 10 client.44287.objecter ms_dispatch 0x7f58e8001550 osd_map(336..336 src has 52..336) v4 2024-02-13T00:44:52.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.024+0000 7f58deffd700 3 client.44287.objecter handle_osd_map got epochs [336,336] > 0 2024-02-13T00:44:52.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.024+0000 7f58deffd700 3 client.44287.objecter handle_osd_map decoding full epoch 336 2024-02-13T00:44:52.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.024+0000 7f58deffd700 20 client.44287.objecter dump_active .. 0 homeless 2024-02-13T00:44:52.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.024+0000 7f58ed3bb700 1 --2- 172.21.15.162:0/4059015823 >> v2:172.21.15.162:6800/2866534053 conn(0x7f58d4062c90 0x7f58d4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:52.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.025+0000 7f58deffd700 10 client.44287.objecter ms_handle_connect 0x7f58d4062c90 2024-02-13T00:44:52.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.025+0000 7f58ed3bb700 1 --2- 172.21.15.162:0/4059015823 >> v2:172.21.15.162:6800/2866534053 conn(0x7f58d4062c90 0x7f58d4065150 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 rx=0x7f58e8109a80 tx=0x7f58d8007b40).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:52.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.028+0000 7f58deffd700 1 -- 172.21.15.162:0/4059015823 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f58e40519c0 con 0x7f58e80f8d10 2024-02-13T00:44:52.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:50.395829+0000 mgr.y (mgr.14544) 307 : cluster [DBG] pgmap v473: 262 pgs: 106 peering, 156 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 72 B/s, 1 objects/s recovering 2024-02-13T00:44:52.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:51.386362+0000 mon.a (mon.0) 2155 : cluster [WRN] Health check failed: Reduced data availability: 16 pgs inactive, 23 pgs peering (PG_AVAILABILITY) 2024-02-13T00:44:52.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:51.386401+0000 mon.a (mon.0) 2156 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/1095 objects degraded (3.562%), 5 pgs degraded) 2024-02-13T00:44:52.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:51.396028+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]': finished 2024-02-13T00:44:52.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:51.396120+0000 mon.a (mon.0) 2158 : cluster [DBG] osdmap e336: 8 total, 8 up, 6 in 2024-02-13T00:44:52.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:51.400153+0000 osd.0 (osd.0) 115 : cluster [DBG] 18.8 starting backfill to osd.1 from (0'0,0'0] MAX to 107'2 2024-02-13T00:44:52.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:51.404629+0000 osd.1 (osd.1) 136 : cluster [DBG] 18.d starting backfill to osd.0 from (0'0,0'0] MAX to 309'9 2024-02-13T00:44:52.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:51.414994+0000 mon.c (mon.2) 325 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:52.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:51.415442+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:52.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:51.437088+0000 mon.b (mon.1) 502 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:52.097 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:52.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:52.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:52.233 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:52.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.251+0000 7f58efe20700 1 -- 172.21.15.162:0/4059015823 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f58e8109850 con 0x7f58e80f8d10 2024-02-13T00:44:52.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:50.395829+0000 mgr.y (mgr.14544) 307 : cluster [DBG] pgmap v473: 262 pgs: 106 peering, 156 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 72 B/s, 1 objects/s recovering 2024-02-13T00:44:52.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:51.386362+0000 mon.a (mon.0) 2155 : cluster [WRN] Health check failed: Reduced data availability: 16 pgs inactive, 23 pgs peering (PG_AVAILABILITY) 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:51.386401+0000 mon.a (mon.0) 2156 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/1095 objects degraded (3.562%), 5 pgs degraded) 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:51.396028+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]': finished 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:51.396120+0000 mon.a (mon.0) 2158 : cluster [DBG] osdmap e336: 8 total, 8 up, 6 in 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:51.400153+0000 osd.0 (osd.0) 115 : cluster [DBG] 18.8 starting backfill to osd.1 from (0'0,0'0] MAX to 107'2 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:51.404629+0000 osd.1 (osd.1) 136 : cluster [DBG] 18.d starting backfill to osd.0 from (0'0,0'0] MAX to 309'9 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:51.414994+0000 mon.c (mon.2) 325 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:51.415442+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:51.437088+0000 mon.b (mon.1) 502 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:50.395829+0000 mgr.y (mgr.14544) 307 : cluster [DBG] pgmap v473: 262 pgs: 106 peering, 156 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 72 B/s, 1 objects/s recovering 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:51.386362+0000 mon.a (mon.0) 2155 : cluster [WRN] Health check failed: Reduced data availability: 16 pgs inactive, 23 pgs peering (PG_AVAILABILITY) 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:51.386401+0000 mon.a (mon.0) 2156 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/1095 objects degraded (3.562%), 5 pgs degraded) 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:51.396028+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-83985-57"}]': finished 2024-02-13T00:44:52.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:51.396120+0000 mon.a (mon.0) 2158 : cluster [DBG] osdmap e336: 8 total, 8 up, 6 in 2024-02-13T00:44:52.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:51.400153+0000 osd.0 (osd.0) 115 : cluster [DBG] 18.8 starting backfill to osd.1 from (0'0,0'0] MAX to 107'2 2024-02-13T00:44:52.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:51.404629+0000 osd.1 (osd.1) 136 : cluster [DBG] 18.d starting backfill to osd.0 from (0'0,0'0] MAX to 309'9 2024-02-13T00:44:52.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:51.414994+0000 mon.c (mon.2) 325 : audit [INF] from='client.? 172.21.15.162:0/3180918714' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:52.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:51.415442+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]: dispatch 2024-02-13T00:44:52.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:51.437088+0000 mon.b (mon.1) 502 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:52.389 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:44:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:44:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:44:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14862.0:3 12.f 12:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:00.557921+0000", 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 231.82628584700001, 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0933016659999999, 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14862", 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/4283548805", 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 3 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557921+0000", 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557921+0000", 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557919+0000", 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9099999999999999e-06 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557926+0000", 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4139e-05 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557940+0000", 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9110000000000004e-06 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557947+0000", 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4711e-05 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557982+0000", 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8269999999999997e-06 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557987+0000", 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.060214974 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.618202+0000", 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016521299999999999 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.618367+0000", 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015337700000000001 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,5", 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.618520+0000", 2024-02-13T00:44:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083500600000000001 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.619355+0000", 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011477988999999999 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.630833+0000", 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020342135000000001 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.651176+0000", 2024-02-13T00:44:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5749e-05 2024-02-13T00:44:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.651201+0000", 2024-02-13T00:44:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1682e-05 2024-02-13T00:44:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.651223+0000", 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0933016659999999 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24814.0:4 27.1f 27:ff7b1f36:::obj1:head [stat] snapc 0=[] ondisk+read+known_if_redirected e107)", 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.629056+0000", 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 230.75515102200001, 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.033158676999999998, 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24814", 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/4028053558", 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 4 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629056+0000", 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629056+0000", 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629054+0000", 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.032e-06 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629060+0000", 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4870000000000002e-06 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629069+0000", 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.793e-06 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629076+0000", 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012851900000000001 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629204+0000", 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5579999999999997e-06 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629211+0000", 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.032924434000000002 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.662135+0000", 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9778999999999996e-05 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.662215+0000", 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.033158676999999998 2024-02-13T00:44:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14862.0:4 12.16 12:681f27cd:::2.obj:head [write 0~105] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.652086+0000", 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 230.732120799, 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014126975, 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14862", 2024-02-13T00:44:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/4283548805", 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 4 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.652086+0000", 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.652086+0000", 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.652085+0000", 2024-02-13T00:44:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1689999999999992e-06 2024-02-13T00:44:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.652093+0000", 2024-02-13T00:44:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5660000000000007e-06 2024-02-13T00:44:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.652102+0000", 2024-02-13T00:44:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6590000000000001e-06 2024-02-13T00:44:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.652108+0000", 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012632240000000001 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.653372+0000", 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5828999999999997e-05 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.653428+0000", 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2966000000000005e-05 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,4", 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.653511+0000", 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00084050800000000001 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.654351+0000", 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025650040000000001 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.656916+0000", 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0092078209999999997 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.666124+0000", 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0042e-05 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.666174+0000", 2024-02-13T00:44:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9520000000000001e-05 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.666213+0000", 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014126975 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24911.0:20 16.1 e108/104 16:83282cb0:::1.obj2:head v 108'14, mlcod=107'12)", 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:02.519083+0000", 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 229.86512380799999, 2024-02-13T00:44:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019362523999999999, 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.519083+0000", 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.519083+0000", 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.519083+0000", 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.339e-06 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.519088+0000", 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4150000000000004e-06 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.519095+0000", 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8859999999999998e-06 2024-02-13T00:44:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.519102+0000", 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015855470000000001 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.520687+0000", 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2031000000000001e-05 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.520699+0000", 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017723135000000001 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.538422+0000", 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3485e-05 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.538446+0000", 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019362523999999999 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24926.0:4 16.12 16:4a6d8757:::3.obj:head [write 0~115] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-13T00:44:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:02.829836+0000", 2024-02-13T00:44:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 229.55437120600001, 2024-02-13T00:44:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020494032999999998, 2024-02-13T00:44:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24926", 2024-02-13T00:44:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2404423687", 2024-02-13T00:44:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 4 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829836+0000", 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829836+0000", 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829835+0000", 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.639e-06 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829839+0000", 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6390000000000003e-06 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829846+0000", 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3669999999999999e-06 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829850+0000", 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2035e-05 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829892+0000", 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0727000000000003e-05 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.829953+0000", 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0786999999999997e-05 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.830044+0000", 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031020700000000001 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.830354+0000", 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013243400000000001 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.830486+0000", 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019756763 2024-02-13T00:44:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850243+0000", 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5442999999999998e-05 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850289+0000", 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1400999999999999e-05 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850330+0000", 2024-02-13T00:44:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020494032999999998 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24997.0:2 36.1a 36:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.022858+0000", 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 228.36134952399999, 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0671349779999999, 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24997", 2024-02-13T00:44:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/1225415695", 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022858+0000", 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022858+0000", 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022855+0000", 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9489999999999996e-06 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022863+0000", 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4700000000000008e-06 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022873+0000", 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2189999999999998e-06 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022880+0000", 2024-02-13T00:44:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9550999999999999e-05 2024-02-13T00:44:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022950+0000", 2024-02-13T00:44:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0539999999999997e-06 2024-02-13T00:44:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:44:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022957+0000", 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0397542660000001 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.062711+0000", 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109751 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.062821+0000", 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015550399999999999 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.062976+0000", 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001836599 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.064813+0000", 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004613102 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.069426+0000", 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020460633999999998 2024-02-13T00:44:52.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.089887+0000", 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5318e-05 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.089952+0000", 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0765e-05 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.089993+0000", 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0671349779999999 2024-02-13T00:44:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24772.0:5 25.d 25:b3257de0:test-rados-api-smithi162-84073-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.030406+0000", 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 228.353801653, 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017494136, 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24772", 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3029464538", 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 5 2024-02-13T00:44:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030406+0000", 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030406+0000", 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030404+0000", 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0809999999999996e-06 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030411+0000", 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3829999999999998e-06 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030420+0000", 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8079999999999999e-06 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030427+0000", 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025939399999999998 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030687+0000", 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119636 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030806+0000", 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017253700000000001 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.030979+0000", 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022362010000000002 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.033215+0000", 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0085149660000000005 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.041730+0000", 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061089999999999998 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.047839+0000", 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3524e-05 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.047873+0000", 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7160000000000001e-05 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.047900+0000", 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017494136 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2070 20.b e109/105 20:d3a22371:test-rados-api-smithi162-84049-17::big2:head v 109'1, mlcod=0'0)", 2024-02-13T00:44:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.754351+0000", 2024-02-13T00:44:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 227.629856115, 2024-02-13T00:44:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088475559999999995, 2024-02-13T00:44:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754351+0000", 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754351+0000", 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754350+0000", 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.045778653000000002 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800129+0000", 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1767999999999999e-05 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800150+0000", 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6830000000000008e-06 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800158+0000", 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7021999999999998e-05 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800205+0000", 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.98e-05 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800225+0000", 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.042577241000000002 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842802+0000", 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4448e-05 2024-02-13T00:44:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842827+0000", 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088475559999999995 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24773.0:48 11.a e109/104 11:55b4a790:test-rados-api-smithi162-84105-9::foo:head v 109'3, mlcod=107'1)", 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.827958+0000", 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 227.556249022, 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014482019, 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.827958+0000", 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.827958+0000", 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.827957+0000", 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4120000000000001e-06 2024-02-13T00:44:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.827966+0000", 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.207e-05 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.827978+0000", 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0089999999999998e-06 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.827983+0000", 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1709999999999999e-05 2024-02-13T00:44:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.828024+0000", 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8414999999999999e-05 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.828043+0000", 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014314863000000001 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842358+0000", 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2528000000000007e-05 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842440+0000", 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014482019 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2073 20.b e109/105 20:d3a22371:test-rados-api-smithi162-84049-17::big2:head v 109'4, mlcod=109'2)", 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.849365+0000", 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 227.534842204, 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012837360000000001, 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.849365+0000", 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.849365+0000", 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.849363+0000", 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048158109999999997 2024-02-13T00:44:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854179+0000", 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4929999999999999e-05 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854194+0000", 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1550000000000004e-06 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854200+0000", 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3585999999999999e-05 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854244+0000", 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3010000000000002e-05 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854267+0000", 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0079004929999999998 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.862167+0000", 2024-02-13T00:44:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5088e-05 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.862202+0000", 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012837360000000001 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:1163 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94514175197056 gen 3] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:39.297056+0000", 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 13.087150958000001, 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.232174933, 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1163 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297056+0000", 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297056+0000", 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999971 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297053+0000", 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3816e-05 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297067+0000", 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8560000000000001e-05 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297096+0000", 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9269000000000001e-05 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297115+0000", 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3229e-05 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297188+0000", 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9490000000000001e-06 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:44:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297194+0000", 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.225866915 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.523061+0000", 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117611 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for missing object", 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.523179+0000", 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044211959999999996 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.527600+0000", 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3129999999999998e-05 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.527693+0000", 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00068676900000000005 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.528380+0000", 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2867999999999995e-05 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.528443+0000", 2024-02-13T00:44:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00078838699999999999 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.529231+0000", 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.232174933 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:1165 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94514175188992 gen 3] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:39.297151+0000", 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 13.087055926, 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23092579999999999, 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1165 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297151+0000", 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297151+0000", 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297151+0000", 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7009999999999998e-06 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297154+0000", 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.985e-06 2024-02-13T00:44:52.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297161+0000", 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8110000000000002e-06 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297166+0000", 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2799000000000002e-05 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297199+0000", 2024-02-13T00:44:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9120000000000003e-06 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297205+0000", 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21097220999999999 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.508177+0000", 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000102681 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for missing object", 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.508280+0000", 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0052417039999999998 2024-02-13T00:44:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.513521+0000", 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3107000000000002e-05 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.513604+0000", 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034221709999999999 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.517027+0000", 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4387000000000005e-05 2024-02-13T00:44:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.517101+0000", 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010976069999999999 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.528077+0000", 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23092579999999999 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25162.0:4731 191.6 191:602f83fe:::foo:head [write 0~1] snapc 0=[] ondisk+write+known_if_redirected e333)", 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:49.420386+0000", 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.9638213090000001, 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015977860999999999, 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25162", 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 4731 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.420386+0000", 2024-02-13T00:44:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.420386+0000", 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.420384+0000", 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.071e-06 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.420390+0000", 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7900000000000005e-06 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.420399+0000", 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5669999999999997e-06 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.420407+0000", 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004263729 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.424670+0000", 2024-02-13T00:44:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2669000000000003e-05 2024-02-13T00:44:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.424763+0000", 2024-02-13T00:44:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000148224 2024-02-13T00:44:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-02-13T00:44:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.424911+0000", 2024-02-13T00:44:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00041209299999999999 2024-02-13T00:44:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.425323+0000", 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058154119999999998 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.431139+0000", 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051326879999999998 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.436272+0000", 2024-02-13T00:44:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3442000000000001e-05 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.436325+0000", 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8829999999999999e-05 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:49.436364+0000", 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015977860999999999 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25162.0:4733 191.6 191:602f83fe:::foo:head [write 0~1] snapc 0=[] ondisk+write+known_if_redirected e334)", 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:50.441367+0000", 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.9428404449999999, 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.053522410999999999, 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25162", 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 4733 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.441367+0000", 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.441367+0000", 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.441365+0000", 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0449999999999997e-06 2024-02-13T00:44:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.441371+0000", 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8289999999999997e-06 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.441380+0000", 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5399999999999998e-06 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.441388+0000", 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00138733 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.442775+0000", 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0977000000000002e-05 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.442836+0000", 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010878 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.442945+0000", 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.022457622 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.465402+0000", 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002867521 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.468270+0000", 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.026506655 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.494777+0000", 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2019000000000004e-05 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.494839+0000", 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0553999999999997e-05 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.494889+0000", 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.053522410999999999 2024-02-13T00:44:52.468 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.16 335/334 [PushOp(4:6dc3b799:reshard::reshard.0000000006:head, version: 64'4, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6dc3b799:reshard::reshard.0000000006:head@64'4, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:51.120970+0000", 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.2632370239999999, 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063347392000000002, 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.120970+0000", 2024-02-13T00:44:52.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.120970+0000", 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.120969+0000", 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3338e-05 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.120983+0000", 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4115e-05 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.120997+0000", 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7559999999999999e-06 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.121005+0000", 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063207902999999996 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.184213+0000", 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.049e-06 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.184219+0000", 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8979000000000002e-05 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.184318+0000", 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063347392000000002 2024-02-13T00:44:52.472 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.16 335/334 [PushOp(4:6849fd06:gc::gc.17:head, version: 64'5, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:6849fd06:gc::gc.17:head@64'5, size: 110, copy_subset: [0~110], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:110, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:51.206879+0000", 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.177327813, 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063759863, 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.206879+0000", 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.206879+0000", 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.206879+0000", 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3208e-05 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.206892+0000", 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3016999999999999e-05 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.206905+0000", 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.596e-06 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.206912+0000", 2024-02-13T00:44:52.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063607217999999993 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.270519+0000", 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8039999999999999e-06 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.270523+0000", 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116678 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.270639+0000", 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063759863 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.16 335/334 [PushOp(4:6a988a9b:reshard::reshard.0000000007:head, version: 64'7, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6a988a9b:reshard::reshard.0000000007:head@64'7, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:51.292782+0000", 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.0914253839999999, 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068626035000000002, 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.477 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.292782+0000", 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.292782+0000", 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.292781+0000", 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0513000000000001e-05 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.292792+0000", 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3203999999999999e-05 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.292805+0000", 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8950000000000001e-06 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.292812+0000", 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068487231999999995 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.361299+0000", 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3909999999999997e-06 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.361304+0000", 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103461 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.361408+0000", 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068626035000000002 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.16 335/334 [PushOp(4:6f6aba0e:::obj_delete_at_hint.0000000045:head, version: 64'17, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6f6aba0e:::obj_delete_at_hint.0000000045:head@64'17, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:51.378708+0000", 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.0054988279999999, 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069001464999999998, 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.481 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.378708+0000", 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.378708+0000", 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.482 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.378707+0000", 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4348999999999999e-05 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.378722+0000", 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5891000000000001e-05 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.378737+0000", 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4080000000000001e-06 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.378745+0000", 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068862920999999994 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.447608+0000", 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0780000000000003e-06 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.447613+0000", 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6967999999999997e-05 2024-02-13T00:44:52.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.447710+0000", 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069001464999999998 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.16 336/334 [PushOp(4:6b0d4d95:lc::lc.20:head, version: 65'19, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6b0d4d95:lc::lc.20:head@65'19, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:51.464684+0000", 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.91952304900000004, 2024-02-13T00:44:52.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069328461999999993, 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.464684+0000", 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.464684+0000", 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:44:52.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.464684+0000", 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0465000000000001e-05 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.464694+0000", 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3033999999999999e-05 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.464707+0000", 2024-02-13T00:44:52.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4029999999999996e-06 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.464716+0000", 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069196293000000006 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.533913+0000", 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9509999999999997e-06 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.533918+0000", 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4919000000000004e-05 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.534013+0000", 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069328461999999993 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.16 336/334 [PushOp(4:6d4431ed:lc::lc.7:head, version: 65'21, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d4431ed:lc::lc.7:head@65'21, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:51.550651+0000", 2024-02-13T00:44:52.489 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.83355667600000005, 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074139102999999998, 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.550651+0000", 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.550651+0000", 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:44:52.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.550650+0000", 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1858999999999999e-05 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.550662+0000", 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3012e-05 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.550675+0000", 2024-02-13T00:44:52.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.511e-06 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.550681+0000", 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074006764000000003 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.624688+0000", 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9620000000000004e-06 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.624692+0000", 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7652999999999995e-05 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.624790+0000", 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074139102999999998 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:44:52.493 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:44:52.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.400+0000 7f58deffd700 1 -- 172.21.15.162:0/4059015823 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v337) v1 ==== 91+0+0 (secure 0 0 0) 0x7f58e40515b0 con 0x7f58e80f8d10 2024-02-13T00:44:52.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.403+0000 7f58efe20700 10 librados: watch_flush enter 2024-02-13T00:44:52.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.404+0000 7f58efe20700 10 librados: watch_flush exit 2024-02-13T00:44:52.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.404+0000 7f58efe20700 20 client.44287.objecter shutdown clearing up homeless session... 2024-02-13T00:44:52.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.404+0000 7f58efe20700 10 client.44287.objecter successfully canceled tick 2024-02-13T00:44:52.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.407+0000 7f58efe20700 1 -- 172.21.15.162:0/4059015823 >> v2:172.21.15.162:6800/2866534053 conn(0x7f58d4062c90 msgr2=0x7f58d4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:52.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.407+0000 7f58efe20700 1 --2- 172.21.15.162:0/4059015823 >> v2:172.21.15.162:6800/2866534053 conn(0x7f58d4062c90 0x7f58d4065150 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 rx=0x7f58e8109a80 tx=0x7f58d8007b40).stop 2024-02-13T00:44:52.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.407+0000 7f58efe20700 1 -- 172.21.15.162:0/4059015823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58e80f8d10 msgr2=0x7f58e810e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:52.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.407+0000 7f58efe20700 1 --2- 172.21.15.162:0/4059015823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58e80f8d10 0x7f58e810e140 secure :-1 s=READY pgs=4853 cs=0 l=1 rev1=1 rx=0x7f58e40043a0 tx=0x7f58e4009cb0).stop 2024-02-13T00:44:52.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.408+0000 7f58efe20700 1 -- 172.21.15.162:0/4059015823 shutdown_connections 2024-02-13T00:44:52.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.408+0000 7f58efe20700 1 --2- 172.21.15.162:0/4059015823 >> v2:172.21.15.162:6800/2866534053 conn(0x7f58d4062c90 0x7f58d4065150 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:52.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.408+0000 7f58efe20700 1 --2- 172.21.15.162:0/4059015823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58e8102270 0x7f58e81082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:52.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.408+0000 7f58efe20700 1 --2- 172.21.15.162:0/4059015823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58e80f8d10 0x7f58e810e140 unknown :-1 s=CLOSED pgs=4853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:52.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.408+0000 7f58efe20700 1 --2- 172.21.15.162:0/4059015823 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e80f95e0 0x7f58e810e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:44:52.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.408+0000 7f58efe20700 1 -- 172.21.15.162:0/4059015823 >> 172.21.15.162:0/4059015823 conn(0x7f58e806eb90 msgr2=0x7f58e8102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:52.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.410+0000 7f58efe20700 1 -- 172.21.15.162:0/4059015823 shutdown_connections 2024-02-13T00:44:52.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.410+0000 7f58efe20700 1 -- 172.21.15.162:0/4059015823 wait complete. 2024-02-13T00:44:52.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:52.410+0000 7f58efe20700 1 librados: shutdown 2024-02-13T00:44:52.496 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:44:53.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:51.399312+0000 osd.5 (osd.5) 147 : cluster [DBG] 18.f starting backfill to osd.6 from (0'0,0'0] MAX to 309'7 2024-02-13T00:44:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:52.252113+0000 mon.b (mon.1) 503 : audit [INF] from='client.? 172.21.15.162:0/4059015823' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:44:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:52.252606+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:44:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:52.394499+0000 mon.a (mon.0) 2161 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:52.398327+0000 mon.a (mon.0) 2162 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]': finished 2024-02-13T00:44:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:52.398408+0000 mon.a (mon.0) 2163 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:44:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:52.398452+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e337: 8 total, 8 up, 6 in 2024-02-13T00:44:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:52.399573+0000 mon.c (mon.2) 326 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-13T00:44:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:52.402538+0000 mon.a (mon.0) 2165 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-13T00:44:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:52.432178+0000 mon.a (mon.0) 2166 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:52.434072+0000 mon.a (mon.0) 2167 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:52.434442+0000 mon.a (mon.0) 2168 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:52.442451+0000 mon.b (mon.1) 504 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:51.399312+0000 osd.5 (osd.5) 147 : cluster [DBG] 18.f starting backfill to osd.6 from (0'0,0'0] MAX to 309'7 2024-02-13T00:44:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:52.252113+0000 mon.b (mon.1) 503 : audit [INF] from='client.? 172.21.15.162:0/4059015823' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:44:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:52.252606+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:44:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:52.394499+0000 mon.a (mon.0) 2161 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:52.398327+0000 mon.a (mon.0) 2162 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]': finished 2024-02-13T00:44:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:52.398408+0000 mon.a (mon.0) 2163 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:44:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:52.398452+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e337: 8 total, 8 up, 6 in 2024-02-13T00:44:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:52.399573+0000 mon.c (mon.2) 326 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-13T00:44:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:52.402538+0000 mon.a (mon.0) 2165 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-13T00:44:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:52.432178+0000 mon.a (mon.0) 2166 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:52.434072+0000 mon.a (mon.0) 2167 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:53.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:52.434442+0000 mon.a (mon.0) 2168 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:53.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:52.442451+0000 mon.b (mon.1) 504 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:51.399312+0000 osd.5 (osd.5) 147 : cluster [DBG] 18.f starting backfill to osd.6 from (0'0,0'0] MAX to 309'7 2024-02-13T00:44:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:52.252113+0000 mon.b (mon.1) 503 : audit [INF] from='client.? 172.21.15.162:0/4059015823' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:44:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:52.252606+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:44:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:52.394499+0000 mon.a (mon.0) 2161 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:44:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:52.398327+0000 mon.a (mon.0) 2162 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-83985-57"}]': finished 2024-02-13T00:44:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:52.398408+0000 mon.a (mon.0) 2163 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:44:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:52.398452+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e337: 8 total, 8 up, 6 in 2024-02-13T00:44:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:52.399573+0000 mon.c (mon.2) 326 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-13T00:44:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:52.402538+0000 mon.a (mon.0) 2165 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-13T00:44:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:52.432178+0000 mon.a (mon.0) 2166 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:53.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:52.434072+0000 mon.a (mon.0) 2167 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:53.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:52.434442+0000 mon.a (mon.0) 2168 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:44:53.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:52.442451+0000 mon.b (mon.1) 504 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:53.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-13T00:44:53.187 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:53.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:53.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:53.193 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:53.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:53.319 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:53.443 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:44:53.445 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:53.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:53.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:53.586 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:53.715 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:53.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:53.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:44:53.847 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:52.396422+0000 mgr.y (mgr.14544) 308 : cluster [DBG] pgmap v475: 261 pgs: 105 peering, 156 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 40 B/s, 2 objects/s recovering 2024-02-13T00:44:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:52.736798+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:44:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:52.736959+0000 osd.0 (osd.0) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:44:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:53.410154+0000 mon.a (mon.0) 2169 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-13T00:44:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:53.410259+0000 mon.a (mon.0) 2170 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:53.410307+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e338: 8 total, 8 up, 6 in 2024-02-13T00:44:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:53.412601+0000 mon.a (mon.0) 2172 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi162-83985-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:54.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:53.443335+0000 mon.b (mon.1) 505 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:54.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:52.396422+0000 mgr.y (mgr.14544) 308 : cluster [DBG] pgmap v475: 261 pgs: 105 peering, 156 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 40 B/s, 2 objects/s recovering 2024-02-13T00:44:54.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:52.736798+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:44:54.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:52.736959+0000 osd.0 (osd.0) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:44:54.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:53.410154+0000 mon.a (mon.0) 2169 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-13T00:44:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:53.410259+0000 mon.a (mon.0) 2170 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:53.410307+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e338: 8 total, 8 up, 6 in 2024-02-13T00:44:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:53.412601+0000 mon.a (mon.0) 2172 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi162-83985-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:54.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:53.443335+0000 mon.b (mon.1) 505 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:52.396422+0000 mgr.y (mgr.14544) 308 : cluster [DBG] pgmap v475: 261 pgs: 105 peering, 156 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 40 B/s, 2 objects/s recovering 2024-02-13T00:44:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:52.736798+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:44:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:52.736959+0000 osd.0 (osd.0) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:44:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:53.410154+0000 mon.a (mon.0) 2169 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-13T00:44:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:53.410259+0000 mon.a (mon.0) 2170 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:44:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:53.410307+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e338: 8 total, 8 up, 6 in 2024-02-13T00:44:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:53.412601+0000 mon.a (mon.0) 2172 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi162-83985-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:53.443335+0000 mon.b (mon.1) 505 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:54.975 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:44:54.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:54.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:44:55.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:55.112 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:55.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.116+0000 7ff32868b700 1 -- 172.21.15.162:0/3777903980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200f73a0 msgr2=0x7ff3201039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.116+0000 7ff32868b700 1 --2- 172.21.15.162:0/3777903980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200f73a0 0x7ff3201039a0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 rx=0x7ff318009be0 tx=0x7ff318004d10).stop 2024-02-13T00:44:55.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.116+0000 7ff32868b700 1 -- 172.21.15.162:0/3777903980 shutdown_connections 2024-02-13T00:44:55.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.116+0000 7ff32868b700 1 --2- 172.21.15.162:0/3777903980 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200f73a0 0x7ff3201039a0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:55.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.116+0000 7ff32868b700 1 --2- 172.21.15.162:0/3777903980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3200fcb30 0x7ff3200fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:55.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.116+0000 7ff32868b700 1 --2- 172.21.15.162:0/3777903980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200fd490 0x7ff3200f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:44:55.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.116+0000 7ff32868b700 1 -- 172.21.15.162:0/3777903980 >> 172.21.15.162:0/3777903980 conn(0x7ff32006eb90 msgr2=0x7ff3200f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:55.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.116+0000 7ff32868b700 1 -- 172.21.15.162:0/3777903980 shutdown_connections 2024-02-13T00:44:55.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.116+0000 7ff32868b700 1 -- 172.21.15.162:0/3777903980 wait complete. 2024-02-13T00:44:55.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.116+0000 7ff32868b700 1 librados: starting msgr at 2024-02-13T00:44:55.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.116+0000 7ff32868b700 1 librados: starting objecter 2024-02-13T00:44:55.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff32868b700 1 Processor -- start 2024-02-13T00:44:55.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff32868b700 1 -- start start 2024-02-13T00:44:55.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff32868b700 1 librados: setting wanted keys 2024-02-13T00:44:55.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff32868b700 1 librados: calling monclient init 2024-02-13T00:44:55.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff32868b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200f73a0 0x7ff3200f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:55.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff32868b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fcb30 0x7ff3200f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:55.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff32868b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3200fd490 0x7ff3200f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:55.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff32868b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3201163f0 con 0x7ff3200f73a0 2024-02-13T00:44:55.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff32868b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff320116560 con 0x7ff3200fd490 2024-02-13T00:44:55.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff32868b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff320116720 con 0x7ff3200fcb30 2024-02-13T00:44:55.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff325c26700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fcb30 0x7ff3200f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:55.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff325c26700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fcb30 0x7ff3200f8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:57478/0 (socket says 172.21.15.162:57478) 2024-02-13T00:44:55.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff325c26700 1 -- 172.21.15.162:0/1542193523 learned_addr learned my addr 172.21.15.162:0/1542193523 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:55.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff326c28700 1 --2- 172.21.15.162:0/1542193523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3200fd490 0x7ff3200f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:55.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff326427700 1 --2- 172.21.15.162:0/1542193523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200f73a0 0x7ff3200f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:55.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff325c26700 1 -- 172.21.15.162:0/1542193523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3200fd490 msgr2=0x7ff3200f9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff325c26700 1 --2- 172.21.15.162:0/1542193523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3200fd490 0x7ff3200f9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:55.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff325c26700 1 -- 172.21.15.162:0/1542193523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200f73a0 msgr2=0x7ff3200f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff325c26700 1 --2- 172.21.15.162:0/1542193523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200f73a0 0x7ff3200f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:55.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.117+0000 7ff325c26700 1 -- 172.21.15.162:0/1542193523 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3180097c0 con 0x7ff3200fcb30 2024-02-13T00:44:55.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.118+0000 7ff325c26700 1 --2- 172.21.15.162:0/1542193523 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fcb30 0x7ff3200f8e20 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 rx=0x7ff31000b530 tx=0x7ff3100090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:55.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.118+0000 7ff3177fe700 10 client.?.objecter ms_handle_connect 0x7ff3200fcb30 2024-02-13T00:44:55.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.118+0000 7ff3177fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:55.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.118+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff310002320 con 0x7ff3200fcb30 2024-02-13T00:44:55.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.118+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff310002950 con 0x7ff3200fcb30 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.118+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff310012950 con 0x7ff3200fcb30 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.118+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff320116a00 con 0x7ff3200fcb30 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.118+0000 7ff32868b700 10 client.37232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.118+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff320116ea0 con 0x7ff3200fcb30 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.118+0000 7ff32868b700 1 librados: init done 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.119+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff320106090 con 0x7ff3200fcb30 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.120+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff310002480 con 0x7ff3200fcb30 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.120+0000 7ff3177fe700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff30c062c90 0x7ff30c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.120+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(339..339 src has 52..339) v4 ==== 8782+0+0 (secure 0 0 0) 0x7ff31007d4a0 con 0x7ff3200fcb30 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.120+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 osd_map(339..339 src has 52..339) v4 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.120+0000 7ff3177fe700 3 client.37232.objecter handle_osd_map got epochs [339,339] > 0 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.120+0000 7ff3177fe700 3 client.37232.objecter handle_osd_map decoding full epoch 339 2024-02-13T00:44:55.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.120+0000 7ff326427700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff30c062c90 0x7ff30c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:55.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.120+0000 7ff3177fe700 20 client.37232.objecter dump_active .. 0 homeless 2024-02-13T00:44:55.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.120+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v339) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff310084070 con 0x7ff3200fcb30 2024-02-13T00:44:55.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.120+0000 7ff3177fe700 10 client.37232.objecter ms_handle_connect 0x7ff30c062c90 2024-02-13T00:44:55.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.120+0000 7ff326427700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff30c062c90 0x7ff30c065150 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7ff31c004e60 tx=0x7ff31c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:55.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff32868b700 10 client.37232.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:55.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 1 2024-02-13T00:44:55.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff3200ffe10 0x7ff320100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:55.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff3200f9e10 osd=0 3 2024-02-13T00:44:55.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff3200f9e10 osd=0 3 2024-02-13T00:44:55.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff3200f9e10 osd=0 3 2024-02-13T00:44:55.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 1 2024-02-13T00:44:55.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff3200f9e10 osd=0 3 2024-02-13T00:44:55.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 0 1 2024-02-13T00:44:55.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff3200f9e10 osd=0 4 2024-02-13T00:44:55.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff32868b700 10 client.37232.objecter _send_command 1 2024-02-13T00:44:55.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff320100700 con 0x7ff3200ffe10 2024-02-13T00:44:55.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff326c28700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff3200ffe10 0x7ff320100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:55.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff3177fe700 10 client.37232.objecter ms_handle_connect 0x7ff3200ffe10 2024-02-13T00:44:55.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.121+0000 7ff326c28700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff3200ffe10 0x7ff320100210 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:55.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.122+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff320100700 con 0x7ff3200ffe10 2024-02-13T00:44:55.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.122+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:44:55.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.122+0000 7ff3177fe700 10 client.37232.objecter _finish_command 1 = system:0 2024-02-13T00:44:55.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.122+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff3200f9e10 osd=0 4 2024-02-13T00:44:55.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.122+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 0 1 2024-02-13T00:44:55.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.136+0000 7ff32868b700 10 client.37232.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:55.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.136+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 2 2024-02-13T00:44:55.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.136+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff3200f9e10 osd=0 3 2024-02-13T00:44:55.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.136+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff3200f9e10 osd=0 3 2024-02-13T00:44:55.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.136+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff3200f9e10 osd=0 3 2024-02-13T00:44:55.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.136+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 2 2024-02-13T00:44:55.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.136+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff3200f9e10 osd=0 3 2024-02-13T00:44:55.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.136+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 0 2 2024-02-13T00:44:55.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.136+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff3200f9e10 osd=0 4 2024-02-13T00:44:55.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.136+0000 7ff32868b700 10 client.37232.objecter _send_command 2 2024-02-13T00:44:55.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.136+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff3201008f0 con 0x7ff3200ffe10 2024-02-13T00:44:55.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff3201008f0 con 0x7ff3200ffe10 2024-02-13T00:44:55.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:55.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff3177fe700 10 client.37232.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff3200f9e10 osd=0 4 2024-02-13T00:44:55.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 0 2 2024-02-13T00:44:55.142 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:44:55.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff32868b700 10 client.37232.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:55.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 3 2024-02-13T00:44:55.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6805/150599780 conn(0x7ff320028590 0x7ff32011b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:55.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320061a10 osd=1 3 2024-02-13T00:44:55.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320061a10 osd=1 3 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320061a10 osd=1 3 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 3 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff320061a10 osd=1 3 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 1 3 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320061a10 osd=1 4 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff32868b700 10 client.37232.objecter _send_command 3 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff320061b80 con 0x7ff320028590 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff326427700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6805/150599780 conn(0x7ff320028590 0x7ff32011b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff326427700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6805/150599780 conn(0x7ff320028590 0x7ff32011b0b0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.139+0000 7ff3177fe700 10 client.37232.objecter ms_handle_connect 0x7ff320028590 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.140+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff320061b80 con 0x7ff320028590 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.140+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.140+0000 7ff3177fe700 10 client.37232.objecter _finish_command 3 = system:0 2024-02-13T00:44:55.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.140+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff320061a10 osd=1 4 2024-02-13T00:44:55.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.140+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 1 3 2024-02-13T00:44:55.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:44:55.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.155+0000 7ff32868b700 10 client.37232.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:55.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.155+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 4 2024-02-13T00:44:55.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.155+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320061a10 osd=1 3 2024-02-13T00:44:55.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.155+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320061a10 osd=1 3 2024-02-13T00:44:55.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.155+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320061a10 osd=1 3 2024-02-13T00:44:55.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.155+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 4 2024-02-13T00:44:55.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.155+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff320061a10 osd=1 3 2024-02-13T00:44:55.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.155+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 1 4 2024-02-13T00:44:55.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.155+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320061a10 osd=1 4 2024-02-13T00:44:55.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.155+0000 7ff32868b700 10 client.37232.objecter _send_command 4 2024-02-13T00:44:55.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.155+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff32005a0e0 con 0x7ff320028590 2024-02-13T00:44:55.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff32005a0e0 con 0x7ff320028590 2024-02-13T00:44:55.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:55.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff3177fe700 10 client.37232.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff320061a10 osd=1 4 2024-02-13T00:44:55.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 1 4 2024-02-13T00:44:55.161 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:44:55.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff32868b700 10 client.37232.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:55.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 5 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff320059570 0x7ff32011e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320059400 osd=2 3 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320059400 osd=2 3 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320059400 osd=2 3 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 5 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff320059400 osd=2 3 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 2 5 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320059400 osd=2 4 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff32868b700 10 client.37232.objecter _send_command 5 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3200281b0 con 0x7ff320059570 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.156+0000 7ff325c26700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff320059570 0x7ff32011e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.157+0000 7ff3177fe700 10 client.37232.objecter ms_handle_connect 0x7ff320059570 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.157+0000 7ff325c26700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff320059570 0x7ff32011e730 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:55.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.158+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff3200281b0 con 0x7ff320059570 2024-02-13T00:44:55.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.158+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:44:55.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.158+0000 7ff3177fe700 10 client.37232.objecter _finish_command 5 = system:0 2024-02-13T00:44:55.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.158+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff320059400 osd=2 4 2024-02-13T00:44:55.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.158+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 2 5 2024-02-13T00:44:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.174+0000 7ff32868b700 10 client.37232.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.174+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 6 2024-02-13T00:44:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.174+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320059400 osd=2 3 2024-02-13T00:44:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.174+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320059400 osd=2 3 2024-02-13T00:44:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.174+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320059400 osd=2 3 2024-02-13T00:44:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.174+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 6 2024-02-13T00:44:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.174+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff320059400 osd=2 3 2024-02-13T00:44:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.174+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 2 6 2024-02-13T00:44:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.174+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320059400 osd=2 4 2024-02-13T00:44:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.174+0000 7ff32868b700 10 client.37232.objecter _send_command 6 2024-02-13T00:44:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.174+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff320027650 con 0x7ff320059570 2024-02-13T00:44:55.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff320027650 con 0x7ff320059570 2024-02-13T00:44:55.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:55.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff3177fe700 10 client.37232.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff320059400 osd=2 4 2024-02-13T00:44:55.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 2 6 2024-02-13T00:44:55.180 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:44:55.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff32868b700 10 client.37232.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:55.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 7 2024-02-13T00:44:55.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6813/604971011 conn(0x7ff32011c280 0x7ff320121db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:55.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320027a60 osd=3 3 2024-02-13T00:44:55.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320027a60 osd=3 3 2024-02-13T00:44:55.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320027a60 osd=3 3 2024-02-13T00:44:55.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 7 2024-02-13T00:44:55.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff320027a60 osd=3 3 2024-02-13T00:44:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 3 7 2024-02-13T00:44:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320027a60 osd=3 4 2024-02-13T00:44:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff32868b700 10 client.37232.objecter _send_command 7 2024-02-13T00:44:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff320124a70 con 0x7ff32011c280 2024-02-13T00:44:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.177+0000 7ff326c28700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6813/604971011 conn(0x7ff32011c280 0x7ff320121db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.178+0000 7ff3177fe700 10 client.37232.objecter ms_handle_connect 0x7ff32011c280 2024-02-13T00:44:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.178+0000 7ff326c28700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6813/604971011 conn(0x7ff32011c280 0x7ff320121db0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.179+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff320124a70 con 0x7ff32011c280 2024-02-13T00:44:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.179+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:44:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.179+0000 7ff3177fe700 10 client.37232.objecter _finish_command 7 = system:0 2024-02-13T00:44:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.179+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff320027a60 osd=3 4 2024-02-13T00:44:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.179+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 3 7 2024-02-13T00:44:55.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.194+0000 7ff32868b700 10 client.37232.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:55.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.194+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 8 2024-02-13T00:44:55.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.194+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320027a60 osd=3 3 2024-02-13T00:44:55.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.194+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320027a60 osd=3 3 2024-02-13T00:44:55.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.194+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320027a60 osd=3 3 2024-02-13T00:44:55.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.194+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 8 2024-02-13T00:44:55.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.194+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff320027a60 osd=3 3 2024-02-13T00:44:55.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.194+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 3 8 2024-02-13T00:44:55.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.194+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320027a60 osd=3 4 2024-02-13T00:44:55.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.194+0000 7ff32868b700 10 client.37232.objecter _send_command 8 2024-02-13T00:44:55.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.194+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff32011b7e0 con 0x7ff32011c280 2024-02-13T00:44:55.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.196+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff32011b7e0 con 0x7ff32011c280 2024-02-13T00:44:55.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.196+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:55.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.196+0000 7ff3177fe700 10 client.37232.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.196+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff320027a60 osd=3 4 2024-02-13T00:44:55.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.196+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 3 8 2024-02-13T00:44:55.203 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:44:55.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff32868b700 10 client.37232.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:55.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 9 2024-02-13T00:44:55.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff32011b950 0x7ff320123420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:55.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff32011b7e0 osd=4 3 2024-02-13T00:44:55.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff32011b7e0 osd=4 3 2024-02-13T00:44:55.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff32011b7e0 osd=4 3 2024-02-13T00:44:55.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 9 2024-02-13T00:44:55.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff32011b7e0 osd=4 3 2024-02-13T00:44:55.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 4 9 2024-02-13T00:44:55.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff32011b7e0 osd=4 4 2024-02-13T00:44:55.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff32868b700 10 client.37232.objecter _send_command 9 2024-02-13T00:44:55.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff32011fa00 con 0x7ff32011b950 2024-02-13T00:44:55.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff326427700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff32011b950 0x7ff320123420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:55.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff3177fe700 10 client.37232.objecter ms_handle_connect 0x7ff32011b950 2024-02-13T00:44:55.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.197+0000 7ff326427700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff32011b950 0x7ff320123420 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:55.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.198+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff32011fa00 con 0x7ff32011b950 2024-02-13T00:44:55.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.198+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:44:55.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.198+0000 7ff3177fe700 10 client.37232.objecter _finish_command 9 = system:0 2024-02-13T00:44:55.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.198+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff32011b7e0 osd=4 4 2024-02-13T00:44:55.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.198+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 4 9 2024-02-13T00:44:55.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.213+0000 7ff32868b700 10 client.37232.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:55.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.213+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 10 2024-02-13T00:44:55.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.213+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff32011b7e0 osd=4 3 2024-02-13T00:44:55.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.213+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff32011b7e0 osd=4 3 2024-02-13T00:44:55.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.213+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff32011b7e0 osd=4 3 2024-02-13T00:44:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.213+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 10 2024-02-13T00:44:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.213+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff32011b7e0 osd=4 3 2024-02-13T00:44:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.213+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 4 10 2024-02-13T00:44:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.213+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff32011b7e0 osd=4 4 2024-02-13T00:44:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.213+0000 7ff32868b700 10 client.37232.objecter _send_command 10 2024-02-13T00:44:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.213+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff32011fa00 con 0x7ff32011b950 2024-02-13T00:44:55.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff32011fa00 con 0x7ff32011b950 2024-02-13T00:44:55.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:55.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff3177fe700 10 client.37232.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff32011b7e0 osd=4 4 2024-02-13T00:44:55.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 4 10 2024-02-13T00:44:55.219 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:44:55.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff32868b700 10 client.37232.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 11 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff320123e20 0x7ff320124260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff32011fa00 osd=5 3 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff32011fa00 osd=5 3 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff32011fa00 osd=5 3 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 11 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff32011fa00 osd=5 3 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 5 11 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff32011fa00 osd=5 4 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff32868b700 10 client.37232.objecter _send_command 11 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.215+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff320124810 con 0x7ff320123e20 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.216+0000 7ff325c26700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff320123e20 0x7ff320124260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.216+0000 7ff3177fe700 10 client.37232.objecter ms_handle_connect 0x7ff320123e20 2024-02-13T00:44:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.216+0000 7ff325c26700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff320123e20 0x7ff320124260 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.217+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff320124810 con 0x7ff320123e20 2024-02-13T00:44:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.217+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:44:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.217+0000 7ff3177fe700 10 client.37232.objecter _finish_command 11 = system:0 2024-02-13T00:44:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.217+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff32011fa00 osd=5 4 2024-02-13T00:44:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.217+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 5 11 2024-02-13T00:44:55.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.245+0000 7ff32868b700 10 client.37232.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:55.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.245+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 12 2024-02-13T00:44:55.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.245+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff32011fa00 osd=5 3 2024-02-13T00:44:55.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.245+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff32011fa00 osd=5 3 2024-02-13T00:44:55.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.245+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff32011fa00 osd=5 3 2024-02-13T00:44:55.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.245+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 12 2024-02-13T00:44:55.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.245+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff32011fa00 osd=5 3 2024-02-13T00:44:55.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.245+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 5 12 2024-02-13T00:44:55.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.245+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff32011fa00 osd=5 4 2024-02-13T00:44:55.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.245+0000 7ff32868b700 10 client.37232.objecter _send_command 12 2024-02-13T00:44:55.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.245+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff320124810 con 0x7ff320123e20 2024-02-13T00:44:55.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.246+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff320124810 con 0x7ff320123e20 2024-02-13T00:44:55.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.246+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:55.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.246+0000 7ff3177fe700 10 client.37232.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff32011fa00 osd=5 4 2024-02-13T00:44:55.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 5 12 2024-02-13T00:44:55.250 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:44:55.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff32868b700 10 client.37232.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:55.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 13 2024-02-13T00:44:55.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff320129220 0x7ff320129680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:55.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320124810 osd=6 3 2024-02-13T00:44:55.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320124810 osd=6 3 2024-02-13T00:44:55.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320124810 osd=6 3 2024-02-13T00:44:55.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 13 2024-02-13T00:44:55.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff320124810 osd=6 3 2024-02-13T00:44:55.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 6 13 2024-02-13T00:44:55.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320124810 osd=6 4 2024-02-13T00:44:55.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff32868b700 10 client.37232.objecter _send_command 13 2024-02-13T00:44:55.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff32012e5d0 con 0x7ff320129220 2024-02-13T00:44:55.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff326c28700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff320129220 0x7ff320129680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:55.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff3177fe700 10 client.37232.objecter ms_handle_connect 0x7ff320129220 2024-02-13T00:44:55.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.247+0000 7ff326c28700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff320129220 0x7ff320129680 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:55.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.248+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff32012e5d0 con 0x7ff320129220 2024-02-13T00:44:55.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.248+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:44:55.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.248+0000 7ff3177fe700 10 client.37232.objecter _finish_command 13 = system:0 2024-02-13T00:44:55.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.248+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff320124810 osd=6 4 2024-02-13T00:44:55.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.248+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 6 13 2024-02-13T00:44:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.263+0000 7ff32868b700 10 client.37232.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.263+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 14 2024-02-13T00:44:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.263+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320124810 osd=6 3 2024-02-13T00:44:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.263+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320124810 osd=6 3 2024-02-13T00:44:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.263+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320124810 osd=6 3 2024-02-13T00:44:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.263+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 14 2024-02-13T00:44:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.263+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff320124810 osd=6 3 2024-02-13T00:44:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.263+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 6 14 2024-02-13T00:44:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.263+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320124810 osd=6 4 2024-02-13T00:44:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.263+0000 7ff32868b700 10 client.37232.objecter _send_command 14 2024-02-13T00:44:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.263+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff320129c60 con 0x7ff320129220 2024-02-13T00:44:55.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff320129c60 con 0x7ff320129220 2024-02-13T00:44:55.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:55.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff3177fe700 10 client.37232.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff320124810 osd=6 4 2024-02-13T00:44:55.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 6 14 2024-02-13T00:44:55.269 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:44:55.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff32868b700 10 client.37232.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:44:55.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 15 2024-02-13T00:44:55.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff32012a2e0 0x7ff32012a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:55.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320129c60 osd=7 3 2024-02-13T00:44:55.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320129c60 osd=7 3 2024-02-13T00:44:55.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320129c60 osd=7 3 2024-02-13T00:44:55.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 15 2024-02-13T00:44:55.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff320129c60 osd=7 3 2024-02-13T00:44:55.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 7 15 2024-02-13T00:44:55.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320129c60 osd=7 4 2024-02-13T00:44:55.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff32868b700 10 client.37232.objecter _send_command 15 2024-02-13T00:44:55.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff32012adc0 con 0x7ff32012a2e0 2024-02-13T00:44:55.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.265+0000 7ff326427700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff32012a2e0 0x7ff32012a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:55.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.266+0000 7ff3177fe700 10 client.37232.objecter ms_handle_connect 0x7ff32012a2e0 2024-02-13T00:44:55.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.266+0000 7ff326427700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff32012a2e0 0x7ff32012a720 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:55.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.266+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff32012adc0 con 0x7ff32012a2e0 2024-02-13T00:44:55.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.266+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:44:55.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.266+0000 7ff3177fe700 10 client.37232.objecter _finish_command 15 = system:0 2024-02-13T00:44:55.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.266+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff320129c60 osd=7 4 2024-02-13T00:44:55.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.266+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 7 15 2024-02-13T00:44:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.281+0000 7ff32868b700 10 client.37232.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:44:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.281+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign -1 16 2024-02-13T00:44:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.281+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320129c60 osd=7 3 2024-02-13T00:44:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.281+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320129c60 osd=7 3 2024-02-13T00:44:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.281+0000 7ff32868b700 20 client.37232.objecter _get_session s=0x7ff320129c60 osd=7 3 2024-02-13T00:44:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.281+0000 7ff32868b700 15 client.37232.objecter _session_command_op_remove -1 16 2024-02-13T00:44:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.281+0000 7ff32868b700 20 client.37232.objecter get_session s=0x7ff320129c60 osd=7 3 2024-02-13T00:44:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.281+0000 7ff32868b700 15 client.37232.objecter _session_command_op_assign 7 16 2024-02-13T00:44:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.281+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320129c60 osd=7 4 2024-02-13T00:44:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.281+0000 7ff32868b700 10 client.37232.objecter _send_command 16 2024-02-13T00:44:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.281+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff32012adc0 con 0x7ff32012a2e0 2024-02-13T00:44:55.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff3177fe700 1 -- 172.21.15.162:0/1542193523 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff32012adc0 con 0x7ff32012a2e0 2024-02-13T00:44:55.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff3177fe700 10 client.37232.objecter ms_dispatch 0x7ff320001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:44:55.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff3177fe700 10 client.37232.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff3177fe700 20 client.37232.objecter put_session s=0x7ff320129c60 osd=7 4 2024-02-13T00:44:55.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff3177fe700 15 client.37232.objecter _session_command_op_remove 7 16 2024-02-13T00:44:55.286 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:44:55.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 10 librados: watch_flush enter 2024-02-13T00:44:55.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 10 librados: watch_flush exit 2024-02-13T00:44:55.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 10 client.37232.objecter close_session for osd.0 2024-02-13T00:44:55.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff3200ffe10 msgr2=0x7ff320100210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff3200ffe10 0x7ff320100210 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff3200f9e10 osd=0 1 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 10 client.37232.objecter close_session for osd.1 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6805/150599780 conn(0x7ff320028590 msgr2=0x7ff32011b0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6805/150599780 conn(0x7ff320028590 0x7ff32011b0b0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320061a10 osd=1 1 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 10 client.37232.objecter close_session for osd.2 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff320059570 msgr2=0x7ff32011e730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff320059570 0x7ff32011e730 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320059400 osd=2 1 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 10 client.37232.objecter close_session for osd.3 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6813/604971011 conn(0x7ff32011c280 msgr2=0x7ff320121db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6813/604971011 conn(0x7ff32011c280 0x7ff320121db0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:55.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff326c28700 1 -- 172.21.15.162:0/1542193523 reap_dead start 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320027a60 osd=3 1 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 10 client.37232.objecter close_session for osd.4 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff32011b950 msgr2=0x7ff320123420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff32011b950 0x7ff320123420 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff32011b7e0 osd=4 1 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 10 client.37232.objecter close_session for osd.5 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff320123e20 msgr2=0x7ff320124260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6804/2212296748 conn(0x7ff320123e20 0x7ff320124260 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff32011fa00 osd=5 1 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 10 client.37232.objecter close_session for osd.6 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff320129220 msgr2=0x7ff320129680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff320129220 0x7ff320129680 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320124810 osd=6 1 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 10 client.37232.objecter close_session for osd.7 2024-02-13T00:44:55.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff32012a2e0 msgr2=0x7ff32012a720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff32012a2e0 0x7ff32012a720 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 20 client.37232.objecter put_session s=0x7ff320129c60 osd=7 1 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 20 client.37232.objecter shutdown clearing up homeless session... 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 10 client.37232.objecter successfully canceled tick 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff30c062c90 msgr2=0x7ff30c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff30c062c90 0x7ff30c065150 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7ff31c004e60 tx=0x7ff31c00b040).stop 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fcb30 msgr2=0x7ff3200f8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff326c28700 1 -- 172.21.15.162:0/1542193523 reap_dead start 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff32868b700 1 --2- 172.21.15.162:0/1542193523 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fcb30 0x7ff3200f8e20 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 rx=0x7ff31000b530 tx=0x7ff3100090f0).stop 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff326c28700 1 -- 172.21.15.162:0/1542193523 reap_dead start 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff326c28700 1 -- 172.21.15.162:0/1542193523 reap_dead start 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.283+0000 7ff326c28700 1 -- 172.21.15.162:0/1542193523 reap_dead start 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.284+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 shutdown_connections 2024-02-13T00:44:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.284+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 >> 172.21.15.162:0/1542193523 conn(0x7ff32006eb90 msgr2=0x7ff320060e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.284+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 shutdown_connections 2024-02-13T00:44:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.284+0000 7ff32868b700 1 -- 172.21.15.162:0/1542193523 wait complete. 2024-02-13T00:44:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:55.284+0000 7ff32868b700 1 librados: shutdown 2024-02-13T00:44:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.291 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.292 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:44:55.562 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:55.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:55.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:54.397127+0000 mgr.y (mgr.14544) 309 : cluster [DBG] pgmap v478: 261 pgs: 105 peering, 156 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:55.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:54.444334+0000 mon.b (mon.1) 506 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:55.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:54.522423+0000 mon.b (mon.1) 507 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:55.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:54.574213+0000 osd.1 (osd.1) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:44:55.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:54.574408+0000 osd.1 (osd.1) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:44:55.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:54.661370+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e339: 8 total, 8 up, 6 in 2024-02-13T00:44:55.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:54.662371+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:55.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:54.663879+0000 mon.b (mon.1) 508 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43"}]: dispatch 2024-02-13T00:44:55.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:54.664328+0000 mon.a (mon.0) 2175 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43"}]: dispatch 2024-02-13T00:44:55.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:55.121056+0000 mon.c (mon.2) 327 : audit [DBG] from='client.? 172.21.15.162:0/1542193523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:55.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:54.397127+0000 mgr.y (mgr.14544) 309 : cluster [DBG] pgmap v478: 261 pgs: 105 peering, 156 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:55.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:54.444334+0000 mon.b (mon.1) 506 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:55.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:54.522423+0000 mon.b (mon.1) 507 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:55.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:54.574213+0000 osd.1 (osd.1) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:44:55.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:54.574408+0000 osd.1 (osd.1) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:44:55.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:54.661370+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e339: 8 total, 8 up, 6 in 2024-02-13T00:44:55.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:54.662371+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:55.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:54.663879+0000 mon.b (mon.1) 508 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43"}]: dispatch 2024-02-13T00:44:55.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:54.664328+0000 mon.a (mon.0) 2175 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43"}]: dispatch 2024-02-13T00:44:55.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:55.121056+0000 mon.c (mon.2) 327 : audit [DBG] from='client.? 172.21.15.162:0/1542193523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:55.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:55.698 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:55.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:44:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:44:55.786 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:54.397127+0000 mgr.y (mgr.14544) 309 : cluster [DBG] pgmap v478: 261 pgs: 105 peering, 156 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:44:55.786 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:54.444334+0000 mon.b (mon.1) 506 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:55.786 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:54.522423+0000 mon.b (mon.1) 507 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:55.787 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:54.574213+0000 osd.1 (osd.1) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:44:55.787 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:54.574408+0000 osd.1 (osd.1) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:44:55.787 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:54.661370+0000 mon.a (mon.0) 2173 : cluster [DBG] osdmap e339: 8 total, 8 up, 6 in 2024-02-13T00:44:55.787 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:54.662371+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:44:55.787 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:54.663879+0000 mon.b (mon.1) 508 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43"}]: dispatch 2024-02-13T00:44:55.787 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:54.664328+0000 mon.a (mon.0) 2175 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43"}]: dispatch 2024-02-13T00:44:55.787 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:55.121056+0000 mon.c (mon.2) 327 : audit [DBG] from='client.? 172.21.15.162:0/1542193523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:44:56.043 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:56.045 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:56.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:56.085 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:44:56.085 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:44:56.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:44:56.175 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:56.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:44:56] "GET /metrics HTTP/1.1" 200 191786 "" "Prometheus/2.33.4" 2024-02-13T00:44:56.376 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:44:56.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:56.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:44:56.425 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:55.415382+0000 mon.a (mon.0) 2176 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi162-83985-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]': finished 2024-02-13T00:44:56.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:56.509 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:56.603 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:55.415382+0000 mon.a (mon.0) 2176 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi162-83985-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]': finished 2024-02-13T00:44:56.603 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:55.415487+0000 mon.a (mon.0) 2177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43"}]': finished 2024-02-13T00:44:56.603 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:55.415553+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e340: 8 total, 8 up, 6 in 2024-02-13T00:44:56.603 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:55.445088+0000 mon.b (mon.1) 509 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:56.603 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:55.541370+0000 mgr.y (mgr.14544) 310 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:56.603 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:56.286368+0000 mon.c (mon.2) 328 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:44:56.604 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:56.397960+0000 mon.c (mon.2) 329 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-13T00:44:56.604 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:56.399620+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-13T00:44:56.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:55.415382+0000 mon.a (mon.0) 2176 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi162-83985-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]': finished 2024-02-13T00:44:56.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:55.415487+0000 mon.a (mon.0) 2177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43"}]': finished 2024-02-13T00:44:56.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:55.415553+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e340: 8 total, 8 up, 6 in 2024-02-13T00:44:56.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:55.445088+0000 mon.b (mon.1) 509 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:56.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:55.541370+0000 mgr.y (mgr.14544) 310 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:56.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:56.286368+0000 mon.c (mon.2) 328 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:44:56.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:56.397960+0000 mon.c (mon.2) 329 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-13T00:44:56.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:56.399620+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-13T00:44:56.754 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:55.415487+0000 mon.a (mon.0) 2177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-43"}]': finished 2024-02-13T00:44:56.754 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:55.415553+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e340: 8 total, 8 up, 6 in 2024-02-13T00:44:56.754 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:55.445088+0000 mon.b (mon.1) 509 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:56.754 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:55.541370+0000 mgr.y (mgr.14544) 310 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:44:56.754 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:56.286368+0000 mon.c (mon.2) 328 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:44:56.754 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:56.397960+0000 mon.c (mon.2) 329 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-13T00:44:56.754 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:56.399620+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-13T00:44:56.757 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:56.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:56.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-13T00:44:56.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:44:56.889 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:56.910 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:44:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:56.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:44:57.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:44:57.042 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:57.063 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:44:57.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:57.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:44:57.091 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:44:57.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:57.194 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:57.542 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:57.543 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 6] out_osds: [3, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:44:57.543 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:44:57.543 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-02-13T00:44:57.543 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-13T00:44:57.543 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-13T00:44:57.543 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-02-13T00:44:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:57.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:57.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:44:57.675 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:56.397541+0000 mgr.y (mgr.14544) 311 : cluster [DBG] pgmap v481: 268 pgs: 8 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 clean+premerge+peered, 3 active+recovering, 254 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 6.0 KiB/s wr, 18 op/s; 31/1101 objects degraded (2.816%); 11/1101 objects misplaced (0.999%); 757 B/s, 65 objects/s recovering 2024-02-13T00:44:57.675 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:56.411245+0000 mon.a (mon.0) 2180 : cluster [WRN] Health check failed: Degraded data redundancy: 31/1101 objects degraded (2.816%), 1 pg degraded (PG_DEGRADED) 2024-02-13T00:44:57.675 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:56.411271+0000 mon.a (mon.0) 2181 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive, 23 pgs peering) 2024-02-13T00:44:57.675 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:56.421304+0000 mon.a (mon.0) 2182 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-13T00:44:57.675 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:56.421409+0000 mon.a (mon.0) 2183 : cluster [DBG] osdmap e341: 8 total, 8 up, 6 in 2024-02-13T00:44:57.675 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:56.447172+0000 mon.b (mon.1) 510 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:57.675 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:56.931884+0000 mon.c (mon.2) 330 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:44:57.675 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:56.932532+0000 mon.c (mon.2) 331 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:44:57.675 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:56.937550+0000 mon.a (mon.0) 2184 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:57.677 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:57.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:56.397541+0000 mgr.y (mgr.14544) 311 : cluster [DBG] pgmap v481: 268 pgs: 8 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 clean+premerge+peered, 3 active+recovering, 254 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 6.0 KiB/s wr, 18 op/s; 31/1101 objects degraded (2.816%); 11/1101 objects misplaced (0.999%); 757 B/s, 65 objects/s recovering 2024-02-13T00:44:57.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:56.411245+0000 mon.a (mon.0) 2180 : cluster [WRN] Health check failed: Degraded data redundancy: 31/1101 objects degraded (2.816%), 1 pg degraded (PG_DEGRADED) 2024-02-13T00:44:57.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:56.411271+0000 mon.a (mon.0) 2181 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive, 23 pgs peering) 2024-02-13T00:44:57.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:56.421304+0000 mon.a (mon.0) 2182 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-13T00:44:57.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:56.421409+0000 mon.a (mon.0) 2183 : cluster [DBG] osdmap e341: 8 total, 8 up, 6 in 2024-02-13T00:44:57.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:56.447172+0000 mon.b (mon.1) 510 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:57.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:56.931884+0000 mon.c (mon.2) 330 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:44:57.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:56.932532+0000 mon.c (mon.2) 331 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:44:57.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:56.937550+0000 mon.a (mon.0) 2184 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:57.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:56.397541+0000 mgr.y (mgr.14544) 311 : cluster [DBG] pgmap v481: 268 pgs: 8 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 clean+premerge+peered, 3 active+recovering, 254 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 6.0 KiB/s wr, 18 op/s; 31/1101 objects degraded (2.816%); 11/1101 objects misplaced (0.999%); 757 B/s, 65 objects/s recovering 2024-02-13T00:44:57.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:56.411245+0000 mon.a (mon.0) 2180 : cluster [WRN] Health check failed: Degraded data redundancy: 31/1101 objects degraded (2.816%), 1 pg degraded (PG_DEGRADED) 2024-02-13T00:44:57.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:56.411271+0000 mon.a (mon.0) 2181 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive, 23 pgs peering) 2024-02-13T00:44:57.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:56.421304+0000 mon.a (mon.0) 2182 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-13T00:44:57.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:56.421409+0000 mon.a (mon.0) 2183 : cluster [DBG] osdmap e341: 8 total, 8 up, 6 in 2024-02-13T00:44:57.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:56.447172+0000 mon.b (mon.1) 510 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:57.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:56.931884+0000 mon.c (mon.2) 330 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:44:57.788 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:56.932532+0000 mon.c (mon.2) 331 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:44:57.788 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:56.937550+0000 mon.a (mon.0) 2184 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:44:57.877 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:44:57.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:57.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:44:57.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:44:58.011 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:58.031 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:44:58.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.028+0000 7ff422c97700 1 -- 172.21.15.162:0/350698184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff41c0fde50 msgr2=0x7ff41c0fe230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:58.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.028+0000 7ff422c97700 1 --2- 172.21.15.162:0/350698184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff41c0fde50 0x7ff41c0fe230 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7ff410004870 tx=0x7ff410009cb0).stop 2024-02-13T00:44:58.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.028+0000 7ff422c97700 1 -- 172.21.15.162:0/350698184 shutdown_connections 2024-02-13T00:44:58.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.028+0000 7ff422c97700 1 --2- 172.21.15.162:0/350698184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff41c0f6880 0x7ff41c104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:58.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.028+0000 7ff422c97700 1 --2- 172.21.15.162:0/350698184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff41c0fde50 0x7ff41c0fe230 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:58.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.028+0000 7ff422c97700 1 --2- 172.21.15.162:0/350698184 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff41c0fe7b0 0x7ff41c0f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:58.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.028+0000 7ff422c97700 1 -- 172.21.15.162:0/350698184 >> 172.21.15.162:0/350698184 conn(0x7ff41c070080 msgr2=0x7ff41c0f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:58.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.028+0000 7ff422c97700 1 -- 172.21.15.162:0/350698184 shutdown_connections 2024-02-13T00:44:58.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.028+0000 7ff422c97700 1 -- 172.21.15.162:0/350698184 wait complete. 2024-02-13T00:44:58.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:44:58.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.029+0000 7ff422c97700 1 librados: starting msgr at 2024-02-13T00:44:58.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.029+0000 7ff422c97700 1 librados: starting objecter 2024-02-13T00:44:58.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.029+0000 7ff422c97700 1 Processor -- start 2024-02-13T00:44:58.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.029+0000 7ff422c97700 1 -- start start 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.029+0000 7ff422c97700 1 librados: setting wanted keys 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.029+0000 7ff422c97700 1 librados: calling monclient init 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.029+0000 7ff422c97700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff41c0f6880 0x7ff41c0f8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.029+0000 7ff422c97700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff41c0fde50 0x7ff41c0f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff422c97700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff41c0fe7b0 0x7ff41c0f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff422c97700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0f8160 con 0x7ff41c0f6880 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff422c97700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0f82d0 con 0x7ff41c0fde50 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff422c97700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c10c7e0 con 0x7ff41c0fe7b0 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff420a33700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff41c0f6880 0x7ff41c0f8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff420a33700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff41c0f6880 0x7ff41c0f8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:56650/0 (socket says 172.21.15.162:56650) 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff420a33700 1 -- 172.21.15.162:0/1725121729 learned_addr learned my addr 172.21.15.162:0/1725121729 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff421234700 1 --2- 172.21.15.162:0/1725121729 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff41c0fe7b0 0x7ff41c0f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:58.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff41bfff700 1 --2- 172.21.15.162:0/1725121729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff41c0fde50 0x7ff41c0f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:58.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff420a33700 1 -- 172.21.15.162:0/1725121729 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff41c0fe7b0 msgr2=0x7ff41c0f7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:58.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff420a33700 1 --2- 172.21.15.162:0/1725121729 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff41c0fe7b0 0x7ff41c0f7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:58.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff420a33700 1 -- 172.21.15.162:0/1725121729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff41c0fde50 msgr2=0x7ff41c0f7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:58.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff420a33700 1 --2- 172.21.15.162:0/1725121729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff41c0fde50 0x7ff41c0f7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:58.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff420a33700 1 -- 172.21.15.162:0/1725121729 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff410005040 con 0x7ff41c0f6880 2024-02-13T00:44:58.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff41bfff700 1 --2- 172.21.15.162:0/1725121729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff41c0fde50 0x7ff41c0f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:44:58.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff420a33700 1 --2- 172.21.15.162:0/1725121729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff41c0f6880 0x7ff41c0f8ed0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 rx=0x7ff4100043a0 tx=0x7ff41000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:58.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff419ffb700 10 client.?.objecter ms_handle_connect 0x7ff41c0f6880 2024-02-13T00:44:58.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.030+0000 7ff419ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:44:58.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.031+0000 7ff419ffb700 1 -- 172.21.15.162:0/1725121729 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41000e070 con 0x7ff41c0f6880 2024-02-13T00:44:58.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.031+0000 7ff419ffb700 1 -- 172.21.15.162:0/1725121729 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff410009710 con 0x7ff41c0f6880 2024-02-13T00:44:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:58.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:44:58.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.031+0000 7ff422c97700 1 -- 172.21.15.162:0/1725121729 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41c10ca60 con 0x7ff41c0f6880 2024-02-13T00:44:58.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.031+0000 7ff419ffb700 1 -- 172.21.15.162:0/1725121729 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff410019070 con 0x7ff41c0f6880 2024-02-13T00:44:58.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.031+0000 7ff422c97700 10 client.27129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:44:58.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.031+0000 7ff422c97700 1 -- 172.21.15.162:0/1725121729 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41c10cf50 con 0x7ff41c0f6880 2024-02-13T00:44:58.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.031+0000 7ff422c97700 1 librados: init done 2024-02-13T00:44:58.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.032+0000 7ff422c97700 1 -- 172.21.15.162:0/1725121729 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff41c05fb80 con 0x7ff41c0f6880 2024-02-13T00:44:58.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.033+0000 7ff419ffb700 1 -- 172.21.15.162:0/1725121729 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff4100090d0 con 0x7ff41c0f6880 2024-02-13T00:44:58.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.034+0000 7ff419ffb700 1 --2- 172.21.15.162:0/1725121729 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff40c062ce0 0x7ff40c0651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:44:58.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.034+0000 7ff419ffb700 1 -- 172.21.15.162:0/1725121729 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(342..342 src has 52..342) v4 ==== 8702+0+0 (secure 0 0 0) 0x7ff41007de60 con 0x7ff41c0f6880 2024-02-13T00:44:58.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.034+0000 7ff419ffb700 10 client.27129.objecter ms_dispatch 0x7ff41c001550 osd_map(342..342 src has 52..342) v4 2024-02-13T00:44:58.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.034+0000 7ff419ffb700 3 client.27129.objecter handle_osd_map got epochs [342,342] > 0 2024-02-13T00:44:58.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.034+0000 7ff419ffb700 3 client.27129.objecter handle_osd_map decoding full epoch 342 2024-02-13T00:44:58.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.034+0000 7ff41bfff700 1 --2- 172.21.15.162:0/1725121729 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff40c062ce0 0x7ff40c0651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:44:58.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.034+0000 7ff419ffb700 20 client.27129.objecter dump_active .. 0 homeless 2024-02-13T00:44:58.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.034+0000 7ff41bfff700 1 --2- 172.21.15.162:0/1725121729 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff40c062ce0 0x7ff40c0651a0 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 rx=0x7ff41c0f8c10 tx=0x7ff40800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:44:58.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.037+0000 7ff419ffb700 10 client.27129.objecter ms_handle_connect 0x7ff40c062ce0 2024-02-13T00:44:58.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.043+0000 7ff419ffb700 1 -- 172.21.15.162:0/1725121729 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff410051200 con 0x7ff41c0f6880 2024-02-13T00:44:58.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:58.166 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:58.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.269+0000 7ff422c97700 1 -- 172.21.15.162:0/1725121729 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7ff41c107390 con 0x7ff41c0f6880 2024-02-13T00:44:58.302 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:44:58.346 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:44:58.365 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:58.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:58.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.429+0000 7ff419ffb700 1 -- 172.21.15.162:0/1725121729 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v343) v1 ==== 101+0+0 (secure 0 0 0) 0x7ff410050df0 con 0x7ff41c0f6880 2024-02-13T00:44:58.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.433+0000 7ff422c97700 10 librados: watch_flush enter 2024-02-13T00:44:58.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.433+0000 7ff422c97700 10 librados: watch_flush exit 2024-02-13T00:44:58.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.433+0000 7ff422c97700 20 client.27129.objecter shutdown clearing up homeless session... 2024-02-13T00:44:58.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.433+0000 7ff422c97700 10 client.27129.objecter successfully canceled tick 2024-02-13T00:44:58.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.433+0000 7ff422c97700 1 -- 172.21.15.162:0/1725121729 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff40c062ce0 msgr2=0x7ff40c0651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:58.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.433+0000 7ff422c97700 1 --2- 172.21.15.162:0/1725121729 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff40c062ce0 0x7ff40c0651a0 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 rx=0x7ff41c0f8c10 tx=0x7ff40800b040).stop 2024-02-13T00:44:58.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.433+0000 7ff422c97700 1 -- 172.21.15.162:0/1725121729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff41c0f6880 msgr2=0x7ff41c0f8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:44:58.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.433+0000 7ff422c97700 1 --2- 172.21.15.162:0/1725121729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff41c0f6880 0x7ff41c0f8ed0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 rx=0x7ff4100043a0 tx=0x7ff41000b040).stop 2024-02-13T00:44:58.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.435+0000 7ff422c97700 1 -- 172.21.15.162:0/1725121729 shutdown_connections 2024-02-13T00:44:58.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.435+0000 7ff422c97700 1 --2- 172.21.15.162:0/1725121729 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff40c062ce0 0x7ff40c0651a0 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:58.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.435+0000 7ff422c97700 1 --2- 172.21.15.162:0/1725121729 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff41c0fe7b0 0x7ff41c0f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:58.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.435+0000 7ff422c97700 1 --2- 172.21.15.162:0/1725121729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff41c0f6880 0x7ff41c0f8ed0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:58.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.435+0000 7ff422c97700 1 --2- 172.21.15.162:0/1725121729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff41c0fde50 0x7ff41c0f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:44:58.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.435+0000 7ff422c97700 1 -- 172.21.15.162:0/1725121729 >> 172.21.15.162:0/1725121729 conn(0x7ff41c070080 msgr2=0x7ff41c06d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:44:58.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.438+0000 7ff422c97700 1 -- 172.21.15.162:0/1725121729 shutdown_connections 2024-02-13T00:44:58.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.440+0000 7ff422c97700 1 -- 172.21.15.162:0/1725121729 wait complete. 2024-02-13T00:44:58.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:44:58.441+0000 7ff422c97700 1 librados: shutdown 2024-02-13T00:44:58.446 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:44:58.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:58.495 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:58.616 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:44:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:44:58.617 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:57.422397+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e342: 8 total, 8 up, 6 in 2024-02-13T00:44:58.617 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:57.425135+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:58.617 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:57.429045+0000 mon.b (mon.1) 511 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:58.617 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:57.438766+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:58.617 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:57.448197+0000 mon.b (mon.1) 512 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:58.617 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:58.270270+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? 172.21.15.162:0/1725121729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:58.617 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:58.419495+0000 mon.a (mon.0) 2189 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:44:58.617 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:58.425688+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]': finished 2024-02-13T00:44:58.617 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:58.425813+0000 mon.a (mon.0) 2191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:58.617 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:58.425898+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? 172.21.15.162:0/1725121729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:44:58.617 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:58.425941+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e343: 8 total, 8 up, 6 in 2024-02-13T00:44:58.617 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:58.428687+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:58.618 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:58.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:58.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:44:58.749 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:57.422397+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e342: 8 total, 8 up, 6 in 2024-02-13T00:44:58.749 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:57.425135+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:58.749 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:57.429045+0000 mon.b (mon.1) 511 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:58.750 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:57.438766+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:58.750 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:57.448197+0000 mon.b (mon.1) 512 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:58.750 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:58.270270+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? 172.21.15.162:0/1725121729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:58.750 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:58.419495+0000 mon.a (mon.0) 2189 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:44:58.750 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:58.425688+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]': finished 2024-02-13T00:44:58.750 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:58.425813+0000 mon.a (mon.0) 2191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:58.750 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:58.425898+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? 172.21.15.162:0/1725121729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:44:58.750 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:58.425941+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e343: 8 total, 8 up, 6 in 2024-02-13T00:44:58.750 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:58.428687+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:58.750 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:57.422397+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e342: 8 total, 8 up, 6 in 2024-02-13T00:44:58.750 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:57.425135+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:58.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:57.429045+0000 mon.b (mon.1) 511 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:58.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:57.438766+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:44:58.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:57.448197+0000 mon.b (mon.1) 512 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:58.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:58.270270+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? 172.21.15.162:0/1725121729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:44:58.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:58.419495+0000 mon.a (mon.0) 2189 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:44:58.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:58.425688+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-83985-58"}]': finished 2024-02-13T00:44:58.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:58.425813+0000 mon.a (mon.0) 2191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:44:58.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:58.425898+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? 172.21.15.162:0/1725121729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:44:58.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:58.425941+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e343: 8 total, 8 up, 6 in 2024-02-13T00:44:58.752 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:58.428687+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-83985-58"}]: dispatch 2024-02-13T00:44:58.752 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:58.825 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:44:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:58.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:44:58.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:58.956 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:59.218 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:44:59.218 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:44:59.304 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:59.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:59.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:44:59.442 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:59.464 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:44:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:59.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:44:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:58.398250+0000 mgr.y (mgr.14544) 312 : cluster [DBG] pgmap v484: 260 pgs: 32 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 clean+premerge+peered, 3 active+recovering, 222 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 1.7 KiB/s wr, 14 op/s; 31/1089 objects degraded (2.847%); 11/1089 objects misplaced (1.010%); 757 B/s, 65 objects/s recovering 2024-02-13T00:44:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:58.451580+0000 mon.b (mon.1) 513 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:58.475159+0000 mon.b (mon.1) 514 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:58.475645+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:59.430410+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-83985-58"}]': finished 2024-02-13T00:44:59.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:59.430507+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:59.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:59.430579+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e344: 8 total, 8 up, 6 in 2024-02-13T00:44:59.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:59.432031+0000 mon.b (mon.1) 515 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:44:59.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:44:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:59.434040+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:44:59.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:44:59.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:58.398250+0000 mgr.y (mgr.14544) 312 : cluster [DBG] pgmap v484: 260 pgs: 32 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 clean+premerge+peered, 3 active+recovering, 222 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 1.7 KiB/s wr, 14 op/s; 31/1089 objects degraded (2.847%); 11/1089 objects misplaced (1.010%); 757 B/s, 65 objects/s recovering 2024-02-13T00:44:59.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:58.451580+0000 mon.b (mon.1) 513 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:59.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:58.475159+0000 mon.b (mon.1) 514 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:59.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:58.475645+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:59.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:59.430410+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-83985-58"}]': finished 2024-02-13T00:44:59.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:59.430507+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:59.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:59.430579+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e344: 8 total, 8 up, 6 in 2024-02-13T00:44:59.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:59.432031+0000 mon.b (mon.1) 515 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:44:59.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:59.434040+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:44:59.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:58.398250+0000 mgr.y (mgr.14544) 312 : cluster [DBG] pgmap v484: 260 pgs: 32 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 clean+premerge+peered, 3 active+recovering, 222 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 1.7 KiB/s wr, 14 op/s; 31/1089 objects degraded (2.847%); 11/1089 objects misplaced (1.010%); 757 B/s, 65 objects/s recovering 2024-02-13T00:44:59.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:58.451580+0000 mon.b (mon.1) 513 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:44:59.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:58.475159+0000 mon.b (mon.1) 514 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:59.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:58.475645+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:44:59.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:59.430410+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? 172.21.15.162:0/233748748' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-83985-58"}]': finished 2024-02-13T00:44:59.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:59.430507+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:44:59.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:59.430579+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e344: 8 total, 8 up, 6 in 2024-02-13T00:44:59.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:59.432031+0000 mon.b (mon.1) 515 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:44:59.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:44:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:59.434040+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:44:59.596 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:44:59.946 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:44:59.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:44:59.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:44:59.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-13T00:45:00.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:00.076 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:00.526 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:00.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:00.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:00.656 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:58.966778+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:45:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:58.969685+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:45:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:59.225826+0000 osd.1 (osd.1) 139 : cluster [DBG] 18.3 scrub starts 2024-02-13T00:45:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:59.229285+0000 osd.1 (osd.1) 140 : cluster [DBG] 18.3 scrub ok 2024-02-13T00:45:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:44:59.452543+0000 mon.b (mon.1) 516 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:00.046034+0000 mon.a (mon.0) 2200 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:00.381112+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-45"}]': finished 2024-02-13T00:45:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:00.381238+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e345: 8 total, 8 up, 6 in 2024-02-13T00:45:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:00.386499+0000 mon.c (mon.2) 332 : audit [INF] from='client.? 172.21.15.162:0/1025848595' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:00.386896+0000 mon.b (mon.1) 517 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-45", "mode": "writeback"}]: dispatch 2024-02-13T00:45:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:00.392755+0000 mon.a (mon.0) 2203 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:00.395287+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-45", "mode": "writeback"}]: dispatch 2024-02-13T00:45:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:00.399122+0000 mon.c (mon.2) 333 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-13T00:45:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:00.400130+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-13T00:45:00.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:58.966778+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:45:00.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:58.969685+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:45:00.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:59.225826+0000 osd.1 (osd.1) 139 : cluster [DBG] 18.3 scrub starts 2024-02-13T00:45:00.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:59.229285+0000 osd.1 (osd.1) 140 : cluster [DBG] 18.3 scrub ok 2024-02-13T00:45:00.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:44:59.452543+0000 mon.b (mon.1) 516 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:00.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:00.046034+0000 mon.a (mon.0) 2200 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:00.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:00.381112+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-45"}]': finished 2024-02-13T00:45:00.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:00.381238+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e345: 8 total, 8 up, 6 in 2024-02-13T00:45:00.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:00.386499+0000 mon.c (mon.2) 332 : audit [INF] from='client.? 172.21.15.162:0/1025848595' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:00.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:00.386896+0000 mon.b (mon.1) 517 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-45", "mode": "writeback"}]: dispatch 2024-02-13T00:45:00.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:00.392755+0000 mon.a (mon.0) 2203 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:00.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:00.395287+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-45", "mode": "writeback"}]: dispatch 2024-02-13T00:45:00.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:00.399122+0000 mon.c (mon.2) 333 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-13T00:45:00.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:00.400130+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-13T00:45:00.809 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:58.966778+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:45:00.809 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:58.969685+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:45:00.809 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:59.225826+0000 osd.1 (osd.1) 139 : cluster [DBG] 18.3 scrub starts 2024-02-13T00:45:00.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:59.229285+0000 osd.1 (osd.1) 140 : cluster [DBG] 18.3 scrub ok 2024-02-13T00:45:00.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:44:59.452543+0000 mon.b (mon.1) 516 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:00.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:00.046034+0000 mon.a (mon.0) 2200 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:00.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:00.381112+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-45"}]': finished 2024-02-13T00:45:00.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:00.381238+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e345: 8 total, 8 up, 6 in 2024-02-13T00:45:00.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:00.386499+0000 mon.c (mon.2) 332 : audit [INF] from='client.? 172.21.15.162:0/1025848595' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:00.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:00.386896+0000 mon.b (mon.1) 517 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-45", "mode": "writeback"}]: dispatch 2024-02-13T00:45:00.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:00.392755+0000 mon.a (mon.0) 2203 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:00.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:00.395287+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-45", "mode": "writeback"}]: dispatch 2024-02-13T00:45:00.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:00.399122+0000 mon.c (mon.2) 333 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-13T00:45:00.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:00.400130+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-13T00:45:00.853 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:00.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:00.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:00.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:00.983 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:01.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.042+0000 7f2686ddf700 1 -- 172.21.15.162:0/1422753118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26800fd490 msgr2=0x7f26800f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.042+0000 7f2686ddf700 1 --2- 172.21.15.162:0/1422753118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26800fd490 0x7f26800f6cc0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 rx=0x7f266c004870 tx=0x7f266c009cb0).stop 2024-02-13T00:45:01.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.042+0000 7f2686ddf700 1 -- 172.21.15.162:0/1422753118 shutdown_connections 2024-02-13T00:45:01.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.042+0000 7f2686ddf700 1 --2- 172.21.15.162:0/1422753118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26800f73a0 0x7f26801039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.042+0000 7f2686ddf700 1 --2- 172.21.15.162:0/1422753118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26800fcb30 0x7f26800fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.042+0000 7f2686ddf700 1 --2- 172.21.15.162:0/1422753118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26800fd490 0x7f26800f6cc0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.042+0000 7f2686ddf700 1 -- 172.21.15.162:0/1422753118 >> 172.21.15.162:0/1422753118 conn(0x7f268006eb90 msgr2=0x7f26800f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.042+0000 7f2686ddf700 1 -- 172.21.15.162:0/1422753118 shutdown_connections 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.042+0000 7f2686ddf700 1 -- 172.21.15.162:0/1422753118 wait complete. 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.042+0000 7f2686ddf700 1 librados: starting msgr at 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.042+0000 7f2686ddf700 1 librados: starting objecter 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f2686ddf700 1 Processor -- start 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f2686ddf700 1 -- start start 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f2686ddf700 1 librados: setting wanted keys 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f2686ddf700 1 librados: calling monclient init 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f2686ddf700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26800f73a0 0x7f26800f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f2686ddf700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26800fcb30 0x7f26800f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:01.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f2686ddf700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26800fd490 0x7f26800f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f2686ddf700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2680116650 con 0x7f26800f73a0 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f2686ddf700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26801167c0 con 0x7f26800fcb30 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f2686ddf700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2680116980 con 0x7f26800fd490 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f268537c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26800fd490 0x7f26800f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f268537c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26800fd490 0x7f26800f9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:57526/0 (socket says 172.21.15.162:57526) 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f2684b7b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26800f73a0 0x7f26800f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f268537c700 1 -- 172.21.15.162:0/2807854082 learned_addr learned my addr 172.21.15.162:0/2807854082 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f267ffff700 1 --2- 172.21.15.162:0/2807854082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26800fcb30 0x7f26800f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f268537c700 1 -- 172.21.15.162:0/2807854082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26800fcb30 msgr2=0x7f26800f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f268537c700 1 --2- 172.21.15.162:0/2807854082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26800fcb30 0x7f26800f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f268537c700 1 -- 172.21.15.162:0/2807854082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26800f73a0 msgr2=0x7f26800f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f268537c700 1 --2- 172.21.15.162:0/2807854082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26800f73a0 0x7f26800f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f268537c700 1 -- 172.21.15.162:0/2807854082 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2674005040 con 0x7f26800fd490 2024-02-13T00:45:01.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f2684b7b700 1 --2- 172.21.15.162:0/2807854082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26800f73a0 0x7f26800f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f268537c700 1 --2- 172.21.15.162:0/2807854082 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26800fd490 0x7f26800f9310 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f267800c870 tx=0x7f2678007dd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f267dffb700 10 client.?.objecter ms_handle_connect 0x7f26800fd490 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f267dffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.043+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2678010510 con 0x7f26800fd490 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.044+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2678010b40 con 0x7f26800fd490 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.044+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c005040 con 0x7f26800fd490 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.044+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2678012ae0 con 0x7f26800fd490 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.044+0000 7f2686ddf700 10 client.37250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.044+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2680116ff0 con 0x7f26800fd490 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.044+0000 7f2686ddf700 1 librados: init done 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.044+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f2680106020 con 0x7f26800fd490 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.045+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f2678010670 con 0x7f26800fd490 2024-02-13T00:45:01.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.045+0000 7f267dffb700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2670062c90 0x7f2670065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.045+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(345..345 src has 52..345) v4 ==== 8662+0+0 (secure 0 0 0) 0x7f267807d520 con 0x7f26800fd490 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.045+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 osd_map(345..345 src has 52..345) v4 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.045+0000 7f267dffb700 3 client.37250.objecter handle_osd_map got epochs [345,345] > 0 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.045+0000 7f267dffb700 3 client.37250.objecter handle_osd_map decoding full epoch 345 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.045+0000 7f2684b7b700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2670062c90 0x7f2670065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f267dffb700 20 client.37250.objecter dump_active .. 0 homeless 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v345) v1 ==== 54+0+16 (secure 0 0 0) 0x7f2678084070 con 0x7f26800fd490 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2684b7b700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2670062c90 0x7f2670065150 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 rx=0x7f2674009760 tx=0x7f267400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f267dffb700 10 client.37250.objecter ms_handle_connect 0x7f2670062c90 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2686ddf700 10 client.37250.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 1 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6801/2103743338 conn(0x7f26800ffdb0 0x7f26801001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f26800f9dc0 osd=0 3 2024-02-13T00:45:01.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f26800f9dc0 osd=0 3 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f26800f9dc0 osd=0 3 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 1 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f26800f9dc0 osd=0 3 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 0 1 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f26800f9dc0 osd=0 4 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2686ddf700 10 client.37250.objecter _send_command 1 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26801006c0 con 0x7f26800ffdb0 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f267ffff700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6801/2103743338 conn(0x7f26800ffdb0 0x7f26801001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f267dffb700 10 client.37250.objecter ms_handle_connect 0x7f26800ffdb0 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.046+0000 7f267ffff700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6801/2103743338 conn(0x7f26800ffdb0 0x7f26801001d0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.047+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f26801006c0 con 0x7f26800ffdb0 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.047+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.047+0000 7f267dffb700 10 client.37250.objecter _finish_command 1 = system:0 2024-02-13T00:45:01.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.047+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f26800f9dc0 osd=0 4 2024-02-13T00:45:01.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.047+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 0 1 2024-02-13T00:45:01.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.062+0000 7f2686ddf700 10 client.37250.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:01.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.062+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 2 2024-02-13T00:45:01.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.062+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f26800f9dc0 osd=0 3 2024-02-13T00:45:01.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.062+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f26800f9dc0 osd=0 3 2024-02-13T00:45:01.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.062+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f26800f9dc0 osd=0 3 2024-02-13T00:45:01.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.062+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 2 2024-02-13T00:45:01.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.062+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f26800f9dc0 osd=0 3 2024-02-13T00:45:01.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.062+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 0 2 2024-02-13T00:45:01.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.062+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f26800f9dc0 osd=0 4 2024-02-13T00:45:01.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.062+0000 7f2686ddf700 10 client.37250.objecter _send_command 2 2024-02-13T00:45:01.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.062+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2680100860 con 0x7f26800ffdb0 2024-02-13T00:45:01.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.063+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f2680100860 con 0x7f26800ffdb0 2024-02-13T00:45:01.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.063+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:01.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.063+0000 7f267dffb700 10 client.37250.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.063+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f26800f9dc0 osd=0 4 2024-02-13T00:45:01.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.063+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 0 2 2024-02-13T00:45:01.068 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:45:01.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2686ddf700 10 client.37250.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:01.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 3 2024-02-13T00:45:01.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6805/150599780 conn(0x7f2680028590 0x7f268011b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:01.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f2680061a70 osd=1 3 2024-02-13T00:45:01.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680061a70 osd=1 3 2024-02-13T00:45:01.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f2680061a70 osd=1 3 2024-02-13T00:45:01.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 3 2024-02-13T00:45:01.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f2680061a70 osd=1 3 2024-02-13T00:45:01.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 1 3 2024-02-13T00:45:01.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680061a70 osd=1 4 2024-02-13T00:45:01.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2686ddf700 10 client.37250.objecter _send_command 3 2024-02-13T00:45:01.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2680061be0 con 0x7f2680028590 2024-02-13T00:45:01.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2684b7b700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6805/150599780 conn(0x7f2680028590 0x7f268011b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:01.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f2684b7b700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6805/150599780 conn(0x7f2680028590 0x7f268011b260 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:01.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.064+0000 7f267dffb700 10 client.37250.objecter ms_handle_connect 0x7f2680028590 2024-02-13T00:45:01.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.065+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2680061be0 con 0x7f2680028590 2024-02-13T00:45:01.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.065+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:45:01.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.065+0000 7f267dffb700 10 client.37250.objecter _finish_command 3 = system:0 2024-02-13T00:45:01.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.065+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f2680061a70 osd=1 4 2024-02-13T00:45:01.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.065+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 1 3 2024-02-13T00:45:01.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.079+0000 7f2686ddf700 10 client.37250.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:01.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.079+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 4 2024-02-13T00:45:01.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.079+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f2680061a70 osd=1 3 2024-02-13T00:45:01.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.079+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680061a70 osd=1 3 2024-02-13T00:45:01.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.079+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f2680061a70 osd=1 3 2024-02-13T00:45:01.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.079+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 4 2024-02-13T00:45:01.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.079+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f2680061a70 osd=1 3 2024-02-13T00:45:01.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.079+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 1 4 2024-02-13T00:45:01.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.079+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680061a70 osd=1 4 2024-02-13T00:45:01.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.079+0000 7f2686ddf700 10 client.37250.objecter _send_command 4 2024-02-13T00:45:01.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.079+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f268005a0e0 con 0x7f2680028590 2024-02-13T00:45:01.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f268005a0e0 con 0x7f2680028590 2024-02-13T00:45:01.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:01.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f267dffb700 10 client.37250.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f2680061a70 osd=1 4 2024-02-13T00:45:01.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 1 4 2024-02-13T00:45:01.086 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:45:01.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f2686ddf700 10 client.37250.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:01.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 5 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6809/3689719376 conn(0x7f2680059570 0x7f268011e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f2680059400 osd=2 3 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680059400 osd=2 3 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f2680059400 osd=2 3 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 5 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f2680059400 osd=2 3 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 2 5 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680059400 osd=2 4 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f2686ddf700 10 client.37250.objecter _send_command 5 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26800281b0 con 0x7f2680059570 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f267ffff700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6809/3689719376 conn(0x7f2680059570 0x7f268011e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f267dffb700 10 client.37250.objecter ms_handle_connect 0x7f2680059570 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.081+0000 7f267ffff700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6809/3689719376 conn(0x7f2680059570 0x7f268011e8e0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.082+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f26800281b0 con 0x7f2680059570 2024-02-13T00:45:01.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.082+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:45:01.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.082+0000 7f267dffb700 10 client.37250.objecter _finish_command 5 = system:0 2024-02-13T00:45:01.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.082+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f2680059400 osd=2 4 2024-02-13T00:45:01.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.082+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 2 5 2024-02-13T00:45:01.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.096+0000 7f2686ddf700 10 client.37250.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:01.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.096+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 6 2024-02-13T00:45:01.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.096+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f2680059400 osd=2 3 2024-02-13T00:45:01.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.096+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680059400 osd=2 3 2024-02-13T00:45:01.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.096+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f2680059400 osd=2 3 2024-02-13T00:45:01.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.096+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 6 2024-02-13T00:45:01.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.096+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f2680059400 osd=2 3 2024-02-13T00:45:01.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.096+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 2 6 2024-02-13T00:45:01.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.096+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680059400 osd=2 4 2024-02-13T00:45:01.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.096+0000 7f2686ddf700 10 client.37250.objecter _send_command 6 2024-02-13T00:45:01.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.096+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f26800283d0 con 0x7f2680059570 2024-02-13T00:45:01.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f26800283d0 con 0x7f2680059570 2024-02-13T00:45:01.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:01.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f267dffb700 10 client.37250.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f2680059400 osd=2 4 2024-02-13T00:45:01.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 2 6 2024-02-13T00:45:01.103 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:45:01.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f2686ddf700 10 client.37250.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:01.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 7 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6813/604971011 conn(0x7f268011c430 0x7f2680121f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f26800278f0 osd=3 3 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f26800278f0 osd=3 3 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f26800278f0 osd=3 3 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 7 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f26800278f0 osd=3 3 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 3 7 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f26800278f0 osd=3 4 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f2686ddf700 10 client.37250.objecter _send_command 7 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2680124bb0 con 0x7f268011c430 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f268537c700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6813/604971011 conn(0x7f268011c430 0x7f2680121f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f267dffb700 10 client.37250.objecter ms_handle_connect 0x7f268011c430 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.098+0000 7f268537c700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6813/604971011 conn(0x7f268011c430 0x7f2680121f60 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:01.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.099+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2680124bb0 con 0x7f268011c430 2024-02-13T00:45:01.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.099+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:45:01.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.099+0000 7f267dffb700 10 client.37250.objecter _finish_command 7 = system:0 2024-02-13T00:45:01.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.099+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f26800278f0 osd=3 4 2024-02-13T00:45:01.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.099+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 3 7 2024-02-13T00:45:01.106 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:01.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:01.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:01.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.113+0000 7f2686ddf700 10 client.37250.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:01.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.113+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 8 2024-02-13T00:45:01.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.113+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f26800278f0 osd=3 3 2024-02-13T00:45:01.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.113+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f26800278f0 osd=3 3 2024-02-13T00:45:01.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.113+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f26800278f0 osd=3 3 2024-02-13T00:45:01.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.113+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 8 2024-02-13T00:45:01.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.113+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f26800278f0 osd=3 3 2024-02-13T00:45:01.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.113+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 3 8 2024-02-13T00:45:01.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.113+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f26800278f0 osd=3 4 2024-02-13T00:45:01.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.113+0000 7f2686ddf700 10 client.37250.objecter _send_command 8 2024-02-13T00:45:01.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.113+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f268011b990 con 0x7f268011c430 2024-02-13T00:45:01.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f268011b990 con 0x7f268011c430 2024-02-13T00:45:01.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:01.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f267dffb700 10 client.37250.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f26800278f0 osd=3 4 2024-02-13T00:45:01.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 3 8 2024-02-13T00:45:01.120 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:45:01.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2686ddf700 10 client.37250.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 9 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6800/3344263388 conn(0x7f268011bb30 0x7f26801235d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f268011b990 osd=4 3 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268011b990 osd=4 3 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f268011b990 osd=4 3 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 9 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f268011b990 osd=4 3 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 4 9 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268011b990 osd=4 4 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2686ddf700 10 client.37250.objecter _send_command 9 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f268011fca0 con 0x7f268011bb30 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2684b7b700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6800/3344263388 conn(0x7f268011bb30 0x7f26801235d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f267dffb700 10 client.37250.objecter ms_handle_connect 0x7f268011bb30 2024-02-13T00:45:01.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.115+0000 7f2684b7b700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6800/3344263388 conn(0x7f268011bb30 0x7f26801235d0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:01.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.116+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f268011fca0 con 0x7f268011bb30 2024-02-13T00:45:01.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.116+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:45:01.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.116+0000 7f267dffb700 10 client.37250.objecter _finish_command 9 = system:0 2024-02-13T00:45:01.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.116+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f268011b990 osd=4 4 2024-02-13T00:45:01.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.116+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 4 9 2024-02-13T00:45:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.131+0000 7f2686ddf700 10 client.37250.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.131+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 10 2024-02-13T00:45:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.131+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f268011b990 osd=4 3 2024-02-13T00:45:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.131+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268011b990 osd=4 3 2024-02-13T00:45:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.131+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f268011b990 osd=4 3 2024-02-13T00:45:01.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.131+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 10 2024-02-13T00:45:01.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.131+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f268011b990 osd=4 3 2024-02-13T00:45:01.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.131+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 4 10 2024-02-13T00:45:01.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.131+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268011b990 osd=4 4 2024-02-13T00:45:01.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.131+0000 7f2686ddf700 10 client.37250.objecter _send_command 10 2024-02-13T00:45:01.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.131+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f268011fca0 con 0x7f268011bb30 2024-02-13T00:45:01.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f268011fca0 con 0x7f268011bb30 2024-02-13T00:45:01.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:01.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f267dffb700 10 client.37250.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f268011b990 osd=4 4 2024-02-13T00:45:01.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 4 10 2024-02-13T00:45:01.140 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:45:01.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f2686ddf700 10 client.37250.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 11 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6804/2212296748 conn(0x7f2680123fd0 0x7f2680124410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f268011fca0 osd=5 3 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268011fca0 osd=5 3 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f268011fca0 osd=5 3 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 11 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f268011fca0 osd=5 3 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 5 11 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268011fca0 osd=5 4 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f2686ddf700 10 client.37250.objecter _send_command 11 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f268012b4a0 con 0x7f2680123fd0 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.133+0000 7f267ffff700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6804/2212296748 conn(0x7f2680123fd0 0x7f2680124410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.134+0000 7f267dffb700 10 client.37250.objecter ms_handle_connect 0x7f2680123fd0 2024-02-13T00:45:01.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.134+0000 7f267ffff700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6804/2212296748 conn(0x7f2680123fd0 0x7f2680124410 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:01.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.134+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f268012b4a0 con 0x7f2680123fd0 2024-02-13T00:45:01.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.134+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:45:01.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.134+0000 7f267dffb700 10 client.37250.objecter _finish_command 11 = system:0 2024-02-13T00:45:01.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.134+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f268011fca0 osd=5 4 2024-02-13T00:45:01.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.134+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 5 11 2024-02-13T00:45:01.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.148+0000 7f2686ddf700 10 client.37250.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:01.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.148+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 12 2024-02-13T00:45:01.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.148+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f268011fca0 osd=5 3 2024-02-13T00:45:01.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.148+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268011fca0 osd=5 3 2024-02-13T00:45:01.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.148+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f268011fca0 osd=5 3 2024-02-13T00:45:01.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.148+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 12 2024-02-13T00:45:01.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.148+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f268011fca0 osd=5 3 2024-02-13T00:45:01.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.148+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 5 12 2024-02-13T00:45:01.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.148+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268011fca0 osd=5 4 2024-02-13T00:45:01.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.148+0000 7f2686ddf700 10 client.37250.objecter _send_command 12 2024-02-13T00:45:01.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.148+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2680128d40 con 0x7f2680123fd0 2024-02-13T00:45:01.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f2680128d40 con 0x7f2680123fd0 2024-02-13T00:45:01.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:01.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f267dffb700 10 client.37250.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f268011fca0 osd=5 4 2024-02-13T00:45:01.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 5 12 2024-02-13T00:45:01.155 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f2686ddf700 10 client.37250.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 13 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6808/1506640341 conn(0x7f2680129420 0x7f268012b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f2680128d40 osd=6 3 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680128d40 osd=6 3 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f2680128d40 osd=6 3 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 13 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f2680128d40 osd=6 3 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 6 13 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680128d40 osd=6 4 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f2686ddf700 10 client.37250.objecter _send_command 13 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.150+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f268012e600 con 0x7f2680129420 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.151+0000 7f268537c700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6808/1506640341 conn(0x7f2680129420 0x7f268012b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:01.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.151+0000 7f267dffb700 10 client.37250.objecter ms_handle_connect 0x7f2680129420 2024-02-13T00:45:01.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.151+0000 7f268537c700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6808/1506640341 conn(0x7f2680129420 0x7f268012b830 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:01.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.152+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f268012e600 con 0x7f2680129420 2024-02-13T00:45:01.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.152+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:45:01.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.152+0000 7f267dffb700 10 client.37250.objecter _finish_command 13 = system:0 2024-02-13T00:45:01.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.152+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f2680128d40 osd=6 4 2024-02-13T00:45:01.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.152+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 6 13 2024-02-13T00:45:01.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.166+0000 7f2686ddf700 10 client.37250.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:01.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.166+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 14 2024-02-13T00:45:01.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.166+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f2680128d40 osd=6 3 2024-02-13T00:45:01.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.166+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680128d40 osd=6 3 2024-02-13T00:45:01.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.166+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f2680128d40 osd=6 3 2024-02-13T00:45:01.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.166+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 14 2024-02-13T00:45:01.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.166+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f2680128d40 osd=6 3 2024-02-13T00:45:01.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.166+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 6 14 2024-02-13T00:45:01.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.166+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680128d40 osd=6 4 2024-02-13T00:45:01.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.166+0000 7f2686ddf700 10 client.37250.objecter _send_command 14 2024-02-13T00:45:01.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.166+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f268012e600 con 0x7f2680129420 2024-02-13T00:45:01.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.167+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f268012e600 con 0x7f2680129420 2024-02-13T00:45:01.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.167+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:01.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.167+0000 7f267dffb700 10 client.37250.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.167+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f2680128d40 osd=6 4 2024-02-13T00:45:01.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.167+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 6 14 2024-02-13T00:45:01.172 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:45:01.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2686ddf700 10 client.37250.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:01.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 15 2024-02-13T00:45:01.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6812/1964192772 conn(0x7f268012d900 0x7f268012dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:01.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f268012e600 osd=7 3 2024-02-13T00:45:01.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268012e600 osd=7 3 2024-02-13T00:45:01.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f268012e600 osd=7 3 2024-02-13T00:45:01.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 15 2024-02-13T00:45:01.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f268012e600 osd=7 3 2024-02-13T00:45:01.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 7 15 2024-02-13T00:45:01.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268012e600 osd=7 4 2024-02-13T00:45:01.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2686ddf700 10 client.37250.objecter _send_command 15 2024-02-13T00:45:01.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f268012e3a0 con 0x7f268012d900 2024-02-13T00:45:01.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2684b7b700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6812/1964192772 conn(0x7f268012d900 0x7f268012dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:01.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f267dffb700 10 client.37250.objecter ms_handle_connect 0x7f268012d900 2024-02-13T00:45:01.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.168+0000 7f2684b7b700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6812/1964192772 conn(0x7f268012d900 0x7f268012dd80 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:01.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.169+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f268012e3a0 con 0x7f268012d900 2024-02-13T00:45:01.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.169+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:45:01.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.169+0000 7f267dffb700 10 client.37250.objecter _finish_command 15 = system:0 2024-02-13T00:45:01.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.169+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f268012e600 osd=7 4 2024-02-13T00:45:01.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.169+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 7 15 2024-02-13T00:45:01.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.183+0000 7f2686ddf700 10 client.37250.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:01.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.183+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign -1 16 2024-02-13T00:45:01.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.183+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f268012e600 osd=7 3 2024-02-13T00:45:01.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.183+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268012e600 osd=7 3 2024-02-13T00:45:01.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.183+0000 7f2686ddf700 20 client.37250.objecter _get_session s=0x7f268012e600 osd=7 3 2024-02-13T00:45:01.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.183+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_remove -1 16 2024-02-13T00:45:01.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.183+0000 7f2686ddf700 20 client.37250.objecter get_session s=0x7f268012e600 osd=7 3 2024-02-13T00:45:01.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.183+0000 7f2686ddf700 15 client.37250.objecter _session_command_op_assign 7 16 2024-02-13T00:45:01.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.183+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268012e600 osd=7 4 2024-02-13T00:45:01.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.183+0000 7f2686ddf700 10 client.37250.objecter _send_command 16 2024-02-13T00:45:01.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.183+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f268012e3a0 con 0x7f268012d900 2024-02-13T00:45:01.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f267dffb700 1 -- 172.21.15.162:0/2807854082 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f268012e3a0 con 0x7f268012d900 2024-02-13T00:45:01.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f267dffb700 10 client.37250.objecter ms_dispatch 0x7f2680001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:01.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f267dffb700 10 client.37250.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f267dffb700 20 client.37250.objecter put_session s=0x7f268012e600 osd=7 4 2024-02-13T00:45:01.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f267dffb700 15 client.37250.objecter _session_command_op_remove 7 16 2024-02-13T00:45:01.190 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:45:01.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 10 librados: watch_flush enter 2024-02-13T00:45:01.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 10 librados: watch_flush exit 2024-02-13T00:45:01.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 10 client.37250.objecter close_session for osd.0 2024-02-13T00:45:01.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6801/2103743338 conn(0x7f26800ffdb0 msgr2=0x7f26801001d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6801/2103743338 conn(0x7f26800ffdb0 0x7f26801001d0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f26800f9dc0 osd=0 1 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 10 client.37250.objecter close_session for osd.1 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6805/150599780 conn(0x7f2680028590 msgr2=0x7f268011b260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6805/150599780 conn(0x7f2680028590 0x7f268011b260 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680061a70 osd=1 1 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 10 client.37250.objecter close_session for osd.2 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6809/3689719376 conn(0x7f2680059570 msgr2=0x7f268011e8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6809/3689719376 conn(0x7f2680059570 0x7f268011e8e0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680059400 osd=2 1 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 10 client.37250.objecter close_session for osd.3 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6813/604971011 conn(0x7f268011c430 msgr2=0x7f2680121f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6813/604971011 conn(0x7f268011c430 0x7f2680121f60 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f268537c700 1 -- 172.21.15.162:0/2807854082 reap_dead start 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f26800278f0 osd=3 1 2024-02-13T00:45:01.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 10 client.37250.objecter close_session for osd.4 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6800/3344263388 conn(0x7f268011bb30 msgr2=0x7f26801235d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6800/3344263388 conn(0x7f268011bb30 0x7f26801235d0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f268537c700 1 -- 172.21.15.162:0/2807854082 reap_dead start 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268011b990 osd=4 1 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 10 client.37250.objecter close_session for osd.5 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6804/2212296748 conn(0x7f2680123fd0 msgr2=0x7f2680124410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6804/2212296748 conn(0x7f2680123fd0 0x7f2680124410 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268011fca0 osd=5 1 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 10 client.37250.objecter close_session for osd.6 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6808/1506640341 conn(0x7f2680129420 msgr2=0x7f268012b830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6808/1506640341 conn(0x7f2680129420 0x7f268012b830 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f2680128d40 osd=6 1 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 10 client.37250.objecter close_session for osd.7 2024-02-13T00:45:01.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6812/1964192772 conn(0x7f268012d900 msgr2=0x7f268012dd80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.181:6812/1964192772 conn(0x7f268012d900 0x7f268012dd80 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:01.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 20 client.37250.objecter put_session s=0x7f268012e600 osd=7 1 2024-02-13T00:45:01.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 20 client.37250.objecter shutdown clearing up homeless session... 2024-02-13T00:45:01.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 10 client.37250.objecter successfully canceled tick 2024-02-13T00:45:01.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2670062c90 msgr2=0x7f2670065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2670062c90 0x7f2670065150 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 rx=0x7f2674009760 tx=0x7f267400b040).stop 2024-02-13T00:45:01.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26800fd490 msgr2=0x7f26800f9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:01.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f2686ddf700 1 --2- 172.21.15.162:0/2807854082 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26800fd490 0x7f26800f9310 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f267800c870 tx=0x7f2678007dd0).stop 2024-02-13T00:45:01.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.185+0000 7f268537c700 1 -- 172.21.15.162:0/2807854082 reap_dead start 2024-02-13T00:45:01.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.186+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 shutdown_connections 2024-02-13T00:45:01.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.186+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 >> 172.21.15.162:0/2807854082 conn(0x7f268006eb90 msgr2=0x7f2680060ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:01.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.186+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 shutdown_connections 2024-02-13T00:45:01.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.186+0000 7f2686ddf700 1 -- 172.21.15.162:0/2807854082 wait complete. 2024-02-13T00:45:01.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:01.186+0000 7f2686ddf700 1 librados: shutdown 2024-02-13T00:45:01.195 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.195 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.195 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.195 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.195 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.195 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.195 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.195 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:45:01.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:01.236 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:01.263 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:01.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:01.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:01.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:01.397 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:45:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-13T00:45:01.397 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:45:01.405 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:01.432 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:45:01.459 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:01.461 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:59.960751+0000 osd.0 (osd.0) 120 : cluster [DBG] 28.16 deep-scrub starts 2024-02-13T00:45:01.461 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:44:59.963612+0000 osd.0 (osd.0) 121 : cluster [DBG] 28.16 deep-scrub ok 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:00.148437+0000 osd.5 (osd.5) 148 : cluster [DBG] 18.9 deep-scrub starts 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:00.151319+0000 osd.5 (osd.5) 149 : cluster [DBG] 18.9 deep-scrub ok 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:00.246306+0000 osd.1 (osd.1) 141 : cluster [DBG] 18.6 scrub starts 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:00.247710+0000 osd.1 (osd.1) 142 : cluster [DBG] 18.6 scrub ok 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:00.308677+0000 osd.6 (osd.6) 77 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:00.311553+0000 osd.6 (osd.6) 78 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:00.398831+0000 mgr.y (mgr.14544) 313 : cluster [DBG] pgmap v488: 291 pgs: 32 unknown, 259 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 6.0 KiB/s wr, 8 op/s; 179 B/s, 12 objects/s recovering 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:00.453283+0000 mon.b (mon.1) 518 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:01.047350+0000 mon.c (mon.2) 334 : audit [DBG] from='client.? 172.21.15.162:0/2807854082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:01.378543+0000 mon.a (mon.0) 2206 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:01.383218+0000 mon.a (mon.0) 2207 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:01.383327+0000 mon.a (mon.0) 2208 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-45", "mode": "writeback"}]': finished 2024-02-13T00:45:01.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:01.383392+0000 mon.a (mon.0) 2209 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-13T00:45:01.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:01.383446+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e346: 8 total, 8 up, 6 in 2024-02-13T00:45:01.463 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:01.392877+0000 mon.a (mon.0) 2211 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/1089 objects degraded (2.847%), 1 pg degraded) 2024-02-13T00:45:01.463 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:59.960751+0000 osd.0 (osd.0) 120 : cluster [DBG] 28.16 deep-scrub starts 2024-02-13T00:45:01.463 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:44:59.963612+0000 osd.0 (osd.0) 121 : cluster [DBG] 28.16 deep-scrub ok 2024-02-13T00:45:01.463 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:00.148437+0000 osd.5 (osd.5) 148 : cluster [DBG] 18.9 deep-scrub starts 2024-02-13T00:45:01.463 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:00.151319+0000 osd.5 (osd.5) 149 : cluster [DBG] 18.9 deep-scrub ok 2024-02-13T00:45:01.463 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:00.246306+0000 osd.1 (osd.1) 141 : cluster [DBG] 18.6 scrub starts 2024-02-13T00:45:01.463 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:00.247710+0000 osd.1 (osd.1) 142 : cluster [DBG] 18.6 scrub ok 2024-02-13T00:45:01.463 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:00.308677+0000 osd.6 (osd.6) 77 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:45:01.463 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:00.311553+0000 osd.6 (osd.6) 78 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:45:01.463 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:00.398831+0000 mgr.y (mgr.14544) 313 : cluster [DBG] pgmap v488: 291 pgs: 32 unknown, 259 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 6.0 KiB/s wr, 8 op/s; 179 B/s, 12 objects/s recovering 2024-02-13T00:45:01.464 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:00.453283+0000 mon.b (mon.1) 518 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:01.464 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:01.047350+0000 mon.c (mon.2) 334 : audit [DBG] from='client.? 172.21.15.162:0/2807854082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:01.464 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:01.378543+0000 mon.a (mon.0) 2206 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:45:01.464 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:01.383218+0000 mon.a (mon.0) 2207 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:01.464 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:01.383327+0000 mon.a (mon.0) 2208 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-45", "mode": "writeback"}]': finished 2024-02-13T00:45:01.464 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:01.383392+0000 mon.a (mon.0) 2209 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-13T00:45:01.464 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:01.383446+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e346: 8 total, 8 up, 6 in 2024-02-13T00:45:01.464 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:01.392877+0000 mon.a (mon.0) 2211 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/1089 objects degraded (2.847%), 1 pg degraded) 2024-02-13T00:45:01.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:01.398084+0000 mon.b (mon.1) 519 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:45:01.464 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:01.398147+0000 osd.4 (osd.4) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:01.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:01.398494+0000 osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:01.465 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:01.403528+0000 mon.a (mon.0) 2212 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:45:01.465 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:01.398084+0000 mon.b (mon.1) 519 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:45:01.465 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:01.398147+0000 osd.4 (osd.4) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:01.465 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:01.398494+0000 osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:01.465 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:01.403528+0000 mon.a (mon.0) 2212 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:45:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:01.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:01.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:45:01.595 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:59.960751+0000 osd.0 (osd.0) 120 : cluster [DBG] 28.16 deep-scrub starts 2024-02-13T00:45:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:44:59.963612+0000 osd.0 (osd.0) 121 : cluster [DBG] 28.16 deep-scrub ok 2024-02-13T00:45:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:00.148437+0000 osd.5 (osd.5) 148 : cluster [DBG] 18.9 deep-scrub starts 2024-02-13T00:45:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:00.151319+0000 osd.5 (osd.5) 149 : cluster [DBG] 18.9 deep-scrub ok 2024-02-13T00:45:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:00.246306+0000 osd.1 (osd.1) 141 : cluster [DBG] 18.6 scrub starts 2024-02-13T00:45:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:00.247710+0000 osd.1 (osd.1) 142 : cluster [DBG] 18.6 scrub ok 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:00.308677+0000 osd.6 (osd.6) 77 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:00.311553+0000 osd.6 (osd.6) 78 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:00.398831+0000 mgr.y (mgr.14544) 313 : cluster [DBG] pgmap v488: 291 pgs: 32 unknown, 259 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 6.0 KiB/s wr, 8 op/s; 179 B/s, 12 objects/s recovering 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:00.453283+0000 mon.b (mon.1) 518 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:01.047350+0000 mon.c (mon.2) 334 : audit [DBG] from='client.? 172.21.15.162:0/2807854082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:01.378543+0000 mon.a (mon.0) 2206 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:01.383218+0000 mon.a (mon.0) 2207 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-83985-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:01.383327+0000 mon.a (mon.0) 2208 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-45", "mode": "writeback"}]': finished 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:01.383392+0000 mon.a (mon.0) 2209 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:01.383446+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e346: 8 total, 8 up, 6 in 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:01.392877+0000 mon.a (mon.0) 2211 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/1089 objects degraded (2.847%), 1 pg degraded) 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:01.398084+0000 mon.b (mon.1) 519 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:01.398147+0000 osd.4 (osd.4) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:01.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:01.398494+0000 osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:01.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:01.403528+0000 mon.a (mon.0) 2212 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:45:01.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:45:01.944 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:45:01.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:01.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:45:02.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:02.079 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:02.161 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:02.528 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:02.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:02.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:02.659 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:02.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:01.178106+0000 osd.5 (osd.5) 150 : cluster [DBG] 28.1d scrub starts 2024-02-13T00:45:02.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:01.179021+0000 osd.5 (osd.5) 151 : cluster [DBG] 28.1d scrub ok 2024-02-13T00:45:02.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:01.454266+0000 mon.b (mon.1) 520 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:02.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:02.384266+0000 mon.a (mon.0) 2213 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:45:02.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:02.384357+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e347: 8 total, 8 up, 6 in 2024-02-13T00:45:02.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:02.385743+0000 mon.b (mon.1) 521 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:45:02.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:02.388082+0000 mon.a (mon.0) 2215 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:45:02.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:02.406267+0000 mon.c (mon.2) 335 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:02.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:02.406593+0000 mon.a (mon.0) 2216 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:02.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:02.407724+0000 mon.c (mon.2) 336 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:02.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:02.408009+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:02.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:02.408449+0000 mon.c (mon.2) 337 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-83985-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:02.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:02.408700+0000 mon.a (mon.0) 2218 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-83985-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:02.677 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:01.178106+0000 osd.5 (osd.5) 150 : cluster [DBG] 28.1d scrub starts 2024-02-13T00:45:02.677 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:01.179021+0000 osd.5 (osd.5) 151 : cluster [DBG] 28.1d scrub ok 2024-02-13T00:45:02.677 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:01.454266+0000 mon.b (mon.1) 520 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:02.384266+0000 mon.a (mon.0) 2213 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:45:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:02.384357+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e347: 8 total, 8 up, 6 in 2024-02-13T00:45:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:02.385743+0000 mon.b (mon.1) 521 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:45:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:02.388082+0000 mon.a (mon.0) 2215 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:45:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:02.406267+0000 mon.c (mon.2) 335 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:02.406593+0000 mon.a (mon.0) 2216 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:02.407724+0000 mon.c (mon.2) 336 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:02.408009+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:02.408449+0000 mon.c (mon.2) 337 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-83985-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:02.408700+0000 mon.a (mon.0) 2218 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-83985-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:01.178106+0000 osd.5 (osd.5) 150 : cluster [DBG] 28.1d scrub starts 2024-02-13T00:45:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:01.179021+0000 osd.5 (osd.5) 151 : cluster [DBG] 28.1d scrub ok 2024-02-13T00:45:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:01.454266+0000 mon.b (mon.1) 520 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:02.384266+0000 mon.a (mon.0) 2213 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:45:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:02.384357+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e347: 8 total, 8 up, 6 in 2024-02-13T00:45:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:02.385743+0000 mon.b (mon.1) 521 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:45:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:02.388082+0000 mon.a (mon.0) 2215 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:45:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:02.406267+0000 mon.c (mon.2) 335 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:02.406593+0000 mon.a (mon.0) 2216 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:02.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:02.407724+0000 mon.c (mon.2) 336 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:02.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:02.408009+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:02.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:02.408449+0000 mon.c (mon.2) 337 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-83985-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:02.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:02.408700+0000 mon.a (mon.0) 2218 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-83985-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:03.508 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:03.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:03.519 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:02.228279+0000 osd.5 (osd.5) 152 : cluster [DBG] 61.7 scrub starts 2024-02-13T00:45:03.519 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:02.230147+0000 osd.5 (osd.5) 153 : cluster [DBG] 61.7 scrub ok 2024-02-13T00:45:03.519 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:02.247400+0000 osd.1 (osd.1) 143 : cluster [DBG] 18.e scrub starts 2024-02-13T00:45:03.519 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:02.248216+0000 osd.1 (osd.1) 144 : cluster [DBG] 18.e scrub ok 2024-02-13T00:45:03.519 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:02.399239+0000 mgr.y (mgr.14544) 314 : cluster [DBG] pgmap v491: 259 pgs: 259 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 6.0 KiB/s wr, 8 op/s; 179 B/s, 12 objects/s recovering 2024-02-13T00:45:03.519 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:02.455033+0000 mon.b (mon.1) 522 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:03.519 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:03.393613+0000 mon.a (mon.0) 2219 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:45:03.519 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:03.393717+0000 mon.a (mon.0) 2220 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-83985-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:03.519 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:03.393788+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e348: 8 total, 8 up, 6 in 2024-02-13T00:45:03.519 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:03.397130+0000 mon.c (mon.2) 338 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-83985-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:03.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:03.397450+0000 mon.b (mon.1) 523 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:45:03.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:03.397571+0000 mon.a (mon.0) 2222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-83985-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:03.520 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:03.398318+0000 mon.a (mon.0) 2223 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:45:03.612 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:03.641 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:03.661 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:03.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:03.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:02.228279+0000 osd.5 (osd.5) 152 : cluster [DBG] 61.7 scrub starts 2024-02-13T00:45:03.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:02.230147+0000 osd.5 (osd.5) 153 : cluster [DBG] 61.7 scrub ok 2024-02-13T00:45:03.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:02.247400+0000 osd.1 (osd.1) 143 : cluster [DBG] 18.e scrub starts 2024-02-13T00:45:03.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:02.248216+0000 osd.1 (osd.1) 144 : cluster [DBG] 18.e scrub ok 2024-02-13T00:45:03.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:02.399239+0000 mgr.y (mgr.14544) 314 : cluster [DBG] pgmap v491: 259 pgs: 259 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 6.0 KiB/s wr, 8 op/s; 179 B/s, 12 objects/s recovering 2024-02-13T00:45:03.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:02.455033+0000 mon.b (mon.1) 522 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:03.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:03.393613+0000 mon.a (mon.0) 2219 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:45:03.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:03.393717+0000 mon.a (mon.0) 2220 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-83985-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:03.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:03.393788+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e348: 8 total, 8 up, 6 in 2024-02-13T00:45:03.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:03.397130+0000 mon.c (mon.2) 338 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-83985-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:03.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:03.397450+0000 mon.b (mon.1) 523 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:45:03.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:03.397571+0000 mon.a (mon.0) 2222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-83985-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:03.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:03.398318+0000 mon.a (mon.0) 2223 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:45:03.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:02.228279+0000 osd.5 (osd.5) 152 : cluster [DBG] 61.7 scrub starts 2024-02-13T00:45:03.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:02.230147+0000 osd.5 (osd.5) 153 : cluster [DBG] 61.7 scrub ok 2024-02-13T00:45:03.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:02.247400+0000 osd.1 (osd.1) 143 : cluster [DBG] 18.e scrub starts 2024-02-13T00:45:03.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:02.248216+0000 osd.1 (osd.1) 144 : cluster [DBG] 18.e scrub ok 2024-02-13T00:45:03.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:02.399239+0000 mgr.y (mgr.14544) 314 : cluster [DBG] pgmap v491: 259 pgs: 259 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 6.0 KiB/s wr, 8 op/s; 179 B/s, 12 objects/s recovering 2024-02-13T00:45:03.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:02.455033+0000 mon.b (mon.1) 522 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:03.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:03.393613+0000 mon.a (mon.0) 2219 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:45:03.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:03.393717+0000 mon.a (mon.0) 2220 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-83985-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:03.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:03.393788+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e348: 8 total, 8 up, 6 in 2024-02-13T00:45:03.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:03.397130+0000 mon.c (mon.2) 338 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-83985-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:03.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:03.397450+0000 mon.b (mon.1) 523 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:45:03.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:03.397571+0000 mon.a (mon.0) 2222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-83985-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:03.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:03.398318+0000 mon.a (mon.0) 2223 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:45:03.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:03.791 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:45:03.791 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:45:03.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:45:03.791 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:45:03.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.791 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24748.0:2 18.6 18:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:45:03.791 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.502560+0000", 2024-02-13T00:45:03.791 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 242.28274203699999, 2024-02-13T00:45:03.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.238140929, 2024-02-13T00:45:03.791 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.791 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24748", 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2831164779", 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.502560+0000", 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.502560+0000", 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.502557+0000", 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0930999999999999e-05 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.502568+0000", 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3305999999999999e-05 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.502582+0000", 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2409999999999995e-06 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.502590+0000", 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012993519999999999 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.503889+0000", 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6420000000000007e-06 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.503897+0000", 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12742637300000001 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.631323+0000", 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000110917 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.631434+0000", 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013954000000000001 2024-02-13T00:45:03.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.631574+0000", 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005929938 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.637504+0000", 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011763342 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.649267+0000", 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.091359251000000002 2024-02-13T00:45:03.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.740626+0000", 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8794000000000002e-05 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.740665+0000", 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5556999999999999e-05 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.740701+0000", 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.238140929 2024-02-13T00:45:03.797 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24727.0:2 9.13 9:ce900539:test-rados-api-smithi162-84097-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.504427+0000", 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 242.280874385, 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094734222000000007, 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24727", 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/1035949604", 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2 2024-02-13T00:45:03.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.504427+0000", 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.504427+0000", 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.504425+0000", 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1124999999999999e-05 2024-02-13T00:45:03.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.504436+0000", 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4211e-05 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.504451+0000", 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4950000000000008e-06 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.504459+0000", 2024-02-13T00:45:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9593999999999998e-05 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.504529+0000", 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6680000000000001e-06 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.504536+0000", 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.075833254000000003 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.580370+0000", 2024-02-13T00:45:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013298100000000001 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.580503+0000", 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015746299999999999 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 5,6", 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.580660+0000", 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013761282999999999 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.594421+0000", 2024-02-13T00:45:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044847089999999999 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.598906+0000", 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000170347 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.599076+0000", 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5948999999999997e-05 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.599122+0000", 2024-02-13T00:45:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9201e-05 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.599161+0000", 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094734222000000007 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14880.0:2 19.9 19:903c8d0a:test-rados-api-smithi162-84033-2::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.582470+0000", 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 242.20283128, 2024-02-13T00:45:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.158011654, 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14880", 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/79393202", 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.582470+0000", 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.582470+0000", 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.582469+0000", 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6120000000000005e-06 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.582478+0000", 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0898e-05 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.582489+0000", 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9039999999999992e-06 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.582497+0000", 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014164700000000001 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.582639+0000", 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7079999999999996e-06 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.582643+0000", 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.05075727 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.633401+0000", 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3500999999999998e-05 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.633494+0000", 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018896900000000001 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2", 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.633683+0000", 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0083379539999999999 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642021+0000", 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.007796529 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.649817+0000", 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.090580167000000003 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.740398+0000", 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.702e-05 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.740445+0000", 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7345000000000002e-05 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.740482+0000", 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.158011654 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14880.0:3 19.9 19:903c8d0a:test-rados-api-smithi162-84033-2::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.741199+0000", 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 242.04410253, 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021832567000000001, 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14880", 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/79393202", 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 3 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.811 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.741199+0000", 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.741199+0000", 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.741198+0000", 2024-02-13T00:45:03.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4100000000000002e-06 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.741205+0000", 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.275e-05 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.741218+0000", 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3509999999999998e-06 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.741224+0000", 2024-02-13T00:45:03.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1019999999999997e-05 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.741265+0000", 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4678000000000004e-05 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.741360+0000", 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021671896 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.763032+0000", 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021832567000000001 2024-02-13T00:45:03.814 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24727.0:10 9.d 9:b19c9ec2:test-rados-api-smithi162-84097-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.746740+0000", 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 242.038561945, 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016428484, 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24727", 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/1035949604", 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 10 2024-02-13T00:45:03.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.746740+0000", 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.746740+0000", 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.746739+0000", 2024-02-13T00:45:03.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0939999999999996e-06 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.746745+0000", 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7459999999999995e-06 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.746755+0000", 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0150000000000003e-06 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.746760+0000", 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0880999999999998e-05 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.746790+0000", 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1297000000000004e-05 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.746872+0000", 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4835999999999998e-05 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.746967+0000", 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033577800000000001 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.747302+0000", 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018959900000000001 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.747492+0000", 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015593769 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.763086+0000", 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4109000000000001e-05 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.763130+0000", 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8327000000000001e-05 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.763168+0000", 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016428484 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.820 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24748.0:859 18.13 18:c8000000::::head [pgnls start_epoch 107] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e107)", 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:02.497297+0000", 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 241.288005086, 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.043455169000000002, 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24748", 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2831164779", 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 859 2024-02-13T00:45:03.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.497297+0000", 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.497297+0000", 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.497294+0000", 2024-02-13T00:45:03.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8779999999999993e-06 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.497304+0000", 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.042926706000000002 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.540231+0000", 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4079999999999998e-06 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.540239+0000", 2024-02-13T00:45:03.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00038690699999999997 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.540626+0000", 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2855000000000003e-05 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.540659+0000", 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2670000000000006e-05 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.540752+0000", 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.043455169000000002 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24797.0:2 29.17 29:ee53464d:::PoolCreationRaceObj:head [read 0~100] snapc 0=[] ondisk+read+known_if_redirected e106)", 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:02.508998+0000", 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 241.27630354999999, 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.031108295000000001, 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24797", 2024-02-13T00:45:03.825 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/4056203793", 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.508998+0000", 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.508998+0000", 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.508996+0000", 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1669999999999999e-06 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.509004+0000", 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1097999999999999e-05 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.509015+0000", 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0749999999999998e-06 2024-02-13T00:45:03.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.509023+0000", 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.030999894 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.540023+0000", 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3009000000000004e-05 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.540106+0000", 2024-02-13T00:45:03.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.031108295000000001 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24724.0:8 15.15 15:ad0c1634:test-rados-api-smithi162-84025-4::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:02.797840+0000", 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 240.98746119399999, 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.052489598999999998, 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24724", 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/832124837", 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 8 2024-02-13T00:45:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.797840+0000", 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.797840+0000", 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.797839+0000", 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.229e-06 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.797846+0000", 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0203999999999999e-05 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.797856+0000", 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6609999999999999e-06 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.797860+0000", 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8081999999999997e-05 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.797918+0000", 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1037000000000004e-05 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.797990+0000", 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000132049 2024-02-13T00:45:03.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1", 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.798122+0000", 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027042499999999997 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.798392+0000", 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.023598917000000001 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.821991+0000", 2024-02-13T00:45:03.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.028238057 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850229+0000", 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0513000000000002e-05 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850289+0000", 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0598e-05 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:02.850330+0000", 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.052489598999999998 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2059 20.6 e109/105 20:661c4854:test-rados-api-smithi162-84049-16::big:head v 109'1, mlcod=0'0)", 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.270703+0000", 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 239.51459829800001, 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.062060650000000002, 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.270703+0000", 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.270703+0000", 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.270701+0000", 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025154453 2024-02-13T00:45:03.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.295856+0000", 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5077999999999999e-05 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.295881+0000", 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0148e-05 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.295891+0000", 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7239999999999997e-05 2024-02-13T00:45:03.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.295988+0000", 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9803999999999999e-05 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.296008+0000", 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.036716217000000002 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.332724+0000", 2024-02-13T00:45:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9715999999999997e-05 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.332764+0000", 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.062060650000000002 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.3.0:340 20.17 e109/105 -22:e88892cc:::temp_20.17_0_14313_1:head v 0'0, mlcod=0'0)", 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.344527+0000", 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 239.440775092, 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.029699428999999999, 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.839 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.344527+0000", 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.344527+0000", 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.344526+0000", 2024-02-13T00:45:03.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017162958999999998 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.361689+0000", 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1387e-05 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.361710+0000", 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8300000000000008e-06 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.361720+0000", 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5889000000000005e-05 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.361796+0000", 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8464000000000001e-05 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.361814+0000", 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012371013 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.374185+0000", 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0908000000000002e-05 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.374226+0000", 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.029699428999999999 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.3.0:341 20.17 e109/105 -22:e88892cc:::temp_20.17_0_14313_1:head v 0'0, mlcod=0'0)", 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.362551+0000", 2024-02-13T00:45:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 239.422750912, 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.018100394999999998, 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.362551+0000", 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.362551+0000", 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:45:03.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.362549+0000", 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0082291680000000002 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.370778+0000", 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8583000000000001e-05 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.370797+0000", 2024-02-13T00:45:03.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1430000000000004e-06 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.370806+0000", 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9081e-05 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.370855+0000", 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0055999999999999e-05 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.370875+0000", 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0097518550000000002 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.380627+0000", 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4459000000000001e-05 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.380651+0000", 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.018100394999999998 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.3.0:342 20.17 e109/105 -22:e88892cc:::temp_20.17_0_14313_1:head v 0'0, mlcod=0'0)", 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.382287+0000", 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 239.403014708, 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.022976297999999999, 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.382287+0000", 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.382287+0000", 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.382285+0000", 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012993635999999999 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.395279+0000", 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7807000000000001e-05 2024-02-13T00:45:03.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.395297+0000", 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3359999999999997e-06 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.395304+0000", 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6358000000000003e-05 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.395381+0000", 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7651000000000001e-05 2024-02-13T00:45:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.395398+0000", 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0098245079999999992 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.405223+0000", 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0543000000000001e-05 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.405263+0000", 2024-02-13T00:45:03.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.022976297999999999 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2069 20.1b e109/105 20:de4afda3:test-rados-api-smithi162-84049-17::big:head v 109'2, mlcod=0'0)", 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.641902+0000", 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 239.14339963099999, 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099833579000000006, 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.641902+0000", 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.641902+0000", 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.641901+0000", 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049776414999999997 2024-02-13T00:45:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.691678+0000", 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7985999999999999e-05 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.691706+0000", 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7790000000000001e-06 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.691714+0000", 2024-02-13T00:45:03.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4009000000000002e-05 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.691788+0000", 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0298999999999999e-05 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.691819+0000", 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049884388000000002 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.741703+0000", 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2441000000000003e-05 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.741736+0000", 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099833579000000006 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2074 20.1b e109/105 20:de8a7420:test-rados-api-smithi162-84049-17::big3:head v 109'3, mlcod=107'1)", 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.870265+0000", 2024-02-13T00:45:03.856 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 238.915036405, 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.093686992999999996, 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.870265+0000", 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.870265+0000", 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:45:03.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.870263+0000", 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.038709031999999997 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.908972+0000", 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3113e-05 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.908995+0000", 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7369999999999995e-06 2024-02-13T00:45:03.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.909004+0000", 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.371e-05 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.909048+0000", 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5187000000000001e-05 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.909073+0000", 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.054821231999999998 2024-02-13T00:45:03.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.963894+0000", 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8007000000000002e-05 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.963952+0000", 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.093686992999999996 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.860 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.29851.0:2 72.6 72:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:24.983112+0000", 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 218.80218960100001, 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.056797175999999998, 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.29851", 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3524561227", 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983112+0000", 2024-02-13T00:45:03.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983112+0000", 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983111+0000", 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.57e-06 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983116+0000", 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8830000000000006e-06 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983124+0000", 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5900000000000002e-06 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983132+0000", 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8680000000000001e-05 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983180+0000", 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1040000000000001e-06 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983187+0000", 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.037057128000000002 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.020244+0000", 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108541 2024-02-13T00:45:03.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.020353+0000", 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000146265 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.020499+0000", 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001090929 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.021590+0000", 2024-02-13T00:45:03.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000159708 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.021750+0000", 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.018075356000000001 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.039825+0000", 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2278000000000003e-05 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.039878+0000", 2024-02-13T00:45:03.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1665000000000003e-05 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.039909+0000", 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.056797175999999998 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.19047.0:1 78.4s0 78:2fee9e45:test-rados-api-smithi162-84073-10::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:28.235955+0000", 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 215.549346472, 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0634874270000001, 2024-02-13T00:45:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.19047", 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3520430640", 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.235955+0000", 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.235955+0000", 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999967 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.235952+0000", 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3636e-05 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.235965+0000", 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8223000000000002e-05 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.235984+0000", 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2216e-05 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.235996+0000", 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012535500000000001 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.236121+0000", 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.976e-06 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.236127+0000", 2024-02-13T00:45:03.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0553561279999999 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:29.291483+0000", 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000100157 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:29.291583+0000", 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027292499999999998 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:45:03.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:29.291856+0000", 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00043762800000000002 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:29.292294+0000", 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071099309999999999 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:29.299404+0000", 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8661999999999997e-05 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:29.299443+0000", 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0634874270000001 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25162.0:110 73.1f 73:fec11214:test-rados-api-smithi162-84057-10::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e135)", 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:30.310984+0000", 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 213.474317531, 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.050975678000000003, 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:45:03.873 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25162", 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 110 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.310984+0000", 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.310984+0000", 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.310982+0000", 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4608e-05 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.310997+0000", 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3023999999999999e-05 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.311010+0000", 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8660000000000002e-06 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.311016+0000", 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7796999999999997e-05 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.311054+0000", 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020388700000000001 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.311258+0000", 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.045311125000000001 2024-02-13T00:45:03.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.356569+0000", 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9762000000000001e-05 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.356639+0000", 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030073029999999998 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.359646+0000", 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7300999999999999e-05 2024-02-13T00:45:03.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.359703+0000", 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022566830000000002 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.361960+0000", 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.050975678000000003 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.878 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.43639.0:2 102.d 102:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:54.329646+0000", 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 189.455656073, 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.034329884999999997, 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.43639", 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/891139537", 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329646+0000", 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329646+0000", 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329644+0000", 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6475000000000001e-05 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329660+0000", 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0299000000000001e-05 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329671+0000", 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.25e-06 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329678+0000", 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3374000000000001e-05 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329731+0000", 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000122748 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329854+0000", 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000161397 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6", 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.330015+0000", 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039438199999999998 2024-02-13T00:45:03.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.330410+0000", 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014673799999999999 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.330556+0000", 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.033313427999999999 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.363870+0000", 2024-02-13T00:45:03.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2893000000000003e-05 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.363923+0000", 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2667999999999998e-05 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.363975+0000", 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.034329884999999997 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPushReply(20.2 172/167 [PushReplyOp(20:40ede249:test-rados-api-smithi162-84049-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head)])", 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:42:07.547345+0000", 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 176.23795660100001, 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.082099599999999995, 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.547345+0000", 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.547345+0000", 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.547344+0000", 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4939999999999992e-06 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.547354+0000", 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9225999999999999e-05 2024-02-13T00:45:03.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.547373+0000", 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0322999999999999e-05 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.547383+0000", 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.081862927000000002 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.629246+0000", 2024-02-13T00:45:03.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019853000000000001 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.629445+0000", 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.082099599999999995 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.25162.0:4733 191.6 e335/323 191:602f83fe:::foo:head v 335'11, mlcod=333'9)", 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:50.443481+0000", 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 13.341820636, 2024-02-13T00:45:03.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.024653034000000001, 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443481+0000", 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443481+0000", 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443481+0000", 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7215999999999999e-05 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443498+0000", 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1439999999999997e-06 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443505+0000", 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.409e-06 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443511+0000", 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039672500000000001 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443908+0000", 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9888999999999999e-05 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443928+0000", 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.024168541000000002 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.468097+0000", 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7477e-05 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.468134+0000", 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.024653034000000001 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:45:03.892 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:45:03.893 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:03.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:45:04.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.011+0000 7f18bce11700 1 -- 172.21.15.162:0/3470361762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18b80fe590 msgr2=0x7f18b80f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:04.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.011+0000 7f18bce11700 1 --2- 172.21.15.162:0/3470361762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18b80fe590 0x7f18b80f7dc0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 rx=0x7f18ac004870 tx=0x7f18ac009cb0).stop 2024-02-13T00:45:04.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.011+0000 7f18bce11700 1 -- 172.21.15.162:0/3470361762 shutdown_connections 2024-02-13T00:45:04.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.011+0000 7f18bce11700 1 --2- 172.21.15.162:0/3470361762 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18b80f84a0 0x7f18b8104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:04.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.011+0000 7f18bce11700 1 --2- 172.21.15.162:0/3470361762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18b80fdc30 0x7f18b80fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:04.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.011+0000 7f18bce11700 1 --2- 172.21.15.162:0/3470361762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18b80fe590 0x7f18b80f7dc0 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:04.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.011+0000 7f18bce11700 1 -- 172.21.15.162:0/3470361762 >> 172.21.15.162:0/3470361762 conn(0x7f18b80f3880 msgr2=0x7f18b80f5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.011+0000 7f18bce11700 1 -- 172.21.15.162:0/3470361762 shutdown_connections 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.011+0000 7f18bce11700 1 -- 172.21.15.162:0/3470361762 wait complete. 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.012+0000 7f18bce11700 1 librados: starting msgr at 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.012+0000 7f18bce11700 1 librados: starting objecter 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.012+0000 7f18bce11700 1 Processor -- start 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.012+0000 7f18bce11700 1 -- start start 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.012+0000 7f18bce11700 1 librados: setting wanted keys 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.012+0000 7f18bce11700 1 librados: calling monclient init 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.012+0000 7f18bce11700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18b80f84a0 0x7f18b806b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.012+0000 7f18bce11700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18b80fdc30 0x7f18b806b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.012+0000 7f18bce11700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18b80fe590 0x7f18b806bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.012+0000 7f18bce11700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8116550 con 0x7f18b80f84a0 2024-02-13T00:45:04.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.012+0000 7f18bce11700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b81166c0 con 0x7f18b80fdc30 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.012+0000 7f18bce11700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8116880 con 0x7f18b80fe590 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b6d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18b80fe590 0x7f18b806bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18b80f84a0 0x7f18b806b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b6d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18b80fe590 0x7f18b806bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:57572/0 (socket says 172.21.15.162:57572) 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b6d9d700 1 -- 172.21.15.162:0/43155753 learned_addr learned my addr 172.21.15.162:0/43155753 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18b80f84a0 0x7f18b806b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:56736/0 (socket says 172.21.15.162:56736) 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b5d9b700 1 --2- 172.21.15.162:0/43155753 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18b80fdc30 0x7f18b806b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b659c700 1 -- 172.21.15.162:0/43155753 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18b80fe590 msgr2=0x7f18b806bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b659c700 1 --2- 172.21.15.162:0/43155753 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18b80fe590 0x7f18b806bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b659c700 1 -- 172.21.15.162:0/43155753 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18b80fdc30 msgr2=0x7f18b806b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b659c700 1 --2- 172.21.15.162:0/43155753 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18b80fdc30 0x7f18b806b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b659c700 1 -- 172.21.15.162:0/43155753 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b00097c0 con 0x7f18b80f84a0 2024-02-13T00:45:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b6d9d700 1 --2- 172.21.15.162:0/43155753 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18b80fe590 0x7f18b806bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18b659c700 1 --2- 172.21.15.162:0/43155753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18b80f84a0 0x7f18b806b170 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f18a800c810 tx=0x7f18a8007ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18a77fe700 10 client.?.objecter ms_handle_connect 0x7f18b80f84a0 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18a77fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18a77fe700 1 -- 172.21.15.162:0/43155753 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a800e4a0 con 0x7f18b80f84a0 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18bce11700 1 -- 172.21.15.162:0/43155753 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac005040 con 0x7f18b80f84a0 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18a77fe700 1 -- 172.21.15.162:0/43155753 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18a800ead0 con 0x7f18b80f84a0 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18a77fe700 1 -- 172.21.15.162:0/43155753 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8012a50 con 0x7f18b80f84a0 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18bce11700 10 client.27153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18bce11700 1 -- 172.21.15.162:0/43155753 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b8116f90 con 0x7f18b80f84a0 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.013+0000 7f18bce11700 1 librados: init done 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.014+0000 7f18bce11700 1 -- 172.21.15.162:0/43155753 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b805fae0 con 0x7f18b80f84a0 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.015+0000 7f18a77fe700 1 -- 172.21.15.162:0/43155753 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f18a800ec40 con 0x7f18b80f84a0 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.015+0000 7f18a77fe700 1 --2- 172.21.15.162:0/43155753 >> v2:172.21.15.162:6800/2866534053 conn(0x7f18a0062c90 0x7f18a0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:04.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.015+0000 7f18a77fe700 1 -- 172.21.15.162:0/43155753 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(348..348 src has 52..348) v4 ==== 8571+0+0 (secure 0 0 0) 0x7f18a807cb90 con 0x7f18b80f84a0 2024-02-13T00:45:04.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.015+0000 7f18a77fe700 10 client.27153.objecter ms_dispatch 0x7f18b8001550 osd_map(348..348 src has 52..348) v4 2024-02-13T00:45:04.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.015+0000 7f18a77fe700 3 client.27153.objecter handle_osd_map got epochs [348,348] > 0 2024-02-13T00:45:04.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.015+0000 7f18a77fe700 3 client.27153.objecter handle_osd_map decoding full epoch 348 2024-02-13T00:45:04.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.015+0000 7f18b5d9b700 1 --2- 172.21.15.162:0/43155753 >> v2:172.21.15.162:6800/2866534053 conn(0x7f18a0062c90 0x7f18a0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:04.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.015+0000 7f18a77fe700 20 client.27153.objecter dump_active .. 0 homeless 2024-02-13T00:45:04.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.015+0000 7f18a77fe700 10 client.27153.objecter ms_handle_connect 0x7f18a0062c90 2024-02-13T00:45:04.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.015+0000 7f18b5d9b700 1 --2- 172.21.15.162:0/43155753 >> v2:172.21.15.162:6800/2866534053 conn(0x7f18a0062c90 0x7f18a0065150 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 rx=0x7f18ac0043d0 tx=0x7f18ac00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:04.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.020+0000 7f18a77fe700 1 -- 172.21.15.162:0/43155753 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f18a8048f60 con 0x7f18b80f84a0 2024-02-13T00:45:04.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:04.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:04.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:45:04.180 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:04.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:45:04.201 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:04.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:45:04.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.225+0000 7f18bce11700 1 -- 172.21.15.162:0/43155753 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f18b806c650 con 0x7f18b80f84a0 2024-02-13T00:45:04.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:04.331 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:04.435 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:04.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:04.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:04.529 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-13T00:45:04.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:04.600 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:04.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.655+0000 7f18a77fe700 1 -- 172.21.15.162:0/43155753 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v349) v1 ==== 87+0+0 (secure 0 0 0) 0x7f18a8017030 con 0x7f18b80f84a0 2024-02-13T00:45:04.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 10 librados: watch_flush enter 2024-02-13T00:45:04.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 10 librados: watch_flush exit 2024-02-13T00:45:04.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 20 client.27153.objecter shutdown clearing up homeless session... 2024-02-13T00:45:04.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 10 client.27153.objecter successfully canceled tick 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 -- 172.21.15.162:0/43155753 >> v2:172.21.15.162:6800/2866534053 conn(0x7f18a0062c90 msgr2=0x7f18a0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 --2- 172.21.15.162:0/43155753 >> v2:172.21.15.162:6800/2866534053 conn(0x7f18a0062c90 0x7f18a0065150 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 rx=0x7f18ac0043d0 tx=0x7f18ac00b040).stop 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 -- 172.21.15.162:0/43155753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18b80f84a0 msgr2=0x7f18b806b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 --2- 172.21.15.162:0/43155753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18b80f84a0 0x7f18b806b170 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f18a800c810 tx=0x7f18a8007ee0).stop 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 -- 172.21.15.162:0/43155753 shutdown_connections 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 --2- 172.21.15.162:0/43155753 >> v2:172.21.15.162:6800/2866534053 conn(0x7f18a0062c90 0x7f18a0065150 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 --2- 172.21.15.162:0/43155753 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18b80fe590 0x7f18b806bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 --2- 172.21.15.162:0/43155753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18b80f84a0 0x7f18b806b170 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 --2- 172.21.15.162:0/43155753 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18b80fdc30 0x7f18b806b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 -- 172.21.15.162:0/43155753 >> 172.21.15.162:0/43155753 conn(0x7f18b80f3880 msgr2=0x7f18b80f5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 -- 172.21.15.162:0/43155753 shutdown_connections 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 -- 172.21.15.162:0/43155753 wait complete. 2024-02-13T00:45:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:04.658+0000 7f18bce11700 1 librados: shutdown 2024-02-13T00:45:04.664 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:45:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:03.455963+0000 mon.b (mon.1) 524 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:04.227992+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? 172.21.15.162:0/43155753' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:45:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:04.334857+0000 mon.a (mon.0) 2225 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:04.391740+0000 mon.a (mon.0) 2226 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:45:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:04.391773+0000 mon.a (mon.0) 2227 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:45:05.039 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:03.455963+0000 mon.b (mon.1) 524 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:05.039 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:04.227992+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? 172.21.15.162:0/43155753' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:45:05.039 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:04.334857+0000 mon.a (mon.0) 2225 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:05.039 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:04.391740+0000 mon.a (mon.0) 2226 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:45:05.039 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:04.391773+0000 mon.a (mon.0) 2227 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:45:05.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:03.455963+0000 mon.b (mon.1) 524 : audit [DBG] from='client.? 172.21.15.162:0/2831164779' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-13T00:45:05.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:04.227992+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? 172.21.15.162:0/43155753' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:45:05.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:04.334857+0000 mon.a (mon.0) 2225 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:05.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:04.391740+0000 mon.a (mon.0) 2226 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:45:05.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:04.391773+0000 mon.a (mon.0) 2227 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:45:06.033 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:04.399855+0000 mgr.y (mgr.14544) 315 : cluster [DBG] pgmap v493: 258 pgs: 258 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:45:06.034 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:04.403528+0000 mon.c (mon.2) 339 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-13T00:45:06.034 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:04.657203+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:45:06.034 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:04.657283+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? 172.21.15.162:0/43155753' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:45:06.034 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:04.657326+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e349: 8 total, 8 up, 6 in 2024-02-13T00:45:06.034 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:04.658353+0000 mon.b (mon.1) 525 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:45:06.034 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:04.659516+0000 mon.a (mon.0) 2231 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-13T00:45:06.034 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:04.659816+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:45:06.034 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.362331+0000 mon.c (mon.2) 340 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]: dispatch 2024-02-13T00:45:06.034 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.362558+0000 mon.c (mon.2) 341 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-02-13T00:45:06.034 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.362754+0000 mon.a (mon.0) 2233 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]: dispatch 2024-02-13T00:45:06.034 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.362831+0000 mon.c (mon.2) 342 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.363094+0000 mon.c (mon.2) 343 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.363178+0000 mon.a (mon.0) 2234 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.363335+0000 mon.c (mon.2) 344 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 6]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.363541+0000 mon.a (mon.0) 2235 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.363584+0000 mon.c (mon.2) 345 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.363847+0000 mon.a (mon.0) 2236 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.363855+0000 mon.c (mon.2) 346 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.12", "id": [0, 4]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.364076+0000 mon.c (mon.2) 347 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "195.6", "id": [4, 6]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.364086+0000 mon.a (mon.0) 2237 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 6]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.364296+0000 mon.a (mon.0) 2238 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.364517+0000 mon.a (mon.0) 2239 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.12", "id": [0, 4]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.364686+0000 mon.a (mon.0) 2240 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "195.6", "id": [4, 6]}]: dispatch 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.403269+0000 mon.a (mon.0) 2241 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-83985-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-83985-60"}]': finished 2024-02-13T00:45:06.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.403426+0000 mon.a (mon.0) 2242 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-13T00:45:06.036 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.403505+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-13T00:45:06.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:04.399855+0000 mgr.y (mgr.14544) 315 : cluster [DBG] pgmap v493: 258 pgs: 258 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:45:06.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:04.403528+0000 mon.c (mon.2) 339 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-13T00:45:06.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:04.657203+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:45:06.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:04.657283+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? 172.21.15.162:0/43155753' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:45:06.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:04.657326+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e349: 8 total, 8 up, 6 in 2024-02-13T00:45:06.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:04.658353+0000 mon.b (mon.1) 525 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:45:06.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:04.659516+0000 mon.a (mon.0) 2231 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-13T00:45:06.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:04.659816+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.362331+0000 mon.c (mon.2) 340 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.362558+0000 mon.c (mon.2) 341 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.362754+0000 mon.a (mon.0) 2233 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.362831+0000 mon.c (mon.2) 342 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.363094+0000 mon.c (mon.2) 343 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.363178+0000 mon.a (mon.0) 2234 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.363335+0000 mon.c (mon.2) 344 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 6]}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.363541+0000 mon.a (mon.0) 2235 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.363584+0000 mon.c (mon.2) 345 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.363847+0000 mon.a (mon.0) 2236 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.363855+0000 mon.c (mon.2) 346 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.12", "id": [0, 4]}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.364076+0000 mon.c (mon.2) 347 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "195.6", "id": [4, 6]}]: dispatch 2024-02-13T00:45:06.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.364086+0000 mon.a (mon.0) 2237 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 6]}]: dispatch 2024-02-13T00:45:06.038 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.364296+0000 mon.a (mon.0) 2238 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]}]: dispatch 2024-02-13T00:45:06.038 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.364517+0000 mon.a (mon.0) 2239 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.12", "id": [0, 4]}]: dispatch 2024-02-13T00:45:06.038 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.364686+0000 mon.a (mon.0) 2240 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "195.6", "id": [4, 6]}]: dispatch 2024-02-13T00:45:06.038 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.403269+0000 mon.a (mon.0) 2241 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-83985-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-83985-60"}]': finished 2024-02-13T00:45:06.038 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.403426+0000 mon.a (mon.0) 2242 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-13T00:45:06.038 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.403505+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-13T00:45:06.038 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.403590+0000 mon.a (mon.0) 2244 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]': finished 2024-02-13T00:45:06.038 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.403665+0000 mon.a (mon.0) 2245 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]': finished 2024-02-13T00:45:06.038 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.403737+0000 mon.a (mon.0) 2246 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-02-13T00:45:06.038 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.403816+0000 mon.a (mon.0) 2247 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]': finished 2024-02-13T00:45:06.038 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.403882+0000 mon.a (mon.0) 2248 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 6]}]': finished 2024-02-13T00:45:06.038 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.403951+0000 mon.a (mon.0) 2249 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]}]': finished 2024-02-13T00:45:06.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.404017+0000 mon.a (mon.0) 2250 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.12", "id": [0, 4]}]': finished 2024-02-13T00:45:06.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.404089+0000 mon.a (mon.0) 2251 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "195.6", "id": [4, 6]}]': finished 2024-02-13T00:45:06.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:05.404171+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e350: 8 total, 8 up, 6 in 2024-02-13T00:45:06.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.417297+0000 mon.b (mon.1) 526 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-13T00:45:06.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.417883+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-13T00:45:06.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.439593+0000 mon.a (mon.0) 2254 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:06.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.440499+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:06.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.440805+0000 mon.a (mon.0) 2256 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi162-84017-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:06.039 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.403590+0000 mon.a (mon.0) 2244 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]': finished 2024-02-13T00:45:06.039 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.403665+0000 mon.a (mon.0) 2245 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]': finished 2024-02-13T00:45:06.039 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.403737+0000 mon.a (mon.0) 2246 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-02-13T00:45:06.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.403816+0000 mon.a (mon.0) 2247 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]': finished 2024-02-13T00:45:06.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.403882+0000 mon.a (mon.0) 2248 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 6]}]': finished 2024-02-13T00:45:06.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.403951+0000 mon.a (mon.0) 2249 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]}]': finished 2024-02-13T00:45:06.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.404017+0000 mon.a (mon.0) 2250 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.12", "id": [0, 4]}]': finished 2024-02-13T00:45:06.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.404089+0000 mon.a (mon.0) 2251 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "195.6", "id": [4, 6]}]': finished 2024-02-13T00:45:06.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:05.404171+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e350: 8 total, 8 up, 6 in 2024-02-13T00:45:06.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.417297+0000 mon.b (mon.1) 526 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-13T00:45:06.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.417883+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-13T00:45:06.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.439593+0000 mon.a (mon.0) 2254 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:06.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.440499+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:06.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.440805+0000 mon.a (mon.0) 2256 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi162-84017-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:06.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:45:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:04.399855+0000 mgr.y (mgr.14544) 315 : cluster [DBG] pgmap v493: 258 pgs: 258 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:45:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:04.403528+0000 mon.c (mon.2) 339 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-13T00:45:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:04.657203+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:45:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:04.657283+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? 172.21.15.162:0/43155753' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:45:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:04.657326+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e349: 8 total, 8 up, 6 in 2024-02-13T00:45:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:04.658353+0000 mon.b (mon.1) 525 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:45:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:04.659516+0000 mon.a (mon.0) 2231 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:04.659816+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.362331+0000 mon.c (mon.2) 340 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.362558+0000 mon.c (mon.2) 341 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.362754+0000 mon.a (mon.0) 2233 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.362831+0000 mon.c (mon.2) 342 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.363094+0000 mon.c (mon.2) 343 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.363178+0000 mon.a (mon.0) 2234 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.363335+0000 mon.c (mon.2) 344 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 6]}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.363541+0000 mon.a (mon.0) 2235 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.363584+0000 mon.c (mon.2) 345 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.363847+0000 mon.a (mon.0) 2236 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.363855+0000 mon.c (mon.2) 346 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.12", "id": [0, 4]}]: dispatch 2024-02-13T00:45:06.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.364076+0000 mon.c (mon.2) 347 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "195.6", "id": [4, 6]}]: dispatch 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.364086+0000 mon.a (mon.0) 2237 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 6]}]: dispatch 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.364296+0000 mon.a (mon.0) 2238 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]}]: dispatch 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.364517+0000 mon.a (mon.0) 2239 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.12", "id": [0, 4]}]: dispatch 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.364686+0000 mon.a (mon.0) 2240 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "195.6", "id": [4, 6]}]: dispatch 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.403269+0000 mon.a (mon.0) 2241 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-83985-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-83985-60"}]': finished 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.403426+0000 mon.a (mon.0) 2242 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-84017-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.403505+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.403590+0000 mon.a (mon.0) 2244 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 4]}]': finished 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.403665+0000 mon.a (mon.0) 2245 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]': finished 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.403737+0000 mon.a (mon.0) 2246 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.403816+0000 mon.a (mon.0) 2247 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]': finished 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.403882+0000 mon.a (mon.0) 2248 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 6]}]': finished 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.403951+0000 mon.a (mon.0) 2249 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]}]': finished 2024-02-13T00:45:06.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.404017+0000 mon.a (mon.0) 2250 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.12", "id": [0, 4]}]': finished 2024-02-13T00:45:06.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.404089+0000 mon.a (mon.0) 2251 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "195.6", "id": [4, 6]}]': finished 2024-02-13T00:45:06.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:05.404171+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e350: 8 total, 8 up, 6 in 2024-02-13T00:45:06.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.417297+0000 mon.b (mon.1) 526 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-13T00:45:06.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.417883+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-13T00:45:06.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.439593+0000 mon.a (mon.0) 2254 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:06.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.440499+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:06.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.440805+0000 mon.a (mon.0) 2256 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi162-84017-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:06.284 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:06.287 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:45:06] "GET /metrics HTTP/1.1" 200 189456 "" "Prometheus/2.33.4" 2024-02-13T00:45:06.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:06.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:06.303 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:06.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:45:06.427 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:06.578 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:45:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:06.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:45:06.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:06.713 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:06.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:05.371488+0000 osd.7 (osd.7) 102 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:06.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:05.371856+0000 osd.7 (osd.7) 103 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:06.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:05.548041+0000 mgr.y (mgr.14544) 316 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:06.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:06.407029+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-13T00:45:06.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:06.407138+0000 mon.a (mon.0) 2258 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi162-84017-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:06.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:06.407201+0000 mon.a (mon.0) 2259 : cluster [DBG] osdmap e351: 8 total, 8 up, 6 in 2024-02-13T00:45:06.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:06.408140+0000 mon.a (mon.0) 2260 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi162-84017-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:06.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:06.411443+0000 mon.b (mon.1) 527 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-13T00:45:06.975 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:06.413853+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-13T00:45:06.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:05.371488+0000 osd.7 (osd.7) 102 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:06.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:05.371856+0000 osd.7 (osd.7) 103 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:06.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:05.548041+0000 mgr.y (mgr.14544) 316 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:06.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:06.407029+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-13T00:45:06.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:06.407138+0000 mon.a (mon.0) 2258 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi162-84017-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:06.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:06.407201+0000 mon.a (mon.0) 2259 : cluster [DBG] osdmap e351: 8 total, 8 up, 6 in 2024-02-13T00:45:06.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:06.408140+0000 mon.a (mon.0) 2260 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi162-84017-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:06.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:06.411443+0000 mon.b (mon.1) 527 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-13T00:45:06.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:06.413853+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-13T00:45:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:05.371488+0000 osd.7 (osd.7) 102 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:05.371856+0000 osd.7 (osd.7) 103 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:05.548041+0000 mgr.y (mgr.14544) 316 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:06.407029+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-13T00:45:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:06.407138+0000 mon.a (mon.0) 2258 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi162-84017-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:06.407201+0000 mon.a (mon.0) 2259 : cluster [DBG] osdmap e351: 8 total, 8 up, 6 in 2024-02-13T00:45:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:06.408140+0000 mon.a (mon.0) 2260 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi162-84017-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:06.411443+0000 mon.b (mon.1) 527 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-13T00:45:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:06.413853+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-13T00:45:07.161 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:07.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:07.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:07.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:07.295 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:07.314 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:07.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:07.418 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:07.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.801+0000 7fb514aa9700 1 -- 172.21.15.162:0/3421552053 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb510102270 msgr2=0x7fb510104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.801+0000 7fb514aa9700 1 --2- 172.21.15.162:0/3421552053 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb510102270 0x7fb510104660 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 rx=0x7fb508009be0 tx=0x7fb508004d10).stop 2024-02-13T00:45:07.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 -- 172.21.15.162:0/3421552053 shutdown_connections 2024-02-13T00:45:07.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 --2- 172.21.15.162:0/3421552053 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb510102270 0x7fb510104660 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 --2- 172.21.15.162:0/3421552053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5100f8d10 0x7fb5100f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 --2- 172.21.15.162:0/3421552053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5100f95e0 0x7fb510101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 -- 172.21.15.162:0/3421552053 >> 172.21.15.162:0/3421552053 conn(0x7fb51006eb90 msgr2=0x7fb5100f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:07.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 -- 172.21.15.162:0/3421552053 shutdown_connections 2024-02-13T00:45:07.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 -- 172.21.15.162:0/3421552053 wait complete. 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 librados: starting msgr at 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 librados: starting objecter 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 Processor -- start 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 -- start start 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 librados: setting wanted keys 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.802+0000 7fb514aa9700 1 librados: calling monclient init 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb514aa9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5100f8d10 0x7fb51010e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb514aa9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb5100f95e0 0x7fb51010e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb514aa9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb510102270 0x7fb510108290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb514aa9700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb510108780 con 0x7fb5100f8d10 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb514aa9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5101088f0 con 0x7fb510102270 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb514aa9700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb510108ab0 con 0x7fb5100f95e0 2024-02-13T00:45:07.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb50dd9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb5100f95e0 0x7fb51010e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb50dd9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb5100f95e0 0x7fb51010e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:56860/0 (socket says 172.21.15.162:56860) 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb50e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5100f8d10 0x7fb51010e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb50dd9b700 1 -- 172.21.15.162:0/168161704 learned_addr learned my addr 172.21.15.162:0/168161704 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb50ed9d700 1 --2- 172.21.15.162:0/168161704 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb510102270 0x7fb510108290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb50dd9b700 1 -- 172.21.15.162:0/168161704 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb510102270 msgr2=0x7fb510108290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb50dd9b700 1 --2- 172.21.15.162:0/168161704 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb510102270 0x7fb510108290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb50dd9b700 1 -- 172.21.15.162:0/168161704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5100f8d10 msgr2=0x7fb51010e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb50dd9b700 1 --2- 172.21.15.162:0/168161704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5100f8d10 0x7fb51010e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb50dd9b700 1 -- 172.21.15.162:0/168161704 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5080097c0 con 0x7fb5100f95e0 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb50e59c700 1 --2- 172.21.15.162:0/168161704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5100f8d10 0x7fb51010e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb50dd9b700 1 --2- 172.21.15.162:0/168161704 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb5100f95e0 0x7fb51010e5e0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7fb50400e970 tx=0x7fb50400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb4ff7fe700 10 client.?.objecter ms_handle_connect 0x7fb5100f95e0 2024-02-13T00:45:07.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb4ff7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb504004660 con 0x7fb5100f95e0 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb50400f040 con 0x7fb5100f95e0 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb510108d90 con 0x7fb5100f95e0 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb504014790 con 0x7fb5100f95e0 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb514aa9700 10 client.37274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb510109110 con 0x7fb5100f95e0 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.803+0000 7fb514aa9700 1 librados: init done 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.804+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb510107170 con 0x7fb5100f95e0 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fb5040090d0 con 0x7fb5100f95e0 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb4ff7fe700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb4f8062c90 0x7fb4f8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(351..351 src has 52..351) v4 ==== 9256+0+0 (secure 0 0 0) 0x7fb50407d710 con 0x7fb5100f95e0 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 osd_map(351..351 src has 52..351) v4 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb4ff7fe700 3 client.37274.objecter handle_osd_map got epochs [351,351] > 0 2024-02-13T00:45:07.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb4ff7fe700 3 client.37274.objecter handle_osd_map decoding full epoch 351 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb50e59c700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb4f8062c90 0x7fb4f8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb4ff7fe700 20 client.37274.objecter dump_active .. 0 homeless 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v351) v1 ==== 54+0+16 (secure 0 0 0) 0x7fb50407dbb0 con 0x7fb5100f95e0 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb4ff7fe700 10 client.37274.objecter ms_handle_connect 0x7fb4f8062c90 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb50e59c700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb4f8062c90 0x7fb4f8065150 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 rx=0x7fb500007bf0 tx=0x7fb50000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb514aa9700 10 client.37274.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.805+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 1 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb510059990 0x7fb510059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510109770 osd=0 3 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510109770 osd=0 3 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510109770 osd=0 3 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 1 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510109770 osd=0 3 2024-02-13T00:45:07.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 0 1 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510109770 osd=0 4 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb514aa9700 10 client.37274.objecter _send_command 1 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb51005fb80 con 0x7fb510059990 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb50ed9d700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb510059990 0x7fb510059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb4ff7fe700 10 client.37274.objecter ms_handle_connect 0x7fb510059990 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb50ed9d700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb510059990 0x7fb510059d70 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb51005fb80 con 0x7fb510059990 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.806+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 1 = system:0 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.807+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510109770 osd=0 4 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.807+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 0 1 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.822+0000 7fb514aa9700 10 client.37274.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.822+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 2 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.822+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510109770 osd=0 3 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.822+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510109770 osd=0 3 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.822+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510109770 osd=0 3 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.822+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 2 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.822+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510109770 osd=0 3 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.822+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 0 2 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.822+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510109770 osd=0 4 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.822+0000 7fb514aa9700 10 client.37274.objecter _send_command 2 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.822+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb510028590 con 0x7fb510059990 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.823+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb510028590 con 0x7fb510059990 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.823+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.823+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.823+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510109770 osd=0 4 2024-02-13T00:45:07.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.823+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 0 2 2024-02-13T00:45:07.443 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:45:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.823+0000 7fb514aa9700 10 client.37274.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 3 2024-02-13T00:45:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6805/150599780 conn(0x7fb510059570 0x7fb510119100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510059400 osd=1 3 2024-02-13T00:45:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510059400 osd=1 3 2024-02-13T00:45:07.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510059400 osd=1 3 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 3 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510059400 osd=1 3 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 1 3 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510059400 osd=1 4 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb514aa9700 10 client.37274.objecter _send_command 3 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5100fc9e0 con 0x7fb510059570 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb50e59c700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6805/150599780 conn(0x7fb510059570 0x7fb510119100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb4ff7fe700 10 client.37274.objecter ms_handle_connect 0x7fb510059570 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.824+0000 7fb50e59c700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6805/150599780 conn(0x7fb510059570 0x7fb510119100 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.825+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb5100fc9e0 con 0x7fb510059570 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.825+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.825+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 3 = system:0 2024-02-13T00:45:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.825+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510059400 osd=1 4 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.825+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 1 3 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.839+0000 7fb514aa9700 10 client.37274.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.839+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 4 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.839+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510059400 osd=1 3 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.839+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510059400 osd=1 3 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.839+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510059400 osd=1 3 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.839+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 4 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.839+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510059400 osd=1 3 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.839+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 1 4 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.839+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510059400 osd=1 4 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.839+0000 7fb514aa9700 10 client.37274.objecter _send_command 4 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.839+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb5101195f0 con 0x7fb510059570 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb5101195f0 con 0x7fb510059570 2024-02-13T00:45:07.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510059400 osd=1 4 2024-02-13T00:45:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 1 4 2024-02-13T00:45:07.446 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:45:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb514aa9700 10 client.37274.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 5 2024-02-13T00:45:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb5100fc0b0 0x7fb51011c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:07.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb5100fbf40 osd=2 3 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb5100fbf40 osd=2 3 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb5100fbf40 osd=2 3 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 5 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb5100fbf40 osd=2 3 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 2 5 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb5100fbf40 osd=2 4 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb514aa9700 10 client.37274.objecter _send_command 5 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5100280f0 con 0x7fb5100fc0b0 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb50dd9b700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb5100fc0b0 0x7fb51011c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb4ff7fe700 10 client.37274.objecter ms_handle_connect 0x7fb5100fc0b0 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.842+0000 7fb50dd9b700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb5100fc0b0 0x7fb51011c950 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.843+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb5100280f0 con 0x7fb5100fc0b0 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.843+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:45:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.843+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 5 = system:0 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.843+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb5100fbf40 osd=2 4 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.843+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 2 5 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.858+0000 7fb514aa9700 10 client.37274.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.858+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 6 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.858+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb5100fbf40 osd=2 3 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.858+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb5100fbf40 osd=2 3 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.858+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb5100fbf40 osd=2 3 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.858+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 6 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.858+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb5100fbf40 osd=2 3 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.858+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 2 6 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.858+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb5100fbf40 osd=2 4 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.858+0000 7fb514aa9700 10 client.37274.objecter _send_command 6 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.858+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb51011ce40 con 0x7fb5100fc0b0 2024-02-13T00:45:07.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb51011ce40 con 0x7fb5100fc0b0 2024-02-13T00:45:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.449 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:45:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb5100fbf40 osd=2 4 2024-02-13T00:45:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 2 6 2024-02-13T00:45:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb514aa9700 10 client.37274.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 7 2024-02-13T00:45:07.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6813/604971011 conn(0x7fb5100277c0 0x7fb510120140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510027650 osd=3 3 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510027650 osd=3 3 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510027650 osd=3 3 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 7 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510027650 osd=3 3 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 3 7 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510027650 osd=3 4 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb514aa9700 10 client.37274.objecter _send_command 7 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb51011a470 con 0x7fb5100277c0 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb50ed9d700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6813/604971011 conn(0x7fb5100277c0 0x7fb510120140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb4ff7fe700 10 client.37274.objecter ms_handle_connect 0x7fb5100277c0 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.860+0000 7fb50ed9d700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6813/604971011 conn(0x7fb5100277c0 0x7fb510120140 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.861+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb51011a470 con 0x7fb5100277c0 2024-02-13T00:45:07.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.861+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.861+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 7 = system:0 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.861+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510027650 osd=3 4 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.861+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 3 7 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.877+0000 7fb514aa9700 10 client.37274.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.877+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 8 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.877+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510027650 osd=3 3 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.877+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510027650 osd=3 3 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.877+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510027650 osd=3 3 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.877+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 8 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.877+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510027650 osd=3 3 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.877+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 3 8 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.877+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510027650 osd=3 4 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.877+0000 7fb514aa9700 10 client.37274.objecter _send_command 8 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.877+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb510120630 con 0x7fb5100277c0 2024-02-13T00:45:07.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb510120630 con 0x7fb5100277c0 2024-02-13T00:45:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.452 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:45:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510027650 osd=3 4 2024-02-13T00:45:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 3 8 2024-02-13T00:45:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb514aa9700 10 client.37274.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:07.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 9 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb510120d10 0x7fb510123100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510120630 osd=4 3 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510120630 osd=4 3 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510120630 osd=4 3 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 9 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510120630 osd=4 3 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 4 9 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510120630 osd=4 4 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb514aa9700 10 client.37274.objecter _send_command 9 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.878+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb510125eb0 con 0x7fb510120d10 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.879+0000 7fb50e59c700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb510120d10 0x7fb510123100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.879+0000 7fb50e59c700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb510120d10 0x7fb510123100 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.879+0000 7fb4ff7fe700 10 client.37274.objecter ms_handle_connect 0x7fb510120d10 2024-02-13T00:45:07.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.880+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb510125eb0 con 0x7fb510120d10 2024-02-13T00:45:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.880+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:45:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.880+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 9 = system:0 2024-02-13T00:45:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.880+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510120630 osd=4 4 2024-02-13T00:45:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.880+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 4 9 2024-02-13T00:45:07.454 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:45:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.895+0000 7fb514aa9700 10 client.37274.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.895+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 10 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.895+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510120630 osd=4 3 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.895+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510120630 osd=4 3 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.895+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510120630 osd=4 3 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.895+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 10 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.895+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510120630 osd=4 3 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.895+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 4 10 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.895+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510120630 osd=4 4 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.895+0000 7fb514aa9700 10 client.37274.objecter _send_command 10 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.895+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb510125eb0 con 0x7fb510120d10 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb510125eb0 con 0x7fb510120d10 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:07.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510120630 osd=4 4 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 4 10 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb514aa9700 10 client.37274.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 11 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6804/2212296748 conn(0x7fb5101251b0 0x7fb510125610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510125eb0 osd=5 3 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510125eb0 osd=5 3 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510125eb0 osd=5 3 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 11 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510125eb0 osd=5 3 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 5 11 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510125eb0 osd=5 4 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb514aa9700 10 client.37274.objecter _send_command 11 2024-02-13T00:45:07.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.896+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb510125cb0 con 0x7fb5101251b0 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.897+0000 7fb50dd9b700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6804/2212296748 conn(0x7fb5101251b0 0x7fb510125610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.897+0000 7fb4ff7fe700 10 client.37274.objecter ms_handle_connect 0x7fb5101251b0 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.897+0000 7fb50dd9b700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6804/2212296748 conn(0x7fb5101251b0 0x7fb510125610 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.898+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb510125cb0 con 0x7fb5101251b0 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.898+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.898+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 11 = system:0 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.898+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510125eb0 osd=5 4 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.898+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 5 11 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.912+0000 7fb514aa9700 10 client.37274.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.912+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 12 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.912+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510125eb0 osd=5 3 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.912+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510125eb0 osd=5 3 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.912+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510125eb0 osd=5 3 2024-02-13T00:45:07.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.912+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 12 2024-02-13T00:45:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.912+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510125eb0 osd=5 3 2024-02-13T00:45:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.912+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 5 12 2024-02-13T00:45:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.912+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510125eb0 osd=5 4 2024-02-13T00:45:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.912+0000 7fb514aa9700 10 client.37274.objecter _send_command 12 2024-02-13T00:45:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.912+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb510125cb0 con 0x7fb5101251b0 2024-02-13T00:45:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb510125cb0 con 0x7fb5101251b0 2024-02-13T00:45:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510125eb0 osd=5 4 2024-02-13T00:45:07.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 5 12 2024-02-13T00:45:07.458 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:45:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb514aa9700 10 client.37274.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 13 2024-02-13T00:45:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb5101286b0 0x7fb510128af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510125cb0 osd=6 3 2024-02-13T00:45:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510125cb0 osd=6 3 2024-02-13T00:45:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510125cb0 osd=6 3 2024-02-13T00:45:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 13 2024-02-13T00:45:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510125cb0 osd=6 3 2024-02-13T00:45:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 6 13 2024-02-13T00:45:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510125cb0 osd=6 4 2024-02-13T00:45:07.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb514aa9700 10 client.37274.objecter _send_command 13 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb51012c570 con 0x7fb5101286b0 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.914+0000 7fb50ed9d700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb5101286b0 0x7fb510128af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.915+0000 7fb4ff7fe700 10 client.37274.objecter ms_handle_connect 0x7fb5101286b0 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.915+0000 7fb50ed9d700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb5101286b0 0x7fb510128af0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.915+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb51012c570 con 0x7fb5101286b0 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.915+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.915+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 13 = system:0 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.915+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510125cb0 osd=6 4 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.915+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 6 13 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.930+0000 7fb514aa9700 10 client.37274.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.930+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 14 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.930+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510125cb0 osd=6 3 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.930+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510125cb0 osd=6 3 2024-02-13T00:45:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.930+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510125cb0 osd=6 3 2024-02-13T00:45:07.461 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:45:07.461 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:45:07.461 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.930+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 14 2024-02-13T00:45:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.930+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510125cb0 osd=6 3 2024-02-13T00:45:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.930+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 6 14 2024-02-13T00:45:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.930+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510125cb0 osd=6 4 2024-02-13T00:45:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.930+0000 7fb514aa9700 10 client.37274.objecter _send_command 14 2024-02-13T00:45:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.930+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb510123690 con 0x7fb5101286b0 2024-02-13T00:45:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.931+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb510123690 con 0x7fb5101286b0 2024-02-13T00:45:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.931+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.931+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.931+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510125cb0 osd=6 4 2024-02-13T00:45:07.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.931+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 6 14 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb514aa9700 10 client.37274.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 15 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb510123d70 0x7fb510124170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510123690 osd=7 3 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510123690 osd=7 3 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510123690 osd=7 3 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 15 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510123690 osd=7 3 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 7 15 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510123690 osd=7 4 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb514aa9700 10 client.37274.objecter _send_command 15 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb510124770 con 0x7fb510123d70 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb50e59c700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb510123d70 0x7fb510124170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:07.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb4ff7fe700 10 client.37274.objecter ms_handle_connect 0x7fb510123d70 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.932+0000 7fb50e59c700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb510123d70 0x7fb510124170 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.933+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb510124770 con 0x7fb510123d70 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.933+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.933+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 15 = system:0 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.933+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510123690 osd=7 4 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.933+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 7 15 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.947+0000 7fb514aa9700 10 client.37274.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.947+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign -1 16 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.947+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510123690 osd=7 3 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.947+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510123690 osd=7 3 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.947+0000 7fb514aa9700 20 client.37274.objecter _get_session s=0x7fb510123690 osd=7 3 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.947+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_remove -1 16 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.947+0000 7fb514aa9700 20 client.37274.objecter get_session s=0x7fb510123690 osd=7 3 2024-02-13T00:45:07.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.947+0000 7fb514aa9700 15 client.37274.objecter _session_command_op_assign 7 16 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.947+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510123690 osd=7 4 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.947+0000 7fb514aa9700 10 client.37274.objecter _send_command 16 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.947+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb510124730 con 0x7fb510123d70 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb4ff7fe700 1 -- 172.21.15.162:0/168161704 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb510124730 con 0x7fb510123d70 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb4ff7fe700 10 client.37274.objecter ms_dispatch 0x7fb510001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb4ff7fe700 10 client.37274.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb4ff7fe700 20 client.37274.objecter put_session s=0x7fb510123690 osd=7 4 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb4ff7fe700 15 client.37274.objecter _session_command_op_remove 7 16 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 10 librados: watch_flush enter 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 10 librados: watch_flush exit 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 10 client.37274.objecter close_session for osd.0 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb510059990 msgr2=0x7fb510059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb510059990 0x7fb510059d70 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510109770 osd=0 1 2024-02-13T00:45:07.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 10 client.37274.objecter close_session for osd.1 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6805/150599780 conn(0x7fb510059570 msgr2=0x7fb510119100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6805/150599780 conn(0x7fb510059570 0x7fb510119100 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510059400 osd=1 1 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 10 client.37274.objecter close_session for osd.2 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb5100fc0b0 msgr2=0x7fb51011c950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb5100fc0b0 0x7fb51011c950 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb5100fbf40 osd=2 1 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 10 client.37274.objecter close_session for osd.3 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6813/604971011 conn(0x7fb5100277c0 msgr2=0x7fb510120140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6813/604971011 conn(0x7fb5100277c0 0x7fb510120140 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510027650 osd=3 1 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 10 client.37274.objecter close_session for osd.4 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb510120d10 msgr2=0x7fb510123100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb510120d10 0x7fb510123100 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb50ed9d700 1 -- 172.21.15.162:0/168161704 reap_dead start 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510120630 osd=4 1 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 10 client.37274.objecter close_session for osd.5 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6804/2212296748 conn(0x7fb5101251b0 msgr2=0x7fb510125610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6804/2212296748 conn(0x7fb5101251b0 0x7fb510125610 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb50ed9d700 1 -- 172.21.15.162:0/168161704 reap_dead start 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510125eb0 osd=5 1 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 10 client.37274.objecter close_session for osd.6 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb5101286b0 msgr2=0x7fb510128af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb5101286b0 0x7fb510128af0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510125cb0 osd=6 1 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 10 client.37274.objecter close_session for osd.7 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb510123d70 msgr2=0x7fb510124170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb510123d70 0x7fb510124170 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 20 client.37274.objecter put_session s=0x7fb510123690 osd=7 1 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 20 client.37274.objecter shutdown clearing up homeless session... 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb514aa9700 10 client.37274.objecter successfully canceled tick 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.949+0000 7fb50ed9d700 1 -- 172.21.15.162:0/168161704 reap_dead start 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.950+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb4f8062c90 msgr2=0x7fb4f8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.950+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb4f8062c90 0x7fb4f8065150 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 rx=0x7fb500007bf0 tx=0x7fb50000b040).stop 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.950+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb5100f95e0 msgr2=0x7fb51010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.950+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb5100f95e0 0x7fb51010e5e0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7fb50400e970 tx=0x7fb50400b040).stop 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.950+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 shutdown_connections 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.950+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb5100f95e0 0x7fb51010e5e0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.950+0000 7fb514aa9700 1 --2- 172.21.15.162:0/168161704 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb4f8062c90 0x7fb4f8065150 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.950+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 >> 172.21.15.162:0/168161704 conn(0x7fb51006eb90 msgr2=0x7fb5100708f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.950+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 shutdown_connections 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.950+0000 7fb514aa9700 1 -- 172.21.15.162:0/168161704 wait complete. 2024-02-13T00:45:07.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:06.950+0000 7fb514aa9700 1 librados: shutdown 2024-02-13T00:45:07.469 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.469 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.469 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.469 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.469 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.469 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.469 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.469 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:45:07.469 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 6] out_osds: [3, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:45:07.469 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:45:07.469 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 4, 5, 7, 6] 2024-02-13T00:45:07.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd out 7 2024-02-13T00:45:07.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:07.471 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:07.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:07.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:07.598 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:07.616 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:07.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:07.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:07.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:06.400309+0000 mgr.y (mgr.14544) 317 : cluster [DBG] pgmap v496: 246 pgs: 8 unknown, 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:45:07.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:06.806558+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? 172.21.15.162:0/168161704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:07.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:07.671548+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-13T00:45:07.682 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:06.400309+0000 mgr.y (mgr.14544) 317 : cluster [DBG] pgmap v496: 246 pgs: 8 unknown, 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:45:07.682 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:06.806558+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? 172.21.15.162:0/168161704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:07.682 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:07.671548+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-13T00:45:07.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:07.746 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:07.764 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:07.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:07.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:07.895 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:06.400309+0000 mgr.y (mgr.14544) 317 : cluster [DBG] pgmap v496: 246 pgs: 8 unknown, 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:45:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:06.806558+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? 172.21.15.162:0/168161704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:07.671548+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-45","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-13T00:45:08.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:45:08.544 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:08.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:08.648 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:45:08.676 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:08.704 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:45:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:08.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:45:08.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:08.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:07.671661+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e352: 8 total, 8 up, 6 in 2024-02-13T00:45:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:07.672483+0000 mon.c (mon.2) 349 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:07.673206+0000 mon.a (mon.0) 2264 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:07.696058+0000 mon.b (mon.1) 528 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:07.696495+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:08.410734+0000 mon.a (mon.0) 2266 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi162-84017-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi162-84017-2"}]': finished 2024-02-13T00:45:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:08.410825+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]': finished 2024-02-13T00:45:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:08.410889+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:45:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:08.410955+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e353: 8 total, 8 up, 6 in 2024-02-13T00:45:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:08.411740+0000 mon.b (mon.1) 529 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:45:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:08.412695+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:45:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:08.413040+0000 mon.c (mon.2) 350 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:08.413454+0000 mon.a (mon.0) 2271 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:07.671661+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e352: 8 total, 8 up, 6 in 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:07.672483+0000 mon.c (mon.2) 349 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:07.673206+0000 mon.a (mon.0) 2264 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:07.696058+0000 mon.b (mon.1) 528 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:07.696495+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:08.410734+0000 mon.a (mon.0) 2266 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi162-84017-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi162-84017-2"}]': finished 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:08.410825+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]': finished 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:08.410889+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:08.410955+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e353: 8 total, 8 up, 6 in 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:08.411740+0000 mon.b (mon.1) 529 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:08.412695+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:08.413040+0000 mon.c (mon.2) 350 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:08.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:08.413454+0000 mon.a (mon.0) 2271 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:08.838 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:45:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:45:08.838 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:07.671661+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e352: 8 total, 8 up, 6 in 2024-02-13T00:45:08.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:07.672483+0000 mon.c (mon.2) 349 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:08.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:07.673206+0000 mon.a (mon.0) 2264 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:08.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:07.696058+0000 mon.b (mon.1) 528 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:08.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:07.696495+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:08.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:08.410734+0000 mon.a (mon.0) 2266 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi162-84017-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi162-84017-2"}]': finished 2024-02-13T00:45:08.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:08.410825+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-83985-60"}]': finished 2024-02-13T00:45:08.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:08.410889+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:45:08.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:08.410955+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e353: 8 total, 8 up, 6 in 2024-02-13T00:45:08.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:08.411740+0000 mon.b (mon.1) 529 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:45:08.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:08.412695+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:45:08.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:08.413040+0000 mon.c (mon.2) 350 : audit [INF] from='client.? 172.21.15.162:0/1003782898' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:08.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:08.413454+0000 mon.a (mon.0) 2271 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]: dispatch 2024-02-13T00:45:08.841 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:08.862 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:08.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:08.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:08.995 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:09.120 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:09.121 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:45:09.121 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:45:09.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:09.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:09.150 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:09.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:09.254 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:09.413 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:09.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:b.RoundTripAppendPP (3213 ms) 2024-02-13T00:45:09.426 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-02-13T00:45:09.426 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2787 ms) 2024-02-13T00:45:09.426 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-02-13T00:45:09.426 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3016 ms) 2024-02-13T00:45:09.426 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-02-13T00:45:09.426 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3017 ms) 2024-02-13T00:45:09.426 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-02-13T00:45:09.426 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (2962 ms) 2024-02-13T00:45:09.426 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 31 tests from LibRadosAio (117659 ms total) 2024-02-13T00:45:09.426 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: 2024-02-13T00:45:09.426 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-02-13T00:45:09.426 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2989 ms) 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9018 ms) 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15006 ms) 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3231 ms) 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30244 ms total) 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2818 ms) 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2818 ms total) 2024-02-13T00:45:09.427 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13966 ms) 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6982 ms) 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7015 ms) 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6996 ms) 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3024 ms) 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6985 ms) 2024-02-13T00:45:09.428 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7010 ms) 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6992 ms) 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7018 ms) 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7097 ms) 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6883 ms) 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7026 ms) 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7030 ms) 2024-02-13T00:45:09.429 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-02-13T00:45:09.430 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (2953 ms) 2024-02-13T00:45:09.430 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-02-13T00:45:09.430 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7035 ms) 2024-02-13T00:45:09.871 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:08.400747+0000 mgr.y (mgr.14544) 318 : cluster [DBG] pgmap v499: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:45:09.871 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:08.985724+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:09.872 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:08.985896+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:09.872 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:09.412836+0000 mon.a (mon.0) 2272 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]': finished 2024-02-13T00:45:09.872 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:09.412919+0000 mon.a (mon.0) 2273 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]': finished 2024-02-13T00:45:09.872 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:09.413017+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e354: 8 total, 8 up, 6 in 2024-02-13T00:45:09.872 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:09.432342+0000 mon.b (mon.1) 530 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:09.872 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:09.432900+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:09.872 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:09.433498+0000 mon.b (mon.1) 531 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:45:09.872 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:09.434074+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:45:09.872 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:09.441290+0000 mon.c (mon.2) 351 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:09.872 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:09.441779+0000 mon.a (mon.0) 2277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:09.872 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:09.442197+0000 mon.c (mon.2) 352 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:09.872 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:09.442420+0000 mon.a (mon.0) 2278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:09.873 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:09.442836+0000 mon.c (mon.2) 353 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:09.873 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:09.443091+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:09.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:08.400747+0000 mgr.y (mgr.14544) 318 : cluster [DBG] pgmap v499: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:45:09.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:08.985724+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:09.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:08.985896+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:09.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:09.412836+0000 mon.a (mon.0) 2272 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]': finished 2024-02-13T00:45:09.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:09.412919+0000 mon.a (mon.0) 2273 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]': finished 2024-02-13T00:45:09.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:09.413017+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e354: 8 total, 8 up, 6 in 2024-02-13T00:45:09.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:09.432342+0000 mon.b (mon.1) 530 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:09.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:09.432900+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:09.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:09.433498+0000 mon.b (mon.1) 531 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:45:09.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:09.434074+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:45:09.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:09.441290+0000 mon.c (mon.2) 351 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:09.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:09.441779+0000 mon.a (mon.0) 2277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:09.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:09.442197+0000 mon.c (mon.2) 352 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:09.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:09.442420+0000 mon.a (mon.0) 2278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:09.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:09.442836+0000 mon.c (mon.2) 353 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:09.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:09.443091+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:08.400747+0000 mgr.y (mgr.14544) 318 : cluster [DBG] pgmap v499: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:45:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:08.985724+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:08.985896+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:09.412836+0000 mon.a (mon.0) 2272 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]': finished 2024-02-13T00:45:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:09.412919+0000 mon.a (mon.0) 2273 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-83985-60"}]': finished 2024-02-13T00:45:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:09.413017+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e354: 8 total, 8 up, 6 in 2024-02-13T00:45:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:09.432342+0000 mon.b (mon.1) 530 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:09.432900+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:09.433498+0000 mon.b (mon.1) 531 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:45:10.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:09.434074+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-45"}]: dispatch 2024-02-13T00:45:10.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:09.441290+0000 mon.c (mon.2) 351 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:10.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:09.441779+0000 mon.a (mon.0) 2277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:10.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:09.442197+0000 mon.c (mon.2) 352 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:10.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:09.442420+0000 mon.a (mon.0) 2278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:10.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:09.442836+0000 mon.c (mon.2) 353 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:10.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:09.443091+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:10.151 INFO:tasks.workunit.client.0.smithi162.stdout: 8 expected=8 2024-02-13T00:45:10.152 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:b29083e3:::5:head 2024-02-13T00:45:10.152 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:b29083e3:::5:head expected=18:b29083e3:::5:head 2024-02-13T00:45:10.152 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:b29083e3:::5:head -> 5 2024-02-13T00:45:10.152 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=5 expected=5 2024-02-13T00:45:10.152 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:89d3ae78:::11:head 2024-02-13T00:45:10.152 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:89d3ae78:::11:head expected=18:89d3ae78:::11:head 2024-02-13T00:45:10.152 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:89d3ae78:::11:head -> 11 2024-02-13T00:45:10.152 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=11 expected=11 2024-02-13T00:45:10.152 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:863748b0:::15:head 2024-02-13T00:45:10.152 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:863748b0:::15:head expected=18:863748b0:::15:head 2024-02-13T00:45:10.152 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:863748b0:::15:head -> 15 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=15 expected=15 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:6cac518f:::0:head 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:6cac518f:::0:head expected=18:6cac518f:::0:head 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:6cac518f:::0:head -> 0 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=0 expected=0 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:62a1935d:::14:head 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:62a1935d:::14:head expected=18:62a1935d:::14:head 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:62a1935d:::14:head -> 14 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=14 expected=14 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:5c6b0b28:::7:head 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:5c6b0b28:::7:head expected=18:5c6b0b28:::7:head 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:5c6b0b28:::7:head -> 7 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=7 expected=7 2024-02-13T00:45:10.153 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:02547ec2:::1:head 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:02547ec2:::1:head expected=18:02547ec2:::1:head 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:02547ec2:::1:head -> 1 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=1 expected=1 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 18:52ea6a34:::10:head 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=18:52ea6a34:::10:head expected=18:52ea6a34:::10:head 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 18:52ea6a34:::10:head -> 10 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=10 expected=10 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (748 ms) 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (202040 ms) 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: split 0/5 -> MIN 18:33333333::::head 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: split 1/5 -> 18:33333333::::head 18:66666666::::head 2024-02-13T00:45:10.154 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: split 2/5 -> 18:66666666::::head 18:99999999::::head 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: split 3/5 -> 18:99999999::::head 18:cccccccc::::head 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: split 4/5 -> 18:cccccccc::::head MAX 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (39524 ms) 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] 7 tests from LibRadosList (242957 ms total) 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1704 ms) 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset foo1,foo2,foo3 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo1 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo2 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo3 2024-02-13T00:45:10.155 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset foo1,foo4,foo5 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo4 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo5 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo1 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset foo6,foo7 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo7 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo6 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns1:foo4 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns1:foo5 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns2:foo7 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns2:foo6 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns1:foo1 2024-02-13T00:45:10.156 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: :foo1 2024-02-13T00:45:10.157 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: :foo2 2024-02-13T00:45:10.157 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: :foo3 2024-02-13T00:45:10.157 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (31 ms) 2024-02-13T00:45:10.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:45:10.403 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:10.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:10.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:10.540 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:10.559 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:10.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:10.662 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:10.691 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:10.710 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:10.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:10.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:10.840 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:10.859 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:10.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:10.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:10.989 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:11.008 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:11.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:11.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:11.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:11.142 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:10.063667+0000 mon.a (mon.0) 2280 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:10.385274+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:10.385395+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e355: 8 total, 8 up, 6 in 2024-02-13T00:45:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:10.389069+0000 mon.c (mon.2) 354 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-83985-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:10.391757+0000 mon.a (mon.0) 2283 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:10.392488+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-83985-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:11.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:10.063667+0000 mon.a (mon.0) 2280 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:11.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:10.385274+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:11.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:10.385395+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e355: 8 total, 8 up, 6 in 2024-02-13T00:45:11.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:10.389069+0000 mon.c (mon.2) 354 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-83985-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:11.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:10.391757+0000 mon.a (mon.0) 2283 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:11.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:10.392488+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-83985-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:11.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:10.063667+0000 mon.a (mon.0) 2280 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:11.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:10.385274+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:11.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:10.385395+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e355: 8 total, 8 up, 6 in 2024-02-13T00:45:11.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:10.389069+0000 mon.c (mon.2) 354 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-83985-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:11.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:10.391757+0000 mon.a (mon.0) 2283 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:11.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:10.392488+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-83985-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:11.340 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:11.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:11.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:11.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-13T00:45:11.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:45:11.473 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:12.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:10.401171+0000 mgr.y (mgr.14544) 319 : cluster [DBG] pgmap v503: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 2 op/s; 206 B/s, 4 objects/s recovering 2024-02-13T00:45:12.329 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:11.396609+0000 mon.a (mon.0) 2285 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-84017-2"}]': finished 2024-02-13T00:45:12.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:11.639995+0000 mon.a (mon.0) 2286 : cluster [DBG] osdmap e356: 8 total, 8 up, 6 in 2024-02-13T00:45:12.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:11.640739+0000 mon.a (mon.0) 2287 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:12.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:11.640988+0000 mon.b (mon.1) 532 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:12.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:11.641371+0000 mon.a (mon.0) 2288 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:10.401171+0000 mgr.y (mgr.14544) 319 : cluster [DBG] pgmap v503: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 2 op/s; 206 B/s, 4 objects/s recovering 2024-02-13T00:45:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:11.396609+0000 mon.a (mon.0) 2285 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-84017-2"}]': finished 2024-02-13T00:45:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:11.639995+0000 mon.a (mon.0) 2286 : cluster [DBG] osdmap e356: 8 total, 8 up, 6 in 2024-02-13T00:45:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:11.640739+0000 mon.a (mon.0) 2287 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:11.640988+0000 mon.b (mon.1) 532 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:12.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:11.641371+0000 mon.a (mon.0) 2288 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:10.401171+0000 mgr.y (mgr.14544) 319 : cluster [DBG] pgmap v503: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 2 op/s; 206 B/s, 4 objects/s recovering 2024-02-13T00:45:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:11.396609+0000 mon.a (mon.0) 2285 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-84017-2"}]': finished 2024-02-13T00:45:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:11.639995+0000 mon.a (mon.0) 2286 : cluster [DBG] osdmap e356: 8 total, 8 up, 6 in 2024-02-13T00:45:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:11.640739+0000 mon.a (mon.0) 2287 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-84017-2"}]: dispatch 2024-02-13T00:45:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:11.640988+0000 mon.b (mon.1) 532 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:11.641371+0000 mon.a (mon.0) 2288 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:12.623 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:45:12.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:12.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:45:12.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.626+0000 7ff5931e0700 1 -- 172.21.15.162:0/3038864824 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff58c0fcb30 msgr2=0x7ff58c0fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:12.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.626+0000 7ff5931e0700 1 --2- 172.21.15.162:0/3038864824 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff58c0fcb30 0x7ff58c0fcf10 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7ff580004870 tx=0x7ff580009cb0).stop 2024-02-13T00:45:12.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.626+0000 7ff5931e0700 1 -- 172.21.15.162:0/3038864824 shutdown_connections 2024-02-13T00:45:12.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.626+0000 7ff5931e0700 1 --2- 172.21.15.162:0/3038864824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff58c0f73a0 0x7ff58c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:12.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.626+0000 7ff5931e0700 1 --2- 172.21.15.162:0/3038864824 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff58c0fcb30 0x7ff58c0fcf10 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:12.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.626+0000 7ff5931e0700 1 --2- 172.21.15.162:0/3038864824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff58c0fd490 0x7ff58c0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:12.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.626+0000 7ff5931e0700 1 -- 172.21.15.162:0/3038864824 >> 172.21.15.162:0/3038864824 conn(0x7ff58c06eb90 msgr2=0x7ff58c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:12.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.626+0000 7ff5931e0700 1 -- 172.21.15.162:0/3038864824 shutdown_connections 2024-02-13T00:45:12.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.626+0000 7ff5931e0700 1 -- 172.21.15.162:0/3038864824 wait complete. 2024-02-13T00:45:12.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff5931e0700 1 librados: starting msgr at 2024-02-13T00:45:12.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff5931e0700 1 librados: starting objecter 2024-02-13T00:45:12.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff5931e0700 1 Processor -- start 2024-02-13T00:45:12.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff5931e0700 1 -- start start 2024-02-13T00:45:12.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff5931e0700 1 librados: setting wanted keys 2024-02-13T00:45:12.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff5931e0700 1 librados: calling monclient init 2024-02-13T00:45:12.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff5931e0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff58c0f73a0 0x7ff58c10e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff5931e0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff58c0fcb30 0x7ff58c10e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff5931e0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff58c0fd490 0x7ff58c1081b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff5931e0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c1086a0 con 0x7ff58c0fcb30 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff5931e0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c108810 con 0x7ff58c0f73a0 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff5931e0700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c1089d0 con 0x7ff58c0fd490 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff58bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff58c0fcb30 0x7ff58c10e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff59177d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff58c0fd490 0x7ff58c1081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff58bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff58c0fcb30 0x7ff58c10e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44328/0 (socket says 172.21.15.162:44328) 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff58bfff700 1 -- 172.21.15.162:0/2483426735 learned_addr learned my addr 172.21.15.162:0/2483426735 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.627+0000 7ff590f7c700 1 --2- 172.21.15.162:0/2483426735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff58c0f73a0 0x7ff58c10e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff59177d700 1 -- 172.21.15.162:0/2483426735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff58c0f73a0 msgr2=0x7ff58c10e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff59177d700 1 --2- 172.21.15.162:0/2483426735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff58c0f73a0 0x7ff58c10e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff59177d700 1 -- 172.21.15.162:0/2483426735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff58c0fcb30 msgr2=0x7ff58c10e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff59177d700 1 --2- 172.21.15.162:0/2483426735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff58c0fcb30 0x7ff58c10e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:12.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff59177d700 1 -- 172.21.15.162:0/2483426735 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff580005040 con 0x7ff58c0fd490 2024-02-13T00:45:12.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff58bfff700 1 --2- 172.21.15.162:0/2483426735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff58c0fcb30 0x7ff58c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:45:12.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff59177d700 1 --2- 172.21.15.162:0/2483426735 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff58c0fd490 0x7ff58c1081b0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 rx=0x7ff58400eb70 tx=0x7ff58400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:12.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff589ffb700 10 client.?.objecter ms_handle_connect 0x7ff58c0fd490 2024-02-13T00:45:12.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff589ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:12.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff589ffb700 1 -- 172.21.15.162:0/2483426735 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584009870 con 0x7ff58c0fd490 2024-02-13T00:45:12.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff589ffb700 1 -- 172.21.15.162:0/2483426735 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff58400f040 con 0x7ff58c0fd490 2024-02-13T00:45:12.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff5931e0700 1 -- 172.21.15.162:0/2483426735 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff58c108c50 con 0x7ff58c0fd490 2024-02-13T00:45:12.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff589ffb700 1 -- 172.21.15.162:0/2483426735 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584007da0 con 0x7ff58c0fd490 2024-02-13T00:45:12.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff5931e0700 10 client.37292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:12.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff5931e0700 1 -- 172.21.15.162:0/2483426735 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58c114e40 con 0x7ff58c0fd490 2024-02-13T00:45:12.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.628+0000 7ff5931e0700 1 librados: init done 2024-02-13T00:45:12.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.629+0000 7ff5931e0700 1 -- 172.21.15.162:0/2483426735 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58c0475e0 con 0x7ff58c0fd490 2024-02-13T00:45:12.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.630+0000 7ff589ffb700 1 -- 172.21.15.162:0/2483426735 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff5840040b0 con 0x7ff58c0fd490 2024-02-13T00:45:12.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.630+0000 7ff589ffb700 1 --2- 172.21.15.162:0/2483426735 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff57c062c90 0x7ff57c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:12.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.630+0000 7ff589ffb700 1 -- 172.21.15.162:0/2483426735 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(357..357 src has 52..357) v4 ==== 8912+0+0 (secure 0 0 0) 0x7ff58407d220 con 0x7ff58c0fd490 2024-02-13T00:45:12.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.630+0000 7ff589ffb700 10 client.37292.objecter ms_dispatch 0x7ff58c001550 osd_map(357..357 src has 52..357) v4 2024-02-13T00:45:12.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.630+0000 7ff589ffb700 3 client.37292.objecter handle_osd_map got epochs [357,357] > 0 2024-02-13T00:45:12.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.630+0000 7ff589ffb700 3 client.37292.objecter handle_osd_map decoding full epoch 357 2024-02-13T00:45:12.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.630+0000 7ff590f7c700 1 --2- 172.21.15.162:0/2483426735 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff57c062c90 0x7ff57c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:12.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.630+0000 7ff589ffb700 20 client.37292.objecter dump_active .. 0 homeless 2024-02-13T00:45:12.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.630+0000 7ff589ffb700 10 client.37292.objecter ms_handle_connect 0x7ff57c062c90 2024-02-13T00:45:12.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.630+0000 7ff590f7c700 1 --2- 172.21.15.162:0/2483426735 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff57c062c90 0x7ff57c065150 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 rx=0x7ff5800043d0 tx=0x7ff58000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:12.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.635+0000 7ff589ffb700 1 -- 172.21.15.162:0/2483426735 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff584049f60 con 0x7ff58c0fd490 2024-02-13T00:45:12.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:12.760 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:12.781 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:12.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:12.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:12.814+0000 7ff5931e0700 1 -- 172.21.15.162:0/2483426735 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7ff58c1097a0 con 0x7ff58c0fd490 2024-02-13T00:45:12.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:12.896 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:12.914 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:12.933 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:12.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:13.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:13.066 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:13.085 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:13.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:13.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:45:13.220 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:13.315 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:13.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.312+0000 7f027bbdb700 1 -- 172.21.15.162:0/2080186469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02740f73c0 msgr2=0x7f02741039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:13.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.312+0000 7f027bbdb700 1 --2- 172.21.15.162:0/2080186469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02740f73c0 0x7f02741039c0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7f026c009be0 tx=0x7f026c004d10).stop 2024-02-13T00:45:13.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.312+0000 7f027bbdb700 1 -- 172.21.15.162:0/2080186469 shutdown_connections 2024-02-13T00:45:13.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.312+0000 7f027bbdb700 1 --2- 172.21.15.162:0/2080186469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02740f73c0 0x7f02741039c0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:13.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.312+0000 7f027bbdb700 1 --2- 172.21.15.162:0/2080186469 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02740fcb50 0x7f02740fcf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:13.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.312+0000 7f027bbdb700 1 --2- 172.21.15.162:0/2080186469 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02740fd4b0 0x7f02740f6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:13.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.312+0000 7f027bbdb700 1 -- 172.21.15.162:0/2080186469 >> 172.21.15.162:0/2080186469 conn(0x7f02740f2760 msgr2=0x7f02740f4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:13.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.312+0000 7f027bbdb700 1 -- 172.21.15.162:0/2080186469 shutdown_connections 2024-02-13T00:45:13.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.312+0000 7f027bbdb700 1 -- 172.21.15.162:0/2080186469 wait complete. 2024-02-13T00:45:13.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f027bbdb700 1 librados: starting msgr at 2024-02-13T00:45:13.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f027bbdb700 1 librados: starting objecter 2024-02-13T00:45:13.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f027bbdb700 1 Processor -- start 2024-02-13T00:45:13.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f027bbdb700 1 -- start start 2024-02-13T00:45:13.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f027bbdb700 1 librados: setting wanted keys 2024-02-13T00:45:13.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f027bbdb700 1 librados: calling monclient init 2024-02-13T00:45:13.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f027bbdb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02740f73c0 0x7f0274106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:13.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f027bbdb700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02740fcb50 0x7f02741073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:13.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f027bbdb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02740fd4b0 0x7f02741078c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f027bbdb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0274116560 con 0x7f02740f73c0 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f027bbdb700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02741166a0 con 0x7f02740fd4b0 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f027bbdb700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02741167e0 con 0x7f02740fcb50 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f0279977700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02740f73c0 0x7f0274106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f0279977700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02740f73c0 0x7f0274106ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44346/0 (socket says 172.21.15.162:44346) 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f0279977700 1 -- 172.21.15.162:0/1767024614 learned_addr learned my addr 172.21.15.162:0/1767024614 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.313+0000 7f0279176700 1 --2- 172.21.15.162:0/1767024614 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02740fcb50 0x7f02741073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f027a178700 1 --2- 172.21.15.162:0/1767024614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02740fd4b0 0x7f02741078c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f0279977700 1 -- 172.21.15.162:0/1767024614 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02740fcb50 msgr2=0x7f02741073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f0279977700 1 --2- 172.21.15.162:0/1767024614 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02740fcb50 0x7f02741073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f0279977700 1 -- 172.21.15.162:0/1767024614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02740fd4b0 msgr2=0x7f02741078c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f0279977700 1 --2- 172.21.15.162:0/1767024614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02740fd4b0 0x7f02741078c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f0279977700 1 -- 172.21.15.162:0/1767024614 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c0097c0 con 0x7f02740f73c0 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f0279977700 1 --2- 172.21.15.162:0/1767024614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02740f73c0 0x7f0274106ee0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 rx=0x7f02700041d0 tx=0x7f0270009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:13.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f026affd700 10 client.?.objecter ms_handle_connect 0x7f02740f73c0 2024-02-13T00:45:13.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f026affd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:13.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f026affd700 1 -- 172.21.15.162:0/1767024614 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0270007b40 con 0x7f02740f73c0 2024-02-13T00:45:13.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f027bbdb700 1 -- 172.21.15.162:0/1767024614 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0274116a90 con 0x7f02740f73c0 2024-02-13T00:45:13.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f026affd700 1 -- 172.21.15.162:0/1767024614 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f027000f040 con 0x7f02740f73c0 2024-02-13T00:45:13.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f026affd700 1 -- 172.21.15.162:0/1767024614 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0270014730 con 0x7f02740f73c0 2024-02-13T00:45:13.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f027bbdb700 10 client.27189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:13.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f027bbdb700 1 -- 172.21.15.162:0/1767024614 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0274116fd0 con 0x7f02740f73c0 2024-02-13T00:45:13.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.314+0000 7f027bbdb700 1 librados: init done 2024-02-13T00:45:13.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.315+0000 7f027bbdb700 1 -- 172.21.15.162:0/1767024614 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0274060790 con 0x7f02740f73c0 2024-02-13T00:45:13.321 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:45:13.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.316+0000 7f026affd700 1 -- 172.21.15.162:0/1767024614 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0270009110 con 0x7f02740f73c0 2024-02-13T00:45:13.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.318+0000 7f026affd700 1 --2- 172.21.15.162:0/1767024614 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0260062c90 0x7f0260065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:13.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.318+0000 7f026affd700 1 -- 172.21.15.162:0/1767024614 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(357..357 src has 52..357) v4 ==== 8912+0+0 (secure 0 0 0) 0x7f027007d550 con 0x7f02740f73c0 2024-02-13T00:45:13.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.318+0000 7f026affd700 10 client.27189.objecter ms_dispatch 0x7f0274001550 osd_map(357..357 src has 52..357) v4 2024-02-13T00:45:13.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.318+0000 7f026affd700 3 client.27189.objecter handle_osd_map got epochs [357,357] > 0 2024-02-13T00:45:13.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.318+0000 7f026affd700 3 client.27189.objecter handle_osd_map decoding full epoch 357 2024-02-13T00:45:13.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.318+0000 7f0279176700 1 --2- 172.21.15.162:0/1767024614 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0260062c90 0x7f0260065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:13.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.319+0000 7f026affd700 20 client.27189.objecter dump_active .. 0 homeless 2024-02-13T00:45:13.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.319+0000 7f026affd700 10 client.27189.objecter ms_handle_connect 0x7f0260062c90 2024-02-13T00:45:13.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.319+0000 7f0279176700 1 --2- 172.21.15.162:0/1767024614 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0260062c90 0x7f0260065150 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 rx=0x7f02740618a0 tx=0x7f026400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:13.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:13.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:45:13.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.322+0000 7f026affd700 1 -- 172.21.15.162:0/1767024614 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f02700506d0 con 0x7f02740f73c0 2024-02-13T00:45:13.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.412+0000 7ff589ffb700 1 -- 172.21.15.162:0/2483426735 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v358) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff584050420 con 0x7ff58c0fd490 2024-02-13T00:45:13.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.415+0000 7ff5931e0700 10 librados: watch_flush enter 2024-02-13T00:45:13.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.415+0000 7ff5931e0700 10 librados: watch_flush exit 2024-02-13T00:45:13.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.415+0000 7ff5931e0700 20 client.37292.objecter shutdown clearing up homeless session... 2024-02-13T00:45:13.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.415+0000 7ff5931e0700 10 client.37292.objecter successfully canceled tick 2024-02-13T00:45:13.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.417+0000 7ff5931e0700 1 -- 172.21.15.162:0/2483426735 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff57c062c90 msgr2=0x7ff57c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:13.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.417+0000 7ff5931e0700 1 --2- 172.21.15.162:0/2483426735 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff57c062c90 0x7ff57c065150 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 rx=0x7ff5800043d0 tx=0x7ff58000b040).stop 2024-02-13T00:45:13.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.417+0000 7ff5931e0700 1 -- 172.21.15.162:0/2483426735 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff58c0fd490 msgr2=0x7ff58c1081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:13.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.417+0000 7ff5931e0700 1 --2- 172.21.15.162:0/2483426735 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff58c0fd490 0x7ff58c1081b0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 rx=0x7ff58400eb70 tx=0x7ff58400b040).stop 2024-02-13T00:45:13.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.418+0000 7ff5931e0700 1 -- 172.21.15.162:0/2483426735 shutdown_connections 2024-02-13T00:45:13.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.418+0000 7ff5931e0700 1 --2- 172.21.15.162:0/2483426735 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff57c062c90 0x7ff57c065150 secure :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 rx=0x7ff5800043d0 tx=0x7ff58000b040).stop 2024-02-13T00:45:13.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.418+0000 7ff5931e0700 1 --2- 172.21.15.162:0/2483426735 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff58c0fd490 0x7ff58c1081b0 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:13.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.418+0000 7ff5931e0700 1 --2- 172.21.15.162:0/2483426735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff58c0f73a0 0x7ff58c10e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:13.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.418+0000 7ff5931e0700 1 --2- 172.21.15.162:0/2483426735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff58c0fcb30 0x7ff58c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:13.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.418+0000 7ff5931e0700 1 -- 172.21.15.162:0/2483426735 >> 172.21.15.162:0/2483426735 conn(0x7ff58c06eb90 msgr2=0x7ff58c06ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:13.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:13.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.422+0000 7ff5931e0700 1 -- 172.21.15.162:0/2483426735 shutdown_connections 2024-02-13T00:45:13.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.423+0000 7ff5931e0700 1 -- 172.21.15.162:0/2483426735 wait complete. 2024-02-13T00:45:13.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.423+0000 7ff5931e0700 1 librados: shutdown 2024-02-13T00:45:13.431 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:45:13.460 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:13.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:13.521+0000 7f027bbdb700 1 -- 172.21.15.162:0/1767024614 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd out", "ids": ["7"]} v 0) v1 -- 0x7f02740615e0 con 0x7f02740f73c0 2024-02-13T00:45:13.525 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:12.400009+0000 mon.a (mon.0) 2289 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-83985-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-83985-61"}]': finished 2024-02-13T00:45:13.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:12.400145+0000 mon.a (mon.0) 2290 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-84017-2"}]': finished 2024-02-13T00:45:13.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:12.400246+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:13.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:12.400329+0000 mon.a (mon.0) 2292 : cluster [DBG] osdmap e357: 8 total, 8 up, 6 in 2024-02-13T00:45:13.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:12.402137+0000 mgr.y (mgr.14544) 320 : cluster [DBG] pgmap v506: 246 pgs: 40 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 2 op/s; 206 B/s, 4 objects/s recovering 2024-02-13T00:45:13.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:12.426090+0000 mon.b (mon.1) 533 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:45:13.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:12.426723+0000 mon.a (mon.0) 2293 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:45:13.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:12.815583+0000 mon.c (mon.2) 355 : audit [INF] from='client.? 172.21.15.162:0/2483426735' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:45:13.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:12.815933+0000 mon.a (mon.0) 2294 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:45:13.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:45:13.522+0000 7f5f6486f700 -1 mon.a@0(leader).osd e358 definitely_dead 0 2024-02-13T00:45:13.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:12.400009+0000 mon.a (mon.0) 2289 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-83985-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-83985-61"}]': finished 2024-02-13T00:45:13.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:12.400145+0000 mon.a (mon.0) 2290 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-84017-2"}]': finished 2024-02-13T00:45:13.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:12.400246+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:13.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:12.400329+0000 mon.a (mon.0) 2292 : cluster [DBG] osdmap e357: 8 total, 8 up, 6 in 2024-02-13T00:45:13.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:12.402137+0000 mgr.y (mgr.14544) 320 : cluster [DBG] pgmap v506: 246 pgs: 40 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 2 op/s; 206 B/s, 4 objects/s recovering 2024-02-13T00:45:13.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:12.426090+0000 mon.b (mon.1) 533 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:45:13.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:12.426723+0000 mon.a (mon.0) 2293 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:45:13.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:12.815583+0000 mon.c (mon.2) 355 : audit [INF] from='client.? 172.21.15.162:0/2483426735' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:45:13.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:12.815933+0000 mon.a (mon.0) 2294 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:45:13.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:13.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:13.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:12.400009+0000 mon.a (mon.0) 2289 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-83985-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-83985-61"}]': finished 2024-02-13T00:45:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:12.400145+0000 mon.a (mon.0) 2290 : audit [INF] from='client.? 172.21.15.162:0/3743230436' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-84017-2"}]': finished 2024-02-13T00:45:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:12.400246+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:12.400329+0000 mon.a (mon.0) 2292 : cluster [DBG] osdmap e357: 8 total, 8 up, 6 in 2024-02-13T00:45:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:12.402137+0000 mgr.y (mgr.14544) 320 : cluster [DBG] pgmap v506: 246 pgs: 40 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 2 op/s; 206 B/s, 4 objects/s recovering 2024-02-13T00:45:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:12.426090+0000 mon.b (mon.1) 533 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:45:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:12.426723+0000 mon.a (mon.0) 2293 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:45:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:12.815583+0000 mon.c (mon.2) 355 : audit [INF] from='client.? 172.21.15.162:0/2483426735' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:45:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:12.815933+0000 mon.a (mon.0) 2294 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:45:13.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:13.845 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:13.866 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:13.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:13.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:13.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:45:13.999 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:14.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.414+0000 7f026affd700 1 -- 172.21.15.162:0/1767024614 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["7"]}]=0 marked out osd.7. v359) v1 ==== 87+0+0 (secure 0 0 0) 0x7f027000d360 con 0x7f02740f73c0 2024-02-13T00:45:14.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.417+0000 7f027bbdb700 10 librados: watch_flush enter 2024-02-13T00:45:14.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.421+0000 7f027bbdb700 10 librados: watch_flush exit 2024-02-13T00:45:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.421+0000 7f027bbdb700 20 client.27189.objecter shutdown clearing up homeless session... 2024-02-13T00:45:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.421+0000 7f027bbdb700 10 client.27189.objecter successfully canceled tick 2024-02-13T00:45:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.422+0000 7f027bbdb700 1 -- 172.21.15.162:0/1767024614 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0260062c90 msgr2=0x7f0260065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.422+0000 7f027bbdb700 1 --2- 172.21.15.162:0/1767024614 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0260062c90 0x7f0260065150 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 rx=0x7f02740618a0 tx=0x7f026400b040).stop 2024-02-13T00:45:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.422+0000 7f027bbdb700 1 -- 172.21.15.162:0/1767024614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02740f73c0 msgr2=0x7f0274106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:14.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.422+0000 7f027bbdb700 1 --2- 172.21.15.162:0/1767024614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02740f73c0 0x7f0274106ee0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 rx=0x7f02700041d0 tx=0x7f0270009f60).stop 2024-02-13T00:45:14.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.425+0000 7f027bbdb700 1 -- 172.21.15.162:0/1767024614 shutdown_connections 2024-02-13T00:45:14.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.425+0000 7f027bbdb700 1 --2- 172.21.15.162:0/1767024614 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0260062c90 0x7f0260065150 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:14.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.425+0000 7f027bbdb700 1 --2- 172.21.15.162:0/1767024614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02740fd4b0 0x7f02741078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:14.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.425+0000 7f027bbdb700 1 --2- 172.21.15.162:0/1767024614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02740f73c0 0x7f0274106ee0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:14.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.425+0000 7f027bbdb700 1 --2- 172.21.15.162:0/1767024614 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02740fcb50 0x7f02741073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:14.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.425+0000 7f027bbdb700 1 -- 172.21.15.162:0/1767024614 >> 172.21.15.162:0/1767024614 conn(0x7f02740f2760 msgr2=0x7f02740f3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:14.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.429+0000 7f027bbdb700 1 -- 172.21.15.162:0/1767024614 shutdown_connections 2024-02-13T00:45:14.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.429+0000 7f027bbdb700 1 -- 172.21.15.162:0/1767024614 wait complete. 2024-02-13T00:45:14.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:14.429+0000 7f027bbdb700 1 librados: shutdown 2024-02-13T00:45:14.433 INFO:teuthology.orchestra.run.smithi162.stderr:marked out osd.7. 2024-02-13T00:45:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:13.396824+0000 mon.a (mon.0) 2295 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:45:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:13.410296+0000 mon.a (mon.0) 2296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:45:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:13.410401+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:45:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:13.410457+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e358: 8 total, 8 up, 6 in 2024-02-13T00:45:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:13.417940+0000 mon.c (mon.2) 356 : audit [INF] from='client.? 172.21.15.162:0/3100108434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84017-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:13.418579+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84017-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:13.419172+0000 mon.b (mon.1) 534 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:13.419612+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:13.523054+0000 mon.a (mon.0) 2301 : audit [INF] from='client.? 172.21.15.162:0/1767024614' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-13T00:45:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:13.523129+0000 mon.a (mon.0) 2302 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2024-02-13T00:45:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:13.396824+0000 mon.a (mon.0) 2295 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:45:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:13.410296+0000 mon.a (mon.0) 2296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:45:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:13.410401+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:45:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:13.410457+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e358: 8 total, 8 up, 6 in 2024-02-13T00:45:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:13.417940+0000 mon.c (mon.2) 356 : audit [INF] from='client.? 172.21.15.162:0/3100108434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84017-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:13.418579+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84017-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:13.419172+0000 mon.b (mon.1) 534 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:13.419612+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:14.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:13.523054+0000 mon.a (mon.0) 2301 : audit [INF] from='client.? 172.21.15.162:0/1767024614' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-13T00:45:14.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:13.523129+0000 mon.a (mon.0) 2302 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2024-02-13T00:45:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:13.396824+0000 mon.a (mon.0) 2295 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:45:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:13.410296+0000 mon.a (mon.0) 2296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:45:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:13.410401+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:45:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:13.410457+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e358: 8 total, 8 up, 6 in 2024-02-13T00:45:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:13.417940+0000 mon.c (mon.2) 356 : audit [INF] from='client.? 172.21.15.162:0/3100108434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84017-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:13.418579+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84017-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:13.419172+0000 mon.b (mon.1) 534 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:13.419612+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:13.523054+0000 mon.a (mon.0) 2301 : audit [INF] from='client.? 172.21.15.162:0/1767024614' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-13T00:45:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:13.523129+0000 mon.a (mon.0) 2302 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2024-02-13T00:45:14.850 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:45:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:14.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:45:14.956 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:14.987 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:15.007 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:15.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:15.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:15.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:15.139 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:15.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:15.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:15.394 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:15.414 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-02-13T00:45:15.414 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 1 0 2024-02-13T00:45:15.414 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 10 0 2024-02-13T00:45:15.414 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 13 0 2024-02-13T00:45:15.414 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 7 0 2024-02-13T00:45:15.414 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 14 0 2024-02-13T00:45:15.414 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 0 0 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 15 0 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 11 0 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 5 0 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 8 0 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 6 0 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 3 0 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 4 0 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 12 0 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 9 0 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 2 0 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (44 ms) 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] 3 tests from LibRadosListEC (1779 ms total) 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 2024-02-13T00:45:15.415 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-02-13T00:45:15.416 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-02-13T00:45:15.416 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2999 ms) 2024-02-13T00:45:15.416 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] 1 test from LibRadosListNP (2999 ms total) 2024-02-13T00:45:15.416 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 2024-02-13T00:45:15.416 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] Global test environment tear-down 2024-02-13T00:45:15.416 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [==========] 11 tests from 3 test suites ran. (260678 ms total) 2024-02-13T00:45:15.416 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ PASSED ] 11 tests. 2024-02-13T00:45:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84100 2024-02-13T00:45:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84100 2024-02-13T00:45:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84092 2024-02-13T00:45:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84092 2024-02-13T00:45:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84156 2024-02-13T00:45:15.418 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84156 2024-02-13T00:45:15.418 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:15.418 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=83980 2024-02-13T00:45:15.418 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 83980 2024-02-13T00:45:15.419 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:14.402571+0000 mgr.y (mgr.14544) 321 : cluster [DBG] pgmap v508: 278 pgs: 72 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:45:15.419 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:14.414744+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84017-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:15.419 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:14.414829+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-47"}]': finished 2024-02-13T00:45:15.419 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:14.414921+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? 172.21.15.162:0/1767024614' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-13T00:45:15.419 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:14.414970+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e359: 8 total, 8 up, 5 in 2024-02-13T00:45:15.419 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:14.419295+0000 mon.c (mon.2) 357 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:15.419 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:14.419533+0000 mon.b (mon.1) 535 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-47", "mode": "readproxy"}]: dispatch 2024-02-13T00:45:15.419 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:14.419676+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:15.420 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:14.421892+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-47", "mode": "readproxy"}]: dispatch 2024-02-13T00:45:15.420 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:14.475651+0000 mon.c (mon.2) 358 : audit [INF] from='client.? 172.21.15.162:0/3100108434' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-84017-3","pool2":"test-rados-api-smithi162-84017-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-13T00:45:15.420 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:14.475936+0000 mon.a (mon.0) 2309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-84017-3","pool2":"test-rados-api-smithi162-84017-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-13T00:45:15.420 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:15.382267+0000 mon.a (mon.0) 2310 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:45:15.420 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:15.391029+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]': finished 2024-02-13T00:45:15.420 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:15.391102+0000 mon.a (mon.0) 2312 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-47", "mode": "readproxy"}]': finished 2024-02-13T00:45:15.420 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:15.391229+0000 mon.a (mon.0) 2313 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-84017-3","pool2":"test-rados-api-smithi162-84017-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-13T00:45:15.420 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:15.391292+0000 mon.a (mon.0) 2314 : cluster [DBG] osdmap e360: 8 total, 8 up, 5 in 2024-02-13T00:45:15.420 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:15.395500+0000 mon.c (mon.2) 359 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:15.420 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:15.404758+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:15.455 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:15.488 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:15.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:15.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:15.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:15.619 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:15.637 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:15.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:15.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:15.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:14.402571+0000 mgr.y (mgr.14544) 321 : cluster [DBG] pgmap v508: 278 pgs: 72 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:45:15.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:14.414744+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84017-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:15.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:14.414829+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-47"}]': finished 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:14.414921+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? 172.21.15.162:0/1767024614' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:14.414970+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e359: 8 total, 8 up, 5 in 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:14.419295+0000 mon.c (mon.2) 357 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:14.419533+0000 mon.b (mon.1) 535 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-47", "mode": "readproxy"}]: dispatch 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:14.419676+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:14.421892+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-47", "mode": "readproxy"}]: dispatch 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:14.475651+0000 mon.c (mon.2) 358 : audit [INF] from='client.? 172.21.15.162:0/3100108434' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-84017-3","pool2":"test-rados-api-smithi162-84017-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:14.475936+0000 mon.a (mon.0) 2309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-84017-3","pool2":"test-rados-api-smithi162-84017-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:15.382267+0000 mon.a (mon.0) 2310 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:15.391029+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]': finished 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:15.391102+0000 mon.a (mon.0) 2312 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-47", "mode": "readproxy"}]': finished 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:15.391229+0000 mon.a (mon.0) 2313 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-84017-3","pool2":"test-rados-api-smithi162-84017-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:15.391292+0000 mon.a (mon.0) 2314 : cluster [DBG] osdmap e360: 8 total, 8 up, 5 in 2024-02-13T00:45:15.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:15.395500+0000 mon.c (mon.2) 359 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:15.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:15.404758+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:15.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:15.773 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:15.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:45:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:14.402571+0000 mgr.y (mgr.14544) 321 : cluster [DBG] pgmap v508: 278 pgs: 72 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-13T00:45:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:14.414744+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84017-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:14.414829+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-47"}]': finished 2024-02-13T00:45:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:14.414921+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? 172.21.15.162:0/1767024614' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-13T00:45:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:14.414970+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e359: 8 total, 8 up, 5 in 2024-02-13T00:45:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:14.419295+0000 mon.c (mon.2) 357 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:14.419533+0000 mon.b (mon.1) 535 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-47", "mode": "readproxy"}]: dispatch 2024-02-13T00:45:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:14.419676+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:14.421892+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-47", "mode": "readproxy"}]: dispatch 2024-02-13T00:45:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:14.475651+0000 mon.c (mon.2) 358 : audit [INF] from='client.? 172.21.15.162:0/3100108434' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-84017-3","pool2":"test-rados-api-smithi162-84017-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-13T00:45:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:14.475936+0000 mon.a (mon.0) 2309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-84017-3","pool2":"test-rados-api-smithi162-84017-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-13T00:45:15.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:15.382267+0000 mon.a (mon.0) 2310 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:45:15.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:15.391029+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-83985-61"}]': finished 2024-02-13T00:45:15.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:15.391102+0000 mon.a (mon.0) 2312 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-47", "mode": "readproxy"}]': finished 2024-02-13T00:45:15.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:15.391229+0000 mon.a (mon.0) 2313 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-84017-3","pool2":"test-rados-api-smithi162-84017-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-13T00:45:15.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:15.391292+0000 mon.a (mon.0) 2314 : cluster [DBG] osdmap e360: 8 total, 8 up, 5 in 2024-02-13T00:45:15.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:15.395500+0000 mon.c (mon.2) 359 : audit [INF] from='client.? 172.21.15.162:0/3575688978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:15.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:15.404758+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]: dispatch 2024-02-13T00:45:16.320 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:45:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:45:16] "GET /metrics HTTP/1.1" 200 189456 "" "Prometheus/2.33.4" 2024-02-13T00:45:16.335 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:16.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:16.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:16.469 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:16.490 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:16.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:16.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:15.555582+0000 mgr.y (mgr.14544) 322 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:16.404274+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]': finished 2024-02-13T00:45:16.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:16.404365+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e361: 8 total, 8 up, 5 in 2024-02-13T00:45:16.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:15.555582+0000 mgr.y (mgr.14544) 322 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:16.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:16.404274+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]': finished 2024-02-13T00:45:16.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:16.404365+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e361: 8 total, 8 up, 5 in 2024-02-13T00:45:16.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:16.623 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:15.555582+0000 mgr.y (mgr.14544) 322 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:16.404274+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-83985-61"}]': finished 2024-02-13T00:45:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:16.404365+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e361: 8 total, 8 up, 5 in 2024-02-13T00:45:16.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.835+0000 7f2444f03700 1 -- 172.21.15.162:0/1074010619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2440102170 msgr2=0x7f2440104560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:16.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.835+0000 7f2444f03700 1 --2- 172.21.15.162:0/1074010619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2440102170 0x7f2440104560 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 rx=0x7f2438009be0 tx=0x7f2438004d10).stop 2024-02-13T00:45:16.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.835+0000 7f2444f03700 1 -- 172.21.15.162:0/1074010619 shutdown_connections 2024-02-13T00:45:16.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.835+0000 7f2444f03700 1 --2- 172.21.15.162:0/1074010619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2440102170 0x7f2440104560 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:16.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.835+0000 7f2444f03700 1 --2- 172.21.15.162:0/1074010619 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24400f8c10 0x7f24400f8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:16.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.835+0000 7f2444f03700 1 --2- 172.21.15.162:0/1074010619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24400f9570 0x7f2440101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:16.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.835+0000 7f2444f03700 1 -- 172.21.15.162:0/1074010619 >> 172.21.15.162:0/1074010619 conn(0x7f24400f2740 msgr2=0x7f24400f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.835+0000 7f2444f03700 1 -- 172.21.15.162:0/1074010619 shutdown_connections 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.835+0000 7f2444f03700 1 -- 172.21.15.162:0/1074010619 wait complete. 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.835+0000 7f2444f03700 1 librados: starting msgr at 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.835+0000 7f2444f03700 1 librados: starting objecter 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f2444f03700 1 Processor -- start 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f2444f03700 1 -- start start 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f2444f03700 1 librados: setting wanted keys 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f2444f03700 1 librados: calling monclient init 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f2444f03700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24400f8c10 0x7f24400f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f2444f03700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24400f9570 0x7f24400f70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f2444f03700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2440102170 0x7f24400f75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f2444f03700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2440116590 con 0x7f24400f9570 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f2444f03700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2440116700 con 0x7f24400f8c10 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f2444f03700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24401168c0 con 0x7f2440102170 2024-02-13T00:45:16.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f243dd9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24400f9570 0x7f24400f70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f243dd9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24400f9570 0x7f24400f70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44896/0 (socket says 172.21.15.162:44896) 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f243dd9b700 1 -- 172.21.15.162:0/642465764 learned_addr learned my addr 172.21.15.162:0/642465764 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f243e59c700 1 --2- 172.21.15.162:0/642465764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24400f8c10 0x7f24400f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f243ed9d700 1 --2- 172.21.15.162:0/642465764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2440102170 0x7f24400f75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f243e59c700 1 -- 172.21.15.162:0/642465764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2440102170 msgr2=0x7f24400f75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f243e59c700 1 --2- 172.21.15.162:0/642465764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2440102170 0x7f24400f75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f243e59c700 1 -- 172.21.15.162:0/642465764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24400f9570 msgr2=0x7f24400f70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f243e59c700 1 --2- 172.21.15.162:0/642465764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24400f9570 0x7f24400f70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f243e59c700 1 -- 172.21.15.162:0/642465764 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24380097c0 con 0x7f24400f8c10 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.836+0000 7f243dd9b700 1 --2- 172.21.15.162:0/642465764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24400f9570 0x7f24400f70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.837+0000 7f243e59c700 1 --2- 172.21.15.162:0/642465764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24400f8c10 0x7f24400f6bc0 secure :-1 s=READY pgs=4854 cs=0 l=1 rev1=1 rx=0x7f243000e8f0 tx=0x7f243000b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.837+0000 7f242f7fe700 10 client.?.objecter ms_handle_connect 0x7f24400f8c10 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.837+0000 7f242f7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.837+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2430004620 con 0x7f24400f8c10 2024-02-13T00:45:16.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.837+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2440116ba0 con 0x7f24400f8c10 2024-02-13T00:45:16.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.837+0000 7f2444f03700 10 client.44356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:16.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.837+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24401170e0 con 0x7f24400f8c10 2024-02-13T00:45:16.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.837+0000 7f2444f03700 1 librados: init done 2024-02-13T00:45:16.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.837+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f243000f040 con 0x7f24400f8c10 2024-02-13T00:45:16.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.837+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2430014830 con 0x7f24400f8c10 2024-02-13T00:45:16.844 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.838+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f2420004f30 con 0x7f24400f8c10 2024-02-13T00:45:16.844 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.839+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f2430007b40 con 0x7f24400f8c10 2024-02-13T00:45:16.844 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.839+0000 7f242f7fe700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2428062c90 0x7f2428065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:16.844 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(361..361 src has 52..361) v4 ==== 8075+0+0 (secure 0 0 0) 0x7f243007cab0 con 0x7f24400f8c10 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 osd_map(361..361 src has 52..361) v4 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f242f7fe700 3 client.44356.objecter handle_osd_map got epochs [361,361] > 0 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f242f7fe700 3 client.44356.objecter handle_osd_map decoding full epoch 361 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f243dd9b700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2428062c90 0x7f2428065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f242f7fe700 20 client.44356.objecter dump_active .. 0 homeless 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v361) v1 ==== 54+0+16 (secure 0 0 0) 0x7f243004bec0 con 0x7f24400f8c10 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f2444f03700 10 client.44356.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 1 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6801/2103743338 conn(0x7f2420004560 0x7f24200049a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f242f7fe700 10 client.44356.objecter ms_handle_connect 0x7f2428062c90 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200051d0 osd=0 3 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200051d0 osd=0 3 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f243dd9b700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2428062c90 0x7f2428065150 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 rx=0x7f24400f8360 tx=0x7f243400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200051d0 osd=0 3 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 1 2024-02-13T00:45:16.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f24200051d0 osd=0 3 2024-02-13T00:45:16.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 0 1 2024-02-13T00:45:16.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200051d0 osd=0 4 2024-02-13T00:45:16.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f2444f03700 10 client.44356.objecter _send_command 1 2024-02-13T00:45:16.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2420008960 con 0x7f2420004560 2024-02-13T00:45:16.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f243ed9d700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6801/2103743338 conn(0x7f2420004560 0x7f24200049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:16.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f242f7fe700 10 client.44356.objecter ms_handle_connect 0x7f2420004560 2024-02-13T00:45:16.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.840+0000 7f243ed9d700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6801/2103743338 conn(0x7f2420004560 0x7f24200049a0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:16.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.841+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2420008960 con 0x7f2420004560 2024-02-13T00:45:16.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.841+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:45:16.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.841+0000 7f242f7fe700 10 client.44356.objecter _finish_command 1 = system:0 2024-02-13T00:45:16.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.841+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f24200051d0 osd=0 4 2024-02-13T00:45:16.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.841+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 0 1 2024-02-13T00:45:16.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:45:16.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.856+0000 7f2444f03700 10 client.44356.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:16.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.856+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 2 2024-02-13T00:45:16.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.856+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200051d0 osd=0 3 2024-02-13T00:45:16.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.856+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200051d0 osd=0 3 2024-02-13T00:45:16.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.856+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200051d0 osd=0 3 2024-02-13T00:45:16.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.856+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 2 2024-02-13T00:45:16.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.856+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f24200051d0 osd=0 3 2024-02-13T00:45:16.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.856+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 0 2 2024-02-13T00:45:16.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.856+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200051d0 osd=0 4 2024-02-13T00:45:16.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.856+0000 7f2444f03700 10 client.44356.objecter _send_command 2 2024-02-13T00:45:16.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.856+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2420003fa0 con 0x7f2420004560 2024-02-13T00:45:16.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.857+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f2420003fa0 con 0x7f2420004560 2024-02-13T00:45:16.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.857+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:16.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.857+0000 7f242f7fe700 10 client.44356.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:16.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.857+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f24200051d0 osd=0 4 2024-02-13T00:45:16.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.857+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 0 2 2024-02-13T00:45:16.861 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:45:16.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f2444f03700 10 client.44356.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:16.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 3 2024-02-13T00:45:16.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6805/150599780 conn(0x7f24200084c0 0x7f2420002740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:16.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200043a0 osd=1 3 2024-02-13T00:45:16.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200043a0 osd=1 3 2024-02-13T00:45:16.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200043a0 osd=1 3 2024-02-13T00:45:16.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 3 2024-02-13T00:45:16.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f24200043a0 osd=1 3 2024-02-13T00:45:16.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 1 3 2024-02-13T00:45:16.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200043a0 osd=1 4 2024-02-13T00:45:16.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f2444f03700 10 client.44356.objecter _send_command 3 2024-02-13T00:45:16.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2420002de0 con 0x7f24200084c0 2024-02-13T00:45:16.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f243e59c700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6805/150599780 conn(0x7f24200084c0 0x7f2420002740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:16.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f242f7fe700 10 client.44356.objecter ms_handle_connect 0x7f24200084c0 2024-02-13T00:45:16.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.858+0000 7f243e59c700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6805/150599780 conn(0x7f24200084c0 0x7f2420002740 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:16.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.859+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2420002de0 con 0x7f24200084c0 2024-02-13T00:45:16.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.859+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:45:16.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.859+0000 7f242f7fe700 10 client.44356.objecter _finish_command 3 = system:0 2024-02-13T00:45:16.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.859+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f24200043a0 osd=1 4 2024-02-13T00:45:16.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.859+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 1 3 2024-02-13T00:45:16.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.886+0000 7f2444f03700 10 client.44356.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:16.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.886+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 4 2024-02-13T00:45:16.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.886+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200043a0 osd=1 3 2024-02-13T00:45:16.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.886+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200043a0 osd=1 3 2024-02-13T00:45:16.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.886+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200043a0 osd=1 3 2024-02-13T00:45:16.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.886+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 4 2024-02-13T00:45:16.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.886+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f24200043a0 osd=1 3 2024-02-13T00:45:16.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.886+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 1 4 2024-02-13T00:45:16.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.886+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200043a0 osd=1 4 2024-02-13T00:45:16.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.886+0000 7f2444f03700 10 client.44356.objecter _send_command 4 2024-02-13T00:45:16.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.886+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2420007ae0 con 0x7f24200084c0 2024-02-13T00:45:16.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.889+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f2420007ae0 con 0x7f24200084c0 2024-02-13T00:45:16.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.889+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:16.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.889+0000 7f242f7fe700 10 client.44356.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:16.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.889+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f24200043a0 osd=1 4 2024-02-13T00:45:16.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.889+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 1 4 2024-02-13T00:45:16.893 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:45:16.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.889+0000 7f2444f03700 10 client.44356.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:16.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.889+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 5 2024-02-13T00:45:16.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.889+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6809/3689719376 conn(0x7f2420003940 0x7f242000d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:16.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.890+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200037d0 osd=2 3 2024-02-13T00:45:16.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.890+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200037d0 osd=2 3 2024-02-13T00:45:16.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.890+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200037d0 osd=2 3 2024-02-13T00:45:16.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.890+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 5 2024-02-13T00:45:16.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.890+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f24200037d0 osd=2 3 2024-02-13T00:45:16.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.890+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 2 5 2024-02-13T00:45:16.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.890+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200037d0 osd=2 4 2024-02-13T00:45:16.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.890+0000 7f2444f03700 10 client.44356.objecter _send_command 5 2024-02-13T00:45:16.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.890+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f242000fe40 con 0x7f2420003940 2024-02-13T00:45:16.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.891+0000 7f243dd9b700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6809/3689719376 conn(0x7f2420003940 0x7f242000d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:16.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.891+0000 7f243dd9b700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6809/3689719376 conn(0x7f2420003940 0x7f242000d000 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:16.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.891+0000 7f242f7fe700 10 client.44356.objecter ms_handle_connect 0x7f2420003940 2024-02-13T00:45:16.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.892+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f242000fe40 con 0x7f2420003940 2024-02-13T00:45:16.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.892+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:45:16.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.892+0000 7f242f7fe700 10 client.44356.objecter _finish_command 5 = system:0 2024-02-13T00:45:16.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.892+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f24200037d0 osd=2 4 2024-02-13T00:45:16.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.892+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 2 5 2024-02-13T00:45:16.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.922+0000 7f2444f03700 10 client.44356.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:16.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.922+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 6 2024-02-13T00:45:16.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.922+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200037d0 osd=2 3 2024-02-13T00:45:16.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.922+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200037d0 osd=2 3 2024-02-13T00:45:16.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.922+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200037d0 osd=2 3 2024-02-13T00:45:16.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.922+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 6 2024-02-13T00:45:16.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.922+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f24200037d0 osd=2 3 2024-02-13T00:45:16.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.922+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 2 6 2024-02-13T00:45:16.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.922+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200037d0 osd=2 4 2024-02-13T00:45:16.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.922+0000 7f2444f03700 10 client.44356.objecter _send_command 6 2024-02-13T00:45:16.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.922+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f24200032a0 con 0x7f2420003940 2024-02-13T00:45:16.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f24200032a0 con 0x7f2420003940 2024-02-13T00:45:16.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:16.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f242f7fe700 10 client.44356.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:16.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f24200037d0 osd=2 4 2024-02-13T00:45:16.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 2 6 2024-02-13T00:45:16.927 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:45:16.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f2444f03700 10 client.44356.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:16.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 7 2024-02-13T00:45:16.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6813/604971011 conn(0x7f242000e710 0x7f242000ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f2420001bb0 osd=3 3 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420001bb0 osd=3 3 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f2420001bb0 osd=3 3 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 7 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f2420001bb0 osd=3 3 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 3 7 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420001bb0 osd=3 4 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f2444f03700 10 client.44356.objecter _send_command 7 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24200023c0 con 0x7f242000e710 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f243ed9d700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6813/604971011 conn(0x7f242000e710 0x7f242000ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f243ed9d700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6813/604971011 conn(0x7f242000e710 0x7f242000ebc0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.924+0000 7f242f7fe700 10 client.44356.objecter ms_handle_connect 0x7f242000e710 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.925+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24200023c0 con 0x7f242000e710 2024-02-13T00:45:16.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.925+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:45:16.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.925+0000 7f242f7fe700 10 client.44356.objecter _finish_command 7 = system:0 2024-02-13T00:45:16.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.925+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f2420001bb0 osd=3 4 2024-02-13T00:45:16.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.925+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 3 7 2024-02-13T00:45:16.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.952+0000 7f2444f03700 10 client.44356.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:16.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.952+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 8 2024-02-13T00:45:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.952+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f2420001bb0 osd=3 3 2024-02-13T00:45:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.952+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420001bb0 osd=3 3 2024-02-13T00:45:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.952+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f2420001bb0 osd=3 3 2024-02-13T00:45:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.952+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 8 2024-02-13T00:45:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.952+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f2420001bb0 osd=3 3 2024-02-13T00:45:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.952+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 3 8 2024-02-13T00:45:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.952+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420001bb0 osd=3 4 2024-02-13T00:45:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.952+0000 7f2444f03700 10 client.44356.objecter _send_command 8 2024-02-13T00:45:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.952+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f24200023c0 con 0x7f242000e710 2024-02-13T00:45:16.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f24200023c0 con 0x7f242000e710 2024-02-13T00:45:16.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:16.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f242f7fe700 10 client.44356.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:16.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f2420001bb0 osd=3 4 2024-02-13T00:45:16.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 3 8 2024-02-13T00:45:16.958 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:45:16.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f2444f03700 10 client.44356.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:16.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 9 2024-02-13T00:45:16.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6800/3344263388 conn(0x7f242000f900 0x7f2420014080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:16.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200023c0 osd=4 3 2024-02-13T00:45:16.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200023c0 osd=4 3 2024-02-13T00:45:16.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200023c0 osd=4 3 2024-02-13T00:45:16.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 9 2024-02-13T00:45:16.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f24200023c0 osd=4 3 2024-02-13T00:45:16.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 4 9 2024-02-13T00:45:16.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200023c0 osd=4 4 2024-02-13T00:45:16.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f2444f03700 10 client.44356.objecter _send_command 9 2024-02-13T00:45:16.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2420016e00 con 0x7f242000f900 2024-02-13T00:45:16.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.955+0000 7f243e59c700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6800/3344263388 conn(0x7f242000f900 0x7f2420014080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:16.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.956+0000 7f242f7fe700 10 client.44356.objecter ms_handle_connect 0x7f242000f900 2024-02-13T00:45:16.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.956+0000 7f243e59c700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6800/3344263388 conn(0x7f242000f900 0x7f2420014080 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:16.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.956+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2420016e00 con 0x7f242000f900 2024-02-13T00:45:16.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.956+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:45:16.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.956+0000 7f242f7fe700 10 client.44356.objecter _finish_command 9 = system:0 2024-02-13T00:45:16.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.956+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f24200023c0 osd=4 4 2024-02-13T00:45:16.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.956+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 4 9 2024-02-13T00:45:16.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.986+0000 7f2444f03700 10 client.44356.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:16.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.986+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 10 2024-02-13T00:45:16.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.986+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200023c0 osd=4 3 2024-02-13T00:45:16.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.986+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200023c0 osd=4 3 2024-02-13T00:45:16.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.986+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f24200023c0 osd=4 3 2024-02-13T00:45:16.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.986+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 10 2024-02-13T00:45:16.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.986+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f24200023c0 osd=4 3 2024-02-13T00:45:16.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.986+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 4 10 2024-02-13T00:45:16.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.986+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200023c0 osd=4 4 2024-02-13T00:45:16.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.986+0000 7f2444f03700 10 client.44356.objecter _send_command 10 2024-02-13T00:45:16.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.986+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2420016e00 con 0x7f242000f900 2024-02-13T00:45:16.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.987+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f2420016e00 con 0x7f242000f900 2024-02-13T00:45:16.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.987+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:16.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.987+0000 7f242f7fe700 10 client.44356.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:16.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.987+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f24200023c0 osd=4 4 2024-02-13T00:45:16.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.987+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 4 10 2024-02-13T00:45:16.991 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:45:16.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f2444f03700 10 client.44356.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:16.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 11 2024-02-13T00:45:16.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6804/2212296748 conn(0x7f2420014b20 0x7f2420014f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:16.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f2420016e00 osd=5 3 2024-02-13T00:45:16.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420016e00 osd=5 3 2024-02-13T00:45:16.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f2420016e00 osd=5 3 2024-02-13T00:45:16.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 11 2024-02-13T00:45:16.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f2420016e00 osd=5 3 2024-02-13T00:45:16.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 5 11 2024-02-13T00:45:16.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420016e00 osd=5 4 2024-02-13T00:45:16.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f2444f03700 10 client.44356.objecter _send_command 11 2024-02-13T00:45:16.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2420016c00 con 0x7f2420014b20 2024-02-13T00:45:16.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f243dd9b700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6804/2212296748 conn(0x7f2420014b20 0x7f2420014f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:16.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f242f7fe700 10 client.44356.objecter ms_handle_connect 0x7f2420014b20 2024-02-13T00:45:16.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.988+0000 7f243dd9b700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6804/2212296748 conn(0x7f2420014b20 0x7f2420014f80 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:16.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.989+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2420016c00 con 0x7f2420014b20 2024-02-13T00:45:16.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.989+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:45:16.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.989+0000 7f242f7fe700 10 client.44356.objecter _finish_command 11 = system:0 2024-02-13T00:45:16.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.989+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f2420016e00 osd=5 4 2024-02-13T00:45:16.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:16.989+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 5 11 2024-02-13T00:45:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.016+0000 7f2444f03700 10 client.44356.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.016+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 12 2024-02-13T00:45:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.016+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f2420016e00 osd=5 3 2024-02-13T00:45:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.016+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420016e00 osd=5 3 2024-02-13T00:45:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.016+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f2420016e00 osd=5 3 2024-02-13T00:45:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.016+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 12 2024-02-13T00:45:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.016+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f2420016e00 osd=5 3 2024-02-13T00:45:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.016+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 5 12 2024-02-13T00:45:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.016+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420016e00 osd=5 4 2024-02-13T00:45:17.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.016+0000 7f2444f03700 10 client.44356.objecter _send_command 12 2024-02-13T00:45:17.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.016+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f2420016c00 con 0x7f2420014b20 2024-02-13T00:45:17.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.017+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f2420016c00 con 0x7f2420014b20 2024-02-13T00:45:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.017+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.017+0000 7f242f7fe700 10 client.44356.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.017+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f2420016e00 osd=5 4 2024-02-13T00:45:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.017+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 5 12 2024-02-13T00:45:17.021 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:45:17.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f2444f03700 10 client.44356.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:17.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 13 2024-02-13T00:45:17.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6808/1506640341 conn(0x7f2420019580 0x7f24200199a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:17.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f2420016c00 osd=6 3 2024-02-13T00:45:17.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420016c00 osd=6 3 2024-02-13T00:45:17.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f2420016c00 osd=6 3 2024-02-13T00:45:17.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 13 2024-02-13T00:45:17.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f2420016c00 osd=6 3 2024-02-13T00:45:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 6 13 2024-02-13T00:45:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420016c00 osd=6 4 2024-02-13T00:45:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f2444f03700 10 client.44356.objecter _send_command 13 2024-02-13T00:45:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f242001a040 con 0x7f2420019580 2024-02-13T00:45:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f243ed9d700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6808/1506640341 conn(0x7f2420019580 0x7f24200199a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f242f7fe700 10 client.44356.objecter ms_handle_connect 0x7f2420019580 2024-02-13T00:45:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.018+0000 7f243ed9d700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6808/1506640341 conn(0x7f2420019580 0x7f24200199a0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.019+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f242001a040 con 0x7f2420019580 2024-02-13T00:45:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.019+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:45:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.019+0000 7f242f7fe700 10 client.44356.objecter _finish_command 13 = system:0 2024-02-13T00:45:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.019+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f2420016c00 osd=6 4 2024-02-13T00:45:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.019+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 6 13 2024-02-13T00:45:17.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.045+0000 7f2444f03700 10 client.44356.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:17.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.045+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 14 2024-02-13T00:45:17.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.045+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f2420016c00 osd=6 3 2024-02-13T00:45:17.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.045+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420016c00 osd=6 3 2024-02-13T00:45:17.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.045+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f2420016c00 osd=6 3 2024-02-13T00:45:17.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.045+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 14 2024-02-13T00:45:17.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.045+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f2420016c00 osd=6 3 2024-02-13T00:45:17.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.045+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 6 14 2024-02-13T00:45:17.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.045+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420016c00 osd=6 4 2024-02-13T00:45:17.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.045+0000 7f2444f03700 10 client.44356.objecter _send_command 14 2024-02-13T00:45:17.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.046+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f242001a040 con 0x7f2420019580 2024-02-13T00:45:17.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f242001a040 con 0x7f2420019580 2024-02-13T00:45:17.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:17.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f242f7fe700 10 client.44356.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:17.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f2420016c00 osd=6 4 2024-02-13T00:45:17.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 6 14 2024-02-13T00:45:17.051 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:45:17.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f2444f03700 10 client.44356.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 15 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6812/1964192772 conn(0x7f242001c800 0x7f242001cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f242001a040 osd=7 3 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f242001a040 osd=7 3 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f242001a040 osd=7 3 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 15 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f242001a040 osd=7 3 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 7 15 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f242001a040 osd=7 4 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f2444f03700 10 client.44356.objecter _send_command 15 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.047+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f242001d2b0 con 0x7f242001c800 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.048+0000 7f243e59c700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6812/1964192772 conn(0x7f242001c800 0x7f242001cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:17.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.048+0000 7f242f7fe700 10 client.44356.objecter ms_handle_connect 0x7f242001c800 2024-02-13T00:45:17.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.048+0000 7f243e59c700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6812/1964192772 conn(0x7f242001c800 0x7f242001cc40 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:17.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.048+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f242001d2b0 con 0x7f242001c800 2024-02-13T00:45:17.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.049+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:45:17.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.049+0000 7f242f7fe700 10 client.44356.objecter _finish_command 15 = system:0 2024-02-13T00:45:17.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.049+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f242001a040 osd=7 4 2024-02-13T00:45:17.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.049+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 7 15 2024-02-13T00:45:17.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.076+0000 7f2444f03700 10 client.44356.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:17.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.076+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign -1 16 2024-02-13T00:45:17.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.076+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f242001a040 osd=7 3 2024-02-13T00:45:17.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.076+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f242001a040 osd=7 3 2024-02-13T00:45:17.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.076+0000 7f2444f03700 20 client.44356.objecter _get_session s=0x7f242001a040 osd=7 3 2024-02-13T00:45:17.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.076+0000 7f2444f03700 15 client.44356.objecter _session_command_op_remove -1 16 2024-02-13T00:45:17.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.076+0000 7f2444f03700 20 client.44356.objecter get_session s=0x7f242001a040 osd=7 3 2024-02-13T00:45:17.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.076+0000 7f2444f03700 15 client.44356.objecter _session_command_op_assign 7 16 2024-02-13T00:45:17.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.076+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f242001a040 osd=7 4 2024-02-13T00:45:17.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.076+0000 7f2444f03700 10 client.44356.objecter _send_command 16 2024-02-13T00:45:17.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.076+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f242001d2b0 con 0x7f242001c800 2024-02-13T00:45:17.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.077+0000 7f242f7fe700 1 -- 172.21.15.162:0/642465764 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f242001d2b0 con 0x7f242001c800 2024-02-13T00:45:17.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.077+0000 7f242f7fe700 10 client.44356.objecter ms_dispatch 0x7f2440001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:17.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.077+0000 7f242f7fe700 10 client.44356.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:17.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.077+0000 7f242f7fe700 20 client.44356.objecter put_session s=0x7f242001a040 osd=7 4 2024-02-13T00:45:17.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.077+0000 7f242f7fe700 15 client.44356.objecter _session_command_op_remove 7 16 2024-02-13T00:45:17.081 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:45:17.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 10 librados: watch_flush enter 2024-02-13T00:45:17.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 10 librados: watch_flush exit 2024-02-13T00:45:17.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 10 client.44356.objecter close_session for osd.0 2024-02-13T00:45:17.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6801/2103743338 conn(0x7f2420004560 msgr2=0x7f24200049a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:17.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6801/2103743338 conn(0x7f2420004560 0x7f24200049a0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200051d0 osd=0 1 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 10 client.44356.objecter close_session for osd.1 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6805/150599780 conn(0x7f24200084c0 msgr2=0x7f2420002740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6805/150599780 conn(0x7f24200084c0 0x7f2420002740 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200043a0 osd=1 1 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 10 client.44356.objecter close_session for osd.2 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6809/3689719376 conn(0x7f2420003940 msgr2=0x7f242000d000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6809/3689719376 conn(0x7f2420003940 0x7f242000d000 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200037d0 osd=2 1 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 10 client.44356.objecter close_session for osd.3 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6813/604971011 conn(0x7f242000e710 msgr2=0x7f242000ebc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6813/604971011 conn(0x7f242000e710 0x7f242000ebc0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:17.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420001bb0 osd=3 1 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 10 client.44356.objecter close_session for osd.4 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6800/3344263388 conn(0x7f242000f900 msgr2=0x7f2420014080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6800/3344263388 conn(0x7f242000f900 0x7f2420014080 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f243ed9d700 1 -- 172.21.15.162:0/642465764 reap_dead start 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f24200023c0 osd=4 1 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 10 client.44356.objecter close_session for osd.5 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6804/2212296748 conn(0x7f2420014b20 msgr2=0x7f2420014f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6804/2212296748 conn(0x7f2420014b20 0x7f2420014f80 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f243ed9d700 1 -- 172.21.15.162:0/642465764 reap_dead start 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420016e00 osd=5 1 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 10 client.44356.objecter close_session for osd.6 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6808/1506640341 conn(0x7f2420019580 msgr2=0x7f24200199a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6808/1506640341 conn(0x7f2420019580 0x7f24200199a0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f2420016c00 osd=6 1 2024-02-13T00:45:17.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 10 client.44356.objecter close_session for osd.7 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6812/1964192772 conn(0x7f242001c800 msgr2=0x7f242001cc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.181:6812/1964192772 conn(0x7f242001c800 0x7f242001cc40 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 20 client.44356.objecter put_session s=0x7f242001a040 osd=7 1 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 20 client.44356.objecter shutdown clearing up homeless session... 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f243ed9d700 1 -- 172.21.15.162:0/642465764 reap_dead start 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 10 client.44356.objecter successfully canceled tick 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2428062c90 msgr2=0x7f2428065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2428062c90 0x7f2428065150 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 rx=0x7f24400f8360 tx=0x7f243400b040).stop 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24400f8c10 msgr2=0x7f24400f6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24400f8c10 0x7f24400f6bc0 secure :-1 s=READY pgs=4854 cs=0 l=1 rev1=1 rx=0x7f243000e8f0 tx=0x7f243000b040).stop 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 shutdown_connections 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24400f8c10 0x7f24400f6bc0 unknown :-1 s=CLOSED pgs=4854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 --2- 172.21.15.162:0/642465764 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2428062c90 0x7f2428065150 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 >> 172.21.15.162:0/642465764 conn(0x7f24400f2740 msgr2=0x7f24400f3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 shutdown_connections 2024-02-13T00:45:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 -- 172.21.15.162:0/642465764 wait complete. 2024-02-13T00:45:17.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:17.078+0000 7f2444f03700 1 librados: shutdown 2024-02-13T00:45:17.086 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:45:17.086 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:45:17.086 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:45:17.086 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:45:17.086 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:45:17.086 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:45:17.086 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:45:17.086 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:45:17.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:16.405024+0000 mgr.y (mgr.14544) 323 : cluster [DBG] pgmap v511: 238 pgs: 125 peering, 113 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:45:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:16.433937+0000 mon.c (mon.2) 360 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:16.434642+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:16.435400+0000 mon.c (mon.2) 361 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:16.435846+0000 mon.a (mon.0) 2319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:16.436303+0000 mon.c (mon.2) 362 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-83985-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:16.436626+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-83985-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:16.840480+0000 mon.b (mon.1) 536 : audit [DBG] from='client.? 172.21.15.162:0/642465764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:17.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:17.400652+0000 mon.a (mon.0) 2321 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-13T00:45:17.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:16.405024+0000 mgr.y (mgr.14544) 323 : cluster [DBG] pgmap v511: 238 pgs: 125 peering, 113 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:45:17.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:16.433937+0000 mon.c (mon.2) 360 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:17.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:16.434642+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:17.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:16.435400+0000 mon.c (mon.2) 361 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:17.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:16.435846+0000 mon.a (mon.0) 2319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:17.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:16.436303+0000 mon.c (mon.2) 362 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-83985-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:17.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:16.436626+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-83985-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:17.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:16.840480+0000 mon.b (mon.1) 536 : audit [DBG] from='client.? 172.21.15.162:0/642465764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:17.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:17.400652+0000 mon.a (mon.0) 2321 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-13T00:45:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:16.405024+0000 mgr.y (mgr.14544) 323 : cluster [DBG] pgmap v511: 238 pgs: 125 peering, 113 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:45:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:16.433937+0000 mon.c (mon.2) 360 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:16.434642+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:16.435400+0000 mon.c (mon.2) 361 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:16.435846+0000 mon.a (mon.0) 2319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:16.436303+0000 mon.c (mon.2) 362 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-83985-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:16.436626+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-83985-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:16.840480+0000 mon.b (mon.1) 536 : audit [DBG] from='client.? 172.21.15.162:0/642465764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:17.400652+0000 mon.a (mon.0) 2321 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-13T00:45:18.474 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:18.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:18.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:45:18.610 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:18.665 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:45:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:18.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:45:18.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:17.420411+0000 mon.a (mon.0) 2322 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-83985-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:18.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:17.420489+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e362: 8 total, 8 up, 5 in 2024-02-13T00:45:18.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:17.423692+0000 mon.c (mon.2) 363 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-83985-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:18.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:17.428507+0000 mon.a (mon.0) 2324 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-83985-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:18.694 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:17.420411+0000 mon.a (mon.0) 2322 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-83985-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:18.694 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:17.420489+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e362: 8 total, 8 up, 5 in 2024-02-13T00:45:18.694 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:17.423692+0000 mon.c (mon.2) 363 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-83985-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:18.694 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:17.428507+0000 mon.a (mon.0) 2324 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-83985-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:18.706 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:18.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:18.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:17.420411+0000 mon.a (mon.0) 2322 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-83985-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:18.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:17.420489+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e362: 8 total, 8 up, 5 in 2024-02-13T00:45:18.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:17.423692+0000 mon.c (mon.2) 363 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-83985-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:18.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:17.428507+0000 mon.a (mon.0) 2324 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-83985-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:18.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:45:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:45:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:45:18.800 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:18.818 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:18.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:18.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:18.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:18.949 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:18.970 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:18.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:19.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:19.100 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:19.119 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:19.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:19.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:19.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:19.248 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:19.267 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:19.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:19.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:19.370 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:19.396 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:19.415 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:19.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:19.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:19.551 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:19.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:18.405508+0000 mgr.y (mgr.14544) 324 : cluster [DBG] pgmap v514: 238 pgs: 125 peering, 113 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:45:19.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:18.684644+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e363: 8 total, 8 up, 5 in 2024-02-13T00:45:19.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:19.334908+0000 mon.a (mon.0) 2326 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:19.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:18.405508+0000 mgr.y (mgr.14544) 324 : cluster [DBG] pgmap v514: 238 pgs: 125 peering, 113 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:45:19.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:18.684644+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e363: 8 total, 8 up, 5 in 2024-02-13T00:45:19.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:19.334908+0000 mon.a (mon.0) 2326 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:19.580 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:45:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-rgw-foo-a[66295]: debug 2024-02-13T00:45:19.302+0000 7fa24b3c3700 -1 RGWWatcher::handle_error cookie 94514175195904 err (110) Connection timed out 2024-02-13T00:45:19.580 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:45:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-rgw-foo-a[66295]: debug 2024-02-13T00:45:19.303+0000 7fa24b3c3700 -1 RGWWatcher::handle_error cookie 94514175197056 err (110) Connection timed out 2024-02-13T00:45:19.581 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:45:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-rgw-foo-a[66295]: debug 2024-02-13T00:45:19.303+0000 7fa24b3c3700 -1 RGWWatcher::handle_error cookie 94514175188992 err (110) Connection timed out 2024-02-13T00:45:19.628 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:45:19.628 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:45:19.628 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:45:19.628 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:45:19.628 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:45:19.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:18.405508+0000 mgr.y (mgr.14544) 324 : cluster [DBG] pgmap v514: 238 pgs: 125 peering, 113 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:45:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:18.684644+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e363: 8 total, 8 up, 5 in 2024-02-13T00:45:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:19.334908+0000 mon.a (mon.0) 2326 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:20.355 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-13T00:45:20.402 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:20.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:20.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:45:20.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:20.536 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:20.555 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:20.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:20.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:20.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:19.442024+0000 mon.a (mon.0) 2327 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-83985-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-83985-62"}]': finished 2024-02-13T00:45:20.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:19.442113+0000 mon.a (mon.0) 2328 : cluster [DBG] osdmap e364: 8 total, 8 up, 5 in 2024-02-13T00:45:20.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:20.389172+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e365: 8 total, 8 up, 5 in 2024-02-13T00:45:20.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:19.442024+0000 mon.a (mon.0) 2327 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-83985-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-83985-62"}]': finished 2024-02-13T00:45:20.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:19.442113+0000 mon.a (mon.0) 2328 : cluster [DBG] osdmap e364: 8 total, 8 up, 5 in 2024-02-13T00:45:20.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:20.389172+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e365: 8 total, 8 up, 5 in 2024-02-13T00:45:20.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:20.684 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:20.703 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:20.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:20.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:19.442024+0000 mon.a (mon.0) 2327 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-83985-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-83985-62"}]': finished 2024-02-13T00:45:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:19.442113+0000 mon.a (mon.0) 2328 : cluster [DBG] osdmap e364: 8 total, 8 up, 5 in 2024-02-13T00:45:20.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:20.389172+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e365: 8 total, 8 up, 5 in 2024-02-13T00:45:20.806 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:20.831 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:20.850 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:20.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:20.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:20.979 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:20.997 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:21.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:21.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:21.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:45:21.128 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:21.375 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:45:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:21.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:45:21.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:21.510 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:21.528 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:21.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:21.603 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:20.405991+0000 mgr.y (mgr.14544) 325 : cluster [DBG] pgmap v518: 246 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering, 4 active+recovery_wait+degraded, 8 creating+peering, 230 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s; 17/1107 objects degraded (1.536%); 11/1107 objects misplaced (0.994%); 574 B/s, 92 objects/s recovering 2024-02-13T00:45:21.603 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:21.398995+0000 mon.a (mon.0) 2330 : cluster [WRN] Health check failed: Degraded data redundancy: 17/1107 objects degraded (1.536%), 4 pgs degraded (PG_DEGRADED) 2024-02-13T00:45:21.603 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:21.399028+0000 mon.a (mon.0) 2331 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-13T00:45:21.603 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:21.411105+0000 mon.a (mon.0) 2332 : cluster [DBG] osdmap e366: 8 total, 8 up, 5 in 2024-02-13T00:45:21.603 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:21.414166+0000 mon.c (mon.2) 364 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:21.603 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:21.414670+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:21.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:20.405991+0000 mgr.y (mgr.14544) 325 : cluster [DBG] pgmap v518: 246 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering, 4 active+recovery_wait+degraded, 8 creating+peering, 230 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s; 17/1107 objects degraded (1.536%); 11/1107 objects misplaced (0.994%); 574 B/s, 92 objects/s recovering 2024-02-13T00:45:21.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:21.398995+0000 mon.a (mon.0) 2330 : cluster [WRN] Health check failed: Degraded data redundancy: 17/1107 objects degraded (1.536%), 4 pgs degraded (PG_DEGRADED) 2024-02-13T00:45:21.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:21.399028+0000 mon.a (mon.0) 2331 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-13T00:45:21.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:21.411105+0000 mon.a (mon.0) 2332 : cluster [DBG] osdmap e366: 8 total, 8 up, 5 in 2024-02-13T00:45:21.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:21.414166+0000 mon.c (mon.2) 364 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:21.604 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:21.414670+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:21.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:21.657 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:21.676 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:21.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:21.779 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:21.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:20.405991+0000 mgr.y (mgr.14544) 325 : cluster [DBG] pgmap v518: 246 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering, 4 active+recovery_wait+degraded, 8 creating+peering, 230 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s; 17/1107 objects degraded (1.536%); 11/1107 objects misplaced (0.994%); 574 B/s, 92 objects/s recovering 2024-02-13T00:45:21.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:21.398995+0000 mon.a (mon.0) 2330 : cluster [WRN] Health check failed: Degraded data redundancy: 17/1107 objects degraded (1.536%), 4 pgs degraded (PG_DEGRADED) 2024-02-13T00:45:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:21.399028+0000 mon.a (mon.0) 2331 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-13T00:45:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:21.411105+0000 mon.a (mon.0) 2332 : cluster [DBG] osdmap e366: 8 total, 8 up, 5 in 2024-02-13T00:45:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:21.414166+0000 mon.c (mon.2) 364 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:21.414670+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:21.811 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:21.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.828+0000 7fba7f7da700 1 -- 172.21.15.162:0/916852981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba780fcb10 msgr2=0x7fba780fcef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:21.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.828+0000 7fba7f7da700 1 --2- 172.21.15.162:0/916852981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba780fcb10 0x7fba780fcef0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7fba74004870 tx=0x7fba74009cb0).stop 2024-02-13T00:45:21.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.828+0000 7fba7f7da700 1 -- 172.21.15.162:0/916852981 shutdown_connections 2024-02-13T00:45:21.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.828+0000 7fba7f7da700 1 --2- 172.21.15.162:0/916852981 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba780f7380 0x7fba78103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:21.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 --2- 172.21.15.162:0/916852981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba780fcb10 0x7fba780fcef0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:21.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 --2- 172.21.15.162:0/916852981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba780fd470 0x7fba780f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:21.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 -- 172.21.15.162:0/916852981 >> 172.21.15.162:0/916852981 conn(0x7fba780f2740 msgr2=0x7fba780f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:21.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 -- 172.21.15.162:0/916852981 shutdown_connections 2024-02-13T00:45:21.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 -- 172.21.15.162:0/916852981 wait complete. 2024-02-13T00:45:21.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 librados: starting msgr at 2024-02-13T00:45:21.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 librados: starting objecter 2024-02-13T00:45:21.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 Processor -- start 2024-02-13T00:45:21.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 -- start start 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 librados: setting wanted keys 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 librados: calling monclient init 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba780f7380 0x7fba7810cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba780fcb10 0x7fba7810d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba780fd470 0x7fba78107130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba78107620 con 0x7fba780fd470 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba78107790 con 0x7fba780f7380 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.829+0000 7fba7f7da700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba78107950 con 0x7fba780fcb10 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7dd77700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba780fd470 0x7fba78107130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7dd77700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba780fd470 0x7fba78107130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44916/0 (socket says 172.21.15.162:44916) 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7dd77700 1 -- 172.21.15.162:0/3297476539 learned_addr learned my addr 172.21.15.162:0/3297476539 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7cd75700 1 --2- 172.21.15.162:0/3297476539 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba780fcb10 0x7fba7810d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7d576700 1 --2- 172.21.15.162:0/3297476539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba780f7380 0x7fba7810cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:21.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7dd77700 1 -- 172.21.15.162:0/3297476539 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba780fcb10 msgr2=0x7fba7810d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:21.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7dd77700 1 --2- 172.21.15.162:0/3297476539 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba780fcb10 0x7fba7810d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:21.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7dd77700 1 -- 172.21.15.162:0/3297476539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba780f7380 msgr2=0x7fba7810cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:21.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7dd77700 1 --2- 172.21.15.162:0/3297476539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba780f7380 0x7fba7810cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:21.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7dd77700 1 -- 172.21.15.162:0/3297476539 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba74005040 con 0x7fba780fd470 2024-02-13T00:45:21.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7dd77700 1 --2- 172.21.15.162:0/3297476539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba780fd470 0x7fba78107130 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7fba7000ec20 tx=0x7fba7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:21.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba6e7fc700 10 client.?.objecter ms_handle_connect 0x7fba780fd470 2024-02-13T00:45:21.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba6e7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:21.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba6e7fc700 1 -- 172.21.15.162:0/3297476539 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba70009950 con 0x7fba780fd470 2024-02-13T00:45:21.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba6e7fc700 1 -- 172.21.15.162:0/3297476539 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba7000f040 con 0x7fba780fd470 2024-02-13T00:45:21.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7f7da700 1 -- 172.21.15.162:0/3297476539 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba78107bd0 con 0x7fba780fd470 2024-02-13T00:45:21.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba6e7fc700 1 -- 172.21.15.162:0/3297476539 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba70007df0 con 0x7fba780fd470 2024-02-13T00:45:21.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7f7da700 10 client.27207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:21.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7f7da700 1 -- 172.21.15.162:0/3297476539 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba78113dc0 con 0x7fba780fd470 2024-02-13T00:45:21.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.830+0000 7fba7f7da700 1 librados: init done 2024-02-13T00:45:21.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.831+0000 7fba7f7da700 1 -- 172.21.15.162:0/3297476539 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba780607d0 con 0x7fba780fd470 2024-02-13T00:45:21.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.832+0000 7fba6e7fc700 1 -- 172.21.15.162:0/3297476539 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fba700092e0 con 0x7fba780fd470 2024-02-13T00:45:21.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.832+0000 7fba6e7fc700 1 --2- 172.21.15.162:0/3297476539 >> v2:172.21.15.162:6800/2866534053 conn(0x7fba64062c90 0x7fba64065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:21.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.832+0000 7fba6e7fc700 1 -- 172.21.15.162:0/3297476539 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(366..366 src has 52..366) v4 ==== 8441+0+0 (secure 0 0 0) 0x7fba7007d050 con 0x7fba780fd470 2024-02-13T00:45:21.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.832+0000 7fba6e7fc700 10 client.27207.objecter ms_dispatch 0x7fba78001550 osd_map(366..366 src has 52..366) v4 2024-02-13T00:45:21.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.832+0000 7fba6e7fc700 3 client.27207.objecter handle_osd_map got epochs [366,366] > 0 2024-02-13T00:45:21.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.832+0000 7fba6e7fc700 3 client.27207.objecter handle_osd_map decoding full epoch 366 2024-02-13T00:45:21.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.832+0000 7fba7d576700 1 --2- 172.21.15.162:0/3297476539 >> v2:172.21.15.162:6800/2866534053 conn(0x7fba64062c90 0x7fba64065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:21.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.832+0000 7fba6e7fc700 20 client.27207.objecter dump_active .. 0 homeless 2024-02-13T00:45:21.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.832+0000 7fba7d576700 1 --2- 172.21.15.162:0/3297476539 >> v2:172.21.15.162:6800/2866534053 conn(0x7fba64062c90 0x7fba64065150 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 rx=0x7fba740043d0 tx=0x7fba7400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:21.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.832+0000 7fba6e7fc700 10 client.27207.objecter ms_handle_connect 0x7fba64062c90 2024-02-13T00:45:21.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:21.836+0000 7fba6e7fc700 1 -- 172.21.15.162:0/3297476539 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fba70049f60 con 0x7fba780fd470 2024-02-13T00:45:21.862 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:21.863 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:21.865 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:21.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:21.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:21.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:22.000 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:22.018 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:22.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:22.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.020+0000 7fba7f7da700 1 -- 172.21.15.162:0/3297476539 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fba78106090 con 0x7fba780fd470 2024-02-13T00:45:22.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:22.297 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:22.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:22.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:22.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.413+0000 7fba6e7fc700 1 -- 172.21.15.162:0/3297476539 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v367) v1 ==== 91+0+0 (secure 0 0 0) 0x7fba700503a0 con 0x7fba780fd470 2024-02-13T00:45:22.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.416+0000 7fba7f7da700 10 librados: watch_flush enter 2024-02-13T00:45:22.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.416+0000 7fba7f7da700 10 librados: watch_flush exit 2024-02-13T00:45:22.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.416+0000 7fba7f7da700 20 client.27207.objecter shutdown clearing up homeless session... 2024-02-13T00:45:22.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.416+0000 7fba7f7da700 10 client.27207.objecter successfully canceled tick 2024-02-13T00:45:22.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.416+0000 7fba7f7da700 1 -- 172.21.15.162:0/3297476539 >> v2:172.21.15.162:6800/2866534053 conn(0x7fba64062c90 msgr2=0x7fba64065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:22.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.416+0000 7fba7f7da700 1 --2- 172.21.15.162:0/3297476539 >> v2:172.21.15.162:6800/2866534053 conn(0x7fba64062c90 0x7fba64065150 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 rx=0x7fba740043d0 tx=0x7fba7400b040).stop 2024-02-13T00:45:22.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.416+0000 7fba7f7da700 1 -- 172.21.15.162:0/3297476539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba780fd470 msgr2=0x7fba78107130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:22.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.416+0000 7fba7f7da700 1 --2- 172.21.15.162:0/3297476539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba780fd470 0x7fba78107130 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7fba7000ec20 tx=0x7fba7000b040).stop 2024-02-13T00:45:22.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.417+0000 7fba7f7da700 1 -- 172.21.15.162:0/3297476539 shutdown_connections 2024-02-13T00:45:22.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.417+0000 7fba7f7da700 1 --2- 172.21.15.162:0/3297476539 >> v2:172.21.15.162:6800/2866534053 conn(0x7fba64062c90 0x7fba64065150 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:22.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.417+0000 7fba7f7da700 1 --2- 172.21.15.162:0/3297476539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba780fd470 0x7fba78107130 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:22.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.417+0000 7fba7f7da700 1 --2- 172.21.15.162:0/3297476539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba780f7380 0x7fba7810cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:22.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.417+0000 7fba7f7da700 1 --2- 172.21.15.162:0/3297476539 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba780fcb10 0x7fba7810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:22.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.417+0000 7fba7f7da700 1 -- 172.21.15.162:0/3297476539 >> 172.21.15.162:0/3297476539 conn(0x7fba780f2740 msgr2=0x7fba780f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:22.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.417+0000 7fba7f7da700 1 -- 172.21.15.162:0/3297476539 shutdown_connections 2024-02-13T00:45:22.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.420+0000 7fba7f7da700 1 -- 172.21.15.162:0/3297476539 wait complete. 2024-02-13T00:45:22.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:22.421+0000 7fba7f7da700 1 librados: shutdown 2024-02-13T00:45:22.425 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:45:22.438 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:22.021964+0000 mon.a (mon.0) 2334 : audit [INF] from='client.? 172.21.15.162:0/3297476539' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:45:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:22.406421+0000 mon.a (mon.0) 2335 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:45:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:22.410787+0000 mon.a (mon.0) 2336 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]': finished 2024-02-13T00:45:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:22.410853+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? 172.21.15.162:0/3297476539' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:45:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:22.410886+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e367: 8 total, 8 up, 5 in 2024-02-13T00:45:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:22.412822+0000 mon.c (mon.2) 365 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:22.414159+0000 mon.a (mon.0) 2339 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:22.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:22.021964+0000 mon.a (mon.0) 2334 : audit [INF] from='client.? 172.21.15.162:0/3297476539' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:45:22.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:22.406421+0000 mon.a (mon.0) 2335 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:45:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:22.410787+0000 mon.a (mon.0) 2336 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]': finished 2024-02-13T00:45:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:22.410853+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? 172.21.15.162:0/3297476539' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:45:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:22.410886+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e367: 8 total, 8 up, 5 in 2024-02-13T00:45:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:22.412822+0000 mon.c (mon.2) 365 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:22.414159+0000 mon.a (mon.0) 2339 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:22.021964+0000 mon.a (mon.0) 2334 : audit [INF] from='client.? 172.21.15.162:0/3297476539' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:45:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:22.406421+0000 mon.a (mon.0) 2335 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:45:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:22.410787+0000 mon.a (mon.0) 2336 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-83985-62"}]': finished 2024-02-13T00:45:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:22.410853+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? 172.21.15.162:0/3297476539' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:45:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:22.410886+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e367: 8 total, 8 up, 5 in 2024-02-13T00:45:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:22.412822+0000 mon.c (mon.2) 365 : audit [INF] from='client.? 172.21.15.162:0/3815326616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:22.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:22.414159+0000 mon.a (mon.0) 2339 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]: dispatch 2024-02-13T00:45:23.694 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 6] out_osds: [3, 2, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:45:23.694 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:45:23.695 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-13T00:45:23.695 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-13T00:45:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:22.406417+0000 mgr.y (mgr.14544) 326 : cluster [DBG] pgmap v520: 238 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering, 4 active+recovery_wait+degraded, 230 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s; 17/1107 objects degraded (1.536%); 11/1107 objects misplaced (0.994%); 574 B/s, 92 objects/s recovering 2024-02-13T00:45:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:23.413663+0000 mon.a (mon.0) 2340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]': finished 2024-02-13T00:45:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:23.413767+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e368: 8 total, 8 up, 5 in 2024-02-13T00:45:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:23.434183+0000 mon.a (mon.0) 2342 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:23.435247+0000 mon.a (mon.0) 2343 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:23.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:23.435886+0000 mon.a (mon.0) 2344 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-83985-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:23.820 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:22.406417+0000 mgr.y (mgr.14544) 326 : cluster [DBG] pgmap v520: 238 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering, 4 active+recovery_wait+degraded, 230 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s; 17/1107 objects degraded (1.536%); 11/1107 objects misplaced (0.994%); 574 B/s, 92 objects/s recovering 2024-02-13T00:45:23.821 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:23.413663+0000 mon.a (mon.0) 2340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]': finished 2024-02-13T00:45:23.821 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:23.413767+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e368: 8 total, 8 up, 5 in 2024-02-13T00:45:23.821 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:23.434183+0000 mon.a (mon.0) 2342 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:23.821 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:23.435247+0000 mon.a (mon.0) 2343 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:23.821 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:23.435886+0000 mon.a (mon.0) 2344 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-83985-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:23.821 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:22.406417+0000 mgr.y (mgr.14544) 326 : cluster [DBG] pgmap v520: 238 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering, 4 active+recovery_wait+degraded, 230 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s; 17/1107 objects degraded (1.536%); 11/1107 objects misplaced (0.994%); 574 B/s, 92 objects/s recovering 2024-02-13T00:45:23.821 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:23.413663+0000 mon.a (mon.0) 2340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-83985-62"}]': finished 2024-02-13T00:45:23.821 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:23.413767+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e368: 8 total, 8 up, 5 in 2024-02-13T00:45:23.822 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:23.434183+0000 mon.a (mon.0) 2342 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:23.822 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:23.435247+0000 mon.a (mon.0) 2343 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:23.822 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:23.435886+0000 mon.a (mon.0) 2344 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-83985-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.509+0000 7f789a219700 1 -- 172.21.15.162:0/254906273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f789406ba80 msgr2=0x7f7894104090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.509+0000 7f789a219700 1 --2- 172.21.15.162:0/254906273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f789406ba80 0x7f7894104090 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 rx=0x7f7888004870 tx=0x7f7888009cb0).stop 2024-02-13T00:45:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.509+0000 7f789a219700 1 -- 172.21.15.162:0/254906273 shutdown_connections 2024-02-13T00:45:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.509+0000 7f789a219700 1 --2- 172.21.15.162:0/254906273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7894104580 0x7f7894106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.509+0000 7f789a219700 1 --2- 172.21.15.162:0/254906273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f789406b1b0 0x7f789406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.509+0000 7f789a219700 1 --2- 172.21.15.162:0/254906273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f789406ba80 0x7f7894104090 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.509+0000 7f789a219700 1 -- 172.21.15.162:0/254906273 >> 172.21.15.162:0/254906273 conn(0x7f78940f38c0 msgr2=0x7f78940f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.509+0000 7f789a219700 1 -- 172.21.15.162:0/254906273 shutdown_connections 2024-02-13T00:45:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.509+0000 7f789a219700 1 -- 172.21.15.162:0/254906273 wait complete. 2024-02-13T00:45:25.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.509+0000 7f789a219700 1 librados: starting msgr at 2024-02-13T00:45:25.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.509+0000 7f789a219700 1 librados: starting objecter 2024-02-13T00:45:25.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.510+0000 7f789a219700 1 Processor -- start 2024-02-13T00:45:25.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.510+0000 7f789a219700 1 -- start start 2024-02-13T00:45:25.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.510+0000 7f789a219700 1 librados: setting wanted keys 2024-02-13T00:45:25.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.510+0000 7f789a219700 1 librados: calling monclient init 2024-02-13T00:45:25.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f789a219700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f789406b1b0 0x7f78941104f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:25.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f789a219700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f789406ba80 0x7f78941109e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:25.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f789a219700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7894104580 0x7f789410a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f789a219700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789410ab80 con 0x7f789406b1b0 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f789a219700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789410acc0 con 0x7f789406ba80 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f789a219700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789410ae80 con 0x7f7894104580 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f78937fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f789406b1b0 0x7f78941104f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f78937fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f789406b1b0 0x7f78941104f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:52492/0 (socket says 172.21.15.162:52492) 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f78937fe700 1 -- 172.21.15.162:0/13031301 learned_addr learned my addr 172.21.15.162:0/13031301 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f7892ffd700 1 --2- 172.21.15.162:0/13031301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f789406ba80 0x7f78941109e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f7893fff700 1 --2- 172.21.15.162:0/13031301 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7894104580 0x7f789410a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f7893fff700 1 -- 172.21.15.162:0/13031301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f789406ba80 msgr2=0x7f78941109e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f7893fff700 1 --2- 172.21.15.162:0/13031301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f789406ba80 0x7f78941109e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f7893fff700 1 -- 172.21.15.162:0/13031301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f789406b1b0 msgr2=0x7f78941104f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f7893fff700 1 --2- 172.21.15.162:0/13031301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f789406b1b0 0x7f78941104f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f7893fff700 1 -- 172.21.15.162:0/13031301 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888005040 con 0x7f7894104580 2024-02-13T00:45:25.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f78937fe700 1 --2- 172.21.15.162:0/13031301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f789406b1b0 0x7f78941104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:45:25.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f7893fff700 1 --2- 172.21.15.162:0/13031301 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7894104580 0x7f789410a690 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7f788c00eb40 tx=0x7f788c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:25.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f7890ff9700 10 client.?.objecter ms_handle_connect 0x7f7894104580 2024-02-13T00:45:25.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f7890ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:25.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.511+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c009870 con 0x7f7894104580 2024-02-13T00:45:25.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.512+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f788c00f040 con 0x7f7894104580 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.512+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789410b100 con 0x7f7894104580 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.512+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c007d60 con 0x7f7894104580 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.512+0000 7f789a219700 10 client.37325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.512+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78941172f0 con 0x7f7894104580 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.512+0000 7f789a219700 1 librados: init done 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.512+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f78940475e0 con 0x7f7894104580 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.513+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f788c0040b0 con 0x7f7894104580 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.513+0000 7f7890ff9700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6800/2866534053 conn(0x7f787c062c90 0x7f787c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(370..370 src has 52..370) v4 ==== 8445+0+0 (secure 0 0 0) 0x7f788c07d6f0 con 0x7f7894104580 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 osd_map(370..370 src has 52..370) v4 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f7890ff9700 3 client.37325.objecter handle_osd_map got epochs [370,370] > 0 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f7890ff9700 3 client.37325.objecter handle_osd_map decoding full epoch 370 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f78937fe700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6800/2866534053 conn(0x7f787c062c90 0x7f787c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:25.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f7890ff9700 20 client.37325.objecter dump_active .. 0 homeless 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v370) v1 ==== 54+0+16 (secure 0 0 0) 0x7f788c07db90 con 0x7f7894104580 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f7890ff9700 10 client.37325.objecter ms_handle_connect 0x7f787c062c90 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f78937fe700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6800/2866534053 conn(0x7f787c062c90 0x7f787c065150 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f7884004fd0 tx=0x7f788400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f789a219700 10 client.37325.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 1 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6801/2103743338 conn(0x7f7894028590 0x7f78940fe040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f78941101c0 osd=0 3 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f78941101c0 osd=0 3 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f78941101c0 osd=0 3 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 1 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f78941101c0 osd=0 3 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 0 1 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f78941101c0 osd=0 4 2024-02-13T00:45:25.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f789a219700 10 client.37325.objecter _send_command 1 2024-02-13T00:45:25.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78941094b0 con 0x7f7894028590 2024-02-13T00:45:25.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f7892ffd700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6801/2103743338 conn(0x7f7894028590 0x7f78940fe040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:25.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f7890ff9700 10 client.37325.objecter ms_handle_connect 0x7f7894028590 2024-02-13T00:45:25.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.514+0000 7f7892ffd700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6801/2103743338 conn(0x7f7894028590 0x7f78940fe040 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:25.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.515+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f78941094b0 con 0x7f7894028590 2024-02-13T00:45:25.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.515+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:45:25.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.515+0000 7f7890ff9700 10 client.37325.objecter _finish_command 1 = system:0 2024-02-13T00:45:25.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.515+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f78941101c0 osd=0 4 2024-02-13T00:45:25.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.515+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 0 1 2024-02-13T00:45:25.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.531+0000 7f789a219700 10 client.37325.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:25.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.531+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 2 2024-02-13T00:45:25.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.531+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f78941101c0 osd=0 3 2024-02-13T00:45:25.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.531+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f78941101c0 osd=0 3 2024-02-13T00:45:25.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.531+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f78941101c0 osd=0 3 2024-02-13T00:45:25.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.531+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 2 2024-02-13T00:45:25.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.531+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f78941101c0 osd=0 3 2024-02-13T00:45:25.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.531+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 0 2 2024-02-13T00:45:25.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.531+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f78941101c0 osd=0 4 2024-02-13T00:45:25.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.531+0000 7f789a219700 10 client.37325.objecter _send_command 2 2024-02-13T00:45:25.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.531+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f789405a0e0 con 0x7f7894028590 2024-02-13T00:45:25.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.533+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f789405a0e0 con 0x7f7894028590 2024-02-13T00:45:25.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.533+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:25.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.533+0000 7f7890ff9700 10 client.37325.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.533+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f78941101c0 osd=0 4 2024-02-13T00:45:25.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.533+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 0 2 2024-02-13T00:45:25.537 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:45:25.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f789a219700 10 client.37325.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:25.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 3 2024-02-13T00:45:25.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6805/150599780 conn(0x7f7894027be0 0x7f7894027fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:25.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f78940ff6b0 osd=1 3 2024-02-13T00:45:25.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f78940ff6b0 osd=1 3 2024-02-13T00:45:25.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f78940ff6b0 osd=1 3 2024-02-13T00:45:25.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 3 2024-02-13T00:45:25.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f78940ff6b0 osd=1 3 2024-02-13T00:45:25.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 1 3 2024-02-13T00:45:25.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f78940ff6b0 osd=1 4 2024-02-13T00:45:25.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f789a219700 10 client.37325.objecter _send_command 3 2024-02-13T00:45:25.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78940ff820 con 0x7f7894027be0 2024-02-13T00:45:25.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f78937fe700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6805/150599780 conn(0x7f7894027be0 0x7f7894027fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:25.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f7890ff9700 10 client.37325.objecter ms_handle_connect 0x7f7894027be0 2024-02-13T00:45:25.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.534+0000 7f78937fe700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6805/150599780 conn(0x7f7894027be0 0x7f7894027fc0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:25.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.535+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f78940ff820 con 0x7f7894027be0 2024-02-13T00:45:25.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.535+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:45:25.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.535+0000 7f7890ff9700 10 client.37325.objecter _finish_command 3 = system:0 2024-02-13T00:45:25.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.535+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f78940ff6b0 osd=1 4 2024-02-13T00:45:25.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.535+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 1 3 2024-02-13T00:45:25.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.549+0000 7f789a219700 10 client.37325.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:25.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.549+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 4 2024-02-13T00:45:25.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.549+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f78940ff6b0 osd=1 3 2024-02-13T00:45:25.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.549+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f78940ff6b0 osd=1 3 2024-02-13T00:45:25.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.549+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f78940ff6b0 osd=1 3 2024-02-13T00:45:25.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.549+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 4 2024-02-13T00:45:25.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.549+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f78940ff6b0 osd=1 3 2024-02-13T00:45:25.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.549+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 1 4 2024-02-13T00:45:25.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.549+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f78940ff6b0 osd=1 4 2024-02-13T00:45:25.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.549+0000 7f789a219700 10 client.37325.objecter _send_command 4 2024-02-13T00:45:25.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.549+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7894027650 con 0x7f7894027be0 2024-02-13T00:45:25.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7894027650 con 0x7f7894027be0 2024-02-13T00:45:25.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:25.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f7890ff9700 10 client.37325.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f78940ff6b0 osd=1 4 2024-02-13T00:45:25.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 1 4 2024-02-13T00:45:25.555 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:45:25.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f789a219700 10 client.37325.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:25.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 5 2024-02-13T00:45:25.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6809/3689719376 conn(0x7f78940ff210 0x7f789411e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:25.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f7894027a60 osd=2 3 2024-02-13T00:45:25.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894027a60 osd=2 3 2024-02-13T00:45:25.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f7894027a60 osd=2 3 2024-02-13T00:45:25.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 5 2024-02-13T00:45:25.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f7894027a60 osd=2 3 2024-02-13T00:45:25.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 2 5 2024-02-13T00:45:25.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894027a60 osd=2 4 2024-02-13T00:45:25.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f789a219700 10 client.37325.objecter _send_command 5 2024-02-13T00:45:25.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7894121280 con 0x7f78940ff210 2024-02-13T00:45:25.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f7892ffd700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6809/3689719376 conn(0x7f78940ff210 0x7f789411e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:25.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f7890ff9700 10 client.37325.objecter ms_handle_connect 0x7f78940ff210 2024-02-13T00:45:25.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.551+0000 7f7892ffd700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6809/3689719376 conn(0x7f78940ff210 0x7f789411e630 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:25.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.552+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7894121280 con 0x7f78940ff210 2024-02-13T00:45:25.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.552+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:45:25.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.552+0000 7f7890ff9700 10 client.37325.objecter _finish_command 5 = system:0 2024-02-13T00:45:25.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.552+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f7894027a60 osd=2 4 2024-02-13T00:45:25.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.552+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 2 5 2024-02-13T00:45:25.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.566+0000 7f789a219700 10 client.37325.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:25.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.566+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 6 2024-02-13T00:45:25.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.566+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f7894027a60 osd=2 3 2024-02-13T00:45:25.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.566+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894027a60 osd=2 3 2024-02-13T00:45:25.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.566+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f7894027a60 osd=2 3 2024-02-13T00:45:25.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.566+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 6 2024-02-13T00:45:25.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.566+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f7894027a60 osd=2 3 2024-02-13T00:45:25.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.566+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 2 6 2024-02-13T00:45:25.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.566+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894027a60 osd=2 4 2024-02-13T00:45:25.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.566+0000 7f789a219700 10 client.37325.objecter _send_command 6 2024-02-13T00:45:25.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.566+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f78940fe770 con 0x7f78940ff210 2024-02-13T00:45:25.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.568+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f78940fe770 con 0x7f78940ff210 2024-02-13T00:45:25.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.568+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:25.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.568+0000 7f7890ff9700 10 client.37325.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.568+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f7894027a60 osd=2 4 2024-02-13T00:45:25.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.568+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 2 6 2024-02-13T00:45:25.573 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:45:25.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f789a219700 10 client.37325.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:25.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 7 2024-02-13T00:45:25.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6813/604971011 conn(0x7f789411fca0 0x7f7894120080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:25.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f789411bc70 osd=3 3 2024-02-13T00:45:25.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789411bc70 osd=3 3 2024-02-13T00:45:25.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f789411bc70 osd=3 3 2024-02-13T00:45:25.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 7 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f789411bc70 osd=3 3 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 3 7 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789411bc70 osd=3 4 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f789a219700 10 client.37325.objecter _send_command 7 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f789411c2f0 con 0x7f789411fca0 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f7893fff700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6813/604971011 conn(0x7f789411fca0 0x7f7894120080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f7890ff9700 10 client.37325.objecter ms_handle_connect 0x7f789411fca0 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.569+0000 7f7893fff700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6813/604971011 conn(0x7f789411fca0 0x7f7894120080 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.570+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f789411c2f0 con 0x7f789411fca0 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.570+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.570+0000 7f7890ff9700 10 client.37325.objecter _finish_command 7 = system:0 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.570+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f789411bc70 osd=3 4 2024-02-13T00:45:25.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.570+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 3 7 2024-02-13T00:45:25.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:24.266122+0000 osd.5 (osd.5) 154 : cluster [DBG] 2.b scrub starts 2024-02-13T00:45:25.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:24.269591+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.b scrub ok 2024-02-13T00:45:25.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:24.406825+0000 mgr.y (mgr.14544) 327 : cluster [DBG] pgmap v523: 238 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering, 4 active+recovery_wait+degraded, 230 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 17/1107 objects degraded (1.536%); 11/1107 objects misplaced (0.994%) 2024-02-13T00:45:25.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:24.414814+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-83985-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:25.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:24.414875+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e369: 8 total, 8 up, 5 in 2024-02-13T00:45:25.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:24.415514+0000 mon.a (mon.0) 2347 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-83985-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:25.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:24.266122+0000 osd.5 (osd.5) 154 : cluster [DBG] 2.b scrub starts 2024-02-13T00:45:25.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:24.269591+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.b scrub ok 2024-02-13T00:45:25.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:24.406825+0000 mgr.y (mgr.14544) 327 : cluster [DBG] pgmap v523: 238 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering, 4 active+recovery_wait+degraded, 230 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 17/1107 objects degraded (1.536%); 11/1107 objects misplaced (0.994%) 2024-02-13T00:45:25.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:24.414814+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-83985-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:25.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:24.414875+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e369: 8 total, 8 up, 5 in 2024-02-13T00:45:25.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:24.415514+0000 mon.a (mon.0) 2347 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-83985-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:25.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.584+0000 7f789a219700 10 client.37325.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:25.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.585+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 8 2024-02-13T00:45:25.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.585+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f789411bc70 osd=3 3 2024-02-13T00:45:25.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.585+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789411bc70 osd=3 3 2024-02-13T00:45:25.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.585+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f789411bc70 osd=3 3 2024-02-13T00:45:25.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.585+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 8 2024-02-13T00:45:25.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.585+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f789411bc70 osd=3 3 2024-02-13T00:45:25.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.585+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 3 8 2024-02-13T00:45:25.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.585+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789411bc70 osd=3 4 2024-02-13T00:45:25.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.585+0000 7f789a219700 10 client.37325.objecter _send_command 8 2024-02-13T00:45:25.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.585+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7894120d40 con 0x7f789411fca0 2024-02-13T00:45:25.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7894120d40 con 0x7f789411fca0 2024-02-13T00:45:25.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:25.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f7890ff9700 10 client.37325.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f789411bc70 osd=3 4 2024-02-13T00:45:25.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 3 8 2024-02-13T00:45:25.590 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:45:25.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f789a219700 10 client.37325.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:25.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 9 2024-02-13T00:45:25.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6800/3344263388 conn(0x7f7894123400 0x7f78941257f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:25.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f7894120d40 osd=4 3 2024-02-13T00:45:25.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894120d40 osd=4 3 2024-02-13T00:45:25.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f7894120d40 osd=4 3 2024-02-13T00:45:25.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 9 2024-02-13T00:45:25.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f7894120d40 osd=4 3 2024-02-13T00:45:25.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 4 9 2024-02-13T00:45:25.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894120d40 osd=4 4 2024-02-13T00:45:25.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f789a219700 10 client.37325.objecter _send_command 9 2024-02-13T00:45:25.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7894120f40 con 0x7f7894123400 2024-02-13T00:45:25.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.586+0000 7f78937fe700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6800/3344263388 conn(0x7f7894123400 0x7f78941257f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:25.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.587+0000 7f7890ff9700 10 client.37325.objecter ms_handle_connect 0x7f7894123400 2024-02-13T00:45:25.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.587+0000 7f78937fe700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6800/3344263388 conn(0x7f7894123400 0x7f78941257f0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:25.592 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.588+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7894120f40 con 0x7f7894123400 2024-02-13T00:45:25.592 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.588+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:45:25.592 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.588+0000 7f7890ff9700 10 client.37325.objecter _finish_command 9 = system:0 2024-02-13T00:45:25.592 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.588+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f7894120d40 osd=4 4 2024-02-13T00:45:25.592 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.588+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 4 9 2024-02-13T00:45:25.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.602+0000 7f789a219700 10 client.37325.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:25.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.602+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 10 2024-02-13T00:45:25.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.602+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f7894120d40 osd=4 3 2024-02-13T00:45:25.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.602+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894120d40 osd=4 3 2024-02-13T00:45:25.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.602+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f7894120d40 osd=4 3 2024-02-13T00:45:25.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.602+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 10 2024-02-13T00:45:25.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.602+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f7894120d40 osd=4 3 2024-02-13T00:45:25.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.602+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 4 10 2024-02-13T00:45:25.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.602+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894120d40 osd=4 4 2024-02-13T00:45:25.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.602+0000 7f789a219700 10 client.37325.objecter _send_command 10 2024-02-13T00:45:25.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.602+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7894120f40 con 0x7f7894123400 2024-02-13T00:45:25.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7894120f40 con 0x7f7894123400 2024-02-13T00:45:25.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:25.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f7890ff9700 10 client.37325.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f7894120d40 osd=4 4 2024-02-13T00:45:25.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 4 10 2024-02-13T00:45:25.608 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:45:25.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f789a219700 10 client.37325.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:25.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 11 2024-02-13T00:45:25.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6804/2212296748 conn(0x7f78941261f0 0x7f78941285e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:25.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f7894120f40 osd=5 3 2024-02-13T00:45:25.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894120f40 osd=5 3 2024-02-13T00:45:25.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f7894120f40 osd=5 3 2024-02-13T00:45:25.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 11 2024-02-13T00:45:25.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f7894120f40 osd=5 3 2024-02-13T00:45:25.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 5 11 2024-02-13T00:45:25.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894120f40 osd=5 4 2024-02-13T00:45:25.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f789a219700 10 client.37325.objecter _send_command 11 2024-02-13T00:45:25.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.604+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f789412b410 con 0x7f78941261f0 2024-02-13T00:45:25.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.605+0000 7f7892ffd700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6804/2212296748 conn(0x7f78941261f0 0x7f78941285e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:25.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.605+0000 7f7890ff9700 10 client.37325.objecter ms_handle_connect 0x7f78941261f0 2024-02-13T00:45:25.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.605+0000 7f7892ffd700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6804/2212296748 conn(0x7f78941261f0 0x7f78941285e0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:25.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.606+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f789412b410 con 0x7f78941261f0 2024-02-13T00:45:25.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.606+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:45:25.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.606+0000 7f7890ff9700 10 client.37325.objecter _finish_command 11 = system:0 2024-02-13T00:45:25.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.606+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f7894120f40 osd=5 4 2024-02-13T00:45:25.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.606+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 5 11 2024-02-13T00:45:25.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.620+0000 7f789a219700 10 client.37325.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:25.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.620+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 12 2024-02-13T00:45:25.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.620+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f7894120f40 osd=5 3 2024-02-13T00:45:25.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.620+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894120f40 osd=5 3 2024-02-13T00:45:25.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.620+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f7894120f40 osd=5 3 2024-02-13T00:45:25.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.620+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 12 2024-02-13T00:45:25.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.620+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f7894120f40 osd=5 3 2024-02-13T00:45:25.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.620+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 5 12 2024-02-13T00:45:25.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.620+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894120f40 osd=5 4 2024-02-13T00:45:25.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.620+0000 7f789a219700 10 client.37325.objecter _send_command 12 2024-02-13T00:45:25.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.620+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f789412b410 con 0x7f78941261f0 2024-02-13T00:45:25.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f789412b410 con 0x7f78941261f0 2024-02-13T00:45:25.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:25.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f7890ff9700 10 client.37325.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f7894120f40 osd=5 4 2024-02-13T00:45:25.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 5 12 2024-02-13T00:45:25.626 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f789a219700 10 client.37325.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 13 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6808/1506640341 conn(0x7f789412a710 0x7f789412ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f789412b410 osd=6 3 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789412b410 osd=6 3 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f789412b410 osd=6 3 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 13 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f789412b410 osd=6 3 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 6 13 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789412b410 osd=6 4 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f789a219700 10 client.37325.objecter _send_command 13 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f789412b1b0 con 0x7f789412a710 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.622+0000 7f7893fff700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6808/1506640341 conn(0x7f789412a710 0x7f789412ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.623+0000 7f7893fff700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6808/1506640341 conn(0x7f789412a710 0x7f789412ab90 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:25.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.623+0000 7f7890ff9700 10 client.37325.objecter ms_handle_connect 0x7f789412a710 2024-02-13T00:45:25.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.623+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f789412b1b0 con 0x7f789412a710 2024-02-13T00:45:25.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.623+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:45:25.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.623+0000 7f7890ff9700 10 client.37325.objecter _finish_command 13 = system:0 2024-02-13T00:45:25.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.623+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f789412b410 osd=6 4 2024-02-13T00:45:25.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.623+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 6 13 2024-02-13T00:45:25.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.638+0000 7f789a219700 10 client.37325.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:25.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.638+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 14 2024-02-13T00:45:25.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.638+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f789412b410 osd=6 3 2024-02-13T00:45:25.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.638+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789412b410 osd=6 3 2024-02-13T00:45:25.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.638+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f789412b410 osd=6 3 2024-02-13T00:45:25.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.638+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 14 2024-02-13T00:45:25.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.638+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f789412b410 osd=6 3 2024-02-13T00:45:25.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.638+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 6 14 2024-02-13T00:45:25.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.638+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789412b410 osd=6 4 2024-02-13T00:45:25.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.638+0000 7f789a219700 10 client.37325.objecter _send_command 14 2024-02-13T00:45:25.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.638+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f789412b1b0 con 0x7f789412a710 2024-02-13T00:45:25.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f789412b1b0 con 0x7f789412a710 2024-02-13T00:45:25.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:25.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f7890ff9700 10 client.37325.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f789412b410 osd=6 4 2024-02-13T00:45:25.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 6 14 2024-02-13T00:45:25.644 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:45:25.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f789a219700 10 client.37325.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:25.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 15 2024-02-13T00:45:25.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6812/1964192772 conn(0x7f789412dc10 0x7f789412e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:25.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f789412b1b0 osd=7 3 2024-02-13T00:45:25.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789412b1b0 osd=7 3 2024-02-13T00:45:25.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f789412b1b0 osd=7 3 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 15 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f789412b1b0 osd=7 3 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 7 15 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789412b1b0 osd=7 4 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f789a219700 10 client.37325.objecter _send_command 15 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.640+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78941319b0 con 0x7f789412dc10 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.641+0000 7f78937fe700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6812/1964192772 conn(0x7f789412dc10 0x7f789412e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.641+0000 7f7890ff9700 10 client.37325.objecter ms_handle_connect 0x7f789412dc10 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.641+0000 7f78937fe700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6812/1964192772 conn(0x7f789412dc10 0x7f789412e070 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.642+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f78941319b0 con 0x7f789412dc10 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.642+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.642+0000 7f7890ff9700 10 client.37325.objecter _finish_command 15 = system:0 2024-02-13T00:45:25.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.642+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f789412b1b0 osd=7 4 2024-02-13T00:45:25.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.642+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 7 15 2024-02-13T00:45:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.657+0000 7f789a219700 10 client.37325.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.657+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign -1 16 2024-02-13T00:45:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.657+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f789412b1b0 osd=7 3 2024-02-13T00:45:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.657+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789412b1b0 osd=7 3 2024-02-13T00:45:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.657+0000 7f789a219700 20 client.37325.objecter _get_session s=0x7f789412b1b0 osd=7 3 2024-02-13T00:45:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.657+0000 7f789a219700 15 client.37325.objecter _session_command_op_remove -1 16 2024-02-13T00:45:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.657+0000 7f789a219700 20 client.37325.objecter get_session s=0x7f789412b1b0 osd=7 3 2024-02-13T00:45:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.657+0000 7f789a219700 15 client.37325.objecter _session_command_op_assign 7 16 2024-02-13T00:45:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.657+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789412b1b0 osd=7 4 2024-02-13T00:45:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.657+0000 7f789a219700 10 client.37325.objecter _send_command 16 2024-02-13T00:45:25.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.657+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7894128bf0 con 0x7f789412dc10 2024-02-13T00:45:25.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.658+0000 7f7890ff9700 1 -- 172.21.15.162:0/13031301 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7894128bf0 con 0x7f789412dc10 2024-02-13T00:45:25.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.658+0000 7f7890ff9700 10 client.37325.objecter ms_dispatch 0x7f7894001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:25.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.658+0000 7f7890ff9700 10 client.37325.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.658+0000 7f7890ff9700 20 client.37325.objecter put_session s=0x7f789412b1b0 osd=7 4 2024-02-13T00:45:25.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.658+0000 7f7890ff9700 15 client.37325.objecter _session_command_op_remove 7 16 2024-02-13T00:45:25.662 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:45:25.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 10 librados: watch_flush enter 2024-02-13T00:45:25.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 10 librados: watch_flush exit 2024-02-13T00:45:25.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 10 client.37325.objecter close_session for osd.0 2024-02-13T00:45:25.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6801/2103743338 conn(0x7f7894028590 msgr2=0x7f78940fe040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6801/2103743338 conn(0x7f7894028590 0x7f78940fe040 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f78941101c0 osd=0 1 2024-02-13T00:45:25.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 10 client.37325.objecter close_session for osd.1 2024-02-13T00:45:25.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6805/150599780 conn(0x7f7894027be0 msgr2=0x7f7894027fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6805/150599780 conn(0x7f7894027be0 0x7f7894027fc0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f78940ff6b0 osd=1 1 2024-02-13T00:45:25.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 10 client.37325.objecter close_session for osd.2 2024-02-13T00:45:25.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6809/3689719376 conn(0x7f78940ff210 msgr2=0x7f789411e630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6809/3689719376 conn(0x7f78940ff210 0x7f789411e630 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894027a60 osd=2 1 2024-02-13T00:45:25.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 10 client.37325.objecter close_session for osd.3 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6813/604971011 conn(0x7f789411fca0 msgr2=0x7f7894120080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6813/604971011 conn(0x7f789411fca0 0x7f7894120080 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789411bc70 osd=3 1 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 10 client.37325.objecter close_session for osd.4 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6800/3344263388 conn(0x7f7894123400 msgr2=0x7f78941257f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6800/3344263388 conn(0x7f7894123400 0x7f78941257f0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f7893fff700 1 -- 172.21.15.162:0/13031301 reap_dead start 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f7893fff700 1 -- 172.21.15.162:0/13031301 reap_dead start 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894120d40 osd=4 1 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 10 client.37325.objecter close_session for osd.5 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6804/2212296748 conn(0x7f78941261f0 msgr2=0x7f78941285e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6804/2212296748 conn(0x7f78941261f0 0x7f78941285e0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f7894120f40 osd=5 1 2024-02-13T00:45:25.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 10 client.37325.objecter close_session for osd.6 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6808/1506640341 conn(0x7f789412a710 msgr2=0x7f789412ab90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6808/1506640341 conn(0x7f789412a710 0x7f789412ab90 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789412b410 osd=6 1 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 10 client.37325.objecter close_session for osd.7 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6812/1964192772 conn(0x7f789412dc10 msgr2=0x7f789412e070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.181:6812/1964192772 conn(0x7f789412dc10 0x7f789412e070 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 20 client.37325.objecter put_session s=0x7f789412b1b0 osd=7 1 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 20 client.37325.objecter shutdown clearing up homeless session... 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 10 client.37325.objecter successfully canceled tick 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6800/2866534053 conn(0x7f787c062c90 msgr2=0x7f787c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> v2:172.21.15.162:6800/2866534053 conn(0x7f787c062c90 0x7f787c065150 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f7884004fd0 tx=0x7f788400b040).stop 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7894104580 msgr2=0x7f789410a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:25.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 --2- 172.21.15.162:0/13031301 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7894104580 0x7f789410a690 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7f788c00eb40 tx=0x7f788c00b040).stop 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f7893fff700 1 -- 172.21.15.162:0/13031301 reap_dead start 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 shutdown_connections 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 >> 172.21.15.162:0/13031301 conn(0x7f78940f38c0 msgr2=0x7f78940f4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 shutdown_connections 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 -- 172.21.15.162:0/13031301 wait complete. 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:25.659+0000 7f789a219700 1 librados: shutdown 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.668 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.669 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:45:25.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:45:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:45:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:24.266122+0000 osd.5 (osd.5) 154 : cluster [DBG] 2.b scrub starts 2024-02-13T00:45:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:24.269591+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.b scrub ok 2024-02-13T00:45:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:24.406825+0000 mgr.y (mgr.14544) 327 : cluster [DBG] pgmap v523: 238 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering, 4 active+recovery_wait+degraded, 230 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 17/1107 objects degraded (1.536%); 11/1107 objects misplaced (0.994%) 2024-02-13T00:45:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:24.414814+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-83985-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:24.414875+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e369: 8 total, 8 up, 5 in 2024-02-13T00:45:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:24.415514+0000 mon.a (mon.0) 2347 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-83985-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:26.169 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:26.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:26.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:26.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:26.305 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:26.378 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:26.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:26.426 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:45:26] "GET /metrics HTTP/1.1" 200 187237 "" "Prometheus/2.33.4" 2024-02-13T00:45:26.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:26.535 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:26.556 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:26.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:26.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:26.688 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:24.985871+0000 osd.2 (osd.2) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:26.689 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:24.986044+0000 osd.2 (osd.2) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:26.689 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:25.513893+0000 mon.c (mon.2) 366 : audit [DBG] from='client.? 172.21.15.162:0/13031301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:26.689 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:25.566525+0000 mgr.y (mgr.14544) 328 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:26.689 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:25.605174+0000 mon.b (mon.1) 537 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:26.689 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:25.666008+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e370: 8 total, 8 up, 5 in 2024-02-13T00:45:26.690 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:25.666583+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:26.690 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:26.712 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:26.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:26.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:24.985871+0000 osd.2 (osd.2) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:26.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:24.986044+0000 osd.2 (osd.2) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:26.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:25.513893+0000 mon.c (mon.2) 366 : audit [DBG] from='client.? 172.21.15.162:0/13031301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:26.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:25.566525+0000 mgr.y (mgr.14544) 328 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:26.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:25.605174+0000 mon.b (mon.1) 537 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:26.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:25.666008+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e370: 8 total, 8 up, 5 in 2024-02-13T00:45:26.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:25.666583+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:26.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:24.985871+0000 osd.2 (osd.2) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:26.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:24.986044+0000 osd.2 (osd.2) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:26.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:25.513893+0000 mon.c (mon.2) 366 : audit [DBG] from='client.? 172.21.15.162:0/13031301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:26.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:25.566525+0000 mgr.y (mgr.14544) 328 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:26.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:25.605174+0000 mon.b (mon.1) 537 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:26.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:25.666008+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e370: 8 total, 8 up, 5 in 2024-02-13T00:45:26.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:25.666583+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:26.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:26.846 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:26.866 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:26.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:26.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:26.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:26.998 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:27.018 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:27.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:27.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:27.150 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:27.169 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:27.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:27.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:27.300 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:27.320 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:27.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:27.351 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:45:27.351 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:45:27.356 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:27.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:27.461 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:27.627 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:26.214035+0000 osd.5 (osd.5) 156 : cluster [DBG] 61.f scrub starts 2024-02-13T00:45:27.628 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:26.215787+0000 osd.5 (osd.5) 157 : cluster [DBG] 61.f scrub ok 2024-02-13T00:45:27.628 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:26.252872+0000 osd.1 (osd.1) 145 : cluster [DBG] 28.15 scrub starts 2024-02-13T00:45:27.628 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:26.255837+0000 osd.1 (osd.1) 146 : cluster [DBG] 28.15 scrub ok 2024-02-13T00:45:27.628 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:26.407205+0000 mgr.y (mgr.14544) 329 : cluster [DBG] pgmap v526: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 394 B/s, 8 objects/s recovering 2024-02-13T00:45:27.628 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:26.417602+0000 mon.a (mon.0) 2350 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/1107 objects degraded (1.536%), 4 pgs degraded) 2024-02-13T00:45:27.629 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:26.427359+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-83985-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-83985-63"}]': finished 2024-02-13T00:45:27.629 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:26.427440+0000 mon.a (mon.0) 2352 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:45:27.629 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:26.427497+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e371: 8 total, 8 up, 5 in 2024-02-13T00:45:27.629 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:26.430160+0000 mon.b (mon.1) 538 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:27.629 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:26.434030+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:27.629 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:26.214035+0000 osd.5 (osd.5) 156 : cluster [DBG] 61.f scrub starts 2024-02-13T00:45:27.630 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:26.215787+0000 osd.5 (osd.5) 157 : cluster [DBG] 61.f scrub ok 2024-02-13T00:45:27.630 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:26.252872+0000 osd.1 (osd.1) 145 : cluster [DBG] 28.15 scrub starts 2024-02-13T00:45:27.630 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:26.255837+0000 osd.1 (osd.1) 146 : cluster [DBG] 28.15 scrub ok 2024-02-13T00:45:27.630 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:26.407205+0000 mgr.y (mgr.14544) 329 : cluster [DBG] pgmap v526: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 394 B/s, 8 objects/s recovering 2024-02-13T00:45:27.630 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:26.417602+0000 mon.a (mon.0) 2350 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/1107 objects degraded (1.536%), 4 pgs degraded) 2024-02-13T00:45:27.630 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:26.427359+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-83985-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-83985-63"}]': finished 2024-02-13T00:45:27.631 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:26.427440+0000 mon.a (mon.0) 2352 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:45:27.631 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:26.427497+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e371: 8 total, 8 up, 5 in 2024-02-13T00:45:27.631 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:26.430160+0000 mon.b (mon.1) 538 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:27.631 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:26.434030+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:26.214035+0000 osd.5 (osd.5) 156 : cluster [DBG] 61.f scrub starts 2024-02-13T00:45:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:26.215787+0000 osd.5 (osd.5) 157 : cluster [DBG] 61.f scrub ok 2024-02-13T00:45:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:26.252872+0000 osd.1 (osd.1) 145 : cluster [DBG] 28.15 scrub starts 2024-02-13T00:45:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:26.255837+0000 osd.1 (osd.1) 146 : cluster [DBG] 28.15 scrub ok 2024-02-13T00:45:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:26.407205+0000 mgr.y (mgr.14544) 329 : cluster [DBG] pgmap v526: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 394 B/s, 8 objects/s recovering 2024-02-13T00:45:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:26.417602+0000 mon.a (mon.0) 2350 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/1107 objects degraded (1.536%), 4 pgs degraded) 2024-02-13T00:45:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:26.427359+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-83985-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-83985-63"}]': finished 2024-02-13T00:45:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:26.427440+0000 mon.a (mon.0) 2352 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:45:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:26.427497+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e371: 8 total, 8 up, 5 in 2024-02-13T00:45:27.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:26.430160+0000 mon.b (mon.1) 538 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:27.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:26.434030+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:28.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:26.648196+0000 osd.0 (osd.0) 122 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:45:28.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:26.651086+0000 osd.0 (osd.0) 123 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:45:28.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:27.171525+0000 osd.5 (osd.5) 158 : cluster [DBG] 61.10 scrub starts 2024-02-13T00:45:28.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:27.173318+0000 osd.5 (osd.5) 159 : cluster [DBG] 61.10 scrub ok 2024-02-13T00:45:28.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:27.423057+0000 mon.a (mon.0) 2355 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:45:28.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:27.430199+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]': finished 2024-02-13T00:45:28.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:27.430284+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e372: 8 total, 8 up, 5 in 2024-02-13T00:45:28.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:27.453777+0000 mon.b (mon.1) 539 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:28.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:27.454290+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:28.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:27.454723+0000 mon.b (mon.1) 540 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:28.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:27.455059+0000 mon.a (mon.0) 2359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:28.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:26.648196+0000 osd.0 (osd.0) 122 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:45:28.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:26.651086+0000 osd.0 (osd.0) 123 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:45:28.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:27.171525+0000 osd.5 (osd.5) 158 : cluster [DBG] 61.10 scrub starts 2024-02-13T00:45:28.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:27.173318+0000 osd.5 (osd.5) 159 : cluster [DBG] 61.10 scrub ok 2024-02-13T00:45:28.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:27.423057+0000 mon.a (mon.0) 2355 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:45:28.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:27.430199+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]': finished 2024-02-13T00:45:28.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:27.430284+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e372: 8 total, 8 up, 5 in 2024-02-13T00:45:28.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:27.453777+0000 mon.b (mon.1) 539 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:28.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:27.454290+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:28.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:27.454723+0000 mon.b (mon.1) 540 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:28.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:27.455059+0000 mon.a (mon.0) 2359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:28.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:45:28.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:45:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:45:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:45:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:26.648196+0000 osd.0 (osd.0) 122 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:45:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:26.651086+0000 osd.0 (osd.0) 123 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:45:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:27.171525+0000 osd.5 (osd.5) 158 : cluster [DBG] 61.10 scrub starts 2024-02-13T00:45:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:27.173318+0000 osd.5 (osd.5) 159 : cluster [DBG] 61.10 scrub ok 2024-02-13T00:45:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:27.423057+0000 mon.a (mon.0) 2355 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:45:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:27.430199+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]': finished 2024-02-13T00:45:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:27.430284+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e372: 8 total, 8 up, 5 in 2024-02-13T00:45:28.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:27.453777+0000 mon.b (mon.1) 539 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:28.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:27.454290+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:28.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:27.454723+0000 mon.b (mon.1) 540 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:28.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:27.455059+0000 mon.a (mon.0) 2359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-47"}]: dispatch 2024-02-13T00:45:29.011 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:29.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:29.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:45:29.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:29.161 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:29.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-13T00:45:29.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:28.407661+0000 mgr.y (mgr.14544) 330 : cluster [DBG] pgmap v529: 246 pgs: 8 unknown, 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 394 B/s, 8 objects/s recovering 2024-02-13T00:45:29.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:28.452558+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e373: 8 total, 8 up, 5 in 2024-02-13T00:45:29.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:28.453688+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:29.757 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:28.407661+0000 mgr.y (mgr.14544) 330 : cluster [DBG] pgmap v529: 246 pgs: 8 unknown, 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 394 B/s, 8 objects/s recovering 2024-02-13T00:45:29.757 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:28.452558+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e373: 8 total, 8 up, 5 in 2024-02-13T00:45:29.757 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:28.453688+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:29.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.335+0000 7fdf679c3700 1 -- 172.21.15.162:0/1211925777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf600fa440 msgr2=0x7fdf600f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.335+0000 7fdf679c3700 1 --2- 172.21.15.162:0/1211925777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf600fa440 0x7fdf600f7e20 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 rx=0x7fdf500047b0 tx=0x7fdf50009cb0).stop 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.335+0000 7fdf679c3700 1 -- 172.21.15.162:0/1211925777 shutdown_connections 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.335+0000 7fdf679c3700 1 --2- 172.21.15.162:0/1211925777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf600f8500 0x7fdf60104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.335+0000 7fdf679c3700 1 --2- 172.21.15.162:0/1211925777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf600f9b70 0x7fdf600f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.335+0000 7fdf679c3700 1 --2- 172.21.15.162:0/1211925777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf600fa440 0x7fdf600f7e20 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.335+0000 7fdf679c3700 1 -- 172.21.15.162:0/1211925777 >> 172.21.15.162:0/1211925777 conn(0x7fdf600f38c0 msgr2=0x7fdf600f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.335+0000 7fdf679c3700 1 -- 172.21.15.162:0/1211925777 shutdown_connections 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.335+0000 7fdf679c3700 1 -- 172.21.15.162:0/1211925777 wait complete. 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf679c3700 1 librados: starting msgr at 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf679c3700 1 librados: starting objecter 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf679c3700 1 Processor -- start 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf679c3700 1 -- start start 2024-02-13T00:45:29.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf679c3700 1 librados: setting wanted keys 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf679c3700 1 librados: calling monclient init 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf679c3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf600f8500 0x7fdf6006b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf679c3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf600f9b70 0x7fdf6006b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf679c3700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf600fa440 0x7fdf6006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf679c3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf601167e0 con 0x7fdf600f9b70 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf679c3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf60116950 con 0x7fdf600f8500 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf679c3700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf60116b10 con 0x7fdf600fa440 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf65f60700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf600fa440 0x7fdf6006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf64f5e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf600f9b70 0x7fdf6006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf65f60700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf600fa440 0x7fdf6006bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:43180/0 (socket says 172.21.15.162:43180) 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf64f5e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf600f9b70 0x7fdf6006b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:52508/0 (socket says 172.21.15.162:52508) 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf65f60700 1 -- 172.21.15.162:0/4261859725 learned_addr learned my addr 172.21.15.162:0/4261859725 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf6575f700 1 --2- 172.21.15.162:0/4261859725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf600f8500 0x7fdf6006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:29.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf65f60700 1 -- 172.21.15.162:0/4261859725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf600f8500 msgr2=0x7fdf6006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.336+0000 7fdf65f60700 1 --2- 172.21.15.162:0/4261859725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf600f8500 0x7fdf6006b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf65f60700 1 -- 172.21.15.162:0/4261859725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf600f9b70 msgr2=0x7fdf6006b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf65f60700 1 --2- 172.21.15.162:0/4261859725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf600f9b70 0x7fdf6006b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf65f60700 1 -- 172.21.15.162:0/4261859725 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c005040 con 0x7fdf600fa440 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf64f5e700 1 --2- 172.21.15.162:0/4261859725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf600f9b70 0x7fdf6006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf65f60700 1 --2- 172.21.15.162:0/4261859725 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf600fa440 0x7fdf6006bb50 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7fdf5800bc30 tx=0x7fdf58009840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf567fc700 10 client.?.objecter ms_handle_connect 0x7fdf600fa440 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf567fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf567fc700 1 -- 172.21.15.162:0/4261859725 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5800c6f0 con 0x7fdf600fa440 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf679c3700 1 -- 172.21.15.162:0/4261859725 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf50005040 con 0x7fdf600fa440 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf567fc700 1 -- 172.21.15.162:0/4261859725 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf58010070 con 0x7fdf600fa440 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf567fc700 1 -- 172.21.15.162:0/4261859725 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf58007880 con 0x7fdf600fa440 2024-02-13T00:45:29.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf679c3700 10 client.37331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf679c3700 1 -- 172.21.15.162:0/4261859725 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf601171a0 con 0x7fdf600fa440 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.337+0000 7fdf679c3700 1 librados: init done 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.338+0000 7fdf679c3700 1 -- 172.21.15.162:0/4261859725 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf600475e0 con 0x7fdf600fa440 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.338+0000 7fdf567fc700 1 -- 172.21.15.162:0/4261859725 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fdf58007c30 con 0x7fdf600fa440 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.339+0000 7fdf567fc700 1 --2- 172.21.15.162:0/4261859725 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf4c062c90 0x7fdf4c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.339+0000 7fdf567fc700 1 -- 172.21.15.162:0/4261859725 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(373..373 src has 52..373) v4 ==== 8052+0+0 (secure 0 0 0) 0x7fdf5807cbc0 con 0x7fdf600fa440 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.339+0000 7fdf567fc700 10 client.37331.objecter ms_dispatch 0x7fdf60001550 osd_map(373..373 src has 52..373) v4 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.339+0000 7fdf6575f700 1 --2- 172.21.15.162:0/4261859725 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf4c062c90 0x7fdf4c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.339+0000 7fdf567fc700 3 client.37331.objecter handle_osd_map got epochs [373,373] > 0 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.339+0000 7fdf567fc700 3 client.37331.objecter handle_osd_map decoding full epoch 373 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.339+0000 7fdf567fc700 20 client.37331.objecter dump_active .. 0 homeless 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.339+0000 7fdf567fc700 10 client.37331.objecter ms_handle_connect 0x7fdf4c062c90 2024-02-13T00:45:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.339+0000 7fdf6575f700 1 --2- 172.21.15.162:0/4261859725 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf4c062c90 0x7fdf4c065150 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 rx=0x7fdf5c009f20 tx=0x7fdf5c009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:29.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.343+0000 7fdf567fc700 1 -- 172.21.15.162:0/4261859725 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fdf58050070 con 0x7fdf600fa440 2024-02-13T00:45:29.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.519+0000 7fdf679c3700 1 -- 172.21.15.162:0/4261859725 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fdf6006c650 con 0x7fdf600fa440 2024-02-13T00:45:29.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.521+0000 7fdf567fc700 1 -- 172.21.15.162:0/4261859725 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v374) v1 ==== 81+0+48102 (secure 0 0 0) 0x7fdf5801a070 con 0x7fdf600fa440 2024-02-13T00:45:29.781 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:45:29.781 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "epoch": 374, 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "created": "2024-02-13T00:31:51.797017+0000", 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "modified": "2024-02-13T00:45:29.445853+0000", 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "last_up_change": "2024-02-13T00:37:33.855829+0000", 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "last_in_change": "2024-02-13T00:45:14.405413+0000", 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_num": 5804032, 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_set": [ 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "nodeep-scrub", 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "pglog_hardlimit", 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "purged_snapdirs", 2024-02-13T00:45:29.782 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_deletes", 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: "sortbitwise" 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_version": 98, 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: "full_ratio": 0.94999998807907104, 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_snapshot": "", 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_max": 204, 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: "max_osd": 8, 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: "require_min_compat_client": "luminous", 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: "min_compat_client": "luminous", 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: "require_osd_release": "pacific", 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: "pools": [ 2024-02-13T00:45:29.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.784 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 1, 2024-02-13T00:45:29.784 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "device_health_metrics", 2024-02-13T00:45:29.784 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:35:34.545390+0000", 2024-02-13T00:45:29.784 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:45:29.784 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:45:29.784 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:45:29.784 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:45:29.784 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:45:29.784 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:45:29.784 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:45:29.784 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 1, 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 1, 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 1, 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 1, 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 1, 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:45:29.785 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "71", 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:45:29.786 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:45:29.787 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:45:29.788 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:45:29.788 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:45:29.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.788 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:45:29.788 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:45:29.788 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:45:29.788 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:45:29.788 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:45:29.788 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:45:29.788 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:45:29.788 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:45:29.788 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:45:29.789 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:45:29.789 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:45:29.789 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:45:29.789 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_max": 32, 2024-02-13T00:45:29.789 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_min": 1 2024-02-13T00:45:29.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.790 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:45:29.790 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_devicehealth": {} 2024-02-13T00:45:29.790 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.790 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 2, 2024-02-13T00:45:29.790 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": ".rgw.root", 2024-02-13T00:45:29.790 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.459980+0000", 2024-02-13T00:45:29.790 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:45:29.790 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:45:29.790 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:45:29.791 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:45:29.792 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:45:29.792 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:45:29.792 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:45:29.792 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:45:29.792 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:45:29.792 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:45:29.792 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:45:29.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.792 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "58", 2024-02-13T00:45:29.792 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:45:29.792 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:45:29.792 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:45:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:45:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.796 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 3, 2024-02-13T00:45:29.796 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "iscsi", 2024-02-13T00:45:29.796 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.740219+0000", 2024-02-13T00:45:29.796 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:45:29.796 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:45:29.796 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:45:29.796 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:45:29.796 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:45:29.796 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:45:29.796 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:45:29.796 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:45:29.796 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 3, 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 3, 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 3, 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 3, 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 3, 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:45:29.797 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:45:29.798 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:45:29.799 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:45:29.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.799 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "102", 2024-02-13T00:45:29.799 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:45:29.799 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:45:29.799 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:45:29.799 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:45:29.799 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:45:29.799 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:45:29.800 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:45:29.801 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd": {} 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 4, 2024-02-13T00:45:29.802 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.log", 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:41.489292+0000", 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:45:29.803 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "60", 2024-02-13T00:45:29.804 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:45:29.805 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:45:29.806 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 5, 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.control", 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:43.466304+0000", 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:45:29.808 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:45:29.809 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:45:29.810 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:45:29.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.810 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "62", 2024-02-13T00:45:29.810 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:45:29.810 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:45:29.810 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:45:29.810 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:45:29.810 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:45:29.810 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:45:29.810 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:45:29.810 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:45:29.810 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:45:29.811 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:45:29.812 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 6, 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.meta", 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:45.542442+0000", 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:45:29.813 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 42, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 42, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 42, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 42, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 42, 2024-02-13T00:45:29.814 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "248", 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "246", 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:45:29.815 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:45:29.816 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:45:29.817 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_bias": 4 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 28, 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-13T00:45:29.818 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:00.190777+0000", 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:45:29.819 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:45:29.820 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:45:29.820 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:45:29.820 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:45:29.820 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:45:29.820 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:45:29.820 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:45:29.820 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:45:29.820 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:45:29.820 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:45:29.820 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:45:29.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.820 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "108", 2024-02-13T00:45:29.821 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:45:29.821 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:45:29.821 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:45:29.821 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:45:29.821 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:45:29.821 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:45:29.821 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:45:29.821 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:45:29.821 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:45:29.821 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:45:29.821 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:45:29.821 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:45:29.822 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:45:29.823 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 61, 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-6", 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:14.963279+0000", 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8193, 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:45:29.824 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:45:29.825 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "372", 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "371", 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "371", 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "371", 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 21, 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 358, 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:45:29.826 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.827 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:45:29.828 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:45:29.828 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:45:29.828 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:45:29.828 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:45:29.828 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:45:29.828 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:45:29.828 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:45:29.828 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:45:29.828 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:45:29.828 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:45:29.828 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:45:29.828 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:45:29.829 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:45:29.829 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.829 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 204, 2024-02-13T00:45:29.829 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-49", 2024-02-13T00:45:29.829 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:45:28.693403+0000", 2024-02-13T00:45:29.829 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 32769, 2024-02-13T00:45:29.829 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,creating", 2024-02-13T00:45:29.829 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:45:29.829 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:45:29.829 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:45:29.830 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "374", 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:45:29.831 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:45:29.832 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:45:29.833 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99", 2024-02-13T00:45:29.834 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 8, 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 371, 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.835 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6801", 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6802", 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.836 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6804", 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.837 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6803", 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6801/2103743338", 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6802/2103743338", 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6804/2103743338", 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6803/2103743338", 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886", 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 13, 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 371, 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:45:29.839 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6805", 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.840 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6806", 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6808", 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.841 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6807", 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6805/150599780", 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6806/150599780", 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6808/150599780", 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6807/150599780", 2024-02-13T00:45:29.842 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e", 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:45:29.843 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 18, 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 333, 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6809", 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6810", 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.845 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6812", 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6811", 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6809/3689719376", 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6810/3689719376", 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6812/3689719376", 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6811/3689719376", 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001", 2024-02-13T00:45:29.847 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 25, 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 255, 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.848 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6813", 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6814", 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6816", 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.850 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6815", 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6813/604971011", 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6814/604971011", 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6816/604971011", 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6815/604971011", 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3", 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 31, 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 81, 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:45:29.852 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6800", 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6801", 2024-02-13T00:45:29.853 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:45:29.854 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.854 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.854 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:45:29.854 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.854 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.854 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6803", 2024-02-13T00:45:29.854 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:45:29.854 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.854 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 10 librados: watch_flush enter 2024-02-13T00:45:29.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 10 librados: watch_flush exit 2024-02-13T00:45:29.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 20 client.37331.objecter shutdown clearing up homeless session... 2024-02-13T00:45:29.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 10 client.37331.objecter successfully canceled tick 2024-02-13T00:45:29.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 -- 172.21.15.162:0/4261859725 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf4c062c90 msgr2=0x7fdf4c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:29.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 --2- 172.21.15.162:0/4261859725 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf4c062c90 0x7fdf4c065150 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 rx=0x7fdf5c009f20 tx=0x7fdf5c009cb0).stop 2024-02-13T00:45:29.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 -- 172.21.15.162:0/4261859725 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf600fa440 msgr2=0x7fdf6006bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:29.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 --2- 172.21.15.162:0/4261859725 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf600fa440 0x7fdf6006bb50 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7fdf5800bc30 tx=0x7fdf58009840).stop 2024-02-13T00:45:29.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 -- 172.21.15.162:0/4261859725 shutdown_connections 2024-02-13T00:45:29.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 --2- 172.21.15.162:0/4261859725 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf4c062c90 0x7fdf4c065150 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:29.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 --2- 172.21.15.162:0/4261859725 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf600fa440 0x7fdf6006bb50 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:29.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 --2- 172.21.15.162:0/4261859725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf600f8500 0x7fdf6006b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:29.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 --2- 172.21.15.162:0/4261859725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf600f9b70 0x7fdf6006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:29.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 -- 172.21.15.162:0/4261859725 >> 172.21.15.162:0/4261859725 conn(0x7fdf600f38c0 msgr2=0x7fdf600f4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:29.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 -- 172.21.15.162:0/4261859725 shutdown_connections 2024-02-13T00:45:29.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 -- 172.21.15.162:0/4261859725 wait complete. 2024-02-13T00:45:29.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:29.523+0000 7fdf679c3700 1 librados: shutdown 2024-02-13T00:45:29.856 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:45:29.856 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6802", 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6800/3344263388", 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6801/3344263388", 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6803/3344263388", 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6802/3344263388", 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:45:29.857 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723", 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 39, 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 371, 2024-02-13T00:45:29.858 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:45:29.859 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:45:29.859 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:45:29.859 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.859 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.859 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6804", 2024-02-13T00:45:29.859 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:45:29.859 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.859 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.859 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:45:29.860 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.860 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.860 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6805", 2024-02-13T00:45:29.860 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:45:29.860 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.860 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.860 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:45:29.860 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.861 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.861 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6807", 2024-02-13T00:45:29.861 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:45:29.861 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.861 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.861 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:45:29.861 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.861 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.861 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6806", 2024-02-13T00:45:29.861 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:45:29.862 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.862 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.862 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6804/2212296748", 2024-02-13T00:45:29.862 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6805/2212296748", 2024-02-13T00:45:29.862 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2212296748", 2024-02-13T00:45:29.862 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2212296748", 2024-02-13T00:45:29.862 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:45:29.862 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:45:29.862 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:45:29.862 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474", 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 45, 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 371, 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:45:29.863 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6808", 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.864 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6809", 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6811", 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6810", 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6808/1506640341", 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6809/1506640341", 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1506640341", 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1506640341", 2024-02-13T00:45:29.866 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b", 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:45:29.867 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 53, 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 215, 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6812", 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.868 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6813", 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.869 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6815", 2024-02-13T00:45:29.870 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:45:29.870 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.870 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.870 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:45:29.870 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:45:29.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.870 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:45:29.870 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6814", 2024-02-13T00:45:29.870 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:45:29.870 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.870 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6812/1964192772", 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6813/1964192772", 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6815/1964192772", 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6814/1964192772", 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_xinfo": [ 2024-02-13T00:45:29.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.872 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:45:29.872 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:45:29.872 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:45:29.872 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:45:29.872 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:45:29.872 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:45:29.872 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:44:52.736961+0000", 2024-02-13T00:45:29.872 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:45:29.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.872 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:44:54.574410+0000", 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:45:29.873 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:45:29.874 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:45:29.874 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:45:29.874 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:45:24.986046+0000", 2024-02-13T00:45:29.874 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:45:29.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.874 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:45:29.874 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:45:29.874 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:45:29.874 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:45:29.874 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:45:29.874 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:45:08.985898+0000", 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:45:01.398497+0000", 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:44:48.972410+0000", 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:45:29.876 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:44:47.050701+0000", 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:45:29.877 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:45:05.371859+0000", 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap": [], 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap_items": [ 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.4", 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 0 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.6", 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 5 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.18", 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:45:29.879 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 5 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "5.5", 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 6 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "28.5", 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 5 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "28.12", 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:45:29.881 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 4 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_temp": [], 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_temp": [], 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: "blocklist": { 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/1909610488": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2370739667": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/1502590187": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/4121673848": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2412382714": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:45:29.882 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3041562701": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/2775563314": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3135632190": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/197307636": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2653247729": "2024-02-14T00:32:24.380625+0000" 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "range_blocklist": {}, 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profiles": { 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "default": { 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:45:29.883 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van" 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps_queue": [ 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 24, 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 9, 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 25, 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 60, 2024-02-13T00:45:29.886 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 78, 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.888 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 85, 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 94, 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.890 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 95, 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 4, 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 4 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 107, 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:45:29.891 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 113, 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 125, 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 10, 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.894 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 133, 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 12, 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 154, 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 14, 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.895 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 166, 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 17, 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 176, 2024-02-13T00:45:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 20, 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: "new_removed_snaps": [], 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: "new_purged_snaps": [], 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_node_flags": {}, 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: "device_class_flags": {}, 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode": { 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_enabled": false, 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_bucket_count": 0, 2024-02-13T00:45:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: "degraded_stretch_mode": 0, 2024-02-13T00:45:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: "recovering_stretch_mode": 0, 2024-02-13T00:45:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_bucket": 0 2024-02-13T00:45:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:45:29.898 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:45:29.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:28.407661+0000 mgr.y (mgr.14544) 330 : cluster [DBG] pgmap v529: 246 pgs: 8 unknown, 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 394 B/s, 8 objects/s recovering 2024-02-13T00:45:29.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:28.452558+0000 mon.a (mon.0) 2360 : cluster [DBG] osdmap e373: 8 total, 8 up, 5 in 2024-02-13T00:45:29.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:28.453688+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:30.019 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:30.503 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:29.216833+0000 osd.5 (osd.5) 160 : cluster [DBG] 61.18 scrub starts 2024-02-13T00:45:30.503 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:29.218587+0000 osd.5 (osd.5) 161 : cluster [DBG] 61.18 scrub ok 2024-02-13T00:45:30.503 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:29.238052+0000 osd.1 (osd.1) 147 : cluster [DBG] 61.1 scrub starts 2024-02-13T00:45:30.503 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:29.241601+0000 osd.1 (osd.1) 148 : cluster [DBG] 61.1 scrub ok 2024-02-13T00:45:30.503 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:29.452197+0000 mon.a (mon.0) 2362 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-83985-63"}]': finished 2024-02-13T00:45:30.503 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:29.452475+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e374: 8 total, 8 up, 5 in 2024-02-13T00:45:30.504 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:29.457817+0000 mon.b (mon.1) 541 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:30.504 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:29.459289+0000 mon.a (mon.0) 2364 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:30.504 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:29.459874+0000 mon.a (mon.0) 2365 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:30.504 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:29.521002+0000 mon.c (mon.2) 367 : audit [DBG] from='client.? 172.21.15.162:0/4261859725' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:45:30.504 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:30.393613+0000 mon.a (mon.0) 2366 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-83985-63"}]': finished 2024-02-13T00:45:30.504 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:30.393723+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:30.504 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:30.393793+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e375: 8 total, 8 up, 5 in 2024-02-13T00:45:30.504 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:30.424191+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:30.566 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:30.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:30.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:30.589 INFO:tasks.thrashosds.thrasher:j is {'epoch': 374, 'fsid': 'd4ee01fc-ca06-11ee-95b9-87774f69a715', 'created': '2024-02-13T00:31:51.797017+0000', 'modified': '2024-02-13T00:45:29.445853+0000', 'last_up_change': '2024-02-13T00:37:33.855829+0000', 'last_in_change': '2024-02-13T00:45:14.405413+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 98, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 204, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-13T00:35:34.545390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-13T00:37:39.459980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-13T00:37:39.740219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-13T00:37:41.489292+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-13T00:37:43.466304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-13T00:37:45.542442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '246', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 28, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-13T00:41:00.190777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 61, 'pool_name': 'test-rados-api-smithi162-84057-6', 'create_time': '2024-02-13T00:41:14.963279+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '372', 'last_force_op_resend': '371', 'last_force_op_resend_prenautilus': '371', 'last_force_op_resend_preluminous': '371', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 358, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 204, 'pool_name': 'test-rados-api-smithi162-84057-49', 'create_time': '2024-02-13T00:45:28.693403+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '374', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 371, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6801', 'nonce': 2103743338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 2103743338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 2103743338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6803', 'nonce': 2103743338}]}, 'public_addr': '172.21.15.162:6801/2103743338', 'cluster_addr': '172.21.15.162:6802/2103743338', 'heartbeat_back_addr': '172.21.15.162:6804/2103743338', 'heartbeat_front_addr': '172.21.15.162:6803/2103743338', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '39e9d2ef-9aa4-45bb-8a47-b0e0ec222886', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 371, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6805', 'nonce': 150599780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 150599780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 150599780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6807', 'nonce': 150599780}]}, 'public_addr': '172.21.15.162:6805/150599780', 'cluster_addr': '172.21.15.162:6806/150599780', 'heartbeat_back_addr': '172.21.15.162:6808/150599780', 'heartbeat_front_addr': '172.21.15.162:6807/150599780', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 333, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6809', 'nonce': 3689719376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3689719376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3689719376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6811', 'nonce': 3689719376}]}, 'public_addr': '172.21.15.162:6809/3689719376', 'cluster_addr': '172.21.15.162:6810/3689719376', 'heartbeat_back_addr': '172.21.15.162:6812/3689719376', 'heartbeat_front_addr': '172.21.15.162:6811/3689719376', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e6b9f0e-7a18-4cf8-9331-1a973fadc001', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 255, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6813', 'nonce': 604971011}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 604971011}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 604971011}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6815', 'nonce': 604971011}]}, 'public_addr': '172.21.15.162:6813/604971011', 'cluster_addr': '172.21.15.162:6814/604971011', 'heartbeat_back_addr': '172.21.15.162:6816/604971011', 'heartbeat_front_addr': '172.21.15.162:6815/604971011', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 3344263388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 3344263388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 3344263388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 3344263388}]}, 'public_addr': '172.21.15.181:6800/3344263388', 'cluster_addr': '172.21.15.181:6801/3344263388', 'heartbeat_back_addr': '172.21.15.181:6803/3344263388', 'heartbeat_front_addr': '172.21.15.181:6802/3344263388', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6aa2bd68-7f4b-4973-ac49-7fbc57c1a723', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 371, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2212296748}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2212296748}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2212296748}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2212296748}]}, 'public_addr': '172.21.15.181:6804/2212296748', 'cluster_addr': '172.21.15.181:6805/2212296748', 'heartbeat_back_addr': '172.21.15.181:6807/2212296748', 'heartbeat_front_addr': '172.21.15.181:6806/2212296748', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '791ba814-5284-432e-addc-d6b7f5f3d474', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 371, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1506640341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1506640341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1506640341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1506640341}]}, 'public_addr': '172.21.15.181:6808/1506640341', 'cluster_addr': '172.21.15.181:6809/1506640341', 'heartbeat_back_addr': '172.21.15.181:6811/1506640341', 'heartbeat_front_addr': '172.21.15.181:6810/1506640341', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f369c65-e663-4eb2-87a5-f2900f15b35b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 215, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 1964192772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 1964192772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 1964192772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 1964192772}]}, 'public_addr': '172.21.15.181:6812/1964192772', 'cluster_addr': '172.21.15.181:6813/1964192772', 'heartbeat_back_addr': '172.21.15.181:6815/1964192772', 'heartbeat_front_addr': '172.21.15.181:6814/1964192772', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:44:52.736961+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:44:54.574410+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:45:24.986046+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:45:08.985898+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:45:01.398497+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:44:48.972410+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:44:47.050701+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:45:05.371859+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.5', 'mappings': [{'from': 0, 'to': 6}]}, {'pgid': '28.5', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '28.12', 'mappings': [{'from': 0, 'to': 4}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.162:6800/1909610488': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2370739667': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/1502590187': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/4121673848': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:0/2412382714': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/3041562701': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/2775563314': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/3135632190': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/197307636': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2653247729': '2024-02-14T00:32:24.380625+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 24, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 60, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 78, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 94, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 113, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 166, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-13T00:45:30.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:45:30.670 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:30.703 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:30.724 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:30.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:29.216833+0000 osd.5 (osd.5) 160 : cluster [DBG] 61.18 scrub starts 2024-02-13T00:45:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:29.218587+0000 osd.5 (osd.5) 161 : cluster [DBG] 61.18 scrub ok 2024-02-13T00:45:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:29.238052+0000 osd.1 (osd.1) 147 : cluster [DBG] 61.1 scrub starts 2024-02-13T00:45:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:29.241601+0000 osd.1 (osd.1) 148 : cluster [DBG] 61.1 scrub ok 2024-02-13T00:45:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:29.452197+0000 mon.a (mon.0) 2362 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-83985-63"}]': finished 2024-02-13T00:45:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:29.452475+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e374: 8 total, 8 up, 5 in 2024-02-13T00:45:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:29.457817+0000 mon.b (mon.1) 541 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:29.459289+0000 mon.a (mon.0) 2364 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:30.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:29.459874+0000 mon.a (mon.0) 2365 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:30.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:29.521002+0000 mon.c (mon.2) 367 : audit [DBG] from='client.? 172.21.15.162:0/4261859725' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:45:30.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:30.393613+0000 mon.a (mon.0) 2366 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-83985-63"}]': finished 2024-02-13T00:45:30.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:30.393723+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:30.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:30.393793+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e375: 8 total, 8 up, 5 in 2024-02-13T00:45:30.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:30.424191+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:30.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:30.425618+0000 mon.a (mon.0) 2370 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:30.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:30.426067+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi162-83985-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:30.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:30.435295+0000 mon.b (mon.1) 542 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:45:30.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:30.435914+0000 mon.a (mon.0) 2372 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:45:30.799 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:30.425618+0000 mon.a (mon.0) 2370 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:30.799 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:30.426067+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi162-83985-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:30.799 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:30.435295+0000 mon.b (mon.1) 542 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:45:30.800 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:30.435914+0000 mon.a (mon.0) 2372 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:45:30.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:29.216833+0000 osd.5 (osd.5) 160 : cluster [DBG] 61.18 scrub starts 2024-02-13T00:45:30.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:29.218587+0000 osd.5 (osd.5) 161 : cluster [DBG] 61.18 scrub ok 2024-02-13T00:45:30.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:29.238052+0000 osd.1 (osd.1) 147 : cluster [DBG] 61.1 scrub starts 2024-02-13T00:45:30.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:29.241601+0000 osd.1 (osd.1) 148 : cluster [DBG] 61.1 scrub ok 2024-02-13T00:45:30.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:29.452197+0000 mon.a (mon.0) 2362 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-83985-63"}]': finished 2024-02-13T00:45:30.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:29.452475+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e374: 8 total, 8 up, 5 in 2024-02-13T00:45:30.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:29.457817+0000 mon.b (mon.1) 541 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:30.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:29.459289+0000 mon.a (mon.0) 2364 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-83985-63"}]: dispatch 2024-02-13T00:45:30.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:29.459874+0000 mon.a (mon.0) 2365 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:30.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:29.521002+0000 mon.c (mon.2) 367 : audit [DBG] from='client.? 172.21.15.162:0/4261859725' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:45:30.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:30.393613+0000 mon.a (mon.0) 2366 : audit [INF] from='client.? 172.21.15.162:0/1895266235' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-83985-63"}]': finished 2024-02-13T00:45:30.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:30.393723+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:30.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:30.393793+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e375: 8 total, 8 up, 5 in 2024-02-13T00:45:30.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:30.424191+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:30.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:30.425618+0000 mon.a (mon.0) 2370 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:30.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:30.426067+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi162-83985-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:30.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:30.435295+0000 mon.b (mon.1) 542 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:45:30.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:30.435914+0000 mon.a (mon.0) 2372 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:45:30.827 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:30.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:30.866 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:31.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:31.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:31.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:31.621 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:30.183021+0000 osd.6 (osd.6) 79 : cluster [DBG] 28.0 scrub starts 2024-02-13T00:45:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:30.186125+0000 osd.6 (osd.6) 80 : cluster [DBG] 28.0 scrub ok 2024-02-13T00:45:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:30.408180+0000 mgr.y (mgr.14544) 331 : cluster [DBG] pgmap v533: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:45:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:31.407750+0000 mon.a (mon.0) 2373 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi162-83985-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:31.407826+0000 mon.a (mon.0) 2374 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:45:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:31.407892+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e376: 8 total, 8 up, 5 in 2024-02-13T00:45:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:31.412041+0000 mon.b (mon.1) 543 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:31.412094+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi162-83985-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:31.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:31.412776+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:31.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:30.183021+0000 osd.6 (osd.6) 79 : cluster [DBG] 28.0 scrub starts 2024-02-13T00:45:31.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:30.186125+0000 osd.6 (osd.6) 80 : cluster [DBG] 28.0 scrub ok 2024-02-13T00:45:31.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:30.408180+0000 mgr.y (mgr.14544) 331 : cluster [DBG] pgmap v533: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:45:31.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:31.407750+0000 mon.a (mon.0) 2373 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi162-83985-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:31.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:31.407826+0000 mon.a (mon.0) 2374 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:45:31.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:31.407892+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e376: 8 total, 8 up, 5 in 2024-02-13T00:45:31.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:31.412041+0000 mon.b (mon.1) 543 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:31.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:31.412094+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi162-83985-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:31.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:31.412776+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:30.183021+0000 osd.6 (osd.6) 79 : cluster [DBG] 28.0 scrub starts 2024-02-13T00:45:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:30.186125+0000 osd.6 (osd.6) 80 : cluster [DBG] 28.0 scrub ok 2024-02-13T00:45:31.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:30.408180+0000 mgr.y (mgr.14544) 331 : cluster [DBG] pgmap v533: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:45:31.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:31.407750+0000 mon.a (mon.0) 2373 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi162-83985-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:31.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:31.407826+0000 mon.a (mon.0) 2374 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:45:31.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:31.407892+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e376: 8 total, 8 up, 5 in 2024-02-13T00:45:31.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:31.412041+0000 mon.b (mon.1) 543 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:31.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:31.412094+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi162-83985-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:31.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:31.412776+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:31.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:31.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:31.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:32.010 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:32.608 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:33.009 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:31.180777+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:45:33.009 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:31.183678+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:45:33.010 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:31.251370+0000 osd.5 (osd.5) 162 : cluster [DBG] 61.c scrub starts 2024-02-13T00:45:33.010 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:31.255144+0000 osd.5 (osd.5) 163 : cluster [DBG] 61.c scrub ok 2024-02-13T00:45:33.010 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:31.180777+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:45:33.010 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:31.183678+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:45:33.010 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:31.251370+0000 osd.5 (osd.5) 162 : cluster [DBG] 61.c scrub starts 2024-02-13T00:45:33.010 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:31.255144+0000 osd.5 (osd.5) 163 : cluster [DBG] 61.c scrub ok 2024-02-13T00:45:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:31.180777+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:45:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:31.183678+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:45:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:31.251370+0000 osd.5 (osd.5) 162 : cluster [DBG] 61.c scrub starts 2024-02-13T00:45:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:31.255144+0000 osd.5 (osd.5) 163 : cluster [DBG] 61.c scrub ok 2024-02-13T00:45:33.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.723+0000 7f66899fe700 1 -- 172.21.15.162:0/1295099564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6684102170 msgr2=0x7f6684104560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.723+0000 7f66899fe700 1 --2- 172.21.15.162:0/1295099564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6684102170 0x7f6684104560 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7f667c009c30 tx=0x7f667c004c50).stop 2024-02-13T00:45:33.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.723+0000 7f66899fe700 1 -- 172.21.15.162:0/1295099564 shutdown_connections 2024-02-13T00:45:33.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.723+0000 7f66899fe700 1 --2- 172.21.15.162:0/1295099564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6684102170 0x7f6684104560 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.723+0000 7f66899fe700 1 --2- 172.21.15.162:0/1295099564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66840f8c10 0x7f66840f8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.723+0000 7f66899fe700 1 --2- 172.21.15.162:0/1295099564 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66840f9570 0x7f6684101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.723+0000 7f66899fe700 1 -- 172.21.15.162:0/1295099564 >> 172.21.15.162:0/1295099564 conn(0x7f66840f2740 msgr2=0x7f66840f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:33.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.723+0000 7f66899fe700 1 -- 172.21.15.162:0/1295099564 shutdown_connections 2024-02-13T00:45:33.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.723+0000 7f66899fe700 1 -- 172.21.15.162:0/1295099564 wait complete. 2024-02-13T00:45:33.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.724+0000 7f66899fe700 1 librados: starting msgr at 2024-02-13T00:45:33.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.724+0000 7f66899fe700 1 librados: starting objecter 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.724+0000 7f66899fe700 1 Processor -- start 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.724+0000 7f66899fe700 1 -- start start 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.724+0000 7f66899fe700 1 librados: setting wanted keys 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.724+0000 7f66899fe700 1 librados: calling monclient init 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.724+0000 7f66899fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66840f8c10 0x7f66840f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.724+0000 7f66899fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66840f9570 0x7f66840f70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.724+0000 7f66899fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6684102170 0x7f66840f75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.724+0000 7f66899fe700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6684116600 con 0x7f66840f9570 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.724+0000 7f66899fe700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6684116770 con 0x7f6684102170 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.724+0000 7f66899fe700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6684116930 con 0x7f66840f8c10 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66827fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66840f9570 0x7f66840f70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66827fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66840f9570 0x7f66840f70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:52542/0 (socket says 172.21.15.162:52542) 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f6682ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66840f8c10 0x7f66840f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:33.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66827fc700 1 -- 172.21.15.162:0/917498487 learned_addr learned my addr 172.21.15.162:0/917498487 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:33.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66837fe700 1 --2- 172.21.15.162:0/917498487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6684102170 0x7f66840f75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66827fc700 1 -- 172.21.15.162:0/917498487 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66840f8c10 msgr2=0x7f66840f6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66827fc700 1 --2- 172.21.15.162:0/917498487 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66840f8c10 0x7f66840f6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66827fc700 1 -- 172.21.15.162:0/917498487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6684102170 msgr2=0x7f66840f75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66827fc700 1 --2- 172.21.15.162:0/917498487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6684102170 0x7f66840f75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66827fc700 1 -- 172.21.15.162:0/917498487 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6678005040 con 0x7f66840f9570 2024-02-13T00:45:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f6682ffd700 1 --2- 172.21.15.162:0/917498487 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66840f8c10 0x7f66840f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:45:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f666bfff700 10 client.?.objecter ms_handle_connect 0x7f66840f9570 2024-02-13T00:45:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f666bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66827fc700 1 --2- 172.21.15.162:0/917498487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66840f9570 0x7f66840f70b0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 rx=0x7f6678009400 tx=0x7f6678009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6678007a40 con 0x7f66840f9570 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f667800f040 con 0x7f66840f9570 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667c0097c0 con 0x7f66840f9570 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6678014a70 con 0x7f66840f9570 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66899fe700 10 client.27237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6684116f90 con 0x7f66840f9570 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.725+0000 7f66899fe700 1 librados: init done 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.726+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f66840607b0 con 0x7f66840f9570 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.727+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6678010530 con 0x7f66840f9570 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.727+0000 7f666bfff700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6800/2866534053 conn(0x7f666c062c90 0x7f666c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.727+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(378..378 src has 52..378) v4 ==== 8830+0+0 (secure 0 0 0) 0x7f667807d4f0 con 0x7f66840f9570 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.727+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 osd_map(378..378 src has 52..378) v4 2024-02-13T00:45:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.727+0000 7f666bfff700 3 client.27237.objecter handle_osd_map got epochs [378,378] > 0 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.727+0000 7f666bfff700 3 client.27237.objecter handle_osd_map decoding full epoch 378 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.727+0000 7f6682ffd700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6800/2866534053 conn(0x7f666c062c90 0x7f666c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.727+0000 7f666bfff700 20 client.27237.objecter dump_active .. 0 homeless 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.727+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v378) v1 ==== 54+0+16 (secure 0 0 0) 0x7f667807d9d0 con 0x7f66840f9570 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f666bfff700 10 client.27237.objecter ms_handle_connect 0x7f666c062c90 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f6682ffd700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6800/2866534053 conn(0x7f666c062c90 0x7f666c065150 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 rx=0x7f66740103c0 tx=0x7f667400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66899fe700 10 client.27237.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 1 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6801/2103743338 conn(0x7f66840fc100 0x7f66840fc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f66840475e0 osd=0 3 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f66840475e0 osd=0 3 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f66840475e0 osd=0 3 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 1 2024-02-13T00:45:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f66840475e0 osd=0 3 2024-02-13T00:45:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 0 1 2024-02-13T00:45:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f66840475e0 osd=0 4 2024-02-13T00:45:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66899fe700 10 client.27237.objecter _send_command 1 2024-02-13T00:45:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66841071a0 con 0x7f66840fc100 2024-02-13T00:45:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66837fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6801/2103743338 conn(0x7f66840fc100 0x7f66840fc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f666bfff700 10 client.27237.objecter ms_handle_connect 0x7f66840fc100 2024-02-13T00:45:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f66837fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6801/2103743338 conn(0x7f66840fc100 0x7f66840fc520 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f66841071a0 con 0x7f66840fc100 2024-02-13T00:45:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:45:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f666bfff700 10 client.27237.objecter _finish_command 1 = system:0 2024-02-13T00:45:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f66840475e0 osd=0 4 2024-02-13T00:45:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.728+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 0 1 2024-02-13T00:45:33.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.743+0000 7f66899fe700 10 client.27237.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:33.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.743+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 2 2024-02-13T00:45:33.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.743+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f66840475e0 osd=0 3 2024-02-13T00:45:33.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.743+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f66840475e0 osd=0 3 2024-02-13T00:45:33.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.743+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f66840475e0 osd=0 3 2024-02-13T00:45:33.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.743+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 2 2024-02-13T00:45:33.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.743+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f66840475e0 osd=0 3 2024-02-13T00:45:33.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.743+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 0 2 2024-02-13T00:45:33.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.743+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f66840475e0 osd=0 4 2024-02-13T00:45:33.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.743+0000 7f66899fe700 10 client.27237.objecter _send_command 2 2024-02-13T00:45:33.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.743+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6684027410 con 0x7f66840fc100 2024-02-13T00:45:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6684027410 con 0x7f66840fc100 2024-02-13T00:45:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f666bfff700 10 client.27237.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f66840475e0 osd=0 4 2024-02-13T00:45:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 0 2 2024-02-13T00:45:33.748 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:45:33.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f66899fe700 10 client.27237.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:33.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 3 2024-02-13T00:45:33.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6805/150599780 conn(0x7f6684059990 0x7f6684059db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:33.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f6684028760 osd=1 3 2024-02-13T00:45:33.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684028760 osd=1 3 2024-02-13T00:45:33.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f6684028760 osd=1 3 2024-02-13T00:45:33.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 3 2024-02-13T00:45:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f6684028760 osd=1 3 2024-02-13T00:45:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 1 3 2024-02-13T00:45:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684028760 osd=1 4 2024-02-13T00:45:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f66899fe700 10 client.27237.objecter _send_command 3 2024-02-13T00:45:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f668411d8a0 con 0x7f6684059990 2024-02-13T00:45:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f6682ffd700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6805/150599780 conn(0x7f6684059990 0x7f6684059db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f666bfff700 10 client.27237.objecter ms_handle_connect 0x7f6684059990 2024-02-13T00:45:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.745+0000 7f6682ffd700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6805/150599780 conn(0x7f6684059990 0x7f6684059db0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.746+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f668411d8a0 con 0x7f6684059990 2024-02-13T00:45:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.746+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:45:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.746+0000 7f666bfff700 10 client.27237.objecter _finish_command 3 = system:0 2024-02-13T00:45:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.746+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f6684028760 osd=1 4 2024-02-13T00:45:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.746+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 1 3 2024-02-13T00:45:33.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.760+0000 7f66899fe700 10 client.27237.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:33.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.760+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 4 2024-02-13T00:45:33.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.760+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f6684028760 osd=1 3 2024-02-13T00:45:33.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.760+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684028760 osd=1 3 2024-02-13T00:45:33.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.760+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f6684028760 osd=1 3 2024-02-13T00:45:33.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.760+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 4 2024-02-13T00:45:33.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.760+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f6684028760 osd=1 3 2024-02-13T00:45:33.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.760+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 1 4 2024-02-13T00:45:33.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.760+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684028760 osd=1 4 2024-02-13T00:45:33.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.760+0000 7f66899fe700 10 client.27237.objecter _send_command 4 2024-02-13T00:45:33.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.760+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6684059400 con 0x7f6684059990 2024-02-13T00:45:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6684059400 con 0x7f6684059990 2024-02-13T00:45:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f666bfff700 10 client.27237.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f6684028760 osd=1 4 2024-02-13T00:45:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 1 4 2024-02-13T00:45:33.765 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:45:33.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66899fe700 10 client.27237.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:33.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 5 2024-02-13T00:45:33.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6684028260 0x7f668411c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:33.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f6684059810 osd=2 3 2024-02-13T00:45:33.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684059810 osd=2 3 2024-02-13T00:45:33.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f6684059810 osd=2 3 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 5 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f6684059810 osd=2 3 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 2 5 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684059810 osd=2 4 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66899fe700 10 client.27237.objecter _send_command 5 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f668411c7b0 con 0x7f6684028260 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66827fc700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6684028260 0x7f668411c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f666bfff700 10 client.27237.objecter ms_handle_connect 0x7f6684028260 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.762+0000 7f66827fc700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6684028260 0x7f668411c2c0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.763+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f668411c7b0 con 0x7f6684028260 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.763+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.763+0000 7f666bfff700 10 client.27237.objecter _finish_command 5 = system:0 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.763+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f6684059810 osd=2 4 2024-02-13T00:45:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.763+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 2 5 2024-02-13T00:45:33.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.777+0000 7f66899fe700 10 client.27237.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:33.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.777+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 6 2024-02-13T00:45:33.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.777+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f6684059810 osd=2 3 2024-02-13T00:45:33.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.777+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684059810 osd=2 3 2024-02-13T00:45:33.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.777+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f6684059810 osd=2 3 2024-02-13T00:45:33.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.777+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 6 2024-02-13T00:45:33.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.777+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f6684059810 osd=2 3 2024-02-13T00:45:33.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.777+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 2 6 2024-02-13T00:45:33.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.777+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684059810 osd=2 4 2024-02-13T00:45:33.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.777+0000 7f66899fe700 10 client.27237.objecter _send_command 6 2024-02-13T00:45:33.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.777+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f66840277c0 con 0x7f6684028260 2024-02-13T00:45:33.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f66840277c0 con 0x7f6684028260 2024-02-13T00:45:33.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:33.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f666bfff700 10 client.27237.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f6684059810 osd=2 4 2024-02-13T00:45:33.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 2 6 2024-02-13T00:45:33.782 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:45:33.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66899fe700 10 client.27237.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 7 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6813/604971011 conn(0x7f668411fa20 0x7f6684121e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f6684027b70 osd=3 3 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684027b70 osd=3 3 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f6684027b70 osd=3 3 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 7 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f6684027b70 osd=3 3 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 3 7 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684027b70 osd=3 4 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66899fe700 10 client.27237.objecter _send_command 7 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f668411d680 con 0x7f668411fa20 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.778+0000 7f66837fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6813/604971011 conn(0x7f668411fa20 0x7f6684121e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.779+0000 7f666bfff700 10 client.27237.objecter ms_handle_connect 0x7f668411fa20 2024-02-13T00:45:33.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.779+0000 7f66837fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6813/604971011 conn(0x7f668411fa20 0x7f6684121e10 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:33.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.779+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f668411d680 con 0x7f668411fa20 2024-02-13T00:45:33.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.779+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:45:33.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.779+0000 7f666bfff700 10 client.27237.objecter _finish_command 7 = system:0 2024-02-13T00:45:33.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.779+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f6684027b70 osd=3 4 2024-02-13T00:45:33.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.779+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 3 7 2024-02-13T00:45:33.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.793+0000 7f66899fe700 10 client.27237.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:33.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.793+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 8 2024-02-13T00:45:33.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.793+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f6684027b70 osd=3 3 2024-02-13T00:45:33.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.793+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684027b70 osd=3 3 2024-02-13T00:45:33.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.793+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f6684027b70 osd=3 3 2024-02-13T00:45:33.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.793+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 8 2024-02-13T00:45:33.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.793+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f6684027b70 osd=3 3 2024-02-13T00:45:33.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.793+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 3 8 2024-02-13T00:45:33.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.793+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684027b70 osd=3 4 2024-02-13T00:45:33.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.793+0000 7f66899fe700 10 client.27237.objecter _send_command 8 2024-02-13T00:45:33.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.793+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f668411cbe0 con 0x7f668411fa20 2024-02-13T00:45:33.798 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f668411cbe0 con 0x7f668411fa20 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f666bfff700 10 client.27237.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f6684027b70 osd=3 4 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 3 8 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f66899fe700 10 client.27237.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 9 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6800/3344263388 conn(0x7f668411cd50 0x7f6684125490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668411cbe0 osd=4 3 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411cbe0 osd=4 3 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668411cbe0 osd=4 3 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 9 2024-02-13T00:45:33.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f668411cbe0 osd=4 3 2024-02-13T00:45:33.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 4 9 2024-02-13T00:45:33.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411cbe0 osd=4 4 2024-02-13T00:45:33.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f66899fe700 10 client.27237.objecter _send_command 9 2024-02-13T00:45:33.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f668411beb0 con 0x7f668411cd50 2024-02-13T00:45:33.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.795+0000 7f6682ffd700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6800/3344263388 conn(0x7f668411cd50 0x7f6684125490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:33.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.796+0000 7f666bfff700 10 client.27237.objecter ms_handle_connect 0x7f668411cd50 2024-02-13T00:45:33.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.796+0000 7f6682ffd700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6800/3344263388 conn(0x7f668411cd50 0x7f6684125490 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:33.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.796+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f668411beb0 con 0x7f668411cd50 2024-02-13T00:45:33.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.796+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:45:33.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.796+0000 7f666bfff700 10 client.27237.objecter _finish_command 9 = system:0 2024-02-13T00:45:33.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.796+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f668411cbe0 osd=4 4 2024-02-13T00:45:33.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.796+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 4 9 2024-02-13T00:45:33.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.811+0000 7f66899fe700 10 client.27237.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:33.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.811+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 10 2024-02-13T00:45:33.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.811+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668411cbe0 osd=4 3 2024-02-13T00:45:33.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.811+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411cbe0 osd=4 3 2024-02-13T00:45:33.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.811+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668411cbe0 osd=4 3 2024-02-13T00:45:33.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.811+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 10 2024-02-13T00:45:33.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.811+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f668411cbe0 osd=4 3 2024-02-13T00:45:33.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.811+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 4 10 2024-02-13T00:45:33.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.811+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411cbe0 osd=4 4 2024-02-13T00:45:33.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.811+0000 7f66899fe700 10 client.27237.objecter _send_command 10 2024-02-13T00:45:33.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.811+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f668411beb0 con 0x7f668411cd50 2024-02-13T00:45:33.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f668411beb0 con 0x7f668411cd50 2024-02-13T00:45:33.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:33.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f666bfff700 10 client.27237.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f668411cbe0 osd=4 4 2024-02-13T00:45:33.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 4 10 2024-02-13T00:45:33.817 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:45:33.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66899fe700 10 client.27237.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:33.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 11 2024-02-13T00:45:33.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6804/2212296748 conn(0x7f6684125ed0 0x7f6684128320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:33.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668411c150 osd=5 3 2024-02-13T00:45:33.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411c150 osd=5 3 2024-02-13T00:45:33.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668411c150 osd=5 3 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 11 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f668411c150 osd=5 3 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 5 11 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411c150 osd=5 4 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66899fe700 10 client.27237.objecter _send_command 11 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f668411beb0 con 0x7f6684125ed0 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.813+0000 7f66827fc700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6804/2212296748 conn(0x7f6684125ed0 0x7f6684128320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.814+0000 7f66827fc700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6804/2212296748 conn(0x7f6684125ed0 0x7f6684128320 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.814+0000 7f666bfff700 10 client.27237.objecter ms_handle_connect 0x7f6684125ed0 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.815+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f668411beb0 con 0x7f6684125ed0 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.815+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.815+0000 7f666bfff700 10 client.27237.objecter _finish_command 11 = system:0 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.815+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f668411c150 osd=5 4 2024-02-13T00:45:33.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.815+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 5 11 2024-02-13T00:45:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.829+0000 7f66899fe700 10 client.27237.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.829+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 12 2024-02-13T00:45:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.829+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668411c150 osd=5 3 2024-02-13T00:45:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.829+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411c150 osd=5 3 2024-02-13T00:45:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.829+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668411c150 osd=5 3 2024-02-13T00:45:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.829+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 12 2024-02-13T00:45:33.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.829+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f668411c150 osd=5 3 2024-02-13T00:45:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.829+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 5 12 2024-02-13T00:45:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.829+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411c150 osd=5 4 2024-02-13T00:45:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.829+0000 7f66899fe700 10 client.27237.objecter _send_command 12 2024-02-13T00:45:33.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.829+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f668411beb0 con 0x7f6684125ed0 2024-02-13T00:45:33.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f668411beb0 con 0x7f6684125ed0 2024-02-13T00:45:33.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:33.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f666bfff700 10 client.27237.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f668411c150 osd=5 4 2024-02-13T00:45:33.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 5 12 2024-02-13T00:45:33.834 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:45:33.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66899fe700 10 client.27237.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:33.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 13 2024-02-13T00:45:33.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6808/1506640341 conn(0x7f668412a420 0x7f668412a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:33.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668411beb0 osd=6 3 2024-02-13T00:45:33.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411beb0 osd=6 3 2024-02-13T00:45:33.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668411beb0 osd=6 3 2024-02-13T00:45:33.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 13 2024-02-13T00:45:33.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f668411beb0 osd=6 3 2024-02-13T00:45:33.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 6 13 2024-02-13T00:45:33.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411beb0 osd=6 4 2024-02-13T00:45:33.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66899fe700 10 client.27237.objecter _send_command 13 2024-02-13T00:45:33.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f668412af10 con 0x7f668412a420 2024-02-13T00:45:33.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.831+0000 7f66837fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6808/1506640341 conn(0x7f668412a420 0x7f668412a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:33.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.832+0000 7f666bfff700 10 client.27237.objecter ms_handle_connect 0x7f668412a420 2024-02-13T00:45:33.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.832+0000 7f66837fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6808/1506640341 conn(0x7f668412a420 0x7f668412a8a0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:33.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.832+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f668412af10 con 0x7f668412a420 2024-02-13T00:45:33.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.832+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:45:33.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.832+0000 7f666bfff700 10 client.27237.objecter _finish_command 13 = system:0 2024-02-13T00:45:33.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.832+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f668411beb0 osd=6 4 2024-02-13T00:45:33.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.832+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 6 13 2024-02-13T00:45:33.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.846+0000 7f66899fe700 10 client.27237.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:33.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.846+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 14 2024-02-13T00:45:33.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.846+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668411beb0 osd=6 3 2024-02-13T00:45:33.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.846+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411beb0 osd=6 3 2024-02-13T00:45:33.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.846+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668411beb0 osd=6 3 2024-02-13T00:45:33.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.846+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 14 2024-02-13T00:45:33.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.846+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f668411beb0 osd=6 3 2024-02-13T00:45:33.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.846+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 6 14 2024-02-13T00:45:33.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.846+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411beb0 osd=6 4 2024-02-13T00:45:33.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.846+0000 7f66899fe700 10 client.27237.objecter _send_command 14 2024-02-13T00:45:33.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.846+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f668412af10 con 0x7f668412a420 2024-02-13T00:45:33.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.848+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f668412af10 con 0x7f668412a420 2024-02-13T00:45:33.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.848+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:33.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.848+0000 7f666bfff700 10 client.27237.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.848+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f668411beb0 osd=6 4 2024-02-13T00:45:33.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.848+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 6 14 2024-02-13T00:45:33.852 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:45:33.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f66899fe700 10 client.27237.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:33.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 15 2024-02-13T00:45:33.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6812/1964192772 conn(0x7f668412d720 0x7f668412db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:33.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668412af10 osd=7 3 2024-02-13T00:45:33.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668412af10 osd=7 3 2024-02-13T00:45:33.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668412af10 osd=7 3 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 15 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f668412af10 osd=7 3 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 7 15 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668412af10 osd=7 4 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f66899fe700 10 client.27237.objecter _send_command 15 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f668412e1f0 con 0x7f668412d720 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f6682ffd700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6812/1964192772 conn(0x7f668412d720 0x7f668412db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f666bfff700 10 client.27237.objecter ms_handle_connect 0x7f668412d720 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.849+0000 7f6682ffd700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6812/1964192772 conn(0x7f668412d720 0x7f668412db80 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.850+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f668412e1f0 con 0x7f668412d720 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.850+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.850+0000 7f666bfff700 10 client.27237.objecter _finish_command 15 = system:0 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.850+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f668412af10 osd=7 4 2024-02-13T00:45:33.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.850+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 7 15 2024-02-13T00:45:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:33.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:33.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:33.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.867+0000 7f66899fe700 10 client.27237.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:33.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.867+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign -1 16 2024-02-13T00:45:33.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.867+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668412af10 osd=7 3 2024-02-13T00:45:33.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.867+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668412af10 osd=7 3 2024-02-13T00:45:33.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.867+0000 7f66899fe700 20 client.27237.objecter _get_session s=0x7f668412af10 osd=7 3 2024-02-13T00:45:33.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.867+0000 7f66899fe700 15 client.27237.objecter _session_command_op_remove -1 16 2024-02-13T00:45:33.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.867+0000 7f66899fe700 20 client.27237.objecter get_session s=0x7f668412af10 osd=7 3 2024-02-13T00:45:33.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.867+0000 7f66899fe700 15 client.27237.objecter _session_command_op_assign 7 16 2024-02-13T00:45:33.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.867+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668412af10 osd=7 4 2024-02-13T00:45:33.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.867+0000 7f66899fe700 10 client.27237.objecter _send_command 16 2024-02-13T00:45:33.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.867+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f668412e1f0 con 0x7f668412d720 2024-02-13T00:45:33.874 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:45:33.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f666bfff700 1 -- 172.21.15.162:0/917498487 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f668412e1f0 con 0x7f668412d720 2024-02-13T00:45:33.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f666bfff700 10 client.27237.objecter ms_dispatch 0x7f6684001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:33.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f666bfff700 10 client.27237.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f666bfff700 20 client.27237.objecter put_session s=0x7f668412af10 osd=7 4 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f666bfff700 15 client.27237.objecter _session_command_op_remove 7 16 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 10 librados: watch_flush enter 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 10 librados: watch_flush exit 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 10 client.27237.objecter close_session for osd.0 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6801/2103743338 conn(0x7f66840fc100 msgr2=0x7f66840fc520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6801/2103743338 conn(0x7f66840fc100 0x7f66840fc520 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f66840475e0 osd=0 1 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 10 client.27237.objecter close_session for osd.1 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6805/150599780 conn(0x7f6684059990 msgr2=0x7f6684059db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6805/150599780 conn(0x7f6684059990 0x7f6684059db0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684028760 osd=1 1 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 10 client.27237.objecter close_session for osd.2 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6684028260 msgr2=0x7f668411c2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6684028260 0x7f668411c2c0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684059810 osd=2 1 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 10 client.27237.objecter close_session for osd.3 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6813/604971011 conn(0x7f668411fa20 msgr2=0x7f6684121e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6813/604971011 conn(0x7f668411fa20 0x7f6684121e10 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f6684027b70 osd=3 1 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 10 client.27237.objecter close_session for osd.4 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6800/3344263388 conn(0x7f668411cd50 msgr2=0x7f6684125490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6800/3344263388 conn(0x7f668411cd50 0x7f6684125490 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411cbe0 osd=4 1 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 10 client.27237.objecter close_session for osd.5 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6804/2212296748 conn(0x7f6684125ed0 msgr2=0x7f6684128320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6804/2212296748 conn(0x7f6684125ed0 0x7f6684128320 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411c150 osd=5 1 2024-02-13T00:45:33.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 10 client.27237.objecter close_session for osd.6 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6808/1506640341 conn(0x7f668412a420 msgr2=0x7f668412a8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6808/1506640341 conn(0x7f668412a420 0x7f668412a8a0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668411beb0 osd=6 1 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 10 client.27237.objecter close_session for osd.7 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6812/1964192772 conn(0x7f668412d720 msgr2=0x7f668412db80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.181:6812/1964192772 conn(0x7f668412d720 0x7f668412db80 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 20 client.27237.objecter put_session s=0x7f668412af10 osd=7 1 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 20 client.27237.objecter shutdown clearing up homeless session... 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.869+0000 7f66899fe700 10 client.27237.objecter successfully canceled tick 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66837fe700 1 -- 172.21.15.162:0/917498487 reap_dead start 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66837fe700 1 -- 172.21.15.162:0/917498487 reap_dead start 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66837fe700 1 -- 172.21.15.162:0/917498487 reap_dead start 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66837fe700 1 -- 172.21.15.162:0/917498487 reap_dead start 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66837fe700 1 -- 172.21.15.162:0/917498487 reap_dead start 2024-02-13T00:45:33.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6800/2866534053 conn(0x7f666c062c90 msgr2=0x7f666c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6800/2866534053 conn(0x7f666c062c90 0x7f666c065150 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 rx=0x7f66740103c0 tx=0x7f667400b040).stop 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66840f9570 msgr2=0x7f66840f70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66840f9570 0x7f66840f70b0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 rx=0x7f6678009400 tx=0x7f6678009cb0).stop 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 shutdown_connections 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66840f9570 0x7f66840f70b0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66899fe700 1 --2- 172.21.15.162:0/917498487 >> v2:172.21.15.162:6800/2866534053 conn(0x7f666c062c90 0x7f666c065150 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 >> 172.21.15.162:0/917498487 conn(0x7f66840f2740 msgr2=0x7f66840f3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 shutdown_connections 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66899fe700 1 -- 172.21.15.162:0/917498487 wait complete. 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:33.870+0000 7f66899fe700 1 librados: shutdown 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.878 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.879 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.879 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.879 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.879 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:45:33.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:33.987 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:45:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:45:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-13T00:45:33.988 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:45:33.998 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:31.619507+0000 osd.0 (osd.0) 124 : cluster [DBG] 28.2 scrub starts 2024-02-13T00:45:33.998 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:31.621162+0000 osd.0 (osd.0) 125 : cluster [DBG] 28.2 scrub ok 2024-02-13T00:45:33.998 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:32.229644+0000 osd.1 (osd.1) 149 : cluster [DBG] 28.1 scrub starts 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:32.233193+0000 osd.1 (osd.1) 150 : cluster [DBG] 28.1 scrub ok 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:32.234450+0000 osd.5 (osd.5) 164 : cluster [DBG] 61.4 scrub starts 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:32.238120+0000 osd.5 (osd.5) 165 : cluster [DBG] 61.4 scrub ok 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:32.408581+0000 mgr.y (mgr.14544) 332 : cluster [DBG] pgmap v535: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:32.650893+0000 mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-49"}]': finished 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:32.650971+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e377: 8 total, 8 up, 5 in 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:32.652204+0000 mon.b (mon.1) 544 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-49", "mode": "writeback"}]: dispatch 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:32.656473+0000 mon.a (mon.0) 2380 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-49", "mode": "writeback"}]: dispatch 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:33.201900+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:33.204965+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:33.408223+0000 mon.a (mon.0) 2381 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:33.414802+0000 mon.a (mon.0) 2382 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi162-83985-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi162-83985-64"}]': finished 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:33.414875+0000 mon.a (mon.0) 2383 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-49", "mode": "writeback"}]': finished 2024-02-13T00:45:33.999 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:33.414932+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e378: 8 total, 8 up, 5 in 2024-02-13T00:45:34.000 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:33.452049+0000 mon.b (mon.1) 545 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:45:34.000 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:33.452562+0000 mon.a (mon.0) 2385 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:45:34.000 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:34.017 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:31.619507+0000 osd.0 (osd.0) 124 : cluster [DBG] 28.2 scrub starts 2024-02-13T00:45:34.018 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:31.621162+0000 osd.0 (osd.0) 125 : cluster [DBG] 28.2 scrub ok 2024-02-13T00:45:34.018 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:32.229644+0000 osd.1 (osd.1) 149 : cluster [DBG] 28.1 scrub starts 2024-02-13T00:45:34.018 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:32.233193+0000 osd.1 (osd.1) 150 : cluster [DBG] 28.1 scrub ok 2024-02-13T00:45:34.018 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:32.234450+0000 osd.5 (osd.5) 164 : cluster [DBG] 61.4 scrub starts 2024-02-13T00:45:34.018 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:32.238120+0000 osd.5 (osd.5) 165 : cluster [DBG] 61.4 scrub ok 2024-02-13T00:45:34.018 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:32.408581+0000 mgr.y (mgr.14544) 332 : cluster [DBG] pgmap v535: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:45:34.018 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:32.650893+0000 mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-49"}]': finished 2024-02-13T00:45:34.018 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:32.650971+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e377: 8 total, 8 up, 5 in 2024-02-13T00:45:34.018 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:32.652204+0000 mon.b (mon.1) 544 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-49", "mode": "writeback"}]: dispatch 2024-02-13T00:45:34.018 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:32.656473+0000 mon.a (mon.0) 2380 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-49", "mode": "writeback"}]: dispatch 2024-02-13T00:45:34.018 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:33.201900+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:45:34.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:33.204965+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:45:34.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:33.408223+0000 mon.a (mon.0) 2381 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:45:34.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:33.414802+0000 mon.a (mon.0) 2382 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi162-83985-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi162-83985-64"}]': finished 2024-02-13T00:45:34.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:33.414875+0000 mon.a (mon.0) 2383 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-49", "mode": "writeback"}]': finished 2024-02-13T00:45:34.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:33.414932+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e378: 8 total, 8 up, 5 in 2024-02-13T00:45:34.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:33.452049+0000 mon.b (mon.1) 545 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:45:34.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:33.452562+0000 mon.a (mon.0) 2385 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:45:34.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:31.619507+0000 osd.0 (osd.0) 124 : cluster [DBG] 28.2 scrub starts 2024-02-13T00:45:34.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:31.621162+0000 osd.0 (osd.0) 125 : cluster [DBG] 28.2 scrub ok 2024-02-13T00:45:34.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:32.229644+0000 osd.1 (osd.1) 149 : cluster [DBG] 28.1 scrub starts 2024-02-13T00:45:34.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:32.233193+0000 osd.1 (osd.1) 150 : cluster [DBG] 28.1 scrub ok 2024-02-13T00:45:34.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:32.234450+0000 osd.5 (osd.5) 164 : cluster [DBG] 61.4 scrub starts 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:32.238120+0000 osd.5 (osd.5) 165 : cluster [DBG] 61.4 scrub ok 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:32.408581+0000 mgr.y (mgr.14544) 332 : cluster [DBG] pgmap v535: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:32.650893+0000 mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-49"}]': finished 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:32.650971+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e377: 8 total, 8 up, 5 in 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:32.652204+0000 mon.b (mon.1) 544 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-49", "mode": "writeback"}]: dispatch 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:32.656473+0000 mon.a (mon.0) 2380 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-49", "mode": "writeback"}]: dispatch 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:33.201900+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:33.204965+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:33.408223+0000 mon.a (mon.0) 2381 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:33.414802+0000 mon.a (mon.0) 2382 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi162-83985-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi162-83985-64"}]': finished 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:33.414875+0000 mon.a (mon.0) 2383 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-49", "mode": "writeback"}]': finished 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:33.414932+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e378: 8 total, 8 up, 5 in 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:33.452049+0000 mon.b (mon.1) 545 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:45:34.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:33.452562+0000 mon.a (mon.0) 2385 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:45:34.021 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:34.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:34.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:34.156 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:34.177 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:34.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:34.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:45:34.312 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:34.434 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:34.436 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:45:34.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:34.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:45:34.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:34.568 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:34.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:33.727650+0000 mon.a (mon.0) 2386 : audit [DBG] from='client.? 172.21.15.162:0/917498487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:34.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:34.335024+0000 mon.a (mon.0) 2387 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:34.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:34.416689+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:45:34.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:34.416771+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e379: 8 total, 8 up, 5 in 2024-02-13T00:45:34.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:34.422189+0000 mon.b (mon.1) 546 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:45:34.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:34.425448+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:45:34.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:33.727650+0000 mon.a (mon.0) 2386 : audit [DBG] from='client.? 172.21.15.162:0/917498487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:34.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:34.335024+0000 mon.a (mon.0) 2387 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:34.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:34.416689+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:45:34.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:34.416771+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e379: 8 total, 8 up, 5 in 2024-02-13T00:45:34.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:34.422189+0000 mon.b (mon.1) 546 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:45:34.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:34.425448+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:45:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:33.727650+0000 mon.a (mon.0) 2386 : audit [DBG] from='client.? 172.21.15.162:0/917498487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:34.335024+0000 mon.a (mon.0) 2387 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:34.416689+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:45:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:34.416771+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e379: 8 total, 8 up, 5 in 2024-02-13T00:45:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:34.422189+0000 mon.b (mon.1) 546 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:45:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:34.425448+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:45:35.719 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:35.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:34.221454+0000 osd.1 (osd.1) 151 : cluster [DBG] 61.3 scrub starts 2024-02-13T00:45:35.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:34.225329+0000 osd.1 (osd.1) 152 : cluster [DBG] 61.3 scrub ok 2024-02-13T00:45:35.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:34.408996+0000 mgr.y (mgr.14544) 333 : cluster [DBG] pgmap v538: 246 pgs: 8 unknown, 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:45:35.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:34.520700+0000 osd.0 (osd.0) 126 : cluster [DBG] 28.c scrub starts 2024-02-13T00:45:35.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:34.523590+0000 osd.0 (osd.0) 127 : cluster [DBG] 28.c scrub ok 2024-02-13T00:45:35.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:35.106426+0000 mon.a (mon.0) 2391 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:35.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:35.419598+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:45:35.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:35.419715+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e380: 8 total, 8 up, 5 in 2024-02-13T00:45:35.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:35.421778+0000 mon.a (mon.0) 2394 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:35.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:35.422252+0000 mon.b (mon.1) 547 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:45:35.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:35.424107+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:45:35.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:34.221454+0000 osd.1 (osd.1) 151 : cluster [DBG] 61.3 scrub starts 2024-02-13T00:45:35.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:34.225329+0000 osd.1 (osd.1) 152 : cluster [DBG] 61.3 scrub ok 2024-02-13T00:45:35.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:34.408996+0000 mgr.y (mgr.14544) 333 : cluster [DBG] pgmap v538: 246 pgs: 8 unknown, 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:45:35.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:34.520700+0000 osd.0 (osd.0) 126 : cluster [DBG] 28.c scrub starts 2024-02-13T00:45:35.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:34.523590+0000 osd.0 (osd.0) 127 : cluster [DBG] 28.c scrub ok 2024-02-13T00:45:35.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:35.106426+0000 mon.a (mon.0) 2391 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:35.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:35.419598+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:45:35.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:35.419715+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e380: 8 total, 8 up, 5 in 2024-02-13T00:45:35.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:35.421778+0000 mon.a (mon.0) 2394 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:35.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:35.422252+0000 mon.b (mon.1) 547 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:45:35.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:35.424107+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:45:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:35.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:35.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:35.855 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:45:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:45:35.855 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:34.221454+0000 osd.1 (osd.1) 151 : cluster [DBG] 61.3 scrub starts 2024-02-13T00:45:35.855 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:34.225329+0000 osd.1 (osd.1) 152 : cluster [DBG] 61.3 scrub ok 2024-02-13T00:45:35.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:34.408996+0000 mgr.y (mgr.14544) 333 : cluster [DBG] pgmap v538: 246 pgs: 8 unknown, 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:45:35.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:34.520700+0000 osd.0 (osd.0) 126 : cluster [DBG] 28.c scrub starts 2024-02-13T00:45:35.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:34.523590+0000 osd.0 (osd.0) 127 : cluster [DBG] 28.c scrub ok 2024-02-13T00:45:35.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:35.106426+0000 mon.a (mon.0) 2391 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:35.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:35.419598+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:45:35.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:35.419715+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e380: 8 total, 8 up, 5 in 2024-02-13T00:45:35.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:35.421778+0000 mon.a (mon.0) 2394 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:35.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:35.422252+0000 mon.b (mon.1) 547 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:45:35.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:35.424107+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:45:35.857 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:35.879 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:35.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:35.983 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:36.014 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:36.034 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:36.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:36.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:36.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:36.173 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:36.194 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:36.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:36.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:36.326 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:36.345 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:36.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:36.366 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:45:36] "GET /metrics HTTP/1.1" 200 189883 "" "Prometheus/2.33.4" 2024-02-13T00:45:36.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:36.478 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:36.496 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:36.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:36.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:36.627 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:36.645 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:36.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:36.749 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:36.775 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:35.194134+0000 osd.5 (osd.5) 166 : cluster [DBG] 28.12 scrub starts 2024-02-13T00:45:36.775 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:35.197036+0000 osd.5 (osd.5) 167 : cluster [DBG] 28.12 scrub ok 2024-02-13T00:45:36.776 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:35.577328+0000 mgr.y (mgr.14544) 334 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:36.776 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:36.416174+0000 mon.a (mon.0) 2396 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:45:36.776 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:36.420662+0000 mon.a (mon.0) 2397 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-83985-64"}]': finished 2024-02-13T00:45:36.776 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:36.420736+0000 mon.a (mon.0) 2398 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:45:36.776 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:36.420803+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e381: 8 total, 8 up, 5 in 2024-02-13T00:45:36.776 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:36.422286+0000 mon.a (mon.0) 2400 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:36.776 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:36.422699+0000 mon.b (mon.1) 548 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:45:36.776 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:36.423144+0000 mon.a (mon.0) 2401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:45:36.776 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:36.795 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:36.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:36.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:36.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:36.925 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:36.943 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:36.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:36.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:37.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:35.194134+0000 osd.5 (osd.5) 166 : cluster [DBG] 28.12 scrub starts 2024-02-13T00:45:37.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:35.197036+0000 osd.5 (osd.5) 167 : cluster [DBG] 28.12 scrub ok 2024-02-13T00:45:37.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:35.577328+0000 mgr.y (mgr.14544) 334 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:37.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:36.416174+0000 mon.a (mon.0) 2396 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:45:37.013 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:36.420662+0000 mon.a (mon.0) 2397 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-83985-64"}]': finished 2024-02-13T00:45:37.014 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:36.420736+0000 mon.a (mon.0) 2398 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:45:37.014 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:36.420803+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e381: 8 total, 8 up, 5 in 2024-02-13T00:45:37.014 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:36.422286+0000 mon.a (mon.0) 2400 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:37.014 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:36.422699+0000 mon.b (mon.1) 548 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:45:37.014 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:36.423144+0000 mon.a (mon.0) 2401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:45:37.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:35.194134+0000 osd.5 (osd.5) 166 : cluster [DBG] 28.12 scrub starts 2024-02-13T00:45:37.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:35.197036+0000 osd.5 (osd.5) 167 : cluster [DBG] 28.12 scrub ok 2024-02-13T00:45:37.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:35.577328+0000 mgr.y (mgr.14544) 334 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:37.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:36.416174+0000 mon.a (mon.0) 2396 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:45:37.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:36.420662+0000 mon.a (mon.0) 2397 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-83985-64"}]': finished 2024-02-13T00:45:37.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:36.420736+0000 mon.a (mon.0) 2398 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:45:37.015 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:36.420803+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e381: 8 total, 8 up, 5 in 2024-02-13T00:45:37.015 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:36.422286+0000 mon.a (mon.0) 2400 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-83985-64"}]: dispatch 2024-02-13T00:45:37.015 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:36.422699+0000 mon.b (mon.1) 548 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:45:37.015 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:36.423144+0000 mon.a (mon.0) 2401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:45:37.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-13T00:45:37.047 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:37.073 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:37.091 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:37.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:37.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:37.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:37.221 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:37.239 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:37.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:37.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:37.371 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:37.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:45:37.469 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:37.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:37.582 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:37.608 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:37.626 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:37.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:37.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:37.757 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:37.904 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:37.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:37.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:38.009 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:38.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:36.186344+0000 osd.5 (osd.5) 168 : cluster [DBG] 61.15 scrub starts 2024-02-13T00:45:38.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:36.188461+0000 osd.5 (osd.5) 169 : cluster [DBG] 61.15 scrub ok 2024-02-13T00:45:38.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:36.409395+0000 mgr.y (mgr.14544) 335 : cluster [DBG] pgmap v541: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-13T00:45:38.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:36.514867+0000 osd.0 (osd.0) 128 : cluster [DBG] 61.17 scrub starts 2024-02-13T00:45:38.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:36.518971+0000 osd.0 (osd.0) 129 : cluster [DBG] 61.17 scrub ok 2024-02-13T00:45:38.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:37.277995+0000 osd.6 (osd.6) 85 : cluster [DBG] 28.4 scrub starts 2024-02-13T00:45:38.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:37.280862+0000 osd.6 (osd.6) 86 : cluster [DBG] 28.4 scrub ok 2024-02-13T00:45:38.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:37.424698+0000 mon.a (mon.0) 2402 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-83985-64"}]': finished 2024-02-13T00:45:38.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:37.424778+0000 mon.a (mon.0) 2403 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-13T00:45:38.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:37.424847+0000 mon.a (mon.0) 2404 : cluster [DBG] osdmap e382: 8 total, 8 up, 5 in 2024-02-13T00:45:38.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:37.426434+0000 mon.b (mon.1) 549 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-13T00:45:38.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:37.426863+0000 mon.a (mon.0) 2405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-13T00:45:38.036 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:37.441437+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:38.036 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:37.442230+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:38.036 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:37.442523+0000 mon.a (mon.0) 2408 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi162-83985-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:38.036 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:38.055 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:38.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:38.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:36.186344+0000 osd.5 (osd.5) 168 : cluster [DBG] 61.15 scrub starts 2024-02-13T00:45:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:36.188461+0000 osd.5 (osd.5) 169 : cluster [DBG] 61.15 scrub ok 2024-02-13T00:45:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:36.409395+0000 mgr.y (mgr.14544) 335 : cluster [DBG] pgmap v541: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-13T00:45:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:36.514867+0000 osd.0 (osd.0) 128 : cluster [DBG] 61.17 scrub starts 2024-02-13T00:45:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:36.518971+0000 osd.0 (osd.0) 129 : cluster [DBG] 61.17 scrub ok 2024-02-13T00:45:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:37.277995+0000 osd.6 (osd.6) 85 : cluster [DBG] 28.4 scrub starts 2024-02-13T00:45:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:37.280862+0000 osd.6 (osd.6) 86 : cluster [DBG] 28.4 scrub ok 2024-02-13T00:45:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:37.424698+0000 mon.a (mon.0) 2402 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-83985-64"}]': finished 2024-02-13T00:45:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:37.424778+0000 mon.a (mon.0) 2403 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-13T00:45:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:37.424847+0000 mon.a (mon.0) 2404 : cluster [DBG] osdmap e382: 8 total, 8 up, 5 in 2024-02-13T00:45:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:37.426434+0000 mon.b (mon.1) 549 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-13T00:45:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:37.426863+0000 mon.a (mon.0) 2405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-13T00:45:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:37.441437+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:37.442230+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:37.442523+0000 mon.a (mon.0) 2408 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi162-83985-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:36.186344+0000 osd.5 (osd.5) 168 : cluster [DBG] 61.15 scrub starts 2024-02-13T00:45:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:36.188461+0000 osd.5 (osd.5) 169 : cluster [DBG] 61.15 scrub ok 2024-02-13T00:45:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:36.409395+0000 mgr.y (mgr.14544) 335 : cluster [DBG] pgmap v541: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-13T00:45:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:36.514867+0000 osd.0 (osd.0) 128 : cluster [DBG] 61.17 scrub starts 2024-02-13T00:45:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:36.518971+0000 osd.0 (osd.0) 129 : cluster [DBG] 61.17 scrub ok 2024-02-13T00:45:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:37.277995+0000 osd.6 (osd.6) 85 : cluster [DBG] 28.4 scrub starts 2024-02-13T00:45:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:37.280862+0000 osd.6 (osd.6) 86 : cluster [DBG] 28.4 scrub ok 2024-02-13T00:45:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:37.424698+0000 mon.a (mon.0) 2402 : audit [INF] from='client.? 172.21.15.162:0/2630050469' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-83985-64"}]': finished 2024-02-13T00:45:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:37.424778+0000 mon.a (mon.0) 2403 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-13T00:45:38.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:37.424847+0000 mon.a (mon.0) 2404 : cluster [DBG] osdmap e382: 8 total, 8 up, 5 in 2024-02-13T00:45:38.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:37.426434+0000 mon.b (mon.1) 549 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-13T00:45:38.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:37.426863+0000 mon.a (mon.0) 2405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-13T00:45:38.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:37.441437+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:38.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:37.442230+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:38.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:37.442523+0000 mon.a (mon.0) 2408 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi162-83985-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:45:38.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.136+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/301639661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b280fcb30 msgr2=0x7f6b280fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:38.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.136+0000 7f6b2d1e8700 1 --2- 172.21.15.162:0/301639661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b280fcb30 0x7f6b280fcf10 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7f6b1c0047b0 tx=0x7f6b1c009cb0).stop 2024-02-13T00:45:38.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.136+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/301639661 shutdown_connections 2024-02-13T00:45:38.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.136+0000 7f6b2d1e8700 1 --2- 172.21.15.162:0/301639661 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6b280f73a0 0x7f6b281039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:38.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.136+0000 7f6b2d1e8700 1 --2- 172.21.15.162:0/301639661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b280fcb30 0x7f6b280fcf10 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:38.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.136+0000 7f6b2d1e8700 1 --2- 172.21.15.162:0/301639661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b280fd490 0x7f6b280f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:38.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.136+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/301639661 >> 172.21.15.162:0/301639661 conn(0x7f6b2806eb90 msgr2=0x7f6b280f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:38.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.136+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/301639661 shutdown_connections 2024-02-13T00:45:38.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.136+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/301639661 wait complete. 2024-02-13T00:45:38.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.137+0000 7f6b2d1e8700 1 librados: starting msgr at 2024-02-13T00:45:38.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.137+0000 7f6b2d1e8700 1 librados: starting objecter 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.137+0000 7f6b2d1e8700 1 Processor -- start 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.137+0000 7f6b2d1e8700 1 -- start start 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.137+0000 7f6b2d1e8700 1 librados: setting wanted keys 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.137+0000 7f6b2d1e8700 1 librados: calling monclient init 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.137+0000 7f6b2d1e8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b280f73a0 0x7f6b2810e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2d1e8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b280fcb30 0x7f6b2810e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2d1e8700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6b280fd490 0x7f6b28108310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2d1e8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b28108800 con 0x7f6b280fcb30 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2d1e8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b28108970 con 0x7f6b280f73a0 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2d1e8700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b28108b30 con 0x7f6b280fd490 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b280fcb30 0x7f6b2810e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b280fcb30 0x7f6b2810e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:54142/0 (socket says 172.21.15.162:54142) 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2659c700 1 -- 172.21.15.162:0/1413540504 learned_addr learned my addr 172.21.15.162:0/1413540504 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2759e700 1 --2- 172.21.15.162:0/1413540504 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6b280fd490 0x7f6b28108310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:38.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b26d9d700 1 --2- 172.21.15.162:0/1413540504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b280f73a0 0x7f6b2810e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:38.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2659c700 1 -- 172.21.15.162:0/1413540504 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6b280fd490 msgr2=0x7f6b28108310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:38.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2659c700 1 --2- 172.21.15.162:0/1413540504 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6b280fd490 0x7f6b28108310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:38.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2659c700 1 -- 172.21.15.162:0/1413540504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b280f73a0 msgr2=0x7f6b2810e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:38.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2659c700 1 --2- 172.21.15.162:0/1413540504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b280f73a0 0x7f6b2810e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:38.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2659c700 1 -- 172.21.15.162:0/1413540504 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b200097c0 con 0x7f6b280fcb30 2024-02-13T00:45:38.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2659c700 1 --2- 172.21.15.162:0/1413540504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b280fcb30 0x7f6b2810e660 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 rx=0x7f6b180041d0 tx=0x7f6b18009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:38.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b0ffff700 10 client.?.objecter ms_handle_connect 0x7f6b280fcb30 2024-02-13T00:45:38.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b0ffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:38.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b0ffff700 1 -- 172.21.15.162:0/1413540504 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18007b40 con 0x7f6b280fcb30 2024-02-13T00:45:38.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/1413540504 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1c005040 con 0x7f6b280fcb30 2024-02-13T00:45:38.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b0ffff700 1 -- 172.21.15.162:0/1413540504 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b1800f040 con 0x7f6b280fcb30 2024-02-13T00:45:38.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b0ffff700 1 -- 172.21.15.162:0/1413540504 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18014730 con 0x7f6b280fcb30 2024-02-13T00:45:38.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.138+0000 7f6b2d1e8700 10 client.27249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:38.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.139+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/1413540504 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b28109170 con 0x7f6b280fcb30 2024-02-13T00:45:38.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.139+0000 7f6b2d1e8700 1 librados: init done 2024-02-13T00:45:38.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.140+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/1413540504 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b280475e0 con 0x7f6b280fcb30 2024-02-13T00:45:38.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.140+0000 7f6b0ffff700 1 -- 172.21.15.162:0/1413540504 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6b18014b60 con 0x7f6b280fcb30 2024-02-13T00:45:38.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.140+0000 7f6b0ffff700 1 --2- 172.21.15.162:0/1413540504 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6b10062c90 0x7f6b10065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:38.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.140+0000 7f6b0ffff700 1 -- 172.21.15.162:0/1413540504 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(382..382 src has 52..382) v4 ==== 8072+0+0 (secure 0 0 0) 0x7f6b1807d1a0 con 0x7f6b280fcb30 2024-02-13T00:45:38.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.140+0000 7f6b0ffff700 10 client.27249.objecter ms_dispatch 0x7f6b28001550 osd_map(382..382 src has 52..382) v4 2024-02-13T00:45:38.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.140+0000 7f6b0ffff700 3 client.27249.objecter handle_osd_map got epochs [382,382] > 0 2024-02-13T00:45:38.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.140+0000 7f6b0ffff700 3 client.27249.objecter handle_osd_map decoding full epoch 382 2024-02-13T00:45:38.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.141+0000 7f6b26d9d700 1 --2- 172.21.15.162:0/1413540504 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6b10062c90 0x7f6b10065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:38.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.141+0000 7f6b0ffff700 20 client.27249.objecter dump_active .. 0 homeless 2024-02-13T00:45:38.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.141+0000 7f6b0ffff700 10 client.27249.objecter ms_handle_connect 0x7f6b10062c90 2024-02-13T00:45:38.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.141+0000 7f6b26d9d700 1 --2- 172.21.15.162:0/1413540504 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6b10062c90 0x7f6b10065150 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 rx=0x7f6b1c005010 tx=0x7f6b1c007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:38.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.145+0000 7f6b0ffff700 1 -- 172.21.15.162:0/1413540504 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6b180506a0 con 0x7f6b280fcb30 2024-02-13T00:45:38.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:38.187 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:38.195 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:38.200 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:38.215 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:38.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:38.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:38.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.331+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/1413540504 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b2805fb80 con 0x7f6b10062c90 2024-02-13T00:45:38.349 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:38.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.347+0000 7f6b0ffff700 1 -- 172.21.15.162:0/1413540504 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+588321 (secure 0 0 0) 0x7f6b2805fb80 con 0x7f6b10062c90 2024-02-13T00:45:38.358 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:45:38.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 10 librados: watch_flush enter 2024-02-13T00:45:38.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 10 librados: watch_flush exit 2024-02-13T00:45:38.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 20 client.27249.objecter shutdown clearing up homeless session... 2024-02-13T00:45:38.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 10 client.27249.objecter successfully canceled tick 2024-02-13T00:45:38.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/1413540504 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6b10062c90 msgr2=0x7f6b10065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:38.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 --2- 172.21.15.162:0/1413540504 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6b10062c90 0x7f6b10065150 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 rx=0x7f6b1c005010 tx=0x7f6b1c007f90).stop 2024-02-13T00:45:38.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/1413540504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b280fcb30 msgr2=0x7f6b2810e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:38.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 --2- 172.21.15.162:0/1413540504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b280fcb30 0x7f6b2810e660 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 rx=0x7f6b180041d0 tx=0x7f6b18009f60).stop 2024-02-13T00:45:38.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/1413540504 shutdown_connections 2024-02-13T00:45:38.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 --2- 172.21.15.162:0/1413540504 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6b10062c90 0x7f6b10065150 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:38.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 --2- 172.21.15.162:0/1413540504 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6b280fd490 0x7f6b28108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:38.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 --2- 172.21.15.162:0/1413540504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b280f73a0 0x7f6b2810e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:38.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 --2- 172.21.15.162:0/1413540504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b280fcb30 0x7f6b2810e660 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:38.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/1413540504 >> 172.21.15.162:0/1413540504 conn(0x7f6b2806eb90 msgr2=0x7f6b2806ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:38.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/1413540504 shutdown_connections 2024-02-13T00:45:38.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 -- 172.21.15.162:0/1413540504 wait complete. 2024-02-13T00:45:38.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:38.351+0000 7f6b2d1e8700 1 librados: shutdown 2024-02-13T00:45:38.365 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:45:38.368 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:38.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:38.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:38.498 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:38.516 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:38.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:38.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:38.646 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:45:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:45:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:45:38.648 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:38.668 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:38.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:38.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:38.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:38.798 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:38.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:37.132312+0000 osd.1 (osd.1) 153 : cluster [DBG] 28.1c scrub starts 2024-02-13T00:45:38.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:37.135328+0000 osd.1 (osd.1) 154 : cluster [DBG] 28.1c scrub ok 2024-02-13T00:45:38.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:37.545521+0000 osd.0 (osd.0) 130 : cluster [DBG] 61.14 scrub starts 2024-02-13T00:45:38.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:37.548969+0000 osd.0 (osd.0) 131 : cluster [DBG] 61.14 scrub ok 2024-02-13T00:45:38.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:38.257981+0000 osd.6 (osd.6) 87 : cluster [DBG] 28.13 scrub starts 2024-02-13T00:45:38.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:38.260867+0000 osd.6 (osd.6) 88 : cluster [DBG] 28.13 scrub ok 2024-02-13T00:45:38.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:38.428647+0000 mon.a (mon.0) 2409 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "target_max_objects","val": "1"}]': finished 2024-02-13T00:45:38.802 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:38.428776+0000 mon.a (mon.0) 2410 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi162-83985-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:38.803 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:38.428850+0000 mon.a (mon.0) 2411 : cluster [DBG] osdmap e383: 8 total, 8 up, 5 in 2024-02-13T00:45:38.803 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:38.429889+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi162-83985-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:38.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:37.132312+0000 osd.1 (osd.1) 153 : cluster [DBG] 28.1c scrub starts 2024-02-13T00:45:38.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:37.135328+0000 osd.1 (osd.1) 154 : cluster [DBG] 28.1c scrub ok 2024-02-13T00:45:38.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:37.545521+0000 osd.0 (osd.0) 130 : cluster [DBG] 61.14 scrub starts 2024-02-13T00:45:38.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:37.548969+0000 osd.0 (osd.0) 131 : cluster [DBG] 61.14 scrub ok 2024-02-13T00:45:38.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:38.257981+0000 osd.6 (osd.6) 87 : cluster [DBG] 28.13 scrub starts 2024-02-13T00:45:38.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:38.260867+0000 osd.6 (osd.6) 88 : cluster [DBG] 28.13 scrub ok 2024-02-13T00:45:38.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:38.428647+0000 mon.a (mon.0) 2409 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "target_max_objects","val": "1"}]': finished 2024-02-13T00:45:38.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:38.428776+0000 mon.a (mon.0) 2410 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi162-83985-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:38.803 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:38.428850+0000 mon.a (mon.0) 2411 : cluster [DBG] osdmap e383: 8 total, 8 up, 5 in 2024-02-13T00:45:38.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:38.429889+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi162-83985-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:38.816 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:38.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:38.920 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:38.948 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:37.132312+0000 osd.1 (osd.1) 153 : cluster [DBG] 28.1c scrub starts 2024-02-13T00:45:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:37.135328+0000 osd.1 (osd.1) 154 : cluster [DBG] 28.1c scrub ok 2024-02-13T00:45:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:37.545521+0000 osd.0 (osd.0) 130 : cluster [DBG] 61.14 scrub starts 2024-02-13T00:45:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:37.548969+0000 osd.0 (osd.0) 131 : cluster [DBG] 61.14 scrub ok 2024-02-13T00:45:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:38.257981+0000 osd.6 (osd.6) 87 : cluster [DBG] 28.13 scrub starts 2024-02-13T00:45:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:38.260867+0000 osd.6 (osd.6) 88 : cluster [DBG] 28.13 scrub ok 2024-02-13T00:45:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:38.428647+0000 mon.a (mon.0) 2409 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-49","var": "target_max_objects","val": "1"}]': finished 2024-02-13T00:45:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:38.428776+0000 mon.a (mon.0) 2410 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi162-83985-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:45:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:38.428850+0000 mon.a (mon.0) 2411 : cluster [DBG] osdmap e383: 8 total, 8 up, 5 in 2024-02-13T00:45:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:38.429889+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi162-83985-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:39.096 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:39.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:39.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:39.200 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:39.228 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:39.676 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:39.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:39.694 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":543,"stamp":"2024-02-13T00:45:37.422056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007253,"num_objects":377,"num_object_clones":2,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":375,"num_whiteouts":0,"num_read":1642,"num_read_kb":1266,"num_write":1202,"num_write_kb":4522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1455,"num_bytes_recovered":15268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2617,"ondisk_log_size":2617,"up":714,"acting":714,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":724,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":1562424,"kb_used_data":43484,"kb_used_omap":0,"kb_used_meta":1518848,"kb_avail":467163336,"statfs":{"total":479975178240,"available":478375256064,"internally_reserved":0,"allocated":44527616,"data_stored":29702185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1555300352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":64,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008160"},"pg_stats":[{"pgid":"204.1f","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423403+0000","last_change":"2024-02-13T00:45:30.411541+0000","last_active":"2024-02-13T00:45:33.423403+0000","last_peered":"2024-02-13T00:45:33.423403+0000","last_clean":"2024-02-13T00:45:33.423403+0000","last_became_active":"2024-02-13T00:45:30.411186+0000","last_became_peered":"2024-02-13T00:45:30.411186+0000","last_unstale":"2024-02-13T00:45:33.423403+0000","last_undegraded":"2024-02-13T00:45:33.423403+0000","last_fullsized":"2024-02-13T00:45:33.423403+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.1e","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422080+0000","last_change":"2024-02-13T00:45:30.426649+0000","last_active":"2024-02-13T00:45:33.422080+0000","last_peered":"2024-02-13T00:45:33.422080+0000","last_clean":"2024-02-13T00:45:33.422080+0000","last_became_active":"2024-02-13T00:45:30.426310+0000","last_became_peered":"2024-02-13T00:45:30.426310+0000","last_unstale":"2024-02-13T00:45:33.422080+0000","last_undegraded":"2024-02-13T00:45:33.422080+0000","last_fullsized":"2024-02-13T00:45:33.422080+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.1d","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422204+0000","last_change":"2024-02-13T00:45:30.416648+0000","last_active":"2024-02-13T00:45:33.422204+0000","last_peered":"2024-02-13T00:45:33.422204+0000","last_clean":"2024-02-13T00:45:33.422204+0000","last_became_active":"2024-02-13T00:45:30.416347+0000","last_became_peered":"2024-02-13T00:45:30.416347+0000","last_unstale":"2024-02-13T00:45:33.422204+0000","last_undegraded":"2024-02-13T00:45:33.422204+0000","last_fullsized":"2024-02-13T00:45:33.422204+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.1c","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421585+0000","last_change":"2024-02-13T00:45:30.409140+0000","last_active":"2024-02-13T00:45:33.421585+0000","last_peered":"2024-02-13T00:45:33.421585+0000","last_clean":"2024-02-13T00:45:33.421585+0000","last_became_active":"2024-02-13T00:45:30.408777+0000","last_became_peered":"2024-02-13T00:45:30.408777+0000","last_unstale":"2024-02-13T00:45:33.421585+0000","last_undegraded":"2024-02-13T00:45:33.421585+0000","last_fullsized":"2024-02-13T00:45:33.421585+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.1b","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417187+0000","last_change":"2024-02-13T00:45:30.403684+0000","last_active":"2024-02-13T00:45:33.417187+0000","last_peered":"2024-02-13T00:45:33.417187+0000","last_clean":"2024-02-13T00:45:33.417187+0000","last_became_active":"2024-02-13T00:45:30.403390+0000","last_became_peered":"2024-02-13T00:45:30.403390+0000","last_unstale":"2024-02-13T00:45:33.417187+0000","last_undegraded":"2024-02-13T00:45:33.417187+0000","last_fullsized":"2024-02-13T00:45:33.417187+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"204.1a","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419458+0000","last_change":"2024-02-13T00:45:30.431644+0000","last_active":"2024-02-13T00:45:33.419458+0000","last_peered":"2024-02-13T00:45:33.419458+0000","last_clean":"2024-02-13T00:45:33.419458+0000","last_became_active":"2024-02-13T00:45:30.431336+0000","last_became_peered":"2024-02-13T00:45:30.431336+0000","last_unstale":"2024-02-13T00:45:33.419458+0000","last_undegraded":"2024-02-13T00:45:33.419458+0000","last_fullsized":"2024-02-13T00:45:33.419458+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.19","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418238+0000","last_change":"2024-02-13T00:45:30.431649+0000","last_active":"2024-02-13T00:45:33.418238+0000","last_peered":"2024-02-13T00:45:33.418238+0000","last_clean":"2024-02-13T00:45:33.418238+0000","last_became_active":"2024-02-13T00:45:30.431333+0000","last_became_peered":"2024-02-13T00:45:30.431333+0000","last_unstale":"2024-02-13T00:45:33.418238+0000","last_undegraded":"2024-02-13T00:45:33.418238+0000","last_fullsized":"2024-02-13T00:45:33.418238+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.18","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.415271+0000","last_change":"2024-02-13T00:45:30.407838+0000","last_active":"2024-02-13T00:45:33.415271+0000","last_peered":"2024-02-13T00:45:33.415271+0000","last_clean":"2024-02-13T00:45:33.415271+0000","last_became_active":"2024-02-13T00:45:30.407543+0000","last_became_peered":"2024-02-13T00:45:30.407543+0000","last_unstale":"2024-02-13T00:45:33.415271+0000","last_undegraded":"2024-02-13T00:45:33.415271+0000","last_fullsized":"2024-02-13T00:45:33.415271+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.17","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424230+0000","last_change":"2024-02-13T00:45:30.409703+0000","last_active":"2024-02-13T00:45:33.424230+0000","last_peered":"2024-02-13T00:45:33.424230+0000","last_clean":"2024-02-13T00:45:33.424230+0000","last_became_active":"2024-02-13T00:45:30.409398+0000","last_became_peered":"2024-02-13T00:45:30.409398+0000","last_unstale":"2024-02-13T00:45:33.424230+0000","last_undegraded":"2024-02-13T00:45:33.424230+0000","last_fullsized":"2024-02-13T00:45:33.424230+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.16","version":"378'2","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.451701+0000","last_change":"2024-02-13T00:45:30.412114+0000","last_active":"2024-02-13T00:45:33.451701+0000","last_peered":"2024-02-13T00:45:33.451701+0000","last_clean":"2024-02-13T00:45:33.451701+0000","last_became_active":"2024-02-13T00:45:30.411747+0000","last_became_peered":"2024-02-13T00:45:30.411747+0000","last_unstale":"2024-02-13T00:45:33.451701+0000","last_undegraded":"2024-02-13T00:45:33.451701+0000","last_fullsized":"2024-02-13T00:45:33.451701+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":1,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.15","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422292+0000","last_change":"2024-02-13T00:45:30.415232+0000","last_active":"2024-02-13T00:45:33.422292+0000","last_peered":"2024-02-13T00:45:33.422292+0000","last_clean":"2024-02-13T00:45:33.422292+0000","last_became_active":"2024-02-13T00:45:30.414904+0000","last_became_peered":"2024-02-13T00:45:30.414904+0000","last_unstale":"2024-02-13T00:45:33.422292+0000","last_undegraded":"2024-02-13T00:45:33.422292+0000","last_fullsized":"2024-02-13T00:45:33.422292+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.14","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417369+0000","last_change":"2024-02-13T00:45:30.431800+0000","last_active":"2024-02-13T00:45:33.417369+0000","last_peered":"2024-02-13T00:45:33.417369+0000","last_clean":"2024-02-13T00:45:33.417369+0000","last_became_active":"2024-02-13T00:45:30.431392+0000","last_became_peered":"2024-02-13T00:45:30.431392+0000","last_unstale":"2024-02-13T00:45:33.417369+0000","last_undegraded":"2024-02-13T00:45:33.417369+0000","last_fullsized":"2024-02-13T00:45:33.417369+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"204.13","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421690+0000","last_change":"2024-02-13T00:45:30.413450+0000","last_active":"2024-02-13T00:45:33.421690+0000","last_peered":"2024-02-13T00:45:33.421690+0000","last_clean":"2024-02-13T00:45:33.421690+0000","last_became_active":"2024-02-13T00:45:30.413198+0000","last_became_peered":"2024-02-13T00:45:30.413198+0000","last_unstale":"2024-02-13T00:45:33.421690+0000","last_undegraded":"2024-02-13T00:45:33.421690+0000","last_fullsized":"2024-02-13T00:45:33.421690+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.12","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421445+0000","last_change":"2024-02-13T00:45:30.412598+0000","last_active":"2024-02-13T00:45:33.421445+0000","last_peered":"2024-02-13T00:45:33.421445+0000","last_clean":"2024-02-13T00:45:33.421445+0000","last_became_active":"2024-02-13T00:45:30.412320+0000","last_became_peered":"2024-02-13T00:45:30.412320+0000","last_unstale":"2024-02-13T00:45:33.421445+0000","last_undegraded":"2024-02-13T00:45:33.421445+0000","last_fullsized":"2024-02-13T00:45:33.421445+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.11","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419106+0000","last_change":"2024-02-13T00:45:30.426653+0000","last_active":"2024-02-13T00:45:33.419106+0000","last_peered":"2024-02-13T00:45:33.419106+0000","last_clean":"2024-02-13T00:45:33.419106+0000","last_became_active":"2024-02-13T00:45:30.426338+0000","last_became_peered":"2024-02-13T00:45:30.426338+0000","last_unstale":"2024-02-13T00:45:33.419106+0000","last_undegraded":"2024-02-13T00:45:33.419106+0000","last_fullsized":"2024-02-13T00:45:33.419106+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.10","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.415866+0000","last_change":"2024-02-13T00:45:30.428445+0000","last_active":"2024-02-13T00:45:33.415866+0000","last_peered":"2024-02-13T00:45:33.415866+0000","last_clean":"2024-02-13T00:45:33.415866+0000","last_became_active":"2024-02-13T00:45:30.428262+0000","last_became_peered":"2024-02-13T00:45:30.428262+0000","last_unstale":"2024-02-13T00:45:33.415866+0000","last_undegraded":"2024-02-13T00:45:33.415866+0000","last_fullsized":"2024-02-13T00:45:33.415866+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"204.f","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422161+0000","last_change":"2024-02-13T00:45:30.429094+0000","last_active":"2024-02-13T00:45:33.422161+0000","last_peered":"2024-02-13T00:45:33.422161+0000","last_clean":"2024-02-13T00:45:33.422161+0000","last_became_active":"2024-02-13T00:45:30.428791+0000","last_became_peered":"2024-02-13T00:45:30.428791+0000","last_unstale":"2024-02-13T00:45:33.422161+0000","last_undegraded":"2024-02-13T00:45:33.422161+0000","last_fullsized":"2024-02-13T00:45:33.422161+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.e","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423260+0000","last_change":"2024-02-13T00:45:30.427271+0000","last_active":"2024-02-13T00:45:33.423260+0000","last_peered":"2024-02-13T00:45:33.423260+0000","last_clean":"2024-02-13T00:45:33.423260+0000","last_became_active":"2024-02-13T00:45:30.426938+0000","last_became_peered":"2024-02-13T00:45:30.426938+0000","last_unstale":"2024-02-13T00:45:33.423260+0000","last_undegraded":"2024-02-13T00:45:33.423260+0000","last_fullsized":"2024-02-13T00:45:33.423260+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.d","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423323+0000","last_change":"2024-02-13T00:45:30.408864+0000","last_active":"2024-02-13T00:45:33.423323+0000","last_peered":"2024-02-13T00:45:33.423323+0000","last_clean":"2024-02-13T00:45:33.423323+0000","last_became_active":"2024-02-13T00:45:30.408185+0000","last_became_peered":"2024-02-13T00:45:30.408185+0000","last_unstale":"2024-02-13T00:45:33.423323+0000","last_undegraded":"2024-02-13T00:45:33.423323+0000","last_fullsized":"2024-02-13T00:45:33.423323+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.c","version":"378'1","reported_seq":15,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.439145+0000","last_change":"2024-02-13T00:45:30.407987+0000","last_active":"2024-02-13T00:45:33.439145+0000","last_peered":"2024-02-13T00:45:33.439145+0000","last_clean":"2024-02-13T00:45:33.439145+0000","last_became_active":"2024-02-13T00:45:30.407658+0000","last_became_peered":"2024-02-13T00:45:30.407658+0000","last_unstale":"2024-02-13T00:45:33.439145+0000","last_undegraded":"2024-02-13T00:45:33.439145+0000","last_fullsized":"2024-02-13T00:45:33.439145+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.b","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417264+0000","last_change":"2024-02-13T00:45:30.426548+0000","last_active":"2024-02-13T00:45:33.417264+0000","last_peered":"2024-02-13T00:45:33.417264+0000","last_clean":"2024-02-13T00:45:33.417264+0000","last_became_active":"2024-02-13T00:45:30.426225+0000","last_became_peered":"2024-02-13T00:45:30.426225+0000","last_unstale":"2024-02-13T00:45:33.417264+0000","last_undegraded":"2024-02-13T00:45:33.417264+0000","last_fullsized":"2024-02-13T00:45:33.417264+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"204.a","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421561+0000","last_change":"2024-02-13T00:45:30.426500+0000","last_active":"2024-02-13T00:45:33.421561+0000","last_peered":"2024-02-13T00:45:33.421561+0000","last_clean":"2024-02-13T00:45:33.421561+0000","last_became_active":"2024-02-13T00:45:30.426183+0000","last_became_peered":"2024-02-13T00:45:30.426183+0000","last_unstale":"2024-02-13T00:45:33.421561+0000","last_undegraded":"2024-02-13T00:45:33.421561+0000","last_fullsized":"2024-02-13T00:45:33.421561+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.9","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421522+0000","last_change":"2024-02-13T00:45:30.427685+0000","last_active":"2024-02-13T00:45:33.421522+0000","last_peered":"2024-02-13T00:45:33.421522+0000","last_clean":"2024-02-13T00:45:33.421522+0000","last_became_active":"2024-02-13T00:45:30.427353+0000","last_became_peered":"2024-02-13T00:45:30.427353+0000","last_unstale":"2024-02-13T00:45:33.421522+0000","last_undegraded":"2024-02-13T00:45:33.421522+0000","last_fullsized":"2024-02-13T00:45:33.421522+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.8","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.415855+0000","last_change":"2024-02-13T00:45:30.400891+0000","last_active":"2024-02-13T00:45:33.415855+0000","last_peered":"2024-02-13T00:45:33.415855+0000","last_clean":"2024-02-13T00:45:33.415855+0000","last_became_active":"2024-02-13T00:45:30.400527+0000","last_became_peered":"2024-02-13T00:45:30.400527+0000","last_unstale":"2024-02-13T00:45:33.415855+0000","last_undegraded":"2024-02-13T00:45:33.415855+0000","last_fullsized":"2024-02-13T00:45:33.415855+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"204.7","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423394+0000","last_change":"2024-02-13T00:45:30.427332+0000","last_active":"2024-02-13T00:45:33.423394+0000","last_peered":"2024-02-13T00:45:33.423394+0000","last_clean":"2024-02-13T00:45:33.423394+0000","last_became_active":"2024-02-13T00:45:30.427040+0000","last_became_peered":"2024-02-13T00:45:30.427040+0000","last_unstale":"2024-02-13T00:45:33.423394+0000","last_undegraded":"2024-02-13T00:45:33.423394+0000","last_fullsized":"2024-02-13T00:45:33.423394+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"204.6","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423659+0000","last_change":"2024-02-13T00:45:30.431715+0000","last_active":"2024-02-13T00:45:33.423659+0000","last_peered":"2024-02-13T00:45:33.423659+0000","last_clean":"2024-02-13T00:45:33.423659+0000","last_became_active":"2024-02-13T00:45:30.431374+0000","last_became_peered":"2024-02-13T00:45:30.431374+0000","last_unstale":"2024-02-13T00:45:33.423659+0000","last_undegraded":"2024-02-13T00:45:33.423659+0000","last_fullsized":"2024-02-13T00:45:33.423659+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.5","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422318+0000","last_change":"2024-02-13T00:45:30.426690+0000","last_active":"2024-02-13T00:45:33.422318+0000","last_peered":"2024-02-13T00:45:33.422318+0000","last_clean":"2024-02-13T00:45:33.422318+0000","last_became_active":"2024-02-13T00:45:30.426209+0000","last_became_peered":"2024-02-13T00:45:30.426209+0000","last_unstale":"2024-02-13T00:45:33.422318+0000","last_undegraded":"2024-02-13T00:45:33.422318+0000","last_fullsized":"2024-02-13T00:45:33.422318+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.4","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421796+0000","last_change":"2024-02-13T00:45:30.409163+0000","last_active":"2024-02-13T00:45:33.421796+0000","last_peered":"2024-02-13T00:45:33.421796+0000","last_clean":"2024-02-13T00:45:33.421796+0000","last_became_active":"2024-02-13T00:45:30.408788+0000","last_became_peered":"2024-02-13T00:45:30.408788+0000","last_unstale":"2024-02-13T00:45:33.421796+0000","last_undegraded":"2024-02-13T00:45:33.421796+0000","last_fullsized":"2024-02-13T00:45:33.421796+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.3","version":"378'1","reported_seq":15,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.431937+0000","last_change":"2024-02-13T00:45:30.432277+0000","last_active":"2024-02-13T00:45:33.431937+0000","last_peered":"2024-02-13T00:45:33.431937+0000","last_clean":"2024-02-13T00:45:33.431937+0000","last_became_active":"2024-02-13T00:45:30.431555+0000","last_became_peered":"2024-02-13T00:45:30.431555+0000","last_unstale":"2024-02-13T00:45:33.431937+0000","last_undegraded":"2024-02-13T00:45:33.431937+0000","last_fullsized":"2024-02-13T00:45:33.431937+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.2","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.420319+0000","last_change":"2024-02-13T00:45:30.411194+0000","last_active":"2024-02-13T00:45:33.420319+0000","last_peered":"2024-02-13T00:45:33.420319+0000","last_clean":"2024-02-13T00:45:33.420319+0000","last_became_active":"2024-02-13T00:45:30.410898+0000","last_became_peered":"2024-02-13T00:45:30.410898+0000","last_unstale":"2024-02-13T00:45:33.420319+0000","last_undegraded":"2024-02-13T00:45:33.420319+0000","last_fullsized":"2024-02-13T00:45:33.420319+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"204.1","version":"378'2","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.450710+0000","last_change":"2024-02-13T00:45:30.431874+0000","last_active":"2024-02-13T00:45:33.450710+0000","last_peered":"2024-02-13T00:45:33.450710+0000","last_clean":"2024-02-13T00:45:33.450710+0000","last_became_active":"2024-02-13T00:45:30.431237+0000","last_became_peered":"2024-02-13T00:45:30.431237+0000","last_unstale":"2024-02-13T00:45:33.450710+0000","last_undegraded":"2024-02-13T00:45:33.450710+0000","last_fullsized":"2024-02-13T00:45:33.450710+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":1,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"204.0","version":"0'0","reported_seq":13,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418132+0000","last_change":"2024-02-13T00:45:30.424845+0000","last_active":"2024-02-13T00:45:33.418132+0000","last_peered":"2024-02-13T00:45:33.418132+0000","last_clean":"2024-02-13T00:45:33.418132+0000","last_became_active":"2024-02-13T00:45:30.424532+0000","last_became_peered":"2024-02-13T00:45:30.424532+0000","last_unstale":"2024-02-13T00:45:33.418132+0000","last_undegraded":"2024-02-13T00:45:33.418132+0000","last_fullsized":"2024-02-13T00:45:33.418132+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:29.445853+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.445853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.2","version":"204'34","reported_seq":118,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421151+0000","last_change":"2024-02-13T00:45:20.814021+0000","last_active":"2024-02-13T00:45:33.421151+0000","last_peered":"2024-02-13T00:45:33.421151+0000","last_clean":"2024-02-13T00:45:33.421151+0000","last_became_active":"2024-02-13T00:45:15.469717+0000","last_became_peered":"2024-02-13T00:45:15.469717+0000","last_unstale":"2024-02-13T00:45:33.421151+0000","last_undegraded":"2024-02-13T00:45:33.421151+0000","last_fullsized":"2024-02-13T00:45:33.421151+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.3","version":"375'104","reported_seq":560,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:34.225455+0000","last_change":"2024-02-13T00:45:34.225416+0000","last_active":"2024-02-13T00:45:34.225455+0000","last_peered":"2024-02-13T00:45:34.225455+0000","last_clean":"2024-02-13T00:45:34.225455+0000","last_became_active":"2024-02-13T00:45:22.426662+0000","last_became_peered":"2024-02-13T00:45:22.426662+0000","last_unstale":"2024-02-13T00:45:34.225455+0000","last_undegraded":"2024-02-13T00:45:34.225455+0000","last_fullsized":"2024-02-13T00:45:34.225455+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'104","last_scrub_stamp":"2024-02-13T00:45:34.225334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.225334+0000","log_size":104,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.0","version":"300'35","reported_seq":340,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416151+0000","last_change":"2024-02-13T00:43:59.387105+0000","last_active":"2024-02-13T00:45:33.416151+0000","last_peered":"2024-02-13T00:45:33.416151+0000","last_clean":"2024-02-13T00:45:33.416151+0000","last_became_active":"2024-02-13T00:41:16.768067+0000","last_became_peered":"2024-02-13T00:41:16.768067+0000","last_unstale":"2024-02-13T00:45:33.416151+0000","last_undegraded":"2024-02-13T00:45:33.416151+0000","last_fullsized":"2024-02-13T00:45:33.416151+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000030,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.1","version":"375'35","reported_seq":374,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423112+0000","last_change":"2024-02-13T00:45:29.241690+0000","last_active":"2024-02-13T00:45:33.423112+0000","last_peered":"2024-02-13T00:45:33.423112+0000","last_clean":"2024-02-13T00:45:33.423112+0000","last_became_active":"2024-02-13T00:45:15.420934+0000","last_became_peered":"2024-02-13T00:45:15.420934+0000","last_unstale":"2024-02-13T00:45:33.423112+0000","last_undegraded":"2024-02-13T00:45:33.423112+0000","last_fullsized":"2024-02-13T00:45:33.423112+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'34","last_scrub_stamp":"2024-02-13T00:45:29.241606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.241606+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.6","version":"263'18","reported_seq":271,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424168+0000","last_change":"2024-02-13T00:44:50.590909+0000","last_active":"2024-02-13T00:45:33.424168+0000","last_peered":"2024-02-13T00:45:33.424168+0000","last_clean":"2024-02-13T00:45:33.424168+0000","last_became_active":"2024-02-13T00:44:50.423503+0000","last_became_peered":"2024-02-13T00:44:50.423503+0000","last_unstale":"2024-02-13T00:45:33.424168+0000","last_undegraded":"2024-02-13T00:45:33.424168+0000","last_fullsized":"2024-02-13T00:45:33.424168+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.7","version":"270'85","reported_seq":452,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424344+0000","last_change":"2024-02-13T00:45:17.447744+0000","last_active":"2024-02-13T00:45:33.424344+0000","last_peered":"2024-02-13T00:45:33.424344+0000","last_clean":"2024-02-13T00:45:33.424344+0000","last_became_active":"2024-02-13T00:45:15.458968+0000","last_became_peered":"2024-02-13T00:45:15.458968+0000","last_unstale":"2024-02-13T00:45:33.424344+0000","last_undegraded":"2024-02-13T00:45:33.424344+0000","last_fullsized":"2024-02-13T00:45:33.424344+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'85","last_scrub_stamp":"2024-02-13T00:45:02.230151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:02.230151+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.4","version":"343'57","reported_seq":143,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422168+0000","last_change":"2024-02-13T00:45:32.238236+0000","last_active":"2024-02-13T00:45:33.422168+0000","last_peered":"2024-02-13T00:45:33.422168+0000","last_clean":"2024-02-13T00:45:33.422168+0000","last_became_active":"2024-02-13T00:45:15.457755+0000","last_became_peered":"2024-02-13T00:45:15.457755+0000","last_unstale":"2024-02-13T00:45:33.422168+0000","last_undegraded":"2024-02-13T00:45:33.422168+0000","last_fullsized":"2024-02-13T00:45:33.422168+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'57","last_scrub_stamp":"2024-02-13T00:45:32.238130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.238130+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":84,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.5","version":"224'56","reported_seq":448,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423302+0000","last_change":"2024-02-13T00:44:44.534359+0000","last_active":"2024-02-13T00:45:33.423302+0000","last_peered":"2024-02-13T00:45:33.423302+0000","last_clean":"2024-02-13T00:45:33.423302+0000","last_became_active":"2024-02-13T00:44:39.504316+0000","last_became_peered":"2024-02-13T00:44:39.504316+0000","last_unstale":"2024-02-13T00:45:33.423302+0000","last_undegraded":"2024-02-13T00:45:33.423302+0000","last_fullsized":"2024-02-13T00:45:33.423302+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":189,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.a","version":"343'92","reported_seq":427,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419753+0000","last_change":"2024-02-13T00:43:59.375752+0000","last_active":"2024-02-13T00:45:33.419753+0000","last_peered":"2024-02-13T00:45:33.419753+0000","last_clean":"2024-02-13T00:45:33.419753+0000","last_became_active":"2024-02-13T00:43:35.450504+0000","last_became_peered":"2024-02-13T00:43:35.450504+0000","last_unstale":"2024-02-13T00:45:33.419753+0000","last_undegraded":"2024-02-13T00:45:33.419753+0000","last_fullsized":"2024-02-13T00:45:33.419753+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":92,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":60,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.b","version":"222'17","reported_seq":384,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422447+0000","last_change":"2024-02-13T00:44:53.182287+0000","last_active":"2024-02-13T00:45:33.422447+0000","last_peered":"2024-02-13T00:45:33.422447+0000","last_clean":"2024-02-13T00:45:33.422447+0000","last_became_active":"2024-02-13T00:44:50.433096+0000","last_became_peered":"2024-02-13T00:44:50.433096+0000","last_unstale":"2024-02-13T00:45:33.422447+0000","last_undegraded":"2024-02-13T00:45:33.422447+0000","last_fullsized":"2024-02-13T00:45:33.422447+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":74,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.8","version":"343'54","reported_seq":457,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418479+0000","last_change":"2024-02-13T00:45:19.168513+0000","last_active":"2024-02-13T00:45:33.418479+0000","last_peered":"2024-02-13T00:45:33.418479+0000","last_clean":"2024-02-13T00:45:33.418479+0000","last_became_active":"2024-02-13T00:45:15.462298+0000","last_became_peered":"2024-02-13T00:45:15.462298+0000","last_unstale":"2024-02-13T00:45:33.418479+0000","last_undegraded":"2024-02-13T00:45:33.418479+0000","last_fullsized":"2024-02-13T00:45:33.418479+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'53","last_scrub_stamp":"2024-02-13T00:44:30.046222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:30.046222+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.9","version":"343'36","reported_seq":374,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421624+0000","last_change":"2024-02-13T00:45:17.411322+0000","last_active":"2024-02-13T00:45:33.421624+0000","last_peered":"2024-02-13T00:45:33.421624+0000","last_clean":"2024-02-13T00:45:33.421624+0000","last_became_active":"2024-02-13T00:45:15.421026+0000","last_became_peered":"2024-02-13T00:45:15.421026+0000","last_unstale":"2024-02-13T00:45:33.421624+0000","last_undegraded":"2024-02-13T00:45:33.421624+0000","last_fullsized":"2024-02-13T00:45:33.421624+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.e","version":"272'107","reported_seq":378,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422524+0000","last_change":"2024-02-13T00:44:46.388608+0000","last_active":"2024-02-13T00:45:33.422524+0000","last_peered":"2024-02-13T00:45:33.422524+0000","last_clean":"2024-02-13T00:45:33.422524+0000","last_became_active":"2024-02-13T00:44:46.387990+0000","last_became_peered":"2024-02-13T00:44:46.387990+0000","last_unstale":"2024-02-13T00:45:33.422524+0000","last_undegraded":"2024-02-13T00:45:33.422524+0000","last_fullsized":"2024-02-13T00:45:33.422524+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":107,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.f","version":"252'51","reported_seq":406,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424439+0000","last_change":"2024-02-13T00:45:26.215878+0000","last_active":"2024-02-13T00:45:33.424439+0000","last_peered":"2024-02-13T00:45:33.424439+0000","last_clean":"2024-02-13T00:45:33.424439+0000","last_became_active":"2024-02-13T00:45:15.466657+0000","last_became_peered":"2024-02-13T00:45:15.466657+0000","last_unstale":"2024-02-13T00:45:33.424439+0000","last_undegraded":"2024-02-13T00:45:33.424439+0000","last_fullsized":"2024-02-13T00:45:33.424439+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'51","last_scrub_stamp":"2024-02-13T00:45:26.215792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.215792+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.c","version":"375'106","reported_seq":564,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.435571+0000","last_change":"2024-02-13T00:45:31.255260+0000","last_active":"2024-02-13T00:45:33.435571+0000","last_peered":"2024-02-13T00:45:33.435571+0000","last_clean":"2024-02-13T00:45:33.435571+0000","last_became_active":"2024-02-13T00:44:45.385325+0000","last_became_peered":"2024-02-13T00:44:45.385325+0000","last_unstale":"2024-02-13T00:45:33.435571+0000","last_undegraded":"2024-02-13T00:45:33.435571+0000","last_fullsized":"2024-02-13T00:45:33.435571+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'106","last_scrub_stamp":"2024-02-13T00:45:31.255154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.255154+0000","log_size":106,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.d","version":"343'91","reported_seq":462,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423588+0000","last_change":"2024-02-13T00:44:51.014582+0000","last_active":"2024-02-13T00:45:33.423588+0000","last_peered":"2024-02-13T00:45:33.423588+0000","last_clean":"2024-02-13T00:45:33.423588+0000","last_became_active":"2024-02-13T00:44:50.431715+0000","last_became_peered":"2024-02-13T00:44:50.431715+0000","last_unstale":"2024-02-13T00:45:33.423588+0000","last_undegraded":"2024-02-13T00:45:33.423588+0000","last_fullsized":"2024-02-13T00:45:33.423588+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'90","last_scrub_stamp":"2024-02-13T00:44:31.080244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:31.080244+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":68,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":274,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416619+0000","last_change":"2024-02-13T00:45:15.442032+0000","last_active":"2024-02-13T00:45:33.416619+0000","last_peered":"2024-02-13T00:45:33.416619+0000","last_clean":"2024-02-13T00:45:33.416619+0000","last_became_active":"2024-02-13T00:45:15.441786+0000","last_became_peered":"2024-02-13T00:45:15.441786+0000","last_unstale":"2024-02-13T00:45:33.416619+0000","last_undegraded":"2024-02-13T00:45:33.416619+0000","last_fullsized":"2024-02-13T00:45:33.416619+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.12","version":"343'24","reported_seq":298,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417380+0000","last_change":"2024-02-13T00:45:15.490595+0000","last_active":"2024-02-13T00:45:33.417380+0000","last_peered":"2024-02-13T00:45:33.417380+0000","last_clean":"2024-02-13T00:45:33.417380+0000","last_became_active":"2024-02-13T00:45:15.441163+0000","last_became_peered":"2024-02-13T00:45:15.441163+0000","last_unstale":"2024-02-13T00:45:33.417380+0000","last_undegraded":"2024-02-13T00:45:33.417380+0000","last_fullsized":"2024-02-13T00:45:33.417380+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'22","last_scrub_stamp":"2024-02-13T00:44:34.629429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.629429+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":200,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416184+0000","last_change":"2024-02-13T00:45:15.468485+0000","last_active":"2024-02-13T00:45:33.416184+0000","last_peered":"2024-02-13T00:45:33.416184+0000","last_clean":"2024-02-13T00:45:33.416184+0000","last_became_active":"2024-02-13T00:45:15.467997+0000","last_became_peered":"2024-02-13T00:45:15.467997+0000","last_unstale":"2024-02-13T00:45:33.416184+0000","last_undegraded":"2024-02-13T00:45:33.416184+0000","last_fullsized":"2024-02-13T00:45:33.416184+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.13","version":"236'73","reported_seq":166,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421800+0000","last_change":"2024-02-13T00:45:20.245873+0000","last_active":"2024-02-13T00:45:33.421800+0000","last_peered":"2024-02-13T00:45:33.421800+0000","last_clean":"2024-02-13T00:45:33.421800+0000","last_became_active":"2024-02-13T00:45:15.447922+0000","last_became_peered":"2024-02-13T00:45:15.447922+0000","last_unstale":"2024-02-13T00:45:33.421800+0000","last_undegraded":"2024-02-13T00:45:33.421800+0000","last_fullsized":"2024-02-13T00:45:33.421800+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.10","version":"274'73","reported_seq":484,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418550+0000","last_change":"2024-02-13T00:45:27.173400+0000","last_active":"2024-02-13T00:45:33.418550+0000","last_peered":"2024-02-13T00:45:33.418550+0000","last_clean":"2024-02-13T00:45:33.418550+0000","last_became_active":"2024-02-13T00:45:15.470892+0000","last_became_peered":"2024-02-13T00:45:15.470892+0000","last_unstale":"2024-02-13T00:45:33.418550+0000","last_undegraded":"2024-02-13T00:45:33.418550+0000","last_fullsized":"2024-02-13T00:45:33.418550+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'73","last_scrub_stamp":"2024-02-13T00:45:27.173322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:27.173322+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":98,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.11","version":"343'37","reported_seq":253,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421608+0000","last_change":"2024-02-13T00:44:55.941713+0000","last_active":"2024-02-13T00:45:33.421608+0000","last_peered":"2024-02-13T00:45:33.421608+0000","last_clean":"2024-02-13T00:45:33.421608+0000","last_became_active":"2024-02-13T00:44:50.443606+0000","last_became_peered":"2024-02-13T00:44:50.443606+0000","last_unstale":"2024-02-13T00:45:33.421608+0000","last_undegraded":"2024-02-13T00:45:33.421608+0000","last_fullsized":"2024-02-13T00:45:33.421608+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.16","version":"375'75","reported_seq":429,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.433158+0000","last_change":"2024-02-13T00:45:17.602497+0000","last_active":"2024-02-13T00:45:33.433158+0000","last_peered":"2024-02-13T00:45:33.433158+0000","last_clean":"2024-02-13T00:45:33.433158+0000","last_became_active":"2024-02-13T00:45:15.461232+0000","last_became_peered":"2024-02-13T00:45:15.461232+0000","last_unstale":"2024-02-13T00:45:33.433158+0000","last_undegraded":"2024-02-13T00:45:33.433158+0000","last_fullsized":"2024-02-13T00:45:33.433158+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.17","version":"343'75","reported_seq":428,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418311+0000","last_change":"2024-02-13T00:44:52.394531+0000","last_active":"2024-02-13T00:45:33.418311+0000","last_peered":"2024-02-13T00:45:33.418311+0000","last_clean":"2024-02-13T00:45:33.418311+0000","last_became_active":"2024-02-13T00:44:50.412656+0000","last_became_peered":"2024-02-13T00:44:50.412656+0000","last_unstale":"2024-02-13T00:45:33.418311+0000","last_undegraded":"2024-02-13T00:45:33.418311+0000","last_fullsized":"2024-02-13T00:45:33.418311+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.14","version":"200'17","reported_seq":346,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417788+0000","last_change":"2024-02-13T00:44:52.067036+0000","last_active":"2024-02-13T00:45:33.417788+0000","last_peered":"2024-02-13T00:45:33.417788+0000","last_clean":"2024-02-13T00:45:33.417788+0000","last_became_active":"2024-02-13T00:44:50.442404+0000","last_became_peered":"2024-02-13T00:44:50.442404+0000","last_unstale":"2024-02-13T00:45:33.417788+0000","last_undegraded":"2024-02-13T00:45:33.417788+0000","last_fullsized":"2024-02-13T00:45:33.417788+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.15","version":"224'78","reported_seq":246,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423500+0000","last_change":"2024-02-13T00:45:19.173463+0000","last_active":"2024-02-13T00:45:33.423500+0000","last_peered":"2024-02-13T00:45:33.423500+0000","last_clean":"2024-02-13T00:45:33.423500+0000","last_became_active":"2024-02-13T00:45:15.459943+0000","last_became_peered":"2024-02-13T00:45:15.459943+0000","last_unstale":"2024-02-13T00:45:33.423500+0000","last_undegraded":"2024-02-13T00:45:33.423500+0000","last_fullsized":"2024-02-13T00:45:33.423500+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":63,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421498+0000","last_change":"2024-02-13T00:45:15.449660+0000","last_active":"2024-02-13T00:45:33.421498+0000","last_peered":"2024-02-13T00:45:33.421498+0000","last_clean":"2024-02-13T00:45:33.421498+0000","last_became_active":"2024-02-13T00:45:15.448446+0000","last_became_peered":"2024-02-13T00:45:15.448446+0000","last_unstale":"2024-02-13T00:45:33.421498+0000","last_undegraded":"2024-02-13T00:45:33.421498+0000","last_fullsized":"2024-02-13T00:45:33.421498+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1a","version":"220'51","reported_seq":138,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421048+0000","last_change":"2024-02-13T00:45:17.421913+0000","last_active":"2024-02-13T00:45:33.421048+0000","last_peered":"2024-02-13T00:45:33.421048+0000","last_clean":"2024-02-13T00:45:33.421048+0000","last_became_active":"2024-02-13T00:45:15.436908+0000","last_became_peered":"2024-02-13T00:45:15.436908+0000","last_unstale":"2024-02-13T00:45:33.421048+0000","last_undegraded":"2024-02-13T00:45:33.421048+0000","last_fullsized":"2024-02-13T00:45:33.421048+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":255,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":157,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421672+0000","last_change":"2024-02-13T00:45:15.426658+0000","last_active":"2024-02-13T00:45:33.421672+0000","last_peered":"2024-02-13T00:45:33.421672+0000","last_clean":"2024-02-13T00:45:33.421672+0000","last_became_active":"2024-02-13T00:45:15.426109+0000","last_became_peered":"2024-02-13T00:45:15.426109+0000","last_unstale":"2024-02-13T00:45:33.421672+0000","last_undegraded":"2024-02-13T00:45:33.421672+0000","last_fullsized":"2024-02-13T00:45:33.421672+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1b","version":"284'48","reported_seq":99,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421384+0000","last_change":"2024-02-13T00:44:48.806217+0000","last_active":"2024-02-13T00:45:33.421384+0000","last_peered":"2024-02-13T00:45:33.421384+0000","last_clean":"2024-02-13T00:45:33.421384+0000","last_became_active":"2024-02-13T00:44:39.525488+0000","last_became_peered":"2024-02-13T00:44:39.525488+0000","last_unstale":"2024-02-13T00:45:33.421384+0000","last_undegraded":"2024-02-13T00:45:33.421384+0000","last_fullsized":"2024-02-13T00:45:33.421384+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":16,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":65,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421261+0000","last_change":"2024-02-13T00:44:39.526065+0000","last_active":"2024-02-13T00:45:33.421261+0000","last_peered":"2024-02-13T00:45:33.421261+0000","last_clean":"2024-02-13T00:45:33.421261+0000","last_became_active":"2024-02-13T00:44:39.525161+0000","last_became_peered":"2024-02-13T00:44:39.525161+0000","last_unstale":"2024-02-13T00:45:33.421261+0000","last_undegraded":"2024-02-13T00:45:33.421261+0000","last_fullsized":"2024-02-13T00:45:33.421261+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.18","version":"204'34","reported_seq":304,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418586+0000","last_change":"2024-02-13T00:45:29.218673+0000","last_active":"2024-02-13T00:45:33.418586+0000","last_peered":"2024-02-13T00:45:33.418586+0000","last_clean":"2024-02-13T00:45:33.418586+0000","last_became_active":"2024-02-13T00:43:35.437411+0000","last_became_peered":"2024-02-13T00:43:35.437411+0000","last_unstale":"2024-02-13T00:45:33.418586+0000","last_undegraded":"2024-02-13T00:45:33.418586+0000","last_fullsized":"2024-02-13T00:45:33.418586+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-02-13T00:45:29.218592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.218592+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421154+0000","last_change":"2024-02-13T00:44:50.439266+0000","last_active":"2024-02-13T00:45:33.421154+0000","last_peered":"2024-02-13T00:45:33.421154+0000","last_clean":"2024-02-13T00:45:33.421154+0000","last_became_active":"2024-02-13T00:44:50.438621+0000","last_became_peered":"2024-02-13T00:44:50.438621+0000","last_unstale":"2024-02-13T00:45:33.421154+0000","last_undegraded":"2024-02-13T00:45:33.421154+0000","last_fullsized":"2024-02-13T00:45:33.421154+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.19","version":"357'18","reported_seq":328,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418861+0000","last_change":"2024-02-13T00:43:59.375182+0000","last_active":"2024-02-13T00:45:33.418861+0000","last_peered":"2024-02-13T00:45:33.418861+0000","last_clean":"2024-02-13T00:45:33.418861+0000","last_became_active":"2024-02-13T00:41:16.749013+0000","last_became_peered":"2024-02-13T00:41:16.749013+0000","last_unstale":"2024-02-13T00:45:33.418861+0000","last_undegraded":"2024-02-13T00:45:33.418861+0000","last_fullsized":"2024-02-13T00:45:33.418861+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422832+0000","last_change":"2024-02-13T00:44:50.436271+0000","last_active":"2024-02-13T00:45:33.422832+0000","last_peered":"2024-02-13T00:45:33.422832+0000","last_clean":"2024-02-13T00:45:33.422832+0000","last_became_active":"2024-02-13T00:44:50.435974+0000","last_became_peered":"2024-02-13T00:44:50.435974+0000","last_unstale":"2024-02-13T00:45:33.422832+0000","last_undegraded":"2024-02-13T00:45:33.422832+0000","last_fullsized":"2024-02-13T00:45:33.422832+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1e","version":"343'54","reported_seq":191,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422600+0000","last_change":"2024-02-13T00:45:20.751154+0000","last_active":"2024-02-13T00:45:33.422600+0000","last_peered":"2024-02-13T00:45:33.422600+0000","last_clean":"2024-02-13T00:45:33.422600+0000","last_became_active":"2024-02-13T00:45:15.446531+0000","last_became_peered":"2024-02-13T00:45:15.446531+0000","last_unstale":"2024-02-13T00:45:33.422600+0000","last_undegraded":"2024-02-13T00:45:33.422600+0000","last_fullsized":"2024-02-13T00:45:33.422600+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":380,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422325+0000","last_change":"2024-02-13T00:45:15.445947+0000","last_active":"2024-02-13T00:45:33.422325+0000","last_peered":"2024-02-13T00:45:33.422325+0000","last_clean":"2024-02-13T00:45:33.422325+0000","last_became_active":"2024-02-13T00:45:15.445625+0000","last_became_peered":"2024-02-13T00:45:15.445625+0000","last_unstale":"2024-02-13T00:45:33.422325+0000","last_undegraded":"2024-02-13T00:45:33.422325+0000","last_fullsized":"2024-02-13T00:45:33.422325+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1f","version":"292'62","reported_seq":374,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418342+0000","last_change":"2024-02-13T00:44:54.113387+0000","last_active":"2024-02-13T00:45:33.418342+0000","last_peered":"2024-02-13T00:45:33.418342+0000","last_clean":"2024-02-13T00:45:33.418342+0000","last_became_active":"2024-02-13T00:44:50.441446+0000","last_became_peered":"2024-02-13T00:44:50.441446+0000","last_unstale":"2024-02-13T00:45:33.418342+0000","last_undegraded":"2024-02-13T00:45:33.418342+0000","last_fullsized":"2024-02-13T00:45:33.418342+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":65,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422924+0000","last_change":"2024-02-13T00:44:39.525601+0000","last_active":"2024-02-13T00:45:33.422924+0000","last_peered":"2024-02-13T00:45:33.422924+0000","last_clean":"2024-02-13T00:45:33.422924+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:45:33.422924+0000","last_undegraded":"2024-02-13T00:45:33.422924+0000","last_fullsized":"2024-02-13T00:45:33.422924+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1c","version":"343'52","reported_seq":435,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422412+0000","last_change":"2024-02-13T00:44:40.899133+0000","last_active":"2024-02-13T00:45:33.422412+0000","last_peered":"2024-02-13T00:45:33.422412+0000","last_clean":"2024-02-13T00:45:33.422412+0000","last_became_active":"2024-02-13T00:44:39.503617+0000","last_became_peered":"2024-02-13T00:44:39.503617+0000","last_unstale":"2024-02-13T00:45:33.422412+0000","last_undegraded":"2024-02-13T00:45:33.422412+0000","last_fullsized":"2024-02-13T00:45:33.422412+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":189,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":139,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424280+0000","last_change":"2024-02-13T00:43:24.381012+0000","last_active":"2024-02-13T00:45:33.424280+0000","last_peered":"2024-02-13T00:45:33.424280+0000","last_clean":"2024-02-13T00:45:33.424280+0000","last_became_active":"2024-02-13T00:43:24.380689+0000","last_became_peered":"2024-02-13T00:43:24.380689+0000","last_unstale":"2024-02-13T00:45:33.424280+0000","last_undegraded":"2024-02-13T00:45:33.424280+0000","last_fullsized":"2024-02-13T00:45:33.424280+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1d","version":"343'69","reported_seq":469,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423235+0000","last_change":"2024-02-13T00:45:19.696168+0000","last_active":"2024-02-13T00:45:33.423235+0000","last_peered":"2024-02-13T00:45:33.423235+0000","last_clean":"2024-02-13T00:45:33.423235+0000","last_became_active":"2024-02-13T00:45:15.435242+0000","last_became_peered":"2024-02-13T00:45:15.435242+0000","last_unstale":"2024-02-13T00:45:33.423235+0000","last_undegraded":"2024-02-13T00:45:33.423235+0000","last_fullsized":"2024-02-13T00:45:33.423235+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:41:15.634020+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":299,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416608+0000","last_change":"2024-02-13T00:44:39.505474+0000","last_active":"2024-02-13T00:45:33.416608+0000","last_peered":"2024-02-13T00:45:33.416608+0000","last_clean":"2024-02-13T00:45:33.416608+0000","last_became_active":"2024-02-13T00:44:39.505121+0000","last_became_peered":"2024-02-13T00:44:39.505121+0000","last_unstale":"2024-02-13T00:45:33.416608+0000","last_undegraded":"2024-02-13T00:45:33.416608+0000","last_fullsized":"2024-02-13T00:45:33.416608+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"363'14","reported_seq":96,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:34.306311+0000","last_change":"2024-02-13T00:45:15.960519+0000","last_active":"2024-02-13T00:45:34.306311+0000","last_peered":"2024-02-13T00:45:34.306311+0000","last_clean":"2024-02-13T00:45:34.306311+0000","last_became_active":"2024-02-13T00:45:15.435476+0000","last_became_peered":"2024-02-13T00:45:15.435476+0000","last_unstale":"2024-02-13T00:45:34.306311+0000","last_undegraded":"2024-02-13T00:45:34.306311+0000","last_fullsized":"2024-02-13T00:45:34.306311+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:40:58.179446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:40:58.179446+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":489,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418688+0000","last_change":"2024-02-13T00:45:19.203705+0000","last_active":"2024-02-13T00:45:33.418688+0000","last_peered":"2024-02-13T00:45:33.418688+0000","last_clean":"2024-02-13T00:45:33.418688+0000","last_became_active":"2024-02-13T00:45:15.470446+0000","last_became_peered":"2024-02-13T00:45:15.470446+0000","last_unstale":"2024-02-13T00:45:33.418688+0000","last_undegraded":"2024-02-13T00:45:33.418688+0000","last_fullsized":"2024-02-13T00:45:33.418688+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:41:31.413581+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_clean_scrub_stamp":"2024-02-13T00:41:31.413581+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":365,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422343+0000","last_change":"2024-02-13T00:45:15.439083+0000","last_active":"2024-02-13T00:45:33.422343+0000","last_peered":"2024-02-13T00:45:33.422343+0000","last_clean":"2024-02-13T00:45:33.422343+0000","last_became_active":"2024-02-13T00:45:15.438392+0000","last_became_peered":"2024-02-13T00:45:15.438392+0000","last_unstale":"2024-02-13T00:45:33.422343+0000","last_undegraded":"2024-02-13T00:45:33.422343+0000","last_fullsized":"2024-02-13T00:45:33.422343+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.340080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.340080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":354,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419848+0000","last_change":"2024-02-13T00:44:39.503788+0000","last_active":"2024-02-13T00:45:33.419848+0000","last_peered":"2024-02-13T00:45:33.419848+0000","last_clean":"2024-02-13T00:45:33.419848+0000","last_became_active":"2024-02-13T00:44:39.503584+0000","last_became_peered":"2024-02-13T00:44:39.503584+0000","last_unstale":"2024-02-13T00:45:33.419848+0000","last_undegraded":"2024-02-13T00:45:33.419848+0000","last_fullsized":"2024-02-13T00:45:33.419848+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:45.416238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:45.416238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":309,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421100+0000","last_change":"2024-02-13T00:45:15.442861+0000","last_active":"2024-02-13T00:45:33.421100+0000","last_peered":"2024-02-13T00:45:33.421100+0000","last_clean":"2024-02-13T00:45:33.421100+0000","last_became_active":"2024-02-13T00:45:15.441991+0000","last_became_peered":"2024-02-13T00:45:15.441991+0000","last_unstale":"2024-02-13T00:45:33.421100+0000","last_undegraded":"2024-02-13T00:45:33.421100+0000","last_fullsized":"2024-02-13T00:45:33.421100+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":369,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417728+0000","last_change":"2024-02-13T00:42:19.652772+0000","last_active":"2024-02-13T00:45:33.417728+0000","last_peered":"2024-02-13T00:45:33.417728+0000","last_clean":"2024-02-13T00:45:33.417728+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:45:33.417728+0000","last_undegraded":"2024-02-13T00:45:33.417728+0000","last_fullsized":"2024-02-13T00:45:33.417728+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:42:19.652689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:19.652689+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":358,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424320+0000","last_change":"2024-02-13T00:44:50.430435+0000","last_active":"2024-02-13T00:45:33.424320+0000","last_peered":"2024-02-13T00:45:33.424320+0000","last_clean":"2024-02-13T00:45:33.424320+0000","last_became_active":"2024-02-13T00:44:50.429994+0000","last_became_peered":"2024-02-13T00:44:50.429994+0000","last_unstale":"2024-02-13T00:45:33.424320+0000","last_undegraded":"2024-02-13T00:45:33.424320+0000","last_fullsized":"2024-02-13T00:45:33.424320+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:20.102471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:20.102471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":274,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417219+0000","last_change":"2024-02-13T00:44:50.434925+0000","last_active":"2024-02-13T00:45:33.417219+0000","last_peered":"2024-02-13T00:45:33.417219+0000","last_clean":"2024-02-13T00:45:33.417219+0000","last_became_active":"2024-02-13T00:44:50.433984+0000","last_became_peered":"2024-02-13T00:44:50.433984+0000","last_unstale":"2024-02-13T00:45:33.417219+0000","last_undegraded":"2024-02-13T00:45:33.417219+0000","last_fullsized":"2024-02-13T00:45:33.417219+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.512181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.512181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":361,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417122+0000","last_change":"2024-02-13T00:45:15.435610+0000","last_active":"2024-02-13T00:45:33.417122+0000","last_peered":"2024-02-13T00:45:33.417122+0000","last_clean":"2024-02-13T00:45:33.417122+0000","last_became_active":"2024-02-13T00:45:15.434853+0000","last_became_peered":"2024-02-13T00:45:15.434853+0000","last_unstale":"2024-02-13T00:45:33.417122+0000","last_undegraded":"2024-02-13T00:45:33.417122+0000","last_fullsized":"2024-02-13T00:45:33.417122+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":280,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-02-13T00:45:35.197147+0000","last_change":"2024-02-13T00:45:35.197147+0000","last_active":"2024-02-13T00:45:35.197147+0000","last_peered":"2024-02-13T00:45:35.197147+0000","last_clean":"2024-02-13T00:45:35.197147+0000","last_became_active":"2024-02-13T00:45:06.416758+0000","last_became_peered":"2024-02-13T00:45:06.416758+0000","last_unstale":"2024-02-13T00:45:35.197147+0000","last_undegraded":"2024-02-13T00:45:35.197147+0000","last_fullsized":"2024-02-13T00:45:35.197147+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:35.197045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:35.197045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"91'35","reported_seq":118,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.420943+0000","last_change":"2024-02-13T00:45:21.501530+0000","last_active":"2024-02-13T00:45:33.420943+0000","last_peered":"2024-02-13T00:45:33.420943+0000","last_clean":"2024-02-13T00:45:33.420943+0000","last_became_active":"2024-02-13T00:45:15.470272+0000","last_became_peered":"2024-02-13T00:45:15.470272+0000","last_unstale":"2024-02-13T00:45:33.420943+0000","last_undegraded":"2024-02-13T00:45:33.420943+0000","last_fullsized":"2024-02-13T00:45:33.420943+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:44:24.196214+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:44:24.196214+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":56,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422869+0000","last_change":"2024-02-13T00:44:50.398271+0000","last_active":"2024-02-13T00:45:33.422869+0000","last_peered":"2024-02-13T00:45:33.422869+0000","last_clean":"2024-02-13T00:45:33.422869+0000","last_became_active":"2024-02-13T00:44:50.397952+0000","last_became_peered":"2024-02-13T00:44:50.397952+0000","last_unstale":"2024-02-13T00:45:33.422869+0000","last_undegraded":"2024-02-13T00:45:33.422869+0000","last_fullsized":"2024-02-13T00:45:33.422869+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:47.446012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:47.446012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":65,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417530+0000","last_change":"2024-02-13T00:45:15.460315+0000","last_active":"2024-02-13T00:45:33.417530+0000","last_peered":"2024-02-13T00:45:33.417530+0000","last_clean":"2024-02-13T00:45:33.417530+0000","last_became_active":"2024-02-13T00:45:15.459878+0000","last_became_peered":"2024-02-13T00:45:15.459878+0000","last_unstale":"2024-02-13T00:45:33.417530+0000","last_undegraded":"2024-02-13T00:45:33.417530+0000","last_fullsized":"2024-02-13T00:45:33.417530+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:03.411257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:17.631877+0000","last_clean_scrub_stamp":"2024-02-13T00:41:03.411257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":201,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.415992+0000","last_change":"2024-02-13T00:45:15.429026+0000","last_active":"2024-02-13T00:45:33.415992+0000","last_peered":"2024-02-13T00:45:33.415992+0000","last_clean":"2024-02-13T00:45:33.415992+0000","last_became_active":"2024-02-13T00:45:15.427945+0000","last_became_peered":"2024-02-13T00:45:15.427945+0000","last_unstale":"2024-02-13T00:45:33.415992+0000","last_undegraded":"2024-02-13T00:45:33.415992+0000","last_fullsized":"2024-02-13T00:45:33.415992+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":303,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423518+0000","last_change":"2024-02-13T00:45:26.255907+0000","last_active":"2024-02-13T00:45:33.423518+0000","last_peered":"2024-02-13T00:45:33.423518+0000","last_clean":"2024-02-13T00:45:33.423518+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:45:33.423518+0000","last_undegraded":"2024-02-13T00:45:33.423518+0000","last_fullsized":"2024-02-13T00:45:33.423518+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.255842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.255842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":114,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424253+0000","last_change":"2024-02-13T00:44:54.041282+0000","last_active":"2024-02-13T00:45:33.424253+0000","last_peered":"2024-02-13T00:45:33.424253+0000","last_clean":"2024-02-13T00:45:33.424253+0000","last_became_active":"2024-02-13T00:44:50.432807+0000","last_became_peered":"2024-02-13T00:44:50.432807+0000","last_unstale":"2024-02-13T00:45:33.424253+0000","last_undegraded":"2024-02-13T00:45:33.424253+0000","last_fullsized":"2024-02-13T00:45:33.424253+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:42:17.413826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.413826+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":199,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421102+0000","last_change":"2024-02-13T00:45:24.269675+0000","last_active":"2024-02-13T00:45:33.421102+0000","last_peered":"2024-02-13T00:45:33.421102+0000","last_clean":"2024-02-13T00:45:33.421102+0000","last_became_active":"2024-02-13T00:45:15.472471+0000","last_became_peered":"2024-02-13T00:45:15.472471+0000","last_unstale":"2024-02-13T00:45:33.421102+0000","last_undegraded":"2024-02-13T00:45:33.421102+0000","last_fullsized":"2024-02-13T00:45:33.421102+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:45:24.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:24.269595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":338,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422905+0000","last_change":"2024-02-13T00:42:28.037452+0000","last_active":"2024-02-13T00:45:33.422905+0000","last_peered":"2024-02-13T00:45:33.422905+0000","last_clean":"2024-02-13T00:45:33.422905+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:45:33.422905+0000","last_undegraded":"2024-02-13T00:45:33.422905+0000","last_fullsized":"2024-02-13T00:45:33.422905+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:28.037365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:28.037365+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":350,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422363+0000","last_change":"2024-02-13T00:44:50.441408+0000","last_active":"2024-02-13T00:45:33.422363+0000","last_peered":"2024-02-13T00:45:33.422363+0000","last_clean":"2024-02-13T00:45:33.422363+0000","last_became_active":"2024-02-13T00:44:50.440301+0000","last_became_peered":"2024-02-13T00:44:50.440301+0000","last_unstale":"2024-02-13T00:45:33.422363+0000","last_undegraded":"2024-02-13T00:45:33.422363+0000","last_fullsized":"2024-02-13T00:45:33.422363+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:25.502311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.502311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":59,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422243+0000","last_change":"2024-02-13T00:44:50.403161+0000","last_active":"2024-02-13T00:45:33.422243+0000","last_peered":"2024-02-13T00:45:33.422243+0000","last_clean":"2024-02-13T00:45:33.422243+0000","last_became_active":"2024-02-13T00:44:50.402280+0000","last_became_peered":"2024-02-13T00:44:50.402280+0000","last_unstale":"2024-02-13T00:45:33.422243+0000","last_undegraded":"2024-02-13T00:45:33.422243+0000","last_fullsized":"2024-02-13T00:45:33.422243+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.488353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.488353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":77,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421965+0000","last_change":"2024-02-13T00:44:46.481036+0000","last_active":"2024-02-13T00:45:33.421965+0000","last_peered":"2024-02-13T00:45:33.421965+0000","last_clean":"2024-02-13T00:45:33.421965+0000","last_became_active":"2024-02-13T00:44:39.523138+0000","last_became_peered":"2024-02-13T00:44:39.523138+0000","last_unstale":"2024-02-13T00:45:33.421965+0000","last_undegraded":"2024-02-13T00:45:33.421965+0000","last_fullsized":"2024-02-13T00:45:33.421965+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:43:00.776560+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.776560+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":358,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422018+0000","last_change":"2024-02-13T00:45:15.450371+0000","last_active":"2024-02-13T00:45:33.422018+0000","last_peered":"2024-02-13T00:45:33.422018+0000","last_clean":"2024-02-13T00:45:33.422018+0000","last_became_active":"2024-02-13T00:45:15.448091+0000","last_became_peered":"2024-02-13T00:45:15.448091+0000","last_unstale":"2024-02-13T00:45:33.422018+0000","last_undegraded":"2024-02-13T00:45:33.422018+0000","last_fullsized":"2024-02-13T00:45:33.422018+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:24.059313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:24.059313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"363'10","reported_seq":101,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:34.306349+0000","last_change":"2024-02-13T00:45:19.174724+0000","last_active":"2024-02-13T00:45:34.306349+0000","last_peered":"2024-02-13T00:45:34.306349+0000","last_clean":"2024-02-13T00:45:34.306349+0000","last_became_active":"2024-02-13T00:45:15.465429+0000","last_became_peered":"2024-02-13T00:45:15.465429+0000","last_unstale":"2024-02-13T00:45:34.306349+0000","last_undegraded":"2024-02-13T00:45:34.306349+0000","last_fullsized":"2024-02-13T00:45:34.306349+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:56.422996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.422996+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":326,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424544+0000","last_change":"2024-02-13T00:44:50.599996+0000","last_active":"2024-02-13T00:45:33.424544+0000","last_peered":"2024-02-13T00:45:33.424544+0000","last_clean":"2024-02-13T00:45:33.424544+0000","last_became_active":"2024-02-13T00:44:50.419338+0000","last_became_peered":"2024-02-13T00:44:50.419338+0000","last_unstale":"2024-02-13T00:45:33.424544+0000","last_undegraded":"2024-02-13T00:45:33.424544+0000","last_fullsized":"2024-02-13T00:45:33.424544+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:44:25.230980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.230980+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":187,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424857+0000","last_change":"2024-02-13T00:45:15.467349+0000","last_active":"2024-02-13T00:45:33.424857+0000","last_peered":"2024-02-13T00:45:33.424857+0000","last_clean":"2024-02-13T00:45:33.424857+0000","last_became_active":"2024-02-13T00:45:15.467094+0000","last_became_peered":"2024-02-13T00:45:15.467094+0000","last_unstale":"2024-02-13T00:45:33.424857+0000","last_undegraded":"2024-02-13T00:45:33.424857+0000","last_fullsized":"2024-02-13T00:45:33.424857+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:18.089252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:18.089252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":413,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424533+0000","last_change":"2024-02-13T00:45:18.339581+0000","last_active":"2024-02-13T00:45:33.424533+0000","last_peered":"2024-02-13T00:45:33.424533+0000","last_clean":"2024-02-13T00:45:33.424533+0000","last_became_active":"2024-02-13T00:45:15.449867+0000","last_became_peered":"2024-02-13T00:45:15.449867+0000","last_unstale":"2024-02-13T00:45:33.424533+0000","last_undegraded":"2024-02-13T00:45:33.424533+0000","last_fullsized":"2024-02-13T00:45:33.424533+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:42:21.069482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.069482+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":399,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422900+0000","last_change":"2024-02-13T00:45:15.445834+0000","last_active":"2024-02-13T00:45:33.422900+0000","last_peered":"2024-02-13T00:45:33.422900+0000","last_clean":"2024-02-13T00:45:33.422900+0000","last_became_active":"2024-02-13T00:45:15.445228+0000","last_became_peered":"2024-02-13T00:45:15.445228+0000","last_unstale":"2024-02-13T00:45:33.422900+0000","last_undegraded":"2024-02-13T00:45:33.422900+0000","last_fullsized":"2024-02-13T00:45:33.422900+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:43:34.320637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.320637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"96'9","reported_seq":384,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:34.306356+0000","last_change":"2024-02-13T00:42:23.496877+0000","last_active":"2024-02-13T00:45:34.306356+0000","last_peered":"2024-02-13T00:45:34.306356+0000","last_clean":"2024-02-13T00:45:34.306356+0000","last_became_active":"2024-02-13T00:40:35.847293+0000","last_became_peered":"2024-02-13T00:40:35.847293+0000","last_unstale":"2024-02-13T00:45:34.306356+0000","last_undegraded":"2024-02-13T00:45:34.306356+0000","last_fullsized":"2024-02-13T00:45:34.306356+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-02-13T00:42:23.496800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:23.496800+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":339,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423851+0000","last_change":"2024-02-13T00:43:47.092345+0000","last_active":"2024-02-13T00:45:33.423851+0000","last_peered":"2024-02-13T00:45:33.423851+0000","last_clean":"2024-02-13T00:45:33.423851+0000","last_became_active":"2024-02-13T00:37:47.260477+0000","last_became_peered":"2024-02-13T00:37:47.260477+0000","last_unstale":"2024-02-13T00:45:33.423851+0000","last_undegraded":"2024-02-13T00:45:33.423851+0000","last_fullsized":"2024-02-13T00:45:33.423851+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.092250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.092250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":295,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419067+0000","last_change":"2024-02-13T00:44:59.963685+0000","last_active":"2024-02-13T00:45:33.419067+0000","last_peered":"2024-02-13T00:45:33.419067+0000","last_clean":"2024-02-13T00:45:33.419067+0000","last_became_active":"2024-02-13T00:44:50.443007+0000","last_became_peered":"2024-02-13T00:44:50.443007+0000","last_unstale":"2024-02-13T00:45:33.419067+0000","last_undegraded":"2024-02-13T00:45:33.419067+0000","last_fullsized":"2024-02-13T00:45:33.419067+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:44:59.963616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":79,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422720+0000","last_change":"2024-02-13T00:44:45.962479+0000","last_active":"2024-02-13T00:45:33.422720+0000","last_peered":"2024-02-13T00:45:33.422720+0000","last_clean":"2024-02-13T00:45:33.422720+0000","last_became_active":"2024-02-13T00:44:39.521984+0000","last_became_peered":"2024-02-13T00:44:39.521984+0000","last_unstale":"2024-02-13T00:45:33.422720+0000","last_undegraded":"2024-02-13T00:45:33.422720+0000","last_fullsized":"2024-02-13T00:45:33.422720+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:42:52.245750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.245750+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":182,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416488+0000","last_change":"2024-02-13T00:45:15.419983+0000","last_active":"2024-02-13T00:45:33.416488+0000","last_peered":"2024-02-13T00:45:33.416488+0000","last_clean":"2024-02-13T00:45:33.416488+0000","last_became_active":"2024-02-13T00:45:15.419239+0000","last_became_peered":"2024-02-13T00:45:15.419239+0000","last_unstale":"2024-02-13T00:45:33.416488+0000","last_undegraded":"2024-02-13T00:45:33.416488+0000","last_fullsized":"2024-02-13T00:45:33.416488+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:32.399475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.399475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":359,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424726+0000","last_change":"2024-02-13T00:44:39.505239+0000","last_active":"2024-02-13T00:45:33.424726+0000","last_peered":"2024-02-13T00:45:33.424726+0000","last_clean":"2024-02-13T00:45:33.424726+0000","last_became_active":"2024-02-13T00:44:39.504786+0000","last_became_peered":"2024-02-13T00:44:39.504786+0000","last_unstale":"2024-02-13T00:45:33.424726+0000","last_undegraded":"2024-02-13T00:45:33.424726+0000","last_fullsized":"2024-02-13T00:45:33.424726+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:57.546517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.546517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421801+0000","last_change":"2024-02-13T00:45:15.439949+0000","last_active":"2024-02-13T00:45:33.421801+0000","last_peered":"2024-02-13T00:45:33.421801+0000","last_clean":"2024-02-13T00:45:33.421801+0000","last_became_active":"2024-02-13T00:45:15.439369+0000","last_became_peered":"2024-02-13T00:45:15.439369+0000","last_unstale":"2024-02-13T00:45:33.421801+0000","last_undegraded":"2024-02-13T00:45:33.421801+0000","last_fullsized":"2024-02-13T00:45:33.421801+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.782345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.782345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":295,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419192+0000","last_change":"2024-02-13T00:45:15.458497+0000","last_active":"2024-02-13T00:45:33.419192+0000","last_peered":"2024-02-13T00:45:33.419192+0000","last_clean":"2024-02-13T00:45:33.419192+0000","last_became_active":"2024-02-13T00:45:15.458091+0000","last_became_peered":"2024-02-13T00:45:15.458091+0000","last_unstale":"2024-02-13T00:45:33.419192+0000","last_undegraded":"2024-02-13T00:45:33.419192+0000","last_fullsized":"2024-02-13T00:45:33.419192+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.281102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.281102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"263'10","reported_seq":320,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:34.306318+0000","last_change":"2024-02-13T00:43:38.251836+0000","last_active":"2024-02-13T00:45:34.306318+0000","last_peered":"2024-02-13T00:45:34.306318+0000","last_clean":"2024-02-13T00:45:34.306318+0000","last_became_active":"2024-02-13T00:43:35.454087+0000","last_became_peered":"2024-02-13T00:43:35.454087+0000","last_unstale":"2024-02-13T00:45:34.306318+0000","last_undegraded":"2024-02-13T00:45:34.306318+0000","last_fullsized":"2024-02-13T00:45:34.306318+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'9","last_scrub_stamp":"2024-02-13T00:41:44.698906+0000","last_deep_scrub":"149'9","last_deep_scrub_stamp":"2024-02-13T00:41:44.698906+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.698906+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":408,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419059+0000","last_change":"2024-02-13T00:44:40.199011+0000","last_active":"2024-02-13T00:45:33.419059+0000","last_peered":"2024-02-13T00:45:33.419059+0000","last_clean":"2024-02-13T00:45:33.419059+0000","last_became_active":"2024-02-13T00:44:39.493872+0000","last_became_peered":"2024-02-13T00:44:39.493872+0000","last_unstale":"2024-02-13T00:45:33.419059+0000","last_undegraded":"2024-02-13T00:45:33.419059+0000","last_fullsized":"2024-02-13T00:45:33.419059+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2024-02-13T00:43:00.636798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.636798+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":83,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422774+0000","last_change":"2024-02-13T00:45:15.446786+0000","last_active":"2024-02-13T00:45:33.422774+0000","last_peered":"2024-02-13T00:45:33.422774+0000","last_clean":"2024-02-13T00:45:33.422774+0000","last_became_active":"2024-02-13T00:45:15.446443+0000","last_became_peered":"2024-02-13T00:45:15.446443+0000","last_unstale":"2024-02-13T00:45:33.422774+0000","last_undegraded":"2024-02-13T00:45:33.422774+0000","last_fullsized":"2024-02-13T00:45:33.422774+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.557475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.557475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":83,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421312+0000","last_change":"2024-02-13T00:45:15.436286+0000","last_active":"2024-02-13T00:45:33.421312+0000","last_peered":"2024-02-13T00:45:33.421312+0000","last_clean":"2024-02-13T00:45:33.421312+0000","last_became_active":"2024-02-13T00:45:15.434559+0000","last_became_peered":"2024-02-13T00:45:15.434559+0000","last_unstale":"2024-02-13T00:45:33.421312+0000","last_undegraded":"2024-02-13T00:45:33.421312+0000","last_fullsized":"2024-02-13T00:45:33.421312+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":145,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418879+0000","last_change":"2024-02-13T00:45:15.471913+0000","last_active":"2024-02-13T00:45:33.418879+0000","last_peered":"2024-02-13T00:45:33.418879+0000","last_clean":"2024-02-13T00:45:33.418879+0000","last_became_active":"2024-02-13T00:45:15.471290+0000","last_became_peered":"2024-02-13T00:45:15.471290+0000","last_unstale":"2024-02-13T00:45:33.418879+0000","last_undegraded":"2024-02-13T00:45:33.418879+0000","last_fullsized":"2024-02-13T00:45:33.418879+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:43.049231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:43.049231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":138,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416564+0000","last_change":"2024-02-13T00:45:15.441934+0000","last_active":"2024-02-13T00:45:33.416564+0000","last_peered":"2024-02-13T00:45:33.416564+0000","last_clean":"2024-02-13T00:45:33.416564+0000","last_became_active":"2024-02-13T00:45:15.441539+0000","last_became_peered":"2024-02-13T00:45:15.441539+0000","last_unstale":"2024-02-13T00:45:33.416564+0000","last_undegraded":"2024-02-13T00:45:33.416564+0000","last_fullsized":"2024-02-13T00:45:33.416564+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:54.842056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:54.842056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":259,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416446+0000","last_change":"2024-02-13T00:45:15.491188+0000","last_active":"2024-02-13T00:45:33.416446+0000","last_peered":"2024-02-13T00:45:33.416446+0000","last_clean":"2024-02-13T00:45:33.416446+0000","last_became_active":"2024-02-13T00:45:15.441092+0000","last_became_peered":"2024-02-13T00:45:15.441092+0000","last_unstale":"2024-02-13T00:45:33.416446+0000","last_undegraded":"2024-02-13T00:45:33.416446+0000","last_fullsized":"2024-02-13T00:45:33.416446+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:42:53.881594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.881594+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":367,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424376+0000","last_change":"2024-02-13T00:44:39.503982+0000","last_active":"2024-02-13T00:45:33.424376+0000","last_peered":"2024-02-13T00:45:33.424376+0000","last_clean":"2024-02-13T00:45:33.424376+0000","last_became_active":"2024-02-13T00:44:39.503590+0000","last_became_peered":"2024-02-13T00:44:39.503590+0000","last_unstale":"2024-02-13T00:45:33.424376+0000","last_undegraded":"2024-02-13T00:45:33.424376+0000","last_fullsized":"2024-02-13T00:45:33.424376+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.236727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.236727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":309,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.420541+0000","last_change":"2024-02-13T00:45:15.470624+0000","last_active":"2024-02-13T00:45:33.420541+0000","last_peered":"2024-02-13T00:45:33.420541+0000","last_clean":"2024-02-13T00:45:33.420541+0000","last_became_active":"2024-02-13T00:45:15.469334+0000","last_became_peered":"2024-02-13T00:45:15.469334+0000","last_unstale":"2024-02-13T00:45:33.420541+0000","last_undegraded":"2024-02-13T00:45:33.420541+0000","last_fullsized":"2024-02-13T00:45:33.420541+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":305,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.420845+0000","last_change":"2024-02-13T00:44:50.436745+0000","last_active":"2024-02-13T00:45:33.420845+0000","last_peered":"2024-02-13T00:45:33.420845+0000","last_clean":"2024-02-13T00:45:33.420845+0000","last_became_active":"2024-02-13T00:44:50.436058+0000","last_became_peered":"2024-02-13T00:44:50.436058+0000","last_unstale":"2024-02-13T00:45:33.420845+0000","last_undegraded":"2024-02-13T00:45:33.420845+0000","last_fullsized":"2024-02-13T00:45:33.420845+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:48.300400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.300400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":70,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.420921+0000","last_change":"2024-02-13T00:45:33.205044+0000","last_active":"2024-02-13T00:45:33.420921+0000","last_peered":"2024-02-13T00:45:33.420921+0000","last_clean":"2024-02-13T00:45:33.420921+0000","last_became_active":"2024-02-13T00:44:39.509737+0000","last_became_peered":"2024-02-13T00:44:39.509737+0000","last_unstale":"2024-02-13T00:45:33.420921+0000","last_undegraded":"2024-02-13T00:45:33.420921+0000","last_fullsized":"2024-02-13T00:45:33.420921+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:33.204969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:33.204969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":427,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417619+0000","last_change":"2024-02-13T00:45:16.265607+0000","last_active":"2024-02-13T00:45:33.417619+0000","last_peered":"2024-02-13T00:45:33.417619+0000","last_clean":"2024-02-13T00:45:33.417619+0000","last_became_active":"2024-02-13T00:45:15.459395+0000","last_became_peered":"2024-02-13T00:45:15.459395+0000","last_unstale":"2024-02-13T00:45:33.417619+0000","last_undegraded":"2024-02-13T00:45:33.417619+0000","last_fullsized":"2024-02-13T00:45:33.417619+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:42:20.688132+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":187,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418293+0000","last_change":"2024-02-13T00:45:15.423607+0000","last_active":"2024-02-13T00:45:33.418293+0000","last_peered":"2024-02-13T00:45:33.418293+0000","last_clean":"2024-02-13T00:45:33.418293+0000","last_became_active":"2024-02-13T00:45:15.423415+0000","last_became_peered":"2024-02-13T00:45:15.423415+0000","last_unstale":"2024-02-13T00:45:33.418293+0000","last_undegraded":"2024-02-13T00:45:33.418293+0000","last_fullsized":"2024-02-13T00:45:33.418293+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:17.295459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:17.295459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":379,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422976+0000","last_change":"2024-02-13T00:45:15.445862+0000","last_active":"2024-02-13T00:45:33.422976+0000","last_peered":"2024-02-13T00:45:33.422976+0000","last_clean":"2024-02-13T00:45:33.422976+0000","last_became_active":"2024-02-13T00:45:15.445557+0000","last_became_peered":"2024-02-13T00:45:15.445557+0000","last_unstale":"2024-02-13T00:45:33.422976+0000","last_undegraded":"2024-02-13T00:45:33.422976+0000","last_fullsized":"2024-02-13T00:45:33.422976+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":56,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424326+0000","last_change":"2024-02-13T00:44:50.402229+0000","last_active":"2024-02-13T00:45:33.424326+0000","last_peered":"2024-02-13T00:45:33.424326+0000","last_clean":"2024-02-13T00:45:33.424326+0000","last_became_active":"2024-02-13T00:44:50.401365+0000","last_became_peered":"2024-02-13T00:44:50.401365+0000","last_unstale":"2024-02-13T00:45:33.424326+0000","last_undegraded":"2024-02-13T00:45:33.424326+0000","last_fullsized":"2024-02-13T00:45:33.424326+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:50.325063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:50.325063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":341,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424767+0000","last_change":"2024-02-13T00:41:54.422549+0000","last_active":"2024-02-13T00:45:33.424767+0000","last_peered":"2024-02-13T00:45:33.424767+0000","last_clean":"2024-02-13T00:45:33.424767+0000","last_became_active":"2024-02-13T00:37:45.245956+0000","last_became_peered":"2024-02-13T00:37:45.245956+0000","last_unstale":"2024-02-13T00:45:33.424767+0000","last_undegraded":"2024-02-13T00:45:33.424767+0000","last_fullsized":"2024-02-13T00:45:33.424767+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:54.422484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:54.422484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":425,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418598+0000","last_change":"2024-02-13T00:45:16.093036+0000","last_active":"2024-02-13T00:45:33.418598+0000","last_peered":"2024-02-13T00:45:33.418598+0000","last_clean":"2024-02-13T00:45:33.418598+0000","last_became_active":"2024-02-13T00:45:15.449882+0000","last_became_peered":"2024-02-13T00:45:15.449882+0000","last_unstale":"2024-02-13T00:45:33.418598+0000","last_undegraded":"2024-02-13T00:45:33.418598+0000","last_fullsized":"2024-02-13T00:45:33.418598+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:41:52.739833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.739833+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":66,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422473+0000","last_change":"2024-02-13T00:45:15.450602+0000","last_active":"2024-02-13T00:45:33.422473+0000","last_peered":"2024-02-13T00:45:33.422473+0000","last_clean":"2024-02-13T00:45:33.422473+0000","last_became_active":"2024-02-13T00:45:15.450015+0000","last_became_peered":"2024-02-13T00:45:15.450015+0000","last_unstale":"2024-02-13T00:45:33.422473+0000","last_undegraded":"2024-02-13T00:45:33.422473+0000","last_fullsized":"2024-02-13T00:45:33.422473+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:54.457605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:54.457605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":349,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416907+0000","last_change":"2024-02-13T00:43:35.437674+0000","last_active":"2024-02-13T00:45:33.416907+0000","last_peered":"2024-02-13T00:45:33.416907+0000","last_clean":"2024-02-13T00:45:33.416907+0000","last_became_active":"2024-02-13T00:43:35.436912+0000","last_became_peered":"2024-02-13T00:43:35.436912+0000","last_unstale":"2024-02-13T00:45:33.416907+0000","last_undegraded":"2024-02-13T00:45:33.416907+0000","last_fullsized":"2024-02-13T00:45:33.416907+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.352030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.352030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":361,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422932+0000","last_change":"2024-02-13T00:44:39.504579+0000","last_active":"2024-02-13T00:45:33.422932+0000","last_peered":"2024-02-13T00:45:33.422932+0000","last_clean":"2024-02-13T00:45:33.422932+0000","last_became_active":"2024-02-13T00:44:39.504221+0000","last_became_peered":"2024-02-13T00:44:39.504221+0000","last_unstale":"2024-02-13T00:45:33.422932+0000","last_undegraded":"2024-02-13T00:45:33.422932+0000","last_fullsized":"2024-02-13T00:45:33.422932+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":304,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423922+0000","last_change":"2024-02-13T00:45:01.179131+0000","last_active":"2024-02-13T00:45:33.423922+0000","last_peered":"2024-02-13T00:45:33.423922+0000","last_clean":"2024-02-13T00:45:33.423922+0000","last_became_active":"2024-02-13T00:44:39.504724+0000","last_became_peered":"2024-02-13T00:44:39.504724+0000","last_unstale":"2024-02-13T00:45:33.423922+0000","last_undegraded":"2024-02-13T00:45:33.423922+0000","last_fullsized":"2024-02-13T00:45:33.423922+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:01.179031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:01.179031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":59,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422731+0000","last_change":"2024-02-13T00:45:15.457357+0000","last_active":"2024-02-13T00:45:33.422731+0000","last_peered":"2024-02-13T00:45:33.422731+0000","last_clean":"2024-02-13T00:45:33.422731+0000","last_became_active":"2024-02-13T00:45:15.456997+0000","last_became_peered":"2024-02-13T00:45:15.456997+0000","last_unstale":"2024-02-13T00:45:33.422731+0000","last_undegraded":"2024-02-13T00:45:33.422731+0000","last_fullsized":"2024-02-13T00:45:33.422731+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:56.666003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.666003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"157'21","reported_seq":106,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423069+0000","last_change":"2024-02-13T00:45:18.327225+0000","last_active":"2024-02-13T00:45:33.423069+0000","last_peered":"2024-02-13T00:45:33.423069+0000","last_clean":"2024-02-13T00:45:33.423069+0000","last_became_active":"2024-02-13T00:45:15.446969+0000","last_became_peered":"2024-02-13T00:45:15.446969+0000","last_unstale":"2024-02-13T00:45:33.423069+0000","last_undegraded":"2024-02-13T00:45:33.423069+0000","last_fullsized":"2024-02-13T00:45:33.423069+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.415907+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":323,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416349+0000","last_change":"2024-02-13T00:45:15.434844+0000","last_active":"2024-02-13T00:45:33.416349+0000","last_peered":"2024-02-13T00:45:33.416349+0000","last_clean":"2024-02-13T00:45:33.416349+0000","last_became_active":"2024-02-13T00:45:15.434498+0000","last_became_peered":"2024-02-13T00:45:15.434498+0000","last_unstale":"2024-02-13T00:45:33.416349+0000","last_undegraded":"2024-02-13T00:45:33.416349+0000","last_fullsized":"2024-02-13T00:45:33.416349+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:23.561163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:42:23.561163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":360,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421232+0000","last_change":"2024-02-13T00:45:15.474154+0000","last_active":"2024-02-13T00:45:33.421232+0000","last_peered":"2024-02-13T00:45:33.421232+0000","last_clean":"2024-02-13T00:45:33.421232+0000","last_became_active":"2024-02-13T00:45:15.473214+0000","last_became_peered":"2024-02-13T00:45:15.473214+0000","last_unstale":"2024-02-13T00:45:33.421232+0000","last_undegraded":"2024-02-13T00:45:33.421232+0000","last_fullsized":"2024-02-13T00:45:33.421232+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.472312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.472312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":269,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.420088+0000","last_change":"2024-02-13T00:44:39.504415+0000","last_active":"2024-02-13T00:45:33.420088+0000","last_peered":"2024-02-13T00:45:33.420088+0000","last_clean":"2024-02-13T00:45:33.420088+0000","last_became_active":"2024-02-13T00:44:39.504055+0000","last_became_peered":"2024-02-13T00:44:39.504055+0000","last_unstale":"2024-02-13T00:45:33.420088+0000","last_undegraded":"2024-02-13T00:45:33.420088+0000","last_fullsized":"2024-02-13T00:45:33.420088+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.544320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.544320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":357,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424801+0000","last_change":"2024-02-13T00:43:35.440823+0000","last_active":"2024-02-13T00:45:33.424801+0000","last_peered":"2024-02-13T00:45:33.424801+0000","last_clean":"2024-02-13T00:45:33.424801+0000","last_became_active":"2024-02-13T00:43:35.440514+0000","last_became_peered":"2024-02-13T00:43:35.440514+0000","last_unstale":"2024-02-13T00:45:33.424801+0000","last_undegraded":"2024-02-13T00:45:33.424801+0000","last_fullsized":"2024-02-13T00:45:33.424801+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":295,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418444+0000","last_change":"2024-02-13T00:45:15.460831+0000","last_active":"2024-02-13T00:45:33.418444+0000","last_peered":"2024-02-13T00:45:33.418444+0000","last_clean":"2024-02-13T00:45:33.418444+0000","last_became_active":"2024-02-13T00:45:15.460376+0000","last_became_peered":"2024-02-13T00:45:15.460376+0000","last_unstale":"2024-02-13T00:45:33.418444+0000","last_undegraded":"2024-02-13T00:45:33.418444+0000","last_fullsized":"2024-02-13T00:45:33.418444+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:16.256516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:16.256516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":65,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424423+0000","last_change":"2024-02-13T00:45:15.465198+0000","last_active":"2024-02-13T00:45:33.424423+0000","last_peered":"2024-02-13T00:45:33.424423+0000","last_clean":"2024-02-13T00:45:33.424423+0000","last_became_active":"2024-02-13T00:45:15.460288+0000","last_became_peered":"2024-02-13T00:45:15.460288+0000","last_unstale":"2024-02-13T00:45:33.424423+0000","last_undegraded":"2024-02-13T00:45:33.424423+0000","last_fullsized":"2024-02-13T00:45:33.424423+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:26.313580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:26.313580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":380,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424608+0000","last_change":"2024-02-13T00:42:54.175029+0000","last_active":"2024-02-13T00:45:33.424608+0000","last_peered":"2024-02-13T00:45:33.424608+0000","last_clean":"2024-02-13T00:45:33.424608+0000","last_became_active":"2024-02-13T00:37:43.231120+0000","last_became_peered":"2024-02-13T00:37:43.231120+0000","last_unstale":"2024-02-13T00:45:33.424608+0000","last_undegraded":"2024-02-13T00:45:33.424608+0000","last_fullsized":"2024-02-13T00:45:33.424608+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:42:54.174950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:54.174950+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":278,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416795+0000","last_change":"2024-02-13T00:44:50.426201+0000","last_active":"2024-02-13T00:45:33.416795+0000","last_peered":"2024-02-13T00:45:33.416795+0000","last_clean":"2024-02-13T00:45:33.416795+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:45:33.416795+0000","last_undegraded":"2024-02-13T00:45:33.416795+0000","last_fullsized":"2024-02-13T00:45:33.416795+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:20.481083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:44:20.481083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":848,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-02-13T00:45:34.417555+0000","last_change":"2024-02-13T00:44:55.017869+0000","last_active":"2024-02-13T00:45:34.417555+0000","last_peered":"2024-02-13T00:45:34.417555+0000","last_clean":"2024-02-13T00:45:34.417555+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:45:34.417555+0000","last_undegraded":"2024-02-13T00:45:34.417555+0000","last_fullsized":"2024-02-13T00:45:34.417555+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-13T00:41:35.787179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:35.787179+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":461,"num_read_kb":456,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":83,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422815+0000","last_change":"2024-02-13T00:45:15.468925+0000","last_active":"2024-02-13T00:45:33.422815+0000","last_peered":"2024-02-13T00:45:33.422815+0000","last_clean":"2024-02-13T00:45:33.422815+0000","last_became_active":"2024-02-13T00:45:15.468176+0000","last_became_peered":"2024-02-13T00:45:15.468176+0000","last_unstale":"2024-02-13T00:45:33.422815+0000","last_undegraded":"2024-02-13T00:45:33.422815+0000","last_fullsized":"2024-02-13T00:45:33.422815+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":138,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422225+0000","last_change":"2024-02-13T00:45:15.436179+0000","last_active":"2024-02-13T00:45:33.422225+0000","last_peered":"2024-02-13T00:45:33.422225+0000","last_clean":"2024-02-13T00:45:33.422225+0000","last_became_active":"2024-02-13T00:45:15.435588+0000","last_became_peered":"2024-02-13T00:45:15.435588+0000","last_unstale":"2024-02-13T00:45:33.422225+0000","last_undegraded":"2024-02-13T00:45:33.422225+0000","last_fullsized":"2024-02-13T00:45:33.422225+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":43,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422410+0000","last_change":"2024-02-13T00:45:31.183763+0000","last_active":"2024-02-13T00:45:33.422410+0000","last_peered":"2024-02-13T00:45:33.422410+0000","last_clean":"2024-02-13T00:45:33.422410+0000","last_became_active":"2024-02-13T00:45:06.421025+0000","last_became_peered":"2024-02-13T00:45:06.421025+0000","last_unstale":"2024-02-13T00:45:33.422410+0000","last_undegraded":"2024-02-13T00:45:33.422410+0000","last_fullsized":"2024-02-13T00:45:33.422410+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.183682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.183682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"363'52","reported_seq":101,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:34.306495+0000","last_change":"2024-02-13T00:45:17.213174+0000","last_active":"2024-02-13T00:45:34.306495+0000","last_peered":"2024-02-13T00:45:34.306495+0000","last_clean":"2024-02-13T00:45:34.306495+0000","last_became_active":"2024-02-13T00:45:15.461880+0000","last_became_peered":"2024-02-13T00:45:15.461880+0000","last_unstale":"2024-02-13T00:45:34.306495+0000","last_undegraded":"2024-02-13T00:45:34.306495+0000","last_fullsized":"2024-02-13T00:45:34.306495+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'48","last_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:43:48.212428+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":363,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419952+0000","last_change":"2024-02-13T00:44:39.503774+0000","last_active":"2024-02-13T00:45:33.419952+0000","last_peered":"2024-02-13T00:45:33.419952+0000","last_clean":"2024-02-13T00:45:33.419952+0000","last_became_active":"2024-02-13T00:44:39.503527+0000","last_became_peered":"2024-02-13T00:44:39.503527+0000","last_unstale":"2024-02-13T00:45:33.419952+0000","last_undegraded":"2024-02-13T00:45:33.419952+0000","last_fullsized":"2024-02-13T00:45:33.419952+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.045912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.045912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422704+0000","last_change":"2024-02-13T00:45:15.442051+0000","last_active":"2024-02-13T00:45:33.422704+0000","last_peered":"2024-02-13T00:45:33.422704+0000","last_clean":"2024-02-13T00:45:33.422704+0000","last_became_active":"2024-02-13T00:45:15.441698+0000","last_became_peered":"2024-02-13T00:45:15.441698+0000","last_unstale":"2024-02-13T00:45:33.422704+0000","last_undegraded":"2024-02-13T00:45:33.422704+0000","last_fullsized":"2024-02-13T00:45:33.422704+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421023+0000","last_change":"2024-02-13T00:44:50.439369+0000","last_active":"2024-02-13T00:45:33.421023+0000","last_peered":"2024-02-13T00:45:33.421023+0000","last_clean":"2024-02-13T00:45:33.421023+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:45:33.421023+0000","last_undegraded":"2024-02-13T00:45:33.421023+0000","last_fullsized":"2024-02-13T00:45:33.421023+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:19.407011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:19.407011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"324'13","reported_seq":480,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:34.306422+0000","last_change":"2024-02-13T00:44:39.605948+0000","last_active":"2024-02-13T00:45:34.306422+0000","last_peered":"2024-02-13T00:45:34.306422+0000","last_clean":"2024-02-13T00:45:34.306422+0000","last_became_active":"2024-02-13T00:44:39.493784+0000","last_became_peered":"2024-02-13T00:44:39.493784+0000","last_unstale":"2024-02-13T00:45:34.306422+0000","last_undegraded":"2024-02-13T00:45:34.306422+0000","last_fullsized":"2024-02-13T00:45:34.306422+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'11","last_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:43:02.423358+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":68,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422220+0000","last_change":"2024-02-13T00:44:52.062902+0000","last_active":"2024-02-13T00:45:33.422220+0000","last_peered":"2024-02-13T00:45:33.422220+0000","last_clean":"2024-02-13T00:45:33.422220+0000","last_became_active":"2024-02-13T00:44:50.420104+0000","last_became_peered":"2024-02-13T00:44:50.420104+0000","last_unstale":"2024-02-13T00:45:33.422220+0000","last_undegraded":"2024-02-13T00:45:33.422220+0000","last_fullsized":"2024-02-13T00:45:33.422220+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:52.517392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.517392+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":354,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422318+0000","last_change":"2024-02-13T00:45:15.423417+0000","last_active":"2024-02-13T00:45:33.422318+0000","last_peered":"2024-02-13T00:45:33.422318+0000","last_clean":"2024-02-13T00:45:33.422318+0000","last_became_active":"2024-02-13T00:45:15.422546+0000","last_became_peered":"2024-02-13T00:45:15.422546+0000","last_unstale":"2024-02-13T00:45:33.422318+0000","last_undegraded":"2024-02-13T00:45:33.422318+0000","last_fullsized":"2024-02-13T00:45:33.422318+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:41.223028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:43:41.223028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":361,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416289+0000","last_change":"2024-02-13T00:44:50.426865+0000","last_active":"2024-02-13T00:45:33.416289+0000","last_peered":"2024-02-13T00:45:33.416289+0000","last_clean":"2024-02-13T00:45:33.416289+0000","last_became_active":"2024-02-13T00:44:50.426131+0000","last_became_peered":"2024-02-13T00:44:50.426131+0000","last_unstale":"2024-02-13T00:45:33.416289+0000","last_undegraded":"2024-02-13T00:45:33.416289+0000","last_fullsized":"2024-02-13T00:45:33.416289+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":303,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422140+0000","last_change":"2024-02-13T00:45:15.445626+0000","last_active":"2024-02-13T00:45:33.422140+0000","last_peered":"2024-02-13T00:45:33.422140+0000","last_clean":"2024-02-13T00:45:33.422140+0000","last_became_active":"2024-02-13T00:45:15.445117+0000","last_became_peered":"2024-02-13T00:45:15.445117+0000","last_unstale":"2024-02-13T00:45:33.422140+0000","last_undegraded":"2024-02-13T00:45:33.422140+0000","last_fullsized":"2024-02-13T00:45:33.422140+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.171185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":56,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416587+0000","last_change":"2024-02-13T00:44:50.427045+0000","last_active":"2024-02-13T00:45:33.416587+0000","last_peered":"2024-02-13T00:45:33.416587+0000","last_clean":"2024-02-13T00:45:33.416587+0000","last_became_active":"2024-02-13T00:44:50.426549+0000","last_became_peered":"2024-02-13T00:44:50.426549+0000","last_unstale":"2024-02-13T00:45:33.416587+0000","last_undegraded":"2024-02-13T00:45:33.416587+0000","last_fullsized":"2024-02-13T00:45:33.416587+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:43.426982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:43.426982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":415,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422050+0000","last_change":"2024-02-13T00:44:41.502069+0000","last_active":"2024-02-13T00:45:33.422050+0000","last_peered":"2024-02-13T00:45:33.422050+0000","last_clean":"2024-02-13T00:45:33.422050+0000","last_became_active":"2024-02-13T00:44:39.501886+0000","last_became_peered":"2024-02-13T00:44:39.501886+0000","last_unstale":"2024-02-13T00:45:33.422050+0000","last_undegraded":"2024-02-13T00:45:33.422050+0000","last_fullsized":"2024-02-13T00:45:33.422050+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'24","last_scrub_stamp":"2024-02-13T00:43:30.413372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.413372+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":36,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":329,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418361+0000","last_change":"2024-02-13T00:45:15.874072+0000","last_active":"2024-02-13T00:45:33.418361+0000","last_peered":"2024-02-13T00:45:33.418361+0000","last_clean":"2024-02-13T00:45:33.418361+0000","last_became_active":"2024-02-13T00:45:15.448935+0000","last_became_peered":"2024-02-13T00:45:15.448935+0000","last_unstale":"2024-02-13T00:45:33.418361+0000","last_undegraded":"2024-02-13T00:45:33.418361+0000","last_fullsized":"2024-02-13T00:45:33.418361+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:43:35.309308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:35.309308+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":149,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423106+0000","last_change":"2024-02-13T00:45:15.435701+0000","last_active":"2024-02-13T00:45:33.423106+0000","last_peered":"2024-02-13T00:45:33.423106+0000","last_clean":"2024-02-13T00:45:33.423106+0000","last_became_active":"2024-02-13T00:45:15.435404+0000","last_became_peered":"2024-02-13T00:45:15.435404+0000","last_unstale":"2024-02-13T00:45:33.423106+0000","last_undegraded":"2024-02-13T00:45:33.423106+0000","last_fullsized":"2024-02-13T00:45:33.423106+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:41:45.660653+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":295,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418886+0000","last_change":"2024-02-13T00:45:15.460066+0000","last_active":"2024-02-13T00:45:33.418886+0000","last_peered":"2024-02-13T00:45:33.418886+0000","last_clean":"2024-02-13T00:45:33.418886+0000","last_became_active":"2024-02-13T00:45:15.459235+0000","last_became_peered":"2024-02-13T00:45:15.459235+0000","last_unstale":"2024-02-13T00:45:33.418886+0000","last_undegraded":"2024-02-13T00:45:33.418886+0000","last_fullsized":"2024-02-13T00:45:33.418886+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.110742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.110742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":458,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418891+0000","last_change":"2024-02-13T00:45:17.038411+0000","last_active":"2024-02-13T00:45:33.418891+0000","last_peered":"2024-02-13T00:45:33.418891+0000","last_clean":"2024-02-13T00:45:33.418891+0000","last_became_active":"2024-02-13T00:45:15.459710+0000","last_became_peered":"2024-02-13T00:45:15.459710+0000","last_unstale":"2024-02-13T00:45:33.418891+0000","last_undegraded":"2024-02-13T00:45:33.418891+0000","last_fullsized":"2024-02-13T00:45:33.418891+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:43:30.365536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.365536+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":56,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422340+0000","last_change":"2024-02-13T00:44:50.442865+0000","last_active":"2024-02-13T00:45:33.422340+0000","last_peered":"2024-02-13T00:45:33.422340+0000","last_clean":"2024-02-13T00:45:33.422340+0000","last_became_active":"2024-02-13T00:44:50.441300+0000","last_became_peered":"2024-02-13T00:44:50.441300+0000","last_unstale":"2024-02-13T00:45:33.422340+0000","last_undegraded":"2024-02-13T00:45:33.422340+0000","last_fullsized":"2024-02-13T00:45:33.422340+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:49.468410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:49.468410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":140,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424599+0000","last_change":"2024-02-13T00:45:15.416946+0000","last_active":"2024-02-13T00:45:33.424599+0000","last_peered":"2024-02-13T00:45:33.424599+0000","last_clean":"2024-02-13T00:45:33.424599+0000","last_became_active":"2024-02-13T00:45:15.416364+0000","last_became_peered":"2024-02-13T00:45:15.416364+0000","last_unstale":"2024-02-13T00:45:33.424599+0000","last_undegraded":"2024-02-13T00:45:33.424599+0000","last_fullsized":"2024-02-13T00:45:33.424599+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.458752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.458752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":68,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417633+0000","last_change":"2024-02-13T00:45:26.651154+0000","last_active":"2024-02-13T00:45:33.417633+0000","last_peered":"2024-02-13T00:45:33.417633+0000","last_clean":"2024-02-13T00:45:33.417633+0000","last_became_active":"2024-02-13T00:45:15.461515+0000","last_became_peered":"2024-02-13T00:45:15.461515+0000","last_unstale":"2024-02-13T00:45:33.417633+0000","last_undegraded":"2024-02-13T00:45:33.417633+0000","last_fullsized":"2024-02-13T00:45:33.417633+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.651091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.651091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422685+0000","last_change":"2024-02-13T00:44:50.441196+0000","last_active":"2024-02-13T00:45:33.422685+0000","last_peered":"2024-02-13T00:45:33.422685+0000","last_clean":"2024-02-13T00:45:33.422685+0000","last_became_active":"2024-02-13T00:44:50.438274+0000","last_became_peered":"2024-02-13T00:44:50.438274+0000","last_unstale":"2024-02-13T00:45:33.422685+0000","last_undegraded":"2024-02-13T00:45:33.422685+0000","last_fullsized":"2024-02-13T00:45:33.422685+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.510219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.510219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":216,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424840+0000","last_change":"2024-02-13T00:45:18.331910+0000","last_active":"2024-02-13T00:45:33.424840+0000","last_peered":"2024-02-13T00:45:33.424840+0000","last_clean":"2024-02-13T00:45:33.424840+0000","last_became_active":"2024-02-13T00:45:15.436020+0000","last_became_peered":"2024-02-13T00:45:15.436020+0000","last_unstale":"2024-02-13T00:45:33.424840+0000","last_undegraded":"2024-02-13T00:45:33.424840+0000","last_fullsized":"2024-02-13T00:45:33.424840+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:41:42.695521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:42.695521+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":70,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419918+0000","last_change":"2024-02-13T00:45:00.311635+0000","last_active":"2024-02-13T00:45:33.419918+0000","last_peered":"2024-02-13T00:45:33.419918+0000","last_clean":"2024-02-13T00:45:33.419918+0000","last_became_active":"2024-02-13T00:44:39.512402+0000","last_became_peered":"2024-02-13T00:44:39.512402+0000","last_unstale":"2024-02-13T00:45:33.419918+0000","last_undegraded":"2024-02-13T00:45:33.419918+0000","last_fullsized":"2024-02-13T00:45:33.419918+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:00.311558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:45:00.311558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":359,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424375+0000","last_change":"2024-02-13T00:44:39.493677+0000","last_active":"2024-02-13T00:45:33.424375+0000","last_peered":"2024-02-13T00:45:33.424375+0000","last_clean":"2024-02-13T00:45:33.424375+0000","last_became_active":"2024-02-13T00:44:39.493304+0000","last_became_peered":"2024-02-13T00:44:39.493304+0000","last_unstale":"2024-02-13T00:45:33.424375+0000","last_undegraded":"2024-02-13T00:45:33.424375+0000","last_fullsized":"2024-02-13T00:45:33.424375+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.106639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.106639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422906+0000","last_change":"2024-02-13T00:45:15.469051+0000","last_active":"2024-02-13T00:45:33.422906+0000","last_peered":"2024-02-13T00:45:33.422906+0000","last_clean":"2024-02-13T00:45:33.422906+0000","last_became_active":"2024-02-13T00:45:15.468623+0000","last_became_peered":"2024-02-13T00:45:15.468623+0000","last_unstale":"2024-02-13T00:45:33.422906+0000","last_undegraded":"2024-02-13T00:45:33.422906+0000","last_fullsized":"2024-02-13T00:45:33.422906+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.042291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.042291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":145,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-02-13T00:45:34.523653+0000","last_change":"2024-02-13T00:45:34.523653+0000","last_active":"2024-02-13T00:45:34.523653+0000","last_peered":"2024-02-13T00:45:34.523653+0000","last_clean":"2024-02-13T00:45:34.523653+0000","last_became_active":"2024-02-13T00:45:15.461606+0000","last_became_peered":"2024-02-13T00:45:15.461606+0000","last_unstale":"2024-02-13T00:45:34.523653+0000","last_undegraded":"2024-02-13T00:45:34.523653+0000","last_fullsized":"2024-02-13T00:45:34.523653+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:34.523594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.523594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":268,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422428+0000","last_change":"2024-02-13T00:45:18.381054+0000","last_active":"2024-02-13T00:45:33.422428+0000","last_peered":"2024-02-13T00:45:33.422428+0000","last_clean":"2024-02-13T00:45:33.422428+0000","last_became_active":"2024-02-13T00:45:15.447151+0000","last_became_peered":"2024-02-13T00:45:15.447151+0000","last_unstale":"2024-02-13T00:45:33.422428+0000","last_undegraded":"2024-02-13T00:45:33.422428+0000","last_fullsized":"2024-02-13T00:45:33.422428+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:42:48.774168+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.774168+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":363,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419893+0000","last_change":"2024-02-13T00:45:15.460882+0000","last_active":"2024-02-13T00:45:33.419893+0000","last_peered":"2024-02-13T00:45:33.419893+0000","last_clean":"2024-02-13T00:45:33.419893+0000","last_became_active":"2024-02-13T00:45:15.459707+0000","last_became_peered":"2024-02-13T00:45:15.459707+0000","last_unstale":"2024-02-13T00:45:33.419893+0000","last_undegraded":"2024-02-13T00:45:33.419893+0000","last_fullsized":"2024-02-13T00:45:33.419893+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.230330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.230330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"64'8","reported_seq":439,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:34.306351+0000","last_change":"2024-02-13T00:41:37.453007+0000","last_active":"2024-02-13T00:45:34.306351+0000","last_peered":"2024-02-13T00:45:34.306351+0000","last_clean":"2024-02-13T00:45:34.306351+0000","last_became_active":"2024-02-13T00:37:45.246757+0000","last_became_peered":"2024-02-13T00:37:45.246757+0000","last_unstale":"2024-02-13T00:45:34.306351+0000","last_undegraded":"2024-02-13T00:45:34.306351+0000","last_fullsized":"2024-02-13T00:45:34.306351+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:37.452882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:37.452882+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":352,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418595+0000","last_change":"2024-02-13T00:45:15.450249+0000","last_active":"2024-02-13T00:45:33.418595+0000","last_peered":"2024-02-13T00:45:33.418595+0000","last_clean":"2024-02-13T00:45:33.418595+0000","last_became_active":"2024-02-13T00:45:15.449361+0000","last_became_peered":"2024-02-13T00:45:15.449361+0000","last_unstale":"2024-02-13T00:45:33.418595+0000","last_undegraded":"2024-02-13T00:45:33.418595+0000","last_fullsized":"2024-02-13T00:45:33.418595+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:52.391906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.391906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":71,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423589+0000","last_change":"2024-02-13T00:45:15.459158+0000","last_active":"2024-02-13T00:45:33.423589+0000","last_peered":"2024-02-13T00:45:33.423589+0000","last_clean":"2024-02-13T00:45:33.423589+0000","last_became_active":"2024-02-13T00:45:15.458790+0000","last_became_peered":"2024-02-13T00:45:15.458790+0000","last_unstale":"2024-02-13T00:45:33.423589+0000","last_undegraded":"2024-02-13T00:45:33.423589+0000","last_fullsized":"2024-02-13T00:45:33.423589+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:16.499780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:16.499780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":433,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423687+0000","last_change":"2024-02-13T00:45:20.251455+0000","last_active":"2024-02-13T00:45:33.423687+0000","last_peered":"2024-02-13T00:45:33.423687+0000","last_clean":"2024-02-13T00:45:33.423687+0000","last_became_active":"2024-02-13T00:45:15.472395+0000","last_became_peered":"2024-02-13T00:45:15.472395+0000","last_unstale":"2024-02-13T00:45:33.423687+0000","last_undegraded":"2024-02-13T00:45:33.423687+0000","last_fullsized":"2024-02-13T00:45:33.423687+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:36.477034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.477034+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":367,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417616+0000","last_change":"2024-02-13T00:45:15.459856+0000","last_active":"2024-02-13T00:45:33.417616+0000","last_peered":"2024-02-13T00:45:33.417616+0000","last_clean":"2024-02-13T00:45:33.417616+0000","last_became_active":"2024-02-13T00:45:15.459393+0000","last_became_peered":"2024-02-13T00:45:15.459393+0000","last_unstale":"2024-02-13T00:45:33.417616+0000","last_undegraded":"2024-02-13T00:45:33.417616+0000","last_fullsized":"2024-02-13T00:45:33.417616+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:40.335892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:40.335892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"363'14","reported_seq":150,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:34.306323+0000","last_change":"2024-02-13T00:45:17.693600+0000","last_active":"2024-02-13T00:45:34.306323+0000","last_peered":"2024-02-13T00:45:34.306323+0000","last_clean":"2024-02-13T00:45:34.306323+0000","last_became_active":"2024-02-13T00:45:15.423066+0000","last_became_peered":"2024-02-13T00:45:15.423066+0000","last_unstale":"2024-02-13T00:45:34.306323+0000","last_undegraded":"2024-02-13T00:45:34.306323+0000","last_fullsized":"2024-02-13T00:45:34.306323+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-02-13T00:41:32.715550+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.715550+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":59,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424564+0000","last_change":"2024-02-13T00:44:50.401576+0000","last_active":"2024-02-13T00:45:33.424564+0000","last_peered":"2024-02-13T00:45:33.424564+0000","last_clean":"2024-02-13T00:45:33.424564+0000","last_became_active":"2024-02-13T00:44:50.401229+0000","last_became_peered":"2024-02-13T00:44:50.401229+0000","last_unstale":"2024-02-13T00:45:33.424564+0000","last_undegraded":"2024-02-13T00:45:33.424564+0000","last_fullsized":"2024-02-13T00:45:33.424564+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.421001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.421001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":313,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.420408+0000","last_change":"2024-02-13T00:44:39.504470+0000","last_active":"2024-02-13T00:45:33.420408+0000","last_peered":"2024-02-13T00:45:33.420408+0000","last_clean":"2024-02-13T00:45:33.420408+0000","last_became_active":"2024-02-13T00:44:39.504102+0000","last_became_peered":"2024-02-13T00:44:39.504102+0000","last_unstale":"2024-02-13T00:45:33.420408+0000","last_undegraded":"2024-02-13T00:45:33.420408+0000","last_fullsized":"2024-02-13T00:45:33.420408+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:27.073034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:27.073034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":420,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422117+0000","last_change":"2024-02-13T00:44:54.926465+0000","last_active":"2024-02-13T00:45:33.422117+0000","last_peered":"2024-02-13T00:45:33.422117+0000","last_clean":"2024-02-13T00:45:33.422117+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:45:33.422117+0000","last_undegraded":"2024-02-13T00:45:33.422117+0000","last_fullsized":"2024-02-13T00:45:33.422117+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:43:40.244930+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:43:40.244930+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421850+0000","last_change":"2024-02-13T00:45:15.439966+0000","last_active":"2024-02-13T00:45:33.421850+0000","last_peered":"2024-02-13T00:45:33.421850+0000","last_clean":"2024-02-13T00:45:33.421850+0000","last_became_active":"2024-02-13T00:45:15.439560+0000","last_became_peered":"2024-02-13T00:45:15.439560+0000","last_unstale":"2024-02-13T00:45:33.421850+0000","last_undegraded":"2024-02-13T00:45:33.421850+0000","last_fullsized":"2024-02-13T00:45:33.421850+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.095888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":131,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417516+0000","last_change":"2024-02-13T00:43:35.462293+0000","last_active":"2024-02-13T00:45:33.417516+0000","last_peered":"2024-02-13T00:45:33.417516+0000","last_clean":"2024-02-13T00:45:33.417516+0000","last_became_active":"2024-02-13T00:43:35.461789+0000","last_became_peered":"2024-02-13T00:43:35.461789+0000","last_unstale":"2024-02-13T00:45:33.417516+0000","last_undegraded":"2024-02-13T00:45:33.417516+0000","last_fullsized":"2024-02-13T00:45:33.417516+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:43.716891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:29.229676+0000","last_clean_scrub_stamp":"2024-02-13T00:41:43.716891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":334,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416262+0000","last_change":"2024-02-13T00:42:56.413422+0000","last_active":"2024-02-13T00:45:33.416262+0000","last_peered":"2024-02-13T00:45:33.416262+0000","last_clean":"2024-02-13T00:45:33.416262+0000","last_became_active":"2024-02-13T00:37:47.259792+0000","last_became_peered":"2024-02-13T00:37:47.259792+0000","last_unstale":"2024-02-13T00:45:33.416262+0000","last_undegraded":"2024-02-13T00:45:33.416262+0000","last_fullsized":"2024-02-13T00:45:33.416262+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.413326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421870+0000","last_change":"2024-02-13T00:45:15.448518+0000","last_active":"2024-02-13T00:45:33.421870+0000","last_peered":"2024-02-13T00:45:33.421870+0000","last_clean":"2024-02-13T00:45:33.421870+0000","last_became_active":"2024-02-13T00:45:15.447051+0000","last_became_peered":"2024-02-13T00:45:15.447051+0000","last_unstale":"2024-02-13T00:45:33.421870+0000","last_undegraded":"2024-02-13T00:45:33.421870+0000","last_fullsized":"2024-02-13T00:45:33.421870+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":98,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421483+0000","last_change":"2024-02-13T00:44:52.751765+0000","last_active":"2024-02-13T00:45:33.421483+0000","last_peered":"2024-02-13T00:45:33.421483+0000","last_clean":"2024-02-13T00:45:33.421483+0000","last_became_active":"2024-02-13T00:44:50.435203+0000","last_became_peered":"2024-02-13T00:44:50.435203+0000","last_unstale":"2024-02-13T00:45:33.421483+0000","last_undegraded":"2024-02-13T00:45:33.421483+0000","last_fullsized":"2024-02-13T00:45:33.421483+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:42:51.361489+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:42:51.361489+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":340,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423494+0000","last_change":"2024-02-13T00:41:03.608458+0000","last_active":"2024-02-13T00:45:33.423494+0000","last_peered":"2024-02-13T00:45:33.423494+0000","last_clean":"2024-02-13T00:45:33.423494+0000","last_became_active":"2024-02-13T00:37:41.215531+0000","last_became_peered":"2024-02-13T00:37:41.215531+0000","last_unstale":"2024-02-13T00:45:33.423494+0000","last_undegraded":"2024-02-13T00:45:33.423494+0000","last_fullsized":"2024-02-13T00:45:33.423494+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:03.608379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:03.608379+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":359,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422168+0000","last_change":"2024-02-13T00:45:15.450693+0000","last_active":"2024-02-13T00:45:33.422168+0000","last_peered":"2024-02-13T00:45:33.422168+0000","last_clean":"2024-02-13T00:45:33.422168+0000","last_became_active":"2024-02-13T00:45:15.450084+0000","last_became_peered":"2024-02-13T00:45:15.450084+0000","last_unstale":"2024-02-13T00:45:33.422168+0000","last_undegraded":"2024-02-13T00:45:33.422168+0000","last_fullsized":"2024-02-13T00:45:33.422168+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:26.218807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:44:26.218807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":140,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416955+0000","last_change":"2024-02-13T00:44:50.416899+0000","last_active":"2024-02-13T00:45:33.416955+0000","last_peered":"2024-02-13T00:45:33.416955+0000","last_clean":"2024-02-13T00:45:33.416955+0000","last_became_active":"2024-02-13T00:44:50.415958+0000","last_became_peered":"2024-02-13T00:44:50.415958+0000","last_unstale":"2024-02-13T00:45:33.416955+0000","last_undegraded":"2024-02-13T00:45:33.416955+0000","last_fullsized":"2024-02-13T00:45:33.416955+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:50.828264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:50.828264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416337+0000","last_change":"2024-02-13T00:45:15.468716+0000","last_active":"2024-02-13T00:45:33.416337+0000","last_peered":"2024-02-13T00:45:33.416337+0000","last_clean":"2024-02-13T00:45:33.416337+0000","last_became_active":"2024-02-13T00:45:15.468433+0000","last_became_peered":"2024-02-13T00:45:15.468433+0000","last_unstale":"2024-02-13T00:45:33.416337+0000","last_undegraded":"2024-02-13T00:45:33.416337+0000","last_fullsized":"2024-02-13T00:45:33.416337+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.820327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.820327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":175,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423314+0000","last_change":"2024-02-13T00:44:40.371566+0000","last_active":"2024-02-13T00:45:33.423314+0000","last_peered":"2024-02-13T00:45:33.423314+0000","last_clean":"2024-02-13T00:45:33.423314+0000","last_became_active":"2024-02-13T00:44:39.501040+0000","last_became_peered":"2024-02-13T00:44:39.501040+0000","last_unstale":"2024-02-13T00:45:33.423314+0000","last_undegraded":"2024-02-13T00:45:33.423314+0000","last_fullsized":"2024-02-13T00:45:33.423314+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:44:32.180113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:44:32.180113+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":372,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424453+0000","last_change":"2024-02-13T00:44:50.423902+0000","last_active":"2024-02-13T00:45:33.424453+0000","last_peered":"2024-02-13T00:45:33.424453+0000","last_clean":"2024-02-13T00:45:33.424453+0000","last_became_active":"2024-02-13T00:44:50.423393+0000","last_became_peered":"2024-02-13T00:44:50.423393+0000","last_unstale":"2024-02-13T00:45:33.424453+0000","last_undegraded":"2024-02-13T00:45:33.424453+0000","last_fullsized":"2024-02-13T00:45:33.424453+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.499576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419707+0000","last_change":"2024-02-13T00:45:15.436871+0000","last_active":"2024-02-13T00:45:33.419707+0000","last_peered":"2024-02-13T00:45:33.419707+0000","last_clean":"2024-02-13T00:45:33.419707+0000","last_became_active":"2024-02-13T00:45:15.435820+0000","last_became_peered":"2024-02-13T00:45:15.435820+0000","last_unstale":"2024-02-13T00:45:33.419707+0000","last_undegraded":"2024-02-13T00:45:33.419707+0000","last_fullsized":"2024-02-13T00:45:33.419707+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.646321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.646321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":63,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421684+0000","last_change":"2024-02-13T00:45:15.445131+0000","last_active":"2024-02-13T00:45:33.421684+0000","last_peered":"2024-02-13T00:45:33.421684+0000","last_clean":"2024-02-13T00:45:33.421684+0000","last_became_active":"2024-02-13T00:45:15.444818+0000","last_became_peered":"2024-02-13T00:45:15.444818+0000","last_unstale":"2024-02-13T00:45:33.421684+0000","last_undegraded":"2024-02-13T00:45:33.421684+0000","last_fullsized":"2024-02-13T00:45:33.421684+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.479450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":304,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419392+0000","last_change":"2024-02-13T00:44:39.504976+0000","last_active":"2024-02-13T00:45:33.419392+0000","last_peered":"2024-02-13T00:45:33.419392+0000","last_clean":"2024-02-13T00:45:33.419392+0000","last_became_active":"2024-02-13T00:44:39.503358+0000","last_became_peered":"2024-02-13T00:44:39.503358+0000","last_unstale":"2024-02-13T00:45:33.419392+0000","last_undegraded":"2024-02-13T00:45:33.419392+0000","last_fullsized":"2024-02-13T00:45:33.419392+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.104906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.104906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":102,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419785+0000","last_change":"2024-02-13T00:45:17.868374+0000","last_active":"2024-02-13T00:45:33.419785+0000","last_peered":"2024-02-13T00:45:33.419785+0000","last_clean":"2024-02-13T00:45:33.419785+0000","last_became_active":"2024-02-13T00:45:15.440063+0000","last_became_peered":"2024-02-13T00:45:15.440063+0000","last_unstale":"2024-02-13T00:45:33.419785+0000","last_undegraded":"2024-02-13T00:45:33.419785+0000","last_fullsized":"2024-02-13T00:45:33.419785+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:41:50.675154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:50.675154+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423098+0000","last_change":"2024-02-13T00:44:50.441254+0000","last_active":"2024-02-13T00:45:33.423098+0000","last_peered":"2024-02-13T00:45:33.423098+0000","last_clean":"2024-02-13T00:45:33.423098+0000","last_became_active":"2024-02-13T00:44:50.440140+0000","last_became_peered":"2024-02-13T00:44:50.440140+0000","last_unstale":"2024-02-13T00:45:33.423098+0000","last_undegraded":"2024-02-13T00:45:33.423098+0000","last_fullsized":"2024-02-13T00:45:33.423098+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.457672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.457672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.415896+0000","last_change":"2024-02-13T00:45:15.439752+0000","last_active":"2024-02-13T00:45:33.415896+0000","last_peered":"2024-02-13T00:45:33.415896+0000","last_clean":"2024-02-13T00:45:33.415896+0000","last_became_active":"2024-02-13T00:45:15.437943+0000","last_became_peered":"2024-02-13T00:45:15.437943+0000","last_unstale":"2024-02-13T00:45:33.415896+0000","last_undegraded":"2024-02-13T00:45:33.415896+0000","last_fullsized":"2024-02-13T00:45:33.415896+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.284740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.284740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":281,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417431+0000","last_change":"2024-02-13T00:44:58.969806+0000","last_active":"2024-02-13T00:45:33.417431+0000","last_peered":"2024-02-13T00:45:33.417431+0000","last_clean":"2024-02-13T00:45:33.417431+0000","last_became_active":"2024-02-13T00:44:50.439512+0000","last_became_peered":"2024-02-13T00:44:50.439512+0000","last_unstale":"2024-02-13T00:45:33.417431+0000","last_undegraded":"2024-02-13T00:45:33.417431+0000","last_fullsized":"2024-02-13T00:45:33.417431+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:58.969692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:44:58.969692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":197,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422331+0000","last_change":"2024-02-13T00:45:15.431130+0000","last_active":"2024-02-13T00:45:33.422331+0000","last_peered":"2024-02-13T00:45:33.422331+0000","last_clean":"2024-02-13T00:45:33.422331+0000","last_became_active":"2024-02-13T00:45:15.430749+0000","last_became_peered":"2024-02-13T00:45:15.430749+0000","last_unstale":"2024-02-13T00:45:33.422331+0000","last_undegraded":"2024-02-13T00:45:33.422331+0000","last_fullsized":"2024-02-13T00:45:33.422331+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.605111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.605111+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":427,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419037+0000","last_change":"2024-02-13T00:45:17.210141+0000","last_active":"2024-02-13T00:45:33.419037+0000","last_peered":"2024-02-13T00:45:33.419037+0000","last_clean":"2024-02-13T00:45:33.419037+0000","last_became_active":"2024-02-13T00:45:15.459952+0000","last_became_peered":"2024-02-13T00:45:15.459952+0000","last_unstale":"2024-02-13T00:45:33.419037+0000","last_undegraded":"2024-02-13T00:45:33.419037+0000","last_fullsized":"2024-02-13T00:45:33.419037+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:43:07.393197+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.393197+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":349,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418707+0000","last_change":"2024-02-13T00:45:15.472213+0000","last_active":"2024-02-13T00:45:33.418707+0000","last_peered":"2024-02-13T00:45:33.418707+0000","last_clean":"2024-02-13T00:45:33.418707+0000","last_became_active":"2024-02-13T00:45:15.471558+0000","last_became_peered":"2024-02-13T00:45:15.471558+0000","last_unstale":"2024-02-13T00:45:33.418707+0000","last_undegraded":"2024-02-13T00:45:33.418707+0000","last_fullsized":"2024-02-13T00:45:33.418707+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:32.424844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.424844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"363'14","reported_seq":96,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:34.306316+0000","last_change":"2024-02-13T00:45:18.328879+0000","last_active":"2024-02-13T00:45:34.306316+0000","last_peered":"2024-02-13T00:45:34.306316+0000","last_clean":"2024-02-13T00:45:34.306316+0000","last_became_active":"2024-02-13T00:45:15.455825+0000","last_became_peered":"2024-02-13T00:45:15.455825+0000","last_unstale":"2024-02-13T00:45:34.306316+0000","last_undegraded":"2024-02-13T00:45:34.306316+0000","last_fullsized":"2024-02-13T00:45:34.306316+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'10","last_scrub_stamp":"2024-02-13T00:43:05.604461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.604461+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":213,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423042+0000","last_change":"2024-02-13T00:45:20.332741+0000","last_active":"2024-02-13T00:45:33.423042+0000","last_peered":"2024-02-13T00:45:33.423042+0000","last_clean":"2024-02-13T00:45:33.423042+0000","last_became_active":"2024-02-13T00:45:15.469508+0000","last_became_peered":"2024-02-13T00:45:15.469508+0000","last_unstale":"2024-02-13T00:45:33.423042+0000","last_undegraded":"2024-02-13T00:45:33.423042+0000","last_fullsized":"2024-02-13T00:45:33.423042+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:42:55.481128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:55.481128+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":307,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424746+0000","last_change":"2024-02-13T00:45:15.436295+0000","last_active":"2024-02-13T00:45:33.424746+0000","last_peered":"2024-02-13T00:45:33.424746+0000","last_clean":"2024-02-13T00:45:33.424746+0000","last_became_active":"2024-02-13T00:45:15.435550+0000","last_became_peered":"2024-02-13T00:45:15.435550+0000","last_unstale":"2024-02-13T00:45:33.424746+0000","last_undegraded":"2024-02-13T00:45:33.424746+0000","last_fullsized":"2024-02-13T00:45:33.424746+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":79,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422866+0000","last_change":"2024-02-13T00:44:45.969595+0000","last_active":"2024-02-13T00:45:33.422866+0000","last_peered":"2024-02-13T00:45:33.422866+0000","last_clean":"2024-02-13T00:45:33.422866+0000","last_became_active":"2024-02-13T00:44:39.524490+0000","last_became_peered":"2024-02-13T00:44:39.524490+0000","last_unstale":"2024-02-13T00:45:33.422866+0000","last_undegraded":"2024-02-13T00:45:33.422866+0000","last_fullsized":"2024-02-13T00:45:33.422866+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:42:59.604002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:59.604002+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":83,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419875+0000","last_change":"2024-02-13T00:45:15.437121+0000","last_active":"2024-02-13T00:45:33.419875+0000","last_peered":"2024-02-13T00:45:33.419875+0000","last_clean":"2024-02-13T00:45:33.419875+0000","last_became_active":"2024-02-13T00:45:15.436239+0000","last_became_peered":"2024-02-13T00:45:15.436239+0000","last_unstale":"2024-02-13T00:45:33.419875+0000","last_undegraded":"2024-02-13T00:45:33.419875+0000","last_fullsized":"2024-02-13T00:45:33.419875+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:46.698599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:46.698599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":356,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419025+0000","last_change":"2024-02-13T00:45:15.423191+0000","last_active":"2024-02-13T00:45:33.419025+0000","last_peered":"2024-02-13T00:45:33.419025+0000","last_clean":"2024-02-13T00:45:33.419025+0000","last_became_active":"2024-02-13T00:45:15.422870+0000","last_became_peered":"2024-02-13T00:45:15.422870+0000","last_unstale":"2024-02-13T00:45:33.419025+0000","last_undegraded":"2024-02-13T00:45:33.419025+0000","last_fullsized":"2024-02-13T00:45:33.419025+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.596334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.596334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":339,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423394+0000","last_change":"2024-02-13T00:43:49.245309+0000","last_active":"2024-02-13T00:45:33.423394+0000","last_peered":"2024-02-13T00:45:33.423394+0000","last_clean":"2024-02-13T00:45:33.423394+0000","last_became_active":"2024-02-13T00:37:47.260182+0000","last_became_peered":"2024-02-13T00:37:47.260182+0000","last_unstale":"2024-02-13T00:45:33.423394+0000","last_undegraded":"2024-02-13T00:45:33.423394+0000","last_fullsized":"2024-02-13T00:45:33.423394+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.245237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.245237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422046+0000","last_change":"2024-02-13T00:44:50.436204+0000","last_active":"2024-02-13T00:45:33.422046+0000","last_peered":"2024-02-13T00:45:33.422046+0000","last_clean":"2024-02-13T00:45:33.422046+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:45:33.422046+0000","last_undegraded":"2024-02-13T00:45:33.422046+0000","last_fullsized":"2024-02-13T00:45:33.422046+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:41:00.445926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":109,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422418+0000","last_change":"2024-02-13T00:44:55.939050+0000","last_active":"2024-02-13T00:45:33.422418+0000","last_peered":"2024-02-13T00:45:33.422418+0000","last_clean":"2024-02-13T00:45:33.422418+0000","last_became_active":"2024-02-13T00:44:50.442894+0000","last_became_peered":"2024-02-13T00:44:50.442894+0000","last_unstale":"2024-02-13T00:45:33.422418+0000","last_undegraded":"2024-02-13T00:45:33.422418+0000","last_fullsized":"2024-02-13T00:45:33.422418+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:43:30.578602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.578602+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.420821+0000","last_change":"2024-02-13T00:45:15.446020+0000","last_active":"2024-02-13T00:45:33.420821+0000","last_peered":"2024-02-13T00:45:33.420821+0000","last_clean":"2024-02-13T00:45:33.420821+0000","last_became_active":"2024-02-13T00:45:15.445403+0000","last_became_peered":"2024-02-13T00:45:15.445403+0000","last_unstale":"2024-02-13T00:45:33.420821+0000","last_undegraded":"2024-02-13T00:45:33.420821+0000","last_fullsized":"2024-02-13T00:45:33.420821+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:06.651077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:06.651077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":336,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424387+0000","last_change":"2024-02-13T00:41:56.204734+0000","last_active":"2024-02-13T00:45:33.424387+0000","last_peered":"2024-02-13T00:45:33.424387+0000","last_clean":"2024-02-13T00:45:33.424387+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:45:33.424387+0000","last_undegraded":"2024-02-13T00:45:33.424387+0000","last_fullsized":"2024-02-13T00:45:33.424387+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:56.204671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.204671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":65,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422620+0000","last_change":"2024-02-13T00:44:39.508525+0000","last_active":"2024-02-13T00:45:33.422620+0000","last_peered":"2024-02-13T00:45:33.422620+0000","last_clean":"2024-02-13T00:45:33.422620+0000","last_became_active":"2024-02-13T00:44:39.508082+0000","last_became_peered":"2024-02-13T00:44:39.508082+0000","last_unstale":"2024-02-13T00:45:33.422620+0000","last_undegraded":"2024-02-13T00:45:33.422620+0000","last_fullsized":"2024-02-13T00:45:33.422620+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.410777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.410777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":47,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423292+0000","last_change":"2024-02-13T00:45:15.465740+0000","last_active":"2024-02-13T00:45:33.423292+0000","last_peered":"2024-02-13T00:45:33.423292+0000","last_clean":"2024-02-13T00:45:33.423292+0000","last_became_active":"2024-02-13T00:45:15.459924+0000","last_became_peered":"2024-02-13T00:45:15.459924+0000","last_unstale":"2024-02-13T00:45:33.423292+0000","last_undegraded":"2024-02-13T00:45:33.423292+0000","last_fullsized":"2024-02-13T00:45:33.423292+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.554065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.554065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":219,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.424366+0000","last_change":"2024-02-13T00:44:44.964075+0000","last_active":"2024-02-13T00:45:33.424366+0000","last_peered":"2024-02-13T00:45:33.424366+0000","last_clean":"2024-02-13T00:45:33.424366+0000","last_became_active":"2024-02-13T00:44:39.504670+0000","last_became_peered":"2024-02-13T00:44:39.504670+0000","last_unstale":"2024-02-13T00:45:33.424366+0000","last_undegraded":"2024-02-13T00:45:33.424366+0000","last_fullsized":"2024-02-13T00:45:33.424366+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:41:41.686058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.686058+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421682+0000","last_change":"2024-02-13T00:45:15.455574+0000","last_active":"2024-02-13T00:45:33.421682+0000","last_peered":"2024-02-13T00:45:33.421682+0000","last_clean":"2024-02-13T00:45:33.421682+0000","last_became_active":"2024-02-13T00:45:15.455234+0000","last_became_peered":"2024-02-13T00:45:15.455234+0000","last_unstale":"2024-02-13T00:45:33.421682+0000","last_undegraded":"2024-02-13T00:45:33.421682+0000","last_fullsized":"2024-02-13T00:45:33.421682+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:28.101254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:28.101254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":63,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422030+0000","last_change":"2024-02-13T00:45:15.435977+0000","last_active":"2024-02-13T00:45:33.422030+0000","last_peered":"2024-02-13T00:45:33.422030+0000","last_clean":"2024-02-13T00:45:33.422030+0000","last_became_active":"2024-02-13T00:45:15.435206+0000","last_became_peered":"2024-02-13T00:45:15.435206+0000","last_unstale":"2024-02-13T00:45:33.422030+0000","last_undegraded":"2024-02-13T00:45:33.422030+0000","last_fullsized":"2024-02-13T00:45:33.422030+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:57.802417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.802417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":65,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422507+0000","last_change":"2024-02-13T00:44:39.526973+0000","last_active":"2024-02-13T00:45:33.422507+0000","last_peered":"2024-02-13T00:45:33.422507+0000","last_clean":"2024-02-13T00:45:33.422507+0000","last_became_active":"2024-02-13T00:44:39.526636+0000","last_became_peered":"2024-02-13T00:44:39.526636+0000","last_unstale":"2024-02-13T00:45:33.422507+0000","last_undegraded":"2024-02-13T00:45:33.422507+0000","last_fullsized":"2024-02-13T00:45:33.422507+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:57.950483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:57.950483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":313,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417531+0000","last_change":"2024-02-13T00:45:31.621294+0000","last_active":"2024-02-13T00:45:33.417531+0000","last_peered":"2024-02-13T00:45:33.417531+0000","last_clean":"2024-02-13T00:45:33.417531+0000","last_became_active":"2024-02-13T00:44:50.415557+0000","last_became_peered":"2024-02-13T00:44:50.415557+0000","last_unstale":"2024-02-13T00:45:33.417531+0000","last_undegraded":"2024-02-13T00:45:33.417531+0000","last_fullsized":"2024-02-13T00:45:33.417531+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.621170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.621170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":213,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417523+0000","last_change":"2024-02-13T00:44:50.589140+0000","last_active":"2024-02-13T00:45:33.417523+0000","last_peered":"2024-02-13T00:45:33.417523+0000","last_clean":"2024-02-13T00:45:33.417523+0000","last_became_active":"2024-02-13T00:44:50.415894+0000","last_became_peered":"2024-02-13T00:44:50.415894+0000","last_unstale":"2024-02-13T00:45:33.417523+0000","last_undegraded":"2024-02-13T00:45:33.417523+0000","last_fullsized":"2024-02-13T00:45:33.417523+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:41:31.648217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:31.648217+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":196,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422647+0000","last_change":"2024-02-13T00:45:15.450605+0000","last_active":"2024-02-13T00:45:33.422647+0000","last_peered":"2024-02-13T00:45:33.422647+0000","last_clean":"2024-02-13T00:45:33.422647+0000","last_became_active":"2024-02-13T00:45:15.450251+0000","last_became_peered":"2024-02-13T00:45:15.450251+0000","last_unstale":"2024-02-13T00:45:33.422647+0000","last_undegraded":"2024-02-13T00:45:33.422647+0000","last_fullsized":"2024-02-13T00:45:33.422647+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:03.665973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:03.665973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":350,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421192+0000","last_change":"2024-02-13T00:45:15.460138+0000","last_active":"2024-02-13T00:45:33.421192+0000","last_peered":"2024-02-13T00:45:33.421192+0000","last_clean":"2024-02-13T00:45:33.421192+0000","last_became_active":"2024-02-13T00:45:15.459788+0000","last_became_peered":"2024-02-13T00:45:15.459788+0000","last_unstale":"2024-02-13T00:45:33.421192+0000","last_undegraded":"2024-02-13T00:45:33.421192+0000","last_fullsized":"2024-02-13T00:45:33.421192+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.415414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.415414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":343,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416540+0000","last_change":"2024-02-13T00:45:15.420228+0000","last_active":"2024-02-13T00:45:33.416540+0000","last_peered":"2024-02-13T00:45:33.416540+0000","last_clean":"2024-02-13T00:45:33.416540+0000","last_became_active":"2024-02-13T00:45:15.419668+0000","last_became_peered":"2024-02-13T00:45:15.419668+0000","last_unstale":"2024-02-13T00:45:33.416540+0000","last_undegraded":"2024-02-13T00:45:33.416540+0000","last_fullsized":"2024-02-13T00:45:33.416540+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:21.648002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.648002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":295,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421431+0000","last_change":"2024-02-13T00:45:15.474301+0000","last_active":"2024-02-13T00:45:33.421431+0000","last_peered":"2024-02-13T00:45:33.421431+0000","last_clean":"2024-02-13T00:45:33.421431+0000","last_became_active":"2024-02-13T00:45:15.473618+0000","last_became_peered":"2024-02-13T00:45:15.473618+0000","last_unstale":"2024-02-13T00:45:33.421431+0000","last_undegraded":"2024-02-13T00:45:33.421431+0000","last_fullsized":"2024-02-13T00:45:33.421431+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:21.087199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:21.087199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"85'14","reported_seq":233,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422998+0000","last_change":"2024-02-13T00:45:15.701409+0000","last_active":"2024-02-13T00:45:33.422998+0000","last_peered":"2024-02-13T00:45:33.422998+0000","last_clean":"2024-02-13T00:45:33.422998+0000","last_became_active":"2024-02-13T00:45:15.419364+0000","last_became_peered":"2024-02-13T00:45:15.419364+0000","last_unstale":"2024-02-13T00:45:33.422998+0000","last_undegraded":"2024-02-13T00:45:33.422998+0000","last_fullsized":"2024-02-13T00:45:33.422998+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:41:47.734904+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422946+0000","last_change":"2024-02-13T00:45:15.469349+0000","last_active":"2024-02-13T00:45:33.422946+0000","last_peered":"2024-02-13T00:45:33.422946+0000","last_clean":"2024-02-13T00:45:33.422946+0000","last_became_active":"2024-02-13T00:45:15.469031+0000","last_became_peered":"2024-02-13T00:45:15.469031+0000","last_unstale":"2024-02-13T00:45:33.422946+0000","last_undegraded":"2024-02-13T00:45:33.422946+0000","last_fullsized":"2024-02-13T00:45:33.422946+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:21.630532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.630532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":181,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.421930+0000","last_change":"2024-02-13T00:45:15.416632+0000","last_active":"2024-02-13T00:45:33.421930+0000","last_peered":"2024-02-13T00:45:33.421930+0000","last_clean":"2024-02-13T00:45:33.421930+0000","last_became_active":"2024-02-13T00:45:15.416307+0000","last_became_peered":"2024-02-13T00:45:15.416307+0000","last_unstale":"2024-02-13T00:45:33.421930+0000","last_undegraded":"2024-02-13T00:45:33.421930+0000","last_fullsized":"2024-02-13T00:45:33.421930+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:48.629024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.629024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":357,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419262+0000","last_change":"2024-02-13T00:44:50.419677+0000","last_active":"2024-02-13T00:45:33.419262+0000","last_peered":"2024-02-13T00:45:33.419262+0000","last_clean":"2024-02-13T00:45:33.419262+0000","last_became_active":"2024-02-13T00:44:50.419353+0000","last_became_peered":"2024-02-13T00:44:50.419353+0000","last_unstale":"2024-02-13T00:45:33.419262+0000","last_undegraded":"2024-02-13T00:45:33.419262+0000","last_fullsized":"2024-02-13T00:45:33.419262+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:52.447698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.447698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":70,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416417+0000","last_change":"2024-02-13T00:45:30.186208+0000","last_active":"2024-02-13T00:45:33.416417+0000","last_peered":"2024-02-13T00:45:33.416417+0000","last_clean":"2024-02-13T00:45:33.416417+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:45:33.416417+0000","last_undegraded":"2024-02-13T00:45:33.416417+0000","last_fullsized":"2024-02-13T00:45:33.416417+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:30.186129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:30.186129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"90'20","reported_seq":98,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416510+0000","last_change":"2024-02-13T00:45:06.850147+0000","last_active":"2024-02-13T00:45:33.416510+0000","last_peered":"2024-02-13T00:45:33.416510+0000","last_clean":"2024-02-13T00:45:33.416510+0000","last_became_active":"2024-02-13T00:45:06.415860+0000","last_became_peered":"2024-02-13T00:45:06.415860+0000","last_unstale":"2024-02-13T00:45:33.416510+0000","last_undegraded":"2024-02-13T00:45:33.416510+0000","last_fullsized":"2024-02-13T00:45:33.416510+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:43:44.241358+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:43:44.241358+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":149,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418934+0000","last_change":"2024-02-13T00:45:15.460238+0000","last_active":"2024-02-13T00:45:33.418934+0000","last_peered":"2024-02-13T00:45:33.418934+0000","last_clean":"2024-02-13T00:45:33.418934+0000","last_became_active":"2024-02-13T00:45:15.459267+0000","last_became_peered":"2024-02-13T00:45:15.459267+0000","last_unstale":"2024-02-13T00:45:33.418934+0000","last_undegraded":"2024-02-13T00:45:33.418934+0000","last_fullsized":"2024-02-13T00:45:33.418934+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:18.464402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:42:18.464402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":345,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422391+0000","last_change":"2024-02-13T00:45:15.448461+0000","last_active":"2024-02-13T00:45:33.422391+0000","last_peered":"2024-02-13T00:45:33.422391+0000","last_clean":"2024-02-13T00:45:33.422391+0000","last_became_active":"2024-02-13T00:45:15.447981+0000","last_became_peered":"2024-02-13T00:45:15.447981+0000","last_unstale":"2024-02-13T00:45:33.422391+0000","last_undegraded":"2024-02-13T00:45:33.422391+0000","last_fullsized":"2024-02-13T00:45:33.422391+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.693214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.693214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":300,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.419802+0000","last_change":"2024-02-13T00:41:47.487953+0000","last_active":"2024-02-13T00:45:33.419802+0000","last_peered":"2024-02-13T00:45:33.419802+0000","last_clean":"2024-02-13T00:45:33.419802+0000","last_became_active":"2024-02-13T00:40:35.845664+0000","last_became_peered":"2024-02-13T00:40:35.845664+0000","last_unstale":"2024-02-13T00:45:33.419802+0000","last_undegraded":"2024-02-13T00:45:33.419802+0000","last_fullsized":"2024-02-13T00:45:33.419802+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:47.487860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:47.487860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":324,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.423183+0000","last_change":"2024-02-13T00:45:32.233265+0000","last_active":"2024-02-13T00:45:33.423183+0000","last_peered":"2024-02-13T00:45:33.423183+0000","last_clean":"2024-02-13T00:45:33.423183+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:45:33.423183+0000","last_undegraded":"2024-02-13T00:45:33.423183+0000","last_fullsized":"2024-02-13T00:45:33.423183+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:45:32.233197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.233197+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"86'50","reported_seq":206,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.417032+0000","last_change":"2024-02-13T00:44:50.598488+0000","last_active":"2024-02-13T00:45:33.417032+0000","last_peered":"2024-02-13T00:45:33.417032+0000","last_clean":"2024-02-13T00:45:33.417032+0000","last_became_active":"2024-02-13T00:44:50.425810+0000","last_became_peered":"2024-02-13T00:44:50.425810+0000","last_unstale":"2024-02-13T00:45:33.417032+0000","last_undegraded":"2024-02-13T00:45:33.417032+0000","last_fullsized":"2024-02-13T00:45:33.417032+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:42:17.454232+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.454232+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":349,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.418426+0000","last_change":"2024-02-13T00:45:15.434575+0000","last_active":"2024-02-13T00:45:33.418426+0000","last_peered":"2024-02-13T00:45:33.418426+0000","last_clean":"2024-02-13T00:45:33.418426+0000","last_became_active":"2024-02-13T00:45:15.434224+0000","last_became_peered":"2024-02-13T00:45:15.434224+0000","last_unstale":"2024-02-13T00:45:33.418426+0000","last_undegraded":"2024-02-13T00:45:33.418426+0000","last_fullsized":"2024-02-13T00:45:33.418426+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:43:03.464398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":65,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.416080+0000","last_change":"2024-02-13T00:44:39.510101+0000","last_active":"2024-02-13T00:45:33.416080+0000","last_peered":"2024-02-13T00:45:33.416080+0000","last_clean":"2024-02-13T00:45:33.416080+0000","last_became_active":"2024-02-13T00:44:39.508197+0000","last_became_peered":"2024-02-13T00:44:39.508197+0000","last_unstale":"2024-02-13T00:45:33.416080+0000","last_undegraded":"2024-02-13T00:45:33.416080+0000","last_fullsized":"2024-02-13T00:45:33.416080+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:46.418859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:41:46.418859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":74,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-02-13T00:45:33.422036+0000","last_change":"2024-02-13T00:45:15.435955+0000","last_active":"2024-02-13T00:45:33.422036+0000","last_peered":"2024-02-13T00:45:33.422036+0000","last_clean":"2024-02-13T00:45:33.422036+0000","last_became_active":"2024-02-13T00:45:15.435259+0000","last_became_peered":"2024-02-13T00:45:15.435259+0000","last_unstale":"2024-02-13T00:45:33.422036+0000","last_undegraded":"2024-02-13T00:45:33.422036+0000","last_fullsized":"2024-02-13T00:45:33.422036+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.953989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.953989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":204,"num_pg":32,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":96,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":5},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":4001420,"num_objects":148,"num_object_clones":2,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":103,"num_read_kb":7,"num_write":573,"num_write_kb":4474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":3483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13881344,"data_stored":12004453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1818,"ondisk_log_size":1818,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1063,"num_read_kb":788,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":7818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":11436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":461,"num_read_kb":456,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2039,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":227633266785,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":45,"seq":193273528423,"num_pgs":146,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311688,"kb_used_data":6264,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433464,"statfs":{"total":95995035648,"available":95675867136,"internally_reserved":0,"allocated":6414336,"data_stored":3533333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.628,"15min":0.56799999999999995},"min":{"1min":0.26000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.77500000000000002,"5min":8.1340000000000003,"15min":8.1340000000000003},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.86099999999999999,"15min":0.60099999999999998},"min":{"1min":0.23100000000000001,"5min":0.221,"15min":0.221},"max":{"1min":4.2910000000000004,"5min":11.367000000000001,"15min":11.367000000000001},"last":0.58599999999999997}]},{"osd":1,"last update":"Tue Feb 13 00:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":1.6639999999999999,"15min":0.89100000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.27200000000000002},"max":{"1min":4.5720000000000001,"5min":54.002000000000002,"15min":54.002000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":1.0389999999999999,"15min":0.68899999999999995},"min":{"1min":0.28799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":5.4989999999999997,"5min":12.403,"15min":12.403},"last":0.46200000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.1200000000000001,"15min":0.70599999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":2.8399999999999999,"5min":12.477,"15min":12.477},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":1.054,"15min":0.67700000000000005},"min":{"1min":0.34000000000000002,"5min":0.33900000000000002,"15min":0.32000000000000001},"max":{"1min":2.1659999999999999,"5min":11.702,"15min":11.702},"last":0.35599999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.95599999999999996,"15min":0.67700000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":5.0990000000000002,"5min":8.0519999999999996,"15min":8.0519999999999996},"last":0.379},{"interface":"front","average":{"1min":0.58199999999999996,"5min":1.232,"15min":0.77400000000000002},"min":{"1min":0.32600000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.8719999999999999,"5min":19.081,"15min":19.081},"last":0.54200000000000004}]},{"osd":4,"last update":"Tue Feb 13 00:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.41899999999999998,"15min":0.40899999999999997},"min":{"1min":0.249,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56100000000000005,"5min":0.71599999999999997,"15min":1.0649999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.42999999999999999,"15min":0.42899999999999999},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.58699999999999997,"5min":0.69799999999999995,"15min":0.83299999999999996},"last":0.59399999999999997}]},{"osd":5,"last update":"Tue Feb 13 00:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.44,"15min":0.41099999999999998},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60999999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48399999999999999,"15min":0.46500000000000002},"min":{"1min":0.30299999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57599999999999996,"5min":0.73899999999999999,"15min":0.86299999999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Tue Feb 13 00:44:44 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48699999999999999,"15min":0.498},"min":{"1min":0.34300000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65400000000000003,"5min":0.91600000000000004,"15min":1.173},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52400000000000002,"15min":0.54800000000000004},"min":{"1min":0.33300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":1.262},"last":0.627}]}]},{"osd":1,"up_from":13,"seq":55834574976,"num_pgs":159,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311624,"kb_used_data":10292,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93433528,"statfs":{"total":95995035648,"available":95675932672,"internally_reserved":0,"allocated":10539008,"data_stored":7541999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:45:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":2.1789999999999998,"15min":0.94499999999999995},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":1.1080000000000001,"5min":61.856000000000002,"15min":61.856000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":1.331,"15min":0.65800000000000003},"min":{"1min":0.23000000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.052,"5min":61.908999999999999,"15min":61.908999999999999},"last":0.38600000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:44:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":1.8260000000000001,"15min":0.85699999999999998},"min":{"1min":0.20699999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.90100000000000002,"5min":32.161000000000001,"15min":32.161000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":1.7210000000000001,"15min":0.83099999999999996},"min":{"1min":0.219,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.9350000000000001,"5min":29.693000000000001,"15min":29.693000000000001},"last":0.28799999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:45:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":3.0009999999999999,"15min":1.29},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.0900000000000001,"5min":131.23099999999999,"15min":131.23099999999999},"last":0.376},{"interface":"front","average":{"1min":0.46500000000000002,"5min":1.7549999999999999,"15min":0.88600000000000001},"min":{"1min":0.26800000000000002,"5min":0.26000000000000001,"15min":0.246},"max":{"1min":1.214,"5min":49.055,"15min":49.055},"last":0.39800000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:44:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.165,"15min":0.76800000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.80500000000000005,"5min":16.547000000000001,"15min":16.547000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.998,"15min":0.71099999999999997},"min":{"1min":0.27800000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.81299999999999994,"5min":14.731999999999999,"15min":14.731999999999999},"last":0.35099999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:44:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":1.5760000000000001,"15min":0.90500000000000003},"min":{"1min":0.38900000000000001,"5min":0.34000000000000002,"15min":0.33700000000000002},"max":{"1min":0.72999999999999998,"5min":26.561,"15min":26.561},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":1.139,"15min":0.76900000000000002},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.27,"5min":16.494,"15min":16.494},"last":0.45700000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:45:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":1.0049999999999999,"15min":0.72999999999999998},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76000000000000001,"5min":14.772,"15min":14.772},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":1.611,"15min":0.94299999999999995},"min":{"1min":0.41699999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.77900000000000003,"5min":26.600000000000001,"15min":26.600000000000001},"last":0.55500000000000005}]},{"osd":7,"last update":"Tue Feb 13 00:44:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":1.008,"15min":0.73399999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.79800000000000004,"5min":14.794,"15min":14.794},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.595,"15min":0.94699999999999995},"min":{"1min":0.41299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78100000000000003,"5min":26.693999999999999,"15min":26.693999999999999},"last":0.46300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738500,"num_pgs":141,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315672,"kb_used_data":10248,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93429480,"statfs":{"total":95995035648,"available":95671787520,"internally_reserved":0,"allocated":10493952,"data_stored":7541785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:45:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":1.169,"15min":0.61099999999999999},"min":{"1min":0.221,"5min":0.192,"15min":0.189},"max":{"1min":2.0379999999999998,"5min":30.184000000000001,"15min":30.184000000000001},"last":0.314},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.60599999999999998,"15min":0.439},"min":{"1min":0.217,"5min":0.187,"15min":0.17599999999999999},"max":{"1min":5.6959999999999997,"5min":5.6959999999999997,"15min":5.6959999999999997},"last":0.26100000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:44:53 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.68899999999999995,"15min":0.496},"min":{"1min":0.26100000000000001,"5min":0.216,"15min":0.20999999999999999},"max":{"1min":7.0839999999999996,"5min":7.0839999999999996,"15min":7.0839999999999996},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":1.2,"15min":0.65200000000000002},"min":{"1min":0.248,"5min":0.20899999999999999,"15min":0.185},"max":{"1min":2.3220000000000001,"5min":24.459,"15min":24.459},"last":0.35899999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:45:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64000000000000001,"15min":0.47599999999999998},"min":{"1min":0.26000000000000001,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":4.7560000000000002,"5min":8.718,"15min":8.718},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.72299999999999998,"15min":0.54600000000000004},"min":{"1min":0.27000000000000002,"5min":0.222,"15min":0.222},"max":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"last":0.36899999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:44:39 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":1.2470000000000001,"15min":0.75900000000000001},"min":{"1min":0.34999999999999998,"5min":0.29599999999999999,"15min":0.24399999999999999},"max":{"1min":1.417,"5min":24.381,"15min":24.381},"last":0.622},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.82299999999999995,"15min":0.61599999999999999},"min":{"1min":0.38700000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.6130000000000004,"5min":8.6950000000000003,"15min":8.6950000000000003},"last":0.63300000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:45:01 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.72599999999999998,"15min":0.622},"min":{"1min":0.40300000000000002,"5min":0.309,"15min":0.309},"max":{"1min":7.0990000000000002,"5min":7.0990000000000002,"15min":7.0990000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.498,"5min":1.337,"15min":0.81699999999999995},"min":{"1min":0.314,"5min":0.307,"15min":0.30599999999999999},"max":{"1min":0.67600000000000005,"5min":24.427,"15min":24.427},"last":0.52900000000000003}]},{"osd":6,"last update":"Tue Feb 13 00:45:33 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.82799999999999996,"15min":0.61699999999999999},"min":{"1min":0.38300000000000001,"5min":0.32900000000000001,"15min":0.31},"max":{"1min":3.302,"5min":8.6129999999999995,"15min":8.6129999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.2629999999999999,"5min":0.84299999999999997,"15min":0.64200000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":14.263,"5min":14.263,"15min":14.263},"last":0.67900000000000005}]},{"osd":7,"last update":"Tue Feb 13 00:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79300000000000004,"15min":0.627},"min":{"1min":0.42399999999999999,"5min":0.317,"15min":0.317},"max":{"1min":4.7149999999999999,"5min":8.6679999999999993,"15min":8.6679999999999993},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.3979999999999999,"5min":0.84499999999999997,"15min":0.66800000000000004},"min":{"1min":0.39400000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":8.1609999999999996,"5min":8.1609999999999996,"15min":8.1609999999999996},"last":0.58399999999999996}]}]},{"osd":2,"up_from":0,"seq":77309411451,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182517,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986289,"num_pgs":128,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307692,"kb_used_data":6356,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437460,"statfs":{"total":95995035648,"available":95679959040,"internally_reserved":0,"allocated":6508544,"data_stored":3542394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:44:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.74299999999999999,"15min":0.58999999999999997},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.66600000000000004,"5min":9.3399999999999999,"15min":9.3399999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.66000000000000003,"15min":0.58099999999999996},"min":{"1min":0.33400000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73599999999999999,"5min":13.833,"15min":13.833},"last":0.36899999999999999}]},{"osd":1,"last update":"Tue Feb 13 00:44:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1419999999999999,"15min":0.73199999999999998},"min":{"1min":0.29699999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.745,"5min":53.188000000000002,"15min":53.188000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.97099999999999997,"15min":0.67500000000000004},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66000000000000003,"5min":44.793999999999997,"15min":44.793999999999997},"last":0.38600000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:44:44 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68100000000000005,"15min":0.57799999999999996},"min":{"1min":0.34899999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.6859999999999999,"5min":8.6829999999999998,"15min":8.6829999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.77300000000000002,"15min":0.60799999999999998},"min":{"1min":0.29999999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.9470000000000001,"5min":18.911000000000001,"15min":18.911000000000001},"last":0.41099999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:44:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.879,"15min":0.65300000000000002},"min":{"1min":0.34699999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72699999999999998,"5min":23.5,"15min":23.5},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":1.0600000000000001,"15min":0.70899999999999996},"min":{"1min":0.36899999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79600000000000004,"5min":36.152999999999999,"15min":36.152999999999999},"last":0.48699999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:44:54 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41499999999999998,"15min":0.375},"min":{"1min":0.25700000000000001,"5min":0.24399999999999999,"15min":0.20599999999999999},"max":{"1min":0.69399999999999995,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.436,"15min":0.39500000000000002},"min":{"1min":0.312,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.67100000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.52000000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:45:16 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.47199999999999998,"15min":0.42899999999999999},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.153,"5min":3.153,"15min":3.153},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.51500000000000001,"15min":0.45100000000000001},"min":{"1min":0.31,"5min":0.254,"15min":0.254},"max":{"1min":4.9390000000000001,"5min":4.9390000000000001,"15min":4.9390000000000001},"last":0.46899999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:44:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48999999999999999,"15min":0.46999999999999997},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.47799999999999998,"15min":0.46000000000000002},"min":{"1min":0.29799999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.54100000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724652,"num_pgs":150,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315748,"kb_used_data":10324,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93429404,"statfs":{"total":95995035648,"available":95671709696,"internally_reserved":0,"allocated":10571776,"data_stored":7542674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:45:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":1.6870000000000001,"15min":0.88700000000000001},"min":{"1min":0.32000000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.363,"5min":72.444999999999993,"15min":72.444999999999993},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.234,"15min":0.75600000000000001},"min":{"1min":0.29999999999999999,"5min":0.215,"15min":0.215},"max":{"1min":4.2969999999999997,"5min":23.949000000000002,"15min":23.949000000000002},"last":0.51800000000000002}]},{"osd":1,"last update":"Tue Feb 13 00:45:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.0549999999999999,"15min":0.68400000000000005},"min":{"1min":0.311,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.1429999999999998,"5min":23.937000000000001,"15min":23.937000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":1.405,"15min":0.80700000000000005},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.254},"max":{"1min":0.63900000000000001,"5min":42.826999999999998,"15min":42.826999999999998},"last":0.60899999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:45:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.1269999999999998,"15min":1.0469999999999999},"min":{"1min":0.32600000000000001,"5min":0.247,"15min":0.247},"max":{"1min":4.9420000000000002,"5min":54.683999999999997,"15min":54.683999999999997},"last":0.501},{"interface":"front","average":{"1min":0.67400000000000004,"5min":1.7709999999999999,"15min":0.92800000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.5760000000000001,"5min":31.422999999999998,"15min":31.422999999999998},"last":0.68700000000000006}]},{"osd":3,"last update":"Tue Feb 13 00:45:04 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":2.5910000000000002,"15min":1.204},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.30499999999999999},"max":{"1min":4.8959999999999999,"5min":61.573,"15min":61.573},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.65600000000000003,"5min":1.954,"15min":0.98899999999999999},"min":{"1min":0.38900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.2389999999999999,"5min":54.848999999999997,"15min":54.848999999999997},"last":0.64000000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:45:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.496,"15min":0.41799999999999998},"min":{"1min":0.27700000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":7.8570000000000002,"15min":7.8570000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.57799999999999996,"15min":0.48199999999999998},"min":{"1min":0.36199999999999999,"5min":0.27900000000000003,"15min":0.27300000000000002},"max":{"1min":0.71399999999999997,"5min":8.0630000000000006,"15min":8.0630000000000006},"last":0.626}]},{"osd":6,"last update":"Tue Feb 13 00:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.5,"15min":0.45900000000000002},"min":{"1min":0.29199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.56699999999999995,"5min":8.1020000000000003,"15min":8.1020000000000003},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.59899999999999998,"15min":0.504},"min":{"1min":0.309,"5min":0.248,"15min":0.248},"max":{"1min":0.69599999999999995,"5min":7.7389999999999999,"15min":7.7389999999999999},"last":0.72299999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:44:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55100000000000005,"15min":0.71099999999999997},"min":{"1min":0.32000000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.2589999999999999,"5min":7.915,"15min":7.915},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.36299999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.90000000000000002,"5min":8.0169999999999995,"15min":8.0169999999999995},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4345856,"data_stored":4000819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4382720,"data_stored":4000905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4411392,"data_stored":4000977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":204,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":204,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":204,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":204,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":32,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":204,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:45:39.704 INFO:tasks.thrashosds.thrasher:Setting 28.18 to [5, 7, 0, 2, 4, 6] 2024-02-13T00:45:39.704 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '28.18', '5', '7', '0', '2', '4', '6'] 2024-02-13T00:45:39.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pg-upmap-items 28.18 5 7 0 2 4 6 2024-02-13T00:45:39.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:39.812 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:39.833 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:39.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:39.923 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:38.083419+0000 osd.1 (osd.1) 155 : cluster [DBG] 28.7 scrub starts 2024-02-13T00:45:39.923 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:38.084353+0000 osd.1 (osd.1) 156 : cluster [DBG] 28.7 scrub ok 2024-02-13T00:45:39.923 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:38.332432+0000 mgr.y (mgr.14544) 336 : audit [DBG] from='client.27249 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:45:39.923 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:38.409768+0000 mgr.y (mgr.14544) 337 : cluster [DBG] pgmap v544: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-13T00:45:39.923 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:39.661674+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e384: 8 total, 8 up, 5 in 2024-02-13T00:45:39.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:38.083419+0000 osd.1 (osd.1) 155 : cluster [DBG] 28.7 scrub starts 2024-02-13T00:45:39.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:38.084353+0000 osd.1 (osd.1) 156 : cluster [DBG] 28.7 scrub ok 2024-02-13T00:45:39.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:38.332432+0000 mgr.y (mgr.14544) 336 : audit [DBG] from='client.27249 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:45:39.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:38.409768+0000 mgr.y (mgr.14544) 337 : cluster [DBG] pgmap v544: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-13T00:45:39.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:39.661674+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e384: 8 total, 8 up, 5 in 2024-02-13T00:45:39.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:39.973 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:38.083419+0000 osd.1 (osd.1) 155 : cluster [DBG] 28.7 scrub starts 2024-02-13T00:45:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:38.084353+0000 osd.1 (osd.1) 156 : cluster [DBG] 28.7 scrub ok 2024-02-13T00:45:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:38.332432+0000 mgr.y (mgr.14544) 336 : audit [DBG] from='client.27249 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:45:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:38.409768+0000 mgr.y (mgr.14544) 337 : cluster [DBG] pgmap v544: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-13T00:45:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:39.661674+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e384: 8 total, 8 up, 5 in 2024-02-13T00:45:40.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:40.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:40.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:40.302 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:40.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 -- 172.21.15.162:0/2001730770 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b04102270 msgr2=0x7f7b04104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:40.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 --2- 172.21.15.162:0/2001730770 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b04102270 0x7f7b04104660 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f7afc009be0 tx=0x7f7afc004d10).stop 2024-02-13T00:45:40.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 -- 172.21.15.162:0/2001730770 shutdown_connections 2024-02-13T00:45:40.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 --2- 172.21.15.162:0/2001730770 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b04102270 0x7f7b04104660 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:40.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 --2- 172.21.15.162:0/2001730770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b040f8d10 0x7f7b040f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:40.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 --2- 172.21.15.162:0/2001730770 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b040f95e0 0x7f7b04101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:40.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 -- 172.21.15.162:0/2001730770 >> 172.21.15.162:0/2001730770 conn(0x7f7b0406eb90 msgr2=0x7f7b040f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:40.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 -- 172.21.15.162:0/2001730770 shutdown_connections 2024-02-13T00:45:40.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 -- 172.21.15.162:0/2001730770 wait complete. 2024-02-13T00:45:40.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 librados: starting msgr at 2024-02-13T00:45:40.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 librados: starting objecter 2024-02-13T00:45:40.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 Processor -- start 2024-02-13T00:45:40.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 -- start start 2024-02-13T00:45:40.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 librados: setting wanted keys 2024-02-13T00:45:40.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.873+0000 7f7b0b358700 1 librados: calling monclient init 2024-02-13T00:45:40.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b0b358700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b040f8d10 0x7f7b040f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:40.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b0b358700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b040f95e0 0x7f7b040f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:40.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b0b358700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b04102270 0x7f7b040f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:40.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b0b358700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b04116650 con 0x7f7b040f95e0 2024-02-13T00:45:40.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b0b358700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b041167c0 con 0x7f7b04102270 2024-02-13T00:45:40.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b0b358700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b04116980 con 0x7f7b040f8d10 2024-02-13T00:45:40.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b090f4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b040f8d10 0x7f7b040f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:40.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b090f4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b040f8d10 0x7f7b040f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:46034/0 (socket says 172.21.15.162:46034) 2024-02-13T00:45:40.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b088f3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b040f95e0 0x7f7b040f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:40.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b090f4700 1 -- 172.21.15.162:0/2462618262 learned_addr learned my addr 172.21.15.162:0/2462618262 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:40.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b098f5700 1 --2- 172.21.15.162:0/2462618262 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b04102270 0x7f7b040f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:40.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b090f4700 1 -- 172.21.15.162:0/2462618262 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b04102270 msgr2=0x7f7b040f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:40.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b090f4700 1 --2- 172.21.15.162:0/2462618262 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b04102270 0x7f7b040f76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:40.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b090f4700 1 -- 172.21.15.162:0/2462618262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b040f95e0 msgr2=0x7f7b040f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:40.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b090f4700 1 --2- 172.21.15.162:0/2462618262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b040f95e0 0x7f7b040f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:40.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b090f4700 1 -- 172.21.15.162:0/2462618262 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b00005040 con 0x7f7b040f8d10 2024-02-13T00:45:40.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7afa7fc700 10 client.?.objecter ms_handle_connect 0x7f7b040f8d10 2024-02-13T00:45:40.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7afa7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:40.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b090f4700 1 --2- 172.21.15.162:0/2462618262 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b040f8d10 0x7f7b040f6cc0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 rx=0x7f7b000048f0 tx=0x7f7b00009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:40.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7afa7fc700 1 -- 172.21.15.162:0/2462618262 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00007b00 con 0x7f7b040f8d10 2024-02-13T00:45:40.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7afa7fc700 1 -- 172.21.15.162:0/2462618262 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b0000f040 con 0x7f7b040f8d10 2024-02-13T00:45:40.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b0b358700 1 -- 172.21.15.162:0/2462618262 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7afc0097c0 con 0x7f7b040f8d10 2024-02-13T00:45:40.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7afa7fc700 1 -- 172.21.15.162:0/2462618262 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00014a00 con 0x7f7b040f8d10 2024-02-13T00:45:40.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b0b358700 10 client.37358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:40.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b0b358700 1 -- 172.21.15.162:0/2462618262 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b04116f90 con 0x7f7b040f8d10 2024-02-13T00:45:40.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.874+0000 7f7b0b358700 1 librados: init done 2024-02-13T00:45:40.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.876+0000 7f7afa7fc700 1 -- 172.21.15.162:0/2462618262 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f7b0000e580 con 0x7f7b040f8d10 2024-02-13T00:45:40.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.876+0000 7f7b0b358700 1 -- 172.21.15.162:0/2462618262 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b041071b0 con 0x7f7b040f8d10 2024-02-13T00:45:40.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.876+0000 7f7afa7fc700 1 --2- 172.21.15.162:0/2462618262 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7af0062c90 0x7f7af0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:40.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.876+0000 7f7afa7fc700 1 -- 172.21.15.162:0/2462618262 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(385..385 src has 52..385) v4 ==== 8880+0+0 (secure 0 0 0) 0x7f7b0007d0f0 con 0x7f7b040f8d10 2024-02-13T00:45:40.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.876+0000 7f7afa7fc700 10 client.37358.objecter ms_dispatch 0x7f7b04001550 osd_map(385..385 src has 52..385) v4 2024-02-13T00:45:40.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.876+0000 7f7afa7fc700 3 client.37358.objecter handle_osd_map got epochs [385,385] > 0 2024-02-13T00:45:40.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.876+0000 7f7afa7fc700 3 client.37358.objecter handle_osd_map decoding full epoch 385 2024-02-13T00:45:40.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.876+0000 7f7b088f3700 1 --2- 172.21.15.162:0/2462618262 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7af0062c90 0x7f7af0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:40.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.876+0000 7f7afa7fc700 20 client.37358.objecter dump_active .. 0 homeless 2024-02-13T00:45:40.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.876+0000 7f7afa7fc700 10 client.37358.objecter ms_handle_connect 0x7f7af0062c90 2024-02-13T00:45:40.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.876+0000 7f7b088f3700 1 --2- 172.21.15.162:0/2462618262 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7af0062c90 0x7f7af0065150 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 rx=0x7f7b040f8460 tx=0x7f7af4009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:40.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:40.881+0000 7f7afa7fc700 1 -- 172.21.15.162:0/2462618262 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7b00049f60 con 0x7f7b040f8d10 2024-02-13T00:45:41.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.057+0000 7f7b0b358700 1 -- 172.21.15.162:0/2462618262 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f7b0405fb80 con 0x7f7b040f8d10 2024-02-13T00:45:41.150 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:41.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:41.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:41.182 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:41.256 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:41.289 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:41.309 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:41.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.398+0000 7f7afa7fc700 1 -- 172.21.15.162:0/2462618262 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v386) v1 ==== 101+0+0 (secure 0 0 0) 0x7f7b00050290 con 0x7f7b040f8d10 2024-02-13T00:45:41.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.403+0000 7f7b0b358700 10 librados: watch_flush enter 2024-02-13T00:45:41.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.403+0000 7f7b0b358700 10 librados: watch_flush exit 2024-02-13T00:45:41.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.403+0000 7f7b0b358700 20 client.37358.objecter shutdown clearing up homeless session... 2024-02-13T00:45:41.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.403+0000 7f7b0b358700 10 client.37358.objecter successfully canceled tick 2024-02-13T00:45:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.407+0000 7f7b0b358700 1 -- 172.21.15.162:0/2462618262 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7af0062c90 msgr2=0x7f7af0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.407+0000 7f7b0b358700 1 --2- 172.21.15.162:0/2462618262 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7af0062c90 0x7f7af0065150 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 rx=0x7f7b040f8460 tx=0x7f7af4009040).stop 2024-02-13T00:45:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.407+0000 7f7b0b358700 1 -- 172.21.15.162:0/2462618262 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b040f8d10 msgr2=0x7f7b040f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.407+0000 7f7b0b358700 1 --2- 172.21.15.162:0/2462618262 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b040f8d10 0x7f7b040f6cc0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 rx=0x7f7b000048f0 tx=0x7f7b00009cb0).stop 2024-02-13T00:45:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.407+0000 7f7b0b358700 1 -- 172.21.15.162:0/2462618262 shutdown_connections 2024-02-13T00:45:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.407+0000 7f7b0b358700 1 --2- 172.21.15.162:0/2462618262 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7af0062c90 0x7f7af0065150 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.407+0000 7f7b0b358700 1 --2- 172.21.15.162:0/2462618262 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b04102270 0x7f7b040f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.407+0000 7f7b0b358700 1 --2- 172.21.15.162:0/2462618262 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b040f8d10 0x7f7b040f6cc0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.407+0000 7f7b0b358700 1 --2- 172.21.15.162:0/2462618262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b040f95e0 0x7f7b040f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:41.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.407+0000 7f7b0b358700 1 -- 172.21.15.162:0/2462618262 >> 172.21.15.162:0/2462618262 conn(0x7f7b0406eb90 msgr2=0x7f7b04102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:41.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:41.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.411+0000 7f7b0b358700 1 -- 172.21.15.162:0/2462618262 shutdown_connections 2024-02-13T00:45:41.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.411+0000 7f7b0b358700 1 -- 172.21.15.162:0/2462618262 wait complete. 2024-02-13T00:45:41.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:41.412+0000 7f7b0b358700 1 librados: shutdown 2024-02-13T00:45:41.417 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:45:41.443 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:41.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:39.558138+0000 osd.0 (osd.0) 132 : cluster [DBG] 28.10 scrub starts 2024-02-13T00:45:41.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:39.561988+0000 osd.0 (osd.0) 133 : cluster [DBG] 28.10 scrub ok 2024-02-13T00:45:41.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:40.230241+0000 osd.6 (osd.6) 89 : cluster [DBG] 28.b scrub starts 2024-02-13T00:45:41.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:40.233180+0000 osd.6 (osd.6) 90 : cluster [DBG] 28.b scrub ok 2024-02-13T00:45:41.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:40.404226+0000 mon.a (mon.0) 2414 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi162-83985-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi162-83985-65"}]': finished 2024-02-13T00:45:41.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:40.404302+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e385: 8 total, 8 up, 5 in 2024-02-13T00:45:41.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:40.410127+0000 mgr.y (mgr.14544) 338 : cluster [DBG] pgmap v548: 246 pgs: 8 unknown, 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:45:41.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:41.058461+0000 mon.c (mon.2) 368 : audit [INF] from='client.? 172.21.15.162:0/2462618262' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:45:41.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:41.058784+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:45:41.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:39.558138+0000 osd.0 (osd.0) 132 : cluster [DBG] 28.10 scrub starts 2024-02-13T00:45:41.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:39.561988+0000 osd.0 (osd.0) 133 : cluster [DBG] 28.10 scrub ok 2024-02-13T00:45:41.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:40.230241+0000 osd.6 (osd.6) 89 : cluster [DBG] 28.b scrub starts 2024-02-13T00:45:41.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:40.233180+0000 osd.6 (osd.6) 90 : cluster [DBG] 28.b scrub ok 2024-02-13T00:45:41.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:40.404226+0000 mon.a (mon.0) 2414 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi162-83985-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi162-83985-65"}]': finished 2024-02-13T00:45:41.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:40.404302+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e385: 8 total, 8 up, 5 in 2024-02-13T00:45:41.765 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:40.410127+0000 mgr.y (mgr.14544) 338 : cluster [DBG] pgmap v548: 246 pgs: 8 unknown, 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:45:41.765 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:41.058461+0000 mon.c (mon.2) 368 : audit [INF] from='client.? 172.21.15.162:0/2462618262' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:45:41.765 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:41.058784+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:45:41.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:39.558138+0000 osd.0 (osd.0) 132 : cluster [DBG] 28.10 scrub starts 2024-02-13T00:45:41.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:39.561988+0000 osd.0 (osd.0) 133 : cluster [DBG] 28.10 scrub ok 2024-02-13T00:45:41.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:40.230241+0000 osd.6 (osd.6) 89 : cluster [DBG] 28.b scrub starts 2024-02-13T00:45:41.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:40.233180+0000 osd.6 (osd.6) 90 : cluster [DBG] 28.b scrub ok 2024-02-13T00:45:41.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:40.404226+0000 mon.a (mon.0) 2414 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi162-83985-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi162-83985-65"}]': finished 2024-02-13T00:45:41.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:40.404302+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e385: 8 total, 8 up, 5 in 2024-02-13T00:45:41.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:40.410127+0000 mgr.y (mgr.14544) 338 : cluster [DBG] pgmap v548: 246 pgs: 8 unknown, 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:45:41.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:41.058461+0000 mon.c (mon.2) 368 : audit [INF] from='client.? 172.21.15.162:0/2462618262' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:45:41.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:41.058784+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:45:41.934 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:45:41.934 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:45:41.934 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:45:41.934 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:45:41.934 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:45:42.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:41.393949+0000 mon.a (mon.0) 2417 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:45:42.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:41.397719+0000 mon.a (mon.0) 2418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:45:42.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:41.397786+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e386: 8 total, 8 up, 5 in 2024-02-13T00:45:42.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:41.406842+0000 mon.a (mon.0) 2420 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-13T00:45:42.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:41.393949+0000 mon.a (mon.0) 2417 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:45:42.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:41.397719+0000 mon.a (mon.0) 2418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:45:42.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:41.397786+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e386: 8 total, 8 up, 5 in 2024-02-13T00:45:42.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:41.406842+0000 mon.a (mon.0) 2420 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-13T00:45:42.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-13T00:45:42.793 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:42.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:41.393949+0000 mon.a (mon.0) 2417 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:45:42.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:41.397719+0000 mon.a (mon.0) 2418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:45:42.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:41.397786+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e386: 8 total, 8 up, 5 in 2024-02-13T00:45:42.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:41.406842+0000 mon.a (mon.0) 2420 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-13T00:45:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:42.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:42.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:42.931 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:42.951 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:42.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:42.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:43.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:43.086 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:43.106 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:43.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:43.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:43.241 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:43.262 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:43.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:43.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:43.395 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:43.415 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:43.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:43.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:43.523 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:43.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:42.192536+0000 osd.1 (osd.1) 157 : cluster [DBG] 61.9 scrub starts 2024-02-13T00:45:43.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:42.196128+0000 osd.1 (osd.1) 158 : cluster [DBG] 61.9 scrub ok 2024-02-13T00:45:43.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:42.236879+0000 osd.6 (osd.6) 91 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:45:43.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:42.238148+0000 osd.6 (osd.6) 92 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:45:43.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:42.410433+0000 mgr.y (mgr.14544) 339 : cluster [DBG] pgmap v551: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:45:43.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:42.411362+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e387: 8 total, 8 up, 5 in 2024-02-13T00:45:43.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:42.412585+0000 mon.a (mon.0) 2422 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:43.551 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:43.570 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:43.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:43.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:42.192536+0000 osd.1 (osd.1) 157 : cluster [DBG] 61.9 scrub starts 2024-02-13T00:45:43.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:42.196128+0000 osd.1 (osd.1) 158 : cluster [DBG] 61.9 scrub ok 2024-02-13T00:45:43.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:42.236879+0000 osd.6 (osd.6) 91 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:45:43.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:42.238148+0000 osd.6 (osd.6) 92 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:45:43.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:42.410433+0000 mgr.y (mgr.14544) 339 : cluster [DBG] pgmap v551: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:45:43.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:42.411362+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e387: 8 total, 8 up, 5 in 2024-02-13T00:45:43.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:42.412585+0000 mon.a (mon.0) 2422 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:43.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:42.192536+0000 osd.1 (osd.1) 157 : cluster [DBG] 61.9 scrub starts 2024-02-13T00:45:43.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:42.196128+0000 osd.1 (osd.1) 158 : cluster [DBG] 61.9 scrub ok 2024-02-13T00:45:43.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:42.236879+0000 osd.6 (osd.6) 91 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:45:43.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:42.238148+0000 osd.6 (osd.6) 92 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:45:43.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:42.410433+0000 mgr.y (mgr.14544) 339 : cluster [DBG] pgmap v551: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:45:43.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:42.411362+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e387: 8 total, 8 up, 5 in 2024-02-13T00:45:43.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:42.412585+0000 mon.a (mon.0) 2422 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:43.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:43.700 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:43.718 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:43.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:43.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:43.850 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:43.924 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:43.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:43.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:44.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:44.055 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:44.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-02-13T00:45:44.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6989 ms) 2024-02-13T00:45:44.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-02-13T00:45:44.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7005 ms) 2024-02-13T00:45:44.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-02-13T00:45:44.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6982 ms) 2024-02-13T00:45:44.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-02-13T00:45:44.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7028 ms) 2024-02-13T00:45:44.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-02-13T00:45:44.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (6992 ms) 2024-02-13T00:45:44.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139008 ms total) 2024-02-13T00:45:44.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: 2024-02-13T00:45:44.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-02-13T00:45:44.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [==========] 56 tests from 4 test suites ran. (289729 ms total) 2024-02-13T00:45:44.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ PASSED ] 56 tests. 2024-02-13T00:45:44.427 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=83996 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 83996 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84060 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84060 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84044 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84044 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84020 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84020 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84116 2024-02-13T00:45:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84116 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=83988 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 83988 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84132 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84132 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84084 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84084 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84164 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84164 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.429 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84124 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84124 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84180 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84180 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=83972 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 83972 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84172 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84172 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84140 2024-02-13T00:45:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84140 2024-02-13T00:45:44.431 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-02-13T00:45:44.431 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=84052 2024-02-13T00:45:44.431 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 84052 2024-02-13T00:45:44.456 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:44.688 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:42.589791+0000 osd.0 (osd.0) 134 : cluster [DBG] 5.b deep-scrub starts 2024-02-13T00:45:44.688 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:42.592668+0000 osd.0 (osd.0) 135 : cluster [DBG] 5.b deep-scrub ok 2024-02-13T00:45:44.688 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:43.413168+0000 mon.a (mon.0) 2423 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-83985-65"}]': finished 2024-02-13T00:45:44.688 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:43.413232+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e388: 8 total, 8 up, 5 in 2024-02-13T00:45:44.689 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:43.414785+0000 mon.a (mon.0) 2425 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:44.689 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:42.589791+0000 osd.0 (osd.0) 134 : cluster [DBG] 5.b deep-scrub starts 2024-02-13T00:45:44.689 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:42.592668+0000 osd.0 (osd.0) 135 : cluster [DBG] 5.b deep-scrub ok 2024-02-13T00:45:44.689 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:43.413168+0000 mon.a (mon.0) 2423 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-83985-65"}]': finished 2024-02-13T00:45:44.689 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:43.413232+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e388: 8 total, 8 up, 5 in 2024-02-13T00:45:44.689 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:43.414785+0000 mon.a (mon.0) 2425 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:42.589791+0000 osd.0 (osd.0) 134 : cluster [DBG] 5.b deep-scrub starts 2024-02-13T00:45:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:42.592668+0000 osd.0 (osd.0) 135 : cluster [DBG] 5.b deep-scrub ok 2024-02-13T00:45:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:43.413168+0000 mon.a (mon.0) 2423 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-83985-65"}]': finished 2024-02-13T00:45:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:43.413232+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e388: 8 total, 8 up, 5 in 2024-02-13T00:45:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:43.414785+0000 mon.a (mon.0) 2425 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-83985-65"}]: dispatch 2024-02-13T00:45:45.304 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:45.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.301+0000 7fae9010f700 1 -- 172.21.15.162:0/3956520903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae880fd490 msgr2=0x7fae880f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.301+0000 7fae9010f700 1 --2- 172.21.15.162:0/3956520903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae880fd490 0x7fae880f6cc0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7fae780048f0 tx=0x7fae78009cb0).stop 2024-02-13T00:45:45.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.301+0000 7fae9010f700 1 -- 172.21.15.162:0/3956520903 shutdown_connections 2024-02-13T00:45:45.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.301+0000 7fae9010f700 1 --2- 172.21.15.162:0/3956520903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae880f73a0 0x7fae881039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.301+0000 7fae9010f700 1 --2- 172.21.15.162:0/3956520903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae880fcb30 0x7fae880fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.301+0000 7fae9010f700 1 --2- 172.21.15.162:0/3956520903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae880fd490 0x7fae880f6cc0 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.301+0000 7fae9010f700 1 -- 172.21.15.162:0/3956520903 >> 172.21.15.162:0/3956520903 conn(0x7fae8806eb90 msgr2=0x7fae880f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:45.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.301+0000 7fae9010f700 1 -- 172.21.15.162:0/3956520903 shutdown_connections 2024-02-13T00:45:45.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.301+0000 7fae9010f700 1 -- 172.21.15.162:0/3956520903 wait complete. 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.301+0000 7fae9010f700 1 librados: starting msgr at 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.301+0000 7fae9010f700 1 librados: starting objecter 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae9010f700 1 Processor -- start 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae9010f700 1 -- start start 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae9010f700 1 librados: setting wanted keys 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae9010f700 1 librados: calling monclient init 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae9010f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae880f73a0 0x7fae8810bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae9010f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae880fcb30 0x7fae8810c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae9010f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae880fd490 0x7fae88106070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae9010f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae88106560 con 0x7fae880f73a0 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae9010f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae881066d0 con 0x7fae880fd490 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae9010f700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae88106890 con 0x7fae880fcb30 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae8deab700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae880f73a0 0x7fae8810bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:45.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae8deab700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae880f73a0 0x7fae8810bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:48300/0 (socket says 172.21.15.162:48300) 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae8deab700 1 -- 172.21.15.162:0/893101610 learned_addr learned my addr 172.21.15.162:0/893101610 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae8d6aa700 1 --2- 172.21.15.162:0/893101610 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae880fcb30 0x7fae8810c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae8e6ac700 1 --2- 172.21.15.162:0/893101610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae880fd490 0x7fae88106070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae8d6aa700 1 -- 172.21.15.162:0/893101610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae880fd490 msgr2=0x7fae88106070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae8d6aa700 1 --2- 172.21.15.162:0/893101610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae880fd490 0x7fae88106070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae8d6aa700 1 -- 172.21.15.162:0/893101610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae880f73a0 msgr2=0x7fae8810bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae8d6aa700 1 --2- 172.21.15.162:0/893101610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae880f73a0 0x7fae8810bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae8d6aa700 1 -- 172.21.15.162:0/893101610 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae78005040 con 0x7fae880fcb30 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae8d6aa700 1 --2- 172.21.15.162:0/893101610 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae880fcb30 0x7fae8810c3c0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 rx=0x7fae7800d270 tx=0x7fae7800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae7effd700 10 client.?.objecter ms_handle_connect 0x7fae880fcb30 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.302+0000 7fae7effd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.303+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7800d970 con 0x7fae880fcb30 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.303+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae88106b10 con 0x7fae880fcb30 2024-02-13T00:45:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.303+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae78000bc0 con 0x7fae880fcb30 2024-02-13T00:45:45.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.303+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78011e10 con 0x7fae880fcb30 2024-02-13T00:45:45.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.303+0000 7fae9010f700 10 client.37364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:45.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.303+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae88112d00 con 0x7fae880fcb30 2024-02-13T00:45:45.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.303+0000 7fae9010f700 1 librados: init done 2024-02-13T00:45:45.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.304+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fae880475e0 con 0x7fae880fcb30 2024-02-13T00:45:45.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.304+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fae78011420 con 0x7fae880fcb30 2024-02-13T00:45:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:45.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:45.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.304+0000 7fae7effd700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6800/2866534053 conn(0x7fae74062c90 0x7fae74065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:45.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(389..389 src has 52..389) v4 ==== 8072+0+0 (secure 0 0 0) 0x7fae7807e1d0 con 0x7fae880fcb30 2024-02-13T00:45:45.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 osd_map(389..389 src has 52..389) v4 2024-02-13T00:45:45.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae7effd700 3 client.37364.objecter handle_osd_map got epochs [389,389] > 0 2024-02-13T00:45:45.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae7effd700 3 client.37364.objecter handle_osd_map decoding full epoch 389 2024-02-13T00:45:45.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae8deab700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6800/2866534053 conn(0x7fae74062c90 0x7fae74065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae7effd700 20 client.37364.objecter dump_active .. 0 homeless 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v389) v1 ==== 54+0+16 (secure 0 0 0) 0x7fae78085070 con 0x7fae880fcb30 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae9010f700 10 client.37364.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 1 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6801/2103743338 conn(0x7fae880fdda0 0x7fae880fe180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88104f30 osd=0 3 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88104f30 osd=0 3 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88104f30 osd=0 3 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 1 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae88104f30 osd=0 3 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 0 1 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88104f30 osd=0 4 2024-02-13T00:45:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae9010f700 10 client.37364.objecter _send_command 1 2024-02-13T00:45:45.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae88107550 con 0x7fae880fdda0 2024-02-13T00:45:45.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae7effd700 10 client.37364.objecter ms_handle_connect 0x7fae74062c90 2024-02-13T00:45:45.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae8deab700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6800/2866534053 conn(0x7fae74062c90 0x7fae74065150 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 rx=0x7fae84004fd0 tx=0x7fae8400f040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:45.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.305+0000 7fae8e6ac700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6801/2103743338 conn(0x7fae880fdda0 0x7fae880fe180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:45.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.306+0000 7fae8e6ac700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6801/2103743338 conn(0x7fae880fdda0 0x7fae880fe180 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:45.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.306+0000 7fae7effd700 10 client.37364.objecter ms_handle_connect 0x7fae880fdda0 2024-02-13T00:45:45.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.306+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fae88107550 con 0x7fae880fdda0 2024-02-13T00:45:45.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.306+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:45:45.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.306+0000 7fae7effd700 10 client.37364.objecter _finish_command 1 = system:0 2024-02-13T00:45:45.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.306+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae88104f30 osd=0 4 2024-02-13T00:45:45.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.306+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 0 1 2024-02-13T00:45:45.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.321+0000 7fae9010f700 10 client.37364.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:45.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.321+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 2 2024-02-13T00:45:45.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.321+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88104f30 osd=0 3 2024-02-13T00:45:45.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.321+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88104f30 osd=0 3 2024-02-13T00:45:45.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.321+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88104f30 osd=0 3 2024-02-13T00:45:45.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.321+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 2 2024-02-13T00:45:45.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.321+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae88104f30 osd=0 3 2024-02-13T00:45:45.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.321+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 0 2 2024-02-13T00:45:45.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.321+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88104f30 osd=0 4 2024-02-13T00:45:45.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.321+0000 7fae9010f700 10 client.37364.objecter _send_command 2 2024-02-13T00:45:45.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.321+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fae88027410 con 0x7fae880fdda0 2024-02-13T00:45:45.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fae88027410 con 0x7fae880fdda0 2024-02-13T00:45:45.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:45.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae7effd700 10 client.37364.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae88104f30 osd=0 4 2024-02-13T00:45:45.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 0 2 2024-02-13T00:45:45.326 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:45:45.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae9010f700 10 client.37364.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:45.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 3 2024-02-13T00:45:45.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6805/150599780 conn(0x7fae88059990 0x7fae88059d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:45.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88028760 osd=1 3 2024-02-13T00:45:45.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88028760 osd=1 3 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88028760 osd=1 3 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 3 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae88028760 osd=1 3 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 1 3 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88028760 osd=1 4 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae9010f700 10 client.37364.objecter _send_command 3 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae88119610 con 0x7fae88059990 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae8deab700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6805/150599780 conn(0x7fae88059990 0x7fae88059d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae7effd700 10 client.37364.objecter ms_handle_connect 0x7fae88059990 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.323+0000 7fae8deab700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6805/150599780 conn(0x7fae88059990 0x7fae88059d90 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.324+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fae88119610 con 0x7fae88059990 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.324+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.324+0000 7fae7effd700 10 client.37364.objecter _finish_command 3 = system:0 2024-02-13T00:45:45.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.324+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae88028760 osd=1 4 2024-02-13T00:45:45.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.324+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 1 3 2024-02-13T00:45:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.339+0000 7fae9010f700 10 client.37364.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.339+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 4 2024-02-13T00:45:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.339+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88028760 osd=1 3 2024-02-13T00:45:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.339+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88028760 osd=1 3 2024-02-13T00:45:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.339+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88028760 osd=1 3 2024-02-13T00:45:45.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.339+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 4 2024-02-13T00:45:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.339+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae88028760 osd=1 3 2024-02-13T00:45:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.339+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 1 4 2024-02-13T00:45:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.339+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88028760 osd=1 4 2024-02-13T00:45:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.339+0000 7fae9010f700 10 client.37364.objecter _send_command 4 2024-02-13T00:45:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.339+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fae88059400 con 0x7fae88059990 2024-02-13T00:45:45.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.340+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fae88059400 con 0x7fae88059990 2024-02-13T00:45:45.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.340+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:45.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.340+0000 7fae7effd700 10 client.37364.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.340+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae88028760 osd=1 4 2024-02-13T00:45:45.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.340+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 1 4 2024-02-13T00:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:45:45.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae9010f700 10 client.37364.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:45.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 5 2024-02-13T00:45:45.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6809/3689719376 conn(0x7fae880282c0 0x7fae88118030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:45.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae880597b0 osd=2 3 2024-02-13T00:45:45.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae880597b0 osd=2 3 2024-02-13T00:45:45.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae880597b0 osd=2 3 2024-02-13T00:45:45.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 5 2024-02-13T00:45:45.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae880597b0 osd=2 3 2024-02-13T00:45:45.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 2 5 2024-02-13T00:45:45.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae880597b0 osd=2 4 2024-02-13T00:45:45.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae9010f700 10 client.37364.objecter _send_command 5 2024-02-13T00:45:45.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae88118520 con 0x7fae880282c0 2024-02-13T00:45:45.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae8d6aa700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6809/3689719376 conn(0x7fae880282c0 0x7fae88118030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:45.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae7effd700 10 client.37364.objecter ms_handle_connect 0x7fae880282c0 2024-02-13T00:45:45.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.341+0000 7fae8d6aa700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6809/3689719376 conn(0x7fae880282c0 0x7fae88118030 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:45.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.342+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fae88118520 con 0x7fae880282c0 2024-02-13T00:45:45.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.342+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:45:45.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.342+0000 7fae7effd700 10 client.37364.objecter _finish_command 5 = system:0 2024-02-13T00:45:45.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.342+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae880597b0 osd=2 4 2024-02-13T00:45:45.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.342+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 2 5 2024-02-13T00:45:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.356+0000 7fae9010f700 10 client.37364.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.356+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 6 2024-02-13T00:45:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.356+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae880597b0 osd=2 3 2024-02-13T00:45:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.356+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae880597b0 osd=2 3 2024-02-13T00:45:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.356+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae880597b0 osd=2 3 2024-02-13T00:45:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.356+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 6 2024-02-13T00:45:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.356+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae880597b0 osd=2 3 2024-02-13T00:45:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.356+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 2 6 2024-02-13T00:45:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.356+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae880597b0 osd=2 4 2024-02-13T00:45:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.356+0000 7fae9010f700 10 client.37364.objecter _send_command 6 2024-02-13T00:45:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.356+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fae88027820 con 0x7fae880282c0 2024-02-13T00:45:45.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fae88027820 con 0x7fae880282c0 2024-02-13T00:45:45.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:45.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae7effd700 10 client.37364.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae880597b0 osd=2 4 2024-02-13T00:45:45.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 2 6 2024-02-13T00:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:45:45.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae9010f700 10 client.37364.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 7 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6813/604971011 conn(0x7fae8811b790 0x7fae8811db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88118f10 osd=3 3 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88118f10 osd=3 3 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88118f10 osd=3 3 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 7 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae88118f10 osd=3 3 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 3 7 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88118f10 osd=3 4 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae9010f700 10 client.37364.objecter _send_command 7 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae881207d0 con 0x7fae8811b790 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.357+0000 7fae8e6ac700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6813/604971011 conn(0x7fae8811b790 0x7fae8811db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.358+0000 7fae7effd700 10 client.37364.objecter ms_handle_connect 0x7fae8811b790 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.358+0000 7fae8e6ac700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6813/604971011 conn(0x7fae8811b790 0x7fae8811db80 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:45.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.358+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fae881207d0 con 0x7fae8811b790 2024-02-13T00:45:45.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.358+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:45:45.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.358+0000 7fae7effd700 10 client.37364.objecter _finish_command 7 = system:0 2024-02-13T00:45:45.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.358+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae88118f10 osd=3 4 2024-02-13T00:45:45.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.358+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 3 7 2024-02-13T00:45:45.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.373+0000 7fae9010f700 10 client.37364.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:45.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.373+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 8 2024-02-13T00:45:45.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.373+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88118f10 osd=3 3 2024-02-13T00:45:45.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.373+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88118f10 osd=3 3 2024-02-13T00:45:45.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.373+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88118f10 osd=3 3 2024-02-13T00:45:45.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.373+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 8 2024-02-13T00:45:45.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.373+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae88118f10 osd=3 3 2024-02-13T00:45:45.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.373+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 3 8 2024-02-13T00:45:45.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.373+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88118f10 osd=3 4 2024-02-13T00:45:45.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.373+0000 7fae9010f700 10 client.37364.objecter _send_command 8 2024-02-13T00:45:45.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.373+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fae88118a40 con 0x7fae8811b790 2024-02-13T00:45:45.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fae88118a40 con 0x7fae8811b790 2024-02-13T00:45:45.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:45.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae7effd700 10 client.37364.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae88118f10 osd=3 4 2024-02-13T00:45:45.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 3 8 2024-02-13T00:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:45:45.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae9010f700 10 client.37364.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 9 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6800/3344263388 conn(0x7fae88117b90 0x7fae8811f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88118800 osd=4 3 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88118800 osd=4 3 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88118800 osd=4 3 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 9 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae88118800 osd=4 3 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 4 9 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88118800 osd=4 4 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae9010f700 10 client.37364.objecter _send_command 9 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.374+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae8811f6e0 con 0x7fae88117b90 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.375+0000 7fae8deab700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6800/3344263388 conn(0x7fae88117b90 0x7fae8811f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.375+0000 7fae8deab700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6800/3344263388 conn(0x7fae88117b90 0x7fae8811f1f0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:45.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.375+0000 7fae7effd700 10 client.37364.objecter ms_handle_connect 0x7fae88117b90 2024-02-13T00:45:45.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.376+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fae8811f6e0 con 0x7fae88117b90 2024-02-13T00:45:45.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.376+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:45:45.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.376+0000 7fae7effd700 10 client.37364.objecter _finish_command 9 = system:0 2024-02-13T00:45:45.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.376+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae88118800 osd=4 4 2024-02-13T00:45:45.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.376+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 4 9 2024-02-13T00:45:45.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.390+0000 7fae9010f700 10 client.37364.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:45.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.390+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 10 2024-02-13T00:45:45.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.390+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88118800 osd=4 3 2024-02-13T00:45:45.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.390+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88118800 osd=4 3 2024-02-13T00:45:45.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.390+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88118800 osd=4 3 2024-02-13T00:45:45.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.390+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 10 2024-02-13T00:45:45.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.390+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae88118800 osd=4 3 2024-02-13T00:45:45.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.390+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 4 10 2024-02-13T00:45:45.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.390+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88118800 osd=4 4 2024-02-13T00:45:45.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.390+0000 7fae9010f700 10 client.37364.objecter _send_command 10 2024-02-13T00:45:45.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.390+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fae8811f6e0 con 0x7fae88117b90 2024-02-13T00:45:45.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.392+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fae8811f6e0 con 0x7fae88117b90 2024-02-13T00:45:45.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.392+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:45.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.392+0000 7fae7effd700 10 client.37364.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.392+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae88118800 osd=4 4 2024-02-13T00:45:45.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.392+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 4 10 2024-02-13T00:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:45:45.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae9010f700 10 client.37364.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:45.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 11 2024-02-13T00:45:45.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6804/2212296748 conn(0x7fae8811fdc0 0x7fae88120220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:45.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae8811f6e0 osd=5 3 2024-02-13T00:45:45.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae8811f6e0 osd=5 3 2024-02-13T00:45:45.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae8811f6e0 osd=5 3 2024-02-13T00:45:45.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 11 2024-02-13T00:45:45.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae8811f6e0 osd=5 3 2024-02-13T00:45:45.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 5 11 2024-02-13T00:45:45.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae8811f6e0 osd=5 4 2024-02-13T00:45:45.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae9010f700 10 client.37364.objecter _send_command 11 2024-02-13T00:45:45.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae88127120 con 0x7fae8811fdc0 2024-02-13T00:45:45.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.393+0000 7fae8d6aa700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6804/2212296748 conn(0x7fae8811fdc0 0x7fae88120220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:45.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.394+0000 7fae7effd700 10 client.37364.objecter ms_handle_connect 0x7fae8811fdc0 2024-02-13T00:45:45.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.394+0000 7fae8d6aa700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6804/2212296748 conn(0x7fae8811fdc0 0x7fae88120220 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:45.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.394+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fae88127120 con 0x7fae8811fdc0 2024-02-13T00:45:45.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.394+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:45:45.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.394+0000 7fae7effd700 10 client.37364.objecter _finish_command 11 = system:0 2024-02-13T00:45:45.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.394+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae8811f6e0 osd=5 4 2024-02-13T00:45:45.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.394+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 5 11 2024-02-13T00:45:45.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:45.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.413+0000 7fae9010f700 10 client.37364.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:45.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.414+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 12 2024-02-13T00:45:45.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.414+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae8811f6e0 osd=5 3 2024-02-13T00:45:45.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.414+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae8811f6e0 osd=5 3 2024-02-13T00:45:45.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.414+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae8811f6e0 osd=5 3 2024-02-13T00:45:45.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.414+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 12 2024-02-13T00:45:45.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.414+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae8811f6e0 osd=5 3 2024-02-13T00:45:45.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.414+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 5 12 2024-02-13T00:45:45.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.414+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae8811f6e0 osd=5 4 2024-02-13T00:45:45.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.414+0000 7fae9010f700 10 client.37364.objecter _send_command 12 2024-02-13T00:45:45.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.414+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fae88127120 con 0x7fae8811fdc0 2024-02-13T00:45:45.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fae88127120 con 0x7fae8811fdc0 2024-02-13T00:45:45.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:45.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae7effd700 10 client.37364.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae8811f6e0 osd=5 4 2024-02-13T00:45:45.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 5 12 2024-02-13T00:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:45:45.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae9010f700 10 client.37364.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:45.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 13 2024-02-13T00:45:45.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6808/1506640341 conn(0x7fae88124ea0 0x7fae881252c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:45.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88127120 osd=6 3 2024-02-13T00:45:45.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88127120 osd=6 3 2024-02-13T00:45:45.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88127120 osd=6 3 2024-02-13T00:45:45.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 13 2024-02-13T00:45:45.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae88127120 osd=6 3 2024-02-13T00:45:45.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 6 13 2024-02-13T00:45:45.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88127120 osd=6 4 2024-02-13T00:45:45.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae9010f700 10 client.37364.objecter _send_command 13 2024-02-13T00:45:45.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae88126f70 con 0x7fae88124ea0 2024-02-13T00:45:45.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.416+0000 7fae8e6ac700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6808/1506640341 conn(0x7fae88124ea0 0x7fae881252c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:45.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.417+0000 7fae7effd700 10 client.37364.objecter ms_handle_connect 0x7fae88124ea0 2024-02-13T00:45:45.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.417+0000 7fae8e6ac700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6808/1506640341 conn(0x7fae88124ea0 0x7fae881252c0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:45.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.417+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fae80007b20 con 0x7fae88124ea0 2024-02-13T00:45:45.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.417+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:45:45.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.417+0000 7fae7effd700 10 client.37364.objecter _finish_command 13 = system:0 2024-02-13T00:45:45.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.417+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae88127120 osd=6 4 2024-02-13T00:45:45.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.417+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 6 13 2024-02-13T00:45:45.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.434+0000 7fae9010f700 10 client.37364.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:45.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.434+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 14 2024-02-13T00:45:45.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.434+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88127120 osd=6 3 2024-02-13T00:45:45.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.434+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88127120 osd=6 3 2024-02-13T00:45:45.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.434+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88127120 osd=6 3 2024-02-13T00:45:45.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.434+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 14 2024-02-13T00:45:45.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.434+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae88127120 osd=6 3 2024-02-13T00:45:45.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.434+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 6 14 2024-02-13T00:45:45.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.434+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88127120 osd=6 4 2024-02-13T00:45:45.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.434+0000 7fae9010f700 10 client.37364.objecter _send_command 14 2024-02-13T00:45:45.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.434+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fae88129350 con 0x7fae88124ea0 2024-02-13T00:45:45.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fae88129350 con 0x7fae88124ea0 2024-02-13T00:45:45.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:45.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae7effd700 10 client.37364.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae88127120 osd=6 4 2024-02-13T00:45:45.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 6 14 2024-02-13T00:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:45:45.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae9010f700 10 client.37364.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:45.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 15 2024-02-13T00:45:45.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6812/1964192772 conn(0x7fae88129a00 0x7fae88129e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:45.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88129350 osd=7 3 2024-02-13T00:45:45.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88129350 osd=7 3 2024-02-13T00:45:45.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88129350 osd=7 3 2024-02-13T00:45:45.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 15 2024-02-13T00:45:45.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae88129350 osd=7 3 2024-02-13T00:45:45.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 7 15 2024-02-13T00:45:45.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88129350 osd=7 4 2024-02-13T00:45:45.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae9010f700 10 client.37364.objecter _send_command 15 2024-02-13T00:45:45.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae8812d710 con 0x7fae88129a00 2024-02-13T00:45:45.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.436+0000 7fae8deab700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6812/1964192772 conn(0x7fae88129a00 0x7fae88129e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:45.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.437+0000 7fae7effd700 10 client.37364.objecter ms_handle_connect 0x7fae88129a00 2024-02-13T00:45:45.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.437+0000 7fae8deab700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6812/1964192772 conn(0x7fae88129a00 0x7fae88129e60 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:45.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.438+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fae8812d710 con 0x7fae88129a00 2024-02-13T00:45:45.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.438+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:45:45.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.438+0000 7fae7effd700 10 client.37364.objecter _finish_command 15 = system:0 2024-02-13T00:45:45.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.438+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae88129350 osd=7 4 2024-02-13T00:45:45.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.438+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 7 15 2024-02-13T00:45:45.443 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:45.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.452+0000 7fae9010f700 10 client.37364.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:45:45.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.452+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign -1 16 2024-02-13T00:45:45.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.452+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88129350 osd=7 3 2024-02-13T00:45:45.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.452+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88129350 osd=7 3 2024-02-13T00:45:45.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.452+0000 7fae9010f700 20 client.37364.objecter _get_session s=0x7fae88129350 osd=7 3 2024-02-13T00:45:45.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.452+0000 7fae9010f700 15 client.37364.objecter _session_command_op_remove -1 16 2024-02-13T00:45:45.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.452+0000 7fae9010f700 20 client.37364.objecter get_session s=0x7fae88129350 osd=7 3 2024-02-13T00:45:45.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.452+0000 7fae9010f700 15 client.37364.objecter _session_command_op_assign 7 16 2024-02-13T00:45:45.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.452+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88129350 osd=7 4 2024-02-13T00:45:45.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.452+0000 7fae9010f700 10 client.37364.objecter _send_command 16 2024-02-13T00:45:45.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.452+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fae8812c590 con 0x7fae88129a00 2024-02-13T00:45:45.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae7effd700 1 -- 172.21.15.162:0/893101610 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fae8812c590 con 0x7fae88129a00 2024-02-13T00:45:45.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae7effd700 10 client.37364.objecter ms_dispatch 0x7fae88001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:45:45.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae7effd700 10 client.37364.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae7effd700 20 client.37364.objecter put_session s=0x7fae88129350 osd=7 4 2024-02-13T00:45:45.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae7effd700 15 client.37364.objecter _session_command_op_remove 7 16 2024-02-13T00:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:45:45.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 10 librados: watch_flush enter 2024-02-13T00:45:45.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 10 librados: watch_flush exit 2024-02-13T00:45:45.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 10 client.37364.objecter close_session for osd.0 2024-02-13T00:45:45.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6801/2103743338 conn(0x7fae880fdda0 msgr2=0x7fae880fe180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6801/2103743338 conn(0x7fae880fdda0 0x7fae880fe180 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88104f30 osd=0 1 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 10 client.37364.objecter close_session for osd.1 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6805/150599780 conn(0x7fae88059990 msgr2=0x7fae88059d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6805/150599780 conn(0x7fae88059990 0x7fae88059d90 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88028760 osd=1 1 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 10 client.37364.objecter close_session for osd.2 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6809/3689719376 conn(0x7fae880282c0 msgr2=0x7fae88118030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6809/3689719376 conn(0x7fae880282c0 0x7fae88118030 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae880597b0 osd=2 1 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 10 client.37364.objecter close_session for osd.3 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6813/604971011 conn(0x7fae8811b790 msgr2=0x7fae8811db80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae8e6ac700 1 -- 172.21.15.162:0/893101610 reap_dead start 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.454+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6813/604971011 conn(0x7fae8811b790 0x7fae8811db80 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88118f10 osd=3 1 2024-02-13T00:45:45.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 10 client.37364.objecter close_session for osd.4 2024-02-13T00:45:45.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6800/3344263388 conn(0x7fae88117b90 msgr2=0x7fae8811f1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6800/3344263388 conn(0x7fae88117b90 0x7fae8811f1f0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88118800 osd=4 1 2024-02-13T00:45:45.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 10 client.37364.objecter close_session for osd.5 2024-02-13T00:45:45.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6804/2212296748 conn(0x7fae8811fdc0 msgr2=0x7fae88120220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6804/2212296748 conn(0x7fae8811fdc0 0x7fae88120220 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae8811f6e0 osd=5 1 2024-02-13T00:45:45.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 10 client.37364.objecter close_session for osd.6 2024-02-13T00:45:45.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6808/1506640341 conn(0x7fae88124ea0 msgr2=0x7fae881252c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6808/1506640341 conn(0x7fae88124ea0 0x7fae881252c0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88127120 osd=6 1 2024-02-13T00:45:45.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 10 client.37364.objecter close_session for osd.7 2024-02-13T00:45:45.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6812/1964192772 conn(0x7fae88129a00 msgr2=0x7fae88129e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.181:6812/1964192772 conn(0x7fae88129a00 0x7fae88129e60 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 20 client.37364.objecter put_session s=0x7fae88129350 osd=7 1 2024-02-13T00:45:45.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 20 client.37364.objecter shutdown clearing up homeless session... 2024-02-13T00:45:45.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 10 client.37364.objecter successfully canceled tick 2024-02-13T00:45:45.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae8e6ac700 1 -- 172.21.15.162:0/893101610 reap_dead start 2024-02-13T00:45:45.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6800/2866534053 conn(0x7fae74062c90 msgr2=0x7fae74065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6800/2866534053 conn(0x7fae74062c90 0x7fae74065150 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 rx=0x7fae84004fd0 tx=0x7fae8400f040).stop 2024-02-13T00:45:45.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae880fcb30 msgr2=0x7fae8810c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:45.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae880fcb30 0x7fae8810c3c0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 rx=0x7fae7800d270 tx=0x7fae7800b040).stop 2024-02-13T00:45:45.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 shutdown_connections 2024-02-13T00:45:45.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae880fcb30 0x7fae8810c3c0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 --2- 172.21.15.162:0/893101610 >> v2:172.21.15.162:6800/2866534053 conn(0x7fae74062c90 0x7fae74065150 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:45.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 >> 172.21.15.162:0/893101610 conn(0x7fae8806eb90 msgr2=0x7fae880fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:45.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 shutdown_connections 2024-02-13T00:45:45.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 -- 172.21.15.162:0/893101610 wait complete. 2024-02-13T00:45:45.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:45.455+0000 7fae9010f700 1 librados: shutdown 2024-02-13T00:45:45.464 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.464 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.464 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.464 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.465 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.465 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.465 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.465 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:45:45.465 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:45.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:45.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:45.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:43.545018+0000 osd.0 (osd.0) 136 : cluster [DBG] 61.0 deep-scrub starts 2024-02-13T00:45:45.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:43.553737+0000 osd.0 (osd.0) 137 : cluster [DBG] 61.0 deep-scrub ok 2024-02-13T00:45:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:44.201317+0000 osd.5 (osd.5) 170 : cluster [DBG] 61.1c scrub starts 2024-02-13T00:45:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:44.205374+0000 osd.5 (osd.5) 171 : cluster [DBG] 61.1c scrub ok 2024-02-13T00:45:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:44.410832+0000 mgr.y (mgr.14544) 340 : cluster [DBG] pgmap v553: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:45:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:44.415604+0000 mon.a (mon.0) 2426 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-83985-65"}]': finished 2024-02-13T00:45:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:44.415670+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e389: 8 total, 8 up, 5 in 2024-02-13T00:45:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:45.115495+0000 mon.a (mon.0) 2428 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:45.305586+0000 mon.c (mon.2) 369 : audit [DBG] from='client.? 172.21.15.162:0/893101610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:43.545018+0000 osd.0 (osd.0) 136 : cluster [DBG] 61.0 deep-scrub starts 2024-02-13T00:45:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:43.553737+0000 osd.0 (osd.0) 137 : cluster [DBG] 61.0 deep-scrub ok 2024-02-13T00:45:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:44.201317+0000 osd.5 (osd.5) 170 : cluster [DBG] 61.1c scrub starts 2024-02-13T00:45:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:44.205374+0000 osd.5 (osd.5) 171 : cluster [DBG] 61.1c scrub ok 2024-02-13T00:45:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:44.410832+0000 mgr.y (mgr.14544) 340 : cluster [DBG] pgmap v553: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:45:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:44.415604+0000 mon.a (mon.0) 2426 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-83985-65"}]': finished 2024-02-13T00:45:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:44.415670+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e389: 8 total, 8 up, 5 in 2024-02-13T00:45:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:45.115495+0000 mon.a (mon.0) 2428 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:45.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:45.305586+0000 mon.c (mon.2) 369 : audit [DBG] from='client.? 172.21.15.162:0/893101610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:45.597 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:45.616 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:45.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:45.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:45.746 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:45:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:45:45.746 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:43.545018+0000 osd.0 (osd.0) 136 : cluster [DBG] 61.0 deep-scrub starts 2024-02-13T00:45:45.746 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:43.553737+0000 osd.0 (osd.0) 137 : cluster [DBG] 61.0 deep-scrub ok 2024-02-13T00:45:45.746 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:44.201317+0000 osd.5 (osd.5) 170 : cluster [DBG] 61.1c scrub starts 2024-02-13T00:45:45.746 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:44.205374+0000 osd.5 (osd.5) 171 : cluster [DBG] 61.1c scrub ok 2024-02-13T00:45:45.746 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:44.410832+0000 mgr.y (mgr.14544) 340 : cluster [DBG] pgmap v553: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:45:45.746 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:44.415604+0000 mon.a (mon.0) 2426 : audit [INF] from='client.? 172.21.15.162:0/879748064' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-83985-65"}]': finished 2024-02-13T00:45:45.746 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:44.415670+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e389: 8 total, 8 up, 5 in 2024-02-13T00:45:45.746 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:45.115495+0000 mon.a (mon.0) 2428 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:45.746 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:45.305586+0000 mon.c (mon.2) 369 : audit [DBG] from='client.? 172.21.15.162:0/893101610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:45.747 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:45.767 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:45.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:45.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:45.902 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:45.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:45:46.351 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:46.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:46.431 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:45:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:45:46] "GET /metrics HTTP/1.1" 200 189883 "" "Prometheus/2.33.4" 2024-02-13T00:45:46.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:46.483 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:46.503 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:46.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:46.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:46.634 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:46.671 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:46.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:46.698 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:45.246104+0000 osd.6 (osd.6) 93 : cluster [DBG] 61.e deep-scrub starts 2024-02-13T00:45:46.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:45.254003+0000 osd.6 (osd.6) 94 : cluster [DBG] 61.e deep-scrub ok 2024-02-13T00:45:46.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:45.587499+0000 mgr.y (mgr.14544) 341 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:46.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:45.246104+0000 osd.6 (osd.6) 93 : cluster [DBG] 61.e deep-scrub starts 2024-02-13T00:45:46.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:45.254003+0000 osd.6 (osd.6) 94 : cluster [DBG] 61.e deep-scrub ok 2024-02-13T00:45:46.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:45.587499+0000 mgr.y (mgr.14544) 341 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:46.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:46.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:45.246104+0000 osd.6 (osd.6) 93 : cluster [DBG] 61.e deep-scrub starts 2024-02-13T00:45:46.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:45.254003+0000 osd.6 (osd.6) 94 : cluster [DBG] 61.e deep-scrub ok 2024-02-13T00:45:46.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:45.587499+0000 mgr.y (mgr.14544) 341 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:46.803 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:47.130 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:47.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:47.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:47.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:47.261 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:47.380 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:45:47.380 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:45:47.383 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:47.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:47.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:47.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:47.518 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:47.539 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:47.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:47.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:47.671 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:47.690 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:47.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:46.155121+0000 osd.5 (osd.5) 172 : cluster [DBG] 61.16 scrub starts 2024-02-13T00:45:47.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:46.157225+0000 osd.5 (osd.5) 173 : cluster [DBG] 61.16 scrub ok 2024-02-13T00:45:47.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:46.189306+0000 osd.1 (osd.1) 159 : cluster [DBG] 61.b scrub starts 2024-02-13T00:45:47.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:46.193083+0000 osd.1 (osd.1) 160 : cluster [DBG] 61.b scrub ok 2024-02-13T00:45:47.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:46.411802+0000 mgr.y (mgr.14544) 342 : cluster [DBG] pgmap v555: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1021 B/s rd, 0 op/s 2024-02-13T00:45:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:47.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:47.794 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:47.822 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:47.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:46.155121+0000 osd.5 (osd.5) 172 : cluster [DBG] 61.16 scrub starts 2024-02-13T00:45:47.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:46.157225+0000 osd.5 (osd.5) 173 : cluster [DBG] 61.16 scrub ok 2024-02-13T00:45:47.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:46.189306+0000 osd.1 (osd.1) 159 : cluster [DBG] 61.b scrub starts 2024-02-13T00:45:47.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:46.193083+0000 osd.1 (osd.1) 160 : cluster [DBG] 61.b scrub ok 2024-02-13T00:45:47.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:46.411802+0000 mgr.y (mgr.14544) 342 : cluster [DBG] pgmap v555: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1021 B/s rd, 0 op/s 2024-02-13T00:45:47.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:46.155121+0000 osd.5 (osd.5) 172 : cluster [DBG] 61.16 scrub starts 2024-02-13T00:45:47.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:46.157225+0000 osd.5 (osd.5) 173 : cluster [DBG] 61.16 scrub ok 2024-02-13T00:45:47.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:46.189306+0000 osd.1 (osd.1) 159 : cluster [DBG] 61.b scrub starts 2024-02-13T00:45:47.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:46.193083+0000 osd.1 (osd.1) 160 : cluster [DBG] 61.b scrub ok 2024-02-13T00:45:47.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:46.411802+0000 mgr.y (mgr.14544) 342 : cluster [DBG] pgmap v555: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1021 B/s rd, 0 op/s 2024-02-13T00:45:47.840 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:47.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:47.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:45:47.972 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:47.990 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:45:47.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:47.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:45:48.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:48.126 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:48.146 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:48.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:48.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:48.277 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:48.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.547+0000 7fbc424c9700 1 -- 172.21.15.162:0/647825612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0f8480 msgr2=0x7fbc3c104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:48.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.547+0000 7fbc424c9700 1 --2- 172.21.15.162:0/647825612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0f8480 0x7fbc3c104a80 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 rx=0x7fbc34009be0 tx=0x7fbc34004d10).stop 2024-02-13T00:45:48.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.547+0000 7fbc424c9700 1 -- 172.21.15.162:0/647825612 shutdown_connections 2024-02-13T00:45:48.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.547+0000 7fbc424c9700 1 --2- 172.21.15.162:0/647825612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0f8480 0x7fbc3c104a80 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:48.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.547+0000 7fbc424c9700 1 --2- 172.21.15.162:0/647825612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc3c0fdc10 0x7fbc3c0fdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:45:48.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.547+0000 7fbc424c9700 1 --2- 172.21.15.162:0/647825612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc3c0fe570 0x7fbc3c0f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.547+0000 7fbc424c9700 1 -- 172.21.15.162:0/647825612 >> 172.21.15.162:0/647825612 conn(0x7fbc3c0f3800 msgr2=0x7fbc3c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.547+0000 7fbc424c9700 1 -- 172.21.15.162:0/647825612 shutdown_connections 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.547+0000 7fbc424c9700 1 -- 172.21.15.162:0/647825612 wait complete. 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.547+0000 7fbc424c9700 1 librados: starting msgr at 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.547+0000 7fbc424c9700 1 librados: starting objecter 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc424c9700 1 Processor -- start 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc424c9700 1 -- start start 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc424c9700 1 librados: setting wanted keys 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc424c9700 1 librados: calling monclient init 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc424c9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc3c0f8480 0x7fbc3c10e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc424c9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0fdc10 0x7fbc3c10e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc424c9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc3c0fe570 0x7fbc3c108230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc424c9700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c108720 con 0x7fbc3c0fdc10 2024-02-13T00:45:48.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc424c9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c108890 con 0x7fbc3c0fe570 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc424c9700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c108a00 con 0x7fbc3c0f8480 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc3b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0fdc10 0x7fbc3c10e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc3b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0fdc10 0x7fbc3c10e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:48326/0 (socket says 172.21.15.162:48326) 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc3b7fe700 1 -- 172.21.15.162:0/1605936927 learned_addr learned my addr 172.21.15.162:0/1605936927 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc3bfff700 1 --2- 172.21.15.162:0/1605936927 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc3c0f8480 0x7fbc3c10e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc40a66700 1 --2- 172.21.15.162:0/1605936927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc3c0fe570 0x7fbc3c108230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc3b7fe700 1 -- 172.21.15.162:0/1605936927 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc3c0f8480 msgr2=0x7fbc3c10e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc3b7fe700 1 --2- 172.21.15.162:0/1605936927 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc3c0f8480 0x7fbc3c10e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc3b7fe700 1 -- 172.21.15.162:0/1605936927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc3c0fe570 msgr2=0x7fbc3c108230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc3b7fe700 1 --2- 172.21.15.162:0/1605936927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc3c0fe570 0x7fbc3c108230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.548+0000 7fbc3b7fe700 1 -- 172.21.15.162:0/1605936927 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c005040 con 0x7fbc3c0fdc10 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.549+0000 7fbc3b7fe700 1 --2- 172.21.15.162:0/1605936927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0fdc10 0x7fbc3c10e4f0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7fbc2c0048f0 tx=0x7fbc2c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.549+0000 7fbc397fa700 10 client.?.objecter ms_handle_connect 0x7fbc3c0fdc10 2024-02-13T00:45:48.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.549+0000 7fbc397fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.549+0000 7fbc397fa700 1 -- 172.21.15.162:0/1605936927 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c007b00 con 0x7fbc3c0fdc10 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.549+0000 7fbc397fa700 1 -- 172.21.15.162:0/1605936927 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc2c00f040 con 0x7fbc3c0fdc10 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.549+0000 7fbc424c9700 1 -- 172.21.15.162:0/1605936927 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc340097c0 con 0x7fbc3c0fdc10 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.549+0000 7fbc397fa700 1 -- 172.21.15.162:0/1605936927 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c014990 con 0x7fbc3c0fdc10 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.549+0000 7fbc424c9700 10 client.27267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.549+0000 7fbc424c9700 1 -- 172.21.15.162:0/1605936927 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc3c109040 con 0x7fbc3c0fdc10 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.549+0000 7fbc424c9700 1 librados: init done 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.550+0000 7fbc424c9700 1 -- 172.21.15.162:0/1605936927 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3c05fb20 con 0x7fbc3c0fdc10 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.550+0000 7fbc397fa700 1 -- 172.21.15.162:0/1605936927 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fbc2c0133c0 con 0x7fbc3c0fdc10 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.550+0000 7fbc397fa700 1 --2- 172.21.15.162:0/1605936927 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbc24062c90 0x7fbc24065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.550+0000 7fbc397fa700 1 -- 172.21.15.162:0/1605936927 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(389..389 src has 52..389) v4 ==== 8072+0+0 (secure 0 0 0) 0x7fbc2c07ccf0 con 0x7fbc3c0fdc10 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.550+0000 7fbc397fa700 10 client.27267.objecter ms_dispatch 0x7fbc3c001550 osd_map(389..389 src has 52..389) v4 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.550+0000 7fbc397fa700 3 client.27267.objecter handle_osd_map got epochs [389,389] > 0 2024-02-13T00:45:48.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.550+0000 7fbc397fa700 3 client.27267.objecter handle_osd_map decoding full epoch 389 2024-02-13T00:45:48.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.551+0000 7fbc3bfff700 1 --2- 172.21.15.162:0/1605936927 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbc24062c90 0x7fbc24065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:48.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.551+0000 7fbc397fa700 20 client.27267.objecter dump_active .. 0 homeless 2024-02-13T00:45:48.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.551+0000 7fbc397fa700 10 client.27267.objecter ms_handle_connect 0x7fbc24062c90 2024-02-13T00:45:48.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.551+0000 7fbc3bfff700 1 --2- 172.21.15.162:0/1605936927 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbc24062c90 0x7fbc24065150 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 rx=0x7fbc30006fd0 tx=0x7fbc30009060).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:48.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.555+0000 7fbc397fa700 1 -- 172.21.15.162:0/1605936927 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fbc2c049f60 con 0x7fbc3c0fdc10 2024-02-13T00:45:48.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:48.731+0000 7fbc424c9700 1 -- 172.21.15.162:0/1605936927 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "28.18", "id": [5, 7, 0, 2, 4, 6]} v 0) v1 -- 0x7fbc3c1096c0 con 0x7fbc3c0fdc10 2024-02-13T00:45:48.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:45:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:45:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:45:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:46.555243+0000 osd.0 (osd.0) 138 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:45:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:46.556222+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:45:48.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:46.555243+0000 osd.0 (osd.0) 138 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:45:48.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:46.556222+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:45:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:46.555243+0000 osd.0 (osd.0) 138 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:45:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:46.556222+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:45:49.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-13T00:45:49.033 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:49.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:49.043 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:49.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:49.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:45:49.170 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:49.191 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:49.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:49.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:49.322 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:49.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.448+0000 7fbc397fa700 1 -- 172.21.15.162:0/1605936927 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "28.18", "id": [5, 7, 0, 2, 4, 6]}]=0 set 28.18 pg_upmap_items mapping to [5->7,0->2,4->6] v390) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbc2c081070 con 0x7fbc3c0fdc10 2024-02-13T00:45:49.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 10 librados: watch_flush enter 2024-02-13T00:45:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 10 librados: watch_flush exit 2024-02-13T00:45:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 20 client.27267.objecter shutdown clearing up homeless session... 2024-02-13T00:45:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 10 client.27267.objecter successfully canceled tick 2024-02-13T00:45:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 1 -- 172.21.15.162:0/1605936927 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbc24062c90 msgr2=0x7fbc24065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 1 --2- 172.21.15.162:0/1605936927 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbc24062c90 0x7fbc24065150 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 rx=0x7fbc30006fd0 tx=0x7fbc30009060).stop 2024-02-13T00:45:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 1 -- 172.21.15.162:0/1605936927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0fdc10 msgr2=0x7fbc3c10e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 1 --2- 172.21.15.162:0/1605936927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0fdc10 0x7fbc3c10e4f0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7fbc2c0048f0 tx=0x7fbc2c009cb0).stop 2024-02-13T00:45:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 1 -- 172.21.15.162:0/1605936927 shutdown_connections 2024-02-13T00:45:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 1 --2- 172.21.15.162:0/1605936927 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbc24062c90 0x7fbc24065150 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 1 --2- 172.21.15.162:0/1605936927 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc3c0fe570 0x7fbc3c108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 1 --2- 172.21.15.162:0/1605936927 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc3c0f8480 0x7fbc3c10e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:49.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 1 --2- 172.21.15.162:0/1605936927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0fdc10 0x7fbc3c10e4f0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:49.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.451+0000 7fbc424c9700 1 -- 172.21.15.162:0/1605936927 >> 172.21.15.162:0/1605936927 conn(0x7fbc3c0f3800 msgr2=0x7fbc3c0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:49.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.452+0000 7fbc424c9700 1 -- 172.21.15.162:0/1605936927 shutdown_connections 2024-02-13T00:45:49.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.452+0000 7fbc424c9700 1 -- 172.21.15.162:0/1605936927 wait complete. 2024-02-13T00:45:49.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:49.452+0000 7fbc424c9700 1 librados: shutdown 2024-02-13T00:45:49.457 INFO:teuthology.orchestra.run.smithi162.stderr:set 28.18 pg_upmap_items mapping to [5->7,0->2,4->6] 2024-02-13T00:45:49.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:48.412344+0000 mgr.y (mgr.14544) 343 : cluster [DBG] pgmap v556: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:45:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:48.443443+0000 mon.b (mon.1) 550 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:48.443962+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:48.733864+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? 172.21.15.162:0/1605936927' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "28.18", "id": [5, 7, 0, 2, 4, 6]}]: dispatch 2024-02-13T00:45:49.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:49.335502+0000 mon.a (mon.0) 2431 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:49.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:48.412344+0000 mgr.y (mgr.14544) 343 : cluster [DBG] pgmap v556: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:45:49.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:48.443443+0000 mon.b (mon.1) 550 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:49.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:48.443962+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:49.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:48.733864+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? 172.21.15.162:0/1605936927' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "28.18", "id": [5, 7, 0, 2, 4, 6]}]: dispatch 2024-02-13T00:45:49.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:49.335502+0000 mon.a (mon.0) 2431 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:48.412344+0000 mgr.y (mgr.14544) 343 : cluster [DBG] pgmap v556: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:45:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:48.443443+0000 mon.b (mon.1) 550 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:48.443962+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:48.733864+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? 172.21.15.162:0/1605936927' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "28.18", "id": [5, 7, 0, 2, 4, 6]}]: dispatch 2024-02-13T00:45:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:49.335502+0000 mon.a (mon.0) 2431 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:50.372 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:50.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:50.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:45:50.505 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:50.532 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:45:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:50.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:45:50.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:49.450071+0000 mon.a (mon.0) 2432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:45:50.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:49.450161+0000 mon.a (mon.0) 2433 : audit [INF] from='client.? 172.21.15.162:0/1605936927' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "28.18", "id": [5, 7, 0, 2, 4, 6]}]': finished 2024-02-13T00:45:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:49.450226+0000 mon.a (mon.0) 2434 : cluster [DBG] osdmap e390: 8 total, 8 up, 5 in 2024-02-13T00:45:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:49.451260+0000 mon.b (mon.1) 551 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:49.452353+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:50.061066+0000 osd.5 (osd.5) 174 : cluster [DBG] 61.a scrub starts 2024-02-13T00:45:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:50.405630+0000 mon.a (mon.0) 2436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]': finished 2024-02-13T00:45:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:50.405710+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e391: 8 total, 8 up, 5 in 2024-02-13T00:45:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:50.434511+0000 mon.b (mon.1) 552 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:50.435156+0000 mon.a (mon.0) 2438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:50.435710+0000 mon.b (mon.1) 553 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:50.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:50.436191+0000 mon.a (mon.0) 2439 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:50.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:49.450071+0000 mon.a (mon.0) 2432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:45:50.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:49.450161+0000 mon.a (mon.0) 2433 : audit [INF] from='client.? 172.21.15.162:0/1605936927' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "28.18", "id": [5, 7, 0, 2, 4, 6]}]': finished 2024-02-13T00:45:50.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:49.450226+0000 mon.a (mon.0) 2434 : cluster [DBG] osdmap e390: 8 total, 8 up, 5 in 2024-02-13T00:45:50.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:49.451260+0000 mon.b (mon.1) 551 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:50.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:49.452353+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:50.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:50.061066+0000 osd.5 (osd.5) 174 : cluster [DBG] 61.a scrub starts 2024-02-13T00:45:50.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:50.405630+0000 mon.a (mon.0) 2436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]': finished 2024-02-13T00:45:50.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:50.405710+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e391: 8 total, 8 up, 5 in 2024-02-13T00:45:50.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:50.434511+0000 mon.b (mon.1) 552 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:50.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:50.435156+0000 mon.a (mon.0) 2438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:50.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:50.435710+0000 mon.b (mon.1) 553 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:50.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:50.436191+0000 mon.a (mon.0) 2439 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:50.636 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:50.666 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:50.686 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:50.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:50.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:50.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:49.450071+0000 mon.a (mon.0) 2432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:45:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:49.450161+0000 mon.a (mon.0) 2433 : audit [INF] from='client.? 172.21.15.162:0/1605936927' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "28.18", "id": [5, 7, 0, 2, 4, 6]}]': finished 2024-02-13T00:45:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:49.450226+0000 mon.a (mon.0) 2434 : cluster [DBG] osdmap e390: 8 total, 8 up, 5 in 2024-02-13T00:45:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:49.451260+0000 mon.b (mon.1) 551 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:49.452353+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:50.061066+0000 osd.5 (osd.5) 174 : cluster [DBG] 61.a scrub starts 2024-02-13T00:45:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:50.405630+0000 mon.a (mon.0) 2436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]': finished 2024-02-13T00:45:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:50.405710+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e391: 8 total, 8 up, 5 in 2024-02-13T00:45:50.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:50.434511+0000 mon.b (mon.1) 552 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:50.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:50.435156+0000 mon.a (mon.0) 2438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:50.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:50.435710+0000 mon.b (mon.1) 553 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:50.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:50.436191+0000 mon.a (mon.0) 2439 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-49"}]: dispatch 2024-02-13T00:45:50.818 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:50.837 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:50.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:50.888 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:50.890 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:50.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:45:50.969 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:50.987 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:45:50.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:50.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:45:51.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:51.117 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:51.505 INFO:tasks.workunit.client.0.smithi162.stdout:4c720e9:::166:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:e694b040:::167:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:afa38db2:::168:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:77ba9f53:::169:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:87495034:::170:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:7c96bf0e:::171:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:dbe346cc:::172:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:e943ec24:::173:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:f97a9c0c:::174:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:6f26e74d:::175:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:4f95e106:::176:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:0e6f2f8f:::177:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:05db05f1:::178:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:38a78d66:::179:head 2024-02-13T00:45:51.506 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:d095610b:::180:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:a1a9d709:::181:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:1e5d39db:::182:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:f7df4fb9:::183:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:03a7f161:::184:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:ba70721e:::185:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:28e5662d:::186:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:973d52de:::187:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:4303eb1c:::188:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:b990b48e:::189:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:29b8165b:::190:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:3547f197:::191:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:7e260936:::192:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:1abec7b1:::193:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:10fdda93:::194:head 2024-02-13T00:45:51.507 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:15817eea:::195:head 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:770bab57:::196:head 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:ed9e13e7:::197:head 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:71471a8f:::198:head 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 189:10fb1d02:::199:head 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8066 ms) 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785083,0 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: first is 1707785083 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785083,0 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785083,0 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785083,1707785085,1707785086,0 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785083,1707785085,1707785086,0 2024-02-13T00:45:51.508 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785083,1707785085,1707785086,0 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785083,1707785085,1707785086,1707785088,1707785089,0 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785083,1707785085,1707785086,1707785088,1707785089,0 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785083,1707785085,1707785086,1707785088,1707785089,0 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785083,1707785085,1707785086,1707785088,1707785089,1707785091,1707785092,0 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785083,1707785085,1707785086,1707785088,1707785089,1707785091,1707785092,0 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785083,1707785085,1707785086,1707785088,1707785089,1707785091,1707785092,0 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785086,1707785088,1707785089,1707785091,1707785092,1707785094,1707785095,0 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: first now 1707785086, trimmed 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20957 ms) 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: foo0 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (13961 ms) 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-02-13T00:45:51.509 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (18067 ms) 2024-02-13T00:45:51.510 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-02-13T00:45:51.510 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:49.963613+0000 osd.6 (osd.6) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:51.510 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:49.963775+0000 osd.6 (osd.6) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:51.510 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:50.066050+0000 osd.5 (osd.5) 175 : cluster [DBG] 61.a scrub ok 2024-02-13T00:45:51.510 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:50.412775+0000 mgr.y (mgr.14544) 344 : cluster [DBG] pgmap v559: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-13T00:45:51.510 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:51.409420+0000 mon.a (mon.0) 2440 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-13T00:45:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:49.963613+0000 osd.6 (osd.6) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:49.963775+0000 osd.6 (osd.6) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:50.066050+0000 osd.5 (osd.5) 175 : cluster [DBG] 61.a scrub ok 2024-02-13T00:45:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:50.412775+0000 mgr.y (mgr.14544) 344 : cluster [DBG] pgmap v559: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-13T00:45:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:51.409420+0000 mon.a (mon.0) 2440 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-13T00:45:51.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:49.963613+0000 osd.6 (osd.6) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:51.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:49.963775+0000 osd.6 (osd.6) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:51.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:50.066050+0000 osd.5 (osd.5) 175 : cluster [DBG] 61.a scrub ok 2024-02-13T00:45:51.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:50.412775+0000 mgr.y (mgr.14544) 344 : cluster [DBG] pgmap v559: 238 pgs: 238 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-13T00:45:51.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:51.409420+0000 mon.a (mon.0) 2440 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-13T00:45:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:51.095444+0000 osd.5 (osd.5) 176 : cluster [DBG] 61.19 scrub starts 2024-02-13T00:45:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:51.098910+0000 osd.5 (osd.5) 177 : cluster [DBG] 61.19 scrub ok 2024-02-13T00:45:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:51.501787+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e392: 8 total, 8 up, 5 in 2024-02-13T00:45:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:51.711480+0000 osd.5 (osd.5) 178 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:51.711773+0000 osd.5 (osd.5) 179 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:52.819 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:51.095444+0000 osd.5 (osd.5) 176 : cluster [DBG] 61.19 scrub starts 2024-02-13T00:45:52.820 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:51.098910+0000 osd.5 (osd.5) 177 : cluster [DBG] 61.19 scrub ok 2024-02-13T00:45:52.820 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:51.501787+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e392: 8 total, 8 up, 5 in 2024-02-13T00:45:52.820 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:51.711480+0000 osd.5 (osd.5) 178 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:52.820 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:51.711773+0000 osd.5 (osd.5) 179 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:52.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:51.095444+0000 osd.5 (osd.5) 176 : cluster [DBG] 61.19 scrub starts 2024-02-13T00:45:52.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:51.098910+0000 osd.5 (osd.5) 177 : cluster [DBG] 61.19 scrub ok 2024-02-13T00:45:52.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:51.501787+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e392: 8 total, 8 up, 5 in 2024-02-13T00:45:52.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:51.711480+0000 osd.5 (osd.5) 178 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:45:52.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:51.711773+0000 osd.5 (osd.5) 179 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:45:53.333 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:45:53.333 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:45:53.667 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:53.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:53.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:51.639314+0000 osd.0 (osd.0) 140 : cluster [DBG] 61.1f deep-scrub starts 2024-02-13T00:45:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:51.651221+0000 osd.0 (osd.0) 141 : cluster [DBG] 61.1f deep-scrub ok 2024-02-13T00:45:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:52.413317+0000 mgr.y (mgr.14544) 345 : cluster [DBG] pgmap v561: 206 pgs: 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 170 B/s rd, 0 op/s 2024-02-13T00:45:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:52.515367+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e393: 8 total, 8 up, 5 in 2024-02-13T00:45:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:52.522214+0000 mon.b (mon.1) 554 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:52.523214+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:53.804 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:53.824 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:53.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:53.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:53.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:51.639314+0000 osd.0 (osd.0) 140 : cluster [DBG] 61.1f deep-scrub starts 2024-02-13T00:45:53.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:51.651221+0000 osd.0 (osd.0) 141 : cluster [DBG] 61.1f deep-scrub ok 2024-02-13T00:45:53.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:52.413317+0000 mgr.y (mgr.14544) 345 : cluster [DBG] pgmap v561: 206 pgs: 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 170 B/s rd, 0 op/s 2024-02-13T00:45:53.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:52.515367+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e393: 8 total, 8 up, 5 in 2024-02-13T00:45:53.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:52.522214+0000 mon.b (mon.1) 554 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:53.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:52.523214+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:51.639314+0000 osd.0 (osd.0) 140 : cluster [DBG] 61.1f deep-scrub starts 2024-02-13T00:45:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:51.651221+0000 osd.0 (osd.0) 141 : cluster [DBG] 61.1f deep-scrub ok 2024-02-13T00:45:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:52.413317+0000 mgr.y (mgr.14544) 345 : cluster [DBG] pgmap v561: 206 pgs: 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 170 B/s rd, 0 op/s 2024-02-13T00:45:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:52.515367+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e393: 8 total, 8 up, 5 in 2024-02-13T00:45:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:52.522214+0000 mon.b (mon.1) 554 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:52.523214+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:53.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:53.956 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:53.979 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:53.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:54.090 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:54.120 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:54.141 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:54.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:54.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:45:54.273 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:54.293 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:45:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:54.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:45:54.396 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:45:54.427 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:54.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-13T00:45:54.451 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:45:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:54.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:45:54.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:54.585 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:53.094557+0000 osd.1 (osd.1) 161 : cluster [DBG] 2.15 deep-scrub starts 2024-02-13T00:45:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:53.097477+0000 osd.1 (osd.1) 162 : cluster [DBG] 2.15 deep-scrub ok 2024-02-13T00:45:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:53.111741+0000 osd.6 (osd.6) 97 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:45:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:53.115255+0000 osd.6 (osd.6) 98 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:45:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:53.516441+0000 mon.a (mon.0) 2444 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:53.516534+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e394: 8 total, 8 up, 5 in 2024-02-13T00:45:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:53.591475+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:45:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:53.592316+0000 osd.0 (osd.0) 143 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:45:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:53.866111+0000 mon.b (mon.1) 555 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:53.866663+0000 mon.a (mon.0) 2446 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:53.867225+0000 mon.b (mon.1) 556 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-51"}]: dispatch 2024-02-13T00:45:54.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:53.867680+0000 mon.a (mon.0) 2447 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-51"}]: dispatch 2024-02-13T00:45:54.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:53.094557+0000 osd.1 (osd.1) 161 : cluster [DBG] 2.15 deep-scrub starts 2024-02-13T00:45:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:53.097477+0000 osd.1 (osd.1) 162 : cluster [DBG] 2.15 deep-scrub ok 2024-02-13T00:45:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:53.111741+0000 osd.6 (osd.6) 97 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:45:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:53.115255+0000 osd.6 (osd.6) 98 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:45:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:53.516441+0000 mon.a (mon.0) 2444 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:53.516534+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e394: 8 total, 8 up, 5 in 2024-02-13T00:45:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:53.591475+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:45:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:53.592316+0000 osd.0 (osd.0) 143 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:45:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:53.866111+0000 mon.b (mon.1) 555 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:53.866663+0000 mon.a (mon.0) 2446 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:53.867225+0000 mon.b (mon.1) 556 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-51"}]: dispatch 2024-02-13T00:45:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:53.867680+0000 mon.a (mon.0) 2447 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-51"}]: dispatch 2024-02-13T00:45:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:53.094557+0000 osd.1 (osd.1) 161 : cluster [DBG] 2.15 deep-scrub starts 2024-02-13T00:45:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:53.097477+0000 osd.1 (osd.1) 162 : cluster [DBG] 2.15 deep-scrub ok 2024-02-13T00:45:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:53.111741+0000 osd.6 (osd.6) 97 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:45:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:53.115255+0000 osd.6 (osd.6) 98 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:45:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:53.516441+0000 mon.a (mon.0) 2444 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:53.516534+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e394: 8 total, 8 up, 5 in 2024-02-13T00:45:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:53.591475+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:45:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:53.592316+0000 osd.0 (osd.0) 143 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:45:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:53.866111+0000 mon.b (mon.1) 555 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:53.866663+0000 mon.a (mon.0) 2446 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:53.867225+0000 mon.b (mon.1) 556 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-51"}]: dispatch 2024-02-13T00:45:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:53.867680+0000 mon.a (mon.0) 2447 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-51"}]: dispatch 2024-02-13T00:45:55.348 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 6] out_osds: [3, 2, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:45:55.349 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:45:55.349 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 4, 5, 6] 2024-02-13T00:45:55.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd out 5 2024-02-13T00:45:55.642 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:45:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:55.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:45:55.646 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:54.070900+0000 osd.1 (osd.1) 163 : cluster [DBG] 3.0 deep-scrub starts 2024-02-13T00:45:55.646 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:54.075190+0000 osd.1 (osd.1) 164 : cluster [DBG] 3.0 deep-scrub ok 2024-02-13T00:45:55.646 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:54.413751+0000 mgr.y (mgr.14544) 346 : cluster [DBG] pgmap v564: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:45:55.646 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:54.540513+0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e395: 8 total, 8 up, 5 in 2024-02-13T00:45:55.647 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:54.070900+0000 osd.1 (osd.1) 163 : cluster [DBG] 3.0 deep-scrub starts 2024-02-13T00:45:55.647 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:54.075190+0000 osd.1 (osd.1) 164 : cluster [DBG] 3.0 deep-scrub ok 2024-02-13T00:45:55.647 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:54.413751+0000 mgr.y (mgr.14544) 346 : cluster [DBG] pgmap v564: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:45:55.647 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:54.540513+0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e395: 8 total, 8 up, 5 in 2024-02-13T00:45:55.682 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:55.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:55.779 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:55.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:45:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:45:55.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:54.070900+0000 osd.1 (osd.1) 163 : cluster [DBG] 3.0 deep-scrub starts 2024-02-13T00:45:55.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:54.075190+0000 osd.1 (osd.1) 164 : cluster [DBG] 3.0 deep-scrub ok 2024-02-13T00:45:55.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:54.413751+0000 mgr.y (mgr.14544) 346 : cluster [DBG] pgmap v564: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:45:55.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:54.540513+0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e395: 8 total, 8 up, 5 in 2024-02-13T00:45:55.800 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:55.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:55.904 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:45:55.933 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:55.952 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:45:55.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:55.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:45:56.056 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:45:56.086 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:56.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:45:56] "GET /metrics HTTP/1.1" 200 187580 "" "Prometheus/2.33.4" 2024-02-13T00:45:56.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.565+0000 7f0e713eb700 1 -- 172.21.15.162:0/190792832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6c0ffd80 msgr2=0x7f0e6c100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:56.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.565+0000 7f0e713eb700 1 --2- 172.21.15.162:0/190792832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6c0ffd80 0x7f0e6c100160 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 rx=0x7f0e6400acf0 tx=0x7f0e64004d10).stop 2024-02-13T00:45:56.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.565+0000 7f0e713eb700 1 -- 172.21.15.162:0/190792832 shutdown_connections 2024-02-13T00:45:56.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.565+0000 7f0e713eb700 1 --2- 172.21.15.162:0/190792832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e6c1092e0 0x7f0e6c10b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:45:56.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.565+0000 7f0e713eb700 1 --2- 172.21.15.162:0/190792832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6c0ffd80 0x7f0e6c100160 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:56.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.565+0000 7f0e713eb700 1 --2- 172.21.15.162:0/190792832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e6c100650 0x7f0e6c108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:45:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.565+0000 7f0e713eb700 1 -- 172.21.15.162:0/190792832 >> 172.21.15.162:0/190792832 conn(0x7f0e6c063df0 msgr2=0x7f0e6c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.566+0000 7f0e713eb700 1 -- 172.21.15.162:0/190792832 shutdown_connections 2024-02-13T00:45:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.566+0000 7f0e713eb700 1 -- 172.21.15.162:0/190792832 wait complete. 2024-02-13T00:45:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.566+0000 7f0e713eb700 1 librados: starting msgr at 2024-02-13T00:45:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.566+0000 7f0e713eb700 1 librados: starting objecter 2024-02-13T00:45:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.566+0000 7f0e713eb700 1 Processor -- start 2024-02-13T00:45:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.567+0000 7f0e713eb700 1 -- start start 2024-02-13T00:45:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.567+0000 7f0e713eb700 1 librados: setting wanted keys 2024-02-13T00:45:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.567+0000 7f0e713eb700 1 librados: calling monclient init 2024-02-13T00:45:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.567+0000 7f0e713eb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e6c0ffd80 0x7f0e6c113880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.567+0000 7f0e713eb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6c100650 0x7f0e6c113db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.567+0000 7f0e713eb700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e6c1092e0 0x7f0e6c10eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.567+0000 7f0e713eb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c10f0d0 con 0x7f0e6c100650 2024-02-13T00:45:56.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.567+0000 7f0e713eb700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c10f240 con 0x7f0e6c0ffd80 2024-02-13T00:45:56.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.567+0000 7f0e713eb700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c10f400 con 0x7f0e6c1092e0 2024-02-13T00:45:56.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.568+0000 7f0e6a7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6c100650 0x7f0e6c113db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.568+0000 7f0e6a7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6c100650 0x7f0e6c113db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60210/0 (socket says 172.21.15.162:60210) 2024-02-13T00:45:56.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.568+0000 7f0e6a7fc700 1 -- 172.21.15.162:0/3259528496 learned_addr learned my addr 172.21.15.162:0/3259528496 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:56.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.568+0000 7f0e6b7fe700 1 --2- 172.21.15.162:0/3259528496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e6c1092e0 0x7f0e6c10eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.568+0000 7f0e6affd700 1 --2- 172.21.15.162:0/3259528496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e6c0ffd80 0x7f0e6c113880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.568+0000 7f0e6a7fc700 1 -- 172.21.15.162:0/3259528496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e6c1092e0 msgr2=0x7f0e6c10eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:56.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.568+0000 7f0e6a7fc700 1 --2- 172.21.15.162:0/3259528496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e6c1092e0 0x7f0e6c10eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:56.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e6a7fc700 1 -- 172.21.15.162:0/3259528496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e6c0ffd80 msgr2=0x7f0e6c113880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:56.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e6a7fc700 1 --2- 172.21.15.162:0/3259528496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e6c0ffd80 0x7f0e6c113880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:56.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e6a7fc700 1 -- 172.21.15.162:0/3259528496 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6400a8d0 con 0x7f0e6c100650 2024-02-13T00:45:56.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e6affd700 1 --2- 172.21.15.162:0/3259528496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e6c0ffd80 0x7f0e6c113880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:45:56.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e6a7fc700 1 --2- 172.21.15.162:0/3259528496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6c100650 0x7f0e6c113db0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f0e6000b530 tx=0x7f0e600090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:56.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e53fff700 10 client.?.objecter ms_handle_connect 0x7f0e6c100650 2024-02-13T00:45:56.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e53fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:56.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e53fff700 1 -- 172.21.15.162:0/3259528496 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e60002320 con 0x7f0e6c100650 2024-02-13T00:45:56.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e53fff700 1 -- 172.21.15.162:0/3259528496 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e60002950 con 0x7f0e6c100650 2024-02-13T00:45:56.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e53fff700 1 -- 172.21.15.162:0/3259528496 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e600129a0 con 0x7f0e6c100650 2024-02-13T00:45:56.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e713eb700 1 -- 172.21.15.162:0/3259528496 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6c10f6e0 con 0x7f0e6c100650 2024-02-13T00:45:56.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e713eb700 10 client.27273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:56.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e713eb700 1 -- 172.21.15.162:0/3259528496 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6c117860 con 0x7f0e6c100650 2024-02-13T00:45:56.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.569+0000 7f0e713eb700 1 librados: init done 2024-02-13T00:45:56.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.572+0000 7f0e713eb700 1 -- 172.21.15.162:0/3259528496 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6c110440 con 0x7f0e6c100650 2024-02-13T00:45:56.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.572+0000 7f0e53fff700 1 -- 172.21.15.162:0/3259528496 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0e60002480 con 0x7f0e6c100650 2024-02-13T00:45:56.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.573+0000 7f0e53fff700 1 --2- 172.21.15.162:0/3259528496 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0e54062c80 0x7f0e54065140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.573+0000 7f0e53fff700 1 -- 172.21.15.162:0/3259528496 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(397..397 src has 52..397) v4 ==== 8038+0+0 (secure 0 0 0) 0x7f0e6007d1d0 con 0x7f0e6c100650 2024-02-13T00:45:56.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.573+0000 7f0e53fff700 10 client.27273.objecter ms_dispatch 0x7f0e6c001550 osd_map(397..397 src has 52..397) v4 2024-02-13T00:45:56.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.573+0000 7f0e53fff700 3 client.27273.objecter handle_osd_map got epochs [397,397] > 0 2024-02-13T00:45:56.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.573+0000 7f0e53fff700 3 client.27273.objecter handle_osd_map decoding full epoch 397 2024-02-13T00:45:56.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.573+0000 7f0e6affd700 1 --2- 172.21.15.162:0/3259528496 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0e54062c80 0x7f0e54065140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.573+0000 7f0e53fff700 20 client.27273.objecter dump_active .. 0 homeless 2024-02-13T00:45:56.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.573+0000 7f0e53fff700 10 client.27273.objecter ms_handle_connect 0x7f0e54062c80 2024-02-13T00:45:56.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.573+0000 7f0e6affd700 1 --2- 172.21.15.162:0/3259528496 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0e54062c80 0x7f0e54065140 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7f0e64005010 tx=0x7f0e64004d10).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:56.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.578+0000 7f0e53fff700 1 -- 172.21.15.162:0/3259528496 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0e60050790 con 0x7f0e6c100650 2024-02-13T00:45:56.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.765+0000 7f0e713eb700 1 -- 172.21.15.162:0/3259528496 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f0e6c118cc0 con 0x7f0e6c100650 2024-02-13T00:45:56.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:55.067605+0000 osd.1 (osd.1) 165 : cluster [DBG] 61.5 scrub starts 2024-02-13T00:45:56.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:55.071113+0000 osd.1 (osd.1) 166 : cluster [DBG] 61.5 scrub ok 2024-02-13T00:45:56.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:55.546464+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e396: 8 total, 8 up, 5 in 2024-02-13T00:45:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:55.556713+0000 mon.b (mon.1) 557 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:55.557727+0000 mon.a (mon.0) 2450 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:55.598246+0000 mgr.y (mgr.14544) 347 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:55.067605+0000 osd.1 (osd.1) 165 : cluster [DBG] 61.5 scrub starts 2024-02-13T00:45:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:55.071113+0000 osd.1 (osd.1) 166 : cluster [DBG] 61.5 scrub ok 2024-02-13T00:45:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:55.546464+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e396: 8 total, 8 up, 5 in 2024-02-13T00:45:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:55.556713+0000 mon.b (mon.1) 557 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:55.557727+0000 mon.a (mon.0) 2450 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:56.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:55.598246+0000 mgr.y (mgr.14544) 347 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:56.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 -- 172.21.15.162:0/3598952603 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63040f9b70 msgr2=0x7f63040f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:56.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3598952603 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63040f9b70 0x7f63040f9f50 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7f62f4004870 tx=0x7f62f4009cb0).stop 2024-02-13T00:45:56.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 -- 172.21.15.162:0/3598952603 shutdown_connections 2024-02-13T00:45:56.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3598952603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63040f8500 0x7f6304104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:56.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3598952603 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63040f9b70 0x7f63040f9f50 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:56.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3598952603 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63040fa440 0x7f63040f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:56.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 -- 172.21.15.162:0/3598952603 >> 172.21.15.162:0/3598952603 conn(0x7f63040f38c0 msgr2=0x7f63040f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:56.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 -- 172.21.15.162:0/3598952603 shutdown_connections 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 -- 172.21.15.162:0/3598952603 wait complete. 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 librados: starting msgr at 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 librados: starting objecter 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 Processor -- start 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 -- start start 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 librados: setting wanted keys 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.879+0000 7f630bfb6700 1 librados: calling monclient init 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f630bfb6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63040f8500 0x7f630410e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f630bfb6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63040f9b70 0x7f630410e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f630bfb6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63040fa440 0x7f6304108310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f630bfb6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6304108800 con 0x7f63040fa440 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f630bfb6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6304108970 con 0x7f63040f8500 2024-02-13T00:45:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f630bfb6700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6304108ae0 con 0x7f63040f9b70 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f6309551700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63040f9b70 0x7f630410e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f630a553700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63040fa440 0x7f6304108310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f6309551700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63040f9b70 0x7f630410e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:49168/0 (socket says 172.21.15.162:49168) 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f6309551700 1 -- 172.21.15.162:0/3909254915 learned_addr learned my addr 172.21.15.162:0/3909254915 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f6309d52700 1 --2- 172.21.15.162:0/3909254915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63040f8500 0x7f630410e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f6309551700 1 -- 172.21.15.162:0/3909254915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63040f8500 msgr2=0x7f630410e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f6309551700 1 --2- 172.21.15.162:0/3909254915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63040f8500 0x7f630410e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f6309551700 1 -- 172.21.15.162:0/3909254915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63040fa440 msgr2=0x7f6304108310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f6309551700 1 --2- 172.21.15.162:0/3909254915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63040fa440 0x7f6304108310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f6309551700 1 -- 172.21.15.162:0/3909254915 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f4005040 con 0x7f63040f9b70 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f630a553700 1 --2- 172.21.15.162:0/3909254915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63040fa440 0x7f6304108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f6309551700 1 --2- 172.21.15.162:0/3909254915 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63040f9b70 0x7f630410e610 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 rx=0x7f630000c7c0 tx=0x7f6300007ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f62faffd700 10 client.?.objecter ms_handle_connect 0x7f63040f9b70 2024-02-13T00:45:56.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f62faffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:45:56.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300004430 con 0x7f63040f9b70 2024-02-13T00:45:56.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f630000e970 con 0x7f63040f9b70 2024-02-13T00:45:56.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6304108dc0 con 0x7f63040f9b70 2024-02-13T00:45:56.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300013a50 con 0x7f63040f9b70 2024-02-13T00:45:56.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f630bfb6700 10 client.37379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:45:56.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6304114fd0 con 0x7f63040f9b70 2024-02-13T00:45:56.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.880+0000 7f630bfb6700 1 librados: init done 2024-02-13T00:45:56.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.881+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f63040475e0 con 0x7f63040f9b70 2024-02-13T00:45:56.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f630000eae0 con 0x7f63040f9b70 2024-02-13T00:45:56.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f62faffd700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6800/2866534053 conn(0x7f62f0062cc0 0x7f62f0065180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(397..397 src has 52..397) v4 ==== 8038+0+0 (secure 0 0 0) 0x7f630007d3e0 con 0x7f63040f9b70 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 osd_map(397..397 src has 52..397) v4 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f62faffd700 3 client.37379.objecter handle_osd_map got epochs [397,397] > 0 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f62faffd700 3 client.37379.objecter handle_osd_map decoding full epoch 397 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f6309d52700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6800/2866534053 conn(0x7f62f0062cc0 0x7f62f0065180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f62faffd700 20 client.37379.objecter dump_active .. 0 homeless 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v397) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6300084070 con 0x7f63040f9b70 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f630bfb6700 10 client.37379.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 1 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6304028590 0x7f63040fea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304107210 osd=0 3 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304107210 osd=0 3 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304107210 osd=0 3 2024-02-13T00:45:56.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 1 2024-02-13T00:45:56.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f6304107210 osd=0 3 2024-02-13T00:45:56.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 0 1 2024-02-13T00:45:56.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304107210 osd=0 4 2024-02-13T00:45:56.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f630bfb6700 10 client.37379.objecter _send_command 1 2024-02-13T00:45:56.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63041000d0 con 0x7f6304028590 2024-02-13T00:45:56.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.882+0000 7f6309d52700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6800/2866534053 conn(0x7f62f0062cc0 0x7f62f0065180 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7f62f40043d0 tx=0x7f62f400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:56.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.883+0000 7f62faffd700 10 client.37379.objecter ms_handle_connect 0x7f62f0062cc0 2024-02-13T00:45:56.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.883+0000 7f630a553700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6304028590 0x7f63040fea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.883+0000 7f62faffd700 10 client.37379.objecter ms_handle_connect 0x7f6304028590 2024-02-13T00:45:56.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.883+0000 7f630a553700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6304028590 0x7f63040fea60 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:56.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.883+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f63041000d0 con 0x7f6304028590 2024-02-13T00:45:56.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.883+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:45:56.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.883+0000 7f62faffd700 10 client.37379.objecter _finish_command 1 = system:0 2024-02-13T00:45:56.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.883+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f6304107210 osd=0 4 2024-02-13T00:45:56.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.883+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 0 1 2024-02-13T00:45:56.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.898+0000 7f630bfb6700 10 client.37379.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:56.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.898+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 2 2024-02-13T00:45:56.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.898+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304107210 osd=0 3 2024-02-13T00:45:56.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.898+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304107210 osd=0 3 2024-02-13T00:45:56.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.898+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304107210 osd=0 3 2024-02-13T00:45:56.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.898+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 2 2024-02-13T00:45:56.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.898+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f6304107210 osd=0 3 2024-02-13T00:45:56.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.898+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 0 2 2024-02-13T00:45:56.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.898+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304107210 osd=0 4 2024-02-13T00:45:56.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.898+0000 7f630bfb6700 10 client.37379.objecter _send_command 2 2024-02-13T00:45:56.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.898+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f630405a0e0 con 0x7f6304028590 2024-02-13T00:45:56.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f630405a0e0 con 0x7f6304028590 2024-02-13T00:45:56.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:56.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f62faffd700 10 client.37379.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:56.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f6304107210 osd=0 4 2024-02-13T00:45:56.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 0 2 2024-02-13T00:45:56.904 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:45:56.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f630bfb6700 10 client.37379.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:56.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 3 2024-02-13T00:45:56.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6805/150599780 conn(0x7f6304059570 0x7f6304119180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304059400 osd=1 3 2024-02-13T00:45:56.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304059400 osd=1 3 2024-02-13T00:45:56.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304059400 osd=1 3 2024-02-13T00:45:56.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 3 2024-02-13T00:45:56.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f6304059400 osd=1 3 2024-02-13T00:45:56.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 1 3 2024-02-13T00:45:56.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304059400 osd=1 4 2024-02-13T00:45:56.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f630bfb6700 10 client.37379.objecter _send_command 3 2024-02-13T00:45:56.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63040280f0 con 0x7f6304059570 2024-02-13T00:45:56.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f6309d52700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6805/150599780 conn(0x7f6304059570 0x7f6304119180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f62faffd700 10 client.37379.objecter ms_handle_connect 0x7f6304059570 2024-02-13T00:45:56.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.900+0000 7f6309d52700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6805/150599780 conn(0x7f6304059570 0x7f6304119180 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:56.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.901+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f63040280f0 con 0x7f6304059570 2024-02-13T00:45:56.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.901+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:45:56.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.901+0000 7f62faffd700 10 client.37379.objecter _finish_command 3 = system:0 2024-02-13T00:45:56.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.901+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f6304059400 osd=1 4 2024-02-13T00:45:56.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.901+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 1 3 2024-02-13T00:45:56.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.915+0000 7f630bfb6700 10 client.37379.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:56.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.915+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 4 2024-02-13T00:45:56.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.915+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304059400 osd=1 3 2024-02-13T00:45:56.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.915+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304059400 osd=1 3 2024-02-13T00:45:56.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.915+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304059400 osd=1 3 2024-02-13T00:45:56.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.915+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 4 2024-02-13T00:45:56.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.915+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f6304059400 osd=1 3 2024-02-13T00:45:56.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.915+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 1 4 2024-02-13T00:45:56.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.915+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304059400 osd=1 4 2024-02-13T00:45:56.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.915+0000 7f630bfb6700 10 client.37379.objecter _send_command 4 2024-02-13T00:45:56.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.915+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f630406afb0 con 0x7f6304059570 2024-02-13T00:45:56.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f630406afb0 con 0x7f6304059570 2024-02-13T00:45:56.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:56.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f62faffd700 10 client.37379.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:56.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f6304059400 osd=1 4 2024-02-13T00:45:56.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 1 4 2024-02-13T00:45:56.922 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:45:56.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f630bfb6700 10 client.37379.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:56.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 5 2024-02-13T00:45:56.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6809/3689719376 conn(0x7f63040277c0 0x7f630406c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304027650 osd=2 3 2024-02-13T00:45:56.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304027650 osd=2 3 2024-02-13T00:45:56.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304027650 osd=2 3 2024-02-13T00:45:56.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 5 2024-02-13T00:45:56.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f6304027650 osd=2 3 2024-02-13T00:45:56.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 2 5 2024-02-13T00:45:56.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304027650 osd=2 4 2024-02-13T00:45:56.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f630bfb6700 10 client.37379.objecter _send_command 5 2024-02-13T00:45:56.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63040ffc30 con 0x7f63040277c0 2024-02-13T00:45:56.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f6309551700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6809/3689719376 conn(0x7f63040277c0 0x7f630406c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f62faffd700 10 client.37379.objecter ms_handle_connect 0x7f63040277c0 2024-02-13T00:45:56.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.917+0000 7f6309551700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6809/3689719376 conn(0x7f63040277c0 0x7f630406c330 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:56.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.918+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f63040ffc30 con 0x7f63040277c0 2024-02-13T00:45:56.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.918+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:45:56.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.918+0000 7f62faffd700 10 client.37379.objecter _finish_command 5 = system:0 2024-02-13T00:45:56.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.918+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f6304027650 osd=2 4 2024-02-13T00:45:56.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.918+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 2 5 2024-02-13T00:45:56.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.932+0000 7f630bfb6700 10 client.37379.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:56.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.932+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 6 2024-02-13T00:45:56.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.932+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304027650 osd=2 3 2024-02-13T00:45:56.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.932+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304027650 osd=2 3 2024-02-13T00:45:56.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.932+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304027650 osd=2 3 2024-02-13T00:45:56.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.932+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 6 2024-02-13T00:45:56.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.932+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f6304027650 osd=2 3 2024-02-13T00:45:56.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.932+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 2 6 2024-02-13T00:45:56.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.932+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304027650 osd=2 4 2024-02-13T00:45:56.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.932+0000 7f630bfb6700 10 client.37379.objecter _send_command 6 2024-02-13T00:45:56.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.932+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f630406c820 con 0x7f63040277c0 2024-02-13T00:45:56.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.933+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f630406c820 con 0x7f63040277c0 2024-02-13T00:45:56.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.933+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:56.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.933+0000 7f62faffd700 10 client.37379.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:56.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.933+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f6304027650 osd=2 4 2024-02-13T00:45:56.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.933+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 2 6 2024-02-13T00:45:56.939 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:45:56.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630bfb6700 10 client.37379.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:56.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 7 2024-02-13T00:45:56.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6813/604971011 conn(0x7f63040ff300 0x7f6304121fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f63040ff190 osd=3 3 2024-02-13T00:45:56.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f63040ff190 osd=3 3 2024-02-13T00:45:56.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f63040ff190 osd=3 3 2024-02-13T00:45:56.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 7 2024-02-13T00:45:56.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f63040ff190 osd=3 3 2024-02-13T00:45:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 3 7 2024-02-13T00:45:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f63040ff190 osd=3 4 2024-02-13T00:45:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630bfb6700 10 client.37379.objecter _send_command 7 2024-02-13T00:45:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6304124c00 con 0x7f63040ff300 2024-02-13T00:45:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630a553700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6813/604971011 conn(0x7f63040ff300 0x7f6304121fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f62faffd700 10 client.37379.objecter ms_handle_connect 0x7f63040ff300 2024-02-13T00:45:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.934+0000 7f630a553700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6813/604971011 conn(0x7f63040ff300 0x7f6304121fb0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.935+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6304124c00 con 0x7f63040ff300 2024-02-13T00:45:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.935+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:45:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.935+0000 7f62faffd700 10 client.37379.objecter _finish_command 7 = system:0 2024-02-13T00:45:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.935+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f63040ff190 osd=3 4 2024-02-13T00:45:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.935+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 3 7 2024-02-13T00:45:56.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.950+0000 7f630bfb6700 10 client.37379.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:56.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.950+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 8 2024-02-13T00:45:56.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.950+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f63040ff190 osd=3 3 2024-02-13T00:45:56.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.950+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f63040ff190 osd=3 3 2024-02-13T00:45:56.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.950+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f63040ff190 osd=3 3 2024-02-13T00:45:56.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.950+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 8 2024-02-13T00:45:56.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.950+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f63040ff190 osd=3 3 2024-02-13T00:45:56.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.950+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 3 8 2024-02-13T00:45:56.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.950+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f63040ff190 osd=3 4 2024-02-13T00:45:56.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.950+0000 7f630bfb6700 10 client.37379.objecter _send_command 8 2024-02-13T00:45:56.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.950+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f630406cbd0 con 0x7f63040ff300 2024-02-13T00:45:56.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f630406cbd0 con 0x7f63040ff300 2024-02-13T00:45:56.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:56.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f62faffd700 10 client.37379.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:56.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f63040ff190 osd=3 4 2024-02-13T00:45:56.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 3 8 2024-02-13T00:45:56.956 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:45:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f630bfb6700 10 client.37379.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 9 2024-02-13T00:45:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6800/3344263388 conn(0x7f630406cdd0 0x7f6304123620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f630406cbd0 osd=4 3 2024-02-13T00:45:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630406cbd0 osd=4 3 2024-02-13T00:45:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f630406cbd0 osd=4 3 2024-02-13T00:45:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 9 2024-02-13T00:45:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f630406cbd0 osd=4 3 2024-02-13T00:45:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 4 9 2024-02-13T00:45:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630406cbd0 osd=4 4 2024-02-13T00:45:56.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f630bfb6700 10 client.37379.objecter _send_command 9 2024-02-13T00:45:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.951+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f630406c0a0 con 0x7f630406cdd0 2024-02-13T00:45:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.952+0000 7f6309d52700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6800/3344263388 conn(0x7f630406cdd0 0x7f6304123620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.952+0000 7f62faffd700 10 client.37379.objecter ms_handle_connect 0x7f630406cdd0 2024-02-13T00:45:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.952+0000 7f6309d52700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6800/3344263388 conn(0x7f630406cdd0 0x7f6304123620 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.953+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f630406c0a0 con 0x7f630406cdd0 2024-02-13T00:45:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.953+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:45:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.953+0000 7f62faffd700 10 client.37379.objecter _finish_command 9 = system:0 2024-02-13T00:45:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.953+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f630406cbd0 osd=4 4 2024-02-13T00:45:56.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.953+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 4 9 2024-02-13T00:45:56.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.968+0000 7f630bfb6700 10 client.37379.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:56.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.968+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 10 2024-02-13T00:45:56.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.968+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f630406cbd0 osd=4 3 2024-02-13T00:45:56.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.968+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630406cbd0 osd=4 3 2024-02-13T00:45:56.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.968+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f630406cbd0 osd=4 3 2024-02-13T00:45:56.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.968+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 10 2024-02-13T00:45:56.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.968+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f630406cbd0 osd=4 3 2024-02-13T00:45:56.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.968+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 4 10 2024-02-13T00:45:56.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.968+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630406cbd0 osd=4 4 2024-02-13T00:45:56.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.968+0000 7f630bfb6700 10 client.37379.objecter _send_command 10 2024-02-13T00:45:56.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.968+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f630406c0a0 con 0x7f630406cdd0 2024-02-13T00:45:56.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.969+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f630406c0a0 con 0x7f630406cdd0 2024-02-13T00:45:56.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.969+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:56.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.969+0000 7f62faffd700 10 client.37379.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:56.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.969+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f630406cbd0 osd=4 4 2024-02-13T00:45:56.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.969+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 4 10 2024-02-13T00:45:56.974 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:45:56.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f630bfb6700 10 client.37379.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:56.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 11 2024-02-13T00:45:56.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6804/2212296748 conn(0x7f6304124020 0x7f6304124460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f630406c0a0 osd=5 3 2024-02-13T00:45:56.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630406c0a0 osd=5 3 2024-02-13T00:45:56.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f630406c0a0 osd=5 3 2024-02-13T00:45:56.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 11 2024-02-13T00:45:56.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f630406c0a0 osd=5 3 2024-02-13T00:45:56.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 5 11 2024-02-13T00:45:56.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630406c0a0 osd=5 4 2024-02-13T00:45:56.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f630bfb6700 10 client.37379.objecter _send_command 11 2024-02-13T00:45:56.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f630412b4f0 con 0x7f6304124020 2024-02-13T00:45:56.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f6309551700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6804/2212296748 conn(0x7f6304124020 0x7f6304124460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f62faffd700 10 client.37379.objecter ms_handle_connect 0x7f6304124020 2024-02-13T00:45:56.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.970+0000 7f6309551700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6804/2212296748 conn(0x7f6304124020 0x7f6304124460 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:56.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.971+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f630412b4f0 con 0x7f6304124020 2024-02-13T00:45:56.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.971+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:45:56.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.971+0000 7f62faffd700 10 client.37379.objecter _finish_command 11 = system:0 2024-02-13T00:45:56.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.971+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f630406c0a0 osd=5 4 2024-02-13T00:45:56.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.971+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 5 11 2024-02-13T00:45:56.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.985+0000 7f630bfb6700 10 client.37379.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:56.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.985+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 12 2024-02-13T00:45:56.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.985+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f630406c0a0 osd=5 3 2024-02-13T00:45:56.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.985+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630406c0a0 osd=5 3 2024-02-13T00:45:56.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.985+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f630406c0a0 osd=5 3 2024-02-13T00:45:56.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.985+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 12 2024-02-13T00:45:56.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.985+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f630406c0a0 osd=5 3 2024-02-13T00:45:56.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.985+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 5 12 2024-02-13T00:45:56.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.985+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630406c0a0 osd=5 4 2024-02-13T00:45:56.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.985+0000 7f630bfb6700 10 client.37379.objecter _send_command 12 2024-02-13T00:45:56.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.985+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6304128d90 con 0x7f6304124020 2024-02-13T00:45:56.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.987+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6304128d90 con 0x7f6304124020 2024-02-13T00:45:56.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.987+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:56.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.987+0000 7f62faffd700 10 client.37379.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:56.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.987+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f630406c0a0 osd=5 4 2024-02-13T00:45:56.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.987+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 5 12 2024-02-13T00:45:56.992 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:45:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630bfb6700 10 client.37379.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 13 2024-02-13T00:45:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6304129410 0x7f630412b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304128d90 osd=6 3 2024-02-13T00:45:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304128d90 osd=6 3 2024-02-13T00:45:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304128d90 osd=6 3 2024-02-13T00:45:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 13 2024-02-13T00:45:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f6304128d90 osd=6 3 2024-02-13T00:45:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 6 13 2024-02-13T00:45:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304128d90 osd=6 4 2024-02-13T00:45:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630bfb6700 10 client.37379.objecter _send_command 13 2024-02-13T00:45:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f630412e630 con 0x7f6304129410 2024-02-13T00:45:56.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630a553700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6304129410 0x7f630412b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:56.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f62faffd700 10 client.37379.objecter ms_handle_connect 0x7f6304129410 2024-02-13T00:45:56.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.988+0000 7f630a553700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6304129410 0x7f630412b860 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:56.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.989+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f630412e630 con 0x7f6304129410 2024-02-13T00:45:56.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.989+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:45:56.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.989+0000 7f62faffd700 10 client.37379.objecter _finish_command 13 = system:0 2024-02-13T00:45:56.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.989+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f6304128d90 osd=6 4 2024-02-13T00:45:56.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:56.989+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 6 13 2024-02-13T00:45:57.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.004+0000 7f630bfb6700 10 client.37379.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:57.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.004+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 14 2024-02-13T00:45:57.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.004+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304128d90 osd=6 3 2024-02-13T00:45:57.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.004+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304128d90 osd=6 3 2024-02-13T00:45:57.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.004+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f6304128d90 osd=6 3 2024-02-13T00:45:57.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.004+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 14 2024-02-13T00:45:57.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.004+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f6304128d90 osd=6 3 2024-02-13T00:45:57.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.004+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 6 14 2024-02-13T00:45:57.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.004+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304128d90 osd=6 4 2024-02-13T00:45:57.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.004+0000 7f630bfb6700 10 client.37379.objecter _send_command 14 2024-02-13T00:45:57.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.004+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f630412e630 con 0x7f6304129410 2024-02-13T00:45:57.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f630412e630 con 0x7f6304129410 2024-02-13T00:45:57.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:57.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f62faffd700 10 client.37379.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:57.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f6304128d90 osd=6 4 2024-02-13T00:45:57.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 6 14 2024-02-13T00:45:57.010 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:45:57.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f630bfb6700 10 client.37379.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:45:57.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 15 2024-02-13T00:45:57.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6812/1964192772 conn(0x7f630412d930 0x7f630412ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:45:57.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f630412e630 osd=7 3 2024-02-13T00:45:57.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630412e630 osd=7 3 2024-02-13T00:45:57.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f630412e630 osd=7 3 2024-02-13T00:45:57.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 15 2024-02-13T00:45:57.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f630412e630 osd=7 3 2024-02-13T00:45:57.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 7 15 2024-02-13T00:45:57.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630412e630 osd=7 4 2024-02-13T00:45:57.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f630bfb6700 10 client.37379.objecter _send_command 15 2024-02-13T00:45:57.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f630412e420 con 0x7f630412d930 2024-02-13T00:45:57.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f6309d52700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6812/1964192772 conn(0x7f630412d930 0x7f630412ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:45:57.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f62faffd700 10 client.37379.objecter ms_handle_connect 0x7f630412d930 2024-02-13T00:45:57.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.006+0000 7f6309d52700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6812/1964192772 conn(0x7f630412d930 0x7f630412ddb0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:45:57.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.008+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f630412e420 con 0x7f630412d930 2024-02-13T00:45:57.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.008+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:45:57.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.008+0000 7f62faffd700 10 client.37379.objecter _finish_command 15 = system:0 2024-02-13T00:45:57.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.008+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f630412e630 osd=7 4 2024-02-13T00:45:57.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.008+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 7 15 2024-02-13T00:45:57.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.022+0000 7f630bfb6700 10 client.37379.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:45:57.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.022+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign -1 16 2024-02-13T00:45:57.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.022+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f630412e630 osd=7 3 2024-02-13T00:45:57.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.022+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630412e630 osd=7 3 2024-02-13T00:45:57.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.022+0000 7f630bfb6700 20 client.37379.objecter _get_session s=0x7f630412e630 osd=7 3 2024-02-13T00:45:57.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.022+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_remove -1 16 2024-02-13T00:45:57.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.022+0000 7f630bfb6700 20 client.37379.objecter get_session s=0x7f630412e630 osd=7 3 2024-02-13T00:45:57.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.022+0000 7f630bfb6700 15 client.37379.objecter _session_command_op_assign 7 16 2024-02-13T00:45:57.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.022+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630412e630 osd=7 4 2024-02-13T00:45:57.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.022+0000 7f630bfb6700 10 client.37379.objecter _send_command 16 2024-02-13T00:45:57.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.022+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f630412e420 con 0x7f630412d930 2024-02-13T00:45:57.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f62faffd700 1 -- 172.21.15.162:0/3909254915 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f630412e420 con 0x7f630412d930 2024-02-13T00:45:57.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f62faffd700 10 client.37379.objecter ms_dispatch 0x7f6304001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:45:57.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f62faffd700 10 client.37379.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:45:57.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f62faffd700 20 client.37379.objecter put_session s=0x7f630412e630 osd=7 4 2024-02-13T00:45:57.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f62faffd700 15 client.37379.objecter _session_command_op_remove 7 16 2024-02-13T00:45:57.029 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:45:57.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 10 librados: watch_flush enter 2024-02-13T00:45:57.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 10 librados: watch_flush exit 2024-02-13T00:45:57.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 10 client.37379.objecter close_session for osd.0 2024-02-13T00:45:57.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6304028590 msgr2=0x7f63040fea60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:57.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6304028590 0x7f63040fea60 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:57.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304107210 osd=0 1 2024-02-13T00:45:57.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 10 client.37379.objecter close_session for osd.1 2024-02-13T00:45:57.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6805/150599780 conn(0x7f6304059570 msgr2=0x7f6304119180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:57.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6805/150599780 conn(0x7f6304059570 0x7f6304119180 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:57.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304059400 osd=1 1 2024-02-13T00:45:57.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 10 client.37379.objecter close_session for osd.2 2024-02-13T00:45:57.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6809/3689719376 conn(0x7f63040277c0 msgr2=0x7f630406c330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:57.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6809/3689719376 conn(0x7f63040277c0 0x7f630406c330 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:57.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304027650 osd=2 1 2024-02-13T00:45:57.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 10 client.37379.objecter close_session for osd.3 2024-02-13T00:45:57.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6813/604971011 conn(0x7f63040ff300 msgr2=0x7f6304121fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:57.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6813/604971011 conn(0x7f63040ff300 0x7f6304121fb0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:57.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f63040ff190 osd=3 1 2024-02-13T00:45:57.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 10 client.37379.objecter close_session for osd.4 2024-02-13T00:45:57.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6800/3344263388 conn(0x7f630406cdd0 msgr2=0x7f6304123620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:57.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6800/3344263388 conn(0x7f630406cdd0 0x7f6304123620 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:57.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630406cbd0 osd=4 1 2024-02-13T00:45:57.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 10 client.37379.objecter close_session for osd.5 2024-02-13T00:45:57.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6804/2212296748 conn(0x7f6304124020 msgr2=0x7f6304124460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:57.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6804/2212296748 conn(0x7f6304124020 0x7f6304124460 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:57.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630a553700 1 -- 172.21.15.162:0/3909254915 reap_dead start 2024-02-13T00:45:57.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630a553700 1 -- 172.21.15.162:0/3909254915 reap_dead start 2024-02-13T00:45:57.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630a553700 1 -- 172.21.15.162:0/3909254915 reap_dead start 2024-02-13T00:45:57.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630406c0a0 osd=5 1 2024-02-13T00:45:57.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 10 client.37379.objecter close_session for osd.6 2024-02-13T00:45:57.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6304129410 msgr2=0x7f630412b860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:57.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6304129410 0x7f630412b860 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:57.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f6304128d90 osd=6 1 2024-02-13T00:45:57.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 10 client.37379.objecter close_session for osd.7 2024-02-13T00:45:57.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6812/1964192772 conn(0x7f630412d930 msgr2=0x7f630412ddb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:57.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.181:6812/1964192772 conn(0x7f630412d930 0x7f630412ddb0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:57.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 20 client.37379.objecter put_session s=0x7f630412e630 osd=7 1 2024-02-13T00:45:57.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 20 client.37379.objecter shutdown clearing up homeless session... 2024-02-13T00:45:57.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 10 client.37379.objecter successfully canceled tick 2024-02-13T00:45:57.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6800/2866534053 conn(0x7f62f0062cc0 msgr2=0x7f62f0065180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:57.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> v2:172.21.15.162:6800/2866534053 conn(0x7f62f0062cc0 0x7f62f0065180 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7f62f40043d0 tx=0x7f62f400b040).stop 2024-02-13T00:45:57.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63040f9b70 msgr2=0x7f630410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:57.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 --2- 172.21.15.162:0/3909254915 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63040f9b70 0x7f630410e610 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 rx=0x7f630000c7c0 tx=0x7f6300007ef0).stop 2024-02-13T00:45:57.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630a553700 1 -- 172.21.15.162:0/3909254915 reap_dead start 2024-02-13T00:45:57.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 shutdown_connections 2024-02-13T00:45:57.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 >> 172.21.15.162:0/3909254915 conn(0x7f63040f38c0 msgr2=0x7f63040f5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:57.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 shutdown_connections 2024-02-13T00:45:57.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 -- 172.21.15.162:0/3909254915 wait complete. 2024-02-13T00:45:57.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.025+0000 7f630bfb6700 1 librados: shutdown 2024-02-13T00:45:57.035 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:45:57.035 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:45:57.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:45:57.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:45:57.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:45:57.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:45:57.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:45:57.036 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:45:57.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:55.067605+0000 osd.1 (osd.1) 165 : cluster [DBG] 61.5 scrub starts 2024-02-13T00:45:57.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:55.071113+0000 osd.1 (osd.1) 166 : cluster [DBG] 61.5 scrub ok 2024-02-13T00:45:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:55.546464+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e396: 8 total, 8 up, 5 in 2024-02-13T00:45:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:55.556713+0000 mon.b (mon.1) 557 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:55.557727+0000 mon.a (mon.0) 2450 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:55.598246+0000 mgr.y (mgr.14544) 347 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:45:57.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.552+0000 7f0e53fff700 1 -- 172.21.15.162:0/3259528496 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v398) v1 ==== 87+0+0 (secure 0 0 0) 0x7f0e60050380 con 0x7f0e6c100650 2024-02-13T00:45:57.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 10 librados: watch_flush enter 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 10 librados: watch_flush exit 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 20 client.27273.objecter shutdown clearing up homeless session... 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 10 client.27273.objecter successfully canceled tick 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 1 -- 172.21.15.162:0/3259528496 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0e54062c80 msgr2=0x7f0e54065140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 1 --2- 172.21.15.162:0/3259528496 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0e54062c80 0x7f0e54065140 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7f0e64005010 tx=0x7f0e64004d10).stop 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 1 -- 172.21.15.162:0/3259528496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6c100650 msgr2=0x7f0e6c113db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 1 --2- 172.21.15.162:0/3259528496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6c100650 0x7f0e6c113db0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f0e6000b530 tx=0x7f0e600090f0).stop 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 1 -- 172.21.15.162:0/3259528496 shutdown_connections 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 1 --2- 172.21.15.162:0/3259528496 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0e54062c80 0x7f0e54065140 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 1 --2- 172.21.15.162:0/3259528496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e6c1092e0 0x7f0e6c10eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 1 --2- 172.21.15.162:0/3259528496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e6c0ffd80 0x7f0e6c113880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 1 --2- 172.21.15.162:0/3259528496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6c100650 0x7f0e6c113db0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:45:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 1 -- 172.21.15.162:0/3259528496 >> 172.21.15.162:0/3259528496 conn(0x7f0e6c063df0 msgr2=0x7f0e6c109ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:45:57.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 1 -- 172.21.15.162:0/3259528496 shutdown_connections 2024-02-13T00:45:57.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.554+0000 7f0e713eb700 1 -- 172.21.15.162:0/3259528496 wait complete. 2024-02-13T00:45:57.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:57.555+0000 7f0e713eb700 1 librados: shutdown 2024-02-13T00:45:57.562 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:45:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:56.414202+0000 mgr.y (mgr.14544) 348 : cluster [DBG] pgmap v567: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:45:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:56.552207+0000 mon.a (mon.0) 2451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:56.552289+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e397: 8 total, 8 up, 5 in 2024-02-13T00:45:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:56.562203+0000 mon.b (mon.1) 558 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:45:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:56.598000+0000 mon.b (mon.1) 559 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:56.598459+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:56.598891+0000 mon.b (mon.1) 560 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-53"}]: dispatch 2024-02-13T00:45:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:56.599208+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-53"}]: dispatch 2024-02-13T00:45:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:56.767671+0000 mon.a (mon.0) 2455 : audit [INF] from='client.? 172.21.15.162:0/3259528496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:45:57.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:56.884080+0000 mon.c (mon.2) 370 : audit [DBG] from='client.? 172.21.15.162:0/3909254915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:57.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:56.941899+0000 mon.c (mon.2) 371 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:45:57.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:56.414202+0000 mgr.y (mgr.14544) 348 : cluster [DBG] pgmap v567: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:45:57.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:56.552207+0000 mon.a (mon.0) 2451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:57.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:56.552289+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e397: 8 total, 8 up, 5 in 2024-02-13T00:45:57.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:56.562203+0000 mon.b (mon.1) 558 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:45:57.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:56.598000+0000 mon.b (mon.1) 559 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:57.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:56.598459+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:57.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:56.598891+0000 mon.b (mon.1) 560 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-53"}]: dispatch 2024-02-13T00:45:57.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:56.599208+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-53"}]: dispatch 2024-02-13T00:45:57.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:56.767671+0000 mon.a (mon.0) 2455 : audit [INF] from='client.? 172.21.15.162:0/3259528496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:45:57.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:56.884080+0000 mon.c (mon.2) 370 : audit [DBG] from='client.? 172.21.15.162:0/3909254915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:57.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:56.941899+0000 mon.c (mon.2) 371 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:45:58.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:56.414202+0000 mgr.y (mgr.14544) 348 : cluster [DBG] pgmap v567: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:45:58.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:56.552207+0000 mon.a (mon.0) 2451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:45:58.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:56.552289+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e397: 8 total, 8 up, 5 in 2024-02-13T00:45:58.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:56.562203+0000 mon.b (mon.1) 558 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:45:58.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:56.598000+0000 mon.b (mon.1) 559 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:58.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:56.598459+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:45:58.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:56.598891+0000 mon.b (mon.1) 560 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-53"}]: dispatch 2024-02-13T00:45:58.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:56.599208+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-53"}]: dispatch 2024-02-13T00:45:58.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:56.767671+0000 mon.a (mon.0) 2455 : audit [INF] from='client.? 172.21.15.162:0/3259528496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:45:58.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:56.884080+0000 mon.c (mon.2) 370 : audit [DBG] from='client.? 172.21.15.162:0/3909254915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:45:58.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:56.941899+0000 mon.c (mon.2) 371 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:45:58.217 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:45:58.571 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:57.123609+0000 osd.6 (osd.6) 99 : cluster [DBG] 61.1b deep-scrub starts 2024-02-13T00:45:58.572 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:57.130504+0000 osd.6 (osd.6) 100 : cluster [DBG] 61.1b deep-scrub ok 2024-02-13T00:45:58.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:57.123609+0000 osd.6 (osd.6) 99 : cluster [DBG] 61.1b deep-scrub starts 2024-02-13T00:45:58.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:57.130504+0000 osd.6 (osd.6) 100 : cluster [DBG] 61.1b deep-scrub ok 2024-02-13T00:45:58.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:57.548122+0000 mon.a (mon.0) 2456 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:45:58.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:57.553672+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? 172.21.15.162:0/3259528496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:45:58.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:57.553716+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e398: 8 total, 8 up, 5 in 2024-02-13T00:45:58.765 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:45:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:58.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:45:58.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:57.548122+0000 mon.a (mon.0) 2456 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:45:58.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:57.553672+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? 172.21.15.162:0/3259528496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:45:58.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:57.553716+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e398: 8 total, 8 up, 5 in 2024-02-13T00:45:58.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:45:58.902 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:59.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:45:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:45:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:45:59.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:57.123609+0000 osd.6 (osd.6) 99 : cluster [DBG] 61.1b deep-scrub starts 2024-02-13T00:45:59.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:57.130504+0000 osd.6 (osd.6) 100 : cluster [DBG] 61.1b deep-scrub ok 2024-02-13T00:45:59.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:57.548122+0000 mon.a (mon.0) 2456 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:45:59.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:57.553672+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? 172.21.15.162:0/3259528496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:45:59.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:57.553716+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e398: 8 total, 8 up, 5 in 2024-02-13T00:45:59.251 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:45:59.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:45:59.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:45:59.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:45:59.394 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:45:59.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:45:59.918 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:58.414700+0000 mgr.y (mgr.14544) 349 : cluster [DBG] pgmap v570: 206 pgs: 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:45:59.918 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:58.570001+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e399: 8 total, 8 up, 5 in 2024-02-13T00:45:59.918 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:58.581499+0000 mon.b (mon.1) 561 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:59.918 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:58.582099+0000 mon.a (mon.0) 2460 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:59.918 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:58.870605+0000 mon.c (mon.2) 372 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:45:59.919 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:58.871341+0000 mon.c (mon.2) 373 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:45:59.919 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:58.875987+0000 mon.a (mon.0) 2461 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:45:59.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:58.414700+0000 mgr.y (mgr.14544) 349 : cluster [DBG] pgmap v570: 206 pgs: 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:45:59.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:58.570001+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e399: 8 total, 8 up, 5 in 2024-02-13T00:45:59.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:58.581499+0000 mon.b (mon.1) 561 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:59.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:58.582099+0000 mon.a (mon.0) 2460 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:45:59.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:58.870605+0000 mon.c (mon.2) 372 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:45:59.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:58.871341+0000 mon.c (mon.2) 373 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:45:59.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:45:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:58.875987+0000 mon.a (mon.0) 2461 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:00.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.896+0000 7f5b8526d700 1 -- 172.21.15.162:0/2957041011 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b80104660 msgr2=0x7f5b80106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:00.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.896+0000 7f5b8526d700 1 --2- 172.21.15.162:0/2957041011 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b80104660 0x7f5b80106a50 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 rx=0x7f5b78009c10 tx=0x7f5b78004d10).stop 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.897+0000 7f5b8526d700 1 -- 172.21.15.162:0/2957041011 shutdown_connections 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.897+0000 7f5b8526d700 1 --2- 172.21.15.162:0/2957041011 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b80104660 0x7f5b80106a50 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.897+0000 7f5b8526d700 1 --2- 172.21.15.162:0/2957041011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b800f6b00 0x7f5b800f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.897+0000 7f5b8526d700 1 --2- 172.21.15.162:0/2957041011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b800f73d0 0x7f5b80104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.897+0000 7f5b8526d700 1 -- 172.21.15.162:0/2957041011 >> 172.21.15.162:0/2957041011 conn(0x7f5b8006eb90 msgr2=0x7f5b800f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.897+0000 7f5b8526d700 1 -- 172.21.15.162:0/2957041011 shutdown_connections 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.897+0000 7f5b8526d700 1 -- 172.21.15.162:0/2957041011 wait complete. 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.897+0000 7f5b8526d700 1 librados: starting msgr at 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.897+0000 7f5b8526d700 1 librados: starting objecter 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.897+0000 7f5b8526d700 1 Processor -- start 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b8526d700 1 -- start start 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b8526d700 1 librados: setting wanted keys 2024-02-13T00:46:00.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b8526d700 1 librados: calling monclient init 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b8526d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b800f6b00 0x7f5b801101c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b8526d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b800f73d0 0x7f5b801106b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b8526d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b80104660 0x7f5b8010a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b8526d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8010a850 con 0x7f5b800f73d0 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b8526d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8010a9c0 con 0x7f5b80104660 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b8526d700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8010ab80 con 0x7f5b800f6b00 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b7effd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b800f6b00 0x7f5b801101c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b7effd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b800f6b00 0x7f5b801101c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:49188/0 (socket says 172.21.15.162:49188) 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b7effd700 1 -- 172.21.15.162:0/3278138012 learned_addr learned my addr 172.21.15.162:0/3278138012 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b7e7fc700 1 --2- 172.21.15.162:0/3278138012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b800f73d0 0x7f5b801106b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b7f7fe700 1 --2- 172.21.15.162:0/3278138012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b80104660 0x7f5b8010a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b7effd700 1 -- 172.21.15.162:0/3278138012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b80104660 msgr2=0x7f5b8010a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b7effd700 1 --2- 172.21.15.162:0/3278138012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b80104660 0x7f5b8010a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:00.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b7effd700 1 -- 172.21.15.162:0/3278138012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b800f73d0 msgr2=0x7f5b801106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b7effd700 1 --2- 172.21.15.162:0/3278138012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b800f73d0 0x7f5b801106b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b7effd700 1 -- 172.21.15.162:0/3278138012 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b74005040 con 0x7f5b800f6b00 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b7f7fe700 1 --2- 172.21.15.162:0/3278138012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b80104660 0x7f5b8010a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.898+0000 7f5b7e7fc700 1 --2- 172.21.15.162:0/3278138012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b800f73d0 0x7f5b801106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.899+0000 7f5b7effd700 1 --2- 172.21.15.162:0/3278138012 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b800f6b00 0x7f5b801101c0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 rx=0x7f5b7000c810 tx=0x7f5b7000cb20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.899+0000 7f5b67fff700 10 client.?.objecter ms_handle_connect 0x7f5b800f6b00 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.899+0000 7f5b67fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.899+0000 7f5b67fff700 1 -- 172.21.15.162:0/3278138012 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b70010510 con 0x7f5b800f6b00 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.899+0000 7f5b8526d700 1 -- 172.21.15.162:0/3278138012 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b780097c0 con 0x7f5b800f6b00 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.899+0000 7f5b67fff700 1 -- 172.21.15.162:0/3278138012 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b70010b40 con 0x7f5b800f6b00 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.899+0000 7f5b67fff700 1 -- 172.21.15.162:0/3278138012 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b70012a90 con 0x7f5b800f6b00 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.899+0000 7f5b8526d700 10 client.37385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.899+0000 7f5b8526d700 1 -- 172.21.15.162:0/3278138012 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8010b1c0 con 0x7f5b800f6b00 2024-02-13T00:46:00.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.899+0000 7f5b8526d700 1 librados: init done 2024-02-13T00:46:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.900+0000 7f5b8526d700 1 -- 172.21.15.162:0/3278138012 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b801093b0 con 0x7f5b800f6b00 2024-02-13T00:46:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.900+0000 7f5b67fff700 1 -- 172.21.15.162:0/3278138012 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f5b70010cb0 con 0x7f5b800f6b00 2024-02-13T00:46:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.901+0000 7f5b67fff700 1 --2- 172.21.15.162:0/3278138012 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5b68062c90 0x7f5b68065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.901+0000 7f5b67fff700 1 -- 172.21.15.162:0/3278138012 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(400..400 src has 52..400) v4 ==== 8038+0+0 (secure 0 0 0) 0x7f5b7007d110 con 0x7f5b800f6b00 2024-02-13T00:46:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.901+0000 7f5b67fff700 10 client.37385.objecter ms_dispatch 0x7f5b80001550 osd_map(400..400 src has 52..400) v4 2024-02-13T00:46:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.901+0000 7f5b67fff700 3 client.37385.objecter handle_osd_map got epochs [400,400] > 0 2024-02-13T00:46:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.901+0000 7f5b67fff700 3 client.37385.objecter handle_osd_map decoding full epoch 400 2024-02-13T00:46:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.901+0000 7f5b7e7fc700 1 --2- 172.21.15.162:0/3278138012 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5b68062c90 0x7f5b68065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.901+0000 7f5b67fff700 20 client.37385.objecter dump_active .. 0 homeless 2024-02-13T00:46:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.901+0000 7f5b67fff700 10 client.37385.objecter ms_handle_connect 0x7f5b68062c90 2024-02-13T00:46:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.901+0000 7f5b7e7fc700 1 --2- 172.21.15.162:0/3278138012 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5b68062c90 0x7f5b68065150 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 rx=0x7f5b8010bb00 tx=0x7f5b7400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:45:59.905+0000 7f5b67fff700 1 -- 172.21.15.162:0/3278138012 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5b70050630 con 0x7f5b800f6b00 2024-02-13T00:46:00.043 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:58.414700+0000 mgr.y (mgr.14544) 349 : cluster [DBG] pgmap v570: 206 pgs: 206 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:46:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:58.570001+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e399: 8 total, 8 up, 5 in 2024-02-13T00:46:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:58.581499+0000 mon.b (mon.1) 561 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:58.582099+0000 mon.a (mon.0) 2460 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:58.870605+0000 mon.c (mon.2) 372 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:46:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:58.871341+0000 mon.c (mon.2) 373 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:46:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:45:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:58.875987+0000 mon.a (mon.0) 2461 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:00.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:00.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:00.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.095+0000 7f5b8526d700 1 -- 172.21.15.162:0/3278138012 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd out", "ids": ["5"]} v 0) v1 -- 0x7f5b8010b840 con 0x7f5b800f6b00 2024-02-13T00:46:00.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:46:00.181 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:00.202 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:46:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:00.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:46:00.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:00.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:46:00.096+0000 7f5f6486f700 -1 mon.a@0(leader).osd e400 definitely_dead 0 2024-02-13T00:46:00.335 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:00.354 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:00.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:00.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.407+0000 7f5b67fff700 1 -- 172.21.15.162:0/3278138012 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["5"]}]=0 marked out osd.5. v401) v1 ==== 87+0+0 (secure 0 0 0) 0x7f5b8010b840 con 0x7f5b800f6b00 2024-02-13T00:46:00.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.411+0000 7f5b8526d700 10 librados: watch_flush enter 2024-02-13T00:46:00.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.411+0000 7f5b8526d700 10 librados: watch_flush exit 2024-02-13T00:46:00.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.411+0000 7f5b8526d700 20 client.37385.objecter shutdown clearing up homeless session... 2024-02-13T00:46:00.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.411+0000 7f5b8526d700 10 client.37385.objecter successfully canceled tick 2024-02-13T00:46:00.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.416+0000 7f5b8526d700 1 -- 172.21.15.162:0/3278138012 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5b68062c90 msgr2=0x7f5b68065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:00.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.416+0000 7f5b8526d700 1 --2- 172.21.15.162:0/3278138012 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5b68062c90 0x7f5b68065150 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 rx=0x7f5b8010bb00 tx=0x7f5b7400b040).stop 2024-02-13T00:46:00.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.416+0000 7f5b8526d700 1 -- 172.21.15.162:0/3278138012 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b800f6b00 msgr2=0x7f5b801101c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:00.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.416+0000 7f5b8526d700 1 --2- 172.21.15.162:0/3278138012 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b800f6b00 0x7f5b801101c0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 rx=0x7f5b7000c810 tx=0x7f5b7000cb20).stop 2024-02-13T00:46:00.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.419+0000 7f5b8526d700 1 -- 172.21.15.162:0/3278138012 shutdown_connections 2024-02-13T00:46:00.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.419+0000 7f5b8526d700 1 --2- 172.21.15.162:0/3278138012 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5b68062c90 0x7f5b68065150 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:00.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.419+0000 7f5b8526d700 1 --2- 172.21.15.162:0/3278138012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b80104660 0x7f5b8010a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:00.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.419+0000 7f5b8526d700 1 --2- 172.21.15.162:0/3278138012 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b800f6b00 0x7f5b801101c0 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:00.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.419+0000 7f5b8526d700 1 --2- 172.21.15.162:0/3278138012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b800f73d0 0x7f5b801106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:00.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.419+0000 7f5b8526d700 1 -- 172.21.15.162:0/3278138012 >> 172.21.15.162:0/3278138012 conn(0x7f5b8006eb90 msgr2=0x7f5b800708a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:00.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.424+0000 7f5b8526d700 1 -- 172.21.15.162:0/3278138012 shutdown_connections 2024-02-13T00:46:00.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.430+0000 7f5b8526d700 1 -- 172.21.15.162:0/3278138012 wait complete. 2024-02-13T00:46:00.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:00.432+0000 7f5b8526d700 1 librados: shutdown 2024-02-13T00:46:00.442 INFO:teuthology.orchestra.run.smithi162.stderr:marked out osd.5. 2024-02-13T00:46:00.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:46:00.486 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:00.505 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:46:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:00.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:46:00.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:46:00.634 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:00.653 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:46:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:00.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:46:00.703 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:00.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:00.781 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:59.571407+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:00.781 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:45:59.571467+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e400: 8 total, 8 up, 5 in 2024-02-13T00:46:00.781 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:59.614046+0000 mon.b (mon.1) 562 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:00.781 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:59.614490+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:00.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:59.614908+0000 mon.b (mon.1) 563 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-55"}]: dispatch 2024-02-13T00:46:00.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:45:59.615285+0000 mon.a (mon.0) 2465 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-55"}]: dispatch 2024-02-13T00:46:00.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:00.097237+0000 mon.c (mon.2) 374 : audit [INF] from='client.? 172.21.15.162:0/3278138012' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-13T00:46:00.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:00.097786+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-13T00:46:00.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:00.097859+0000 mon.a (mon.0) 2467 : cluster [INF] Client client.admin marked osd.5 out, while it was still marked up 2024-02-13T00:46:00.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:00.404359+0000 mon.a (mon.0) 2468 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-02-13T00:46:00.782 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:00.404429+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e401: 8 total, 8 up, 4 in 2024-02-13T00:46:00.783 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:00.802 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:00.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:00.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:59.571407+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:00.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:45:59.571467+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e400: 8 total, 8 up, 5 in 2024-02-13T00:46:00.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:59.614046+0000 mon.b (mon.1) 562 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:00.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:59.614490+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:00.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:59.614908+0000 mon.b (mon.1) 563 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-55"}]: dispatch 2024-02-13T00:46:00.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:45:59.615285+0000 mon.a (mon.0) 2465 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-55"}]: dispatch 2024-02-13T00:46:00.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:00.097237+0000 mon.c (mon.2) 374 : audit [INF] from='client.? 172.21.15.162:0/3278138012' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-13T00:46:00.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:00.097786+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-13T00:46:00.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:00.097859+0000 mon.a (mon.0) 2467 : cluster [INF] Client client.admin marked osd.5 out, while it was still marked up 2024-02-13T00:46:00.806 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:00.404359+0000 mon.a (mon.0) 2468 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-02-13T00:46:00.806 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:00.404429+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e401: 8 total, 8 up, 4 in 2024-02-13T00:46:00.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:59.571407+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:00.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:45:59.571467+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e400: 8 total, 8 up, 5 in 2024-02-13T00:46:00.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:59.614046+0000 mon.b (mon.1) 562 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:00.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:59.614490+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:00.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:59.614908+0000 mon.b (mon.1) 563 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-55"}]: dispatch 2024-02-13T00:46:00.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:45:59.615285+0000 mon.a (mon.0) 2465 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-55"}]: dispatch 2024-02-13T00:46:00.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:00.097237+0000 mon.c (mon.2) 374 : audit [INF] from='client.? 172.21.15.162:0/3278138012' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-13T00:46:00.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:00.097786+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-13T00:46:00.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:00.097859+0000 mon.a (mon.0) 2467 : cluster [INF] Client client.admin marked osd.5 out, while it was still marked up 2024-02-13T00:46:00.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:00.404359+0000 mon.a (mon.0) 2468 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-02-13T00:46:00.807 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:00.404429+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e401: 8 total, 8 up, 4 in 2024-02-13T00:46:00.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:00.934 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:01.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:01.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:01.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:01.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:01.368 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:01.388 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:01.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:01.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:46:01.519 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:01.536 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:46:01.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:01.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:46:01.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:01.666 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:01.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:46:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:01.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:01.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:01.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:00.358261+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:00.358422+0000 osd.0 (osd.0) 145 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:00.415176+0000 mgr.y (mgr.14544) 350 : cluster [DBG] pgmap v574: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-02-13T00:46:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:01.409124+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e402: 8 total, 8 up, 4 in 2024-02-13T00:46:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:01.422620+0000 mon.b (mon.1) 564 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:01.423070+0000 mon.a (mon.0) 2471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:01.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:00.358261+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:01.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:00.358422+0000 osd.0 (osd.0) 145 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:01.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:00.415176+0000 mgr.y (mgr.14544) 350 : cluster [DBG] pgmap v574: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-02-13T00:46:01.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:01.409124+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e402: 8 total, 8 up, 4 in 2024-02-13T00:46:01.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:01.422620+0000 mon.b (mon.1) 564 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:01.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:01.423070+0000 mon.a (mon.0) 2471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:01.863 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:00.358261+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:00.358422+0000 osd.0 (osd.0) 145 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:00.415176+0000 mgr.y (mgr.14544) 350 : cluster [DBG] pgmap v574: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-02-13T00:46:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:01.409124+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e402: 8 total, 8 up, 4 in 2024-02-13T00:46:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:01.422620+0000 mon.b (mon.1) 564 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:01.423070+0000 mon.a (mon.0) 2471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:02.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:02.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:02.187 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:02.214 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:02.338 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:02.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:02.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:02.538 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:02.619 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:02.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:02.724 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:02.751 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:02.754 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:46:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:46:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:46:02.755 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:46:02.755 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:46:03.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:03.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:03.303 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:03.329 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:03.347 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:03.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:03.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:03.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:03.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:02.415532+0000 mgr.y (mgr.14544) 351 : cluster [DBG] pgmap v576: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-13T00:46:03.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:02.420534+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:03.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:02.420609+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e403: 8 total, 8 up, 4 in 2024-02-13T00:46:03.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:02.423048+0000 mon.b (mon.1) 565 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:46:03.476 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:02.605521+0000 mon.b (mon.1) 566 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:03.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:02.606168+0000 mon.a (mon.0) 2474 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:03.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:02.606630+0000 mon.b (mon.1) 567 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-57"}]: dispatch 2024-02-13T00:46:03.477 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:02.607092+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-57"}]: dispatch 2024-02-13T00:46:03.477 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:02.415532+0000 mgr.y (mgr.14544) 351 : cluster [DBG] pgmap v576: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-13T00:46:03.477 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:02.420534+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:03.477 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:02.420609+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e403: 8 total, 8 up, 4 in 2024-02-13T00:46:03.478 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:02.423048+0000 mon.b (mon.1) 565 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:46:03.478 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:02.605521+0000 mon.b (mon.1) 566 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:03.478 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:02.606168+0000 mon.a (mon.0) 2474 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:03.478 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:02.606630+0000 mon.b (mon.1) 567 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-57"}]: dispatch 2024-02-13T00:46:03.478 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:02.607092+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-57"}]: dispatch 2024-02-13T00:46:03.478 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:03.732 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:03.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:03.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:02.415532+0000 mgr.y (mgr.14544) 351 : cluster [DBG] pgmap v576: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-13T00:46:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:02.420534+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:02.420609+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e403: 8 total, 8 up, 4 in 2024-02-13T00:46:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:02.423048+0000 mon.b (mon.1) 565 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:46:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:02.605521+0000 mon.b (mon.1) 566 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:02.606168+0000 mon.a (mon.0) 2474 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:02.606630+0000 mon.b (mon.1) 567 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-57"}]: dispatch 2024-02-13T00:46:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:02.607092+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-57"}]: dispatch 2024-02-13T00:46:03.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:03.870 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:04.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:04.104 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:46:04.129 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:04.148 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:46:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:04.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:46:04.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.223+0000 7f111d0eb700 1 -- 172.21.15.162:0/3606527673 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11180fde50 msgr2=0x7f11180fe230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.223+0000 7f111d0eb700 1 --2- 172.21.15.162:0/3606527673 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11180fde50 0x7f11180fe230 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7f11080047b0 tx=0x7f1108009cb0).stop 2024-02-13T00:46:04.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.223+0000 7f111d0eb700 1 -- 172.21.15.162:0/3606527673 shutdown_connections 2024-02-13T00:46:04.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.223+0000 7f111d0eb700 1 --2- 172.21.15.162:0/3606527673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11180f6880 0x7f1118104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.223+0000 7f111d0eb700 1 --2- 172.21.15.162:0/3606527673 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11180fde50 0x7f11180fe230 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.223+0000 7f111d0eb700 1 --2- 172.21.15.162:0/3606527673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11180fe7b0 0x7f11180f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.223+0000 7f111d0eb700 1 -- 172.21.15.162:0/3606527673 >> 172.21.15.162:0/3606527673 conn(0x7f1118070080 msgr2=0x7f11180f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:04.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.223+0000 7f111d0eb700 1 -- 172.21.15.162:0/3606527673 shutdown_connections 2024-02-13T00:46:04.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.223+0000 7f111d0eb700 1 -- 172.21.15.162:0/3606527673 wait complete. 2024-02-13T00:46:04.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.224+0000 7f111d0eb700 1 librados: starting msgr at 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.224+0000 7f111d0eb700 1 librados: starting objecter 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.224+0000 7f111d0eb700 1 Processor -- start 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111d0eb700 1 -- start start 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111d0eb700 1 librados: setting wanted keys 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111d0eb700 1 librados: calling monclient init 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111d0eb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11180f6880 0x7f11180f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111d0eb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11180fde50 0x7f11180f7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111d0eb700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11180fe7b0 0x7f11180f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111d0eb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1118116860 con 0x7f11180f6880 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111d0eb700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11181169d0 con 0x7f11180fde50 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111d0eb700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1118116b90 con 0x7f11180fe7b0 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111759e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11180fe7b0 0x7f11180f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:04.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111759e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11180fe7b0 0x7f11180f7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:49192/0 (socket says 172.21.15.162:49192) 2024-02-13T00:46:04.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111759e700 1 -- 172.21.15.162:0/682096350 learned_addr learned my addr 172.21.15.162:0/682096350 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:04.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f1116d9d700 1 --2- 172.21.15.162:0/682096350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11180f6880 0x7f11180f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:04.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111759e700 1 -- 172.21.15.162:0/682096350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11180fde50 msgr2=0x7f11180f7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111759e700 1 --2- 172.21.15.162:0/682096350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11180fde50 0x7f11180f7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:46:04.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111759e700 1 -- 172.21.15.162:0/682096350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11180f6880 msgr2=0x7f11180f7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111759e700 1 --2- 172.21.15.162:0/682096350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11180f6880 0x7f11180f7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f111759e700 1 -- 172.21.15.162:0/682096350 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c005040 con 0x7f11180fe7b0 2024-02-13T00:46:04.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.225+0000 7f1116d9d700 1 --2- 172.21.15.162:0/682096350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11180f6880 0x7f11180f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:46:04.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.226+0000 7f111759e700 1 --2- 172.21.15.162:0/682096350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11180fe7b0 0x7f11180f7e50 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 rx=0x7f111000d970 tx=0x7f1110004e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:04.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.226+0000 7f10fffff700 10 client.?.objecter ms_handle_connect 0x7f11180fe7b0 2024-02-13T00:46:04.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.226+0000 7f10fffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:04.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.226+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1110004020 con 0x7f11180fe7b0 2024-02-13T00:46:04.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.226+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f111000f040 con 0x7f11180fe7b0 2024-02-13T00:46:04.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.226+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11100036a0 con 0x7f11180fe7b0 2024-02-13T00:46:04.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.226+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1108005040 con 0x7f11180fe7b0 2024-02-13T00:46:04.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.226+0000 7f111d0eb700 10 client.37391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:04.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.226+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1118108780 con 0x7f11180fe7b0 2024-02-13T00:46:04.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.226+0000 7f111d0eb700 1 librados: init done 2024-02-13T00:46:04.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.227+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f111805fb80 con 0x7f11180fe7b0 2024-02-13T00:46:04.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.228+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1110003ad0 con 0x7f11180fe7b0 2024-02-13T00:46:04.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f10fffff700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1100062c90 0x7f1100065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:04.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(404..404 src has 52..404) v4 ==== 7624+0+0 (secure 0 0 0) 0x7f111007cc80 con 0x7f11180fe7b0 2024-02-13T00:46:04.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 osd_map(404..404 src has 52..404) v4 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f10fffff700 3 client.37391.objecter handle_osd_map got epochs [404,404] > 0 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f10fffff700 3 client.37391.objecter handle_osd_map decoding full epoch 404 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f1116d9d700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1100062c90 0x7f1100065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f10fffff700 20 client.37391.objecter dump_active .. 0 homeless 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v404) v1 ==== 54+0+16 (secure 0 0 0) 0x7f1110081220 con 0x7f11180fe7b0 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111d0eb700 10 client.37391.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 1 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6801/2103743338 conn(0x7f11180ff120 0x7f11180ff500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f11180475e0 osd=0 3 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f11180475e0 osd=0 3 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f11180475e0 osd=0 3 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 1 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f11180475e0 osd=0 3 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 0 1 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f11180475e0 osd=0 4 2024-02-13T00:46:04.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111d0eb700 10 client.37391.objecter _send_command 1 2024-02-13T00:46:04.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1118107410 con 0x7f11180ff120 2024-02-13T00:46:04.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f1116d9d700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1100062c90 0x7f1100065150 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 rx=0x7f1108009da0 tx=0x7f1108007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:04.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f10fffff700 10 client.37391.objecter ms_handle_connect 0x7f1100062c90 2024-02-13T00:46:04.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.229+0000 7f111659c700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6801/2103743338 conn(0x7f11180ff120 0x7f11180ff500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:04.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.230+0000 7f10fffff700 10 client.37391.objecter ms_handle_connect 0x7f11180ff120 2024-02-13T00:46:04.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.230+0000 7f111659c700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6801/2103743338 conn(0x7f11180ff120 0x7f11180ff500 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:04.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.231+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1118107410 con 0x7f11180ff120 2024-02-13T00:46:04.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.231+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:46:04.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.231+0000 7f10fffff700 10 client.37391.objecter _finish_command 1 = system:0 2024-02-13T00:46:04.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.231+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f11180475e0 osd=0 4 2024-02-13T00:46:04.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.231+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 0 1 2024-02-13T00:46:04.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:04.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.258+0000 7f111d0eb700 10 client.37391.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:04.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.258+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 2 2024-02-13T00:46:04.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.258+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f11180475e0 osd=0 3 2024-02-13T00:46:04.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.258+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f11180475e0 osd=0 3 2024-02-13T00:46:04.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.258+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f11180475e0 osd=0 3 2024-02-13T00:46:04.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.258+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 2 2024-02-13T00:46:04.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.258+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f11180475e0 osd=0 3 2024-02-13T00:46:04.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.258+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 0 2 2024-02-13T00:46:04.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.258+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f11180475e0 osd=0 4 2024-02-13T00:46:04.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.258+0000 7f111d0eb700 10 client.37391.objecter _send_command 2 2024-02-13T00:46:04.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.258+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1118060420 con 0x7f11180ff120 2024-02-13T00:46:04.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f1118060420 con 0x7f11180ff120 2024-02-13T00:46:04.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:04.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f10fffff700 10 client.37391.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f11180475e0 osd=0 4 2024-02-13T00:46:04.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 0 2 2024-02-13T00:46:04.264 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:46:04.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f111d0eb700 10 client.37391.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:04.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 3 2024-02-13T00:46:04.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6805/150599780 conn(0x7f1118059990 0x7f1118059d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:04.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118028590 osd=1 3 2024-02-13T00:46:04.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118028590 osd=1 3 2024-02-13T00:46:04.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118028590 osd=1 3 2024-02-13T00:46:04.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 3 2024-02-13T00:46:04.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f1118028590 osd=1 3 2024-02-13T00:46:04.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 1 3 2024-02-13T00:46:04.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118028590 osd=1 4 2024-02-13T00:46:04.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f111d0eb700 10 client.37391.objecter _send_command 3 2024-02-13T00:46:04.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1118028700 con 0x7f1118059990 2024-02-13T00:46:04.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f1116d9d700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6805/150599780 conn(0x7f1118059990 0x7f1118059d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:04.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.261+0000 7f1116d9d700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6805/150599780 conn(0x7f1118059990 0x7f1118059d90 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:04.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.262+0000 7f10fffff700 10 client.37391.objecter ms_handle_connect 0x7f1118059990 2024-02-13T00:46:04.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.262+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1118028700 con 0x7f1118059990 2024-02-13T00:46:04.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.262+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:46:04.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.262+0000 7f10fffff700 10 client.37391.objecter _finish_command 3 = system:0 2024-02-13T00:46:04.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.262+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f1118028590 osd=1 4 2024-02-13T00:46:04.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.262+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 1 3 2024-02-13T00:46:04.283 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:04.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.290+0000 7f111d0eb700 10 client.37391.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:04.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.290+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 4 2024-02-13T00:46:04.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.290+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118028590 osd=1 3 2024-02-13T00:46:04.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.290+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118028590 osd=1 3 2024-02-13T00:46:04.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.290+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118028590 osd=1 3 2024-02-13T00:46:04.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.290+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 4 2024-02-13T00:46:04.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.290+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f1118028590 osd=1 3 2024-02-13T00:46:04.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.290+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 1 4 2024-02-13T00:46:04.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.290+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118028590 osd=1 4 2024-02-13T00:46:04.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.290+0000 7f111d0eb700 10 client.37391.objecter _send_command 4 2024-02-13T00:46:04.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.290+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1118059400 con 0x7f1118059990 2024-02-13T00:46:04.299 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:46:04.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f1118059400 con 0x7f1118059990 2024-02-13T00:46:04.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:04.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f10fffff700 10 client.37391.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f1118028590 osd=1 4 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 1 4 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f111d0eb700 10 client.37391.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 5 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6809/3689719376 conn(0x7f111810bbf0 0x7f111810c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118027be0 osd=2 3 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118027be0 osd=2 3 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118027be0 osd=2 3 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 5 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f1118027be0 osd=2 3 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 2 5 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118027be0 osd=2 4 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f111d0eb700 10 client.37391.objecter _send_command 5 2024-02-13T00:46:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.293+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f111810c500 con 0x7f111810bbf0 2024-02-13T00:46:04.301 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:04.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.296+0000 7f111659c700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6809/3689719376 conn(0x7f111810bbf0 0x7f111810c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:04.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.296+0000 7f111659c700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6809/3689719376 conn(0x7f111810bbf0 0x7f111810c010 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:04.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.296+0000 7f10fffff700 10 client.37391.objecter ms_handle_connect 0x7f111810bbf0 2024-02-13T00:46:04.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.298+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f111810c500 con 0x7f111810bbf0 2024-02-13T00:46:04.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.298+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:46:04.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.298+0000 7f10fffff700 10 client.37391.objecter _finish_command 5 = system:0 2024-02-13T00:46:04.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.298+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f1118027be0 osd=2 4 2024-02-13T00:46:04.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.298+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 2 5 2024-02-13T00:46:04.323 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:04.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.322+0000 7f111d0eb700 10 client.37391.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:04.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.322+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 6 2024-02-13T00:46:04.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.322+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118027be0 osd=2 3 2024-02-13T00:46:04.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.322+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118027be0 osd=2 3 2024-02-13T00:46:04.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.322+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118027be0 osd=2 3 2024-02-13T00:46:04.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.322+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 6 2024-02-13T00:46:04.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.322+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f1118027be0 osd=2 3 2024-02-13T00:46:04.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.322+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 2 6 2024-02-13T00:46:04.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.322+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118027be0 osd=2 4 2024-02-13T00:46:04.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.322+0000 7f111d0eb700 10 client.37391.objecter _send_command 6 2024-02-13T00:46:04.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.322+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1118123640 con 0x7f111810bbf0 2024-02-13T00:46:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:04.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:04.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f1118123640 con 0x7f111810bbf0 2024-02-13T00:46:04.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:04.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f10fffff700 10 client.37391.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f1118027be0 osd=2 4 2024-02-13T00:46:04.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 2 6 2024-02-13T00:46:04.333 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:46:04.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f111d0eb700 10 client.37391.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:04.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 7 2024-02-13T00:46:04.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6813/604971011 conn(0x7f11180277c0 0x7f11181269a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:04.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118027650 osd=3 3 2024-02-13T00:46:04.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118027650 osd=3 3 2024-02-13T00:46:04.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118027650 osd=3 3 2024-02-13T00:46:04.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 7 2024-02-13T00:46:04.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f1118027650 osd=3 3 2024-02-13T00:46:04.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 3 7 2024-02-13T00:46:04.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118027650 osd=3 4 2024-02-13T00:46:04.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f111d0eb700 10 client.37391.objecter _send_command 7 2024-02-13T00:46:04.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.330+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f111810b750 con 0x7f11180277c0 2024-02-13T00:46:04.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.331+0000 7f111759e700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6813/604971011 conn(0x7f11180277c0 0x7f11181269a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:04.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.331+0000 7f10fffff700 10 client.37391.objecter ms_handle_connect 0x7f11180277c0 2024-02-13T00:46:04.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.331+0000 7f111759e700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6813/604971011 conn(0x7f11180277c0 0x7f11181269a0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:04.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.332+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f111810b750 con 0x7f11180277c0 2024-02-13T00:46:04.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.332+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:46:04.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.332+0000 7f10fffff700 10 client.37391.objecter _finish_command 7 = system:0 2024-02-13T00:46:04.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.332+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f1118027650 osd=3 4 2024-02-13T00:46:04.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.332+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 3 7 2024-02-13T00:46:04.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.355+0000 7f111d0eb700 10 client.37391.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:04.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.355+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 8 2024-02-13T00:46:04.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.355+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118027650 osd=3 3 2024-02-13T00:46:04.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.355+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118027650 osd=3 3 2024-02-13T00:46:04.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.355+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118027650 osd=3 3 2024-02-13T00:46:04.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.355+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 8 2024-02-13T00:46:04.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.355+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f1118027650 osd=3 3 2024-02-13T00:46:04.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.355+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 3 8 2024-02-13T00:46:04.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.355+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118027650 osd=3 4 2024-02-13T00:46:04.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.355+0000 7f111d0eb700 10 client.37391.objecter _send_command 8 2024-02-13T00:46:04.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.355+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1118126e90 con 0x7f11180277c0 2024-02-13T00:46:04.360 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:46:04.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f1118126e90 con 0x7f11180277c0 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f10fffff700 10 client.37391.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f1118027650 osd=3 4 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 3 8 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f111d0eb700 10 client.37391.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 9 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6800/3344263388 conn(0x7f1118127510 0x7f1118129900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118126e90 osd=4 3 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118126e90 osd=4 3 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118126e90 osd=4 3 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 9 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f1118126e90 osd=4 3 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 4 9 2024-02-13T00:46:04.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118126e90 osd=4 4 2024-02-13T00:46:04.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f111d0eb700 10 client.37391.objecter _send_command 9 2024-02-13T00:46:04.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.357+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f111812c550 con 0x7f1118127510 2024-02-13T00:46:04.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.358+0000 7f1116d9d700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6800/3344263388 conn(0x7f1118127510 0x7f1118129900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:04.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.358+0000 7f10fffff700 10 client.37391.objecter ms_handle_connect 0x7f1118127510 2024-02-13T00:46:04.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.358+0000 7f1116d9d700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6800/3344263388 conn(0x7f1118127510 0x7f1118129900 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:04.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.359+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f111812c550 con 0x7f1118127510 2024-02-13T00:46:04.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.359+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:46:04.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.359+0000 7f10fffff700 10 client.37391.objecter _finish_command 9 = system:0 2024-02-13T00:46:04.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.359+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f1118126e90 osd=4 4 2024-02-13T00:46:04.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.359+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 4 9 2024-02-13T00:46:04.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.388+0000 7f111d0eb700 10 client.37391.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:04.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.388+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 10 2024-02-13T00:46:04.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.388+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118126e90 osd=4 3 2024-02-13T00:46:04.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.388+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118126e90 osd=4 3 2024-02-13T00:46:04.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.388+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118126e90 osd=4 3 2024-02-13T00:46:04.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.388+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 10 2024-02-13T00:46:04.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.388+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f1118126e90 osd=4 3 2024-02-13T00:46:04.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.388+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 4 10 2024-02-13T00:46:04.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.388+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118126e90 osd=4 4 2024-02-13T00:46:04.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.388+0000 7f111d0eb700 10 client.37391.objecter _send_command 10 2024-02-13T00:46:04.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.388+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1118129df0 con 0x7f1118127510 2024-02-13T00:46:04.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f1118129df0 con 0x7f1118127510 2024-02-13T00:46:04.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:04.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f10fffff700 10 client.37391.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f1118126e90 osd=4 4 2024-02-13T00:46:04.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 4 10 2024-02-13T00:46:04.394 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:46:04.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111d0eb700 10 client.37391.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:04.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 11 2024-02-13T00:46:04.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6804/2212296748 conn(0x7f111812a470 0x7f111812a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:04.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118129df0 osd=5 3 2024-02-13T00:46:04.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118129df0 osd=5 3 2024-02-13T00:46:04.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118129df0 osd=5 3 2024-02-13T00:46:04.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 11 2024-02-13T00:46:04.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f1118129df0 osd=5 3 2024-02-13T00:46:04.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 5 11 2024-02-13T00:46:04.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118129df0 osd=5 4 2024-02-13T00:46:04.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111d0eb700 10 client.37391.objecter _send_command 11 2024-02-13T00:46:04.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f111812c4b0 con 0x7f111812a470 2024-02-13T00:46:04.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.390+0000 7f111659c700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6804/2212296748 conn(0x7f111812a470 0x7f111812a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:04.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.391+0000 7f10fffff700 10 client.37391.objecter ms_handle_connect 0x7f111812a470 2024-02-13T00:46:04.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.391+0000 7f111659c700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6804/2212296748 conn(0x7f111812a470 0x7f111812a8b0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:04.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.391+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f111812c4b0 con 0x7f111812a470 2024-02-13T00:46:04.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.391+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:46:04.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.391+0000 7f10fffff700 10 client.37391.objecter _finish_command 11 = system:0 2024-02-13T00:46:04.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.391+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f1118129df0 osd=5 4 2024-02-13T00:46:04.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.391+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 5 11 2024-02-13T00:46:04.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.406+0000 7f111d0eb700 10 client.37391.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:04.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.406+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 12 2024-02-13T00:46:04.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.406+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118129df0 osd=5 3 2024-02-13T00:46:04.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.406+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118129df0 osd=5 3 2024-02-13T00:46:04.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.406+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f1118129df0 osd=5 3 2024-02-13T00:46:04.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.406+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 12 2024-02-13T00:46:04.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.406+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f1118129df0 osd=5 3 2024-02-13T00:46:04.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.406+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 5 12 2024-02-13T00:46:04.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.406+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118129df0 osd=5 4 2024-02-13T00:46:04.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.406+0000 7f111d0eb700 10 client.37391.objecter _send_command 12 2024-02-13T00:46:04.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.406+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f111812c4b0 con 0x7f111812a470 2024-02-13T00:46:04.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f111812c4b0 con 0x7f111812a470 2024-02-13T00:46:04.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:04.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f10fffff700 10 client.37391.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f1118129df0 osd=5 4 2024-02-13T00:46:04.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 5 12 2024-02-13T00:46:04.414 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:46:04.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111d0eb700 10 client.37391.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:04.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 13 2024-02-13T00:46:04.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6808/1506640341 conn(0x7f111812ecd0 0x7f111812f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:04.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f111812c4b0 osd=6 3 2024-02-13T00:46:04.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f111812c4b0 osd=6 3 2024-02-13T00:46:04.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f111812c4b0 osd=6 3 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 13 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f111812c4b0 osd=6 3 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 6 13 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f111812c4b0 osd=6 4 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111d0eb700 10 client.37391.objecter _send_command 13 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1118132a70 con 0x7f111812ecd0 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111759e700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6808/1506640341 conn(0x7f111812ecd0 0x7f111812f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f10fffff700 10 client.37391.objecter ms_handle_connect 0x7f111812ecd0 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.408+0000 7f111759e700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6808/1506640341 conn(0x7f111812ecd0 0x7f111812f130 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.409+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1118132a70 con 0x7f111812ecd0 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.409+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.409+0000 7f10fffff700 10 client.37391.objecter _finish_command 13 = system:0 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.409+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f111812c4b0 osd=6 4 2024-02-13T00:46:04.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.409+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 6 13 2024-02-13T00:46:04.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.423+0000 7f111d0eb700 10 client.37391.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:04.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.423+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 14 2024-02-13T00:46:04.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.423+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f111812c4b0 osd=6 3 2024-02-13T00:46:04.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.423+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f111812c4b0 osd=6 3 2024-02-13T00:46:04.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.423+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f111812c4b0 osd=6 3 2024-02-13T00:46:04.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.423+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 14 2024-02-13T00:46:04.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.423+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f111812c4b0 osd=6 3 2024-02-13T00:46:04.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.423+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 6 14 2024-02-13T00:46:04.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.423+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f111812c4b0 osd=6 4 2024-02-13T00:46:04.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.423+0000 7f111d0eb700 10 client.37391.objecter _send_command 14 2024-02-13T00:46:04.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.423+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f111812ae60 con 0x7f111812ecd0 2024-02-13T00:46:04.428 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:46:04.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f111812ae60 con 0x7f111812ecd0 2024-02-13T00:46:04.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:04.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f10fffff700 10 client.37391.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f111812c4b0 osd=6 4 2024-02-13T00:46:04.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 6 14 2024-02-13T00:46:04.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f111d0eb700 10 client.37391.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:04.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 15 2024-02-13T00:46:04.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6812/1964192772 conn(0x7f111812b540 0x7f111812b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:04.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f111812ae60 osd=7 3 2024-02-13T00:46:04.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f111812ae60 osd=7 3 2024-02-13T00:46:04.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f111812ae60 osd=7 3 2024-02-13T00:46:04.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 15 2024-02-13T00:46:04.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f111812ae60 osd=7 3 2024-02-13T00:46:04.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 7 15 2024-02-13T00:46:04.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f111812ae60 osd=7 4 2024-02-13T00:46:04.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f111d0eb700 10 client.37391.objecter _send_command 15 2024-02-13T00:46:04.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f111812bf50 con 0x7f111812b540 2024-02-13T00:46:04.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.425+0000 7f1116d9d700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6812/1964192772 conn(0x7f111812b540 0x7f111812b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:04.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.426+0000 7f10fffff700 10 client.37391.objecter ms_handle_connect 0x7f111812b540 2024-02-13T00:46:04.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.426+0000 7f1116d9d700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6812/1964192772 conn(0x7f111812b540 0x7f111812b9a0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:04.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.426+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f111812bf50 con 0x7f111812b540 2024-02-13T00:46:04.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.426+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:46:04.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.426+0000 7f10fffff700 10 client.37391.objecter _finish_command 15 = system:0 2024-02-13T00:46:04.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.426+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f111812ae60 osd=7 4 2024-02-13T00:46:04.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.426+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 7 15 2024-02-13T00:46:04.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:04.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.454+0000 7f111d0eb700 10 client.37391.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:04.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.454+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign -1 16 2024-02-13T00:46:04.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.454+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f111812ae60 osd=7 3 2024-02-13T00:46:04.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.454+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f111812ae60 osd=7 3 2024-02-13T00:46:04.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.454+0000 7f111d0eb700 20 client.37391.objecter _get_session s=0x7f111812ae60 osd=7 3 2024-02-13T00:46:04.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.454+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_remove -1 16 2024-02-13T00:46:04.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.454+0000 7f111d0eb700 20 client.37391.objecter get_session s=0x7f111812ae60 osd=7 3 2024-02-13T00:46:04.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.454+0000 7f111d0eb700 15 client.37391.objecter _session_command_op_assign 7 16 2024-02-13T00:46:04.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.454+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f111812ae60 osd=7 4 2024-02-13T00:46:04.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.454+0000 7f111d0eb700 10 client.37391.objecter _send_command 16 2024-02-13T00:46:04.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.454+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f111812bf50 con 0x7f111812b540 2024-02-13T00:46:04.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.458+0000 7f10fffff700 1 -- 172.21.15.162:0/682096350 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f111812bf50 con 0x7f111812b540 2024-02-13T00:46:04.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.458+0000 7f10fffff700 10 client.37391.objecter ms_dispatch 0x7f1118001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:04.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.458+0000 7f10fffff700 10 client.37391.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.458+0000 7f10fffff700 20 client.37391.objecter put_session s=0x7f111812ae60 osd=7 4 2024-02-13T00:46:04.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.458+0000 7f10fffff700 15 client.37391.objecter _session_command_op_remove 7 16 2024-02-13T00:46:04.465 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:46:04.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.463+0000 7f111d0eb700 10 librados: watch_flush enter 2024-02-13T00:46:04.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 10 librados: watch_flush exit 2024-02-13T00:46:04.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 10 client.37391.objecter close_session for osd.0 2024-02-13T00:46:04.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6801/2103743338 conn(0x7f11180ff120 msgr2=0x7f11180ff500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6801/2103743338 conn(0x7f11180ff120 0x7f11180ff500 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f11180475e0 osd=0 1 2024-02-13T00:46:04.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 10 client.37391.objecter close_session for osd.1 2024-02-13T00:46:04.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6805/150599780 conn(0x7f1118059990 msgr2=0x7f1118059d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6805/150599780 conn(0x7f1118059990 0x7f1118059d90 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118028590 osd=1 1 2024-02-13T00:46:04.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 10 client.37391.objecter close_session for osd.2 2024-02-13T00:46:04.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6809/3689719376 conn(0x7f111810bbf0 msgr2=0x7f111810c010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6809/3689719376 conn(0x7f111810bbf0 0x7f111810c010 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118027be0 osd=2 1 2024-02-13T00:46:04.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 10 client.37391.objecter close_session for osd.3 2024-02-13T00:46:04.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6813/604971011 conn(0x7f11180277c0 msgr2=0x7f11181269a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6813/604971011 conn(0x7f11180277c0 0x7f11181269a0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118027650 osd=3 1 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 10 client.37391.objecter close_session for osd.4 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6800/3344263388 conn(0x7f1118127510 msgr2=0x7f1118129900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6800/3344263388 conn(0x7f1118127510 0x7f1118129900 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118126e90 osd=4 1 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 10 client.37391.objecter close_session for osd.5 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6804/2212296748 conn(0x7f111812a470 msgr2=0x7f111812a8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6804/2212296748 conn(0x7f111812a470 0x7f111812a8b0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f1118129df0 osd=5 1 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 10 client.37391.objecter close_session for osd.6 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6808/1506640341 conn(0x7f111812ecd0 msgr2=0x7f111812f130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6808/1506640341 conn(0x7f111812ecd0 0x7f111812f130 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f111812c4b0 osd=6 1 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 10 client.37391.objecter close_session for osd.7 2024-02-13T00:46:04.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6812/1964192772 conn(0x7f111812b540 msgr2=0x7f111812b9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.181:6812/1964192772 conn(0x7f111812b540 0x7f111812b9a0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 20 client.37391.objecter put_session s=0x7f111812ae60 osd=7 1 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 20 client.37391.objecter shutdown clearing up homeless session... 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 10 client.37391.objecter successfully canceled tick 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111759e700 1 -- 172.21.15.162:0/682096350 reap_dead start 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111759e700 1 -- 172.21.15.162:0/682096350 reap_dead start 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111759e700 1 -- 172.21.15.162:0/682096350 reap_dead start 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111759e700 1 -- 172.21.15.162:0/682096350 reap_dead start 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111759e700 1 -- 172.21.15.162:0/682096350 reap_dead start 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1100062c90 msgr2=0x7f1100065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1100062c90 0x7f1100065150 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 rx=0x7f1108009da0 tx=0x7f1108007f90).stop 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11180fe7b0 msgr2=0x7f11180f7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11180fe7b0 0x7f11180f7e50 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 rx=0x7f111000d970 tx=0x7f1110004e60).stop 2024-02-13T00:46:04.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 shutdown_connections 2024-02-13T00:46:04.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11180fe7b0 0x7f11180f7e50 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:04.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 --2- 172.21.15.162:0/682096350 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1100062c90 0x7f1100065150 secure :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 rx=0x7f1108009da0 tx=0x7f1108007f90).stop 2024-02-13T00:46:04.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.465+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 >> 172.21.15.162:0/682096350 conn(0x7f1118070080 msgr2=0x7f111806d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:04.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.466+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 shutdown_connections 2024-02-13T00:46:04.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.466+0000 7f111d0eb700 1 -- 172.21.15.162:0/682096350 wait complete. 2024-02-13T00:46:04.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:04.466+0000 7f111d0eb700 1 librados: shutdown 2024-02-13T00:46:04.474 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:04.480 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.481 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.481 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.481 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.481 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.481 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.481 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.481 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:46:04.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:03.430131+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e404: 8 total, 8 up, 4 in 2024-02-13T00:46:04.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:04.229707+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? 172.21.15.162:0/682096350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:04.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:04.337631+0000 mon.a (mon.0) 2477 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:04.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:03.430131+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e404: 8 total, 8 up, 4 in 2024-02-13T00:46:04.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:04.229707+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? 172.21.15.162:0/682096350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:04.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:04.337631+0000 mon.a (mon.0) 2477 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:04.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:03.430131+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e404: 8 total, 8 up, 4 in 2024-02-13T00:46:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:04.229707+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? 172.21.15.162:0/682096350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:04.337631+0000 mon.a (mon.0) 2477 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:04.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-13T00:46:05.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:04.415878+0000 mgr.y (mgr.14544) 352 : cluster [DBG] pgmap v579: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:46:05.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:04.438631+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e405: 8 total, 8 up, 4 in 2024-02-13T00:46:05.781 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:04.441086+0000 mon.b (mon.1) 568 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:05.781 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:04.442777+0000 mon.a (mon.0) 2479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:05.781 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:04.415878+0000 mgr.y (mgr.14544) 352 : cluster [DBG] pgmap v579: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:46:05.781 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:04.438631+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e405: 8 total, 8 up, 4 in 2024-02-13T00:46:05.781 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:04.441086+0000 mon.b (mon.1) 568 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:05.781 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:04.442777+0000 mon.a (mon.0) 2479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:05.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:46:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:46:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:04.415878+0000 mgr.y (mgr.14544) 352 : cluster [DBG] pgmap v579: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:46:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:04.438631+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e405: 8 total, 8 up, 4 in 2024-02-13T00:46:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:04.441086+0000 mon.b (mon.1) 568 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:04.442777+0000 mon.a (mon.0) 2479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:06.238 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6] out_osds: [3, 2, 7, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:46:06.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-13T00:46:06.246 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:06.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:06.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:06.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:06.377 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:06.395 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:06.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:06.450 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:46:06] "GET /metrics HTTP/1.1" 200 185129 "" "Prometheus/2.33.4" 2024-02-13T00:46:06.451 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:05.445813+0000 mon.a (mon.0) 2480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:06.451 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:05.445938+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e406: 8 total, 8 up, 4 in 2024-02-13T00:46:06.451 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:05.447848+0000 mon.b (mon.1) 569 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:46:06.452 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:05.452796+0000 mon.b (mon.1) 570 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:06.452 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:05.456015+0000 mon.a (mon.0) 2482 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:06.452 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:05.599289+0000 mgr.y (mgr.14544) 353 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:06.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:46:06.526 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:06.549 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:46:06.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:06.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:46:06.652 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:06.680 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:06.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:06.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:05.445813+0000 mon.a (mon.0) 2480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:06.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:05.445938+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e406: 8 total, 8 up, 4 in 2024-02-13T00:46:06.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:05.447848+0000 mon.b (mon.1) 569 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:46:06.718 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:05.452796+0000 mon.b (mon.1) 570 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:06.718 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:05.456015+0000 mon.a (mon.0) 2482 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:06.718 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:05.599289+0000 mgr.y (mgr.14544) 353 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:06.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:06.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:05.445813+0000 mon.a (mon.0) 2480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:06.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:05.445938+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e406: 8 total, 8 up, 4 in 2024-02-13T00:46:06.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:05.447848+0000 mon.b (mon.1) 569 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:46:06.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:05.452796+0000 mon.b (mon.1) 570 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:06.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:05.456015+0000 mon.a (mon.0) 2482 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:06.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:05.599289+0000 mgr.y (mgr.14544) 353 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:06.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:06.846 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:06.865 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:06.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:06.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:06.995 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:07.013 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:07.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:07.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:07.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:07.144 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:07.217 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:07.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:07.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:07.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.341+0000 7f34e004b700 1 -- 172.21.15.162:0/1477247559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34d80fdc50 msgr2=0x7f34d80fe030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:07.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.341+0000 7f34e004b700 1 --2- 172.21.15.162:0/1477247559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34d80fdc50 0x7f34d80fe030 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7f34d40047b0 tx=0x7f34d4009cb0).stop 2024-02-13T00:46:07.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.341+0000 7f34e004b700 1 -- 172.21.15.162:0/1477247559 shutdown_connections 2024-02-13T00:46:07.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.341+0000 7f34e004b700 1 --2- 172.21.15.162:0/1477247559 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34d80f6840 0x7f34d8104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:07.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.341+0000 7f34e004b700 1 --2- 172.21.15.162:0/1477247559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34d80fdc50 0x7f34d80fe030 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.341+0000 7f34e004b700 1 --2- 172.21.15.162:0/1477247559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d80fe5b0 0x7f34d80f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.341+0000 7f34e004b700 1 -- 172.21.15.162:0/1477247559 >> 172.21.15.162:0/1477247559 conn(0x7f34d80700d0 msgr2=0x7f34d80f4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.341+0000 7f34e004b700 1 -- 172.21.15.162:0/1477247559 shutdown_connections 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.341+0000 7f34e004b700 1 -- 172.21.15.162:0/1477247559 wait complete. 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.341+0000 7f34e004b700 1 librados: starting msgr at 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.341+0000 7f34e004b700 1 librados: starting objecter 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34e004b700 1 Processor -- start 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34e004b700 1 -- start start 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34e004b700 1 librados: setting wanted keys 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34e004b700 1 librados: calling monclient init 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34e004b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d80f6840 0x7f34d810c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34e004b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34d80fdc50 0x7f34d810c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34e004b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34d80fe5b0 0x7f34d8106210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:07.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34e004b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8106700 con 0x7f34d80fe5b0 2024-02-13T00:46:07.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34e004b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8106870 con 0x7f34d80f6840 2024-02-13T00:46:07.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34e004b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8106a30 con 0x7f34d80fdc50 2024-02-13T00:46:07.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34dd5e6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34d80fdc50 0x7f34d810c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:07.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34dd5e6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34d80fdc50 0x7f34d810c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:32926/0 (socket says 172.21.15.162:32926) 2024-02-13T00:46:07.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34dd5e6700 1 -- 172.21.15.162:0/480967909 learned_addr learned my addr 172.21.15.162:0/480967909 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:07.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34ddde7700 1 --2- 172.21.15.162:0/480967909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d80f6840 0x7f34d810c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:07.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34de5e8700 1 --2- 172.21.15.162:0/480967909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34d80fe5b0 0x7f34d8106210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:07.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34dd5e6700 1 -- 172.21.15.162:0/480967909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d80f6840 msgr2=0x7f34d810c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34dd5e6700 1 --2- 172.21.15.162:0/480967909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d80f6840 0x7f34d810c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34dd5e6700 1 -- 172.21.15.162:0/480967909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34d80fe5b0 msgr2=0x7f34d8106210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34dd5e6700 1 --2- 172.21.15.162:0/480967909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34d80fe5b0 0x7f34d8106210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.342+0000 7f34dd5e6700 1 -- 172.21.15.162:0/480967909 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d00097c0 con 0x7f34d80fdc50 2024-02-13T00:46:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.343+0000 7f34dd5e6700 1 --2- 172.21.15.162:0/480967909 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34d80fdc50 0x7f34d810c560 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 rx=0x7f34c800ef00 tx=0x7f34c8011040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.343+0000 7f34ceffd700 10 client.?.objecter ms_handle_connect 0x7f34d80fdc50 2024-02-13T00:46:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.343+0000 7f34ceffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.343+0000 7f34ceffd700 1 -- 172.21.15.162:0/480967909 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8014040 con 0x7f34d80fdc50 2024-02-13T00:46:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.343+0000 7f34ceffd700 1 -- 172.21.15.162:0/480967909 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34c8004660 con 0x7f34d80fdc50 2024-02-13T00:46:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.343+0000 7f34e004b700 1 -- 172.21.15.162:0/480967909 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d4005040 con 0x7f34d80fdc50 2024-02-13T00:46:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.343+0000 7f34ceffd700 1 -- 172.21.15.162:0/480967909 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8009360 con 0x7f34d80fdc50 2024-02-13T00:46:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.343+0000 7f34e004b700 10 client.37397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.343+0000 7f34e004b700 1 -- 172.21.15.162:0/480967909 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d8107010 con 0x7f34d80fdc50 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.343+0000 7f34e004b700 1 librados: init done 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.344+0000 7f34e004b700 1 -- 172.21.15.162:0/480967909 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d805f960 con 0x7f34d80fdc50 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.344+0000 7f34ceffd700 1 -- 172.21.15.162:0/480967909 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f34c8012890 con 0x7f34d80fdc50 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.344+0000 7f34ceffd700 1 --2- 172.21.15.162:0/480967909 >> v2:172.21.15.162:6800/2866534053 conn(0x7f34c4062c90 0x7f34c4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.345+0000 7f34ddde7700 1 --2- 172.21.15.162:0/480967909 >> v2:172.21.15.162:6800/2866534053 conn(0x7f34c4062c90 0x7f34c4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.345+0000 7f34ceffd700 1 -- 172.21.15.162:0/480967909 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(407..407 src has 52..407) v4 ==== 8025+0+0 (secure 0 0 0) 0x7f34c8081120 con 0x7f34d80fdc50 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.345+0000 7f34ceffd700 10 client.37397.objecter ms_dispatch 0x7f34d8001550 osd_map(407..407 src has 52..407) v4 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.345+0000 7f34ceffd700 3 client.37397.objecter handle_osd_map got epochs [407,407] > 0 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.345+0000 7f34ceffd700 3 client.37397.objecter handle_osd_map decoding full epoch 407 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.345+0000 7f34ceffd700 20 client.37397.objecter dump_active .. 0 homeless 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.345+0000 7f34ceffd700 10 client.37397.objecter ms_handle_connect 0x7f34c4062c90 2024-02-13T00:46:07.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.345+0000 7f34ddde7700 1 --2- 172.21.15.162:0/480967909 >> v2:172.21.15.162:6800/2866534053 conn(0x7f34c4062c90 0x7f34c4065150 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 rx=0x7f34d4005010 tx=0x7f34d4009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:07.351 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:07.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.349+0000 7f34ceffd700 1 -- 172.21.15.162:0/480967909 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f34c8054650 con 0x7f34d80fdc50 2024-02-13T00:46:07.370 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:07.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:07.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:07.399 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:07.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:07.527 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:07.564 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:07.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:07.580 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:07.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:07.612+0000 7f34e004b700 1 -- 172.21.15.162:0/480967909 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f34d80f9020 con 0x7f34d80fdc50 2024-02-13T00:46:07.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:07.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:05.482757+0000 osd.4 (osd.4) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:07.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:05.483025+0000 osd.4 (osd.4) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:07.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:06.416503+0000 mgr.y (mgr.14544) 354 : cluster [DBG] pgmap v582: 238 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering, 3 active+recovery_wait, 22 active+recovery_wait+degraded, 32 creating+peering, 176 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 511 B/s rd, 767 B/s wr, 1 op/s; 315/1140 objects degraded (27.632%); 13/1140 objects misplaced (1.140%); 977 KiB/s, 12 objects/s recovering 2024-02-13T00:46:07.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:06.441018+0000 mon.a (mon.0) 2483 : cluster [WRN] Health check failed: Degraded data redundancy: 315/1140 objects degraded (27.632%), 24 pgs degraded (PG_DEGRADED) 2024-02-13T00:46:07.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:06.450630+0000 mon.a (mon.0) 2484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:46:07.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:06.450703+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e407: 8 total, 8 up, 4 in 2024-02-13T00:46:07.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:06.521442+0000 mon.b (mon.1) 571 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:07.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:06.521972+0000 mon.a (mon.0) 2486 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:07.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:06.522460+0000 mon.b (mon.1) 572 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-59"}]: dispatch 2024-02-13T00:46:07.700 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:06.522842+0000 mon.a (mon.0) 2487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-59"}]: dispatch 2024-02-13T00:46:07.701 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:07.706 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:05.482757+0000 osd.4 (osd.4) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:07.706 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:05.483025+0000 osd.4 (osd.4) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:07.706 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:06.416503+0000 mgr.y (mgr.14544) 354 : cluster [DBG] pgmap v582: 238 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering, 3 active+recovery_wait, 22 active+recovery_wait+degraded, 32 creating+peering, 176 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 511 B/s rd, 767 B/s wr, 1 op/s; 315/1140 objects degraded (27.632%); 13/1140 objects misplaced (1.140%); 977 KiB/s, 12 objects/s recovering 2024-02-13T00:46:07.706 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:06.441018+0000 mon.a (mon.0) 2483 : cluster [WRN] Health check failed: Degraded data redundancy: 315/1140 objects degraded (27.632%), 24 pgs degraded (PG_DEGRADED) 2024-02-13T00:46:07.706 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:06.450630+0000 mon.a (mon.0) 2484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:46:07.706 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:06.450703+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e407: 8 total, 8 up, 4 in 2024-02-13T00:46:07.706 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:06.521442+0000 mon.b (mon.1) 571 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:07.706 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:06.521972+0000 mon.a (mon.0) 2486 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:07.707 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:06.522460+0000 mon.b (mon.1) 572 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-59"}]: dispatch 2024-02-13T00:46:07.707 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:06.522842+0000 mon.a (mon.0) 2487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-59"}]: dispatch 2024-02-13T00:46:07.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:05.482757+0000 osd.4 (osd.4) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:07.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:05.483025+0000 osd.4 (osd.4) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:07.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:06.416503+0000 mgr.y (mgr.14544) 354 : cluster [DBG] pgmap v582: 238 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering, 3 active+recovery_wait, 22 active+recovery_wait+degraded, 32 creating+peering, 176 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 511 B/s rd, 767 B/s wr, 1 op/s; 315/1140 objects degraded (27.632%); 13/1140 objects misplaced (1.140%); 977 KiB/s, 12 objects/s recovering 2024-02-13T00:46:07.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:06.441018+0000 mon.a (mon.0) 2483 : cluster [WRN] Health check failed: Degraded data redundancy: 315/1140 objects degraded (27.632%), 24 pgs degraded (PG_DEGRADED) 2024-02-13T00:46:07.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:06.450630+0000 mon.a (mon.0) 2484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:46:07.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:06.450703+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e407: 8 total, 8 up, 4 in 2024-02-13T00:46:07.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:06.521442+0000 mon.b (mon.1) 571 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:07.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:06.521972+0000 mon.a (mon.0) 2486 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:07.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:06.522460+0000 mon.b (mon.1) 572 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-59"}]: dispatch 2024-02-13T00:46:07.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:06.522842+0000 mon.a (mon.0) 2487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-59"}]: dispatch 2024-02-13T00:46:07.719 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:07.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:07.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:07.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:46:07.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:07.847 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:07.865 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:07.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:07.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:07.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:08.000 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:08.026 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:08.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:08.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:08.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:08.156 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:08.360 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:08.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:08.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:08.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.464+0000 7f34ceffd700 1 -- 172.21.15.162:0/480967909 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v409) v1 ==== 97+0+0 (secure 0 0 0) 0x7f34c8054240 con 0x7f34d80fdc50 2024-02-13T00:46:08.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.467+0000 7f34e004b700 10 librados: watch_flush enter 2024-02-13T00:46:08.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.469+0000 7f34e004b700 10 librados: watch_flush exit 2024-02-13T00:46:08.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.469+0000 7f34e004b700 20 client.37397.objecter shutdown clearing up homeless session... 2024-02-13T00:46:08.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.469+0000 7f34e004b700 10 client.37397.objecter successfully canceled tick 2024-02-13T00:46:08.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.469+0000 7f34e004b700 1 -- 172.21.15.162:0/480967909 >> v2:172.21.15.162:6800/2866534053 conn(0x7f34c4062c90 msgr2=0x7f34c4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:08.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.469+0000 7f34e004b700 1 --2- 172.21.15.162:0/480967909 >> v2:172.21.15.162:6800/2866534053 conn(0x7f34c4062c90 0x7f34c4065150 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 rx=0x7f34d4005010 tx=0x7f34d4009cb0).stop 2024-02-13T00:46:08.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.470+0000 7f34e004b700 1 -- 172.21.15.162:0/480967909 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34d80fdc50 msgr2=0x7f34d810c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:08.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.470+0000 7f34e004b700 1 --2- 172.21.15.162:0/480967909 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34d80fdc50 0x7f34d810c560 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 rx=0x7f34c800ef00 tx=0x7f34c8011040).stop 2024-02-13T00:46:08.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.478+0000 7f34e004b700 1 -- 172.21.15.162:0/480967909 shutdown_connections 2024-02-13T00:46:08.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.478+0000 7f34e004b700 1 --2- 172.21.15.162:0/480967909 >> v2:172.21.15.162:6800/2866534053 conn(0x7f34c4062c90 0x7f34c4065150 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:08.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.478+0000 7f34e004b700 1 --2- 172.21.15.162:0/480967909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34d80fe5b0 0x7f34d8106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:08.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.478+0000 7f34e004b700 1 --2- 172.21.15.162:0/480967909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d80f6840 0x7f34d810c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:08.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.478+0000 7f34e004b700 1 --2- 172.21.15.162:0/480967909 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34d80fdc50 0x7f34d810c560 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:08.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.478+0000 7f34e004b700 1 -- 172.21.15.162:0/480967909 >> 172.21.15.162:0/480967909 conn(0x7f34d80700d0 msgr2=0x7f34d806d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:08.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.484+0000 7f34e004b700 1 -- 172.21.15.162:0/480967909 shutdown_connections 2024-02-13T00:46:08.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.484+0000 7f34e004b700 1 -- 172.21.15.162:0/480967909 wait complete. 2024-02-13T00:46:08.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:08.486+0000 7f34e004b700 1 librados: shutdown 2024-02-13T00:46:08.490 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:46:08.496 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:08.520 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:08.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:08.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:08.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:08.653 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:08.673 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:08.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:08.777 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:08.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:07.457263+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e408: 8 total, 8 up, 4 in 2024-02-13T00:46:08.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:07.613251+0000 mon.c (mon.2) 376 : audit [INF] from='client.? 172.21.15.162:0/480967909' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:08.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:07.730955+0000 mon.a (mon.0) 2489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:08.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:46:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:46:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:46:08.803 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:08.822 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:08.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:08.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:07.457263+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e408: 8 total, 8 up, 4 in 2024-02-13T00:46:08.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:07.613251+0000 mon.c (mon.2) 376 : audit [INF] from='client.? 172.21.15.162:0/480967909' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:08.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:07.730955+0000 mon.a (mon.0) 2489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:08.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:07.457263+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e408: 8 total, 8 up, 4 in 2024-02-13T00:46:08.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:07.613251+0000 mon.c (mon.2) 376 : audit [INF] from='client.? 172.21.15.162:0/480967909' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:08.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:07.730955+0000 mon.a (mon.0) 2489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:08.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:08.953 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:08.416957+0000 mgr.y (mgr.14544) 355 : cluster [DBG] pgmap v585: 206 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering, 3 active+recovery_wait, 22 active+recovery_wait+degraded, 176 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 511 B/s rd, 767 B/s wr, 1 op/s; 315/1140 objects degraded (27.632%); 13/1140 objects misplaced (1.140%); 977 KiB/s, 12 objects/s recovering 2024-02-13T00:46:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:08.455548+0000 mon.a (mon.0) 2490 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:08.463586+0000 mon.a (mon.0) 2491 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:46:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:08.463635+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e409: 8 total, 8 up, 4 in 2024-02-13T00:46:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:08.465088+0000 mon.b (mon.1) 573 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:08.465724+0000 mon.a (mon.0) 2493 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:09.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:08.416957+0000 mgr.y (mgr.14544) 355 : cluster [DBG] pgmap v585: 206 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering, 3 active+recovery_wait, 22 active+recovery_wait+degraded, 176 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 511 B/s rd, 767 B/s wr, 1 op/s; 315/1140 objects degraded (27.632%); 13/1140 objects misplaced (1.140%); 977 KiB/s, 12 objects/s recovering 2024-02-13T00:46:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:08.455548+0000 mon.a (mon.0) 2490 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:08.463586+0000 mon.a (mon.0) 2491 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:46:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:08.463635+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e409: 8 total, 8 up, 4 in 2024-02-13T00:46:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:08.465088+0000 mon.b (mon.1) 573 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:08.465724+0000 mon.a (mon.0) 2493 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:08.416957+0000 mgr.y (mgr.14544) 355 : cluster [DBG] pgmap v585: 206 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering, 3 active+recovery_wait, 22 active+recovery_wait+degraded, 176 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 511 B/s rd, 767 B/s wr, 1 op/s; 315/1140 objects degraded (27.632%); 13/1140 objects misplaced (1.140%); 977 KiB/s, 12 objects/s recovering 2024-02-13T00:46:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:08.455548+0000 mon.a (mon.0) 2490 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:08.463586+0000 mon.a (mon.0) 2491 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:46:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:08.463635+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e409: 8 total, 8 up, 4 in 2024-02-13T00:46:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:08.465088+0000 mon.b (mon.1) 573 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:09.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:08.465724+0000 mon.a (mon.0) 2493 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:10.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.225+0000 7f42002cb700 1 -- 172.21.15.162:0/3567446570 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41f80fcb30 msgr2=0x7f41f80fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:10.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.225+0000 7f42002cb700 1 --2- 172.21.15.162:0/3567446570 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41f80fcb30 0x7f41f80fcf10 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7f41f4004870 tx=0x7f41f4009cb0).stop 2024-02-13T00:46:10.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.225+0000 7f42002cb700 1 -- 172.21.15.162:0/3567446570 shutdown_connections 2024-02-13T00:46:10.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.225+0000 7f42002cb700 1 --2- 172.21.15.162:0/3567446570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41f80f73a0 0x7f41f81039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:10.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.225+0000 7f42002cb700 1 --2- 172.21.15.162:0/3567446570 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41f80fcb30 0x7f41f80fcf10 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:10.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.225+0000 7f42002cb700 1 --2- 172.21.15.162:0/3567446570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41f80fd490 0x7f41f80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:46:10.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.225+0000 7f42002cb700 1 -- 172.21.15.162:0/3567446570 >> 172.21.15.162:0/3567446570 conn(0x7f41f806eb90 msgr2=0x7f41f80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:10.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.226+0000 7f42002cb700 1 -- 172.21.15.162:0/3567446570 shutdown_connections 2024-02-13T00:46:10.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.226+0000 7f42002cb700 1 -- 172.21.15.162:0/3567446570 wait complete. 2024-02-13T00:46:10.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.226+0000 7f42002cb700 1 librados: starting msgr at 2024-02-13T00:46:10.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.226+0000 7f42002cb700 1 librados: starting objecter 2024-02-13T00:46:10.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.226+0000 7f42002cb700 1 Processor -- start 2024-02-13T00:46:10.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f42002cb700 1 -- start start 2024-02-13T00:46:10.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f42002cb700 1 librados: setting wanted keys 2024-02-13T00:46:10.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f42002cb700 1 librados: calling monclient init 2024-02-13T00:46:10.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f42002cb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41f80f73a0 0x7f41f8104190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:10.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f42002cb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41f80fcb30 0x7f41f8104680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:10.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f42002cb700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41f8104b70 0x7f41f81167e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:10.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f42002cb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f8116cd0 con 0x7f41f80fcb30 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f42002cb700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f8116e10 con 0x7f41f80f73a0 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f42002cb700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f8116f80 con 0x7f41f8104b70 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f41fd866700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41f80fcb30 0x7f41f8104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f41fd866700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41f80fcb30 0x7f41f8104680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35890/0 (socket says 172.21.15.162:35890) 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f41fe067700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41f80f73a0 0x7f41f8104190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f41fd866700 1 -- 172.21.15.162:0/1345951736 learned_addr learned my addr 172.21.15.162:0/1345951736 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f41fe868700 1 --2- 172.21.15.162:0/1345951736 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41f8104b70 0x7f41f81167e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f41fd866700 1 -- 172.21.15.162:0/1345951736 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41f8104b70 msgr2=0x7f41f81167e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f41fd866700 1 --2- 172.21.15.162:0/1345951736 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41f8104b70 0x7f41f81167e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f41fd866700 1 -- 172.21.15.162:0/1345951736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41f80f73a0 msgr2=0x7f41f8104190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f41fd866700 1 --2- 172.21.15.162:0/1345951736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41f80f73a0 0x7f41f8104190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.227+0000 7f41fd866700 1 -- 172.21.15.162:0/1345951736 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f00097c0 con 0x7f41f80fcb30 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.228+0000 7f41fe067700 1 --2- 172.21.15.162:0/1345951736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41f80f73a0 0x7f41f8104190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:10.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.228+0000 7f41fd866700 1 --2- 172.21.15.162:0/1345951736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41f80fcb30 0x7f41f8104680 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 rx=0x7f41e800b3f0 tx=0x7f41e8006f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:10.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.228+0000 7f41ef779700 10 client.?.objecter ms_handle_connect 0x7f41f80fcb30 2024-02-13T00:46:10.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.228+0000 7f41ef779700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:10.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.228+0000 7f41ef779700 1 -- 172.21.15.162:0/1345951736 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8012410 con 0x7f41f80fcb30 2024-02-13T00:46:10.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.228+0000 7f42002cb700 1 -- 172.21.15.162:0/1345951736 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f4005040 con 0x7f41f80fcb30 2024-02-13T00:46:10.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.228+0000 7f41ef779700 1 -- 172.21.15.162:0/1345951736 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41e8012a40 con 0x7f41f80fcb30 2024-02-13T00:46:10.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.228+0000 7f41ef779700 1 -- 172.21.15.162:0/1345951736 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8011a80 con 0x7f41f80fcb30 2024-02-13T00:46:10.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.228+0000 7f42002cb700 10 client.27303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:10.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.228+0000 7f42002cb700 1 -- 172.21.15.162:0/1345951736 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f8117640 con 0x7f41f80fcb30 2024-02-13T00:46:10.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.228+0000 7f42002cb700 1 librados: init done 2024-02-13T00:46:10.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.229+0000 7f42002cb700 1 -- 172.21.15.162:0/1345951736 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f81071a0 con 0x7f41f80fcb30 2024-02-13T00:46:10.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.230+0000 7f41ef779700 1 -- 172.21.15.162:0/1345951736 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f41e8012570 con 0x7f41f80fcb30 2024-02-13T00:46:10.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.230+0000 7f41ef779700 1 --2- 172.21.15.162:0/1345951736 >> v2:172.21.15.162:6800/2866534053 conn(0x7f41e4062c90 0x7f41e4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:10.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.230+0000 7f41ef779700 1 -- 172.21.15.162:0/1345951736 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(410..410 src has 52..410) v4 ==== 8025+0+0 (secure 0 0 0) 0x7f41e807d1e0 con 0x7f41f80fcb30 2024-02-13T00:46:10.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.230+0000 7f41ef779700 10 client.27303.objecter ms_dispatch 0x7f41f8001550 osd_map(410..410 src has 52..410) v4 2024-02-13T00:46:10.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.230+0000 7f41ef779700 3 client.27303.objecter handle_osd_map got epochs [410,410] > 0 2024-02-13T00:46:10.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.230+0000 7f41ef779700 3 client.27303.objecter handle_osd_map decoding full epoch 410 2024-02-13T00:46:10.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.230+0000 7f41fe067700 1 --2- 172.21.15.162:0/1345951736 >> v2:172.21.15.162:6800/2866534053 conn(0x7f41e4062c90 0x7f41e4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:10.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.231+0000 7f41ef779700 20 client.27303.objecter dump_active .. 0 homeless 2024-02-13T00:46:10.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.231+0000 7f41ef779700 10 client.27303.objecter ms_handle_connect 0x7f41e4062c90 2024-02-13T00:46:10.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.231+0000 7f41fe067700 1 --2- 172.21.15.162:0/1345951736 >> v2:172.21.15.162:6800/2866534053 conn(0x7f41e4062c90 0x7f41e4065150 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 rx=0x7f41f40043d0 tx=0x7f41f400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:10.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.238+0000 7f41ef779700 1 -- 172.21.15.162:0/1345951736 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f41e80507b0 con 0x7f41f80fcb30 2024-02-13T00:46:10.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:10.441+0000 7f42002cb700 1 -- 172.21.15.162:0/1345951736 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f41f8105670 con 0x7f41f80fcb30 2024-02-13T00:46:10.608 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:10.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:10.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:10.739 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:10.756 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:10.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:09.470418+0000 mon.a (mon.0) 2494 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:09.470553+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e410: 8 total, 8 up, 4 in 2024-02-13T00:46:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:09.481872+0000 mon.b (mon.1) 574 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:46:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:09.504238+0000 mon.b (mon.1) 575 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:09.504767+0000 mon.a (mon.0) 2496 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:09.637505+0000 osd.1 (osd.1) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:09.637650+0000 osd.1 (osd.1) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:10.412834+0000 mon.a (mon.0) 2497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:46:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:10.412913+0000 mon.a (mon.0) 2498 : cluster [DBG] osdmap e411: 8 total, 8 up, 4 in 2024-02-13T00:46:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:10.443120+0000 mon.a (mon.0) 2499 : audit [INF] from='client.? 172.21.15.162:0/1345951736' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:46:10.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:09.470418+0000 mon.a (mon.0) 2494 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:09.470553+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e410: 8 total, 8 up, 4 in 2024-02-13T00:46:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:09.481872+0000 mon.b (mon.1) 574 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:46:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:09.504238+0000 mon.b (mon.1) 575 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:09.504767+0000 mon.a (mon.0) 2496 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:09.637505+0000 osd.1 (osd.1) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:09.637650+0000 osd.1 (osd.1) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:10.412834+0000 mon.a (mon.0) 2497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:46:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:10.412913+0000 mon.a (mon.0) 2498 : cluster [DBG] osdmap e411: 8 total, 8 up, 4 in 2024-02-13T00:46:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:10.443120+0000 mon.a (mon.0) 2499 : audit [INF] from='client.? 172.21.15.162:0/1345951736' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:46:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:09.470418+0000 mon.a (mon.0) 2494 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:10.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:09.470553+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e410: 8 total, 8 up, 4 in 2024-02-13T00:46:10.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:09.481872+0000 mon.b (mon.1) 574 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:46:10.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:09.504238+0000 mon.b (mon.1) 575 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:10.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:09.504767+0000 mon.a (mon.0) 2496 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:10.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:09.637505+0000 osd.1 (osd.1) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:10.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:09.637650+0000 osd.1 (osd.1) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:10.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:10.412834+0000 mon.a (mon.0) 2497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:46:10.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:10.412913+0000 mon.a (mon.0) 2498 : cluster [DBG] osdmap e411: 8 total, 8 up, 4 in 2024-02-13T00:46:10.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:10.443120+0000 mon.a (mon.0) 2499 : audit [INF] from='client.? 172.21.15.162:0/1345951736' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:46:10.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:10.887 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:10.905 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:10.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:11.009 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:11.038 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:11.060 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:11.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:11.067 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:11.168 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:11.193 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:11.341 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:11.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:11.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:11.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.427+0000 7f41ef779700 1 -- 172.21.15.162:0/1345951736 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v412) v1 ==== 113+0+0 (secure 0 0 0) 0x7f41e80503a0 con 0x7f41f80fcb30 2024-02-13T00:46:11.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.429+0000 7f42002cb700 10 librados: watch_flush enter 2024-02-13T00:46:11.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 10 librados: watch_flush exit 2024-02-13T00:46:11.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 20 client.27303.objecter shutdown clearing up homeless session... 2024-02-13T00:46:11.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 10 client.27303.objecter successfully canceled tick 2024-02-13T00:46:11.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 1 -- 172.21.15.162:0/1345951736 >> v2:172.21.15.162:6800/2866534053 conn(0x7f41e4062c90 msgr2=0x7f41e4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:11.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 1 --2- 172.21.15.162:0/1345951736 >> v2:172.21.15.162:6800/2866534053 conn(0x7f41e4062c90 0x7f41e4065150 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 rx=0x7f41f40043d0 tx=0x7f41f400b040).stop 2024-02-13T00:46:11.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 1 -- 172.21.15.162:0/1345951736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41f80fcb30 msgr2=0x7f41f8104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:11.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 1 --2- 172.21.15.162:0/1345951736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41f80fcb30 0x7f41f8104680 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 rx=0x7f41e800b3f0 tx=0x7f41e8006f60).stop 2024-02-13T00:46:11.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 1 -- 172.21.15.162:0/1345951736 shutdown_connections 2024-02-13T00:46:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 1 --2- 172.21.15.162:0/1345951736 >> v2:172.21.15.162:6800/2866534053 conn(0x7f41e4062c90 0x7f41e4065150 secure :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 rx=0x7f41f40043d0 tx=0x7f41f400b040).stop 2024-02-13T00:46:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 1 --2- 172.21.15.162:0/1345951736 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41f8104b70 0x7f41f81167e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 1 --2- 172.21.15.162:0/1345951736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41f80f73a0 0x7f41f8104190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 1 --2- 172.21.15.162:0/1345951736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41f80fcb30 0x7f41f8104680 secure :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 rx=0x7f41e800b3f0 tx=0x7f41e8006f60).stop 2024-02-13T00:46:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.430+0000 7f42002cb700 1 -- 172.21.15.162:0/1345951736 >> 172.21.15.162:0/1345951736 conn(0x7f41f806eb90 msgr2=0x7f41f80fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.433+0000 7f42002cb700 1 -- 172.21.15.162:0/1345951736 shutdown_connections 2024-02-13T00:46:11.447 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:11.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.444+0000 7f42002cb700 1 -- 172.21.15.162:0/1345951736 wait complete. 2024-02-13T00:46:11.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:11.444+0000 7f42002cb700 1 librados: shutdown 2024-02-13T00:46:11.450 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.0 to 1 (10000) 2024-02-13T00:46:11.475 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:11.495 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:11.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:11.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:11.615 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:10.417328+0000 mgr.y (mgr.14544) 356 : cluster [DBG] pgmap v589: 238 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering, 2 active+recovery_wait, 7 active+recovery_wait+degraded, 223 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 516 B/s wr, 1 op/s; 146/1146 objects degraded (12.740%); 13/1146 objects misplaced (1.134%); 228 B/s, 43 objects/s recovering 2024-02-13T00:46:11.615 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:11.427808+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? 172.21.15.162:0/1345951736' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:46:11.615 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:11.427881+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e412: 8 total, 8 up, 4 in 2024-02-13T00:46:11.616 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:10.417328+0000 mgr.y (mgr.14544) 356 : cluster [DBG] pgmap v589: 238 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering, 2 active+recovery_wait, 7 active+recovery_wait+degraded, 223 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 516 B/s wr, 1 op/s; 146/1146 objects degraded (12.740%); 13/1146 objects misplaced (1.134%); 228 B/s, 43 objects/s recovering 2024-02-13T00:46:11.616 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:11.427808+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? 172.21.15.162:0/1345951736' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:46:11.616 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:11.427881+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e412: 8 total, 8 up, 4 in 2024-02-13T00:46:11.628 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:11.666 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:11.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:11.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:11.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:10.417328+0000 mgr.y (mgr.14544) 356 : cluster [DBG] pgmap v589: 238 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering, 2 active+recovery_wait, 7 active+recovery_wait+degraded, 223 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 516 B/s wr, 1 op/s; 146/1146 objects degraded (12.740%); 13/1146 objects misplaced (1.134%); 228 B/s, 43 objects/s recovering 2024-02-13T00:46:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:11.427808+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? 172.21.15.162:0/1345951736' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:46:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:11.427881+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e412: 8 total, 8 up, 4 in 2024-02-13T00:46:11.798 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:11.816 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:11.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:11.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:11.946 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:11.969 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:11.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:12.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:12.098 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:12.117 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:12.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:12.220 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:46:12.220 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:46:12.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:12.246 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:12.264 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:12.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:12.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:12.368 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:12.393 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:12.412 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:12.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:12.516 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:12.542 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:12.567 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:12.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:12.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:12.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:46:12.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:12.696 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:12.714 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:12.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:12.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:12.845 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:13.093 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:13.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:13.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:13.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 1 1 2024-02-13T00:46:13.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:13.223 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:13.242 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:13.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:13.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:13.373 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:13.523 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:13.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:13.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-13T00:46:13.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:12.417855+0000 mgr.y (mgr.14544) 357 : cluster [DBG] pgmap v591: 238 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering, 2 active+recovery_wait, 7 active+recovery_wait+degraded, 223 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 146/1146 objects degraded (12.740%); 13/1146 objects misplaced (1.134%); 226 B/s, 42 objects/s recovering 2024-02-13T00:46:13.544 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:12.429932+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e413: 8 total, 8 up, 4 in 2024-02-13T00:46:13.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:13.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:12.417855+0000 mgr.y (mgr.14544) 357 : cluster [DBG] pgmap v591: 238 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering, 2 active+recovery_wait, 7 active+recovery_wait+degraded, 223 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 146/1146 objects degraded (12.740%); 13/1146 objects misplaced (1.134%); 226 B/s, 42 objects/s recovering 2024-02-13T00:46:13.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:12.429932+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e413: 8 total, 8 up, 4 in 2024-02-13T00:46:13.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:12.417855+0000 mgr.y (mgr.14544) 357 : cluster [DBG] pgmap v591: 238 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering, 2 active+recovery_wait, 7 active+recovery_wait+degraded, 223 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 146/1146 objects degraded (12.740%); 13/1146 objects misplaced (1.134%); 226 B/s, 42 objects/s recovering 2024-02-13T00:46:13.652 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:12.429932+0000 mon.a (mon.0) 2502 : cluster [DBG] osdmap e413: 8 total, 8 up, 4 in 2024-02-13T00:46:13.653 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:14.010 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:14.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:14.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:14.143 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:14.283 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:14.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:14.296 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:14.299 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:14.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:14.416 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:14.451 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:14.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:14.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:14.558 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:14.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:13.431158+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e414: 8 total, 8 up, 4 in 2024-02-13T00:46:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:13.431158+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e414: 8 total, 8 up, 4 in 2024-02-13T00:46:14.590 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:14.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:14.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:14.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:13.431158+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e414: 8 total, 8 up, 4 in 2024-02-13T00:46:14.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:14.850 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:15.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:46:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:46:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:14.418326+0000 mgr.y (mgr.14544) 358 : cluster [DBG] pgmap v594: 238 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering, 2 active+recovery_wait, 7 active+recovery_wait+degraded, 223 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 146/1146 objects degraded (12.740%); 13/1146 objects misplaced (1.134%) 2024-02-13T00:46:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:14.442386+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e415: 8 total, 8 up, 4 in 2024-02-13T00:46:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:14.552869+0000 mon.c (mon.2) 377 : audit [DBG] from='client.? 172.21.15.162:0/2308198972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:15.179526+0000 mon.a (mon.0) 2505 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:15.404807+0000 mon.a (mon.0) 2506 : cluster [WRN] Health check update: Degraded data redundancy: 146/1146 objects degraded (12.740%), 9 pgs degraded (PG_DEGRADED) 2024-02-13T00:46:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:14.418326+0000 mgr.y (mgr.14544) 358 : cluster [DBG] pgmap v594: 238 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering, 2 active+recovery_wait, 7 active+recovery_wait+degraded, 223 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 146/1146 objects degraded (12.740%); 13/1146 objects misplaced (1.134%) 2024-02-13T00:46:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:14.442386+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e415: 8 total, 8 up, 4 in 2024-02-13T00:46:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:14.552869+0000 mon.c (mon.2) 377 : audit [DBG] from='client.? 172.21.15.162:0/2308198972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:15.179526+0000 mon.a (mon.0) 2505 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:15.404807+0000 mon.a (mon.0) 2506 : cluster [WRN] Health check update: Degraded data redundancy: 146/1146 objects degraded (12.740%), 9 pgs degraded (PG_DEGRADED) 2024-02-13T00:46:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:14.418326+0000 mgr.y (mgr.14544) 358 : cluster [DBG] pgmap v594: 238 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering, 2 active+recovery_wait, 7 active+recovery_wait+degraded, 223 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 146/1146 objects degraded (12.740%); 13/1146 objects misplaced (1.134%) 2024-02-13T00:46:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:14.442386+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e415: 8 total, 8 up, 4 in 2024-02-13T00:46:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:14.552869+0000 mon.c (mon.2) 377 : audit [DBG] from='client.? 172.21.15.162:0/2308198972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:15.179526+0000 mon.a (mon.0) 2505 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:15.404807+0000 mon.a (mon.0) 2506 : cluster [WRN] Health check update: Degraded data redundancy: 146/1146 objects degraded (12.740%), 9 pgs degraded (PG_DEGRADED) 2024-02-13T00:46:15.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.549+0000 7fe67d4e6700 1 -- 172.21.15.162:0/1531637538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6780f8d10 msgr2=0x7fe6780f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:15.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.549+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/1531637538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6780f8d10 0x7fe6780f90f0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 rx=0x7fe668004870 tx=0x7fe668009cb0).stop 2024-02-13T00:46:15.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.549+0000 7fe67d4e6700 1 -- 172.21.15.162:0/1531637538 shutdown_connections 2024-02-13T00:46:15.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.549+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/1531637538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe678102270 0x7fe678104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.549+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/1531637538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6780f8d10 0x7fe6780f90f0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.549+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/1531637538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6780f95e0 0x7fe678101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.549+0000 7fe67d4e6700 1 -- 172.21.15.162:0/1531637538 >> 172.21.15.162:0/1531637538 conn(0x7fe67806eb90 msgr2=0x7fe6780f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.549+0000 7fe67d4e6700 1 -- 172.21.15.162:0/1531637538 shutdown_connections 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.549+0000 7fe67d4e6700 1 -- 172.21.15.162:0/1531637538 wait complete. 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.549+0000 7fe67d4e6700 1 librados: starting msgr at 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.549+0000 7fe67d4e6700 1 librados: starting objecter 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.549+0000 7fe67d4e6700 1 Processor -- start 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe67d4e6700 1 -- start start 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe67d4e6700 1 librados: setting wanted keys 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe67d4e6700 1 librados: calling monclient init 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe67d4e6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6780f8d10 0x7fe6780f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe67d4e6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6780f95e0 0x7fe6780f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:15.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe67d4e6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe678102270 0x7fe6780f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe67d4e6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781167a0 con 0x7fe6780f8d10 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe67d4e6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe678116910 con 0x7fe678102270 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe67d4e6700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe678116ad0 con 0x7fe6780f95e0 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe6767fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6780f95e0 0x7fe6780f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe676ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6780f8d10 0x7fe6780f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe6767fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6780f95e0 0x7fe6780f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:42978/0 (socket says 172.21.15.162:42978) 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe6767fc700 1 -- 172.21.15.162:0/2308198972 learned_addr learned my addr 172.21.15.162:0/2308198972 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe6777fe700 1 --2- 172.21.15.162:0/2308198972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe678102270 0x7fe6780f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe6767fc700 1 -- 172.21.15.162:0/2308198972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe678102270 msgr2=0x7fe6780f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe6767fc700 1 --2- 172.21.15.162:0/2308198972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe678102270 0x7fe6780f76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe6767fc700 1 -- 172.21.15.162:0/2308198972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6780f8d10 msgr2=0x7fe6780f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe6767fc700 1 --2- 172.21.15.162:0/2308198972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6780f8d10 0x7fe6780f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe6767fc700 1 -- 172.21.15.162:0/2308198972 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6700097c0 con 0x7fe6780f95e0 2024-02-13T00:46:15.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe676ffd700 1 --2- 172.21.15.162:0/2308198972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6780f8d10 0x7fe6780f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe6777fe700 1 --2- 172.21.15.162:0/2308198972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe678102270 0x7fe6780f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe6767fc700 1 --2- 172.21.15.162:0/2308198972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6780f95e0 0x7fe6780f71b0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 rx=0x7fe66c00e990 tx=0x7fe66c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe65ffff700 10 client.?.objecter ms_handle_connect 0x7fe6780f95e0 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe65ffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.550+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c004680 con 0x7fe6780f95e0 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.551+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe66c00f040 con 0x7fe6780f95e0 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.551+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c0146b0 con 0x7fe6780f95e0 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.551+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe668005040 con 0x7fe6780f95e0 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.551+0000 7fe67d4e6700 10 client.37409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.551+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe678117060 con 0x7fe6780f95e0 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.551+0000 7fe67d4e6700 1 librados: init done 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.551+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe6780475e0 con 0x7fe6780f95e0 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.552+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe66c014ae0 con 0x7fe6780f95e0 2024-02-13T00:46:15.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.552+0000 7fe65ffff700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe660062c90 0x7fe660065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:15.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.552+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(415..415 src has 52..415) v4 ==== 8081+0+0 (secure 0 0 0) 0x7fe66c07d030 con 0x7fe6780f95e0 2024-02-13T00:46:15.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.552+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 osd_map(415..415 src has 52..415) v4 2024-02-13T00:46:15.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.552+0000 7fe65ffff700 3 client.37409.objecter handle_osd_map got epochs [415,415] > 0 2024-02-13T00:46:15.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.552+0000 7fe65ffff700 3 client.37409.objecter handle_osd_map decoding full epoch 415 2024-02-13T00:46:15.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe676ffd700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe660062c90 0x7fe660065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:15.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe65ffff700 20 client.37409.objecter dump_active .. 0 homeless 2024-02-13T00:46:15.994 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v415) v1 ==== 54+0+16 (secure 0 0 0) 0x7fe66c07d560 con 0x7fe6780f95e0 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 1 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe678028590 0x7fe6780fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe6781071b0 osd=0 3 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe6781071b0 osd=0 3 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe6781071b0 osd=0 3 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 1 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe6781071b0 osd=0 3 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 0 1 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe6781071b0 osd=0 4 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe67d4e6700 10 client.37409.objecter _send_command 1 2024-02-13T00:46:15.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6780f8200 con 0x7fe678028590 2024-02-13T00:46:15.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe65ffff700 10 client.37409.objecter ms_handle_connect 0x7fe660062c90 2024-02-13T00:46:15.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe676ffd700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe660062c90 0x7fe660065150 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 rx=0x7fe668004ee0 tx=0x7fe66800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:15.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe6777fe700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe678028590 0x7fe6780fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:15.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe65ffff700 10 client.37409.objecter ms_handle_connect 0x7fe678028590 2024-02-13T00:46:15.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.553+0000 7fe6777fe700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe678028590 0x7fe6780fbd00 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:15.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.554+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe6780f8200 con 0x7fe678028590 2024-02-13T00:46:15.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.554+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:46:15.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.554+0000 7fe65ffff700 10 client.37409.objecter _finish_command 1 = system:0 2024-02-13T00:46:15.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.554+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe6781071b0 osd=0 4 2024-02-13T00:46:15.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.554+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 0 1 2024-02-13T00:46:15.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.571+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.571+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 2 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.571+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe6781071b0 osd=0 3 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.571+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe6781071b0 osd=0 3 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.571+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe6781071b0 osd=0 3 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.571+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 2 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.571+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe6781071b0 osd=0 3 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.571+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 0 2 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.571+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe6781071b0 osd=0 4 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.571+0000 7fe67d4e6700 10 client.37409.objecter _send_command 2 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.571+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe67805a0e0 con 0x7fe678028590 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.573+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe67805a0e0 con 0x7fe678028590 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.573+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.573+0000 7fe65ffff700 10 client.37409.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.573+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe6781071b0 osd=0 4 2024-02-13T00:46:15.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.573+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 0 2 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.573+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.573+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 3 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.573+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6805/150599780 conn(0x7fe678027be0 0x7fe678028000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.574+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe6780fd370 osd=1 3 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.574+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe6780fd370 osd=1 3 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.574+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe6780fd370 osd=1 3 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.574+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 3 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.574+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe6780fd370 osd=1 3 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.574+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 1 3 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.574+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe6780fd370 osd=1 4 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.574+0000 7fe67d4e6700 10 client.37409.objecter _send_command 3 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.574+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6780fd4e0 con 0x7fe678027be0 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.574+0000 7fe676ffd700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6805/150599780 conn(0x7fe678027be0 0x7fe678028000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.574+0000 7fe65ffff700 10 client.37409.objecter ms_handle_connect 0x7fe678027be0 2024-02-13T00:46:15.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.574+0000 7fe676ffd700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6805/150599780 conn(0x7fe678027be0 0x7fe678028000 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:16.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.575+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe6780fd4e0 con 0x7fe678027be0 2024-02-13T00:46:16.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.575+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:46:16.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.575+0000 7fe65ffff700 10 client.37409.objecter _finish_command 3 = system:0 2024-02-13T00:46:16.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.575+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe6780fd370 osd=1 4 2024-02-13T00:46:16.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.575+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 1 3 2024-02-13T00:46:16.000 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:46:16.000 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.590+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.590+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 4 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.590+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe6780fd370 osd=1 3 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.590+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe6780fd370 osd=1 3 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.590+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe6780fd370 osd=1 3 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.590+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 4 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.590+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe6780fd370 osd=1 3 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.590+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 1 4 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.590+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe6780fd370 osd=1 4 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.590+0000 7fe67d4e6700 10 client.37409.objecter _send_command 4 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.590+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe678027650 con 0x7fe678027be0 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe678027650 con 0x7fe678027be0 2024-02-13T00:46:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe65ffff700 10 client.37409.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe6780fd370 osd=1 4 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 1 4 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 5 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe6780fced0 0x7fe67811e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678027a60 osd=2 3 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678027a60 osd=2 3 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678027a60 osd=2 3 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 5 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe678027a60 osd=2 3 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 2 5 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678027a60 osd=2 4 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe67d4e6700 10 client.37409.objecter _send_command 5 2024-02-13T00:46:16.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe678121080 con 0x7fe6780fced0 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.592+0000 7fe6767fc700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe6780fced0 0x7fe67811e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.593+0000 7fe65ffff700 10 client.37409.objecter ms_handle_connect 0x7fe6780fced0 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.593+0000 7fe6767fc700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe6780fced0 0x7fe67811e3a0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.593+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe678121080 con 0x7fe6780fced0 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.593+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.593+0000 7fe65ffff700 10 client.37409.objecter _finish_command 5 = system:0 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.593+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe678027a60 osd=2 4 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.593+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 2 5 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.611+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.611+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 6 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.611+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678027a60 osd=2 3 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.611+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678027a60 osd=2 3 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.611+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678027a60 osd=2 3 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.611+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 6 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.611+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe678027a60 osd=2 3 2024-02-13T00:46:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.611+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 2 6 2024-02-13T00:46:16.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.611+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678027a60 osd=2 4 2024-02-13T00:46:16.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.611+0000 7fe67d4e6700 10 client.37409.objecter _send_command 6 2024-02-13T00:46:16.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.611+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6780fc490 con 0x7fe6780fced0 2024-02-13T00:46:16.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe6780fc490 con 0x7fe6780fced0 2024-02-13T00:46:16.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:16.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe65ffff700 10 client.37409.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe678027a60 osd=2 4 2024-02-13T00:46:16.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 2 6 2024-02-13T00:46:16.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:16.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 7 2024-02-13T00:46:16.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6813/604971011 conn(0x7fe67811fa10 0x7fe67811fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:16.005 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:46:16.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe67811b9e0 osd=3 3 2024-02-13T00:46:16.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe67811b9e0 osd=3 3 2024-02-13T00:46:16.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe67811b9e0 osd=3 3 2024-02-13T00:46:16.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 7 2024-02-13T00:46:16.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe67811b9e0 osd=3 3 2024-02-13T00:46:16.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 3 7 2024-02-13T00:46:16.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe67811b9e0 osd=3 4 2024-02-13T00:46:16.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe67d4e6700 10 client.37409.objecter _send_command 7 2024-02-13T00:46:16.005 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe67811c120 con 0x7fe67811fa10 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe6777fe700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6813/604971011 conn(0x7fe67811fa10 0x7fe67811fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe65ffff700 10 client.37409.objecter ms_handle_connect 0x7fe67811fa10 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.613+0000 7fe6777fe700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6813/604971011 conn(0x7fe67811fa10 0x7fe67811fe50 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.614+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe67811c120 con 0x7fe67811fa10 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.614+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.614+0000 7fe65ffff700 10 client.37409.objecter _finish_command 7 = system:0 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.614+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe67811b9e0 osd=3 4 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.614+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 3 7 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.629+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.629+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 8 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.629+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe67811b9e0 osd=3 3 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.629+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe67811b9e0 osd=3 3 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.629+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe67811b9e0 osd=3 3 2024-02-13T00:46:16.006 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.629+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 8 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.629+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe67811b9e0 osd=3 3 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.629+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 3 8 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.629+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe67811b9e0 osd=3 4 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.629+0000 7fe67d4e6700 10 client.37409.objecter _send_command 8 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.629+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe678120b10 con 0x7fe67811fa10 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe678120b10 con 0x7fe67811fa10 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe65ffff700 10 client.37409.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe67811b9e0 osd=3 4 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 3 8 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 9 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe678123230 0x7fe678125620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678120340 osd=4 3 2024-02-13T00:46:16.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678120340 osd=4 3 2024-02-13T00:46:16.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678120340 osd=4 3 2024-02-13T00:46:16.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 9 2024-02-13T00:46:16.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe678120340 osd=4 3 2024-02-13T00:46:16.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 4 9 2024-02-13T00:46:16.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678120340 osd=4 4 2024-02-13T00:46:16.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe67d4e6700 10 client.37409.objecter _send_command 9 2024-02-13T00:46:16.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe678120a90 con 0x7fe678123230 2024-02-13T00:46:16.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe676ffd700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe678123230 0x7fe678125620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:16.008 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe65ffff700 10 client.37409.objecter ms_handle_connect 0x7fe678123230 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.631+0000 7fe676ffd700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe678123230 0x7fe678125620 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.632+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe678120a90 con 0x7fe678123230 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.632+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.632+0000 7fe65ffff700 10 client.37409.objecter _finish_command 9 = system:0 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.632+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe678120340 osd=4 4 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.632+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 4 9 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.647+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.647+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 10 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.647+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678120340 osd=4 3 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.647+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678120340 osd=4 3 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.647+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678120340 osd=4 3 2024-02-13T00:46:16.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.647+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 10 2024-02-13T00:46:16.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.647+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe678120340 osd=4 3 2024-02-13T00:46:16.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.647+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 4 10 2024-02-13T00:46:16.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.647+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678120340 osd=4 4 2024-02-13T00:46:16.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.647+0000 7fe67d4e6700 10 client.37409.objecter _send_command 10 2024-02-13T00:46:16.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.647+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe678120a90 con 0x7fe678123230 2024-02-13T00:46:16.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe678120a90 con 0x7fe678123230 2024-02-13T00:46:16.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:16.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe65ffff700 10 client.37409.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe678120340 osd=4 4 2024-02-13T00:46:16.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 4 10 2024-02-13T00:46:16.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 11 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6804/2212296748 conn(0x7fe678126060 0x7fe6781264c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678120a90 osd=5 3 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678120a90 osd=5 3 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678120a90 osd=5 3 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 11 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe678120a90 osd=5 3 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 5 11 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678120a90 osd=5 4 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe67d4e6700 10 client.37409.objecter _send_command 11 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe67812b480 con 0x7fe678126060 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe6767fc700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6804/2212296748 conn(0x7fe678126060 0x7fe6781264c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe6767fc700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6804/2212296748 conn(0x7fe678126060 0x7fe6781264c0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:16.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.650+0000 7fe65ffff700 10 client.37409.objecter ms_handle_connect 0x7fe678126060 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.651+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe67812b480 con 0x7fe678126060 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.651+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.651+0000 7fe65ffff700 10 client.37409.objecter _finish_command 11 = system:0 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.651+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe678120a90 osd=5 4 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.651+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 5 11 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.666+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.666+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 12 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.666+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678120a90 osd=5 3 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.666+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678120a90 osd=5 3 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.666+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678120a90 osd=5 3 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.666+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 12 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.666+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe678120a90 osd=5 3 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.666+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 5 12 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.666+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678120a90 osd=5 4 2024-02-13T00:46:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.666+0000 7fe67d4e6700 10 client.37409.objecter _send_command 12 2024-02-13T00:46:16.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.666+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe678126aa0 con 0x7fe678126060 2024-02-13T00:46:16.013 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:46:16.013 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:46:16.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.668+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe678126aa0 con 0x7fe678126060 2024-02-13T00:46:16.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.668+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:16.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.668+0000 7fe65ffff700 10 client.37409.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.668+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe678120a90 osd=5 4 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.668+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 5 12 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 13 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe678127120 0x7fe678127560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678126aa0 osd=6 3 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678126aa0 osd=6 3 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678126aa0 osd=6 3 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 13 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe678126aa0 osd=6 3 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 6 13 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678126aa0 osd=6 4 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe67d4e6700 10 client.37409.objecter _send_command 13 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe678127ba0 con 0x7fe678127120 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe6777fe700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe678127120 0x7fe678127560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe65ffff700 10 client.37409.objecter ms_handle_connect 0x7fe678127120 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.669+0000 7fe6777fe700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe678127120 0x7fe678127560 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.670+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe678127ba0 con 0x7fe678127120 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.670+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.670+0000 7fe65ffff700 10 client.37409.objecter _finish_command 13 = system:0 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.670+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe678126aa0 osd=6 4 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.670+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 6 13 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.684+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.684+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 14 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.684+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678126aa0 osd=6 3 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.684+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678126aa0 osd=6 3 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.684+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678126aa0 osd=6 3 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.684+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 14 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.684+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe678126aa0 osd=6 3 2024-02-13T00:46:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.684+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 6 14 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.684+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678126aa0 osd=6 4 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.684+0000 7fe67d4e6700 10 client.37409.objecter _send_command 14 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.684+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe678127ba0 con 0x7fe678127120 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe678127ba0 con 0x7fe678127120 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe65ffff700 10 client.37409.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe678126aa0 osd=6 4 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 6 14 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 15 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe67812c820 0x7fe67812ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678127ba0 osd=7 3 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678127ba0 osd=7 3 2024-02-13T00:46:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678127ba0 osd=7 3 2024-02-13T00:46:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 15 2024-02-13T00:46:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe678127ba0 osd=7 3 2024-02-13T00:46:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 7 15 2024-02-13T00:46:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678127ba0 osd=7 4 2024-02-13T00:46:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe67d4e6700 10 client.37409.objecter _send_command 15 2024-02-13T00:46:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe678131860 con 0x7fe67812c820 2024-02-13T00:46:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.687+0000 7fe676ffd700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe67812c820 0x7fe67812ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.688+0000 7fe65ffff700 10 client.37409.objecter ms_handle_connect 0x7fe67812c820 2024-02-13T00:46:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.688+0000 7fe676ffd700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe67812c820 0x7fe67812ec10 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:16.017 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:46:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.688+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe678131860 con 0x7fe67812c820 2024-02-13T00:46:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.688+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:46:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.688+0000 7fe65ffff700 10 client.37409.objecter _finish_command 15 = system:0 2024-02-13T00:46:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.688+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe678127ba0 osd=7 4 2024-02-13T00:46:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.688+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 7 15 2024-02-13T00:46:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.703+0000 7fe67d4e6700 10 client.37409.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.703+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign -1 16 2024-02-13T00:46:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.703+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678127ba0 osd=7 3 2024-02-13T00:46:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.703+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678127ba0 osd=7 3 2024-02-13T00:46:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.703+0000 7fe67d4e6700 20 client.37409.objecter _get_session s=0x7fe678127ba0 osd=7 3 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.703+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_remove -1 16 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.703+0000 7fe67d4e6700 20 client.37409.objecter get_session s=0x7fe678127ba0 osd=7 3 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.703+0000 7fe67d4e6700 15 client.37409.objecter _session_command_op_assign 7 16 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.703+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678127ba0 osd=7 4 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.703+0000 7fe67d4e6700 10 client.37409.objecter _send_command 16 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.703+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe67812f100 con 0x7fe67812c820 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.704+0000 7fe65ffff700 1 -- 172.21.15.162:0/2308198972 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe67812f100 con 0x7fe67812c820 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.704+0000 7fe65ffff700 10 client.37409.objecter ms_dispatch 0x7fe678001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.704+0000 7fe65ffff700 10 client.37409.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.704+0000 7fe65ffff700 20 client.37409.objecter put_session s=0x7fe678127ba0 osd=7 4 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.704+0000 7fe65ffff700 15 client.37409.objecter _session_command_op_remove 7 16 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.704+0000 7fe67d4e6700 10 librados: watch_flush enter 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 10 librados: watch_flush exit 2024-02-13T00:46:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 10 client.37409.objecter close_session for osd.0 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe678028590 msgr2=0x7fe6780fbd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe678028590 0x7fe6780fbd00 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe6781071b0 osd=0 1 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 10 client.37409.objecter close_session for osd.1 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6805/150599780 conn(0x7fe678027be0 msgr2=0x7fe678028000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6805/150599780 conn(0x7fe678027be0 0x7fe678028000 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe6780fd370 osd=1 1 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 10 client.37409.objecter close_session for osd.2 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe6780fced0 msgr2=0x7fe67811e3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe6780fced0 0x7fe67811e3a0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678027a60 osd=2 1 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 10 client.37409.objecter close_session for osd.3 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe6777fe700 1 -- 172.21.15.162:0/2308198972 reap_dead start 2024-02-13T00:46:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6813/604971011 conn(0x7fe67811fa10 msgr2=0x7fe67811fe50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6813/604971011 conn(0x7fe67811fa10 0x7fe67811fe50 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe67811b9e0 osd=3 1 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 10 client.37409.objecter close_session for osd.4 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe678123230 msgr2=0x7fe678125620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe678123230 0x7fe678125620 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678120340 osd=4 1 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 10 client.37409.objecter close_session for osd.5 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6804/2212296748 conn(0x7fe678126060 msgr2=0x7fe6781264c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6804/2212296748 conn(0x7fe678126060 0x7fe6781264c0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678120a90 osd=5 1 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 10 client.37409.objecter close_session for osd.6 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe678127120 msgr2=0x7fe678127560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe678127120 0x7fe678127560 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678126aa0 osd=6 1 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 10 client.37409.objecter close_session for osd.7 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe67812c820 msgr2=0x7fe67812ec10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe67812c820 0x7fe67812ec10 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 20 client.37409.objecter put_session s=0x7fe678127ba0 osd=7 1 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 20 client.37409.objecter shutdown clearing up homeless session... 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 10 client.37409.objecter successfully canceled tick 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe6777fe700 1 -- 172.21.15.162:0/2308198972 reap_dead start 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe660062c90 msgr2=0x7fe660065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe660062c90 0x7fe660065150 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 rx=0x7fe668004ee0 tx=0x7fe66800b040).stop 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6780f95e0 msgr2=0x7fe6780f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6780f95e0 0x7fe6780f71b0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 rx=0x7fe66c00e990 tx=0x7fe66c00b040).stop 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 shutdown_connections 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6780f95e0 0x7fe6780f71b0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 --2- 172.21.15.162:0/2308198972 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe660062c90 0x7fe660065150 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 >> 172.21.15.162:0/2308198972 conn(0x7fe67806eb90 msgr2=0x7fe678102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 shutdown_connections 2024-02-13T00:46:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 -- 172.21.15.162:0/2308198972 wait complete. 2024-02-13T00:46:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:14.705+0000 7fe67d4e6700 1 librados: shutdown 2024-02-13T00:46:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:46:16.501 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:46:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:46:16] "GET /metrics HTTP/1.1" 200 185129 "" "Prometheus/2.33.4" 2024-02-13T00:46:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:15.609316+0000 mgr.y (mgr.14544) 359 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:16.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:15.609316+0000 mgr.y (mgr.14544) 359 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:16.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:15.609316+0000 mgr.y (mgr.14544) 359 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:17.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 -- 172.21.15.162:0/1183035256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6240f73a0 msgr2=0x7fe6241039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:17.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 --2- 172.21.15.162:0/1183035256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6240f73a0 0x7fe6241039a0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7fe61c009be0 tx=0x7fe61c004d10).stop 2024-02-13T00:46:17.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 -- 172.21.15.162:0/1183035256 shutdown_connections 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 --2- 172.21.15.162:0/1183035256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6240f73a0 0x7fe6241039a0 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 --2- 172.21.15.162:0/1183035256 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6240fcb30 0x7fe6240fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 --2- 172.21.15.162:0/1183035256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6240fd490 0x7fe6240f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 -- 172.21.15.162:0/1183035256 >> 172.21.15.162:0/1183035256 conn(0x7fe62406eb90 msgr2=0x7fe6240f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 -- 172.21.15.162:0/1183035256 shutdown_connections 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 -- 172.21.15.162:0/1183035256 wait complete. 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 librados: starting msgr at 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 librados: starting objecter 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 Processor -- start 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 -- start start 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 librados: setting wanted keys 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 librados: calling monclient init 2024-02-13T00:46:17.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.287+0000 7fe62b101700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6240f73a0 0x7fe6240f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62b101700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6240fcb30 0x7fe6240f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62b101700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6240fd490 0x7fe6240f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62b101700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe624116440 con 0x7fe6240fd490 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62b101700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6241165b0 con 0x7fe6240fcb30 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62b101700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe624116770 con 0x7fe6240f73a0 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe628e9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6240f73a0 0x7fe6240f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe628e9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6240f73a0 0x7fe6240f8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:42996/0 (socket says 172.21.15.162:42996) 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62969e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6240fd490 0x7fe6240f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe628e9d700 1 -- 172.21.15.162:0/2147348376 learned_addr learned my addr 172.21.15.162:0/2147348376 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe623fff700 1 --2- 172.21.15.162:0/2147348376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6240fcb30 0x7fe6240f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62969e700 1 -- 172.21.15.162:0/2147348376 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6240f73a0 msgr2=0x7fe6240f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62969e700 1 --2- 172.21.15.162:0/2147348376 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6240f73a0 0x7fe6240f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62969e700 1 -- 172.21.15.162:0/2147348376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6240fcb30 msgr2=0x7fe6240f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:17.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62969e700 1 --2- 172.21.15.162:0/2147348376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6240fcb30 0x7fe6240f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62969e700 1 -- 172.21.15.162:0/2147348376 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c0097c0 con 0x7fe6240fd490 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe628e9d700 1 --2- 172.21.15.162:0/2147348376 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6240f73a0 0x7fe6240f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62969e700 1 --2- 172.21.15.162:0/2147348376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6240fd490 0x7fe6240f9310 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7fe61c00b6d0 tx=0x7fe61c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe621ffb700 10 client.?.objecter ms_handle_connect 0x7fe6240fd490 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe621ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe621ffb700 1 -- 172.21.15.162:0/2147348376 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c00e040 con 0x7fe6240fd490 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe621ffb700 1 -- 172.21.15.162:0/2147348376 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe61c018040 con 0x7fe6240fd490 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62b101700 1 -- 172.21.15.162:0/2147348376 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6241169f0 con 0x7fe6240fd490 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe621ffb700 1 -- 172.21.15.162:0/2147348376 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c013e90 con 0x7fe6240fd490 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62b101700 10 client.27315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62b101700 1 -- 172.21.15.162:0/2147348376 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe624116e70 con 0x7fe6240fd490 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.288+0000 7fe62b101700 1 librados: init done 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.290+0000 7fe62b101700 1 -- 172.21.15.162:0/2147348376 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe624106050 con 0x7fe6240fd490 2024-02-13T00:46:17.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.290+0000 7fe621ffb700 1 -- 172.21.15.162:0/2147348376 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe61c013590 con 0x7fe6240fd490 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.290+0000 7fe621ffb700 1 --2- 172.21.15.162:0/2147348376 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe614062c90 0x7fe614065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.290+0000 7fe621ffb700 1 -- 172.21.15.162:0/2147348376 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(416..416 src has 52..416) v4 ==== 7995+0+0 (secure 0 0 0) 0x7fe61c07dce0 con 0x7fe6240fd490 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.290+0000 7fe621ffb700 10 client.27315.objecter ms_dispatch 0x7fe624001550 osd_map(416..416 src has 52..416) v4 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.290+0000 7fe621ffb700 3 client.27315.objecter handle_osd_map got epochs [416,416] > 0 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.290+0000 7fe621ffb700 3 client.27315.objecter handle_osd_map decoding full epoch 416 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.290+0000 7fe628e9d700 1 --2- 172.21.15.162:0/2147348376 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe614062c90 0x7fe614065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.290+0000 7fe621ffb700 20 client.27315.objecter dump_active .. 0 homeless 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.290+0000 7fe621ffb700 10 client.27315.objecter ms_handle_connect 0x7fe614062c90 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.290+0000 7fe628e9d700 1 --2- 172.21.15.162:0/2147348376 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe614062c90 0x7fe614065150 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 rx=0x7fe618007d30 tx=0x7fe61800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.294+0000 7fe621ffb700 1 -- 172.21.15.162:0/2147348376 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe61c0512e0 con 0x7fe6240fd490 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.489+0000 7fe62b101700 1 -- 172.21.15.162:0/2147348376 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fe62405fb80 con 0x7fe6240fd490 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.506+0000 7fe621ffb700 1 -- 172.21.15.162:0/2147348376 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v417) v1 ==== 91+0+0 (secure 0 0 0) 0x7fe61c021030 con 0x7fe6240fd490 2024-02-13T00:46:17.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 10 librados: watch_flush enter 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 10 librados: watch_flush exit 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 20 client.27315.objecter shutdown clearing up homeless session... 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 10 client.27315.objecter successfully canceled tick 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 1 -- 172.21.15.162:0/2147348376 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe614062c90 msgr2=0x7fe614065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 1 --2- 172.21.15.162:0/2147348376 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe614062c90 0x7fe614065150 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 rx=0x7fe618007d30 tx=0x7fe61800b040).stop 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 1 -- 172.21.15.162:0/2147348376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6240fd490 msgr2=0x7fe6240f9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 1 --2- 172.21.15.162:0/2147348376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6240fd490 0x7fe6240f9310 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7fe61c00b6d0 tx=0x7fe61c004d10).stop 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 1 -- 172.21.15.162:0/2147348376 shutdown_connections 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 1 --2- 172.21.15.162:0/2147348376 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe614062c90 0x7fe614065150 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 1 --2- 172.21.15.162:0/2147348376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6240fd490 0x7fe6240f9310 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 1 --2- 172.21.15.162:0/2147348376 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6240f73a0 0x7fe6240f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 1 --2- 172.21.15.162:0/2147348376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6240fcb30 0x7fe6240f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.509+0000 7fe62b101700 1 -- 172.21.15.162:0/2147348376 >> 172.21.15.162:0/2147348376 conn(0x7fe62406eb90 msgr2=0x7fe624060e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:17.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.510+0000 7fe62b101700 1 -- 172.21.15.162:0/2147348376 shutdown_connections 2024-02-13T00:46:17.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.510+0000 7fe62b101700 1 -- 172.21.15.162:0/2147348376 wait complete. 2024-02-13T00:46:17.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:17.510+0000 7fe62b101700 1 librados: shutdown 2024-02-13T00:46:17.525 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:46:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:16.419335+0000 mgr.y (mgr.14544) 360 : cluster [DBG] pgmap v596: 238 pgs: 1 active+clean+snaptrim, 2 active+recovering+undersized+remapped, 1 active+recovering, 234 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.4 KiB/s rd, 2.2 KiB/s wr, 6 op/s; 15/1167 objects misplaced (1.285%); 124 B/s, 29 objects/s recovering 2024-02-13T00:46:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:16.492830+0000 mon.a (mon.0) 2507 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 146/1146 objects degraded (12.740%), 9 pgs degraded) 2024-02-13T00:46:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:16.501409+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e416: 8 total, 8 up, 4 in 2024-02-13T00:46:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:17.490603+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? 172.21.15.162:0/2147348376' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:46:17.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:16.419335+0000 mgr.y (mgr.14544) 360 : cluster [DBG] pgmap v596: 238 pgs: 1 active+clean+snaptrim, 2 active+recovering+undersized+remapped, 1 active+recovering, 234 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.4 KiB/s rd, 2.2 KiB/s wr, 6 op/s; 15/1167 objects misplaced (1.285%); 124 B/s, 29 objects/s recovering 2024-02-13T00:46:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:16.492830+0000 mon.a (mon.0) 2507 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 146/1146 objects degraded (12.740%), 9 pgs degraded) 2024-02-13T00:46:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:16.501409+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e416: 8 total, 8 up, 4 in 2024-02-13T00:46:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:17.490603+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? 172.21.15.162:0/2147348376' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:46:17.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:16.419335+0000 mgr.y (mgr.14544) 360 : cluster [DBG] pgmap v596: 238 pgs: 1 active+clean+snaptrim, 2 active+recovering+undersized+remapped, 1 active+recovering, 234 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.4 KiB/s rd, 2.2 KiB/s wr, 6 op/s; 15/1167 objects misplaced (1.285%); 124 B/s, 29 objects/s recovering 2024-02-13T00:46:17.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:16.492830+0000 mon.a (mon.0) 2507 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 146/1146 objects degraded (12.740%), 9 pgs degraded) 2024-02-13T00:46:17.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:16.501409+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e416: 8 total, 8 up, 4 in 2024-02-13T00:46:17.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:17.490603+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? 172.21.15.162:0/2147348376' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:46:18.789 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:17.499719+0000 mon.a (mon.0) 2510 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:18.790 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:17.507239+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? 172.21.15.162:0/2147348376' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:46:18.790 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:17.507311+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e417: 8 total, 8 up, 4 in 2024-02-13T00:46:18.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:17.499719+0000 mon.a (mon.0) 2510 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:18.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:17.507239+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? 172.21.15.162:0/2147348376' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:46:18.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:17.507311+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e417: 8 total, 8 up, 4 in 2024-02-13T00:46:18.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:17.499719+0000 mon.a (mon.0) 2510 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:18.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:17.507239+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? 172.21.15.162:0/2147348376' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:46:18.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:17.507311+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e417: 8 total, 8 up, 4 in 2024-02-13T00:46:18.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:46:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:46:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:46:19.403 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:19.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:19.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:18.419992+0000 mgr.y (mgr.14544) 361 : cluster [DBG] pgmap v599: 238 pgs: 1 active+clean+snaptrim, 2 active+recovering+undersized+remapped, 1 active+recovering, 234 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.4 KiB/s rd, 2.2 KiB/s wr, 6 op/s; 15/1167 objects misplaced (1.285%); 124 B/s, 29 objects/s recovering 2024-02-13T00:46:19.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:18.419992+0000 mgr.y (mgr.14544) 361 : cluster [DBG] pgmap v599: 238 pgs: 1 active+clean+snaptrim, 2 active+recovering+undersized+remapped, 1 active+recovering, 234 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.4 KiB/s rd, 2.2 KiB/s wr, 6 op/s; 15/1167 objects misplaced (1.285%); 124 B/s, 29 objects/s recovering 2024-02-13T00:46:19.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:18.419992+0000 mgr.y (mgr.14544) 361 : cluster [DBG] pgmap v599: 238 pgs: 1 active+clean+snaptrim, 2 active+recovering+undersized+remapped, 1 active+recovering, 234 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.4 KiB/s rd, 2.2 KiB/s wr, 6 op/s; 15/1167 objects misplaced (1.285%); 124 B/s, 29 objects/s recovering 2024-02-13T00:46:20.087 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:46:20.088 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:46:20.088 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-13T00:46:20.088 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:46:20.088 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:20.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:20.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:20.220 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:20.369 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:20.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:20.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:20.501 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:19.264051+0000 osd.6 (osd.6) 101 : cluster [DBG] 61.2 scrub starts 2024-02-13T00:46:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:19.267504+0000 osd.6 (osd.6) 102 : cluster [DBG] 61.2 scrub ok 2024-02-13T00:46:20.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:19.264051+0000 osd.6 (osd.6) 101 : cluster [DBG] 61.2 scrub starts 2024-02-13T00:46:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:19.267504+0000 osd.6 (osd.6) 102 : cluster [DBG] 61.2 scrub ok 2024-02-13T00:46:20.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:19.264051+0000 osd.6 (osd.6) 101 : cluster [DBG] 61.2 scrub starts 2024-02-13T00:46:20.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:19.267504+0000 osd.6 (osd.6) 102 : cluster [DBG] 61.2 scrub ok 2024-02-13T00:46:21.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:46:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:20.164511+0000 osd.1 (osd.1) 169 : cluster [DBG] 61.6 scrub starts 2024-02-13T00:46:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:20.166292+0000 osd.1 (osd.1) 170 : cluster [DBG] 61.6 scrub ok 2024-02-13T00:46:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:20.420930+0000 mgr.y (mgr.14544) 362 : cluster [DBG] pgmap v600: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 6 op/s; 129 B/s, 29 objects/s recovering 2024-02-13T00:46:21.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:20.164511+0000 osd.1 (osd.1) 169 : cluster [DBG] 61.6 scrub starts 2024-02-13T00:46:21.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:20.166292+0000 osd.1 (osd.1) 170 : cluster [DBG] 61.6 scrub ok 2024-02-13T00:46:21.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:20.420930+0000 mgr.y (mgr.14544) 362 : cluster [DBG] pgmap v600: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 6 op/s; 129 B/s, 29 objects/s recovering 2024-02-13T00:46:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:20.164511+0000 osd.1 (osd.1) 169 : cluster [DBG] 61.6 scrub starts 2024-02-13T00:46:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:20.166292+0000 osd.1 (osd.1) 170 : cluster [DBG] 61.6 scrub ok 2024-02-13T00:46:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:20.420930+0000 mgr.y (mgr.14544) 362 : cluster [DBG] pgmap v600: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 6 op/s; 129 B/s, 29 objects/s recovering 2024-02-13T00:46:21.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-13T00:46:21.951 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:21.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:21.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:22.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:46:22.087 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:22.108 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:46:22.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:22.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:46:22.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.188+0000 7f9481ce4700 1 -- 172.21.15.162:0/2484806663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f73a0 msgr2=0x7f947c1039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:22.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.188+0000 7f9481ce4700 1 --2- 172.21.15.162:0/2484806663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f73a0 0x7f947c1039a0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 rx=0x7f946c004870 tx=0x7f946c009cb0).stop 2024-02-13T00:46:22.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.188+0000 7f9481ce4700 1 -- 172.21.15.162:0/2484806663 shutdown_connections 2024-02-13T00:46:22.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.188+0000 7f9481ce4700 1 --2- 172.21.15.162:0/2484806663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f73a0 0x7f947c1039a0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:22.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.188+0000 7f9481ce4700 1 --2- 172.21.15.162:0/2484806663 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c0fcb30 0x7f947c0fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:22.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.188+0000 7f9481ce4700 1 --2- 172.21.15.162:0/2484806663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f947c0fd490 0x7f947c0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:22.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.188+0000 7f9481ce4700 1 -- 172.21.15.162:0/2484806663 >> 172.21.15.162:0/2484806663 conn(0x7f947c06eb90 msgr2=0x7f947c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:22.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.188+0000 7f9481ce4700 1 -- 172.21.15.162:0/2484806663 shutdown_connections 2024-02-13T00:46:22.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.188+0000 7f9481ce4700 1 -- 172.21.15.162:0/2484806663 wait complete. 2024-02-13T00:46:22.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.189+0000 7f9481ce4700 1 librados: starting msgr at 2024-02-13T00:46:22.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.189+0000 7f9481ce4700 1 librados: starting objecter 2024-02-13T00:46:22.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.189+0000 7f9481ce4700 1 Processor -- start 2024-02-13T00:46:22.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.189+0000 7f9481ce4700 1 -- start start 2024-02-13T00:46:22.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.189+0000 7f9481ce4700 1 librados: setting wanted keys 2024-02-13T00:46:22.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.189+0000 7f9481ce4700 1 librados: calling monclient init 2024-02-13T00:46:22.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.189+0000 7f9481ce4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f73a0 0x7f947c106d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:22.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.189+0000 7f9481ce4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c0fcb30 0x7f947c107220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:22.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.189+0000 7f9481ce4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f947c0fd490 0x7f947c107850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:22.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.189+0000 7f9481ce4700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c10bf20 con 0x7f947c0f73a0 2024-02-13T00:46:22.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.189+0000 7f9481ce4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c10c090 con 0x7f947c0fd490 2024-02-13T00:46:22.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.189+0000 7f9481ce4700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947c10c250 con 0x7f947c0fcb30 2024-02-13T00:46:22.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f947affd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c0fcb30 0x7f947c107220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f947affd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c0fcb30 0x7f947c107220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:43022/0 (socket says 172.21.15.162:43022) 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f947affd700 1 -- 172.21.15.162:0/3457461224 learned_addr learned my addr 172.21.15.162:0/3457461224 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f947bfff700 1 --2- 172.21.15.162:0/3457461224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f947c0fd490 0x7f947c107850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f947b7fe700 1 --2- 172.21.15.162:0/3457461224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f73a0 0x7f947c106d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f947affd700 1 -- 172.21.15.162:0/3457461224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f947c0fd490 msgr2=0x7f947c107850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f947affd700 1 --2- 172.21.15.162:0/3457461224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f947c0fd490 0x7f947c107850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f947affd700 1 -- 172.21.15.162:0/3457461224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f73a0 msgr2=0x7f947c106d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f947affd700 1 --2- 172.21.15.162:0/3457461224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f73a0 0x7f947c106d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f947affd700 1 -- 172.21.15.162:0/3457461224 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c005040 con 0x7f947c0fcb30 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f947b7fe700 1 --2- 172.21.15.162:0/3457461224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f73a0 0x7f947c106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f947affd700 1 --2- 172.21.15.162:0/3457461224 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c0fcb30 0x7f947c107220 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f947400c930 tx=0x7f9474007e10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f9478ff9700 10 client.?.objecter ms_handle_connect 0x7f947c0fcb30 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f9478ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:22.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f9478ff9700 1 -- 172.21.15.162:0/3457461224 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947400a5c0 con 0x7f947c0fcb30 2024-02-13T00:46:22.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f9481ce4700 1 -- 172.21.15.162:0/3457461224 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947c10c530 con 0x7f947c0fcb30 2024-02-13T00:46:22.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f9481ce4700 10 client.37421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:22.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.190+0000 7f9481ce4700 1 -- 172.21.15.162:0/3457461224 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947c10caf0 con 0x7f947c0fcb30 2024-02-13T00:46:22.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.191+0000 7f9481ce4700 1 librados: init done 2024-02-13T00:46:22.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.191+0000 7f9478ff9700 1 -- 172.21.15.162:0/3457461224 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f947400abf0 con 0x7f947c0fcb30 2024-02-13T00:46:22.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.191+0000 7f9478ff9700 1 -- 172.21.15.162:0/3457461224 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9474012ba0 con 0x7f947c0fcb30 2024-02-13T00:46:22.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.191+0000 7f9481ce4700 1 -- 172.21.15.162:0/3457461224 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9468004f50 con 0x7f947c0fcb30 2024-02-13T00:46:22.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.193+0000 7f9478ff9700 1 -- 172.21.15.162:0/3457461224 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f9474012400 con 0x7f947c0fcb30 2024-02-13T00:46:22.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.193+0000 7f9478ff9700 1 --2- 172.21.15.162:0/3457461224 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9464062c90 0x7f9464065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:22.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.193+0000 7f9478ff9700 1 -- 172.21.15.162:0/3457461224 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(417..417 src has 52..417) v4 ==== 7995+0+0 (secure 0 0 0) 0x7f947407d0d0 con 0x7f947c0fcb30 2024-02-13T00:46:22.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.193+0000 7f9478ff9700 10 client.37421.objecter ms_dispatch 0x7f947c001550 osd_map(417..417 src has 52..417) v4 2024-02-13T00:46:22.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.193+0000 7f9478ff9700 3 client.37421.objecter handle_osd_map got epochs [417,417] > 0 2024-02-13T00:46:22.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.193+0000 7f9478ff9700 3 client.37421.objecter handle_osd_map decoding full epoch 417 2024-02-13T00:46:22.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.193+0000 7f9478ff9700 20 client.37421.objecter dump_active .. 0 homeless 2024-02-13T00:46:22.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.194+0000 7f947b7fe700 1 --2- 172.21.15.162:0/3457461224 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9464062c90 0x7f9464065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:22.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.194+0000 7f947b7fe700 1 --2- 172.21.15.162:0/3457461224 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9464062c90 0x7f9464065150 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f9470007fd0 tx=0x7f947000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:22.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.194+0000 7f9478ff9700 10 client.37421.objecter ms_handle_connect 0x7f9464062c90 2024-02-13T00:46:22.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.198+0000 7f9478ff9700 1 -- 172.21.15.162:0/3457461224 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9474050810 con 0x7f947c0fcb30 2024-02-13T00:46:22.205 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:22.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:22.243 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:22.259 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:22.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.390+0000 7f9481ce4700 1 -- 172.21.15.162:0/3457461224 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f9468005200 con 0x7f947c0fcb30 2024-02-13T00:46:22.494 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:22.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:22.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.525+0000 7f9478ff9700 1 -- 172.21.15.162:0/3457461224 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v418) v1 ==== 113+0+0 (secure 0 0 0) 0x7f9474050400 con 0x7f947c0fcb30 2024-02-13T00:46:22.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 10 librados: watch_flush enter 2024-02-13T00:46:22.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 10 librados: watch_flush exit 2024-02-13T00:46:22.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 20 client.37421.objecter shutdown clearing up homeless session... 2024-02-13T00:46:22.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 10 client.37421.objecter successfully canceled tick 2024-02-13T00:46:22.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 1 -- 172.21.15.162:0/3457461224 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9464062c90 msgr2=0x7f9464065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:22.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 1 --2- 172.21.15.162:0/3457461224 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9464062c90 0x7f9464065150 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f9470007fd0 tx=0x7f947000b040).stop 2024-02-13T00:46:22.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 1 -- 172.21.15.162:0/3457461224 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c0fcb30 msgr2=0x7f947c107220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:22.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 1 --2- 172.21.15.162:0/3457461224 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c0fcb30 0x7f947c107220 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f947400c930 tx=0x7f9474007e10).stop 2024-02-13T00:46:22.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 1 -- 172.21.15.162:0/3457461224 shutdown_connections 2024-02-13T00:46:22.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 1 --2- 172.21.15.162:0/3457461224 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9464062c90 0x7f9464065150 secure :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 rx=0x7f9470007fd0 tx=0x7f947000b040).stop 2024-02-13T00:46:22.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 1 --2- 172.21.15.162:0/3457461224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f947c0fd490 0x7f947c107850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:22.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 1 --2- 172.21.15.162:0/3457461224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f73a0 0x7f947c106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:22.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 1 --2- 172.21.15.162:0/3457461224 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c0fcb30 0x7f947c107220 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:22.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.527+0000 7f9481ce4700 1 -- 172.21.15.162:0/3457461224 >> 172.21.15.162:0/3457461224 conn(0x7f947c06eb90 msgr2=0x7f947c06fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:22.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.529+0000 7f9481ce4700 1 -- 172.21.15.162:0/3457461224 shutdown_connections 2024-02-13T00:46:22.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.530+0000 7f9481ce4700 1 -- 172.21.15.162:0/3457461224 wait complete. 2024-02-13T00:46:22.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:22.530+0000 7f9481ce4700 1 librados: shutdown 2024-02-13T00:46:22.533 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.1 to 1 (10000) 2024-02-13T00:46:22.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:22.631 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:22.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:22.391549+0000 mon.c (mon.2) 378 : audit [INF] from='client.? 172.21.15.162:0/3457461224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:46:22.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:22.391858+0000 mon.a (mon.0) 2513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:46:22.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:22.391549+0000 mon.c (mon.2) 378 : audit [INF] from='client.? 172.21.15.162:0/3457461224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:46:22.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:22.391858+0000 mon.a (mon.0) 2513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:46:22.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:22.391549+0000 mon.c (mon.2) 378 : audit [INF] from='client.? 172.21.15.162:0/3457461224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:46:22.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:22.391858+0000 mon.a (mon.0) 2513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:46:23.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:46:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:22.166348+0000 osd.1 (osd.1) 171 : cluster [DBG] 61.11 scrub starts 2024-02-13T00:46:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:22.170061+0000 osd.1 (osd.1) 172 : cluster [DBG] 61.11 scrub ok 2024-02-13T00:46:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:22.287658+0000 osd.6 (osd.6) 103 : cluster [DBG] 61.13 scrub starts 2024-02-13T00:46:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:22.291164+0000 osd.6 (osd.6) 104 : cluster [DBG] 61.13 scrub ok 2024-02-13T00:46:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:22.421490+0000 mgr.y (mgr.14544) 363 : cluster [DBG] pgmap v601: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s; 97 B/s, 22 objects/s recovering 2024-02-13T00:46:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:22.520940+0000 mon.a (mon.0) 2514 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:46:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:22.520989+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e418: 8 total, 8 up, 4 in 2024-02-13T00:46:23.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:22.166348+0000 osd.1 (osd.1) 171 : cluster [DBG] 61.11 scrub starts 2024-02-13T00:46:23.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:22.170061+0000 osd.1 (osd.1) 172 : cluster [DBG] 61.11 scrub ok 2024-02-13T00:46:23.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:22.287658+0000 osd.6 (osd.6) 103 : cluster [DBG] 61.13 scrub starts 2024-02-13T00:46:23.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:22.291164+0000 osd.6 (osd.6) 104 : cluster [DBG] 61.13 scrub ok 2024-02-13T00:46:23.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:22.421490+0000 mgr.y (mgr.14544) 363 : cluster [DBG] pgmap v601: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s; 97 B/s, 22 objects/s recovering 2024-02-13T00:46:23.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:22.520940+0000 mon.a (mon.0) 2514 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:46:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:22.520989+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e418: 8 total, 8 up, 4 in 2024-02-13T00:46:23.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:22.166348+0000 osd.1 (osd.1) 171 : cluster [DBG] 61.11 scrub starts 2024-02-13T00:46:23.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:22.170061+0000 osd.1 (osd.1) 172 : cluster [DBG] 61.11 scrub ok 2024-02-13T00:46:23.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:22.287658+0000 osd.6 (osd.6) 103 : cluster [DBG] 61.13 scrub starts 2024-02-13T00:46:23.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:22.291164+0000 osd.6 (osd.6) 104 : cluster [DBG] 61.13 scrub ok 2024-02-13T00:46:23.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:22.421490+0000 mgr.y (mgr.14544) 363 : cluster [DBG] pgmap v601: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s; 97 B/s, 22 objects/s recovering 2024-02-13T00:46:23.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:22.520940+0000 mon.a (mon.0) 2514 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:46:23.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:22.520989+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e418: 8 total, 8 up, 4 in 2024-02-13T00:46:24.686 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:23.756385+0000 osd.0 (osd.0) 146 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:46:24.687 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:23.757347+0000 osd.0 (osd.0) 147 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:46:24.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:23.756385+0000 osd.0 (osd.0) 146 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:46:24.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:23.757347+0000 osd.0 (osd.0) 147 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:46:24.780 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:24.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:24.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:23.756385+0000 osd.0 (osd.0) 146 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:46:24.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:23.757347+0000 osd.0 (osd.0) 147 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:46:24.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:24.925 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:25.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 4 1 2024-02-13T00:46:25.533 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:24.421943+0000 mgr.y (mgr.14544) 364 : cluster [DBG] pgmap v603: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 646 B/s rd, 0 op/s; 19 B/s, 3 objects/s recovering 2024-02-13T00:46:25.533 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:24.528362+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e419: 8 total, 8 up, 4 in 2024-02-13T00:46:25.533 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:25.195802+0000 mon.a (mon.0) 2517 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:25.533 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:25.413826+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e420: 8 total, 8 up, 4 in 2024-02-13T00:46:25.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:46:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:46:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:24.421943+0000 mgr.y (mgr.14544) 364 : cluster [DBG] pgmap v603: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 646 B/s rd, 0 op/s; 19 B/s, 3 objects/s recovering 2024-02-13T00:46:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:24.528362+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e419: 8 total, 8 up, 4 in 2024-02-13T00:46:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:25.195802+0000 mon.a (mon.0) 2517 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:25.413826+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e420: 8 total, 8 up, 4 in 2024-02-13T00:46:25.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:24.421943+0000 mgr.y (mgr.14544) 364 : cluster [DBG] pgmap v603: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 646 B/s rd, 0 op/s; 19 B/s, 3 objects/s recovering 2024-02-13T00:46:25.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:24.528362+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e419: 8 total, 8 up, 4 in 2024-02-13T00:46:25.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:25.195802+0000 mon.a (mon.0) 2517 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:25.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:25.413826+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e420: 8 total, 8 up, 4 in 2024-02-13T00:46:25.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.426+0000 7f1fc1695700 1 -- 172.21.15.162:0/3858910775 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fbc0fde50 msgr2=0x7f1fbc0fe230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:25.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.426+0000 7f1fc1695700 1 --2- 172.21.15.162:0/3858910775 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fbc0fde50 0x7f1fbc0fe230 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7f1fac0047b0 tx=0x7f1fac009cb0).stop 2024-02-13T00:46:25.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.427+0000 7f1fc1695700 1 -- 172.21.15.162:0/3858910775 shutdown_connections 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.427+0000 7f1fc1695700 1 --2- 172.21.15.162:0/3858910775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fbc0f6880 0x7f1fbc104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.427+0000 7f1fc1695700 1 --2- 172.21.15.162:0/3858910775 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fbc0fde50 0x7f1fbc0fe230 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.427+0000 7f1fc1695700 1 --2- 172.21.15.162:0/3858910775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fbc0fe7b0 0x7f1fbc0f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.427+0000 7f1fc1695700 1 -- 172.21.15.162:0/3858910775 >> 172.21.15.162:0/3858910775 conn(0x7f1fbc070080 msgr2=0x7f1fbc0f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.427+0000 7f1fc1695700 1 -- 172.21.15.162:0/3858910775 shutdown_connections 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.427+0000 7f1fc1695700 1 -- 172.21.15.162:0/3858910775 wait complete. 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.427+0000 7f1fc1695700 1 librados: starting msgr at 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.427+0000 7f1fc1695700 1 librados: starting objecter 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.427+0000 7f1fc1695700 1 Processor -- start 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fc1695700 1 -- start start 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fc1695700 1 librados: setting wanted keys 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fc1695700 1 librados: calling monclient init 2024-02-13T00:46:25.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fc1695700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fbc0f6880 0x7f1fbc10e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fc1695700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fbc0fde50 0x7f1fbc10e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fc1695700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fbc0fe7b0 0x7f1fbc1084c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fc1695700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc1089b0 con 0x7f1fbc0fe7b0 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fc1695700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc108b20 con 0x7f1fbc0f6880 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fc1695700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc108ce0 con 0x7f1fbc0fde50 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fbb7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fbc0fe7b0 0x7f1fbc1084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fba7fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fbc0fde50 0x7f1fbc10e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fbb7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fbc0fe7b0 0x7f1fbc1084c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60818/0 (socket says 172.21.15.162:60818) 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fba7fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fbc0fde50 0x7f1fbc10e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:40208/0 (socket says 172.21.15.162:40208) 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fbb7fe700 1 -- 172.21.15.162:0/3051397739 learned_addr learned my addr 172.21.15.162:0/3051397739 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fbaffd700 1 --2- 172.21.15.162:0/3051397739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fbc0f6880 0x7f1fbc10e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fba7fc700 1 -- 172.21.15.162:0/3051397739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fbc0f6880 msgr2=0x7f1fbc10e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:25.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fba7fc700 1 --2- 172.21.15.162:0/3051397739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fbc0f6880 0x7f1fbc10e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fba7fc700 1 -- 172.21.15.162:0/3051397739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fbc0fe7b0 msgr2=0x7f1fbc1084c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fba7fc700 1 --2- 172.21.15.162:0/3051397739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fbc0fe7b0 0x7f1fbc1084c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fba7fc700 1 -- 172.21.15.162:0/3051397739 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb40097c0 con 0x7f1fbc0fde50 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.428+0000 7f1fbb7fe700 1 --2- 172.21.15.162:0/3051397739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fbc0fe7b0 0x7f1fbc1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.429+0000 7f1fba7fc700 1 --2- 172.21.15.162:0/3051397739 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fbc0fde50 0x7f1fbc10e810 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 rx=0x7f1fb0004fa0 tx=0x7f1fb0009e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.429+0000 7f1fa3fff700 10 client.?.objecter ms_handle_connect 0x7f1fbc0fde50 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.429+0000 7f1fa3fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.429+0000 7f1fa3fff700 1 -- 172.21.15.162:0/3051397739 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0007b40 con 0x7f1fbc0fde50 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.429+0000 7f1fa3fff700 1 -- 172.21.15.162:0/3051397739 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1fb0016070 con 0x7f1fbc0fde50 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.429+0000 7f1fc1695700 1 -- 172.21.15.162:0/3051397739 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fac005040 con 0x7f1fbc0fde50 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.429+0000 7f1fa3fff700 1 -- 172.21.15.162:0/3051397739 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0012720 con 0x7f1fbc0fde50 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.429+0000 7f1fc1695700 10 client.37427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.429+0000 7f1fc1695700 1 -- 172.21.15.162:0/3051397739 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fbc0f72b0 con 0x7f1fbc0fde50 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.429+0000 7f1fc1695700 1 librados: init done 2024-02-13T00:46:25.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.431+0000 7f1fa3fff700 1 -- 172.21.15.162:0/3051397739 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1fb0007ca0 con 0x7f1fbc0fde50 2024-02-13T00:46:25.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.431+0000 7f1fa3fff700 1 --2- 172.21.15.162:0/3051397739 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fa4062c90 0x7f1fa4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:25.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.431+0000 7f1fa3fff700 1 -- 172.21.15.162:0/3051397739 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(420..420 src has 52..420) v4 ==== 8023+0+0 (secure 0 0 0) 0x7f1fb0080c70 con 0x7f1fbc0fde50 2024-02-13T00:46:25.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.431+0000 7f1fa3fff700 10 client.37427.objecter ms_dispatch 0x7f1fbc001550 osd_map(420..420 src has 52..420) v4 2024-02-13T00:46:25.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.431+0000 7f1fa3fff700 3 client.37427.objecter handle_osd_map got epochs [420,420] > 0 2024-02-13T00:46:25.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.431+0000 7f1fa3fff700 3 client.37427.objecter handle_osd_map decoding full epoch 420 2024-02-13T00:46:25.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.432+0000 7f1fa3fff700 20 client.37427.objecter dump_active .. 0 homeless 2024-02-13T00:46:25.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.432+0000 7f1fbaffd700 1 --2- 172.21.15.162:0/3051397739 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fa4062c90 0x7f1fa4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:25.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.432+0000 7f1fbaffd700 1 --2- 172.21.15.162:0/3051397739 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fa4062c90 0x7f1fa4065150 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7f1fac005010 tx=0x7f1fac007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:25.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.432+0000 7f1fa3fff700 10 client.37427.objecter ms_handle_connect 0x7f1fa4062c90 2024-02-13T00:46:25.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.435+0000 7f1fc1695700 1 -- 172.21.15.162:0/3051397739 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fbc109390 con 0x7f1fbc0fde50 2024-02-13T00:46:25.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.444+0000 7f1fa3fff700 1 -- 172.21.15.162:0/3051397739 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1fbc109390 con 0x7f1fbc0fde50 2024-02-13T00:46:25.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:25.632+0000 7f1fc1695700 1 -- 172.21.15.162:0/3051397739 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f1fbc109390 con 0x7f1fbc0fde50 2024-02-13T00:46:26.075 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:26.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:26.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:26.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:26.207 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:26.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:46:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:46:26] "GET /metrics HTTP/1.1" 200 187736 "" "Prometheus/2.33.4" 2024-02-13T00:46:26.375 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:46:26.375 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:46:26.375 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:46:26.375 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:46:26.375 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:46:26.400 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:26.405 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:26.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:26.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:26.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:26.543 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:26.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.540+0000 7f1fa3fff700 1 -- 172.21.15.162:0/3051397739 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v421) v1 ==== 101+0+0 (secure 0 0 0) 0x7f1fbc109390 con 0x7f1fbc0fde50 2024-02-13T00:46:26.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 10 librados: watch_flush enter 2024-02-13T00:46:26.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 10 librados: watch_flush exit 2024-02-13T00:46:26.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 20 client.37427.objecter shutdown clearing up homeless session... 2024-02-13T00:46:26.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 10 client.37427.objecter successfully canceled tick 2024-02-13T00:46:26.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 1 -- 172.21.15.162:0/3051397739 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fa4062c90 msgr2=0x7f1fa4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:26.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 1 --2- 172.21.15.162:0/3051397739 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fa4062c90 0x7f1fa4065150 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7f1fac005010 tx=0x7f1fac007f90).stop 2024-02-13T00:46:26.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 1 -- 172.21.15.162:0/3051397739 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fbc0fde50 msgr2=0x7f1fbc10e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:26.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 1 --2- 172.21.15.162:0/3051397739 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fbc0fde50 0x7f1fbc10e810 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 rx=0x7f1fb0004fa0 tx=0x7f1fb0009e40).stop 2024-02-13T00:46:26.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 1 -- 172.21.15.162:0/3051397739 shutdown_connections 2024-02-13T00:46:26.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 1 --2- 172.21.15.162:0/3051397739 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1fa4062c90 0x7f1fa4065150 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:26.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 1 --2- 172.21.15.162:0/3051397739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fbc0fe7b0 0x7f1fbc1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:26.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 1 --2- 172.21.15.162:0/3051397739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fbc0f6880 0x7f1fbc10e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:26.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 1 --2- 172.21.15.162:0/3051397739 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fbc0fde50 0x7f1fbc10e810 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:26.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 1 -- 172.21.15.162:0/3051397739 >> 172.21.15.162:0/3051397739 conn(0x7f1fbc070080 msgr2=0x7f1fbc06d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:26.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.543+0000 7f1fc1695700 1 -- 172.21.15.162:0/3051397739 shutdown_connections 2024-02-13T00:46:26.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.544+0000 7f1fc1695700 1 -- 172.21.15.162:0/3051397739 wait complete. 2024-02-13T00:46:26.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:26.544+0000 7f1fc1695700 1 librados: shutdown 2024-02-13T00:46:26.548 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:46:26.563 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:26.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:26.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:26.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:25.618494+0000 mgr.y (mgr.14544) 365 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:26.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:25.633686+0000 mon.c (mon.2) 379 : audit [INF] from='client.? 172.21.15.162:0/3051397739' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:26.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:25.634011+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:26.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:25.961352+0000 osd.7 (osd.7) 104 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:26.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:25.961579+0000 osd.7 (osd.7) 105 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:26.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:25.618494+0000 mgr.y (mgr.14544) 365 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:26.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:25.633686+0000 mon.c (mon.2) 379 : audit [INF] from='client.? 172.21.15.162:0/3051397739' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:26.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:25.634011+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:26.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:25.961352+0000 osd.7 (osd.7) 104 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:26.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:25.961579+0000 osd.7 (osd.7) 105 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:26.694 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:26.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:25.618494+0000 mgr.y (mgr.14544) 365 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:26.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:25.633686+0000 mon.c (mon.2) 379 : audit [INF] from='client.? 172.21.15.162:0/3051397739' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:26.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:25.634011+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:26.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:25.961352+0000 osd.7 (osd.7) 104 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:26.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:25.961579+0000 osd.7 (osd.7) 105 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:25.732835+0000 osd.3 (osd.3) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:25.732997+0000 osd.3 (osd.3) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:26.228917+0000 osd.6 (osd.6) 105 : cluster [DBG] 61.1a scrub starts 2024-02-13T00:46:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:26.235251+0000 osd.6 (osd.6) 106 : cluster [DBG] 61.1a scrub ok 2024-02-13T00:46:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:26.422304+0000 mgr.y (mgr.14544) 366 : cluster [DBG] pgmap v606: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:46:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:26.528074+0000 mon.a (mon.0) 2520 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:46:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:26.540667+0000 mon.a (mon.0) 2521 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:46:27.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:26.540722+0000 mon.a (mon.0) 2522 : cluster [DBG] osdmap e421: 8 total, 8 up, 4 in 2024-02-13T00:46:27.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:25.732835+0000 osd.3 (osd.3) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:25.732997+0000 osd.3 (osd.3) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:26.228917+0000 osd.6 (osd.6) 105 : cluster [DBG] 61.1a scrub starts 2024-02-13T00:46:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:26.235251+0000 osd.6 (osd.6) 106 : cluster [DBG] 61.1a scrub ok 2024-02-13T00:46:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:26.422304+0000 mgr.y (mgr.14544) 366 : cluster [DBG] pgmap v606: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:46:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:26.528074+0000 mon.a (mon.0) 2520 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:46:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:26.540667+0000 mon.a (mon.0) 2521 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:46:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:26.540722+0000 mon.a (mon.0) 2522 : cluster [DBG] osdmap e421: 8 total, 8 up, 4 in 2024-02-13T00:46:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:25.732835+0000 osd.3 (osd.3) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:25.732997+0000 osd.3 (osd.3) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:26.228917+0000 osd.6 (osd.6) 105 : cluster [DBG] 61.1a scrub starts 2024-02-13T00:46:27.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:26.235251+0000 osd.6 (osd.6) 106 : cluster [DBG] 61.1a scrub ok 2024-02-13T00:46:27.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:26.422304+0000 mgr.y (mgr.14544) 366 : cluster [DBG] pgmap v606: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:46:27.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:26.528074+0000 mon.a (mon.0) 2520 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:46:27.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:26.540667+0000 mon.a (mon.0) 2521 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:46:27.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:26.540722+0000 mon.a (mon.0) 2522 : cluster [DBG] osdmap e421: 8 total, 8 up, 4 in 2024-02-13T00:46:28.608 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:27.703470+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:46:28.608 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:27.706001+0000 osd.0 (osd.0) 149 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:46:28.608 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:27.703470+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:46:28.608 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:27.706001+0000 osd.0 (osd.0) 149 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:46:28.973 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:28.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:28.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:28.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-13T00:46:29.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:46:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:46:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:46:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:27.703470+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:46:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:27.706001+0000 osd.0 (osd.0) 149 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:46:29.078 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:29.110 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:29.132 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:29.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:29.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:29.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:29.267 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:29.287 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:29.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:29.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:29.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:29.414 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:29.418 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:29.438 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:29.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:29.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:29.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:29.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:28.114363+0000 osd.1 (osd.1) 173 : cluster [DBG] 61.1d scrub starts 2024-02-13T00:46:29.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:28.118112+0000 osd.1 (osd.1) 174 : cluster [DBG] 61.1d scrub ok 2024-02-13T00:46:29.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:28.422712+0000 mgr.y (mgr.14544) 367 : cluster [DBG] pgmap v608: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:46:29.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:28.114363+0000 osd.1 (osd.1) 173 : cluster [DBG] 61.1d scrub starts 2024-02-13T00:46:29.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:28.118112+0000 osd.1 (osd.1) 174 : cluster [DBG] 61.1d scrub ok 2024-02-13T00:46:29.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:28.422712+0000 mgr.y (mgr.14544) 367 : cluster [DBG] pgmap v608: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:46:29.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:28.114363+0000 osd.1 (osd.1) 173 : cluster [DBG] 61.1d scrub starts 2024-02-13T00:46:29.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:28.118112+0000 osd.1 (osd.1) 174 : cluster [DBG] 61.1d scrub ok 2024-02-13T00:46:29.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:28.422712+0000 mgr.y (mgr.14544) 367 : cluster [DBG] pgmap v608: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:46:30.383 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:46:31.504 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:30.092263+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.10 deep-scrub starts 2024-02-13T00:46:31.505 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:30.095302+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.10 deep-scrub ok 2024-02-13T00:46:31.505 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:30.203687+0000 mon.a (mon.0) 2523 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:31.505 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:30.424070+0000 mgr.y (mgr.14544) 368 : cluster [DBG] pgmap v609: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-13T00:46:31.505 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:30.786724+0000 mon.a (mon.0) 2524 : audit [DBG] from='client.? 172.21.15.162:0/1889542406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:31.505 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:30.092263+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.10 deep-scrub starts 2024-02-13T00:46:31.505 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:30.095302+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.10 deep-scrub ok 2024-02-13T00:46:31.505 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:30.203687+0000 mon.a (mon.0) 2523 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:31.505 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:30.424070+0000 mgr.y (mgr.14544) 368 : cluster [DBG] pgmap v609: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-13T00:46:31.505 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:30.786724+0000 mon.a (mon.0) 2524 : audit [DBG] from='client.? 172.21.15.162:0/1889542406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:31.506 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:31.518 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:31.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:31.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:31.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:30.092263+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.10 deep-scrub starts 2024-02-13T00:46:31.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:30.095302+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.10 deep-scrub ok 2024-02-13T00:46:31.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:30.203687+0000 mon.a (mon.0) 2523 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:31.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:30.424070+0000 mgr.y (mgr.14544) 368 : cluster [DBG] pgmap v609: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-13T00:46:31.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:30.786724+0000 mon.a (mon.0) 2524 : audit [DBG] from='client.? 172.21.15.162:0/1889542406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:31.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:46:31.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.782+0000 7f3563e6d700 1 -- 172.21.15.162:0/181041382 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f355c0fa440 msgr2=0x7f355c0f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.782+0000 7f3563e6d700 1 --2- 172.21.15.162:0/181041382 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f355c0fa440 0x7f355c0f7e20 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7f354c004870 tx=0x7f354c009cb0).stop 2024-02-13T00:46:31.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.782+0000 7f3563e6d700 1 -- 172.21.15.162:0/181041382 shutdown_connections 2024-02-13T00:46:31.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.782+0000 7f3563e6d700 1 --2- 172.21.15.162:0/181041382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f355c0f8500 0x7f355c104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:31.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.782+0000 7f3563e6d700 1 --2- 172.21.15.162:0/181041382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f355c0f9b70 0x7f355c0f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:46:31.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.782+0000 7f3563e6d700 1 --2- 172.21.15.162:0/181041382 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f355c0fa440 0x7f355c0f7e20 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:31.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.782+0000 7f3563e6d700 1 -- 172.21.15.162:0/181041382 >> 172.21.15.162:0/181041382 conn(0x7f355c0f38c0 msgr2=0x7f355c0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:31.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 -- 172.21.15.162:0/181041382 shutdown_connections 2024-02-13T00:46:31.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 -- 172.21.15.162:0/181041382 wait complete. 2024-02-13T00:46:31.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 librados: starting msgr at 2024-02-13T00:46:31.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 librados: starting objecter 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 Processor -- start 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 -- start start 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 librados: setting wanted keys 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 librados: calling monclient init 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f355c0f8500 0x7f355c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f355c0f9b70 0x7f355c06b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f355c0fa440 0x7f355c06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c116780 con 0x7f355c0fa440 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c1168f0 con 0x7f355c0f8500 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3563e6d700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c116ab0 con 0x7f355c0f9b70 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f3561408700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f355c0f9b70 0x7f355c06b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f356240a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f355c0fa440 0x7f355c06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:31.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f356240a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f355c0fa440 0x7f355c06bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60844/0 (socket says 172.21.15.162:60844) 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.783+0000 7f356240a700 1 -- 172.21.15.162:0/1889542406 learned_addr learned my addr 172.21.15.162:0/1889542406 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f3561c09700 1 --2- 172.21.15.162:0/1889542406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f355c0f8500 0x7f355c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f356240a700 1 -- 172.21.15.162:0/1889542406 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f355c0f9b70 msgr2=0x7f355c06b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f356240a700 1 --2- 172.21.15.162:0/1889542406 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f355c0f9b70 0x7f355c06b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f356240a700 1 -- 172.21.15.162:0/1889542406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f355c0f8500 msgr2=0x7f355c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f356240a700 1 --2- 172.21.15.162:0/1889542406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f355c0f8500 0x7f355c06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f356240a700 1 -- 172.21.15.162:0/1889542406 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35540097c0 con 0x7f355c0fa440 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f3561408700 1 --2- 172.21.15.162:0/1889542406 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f355c0f9b70 0x7f355c06b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f3561c09700 1 --2- 172.21.15.162:0/1889542406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f355c0f8500 0x7f355c06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f356240a700 1 --2- 172.21.15.162:0/1889542406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f355c0fa440 0x7f355c06bb50 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7f3554001df0 tx=0x7f3554004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f3552ffd700 10 client.?.objecter ms_handle_connect 0x7f355c0fa440 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f3552ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554009a10 con 0x7f355c0fa440 2024-02-13T00:46:31.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f354c005040 con 0x7f355c0fa440 2024-02-13T00:46:31.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35540095a0 con 0x7f355c0fa440 2024-02-13T00:46:31.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554007da0 con 0x7f355c0fa440 2024-02-13T00:46:31.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f3563e6d700 10 client.27333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:31.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c1170a0 con 0x7f355c0fa440 2024-02-13T00:46:31.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.784+0000 7f3563e6d700 1 librados: init done 2024-02-13T00:46:31.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.785+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f355c0475e0 con 0x7f355c0fa440 2024-02-13T00:46:31.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.785+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f35540090d0 con 0x7f355c0fa440 2024-02-13T00:46:31.628 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3552ffd700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3548062c90 0x7f3548065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(421..421 src has 52..421) v4 ==== 7995+0+0 (secure 0 0 0) 0x7f355407db30 con 0x7f355c0fa440 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 osd_map(421..421 src has 52..421) v4 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3552ffd700 3 client.27333.objecter handle_osd_map got epochs [421,421] > 0 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3552ffd700 3 client.27333.objecter handle_osd_map decoding full epoch 421 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3561c09700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3548062c90 0x7f3548065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3552ffd700 20 client.27333.objecter dump_active .. 0 homeless 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v421) v1 ==== 54+0+16 (secure 0 0 0) 0x7f3554082220 con 0x7f355c0fa440 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3552ffd700 10 client.27333.objecter ms_handle_connect 0x7f3548062c90 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3561c09700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3548062c90 0x7f3548065150 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 rx=0x7f3558009990 tx=0x7f3558008040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3563e6d700 10 client.27333.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 1 2024-02-13T00:46:31.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6801/2103743338 conn(0x7f355c028590 0x7f355c0feb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c107250 osd=0 3 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c107250 osd=0 3 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c107250 osd=0 3 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 1 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c107250 osd=0 3 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 0 1 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c107250 osd=0 4 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3563e6d700 10 client.27333.objecter _send_command 1 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f355c1001b0 con 0x7f355c028590 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3561408700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6801/2103743338 conn(0x7f355c028590 0x7f355c0feb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3552ffd700 10 client.27333.objecter ms_handle_connect 0x7f355c028590 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.786+0000 7f3561408700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6801/2103743338 conn(0x7f355c028590 0x7f355c0feb40 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.787+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f355c1001b0 con 0x7f355c028590 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.787+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:46:31.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.787+0000 7f3552ffd700 10 client.27333.objecter _finish_command 1 = system:0 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.787+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c107250 osd=0 4 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.787+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 0 1 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.802+0000 7f3563e6d700 10 client.27333.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.802+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 2 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.802+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c107250 osd=0 3 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.802+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c107250 osd=0 3 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.802+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c107250 osd=0 3 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.802+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 2 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.802+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c107250 osd=0 3 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.802+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 0 2 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.802+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c107250 osd=0 4 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.802+0000 7f3563e6d700 10 client.27333.objecter _send_command 2 2024-02-13T00:46:31.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.802+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f355c05a0e0 con 0x7f355c028590 2024-02-13T00:46:31.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.803+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f355c05a0e0 con 0x7f355c028590 2024-02-13T00:46:31.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.803+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:31.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.803+0000 7f3552ffd700 10 client.27333.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.803+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c107250 osd=0 4 2024-02-13T00:46:31.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.803+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 0 2 2024-02-13T00:46:31.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3563e6d700 10 client.27333.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:31.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 3 2024-02-13T00:46:31.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6805/150599780 conn(0x7f355c059570 0x7f355c11b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:31.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c059400 osd=1 3 2024-02-13T00:46:31.633 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:46:31.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c059400 osd=1 3 2024-02-13T00:46:31.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c059400 osd=1 3 2024-02-13T00:46:31.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 3 2024-02-13T00:46:31.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c059400 osd=1 3 2024-02-13T00:46:31.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 1 3 2024-02-13T00:46:31.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c059400 osd=1 4 2024-02-13T00:46:31.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3563e6d700 10 client.27333.objecter _send_command 3 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f355c0280f0 con 0x7f355c059570 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3561c09700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6805/150599780 conn(0x7f355c059570 0x7f355c11b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3561c09700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6805/150599780 conn(0x7f355c059570 0x7f355c11b310 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.804+0000 7f3552ffd700 10 client.27333.objecter ms_handle_connect 0x7f355c059570 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.805+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f355c0280f0 con 0x7f355c059570 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.805+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.805+0000 7f3552ffd700 10 client.27333.objecter _finish_command 3 = system:0 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.805+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c059400 osd=1 4 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.805+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 1 3 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.819+0000 7f3563e6d700 10 client.27333.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.819+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 4 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.819+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c059400 osd=1 3 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.819+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c059400 osd=1 3 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.819+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c059400 osd=1 3 2024-02-13T00:46:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.819+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 4 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.819+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c059400 osd=1 3 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.819+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 1 4 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.819+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c059400 osd=1 4 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.819+0000 7f3563e6d700 10 client.27333.objecter _send_command 4 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.819+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f355c11b800 con 0x7f355c059570 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f355c11b800 con 0x7f355c059570 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3552ffd700 10 client.27333.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c059400 osd=1 4 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 1 4 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3563e6d700 10 client.27333.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 5 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6809/3689719376 conn(0x7f355c0277c0 0x7f355c11eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:31.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c027650 osd=2 3 2024-02-13T00:46:31.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c027650 osd=2 3 2024-02-13T00:46:31.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c027650 osd=2 3 2024-02-13T00:46:31.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 5 2024-02-13T00:46:31.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c027650 osd=2 3 2024-02-13T00:46:31.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 2 5 2024-02-13T00:46:31.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c027650 osd=2 4 2024-02-13T00:46:31.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3563e6d700 10 client.27333.objecter _send_command 5 2024-02-13T00:46:31.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f355c0ffd10 con 0x7f355c0277c0 2024-02-13T00:46:31.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3561408700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6809/3689719376 conn(0x7f355c0277c0 0x7f355c11eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:31.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3561408700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6809/3689719376 conn(0x7f355c0277c0 0x7f355c11eb60 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:31.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.821+0000 7f3552ffd700 10 client.27333.objecter ms_handle_connect 0x7f355c0277c0 2024-02-13T00:46:31.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.822+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f355c0ffd10 con 0x7f355c0277c0 2024-02-13T00:46:31.640 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:46:31.640 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:46:31.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.822+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:46:31.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.822+0000 7f3552ffd700 10 client.27333.objecter _finish_command 5 = system:0 2024-02-13T00:46:31.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.822+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c027650 osd=2 4 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.822+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 2 5 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.836+0000 7f3563e6d700 10 client.27333.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.836+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 6 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.836+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c027650 osd=2 3 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.836+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c027650 osd=2 3 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.836+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c027650 osd=2 3 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.836+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 6 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.836+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c027650 osd=2 3 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.836+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 2 6 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.836+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c027650 osd=2 4 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.836+0000 7f3563e6d700 10 client.27333.objecter _send_command 6 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.836+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f355c11f050 con 0x7f355c0277c0 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f355c11f050 con 0x7f355c0277c0 2024-02-13T00:46:31.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:31.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3552ffd700 10 client.27333.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c027650 osd=2 4 2024-02-13T00:46:31.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 2 6 2024-02-13T00:46:31.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3563e6d700 10 client.27333.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:31.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 7 2024-02-13T00:46:31.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6813/604971011 conn(0x7f355c0ff3e0 0x7f355c122350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:31.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c0ff270 osd=3 3 2024-02-13T00:46:31.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c0ff270 osd=3 3 2024-02-13T00:46:31.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c0ff270 osd=3 3 2024-02-13T00:46:31.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 7 2024-02-13T00:46:31.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c0ff270 osd=3 3 2024-02-13T00:46:31.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 3 7 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c0ff270 osd=3 4 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3563e6d700 10 client.27333.objecter _send_command 7 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f355c11c650 con 0x7f355c0ff3e0 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f356240a700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6813/604971011 conn(0x7f355c0ff3e0 0x7f355c122350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f3552ffd700 10 client.27333.objecter ms_handle_connect 0x7f355c0ff3e0 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.838+0000 7f356240a700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6813/604971011 conn(0x7f355c0ff3e0 0x7f355c122350 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.839+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f355c11c650 con 0x7f355c0ff3e0 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.839+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.839+0000 7f3552ffd700 10 client.27333.objecter _finish_command 7 = system:0 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.839+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c0ff270 osd=3 4 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.839+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 3 7 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.853+0000 7f3563e6d700 10 client.27333.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.853+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 8 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.853+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c0ff270 osd=3 3 2024-02-13T00:46:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.853+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c0ff270 osd=3 3 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.853+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c0ff270 osd=3 3 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.853+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 8 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.853+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c0ff270 osd=3 3 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.853+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 3 8 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.853+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c0ff270 osd=3 4 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.853+0000 7f3563e6d700 10 client.27333.objecter _send_command 8 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.853+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f355c122840 con 0x7f355c0ff3e0 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f355c122840 con 0x7f355c0ff3e0 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3552ffd700 10 client.27333.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c0ff270 osd=3 4 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 3 8 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3563e6d700 10 client.27333.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:31.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 9 2024-02-13T00:46:31.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6800/3344263388 conn(0x7f355c122f20 0x7f355c125310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:31.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c122840 osd=4 3 2024-02-13T00:46:31.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c122840 osd=4 3 2024-02-13T00:46:31.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c122840 osd=4 3 2024-02-13T00:46:31.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 9 2024-02-13T00:46:31.645 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:46:31.645 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c122840 osd=4 3 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 4 9 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c122840 osd=4 4 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3563e6d700 10 client.27333.objecter _send_command 9 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f355c1280f0 con 0x7f355c122f20 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3561c09700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6800/3344263388 conn(0x7f355c122f20 0x7f355c125310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3561c09700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6800/3344263388 conn(0x7f355c122f20 0x7f355c125310 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.855+0000 7f3552ffd700 10 client.27333.objecter ms_handle_connect 0x7f355c122f20 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.856+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f355c1280f0 con 0x7f355c122f20 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.856+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.856+0000 7f3552ffd700 10 client.27333.objecter _finish_command 9 = system:0 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.856+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c122840 osd=4 4 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.856+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 4 9 2024-02-13T00:46:31.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.871+0000 7f3563e6d700 10 client.27333.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.871+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 10 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.871+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c122840 osd=4 3 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.871+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c122840 osd=4 3 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.871+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c122840 osd=4 3 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.871+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 10 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.871+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c122840 osd=4 3 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.871+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 4 10 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.871+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c122840 osd=4 4 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.871+0000 7f3563e6d700 10 client.27333.objecter _send_command 10 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.871+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f355c1280f0 con 0x7f355c122f20 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f355c1280f0 con 0x7f355c122f20 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3552ffd700 10 client.27333.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c122840 osd=4 4 2024-02-13T00:46:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 4 10 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3563e6d700 10 client.27333.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 11 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6804/2212296748 conn(0x7f355c1273f0 0x7f355c127870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c1280f0 osd=5 3 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c1280f0 osd=5 3 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c1280f0 osd=5 3 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 11 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c1280f0 osd=5 3 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 5 11 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c1280f0 osd=5 4 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3563e6d700 10 client.27333.objecter _send_command 11 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f355c127ee0 con 0x7f355c1273f0 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.873+0000 7f3561408700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6804/2212296748 conn(0x7f355c1273f0 0x7f355c127870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:31.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.874+0000 7f3552ffd700 10 client.27333.objecter ms_handle_connect 0x7f355c1273f0 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.874+0000 7f3561408700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6804/2212296748 conn(0x7f355c1273f0 0x7f355c127870 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.874+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f355c127ee0 con 0x7f355c1273f0 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.874+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.874+0000 7f3552ffd700 10 client.27333.objecter _finish_command 11 = system:0 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.874+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c1280f0 osd=5 4 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.874+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 5 11 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.889+0000 7f3563e6d700 10 client.27333.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.889+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 12 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.889+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c1280f0 osd=5 3 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.889+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c1280f0 osd=5 3 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.889+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c1280f0 osd=5 3 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.889+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 12 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.889+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c1280f0 osd=5 3 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.889+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 5 12 2024-02-13T00:46:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.889+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c1280f0 osd=5 4 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.889+0000 7f3563e6d700 10 client.27333.objecter _send_command 12 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.889+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f355c127ee0 con 0x7f355c1273f0 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f355c127ee0 con 0x7f355c1273f0 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3552ffd700 10 client.27333.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c1280f0 osd=5 4 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 5 12 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3563e6d700 10 client.27333.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 13 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6808/1506640341 conn(0x7f355c12a8f0 0x7f355c12ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c127ee0 osd=6 3 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c127ee0 osd=6 3 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c127ee0 osd=6 3 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 13 2024-02-13T00:46:31.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c127ee0 osd=6 3 2024-02-13T00:46:31.651 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:46:31.651 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:46:31.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 6 13 2024-02-13T00:46:31.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c127ee0 osd=6 4 2024-02-13T00:46:31.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3563e6d700 10 client.27333.objecter _send_command 13 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.891+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f355c12e6e0 con 0x7f355c12a8f0 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.892+0000 7f356240a700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6808/1506640341 conn(0x7f355c12a8f0 0x7f355c12ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.892+0000 7f356240a700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6808/1506640341 conn(0x7f355c12a8f0 0x7f355c12ad50 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.892+0000 7f3552ffd700 10 client.27333.objecter ms_handle_connect 0x7f355c12a8f0 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.893+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f355c12e6e0 con 0x7f355c12a8f0 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.893+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.893+0000 7f3552ffd700 10 client.27333.objecter _finish_command 13 = system:0 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.893+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c127ee0 osd=6 4 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.893+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 6 13 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.907+0000 7f3563e6d700 10 client.27333.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.907+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 14 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.907+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c127ee0 osd=6 3 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.907+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c127ee0 osd=6 3 2024-02-13T00:46:31.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.907+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c127ee0 osd=6 3 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.907+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 14 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.907+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c127ee0 osd=6 3 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.907+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 6 14 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.907+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c127ee0 osd=6 4 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.907+0000 7f3563e6d700 10 client.27333.objecter _send_command 14 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.907+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f355c1258d0 con 0x7f355c12a8f0 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f355c1258d0 con 0x7f355c12a8f0 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3552ffd700 10 client.27333.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c127ee0 osd=6 4 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 6 14 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3563e6d700 10 client.27333.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 15 2024-02-13T00:46:31.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6812/1964192772 conn(0x7f355c125fb0 0x7f355c1263f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c1258d0 osd=7 3 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c1258d0 osd=7 3 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c1258d0 osd=7 3 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 15 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c1258d0 osd=7 3 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 7 15 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c1258d0 osd=7 4 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3563e6d700 10 client.27333.objecter _send_command 15 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f355c126a30 con 0x7f355c125fb0 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.909+0000 7f3561c09700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6812/1964192772 conn(0x7f355c125fb0 0x7f355c1263f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.910+0000 7f3552ffd700 10 client.27333.objecter ms_handle_connect 0x7f355c125fb0 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.910+0000 7f3561c09700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6812/1964192772 conn(0x7f355c125fb0 0x7f355c1263f0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.910+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f355c126a30 con 0x7f355c125fb0 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.910+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:46:31.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.910+0000 7f3552ffd700 10 client.27333.objecter _finish_command 15 = system:0 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.910+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c1258d0 osd=7 4 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.910+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 7 15 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.924+0000 7f3563e6d700 10 client.27333.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.924+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign -1 16 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.924+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c1258d0 osd=7 3 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.924+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c1258d0 osd=7 3 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.924+0000 7f3563e6d700 20 client.27333.objecter _get_session s=0x7f355c1258d0 osd=7 3 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.924+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_remove -1 16 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.924+0000 7f3563e6d700 20 client.27333.objecter get_session s=0x7f355c1258d0 osd=7 3 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.924+0000 7f3563e6d700 15 client.27333.objecter _session_command_op_assign 7 16 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.924+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c1258d0 osd=7 4 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.924+0000 7f3563e6d700 10 client.27333.objecter _send_command 16 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.924+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f355c126a30 con 0x7f355c125fb0 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.926+0000 7f3552ffd700 1 -- 172.21.15.162:0/1889542406 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f355c126a30 con 0x7f355c125fb0 2024-02-13T00:46:31.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.926+0000 7f3552ffd700 10 client.27333.objecter ms_dispatch 0x7f355c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.926+0000 7f3552ffd700 10 client.27333.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.926+0000 7f3552ffd700 20 client.27333.objecter put_session s=0x7f355c1258d0 osd=7 4 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.926+0000 7f3552ffd700 15 client.27333.objecter _session_command_op_remove 7 16 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 10 librados: watch_flush enter 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 10 librados: watch_flush exit 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 10 client.27333.objecter close_session for osd.0 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6801/2103743338 conn(0x7f355c028590 msgr2=0x7f355c0feb40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6801/2103743338 conn(0x7f355c028590 0x7f355c0feb40 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c107250 osd=0 1 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 10 client.27333.objecter close_session for osd.1 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6805/150599780 conn(0x7f355c059570 msgr2=0x7f355c11b310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6805/150599780 conn(0x7f355c059570 0x7f355c11b310 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c059400 osd=1 1 2024-02-13T00:46:31.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 10 client.27333.objecter close_session for osd.2 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6809/3689719376 conn(0x7f355c0277c0 msgr2=0x7f355c11eb60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6809/3689719376 conn(0x7f355c0277c0 0x7f355c11eb60 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c027650 osd=2 1 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 10 client.27333.objecter close_session for osd.3 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6813/604971011 conn(0x7f355c0ff3e0 msgr2=0x7f355c122350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6813/604971011 conn(0x7f355c0ff3e0 0x7f355c122350 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f356240a700 1 -- 172.21.15.162:0/1889542406 reap_dead start 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c0ff270 osd=3 1 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 10 client.27333.objecter close_session for osd.4 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6800/3344263388 conn(0x7f355c122f20 msgr2=0x7f355c125310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6800/3344263388 conn(0x7f355c122f20 0x7f355c125310 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f356240a700 1 -- 172.21.15.162:0/1889542406 reap_dead start 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c122840 osd=4 1 2024-02-13T00:46:31.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 10 client.27333.objecter close_session for osd.5 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6804/2212296748 conn(0x7f355c1273f0 msgr2=0x7f355c127870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6804/2212296748 conn(0x7f355c1273f0 0x7f355c127870 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c1280f0 osd=5 1 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 10 client.27333.objecter close_session for osd.6 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6808/1506640341 conn(0x7f355c12a8f0 msgr2=0x7f355c12ad50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6808/1506640341 conn(0x7f355c12a8f0 0x7f355c12ad50 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c127ee0 osd=6 1 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 10 client.27333.objecter close_session for osd.7 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6812/1964192772 conn(0x7f355c125fb0 msgr2=0x7f355c1263f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.181:6812/1964192772 conn(0x7f355c125fb0 0x7f355c1263f0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 20 client.27333.objecter put_session s=0x7f355c1258d0 osd=7 1 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 20 client.27333.objecter shutdown clearing up homeless session... 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 10 client.27333.objecter successfully canceled tick 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3548062c90 msgr2=0x7f3548065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3548062c90 0x7f3548065150 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 rx=0x7f3558009990 tx=0x7f3558008040).stop 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f355c0fa440 msgr2=0x7f355c06bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 --2- 172.21.15.162:0/1889542406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f355c0fa440 0x7f355c06bb50 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7f3554001df0 tx=0x7f3554004c50).stop 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f356240a700 1 -- 172.21.15.162:0/1889542406 reap_dead start 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 shutdown_connections 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 >> 172.21.15.162:0/1889542406 conn(0x7f355c0f38c0 msgr2=0x7f355c0f4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 shutdown_connections 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 -- 172.21.15.162:0/1889542406 wait complete. 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:30.927+0000 7f3563e6d700 1 librados: shutdown 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.659 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.660 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:46:31.691 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:31.712 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:46:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:31.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:46:31.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:31.845 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:31.865 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:31.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:31.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:31.999 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:32.248 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:32.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:32.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:32.379 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:32.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:30.638445+0000 osd.0 (osd.0) 150 : cluster [DBG] 2.c scrub starts 2024-02-13T00:46:32.473 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:30.641456+0000 osd.0 (osd.0) 151 : cluster [DBG] 2.c scrub ok 2024-02-13T00:46:32.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:30.638445+0000 osd.0 (osd.0) 150 : cluster [DBG] 2.c scrub starts 2024-02-13T00:46:32.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:30.641456+0000 osd.0 (osd.0) 151 : cluster [DBG] 2.c scrub ok 2024-02-13T00:46:32.478 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:32.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:32.583 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:32.611 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:32.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:30.638445+0000 osd.0 (osd.0) 150 : cluster [DBG] 2.c scrub starts 2024-02-13T00:46:32.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:30.641456+0000 osd.0 (osd.0) 151 : cluster [DBG] 2.c scrub ok 2024-02-13T00:46:33.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:32.294853+0000 osd.6 (osd.6) 107 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:46:33.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:32.297910+0000 osd.6 (osd.6) 108 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:46:33.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:32.424468+0000 mgr.y (mgr.14544) 369 : cluster [DBG] pgmap v610: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:46:33.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:32.294853+0000 osd.6 (osd.6) 107 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:46:33.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:32.297910+0000 osd.6 (osd.6) 108 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:46:33.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:32.424468+0000 mgr.y (mgr.14544) 369 : cluster [DBG] pgmap v610: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:46:33.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:32.294853+0000 osd.6 (osd.6) 107 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:46:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:32.297910+0000 osd.6 (osd.6) 108 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:46:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:32.424468+0000 mgr.y (mgr.14544) 369 : cluster [DBG] pgmap v610: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:46:33.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:33.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:33.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.852+0000 7fdb1ae21700 1 -- 172.21.15.162:0/1823808006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb140fe590 msgr2=0x7fdb140f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:33.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.852+0000 7fdb1ae21700 1 --2- 172.21.15.162:0/1823808006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb140fe590 0x7fdb140f7dc0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7fdb00004870 tx=0x7fdb00009cb0).stop 2024-02-13T00:46:33.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.852+0000 7fdb1ae21700 1 -- 172.21.15.162:0/1823808006 shutdown_connections 2024-02-13T00:46:33.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.852+0000 7fdb1ae21700 1 --2- 172.21.15.162:0/1823808006 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb140f84a0 0x7fdb14104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:33.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.852+0000 7fdb1ae21700 1 --2- 172.21.15.162:0/1823808006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb140fdc30 0x7fdb140fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:46:33.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.852+0000 7fdb1ae21700 1 --2- 172.21.15.162:0/1823808006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb140fe590 0x7fdb140f7dc0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:33.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.852+0000 7fdb1ae21700 1 -- 172.21.15.162:0/1823808006 >> 172.21.15.162:0/1823808006 conn(0x7fdb140f3880 msgr2=0x7fdb140f5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:33.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.852+0000 7fdb1ae21700 1 -- 172.21.15.162:0/1823808006 shutdown_connections 2024-02-13T00:46:33.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.852+0000 7fdb1ae21700 1 -- 172.21.15.162:0/1823808006 wait complete. 2024-02-13T00:46:33.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb1ae21700 1 librados: starting msgr at 2024-02-13T00:46:33.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb1ae21700 1 librados: starting objecter 2024-02-13T00:46:33.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb1ae21700 1 Processor -- start 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb1ae21700 1 -- start start 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb1ae21700 1 librados: setting wanted keys 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb1ae21700 1 librados: calling monclient init 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb1ae21700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb140f84a0 0x7fdb1406b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb1ae21700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb140fdc30 0x7fdb1406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb1ae21700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb140fe590 0x7fdb1406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb1ae21700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb14116650 con 0x7fdb140fe590 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb1ae21700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb141167c0 con 0x7fdb140f84a0 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb1ae21700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb14116980 con 0x7fdb140fdc30 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb193be700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb140fe590 0x7fdb1406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb13fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb140fdc30 0x7fdb1406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb193be700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb140fe590 0x7fdb1406bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60864/0 (socket says 172.21.15.162:60864) 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb13fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb140fdc30 0x7fdb1406b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:40244/0 (socket says 172.21.15.162:40244) 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb193be700 1 -- 172.21.15.162:0/943728602 learned_addr learned my addr 172.21.15.162:0/943728602 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:33.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb18bbd700 1 --2- 172.21.15.162:0/943728602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb140f84a0 0x7fdb1406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb193be700 1 -- 172.21.15.162:0/943728602 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb140fdc30 msgr2=0x7fdb1406b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb193be700 1 --2- 172.21.15.162:0/943728602 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb140fdc30 0x7fdb1406b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb193be700 1 -- 172.21.15.162:0/943728602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb140f84a0 msgr2=0x7fdb1406b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb193be700 1 --2- 172.21.15.162:0/943728602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb140f84a0 0x7fdb1406b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb193be700 1 -- 172.21.15.162:0/943728602 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb00005040 con 0x7fdb140fe590 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb193be700 1 --2- 172.21.15.162:0/943728602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb140fe590 0x7fdb1406bb50 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 rx=0x7fdb0c00ec20 tx=0x7fdb0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb11ffb700 10 client.?.objecter ms_handle_connect 0x7fdb140fe590 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.853+0000 7fdb11ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.854+0000 7fdb11ffb700 1 -- 172.21.15.162:0/943728602 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c009950 con 0x7fdb140fe590 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.854+0000 7fdb11ffb700 1 -- 172.21.15.162:0/943728602 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb0c00f040 con 0x7fdb140fe590 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.854+0000 7fdb1ae21700 1 -- 172.21.15.162:0/943728602 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb14116c00 con 0x7fdb140fe590 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.854+0000 7fdb11ffb700 1 -- 172.21.15.162:0/943728602 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c007df0 con 0x7fdb140fe590 2024-02-13T00:46:33.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.854+0000 7fdb1ae21700 10 client.27339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:33.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.854+0000 7fdb1ae21700 1 -- 172.21.15.162:0/943728602 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb14117140 con 0x7fdb140fe590 2024-02-13T00:46:33.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.854+0000 7fdb1ae21700 1 librados: init done 2024-02-13T00:46:33.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.855+0000 7fdb1ae21700 1 -- 172.21.15.162:0/943728602 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb1405fae0 con 0x7fdb140fe590 2024-02-13T00:46:33.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.855+0000 7fdb11ffb700 1 -- 172.21.15.162:0/943728602 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fdb0c0092e0 con 0x7fdb140fe590 2024-02-13T00:46:33.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.855+0000 7fdb11ffb700 1 --2- 172.21.15.162:0/943728602 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdb04062c90 0x7fdb04065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:33.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.855+0000 7fdb11ffb700 1 -- 172.21.15.162:0/943728602 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(421..421 src has 52..421) v4 ==== 7995+0+0 (secure 0 0 0) 0x7fdb0c07cd80 con 0x7fdb140fe590 2024-02-13T00:46:33.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.855+0000 7fdb11ffb700 10 client.27339.objecter ms_dispatch 0x7fdb14001550 osd_map(421..421 src has 52..421) v4 2024-02-13T00:46:33.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.855+0000 7fdb11ffb700 3 client.27339.objecter handle_osd_map got epochs [421,421] > 0 2024-02-13T00:46:33.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.855+0000 7fdb11ffb700 3 client.27339.objecter handle_osd_map decoding full epoch 421 2024-02-13T00:46:33.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.855+0000 7fdb18bbd700 1 --2- 172.21.15.162:0/943728602 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdb04062c90 0x7fdb04065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:33.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.856+0000 7fdb11ffb700 20 client.27339.objecter dump_active .. 0 homeless 2024-02-13T00:46:33.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.856+0000 7fdb11ffb700 10 client.27339.objecter ms_handle_connect 0x7fdb04062c90 2024-02-13T00:46:33.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.856+0000 7fdb18bbd700 1 --2- 172.21.15.162:0/943728602 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdb04062c90 0x7fdb04065150 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 rx=0x7fdb08006fd0 tx=0x7fdb08009060).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:33.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:33.860+0000 7fdb11ffb700 1 -- 172.21.15.162:0/943728602 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fdb0c049f60 con 0x7fdb140fe590 2024-02-13T00:46:33.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:33.976 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:33.997 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:34.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:34.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:34.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.038+0000 7fdb1ae21700 1 -- 172.21.15.162:0/943728602 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fdb141071b0 con 0x7fdb140fe590 2024-02-13T00:46:34.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:46:34.130 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:34.150 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:46:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:34.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:46:34.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:46:34.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:34.301 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:34.360 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:34.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:34.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:34.380 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:34.390 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:46:34.390 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:46:34.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.432+0000 7fdb11ffb700 1 -- 172.21.15.162:0/943728602 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v422) v1 ==== 113+0+0 (secure 0 0 0) 0x7fdb0c07e030 con 0x7fdb140fe590 2024-02-13T00:46:34.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.438+0000 7fdb1ae21700 10 librados: watch_flush enter 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.438+0000 7fdb1ae21700 10 librados: watch_flush exit 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.438+0000 7fdb1ae21700 20 client.27339.objecter shutdown clearing up homeless session... 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.438+0000 7fdb1ae21700 10 client.27339.objecter successfully canceled tick 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.438+0000 7fdb1ae21700 1 -- 172.21.15.162:0/943728602 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdb04062c90 msgr2=0x7fdb04065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.438+0000 7fdb1ae21700 1 --2- 172.21.15.162:0/943728602 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdb04062c90 0x7fdb04065150 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 rx=0x7fdb08006fd0 tx=0x7fdb08009060).stop 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.438+0000 7fdb1ae21700 1 -- 172.21.15.162:0/943728602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb140fe590 msgr2=0x7fdb1406bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.438+0000 7fdb1ae21700 1 --2- 172.21.15.162:0/943728602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb140fe590 0x7fdb1406bb50 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 rx=0x7fdb0c00ec20 tx=0x7fdb0c00b040).stop 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.439+0000 7fdb1ae21700 1 -- 172.21.15.162:0/943728602 shutdown_connections 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.439+0000 7fdb1ae21700 1 --2- 172.21.15.162:0/943728602 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdb04062c90 0x7fdb04065150 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.439+0000 7fdb1ae21700 1 --2- 172.21.15.162:0/943728602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb140fe590 0x7fdb1406bb50 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.439+0000 7fdb1ae21700 1 --2- 172.21.15.162:0/943728602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb140f84a0 0x7fdb1406b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.439+0000 7fdb1ae21700 1 --2- 172.21.15.162:0/943728602 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb140fdc30 0x7fdb1406b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.439+0000 7fdb1ae21700 1 -- 172.21.15.162:0/943728602 >> 172.21.15.162:0/943728602 conn(0x7fdb140f3880 msgr2=0x7fdb140fba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:34.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.439+0000 7fdb1ae21700 1 -- 172.21.15.162:0/943728602 shutdown_connections 2024-02-13T00:46:34.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.439+0000 7fdb1ae21700 1 -- 172.21.15.162:0/943728602 wait complete. 2024-02-13T00:46:34.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:34.439+0000 7fdb1ae21700 1 librados: shutdown 2024-02-13T00:46:34.444 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.4 to 1 (10000) 2024-02-13T00:46:34.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:34.496 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:34.689 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:32.635209+0000 osd.0 (osd.0) 152 : cluster [DBG] 5.13 deep-scrub starts 2024-02-13T00:46:34.689 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:32.638343+0000 osd.0 (osd.0) 153 : cluster [DBG] 5.13 deep-scrub ok 2024-02-13T00:46:34.689 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:33.283065+0000 osd.6 (osd.6) 109 : cluster [DBG] 61.1e scrub starts 2024-02-13T00:46:34.689 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:33.286808+0000 osd.6 (osd.6) 110 : cluster [DBG] 61.1e scrub ok 2024-02-13T00:46:34.690 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:34.039408+0000 mon.a (mon.0) 2525 : audit [INF] from='client.? 172.21.15.162:0/943728602' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:46:34.690 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:34.337253+0000 mon.a (mon.0) 2526 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:34.690 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:32.635209+0000 osd.0 (osd.0) 152 : cluster [DBG] 5.13 deep-scrub starts 2024-02-13T00:46:34.690 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:32.638343+0000 osd.0 (osd.0) 153 : cluster [DBG] 5.13 deep-scrub ok 2024-02-13T00:46:34.690 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:33.283065+0000 osd.6 (osd.6) 109 : cluster [DBG] 61.1e scrub starts 2024-02-13T00:46:34.690 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:33.286808+0000 osd.6 (osd.6) 110 : cluster [DBG] 61.1e scrub ok 2024-02-13T00:46:34.690 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:34.039408+0000 mon.a (mon.0) 2525 : audit [INF] from='client.? 172.21.15.162:0/943728602' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:46:34.690 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:34.337253+0000 mon.a (mon.0) 2526 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:32.635209+0000 osd.0 (osd.0) 152 : cluster [DBG] 5.13 deep-scrub starts 2024-02-13T00:46:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:32.638343+0000 osd.0 (osd.0) 153 : cluster [DBG] 5.13 deep-scrub ok 2024-02-13T00:46:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:33.283065+0000 osd.6 (osd.6) 109 : cluster [DBG] 61.1e scrub starts 2024-02-13T00:46:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:33.286808+0000 osd.6 (osd.6) 110 : cluster [DBG] 61.1e scrub ok 2024-02-13T00:46:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:34.039408+0000 mon.a (mon.0) 2525 : audit [INF] from='client.? 172.21.15.162:0/943728602' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:46:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:34.337253+0000 mon.a (mon.0) 2526 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:35.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:34.139347+0000 osd.1 (osd.1) 177 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:46:35.720 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:34.140922+0000 osd.1 (osd.1) 178 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:46:35.720 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:34.424870+0000 mgr.y (mgr.14544) 370 : cluster [DBG] pgmap v611: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 568 B/s rd, 0 op/s 2024-02-13T00:46:35.720 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:34.432440+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? 172.21.15.162:0/943728602' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-13T00:46:35.720 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:34.432501+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e422: 8 total, 8 up, 4 in 2024-02-13T00:46:35.720 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:35.416810+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e423: 8 total, 8 up, 4 in 2024-02-13T00:46:35.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:34.139347+0000 osd.1 (osd.1) 177 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:46:35.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:34.140922+0000 osd.1 (osd.1) 178 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:46:35.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:34.424870+0000 mgr.y (mgr.14544) 370 : cluster [DBG] pgmap v611: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 568 B/s rd, 0 op/s 2024-02-13T00:46:35.721 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:34.432440+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? 172.21.15.162:0/943728602' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-13T00:46:35.721 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:34.432501+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e422: 8 total, 8 up, 4 in 2024-02-13T00:46:35.721 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:35.416810+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e423: 8 total, 8 up, 4 in 2024-02-13T00:46:35.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-13T00:46:35.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:46:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:46:35.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:34.139347+0000 osd.1 (osd.1) 177 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:46:35.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:34.140922+0000 osd.1 (osd.1) 178 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:46:35.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:34.424870+0000 mgr.y (mgr.14544) 370 : cluster [DBG] pgmap v611: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 568 B/s rd, 0 op/s 2024-02-13T00:46:35.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:34.432440+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? 172.21.15.162:0/943728602' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-13T00:46:35.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:34.432501+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e422: 8 total, 8 up, 4 in 2024-02-13T00:46:35.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:35.416810+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e423: 8 total, 8 up, 4 in 2024-02-13T00:46:36.012 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:46:36.056 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:36.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:36.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:36.186 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:36.285 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:36.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:36.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:36.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:46:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:46:36] "GET /metrics HTTP/1.1" 200 187716 "" "Prometheus/2.33.4" 2024-02-13T00:46:36.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:36.416 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:36.513 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:36.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:36.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:36.652 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:36.673 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:36.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:36.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:36.779 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:36.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:34.687210+0000 osd.0 (osd.0) 154 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:46:36.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:34.690970+0000 osd.0 (osd.0) 155 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:46:36.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:35.621508+0000 mgr.y (mgr.14544) 371 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:36.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:34.687210+0000 osd.0 (osd.0) 154 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:46:36.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:34.690970+0000 osd.0 (osd.0) 155 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:46:36.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:35.621508+0000 mgr.y (mgr.14544) 371 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:36.805 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:34.687210+0000 osd.0 (osd.0) 154 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:46:36.805 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:34.690970+0000 osd.0 (osd.0) 155 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:46:36.805 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:35.621508+0000 mgr.y (mgr.14544) 371 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:36.807 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:37.006 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:37.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:37.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:37.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 6 1 2024-02-13T00:46:37.040 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:37.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:37.141 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:37.340 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:37.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:37.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:37.444 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:37.471 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:37.497 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:37.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:37.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.588+0000 7f15c01a6700 1 -- 172.21.15.162:0/1697884257 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15b80fa440 msgr2=0x7f15b80f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:37.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.588+0000 7f15c01a6700 1 --2- 172.21.15.162:0/1697884257 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15b80fa440 0x7f15b80f7e20 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 rx=0x7f15b4004870 tx=0x7f15b4009cb0).stop 2024-02-13T00:46:37.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.588+0000 7f15c01a6700 1 -- 172.21.15.162:0/1697884257 shutdown_connections 2024-02-13T00:46:37.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.588+0000 7f15c01a6700 1 --2- 172.21.15.162:0/1697884257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b80f8500 0x7f15b8104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:37.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.588+0000 7f15c01a6700 1 --2- 172.21.15.162:0/1697884257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b80f9b70 0x7f15b80f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.588+0000 7f15c01a6700 1 --2- 172.21.15.162:0/1697884257 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15b80fa440 0x7f15b80f7e20 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.588+0000 7f15c01a6700 1 -- 172.21.15.162:0/1697884257 >> 172.21.15.162:0/1697884257 conn(0x7f15b80f38c0 msgr2=0x7f15b80f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.588+0000 7f15c01a6700 1 -- 172.21.15.162:0/1697884257 shutdown_connections 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.588+0000 7f15c01a6700 1 -- 172.21.15.162:0/1697884257 wait complete. 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15c01a6700 1 librados: starting msgr at 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15c01a6700 1 librados: starting objecter 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15c01a6700 1 Processor -- start 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15c01a6700 1 -- start start 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15c01a6700 1 librados: setting wanted keys 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15c01a6700 1 librados: calling monclient init 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15c01a6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15b80f8500 0x7f15b810e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15c01a6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b80f9b70 0x7f15b810e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15c01a6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b80fa440 0x7f15b8108490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:37.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15c01a6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8108980 con 0x7f15b80fa440 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15c01a6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8108af0 con 0x7f15b80f9b70 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15c01a6700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8108cb0 con 0x7f15b80f8500 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15bdf42700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15b80f8500 0x7f15b810e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15bdf42700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15b80f8500 0x7f15b810e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:46322/0 (socket says 172.21.15.162:46322) 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15bdf42700 1 -- 172.21.15.162:0/1034840363 learned_addr learned my addr 172.21.15.162:0/1034840363 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15be743700 1 --2- 172.21.15.162:0/1034840363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b80fa440 0x7f15b8108490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.589+0000 7f15bd741700 1 --2- 172.21.15.162:0/1034840363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b80f9b70 0x7f15b810e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15bdf42700 1 -- 172.21.15.162:0/1034840363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b80f9b70 msgr2=0x7f15b810e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15bdf42700 1 --2- 172.21.15.162:0/1034840363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b80f9b70 0x7f15b810e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15bdf42700 1 -- 172.21.15.162:0/1034840363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b80fa440 msgr2=0x7f15b8108490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15bdf42700 1 --2- 172.21.15.162:0/1034840363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b80fa440 0x7f15b8108490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15bdf42700 1 -- 172.21.15.162:0/1034840363 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b00097c0 con 0x7f15b80f8500 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15be743700 1 --2- 172.21.15.162:0/1034840363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b80fa440 0x7f15b8108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:37.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15bdf42700 1 --2- 172.21.15.162:0/1034840363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15b80f8500 0x7f15b810e2a0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7f15a800e990 tx=0x7f15a800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15aeffd700 10 client.?.objecter ms_handle_connect 0x7f15b80f8500 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15aeffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15aeffd700 1 -- 172.21.15.162:0/1034840363 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8004680 con 0x7f15b80f8500 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15aeffd700 1 -- 172.21.15.162:0/1034840363 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15a800f040 con 0x7f15b80f8500 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15c01a6700 1 -- 172.21.15.162:0/1034840363 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b4005040 con 0x7f15b80f8500 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15aeffd700 1 -- 172.21.15.162:0/1034840363 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a80146b0 con 0x7f15b80f8500 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15c01a6700 10 client.37445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15c01a6700 1 -- 172.21.15.162:0/1034840363 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b8109240 con 0x7f15b80f8500 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.590+0000 7f15c01a6700 1 librados: init done 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.591+0000 7f15c01a6700 1 -- 172.21.15.162:0/1034840363 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b80475e0 con 0x7f15b80f8500 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.591+0000 7f15aeffd700 1 -- 172.21.15.162:0/1034840363 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f15a8014ae0 con 0x7f15b80f8500 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.592+0000 7f15aeffd700 1 --2- 172.21.15.162:0/1034840363 >> v2:172.21.15.162:6800/2866534053 conn(0x7f15a4062c90 0x7f15a4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.592+0000 7f15aeffd700 1 -- 172.21.15.162:0/1034840363 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(423..423 src has 52..423) v4 ==== 8051+0+0 (secure 0 0 0) 0x7f15a807cd10 con 0x7f15b80f8500 2024-02-13T00:46:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.592+0000 7f15aeffd700 10 client.37445.objecter ms_dispatch 0x7f15b8001550 osd_map(423..423 src has 52..423) v4 2024-02-13T00:46:37.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.592+0000 7f15aeffd700 3 client.37445.objecter handle_osd_map got epochs [423,423] > 0 2024-02-13T00:46:37.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.592+0000 7f15aeffd700 3 client.37445.objecter handle_osd_map decoding full epoch 423 2024-02-13T00:46:37.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.592+0000 7f15bd741700 1 --2- 172.21.15.162:0/1034840363 >> v2:172.21.15.162:6800/2866534053 conn(0x7f15a4062c90 0x7f15a4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:37.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.592+0000 7f15aeffd700 20 client.37445.objecter dump_active .. 0 homeless 2024-02-13T00:46:37.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.592+0000 7f15bd741700 1 --2- 172.21.15.162:0/1034840363 >> v2:172.21.15.162:6800/2866534053 conn(0x7f15a4062c90 0x7f15a4065150 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 rx=0x7f15b4004ee0 tx=0x7f15b400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:37.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.592+0000 7f15aeffd700 10 client.37445.objecter ms_handle_connect 0x7f15a4062c90 2024-02-13T00:46:37.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.596+0000 7f15aeffd700 1 -- 172.21.15.162:0/1034840363 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f15a8049f60 con 0x7f15b80f8500 2024-02-13T00:46:37.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:37.605 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:36.085839+0000 osd.1 (osd.1) 179 : cluster [DBG] 4.1b scrub starts 2024-02-13T00:46:37.605 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:36.089397+0000 osd.1 (osd.1) 180 : cluster [DBG] 4.1b scrub ok 2024-02-13T00:46:37.605 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:36.425269+0000 mgr.y (mgr.14544) 372 : cluster [DBG] pgmap v614: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:46:37.631 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:37.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:37.775+0000 7f15c01a6700 1 -- 172.21.15.162:0/1034840363 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f15b8109920 con 0x7f15b80f8500 2024-02-13T00:46:37.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:36.085839+0000 osd.1 (osd.1) 179 : cluster [DBG] 4.1b scrub starts 2024-02-13T00:46:37.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:36.089397+0000 osd.1 (osd.1) 180 : cluster [DBG] 4.1b scrub ok 2024-02-13T00:46:37.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:36.425269+0000 mgr.y (mgr.14544) 372 : cluster [DBG] pgmap v614: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:46:37.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:36.085839+0000 osd.1 (osd.1) 179 : cluster [DBG] 4.1b scrub starts 2024-02-13T00:46:37.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:36.089397+0000 osd.1 (osd.1) 180 : cluster [DBG] 4.1b scrub ok 2024-02-13T00:46:37.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:36.425269+0000 mgr.y (mgr.14544) 372 : cluster [DBG] pgmap v614: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:46:37.857 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:46:37.858 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:46:37.880 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:37.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:37.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:38.013 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:38.361 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:38.362 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:38.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:38.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.455+0000 7f15aeffd700 1 -- 172.21.15.162:0/1034840363 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v424) v1 ==== 87+0+0 (secure 0 0 0) 0x7f15a8050290 con 0x7f15b80f8500 2024-02-13T00:46:38.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 10 librados: watch_flush enter 2024-02-13T00:46:38.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 10 librados: watch_flush exit 2024-02-13T00:46:38.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 20 client.37445.objecter shutdown clearing up homeless session... 2024-02-13T00:46:38.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 10 client.37445.objecter successfully canceled tick 2024-02-13T00:46:38.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 -- 172.21.15.162:0/1034840363 >> v2:172.21.15.162:6800/2866534053 conn(0x7f15a4062c90 msgr2=0x7f15a4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:38.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 --2- 172.21.15.162:0/1034840363 >> v2:172.21.15.162:6800/2866534053 conn(0x7f15a4062c90 0x7f15a4065150 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 rx=0x7f15b4004ee0 tx=0x7f15b400b040).stop 2024-02-13T00:46:38.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 -- 172.21.15.162:0/1034840363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15b80f8500 msgr2=0x7f15b810e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:38.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 --2- 172.21.15.162:0/1034840363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15b80f8500 0x7f15b810e2a0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7f15a800e990 tx=0x7f15a800b040).stop 2024-02-13T00:46:38.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 -- 172.21.15.162:0/1034840363 shutdown_connections 2024-02-13T00:46:38.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 --2- 172.21.15.162:0/1034840363 >> v2:172.21.15.162:6800/2866534053 conn(0x7f15a4062c90 0x7f15a4065150 unknown :-1 s=CLOSED pgs=2903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:38.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 --2- 172.21.15.162:0/1034840363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b80fa440 0x7f15b8108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:38.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 --2- 172.21.15.162:0/1034840363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15b80f8500 0x7f15b810e2a0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:38.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 --2- 172.21.15.162:0/1034840363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b80f9b70 0x7f15b810e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:38.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 -- 172.21.15.162:0/1034840363 >> 172.21.15.162:0/1034840363 conn(0x7f15b80f38c0 msgr2=0x7f15b80f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:38.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 -- 172.21.15.162:0/1034840363 shutdown_connections 2024-02-13T00:46:38.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 -- 172.21.15.162:0/1034840363 wait complete. 2024-02-13T00:46:38.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:38.458+0000 7f15c01a6700 1 librados: shutdown 2024-02-13T00:46:38.463 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:46:38.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:38.497 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:38.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:37.052888+0000 osd.1 (osd.1) 181 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:46:38.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:37.056172+0000 osd.1 (osd.1) 182 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:46:38.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:37.776851+0000 mon.c (mon.2) 380 : audit [INF] from='client.? 172.21.15.162:0/1034840363' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:46:38.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:37.777142+0000 mon.a (mon.0) 2530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:46:38.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-13T00:46:38.595 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:38.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:38.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:38.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:38.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:37.052888+0000 osd.1 (osd.1) 181 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:46:38.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:37.056172+0000 osd.1 (osd.1) 182 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:46:38.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:37.776851+0000 mon.c (mon.2) 380 : audit [INF] from='client.? 172.21.15.162:0/1034840363' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:46:38.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:37.777142+0000 mon.a (mon.0) 2530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:46:38.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:37.052888+0000 osd.1 (osd.1) 181 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:46:38.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:37.056172+0000 osd.1 (osd.1) 182 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:46:38.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:37.776851+0000 mon.c (mon.2) 380 : audit [INF] from='client.? 172.21.15.162:0/1034840363' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:46:38.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:37.777142+0000 mon.a (mon.0) 2530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:46:38.730 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:38.782 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:38.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:38.862 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:46:38.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:38.915 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:39.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:46:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:46:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:46:39.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:37.658773+0000 osd.0 (osd.0) 156 : cluster [DBG] 5.d scrub starts 2024-02-13T00:46:39.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:37.662408+0000 osd.0 (osd.0) 157 : cluster [DBG] 5.d scrub ok 2024-02-13T00:46:39.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:38.425776+0000 mgr.y (mgr.14544) 373 : cluster [DBG] pgmap v615: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T00:46:39.711 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:38.444204+0000 mon.a (mon.0) 2531 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:39.711 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:38.452845+0000 mon.a (mon.0) 2532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:46:39.711 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:38.452917+0000 mon.a (mon.0) 2533 : cluster [DBG] osdmap e424: 8 total, 8 up, 4 in 2024-02-13T00:46:39.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:37.658773+0000 osd.0 (osd.0) 156 : cluster [DBG] 5.d scrub starts 2024-02-13T00:46:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:37.662408+0000 osd.0 (osd.0) 157 : cluster [DBG] 5.d scrub ok 2024-02-13T00:46:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:38.425776+0000 mgr.y (mgr.14544) 373 : cluster [DBG] pgmap v615: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T00:46:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:38.444204+0000 mon.a (mon.0) 2531 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:38.452845+0000 mon.a (mon.0) 2532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:46:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:38.452917+0000 mon.a (mon.0) 2533 : cluster [DBG] osdmap e424: 8 total, 8 up, 4 in 2024-02-13T00:46:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:37.658773+0000 osd.0 (osd.0) 156 : cluster [DBG] 5.d scrub starts 2024-02-13T00:46:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:37.662408+0000 osd.0 (osd.0) 157 : cluster [DBG] 5.d scrub ok 2024-02-13T00:46:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:38.425776+0000 mgr.y (mgr.14544) 373 : cluster [DBG] pgmap v615: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 767 B/s rd, 0 op/s 2024-02-13T00:46:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:38.444204+0000 mon.a (mon.0) 2531 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:38.452845+0000 mon.a (mon.0) 2532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:46:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:38.452917+0000 mon.a (mon.0) 2533 : cluster [DBG] osdmap e424: 8 total, 8 up, 4 in 2024-02-13T00:46:40.466 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:40.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:40.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:40.601 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:40.644 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:40.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:40.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:40.650 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:46:40.650 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:46:40.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:40.776 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:40.796 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:40.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:40.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:40.900 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:40.928 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:41.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.392+0000 7ffab085d700 1 -- 172.21.15.162:0/2289283266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaac0fd490 msgr2=0x7ffaac0f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:41.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.392+0000 7ffab085d700 1 --2- 172.21.15.162:0/2289283266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaac0fd490 0x7ffaac0f6cc0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 rx=0x7ffaa0004870 tx=0x7ffaa0009cb0).stop 2024-02-13T00:46:41.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 -- 172.21.15.162:0/2289283266 shutdown_connections 2024-02-13T00:46:41.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 --2- 172.21.15.162:0/2289283266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaac0f73a0 0x7ffaac1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:41.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 --2- 172.21.15.162:0/2289283266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaac0fcb30 0x7ffaac0fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 --2- 172.21.15.162:0/2289283266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaac0fd490 0x7ffaac0f6cc0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 -- 172.21.15.162:0/2289283266 >> 172.21.15.162:0/2289283266 conn(0x7ffaac06eb90 msgr2=0x7ffaac0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 -- 172.21.15.162:0/2289283266 shutdown_connections 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 -- 172.21.15.162:0/2289283266 wait complete. 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 librados: starting msgr at 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 librados: starting objecter 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 Processor -- start 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 -- start start 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 librados: setting wanted keys 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.393+0000 7ffab085d700 1 librados: calling monclient init 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffab085d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaac0f73a0 0x7ffaac0f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffab085d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaac0fcb30 0x7ffaac0f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffab085d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaac0fd490 0x7ffaac0f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:41.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffab085d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac116570 con 0x7ffaac0fcb30 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffab085d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac1166e0 con 0x7ffaac0fd490 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffab085d700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac1168a0 con 0x7ffaac0f73a0 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffaa9d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaac0fcb30 0x7ffaac0f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffaa9d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaac0fcb30 0x7ffaac0f8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:39476/0 (socket says 172.21.15.162:39476) 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffaa9d9b700 1 -- 172.21.15.162:0/2771150087 learned_addr learned my addr 172.21.15.162:0/2771150087 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffaaa59c700 1 --2- 172.21.15.162:0/2771150087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaac0f73a0 0x7ffaac0f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffaaad9d700 1 --2- 172.21.15.162:0/2771150087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaac0fd490 0x7ffaac0f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffaa9d9b700 1 -- 172.21.15.162:0/2771150087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaac0f73a0 msgr2=0x7ffaac0f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffaa9d9b700 1 --2- 172.21.15.162:0/2771150087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaac0f73a0 0x7ffaac0f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffaa9d9b700 1 -- 172.21.15.162:0/2771150087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaac0fd490 msgr2=0x7ffaac0f9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffaa9d9b700 1 --2- 172.21.15.162:0/2771150087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaac0fd490 0x7ffaac0f9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffaa9d9b700 1 -- 172.21.15.162:0/2771150087 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa40097c0 con 0x7ffaac0fcb30 2024-02-13T00:46:41.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffaa9d9b700 1 --2- 172.21.15.162:0/2771150087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaac0fcb30 0x7ffaac0f8e20 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 rx=0x7ffaa0009eb0 tx=0x7ffaa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffa9b7fe700 10 client.?.objecter ms_handle_connect 0x7ffaac0fcb30 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffa9b7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffa9b7fe700 1 -- 172.21.15.162:0/2771150087 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa000e070 con 0x7ffaac0fcb30 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffa9b7fe700 1 -- 172.21.15.162:0/2771150087 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffaa0013da0 con 0x7ffaac0fcb30 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffab085d700 1 -- 172.21.15.162:0/2771150087 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa0005040 con 0x7ffaac0fcb30 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffa9b7fe700 1 -- 172.21.15.162:0/2771150087 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0019070 con 0x7ffaac0fcb30 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffab085d700 10 client.27351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffab085d700 1 -- 172.21.15.162:0/2771150087 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaac116e70 con 0x7ffaac0fcb30 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.394+0000 7ffab085d700 1 librados: init done 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.395+0000 7ffab085d700 1 -- 172.21.15.162:0/2771150087 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaac106020 con 0x7ffaac0fcb30 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.396+0000 7ffa9b7fe700 1 -- 172.21.15.162:0/2771150087 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ffaa001e420 con 0x7ffaac0fcb30 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.396+0000 7ffa9b7fe700 1 --2- 172.21.15.162:0/2771150087 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffa94062c90 0x7ffa94065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.396+0000 7ffa9b7fe700 1 -- 172.21.15.162:0/2771150087 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(425..425 src has 52..425) v4 ==== 7995+0+0 (secure 0 0 0) 0x7ffaa007dd50 con 0x7ffaac0fcb30 2024-02-13T00:46:41.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.396+0000 7ffa9b7fe700 10 client.27351.objecter ms_dispatch 0x7ffaac001550 osd_map(425..425 src has 52..425) v4 2024-02-13T00:46:41.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.396+0000 7ffa9b7fe700 3 client.27351.objecter handle_osd_map got epochs [425,425] > 0 2024-02-13T00:46:41.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.396+0000 7ffa9b7fe700 3 client.27351.objecter handle_osd_map decoding full epoch 425 2024-02-13T00:46:41.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.396+0000 7ffaaa59c700 1 --2- 172.21.15.162:0/2771150087 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffa94062c90 0x7ffa94065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:41.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.396+0000 7ffa9b7fe700 20 client.27351.objecter dump_active .. 0 homeless 2024-02-13T00:46:41.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.396+0000 7ffa9b7fe700 10 client.27351.objecter ms_handle_connect 0x7ffa94062c90 2024-02-13T00:46:41.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.396+0000 7ffaaa59c700 1 --2- 172.21.15.162:0/2771150087 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffa94062c90 0x7ffa94065150 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 rx=0x7ffa9c0041d0 tx=0x7ffa9c009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:41.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.401+0000 7ffa9b7fe700 1 -- 172.21.15.162:0/2771150087 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ffaa0081030 con 0x7ffaac0fcb30 2024-02-13T00:46:41.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-13T00:46:41.576 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:41.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:41.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:41.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.588+0000 7ffab085d700 1 -- 172.21.15.162:0/2771150087 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7ffaac05fc10 con 0x7ffaac0fcb30 2024-02-13T00:46:41.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:46:41.703 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:40.421587+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e425: 8 total, 8 up, 4 in 2024-02-13T00:46:41.703 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:40.426226+0000 mgr.y (mgr.14544) 374 : cluster [DBG] pgmap v618: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-13T00:46:41.705 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:41.720 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:40.421587+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e425: 8 total, 8 up, 4 in 2024-02-13T00:46:41.720 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:40.426226+0000 mgr.y (mgr.14544) 374 : cluster [DBG] pgmap v618: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-13T00:46:41.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:40.421587+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e425: 8 total, 8 up, 4 in 2024-02-13T00:46:41.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:40.426226+0000 mgr.y (mgr.14544) 374 : cluster [DBG] pgmap v618: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-13T00:46:41.724 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:46:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:41.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:46:41.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:46:41.829 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:46:41.856 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:41.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.953+0000 7fe30051e700 1 -- 172.21.15.162:0/3982985147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2f80f95e0 msgr2=0x7fe2f8101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:41.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.953+0000 7fe30051e700 1 --2- 172.21.15.162:0/3982985147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2f80f95e0 0x7fe2f8101d80 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 rx=0x7fe2f40047b0 tx=0x7fe2f4009cb0).stop 2024-02-13T00:46:41.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.953+0000 7fe30051e700 1 -- 172.21.15.162:0/3982985147 shutdown_connections 2024-02-13T00:46:41.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.953+0000 7fe30051e700 1 --2- 172.21.15.162:0/3982985147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2f8102270 0x7fe2f8104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:41.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.953+0000 7fe30051e700 1 --2- 172.21.15.162:0/3982985147 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f80f8d10 0x7fe2f80f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:41.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.953+0000 7fe30051e700 1 --2- 172.21.15.162:0/3982985147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2f80f95e0 0x7fe2f8101d80 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:41.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.953+0000 7fe30051e700 1 -- 172.21.15.162:0/3982985147 >> 172.21.15.162:0/3982985147 conn(0x7fe2f806eb90 msgr2=0x7fe2f80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:41.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.953+0000 7fe30051e700 1 -- 172.21.15.162:0/3982985147 shutdown_connections 2024-02-13T00:46:41.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.953+0000 7fe30051e700 1 -- 172.21.15.162:0/3982985147 wait complete. 2024-02-13T00:46:41.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe30051e700 1 librados: starting msgr at 2024-02-13T00:46:41.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe30051e700 1 librados: starting objecter 2024-02-13T00:46:41.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe30051e700 1 Processor -- start 2024-02-13T00:46:41.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe30051e700 1 -- start start 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe30051e700 1 librados: setting wanted keys 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe30051e700 1 librados: calling monclient init 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe30051e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2f80f8d10 0x7fe2f80f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe30051e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f80f95e0 0x7fe2f80f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe30051e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2f8102270 0x7fe2f80f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe30051e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f81166d0 con 0x7fe2f8102270 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe30051e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f8116840 con 0x7fe2f80f95e0 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe30051e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f8116a00 con 0x7fe2f80f8d10 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2fe2ba700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2f80f8d10 0x7fe2f80f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2feabb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2f8102270 0x7fe2f80f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2fe2ba700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2f80f8d10 0x7fe2f80f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:46350/0 (socket says 172.21.15.162:46350) 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2feabb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2f8102270 0x7fe2f80f76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:39486/0 (socket says 172.21.15.162:39486) 2024-02-13T00:46:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2fe2ba700 1 -- 172.21.15.162:0/2164082683 learned_addr learned my addr 172.21.15.162:0/2164082683 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2fdab9700 1 --2- 172.21.15.162:0/2164082683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f80f95e0 0x7fe2f80f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2feabb700 1 -- 172.21.15.162:0/2164082683 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2f80f8d10 msgr2=0x7fe2f80f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2feabb700 1 --2- 172.21.15.162:0/2164082683 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2f80f8d10 0x7fe2f80f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2feabb700 1 -- 172.21.15.162:0/2164082683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f80f95e0 msgr2=0x7fe2f80f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2feabb700 1 --2- 172.21.15.162:0/2164082683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f80f95e0 0x7fe2f80f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2feabb700 1 -- 172.21.15.162:0/2164082683 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f00097c0 con 0x7fe2f8102270 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2fe2ba700 1 --2- 172.21.15.162:0/2164082683 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2f80f8d10 0x7fe2f80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2feabb700 1 --2- 172.21.15.162:0/2164082683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2f8102270 0x7fe2f80f76a0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 rx=0x7fe2f0001dc0 tx=0x7fe2f0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2ef7fe700 10 client.?.objecter ms_handle_connect 0x7fe2f8102270 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.954+0000 7fe2ef7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.955+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0009a10 con 0x7fe2f8102270 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.955+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2f00090d0 con 0x7fe2f8102270 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.955+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0007c90 con 0x7fe2f8102270 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.955+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f4005040 con 0x7fe2f8102270 2024-02-13T00:46:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.955+0000 7fe30051e700 10 client.27357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.955+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f8116ff0 con 0x7fe2f8102270 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.955+0000 7fe30051e700 1 librados: init done 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.955+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe2f81071b0 con 0x7fe2f8102270 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.956+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe2f000fb30 con 0x7fe2f8102270 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.956+0000 7fe2ef7fe700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe2e4062c90 0x7fe2e4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.956+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(425..425 src has 52..425) v4 ==== 7995+0+0 (secure 0 0 0) 0x7fe2f007dc30 con 0x7fe2f8102270 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.956+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 osd_map(425..425 src has 52..425) v4 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.956+0000 7fe2fe2ba700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe2e4062c90 0x7fe2e4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.956+0000 7fe2ef7fe700 3 client.27357.objecter handle_osd_map got epochs [425,425] > 0 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.956+0000 7fe2ef7fe700 3 client.27357.objecter handle_osd_map decoding full epoch 425 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe2ef7fe700 20 client.27357.objecter dump_active .. 0 homeless 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v425) v1 ==== 54+0+16 (secure 0 0 0) 0x7fe2f007e160 con 0x7fe2f8102270 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe2ef7fe700 10 client.27357.objecter ms_handle_connect 0x7fe2e4062c90 2024-02-13T00:46:41.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe2fe2ba700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe2e4062c90 0x7fe2e4065150 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 rx=0x7fe2e8007d30 tx=0x7fe2e800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe30051e700 10 client.27357.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 1 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe2f8028590 0x7fe2f80fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f80f81a0 osd=0 3 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f80f81a0 osd=0 3 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f80f81a0 osd=0 3 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 1 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f80f81a0 osd=0 3 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 0 1 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f80f81a0 osd=0 4 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe30051e700 10 client.27357.objecter _send_command 1 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2f80fd370 con 0x7fe2f8028590 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe2fdab9700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe2f8028590 0x7fe2f80fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe2fdab9700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe2f8028590 0x7fe2f80fbd00 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.957+0000 7fe2ef7fe700 10 client.27357.objecter ms_handle_connect 0x7fe2f8028590 2024-02-13T00:46:41.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.958+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe2f80fd370 con 0x7fe2f8028590 2024-02-13T00:46:41.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.958+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:46:41.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.958+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 1 = system:0 2024-02-13T00:46:41.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.958+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f80f81a0 osd=0 4 2024-02-13T00:46:41.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.958+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 0 1 2024-02-13T00:46:41.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.972+0000 7fe30051e700 10 client.27357.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:41.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.972+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 2 2024-02-13T00:46:41.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.972+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f80f81a0 osd=0 3 2024-02-13T00:46:41.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.972+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f80f81a0 osd=0 3 2024-02-13T00:46:41.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.972+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f80f81a0 osd=0 3 2024-02-13T00:46:41.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.972+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 2 2024-02-13T00:46:41.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.972+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f80f81a0 osd=0 3 2024-02-13T00:46:41.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.972+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 0 2 2024-02-13T00:46:41.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.972+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f80f81a0 osd=0 4 2024-02-13T00:46:41.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.972+0000 7fe30051e700 10 client.27357.objecter _send_command 2 2024-02-13T00:46:41.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.972+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe2f80fd4e0 con 0x7fe2f8028590 2024-02-13T00:46:41.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe2f80fd4e0 con 0x7fe2f8028590 2024-02-13T00:46:41.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:41.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:41.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f80f81a0 osd=0 4 2024-02-13T00:46:41.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 0 2 2024-02-13T00:46:41.979 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:46:41.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe30051e700 10 client.27357.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:41.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 3 2024-02-13T00:46:41.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6805/150599780 conn(0x7fe2f80fc870 0x7fe2f80fcc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:41.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f80fc1f0 osd=1 3 2024-02-13T00:46:41.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f80fc1f0 osd=1 3 2024-02-13T00:46:41.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f80fc1f0 osd=1 3 2024-02-13T00:46:41.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 3 2024-02-13T00:46:41.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f80fc1f0 osd=1 3 2024-02-13T00:46:41.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 1 3 2024-02-13T00:46:41.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f80fc1f0 osd=1 4 2024-02-13T00:46:41.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe30051e700 10 client.27357.objecter _send_command 3 2024-02-13T00:46:41.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2f80fd140 con 0x7fe2f80fc870 2024-02-13T00:46:41.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.974+0000 7fe2fe2ba700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6805/150599780 conn(0x7fe2f80fc870 0x7fe2f80fcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:41.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.975+0000 7fe2fe2ba700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6805/150599780 conn(0x7fe2f80fc870 0x7fe2f80fcc50 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:41.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.975+0000 7fe2ef7fe700 10 client.27357.objecter ms_handle_connect 0x7fe2f80fc870 2024-02-13T00:46:41.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.975+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe2f80fd140 con 0x7fe2f80fc870 2024-02-13T00:46:41.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.975+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:46:41.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.975+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 3 = system:0 2024-02-13T00:46:41.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.975+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f80fc1f0 osd=1 4 2024-02-13T00:46:41.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.975+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 1 3 2024-02-13T00:46:41.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.989+0000 7fe30051e700 10 client.27357.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:41.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.989+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 4 2024-02-13T00:46:41.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.989+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f80fc1f0 osd=1 3 2024-02-13T00:46:41.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.989+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f80fc1f0 osd=1 3 2024-02-13T00:46:41.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.989+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f80fc1f0 osd=1 3 2024-02-13T00:46:41.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.989+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 4 2024-02-13T00:46:41.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.989+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f80fc1f0 osd=1 3 2024-02-13T00:46:41.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.989+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 1 4 2024-02-13T00:46:41.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.989+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f80fc1f0 osd=1 4 2024-02-13T00:46:41.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.989+0000 7fe30051e700 10 client.27357.objecter _send_command 4 2024-02-13T00:46:41.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.989+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe2f8059400 con 0x7fe2f80fc870 2024-02-13T00:46:41.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe2f8059400 con 0x7fe2f80fc870 2024-02-13T00:46:41.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:41.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:41.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f80fc1f0 osd=1 4 2024-02-13T00:46:41.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 1 4 2024-02-13T00:46:41.996 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:46:41.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe30051e700 10 client.27357.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:41.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 5 2024-02-13T00:46:41.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe2f80280f0 0x7fe2f811e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:41.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8059810 osd=2 3 2024-02-13T00:46:41.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8059810 osd=2 3 2024-02-13T00:46:41.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8059810 osd=2 3 2024-02-13T00:46:41.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 5 2024-02-13T00:46:41.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f8059810 osd=2 3 2024-02-13T00:46:41.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 2 5 2024-02-13T00:46:41.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8059810 osd=2 4 2024-02-13T00:46:41.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe30051e700 10 client.27357.objecter _send_command 5 2024-02-13T00:46:41.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2f8121040 con 0x7fe2f80280f0 2024-02-13T00:46:41.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.991+0000 7fe2fdab9700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe2f80280f0 0x7fe2f811e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:41.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.992+0000 7fe2ef7fe700 10 client.27357.objecter ms_handle_connect 0x7fe2f80280f0 2024-02-13T00:46:41.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.992+0000 7fe2fdab9700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe2f80280f0 0x7fe2f811e3f0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:41.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.992+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe2f8121040 con 0x7fe2f80280f0 2024-02-13T00:46:41.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.992+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:46:41.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.992+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 5 = system:0 2024-02-13T00:46:41.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.992+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f8059810 osd=2 4 2024-02-13T00:46:41.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:41.992+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 2 5 2024-02-13T00:46:42.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.006+0000 7fe30051e700 10 client.27357.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:42.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.006+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 6 2024-02-13T00:46:42.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.006+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8059810 osd=2 3 2024-02-13T00:46:42.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.006+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8059810 osd=2 3 2024-02-13T00:46:42.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.006+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8059810 osd=2 3 2024-02-13T00:46:42.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.006+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 6 2024-02-13T00:46:42.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.006+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f8059810 osd=2 3 2024-02-13T00:46:42.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.006+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 2 6 2024-02-13T00:46:42.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.006+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8059810 osd=2 4 2024-02-13T00:46:42.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.006+0000 7fe30051e700 10 client.27357.objecter _send_command 6 2024-02-13T00:46:42.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.006+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe2f80276b0 con 0x7fe2f80280f0 2024-02-13T00:46:42.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.007+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe2f80276b0 con 0x7fe2f80280f0 2024-02-13T00:46:42.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.007+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:42.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.007+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.007+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f8059810 osd=2 4 2024-02-13T00:46:42.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.007+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 2 6 2024-02-13T00:46:42.012 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:46:42.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe30051e700 10 client.27357.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:42.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 7 2024-02-13T00:46:42.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6813/604971011 conn(0x7fe2f811fa60 0x7fe2f811fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:42.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f811ba30 osd=3 3 2024-02-13T00:46:42.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f811ba30 osd=3 3 2024-02-13T00:46:42.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f811ba30 osd=3 3 2024-02-13T00:46:42.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 7 2024-02-13T00:46:42.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f811ba30 osd=3 3 2024-02-13T00:46:42.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 3 7 2024-02-13T00:46:42.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f811ba30 osd=3 4 2024-02-13T00:46:42.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe30051e700 10 client.27357.objecter _send_command 7 2024-02-13T00:46:42.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2f811c130 con 0x7fe2f811fa60 2024-02-13T00:46:42.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe2feabb700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6813/604971011 conn(0x7fe2f811fa60 0x7fe2f811fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:42.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe2ef7fe700 10 client.27357.objecter ms_handle_connect 0x7fe2f811fa60 2024-02-13T00:46:42.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.008+0000 7fe2feabb700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6813/604971011 conn(0x7fe2f811fa60 0x7fe2f811fea0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:42.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.009+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe2f811c130 con 0x7fe2f811fa60 2024-02-13T00:46:42.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.009+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:46:42.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.009+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 7 = system:0 2024-02-13T00:46:42.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.009+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f811ba30 osd=3 4 2024-02-13T00:46:42.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.009+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 3 7 2024-02-13T00:46:42.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.023+0000 7fe30051e700 10 client.27357.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:42.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.023+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 8 2024-02-13T00:46:42.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.023+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f811ba30 osd=3 3 2024-02-13T00:46:42.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.023+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f811ba30 osd=3 3 2024-02-13T00:46:42.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.023+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f811ba30 osd=3 3 2024-02-13T00:46:42.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.023+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 8 2024-02-13T00:46:42.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.023+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f811ba30 osd=3 3 2024-02-13T00:46:42.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.023+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 3 8 2024-02-13T00:46:42.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.023+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f811ba30 osd=3 4 2024-02-13T00:46:42.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.023+0000 7fe30051e700 10 client.27357.objecter _send_command 8 2024-02-13T00:46:42.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.023+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe2f8120b60 con 0x7fe2f811fa60 2024-02-13T00:46:42.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe2f8120b60 con 0x7fe2f811fa60 2024-02-13T00:46:42.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:42.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f811ba30 osd=3 4 2024-02-13T00:46:42.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 3 8 2024-02-13T00:46:42.029 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:46:42.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe30051e700 10 client.27357.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:42.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 9 2024-02-13T00:46:42.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe2f81231c0 0x7fe2f81255b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:42.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8120b60 osd=4 3 2024-02-13T00:46:42.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8120b60 osd=4 3 2024-02-13T00:46:42.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8120b60 osd=4 3 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 9 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f8120b60 osd=4 3 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 4 9 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8120b60 osd=4 4 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe30051e700 10 client.27357.objecter _send_command 9 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.024+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2f8120dc0 con 0x7fe2f81231c0 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.025+0000 7fe2fe2ba700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe2f81231c0 0x7fe2f81255b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.025+0000 7fe2ef7fe700 10 client.27357.objecter ms_handle_connect 0x7fe2f81231c0 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.025+0000 7fe2fe2ba700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe2f81231c0 0x7fe2f81255b0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.026+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe2f8120dc0 con 0x7fe2f81231c0 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.026+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.026+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 9 = system:0 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.026+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f8120b60 osd=4 4 2024-02-13T00:46:42.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.026+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 4 9 2024-02-13T00:46:42.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.040+0000 7fe30051e700 10 client.27357.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:42.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.040+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 10 2024-02-13T00:46:42.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.040+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8120b60 osd=4 3 2024-02-13T00:46:42.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.040+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8120b60 osd=4 3 2024-02-13T00:46:42.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.040+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8120b60 osd=4 3 2024-02-13T00:46:42.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.040+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 10 2024-02-13T00:46:42.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.040+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f8120b60 osd=4 3 2024-02-13T00:46:42.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.040+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 4 10 2024-02-13T00:46:42.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.040+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8120b60 osd=4 4 2024-02-13T00:46:42.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.040+0000 7fe30051e700 10 client.27357.objecter _send_command 10 2024-02-13T00:46:42.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.040+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe2f8120dc0 con 0x7fe2f81231c0 2024-02-13T00:46:42.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe2f8120dc0 con 0x7fe2f81231c0 2024-02-13T00:46:42.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:42.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f8120b60 osd=4 4 2024-02-13T00:46:42.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 4 10 2024-02-13T00:46:42.047 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:46:42.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe30051e700 10 client.27357.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:42.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 11 2024-02-13T00:46:42.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6804/2212296748 conn(0x7fe2f8125fb0 0x7fe2f81263d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:42.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8120dc0 osd=5 3 2024-02-13T00:46:42.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8120dc0 osd=5 3 2024-02-13T00:46:42.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8120dc0 osd=5 3 2024-02-13T00:46:42.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 11 2024-02-13T00:46:42.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f8120dc0 osd=5 3 2024-02-13T00:46:42.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 5 11 2024-02-13T00:46:42.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8120dc0 osd=5 4 2024-02-13T00:46:42.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe30051e700 10 client.27357.objecter _send_command 11 2024-02-13T00:46:42.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.042+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2f8128050 con 0x7fe2f8125fb0 2024-02-13T00:46:42.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.043+0000 7fe2fdab9700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6804/2212296748 conn(0x7fe2f8125fb0 0x7fe2f81263d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:42.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.043+0000 7fe2fdab9700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6804/2212296748 conn(0x7fe2f8125fb0 0x7fe2f81263d0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:42.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.043+0000 7fe2ef7fe700 10 client.27357.objecter ms_handle_connect 0x7fe2f8125fb0 2024-02-13T00:46:42.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.043+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe2f8128050 con 0x7fe2f8125fb0 2024-02-13T00:46:42.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.043+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:46:42.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.043+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 11 = system:0 2024-02-13T00:46:42.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.043+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f8120dc0 osd=5 4 2024-02-13T00:46:42.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.043+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 5 11 2024-02-13T00:46:42.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.058+0000 7fe30051e700 10 client.27357.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:42.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.058+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 12 2024-02-13T00:46:42.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.058+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8120dc0 osd=5 3 2024-02-13T00:46:42.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.058+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8120dc0 osd=5 3 2024-02-13T00:46:42.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.058+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8120dc0 osd=5 3 2024-02-13T00:46:42.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.058+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 12 2024-02-13T00:46:42.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.058+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f8120dc0 osd=5 3 2024-02-13T00:46:42.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.058+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 5 12 2024-02-13T00:46:42.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.058+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8120dc0 osd=5 4 2024-02-13T00:46:42.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.058+0000 7fe30051e700 10 client.27357.objecter _send_command 12 2024-02-13T00:46:42.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.058+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe2f8128050 con 0x7fe2f8125fb0 2024-02-13T00:46:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe2f8128050 con 0x7fe2f8125fb0 2024-02-13T00:46:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f8120dc0 osd=5 4 2024-02-13T00:46:42.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 5 12 2024-02-13T00:46:42.064 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:46:42.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe30051e700 10 client.27357.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:42.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 13 2024-02-13T00:46:42.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe2f812a7a0 0x7fe2f812ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8128050 osd=6 3 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8128050 osd=6 3 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8128050 osd=6 3 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 13 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f8128050 osd=6 3 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 6 13 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8128050 osd=6 4 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe30051e700 10 client.27357.objecter _send_command 13 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2f812b220 con 0x7fe2f812a7a0 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe2feabb700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe2f812a7a0 0x7fe2f812ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe2ef7fe700 10 client.27357.objecter ms_handle_connect 0x7fe2f812a7a0 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.060+0000 7fe2feabb700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe2f812a7a0 0x7fe2f812ac00 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.061+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe2f812b220 con 0x7fe2f812a7a0 2024-02-13T00:46:42.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.061+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:46:42.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.061+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 13 = system:0 2024-02-13T00:46:42.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.061+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f8128050 osd=6 4 2024-02-13T00:46:42.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.061+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 6 13 2024-02-13T00:46:42.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.075+0000 7fe30051e700 10 client.27357.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:42.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.075+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 14 2024-02-13T00:46:42.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.075+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8128050 osd=6 3 2024-02-13T00:46:42.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.075+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8128050 osd=6 3 2024-02-13T00:46:42.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.075+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f8128050 osd=6 3 2024-02-13T00:46:42.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.075+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 14 2024-02-13T00:46:42.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.075+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f8128050 osd=6 3 2024-02-13T00:46:42.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.075+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 6 14 2024-02-13T00:46:42.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.075+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8128050 osd=6 4 2024-02-13T00:46:42.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.075+0000 7fe30051e700 10 client.27357.objecter _send_command 14 2024-02-13T00:46:42.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.075+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe2f812b220 con 0x7fe2f812a7a0 2024-02-13T00:46:42.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe2f812b220 con 0x7fe2f812a7a0 2024-02-13T00:46:42.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:42.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f8128050 osd=6 4 2024-02-13T00:46:42.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 6 14 2024-02-13T00:46:42.084 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:46:42.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe30051e700 10 client.27357.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:42.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 15 2024-02-13T00:46:42.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe2f812da60 0x7fe2f812dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:42.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f812b220 osd=7 3 2024-02-13T00:46:42.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f812b220 osd=7 3 2024-02-13T00:46:42.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f812b220 osd=7 3 2024-02-13T00:46:42.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 15 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f812b220 osd=7 3 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 7 15 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f812b220 osd=7 4 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe30051e700 10 client.27357.objecter _send_command 15 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2f81317c0 con 0x7fe2f812da60 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe2fe2ba700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe2f812da60 0x7fe2f812dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe2fe2ba700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe2f812da60 0x7fe2f812dec0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.077+0000 7fe2ef7fe700 10 client.27357.objecter ms_handle_connect 0x7fe2f812da60 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.078+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe2f81317c0 con 0x7fe2f812da60 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.078+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.078+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 15 = system:0 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.078+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f812b220 osd=7 4 2024-02-13T00:46:42.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.078+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 7 15 2024-02-13T00:46:42.096 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.092+0000 7fe30051e700 10 client.27357.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:42.096 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.092+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign -1 16 2024-02-13T00:46:42.096 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.092+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f812b220 osd=7 3 2024-02-13T00:46:42.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.092+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f812b220 osd=7 3 2024-02-13T00:46:42.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.092+0000 7fe30051e700 20 client.27357.objecter _get_session s=0x7fe2f812b220 osd=7 3 2024-02-13T00:46:42.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.092+0000 7fe30051e700 15 client.27357.objecter _session_command_op_remove -1 16 2024-02-13T00:46:42.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.092+0000 7fe30051e700 20 client.27357.objecter get_session s=0x7fe2f812b220 osd=7 3 2024-02-13T00:46:42.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.092+0000 7fe30051e700 15 client.27357.objecter _session_command_op_assign 7 16 2024-02-13T00:46:42.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.092+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f812b220 osd=7 4 2024-02-13T00:46:42.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.092+0000 7fe30051e700 10 client.27357.objecter _send_command 16 2024-02-13T00:46:42.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.092+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe2f81269b0 con 0x7fe2f812da60 2024-02-13T00:46:42.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe2ef7fe700 1 -- 172.21.15.162:0/2164082683 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe2f81269b0 con 0x7fe2f812da60 2024-02-13T00:46:42.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe2ef7fe700 10 client.27357.objecter ms_dispatch 0x7fe2f8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:42.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe2ef7fe700 10 client.27357.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe2ef7fe700 20 client.27357.objecter put_session s=0x7fe2f812b220 osd=7 4 2024-02-13T00:46:42.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe2ef7fe700 15 client.27357.objecter _session_command_op_remove 7 16 2024-02-13T00:46:42.099 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:46:42.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 10 librados: watch_flush enter 2024-02-13T00:46:42.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 10 librados: watch_flush exit 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 10 client.27357.objecter close_session for osd.0 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe2f8028590 msgr2=0x7fe2f80fbd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe2f8028590 0x7fe2f80fbd00 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f80f81a0 osd=0 1 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 10 client.27357.objecter close_session for osd.1 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6805/150599780 conn(0x7fe2f80fc870 msgr2=0x7fe2f80fcc50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6805/150599780 conn(0x7fe2f80fc870 0x7fe2f80fcc50 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f80fc1f0 osd=1 1 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 10 client.27357.objecter close_session for osd.2 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe2f80280f0 msgr2=0x7fe2f811e3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe2f80280f0 0x7fe2f811e3f0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8059810 osd=2 1 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 10 client.27357.objecter close_session for osd.3 2024-02-13T00:46:42.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6813/604971011 conn(0x7fe2f811fa60 msgr2=0x7fe2f811fea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6813/604971011 conn(0x7fe2f811fa60 0x7fe2f811fea0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe2feabb700 1 -- 172.21.15.162:0/2164082683 reap_dead start 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f811ba30 osd=3 1 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 10 client.27357.objecter close_session for osd.4 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe2f81231c0 msgr2=0x7fe2f81255b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe2f81231c0 0x7fe2f81255b0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe2feabb700 1 -- 172.21.15.162:0/2164082683 reap_dead start 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8120b60 osd=4 1 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 10 client.27357.objecter close_session for osd.5 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6804/2212296748 conn(0x7fe2f8125fb0 msgr2=0x7fe2f81263d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6804/2212296748 conn(0x7fe2f8125fb0 0x7fe2f81263d0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8120dc0 osd=5 1 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 10 client.27357.objecter close_session for osd.6 2024-02-13T00:46:42.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe2f812a7a0 msgr2=0x7fe2f812ac00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe2f812a7a0 0x7fe2f812ac00 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f8128050 osd=6 1 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 10 client.27357.objecter close_session for osd.7 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe2f812da60 msgr2=0x7fe2f812dec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe2f812da60 0x7fe2f812dec0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 20 client.27357.objecter put_session s=0x7fe2f812b220 osd=7 1 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 20 client.27357.objecter shutdown clearing up homeless session... 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.094+0000 7fe30051e700 10 client.27357.objecter successfully canceled tick 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.095+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe2e4062c90 msgr2=0x7fe2e4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.095+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe2e4062c90 0x7fe2e4065150 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 rx=0x7fe2e8007d30 tx=0x7fe2e800b040).stop 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.095+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2f8102270 msgr2=0x7fe2f80f76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.095+0000 7fe30051e700 1 --2- 172.21.15.162:0/2164082683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2f8102270 0x7fe2f80f76a0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 rx=0x7fe2f0001dc0 tx=0x7fe2f0004d10).stop 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.095+0000 7fe2feabb700 1 -- 172.21.15.162:0/2164082683 reap_dead start 2024-02-13T00:46:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.095+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 shutdown_connections 2024-02-13T00:46:42.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.095+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 >> 172.21.15.162:0/2164082683 conn(0x7fe2f806eb90 msgr2=0x7fe2f8102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:42.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.095+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 shutdown_connections 2024-02-13T00:46:42.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.095+0000 7fe30051e700 1 -- 172.21.15.162:0/2164082683 wait complete. 2024-02-13T00:46:42.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.095+0000 7fe30051e700 1 librados: shutdown 2024-02-13T00:46:42.104 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.104 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.104 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.104 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.104 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.104 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.104 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.104 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:46:42.105 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:46:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:42.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:46:42.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:42.236 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:42.308 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:42.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:42.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:42.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.435+0000 7ffa9b7fe700 1 -- 172.21.15.162:0/2771150087 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v426) v1 ==== 113+0+0 (secure 0 0 0) 0x7ffaa0023030 con 0x7ffaac0fcb30 2024-02-13T00:46:42.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.438+0000 7ffab085d700 10 librados: watch_flush enter 2024-02-13T00:46:42.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.438+0000 7ffab085d700 10 librados: watch_flush exit 2024-02-13T00:46:42.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.438+0000 7ffab085d700 20 client.27351.objecter shutdown clearing up homeless session... 2024-02-13T00:46:42.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.438+0000 7ffab085d700 10 client.27351.objecter successfully canceled tick 2024-02-13T00:46:42.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.438+0000 7ffab085d700 1 -- 172.21.15.162:0/2771150087 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffa94062c90 msgr2=0x7ffa94065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:42.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.438+0000 7ffab085d700 1 --2- 172.21.15.162:0/2771150087 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffa94062c90 0x7ffa94065150 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 rx=0x7ffa9c0041d0 tx=0x7ffa9c009040).stop 2024-02-13T00:46:42.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.438+0000 7ffab085d700 1 -- 172.21.15.162:0/2771150087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaac0fcb30 msgr2=0x7ffaac0f8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:42.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.438+0000 7ffab085d700 1 --2- 172.21.15.162:0/2771150087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaac0fcb30 0x7ffaac0f8e20 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 rx=0x7ffaa0009eb0 tx=0x7ffaa000b040).stop 2024-02-13T00:46:42.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.439+0000 7ffab085d700 1 -- 172.21.15.162:0/2771150087 shutdown_connections 2024-02-13T00:46:42.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.439+0000 7ffab085d700 1 --2- 172.21.15.162:0/2771150087 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffa94062c90 0x7ffa94065150 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:42.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.439+0000 7ffab085d700 1 --2- 172.21.15.162:0/2771150087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffaac0fd490 0x7ffaac0f9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:42.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.439+0000 7ffab085d700 1 --2- 172.21.15.162:0/2771150087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaac0f73a0 0x7ffaac0f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:42.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.439+0000 7ffab085d700 1 --2- 172.21.15.162:0/2771150087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaac0fcb30 0x7ffaac0f8e20 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:42.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.439+0000 7ffab085d700 1 -- 172.21.15.162:0/2771150087 >> 172.21.15.162:0/2771150087 conn(0x7ffaac06eb90 msgr2=0x7ffaac070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:42.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.439+0000 7ffab085d700 1 -- 172.21.15.162:0/2771150087 shutdown_connections 2024-02-13T00:46:42.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.439+0000 7ffab085d700 1 -- 172.21.15.162:0/2771150087 wait complete. 2024-02-13T00:46:42.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:42.439+0000 7ffab085d700 1 librados: shutdown 2024-02-13T00:46:42.445 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.6 to 1 (10000) 2024-02-13T00:46:42.447 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:42.572 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:42.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:42.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:42.685 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:42.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:41.590693+0000 mon.a (mon.0) 2535 : audit [INF] from='client.? 172.21.15.162:0/2771150087' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:46:42.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:41.957874+0000 mon.a (mon.0) 2536 : audit [DBG] from='client.? 172.21.15.162:0/2164082683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:42.712 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:42.732 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:42.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:42.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:46:42.799 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:41.590693+0000 mon.a (mon.0) 2535 : audit [INF] from='client.? 172.21.15.162:0/2771150087' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:46:42.799 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:41.957874+0000 mon.a (mon.0) 2536 : audit [DBG] from='client.? 172.21.15.162:0/2164082683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:42.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:41.590693+0000 mon.a (mon.0) 2535 : audit [INF] from='client.? 172.21.15.162:0/2771150087' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:46:42.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:41.957874+0000 mon.a (mon.0) 2536 : audit [DBG] from='client.? 172.21.15.162:0/2164082683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:42.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:42.862 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:43.210 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:43.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:43.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:43.341 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:43.440 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:43.442 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:42.426713+0000 mgr.y (mgr.14544) 375 : cluster [DBG] pgmap v619: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 730 B/s rd, 0 op/s 2024-02-13T00:46:43.442 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:42.436338+0000 mon.a (mon.0) 2537 : audit [INF] from='client.? 172.21.15.162:0/2771150087' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:46:43.442 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:42.436426+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e426: 8 total, 8 up, 4 in 2024-02-13T00:46:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:43.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:43.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:43.572 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:43.609 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:43.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:43.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:42.426713+0000 mgr.y (mgr.14544) 375 : cluster [DBG] pgmap v619: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 730 B/s rd, 0 op/s 2024-02-13T00:46:43.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:42.436338+0000 mon.a (mon.0) 2537 : audit [INF] from='client.? 172.21.15.162:0/2771150087' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:46:43.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:42.436426+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e426: 8 total, 8 up, 4 in 2024-02-13T00:46:43.671 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:42.426713+0000 mgr.y (mgr.14544) 375 : cluster [DBG] pgmap v619: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 730 B/s rd, 0 op/s 2024-02-13T00:46:43.671 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:42.436338+0000 mon.a (mon.0) 2537 : audit [INF] from='client.? 172.21.15.162:0/2771150087' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:46:43.671 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:42.436426+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e426: 8 total, 8 up, 4 in 2024-02-13T00:46:43.698 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-13T00:46:43.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:46:43.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:43.718 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:43.734 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:46:43.735 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:46:43.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:46:43.735 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:46:43.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.735 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24377.0:623 4.9 4:954be8d2:gc::gc.5:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e64)", 2024-02-13T00:46:43.735 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:37:48.456190+0000", 2024-02-13T00:46:43.735 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 535.27368561499998, 2024-02-13T00:46:43.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0037092359999999999, 2024-02-13T00:46:43.735 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.735 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.735 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24377", 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 623 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456190+0000", 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456190+0000", 2024-02-13T00:46:43.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456189+0000", 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6030000000000001e-06 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456196+0000", 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1809e-05 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456207+0000", 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4989999999999999e-06 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456214+0000", 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1881e-05 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456266+0000", 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3955000000000005e-05 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456350+0000", 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013941100000000001 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.456489+0000", 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061411899999999999 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.457103+0000", 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001897262 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.459001+0000", 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00084291699999999999 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.459843+0000", 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5435000000000002e-05 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.459879+0000", 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0389000000000001e-05 2024-02-13T00:46:43.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:48.459899+0000", 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0037092359999999999 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24377.0:944 4.e 4:71a6e5bc:lc::lc.2:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:37:53.331569+0000", 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 530.39830673599999, 2024-02-13T00:46:43.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056003779999999996, 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24377", 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2326056861", 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 944 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331569+0000", 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331569+0000", 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331566+0000", 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0927000000000001e-05 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331577+0000", 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6793e-05 2024-02-13T00:46:43.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331594+0000", 2024-02-13T00:46:43.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1973999999999999e-05 2024-02-13T00:46:43.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331606+0000", 2024-02-13T00:46:43.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8511999999999999e-05 2024-02-13T00:46:43.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331655+0000", 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6979999999999997e-06 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.331659+0000", 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002240658 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.333900+0000", 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103423 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.334003+0000", 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0002767 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.334280+0000", 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027515080000000002 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.337032+0000", 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8433000000000002e-05 2024-02-13T00:46:43.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:46:43.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.337080+0000", 2024-02-13T00:46:43.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4619999999999999e-06 2024-02-13T00:46:43.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:46:43.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.337088+0000", 2024-02-13T00:46:43.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7343999999999999e-05 2024-02-13T00:46:43.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.337135+0000", 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4400000000000003e-05 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:37:53.337169+0000", 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056003779999999996 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24748.0:2 18.6 e107/105 18:602f83fe:::foo:head v 107'1, mlcod=0'0)", 2024-02-13T00:46:43.748 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.642263+0000", 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 342.08761218500001, 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057417939999999997, 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642263+0000", 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642263+0000", 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642263+0000", 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2369999999999998e-06 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642272+0000", 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9379999999999997e-06 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642278+0000", 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2829999999999998e-06 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642285+0000", 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000456229 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642741+0000", 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3309e-05 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.642764+0000", 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0052050890000000004 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.647969+0000", 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6060999999999999e-05 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.648005+0000", 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057417939999999997 2024-02-13T00:46:43.752 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14862.0:4 12.16 e107/104 12:681f27cd:::2.obj:head v 107'1, mlcod=0'0)", 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.655596+0000", 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 342.07427990299999, 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0085331920000000002, 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.655596+0000", 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.655596+0000", 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.655595+0000", 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1340000000000004e-06 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.655603+0000", 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.273e-05 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.655616+0000", 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0099999999999998e-06 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.655623+0000", 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.007108006 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.662731+0000", 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2544999999999999e-05 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.662743+0000", 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001350221 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.664094+0000", 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5250000000000003e-05 2024-02-13T00:46:43.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.664129+0000", 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0085331920000000002 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.2.0:191 20.19 e109/105 -22:9c8ffe76:::temp_20.19_0_14286_1:head v 0'0, mlcod=0'0)", 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.448877+0000", 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 339.28099830299999, 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057776421000000001, 2024-02-13T00:46:43.757 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.448877+0000", 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.448877+0000", 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.448877+0000", 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.026996183 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475873+0000", 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2881e-05 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475896+0000", 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8559999999999996e-06 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475905+0000", 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6680000000000004e-05 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475981+0000", 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0239999999999999e-05 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.476002+0000", 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.030597243999999999 2024-02-13T00:46:43.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.506599+0000", 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4880000000000003e-05 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.506654+0000", 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057776421000000001 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.761 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.2.0:192 20.19 e109/105 -22:9c8ffe76:::temp_20.19_0_14286_1:head v 0'0, mlcod=0'0)", 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.475919+0000", 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 339.25395710700002, 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.046592573999999998, 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475919+0000", 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475919+0000", 2024-02-13T00:46:43.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.475918+0000", 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014380096 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.490298+0000", 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8105e-05 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.490316+0000", 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7569999999999999e-06 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.490325+0000", 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4147000000000002e-05 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.490380+0000", 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8797999999999999e-05 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.490398+0000", 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.03208714 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.522486+0000", 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5590999999999999e-05 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.522511+0000", 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.046592573999999998 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.765 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.2.0:193 20.19 e109/105 -22:9c8ffe76:::temp_20.19_0_14286_1:head v 0'0, mlcod=0'0)", 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.491530+0000", 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 339.23834600999999, 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.033860855000000002, 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.491530+0000", 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.491530+0000", 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.491528+0000", 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.01643648 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507965+0000", 2024-02-13T00:46:43.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2159e-05 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507977+0000", 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1469999999999998e-06 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507983+0000", 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2548000000000001e-05 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.508016+0000", 2024-02-13T00:46:43.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7958999999999999e-05 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.508034+0000", 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017333595 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.525367+0000", 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3408999999999999e-05 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.525391+0000", 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.033860855000000002 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2064 20.19 e109/105 20:9c8ffe76:test-rados-api-smithi162-84049-16::big.copy2:head v 109'1, mlcod=0'0)", 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.507993+0000", 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 339.22188272400001, 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017450688999999998, 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507993+0000", 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507993+0000", 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507992+0000", 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.891e-06 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.507998+0000", 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4490000000000001e-06 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.508002+0000", 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7609999999999999e-06 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.508005+0000", 2024-02-13T00:46:43.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003349201 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.511354+0000", 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1878999999999998e-05 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.511376+0000", 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014054255999999999 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.525430+0000", 2024-02-13T00:46:43.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3195e-05 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.525444+0000", 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017450688999999998 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2069 20.1b e109/105 20:de4afda3:test-rados-api-smithi162-84049-17::big:head v 109'2, mlcod=0'0)", 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.641911+0000", 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 339.08796462499998, 2024-02-13T00:46:43.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099432847000000005, 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.641911+0000", 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.641911+0000", 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.641910+0000", 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.044861537 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.686771+0000", 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9842e-05 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.686801+0000", 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8167e-05 2024-02-13T00:46:43.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.686829+0000", 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5619999999999999e-05 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.686885+0000", 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7118999999999999e-05 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.686912+0000", 2024-02-13T00:46:43.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.054351272999999999 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.741263+0000", 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0814000000000002e-05 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.741344+0000", 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099432847000000005 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2070 20.b e109/105 20:d3a22371:test-rados-api-smithi162-84049-17::big2:head v 109'1, mlcod=0'0)", 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.754348+0000", 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 338.975527618, 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088186930999999996, 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754348+0000", 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754348+0000", 2024-02-13T00:46:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:46:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754347+0000", 2024-02-13T00:46:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.034034943999999998 2024-02-13T00:46:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.788381+0000", 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3618e-05 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.788405+0000", 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4919999999999993e-06 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.788414+0000", 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6088000000000003e-05 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.788460+0000", 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9964999999999999e-05 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.788480+0000", 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.053965705000000003 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842445+0000", 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9630999999999996e-05 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842535+0000", 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088186930999999996 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2073 20.b e109/105 20:d3a22371:test-rados-api-smithi162-84049-17::big2:head v 109'4, mlcod=109'2)", 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.849406+0000", 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 338.880470113, 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.01207979, 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.783 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.849406+0000", 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.849406+0000", 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.849404+0000", 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047755940000000002 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854179+0000", 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4504999999999999e-05 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854194+0000", 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9820000000000002e-06 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854201+0000", 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9499000000000002e-05 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854240+0000", 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7659000000000002e-05 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.854258+0000", 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071836679999999998 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.861442+0000", 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3723000000000001e-05 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.861485+0000", 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.01207979 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2074 20.1b e109/105 20:de8a7420:test-rados-api-smithi162-84049-17::big3:head v 109'3, mlcod=107'1)", 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.870270+0000", 2024-02-13T00:46:43.787 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 338.85960617199999, 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094008466999999998, 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.870270+0000", 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.870270+0000", 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999938 2024-02-13T00:46:43.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.870263+0000", 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049442282999999997 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.919706+0000", 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8632e-05 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.919734+0000", 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8073e-05 2024-02-13T00:46:43.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.919762+0000", 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5293000000000001e-05 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.919818+0000", 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1254e-05 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.919839+0000", 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.044407744999999998 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.964247+0000", 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1446999999999999e-05 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.964278+0000", 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094008466999999998 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.791 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2076 20.1b e109/105 20:de8a7420:test-rados-api-smithi162-84049-17::big3:head v 109'5, mlcod=109'3)", 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.970516+0000", 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 338.75936004800002, 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0122755, 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.970516+0000", 2024-02-13T00:46:43.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.970516+0000", 2024-02-13T00:46:43.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:46:43.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.970514+0000", 2024-02-13T00:46:43.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048866400000000003 2024-02-13T00:46:43.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.975401+0000", 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3018e-05 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.975414+0000", 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.849e-06 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.975420+0000", 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3612000000000001e-05 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.975474+0000", 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3119e-05 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.975497+0000", 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0072508829999999996 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.982748+0000", 2024-02-13T00:46:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3099000000000002e-05 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.982791+0000", 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0122755 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.5.0:282 60.a e128/120 60:51f054f9:test-rados-api-smithi162-84073-8::foo:7 v 128'6, mlcod=127'3)", 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:22.774235+0000", 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 320.95564102999998, 2024-02-13T00:46:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.007417252, 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774235+0000", 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774235+0000", 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774234+0000", 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4309999999999996e-06 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774239+0000", 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3290000000000001e-06 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774244+0000", 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8940000000000003e-06 2024-02-13T00:46:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774248+0000", 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000423449 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774671+0000", 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7167e-05 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774689+0000", 2024-02-13T00:46:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0069276140000000003 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.781616+0000", 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5657000000000001e-05 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.781652+0000", 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.007417252 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.16275.0:11 66.6 e130/124 66:629bc174:test-rados-api-smithi162-84081-9::foo:head v 130'7, mlcod=128'5)", 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:24.903032+0000", 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 318.826843752, 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041090170000000004, 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903032+0000", 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903032+0000", 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903031+0000", 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3899999999999999e-06 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903040+0000", 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0533999999999999e-05 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903051+0000", 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0879999999999997e-06 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903058+0000", 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6895000000000002e-05 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903135+0000", 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2158999999999999e-05 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903157+0000", 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039450989999999997 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.907102+0000", 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9041000000000001e-05 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.907141+0000", 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041090170000000004 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.5.0:293 66.6 e132/124 66:629bc174:test-rados-api-smithi162-84081-9::foo:7 v 132'15, mlcod=131'12)", 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:27.091552+0000", 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 316.63832375499999, 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084929270000000008, 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091552+0000", 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091552+0000", 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091550+0000", 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1979e-05 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091562+0000", 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2990000000000002e-06 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091572+0000", 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1720000000000001e-06 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091579+0000", 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000886285 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.092465+0000", 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2421000000000001e-05 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.092488+0000", 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075259120000000001 2024-02-13T00:46:43.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.100013+0000", 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1429e-05 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.100045+0000", 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084929270000000008 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.809 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.2.0:365 61.4 e188/121 61:21c2c914:test-rados-api-smithi162-84057-14::foo8:7 v 188'36, mlcod=154'30)", 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:42:23.329308+0000", 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 260.400568142, 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0089267670000000004, 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329308+0000", 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329308+0000", 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329307+0000", 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3189999999999996e-06 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329312+0000", 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.648e-06 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329317+0000", 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3749999999999996e-06 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329322+0000", 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00029372900000000001 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329615+0000", 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7045999999999999e-05 2024-02-13T00:46:43.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329632+0000", 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0085733790000000008 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.338206+0000", 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8561e-05 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.338234+0000", 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0089267670000000004 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.813 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.5.0:512 61.19 e188/121 61:9a2cc200:test-rados-api-smithi162-84057-14::foo19:7 v 188'11, mlcod=153'7)", 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:42:23.330027+0000", 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 260.39984847599999, 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073812160000000003, 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330027+0000", 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330027+0000", 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330026+0000", 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5390000000000002e-06 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330034+0000", 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0818999999999999e-05 2024-02-13T00:46:43.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330045+0000", 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1320000000000002e-06 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330050+0000", 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000152163 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330202+0000", 2024-02-13T00:46:43.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9652000000000001e-05 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330222+0000", 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071553379999999998 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.337377+0000", 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1399999999999998e-05 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.337408+0000", 2024-02-13T00:46:43.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073812160000000003 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.25162.0:3128 184.0 e300/296 184:00adb854:test-rados-api-smithi162-84057-38::foo:head v 300'2, mlcod=0'0)", 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:15.383766+0000", 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 148.34610946199999, 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013839536, 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.383766+0000", 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.383766+0000", 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.383765+0000", 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002561158 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386326+0000", 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7059e-05 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386344+0000", 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.259e-06 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386352+0000", 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014103499999999999 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386493+0000", 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2059999999999999e-05 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386515+0000", 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011046402 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.397561+0000", 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4487000000000003e-05 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.397606+0000", 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013839536 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(28.c REQUEST e379)", 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:45:34.516696+0000", 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 69.213179589999996, 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003757631, 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:46:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516696+0000", 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516696+0000", 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516695+0000", 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1699999999999997e-06 2024-02-13T00:46:43.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516703+0000", 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0827000000000001e-05 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516734+0000", 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5639000000000001e-05 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516750+0000", 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011122500000000001 2024-02-13T00:46:43.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516861+0000", 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6915999999999999e-05 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516878+0000", 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035454750000000002 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.520424+0000", 2024-02-13T00:46:43.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0348e-05 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.520434+0000", 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9860999999999998e-05 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.520454+0000", 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003757631 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:46:43.826 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:46:43.827 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:46:43.827 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:43.954 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:43.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:44.058 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:44.085 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:44.158 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:44.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:44.262 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:44.289 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:44.414 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-13T00:46:44.482 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:44.487 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:44.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:44.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:44.619 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:44.679 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:46:44.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:44.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:44.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:44.873 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:44.944 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:44.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:44.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:45.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:46:45.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:45.102 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:45.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:44.427239+0000 mgr.y (mgr.14544) 376 : cluster [DBG] pgmap v621: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:46:45.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:45.421986+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e427: 8 total, 8 up, 4 in 2024-02-13T00:46:45.777 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:44.427239+0000 mgr.y (mgr.14544) 376 : cluster [DBG] pgmap v621: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:46:45.777 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:45.421986+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e427: 8 total, 8 up, 4 in 2024-02-13T00:46:45.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:44.427239+0000 mgr.y (mgr.14544) 376 : cluster [DBG] pgmap v621: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:46:45.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:45.421986+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e427: 8 total, 8 up, 4 in 2024-02-13T00:46:45.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:46:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:46:46.051 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:46.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:46.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:46.155 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:46.183 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:46.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:46:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:46:46] "GET /metrics HTTP/1.1" 200 187716 "" "Prometheus/2.33.4" 2024-02-13T00:46:46.330 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:46.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:46.435 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:46.441 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:46:46.441 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:46:46.441 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:46:46.441 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:46:46.465 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:46.484 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:46.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:46.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:46:46.618 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:46.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.617+0000 7f2610b6b700 1 -- 172.21.15.162:0/3049172594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f260c0fcb30 msgr2=0x7f260c0fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:46.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.617+0000 7f2610b6b700 1 --2- 172.21.15.162:0/3049172594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f260c0fcb30 0x7f260c0fcf10 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 rx=0x7f25fc004870 tx=0x7f25fc009cb0).stop 2024-02-13T00:46:46.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.617+0000 7f2610b6b700 1 -- 172.21.15.162:0/3049172594 shutdown_connections 2024-02-13T00:46:46.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.617+0000 7f2610b6b700 1 --2- 172.21.15.162:0/3049172594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f260c0f73a0 0x7f260c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:46.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.617+0000 7f2610b6b700 1 --2- 172.21.15.162:0/3049172594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f260c0fcb30 0x7f260c0fcf10 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:46.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.617+0000 7f2610b6b700 1 --2- 172.21.15.162:0/3049172594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f260c0fd490 0x7f260c0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:46.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.617+0000 7f2610b6b700 1 -- 172.21.15.162:0/3049172594 >> 172.21.15.162:0/3049172594 conn(0x7f260c06eb90 msgr2=0x7f260c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:46.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.617+0000 7f2610b6b700 1 -- 172.21.15.162:0/3049172594 shutdown_connections 2024-02-13T00:46:46.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.617+0000 7f2610b6b700 1 -- 172.21.15.162:0/3049172594 wait complete. 2024-02-13T00:46:46.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.618+0000 7f2610b6b700 1 librados: starting msgr at 2024-02-13T00:46:46.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.618+0000 7f2610b6b700 1 librados: starting objecter 2024-02-13T00:46:46.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.618+0000 7f2610b6b700 1 Processor -- start 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.618+0000 7f2610b6b700 1 -- start start 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.618+0000 7f2610b6b700 1 librados: setting wanted keys 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.618+0000 7f2610b6b700 1 librados: calling monclient init 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.618+0000 7f2610b6b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f260c0f73a0 0x7f260c1091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.618+0000 7f2610b6b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f260c0fcb30 0x7f260c1096d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.618+0000 7f2610b6b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f260c0fd490 0x7f260c109d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f2610b6b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c114410 con 0x7f260c0fcb30 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f2610b6b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c114580 con 0x7f260c0fd490 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f2610b6b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260c114740 con 0x7f260c0f73a0 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f2609d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f260c0fcb30 0x7f260c1096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f2609d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f260c0fcb30 0x7f260c1096d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38040/0 (socket says 172.21.15.162:38040) 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f2609d9b700 1 -- 172.21.15.162:0/585817790 learned_addr learned my addr 172.21.15.162:0/585817790 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:46.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f260ad9d700 1 --2- 172.21.15.162:0/585817790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f260c0fd490 0x7f260c109d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:46.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f260a59c700 1 --2- 172.21.15.162:0/585817790 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f260c0f73a0 0x7f260c1091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:46.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f260a59c700 1 -- 172.21.15.162:0/585817790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f260c0fd490 msgr2=0x7f260c109d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:46.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f260a59c700 1 --2- 172.21.15.162:0/585817790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f260c0fd490 0x7f260c109d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:46.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f260a59c700 1 -- 172.21.15.162:0/585817790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f260c0fcb30 msgr2=0x7f260c1096d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:46.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f260a59c700 1 --2- 172.21.15.162:0/585817790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f260c0fcb30 0x7f260c1096d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:46.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f260a59c700 1 -- 172.21.15.162:0/585817790 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc005040 con 0x7f260c0f73a0 2024-02-13T00:46:46.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f260ad9d700 1 --2- 172.21.15.162:0/585817790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f260c0fd490 0x7f260c109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:46.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f260a59c700 1 --2- 172.21.15.162:0/585817790 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f260c0f73a0 0x7f260c1091e0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 rx=0x7f25fc004870 tx=0x7f25fc009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:46.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f25fb7fe700 10 client.?.objecter ms_handle_connect 0x7f260c0f73a0 2024-02-13T00:46:46.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f25fb7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:46.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f25fb7fe700 1 -- 172.21.15.162:0/585817790 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc00e030 con 0x7f260c0f73a0 2024-02-13T00:46:46.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f2610b6b700 1 -- 172.21.15.162:0/585817790 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f260c1149c0 con 0x7f260c0f73a0 2024-02-13T00:46:46.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f25fb7fe700 1 -- 172.21.15.162:0/585817790 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25fc009710 con 0x7f260c0f73a0 2024-02-13T00:46:46.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f25fb7fe700 1 -- 172.21.15.162:0/585817790 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc018070 con 0x7f260c0f73a0 2024-02-13T00:46:46.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f2610b6b700 10 client.37463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:46.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f2610b6b700 1 -- 172.21.15.162:0/585817790 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f260c114ea0 con 0x7f260c0f73a0 2024-02-13T00:46:46.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.619+0000 7f2610b6b700 1 librados: init done 2024-02-13T00:46:46.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.620+0000 7f2610b6b700 1 -- 172.21.15.162:0/585817790 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f260c106030 con 0x7f260c0f73a0 2024-02-13T00:46:46.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.621+0000 7f25fb7fe700 1 -- 172.21.15.162:0/585817790 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f25fc0090d0 con 0x7f260c0f73a0 2024-02-13T00:46:46.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.621+0000 7f25fb7fe700 1 --2- 172.21.15.162:0/585817790 >> v2:172.21.15.162:6800/2866534053 conn(0x7f25f4062c90 0x7f25f4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:46.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.621+0000 7f25fb7fe700 1 -- 172.21.15.162:0/585817790 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(428..428 src has 52..428) v4 ==== 7582+0+0 (secure 0 0 0) 0x7f25fc07e0d0 con 0x7f260c0f73a0 2024-02-13T00:46:46.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.621+0000 7f25fb7fe700 10 client.37463.objecter ms_dispatch 0x7f260c001550 osd_map(428..428 src has 52..428) v4 2024-02-13T00:46:46.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.621+0000 7f25fb7fe700 3 client.37463.objecter handle_osd_map got epochs [428,428] > 0 2024-02-13T00:46:46.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.621+0000 7f25fb7fe700 3 client.37463.objecter handle_osd_map decoding full epoch 428 2024-02-13T00:46:46.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.621+0000 7f2609d9b700 1 --2- 172.21.15.162:0/585817790 >> v2:172.21.15.162:6800/2866534053 conn(0x7f25f4062c90 0x7f25f4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:46.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.621+0000 7f25fb7fe700 20 client.37463.objecter dump_active .. 0 homeless 2024-02-13T00:46:46.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.621+0000 7f25fb7fe700 10 client.37463.objecter ms_handle_connect 0x7f25f4062c90 2024-02-13T00:46:46.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.621+0000 7f2609d9b700 1 --2- 172.21.15.162:0/585817790 >> v2:172.21.15.162:6800/2866534053 conn(0x7f25f4062c90 0x7f25f4065150 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 rx=0x7f260c061810 tx=0x7f260000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:46.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.626+0000 7f25fb7fe700 1 -- 172.21.15.162:0/585817790 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f25fc0517c0 con 0x7f260c0f73a0 2024-02-13T00:46:46.715 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:46.716 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:45.444006+0000 mon.b (mon.1) 576 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:46.716 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:45.444354+0000 mon.a (mon.0) 2540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:46.716 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:45.444697+0000 mon.b (mon.1) 577 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-61"}]: dispatch 2024-02-13T00:46:46.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:45.444980+0000 mon.a (mon.0) 2541 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-61"}]: dispatch 2024-02-13T00:46:46.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:45.632579+0000 mgr.y (mgr.14544) 377 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:46.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:46.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:46:46.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.792+0000 7f2610b6b700 1 -- 172.21.15.162:0/585817790 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f260c05fc90 con 0x7f25f4062c90 2024-02-13T00:46:46.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.806+0000 7f25fb7fe700 1 -- 172.21.15.162:0/585817790 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+510749 (secure 0 0 0) 0x7f260c05fc90 con 0x7f25f4062c90 2024-02-13T00:46:46.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:46.821 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:46:46.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.810+0000 7f2610b6b700 10 librados: watch_flush enter 2024-02-13T00:46:46.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.810+0000 7f2610b6b700 10 librados: watch_flush exit 2024-02-13T00:46:46.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.810+0000 7f2610b6b700 20 client.37463.objecter shutdown clearing up homeless session... 2024-02-13T00:46:46.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.810+0000 7f2610b6b700 10 client.37463.objecter successfully canceled tick 2024-02-13T00:46:46.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.810+0000 7f2610b6b700 1 -- 172.21.15.162:0/585817790 >> v2:172.21.15.162:6800/2866534053 conn(0x7f25f4062c90 msgr2=0x7f25f4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:46.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.810+0000 7f2610b6b700 1 --2- 172.21.15.162:0/585817790 >> v2:172.21.15.162:6800/2866534053 conn(0x7f25f4062c90 0x7f25f4065150 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 rx=0x7f260c061810 tx=0x7f260000b040).stop 2024-02-13T00:46:46.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.811+0000 7f2610b6b700 1 -- 172.21.15.162:0/585817790 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f260c0f73a0 msgr2=0x7f260c1091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:46.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.811+0000 7f2610b6b700 1 --2- 172.21.15.162:0/585817790 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f260c0f73a0 0x7f260c1091e0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 rx=0x7f25fc004870 tx=0x7f25fc009cb0).stop 2024-02-13T00:46:46.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.811+0000 7f2610b6b700 1 -- 172.21.15.162:0/585817790 shutdown_connections 2024-02-13T00:46:46.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.811+0000 7f2610b6b700 1 --2- 172.21.15.162:0/585817790 >> v2:172.21.15.162:6800/2866534053 conn(0x7f25f4062c90 0x7f25f4065150 unknown :-1 s=CLOSED pgs=2906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:46.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.811+0000 7f2610b6b700 1 --2- 172.21.15.162:0/585817790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f260c0fd490 0x7f260c109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:46.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.811+0000 7f2610b6b700 1 --2- 172.21.15.162:0/585817790 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f260c0f73a0 0x7f260c1091e0 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:46.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.811+0000 7f2610b6b700 1 --2- 172.21.15.162:0/585817790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f260c0fcb30 0x7f260c1096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:46.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.811+0000 7f2610b6b700 1 -- 172.21.15.162:0/585817790 >> 172.21.15.162:0/585817790 conn(0x7f260c06eb90 msgr2=0x7f260c0fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:46.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.811+0000 7f2610b6b700 1 -- 172.21.15.162:0/585817790 shutdown_connections 2024-02-13T00:46:46.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.811+0000 7f2610b6b700 1 -- 172.21.15.162:0/585817790 wait complete. 2024-02-13T00:46:46.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:46.811+0000 7f2610b6b700 1 librados: shutdown 2024-02-13T00:46:46.829 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:46:46.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:45.444006+0000 mon.b (mon.1) 576 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:46.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:45.444354+0000 mon.a (mon.0) 2540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:45.444697+0000 mon.b (mon.1) 577 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-61"}]: dispatch 2024-02-13T00:46:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:45.444980+0000 mon.a (mon.0) 2541 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-61"}]: dispatch 2024-02-13T00:46:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:45.632579+0000 mgr.y (mgr.14544) 377 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:46.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:45.444006+0000 mon.b (mon.1) 576 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:46.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:45.444354+0000 mon.a (mon.0) 2540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:46:46.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:45.444697+0000 mon.b (mon.1) 577 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-61"}]: dispatch 2024-02-13T00:46:46.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:45.444980+0000 mon.a (mon.0) 2541 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-61"}]: dispatch 2024-02-13T00:46:46.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:45.632579+0000 mgr.y (mgr.14544) 377 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:46.898 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:47.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-13T00:46:47.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.293+0000 7f42cf93b700 1 -- 172.21.15.162:0/4294103334 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42c80f9b70 msgr2=0x7f42c80f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:47.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.293+0000 7f42cf93b700 1 --2- 172.21.15.162:0/4294103334 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42c80f9b70 0x7f42c80f9f50 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 rx=0x7f42b80047b0 tx=0x7f42b8009cb0).stop 2024-02-13T00:46:47.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.293+0000 7f42cf93b700 1 -- 172.21.15.162:0/4294103334 shutdown_connections 2024-02-13T00:46:47.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.293+0000 7f42cf93b700 1 --2- 172.21.15.162:0/4294103334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c80f8500 0x7f42c8104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:47.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.293+0000 7f42cf93b700 1 --2- 172.21.15.162:0/4294103334 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42c80f9b70 0x7f42c80f9f50 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:47.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.293+0000 7f42cf93b700 1 --2- 172.21.15.162:0/4294103334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42c80fa440 0x7f42c80f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:47.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.293+0000 7f42cf93b700 1 -- 172.21.15.162:0/4294103334 >> 172.21.15.162:0/4294103334 conn(0x7f42c80f38c0 msgr2=0x7f42c80f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:47.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.293+0000 7f42cf93b700 1 -- 172.21.15.162:0/4294103334 shutdown_connections 2024-02-13T00:46:47.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.293+0000 7f42cf93b700 1 -- 172.21.15.162:0/4294103334 wait complete. 2024-02-13T00:46:47.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.294+0000 7f42cf93b700 1 librados: starting msgr at 2024-02-13T00:46:47.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.294+0000 7f42cf93b700 1 librados: starting objecter 2024-02-13T00:46:47.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.294+0000 7f42cf93b700 1 Processor -- start 2024-02-13T00:46:47.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.294+0000 7f42cf93b700 1 -- start start 2024-02-13T00:46:47.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.294+0000 7f42cf93b700 1 librados: setting wanted keys 2024-02-13T00:46:47.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.294+0000 7f42cf93b700 1 librados: calling monclient init 2024-02-13T00:46:47.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cf93b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c80f8500 0x7f42c806b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:47.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cf93b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42c80f9b70 0x7f42c806b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:47.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cf93b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42c80fa440 0x7f42c806bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:47.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cf93b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c81166a0 con 0x7f42c80f8500 2024-02-13T00:46:47.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cf93b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c8116810 con 0x7f42c80fa440 2024-02-13T00:46:47.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cf93b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c81169d0 con 0x7f42c80f9b70 2024-02-13T00:46:47.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cd6d7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c80f8500 0x7f42c806b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cd6d7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c80f8500 0x7f42c806b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38062/0 (socket says 172.21.15.162:38062) 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cd6d7700 1 -- 172.21.15.162:0/2488015535 learned_addr learned my addr 172.21.15.162:0/2488015535 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cced6700 1 --2- 172.21.15.162:0/2488015535 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42c80f9b70 0x7f42c806b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cded8700 1 --2- 172.21.15.162:0/2488015535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42c80fa440 0x7f42c806bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cd6d7700 1 -- 172.21.15.162:0/2488015535 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42c80f9b70 msgr2=0x7f42c806b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cd6d7700 1 --2- 172.21.15.162:0/2488015535 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42c80f9b70 0x7f42c806b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cd6d7700 1 -- 172.21.15.162:0/2488015535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42c80fa440 msgr2=0x7f42c806bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cd6d7700 1 --2- 172.21.15.162:0/2488015535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42c80fa440 0x7f42c806bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cd6d7700 1 -- 172.21.15.162:0/2488015535 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c00097c0 con 0x7f42c80f8500 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cded8700 1 --2- 172.21.15.162:0/2488015535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42c80fa440 0x7f42c806bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cd6d7700 1 --2- 172.21.15.162:0/2488015535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c80f8500 0x7f42c806b170 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7f42b80044e0 tx=0x7f42b8007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42be7fc700 10 client.?.objecter ms_handle_connect 0x7f42c80f8500 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42be7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:47.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42be7fc700 1 -- 172.21.15.162:0/2488015535 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8018070 con 0x7f42c80f8500 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cf93b700 1 -- 172.21.15.162:0/2488015535 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b8005040 con 0x7f42c80f8500 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42be7fc700 1 -- 172.21.15.162:0/2488015535 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f42b80093b0 con 0x7f42c80f8500 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42be7fc700 1 -- 172.21.15.162:0/2488015535 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b801de40 con 0x7f42c80f8500 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cf93b700 10 client.27369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cf93b700 1 -- 172.21.15.162:0/2488015535 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c8116ec0 con 0x7f42c80f8500 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.295+0000 7f42cf93b700 1 librados: init done 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.296+0000 7f42cf93b700 1 -- 172.21.15.162:0/2488015535 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c80475e0 con 0x7f42c80f8500 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.297+0000 7f42be7fc700 1 -- 172.21.15.162:0/2488015535 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f42b8013ad0 con 0x7f42c80f8500 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.297+0000 7f42be7fc700 1 --2- 172.21.15.162:0/2488015535 >> v2:172.21.15.162:6800/2866534053 conn(0x7f42b4062c90 0x7f42b4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.297+0000 7f42be7fc700 1 -- 172.21.15.162:0/2488015535 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(428..428 src has 52..428) v4 ==== 7582+0+0 (secure 0 0 0) 0x7f42b807e520 con 0x7f42c80f8500 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.297+0000 7f42be7fc700 10 client.27369.objecter ms_dispatch 0x7f42c8001550 osd_map(428..428 src has 52..428) v4 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.297+0000 7f42be7fc700 3 client.27369.objecter handle_osd_map got epochs [428,428] > 0 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.297+0000 7f42be7fc700 3 client.27369.objecter handle_osd_map decoding full epoch 428 2024-02-13T00:46:47.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.297+0000 7f42cced6700 1 --2- 172.21.15.162:0/2488015535 >> v2:172.21.15.162:6800/2866534053 conn(0x7f42b4062c90 0x7f42b4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:47.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.297+0000 7f42be7fc700 20 client.27369.objecter dump_active .. 0 homeless 2024-02-13T00:46:47.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.298+0000 7f42be7fc700 10 client.27369.objecter ms_handle_connect 0x7f42b4062c90 2024-02-13T00:46:47.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.298+0000 7f42cced6700 1 --2- 172.21.15.162:0/2488015535 >> v2:172.21.15.162:6800/2866534053 conn(0x7f42b4062c90 0x7f42b4065150 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 rx=0x7f42c806c910 tx=0x7f42c400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:47.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.302+0000 7f42be7fc700 1 -- 172.21.15.162:0/2488015535 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f42b8051bb0 con 0x7f42c80f8500 2024-02-13T00:46:47.322 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:47.400 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:46:47.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:47.493+0000 7f42cf93b700 1 -- 172.21.15.162:0/2488015535 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f42c806c650 con 0x7f42c80f8500 2024-02-13T00:46:47.697 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:46.427927+0000 mgr.y (mgr.14544) 378 : cluster [DBG] pgmap v623: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 852 B/s rd, 0 op/s 2024-02-13T00:46:47.697 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:46.444728+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e428: 8 total, 8 up, 4 in 2024-02-13T00:46:47.698 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:46.794677+0000 mgr.y (mgr.14544) 379 : audit [DBG] from='client.37463 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:46:47.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:46.427927+0000 mgr.y (mgr.14544) 378 : cluster [DBG] pgmap v623: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 852 B/s rd, 0 op/s 2024-02-13T00:46:47.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:46.444728+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e428: 8 total, 8 up, 4 in 2024-02-13T00:46:47.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:46.794677+0000 mgr.y (mgr.14544) 379 : audit [DBG] from='client.37463 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:46:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:46.427927+0000 mgr.y (mgr.14544) 378 : cluster [DBG] pgmap v623: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 852 B/s rd, 0 op/s 2024-02-13T00:46:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:46.444728+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e428: 8 total, 8 up, 4 in 2024-02-13T00:46:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:46.794677+0000 mgr.y (mgr.14544) 379 : audit [DBG] from='client.37463 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:46:48.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.454+0000 7f42be7fc700 1 -- 172.21.15.162:0/2488015535 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v430) v1 ==== 97+0+0 (secure 0 0 0) 0x7f42b80517a0 con 0x7f42c80f8500 2024-02-13T00:46:48.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.458+0000 7f42cf93b700 10 librados: watch_flush enter 2024-02-13T00:46:48.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.460+0000 7f42cf93b700 10 librados: watch_flush exit 2024-02-13T00:46:48.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.460+0000 7f42cf93b700 20 client.27369.objecter shutdown clearing up homeless session... 2024-02-13T00:46:48.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.460+0000 7f42cf93b700 10 client.27369.objecter successfully canceled tick 2024-02-13T00:46:48.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.460+0000 7f42cf93b700 1 -- 172.21.15.162:0/2488015535 >> v2:172.21.15.162:6800/2866534053 conn(0x7f42b4062c90 msgr2=0x7f42b4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:48.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.460+0000 7f42cf93b700 1 --2- 172.21.15.162:0/2488015535 >> v2:172.21.15.162:6800/2866534053 conn(0x7f42b4062c90 0x7f42b4065150 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 rx=0x7f42c806c910 tx=0x7f42c400b040).stop 2024-02-13T00:46:48.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.460+0000 7f42cf93b700 1 -- 172.21.15.162:0/2488015535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c80f8500 msgr2=0x7f42c806b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:48.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.460+0000 7f42cf93b700 1 --2- 172.21.15.162:0/2488015535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c80f8500 0x7f42c806b170 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7f42b80044e0 tx=0x7f42b8007f90).stop 2024-02-13T00:46:48.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.467+0000 7f42cf93b700 1 -- 172.21.15.162:0/2488015535 shutdown_connections 2024-02-13T00:46:48.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.467+0000 7f42cf93b700 1 --2- 172.21.15.162:0/2488015535 >> v2:172.21.15.162:6800/2866534053 conn(0x7f42b4062c90 0x7f42b4065150 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:48.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.468+0000 7f42cf93b700 1 --2- 172.21.15.162:0/2488015535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42c80fa440 0x7f42c806bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:48.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.468+0000 7f42cf93b700 1 --2- 172.21.15.162:0/2488015535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42c80f8500 0x7f42c806b170 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:48.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.468+0000 7f42cf93b700 1 --2- 172.21.15.162:0/2488015535 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42c80f9b70 0x7f42c806b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:48.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.468+0000 7f42cf93b700 1 -- 172.21.15.162:0/2488015535 >> 172.21.15.162:0/2488015535 conn(0x7f42c80f38c0 msgr2=0x7f42c80fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:48.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.469+0000 7f42cf93b700 1 -- 172.21.15.162:0/2488015535 shutdown_connections 2024-02-13T00:46:48.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.469+0000 7f42cf93b700 1 -- 172.21.15.162:0/2488015535 wait complete. 2024-02-13T00:46:48.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:48.469+0000 7f42cf93b700 1 librados: shutdown 2024-02-13T00:46:48.474 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:46:48.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:47.449837+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e429: 8 total, 8 up, 4 in 2024-02-13T00:46:48.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:47.451094+0000 mon.b (mon.1) 578 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:48.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:47.451672+0000 mon.a (mon.0) 2544 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:48.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:47.495567+0000 mon.a (mon.0) 2545 : audit [INF] from='client.? 172.21.15.162:0/2488015535' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:48.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:47.449837+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e429: 8 total, 8 up, 4 in 2024-02-13T00:46:48.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:47.451094+0000 mon.b (mon.1) 578 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:48.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:47.451672+0000 mon.a (mon.0) 2544 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:48.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:47.495567+0000 mon.a (mon.0) 2545 : audit [INF] from='client.? 172.21.15.162:0/2488015535' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:48.600 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:47.449837+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e429: 8 total, 8 up, 4 in 2024-02-13T00:46:48.600 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:47.451094+0000 mon.b (mon.1) 578 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:48.600 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:47.451672+0000 mon.a (mon.0) 2544 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:46:48.600 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:47.495567+0000 mon.a (mon.0) 2545 : audit [INF] from='client.? 172.21.15.162:0/2488015535' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:46:48.819 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:46:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:46:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:46:49.085 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:46:49.086 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:46:49.086 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-02-13T00:46:49.086 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-02-13T00:46:49.086 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:46:49.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.298+0000 7fab930c7700 1 -- 172.21.15.162:0/3470629697 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab8c0f8490 msgr2=0x7fab8c104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.298+0000 7fab930c7700 1 --2- 172.21.15.162:0/3470629697 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab8c0f8490 0x7fab8c104a90 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 rx=0x7fab7c004870 tx=0x7fab7c009cb0).stop 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.298+0000 7fab930c7700 1 -- 172.21.15.162:0/3470629697 shutdown_connections 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.298+0000 7fab930c7700 1 --2- 172.21.15.162:0/3470629697 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab8c0f8490 0x7fab8c104a90 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.298+0000 7fab930c7700 1 --2- 172.21.15.162:0/3470629697 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab8c0fdc20 0x7fab8c0fe000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.298+0000 7fab930c7700 1 --2- 172.21.15.162:0/3470629697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8c0fe580 0x7fab8c0f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.298+0000 7fab930c7700 1 -- 172.21.15.162:0/3470629697 >> 172.21.15.162:0/3470629697 conn(0x7fab8c0f3850 msgr2=0x7fab8c0f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.298+0000 7fab930c7700 1 -- 172.21.15.162:0/3470629697 shutdown_connections 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.298+0000 7fab930c7700 1 -- 172.21.15.162:0/3470629697 wait complete. 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.298+0000 7fab930c7700 1 librados: starting msgr at 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.298+0000 7fab930c7700 1 librados: starting objecter 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.298+0000 7fab930c7700 1 Processor -- start 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab930c7700 1 -- start start 2024-02-13T00:46:49.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab930c7700 1 librados: setting wanted keys 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab930c7700 1 librados: calling monclient init 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab930c7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab8c0f8490 0x7fab8c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab930c7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8c0fdc20 0x7fab8c06b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab930c7700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab8c0fe580 0x7fab8c06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab930c7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c1163a0 con 0x7fab8c0f8490 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab930c7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c116510 con 0x7fab8c0fdc20 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab930c7700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c1166d0 con 0x7fab8c0fe580 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab90e63700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab8c0f8490 0x7fab8c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab91664700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab8c0fe580 0x7fab8c06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab90e63700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab8c0f8490 0x7fab8c06b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38074/0 (socket says 172.21.15.162:38074) 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab90e63700 1 -- 172.21.15.162:0/328612703 learned_addr learned my addr 172.21.15.162:0/328612703 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab8bfff700 1 --2- 172.21.15.162:0/328612703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8c0fdc20 0x7fab8c06b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:49.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab90e63700 1 -- 172.21.15.162:0/328612703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab8c0fe580 msgr2=0x7fab8c06bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab90e63700 1 --2- 172.21.15.162:0/328612703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab8c0fe580 0x7fab8c06bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab90e63700 1 -- 172.21.15.162:0/328612703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8c0fdc20 msgr2=0x7fab8c06b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab90e63700 1 --2- 172.21.15.162:0/328612703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8c0fdc20 0x7fab8c06b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab90e63700 1 -- 172.21.15.162:0/328612703 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c005040 con 0x7fab8c0f8490 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab91664700 1 --2- 172.21.15.162:0/328612703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab8c0fe580 0x7fab8c06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab90e63700 1 --2- 172.21.15.162:0/328612703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab8c0f8490 0x7fab8c06b170 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 rx=0x7fab8400eca0 tx=0x7fab8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab89ffb700 10 client.?.objecter ms_handle_connect 0x7fab8c0f8490 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab89ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84009930 con 0x7fab8c0f8490 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab8400f040 con 0x7fab8c0f8490 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8c1169b0 con 0x7fab8c0f8490 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.299+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8400c8f0 con 0x7fab8c0f8490 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.300+0000 7fab930c7700 10 client.27375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:49.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.300+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8c116f70 con 0x7fab8c0f8490 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.300+0000 7fab930c7700 1 librados: init done 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.300+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fab8c0607d0 con 0x7fab8c0f8490 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.301+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fab8400cd20 con 0x7fab8c0f8490 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.301+0000 7fab89ffb700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab74062c90 0x7fab74065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.301+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(430..430 src has 52..430) v4 ==== 7967+0+0 (secure 0 0 0) 0x7fab8407cf40 con 0x7fab8c0f8490 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.301+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 osd_map(430..430 src has 52..430) v4 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.301+0000 7fab89ffb700 3 client.27375.objecter handle_osd_map got epochs [430,430] > 0 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.301+0000 7fab89ffb700 3 client.27375.objecter handle_osd_map decoding full epoch 430 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.301+0000 7fab8bfff700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab74062c90 0x7fab74065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.301+0000 7fab89ffb700 20 client.27375.objecter dump_active .. 0 homeless 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.301+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v430) v1 ==== 54+0+16 (secure 0 0 0) 0x7fab8407f550 con 0x7fab8c0f8490 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab89ffb700 10 client.27375.objecter ms_handle_connect 0x7fab74062c90 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab8bfff700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab74062c90 0x7fab74065150 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 rx=0x7fab8c06c910 tx=0x7fab8000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:49.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab930c7700 10 client.27375.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 1 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6801/2103743338 conn(0x7fab8c0feef0 0x7fab8c0ff310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c0475e0 osd=0 3 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c0475e0 osd=0 3 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c0475e0 osd=0 3 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 1 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c0475e0 osd=0 3 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 0 1 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c0475e0 osd=0 4 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab930c7700 10 client.27375.objecter _send_command 1 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab8c107110 con 0x7fab8c0feef0 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab91664700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6801/2103743338 conn(0x7fab8c0feef0 0x7fab8c0ff310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab89ffb700 10 client.27375.objecter ms_handle_connect 0x7fab8c0feef0 2024-02-13T00:46:49.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.302+0000 7fab91664700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6801/2103743338 conn(0x7fab8c0feef0 0x7fab8c0ff310 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.303+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab8c107110 con 0x7fab8c0feef0 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.303+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.303+0000 7fab89ffb700 10 client.27375.objecter _finish_command 1 = system:0 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.303+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c0475e0 osd=0 4 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.303+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 0 1 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.318+0000 7fab930c7700 10 client.27375.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.318+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 2 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.318+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c0475e0 osd=0 3 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.318+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c0475e0 osd=0 3 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.318+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c0475e0 osd=0 3 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.318+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 2 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.318+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c0475e0 osd=0 3 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.318+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 0 2 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.318+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c0475e0 osd=0 4 2024-02-13T00:46:49.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.318+0000 7fab930c7700 10 client.27375.objecter _send_command 2 2024-02-13T00:46:49.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.318+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab8c027410 con 0x7fab8c0feef0 2024-02-13T00:46:49.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.319+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab8c027410 con 0x7fab8c0feef0 2024-02-13T00:46:49.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.319+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:49.640 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:46:49.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.319+0000 7fab89ffb700 10 client.27375.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.319+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c0475e0 osd=0 4 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.319+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 0 2 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab930c7700 10 client.27375.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 3 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6805/150599780 conn(0x7fab8c059990 0x7fab8c059db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c028760 osd=1 3 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c028760 osd=1 3 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c028760 osd=1 3 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 3 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c028760 osd=1 3 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 1 3 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c028760 osd=1 4 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab930c7700 10 client.27375.objecter _send_command 3 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab8c11d880 con 0x7fab8c059990 2024-02-13T00:46:49.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab90e63700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6805/150599780 conn(0x7fab8c059990 0x7fab8c059db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab89ffb700 10 client.27375.objecter ms_handle_connect 0x7fab8c059990 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab90e63700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6805/150599780 conn(0x7fab8c059990 0x7fab8c059db0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.320+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab8c11d880 con 0x7fab8c059990 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.321+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.321+0000 7fab89ffb700 10 client.27375.objecter _finish_command 3 = system:0 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.321+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c028760 osd=1 4 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.321+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 1 3 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.336+0000 7fab930c7700 10 client.27375.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.336+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 4 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.336+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c028760 osd=1 3 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.336+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c028760 osd=1 3 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.336+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c028760 osd=1 3 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.336+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 4 2024-02-13T00:46:49.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.336+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c028760 osd=1 3 2024-02-13T00:46:49.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.336+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 1 4 2024-02-13T00:46:49.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.336+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c028760 osd=1 4 2024-02-13T00:46:49.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.336+0000 7fab930c7700 10 client.27375.objecter _send_command 4 2024-02-13T00:46:49.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.336+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab8c059400 con 0x7fab8c059990 2024-02-13T00:46:49.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab8c059400 con 0x7fab8c059990 2024-02-13T00:46:49.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:49.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab89ffb700 10 client.27375.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c028760 osd=1 4 2024-02-13T00:46:49.643 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 1 4 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab930c7700 10 client.27375.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 5 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6809/3689719376 conn(0x7fab8c0282c0 0x7fab8c11c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c059810 osd=2 3 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c059810 osd=2 3 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c059810 osd=2 3 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 5 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c059810 osd=2 3 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 2 5 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c059810 osd=2 4 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab930c7700 10 client.27375.objecter _send_command 5 2024-02-13T00:46:49.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab8c11c790 con 0x7fab8c0282c0 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.337+0000 7fab8bfff700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6809/3689719376 conn(0x7fab8c0282c0 0x7fab8c11c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.338+0000 7fab89ffb700 10 client.27375.objecter ms_handle_connect 0x7fab8c0282c0 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.338+0000 7fab8bfff700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6809/3689719376 conn(0x7fab8c0282c0 0x7fab8c11c2a0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.338+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab8c11c790 con 0x7fab8c0282c0 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.338+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.338+0000 7fab89ffb700 10 client.27375.objecter _finish_command 5 = system:0 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.338+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c059810 osd=2 4 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.338+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 2 5 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.353+0000 7fab930c7700 10 client.27375.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.353+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 6 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.353+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c059810 osd=2 3 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.353+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c059810 osd=2 3 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.353+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c059810 osd=2 3 2024-02-13T00:46:49.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.353+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 6 2024-02-13T00:46:49.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.353+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c059810 osd=2 3 2024-02-13T00:46:49.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.353+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 2 6 2024-02-13T00:46:49.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.353+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c059810 osd=2 4 2024-02-13T00:46:49.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.353+0000 7fab930c7700 10 client.27375.objecter _send_command 6 2024-02-13T00:46:49.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.353+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab8c027820 con 0x7fab8c0282c0 2024-02-13T00:46:49.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab8c027820 con 0x7fab8c0282c0 2024-02-13T00:46:49.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:49.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab89ffb700 10 client.27375.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c059810 osd=2 4 2024-02-13T00:46:49.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 2 6 2024-02-13T00:46:49.646 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:46:49.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab930c7700 10 client.27375.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:49.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 7 2024-02-13T00:46:49.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6813/604971011 conn(0x7fab8c11fa00 0x7fab8c121df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:49.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c027c30 osd=3 3 2024-02-13T00:46:49.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c027c30 osd=3 3 2024-02-13T00:46:49.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c027c30 osd=3 3 2024-02-13T00:46:49.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 7 2024-02-13T00:46:49.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c027c30 osd=3 3 2024-02-13T00:46:49.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 3 7 2024-02-13T00:46:49.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c027c30 osd=3 4 2024-02-13T00:46:49.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab930c7700 10 client.27375.objecter _send_command 7 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab8c124a80 con 0x7fab8c11fa00 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab91664700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6813/604971011 conn(0x7fab8c11fa00 0x7fab8c121df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab89ffb700 10 client.27375.objecter ms_handle_connect 0x7fab8c11fa00 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.355+0000 7fab91664700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6813/604971011 conn(0x7fab8c11fa00 0x7fab8c121df0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.356+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab8c124a80 con 0x7fab8c11fa00 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.356+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.356+0000 7fab89ffb700 10 client.27375.objecter _finish_command 7 = system:0 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.356+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c027c30 osd=3 4 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.356+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 3 7 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.371+0000 7fab930c7700 10 client.27375.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.371+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 8 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.371+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c027c30 osd=3 3 2024-02-13T00:46:49.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.371+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c027c30 osd=3 3 2024-02-13T00:46:49.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.371+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c027c30 osd=3 3 2024-02-13T00:46:49.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.371+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 8 2024-02-13T00:46:49.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.371+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c027c30 osd=3 3 2024-02-13T00:46:49.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.371+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 3 8 2024-02-13T00:46:49.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.371+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c027c30 osd=3 4 2024-02-13T00:46:49.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.371+0000 7fab930c7700 10 client.27375.objecter _send_command 8 2024-02-13T00:46:49.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.371+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab8c11cba0 con 0x7fab8c11fa00 2024-02-13T00:46:49.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab8c11cba0 con 0x7fab8c11fa00 2024-02-13T00:46:49.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:49.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab89ffb700 10 client.27375.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.650 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:46:49.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c027c30 osd=3 4 2024-02-13T00:46:49.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 3 8 2024-02-13T00:46:49.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab930c7700 10 client.27375.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:49.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 9 2024-02-13T00:46:49.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6800/3344263388 conn(0x7fab8c11be00 0x7fab8c123460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:49.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c11cba0 osd=4 3 2024-02-13T00:46:49.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c11cba0 osd=4 3 2024-02-13T00:46:49.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c11cba0 osd=4 3 2024-02-13T00:46:49.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 9 2024-02-13T00:46:49.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c11cba0 osd=4 3 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 4 9 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c11cba0 osd=4 4 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab930c7700 10 client.27375.objecter _send_command 9 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab8c11ce30 con 0x7fab8c11be00 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.373+0000 7fab90e63700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6800/3344263388 conn(0x7fab8c11be00 0x7fab8c123460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.374+0000 7fab89ffb700 10 client.27375.objecter ms_handle_connect 0x7fab8c11be00 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.374+0000 7fab90e63700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6800/3344263388 conn(0x7fab8c11be00 0x7fab8c123460 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.374+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab8c11ce30 con 0x7fab8c11be00 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.374+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.374+0000 7fab89ffb700 10 client.27375.objecter _finish_command 9 = system:0 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.374+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c11cba0 osd=4 4 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.374+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 4 9 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.389+0000 7fab930c7700 10 client.27375.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:49.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.389+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 10 2024-02-13T00:46:49.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.389+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c11cba0 osd=4 3 2024-02-13T00:46:49.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.389+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c11cba0 osd=4 3 2024-02-13T00:46:49.652 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:46:49.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.389+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c11cba0 osd=4 3 2024-02-13T00:46:49.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.389+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 10 2024-02-13T00:46:49.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.389+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c11cba0 osd=4 3 2024-02-13T00:46:49.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.389+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 4 10 2024-02-13T00:46:49.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.389+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c11cba0 osd=4 4 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.389+0000 7fab930c7700 10 client.27375.objecter _send_command 10 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.389+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab8c11ce30 con 0x7fab8c11be00 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab8c11ce30 con 0x7fab8c11be00 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab89ffb700 10 client.27375.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c11cba0 osd=4 4 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 4 10 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab930c7700 10 client.27375.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 11 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6804/2212296748 conn(0x7fab8c123e60 0x7fab8c1242e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c11ce30 osd=5 3 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c11ce30 osd=5 3 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c11ce30 osd=5 3 2024-02-13T00:46:49.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 11 2024-02-13T00:46:49.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c11ce30 osd=5 3 2024-02-13T00:46:49.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 5 11 2024-02-13T00:46:49.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c11ce30 osd=5 4 2024-02-13T00:46:49.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab930c7700 10 client.27375.objecter _send_command 11 2024-02-13T00:46:49.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.181:6804/2212296748 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab8c1248d0 con 0x7fab8c123e60 2024-02-13T00:46:49.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab8bfff700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6804/2212296748 conn(0x7fab8c123e60 0x7fab8c1242e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:49.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab89ffb700 10 client.27375.objecter ms_handle_connect 0x7fab8c123e60 2024-02-13T00:46:49.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.391+0000 7fab8bfff700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6804/2212296748 conn(0x7fab8c123e60 0x7fab8c1242e0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:49.654 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:46:49.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.392+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.5 v2:172.21.15.181:6804/2212296748 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab8c1248d0 con 0x7fab8c123e60 2024-02-13T00:46:49.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.392+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:46:49.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.392+0000 7fab89ffb700 10 client.27375.objecter _finish_command 11 = system:0 2024-02-13T00:46:49.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.392+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c11ce30 osd=5 4 2024-02-13T00:46:49.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.392+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 5 11 2024-02-13T00:46:49.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.406+0000 7fab930c7700 10 client.27375.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:49.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.406+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 12 2024-02-13T00:46:49.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.406+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c11ce30 osd=5 3 2024-02-13T00:46:49.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.406+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c11ce30 osd=5 3 2024-02-13T00:46:49.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.406+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c11ce30 osd=5 3 2024-02-13T00:46:49.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.406+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 12 2024-02-13T00:46:49.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.406+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c11ce30 osd=5 3 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.406+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 5 12 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.406+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c11ce30 osd=5 4 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.406+0000 7fab930c7700 10 client.27375.objecter _send_command 12 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.406+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.181:6804/2212296748 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab8c1248d0 con 0x7fab8c123e60 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.5 v2:172.21.15.181:6804/2212296748 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab8c1248d0 con 0x7fab8c123e60 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab89ffb700 10 client.27375.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c11ce30 osd=5 4 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 5 12 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab930c7700 10 client.27375.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 13 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6808/1506640341 conn(0x7fab8c1291d0 0x7fab8c129630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c1248d0 osd=6 3 2024-02-13T00:46:49.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c1248d0 osd=6 3 2024-02-13T00:46:49.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c1248d0 osd=6 3 2024-02-13T00:46:49.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 13 2024-02-13T00:46:49.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c1248d0 osd=6 3 2024-02-13T00:46:49.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 6 13 2024-02-13T00:46:49.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c1248d0 osd=6 4 2024-02-13T00:46:49.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab930c7700 10 client.27375.objecter _send_command 13 2024-02-13T00:46:49.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab8c12b280 con 0x7fab8c1291d0 2024-02-13T00:46:49.657 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab91664700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6808/1506640341 conn(0x7fab8c1291d0 0x7fab8c129630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab91664700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6808/1506640341 conn(0x7fab8c1291d0 0x7fab8c129630 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab89ffb700 10 client.27375.objecter ms_handle_connect 0x7fab8c1291d0 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab8c12b280 con 0x7fab8c1291d0 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab89ffb700 10 client.27375.objecter _finish_command 13 = system:0 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c1248d0 osd=6 4 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.409+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 6 13 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.425+0000 7fab930c7700 10 client.27375.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.425+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 14 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.425+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c1248d0 osd=6 3 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.425+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c1248d0 osd=6 3 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.425+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c1248d0 osd=6 3 2024-02-13T00:46:49.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.425+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 14 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.425+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c1248d0 osd=6 3 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.425+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 6 14 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.425+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c1248d0 osd=6 4 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.425+0000 7fab930c7700 10 client.27375.objecter _send_command 14 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.425+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab8c12b280 con 0x7fab8c1291d0 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab8c12b280 con 0x7fab8c1291d0 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab89ffb700 10 client.27375.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c1248d0 osd=6 4 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 6 14 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab930c7700 10 client.27375.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 15 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6812/1964192772 conn(0x7fab8c12da20 0x7fab8c12de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:49.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c12b280 osd=7 3 2024-02-13T00:46:49.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c12b280 osd=7 3 2024-02-13T00:46:49.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c12b280 osd=7 3 2024-02-13T00:46:49.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 15 2024-02-13T00:46:49.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c12b280 osd=7 3 2024-02-13T00:46:49.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 7 15 2024-02-13T00:46:49.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c12b280 osd=7 4 2024-02-13T00:46:49.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab930c7700 10 client.27375.objecter _send_command 15 2024-02-13T00:46:49.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab8c12e500 con 0x7fab8c12da20 2024-02-13T00:46:49.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab90e63700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6812/1964192772 conn(0x7fab8c12da20 0x7fab8c12de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:49.660 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:46:49.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab90e63700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6812/1964192772 conn(0x7fab8c12da20 0x7fab8c12de60 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:49.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.428+0000 7fab89ffb700 10 client.27375.objecter ms_handle_connect 0x7fab8c12da20 2024-02-13T00:46:49.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.430+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab8c12e500 con 0x7fab8c12da20 2024-02-13T00:46:49.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.430+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:46:49.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.430+0000 7fab89ffb700 10 client.27375.objecter _finish_command 15 = system:0 2024-02-13T00:46:49.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.430+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c12b280 osd=7 4 2024-02-13T00:46:49.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.430+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 7 15 2024-02-13T00:46:49.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.445+0000 7fab930c7700 10 client.27375.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:46:49.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.445+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign -1 16 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.445+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c12b280 osd=7 3 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.445+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c12b280 osd=7 3 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.445+0000 7fab930c7700 20 client.27375.objecter _get_session s=0x7fab8c12b280 osd=7 3 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.445+0000 7fab930c7700 15 client.27375.objecter _session_command_op_remove -1 16 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.445+0000 7fab930c7700 20 client.27375.objecter get_session s=0x7fab8c12b280 osd=7 3 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.445+0000 7fab930c7700 15 client.27375.objecter _session_command_op_assign 7 16 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.445+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c12b280 osd=7 4 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.445+0000 7fab930c7700 10 client.27375.objecter _send_command 16 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.445+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab8c12e500 con 0x7fab8c12da20 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab89ffb700 1 -- 172.21.15.162:0/328612703 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab8c12e500 con 0x7fab8c12da20 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab89ffb700 10 client.27375.objecter ms_dispatch 0x7fab8c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab89ffb700 10 client.27375.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab89ffb700 20 client.27375.objecter put_session s=0x7fab8c12b280 osd=7 4 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab89ffb700 15 client.27375.objecter _session_command_op_remove 7 16 2024-02-13T00:46:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 10 librados: watch_flush enter 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 10 librados: watch_flush exit 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 10 client.27375.objecter close_session for osd.0 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6801/2103743338 conn(0x7fab8c0feef0 msgr2=0x7fab8c0ff310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6801/2103743338 conn(0x7fab8c0feef0 0x7fab8c0ff310 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c0475e0 osd=0 1 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 10 client.27375.objecter close_session for osd.1 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6805/150599780 conn(0x7fab8c059990 msgr2=0x7fab8c059db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6805/150599780 conn(0x7fab8c059990 0x7fab8c059db0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c028760 osd=1 1 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 10 client.27375.objecter close_session for osd.2 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6809/3689719376 conn(0x7fab8c0282c0 msgr2=0x7fab8c11c2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6809/3689719376 conn(0x7fab8c0282c0 0x7fab8c11c2a0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c059810 osd=2 1 2024-02-13T00:46:49.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 10 client.27375.objecter close_session for osd.3 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6813/604971011 conn(0x7fab8c11fa00 msgr2=0x7fab8c121df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6813/604971011 conn(0x7fab8c11fa00 0x7fab8c121df0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c027c30 osd=3 1 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 10 client.27375.objecter close_session for osd.4 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6800/3344263388 conn(0x7fab8c11be00 msgr2=0x7fab8c123460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6800/3344263388 conn(0x7fab8c11be00 0x7fab8c123460 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c11cba0 osd=4 1 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 10 client.27375.objecter close_session for osd.5 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6804/2212296748 conn(0x7fab8c123e60 msgr2=0x7fab8c1242e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6804/2212296748 conn(0x7fab8c123e60 0x7fab8c1242e0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c11ce30 osd=5 1 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 10 client.27375.objecter close_session for osd.6 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6808/1506640341 conn(0x7fab8c1291d0 msgr2=0x7fab8c129630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6808/1506640341 conn(0x7fab8c1291d0 0x7fab8c129630 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c1248d0 osd=6 1 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 10 client.27375.objecter close_session for osd.7 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6812/1964192772 conn(0x7fab8c12da20 msgr2=0x7fab8c12de60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.181:6812/1964192772 conn(0x7fab8c12da20 0x7fab8c12de60 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 20 client.27375.objecter put_session s=0x7fab8c12b280 osd=7 1 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 20 client.27375.objecter shutdown clearing up homeless session... 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab930c7700 10 client.27375.objecter successfully canceled tick 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab91664700 1 -- 172.21.15.162:0/328612703 reap_dead start 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab91664700 1 -- 172.21.15.162:0/328612703 reap_dead start 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab91664700 1 -- 172.21.15.162:0/328612703 reap_dead start 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab91664700 1 -- 172.21.15.162:0/328612703 reap_dead start 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.447+0000 7fab91664700 1 -- 172.21.15.162:0/328612703 reap_dead start 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.448+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab74062c90 msgr2=0x7fab74065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.448+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab74062c90 0x7fab74065150 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 rx=0x7fab8c06c910 tx=0x7fab8000b040).stop 2024-02-13T00:46:49.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.448+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab8c0f8490 msgr2=0x7fab8c06b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.448+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab8c0f8490 0x7fab8c06b170 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 rx=0x7fab8400eca0 tx=0x7fab8400b040).stop 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.448+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 shutdown_connections 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.448+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab8c0f8490 0x7fab8c06b170 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.448+0000 7fab930c7700 1 --2- 172.21.15.162:0/328612703 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab74062c90 0x7fab74065150 unknown :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.448+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 >> 172.21.15.162:0/328612703 conn(0x7fab8c0f3850 msgr2=0x7fab8c0f50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.448+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 shutdown_connections 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.448+0000 7fab930c7700 1 -- 172.21.15.162:0/328612703 wait complete. 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:49.448+0000 7fab930c7700 1 librados: shutdown 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.666 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.667 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:46:49.768 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:48.428524+0000 mgr.y (mgr.14544) 380 : cluster [DBG] pgmap v626: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:46:49.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:48.447985+0000 mon.a (mon.0) 2546 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:49.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:48.455641+0000 mon.a (mon.0) 2547 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:49.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:48.455789+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? 172.21.15.162:0/2488015535' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:46:49.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:48.455846+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e430: 8 total, 8 up, 4 in 2024-02-13T00:46:49.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:48.461059+0000 mon.b (mon.1) 579 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:46:49.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:48.465137+0000 mon.b (mon.1) 580 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:49.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:48.469265+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:49.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:49.303257+0000 mon.a (mon.0) 2551 : audit [DBG] from='client.? 172.21.15.162:0/328612703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:49.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:49.336863+0000 mon.a (mon.0) 2552 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:49.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-13T00:46:49.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:48.428524+0000 mgr.y (mgr.14544) 380 : cluster [DBG] pgmap v626: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:46:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:48.447985+0000 mon.a (mon.0) 2546 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:48.455641+0000 mon.a (mon.0) 2547 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:48.455789+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? 172.21.15.162:0/2488015535' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:46:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:48.455846+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e430: 8 total, 8 up, 4 in 2024-02-13T00:46:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:48.461059+0000 mon.b (mon.1) 579 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:46:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:48.465137+0000 mon.b (mon.1) 580 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:48.469265+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:49.303257+0000 mon.a (mon.0) 2551 : audit [DBG] from='client.? 172.21.15.162:0/328612703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:49.336863+0000 mon.a (mon.0) 2552 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:49.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:48.428524+0000 mgr.y (mgr.14544) 380 : cluster [DBG] pgmap v626: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:46:49.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:48.447985+0000 mon.a (mon.0) 2546 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:48.455641+0000 mon.a (mon.0) 2547 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:46:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:48.455789+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? 172.21.15.162:0/2488015535' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:46:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:48.455846+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e430: 8 total, 8 up, 4 in 2024-02-13T00:46:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:48.461059+0000 mon.b (mon.1) 579 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:46:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:48.465137+0000 mon.b (mon.1) 580 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:48.469265+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:46:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:49.303257+0000 mon.a (mon.0) 2551 : audit [DBG] from='client.? 172.21.15.162:0/328612703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:49.336863+0000 mon.a (mon.0) 2552 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:46:50.052 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:46:50.228 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":624,"stamp":"2024-02-13T00:46:46.442700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007280,"num_objects":383,"num_object_clones":2,"num_object_copies":1149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":1706,"num_read_kb":1331,"num_write":1223,"num_write_kb":4543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1852,"num_bytes_recovered":4017904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2683,"ondisk_log_size":2683,"up":618,"acting":618,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":714,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1270044,"kb_used_data":40168,"kb_used_omap":0,"kb_used_meta":1229824,"kb_avail":373710564,"statfs":{"total":383980142592,"available":382679617536,"internally_reserved":0,"allocated":41132032,"data_stored":27810722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1259339776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12,"num_objects":-3,"num_object_clones":0,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":-9,"num_read_kb":-6,"num_write":-12,"num_write_kb":-4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.016752"},"pg_stats":[{"pgid":"61.2","version":"204'34","reported_seq":214,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.528899+0000","last_change":"2024-02-13T00:46:35.432364+0000","last_active":"2024-02-13T00:46:41.528899+0000","last_peered":"2024-02-13T00:46:41.528899+0000","last_clean":"2024-02-13T00:46:41.528899+0000","last_became_active":"2024-02-13T00:46:01.448503+0000","last_became_peered":"2024-02-13T00:46:01.448503+0000","last_unstale":"2024-02-13T00:46:41.528899+0000","last_undegraded":"2024-02-13T00:46:41.528899+0000","last_fullsized":"2024-02-13T00:46:41.528899+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-02-13T00:46:19.267508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:19.267508+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.3","version":"375'104","reported_seq":621,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.536472+0000","last_change":"2024-02-13T00:46:35.421257+0000","last_active":"2024-02-13T00:46:40.536472+0000","last_peered":"2024-02-13T00:46:40.536472+0000","last_clean":"2024-02-13T00:46:40.536472+0000","last_became_active":"2024-02-13T00:45:22.426662+0000","last_became_peered":"2024-02-13T00:45:22.426662+0000","last_unstale":"2024-02-13T00:46:40.536472+0000","last_undegraded":"2024-02-13T00:46:40.536472+0000","last_fullsized":"2024-02-13T00:46:40.536472+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'104","last_scrub_stamp":"2024-02-13T00:45:34.225334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.225334+0000","log_size":104,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.0","version":"388'38","reported_seq":442,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421369+0000","last_change":"2024-02-13T00:46:35.670480+0000","last_active":"2024-02-13T00:46:40.421369+0000","last_peered":"2024-02-13T00:46:40.421369+0000","last_clean":"2024-02-13T00:46:40.421369+0000","last_became_active":"2024-02-13T00:46:01.436132+0000","last_became_peered":"2024-02-13T00:46:01.436132+0000","last_unstale":"2024-02-13T00:46:40.421369+0000","last_undegraded":"2024-02-13T00:46:40.421369+0000","last_fullsized":"2024-02-13T00:46:40.421369+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'38","last_scrub_stamp":"2024-02-13T00:45:43.553743+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-02-13T00:45:43.553743+0000","last_clean_scrub_stamp":"2024-02-13T00:45:43.553743+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000030,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4000030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.1","version":"375'35","reported_seq":435,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535281+0000","last_change":"2024-02-13T00:46:35.421463+0000","last_active":"2024-02-13T00:46:40.535281+0000","last_peered":"2024-02-13T00:46:40.535281+0000","last_clean":"2024-02-13T00:46:40.535281+0000","last_became_active":"2024-02-13T00:45:15.420934+0000","last_became_peered":"2024-02-13T00:45:15.420934+0000","last_unstale":"2024-02-13T00:46:40.535281+0000","last_undegraded":"2024-02-13T00:46:40.535281+0000","last_fullsized":"2024-02-13T00:46:40.535281+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'34","last_scrub_stamp":"2024-02-13T00:45:29.241606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.241606+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.6","version":"400'20","reported_seq":363,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537972+0000","last_change":"2024-02-13T00:46:35.422997+0000","last_active":"2024-02-13T00:46:40.537972+0000","last_peered":"2024-02-13T00:46:40.537972+0000","last_clean":"2024-02-13T00:46:40.537972+0000","last_became_active":"2024-02-13T00:46:01.452805+0000","last_became_peered":"2024-02-13T00:46:01.452805+0000","last_unstale":"2024-02-13T00:46:40.537972+0000","last_undegraded":"2024-02-13T00:46:40.537972+0000","last_fullsized":"2024-02-13T00:46:40.537972+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'20","last_scrub_stamp":"2024-02-13T00:46:20.166298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:20.166298+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.7","version":"270'85","reported_seq":356,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424090+0000","last_change":"2024-02-13T00:46:35.674133+0000","last_active":"2024-02-13T00:46:40.424090+0000","last_peered":"2024-02-13T00:46:40.424090+0000","last_clean":"2024-02-13T00:46:40.424090+0000","last_became_active":"2024-02-13T00:46:01.458985+0000","last_became_peered":"2024-02-13T00:46:01.458985+0000","last_unstale":"2024-02-13T00:46:40.424090+0000","last_undegraded":"2024-02-13T00:46:40.424090+0000","last_fullsized":"2024-02-13T00:46:40.424090+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'85","last_scrub_stamp":"2024-02-13T00:45:02.230151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:02.230151+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.4","version":"407'59","reported_seq":162,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423082+0000","last_change":"2024-02-13T00:46:35.673002+0000","last_active":"2024-02-13T00:46:40.423082+0000","last_peered":"2024-02-13T00:46:40.423082+0000","last_clean":"2024-02-13T00:46:40.423082+0000","last_became_active":"2024-02-13T00:46:01.455621+0000","last_became_peered":"2024-02-13T00:46:01.455621+0000","last_unstale":"2024-02-13T00:46:40.423082+0000","last_undegraded":"2024-02-13T00:46:40.423082+0000","last_fullsized":"2024-02-13T00:46:40.423082+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'57","last_scrub_stamp":"2024-02-13T00:45:32.238130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.238130+0000","log_size":59,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":84,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.5","version":"224'56","reported_seq":559,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537675+0000","last_change":"2024-02-13T00:46:35.423606+0000","last_active":"2024-02-13T00:46:40.537675+0000","last_peered":"2024-02-13T00:46:40.537675+0000","last_clean":"2024-02-13T00:46:40.537675+0000","last_became_active":"2024-02-13T00:46:01.456637+0000","last_became_peered":"2024-02-13T00:46:01.456637+0000","last_unstale":"2024-02-13T00:46:40.537675+0000","last_undegraded":"2024-02-13T00:46:40.537675+0000","last_fullsized":"2024-02-13T00:46:40.537675+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'56","last_scrub_stamp":"2024-02-13T00:45:55.071118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:55.071118+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.a","version":"343'92","reported_seq":499,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535489+0000","last_change":"2024-02-13T00:46:35.422709+0000","last_active":"2024-02-13T00:46:40.535489+0000","last_peered":"2024-02-13T00:46:40.535489+0000","last_clean":"2024-02-13T00:46:40.535489+0000","last_became_active":"2024-02-13T00:46:01.453135+0000","last_became_peered":"2024-02-13T00:46:01.453135+0000","last_unstale":"2024-02-13T00:46:40.535489+0000","last_undegraded":"2024-02-13T00:46:40.535489+0000","last_fullsized":"2024-02-13T00:46:40.535489+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'92","last_scrub_stamp":"2024-02-13T00:45:50.066059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:50.066059+0000","log_size":92,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.b","version":"222'17","reported_seq":450,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.536449+0000","last_change":"2024-02-13T00:46:35.422505+0000","last_active":"2024-02-13T00:46:40.536449+0000","last_peered":"2024-02-13T00:46:40.536449+0000","last_clean":"2024-02-13T00:46:40.536449+0000","last_became_active":"2024-02-13T00:44:50.433096+0000","last_became_peered":"2024-02-13T00:44:50.433096+0000","last_unstale":"2024-02-13T00:46:40.536449+0000","last_undegraded":"2024-02-13T00:46:40.536449+0000","last_fullsized":"2024-02-13T00:46:40.536449+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'17","last_scrub_stamp":"2024-02-13T00:45:46.193088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.193088+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":74,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.8","version":"407'56","reported_seq":462,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421499+0000","last_change":"2024-02-13T00:46:35.670447+0000","last_active":"2024-02-13T00:46:40.421499+0000","last_peered":"2024-02-13T00:46:40.421499+0000","last_clean":"2024-02-13T00:46:40.421499+0000","last_became_active":"2024-02-13T00:46:01.445063+0000","last_became_peered":"2024-02-13T00:46:01.445063+0000","last_unstale":"2024-02-13T00:46:40.421499+0000","last_undegraded":"2024-02-13T00:46:40.421499+0000","last_fullsized":"2024-02-13T00:46:40.421499+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'53","last_scrub_stamp":"2024-02-13T00:44:30.046222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:30.046222+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.9","version":"343'36","reported_seq":440,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535261+0000","last_change":"2024-02-13T00:46:35.421109+0000","last_active":"2024-02-13T00:46:40.535261+0000","last_peered":"2024-02-13T00:46:40.535261+0000","last_clean":"2024-02-13T00:46:40.535261+0000","last_became_active":"2024-02-13T00:45:15.421026+0000","last_became_peered":"2024-02-13T00:45:15.421026+0000","last_unstale":"2024-02-13T00:46:40.535261+0000","last_undegraded":"2024-02-13T00:46:40.535261+0000","last_fullsized":"2024-02-13T00:46:40.535261+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'36","last_scrub_stamp":"2024-02-13T00:45:42.196133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.196133+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.e","version":"389'113","reported_seq":532,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530408+0000","last_change":"2024-02-13T00:46:35.432063+0000","last_active":"2024-02-13T00:46:41.530408+0000","last_peered":"2024-02-13T00:46:41.530408+0000","last_clean":"2024-02-13T00:46:41.530408+0000","last_became_active":"2024-02-13T00:46:17.513379+0000","last_became_peered":"2024-02-13T00:46:17.513379+0000","last_unstale":"2024-02-13T00:46:41.530408+0000","last_undegraded":"2024-02-13T00:46:41.530408+0000","last_fullsized":"2024-02-13T00:46:41.530408+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'113","last_scrub_stamp":"2024-02-13T00:45:45.254008+0000","last_deep_scrub":"389'113","last_deep_scrub_stamp":"2024-02-13T00:45:45.254008+0000","last_clean_scrub_stamp":"2024-02-13T00:45:45.254008+0000","log_size":113,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.f","version":"252'51","reported_seq":423,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537580+0000","last_change":"2024-02-13T00:46:35.422750+0000","last_active":"2024-02-13T00:46:40.537580+0000","last_peered":"2024-02-13T00:46:40.537580+0000","last_clean":"2024-02-13T00:46:40.537580+0000","last_became_active":"2024-02-13T00:46:01.440972+0000","last_became_peered":"2024-02-13T00:46:01.440972+0000","last_unstale":"2024-02-13T00:46:40.537580+0000","last_undegraded":"2024-02-13T00:46:40.537580+0000","last_fullsized":"2024-02-13T00:46:40.537580+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'51","last_scrub_stamp":"2024-02-13T00:45:26.215792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.215792+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.c","version":"400'108","reported_seq":713,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530196+0000","last_change":"2024-02-13T00:46:35.431163+0000","last_active":"2024-02-13T00:46:41.530196+0000","last_peered":"2024-02-13T00:46:41.530196+0000","last_clean":"2024-02-13T00:46:41.530196+0000","last_became_active":"2024-02-13T00:46:17.514719+0000","last_became_peered":"2024-02-13T00:46:17.514719+0000","last_unstale":"2024-02-13T00:46:41.530196+0000","last_undegraded":"2024-02-13T00:46:41.530196+0000","last_fullsized":"2024-02-13T00:46:41.530196+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'106","last_scrub_stamp":"2024-02-13T00:45:31.255154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.255154+0000","log_size":108,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":100,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.d","version":"343'91","reported_seq":534,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537561+0000","last_change":"2024-02-13T00:46:35.423176+0000","last_active":"2024-02-13T00:46:40.537561+0000","last_peered":"2024-02-13T00:46:40.537561+0000","last_clean":"2024-02-13T00:46:40.537561+0000","last_became_active":"2024-02-13T00:46:01.442967+0000","last_became_peered":"2024-02-13T00:46:01.442967+0000","last_unstale":"2024-02-13T00:46:40.537561+0000","last_undegraded":"2024-02-13T00:46:40.537561+0000","last_fullsized":"2024-02-13T00:46:40.537561+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'90","last_scrub_stamp":"2024-02-13T00:44:31.080244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:31.080244+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"6.29","version":"0'0","reported_seq":330,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421609+0000","last_change":"2024-02-13T00:46:01.448309+0000","last_active":"2024-02-13T00:46:40.421609+0000","last_peered":"2024-02-13T00:46:40.421609+0000","last_clean":"2024-02-13T00:46:40.421609+0000","last_became_active":"2024-02-13T00:46:01.445363+0000","last_became_peered":"2024-02-13T00:46:01.445363+0000","last_unstale":"2024-02-13T00:46:40.421609+0000","last_undegraded":"2024-02-13T00:46:40.421609+0000","last_fullsized":"2024-02-13T00:46:40.421609+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.12","version":"343'24","reported_seq":388,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.422706+0000","last_change":"2024-02-13T00:46:35.674236+0000","last_active":"2024-02-13T00:46:40.422706+0000","last_peered":"2024-02-13T00:46:40.422706+0000","last_clean":"2024-02-13T00:46:40.422706+0000","last_became_active":"2024-02-13T00:46:01.433618+0000","last_became_peered":"2024-02-13T00:46:01.433618+0000","last_unstale":"2024-02-13T00:46:40.422706+0000","last_undegraded":"2024-02-13T00:46:40.422706+0000","last_fullsized":"2024-02-13T00:46:40.422706+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'22","last_scrub_stamp":"2024-02-13T00:44:34.629429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.629429+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"6.28","version":"0'0","reported_seq":255,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.528822+0000","last_change":"2024-02-13T00:46:01.449894+0000","last_active":"2024-02-13T00:46:41.528822+0000","last_peered":"2024-02-13T00:46:41.528822+0000","last_clean":"2024-02-13T00:46:41.528822+0000","last_became_active":"2024-02-13T00:46:01.449570+0000","last_became_peered":"2024-02-13T00:46:01.449570+0000","last_unstale":"2024-02-13T00:46:41.528822+0000","last_undegraded":"2024-02-13T00:46:41.528822+0000","last_fullsized":"2024-02-13T00:46:41.528822+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.13","version":"236'73","reported_seq":231,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533540+0000","last_change":"2024-02-13T00:46:35.429460+0000","last_active":"2024-02-13T00:46:41.533540+0000","last_peered":"2024-02-13T00:46:41.533540+0000","last_clean":"2024-02-13T00:46:41.533540+0000","last_became_active":"2024-02-13T00:45:15.447922+0000","last_became_peered":"2024-02-13T00:45:15.447922+0000","last_unstale":"2024-02-13T00:46:41.533540+0000","last_undegraded":"2024-02-13T00:46:41.533540+0000","last_fullsized":"2024-02-13T00:46:41.533540+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'73","last_scrub_stamp":"2024-02-13T00:46:22.291170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:22.291170+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.10","version":"397'79","reported_seq":605,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529212+0000","last_change":"2024-02-13T00:46:35.429803+0000","last_active":"2024-02-13T00:46:41.529212+0000","last_peered":"2024-02-13T00:46:41.529212+0000","last_clean":"2024-02-13T00:46:41.529212+0000","last_became_active":"2024-02-13T00:46:01.440561+0000","last_became_peered":"2024-02-13T00:46:01.440561+0000","last_unstale":"2024-02-13T00:46:41.529212+0000","last_undegraded":"2024-02-13T00:46:41.529212+0000","last_fullsized":"2024-02-13T00:46:41.529212+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'73","last_scrub_stamp":"2024-02-13T00:45:27.173322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:27.173322+0000","log_size":79,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.11","version":"343'37","reported_seq":319,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535156+0000","last_change":"2024-02-13T00:46:35.420913+0000","last_active":"2024-02-13T00:46:40.535156+0000","last_peered":"2024-02-13T00:46:40.535156+0000","last_clean":"2024-02-13T00:46:40.535156+0000","last_became_active":"2024-02-13T00:44:50.443606+0000","last_became_peered":"2024-02-13T00:44:50.443606+0000","last_unstale":"2024-02-13T00:46:40.535156+0000","last_undegraded":"2024-02-13T00:46:40.535156+0000","last_fullsized":"2024-02-13T00:46:40.535156+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'37","last_scrub_stamp":"2024-02-13T00:46:22.170067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:22.170067+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.16","version":"375'75","reported_seq":521,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424183+0000","last_change":"2024-02-13T00:46:35.673407+0000","last_active":"2024-02-13T00:46:40.424183+0000","last_peered":"2024-02-13T00:46:40.424183+0000","last_clean":"2024-02-13T00:46:40.424183+0000","last_became_active":"2024-02-13T00:46:01.449774+0000","last_became_peered":"2024-02-13T00:46:01.449774+0000","last_unstale":"2024-02-13T00:46:40.424183+0000","last_undegraded":"2024-02-13T00:46:40.424183+0000","last_fullsized":"2024-02-13T00:46:40.424183+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'75","last_scrub_stamp":"2024-02-13T00:45:46.157235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.157235+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.17","version":"343'75","reported_seq":553,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424147+0000","last_change":"2024-02-13T00:46:35.674344+0000","last_active":"2024-02-13T00:46:40.424147+0000","last_peered":"2024-02-13T00:46:40.424147+0000","last_clean":"2024-02-13T00:46:40.424147+0000","last_became_active":"2024-02-13T00:46:01.447341+0000","last_became_peered":"2024-02-13T00:46:01.447341+0000","last_unstale":"2024-02-13T00:46:40.424147+0000","last_undegraded":"2024-02-13T00:46:40.424147+0000","last_fullsized":"2024-02-13T00:46:40.424147+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'75","last_scrub_stamp":"2024-02-13T00:45:36.518979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:36.518979+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.14","version":"200'17","reported_seq":433,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423923+0000","last_change":"2024-02-13T00:46:35.673208+0000","last_active":"2024-02-13T00:46:40.423923+0000","last_peered":"2024-02-13T00:46:40.423923+0000","last_clean":"2024-02-13T00:46:40.423923+0000","last_became_active":"2024-02-13T00:46:01.455179+0000","last_became_peered":"2024-02-13T00:46:01.455179+0000","last_unstale":"2024-02-13T00:46:40.423923+0000","last_undegraded":"2024-02-13T00:46:40.423923+0000","last_fullsized":"2024-02-13T00:46:40.423923+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'17","last_scrub_stamp":"2024-02-13T00:45:37.548974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.548974+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"61.15","version":"423'99","reported_seq":289,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537307+0000","last_change":"2024-02-13T00:46:37.425014+0000","last_active":"2024-02-13T00:46:40.537307+0000","last_peered":"2024-02-13T00:46:40.537307+0000","last_clean":"2024-02-13T00:46:40.537307+0000","last_became_active":"2024-02-13T00:46:01.452384+0000","last_became_peered":"2024-02-13T00:46:01.452384+0000","last_unstale":"2024-02-13T00:46:40.537307+0000","last_undegraded":"2024-02-13T00:46:40.537307+0000","last_fullsized":"2024-02-13T00:46:40.537307+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'78","last_scrub_stamp":"2024-02-13T00:45:36.188466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:36.188466+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"6.21","version":"0'0","reported_seq":119,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535120+0000","last_change":"2024-02-13T00:46:01.445141+0000","last_active":"2024-02-13T00:46:40.535120+0000","last_peered":"2024-02-13T00:46:40.535120+0000","last_clean":"2024-02-13T00:46:40.535120+0000","last_became_active":"2024-02-13T00:46:01.444729+0000","last_became_peered":"2024-02-13T00:46:01.444729+0000","last_unstale":"2024-02-13T00:46:40.535120+0000","last_undegraded":"2024-02-13T00:46:40.535120+0000","last_fullsized":"2024-02-13T00:46:40.535120+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1a","version":"403'54","reported_seq":206,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533149+0000","last_change":"2024-02-13T00:46:35.431980+0000","last_active":"2024-02-13T00:46:41.533149+0000","last_peered":"2024-02-13T00:46:41.533149+0000","last_clean":"2024-02-13T00:46:41.533149+0000","last_became_active":"2024-02-13T00:45:15.436908+0000","last_became_peered":"2024-02-13T00:45:15.436908+0000","last_unstale":"2024-02-13T00:46:41.533149+0000","last_undegraded":"2024-02-13T00:46:41.533149+0000","last_fullsized":"2024-02-13T00:46:41.533149+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'54","last_scrub_stamp":"2024-02-13T00:46:26.235259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:26.235259+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":255,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"6.20","version":"0'0","reported_seq":204,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535165+0000","last_change":"2024-02-13T00:45:15.426658+0000","last_active":"2024-02-13T00:46:40.535165+0000","last_peered":"2024-02-13T00:46:40.535165+0000","last_clean":"2024-02-13T00:46:40.535165+0000","last_became_active":"2024-02-13T00:45:15.426109+0000","last_became_peered":"2024-02-13T00:45:15.426109+0000","last_unstale":"2024-02-13T00:46:40.535165+0000","last_undegraded":"2024-02-13T00:46:40.535165+0000","last_fullsized":"2024-02-13T00:46:40.535165+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1b","version":"397'54","reported_seq":213,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.528961+0000","last_change":"2024-02-13T00:46:35.428298+0000","last_active":"2024-02-13T00:46:41.528961+0000","last_peered":"2024-02-13T00:46:41.528961+0000","last_clean":"2024-02-13T00:46:41.528961+0000","last_became_active":"2024-02-13T00:46:01.458733+0000","last_became_peered":"2024-02-13T00:46:01.458733+0000","last_unstale":"2024-02-13T00:46:41.528961+0000","last_undegraded":"2024-02-13T00:46:41.528961+0000","last_fullsized":"2024-02-13T00:46:41.528961+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'54","last_scrub_stamp":"2024-02-13T00:45:57.130510+0000","last_deep_scrub":"397'54","last_deep_scrub_stamp":"2024-02-13T00:45:57.130510+0000","last_clean_scrub_stamp":"2024-02-13T00:45:57.130510+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":16,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":189,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"6.23","version":"0'0","reported_seq":111,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533280+0000","last_change":"2024-02-13T00:44:39.526065+0000","last_active":"2024-02-13T00:46:41.533280+0000","last_peered":"2024-02-13T00:46:41.533280+0000","last_clean":"2024-02-13T00:46:41.533280+0000","last_became_active":"2024-02-13T00:44:39.525161+0000","last_became_peered":"2024-02-13T00:44:39.525161+0000","last_unstale":"2024-02-13T00:46:41.533280+0000","last_undegraded":"2024-02-13T00:46:41.533280+0000","last_fullsized":"2024-02-13T00:46:41.533280+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.18","version":"204'34","reported_seq":213,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535017+0000","last_change":"2024-02-13T00:46:35.422472+0000","last_active":"2024-02-13T00:46:40.535017+0000","last_peered":"2024-02-13T00:46:40.535017+0000","last_clean":"2024-02-13T00:46:40.535017+0000","last_became_active":"2024-02-13T00:46:01.438318+0000","last_became_peered":"2024-02-13T00:46:01.438318+0000","last_unstale":"2024-02-13T00:46:40.535017+0000","last_undegraded":"2024-02-13T00:46:40.535017+0000","last_fullsized":"2024-02-13T00:46:40.535017+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-02-13T00:45:29.218592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.218592+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"6.22","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.528757+0000","last_change":"2024-02-13T00:46:01.448133+0000","last_active":"2024-02-13T00:46:41.528757+0000","last_peered":"2024-02-13T00:46:41.528757+0000","last_clean":"2024-02-13T00:46:41.528757+0000","last_became_active":"2024-02-13T00:46:01.447415+0000","last_became_peered":"2024-02-13T00:46:01.447415+0000","last_unstale":"2024-02-13T00:46:41.528757+0000","last_undegraded":"2024-02-13T00:46:41.528757+0000","last_fullsized":"2024-02-13T00:46:41.528757+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.19","version":"357'18","reported_seq":367,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421704+0000","last_change":"2024-02-13T00:46:35.671847+0000","last_active":"2024-02-13T00:46:40.421704+0000","last_peered":"2024-02-13T00:46:40.421704+0000","last_clean":"2024-02-13T00:46:40.421704+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:46:40.421704+0000","last_undegraded":"2024-02-13T00:46:40.421704+0000","last_fullsized":"2024-02-13T00:46:40.421704+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'18","last_scrub_stamp":"2024-02-13T00:45:51.098914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:51.098914+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"6.25","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530423+0000","last_change":"2024-02-13T00:46:01.459772+0000","last_active":"2024-02-13T00:46:41.530423+0000","last_peered":"2024-02-13T00:46:41.530423+0000","last_clean":"2024-02-13T00:46:41.530423+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:46:41.530423+0000","last_undegraded":"2024-02-13T00:46:41.530423+0000","last_fullsized":"2024-02-13T00:46:41.530423+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1e","version":"403'57","reported_seq":259,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534648+0000","last_change":"2024-02-13T00:46:35.432270+0000","last_active":"2024-02-13T00:46:41.534648+0000","last_peered":"2024-02-13T00:46:41.534648+0000","last_clean":"2024-02-13T00:46:41.534648+0000","last_became_active":"2024-02-13T00:45:15.446531+0000","last_became_peered":"2024-02-13T00:45:15.446531+0000","last_unstale":"2024-02-13T00:46:41.534648+0000","last_undegraded":"2024-02-13T00:46:41.534648+0000","last_fullsized":"2024-02-13T00:46:41.534648+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'57","last_scrub_stamp":"2024-02-13T00:46:33.286813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:33.286813+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"6.24","version":"0'0","reported_seq":388,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529366+0000","last_change":"2024-02-13T00:46:01.463062+0000","last_active":"2024-02-13T00:46:41.529366+0000","last_peered":"2024-02-13T00:46:41.529366+0000","last_clean":"2024-02-13T00:46:41.529366+0000","last_became_active":"2024-02-13T00:46:01.462319+0000","last_became_peered":"2024-02-13T00:46:01.462319+0000","last_unstale":"2024-02-13T00:46:41.529366+0000","last_undegraded":"2024-02-13T00:46:41.529366+0000","last_fullsized":"2024-02-13T00:46:41.529366+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1f","version":"392'68","reported_seq":446,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424167+0000","last_change":"2024-02-13T00:46:35.674461+0000","last_active":"2024-02-13T00:46:40.424167+0000","last_peered":"2024-02-13T00:46:40.424167+0000","last_clean":"2024-02-13T00:46:40.424167+0000","last_became_active":"2024-02-13T00:44:50.441446+0000","last_became_peered":"2024-02-13T00:44:50.441446+0000","last_unstale":"2024-02-13T00:46:40.424167+0000","last_undegraded":"2024-02-13T00:46:40.424167+0000","last_fullsized":"2024-02-13T00:46:40.424167+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'68","last_scrub_stamp":"2024-02-13T00:45:51.651226+0000","last_deep_scrub":"392'68","last_deep_scrub_stamp":"2024-02-13T00:45:51.651226+0000","last_clean_scrub_stamp":"2024-02-13T00:45:51.651226+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"6.27","version":"0'0","reported_seq":111,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534829+0000","last_change":"2024-02-13T00:44:39.525601+0000","last_active":"2024-02-13T00:46:41.534829+0000","last_peered":"2024-02-13T00:46:41.534829+0000","last_clean":"2024-02-13T00:46:41.534829+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:46:41.534829+0000","last_undegraded":"2024-02-13T00:46:41.534829+0000","last_fullsized":"2024-02-13T00:46:41.534829+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1c","version":"397'56","reported_seq":545,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529719+0000","last_change":"2024-02-13T00:46:35.431168+0000","last_active":"2024-02-13T00:46:41.529719+0000","last_peered":"2024-02-13T00:46:41.529719+0000","last_clean":"2024-02-13T00:46:41.529719+0000","last_became_active":"2024-02-13T00:46:01.462341+0000","last_became_peered":"2024-02-13T00:46:01.462341+0000","last_unstale":"2024-02-13T00:46:41.529719+0000","last_undegraded":"2024-02-13T00:46:41.529719+0000","last_fullsized":"2024-02-13T00:46:41.529719+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'52","last_scrub_stamp":"2024-02-13T00:45:44.205380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:44.205380+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"6.26","version":"0'0","reported_seq":195,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537946+0000","last_change":"2024-02-13T00:46:01.455629+0000","last_active":"2024-02-13T00:46:40.537946+0000","last_peered":"2024-02-13T00:46:40.537946+0000","last_clean":"2024-02-13T00:46:40.537946+0000","last_became_active":"2024-02-13T00:46:01.452801+0000","last_became_peered":"2024-02-13T00:46:01.452801+0000","last_unstale":"2024-02-13T00:46:40.537946+0000","last_undegraded":"2024-02-13T00:46:40.537946+0000","last_fullsized":"2024-02-13T00:46:40.537946+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1d","version":"343'69","reported_seq":535,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537577+0000","last_change":"2024-02-13T00:46:35.423342+0000","last_active":"2024-02-13T00:46:40.537577+0000","last_peered":"2024-02-13T00:46:40.537577+0000","last_clean":"2024-02-13T00:46:40.537577+0000","last_became_active":"2024-02-13T00:45:15.435242+0000","last_became_peered":"2024-02-13T00:45:15.435242+0000","last_unstale":"2024-02-13T00:46:40.537577+0000","last_undegraded":"2024-02-13T00:46:40.537577+0000","last_fullsized":"2024-02-13T00:46:40.537577+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'69","last_scrub_stamp":"2024-02-13T00:46:28.118117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:28.118117+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"}]},{"pgid":"28.10","version":"0'0","reported_seq":351,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.422557+0000","last_change":"2024-02-13T00:45:39.562095+0000","last_active":"2024-02-13T00:46:40.422557+0000","last_peered":"2024-02-13T00:46:40.422557+0000","last_clean":"2024-02-13T00:46:40.422557+0000","last_became_active":"2024-02-13T00:44:39.505121+0000","last_became_peered":"2024-02-13T00:44:39.505121+0000","last_unstale":"2024-02-13T00:46:40.422557+0000","last_undegraded":"2024-02-13T00:46:40.422557+0000","last_fullsized":"2024-02-13T00:46:40.422557+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:39.561994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:39.561994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"363'14","reported_seq":161,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-13T00:46:42.434681+0000","last_change":"2024-02-13T00:45:53.115355+0000","last_active":"2024-02-13T00:46:42.434681+0000","last_peered":"2024-02-13T00:46:42.434681+0000","last_clean":"2024-02-13T00:46:42.434681+0000","last_became_active":"2024-02-13T00:45:15.435476+0000","last_became_peered":"2024-02-13T00:45:15.435476+0000","last_unstale":"2024-02-13T00:46:42.434681+0000","last_undegraded":"2024-02-13T00:46:42.434681+0000","last_fullsized":"2024-02-13T00:46:42.434681+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:45:53.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.115260+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":432,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.528958+0000","last_change":"2024-02-13T00:46:03.880562+0000","last_active":"2024-02-13T00:46:41.528958+0000","last_peered":"2024-02-13T00:46:41.528958+0000","last_clean":"2024-02-13T00:46:41.528958+0000","last_became_active":"2024-02-13T00:46:01.438056+0000","last_became_peered":"2024-02-13T00:46:01.438056+0000","last_unstale":"2024-02-13T00:46:41.528958+0000","last_undegraded":"2024-02-13T00:46:41.528958+0000","last_fullsized":"2024-02-13T00:46:41.528958+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:41:31.413581+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_clean_scrub_stamp":"2024-02-13T00:41:31.413581+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":420,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530648+0000","last_change":"2024-02-13T00:46:01.464695+0000","last_active":"2024-02-13T00:46:41.530648+0000","last_peered":"2024-02-13T00:46:41.530648+0000","last_clean":"2024-02-13T00:46:41.530648+0000","last_became_active":"2024-02-13T00:46:01.460811+0000","last_became_peered":"2024-02-13T00:46:01.460811+0000","last_unstale":"2024-02-13T00:46:41.530648+0000","last_undegraded":"2024-02-13T00:46:41.530648+0000","last_fullsized":"2024-02-13T00:46:41.530648+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.340080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.340080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":326,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530210+0000","last_change":"2024-02-13T00:46:01.451838+0000","last_active":"2024-02-13T00:46:41.530210+0000","last_peered":"2024-02-13T00:46:41.530210+0000","last_clean":"2024-02-13T00:46:41.530210+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:46:41.530210+0000","last_undegraded":"2024-02-13T00:46:41.530210+0000","last_fullsized":"2024-02-13T00:46:41.530210+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:45.416238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:45.416238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":369,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529174+0000","last_change":"2024-02-13T00:46:01.448237+0000","last_active":"2024-02-13T00:46:41.529174+0000","last_peered":"2024-02-13T00:46:41.529174+0000","last_clean":"2024-02-13T00:46:41.529174+0000","last_became_active":"2024-02-13T00:46:01.446660+0000","last_became_peered":"2024-02-13T00:46:01.446660+0000","last_unstale":"2024-02-13T00:46:41.529174+0000","last_undegraded":"2024-02-13T00:46:41.529174+0000","last_fullsized":"2024-02-13T00:46:41.529174+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.260871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.260871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":416,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423479+0000","last_change":"2024-02-13T00:42:19.652772+0000","last_active":"2024-02-13T00:46:40.423479+0000","last_peered":"2024-02-13T00:46:40.423479+0000","last_clean":"2024-02-13T00:46:40.423479+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:46:40.423479+0000","last_undegraded":"2024-02-13T00:46:40.423479+0000","last_fullsized":"2024-02-13T00:46:40.423479+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:42:19.652689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:19.652689+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":405,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538668+0000","last_change":"2024-02-13T00:44:50.430435+0000","last_active":"2024-02-13T00:46:40.538668+0000","last_peered":"2024-02-13T00:46:40.538668+0000","last_clean":"2024-02-13T00:46:40.538668+0000","last_became_active":"2024-02-13T00:44:50.429994+0000","last_became_peered":"2024-02-13T00:44:50.429994+0000","last_unstale":"2024-02-13T00:46:40.538668+0000","last_undegraded":"2024-02-13T00:46:40.538668+0000","last_fullsized":"2024-02-13T00:46:40.538668+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:20.102471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:20.102471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":321,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.422520+0000","last_change":"2024-02-13T00:44:50.434925+0000","last_active":"2024-02-13T00:46:40.422520+0000","last_peered":"2024-02-13T00:46:40.422520+0000","last_clean":"2024-02-13T00:46:40.422520+0000","last_became_active":"2024-02-13T00:44:50.433984+0000","last_became_peered":"2024-02-13T00:44:50.433984+0000","last_unstale":"2024-02-13T00:46:40.422520+0000","last_undegraded":"2024-02-13T00:46:40.422520+0000","last_fullsized":"2024-02-13T00:46:40.422520+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.512181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.512181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":408,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421996+0000","last_change":"2024-02-13T00:45:15.435610+0000","last_active":"2024-02-13T00:46:40.421996+0000","last_peered":"2024-02-13T00:46:40.421996+0000","last_clean":"2024-02-13T00:46:40.421996+0000","last_became_active":"2024-02-13T00:45:15.434853+0000","last_became_peered":"2024-02-13T00:45:15.434853+0000","last_unstale":"2024-02-13T00:46:40.421996+0000","last_undegraded":"2024-02-13T00:46:40.421996+0000","last_fullsized":"2024-02-13T00:46:40.421996+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":34,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.422091+0000","last_change":"2024-02-13T00:46:01.455201+0000","last_active":"2024-02-13T00:46:40.422091+0000","last_peered":"2024-02-13T00:46:40.422091+0000","last_clean":"2024-02-13T00:46:40.422091+0000","last_became_active":"2024-02-13T00:46:01.454882+0000","last_became_peered":"2024-02-13T00:46:01.454882+0000","last_unstale":"2024-02-13T00:46:40.422091+0000","last_undegraded":"2024-02-13T00:46:40.422091+0000","last_fullsized":"2024-02-13T00:46:40.422091+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:35.197045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:35.197045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"91'35","reported_seq":195,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529192+0000","last_change":"2024-02-13T00:46:09.256189+0000","last_active":"2024-02-13T00:46:41.529192+0000","last_peered":"2024-02-13T00:46:41.529192+0000","last_clean":"2024-02-13T00:46:41.529192+0000","last_became_active":"2024-02-13T00:46:01.459978+0000","last_became_peered":"2024-02-13T00:46:01.459978+0000","last_unstale":"2024-02-13T00:46:41.529192+0000","last_undegraded":"2024-02-13T00:46:41.529192+0000","last_fullsized":"2024-02-13T00:46:41.529192+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:44:24.196214+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:44:24.196214+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":44,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424237+0000","last_change":"2024-02-13T00:46:30.641539+0000","last_active":"2024-02-13T00:46:40.424237+0000","last_peered":"2024-02-13T00:46:40.424237+0000","last_clean":"2024-02-13T00:46:40.424237+0000","last_became_active":"2024-02-13T00:46:01.444548+0000","last_became_peered":"2024-02-13T00:46:01.444548+0000","last_unstale":"2024-02-13T00:46:40.424237+0000","last_undegraded":"2024-02-13T00:46:40.424237+0000","last_fullsized":"2024-02-13T00:46:40.424237+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.641460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.641460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":126,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423287+0000","last_change":"2024-02-13T00:46:01.455578+0000","last_active":"2024-02-13T00:46:40.423287+0000","last_peered":"2024-02-13T00:46:40.423287+0000","last_clean":"2024-02-13T00:46:40.423287+0000","last_became_active":"2024-02-13T00:46:01.454642+0000","last_became_peered":"2024-02-13T00:46:01.454642+0000","last_unstale":"2024-02-13T00:46:40.423287+0000","last_undegraded":"2024-02-13T00:46:40.423287+0000","last_fullsized":"2024-02-13T00:46:40.423287+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.592672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":247,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.531986+0000","last_change":"2024-02-13T00:45:15.429026+0000","last_active":"2024-02-13T00:46:41.531986+0000","last_peered":"2024-02-13T00:46:41.531986+0000","last_clean":"2024-02-13T00:46:41.531986+0000","last_became_active":"2024-02-13T00:45:15.427945+0000","last_became_peered":"2024-02-13T00:45:15.427945+0000","last_unstale":"2024-02-13T00:46:41.531986+0000","last_undegraded":"2024-02-13T00:46:41.531986+0000","last_fullsized":"2024-02-13T00:46:41.531986+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":350,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537891+0000","last_change":"2024-02-13T00:45:26.255907+0000","last_active":"2024-02-13T00:46:40.537891+0000","last_peered":"2024-02-13T00:46:40.537891+0000","last_clean":"2024-02-13T00:46:40.537891+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:46:40.537891+0000","last_undegraded":"2024-02-13T00:46:40.537891+0000","last_fullsized":"2024-02-13T00:46:40.537891+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.255842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.255842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":161,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537978+0000","last_change":"2024-02-13T00:44:54.041282+0000","last_active":"2024-02-13T00:46:40.537978+0000","last_peered":"2024-02-13T00:46:40.537978+0000","last_clean":"2024-02-13T00:46:40.537978+0000","last_became_active":"2024-02-13T00:44:50.432807+0000","last_became_peered":"2024-02-13T00:44:50.432807+0000","last_unstale":"2024-02-13T00:46:40.537978+0000","last_undegraded":"2024-02-13T00:46:40.537978+0000","last_fullsized":"2024-02-13T00:46:40.537978+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:42:17.413826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.413826+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":202,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529408+0000","last_change":"2024-02-13T00:46:04.708510+0000","last_active":"2024-02-13T00:46:41.529408+0000","last_peered":"2024-02-13T00:46:41.529408+0000","last_clean":"2024-02-13T00:46:41.529408+0000","last_became_active":"2024-02-13T00:46:01.447801+0000","last_became_peered":"2024-02-13T00:46:01.447801+0000","last_unstale":"2024-02-13T00:46:41.529408+0000","last_undegraded":"2024-02-13T00:46:41.529408+0000","last_fullsized":"2024-02-13T00:46:41.529408+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:45:24.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:24.269595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":385,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537607+0000","last_change":"2024-02-13T00:42:28.037452+0000","last_active":"2024-02-13T00:46:40.537607+0000","last_peered":"2024-02-13T00:46:40.537607+0000","last_clean":"2024-02-13T00:46:40.537607+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:46:40.537607+0000","last_undegraded":"2024-02-13T00:46:40.537607+0000","last_fullsized":"2024-02-13T00:46:40.537607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:28.037365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:28.037365+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":396,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534685+0000","last_change":"2024-02-13T00:44:50.441408+0000","last_active":"2024-02-13T00:46:41.534685+0000","last_peered":"2024-02-13T00:46:41.534685+0000","last_clean":"2024-02-13T00:46:41.534685+0000","last_became_active":"2024-02-13T00:44:50.440301+0000","last_became_peered":"2024-02-13T00:44:50.440301+0000","last_unstale":"2024-02-13T00:46:41.534685+0000","last_undegraded":"2024-02-13T00:46:41.534685+0000","last_fullsized":"2024-02-13T00:46:41.534685+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:25.502311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.502311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":42,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423595+0000","last_change":"2024-02-13T00:46:01.454885+0000","last_active":"2024-02-13T00:46:40.423595+0000","last_peered":"2024-02-13T00:46:40.423595+0000","last_clean":"2024-02-13T00:46:40.423595+0000","last_became_active":"2024-02-13T00:46:01.454535+0000","last_became_peered":"2024-02-13T00:46:01.454535+0000","last_unstale":"2024-02-13T00:46:40.423595+0000","last_undegraded":"2024-02-13T00:46:40.423595+0000","last_fullsized":"2024-02-13T00:46:40.423595+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.488353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.488353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":123,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534341+0000","last_change":"2024-02-13T00:44:46.481036+0000","last_active":"2024-02-13T00:46:41.534341+0000","last_peered":"2024-02-13T00:46:41.534341+0000","last_clean":"2024-02-13T00:46:41.534341+0000","last_became_active":"2024-02-13T00:44:39.523138+0000","last_became_peered":"2024-02-13T00:44:39.523138+0000","last_unstale":"2024-02-13T00:46:41.534341+0000","last_undegraded":"2024-02-13T00:46:41.534341+0000","last_fullsized":"2024-02-13T00:46:41.534341+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:43:00.776560+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.776560+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":414,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535782+0000","last_change":"2024-02-13T00:46:01.453216+0000","last_active":"2024-02-13T00:46:40.535782+0000","last_peered":"2024-02-13T00:46:40.535782+0000","last_clean":"2024-02-13T00:46:40.535782+0000","last_became_active":"2024-02-13T00:46:01.452674+0000","last_became_peered":"2024-02-13T00:46:01.452674+0000","last_unstale":"2024-02-13T00:46:40.535782+0000","last_undegraded":"2024-02-13T00:46:40.535782+0000","last_fullsized":"2024-02-13T00:46:40.535782+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:24.059313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:24.059313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"404'11","reported_seq":76,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-13T00:46:42.547278+0000","last_change":"2024-02-13T00:46:37.662485+0000","last_active":"2024-02-13T00:46:42.547278+0000","last_peered":"2024-02-13T00:46:42.547278+0000","last_clean":"2024-02-13T00:46:42.547278+0000","last_became_active":"2024-02-13T00:46:01.456187+0000","last_became_peered":"2024-02-13T00:46:01.456187+0000","last_unstale":"2024-02-13T00:46:42.547278+0000","last_undegraded":"2024-02-13T00:46:42.547278+0000","last_fullsized":"2024-02-13T00:46:42.547278+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'11","last_scrub_stamp":"2024-02-13T00:46:37.662413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.662413+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":386,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538812+0000","last_change":"2024-02-13T00:46:05.839833+0000","last_active":"2024-02-13T00:46:40.538812+0000","last_peered":"2024-02-13T00:46:40.538812+0000","last_clean":"2024-02-13T00:46:40.538812+0000","last_became_active":"2024-02-13T00:46:01.455274+0000","last_became_peered":"2024-02-13T00:46:01.455274+0000","last_unstale":"2024-02-13T00:46:40.538812+0000","last_undegraded":"2024-02-13T00:46:40.538812+0000","last_fullsized":"2024-02-13T00:46:40.538812+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:44:25.230980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.230980+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":195,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538709+0000","last_change":"2024-02-13T00:46:01.457421+0000","last_active":"2024-02-13T00:46:40.538709+0000","last_peered":"2024-02-13T00:46:40.538709+0000","last_clean":"2024-02-13T00:46:40.538709+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:46:40.538709+0000","last_undegraded":"2024-02-13T00:46:40.538709+0000","last_fullsized":"2024-02-13T00:46:40.538709+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:18.089252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:18.089252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":485,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538686+0000","last_change":"2024-02-13T00:46:05.836637+0000","last_active":"2024-02-13T00:46:40.538686+0000","last_peered":"2024-02-13T00:46:40.538686+0000","last_clean":"2024-02-13T00:46:40.538686+0000","last_became_active":"2024-02-13T00:46:01.448705+0000","last_became_peered":"2024-02-13T00:46:01.448705+0000","last_unstale":"2024-02-13T00:46:40.538686+0000","last_undegraded":"2024-02-13T00:46:40.538686+0000","last_fullsized":"2024-02-13T00:46:40.538686+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:42:21.069482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.069482+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":446,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535597+0000","last_change":"2024-02-13T00:45:15.445834+0000","last_active":"2024-02-13T00:46:40.535597+0000","last_peered":"2024-02-13T00:46:40.535597+0000","last_clean":"2024-02-13T00:46:40.535597+0000","last_became_active":"2024-02-13T00:45:15.445228+0000","last_became_peered":"2024-02-13T00:45:15.445228+0000","last_unstale":"2024-02-13T00:46:40.535597+0000","last_undegraded":"2024-02-13T00:46:40.535597+0000","last_fullsized":"2024-02-13T00:46:40.535597+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:43:34.320637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.320637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"404'10","reported_seq":68,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-13T00:46:42.548505+0000","last_change":"2024-02-13T00:46:08.351827+0000","last_active":"2024-02-13T00:46:42.548505+0000","last_peered":"2024-02-13T00:46:42.548505+0000","last_clean":"2024-02-13T00:46:42.548505+0000","last_became_active":"2024-02-13T00:46:01.448785+0000","last_became_peered":"2024-02-13T00:46:01.448785+0000","last_unstale":"2024-02-13T00:46:42.548505+0000","last_undegraded":"2024-02-13T00:46:42.548505+0000","last_fullsized":"2024-02-13T00:46:42.548505+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-02-13T00:42:23.496800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:23.496800+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":36,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537739+0000","last_change":"2024-02-13T00:46:01.457110+0000","last_active":"2024-02-13T00:46:40.537739+0000","last_peered":"2024-02-13T00:46:40.537739+0000","last_clean":"2024-02-13T00:46:40.537739+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:46:40.537739+0000","last_undegraded":"2024-02-13T00:46:40.537739+0000","last_fullsized":"2024-02-13T00:46:40.537739+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.092250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.092250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":351,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424678+0000","last_change":"2024-02-13T00:46:01.456531+0000","last_active":"2024-02-13T00:46:40.424678+0000","last_peered":"2024-02-13T00:46:40.424678+0000","last_clean":"2024-02-13T00:46:40.424678+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:46:40.424678+0000","last_undegraded":"2024-02-13T00:46:40.424678+0000","last_fullsized":"2024-02-13T00:46:40.424678+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:44:59.963616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":125,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534664+0000","last_change":"2024-02-13T00:44:45.962479+0000","last_active":"2024-02-13T00:46:41.534664+0000","last_peered":"2024-02-13T00:46:41.534664+0000","last_clean":"2024-02-13T00:46:41.534664+0000","last_became_active":"2024-02-13T00:44:39.521984+0000","last_became_peered":"2024-02-13T00:44:39.521984+0000","last_unstale":"2024-02-13T00:46:41.534664+0000","last_undegraded":"2024-02-13T00:46:41.534664+0000","last_fullsized":"2024-02-13T00:46:41.534664+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:42:52.245750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.245750+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":234,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.422350+0000","last_change":"2024-02-13T00:46:23.757417+0000","last_active":"2024-02-13T00:46:40.422350+0000","last_peered":"2024-02-13T00:46:40.422350+0000","last_clean":"2024-02-13T00:46:40.422350+0000","last_became_active":"2024-02-13T00:45:15.419239+0000","last_became_peered":"2024-02-13T00:45:15.419239+0000","last_unstale":"2024-02-13T00:46:40.422350+0000","last_undegraded":"2024-02-13T00:46:40.422350+0000","last_fullsized":"2024-02-13T00:46:40.422350+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:23.757351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:23.757351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":331,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530537+0000","last_change":"2024-02-13T00:46:01.460527+0000","last_active":"2024-02-13T00:46:41.530537+0000","last_peered":"2024-02-13T00:46:41.530537+0000","last_clean":"2024-02-13T00:46:41.530537+0000","last_became_active":"2024-02-13T00:46:01.459122+0000","last_became_peered":"2024-02-13T00:46:01.459122+0000","last_unstale":"2024-02-13T00:46:41.530537+0000","last_undegraded":"2024-02-13T00:46:41.530537+0000","last_fullsized":"2024-02-13T00:46:41.530537+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:57.546517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.546517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529217+0000","last_change":"2024-02-13T00:46:01.463229+0000","last_active":"2024-02-13T00:46:41.529217+0000","last_peered":"2024-02-13T00:46:41.529217+0000","last_clean":"2024-02-13T00:46:41.529217+0000","last_became_active":"2024-02-13T00:46:01.462717+0000","last_became_peered":"2024-02-13T00:46:01.462717+0000","last_unstale":"2024-02-13T00:46:41.529217+0000","last_undegraded":"2024-02-13T00:46:41.529217+0000","last_fullsized":"2024-02-13T00:46:41.529217+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.782345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.782345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":303,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421359+0000","last_change":"2024-02-13T00:46:01.457255+0000","last_active":"2024-02-13T00:46:40.421359+0000","last_peered":"2024-02-13T00:46:40.421359+0000","last_clean":"2024-02-13T00:46:40.421359+0000","last_became_active":"2024-02-13T00:46:01.456959+0000","last_became_peered":"2024-02-13T00:46:01.456959+0000","last_unstale":"2024-02-13T00:46:40.421359+0000","last_undegraded":"2024-02-13T00:46:40.421359+0000","last_fullsized":"2024-02-13T00:46:40.421359+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.281102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.281102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"263'10","reported_seq":381,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-13T00:46:42.437746+0000","last_change":"2024-02-13T00:43:38.251836+0000","last_active":"2024-02-13T00:46:42.437746+0000","last_peered":"2024-02-13T00:46:42.437746+0000","last_clean":"2024-02-13T00:46:42.437746+0000","last_became_active":"2024-02-13T00:43:35.454087+0000","last_became_peered":"2024-02-13T00:43:35.454087+0000","last_unstale":"2024-02-13T00:46:42.437746+0000","last_undegraded":"2024-02-13T00:46:42.437746+0000","last_fullsized":"2024-02-13T00:46:42.437746+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'9","last_scrub_stamp":"2024-02-13T00:41:44.698906+0000","last_deep_scrub":"149'9","last_deep_scrub_stamp":"2024-02-13T00:41:44.698906+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.698906+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":380,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.528905+0000","last_change":"2024-02-13T00:46:08.351239+0000","last_active":"2024-02-13T00:46:41.528905+0000","last_peered":"2024-02-13T00:46:41.528905+0000","last_clean":"2024-02-13T00:46:41.528905+0000","last_became_active":"2024-02-13T00:46:01.449654+0000","last_became_peered":"2024-02-13T00:46:01.449654+0000","last_unstale":"2024-02-13T00:46:41.528905+0000","last_undegraded":"2024-02-13T00:46:41.528905+0000","last_fullsized":"2024-02-13T00:46:41.528905+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2024-02-13T00:43:00.636798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.636798+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534802+0000","last_change":"2024-02-13T00:45:15.446786+0000","last_active":"2024-02-13T00:46:41.534802+0000","last_peered":"2024-02-13T00:46:41.534802+0000","last_clean":"2024-02-13T00:46:41.534802+0000","last_became_active":"2024-02-13T00:45:15.446443+0000","last_became_peered":"2024-02-13T00:45:15.446443+0000","last_unstale":"2024-02-13T00:46:41.534802+0000","last_undegraded":"2024-02-13T00:46:41.534802+0000","last_fullsized":"2024-02-13T00:46:41.534802+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.557475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.557475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533381+0000","last_change":"2024-02-13T00:45:15.436286+0000","last_active":"2024-02-13T00:46:41.533381+0000","last_peered":"2024-02-13T00:46:41.533381+0000","last_clean":"2024-02-13T00:46:41.533381+0000","last_became_active":"2024-02-13T00:45:15.434559+0000","last_became_peered":"2024-02-13T00:45:15.434559+0000","last_unstale":"2024-02-13T00:46:41.533381+0000","last_undegraded":"2024-02-13T00:46:41.533381+0000","last_fullsized":"2024-02-13T00:46:41.533381+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":153,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.422282+0000","last_change":"2024-02-13T00:46:01.445822+0000","last_active":"2024-02-13T00:46:40.422282+0000","last_peered":"2024-02-13T00:46:40.422282+0000","last_clean":"2024-02-13T00:46:40.422282+0000","last_became_active":"2024-02-13T00:46:01.444661+0000","last_became_peered":"2024-02-13T00:46:01.444661+0000","last_unstale":"2024-02-13T00:46:40.422282+0000","last_undegraded":"2024-02-13T00:46:40.422282+0000","last_fullsized":"2024-02-13T00:46:40.422282+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:43.049231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:43.049231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":194,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421502+0000","last_change":"2024-02-13T00:46:01.454510+0000","last_active":"2024-02-13T00:46:40.421502+0000","last_peered":"2024-02-13T00:46:40.421502+0000","last_clean":"2024-02-13T00:46:40.421502+0000","last_became_active":"2024-02-13T00:46:01.453842+0000","last_became_peered":"2024-02-13T00:46:01.453842+0000","last_unstale":"2024-02-13T00:46:40.421502+0000","last_undegraded":"2024-02-13T00:46:40.421502+0000","last_fullsized":"2024-02-13T00:46:40.421502+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:54.842056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:54.842056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":337,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.422327+0000","last_change":"2024-02-13T00:46:04.744348+0000","last_active":"2024-02-13T00:46:40.422327+0000","last_peered":"2024-02-13T00:46:40.422327+0000","last_clean":"2024-02-13T00:46:40.422327+0000","last_became_active":"2024-02-13T00:46:01.445058+0000","last_became_peered":"2024-02-13T00:46:01.445058+0000","last_unstale":"2024-02-13T00:46:40.422327+0000","last_undegraded":"2024-02-13T00:46:40.422327+0000","last_fullsized":"2024-02-13T00:46:40.422327+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:42:53.881594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.881594+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":414,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538626+0000","last_change":"2024-02-13T00:44:39.503982+0000","last_active":"2024-02-13T00:46:40.538626+0000","last_peered":"2024-02-13T00:46:40.538626+0000","last_clean":"2024-02-13T00:46:40.538626+0000","last_became_active":"2024-02-13T00:44:39.503590+0000","last_became_peered":"2024-02-13T00:44:39.503590+0000","last_unstale":"2024-02-13T00:46:40.538626+0000","last_undegraded":"2024-02-13T00:46:40.538626+0000","last_fullsized":"2024-02-13T00:46:40.538626+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.236727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.236727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":364,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529898+0000","last_change":"2024-02-13T00:46:01.448313+0000","last_active":"2024-02-13T00:46:41.529898+0000","last_peered":"2024-02-13T00:46:41.529898+0000","last_clean":"2024-02-13T00:46:41.529898+0000","last_became_active":"2024-02-13T00:46:01.447815+0000","last_became_peered":"2024-02-13T00:46:01.447815+0000","last_unstale":"2024-02-13T00:46:41.529898+0000","last_undegraded":"2024-02-13T00:46:41.529898+0000","last_fullsized":"2024-02-13T00:46:41.529898+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":351,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533165+0000","last_change":"2024-02-13T00:44:50.436745+0000","last_active":"2024-02-13T00:46:41.533165+0000","last_peered":"2024-02-13T00:46:41.533165+0000","last_clean":"2024-02-13T00:46:41.533165+0000","last_became_active":"2024-02-13T00:44:50.436058+0000","last_became_peered":"2024-02-13T00:44:50.436058+0000","last_unstale":"2024-02-13T00:46:41.533165+0000","last_undegraded":"2024-02-13T00:46:41.533165+0000","last_fullsized":"2024-02-13T00:46:41.533165+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:48.300400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.300400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":125,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529033+0000","last_change":"2024-02-13T00:46:01.451625+0000","last_active":"2024-02-13T00:46:41.529033+0000","last_peered":"2024-02-13T00:46:41.529033+0000","last_clean":"2024-02-13T00:46:41.529033+0000","last_became_active":"2024-02-13T00:46:01.450587+0000","last_became_peered":"2024-02-13T00:46:01.450587+0000","last_unstale":"2024-02-13T00:46:41.529033+0000","last_undegraded":"2024-02-13T00:46:41.529033+0000","last_fullsized":"2024-02-13T00:46:41.529033+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:33.204969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:33.204969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":503,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423152+0000","last_change":"2024-02-13T00:46:04.826799+0000","last_active":"2024-02-13T00:46:40.423152+0000","last_peered":"2024-02-13T00:46:40.423152+0000","last_clean":"2024-02-13T00:46:40.423152+0000","last_became_active":"2024-02-13T00:46:01.443777+0000","last_became_peered":"2024-02-13T00:46:01.443777+0000","last_unstale":"2024-02-13T00:46:40.423152+0000","last_undegraded":"2024-02-13T00:46:40.423152+0000","last_fullsized":"2024-02-13T00:46:40.423152+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:42:20.688132+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":234,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424148+0000","last_change":"2024-02-13T00:45:15.423607+0000","last_active":"2024-02-13T00:46:40.424148+0000","last_peered":"2024-02-13T00:46:40.424148+0000","last_clean":"2024-02-13T00:46:40.424148+0000","last_became_active":"2024-02-13T00:45:15.423415+0000","last_became_peered":"2024-02-13T00:45:15.423415+0000","last_unstale":"2024-02-13T00:46:40.424148+0000","last_undegraded":"2024-02-13T00:46:40.424148+0000","last_fullsized":"2024-02-13T00:46:40.424148+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:17.295459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:17.295459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":426,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535693+0000","last_change":"2024-02-13T00:45:15.445862+0000","last_active":"2024-02-13T00:46:40.535693+0000","last_peered":"2024-02-13T00:46:40.535693+0000","last_clean":"2024-02-13T00:46:40.535693+0000","last_became_active":"2024-02-13T00:45:15.445557+0000","last_became_peered":"2024-02-13T00:45:15.445557+0000","last_unstale":"2024-02-13T00:46:40.535693+0000","last_undegraded":"2024-02-13T00:46:40.535693+0000","last_fullsized":"2024-02-13T00:46:40.535693+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":39,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537831+0000","last_change":"2024-02-13T00:46:01.442629+0000","last_active":"2024-02-13T00:46:40.537831+0000","last_peered":"2024-02-13T00:46:40.537831+0000","last_clean":"2024-02-13T00:46:40.537831+0000","last_became_active":"2024-02-13T00:46:01.441421+0000","last_became_peered":"2024-02-13T00:46:01.441421+0000","last_unstale":"2024-02-13T00:46:40.537831+0000","last_undegraded":"2024-02-13T00:46:40.537831+0000","last_fullsized":"2024-02-13T00:46:40.537831+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:50.325063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:50.325063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":36,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538983+0000","last_change":"2024-02-13T00:46:01.450756+0000","last_active":"2024-02-13T00:46:40.538983+0000","last_peered":"2024-02-13T00:46:40.538983+0000","last_clean":"2024-02-13T00:46:40.538983+0000","last_became_active":"2024-02-13T00:46:01.449142+0000","last_became_peered":"2024-02-13T00:46:01.449142+0000","last_unstale":"2024-02-13T00:46:40.538983+0000","last_undegraded":"2024-02-13T00:46:40.538983+0000","last_fullsized":"2024-02-13T00:46:40.538983+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:54.422484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:54.422484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":500,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424433+0000","last_change":"2024-02-13T00:46:34.691109+0000","last_active":"2024-02-13T00:46:40.424433+0000","last_peered":"2024-02-13T00:46:40.424433+0000","last_clean":"2024-02-13T00:46:40.424433+0000","last_became_active":"2024-02-13T00:46:01.448163+0000","last_became_peered":"2024-02-13T00:46:01.448163+0000","last_unstale":"2024-02-13T00:46:40.424433+0000","last_undegraded":"2024-02-13T00:46:40.424433+0000","last_fullsized":"2024-02-13T00:46:40.424433+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:46:34.690980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.690980+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":127,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535674+0000","last_change":"2024-02-13T00:46:37.056299+0000","last_active":"2024-02-13T00:46:40.535674+0000","last_peered":"2024-02-13T00:46:40.535674+0000","last_clean":"2024-02-13T00:46:40.535674+0000","last_became_active":"2024-02-13T00:46:01.446104+0000","last_became_peered":"2024-02-13T00:46:01.446104+0000","last_unstale":"2024-02-13T00:46:40.535674+0000","last_undegraded":"2024-02-13T00:46:40.535674+0000","last_fullsized":"2024-02-13T00:46:40.535674+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:37.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":410,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421825+0000","last_change":"2024-02-13T00:46:01.431899+0000","last_active":"2024-02-13T00:46:40.421825+0000","last_peered":"2024-02-13T00:46:40.421825+0000","last_clean":"2024-02-13T00:46:40.421825+0000","last_became_active":"2024-02-13T00:46:01.430623+0000","last_became_peered":"2024-02-13T00:46:01.430623+0000","last_unstale":"2024-02-13T00:46:40.421825+0000","last_undegraded":"2024-02-13T00:46:40.421825+0000","last_fullsized":"2024-02-13T00:46:40.421825+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:46.556229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.556229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":417,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537473+0000","last_change":"2024-02-13T00:46:01.450548+0000","last_active":"2024-02-13T00:46:40.537473+0000","last_peered":"2024-02-13T00:46:40.537473+0000","last_clean":"2024-02-13T00:46:40.537473+0000","last_became_active":"2024-02-13T00:46:01.450176+0000","last_became_peered":"2024-02-13T00:46:01.450176+0000","last_unstale":"2024-02-13T00:46:40.537473+0000","last_undegraded":"2024-02-13T00:46:40.537473+0000","last_fullsized":"2024-02-13T00:46:40.537473+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":271,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529605+0000","last_change":"2024-02-13T00:46:01.460470+0000","last_active":"2024-02-13T00:46:41.529605+0000","last_peered":"2024-02-13T00:46:41.529605+0000","last_clean":"2024-02-13T00:46:41.529605+0000","last_became_active":"2024-02-13T00:46:01.459859+0000","last_became_peered":"2024-02-13T00:46:01.459859+0000","last_unstale":"2024-02-13T00:46:41.529605+0000","last_undegraded":"2024-02-13T00:46:41.529605+0000","last_fullsized":"2024-02-13T00:46:41.529605+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:01.179031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:01.179031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":67,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423298+0000","last_change":"2024-02-13T00:46:01.456508+0000","last_active":"2024-02-13T00:46:40.423298+0000","last_peered":"2024-02-13T00:46:40.423298+0000","last_clean":"2024-02-13T00:46:40.423298+0000","last_became_active":"2024-02-13T00:46:01.455170+0000","last_became_peered":"2024-02-13T00:46:01.455170+0000","last_unstale":"2024-02-13T00:46:40.423298+0000","last_undegraded":"2024-02-13T00:46:40.423298+0000","last_fullsized":"2024-02-13T00:46:40.423298+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:56.666003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.666003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"157'21","reported_seq":152,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534606+0000","last_change":"2024-02-13T00:45:18.327225+0000","last_active":"2024-02-13T00:46:41.534606+0000","last_peered":"2024-02-13T00:46:41.534606+0000","last_clean":"2024-02-13T00:46:41.534606+0000","last_became_active":"2024-02-13T00:45:15.446969+0000","last_became_peered":"2024-02-13T00:45:15.446969+0000","last_unstale":"2024-02-13T00:46:41.534606+0000","last_undegraded":"2024-02-13T00:46:41.534606+0000","last_fullsized":"2024-02-13T00:46:41.534606+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.415907+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":370,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421642+0000","last_change":"2024-02-13T00:45:15.434844+0000","last_active":"2024-02-13T00:46:40.421642+0000","last_peered":"2024-02-13T00:46:40.421642+0000","last_clean":"2024-02-13T00:46:40.421642+0000","last_became_active":"2024-02-13T00:45:15.434498+0000","last_became_peered":"2024-02-13T00:45:15.434498+0000","last_unstale":"2024-02-13T00:46:40.421642+0000","last_undegraded":"2024-02-13T00:46:40.421642+0000","last_fullsized":"2024-02-13T00:46:40.421642+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:23.561163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:42:23.561163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":368,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529287+0000","last_change":"2024-02-13T00:46:01.448309+0000","last_active":"2024-02-13T00:46:41.529287+0000","last_peered":"2024-02-13T00:46:41.529287+0000","last_clean":"2024-02-13T00:46:41.529287+0000","last_became_active":"2024-02-13T00:46:01.447823+0000","last_became_peered":"2024-02-13T00:46:01.447823+0000","last_unstale":"2024-02-13T00:46:41.529287+0000","last_undegraded":"2024-02-13T00:46:41.529287+0000","last_fullsized":"2024-02-13T00:46:41.529287+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.472312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.472312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":241,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530358+0000","last_change":"2024-02-13T00:46:01.460412+0000","last_active":"2024-02-13T00:46:41.530358+0000","last_peered":"2024-02-13T00:46:41.530358+0000","last_clean":"2024-02-13T00:46:41.530358+0000","last_became_active":"2024-02-13T00:46:01.460001+0000","last_became_peered":"2024-02-13T00:46:01.460001+0000","last_unstale":"2024-02-13T00:46:41.530358+0000","last_undegraded":"2024-02-13T00:46:41.530358+0000","last_fullsized":"2024-02-13T00:46:41.530358+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.544320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.544320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":265,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537925+0000","last_change":"2024-02-13T00:46:01.450879+0000","last_active":"2024-02-13T00:46:40.537925+0000","last_peered":"2024-02-13T00:46:40.537925+0000","last_clean":"2024-02-13T00:46:40.537925+0000","last_became_active":"2024-02-13T00:46:01.450088+0000","last_became_peered":"2024-02-13T00:46:01.450088+0000","last_unstale":"2024-02-13T00:46:40.537925+0000","last_undegraded":"2024-02-13T00:46:40.537925+0000","last_fullsized":"2024-02-13T00:46:40.537925+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":351,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424360+0000","last_change":"2024-02-13T00:46:01.459317+0000","last_active":"2024-02-13T00:46:40.424360+0000","last_peered":"2024-02-13T00:46:40.424360+0000","last_clean":"2024-02-13T00:46:40.424360+0000","last_became_active":"2024-02-13T00:46:01.458408+0000","last_became_peered":"2024-02-13T00:46:01.458408+0000","last_unstale":"2024-02-13T00:46:40.424360+0000","last_undegraded":"2024-02-13T00:46:40.424360+0000","last_fullsized":"2024-02-13T00:46:40.424360+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:16.256516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:16.256516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":73,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537656+0000","last_change":"2024-02-13T00:46:01.442001+0000","last_active":"2024-02-13T00:46:40.537656+0000","last_peered":"2024-02-13T00:46:40.537656+0000","last_clean":"2024-02-13T00:46:40.537656+0000","last_became_active":"2024-02-13T00:46:01.441013+0000","last_became_peered":"2024-02-13T00:46:01.441013+0000","last_unstale":"2024-02-13T00:46:40.537656+0000","last_undegraded":"2024-02-13T00:46:40.537656+0000","last_fullsized":"2024-02-13T00:46:40.537656+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:26.313580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:26.313580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":450,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538805+0000","last_change":"2024-02-13T00:46:04.714305+0000","last_active":"2024-02-13T00:46:40.538805+0000","last_peered":"2024-02-13T00:46:40.538805+0000","last_clean":"2024-02-13T00:46:40.538805+0000","last_became_active":"2024-02-13T00:46:01.448219+0000","last_became_peered":"2024-02-13T00:46:01.448219+0000","last_unstale":"2024-02-13T00:46:40.538805+0000","last_undegraded":"2024-02-13T00:46:40.538805+0000","last_fullsized":"2024-02-13T00:46:40.538805+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:42:54.174950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:54.174950+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":325,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421727+0000","last_change":"2024-02-13T00:44:50.426201+0000","last_active":"2024-02-13T00:46:40.421727+0000","last_peered":"2024-02-13T00:46:40.421727+0000","last_clean":"2024-02-13T00:46:40.421727+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:46:40.421727+0000","last_undegraded":"2024-02-13T00:46:40.421727+0000","last_fullsized":"2024-02-13T00:46:40.421727+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:20.481083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:44:20.481083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":971,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-13T00:46:43.639615+0000","last_change":"2024-02-13T00:45:54.075270+0000","last_active":"2024-02-13T00:46:43.639615+0000","last_peered":"2024-02-13T00:46:43.639615+0000","last_clean":"2024-02-13T00:46:43.639615+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:46:43.639615+0000","last_undegraded":"2024-02-13T00:46:43.639615+0000","last_fullsized":"2024-02-13T00:46:43.639615+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:45:54.075194+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":531,"num_read_kb":526,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":138,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530257+0000","last_change":"2024-02-13T00:46:01.460282+0000","last_active":"2024-02-13T00:46:41.530257+0000","last_peered":"2024-02-13T00:46:41.530257+0000","last_clean":"2024-02-13T00:46:41.530257+0000","last_became_active":"2024-02-13T00:46:01.459473+0000","last_became_peered":"2024-02-13T00:46:01.459473+0000","last_unstale":"2024-02-13T00:46:41.530257+0000","last_undegraded":"2024-02-13T00:46:41.530257+0000","last_fullsized":"2024-02-13T00:46:41.530257+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":190,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.536900+0000","last_change":"2024-02-13T00:45:37.135398+0000","last_active":"2024-02-13T00:46:40.536900+0000","last_peered":"2024-02-13T00:46:40.536900+0000","last_clean":"2024-02-13T00:46:40.536900+0000","last_became_active":"2024-02-13T00:45:15.435588+0000","last_became_peered":"2024-02-13T00:45:15.435588+0000","last_unstale":"2024-02-13T00:46:40.536900+0000","last_undegraded":"2024-02-13T00:46:40.536900+0000","last_fullsized":"2024-02-13T00:46:40.536900+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.135332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.135332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":89,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534312+0000","last_change":"2024-02-13T00:45:31.183763+0000","last_active":"2024-02-13T00:46:41.534312+0000","last_peered":"2024-02-13T00:46:41.534312+0000","last_clean":"2024-02-13T00:46:41.534312+0000","last_became_active":"2024-02-13T00:45:06.421025+0000","last_became_peered":"2024-02-13T00:45:06.421025+0000","last_unstale":"2024-02-13T00:46:41.534312+0000","last_undegraded":"2024-02-13T00:46:41.534312+0000","last_fullsized":"2024-02-13T00:46:41.534312+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.183682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.183682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"405'53","reported_seq":67,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-13T00:46:42.441662+0000","last_change":"2024-02-13T00:46:04.950162+0000","last_active":"2024-02-13T00:46:42.441662+0000","last_peered":"2024-02-13T00:46:42.441662+0000","last_clean":"2024-02-13T00:46:42.441662+0000","last_became_active":"2024-02-13T00:46:01.449059+0000","last_became_peered":"2024-02-13T00:46:01.449059+0000","last_unstale":"2024-02-13T00:46:42.441662+0000","last_undegraded":"2024-02-13T00:46:42.441662+0000","last_fullsized":"2024-02-13T00:46:42.441662+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'48","last_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:43:48.212428+0000","log_size":53,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":2148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":335,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535628+0000","last_change":"2024-02-13T00:46:01.453042+0000","last_active":"2024-02-13T00:46:40.535628+0000","last_peered":"2024-02-13T00:46:40.535628+0000","last_clean":"2024-02-13T00:46:40.535628+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:46:40.535628+0000","last_undegraded":"2024-02-13T00:46:40.535628+0000","last_fullsized":"2024-02-13T00:46:40.535628+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.045912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.045912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529799+0000","last_change":"2024-02-13T00:46:01.461717+0000","last_active":"2024-02-13T00:46:41.529799+0000","last_peered":"2024-02-13T00:46:41.529799+0000","last_clean":"2024-02-13T00:46:41.529799+0000","last_became_active":"2024-02-13T00:46:01.460410+0000","last_became_peered":"2024-02-13T00:46:01.460410+0000","last_unstale":"2024-02-13T00:46:41.529799+0000","last_undegraded":"2024-02-13T00:46:41.529799+0000","last_fullsized":"2024-02-13T00:46:41.529799+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":120,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533265+0000","last_change":"2024-02-13T00:44:50.439369+0000","last_active":"2024-02-13T00:46:41.533265+0000","last_peered":"2024-02-13T00:46:41.533265+0000","last_clean":"2024-02-13T00:46:41.533265+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:46:41.533265+0000","last_undegraded":"2024-02-13T00:46:41.533265+0000","last_fullsized":"2024-02-13T00:46:41.533265+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:19.407011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:19.407011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":554,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-13T00:46:42.547496+0000","last_change":"2024-02-13T00:46:11.962872+0000","last_active":"2024-02-13T00:46:42.547496+0000","last_peered":"2024-02-13T00:46:42.547496+0000","last_clean":"2024-02-13T00:46:42.547496+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:46:42.547496+0000","last_undegraded":"2024-02-13T00:46:42.547496+0000","last_fullsized":"2024-02-13T00:46:42.547496+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'11","last_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:43:02.423358+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":138,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535974+0000","last_change":"2024-02-13T00:46:02.664116+0000","last_active":"2024-02-13T00:46:40.535974+0000","last_peered":"2024-02-13T00:46:40.535974+0000","last_clean":"2024-02-13T00:46:40.535974+0000","last_became_active":"2024-02-13T00:46:01.442003+0000","last_became_peered":"2024-02-13T00:46:01.442003+0000","last_unstale":"2024-02-13T00:46:40.535974+0000","last_undegraded":"2024-02-13T00:46:40.535974+0000","last_fullsized":"2024-02-13T00:46:40.535974+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:52.517392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.517392+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.536994+0000","last_change":"2024-02-13T00:45:15.423417+0000","last_active":"2024-02-13T00:46:40.536994+0000","last_peered":"2024-02-13T00:46:40.536994+0000","last_clean":"2024-02-13T00:46:40.536994+0000","last_became_active":"2024-02-13T00:45:15.422546+0000","last_became_peered":"2024-02-13T00:45:15.422546+0000","last_unstale":"2024-02-13T00:46:40.536994+0000","last_undegraded":"2024-02-13T00:46:40.536994+0000","last_fullsized":"2024-02-13T00:46:40.536994+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:41.223028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:43:41.223028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":417,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421611+0000","last_change":"2024-02-13T00:46:01.436997+0000","last_active":"2024-02-13T00:46:40.421611+0000","last_peered":"2024-02-13T00:46:40.421611+0000","last_clean":"2024-02-13T00:46:40.421611+0000","last_became_active":"2024-02-13T00:46:01.435178+0000","last_became_peered":"2024-02-13T00:46:01.435178+0000","last_unstale":"2024-02-13T00:46:40.421611+0000","last_undegraded":"2024-02-13T00:46:40.421611+0000","last_fullsized":"2024-02-13T00:46:40.421611+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":350,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535433+0000","last_change":"2024-02-13T00:45:15.445626+0000","last_active":"2024-02-13T00:46:40.535433+0000","last_peered":"2024-02-13T00:46:40.535433+0000","last_clean":"2024-02-13T00:46:40.535433+0000","last_became_active":"2024-02-13T00:45:15.445117+0000","last_became_peered":"2024-02-13T00:45:15.445117+0000","last_unstale":"2024-02-13T00:46:40.535433+0000","last_undegraded":"2024-02-13T00:46:40.535433+0000","last_fullsized":"2024-02-13T00:46:40.535433+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.171185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":117,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.422485+0000","last_change":"2024-02-13T00:46:01.436795+0000","last_active":"2024-02-13T00:46:40.422485+0000","last_peered":"2024-02-13T00:46:40.422485+0000","last_clean":"2024-02-13T00:46:40.422485+0000","last_became_active":"2024-02-13T00:46:01.434431+0000","last_became_peered":"2024-02-13T00:46:01.434431+0000","last_unstale":"2024-02-13T00:46:40.422485+0000","last_undegraded":"2024-02-13T00:46:40.422485+0000","last_fullsized":"2024-02-13T00:46:40.422485+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.592320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.592320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":462,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535409+0000","last_change":"2024-02-13T00:44:41.502069+0000","last_active":"2024-02-13T00:46:40.535409+0000","last_peered":"2024-02-13T00:46:40.535409+0000","last_clean":"2024-02-13T00:46:40.535409+0000","last_became_active":"2024-02-13T00:44:39.501886+0000","last_became_peered":"2024-02-13T00:44:39.501886+0000","last_unstale":"2024-02-13T00:46:40.535409+0000","last_undegraded":"2024-02-13T00:46:40.535409+0000","last_fullsized":"2024-02-13T00:46:40.535409+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'24","last_scrub_stamp":"2024-02-13T00:43:30.413372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.413372+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":36,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":391,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424244+0000","last_change":"2024-02-13T00:46:14.266858+0000","last_active":"2024-02-13T00:46:40.424244+0000","last_peered":"2024-02-13T00:46:40.424244+0000","last_clean":"2024-02-13T00:46:40.424244+0000","last_became_active":"2024-02-13T00:46:01.457926+0000","last_became_peered":"2024-02-13T00:46:01.457926+0000","last_unstale":"2024-02-13T00:46:40.424244+0000","last_undegraded":"2024-02-13T00:46:40.424244+0000","last_fullsized":"2024-02-13T00:46:40.424244+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:43:35.309308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:35.309308+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":196,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.536730+0000","last_change":"2024-02-13T00:45:15.435701+0000","last_active":"2024-02-13T00:46:40.536730+0000","last_peered":"2024-02-13T00:46:40.536730+0000","last_clean":"2024-02-13T00:46:40.536730+0000","last_became_active":"2024-02-13T00:45:15.435404+0000","last_became_peered":"2024-02-13T00:45:15.435404+0000","last_unstale":"2024-02-13T00:46:40.536730+0000","last_undegraded":"2024-02-13T00:46:40.536730+0000","last_fullsized":"2024-02-13T00:46:40.536730+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:41:45.660653+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":351,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424486+0000","last_change":"2024-02-13T00:46:01.450436+0000","last_active":"2024-02-13T00:46:40.424486+0000","last_peered":"2024-02-13T00:46:40.424486+0000","last_clean":"2024-02-13T00:46:40.424486+0000","last_became_active":"2024-02-13T00:46:01.449234+0000","last_became_peered":"2024-02-13T00:46:01.449234+0000","last_unstale":"2024-02-13T00:46:40.424486+0000","last_undegraded":"2024-02-13T00:46:40.424486+0000","last_fullsized":"2024-02-13T00:46:40.424486+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.110742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.110742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":528,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424429+0000","last_change":"2024-02-13T00:46:12.563543+0000","last_active":"2024-02-13T00:46:40.424429+0000","last_peered":"2024-02-13T00:46:40.424429+0000","last_clean":"2024-02-13T00:46:40.424429+0000","last_became_active":"2024-02-13T00:46:01.456876+0000","last_became_peered":"2024-02-13T00:46:01.456876+0000","last_unstale":"2024-02-13T00:46:40.424429+0000","last_undegraded":"2024-02-13T00:46:40.424429+0000","last_fullsized":"2024-02-13T00:46:40.424429+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:43:30.365536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.365536+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":108,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535664+0000","last_change":"2024-02-13T00:46:30.095387+0000","last_active":"2024-02-13T00:46:40.535664+0000","last_peered":"2024-02-13T00:46:40.535664+0000","last_clean":"2024-02-13T00:46:40.535664+0000","last_became_active":"2024-02-13T00:44:50.441300+0000","last_became_peered":"2024-02-13T00:44:50.441300+0000","last_unstale":"2024-02-13T00:46:40.535664+0000","last_undegraded":"2024-02-13T00:46:40.535664+0000","last_fullsized":"2024-02-13T00:46:40.535664+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.095307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":187,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538779+0000","last_change":"2024-02-13T00:45:15.416946+0000","last_active":"2024-02-13T00:46:40.538779+0000","last_peered":"2024-02-13T00:46:40.538779+0000","last_clean":"2024-02-13T00:46:40.538779+0000","last_became_active":"2024-02-13T00:45:15.416364+0000","last_became_peered":"2024-02-13T00:45:15.416364+0000","last_unstale":"2024-02-13T00:46:40.538779+0000","last_undegraded":"2024-02-13T00:46:40.538779+0000","last_fullsized":"2024-02-13T00:46:40.538779+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.458752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.458752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":124,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423479+0000","last_change":"2024-02-13T00:46:01.454597+0000","last_active":"2024-02-13T00:46:40.423479+0000","last_peered":"2024-02-13T00:46:40.423479+0000","last_clean":"2024-02-13T00:46:40.423479+0000","last_became_active":"2024-02-13T00:46:01.454101+0000","last_became_peered":"2024-02-13T00:46:01.454101+0000","last_unstale":"2024-02-13T00:46:40.423479+0000","last_undegraded":"2024-02-13T00:46:40.423479+0000","last_fullsized":"2024-02-13T00:46:40.423479+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.651091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.651091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":120,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534737+0000","last_change":"2024-02-13T00:44:50.441196+0000","last_active":"2024-02-13T00:46:41.534737+0000","last_peered":"2024-02-13T00:46:41.534737+0000","last_clean":"2024-02-13T00:46:41.534737+0000","last_became_active":"2024-02-13T00:44:50.438274+0000","last_became_peered":"2024-02-13T00:44:50.438274+0000","last_unstale":"2024-02-13T00:46:41.534737+0000","last_undegraded":"2024-02-13T00:46:41.534737+0000","last_fullsized":"2024-02-13T00:46:41.534737+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.510219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.510219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":268,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538922+0000","last_change":"2024-02-13T00:46:34.141015+0000","last_active":"2024-02-13T00:46:40.538922+0000","last_peered":"2024-02-13T00:46:40.538922+0000","last_clean":"2024-02-13T00:46:40.538922+0000","last_became_active":"2024-02-13T00:45:15.436020+0000","last_became_peered":"2024-02-13T00:45:15.436020+0000","last_unstale":"2024-02-13T00:46:40.538922+0000","last_undegraded":"2024-02-13T00:46:40.538922+0000","last_fullsized":"2024-02-13T00:46:40.538922+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:46:34.140928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.140928+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":116,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534193+0000","last_change":"2024-02-13T00:45:00.311635+0000","last_active":"2024-02-13T00:46:41.534193+0000","last_peered":"2024-02-13T00:46:41.534193+0000","last_clean":"2024-02-13T00:46:41.534193+0000","last_became_active":"2024-02-13T00:44:39.512402+0000","last_became_peered":"2024-02-13T00:44:39.512402+0000","last_unstale":"2024-02-13T00:46:41.534193+0000","last_undegraded":"2024-02-13T00:46:41.534193+0000","last_fullsized":"2024-02-13T00:46:41.534193+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:00.311558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:45:00.311558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":331,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538725+0000","last_change":"2024-02-13T00:46:01.455806+0000","last_active":"2024-02-13T00:46:40.538725+0000","last_peered":"2024-02-13T00:46:40.538725+0000","last_clean":"2024-02-13T00:46:40.538725+0000","last_became_active":"2024-02-13T00:46:01.453265+0000","last_became_peered":"2024-02-13T00:46:01.453265+0000","last_unstale":"2024-02-13T00:46:40.538725+0000","last_undegraded":"2024-02-13T00:46:40.538725+0000","last_fullsized":"2024-02-13T00:46:40.538725+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.106639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.106639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529914+0000","last_change":"2024-02-13T00:46:01.451840+0000","last_active":"2024-02-13T00:46:41.529914+0000","last_peered":"2024-02-13T00:46:41.529914+0000","last_clean":"2024-02-13T00:46:41.529914+0000","last_became_active":"2024-02-13T00:46:01.450312+0000","last_became_peered":"2024-02-13T00:46:01.450312+0000","last_unstale":"2024-02-13T00:46:41.529914+0000","last_undegraded":"2024-02-13T00:46:41.529914+0000","last_fullsized":"2024-02-13T00:46:41.529914+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.042291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.042291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":201,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424082+0000","last_change":"2024-02-13T00:46:01.454858+0000","last_active":"2024-02-13T00:46:40.424082+0000","last_peered":"2024-02-13T00:46:40.424082+0000","last_clean":"2024-02-13T00:46:40.424082+0000","last_became_active":"2024-02-13T00:46:01.454284+0000","last_became_peered":"2024-02-13T00:46:01.454284+0000","last_unstale":"2024-02-13T00:46:40.424082+0000","last_undegraded":"2024-02-13T00:46:40.424082+0000","last_fullsized":"2024-02-13T00:46:40.424082+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:34.523594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.523594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":340,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537371+0000","last_change":"2024-02-13T00:46:04.434215+0000","last_active":"2024-02-13T00:46:40.537371+0000","last_peered":"2024-02-13T00:46:40.537371+0000","last_clean":"2024-02-13T00:46:40.537371+0000","last_became_active":"2024-02-13T00:46:01.450113+0000","last_became_peered":"2024-02-13T00:46:01.450113+0000","last_unstale":"2024-02-13T00:46:40.537371+0000","last_undegraded":"2024-02-13T00:46:40.537371+0000","last_fullsized":"2024-02-13T00:46:40.537371+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:42:48.774168+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.774168+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":371,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423258+0000","last_change":"2024-02-13T00:46:01.454689+0000","last_active":"2024-02-13T00:46:40.423258+0000","last_peered":"2024-02-13T00:46:40.423258+0000","last_clean":"2024-02-13T00:46:40.423258+0000","last_became_active":"2024-02-13T00:46:01.454310+0000","last_became_peered":"2024-02-13T00:46:01.454310+0000","last_unstale":"2024-02-13T00:46:40.423258+0000","last_undegraded":"2024-02-13T00:46:40.423258+0000","last_fullsized":"2024-02-13T00:46:40.423258+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.230330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.230330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"404'9","reported_seq":68,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-13T00:46:42.441410+0000","last_change":"2024-02-13T00:46:04.816167+0000","last_active":"2024-02-13T00:46:42.441410+0000","last_peered":"2024-02-13T00:46:42.441410+0000","last_clean":"2024-02-13T00:46:42.441410+0000","last_became_active":"2024-02-13T00:46:01.448144+0000","last_became_peered":"2024-02-13T00:46:01.448144+0000","last_unstale":"2024-02-13T00:46:42.441410+0000","last_undegraded":"2024-02-13T00:46:42.441410+0000","last_fullsized":"2024-02-13T00:46:42.441410+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:37.452882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:37.452882+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":408,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424367+0000","last_change":"2024-02-13T00:46:01.449886+0000","last_active":"2024-02-13T00:46:40.424367+0000","last_peered":"2024-02-13T00:46:40.424367+0000","last_clean":"2024-02-13T00:46:40.424367+0000","last_became_active":"2024-02-13T00:46:01.447236+0000","last_became_peered":"2024-02-13T00:46:01.447236+0000","last_unstale":"2024-02-13T00:46:40.424367+0000","last_undegraded":"2024-02-13T00:46:40.424367+0000","last_fullsized":"2024-02-13T00:46:40.424367+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:52.391906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.391906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":79,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423895+0000","last_change":"2024-02-13T00:46:01.456618+0000","last_active":"2024-02-13T00:46:40.423895+0000","last_peered":"2024-02-13T00:46:40.423895+0000","last_clean":"2024-02-13T00:46:40.423895+0000","last_became_active":"2024-02-13T00:46:01.455989+0000","last_became_peered":"2024-02-13T00:46:01.455989+0000","last_unstale":"2024-02-13T00:46:40.423895+0000","last_undegraded":"2024-02-13T00:46:40.423895+0000","last_fullsized":"2024-02-13T00:46:40.423895+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:16.499780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:16.499780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":384,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.531632+0000","last_change":"2024-02-13T00:46:10.157189+0000","last_active":"2024-02-13T00:46:41.531632+0000","last_peered":"2024-02-13T00:46:41.531632+0000","last_clean":"2024-02-13T00:46:41.531632+0000","last_became_active":"2024-02-13T00:46:01.460321+0000","last_became_peered":"2024-02-13T00:46:01.460321+0000","last_unstale":"2024-02-13T00:46:41.531632+0000","last_undegraded":"2024-02-13T00:46:41.531632+0000","last_fullsized":"2024-02-13T00:46:41.531632+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:36.477034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.477034+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":423,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424140+0000","last_change":"2024-02-13T00:46:01.432179+0000","last_active":"2024-02-13T00:46:40.424140+0000","last_peered":"2024-02-13T00:46:40.424140+0000","last_clean":"2024-02-13T00:46:40.424140+0000","last_became_active":"2024-02-13T00:46:01.431926+0000","last_became_peered":"2024-02-13T00:46:01.431926+0000","last_unstale":"2024-02-13T00:46:40.424140+0000","last_undegraded":"2024-02-13T00:46:40.424140+0000","last_fullsized":"2024-02-13T00:46:40.424140+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:40.335892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:40.335892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"363'14","reported_seq":211,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-13T00:46:42.441609+0000","last_change":"2024-02-13T00:45:17.693600+0000","last_active":"2024-02-13T00:46:42.441609+0000","last_peered":"2024-02-13T00:46:42.441609+0000","last_clean":"2024-02-13T00:46:42.441609+0000","last_became_active":"2024-02-13T00:45:15.423066+0000","last_became_peered":"2024-02-13T00:45:15.423066+0000","last_unstale":"2024-02-13T00:46:42.441609+0000","last_undegraded":"2024-02-13T00:46:42.441609+0000","last_fullsized":"2024-02-13T00:46:42.441609+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-02-13T00:41:32.715550+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.715550+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":42,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423980+0000","last_change":"2024-02-13T00:46:01.447742+0000","last_active":"2024-02-13T00:46:40.423980+0000","last_peered":"2024-02-13T00:46:40.423980+0000","last_clean":"2024-02-13T00:46:40.423980+0000","last_became_active":"2024-02-13T00:46:01.446960+0000","last_became_peered":"2024-02-13T00:46:01.446960+0000","last_unstale":"2024-02-13T00:46:40.423980+0000","last_undegraded":"2024-02-13T00:46:40.423980+0000","last_fullsized":"2024-02-13T00:46:40.423980+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.421001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.421001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":285,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529406+0000","last_change":"2024-02-13T00:46:01.450308+0000","last_active":"2024-02-13T00:46:41.529406+0000","last_peered":"2024-02-13T00:46:41.529406+0000","last_clean":"2024-02-13T00:46:41.529406+0000","last_became_active":"2024-02-13T00:46:01.449694+0000","last_became_peered":"2024-02-13T00:46:01.449694+0000","last_unstale":"2024-02-13T00:46:41.529406+0000","last_undegraded":"2024-02-13T00:46:41.529406+0000","last_fullsized":"2024-02-13T00:46:41.529406+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:27.073034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:27.073034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":467,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535881+0000","last_change":"2024-02-13T00:44:54.926465+0000","last_active":"2024-02-13T00:46:40.535881+0000","last_peered":"2024-02-13T00:46:40.535881+0000","last_clean":"2024-02-13T00:46:40.535881+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:46:40.535881+0000","last_undegraded":"2024-02-13T00:46:40.535881+0000","last_fullsized":"2024-02-13T00:46:40.535881+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:43:40.244930+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:43:40.244930+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529054+0000","last_change":"2024-02-13T00:46:01.463310+0000","last_active":"2024-02-13T00:46:41.529054+0000","last_peered":"2024-02-13T00:46:41.529054+0000","last_clean":"2024-02-13T00:46:41.529054+0000","last_became_active":"2024-02-13T00:46:01.462898+0000","last_became_peered":"2024-02-13T00:46:01.462898+0000","last_unstale":"2024-02-13T00:46:41.529054+0000","last_undegraded":"2024-02-13T00:46:41.529054+0000","last_fullsized":"2024-02-13T00:46:41.529054+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.095888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":192,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423006+0000","last_change":"2024-02-13T00:46:32.638418+0000","last_active":"2024-02-13T00:46:40.423006+0000","last_peered":"2024-02-13T00:46:40.423006+0000","last_clean":"2024-02-13T00:46:40.423006+0000","last_became_active":"2024-02-13T00:46:01.443279+0000","last_became_peered":"2024-02-13T00:46:01.443279+0000","last_unstale":"2024-02-13T00:46:40.423006+0000","last_undegraded":"2024-02-13T00:46:40.423006+0000","last_fullsized":"2024-02-13T00:46:40.423006+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.638349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":390,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421550+0000","last_change":"2024-02-13T00:46:01.446071+0000","last_active":"2024-02-13T00:46:40.421550+0000","last_peered":"2024-02-13T00:46:40.421550+0000","last_clean":"2024-02-13T00:46:40.421550+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:46:40.421550+0000","last_undegraded":"2024-02-13T00:46:40.421550+0000","last_fullsized":"2024-02-13T00:46:40.421550+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.413326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":125,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533468+0000","last_change":"2024-02-13T00:45:40.233264+0000","last_active":"2024-02-13T00:46:41.533468+0000","last_peered":"2024-02-13T00:46:41.533468+0000","last_clean":"2024-02-13T00:46:41.533468+0000","last_became_active":"2024-02-13T00:45:15.447051+0000","last_became_peered":"2024-02-13T00:45:15.447051+0000","last_unstale":"2024-02-13T00:46:41.533468+0000","last_undegraded":"2024-02-13T00:46:41.533468+0000","last_fullsized":"2024-02-13T00:46:41.533468+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:40.233185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:40.233185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":144,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533417+0000","last_change":"2024-02-13T00:44:52.751765+0000","last_active":"2024-02-13T00:46:41.533417+0000","last_peered":"2024-02-13T00:46:41.533417+0000","last_clean":"2024-02-13T00:46:41.533417+0000","last_became_active":"2024-02-13T00:44:50.435203+0000","last_became_peered":"2024-02-13T00:44:50.435203+0000","last_unstale":"2024-02-13T00:46:41.533417+0000","last_undegraded":"2024-02-13T00:46:41.533417+0000","last_fullsized":"2024-02-13T00:46:41.533417+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:42:51.361489+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:42:51.361489+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":401,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537858+0000","last_change":"2024-02-13T00:46:01.457220+0000","last_active":"2024-02-13T00:46:40.537858+0000","last_peered":"2024-02-13T00:46:40.537858+0000","last_clean":"2024-02-13T00:46:40.537858+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:46:40.537858+0000","last_undegraded":"2024-02-13T00:46:40.537858+0000","last_fullsized":"2024-02-13T00:46:40.537858+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.097481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":415,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535881+0000","last_change":"2024-02-13T00:46:01.453391+0000","last_active":"2024-02-13T00:46:40.535881+0000","last_peered":"2024-02-13T00:46:40.535881+0000","last_clean":"2024-02-13T00:46:40.535881+0000","last_became_active":"2024-02-13T00:46:01.452730+0000","last_became_peered":"2024-02-13T00:46:01.452730+0000","last_unstale":"2024-02-13T00:46:40.535881+0000","last_undegraded":"2024-02-13T00:46:40.535881+0000","last_fullsized":"2024-02-13T00:46:40.535881+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:26.218807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:44:26.218807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":196,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421837+0000","last_change":"2024-02-13T00:46:01.454624+0000","last_active":"2024-02-13T00:46:40.421837+0000","last_peered":"2024-02-13T00:46:40.421837+0000","last_clean":"2024-02-13T00:46:40.421837+0000","last_became_active":"2024-02-13T00:46:01.454287+0000","last_became_peered":"2024-02-13T00:46:01.454287+0000","last_unstale":"2024-02-13T00:46:40.421837+0000","last_undegraded":"2024-02-13T00:46:40.421837+0000","last_fullsized":"2024-02-13T00:46:40.421837+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:50.828264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:50.828264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.528721+0000","last_change":"2024-02-13T00:46:01.449548+0000","last_active":"2024-02-13T00:46:41.528721+0000","last_peered":"2024-02-13T00:46:41.528721+0000","last_clean":"2024-02-13T00:46:41.528721+0000","last_became_active":"2024-02-13T00:46:01.448671+0000","last_became_peered":"2024-02-13T00:46:01.448671+0000","last_unstale":"2024-02-13T00:46:41.528721+0000","last_undegraded":"2024-02-13T00:46:41.528721+0000","last_fullsized":"2024-02-13T00:46:41.528721+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.820327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.820327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":222,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.536016+0000","last_change":"2024-02-13T00:44:40.371566+0000","last_active":"2024-02-13T00:46:40.536016+0000","last_peered":"2024-02-13T00:46:40.536016+0000","last_clean":"2024-02-13T00:46:40.536016+0000","last_became_active":"2024-02-13T00:44:39.501040+0000","last_became_peered":"2024-02-13T00:44:39.501040+0000","last_unstale":"2024-02-13T00:46:40.536016+0000","last_undegraded":"2024-02-13T00:46:40.536016+0000","last_fullsized":"2024-02-13T00:46:40.536016+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:44:32.180113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:44:32.180113+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":355,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530766+0000","last_change":"2024-02-13T00:46:01.464539+0000","last_active":"2024-02-13T00:46:41.530766+0000","last_peered":"2024-02-13T00:46:41.530766+0000","last_clean":"2024-02-13T00:46:41.530766+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:46:41.530766+0000","last_undegraded":"2024-02-13T00:46:41.530766+0000","last_fullsized":"2024-02-13T00:46:41.530766+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.499576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":120,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533169+0000","last_change":"2024-02-13T00:45:15.436871+0000","last_active":"2024-02-13T00:46:41.533169+0000","last_peered":"2024-02-13T00:46:41.533169+0000","last_clean":"2024-02-13T00:46:41.533169+0000","last_became_active":"2024-02-13T00:45:15.435820+0000","last_became_peered":"2024-02-13T00:45:15.435820+0000","last_unstale":"2024-02-13T00:46:41.533169+0000","last_undegraded":"2024-02-13T00:46:41.533169+0000","last_fullsized":"2024-02-13T00:46:41.533169+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.646321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.646321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":110,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535408+0000","last_change":"2024-02-13T00:45:15.445131+0000","last_active":"2024-02-13T00:46:40.535408+0000","last_peered":"2024-02-13T00:46:40.535408+0000","last_clean":"2024-02-13T00:46:40.535408+0000","last_became_active":"2024-02-13T00:45:15.444818+0000","last_became_peered":"2024-02-13T00:45:15.444818+0000","last_unstale":"2024-02-13T00:46:40.535408+0000","last_undegraded":"2024-02-13T00:46:40.535408+0000","last_fullsized":"2024-02-13T00:46:40.535408+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.479450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":276,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.528780+0000","last_change":"2024-02-13T00:46:01.439915+0000","last_active":"2024-02-13T00:46:41.528780+0000","last_peered":"2024-02-13T00:46:41.528780+0000","last_clean":"2024-02-13T00:46:41.528780+0000","last_became_active":"2024-02-13T00:46:01.438745+0000","last_became_peered":"2024-02-13T00:46:01.438745+0000","last_unstale":"2024-02-13T00:46:41.528780+0000","last_undegraded":"2024-02-13T00:46:41.528780+0000","last_fullsized":"2024-02-13T00:46:41.528780+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.104906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.104906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":171,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.528652+0000","last_change":"2024-02-13T00:46:07.632448+0000","last_active":"2024-02-13T00:46:41.528652+0000","last_peered":"2024-02-13T00:46:41.528652+0000","last_clean":"2024-02-13T00:46:41.528652+0000","last_became_active":"2024-02-13T00:46:01.450079+0000","last_became_peered":"2024-02-13T00:46:01.450079+0000","last_unstale":"2024-02-13T00:46:41.528652+0000","last_undegraded":"2024-02-13T00:46:41.528652+0000","last_fullsized":"2024-02-13T00:46:41.528652+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:41:50.675154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:50.675154+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530010+0000","last_change":"2024-02-13T00:46:01.460250+0000","last_active":"2024-02-13T00:46:41.530010+0000","last_peered":"2024-02-13T00:46:41.530010+0000","last_clean":"2024-02-13T00:46:41.530010+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:46:41.530010+0000","last_undegraded":"2024-02-13T00:46:41.530010+0000","last_fullsized":"2024-02-13T00:46:41.530010+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.457672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.457672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":120,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.531868+0000","last_change":"2024-02-13T00:45:15.439752+0000","last_active":"2024-02-13T00:46:41.531868+0000","last_peered":"2024-02-13T00:46:41.531868+0000","last_clean":"2024-02-13T00:46:41.531868+0000","last_became_active":"2024-02-13T00:45:15.437943+0000","last_became_peered":"2024-02-13T00:45:15.437943+0000","last_unstale":"2024-02-13T00:46:41.531868+0000","last_undegraded":"2024-02-13T00:46:41.531868+0000","last_fullsized":"2024-02-13T00:46:41.531868+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.284740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.284740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":328,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.422987+0000","last_change":"2024-02-13T00:44:58.969806+0000","last_active":"2024-02-13T00:46:40.422987+0000","last_peered":"2024-02-13T00:46:40.422987+0000","last_clean":"2024-02-13T00:46:40.422987+0000","last_became_active":"2024-02-13T00:44:50.439512+0000","last_became_peered":"2024-02-13T00:44:50.439512+0000","last_unstale":"2024-02-13T00:46:40.422987+0000","last_undegraded":"2024-02-13T00:46:40.422987+0000","last_fullsized":"2024-02-13T00:46:40.422987+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:58.969692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:44:58.969692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":243,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534524+0000","last_change":"2024-02-13T00:45:15.431130+0000","last_active":"2024-02-13T00:46:41.534524+0000","last_peered":"2024-02-13T00:46:41.534524+0000","last_clean":"2024-02-13T00:46:41.534524+0000","last_became_active":"2024-02-13T00:45:15.430749+0000","last_became_peered":"2024-02-13T00:45:15.430749+0000","last_unstale":"2024-02-13T00:46:41.534524+0000","last_undegraded":"2024-02-13T00:46:41.534524+0000","last_fullsized":"2024-02-13T00:46:41.534524+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.605111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.605111+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":497,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424592+0000","last_change":"2024-02-13T00:46:02.034931+0000","last_active":"2024-02-13T00:46:40.424592+0000","last_peered":"2024-02-13T00:46:40.424592+0000","last_clean":"2024-02-13T00:46:40.424592+0000","last_became_active":"2024-02-13T00:46:01.443721+0000","last_became_peered":"2024-02-13T00:46:01.443721+0000","last_unstale":"2024-02-13T00:46:40.424592+0000","last_undegraded":"2024-02-13T00:46:40.424592+0000","last_fullsized":"2024-02-13T00:46:40.424592+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:43:07.393197+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.393197+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":362,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529616+0000","last_change":"2024-02-13T00:46:32.298033+0000","last_active":"2024-02-13T00:46:41.529616+0000","last_peered":"2024-02-13T00:46:41.529616+0000","last_clean":"2024-02-13T00:46:41.529616+0000","last_became_active":"2024-02-13T00:46:01.442231+0000","last_became_peered":"2024-02-13T00:46:01.442231+0000","last_unstale":"2024-02-13T00:46:41.529616+0000","last_undegraded":"2024-02-13T00:46:41.529616+0000","last_fullsized":"2024-02-13T00:46:41.529616+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.297918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.297918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"363'14","reported_seq":156,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-02-13T00:46:42.437744+0000","last_change":"2024-02-13T00:45:18.328879+0000","last_active":"2024-02-13T00:46:42.437744+0000","last_peered":"2024-02-13T00:46:42.437744+0000","last_clean":"2024-02-13T00:46:42.437744+0000","last_became_active":"2024-02-13T00:45:15.455825+0000","last_became_peered":"2024-02-13T00:45:15.455825+0000","last_unstale":"2024-02-13T00:46:42.437744+0000","last_undegraded":"2024-02-13T00:46:42.437744+0000","last_fullsized":"2024-02-13T00:46:42.437744+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'10","last_scrub_stamp":"2024-02-13T00:43:05.604461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.604461+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":221,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.531589+0000","last_change":"2024-02-13T00:46:03.966962+0000","last_active":"2024-02-13T00:46:41.531589+0000","last_peered":"2024-02-13T00:46:41.531589+0000","last_clean":"2024-02-13T00:46:41.531589+0000","last_became_active":"2024-02-13T00:46:01.439984+0000","last_became_peered":"2024-02-13T00:46:01.439984+0000","last_unstale":"2024-02-13T00:46:41.531589+0000","last_undegraded":"2024-02-13T00:46:41.531589+0000","last_fullsized":"2024-02-13T00:46:41.531589+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:42:55.481128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:55.481128+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":359,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538878+0000","last_change":"2024-02-13T00:45:38.084426+0000","last_active":"2024-02-13T00:46:40.538878+0000","last_peered":"2024-02-13T00:46:40.538878+0000","last_clean":"2024-02-13T00:46:40.538878+0000","last_became_active":"2024-02-13T00:45:15.435550+0000","last_became_peered":"2024-02-13T00:45:15.435550+0000","last_unstale":"2024-02-13T00:46:40.538878+0000","last_undegraded":"2024-02-13T00:46:40.538878+0000","last_fullsized":"2024-02-13T00:46:40.538878+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.084358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.084358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":148,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529870+0000","last_change":"2024-02-13T00:46:10.679144+0000","last_active":"2024-02-13T00:46:41.529870+0000","last_peered":"2024-02-13T00:46:41.529870+0000","last_clean":"2024-02-13T00:46:41.529870+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:46:41.529870+0000","last_undegraded":"2024-02-13T00:46:41.529870+0000","last_fullsized":"2024-02-13T00:46:41.529870+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:42:59.604002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:59.604002+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533442+0000","last_change":"2024-02-13T00:45:15.437121+0000","last_active":"2024-02-13T00:46:41.533442+0000","last_peered":"2024-02-13T00:46:41.533442+0000","last_clean":"2024-02-13T00:46:41.533442+0000","last_became_active":"2024-02-13T00:45:15.436239+0000","last_became_peered":"2024-02-13T00:45:15.436239+0000","last_unstale":"2024-02-13T00:46:41.533442+0000","last_undegraded":"2024-02-13T00:46:41.533442+0000","last_fullsized":"2024-02-13T00:46:41.533442+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:46.698599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:46.698599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":403,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424603+0000","last_change":"2024-02-13T00:45:15.423191+0000","last_active":"2024-02-13T00:46:40.424603+0000","last_peered":"2024-02-13T00:46:40.424603+0000","last_clean":"2024-02-13T00:46:40.424603+0000","last_became_active":"2024-02-13T00:45:15.422870+0000","last_became_peered":"2024-02-13T00:45:15.422870+0000","last_unstale":"2024-02-13T00:46:40.424603+0000","last_undegraded":"2024-02-13T00:46:40.424603+0000","last_fullsized":"2024-02-13T00:46:40.424603+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.596334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.596334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":395,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537773+0000","last_change":"2024-02-13T00:46:01.457036+0000","last_active":"2024-02-13T00:46:40.537773+0000","last_peered":"2024-02-13T00:46:40.537773+0000","last_clean":"2024-02-13T00:46:40.537773+0000","last_became_active":"2024-02-13T00:46:01.453269+0000","last_became_peered":"2024-02-13T00:46:01.453269+0000","last_unstale":"2024-02-13T00:46:40.537773+0000","last_undegraded":"2024-02-13T00:46:40.537773+0000","last_fullsized":"2024-02-13T00:46:40.537773+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.245237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.245237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":125,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534342+0000","last_change":"2024-02-13T00:45:37.280947+0000","last_active":"2024-02-13T00:46:41.534342+0000","last_peered":"2024-02-13T00:46:41.534342+0000","last_clean":"2024-02-13T00:46:41.534342+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:46:41.534342+0000","last_undegraded":"2024-02-13T00:46:41.534342+0000","last_fullsized":"2024-02-13T00:46:41.534342+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.280865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.280865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":156,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.537355+0000","last_change":"2024-02-13T00:44:55.939050+0000","last_active":"2024-02-13T00:46:40.537355+0000","last_peered":"2024-02-13T00:46:40.537355+0000","last_clean":"2024-02-13T00:46:40.537355+0000","last_became_active":"2024-02-13T00:44:50.442894+0000","last_became_peered":"2024-02-13T00:44:50.442894+0000","last_unstale":"2024-02-13T00:46:40.537355+0000","last_undegraded":"2024-02-13T00:46:40.537355+0000","last_fullsized":"2024-02-13T00:46:40.537355+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:43:30.578602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.578602+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":120,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533022+0000","last_change":"2024-02-13T00:45:15.446020+0000","last_active":"2024-02-13T00:46:41.533022+0000","last_peered":"2024-02-13T00:46:41.533022+0000","last_clean":"2024-02-13T00:46:41.533022+0000","last_became_active":"2024-02-13T00:45:15.445403+0000","last_became_peered":"2024-02-13T00:45:15.445403+0000","last_unstale":"2024-02-13T00:46:41.533022+0000","last_undegraded":"2024-02-13T00:46:41.533022+0000","last_fullsized":"2024-02-13T00:46:41.533022+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:06.651077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:06.651077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":383,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538757+0000","last_change":"2024-02-13T00:41:56.204734+0000","last_active":"2024-02-13T00:46:40.538757+0000","last_peered":"2024-02-13T00:46:40.538757+0000","last_clean":"2024-02-13T00:46:40.538757+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:46:40.538757+0000","last_undegraded":"2024-02-13T00:46:40.538757+0000","last_fullsized":"2024-02-13T00:46:40.538757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:56.204671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.204671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":120,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529973+0000","last_change":"2024-02-13T00:46:01.451373+0000","last_active":"2024-02-13T00:46:41.529973+0000","last_peered":"2024-02-13T00:46:41.529973+0000","last_clean":"2024-02-13T00:46:41.529973+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:46:41.529973+0000","last_undegraded":"2024-02-13T00:46:41.529973+0000","last_fullsized":"2024-02-13T00:46:41.529973+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.410777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.410777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":34,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530844+0000","last_change":"2024-02-13T00:46:01.451610+0000","last_active":"2024-02-13T00:46:41.530844+0000","last_peered":"2024-02-13T00:46:41.530844+0000","last_clean":"2024-02-13T00:46:41.530844+0000","last_became_active":"2024-02-13T00:46:01.449873+0000","last_became_peered":"2024-02-13T00:46:01.449873+0000","last_unstale":"2024-02-13T00:46:41.530844+0000","last_undegraded":"2024-02-13T00:46:41.530844+0000","last_fullsized":"2024-02-13T00:46:41.530844+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.554065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.554065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":266,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.538678+0000","last_change":"2024-02-13T00:44:44.964075+0000","last_active":"2024-02-13T00:46:40.538678+0000","last_peered":"2024-02-13T00:46:40.538678+0000","last_clean":"2024-02-13T00:46:40.538678+0000","last_became_active":"2024-02-13T00:44:39.504670+0000","last_became_peered":"2024-02-13T00:44:39.504670+0000","last_unstale":"2024-02-13T00:46:40.538678+0000","last_undegraded":"2024-02-13T00:46:40.538678+0000","last_fullsized":"2024-02-13T00:46:40.538678+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:41:41.686058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.686058+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.528813+0000","last_change":"2024-02-13T00:46:01.448134+0000","last_active":"2024-02-13T00:46:41.528813+0000","last_peered":"2024-02-13T00:46:41.528813+0000","last_clean":"2024-02-13T00:46:41.528813+0000","last_became_active":"2024-02-13T00:46:01.446635+0000","last_became_peered":"2024-02-13T00:46:01.446635+0000","last_unstale":"2024-02-13T00:46:41.528813+0000","last_undegraded":"2024-02-13T00:46:41.528813+0000","last_fullsized":"2024-02-13T00:46:41.528813+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:28.101254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:28.101254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":110,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.536624+0000","last_change":"2024-02-13T00:45:15.435977+0000","last_active":"2024-02-13T00:46:40.536624+0000","last_peered":"2024-02-13T00:46:40.536624+0000","last_clean":"2024-02-13T00:46:40.536624+0000","last_became_active":"2024-02-13T00:45:15.435206+0000","last_became_peered":"2024-02-13T00:45:15.435206+0000","last_unstale":"2024-02-13T00:46:40.536624+0000","last_undegraded":"2024-02-13T00:46:40.536624+0000","last_fullsized":"2024-02-13T00:46:40.536624+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:57.802417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.802417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":125,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530415+0000","last_change":"2024-02-13T00:46:01.461188+0000","last_active":"2024-02-13T00:46:41.530415+0000","last_peered":"2024-02-13T00:46:41.530415+0000","last_clean":"2024-02-13T00:46:41.530415+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:46:41.530415+0000","last_undegraded":"2024-02-13T00:46:41.530415+0000","last_fullsized":"2024-02-13T00:46:41.530415+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.238152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.238152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":369,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423046+0000","last_change":"2024-02-13T00:46:01.443734+0000","last_active":"2024-02-13T00:46:40.423046+0000","last_peered":"2024-02-13T00:46:40.423046+0000","last_clean":"2024-02-13T00:46:40.423046+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:46:40.423046+0000","last_undegraded":"2024-02-13T00:46:40.423046+0000","last_fullsized":"2024-02-13T00:46:40.423046+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.621170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.621170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":265,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423246+0000","last_change":"2024-02-13T00:46:27.706096+0000","last_active":"2024-02-13T00:46:40.423246+0000","last_peered":"2024-02-13T00:46:40.423246+0000","last_clean":"2024-02-13T00:46:40.423246+0000","last_became_active":"2024-02-13T00:44:50.415894+0000","last_became_peered":"2024-02-13T00:44:50.415894+0000","last_unstale":"2024-02-13T00:46:40.423246+0000","last_undegraded":"2024-02-13T00:46:40.423246+0000","last_fullsized":"2024-02-13T00:46:40.423246+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:46:27.706006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:27.706006+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":204,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.530320+0000","last_change":"2024-02-13T00:46:01.452098+0000","last_active":"2024-02-13T00:46:41.530320+0000","last_peered":"2024-02-13T00:46:41.530320+0000","last_clean":"2024-02-13T00:46:41.530320+0000","last_became_active":"2024-02-13T00:46:01.451790+0000","last_became_peered":"2024-02-13T00:46:01.451790+0000","last_unstale":"2024-02-13T00:46:41.530320+0000","last_undegraded":"2024-02-13T00:46:41.530320+0000","last_fullsized":"2024-02-13T00:46:41.530320+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:03.665973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:03.665973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":358,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.423830+0000","last_change":"2024-02-13T00:46:01.454932+0000","last_active":"2024-02-13T00:46:40.423830+0000","last_peered":"2024-02-13T00:46:40.423830+0000","last_clean":"2024-02-13T00:46:40.423830+0000","last_became_active":"2024-02-13T00:46:01.454163+0000","last_became_peered":"2024-02-13T00:46:01.454163+0000","last_unstale":"2024-02-13T00:46:40.423830+0000","last_undegraded":"2024-02-13T00:46:40.423830+0000","last_fullsized":"2024-02-13T00:46:40.423830+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.415414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.415414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":390,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.422439+0000","last_change":"2024-02-13T00:45:15.420228+0000","last_active":"2024-02-13T00:46:40.422439+0000","last_peered":"2024-02-13T00:46:40.422439+0000","last_clean":"2024-02-13T00:46:40.422439+0000","last_became_active":"2024-02-13T00:45:15.419668+0000","last_became_peered":"2024-02-13T00:45:15.419668+0000","last_unstale":"2024-02-13T00:46:40.422439+0000","last_undegraded":"2024-02-13T00:46:40.422439+0000","last_fullsized":"2024-02-13T00:46:40.422439+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:21.648002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.648002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":303,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424059+0000","last_change":"2024-02-13T00:46:01.457482+0000","last_active":"2024-02-13T00:46:40.424059+0000","last_peered":"2024-02-13T00:46:40.424059+0000","last_clean":"2024-02-13T00:46:40.424059+0000","last_became_active":"2024-02-13T00:46:01.455940+0000","last_became_peered":"2024-02-13T00:46:01.455940+0000","last_unstale":"2024-02-13T00:46:40.424059+0000","last_undegraded":"2024-02-13T00:46:40.424059+0000","last_fullsized":"2024-02-13T00:46:40.424059+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:21.087199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:21.087199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"85'14","reported_seq":285,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.536571+0000","last_change":"2024-02-13T00:46:36.089483+0000","last_active":"2024-02-13T00:46:40.536571+0000","last_peered":"2024-02-13T00:46:40.536571+0000","last_clean":"2024-02-13T00:46:40.536571+0000","last_became_active":"2024-02-13T00:45:15.419364+0000","last_became_peered":"2024-02-13T00:45:15.419364+0000","last_unstale":"2024-02-13T00:46:40.536571+0000","last_undegraded":"2024-02-13T00:46:40.536571+0000","last_fullsized":"2024-02-13T00:46:40.536571+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:46:36.089402+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:46:36.089402+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":129,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.529780+0000","last_change":"2024-02-13T00:46:01.452313+0000","last_active":"2024-02-13T00:46:41.529780+0000","last_peered":"2024-02-13T00:46:41.529780+0000","last_clean":"2024-02-13T00:46:41.529780+0000","last_became_active":"2024-02-13T00:46:01.451997+0000","last_became_peered":"2024-02-13T00:46:01.451997+0000","last_unstale":"2024-02-13T00:46:41.529780+0000","last_undegraded":"2024-02-13T00:46:41.529780+0000","last_fullsized":"2024-02-13T00:46:41.529780+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:21.630532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.630532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":228,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535678+0000","last_change":"2024-02-13T00:45:15.416632+0000","last_active":"2024-02-13T00:46:40.535678+0000","last_peered":"2024-02-13T00:46:40.535678+0000","last_clean":"2024-02-13T00:46:40.535678+0000","last_became_active":"2024-02-13T00:45:15.416307+0000","last_became_peered":"2024-02-13T00:45:15.416307+0000","last_unstale":"2024-02-13T00:46:40.535678+0000","last_undegraded":"2024-02-13T00:46:40.535678+0000","last_fullsized":"2024-02-13T00:46:40.535678+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:48.629024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.629024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":340,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535824+0000","last_change":"2024-02-13T00:46:01.445422+0000","last_active":"2024-02-13T00:46:40.535824+0000","last_peered":"2024-02-13T00:46:40.535824+0000","last_clean":"2024-02-13T00:46:40.535824+0000","last_became_active":"2024-02-13T00:46:01.445141+0000","last_became_peered":"2024-02-13T00:46:01.445141+0000","last_unstale":"2024-02-13T00:46:40.535824+0000","last_undegraded":"2024-02-13T00:46:40.535824+0000","last_fullsized":"2024-02-13T00:46:40.535824+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:52.447698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.447698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":116,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533202+0000","last_change":"2024-02-13T00:45:30.186208+0000","last_active":"2024-02-13T00:46:41.533202+0000","last_peered":"2024-02-13T00:46:41.533202+0000","last_clean":"2024-02-13T00:46:41.533202+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:46:41.533202+0000","last_undegraded":"2024-02-13T00:46:41.533202+0000","last_fullsized":"2024-02-13T00:46:41.533202+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:30.186129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:30.186129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"90'20","reported_seq":165,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.528610+0000","last_change":"2024-02-13T00:46:06.251713+0000","last_active":"2024-02-13T00:46:41.528610+0000","last_peered":"2024-02-13T00:46:41.528610+0000","last_clean":"2024-02-13T00:46:41.528610+0000","last_became_active":"2024-02-13T00:46:01.442464+0000","last_became_peered":"2024-02-13T00:46:01.442464+0000","last_unstale":"2024-02-13T00:46:41.528610+0000","last_undegraded":"2024-02-13T00:46:41.528610+0000","last_fullsized":"2024-02-13T00:46:41.528610+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:43:44.241358+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:43:44.241358+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":205,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424502+0000","last_change":"2024-02-13T00:46:01.450078+0000","last_active":"2024-02-13T00:46:40.424502+0000","last_peered":"2024-02-13T00:46:40.424502+0000","last_clean":"2024-02-13T00:46:40.424502+0000","last_became_active":"2024-02-13T00:46:01.447654+0000","last_became_peered":"2024-02-13T00:46:01.447654+0000","last_unstale":"2024-02-13T00:46:40.424502+0000","last_undegraded":"2024-02-13T00:46:40.424502+0000","last_fullsized":"2024-02-13T00:46:40.424502+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:18.464402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:42:18.464402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":401,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535509+0000","last_change":"2024-02-13T00:46:01.445302+0000","last_active":"2024-02-13T00:46:40.535509+0000","last_peered":"2024-02-13T00:46:40.535509+0000","last_clean":"2024-02-13T00:46:40.535509+0000","last_became_active":"2024-02-13T00:46:01.444842+0000","last_became_peered":"2024-02-13T00:46:01.444842+0000","last_unstale":"2024-02-13T00:46:40.535509+0000","last_undegraded":"2024-02-13T00:46:40.535509+0000","last_fullsized":"2024-02-13T00:46:40.535509+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.693214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.693214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":39,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.535589+0000","last_change":"2024-02-13T00:46:01.449363+0000","last_active":"2024-02-13T00:46:40.535589+0000","last_peered":"2024-02-13T00:46:40.535589+0000","last_clean":"2024-02-13T00:46:40.535589+0000","last_became_active":"2024-02-13T00:46:01.448971+0000","last_became_peered":"2024-02-13T00:46:01.448971+0000","last_unstale":"2024-02-13T00:46:40.535589+0000","last_undegraded":"2024-02-13T00:46:40.535589+0000","last_fullsized":"2024-02-13T00:46:40.535589+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:47.487860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:47.487860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":371,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.536036+0000","last_change":"2024-02-13T00:45:32.233265+0000","last_active":"2024-02-13T00:46:40.536036+0000","last_peered":"2024-02-13T00:46:40.536036+0000","last_clean":"2024-02-13T00:46:40.536036+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:46:40.536036+0000","last_undegraded":"2024-02-13T00:46:40.536036+0000","last_fullsized":"2024-02-13T00:46:40.536036+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:45:32.233197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.233197+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"86'50","reported_seq":286,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.421925+0000","last_change":"2024-02-13T00:46:05.700233+0000","last_active":"2024-02-13T00:46:40.421925+0000","last_peered":"2024-02-13T00:46:40.421925+0000","last_clean":"2024-02-13T00:46:40.421925+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:46:40.421925+0000","last_undegraded":"2024-02-13T00:46:40.421925+0000","last_fullsized":"2024-02-13T00:46:40.421925+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:42:17.454232+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.454232+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":396,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:40.424269+0000","last_change":"2024-02-13T00:45:15.434575+0000","last_active":"2024-02-13T00:46:40.424269+0000","last_peered":"2024-02-13T00:46:40.424269+0000","last_clean":"2024-02-13T00:46:40.424269+0000","last_became_active":"2024-02-13T00:45:15.434224+0000","last_became_peered":"2024-02-13T00:45:15.434224+0000","last_unstale":"2024-02-13T00:46:40.424269+0000","last_undegraded":"2024-02-13T00:46:40.424269+0000","last_fullsized":"2024-02-13T00:46:40.424269+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:43:03.464398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":111,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.533070+0000","last_change":"2024-02-13T00:44:39.510101+0000","last_active":"2024-02-13T00:46:41.533070+0000","last_peered":"2024-02-13T00:46:41.533070+0000","last_clean":"2024-02-13T00:46:41.533070+0000","last_became_active":"2024-02-13T00:44:39.508197+0000","last_became_peered":"2024-02-13T00:44:39.508197+0000","last_unstale":"2024-02-13T00:46:41.533070+0000","last_undegraded":"2024-02-13T00:46:41.533070+0000","last_fullsized":"2024-02-13T00:46:41.533070+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:46.418859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:41:46.418859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":120,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-02-13T00:46:41.534232+0000","last_change":"2024-02-13T00:45:15.435955+0000","last_active":"2024-02-13T00:46:41.534232+0000","last_peered":"2024-02-13T00:46:41.534232+0000","last_clean":"2024-02-13T00:46:41.534232+0000","last_became_active":"2024-02-13T00:45:15.435259+0000","last_became_peered":"2024-02-13T00:45:15.435259+0000","last_unstale":"2024-02-13T00:46:41.534232+0000","last_undegraded":"2024-02-13T00:46:41.534232+0000","last_fullsized":"2024-02-13T00:46:41.534232+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.953989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.953989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":4001479,"num_objects":158,"num_object_clones":2,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":101,"num_read_kb":6,"num_write":596,"num_write_kb":4495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1266,"num_bytes_recovered":4004226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14065664,"data_stored":12004851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1884,"ondisk_log_size":1884,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1063,"num_read_kb":788,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":9606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":11656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":531,"num_read_kb":526,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":227633266799,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":45,"seq":193273528437,"num_pgs":184,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316600,"kb_used_data":7080,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428552,"statfs":{"total":95995035648,"available":95670837248,"internally_reserved":0,"allocated":7249920,"data_stored":3952787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.62,"15min":0.56000000000000005},"min":{"1min":0.25600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.61199999999999999,"5min":8.1340000000000003,"15min":8.1340000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.85599999999999998,"15min":0.60499999999999998},"min":{"1min":0.309,"5min":0.221,"15min":0.221},"max":{"1min":1.0309999999999999,"5min":11.367000000000001,"15min":11.367000000000001},"last":0.45300000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.03,"15min":0.88700000000000001},"min":{"1min":0.317,"5min":0.29699999999999999,"15min":0.27200000000000002},"max":{"1min":0.64500000000000002,"5min":24.209,"15min":54.002000000000002},"last":0.435},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.88300000000000001,"15min":0.68300000000000005},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63900000000000001,"5min":12.403,"15min":12.403},"last":0.52700000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.98399999999999999,"15min":0.70699999999999996},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.002,"5min":12.477,"15min":12.477},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.99099999999999999,"15min":0.67600000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.95199999999999996,"5min":11.702,"15min":11.702},"last":0.59099999999999997}]},{"osd":3,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.87,"15min":0.67100000000000004},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.64600000000000002,"5min":8.0519999999999996,"15min":8.0519999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58099999999999996,"5min":1,"15min":0.77600000000000002},"min":{"1min":0.33700000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.2330000000000001,"5min":13.257,"15min":19.081},"last":0.67100000000000004}]},{"osd":4,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41399999999999998,"15min":0.41099999999999998},"min":{"1min":0.26100000000000001,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":1.0649999999999999},"last":0.373},{"interface":"front","average":{"1min":0.438,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.27600000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.83299999999999996},"last":0.311}]},{"osd":5,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.434,"15min":0.41499999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.61199999999999999,"5min":0.66500000000000004,"15min":0.94399999999999995},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.48199999999999998,"15min":0.46999999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.86299999999999999},"last":0.33900000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46500000000000002,"15min":0.49399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.59899999999999998,"5min":0.91600000000000004,"15min":1.173},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.502,"15min":0.54400000000000004},"min":{"1min":0.35899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67900000000000005,"5min":0.77400000000000002,"15min":1.262},"last":0.35599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574990,"num_pgs":191,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316488,"kb_used_data":11068,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93428664,"statfs":{"total":95995035648,"available":95670951936,"internally_reserved":0,"allocated":11333632,"data_stored":7952718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":1.7070000000000001,"15min":0.94699999999999995},"min":{"1min":0.223,"5min":0.189,"15min":0.189},"max":{"1min":0.60899999999999999,"5min":61.856000000000002,"15min":61.856000000000002},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":1.244,"15min":0.66200000000000003},"min":{"1min":0.26500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.44800000000000001,"5min":61.908999999999999,"15min":61.908999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":1.825,"15min":0.85699999999999998},"min":{"1min":0.22900000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.80000000000000004,"5min":32.161000000000001,"15min":32.161000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":1.716,"15min":0.83199999999999996},"min":{"1min":0.246,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.21,"5min":29.693000000000001,"15min":29.693000000000001},"last":0.38300000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":2.5379999999999998,"15min":1.2849999999999999},"min":{"1min":0.26800000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.46600000000000003,"5min":131.23099999999999,"15min":131.23099999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.40600000000000003,"5min":1.5669999999999999,"15min":0.88100000000000001},"min":{"1min":0.307,"5min":0.26800000000000002,"15min":0.246},"max":{"1min":0.61799999999999999,"5min":49.055,"15min":49.055},"last":0.41899999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":1.1739999999999999,"15min":0.76900000000000002},"min":{"1min":0.38900000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.246,"5min":16.547000000000001,"15min":16.547000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.98399999999999999,"15min":0.70799999999999996},"min":{"1min":0.34100000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":14.731999999999999,"15min":14.731999999999999},"last":0.53600000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.5680000000000001,"15min":0.90200000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.90500000000000003,"5min":26.561,"15min":26.561},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.137,"15min":0.76900000000000002},"min":{"1min":0.377,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.163,"5min":16.494,"15min":16.494},"last":0.56000000000000005}]},{"osd":6,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.84299999999999997,"15min":0.72699999999999998},"min":{"1min":0.41999999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.92700000000000005,"5min":14.772,"15min":14.772},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":1.208,"15min":0.93899999999999995},"min":{"1min":0.41399999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.78500000000000003,"5min":26.600000000000001,"15min":26.600000000000001},"last":0.52900000000000003}]},{"osd":7,"last update":"Tue Feb 13 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":1.006,"15min":0.73099999999999998},"min":{"1min":0.432,"5min":0.313,"15min":0.313},"max":{"1min":0.95899999999999996,"5min":14.794,"15min":14.794},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.591,"15min":0.94299999999999995},"min":{"1min":0.40600000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80100000000000005,"5min":26.693999999999999,"15min":26.693999999999999},"last":0.503}]}]},{"osd":0,"up_from":8,"seq":34359738514,"num_pgs":172,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320480,"kb_used_data":10964,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93424672,"statfs":{"total":95995035648,"available":95666864128,"internally_reserved":0,"allocated":11227136,"data_stored":7952461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:46:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":1.157,"15min":0.62},"min":{"1min":0.20999999999999999,"5min":0.192,"15min":0.189},"max":{"1min":1.7370000000000001,"5min":30.184000000000001,"15min":30.184000000000001},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.55000000000000004,"15min":0.44700000000000001},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.17599999999999999},"max":{"1min":2.839,"5min":5.6959999999999997,"15min":5.6959999999999997},"last":0.253}]},{"osd":2,"last update":"Tue Feb 13 00:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.67300000000000004,"15min":0.497},"min":{"1min":0.25,"5min":0.216,"15min":0.20999999999999999},"max":{"1min":0.94799999999999995,"5min":7.0839999999999996,"15min":7.0839999999999996},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.625,"5min":1.2310000000000001,"15min":0.66900000000000004},"min":{"1min":0.26800000000000002,"5min":0.20899999999999999,"15min":0.185},"max":{"1min":3.6949999999999998,"5min":24.459,"15min":24.459},"last":0.28799999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67700000000000005,"15min":0.49399999999999999},"min":{"1min":0.26500000000000001,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":4.8710000000000004,"5min":8.718,"15min":8.718},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.72499999999999998,"15min":0.55300000000000005},"min":{"1min":0.316,"5min":0.222,"15min":0.222},"max":{"1min":2.3140000000000001,"5min":7.1139999999999999,"15min":7.1139999999999999},"last":0.34000000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:45:42 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.3200000000000001,"15min":0.78900000000000003},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.24399999999999999},"max":{"1min":7.056,"5min":24.381,"15min":24.381},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.90100000000000002,"15min":0.64800000000000002},"min":{"1min":0.38700000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.1859999999999999,"5min":8.6950000000000003,"15min":8.6950000000000003},"last":0.34899999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:46:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.749,"15min":0.63},"min":{"1min":0.36399999999999999,"5min":0.309,"15min":0.309},"max":{"1min":3.331,"5min":7.0990000000000002,"15min":7.0990000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.4330000000000001,"15min":0.85099999999999998},"min":{"1min":0.33900000000000002,"5min":0.307,"15min":0.30599999999999999},"max":{"1min":10.686,"5min":24.427,"15min":24.427},"last":0.432}]},{"osd":6,"last update":"Tue Feb 13 00:46:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.81100000000000005,"15min":0.622},"min":{"1min":0.376,"5min":0.32900000000000001,"15min":0.31},"max":{"1min":1.4990000000000001,"5min":8.6129999999999995,"15min":8.6129999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.79200000000000004,"15min":0.64200000000000002},"min":{"1min":0.35099999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.83399999999999996,"5min":14.263,"15min":14.263},"last":0.48799999999999999}]},{"osd":7,"last update":"Tue Feb 13 00:45:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.89000000000000001,"15min":0.66500000000000004},"min":{"1min":0.441,"5min":0.317,"15min":0.317},"max":{"1min":7.7720000000000002,"5min":8.6679999999999993,"15min":8.6679999999999993},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.84399999999999997,"15min":0.67000000000000004},"min":{"1min":0.41199999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.006,"5min":8.1609999999999996,"15min":8.1609999999999996},"last":0.47199999999999998}]}]},{"osd":2,"up_from":0,"seq":77309411465,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182531,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986303,"num_pgs":167,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316476,"kb_used_data":11056,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93428676,"statfs":{"total":95995035648,"available":95670964224,"internally_reserved":0,"allocated":11321344,"data_stored":7952756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.79200000000000004,"15min":0.60699999999999998},"min":{"1min":0.34499999999999997,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":5.3319999999999999,"5min":9.3399999999999999,"15min":9.3399999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.71199999999999997,"15min":0.59499999999999997},"min":{"1min":0.28199999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.5700000000000003,"5min":13.833,"15min":13.833},"last":0.39000000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:45:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.2609999999999999,"15min":0.77200000000000002},"min":{"1min":0.26700000000000002,"5min":0.25,"15min":0.25},"max":{"1min":7.5410000000000004,"5min":53.188000000000002,"15min":53.188000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.97799999999999998,"15min":0.67700000000000005},"min":{"1min":0.315,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.6559999999999999,"5min":44.793999999999997,"15min":44.793999999999997},"last":0.442}]},{"osd":2,"last update":"Tue Feb 13 00:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.70999999999999996,"15min":0.58999999999999997},"min":{"1min":0.34399999999999997,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.827,"5min":8.6829999999999998,"15min":8.6829999999999998},"last":0.496},{"interface":"front","average":{"1min":0.877,"5min":0.84399999999999997,"15min":0.63100000000000001},"min":{"1min":0.33300000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.3340000000000001,"5min":18.911000000000001,"15min":18.911000000000001},"last":0.42299999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.88800000000000001,"15min":0.65600000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.528,"5min":23.5,"15min":23.5},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":1.081,"15min":0.71499999999999997},"min":{"1min":0.32100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5720000000000001,"5min":36.152999999999999,"15min":36.152999999999999},"last":0.40400000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.42299999999999999,"15min":0.38500000000000001},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.20599999999999999},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.371},{"interface":"front","average":{"1min":0.505,"5min":0.443,"15min":0.40500000000000003},"min":{"1min":0.27100000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.41099999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:46:19 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46700000000000003,"15min":0.432},"min":{"1min":0.28000000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67200000000000004,"5min":3.153,"15min":3.153},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.50800000000000001,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.254,"15min":0.254},"max":{"1min":0.70099999999999996,"5min":4.9390000000000001,"15min":4.9390000000000001},"last":0.46800000000000003}]},{"osd":7,"last update":"Tue Feb 13 00:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.49299999999999999,"15min":0.47699999999999998},"min":{"1min":0.34599999999999997,"5min":0.312,"15min":0.312},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.49299999999999999,"15min":0.47099999999999997},"min":{"1min":0.29699999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":0.34300000000000003}]}]},{"osd":5,"up_from":0,"seq":167503724666,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":3328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4427776,"data_stored":4001019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4509696,"data_stored":4001192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4521984,"data_stored":4001206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":1020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:46:50.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:46:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:50.287 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:50.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:50.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:50.416 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:50.515 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:49.457023+0000 mon.a (mon.0) 2553 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:46:50.515 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:49.457103+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e431: 8 total, 8 up, 4 in 2024-02-13T00:46:50.607 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:49.457023+0000 mon.a (mon.0) 2553 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:46:50.607 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:49.457103+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e431: 8 total, 8 up, 4 in 2024-02-13T00:46:50.772 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:50.773 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:49.457023+0000 mon.a (mon.0) 2553 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:46:50.774 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:49.457103+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e431: 8 total, 8 up, 4 in 2024-02-13T00:46:50.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:50.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:50.849 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:50.876 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:50.904 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:51.102 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:51.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:51.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:51.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:51.236 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:51.435 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:51.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:51.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:51.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:51.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:46:51.568 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:51.589 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:51.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:51.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:51.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:50.429738+0000 mgr.y (mgr.14544) 381 : cluster [DBG] pgmap v629: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:46:51.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:50.522350+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e432: 8 total, 8 up, 4 in 2024-02-13T00:46:51.724 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:51.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:50.429738+0000 mgr.y (mgr.14544) 381 : cluster [DBG] pgmap v629: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:46:51.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:50.522350+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e432: 8 total, 8 up, 4 in 2024-02-13T00:46:51.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:50.429738+0000 mgr.y (mgr.14544) 381 : cluster [DBG] pgmap v629: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:46:51.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:50.522350+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e432: 8 total, 8 up, 4 in 2024-02-13T00:46:51.971 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:51.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:51.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:51.993 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:46:51.994 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:46:52.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:52.103 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:52.227 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:52.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:52.262 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:52.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:46:52.361 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:52.381 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:46:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:52.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:46:52.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.440+0000 7f948473b700 1 -- 172.21.15.162:0/511181281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c102270 msgr2=0x7f947c104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:52.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.440+0000 7f948473b700 1 --2- 172.21.15.162:0/511181281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c102270 0x7f947c104660 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7f9474009be0 tx=0x7f9474004d10).stop 2024-02-13T00:46:52.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.440+0000 7f948473b700 1 -- 172.21.15.162:0/511181281 shutdown_connections 2024-02-13T00:46:52.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.440+0000 7f948473b700 1 --2- 172.21.15.162:0/511181281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c102270 0x7f947c104660 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:52.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.440+0000 7f948473b700 1 --2- 172.21.15.162:0/511181281 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f947c0f8d10 0x7f947c0f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:52.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.440+0000 7f948473b700 1 --2- 172.21.15.162:0/511181281 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c0f95e0 0x7f947c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:52.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.440+0000 7f948473b700 1 -- 172.21.15.162:0/511181281 >> 172.21.15.162:0/511181281 conn(0x7f947c06eb90 msgr2=0x7f947c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:52.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.440+0000 7f948473b700 1 -- 172.21.15.162:0/511181281 shutdown_connections 2024-02-13T00:46:52.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.440+0000 7f948473b700 1 -- 172.21.15.162:0/511181281 wait complete. 2024-02-13T00:46:52.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f948473b700 1 librados: starting msgr at 2024-02-13T00:46:52.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f948473b700 1 librados: starting objecter 2024-02-13T00:46:52.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f948473b700 1 Processor -- start 2024-02-13T00:46:52.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f948473b700 1 -- start start 2024-02-13T00:46:52.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f948473b700 1 librados: setting wanted keys 2024-02-13T00:46:52.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f948473b700 1 librados: calling monclient init 2024-02-13T00:46:52.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f948473b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f947c0f8d10 0x7f947c10e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:52.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f948473b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f95e0 0x7f947c10e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:52.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f948473b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c102270 0x7f947c108250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:52.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f948473b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c108740 con 0x7f947c0f95e0 2024-02-13T00:46:52.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f948473b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c1088b0 con 0x7f947c0f8d10 2024-02-13T00:46:52.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f948473b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947c108a20 con 0x7f947c102270 2024-02-13T00:46:52.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f9481cd6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f95e0 0x7f947c10e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f9482cd8700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c102270 0x7f947c108250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f9481cd6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f95e0 0x7f947c10e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38088/0 (socket says 172.21.15.162:38088) 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f9481cd6700 1 -- 172.21.15.162:0/40510515 learned_addr learned my addr 172.21.15.162:0/40510515 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f94824d7700 1 --2- 172.21.15.162:0/40510515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f947c0f8d10 0x7f947c10e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f9482cd8700 1 -- 172.21.15.162:0/40510515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f947c0f8d10 msgr2=0x7f947c10e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f9482cd8700 1 --2- 172.21.15.162:0/40510515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f947c0f8d10 0x7f947c10e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f9482cd8700 1 -- 172.21.15.162:0/40510515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f95e0 msgr2=0x7f947c10e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.441+0000 7f9482cd8700 1 --2- 172.21.15.162:0/40510515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f95e0 0x7f947c10e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.442+0000 7f9482cd8700 1 -- 172.21.15.162:0/40510515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9478005040 con 0x7f947c102270 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.442+0000 7f9481cd6700 1 --2- 172.21.15.162:0/40510515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f95e0 0x7f947c10e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.442+0000 7f9482cd8700 1 --2- 172.21.15.162:0/40510515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c102270 0x7f947c108250 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7f9474009bb0 tx=0x7f9474004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.442+0000 7f94737fe700 10 client.?.objecter ms_handle_connect 0x7f947c102270 2024-02-13T00:46:52.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.442+0000 7f94737fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.442+0000 7f94737fe700 1 -- 172.21.15.162:0/40510515 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947400e040 con 0x7f947c102270 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.442+0000 7f94737fe700 1 -- 172.21.15.162:0/40510515 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9474018040 con 0x7f947c102270 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.442+0000 7f948473b700 1 -- 172.21.15.162:0/40510515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94740097c0 con 0x7f947c102270 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.442+0000 7f94737fe700 1 -- 172.21.15.162:0/40510515 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9474013e90 con 0x7f947c102270 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.442+0000 7f948473b700 10 client.37481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.442+0000 7f948473b700 1 -- 172.21.15.162:0/40510515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947c108f10 con 0x7f947c102270 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.442+0000 7f948473b700 1 librados: init done 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.443+0000 7f948473b700 1 -- 172.21.15.162:0/40510515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947c0475e0 con 0x7f947c102270 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.443+0000 7f94737fe700 1 -- 172.21.15.162:0/40510515 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f9474013520 con 0x7f947c102270 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.444+0000 7f94737fe700 1 --2- 172.21.15.162:0/40510515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9468062c90 0x7f9468065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.444+0000 7f94824d7700 1 --2- 172.21.15.162:0/40510515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9468062c90 0x7f9468065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.444+0000 7f94737fe700 1 -- 172.21.15.162:0/40510515 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(433..433 src has 52..433) v4 ==== 7967+0+0 (secure 0 0 0) 0x7f947407dbe0 con 0x7f947c102270 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.444+0000 7f94737fe700 10 client.37481.objecter ms_dispatch 0x7f947c001550 osd_map(433..433 src has 52..433) v4 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.444+0000 7f94737fe700 3 client.37481.objecter handle_osd_map got epochs [433,433] > 0 2024-02-13T00:46:52.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.444+0000 7f94737fe700 3 client.37481.objecter handle_osd_map decoding full epoch 433 2024-02-13T00:46:52.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.444+0000 7f94737fe700 20 client.37481.objecter dump_active .. 0 homeless 2024-02-13T00:46:52.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.444+0000 7f94737fe700 10 client.37481.objecter ms_handle_connect 0x7f9468062c90 2024-02-13T00:46:52.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.444+0000 7f94824d7700 1 --2- 172.21.15.162:0/40510515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9468062c90 0x7f9468065150 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 rx=0x7f946c004dc0 tx=0x7f946c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:52.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.448+0000 7f94737fe700 1 -- 172.21.15.162:0/40510515 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9474051200 con 0x7f947c102270 2024-02-13T00:46:52.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:52.521 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:52.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.619+0000 7f948473b700 1 -- 172.21.15.162:0/40510515 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f947c1097d0 con 0x7f9468062c90 2024-02-13T00:46:52.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.635+0000 7f94737fe700 1 -- 172.21.15.162:0/40510515 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+583379 (secure 0 0 0) 0x7f947c1097d0 con 0x7f9468062c90 2024-02-13T00:46:52.645 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:46:52.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 10 librados: watch_flush enter 2024-02-13T00:46:52.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 10 librados: watch_flush exit 2024-02-13T00:46:52.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 20 client.37481.objecter shutdown clearing up homeless session... 2024-02-13T00:46:52.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 10 client.37481.objecter successfully canceled tick 2024-02-13T00:46:52.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 -- 172.21.15.162:0/40510515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9468062c90 msgr2=0x7f9468065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:52.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 --2- 172.21.15.162:0/40510515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9468062c90 0x7f9468065150 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 rx=0x7f946c004dc0 tx=0x7f946c00b040).stop 2024-02-13T00:46:52.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 -- 172.21.15.162:0/40510515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c102270 msgr2=0x7f947c108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:52.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 --2- 172.21.15.162:0/40510515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c102270 0x7f947c108250 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7f9474009bb0 tx=0x7f9474004d10).stop 2024-02-13T00:46:52.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 -- 172.21.15.162:0/40510515 shutdown_connections 2024-02-13T00:46:52.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 --2- 172.21.15.162:0/40510515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9468062c90 0x7f9468065150 unknown :-1 s=CLOSED pgs=2909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:52.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 --2- 172.21.15.162:0/40510515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f947c102270 0x7f947c108250 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:52.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 --2- 172.21.15.162:0/40510515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f947c0f8d10 0x7f947c10e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:52.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 --2- 172.21.15.162:0/40510515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f947c0f95e0 0x7f947c10e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:52.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 -- 172.21.15.162:0/40510515 >> 172.21.15.162:0/40510515 conn(0x7f947c06eb90 msgr2=0x7f947c0708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:52.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 -- 172.21.15.162:0/40510515 shutdown_connections 2024-02-13T00:46:52.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 -- 172.21.15.162:0/40510515 wait complete. 2024-02-13T00:46:52.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:52.639+0000 7f948473b700 1 librados: shutdown 2024-02-13T00:46:52.653 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:46:52.692 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:51.525569+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e433: 8 total, 8 up, 4 in 2024-02-13T00:46:52.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-13T00:46:52.722 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:52.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:52.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:52.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:46:52.808 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:51.525569+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e433: 8 total, 8 up, 4 in 2024-02-13T00:46:52.809 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:51.525569+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e433: 8 total, 8 up, 4 in 2024-02-13T00:46:52.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:52.853 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:52.988 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:46:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:52.430138+0000 mgr.y (mgr.14544) 382 : cluster [DBG] pgmap v632: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-13T00:46:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:52.535294+0000 mon.a (mon.0) 2557 : cluster [DBG] osdmap e434: 8 total, 8 up, 4 in 2024-02-13T00:46:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:52.620785+0000 mgr.y (mgr.14544) 383 : audit [DBG] from='client.37481 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:46:53.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:52.430138+0000 mgr.y (mgr.14544) 382 : cluster [DBG] pgmap v632: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-13T00:46:53.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:52.535294+0000 mon.a (mon.0) 2557 : cluster [DBG] osdmap e434: 8 total, 8 up, 4 in 2024-02-13T00:46:53.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:52.620785+0000 mgr.y (mgr.14544) 383 : audit [DBG] from='client.37481 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:46:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:52.430138+0000 mgr.y (mgr.14544) 382 : cluster [DBG] pgmap v632: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-13T00:46:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:52.535294+0000 mon.a (mon.0) 2557 : cluster [DBG] osdmap e434: 8 total, 8 up, 4 in 2024-02-13T00:46:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:52.620785+0000 mgr.y (mgr.14544) 383 : audit [DBG] from='client.37481 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:46:54.102 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:54.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:54.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:54.234 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:54.359 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:54.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:54.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:54.493 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:54.515 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:54.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:54.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:54.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:54.647 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:53.107701+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:54.647 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:53.108059+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:54.647 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:53.540234+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e435: 8 total, 8 up, 4 in 2024-02-13T00:46:54.647 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:53.107701+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:54.647 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:53.108059+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:54.647 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:53.540234+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e435: 8 total, 8 up, 4 in 2024-02-13T00:46:54.649 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:54.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.719+0000 7f097c2db700 1 -- 172.21.15.162:0/554449478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09740f9b70 msgr2=0x7f09740f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:54.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.719+0000 7f097c2db700 1 --2- 172.21.15.162:0/554449478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09740f9b70 0x7f09740f9f50 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 rx=0x7f09700047b0 tx=0x7f0970009cb0).stop 2024-02-13T00:46:54.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 -- 172.21.15.162:0/554449478 shutdown_connections 2024-02-13T00:46:54.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 --2- 172.21.15.162:0/554449478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09740f8500 0x7f0974104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:46:54.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 --2- 172.21.15.162:0/554449478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09740f9b70 0x7f09740f9f50 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:54.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 --2- 172.21.15.162:0/554449478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09740fa440 0x7f09740f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:54.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 -- 172.21.15.162:0/554449478 >> 172.21.15.162:0/554449478 conn(0x7f09740f38c0 msgr2=0x7f09740f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:54.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 -- 172.21.15.162:0/554449478 shutdown_connections 2024-02-13T00:46:54.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 -- 172.21.15.162:0/554449478 wait complete. 2024-02-13T00:46:54.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 librados: starting msgr at 2024-02-13T00:46:54.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 librados: starting objecter 2024-02-13T00:46:54.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 Processor -- start 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 -- start start 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 librados: setting wanted keys 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.720+0000 7f097c2db700 1 librados: calling monclient init 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097c2db700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09740f8500 0x7f097406b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097c2db700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09740f9b70 0x7f097406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097c2db700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09740fa440 0x7f097406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097c2db700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09741167e0 con 0x7f09740f8500 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097c2db700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0974116950 con 0x7f09740fa440 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097c2db700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0974116b10 con 0x7f09740f9b70 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f0979876700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09740f9b70 0x7f097406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097a077700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09740f8500 0x7f097406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097a878700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09740fa440 0x7f097406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:54.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097a077700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09740f8500 0x7f097406b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38168/0 (socket says 172.21.15.162:38168) 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f0979876700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09740f9b70 0x7f097406b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:58242/0 (socket says 172.21.15.162:58242) 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097a077700 1 -- 172.21.15.162:0/3177053083 learned_addr learned my addr 172.21.15.162:0/3177053083 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097a077700 1 -- 172.21.15.162:0/3177053083 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09740f9b70 msgr2=0x7f097406b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097a077700 1 --2- 172.21.15.162:0/3177053083 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09740f9b70 0x7f097406b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097a077700 1 -- 172.21.15.162:0/3177053083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09740fa440 msgr2=0x7f097406bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097a077700 1 --2- 172.21.15.162:0/3177053083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09740fa440 0x7f097406bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097a077700 1 -- 172.21.15.162:0/3177053083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0964005040 con 0x7f09740f8500 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097a878700 1 --2- 172.21.15.162:0/3177053083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09740fa440 0x7f097406bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f097a077700 1 --2- 172.21.15.162:0/3177053083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09740f8500 0x7f097406b170 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f09700044e0 tx=0x7f0970007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f096b779700 10 client.?.objecter ms_handle_connect 0x7f09740f8500 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.721+0000 7f096b779700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.722+0000 7f096b779700 1 -- 172.21.15.162:0/3177053083 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0970018070 con 0x7f09740f8500 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.722+0000 7f097c2db700 1 -- 172.21.15.162:0/3177053083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0970005040 con 0x7f09740f8500 2024-02-13T00:46:54.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.722+0000 7f096b779700 1 -- 172.21.15.162:0/3177053083 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09700093b0 con 0x7f09740f8500 2024-02-13T00:46:54.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.722+0000 7f096b779700 1 -- 172.21.15.162:0/3177053083 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097001dd90 con 0x7f09740f8500 2024-02-13T00:46:54.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.722+0000 7f097c2db700 10 client.27387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:54.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.722+0000 7f097c2db700 1 -- 172.21.15.162:0/3177053083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0974117000 con 0x7f09740f8500 2024-02-13T00:46:54.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.722+0000 7f097c2db700 1 librados: init done 2024-02-13T00:46:54.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.723+0000 7f097c2db700 1 -- 172.21.15.162:0/3177053083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09740475e0 con 0x7f09740f8500 2024-02-13T00:46:54.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.723+0000 7f096b779700 1 -- 172.21.15.162:0/3177053083 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0970013ad0 con 0x7f09740f8500 2024-02-13T00:46:54.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.724+0000 7f096b779700 1 --2- 172.21.15.162:0/3177053083 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0960062c90 0x7f0960065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:54.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.724+0000 7f096b779700 1 -- 172.21.15.162:0/3177053083 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(435..435 src has 52..435) v4 ==== 7995+0+0 (secure 0 0 0) 0x7f097007e4b0 con 0x7f09740f8500 2024-02-13T00:46:54.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.724+0000 7f096b779700 10 client.27387.objecter ms_dispatch 0x7f0974001550 osd_map(435..435 src has 52..435) v4 2024-02-13T00:46:54.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.724+0000 7f096b779700 3 client.27387.objecter handle_osd_map got epochs [435,435] > 0 2024-02-13T00:46:54.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.724+0000 7f096b779700 3 client.27387.objecter handle_osd_map decoding full epoch 435 2024-02-13T00:46:54.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.724+0000 7f0979876700 1 --2- 172.21.15.162:0/3177053083 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0960062c90 0x7f0960065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:54.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.724+0000 7f096b779700 20 client.27387.objecter dump_active .. 0 homeless 2024-02-13T00:46:54.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.724+0000 7f096b779700 10 client.27387.objecter ms_handle_connect 0x7f0960062c90 2024-02-13T00:46:54.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.724+0000 7f0979876700 1 --2- 172.21.15.162:0/3177053083 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0960062c90 0x7f0960065150 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 rx=0x7f097406c8c0 tx=0x7f096400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:54.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.729+0000 7f096b779700 1 -- 172.21.15.162:0/3177053083 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0970051aa0 con 0x7f09740f8500 2024-02-13T00:46:54.807 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:46:54.807 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:46:54.827 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:53.107701+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:46:54.827 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:53.108059+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:46:54.828 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:53.540234+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e435: 8 total, 8 up, 4 in 2024-02-13T00:46:54.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:54.910+0000 7f097c2db700 1 -- 172.21.15.162:0/3177053083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f097406c600 con 0x7f09740f8500 2024-02-13T00:46:54.999 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:55.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:55.018 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":633,"stamp":"2024-02-13T00:46:52.536360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007308,"num_objects":389,"num_object_clones":2,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":1714,"num_read_kb":1336,"num_write":1232,"num_write_kb":4549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1852,"num_bytes_recovered":4017904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2695,"ondisk_log_size":2695,"up":714,"acting":714,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":705,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1274424,"kb_used_data":40424,"kb_used_omap":0,"kb_used_meta":1233920,"kb_avail":373706184,"statfs":{"total":383980142592,"available":382675132416,"internally_reserved":0,"allocated":41394176,"data_stored":27971079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1263534080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.082439"},"pg_stats":[{"pgid":"213.1f","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.593159+0000","last_change":"2024-02-13T00:46:48.592858+0000","last_active":"2024-02-13T00:46:48.593159+0000","last_peered":"2024-02-13T00:46:48.593159+0000","last_clean":"2024-02-13T00:46:48.593159+0000","last_became_active":"2024-02-13T00:46:48.592484+0000","last_became_peered":"2024-02-13T00:46:48.592484+0000","last_unstale":"2024-02-13T00:46:48.593159+0000","last_undegraded":"2024-02-13T00:46:48.593159+0000","last_fullsized":"2024-02-13T00:46:48.593159+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"213.1e","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.598084+0000","last_change":"2024-02-13T00:46:48.597599+0000","last_active":"2024-02-13T00:46:48.598084+0000","last_peered":"2024-02-13T00:46:48.598084+0000","last_clean":"2024-02-13T00:46:48.598084+0000","last_became_active":"2024-02-13T00:46:48.597320+0000","last_became_peered":"2024-02-13T00:46:48.597320+0000","last_unstale":"2024-02-13T00:46:48.598084+0000","last_undegraded":"2024-02-13T00:46:48.598084+0000","last_fullsized":"2024-02-13T00:46:48.598084+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"213.1d","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.597074+0000","last_change":"2024-02-13T00:46:48.595926+0000","last_active":"2024-02-13T00:46:48.597074+0000","last_peered":"2024-02-13T00:46:48.597074+0000","last_clean":"2024-02-13T00:46:48.597074+0000","last_became_active":"2024-02-13T00:46:48.595456+0000","last_became_peered":"2024-02-13T00:46:48.595456+0000","last_unstale":"2024-02-13T00:46:48.597074+0000","last_undegraded":"2024-02-13T00:46:48.597074+0000","last_fullsized":"2024-02-13T00:46:48.597074+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.1c","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.471170+0000","last_change":"2024-02-13T00:46:48.470986+0000","last_active":"2024-02-13T00:46:48.471170+0000","last_peered":"2024-02-13T00:46:48.471170+0000","last_clean":"2024-02-13T00:46:48.471170+0000","last_became_active":"2024-02-13T00:46:48.470685+0000","last_became_peered":"2024-02-13T00:46:48.470685+0000","last_unstale":"2024-02-13T00:46:48.471170+0000","last_undegraded":"2024-02-13T00:46:48.471170+0000","last_fullsized":"2024-02-13T00:46:48.471170+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"213.1b","version":"431'2","reported_seq":12,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.477523+0000","last_change":"2024-02-13T00:46:48.472880+0000","last_active":"2024-02-13T00:46:49.477523+0000","last_peered":"2024-02-13T00:46:49.477523+0000","last_clean":"2024-02-13T00:46:49.477523+0000","last_became_active":"2024-02-13T00:46:48.472574+0000","last_became_peered":"2024-02-13T00:46:48.472574+0000","last_unstale":"2024-02-13T00:46:49.477523+0000","last_undegraded":"2024-02-13T00:46:49.477523+0000","last_fullsized":"2024-02-13T00:46:49.477523+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"213.1a","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.596438+0000","last_change":"2024-02-13T00:46:48.594846+0000","last_active":"2024-02-13T00:46:48.596438+0000","last_peered":"2024-02-13T00:46:48.596438+0000","last_clean":"2024-02-13T00:46:48.596438+0000","last_became_active":"2024-02-13T00:46:48.594448+0000","last_became_peered":"2024-02-13T00:46:48.594448+0000","last_unstale":"2024-02-13T00:46:48.596438+0000","last_undegraded":"2024-02-13T00:46:48.596438+0000","last_fullsized":"2024-02-13T00:46:48.596438+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.19","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.596667+0000","last_change":"2024-02-13T00:46:48.594924+0000","last_active":"2024-02-13T00:46:48.596667+0000","last_peered":"2024-02-13T00:46:48.596667+0000","last_clean":"2024-02-13T00:46:48.596667+0000","last_became_active":"2024-02-13T00:46:48.594207+0000","last_became_peered":"2024-02-13T00:46:48.594207+0000","last_unstale":"2024-02-13T00:46:48.596667+0000","last_undegraded":"2024-02-13T00:46:48.596667+0000","last_fullsized":"2024-02-13T00:46:48.596667+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.18","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.472744+0000","last_change":"2024-02-13T00:46:48.472112+0000","last_active":"2024-02-13T00:46:48.472744+0000","last_peered":"2024-02-13T00:46:48.472744+0000","last_clean":"2024-02-13T00:46:48.472744+0000","last_became_active":"2024-02-13T00:46:48.471868+0000","last_became_peered":"2024-02-13T00:46:48.471868+0000","last_unstale":"2024-02-13T00:46:48.472744+0000","last_undegraded":"2024-02-13T00:46:48.472744+0000","last_fullsized":"2024-02-13T00:46:48.472744+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"213.17","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.472694+0000","last_change":"2024-02-13T00:46:48.471567+0000","last_active":"2024-02-13T00:46:48.472694+0000","last_peered":"2024-02-13T00:46:48.472694+0000","last_clean":"2024-02-13T00:46:48.472694+0000","last_became_active":"2024-02-13T00:46:48.471340+0000","last_became_peered":"2024-02-13T00:46:48.471340+0000","last_unstale":"2024-02-13T00:46:48.472694+0000","last_undegraded":"2024-02-13T00:46:48.472694+0000","last_fullsized":"2024-02-13T00:46:48.472694+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"213.16","version":"431'2","reported_seq":12,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.478693+0000","last_change":"2024-02-13T00:46:48.599050+0000","last_active":"2024-02-13T00:46:49.478693+0000","last_peered":"2024-02-13T00:46:49.478693+0000","last_clean":"2024-02-13T00:46:49.478693+0000","last_became_active":"2024-02-13T00:46:48.598758+0000","last_became_peered":"2024-02-13T00:46:48.598758+0000","last_unstale":"2024-02-13T00:46:49.478693+0000","last_undegraded":"2024-02-13T00:46:49.478693+0000","last_fullsized":"2024-02-13T00:46:49.478693+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.15","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.594088+0000","last_change":"2024-02-13T00:46:48.593533+0000","last_active":"2024-02-13T00:46:48.594088+0000","last_peered":"2024-02-13T00:46:48.594088+0000","last_clean":"2024-02-13T00:46:48.594088+0000","last_became_active":"2024-02-13T00:46:48.593220+0000","last_became_peered":"2024-02-13T00:46:48.593220+0000","last_unstale":"2024-02-13T00:46:48.594088+0000","last_undegraded":"2024-02-13T00:46:48.594088+0000","last_fullsized":"2024-02-13T00:46:48.594088+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"213.14","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.473912+0000","last_change":"2024-02-13T00:46:48.473713+0000","last_active":"2024-02-13T00:46:48.473912+0000","last_peered":"2024-02-13T00:46:48.473912+0000","last_clean":"2024-02-13T00:46:48.473912+0000","last_became_active":"2024-02-13T00:46:48.473228+0000","last_became_peered":"2024-02-13T00:46:48.473228+0000","last_unstale":"2024-02-13T00:46:48.473912+0000","last_undegraded":"2024-02-13T00:46:48.473912+0000","last_fullsized":"2024-02-13T00:46:48.473912+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"213.13","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.596862+0000","last_change":"2024-02-13T00:46:48.595535+0000","last_active":"2024-02-13T00:46:48.596862+0000","last_peered":"2024-02-13T00:46:48.596862+0000","last_clean":"2024-02-13T00:46:48.596862+0000","last_became_active":"2024-02-13T00:46:48.595253+0000","last_became_peered":"2024-02-13T00:46:48.595253+0000","last_unstale":"2024-02-13T00:46:48.596862+0000","last_undegraded":"2024-02-13T00:46:48.596862+0000","last_fullsized":"2024-02-13T00:46:48.596862+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.12","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.596422+0000","last_change":"2024-02-13T00:46:48.594748+0000","last_active":"2024-02-13T00:46:48.596422+0000","last_peered":"2024-02-13T00:46:48.596422+0000","last_clean":"2024-02-13T00:46:48.596422+0000","last_became_active":"2024-02-13T00:46:48.594197+0000","last_became_peered":"2024-02-13T00:46:48.594197+0000","last_unstale":"2024-02-13T00:46:48.596422+0000","last_undegraded":"2024-02-13T00:46:48.596422+0000","last_fullsized":"2024-02-13T00:46:48.596422+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.11","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.594252+0000","last_change":"2024-02-13T00:46:48.593675+0000","last_active":"2024-02-13T00:46:48.594252+0000","last_peered":"2024-02-13T00:46:48.594252+0000","last_clean":"2024-02-13T00:46:48.594252+0000","last_became_active":"2024-02-13T00:46:48.593124+0000","last_became_peered":"2024-02-13T00:46:48.593124+0000","last_unstale":"2024-02-13T00:46:48.594252+0000","last_undegraded":"2024-02-13T00:46:48.594252+0000","last_fullsized":"2024-02-13T00:46:48.594252+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"213.10","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.596927+0000","last_change":"2024-02-13T00:46:48.594932+0000","last_active":"2024-02-13T00:46:48.596927+0000","last_peered":"2024-02-13T00:46:48.596927+0000","last_clean":"2024-02-13T00:46:48.596927+0000","last_became_active":"2024-02-13T00:46:48.594439+0000","last_became_peered":"2024-02-13T00:46:48.594439+0000","last_unstale":"2024-02-13T00:46:48.596927+0000","last_undegraded":"2024-02-13T00:46:48.596927+0000","last_fullsized":"2024-02-13T00:46:48.596927+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.f","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.594071+0000","last_change":"2024-02-13T00:46:48.592731+0000","last_active":"2024-02-13T00:46:48.594071+0000","last_peered":"2024-02-13T00:46:48.594071+0000","last_clean":"2024-02-13T00:46:48.594071+0000","last_became_active":"2024-02-13T00:46:48.592446+0000","last_became_peered":"2024-02-13T00:46:48.592446+0000","last_unstale":"2024-02-13T00:46:48.594071+0000","last_undegraded":"2024-02-13T00:46:48.594071+0000","last_fullsized":"2024-02-13T00:46:48.594071+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"213.e","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.597117+0000","last_change":"2024-02-13T00:46:48.596299+0000","last_active":"2024-02-13T00:46:48.597117+0000","last_peered":"2024-02-13T00:46:48.597117+0000","last_clean":"2024-02-13T00:46:48.597117+0000","last_became_active":"2024-02-13T00:46:48.596035+0000","last_became_peered":"2024-02-13T00:46:48.596035+0000","last_unstale":"2024-02-13T00:46:48.597117+0000","last_undegraded":"2024-02-13T00:46:48.597117+0000","last_fullsized":"2024-02-13T00:46:48.597117+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.d","version":"431'2","reported_seq":12,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.471175+0000","last_change":"2024-02-13T00:46:48.594966+0000","last_active":"2024-02-13T00:46:49.471175+0000","last_peered":"2024-02-13T00:46:49.471175+0000","last_clean":"2024-02-13T00:46:49.471175+0000","last_became_active":"2024-02-13T00:46:48.594361+0000","last_became_peered":"2024-02-13T00:46:48.594361+0000","last_unstale":"2024-02-13T00:46:49.471175+0000","last_undegraded":"2024-02-13T00:46:49.471175+0000","last_fullsized":"2024-02-13T00:46:49.471175+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"213.c","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.473815+0000","last_change":"2024-02-13T00:46:48.473539+0000","last_active":"2024-02-13T00:46:48.473815+0000","last_peered":"2024-02-13T00:46:48.473815+0000","last_clean":"2024-02-13T00:46:48.473815+0000","last_became_active":"2024-02-13T00:46:48.472837+0000","last_became_peered":"2024-02-13T00:46:48.472837+0000","last_unstale":"2024-02-13T00:46:48.473815+0000","last_undegraded":"2024-02-13T00:46:48.473815+0000","last_fullsized":"2024-02-13T00:46:48.473815+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"213.b","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.593111+0000","last_change":"2024-02-13T00:46:48.592587+0000","last_active":"2024-02-13T00:46:48.593111+0000","last_peered":"2024-02-13T00:46:48.593111+0000","last_clean":"2024-02-13T00:46:48.593111+0000","last_became_active":"2024-02-13T00:46:48.592261+0000","last_became_peered":"2024-02-13T00:46:48.592261+0000","last_unstale":"2024-02-13T00:46:48.593111+0000","last_undegraded":"2024-02-13T00:46:48.593111+0000","last_fullsized":"2024-02-13T00:46:48.593111+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.a","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.596405+0000","last_change":"2024-02-13T00:46:48.594655+0000","last_active":"2024-02-13T00:46:48.596405+0000","last_peered":"2024-02-13T00:46:48.596405+0000","last_clean":"2024-02-13T00:46:48.596405+0000","last_became_active":"2024-02-13T00:46:48.593862+0000","last_became_peered":"2024-02-13T00:46:48.593862+0000","last_unstale":"2024-02-13T00:46:48.596405+0000","last_undegraded":"2024-02-13T00:46:48.596405+0000","last_fullsized":"2024-02-13T00:46:48.596405+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.9","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.473505+0000","last_change":"2024-02-13T00:46:48.473046+0000","last_active":"2024-02-13T00:46:48.473505+0000","last_peered":"2024-02-13T00:46:48.473505+0000","last_clean":"2024-02-13T00:46:48.473505+0000","last_became_active":"2024-02-13T00:46:48.472624+0000","last_became_peered":"2024-02-13T00:46:48.472624+0000","last_unstale":"2024-02-13T00:46:48.473505+0000","last_undegraded":"2024-02-13T00:46:48.473505+0000","last_fullsized":"2024-02-13T00:46:48.473505+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"213.8","version":"431'1","reported_seq":11,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.456634+0000","last_change":"2024-02-13T00:46:48.594851+0000","last_active":"2024-02-13T00:46:49.456634+0000","last_peered":"2024-02-13T00:46:49.456634+0000","last_clean":"2024-02-13T00:46:49.456634+0000","last_became_active":"2024-02-13T00:46:48.594220+0000","last_became_peered":"2024-02-13T00:46:48.594220+0000","last_unstale":"2024-02-13T00:46:49.456634+0000","last_undegraded":"2024-02-13T00:46:49.456634+0000","last_fullsized":"2024-02-13T00:46:49.456634+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.7","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.470935+0000","last_change":"2024-02-13T00:46:48.470512+0000","last_active":"2024-02-13T00:46:48.470935+0000","last_peered":"2024-02-13T00:46:48.470935+0000","last_clean":"2024-02-13T00:46:48.470935+0000","last_became_active":"2024-02-13T00:46:48.470247+0000","last_became_peered":"2024-02-13T00:46:48.470247+0000","last_unstale":"2024-02-13T00:46:48.470935+0000","last_undegraded":"2024-02-13T00:46:48.470935+0000","last_fullsized":"2024-02-13T00:46:48.470935+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.6","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.597814+0000","last_change":"2024-02-13T00:46:48.597264+0000","last_active":"2024-02-13T00:46:48.597814+0000","last_peered":"2024-02-13T00:46:48.597814+0000","last_clean":"2024-02-13T00:46:48.597814+0000","last_became_active":"2024-02-13T00:46:48.596974+0000","last_became_peered":"2024-02-13T00:46:48.596974+0000","last_unstale":"2024-02-13T00:46:48.597814+0000","last_undegraded":"2024-02-13T00:46:48.597814+0000","last_fullsized":"2024-02-13T00:46:48.597814+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"213.5","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.594069+0000","last_change":"2024-02-13T00:46:48.593496+0000","last_active":"2024-02-13T00:46:48.594069+0000","last_peered":"2024-02-13T00:46:48.594069+0000","last_clean":"2024-02-13T00:46:48.594069+0000","last_became_active":"2024-02-13T00:46:48.593201+0000","last_became_peered":"2024-02-13T00:46:48.593201+0000","last_unstale":"2024-02-13T00:46:48.594069+0000","last_undegraded":"2024-02-13T00:46:48.594069+0000","last_fullsized":"2024-02-13T00:46:48.594069+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"213.4","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.592823+0000","last_change":"2024-02-13T00:46:48.592254+0000","last_active":"2024-02-13T00:46:48.592823+0000","last_peered":"2024-02-13T00:46:48.592823+0000","last_clean":"2024-02-13T00:46:48.592823+0000","last_became_active":"2024-02-13T00:46:48.591937+0000","last_became_peered":"2024-02-13T00:46:48.591937+0000","last_unstale":"2024-02-13T00:46:48.592823+0000","last_undegraded":"2024-02-13T00:46:48.592823+0000","last_fullsized":"2024-02-13T00:46:48.592823+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"213.3","version":"431'1","reported_seq":11,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.464778+0000","last_change":"2024-02-13T00:46:48.592544+0000","last_active":"2024-02-13T00:46:49.464778+0000","last_peered":"2024-02-13T00:46:49.464778+0000","last_clean":"2024-02-13T00:46:49.464778+0000","last_became_active":"2024-02-13T00:46:48.592245+0000","last_became_peered":"2024-02-13T00:46:48.592245+0000","last_unstale":"2024-02-13T00:46:49.464778+0000","last_undegraded":"2024-02-13T00:46:49.464778+0000","last_fullsized":"2024-02-13T00:46:49.464778+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"213.2","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.470418+0000","last_change":"2024-02-13T00:46:48.470125+0000","last_active":"2024-02-13T00:46:48.470418+0000","last_peered":"2024-02-13T00:46:48.470418+0000","last_clean":"2024-02-13T00:46:48.470418+0000","last_became_active":"2024-02-13T00:46:48.469821+0000","last_became_peered":"2024-02-13T00:46:48.469821+0000","last_unstale":"2024-02-13T00:46:48.470418+0000","last_undegraded":"2024-02-13T00:46:48.470418+0000","last_fullsized":"2024-02-13T00:46:48.470418+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"213.1","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.470409+0000","last_change":"2024-02-13T00:46:48.470083+0000","last_active":"2024-02-13T00:46:48.470409+0000","last_peered":"2024-02-13T00:46:48.470409+0000","last_clean":"2024-02-13T00:46:48.470409+0000","last_became_active":"2024-02-13T00:46:48.469845+0000","last_became_peered":"2024-02-13T00:46:48.469845+0000","last_unstale":"2024-02-13T00:46:48.470409+0000","last_undegraded":"2024-02-13T00:46:48.470409+0000","last_fullsized":"2024-02-13T00:46:48.470409+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"213.0","version":"0'0","reported_seq":10,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.596891+0000","last_change":"2024-02-13T00:46:48.594684+0000","last_active":"2024-02-13T00:46:48.596891+0000","last_peered":"2024-02-13T00:46:48.596891+0000","last_clean":"2024-02-13T00:46:48.596891+0000","last_became_active":"2024-02-13T00:46:48.593891+0000","last_became_peered":"2024-02-13T00:46:48.593891+0000","last_unstale":"2024-02-13T00:46:48.596891+0000","last_undegraded":"2024-02-13T00:46:48.596891+0000","last_fullsized":"2024-02-13T00:46:48.596891+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:47.441957+0000","last_clean_scrub_stamp":"2024-02-13T00:46:47.441957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.2","version":"204'34","reported_seq":220,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462005+0000","last_change":"2024-02-13T00:46:35.432364+0000","last_active":"2024-02-13T00:46:48.462005+0000","last_peered":"2024-02-13T00:46:48.462005+0000","last_clean":"2024-02-13T00:46:48.462005+0000","last_became_active":"2024-02-13T00:46:01.448503+0000","last_became_peered":"2024-02-13T00:46:01.448503+0000","last_unstale":"2024-02-13T00:46:48.462005+0000","last_undegraded":"2024-02-13T00:46:48.462005+0000","last_fullsized":"2024-02-13T00:46:48.462005+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-02-13T00:46:19.267508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:19.267508+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.3","version":"375'104","reported_seq":627,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463061+0000","last_change":"2024-02-13T00:46:35.421257+0000","last_active":"2024-02-13T00:46:48.463061+0000","last_peered":"2024-02-13T00:46:48.463061+0000","last_clean":"2024-02-13T00:46:48.463061+0000","last_became_active":"2024-02-13T00:45:22.426662+0000","last_became_peered":"2024-02-13T00:45:22.426662+0000","last_unstale":"2024-02-13T00:46:48.463061+0000","last_undegraded":"2024-02-13T00:46:48.463061+0000","last_fullsized":"2024-02-13T00:46:48.463061+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'104","last_scrub_stamp":"2024-02-13T00:45:34.225334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.225334+0000","log_size":104,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.0","version":"388'38","reported_seq":448,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462001+0000","last_change":"2024-02-13T00:46:35.670480+0000","last_active":"2024-02-13T00:46:48.462001+0000","last_peered":"2024-02-13T00:46:48.462001+0000","last_clean":"2024-02-13T00:46:48.462001+0000","last_became_active":"2024-02-13T00:46:01.436132+0000","last_became_peered":"2024-02-13T00:46:01.436132+0000","last_unstale":"2024-02-13T00:46:48.462001+0000","last_undegraded":"2024-02-13T00:46:48.462001+0000","last_fullsized":"2024-02-13T00:46:48.462001+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'38","last_scrub_stamp":"2024-02-13T00:45:43.553743+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-02-13T00:45:43.553743+0000","last_clean_scrub_stamp":"2024-02-13T00:45:43.553743+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000030,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4000030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.1","version":"375'35","reported_seq":441,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458947+0000","last_change":"2024-02-13T00:46:35.421463+0000","last_active":"2024-02-13T00:46:48.458947+0000","last_peered":"2024-02-13T00:46:48.458947+0000","last_clean":"2024-02-13T00:46:48.458947+0000","last_became_active":"2024-02-13T00:45:15.420934+0000","last_became_peered":"2024-02-13T00:45:15.420934+0000","last_unstale":"2024-02-13T00:46:48.458947+0000","last_undegraded":"2024-02-13T00:46:48.458947+0000","last_fullsized":"2024-02-13T00:46:48.458947+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'34","last_scrub_stamp":"2024-02-13T00:45:29.241606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.241606+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.6","version":"400'20","reported_seq":369,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466737+0000","last_change":"2024-02-13T00:46:35.422997+0000","last_active":"2024-02-13T00:46:48.466737+0000","last_peered":"2024-02-13T00:46:48.466737+0000","last_clean":"2024-02-13T00:46:48.466737+0000","last_became_active":"2024-02-13T00:46:01.452805+0000","last_became_peered":"2024-02-13T00:46:01.452805+0000","last_unstale":"2024-02-13T00:46:48.466737+0000","last_undegraded":"2024-02-13T00:46:48.466737+0000","last_fullsized":"2024-02-13T00:46:48.466737+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'20","last_scrub_stamp":"2024-02-13T00:46:20.166298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:20.166298+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.7","version":"270'85","reported_seq":362,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466236+0000","last_change":"2024-02-13T00:46:35.674133+0000","last_active":"2024-02-13T00:46:48.466236+0000","last_peered":"2024-02-13T00:46:48.466236+0000","last_clean":"2024-02-13T00:46:48.466236+0000","last_became_active":"2024-02-13T00:46:01.458985+0000","last_became_peered":"2024-02-13T00:46:01.458985+0000","last_unstale":"2024-02-13T00:46:48.466236+0000","last_undegraded":"2024-02-13T00:46:48.466236+0000","last_fullsized":"2024-02-13T00:46:48.466236+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'85","last_scrub_stamp":"2024-02-13T00:45:02.230151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:02.230151+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.4","version":"407'59","reported_seq":168,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463967+0000","last_change":"2024-02-13T00:46:35.673002+0000","last_active":"2024-02-13T00:46:48.463967+0000","last_peered":"2024-02-13T00:46:48.463967+0000","last_clean":"2024-02-13T00:46:48.463967+0000","last_became_active":"2024-02-13T00:46:01.455621+0000","last_became_peered":"2024-02-13T00:46:01.455621+0000","last_unstale":"2024-02-13T00:46:48.463967+0000","last_undegraded":"2024-02-13T00:46:48.463967+0000","last_fullsized":"2024-02-13T00:46:48.463967+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'57","last_scrub_stamp":"2024-02-13T00:45:32.238130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.238130+0000","log_size":59,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":84,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.5","version":"224'56","reported_seq":565,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461338+0000","last_change":"2024-02-13T00:46:35.423606+0000","last_active":"2024-02-13T00:46:48.461338+0000","last_peered":"2024-02-13T00:46:48.461338+0000","last_clean":"2024-02-13T00:46:48.461338+0000","last_became_active":"2024-02-13T00:46:01.456637+0000","last_became_peered":"2024-02-13T00:46:01.456637+0000","last_unstale":"2024-02-13T00:46:48.461338+0000","last_undegraded":"2024-02-13T00:46:48.461338+0000","last_fullsized":"2024-02-13T00:46:48.461338+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'56","last_scrub_stamp":"2024-02-13T00:45:55.071118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:55.071118+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.a","version":"343'92","reported_seq":505,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.465646+0000","last_change":"2024-02-13T00:46:35.422709+0000","last_active":"2024-02-13T00:46:48.465646+0000","last_peered":"2024-02-13T00:46:48.465646+0000","last_clean":"2024-02-13T00:46:48.465646+0000","last_became_active":"2024-02-13T00:46:01.453135+0000","last_became_peered":"2024-02-13T00:46:01.453135+0000","last_unstale":"2024-02-13T00:46:48.465646+0000","last_undegraded":"2024-02-13T00:46:48.465646+0000","last_fullsized":"2024-02-13T00:46:48.465646+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'92","last_scrub_stamp":"2024-02-13T00:45:50.066059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:50.066059+0000","log_size":92,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.b","version":"222'17","reported_seq":456,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462970+0000","last_change":"2024-02-13T00:46:35.422505+0000","last_active":"2024-02-13T00:46:48.462970+0000","last_peered":"2024-02-13T00:46:48.462970+0000","last_clean":"2024-02-13T00:46:48.462970+0000","last_became_active":"2024-02-13T00:44:50.433096+0000","last_became_peered":"2024-02-13T00:44:50.433096+0000","last_unstale":"2024-02-13T00:46:48.462970+0000","last_undegraded":"2024-02-13T00:46:48.462970+0000","last_fullsized":"2024-02-13T00:46:48.462970+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'17","last_scrub_stamp":"2024-02-13T00:45:46.193088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.193088+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":74,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.8","version":"407'56","reported_seq":468,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462098+0000","last_change":"2024-02-13T00:46:35.670447+0000","last_active":"2024-02-13T00:46:48.462098+0000","last_peered":"2024-02-13T00:46:48.462098+0000","last_clean":"2024-02-13T00:46:48.462098+0000","last_became_active":"2024-02-13T00:46:01.445063+0000","last_became_peered":"2024-02-13T00:46:01.445063+0000","last_unstale":"2024-02-13T00:46:48.462098+0000","last_undegraded":"2024-02-13T00:46:48.462098+0000","last_fullsized":"2024-02-13T00:46:48.462098+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'53","last_scrub_stamp":"2024-02-13T00:44:30.046222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:30.046222+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.9","version":"343'36","reported_seq":446,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458884+0000","last_change":"2024-02-13T00:46:35.421109+0000","last_active":"2024-02-13T00:46:48.458884+0000","last_peered":"2024-02-13T00:46:48.458884+0000","last_clean":"2024-02-13T00:46:48.458884+0000","last_became_active":"2024-02-13T00:45:15.421026+0000","last_became_peered":"2024-02-13T00:45:15.421026+0000","last_unstale":"2024-02-13T00:46:48.458884+0000","last_undegraded":"2024-02-13T00:46:48.458884+0000","last_fullsized":"2024-02-13T00:46:48.458884+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'36","last_scrub_stamp":"2024-02-13T00:45:42.196133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.196133+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.e","version":"431'117","reported_seq":542,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.483608+0000","last_change":"2024-02-13T00:46:35.432063+0000","last_active":"2024-02-13T00:46:49.483608+0000","last_peered":"2024-02-13T00:46:49.483608+0000","last_clean":"2024-02-13T00:46:49.483608+0000","last_became_active":"2024-02-13T00:46:17.513379+0000","last_became_peered":"2024-02-13T00:46:17.513379+0000","last_unstale":"2024-02-13T00:46:49.483608+0000","last_undegraded":"2024-02-13T00:46:49.483608+0000","last_fullsized":"2024-02-13T00:46:49.483608+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'113","last_scrub_stamp":"2024-02-13T00:45:45.254008+0000","last_deep_scrub":"389'113","last_deep_scrub_stamp":"2024-02-13T00:45:45.254008+0000","last_clean_scrub_stamp":"2024-02-13T00:45:45.254008+0000","log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.f","version":"252'51","reported_seq":429,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466664+0000","last_change":"2024-02-13T00:46:35.422750+0000","last_active":"2024-02-13T00:46:48.466664+0000","last_peered":"2024-02-13T00:46:48.466664+0000","last_clean":"2024-02-13T00:46:48.466664+0000","last_became_active":"2024-02-13T00:46:01.440972+0000","last_became_peered":"2024-02-13T00:46:01.440972+0000","last_unstale":"2024-02-13T00:46:48.466664+0000","last_undegraded":"2024-02-13T00:46:48.466664+0000","last_fullsized":"2024-02-13T00:46:48.466664+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'51","last_scrub_stamp":"2024-02-13T00:45:26.215792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.215792+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.c","version":"400'108","reported_seq":719,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459082+0000","last_change":"2024-02-13T00:46:35.431163+0000","last_active":"2024-02-13T00:46:48.459082+0000","last_peered":"2024-02-13T00:46:48.459082+0000","last_clean":"2024-02-13T00:46:48.459082+0000","last_became_active":"2024-02-13T00:46:17.514719+0000","last_became_peered":"2024-02-13T00:46:17.514719+0000","last_unstale":"2024-02-13T00:46:48.459082+0000","last_undegraded":"2024-02-13T00:46:48.459082+0000","last_fullsized":"2024-02-13T00:46:48.459082+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'106","last_scrub_stamp":"2024-02-13T00:45:31.255154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.255154+0000","log_size":108,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":100,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.d","version":"343'91","reported_seq":540,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461151+0000","last_change":"2024-02-13T00:46:35.423176+0000","last_active":"2024-02-13T00:46:48.461151+0000","last_peered":"2024-02-13T00:46:48.461151+0000","last_clean":"2024-02-13T00:46:48.461151+0000","last_became_active":"2024-02-13T00:46:01.442967+0000","last_became_peered":"2024-02-13T00:46:01.442967+0000","last_unstale":"2024-02-13T00:46:48.461151+0000","last_undegraded":"2024-02-13T00:46:48.461151+0000","last_fullsized":"2024-02-13T00:46:48.461151+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'90","last_scrub_stamp":"2024-02-13T00:44:31.080244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:31.080244+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":335,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462975+0000","last_change":"2024-02-13T00:46:01.448309+0000","last_active":"2024-02-13T00:46:48.462975+0000","last_peered":"2024-02-13T00:46:48.462975+0000","last_clean":"2024-02-13T00:46:48.462975+0000","last_became_active":"2024-02-13T00:46:01.445363+0000","last_became_peered":"2024-02-13T00:46:01.445363+0000","last_unstale":"2024-02-13T00:46:48.462975+0000","last_undegraded":"2024-02-13T00:46:48.462975+0000","last_fullsized":"2024-02-13T00:46:48.462975+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.12","version":"343'24","reported_seq":394,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462660+0000","last_change":"2024-02-13T00:46:35.674236+0000","last_active":"2024-02-13T00:46:48.462660+0000","last_peered":"2024-02-13T00:46:48.462660+0000","last_clean":"2024-02-13T00:46:48.462660+0000","last_became_active":"2024-02-13T00:46:01.433618+0000","last_became_peered":"2024-02-13T00:46:01.433618+0000","last_unstale":"2024-02-13T00:46:48.462660+0000","last_undegraded":"2024-02-13T00:46:48.462660+0000","last_fullsized":"2024-02-13T00:46:48.462660+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'22","last_scrub_stamp":"2024-02-13T00:44:34.629429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.629429+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":260,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462411+0000","last_change":"2024-02-13T00:46:01.449894+0000","last_active":"2024-02-13T00:46:48.462411+0000","last_peered":"2024-02-13T00:46:48.462411+0000","last_clean":"2024-02-13T00:46:48.462411+0000","last_became_active":"2024-02-13T00:46:01.449570+0000","last_became_peered":"2024-02-13T00:46:01.449570+0000","last_unstale":"2024-02-13T00:46:48.462411+0000","last_undegraded":"2024-02-13T00:46:48.462411+0000","last_fullsized":"2024-02-13T00:46:48.462411+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.13","version":"236'73","reported_seq":237,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460406+0000","last_change":"2024-02-13T00:46:35.429460+0000","last_active":"2024-02-13T00:46:48.460406+0000","last_peered":"2024-02-13T00:46:48.460406+0000","last_clean":"2024-02-13T00:46:48.460406+0000","last_became_active":"2024-02-13T00:45:15.447922+0000","last_became_peered":"2024-02-13T00:45:15.447922+0000","last_unstale":"2024-02-13T00:46:48.460406+0000","last_undegraded":"2024-02-13T00:46:48.460406+0000","last_fullsized":"2024-02-13T00:46:48.460406+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'73","last_scrub_stamp":"2024-02-13T00:46:22.291170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:22.291170+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.10","version":"397'79","reported_seq":611,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460379+0000","last_change":"2024-02-13T00:46:35.429803+0000","last_active":"2024-02-13T00:46:48.460379+0000","last_peered":"2024-02-13T00:46:48.460379+0000","last_clean":"2024-02-13T00:46:48.460379+0000","last_became_active":"2024-02-13T00:46:01.440561+0000","last_became_peered":"2024-02-13T00:46:01.440561+0000","last_unstale":"2024-02-13T00:46:48.460379+0000","last_undegraded":"2024-02-13T00:46:48.460379+0000","last_fullsized":"2024-02-13T00:46:48.460379+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'73","last_scrub_stamp":"2024-02-13T00:45:27.173322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:27.173322+0000","log_size":79,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.11","version":"343'37","reported_seq":325,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458831+0000","last_change":"2024-02-13T00:46:35.420913+0000","last_active":"2024-02-13T00:46:48.458831+0000","last_peered":"2024-02-13T00:46:48.458831+0000","last_clean":"2024-02-13T00:46:48.458831+0000","last_became_active":"2024-02-13T00:44:50.443606+0000","last_became_peered":"2024-02-13T00:44:50.443606+0000","last_unstale":"2024-02-13T00:46:48.458831+0000","last_undegraded":"2024-02-13T00:46:48.458831+0000","last_fullsized":"2024-02-13T00:46:48.458831+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'37","last_scrub_stamp":"2024-02-13T00:46:22.170067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:22.170067+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.16","version":"375'75","reported_seq":527,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.465886+0000","last_change":"2024-02-13T00:46:35.673407+0000","last_active":"2024-02-13T00:46:48.465886+0000","last_peered":"2024-02-13T00:46:48.465886+0000","last_clean":"2024-02-13T00:46:48.465886+0000","last_became_active":"2024-02-13T00:46:01.449774+0000","last_became_peered":"2024-02-13T00:46:01.449774+0000","last_unstale":"2024-02-13T00:46:48.465886+0000","last_undegraded":"2024-02-13T00:46:48.465886+0000","last_fullsized":"2024-02-13T00:46:48.465886+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'75","last_scrub_stamp":"2024-02-13T00:45:46.157235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.157235+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.17","version":"343'75","reported_seq":559,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466318+0000","last_change":"2024-02-13T00:46:35.674344+0000","last_active":"2024-02-13T00:46:48.466318+0000","last_peered":"2024-02-13T00:46:48.466318+0000","last_clean":"2024-02-13T00:46:48.466318+0000","last_became_active":"2024-02-13T00:46:01.447341+0000","last_became_peered":"2024-02-13T00:46:01.447341+0000","last_unstale":"2024-02-13T00:46:48.466318+0000","last_undegraded":"2024-02-13T00:46:48.466318+0000","last_fullsized":"2024-02-13T00:46:48.466318+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'75","last_scrub_stamp":"2024-02-13T00:45:36.518979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:36.518979+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.14","version":"200'17","reported_seq":439,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.464626+0000","last_change":"2024-02-13T00:46:35.673208+0000","last_active":"2024-02-13T00:46:48.464626+0000","last_peered":"2024-02-13T00:46:48.464626+0000","last_clean":"2024-02-13T00:46:48.464626+0000","last_became_active":"2024-02-13T00:46:01.455179+0000","last_became_peered":"2024-02-13T00:46:01.455179+0000","last_unstale":"2024-02-13T00:46:48.464626+0000","last_undegraded":"2024-02-13T00:46:48.464626+0000","last_fullsized":"2024-02-13T00:46:48.464626+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'17","last_scrub_stamp":"2024-02-13T00:45:37.548974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.548974+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.15","version":"423'99","reported_seq":297,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460871+0000","last_change":"2024-02-13T00:46:37.425014+0000","last_active":"2024-02-13T00:46:48.460871+0000","last_peered":"2024-02-13T00:46:48.460871+0000","last_clean":"2024-02-13T00:46:48.460871+0000","last_became_active":"2024-02-13T00:46:01.452384+0000","last_became_peered":"2024-02-13T00:46:01.452384+0000","last_unstale":"2024-02-13T00:46:48.460871+0000","last_undegraded":"2024-02-13T00:46:48.460871+0000","last_fullsized":"2024-02-13T00:46:48.460871+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'78","last_scrub_stamp":"2024-02-13T00:45:36.188466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:36.188466+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":124,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458753+0000","last_change":"2024-02-13T00:46:01.445141+0000","last_active":"2024-02-13T00:46:48.458753+0000","last_peered":"2024-02-13T00:46:48.458753+0000","last_clean":"2024-02-13T00:46:48.458753+0000","last_became_active":"2024-02-13T00:46:01.444729+0000","last_became_peered":"2024-02-13T00:46:01.444729+0000","last_unstale":"2024-02-13T00:46:48.458753+0000","last_undegraded":"2024-02-13T00:46:48.458753+0000","last_fullsized":"2024-02-13T00:46:48.458753+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1a","version":"403'54","reported_seq":212,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461898+0000","last_change":"2024-02-13T00:46:35.431980+0000","last_active":"2024-02-13T00:46:48.461898+0000","last_peered":"2024-02-13T00:46:48.461898+0000","last_clean":"2024-02-13T00:46:48.461898+0000","last_became_active":"2024-02-13T00:45:15.436908+0000","last_became_peered":"2024-02-13T00:45:15.436908+0000","last_unstale":"2024-02-13T00:46:48.461898+0000","last_undegraded":"2024-02-13T00:46:48.461898+0000","last_fullsized":"2024-02-13T00:46:48.461898+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'54","last_scrub_stamp":"2024-02-13T00:46:26.235259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:26.235259+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":255,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":209,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458844+0000","last_change":"2024-02-13T00:45:15.426658+0000","last_active":"2024-02-13T00:46:48.458844+0000","last_peered":"2024-02-13T00:46:48.458844+0000","last_clean":"2024-02-13T00:46:48.458844+0000","last_became_active":"2024-02-13T00:45:15.426109+0000","last_became_peered":"2024-02-13T00:45:15.426109+0000","last_unstale":"2024-02-13T00:46:48.458844+0000","last_undegraded":"2024-02-13T00:46:48.458844+0000","last_fullsized":"2024-02-13T00:46:48.458844+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1b","version":"397'54","reported_seq":219,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460246+0000","last_change":"2024-02-13T00:46:35.428298+0000","last_active":"2024-02-13T00:46:48.460246+0000","last_peered":"2024-02-13T00:46:48.460246+0000","last_clean":"2024-02-13T00:46:48.460246+0000","last_became_active":"2024-02-13T00:46:01.458733+0000","last_became_peered":"2024-02-13T00:46:01.458733+0000","last_unstale":"2024-02-13T00:46:48.460246+0000","last_undegraded":"2024-02-13T00:46:48.460246+0000","last_fullsized":"2024-02-13T00:46:48.460246+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'54","last_scrub_stamp":"2024-02-13T00:45:57.130510+0000","last_deep_scrub":"397'54","last_deep_scrub_stamp":"2024-02-13T00:45:57.130510+0000","last_clean_scrub_stamp":"2024-02-13T00:45:57.130510+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":16,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":189,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":116,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459413+0000","last_change":"2024-02-13T00:44:39.526065+0000","last_active":"2024-02-13T00:46:48.459413+0000","last_peered":"2024-02-13T00:46:48.459413+0000","last_clean":"2024-02-13T00:46:48.459413+0000","last_became_active":"2024-02-13T00:44:39.525161+0000","last_became_peered":"2024-02-13T00:44:39.525161+0000","last_unstale":"2024-02-13T00:46:48.459413+0000","last_undegraded":"2024-02-13T00:46:48.459413+0000","last_fullsized":"2024-02-13T00:46:48.459413+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.18","version":"204'34","reported_seq":219,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458739+0000","last_change":"2024-02-13T00:46:35.422472+0000","last_active":"2024-02-13T00:46:48.458739+0000","last_peered":"2024-02-13T00:46:48.458739+0000","last_clean":"2024-02-13T00:46:48.458739+0000","last_became_active":"2024-02-13T00:46:01.438318+0000","last_became_peered":"2024-02-13T00:46:01.438318+0000","last_unstale":"2024-02-13T00:46:48.458739+0000","last_undegraded":"2024-02-13T00:46:48.458739+0000","last_fullsized":"2024-02-13T00:46:48.458739+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-02-13T00:45:29.218592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.218592+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462096+0000","last_change":"2024-02-13T00:46:01.448133+0000","last_active":"2024-02-13T00:46:48.462096+0000","last_peered":"2024-02-13T00:46:48.462096+0000","last_clean":"2024-02-13T00:46:48.462096+0000","last_became_active":"2024-02-13T00:46:01.447415+0000","last_became_peered":"2024-02-13T00:46:01.447415+0000","last_unstale":"2024-02-13T00:46:48.462096+0000","last_undegraded":"2024-02-13T00:46:48.462096+0000","last_fullsized":"2024-02-13T00:46:48.462096+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.19","version":"357'18","reported_seq":373,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463134+0000","last_change":"2024-02-13T00:46:35.671847+0000","last_active":"2024-02-13T00:46:48.463134+0000","last_peered":"2024-02-13T00:46:48.463134+0000","last_clean":"2024-02-13T00:46:48.463134+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:46:48.463134+0000","last_undegraded":"2024-02-13T00:46:48.463134+0000","last_fullsized":"2024-02-13T00:46:48.463134+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'18","last_scrub_stamp":"2024-02-13T00:45:51.098914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:51.098914+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459900+0000","last_change":"2024-02-13T00:46:01.459772+0000","last_active":"2024-02-13T00:46:48.459900+0000","last_peered":"2024-02-13T00:46:48.459900+0000","last_clean":"2024-02-13T00:46:48.459900+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:46:48.459900+0000","last_undegraded":"2024-02-13T00:46:48.459900+0000","last_fullsized":"2024-02-13T00:46:48.459900+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1e","version":"403'57","reported_seq":265,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460091+0000","last_change":"2024-02-13T00:46:35.432270+0000","last_active":"2024-02-13T00:46:48.460091+0000","last_peered":"2024-02-13T00:46:48.460091+0000","last_clean":"2024-02-13T00:46:48.460091+0000","last_became_active":"2024-02-13T00:45:15.446531+0000","last_became_peered":"2024-02-13T00:45:15.446531+0000","last_unstale":"2024-02-13T00:46:48.460091+0000","last_undegraded":"2024-02-13T00:46:48.460091+0000","last_fullsized":"2024-02-13T00:46:48.460091+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'57","last_scrub_stamp":"2024-02-13T00:46:33.286813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:33.286813+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":393,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459397+0000","last_change":"2024-02-13T00:46:01.463062+0000","last_active":"2024-02-13T00:46:48.459397+0000","last_peered":"2024-02-13T00:46:48.459397+0000","last_clean":"2024-02-13T00:46:48.459397+0000","last_became_active":"2024-02-13T00:46:01.462319+0000","last_became_peered":"2024-02-13T00:46:01.462319+0000","last_unstale":"2024-02-13T00:46:48.459397+0000","last_undegraded":"2024-02-13T00:46:48.459397+0000","last_fullsized":"2024-02-13T00:46:48.459397+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1f","version":"392'68","reported_seq":452,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466417+0000","last_change":"2024-02-13T00:46:35.674461+0000","last_active":"2024-02-13T00:46:48.466417+0000","last_peered":"2024-02-13T00:46:48.466417+0000","last_clean":"2024-02-13T00:46:48.466417+0000","last_became_active":"2024-02-13T00:44:50.441446+0000","last_became_peered":"2024-02-13T00:44:50.441446+0000","last_unstale":"2024-02-13T00:46:48.466417+0000","last_undegraded":"2024-02-13T00:46:48.466417+0000","last_fullsized":"2024-02-13T00:46:48.466417+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'68","last_scrub_stamp":"2024-02-13T00:45:51.651226+0000","last_deep_scrub":"392'68","last_deep_scrub_stamp":"2024-02-13T00:45:51.651226+0000","last_clean_scrub_stamp":"2024-02-13T00:45:51.651226+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":116,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461364+0000","last_change":"2024-02-13T00:44:39.525601+0000","last_active":"2024-02-13T00:46:48.461364+0000","last_peered":"2024-02-13T00:46:48.461364+0000","last_clean":"2024-02-13T00:46:48.461364+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:46:48.461364+0000","last_undegraded":"2024-02-13T00:46:48.461364+0000","last_fullsized":"2024-02-13T00:46:48.461364+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1c","version":"397'56","reported_seq":551,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459205+0000","last_change":"2024-02-13T00:46:35.431168+0000","last_active":"2024-02-13T00:46:48.459205+0000","last_peered":"2024-02-13T00:46:48.459205+0000","last_clean":"2024-02-13T00:46:48.459205+0000","last_became_active":"2024-02-13T00:46:01.462341+0000","last_became_peered":"2024-02-13T00:46:01.462341+0000","last_unstale":"2024-02-13T00:46:48.459205+0000","last_undegraded":"2024-02-13T00:46:48.459205+0000","last_fullsized":"2024-02-13T00:46:48.459205+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'52","last_scrub_stamp":"2024-02-13T00:45:44.205380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:44.205380+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":200,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466619+0000","last_change":"2024-02-13T00:46:01.455629+0000","last_active":"2024-02-13T00:46:48.466619+0000","last_peered":"2024-02-13T00:46:48.466619+0000","last_clean":"2024-02-13T00:46:48.466619+0000","last_became_active":"2024-02-13T00:46:01.452801+0000","last_became_peered":"2024-02-13T00:46:01.452801+0000","last_unstale":"2024-02-13T00:46:48.466619+0000","last_undegraded":"2024-02-13T00:46:48.466619+0000","last_fullsized":"2024-02-13T00:46:48.466619+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1d","version":"343'69","reported_seq":541,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461246+0000","last_change":"2024-02-13T00:46:35.423342+0000","last_active":"2024-02-13T00:46:48.461246+0000","last_peered":"2024-02-13T00:46:48.461246+0000","last_clean":"2024-02-13T00:46:48.461246+0000","last_became_active":"2024-02-13T00:45:15.435242+0000","last_became_peered":"2024-02-13T00:45:15.435242+0000","last_unstale":"2024-02-13T00:46:48.461246+0000","last_undegraded":"2024-02-13T00:46:48.461246+0000","last_fullsized":"2024-02-13T00:46:48.461246+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'69","last_scrub_stamp":"2024-02-13T00:46:28.118117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:28.118117+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":356,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462889+0000","last_change":"2024-02-13T00:45:39.562095+0000","last_active":"2024-02-13T00:46:48.462889+0000","last_peered":"2024-02-13T00:46:48.462889+0000","last_clean":"2024-02-13T00:46:48.462889+0000","last_became_active":"2024-02-13T00:44:39.505121+0000","last_became_peered":"2024-02-13T00:44:39.505121+0000","last_unstale":"2024-02-13T00:46:48.462889+0000","last_undegraded":"2024-02-13T00:46:48.462889+0000","last_fullsized":"2024-02-13T00:46:48.462889+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:39.561994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:39.561994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"363'14","reported_seq":167,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.320268+0000","last_change":"2024-02-13T00:45:53.115355+0000","last_active":"2024-02-13T00:46:49.320268+0000","last_peered":"2024-02-13T00:46:49.320268+0000","last_clean":"2024-02-13T00:46:49.320268+0000","last_became_active":"2024-02-13T00:45:15.435476+0000","last_became_peered":"2024-02-13T00:45:15.435476+0000","last_unstale":"2024-02-13T00:46:49.320268+0000","last_undegraded":"2024-02-13T00:46:49.320268+0000","last_fullsized":"2024-02-13T00:46:49.320268+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:45:53.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.115260+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":437,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462247+0000","last_change":"2024-02-13T00:46:03.880562+0000","last_active":"2024-02-13T00:46:48.462247+0000","last_peered":"2024-02-13T00:46:48.462247+0000","last_clean":"2024-02-13T00:46:48.462247+0000","last_became_active":"2024-02-13T00:46:01.438056+0000","last_became_peered":"2024-02-13T00:46:01.438056+0000","last_unstale":"2024-02-13T00:46:48.462247+0000","last_undegraded":"2024-02-13T00:46:48.462247+0000","last_fullsized":"2024-02-13T00:46:48.462247+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:41:31.413581+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:38:59.548872+0000","last_clean_scrub_stamp":"2024-02-13T00:41:31.413581+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":425,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459102+0000","last_change":"2024-02-13T00:46:01.464695+0000","last_active":"2024-02-13T00:46:48.459102+0000","last_peered":"2024-02-13T00:46:48.459102+0000","last_clean":"2024-02-13T00:46:48.459102+0000","last_became_active":"2024-02-13T00:46:01.460811+0000","last_became_peered":"2024-02-13T00:46:01.460811+0000","last_unstale":"2024-02-13T00:46:48.459102+0000","last_undegraded":"2024-02-13T00:46:48.459102+0000","last_fullsized":"2024-02-13T00:46:48.459102+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.340080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.340080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":331,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460292+0000","last_change":"2024-02-13T00:46:01.451838+0000","last_active":"2024-02-13T00:46:48.460292+0000","last_peered":"2024-02-13T00:46:48.460292+0000","last_clean":"2024-02-13T00:46:48.460292+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:46:48.460292+0000","last_undegraded":"2024-02-13T00:46:48.460292+0000","last_fullsized":"2024-02-13T00:46:48.460292+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:45.416238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:45.416238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":374,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458914+0000","last_change":"2024-02-13T00:46:01.448237+0000","last_active":"2024-02-13T00:46:48.458914+0000","last_peered":"2024-02-13T00:46:48.458914+0000","last_clean":"2024-02-13T00:46:48.458914+0000","last_became_active":"2024-02-13T00:46:01.446660+0000","last_became_peered":"2024-02-13T00:46:01.446660+0000","last_unstale":"2024-02-13T00:46:48.458914+0000","last_undegraded":"2024-02-13T00:46:48.458914+0000","last_fullsized":"2024-02-13T00:46:48.458914+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.260871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.260871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":421,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.464262+0000","last_change":"2024-02-13T00:42:19.652772+0000","last_active":"2024-02-13T00:46:48.464262+0000","last_peered":"2024-02-13T00:46:48.464262+0000","last_clean":"2024-02-13T00:46:48.464262+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:46:48.464262+0000","last_undegraded":"2024-02-13T00:46:48.464262+0000","last_fullsized":"2024-02-13T00:46:48.464262+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:42:19.652689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:19.652689+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":410,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462069+0000","last_change":"2024-02-13T00:44:50.430435+0000","last_active":"2024-02-13T00:46:48.462069+0000","last_peered":"2024-02-13T00:46:48.462069+0000","last_clean":"2024-02-13T00:46:48.462069+0000","last_became_active":"2024-02-13T00:44:50.429994+0000","last_became_peered":"2024-02-13T00:44:50.429994+0000","last_unstale":"2024-02-13T00:46:48.462069+0000","last_undegraded":"2024-02-13T00:46:48.462069+0000","last_fullsized":"2024-02-13T00:46:48.462069+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:20.102471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:20.102471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":326,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462508+0000","last_change":"2024-02-13T00:44:50.434925+0000","last_active":"2024-02-13T00:46:48.462508+0000","last_peered":"2024-02-13T00:46:48.462508+0000","last_clean":"2024-02-13T00:46:48.462508+0000","last_became_active":"2024-02-13T00:44:50.433984+0000","last_became_peered":"2024-02-13T00:44:50.433984+0000","last_unstale":"2024-02-13T00:46:48.462508+0000","last_undegraded":"2024-02-13T00:46:48.462508+0000","last_fullsized":"2024-02-13T00:46:48.462508+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.512181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.512181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":413,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463755+0000","last_change":"2024-02-13T00:45:15.435610+0000","last_active":"2024-02-13T00:46:48.463755+0000","last_peered":"2024-02-13T00:46:48.463755+0000","last_clean":"2024-02-13T00:46:48.463755+0000","last_became_active":"2024-02-13T00:45:15.434853+0000","last_became_peered":"2024-02-13T00:45:15.434853+0000","last_unstale":"2024-02-13T00:46:48.463755+0000","last_undegraded":"2024-02-13T00:46:48.463755+0000","last_fullsized":"2024-02-13T00:46:48.463755+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":39,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462108+0000","last_change":"2024-02-13T00:46:01.455201+0000","last_active":"2024-02-13T00:46:48.462108+0000","last_peered":"2024-02-13T00:46:48.462108+0000","last_clean":"2024-02-13T00:46:48.462108+0000","last_became_active":"2024-02-13T00:46:01.454882+0000","last_became_peered":"2024-02-13T00:46:01.454882+0000","last_unstale":"2024-02-13T00:46:48.462108+0000","last_undegraded":"2024-02-13T00:46:48.462108+0000","last_fullsized":"2024-02-13T00:46:48.462108+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:35.197045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:35.197045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"91'35","reported_seq":200,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461628+0000","last_change":"2024-02-13T00:46:09.256189+0000","last_active":"2024-02-13T00:46:48.461628+0000","last_peered":"2024-02-13T00:46:48.461628+0000","last_clean":"2024-02-13T00:46:48.461628+0000","last_became_active":"2024-02-13T00:46:01.459978+0000","last_became_peered":"2024-02-13T00:46:01.459978+0000","last_unstale":"2024-02-13T00:46:48.461628+0000","last_undegraded":"2024-02-13T00:46:48.461628+0000","last_fullsized":"2024-02-13T00:46:48.461628+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:44:24.196214+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:44:24.196214+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":49,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.465284+0000","last_change":"2024-02-13T00:46:30.641539+0000","last_active":"2024-02-13T00:46:48.465284+0000","last_peered":"2024-02-13T00:46:48.465284+0000","last_clean":"2024-02-13T00:46:48.465284+0000","last_became_active":"2024-02-13T00:46:01.444548+0000","last_became_peered":"2024-02-13T00:46:01.444548+0000","last_unstale":"2024-02-13T00:46:48.465284+0000","last_undegraded":"2024-02-13T00:46:48.465284+0000","last_fullsized":"2024-02-13T00:46:48.465284+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.641460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.641460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":131,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463831+0000","last_change":"2024-02-13T00:46:01.455578+0000","last_active":"2024-02-13T00:46:48.463831+0000","last_peered":"2024-02-13T00:46:48.463831+0000","last_clean":"2024-02-13T00:46:48.463831+0000","last_became_active":"2024-02-13T00:46:01.454642+0000","last_became_peered":"2024-02-13T00:46:01.454642+0000","last_unstale":"2024-02-13T00:46:48.463831+0000","last_undegraded":"2024-02-13T00:46:48.463831+0000","last_fullsized":"2024-02-13T00:46:48.463831+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.592672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":252,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462324+0000","last_change":"2024-02-13T00:45:15.429026+0000","last_active":"2024-02-13T00:46:48.462324+0000","last_peered":"2024-02-13T00:46:48.462324+0000","last_clean":"2024-02-13T00:46:48.462324+0000","last_became_active":"2024-02-13T00:45:15.427945+0000","last_became_peered":"2024-02-13T00:45:15.427945+0000","last_unstale":"2024-02-13T00:46:48.462324+0000","last_undegraded":"2024-02-13T00:46:48.462324+0000","last_fullsized":"2024-02-13T00:46:48.462324+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":355,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461810+0000","last_change":"2024-02-13T00:45:26.255907+0000","last_active":"2024-02-13T00:46:48.461810+0000","last_peered":"2024-02-13T00:46:48.461810+0000","last_clean":"2024-02-13T00:46:48.461810+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:46:48.461810+0000","last_undegraded":"2024-02-13T00:46:48.461810+0000","last_fullsized":"2024-02-13T00:46:48.461810+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.255842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.255842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":166,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461899+0000","last_change":"2024-02-13T00:44:54.041282+0000","last_active":"2024-02-13T00:46:48.461899+0000","last_peered":"2024-02-13T00:46:48.461899+0000","last_clean":"2024-02-13T00:46:48.461899+0000","last_became_active":"2024-02-13T00:44:50.432807+0000","last_became_peered":"2024-02-13T00:44:50.432807+0000","last_unstale":"2024-02-13T00:46:48.461899+0000","last_undegraded":"2024-02-13T00:46:48.461899+0000","last_fullsized":"2024-02-13T00:46:48.461899+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:42:17.413826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.413826+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":207,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458453+0000","last_change":"2024-02-13T00:46:04.708510+0000","last_active":"2024-02-13T00:46:48.458453+0000","last_peered":"2024-02-13T00:46:48.458453+0000","last_clean":"2024-02-13T00:46:48.458453+0000","last_became_active":"2024-02-13T00:46:01.447801+0000","last_became_peered":"2024-02-13T00:46:01.447801+0000","last_unstale":"2024-02-13T00:46:48.458453+0000","last_undegraded":"2024-02-13T00:46:48.458453+0000","last_fullsized":"2024-02-13T00:46:48.458453+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:45:24.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:24.269595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":390,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461628+0000","last_change":"2024-02-13T00:42:28.037452+0000","last_active":"2024-02-13T00:46:48.461628+0000","last_peered":"2024-02-13T00:46:48.461628+0000","last_clean":"2024-02-13T00:46:48.461628+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:46:48.461628+0000","last_undegraded":"2024-02-13T00:46:48.461628+0000","last_fullsized":"2024-02-13T00:46:48.461628+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:28.037365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:28.037365+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":401,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459479+0000","last_change":"2024-02-13T00:44:50.441408+0000","last_active":"2024-02-13T00:46:48.459479+0000","last_peered":"2024-02-13T00:46:48.459479+0000","last_clean":"2024-02-13T00:46:48.459479+0000","last_became_active":"2024-02-13T00:44:50.440301+0000","last_became_peered":"2024-02-13T00:44:50.440301+0000","last_unstale":"2024-02-13T00:46:48.459479+0000","last_undegraded":"2024-02-13T00:46:48.459479+0000","last_fullsized":"2024-02-13T00:46:48.459479+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:25.502311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.502311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":47,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.464476+0000","last_change":"2024-02-13T00:46:01.454885+0000","last_active":"2024-02-13T00:46:48.464476+0000","last_peered":"2024-02-13T00:46:48.464476+0000","last_clean":"2024-02-13T00:46:48.464476+0000","last_became_active":"2024-02-13T00:46:01.454535+0000","last_became_peered":"2024-02-13T00:46:01.454535+0000","last_unstale":"2024-02-13T00:46:48.464476+0000","last_undegraded":"2024-02-13T00:46:48.464476+0000","last_fullsized":"2024-02-13T00:46:48.464476+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.488353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.488353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":128,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459960+0000","last_change":"2024-02-13T00:44:46.481036+0000","last_active":"2024-02-13T00:46:48.459960+0000","last_peered":"2024-02-13T00:46:48.459960+0000","last_clean":"2024-02-13T00:46:48.459960+0000","last_became_active":"2024-02-13T00:44:39.523138+0000","last_became_peered":"2024-02-13T00:44:39.523138+0000","last_unstale":"2024-02-13T00:46:48.459960+0000","last_undegraded":"2024-02-13T00:46:48.459960+0000","last_fullsized":"2024-02-13T00:46:48.459960+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:43:00.776560+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.776560+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":419,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.465971+0000","last_change":"2024-02-13T00:46:01.453216+0000","last_active":"2024-02-13T00:46:48.465971+0000","last_peered":"2024-02-13T00:46:48.465971+0000","last_clean":"2024-02-13T00:46:48.465971+0000","last_became_active":"2024-02-13T00:46:01.452674+0000","last_became_peered":"2024-02-13T00:46:01.452674+0000","last_unstale":"2024-02-13T00:46:48.465971+0000","last_undegraded":"2024-02-13T00:46:48.465971+0000","last_fullsized":"2024-02-13T00:46:48.465971+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:24.059313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:24.059313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"404'11","reported_seq":82,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.320367+0000","last_change":"2024-02-13T00:46:37.662485+0000","last_active":"2024-02-13T00:46:49.320367+0000","last_peered":"2024-02-13T00:46:49.320367+0000","last_clean":"2024-02-13T00:46:49.320367+0000","last_became_active":"2024-02-13T00:46:01.456187+0000","last_became_peered":"2024-02-13T00:46:01.456187+0000","last_unstale":"2024-02-13T00:46:49.320367+0000","last_undegraded":"2024-02-13T00:46:49.320367+0000","last_fullsized":"2024-02-13T00:46:49.320367+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'11","last_scrub_stamp":"2024-02-13T00:46:37.662413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.662413+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":391,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467161+0000","last_change":"2024-02-13T00:46:05.839833+0000","last_active":"2024-02-13T00:46:48.467161+0000","last_peered":"2024-02-13T00:46:48.467161+0000","last_clean":"2024-02-13T00:46:48.467161+0000","last_became_active":"2024-02-13T00:46:01.455274+0000","last_became_peered":"2024-02-13T00:46:01.455274+0000","last_unstale":"2024-02-13T00:46:48.467161+0000","last_undegraded":"2024-02-13T00:46:48.467161+0000","last_fullsized":"2024-02-13T00:46:48.467161+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:44:25.230980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.230980+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":200,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467212+0000","last_change":"2024-02-13T00:46:01.457421+0000","last_active":"2024-02-13T00:46:48.467212+0000","last_peered":"2024-02-13T00:46:48.467212+0000","last_clean":"2024-02-13T00:46:48.467212+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:46:48.467212+0000","last_undegraded":"2024-02-13T00:46:48.467212+0000","last_fullsized":"2024-02-13T00:46:48.467212+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:18.089252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:18.089252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":490,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467117+0000","last_change":"2024-02-13T00:46:05.836637+0000","last_active":"2024-02-13T00:46:48.467117+0000","last_peered":"2024-02-13T00:46:48.467117+0000","last_clean":"2024-02-13T00:46:48.467117+0000","last_became_active":"2024-02-13T00:46:01.448705+0000","last_became_peered":"2024-02-13T00:46:01.448705+0000","last_unstale":"2024-02-13T00:46:48.467117+0000","last_undegraded":"2024-02-13T00:46:48.467117+0000","last_fullsized":"2024-02-13T00:46:48.467117+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:42:21.069482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.069482+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":451,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459399+0000","last_change":"2024-02-13T00:45:15.445834+0000","last_active":"2024-02-13T00:46:48.459399+0000","last_peered":"2024-02-13T00:46:48.459399+0000","last_clean":"2024-02-13T00:46:48.459399+0000","last_became_active":"2024-02-13T00:45:15.445228+0000","last_became_peered":"2024-02-13T00:45:15.445228+0000","last_unstale":"2024-02-13T00:46:48.459399+0000","last_undegraded":"2024-02-13T00:46:48.459399+0000","last_fullsized":"2024-02-13T00:46:48.459399+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:43:34.320637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.320637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"404'10","reported_seq":74,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.320304+0000","last_change":"2024-02-13T00:46:08.351827+0000","last_active":"2024-02-13T00:46:49.320304+0000","last_peered":"2024-02-13T00:46:49.320304+0000","last_clean":"2024-02-13T00:46:49.320304+0000","last_became_active":"2024-02-13T00:46:01.448785+0000","last_became_peered":"2024-02-13T00:46:01.448785+0000","last_unstale":"2024-02-13T00:46:49.320304+0000","last_undegraded":"2024-02-13T00:46:49.320304+0000","last_fullsized":"2024-02-13T00:46:49.320304+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-02-13T00:42:23.496800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:23.496800+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":41,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461481+0000","last_change":"2024-02-13T00:46:01.457110+0000","last_active":"2024-02-13T00:46:48.461481+0000","last_peered":"2024-02-13T00:46:48.461481+0000","last_clean":"2024-02-13T00:46:48.461481+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:46:48.461481+0000","last_undegraded":"2024-02-13T00:46:48.461481+0000","last_fullsized":"2024-02-13T00:46:48.461481+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.092250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.092250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":356,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467243+0000","last_change":"2024-02-13T00:46:01.456531+0000","last_active":"2024-02-13T00:46:48.467243+0000","last_peered":"2024-02-13T00:46:48.467243+0000","last_clean":"2024-02-13T00:46:48.467243+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:46:48.467243+0000","last_undegraded":"2024-02-13T00:46:48.467243+0000","last_fullsized":"2024-02-13T00:46:48.467243+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:44:59.963616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":130,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460377+0000","last_change":"2024-02-13T00:44:45.962479+0000","last_active":"2024-02-13T00:46:48.460377+0000","last_peered":"2024-02-13T00:46:48.460377+0000","last_clean":"2024-02-13T00:46:48.460377+0000","last_became_active":"2024-02-13T00:44:39.521984+0000","last_became_peered":"2024-02-13T00:44:39.521984+0000","last_unstale":"2024-02-13T00:46:48.460377+0000","last_undegraded":"2024-02-13T00:46:48.460377+0000","last_fullsized":"2024-02-13T00:46:48.460377+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:42:52.245750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.245750+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":239,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462667+0000","last_change":"2024-02-13T00:46:23.757417+0000","last_active":"2024-02-13T00:46:48.462667+0000","last_peered":"2024-02-13T00:46:48.462667+0000","last_clean":"2024-02-13T00:46:48.462667+0000","last_became_active":"2024-02-13T00:45:15.419239+0000","last_became_peered":"2024-02-13T00:45:15.419239+0000","last_unstale":"2024-02-13T00:46:48.462667+0000","last_undegraded":"2024-02-13T00:46:48.462667+0000","last_fullsized":"2024-02-13T00:46:48.462667+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:23.757351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:23.757351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":336,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459992+0000","last_change":"2024-02-13T00:46:01.460527+0000","last_active":"2024-02-13T00:46:48.459992+0000","last_peered":"2024-02-13T00:46:48.459992+0000","last_clean":"2024-02-13T00:46:48.459992+0000","last_became_active":"2024-02-13T00:46:01.459122+0000","last_became_peered":"2024-02-13T00:46:01.459122+0000","last_unstale":"2024-02-13T00:46:48.459992+0000","last_undegraded":"2024-02-13T00:46:48.459992+0000","last_fullsized":"2024-02-13T00:46:48.459992+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:57.546517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.546517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459454+0000","last_change":"2024-02-13T00:46:01.463229+0000","last_active":"2024-02-13T00:46:48.459454+0000","last_peered":"2024-02-13T00:46:48.459454+0000","last_clean":"2024-02-13T00:46:48.459454+0000","last_became_active":"2024-02-13T00:46:01.462717+0000","last_became_peered":"2024-02-13T00:46:01.462717+0000","last_unstale":"2024-02-13T00:46:48.459454+0000","last_undegraded":"2024-02-13T00:46:48.459454+0000","last_fullsized":"2024-02-13T00:46:48.459454+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.782345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.782345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":308,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462611+0000","last_change":"2024-02-13T00:46:01.457255+0000","last_active":"2024-02-13T00:46:48.462611+0000","last_peered":"2024-02-13T00:46:48.462611+0000","last_clean":"2024-02-13T00:46:48.462611+0000","last_became_active":"2024-02-13T00:46:01.456959+0000","last_became_peered":"2024-02-13T00:46:01.456959+0000","last_unstale":"2024-02-13T00:46:48.462611+0000","last_undegraded":"2024-02-13T00:46:48.462611+0000","last_fullsized":"2024-02-13T00:46:48.462611+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.281102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.281102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"263'10","reported_seq":387,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.320657+0000","last_change":"2024-02-13T00:43:38.251836+0000","last_active":"2024-02-13T00:46:49.320657+0000","last_peered":"2024-02-13T00:46:49.320657+0000","last_clean":"2024-02-13T00:46:49.320657+0000","last_became_active":"2024-02-13T00:43:35.454087+0000","last_became_peered":"2024-02-13T00:43:35.454087+0000","last_unstale":"2024-02-13T00:46:49.320657+0000","last_undegraded":"2024-02-13T00:46:49.320657+0000","last_fullsized":"2024-02-13T00:46:49.320657+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'9","last_scrub_stamp":"2024-02-13T00:41:44.698906+0000","last_deep_scrub":"149'9","last_deep_scrub_stamp":"2024-02-13T00:41:44.698906+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.698906+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":385,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458340+0000","last_change":"2024-02-13T00:46:08.351239+0000","last_active":"2024-02-13T00:46:48.458340+0000","last_peered":"2024-02-13T00:46:48.458340+0000","last_clean":"2024-02-13T00:46:48.458340+0000","last_became_active":"2024-02-13T00:46:01.449654+0000","last_became_peered":"2024-02-13T00:46:01.449654+0000","last_unstale":"2024-02-13T00:46:48.458340+0000","last_undegraded":"2024-02-13T00:46:48.458340+0000","last_fullsized":"2024-02-13T00:46:48.458340+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2024-02-13T00:43:00.636798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.636798+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460431+0000","last_change":"2024-02-13T00:45:15.446786+0000","last_active":"2024-02-13T00:46:48.460431+0000","last_peered":"2024-02-13T00:46:48.460431+0000","last_clean":"2024-02-13T00:46:48.460431+0000","last_became_active":"2024-02-13T00:45:15.446443+0000","last_became_peered":"2024-02-13T00:45:15.446443+0000","last_unstale":"2024-02-13T00:46:48.460431+0000","last_undegraded":"2024-02-13T00:46:48.460431+0000","last_fullsized":"2024-02-13T00:46:48.460431+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.557475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.557475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459326+0000","last_change":"2024-02-13T00:45:15.436286+0000","last_active":"2024-02-13T00:46:48.459326+0000","last_peered":"2024-02-13T00:46:48.459326+0000","last_clean":"2024-02-13T00:46:48.459326+0000","last_became_active":"2024-02-13T00:45:15.434559+0000","last_became_peered":"2024-02-13T00:45:15.434559+0000","last_unstale":"2024-02-13T00:46:48.459326+0000","last_undegraded":"2024-02-13T00:46:48.459326+0000","last_fullsized":"2024-02-13T00:46:48.459326+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":158,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462508+0000","last_change":"2024-02-13T00:46:01.445822+0000","last_active":"2024-02-13T00:46:48.462508+0000","last_peered":"2024-02-13T00:46:48.462508+0000","last_clean":"2024-02-13T00:46:48.462508+0000","last_became_active":"2024-02-13T00:46:01.444661+0000","last_became_peered":"2024-02-13T00:46:01.444661+0000","last_unstale":"2024-02-13T00:46:48.462508+0000","last_undegraded":"2024-02-13T00:46:48.462508+0000","last_fullsized":"2024-02-13T00:46:48.462508+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:43.049231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:43.049231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":199,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462889+0000","last_change":"2024-02-13T00:46:01.454510+0000","last_active":"2024-02-13T00:46:48.462889+0000","last_peered":"2024-02-13T00:46:48.462889+0000","last_clean":"2024-02-13T00:46:48.462889+0000","last_became_active":"2024-02-13T00:46:01.453842+0000","last_became_peered":"2024-02-13T00:46:01.453842+0000","last_unstale":"2024-02-13T00:46:48.462889+0000","last_undegraded":"2024-02-13T00:46:48.462889+0000","last_fullsized":"2024-02-13T00:46:48.462889+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:54.842056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:54.842056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":342,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462592+0000","last_change":"2024-02-13T00:46:04.744348+0000","last_active":"2024-02-13T00:46:48.462592+0000","last_peered":"2024-02-13T00:46:48.462592+0000","last_clean":"2024-02-13T00:46:48.462592+0000","last_became_active":"2024-02-13T00:46:01.445058+0000","last_became_peered":"2024-02-13T00:46:01.445058+0000","last_unstale":"2024-02-13T00:46:48.462592+0000","last_undegraded":"2024-02-13T00:46:48.462592+0000","last_fullsized":"2024-02-13T00:46:48.462592+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:42:53.881594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.881594+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":419,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466843+0000","last_change":"2024-02-13T00:44:39.503982+0000","last_active":"2024-02-13T00:46:48.466843+0000","last_peered":"2024-02-13T00:46:48.466843+0000","last_clean":"2024-02-13T00:46:48.466843+0000","last_became_active":"2024-02-13T00:44:39.503590+0000","last_became_peered":"2024-02-13T00:44:39.503590+0000","last_unstale":"2024-02-13T00:46:48.466843+0000","last_undegraded":"2024-02-13T00:46:48.466843+0000","last_fullsized":"2024-02-13T00:46:48.466843+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.236727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.236727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":369,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460325+0000","last_change":"2024-02-13T00:46:01.448313+0000","last_active":"2024-02-13T00:46:48.460325+0000","last_peered":"2024-02-13T00:46:48.460325+0000","last_clean":"2024-02-13T00:46:48.460325+0000","last_became_active":"2024-02-13T00:46:01.447815+0000","last_became_peered":"2024-02-13T00:46:01.447815+0000","last_unstale":"2024-02-13T00:46:48.460325+0000","last_undegraded":"2024-02-13T00:46:48.460325+0000","last_fullsized":"2024-02-13T00:46:48.460325+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":356,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458765+0000","last_change":"2024-02-13T00:44:50.436745+0000","last_active":"2024-02-13T00:46:48.458765+0000","last_peered":"2024-02-13T00:46:48.458765+0000","last_clean":"2024-02-13T00:46:48.458765+0000","last_became_active":"2024-02-13T00:44:50.436058+0000","last_became_peered":"2024-02-13T00:44:50.436058+0000","last_unstale":"2024-02-13T00:46:48.458765+0000","last_undegraded":"2024-02-13T00:46:48.458765+0000","last_fullsized":"2024-02-13T00:46:48.458765+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:48.300400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.300400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":130,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461722+0000","last_change":"2024-02-13T00:46:01.451625+0000","last_active":"2024-02-13T00:46:48.461722+0000","last_peered":"2024-02-13T00:46:48.461722+0000","last_clean":"2024-02-13T00:46:48.461722+0000","last_became_active":"2024-02-13T00:46:01.450587+0000","last_became_peered":"2024-02-13T00:46:01.450587+0000","last_unstale":"2024-02-13T00:46:48.461722+0000","last_undegraded":"2024-02-13T00:46:48.461722+0000","last_fullsized":"2024-02-13T00:46:48.461722+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:33.204969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:33.204969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":508,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463928+0000","last_change":"2024-02-13T00:46:04.826799+0000","last_active":"2024-02-13T00:46:48.463928+0000","last_peered":"2024-02-13T00:46:48.463928+0000","last_clean":"2024-02-13T00:46:48.463928+0000","last_became_active":"2024-02-13T00:46:01.443777+0000","last_became_peered":"2024-02-13T00:46:01.443777+0000","last_unstale":"2024-02-13T00:46:48.463928+0000","last_undegraded":"2024-02-13T00:46:48.463928+0000","last_fullsized":"2024-02-13T00:46:48.463928+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:42:20.688132+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":239,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.465295+0000","last_change":"2024-02-13T00:45:15.423607+0000","last_active":"2024-02-13T00:46:48.465295+0000","last_peered":"2024-02-13T00:46:48.465295+0000","last_clean":"2024-02-13T00:46:48.465295+0000","last_became_active":"2024-02-13T00:45:15.423415+0000","last_became_peered":"2024-02-13T00:45:15.423415+0000","last_unstale":"2024-02-13T00:46:48.465295+0000","last_undegraded":"2024-02-13T00:46:48.465295+0000","last_fullsized":"2024-02-13T00:46:48.465295+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:17.295459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:17.295459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":431,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459459+0000","last_change":"2024-02-13T00:45:15.445862+0000","last_active":"2024-02-13T00:46:48.459459+0000","last_peered":"2024-02-13T00:46:48.459459+0000","last_clean":"2024-02-13T00:46:48.459459+0000","last_became_active":"2024-02-13T00:45:15.445557+0000","last_became_peered":"2024-02-13T00:45:15.445557+0000","last_unstale":"2024-02-13T00:46:48.459459+0000","last_undegraded":"2024-02-13T00:46:48.459459+0000","last_fullsized":"2024-02-13T00:46:48.459459+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":44,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466381+0000","last_change":"2024-02-13T00:46:01.442629+0000","last_active":"2024-02-13T00:46:48.466381+0000","last_peered":"2024-02-13T00:46:48.466381+0000","last_clean":"2024-02-13T00:46:48.466381+0000","last_became_active":"2024-02-13T00:46:01.441421+0000","last_became_peered":"2024-02-13T00:46:01.441421+0000","last_unstale":"2024-02-13T00:46:48.466381+0000","last_undegraded":"2024-02-13T00:46:48.466381+0000","last_fullsized":"2024-02-13T00:46:48.466381+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:50.325063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:50.325063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":41,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.468152+0000","last_change":"2024-02-13T00:46:01.450756+0000","last_active":"2024-02-13T00:46:48.468152+0000","last_peered":"2024-02-13T00:46:48.468152+0000","last_clean":"2024-02-13T00:46:48.468152+0000","last_became_active":"2024-02-13T00:46:01.449142+0000","last_became_peered":"2024-02-13T00:46:01.449142+0000","last_unstale":"2024-02-13T00:46:48.468152+0000","last_undegraded":"2024-02-13T00:46:48.468152+0000","last_fullsized":"2024-02-13T00:46:48.468152+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:54.422484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:54.422484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":505,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466201+0000","last_change":"2024-02-13T00:46:34.691109+0000","last_active":"2024-02-13T00:46:48.466201+0000","last_peered":"2024-02-13T00:46:48.466201+0000","last_clean":"2024-02-13T00:46:48.466201+0000","last_became_active":"2024-02-13T00:46:01.448163+0000","last_became_peered":"2024-02-13T00:46:01.448163+0000","last_unstale":"2024-02-13T00:46:48.466201+0000","last_undegraded":"2024-02-13T00:46:48.466201+0000","last_fullsized":"2024-02-13T00:46:48.466201+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:46:34.690980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.690980+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":132,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459262+0000","last_change":"2024-02-13T00:46:37.056299+0000","last_active":"2024-02-13T00:46:48.459262+0000","last_peered":"2024-02-13T00:46:48.459262+0000","last_clean":"2024-02-13T00:46:48.459262+0000","last_became_active":"2024-02-13T00:46:01.446104+0000","last_became_peered":"2024-02-13T00:46:01.446104+0000","last_unstale":"2024-02-13T00:46:48.459262+0000","last_undegraded":"2024-02-13T00:46:48.459262+0000","last_fullsized":"2024-02-13T00:46:48.459262+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:37.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":415,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463294+0000","last_change":"2024-02-13T00:46:01.431899+0000","last_active":"2024-02-13T00:46:48.463294+0000","last_peered":"2024-02-13T00:46:48.463294+0000","last_clean":"2024-02-13T00:46:48.463294+0000","last_became_active":"2024-02-13T00:46:01.430623+0000","last_became_peered":"2024-02-13T00:46:01.430623+0000","last_unstale":"2024-02-13T00:46:48.463294+0000","last_undegraded":"2024-02-13T00:46:48.463294+0000","last_fullsized":"2024-02-13T00:46:48.463294+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:46.556229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.556229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":422,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461715+0000","last_change":"2024-02-13T00:46:01.450548+0000","last_active":"2024-02-13T00:46:48.461715+0000","last_peered":"2024-02-13T00:46:48.461715+0000","last_clean":"2024-02-13T00:46:48.461715+0000","last_became_active":"2024-02-13T00:46:01.450176+0000","last_became_peered":"2024-02-13T00:46:01.450176+0000","last_unstale":"2024-02-13T00:46:48.461715+0000","last_undegraded":"2024-02-13T00:46:48.461715+0000","last_fullsized":"2024-02-13T00:46:48.461715+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":276,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460095+0000","last_change":"2024-02-13T00:46:01.460470+0000","last_active":"2024-02-13T00:46:48.460095+0000","last_peered":"2024-02-13T00:46:48.460095+0000","last_clean":"2024-02-13T00:46:48.460095+0000","last_became_active":"2024-02-13T00:46:01.459859+0000","last_became_peered":"2024-02-13T00:46:01.459859+0000","last_unstale":"2024-02-13T00:46:48.460095+0000","last_undegraded":"2024-02-13T00:46:48.460095+0000","last_fullsized":"2024-02-13T00:46:48.460095+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:01.179031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:01.179031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":72,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.464359+0000","last_change":"2024-02-13T00:46:01.456508+0000","last_active":"2024-02-13T00:46:48.464359+0000","last_peered":"2024-02-13T00:46:48.464359+0000","last_clean":"2024-02-13T00:46:48.464359+0000","last_became_active":"2024-02-13T00:46:01.455170+0000","last_became_peered":"2024-02-13T00:46:01.455170+0000","last_unstale":"2024-02-13T00:46:48.464359+0000","last_undegraded":"2024-02-13T00:46:48.464359+0000","last_fullsized":"2024-02-13T00:46:48.464359+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:56.666003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.666003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"157'21","reported_seq":157,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460200+0000","last_change":"2024-02-13T00:45:18.327225+0000","last_active":"2024-02-13T00:46:48.460200+0000","last_peered":"2024-02-13T00:46:48.460200+0000","last_clean":"2024-02-13T00:46:48.460200+0000","last_became_active":"2024-02-13T00:45:15.446969+0000","last_became_peered":"2024-02-13T00:45:15.446969+0000","last_unstale":"2024-02-13T00:46:48.460200+0000","last_undegraded":"2024-02-13T00:46:48.460200+0000","last_fullsized":"2024-02-13T00:46:48.460200+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.415907+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":375,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462379+0000","last_change":"2024-02-13T00:45:15.434844+0000","last_active":"2024-02-13T00:46:48.462379+0000","last_peered":"2024-02-13T00:46:48.462379+0000","last_clean":"2024-02-13T00:46:48.462379+0000","last_became_active":"2024-02-13T00:45:15.434498+0000","last_became_peered":"2024-02-13T00:45:15.434498+0000","last_unstale":"2024-02-13T00:46:48.462379+0000","last_undegraded":"2024-02-13T00:46:48.462379+0000","last_fullsized":"2024-02-13T00:46:48.462379+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:23.561163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:42:23.561163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":373,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458620+0000","last_change":"2024-02-13T00:46:01.448309+0000","last_active":"2024-02-13T00:46:48.458620+0000","last_peered":"2024-02-13T00:46:48.458620+0000","last_clean":"2024-02-13T00:46:48.458620+0000","last_became_active":"2024-02-13T00:46:01.447823+0000","last_became_peered":"2024-02-13T00:46:01.447823+0000","last_unstale":"2024-02-13T00:46:48.458620+0000","last_undegraded":"2024-02-13T00:46:48.458620+0000","last_fullsized":"2024-02-13T00:46:48.458620+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.472312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.472312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":246,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460188+0000","last_change":"2024-02-13T00:46:01.460412+0000","last_active":"2024-02-13T00:46:48.460188+0000","last_peered":"2024-02-13T00:46:48.460188+0000","last_clean":"2024-02-13T00:46:48.460188+0000","last_became_active":"2024-02-13T00:46:01.460001+0000","last_became_peered":"2024-02-13T00:46:01.460001+0000","last_unstale":"2024-02-13T00:46:48.460188+0000","last_undegraded":"2024-02-13T00:46:48.460188+0000","last_fullsized":"2024-02-13T00:46:48.460188+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.544320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.544320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":270,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466930+0000","last_change":"2024-02-13T00:46:01.450879+0000","last_active":"2024-02-13T00:46:48.466930+0000","last_peered":"2024-02-13T00:46:48.466930+0000","last_clean":"2024-02-13T00:46:48.466930+0000","last_became_active":"2024-02-13T00:46:01.450088+0000","last_became_peered":"2024-02-13T00:46:01.450088+0000","last_unstale":"2024-02-13T00:46:48.466930+0000","last_undegraded":"2024-02-13T00:46:48.466930+0000","last_fullsized":"2024-02-13T00:46:48.466930+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":356,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467573+0000","last_change":"2024-02-13T00:46:01.459317+0000","last_active":"2024-02-13T00:46:48.467573+0000","last_peered":"2024-02-13T00:46:48.467573+0000","last_clean":"2024-02-13T00:46:48.467573+0000","last_became_active":"2024-02-13T00:46:01.458408+0000","last_became_peered":"2024-02-13T00:46:01.458408+0000","last_unstale":"2024-02-13T00:46:48.467573+0000","last_undegraded":"2024-02-13T00:46:48.467573+0000","last_fullsized":"2024-02-13T00:46:48.467573+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:16.256516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:16.256516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":78,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466131+0000","last_change":"2024-02-13T00:46:01.442001+0000","last_active":"2024-02-13T00:46:48.466131+0000","last_peered":"2024-02-13T00:46:48.466131+0000","last_clean":"2024-02-13T00:46:48.466131+0000","last_became_active":"2024-02-13T00:46:01.441013+0000","last_became_peered":"2024-02-13T00:46:01.441013+0000","last_unstale":"2024-02-13T00:46:48.466131+0000","last_undegraded":"2024-02-13T00:46:48.466131+0000","last_fullsized":"2024-02-13T00:46:48.466131+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:26.313580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:26.313580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":455,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467390+0000","last_change":"2024-02-13T00:46:04.714305+0000","last_active":"2024-02-13T00:46:48.467390+0000","last_peered":"2024-02-13T00:46:48.467390+0000","last_clean":"2024-02-13T00:46:48.467390+0000","last_became_active":"2024-02-13T00:46:01.448219+0000","last_became_peered":"2024-02-13T00:46:01.448219+0000","last_unstale":"2024-02-13T00:46:48.467390+0000","last_undegraded":"2024-02-13T00:46:48.467390+0000","last_fullsized":"2024-02-13T00:46:48.467390+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:42:54.174950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:54.174950+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":330,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463216+0000","last_change":"2024-02-13T00:44:50.426201+0000","last_active":"2024-02-13T00:46:48.463216+0000","last_peered":"2024-02-13T00:46:48.463216+0000","last_clean":"2024-02-13T00:46:48.463216+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:46:48.463216+0000","last_undegraded":"2024-02-13T00:46:48.463216+0000","last_fullsized":"2024-02-13T00:46:48.463216+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:20.481083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:44:20.481083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":981,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.459042+0000","last_change":"2024-02-13T00:45:54.075270+0000","last_active":"2024-02-13T00:46:49.459042+0000","last_peered":"2024-02-13T00:46:49.459042+0000","last_clean":"2024-02-13T00:46:49.459042+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:46:49.459042+0000","last_undegraded":"2024-02-13T00:46:49.459042+0000","last_fullsized":"2024-02-13T00:46:49.459042+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:45:54.075194+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":536,"num_read_kb":531,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":143,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459999+0000","last_change":"2024-02-13T00:46:01.460282+0000","last_active":"2024-02-13T00:46:48.459999+0000","last_peered":"2024-02-13T00:46:48.459999+0000","last_clean":"2024-02-13T00:46:48.459999+0000","last_became_active":"2024-02-13T00:46:01.459473+0000","last_became_peered":"2024-02-13T00:46:01.459473+0000","last_unstale":"2024-02-13T00:46:48.459999+0000","last_undegraded":"2024-02-13T00:46:48.459999+0000","last_fullsized":"2024-02-13T00:46:48.459999+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":195,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461227+0000","last_change":"2024-02-13T00:45:37.135398+0000","last_active":"2024-02-13T00:46:48.461227+0000","last_peered":"2024-02-13T00:46:48.461227+0000","last_clean":"2024-02-13T00:46:48.461227+0000","last_became_active":"2024-02-13T00:45:15.435588+0000","last_became_peered":"2024-02-13T00:45:15.435588+0000","last_unstale":"2024-02-13T00:46:48.461227+0000","last_undegraded":"2024-02-13T00:46:48.461227+0000","last_fullsized":"2024-02-13T00:46:48.461227+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.135332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.135332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":94,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458851+0000","last_change":"2024-02-13T00:45:31.183763+0000","last_active":"2024-02-13T00:46:48.458851+0000","last_peered":"2024-02-13T00:46:48.458851+0000","last_clean":"2024-02-13T00:46:48.458851+0000","last_became_active":"2024-02-13T00:45:06.421025+0000","last_became_peered":"2024-02-13T00:45:06.421025+0000","last_unstale":"2024-02-13T00:46:48.458851+0000","last_undegraded":"2024-02-13T00:46:48.458851+0000","last_fullsized":"2024-02-13T00:46:48.458851+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.183682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.183682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"405'53","reported_seq":73,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.320403+0000","last_change":"2024-02-13T00:46:04.950162+0000","last_active":"2024-02-13T00:46:49.320403+0000","last_peered":"2024-02-13T00:46:49.320403+0000","last_clean":"2024-02-13T00:46:49.320403+0000","last_became_active":"2024-02-13T00:46:01.449059+0000","last_became_peered":"2024-02-13T00:46:01.449059+0000","last_unstale":"2024-02-13T00:46:49.320403+0000","last_undegraded":"2024-02-13T00:46:49.320403+0000","last_fullsized":"2024-02-13T00:46:49.320403+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'48","last_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:43:48.212428+0000","log_size":53,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":2148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":340,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.465868+0000","last_change":"2024-02-13T00:46:01.453042+0000","last_active":"2024-02-13T00:46:48.465868+0000","last_peered":"2024-02-13T00:46:48.465868+0000","last_clean":"2024-02-13T00:46:48.465868+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:46:48.465868+0000","last_undegraded":"2024-02-13T00:46:48.465868+0000","last_fullsized":"2024-02-13T00:46:48.465868+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.045912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.045912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460473+0000","last_change":"2024-02-13T00:46:01.461717+0000","last_active":"2024-02-13T00:46:48.460473+0000","last_peered":"2024-02-13T00:46:48.460473+0000","last_clean":"2024-02-13T00:46:48.460473+0000","last_became_active":"2024-02-13T00:46:01.460410+0000","last_became_peered":"2024-02-13T00:46:01.460410+0000","last_unstale":"2024-02-13T00:46:48.460473+0000","last_undegraded":"2024-02-13T00:46:48.460473+0000","last_fullsized":"2024-02-13T00:46:48.460473+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":125,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461805+0000","last_change":"2024-02-13T00:44:50.439369+0000","last_active":"2024-02-13T00:46:48.461805+0000","last_peered":"2024-02-13T00:46:48.461805+0000","last_clean":"2024-02-13T00:46:48.461805+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:46:48.461805+0000","last_undegraded":"2024-02-13T00:46:48.461805+0000","last_fullsized":"2024-02-13T00:46:48.461805+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:19.407011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:19.407011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":560,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.320351+0000","last_change":"2024-02-13T00:46:11.962872+0000","last_active":"2024-02-13T00:46:49.320351+0000","last_peered":"2024-02-13T00:46:49.320351+0000","last_clean":"2024-02-13T00:46:49.320351+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:46:49.320351+0000","last_undegraded":"2024-02-13T00:46:49.320351+0000","last_fullsized":"2024-02-13T00:46:49.320351+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'11","last_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:43:02.423358+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":143,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466152+0000","last_change":"2024-02-13T00:46:02.664116+0000","last_active":"2024-02-13T00:46:48.466152+0000","last_peered":"2024-02-13T00:46:48.466152+0000","last_clean":"2024-02-13T00:46:48.466152+0000","last_became_active":"2024-02-13T00:46:01.442003+0000","last_became_peered":"2024-02-13T00:46:01.442003+0000","last_unstale":"2024-02-13T00:46:48.466152+0000","last_undegraded":"2024-02-13T00:46:48.466152+0000","last_fullsized":"2024-02-13T00:46:48.466152+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:52.517392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.517392+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":406,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461307+0000","last_change":"2024-02-13T00:45:15.423417+0000","last_active":"2024-02-13T00:46:48.461307+0000","last_peered":"2024-02-13T00:46:48.461307+0000","last_clean":"2024-02-13T00:46:48.461307+0000","last_became_active":"2024-02-13T00:45:15.422546+0000","last_became_peered":"2024-02-13T00:45:15.422546+0000","last_unstale":"2024-02-13T00:46:48.461307+0000","last_undegraded":"2024-02-13T00:46:48.461307+0000","last_fullsized":"2024-02-13T00:46:48.461307+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:41.223028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:43:41.223028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":422,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462306+0000","last_change":"2024-02-13T00:46:01.436997+0000","last_active":"2024-02-13T00:46:48.462306+0000","last_peered":"2024-02-13T00:46:48.462306+0000","last_clean":"2024-02-13T00:46:48.462306+0000","last_became_active":"2024-02-13T00:46:01.435178+0000","last_became_peered":"2024-02-13T00:46:01.435178+0000","last_unstale":"2024-02-13T00:46:48.462306+0000","last_undegraded":"2024-02-13T00:46:48.462306+0000","last_fullsized":"2024-02-13T00:46:48.462306+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":355,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459228+0000","last_change":"2024-02-13T00:45:15.445626+0000","last_active":"2024-02-13T00:46:48.459228+0000","last_peered":"2024-02-13T00:46:48.459228+0000","last_clean":"2024-02-13T00:46:48.459228+0000","last_became_active":"2024-02-13T00:45:15.445117+0000","last_became_peered":"2024-02-13T00:45:15.445117+0000","last_unstale":"2024-02-13T00:46:48.459228+0000","last_undegraded":"2024-02-13T00:46:48.459228+0000","last_fullsized":"2024-02-13T00:46:48.459228+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.171185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":122,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463033+0000","last_change":"2024-02-13T00:46:01.436795+0000","last_active":"2024-02-13T00:46:48.463033+0000","last_peered":"2024-02-13T00:46:48.463033+0000","last_clean":"2024-02-13T00:46:48.463033+0000","last_became_active":"2024-02-13T00:46:01.434431+0000","last_became_peered":"2024-02-13T00:46:01.434431+0000","last_unstale":"2024-02-13T00:46:48.463033+0000","last_undegraded":"2024-02-13T00:46:48.463033+0000","last_fullsized":"2024-02-13T00:46:48.463033+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.592320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.592320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":467,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459148+0000","last_change":"2024-02-13T00:44:41.502069+0000","last_active":"2024-02-13T00:46:48.459148+0000","last_peered":"2024-02-13T00:46:48.459148+0000","last_clean":"2024-02-13T00:46:48.459148+0000","last_became_active":"2024-02-13T00:44:39.501886+0000","last_became_peered":"2024-02-13T00:44:39.501886+0000","last_unstale":"2024-02-13T00:46:48.459148+0000","last_undegraded":"2024-02-13T00:46:48.459148+0000","last_fullsized":"2024-02-13T00:46:48.459148+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'24","last_scrub_stamp":"2024-02-13T00:43:30.413372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.413372+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":36,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":396,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466500+0000","last_change":"2024-02-13T00:46:14.266858+0000","last_active":"2024-02-13T00:46:48.466500+0000","last_peered":"2024-02-13T00:46:48.466500+0000","last_clean":"2024-02-13T00:46:48.466500+0000","last_became_active":"2024-02-13T00:46:01.457926+0000","last_became_peered":"2024-02-13T00:46:01.457926+0000","last_unstale":"2024-02-13T00:46:48.466500+0000","last_undegraded":"2024-02-13T00:46:48.466500+0000","last_fullsized":"2024-02-13T00:46:48.466500+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:43:35.309308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:35.309308+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":201,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463388+0000","last_change":"2024-02-13T00:45:15.435701+0000","last_active":"2024-02-13T00:46:48.463388+0000","last_peered":"2024-02-13T00:46:48.463388+0000","last_clean":"2024-02-13T00:46:48.463388+0000","last_became_active":"2024-02-13T00:45:15.435404+0000","last_became_peered":"2024-02-13T00:45:15.435404+0000","last_unstale":"2024-02-13T00:46:48.463388+0000","last_undegraded":"2024-02-13T00:46:48.463388+0000","last_fullsized":"2024-02-13T00:46:48.463388+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:41:45.660653+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":356,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467056+0000","last_change":"2024-02-13T00:46:01.450436+0000","last_active":"2024-02-13T00:46:48.467056+0000","last_peered":"2024-02-13T00:46:48.467056+0000","last_clean":"2024-02-13T00:46:48.467056+0000","last_became_active":"2024-02-13T00:46:01.449234+0000","last_became_peered":"2024-02-13T00:46:01.449234+0000","last_unstale":"2024-02-13T00:46:48.467056+0000","last_undegraded":"2024-02-13T00:46:48.467056+0000","last_fullsized":"2024-02-13T00:46:48.467056+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.110742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.110742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":533,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466290+0000","last_change":"2024-02-13T00:46:12.563543+0000","last_active":"2024-02-13T00:46:48.466290+0000","last_peered":"2024-02-13T00:46:48.466290+0000","last_clean":"2024-02-13T00:46:48.466290+0000","last_became_active":"2024-02-13T00:46:01.456876+0000","last_became_peered":"2024-02-13T00:46:01.456876+0000","last_unstale":"2024-02-13T00:46:48.466290+0000","last_undegraded":"2024-02-13T00:46:48.466290+0000","last_fullsized":"2024-02-13T00:46:48.466290+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:43:30.365536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.365536+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":113,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459216+0000","last_change":"2024-02-13T00:46:30.095387+0000","last_active":"2024-02-13T00:46:48.459216+0000","last_peered":"2024-02-13T00:46:48.459216+0000","last_clean":"2024-02-13T00:46:48.459216+0000","last_became_active":"2024-02-13T00:44:50.441300+0000","last_became_peered":"2024-02-13T00:44:50.441300+0000","last_unstale":"2024-02-13T00:46:48.459216+0000","last_undegraded":"2024-02-13T00:46:48.459216+0000","last_fullsized":"2024-02-13T00:46:48.459216+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.095307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":192,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467300+0000","last_change":"2024-02-13T00:45:15.416946+0000","last_active":"2024-02-13T00:46:48.467300+0000","last_peered":"2024-02-13T00:46:48.467300+0000","last_clean":"2024-02-13T00:46:48.467300+0000","last_became_active":"2024-02-13T00:45:15.416364+0000","last_became_peered":"2024-02-13T00:45:15.416364+0000","last_unstale":"2024-02-13T00:46:48.467300+0000","last_undegraded":"2024-02-13T00:46:48.467300+0000","last_fullsized":"2024-02-13T00:46:48.467300+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.458752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.458752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":129,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.464423+0000","last_change":"2024-02-13T00:46:01.454597+0000","last_active":"2024-02-13T00:46:48.464423+0000","last_peered":"2024-02-13T00:46:48.464423+0000","last_clean":"2024-02-13T00:46:48.464423+0000","last_became_active":"2024-02-13T00:46:01.454101+0000","last_became_peered":"2024-02-13T00:46:01.454101+0000","last_unstale":"2024-02-13T00:46:48.464423+0000","last_undegraded":"2024-02-13T00:46:48.464423+0000","last_fullsized":"2024-02-13T00:46:48.464423+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.651091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.651091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":125,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460312+0000","last_change":"2024-02-13T00:44:50.441196+0000","last_active":"2024-02-13T00:46:48.460312+0000","last_peered":"2024-02-13T00:46:48.460312+0000","last_clean":"2024-02-13T00:46:48.460312+0000","last_became_active":"2024-02-13T00:44:50.438274+0000","last_became_peered":"2024-02-13T00:44:50.438274+0000","last_unstale":"2024-02-13T00:46:48.460312+0000","last_undegraded":"2024-02-13T00:46:48.460312+0000","last_fullsized":"2024-02-13T00:46:48.460312+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.510219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.510219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":273,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467631+0000","last_change":"2024-02-13T00:46:34.141015+0000","last_active":"2024-02-13T00:46:48.467631+0000","last_peered":"2024-02-13T00:46:48.467631+0000","last_clean":"2024-02-13T00:46:48.467631+0000","last_became_active":"2024-02-13T00:45:15.436020+0000","last_became_peered":"2024-02-13T00:45:15.436020+0000","last_unstale":"2024-02-13T00:46:48.467631+0000","last_undegraded":"2024-02-13T00:46:48.467631+0000","last_fullsized":"2024-02-13T00:46:48.467631+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:46:34.140928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.140928+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":121,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459953+0000","last_change":"2024-02-13T00:45:00.311635+0000","last_active":"2024-02-13T00:46:48.459953+0000","last_peered":"2024-02-13T00:46:48.459953+0000","last_clean":"2024-02-13T00:46:48.459953+0000","last_became_active":"2024-02-13T00:44:39.512402+0000","last_became_peered":"2024-02-13T00:44:39.512402+0000","last_unstale":"2024-02-13T00:46:48.459953+0000","last_undegraded":"2024-02-13T00:46:48.459953+0000","last_fullsized":"2024-02-13T00:46:48.459953+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:00.311558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:45:00.311558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":336,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467062+0000","last_change":"2024-02-13T00:46:01.455806+0000","last_active":"2024-02-13T00:46:48.467062+0000","last_peered":"2024-02-13T00:46:48.467062+0000","last_clean":"2024-02-13T00:46:48.467062+0000","last_became_active":"2024-02-13T00:46:01.453265+0000","last_became_peered":"2024-02-13T00:46:01.453265+0000","last_unstale":"2024-02-13T00:46:48.467062+0000","last_undegraded":"2024-02-13T00:46:48.467062+0000","last_fullsized":"2024-02-13T00:46:48.467062+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.106639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.106639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460017+0000","last_change":"2024-02-13T00:46:01.451840+0000","last_active":"2024-02-13T00:46:48.460017+0000","last_peered":"2024-02-13T00:46:48.460017+0000","last_clean":"2024-02-13T00:46:48.460017+0000","last_became_active":"2024-02-13T00:46:01.450312+0000","last_became_peered":"2024-02-13T00:46:01.450312+0000","last_unstale":"2024-02-13T00:46:48.460017+0000","last_undegraded":"2024-02-13T00:46:48.460017+0000","last_fullsized":"2024-02-13T00:46:48.460017+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.042291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.042291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":206,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.464719+0000","last_change":"2024-02-13T00:46:01.454858+0000","last_active":"2024-02-13T00:46:48.464719+0000","last_peered":"2024-02-13T00:46:48.464719+0000","last_clean":"2024-02-13T00:46:48.464719+0000","last_became_active":"2024-02-13T00:46:01.454284+0000","last_became_peered":"2024-02-13T00:46:01.454284+0000","last_unstale":"2024-02-13T00:46:48.464719+0000","last_undegraded":"2024-02-13T00:46:48.464719+0000","last_fullsized":"2024-02-13T00:46:48.464719+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:34.523594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.523594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":345,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461546+0000","last_change":"2024-02-13T00:46:04.434215+0000","last_active":"2024-02-13T00:46:48.461546+0000","last_peered":"2024-02-13T00:46:48.461546+0000","last_clean":"2024-02-13T00:46:48.461546+0000","last_became_active":"2024-02-13T00:46:01.450113+0000","last_became_peered":"2024-02-13T00:46:01.450113+0000","last_unstale":"2024-02-13T00:46:48.461546+0000","last_undegraded":"2024-02-13T00:46:48.461546+0000","last_fullsized":"2024-02-13T00:46:48.461546+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:42:48.774168+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.774168+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":376,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463100+0000","last_change":"2024-02-13T00:46:01.454689+0000","last_active":"2024-02-13T00:46:48.463100+0000","last_peered":"2024-02-13T00:46:48.463100+0000","last_clean":"2024-02-13T00:46:48.463100+0000","last_became_active":"2024-02-13T00:46:01.454310+0000","last_became_peered":"2024-02-13T00:46:01.454310+0000","last_unstale":"2024-02-13T00:46:48.463100+0000","last_undegraded":"2024-02-13T00:46:48.463100+0000","last_fullsized":"2024-02-13T00:46:48.463100+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.230330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.230330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"404'9","reported_seq":74,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.320318+0000","last_change":"2024-02-13T00:46:04.816167+0000","last_active":"2024-02-13T00:46:49.320318+0000","last_peered":"2024-02-13T00:46:49.320318+0000","last_clean":"2024-02-13T00:46:49.320318+0000","last_became_active":"2024-02-13T00:46:01.448144+0000","last_became_peered":"2024-02-13T00:46:01.448144+0000","last_unstale":"2024-02-13T00:46:49.320318+0000","last_undegraded":"2024-02-13T00:46:49.320318+0000","last_fullsized":"2024-02-13T00:46:49.320318+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:37.452882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:37.452882+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":413,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466073+0000","last_change":"2024-02-13T00:46:01.449886+0000","last_active":"2024-02-13T00:46:48.466073+0000","last_peered":"2024-02-13T00:46:48.466073+0000","last_clean":"2024-02-13T00:46:48.466073+0000","last_became_active":"2024-02-13T00:46:01.447236+0000","last_became_peered":"2024-02-13T00:46:01.447236+0000","last_unstale":"2024-02-13T00:46:48.466073+0000","last_undegraded":"2024-02-13T00:46:48.466073+0000","last_fullsized":"2024-02-13T00:46:48.466073+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:52.391906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.391906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":84,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.464329+0000","last_change":"2024-02-13T00:46:01.456618+0000","last_active":"2024-02-13T00:46:48.464329+0000","last_peered":"2024-02-13T00:46:48.464329+0000","last_clean":"2024-02-13T00:46:48.464329+0000","last_became_active":"2024-02-13T00:46:01.455989+0000","last_became_peered":"2024-02-13T00:46:01.455989+0000","last_unstale":"2024-02-13T00:46:48.464329+0000","last_undegraded":"2024-02-13T00:46:48.464329+0000","last_fullsized":"2024-02-13T00:46:48.464329+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:16.499780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:16.499780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":389,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459190+0000","last_change":"2024-02-13T00:46:10.157189+0000","last_active":"2024-02-13T00:46:48.459190+0000","last_peered":"2024-02-13T00:46:48.459190+0000","last_clean":"2024-02-13T00:46:48.459190+0000","last_became_active":"2024-02-13T00:46:01.460321+0000","last_became_peered":"2024-02-13T00:46:01.460321+0000","last_unstale":"2024-02-13T00:46:48.459190+0000","last_undegraded":"2024-02-13T00:46:48.459190+0000","last_fullsized":"2024-02-13T00:46:48.459190+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:36.477034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.477034+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":428,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.464026+0000","last_change":"2024-02-13T00:46:01.432179+0000","last_active":"2024-02-13T00:46:48.464026+0000","last_peered":"2024-02-13T00:46:48.464026+0000","last_clean":"2024-02-13T00:46:48.464026+0000","last_became_active":"2024-02-13T00:46:01.431926+0000","last_became_peered":"2024-02-13T00:46:01.431926+0000","last_unstale":"2024-02-13T00:46:48.464026+0000","last_undegraded":"2024-02-13T00:46:48.464026+0000","last_fullsized":"2024-02-13T00:46:48.464026+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:40.335892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:40.335892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"363'14","reported_seq":217,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.320579+0000","last_change":"2024-02-13T00:45:17.693600+0000","last_active":"2024-02-13T00:46:49.320579+0000","last_peered":"2024-02-13T00:46:49.320579+0000","last_clean":"2024-02-13T00:46:49.320579+0000","last_became_active":"2024-02-13T00:45:15.423066+0000","last_became_peered":"2024-02-13T00:45:15.423066+0000","last_unstale":"2024-02-13T00:46:49.320579+0000","last_undegraded":"2024-02-13T00:46:49.320579+0000","last_fullsized":"2024-02-13T00:46:49.320579+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-02-13T00:41:32.715550+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:41:32.715550+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":47,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466051+0000","last_change":"2024-02-13T00:46:01.447742+0000","last_active":"2024-02-13T00:46:48.466051+0000","last_peered":"2024-02-13T00:46:48.466051+0000","last_clean":"2024-02-13T00:46:48.466051+0000","last_became_active":"2024-02-13T00:46:01.446960+0000","last_became_peered":"2024-02-13T00:46:01.446960+0000","last_unstale":"2024-02-13T00:46:48.466051+0000","last_undegraded":"2024-02-13T00:46:48.466051+0000","last_fullsized":"2024-02-13T00:46:48.466051+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.421001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.421001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":290,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461480+0000","last_change":"2024-02-13T00:46:01.450308+0000","last_active":"2024-02-13T00:46:48.461480+0000","last_peered":"2024-02-13T00:46:48.461480+0000","last_clean":"2024-02-13T00:46:48.461480+0000","last_became_active":"2024-02-13T00:46:01.449694+0000","last_became_peered":"2024-02-13T00:46:01.449694+0000","last_unstale":"2024-02-13T00:46:48.461480+0000","last_undegraded":"2024-02-13T00:46:48.461480+0000","last_fullsized":"2024-02-13T00:46:48.461480+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:27.073034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:27.073034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":472,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466061+0000","last_change":"2024-02-13T00:44:54.926465+0000","last_active":"2024-02-13T00:46:48.466061+0000","last_peered":"2024-02-13T00:46:48.466061+0000","last_clean":"2024-02-13T00:46:48.466061+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:46:48.466061+0000","last_undegraded":"2024-02-13T00:46:48.466061+0000","last_fullsized":"2024-02-13T00:46:48.466061+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:43:40.244930+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:43:40.244930+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459482+0000","last_change":"2024-02-13T00:46:01.463310+0000","last_active":"2024-02-13T00:46:48.459482+0000","last_peered":"2024-02-13T00:46:48.459482+0000","last_clean":"2024-02-13T00:46:48.459482+0000","last_became_active":"2024-02-13T00:46:01.462898+0000","last_became_peered":"2024-02-13T00:46:01.462898+0000","last_unstale":"2024-02-13T00:46:48.459482+0000","last_undegraded":"2024-02-13T00:46:48.459482+0000","last_fullsized":"2024-02-13T00:46:48.459482+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.095888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":197,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.464172+0000","last_change":"2024-02-13T00:46:32.638418+0000","last_active":"2024-02-13T00:46:48.464172+0000","last_peered":"2024-02-13T00:46:48.464172+0000","last_clean":"2024-02-13T00:46:48.464172+0000","last_became_active":"2024-02-13T00:46:01.443279+0000","last_became_peered":"2024-02-13T00:46:01.443279+0000","last_unstale":"2024-02-13T00:46:48.464172+0000","last_undegraded":"2024-02-13T00:46:48.464172+0000","last_fullsized":"2024-02-13T00:46:48.464172+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.638349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":395,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462232+0000","last_change":"2024-02-13T00:46:01.446071+0000","last_active":"2024-02-13T00:46:48.462232+0000","last_peered":"2024-02-13T00:46:48.462232+0000","last_clean":"2024-02-13T00:46:48.462232+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:46:48.462232+0000","last_undegraded":"2024-02-13T00:46:48.462232+0000","last_fullsized":"2024-02-13T00:46:48.462232+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.413326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":130,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460202+0000","last_change":"2024-02-13T00:45:40.233264+0000","last_active":"2024-02-13T00:46:48.460202+0000","last_peered":"2024-02-13T00:46:48.460202+0000","last_clean":"2024-02-13T00:46:48.460202+0000","last_became_active":"2024-02-13T00:45:15.447051+0000","last_became_peered":"2024-02-13T00:45:15.447051+0000","last_unstale":"2024-02-13T00:46:48.460202+0000","last_undegraded":"2024-02-13T00:46:48.460202+0000","last_fullsized":"2024-02-13T00:46:48.460202+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:40.233185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:40.233185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":149,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459894+0000","last_change":"2024-02-13T00:44:52.751765+0000","last_active":"2024-02-13T00:46:48.459894+0000","last_peered":"2024-02-13T00:46:48.459894+0000","last_clean":"2024-02-13T00:46:48.459894+0000","last_became_active":"2024-02-13T00:44:50.435203+0000","last_became_peered":"2024-02-13T00:44:50.435203+0000","last_unstale":"2024-02-13T00:46:48.459894+0000","last_undegraded":"2024-02-13T00:46:48.459894+0000","last_fullsized":"2024-02-13T00:46:48.459894+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:42:51.361489+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:42:51.361489+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":406,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461724+0000","last_change":"2024-02-13T00:46:01.457220+0000","last_active":"2024-02-13T00:46:48.461724+0000","last_peered":"2024-02-13T00:46:48.461724+0000","last_clean":"2024-02-13T00:46:48.461724+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:46:48.461724+0000","last_undegraded":"2024-02-13T00:46:48.461724+0000","last_fullsized":"2024-02-13T00:46:48.461724+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.097481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":420,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466085+0000","last_change":"2024-02-13T00:46:01.453391+0000","last_active":"2024-02-13T00:46:48.466085+0000","last_peered":"2024-02-13T00:46:48.466085+0000","last_clean":"2024-02-13T00:46:48.466085+0000","last_became_active":"2024-02-13T00:46:01.452730+0000","last_became_peered":"2024-02-13T00:46:01.452730+0000","last_unstale":"2024-02-13T00:46:48.466085+0000","last_undegraded":"2024-02-13T00:46:48.466085+0000","last_fullsized":"2024-02-13T00:46:48.466085+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:26.218807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:44:26.218807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":201,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463388+0000","last_change":"2024-02-13T00:46:01.454624+0000","last_active":"2024-02-13T00:46:48.463388+0000","last_peered":"2024-02-13T00:46:48.463388+0000","last_clean":"2024-02-13T00:46:48.463388+0000","last_became_active":"2024-02-13T00:46:01.454287+0000","last_became_peered":"2024-02-13T00:46:01.454287+0000","last_unstale":"2024-02-13T00:46:48.463388+0000","last_undegraded":"2024-02-13T00:46:48.463388+0000","last_fullsized":"2024-02-13T00:46:48.463388+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:50.828264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:50.828264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462478+0000","last_change":"2024-02-13T00:46:01.449548+0000","last_active":"2024-02-13T00:46:48.462478+0000","last_peered":"2024-02-13T00:46:48.462478+0000","last_clean":"2024-02-13T00:46:48.462478+0000","last_became_active":"2024-02-13T00:46:01.448671+0000","last_became_peered":"2024-02-13T00:46:01.448671+0000","last_unstale":"2024-02-13T00:46:48.462478+0000","last_undegraded":"2024-02-13T00:46:48.462478+0000","last_fullsized":"2024-02-13T00:46:48.462478+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.820327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.820327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":227,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459401+0000","last_change":"2024-02-13T00:44:40.371566+0000","last_active":"2024-02-13T00:46:48.459401+0000","last_peered":"2024-02-13T00:46:48.459401+0000","last_clean":"2024-02-13T00:46:48.459401+0000","last_became_active":"2024-02-13T00:44:39.501040+0000","last_became_peered":"2024-02-13T00:44:39.501040+0000","last_unstale":"2024-02-13T00:46:48.459401+0000","last_undegraded":"2024-02-13T00:46:48.459401+0000","last_fullsized":"2024-02-13T00:46:48.459401+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:44:32.180113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:44:32.180113+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":360,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458874+0000","last_change":"2024-02-13T00:46:01.464539+0000","last_active":"2024-02-13T00:46:48.458874+0000","last_peered":"2024-02-13T00:46:48.458874+0000","last_clean":"2024-02-13T00:46:48.458874+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:46:48.458874+0000","last_undegraded":"2024-02-13T00:46:48.458874+0000","last_fullsized":"2024-02-13T00:46:48.458874+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.499576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":125,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458443+0000","last_change":"2024-02-13T00:45:15.436871+0000","last_active":"2024-02-13T00:46:48.458443+0000","last_peered":"2024-02-13T00:46:48.458443+0000","last_clean":"2024-02-13T00:46:48.458443+0000","last_became_active":"2024-02-13T00:45:15.435820+0000","last_became_peered":"2024-02-13T00:45:15.435820+0000","last_unstale":"2024-02-13T00:46:48.458443+0000","last_undegraded":"2024-02-13T00:46:48.458443+0000","last_fullsized":"2024-02-13T00:46:48.458443+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.646321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.646321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":115,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458948+0000","last_change":"2024-02-13T00:45:15.445131+0000","last_active":"2024-02-13T00:46:48.458948+0000","last_peered":"2024-02-13T00:46:48.458948+0000","last_clean":"2024-02-13T00:46:48.458948+0000","last_became_active":"2024-02-13T00:45:15.444818+0000","last_became_peered":"2024-02-13T00:45:15.444818+0000","last_unstale":"2024-02-13T00:46:48.458948+0000","last_undegraded":"2024-02-13T00:46:48.458948+0000","last_fullsized":"2024-02-13T00:46:48.458948+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.479450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":281,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459141+0000","last_change":"2024-02-13T00:46:01.439915+0000","last_active":"2024-02-13T00:46:48.459141+0000","last_peered":"2024-02-13T00:46:48.459141+0000","last_clean":"2024-02-13T00:46:48.459141+0000","last_became_active":"2024-02-13T00:46:01.438745+0000","last_became_peered":"2024-02-13T00:46:01.438745+0000","last_unstale":"2024-02-13T00:46:48.459141+0000","last_undegraded":"2024-02-13T00:46:48.459141+0000","last_fullsized":"2024-02-13T00:46:48.459141+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.104906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.104906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":176,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459407+0000","last_change":"2024-02-13T00:46:07.632448+0000","last_active":"2024-02-13T00:46:48.459407+0000","last_peered":"2024-02-13T00:46:48.459407+0000","last_clean":"2024-02-13T00:46:48.459407+0000","last_became_active":"2024-02-13T00:46:01.450079+0000","last_became_peered":"2024-02-13T00:46:01.450079+0000","last_unstale":"2024-02-13T00:46:48.459407+0000","last_undegraded":"2024-02-13T00:46:48.459407+0000","last_fullsized":"2024-02-13T00:46:48.459407+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:41:50.675154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:50.675154+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460191+0000","last_change":"2024-02-13T00:46:01.460250+0000","last_active":"2024-02-13T00:46:48.460191+0000","last_peered":"2024-02-13T00:46:48.460191+0000","last_clean":"2024-02-13T00:46:48.460191+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:46:48.460191+0000","last_undegraded":"2024-02-13T00:46:48.460191+0000","last_fullsized":"2024-02-13T00:46:48.460191+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:44.457672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:44.457672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":125,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462240+0000","last_change":"2024-02-13T00:45:15.439752+0000","last_active":"2024-02-13T00:46:48.462240+0000","last_peered":"2024-02-13T00:46:48.462240+0000","last_clean":"2024-02-13T00:46:48.462240+0000","last_became_active":"2024-02-13T00:45:15.437943+0000","last_became_peered":"2024-02-13T00:45:15.437943+0000","last_unstale":"2024-02-13T00:46:48.462240+0000","last_undegraded":"2024-02-13T00:46:48.462240+0000","last_fullsized":"2024-02-13T00:46:48.462240+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.284740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.284740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":333,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463606+0000","last_change":"2024-02-13T00:44:58.969806+0000","last_active":"2024-02-13T00:46:48.463606+0000","last_peered":"2024-02-13T00:46:48.463606+0000","last_clean":"2024-02-13T00:46:48.463606+0000","last_became_active":"2024-02-13T00:44:50.439512+0000","last_became_peered":"2024-02-13T00:44:50.439512+0000","last_unstale":"2024-02-13T00:46:48.463606+0000","last_undegraded":"2024-02-13T00:46:48.463606+0000","last_fullsized":"2024-02-13T00:46:48.463606+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:58.969692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:44:58.969692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":248,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458969+0000","last_change":"2024-02-13T00:45:15.431130+0000","last_active":"2024-02-13T00:46:48.458969+0000","last_peered":"2024-02-13T00:46:48.458969+0000","last_clean":"2024-02-13T00:46:48.458969+0000","last_became_active":"2024-02-13T00:45:15.430749+0000","last_became_peered":"2024-02-13T00:45:15.430749+0000","last_unstale":"2024-02-13T00:46:48.458969+0000","last_undegraded":"2024-02-13T00:46:48.458969+0000","last_fullsized":"2024-02-13T00:46:48.458969+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.605111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.605111+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":502,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467291+0000","last_change":"2024-02-13T00:46:02.034931+0000","last_active":"2024-02-13T00:46:48.467291+0000","last_peered":"2024-02-13T00:46:48.467291+0000","last_clean":"2024-02-13T00:46:48.467291+0000","last_became_active":"2024-02-13T00:46:01.443721+0000","last_became_peered":"2024-02-13T00:46:01.443721+0000","last_unstale":"2024-02-13T00:46:48.467291+0000","last_undegraded":"2024-02-13T00:46:48.467291+0000","last_fullsized":"2024-02-13T00:46:48.467291+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:43:07.393197+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.393197+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":367,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460172+0000","last_change":"2024-02-13T00:46:32.298033+0000","last_active":"2024-02-13T00:46:48.460172+0000","last_peered":"2024-02-13T00:46:48.460172+0000","last_clean":"2024-02-13T00:46:48.460172+0000","last_became_active":"2024-02-13T00:46:01.442231+0000","last_became_peered":"2024-02-13T00:46:01.442231+0000","last_unstale":"2024-02-13T00:46:48.460172+0000","last_undegraded":"2024-02-13T00:46:48.460172+0000","last_fullsized":"2024-02-13T00:46:48.460172+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.297918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.297918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"363'14","reported_seq":162,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:49.320273+0000","last_change":"2024-02-13T00:45:18.328879+0000","last_active":"2024-02-13T00:46:49.320273+0000","last_peered":"2024-02-13T00:46:49.320273+0000","last_clean":"2024-02-13T00:46:49.320273+0000","last_became_active":"2024-02-13T00:45:15.455825+0000","last_became_peered":"2024-02-13T00:45:15.455825+0000","last_unstale":"2024-02-13T00:46:49.320273+0000","last_undegraded":"2024-02-13T00:46:49.320273+0000","last_fullsized":"2024-02-13T00:46:49.320273+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'10","last_scrub_stamp":"2024-02-13T00:43:05.604461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.604461+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":226,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458703+0000","last_change":"2024-02-13T00:46:03.966962+0000","last_active":"2024-02-13T00:46:48.458703+0000","last_peered":"2024-02-13T00:46:48.458703+0000","last_clean":"2024-02-13T00:46:48.458703+0000","last_became_active":"2024-02-13T00:46:01.439984+0000","last_became_peered":"2024-02-13T00:46:01.439984+0000","last_unstale":"2024-02-13T00:46:48.458703+0000","last_undegraded":"2024-02-13T00:46:48.458703+0000","last_fullsized":"2024-02-13T00:46:48.458703+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:42:55.481128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:55.481128+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":364,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467572+0000","last_change":"2024-02-13T00:45:38.084426+0000","last_active":"2024-02-13T00:46:48.467572+0000","last_peered":"2024-02-13T00:46:48.467572+0000","last_clean":"2024-02-13T00:46:48.467572+0000","last_became_active":"2024-02-13T00:45:15.435550+0000","last_became_peered":"2024-02-13T00:45:15.435550+0000","last_unstale":"2024-02-13T00:46:48.467572+0000","last_undegraded":"2024-02-13T00:46:48.467572+0000","last_fullsized":"2024-02-13T00:46:48.467572+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.084358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.084358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":153,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461318+0000","last_change":"2024-02-13T00:46:10.679144+0000","last_active":"2024-02-13T00:46:48.461318+0000","last_peered":"2024-02-13T00:46:48.461318+0000","last_clean":"2024-02-13T00:46:48.461318+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:46:48.461318+0000","last_undegraded":"2024-02-13T00:46:48.461318+0000","last_fullsized":"2024-02-13T00:46:48.461318+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:42:59.604002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:59.604002+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459920+0000","last_change":"2024-02-13T00:45:15.437121+0000","last_active":"2024-02-13T00:46:48.459920+0000","last_peered":"2024-02-13T00:46:48.459920+0000","last_clean":"2024-02-13T00:46:48.459920+0000","last_became_active":"2024-02-13T00:45:15.436239+0000","last_became_peered":"2024-02-13T00:45:15.436239+0000","last_unstale":"2024-02-13T00:46:48.459920+0000","last_undegraded":"2024-02-13T00:46:48.459920+0000","last_fullsized":"2024-02-13T00:46:48.459920+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:46.698599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:41:46.698599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":408,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.467167+0000","last_change":"2024-02-13T00:45:15.423191+0000","last_active":"2024-02-13T00:46:48.467167+0000","last_peered":"2024-02-13T00:46:48.467167+0000","last_clean":"2024-02-13T00:46:48.467167+0000","last_became_active":"2024-02-13T00:45:15.422870+0000","last_became_peered":"2024-02-13T00:45:15.422870+0000","last_unstale":"2024-02-13T00:46:48.467167+0000","last_undegraded":"2024-02-13T00:46:48.467167+0000","last_fullsized":"2024-02-13T00:46:48.467167+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.596334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.596334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":400,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461588+0000","last_change":"2024-02-13T00:46:01.457036+0000","last_active":"2024-02-13T00:46:48.461588+0000","last_peered":"2024-02-13T00:46:48.461588+0000","last_clean":"2024-02-13T00:46:48.461588+0000","last_became_active":"2024-02-13T00:46:01.453269+0000","last_became_peered":"2024-02-13T00:46:01.453269+0000","last_unstale":"2024-02-13T00:46:48.461588+0000","last_undegraded":"2024-02-13T00:46:48.461588+0000","last_fullsized":"2024-02-13T00:46:48.461588+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.245237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.245237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":130,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460048+0000","last_change":"2024-02-13T00:45:37.280947+0000","last_active":"2024-02-13T00:46:48.460048+0000","last_peered":"2024-02-13T00:46:48.460048+0000","last_clean":"2024-02-13T00:46:48.460048+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:46:48.460048+0000","last_undegraded":"2024-02-13T00:46:48.460048+0000","last_fullsized":"2024-02-13T00:46:48.460048+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.280865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.280865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":161,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.461453+0000","last_change":"2024-02-13T00:44:55.939050+0000","last_active":"2024-02-13T00:46:48.461453+0000","last_peered":"2024-02-13T00:46:48.461453+0000","last_clean":"2024-02-13T00:46:48.461453+0000","last_became_active":"2024-02-13T00:44:50.442894+0000","last_became_peered":"2024-02-13T00:44:50.442894+0000","last_unstale":"2024-02-13T00:46:48.461453+0000","last_undegraded":"2024-02-13T00:46:48.461453+0000","last_fullsized":"2024-02-13T00:46:48.461453+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:43:30.578602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.578602+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":125,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462779+0000","last_change":"2024-02-13T00:45:15.446020+0000","last_active":"2024-02-13T00:46:48.462779+0000","last_peered":"2024-02-13T00:46:48.462779+0000","last_clean":"2024-02-13T00:46:48.462779+0000","last_became_active":"2024-02-13T00:45:15.445403+0000","last_became_peered":"2024-02-13T00:45:15.445403+0000","last_unstale":"2024-02-13T00:46:48.462779+0000","last_undegraded":"2024-02-13T00:46:48.462779+0000","last_fullsized":"2024-02-13T00:46:48.462779+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:06.651077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:06.651077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":388,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462247+0000","last_change":"2024-02-13T00:41:56.204734+0000","last_active":"2024-02-13T00:46:48.462247+0000","last_peered":"2024-02-13T00:46:48.462247+0000","last_clean":"2024-02-13T00:46:48.462247+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:46:48.462247+0000","last_undegraded":"2024-02-13T00:46:48.462247+0000","last_fullsized":"2024-02-13T00:46:48.462247+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:56.204671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.204671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":125,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460225+0000","last_change":"2024-02-13T00:46:01.451373+0000","last_active":"2024-02-13T00:46:48.460225+0000","last_peered":"2024-02-13T00:46:48.460225+0000","last_clean":"2024-02-13T00:46:48.460225+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:46:48.460225+0000","last_undegraded":"2024-02-13T00:46:48.460225+0000","last_fullsized":"2024-02-13T00:46:48.460225+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.410777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.410777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":39,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459317+0000","last_change":"2024-02-13T00:46:01.451610+0000","last_active":"2024-02-13T00:46:48.459317+0000","last_peered":"2024-02-13T00:46:48.459317+0000","last_clean":"2024-02-13T00:46:48.459317+0000","last_became_active":"2024-02-13T00:46:01.449873+0000","last_became_peered":"2024-02-13T00:46:01.449873+0000","last_unstale":"2024-02-13T00:46:48.459317+0000","last_undegraded":"2024-02-13T00:46:48.459317+0000","last_fullsized":"2024-02-13T00:46:48.459317+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.554065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.554065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":271,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462154+0000","last_change":"2024-02-13T00:44:44.964075+0000","last_active":"2024-02-13T00:46:48.462154+0000","last_peered":"2024-02-13T00:46:48.462154+0000","last_clean":"2024-02-13T00:46:48.462154+0000","last_became_active":"2024-02-13T00:44:39.504670+0000","last_became_peered":"2024-02-13T00:44:39.504670+0000","last_unstale":"2024-02-13T00:46:48.462154+0000","last_undegraded":"2024-02-13T00:46:48.462154+0000","last_fullsized":"2024-02-13T00:46:48.462154+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:41:41.686058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.686058+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460335+0000","last_change":"2024-02-13T00:46:01.448134+0000","last_active":"2024-02-13T00:46:48.460335+0000","last_peered":"2024-02-13T00:46:48.460335+0000","last_clean":"2024-02-13T00:46:48.460335+0000","last_became_active":"2024-02-13T00:46:01.446635+0000","last_became_peered":"2024-02-13T00:46:01.446635+0000","last_unstale":"2024-02-13T00:46:48.460335+0000","last_undegraded":"2024-02-13T00:46:48.460335+0000","last_fullsized":"2024-02-13T00:46:48.460335+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:28.101254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:28.101254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":115,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460871+0000","last_change":"2024-02-13T00:45:15.435977+0000","last_active":"2024-02-13T00:46:48.460871+0000","last_peered":"2024-02-13T00:46:48.460871+0000","last_clean":"2024-02-13T00:46:48.460871+0000","last_became_active":"2024-02-13T00:45:15.435206+0000","last_became_peered":"2024-02-13T00:45:15.435206+0000","last_unstale":"2024-02-13T00:46:48.460871+0000","last_undegraded":"2024-02-13T00:46:48.460871+0000","last_fullsized":"2024-02-13T00:46:48.460871+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:57.802417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.802417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":130,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460096+0000","last_change":"2024-02-13T00:46:01.461188+0000","last_active":"2024-02-13T00:46:48.460096+0000","last_peered":"2024-02-13T00:46:48.460096+0000","last_clean":"2024-02-13T00:46:48.460096+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:46:48.460096+0000","last_undegraded":"2024-02-13T00:46:48.460096+0000","last_fullsized":"2024-02-13T00:46:48.460096+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.238152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.238152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":374,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462954+0000","last_change":"2024-02-13T00:46:01.443734+0000","last_active":"2024-02-13T00:46:48.462954+0000","last_peered":"2024-02-13T00:46:48.462954+0000","last_clean":"2024-02-13T00:46:48.462954+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:46:48.462954+0000","last_undegraded":"2024-02-13T00:46:48.462954+0000","last_fullsized":"2024-02-13T00:46:48.462954+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.621170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.621170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":270,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463594+0000","last_change":"2024-02-13T00:46:27.706096+0000","last_active":"2024-02-13T00:46:48.463594+0000","last_peered":"2024-02-13T00:46:48.463594+0000","last_clean":"2024-02-13T00:46:48.463594+0000","last_became_active":"2024-02-13T00:44:50.415894+0000","last_became_peered":"2024-02-13T00:44:50.415894+0000","last_unstale":"2024-02-13T00:46:48.463594+0000","last_undegraded":"2024-02-13T00:46:48.463594+0000","last_fullsized":"2024-02-13T00:46:48.463594+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:46:27.706006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:27.706006+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":209,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.458989+0000","last_change":"2024-02-13T00:46:01.452098+0000","last_active":"2024-02-13T00:46:48.458989+0000","last_peered":"2024-02-13T00:46:48.458989+0000","last_clean":"2024-02-13T00:46:48.458989+0000","last_became_active":"2024-02-13T00:46:01.451790+0000","last_became_peered":"2024-02-13T00:46:01.451790+0000","last_unstale":"2024-02-13T00:46:48.458989+0000","last_undegraded":"2024-02-13T00:46:48.458989+0000","last_fullsized":"2024-02-13T00:46:48.458989+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:03.665973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:03.665973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":363,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.464484+0000","last_change":"2024-02-13T00:46:01.454932+0000","last_active":"2024-02-13T00:46:48.464484+0000","last_peered":"2024-02-13T00:46:48.464484+0000","last_clean":"2024-02-13T00:46:48.464484+0000","last_became_active":"2024-02-13T00:46:01.454163+0000","last_became_peered":"2024-02-13T00:46:01.454163+0000","last_unstale":"2024-02-13T00:46:48.464484+0000","last_undegraded":"2024-02-13T00:46:48.464484+0000","last_fullsized":"2024-02-13T00:46:48.464484+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.415414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.415414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":395,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462807+0000","last_change":"2024-02-13T00:45:15.420228+0000","last_active":"2024-02-13T00:46:48.462807+0000","last_peered":"2024-02-13T00:46:48.462807+0000","last_clean":"2024-02-13T00:46:48.462807+0000","last_became_active":"2024-02-13T00:45:15.419668+0000","last_became_peered":"2024-02-13T00:45:15.419668+0000","last_unstale":"2024-02-13T00:46:48.462807+0000","last_undegraded":"2024-02-13T00:46:48.462807+0000","last_fullsized":"2024-02-13T00:46:48.462807+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:21.648002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.648002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":308,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.464699+0000","last_change":"2024-02-13T00:46:01.457482+0000","last_active":"2024-02-13T00:46:48.464699+0000","last_peered":"2024-02-13T00:46:48.464699+0000","last_clean":"2024-02-13T00:46:48.464699+0000","last_became_active":"2024-02-13T00:46:01.455940+0000","last_became_peered":"2024-02-13T00:46:01.455940+0000","last_unstale":"2024-02-13T00:46:48.464699+0000","last_undegraded":"2024-02-13T00:46:48.464699+0000","last_fullsized":"2024-02-13T00:46:48.464699+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:21.087199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:21.087199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"85'14","reported_seq":290,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463164+0000","last_change":"2024-02-13T00:46:36.089483+0000","last_active":"2024-02-13T00:46:48.463164+0000","last_peered":"2024-02-13T00:46:48.463164+0000","last_clean":"2024-02-13T00:46:48.463164+0000","last_became_active":"2024-02-13T00:45:15.419364+0000","last_became_peered":"2024-02-13T00:45:15.419364+0000","last_unstale":"2024-02-13T00:46:48.463164+0000","last_undegraded":"2024-02-13T00:46:48.463164+0000","last_fullsized":"2024-02-13T00:46:48.463164+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:46:36.089402+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:46:36.089402+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":134,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460101+0000","last_change":"2024-02-13T00:46:01.452313+0000","last_active":"2024-02-13T00:46:48.460101+0000","last_peered":"2024-02-13T00:46:48.460101+0000","last_clean":"2024-02-13T00:46:48.460101+0000","last_became_active":"2024-02-13T00:46:01.451997+0000","last_became_peered":"2024-02-13T00:46:01.451997+0000","last_unstale":"2024-02-13T00:46:48.460101+0000","last_undegraded":"2024-02-13T00:46:48.460101+0000","last_fullsized":"2024-02-13T00:46:48.460101+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:21.630532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.630532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":233,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.465879+0000","last_change":"2024-02-13T00:45:15.416632+0000","last_active":"2024-02-13T00:46:48.465879+0000","last_peered":"2024-02-13T00:46:48.465879+0000","last_clean":"2024-02-13T00:46:48.465879+0000","last_became_active":"2024-02-13T00:45:15.416307+0000","last_became_peered":"2024-02-13T00:45:15.416307+0000","last_unstale":"2024-02-13T00:46:48.465879+0000","last_undegraded":"2024-02-13T00:46:48.465879+0000","last_fullsized":"2024-02-13T00:46:48.465879+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:48.629024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.629024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":345,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459504+0000","last_change":"2024-02-13T00:46:01.445422+0000","last_active":"2024-02-13T00:46:48.459504+0000","last_peered":"2024-02-13T00:46:48.459504+0000","last_clean":"2024-02-13T00:46:48.459504+0000","last_became_active":"2024-02-13T00:46:01.445141+0000","last_became_peered":"2024-02-13T00:46:01.445141+0000","last_unstale":"2024-02-13T00:46:48.459504+0000","last_undegraded":"2024-02-13T00:46:48.459504+0000","last_fullsized":"2024-02-13T00:46:48.459504+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:52.447698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.447698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":121,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462514+0000","last_change":"2024-02-13T00:45:30.186208+0000","last_active":"2024-02-13T00:46:48.462514+0000","last_peered":"2024-02-13T00:46:48.462514+0000","last_clean":"2024-02-13T00:46:48.462514+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:46:48.462514+0000","last_undegraded":"2024-02-13T00:46:48.462514+0000","last_fullsized":"2024-02-13T00:46:48.462514+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:30.186129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:30.186129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"90'20","reported_seq":170,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462561+0000","last_change":"2024-02-13T00:46:06.251713+0000","last_active":"2024-02-13T00:46:48.462561+0000","last_peered":"2024-02-13T00:46:48.462561+0000","last_clean":"2024-02-13T00:46:48.462561+0000","last_became_active":"2024-02-13T00:46:01.442464+0000","last_became_peered":"2024-02-13T00:46:01.442464+0000","last_unstale":"2024-02-13T00:46:48.462561+0000","last_undegraded":"2024-02-13T00:46:48.462561+0000","last_fullsized":"2024-02-13T00:46:48.462561+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:43:44.241358+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:43:44.241358+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":210,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466440+0000","last_change":"2024-02-13T00:46:01.450078+0000","last_active":"2024-02-13T00:46:48.466440+0000","last_peered":"2024-02-13T00:46:48.466440+0000","last_clean":"2024-02-13T00:46:48.466440+0000","last_became_active":"2024-02-13T00:46:01.447654+0000","last_became_peered":"2024-02-13T00:46:01.447654+0000","last_unstale":"2024-02-13T00:46:48.466440+0000","last_undegraded":"2024-02-13T00:46:48.466440+0000","last_fullsized":"2024-02-13T00:46:48.466440+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:18.464402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:42:18.464402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":406,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459311+0000","last_change":"2024-02-13T00:46:01.445302+0000","last_active":"2024-02-13T00:46:48.459311+0000","last_peered":"2024-02-13T00:46:48.459311+0000","last_clean":"2024-02-13T00:46:48.459311+0000","last_became_active":"2024-02-13T00:46:01.444842+0000","last_became_peered":"2024-02-13T00:46:01.444842+0000","last_unstale":"2024-02-13T00:46:48.459311+0000","last_undegraded":"2024-02-13T00:46:48.459311+0000","last_fullsized":"2024-02-13T00:46:48.459311+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:34.693214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:34.693214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":44,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.465773+0000","last_change":"2024-02-13T00:46:01.449363+0000","last_active":"2024-02-13T00:46:48.465773+0000","last_peered":"2024-02-13T00:46:48.465773+0000","last_clean":"2024-02-13T00:46:48.465773+0000","last_became_active":"2024-02-13T00:46:01.448971+0000","last_became_peered":"2024-02-13T00:46:01.448971+0000","last_unstale":"2024-02-13T00:46:48.465773+0000","last_undegraded":"2024-02-13T00:46:48.465773+0000","last_fullsized":"2024-02-13T00:46:48.465773+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:47.487860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:47.487860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":376,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.459624+0000","last_change":"2024-02-13T00:45:32.233265+0000","last_active":"2024-02-13T00:46:48.459624+0000","last_peered":"2024-02-13T00:46:48.459624+0000","last_clean":"2024-02-13T00:46:48.459624+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:46:48.459624+0000","last_undegraded":"2024-02-13T00:46:48.459624+0000","last_fullsized":"2024-02-13T00:46:48.459624+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:45:32.233197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.233197+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"86'50","reported_seq":291,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.463574+0000","last_change":"2024-02-13T00:46:05.700233+0000","last_active":"2024-02-13T00:46:48.463574+0000","last_peered":"2024-02-13T00:46:48.463574+0000","last_clean":"2024-02-13T00:46:48.463574+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:46:48.463574+0000","last_undegraded":"2024-02-13T00:46:48.463574+0000","last_fullsized":"2024-02-13T00:46:48.463574+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:42:17.454232+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.454232+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":401,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.466574+0000","last_change":"2024-02-13T00:45:15.434575+0000","last_active":"2024-02-13T00:46:48.466574+0000","last_peered":"2024-02-13T00:46:48.466574+0000","last_clean":"2024-02-13T00:46:48.466574+0000","last_became_active":"2024-02-13T00:45:15.434224+0000","last_became_peered":"2024-02-13T00:45:15.434224+0000","last_unstale":"2024-02-13T00:46:48.466574+0000","last_undegraded":"2024-02-13T00:46:48.466574+0000","last_fullsized":"2024-02-13T00:46:48.466574+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:43:03.464398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":116,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.462372+0000","last_change":"2024-02-13T00:44:39.510101+0000","last_active":"2024-02-13T00:46:48.462372+0000","last_peered":"2024-02-13T00:46:48.462372+0000","last_clean":"2024-02-13T00:46:48.462372+0000","last_became_active":"2024-02-13T00:44:39.508197+0000","last_became_peered":"2024-02-13T00:44:39.508197+0000","last_unstale":"2024-02-13T00:46:48.462372+0000","last_undegraded":"2024-02-13T00:46:48.462372+0000","last_fullsized":"2024-02-13T00:46:48.462372+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:46.418859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:41:46.418859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":125,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-02-13T00:46:48.460499+0000","last_change":"2024-02-13T00:45:15.435955+0000","last_active":"2024-02-13T00:46:48.460499+0000","last_peered":"2024-02-13T00:46:48.460499+0000","last_clean":"2024-02-13T00:46:48.460499+0000","last_became_active":"2024-02-13T00:45:15.435259+0000","last_became_peered":"2024-02-13T00:45:15.435259+0000","last_unstale":"2024-02-13T00:46:48.460499+0000","last_undegraded":"2024-02-13T00:46:48.460499+0000","last_fullsized":"2024-02-13T00:46:48.460499+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.953989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.953989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":213,"num_pg":32,"stat_sum":{"num_bytes":18,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":4},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":4001489,"num_objects":159,"num_object_clones":2,"num_object_copies":477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":101,"num_read_kb":6,"num_write":597,"num_write_kb":4496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1266,"num_bytes_recovered":4004226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14077952,"data_stored":12004881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1888,"ondisk_log_size":1888,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1063,"num_read_kb":788,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":9606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":11656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":536,"num_read_kb":531,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":227633266800,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":45,"seq":193273528438,"num_pgs":185,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320740,"kb_used_data":7120,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93424412,"statfs":{"total":95995035648,"available":95666597888,"internally_reserved":0,"allocated":7290880,"data_stored":3986218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.62,"15min":0.56000000000000005},"min":{"1min":0.25600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.61199999999999999,"5min":8.1340000000000003,"15min":8.1340000000000003},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.85599999999999998,"15min":0.60499999999999998},"min":{"1min":0.309,"5min":0.221,"15min":0.221},"max":{"1min":1.0309999999999999,"5min":11.367000000000001,"15min":11.367000000000001},"last":0.622}]},{"osd":1,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.03,"15min":0.88700000000000001},"min":{"1min":0.317,"5min":0.29699999999999999,"15min":0.27200000000000002},"max":{"1min":0.64500000000000002,"5min":24.209,"15min":54.002000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.88300000000000001,"15min":0.68300000000000005},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63900000000000001,"5min":12.403,"15min":12.403},"last":0.65400000000000003}]},{"osd":2,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.98399999999999999,"15min":0.70699999999999996},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.002,"5min":12.477,"15min":12.477},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.99099999999999999,"15min":0.67600000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.95199999999999996,"5min":11.702,"15min":11.702},"last":0.61399999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.87,"15min":0.67100000000000004},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.64600000000000002,"5min":8.0519999999999996,"15min":8.0519999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":1,"15min":0.77600000000000002},"min":{"1min":0.33700000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.2330000000000001,"5min":13.257,"15min":19.081},"last":0.57499999999999996}]},{"osd":4,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41399999999999998,"15min":0.41099999999999998},"min":{"1min":0.26100000000000001,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":1.0649999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.27600000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.83299999999999996},"last":0.46100000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.434,"15min":0.41499999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.61199999999999999,"5min":0.66500000000000004,"15min":0.94399999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.48199999999999998,"15min":0.46999999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.86299999999999999},"last":0.60099999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:46:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46899999999999997,"15min":0.49199999999999999},"min":{"1min":0.29499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":1.173},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51000000000000001,"15min":0.54000000000000004},"min":{"1min":0.32900000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":1.262},"last":0.63500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574991,"num_pgs":192,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316576,"kb_used_data":11148,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93428576,"statfs":{"total":95995035648,"available":95670861824,"internally_reserved":0,"allocated":11415552,"data_stored":7995034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":1.7070000000000001,"15min":0.94699999999999995},"min":{"1min":0.223,"5min":0.189,"15min":0.189},"max":{"1min":0.60899999999999999,"5min":61.856000000000002,"15min":61.856000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":1.244,"15min":0.66200000000000003},"min":{"1min":0.26500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.44800000000000001,"5min":61.908999999999999,"15min":61.908999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":1.825,"15min":0.85699999999999998},"min":{"1min":0.22900000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.80000000000000004,"5min":32.161000000000001,"15min":32.161000000000001},"last":0.255},{"interface":"front","average":{"1min":0.39300000000000002,"5min":1.716,"15min":0.83199999999999996},"min":{"1min":0.246,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.21,"5min":29.693000000000001,"15min":29.693000000000001},"last":0.36099999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":2.5379999999999998,"15min":1.2849999999999999},"min":{"1min":0.26800000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.46600000000000003,"5min":131.23099999999999,"15min":131.23099999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":1.5669999999999999,"15min":0.88100000000000001},"min":{"1min":0.307,"5min":0.26800000000000002,"15min":0.246},"max":{"1min":0.61799999999999999,"5min":49.055,"15min":49.055},"last":0.30199999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:46:47 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.64100000000000001,"15min":0.76400000000000001},"min":{"1min":0.33500000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61699999999999999,"5min":7.2089999999999996,"15min":16.547000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.70899999999999996,"15min":0.70599999999999996},"min":{"1min":0.40100000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64500000000000002,"5min":10.050000000000001,"15min":14.731999999999999},"last":0.53200000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.5680000000000001,"15min":0.90200000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.90500000000000003,"5min":26.561,"15min":26.561},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.137,"15min":0.76900000000000002},"min":{"1min":0.377,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.163,"5min":16.494,"15min":16.494},"last":0.55300000000000005}]},{"osd":6,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.84299999999999997,"15min":0.72699999999999998},"min":{"1min":0.41999999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.92700000000000005,"5min":14.772,"15min":14.772},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":1.208,"15min":0.93899999999999995},"min":{"1min":0.41399999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.78500000000000003,"5min":26.600000000000001,"15min":26.600000000000001},"last":0.39900000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":1.006,"15min":0.73099999999999998},"min":{"1min":0.432,"5min":0.313,"15min":0.313},"max":{"1min":0.95899999999999996,"5min":14.794,"15min":14.794},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.591,"15min":0.94299999999999995},"min":{"1min":0.40600000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80100000000000005,"5min":26.693999999999999,"15min":26.693999999999999},"last":0.50600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738515,"num_pgs":163,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320548,"kb_used_data":11028,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93424604,"statfs":{"total":95995035648,"available":95666794496,"internally_reserved":0,"allocated":11292672,"data_stored":7994759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:46:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":1.157,"15min":0.62},"min":{"1min":0.20999999999999999,"5min":0.192,"15min":0.189},"max":{"1min":1.7370000000000001,"5min":30.184000000000001,"15min":30.184000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.55000000000000004,"15min":0.44700000000000001},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.17599999999999999},"max":{"1min":2.839,"5min":5.6959999999999997,"15min":5.6959999999999997},"last":0.24399999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.67300000000000004,"15min":0.497},"min":{"1min":0.25,"5min":0.216,"15min":0.20999999999999999},"max":{"1min":0.94799999999999995,"5min":7.0839999999999996,"15min":7.0839999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.625,"5min":1.2310000000000001,"15min":0.66900000000000004},"min":{"1min":0.26800000000000002,"5min":0.20899999999999999,"15min":0.185},"max":{"1min":3.6949999999999998,"5min":24.459,"15min":24.459},"last":0.315}]},{"osd":3,"last update":"Tue Feb 13 00:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67700000000000005,"15min":0.49399999999999999},"min":{"1min":0.26500000000000001,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":4.8710000000000004,"5min":8.718,"15min":8.718},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.72499999999999998,"15min":0.55300000000000005},"min":{"1min":0.316,"5min":0.222,"15min":0.222},"max":{"1min":2.3140000000000001,"5min":7.1139999999999999,"15min":7.1139999999999999},"last":0.32300000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:46:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.84299999999999997,"15min":0.79400000000000004},"min":{"1min":0.38300000000000001,"5min":0.29599999999999999,"15min":0.24399999999999999},"max":{"1min":1.639,"5min":9.2400000000000002,"15min":24.381},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.82599999999999996,"15min":0.65000000000000002},"min":{"1min":0.34899999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.81299999999999994,"5min":8.6950000000000003,"15min":8.6950000000000003},"last":0.46100000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:46:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.749,"15min":0.63},"min":{"1min":0.36399999999999999,"5min":0.309,"15min":0.309},"max":{"1min":3.331,"5min":7.0990000000000002,"15min":7.0990000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.4330000000000001,"15min":0.85099999999999998},"min":{"1min":0.33900000000000002,"5min":0.307,"15min":0.30599999999999999},"max":{"1min":10.686,"5min":24.427,"15min":24.427},"last":0.504}]},{"osd":6,"last update":"Tue Feb 13 00:46:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.81100000000000005,"15min":0.622},"min":{"1min":0.376,"5min":0.32900000000000001,"15min":0.31},"max":{"1min":1.4990000000000001,"5min":8.6129999999999995,"15min":8.6129999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.79200000000000004,"15min":0.64200000000000002},"min":{"1min":0.35099999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.83399999999999996,"5min":14.263,"15min":14.263},"last":0.45000000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:45:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.89000000000000001,"15min":0.66500000000000004},"min":{"1min":0.441,"5min":0.317,"15min":0.317},"max":{"1min":7.7720000000000002,"5min":8.6679999999999993,"15min":8.6679999999999993},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.84399999999999997,"15min":0.67000000000000004},"min":{"1min":0.41199999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.006,"5min":8.1609999999999996,"15min":8.1609999999999996},"last":0.51300000000000001}]}]},{"osd":2,"up_from":0,"seq":77309411466,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182532,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986304,"num_pgs":165,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316560,"kb_used_data":11128,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93428592,"statfs":{"total":95995035648,"available":95670878208,"internally_reserved":0,"allocated":11395072,"data_stored":7995068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:46:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.55500000000000005,"15min":0.60199999999999998},"min":{"1min":0.248,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.70399999999999996,"5min":5.3319999999999999,"15min":9.3399999999999999},"last":0.436},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":11.532,"5min":11.532,"15min":13.833},"last":0.41899999999999998}]},{"osd":1,"last update":"Tue Feb 13 00:46:49 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.59599999999999997,"15min":0.76500000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.75700000000000001,"5min":7.5410000000000004,"15min":53.188000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.998,"5min":0.59799999999999998,"15min":0.70799999999999996},"min":{"1min":0.30499999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":11.497999999999999,"5min":11.497999999999999,"15min":44.793999999999997},"last":0.43099999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:46:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.59099999999999997,"15min":0.59999999999999998},"min":{"1min":0.40300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.2199999999999998,"5min":4.2199999999999998,"15min":8.6829999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":18.911000000000001},"last":0.36899999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:46:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.60499999999999998,"15min":0.66300000000000003},"min":{"1min":0.312,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.0090000000000003,"5min":4.0300000000000002,"15min":23.5},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.57599999999999996,"15min":0.72299999999999998},"min":{"1min":0.39700000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.9729999999999999,"5min":3.9729999999999999,"15min":36.152999999999999},"last":0.40999999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.42299999999999999,"15min":0.38500000000000001},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.20599999999999999},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.377},{"interface":"front","average":{"1min":0.505,"5min":0.443,"15min":0.40500000000000003},"min":{"1min":0.27100000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.34799999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:46:19 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46700000000000003,"15min":0.432},"min":{"1min":0.28000000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67200000000000004,"5min":3.153,"15min":3.153},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.50800000000000001,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.254,"15min":0.254},"max":{"1min":0.70099999999999996,"5min":4.9390000000000001,"15min":4.9390000000000001},"last":0.39900000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:46:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.58399999999999996,"5min":1.575,"15min":1.575},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.497,"15min":0.47099999999999997},"min":{"1min":0.316,"5min":0.29699999999999999,"15min":0.28100000000000003},"max":{"1min":0.72299999999999998,"5min":2.46,"15min":2.46},"last":0.33500000000000002}]}]},{"osd":5,"up_from":0,"seq":167503724667,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":3328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4427776,"data_stored":4001019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4513792,"data_stored":4001202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4526080,"data_stored":4001216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":1030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":213,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":213,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":18,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":213,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":213,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":18,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:46:55.029 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-13T00:46:55.029 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:46:55.030 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-13T00:46:55.030 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-13T00:46:55.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.5 2024-02-13T00:46:55.077 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:55.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:55.133 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:55.160 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:46:55 smithi181 systemd[1]: Stopping Ceph osd.5 for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:46:55.335 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:55.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:55.440 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:55.469 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:55.498 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:46:55 smithi181 bash[89992]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:46:55.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-13T00:46:55.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.552+0000 7f096b779700 1 -- 172.21.15.162:0/3177053083 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v436) v1 ==== 91+0+0 (secure 0 0 0) 0x7f0970051690 con 0x7f09740f8500 2024-02-13T00:46:55.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 10 librados: watch_flush enter 2024-02-13T00:46:55.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 10 librados: watch_flush exit 2024-02-13T00:46:55.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 20 client.27387.objecter shutdown clearing up homeless session... 2024-02-13T00:46:55.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 10 client.27387.objecter successfully canceled tick 2024-02-13T00:46:55.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 -- 172.21.15.162:0/3177053083 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0960062c90 msgr2=0x7f0960065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:55.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 --2- 172.21.15.162:0/3177053083 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0960062c90 0x7f0960065150 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 rx=0x7f097406c8c0 tx=0x7f096400b040).stop 2024-02-13T00:46:55.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 -- 172.21.15.162:0/3177053083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09740f8500 msgr2=0x7f097406b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:55.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 --2- 172.21.15.162:0/3177053083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09740f8500 0x7f097406b170 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f09700044e0 tx=0x7f0970007f90).stop 2024-02-13T00:46:55.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 -- 172.21.15.162:0/3177053083 shutdown_connections 2024-02-13T00:46:55.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 --2- 172.21.15.162:0/3177053083 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0960062c90 0x7f0960065150 unknown :-1 s=CLOSED pgs=2910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:55.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 --2- 172.21.15.162:0/3177053083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09740fa440 0x7f097406bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:55.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 --2- 172.21.15.162:0/3177053083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09740f8500 0x7f097406b170 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:55.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 --2- 172.21.15.162:0/3177053083 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09740f9b70 0x7f097406b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:55.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 -- 172.21.15.162:0/3177053083 >> 172.21.15.162:0/3177053083 conn(0x7f09740f38c0 msgr2=0x7f09740fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:55.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 -- 172.21.15.162:0/3177053083 shutdown_connections 2024-02-13T00:46:55.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 -- 172.21.15.162:0/3177053083 wait complete. 2024-02-13T00:46:55.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:55.554+0000 7f097c2db700 1 librados: shutdown 2024-02-13T00:46:55.560 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:46:55.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:54.430507+0000 mgr.y (mgr.14544) 384 : cluster [DBG] pgmap v635: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:46:55.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:54.911092+0000 mon.a (mon.0) 2559 : audit [INF] from='client.? 172.21.15.162:0/3177053083' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:46:55.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:54.430507+0000 mgr.y (mgr.14544) 384 : cluster [DBG] pgmap v635: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:46:55.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:54.911092+0000 mon.a (mon.0) 2559 : audit [INF] from='client.? 172.21.15.162:0/3177053083' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:46:55.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:46:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:46:55.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:54.430507+0000 mgr.y (mgr.14544) 384 : cluster [DBG] pgmap v635: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:46:55.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:54.911092+0000 mon.a (mon.0) 2559 : audit [INF] from='client.? 172.21.15.162:0/3177053083' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:46:55.795 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:46:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[49633]: debug 2024-02-13T00:46:55.732+0000 7f6c07e7e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-13T00:46:55.796 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:46:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[49633]: debug 2024-02-13T00:46:55.732+0000 7f6c07e7e700 -1 osd.5 436 *** Got signal Terminated *** 2024-02-13T00:46:55.796 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:46:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[49633]: debug 2024-02-13T00:46:55.732+0000 7f6c07e7e700 -1 osd.5 436 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T00:46:55.800 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:46:56.543 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:46:56] "GET /metrics HTTP/1.1" 200 187815 "" "Prometheus/2.33.4" 2024-02-13T00:46:56.576 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:56.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:56.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:56.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:56.712 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:56.811 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:46:56.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:55.542696+0000 mon.a (mon.0) 2560 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:56.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:55.551008+0000 mon.a (mon.0) 2561 : audit [INF] from='client.? 172.21.15.162:0/3177053083' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:46:56.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:55.551073+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e436: 8 total, 8 up, 4 in 2024-02-13T00:46:56.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:55.640573+0000 mgr.y (mgr.14544) 385 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:56.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:55.734516+0000 mon.a (mon.0) 2563 : cluster [INF] osd.5 marked itself down and dead 2024-02-13T00:46:56.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:55.976801+0000 osd.1 (osd.1) 183 : cluster [DBG] 5.19 scrub starts 2024-02-13T00:46:56.813 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:55.979632+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.19 scrub ok 2024-02-13T00:46:56.813 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:55.542696+0000 mon.a (mon.0) 2560 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:56.813 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:55.551008+0000 mon.a (mon.0) 2561 : audit [INF] from='client.? 172.21.15.162:0/3177053083' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:46:56.814 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:55.551073+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e436: 8 total, 8 up, 4 in 2024-02-13T00:46:56.814 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:55.640573+0000 mgr.y (mgr.14544) 385 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:56.814 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:55.734516+0000 mon.a (mon.0) 2563 : cluster [INF] osd.5 marked itself down and dead 2024-02-13T00:46:56.814 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:55.976801+0000 osd.1 (osd.1) 183 : cluster [DBG] 5.19 scrub starts 2024-02-13T00:46:56.814 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:55.979632+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.19 scrub ok 2024-02-13T00:46:56.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:56.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:46:56.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:46:56.945 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:55.542696+0000 mon.a (mon.0) 2560 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:46:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:55.551008+0000 mon.a (mon.0) 2561 : audit [INF] from='client.? 172.21.15.162:0/3177053083' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:46:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:55.551073+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e436: 8 total, 8 up, 4 in 2024-02-13T00:46:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:55.640573+0000 mgr.y (mgr.14544) 385 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:46:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:55.734516+0000 mon.a (mon.0) 2563 : cluster [INF] osd.5 marked itself down and dead 2024-02-13T00:46:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:55.976801+0000 osd.1 (osd.1) 183 : cluster [DBG] 5.19 scrub starts 2024-02-13T00:46:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:55.979632+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.19 scrub ok 2024-02-13T00:46:57.545 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:46:57 smithi181 bash[90059]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5 2024-02-13T00:46:57.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.152+0000 7fbb24022700 1 -- 172.21.15.162:0/2633360192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb1c0f8d10 msgr2=0x7fbb1c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:57.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.152+0000 7fbb24022700 1 --2- 172.21.15.162:0/2633360192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb1c0f8d10 0x7fbb1c0f90f0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7fbb18004870 tx=0x7fbb18009cb0).stop 2024-02-13T00:46:57.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.152+0000 7fbb24022700 1 -- 172.21.15.162:0/2633360192 shutdown_connections 2024-02-13T00:46:57.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.152+0000 7fbb24022700 1 --2- 172.21.15.162:0/2633360192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbb1c102270 0x7fbb1c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:57.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.152+0000 7fbb24022700 1 --2- 172.21.15.162:0/2633360192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb1c0f8d10 0x7fbb1c0f90f0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:57.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.152+0000 7fbb24022700 1 --2- 172.21.15.162:0/2633360192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1c0f95e0 0x7fbb1c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:57.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.152+0000 7fbb24022700 1 -- 172.21.15.162:0/2633360192 >> 172.21.15.162:0/2633360192 conn(0x7fbb1c06eb90 msgr2=0x7fbb1c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:46:57.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 -- 172.21.15.162:0/2633360192 shutdown_connections 2024-02-13T00:46:57.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 -- 172.21.15.162:0/2633360192 wait complete. 2024-02-13T00:46:57.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 librados: starting msgr at 2024-02-13T00:46:57.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 librados: starting objecter 2024-02-13T00:46:57.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 Processor -- start 2024-02-13T00:46:57.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 -- start start 2024-02-13T00:46:57.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 librados: setting wanted keys 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 librados: calling monclient init 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbb1c0f8d10 0x7fbb1c10e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1c0f95e0 0x7fbb1c10e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb1c102270 0x7fbb1c108270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c108760 con 0x7fbb1c102270 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c1088d0 con 0x7fbb1c0f95e0 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb24022700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c108a90 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb21dbe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbb1c0f8d10 0x7fbb1c10e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb21dbe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbb1c0f8d10 0x7fbb1c10e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:58256/0 (socket says 172.21.15.162:58256) 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb21dbe700 1 -- 172.21.15.162:0/1713912343 learned_addr learned my addr 172.21.15.162:0/1713912343 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.153+0000 7fbb225bf700 1 --2- 172.21.15.162:0/1713912343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb1c102270 0x7fbb1c108270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb215bd700 1 --2- 172.21.15.162:0/1713912343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1c0f95e0 0x7fbb1c10e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:57.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb21dbe700 1 -- 172.21.15.162:0/1713912343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1c0f95e0 msgr2=0x7fbb1c10e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb21dbe700 1 --2- 172.21.15.162:0/1713912343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb1c0f95e0 0x7fbb1c10e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb21dbe700 1 -- 172.21.15.162:0/1713912343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb1c102270 msgr2=0x7fbb1c108270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb21dbe700 1 --2- 172.21.15.162:0/1713912343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb1c102270 0x7fbb1c108270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb21dbe700 1 -- 172.21.15.162:0/1713912343 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb140097c0 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb21dbe700 1 --2- 172.21.15.162:0/1713912343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbb1c0f8d10 0x7fbb1c10e0d0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7fbb18009eb0 tx=0x7fbb1800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb12ffd700 10 client.?.objecter ms_handle_connect 0x7fbb1c0f8d10 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb12ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1800e070 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb18005040 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb18013da0 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb18019070 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb24022700 10 client.37493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:57.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1c108f30 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.154+0000 7fbb24022700 1 librados: init done 2024-02-13T00:46:57.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.155+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fbb1c0475e0 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.155+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fbb1801e420 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb12ffd700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbb08062c90 0x7fbb08065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:57.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(437..437 src has 52..437) v4 ==== 7995+0+0 (secure 0 0 0) 0x7fbb1807dc70 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 osd_map(437..437 src has 52..437) v4 2024-02-13T00:46:57.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb12ffd700 3 client.37493.objecter handle_osd_map got epochs [437,437] > 0 2024-02-13T00:46:57.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb12ffd700 3 client.37493.objecter handle_osd_map decoding full epoch 437 2024-02-13T00:46:57.599 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:46:57.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb215bd700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbb08062c90 0x7fbb08065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:57.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb12ffd700 20 client.37493.objecter dump_active .. 0 homeless 2024-02-13T00:46:57.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v437) v1 ==== 54+0+16 (secure 0 0 0) 0x7fbb18080340 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb12ffd700 10 client.37493.objecter ms_handle_connect 0x7fbb08062c90 2024-02-13T00:46:57.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb215bd700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbb08062c90 0x7fbb08065150 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 rx=0x7fbb1c109a10 tx=0x7fbb0c009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:57.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb24022700 10 client.37493.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:57.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 1 2024-02-13T00:46:57.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb24022700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6801/2103743338 conn(0x7fbb1c028590 0x7fbb1c0fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:57.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c107170 osd=0 3 2024-02-13T00:46:57.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c107170 osd=0 3 2024-02-13T00:46:57.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c107170 osd=0 3 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 1 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c107170 osd=0 3 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 0 1 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c107170 osd=0 4 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb24022700 10 client.37493.objecter _send_command 1 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb1c109750 con 0x7fbb1c028590 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.156+0000 7fbb225bf700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6801/2103743338 conn(0x7fbb1c028590 0x7fbb1c0fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.157+0000 7fbb12ffd700 10 client.37493.objecter ms_handle_connect 0x7fbb1c028590 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.157+0000 7fbb225bf700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6801/2103743338 conn(0x7fbb1c028590 0x7fbb1c0fbd00 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.157+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb1c109750 con 0x7fbb1c028590 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.157+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.157+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 1 = system:0 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.157+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c107170 osd=0 4 2024-02-13T00:46:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.157+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 0 1 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.172+0000 7fbb24022700 10 client.37493.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.172+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 2 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.172+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c107170 osd=0 3 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.172+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c107170 osd=0 3 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.172+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c107170 osd=0 3 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.172+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 2 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.172+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c107170 osd=0 3 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.172+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 0 2 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.172+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c107170 osd=0 4 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.172+0000 7fbb24022700 10 client.37493.objecter _send_command 2 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.172+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb1c05a0e0 con 0x7fbb1c028590 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb1c05a0e0 con 0x7fbb1c028590 2024-02-13T00:46:57.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:57.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c107170 osd=0 4 2024-02-13T00:46:57.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 0 2 2024-02-13T00:46:57.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb24022700 10 client.37493.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:57.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 3 2024-02-13T00:46:57.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb24022700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6805/150599780 conn(0x7fbb1c027be0 0x7fbb1c028020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:57.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c0fd370 osd=1 3 2024-02-13T00:46:57.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c0fd370 osd=1 3 2024-02-13T00:46:57.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c0fd370 osd=1 3 2024-02-13T00:46:57.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 3 2024-02-13T00:46:57.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c0fd370 osd=1 3 2024-02-13T00:46:57.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 1 3 2024-02-13T00:46:57.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c0fd370 osd=1 4 2024-02-13T00:46:57.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb24022700 10 client.37493.objecter _send_command 3 2024-02-13T00:46:57.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb1c0fd4e0 con 0x7fbb1c027be0 2024-02-13T00:46:57.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.173+0000 7fbb21dbe700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6805/150599780 conn(0x7fbb1c027be0 0x7fbb1c028020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:57.604 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:46:57.605 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:46:57.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.174+0000 7fbb21dbe700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6805/150599780 conn(0x7fbb1c027be0 0x7fbb1c028020 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:57.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.174+0000 7fbb12ffd700 10 client.37493.objecter ms_handle_connect 0x7fbb1c027be0 2024-02-13T00:46:57.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.174+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb1c0fd4e0 con 0x7fbb1c027be0 2024-02-13T00:46:57.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.174+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:46:57.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.174+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 3 = system:0 2024-02-13T00:46:57.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.174+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c0fd370 osd=1 4 2024-02-13T00:46:57.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.174+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 1 3 2024-02-13T00:46:57.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.189+0000 7fbb24022700 10 client.37493.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.189+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 4 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.189+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c0fd370 osd=1 3 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.189+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c0fd370 osd=1 3 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.189+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c0fd370 osd=1 3 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.189+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 4 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.189+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c0fd370 osd=1 3 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.189+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 1 4 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.189+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c0fd370 osd=1 4 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.189+0000 7fbb24022700 10 client.37493.objecter _send_command 4 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.189+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb1c027650 con 0x7fbb1c027be0 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb1c027650 con 0x7fbb1c027be0 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c0fd370 osd=1 4 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 1 4 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb24022700 10 client.37493.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 5 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb24022700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6809/3689719376 conn(0x7fbb1c0fced0 0x7fbb1c11c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c027a60 osd=2 3 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c027a60 osd=2 3 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c027a60 osd=2 3 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 5 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c027a60 osd=2 3 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 2 5 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c027a60 osd=2 4 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb24022700 10 client.37493.objecter _send_command 5 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb1c11ed00 con 0x7fbb1c0fced0 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.190+0000 7fbb215bd700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6809/3689719376 conn(0x7fbb1c0fced0 0x7fbb1c11c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:57.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.191+0000 7fbb215bd700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6809/3689719376 conn(0x7fbb1c0fced0 0x7fbb1c11c0b0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.191+0000 7fbb12ffd700 10 client.37493.objecter ms_handle_connect 0x7fbb1c0fced0 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.191+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb1c11ed00 con 0x7fbb1c0fced0 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.191+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.191+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 5 = system:0 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.191+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c027a60 osd=2 4 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.191+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 2 5 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.205+0000 7fbb24022700 10 client.37493.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.205+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 6 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.205+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c027a60 osd=2 3 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.205+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c027a60 osd=2 3 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.205+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c027a60 osd=2 3 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.205+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 6 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.205+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c027a60 osd=2 3 2024-02-13T00:46:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.205+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 2 6 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.205+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c027a60 osd=2 4 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.205+0000 7fbb24022700 10 client.37493.objecter _send_command 6 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.205+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb1c0fc4f0 con 0x7fbb1c0fced0 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb1c0fc4f0 con 0x7fbb1c0fced0 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c027a60 osd=2 4 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 2 6 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb24022700 10 client.37493.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 7 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb24022700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6813/604971011 conn(0x7fbb1c11d720 0x7fbb1c11db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c1196f0 osd=3 3 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c1196f0 osd=3 3 2024-02-13T00:46:57.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c1196f0 osd=3 3 2024-02-13T00:46:57.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 7 2024-02-13T00:46:57.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c1196f0 osd=3 3 2024-02-13T00:46:57.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 3 7 2024-02-13T00:46:57.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c1196f0 osd=3 4 2024-02-13T00:46:57.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb24022700 10 client.37493.objecter _send_command 7 2024-02-13T00:46:57.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.207+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb1c119ec0 con 0x7fbb1c11d720 2024-02-13T00:46:57.610 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:46:57.610 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:46:57.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.208+0000 7fbb225bf700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6813/604971011 conn(0x7fbb1c11d720 0x7fbb1c11db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:57.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.208+0000 7fbb12ffd700 10 client.37493.objecter ms_handle_connect 0x7fbb1c11d720 2024-02-13T00:46:57.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.208+0000 7fbb225bf700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.162:6813/604971011 conn(0x7fbb1c11d720 0x7fbb1c11db60 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:57.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.208+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb1c119ec0 con 0x7fbb1c11d720 2024-02-13T00:46:57.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.208+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:46:57.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.208+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 7 = system:0 2024-02-13T00:46:57.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.208+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c1196f0 osd=3 4 2024-02-13T00:46:57.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.208+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 3 7 2024-02-13T00:46:57.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.223+0000 7fbb24022700 10 client.37493.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:57.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.223+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 8 2024-02-13T00:46:57.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.223+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c1196f0 osd=3 3 2024-02-13T00:46:57.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.223+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c1196f0 osd=3 3 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.223+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c1196f0 osd=3 3 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.223+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 8 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.223+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c1196f0 osd=3 3 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.223+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 3 8 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.223+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c1196f0 osd=3 4 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.223+0000 7fbb24022700 10 client.37493.objecter _send_command 8 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.223+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb1c11e8a0 con 0x7fbb1c11d720 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb1c11e8a0 con 0x7fbb1c11d720 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c1196f0 osd=3 4 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 3 8 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb24022700 10 client.37493.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:57.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 9 2024-02-13T00:46:57.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb24022700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.181:6800/3344263388 conn(0x7fbb1c120e80 0x7fbb1c123270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:57.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c11e0d0 osd=4 3 2024-02-13T00:46:57.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c11e0d0 osd=4 3 2024-02-13T00:46:57.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c11e0d0 osd=4 3 2024-02-13T00:46:57.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 9 2024-02-13T00:46:57.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c11e0d0 osd=4 3 2024-02-13T00:46:57.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 4 9 2024-02-13T00:46:57.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c11e0d0 osd=4 4 2024-02-13T00:46:57.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb24022700 10 client.37493.objecter _send_command 9 2024-02-13T00:46:57.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.224+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb1c125ec0 con 0x7fbb1c120e80 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.225+0000 7fbb21dbe700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.181:6800/3344263388 conn(0x7fbb1c120e80 0x7fbb1c123270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.225+0000 7fbb12ffd700 10 client.37493.objecter ms_handle_connect 0x7fbb1c120e80 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.225+0000 7fbb21dbe700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.181:6800/3344263388 conn(0x7fbb1c120e80 0x7fbb1c123270 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.226+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb1c125ec0 con 0x7fbb1c120e80 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.226+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.226+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 9 = system:0 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.226+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c11e0d0 osd=4 4 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.226+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 4 9 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.240+0000 7fbb24022700 10 client.37493.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.241+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 10 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.241+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c11e0d0 osd=4 3 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.241+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c11e0d0 osd=4 3 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.241+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c11e0d0 osd=4 3 2024-02-13T00:46:57.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.241+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 10 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.241+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c11e0d0 osd=4 3 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.241+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 4 10 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.241+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c11e0d0 osd=4 4 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.241+0000 7fbb24022700 10 client.37493.objecter _send_command 10 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.241+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb1c123760 con 0x7fbb1c120e80 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.242+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb1c123760 con 0x7fbb1c120e80 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.242+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.242+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.242+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c11e0d0 osd=4 4 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.242+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 4 10 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.242+0000 7fbb24022700 10 client.37493.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.242+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 11 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.242+0000 7fbb24022700 20 client.37493.objecter _get_session osd=-1 returning homeless 2024-02-13T00:46:57.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.242+0000 7fbb24022700 10 client.37493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:46:57.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.242+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=438}) v3 -- 0x7fbb1c126090 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.242+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbb1c126220 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_get_version_reply(handle=1 version=437) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb1804f600 con 0x7fbb1c0f8d10 2024-02-13T00:46:57.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb13fff700 10 client.37493.objecter _check_command_map_dne tid 11 current 437 map_dne_bound 437 2024-02-13T00:46:57.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb13fff700 10 client.37493.objecter _finish_command 11 = osd:6 osd down 2024-02-13T00:46:57.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb13fff700 15 client.37493.objecter _session_command_op_remove -1 11 2024-02-13T00:46:57.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb24022700 10 client.37493.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:57.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 12 2024-02-13T00:46:57.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb24022700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.181:6808/1506640341 conn(0x7fbb1c1192d0 0x7fbb1c124ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:57.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c119160 osd=6 3 2024-02-13T00:46:57.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c119160 osd=6 3 2024-02-13T00:46:57.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c119160 osd=6 3 2024-02-13T00:46:57.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 12 2024-02-13T00:46:57.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c119160 osd=6 3 2024-02-13T00:46:57.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 6 12 2024-02-13T00:46:57.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c119160 osd=6 4 2024-02-13T00:46:57.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb24022700 10 client.37493.objecter _send_command 12 2024-02-13T00:46:57.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.181:6808/1506640341 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb1c11d3b0 con 0x7fbb1c1192d0 2024-02-13T00:46:57.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb215bd700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.181:6808/1506640341 conn(0x7fbb1c1192d0 0x7fbb1c124ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:57.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb12ffd700 10 client.37493.objecter ms_handle_connect 0x7fbb1c1192d0 2024-02-13T00:46:57.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.243+0000 7fbb215bd700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.181:6808/1506640341 conn(0x7fbb1c1192d0 0x7fbb1c124ab0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:57.618 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:46:57.618 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:46:57.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.244+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb1c11d3b0 con 0x7fbb1c1192d0 2024-02-13T00:46:57.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.244+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 12: 0 ) v1 2024-02-13T00:46:57.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.244+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 12 = system:0 2024-02-13T00:46:57.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.244+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c119160 osd=6 4 2024-02-13T00:46:57.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.244+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 6 12 2024-02-13T00:46:57.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.258+0000 7fbb24022700 10 client.37493.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:57.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.258+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 13 2024-02-13T00:46:57.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.258+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c119160 osd=6 3 2024-02-13T00:46:57.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.258+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c119160 osd=6 3 2024-02-13T00:46:57.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.258+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c119160 osd=6 3 2024-02-13T00:46:57.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.258+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 13 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.258+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c119160 osd=6 3 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.258+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 6 13 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.258+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c119160 osd=6 4 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.258+0000 7fbb24022700 10 client.37493.objecter _send_command 13 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.258+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb1c11d3b0 con 0x7fbb1c1192d0 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb1c11d3b0 con 0x7fbb1c1192d0 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c119160 osd=6 4 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 6 13 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb24022700 10 client.37493.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 14 2024-02-13T00:46:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb24022700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.181:6812/1964192772 conn(0x7fbb1c1254b0 0x7fbb1c1258f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c11d3b0 osd=7 3 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c11d3b0 osd=7 3 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c11d3b0 osd=7 3 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 14 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c11d3b0 osd=7 3 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 7 14 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c11d3b0 osd=7 4 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb24022700 10 client.37493.objecter _send_command 14 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.260+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.181:6812/1964192772 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb1c12cbe0 con 0x7fbb1c1254b0 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.261+0000 7fbb225bf700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.181:6812/1964192772 conn(0x7fbb1c1254b0 0x7fbb1c1258f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.261+0000 7fbb12ffd700 10 client.37493.objecter ms_handle_connect 0x7fbb1c1254b0 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.261+0000 7fbb225bf700 1 --2- 172.21.15.162:0/1713912343 >> v2:172.21.15.181:6812/1964192772 conn(0x7fbb1c1254b0 0x7fbb1c1258f0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.261+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb1c12cbe0 con 0x7fbb1c1254b0 2024-02-13T00:46:57.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.261+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 14: 0 ) v1 2024-02-13T00:46:57.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.261+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 14 = system:0 2024-02-13T00:46:57.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.261+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c11d3b0 osd=7 4 2024-02-13T00:46:57.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.261+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 7 14 2024-02-13T00:46:57.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.276+0000 7fbb24022700 10 client.37493.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:46:57.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.276+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign -1 15 2024-02-13T00:46:57.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.276+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c11d3b0 osd=7 3 2024-02-13T00:46:57.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.276+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c11d3b0 osd=7 3 2024-02-13T00:46:57.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.276+0000 7fbb24022700 20 client.37493.objecter _get_session s=0x7fbb1c11d3b0 osd=7 3 2024-02-13T00:46:57.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.276+0000 7fbb24022700 15 client.37493.objecter _session_command_op_remove -1 15 2024-02-13T00:46:57.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.276+0000 7fbb24022700 20 client.37493.objecter get_session s=0x7fbb1c11d3b0 osd=7 3 2024-02-13T00:46:57.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.276+0000 7fbb24022700 15 client.37493.objecter _session_command_op_assign 7 15 2024-02-13T00:46:57.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.276+0000 7fbb24022700 20 client.37493.objecter put_session s=0x7fbb1c11d3b0 osd=7 4 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.276+0000 7fbb24022700 10 client.37493.objecter _send_command 15 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.276+0000 7fbb24022700 1 -- 172.21.15.162:0/1713912343 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb1c12a480 con 0x7fbb1c1254b0 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.278+0000 7fbb12ffd700 1 -- 172.21.15.162:0/1713912343 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb1c12a480 con 0x7fbb1c1254b0 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.278+0000 7fbb12ffd700 10 client.37493.objecter ms_dispatch 0x7fbb1c001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.278+0000 7fbb12ffd700 10 client.37493.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.278+0000 7fbb12ffd700 20 client.37493.objecter put_session s=0x7fbb1c11d3b0 osd=7 4 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:46:57.278+0000 7fbb12ffd700 15 client.37493.objecter _session_command_op_remove 7 15 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-02-13T00:46:57.623 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.624 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:46:57.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:56.336731+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:46:57.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:56.339882+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:46:57.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:56.431007+0000 mgr.y (mgr.14544) 386 : cluster [DBG] pgmap v637: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 3.1 KiB/s rd, 835 B/s wr, 5 op/s 2024-02-13T00:46:57.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:56.556774+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e437: 8 total, 7 up, 4 in 2024-02-13T00:46:57.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:56.969395+0000 osd.1 (osd.1) 185 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:46:57.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:56.972994+0000 osd.1 (osd.1) 186 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:46:57.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:57.156717+0000 mon.c (mon.2) 381 : audit [DBG] from='client.? 172.21.15.162:0/1713912343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:57.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:56.336731+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:46:57.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:56.339882+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:46:57.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:56.431007+0000 mgr.y (mgr.14544) 386 : cluster [DBG] pgmap v637: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 3.1 KiB/s rd, 835 B/s wr, 5 op/s 2024-02-13T00:46:57.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:56.556774+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e437: 8 total, 7 up, 4 in 2024-02-13T00:46:57.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:56.969395+0000 osd.1 (osd.1) 185 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:46:57.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:56.972994+0000 osd.1 (osd.1) 186 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:46:57.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:57.156717+0000 mon.c (mon.2) 381 : audit [DBG] from='client.? 172.21.15.162:0/1713912343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:57.986 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:56.336731+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:46:57.986 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:56.339882+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:46:57.986 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:56.431007+0000 mgr.y (mgr.14544) 386 : cluster [DBG] pgmap v637: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 3.1 KiB/s rd, 835 B/s wr, 5 op/s 2024-02-13T00:46:57.986 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:56.556774+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e437: 8 total, 7 up, 4 in 2024-02-13T00:46:57.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:56.969395+0000 osd.1 (osd.1) 185 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:46:57.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:56.972994+0000 osd.1 (osd.1) 186 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:46:57.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:57.156717+0000 mon.c (mon.2) 381 : audit [DBG] from='client.? 172.21.15.162:0/1713912343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:46:58.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:46:58.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:58.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:46:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-13T00:46:58.436 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-13T00:46:58.465 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:46:58.466 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:46:58.467 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-02-13T00:46:58.467 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-02-13T00:46:58.470 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:46:58.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:46:58.565 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:58.744 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:46:58.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:58.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:46:58.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:57.378068+0000 osd.6 (osd.6) 113 : cluster [DBG] 4.5 scrub starts 2024-02-13T00:46:58.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:57.381536+0000 osd.6 (osd.6) 114 : cluster [DBG] 4.5 scrub ok 2024-02-13T00:46:58.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:57.378068+0000 osd.6 (osd.6) 113 : cluster [DBG] 4.5 scrub starts 2024-02-13T00:46:58.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:57.381536+0000 osd.6 (osd.6) 114 : cluster [DBG] 4.5 scrub ok 2024-02-13T00:46:58.850 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:46:58.882 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:58.909 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:46:58.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:58.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:46:59.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:59.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:57.378068+0000 osd.6 (osd.6) 113 : cluster [DBG] 4.5 scrub starts 2024-02-13T00:46:59.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:57.381536+0000 osd.6 (osd.6) 114 : cluster [DBG] 4.5 scrub ok 2024-02-13T00:46:59.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:46:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:46:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:46:59.046 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:59.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-13T00:46:59.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:46:59.499 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:59.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:59.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:46:59.631 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:59.649 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:46:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:59.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:46:59.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:46:59.760 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:46:59.779 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:58.431444+0000 mgr.y (mgr.14544) 387 : cluster [DBG] pgmap v639: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 694 B/s wr, 4 op/s 2024-02-13T00:46:59.779 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:46:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:58.881397+0000 mon.c (mon.2) 382 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:46:59.782 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:46:59.793 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:46:59.803 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:46:59.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:46:59.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:46:59.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:58.431444+0000 mgr.y (mgr.14544) 387 : cluster [DBG] pgmap v639: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 694 B/s wr, 4 op/s 2024-02-13T00:46:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:46:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:58.881397+0000 mon.c (mon.2) 382 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:46:59.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:58.431444+0000 mgr.y (mgr.14544) 387 : cluster [DBG] pgmap v639: 238 pgs: 1 active+clean+snaptrim, 237 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 694 B/s wr, 4 op/s 2024-02-13T00:46:59.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:46:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:58.881397+0000 mon.c (mon.2) 382 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:46:59.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:46:59.934 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:00.076 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:00.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:00.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:00.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:00.207 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:00.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:47:00.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.5.service' 2024-02-13T00:47:00.741 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:59.387646+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:47:00.741 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:46:59.388740+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:47:00.741 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:59.616490+0000 mon.c (mon.2) 383 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:47:00.742 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:59.617231+0000 mon.c (mon.2) 384 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:47:00.742 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:46:59.622079+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:47:00.742 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:59.387646+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:47:00.742 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:46:59.388740+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:47:00.742 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:59.616490+0000 mon.c (mon.2) 383 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:47:00.742 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:59.617231+0000 mon.c (mon.2) 384 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:47:00.742 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:46:59.622079+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:47:00.771 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:47:00.772 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-13T00:47:00.772 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-02-13T00:47:00.772 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi181.front.sepia.ceph.com 2024-02-13T00:47:00.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.5 --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-02-13T00:47:00.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:59.387646+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:47:00.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:46:59.388740+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:47:00.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:59.616490+0000 mon.c (mon.2) 383 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:47:00.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:59.617231+0000 mon.c (mon.2) 384 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:47:00.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:46:59.622079+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:47:01.199 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:01.214 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/osd.5/config 2024-02-13T00:47:01.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:01.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:01.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:01.560 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:01.589 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:01.687 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:01.690 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:00.432294+0000 mgr.y (mgr.14544) 388 : cluster [DBG] pgmap v640: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.9 KiB/s rd, 594 B/s wr, 4 op/s 2024-02-13T00:47:01.690 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:00.432294+0000 mgr.y (mgr.14544) 388 : cluster [DBG] pgmap v640: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.9 KiB/s rd, 594 B/s wr, 4 op/s 2024-02-13T00:47:01.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:01.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:01.792 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:00.432294+0000 mgr.y (mgr.14544) 388 : cluster [DBG] pgmap v640: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.9 KiB/s rd, 594 B/s wr, 4 op/s 2024-02-13T00:47:01.820 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:01.893 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:01.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:01.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:02.023 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:02.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.911+0000 7fd6fd714700 1 -- 172.21.15.162:0/3913208640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6f80fd550 msgr2=0x7fd6f80f73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.911+0000 7fd6fd714700 1 --2- 172.21.15.162:0/3913208640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6f80fd550 0x7fd6f80f73c0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 rx=0x7fd6ec0047b0 tx=0x7fd6ec009cb0).stop 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.911+0000 7fd6fd714700 1 -- 172.21.15.162:0/3913208640 shutdown_connections 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.911+0000 7fd6fd714700 1 --2- 172.21.15.162:0/3913208640 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6f80f7aa0 0x7fd6f81039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.911+0000 7fd6fd714700 1 --2- 172.21.15.162:0/3913208640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6f80fcbf0 0x7fd6f80fcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.911+0000 7fd6fd714700 1 --2- 172.21.15.162:0/3913208640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6f80fd550 0x7fd6f80f73c0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.911+0000 7fd6fd714700 1 -- 172.21.15.162:0/3913208640 >> 172.21.15.162:0/3913208640 conn(0x7fd6f8070100 msgr2=0x7fd6f80f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.911+0000 7fd6fd714700 1 -- 172.21.15.162:0/3913208640 shutdown_connections 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.911+0000 7fd6fd714700 1 -- 172.21.15.162:0/3913208640 wait complete. 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.912+0000 7fd6fd714700 1 librados: starting msgr at 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.912+0000 7fd6fd714700 1 librados: starting objecter 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.912+0000 7fd6fd714700 1 Processor -- start 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.912+0000 7fd6fd714700 1 -- start start 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.912+0000 7fd6fd714700 1 librados: setting wanted keys 2024-02-13T00:47:02.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.912+0000 7fd6fd714700 1 librados: calling monclient init 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.912+0000 7fd6fd714700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6f80f7aa0 0x7fd6f810d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.912+0000 7fd6fd714700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6f80fcbf0 0x7fd6f810d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.912+0000 7fd6fd714700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6f80fd550 0x7fd6f8107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6fd714700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f8107780 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6fd714700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f81078f0 con 0x7fd6f80f7aa0 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6fd714700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f8107ab0 con 0x7fd6f80fd550 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f67fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6f80fcbf0 0x7fd6f810d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f77fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6f80fd550 0x7fd6f8107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f67fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6f80fcbf0 0x7fd6f810d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38190/0 (socket says 172.21.15.162:38190) 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f67fc700 1 -- 172.21.15.162:0/474951449 learned_addr learned my addr 172.21.15.162:0/474951449 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f77fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6f80fd550 0x7fd6f8107290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:58276/0 (socket says 172.21.15.162:58276) 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f6ffd700 1 --2- 172.21.15.162:0/474951449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6f80f7aa0 0x7fd6f810d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f67fc700 1 -- 172.21.15.162:0/474951449 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6f80fd550 msgr2=0x7fd6f8107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:02.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f67fc700 1 --2- 172.21.15.162:0/474951449 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6f80fd550 0x7fd6f8107290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f67fc700 1 -- 172.21.15.162:0/474951449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6f80f7aa0 msgr2=0x7fd6f810d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f67fc700 1 --2- 172.21.15.162:0/474951449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6f80f7aa0 0x7fd6f810d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f67fc700 1 -- 172.21.15.162:0/474951449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f00097c0 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f77fe700 1 --2- 172.21.15.162:0/474951449 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6f80fd550 0x7fd6f8107290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6f67fc700 1 --2- 172.21.15.162:0/474951449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6f80fcbf0 0x7fd6f810d5e0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 rx=0x7fd6ec013760 tx=0x7fd6ec009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6dffff700 10 client.?.objecter ms_handle_connect 0x7fd6f80fcbf0 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6dffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6dffff700 1 -- 172.21.15.162:0/474951449 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec018070 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6fd714700 1 -- 172.21.15.162:0/474951449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ec005040 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6dffff700 1 -- 172.21.15.162:0/474951449 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6ec013da0 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6dffff700 1 -- 172.21.15.162:0/474951449 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec01dd50 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6fd714700 10 client.27399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6fd714700 1 -- 172.21.15.162:0/474951449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f8107f50 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.913+0000 7fd6fd714700 1 librados: init done 2024-02-13T00:47:02.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.914+0000 7fd6fd714700 1 -- 172.21.15.162:0/474951449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f8108140 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.915+0000 7fd6dffff700 1 -- 172.21.15.162:0/474951449 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fd6ec01d420 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.915+0000 7fd6dffff700 1 --2- 172.21.15.162:0/474951449 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd6e0062c90 0x7fd6e0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:02.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.915+0000 7fd6dffff700 1 -- 172.21.15.162:0/474951449 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(437..437 src has 52..437) v4 ==== 7995+0+0 (secure 0 0 0) 0x7fd6ec07e6e0 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.915+0000 7fd6dffff700 10 client.27399.objecter ms_dispatch 0x7fd6f8001550 osd_map(437..437 src has 52..437) v4 2024-02-13T00:47:02.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.915+0000 7fd6dffff700 3 client.27399.objecter handle_osd_map got epochs [437,437] > 0 2024-02-13T00:47:02.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.915+0000 7fd6dffff700 3 client.27399.objecter handle_osd_map decoding full epoch 437 2024-02-13T00:47:02.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.915+0000 7fd6f6ffd700 1 --2- 172.21.15.162:0/474951449 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd6e0062c90 0x7fd6e0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:02.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.915+0000 7fd6dffff700 20 client.27399.objecter dump_active .. 0 homeless 2024-02-13T00:47:02.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.915+0000 7fd6dffff700 10 client.27399.objecter ms_handle_connect 0x7fd6e0062c90 2024-02-13T00:47:02.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.915+0000 7fd6f6ffd700 1 --2- 172.21.15.162:0/474951449 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd6e0062c90 0x7fd6e0065150 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 rx=0x7fd6f8060af0 tx=0x7fd6e800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:02.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:01.920+0000 7fd6dffff700 1 -- 172.21.15.162:0/474951449 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd6ec051bd0 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.107+0000 7fd6fd714700 1 -- 172.21.15.162:0/474951449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fd6f8106130 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.377 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:02.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:02.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:02.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:02.508 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:02.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.576+0000 7fd6dffff700 1 -- 172.21.15.162:0/474951449 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v438) v1 ==== 101+0+0 (secure 0 0 0) 0x7fd6ec0517c0 con 0x7fd6f80fcbf0 2024-02-13T00:47:02.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 10 librados: watch_flush enter 2024-02-13T00:47:02.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 10 librados: watch_flush exit 2024-02-13T00:47:02.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 20 client.27399.objecter shutdown clearing up homeless session... 2024-02-13T00:47:02.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 10 client.27399.objecter successfully canceled tick 2024-02-13T00:47:02.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 1 -- 172.21.15.162:0/474951449 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd6e0062c90 msgr2=0x7fd6e0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:02.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 1 --2- 172.21.15.162:0/474951449 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd6e0062c90 0x7fd6e0065150 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 rx=0x7fd6f8060af0 tx=0x7fd6e800b040).stop 2024-02-13T00:47:02.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 1 -- 172.21.15.162:0/474951449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6f80fcbf0 msgr2=0x7fd6f810d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:02.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 1 --2- 172.21.15.162:0/474951449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6f80fcbf0 0x7fd6f810d5e0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 rx=0x7fd6ec013760 tx=0x7fd6ec009cb0).stop 2024-02-13T00:47:02.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 1 -- 172.21.15.162:0/474951449 shutdown_connections 2024-02-13T00:47:02.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 1 --2- 172.21.15.162:0/474951449 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd6e0062c90 0x7fd6e0065150 unknown :-1 s=CLOSED pgs=2912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:02.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 1 --2- 172.21.15.162:0/474951449 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd6f80fd550 0x7fd6f8107290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:02.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 1 --2- 172.21.15.162:0/474951449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6f80f7aa0 0x7fd6f810d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:02.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 1 --2- 172.21.15.162:0/474951449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6f80fcbf0 0x7fd6f810d5e0 secure :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 rx=0x7fd6ec013760 tx=0x7fd6ec009cb0).stop 2024-02-13T00:47:02.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.579+0000 7fd6fd714700 1 -- 172.21.15.162:0/474951449 >> 172.21.15.162:0/474951449 conn(0x7fd6f8070100 msgr2=0x7fd6f80fa8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:02.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.581+0000 7fd6fd714700 1 -- 172.21.15.162:0/474951449 shutdown_connections 2024-02-13T00:47:02.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.581+0000 7fd6fd714700 1 -- 172.21.15.162:0/474951449 wait complete. 2024-02-13T00:47:02.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:02.581+0000 7fd6fd714700 1 librados: shutdown 2024-02-13T00:47:02.589 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:47:02.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:02.108263+0000 mon.a (mon.0) 2566 : audit [INF] from='client.? 172.21.15.162:0/474951449' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:47:02.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:02.549970+0000 mon.b (mon.1) 581 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:02.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:02.550415+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:02.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:02.550831+0000 mon.b (mon.1) 582 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-63"}]: dispatch 2024-02-13T00:47:02.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:02.551137+0000 mon.a (mon.0) 2568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-63"}]: dispatch 2024-02-13T00:47:02.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:02.108263+0000 mon.a (mon.0) 2566 : audit [INF] from='client.? 172.21.15.162:0/474951449' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:47:02.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:02.549970+0000 mon.b (mon.1) 581 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:02.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:02.550415+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:02.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:02.550831+0000 mon.b (mon.1) 582 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-63"}]: dispatch 2024-02-13T00:47:02.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:02.551137+0000 mon.a (mon.0) 2568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-63"}]: dispatch 2024-02-13T00:47:02.921 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:02.108263+0000 mon.a (mon.0) 2566 : audit [INF] from='client.? 172.21.15.162:0/474951449' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:47:02.921 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:02.549970+0000 mon.b (mon.1) 581 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:02.921 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:02.550415+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:02.922 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:02.550831+0000 mon.b (mon.1) 582 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-63"}]: dispatch 2024-02-13T00:47:02.922 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:02.551137+0000 mon.a (mon.0) 2568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-63"}]: dispatch 2024-02-13T00:47:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.426+0000 7f77b5bf0700 1 -- 172.21.15.162:0/3750722999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77b00fd490 msgr2=0x7f77b00f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.426+0000 7f77b5bf0700 1 --2- 172.21.15.162:0/3750722999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77b00fd490 0x7f77b00f6cc0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 rx=0x7f77a4004870 tx=0x7f77a4009cb0).stop 2024-02-13T00:47:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.426+0000 7f77b5bf0700 1 -- 172.21.15.162:0/3750722999 shutdown_connections 2024-02-13T00:47:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.426+0000 7f77b5bf0700 1 --2- 172.21.15.162:0/3750722999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77b00f73a0 0x7f77b01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.426+0000 7f77b5bf0700 1 --2- 172.21.15.162:0/3750722999 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77b00fcb30 0x7f77b00fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.426+0000 7f77b5bf0700 1 --2- 172.21.15.162:0/3750722999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77b00fd490 0x7f77b00f6cc0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.426+0000 7f77b5bf0700 1 -- 172.21.15.162:0/3750722999 >> 172.21.15.162:0/3750722999 conn(0x7f77b006eb90 msgr2=0x7f77b00f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.426+0000 7f77b5bf0700 1 -- 172.21.15.162:0/3750722999 shutdown_connections 2024-02-13T00:47:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.426+0000 7f77b5bf0700 1 -- 172.21.15.162:0/3750722999 wait complete. 2024-02-13T00:47:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77b5bf0700 1 librados: starting msgr at 2024-02-13T00:47:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77b5bf0700 1 librados: starting objecter 2024-02-13T00:47:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77b5bf0700 1 Processor -- start 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77b5bf0700 1 -- start start 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77b5bf0700 1 librados: setting wanted keys 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77b5bf0700 1 librados: calling monclient init 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77b5bf0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77b00f73a0 0x7f77b010d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77b5bf0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77b00fcb30 0x7f77b010d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77b5bf0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77b00fd490 0x7f77b0107260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77b5bf0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b0107750 con 0x7f77b00fd490 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77b5bf0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b01078c0 con 0x7f77b00f73a0 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77b5bf0700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b0107a80 con 0x7f77b00fcb30 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77aeffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77b00fcb30 0x7f77b010d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77aeffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77b00fcb30 0x7f77b010d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:58302/0 (socket says 172.21.15.162:58302) 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.427+0000 7f77aeffd700 1 -- 172.21.15.162:0/2710312432 learned_addr learned my addr 172.21.15.162:0/2710312432 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77affff700 1 --2- 172.21.15.162:0/2710312432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77b00fd490 0x7f77b0107260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77af7fe700 1 --2- 172.21.15.162:0/2710312432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77b00f73a0 0x7f77b010d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77affff700 1 -- 172.21.15.162:0/2710312432 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77b00fcb30 msgr2=0x7f77b010d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77affff700 1 --2- 172.21.15.162:0/2710312432 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77b00fcb30 0x7f77b010d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77affff700 1 -- 172.21.15.162:0/2710312432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77b00f73a0 msgr2=0x7f77b010d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77affff700 1 --2- 172.21.15.162:0/2710312432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77b00f73a0 0x7f77b010d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77affff700 1 -- 172.21.15.162:0/2710312432 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a0005040 con 0x7f77b00fd490 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77aeffd700 1 --2- 172.21.15.162:0/2710312432 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77b00fcb30 0x7f77b010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77affff700 1 --2- 172.21.15.162:0/2710312432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77b00fd490 0x7f77b0107260 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 rx=0x7f77a800d940 tx=0x7f77a8004e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77acff9700 10 client.?.objecter ms_handle_connect 0x7f77b00fd490 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77acff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a800b4f0 con 0x7f77b00fd490 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77a800f040 con 0x7f77b00fd490 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a4005040 con 0x7f77b00fd490 2024-02-13T00:47:03.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a8013970 con 0x7f77b00fd490 2024-02-13T00:47:03.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77b5bf0700 10 client.27405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:03.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b01080c0 con 0x7f77b00fd490 2024-02-13T00:47:03.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.428+0000 7f77b5bf0700 1 librados: init done 2024-02-13T00:47:03.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.429+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f77b0105fe0 con 0x7f77b00fd490 2024-02-13T00:47:03.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.429+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f77a800b650 con 0x7f77b00fd490 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77acff9700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7798062c90 0x7f7798065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(438..438 src has 52..438) v4 ==== 7995+0+0 (secure 0 0 0) 0x7f77a807cc50 con 0x7f77b00fd490 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 osd_map(438..438 src has 52..438) v4 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77acff9700 3 client.27405.objecter handle_osd_map got epochs [438,438] > 0 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77af7fe700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7798062c90 0x7f7798065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77acff9700 3 client.27405.objecter handle_osd_map decoding full epoch 438 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77acff9700 20 client.27405.objecter dump_active .. 0 homeless 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v438) v1 ==== 54+0+16 (secure 0 0 0) 0x7f77a807f320 con 0x7f77b00fd490 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77acff9700 10 client.27405.objecter ms_handle_connect 0x7f7798062c90 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77af7fe700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7798062c90 0x7f7798065150 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 rx=0x7f77a0009350 tx=0x7f77a000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 1 2024-02-13T00:47:03.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77b5bf0700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6801/2103743338 conn(0x7f77b0028590 0x7f77b00fd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b01086f0 osd=0 3 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b01086f0 osd=0 3 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b01086f0 osd=0 3 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 1 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b01086f0 osd=0 3 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 0 1 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b01086f0 osd=0 4 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77b5bf0700 10 client.27405.objecter _send_command 1 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77b00feee0 con 0x7f77b0028590 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.430+0000 7f77aeffd700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6801/2103743338 conn(0x7f77b0028590 0x7f77b00fd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.431+0000 7f77aeffd700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6801/2103743338 conn(0x7f77b0028590 0x7f77b00fd870 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.431+0000 7f77acff9700 10 client.27405.objecter ms_handle_connect 0x7f77b0028590 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.431+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77b00feee0 con 0x7f77b0028590 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.431+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:47:03.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.431+0000 7f77acff9700 10 client.27405.objecter _finish_command 1 = system:0 2024-02-13T00:47:03.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.431+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b01086f0 osd=0 4 2024-02-13T00:47:03.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.431+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 0 1 2024-02-13T00:47:03.449 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.446+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:03.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.446+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 2 2024-02-13T00:47:03.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.446+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b01086f0 osd=0 3 2024-02-13T00:47:03.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.446+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b01086f0 osd=0 3 2024-02-13T00:47:03.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.446+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b01086f0 osd=0 3 2024-02-13T00:47:03.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.446+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 2 2024-02-13T00:47:03.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.446+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b01086f0 osd=0 3 2024-02-13T00:47:03.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.446+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 0 2 2024-02-13T00:47:03.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.446+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b01086f0 osd=0 4 2024-02-13T00:47:03.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.446+0000 7f77b5bf0700 10 client.27405.objecter _send_command 2 2024-02-13T00:47:03.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.446+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f77b005a0e0 con 0x7f77b0028590 2024-02-13T00:47:03.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.447+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f77b005a0e0 con 0x7f77b0028590 2024-02-13T00:47:03.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.447+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:03.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.447+0000 7f77acff9700 10 client.27405.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.447+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b01086f0 osd=0 4 2024-02-13T00:47:03.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.447+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 0 2 2024-02-13T00:47:03.451 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:47:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 3 2024-02-13T00:47:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77b5bf0700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6805/150599780 conn(0x7f77b0059570 0x7f77b0117f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0059400 osd=1 3 2024-02-13T00:47:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0059400 osd=1 3 2024-02-13T00:47:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0059400 osd=1 3 2024-02-13T00:47:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 3 2024-02-13T00:47:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b0059400 osd=1 3 2024-02-13T00:47:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 1 3 2024-02-13T00:47:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0059400 osd=1 4 2024-02-13T00:47:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77b5bf0700 10 client.27405.objecter _send_command 3 2024-02-13T00:47:03.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77b00280f0 con 0x7f77b0059570 2024-02-13T00:47:03.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77af7fe700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6805/150599780 conn(0x7f77b0059570 0x7f77b0117f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:03.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77acff9700 10 client.27405.objecter ms_handle_connect 0x7f77b0059570 2024-02-13T00:47:03.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.448+0000 7f77af7fe700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6805/150599780 conn(0x7f77b0059570 0x7f77b0117f80 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:03.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.449+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77b00280f0 con 0x7f77b0059570 2024-02-13T00:47:03.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.449+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:47:03.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.449+0000 7f77acff9700 10 client.27405.objecter _finish_command 3 = system:0 2024-02-13T00:47:03.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.449+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b0059400 osd=1 4 2024-02-13T00:47:03.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.449+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 1 3 2024-02-13T00:47:03.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.463+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:03.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.463+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 4 2024-02-13T00:47:03.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.463+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0059400 osd=1 3 2024-02-13T00:47:03.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.463+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0059400 osd=1 3 2024-02-13T00:47:03.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.463+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0059400 osd=1 3 2024-02-13T00:47:03.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.463+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 4 2024-02-13T00:47:03.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.463+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b0059400 osd=1 3 2024-02-13T00:47:03.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.463+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 1 4 2024-02-13T00:47:03.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.463+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0059400 osd=1 4 2024-02-13T00:47:03.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.463+0000 7f77b5bf0700 10 client.27405.objecter _send_command 4 2024-02-13T00:47:03.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.463+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f77b0060e60 con 0x7f77b0059570 2024-02-13T00:47:03.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f77b0060e60 con 0x7f77b0059570 2024-02-13T00:47:03.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:03.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77acff9700 10 client.27405.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b0059400 osd=1 4 2024-02-13T00:47:03.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 1 4 2024-02-13T00:47:03.469 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:47:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 5 2024-02-13T00:47:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77b5bf0700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6809/3689719376 conn(0x7f77b00277c0 0x7f77b011c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0027650 osd=2 3 2024-02-13T00:47:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0027650 osd=2 3 2024-02-13T00:47:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0027650 osd=2 3 2024-02-13T00:47:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 5 2024-02-13T00:47:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b0027650 osd=2 3 2024-02-13T00:47:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 2 5 2024-02-13T00:47:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0027650 osd=2 4 2024-02-13T00:47:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77b5bf0700 10 client.27405.objecter _send_command 5 2024-02-13T00:47:03.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77b0061d10 con 0x7f77b00277c0 2024-02-13T00:47:03.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77aeffd700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6809/3689719376 conn(0x7f77b00277c0 0x7f77b011c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:03.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77acff9700 10 client.27405.objecter ms_handle_connect 0x7f77b00277c0 2024-02-13T00:47:03.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.465+0000 7f77aeffd700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6809/3689719376 conn(0x7f77b00277c0 0x7f77b011c690 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:03.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.466+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77b0061d10 con 0x7f77b00277c0 2024-02-13T00:47:03.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.466+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:47:03.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.466+0000 7f77acff9700 10 client.27405.objecter _finish_command 5 = system:0 2024-02-13T00:47:03.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.466+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b0027650 osd=2 4 2024-02-13T00:47:03.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.466+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 2 5 2024-02-13T00:47:03.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.480+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:03.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.480+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 6 2024-02-13T00:47:03.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.480+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0027650 osd=2 3 2024-02-13T00:47:03.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.480+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0027650 osd=2 3 2024-02-13T00:47:03.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.480+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0027650 osd=2 3 2024-02-13T00:47:03.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.480+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 6 2024-02-13T00:47:03.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.480+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b0027650 osd=2 3 2024-02-13T00:47:03.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.480+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 2 6 2024-02-13T00:47:03.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.480+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0027650 osd=2 4 2024-02-13T00:47:03.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.480+0000 7f77b5bf0700 10 client.27405.objecter _send_command 6 2024-02-13T00:47:03.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.480+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f77b0061270 con 0x7f77b00277c0 2024-02-13T00:47:03.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f77b0061270 con 0x7f77b00277c0 2024-02-13T00:47:03.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:03.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77acff9700 10 client.27405.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b0027650 osd=2 4 2024-02-13T00:47:03.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 2 6 2024-02-13T00:47:03.486 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:47:03.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 7 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77b5bf0700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6813/604971011 conn(0x7f77b00fea40 0x7f77b011fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0061680 osd=3 3 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0061680 osd=3 3 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0061680 osd=3 3 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 7 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b0061680 osd=3 3 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 3 7 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0061680 osd=3 4 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77b5bf0700 10 client.27405.objecter _send_command 7 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77b0122960 con 0x7f77b00fea40 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77affff700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6813/604971011 conn(0x7f77b00fea40 0x7f77b011fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77acff9700 10 client.27405.objecter ms_handle_connect 0x7f77b00fea40 2024-02-13T00:47:03.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.482+0000 7f77affff700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.162:6813/604971011 conn(0x7f77b00fea40 0x7f77b011fd10 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:03.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.483+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77b0122960 con 0x7f77b00fea40 2024-02-13T00:47:03.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.483+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:47:03.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.483+0000 7f77acff9700 10 client.27405.objecter _finish_command 7 = system:0 2024-02-13T00:47:03.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.483+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b0061680 osd=3 4 2024-02-13T00:47:03.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.483+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 3 7 2024-02-13T00:47:03.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.498+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:03.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.498+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 8 2024-02-13T00:47:03.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.498+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0061680 osd=3 3 2024-02-13T00:47:03.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.498+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0061680 osd=3 3 2024-02-13T00:47:03.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.498+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0061680 osd=3 3 2024-02-13T00:47:03.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.498+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 8 2024-02-13T00:47:03.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.498+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b0061680 osd=3 3 2024-02-13T00:47:03.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.498+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 3 8 2024-02-13T00:47:03.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.498+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0061680 osd=3 4 2024-02-13T00:47:03.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.498+0000 7f77b5bf0700 10 client.27405.objecter _send_command 8 2024-02-13T00:47:03.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.498+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f77b00fdfa0 con 0x7f77b00fea40 2024-02-13T00:47:03.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f77b00fdfa0 con 0x7f77b00fea40 2024-02-13T00:47:03.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:03.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77acff9700 10 client.27405.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b0061680 osd=3 4 2024-02-13T00:47:03.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 3 8 2024-02-13T00:47:03.503 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:47:03.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:03.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 9 2024-02-13T00:47:03.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77b5bf0700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.181:6800/3344263388 conn(0x7f77b011d860 0x7f77b0121380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:03.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b00fdfa0 osd=4 3 2024-02-13T00:47:03.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b00fdfa0 osd=4 3 2024-02-13T00:47:03.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b00fdfa0 osd=4 3 2024-02-13T00:47:03.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 9 2024-02-13T00:47:03.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b00fdfa0 osd=4 3 2024-02-13T00:47:03.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 4 9 2024-02-13T00:47:03.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b00fdfa0 osd=4 4 2024-02-13T00:47:03.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77b5bf0700 10 client.27405.objecter _send_command 9 2024-02-13T00:47:03.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77b00fe250 con 0x7f77b011d860 2024-02-13T00:47:03.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77af7fe700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.181:6800/3344263388 conn(0x7f77b011d860 0x7f77b0121380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:03.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77acff9700 10 client.27405.objecter ms_handle_connect 0x7f77b011d860 2024-02-13T00:47:03.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.500+0000 7f77af7fe700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.181:6800/3344263388 conn(0x7f77b011d860 0x7f77b0121380 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:03.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.501+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77b00fe250 con 0x7f77b011d860 2024-02-13T00:47:03.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.501+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:47:03.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.501+0000 7f77acff9700 10 client.27405.objecter _finish_command 9 = system:0 2024-02-13T00:47:03.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.501+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b00fdfa0 osd=4 4 2024-02-13T00:47:03.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.501+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 4 9 2024-02-13T00:47:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.516+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.516+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 10 2024-02-13T00:47:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.516+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b00fdfa0 osd=4 3 2024-02-13T00:47:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.516+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b00fdfa0 osd=4 3 2024-02-13T00:47:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.516+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b00fdfa0 osd=4 3 2024-02-13T00:47:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.516+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 10 2024-02-13T00:47:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.516+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b00fdfa0 osd=4 3 2024-02-13T00:47:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.516+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 4 10 2024-02-13T00:47:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.516+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b00fdfa0 osd=4 4 2024-02-13T00:47:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.516+0000 7f77b5bf0700 10 client.27405.objecter _send_command 10 2024-02-13T00:47:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.516+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f77b00fe250 con 0x7f77b011d860 2024-02-13T00:47:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f77b00fe250 con 0x7f77b011d860 2024-02-13T00:47:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77acff9700 10 client.27405.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b00fdfa0 osd=4 4 2024-02-13T00:47:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 4 10 2024-02-13T00:47:03.524 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:47:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 11 2024-02-13T00:47:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 20 client.27405.objecter _get_session osd=-1 returning homeless 2024-02-13T00:47:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 10 client.27405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=439}) v3 -- 0x7f77b0121870 con 0x7f77b00fd490 2024-02-13T00:47:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f77b0121b70 con 0x7f77b00fd490 2024-02-13T00:47:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_get_version_reply(handle=1 version=438) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77a807f320 con 0x7f77b00fd490 2024-02-13T00:47:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77adffb700 10 client.27405.objecter _check_command_map_dne tid 11 current 438 map_dne_bound 438 2024-02-13T00:47:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77adffb700 10 client.27405.objecter _finish_command 11 = osd:6 osd down 2024-02-13T00:47:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77adffb700 15 client.27405.objecter _session_command_op_remove -1 11 2024-02-13T00:47:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 12 2024-02-13T00:47:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.181:6808/1506640341 conn(0x7f77b0124b60 0x7f77b0126f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b01224b0 osd=6 3 2024-02-13T00:47:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b01224b0 osd=6 3 2024-02-13T00:47:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b01224b0 osd=6 3 2024-02-13T00:47:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 12 2024-02-13T00:47:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b01224b0 osd=6 3 2024-02-13T00:47:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 6 12 2024-02-13T00:47:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b01224b0 osd=6 4 2024-02-13T00:47:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 10 client.27405.objecter _send_command 12 2024-02-13T00:47:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.519+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.181:6808/1506640341 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77b0122710 con 0x7f77b0124b60 2024-02-13T00:47:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.520+0000 7f77aeffd700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.181:6808/1506640341 conn(0x7f77b0124b60 0x7f77b0126f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.520+0000 7f77acff9700 10 client.27405.objecter ms_handle_connect 0x7f77b0124b60 2024-02-13T00:47:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.520+0000 7f77aeffd700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.181:6808/1506640341 conn(0x7f77b0124b60 0x7f77b0126f50 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.521+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77b0122710 con 0x7f77b0124b60 2024-02-13T00:47:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.521+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 12: 0 ) v1 2024-02-13T00:47:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.521+0000 7f77acff9700 10 client.27405.objecter _finish_command 12 = system:0 2024-02-13T00:47:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.521+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b01224b0 osd=6 4 2024-02-13T00:47:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.521+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 6 12 2024-02-13T00:47:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.535+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.535+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 13 2024-02-13T00:47:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.535+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b01224b0 osd=6 3 2024-02-13T00:47:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.535+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b01224b0 osd=6 3 2024-02-13T00:47:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.535+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b01224b0 osd=6 3 2024-02-13T00:47:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.535+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 13 2024-02-13T00:47:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.535+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b01224b0 osd=6 3 2024-02-13T00:47:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.535+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 6 13 2024-02-13T00:47:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.535+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b01224b0 osd=6 4 2024-02-13T00:47:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.535+0000 7f77b5bf0700 10 client.27405.objecter _send_command 13 2024-02-13T00:47:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.535+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f77b0122710 con 0x7f77b0124b60 2024-02-13T00:47:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f77b0122710 con 0x7f77b0124b60 2024-02-13T00:47:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77acff9700 10 client.27405.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b01224b0 osd=6 4 2024-02-13T00:47:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 6 13 2024-02-13T00:47:03.541 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:47:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 14 2024-02-13T00:47:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77b5bf0700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.181:6812/1964192772 conn(0x7f77b01279b0 0x7f77b0129e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0122710 osd=7 3 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0122710 osd=7 3 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0122710 osd=7 3 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 14 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b0122710 osd=7 3 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 7 14 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0122710 osd=7 4 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77b5bf0700 10 client.27405.objecter _send_command 14 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.537+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.181:6812/1964192772 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77b012cba0 con 0x7f77b01279b0 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.538+0000 7f77affff700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.181:6812/1964192772 conn(0x7f77b01279b0 0x7f77b0129e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.538+0000 7f77acff9700 10 client.27405.objecter ms_handle_connect 0x7f77b01279b0 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.538+0000 7f77affff700 1 --2- 172.21.15.162:0/2710312432 >> v2:172.21.15.181:6812/1964192772 conn(0x7f77b01279b0 0x7f77b0129e00 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.539+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77b012cba0 con 0x7f77b01279b0 2024-02-13T00:47:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.539+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 14: 0 ) v1 2024-02-13T00:47:03.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.539+0000 7f77acff9700 10 client.27405.objecter _finish_command 14 = system:0 2024-02-13T00:47:03.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.539+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b0122710 osd=7 4 2024-02-13T00:47:03.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.539+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 7 14 2024-02-13T00:47:03.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.553+0000 7f77b5bf0700 10 client.27405.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:03.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.553+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign -1 15 2024-02-13T00:47:03.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.553+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0122710 osd=7 3 2024-02-13T00:47:03.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.553+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0122710 osd=7 3 2024-02-13T00:47:03.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.553+0000 7f77b5bf0700 20 client.27405.objecter _get_session s=0x7f77b0122710 osd=7 3 2024-02-13T00:47:03.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.553+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_remove -1 15 2024-02-13T00:47:03.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.553+0000 7f77b5bf0700 20 client.27405.objecter get_session s=0x7f77b0122710 osd=7 3 2024-02-13T00:47:03.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.553+0000 7f77b5bf0700 15 client.27405.objecter _session_command_op_assign 7 15 2024-02-13T00:47:03.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.553+0000 7f77b5bf0700 20 client.27405.objecter put_session s=0x7f77b0122710 osd=7 4 2024-02-13T00:47:03.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.553+0000 7f77b5bf0700 10 client.27405.objecter _send_command 15 2024-02-13T00:47:03.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.553+0000 7f77b5bf0700 1 -- 172.21.15.162:0/2710312432 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f77b012cba0 con 0x7f77b01279b0 2024-02-13T00:47:03.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.555+0000 7f77acff9700 1 -- 172.21.15.162:0/2710312432 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f77b012cba0 con 0x7f77b01279b0 2024-02-13T00:47:03.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.555+0000 7f77acff9700 10 client.27405.objecter ms_dispatch 0x7f77b0001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:03.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.555+0000 7f77acff9700 10 client.27405.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.555+0000 7f77acff9700 20 client.27405.objecter put_session s=0x7f77b0122710 osd=7 4 2024-02-13T00:47:03.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:03.555+0000 7f77acff9700 15 client.27405.objecter _session_command_op_remove 7 15 2024-02-13T00:47:03.559 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:47:03.559 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.559 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.559 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.559 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.559 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.560 INFO:teuthology.orchestra.run.smithi162.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-02-13T00:47:03.560 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.560 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:47:03.571 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:02.432887+0000 mgr.y (mgr.14544) 389 : cluster [DBG] pgmap v641: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-13T00:47:03.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:02.564975+0000 mon.a (mon.0) 2569 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:47:03.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:02.575482+0000 mon.a (mon.0) 2570 : audit [INF] from='client.? 172.21.15.162:0/474951449' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:47:03.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:02.575548+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e438: 8 total, 7 up, 4 in 2024-02-13T00:47:03.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:02.786097+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:47:03.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:02.789177+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:47:03.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:02.945746+0000 osd.1 (osd.1) 187 : cluster [DBG] 5.0 deep-scrub starts 2024-02-13T00:47:03.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:02.949402+0000 osd.1 (osd.1) 188 : cluster [DBG] 5.0 deep-scrub ok 2024-02-13T00:47:03.572 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:03.430744+0000 mon.a (mon.0) 2572 : audit [DBG] from='client.? 172.21.15.162:0/2710312432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:03.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:02.432887+0000 mgr.y (mgr.14544) 389 : cluster [DBG] pgmap v641: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-13T00:47:03.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:02.564975+0000 mon.a (mon.0) 2569 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:47:03.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:02.575482+0000 mon.a (mon.0) 2570 : audit [INF] from='client.? 172.21.15.162:0/474951449' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:47:03.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:02.575548+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e438: 8 total, 7 up, 4 in 2024-02-13T00:47:03.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:02.786097+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:47:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:02.789177+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:47:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:02.945746+0000 osd.1 (osd.1) 187 : cluster [DBG] 5.0 deep-scrub starts 2024-02-13T00:47:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:02.949402+0000 osd.1 (osd.1) 188 : cluster [DBG] 5.0 deep-scrub ok 2024-02-13T00:47:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:03.430744+0000 mon.a (mon.0) 2572 : audit [DBG] from='client.? 172.21.15.162:0/2710312432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:04.045 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:47:04.045 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:47:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:02.432887+0000 mgr.y (mgr.14544) 389 : cluster [DBG] pgmap v641: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-13T00:47:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:02.564975+0000 mon.a (mon.0) 2569 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:47:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:02.575482+0000 mon.a (mon.0) 2570 : audit [INF] from='client.? 172.21.15.162:0/474951449' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:47:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:02.575548+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e438: 8 total, 7 up, 4 in 2024-02-13T00:47:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:02.786097+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:47:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:02.789177+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:47:04.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:02.945746+0000 osd.1 (osd.1) 187 : cluster [DBG] 5.0 deep-scrub starts 2024-02-13T00:47:04.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:02.949402+0000 osd.1 (osd.1) 188 : cluster [DBG] 5.0 deep-scrub ok 2024-02-13T00:47:04.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:03.430744+0000 mon.a (mon.0) 2572 : audit [DBG] from='client.? 172.21.15.162:0/2710312432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:04.476 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:04.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:04.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:04.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:04.604 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:47:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:47:04.345+0000 7f16fd9f6700 -1 mgr get_metadata_python Requested missing service osd.5 2024-02-13T00:47:04.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:03.487525+0000 osd.6 (osd.6) 117 : cluster [DBG] 4.8 deep-scrub starts 2024-02-13T00:47:04.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:03.491643+0000 osd.6 (osd.6) 118 : cluster [DBG] 4.8 deep-scrub ok 2024-02-13T00:47:04.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:03.574144+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e439: 8 total, 7 up, 4 in 2024-02-13T00:47:04.605 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:04.341051+0000 mon.a (mon.0) 2574 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:47:04.605 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:03.487525+0000 osd.6 (osd.6) 117 : cluster [DBG] 4.8 deep-scrub starts 2024-02-13T00:47:04.605 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:03.491643+0000 osd.6 (osd.6) 118 : cluster [DBG] 4.8 deep-scrub ok 2024-02-13T00:47:04.605 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:03.574144+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e439: 8 total, 7 up, 4 in 2024-02-13T00:47:04.606 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:04.341051+0000 mon.a (mon.0) 2574 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:47:04.623 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:04.786 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:03.487525+0000 osd.6 (osd.6) 117 : cluster [DBG] 4.8 deep-scrub starts 2024-02-13T00:47:04.786 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:03.491643+0000 osd.6 (osd.6) 118 : cluster [DBG] 4.8 deep-scrub ok 2024-02-13T00:47:04.786 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:03.574144+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e439: 8 total, 7 up, 4 in 2024-02-13T00:47:04.786 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:04.341051+0000 mon.a (mon.0) 2574 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:47:04.804 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-13T00:47:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-13T00:47:04.960 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-13T00:47:04.973 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:04.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:04.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:05.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:05.077 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:05.106 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:05.127 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:05.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:05.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:05.231 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:05.258 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:05.356 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:05.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:05.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:05.501 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:05.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:04.433343+0000 mgr.y (mgr.14544) 390 : cluster [DBG] pgmap v644: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:05.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:04.490963+0000 osd.6 (osd.6) 119 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:47:05.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:04.493837+0000 osd.6 (osd.6) 120 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:47:05.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:04.586196+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e440: 8 total, 7 up, 4 in 2024-02-13T00:47:05.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:04.590713+0000 mon.b (mon.1) 583 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:05.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:04.598819+0000 mon.a (mon.0) 2576 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:05.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:05.431383+0000 mon.a (mon.0) 2577 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:05.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:05.431489+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e441: 8 total, 7 up, 4 in 2024-02-13T00:47:05.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:05.436226+0000 mon.b (mon.1) 584 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:05.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:04.433343+0000 mgr.y (mgr.14544) 390 : cluster [DBG] pgmap v644: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:05.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:04.490963+0000 osd.6 (osd.6) 119 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:47:05.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:04.493837+0000 osd.6 (osd.6) 120 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:47:05.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:04.586196+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e440: 8 total, 7 up, 4 in 2024-02-13T00:47:05.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:04.590713+0000 mon.b (mon.1) 583 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:05.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:04.598819+0000 mon.a (mon.0) 2576 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:05.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:05.431383+0000 mon.a (mon.0) 2577 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:05.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:05.431489+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e441: 8 total, 7 up, 4 in 2024-02-13T00:47:05.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:05.436226+0000 mon.b (mon.1) 584 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:05.863 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:04.433343+0000 mgr.y (mgr.14544) 390 : cluster [DBG] pgmap v644: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:05.863 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:04.490963+0000 osd.6 (osd.6) 119 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:47:05.863 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:04.493837+0000 osd.6 (osd.6) 120 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:47:05.863 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:04.586196+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e440: 8 total, 7 up, 4 in 2024-02-13T00:47:05.863 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:04.590713+0000 mon.b (mon.1) 583 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:05.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:04.598819+0000 mon.a (mon.0) 2576 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:05.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:05.431383+0000 mon.a (mon.0) 2577 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:05.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:05.431489+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e441: 8 total, 7 up, 4 in 2024-02-13T00:47:05.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:05.436226+0000 mon.b (mon.1) 584 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:05.864 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:47:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:47:06.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:47:06] "GET /metrics HTTP/1.1" 200 175058 "" "Prometheus/2.33.4" 2024-02-13T00:47:06.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:47:06.452 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:06.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:06.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:06.560 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:06.588 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:06.658 INFO:tasks.thrashosds.thrasher:No PGs found for osd.5 2024-02-13T00:47:06.691 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:06.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:05.583844+0000 osd.0 (osd.0) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:06.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:05.584140+0000 osd.0 (osd.0) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:06.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:05.650346+0000 mgr.y (mgr.14544) 391 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:06.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:05.714457+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.3 scrub starts 2024-02-13T00:47:06.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:05.717983+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.3 scrub ok 2024-02-13T00:47:06.693 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:06.460979+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e442: 8 total, 7 up, 4 in 2024-02-13T00:47:06.693 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:06.487993+0000 mon.b (mon.1) 585 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:06.693 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:06.488475+0000 mon.a (mon.0) 2580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:06.693 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:06.488905+0000 mon.b (mon.1) 586 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-65"}]: dispatch 2024-02-13T00:47:06.693 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:06.489202+0000 mon.a (mon.0) 2581 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-65"}]: dispatch 2024-02-13T00:47:06.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:05.583844+0000 osd.0 (osd.0) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:06.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:05.584140+0000 osd.0 (osd.0) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:06.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:05.650346+0000 mgr.y (mgr.14544) 391 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:06.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:05.714457+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.3 scrub starts 2024-02-13T00:47:06.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:05.717983+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.3 scrub ok 2024-02-13T00:47:06.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:06.460979+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e442: 8 total, 7 up, 4 in 2024-02-13T00:47:06.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:06.487993+0000 mon.b (mon.1) 585 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:06.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:06.488475+0000 mon.a (mon.0) 2580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:06.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:06.488905+0000 mon.b (mon.1) 586 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-65"}]: dispatch 2024-02-13T00:47:06.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:06.489202+0000 mon.a (mon.0) 2581 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-65"}]: dispatch 2024-02-13T00:47:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:06.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:06.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:06.825 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:06.844 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:06.846 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:05.583844+0000 osd.0 (osd.0) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:06.846 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:05.584140+0000 osd.0 (osd.0) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:06.846 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:05.650346+0000 mgr.y (mgr.14544) 391 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:06.846 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:05.714457+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.3 scrub starts 2024-02-13T00:47:06.846 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:05.717983+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.3 scrub ok 2024-02-13T00:47:06.846 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:06.460979+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e442: 8 total, 7 up, 4 in 2024-02-13T00:47:06.846 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:06.487993+0000 mon.b (mon.1) 585 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:06.847 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:06.488475+0000 mon.a (mon.0) 2580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:06.847 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:06.488905+0000 mon.b (mon.1) 586 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-65"}]: dispatch 2024-02-13T00:47:06.847 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:06.489202+0000 mon.a (mon.0) 2581 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-65"}]: dispatch 2024-02-13T00:47:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:06.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:06.923 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:47:06.923 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:47:06.933 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:06.949 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:06.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:47:06.976 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:06.994 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:06.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:07.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:07.127 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:07.226 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:07.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:07.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:07.359 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:07.956 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:06.433775+0000 mgr.y (mgr.14544) 392 : cluster [DBG] pgmap v647: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:07.956 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:06.892663+0000 osd.1 (osd.1) 189 : cluster [DBG] 6.b scrub starts 2024-02-13T00:47:07.956 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:06.895608+0000 osd.1 (osd.1) 190 : cluster [DBG] 6.b scrub ok 2024-02-13T00:47:07.957 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:07.450751+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e443: 8 total, 7 up, 4 in 2024-02-13T00:47:07.957 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:06.433775+0000 mgr.y (mgr.14544) 392 : cluster [DBG] pgmap v647: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:07.957 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:06.892663+0000 osd.1 (osd.1) 189 : cluster [DBG] 6.b scrub starts 2024-02-13T00:47:07.957 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:06.895608+0000 osd.1 (osd.1) 190 : cluster [DBG] 6.b scrub ok 2024-02-13T00:47:07.957 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:07.450751+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e443: 8 total, 7 up, 4 in 2024-02-13T00:47:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:06.433775+0000 mgr.y (mgr.14544) 392 : cluster [DBG] pgmap v647: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:06.892663+0000 osd.1 (osd.1) 189 : cluster [DBG] 6.b scrub starts 2024-02-13T00:47:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:06.895608+0000 osd.1 (osd.1) 190 : cluster [DBG] 6.b scrub ok 2024-02-13T00:47:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:07.450751+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e443: 8 total, 7 up, 4 in 2024-02-13T00:47:08.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-13T00:47:08.808 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:08.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:08.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:08.943 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:47:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:47:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:47:08.943 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:07.939140+0000 osd.1 (osd.1) 191 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:47:08.943 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:07.942664+0000 osd.1 (osd.1) 192 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:47:08.943 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:08.453039+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e444: 8 total, 7 up, 4 in 2024-02-13T00:47:08.943 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:08.458270+0000 mon.b (mon.1) 587 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:08.943 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:08.459020+0000 mon.a (mon.0) 2584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:08.946 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:08.967 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:08.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:08.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:09.032 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:07.939140+0000 osd.1 (osd.1) 191 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:47:09.032 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:07.942664+0000 osd.1 (osd.1) 192 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:47:09.032 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:08.453039+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e444: 8 total, 7 up, 4 in 2024-02-13T00:47:09.032 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:08.458270+0000 mon.b (mon.1) 587 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:09.033 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:08.459020+0000 mon.a (mon.0) 2584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:09.033 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:07.939140+0000 osd.1 (osd.1) 191 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:47:09.033 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:07.942664+0000 osd.1 (osd.1) 192 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:47:09.033 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:08.453039+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e444: 8 total, 7 up, 4 in 2024-02-13T00:47:09.033 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:08.458270+0000 mon.b (mon.1) 587 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:09.033 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:08.459020+0000 mon.a (mon.0) 2584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:09.071 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:09.102 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:09.122 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:09.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:09.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:09.255 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:09.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.293+0000 7f17f30b8700 1 -- 172.21.15.162:0/1066754472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17ec102270 msgr2=0x7f17ec104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:09.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.293+0000 7f17f30b8700 1 --2- 172.21.15.162:0/1066754472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17ec102270 0x7f17ec104660 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7f17e4009c30 tx=0x7f17e4004c50).stop 2024-02-13T00:47:09.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.293+0000 7f17f30b8700 1 -- 172.21.15.162:0/1066754472 shutdown_connections 2024-02-13T00:47:09.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.293+0000 7f17f30b8700 1 --2- 172.21.15.162:0/1066754472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17ec102270 0x7f17ec104660 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:09.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.293+0000 7f17f30b8700 1 --2- 172.21.15.162:0/1066754472 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17ec0f8d10 0x7f17ec0f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:09.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.293+0000 7f17f30b8700 1 --2- 172.21.15.162:0/1066754472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17ec0f95e0 0x7f17ec101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:09.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.293+0000 7f17f30b8700 1 -- 172.21.15.162:0/1066754472 >> 172.21.15.162:0/1066754472 conn(0x7f17ec06eb90 msgr2=0x7f17ec0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:09.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 -- 172.21.15.162:0/1066754472 shutdown_connections 2024-02-13T00:47:09.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 -- 172.21.15.162:0/1066754472 wait complete. 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 librados: starting msgr at 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 librados: starting objecter 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 Processor -- start 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 -- start start 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 librados: setting wanted keys 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 librados: calling monclient init 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17ec0f8d10 0x7f17ec10e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17ec0f95e0 0x7f17ec10e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17ec102270 0x7f17ec108290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec108780 con 0x7f17ec102270 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec1088f0 con 0x7f17ec0f8d10 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f30b8700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec108ab0 con 0x7f17ec0f95e0 2024-02-13T00:47:09.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17ebfff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17ec0f95e0 0x7f17ec10e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f1655700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17ec102270 0x7f17ec108290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17ebfff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17ec0f95e0 0x7f17ec10e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:40726/0 (socket says 172.21.15.162:40726) 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17ebfff700 1 -- 172.21.15.162:0/1257944166 learned_addr learned my addr 172.21.15.162:0/1257944166 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.294+0000 7f17f1655700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17ec102270 0x7f17ec108290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46614/0 (socket says 172.21.15.162:46614) 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17f0e54700 1 --2- 172.21.15.162:0/1257944166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17ec0f8d10 0x7f17ec10e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17ebfff700 1 -- 172.21.15.162:0/1257944166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17ec0f8d10 msgr2=0x7f17ec10e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17ebfff700 1 --2- 172.21.15.162:0/1257944166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17ec0f8d10 0x7f17ec10e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17ebfff700 1 -- 172.21.15.162:0/1257944166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17ec102270 msgr2=0x7f17ec108290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17ebfff700 1 --2- 172.21.15.162:0/1257944166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17ec102270 0x7f17ec108290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17ebfff700 1 -- 172.21.15.162:0/1257944166 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc005040 con 0x7f17ec0f95e0 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17f1655700 1 --2- 172.21.15.162:0/1257944166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17ec102270 0x7f17ec108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17ebfff700 1 --2- 172.21.15.162:0/1257944166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17ec0f95e0 0x7f17ec10e5e0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 rx=0x7f17e000e970 tx=0x7f17e000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17e9ffb700 10 client.?.objecter ms_handle_connect 0x7f17ec0f95e0 2024-02-13T00:47:09.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17e9ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17e9ffb700 1 -- 172.21.15.162:0/1257944166 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0004660 con 0x7f17ec0f95e0 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17f30b8700 1 -- 172.21.15.162:0/1257944166 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e40097c0 con 0x7f17ec0f95e0 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17e9ffb700 1 -- 172.21.15.162:0/1257944166 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17e000f040 con 0x7f17ec0f95e0 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17e9ffb700 1 -- 172.21.15.162:0/1257944166 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0014790 con 0x7f17ec0f95e0 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17f30b8700 10 client.37511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17f30b8700 1 -- 172.21.15.162:0/1257944166 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ec114f20 con 0x7f17ec0f95e0 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.295+0000 7f17f30b8700 1 librados: init done 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.296+0000 7f17f30b8700 1 -- 172.21.15.162:0/1257944166 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ec0475e0 con 0x7f17ec0f95e0 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.296+0000 7f17e9ffb700 1 -- 172.21.15.162:0/1257944166 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f17e00090d0 con 0x7f17ec0f95e0 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.297+0000 7f17e9ffb700 1 --2- 172.21.15.162:0/1257944166 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17d4062c90 0x7f17d4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.297+0000 7f17e9ffb700 1 -- 172.21.15.162:0/1257944166 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(444..444 src has 52..444) v4 ==== 7954+0+0 (secure 0 0 0) 0x7f17e007cc20 con 0x7f17ec0f95e0 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.297+0000 7f17e9ffb700 10 client.37511.objecter ms_dispatch 0x7f17ec001550 osd_map(444..444 src has 52..444) v4 2024-02-13T00:47:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.297+0000 7f17e9ffb700 3 client.37511.objecter handle_osd_map got epochs [444,444] > 0 2024-02-13T00:47:09.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.297+0000 7f17e9ffb700 3 client.37511.objecter handle_osd_map decoding full epoch 444 2024-02-13T00:47:09.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.297+0000 7f17f0e54700 1 --2- 172.21.15.162:0/1257944166 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17d4062c90 0x7f17d4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:09.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.297+0000 7f17e9ffb700 20 client.37511.objecter dump_active .. 0 homeless 2024-02-13T00:47:09.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.297+0000 7f17e9ffb700 10 client.37511.objecter ms_handle_connect 0x7f17d4062c90 2024-02-13T00:47:09.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.297+0000 7f17f0e54700 1 --2- 172.21.15.162:0/1257944166 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17d4062c90 0x7f17d4065150 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 rx=0x7f17dc004870 tx=0x7f17dc00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:09.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.301+0000 7f17e9ffb700 1 -- 172.21.15.162:0/1257944166 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f17e004af60 con 0x7f17ec0f95e0 2024-02-13T00:47:09.322 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:09.328 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:09.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:09.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:09.368 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:09.434 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:09.470 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:09.496 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:09.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:09.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:09.508+0000 7f17f30b8700 1 -- 172.21.15.162:0/1257944166 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f17ec108d70 con 0x7f17ec0f95e0 2024-02-13T00:47:09.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:09.631 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:08.434156+0000 mgr.y (mgr.14544) 393 : cluster [DBG] pgmap v650: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:47:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:09.455198+0000 mon.a (mon.0) 2585 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:09.455275+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e445: 8 total, 7 up, 4 in 2024-02-13T00:47:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:09.459805+0000 mon.b (mon.1) 588 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:09.499960+0000 mon.b (mon.1) 589 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:09.500362+0000 mon.a (mon.0) 2587 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:09.500768+0000 mon.b (mon.1) 590 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-67"}]: dispatch 2024-02-13T00:47:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:09.501025+0000 mon.a (mon.0) 2588 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-67"}]: dispatch 2024-02-13T00:47:09.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:09.509026+0000 mon.c (mon.2) 385 : audit [INF] from='client.? 172.21.15.162:0/1257944166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:47:09.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:09.509290+0000 mon.a (mon.0) 2589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:47:09.857 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:08.434156+0000 mgr.y (mgr.14544) 393 : cluster [DBG] pgmap v650: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:47:09.857 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:09.455198+0000 mon.a (mon.0) 2585 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:09.857 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:09.455275+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e445: 8 total, 7 up, 4 in 2024-02-13T00:47:09.857 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:09.459805+0000 mon.b (mon.1) 588 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:09.857 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:09.499960+0000 mon.b (mon.1) 589 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:09.858 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:09.500362+0000 mon.a (mon.0) 2587 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:09.858 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:09.500768+0000 mon.b (mon.1) 590 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-67"}]: dispatch 2024-02-13T00:47:09.858 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:09.501025+0000 mon.a (mon.0) 2588 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-67"}]: dispatch 2024-02-13T00:47:09.858 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:09.509026+0000 mon.c (mon.2) 385 : audit [INF] from='client.? 172.21.15.162:0/1257944166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:47:09.858 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:09.509290+0000 mon.a (mon.0) 2589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:47:09.858 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:08.434156+0000 mgr.y (mgr.14544) 393 : cluster [DBG] pgmap v650: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:47:09.858 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:09.455198+0000 mon.a (mon.0) 2585 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:09.858 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:09.455275+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e445: 8 total, 7 up, 4 in 2024-02-13T00:47:09.858 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:09.459805+0000 mon.b (mon.1) 588 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:09.858 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:09.499960+0000 mon.b (mon.1) 589 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:09.859 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:09.500362+0000 mon.a (mon.0) 2587 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:09.859 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:09.500768+0000 mon.b (mon.1) 590 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-67"}]: dispatch 2024-02-13T00:47:09.859 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:09.501025+0000 mon.a (mon.0) 2588 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-67"}]: dispatch 2024-02-13T00:47:09.859 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:09.509026+0000 mon.c (mon.2) 385 : audit [INF] from='client.? 172.21.15.162:0/1257944166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:47:09.859 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:09.509290+0000 mon.a (mon.0) 2589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:47:10.279 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:10.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:10.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:10.383 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:10.411 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:10.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.512+0000 7f17e9ffb700 1 -- 172.21.15.162:0/1257944166 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v446) v1 ==== 87+0+0 (secure 0 0 0) 0x7f17e0019070 con 0x7f17ec0f95e0 2024-02-13T00:47:10.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.514+0000 7f17f30b8700 10 librados: watch_flush enter 2024-02-13T00:47:10.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.514+0000 7f17f30b8700 10 librados: watch_flush exit 2024-02-13T00:47:10.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.514+0000 7f17f30b8700 20 client.37511.objecter shutdown clearing up homeless session... 2024-02-13T00:47:10.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.514+0000 7f17f30b8700 10 client.37511.objecter successfully canceled tick 2024-02-13T00:47:10.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.515+0000 7f17f30b8700 1 -- 172.21.15.162:0/1257944166 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17d4062c90 msgr2=0x7f17d4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:10.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.515+0000 7f17f30b8700 1 --2- 172.21.15.162:0/1257944166 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17d4062c90 0x7f17d4065150 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 rx=0x7f17dc004870 tx=0x7f17dc00b040).stop 2024-02-13T00:47:10.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.515+0000 7f17f30b8700 1 -- 172.21.15.162:0/1257944166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17ec0f95e0 msgr2=0x7f17ec10e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:10.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.516+0000 7f17f30b8700 1 --2- 172.21.15.162:0/1257944166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17ec0f95e0 0x7f17ec10e5e0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 rx=0x7f17e000e970 tx=0x7f17e000b040).stop 2024-02-13T00:47:10.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.516+0000 7f17f30b8700 1 -- 172.21.15.162:0/1257944166 shutdown_connections 2024-02-13T00:47:10.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.516+0000 7f17f30b8700 1 --2- 172.21.15.162:0/1257944166 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17d4062c90 0x7f17d4065150 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:10.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.516+0000 7f17f30b8700 1 --2- 172.21.15.162:0/1257944166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17ec102270 0x7f17ec108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:10.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.516+0000 7f17f30b8700 1 --2- 172.21.15.162:0/1257944166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17ec0f8d10 0x7f17ec10e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:10.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.516+0000 7f17f30b8700 1 --2- 172.21.15.162:0/1257944166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17ec0f95e0 0x7f17ec10e5e0 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:10.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.516+0000 7f17f30b8700 1 -- 172.21.15.162:0/1257944166 >> 172.21.15.162:0/1257944166 conn(0x7f17ec06eb90 msgr2=0x7f17ec0708a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:10.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.516+0000 7f17f30b8700 1 -- 172.21.15.162:0/1257944166 shutdown_connections 2024-02-13T00:47:10.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.516+0000 7f17f30b8700 1 -- 172.21.15.162:0/1257944166 wait complete. 2024-02-13T00:47:10.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:10.516+0000 7f17f30b8700 1 librados: shutdown 2024-02-13T00:47:10.524 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:47:10.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:09.939687+0000 osd.1 (osd.1) 193 : cluster [DBG] 2.6 scrub starts 2024-02-13T00:47:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:09.940601+0000 osd.1 (osd.1) 194 : cluster [DBG] 2.6 scrub ok 2024-02-13T00:47:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:10.451568+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:47:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:10.510515+0000 mon.a (mon.0) 2591 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:47:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:10.510603+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e446: 8 total, 7 up, 4 in 2024-02-13T00:47:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:09.939687+0000 osd.1 (osd.1) 193 : cluster [DBG] 2.6 scrub starts 2024-02-13T00:47:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:09.940601+0000 osd.1 (osd.1) 194 : cluster [DBG] 2.6 scrub ok 2024-02-13T00:47:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:10.451568+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:47:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:10.510515+0000 mon.a (mon.0) 2591 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:47:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:10.510603+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e446: 8 total, 7 up, 4 in 2024-02-13T00:47:11.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:09.939687+0000 osd.1 (osd.1) 193 : cluster [DBG] 2.6 scrub starts 2024-02-13T00:47:11.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:09.940601+0000 osd.1 (osd.1) 194 : cluster [DBG] 2.6 scrub ok 2024-02-13T00:47:11.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:10.451568+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:47:11.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:10.510515+0000 mon.a (mon.0) 2591 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:47:11.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:10.510603+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e446: 8 total, 7 up, 4 in 2024-02-13T00:47:11.158 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:11.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:11.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:11.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:11.294 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:11.316 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:11.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:11.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:11.450 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:11.503 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:11.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:11.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:11.651 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:11.659 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6] out_osds: [3, 2, 7, 5] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-02-13T00:47:11.659 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:47:11.660 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-13T00:47:11.660 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-02-13T00:47:11.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-13T00:47:11.672 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:11.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:11.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:11.803 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:10.435117+0000 mgr.y (mgr.14544) 394 : cluster [DBG] pgmap v653: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-13T00:47:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:10.690937+0000 osd.0 (osd.0) 164 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:47:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:11.510665+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e447: 8 total, 7 up, 4 in 2024-02-13T00:47:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:11.518882+0000 mon.b (mon.1) 591 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:11.520599+0000 mon.a (mon.0) 2594 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:12.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:11.716415+0000 mon.c (mon.2) 386 : audit [DBG] from='client.? 172.21.15.162:0/2681259694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:12.066 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:10.435117+0000 mgr.y (mgr.14544) 394 : cluster [DBG] pgmap v653: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-13T00:47:12.066 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:10.690937+0000 osd.0 (osd.0) 164 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:47:12.066 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:11.510665+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e447: 8 total, 7 up, 4 in 2024-02-13T00:47:12.066 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:11.518882+0000 mon.b (mon.1) 591 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:12.066 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:11.520599+0000 mon.a (mon.0) 2594 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:12.066 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:11.716415+0000 mon.c (mon.2) 386 : audit [DBG] from='client.? 172.21.15.162:0/2681259694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:12.067 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:10.435117+0000 mgr.y (mgr.14544) 394 : cluster [DBG] pgmap v653: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-13T00:47:12.067 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:10.690937+0000 osd.0 (osd.0) 164 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:47:12.067 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:11.510665+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e447: 8 total, 7 up, 4 in 2024-02-13T00:47:12.067 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:11.518882+0000 mon.b (mon.1) 591 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:12.067 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:11.520599+0000 mon.a (mon.0) 2594 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:12.067 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:11.716415+0000 mon.c (mon.2) 386 : audit [DBG] from='client.? 172.21.15.162:0/2681259694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:12.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.712+0000 7fc23377e700 1 -- 172.21.15.162:0/2723730677 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc22c0f8d10 msgr2=0x7fc22c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:12.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.712+0000 7fc23377e700 1 --2- 172.21.15.162:0/2723730677 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc22c0f8d10 0x7fc22c0f90f0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 rx=0x7fc228004870 tx=0x7fc228009cb0).stop 2024-02-13T00:47:12.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.712+0000 7fc23377e700 1 -- 172.21.15.162:0/2723730677 shutdown_connections 2024-02-13T00:47:12.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.712+0000 7fc23377e700 1 --2- 172.21.15.162:0/2723730677 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc22c102270 0x7fc22c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:12.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.712+0000 7fc23377e700 1 --2- 172.21.15.162:0/2723730677 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc22c0f8d10 0x7fc22c0f90f0 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:12.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.712+0000 7fc23377e700 1 --2- 172.21.15.162:0/2723730677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc22c0f95e0 0x7fc22c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:12.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.712+0000 7fc23377e700 1 -- 172.21.15.162:0/2723730677 >> 172.21.15.162:0/2723730677 conn(0x7fc22c06eb90 msgr2=0x7fc22c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:12.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.712+0000 7fc23377e700 1 -- 172.21.15.162:0/2723730677 shutdown_connections 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.712+0000 7fc23377e700 1 -- 172.21.15.162:0/2723730677 wait complete. 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23377e700 1 librados: starting msgr at 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23377e700 1 librados: starting objecter 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23377e700 1 Processor -- start 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23377e700 1 -- start start 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23377e700 1 librados: setting wanted keys 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23377e700 1 librados: calling monclient init 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23377e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc22c0f8d10 0x7fc22c10e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23377e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc22c0f95e0 0x7fc22c10e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23377e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc22c102270 0x7fc22c108330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23377e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c108820 con 0x7fc22c102270 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23377e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c108990 con 0x7fc22c0f95e0 2024-02-13T00:47:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23377e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c108b50 con 0x7fc22c0f8d10 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc231d1b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc22c102270 0x7fc22c108330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23151a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc22c0f8d10 0x7fc22c10e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc231d1b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc22c102270 0x7fc22c108330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46622/0 (socket says 172.21.15.162:46622) 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc231d1b700 1 -- 172.21.15.162:0/2681259694 learned_addr learned my addr 172.21.15.162:0/2681259694 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc230d19700 1 --2- 172.21.15.162:0/2681259694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc22c0f95e0 0x7fc22c10e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23151a700 1 -- 172.21.15.162:0/2681259694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc22c0f95e0 msgr2=0x7fc22c10e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23151a700 1 --2- 172.21.15.162:0/2681259694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc22c0f95e0 0x7fc22c10e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23151a700 1 -- 172.21.15.162:0/2681259694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc22c102270 msgr2=0x7fc22c108330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23151a700 1 --2- 172.21.15.162:0/2681259694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc22c102270 0x7fc22c108330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23151a700 1 -- 172.21.15.162:0/2681259694 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2240097c0 con 0x7fc22c0f8d10 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc231d1b700 1 --2- 172.21.15.162:0/2681259694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc22c102270 0x7fc22c108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc23151a700 1 --2- 172.21.15.162:0/2681259694 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc22c0f8d10 0x7fc22c10e190 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 rx=0x7fc228004870 tx=0x7fc22800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc2227fc700 10 client.?.objecter ms_handle_connect 0x7fc22c0f8d10 2024-02-13T00:47:12.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.713+0000 7fc2227fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:12.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.714+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22800e070 con 0x7fc22c0f8d10 2024-02-13T00:47:12.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.714+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc228013da0 con 0x7fc22c0f8d10 2024-02-13T00:47:12.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.714+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc228005040 con 0x7fc22c0f8d10 2024-02-13T00:47:12.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.714+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc228019070 con 0x7fc22c0f8d10 2024-02-13T00:47:12.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.714+0000 7fc23377e700 10 client.37517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:12.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.714+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc22c108ff0 con 0x7fc22c0f8d10 2024-02-13T00:47:12.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.714+0000 7fc23377e700 1 librados: init done 2024-02-13T00:47:12.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.714+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc22c0475e0 con 0x7fc22c0f8d10 2024-02-13T00:47:12.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.715+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc22801e420 con 0x7fc22c0f8d10 2024-02-13T00:47:12.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.715+0000 7fc2227fc700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc218062c90 0x7fc218065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:12.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.715+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(447..447 src has 52..447) v4 ==== 7954+0+0 (secure 0 0 0) 0x7fc22807dac0 con 0x7fc22c0f8d10 2024-02-13T00:47:12.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.715+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 osd_map(447..447 src has 52..447) v4 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.715+0000 7fc2227fc700 3 client.37517.objecter handle_osd_map got epochs [447,447] > 0 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.715+0000 7fc2227fc700 3 client.37517.objecter handle_osd_map decoding full epoch 447 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc230d19700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc218062c90 0x7fc218065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc2227fc700 20 client.37517.objecter dump_active .. 0 homeless 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v447) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc22807de60 con 0x7fc22c0f8d10 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc2227fc700 10 client.37517.objecter ms_handle_connect 0x7fc218062c90 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc230d19700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc218062c90 0x7fc218065150 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 rx=0x7fc22c109ad0 tx=0x7fc21c009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc23377e700 10 client.37517.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 1 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc23377e700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc22c028590 0x7fc22c0fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c107170 osd=0 3 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c107170 osd=0 3 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c107170 osd=0 3 2024-02-13T00:47:12.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 1 2024-02-13T00:47:12.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c107170 osd=0 3 2024-02-13T00:47:12.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 0 1 2024-02-13T00:47:12.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c107170 osd=0 4 2024-02-13T00:47:12.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc23377e700 10 client.37517.objecter _send_command 1 2024-02-13T00:47:12.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc22c109810 con 0x7fc22c028590 2024-02-13T00:47:12.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc231d1b700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc22c028590 0x7fc22c0fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:12.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc2227fc700 10 client.37517.objecter ms_handle_connect 0x7fc22c028590 2024-02-13T00:47:12.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.716+0000 7fc231d1b700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc22c028590 0x7fc22c0fbd00 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:12.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.717+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc22c109810 con 0x7fc22c028590 2024-02-13T00:47:12.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.717+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:47:12.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.717+0000 7fc2227fc700 10 client.37517.objecter _finish_command 1 = system:0 2024-02-13T00:47:12.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.717+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c107170 osd=0 4 2024-02-13T00:47:12.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.717+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 0 1 2024-02-13T00:47:12.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.746+0000 7fc23377e700 10 client.37517.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:12.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.746+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 2 2024-02-13T00:47:12.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.746+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c107170 osd=0 3 2024-02-13T00:47:12.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.746+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c107170 osd=0 3 2024-02-13T00:47:12.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.746+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c107170 osd=0 3 2024-02-13T00:47:12.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.746+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 2 2024-02-13T00:47:12.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.746+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c107170 osd=0 3 2024-02-13T00:47:12.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.746+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 0 2 2024-02-13T00:47:12.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.746+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c107170 osd=0 4 2024-02-13T00:47:12.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.746+0000 7fc23377e700 10 client.37517.objecter _send_command 2 2024-02-13T00:47:12.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.746+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc22c05a0e0 con 0x7fc22c028590 2024-02-13T00:47:12.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc22c05a0e0 con 0x7fc22c028590 2024-02-13T00:47:12.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:12.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc2227fc700 10 client.37517.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c107170 osd=0 4 2024-02-13T00:47:12.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 0 2 2024-02-13T00:47:12.948 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:47:12.949 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:47:12.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23377e700 10 client.37517.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:12.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 3 2024-02-13T00:47:12.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23377e700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6805/150599780 conn(0x7fc22c027be0 0x7fc22c028000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:12.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c0fd370 osd=1 3 2024-02-13T00:47:12.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c0fd370 osd=1 3 2024-02-13T00:47:12.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c0fd370 osd=1 3 2024-02-13T00:47:12.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 3 2024-02-13T00:47:12.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c0fd370 osd=1 3 2024-02-13T00:47:12.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 1 3 2024-02-13T00:47:12.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c0fd370 osd=1 4 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23377e700 10 client.37517.objecter _send_command 3 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc22c0fd4e0 con 0x7fc22c027be0 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.748+0000 7fc23151a700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6805/150599780 conn(0x7fc22c027be0 0x7fc22c028000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.749+0000 7fc2227fc700 10 client.37517.objecter ms_handle_connect 0x7fc22c027be0 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.749+0000 7fc23151a700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6805/150599780 conn(0x7fc22c027be0 0x7fc22c028000 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.749+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc22c0fd4e0 con 0x7fc22c027be0 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.749+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.749+0000 7fc2227fc700 10 client.37517.objecter _finish_command 3 = system:0 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.749+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c0fd370 osd=1 4 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.749+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 1 3 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.772+0000 7fc23377e700 10 client.37517.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.772+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 4 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.772+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c0fd370 osd=1 3 2024-02-13T00:47:12.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.772+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c0fd370 osd=1 3 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.772+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c0fd370 osd=1 3 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.772+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 4 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.772+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c0fd370 osd=1 3 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.772+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 1 4 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.772+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c0fd370 osd=1 4 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.772+0000 7fc23377e700 10 client.37517.objecter _send_command 4 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.772+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc22c0f8b10 con 0x7fc22c027be0 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc22c0f8b10 con 0x7fc22c027be0 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc2227fc700 10 client.37517.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c0fd370 osd=1 4 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 1 4 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc23377e700 10 client.37517.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:12.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 5 2024-02-13T00:47:12.952 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:47:12.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc23377e700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc22c118150 0x7fc22c11e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:12.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c027890 osd=2 3 2024-02-13T00:47:12.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c027890 osd=2 3 2024-02-13T00:47:12.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c027890 osd=2 3 2024-02-13T00:47:12.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 5 2024-02-13T00:47:12.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c027890 osd=2 3 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 2 5 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c027890 osd=2 4 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc23377e700 10 client.37517.objecter _send_command 5 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc22c027a00 con 0x7fc22c118150 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc230d19700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc22c118150 0x7fc22c11e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc2227fc700 10 client.37517.objecter ms_handle_connect 0x7fc22c118150 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.774+0000 7fc230d19700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc22c118150 0x7fc22c11e9d0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.775+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc22c027a00 con 0x7fc22c118150 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.775+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.775+0000 7fc2227fc700 10 client.37517.objecter _finish_command 5 = system:0 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.775+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c027890 osd=2 4 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.775+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 2 5 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.789+0000 7fc23377e700 10 client.37517.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:12.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.789+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 6 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.789+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c027890 osd=2 3 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.789+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c027890 osd=2 3 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.789+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c027890 osd=2 3 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.789+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 6 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.789+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c027890 osd=2 3 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.789+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 2 6 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.789+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c027890 osd=2 4 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.789+0000 7fc23377e700 10 client.37517.objecter _send_command 6 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.789+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc22c0fd110 con 0x7fc22c118150 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc22c0fd110 con 0x7fc22c118150 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc2227fc700 10 client.37517.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c027890 osd=2 4 2024-02-13T00:47:12.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 2 6 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc23377e700 10 client.37517.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 7 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc23377e700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6813/604971011 conn(0x7fc22c0fc5a0 0x7fc22c120040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c0fc430 osd=3 3 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c0fc430 osd=3 3 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c0fc430 osd=3 3 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 7 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c0fc430 osd=3 3 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 3 7 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c0fc430 osd=3 4 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc23377e700 10 client.37517.objecter _send_command 7 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.791+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc22c117e50 con 0x7fc22c0fc5a0 2024-02-13T00:47:12.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.792+0000 7fc231d1b700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6813/604971011 conn(0x7fc22c0fc5a0 0x7fc22c120040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:12.956 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:47:12.956 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:47:12.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.792+0000 7fc2227fc700 10 client.37517.objecter ms_handle_connect 0x7fc22c0fc5a0 2024-02-13T00:47:12.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.792+0000 7fc231d1b700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.162:6813/604971011 conn(0x7fc22c0fc5a0 0x7fc22c120040 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:12.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.792+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc22c117e50 con 0x7fc22c0fc5a0 2024-02-13T00:47:12.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.792+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:47:12.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.792+0000 7fc2227fc700 10 client.37517.objecter _finish_command 7 = system:0 2024-02-13T00:47:12.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.792+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c0fc430 osd=3 4 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.792+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 3 7 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.812+0000 7fc23377e700 10 client.37517.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.812+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 8 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.812+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c0fc430 osd=3 3 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.812+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c0fc430 osd=3 3 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.812+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c0fc430 osd=3 3 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.812+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 8 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.812+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c0fc430 osd=3 3 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.812+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 3 8 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.812+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c0fc430 osd=3 4 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.812+0000 7fc23377e700 10 client.37517.objecter _send_command 8 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.812+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc22c117210 con 0x7fc22c0fc5a0 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc22c117210 con 0x7fc22c0fc5a0 2024-02-13T00:47:12.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc2227fc700 10 client.37517.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c0fc430 osd=3 4 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 3 8 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23377e700 10 client.37517.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 9 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23377e700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc22c121210 0x7fc22c125690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c117210 osd=4 3 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c117210 osd=4 3 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c117210 osd=4 3 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 9 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c117210 osd=4 3 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 4 9 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c117210 osd=4 4 2024-02-13T00:47:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23377e700 10 client.37517.objecter _send_command 9 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc22c1174f0 con 0x7fc22c121210 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.814+0000 7fc23151a700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc22c121210 0x7fc22c125690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.815+0000 7fc2227fc700 10 client.37517.objecter ms_handle_connect 0x7fc22c121210 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.815+0000 7fc23151a700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc22c121210 0x7fc22c125690 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.816+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc22c1174f0 con 0x7fc22c121210 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.816+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.816+0000 7fc2227fc700 10 client.37517.objecter _finish_command 9 = system:0 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.816+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c117210 osd=4 4 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.816+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 4 9 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.831+0000 7fc23377e700 10 client.37517.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.831+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 10 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.831+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c117210 osd=4 3 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.831+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c117210 osd=4 3 2024-02-13T00:47:12.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.831+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c117210 osd=4 3 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.831+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 10 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.831+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c117210 osd=4 3 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.831+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 4 10 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.831+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c117210 osd=4 4 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.831+0000 7fc23377e700 10 client.37517.objecter _send_command 10 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.831+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc22c1174f0 con 0x7fc22c121210 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc22c1174f0 con 0x7fc22c121210 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc2227fc700 10 client.37517.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c117210 osd=4 4 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 4 10 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 10 client.37517.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:12.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 11 2024-02-13T00:47:12.961 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:47:12.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 20 client.37517.objecter _get_session osd=-1 returning homeless 2024-02-13T00:47:12.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 10 client.37517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:12.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=448}) v3 -- 0x7fc22c1282e0 con 0x7fc22c0f8d10 2024-02-13T00:47:12.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc22c1284b0 con 0x7fc22c0f8d10 2024-02-13T00:47:12.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_get_version_reply(handle=1 version=447) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc22804f470 con 0x7fc22c0f8d10 2024-02-13T00:47:12.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc2237fe700 10 client.37517.objecter _check_command_map_dne tid 11 current 447 map_dne_bound 447 2024-02-13T00:47:12.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc2237fe700 10 client.37517.objecter _finish_command 11 = osd:6 osd down 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc2237fe700 15 client.37517.objecter _session_command_op_remove -1 11 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 10 client.37517.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 12 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc22c1208e0 0x7fc22c126d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c120770 osd=6 3 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c120770 osd=6 3 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c120770 osd=6 3 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 12 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c120770 osd=6 3 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 6 12 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c120770 osd=6 4 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 10 client.37517.objecter _send_command 12 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.833+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.181:6808/1506640341 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc22c11fd40 con 0x7fc22c1208e0 2024-02-13T00:47:12.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.834+0000 7fc230d19700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc22c1208e0 0x7fc22c126d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.834+0000 7fc2227fc700 10 client.37517.objecter ms_handle_connect 0x7fc22c1208e0 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.834+0000 7fc230d19700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc22c1208e0 0x7fc22c126d00 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.835+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc22c11fd40 con 0x7fc22c1208e0 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.835+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 12: 0 ) v1 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.835+0000 7fc2227fc700 10 client.37517.objecter _finish_command 12 = system:0 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.835+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c120770 osd=6 4 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.835+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 6 12 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.849+0000 7fc23377e700 10 client.37517.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.849+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 13 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.849+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c120770 osd=6 3 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.849+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c120770 osd=6 3 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.849+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c120770 osd=6 3 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.849+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 13 2024-02-13T00:47:12.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.849+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c120770 osd=6 3 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.849+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 6 13 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.849+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c120770 osd=6 4 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.849+0000 7fc23377e700 10 client.37517.objecter _send_command 13 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.849+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc22c11fd00 con 0x7fc22c1208e0 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc22c11fd00 con 0x7fc22c1208e0 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc2227fc700 10 client.37517.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c120770 osd=6 4 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 6 13 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc23377e700 10 client.37517.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 14 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc23377e700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc22c127700 0x7fc22c127b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c11fd00 osd=7 3 2024-02-13T00:47:12.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c11fd00 osd=7 3 2024-02-13T00:47:12.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c11fd00 osd=7 3 2024-02-13T00:47:12.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 14 2024-02-13T00:47:12.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c11fd00 osd=7 3 2024-02-13T00:47:12.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 7 14 2024-02-13T00:47:12.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c11fd00 osd=7 4 2024-02-13T00:47:12.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc23377e700 10 client.37517.objecter _send_command 14 2024-02-13T00:47:12.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.851+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.181:6812/1964192772 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc22c128140 con 0x7fc22c127700 2024-02-13T00:47:12.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.852+0000 7fc231d1b700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc22c127700 0x7fc22c127b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:12.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.852+0000 7fc2227fc700 10 client.37517.objecter ms_handle_connect 0x7fc22c127700 2024-02-13T00:47:12.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.852+0000 7fc231d1b700 1 --2- 172.21.15.162:0/2681259694 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc22c127700 0x7fc22c127b40 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:12.966 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:47:12.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.853+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc22c128140 con 0x7fc22c127700 2024-02-13T00:47:12.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.853+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 14: 0 ) v1 2024-02-13T00:47:12.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.853+0000 7fc2227fc700 10 client.37517.objecter _finish_command 14 = system:0 2024-02-13T00:47:12.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.853+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c11fd00 osd=7 4 2024-02-13T00:47:12.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.853+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 7 14 2024-02-13T00:47:12.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.867+0000 7fc23377e700 10 client.37517.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:12.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.867+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign -1 15 2024-02-13T00:47:12.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.867+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c11fd00 osd=7 3 2024-02-13T00:47:12.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.867+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c11fd00 osd=7 3 2024-02-13T00:47:12.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.867+0000 7fc23377e700 20 client.37517.objecter _get_session s=0x7fc22c11fd00 osd=7 3 2024-02-13T00:47:12.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.867+0000 7fc23377e700 15 client.37517.objecter _session_command_op_remove -1 15 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.867+0000 7fc23377e700 20 client.37517.objecter get_session s=0x7fc22c11fd00 osd=7 3 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.867+0000 7fc23377e700 15 client.37517.objecter _session_command_op_assign 7 15 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.867+0000 7fc23377e700 20 client.37517.objecter put_session s=0x7fc22c11fd00 osd=7 4 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.867+0000 7fc23377e700 10 client.37517.objecter _send_command 15 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.867+0000 7fc23377e700 1 -- 172.21.15.162:0/2681259694 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc22c128140 con 0x7fc22c127700 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.869+0000 7fc2227fc700 1 -- 172.21.15.162:0/2681259694 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc22c128140 con 0x7fc22c127700 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.869+0000 7fc2227fc700 10 client.37517.objecter ms_dispatch 0x7fc22c001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.869+0000 7fc2227fc700 10 client.37517.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.869+0000 7fc2227fc700 20 client.37517.objecter put_session s=0x7fc22c11fd00 osd=7 4 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:11.869+0000 7fc2227fc700 15 client.37517.objecter _session_command_op_remove 7 15 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.967 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.968 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.968 INFO:teuthology.orchestra.run.smithi162.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-02-13T00:47:12.968 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:47:12.968 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:47:13.361 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:13.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:13.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:13.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:13.496 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:13.550 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:13.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:13.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:13.677 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:47:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:47:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:47:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:47:13.678 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:47:13.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:12.435690+0000 mgr.y (mgr.14544) 395 : cluster [DBG] pgmap v656: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:47:13.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:12.512635+0000 mon.a (mon.0) 2595 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:13.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:12.512728+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e448: 8 total, 7 up, 4 in 2024-02-13T00:47:13.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:12.522824+0000 mon.b (mon.1) 592 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:13.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:12.525996+0000 mon.b (mon.1) 593 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:47:13.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:12.527784+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:47:13.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:12.435690+0000 mgr.y (mgr.14544) 395 : cluster [DBG] pgmap v656: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:47:13.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:12.512635+0000 mon.a (mon.0) 2595 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:13.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:12.512728+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e448: 8 total, 7 up, 4 in 2024-02-13T00:47:13.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:12.522824+0000 mon.b (mon.1) 592 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:13.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:12.525996+0000 mon.b (mon.1) 593 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:47:13.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:12.527784+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:47:13.681 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:12.435690+0000 mgr.y (mgr.14544) 395 : cluster [DBG] pgmap v656: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:47:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:12.512635+0000 mon.a (mon.0) 2595 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:12.512728+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e448: 8 total, 7 up, 4 in 2024-02-13T00:47:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:12.522824+0000 mon.b (mon.1) 592 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:12.525996+0000 mon.b (mon.1) 593 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:47:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:12.527784+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:47:14.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:47:14.125 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-13T00:47:14.125 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-13T00:47:14.130 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:14.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:14.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:14.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:14.279 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:14.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:14.330 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:14.373 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:14.384 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:14.417 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:14.438 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:14.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:14.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:14.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:14.573 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:14.719 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:14.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:13.514805+0000 mon.a (mon.0) 2598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:47:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:13.514880+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e449: 8 total, 7 up, 4 in 2024-02-13T00:47:14.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:14.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:13.514805+0000 mon.a (mon.0) 2598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:47:14.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:13.514880+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e449: 8 total, 7 up, 4 in 2024-02-13T00:47:14.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:13.514805+0000 mon.a (mon.0) 2598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:47:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:13.514880+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e449: 8 total, 7 up, 4 in 2024-02-13T00:47:14.850 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:14.975 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:14.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:15.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:15.112 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:15.133 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:15.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:15.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:15.265 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:15.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:47:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:47:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:14.436055+0000 mgr.y (mgr.14544) 396 : cluster [DBG] pgmap v659: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:14.527606+0000 mon.a (mon.0) 2600 : cluster [DBG] osdmap e450: 8 total, 7 up, 4 in 2024-02-13T00:47:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:15.433155+0000 mon.a (mon.0) 2601 : cluster [DBG] osdmap e451: 8 total, 7 up, 4 in 2024-02-13T00:47:15.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:14.436055+0000 mgr.y (mgr.14544) 396 : cluster [DBG] pgmap v659: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:14.527606+0000 mon.a (mon.0) 2600 : cluster [DBG] osdmap e450: 8 total, 7 up, 4 in 2024-02-13T00:47:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:15.433155+0000 mon.a (mon.0) 2601 : cluster [DBG] osdmap e451: 8 total, 7 up, 4 in 2024-02-13T00:47:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:14.436055+0000 mgr.y (mgr.14544) 396 : cluster [DBG] pgmap v659: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:14.527606+0000 mon.a (mon.0) 2600 : cluster [DBG] osdmap e450: 8 total, 7 up, 4 in 2024-02-13T00:47:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:15.433155+0000 mon.a (mon.0) 2601 : cluster [DBG] osdmap e451: 8 total, 7 up, 4 in 2024-02-13T00:47:16.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:47:16.407 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:47:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:47:16] "GET /metrics HTTP/1.1" 200 175058 "" "Prometheus/2.33.4" 2024-02-13T00:47:16.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-13T00:47:16.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:16.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:16.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:16.547 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:16.580 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:16.599 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:16.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:16.703 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:16.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:15.653573+0000 mgr.y (mgr.14544) 397 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:16.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:15.726907+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:16.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:15.727102+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:16.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:15.653573+0000 mgr.y (mgr.14544) 397 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:16.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:15.726907+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:16.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:15.727102+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:16.732 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:16.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:15.653573+0000 mgr.y (mgr.14544) 397 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:15.726907+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:15.727102+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:16.855 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:16.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:16.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:16.991 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:17.012 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:17.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:17.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:17.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:17.145 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:17.165 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:17.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:17.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:17.269 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:17.299 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:17.319 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:17.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:17.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:17.453 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:17.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.462+0000 7fb698038700 1 -- 172.21.15.162:0/4069289720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6900f95e0 msgr2=0x7fb690101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:17.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.462+0000 7fb698038700 1 --2- 172.21.15.162:0/4069289720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6900f95e0 0x7fb690101d80 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7fb680004870 tx=0x7fb680009cb0).stop 2024-02-13T00:47:17.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.462+0000 7fb698038700 1 -- 172.21.15.162:0/4069289720 shutdown_connections 2024-02-13T00:47:17.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.462+0000 7fb698038700 1 --2- 172.21.15.162:0/4069289720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb690102270 0x7fb690104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:17.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.462+0000 7fb698038700 1 --2- 172.21.15.162:0/4069289720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6900f8d10 0x7fb6900f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:17.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.462+0000 7fb698038700 1 --2- 172.21.15.162:0/4069289720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6900f95e0 0x7fb690101d80 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:17.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.462+0000 7fb698038700 1 -- 172.21.15.162:0/4069289720 >> 172.21.15.162:0/4069289720 conn(0x7fb69006eb90 msgr2=0x7fb6900f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:17.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.462+0000 7fb698038700 1 -- 172.21.15.162:0/4069289720 shutdown_connections 2024-02-13T00:47:17.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.462+0000 7fb698038700 1 -- 172.21.15.162:0/4069289720 wait complete. 2024-02-13T00:47:17.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.462+0000 7fb698038700 1 librados: starting msgr at 2024-02-13T00:47:17.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.462+0000 7fb698038700 1 librados: starting objecter 2024-02-13T00:47:17.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb698038700 1 Processor -- start 2024-02-13T00:47:17.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb698038700 1 -- start start 2024-02-13T00:47:17.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb698038700 1 librados: setting wanted keys 2024-02-13T00:47:17.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb698038700 1 librados: calling monclient init 2024-02-13T00:47:17.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb698038700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6900f8d10 0x7fb6900f8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:17.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb698038700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6900f95e0 0x7fb6900f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb698038700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb690102270 0x7fb6900f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb698038700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900f79b0 con 0x7fb690102270 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb698038700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900f7b20 con 0x7fb6900f95e0 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb698038700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69011ad50 con 0x7fb6900f8d10 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb695dd4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6900f8d10 0x7fb6900f8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb695dd4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6900f8d10 0x7fb6900f8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:35074/0 (socket says 172.21.15.162:35074) 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb695dd4700 1 -- 172.21.15.162:0/4071309867 learned_addr learned my addr 172.21.15.162:0/4071309867 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb6965d5700 1 --2- 172.21.15.162:0/4071309867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb690102270 0x7fb6900f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb6955d3700 1 --2- 172.21.15.162:0/4071309867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6900f95e0 0x7fb6900f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.463+0000 7fb695dd4700 1 -- 172.21.15.162:0/4071309867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6900f95e0 msgr2=0x7fb6900f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb695dd4700 1 --2- 172.21.15.162:0/4071309867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6900f95e0 0x7fb6900f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb695dd4700 1 -- 172.21.15.162:0/4071309867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb690102270 msgr2=0x7fb6900f72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb695dd4700 1 --2- 172.21.15.162:0/4071309867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb690102270 0x7fb6900f72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:17.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb695dd4700 1 -- 172.21.15.162:0/4071309867 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb680005040 con 0x7fb6900f8d10 2024-02-13T00:47:17.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb6955d3700 1 --2- 172.21.15.162:0/4071309867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6900f95e0 0x7fb6900f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:47:17.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb695dd4700 1 --2- 172.21.15.162:0/4071309867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6900f8d10 0x7fb6900f8720 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7fb68c00c7c0 tx=0x7fb68c007ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:17.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb686ffd700 10 client.?.objecter ms_handle_connect 0x7fb6900f8d10 2024-02-13T00:47:17.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb686ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:17.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb686ffd700 1 -- 172.21.15.162:0/4071309867 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c004430 con 0x7fb6900f8d10 2024-02-13T00:47:17.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb686ffd700 1 -- 172.21.15.162:0/4071309867 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb68c00e970 con 0x7fb6900f8d10 2024-02-13T00:47:17.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb698038700 1 -- 172.21.15.162:0/4071309867 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69011afd0 con 0x7fb6900f8d10 2024-02-13T00:47:17.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb698038700 10 client.37523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:17.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb698038700 1 -- 172.21.15.162:0/4071309867 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69011b490 con 0x7fb6900f8d10 2024-02-13T00:47:17.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.464+0000 7fb698038700 1 librados: init done 2024-02-13T00:47:17.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.465+0000 7fb686ffd700 1 -- 172.21.15.162:0/4071309867 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c013a50 con 0x7fb6900f8d10 2024-02-13T00:47:17.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.465+0000 7fb698038700 1 -- 172.21.15.162:0/4071309867 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6900475e0 con 0x7fb6900f8d10 2024-02-13T00:47:17.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.467+0000 7fb686ffd700 1 -- 172.21.15.162:0/4071309867 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fb68c00eae0 con 0x7fb6900f8d10 2024-02-13T00:47:17.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.467+0000 7fb686ffd700 1 --2- 172.21.15.162:0/4071309867 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb67c062c90 0x7fb67c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:17.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.468+0000 7fb686ffd700 1 -- 172.21.15.162:0/4071309867 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(452..452 src has 52..452) v4 ==== 7967+0+0 (secure 0 0 0) 0x7fb68c07d540 con 0x7fb6900f8d10 2024-02-13T00:47:17.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.468+0000 7fb686ffd700 10 client.37523.objecter ms_dispatch 0x7fb690001550 osd_map(452..452 src has 52..452) v4 2024-02-13T00:47:17.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.468+0000 7fb686ffd700 3 client.37523.objecter handle_osd_map got epochs [452,452] > 0 2024-02-13T00:47:17.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.468+0000 7fb686ffd700 3 client.37523.objecter handle_osd_map decoding full epoch 452 2024-02-13T00:47:17.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.468+0000 7fb6955d3700 1 --2- 172.21.15.162:0/4071309867 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb67c062c90 0x7fb67c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:17.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.468+0000 7fb686ffd700 20 client.37523.objecter dump_active .. 0 homeless 2024-02-13T00:47:17.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.468+0000 7fb686ffd700 10 client.37523.objecter ms_handle_connect 0x7fb67c062c90 2024-02-13T00:47:17.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.468+0000 7fb6955d3700 1 --2- 172.21.15.162:0/4071309867 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb67c062c90 0x7fb67c065150 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 rx=0x7fb6800043d0 tx=0x7fb68000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:17.473 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:17.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.471+0000 7fb686ffd700 1 -- 172.21.15.162:0/4071309867 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb68c050a70 con 0x7fb6900f8d10 2024-02-13T00:47:17.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:17.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:17.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:17.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:17.666+0000 7fb698038700 1 -- 172.21.15.162:0/4071309867 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) v1 -- 0x7fb69005fb80 con 0x7fb6900f8d10 2024-02-13T00:47:17.701 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:16.436429+0000 mgr.y (mgr.14544) 398 : cluster [DBG] pgmap v662: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-02-13T00:47:17.702 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:16.538278+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e452: 8 total, 7 up, 4 in 2024-02-13T00:47:17.702 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:16.436429+0000 mgr.y (mgr.14544) 398 : cluster [DBG] pgmap v662: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-02-13T00:47:17.702 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:16.538278+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e452: 8 total, 7 up, 4 in 2024-02-13T00:47:17.704 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:17.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:16.436429+0000 mgr.y (mgr.14544) 398 : cluster [DBG] pgmap v662: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-02-13T00:47:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:16.538278+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e452: 8 total, 7 up, 4 in 2024-02-13T00:47:17.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:17.852 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:18.408 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:18.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:18.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:18.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:18.546 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:18.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.543+0000 7fb686ffd700 1 -- 172.21.15.162:0/4071309867 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v453) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb68c050660 con 0x7fb6900f8d10 2024-02-13T00:47:18.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.546+0000 7fb698038700 10 librados: watch_flush enter 2024-02-13T00:47:18.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.546+0000 7fb698038700 10 librados: watch_flush exit 2024-02-13T00:47:18.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.546+0000 7fb698038700 20 client.37523.objecter shutdown clearing up homeless session... 2024-02-13T00:47:18.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.546+0000 7fb698038700 10 client.37523.objecter successfully canceled tick 2024-02-13T00:47:18.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.546+0000 7fb698038700 1 -- 172.21.15.162:0/4071309867 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb67c062c90 msgr2=0x7fb67c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:18.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.546+0000 7fb698038700 1 --2- 172.21.15.162:0/4071309867 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb67c062c90 0x7fb67c065150 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 rx=0x7fb6800043d0 tx=0x7fb68000b040).stop 2024-02-13T00:47:18.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.546+0000 7fb698038700 1 -- 172.21.15.162:0/4071309867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6900f8d10 msgr2=0x7fb6900f8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:18.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.546+0000 7fb698038700 1 --2- 172.21.15.162:0/4071309867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6900f8d10 0x7fb6900f8720 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7fb68c00c7c0 tx=0x7fb68c007ef0).stop 2024-02-13T00:47:18.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.546+0000 7fb698038700 1 -- 172.21.15.162:0/4071309867 shutdown_connections 2024-02-13T00:47:18.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.546+0000 7fb698038700 1 --2- 172.21.15.162:0/4071309867 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb67c062c90 0x7fb67c065150 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:18.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.546+0000 7fb698038700 1 --2- 172.21.15.162:0/4071309867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb690102270 0x7fb6900f72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:18.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.547+0000 7fb698038700 1 --2- 172.21.15.162:0/4071309867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6900f8d10 0x7fb6900f8720 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:18.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.547+0000 7fb698038700 1 --2- 172.21.15.162:0/4071309867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6900f95e0 0x7fb6900f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:18.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.547+0000 7fb698038700 1 -- 172.21.15.162:0/4071309867 >> 172.21.15.162:0/4071309867 conn(0x7fb69006eb90 msgr2=0x7fb690102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:18.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.547+0000 7fb698038700 1 -- 172.21.15.162:0/4071309867 shutdown_connections 2024-02-13T00:47:18.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.547+0000 7fb698038700 1 -- 172.21.15.162:0/4071309867 wait complete. 2024-02-13T00:47:18.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.547+0000 7fb698038700 1 librados: shutdown 2024-02-13T00:47:18.552 INFO:teuthology.orchestra.run.smithi162.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-13T00:47:18.568 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:18.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:18.579 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:18.580 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:18.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.596+0000 7f70f8462700 1 -- 172.21.15.162:0/4030526526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f00fc430 msgr2=0x7f70f00fc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:18.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.596+0000 7f70f8462700 1 --2- 172.21.15.162:0/4030526526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f00fc430 0x7f70f00fc810 secure :-1 s=READY pgs=4855 cs=0 l=1 rev1=1 rx=0x7f70ec0048b0 tx=0x7f70ec00adc0).stop 2024-02-13T00:47:18.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.596+0000 7f70f8462700 1 -- 172.21.15.162:0/4030526526 shutdown_connections 2024-02-13T00:47:18.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.596+0000 7f70f8462700 1 --2- 172.21.15.162:0/4030526526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70f00f6b90 0x7f70f0103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:18.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.596+0000 7f70f8462700 1 --2- 172.21.15.162:0/4030526526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f00fc430 0x7f70f00fc810 unknown :-1 s=CLOSED pgs=4855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:18.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.596+0000 7f70f8462700 1 --2- 172.21.15.162:0/4030526526 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f70f00fcd90 0x7f70f00f64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:18.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.596+0000 7f70f8462700 1 -- 172.21.15.162:0/4030526526 >> 172.21.15.162:0/4030526526 conn(0x7f70f006c410 msgr2=0x7f70f006c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:18.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.596+0000 7f70f8462700 1 -- 172.21.15.162:0/4030526526 shutdown_connections 2024-02-13T00:47:18.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.596+0000 7f70f8462700 1 -- 172.21.15.162:0/4030526526 wait complete. 2024-02-13T00:47:18.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.597+0000 7f70f8462700 1 librados: starting msgr at 2024-02-13T00:47:18.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.597+0000 7f70f8462700 1 librados: starting objecter 2024-02-13T00:47:18.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.597+0000 7f70f8462700 1 Processor -- start 2024-02-13T00:47:18.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.597+0000 7f70f8462700 1 -- start start 2024-02-13T00:47:18.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.597+0000 7f70f8462700 1 librados: setting wanted keys 2024-02-13T00:47:18.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.597+0000 7f70f8462700 1 librados: calling monclient init 2024-02-13T00:47:18.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.598+0000 7f70f8462700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70f00f6b90 0x7f70f0103d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:18.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.598+0000 7f70f8462700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f00fc430 0x7f70f0104240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:18.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.598+0000 7f70f8462700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f70f00fcd90 0x7f70f0104730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:18.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.598+0000 7f70f8462700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f01167a0 con 0x7f70f00f6b90 2024-02-13T00:47:18.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.598+0000 7f70f8462700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f0116910 con 0x7f70f00fc430 2024-02-13T00:47:18.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.598+0000 7f70f8462700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f0116ad0 con 0x7f70f00fcd90 2024-02-13T00:47:18.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.598+0000 7f70f69ff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f70f00fcd90 0x7f70f0104730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:18.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.598+0000 7f70f69ff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f70f00fcd90 0x7f70f0104730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:35086/0 (socket says 172.21.15.162:35086) 2024-02-13T00:47:18.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.598+0000 7f70f69ff700 1 -- 172.21.15.162:0/2682240135 learned_addr learned my addr 172.21.15.162:0/2682240135 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:18.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.598+0000 7f70f59fd700 1 --2- 172.21.15.162:0/2682240135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f00fc430 0x7f70f0104240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:18.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.598+0000 7f70f61fe700 1 --2- 172.21.15.162:0/2682240135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70f00f6b90 0x7f70f0103d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:18.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70f59fd700 1 -- 172.21.15.162:0/2682240135 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f70f00fcd90 msgr2=0x7f70f0104730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:18.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70f59fd700 1 --2- 172.21.15.162:0/2682240135 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f70f00fcd90 0x7f70f0104730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:18.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70f59fd700 1 -- 172.21.15.162:0/2682240135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70f00f6b90 msgr2=0x7f70f0103d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:18.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70f59fd700 1 --2- 172.21.15.162:0/2682240135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70f00f6b90 0x7f70f0103d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:18.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70f59fd700 1 -- 172.21.15.162:0/2682240135 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e80097c0 con 0x7f70f00fc430 2024-02-13T00:47:18.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70f59fd700 1 --2- 172.21.15.162:0/2682240135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f00fc430 0x7f70f0104240 secure :-1 s=READY pgs=4856 cs=0 l=1 rev1=1 rx=0x7f70f00fd190 tx=0x7f70e0009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:18.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70e77fe700 10 client.?.objecter ms_handle_connect 0x7f70f00fc430 2024-02-13T00:47:18.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70e77fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:18.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70e77fe700 1 -- 172.21.15.162:0/2682240135 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0019070 con 0x7f70f00fc430 2024-02-13T00:47:18.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70f8462700 1 -- 172.21.15.162:0/2682240135 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e0005040 con 0x7f70f00fc430 2024-02-13T00:47:18.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70e77fe700 1 -- 172.21.15.162:0/2682240135 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70e0007a40 con 0x7f70f00fc430 2024-02-13T00:47:18.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70e77fe700 1 -- 172.21.15.162:0/2682240135 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e001c320 con 0x7f70f00fc430 2024-02-13T00:47:18.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.599+0000 7f70f8462700 10 client.44533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:18.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.600+0000 7f70f8462700 1 -- 172.21.15.162:0/2682240135 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ec005040 con 0x7f70f00fc430 2024-02-13T00:47:18.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.600+0000 7f70f8462700 1 librados: init done 2024-02-13T00:47:18.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.601+0000 7f70f8462700 1 -- 172.21.15.162:0/2682240135 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f005f9a0 con 0x7f70f00fc430 2024-02-13T00:47:18.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.601+0000 7f70e77fe700 1 -- 172.21.15.162:0/2682240135 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f70e001c750 con 0x7f70f00fc430 2024-02-13T00:47:18.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.601+0000 7f70e77fe700 1 --2- 172.21.15.162:0/2682240135 >> v2:172.21.15.162:6800/2866534053 conn(0x7f70dc062c90 0x7f70dc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:18.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.601+0000 7f70e77fe700 1 -- 172.21.15.162:0/2682240135 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(453..453 src has 52..453) v4 ==== 7967+0+0 (secure 0 0 0) 0x7f70e0084ff0 con 0x7f70f00fc430 2024-02-13T00:47:18.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.601+0000 7f70e77fe700 10 client.44533.objecter ms_dispatch 0x7f70f0001550 osd_map(453..453 src has 52..453) v4 2024-02-13T00:47:18.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.601+0000 7f70e77fe700 3 client.44533.objecter handle_osd_map got epochs [453,453] > 0 2024-02-13T00:47:18.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.601+0000 7f70e77fe700 3 client.44533.objecter handle_osd_map decoding full epoch 453 2024-02-13T00:47:18.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.601+0000 7f70f61fe700 1 --2- 172.21.15.162:0/2682240135 >> v2:172.21.15.162:6800/2866534053 conn(0x7f70dc062c90 0x7f70dc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:18.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.601+0000 7f70e77fe700 20 client.44533.objecter dump_active .. 0 homeless 2024-02-13T00:47:18.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.602+0000 7f70f61fe700 1 --2- 172.21.15.162:0/2682240135 >> v2:172.21.15.162:6800/2866534053 conn(0x7f70dc062c90 0x7f70dc065150 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 rx=0x7f70ec00aeb0 tx=0x7f70ec00c040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:18.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.602+0000 7f70e77fe700 10 client.44533.objecter ms_handle_connect 0x7f70dc062c90 2024-02-13T00:47:18.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.606+0000 7f70e77fe700 1 -- 172.21.15.162:0/2682240135 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f70e0058600 con 0x7f70f00fc430 2024-02-13T00:47:18.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:18.701 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:18.720 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:18.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:18.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:18.784+0000 7f70f8462700 1 -- 172.21.15.162:0/2682240135 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f70f0107210 con 0x7f70f00fc430 2024-02-13T00:47:18.794 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:47:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:47:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:47:18.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:17.667142+0000 mon.c (mon.2) 387 : audit [INF] from='client.? 172.21.15.162:0/4071309867' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:47:18.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:17.667475+0000 mon.a (mon.0) 2603 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:47:18.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:18.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:17.667142+0000 mon.c (mon.2) 387 : audit [INF] from='client.? 172.21.15.162:0/4071309867' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:47:18.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:17.667475+0000 mon.a (mon.0) 2603 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:47:18.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:17.667142+0000 mon.c (mon.2) 387 : audit [INF] from='client.? 172.21.15.162:0/4071309867' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:47:18.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:17.667475+0000 mon.a (mon.0) 2603 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:47:18.856 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:19.151 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:19.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:19.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:19.282 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:19.301 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:19.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:19.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:19.434 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:19.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.548+0000 7f70e77fe700 1 -- 172.21.15.162:0/2682240135 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v454) v1 ==== 97+0+0 (secure 0 0 0) 0x7f70e0023070 con 0x7f70f00fc430 2024-02-13T00:47:19.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 10 librados: watch_flush enter 2024-02-13T00:47:19.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 10 librados: watch_flush exit 2024-02-13T00:47:19.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 20 client.44533.objecter shutdown clearing up homeless session... 2024-02-13T00:47:19.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 10 client.44533.objecter successfully canceled tick 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 -- 172.21.15.162:0/2682240135 >> v2:172.21.15.162:6800/2866534053 conn(0x7f70dc062c90 msgr2=0x7f70dc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 --2- 172.21.15.162:0/2682240135 >> v2:172.21.15.162:6800/2866534053 conn(0x7f70dc062c90 0x7f70dc065150 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 rx=0x7f70ec00aeb0 tx=0x7f70ec00c040).stop 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 -- 172.21.15.162:0/2682240135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f00fc430 msgr2=0x7f70f0104240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 --2- 172.21.15.162:0/2682240135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f00fc430 0x7f70f0104240 secure :-1 s=READY pgs=4856 cs=0 l=1 rev1=1 rx=0x7f70f00fd190 tx=0x7f70e0009cb0).stop 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 -- 172.21.15.162:0/2682240135 shutdown_connections 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 --2- 172.21.15.162:0/2682240135 >> v2:172.21.15.162:6800/2866534053 conn(0x7f70dc062c90 0x7f70dc065150 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 --2- 172.21.15.162:0/2682240135 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f70f00fcd90 0x7f70f0104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 --2- 172.21.15.162:0/2682240135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70f00f6b90 0x7f70f0103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 --2- 172.21.15.162:0/2682240135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f00fc430 0x7f70f0104240 unknown :-1 s=CLOSED pgs=4856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 -- 172.21.15.162:0/2682240135 >> 172.21.15.162:0/2682240135 conn(0x7f70f006c410 msgr2=0x7f70f00f4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 -- 172.21.15.162:0/2682240135 shutdown_connections 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 -- 172.21.15.162:0/2682240135 wait complete. 2024-02-13T00:47:19.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:19.551+0000 7f70f8462700 1 librados: shutdown 2024-02-13T00:47:19.556 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:47:19.576 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:47:19.344+0000 7f16fd9f6700 -1 mgr get_metadata_python Requested missing service osd.5 2024-02-13T00:47:19.576 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:18.436963+0000 mgr.y (mgr.14544) 399 : cluster [DBG] pgmap v664: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-02-13T00:47:19.577 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:18.543558+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:47:19.577 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:18.543621+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e453: 8 total, 7 up, 4 in 2024-02-13T00:47:19.577 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:18.785500+0000 mon.b (mon.1) 594 : audit [INF] from='client.? 172.21.15.162:0/2682240135' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:47:19.577 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:18.786049+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:47:19.577 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:19.339617+0000 mon.a (mon.0) 2607 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:47:19.577 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:18.436963+0000 mgr.y (mgr.14544) 399 : cluster [DBG] pgmap v664: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-02-13T00:47:19.577 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:18.543558+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:47:19.577 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:18.543621+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e453: 8 total, 7 up, 4 in 2024-02-13T00:47:19.577 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:18.785500+0000 mon.b (mon.1) 594 : audit [INF] from='client.? 172.21.15.162:0/2682240135' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:47:19.577 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:18.786049+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:47:19.577 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:19.339617+0000 mon.a (mon.0) 2607 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:47:19.983 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:19.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:19.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:18.436963+0000 mgr.y (mgr.14544) 399 : cluster [DBG] pgmap v664: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-02-13T00:47:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:18.543558+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:47:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:18.543621+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e453: 8 total, 7 up, 4 in 2024-02-13T00:47:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:18.785500+0000 mon.b (mon.1) 594 : audit [INF] from='client.? 172.21.15.162:0/2682240135' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:47:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:18.786049+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:47:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:19.339617+0000 mon.a (mon.0) 2607 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:47:20.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:20.119 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:20.140 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:20.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:20.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:20.275 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:20.474 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:20.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:20.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:20.605 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:20.624 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:20.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:20.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:20.758 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:20.777 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:20.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:19.221168+0000 osd.6 (osd.6) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:19.221394+0000 osd.6 (osd.6) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:19.541784+0000 mon.a (mon.0) 2608 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:47:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:19.548579+0000 mon.a (mon.0) 2609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:47:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:19.548628+0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e454: 8 total, 7 up, 4 in 2024-02-13T00:47:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:20.434405+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e455: 8 total, 7 up, 4 in 2024-02-13T00:47:20.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:19.221168+0000 osd.6 (osd.6) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:19.221394+0000 osd.6 (osd.6) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:19.541784+0000 mon.a (mon.0) 2608 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:47:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:19.548579+0000 mon.a (mon.0) 2609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:47:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:19.548628+0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e454: 8 total, 7 up, 4 in 2024-02-13T00:47:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:20.434405+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e455: 8 total, 7 up, 4 in 2024-02-13T00:47:20.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:20.907 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:21.006 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:21.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:19.221168+0000 osd.6 (osd.6) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:19.221394+0000 osd.6 (osd.6) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:19.541784+0000 mon.a (mon.0) 2608 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:47:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:19.548579+0000 mon.a (mon.0) 2609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:47:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:19.548628+0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e454: 8 total, 7 up, 4 in 2024-02-13T00:47:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:20.434405+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e455: 8 total, 7 up, 4 in 2024-02-13T00:47:21.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:21.137 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:21.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:20.437503+0000 mgr.y (mgr.14544) 400 : cluster [DBG] pgmap v668: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:47:21.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:21.377476+0000 osd.4 (osd.4) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:21.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:21.377900+0000 osd.4 (osd.4) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:21.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:20.437503+0000 mgr.y (mgr.14544) 400 : cluster [DBG] pgmap v668: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:47:21.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:21.377476+0000 osd.4 (osd.4) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:21.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:21.377900+0000 osd.4 (osd.4) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:21.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:21.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:21.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:21.917 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:21.939 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:21.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:21.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:22.044 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:20.437503+0000 mgr.y (mgr.14544) 400 : cluster [DBG] pgmap v668: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:47:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:21.377476+0000 osd.4 (osd.4) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:21.377900+0000 osd.4 (osd.4) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:22.072 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:47:22.072 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:47:22.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24739.0:3 16.f 16:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:00.558130+0000", 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 381.50812090199997, 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.081761433, 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24739", 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2737601038", 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3 2024-02-13T00:47:22.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558130+0000", 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558130+0000", 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558128+0000", 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2910000000000002e-06 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558135+0000", 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0115999999999999e-05 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558145+0000", 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2560000000000002e-06 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558153+0000", 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0846e-05 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558214+0000", 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8210000000000008e-06 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558223+0000", 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.017115333 2024-02-13T00:47:22.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.575338+0000", 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122046 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.575460+0000", 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001568549 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-02-13T00:47:22.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.577029+0000", 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012430968000000001 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589460+0000", 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015196783 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.604657+0000", 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034770550999999997 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.639427+0000", 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040860799999999997 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.639836+0000", 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5262e-05 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.639891+0000", 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.081761433 2024-02-13T00:47:22.079 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24763.0:2 23.d 23:b36015c1:test-rados-api-smithi162-84089-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.495169+0000", 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 380.57108143900001, 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22330878300000001, 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24763", 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/90739862", 2024-02-13T00:47:22.080 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495169+0000", 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495169+0000", 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495167+0000", 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1899999999999995e-06 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495175+0000", 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0474e-05 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495186+0000", 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9990000000000008e-06 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495194+0000", 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3149000000000002e-05 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495247+0000", 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.533e-06 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495254+0000", 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031521518999999998 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.526775+0000", 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0169999999999996e-06 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.526783+0000", 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.170767906 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697551+0000", 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3193000000000003e-05 2024-02-13T00:47:22.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697614+0000", 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8979000000000002e-05 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,4", 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697713+0000", 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012975179999999999 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.699011+0000", 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003049412 2024-02-13T00:47:22.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.702060+0000", 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016330172 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.718390+0000", 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8174999999999999e-05 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.718438+0000", 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9382999999999999e-05 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.718478+0000", 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22330878300000001 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24767.0:2 14.1f 14:feb32ed4:test-rados-api-smithi162-84161-3::testobj:head [write 0~8] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:47:22.087 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.597458+0000", 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 380.46879212800002, 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.074387914999999999, 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24767", 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2472368574", 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597458+0000", 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597458+0000", 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597456+0000", 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.696e-06 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597466+0000", 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0883999999999999e-05 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597477+0000", 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7740000000000004e-06 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597485+0000", 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029994400000000001 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597785+0000", 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113497 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597898+0000", 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015973699999999999 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.598058+0000", 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020561050000000001 2024-02-13T00:47:22.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.618619+0000", 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033024080000000002 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.621921+0000", 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049844082999999997 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.092 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.671766+0000", 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4239000000000001e-05 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.671810+0000", 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6464999999999997e-05 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.671846+0000", 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.074387914999999999 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.093 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:441 20.14 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head [setxattr a196 in=4b] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.493977+0000", 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 379.57227304600002, 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16342337300000001, 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 441 2024-02-13T00:47:22.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.493977+0000", 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.493977+0000", 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.493975+0000", 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3498e-05 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.493988+0000", 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5116999999999999e-05 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494003+0000", 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3450000000000006e-06 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494012+0000", 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0686999999999998e-05 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494062+0000", 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6800999999999998e-05 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494149+0000", 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147002 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494296+0000", 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001189708 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.495486+0000", 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000325693 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.495812+0000", 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.161531377 2024-02-13T00:47:22.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.657343+0000", 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1778e-05 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.657375+0000", 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5942999999999999e-05 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.657401+0000", 2024-02-13T00:47:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16342337300000001 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:1809 20.14 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head [setxattr a880 in=4b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:03.990416+0000", 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 378.07583432500002, 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055387081999999997, 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:47:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:47:22.101 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1809 2024-02-13T00:47:22.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.101 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990416+0000", 2024-02-13T00:47:22.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990416+0000", 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990415+0000", 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7190000000000004e-06 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990422+0000", 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5320000000000006e-06 2024-02-13T00:47:22.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990430+0000", 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0110000000000003e-06 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990437+0000", 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043852149999999996 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.994823+0000", 2024-02-13T00:47:22.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7717000000000001e-05 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.994900+0000", 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013310199999999999 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.995033+0000", 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089106199999999997 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:47:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.995924+0000", 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047824802 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.043749+0000", 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00194904 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.045698+0000", 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8034e-05 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.045756+0000", 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6875000000000001e-05 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.045803+0000", 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055387081999999997 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14886.0:2067 20.1e e109/105 20:78e1aec5:test-rados-api-smithi162-84049-17::small:head v 109'3, mlcod=109'1)", 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.582878+0000", 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 377.48337284600001, 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043476111999999997, 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582878+0000", 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582878+0000", 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582876+0000", 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2574e-05 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582889+0000", 2024-02-13T00:47:22.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5733999999999999e-05 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582905+0000", 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4379999999999998e-06 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582912+0000", 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9762999999999997e-05 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582972+0000", 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3986999999999999e-05 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582996+0000", 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043311409000000002 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.626307+0000", 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6532000000000001e-05 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.626354+0000", 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043476111999999997 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2070 20.b 20:d3a22371:test-rados-api-smithi162-84049-17::big2:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.742315+0000", 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 377.32393551000001, 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.100851074, 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.111 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.112 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:47:22.112 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:47:22.112 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2070 2024-02-13T00:47:22.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.112 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742315+0000", 2024-02-13T00:47:22.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742315+0000", 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742313+0000", 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011733019000000001 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754046+0000", 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3737e-05 2024-02-13T00:47:22.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754070+0000", 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0747999999999999e-05 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754081+0000", 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1600000000000001e-05 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754132+0000", 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6428999999999994e-05 2024-02-13T00:47:22.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754229+0000", 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106959 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,6", 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754336+0000", 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036365882000000002 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:47:22.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.790702+0000", 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.052175179000000002 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.842877+0000", 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000231101 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.843108+0000", 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3439000000000003e-05 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.843141+0000", 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4810000000000001e-05 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.843166+0000", 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.100851074 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16275.0:35 66.8 66:110c41bf:test-rados-api-smithi162-84081-10::foo:head [write 128~128] snapc b=[b,a] ondisk+write+known_if_redirected e135)", 2024-02-13T00:47:22.117 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:30.279104+0000", 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 351.78714599800003, 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044119924999999997, 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16275", 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/238911824", 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 35 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279104+0000", 2024-02-13T00:47:22.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279104+0000", 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279103+0000", 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1609999999999997e-06 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279110+0000", 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9039999999999998e-06 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279119+0000", 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8399999999999997e-06 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279126+0000", 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026694828 2024-02-13T00:47:22.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.305820+0000", 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2825999999999995e-05 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.305913+0000", 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018201799999999999 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.306095+0000", 2024-02-13T00:47:22.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025887029999999999 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.308684+0000", 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013424475999999999 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.322109+0000", 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010389150000000001 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.323147+0000", 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0759999999999997e-05 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.323178+0000", 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6211000000000002e-05 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.323224+0000", 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044119924999999997 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.123 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16275.0:42 66.8 66:110c41bf:test-rados-api-smithi162-84081-10::foo:head [write 128~128] snapc c=[c,b,a] ondisk+write+known_if_redirected e136)", 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:31.308000+0000", 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 350.75825064100002, 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075133532000000003, 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16275", 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/238911824", 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 42 2024-02-13T00:47:22.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308000+0000", 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308000+0000", 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.307997+0000", 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5245999999999999e-05 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308012+0000", 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8859e-05 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308031+0000", 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0020999999999999e-05 2024-02-13T00:47:22.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308041+0000", 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014614509999999999 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.309503+0000", 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8609999999999998e-06 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.309511+0000", 2024-02-13T00:47:22.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043170606 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.352681+0000", 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010319699999999999 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.352784+0000", 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016862800000000001 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.352953+0000", 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002530109 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.355483+0000", 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0083680739999999997 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.363851+0000", 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019179723999999999 2024-02-13T00:47:22.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383031+0000", 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6895000000000003e-05 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383088+0000", 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5457000000000002e-05 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383133+0000", 2024-02-13T00:47:22.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075133532000000003 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.35941.0:2 84.6 e139/138 84:602f83fe:::foo:head v 139'1, mlcod=0'0)", 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:34.483127+0000", 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 347.58312314300002, 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041127036999999998, 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483127+0000", 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483127+0000", 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483127+0000", 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3149999999999999e-06 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483132+0000", 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0050000000000004e-06 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483137+0000", 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1290000000000001e-06 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483143+0000", 2024-02-13T00:47:22.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022933839999999999 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.485437+0000", 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6535e-05 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.485453+0000", 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038765455999999997 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.524219+0000", 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5552000000000001e-05 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.524254+0000", 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041127036999999998 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.43732.0:2 112.6 112:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e172)", 2024-02-13T00:47:22.135 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:07.426136+0000", 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 314.640114203, 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058639376999999999, 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.43732", 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/981068687", 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426136+0000", 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426136+0000", 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426134+0000", 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5360000000000004e-06 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426142+0000", 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0329999999999999e-05 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426152+0000", 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5750000000000003e-06 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426160+0000", 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4138999999999998e-05 2024-02-13T00:47:22.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426214+0000", 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6590000000000007e-06 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426222+0000", 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014917939 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.441140+0000", 2024-02-13T00:47:22.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109622 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.441249+0000", 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015227799999999999 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.441401+0000", 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028880260000000001 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.444289+0000", 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033112480000000001 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.447601+0000", 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037075416 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.484676+0000", 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2821999999999997e-05 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.484729+0000", 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6717e-05 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.484776+0000", 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058639376999999999 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36767.0:1 140.6s0 140:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:41.361654+0000", 2024-02-13T00:47:22.142 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 280.70459654699999, 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99434686900000002, 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36767", 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1509178592", 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361654+0000", 2024-02-13T00:47:22.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361654+0000", 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361653+0000", 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2099999999999998e-06 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361659+0000", 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1799999999999996e-06 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361667+0000", 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8100000000000003e-06 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361673+0000", 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0318000000000002e-05 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361723+0000", 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5279999999999999e-06 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361727+0000", 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98580988800000002 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.347536+0000", 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149586 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.347686+0000", 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038543399999999998 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.348071+0000", 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0075773510000000004 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.355649+0000", 2024-02-13T00:47:22.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027853600000000002 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.355927+0000", 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3433e-05 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.356001+0000", 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99434686900000002 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36824.0:1 148.6s1 148:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e220)", 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:55.339285+0000", 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 266.72696556099999, 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0318101019999999, 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36824", 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4039137531", 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339285+0000", 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339285+0000", 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339283+0000", 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8839999999999994e-06 2024-02-13T00:47:22.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339291+0000", 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1019000000000001e-05 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339302+0000", 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6029999999999994e-06 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339312+0000", 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061423129999999999 2024-02-13T00:47:22.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.345454+0000", 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3289999999999998e-06 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.345457+0000", 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.020285004 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.365742+0000", 2024-02-13T00:47:22.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137575 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.365880+0000", 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040855500000000001 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.366289+0000", 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051522999999999998 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:47:22.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.366804+0000", 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004224088 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.371028+0000", 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7168999999999993e-05 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.371095+0000", 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0318101019999999 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26868.0:1 170.7s1 170:fd8eaea7:nspace::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:43:35.394195+0000", 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 226.67205571299999, 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.199555962, 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26868", 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1602322330", 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394195+0000", 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394195+0000", 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394193+0000", 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1409999999999999e-06 2024-02-13T00:47:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394200+0000", 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1063000000000001e-05 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394211+0000", 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0160000000000008e-06 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394219+0000", 2024-02-13T00:47:22.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4419999999999996e-05 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394284+0000", 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2720000000000002e-06 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394287+0000", 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1938998459999999 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:36.588187+0000", 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011178799999999999 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:36.588298+0000", 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038829199999999999 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:36.588687+0000", 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050951100000000004 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:47:22.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:36.589196+0000", 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044877179999999999 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:36.593684+0000", 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6774999999999997e-05 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:36.593751+0000", 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.199555962 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.160 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27057.0:1 190.5s2 190:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e319)", 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:34.404917+0000", 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 167.66133385200001, 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.331389991, 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27057", 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4276929747", 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404917+0000", 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404917+0000", 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404915+0000", 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7039999999999997e-06 2024-02-13T00:47:22.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404922+0000", 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0203999999999999e-05 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404932+0000", 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9459999999999998e-06 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404940+0000", 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022950620000000001 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.407235+0000", 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5379999999999998e-06 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.407238+0000", 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3289484380000001 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:35.736187+0000", 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011990899999999999 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:35.736307+0000", 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.331389991 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24329.0:1153 5.3 5:c609908c:::notify.5:head [watch ping cookie 94514175194752 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:39.296113+0000", 2024-02-13T00:47:22.165 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.770137522, 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223829202, 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24329", 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1153 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296113+0000", 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296113+0000", 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296110+0000", 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3057000000000001e-05 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296123+0000", 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0857999999999999e-05 2024-02-13T00:47:22.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296144+0000", 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1107999999999999e-05 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296155+0000", 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112284 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296268+0000", 2024-02-13T00:47:22.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3674e-05 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296281+0000", 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.197784131 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.494065+0000", 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034288400000000001 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.494408+0000", 2024-02-13T00:47:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025363974000000001 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.519772+0000", 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5661999999999994e-05 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.519858+0000", 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4176000000000002e-05 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.519942+0000", 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223829202 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24329.0:1162 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94514175194752 gen 2] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:39.297175+0000", 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.76907557999999, 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22574032099999999, 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.171 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24329", 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1162 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297175+0000", 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297175+0000", 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:47:22.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297174+0000", 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4430000000000002e-06 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297179+0000", 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0529e-05 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297190+0000", 2024-02-13T00:47:22.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2519999999999999e-06 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297195+0000", 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2176999999999999e-05 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297247+0000", 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0348e-05 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297258+0000", 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19719434299999999 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.494452+0000", 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1651999999999999e-05 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.494514+0000", 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025459631 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.519973+0000", 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0639000000000002e-05 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.520014+0000", 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002901299 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.522915+0000", 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22574032099999999 2024-02-13T00:47:22.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25162.0:4733 191.6 e335/323 191:602f83fe:::foo:head v 335'11, mlcod=333'9)", 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:50.443597+0000", 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 151.62265391299999, 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051008379, 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443597+0000", 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443597+0000", 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443596+0000", 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2199999999999992e-06 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443604+0000", 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3690000000000003e-06 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443611+0000", 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4969999999999998e-06 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443617+0000", 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7141999999999999e-05 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443714+0000", 2024-02-13T00:47:22.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9049999999999999e-05 2024-02-13T00:47:22.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443733+0000", 2024-02-13T00:47:22.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050843781999999997 2024-02-13T00:47:22.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.494577+0000", 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7654e-05 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.494605+0000", 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051008379 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24748.0:1139 18.4 18:20000000::::head [pgnls start_epoch 348] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e348)", 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:45:03.458723+0000", 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 138.60752700399999, 2024-02-13T00:47:22.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94893485399999999, 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24748", 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2831164779", 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1139 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458723+0000", 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458723+0000", 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458723+0000", 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9849999999999997e-06 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458728+0000", 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1950000000000003e-06 2024-02-13T00:47:22.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458736+0000", 2024-02-13T00:47:22.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4649999999999996e-06 2024-02-13T00:47:22.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458740+0000", 2024-02-13T00:47:22.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9974999999999993e-05 2024-02-13T00:47:22.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458810+0000", 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0842999999999999e-05 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458821+0000", 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94869083399999998 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:04.407512+0000", 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0757999999999998e-05 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:04.407553+0000", 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105543 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:04.407658+0000", 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94893485399999999 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.186 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.37256.0:1 197.2s1 197:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:45:05.418265+0000", 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 136.64798591499999, 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.004095116, 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.37256", 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1003782898", 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:47:22.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418265+0000", 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418265+0000", 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418262+0000", 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0108999999999999e-05 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418272+0000", 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1304000000000001e-05 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418283+0000", 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7669999999999992e-06 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418292+0000", 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00073741699999999998 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.419030+0000", 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9870000000000001e-06 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.419034+0000", 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99819390399999997 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.417227+0000", 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013077100000000001 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.417358+0000", 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00046316100000000002 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.417821+0000", 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041701299999999999 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.418238+0000", 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040563220000000002 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.422295+0000", 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4969999999999996e-05 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.422360+0000", 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.004095116 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:22.192 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:47:22.193 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:22.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:47:22.621 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:22.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:22.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:22.754 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:21.557781+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e456: 8 total, 7 up, 4 in 2024-02-13T00:47:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:21.557781+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e456: 8 total, 7 up, 4 in 2024-02-13T00:47:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:21.557781+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e456: 8 total, 7 up, 4 in 2024-02-13T00:47:23.102 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:23.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:23.207 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:23.240 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:23.260 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:23.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:23.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:23.392 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:22.437863+0000 mgr.y (mgr.14544) 401 : cluster [DBG] pgmap v670: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:47:23.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:22.437863+0000 mgr.y (mgr.14544) 401 : cluster [DBG] pgmap v670: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:47:23.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:23.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:23.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:23.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:23.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.560+0000 7fcc030b3700 1 -- 172.21.15.162:0/3971041066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0f8480 msgr2=0x7fcbfc104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:23.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.560+0000 7fcc030b3700 1 --2- 172.21.15.162:0/3971041066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0f8480 0x7fcbfc104a80 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7fcbf4009be0 tx=0x7fcbf4004d10).stop 2024-02-13T00:47:23.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.560+0000 7fcc030b3700 1 -- 172.21.15.162:0/3971041066 shutdown_connections 2024-02-13T00:47:23.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.560+0000 7fcc030b3700 1 --2- 172.21.15.162:0/3971041066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0f8480 0x7fcbfc104a80 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:23.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.560+0000 7fcc030b3700 1 --2- 172.21.15.162:0/3971041066 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbfc0fdc10 0x7fcbfc0fdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:23.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.560+0000 7fcc030b3700 1 --2- 172.21.15.162:0/3971041066 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbfc0fe570 0x7fcbfc0f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:23.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.560+0000 7fcc030b3700 1 -- 172.21.15.162:0/3971041066 >> 172.21.15.162:0/3971041066 conn(0x7fcbfc0f3800 msgr2=0x7fcbfc0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:23.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.560+0000 7fcc030b3700 1 -- 172.21.15.162:0/3971041066 shutdown_connections 2024-02-13T00:47:23.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.560+0000 7fcc030b3700 1 -- 172.21.15.162:0/3971041066 wait complete. 2024-02-13T00:47:23.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.560+0000 7fcc030b3700 1 librados: starting msgr at 2024-02-13T00:47:23.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.560+0000 7fcc030b3700 1 librados: starting objecter 2024-02-13T00:47:23.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc030b3700 1 Processor -- start 2024-02-13T00:47:23.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc030b3700 1 -- start start 2024-02-13T00:47:23.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc030b3700 1 librados: setting wanted keys 2024-02-13T00:47:23.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc030b3700 1 librados: calling monclient init 2024-02-13T00:47:23.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc030b3700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbfc0f8480 0x7fcbfc06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:23.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc030b3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbfc0fdc10 0x7fcbfc06b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:23.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc030b3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fe570 0x7fcbfc06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:23.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc030b3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc116410 con 0x7fcbfc0fe570 2024-02-13T00:47:23.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc030b3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc116580 con 0x7fcbfc0fdc10 2024-02-13T00:47:23.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc030b3700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc116740 con 0x7fcbfc0f8480 2024-02-13T00:47:23.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc01650700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fe570 0x7fcbfc06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:23.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc00e4f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbfc0f8480 0x7fcbfc06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:23.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc01650700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fe570 0x7fcbfc06bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44480/0 (socket says 172.21.15.162:44480) 2024-02-13T00:47:23.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc01650700 1 -- 172.21.15.162:0/1697711182 learned_addr learned my addr 172.21.15.162:0/1697711182 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:23.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcbfbfff700 1 --2- 172.21.15.162:0/1697711182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbfc0fdc10 0x7fcbfc06b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:23.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc01650700 1 -- 172.21.15.162:0/1697711182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbfc0f8480 msgr2=0x7fcbfc06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc01650700 1 --2- 172.21.15.162:0/1697711182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbfc0f8480 0x7fcbfc06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc01650700 1 -- 172.21.15.162:0/1697711182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbfc0fdc10 msgr2=0x7fcbfc06b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc01650700 1 --2- 172.21.15.162:0/1697711182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbfc0fdc10 0x7fcbfc06b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc01650700 1 -- 172.21.15.162:0/1697711182 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf40097c0 con 0x7fcbfc0fe570 2024-02-13T00:47:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.561+0000 7fcc00e4f700 1 --2- 172.21.15.162:0/1697711182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbfc0f8480 0x7fcbfc06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:47:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.562+0000 7fcc01650700 1 --2- 172.21.15.162:0/1697711182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc0fe570 0x7fcbfc06bb50 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7fcbf4004a30 tx=0x7fcbf4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.562+0000 7fcbf9ffb700 10 client.?.objecter ms_handle_connect 0x7fcbfc0fe570 2024-02-13T00:47:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.562+0000 7fcbf9ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.563+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf400e040 con 0x7fcbfc0fe570 2024-02-13T00:47:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.563+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbfc1169c0 con 0x7fcbfc0fe570 2024-02-13T00:47:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.563+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbf4018040 con 0x7fcbfc0fe570 2024-02-13T00:47:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.563+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf401c3f0 con 0x7fcbfc0fe570 2024-02-13T00:47:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.563+0000 7fcc030b3700 10 client.27435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.563+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbfc116dc0 con 0x7fcbfc0fe570 2024-02-13T00:47:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.563+0000 7fcc030b3700 1 librados: init done 2024-02-13T00:47:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.564+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fcbfc05fb20 con 0x7fcbfc0fe570 2024-02-13T00:47:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.564+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fcbf4007b20 con 0x7fcbfc0fe570 2024-02-13T00:47:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcbf9ffb700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcbec062c90 0x7fcbec065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(456..456 src has 52..456) v4 ==== 7967+0+0 (secure 0 0 0) 0x7fcbf407e9d0 con 0x7fcbfc0fe570 2024-02-13T00:47:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 osd_map(456..456 src has 52..456) v4 2024-02-13T00:47:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcbf9ffb700 3 client.27435.objecter handle_osd_map got epochs [456,456] > 0 2024-02-13T00:47:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcbf9ffb700 3 client.27435.objecter handle_osd_map decoding full epoch 456 2024-02-13T00:47:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc00e4f700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcbec062c90 0x7fcbec065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcbf9ffb700 20 client.27435.objecter dump_active .. 0 homeless 2024-02-13T00:47:23.776 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:47:23.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v456) v1 ==== 54+0+16 (secure 0 0 0) 0x7fcbf407f0a0 con 0x7fcbfc0fe570 2024-02-13T00:47:23.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc030b3700 10 client.27435.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:23.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 1 2024-02-13T00:47:23.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc030b3700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6801/2103743338 conn(0x7fcbfc0feee0 0x7fcbfc0ff300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc0475e0 osd=0 3 2024-02-13T00:47:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc0475e0 osd=0 3 2024-02-13T00:47:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc0475e0 osd=0 3 2024-02-13T00:47:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 1 2024-02-13T00:47:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc0475e0 osd=0 3 2024-02-13T00:47:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 0 1 2024-02-13T00:47:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc0475e0 osd=0 4 2024-02-13T00:47:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc030b3700 10 client.27435.objecter _send_command 1 2024-02-13T00:47:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbfc107170 con 0x7fcbfc0feee0 2024-02-13T00:47:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcbf9ffb700 10 client.27435.objecter ms_handle_connect 0x7fcbec062c90 2024-02-13T00:47:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcc00e4f700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcbec062c90 0x7fcbec065150 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 rx=0x7fcbf000f2f0 tx=0x7fcbf0009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcbfbfff700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6801/2103743338 conn(0x7fcbfc0feee0 0x7fcbfc0ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.566+0000 7fcbf9ffb700 10 client.27435.objecter ms_handle_connect 0x7fcbfc0feee0 2024-02-13T00:47:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.565+0000 7fcbfbfff700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6801/2103743338 conn(0x7fcbfc0feee0 0x7fcbfc0ff300 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.566+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbfc107170 con 0x7fcbfc0feee0 2024-02-13T00:47:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.566+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:47:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.566+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 1 = system:0 2024-02-13T00:47:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.566+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc0475e0 osd=0 4 2024-02-13T00:47:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.566+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 0 1 2024-02-13T00:47:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.581+0000 7fcc030b3700 10 client.27435.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.581+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 2 2024-02-13T00:47:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.581+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc0475e0 osd=0 3 2024-02-13T00:47:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.581+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc0475e0 osd=0 3 2024-02-13T00:47:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.581+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc0475e0 osd=0 3 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.581+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 2 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.581+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc0475e0 osd=0 3 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.581+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 0 2 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.581+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc0475e0 osd=0 4 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.581+0000 7fcc030b3700 10 client.27435.objecter _send_command 2 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.581+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcbfc027410 con 0x7fcbfc0feee0 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcbfc027410 con 0x7fcbfc0feee0 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc0475e0 osd=0 4 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 0 2 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcc030b3700 10 client.27435.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:23.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 3 2024-02-13T00:47:23.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcc030b3700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6805/150599780 conn(0x7fcbfc059990 0x7fcbfc059dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:23.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc028760 osd=1 3 2024-02-13T00:47:23.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc028760 osd=1 3 2024-02-13T00:47:23.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc028760 osd=1 3 2024-02-13T00:47:23.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 3 2024-02-13T00:47:23.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc028760 osd=1 3 2024-02-13T00:47:23.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 1 3 2024-02-13T00:47:23.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc028760 osd=1 4 2024-02-13T00:47:23.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcc030b3700 10 client.27435.objecter _send_command 3 2024-02-13T00:47:23.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.582+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbfc11d760 con 0x7fcbfc059990 2024-02-13T00:47:23.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.583+0000 7fcc00e4f700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6805/150599780 conn(0x7fcbfc059990 0x7fcbfc059dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:23.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.583+0000 7fcbf9ffb700 10 client.27435.objecter ms_handle_connect 0x7fcbfc059990 2024-02-13T00:47:23.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.583+0000 7fcc00e4f700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6805/150599780 conn(0x7fcbfc059990 0x7fcbfc059dd0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:23.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.583+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbfc11d760 con 0x7fcbfc059990 2024-02-13T00:47:23.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.583+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:47:23.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.583+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 3 = system:0 2024-02-13T00:47:23.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.583+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc028760 osd=1 4 2024-02-13T00:47:23.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.583+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 1 3 2024-02-13T00:47:23.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.598+0000 7fcc030b3700 10 client.27435.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:23.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.598+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 4 2024-02-13T00:47:23.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.598+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc028760 osd=1 3 2024-02-13T00:47:23.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.598+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc028760 osd=1 3 2024-02-13T00:47:23.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.598+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc028760 osd=1 3 2024-02-13T00:47:23.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.598+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 4 2024-02-13T00:47:23.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.598+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc028760 osd=1 3 2024-02-13T00:47:23.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.598+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 1 4 2024-02-13T00:47:23.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.598+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc028760 osd=1 4 2024-02-13T00:47:23.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.598+0000 7fcc030b3700 10 client.27435.objecter _send_command 4 2024-02-13T00:47:23.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.598+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcbfc059400 con 0x7fcbfc059990 2024-02-13T00:47:23.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcbfc059400 con 0x7fcbfc059990 2024-02-13T00:47:23.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:23.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc028760 osd=1 4 2024-02-13T00:47:23.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 1 4 2024-02-13T00:47:23.783 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:47:23.783 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:47:23.783 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:47:23.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcc030b3700 10 client.27435.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:23.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 5 2024-02-13T00:47:23.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcc030b3700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6809/3689719376 conn(0x7fcbfc0282c0 0x7fcbfc11c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:23.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc059810 osd=2 3 2024-02-13T00:47:23.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc059810 osd=2 3 2024-02-13T00:47:23.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc059810 osd=2 3 2024-02-13T00:47:23.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 5 2024-02-13T00:47:23.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc059810 osd=2 3 2024-02-13T00:47:23.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 2 5 2024-02-13T00:47:23.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc059810 osd=2 4 2024-02-13T00:47:23.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcc030b3700 10 client.27435.objecter _send_command 5 2024-02-13T00:47:23.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbfc11c670 con 0x7fcbfc0282c0 2024-02-13T00:47:23.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.599+0000 7fcbfbfff700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6809/3689719376 conn(0x7fcbfc0282c0 0x7fcbfc11c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:23.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.600+0000 7fcbf9ffb700 10 client.27435.objecter ms_handle_connect 0x7fcbfc0282c0 2024-02-13T00:47:23.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.600+0000 7fcbfbfff700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6809/3689719376 conn(0x7fcbfc0282c0 0x7fcbfc11c180 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:23.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.600+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbfc11c670 con 0x7fcbfc0282c0 2024-02-13T00:47:23.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.600+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:47:23.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.600+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 5 = system:0 2024-02-13T00:47:23.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.600+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc059810 osd=2 4 2024-02-13T00:47:23.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.600+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 2 5 2024-02-13T00:47:23.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.615+0000 7fcc030b3700 10 client.27435.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:23.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.615+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 6 2024-02-13T00:47:23.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.615+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc059810 osd=2 3 2024-02-13T00:47:23.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.615+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc059810 osd=2 3 2024-02-13T00:47:23.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.615+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc059810 osd=2 3 2024-02-13T00:47:23.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.615+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 6 2024-02-13T00:47:23.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.615+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc059810 osd=2 3 2024-02-13T00:47:23.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.615+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 2 6 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.615+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc059810 osd=2 4 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.615+0000 7fcc030b3700 10 client.27435.objecter _send_command 6 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.615+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcbfc027820 con 0x7fcbfc0282c0 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcbfc027820 con 0x7fcbfc0282c0 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc059810 osd=2 4 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 2 6 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc030b3700 10 client.27435.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 7 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc030b3700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6813/604971011 conn(0x7fcbfc11f8e0 0x7fcbfc121cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc11d060 osd=3 3 2024-02-13T00:47:23.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc11d060 osd=3 3 2024-02-13T00:47:23.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc11d060 osd=3 3 2024-02-13T00:47:23.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 7 2024-02-13T00:47:23.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc11d060 osd=3 3 2024-02-13T00:47:23.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 3 7 2024-02-13T00:47:23.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc11d060 osd=3 4 2024-02-13T00:47:23.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc030b3700 10 client.27435.objecter _send_command 7 2024-02-13T00:47:23.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbfc124920 con 0x7fcbfc11f8e0 2024-02-13T00:47:23.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.616+0000 7fcc01650700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6813/604971011 conn(0x7fcbfc11f8e0 0x7fcbfc121cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:23.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.617+0000 7fcbf9ffb700 10 client.27435.objecter ms_handle_connect 0x7fcbfc11f8e0 2024-02-13T00:47:23.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.617+0000 7fcc01650700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.162:6813/604971011 conn(0x7fcbfc11f8e0 0x7fcbfc121cd0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:23.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.617+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbfc124920 con 0x7fcbfc11f8e0 2024-02-13T00:47:23.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.617+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:47:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.617+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 7 = system:0 2024-02-13T00:47:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.617+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc11d060 osd=3 4 2024-02-13T00:47:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.617+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 3 7 2024-02-13T00:47:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.631+0000 7fcc030b3700 10 client.27435.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.631+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 8 2024-02-13T00:47:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.631+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc11d060 osd=3 3 2024-02-13T00:47:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.631+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc11d060 osd=3 3 2024-02-13T00:47:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.631+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc11d060 osd=3 3 2024-02-13T00:47:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.631+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 8 2024-02-13T00:47:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.631+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc11d060 osd=3 3 2024-02-13T00:47:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.631+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 3 8 2024-02-13T00:47:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.631+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc11d060 osd=3 4 2024-02-13T00:47:23.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.631+0000 7fcc030b3700 10 client.27435.objecter _send_command 8 2024-02-13T00:47:23.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.631+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcbfc11cad0 con 0x7fcbfc11f8e0 2024-02-13T00:47:23.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcbfc11cad0 con 0x7fcbfc11f8e0 2024-02-13T00:47:23.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:23.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc11d060 osd=3 4 2024-02-13T00:47:23.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 3 8 2024-02-13T00:47:23.789 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:47:23.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc030b3700 10 client.27435.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:23.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 9 2024-02-13T00:47:23.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc030b3700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.181:6800/3344263388 conn(0x7fcbfc11bce0 0x7fcbfc123340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:23.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc11cad0 osd=4 3 2024-02-13T00:47:23.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc11cad0 osd=4 3 2024-02-13T00:47:23.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc11cad0 osd=4 3 2024-02-13T00:47:23.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 9 2024-02-13T00:47:23.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc11cad0 osd=4 3 2024-02-13T00:47:23.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 4 9 2024-02-13T00:47:23.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc11cad0 osd=4 4 2024-02-13T00:47:23.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc030b3700 10 client.27435.objecter _send_command 9 2024-02-13T00:47:23.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbfc123870 con 0x7fcbfc11bce0 2024-02-13T00:47:23.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.633+0000 7fcc00e4f700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.181:6800/3344263388 conn(0x7fcbfc11bce0 0x7fcbfc123340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:23.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.634+0000 7fcbf9ffb700 10 client.27435.objecter ms_handle_connect 0x7fcbfc11bce0 2024-02-13T00:47:23.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.634+0000 7fcc00e4f700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.181:6800/3344263388 conn(0x7fcbfc11bce0 0x7fcbfc123340 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:23.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.634+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbfc123870 con 0x7fcbfc11bce0 2024-02-13T00:47:23.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.634+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:47:23.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.634+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 9 = system:0 2024-02-13T00:47:23.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.634+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc11cad0 osd=4 4 2024-02-13T00:47:23.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.634+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 4 9 2024-02-13T00:47:23.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.651+0000 7fcc030b3700 10 client.27435.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:23.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.651+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 10 2024-02-13T00:47:23.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.651+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc11cad0 osd=4 3 2024-02-13T00:47:23.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.651+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc11cad0 osd=4 3 2024-02-13T00:47:23.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.651+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc11cad0 osd=4 3 2024-02-13T00:47:23.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.651+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 10 2024-02-13T00:47:23.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.651+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc11cad0 osd=4 3 2024-02-13T00:47:23.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.651+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 4 10 2024-02-13T00:47:23.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.651+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc11cad0 osd=4 4 2024-02-13T00:47:23.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.651+0000 7fcc030b3700 10 client.27435.objecter _send_command 10 2024-02-13T00:47:23.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.651+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcbfc123870 con 0x7fcbfc11bce0 2024-02-13T00:47:23.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcbfc123870 con 0x7fcbfc11bce0 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc11cad0 osd=4 4 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 4 10 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 10 client.27435.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 11 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 20 client.27435.objecter _get_session osd=-1 returning homeless 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 10 client.27435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=457}) v3 -- 0x7fcbfc123a90 con 0x7fcbfc0fe570 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcbfc123d90 con 0x7fcbfc0fe570 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_get_version_reply(handle=1 version=456) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbf4004810 con 0x7fcbfc0fe570 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcbfaffd700 10 client.27435.objecter _check_command_map_dne tid 11 current 456 map_dne_bound 456 2024-02-13T00:47:23.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcbfaffd700 10 client.27435.objecter _finish_command 11 = osd:6 osd down 2024-02-13T00:47:23.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcbfaffd700 15 client.27435.objecter _session_command_op_remove -1 11 2024-02-13T00:47:23.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 10 client.27435.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:23.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 12 2024-02-13T00:47:23.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.181:6808/1506640341 conn(0x7fcbfc126aa0 0x7fcbfc128e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:23.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc1246d0 osd=6 3 2024-02-13T00:47:23.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc1246d0 osd=6 3 2024-02-13T00:47:23.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc1246d0 osd=6 3 2024-02-13T00:47:23.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 12 2024-02-13T00:47:23.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc1246d0 osd=6 3 2024-02-13T00:47:23.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 6 12 2024-02-13T00:47:23.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc1246d0 osd=6 4 2024-02-13T00:47:23.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 10 client.27435.objecter _send_command 12 2024-02-13T00:47:23.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.653+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.181:6808/1506640341 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbfc12bae0 con 0x7fcbfc126aa0 2024-02-13T00:47:23.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.654+0000 7fcbfbfff700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.181:6808/1506640341 conn(0x7fcbfc126aa0 0x7fcbfc128e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:23.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.654+0000 7fcbf9ffb700 10 client.27435.objecter ms_handle_connect 0x7fcbfc126aa0 2024-02-13T00:47:23.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.654+0000 7fcbfbfff700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.181:6808/1506640341 conn(0x7fcbfc126aa0 0x7fcbfc128e90 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:23.795 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:47:23.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.655+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbfc12bae0 con 0x7fcbfc126aa0 2024-02-13T00:47:23.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.655+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 12: 0 ) v1 2024-02-13T00:47:23.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.655+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 12 = system:0 2024-02-13T00:47:23.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.655+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc1246d0 osd=6 4 2024-02-13T00:47:23.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.655+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 6 12 2024-02-13T00:47:23.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.669+0000 7fcc030b3700 10 client.27435.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:23.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.669+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 13 2024-02-13T00:47:23.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.669+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc1246d0 osd=6 3 2024-02-13T00:47:23.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.669+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc1246d0 osd=6 3 2024-02-13T00:47:23.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.669+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc1246d0 osd=6 3 2024-02-13T00:47:23.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.669+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 13 2024-02-13T00:47:23.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.669+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc1246d0 osd=6 3 2024-02-13T00:47:23.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.669+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 6 13 2024-02-13T00:47:23.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.669+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc1246d0 osd=6 4 2024-02-13T00:47:23.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.669+0000 7fcc030b3700 10 client.27435.objecter _send_command 13 2024-02-13T00:47:23.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.669+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcbfc129380 con 0x7fcbfc126aa0 2024-02-13T00:47:23.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcbfc129380 con 0x7fcbfc126aa0 2024-02-13T00:47:23.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:23.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc1246d0 osd=6 4 2024-02-13T00:47:23.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 6 13 2024-02-13T00:47:23.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcc030b3700 10 client.27435.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:23.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 14 2024-02-13T00:47:23.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcc030b3700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.181:6812/1964192772 conn(0x7fcbfc129a40 0x7fcbfc129e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:23.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc129380 osd=7 3 2024-02-13T00:47:23.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc129380 osd=7 3 2024-02-13T00:47:23.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc129380 osd=7 3 2024-02-13T00:47:23.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 14 2024-02-13T00:47:23.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc129380 osd=7 3 2024-02-13T00:47:23.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 7 14 2024-02-13T00:47:23.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc129380 osd=7 4 2024-02-13T00:47:23.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcc030b3700 10 client.27435.objecter _send_command 14 2024-02-13T00:47:23.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.671+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.181:6812/1964192772 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbfc12bad0 con 0x7fcbfc129a40 2024-02-13T00:47:23.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.672+0000 7fcc01650700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.181:6812/1964192772 conn(0x7fcbfc129a40 0x7fcbfc129e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:23.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.672+0000 7fcbf9ffb700 10 client.27435.objecter ms_handle_connect 0x7fcbfc129a40 2024-02-13T00:47:23.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.672+0000 7fcc01650700 1 --2- 172.21.15.162:0/1697711182 >> v2:172.21.15.181:6812/1964192772 conn(0x7fcbfc129a40 0x7fcbfc129e80 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:23.799 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:47:23.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.673+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbfc12bad0 con 0x7fcbfc129a40 2024-02-13T00:47:23.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.673+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 14: 0 ) v1 2024-02-13T00:47:23.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.673+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 14 = system:0 2024-02-13T00:47:23.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.673+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc129380 osd=7 4 2024-02-13T00:47:23.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.673+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 7 14 2024-02-13T00:47:23.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.687+0000 7fcc030b3700 10 client.27435.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:23.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.687+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign -1 15 2024-02-13T00:47:23.800 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.687+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc129380 osd=7 3 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.687+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc129380 osd=7 3 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.687+0000 7fcc030b3700 20 client.27435.objecter _get_session s=0x7fcbfc129380 osd=7 3 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.687+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_remove -1 15 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.687+0000 7fcc030b3700 20 client.27435.objecter get_session s=0x7fcbfc129380 osd=7 3 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.687+0000 7fcc030b3700 15 client.27435.objecter _session_command_op_assign 7 15 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.687+0000 7fcc030b3700 20 client.27435.objecter put_session s=0x7fcbfc129380 osd=7 4 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.687+0000 7fcc030b3700 10 client.27435.objecter _send_command 15 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.687+0000 7fcc030b3700 1 -- 172.21.15.162:0/1697711182 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcbfc12bad0 con 0x7fcbfc129a40 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.689+0000 7fcbf9ffb700 1 -- 172.21.15.162:0/1697711182 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcbfc12bad0 con 0x7fcbfc129a40 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.689+0000 7fcbf9ffb700 10 client.27435.objecter ms_dispatch 0x7fcbfc001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.689+0000 7fcbf9ffb700 10 client.27435.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.689+0000 7fcbf9ffb700 20 client.27435.objecter put_session s=0x7fcbfc129380 osd=7 4 2024-02-13T00:47:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:23.689+0000 7fcbf9ffb700 15 client.27435.objecter _session_command_op_remove 7 15 2024-02-13T00:47:23.802 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.802 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.802 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.802 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.802 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.802 INFO:teuthology.orchestra.run.smithi162.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-02-13T00:47:23.802 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.802 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:47:23.802 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:22.437863+0000 mgr.y (mgr.14544) 401 : cluster [DBG] pgmap v670: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:47:24.559 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-13T00:47:24.560 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-13T00:47:24.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:23.565177+0000 mon.a (mon.0) 2613 : audit [DBG] from='client.? 172.21.15.162:0/1697711182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:24.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:23.565177+0000 mon.a (mon.0) 2613 : audit [DBG] from='client.? 172.21.15.162:0/1697711182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:25.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:23.565177+0000 mon.a (mon.0) 2613 : audit [DBG] from='client.? 172.21.15.162:0/1697711182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:25.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-13T00:47:25.374 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:25.639 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:24.438352+0000 mgr.y (mgr.14544) 402 : cluster [DBG] pgmap v671: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 868 B/s rd, 0 op/s 2024-02-13T00:47:25.640 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:25.437888+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e457: 8 total, 7 up, 4 in 2024-02-13T00:47:25.640 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:24.438352+0000 mgr.y (mgr.14544) 402 : cluster [DBG] pgmap v671: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 868 B/s rd, 0 op/s 2024-02-13T00:47:25.640 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:25.437888+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e457: 8 total, 7 up, 4 in 2024-02-13T00:47:25.723 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:25.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:25.826 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:25.856 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:47:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:47:25.857 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:24.438352+0000 mgr.y (mgr.14544) 402 : cluster [DBG] pgmap v671: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 868 B/s rd, 0 op/s 2024-02-13T00:47:25.857 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:25.437888+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e457: 8 total, 7 up, 4 in 2024-02-13T00:47:25.858 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:25.880 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:25.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:25.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:25.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:26.010 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6] out_osds: [3, 2, 7, 5] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-02-13T00:47:26.010 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:47:26.010 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-13T00:47:26.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-13T00:47:26.012 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:26.212 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:26.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:26.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:26.248 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:26.284 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:26.317 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:26.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:47:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:47:26] "GET /metrics HTTP/1.1" 200 177633 "" "Prometheus/2.33.4" 2024-02-13T00:47:26.351 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:26.371 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:26.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:26.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:26.504 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:26.523 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:26.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:26.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:47:26.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:26.655 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:26.744 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:25.664659+0000 mgr.y (mgr.14544) 403 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:26.745 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:25.664659+0000 mgr.y (mgr.14544) 403 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:26.759 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:26.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:26.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:25.664659+0000 mgr.y (mgr.14544) 403 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:26.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:26.894 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:26.915 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:26.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:27.019 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:27.047 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:27.067 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:27.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:27.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:27.171 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:27.199 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:27.346 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:27.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:27.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:27.481 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:27.501 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:27.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:27.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:27.626 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:26.438759+0000 mgr.y (mgr.14544) 404 : cluster [DBG] pgmap v673: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-13T00:47:27.626 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:26.438759+0000 mgr.y (mgr.14544) 404 : cluster [DBG] pgmap v673: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-13T00:47:27.634 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:27.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:26.438759+0000 mgr.y (mgr.14544) 404 : cluster [DBG] pgmap v673: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-13T00:47:29.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:47:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:47:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:47:29.186 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:29.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:29.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:29.318 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:29.390 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:29.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:29.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:29.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:29.524 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:29.665 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:28.439205+0000 mgr.y (mgr.14544) 405 : cluster [DBG] pgmap v674: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 KiB/s rd, 383 B/s wr, 2 op/s 2024-02-13T00:47:29.666 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:28.439205+0000 mgr.y (mgr.14544) 405 : cluster [DBG] pgmap v674: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 KiB/s rd, 383 B/s wr, 2 op/s 2024-02-13T00:47:29.671 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:29.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:29.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:29.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:29.803 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:29.918 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:47:29.919 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:47:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:28.439205+0000 mgr.y (mgr.14544) 405 : cluster [DBG] pgmap v674: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 KiB/s rd, 383 B/s wr, 2 op/s 2024-02-13T00:47:30.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.109+0000 7f4644496700 1 -- 172.21.15.162:0/1753236080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f463c102270 msgr2=0x7f463c104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:30.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.109+0000 7f4644496700 1 --2- 172.21.15.162:0/1753236080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f463c102270 0x7f463c104660 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7f4634009be0 tx=0x7f4634004d10).stop 2024-02-13T00:47:30.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.109+0000 7f4644496700 1 -- 172.21.15.162:0/1753236080 shutdown_connections 2024-02-13T00:47:30.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.109+0000 7f4644496700 1 --2- 172.21.15.162:0/1753236080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f463c102270 0x7f463c104660 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:30.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.109+0000 7f4644496700 1 --2- 172.21.15.162:0/1753236080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f463c0f8d10 0x7f463c0f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:47:30.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.109+0000 7f4644496700 1 --2- 172.21.15.162:0/1753236080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f463c0f95e0 0x7f463c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:30.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.109+0000 7f4644496700 1 -- 172.21.15.162:0/1753236080 >> 172.21.15.162:0/1753236080 conn(0x7f463c06eb90 msgr2=0x7f463c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:30.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.109+0000 7f4644496700 1 -- 172.21.15.162:0/1753236080 shutdown_connections 2024-02-13T00:47:30.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.109+0000 7f4644496700 1 -- 172.21.15.162:0/1753236080 wait complete. 2024-02-13T00:47:30.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.110+0000 7f4644496700 1 librados: starting msgr at 2024-02-13T00:47:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.110+0000 7f4644496700 1 librados: starting objecter 2024-02-13T00:47:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.110+0000 7f4644496700 1 Processor -- start 2024-02-13T00:47:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.110+0000 7f4644496700 1 -- start start 2024-02-13T00:47:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.110+0000 7f4644496700 1 librados: setting wanted keys 2024-02-13T00:47:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.110+0000 7f4644496700 1 librados: calling monclient init 2024-02-13T00:47:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.110+0000 7f4644496700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f463c0f8d10 0x7f463c10e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.110+0000 7f4644496700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f463c0f95e0 0x7f463c10e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.110+0000 7f4644496700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f463c102270 0x7f463c1082c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.110+0000 7f4644496700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c1087b0 con 0x7f463c102270 2024-02-13T00:47:30.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.110+0000 7f4644496700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c108920 con 0x7f463c0f95e0 2024-02-13T00:47:30.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.110+0000 7f4644496700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463c108a90 con 0x7f463c0f8d10 2024-02-13T00:47:30.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4642232700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f463c0f8d10 0x7f463c10e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:30.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4642a33700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f463c102270 0x7f463c1082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:30.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4642232700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f463c0f8d10 0x7f463c10e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:58470/0 (socket says 172.21.15.162:58470) 2024-02-13T00:47:30.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4642232700 1 -- 172.21.15.162:0/2769154100 learned_addr learned my addr 172.21.15.162:0/2769154100 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:30.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4642a33700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f463c102270 0x7f463c1082c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:40838/0 (socket says 172.21.15.162:40838) 2024-02-13T00:47:30.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4641a31700 1 --2- 172.21.15.162:0/2769154100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f463c0f95e0 0x7f463c10e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:30.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4642232700 1 -- 172.21.15.162:0/2769154100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f463c0f95e0 msgr2=0x7f463c10e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:30.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4642232700 1 --2- 172.21.15.162:0/2769154100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f463c0f95e0 0x7f463c10e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:30.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4642232700 1 -- 172.21.15.162:0/2769154100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f463c102270 msgr2=0x7f463c1082c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:30.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4642232700 1 --2- 172.21.15.162:0/2769154100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f463c102270 0x7f463c1082c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:30.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4642232700 1 -- 172.21.15.162:0/2769154100 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46340097c0 con 0x7f463c0f8d10 2024-02-13T00:47:30.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4642a33700 1 --2- 172.21.15.162:0/2769154100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f463c102270 0x7f463c1082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:47:30.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4642232700 1 --2- 172.21.15.162:0/2769154100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f463c0f8d10 0x7f463c10e090 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7f463800b530 tx=0x7f46380090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:30.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f46337fe700 10 client.?.objecter ms_handle_connect 0x7f463c0f8d10 2024-02-13T00:47:30.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f46337fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:30.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f46337fe700 1 -- 172.21.15.162:0/2769154100 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638002320 con 0x7f463c0f8d10 2024-02-13T00:47:30.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f46337fe700 1 -- 172.21.15.162:0/2769154100 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4638002950 con 0x7f463c0f8d10 2024-02-13T00:47:30.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4644496700 1 -- 172.21.15.162:0/2769154100 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f463c108d70 con 0x7f463c0f8d10 2024-02-13T00:47:30.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f46337fe700 1 -- 172.21.15.162:0/2769154100 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638012950 con 0x7f463c0f8d10 2024-02-13T00:47:30.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4644496700 10 client.37541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:30.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4644496700 1 -- 172.21.15.162:0/2769154100 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463c0f6b00 con 0x7f463c0f8d10 2024-02-13T00:47:30.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.111+0000 7f4644496700 1 librados: init done 2024-02-13T00:47:30.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.112+0000 7f4644496700 1 -- 172.21.15.162:0/2769154100 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463c1071b0 con 0x7f463c0f8d10 2024-02-13T00:47:30.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.113+0000 7f46337fe700 1 -- 172.21.15.162:0/2769154100 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4638002480 con 0x7f463c0f8d10 2024-02-13T00:47:30.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.113+0000 7f46337fe700 1 --2- 172.21.15.162:0/2769154100 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4628062c90 0x7f4628065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:30.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.113+0000 7f46337fe700 1 -- 172.21.15.162:0/2769154100 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(457..457 src has 52..457) v4 ==== 7967+0+0 (secure 0 0 0) 0x7f463807d180 con 0x7f463c0f8d10 2024-02-13T00:47:30.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.113+0000 7f46337fe700 10 client.37541.objecter ms_dispatch 0x7f463c001550 osd_map(457..457 src has 52..457) v4 2024-02-13T00:47:30.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.113+0000 7f46337fe700 3 client.37541.objecter handle_osd_map got epochs [457,457] > 0 2024-02-13T00:47:30.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.113+0000 7f46337fe700 3 client.37541.objecter handle_osd_map decoding full epoch 457 2024-02-13T00:47:30.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.113+0000 7f4641a31700 1 --2- 172.21.15.162:0/2769154100 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4628062c90 0x7f4628065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:30.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.113+0000 7f46337fe700 20 client.37541.objecter dump_active .. 0 homeless 2024-02-13T00:47:30.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.113+0000 7f46337fe700 10 client.37541.objecter ms_handle_connect 0x7f4628062c90 2024-02-13T00:47:30.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.113+0000 7f4641a31700 1 --2- 172.21.15.162:0/2769154100 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4628062c90 0x7f4628065150 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 rx=0x7f463c109a10 tx=0x7f462c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:30.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.118+0000 7f46337fe700 1 -- 172.21.15.162:0/2769154100 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f46380506f0 con 0x7f463c0f8d10 2024-02-13T00:47:30.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.288+0000 7f4644496700 1 -- 172.21.15.162:0/2769154100 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f463c109750 con 0x7f463c0f8d10 2024-02-13T00:47:30.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.289+0000 7f46337fe700 1 -- 172.21.15.162:0/2769154100 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v457) v1 ==== 81+0+47654 (secure 0 0 0) 0x7f463c109750 con 0x7f463c0f8d10 2024-02-13T00:47:30.294 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:47:30.294 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:47:30.294 INFO:teuthology.orchestra.run.smithi162.stdout: "epoch": 457, 2024-02-13T00:47:30.294 INFO:teuthology.orchestra.run.smithi162.stdout: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:47:30.294 INFO:teuthology.orchestra.run.smithi162.stdout: "created": "2024-02-13T00:31:51.797017+0000", 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "modified": "2024-02-13T00:47:25.430722+0000", 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "last_up_change": "2024-02-13T00:46:56.548676+0000", 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "last_in_change": "2024-02-13T00:46:00.399226+0000", 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_num": 5806080, 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_set": [ 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "nodeep-scrub", 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "noscrub", 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "pglog_hardlimit", 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "purged_snapdirs", 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_deletes", 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "sortbitwise" 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:47:30.295 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_version": 103, 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: "full_ratio": 0.94999998807907104, 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_snapshot": "", 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_max": 216, 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: "max_osd": 8, 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: "require_min_compat_client": "luminous", 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: "min_compat_client": "luminous", 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: "require_osd_release": "pacific", 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: "pools": [ 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 1, 2024-02-13T00:47:30.296 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "device_health_metrics", 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:35:34.545390+0000", 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 1, 2024-02-13T00:47:30.297 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 1, 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 1, 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 1, 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 1, 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "71", 2024-02-13T00:47:30.298 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:47:30.299 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:47:30.300 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:47:30.301 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:47:30.301 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:47:30.301 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:47:30.301 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:47:30.301 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:47:30.301 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:47:30.301 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:47:30.301 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:47:30.301 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:47:30.301 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:47:30.301 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_max": 32, 2024-02-13T00:47:30.302 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_min": 1 2024-02-13T00:47:30.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.302 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:47:30.302 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_devicehealth": {} 2024-02-13T00:47:30.302 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.302 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 2, 2024-02-13T00:47:30.302 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": ".rgw.root", 2024-02-13T00:47:30.302 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.459980+0000", 2024-02-13T00:47:30.302 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:47:30.302 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:47:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:47:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:47:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:47:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:47:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:47:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:47:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:47:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:47:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:47:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:47:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:47:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "58", 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:47:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:47:30.305 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:47:30.306 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.307 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 3, 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "iscsi", 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.740219+0000", 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:47:30.308 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 3, 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 3, 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 3, 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 3, 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 3, 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "102", 2024-02-13T00:47:30.309 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:47:30.310 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:47:30.311 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:47:30.312 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:47:30.312 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:47:30.312 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:47:30.312 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:47:30.312 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:47:30.312 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:47:30.312 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:47:30.312 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd": {} 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 4, 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.log", 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:41.489292+0000", 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:47:30.313 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:47:30.314 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "60", 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:47:30.315 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:47:30.316 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:47:30.317 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 5, 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.control", 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:43.466304+0000", 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:47:30.318 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:47:30.319 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "62", 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:47:30.320 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:47:30.321 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:47:30.322 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 6, 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.meta", 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:45.542442+0000", 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:47:30.323 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 42, 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 42, 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 42, 2024-02-13T00:47:30.324 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 42, 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 42, 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "248", 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "246", 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:47:30.325 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:47:30.326 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:47:30.327 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_bias": 4 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.328 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 28, 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:00.190777+0000", 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:47:30.329 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.330 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "108", 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:47:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.332 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.333 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 61, 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-6", 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:14.963279+0000", 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8193, 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:47:30.334 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "450", 2024-02-13T00:47:30.335 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "390", 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "390", 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "390", 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 32, 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 450, 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:47:30.336 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:47:30.337 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: "fingerprint_algorithm": "sha1" 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:47:30.338 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 216, 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-69", 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:47:10.762502+0000", 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:47:30.339 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:47:30.340 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "449", 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:47:30.341 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:47:30.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:47:30.343 INFO:teuthology.orchestra.run.smithi162.stdout: "rados": {} 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99", 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:47:30.344 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 8, 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 447, 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6801", 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:47:30.345 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6802", 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6804", 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:47:30.346 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6803", 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6801/2103743338", 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6802/2103743338", 2024-02-13T00:47:30.347 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6804/2103743338", 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6803/2103743338", 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886", 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:47:30.348 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 13, 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 447, 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6805", 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6806", 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.350 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6808", 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6807", 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6805/150599780", 2024-02-13T00:47:30.351 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6806/150599780", 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6808/150599780", 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6807/150599780", 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e", 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:47:30.352 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 18, 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 333, 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6809", 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.353 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6810", 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.354 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6812", 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6811", 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.355 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6809/3689719376", 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6810/3689719376", 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6812/3689719376", 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6811/3689719376", 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001", 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:47:30.356 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 25, 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 255, 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6813", 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.357 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6814", 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.358 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6816", 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6815", 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.359 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6813/604971011", 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6814/604971011", 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6816/604971011", 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6815/604971011", 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3", 2024-02-13T00:47:30.360 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 31, 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 81, 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.361 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6800", 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6801", 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.362 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6803", 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6802", 2024-02-13T00:47:30.363 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6800/3344263388", 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6801/3344263388", 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6803/3344263388", 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6802/3344263388", 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.364 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723", 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 0, 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 39, 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 399, 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 437, 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6804", 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6805", 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6807", 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.367 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6806", 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2212296748 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6804/2212296748", 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6805/2212296748", 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2212296748", 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2212296748", 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: "exists" 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474", 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 45, 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 447, 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:47:30.369 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6808", 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6809", 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:47:30.370 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6811", 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6810", 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6808/1506640341", 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6809/1506640341", 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1506640341", 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1506640341", 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:47:30.372 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b", 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 53, 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 215, 2024-02-13T00:47:30.373 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6812", 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.374 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6813", 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6815", 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6814", 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6812/1964192772", 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6813/1964192772", 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6815/1964192772", 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6814/1964192772", 2024-02-13T00:47:30.376 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_xinfo": [ 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:47:30.377 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:47:05.584144+0000", 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:47:15.727103+0000", 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.378 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:46:53.108061+0000", 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:47:30.379 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:46:25.732999+0000", 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:47:21.377903+0000", 2024-02-13T00:47:30.380 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:45:51.711776+0000", 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 436 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:47:30.381 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:47:19.221397+0000", 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:47:30.382 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:46:25.961583+0000", 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap": [], 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap_items": [ 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.4", 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 0 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "5.5", 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 6 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "28.12", 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:47:30.384 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 4 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_temp": [], 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_temp": [], 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: "blocklist": { 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/1909610488": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2370739667": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/1502590187": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/4121673848": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2412382714": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3041562701": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:47:30.385 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/2775563314": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3135632190": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/197307636": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2653247729": "2024-02-14T00:32:24.380625+0000" 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: "range_blocklist": {}, 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profiles": { 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: "default": { 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van" 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps_queue": [ 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 24, 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.387 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 9, 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 25, 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:47:30.388 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 60, 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 78, 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.390 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 85, 2024-02-13T00:47:30.391 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 94, 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:47:30.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 95, 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 4, 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 4 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 107, 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 113, 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.394 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.395 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 125, 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.396 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 10, 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 133, 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 12, 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 154, 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 14, 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 166, 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 17, 2024-02-13T00:47:30.398 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 176, 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 20, 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:47:30.399 INFO:teuthology.orchestra.run.smithi162.stdout: "new_removed_snaps": [], 2024-02-13T00:47:30.400 INFO:teuthology.orchestra.run.smithi162.stdout: "new_purged_snaps": [], 2024-02-13T00:47:30.400 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_node_flags": {}, 2024-02-13T00:47:30.400 INFO:teuthology.orchestra.run.smithi162.stdout: "device_class_flags": {}, 2024-02-13T00:47:30.400 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode": { 2024-02-13T00:47:30.400 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_enabled": false, 2024-02-13T00:47:30.400 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_bucket_count": 0, 2024-02-13T00:47:30.400 INFO:teuthology.orchestra.run.smithi162.stdout: "degraded_stretch_mode": 0, 2024-02-13T00:47:30.400 INFO:teuthology.orchestra.run.smithi162.stdout: "recovering_stretch_mode": 0, 2024-02-13T00:47:30.400 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_bucket": 0 2024-02-13T00:47:30.400 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:47:30.400 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:47:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 10 librados: watch_flush enter 2024-02-13T00:47:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 10 librados: watch_flush exit 2024-02-13T00:47:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 20 client.37541.objecter shutdown clearing up homeless session... 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 10 client.37541.objecter successfully canceled tick 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 -- 172.21.15.162:0/2769154100 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4628062c90 msgr2=0x7f4628065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 --2- 172.21.15.162:0/2769154100 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4628062c90 0x7f4628065150 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 rx=0x7f463c109a10 tx=0x7f462c00b040).stop 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 -- 172.21.15.162:0/2769154100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f463c0f8d10 msgr2=0x7f463c10e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 --2- 172.21.15.162:0/2769154100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f463c0f8d10 0x7f463c10e090 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7f463800b530 tx=0x7f46380090f0).stop 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 -- 172.21.15.162:0/2769154100 shutdown_connections 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 --2- 172.21.15.162:0/2769154100 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4628062c90 0x7f4628065150 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 --2- 172.21.15.162:0/2769154100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f463c102270 0x7f463c1082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 --2- 172.21.15.162:0/2769154100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f463c0f8d10 0x7f463c10e090 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 --2- 172.21.15.162:0/2769154100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f463c0f95e0 0x7f463c10e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 -- 172.21.15.162:0/2769154100 >> 172.21.15.162:0/2769154100 conn(0x7f463c06eb90 msgr2=0x7f463c102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 -- 172.21.15.162:0/2769154100 shutdown_connections 2024-02-13T00:47:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 -- 172.21.15.162:0/2769154100 wait complete. 2024-02-13T00:47:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:30.291+0000 7f4644496700 1 librados: shutdown 2024-02-13T00:47:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:30.289985+0000 mon.c (mon.2) 388 : audit [DBG] from='client.? 172.21.15.162:0/2769154100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:47:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:30.289985+0000 mon.c (mon.2) 388 : audit [DBG] from='client.? 172.21.15.162:0/2769154100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:47:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:30.289985+0000 mon.c (mon.2) 388 : audit [DBG] from='client.? 172.21.15.162:0/2769154100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:47:31.153 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:31.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:31.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:31.292 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:31.834 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:31.843 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:31.846 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:30.440596+0000 mgr.y (mgr.14544) 406 : cluster [DBG] pgmap v675: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.6 KiB/s rd, 345 B/s wr, 3 op/s 2024-02-13T00:47:31.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:30.440596+0000 mgr.y (mgr.14544) 406 : cluster [DBG] pgmap v675: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.6 KiB/s rd, 345 B/s wr, 3 op/s 2024-02-13T00:47:31.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:31.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:31.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-13T00:47:31.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:31.976 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:30.440596+0000 mgr.y (mgr.14544) 406 : cluster [DBG] pgmap v675: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.6 KiB/s rd, 345 B/s wr, 3 op/s 2024-02-13T00:47:32.053 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:32.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:32.158 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:32.193 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:32.215 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:32.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:32.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:32.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:32.347 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:32.366 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:32.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:32.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:32.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:32.498 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:32.517 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:32.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:32.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:32.622 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:32.484873+0000 mon.c (mon.2) 389 : audit [INF] from='client.? 172.21.15.162:0/671796156' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:47:32.622 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:32.485151+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:47:32.622 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:32.484873+0000 mon.c (mon.2) 389 : audit [INF] from='client.? 172.21.15.162:0/671796156' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:47:32.622 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:32.485151+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:47:32.651 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:32.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.296+0000 7fa4d5657700 1 -- 172.21.15.162:0/3235140461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d00f8d10 msgr2=0x7fa4d00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:32.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.296+0000 7fa4d5657700 1 --2- 172.21.15.162:0/3235140461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d00f8d10 0x7fa4d00f90f0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 rx=0x7fa4c40047b0 tx=0x7fa4c4009cb0).stop 2024-02-13T00:47:32.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.296+0000 7fa4d5657700 1 -- 172.21.15.162:0/3235140461 shutdown_connections 2024-02-13T00:47:32.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.296+0000 7fa4d5657700 1 --2- 172.21.15.162:0/3235140461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4d0102270 0x7fa4d0104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:32.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.296+0000 7fa4d5657700 1 --2- 172.21.15.162:0/3235140461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d00f8d10 0x7fa4d00f90f0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:32.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.296+0000 7fa4d5657700 1 --2- 172.21.15.162:0/3235140461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4d00f95e0 0x7fa4d0101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:32.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.296+0000 7fa4d5657700 1 -- 172.21.15.162:0/3235140461 >> 172.21.15.162:0/3235140461 conn(0x7fa4d006eb90 msgr2=0x7fa4d00f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:32.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.296+0000 7fa4d5657700 1 -- 172.21.15.162:0/3235140461 shutdown_connections 2024-02-13T00:47:32.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.296+0000 7fa4d5657700 1 -- 172.21.15.162:0/3235140461 wait complete. 2024-02-13T00:47:32.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.296+0000 7fa4d5657700 1 librados: starting msgr at 2024-02-13T00:47:32.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.296+0000 7fa4d5657700 1 librados: starting objecter 2024-02-13T00:47:32.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.296+0000 7fa4d5657700 1 Processor -- start 2024-02-13T00:47:32.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4d5657700 1 -- start start 2024-02-13T00:47:32.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4d5657700 1 librados: setting wanted keys 2024-02-13T00:47:32.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4d5657700 1 librados: calling monclient init 2024-02-13T00:47:32.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4d5657700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4d00f8d10 0x7fa4d010e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:32.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4d5657700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d00f95e0 0x7fa4d010e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:32.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4d5657700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4d0102270 0x7fa4d01082c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:32.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4d5657700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d01087b0 con 0x7fa4d00f95e0 2024-02-13T00:47:32.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4d5657700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0108920 con 0x7fa4d00f8d10 2024-02-13T00:47:32.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4d5657700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0108ae0 con 0x7fa4d0102270 2024-02-13T00:47:32.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4ce7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d00f95e0 0x7fa4d010e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:32.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4cf7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4d0102270 0x7fa4d01082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:32.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4ce7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d00f95e0 0x7fa4d010e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:40856/0 (socket says 172.21.15.162:40856) 2024-02-13T00:47:32.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4ce7fc700 1 -- 172.21.15.162:0/671796156 learned_addr learned my addr 172.21.15.162:0/671796156 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:32.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4ceffd700 1 --2- 172.21.15.162:0/671796156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4d00f8d10 0x7fa4d010e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:32.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4cf7fe700 1 -- 172.21.15.162:0/671796156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4d00f8d10 msgr2=0x7fa4d010e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:32.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4cf7fe700 1 --2- 172.21.15.162:0/671796156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4d00f8d10 0x7fa4d010e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:32.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4cf7fe700 1 -- 172.21.15.162:0/671796156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d00f95e0 msgr2=0x7fa4d010e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:32.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4cf7fe700 1 --2- 172.21.15.162:0/671796156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d00f95e0 0x7fa4d010e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:32.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4cf7fe700 1 -- 172.21.15.162:0/671796156 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c80097c0 con 0x7fa4d0102270 2024-02-13T00:47:32.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4ceffd700 1 --2- 172.21.15.162:0/671796156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4d00f8d10 0x7fa4d010e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:47:32.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4ce7fc700 1 --2- 172.21.15.162:0/671796156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d00f95e0 0x7fa4d010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:47:32.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4cf7fe700 1 --2- 172.21.15.162:0/671796156 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4d0102270 0x7fa4d01082c0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7fa4c8001dc0 tx=0x7fa4c8004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:32.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4b7fff700 10 client.?.objecter ms_handle_connect 0x7fa4d0102270 2024-02-13T00:47:32.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.297+0000 7fa4b7fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:32.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.298+0000 7fa4b7fff700 1 -- 172.21.15.162:0/671796156 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8009a10 con 0x7fa4d0102270 2024-02-13T00:47:32.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.298+0000 7fa4d5657700 1 -- 172.21.15.162:0/671796156 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c4005040 con 0x7fa4d0102270 2024-02-13T00:47:32.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.298+0000 7fa4b7fff700 1 -- 172.21.15.162:0/671796156 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4c80095a0 con 0x7fa4d0102270 2024-02-13T00:47:32.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.298+0000 7fa4b7fff700 1 -- 172.21.15.162:0/671796156 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8007d50 con 0x7fa4d0102270 2024-02-13T00:47:32.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.298+0000 7fa4d5657700 10 client.37547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:32.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.298+0000 7fa4d5657700 1 -- 172.21.15.162:0/671796156 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d0109120 con 0x7fa4d0102270 2024-02-13T00:47:32.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.298+0000 7fa4d5657700 1 librados: init done 2024-02-13T00:47:32.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.299+0000 7fa4d5657700 1 -- 172.21.15.162:0/671796156 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d00475e0 con 0x7fa4d0102270 2024-02-13T00:47:32.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.299+0000 7fa4b7fff700 1 -- 172.21.15.162:0/671796156 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fa4c8015630 con 0x7fa4d0102270 2024-02-13T00:47:32.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.299+0000 7fa4b7fff700 1 --2- 172.21.15.162:0/671796156 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa4b8062c90 0x7fa4b8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:32.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.299+0000 7fa4b7fff700 1 -- 172.21.15.162:0/671796156 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(457..457 src has 52..457) v4 ==== 7967+0+0 (secure 0 0 0) 0x7fa4c807dbe0 con 0x7fa4d0102270 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.299+0000 7fa4b7fff700 10 client.37547.objecter ms_dispatch 0x7fa4d0001550 osd_map(457..457 src has 52..457) v4 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.299+0000 7fa4b7fff700 3 client.37547.objecter handle_osd_map got epochs [457,457] > 0 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.299+0000 7fa4b7fff700 3 client.37547.objecter handle_osd_map decoding full epoch 457 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.299+0000 7fa4ceffd700 1 --2- 172.21.15.162:0/671796156 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa4b8062c90 0x7fa4b8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.299+0000 7fa4b7fff700 20 client.37547.objecter dump_active .. 0 homeless 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.300+0000 7fa4b7fff700 10 client.37547.objecter ms_handle_connect 0x7fa4b8062c90 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.300+0000 7fa4ceffd700 1 --2- 172.21.15.162:0/671796156 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa4b8062c90 0x7fa4b8065150 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 rx=0x7fa4c4004e20 tx=0x7fa4c4009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.303+0000 7fa4b7fff700 1 -- 172.21.15.162:0/671796156 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa4c80512a0 con 0x7fa4d0102270 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.482+0000 7fa4d5657700 1 -- 172.21.15.162:0/671796156 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fa4d01098c0 con 0x7fa4d0102270 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.588+0000 7fa4b7fff700 1 -- 172.21.15.162:0/671796156 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v458) v1 ==== 91+0+0 (secure 0 0 0) 0x7fa4d01098c0 con 0x7fa4d0102270 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 10 librados: watch_flush enter 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 10 librados: watch_flush exit 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 20 client.37547.objecter shutdown clearing up homeless session... 2024-02-13T00:47:32.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 10 client.37547.objecter successfully canceled tick 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 -- 172.21.15.162:0/671796156 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa4b8062c90 msgr2=0x7fa4b8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 --2- 172.21.15.162:0/671796156 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa4b8062c90 0x7fa4b8065150 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 rx=0x7fa4c4004e20 tx=0x7fa4c4009cb0).stop 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 -- 172.21.15.162:0/671796156 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4d0102270 msgr2=0x7fa4d01082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 --2- 172.21.15.162:0/671796156 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4d0102270 0x7fa4d01082c0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7fa4c8001dc0 tx=0x7fa4c8004d10).stop 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 -- 172.21.15.162:0/671796156 shutdown_connections 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 --2- 172.21.15.162:0/671796156 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa4b8062c90 0x7fa4b8065150 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 --2- 172.21.15.162:0/671796156 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4d0102270 0x7fa4d01082c0 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 --2- 172.21.15.162:0/671796156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4d00f8d10 0x7fa4d010e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 --2- 172.21.15.162:0/671796156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4d00f95e0 0x7fa4d010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 -- 172.21.15.162:0/671796156 >> 172.21.15.162:0/671796156 conn(0x7fa4d006eb90 msgr2=0x7fa4d0102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 -- 172.21.15.162:0/671796156 shutdown_connections 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 -- 172.21.15.162:0/671796156 wait complete. 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:32.591+0000 7fa4d5657700 1 librados: shutdown 2024-02-13T00:47:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:47:32.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:32.484873+0000 mon.c (mon.2) 389 : audit [INF] from='client.? 172.21.15.162:0/671796156' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:47:32.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:32.485151+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:47:33.458 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:33.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:33.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:33.563 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:33.593 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:33.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:32.441017+0000 mgr.y (mgr.14544) 407 : cluster [DBG] pgmap v676: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.4 KiB/s rd, 307 B/s wr, 2 op/s 2024-02-13T00:47:33.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:32.581342+0000 mon.a (mon.0) 2616 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:47:33.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:32.589684+0000 mon.a (mon.0) 2617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:47:33.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:32.589757+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e458: 8 total, 7 up, 4 in 2024-02-13T00:47:33.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:32.441017+0000 mgr.y (mgr.14544) 407 : cluster [DBG] pgmap v676: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.4 KiB/s rd, 307 B/s wr, 2 op/s 2024-02-13T00:47:33.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:32.581342+0000 mon.a (mon.0) 2616 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:47:33.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:32.589684+0000 mon.a (mon.0) 2617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:47:33.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:32.589757+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e458: 8 total, 7 up, 4 in 2024-02-13T00:47:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:32.441017+0000 mgr.y (mgr.14544) 407 : cluster [DBG] pgmap v676: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.4 KiB/s rd, 307 B/s wr, 2 op/s 2024-02-13T00:47:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:32.581342+0000 mon.a (mon.0) 2616 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:47:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:32.589684+0000 mon.a (mon.0) 2617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:47:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:32.589757+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e458: 8 total, 7 up, 4 in 2024-02-13T00:47:34.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 -- 172.21.15.162:0/2989457679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ba40f84a0 msgr2=0x7f8ba4104aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:34.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 --2- 172.21.15.162:0/2989457679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ba40f84a0 0x7f8ba4104aa0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7f8b9c009be0 tx=0x7f8b9c004d10).stop 2024-02-13T00:47:34.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 -- 172.21.15.162:0/2989457679 shutdown_connections 2024-02-13T00:47:34.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 --2- 172.21.15.162:0/2989457679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ba40f84a0 0x7f8ba4104aa0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:34.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 --2- 172.21.15.162:0/2989457679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ba40fdc30 0x7f8ba40fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:34.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 --2- 172.21.15.162:0/2989457679 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ba40fe590 0x7f8ba40f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 -- 172.21.15.162:0/2989457679 >> 172.21.15.162:0/2989457679 conn(0x7f8ba40f3880 msgr2=0x7f8ba40f5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 -- 172.21.15.162:0/2989457679 shutdown_connections 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 -- 172.21.15.162:0/2989457679 wait complete. 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 librados: starting msgr at 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 librados: starting objecter 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 Processor -- start 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 -- start start 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 librados: setting wanted keys 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.114+0000 7f8ba9466700 1 librados: calling monclient init 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba9466700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ba40f84a0 0x7f8ba406b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba9466700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ba40fdc30 0x7f8ba406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba9466700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ba40fe590 0x7f8ba406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:34.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba9466700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba4116410 con 0x7f8ba40f84a0 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba9466700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba4116580 con 0x7f8ba40fdc30 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba9466700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba4116740 con 0x7f8ba40fe590 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba37fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ba40fe590 0x7f8ba406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba37fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ba40fe590 0x7f8ba406bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:58504/0 (socket says 172.21.15.162:58504) 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba37fe700 1 -- 172.21.15.162:0/2996021839 learned_addr learned my addr 172.21.15.162:0/2996021839 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba2ffd700 1 --2- 172.21.15.162:0/2996021839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ba40f84a0 0x7f8ba406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba27fc700 1 --2- 172.21.15.162:0/2996021839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ba40fdc30 0x7f8ba406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba37fe700 1 -- 172.21.15.162:0/2996021839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ba40fdc30 msgr2=0x7f8ba406b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba37fe700 1 --2- 172.21.15.162:0/2996021839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ba40fdc30 0x7f8ba406b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba37fe700 1 -- 172.21.15.162:0/2996021839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ba40f84a0 msgr2=0x7f8ba406b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba37fe700 1 --2- 172.21.15.162:0/2996021839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ba40f84a0 0x7f8ba406b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba37fe700 1 -- 172.21.15.162:0/2996021839 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b98005040 con 0x7f8ba40fe590 2024-02-13T00:47:34.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba37fe700 1 --2- 172.21.15.162:0/2996021839 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ba40fe590 0x7f8ba406bb50 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 rx=0x7f8b9c004a30 tx=0x7f8b9c004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8b8bfff700 10 client.?.objecter ms_handle_connect 0x7f8ba40fe590 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8b8bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c00e040 con 0x7f8ba40fe590 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b9c018040 con 0x7f8ba40fe590 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba9466700 1 -- 172.21.15.162:0/2996021839 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9c0097c0 con 0x7f8ba40fe590 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c01c3f0 con 0x7f8ba40fe590 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba9466700 10 client.37553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba9466700 1 -- 172.21.15.162:0/2996021839 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba4116cb0 con 0x7f8ba40fe590 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.115+0000 7f8ba9466700 1 librados: init done 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.116+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f8b84004f00 con 0x7f8ba40fe590 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f8b9c013420 con 0x7f8ba40fe590 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b8bfff700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8b8c062c90 0x7f8b8c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(458..458 src has 52..458) v4 ==== 7967+0+0 (secure 0 0 0) 0x7f8b9c07e260 con 0x7f8ba40fe590 2024-02-13T00:47:34.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 osd_map(458..458 src has 52..458) v4 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b8bfff700 3 client.37553.objecter handle_osd_map got epochs [458,458] > 0 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b8bfff700 3 client.37553.objecter handle_osd_map decoding full epoch 458 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8ba2ffd700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8b8c062c90 0x7f8b8c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b8bfff700 20 client.37553.objecter dump_active .. 0 homeless 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v458) v1 ==== 54+0+16 (secure 0 0 0) 0x7f8b9c085070 con 0x7f8ba40fe590 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 1 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b8bfff700 10 client.37553.objecter ms_handle_connect 0x7f8b8c062c90 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8ba2ffd700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8b8c062c90 0x7f8b8c065150 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 rx=0x7f8b94004dc0 tx=0x7f8b9400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b89ffb700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6801/2103743338 conn(0x7f8b84004530 0x7f8b84004970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b840051a0 osd=0 3 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b840051a0 osd=0 3 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b840051a0 osd=0 3 2024-02-13T00:47:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 1 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b840051a0 osd=0 3 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 0 1 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b840051a0 osd=0 4 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b89ffb700 10 client.37553.objecter _send_command 1 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.117+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b84008930 con 0x7f8b84004530 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.118+0000 7f8ba27fc700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6801/2103743338 conn(0x7f8b84004530 0x7f8b84004970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.118+0000 7f8b8bfff700 10 client.37553.objecter ms_handle_connect 0x7f8b84004530 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.118+0000 7f8ba27fc700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6801/2103743338 conn(0x7f8b84004530 0x7f8b84004970 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.118+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8b84008930 con 0x7f8b84004530 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.118+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.118+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 1 = system:0 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.118+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b840051a0 osd=0 4 2024-02-13T00:47:34.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.118+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 0 1 2024-02-13T00:47:34.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.133+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:34.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.133+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 2 2024-02-13T00:47:34.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.133+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b840051a0 osd=0 3 2024-02-13T00:47:34.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.133+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b840051a0 osd=0 3 2024-02-13T00:47:34.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.133+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b840051a0 osd=0 3 2024-02-13T00:47:34.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.133+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 2 2024-02-13T00:47:34.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.133+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b840051a0 osd=0 3 2024-02-13T00:47:34.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.133+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 0 2 2024-02-13T00:47:34.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.133+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b840051a0 osd=0 4 2024-02-13T00:47:34.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.133+0000 7f8b89ffb700 10 client.37553.objecter _send_command 2 2024-02-13T00:47:34.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.133+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8b84003f70 con 0x7f8b84004530 2024-02-13T00:47:34.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8b84003f70 con 0x7f8b84004530 2024-02-13T00:47:34.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:34.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b840051a0 osd=0 4 2024-02-13T00:47:34.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 0 2 2024-02-13T00:47:34.140 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:47:34.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:34.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 3 2024-02-13T00:47:34.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b89ffb700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6805/150599780 conn(0x7f8b84008500 0x7f8b84002770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:34.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b840043a0 osd=1 3 2024-02-13T00:47:34.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b840043a0 osd=1 3 2024-02-13T00:47:34.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b840043a0 osd=1 3 2024-02-13T00:47:34.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 3 2024-02-13T00:47:34.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b840043a0 osd=1 3 2024-02-13T00:47:34.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 1 3 2024-02-13T00:47:34.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b840043a0 osd=1 4 2024-02-13T00:47:34.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b89ffb700 10 client.37553.objecter _send_command 3 2024-02-13T00:47:34.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b84002e40 con 0x7f8b84008500 2024-02-13T00:47:34.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.135+0000 7f8ba2ffd700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6805/150599780 conn(0x7f8b84008500 0x7f8b84002770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:34.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.136+0000 7f8b8bfff700 10 client.37553.objecter ms_handle_connect 0x7f8b84008500 2024-02-13T00:47:34.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.136+0000 7f8ba2ffd700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6805/150599780 conn(0x7f8b84008500 0x7f8b84002770 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:34.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.136+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8b84002e40 con 0x7f8b84008500 2024-02-13T00:47:34.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.136+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:47:34.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.136+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 3 = system:0 2024-02-13T00:47:34.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.136+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b840043a0 osd=1 4 2024-02-13T00:47:34.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.136+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 1 3 2024-02-13T00:47:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:34.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:34.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:34.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.152+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:34.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.152+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 4 2024-02-13T00:47:34.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.152+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b840043a0 osd=1 3 2024-02-13T00:47:34.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.152+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b840043a0 osd=1 3 2024-02-13T00:47:34.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.152+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b840043a0 osd=1 3 2024-02-13T00:47:34.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.152+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 4 2024-02-13T00:47:34.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.152+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b840043a0 osd=1 3 2024-02-13T00:47:34.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.152+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 1 4 2024-02-13T00:47:34.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.152+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b840043a0 osd=1 4 2024-02-13T00:47:34.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.152+0000 7f8b89ffb700 10 client.37553.objecter _send_command 4 2024-02-13T00:47:34.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.152+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8b84007ab0 con 0x7f8b84008500 2024-02-13T00:47:34.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.153+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8b84007ab0 con 0x7f8b84008500 2024-02-13T00:47:34.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.153+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:34.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.153+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.153+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b840043a0 osd=1 4 2024-02-13T00:47:34.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.153+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 1 4 2024-02-13T00:47:34.158 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:47:34.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:34.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 5 2024-02-13T00:47:34.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b89ffb700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6809/3689719376 conn(0x7f8b8400afc0 0x7f8b8400d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:34.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b84003830 osd=2 3 2024-02-13T00:47:34.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b84003830 osd=2 3 2024-02-13T00:47:34.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b84003830 osd=2 3 2024-02-13T00:47:34.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 5 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b84003830 osd=2 3 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 2 5 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b84003830 osd=2 4 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b89ffb700 10 client.37553.objecter _send_command 5 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b84010000 con 0x7f8b8400afc0 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8ba27fc700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6809/3689719376 conn(0x7f8b8400afc0 0x7f8b8400d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8b8bfff700 10 client.37553.objecter ms_handle_connect 0x7f8b8400afc0 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.154+0000 7f8ba27fc700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6809/3689719376 conn(0x7f8b8400afc0 0x7f8b8400d3b0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.155+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8b84010000 con 0x7f8b8400afc0 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.155+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.155+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 5 = system:0 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.155+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b84003830 osd=2 4 2024-02-13T00:47:34.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.155+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 2 5 2024-02-13T00:47:34.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.169+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:34.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.169+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 6 2024-02-13T00:47:34.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.169+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b84003830 osd=2 3 2024-02-13T00:47:34.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.169+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b84003830 osd=2 3 2024-02-13T00:47:34.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.169+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b84003830 osd=2 3 2024-02-13T00:47:34.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.169+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 6 2024-02-13T00:47:34.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.169+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b84003830 osd=2 3 2024-02-13T00:47:34.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.169+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 2 6 2024-02-13T00:47:34.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.169+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b84003830 osd=2 4 2024-02-13T00:47:34.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.169+0000 7f8b89ffb700 10 client.37553.objecter _send_command 6 2024-02-13T00:47:34.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.169+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8b84003300 con 0x7f8b8400afc0 2024-02-13T00:47:34.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8b84003300 con 0x7f8b8400afc0 2024-02-13T00:47:34.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:34.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b84003830 osd=2 4 2024-02-13T00:47:34.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 2 6 2024-02-13T00:47:34.175 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:47:34.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:34.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 7 2024-02-13T00:47:34.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b89ffb700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6813/604971011 conn(0x7f8b8400ea20 0x7f8b8400eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:34.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b84001b80 osd=3 3 2024-02-13T00:47:34.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b84001b80 osd=3 3 2024-02-13T00:47:34.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b84001b80 osd=3 3 2024-02-13T00:47:34.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 7 2024-02-13T00:47:34.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b84001b80 osd=3 3 2024-02-13T00:47:34.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 3 7 2024-02-13T00:47:34.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b84001b80 osd=3 4 2024-02-13T00:47:34.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b89ffb700 10 client.37553.objecter _send_command 7 2024-02-13T00:47:34.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b840022b0 con 0x7f8b8400ea20 2024-02-13T00:47:34.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.170+0000 7f8ba37fe700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6813/604971011 conn(0x7f8b8400ea20 0x7f8b8400eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:34.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.171+0000 7f8b8bfff700 10 client.37553.objecter ms_handle_connect 0x7f8b8400ea20 2024-02-13T00:47:34.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.171+0000 7f8ba37fe700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.162:6813/604971011 conn(0x7f8b8400ea20 0x7f8b8400eeb0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:34.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.171+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8b840022b0 con 0x7f8b8400ea20 2024-02-13T00:47:34.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.171+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:47:34.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.171+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 7 = system:0 2024-02-13T00:47:34.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.171+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b84001b80 osd=3 4 2024-02-13T00:47:34.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.171+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 3 7 2024-02-13T00:47:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.185+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.185+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 8 2024-02-13T00:47:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.185+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b84001b80 osd=3 3 2024-02-13T00:47:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.185+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b84001b80 osd=3 3 2024-02-13T00:47:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.185+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b84001b80 osd=3 3 2024-02-13T00:47:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.185+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 8 2024-02-13T00:47:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.185+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b84001b80 osd=3 3 2024-02-13T00:47:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.185+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 3 8 2024-02-13T00:47:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.185+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b84001b80 osd=3 4 2024-02-13T00:47:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.185+0000 7f8b89ffb700 10 client.37553.objecter _send_command 8 2024-02-13T00:47:34.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.185+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8b8400fb70 con 0x7f8b8400ea20 2024-02-13T00:47:34.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8b8400fb70 con 0x7f8b8400ea20 2024-02-13T00:47:34.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:34.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b84001b80 osd=3 4 2024-02-13T00:47:34.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 3 8 2024-02-13T00:47:34.192 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:47:34.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:34.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 9 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b89ffb700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.181:6800/3344263388 conn(0x7f8b84012180 0x7f8b84014570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b8400f3a0 osd=4 3 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b8400f3a0 osd=4 3 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b8400f3a0 osd=4 3 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 9 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b8400f3a0 osd=4 3 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 4 9 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b8400f3a0 osd=4 4 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b89ffb700 10 client.37553.objecter _send_command 9 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b8400fb50 con 0x7f8b84012180 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.187+0000 7f8ba2ffd700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.181:6800/3344263388 conn(0x7f8b84012180 0x7f8b84014570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.188+0000 7f8ba2ffd700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.181:6800/3344263388 conn(0x7f8b84012180 0x7f8b84014570 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.188+0000 7f8b8bfff700 10 client.37553.objecter ms_handle_connect 0x7f8b84012180 2024-02-13T00:47:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.189+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8b8400fb50 con 0x7f8b84012180 2024-02-13T00:47:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.189+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:47:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.189+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 9 = system:0 2024-02-13T00:47:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.189+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b8400f3a0 osd=4 4 2024-02-13T00:47:34.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.189+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 4 9 2024-02-13T00:47:34.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.203+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:34.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.203+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 10 2024-02-13T00:47:34.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.203+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b8400f3a0 osd=4 3 2024-02-13T00:47:34.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.203+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b8400f3a0 osd=4 3 2024-02-13T00:47:34.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.203+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b8400f3a0 osd=4 3 2024-02-13T00:47:34.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.203+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 10 2024-02-13T00:47:34.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.203+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b8400f3a0 osd=4 3 2024-02-13T00:47:34.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.203+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 4 10 2024-02-13T00:47:34.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.203+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b8400f3a0 osd=4 4 2024-02-13T00:47:34.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.203+0000 7f8b89ffb700 10 client.37553.objecter _send_command 10 2024-02-13T00:47:34.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.203+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8b8400fd90 con 0x7f8b84012180 2024-02-13T00:47:34.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.205+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8b8400fd90 con 0x7f8b84012180 2024-02-13T00:47:34.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.205+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:34.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.205+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.205+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b8400f3a0 osd=4 4 2024-02-13T00:47:34.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.205+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 4 10 2024-02-13T00:47:34.210 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:47:34.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.205+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:34.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.205+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 11 2024-02-13T00:47:34.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.205+0000 7f8b89ffb700 20 client.37553.objecter _get_session osd=-1 returning homeless 2024-02-13T00:47:34.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.205+0000 7f8b89ffb700 10 client.37553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.205+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=459}) v3 -- 0x7f8b840172c0 con 0x7f8ba40fe590 2024-02-13T00:47:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.205+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8b84017550 con 0x7f8ba40fe590 2024-02-13T00:47:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_get_version_reply(handle=1 version=458) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b9c053ea0 con 0x7f8ba40fe590 2024-02-13T00:47:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8ba17fa700 10 client.37553.objecter _check_command_map_dne tid 11 current 458 map_dne_bound 458 2024-02-13T00:47:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8ba17fa700 10 client.37553.objecter _finish_command 11 = osd:6 osd down 2024-02-13T00:47:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8ba17fa700 15 client.37553.objecter _session_command_op_remove -1 11 2024-02-13T00:47:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 12 2024-02-13T00:47:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b89ffb700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.181:6808/1506640341 conn(0x7f8b8400e580 0x7f8b84015be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b84001650 osd=6 3 2024-02-13T00:47:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b84001650 osd=6 3 2024-02-13T00:47:34.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b84001650 osd=6 3 2024-02-13T00:47:34.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 12 2024-02-13T00:47:34.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b84001650 osd=6 3 2024-02-13T00:47:34.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 6 12 2024-02-13T00:47:34.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b84001650 osd=6 4 2024-02-13T00:47:34.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b89ffb700 10 client.37553.objecter _send_command 12 2024-02-13T00:47:34.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.181:6808/1506640341 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b840018a0 con 0x7f8b8400e580 2024-02-13T00:47:34.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8ba27fc700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.181:6808/1506640341 conn(0x7f8b8400e580 0x7f8b84015be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:34.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8b8bfff700 10 client.37553.objecter ms_handle_connect 0x7f8b8400e580 2024-02-13T00:47:34.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.206+0000 7f8ba27fc700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.181:6808/1506640341 conn(0x7f8b8400e580 0x7f8b84015be0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:34.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.208+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8b840018a0 con 0x7f8b8400e580 2024-02-13T00:47:34.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.208+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 12: 0 ) v1 2024-02-13T00:47:34.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.208+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 12 = system:0 2024-02-13T00:47:34.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.208+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b84001650 osd=6 4 2024-02-13T00:47:34.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.208+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 6 12 2024-02-13T00:47:34.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.223+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:34.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.223+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 13 2024-02-13T00:47:34.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.223+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b84001650 osd=6 3 2024-02-13T00:47:34.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.223+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b84001650 osd=6 3 2024-02-13T00:47:34.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.223+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b84001650 osd=6 3 2024-02-13T00:47:34.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.223+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 13 2024-02-13T00:47:34.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.223+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b84001650 osd=6 3 2024-02-13T00:47:34.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.223+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 6 13 2024-02-13T00:47:34.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.223+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b84001650 osd=6 4 2024-02-13T00:47:34.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.223+0000 7f8b89ffb700 10 client.37553.objecter _send_command 13 2024-02-13T00:47:34.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.223+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8b840018a0 con 0x7f8b8400e580 2024-02-13T00:47:34.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8b840018a0 con 0x7f8b8400e580 2024-02-13T00:47:34.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:34.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b84001650 osd=6 4 2024-02-13T00:47:34.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 6 13 2024-02-13T00:47:34.230 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:47:34.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:34.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 14 2024-02-13T00:47:34.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b89ffb700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.181:6812/1964192772 conn(0x7f8b840166b0 0x7f8b84016af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:34.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b840018a0 osd=7 3 2024-02-13T00:47:34.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b840018a0 osd=7 3 2024-02-13T00:47:34.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b840018a0 osd=7 3 2024-02-13T00:47:34.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 14 2024-02-13T00:47:34.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b840018a0 osd=7 3 2024-02-13T00:47:34.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 7 14 2024-02-13T00:47:34.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b840018a0 osd=7 4 2024-02-13T00:47:34.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b89ffb700 10 client.37553.objecter _send_command 14 2024-02-13T00:47:34.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.181:6812/1964192772 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b8401de40 con 0x7f8b840166b0 2024-02-13T00:47:34.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.225+0000 7f8ba37fe700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.181:6812/1964192772 conn(0x7f8b840166b0 0x7f8b84016af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:34.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.226+0000 7f8b8bfff700 10 client.37553.objecter ms_handle_connect 0x7f8b840166b0 2024-02-13T00:47:34.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.226+0000 7f8ba37fe700 1 --2- 172.21.15.162:0/2996021839 >> v2:172.21.15.181:6812/1964192772 conn(0x7f8b840166b0 0x7f8b84016af0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:34.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.226+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8b8401de40 con 0x7f8b840166b0 2024-02-13T00:47:34.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.226+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 14: 0 ) v1 2024-02-13T00:47:34.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.226+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 14 = system:0 2024-02-13T00:47:34.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.226+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b840018a0 osd=7 4 2024-02-13T00:47:34.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.226+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 7 14 2024-02-13T00:47:34.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.243+0000 7f8b89ffb700 10 client.37553.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:34.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.243+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign -1 15 2024-02-13T00:47:34.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.243+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b840018a0 osd=7 3 2024-02-13T00:47:34.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.243+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b840018a0 osd=7 3 2024-02-13T00:47:34.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.243+0000 7f8b89ffb700 20 client.37553.objecter _get_session s=0x7f8b840018a0 osd=7 3 2024-02-13T00:47:34.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.243+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_remove -1 15 2024-02-13T00:47:34.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.243+0000 7f8b89ffb700 20 client.37553.objecter get_session s=0x7f8b840018a0 osd=7 3 2024-02-13T00:47:34.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.243+0000 7f8b89ffb700 15 client.37553.objecter _session_command_op_assign 7 15 2024-02-13T00:47:34.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.243+0000 7f8b89ffb700 20 client.37553.objecter put_session s=0x7f8b840018a0 osd=7 4 2024-02-13T00:47:34.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.243+0000 7f8b89ffb700 10 client.37553.objecter _send_command 15 2024-02-13T00:47:34.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.243+0000 7f8b89ffb700 1 -- 172.21.15.162:0/2996021839 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8b8401b6e0 con 0x7f8b840166b0 2024-02-13T00:47:34.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:34.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.245+0000 7f8b8bfff700 1 -- 172.21.15.162:0/2996021839 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8b8401b6e0 con 0x7f8b840166b0 2024-02-13T00:47:34.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.245+0000 7f8b8bfff700 10 client.37553.objecter ms_dispatch 0x7f8ba4001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:34.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.245+0000 7f8b8bfff700 10 client.37553.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.245+0000 7f8b8bfff700 20 client.37553.objecter put_session s=0x7f8b840018a0 osd=7 4 2024-02-13T00:47:34.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:34.245+0000 7f8b8bfff700 15 client.37553.objecter _session_command_op_remove 7 15 2024-02-13T00:47:34.250 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:47:34.250 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.250 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.251 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.251 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.251 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.251 INFO:teuthology.orchestra.run.smithi162.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-02-13T00:47:34.251 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.251 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:47:34.283 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:34.304 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:34.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:34.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:34.434 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:47:34.335+0000 7f16fd9f6700 -1 mgr get_metadata_python Requested missing service osd.5 2024-02-13T00:47:34.437 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:34.714 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:33.929483+0000 osd.0 (osd.0) 165 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:47:34.714 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:33.932759+0000 osd.0 (osd.0) 166 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:47:34.715 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:34.119068+0000 mon.c (mon.2) 390 : audit [DBG] from='client.? 172.21.15.162:0/2996021839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:34.715 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:34.329656+0000 osd.6 (osd.6) 123 : cluster [DBG] 6.a scrub starts 2024-02-13T00:47:34.715 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:34.332700+0000 osd.6 (osd.6) 124 : cluster [DBG] 6.a scrub ok 2024-02-13T00:47:34.715 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:34.573467+0000 mon.b (mon.1) 595 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:34.715 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:34.573994+0000 mon.a (mon.0) 2619 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:34.715 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:34.574441+0000 mon.b (mon.1) 596 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-69"}]: dispatch 2024-02-13T00:47:34.715 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:34.574731+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-69"}]: dispatch 2024-02-13T00:47:34.715 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:33.929483+0000 osd.0 (osd.0) 165 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:47:34.715 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:33.932759+0000 osd.0 (osd.0) 166 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:47:34.715 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:34.119068+0000 mon.c (mon.2) 390 : audit [DBG] from='client.? 172.21.15.162:0/2996021839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:34.715 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:34.329656+0000 osd.6 (osd.6) 123 : cluster [DBG] 6.a scrub starts 2024-02-13T00:47:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:34.332700+0000 osd.6 (osd.6) 124 : cluster [DBG] 6.a scrub ok 2024-02-13T00:47:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:34.573467+0000 mon.b (mon.1) 595 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:34.573994+0000 mon.a (mon.0) 2619 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:34.574441+0000 mon.b (mon.1) 596 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-69"}]: dispatch 2024-02-13T00:47:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:34.574731+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-69"}]: dispatch 2024-02-13T00:47:34.733 INFO:tasks.thrashosds.thrasher:j is {'epoch': 457, 'fsid': 'd4ee01fc-ca06-11ee-95b9-87774f69a715', 'created': '2024-02-13T00:31:51.797017+0000', 'modified': '2024-02-13T00:47:25.430722+0000', 'last_up_change': '2024-02-13T00:46:56.548676+0000', 'last_in_change': '2024-02-13T00:46:00.399226+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 103, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 216, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-13T00:35:34.545390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-13T00:37:39.459980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-13T00:37:39.740219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-13T00:37:41.489292+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-13T00:37:43.466304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-13T00:37:45.542442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '246', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 28, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-13T00:41:00.190777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 61, 'pool_name': 'test-rados-api-smithi162-84057-6', 'create_time': '2024-02-13T00:41:14.963279+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '450', 'last_force_op_resend': '390', 'last_force_op_resend_prenautilus': '390', 'last_force_op_resend_preluminous': '390', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 32, 'snap_epoch': 450, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}}, {'pool': 216, 'pool_name': 'test-rados-api-smithi162-84057-69', 'create_time': '2024-02-13T00:47:10.762502+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '449', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 447, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6801', 'nonce': 2103743338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 2103743338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 2103743338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6803', 'nonce': 2103743338}]}, 'public_addr': '172.21.15.162:6801/2103743338', 'cluster_addr': '172.21.15.162:6802/2103743338', 'heartbeat_back_addr': '172.21.15.162:6804/2103743338', 'heartbeat_front_addr': '172.21.15.162:6803/2103743338', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '39e9d2ef-9aa4-45bb-8a47-b0e0ec222886', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 447, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6805', 'nonce': 150599780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 150599780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 150599780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6807', 'nonce': 150599780}]}, 'public_addr': '172.21.15.162:6805/150599780', 'cluster_addr': '172.21.15.162:6806/150599780', 'heartbeat_back_addr': '172.21.15.162:6808/150599780', 'heartbeat_front_addr': '172.21.15.162:6807/150599780', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 333, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6809', 'nonce': 3689719376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3689719376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3689719376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6811', 'nonce': 3689719376}]}, 'public_addr': '172.21.15.162:6809/3689719376', 'cluster_addr': '172.21.15.162:6810/3689719376', 'heartbeat_back_addr': '172.21.15.162:6812/3689719376', 'heartbeat_front_addr': '172.21.15.162:6811/3689719376', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e6b9f0e-7a18-4cf8-9331-1a973fadc001', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 255, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6813', 'nonce': 604971011}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 604971011}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 604971011}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6815', 'nonce': 604971011}]}, 'public_addr': '172.21.15.162:6813/604971011', 'cluster_addr': '172.21.15.162:6814/604971011', 'heartbeat_back_addr': '172.21.15.162:6816/604971011', 'heartbeat_front_addr': '172.21.15.162:6815/604971011', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 3344263388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 3344263388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 3344263388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 3344263388}]}, 'public_addr': '172.21.15.181:6800/3344263388', 'cluster_addr': '172.21.15.181:6801/3344263388', 'heartbeat_back_addr': '172.21.15.181:6803/3344263388', 'heartbeat_front_addr': '172.21.15.181:6802/3344263388', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6aa2bd68-7f4b-4973-ac49-7fbc57c1a723', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 399, 'down_at': 437, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2212296748}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2212296748}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2212296748}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2212296748}]}, 'public_addr': '172.21.15.181:6804/2212296748', 'cluster_addr': '172.21.15.181:6805/2212296748', 'heartbeat_back_addr': '172.21.15.181:6807/2212296748', 'heartbeat_front_addr': '172.21.15.181:6806/2212296748', 'state': ['exists']}, {'osd': 6, 'uuid': '791ba814-5284-432e-addc-d6b7f5f3d474', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 447, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1506640341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1506640341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1506640341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1506640341}]}, 'public_addr': '172.21.15.181:6808/1506640341', 'cluster_addr': '172.21.15.181:6809/1506640341', 'heartbeat_back_addr': '172.21.15.181:6811/1506640341', 'heartbeat_front_addr': '172.21.15.181:6810/1506640341', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f369c65-e663-4eb2-87a5-f2900f15b35b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 215, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 1964192772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 1964192772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 1964192772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 1964192772}]}, 'public_addr': '172.21.15.181:6812/1964192772', 'cluster_addr': '172.21.15.181:6813/1964192772', 'heartbeat_back_addr': '172.21.15.181:6815/1964192772', 'heartbeat_front_addr': '172.21.15.181:6814/1964192772', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:47:05.584144+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:47:15.727103+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:46:53.108061+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:46:25.732999+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:47:21.377903+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:45:51.711776+0000', 'dead_epoch': 436}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:47:19.221397+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:46:25.961583+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '5.5', 'mappings': [{'from': 0, 'to': 6}]}, {'pgid': '28.12', 'mappings': [{'from': 0, 'to': 4}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.162:6800/1909610488': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2370739667': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/1502590187': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/4121673848': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:0/2412382714': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/3041562701': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/2775563314': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/3135632190': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/197307636': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2653247729': '2024-02-14T00:32:24.380625+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 24, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 60, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 78, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 94, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 113, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 166, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-13T00:47:34.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:47:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:33.929483+0000 osd.0 (osd.0) 165 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:47:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:33.932759+0000 osd.0 (osd.0) 166 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:47:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:34.119068+0000 mon.c (mon.2) 390 : audit [DBG] from='client.? 172.21.15.162:0/2996021839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:34.329656+0000 osd.6 (osd.6) 123 : cluster [DBG] 6.a scrub starts 2024-02-13T00:47:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:34.332700+0000 osd.6 (osd.6) 124 : cluster [DBG] 6.a scrub ok 2024-02-13T00:47:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:34.573467+0000 mon.b (mon.1) 595 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:35.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:34.573994+0000 mon.a (mon.0) 2619 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:35.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:34.574441+0000 mon.b (mon.1) 596 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-69"}]: dispatch 2024-02-13T00:47:35.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:34.574731+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-69"}]: dispatch 2024-02-13T00:47:35.385 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:35.386 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:35.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:35.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:35.517 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:35.669 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:35.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:35.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:35.800 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:35.873 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:35.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:35.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:35.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:35.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:34.441384+0000 mgr.y (mgr.14544) 408 : cluster [DBG] pgmap v678: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 568 B/s rd, 0 op/s 2024-02-13T00:47:35.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:35.439245+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e459: 8 total, 7 up, 4 in 2024-02-13T00:47:35.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:34.441384+0000 mgr.y (mgr.14544) 408 : cluster [DBG] pgmap v678: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 568 B/s rd, 0 op/s 2024-02-13T00:47:35.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:35.439245+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e459: 8 total, 7 up, 4 in 2024-02-13T00:47:36.008 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:36.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:47:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:47:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:34.441384+0000 mgr.y (mgr.14544) 408 : cluster [DBG] pgmap v678: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 568 B/s rd, 0 op/s 2024-02-13T00:47:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:35.439245+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e459: 8 total, 7 up, 4 in 2024-02-13T00:47:36.330 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:47:36] "GET /metrics HTTP/1.1" 200 177634 "" "Prometheus/2.33.4" 2024-02-13T00:47:36.387 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-13T00:47:36.387 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-13T00:47:36.707 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:36.997 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:35.365952+0000 osd.6 (osd.6) 125 : cluster [DBG] 4.15 scrub starts 2024-02-13T00:47:36.997 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:35.369610+0000 osd.6 (osd.6) 126 : cluster [DBG] 4.15 scrub ok 2024-02-13T00:47:36.997 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:35.671674+0000 mgr.y (mgr.14544) 409 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:36.998 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:36.365320+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:47:36.998 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:36.368858+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:47:36.998 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:36.449609+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e460: 8 total, 7 up, 4 in 2024-02-13T00:47:36.998 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:36.457139+0000 mon.b (mon.1) 597 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:36.998 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:36.457824+0000 mon.a (mon.0) 2623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:36.998 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:35.365952+0000 osd.6 (osd.6) 125 : cluster [DBG] 4.15 scrub starts 2024-02-13T00:47:36.998 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:35.369610+0000 osd.6 (osd.6) 126 : cluster [DBG] 4.15 scrub ok 2024-02-13T00:47:36.998 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:35.671674+0000 mgr.y (mgr.14544) 409 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:36.998 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:36.365320+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:47:36.998 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:36.368858+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:47:36.998 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:36.449609+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e460: 8 total, 7 up, 4 in 2024-02-13T00:47:36.999 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:36.457139+0000 mon.b (mon.1) 597 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:36.999 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:36.457824+0000 mon.a (mon.0) 2623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:35.365952+0000 osd.6 (osd.6) 125 : cluster [DBG] 4.15 scrub starts 2024-02-13T00:47:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:35.369610+0000 osd.6 (osd.6) 126 : cluster [DBG] 4.15 scrub ok 2024-02-13T00:47:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:35.671674+0000 mgr.y (mgr.14544) 409 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:37.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:36.365320+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:47:37.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:36.368858+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:47:37.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:36.449609+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e460: 8 total, 7 up, 4 in 2024-02-13T00:47:37.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:36.457139+0000 mon.b (mon.1) 597 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:37.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:36.457824+0000 mon.a (mon.0) 2623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:37.061 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:37.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:37.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:37.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:37.194 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:37.267 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:37.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:37.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:37.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:47:37.399 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:37.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:36.441718+0000 mgr.y (mgr.14544) 410 : cluster [DBG] pgmap v680: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:37.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:36.884407+0000 osd.0 (osd.0) 167 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:47:37.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:36.885503+0000 osd.0 (osd.0) 168 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:47:37.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:37.452084+0000 mon.a (mon.0) 2624 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:37.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:37.452166+0000 mon.a (mon.0) 2625 : cluster [DBG] osdmap e461: 8 total, 7 up, 4 in 2024-02-13T00:47:37.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:37.454543+0000 mon.b (mon.1) 598 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:37.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:36.441718+0000 mgr.y (mgr.14544) 410 : cluster [DBG] pgmap v680: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:37.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:36.884407+0000 osd.0 (osd.0) 167 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:47:37.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:36.885503+0000 osd.0 (osd.0) 168 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:47:37.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:37.452084+0000 mon.a (mon.0) 2624 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:37.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:37.452166+0000 mon.a (mon.0) 2625 : cluster [DBG] osdmap e461: 8 total, 7 up, 4 in 2024-02-13T00:47:37.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:37.454543+0000 mon.b (mon.1) 598 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:36.441718+0000 mgr.y (mgr.14544) 410 : cluster [DBG] pgmap v680: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:36.884407+0000 osd.0 (osd.0) 167 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:47:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:36.885503+0000 osd.0 (osd.0) 168 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:47:38.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:37.452084+0000 mon.a (mon.0) 2624 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:38.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:37.452166+0000 mon.a (mon.0) 2625 : cluster [DBG] osdmap e461: 8 total, 7 up, 4 in 2024-02-13T00:47:38.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:37.454543+0000 mon.b (mon.1) 598 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:38.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:47:38.885 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:37.009367+0000 osd.1 (osd.1) 197 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:47:38.885 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:37.012928+0000 osd.1 (osd.1) 198 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:47:38.885 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:38.452277+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e462: 8 total, 7 up, 4 in 2024-02-13T00:47:38.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:37.009367+0000 osd.1 (osd.1) 197 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:47:38.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:37.012928+0000 osd.1 (osd.1) 198 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:47:38.886 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:38.452277+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e462: 8 total, 7 up, 4 in 2024-02-13T00:47:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:37.009367+0000 osd.1 (osd.1) 197 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:47:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:37.012928+0000 osd.1 (osd.1) 198 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:47:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:38.452277+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e462: 8 total, 7 up, 4 in 2024-02-13T00:47:39.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:47:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:47:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:47:39.049 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:39.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:39.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:39.186 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:39.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.251+0000 7f026035f700 1 -- 172.21.15.162:0/2613306113 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02580f8500 msgr2=0x7f0258104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:39.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.251+0000 7f026035f700 1 --2- 172.21.15.162:0/2613306113 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02580f8500 0x7f0258104bd0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 rx=0x7f0250009be0 tx=0x7f0250004d10).stop 2024-02-13T00:47:39.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.251+0000 7f026035f700 1 -- 172.21.15.162:0/2613306113 shutdown_connections 2024-02-13T00:47:39.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.251+0000 7f026035f700 1 --2- 172.21.15.162:0/2613306113 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02580f8500 0x7f0258104bd0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:39.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.251+0000 7f026035f700 1 --2- 172.21.15.162:0/2613306113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02580f9b70 0x7f02580f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:39.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.251+0000 7f026035f700 1 --2- 172.21.15.162:0/2613306113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02580fa440 0x7f02580f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:47:39.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.251+0000 7f026035f700 1 -- 172.21.15.162:0/2613306113 >> 172.21.15.162:0/2613306113 conn(0x7f02580f38c0 msgr2=0x7f02580f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:39.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.251+0000 7f026035f700 1 -- 172.21.15.162:0/2613306113 shutdown_connections 2024-02-13T00:47:39.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.251+0000 7f026035f700 1 -- 172.21.15.162:0/2613306113 wait complete. 2024-02-13T00:47:39.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f026035f700 1 librados: starting msgr at 2024-02-13T00:47:39.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f026035f700 1 librados: starting objecter 2024-02-13T00:47:39.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f026035f700 1 Processor -- start 2024-02-13T00:47:39.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f026035f700 1 -- start start 2024-02-13T00:47:39.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f026035f700 1 librados: setting wanted keys 2024-02-13T00:47:39.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f026035f700 1 librados: calling monclient init 2024-02-13T00:47:39.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f026035f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02580f8500 0x7f025810e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:39.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f026035f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02580f9b70 0x7f025810e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:39.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f026035f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02580fa440 0x7f02581082d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:39.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f026035f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02581087c0 con 0x7f02580fa440 2024-02-13T00:47:39.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f026035f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0258108930 con 0x7f02580f8500 2024-02-13T00:47:39.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f026035f700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0258108aa0 con 0x7f02580f9b70 2024-02-13T00:47:39.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f025e8fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02580fa440 0x7f02581082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:39.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f025d8fa700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02580f9b70 0x7f025810e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:39.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f025e8fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02580fa440 0x7f02581082d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46724/0 (socket says 172.21.15.162:46724) 2024-02-13T00:47:39.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f025e8fc700 1 -- 172.21.15.162:0/2664595452 learned_addr learned my addr 172.21.15.162:0/2664595452 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:39.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f025e0fb700 1 --2- 172.21.15.162:0/2664595452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02580f8500 0x7f025810e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:39.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.252+0000 7f025d8fa700 1 -- 172.21.15.162:0/2664595452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02580f8500 msgr2=0x7f025810e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:39.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f025d8fa700 1 --2- 172.21.15.162:0/2664595452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02580f8500 0x7f025810e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:39.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f025d8fa700 1 -- 172.21.15.162:0/2664595452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02580fa440 msgr2=0x7f02581082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:39.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f025d8fa700 1 --2- 172.21.15.162:0/2664595452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02580fa440 0x7f02581082d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:39.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f025d8fa700 1 -- 172.21.15.162:0/2664595452 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0248005040 con 0x7f02580f9b70 2024-02-13T00:47:39.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f025e8fc700 1 --2- 172.21.15.162:0/2664595452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02580fa440 0x7f02581082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:47:39.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f025d8fa700 1 --2- 172.21.15.162:0/2664595452 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02580f9b70 0x7f025810e5e0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7f025400b4e0 tx=0x7f0254009060).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:39.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f024f7fe700 10 client.?.objecter ms_handle_connect 0x7f02580f9b70 2024-02-13T00:47:39.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f024f7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:39.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f024f7fe700 1 -- 172.21.15.162:0/2664595452 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025400be30 con 0x7f02580f9b70 2024-02-13T00:47:39.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f024f7fe700 1 -- 172.21.15.162:0/2664595452 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02540027f0 con 0x7f02580f9b70 2024-02-13T00:47:39.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f026035f700 1 -- 172.21.15.162:0/2664595452 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02500097c0 con 0x7f02580f9b70 2024-02-13T00:47:39.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f024f7fe700 1 -- 172.21.15.162:0/2664595452 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254012950 con 0x7f02580f9b70 2024-02-13T00:47:39.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f026035f700 10 client.37559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:39.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f026035f700 1 -- 172.21.15.162:0/2664595452 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02581090e0 con 0x7f02580f9b70 2024-02-13T00:47:39.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.253+0000 7f026035f700 1 librados: init done 2024-02-13T00:47:39.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.254+0000 7f026035f700 1 -- 172.21.15.162:0/2664595452 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02580475e0 con 0x7f02580f9b70 2024-02-13T00:47:39.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.255+0000 7f024f7fe700 1 -- 172.21.15.162:0/2664595452 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0254002320 con 0x7f02580f9b70 2024-02-13T00:47:39.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.255+0000 7f024f7fe700 1 --2- 172.21.15.162:0/2664595452 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0244062c90 0x7f0244065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:39.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.255+0000 7f024f7fe700 1 -- 172.21.15.162:0/2664595452 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(462..462 src has 52..462) v4 ==== 7967+0+0 (secure 0 0 0) 0x7f025407d000 con 0x7f02580f9b70 2024-02-13T00:47:39.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.255+0000 7f024f7fe700 10 client.37559.objecter ms_dispatch 0x7f0258001550 osd_map(462..462 src has 52..462) v4 2024-02-13T00:47:39.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.255+0000 7f024f7fe700 3 client.37559.objecter handle_osd_map got epochs [462,462] > 0 2024-02-13T00:47:39.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.255+0000 7f024f7fe700 3 client.37559.objecter handle_osd_map decoding full epoch 462 2024-02-13T00:47:39.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.255+0000 7f025e0fb700 1 --2- 172.21.15.162:0/2664595452 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0244062c90 0x7f0244065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:39.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.255+0000 7f024f7fe700 20 client.37559.objecter dump_active .. 0 homeless 2024-02-13T00:47:39.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.255+0000 7f024f7fe700 10 client.37559.objecter ms_handle_connect 0x7f0244062c90 2024-02-13T00:47:39.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.255+0000 7f025e0fb700 1 --2- 172.21.15.162:0/2664595452 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0244062c90 0x7f0244065150 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 rx=0x7f0248009330 tx=0x7f024800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:39.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.263+0000 7f024f7fe700 1 -- 172.21.15.162:0/2664595452 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0254050570 con 0x7f02580f9b70 2024-02-13T00:47:39.290 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:39.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:39.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:39.395 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:39.412 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:47:39.412 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:47:39.412 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:47:39.412 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:47:39.412 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:47:39.428 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:39.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.431+0000 7f026035f700 1 -- 172.21.15.162:0/2664595452 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f02581097b0 con 0x7f0244062c90 2024-02-13T00:47:39.450 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.447+0000 7f024f7fe700 1 -- 172.21.15.162:0/2664595452 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+580530 (secure 0 0 0) 0x7f02581097b0 con 0x7f0244062c90 2024-02-13T00:47:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:39.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:39.462 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:47:39.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 10 librados: watch_flush enter 2024-02-13T00:47:39.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 10 librados: watch_flush exit 2024-02-13T00:47:39.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 20 client.37559.objecter shutdown clearing up homeless session... 2024-02-13T00:47:39.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 10 client.37559.objecter successfully canceled tick 2024-02-13T00:47:39.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 1 -- 172.21.15.162:0/2664595452 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0244062c90 msgr2=0x7f0244065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 1 --2- 172.21.15.162:0/2664595452 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0244062c90 0x7f0244065150 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 rx=0x7f0248009330 tx=0x7f024800b040).stop 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 1 -- 172.21.15.162:0/2664595452 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02580f9b70 msgr2=0x7f025810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 1 --2- 172.21.15.162:0/2664595452 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02580f9b70 0x7f025810e5e0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7f025400b4e0 tx=0x7f0254009060).stop 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 1 -- 172.21.15.162:0/2664595452 shutdown_connections 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 1 --2- 172.21.15.162:0/2664595452 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0244062c90 0x7f0244065150 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 1 --2- 172.21.15.162:0/2664595452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02580fa440 0x7f02581082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 1 --2- 172.21.15.162:0/2664595452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02580f8500 0x7f025810e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 1 --2- 172.21.15.162:0/2664595452 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02580f9b70 0x7f025810e5e0 secure :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 rx=0x7f025400b4e0 tx=0x7f0254009060).stop 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.452+0000 7f026035f700 1 -- 172.21.15.162:0/2664595452 >> 172.21.15.162:0/2664595452 conn(0x7f02580f38c0 msgr2=0x7f02580f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.455+0000 7f026035f700 1 -- 172.21.15.162:0/2664595452 shutdown_connections 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.455+0000 7f026035f700 1 -- 172.21.15.162:0/2664595452 wait complete. 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:39.455+0000 7f026035f700 1 librados: shutdown 2024-02-13T00:47:39.473 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:47:39.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:39.590 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:39.610 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:39.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:39.715 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:39.744 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:39.763 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:39.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:39.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:38.442082+0000 mgr.y (mgr.14544) 411 : cluster [DBG] pgmap v683: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:39.454532+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e463: 8 total, 7 up, 4 in 2024-02-13T00:47:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:38.442082+0000 mgr.y (mgr.14544) 411 : cluster [DBG] pgmap v683: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:39.454532+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e463: 8 total, 7 up, 4 in 2024-02-13T00:47:39.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:39.895 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:39.947 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:39.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:38.442082+0000 mgr.y (mgr.14544) 411 : cluster [DBG] pgmap v683: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:39.454532+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e463: 8 total, 7 up, 4 in 2024-02-13T00:47:40.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:40.088 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:40.108 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:40.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:40.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:40.242 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:40.365 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:40.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:40.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-13T00:47:40.419 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:40.419 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:40.469 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:40.499 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:40.524 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:40.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:40.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:40.668 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:40.689 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:40.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:40.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:40.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:40.822 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:40.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:39.433395+0000 mgr.y (mgr.14544) 412 : audit [DBG] from='client.37559 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:47:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:39.821572+0000 osd.0 (osd.0) 169 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:47:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:39.822611+0000 osd.0 (osd.0) 170 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:47:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:40.319501+0000 osd.6 (osd.6) 129 : cluster [DBG] 5.18 scrub starts 2024-02-13T00:47:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:40.322590+0000 osd.6 (osd.6) 130 : cluster [DBG] 5.18 scrub ok 2024-02-13T00:47:40.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:39.433395+0000 mgr.y (mgr.14544) 412 : audit [DBG] from='client.37559 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:47:40.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:39.821572+0000 osd.0 (osd.0) 169 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:47:40.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:39.822611+0000 osd.0 (osd.0) 170 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:47:40.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:40.319501+0000 osd.6 (osd.6) 129 : cluster [DBG] 5.18 scrub starts 2024-02-13T00:47:40.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:40.322590+0000 osd.6 (osd.6) 130 : cluster [DBG] 5.18 scrub ok 2024-02-13T00:47:40.842 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:40.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:40.884 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":684,"stamp":"2024-02-13T00:47:38.451003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007339,"num_objects":390,"num_object_clones":5,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":0,"num_read":1758,"num_read_kb":1382,"num_write":1241,"num_write_kb":4561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1852,"num_bytes_recovered":4017904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":12,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2727,"ondisk_log_size":2727,"up":618,"acting":618,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":714,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1267592,"kb_used_data":41820,"kb_used_omap":0,"kb_used_meta":1225728,"kb_avail":373713016,"statfs":{"total":383980142592,"available":382682128384,"internally_reserved":0,"allocated":42823680,"data_stored":28884034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1255145472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6,"num_objects":-3,"num_object_clones":0,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":-8,"num_read_kb":0,"num_write":-21,"num_write_kb":-8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009852"},"pg_stats":[{"pgid":"217.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:47:36.442725+0000","last_change":"2024-02-13T00:47:36.442725+0000","last_active":"2024-02-13T00:47:36.442725+0000","last_peered":"2024-02-13T00:47:36.442725+0000","last_clean":"2024-02-13T00:47:36.442725+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:47:36.442725+0000","last_undegraded":"2024-02-13T00:47:36.442725+0000","last_fullsized":"2024-02-13T00:47:36.442725+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:36.442725+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.442725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.2","version":"204'34","reported_seq":251,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436447+0000","last_change":"2024-02-13T00:46:53.301789+0000","last_active":"2024-02-13T00:47:25.436447+0000","last_peered":"2024-02-13T00:47:25.436447+0000","last_clean":"2024-02-13T00:47:25.436447+0000","last_became_active":"2024-02-13T00:46:01.448503+0000","last_became_peered":"2024-02-13T00:46:01.448503+0000","last_unstale":"2024-02-13T00:47:25.436447+0000","last_undegraded":"2024-02-13T00:47:25.436447+0000","last_fullsized":"2024-02-13T00:47:25.436447+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-02-13T00:46:19.267508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:19.267508+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.3","version":"375'104","reported_seq":659,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.156048+0000","last_change":"2024-02-13T00:46:52.541875+0000","last_active":"2024-02-13T00:47:26.156048+0000","last_peered":"2024-02-13T00:47:26.156048+0000","last_clean":"2024-02-13T00:47:26.156048+0000","last_became_active":"2024-02-13T00:45:22.426662+0000","last_became_peered":"2024-02-13T00:45:22.426662+0000","last_unstale":"2024-02-13T00:47:26.156048+0000","last_undegraded":"2024-02-13T00:47:26.156048+0000","last_fullsized":"2024-02-13T00:47:26.156048+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'104","last_scrub_stamp":"2024-02-13T00:45:34.225334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.225334+0000","log_size":104,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.0","version":"388'38","reported_seq":479,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438661+0000","last_change":"2024-02-13T00:46:52.599628+0000","last_active":"2024-02-13T00:47:25.438661+0000","last_peered":"2024-02-13T00:47:25.438661+0000","last_clean":"2024-02-13T00:47:25.438661+0000","last_became_active":"2024-02-13T00:46:01.436132+0000","last_became_peered":"2024-02-13T00:46:01.436132+0000","last_unstale":"2024-02-13T00:47:25.438661+0000","last_undegraded":"2024-02-13T00:47:25.438661+0000","last_fullsized":"2024-02-13T00:47:25.438661+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'38","last_scrub_stamp":"2024-02-13T00:45:43.553743+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-02-13T00:45:43.553743+0000","last_clean_scrub_stamp":"2024-02-13T00:45:43.553743+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000030,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4000030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.1","version":"375'35","reported_seq":473,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155258+0000","last_change":"2024-02-13T00:46:52.539272+0000","last_active":"2024-02-13T00:47:26.155258+0000","last_peered":"2024-02-13T00:47:26.155258+0000","last_clean":"2024-02-13T00:47:26.155258+0000","last_became_active":"2024-02-13T00:45:15.420934+0000","last_became_peered":"2024-02-13T00:45:15.420934+0000","last_unstale":"2024-02-13T00:47:26.155258+0000","last_undegraded":"2024-02-13T00:47:26.155258+0000","last_fullsized":"2024-02-13T00:47:26.155258+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'34","last_scrub_stamp":"2024-02-13T00:45:29.241606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.241606+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.6","version":"400'20","reported_seq":401,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157817+0000","last_change":"2024-02-13T00:46:52.544318+0000","last_active":"2024-02-13T00:47:26.157817+0000","last_peered":"2024-02-13T00:47:26.157817+0000","last_clean":"2024-02-13T00:47:26.157817+0000","last_became_active":"2024-02-13T00:46:01.452805+0000","last_became_peered":"2024-02-13T00:46:01.452805+0000","last_unstale":"2024-02-13T00:47:26.157817+0000","last_undegraded":"2024-02-13T00:47:26.157817+0000","last_fullsized":"2024-02-13T00:47:26.157817+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'20","last_scrub_stamp":"2024-02-13T00:46:20.166298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:20.166298+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.7","version":"270'85","reported_seq":393,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440027+0000","last_change":"2024-02-13T00:46:52.602314+0000","last_active":"2024-02-13T00:47:25.440027+0000","last_peered":"2024-02-13T00:47:25.440027+0000","last_clean":"2024-02-13T00:47:25.440027+0000","last_became_active":"2024-02-13T00:46:01.458985+0000","last_became_peered":"2024-02-13T00:46:01.458985+0000","last_unstale":"2024-02-13T00:47:25.440027+0000","last_undegraded":"2024-02-13T00:47:25.440027+0000","last_fullsized":"2024-02-13T00:47:25.440027+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'85","last_scrub_stamp":"2024-02-13T00:45:02.230151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:02.230151+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.4","version":"445'64","reported_seq":205,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.439199+0000","last_change":"2024-02-13T00:46:52.602005+0000","last_active":"2024-02-13T00:47:25.439199+0000","last_peered":"2024-02-13T00:47:25.439199+0000","last_clean":"2024-02-13T00:47:25.439199+0000","last_became_active":"2024-02-13T00:46:01.455621+0000","last_became_peered":"2024-02-13T00:46:01.455621+0000","last_unstale":"2024-02-13T00:47:25.439199+0000","last_undegraded":"2024-02-13T00:47:25.439199+0000","last_fullsized":"2024-02-13T00:47:25.439199+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'57","last_scrub_stamp":"2024-02-13T00:45:32.238130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.238130+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":64,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":84,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.5","version":"224'56","reported_seq":597,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157531+0000","last_change":"2024-02-13T00:46:52.544183+0000","last_active":"2024-02-13T00:47:26.157531+0000","last_peered":"2024-02-13T00:47:26.157531+0000","last_clean":"2024-02-13T00:47:26.157531+0000","last_became_active":"2024-02-13T00:46:01.456637+0000","last_became_peered":"2024-02-13T00:46:01.456637+0000","last_unstale":"2024-02-13T00:47:26.157531+0000","last_undegraded":"2024-02-13T00:47:26.157531+0000","last_fullsized":"2024-02-13T00:47:26.157531+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'56","last_scrub_stamp":"2024-02-13T00:45:55.071118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:55.071118+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.a","version":"343'92","reported_seq":537,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155610+0000","last_change":"2024-02-13T00:46:52.539511+0000","last_active":"2024-02-13T00:47:26.155610+0000","last_peered":"2024-02-13T00:47:26.155610+0000","last_clean":"2024-02-13T00:47:26.155610+0000","last_became_active":"2024-02-13T00:46:01.453135+0000","last_became_peered":"2024-02-13T00:46:01.453135+0000","last_unstale":"2024-02-13T00:47:26.155610+0000","last_undegraded":"2024-02-13T00:47:26.155610+0000","last_fullsized":"2024-02-13T00:47:26.155610+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'92","last_scrub_stamp":"2024-02-13T00:45:50.066059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:50.066059+0000","log_size":92,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.b","version":"222'17","reported_seq":488,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.156008+0000","last_change":"2024-02-13T00:46:52.541543+0000","last_active":"2024-02-13T00:47:26.156008+0000","last_peered":"2024-02-13T00:47:26.156008+0000","last_clean":"2024-02-13T00:47:26.156008+0000","last_became_active":"2024-02-13T00:44:50.433096+0000","last_became_peered":"2024-02-13T00:44:50.433096+0000","last_unstale":"2024-02-13T00:47:26.156008+0000","last_undegraded":"2024-02-13T00:47:26.156008+0000","last_fullsized":"2024-02-13T00:47:26.156008+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'17","last_scrub_stamp":"2024-02-13T00:45:46.193088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.193088+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":74,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.8","version":"407'56","reported_seq":499,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437362+0000","last_change":"2024-02-13T00:46:52.600028+0000","last_active":"2024-02-13T00:47:25.437362+0000","last_peered":"2024-02-13T00:47:25.437362+0000","last_clean":"2024-02-13T00:47:25.437362+0000","last_became_active":"2024-02-13T00:46:01.445063+0000","last_became_peered":"2024-02-13T00:46:01.445063+0000","last_unstale":"2024-02-13T00:47:25.437362+0000","last_undegraded":"2024-02-13T00:47:25.437362+0000","last_fullsized":"2024-02-13T00:47:25.437362+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'53","last_scrub_stamp":"2024-02-13T00:44:30.046222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:30.046222+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.9","version":"343'36","reported_seq":478,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155194+0000","last_change":"2024-02-13T00:46:52.539250+0000","last_active":"2024-02-13T00:47:26.155194+0000","last_peered":"2024-02-13T00:47:26.155194+0000","last_clean":"2024-02-13T00:47:26.155194+0000","last_became_active":"2024-02-13T00:45:15.421026+0000","last_became_peered":"2024-02-13T00:45:15.421026+0000","last_unstale":"2024-02-13T00:47:26.155194+0000","last_undegraded":"2024-02-13T00:47:26.155194+0000","last_fullsized":"2024-02-13T00:47:26.155194+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'36","last_scrub_stamp":"2024-02-13T00:45:42.196133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.196133+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.e","version":"434'129","reported_seq":583,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438451+0000","last_change":"2024-02-13T00:46:55.305903+0000","last_active":"2024-02-13T00:47:25.438451+0000","last_peered":"2024-02-13T00:47:25.438451+0000","last_clean":"2024-02-13T00:47:25.438451+0000","last_became_active":"2024-02-13T00:46:17.513379+0000","last_became_peered":"2024-02-13T00:46:17.513379+0000","last_unstale":"2024-02-13T00:47:25.438451+0000","last_undegraded":"2024-02-13T00:47:25.438451+0000","last_fullsized":"2024-02-13T00:47:25.438451+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'113","last_scrub_stamp":"2024-02-13T00:45:45.254008+0000","last_deep_scrub":"389'113","last_deep_scrub_stamp":"2024-02-13T00:45:45.254008+0000","last_clean_scrub_stamp":"2024-02-13T00:45:45.254008+0000","log_size":129,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":84,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.f","version":"252'51","reported_seq":461,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157534+0000","last_change":"2024-02-13T00:46:52.543890+0000","last_active":"2024-02-13T00:47:26.157534+0000","last_peered":"2024-02-13T00:47:26.157534+0000","last_clean":"2024-02-13T00:47:26.157534+0000","last_became_active":"2024-02-13T00:46:01.440972+0000","last_became_peered":"2024-02-13T00:46:01.440972+0000","last_unstale":"2024-02-13T00:47:26.157534+0000","last_undegraded":"2024-02-13T00:47:26.157534+0000","last_fullsized":"2024-02-13T00:47:26.157534+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'51","last_scrub_stamp":"2024-02-13T00:45:26.215792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.215792+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.c","version":"400'108","reported_seq":750,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436639+0000","last_change":"2024-02-13T00:46:53.303237+0000","last_active":"2024-02-13T00:47:25.436639+0000","last_peered":"2024-02-13T00:47:25.436639+0000","last_clean":"2024-02-13T00:47:25.436639+0000","last_became_active":"2024-02-13T00:46:17.514719+0000","last_became_peered":"2024-02-13T00:46:17.514719+0000","last_unstale":"2024-02-13T00:47:25.436639+0000","last_undegraded":"2024-02-13T00:47:25.436639+0000","last_fullsized":"2024-02-13T00:47:25.436639+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'106","last_scrub_stamp":"2024-02-13T00:45:31.255154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.255154+0000","log_size":108,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":100,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.d","version":"343'91","reported_seq":572,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.156935+0000","last_change":"2024-02-13T00:46:52.543520+0000","last_active":"2024-02-13T00:47:26.156935+0000","last_peered":"2024-02-13T00:47:26.156935+0000","last_clean":"2024-02-13T00:47:26.156935+0000","last_became_active":"2024-02-13T00:46:01.442967+0000","last_became_peered":"2024-02-13T00:46:01.442967+0000","last_unstale":"2024-02-13T00:47:26.156935+0000","last_undegraded":"2024-02-13T00:47:26.156935+0000","last_fullsized":"2024-02-13T00:47:26.156935+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'90","last_scrub_stamp":"2024-02-13T00:44:31.080244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:31.080244+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":361,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437960+0000","last_change":"2024-02-13T00:46:01.448309+0000","last_active":"2024-02-13T00:47:25.437960+0000","last_peered":"2024-02-13T00:47:25.437960+0000","last_clean":"2024-02-13T00:47:25.437960+0000","last_became_active":"2024-02-13T00:46:01.445363+0000","last_became_peered":"2024-02-13T00:46:01.445363+0000","last_unstale":"2024-02-13T00:47:25.437960+0000","last_undegraded":"2024-02-13T00:47:25.437960+0000","last_fullsized":"2024-02-13T00:47:25.437960+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.12","version":"343'24","reported_seq":425,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438144+0000","last_change":"2024-02-13T00:46:52.600197+0000","last_active":"2024-02-13T00:47:25.438144+0000","last_peered":"2024-02-13T00:47:25.438144+0000","last_clean":"2024-02-13T00:47:25.438144+0000","last_became_active":"2024-02-13T00:46:01.433618+0000","last_became_peered":"2024-02-13T00:46:01.433618+0000","last_unstale":"2024-02-13T00:47:25.438144+0000","last_undegraded":"2024-02-13T00:47:25.438144+0000","last_fullsized":"2024-02-13T00:47:25.438144+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'22","last_scrub_stamp":"2024-02-13T00:44:34.629429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.629429+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":286,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435461+0000","last_change":"2024-02-13T00:46:01.449894+0000","last_active":"2024-02-13T00:47:25.435461+0000","last_peered":"2024-02-13T00:47:25.435461+0000","last_clean":"2024-02-13T00:47:25.435461+0000","last_became_active":"2024-02-13T00:46:01.449570+0000","last_became_peered":"2024-02-13T00:46:01.449570+0000","last_unstale":"2024-02-13T00:47:25.435461+0000","last_undegraded":"2024-02-13T00:47:25.435461+0000","last_fullsized":"2024-02-13T00:47:25.435461+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.13","version":"236'73","reported_seq":268,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437208+0000","last_change":"2024-02-13T00:46:53.304200+0000","last_active":"2024-02-13T00:47:25.437208+0000","last_peered":"2024-02-13T00:47:25.437208+0000","last_clean":"2024-02-13T00:47:25.437208+0000","last_became_active":"2024-02-13T00:45:15.447922+0000","last_became_peered":"2024-02-13T00:45:15.447922+0000","last_unstale":"2024-02-13T00:47:25.437208+0000","last_undegraded":"2024-02-13T00:47:25.437208+0000","last_fullsized":"2024-02-13T00:47:25.437208+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'73","last_scrub_stamp":"2024-02-13T00:46:22.291170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:22.291170+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.10","version":"397'79","reported_seq":642,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435045+0000","last_change":"2024-02-13T00:46:53.300466+0000","last_active":"2024-02-13T00:47:25.435045+0000","last_peered":"2024-02-13T00:47:25.435045+0000","last_clean":"2024-02-13T00:47:25.435045+0000","last_became_active":"2024-02-13T00:46:01.440561+0000","last_became_peered":"2024-02-13T00:46:01.440561+0000","last_unstale":"2024-02-13T00:47:25.435045+0000","last_undegraded":"2024-02-13T00:47:25.435045+0000","last_fullsized":"2024-02-13T00:47:25.435045+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'73","last_scrub_stamp":"2024-02-13T00:45:27.173322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:27.173322+0000","log_size":79,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.11","version":"343'37","reported_seq":357,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155159+0000","last_change":"2024-02-13T00:46:52.538747+0000","last_active":"2024-02-13T00:47:26.155159+0000","last_peered":"2024-02-13T00:47:26.155159+0000","last_clean":"2024-02-13T00:47:26.155159+0000","last_became_active":"2024-02-13T00:44:50.443606+0000","last_became_peered":"2024-02-13T00:44:50.443606+0000","last_unstale":"2024-02-13T00:47:26.155159+0000","last_undegraded":"2024-02-13T00:47:26.155159+0000","last_fullsized":"2024-02-13T00:47:26.155159+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'37","last_scrub_stamp":"2024-02-13T00:46:22.170067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:22.170067+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.16","version":"442'81","reported_seq":564,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.439669+0000","last_change":"2024-02-13T00:46:52.603656+0000","last_active":"2024-02-13T00:47:25.439669+0000","last_peered":"2024-02-13T00:47:25.439669+0000","last_clean":"2024-02-13T00:47:25.439669+0000","last_became_active":"2024-02-13T00:46:01.449774+0000","last_became_peered":"2024-02-13T00:46:01.449774+0000","last_unstale":"2024-02-13T00:47:25.439669+0000","last_undegraded":"2024-02-13T00:47:25.439669+0000","last_fullsized":"2024-02-13T00:47:25.439669+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'75","last_scrub_stamp":"2024-02-13T00:45:46.157235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.157235+0000","log_size":81,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.17","version":"343'75","reported_seq":590,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440078+0000","last_change":"2024-02-13T00:46:52.602227+0000","last_active":"2024-02-13T00:47:25.440078+0000","last_peered":"2024-02-13T00:47:25.440078+0000","last_clean":"2024-02-13T00:47:25.440078+0000","last_became_active":"2024-02-13T00:46:01.447341+0000","last_became_peered":"2024-02-13T00:46:01.447341+0000","last_unstale":"2024-02-13T00:47:25.440078+0000","last_undegraded":"2024-02-13T00:47:25.440078+0000","last_fullsized":"2024-02-13T00:47:25.440078+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'75","last_scrub_stamp":"2024-02-13T00:45:36.518979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:36.518979+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.14","version":"200'17","reported_seq":470,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440994+0000","last_change":"2024-02-13T00:46:52.602069+0000","last_active":"2024-02-13T00:47:25.440994+0000","last_peered":"2024-02-13T00:47:25.440994+0000","last_clean":"2024-02-13T00:47:25.440994+0000","last_became_active":"2024-02-13T00:46:01.455179+0000","last_became_peered":"2024-02-13T00:46:01.455179+0000","last_unstale":"2024-02-13T00:47:25.440994+0000","last_undegraded":"2024-02-13T00:47:25.440994+0000","last_fullsized":"2024-02-13T00:47:25.440994+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'17","last_scrub_stamp":"2024-02-13T00:45:37.548974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.548974+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.15","version":"423'99","reported_seq":329,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.156660+0000","last_change":"2024-02-13T00:46:52.543515+0000","last_active":"2024-02-13T00:47:26.156660+0000","last_peered":"2024-02-13T00:47:26.156660+0000","last_clean":"2024-02-13T00:47:26.156660+0000","last_became_active":"2024-02-13T00:46:01.452384+0000","last_became_peered":"2024-02-13T00:46:01.452384+0000","last_unstale":"2024-02-13T00:47:26.156660+0000","last_undegraded":"2024-02-13T00:47:26.156660+0000","last_fullsized":"2024-02-13T00:47:26.156660+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'78","last_scrub_stamp":"2024-02-13T00:45:36.188466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:36.188466+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":151,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155103+0000","last_change":"2024-02-13T00:46:01.445141+0000","last_active":"2024-02-13T00:47:26.155103+0000","last_peered":"2024-02-13T00:47:26.155103+0000","last_clean":"2024-02-13T00:47:26.155103+0000","last_became_active":"2024-02-13T00:46:01.444729+0000","last_became_peered":"2024-02-13T00:46:01.444729+0000","last_unstale":"2024-02-13T00:47:26.155103+0000","last_undegraded":"2024-02-13T00:47:26.155103+0000","last_fullsized":"2024-02-13T00:47:26.155103+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1a","version":"403'54","reported_seq":243,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436295+0000","last_change":"2024-02-13T00:46:53.301730+0000","last_active":"2024-02-13T00:47:25.436295+0000","last_peered":"2024-02-13T00:47:25.436295+0000","last_clean":"2024-02-13T00:47:25.436295+0000","last_became_active":"2024-02-13T00:45:15.436908+0000","last_became_peered":"2024-02-13T00:45:15.436908+0000","last_unstale":"2024-02-13T00:47:25.436295+0000","last_undegraded":"2024-02-13T00:47:25.436295+0000","last_fullsized":"2024-02-13T00:47:25.436295+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'54","last_scrub_stamp":"2024-02-13T00:46:26.235259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:26.235259+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":255,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":236,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155375+0000","last_change":"2024-02-13T00:45:15.426658+0000","last_active":"2024-02-13T00:47:26.155375+0000","last_peered":"2024-02-13T00:47:26.155375+0000","last_clean":"2024-02-13T00:47:26.155375+0000","last_became_active":"2024-02-13T00:45:15.426109+0000","last_became_peered":"2024-02-13T00:45:15.426109+0000","last_unstale":"2024-02-13T00:47:26.155375+0000","last_undegraded":"2024-02-13T00:47:26.155375+0000","last_fullsized":"2024-02-13T00:47:26.155375+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1b","version":"397'54","reported_seq":250,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436746+0000","last_change":"2024-02-13T00:46:53.302763+0000","last_active":"2024-02-13T00:47:25.436746+0000","last_peered":"2024-02-13T00:47:25.436746+0000","last_clean":"2024-02-13T00:47:25.436746+0000","last_became_active":"2024-02-13T00:46:01.458733+0000","last_became_peered":"2024-02-13T00:46:01.458733+0000","last_unstale":"2024-02-13T00:47:25.436746+0000","last_undegraded":"2024-02-13T00:47:25.436746+0000","last_fullsized":"2024-02-13T00:47:25.436746+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'54","last_scrub_stamp":"2024-02-13T00:45:57.130510+0000","last_deep_scrub":"397'54","last_deep_scrub_stamp":"2024-02-13T00:45:57.130510+0000","last_clean_scrub_stamp":"2024-02-13T00:45:57.130510+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":16,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":189,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":142,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436627+0000","last_change":"2024-02-13T00:44:39.526065+0000","last_active":"2024-02-13T00:47:25.436627+0000","last_peered":"2024-02-13T00:47:25.436627+0000","last_clean":"2024-02-13T00:47:25.436627+0000","last_became_active":"2024-02-13T00:44:39.525161+0000","last_became_peered":"2024-02-13T00:44:39.525161+0000","last_unstale":"2024-02-13T00:47:25.436627+0000","last_undegraded":"2024-02-13T00:47:25.436627+0000","last_fullsized":"2024-02-13T00:47:25.436627+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.18","version":"204'34","reported_seq":251,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155015+0000","last_change":"2024-02-13T00:46:52.538637+0000","last_active":"2024-02-13T00:47:26.155015+0000","last_peered":"2024-02-13T00:47:26.155015+0000","last_clean":"2024-02-13T00:47:26.155015+0000","last_became_active":"2024-02-13T00:46:01.438318+0000","last_became_peered":"2024-02-13T00:46:01.438318+0000","last_unstale":"2024-02-13T00:47:26.155015+0000","last_undegraded":"2024-02-13T00:47:26.155015+0000","last_fullsized":"2024-02-13T00:47:26.155015+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-02-13T00:45:29.218592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.218592+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436501+0000","last_change":"2024-02-13T00:46:01.448133+0000","last_active":"2024-02-13T00:47:25.436501+0000","last_peered":"2024-02-13T00:47:25.436501+0000","last_clean":"2024-02-13T00:47:25.436501+0000","last_became_active":"2024-02-13T00:46:01.447415+0000","last_became_peered":"2024-02-13T00:46:01.447415+0000","last_unstale":"2024-02-13T00:47:25.436501+0000","last_undegraded":"2024-02-13T00:47:25.436501+0000","last_fullsized":"2024-02-13T00:47:25.436501+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.19","version":"456'35","reported_seq":420,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438076+0000","last_change":"2024-02-13T00:46:52.599919+0000","last_active":"2024-02-13T00:47:25.438076+0000","last_peered":"2024-02-13T00:47:25.438076+0000","last_clean":"2024-02-13T00:47:25.438076+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:47:25.438076+0000","last_undegraded":"2024-02-13T00:47:25.438076+0000","last_fullsized":"2024-02-13T00:47:25.438076+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'18","last_scrub_stamp":"2024-02-13T00:45:51.098914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:51.098914+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438018+0000","last_change":"2024-02-13T00:46:01.459772+0000","last_active":"2024-02-13T00:47:25.438018+0000","last_peered":"2024-02-13T00:47:25.438018+0000","last_clean":"2024-02-13T00:47:25.438018+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:47:25.438018+0000","last_undegraded":"2024-02-13T00:47:25.438018+0000","last_fullsized":"2024-02-13T00:47:25.438018+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1e","version":"403'57","reported_seq":296,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438152+0000","last_change":"2024-02-13T00:46:53.304309+0000","last_active":"2024-02-13T00:47:25.438152+0000","last_peered":"2024-02-13T00:47:25.438152+0000","last_clean":"2024-02-13T00:47:25.438152+0000","last_became_active":"2024-02-13T00:45:15.446531+0000","last_became_peered":"2024-02-13T00:45:15.446531+0000","last_unstale":"2024-02-13T00:47:25.438152+0000","last_undegraded":"2024-02-13T00:47:25.438152+0000","last_fullsized":"2024-02-13T00:47:25.438152+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'57","last_scrub_stamp":"2024-02-13T00:46:33.286813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:33.286813+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":419,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436913+0000","last_change":"2024-02-13T00:46:01.463062+0000","last_active":"2024-02-13T00:47:25.436913+0000","last_peered":"2024-02-13T00:47:25.436913+0000","last_clean":"2024-02-13T00:47:25.436913+0000","last_became_active":"2024-02-13T00:46:01.462319+0000","last_became_peered":"2024-02-13T00:46:01.462319+0000","last_unstale":"2024-02-13T00:47:25.436913+0000","last_undegraded":"2024-02-13T00:47:25.436913+0000","last_fullsized":"2024-02-13T00:47:25.436913+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1f","version":"392'68","reported_seq":483,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440120+0000","last_change":"2024-02-13T00:46:52.602905+0000","last_active":"2024-02-13T00:47:25.440120+0000","last_peered":"2024-02-13T00:47:25.440120+0000","last_clean":"2024-02-13T00:47:25.440120+0000","last_became_active":"2024-02-13T00:44:50.441446+0000","last_became_peered":"2024-02-13T00:44:50.441446+0000","last_unstale":"2024-02-13T00:47:25.440120+0000","last_undegraded":"2024-02-13T00:47:25.440120+0000","last_fullsized":"2024-02-13T00:47:25.440120+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'68","last_scrub_stamp":"2024-02-13T00:45:51.651226+0000","last_deep_scrub":"392'68","last_deep_scrub_stamp":"2024-02-13T00:45:51.651226+0000","last_clean_scrub_stamp":"2024-02-13T00:45:51.651226+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":142,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438433+0000","last_change":"2024-02-13T00:44:39.525601+0000","last_active":"2024-02-13T00:47:25.438433+0000","last_peered":"2024-02-13T00:47:25.438433+0000","last_clean":"2024-02-13T00:47:25.438433+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:47:25.438433+0000","last_undegraded":"2024-02-13T00:47:25.438433+0000","last_fullsized":"2024-02-13T00:47:25.438433+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1c","version":"397'56","reported_seq":582,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436904+0000","last_change":"2024-02-13T00:46:53.304111+0000","last_active":"2024-02-13T00:47:25.436904+0000","last_peered":"2024-02-13T00:47:25.436904+0000","last_clean":"2024-02-13T00:47:25.436904+0000","last_became_active":"2024-02-13T00:46:01.462341+0000","last_became_peered":"2024-02-13T00:46:01.462341+0000","last_unstale":"2024-02-13T00:47:25.436904+0000","last_undegraded":"2024-02-13T00:47:25.436904+0000","last_fullsized":"2024-02-13T00:47:25.436904+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'52","last_scrub_stamp":"2024-02-13T00:45:44.205380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:44.205380+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":227,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157751+0000","last_change":"2024-02-13T00:46:01.455629+0000","last_active":"2024-02-13T00:47:26.157751+0000","last_peered":"2024-02-13T00:47:26.157751+0000","last_clean":"2024-02-13T00:47:26.157751+0000","last_became_active":"2024-02-13T00:46:01.452801+0000","last_became_peered":"2024-02-13T00:46:01.452801+0000","last_unstale":"2024-02-13T00:47:26.157751+0000","last_undegraded":"2024-02-13T00:47:26.157751+0000","last_fullsized":"2024-02-13T00:47:26.157751+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1d","version":"343'69","reported_seq":573,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157522+0000","last_change":"2024-02-13T00:46:52.543895+0000","last_active":"2024-02-13T00:47:26.157522+0000","last_peered":"2024-02-13T00:47:26.157522+0000","last_clean":"2024-02-13T00:47:26.157522+0000","last_became_active":"2024-02-13T00:45:15.435242+0000","last_became_peered":"2024-02-13T00:45:15.435242+0000","last_unstale":"2024-02-13T00:47:26.157522+0000","last_undegraded":"2024-02-13T00:47:26.157522+0000","last_fullsized":"2024-02-13T00:47:26.157522+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'69","last_scrub_stamp":"2024-02-13T00:46:28.118117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:28.118117+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":382,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438371+0000","last_change":"2024-02-13T00:45:39.562095+0000","last_active":"2024-02-13T00:47:25.438371+0000","last_peered":"2024-02-13T00:47:25.438371+0000","last_clean":"2024-02-13T00:47:25.438371+0000","last_became_active":"2024-02-13T00:44:39.505121+0000","last_became_peered":"2024-02-13T00:44:39.505121+0000","last_unstale":"2024-02-13T00:47:25.438371+0000","last_undegraded":"2024-02-13T00:47:25.438371+0000","last_fullsized":"2024-02-13T00:47:25.438371+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:39.561994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:39.561994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"363'14","reported_seq":202,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-13T00:47:32.593124+0000","last_change":"2024-02-13T00:45:53.115355+0000","last_active":"2024-02-13T00:47:32.593124+0000","last_peered":"2024-02-13T00:47:32.593124+0000","last_clean":"2024-02-13T00:47:32.593124+0000","last_became_active":"2024-02-13T00:45:15.435476+0000","last_became_peered":"2024-02-13T00:45:15.435476+0000","last_unstale":"2024-02-13T00:47:32.593124+0000","last_undegraded":"2024-02-13T00:47:32.593124+0000","last_fullsized":"2024-02-13T00:47:32.593124+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:45:53.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.115260+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"80'36","reported_seq":468,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435241+0000","last_change":"2024-02-13T00:47:03.491720+0000","last_active":"2024-02-13T00:47:25.435241+0000","last_peered":"2024-02-13T00:47:25.435241+0000","last_clean":"2024-02-13T00:47:25.435241+0000","last_became_active":"2024-02-13T00:46:01.438056+0000","last_became_peered":"2024-02-13T00:46:01.438056+0000","last_unstale":"2024-02-13T00:47:25.435241+0000","last_undegraded":"2024-02-13T00:47:25.435241+0000","last_fullsized":"2024-02-13T00:47:25.435241+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:47:03.491647+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":451,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438035+0000","last_change":"2024-02-13T00:46:01.464695+0000","last_active":"2024-02-13T00:47:25.438035+0000","last_peered":"2024-02-13T00:47:25.438035+0000","last_clean":"2024-02-13T00:47:25.438035+0000","last_became_active":"2024-02-13T00:46:01.460811+0000","last_became_peered":"2024-02-13T00:46:01.460811+0000","last_unstale":"2024-02-13T00:47:25.438035+0000","last_undegraded":"2024-02-13T00:47:25.438035+0000","last_fullsized":"2024-02-13T00:47:25.438035+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.340080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.340080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":362,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-13T00:47:34.332791+0000","last_change":"2024-02-13T00:47:34.332791+0000","last_active":"2024-02-13T00:47:34.332791+0000","last_peered":"2024-02-13T00:47:34.332791+0000","last_clean":"2024-02-13T00:47:34.332791+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:47:34.332791+0000","last_undegraded":"2024-02-13T00:47:34.332791+0000","last_fullsized":"2024-02-13T00:47:34.332791+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:34.332704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:34.332704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":400,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436382+0000","last_change":"2024-02-13T00:46:01.448237+0000","last_active":"2024-02-13T00:47:25.436382+0000","last_peered":"2024-02-13T00:47:25.436382+0000","last_clean":"2024-02-13T00:47:25.436382+0000","last_became_active":"2024-02-13T00:46:01.446660+0000","last_became_peered":"2024-02-13T00:46:01.446660+0000","last_unstale":"2024-02-13T00:47:25.436382+0000","last_undegraded":"2024-02-13T00:47:25.436382+0000","last_fullsized":"2024-02-13T00:47:25.436382+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.260871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.260871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"65'15","reported_seq":447,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438425+0000","last_change":"2024-02-13T00:42:19.652772+0000","last_active":"2024-02-13T00:47:25.438425+0000","last_peered":"2024-02-13T00:47:25.438425+0000","last_clean":"2024-02-13T00:47:25.438425+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:47:25.438425+0000","last_undegraded":"2024-02-13T00:47:25.438425+0000","last_fullsized":"2024-02-13T00:47:25.438425+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:42:19.652689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:19.652689+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":437,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157930+0000","last_change":"2024-02-13T00:44:50.430435+0000","last_active":"2024-02-13T00:47:26.157930+0000","last_peered":"2024-02-13T00:47:26.157930+0000","last_clean":"2024-02-13T00:47:26.157930+0000","last_became_active":"2024-02-13T00:44:50.429994+0000","last_became_peered":"2024-02-13T00:44:50.429994+0000","last_unstale":"2024-02-13T00:47:26.157930+0000","last_undegraded":"2024-02-13T00:47:26.157930+0000","last_fullsized":"2024-02-13T00:47:26.157930+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:20.102471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:20.102471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":352,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437992+0000","last_change":"2024-02-13T00:44:50.434925+0000","last_active":"2024-02-13T00:47:25.437992+0000","last_peered":"2024-02-13T00:47:25.437992+0000","last_clean":"2024-02-13T00:47:25.437992+0000","last_became_active":"2024-02-13T00:44:50.433984+0000","last_became_peered":"2024-02-13T00:44:50.433984+0000","last_unstale":"2024-02-13T00:47:25.437992+0000","last_undegraded":"2024-02-13T00:47:25.437992+0000","last_fullsized":"2024-02-13T00:47:25.437992+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.512181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.512181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":439,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438612+0000","last_change":"2024-02-13T00:45:15.435610+0000","last_active":"2024-02-13T00:47:25.438612+0000","last_peered":"2024-02-13T00:47:25.438612+0000","last_clean":"2024-02-13T00:47:25.438612+0000","last_became_active":"2024-02-13T00:45:15.434853+0000","last_became_peered":"2024-02-13T00:45:15.434853+0000","last_unstale":"2024-02-13T00:47:25.438612+0000","last_undegraded":"2024-02-13T00:47:25.438612+0000","last_fullsized":"2024-02-13T00:47:25.438612+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":65,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437580+0000","last_change":"2024-02-13T00:46:01.455201+0000","last_active":"2024-02-13T00:47:25.437580+0000","last_peered":"2024-02-13T00:47:25.437580+0000","last_clean":"2024-02-13T00:47:25.437580+0000","last_became_active":"2024-02-13T00:46:01.454882+0000","last_became_peered":"2024-02-13T00:46:01.454882+0000","last_unstale":"2024-02-13T00:47:25.437580+0000","last_undegraded":"2024-02-13T00:47:25.437580+0000","last_fullsized":"2024-02-13T00:47:25.437580+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:35.197045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:35.197045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"91'35","reported_seq":226,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436144+0000","last_change":"2024-02-13T00:46:09.256189+0000","last_active":"2024-02-13T00:47:25.436144+0000","last_peered":"2024-02-13T00:47:25.436144+0000","last_clean":"2024-02-13T00:47:25.436144+0000","last_became_active":"2024-02-13T00:46:01.459978+0000","last_became_peered":"2024-02-13T00:46:01.459978+0000","last_unstale":"2024-02-13T00:47:25.436144+0000","last_undegraded":"2024-02-13T00:47:25.436144+0000","last_fullsized":"2024-02-13T00:47:25.436144+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:44:24.196214+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:44:24.196214+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":75,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440508+0000","last_change":"2024-02-13T00:46:30.641539+0000","last_active":"2024-02-13T00:47:25.440508+0000","last_peered":"2024-02-13T00:47:25.440508+0000","last_clean":"2024-02-13T00:47:25.440508+0000","last_became_active":"2024-02-13T00:46:01.444548+0000","last_became_peered":"2024-02-13T00:46:01.444548+0000","last_unstale":"2024-02-13T00:47:25.440508+0000","last_undegraded":"2024-02-13T00:47:25.440508+0000","last_fullsized":"2024-02-13T00:47:25.440508+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.641460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.641460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":157,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438180+0000","last_change":"2024-02-13T00:46:01.455578+0000","last_active":"2024-02-13T00:47:25.438180+0000","last_peered":"2024-02-13T00:47:25.438180+0000","last_clean":"2024-02-13T00:47:25.438180+0000","last_became_active":"2024-02-13T00:46:01.454642+0000","last_became_peered":"2024-02-13T00:46:01.454642+0000","last_unstale":"2024-02-13T00:47:25.438180+0000","last_undegraded":"2024-02-13T00:47:25.438180+0000","last_fullsized":"2024-02-13T00:47:25.438180+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.592672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":278,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435355+0000","last_change":"2024-02-13T00:45:15.429026+0000","last_active":"2024-02-13T00:47:25.435355+0000","last_peered":"2024-02-13T00:47:25.435355+0000","last_clean":"2024-02-13T00:47:25.435355+0000","last_became_active":"2024-02-13T00:45:15.427945+0000","last_became_peered":"2024-02-13T00:45:15.427945+0000","last_unstale":"2024-02-13T00:47:25.435355+0000","last_undegraded":"2024-02-13T00:47:25.435355+0000","last_fullsized":"2024-02-13T00:47:25.435355+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":382,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157815+0000","last_change":"2024-02-13T00:45:26.255907+0000","last_active":"2024-02-13T00:47:26.157815+0000","last_peered":"2024-02-13T00:47:26.157815+0000","last_clean":"2024-02-13T00:47:26.157815+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:47:26.157815+0000","last_undegraded":"2024-02-13T00:47:26.157815+0000","last_fullsized":"2024-02-13T00:47:26.157815+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.255842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.255842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"64'17","reported_seq":193,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157848+0000","last_change":"2024-02-13T00:44:54.041282+0000","last_active":"2024-02-13T00:47:26.157848+0000","last_peered":"2024-02-13T00:47:26.157848+0000","last_clean":"2024-02-13T00:47:26.157848+0000","last_became_active":"2024-02-13T00:44:50.432807+0000","last_became_peered":"2024-02-13T00:44:50.432807+0000","last_unstale":"2024-02-13T00:47:26.157848+0000","last_undegraded":"2024-02-13T00:47:26.157848+0000","last_fullsized":"2024-02-13T00:47:26.157848+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:42:17.413826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.413826+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":233,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436070+0000","last_change":"2024-02-13T00:46:04.708510+0000","last_active":"2024-02-13T00:47:25.436070+0000","last_peered":"2024-02-13T00:47:25.436070+0000","last_clean":"2024-02-13T00:47:25.436070+0000","last_became_active":"2024-02-13T00:46:01.447801+0000","last_became_peered":"2024-02-13T00:46:01.447801+0000","last_unstale":"2024-02-13T00:47:25.436070+0000","last_undegraded":"2024-02-13T00:47:25.436070+0000","last_fullsized":"2024-02-13T00:47:25.436070+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:45:24.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:24.269595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":417,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157797+0000","last_change":"2024-02-13T00:42:28.037452+0000","last_active":"2024-02-13T00:47:26.157797+0000","last_peered":"2024-02-13T00:47:26.157797+0000","last_clean":"2024-02-13T00:47:26.157797+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:47:26.157797+0000","last_undegraded":"2024-02-13T00:47:26.157797+0000","last_fullsized":"2024-02-13T00:47:26.157797+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:28.037365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:28.037365+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":427,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438088+0000","last_change":"2024-02-13T00:44:50.441408+0000","last_active":"2024-02-13T00:47:25.438088+0000","last_peered":"2024-02-13T00:47:25.438088+0000","last_clean":"2024-02-13T00:47:25.438088+0000","last_became_active":"2024-02-13T00:44:50.440301+0000","last_became_peered":"2024-02-13T00:44:50.440301+0000","last_unstale":"2024-02-13T00:47:25.438088+0000","last_undegraded":"2024-02-13T00:47:25.438088+0000","last_fullsized":"2024-02-13T00:47:25.438088+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:25.502311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.502311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":73,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.439917+0000","last_change":"2024-02-13T00:46:01.454885+0000","last_active":"2024-02-13T00:47:25.439917+0000","last_peered":"2024-02-13T00:47:25.439917+0000","last_clean":"2024-02-13T00:47:25.439917+0000","last_became_active":"2024-02-13T00:46:01.454535+0000","last_became_peered":"2024-02-13T00:46:01.454535+0000","last_unstale":"2024-02-13T00:47:25.439917+0000","last_undegraded":"2024-02-13T00:47:25.439917+0000","last_fullsized":"2024-02-13T00:47:25.439917+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.488353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.488353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"85'15","reported_seq":154,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437190+0000","last_change":"2024-02-13T00:44:46.481036+0000","last_active":"2024-02-13T00:47:25.437190+0000","last_peered":"2024-02-13T00:47:25.437190+0000","last_clean":"2024-02-13T00:47:25.437190+0000","last_became_active":"2024-02-13T00:44:39.523138+0000","last_became_peered":"2024-02-13T00:44:39.523138+0000","last_unstale":"2024-02-13T00:47:25.437190+0000","last_undegraded":"2024-02-13T00:47:25.437190+0000","last_fullsized":"2024-02-13T00:47:25.437190+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:43:00.776560+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.776560+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":446,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.156208+0000","last_change":"2024-02-13T00:46:01.453216+0000","last_active":"2024-02-13T00:47:26.156208+0000","last_peered":"2024-02-13T00:47:26.156208+0000","last_clean":"2024-02-13T00:47:26.156208+0000","last_became_active":"2024-02-13T00:46:01.452674+0000","last_became_peered":"2024-02-13T00:46:01.452674+0000","last_unstale":"2024-02-13T00:47:26.156208+0000","last_undegraded":"2024-02-13T00:47:26.156208+0000","last_fullsized":"2024-02-13T00:47:26.156208+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:24.059313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:24.059313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"404'11","reported_seq":117,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-13T00:47:32.591489+0000","last_change":"2024-02-13T00:46:37.662485+0000","last_active":"2024-02-13T00:47:32.591489+0000","last_peered":"2024-02-13T00:47:32.591489+0000","last_clean":"2024-02-13T00:47:32.591489+0000","last_became_active":"2024-02-13T00:46:01.456187+0000","last_became_peered":"2024-02-13T00:46:01.456187+0000","last_unstale":"2024-02-13T00:47:32.591489+0000","last_undegraded":"2024-02-13T00:47:32.591489+0000","last_fullsized":"2024-02-13T00:47:32.591489+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'11","last_scrub_stamp":"2024-02-13T00:46:37.662413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.662413+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":418,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157998+0000","last_change":"2024-02-13T00:46:05.839833+0000","last_active":"2024-02-13T00:47:26.157998+0000","last_peered":"2024-02-13T00:47:26.157998+0000","last_clean":"2024-02-13T00:47:26.157998+0000","last_became_active":"2024-02-13T00:46:01.455274+0000","last_became_peered":"2024-02-13T00:46:01.455274+0000","last_unstale":"2024-02-13T00:47:26.157998+0000","last_undegraded":"2024-02-13T00:47:26.157998+0000","last_fullsized":"2024-02-13T00:47:26.157998+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:44:25.230980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.230980+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":227,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.158381+0000","last_change":"2024-02-13T00:46:01.457421+0000","last_active":"2024-02-13T00:47:26.158381+0000","last_peered":"2024-02-13T00:47:26.158381+0000","last_clean":"2024-02-13T00:47:26.158381+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:47:26.158381+0000","last_undegraded":"2024-02-13T00:47:26.158381+0000","last_fullsized":"2024-02-13T00:47:26.158381+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:18.089252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:18.089252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"64'15","reported_seq":517,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.158297+0000","last_change":"2024-02-13T00:46:05.836637+0000","last_active":"2024-02-13T00:47:26.158297+0000","last_peered":"2024-02-13T00:47:26.158297+0000","last_clean":"2024-02-13T00:47:26.158297+0000","last_became_active":"2024-02-13T00:46:01.448705+0000","last_became_peered":"2024-02-13T00:46:01.448705+0000","last_unstale":"2024-02-13T00:47:26.158297+0000","last_undegraded":"2024-02-13T00:47:26.158297+0000","last_fullsized":"2024-02-13T00:47:26.158297+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:42:21.069482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.069482+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":478,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155507+0000","last_change":"2024-02-13T00:45:15.445834+0000","last_active":"2024-02-13T00:47:26.155507+0000","last_peered":"2024-02-13T00:47:26.155507+0000","last_clean":"2024-02-13T00:47:26.155507+0000","last_became_active":"2024-02-13T00:45:15.445228+0000","last_became_peered":"2024-02-13T00:45:15.445228+0000","last_unstale":"2024-02-13T00:47:26.155507+0000","last_undegraded":"2024-02-13T00:47:26.155507+0000","last_fullsized":"2024-02-13T00:47:26.155507+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:43:34.320637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.320637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"404'10","reported_seq":109,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-13T00:47:32.592613+0000","last_change":"2024-02-13T00:46:08.351827+0000","last_active":"2024-02-13T00:47:32.592613+0000","last_peered":"2024-02-13T00:47:32.592613+0000","last_clean":"2024-02-13T00:47:32.592613+0000","last_became_active":"2024-02-13T00:46:01.448785+0000","last_became_peered":"2024-02-13T00:46:01.448785+0000","last_unstale":"2024-02-13T00:47:32.592613+0000","last_undegraded":"2024-02-13T00:47:32.592613+0000","last_fullsized":"2024-02-13T00:47:32.592613+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-02-13T00:42:23.496800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:23.496800+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":68,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157629+0000","last_change":"2024-02-13T00:46:01.457110+0000","last_active":"2024-02-13T00:47:26.157629+0000","last_peered":"2024-02-13T00:47:26.157629+0000","last_clean":"2024-02-13T00:47:26.157629+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:47:26.157629+0000","last_undegraded":"2024-02-13T00:47:26.157629+0000","last_fullsized":"2024-02-13T00:47:26.157629+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.092250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.092250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":382,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440422+0000","last_change":"2024-02-13T00:46:01.456531+0000","last_active":"2024-02-13T00:47:25.440422+0000","last_peered":"2024-02-13T00:47:25.440422+0000","last_clean":"2024-02-13T00:47:25.440422+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:47:25.440422+0000","last_undegraded":"2024-02-13T00:47:25.440422+0000","last_fullsized":"2024-02-13T00:47:25.440422+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:44:59.963616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"65'17","reported_seq":156,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438405+0000","last_change":"2024-02-13T00:44:45.962479+0000","last_active":"2024-02-13T00:47:25.438405+0000","last_peered":"2024-02-13T00:47:25.438405+0000","last_clean":"2024-02-13T00:47:25.438405+0000","last_became_active":"2024-02-13T00:44:39.521984+0000","last_became_peered":"2024-02-13T00:44:39.521984+0000","last_unstale":"2024-02-13T00:47:25.438405+0000","last_undegraded":"2024-02-13T00:47:25.438405+0000","last_fullsized":"2024-02-13T00:47:25.438405+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:42:52.245750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.245750+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":265,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438027+0000","last_change":"2024-02-13T00:46:23.757417+0000","last_active":"2024-02-13T00:47:25.438027+0000","last_peered":"2024-02-13T00:47:25.438027+0000","last_clean":"2024-02-13T00:47:25.438027+0000","last_became_active":"2024-02-13T00:45:15.419239+0000","last_became_peered":"2024-02-13T00:45:15.419239+0000","last_unstale":"2024-02-13T00:47:25.438027+0000","last_undegraded":"2024-02-13T00:47:25.438027+0000","last_fullsized":"2024-02-13T00:47:25.438027+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:23.757351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:23.757351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":362,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438032+0000","last_change":"2024-02-13T00:46:01.460527+0000","last_active":"2024-02-13T00:47:25.438032+0000","last_peered":"2024-02-13T00:47:25.438032+0000","last_clean":"2024-02-13T00:47:25.438032+0000","last_became_active":"2024-02-13T00:46:01.459122+0000","last_became_peered":"2024-02-13T00:46:01.459122+0000","last_unstale":"2024-02-13T00:47:25.438032+0000","last_undegraded":"2024-02-13T00:47:25.438032+0000","last_fullsized":"2024-02-13T00:47:25.438032+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:57.546517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.546517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437056+0000","last_change":"2024-02-13T00:46:01.463229+0000","last_active":"2024-02-13T00:47:25.437056+0000","last_peered":"2024-02-13T00:47:25.437056+0000","last_clean":"2024-02-13T00:47:25.437056+0000","last_became_active":"2024-02-13T00:46:01.462717+0000","last_became_peered":"2024-02-13T00:46:01.462717+0000","last_unstale":"2024-02-13T00:47:25.437056+0000","last_undegraded":"2024-02-13T00:47:25.437056+0000","last_fullsized":"2024-02-13T00:47:25.437056+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.782345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.782345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":334,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437635+0000","last_change":"2024-02-13T00:46:01.457255+0000","last_active":"2024-02-13T00:47:25.437635+0000","last_peered":"2024-02-13T00:47:25.437635+0000","last_clean":"2024-02-13T00:47:25.437635+0000","last_became_active":"2024-02-13T00:46:01.456959+0000","last_became_peered":"2024-02-13T00:46:01.456959+0000","last_unstale":"2024-02-13T00:47:25.437635+0000","last_undegraded":"2024-02-13T00:47:25.437635+0000","last_fullsized":"2024-02-13T00:47:25.437635+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.281102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.281102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"263'10","reported_seq":428,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-13T00:47:33.237281+0000","last_change":"2024-02-13T00:47:02.949483+0000","last_active":"2024-02-13T00:47:33.237281+0000","last_peered":"2024-02-13T00:47:33.237281+0000","last_clean":"2024-02-13T00:47:33.237281+0000","last_became_active":"2024-02-13T00:43:35.454087+0000","last_became_peered":"2024-02-13T00:43:35.454087+0000","last_unstale":"2024-02-13T00:47:33.237281+0000","last_undegraded":"2024-02-13T00:47:33.237281+0000","last_fullsized":"2024-02-13T00:47:33.237281+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.949406+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":411,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435514+0000","last_change":"2024-02-13T00:46:08.351239+0000","last_active":"2024-02-13T00:47:25.435514+0000","last_peered":"2024-02-13T00:47:25.435514+0000","last_clean":"2024-02-13T00:47:25.435514+0000","last_became_active":"2024-02-13T00:46:01.449654+0000","last_became_peered":"2024-02-13T00:46:01.449654+0000","last_unstale":"2024-02-13T00:47:25.435514+0000","last_undegraded":"2024-02-13T00:47:25.435514+0000","last_fullsized":"2024-02-13T00:47:25.435514+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2024-02-13T00:43:00.636798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.636798+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":31,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438294+0000","last_change":"2024-02-13T00:45:15.446786+0000","last_active":"2024-02-13T00:47:25.438294+0000","last_peered":"2024-02-13T00:47:25.438294+0000","last_clean":"2024-02-13T00:47:25.438294+0000","last_became_active":"2024-02-13T00:45:15.446443+0000","last_became_peered":"2024-02-13T00:45:15.446443+0000","last_unstale":"2024-02-13T00:47:25.438294+0000","last_undegraded":"2024-02-13T00:47:25.438294+0000","last_fullsized":"2024-02-13T00:47:25.438294+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.557475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.557475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436648+0000","last_change":"2024-02-13T00:45:15.436286+0000","last_active":"2024-02-13T00:47:25.436648+0000","last_peered":"2024-02-13T00:47:25.436648+0000","last_clean":"2024-02-13T00:47:25.436648+0000","last_became_active":"2024-02-13T00:45:15.434559+0000","last_became_peered":"2024-02-13T00:45:15.434559+0000","last_unstale":"2024-02-13T00:47:25.436648+0000","last_undegraded":"2024-02-13T00:47:25.436648+0000","last_fullsized":"2024-02-13T00:47:25.436648+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":184,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437841+0000","last_change":"2024-02-13T00:46:01.445822+0000","last_active":"2024-02-13T00:47:25.437841+0000","last_peered":"2024-02-13T00:47:25.437841+0000","last_clean":"2024-02-13T00:47:25.437841+0000","last_became_active":"2024-02-13T00:46:01.444661+0000","last_became_peered":"2024-02-13T00:46:01.444661+0000","last_unstale":"2024-02-13T00:47:25.437841+0000","last_undegraded":"2024-02-13T00:47:25.437841+0000","last_fullsized":"2024-02-13T00:47:25.437841+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:43.049231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:43.049231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":225,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437872+0000","last_change":"2024-02-13T00:46:01.454510+0000","last_active":"2024-02-13T00:47:25.437872+0000","last_peered":"2024-02-13T00:47:25.437872+0000","last_clean":"2024-02-13T00:47:25.437872+0000","last_became_active":"2024-02-13T00:46:01.453842+0000","last_became_peered":"2024-02-13T00:46:01.453842+0000","last_unstale":"2024-02-13T00:47:25.437872+0000","last_undegraded":"2024-02-13T00:47:25.437872+0000","last_fullsized":"2024-02-13T00:47:25.437872+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:54.842056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:54.842056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"65'30","reported_seq":368,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437939+0000","last_change":"2024-02-13T00:46:04.744348+0000","last_active":"2024-02-13T00:47:25.437939+0000","last_peered":"2024-02-13T00:47:25.437939+0000","last_clean":"2024-02-13T00:47:25.437939+0000","last_became_active":"2024-02-13T00:46:01.445058+0000","last_became_peered":"2024-02-13T00:46:01.445058+0000","last_unstale":"2024-02-13T00:47:25.437939+0000","last_undegraded":"2024-02-13T00:47:25.437939+0000","last_fullsized":"2024-02-13T00:47:25.437939+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:42:53.881594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.881594+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":451,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157835+0000","last_change":"2024-02-13T00:47:09.940673+0000","last_active":"2024-02-13T00:47:26.157835+0000","last_peered":"2024-02-13T00:47:26.157835+0000","last_clean":"2024-02-13T00:47:26.157835+0000","last_became_active":"2024-02-13T00:44:39.503590+0000","last_became_peered":"2024-02-13T00:44:39.503590+0000","last_unstale":"2024-02-13T00:47:26.157835+0000","last_undegraded":"2024-02-13T00:47:26.157835+0000","last_fullsized":"2024-02-13T00:47:26.157835+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:09.940605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:09.940605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":395,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435721+0000","last_change":"2024-02-13T00:46:01.448313+0000","last_active":"2024-02-13T00:47:25.435721+0000","last_peered":"2024-02-13T00:47:25.435721+0000","last_clean":"2024-02-13T00:47:25.435721+0000","last_became_active":"2024-02-13T00:46:01.447815+0000","last_became_peered":"2024-02-13T00:46:01.447815+0000","last_unstale":"2024-02-13T00:47:25.435721+0000","last_undegraded":"2024-02-13T00:47:25.435721+0000","last_fullsized":"2024-02-13T00:47:25.435721+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":382,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436278+0000","last_change":"2024-02-13T00:44:50.436745+0000","last_active":"2024-02-13T00:47:25.436278+0000","last_peered":"2024-02-13T00:47:25.436278+0000","last_clean":"2024-02-13T00:47:25.436278+0000","last_became_active":"2024-02-13T00:44:50.436058+0000","last_became_peered":"2024-02-13T00:44:50.436058+0000","last_unstale":"2024-02-13T00:47:25.436278+0000","last_undegraded":"2024-02-13T00:47:25.436278+0000","last_fullsized":"2024-02-13T00:47:25.436278+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:48.300400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.300400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":156,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436168+0000","last_change":"2024-02-13T00:46:01.451625+0000","last_active":"2024-02-13T00:47:25.436168+0000","last_peered":"2024-02-13T00:47:25.436168+0000","last_clean":"2024-02-13T00:47:25.436168+0000","last_became_active":"2024-02-13T00:46:01.450587+0000","last_became_peered":"2024-02-13T00:46:01.450587+0000","last_unstale":"2024-02-13T00:47:25.436168+0000","last_undegraded":"2024-02-13T00:47:25.436168+0000","last_fullsized":"2024-02-13T00:47:25.436168+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:33.204969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:33.204969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"80'34","reported_seq":539,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438304+0000","last_change":"2024-02-13T00:47:05.718055+0000","last_active":"2024-02-13T00:47:25.438304+0000","last_peered":"2024-02-13T00:47:25.438304+0000","last_clean":"2024-02-13T00:47:25.438304+0000","last_became_active":"2024-02-13T00:46:01.443777+0000","last_became_peered":"2024-02-13T00:46:01.443777+0000","last_unstale":"2024-02-13T00:47:25.438304+0000","last_undegraded":"2024-02-13T00:47:25.438304+0000","last_fullsized":"2024-02-13T00:47:25.438304+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:47:05.717987+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:47:05.717987+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":265,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.439666+0000","last_change":"2024-02-13T00:45:15.423607+0000","last_active":"2024-02-13T00:47:25.439666+0000","last_peered":"2024-02-13T00:47:25.439666+0000","last_clean":"2024-02-13T00:47:25.439666+0000","last_became_active":"2024-02-13T00:45:15.423415+0000","last_became_peered":"2024-02-13T00:45:15.423415+0000","last_unstale":"2024-02-13T00:47:25.439666+0000","last_undegraded":"2024-02-13T00:47:25.439666+0000","last_fullsized":"2024-02-13T00:47:25.439666+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:17.295459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:17.295459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155583+0000","last_change":"2024-02-13T00:45:15.445862+0000","last_active":"2024-02-13T00:47:26.155583+0000","last_peered":"2024-02-13T00:47:26.155583+0000","last_clean":"2024-02-13T00:47:26.155583+0000","last_became_active":"2024-02-13T00:45:15.445557+0000","last_became_peered":"2024-02-13T00:45:15.445557+0000","last_unstale":"2024-02-13T00:47:26.155583+0000","last_undegraded":"2024-02-13T00:47:26.155583+0000","last_fullsized":"2024-02-13T00:47:26.155583+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":71,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157657+0000","last_change":"2024-02-13T00:46:01.442629+0000","last_active":"2024-02-13T00:47:26.157657+0000","last_peered":"2024-02-13T00:47:26.157657+0000","last_clean":"2024-02-13T00:47:26.157657+0000","last_became_active":"2024-02-13T00:46:01.441421+0000","last_became_peered":"2024-02-13T00:46:01.441421+0000","last_unstale":"2024-02-13T00:47:26.157657+0000","last_undegraded":"2024-02-13T00:47:26.157657+0000","last_fullsized":"2024-02-13T00:47:26.157657+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:50.325063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:50.325063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":68,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.158812+0000","last_change":"2024-02-13T00:46:01.450756+0000","last_active":"2024-02-13T00:47:26.158812+0000","last_peered":"2024-02-13T00:47:26.158812+0000","last_clean":"2024-02-13T00:47:26.158812+0000","last_became_active":"2024-02-13T00:46:01.449142+0000","last_became_peered":"2024-02-13T00:46:01.449142+0000","last_unstale":"2024-02-13T00:47:26.158812+0000","last_undegraded":"2024-02-13T00:47:26.158812+0000","last_fullsized":"2024-02-13T00:47:26.158812+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:54.422484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:54.422484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"64'12","reported_seq":531,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.439994+0000","last_change":"2024-02-13T00:46:34.691109+0000","last_active":"2024-02-13T00:47:25.439994+0000","last_peered":"2024-02-13T00:47:25.439994+0000","last_clean":"2024-02-13T00:47:25.439994+0000","last_became_active":"2024-02-13T00:46:01.448163+0000","last_became_peered":"2024-02-13T00:46:01.448163+0000","last_unstale":"2024-02-13T00:47:25.439994+0000","last_undegraded":"2024-02-13T00:47:25.439994+0000","last_fullsized":"2024-02-13T00:47:25.439994+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:46:34.690980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.690980+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":159,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155743+0000","last_change":"2024-02-13T00:46:37.056299+0000","last_active":"2024-02-13T00:47:26.155743+0000","last_peered":"2024-02-13T00:47:26.155743+0000","last_clean":"2024-02-13T00:47:26.155743+0000","last_became_active":"2024-02-13T00:46:01.446104+0000","last_became_peered":"2024-02-13T00:46:01.446104+0000","last_unstale":"2024-02-13T00:47:26.155743+0000","last_undegraded":"2024-02-13T00:47:26.155743+0000","last_fullsized":"2024-02-13T00:47:26.155743+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:37.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":441,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438219+0000","last_change":"2024-02-13T00:46:01.431899+0000","last_active":"2024-02-13T00:47:25.438219+0000","last_peered":"2024-02-13T00:47:25.438219+0000","last_clean":"2024-02-13T00:47:25.438219+0000","last_became_active":"2024-02-13T00:46:01.430623+0000","last_became_peered":"2024-02-13T00:46:01.430623+0000","last_unstale":"2024-02-13T00:47:25.438219+0000","last_undegraded":"2024-02-13T00:47:25.438219+0000","last_fullsized":"2024-02-13T00:47:25.438219+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:46.556229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.556229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":449,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157885+0000","last_change":"2024-02-13T00:46:01.450548+0000","last_active":"2024-02-13T00:47:26.157885+0000","last_peered":"2024-02-13T00:47:26.157885+0000","last_clean":"2024-02-13T00:47:26.157885+0000","last_became_active":"2024-02-13T00:46:01.450176+0000","last_became_peered":"2024-02-13T00:46:01.450176+0000","last_unstale":"2024-02-13T00:47:26.157885+0000","last_undegraded":"2024-02-13T00:47:26.157885+0000","last_fullsized":"2024-02-13T00:47:26.157885+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":302,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438251+0000","last_change":"2024-02-13T00:46:01.460470+0000","last_active":"2024-02-13T00:47:25.438251+0000","last_peered":"2024-02-13T00:47:25.438251+0000","last_clean":"2024-02-13T00:47:25.438251+0000","last_became_active":"2024-02-13T00:46:01.459859+0000","last_became_peered":"2024-02-13T00:46:01.459859+0000","last_unstale":"2024-02-13T00:47:25.438251+0000","last_undegraded":"2024-02-13T00:47:25.438251+0000","last_fullsized":"2024-02-13T00:47:25.438251+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:01.179031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:01.179031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":106,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-13T00:47:33.932830+0000","last_change":"2024-02-13T00:47:33.932830+0000","last_active":"2024-02-13T00:47:33.932830+0000","last_peered":"2024-02-13T00:47:33.932830+0000","last_clean":"2024-02-13T00:47:33.932830+0000","last_became_active":"2024-02-13T00:46:01.455170+0000","last_became_peered":"2024-02-13T00:46:01.455170+0000","last_unstale":"2024-02-13T00:47:33.932830+0000","last_undegraded":"2024-02-13T00:47:33.932830+0000","last_fullsized":"2024-02-13T00:47:33.932830+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:33.932766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:33.932766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"157'21","reported_seq":188,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438313+0000","last_change":"2024-02-13T00:46:57.381602+0000","last_active":"2024-02-13T00:47:25.438313+0000","last_peered":"2024-02-13T00:47:25.438313+0000","last_clean":"2024-02-13T00:47:25.438313+0000","last_became_active":"2024-02-13T00:45:15.446969+0000","last_became_peered":"2024-02-13T00:45:15.446969+0000","last_unstale":"2024-02-13T00:47:25.438313+0000","last_undegraded":"2024-02-13T00:47:25.438313+0000","last_fullsized":"2024-02-13T00:47:25.438313+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:46:57.381540+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:46:57.381540+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":401,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437686+0000","last_change":"2024-02-13T00:45:15.434844+0000","last_active":"2024-02-13T00:47:25.437686+0000","last_peered":"2024-02-13T00:47:25.437686+0000","last_clean":"2024-02-13T00:47:25.437686+0000","last_became_active":"2024-02-13T00:45:15.434498+0000","last_became_peered":"2024-02-13T00:45:15.434498+0000","last_unstale":"2024-02-13T00:47:25.437686+0000","last_undegraded":"2024-02-13T00:47:25.437686+0000","last_fullsized":"2024-02-13T00:47:25.437686+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:23.561163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:42:23.561163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":404,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436186+0000","last_change":"2024-02-13T00:47:04.493911+0000","last_active":"2024-02-13T00:47:25.436186+0000","last_peered":"2024-02-13T00:47:25.436186+0000","last_clean":"2024-02-13T00:47:25.436186+0000","last_became_active":"2024-02-13T00:46:01.447823+0000","last_became_peered":"2024-02-13T00:46:01.447823+0000","last_unstale":"2024-02-13T00:47:25.436186+0000","last_undegraded":"2024-02-13T00:47:25.436186+0000","last_fullsized":"2024-02-13T00:47:25.436186+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:04.493841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:04.493841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":272,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435595+0000","last_change":"2024-02-13T00:46:01.460412+0000","last_active":"2024-02-13T00:47:25.435595+0000","last_peered":"2024-02-13T00:47:25.435595+0000","last_clean":"2024-02-13T00:47:25.435595+0000","last_became_active":"2024-02-13T00:46:01.460001+0000","last_became_peered":"2024-02-13T00:46:01.460001+0000","last_unstale":"2024-02-13T00:47:25.435595+0000","last_undegraded":"2024-02-13T00:47:25.435595+0000","last_fullsized":"2024-02-13T00:47:25.435595+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.544320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.544320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":297,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.158013+0000","last_change":"2024-02-13T00:46:01.450879+0000","last_active":"2024-02-13T00:47:26.158013+0000","last_peered":"2024-02-13T00:47:26.158013+0000","last_clean":"2024-02-13T00:47:26.158013+0000","last_became_active":"2024-02-13T00:46:01.450088+0000","last_became_peered":"2024-02-13T00:46:01.450088+0000","last_unstale":"2024-02-13T00:47:26.158013+0000","last_undegraded":"2024-02-13T00:47:26.158013+0000","last_fullsized":"2024-02-13T00:47:26.158013+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":382,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440277+0000","last_change":"2024-02-13T00:46:01.459317+0000","last_active":"2024-02-13T00:47:25.440277+0000","last_peered":"2024-02-13T00:47:25.440277+0000","last_clean":"2024-02-13T00:47:25.440277+0000","last_became_active":"2024-02-13T00:46:01.458408+0000","last_became_peered":"2024-02-13T00:46:01.458408+0000","last_unstale":"2024-02-13T00:47:25.440277+0000","last_undegraded":"2024-02-13T00:47:25.440277+0000","last_fullsized":"2024-02-13T00:47:25.440277+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:16.256516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:16.256516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":105,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157504+0000","last_change":"2024-02-13T00:46:01.442001+0000","last_active":"2024-02-13T00:47:26.157504+0000","last_peered":"2024-02-13T00:47:26.157504+0000","last_clean":"2024-02-13T00:47:26.157504+0000","last_became_active":"2024-02-13T00:46:01.441013+0000","last_became_peered":"2024-02-13T00:46:01.441013+0000","last_unstale":"2024-02-13T00:47:26.157504+0000","last_undegraded":"2024-02-13T00:47:26.157504+0000","last_fullsized":"2024-02-13T00:47:26.157504+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:26.313580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:26.313580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"65'19","reported_seq":482,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.158550+0000","last_change":"2024-02-13T00:46:04.714305+0000","last_active":"2024-02-13T00:47:26.158550+0000","last_peered":"2024-02-13T00:47:26.158550+0000","last_clean":"2024-02-13T00:47:26.158550+0000","last_became_active":"2024-02-13T00:46:01.448219+0000","last_became_peered":"2024-02-13T00:46:01.448219+0000","last_unstale":"2024-02-13T00:47:26.158550+0000","last_undegraded":"2024-02-13T00:47:26.158550+0000","last_fullsized":"2024-02-13T00:47:26.158550+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:42:54.174950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:54.174950+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":356,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438161+0000","last_change":"2024-02-13T00:44:50.426201+0000","last_active":"2024-02-13T00:47:25.438161+0000","last_peered":"2024-02-13T00:47:25.438161+0000","last_clean":"2024-02-13T00:47:25.438161+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:47:25.438161+0000","last_undegraded":"2024-02-13T00:47:25.438161+0000","last_fullsized":"2024-02-13T00:47:25.438161+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:20.481083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:44:20.481083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":1053,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-13T00:47:33.718572+0000","last_change":"2024-02-13T00:45:54.075270+0000","last_active":"2024-02-13T00:47:33.718572+0000","last_peered":"2024-02-13T00:47:33.718572+0000","last_clean":"2024-02-13T00:47:33.718572+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:47:33.718572+0000","last_undegraded":"2024-02-13T00:47:33.718572+0000","last_fullsized":"2024-02-13T00:47:33.718572+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:45:54.075194+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":581,"num_read_kb":576,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":169,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438145+0000","last_change":"2024-02-13T00:46:01.460282+0000","last_active":"2024-02-13T00:47:25.438145+0000","last_peered":"2024-02-13T00:47:25.438145+0000","last_clean":"2024-02-13T00:47:25.438145+0000","last_became_active":"2024-02-13T00:46:01.459473+0000","last_became_peered":"2024-02-13T00:46:01.459473+0000","last_unstale":"2024-02-13T00:47:25.438145+0000","last_undegraded":"2024-02-13T00:47:25.438145+0000","last_fullsized":"2024-02-13T00:47:25.438145+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":222,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.156858+0000","last_change":"2024-02-13T00:45:37.135398+0000","last_active":"2024-02-13T00:47:26.156858+0000","last_peered":"2024-02-13T00:47:26.156858+0000","last_clean":"2024-02-13T00:47:26.156858+0000","last_became_active":"2024-02-13T00:45:15.435588+0000","last_became_peered":"2024-02-13T00:45:15.435588+0000","last_unstale":"2024-02-13T00:47:26.156858+0000","last_undegraded":"2024-02-13T00:47:26.156858+0000","last_fullsized":"2024-02-13T00:47:26.156858+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.135332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.135332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":120,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436814+0000","last_change":"2024-02-13T00:45:31.183763+0000","last_active":"2024-02-13T00:47:25.436814+0000","last_peered":"2024-02-13T00:47:25.436814+0000","last_clean":"2024-02-13T00:47:25.436814+0000","last_became_active":"2024-02-13T00:45:06.421025+0000","last_became_peered":"2024-02-13T00:45:06.421025+0000","last_unstale":"2024-02-13T00:47:25.436814+0000","last_undegraded":"2024-02-13T00:47:25.436814+0000","last_fullsized":"2024-02-13T00:47:25.436814+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.183682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.183682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"405'53","reported_seq":109,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-13T00:47:33.238979+0000","last_change":"2024-02-13T00:46:04.950162+0000","last_active":"2024-02-13T00:47:33.238979+0000","last_peered":"2024-02-13T00:47:33.238979+0000","last_clean":"2024-02-13T00:47:33.238979+0000","last_became_active":"2024-02-13T00:46:01.449059+0000","last_became_peered":"2024-02-13T00:46:01.449059+0000","last_unstale":"2024-02-13T00:47:33.238979+0000","last_undegraded":"2024-02-13T00:47:33.238979+0000","last_fullsized":"2024-02-13T00:47:33.238979+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'48","last_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:43:48.212428+0000","log_size":53,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":2148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":367,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155902+0000","last_change":"2024-02-13T00:46:01.453042+0000","last_active":"2024-02-13T00:47:26.155902+0000","last_peered":"2024-02-13T00:47:26.155902+0000","last_clean":"2024-02-13T00:47:26.155902+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:47:26.155902+0000","last_undegraded":"2024-02-13T00:47:26.155902+0000","last_fullsized":"2024-02-13T00:47:26.155902+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.045912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.045912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438321+0000","last_change":"2024-02-13T00:46:01.461717+0000","last_active":"2024-02-13T00:47:25.438321+0000","last_peered":"2024-02-13T00:47:25.438321+0000","last_clean":"2024-02-13T00:47:25.438321+0000","last_became_active":"2024-02-13T00:46:01.460410+0000","last_became_peered":"2024-02-13T00:46:01.460410+0000","last_unstale":"2024-02-13T00:47:25.438321+0000","last_undegraded":"2024-02-13T00:47:25.438321+0000","last_fullsized":"2024-02-13T00:47:25.438321+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":151,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436273+0000","last_change":"2024-02-13T00:44:50.439369+0000","last_active":"2024-02-13T00:47:25.436273+0000","last_peered":"2024-02-13T00:47:25.436273+0000","last_clean":"2024-02-13T00:47:25.436273+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:47:25.436273+0000","last_undegraded":"2024-02-13T00:47:25.436273+0000","last_fullsized":"2024-02-13T00:47:25.436273+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:19.407011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:19.407011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":595,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-13T00:47:32.591529+0000","last_change":"2024-02-13T00:46:11.962872+0000","last_active":"2024-02-13T00:47:32.591529+0000","last_peered":"2024-02-13T00:47:32.591529+0000","last_clean":"2024-02-13T00:47:32.591529+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:47:32.591529+0000","last_undegraded":"2024-02-13T00:47:32.591529+0000","last_fullsized":"2024-02-13T00:47:32.591529+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'11","last_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:43:02.423358+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":170,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.156525+0000","last_change":"2024-02-13T00:46:02.664116+0000","last_active":"2024-02-13T00:47:26.156525+0000","last_peered":"2024-02-13T00:47:26.156525+0000","last_clean":"2024-02-13T00:47:26.156525+0000","last_became_active":"2024-02-13T00:46:01.442003+0000","last_became_peered":"2024-02-13T00:46:01.442003+0000","last_unstale":"2024-02-13T00:47:26.156525+0000","last_undegraded":"2024-02-13T00:47:26.156525+0000","last_fullsized":"2024-02-13T00:47:26.156525+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:52.517392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.517392+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":433,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.158715+0000","last_change":"2024-02-13T00:45:15.423417+0000","last_active":"2024-02-13T00:47:26.158715+0000","last_peered":"2024-02-13T00:47:26.158715+0000","last_clean":"2024-02-13T00:47:26.158715+0000","last_became_active":"2024-02-13T00:45:15.422546+0000","last_became_peered":"2024-02-13T00:45:15.422546+0000","last_unstale":"2024-02-13T00:47:26.158715+0000","last_undegraded":"2024-02-13T00:47:26.158715+0000","last_fullsized":"2024-02-13T00:47:26.158715+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:41.223028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:43:41.223028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":448,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437605+0000","last_change":"2024-02-13T00:46:01.436997+0000","last_active":"2024-02-13T00:47:25.437605+0000","last_peered":"2024-02-13T00:47:25.437605+0000","last_clean":"2024-02-13T00:47:25.437605+0000","last_became_active":"2024-02-13T00:46:01.435178+0000","last_became_peered":"2024-02-13T00:46:01.435178+0000","last_unstale":"2024-02-13T00:47:25.437605+0000","last_undegraded":"2024-02-13T00:47:25.437605+0000","last_fullsized":"2024-02-13T00:47:25.437605+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":382,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155405+0000","last_change":"2024-02-13T00:45:15.445626+0000","last_active":"2024-02-13T00:47:26.155405+0000","last_peered":"2024-02-13T00:47:26.155405+0000","last_clean":"2024-02-13T00:47:26.155405+0000","last_became_active":"2024-02-13T00:45:15.445117+0000","last_became_peered":"2024-02-13T00:45:15.445117+0000","last_unstale":"2024-02-13T00:47:26.155405+0000","last_undegraded":"2024-02-13T00:47:26.155405+0000","last_fullsized":"2024-02-13T00:47:26.155405+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.171185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":148,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438288+0000","last_change":"2024-02-13T00:46:01.436795+0000","last_active":"2024-02-13T00:47:25.438288+0000","last_peered":"2024-02-13T00:47:25.438288+0000","last_clean":"2024-02-13T00:47:25.438288+0000","last_became_active":"2024-02-13T00:46:01.434431+0000","last_became_peered":"2024-02-13T00:46:01.434431+0000","last_unstale":"2024-02-13T00:47:25.438288+0000","last_undegraded":"2024-02-13T00:47:25.438288+0000","last_fullsized":"2024-02-13T00:47:25.438288+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.592320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.592320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"65'24","reported_seq":494,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155378+0000","last_change":"2024-02-13T00:44:41.502069+0000","last_active":"2024-02-13T00:47:26.155378+0000","last_peered":"2024-02-13T00:47:26.155378+0000","last_clean":"2024-02-13T00:47:26.155378+0000","last_became_active":"2024-02-13T00:44:39.501886+0000","last_became_peered":"2024-02-13T00:44:39.501886+0000","last_unstale":"2024-02-13T00:47:26.155378+0000","last_undegraded":"2024-02-13T00:47:26.155378+0000","last_fullsized":"2024-02-13T00:47:26.155378+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'24","last_scrub_stamp":"2024-02-13T00:43:30.413372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.413372+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":36,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":422,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440170+0000","last_change":"2024-02-13T00:46:14.266858+0000","last_active":"2024-02-13T00:47:25.440170+0000","last_peered":"2024-02-13T00:47:25.440170+0000","last_clean":"2024-02-13T00:47:25.440170+0000","last_became_active":"2024-02-13T00:46:01.457926+0000","last_became_peered":"2024-02-13T00:46:01.457926+0000","last_unstale":"2024-02-13T00:47:25.440170+0000","last_undegraded":"2024-02-13T00:47:25.440170+0000","last_fullsized":"2024-02-13T00:47:25.440170+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:43:35.309308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:35.309308+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":233,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.156434+0000","last_change":"2024-02-13T00:47:06.895693+0000","last_active":"2024-02-13T00:47:26.156434+0000","last_peered":"2024-02-13T00:47:26.156434+0000","last_clean":"2024-02-13T00:47:26.156434+0000","last_became_active":"2024-02-13T00:45:15.435404+0000","last_became_peered":"2024-02-13T00:45:15.435404+0000","last_unstale":"2024-02-13T00:47:26.156434+0000","last_undegraded":"2024-02-13T00:47:26.156434+0000","last_fullsized":"2024-02-13T00:47:26.156434+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:06.895612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:47:06.895612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":382,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440159+0000","last_change":"2024-02-13T00:46:01.450436+0000","last_active":"2024-02-13T00:47:25.440159+0000","last_peered":"2024-02-13T00:47:25.440159+0000","last_clean":"2024-02-13T00:47:25.440159+0000","last_became_active":"2024-02-13T00:46:01.449234+0000","last_became_peered":"2024-02-13T00:46:01.449234+0000","last_unstale":"2024-02-13T00:47:25.440159+0000","last_undegraded":"2024-02-13T00:47:25.440159+0000","last_fullsized":"2024-02-13T00:47:25.440159+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.110742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.110742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"65'21","reported_seq":559,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440082+0000","last_change":"2024-02-13T00:46:12.563543+0000","last_active":"2024-02-13T00:47:25.440082+0000","last_peered":"2024-02-13T00:47:25.440082+0000","last_clean":"2024-02-13T00:47:25.440082+0000","last_became_active":"2024-02-13T00:46:01.456876+0000","last_became_peered":"2024-02-13T00:46:01.456876+0000","last_unstale":"2024-02-13T00:47:25.440082+0000","last_undegraded":"2024-02-13T00:47:25.440082+0000","last_fullsized":"2024-02-13T00:47:25.440082+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:43:30.365536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.365536+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":140,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155491+0000","last_change":"2024-02-13T00:46:30.095387+0000","last_active":"2024-02-13T00:47:26.155491+0000","last_peered":"2024-02-13T00:47:26.155491+0000","last_clean":"2024-02-13T00:47:26.155491+0000","last_became_active":"2024-02-13T00:44:50.441300+0000","last_became_peered":"2024-02-13T00:44:50.441300+0000","last_unstale":"2024-02-13T00:47:26.155491+0000","last_undegraded":"2024-02-13T00:47:26.155491+0000","last_fullsized":"2024-02-13T00:47:26.155491+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.095307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":219,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.158461+0000","last_change":"2024-02-13T00:45:15.416946+0000","last_active":"2024-02-13T00:47:26.158461+0000","last_peered":"2024-02-13T00:47:26.158461+0000","last_clean":"2024-02-13T00:47:26.158461+0000","last_became_active":"2024-02-13T00:45:15.416364+0000","last_became_peered":"2024-02-13T00:45:15.416364+0000","last_unstale":"2024-02-13T00:47:26.158461+0000","last_undegraded":"2024-02-13T00:47:26.158461+0000","last_fullsized":"2024-02-13T00:47:26.158461+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.458752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.458752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":155,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.439883+0000","last_change":"2024-02-13T00:46:01.454597+0000","last_active":"2024-02-13T00:47:25.439883+0000","last_peered":"2024-02-13T00:47:25.439883+0000","last_clean":"2024-02-13T00:47:25.439883+0000","last_became_active":"2024-02-13T00:46:01.454101+0000","last_became_peered":"2024-02-13T00:46:01.454101+0000","last_unstale":"2024-02-13T00:47:25.439883+0000","last_undegraded":"2024-02-13T00:47:25.439883+0000","last_fullsized":"2024-02-13T00:47:25.439883+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.651091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.651091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":151,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438288+0000","last_change":"2024-02-13T00:44:50.441196+0000","last_active":"2024-02-13T00:47:25.438288+0000","last_peered":"2024-02-13T00:47:25.438288+0000","last_clean":"2024-02-13T00:47:25.438288+0000","last_became_active":"2024-02-13T00:44:50.438274+0000","last_became_peered":"2024-02-13T00:44:50.438274+0000","last_unstale":"2024-02-13T00:47:25.438288+0000","last_undegraded":"2024-02-13T00:47:25.438288+0000","last_fullsized":"2024-02-13T00:47:25.438288+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.510219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.510219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'12","reported_seq":300,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.158758+0000","last_change":"2024-02-13T00:46:34.141015+0000","last_active":"2024-02-13T00:47:26.158758+0000","last_peered":"2024-02-13T00:47:26.158758+0000","last_clean":"2024-02-13T00:47:26.158758+0000","last_became_active":"2024-02-13T00:45:15.436020+0000","last_became_peered":"2024-02-13T00:45:15.436020+0000","last_unstale":"2024-02-13T00:47:26.158758+0000","last_undegraded":"2024-02-13T00:47:26.158758+0000","last_fullsized":"2024-02-13T00:47:26.158758+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:46:34.140928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.140928+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":147,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435909+0000","last_change":"2024-02-13T00:45:00.311635+0000","last_active":"2024-02-13T00:47:25.435909+0000","last_peered":"2024-02-13T00:47:25.435909+0000","last_clean":"2024-02-13T00:47:25.435909+0000","last_became_active":"2024-02-13T00:44:39.512402+0000","last_became_peered":"2024-02-13T00:44:39.512402+0000","last_unstale":"2024-02-13T00:47:25.435909+0000","last_undegraded":"2024-02-13T00:47:25.435909+0000","last_fullsized":"2024-02-13T00:47:25.435909+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:00.311558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:45:00.311558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":363,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157971+0000","last_change":"2024-02-13T00:46:01.455806+0000","last_active":"2024-02-13T00:47:26.157971+0000","last_peered":"2024-02-13T00:47:26.157971+0000","last_clean":"2024-02-13T00:47:26.157971+0000","last_became_active":"2024-02-13T00:46:01.453265+0000","last_became_peered":"2024-02-13T00:46:01.453265+0000","last_unstale":"2024-02-13T00:47:26.157971+0000","last_undegraded":"2024-02-13T00:47:26.157971+0000","last_fullsized":"2024-02-13T00:47:26.157971+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.106639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.106639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438173+0000","last_change":"2024-02-13T00:46:01.451840+0000","last_active":"2024-02-13T00:47:25.438173+0000","last_peered":"2024-02-13T00:47:25.438173+0000","last_clean":"2024-02-13T00:47:25.438173+0000","last_became_active":"2024-02-13T00:46:01.450312+0000","last_became_peered":"2024-02-13T00:46:01.450312+0000","last_unstale":"2024-02-13T00:47:25.438173+0000","last_undegraded":"2024-02-13T00:47:25.438173+0000","last_fullsized":"2024-02-13T00:47:25.438173+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.042291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.042291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":232,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440458+0000","last_change":"2024-02-13T00:46:01.454858+0000","last_active":"2024-02-13T00:47:25.440458+0000","last_peered":"2024-02-13T00:47:25.440458+0000","last_clean":"2024-02-13T00:47:25.440458+0000","last_became_active":"2024-02-13T00:46:01.454284+0000","last_became_peered":"2024-02-13T00:46:01.454284+0000","last_unstale":"2024-02-13T00:47:25.440458+0000","last_undegraded":"2024-02-13T00:47:25.440458+0000","last_fullsized":"2024-02-13T00:47:25.440458+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:34.523594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.523594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"76'29","reported_seq":372,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157711+0000","last_change":"2024-02-13T00:46:04.434215+0000","last_active":"2024-02-13T00:47:26.157711+0000","last_peered":"2024-02-13T00:47:26.157711+0000","last_clean":"2024-02-13T00:47:26.157711+0000","last_became_active":"2024-02-13T00:46:01.450113+0000","last_became_peered":"2024-02-13T00:46:01.450113+0000","last_unstale":"2024-02-13T00:47:26.157711+0000","last_undegraded":"2024-02-13T00:47:26.157711+0000","last_fullsized":"2024-02-13T00:47:26.157711+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:42:48.774168+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.774168+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":402,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438560+0000","last_change":"2024-02-13T00:46:01.454689+0000","last_active":"2024-02-13T00:47:25.438560+0000","last_peered":"2024-02-13T00:47:25.438560+0000","last_clean":"2024-02-13T00:47:25.438560+0000","last_became_active":"2024-02-13T00:46:01.454310+0000","last_became_peered":"2024-02-13T00:46:01.454310+0000","last_unstale":"2024-02-13T00:47:25.438560+0000","last_undegraded":"2024-02-13T00:47:25.438560+0000","last_fullsized":"2024-02-13T00:47:25.438560+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.230330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.230330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"404'9","reported_seq":110,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-13T00:47:33.240365+0000","last_change":"2024-02-13T00:46:04.816167+0000","last_active":"2024-02-13T00:47:33.240365+0000","last_peered":"2024-02-13T00:47:33.240365+0000","last_clean":"2024-02-13T00:47:33.240365+0000","last_became_active":"2024-02-13T00:46:01.448144+0000","last_became_peered":"2024-02-13T00:46:01.448144+0000","last_unstale":"2024-02-13T00:47:33.240365+0000","last_undegraded":"2024-02-13T00:47:33.240365+0000","last_fullsized":"2024-02-13T00:47:33.240365+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'8","last_scrub_stamp":"2024-02-13T00:41:37.452882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:37.452882+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":439,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.439904+0000","last_change":"2024-02-13T00:46:01.449886+0000","last_active":"2024-02-13T00:47:25.439904+0000","last_peered":"2024-02-13T00:47:25.439904+0000","last_clean":"2024-02-13T00:47:25.439904+0000","last_became_active":"2024-02-13T00:46:01.447236+0000","last_became_peered":"2024-02-13T00:46:01.447236+0000","last_unstale":"2024-02-13T00:47:25.439904+0000","last_undegraded":"2024-02-13T00:47:25.439904+0000","last_fullsized":"2024-02-13T00:47:25.439904+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:52.391906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.391906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":110,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.439092+0000","last_change":"2024-02-13T00:46:01.456618+0000","last_active":"2024-02-13T00:47:25.439092+0000","last_peered":"2024-02-13T00:47:25.439092+0000","last_clean":"2024-02-13T00:47:25.439092+0000","last_became_active":"2024-02-13T00:46:01.455989+0000","last_became_peered":"2024-02-13T00:46:01.455989+0000","last_unstale":"2024-02-13T00:47:25.439092+0000","last_undegraded":"2024-02-13T00:47:25.439092+0000","last_fullsized":"2024-02-13T00:47:25.439092+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:16.499780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:16.499780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'16","reported_seq":415,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437063+0000","last_change":"2024-02-13T00:46:10.157189+0000","last_active":"2024-02-13T00:47:25.437063+0000","last_peered":"2024-02-13T00:47:25.437063+0000","last_clean":"2024-02-13T00:47:25.437063+0000","last_became_active":"2024-02-13T00:46:01.460321+0000","last_became_peered":"2024-02-13T00:46:01.460321+0000","last_unstale":"2024-02-13T00:47:25.437063+0000","last_undegraded":"2024-02-13T00:47:25.437063+0000","last_fullsized":"2024-02-13T00:47:25.437063+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:36.477034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:36.477034+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":454,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438344+0000","last_change":"2024-02-13T00:46:01.432179+0000","last_active":"2024-02-13T00:47:25.438344+0000","last_peered":"2024-02-13T00:47:25.438344+0000","last_clean":"2024-02-13T00:47:25.438344+0000","last_became_active":"2024-02-13T00:46:01.431926+0000","last_became_peered":"2024-02-13T00:46:01.431926+0000","last_unstale":"2024-02-13T00:47:25.438344+0000","last_undegraded":"2024-02-13T00:47:25.438344+0000","last_fullsized":"2024-02-13T00:47:25.438344+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:40.335892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:40.335892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"363'14","reported_seq":258,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-13T00:47:33.238942+0000","last_change":"2024-02-13T00:47:07.942748+0000","last_active":"2024-02-13T00:47:33.238942+0000","last_peered":"2024-02-13T00:47:33.238942+0000","last_clean":"2024-02-13T00:47:33.238942+0000","last_became_active":"2024-02-13T00:45:15.423066+0000","last_became_peered":"2024-02-13T00:45:15.423066+0000","last_unstale":"2024-02-13T00:47:33.238942+0000","last_undegraded":"2024-02-13T00:47:33.238942+0000","last_fullsized":"2024-02-13T00:47:33.238942+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:47:07.942668+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:47:07.942668+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":78,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.439913+0000","last_change":"2024-02-13T00:47:02.789298+0000","last_active":"2024-02-13T00:47:25.439913+0000","last_peered":"2024-02-13T00:47:25.439913+0000","last_clean":"2024-02-13T00:47:25.439913+0000","last_became_active":"2024-02-13T00:46:01.446960+0000","last_became_peered":"2024-02-13T00:46:01.446960+0000","last_unstale":"2024-02-13T00:47:25.439913+0000","last_undegraded":"2024-02-13T00:47:25.439913+0000","last_fullsized":"2024-02-13T00:47:25.439913+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:02.789189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.789189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":316,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435918+0000","last_change":"2024-02-13T00:46:01.450308+0000","last_active":"2024-02-13T00:47:25.435918+0000","last_peered":"2024-02-13T00:47:25.435918+0000","last_clean":"2024-02-13T00:47:25.435918+0000","last_became_active":"2024-02-13T00:46:01.449694+0000","last_became_peered":"2024-02-13T00:46:01.449694+0000","last_unstale":"2024-02-13T00:47:25.435918+0000","last_undegraded":"2024-02-13T00:47:25.435918+0000","last_fullsized":"2024-02-13T00:47:25.435918+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:27.073034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:27.073034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"83'13","reported_seq":499,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.156353+0000","last_change":"2024-02-13T00:44:54.926465+0000","last_active":"2024-02-13T00:47:26.156353+0000","last_peered":"2024-02-13T00:47:26.156353+0000","last_clean":"2024-02-13T00:47:26.156353+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:47:26.156353+0000","last_undegraded":"2024-02-13T00:47:26.156353+0000","last_fullsized":"2024-02-13T00:47:26.156353+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:43:40.244930+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:43:40.244930+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437051+0000","last_change":"2024-02-13T00:46:01.463310+0000","last_active":"2024-02-13T00:47:25.437051+0000","last_peered":"2024-02-13T00:47:25.437051+0000","last_clean":"2024-02-13T00:47:25.437051+0000","last_became_active":"2024-02-13T00:46:01.462898+0000","last_became_peered":"2024-02-13T00:46:01.462898+0000","last_unstale":"2024-02-13T00:47:25.437051+0000","last_undegraded":"2024-02-13T00:47:25.437051+0000","last_fullsized":"2024-02-13T00:47:25.437051+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.095888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":223,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438103+0000","last_change":"2024-02-13T00:46:32.638418+0000","last_active":"2024-02-13T00:47:25.438103+0000","last_peered":"2024-02-13T00:47:25.438103+0000","last_clean":"2024-02-13T00:47:25.438103+0000","last_became_active":"2024-02-13T00:46:01.443279+0000","last_became_peered":"2024-02-13T00:46:01.443279+0000","last_unstale":"2024-02-13T00:47:25.438103+0000","last_undegraded":"2024-02-13T00:47:25.438103+0000","last_fullsized":"2024-02-13T00:47:25.438103+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.638349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":421,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437518+0000","last_change":"2024-02-13T00:46:01.446071+0000","last_active":"2024-02-13T00:47:25.437518+0000","last_peered":"2024-02-13T00:47:25.437518+0000","last_clean":"2024-02-13T00:47:25.437518+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:47:25.437518+0000","last_undegraded":"2024-02-13T00:47:25.437518+0000","last_fullsized":"2024-02-13T00:47:25.437518+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.413326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":156,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436955+0000","last_change":"2024-02-13T00:45:40.233264+0000","last_active":"2024-02-13T00:47:25.436955+0000","last_peered":"2024-02-13T00:47:25.436955+0000","last_clean":"2024-02-13T00:47:25.436955+0000","last_became_active":"2024-02-13T00:45:15.447051+0000","last_became_peered":"2024-02-13T00:45:15.447051+0000","last_unstale":"2024-02-13T00:47:25.436955+0000","last_undegraded":"2024-02-13T00:47:25.436955+0000","last_fullsized":"2024-02-13T00:47:25.436955+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:40.233185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:40.233185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"80'16","reported_seq":175,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436762+0000","last_change":"2024-02-13T00:44:52.751765+0000","last_active":"2024-02-13T00:47:25.436762+0000","last_peered":"2024-02-13T00:47:25.436762+0000","last_clean":"2024-02-13T00:47:25.436762+0000","last_became_active":"2024-02-13T00:44:50.435203+0000","last_became_peered":"2024-02-13T00:44:50.435203+0000","last_unstale":"2024-02-13T00:47:25.436762+0000","last_undegraded":"2024-02-13T00:47:25.436762+0000","last_fullsized":"2024-02-13T00:47:25.436762+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:42:51.361489+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:42:51.361489+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":433,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157750+0000","last_change":"2024-02-13T00:46:01.457220+0000","last_active":"2024-02-13T00:47:26.157750+0000","last_peered":"2024-02-13T00:47:26.157750+0000","last_clean":"2024-02-13T00:47:26.157750+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:47:26.157750+0000","last_undegraded":"2024-02-13T00:47:26.157750+0000","last_fullsized":"2024-02-13T00:47:26.157750+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.097481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":447,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.156439+0000","last_change":"2024-02-13T00:46:01.453391+0000","last_active":"2024-02-13T00:47:26.156439+0000","last_peered":"2024-02-13T00:47:26.156439+0000","last_clean":"2024-02-13T00:47:26.156439+0000","last_became_active":"2024-02-13T00:46:01.452730+0000","last_became_peered":"2024-02-13T00:46:01.452730+0000","last_unstale":"2024-02-13T00:47:26.156439+0000","last_undegraded":"2024-02-13T00:47:26.156439+0000","last_fullsized":"2024-02-13T00:47:26.156439+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:26.218807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:44:26.218807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":227,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438303+0000","last_change":"2024-02-13T00:46:01.454624+0000","last_active":"2024-02-13T00:47:25.438303+0000","last_peered":"2024-02-13T00:47:25.438303+0000","last_clean":"2024-02-13T00:47:25.438303+0000","last_became_active":"2024-02-13T00:46:01.454287+0000","last_became_peered":"2024-02-13T00:46:01.454287+0000","last_unstale":"2024-02-13T00:47:25.438303+0000","last_undegraded":"2024-02-13T00:47:25.438303+0000","last_fullsized":"2024-02-13T00:47:25.438303+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:50.828264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:50.828264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435554+0000","last_change":"2024-02-13T00:46:01.449548+0000","last_active":"2024-02-13T00:47:25.435554+0000","last_peered":"2024-02-13T00:47:25.435554+0000","last_clean":"2024-02-13T00:47:25.435554+0000","last_became_active":"2024-02-13T00:46:01.448671+0000","last_became_peered":"2024-02-13T00:46:01.448671+0000","last_unstale":"2024-02-13T00:47:25.435554+0000","last_undegraded":"2024-02-13T00:47:25.435554+0000","last_fullsized":"2024-02-13T00:47:25.435554+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.820327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.820327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"64'4","reported_seq":254,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155812+0000","last_change":"2024-02-13T00:44:40.371566+0000","last_active":"2024-02-13T00:47:26.155812+0000","last_peered":"2024-02-13T00:47:26.155812+0000","last_clean":"2024-02-13T00:47:26.155812+0000","last_became_active":"2024-02-13T00:44:39.501040+0000","last_became_peered":"2024-02-13T00:44:39.501040+0000","last_unstale":"2024-02-13T00:47:26.155812+0000","last_undegraded":"2024-02-13T00:47:26.155812+0000","last_fullsized":"2024-02-13T00:47:26.155812+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:44:32.180113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:44:32.180113+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":386,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437923+0000","last_change":"2024-02-13T00:46:01.464539+0000","last_active":"2024-02-13T00:47:25.437923+0000","last_peered":"2024-02-13T00:47:25.437923+0000","last_clean":"2024-02-13T00:47:25.437923+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:47:25.437923+0000","last_undegraded":"2024-02-13T00:47:25.437923+0000","last_fullsized":"2024-02-13T00:47:25.437923+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.499576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":151,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435626+0000","last_change":"2024-02-13T00:45:15.436871+0000","last_active":"2024-02-13T00:47:25.435626+0000","last_peered":"2024-02-13T00:47:25.435626+0000","last_clean":"2024-02-13T00:47:25.435626+0000","last_became_active":"2024-02-13T00:45:15.435820+0000","last_became_peered":"2024-02-13T00:45:15.435820+0000","last_unstale":"2024-02-13T00:47:25.435626+0000","last_undegraded":"2024-02-13T00:47:25.435626+0000","last_fullsized":"2024-02-13T00:47:25.435626+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.646321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.646321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":142,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155437+0000","last_change":"2024-02-13T00:45:15.445131+0000","last_active":"2024-02-13T00:47:26.155437+0000","last_peered":"2024-02-13T00:47:26.155437+0000","last_clean":"2024-02-13T00:47:26.155437+0000","last_became_active":"2024-02-13T00:45:15.444818+0000","last_became_peered":"2024-02-13T00:45:15.444818+0000","last_unstale":"2024-02-13T00:47:26.155437+0000","last_undegraded":"2024-02-13T00:47:26.155437+0000","last_fullsized":"2024-02-13T00:47:26.155437+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.479450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":307,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435708+0000","last_change":"2024-02-13T00:46:01.439915+0000","last_active":"2024-02-13T00:47:25.435708+0000","last_peered":"2024-02-13T00:47:25.435708+0000","last_clean":"2024-02-13T00:47:25.435708+0000","last_became_active":"2024-02-13T00:46:01.438745+0000","last_became_peered":"2024-02-13T00:46:01.438745+0000","last_unstale":"2024-02-13T00:47:25.435708+0000","last_undegraded":"2024-02-13T00:47:25.435708+0000","last_fullsized":"2024-02-13T00:47:25.435708+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.104906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.104906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'17","reported_seq":202,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435721+0000","last_change":"2024-02-13T00:46:07.632448+0000","last_active":"2024-02-13T00:47:25.435721+0000","last_peered":"2024-02-13T00:47:25.435721+0000","last_clean":"2024-02-13T00:47:25.435721+0000","last_became_active":"2024-02-13T00:46:01.450079+0000","last_became_peered":"2024-02-13T00:46:01.450079+0000","last_unstale":"2024-02-13T00:47:25.435721+0000","last_undegraded":"2024-02-13T00:47:25.435721+0000","last_fullsized":"2024-02-13T00:47:25.435721+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:41:50.675154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:50.675154+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":165,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438186+0000","last_change":"2024-02-13T00:46:59.388809+0000","last_active":"2024-02-13T00:47:25.438186+0000","last_peered":"2024-02-13T00:47:25.438186+0000","last_clean":"2024-02-13T00:47:25.438186+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:47:25.438186+0000","last_undegraded":"2024-02-13T00:47:25.438186+0000","last_fullsized":"2024-02-13T00:47:25.438186+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:59.388745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:59.388745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":151,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435266+0000","last_change":"2024-02-13T00:45:15.439752+0000","last_active":"2024-02-13T00:47:25.435266+0000","last_peered":"2024-02-13T00:47:25.435266+0000","last_clean":"2024-02-13T00:47:25.435266+0000","last_became_active":"2024-02-13T00:45:15.437943+0000","last_became_peered":"2024-02-13T00:45:15.437943+0000","last_unstale":"2024-02-13T00:47:25.435266+0000","last_undegraded":"2024-02-13T00:47:25.435266+0000","last_fullsized":"2024-02-13T00:47:25.435266+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.284740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.284740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":359,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438022+0000","last_change":"2024-02-13T00:44:58.969806+0000","last_active":"2024-02-13T00:47:25.438022+0000","last_peered":"2024-02-13T00:47:25.438022+0000","last_clean":"2024-02-13T00:47:25.438022+0000","last_became_active":"2024-02-13T00:44:50.439512+0000","last_became_peered":"2024-02-13T00:44:50.439512+0000","last_unstale":"2024-02-13T00:47:25.438022+0000","last_undegraded":"2024-02-13T00:47:25.438022+0000","last_fullsized":"2024-02-13T00:47:25.438022+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:58.969692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:44:58.969692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":274,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437962+0000","last_change":"2024-02-13T00:45:15.431130+0000","last_active":"2024-02-13T00:47:25.437962+0000","last_peered":"2024-02-13T00:47:25.437962+0000","last_clean":"2024-02-13T00:47:25.437962+0000","last_became_active":"2024-02-13T00:45:15.430749+0000","last_became_peered":"2024-02-13T00:45:15.430749+0000","last_unstale":"2024-02-13T00:47:25.437962+0000","last_undegraded":"2024-02-13T00:47:25.437962+0000","last_fullsized":"2024-02-13T00:47:25.437962+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.605111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.605111+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"79'22","reported_seq":528,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440338+0000","last_change":"2024-02-13T00:46:02.034931+0000","last_active":"2024-02-13T00:47:25.440338+0000","last_peered":"2024-02-13T00:47:25.440338+0000","last_clean":"2024-02-13T00:47:25.440338+0000","last_became_active":"2024-02-13T00:46:01.443721+0000","last_became_peered":"2024-02-13T00:46:01.443721+0000","last_unstale":"2024-02-13T00:47:25.440338+0000","last_undegraded":"2024-02-13T00:47:25.440338+0000","last_fullsized":"2024-02-13T00:47:25.440338+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:43:07.393197+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.393197+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":393,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.434889+0000","last_change":"2024-02-13T00:46:32.298033+0000","last_active":"2024-02-13T00:47:25.434889+0000","last_peered":"2024-02-13T00:47:25.434889+0000","last_clean":"2024-02-13T00:47:25.434889+0000","last_became_active":"2024-02-13T00:46:01.442231+0000","last_became_peered":"2024-02-13T00:46:01.442231+0000","last_unstale":"2024-02-13T00:47:25.434889+0000","last_undegraded":"2024-02-13T00:47:25.434889+0000","last_fullsized":"2024-02-13T00:47:25.434889+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.297918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.297918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"363'14","reported_seq":197,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-02-13T00:47:32.591261+0000","last_change":"2024-02-13T00:45:18.328879+0000","last_active":"2024-02-13T00:47:32.591261+0000","last_peered":"2024-02-13T00:47:32.591261+0000","last_clean":"2024-02-13T00:47:32.591261+0000","last_became_active":"2024-02-13T00:45:15.455825+0000","last_became_peered":"2024-02-13T00:45:15.455825+0000","last_unstale":"2024-02-13T00:47:32.591261+0000","last_undegraded":"2024-02-13T00:47:32.591261+0000","last_fullsized":"2024-02-13T00:47:32.591261+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'10","last_scrub_stamp":"2024-02-13T00:43:05.604461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.604461+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":252,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436341+0000","last_change":"2024-02-13T00:46:03.966962+0000","last_active":"2024-02-13T00:47:25.436341+0000","last_peered":"2024-02-13T00:47:25.436341+0000","last_clean":"2024-02-13T00:47:25.436341+0000","last_became_active":"2024-02-13T00:46:01.439984+0000","last_became_peered":"2024-02-13T00:46:01.439984+0000","last_unstale":"2024-02-13T00:47:25.436341+0000","last_undegraded":"2024-02-13T00:47:25.436341+0000","last_fullsized":"2024-02-13T00:47:25.436341+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:42:55.481128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:55.481128+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":391,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.158715+0000","last_change":"2024-02-13T00:45:38.084426+0000","last_active":"2024-02-13T00:47:26.158715+0000","last_peered":"2024-02-13T00:47:26.158715+0000","last_clean":"2024-02-13T00:47:26.158715+0000","last_became_active":"2024-02-13T00:45:15.435550+0000","last_became_peered":"2024-02-13T00:45:15.435550+0000","last_unstale":"2024-02-13T00:47:26.158715+0000","last_undegraded":"2024-02-13T00:47:26.158715+0000","last_fullsized":"2024-02-13T00:47:26.158715+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.084358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.084358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":179,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438387+0000","last_change":"2024-02-13T00:46:10.679144+0000","last_active":"2024-02-13T00:47:25.438387+0000","last_peered":"2024-02-13T00:47:25.438387+0000","last_clean":"2024-02-13T00:47:25.438387+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:47:25.438387+0000","last_undegraded":"2024-02-13T00:47:25.438387+0000","last_fullsized":"2024-02-13T00:47:25.438387+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:42:59.604002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:59.604002+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":165,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435824+0000","last_change":"2024-02-13T00:46:56.339969+0000","last_active":"2024-02-13T00:47:25.435824+0000","last_peered":"2024-02-13T00:47:25.435824+0000","last_clean":"2024-02-13T00:47:25.435824+0000","last_became_active":"2024-02-13T00:45:15.436239+0000","last_became_peered":"2024-02-13T00:45:15.436239+0000","last_unstale":"2024-02-13T00:47:25.435824+0000","last_undegraded":"2024-02-13T00:47:25.435824+0000","last_fullsized":"2024-02-13T00:47:25.435824+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:56.339885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.339885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":434,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440344+0000","last_change":"2024-02-13T00:45:15.423191+0000","last_active":"2024-02-13T00:47:25.440344+0000","last_peered":"2024-02-13T00:47:25.440344+0000","last_clean":"2024-02-13T00:47:25.440344+0000","last_became_active":"2024-02-13T00:45:15.422870+0000","last_became_peered":"2024-02-13T00:45:15.422870+0000","last_unstale":"2024-02-13T00:47:25.440344+0000","last_undegraded":"2024-02-13T00:47:25.440344+0000","last_fullsized":"2024-02-13T00:47:25.440344+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.596334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.596334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":427,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157670+0000","last_change":"2024-02-13T00:46:01.457036+0000","last_active":"2024-02-13T00:47:26.157670+0000","last_peered":"2024-02-13T00:47:26.157670+0000","last_clean":"2024-02-13T00:47:26.157670+0000","last_became_active":"2024-02-13T00:46:01.453269+0000","last_became_peered":"2024-02-13T00:46:01.453269+0000","last_unstale":"2024-02-13T00:47:26.157670+0000","last_undegraded":"2024-02-13T00:47:26.157670+0000","last_fullsized":"2024-02-13T00:47:26.157670+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.245237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.245237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":156,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437853+0000","last_change":"2024-02-13T00:45:37.280947+0000","last_active":"2024-02-13T00:47:25.437853+0000","last_peered":"2024-02-13T00:47:25.437853+0000","last_clean":"2024-02-13T00:47:25.437853+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:47:25.437853+0000","last_undegraded":"2024-02-13T00:47:25.437853+0000","last_fullsized":"2024-02-13T00:47:25.437853+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.280865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.280865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"64'15","reported_seq":188,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157623+0000","last_change":"2024-02-13T00:44:55.939050+0000","last_active":"2024-02-13T00:47:26.157623+0000","last_peered":"2024-02-13T00:47:26.157623+0000","last_clean":"2024-02-13T00:47:26.157623+0000","last_became_active":"2024-02-13T00:44:50.442894+0000","last_became_peered":"2024-02-13T00:44:50.442894+0000","last_unstale":"2024-02-13T00:47:26.157623+0000","last_undegraded":"2024-02-13T00:47:26.157623+0000","last_fullsized":"2024-02-13T00:47:26.157623+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:43:30.578602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.578602+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":151,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436028+0000","last_change":"2024-02-13T00:45:15.446020+0000","last_active":"2024-02-13T00:47:25.436028+0000","last_peered":"2024-02-13T00:47:25.436028+0000","last_clean":"2024-02-13T00:47:25.436028+0000","last_became_active":"2024-02-13T00:45:15.445403+0000","last_became_peered":"2024-02-13T00:45:15.445403+0000","last_unstale":"2024-02-13T00:47:25.436028+0000","last_undegraded":"2024-02-13T00:47:25.436028+0000","last_fullsized":"2024-02-13T00:47:25.436028+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:06.651077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:06.651077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":415,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.158003+0000","last_change":"2024-02-13T00:41:56.204734+0000","last_active":"2024-02-13T00:47:26.158003+0000","last_peered":"2024-02-13T00:47:26.158003+0000","last_clean":"2024-02-13T00:47:26.158003+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:47:26.158003+0000","last_undegraded":"2024-02-13T00:47:26.158003+0000","last_fullsized":"2024-02-13T00:47:26.158003+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:56.204671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.204671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":151,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438216+0000","last_change":"2024-02-13T00:46:01.451373+0000","last_active":"2024-02-13T00:47:25.438216+0000","last_peered":"2024-02-13T00:47:25.438216+0000","last_clean":"2024-02-13T00:47:25.438216+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:47:25.438216+0000","last_undegraded":"2024-02-13T00:47:25.438216+0000","last_fullsized":"2024-02-13T00:47:25.438216+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.410777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.410777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":65,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437259+0000","last_change":"2024-02-13T00:46:01.451610+0000","last_active":"2024-02-13T00:47:25.437259+0000","last_peered":"2024-02-13T00:47:25.437259+0000","last_clean":"2024-02-13T00:47:25.437259+0000","last_became_active":"2024-02-13T00:46:01.449873+0000","last_became_peered":"2024-02-13T00:46:01.449873+0000","last_unstale":"2024-02-13T00:47:25.437259+0000","last_undegraded":"2024-02-13T00:47:25.437259+0000","last_fullsized":"2024-02-13T00:47:25.437259+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.554065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.554065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"65'18","reported_seq":303,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.157978+0000","last_change":"2024-02-13T00:46:56.973071+0000","last_active":"2024-02-13T00:47:26.157978+0000","last_peered":"2024-02-13T00:47:26.157978+0000","last_clean":"2024-02-13T00:47:26.157978+0000","last_became_active":"2024-02-13T00:44:39.504670+0000","last_became_peered":"2024-02-13T00:44:39.504670+0000","last_unstale":"2024-02-13T00:47:26.157978+0000","last_undegraded":"2024-02-13T00:47:26.157978+0000","last_fullsized":"2024-02-13T00:47:26.157978+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:46:56.973000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.973000+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436914+0000","last_change":"2024-02-13T00:46:01.448134+0000","last_active":"2024-02-13T00:47:25.436914+0000","last_peered":"2024-02-13T00:47:25.436914+0000","last_clean":"2024-02-13T00:47:25.436914+0000","last_became_active":"2024-02-13T00:46:01.446635+0000","last_became_peered":"2024-02-13T00:46:01.446635+0000","last_unstale":"2024-02-13T00:47:25.436914+0000","last_undegraded":"2024-02-13T00:47:25.436914+0000","last_fullsized":"2024-02-13T00:47:25.436914+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:28.101254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:28.101254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":142,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.156474+0000","last_change":"2024-02-13T00:45:15.435977+0000","last_active":"2024-02-13T00:47:26.156474+0000","last_peered":"2024-02-13T00:47:26.156474+0000","last_clean":"2024-02-13T00:47:26.156474+0000","last_became_active":"2024-02-13T00:45:15.435206+0000","last_became_peered":"2024-02-13T00:45:15.435206+0000","last_unstale":"2024-02-13T00:47:26.156474+0000","last_undegraded":"2024-02-13T00:47:26.156474+0000","last_fullsized":"2024-02-13T00:47:26.156474+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:57.802417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.802417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":156,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438142+0000","last_change":"2024-02-13T00:46:01.461188+0000","last_active":"2024-02-13T00:47:25.438142+0000","last_peered":"2024-02-13T00:47:25.438142+0000","last_clean":"2024-02-13T00:47:25.438142+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:47:25.438142+0000","last_undegraded":"2024-02-13T00:47:25.438142+0000","last_fullsized":"2024-02-13T00:47:25.438142+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.238152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.238152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":400,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438430+0000","last_change":"2024-02-13T00:46:01.443734+0000","last_active":"2024-02-13T00:47:25.438430+0000","last_peered":"2024-02-13T00:47:25.438430+0000","last_clean":"2024-02-13T00:47:25.438430+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:47:25.438430+0000","last_undegraded":"2024-02-13T00:47:25.438430+0000","last_fullsized":"2024-02-13T00:47:25.438430+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.621170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.621170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"65'21","reported_seq":296,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438355+0000","last_change":"2024-02-13T00:46:27.706096+0000","last_active":"2024-02-13T00:47:25.438355+0000","last_peered":"2024-02-13T00:47:25.438355+0000","last_clean":"2024-02-13T00:47:25.438355+0000","last_became_active":"2024-02-13T00:44:50.415894+0000","last_became_peered":"2024-02-13T00:44:50.415894+0000","last_unstale":"2024-02-13T00:47:25.438355+0000","last_undegraded":"2024-02-13T00:47:25.438355+0000","last_fullsized":"2024-02-13T00:47:25.438355+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:46:27.706006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:27.706006+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":235,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.436560+0000","last_change":"2024-02-13T00:46:01.452098+0000","last_active":"2024-02-13T00:47:25.436560+0000","last_peered":"2024-02-13T00:47:25.436560+0000","last_clean":"2024-02-13T00:47:25.436560+0000","last_became_active":"2024-02-13T00:46:01.451790+0000","last_became_peered":"2024-02-13T00:46:01.451790+0000","last_unstale":"2024-02-13T00:47:25.436560+0000","last_undegraded":"2024-02-13T00:47:25.436560+0000","last_fullsized":"2024-02-13T00:47:25.436560+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:03.665973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:03.665973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":389,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438504+0000","last_change":"2024-02-13T00:46:01.454932+0000","last_active":"2024-02-13T00:47:25.438504+0000","last_peered":"2024-02-13T00:47:25.438504+0000","last_clean":"2024-02-13T00:47:25.438504+0000","last_became_active":"2024-02-13T00:46:01.454163+0000","last_became_peered":"2024-02-13T00:46:01.454163+0000","last_unstale":"2024-02-13T00:47:25.438504+0000","last_undegraded":"2024-02-13T00:47:25.438504+0000","last_fullsized":"2024-02-13T00:47:25.438504+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.415414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.415414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":421,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438180+0000","last_change":"2024-02-13T00:45:15.420228+0000","last_active":"2024-02-13T00:47:25.438180+0000","last_peered":"2024-02-13T00:47:25.438180+0000","last_clean":"2024-02-13T00:47:25.438180+0000","last_became_active":"2024-02-13T00:45:15.419668+0000","last_became_peered":"2024-02-13T00:45:15.419668+0000","last_unstale":"2024-02-13T00:47:25.438180+0000","last_undegraded":"2024-02-13T00:47:25.438180+0000","last_fullsized":"2024-02-13T00:47:25.438180+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:21.648002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.648002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":334,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438613+0000","last_change":"2024-02-13T00:46:01.457482+0000","last_active":"2024-02-13T00:47:25.438613+0000","last_peered":"2024-02-13T00:47:25.438613+0000","last_clean":"2024-02-13T00:47:25.438613+0000","last_became_active":"2024-02-13T00:46:01.455940+0000","last_became_peered":"2024-02-13T00:46:01.455940+0000","last_unstale":"2024-02-13T00:47:25.438613+0000","last_undegraded":"2024-02-13T00:47:25.438613+0000","last_fullsized":"2024-02-13T00:47:25.438613+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:21.087199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:21.087199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"85'14","reported_seq":317,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.156279+0000","last_change":"2024-02-13T00:46:36.089483+0000","last_active":"2024-02-13T00:47:26.156279+0000","last_peered":"2024-02-13T00:47:26.156279+0000","last_clean":"2024-02-13T00:47:26.156279+0000","last_became_active":"2024-02-13T00:45:15.419364+0000","last_became_peered":"2024-02-13T00:45:15.419364+0000","last_unstale":"2024-02-13T00:47:26.156279+0000","last_undegraded":"2024-02-13T00:47:26.156279+0000","last_fullsized":"2024-02-13T00:47:26.156279+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:46:36.089402+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:46:36.089402+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":160,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438286+0000","last_change":"2024-02-13T00:46:01.452313+0000","last_active":"2024-02-13T00:47:25.438286+0000","last_peered":"2024-02-13T00:47:25.438286+0000","last_clean":"2024-02-13T00:47:25.438286+0000","last_became_active":"2024-02-13T00:46:01.451997+0000","last_became_peered":"2024-02-13T00:46:01.451997+0000","last_unstale":"2024-02-13T00:47:25.438286+0000","last_undegraded":"2024-02-13T00:47:25.438286+0000","last_fullsized":"2024-02-13T00:47:25.438286+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:21.630532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.630532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":260,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155948+0000","last_change":"2024-02-13T00:45:15.416632+0000","last_active":"2024-02-13T00:47:26.155948+0000","last_peered":"2024-02-13T00:47:26.155948+0000","last_clean":"2024-02-13T00:47:26.155948+0000","last_became_active":"2024-02-13T00:45:15.416307+0000","last_became_peered":"2024-02-13T00:45:15.416307+0000","last_unstale":"2024-02-13T00:47:26.155948+0000","last_undegraded":"2024-02-13T00:47:26.155948+0000","last_fullsized":"2024-02-13T00:47:26.155948+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:48.629024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.629024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":372,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155611+0000","last_change":"2024-02-13T00:46:01.445422+0000","last_active":"2024-02-13T00:47:26.155611+0000","last_peered":"2024-02-13T00:47:26.155611+0000","last_clean":"2024-02-13T00:47:26.155611+0000","last_became_active":"2024-02-13T00:46:01.445141+0000","last_became_peered":"2024-02-13T00:46:01.445141+0000","last_unstale":"2024-02-13T00:47:26.155611+0000","last_undegraded":"2024-02-13T00:47:26.155611+0000","last_fullsized":"2024-02-13T00:47:26.155611+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:52.447698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.447698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":147,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435560+0000","last_change":"2024-02-13T00:45:30.186208+0000","last_active":"2024-02-13T00:47:25.435560+0000","last_peered":"2024-02-13T00:47:25.435560+0000","last_clean":"2024-02-13T00:47:25.435560+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:47:25.435560+0000","last_undegraded":"2024-02-13T00:47:25.435560+0000","last_fullsized":"2024-02-13T00:47:25.435560+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:30.186129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:30.186129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"90'20","reported_seq":196,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435652+0000","last_change":"2024-02-13T00:46:06.251713+0000","last_active":"2024-02-13T00:47:25.435652+0000","last_peered":"2024-02-13T00:47:25.435652+0000","last_clean":"2024-02-13T00:47:25.435652+0000","last_became_active":"2024-02-13T00:46:01.442464+0000","last_became_peered":"2024-02-13T00:46:01.442464+0000","last_unstale":"2024-02-13T00:47:25.435652+0000","last_undegraded":"2024-02-13T00:47:25.435652+0000","last_fullsized":"2024-02-13T00:47:25.435652+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:43:44.241358+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:43:44.241358+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":236,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440171+0000","last_change":"2024-02-13T00:46:01.450078+0000","last_active":"2024-02-13T00:47:25.440171+0000","last_peered":"2024-02-13T00:47:25.440171+0000","last_clean":"2024-02-13T00:47:25.440171+0000","last_became_active":"2024-02-13T00:46:01.447654+0000","last_became_peered":"2024-02-13T00:46:01.447654+0000","last_unstale":"2024-02-13T00:47:25.440171+0000","last_undegraded":"2024-02-13T00:47:25.440171+0000","last_fullsized":"2024-02-13T00:47:25.440171+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:18.464402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:42:18.464402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":438,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155476+0000","last_change":"2024-02-13T00:46:55.979707+0000","last_active":"2024-02-13T00:47:26.155476+0000","last_peered":"2024-02-13T00:47:26.155476+0000","last_clean":"2024-02-13T00:47:26.155476+0000","last_became_active":"2024-02-13T00:46:01.444842+0000","last_became_peered":"2024-02-13T00:46:01.444842+0000","last_unstale":"2024-02-13T00:47:26.155476+0000","last_undegraded":"2024-02-13T00:47:26.155476+0000","last_fullsized":"2024-02-13T00:47:26.155476+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:55.979635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:46:55.979635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":71,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155851+0000","last_change":"2024-02-13T00:46:01.449363+0000","last_active":"2024-02-13T00:47:26.155851+0000","last_peered":"2024-02-13T00:47:26.155851+0000","last_clean":"2024-02-13T00:47:26.155851+0000","last_became_active":"2024-02-13T00:46:01.448971+0000","last_became_peered":"2024-02-13T00:46:01.448971+0000","last_unstale":"2024-02-13T00:47:26.155851+0000","last_undegraded":"2024-02-13T00:47:26.155851+0000","last_fullsized":"2024-02-13T00:47:26.155851+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:47.487860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:47.487860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":403,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:26.155773+0000","last_change":"2024-02-13T00:45:32.233265+0000","last_active":"2024-02-13T00:47:26.155773+0000","last_peered":"2024-02-13T00:47:26.155773+0000","last_clean":"2024-02-13T00:47:26.155773+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:47:26.155773+0000","last_undegraded":"2024-02-13T00:47:26.155773+0000","last_fullsized":"2024-02-13T00:47:26.155773+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:45:32.233197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.233197+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"86'50","reported_seq":317,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.438463+0000","last_change":"2024-02-13T00:46:05.700233+0000","last_active":"2024-02-13T00:47:25.438463+0000","last_peered":"2024-02-13T00:47:25.438463+0000","last_clean":"2024-02-13T00:47:25.438463+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:47:25.438463+0000","last_undegraded":"2024-02-13T00:47:25.438463+0000","last_fullsized":"2024-02-13T00:47:25.438463+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:42:17.454232+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.454232+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":427,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.440202+0000","last_change":"2024-02-13T00:45:15.434575+0000","last_active":"2024-02-13T00:47:25.440202+0000","last_peered":"2024-02-13T00:47:25.440202+0000","last_clean":"2024-02-13T00:47:25.440202+0000","last_became_active":"2024-02-13T00:45:15.434224+0000","last_became_peered":"2024-02-13T00:45:15.434224+0000","last_unstale":"2024-02-13T00:47:25.440202+0000","last_undegraded":"2024-02-13T00:47:25.440202+0000","last_fullsized":"2024-02-13T00:47:25.440202+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:43:03.464398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":142,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.435367+0000","last_change":"2024-02-13T00:44:39.510101+0000","last_active":"2024-02-13T00:47:25.435367+0000","last_peered":"2024-02-13T00:47:25.435367+0000","last_clean":"2024-02-13T00:47:25.435367+0000","last_became_active":"2024-02-13T00:44:39.508197+0000","last_became_peered":"2024-02-13T00:44:39.508197+0000","last_unstale":"2024-02-13T00:47:25.435367+0000","last_undegraded":"2024-02-13T00:47:25.435367+0000","last_fullsized":"2024-02-13T00:47:25.435367+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:46.418859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:41:46.418859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":151,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-02-13T00:47:25.437193+0000","last_change":"2024-02-13T00:45:15.435955+0000","last_active":"2024-02-13T00:47:25.437193+0000","last_peered":"2024-02-13T00:47:25.437193+0000","last_clean":"2024-02-13T00:47:25.437193+0000","last_became_active":"2024-02-13T00:45:15.435259+0000","last_became_peered":"2024-02-13T00:45:15.435259+0000","last_unstale":"2024-02-13T00:47:25.437193+0000","last_undegraded":"2024-02-13T00:47:25.437193+0000","last_fullsized":"2024-02-13T00:47:25.437193+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.953989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.953989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":217,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":4001538,"num_objects":165,"num_object_clones":5,"num_object_copies":495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":103,"num_read_kb":7,"num_write":614,"num_write_kb":4513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1266,"num_bytes_recovered":4004226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":12,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14151680,"data_stored":12005061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1928,"ondisk_log_size":1928,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1063,"num_read_kb":788,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":9606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":11656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":581,"num_read_kb":576,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":227633266809,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":45,"seq":193273528447,"num_pgs":182,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304712,"kb_used_data":7484,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93440440,"statfs":{"total":95995035648,"available":95683010560,"internally_reserved":0,"allocated":7663616,"data_stored":4221091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47899999999999998,"15min":0.55400000000000005},"min":{"1min":0.28699999999999998,"5min":0.25600000000000001,"15min":0.19700000000000001},"max":{"1min":0.69799999999999995,"5min":0.77500000000000002,"15min":8.1340000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.52700000000000002,"15min":0.60599999999999998},"min":{"1min":0.312,"5min":0.23100000000000001,"15min":0.221},"max":{"1min":0.77900000000000003,"5min":4.2910000000000004,"15min":11.367000000000001},"last":0.34599999999999997}]},{"osd":1,"last update":"Tue Feb 13 00:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51600000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.68400000000000005,"5min":4.5720000000000001,"15min":54.002000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.68100000000000005},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.69399999999999995,"5min":5.4989999999999997,"15min":12.403},"last":0.501}]},{"osd":2,"last update":"Tue Feb 13 00:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.70599999999999996},"min":{"1min":0.29399999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.72399999999999998,"5min":2.8399999999999999,"15min":12.477},"last":0.437},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.50800000000000001,"15min":0.67500000000000004},"min":{"1min":0.33200000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.66200000000000003,"5min":2.1659999999999999,"15min":11.702},"last":0.46700000000000003}]},{"osd":3,"last update":"Tue Feb 13 00:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.61399999999999999,"15min":0.66600000000000004},"min":{"1min":0.33000000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71099999999999997,"5min":5.0990000000000002,"15min":8.0519999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.55300000000000005,"15min":0.77300000000000002},"min":{"1min":0.379,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":0.67100000000000004,"5min":2.8719999999999999,"15min":19.081},"last":0.54000000000000004}]},{"osd":4,"last update":"Tue Feb 13 00:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.42099999999999999,"15min":0.41099999999999998},"min":{"1min":0.26800000000000002,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":0.67400000000000004,"5min":0.71899999999999997,"15min":1.0649999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.435,"15min":0.42999999999999999},"min":{"1min":0.22800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.623,"5min":0.70799999999999996,"15min":0.83299999999999996},"last":0.55900000000000005}]},{"osd":5,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.434,"15min":0.41499999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.61199999999999999,"5min":0.66500000000000004,"15min":0.94399999999999995},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.48199999999999998,"15min":0.46999999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.86299999999999999},"last":0.316}]},{"osd":7,"last update":"Tue Feb 13 00:46:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46899999999999997,"15min":0.49199999999999999},"min":{"1min":0.29499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":1.173},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51000000000000001,"15min":0.54000000000000004},"min":{"1min":0.32900000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":1.262},"last":0.52800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575000,"num_pgs":191,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321000,"kb_used_data":11484,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93424152,"statfs":{"total":95995035648,"available":95666331648,"internally_reserved":0,"allocated":11759616,"data_stored":8221052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.38200000000000001,"15min":0.94699999999999995},"min":{"1min":0.24099999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.52200000000000002,"5min":5.7199999999999998,"15min":61.856000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.39500000000000002,"15min":0.66500000000000004},"min":{"1min":0.221,"5min":0.19400000000000001,"15min":0.16500000000000001},"max":{"1min":0.57399999999999995,"5min":7.1100000000000003,"15min":61.908999999999999},"last":0.38400000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.56000000000000005,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.42299999999999999,"5min":11.304,"15min":32.161000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.435,"15min":0.83099999999999996},"min":{"1min":0.251,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.56200000000000006,"5min":6.298,"15min":29.693000000000001},"last":0.34799999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.47099999999999997,"15min":1.28},"min":{"1min":0.27500000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.59099999999999997,"5min":10.478,"15min":131.23099999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.42699999999999999,"15min":0.875},"min":{"1min":0.28199999999999997,"5min":0.26800000000000002,"15min":0.246},"max":{"1min":0.52300000000000002,"5min":4.024,"15min":49.055},"last":0.32900000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:46:47 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.64100000000000001,"15min":0.76400000000000001},"min":{"1min":0.33500000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61699999999999999,"5min":7.2089999999999996,"15min":16.547000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.70899999999999996,"15min":0.70599999999999996},"min":{"1min":0.40100000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64500000000000002,"5min":10.050000000000001,"15min":14.731999999999999},"last":0.66000000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.5680000000000001,"15min":0.90200000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.90500000000000003,"5min":26.561,"15min":26.561},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.137,"15min":0.76900000000000002},"min":{"1min":0.377,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.163,"5min":16.494,"15min":16.494},"last":0.47499999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53500000000000003,"15min":0.72099999999999997},"min":{"1min":0.35599999999999998,"5min":0.29699999999999999,"15min":0.29099999999999998},"max":{"1min":0.74199999999999999,"5min":0.92700000000000005,"15min":14.772},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.65100000000000002,"15min":0.93300000000000005},"min":{"1min":0.38100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":10.565,"15min":26.600000000000001},"last":0.498}]},{"osd":7,"last update":"Tue Feb 13 00:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.626,"15min":0.72799999999999998},"min":{"1min":0.437,"5min":0.313,"15min":0.313},"max":{"1min":0.66800000000000004,"5min":7.048,"15min":14.794},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.67900000000000005,"15min":0.93799999999999994},"min":{"1min":0.42399999999999999,"5min":0.36599999999999999,"15min":0.32200000000000001},"max":{"1min":0.71499999999999997,"5min":10.521000000000001,"15min":26.693999999999999},"last":0.56499999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738524,"num_pgs":174,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320892,"kb_used_data":11380,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93424260,"statfs":{"total":95995035648,"available":95666442240,"internally_reserved":0,"allocated":11653120,"data_stored":8220793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:46:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":1.157,"15min":0.62},"min":{"1min":0.20999999999999999,"5min":0.192,"15min":0.189},"max":{"1min":1.7370000000000001,"5min":30.184000000000001,"15min":30.184000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.55000000000000004,"15min":0.44700000000000001},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.17599999999999999},"max":{"1min":2.839,"5min":5.6959999999999997,"15min":5.6959999999999997},"last":0.32700000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:46:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.54400000000000004,"15min":0.501},"min":{"1min":0.245,"5min":0.216,"15min":0.20999999999999999},"max":{"1min":1.3700000000000001,"5min":7.0839999999999996,"15min":7.0839999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.56499999999999995,"15min":0.67200000000000004},"min":{"1min":0.27300000000000002,"5min":0.20899999999999999,"15min":0.185},"max":{"1min":1.6970000000000001,"5min":9.4160000000000004,"15min":24.459},"last":0.36799999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.48799999999999999,"15min":0.49099999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":0.53300000000000003,"5min":4.8710000000000004,"15min":8.718},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.61699999999999999,"15min":0.54900000000000004},"min":{"1min":0.26900000000000002,"5min":0.222,"15min":0.222},"max":{"1min":1.2669999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"last":0.45000000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:46:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.84299999999999997,"15min":0.79400000000000004},"min":{"1min":0.38300000000000001,"5min":0.29599999999999999,"15min":0.24399999999999999},"max":{"1min":1.639,"5min":9.2400000000000002,"15min":24.381},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.82599999999999996,"15min":0.65000000000000002},"min":{"1min":0.34899999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.81299999999999994,"5min":8.6950000000000003,"15min":8.6950000000000003},"last":0.63900000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:46:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.749,"15min":0.63},"min":{"1min":0.36399999999999999,"5min":0.309,"15min":0.309},"max":{"1min":3.331,"5min":7.0990000000000002,"15min":7.0990000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.4330000000000001,"15min":0.85099999999999998},"min":{"1min":0.33900000000000002,"5min":0.307,"15min":0.30599999999999999},"max":{"1min":10.686,"5min":24.427,"15min":24.427},"last":0.48299999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:46:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.81100000000000005,"15min":0.622},"min":{"1min":0.376,"5min":0.32900000000000001,"15min":0.31},"max":{"1min":1.4990000000000001,"5min":8.6129999999999995,"15min":8.6129999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.79200000000000004,"15min":0.64200000000000002},"min":{"1min":0.35099999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.83399999999999996,"5min":14.263,"15min":14.263},"last":0.51700000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:46:51 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.71099999999999997,"15min":0.66200000000000003},"min":{"1min":0.374,"5min":0.317,"15min":0.317},"max":{"1min":0.63700000000000001,"5min":7.7720000000000002,"15min":8.6679999999999993},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.76700000000000002,"15min":0.66800000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86299999999999999,"5min":8.1609999999999996,"15min":8.1609999999999996},"last":0.55500000000000005}]}]},{"osd":2,"up_from":0,"seq":77309411475,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182541,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986313,"num_pgs":167,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320988,"kb_used_data":11472,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93424164,"statfs":{"total":95995035648,"available":95666343936,"internally_reserved":0,"allocated":11747328,"data_stored":8221098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:46:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.55500000000000005,"15min":0.60199999999999998},"min":{"1min":0.248,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.70399999999999996,"5min":5.3319999999999999,"15min":9.3399999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":11.532,"5min":11.532,"15min":13.833},"last":0.38700000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:46:49 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.59599999999999997,"15min":0.76500000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.75700000000000001,"5min":7.5410000000000004,"15min":53.188000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.998,"5min":0.59799999999999998,"15min":0.70799999999999996},"min":{"1min":0.30499999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":11.497999999999999,"5min":11.497999999999999,"15min":44.793999999999997},"last":0.40200000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:46:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.59099999999999997,"15min":0.59999999999999998},"min":{"1min":0.40300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.2199999999999998,"5min":4.2199999999999998,"15min":8.6829999999999998},"last":0.496},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":18.911000000000001},"last":0.44}]},{"osd":3,"last update":"Tue Feb 13 00:46:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.60499999999999998,"15min":0.66300000000000003},"min":{"1min":0.312,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.0090000000000003,"5min":4.0300000000000002,"15min":23.5},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.57599999999999996,"15min":0.72299999999999998},"min":{"1min":0.39700000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.9729999999999999,"5min":3.9729999999999999,"15min":36.152999999999999},"last":0.41499999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.42299999999999999,"15min":0.38500000000000001},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.20599999999999999},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.443,"15min":0.40500000000000003},"min":{"1min":0.27100000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.40200000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.47299999999999998,"15min":0.433},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.25700000000000001},"max":{"1min":0.69899999999999995,"5min":3.153,"15min":3.153},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.51600000000000001,"15min":0.45600000000000002},"min":{"1min":0.32000000000000001,"5min":0.31,"15min":0.254},"max":{"1min":0.59999999999999998,"5min":4.9390000000000001,"15min":4.9390000000000001},"last":0.44900000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:46:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.58399999999999996,"5min":1.575,"15min":1.575},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.497,"15min":0.47099999999999997},"min":{"1min":0.316,"5min":0.29699999999999999,"15min":0.28100000000000003},"max":{"1min":0.72299999999999998,"5min":2.46,"15min":2.46},"last":0.48799999999999999}]}]},{"osd":5,"up_from":0,"seq":167503724668,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":3328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4448256,"data_stored":4001069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4534272,"data_stored":4001252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4550656,"data_stored":4001276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":1050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:47:40.900 INFO:tasks.thrashosds.thrasher:No pools; doing nothing 2024-02-13T00:47:40.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:40.973 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:39.433395+0000 mgr.y (mgr.14544) 412 : audit [DBG] from='client.37559 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:47:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:39.821572+0000 osd.0 (osd.0) 169 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:47:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:39.822611+0000 osd.0 (osd.0) 170 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:47:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:40.319501+0000 osd.6 (osd.6) 129 : cluster [DBG] 5.18 scrub starts 2024-02-13T00:47:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:40.322590+0000 osd.6 (osd.6) 130 : cluster [DBG] 5.18 scrub ok 2024-02-13T00:47:41.079 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:41.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:41.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:41.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:41.211 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:41.409 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:41.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:41.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:41.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:41.541 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:41.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:40.442904+0000 mgr.y (mgr.14544) 413 : cluster [DBG] pgmap v686: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-13T00:47:41.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:40.442904+0000 mgr.y (mgr.14544) 413 : cluster [DBG] pgmap v686: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-13T00:47:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:40.442904+0000 mgr.y (mgr.14544) 413 : cluster [DBG] pgmap v686: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-13T00:47:42.746 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:42.890 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:42.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:42.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:43.026 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:43.275 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:43.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:43.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:43.411 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:43.432 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:43.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:43.532 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:41.646244+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:43.532 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:41.646525+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:43.532 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:41.804599+0000 osd.0 (osd.0) 171 : cluster [DBG] 4.b scrub starts 2024-02-13T00:47:43.532 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:41.808255+0000 osd.0 (osd.0) 172 : cluster [DBG] 4.b scrub ok 2024-02-13T00:47:43.532 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:42.326459+0000 osd.6 (osd.6) 131 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:47:43.532 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:42.329362+0000 osd.6 (osd.6) 132 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:47:43.532 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:42.443242+0000 mgr.y (mgr.14544) 414 : cluster [DBG] pgmap v688: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.0 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-13T00:47:43.532 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:42.444310+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e464: 8 total, 7 up, 4 in 2024-02-13T00:47:43.532 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:42.749869+0000 osd.7 (osd.7) 106 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:43.533 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:42.750165+0000 osd.7 (osd.7) 107 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:43.533 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:41.646244+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:43.533 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:41.646525+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:43.533 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:41.804599+0000 osd.0 (osd.0) 171 : cluster [DBG] 4.b scrub starts 2024-02-13T00:47:43.533 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:41.808255+0000 osd.0 (osd.0) 172 : cluster [DBG] 4.b scrub ok 2024-02-13T00:47:43.533 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:42.326459+0000 osd.6 (osd.6) 131 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:47:43.533 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:42.329362+0000 osd.6 (osd.6) 132 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:47:43.533 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:42.443242+0000 mgr.y (mgr.14544) 414 : cluster [DBG] pgmap v688: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.0 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-13T00:47:43.533 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:42.444310+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e464: 8 total, 7 up, 4 in 2024-02-13T00:47:43.533 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:42.749869+0000 osd.7 (osd.7) 106 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:43.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:42.750165+0000 osd.7 (osd.7) 107 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:43.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:43.565 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:43.585 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:43.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:43.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:43.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:43.718 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:41.646244+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:41.646525+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:41.804599+0000 osd.0 (osd.0) 171 : cluster [DBG] 4.b scrub starts 2024-02-13T00:47:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:41.808255+0000 osd.0 (osd.0) 172 : cluster [DBG] 4.b scrub ok 2024-02-13T00:47:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:42.326459+0000 osd.6 (osd.6) 131 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:47:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:42.329362+0000 osd.6 (osd.6) 132 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:47:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:42.443242+0000 mgr.y (mgr.14544) 414 : cluster [DBG] pgmap v688: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.0 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-13T00:47:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:42.444310+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e464: 8 total, 7 up, 4 in 2024-02-13T00:47:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:42.749869+0000 osd.7 (osd.7) 106 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:47:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:42.750165+0000 osd.7 (osd.7) 107 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:47:44.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.013+0000 7f6f98cb0700 1 -- 172.21.15.162:0/2014642497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f940fd490 msgr2=0x7f6f940f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:44.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.013+0000 7f6f98cb0700 1 --2- 172.21.15.162:0/2014642497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f940fd490 0x7f6f940f6cc0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7f6f840047b0 tx=0x7f6f84009cb0).stop 2024-02-13T00:47:44.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.013+0000 7f6f98cb0700 1 -- 172.21.15.162:0/2014642497 shutdown_connections 2024-02-13T00:47:44.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.013+0000 7f6f98cb0700 1 --2- 172.21.15.162:0/2014642497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f940f73a0 0x7f6f941039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.013+0000 7f6f98cb0700 1 --2- 172.21.15.162:0/2014642497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f940fcb30 0x7f6f940fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:47:44.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.013+0000 7f6f98cb0700 1 --2- 172.21.15.162:0/2014642497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f940fd490 0x7f6f940f6cc0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.013+0000 7f6f98cb0700 1 -- 172.21.15.162:0/2014642497 >> 172.21.15.162:0/2014642497 conn(0x7f6f9406eb90 msgr2=0x7f6f940f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:44.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.013+0000 7f6f98cb0700 1 -- 172.21.15.162:0/2014642497 shutdown_connections 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.013+0000 7f6f98cb0700 1 -- 172.21.15.162:0/2014642497 wait complete. 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.013+0000 7f6f98cb0700 1 librados: starting msgr at 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.013+0000 7f6f98cb0700 1 librados: starting objecter 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f98cb0700 1 Processor -- start 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f98cb0700 1 -- start start 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f98cb0700 1 librados: setting wanted keys 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f98cb0700 1 librados: calling monclient init 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f98cb0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f940f73a0 0x7f6f94109280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f98cb0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f940fcb30 0x7f6f94109770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f98cb0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f940fd490 0x7f6f94109da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f98cb0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f941144b0 con 0x7f6f940fd490 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f98cb0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f94114620 con 0x7f6f940f73a0 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f98cb0700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f941147e0 con 0x7f6f940fcb30 2024-02-13T00:47:44.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f92d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f940fd490 0x7f6f94109da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f92d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f940fd490 0x7f6f94109da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46744/0 (socket says 172.21.15.162:46744) 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f92d9d700 1 -- 172.21.15.162:0/932673507 learned_addr learned my addr 172.21.15.162:0/932673507 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f91d9b700 1 --2- 172.21.15.162:0/932673507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f940fcb30 0x7f6f94109770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f9259c700 1 --2- 172.21.15.162:0/932673507 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f940f73a0 0x7f6f94109280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f92d9d700 1 -- 172.21.15.162:0/932673507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f940fcb30 msgr2=0x7f6f94109770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f92d9d700 1 --2- 172.21.15.162:0/932673507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f940fcb30 0x7f6f94109770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f92d9d700 1 -- 172.21.15.162:0/932673507 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f940f73a0 msgr2=0x7f6f94109280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f92d9d700 1 --2- 172.21.15.162:0/932673507 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f940f73a0 0x7f6f94109280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f92d9d700 1 -- 172.21.15.162:0/932673507 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c0097c0 con 0x7f6f940fd490 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.014+0000 7f6f9259c700 1 --2- 172.21.15.162:0/932673507 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f940f73a0 0x7f6f94109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.015+0000 7f6f92d9d700 1 --2- 172.21.15.162:0/932673507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f940fd490 0x7f6f94109da0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7f6f8c001df0 tx=0x7f6f8c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.015+0000 7f6f837fe700 10 client.?.objecter ms_handle_connect 0x7f6f940fd490 2024-02-13T00:47:44.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.015+0000 7f6f837fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:44.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.015+0000 7f6f837fe700 1 -- 172.21.15.162:0/932673507 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c009a10 con 0x7f6f940fd490 2024-02-13T00:47:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.015+0000 7f6f98cb0700 1 -- 172.21.15.162:0/932673507 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f84005040 con 0x7f6f940fd490 2024-02-13T00:47:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.015+0000 7f6f837fe700 1 -- 172.21.15.162:0/932673507 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f8c0095a0 con 0x7f6f940fd490 2024-02-13T00:47:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.015+0000 7f6f837fe700 1 -- 172.21.15.162:0/932673507 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c007e30 con 0x7f6f940fd490 2024-02-13T00:47:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.015+0000 7f6f98cb0700 10 client.27465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.015+0000 7f6f98cb0700 1 -- 172.21.15.162:0/932673507 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f94114e70 con 0x7f6f940fd490 2024-02-13T00:47:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.015+0000 7f6f98cb0700 1 librados: init done 2024-02-13T00:47:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.016+0000 7f6f98cb0700 1 -- 172.21.15.162:0/932673507 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f940475e0 con 0x7f6f940fd490 2024-02-13T00:47:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.016+0000 7f6f837fe700 1 -- 172.21.15.162:0/932673507 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6f8c0090d0 con 0x7f6f940fd490 2024-02-13T00:47:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.017+0000 7f6f837fe700 1 --2- 172.21.15.162:0/932673507 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6f7c062c90 0x7f6f7c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.017+0000 7f6f9259c700 1 --2- 172.21.15.162:0/932673507 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6f7c062c90 0x7f6f7c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.017+0000 7f6f837fe700 1 -- 172.21.15.162:0/932673507 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(465..465 src has 52..465) v4 ==== 7967+0+0 (secure 0 0 0) 0x7f6f8c07d9c0 con 0x7f6f940fd490 2024-02-13T00:47:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.017+0000 7f6f837fe700 10 client.27465.objecter ms_dispatch 0x7f6f94001550 osd_map(465..465 src has 52..465) v4 2024-02-13T00:47:44.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.017+0000 7f6f837fe700 3 client.27465.objecter handle_osd_map got epochs [465,465] > 0 2024-02-13T00:47:44.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.017+0000 7f6f837fe700 3 client.27465.objecter handle_osd_map decoding full epoch 465 2024-02-13T00:47:44.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.017+0000 7f6f837fe700 20 client.27465.objecter dump_active .. 0 homeless 2024-02-13T00:47:44.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.017+0000 7f6f837fe700 10 client.27465.objecter ms_handle_connect 0x7f6f7c062c90 2024-02-13T00:47:44.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.017+0000 7f6f9259c700 1 --2- 172.21.15.162:0/932673507 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6f7c062c90 0x7f6f7c065150 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 rx=0x7f6f88009990 tx=0x7f6f88008040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:44.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.021+0000 7f6f837fe700 1 -- 172.21.15.162:0/932673507 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6f8c050f30 con 0x7f6f940fd490 2024-02-13T00:47:44.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.200+0000 7f6f98cb0700 1 -- 172.21.15.162:0/932673507 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f6f9405fb80 con 0x7f6f940fd490 2024-02-13T00:47:44.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.226+0000 7f24beeb7700 1 -- 172.21.15.162:0/2121798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24b80fa440 msgr2=0x7f24b80f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:44.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.226+0000 7f24beeb7700 1 --2- 172.21.15.162:0/2121798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24b80fa440 0x7f24b80f7e20 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7f24a40047b0 tx=0x7f24a4009cb0).stop 2024-02-13T00:47:44.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.226+0000 7f24beeb7700 1 -- 172.21.15.162:0/2121798 shutdown_connections 2024-02-13T00:47:44.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.226+0000 7f24beeb7700 1 --2- 172.21.15.162:0/2121798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24b80f8500 0x7f24b8104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.226+0000 7f24beeb7700 1 --2- 172.21.15.162:0/2121798 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24b80f9b70 0x7f24b80f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.226+0000 7f24beeb7700 1 --2- 172.21.15.162:0/2121798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24b80fa440 0x7f24b80f7e20 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.226+0000 7f24beeb7700 1 -- 172.21.15.162:0/2121798 >> 172.21.15.162:0/2121798 conn(0x7f24b80f38c0 msgr2=0x7f24b80f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:44.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.226+0000 7f24beeb7700 1 -- 172.21.15.162:0/2121798 shutdown_connections 2024-02-13T00:47:44.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.226+0000 7f24beeb7700 1 -- 172.21.15.162:0/2121798 wait complete. 2024-02-13T00:47:44.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24beeb7700 1 librados: starting msgr at 2024-02-13T00:47:44.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24beeb7700 1 librados: starting objecter 2024-02-13T00:47:44.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24beeb7700 1 Processor -- start 2024-02-13T00:47:44.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24beeb7700 1 -- start start 2024-02-13T00:47:44.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24beeb7700 1 librados: setting wanted keys 2024-02-13T00:47:44.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24beeb7700 1 librados: calling monclient init 2024-02-13T00:47:44.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24beeb7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24b80f8500 0x7f24b810e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24beeb7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24b80f9b70 0x7f24b810e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24beeb7700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24b80fa440 0x7f24b8108480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24beeb7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b8108970 con 0x7f24b80f8500 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24beeb7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b8108ae0 con 0x7f24b80f9b70 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24beeb7700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b8108c50 con 0x7f24b80fa440 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24bd454700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24b80fa440 0x7f24b8108480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24bd454700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24b80fa440 0x7f24b8108480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:45886/0 (socket says 172.21.15.162:45886) 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24bcc53700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24b80f8500 0x7f24b810e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24bd454700 1 -- 172.21.15.162:0/655776133 learned_addr learned my addr 172.21.15.162:0/655776133 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.227+0000 7f24b7fff700 1 --2- 172.21.15.162:0/655776133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24b80f9b70 0x7f24b810e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24bd454700 1 -- 172.21.15.162:0/655776133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24b80f9b70 msgr2=0x7f24b810e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24bd454700 1 --2- 172.21.15.162:0/655776133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f24b80f9b70 0x7f24b810e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24bd454700 1 -- 172.21.15.162:0/655776133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24b80f8500 msgr2=0x7f24b810e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:44.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24bd454700 1 --2- 172.21.15.162:0/655776133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24b80f8500 0x7f24b810e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24bd454700 1 -- 172.21.15.162:0/655776133 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac005040 con 0x7f24b80fa440 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24bcc53700 1 --2- 172.21.15.162:0/655776133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24b80f8500 0x7f24b810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24bd454700 1 --2- 172.21.15.162:0/655776133 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24b80fa440 0x7f24b8108480 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f24b000d970 tx=0x7f24b0004e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24b5ffb700 10 client.?.objecter ms_handle_connect 0x7f24b80fa440 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24b5ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b0004020 con 0x7f24b80fa440 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24b000f040 con 0x7f24b80fa440 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a4005040 con 0x7f24b80fa440 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b00036c0 con 0x7f24b80fa440 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24beeb7700 10 client.37571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b8109290 con 0x7f24b80fa440 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.228+0000 7f24beeb7700 1 librados: init done 2024-02-13T00:47:44.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.229+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f24b80475e0 con 0x7f24b80fa440 2024-02-13T00:47:44.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.229+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f24b0003af0 con 0x7f24b80fa440 2024-02-13T00:47:44.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.229+0000 7f24b5ffb700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6800/2866534053 conn(0x7f24a8062c90 0x7f24a8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(465..465 src has 52..465) v4 ==== 7967+0+0 (secure 0 0 0) 0x7f24b007cb10 con 0x7f24b80fa440 2024-02-13T00:47:44.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 osd_map(465..465 src has 52..465) v4 2024-02-13T00:47:44.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24b5ffb700 3 client.37571.objecter handle_osd_map got epochs [465,465] > 0 2024-02-13T00:47:44.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24b5ffb700 3 client.37571.objecter handle_osd_map decoding full epoch 465 2024-02-13T00:47:44.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24bcc53700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6800/2866534053 conn(0x7f24a8062c90 0x7f24a8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24b5ffb700 20 client.37571.objecter dump_active .. 0 homeless 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v465) v1 ==== 54+0+16 (secure 0 0 0) 0x7f24b0081220 con 0x7f24b80fa440 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24b5ffb700 10 client.37571.objecter ms_handle_connect 0x7f24a8062c90 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24bcc53700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6800/2866534053 conn(0x7f24a8062c90 0x7f24a8065150 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 rx=0x7f24ac004c80 tx=0x7f24ac00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24beeb7700 10 client.37571.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 1 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24beeb7700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6801/2103743338 conn(0x7f24b80ff070 0x7f24b80ff450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b8107280 osd=0 3 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b8107280 osd=0 3 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b8107280 osd=0 3 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 1 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b8107280 osd=0 3 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 0 1 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b8107280 osd=0 4 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24beeb7700 10 client.37571.objecter _send_command 1 2024-02-13T00:47:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b80ff940 con 0x7f24b80ff070 2024-02-13T00:47:44.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24b7fff700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6801/2103743338 conn(0x7f24b80ff070 0x7f24b80ff450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24b5ffb700 10 client.37571.objecter ms_handle_connect 0x7f24b80ff070 2024-02-13T00:47:44.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.230+0000 7f24b7fff700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6801/2103743338 conn(0x7f24b80ff070 0x7f24b80ff450 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:44.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.231+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b80ff940 con 0x7f24b80ff070 2024-02-13T00:47:44.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.231+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:47:44.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.231+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 1 = system:0 2024-02-13T00:47:44.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.231+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b8107280 osd=0 4 2024-02-13T00:47:44.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.231+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 0 1 2024-02-13T00:47:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.246+0000 7f24beeb7700 10 client.37571.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.246+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 2 2024-02-13T00:47:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.246+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b8107280 osd=0 3 2024-02-13T00:47:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.246+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b8107280 osd=0 3 2024-02-13T00:47:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.246+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b8107280 osd=0 3 2024-02-13T00:47:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.246+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 2 2024-02-13T00:47:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.246+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b8107280 osd=0 3 2024-02-13T00:47:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.246+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 0 2 2024-02-13T00:47:44.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.246+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b8107280 osd=0 4 2024-02-13T00:47:44.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.246+0000 7f24beeb7700 10 client.37571.objecter _send_command 2 2024-02-13T00:47:44.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.246+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b80ffb60 con 0x7f24b80ff070 2024-02-13T00:47:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.247+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24a4019040 con 0x7f24b80ff070 2024-02-13T00:47:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.247+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.247+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.247+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b8107280 osd=0 4 2024-02-13T00:47:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.247+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 0 2 2024-02-13T00:47:44.251 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:47:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24beeb7700 10 client.37571.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 3 2024-02-13T00:47:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24beeb7700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6805/150599780 conn(0x7f24b8028590 0x7f24b81191f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b80f4f80 osd=1 3 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b80f4f80 osd=1 3 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b80f4f80 osd=1 3 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 3 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b80f4f80 osd=1 3 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 1 3 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b80f4f80 osd=1 4 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24beeb7700 10 client.37571.objecter _send_command 3 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b80f50f0 con 0x7f24b8028590 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24bcc53700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6805/150599780 conn(0x7f24b8028590 0x7f24b81191f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24b5ffb700 10 client.37571.objecter ms_handle_connect 0x7f24b8028590 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.248+0000 7f24bcc53700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6805/150599780 conn(0x7f24b8028590 0x7f24b81191f0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.249+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b80f50f0 con 0x7f24b8028590 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.249+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:47:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.249+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 3 = system:0 2024-02-13T00:47:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.249+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b80f4f80 osd=1 4 2024-02-13T00:47:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.249+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 1 3 2024-02-13T00:47:44.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.263+0000 7f24beeb7700 10 client.37571.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:44.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.263+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 4 2024-02-13T00:47:44.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.263+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b80f4f80 osd=1 3 2024-02-13T00:47:44.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.263+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b80f4f80 osd=1 3 2024-02-13T00:47:44.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.263+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b80f4f80 osd=1 3 2024-02-13T00:47:44.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.263+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 4 2024-02-13T00:47:44.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.263+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b80f4f80 osd=1 3 2024-02-13T00:47:44.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.263+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 1 4 2024-02-13T00:47:44.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.263+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b80f4f80 osd=1 4 2024-02-13T00:47:44.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.263+0000 7f24beeb7700 10 client.37571.objecter _send_command 4 2024-02-13T00:47:44.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.263+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b805a0e0 con 0x7f24b8028590 2024-02-13T00:47:44.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.264+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b805a0e0 con 0x7f24b8028590 2024-02-13T00:47:44.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.264+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:44.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.264+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.264+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b80f4f80 osd=1 4 2024-02-13T00:47:44.268 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.264+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 1 4 2024-02-13T00:47:44.269 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:47:44.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24beeb7700 10 client.37571.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:44.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 5 2024-02-13T00:47:44.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24beeb7700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6809/3689719376 conn(0x7f24b8059570 0x7f24b806c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b8059400 osd=2 3 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b8059400 osd=2 3 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b8059400 osd=2 3 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 5 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b8059400 osd=2 3 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 2 5 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b8059400 osd=2 4 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24beeb7700 10 client.37571.objecter _send_command 5 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b80280f0 con 0x7f24b8059570 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24b7fff700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6809/3689719376 conn(0x7f24b8059570 0x7f24b806c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24b5ffb700 10 client.37571.objecter ms_handle_connect 0x7f24b8059570 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.265+0000 7f24b7fff700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6809/3689719376 conn(0x7f24b8059570 0x7f24b806c130 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:44.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.266+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b80280f0 con 0x7f24b8059570 2024-02-13T00:47:44.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.266+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:47:44.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.266+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 5 = system:0 2024-02-13T00:47:44.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.266+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b8059400 osd=2 4 2024-02-13T00:47:44.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.266+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 2 5 2024-02-13T00:47:44.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.280+0000 7f24beeb7700 10 client.37571.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:44.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.280+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 6 2024-02-13T00:47:44.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.280+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b8059400 osd=2 3 2024-02-13T00:47:44.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.280+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b8059400 osd=2 3 2024-02-13T00:47:44.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.280+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b8059400 osd=2 3 2024-02-13T00:47:44.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.280+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 6 2024-02-13T00:47:44.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.280+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b8059400 osd=2 3 2024-02-13T00:47:44.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.280+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 2 6 2024-02-13T00:47:44.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.280+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b8059400 osd=2 4 2024-02-13T00:47:44.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.280+0000 7f24beeb7700 10 client.37571.objecter _send_command 6 2024-02-13T00:47:44.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.280+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b806c620 con 0x7f24b8059570 2024-02-13T00:47:44.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.281+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b806c620 con 0x7f24b8059570 2024-02-13T00:47:44.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.281+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:44.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.281+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.281+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b8059400 osd=2 4 2024-02-13T00:47:44.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.281+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 2 6 2024-02-13T00:47:44.285 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:47:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24beeb7700 10 client.37571.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 7 2024-02-13T00:47:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24beeb7700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6813/604971011 conn(0x7f24b811fb00 0x7f24b8121ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b806cfc0 osd=3 3 2024-02-13T00:47:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b806cfc0 osd=3 3 2024-02-13T00:47:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b806cfc0 osd=3 3 2024-02-13T00:47:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 7 2024-02-13T00:47:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b806cfc0 osd=3 3 2024-02-13T00:47:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 3 7 2024-02-13T00:47:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b806cfc0 osd=3 4 2024-02-13T00:47:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24beeb7700 10 client.37571.objecter _send_command 7 2024-02-13T00:47:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b8124be0 con 0x7f24b811fb00 2024-02-13T00:47:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24bd454700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6813/604971011 conn(0x7f24b811fb00 0x7f24b8121ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24b5ffb700 10 client.37571.objecter ms_handle_connect 0x7f24b811fb00 2024-02-13T00:47:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24bd454700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.162:6813/604971011 conn(0x7f24b811fb00 0x7f24b8121ef0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b8124be0 con 0x7f24b811fb00 2024-02-13T00:47:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:47:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 7 = system:0 2024-02-13T00:47:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b806cfc0 osd=3 4 2024-02-13T00:47:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.282+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 3 7 2024-02-13T00:47:44.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.296+0000 7f24beeb7700 10 client.37571.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:44.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.296+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 8 2024-02-13T00:47:44.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.296+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b806cfc0 osd=3 3 2024-02-13T00:47:44.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.296+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b806cfc0 osd=3 3 2024-02-13T00:47:44.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.296+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b806cfc0 osd=3 3 2024-02-13T00:47:44.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.296+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 8 2024-02-13T00:47:44.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.296+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b806cfc0 osd=3 3 2024-02-13T00:47:44.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.296+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 3 8 2024-02-13T00:47:44.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.296+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b806cfc0 osd=3 4 2024-02-13T00:47:44.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.296+0000 7f24beeb7700 10 client.37571.objecter _send_command 8 2024-02-13T00:47:44.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.296+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b806ca30 con 0x7f24b811fb00 2024-02-13T00:47:44.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:44.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.298+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b806ca30 con 0x7f24b811fb00 2024-02-13T00:47:44.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.298+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:44.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.298+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.298+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b806cfc0 osd=3 4 2024-02-13T00:47:44.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.298+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 3 8 2024-02-13T00:47:44.302 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:47:44.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.298+0000 7f24beeb7700 10 client.37571.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:44.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.298+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 9 2024-02-13T00:47:44.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.298+0000 7f24beeb7700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.181:6800/3344263388 conn(0x7f24b806bc90 0x7f24b8123600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.299+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b806ca30 osd=4 3 2024-02-13T00:47:44.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.299+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b806ca30 osd=4 3 2024-02-13T00:47:44.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.299+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b806ca30 osd=4 3 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.299+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 9 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.299+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b806ca30 osd=4 3 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.299+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 4 9 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.299+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b806ca30 osd=4 4 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.299+0000 7f24beeb7700 10 client.37571.objecter _send_command 9 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.299+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b806cd40 con 0x7f24b806bc90 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.299+0000 7f24bcc53700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.181:6800/3344263388 conn(0x7f24b806bc90 0x7f24b8123600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.299+0000 7f24b5ffb700 10 client.37571.objecter ms_handle_connect 0x7f24b806bc90 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.299+0000 7f24bcc53700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.181:6800/3344263388 conn(0x7f24b806bc90 0x7f24b8123600 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.300+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b806cd40 con 0x7f24b806bc90 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.300+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.300+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 9 = system:0 2024-02-13T00:47:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.300+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b806ca30 osd=4 4 2024-02-13T00:47:44.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.300+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 4 9 2024-02-13T00:47:44.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:44.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:44.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.315+0000 7f24beeb7700 10 client.37571.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:44.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.315+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 10 2024-02-13T00:47:44.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.315+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b806ca30 osd=4 3 2024-02-13T00:47:44.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.315+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b806ca30 osd=4 3 2024-02-13T00:47:44.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.315+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b806ca30 osd=4 3 2024-02-13T00:47:44.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.315+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 10 2024-02-13T00:47:44.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.315+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b806ca30 osd=4 3 2024-02-13T00:47:44.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.315+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 4 10 2024-02-13T00:47:44.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.315+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b806ca30 osd=4 4 2024-02-13T00:47:44.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.315+0000 7f24beeb7700 10 client.37571.objecter _send_command 10 2024-02-13T00:47:44.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.315+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b806cd40 con 0x7f24b806bc90 2024-02-13T00:47:44.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b806cd40 con 0x7f24b806bc90 2024-02-13T00:47:44.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:44.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b806ca30 osd=4 4 2024-02-13T00:47:44.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 4 10 2024-02-13T00:47:44.321 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:47:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 10 client.37571.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 11 2024-02-13T00:47:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 20 client.37571.objecter _get_session osd=-1 returning homeless 2024-02-13T00:47:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 10 client.37571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=466}) v3 -- 0x7f24b8123b20 con 0x7f24b80fa440 2024-02-13T00:47:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f24b806cd40 con 0x7f24b80fa440 2024-02-13T00:47:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_get_version_reply(handle=1 version=465) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24b000baf0 con 0x7f24b80fa440 2024-02-13T00:47:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24b6ffd700 10 client.37571.objecter _check_command_map_dne tid 11 current 465 map_dne_bound 465 2024-02-13T00:47:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24b6ffd700 10 client.37571.objecter _finish_command 11 = osd:6 osd down 2024-02-13T00:47:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24b6ffd700 15 client.37571.objecter _session_command_op_remove -1 11 2024-02-13T00:47:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 10 client.37571.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 12 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.181:6808/1506640341 conn(0x7f24b806b360 0x7f24b8128da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b806b1f0 osd=6 3 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b806b1f0 osd=6 3 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b806b1f0 osd=6 3 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 12 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b806b1f0 osd=6 3 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 6 12 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b806b1f0 osd=6 4 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 10 client.37571.objecter _send_command 12 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.318+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.181:6808/1506640341 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b812bb80 con 0x7f24b806b360 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.319+0000 7f24b7fff700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.181:6808/1506640341 conn(0x7f24b806b360 0x7f24b8128da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.319+0000 7f24b5ffb700 10 client.37571.objecter ms_handle_connect 0x7f24b806b360 2024-02-13T00:47:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.319+0000 7f24b7fff700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.181:6808/1506640341 conn(0x7f24b806b360 0x7f24b8128da0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:44.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.320+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b812bb80 con 0x7f24b806b360 2024-02-13T00:47:44.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.320+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 12: 0 ) v1 2024-02-13T00:47:44.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.320+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 12 = system:0 2024-02-13T00:47:44.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.320+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b806b1f0 osd=6 4 2024-02-13T00:47:44.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.320+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 6 12 2024-02-13T00:47:44.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.334+0000 7f24beeb7700 10 client.37571.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:44.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.334+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 13 2024-02-13T00:47:44.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.334+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b806b1f0 osd=6 3 2024-02-13T00:47:44.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.334+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b806b1f0 osd=6 3 2024-02-13T00:47:44.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.334+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b806b1f0 osd=6 3 2024-02-13T00:47:44.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.334+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 13 2024-02-13T00:47:44.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.334+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b806b1f0 osd=6 3 2024-02-13T00:47:44.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.334+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 6 13 2024-02-13T00:47:44.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.334+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b806b1f0 osd=6 4 2024-02-13T00:47:44.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.334+0000 7f24beeb7700 10 client.37571.objecter _send_command 13 2024-02-13T00:47:44.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.334+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b812bb80 con 0x7f24b806b360 2024-02-13T00:47:44.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.336+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b812bb80 con 0x7f24b806b360 2024-02-13T00:47:44.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.336+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:44.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.336+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.336+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b806b1f0 osd=6 4 2024-02-13T00:47:44.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.336+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 6 13 2024-02-13T00:47:44.340 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:47:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.336+0000 7f24beeb7700 10 client.37571.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.336+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 14 2024-02-13T00:47:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24beeb7700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.181:6812/1964192772 conn(0x7f24b812ae80 0x7f24b812b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b812bb80 osd=7 3 2024-02-13T00:47:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b812bb80 osd=7 3 2024-02-13T00:47:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b812bb80 osd=7 3 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 14 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b812bb80 osd=7 3 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 7 14 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b812bb80 osd=7 4 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24beeb7700 10 client.37571.objecter _send_command 14 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.181:6812/1964192772 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b812b980 con 0x7f24b812ae80 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24bd454700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.181:6812/1964192772 conn(0x7f24b812ae80 0x7f24b812b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24b5ffb700 10 client.37571.objecter ms_handle_connect 0x7f24b812ae80 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.337+0000 7f24bd454700 1 --2- 172.21.15.162:0/655776133 >> v2:172.21.15.181:6812/1964192772 conn(0x7f24b812ae80 0x7f24b812b2e0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.338+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b812b980 con 0x7f24b812ae80 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.338+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 14: 0 ) v1 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.338+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 14 = system:0 2024-02-13T00:47:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.338+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b812bb80 osd=7 4 2024-02-13T00:47:44.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.338+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 7 14 2024-02-13T00:47:44.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.352+0000 7f24beeb7700 10 client.37571.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:47:44.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.352+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign -1 15 2024-02-13T00:47:44.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.352+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b812bb80 osd=7 3 2024-02-13T00:47:44.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.352+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b812bb80 osd=7 3 2024-02-13T00:47:44.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.352+0000 7f24beeb7700 20 client.37571.objecter _get_session s=0x7f24b812bb80 osd=7 3 2024-02-13T00:47:44.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.352+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_remove -1 15 2024-02-13T00:47:44.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.352+0000 7f24beeb7700 20 client.37571.objecter get_session s=0x7f24b812bb80 osd=7 3 2024-02-13T00:47:44.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.352+0000 7f24beeb7700 15 client.37571.objecter _session_command_op_assign 7 15 2024-02-13T00:47:44.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.352+0000 7f24beeb7700 20 client.37571.objecter put_session s=0x7f24b812bb80 osd=7 4 2024-02-13T00:47:44.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.352+0000 7f24beeb7700 10 client.37571.objecter _send_command 15 2024-02-13T00:47:44.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.352+0000 7f24beeb7700 1 -- 172.21.15.162:0/655776133 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b812b980 con 0x7f24b812ae80 2024-02-13T00:47:44.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.354+0000 7f24b5ffb700 1 -- 172.21.15.162:0/655776133 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b812b980 con 0x7f24b812ae80 2024-02-13T00:47:44.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.354+0000 7f24b5ffb700 10 client.37571.objecter ms_dispatch 0x7f24b8001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:47:44.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.354+0000 7f24b5ffb700 10 client.37571.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.354+0000 7f24b5ffb700 20 client.37571.objecter put_session s=0x7f24b812bb80 osd=7 4 2024-02-13T00:47:44.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.354+0000 7f24b5ffb700 15 client.37571.objecter _session_command_op_remove 7 15 2024-02-13T00:47:44.358 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:47:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-02-13T00:47:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:47:44.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:44.438 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:44.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.456+0000 7f6f837fe700 1 -- 172.21.15.162:0/932673507 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v466) v1 ==== 101+0+0 (secure 0 0 0) 0x7f6f8c050b20 con 0x7f6f940fd490 2024-02-13T00:47:44.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.459+0000 7f6f98cb0700 10 librados: watch_flush enter 2024-02-13T00:47:44.462 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:44.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.459+0000 7f6f98cb0700 10 librados: watch_flush exit 2024-02-13T00:47:44.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.459+0000 7f6f98cb0700 20 client.27465.objecter shutdown clearing up homeless session... 2024-02-13T00:47:44.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.459+0000 7f6f98cb0700 10 client.27465.objecter successfully canceled tick 2024-02-13T00:47:44.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.459+0000 7f6f98cb0700 1 -- 172.21.15.162:0/932673507 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6f7c062c90 msgr2=0x7f6f7c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:44.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.459+0000 7f6f98cb0700 1 --2- 172.21.15.162:0/932673507 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6f7c062c90 0x7f6f7c065150 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 rx=0x7f6f88009990 tx=0x7f6f88008040).stop 2024-02-13T00:47:44.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.459+0000 7f6f98cb0700 1 -- 172.21.15.162:0/932673507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f940fd490 msgr2=0x7f6f94109da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:44.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.459+0000 7f6f98cb0700 1 --2- 172.21.15.162:0/932673507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f940fd490 0x7f6f94109da0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7f6f8c001df0 tx=0x7f6f8c004d10).stop 2024-02-13T00:47:44.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.460+0000 7f6f98cb0700 1 -- 172.21.15.162:0/932673507 shutdown_connections 2024-02-13T00:47:44.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.460+0000 7f6f98cb0700 1 --2- 172.21.15.162:0/932673507 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6f7c062c90 0x7f6f7c065150 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.460+0000 7f6f98cb0700 1 --2- 172.21.15.162:0/932673507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f940fd490 0x7f6f94109da0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.460+0000 7f6f98cb0700 1 --2- 172.21.15.162:0/932673507 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f940f73a0 0x7f6f94109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.460+0000 7f6f98cb0700 1 --2- 172.21.15.162:0/932673507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f940fcb30 0x7f6f94109770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:44.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.460+0000 7f6f98cb0700 1 -- 172.21.15.162:0/932673507 >> 172.21.15.162:0/932673507 conn(0x7f6f9406eb90 msgr2=0x7f6f940fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:44.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.460+0000 7f6f98cb0700 1 -- 172.21.15.162:0/932673507 shutdown_connections 2024-02-13T00:47:44.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.460+0000 7f6f98cb0700 1 -- 172.21.15.162:0/932673507 wait complete. 2024-02-13T00:47:44.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:44.460+0000 7f6f98cb0700 1 librados: shutdown 2024-02-13T00:47:44.465 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:47:44.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:44.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:44.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:44.579 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:43.453194+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e465: 8 total, 7 up, 4 in 2024-02-13T00:47:44.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:43.902623+0000 osd.1 (osd.1) 199 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:47:44.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:43.903508+0000 osd.1 (osd.1) 200 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:47:44.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:44.201646+0000 mon.a (mon.0) 2630 : audit [INF] from='client.? 172.21.15.162:0/932673507' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:47:44.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:44.230609+0000 mon.c (mon.2) 391 : audit [DBG] from='client.? 172.21.15.162:0/655776133' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:44.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:43.453194+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e465: 8 total, 7 up, 4 in 2024-02-13T00:47:44.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:43.902623+0000 osd.1 (osd.1) 199 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:47:44.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:43.903508+0000 osd.1 (osd.1) 200 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:47:44.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:44.201646+0000 mon.a (mon.0) 2630 : audit [INF] from='client.? 172.21.15.162:0/932673507' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:47:44.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:44.230609+0000 mon.c (mon.2) 391 : audit [DBG] from='client.? 172.21.15.162:0/655776133' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:44.596 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:44.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:43.453194+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e465: 8 total, 7 up, 4 in 2024-02-13T00:47:44.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:43.902623+0000 osd.1 (osd.1) 199 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:47:44.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:43.903508+0000 osd.1 (osd.1) 200 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:47:44.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:44.201646+0000 mon.a (mon.0) 2630 : audit [INF] from='client.? 172.21.15.162:0/932673507' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:47:44.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:44.230609+0000 mon.c (mon.2) 391 : audit [DBG] from='client.? 172.21.15.162:0/655776133' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:44.946 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:44.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:44.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:45.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:45.081 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:45.102 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:45.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:45.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:45.207 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:45.238 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:45.258 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:45.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:45.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:45.362 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:45.390 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:45.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:44.443594+0000 mgr.y (mgr.14544) 415 : cluster [DBG] pgmap v690: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-13T00:47:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:44.448543+0000 mon.a (mon.0) 2631 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:47:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:44.457583+0000 mon.a (mon.0) 2632 : audit [INF] from='client.? 172.21.15.162:0/932673507' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:47:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:44.457617+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e466: 8 total, 7 up, 4 in 2024-02-13T00:47:45.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:45.441868+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e467: 8 total, 7 up, 4 in 2024-02-13T00:47:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:44.443594+0000 mgr.y (mgr.14544) 415 : cluster [DBG] pgmap v690: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-13T00:47:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:44.448543+0000 mon.a (mon.0) 2631 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:47:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:44.457583+0000 mon.a (mon.0) 2632 : audit [INF] from='client.? 172.21.15.162:0/932673507' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:47:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:44.457617+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e466: 8 total, 7 up, 4 in 2024-02-13T00:47:45.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:45.441868+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e467: 8 total, 7 up, 4 in 2024-02-13T00:47:45.588 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:45.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:45.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:45.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:45.720 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:45.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:44.443594+0000 mgr.y (mgr.14544) 415 : cluster [DBG] pgmap v690: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-13T00:47:45.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:44.448543+0000 mon.a (mon.0) 2631 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:47:45.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:44.457583+0000 mon.a (mon.0) 2632 : audit [INF] from='client.? 172.21.15.162:0/932673507' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:47:45.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:44.457617+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e466: 8 total, 7 up, 4 in 2024-02-13T00:47:45.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:45.441868+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e467: 8 total, 7 up, 4 in 2024-02-13T00:47:45.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:47:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:47:45.902 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6] out_osds: [3, 2, 7, 5] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-02-13T00:47:45.902 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:47:45.903 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-02-13T00:47:45.903 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-13T00:47:45.903 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-13T00:47:45.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-02-13T00:47:46.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:47:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:47:46] "GET /metrics HTTP/1.1" 200 177634 "" "Prometheus/2.33.4" 2024-02-13T00:47:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-13T00:47:46.464 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-13T00:47:46.568 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:46.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:46.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:46.704 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:46.725 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:47:46.727 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:44.752408+0000 osd.0 (osd.0) 173 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:47:46.727 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:44.757254+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:47:46.727 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:45.673032+0000 mgr.y (mgr.14544) 416 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:46.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:47:46.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:44.752408+0000 osd.0 (osd.0) 173 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:47:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:44.757254+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:47:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:45.673032+0000 mgr.y (mgr.14544) 416 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:46.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:44.752408+0000 osd.0 (osd.0) 173 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:47:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:44.757254+0000 osd.0 (osd.0) 174 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:47:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:45.673032+0000 mgr.y (mgr.14544) 416 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:46.861 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:46.880 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:46.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:46.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:46.905 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:46.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:47.011 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:47.210 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:47.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:47.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:47:47.346 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:47.367 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:47:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:47.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:47:47.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:47.500 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:46.444035+0000 mgr.y (mgr.14544) 417 : cluster [DBG] pgmap v693: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:47:47.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:46.444035+0000 mgr.y (mgr.14544) 417 : cluster [DBG] pgmap v693: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:47:47.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:46.444035+0000 mgr.y (mgr.14544) 417 : cluster [DBG] pgmap v693: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:47:48.449 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:48.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:48.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:47:48.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:48.581 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:48.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:47.281066+0000 osd.6 (osd.6) 133 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:47:48.693 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:47.284010+0000 osd.6 (osd.6) 134 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:47:48.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:47.281066+0000 osd.6 (osd.6) 133 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:47:48.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:47.284010+0000 osd.6 (osd.6) 134 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:47:48.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:47.281066+0000 osd.6 (osd.6) 133 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:47:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:47.284010+0000 osd.6 (osd.6) 134 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:47:48.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:47:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:47:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:47:48.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:47:48.957 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:47:48.957 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:47:49.190 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:47:49.190 INFO:teuthology.orchestra.run.smithi162.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-13T00:47:49.190 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:47:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:47.708494+0000 osd.0 (osd.0) 175 : cluster [DBG] 2.1e scrub starts 2024-02-13T00:47:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:47.711482+0000 osd.0 (osd.0) 176 : cluster [DBG] 2.1e scrub ok 2024-02-13T00:47:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:48.444395+0000 mgr.y (mgr.14544) 418 : cluster [DBG] pgmap v694: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:47:49.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:49.337862+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:47:49.829 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:47:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:47:49.341+0000 7f16fd9f6700 -1 mgr get_metadata_python Requested missing service osd.5 2024-02-13T00:47:49.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:47.708494+0000 osd.0 (osd.0) 175 : cluster [DBG] 2.1e scrub starts 2024-02-13T00:47:49.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:47.711482+0000 osd.0 (osd.0) 176 : cluster [DBG] 2.1e scrub ok 2024-02-13T00:47:49.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:48.444395+0000 mgr.y (mgr.14544) 418 : cluster [DBG] pgmap v694: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:47:49.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:49.337862+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:47:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:47.708494+0000 osd.0 (osd.0) 175 : cluster [DBG] 2.1e scrub starts 2024-02-13T00:47:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:47.711482+0000 osd.0 (osd.0) 176 : cluster [DBG] 2.1e scrub ok 2024-02-13T00:47:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:48.444395+0000 mgr.y (mgr.14544) 418 : cluster [DBG] pgmap v694: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:47:49.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:49.337862+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:47:49.876 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-13T00:47:50.097 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:50.560 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:49.490278+0000 mon.b (mon.1) 599 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:50.560 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:49.490635+0000 mon.a (mon.0) 2636 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:50.560 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:49.491037+0000 mon.b (mon.1) 600 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-71"}]: dispatch 2024-02-13T00:47:50.560 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:49.491299+0000 mon.a (mon.0) 2637 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-71"}]: dispatch 2024-02-13T00:47:50.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:49.490278+0000 mon.b (mon.1) 599 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:50.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:49.490635+0000 mon.a (mon.0) 2636 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:50.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:49.491037+0000 mon.b (mon.1) 600 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-71"}]: dispatch 2024-02-13T00:47:50.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:49.491299+0000 mon.a (mon.0) 2637 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-71"}]: dispatch 2024-02-13T00:47:50.830 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:50.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:50.935 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:50.963 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:49.490278+0000 mon.b (mon.1) 599 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:49.490635+0000 mon.a (mon.0) 2636 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:49.491037+0000 mon.b (mon.1) 600 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-71"}]: dispatch 2024-02-13T00:47:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:49.491299+0000 mon.a (mon.0) 2637 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-71"}]: dispatch 2024-02-13T00:47:51.359 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:51.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:49.683486+0000 osd.0 (osd.0) 177 : cluster [DBG] 5.e deep-scrub starts 2024-02-13T00:47:51.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:49.687722+0000 osd.0 (osd.0) 178 : cluster [DBG] 5.e deep-scrub ok 2024-02-13T00:47:51.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:50.445658+0000 mgr.y (mgr.14544) 419 : cluster [DBG] pgmap v695: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-02-13T00:47:51.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:50.529319+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e468: 8 total, 7 up, 4 in 2024-02-13T00:47:51.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:49.683486+0000 osd.0 (osd.0) 177 : cluster [DBG] 5.e deep-scrub starts 2024-02-13T00:47:51.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:49.687722+0000 osd.0 (osd.0) 178 : cluster [DBG] 5.e deep-scrub ok 2024-02-13T00:47:51.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:50.445658+0000 mgr.y (mgr.14544) 419 : cluster [DBG] pgmap v695: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-02-13T00:47:51.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:50.529319+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e468: 8 total, 7 up, 4 in 2024-02-13T00:47:51.997 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:49.683486+0000 osd.0 (osd.0) 177 : cluster [DBG] 5.e deep-scrub starts 2024-02-13T00:47:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:49.687722+0000 osd.0 (osd.0) 178 : cluster [DBG] 5.e deep-scrub ok 2024-02-13T00:47:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:50.445658+0000 mgr.y (mgr.14544) 419 : cluster [DBG] pgmap v695: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-02-13T00:47:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:50.529319+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e468: 8 total, 7 up, 4 in 2024-02-13T00:47:52.574 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:51.567473+0000 mon.a (mon.0) 2639 : cluster [DBG] osdmap e469: 8 total, 7 up, 4 in 2024-02-13T00:47:52.574 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:51.573845+0000 mon.b (mon.1) 601 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:52.574 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:51.576052+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:52.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:51.638322+0000 osd.0 (osd.0) 179 : cluster [DBG] 5.1e scrub starts 2024-02-13T00:47:52.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:51.641458+0000 osd.0 (osd.0) 180 : cluster [DBG] 5.1e scrub ok 2024-02-13T00:47:52.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:51.567473+0000 mon.a (mon.0) 2639 : cluster [DBG] osdmap e469: 8 total, 7 up, 4 in 2024-02-13T00:47:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:51.573845+0000 mon.b (mon.1) 601 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:51.576052+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:51.638322+0000 osd.0 (osd.0) 179 : cluster [DBG] 5.1e scrub starts 2024-02-13T00:47:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:51.641458+0000 osd.0 (osd.0) 180 : cluster [DBG] 5.1e scrub ok 2024-02-13T00:47:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:51.567473+0000 mon.a (mon.0) 2639 : cluster [DBG] osdmap e469: 8 total, 7 up, 4 in 2024-02-13T00:47:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:51.573845+0000 mon.b (mon.1) 601 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:51.576052+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:51.638322+0000 osd.0 (osd.0) 179 : cluster [DBG] 5.1e scrub starts 2024-02-13T00:47:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:51.641458+0000 osd.0 (osd.0) 180 : cluster [DBG] 5.1e scrub ok 2024-02-13T00:47:53.246 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:53.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:53.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:53.351 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:53.384 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:53.405 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:53.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:53.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:53.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:53.538 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:53.793 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:52.446071+0000 mgr.y (mgr.14544) 420 : cluster [DBG] pgmap v698: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-02-13T00:47:53.793 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:52.573103+0000 mon.a (mon.0) 2641 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:53.793 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:52.573189+0000 mon.a (mon.0) 2642 : cluster [DBG] osdmap e470: 8 total, 7 up, 4 in 2024-02-13T00:47:53.793 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:52.579511+0000 mon.b (mon.1) 602 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:53.794 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:52.446071+0000 mgr.y (mgr.14544) 420 : cluster [DBG] pgmap v698: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-02-13T00:47:53.794 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:52.573103+0000 mon.a (mon.0) 2641 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:53.794 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:52.573189+0000 mon.a (mon.0) 2642 : cluster [DBG] osdmap e470: 8 total, 7 up, 4 in 2024-02-13T00:47:53.794 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:52.579511+0000 mon.b (mon.1) 602 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:53.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:52.446071+0000 mgr.y (mgr.14544) 420 : cluster [DBG] pgmap v698: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-02-13T00:47:53.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:52.573103+0000 mon.a (mon.0) 2641 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:52.573189+0000 mon.a (mon.0) 2642 : cluster [DBG] osdmap e470: 8 total, 7 up, 4 in 2024-02-13T00:47:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:52.579511+0000 mon.b (mon.1) 602 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:53.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:53.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:53.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.993+0000 7f309b1c2700 1 -- 172.21.15.162:0/3239913042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30940f9b70 msgr2=0x7f30940f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:53.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.993+0000 7f309b1c2700 1 --2- 172.21.15.162:0/3239913042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30940f9b70 0x7f30940f9f50 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7f3088004870 tx=0x7f3088009cb0).stop 2024-02-13T00:47:53.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.993+0000 7f309b1c2700 1 -- 172.21.15.162:0/3239913042 shutdown_connections 2024-02-13T00:47:53.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.993+0000 7f309b1c2700 1 --2- 172.21.15.162:0/3239913042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30940f8500 0x7f3094104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:53.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.993+0000 7f309b1c2700 1 --2- 172.21.15.162:0/3239913042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30940f9b70 0x7f30940f9f50 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.993+0000 7f309b1c2700 1 --2- 172.21.15.162:0/3239913042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30940fa440 0x7f30940f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.993+0000 7f309b1c2700 1 -- 172.21.15.162:0/3239913042 >> 172.21.15.162:0/3239913042 conn(0x7f30940f38c0 msgr2=0x7f30940f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.993+0000 7f309b1c2700 1 -- 172.21.15.162:0/3239913042 shutdown_connections 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.993+0000 7f309b1c2700 1 -- 172.21.15.162:0/3239913042 wait complete. 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.993+0000 7f309b1c2700 1 librados: starting msgr at 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.993+0000 7f309b1c2700 1 librados: starting objecter 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f309b1c2700 1 Processor -- start 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f309b1c2700 1 -- start start 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f309b1c2700 1 librados: setting wanted keys 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f309b1c2700 1 librados: calling monclient init 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f309b1c2700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30940f8500 0x7f309406b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f309b1c2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30940f9b70 0x7f309406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f309b1c2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30940fa440 0x7f309406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:53.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f309b1c2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30941166c0 con 0x7f30940f9b70 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f309b1c2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3094116830 con 0x7f30940fa440 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f309b1c2700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30941169f0 con 0x7f30940f8500 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3098f5e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30940f8500 0x7f309406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3098f5e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30940f8500 0x7f309406b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:39352/0 (socket says 172.21.15.162:39352) 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3098f5e700 1 -- 172.21.15.162:0/1921171230 learned_addr learned my addr 172.21.15.162:0/1921171230 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f309975f700 1 --2- 172.21.15.162:0/1921171230 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30940fa440 0x7f309406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3093fff700 1 --2- 172.21.15.162:0/1921171230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30940f9b70 0x7f309406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3098f5e700 1 -- 172.21.15.162:0/1921171230 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30940fa440 msgr2=0x7f309406bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3098f5e700 1 --2- 172.21.15.162:0/1921171230 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30940fa440 0x7f309406bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3098f5e700 1 -- 172.21.15.162:0/1921171230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30940f9b70 msgr2=0x7f309406b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3098f5e700 1 --2- 172.21.15.162:0/1921171230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30940f9b70 0x7f309406b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3098f5e700 1 -- 172.21.15.162:0/1921171230 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c0097c0 con 0x7f30940f8500 2024-02-13T00:47:54.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3093fff700 1 --2- 172.21.15.162:0/1921171230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30940f9b70 0x7f309406b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:47:54.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3098f5e700 1 --2- 172.21.15.162:0/1921171230 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30940f8500 0x7f309406b170 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7f3088009eb0 tx=0x7f308800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:54.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3091ffb700 10 client.?.objecter ms_handle_connect 0x7f30940f8500 2024-02-13T00:47:54.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.994+0000 7f3091ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:54.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.995+0000 7f3091ffb700 1 -- 172.21.15.162:0/1921171230 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308800e070 con 0x7f30940f8500 2024-02-13T00:47:54.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.995+0000 7f3091ffb700 1 -- 172.21.15.162:0/1921171230 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3088013da0 con 0x7f30940f8500 2024-02-13T00:47:54.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.995+0000 7f309b1c2700 1 -- 172.21.15.162:0/1921171230 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3088005040 con 0x7f30940f8500 2024-02-13T00:47:54.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.995+0000 7f3091ffb700 1 -- 172.21.15.162:0/1921171230 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088019070 con 0x7f30940f8500 2024-02-13T00:47:54.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.995+0000 7f309b1c2700 10 client.37577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:54.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.995+0000 7f309b1c2700 1 -- 172.21.15.162:0/1921171230 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3094116fc0 con 0x7f30940f8500 2024-02-13T00:47:54.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.995+0000 7f309b1c2700 1 librados: init done 2024-02-13T00:47:54.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.996+0000 7f309b1c2700 1 -- 172.21.15.162:0/1921171230 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30940475e0 con 0x7f30940f8500 2024-02-13T00:47:54.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.996+0000 7f3091ffb700 1 -- 172.21.15.162:0/1921171230 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f308801e420 con 0x7f30940f8500 2024-02-13T00:47:54.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.996+0000 7f3091ffb700 1 --2- 172.21.15.162:0/1921171230 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3084062c90 0x7f3084065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:54.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.997+0000 7f3093fff700 1 --2- 172.21.15.162:0/1921171230 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3084062c90 0x7f3084065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.997+0000 7f3091ffb700 1 -- 172.21.15.162:0/1921171230 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(471..471 src has 52..471) v4 ==== 7967+0+0 (secure 0 0 0) 0x7f308807dc70 con 0x7f30940f8500 2024-02-13T00:47:54.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.997+0000 7f3091ffb700 10 client.37577.objecter ms_dispatch 0x7f3094001550 osd_map(471..471 src has 52..471) v4 2024-02-13T00:47:54.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.997+0000 7f3091ffb700 3 client.37577.objecter handle_osd_map got epochs [471,471] > 0 2024-02-13T00:47:54.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.997+0000 7f3091ffb700 3 client.37577.objecter handle_osd_map decoding full epoch 471 2024-02-13T00:47:54.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.997+0000 7f3091ffb700 20 client.37577.objecter dump_active .. 0 homeless 2024-02-13T00:47:54.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.997+0000 7f3091ffb700 10 client.37577.objecter ms_handle_connect 0x7f3084062c90 2024-02-13T00:47:54.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:53.997+0000 7f3093fff700 1 --2- 172.21.15.162:0/1921171230 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3084062c90 0x7f3084065150 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7f309406c910 tx=0x7f3080009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:54.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.001+0000 7f3091ffb700 1 -- 172.21.15.162:0/1921171230 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3088051280 con 0x7f30940f8500 2024-02-13T00:47:54.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:54.124 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:54.144 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:54.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:54.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:54.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.178+0000 7fd56800e700 1 -- 172.21.15.162:0/3382902863 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd56006bca0 msgr2=0x7fd560104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:54.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.178+0000 7fd56800e700 1 --2- 172.21.15.162:0/3382902863 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd56006bca0 0x7fd560104030 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 rx=0x7fd5500047b0 tx=0x7fd550009cb0).stop 2024-02-13T00:47:54.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.178+0000 7fd56800e700 1 -- 172.21.15.162:0/3382902863 shutdown_connections 2024-02-13T00:47:54.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.178+0000 7fd56800e700 1 --2- 172.21.15.162:0/3382902863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd560104520 0x7fd560106910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:54.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.178+0000 7fd56800e700 1 --2- 172.21.15.162:0/3382902863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd56006b3d0 0x7fd56006b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:47:54.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.178+0000 7fd56800e700 1 --2- 172.21.15.162:0/3382902863 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd56006bca0 0x7fd560104030 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:54.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.178+0000 7fd56800e700 1 -- 172.21.15.162:0/3382902863 >> 172.21.15.162:0/3382902863 conn(0x7fd5600f38c0 msgr2=0x7fd5600f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:54.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 -- 172.21.15.162:0/3382902863 shutdown_connections 2024-02-13T00:47:54.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 -- 172.21.15.162:0/3382902863 wait complete. 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 librados: starting msgr at 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 librados: starting objecter 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 Processor -- start 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 -- start start 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 librados: setting wanted keys 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 librados: calling monclient init 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd56006b3d0 0x7fd5601105e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd56006bca0 0x7fd560110af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd560104520 0x7fd56010a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56010acb0 con 0x7fd56006b3d0 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56010ae20 con 0x7fd560104520 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd56800e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56010afe0 con 0x7fd56006bca0 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd5655a9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd56006bca0 0x7fd560110af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd5655a9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd56006bca0 0x7fd560110af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:39372/0 (socket says 172.21.15.162:39372) 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd5655a9700 1 -- 172.21.15.162:0/2567450119 learned_addr learned my addr 172.21.15.162:0/2567450119 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.179+0000 7fd565daa700 1 --2- 172.21.15.162:0/2567450119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd56006b3d0 0x7fd5601105e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd5665ab700 1 --2- 172.21.15.162:0/2567450119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd560104520 0x7fd56010a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd5655a9700 1 -- 172.21.15.162:0/2567450119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd560104520 msgr2=0x7fd56010a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd5655a9700 1 --2- 172.21.15.162:0/2567450119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd560104520 0x7fd56010a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd5655a9700 1 -- 172.21.15.162:0/2567450119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd56006b3d0 msgr2=0x7fd5601105e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd5655a9700 1 --2- 172.21.15.162:0/2567450119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd56006b3d0 0x7fd5601105e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd5655a9700 1 -- 172.21.15.162:0/2567450119 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5580097c0 con 0x7fd56006bca0 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd5665ab700 1 --2- 172.21.15.162:0/2567450119 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd560104520 0x7fd56010a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd5655a9700 1 --2- 172.21.15.162:0/2567450119 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd56006bca0 0x7fd560110af0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 rx=0x7fd550009b10 tx=0x7fd550009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd556ffd700 10 client.?.objecter ms_handle_connect 0x7fd56006bca0 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd556ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550018070 con 0x7fd56006bca0 2024-02-13T00:47:54.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd550005040 con 0x7fd56006bca0 2024-02-13T00:47:54.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd550004180 con 0x7fd56006bca0 2024-02-13T00:47:54.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55001c3d0 con 0x7fd56006bca0 2024-02-13T00:47:54.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd56800e700 10 client.37583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:54.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56010b560 con 0x7fd56006bca0 2024-02-13T00:47:54.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.180+0000 7fd56800e700 1 librados: init done 2024-02-13T00:47:54.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.181+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd5600475e0 con 0x7fd56006bca0 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.181+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fd55001d500 con 0x7fd56006bca0 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd556ffd700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd54c062c90 0x7fd54c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(471..471 src has 52..471) v4 ==== 7967+0+0 (secure 0 0 0) 0x7fd55007e070 con 0x7fd56006bca0 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 osd_map(471..471 src has 52..471) v4 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd556ffd700 3 client.37583.objecter handle_osd_map got epochs [471,471] > 0 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd556ffd700 3 client.37583.objecter handle_osd_map decoding full epoch 471 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd565daa700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd54c062c90 0x7fd54c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd556ffd700 20 client.37583.objecter dump_active .. 0 homeless 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v471) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd550085070 con 0x7fd56006bca0 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd56800e700 10 client.37583.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 1 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd56800e700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd560028590 0x7fd5600fdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:54.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd5601102b0 osd=0 3 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd5601102b0 osd=0 3 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd5601102b0 osd=0 3 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 1 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd5601102b0 osd=0 3 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 0 1 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd5601102b0 osd=0 4 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd56800e700 10 client.37583.objecter _send_command 1 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5601094b0 con 0x7fd560028590 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd556ffd700 10 client.37583.objecter ms_handle_connect 0x7fd54c062c90 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd565daa700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd54c062c90 0x7fd54c065150 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7fd560105fa0 tx=0x7fd55c009060).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.182+0000 7fd5665ab700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd560028590 0x7fd5600fdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.183+0000 7fd556ffd700 10 client.37583.objecter ms_handle_connect 0x7fd560028590 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.183+0000 7fd5665ab700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd560028590 0x7fd5600fdfe0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:54.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.183+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd5601094b0 con 0x7fd560028590 2024-02-13T00:47:54.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.183+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:47:54.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.183+0000 7fd556ffd700 10 client.37583.objecter _finish_command 1 = system:0 2024-02-13T00:47:54.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.183+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd5601102b0 osd=0 4 2024-02-13T00:47:54.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.183+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 0 1 2024-02-13T00:47:54.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.183+0000 7f309b1c2700 1 -- 172.21.15.162:0/1921171230 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f309406c650 con 0x7f30940f8500 2024-02-13T00:47:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.198+0000 7fd56800e700 10 client.37583.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.198+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 2 2024-02-13T00:47:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.198+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd5601102b0 osd=0 3 2024-02-13T00:47:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.198+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd5601102b0 osd=0 3 2024-02-13T00:47:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.198+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd5601102b0 osd=0 3 2024-02-13T00:47:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.198+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 2 2024-02-13T00:47:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.198+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd5601102b0 osd=0 3 2024-02-13T00:47:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.198+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 0 2 2024-02-13T00:47:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.198+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd5601102b0 osd=0 4 2024-02-13T00:47:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.198+0000 7fd56800e700 10 client.37583.objecter _send_command 2 2024-02-13T00:47:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.198+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd56005a0e0 con 0x7fd560028590 2024-02-13T00:47:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd56005a0e0 con 0x7fd560028590 2024-02-13T00:47:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd556ffd700 10 client.37583.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd5601102b0 osd=0 4 2024-02-13T00:47:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 0 2 2024-02-13T00:47:54.204 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:47:54.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd56800e700 10 client.37583.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:54.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 3 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd56800e700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6805/150599780 conn(0x7fd560027be0 0x7fd560027fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd5600ff650 osd=1 3 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd5600ff650 osd=1 3 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd5600ff650 osd=1 3 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 3 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd5600ff650 osd=1 3 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 1 3 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd5600ff650 osd=1 4 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd56800e700 10 client.37583.objecter _send_command 3 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5600ff7c0 con 0x7fd560027be0 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd565daa700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6805/150599780 conn(0x7fd560027be0 0x7fd560027fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd556ffd700 10 client.37583.objecter ms_handle_connect 0x7fd560027be0 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.200+0000 7fd565daa700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6805/150599780 conn(0x7fd560027be0 0x7fd560027fc0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:54.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.201+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd5600ff7c0 con 0x7fd560027be0 2024-02-13T00:47:54.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.201+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:47:54.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.201+0000 7fd556ffd700 10 client.37583.objecter _finish_command 3 = system:0 2024-02-13T00:47:54.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.201+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd5600ff650 osd=1 4 2024-02-13T00:47:54.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.201+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 1 3 2024-02-13T00:47:54.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.216+0000 7fd56800e700 10 client.37583.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:54.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.216+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 4 2024-02-13T00:47:54.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.216+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd5600ff650 osd=1 3 2024-02-13T00:47:54.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.216+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd5600ff650 osd=1 3 2024-02-13T00:47:54.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.216+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd5600ff650 osd=1 3 2024-02-13T00:47:54.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.216+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 4 2024-02-13T00:47:54.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.216+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd5600ff650 osd=1 3 2024-02-13T00:47:54.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.216+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 1 4 2024-02-13T00:47:54.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.216+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd5600ff650 osd=1 4 2024-02-13T00:47:54.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.216+0000 7fd56800e700 10 client.37583.objecter _send_command 4 2024-02-13T00:47:54.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.216+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd560027650 con 0x7fd560027be0 2024-02-13T00:47:54.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd560027650 con 0x7fd560027be0 2024-02-13T00:47:54.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:54.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd556ffd700 10 client.37583.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd5600ff650 osd=1 4 2024-02-13T00:47:54.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 1 4 2024-02-13T00:47:54.221 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:47:54.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd56800e700 10 client.37583.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:54.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 5 2024-02-13T00:47:54.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd56800e700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd5600ff1b0 0x7fd56011e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:54.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560027a60 osd=2 3 2024-02-13T00:47:54.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560027a60 osd=2 3 2024-02-13T00:47:54.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560027a60 osd=2 3 2024-02-13T00:47:54.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 5 2024-02-13T00:47:54.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd560027a60 osd=2 3 2024-02-13T00:47:54.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 2 5 2024-02-13T00:47:54.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560027a60 osd=2 4 2024-02-13T00:47:54.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd56800e700 10 client.37583.objecter _send_command 5 2024-02-13T00:47:54.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5601211e0 con 0x7fd5600ff1b0 2024-02-13T00:47:54.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.217+0000 7fd5655a9700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd5600ff1b0 0x7fd56011e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.218+0000 7fd556ffd700 10 client.37583.objecter ms_handle_connect 0x7fd5600ff1b0 2024-02-13T00:47:54.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.218+0000 7fd5655a9700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd5600ff1b0 0x7fd56011e590 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:54.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.218+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd5601211e0 con 0x7fd5600ff1b0 2024-02-13T00:47:54.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.218+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:47:54.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.218+0000 7fd556ffd700 10 client.37583.objecter _finish_command 5 = system:0 2024-02-13T00:47:54.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.218+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd560027a60 osd=2 4 2024-02-13T00:47:54.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.218+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 2 5 2024-02-13T00:47:54.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.233+0000 7fd56800e700 10 client.37583.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:54.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.233+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 6 2024-02-13T00:47:54.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.233+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560027a60 osd=2 3 2024-02-13T00:47:54.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.233+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560027a60 osd=2 3 2024-02-13T00:47:54.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.233+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560027a60 osd=2 3 2024-02-13T00:47:54.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.233+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 6 2024-02-13T00:47:54.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.233+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd560027a60 osd=2 3 2024-02-13T00:47:54.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.233+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 2 6 2024-02-13T00:47:54.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.233+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560027a60 osd=2 4 2024-02-13T00:47:54.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.233+0000 7fd56800e700 10 client.37583.objecter _send_command 6 2024-02-13T00:47:54.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.233+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd5600fe7d0 con 0x7fd5600ff1b0 2024-02-13T00:47:54.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd5600fe7d0 con 0x7fd5600ff1b0 2024-02-13T00:47:54.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:54.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd556ffd700 10 client.37583.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd560027a60 osd=2 4 2024-02-13T00:47:54.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 2 6 2024-02-13T00:47:54.238 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:47:54.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd56800e700 10 client.37583.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:54.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 7 2024-02-13T00:47:54.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd56800e700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6813/604971011 conn(0x7fd56011fc00 0x7fd560120040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:54.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd56011bbd0 osd=3 3 2024-02-13T00:47:54.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd56011bbd0 osd=3 3 2024-02-13T00:47:54.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd56011bbd0 osd=3 3 2024-02-13T00:47:54.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 7 2024-02-13T00:47:54.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd56011bbd0 osd=3 3 2024-02-13T00:47:54.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 3 7 2024-02-13T00:47:54.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd56011bbd0 osd=3 4 2024-02-13T00:47:54.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd56800e700 10 client.37583.objecter _send_command 7 2024-02-13T00:47:54.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.234+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd56011c2c0 con 0x7fd56011fc00 2024-02-13T00:47:54.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.235+0000 7fd5665ab700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6813/604971011 conn(0x7fd56011fc00 0x7fd560120040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.235+0000 7fd556ffd700 10 client.37583.objecter ms_handle_connect 0x7fd56011fc00 2024-02-13T00:47:54.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.235+0000 7fd5665ab700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.162:6813/604971011 conn(0x7fd56011fc00 0x7fd560120040 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:54.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.235+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd56011c2c0 con 0x7fd56011fc00 2024-02-13T00:47:54.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.235+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:47:54.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.235+0000 7fd556ffd700 10 client.37583.objecter _finish_command 7 = system:0 2024-02-13T00:47:54.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.235+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd56011bbd0 osd=3 4 2024-02-13T00:47:54.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.235+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 3 7 2024-02-13T00:47:54.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:54.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.250+0000 7fd56800e700 10 client.37583.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:54.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.250+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 8 2024-02-13T00:47:54.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.250+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd56011bbd0 osd=3 3 2024-02-13T00:47:54.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.250+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd56011bbd0 osd=3 3 2024-02-13T00:47:54.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.250+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd56011bbd0 osd=3 3 2024-02-13T00:47:54.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.250+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 8 2024-02-13T00:47:54.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.250+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd56011bbd0 osd=3 3 2024-02-13T00:47:54.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.250+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 3 8 2024-02-13T00:47:54.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.250+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd56011bbd0 osd=3 4 2024-02-13T00:47:54.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.250+0000 7fd56800e700 10 client.37583.objecter _send_command 8 2024-02-13T00:47:54.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.250+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd560120d00 con 0x7fd56011fc00 2024-02-13T00:47:54.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd560120d00 con 0x7fd56011fc00 2024-02-13T00:47:54.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:54.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd556ffd700 10 client.37583.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd56011bbd0 osd=3 4 2024-02-13T00:47:54.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 3 8 2024-02-13T00:47:54.256 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:47:54.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd56800e700 10 client.37583.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 9 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd56800e700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd560123360 0x7fd560125750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560120d00 osd=4 3 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560120d00 osd=4 3 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560120d00 osd=4 3 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 9 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd560120d00 osd=4 3 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 4 9 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560120d00 osd=4 4 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd56800e700 10 client.37583.objecter _send_command 9 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd560120f70 con 0x7fd560123360 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.252+0000 7fd565daa700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd560123360 0x7fd560125750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.253+0000 7fd556ffd700 10 client.37583.objecter ms_handle_connect 0x7fd560123360 2024-02-13T00:47:54.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.253+0000 7fd565daa700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd560123360 0x7fd560125750 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:54.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.253+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd560120f70 con 0x7fd560123360 2024-02-13T00:47:54.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.253+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:47:54.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.253+0000 7fd556ffd700 10 client.37583.objecter _finish_command 9 = system:0 2024-02-13T00:47:54.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.253+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd560120d00 osd=4 4 2024-02-13T00:47:54.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.253+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 4 9 2024-02-13T00:47:54.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.269+0000 7fd56800e700 10 client.37583.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:54.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.269+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 10 2024-02-13T00:47:54.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.269+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560120d00 osd=4 3 2024-02-13T00:47:54.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.269+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560120d00 osd=4 3 2024-02-13T00:47:54.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.269+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560120d00 osd=4 3 2024-02-13T00:47:54.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.269+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 10 2024-02-13T00:47:54.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.269+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd560120d00 osd=4 3 2024-02-13T00:47:54.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.269+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 4 10 2024-02-13T00:47:54.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.269+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560120d00 osd=4 4 2024-02-13T00:47:54.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.269+0000 7fd56800e700 10 client.37583.objecter _send_command 10 2024-02-13T00:47:54.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.269+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd560120f70 con 0x7fd560123360 2024-02-13T00:47:54.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.271+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd560120f70 con 0x7fd560123360 2024-02-13T00:47:54.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.271+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:54.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.271+0000 7fd556ffd700 10 client.37583.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.271+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd560120d00 osd=4 4 2024-02-13T00:47:54.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.271+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 4 10 2024-02-13T00:47:54.275 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:47:54.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 10 client.37583.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:54.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 11 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 20 client.37583.objecter _get_session osd=-1 returning homeless 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 10 client.37583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=472}) v3 -- 0x7fd560128430 con 0x7fd56006bca0 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd560128660 con 0x7fd56006bca0 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_get_version_reply(handle=1 version=471) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5500513b0 con 0x7fd56006bca0 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd557fff700 10 client.37583.objecter _check_command_map_dne tid 11 current 471 map_dne_bound 471 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd557fff700 10 client.37583.objecter _finish_command 11 = osd:6 osd down 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd557fff700 15 client.37583.objecter _session_command_op_remove -1 11 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 10 client.37583.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 12 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd5601208e0 0x7fd560126e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560120770 osd=6 3 2024-02-13T00:47:54.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560120770 osd=6 3 2024-02-13T00:47:54.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560120770 osd=6 3 2024-02-13T00:47:54.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 12 2024-02-13T00:47:54.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd560120770 osd=6 3 2024-02-13T00:47:54.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 6 12 2024-02-13T00:47:54.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560120770 osd=6 4 2024-02-13T00:47:54.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 10 client.37583.objecter _send_command 12 2024-02-13T00:47:54.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.272+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.181:6808/1506640341 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd56011f8f0 con 0x7fd5601208e0 2024-02-13T00:47:54.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.273+0000 7fd5655a9700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd5601208e0 0x7fd560126e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.273+0000 7fd556ffd700 10 client.37583.objecter ms_handle_connect 0x7fd5601208e0 2024-02-13T00:47:54.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.273+0000 7fd5655a9700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd5601208e0 0x7fd560126e20 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:54.279 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:54.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.274+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd56011f8f0 con 0x7fd5601208e0 2024-02-13T00:47:54.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.274+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 12: 0 ) v1 2024-02-13T00:47:54.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.274+0000 7fd556ffd700 10 client.37583.objecter _finish_command 12 = system:0 2024-02-13T00:47:54.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.274+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd560120770 osd=6 4 2024-02-13T00:47:54.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.274+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 6 12 2024-02-13T00:47:54.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.289+0000 7fd56800e700 10 client.37583.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:54.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.289+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 13 2024-02-13T00:47:54.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.289+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560120770 osd=6 3 2024-02-13T00:47:54.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.289+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560120770 osd=6 3 2024-02-13T00:47:54.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.289+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560120770 osd=6 3 2024-02-13T00:47:54.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.289+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 13 2024-02-13T00:47:54.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.289+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd560120770 osd=6 3 2024-02-13T00:47:54.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.289+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 6 13 2024-02-13T00:47:54.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.289+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560120770 osd=6 4 2024-02-13T00:47:54.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.289+0000 7fd56800e700 10 client.37583.objecter _send_command 13 2024-02-13T00:47:54.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.289+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd56011f8f0 con 0x7fd5601208e0 2024-02-13T00:47:54.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.291+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd56011f8f0 con 0x7fd5601208e0 2024-02-13T00:47:54.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.291+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:54.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.291+0000 7fd556ffd700 10 client.37583.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.291+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd560120770 osd=6 4 2024-02-13T00:47:54.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.291+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 6 13 2024-02-13T00:47:54.295 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:47:54.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd56800e700 10 client.37583.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:47:54.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 14 2024-02-13T00:47:54.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd56800e700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd560127990 0x7fd560127df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560127310 osd=7 3 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560127310 osd=7 3 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560127310 osd=7 3 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 14 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd560127310 osd=7 3 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 7 14 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560127310 osd=7 4 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd56800e700 10 client.37583.objecter _send_command 14 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.181:6812/1964192772 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd56012ef50 con 0x7fd560127990 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd5665ab700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd560127990 0x7fd560127df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd5665ab700 1 --2- 172.21.15.162:0/2567450119 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd560127990 0x7fd560127df0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.292+0000 7fd556ffd700 10 client.37583.objecter ms_handle_connect 0x7fd560127990 2024-02-13T00:47:54.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.293+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd56012ef50 con 0x7fd560127990 2024-02-13T00:47:54.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.293+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 14: 0 ) v1 2024-02-13T00:47:54.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.293+0000 7fd556ffd700 10 client.37583.objecter _finish_command 14 = system:0 2024-02-13T00:47:54.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.293+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd560127310 osd=7 4 2024-02-13T00:47:54.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.293+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 7 14 2024-02-13T00:47:54.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.308+0000 7fd56800e700 10 client.37583.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:47:54.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.308+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign -1 15 2024-02-13T00:47:54.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.308+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560127310 osd=7 3 2024-02-13T00:47:54.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.308+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560127310 osd=7 3 2024-02-13T00:47:54.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.308+0000 7fd56800e700 20 client.37583.objecter _get_session s=0x7fd560127310 osd=7 3 2024-02-13T00:47:54.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.308+0000 7fd56800e700 15 client.37583.objecter _session_command_op_remove -1 15 2024-02-13T00:47:54.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.308+0000 7fd56800e700 20 client.37583.objecter get_session s=0x7fd560127310 osd=7 3 2024-02-13T00:47:54.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.308+0000 7fd56800e700 15 client.37583.objecter _session_command_op_assign 7 15 2024-02-13T00:47:54.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.308+0000 7fd56800e700 20 client.37583.objecter put_session s=0x7fd560127310 osd=7 4 2024-02-13T00:47:54.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.308+0000 7fd56800e700 10 client.37583.objecter _send_command 15 2024-02-13T00:47:54.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.308+0000 7fd56800e700 1 -- 172.21.15.162:0/2567450119 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd56012c7f0 con 0x7fd560127990 2024-02-13T00:47:54.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.310+0000 7fd556ffd700 1 -- 172.21.15.162:0/2567450119 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd56012c7f0 con 0x7fd560127990 2024-02-13T00:47:54.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.310+0000 7fd556ffd700 10 client.37583.objecter ms_dispatch 0x7fd560001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:47:54.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.310+0000 7fd556ffd700 10 client.37583.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.310+0000 7fd556ffd700 20 client.37583.objecter put_session s=0x7fd560127310 osd=7 4 2024-02-13T00:47:54.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.310+0000 7fd556ffd700 15 client.37583.objecter _session_command_op_remove 7 15 2024-02-13T00:47:54.314 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:47:54.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.314 INFO:teuthology.orchestra.run.smithi162.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-02-13T00:47:54.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.314 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:47:54.401 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:54.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:54.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:54.532 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:54.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.590+0000 7f3091ffb700 1 -- 172.21.15.162:0/1921171230 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v472) v1 ==== 87+0+0 (secure 0 0 0) 0x7f3088050e70 con 0x7f30940f8500 2024-02-13T00:47:54.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 10 librados: watch_flush enter 2024-02-13T00:47:54.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 10 librados: watch_flush exit 2024-02-13T00:47:54.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 20 client.37577.objecter shutdown clearing up homeless session... 2024-02-13T00:47:54.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 10 client.37577.objecter successfully canceled tick 2024-02-13T00:47:54.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 -- 172.21.15.162:0/1921171230 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3084062c90 msgr2=0x7f3084065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:54.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 --2- 172.21.15.162:0/1921171230 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3084062c90 0x7f3084065150 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7f309406c910 tx=0x7f3080009040).stop 2024-02-13T00:47:54.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 -- 172.21.15.162:0/1921171230 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30940f8500 msgr2=0x7f309406b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:47:54.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 --2- 172.21.15.162:0/1921171230 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30940f8500 0x7f309406b170 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7f3088009eb0 tx=0x7f308800b040).stop 2024-02-13T00:47:54.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 -- 172.21.15.162:0/1921171230 shutdown_connections 2024-02-13T00:47:54.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 --2- 172.21.15.162:0/1921171230 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3084062c90 0x7f3084065150 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:54.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 --2- 172.21.15.162:0/1921171230 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30940fa440 0x7f309406bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:54.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 --2- 172.21.15.162:0/1921171230 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30940f8500 0x7f309406b170 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:54.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 --2- 172.21.15.162:0/1921171230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30940f9b70 0x7f309406b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:47:54.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 -- 172.21.15.162:0/1921171230 >> 172.21.15.162:0/1921171230 conn(0x7f30940f38c0 msgr2=0x7f30940fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:47:54.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 -- 172.21.15.162:0/1921171230 shutdown_connections 2024-02-13T00:47:54.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 -- 172.21.15.162:0/1921171230 wait complete. 2024-02-13T00:47:54.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:47:54.593+0000 7f309b1c2700 1 librados: shutdown 2024-02-13T00:47:54.599 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:47:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:53.574613+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e471: 8 total, 7 up, 4 in 2024-02-13T00:47:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:53.823085+0000 osd.1 (osd.1) 201 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:47:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:53.825966+0000 osd.1 (osd.1) 202 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:47:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:54.182717+0000 mon.c (mon.2) 392 : audit [DBG] from='client.? 172.21.15.162:0/2567450119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:54.184848+0000 mon.c (mon.2) 393 : audit [INF] from='client.? 172.21.15.162:0/1921171230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:47:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:54.185110+0000 mon.a (mon.0) 2644 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:47:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:53.574613+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e471: 8 total, 7 up, 4 in 2024-02-13T00:47:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:53.823085+0000 osd.1 (osd.1) 201 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:47:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:53.825966+0000 osd.1 (osd.1) 202 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:47:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:54.182717+0000 mon.c (mon.2) 392 : audit [DBG] from='client.? 172.21.15.162:0/2567450119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:54.184848+0000 mon.c (mon.2) 393 : audit [INF] from='client.? 172.21.15.162:0/1921171230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:47:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:54.185110+0000 mon.a (mon.0) 2644 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:47:54.831 INFO:journalctl@ceph.osd.1.smithi162.stdout:Feb 13 00:47:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-1[54323]: debug 2024-02-13T00:47:54.620+0000 7fe449bf1700 -1 snap_mapper.add_oid found existing snaps mapped on 61:56e7e5bd:test-rados-api-smithi162-84057-74::foo:24, removing 2024-02-13T00:47:54.980 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:54.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:54.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:55.045 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:47:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:47:54.620+0000 7f54b5f70700 -1 snap_mapper.add_oid found existing snaps mapped on 61:56e7e5bd:test-rados-api-smithi162-84057-74::foo:24, removing 2024-02-13T00:47:55.045 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:47:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:47:54.620+0000 7f8a5bd7d700 -1 snap_mapper.add_oid found existing snaps mapped on 61:56e7e5bd:test-rados-api-smithi162-84057-74::foo:24, removing 2024-02-13T00:47:55.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:53.574613+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e471: 8 total, 7 up, 4 in 2024-02-13T00:47:55.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:53.823085+0000 osd.1 (osd.1) 201 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:47:55.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:53.825966+0000 osd.1 (osd.1) 202 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:47:55.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:54.182717+0000 mon.c (mon.2) 392 : audit [DBG] from='client.? 172.21.15.162:0/2567450119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:47:55.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:54.184848+0000 mon.c (mon.2) 393 : audit [INF] from='client.? 172.21.15.162:0/1921171230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:47:55.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:54.185110+0000 mon.a (mon.0) 2644 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:47:55.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:47:55.113 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-13T00:47:55.420 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-13T00:47:55.685 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6] out_osds: [3, 2, 7, 5] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-02-13T00:47:55.685 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:47:55.685 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-02-13T00:47:55.685 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-02-13T00:47:55.685 DEBUG:teuthology.orchestra.run.smithi181:osd.5> sudo journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.5.service 2024-02-13T00:47:55.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl start ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.5 2024-02-13T00:47:55.721 INFO:journalctl@ceph.osd.5.smithi181.stdout:-- Logs begin at Tue 2024-02-13 00:17:30 UTC. -- 2024-02-13T00:47:56.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:54.446414+0000 mgr.y (mgr.14544) 421 : cluster [DBG] pgmap v701: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:56.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:54.580051+0000 mon.a (mon.0) 2645 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:47:56.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:54.588473+0000 mon.a (mon.0) 2646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:47:56.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:54.588524+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e472: 8 total, 7 up, 4 in 2024-02-13T00:47:56.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:54.635109+0000 mon.b (mon.1) 603 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:56.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:54.635539+0000 mon.a (mon.0) 2648 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:56.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:54.636012+0000 mon.b (mon.1) 604 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-73"}]: dispatch 2024-02-13T00:47:56.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:54.636268+0000 mon.a (mon.0) 2649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-73"}]: dispatch 2024-02-13T00:47:56.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:55.446446+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e473: 8 total, 7 up, 4 in 2024-02-13T00:47:56.046 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:47:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:47:56.046 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:55 smithi181 systemd[1]: Starting Ceph osd.5 for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:47:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:54.446414+0000 mgr.y (mgr.14544) 421 : cluster [DBG] pgmap v701: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:54.580051+0000 mon.a (mon.0) 2645 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:47:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:54.588473+0000 mon.a (mon.0) 2646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:47:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:54.588524+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e472: 8 total, 7 up, 4 in 2024-02-13T00:47:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:54.635109+0000 mon.b (mon.1) 603 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:54.635539+0000 mon.a (mon.0) 2648 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:54.636012+0000 mon.b (mon.1) 604 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-73"}]: dispatch 2024-02-13T00:47:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:54.636268+0000 mon.a (mon.0) 2649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-73"}]: dispatch 2024-02-13T00:47:56.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:55.446446+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e473: 8 total, 7 up, 4 in 2024-02-13T00:47:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:54.446414+0000 mgr.y (mgr.14544) 421 : cluster [DBG] pgmap v701: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:47:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:54.580051+0000 mon.a (mon.0) 2645 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:47:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:54.588473+0000 mon.a (mon.0) 2646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:47:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:54.588524+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e472: 8 total, 7 up, 4 in 2024-02-13T00:47:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:54.635109+0000 mon.b (mon.1) 603 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:54.635539+0000 mon.a (mon.0) 2648 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:47:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:54.636012+0000 mon.b (mon.1) 604 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-73"}]: dispatch 2024-02-13T00:47:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:54.636268+0000 mon.a (mon.0) 2649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-73"}]: dispatch 2024-02-13T00:47:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:55.446446+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e473: 8 total, 7 up, 4 in 2024-02-13T00:47:56.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:47:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:47:56] "GET /metrics HTTP/1.1" 200 177759 "" "Prometheus/2.33.4" 2024-02-13T00:47:56.993 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:47:56.995 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:55.675390+0000 mgr.y (mgr.14544) 422 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:56.995 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:56.454606+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e474: 8 total, 7 up, 4 in 2024-02-13T00:47:56.995 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:56.464555+0000 mon.b (mon.1) 605 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:56.995 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:56.465065+0000 mon.a (mon.0) 2652 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:56.995 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:55.675390+0000 mgr.y (mgr.14544) 422 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:56.995 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:56.454606+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e474: 8 total, 7 up, 4 in 2024-02-13T00:47:56.995 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:56.464555+0000 mon.b (mon.1) 605 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:56.996 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:56.465065+0000 mon.a (mon.0) 2652 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:56.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:47:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:55.675390+0000 mgr.y (mgr.14544) 422 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:47:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:56.454606+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e474: 8 total, 7 up, 4 in 2024-02-13T00:47:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:56.464555+0000 mon.b (mon.1) 605 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:56.465065+0000 mon.a (mon.0) 2652 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:47:57.046 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:56 smithi181 bash[93036]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-13T00:47:57.046 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:56 smithi181 bash[93036]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-13T00:47:57.046 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:56 smithi181 bash[93036]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2024-02-13T00:47:57.046 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:56 smithi181 bash[93036]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-13T00:47:57.046 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:56 smithi181 bash[93036]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2024-02-13T00:47:57.046 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:56 smithi181 bash[93036]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-13T00:47:57.046 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:56 smithi181 bash[93036]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-13T00:47:57.046 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:56 smithi181 bash[93036]: --> ceph-volume lvm activate successful for osd ID: 5 2024-02-13T00:47:57.098 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:47:57.130 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:57.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:47:57.482 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:47:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:57.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:47:57.522 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:47:57.522 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:47:57.522 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-13T00:47:57.522 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:47:57.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:57.613 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:57.761 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:57.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:57.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:57.790 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:56.447034+0000 mgr.y (mgr.14544) 423 : cluster [DBG] pgmap v704: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-13T00:47:57.790 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:57.458846+0000 mon.a (mon.0) 2653 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:57.790 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:57.458913+0000 mon.a (mon.0) 2654 : cluster [DBG] osdmap e475: 8 total, 7 up, 4 in 2024-02-13T00:47:57.790 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:57.470043+0000 mon.b (mon.1) 606 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:57.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:47:57.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:47:57.919 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:57.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-13T00:47:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:56.447034+0000 mgr.y (mgr.14544) 423 : cluster [DBG] pgmap v704: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-13T00:47:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:57.458846+0000 mon.a (mon.0) 2653 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:57.458913+0000 mon.a (mon.0) 2654 : cluster [DBG] osdmap e475: 8 total, 7 up, 4 in 2024-02-13T00:47:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:57.470043+0000 mon.b (mon.1) 606 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:58.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:56.447034+0000 mgr.y (mgr.14544) 423 : cluster [DBG] pgmap v704: 206 pgs: 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-13T00:47:58.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:57.458846+0000 mon.a (mon.0) 2653 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:47:58.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:57.458913+0000 mon.a (mon.0) 2654 : cluster [DBG] osdmap e475: 8 total, 7 up, 4 in 2024-02-13T00:47:58.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:57.470043+0000 mon.b (mon.1) 606 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:47:58.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-13T00:47:58.226 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:58.231 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:47:58.295 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:47:57.908+0000 7ff8c2e55700 1 -- 172.21.15.181:0/4176451377 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564c167ac680 con 0x564c16773400 2024-02-13T00:47:58.295 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:57 smithi181 bash[93314]: 620d70802aaea84ba3e7a25432bad960e30943c3b21abf1c3ba2747277dbbf01 2024-02-13T00:47:58.295 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:57 smithi181 systemd[1]: Started Ceph osd.5 for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:47:58.478 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:47:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:58.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:47:58.584 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:47:58.612 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:58.752 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:47:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:47:58.434+0000 7ff8ca8e3380 -1 Falling back to public interface 2024-02-13T00:47:58.752 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:47:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:47:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:47:58.996 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:59.460 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:47:59.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:59.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:47:59.470 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:47:59.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:47:59.594 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:47:59.692 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:47:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:47:59.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:47:59.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:58.447505+0000 mgr.y (mgr.14544) 424 : cluster [DBG] pgmap v707: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-13T00:47:59.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:47:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:58.458188+0000 mon.a (mon.0) 2655 : cluster [DBG] osdmap e476: 8 total, 7 up, 4 in 2024-02-13T00:47:59.710 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:58.447505+0000 mgr.y (mgr.14544) 424 : cluster [DBG] pgmap v707: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-13T00:47:59.710 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:47:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:58.458188+0000 mon.a (mon.0) 2655 : cluster [DBG] osdmap e476: 8 total, 7 up, 4 in 2024-02-13T00:47:59.734 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:58.447505+0000 mgr.y (mgr.14544) 424 : cluster [DBG] pgmap v707: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-13T00:47:59.734 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:47:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:58.458188+0000 mon.a (mon.0) 2655 : cluster [DBG] osdmap e476: 8 total, 7 up, 4 in 2024-02-13T00:47:59.795 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:47:59.828 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:00.055 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-02-13T00:48:00.055 INFO:teuthology.orchestra.run.smithi181.stderr:0 2024-02-13T00:48:00.055 INFO:teuthology.orchestra.run.smithi181.stderr:1 2024-02-13T00:48:00.056 INFO:teuthology.orchestra.run.smithi181.stderr:2 2024-02-13T00:48:00.056 INFO:teuthology.orchestra.run.smithi181.stderr:abort 2024-02-13T00:48:00.056 INFO:teuthology.orchestra.run.smithi181.stderr:assert 2024-02-13T00:48:00.056 INFO:teuthology.orchestra.run.smithi181.stderr:config diff 2024-02-13T00:48:00.056 INFO:teuthology.orchestra.run.smithi181.stderr:config diff get 2024-02-13T00:48:00.056 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-02-13T00:48:00.056 INFO:teuthology.orchestra.run.smithi181.stderr:config help [] 2024-02-13T00:48:00.056 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-02-13T00:48:00.056 INFO:teuthology.orchestra.run.smithi181.stderr:admin_socket: invalid command 2024-02-13T00:48:00.276 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:00.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:00.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:00.409 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:00.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:59.459628+0000 mon.a (mon.0) 2656 : cluster [DBG] osdmap e477: 8 total, 7 up, 4 in 2024-02-13T00:48:00.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:59.481140+0000 mon.b (mon.1) 607 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1d"}]: dispatch 2024-02-13T00:48:00.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:59.481421+0000 mgr.y (mgr.14544) 425 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1d"}]: dispatch 2024-02-13T00:48:00.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:47:59.626865+0000 mon.c (mon.2) 394 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:48:00.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:59.898478+0000 osd.1 (osd.1) 203 : cluster [DBG] 219.1d deep-scrub starts 2024-02-13T00:48:00.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:47:59.902550+0000 osd.1 (osd.1) 204 : cluster [DBG] 219.1d deep-scrub ok 2024-02-13T00:48:00.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:59.459628+0000 mon.a (mon.0) 2656 : cluster [DBG] osdmap e477: 8 total, 7 up, 4 in 2024-02-13T00:48:00.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:59.481140+0000 mon.b (mon.1) 607 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1d"}]: dispatch 2024-02-13T00:48:00.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:59.481421+0000 mgr.y (mgr.14544) 425 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1d"}]: dispatch 2024-02-13T00:48:00.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:47:59.626865+0000 mon.c (mon.2) 394 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:48:00.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:59.898478+0000 osd.1 (osd.1) 203 : cluster [DBG] 219.1d deep-scrub starts 2024-02-13T00:48:00.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:47:59.902550+0000 osd.1 (osd.1) 204 : cluster [DBG] 219.1d deep-scrub ok 2024-02-13T00:48:00.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:59.459628+0000 mon.a (mon.0) 2656 : cluster [DBG] osdmap e477: 8 total, 7 up, 4 in 2024-02-13T00:48:00.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:59.481140+0000 mon.b (mon.1) 607 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1d"}]: dispatch 2024-02-13T00:48:00.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:59.481421+0000 mgr.y (mgr.14544) 425 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1d"}]: dispatch 2024-02-13T00:48:00.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:47:59.626865+0000 mon.c (mon.2) 394 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:48:00.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:59.898478+0000 osd.1 (osd.1) 203 : cluster [DBG] 219.1d deep-scrub starts 2024-02-13T00:48:00.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:47:59.902550+0000 osd.1 (osd.1) 204 : cluster [DBG] 219.1d deep-scrub ok 2024-02-13T00:48:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-13T00:48:00.774 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-02-13T00:48:00.858 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:00.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:00.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.524+0000 7f4421170700 1 -- 172.21.15.162:0/447312678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0fe5a0 msgr2=0x7f441c0f7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:00.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.524+0000 7f4421170700 1 --2- 172.21.15.162:0/447312678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0fe5a0 0x7f441c0f7dd0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 rx=0x7f4410004870 tx=0x7f4410009cb0).stop 2024-02-13T00:48:00.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.524+0000 7f4421170700 1 -- 172.21.15.162:0/447312678 shutdown_connections 2024-02-13T00:48:00.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.524+0000 7f4421170700 1 --2- 172.21.15.162:0/447312678 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f441c0f84b0 0x7f441c104ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:00.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.524+0000 7f4421170700 1 --2- 172.21.15.162:0/447312678 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0fdc40 0x7f441c0fe020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.524+0000 7f4421170700 1 --2- 172.21.15.162:0/447312678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0fe5a0 0x7f441c0f7dd0 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.524+0000 7f4421170700 1 -- 172.21.15.162:0/447312678 >> 172.21.15.162:0/447312678 conn(0x7f441c0f3850 msgr2=0x7f441c0f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.524+0000 7f4421170700 1 -- 172.21.15.162:0/447312678 shutdown_connections 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.524+0000 7f4421170700 1 -- 172.21.15.162:0/447312678 wait complete. 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.525+0000 7f4421170700 1 librados: starting msgr at 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.525+0000 7f4421170700 1 librados: starting objecter 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.525+0000 7f4421170700 1 Processor -- start 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f4421170700 1 -- start start 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f4421170700 1 librados: setting wanted keys 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f4421170700 1 librados: calling monclient init 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f4421170700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f441c0f84b0 0x7f441c10e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f4421170700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0fdc40 0x7f441c10e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f4421170700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0fe5a0 0x7f441c108300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f4421170700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c1087f0 con 0x7f441c0fdc40 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f4421170700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c108960 con 0x7f441c0f84b0 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f4421170700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c108b20 con 0x7f441c0fe5a0 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f441a59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0fdc40 0x7f441c10e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f441a59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0fdc40 0x7f441c10e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:57144/0 (socket says 172.21.15.162:57144) 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f441a59c700 1 -- 172.21.15.162:0/1296304111 learned_addr learned my addr 172.21.15.162:0/1296304111 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f441ad9d700 1 --2- 172.21.15.162:0/1296304111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f441c0f84b0 0x7f441c10e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f441b59e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0fe5a0 0x7f441c108300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f441a59c700 1 -- 172.21.15.162:0/1296304111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0fe5a0 msgr2=0x7f441c108300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f441a59c700 1 --2- 172.21.15.162:0/1296304111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0fe5a0 0x7f441c108300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f441a59c700 1 -- 172.21.15.162:0/1296304111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f441c0f84b0 msgr2=0x7f441c10e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f441a59c700 1 --2- 172.21.15.162:0/1296304111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f441c0f84b0 0x7f441c10e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f441a59c700 1 -- 172.21.15.162:0/1296304111 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440c005040 con 0x7f441c0fdc40 2024-02-13T00:48:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f441b59e700 1 --2- 172.21.15.162:0/1296304111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0fe5a0 0x7f441c108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f441a59c700 1 --2- 172.21.15.162:0/1296304111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0fdc40 0x7f441c10e650 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 rx=0x7f4410009eb0 tx=0x7f441000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f4403fff700 10 client.?.objecter ms_handle_connect 0x7f441c0fdc40 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.526+0000 7f4403fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.527+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441000e070 con 0x7f441c0fdc40 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.527+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4410013da0 con 0x7f441c0fdc40 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.527+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4410005040 con 0x7f441c0fdc40 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.527+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4410019070 con 0x7f441c0fdc40 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.527+0000 7f4421170700 10 client.27492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.527+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c109090 con 0x7f441c0fdc40 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.527+0000 7f4421170700 1 librados: init done 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.528+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f441c05fb00 con 0x7f441c0fdc40 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.528+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f441001e420 con 0x7f441c0fdc40 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4403fff700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4404062c90 0x7f4404065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:00.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(477..477 src has 52..477) v4 ==== 7967+0+0 (secure 0 0 0) 0x7f441007dba0 con 0x7f441c0fdc40 2024-02-13T00:48:00.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 osd_map(477..477 src has 52..477) v4 2024-02-13T00:48:00.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4403fff700 3 client.27492.objecter handle_osd_map got epochs [477,477] > 0 2024-02-13T00:48:00.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4403fff700 3 client.27492.objecter handle_osd_map decoding full epoch 477 2024-02-13T00:48:00.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f441ad9d700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4404062c90 0x7f4404065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:00.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4403fff700 20 client.27492.objecter dump_active .. 0 homeless 2024-02-13T00:48:00.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v477) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4410082220 con 0x7f441c0fdc40 2024-02-13T00:48:00.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f441ad9d700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4404062c90 0x7f4404065150 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 rx=0x7f440c009350 tx=0x7f440c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:00.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4403fff700 10 client.27492.objecter ms_handle_connect 0x7f4404062c90 2024-02-13T00:48:00.967 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:48:00.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4421170700 10 client.27492.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:00.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 1 2024-02-13T00:48:00.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4421170700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6801/2103743338 conn(0x7f441c028590 0x7f441c0fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:00.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c0475e0 osd=0 3 2024-02-13T00:48:00.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c0475e0 osd=0 3 2024-02-13T00:48:00.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c0475e0 osd=0 3 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 1 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c0475e0 osd=0 3 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 0 1 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c0475e0 osd=0 4 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4421170700 10 client.27492.objecter _send_command 1 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f441c1070f0 con 0x7f441c028590 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f441b59e700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6801/2103743338 conn(0x7f441c028590 0x7f441c0fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f4403fff700 10 client.27492.objecter ms_handle_connect 0x7f441c028590 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.529+0000 7f441b59e700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6801/2103743338 conn(0x7f441c028590 0x7f441c0fe9a0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.530+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f441c1070f0 con 0x7f441c028590 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.530+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.530+0000 7f4403fff700 10 client.27492.objecter _finish_command 1 = system:0 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.530+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c0475e0 osd=0 4 2024-02-13T00:48:00.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.530+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 0 1 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.544+0000 7f4421170700 10 client.27492.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.544+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 2 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.544+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c0475e0 osd=0 3 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.544+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c0475e0 osd=0 3 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.544+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c0475e0 osd=0 3 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.544+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 2 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.544+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c0475e0 osd=0 3 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.544+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 0 2 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.544+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c0475e0 osd=0 4 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.544+0000 7f4421170700 10 client.27492.objecter _send_command 2 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.544+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f441c05a0e0 con 0x7f441c028590 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f441c05a0e0 con 0x7f441c028590 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:00.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4403fff700 10 client.27492.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c0475e0 osd=0 4 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 0 2 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4421170700 10 client.27492.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 3 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4421170700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6805/150599780 conn(0x7f441c027be0 0x7f441c027fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c100010 osd=1 3 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c100010 osd=1 3 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c100010 osd=1 3 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 3 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c100010 osd=1 3 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 1 3 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c100010 osd=1 4 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4421170700 10 client.27492.objecter _send_command 3 2024-02-13T00:48:00.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f441c100180 con 0x7f441c027be0 2024-02-13T00:48:00.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.546+0000 7f441ad9d700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6805/150599780 conn(0x7f441c027be0 0x7f441c027fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:00.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.547+0000 7f4403fff700 10 client.27492.objecter ms_handle_connect 0x7f441c027be0 2024-02-13T00:48:00.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.547+0000 7f441ad9d700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6805/150599780 conn(0x7f441c027be0 0x7f441c027fc0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:00.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.547+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f441c100180 con 0x7f441c027be0 2024-02-13T00:48:00.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.547+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:48:00.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.547+0000 7f4403fff700 10 client.27492.objecter _finish_command 3 = system:0 2024-02-13T00:48:00.972 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.547+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c100010 osd=1 4 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.547+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 1 3 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.562+0000 7f4421170700 10 client.27492.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.562+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 4 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.562+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c100010 osd=1 3 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.562+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c100010 osd=1 3 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.562+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c100010 osd=1 3 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.562+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 4 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.562+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c100010 osd=1 3 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.562+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 1 4 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.562+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c100010 osd=1 4 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.562+0000 7f4421170700 10 client.27492.objecter _send_command 4 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.562+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f441c06cfc0 con 0x7f441c027be0 2024-02-13T00:48:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f441c06cfc0 con 0x7f441c027be0 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4403fff700 10 client.27492.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c100010 osd=1 4 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 1 4 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4421170700 10 client.27492.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 5 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4421170700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6809/3689719376 conn(0x7f441c0ffb70 0x7f441c11e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c027890 osd=2 3 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c027890 osd=2 3 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c027890 osd=2 3 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 5 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c027890 osd=2 3 2024-02-13T00:48:00.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 2 5 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c027890 osd=2 4 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4421170700 10 client.27492.objecter _send_command 5 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f441c121190 con 0x7f441c0ffb70 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.563+0000 7f441a59c700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6809/3689719376 conn(0x7f441c0ffb70 0x7f441c11e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.564+0000 7f4403fff700 10 client.27492.objecter ms_handle_connect 0x7f441c0ffb70 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.564+0000 7f441a59c700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6809/3689719376 conn(0x7f441c0ffb70 0x7f441c11e540 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.564+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f441c121190 con 0x7f441c0ffb70 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.564+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.564+0000 7f4403fff700 10 client.27492.objecter _finish_command 5 = system:0 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.564+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c027890 osd=2 4 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.564+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 2 5 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.579+0000 7f4421170700 10 client.27492.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.579+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 6 2024-02-13T00:48:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.579+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c027890 osd=2 3 2024-02-13T00:48:00.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.579+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c027890 osd=2 3 2024-02-13T00:48:00.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.579+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c027890 osd=2 3 2024-02-13T00:48:00.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.579+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 6 2024-02-13T00:48:00.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.579+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c027890 osd=2 3 2024-02-13T00:48:00.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.579+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 2 6 2024-02-13T00:48:00.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.579+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c027890 osd=2 4 2024-02-13T00:48:00.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.579+0000 7f4421170700 10 client.27492.objecter _send_command 6 2024-02-13T00:48:00.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.579+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f441c0ff130 con 0x7f441c0ffb70 2024-02-13T00:48:00.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f441c0ff130 con 0x7f441c0ffb70 2024-02-13T00:48:00.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:00.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4403fff700 10 client.27492.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c027890 osd=2 4 2024-02-13T00:48:00.977 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:48:00.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 2 6 2024-02-13T00:48:00.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4421170700 10 client.27492.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:00.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 7 2024-02-13T00:48:00.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4421170700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6813/604971011 conn(0x7f441c11fbb0 0x7f441c11fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:00.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c11bb80 osd=3 3 2024-02-13T00:48:00.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c11bb80 osd=3 3 2024-02-13T00:48:00.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c11bb80 osd=3 3 2024-02-13T00:48:00.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 7 2024-02-13T00:48:00.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c11bb80 osd=3 3 2024-02-13T00:48:00.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 3 7 2024-02-13T00:48:00.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c11bb80 osd=3 4 2024-02-13T00:48:00.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4421170700 10 client.27492.objecter _send_command 7 2024-02-13T00:48:00.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f441c11c230 con 0x7f441c11fbb0 2024-02-13T00:48:00.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.582+0000 7f441b59e700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6813/604971011 conn(0x7f441c11fbb0 0x7f441c11fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:00.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.583+0000 7f441b59e700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.162:6813/604971011 conn(0x7f441c11fbb0 0x7f441c11fff0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:00.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.583+0000 7f4403fff700 10 client.27492.objecter ms_handle_connect 0x7f441c11fbb0 2024-02-13T00:48:00.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.583+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f441c11c230 con 0x7f441c11fbb0 2024-02-13T00:48:00.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.583+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.583+0000 7f4403fff700 10 client.27492.objecter _finish_command 7 = system:0 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.583+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c11bb80 osd=3 4 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.583+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 3 7 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.598+0000 7f4421170700 10 client.27492.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.598+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 8 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.598+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c11bb80 osd=3 3 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.598+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c11bb80 osd=3 3 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.598+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c11bb80 osd=3 3 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.598+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 8 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.598+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c11bb80 osd=3 3 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.598+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 3 8 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.598+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c11bb80 osd=3 4 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.598+0000 7f4421170700 10 client.27492.objecter _send_command 8 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.598+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f441c120cb0 con 0x7f441c11fbb0 2024-02-13T00:48:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f441c120cb0 con 0x7f441c11fbb0 2024-02-13T00:48:00.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:00.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4403fff700 10 client.27492.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c11bb80 osd=3 4 2024-02-13T00:48:00.980 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:48:00.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 3 8 2024-02-13T00:48:00.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4421170700 10 client.27492.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 9 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4421170700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.181:6800/3344263388 conn(0x7f441c123310 0x7f441c125700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c1204e0 osd=4 3 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c1204e0 osd=4 3 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c1204e0 osd=4 3 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 9 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c1204e0 osd=4 3 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 4 9 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c1204e0 osd=4 4 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4421170700 10 client.27492.objecter _send_command 9 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f441c120c80 con 0x7f441c123310 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.600+0000 7f441ad9d700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.181:6800/3344263388 conn(0x7f441c123310 0x7f441c125700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.601+0000 7f4403fff700 10 client.27492.objecter ms_handle_connect 0x7f441c123310 2024-02-13T00:48:00.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.601+0000 7f441ad9d700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.181:6800/3344263388 conn(0x7f441c123310 0x7f441c125700 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:00.982 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:48:00.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.601+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f441c120c80 con 0x7f441c123310 2024-02-13T00:48:00.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.601+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:48:00.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.601+0000 7f4403fff700 10 client.27492.objecter _finish_command 9 = system:0 2024-02-13T00:48:00.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.601+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c1204e0 osd=4 4 2024-02-13T00:48:00.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.601+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 4 9 2024-02-13T00:48:00.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.616+0000 7f4421170700 10 client.27492.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:00.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.616+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 10 2024-02-13T00:48:00.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.616+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c1204e0 osd=4 3 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.616+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c1204e0 osd=4 3 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.616+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c1204e0 osd=4 3 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.616+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 10 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.616+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c1204e0 osd=4 3 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.616+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 4 10 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.616+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c1204e0 osd=4 4 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.616+0000 7f4421170700 10 client.27492.objecter _send_command 10 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.616+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f441c120c80 con 0x7f441c123310 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f441c120c80 con 0x7f441c123310 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4403fff700 10 client.27492.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c1204e0 osd=4 4 2024-02-13T00:48:00.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 4 10 2024-02-13T00:48:00.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 10 client.27492.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:00.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 11 2024-02-13T00:48:00.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 20 client.27492.objecter _get_session osd=-1 returning homeless 2024-02-13T00:48:00.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 10 client.27492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:00.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=478}) v3 -- 0x7f441c128450 con 0x7f441c0fdc40 2024-02-13T00:48:00.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f441c128680 con 0x7f441c0fdc40 2024-02-13T00:48:00.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_get_version_reply(handle=1 version=477) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4410050d00 con 0x7f441c0fdc40 2024-02-13T00:48:00.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f441959a700 10 client.27492.objecter _check_command_map_dne tid 11 current 477 map_dne_bound 477 2024-02-13T00:48:00.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f441959a700 10 client.27492.objecter _finish_command 11 = osd:6 osd down 2024-02-13T00:48:00.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f441959a700 15 client.27492.objecter _session_command_op_remove -1 11 2024-02-13T00:48:00.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 10 client.27492.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:00.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 12 2024-02-13T00:48:00.988 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:48:00.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.181:6808/1506640341 conn(0x7f441c11b760 0x7f441c126d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:00.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c11b5f0 osd=6 3 2024-02-13T00:48:00.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c11b5f0 osd=6 3 2024-02-13T00:48:00.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c11b5f0 osd=6 3 2024-02-13T00:48:00.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 12 2024-02-13T00:48:00.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c11b5f0 osd=6 3 2024-02-13T00:48:00.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 6 12 2024-02-13T00:48:00.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c11b5f0 osd=6 4 2024-02-13T00:48:00.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 10 client.27492.objecter _send_command 12 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.618+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.181:6808/1506640341 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f441c11f8e0 con 0x7f441c11b760 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.619+0000 7f441a59c700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.181:6808/1506640341 conn(0x7f441c11b760 0x7f441c126d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.619+0000 7f441a59c700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.181:6808/1506640341 conn(0x7f441c11b760 0x7f441c126d70 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.619+0000 7f4403fff700 10 client.27492.objecter ms_handle_connect 0x7f441c11b760 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.619+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f441c11f8e0 con 0x7f441c11b760 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.619+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 12: 0 ) v1 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.619+0000 7f4403fff700 10 client.27492.objecter _finish_command 12 = system:0 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.619+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c11b5f0 osd=6 4 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.619+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 6 12 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.634+0000 7f4421170700 10 client.27492.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.634+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 13 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.634+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c11b5f0 osd=6 3 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.634+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c11b5f0 osd=6 3 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.634+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c11b5f0 osd=6 3 2024-02-13T00:48:00.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.634+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 13 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.634+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c11b5f0 osd=6 3 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.634+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 6 13 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.634+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c11b5f0 osd=6 4 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.634+0000 7f4421170700 10 client.27492.objecter _send_command 13 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.634+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f441c11f8e0 con 0x7f441c11b760 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f441c11f8e0 con 0x7f441c11b760 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4403fff700 10 client.27492.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c11b5f0 osd=6 4 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 6 13 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4421170700 10 client.27492.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 14 2024-02-13T00:48:00.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4421170700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.181:6812/1964192772 conn(0x7f441c127770 0x7f441c127bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c11f8e0 osd=7 3 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c11f8e0 osd=7 3 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c11f8e0 osd=7 3 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 14 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c11f8e0 osd=7 3 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 7 14 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c11f8e0 osd=7 4 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4421170700 10 client.27492.objecter _send_command 14 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.181:6812/1964192772 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f441c12ef70 con 0x7f441c127770 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.636+0000 7f441b59e700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.181:6812/1964192772 conn(0x7f441c127770 0x7f441c127bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.637+0000 7f441b59e700 1 --2- 172.21.15.162:0/1296304111 >> v2:172.21.15.181:6812/1964192772 conn(0x7f441c127770 0x7f441c127bd0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.637+0000 7f4403fff700 10 client.27492.objecter ms_handle_connect 0x7f441c127770 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.638+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f441c12ef70 con 0x7f441c127770 2024-02-13T00:48:00.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.638+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 14: 0 ) v1 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.638+0000 7f4403fff700 10 client.27492.objecter _finish_command 14 = system:0 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.638+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c11f8e0 osd=7 4 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.638+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 7 14 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.661+0000 7f4421170700 10 client.27492.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.661+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign -1 15 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.661+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c11f8e0 osd=7 3 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.661+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c11f8e0 osd=7 3 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.661+0000 7f4421170700 20 client.27492.objecter _get_session s=0x7f441c11f8e0 osd=7 3 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.661+0000 7f4421170700 15 client.27492.objecter _session_command_op_remove -1 15 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.661+0000 7f4421170700 20 client.27492.objecter get_session s=0x7f441c11f8e0 osd=7 3 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.661+0000 7f4421170700 15 client.27492.objecter _session_command_op_assign 7 15 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.661+0000 7f4421170700 20 client.27492.objecter put_session s=0x7f441c11f8e0 osd=7 4 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.661+0000 7f4421170700 10 client.27492.objecter _send_command 15 2024-02-13T00:48:00.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.661+0000 7f4421170700 1 -- 172.21.15.162:0/1296304111 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f441c12c810 con 0x7f441c127770 2024-02-13T00:48:00.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.663+0000 7f4403fff700 1 -- 172.21.15.162:0/1296304111 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f441c12c810 con 0x7f441c127770 2024-02-13T00:48:00.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.663+0000 7f4403fff700 10 client.27492.objecter ms_dispatch 0x7f441c001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:00.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.663+0000 7f4403fff700 10 client.27492.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.663+0000 7f4403fff700 20 client.27492.objecter put_session s=0x7f441c11f8e0 osd=7 4 2024-02-13T00:48:00.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:00.663+0000 7f4403fff700 15 client.27492.objecter _session_command_op_remove 7 15 2024-02-13T00:48:00.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:00.995 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:48:00.995 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.995 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.995 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.995 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.995 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.995 INFO:teuthology.orchestra.run.smithi162.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-02-13T00:48:00.995 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:48:00.996 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:48:01.023 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:01.043 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:01.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:01.146 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:01.173 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:01.192 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:01.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:01.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:01.325 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:01.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:00.448378+0000 mgr.y (mgr.14544) 426 : cluster [DBG] pgmap v710: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-13T00:48:01.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:00.529619+0000 mon.a (mon.0) 2657 : audit [DBG] from='client.? 172.21.15.162:0/1296304111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:01.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:00.947055+0000 mon.c (mon.2) 395 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:48:01.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:00.947707+0000 mon.c (mon.2) 396 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:48:01.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:00.952551+0000 mon.a (mon.0) 2658 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:48:01.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:00.448378+0000 mgr.y (mgr.14544) 426 : cluster [DBG] pgmap v710: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-13T00:48:01.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:00.529619+0000 mon.a (mon.0) 2657 : audit [DBG] from='client.? 172.21.15.162:0/1296304111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:01.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:00.947055+0000 mon.c (mon.2) 395 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:48:01.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:00.947707+0000 mon.c (mon.2) 396 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:48:01.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:00.952551+0000 mon.a (mon.0) 2658 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:48:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-13T00:48:01.699 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-13T00:48:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:00.448378+0000 mgr.y (mgr.14544) 426 : cluster [DBG] pgmap v710: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-13T00:48:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:00.529619+0000 mon.a (mon.0) 2657 : audit [DBG] from='client.? 172.21.15.162:0/1296304111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:00.947055+0000 mon.c (mon.2) 395 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:48:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:00.947707+0000 mon.c (mon.2) 396 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:48:01.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:00.952551+0000 mon.a (mon.0) 2658 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:48:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:02.424101+0000 mon.b (mon.1) 608 : audit [INF] from='osd.5 v2:172.21.15.181:6804/2277917372' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T00:48:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:02.424540+0000 mon.a (mon.0) 2659 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T00:48:02.795 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:48:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:48:02.417+0000 7ff8ca8e3380 -1 osd.5 436 log_to_monitors {default=true} 2024-02-13T00:48:02.795 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:48:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:48:02.489+0000 7ff8b9dc6700 -1 osd.5 476 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T00:48:02.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:02.424101+0000 mon.b (mon.1) 608 : audit [INF] from='osd.5 v2:172.21.15.181:6804/2277917372' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T00:48:02.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:02.424540+0000 mon.a (mon.0) 2659 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T00:48:02.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:02.424101+0000 mon.b (mon.1) 608 : audit [INF] from='osd.5 v2:172.21.15.181:6804/2277917372' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T00:48:02.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:02.424540+0000 mon.a (mon.0) 2659 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-13T00:48:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:03.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:03.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:03.180 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:03.214 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:03.234 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:03.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:03.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:03.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:03.378 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:03.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:03.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:03.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:03.535 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:03.555 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:03.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:03.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:03.659 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:03.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:02.449019+0000 mgr.y (mgr.14544) 427 : cluster [DBG] pgmap v711: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-02-13T00:48:03.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:02.477527+0000 mon.a (mon.0) 2660 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-13T00:48:03.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:02.477593+0000 mon.a (mon.0) 2661 : cluster [DBG] osdmap e478: 8 total, 7 up, 4 in 2024-02-13T00:48:03.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:02.479831+0000 mon.b (mon.1) 609 : audit [INF] from='osd.5 v2:172.21.15.181:6804/2277917372' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:48:03.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:02.480419+0000 mon.a (mon.0) 2662 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:48:03.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:02.483928+0000 mon.c (mon.2) 397 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:48:03.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:02.449019+0000 mgr.y (mgr.14544) 427 : cluster [DBG] pgmap v711: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-02-13T00:48:03.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:02.477527+0000 mon.a (mon.0) 2660 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-13T00:48:03.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:02.477593+0000 mon.a (mon.0) 2661 : cluster [DBG] osdmap e478: 8 total, 7 up, 4 in 2024-02-13T00:48:03.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:02.479831+0000 mon.b (mon.1) 609 : audit [INF] from='osd.5 v2:172.21.15.181:6804/2277917372' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:48:03.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:02.480419+0000 mon.a (mon.0) 2662 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:48:03.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:02.483928+0000 mon.c (mon.2) 397 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:48:03.687 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:03.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:48:03.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.625+0000 7f481fa91700 1 -- 172.21.15.162:0/1953079306 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180f8d10 msgr2=0x7f48180f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.625+0000 7f481fa91700 1 --2- 172.21.15.162:0/1953079306 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180f8d10 0x7f48180f90f0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 rx=0x7f4808004870 tx=0x7f4808009cb0).stop 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.625+0000 7f481fa91700 1 -- 172.21.15.162:0/1953079306 shutdown_connections 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.625+0000 7f481fa91700 1 --2- 172.21.15.162:0/1953079306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4818102270 0x7f4818104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.625+0000 7f481fa91700 1 --2- 172.21.15.162:0/1953079306 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180f8d10 0x7f48180f90f0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.625+0000 7f481fa91700 1 --2- 172.21.15.162:0/1953079306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48180f95e0 0x7f4818101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.625+0000 7f481fa91700 1 -- 172.21.15.162:0/1953079306 >> 172.21.15.162:0/1953079306 conn(0x7f481806eb90 msgr2=0x7f48180f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.625+0000 7f481fa91700 1 -- 172.21.15.162:0/1953079306 shutdown_connections 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.625+0000 7f481fa91700 1 -- 172.21.15.162:0/1953079306 wait complete. 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.625+0000 7f481fa91700 1 librados: starting msgr at 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.625+0000 7f481fa91700 1 librados: starting objecter 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 1 Processor -- start 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 1 -- start start 2024-02-13T00:48:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 1 librados: setting wanted keys 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 1 librados: calling monclient init 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180f8d10 0x7f481810e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48180f95e0 0x7f481810e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4818102270 0x7f4818108410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4818108900 con 0x7f4818102270 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4818108a70 con 0x7f48180f95e0 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4818108c30 con 0x7f48180f8d10 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481d82d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180f8d10 0x7f481810e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481d82d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180f8d10 0x7f481810e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48468/0 (socket says 172.21.15.162:48468) 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481e02e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4818102270 0x7f4818108410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481d82d700 1 -- 172.21.15.162:0/231699048 learned_addr learned my addr 172.21.15.162:0/231699048 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481d02c700 1 --2- 172.21.15.162:0/231699048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48180f95e0 0x7f481810e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481d82d700 1 -- 172.21.15.162:0/231699048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48180f95e0 msgr2=0x7f481810e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:03.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481d82d700 1 --2- 172.21.15.162:0/231699048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48180f95e0 0x7f481810e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481d82d700 1 -- 172.21.15.162:0/231699048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4818102270 msgr2=0x7f4818108410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481d82d700 1 --2- 172.21.15.162:0/231699048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4818102270 0x7f4818108410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481d82d700 1 -- 172.21.15.162:0/231699048 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4814005040 con 0x7f48180f8d10 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481d02c700 1 --2- 172.21.15.162:0/231699048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48180f95e0 0x7f481810e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481e02e700 1 --2- 172.21.15.162:0/231699048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4818102270 0x7f4818108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481d82d700 1 --2- 172.21.15.162:0/231699048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180f8d10 0x7f481810e270 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7f4808009eb0 tx=0x7f480800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f480ef78700 10 client.?.objecter ms_handle_connect 0x7f48180f8d10 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f480ef78700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f480ef78700 1 -- 172.21.15.162:0/231699048 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480800e070 con 0x7f48180f8d10 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 1 -- 172.21.15.162:0/231699048 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4808005040 con 0x7f48180f8d10 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f480ef78700 1 -- 172.21.15.162:0/231699048 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4808013da0 con 0x7f48180f8d10 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f480ef78700 1 -- 172.21.15.162:0/231699048 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808019070 con 0x7f48180f8d10 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 10 client.37601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.626+0000 7f481fa91700 1 -- 172.21.15.162:0/231699048 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48181090d0 con 0x7f48180f8d10 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.627+0000 7f481fa91700 1 librados: init done 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.628+0000 7f480ef78700 1 -- 172.21.15.162:0/231699048 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f480801e420 con 0x7f48180f8d10 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.628+0000 7f481fa91700 1 -- 172.21.15.162:0/231699048 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48180475e0 con 0x7f48180f8d10 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.628+0000 7f480ef78700 1 --2- 172.21.15.162:0/231699048 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4804062c90 0x7f4804065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.628+0000 7f480ef78700 1 -- 172.21.15.162:0/231699048 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(479..479 src has 52..479) v4 ==== 7967+0+0 (secure 0 0 0) 0x7f480807dc40 con 0x7f48180f8d10 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.628+0000 7f480ef78700 10 client.37601.objecter ms_dispatch 0x7f4818001550 osd_map(479..479 src has 52..479) v4 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.628+0000 7f480ef78700 3 client.37601.objecter handle_osd_map got epochs [479,479] > 0 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.628+0000 7f480ef78700 3 client.37601.objecter handle_osd_map decoding full epoch 479 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.628+0000 7f481d02c700 1 --2- 172.21.15.162:0/231699048 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4804062c90 0x7f4804065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.629+0000 7f480ef78700 20 client.37601.objecter dump_active .. 0 homeless 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.629+0000 7f480ef78700 10 client.37601.objecter ms_handle_connect 0x7f4804062c90 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.629+0000 7f481d02c700 1 --2- 172.21.15.162:0/231699048 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4804062c90 0x7f4804065150 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 rx=0x7f4814007fd0 tx=0x7f4814009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:03.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.633+0000 7f480ef78700 1 -- 172.21.15.162:0/231699048 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f48080511b0 con 0x7f48180f8d10 2024-02-13T00:48:03.710 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:03.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:02.449019+0000 mgr.y (mgr.14544) 427 : cluster [DBG] pgmap v711: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-02-13T00:48:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:02.477527+0000 mon.a (mon.0) 2660 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-13T00:48:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:02.477593+0000 mon.a (mon.0) 2661 : cluster [DBG] osdmap e478: 8 total, 7 up, 4 in 2024-02-13T00:48:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:02.479831+0000 mon.b (mon.1) 609 : audit [INF] from='osd.5 v2:172.21.15.181:6804/2277917372' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:48:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:02.480419+0000 mon.a (mon.0) 2662 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-02-13T00:48:03.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:02.483928+0000 mon.c (mon.2) 397 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:48:03.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:03.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:03.819+0000 7f481fa91700 1 -- 172.21.15.162:0/231699048 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f4818107170 con 0x7f48180f8d10 2024-02-13T00:48:03.839 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:04.389 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:04.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:04.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:04.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.494+0000 7f480ef78700 1 -- 172.21.15.162:0/231699048 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v480) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4808023030 con 0x7f48180f8d10 2024-02-13T00:48:04.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 10 librados: watch_flush enter 2024-02-13T00:48:04.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 10 librados: watch_flush exit 2024-02-13T00:48:04.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 20 client.37601.objecter shutdown clearing up homeless session... 2024-02-13T00:48:04.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 10 client.37601.objecter successfully canceled tick 2024-02-13T00:48:04.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 1 -- 172.21.15.162:0/231699048 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4804062c90 msgr2=0x7f4804065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:04.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 1 --2- 172.21.15.162:0/231699048 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4804062c90 0x7f4804065150 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 rx=0x7f4814007fd0 tx=0x7f4814009cb0).stop 2024-02-13T00:48:04.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 1 -- 172.21.15.162:0/231699048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180f8d10 msgr2=0x7f481810e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:04.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 1 --2- 172.21.15.162:0/231699048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180f8d10 0x7f481810e270 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7f4808009eb0 tx=0x7f480800b040).stop 2024-02-13T00:48:04.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 1 -- 172.21.15.162:0/231699048 shutdown_connections 2024-02-13T00:48:04.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 1 --2- 172.21.15.162:0/231699048 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4804062c90 0x7f4804065150 secure :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 rx=0x7f4814007fd0 tx=0x7f4814009cb0).stop 2024-02-13T00:48:04.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 1 --2- 172.21.15.162:0/231699048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4818102270 0x7f4818108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:04.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 1 --2- 172.21.15.162:0/231699048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180f8d10 0x7f481810e270 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:04.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 1 --2- 172.21.15.162:0/231699048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48180f95e0 0x7f481810e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:04.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.497+0000 7f481fa91700 1 -- 172.21.15.162:0/231699048 >> 172.21.15.162:0/231699048 conn(0x7f481806eb90 msgr2=0x7f4818102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:04.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.498+0000 7f481fa91700 1 -- 172.21.15.162:0/231699048 shutdown_connections 2024-02-13T00:48:04.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.498+0000 7f481fa91700 1 -- 172.21.15.162:0/231699048 wait complete. 2024-02-13T00:48:04.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:04.498+0000 7f481fa91700 1 librados: shutdown 2024-02-13T00:48:04.502 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:48:04.523 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:04.544 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:04.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:04.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:04.648 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:04.674 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:03.479734+0000 mon.a (mon.0) 2663 : cluster [INF] osd.5 v2:172.21.15.181:6804/2277917372 boot 2024-02-13T00:48:04.674 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:03.479822+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e479: 8 total, 8 up, 4 in 2024-02-13T00:48:04.674 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:03.480402+0000 mon.c (mon.2) 398 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:48:04.675 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:03.820229+0000 mon.c (mon.2) 399 : audit [INF] from='client.? 172.21.15.162:0/231699048' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:04.675 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:03.820648+0000 mon.a (mon.0) 2665 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:04.675 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:04.338012+0000 mon.a (mon.0) 2666 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:48:04.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:03.479734+0000 mon.a (mon.0) 2663 : cluster [INF] osd.5 v2:172.21.15.181:6804/2277917372 boot 2024-02-13T00:48:04.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:03.479822+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e479: 8 total, 8 up, 4 in 2024-02-13T00:48:04.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:03.480402+0000 mon.c (mon.2) 398 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:48:04.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:03.820229+0000 mon.c (mon.2) 399 : audit [INF] from='client.? 172.21.15.162:0/231699048' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:04.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:03.820648+0000 mon.a (mon.0) 2665 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:04.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:04.338012+0000 mon.a (mon.0) 2666 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:48:04.677 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:03.479734+0000 mon.a (mon.0) 2663 : cluster [INF] osd.5 v2:172.21.15.181:6804/2277917372 boot 2024-02-13T00:48:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:03.479822+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e479: 8 total, 8 up, 4 in 2024-02-13T00:48:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:03.480402+0000 mon.c (mon.2) 398 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-13T00:48:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:03.820229+0000 mon.c (mon.2) 399 : audit [INF] from='client.? 172.21.15.162:0/231699048' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:03.820648+0000 mon.a (mon.0) 2665 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:04.338012+0000 mon.a (mon.0) 2666 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:48:04.868 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:48:04.868 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:48:05.072 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:05.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:05.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:05.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:05.675 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:05.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:03.401058+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:05.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:03.401332+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:05.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:04.449445+0000 mgr.y (mgr.14544) 428 : cluster [DBG] pgmap v714: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 854 B/s rd, 854 B/s wr, 2 op/s 2024-02-13T00:48:05.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:04.482419+0000 mon.a (mon.0) 2667 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:05.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:04.491779+0000 mon.a (mon.0) 2668 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:48:05.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:04.491829+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e480: 8 total, 8 up, 4 in 2024-02-13T00:48:05.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.428308+0000 mon.c (mon.2) 400 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]: dispatch 2024-02-13T00:48:05.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.428528+0000 mon.c (mon.2) 401 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [6, 0]}]: dispatch 2024-02-13T00:48:05.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.428671+0000 mon.a (mon.0) 2670 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]: dispatch 2024-02-13T00:48:05.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.428773+0000 mon.c (mon.2) 402 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.22", "id": [6, 4]}]: dispatch 2024-02-13T00:48:05.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.428949+0000 mon.a (mon.0) 2671 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [6, 0]}]: dispatch 2024-02-13T00:48:05.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.428972+0000 mon.c (mon.2) 403 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.14", "id": [6, 1]}]: dispatch 2024-02-13T00:48:05.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.429128+0000 mon.a (mon.0) 2672 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.22", "id": [6, 4]}]: dispatch 2024-02-13T00:48:05.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.429303+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.14", "id": [6, 1]}]: dispatch 2024-02-13T00:48:05.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.453251+0000 mon.a (mon.0) 2674 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]': finished 2024-02-13T00:48:05.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:03.401058+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:05.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:03.401332+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:04.449445+0000 mgr.y (mgr.14544) 428 : cluster [DBG] pgmap v714: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 854 B/s rd, 854 B/s wr, 2 op/s 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:04.482419+0000 mon.a (mon.0) 2667 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:04.491779+0000 mon.a (mon.0) 2668 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:04.491829+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e480: 8 total, 8 up, 4 in 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.428308+0000 mon.c (mon.2) 400 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]: dispatch 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.428528+0000 mon.c (mon.2) 401 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [6, 0]}]: dispatch 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.428671+0000 mon.a (mon.0) 2670 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]: dispatch 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.428773+0000 mon.c (mon.2) 402 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.22", "id": [6, 4]}]: dispatch 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.428949+0000 mon.a (mon.0) 2671 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [6, 0]}]: dispatch 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.428972+0000 mon.c (mon.2) 403 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.14", "id": [6, 1]}]: dispatch 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.429128+0000 mon.a (mon.0) 2672 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.22", "id": [6, 4]}]: dispatch 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.429303+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.14", "id": [6, 1]}]: dispatch 2024-02-13T00:48:05.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.453251+0000 mon.a (mon.0) 2674 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]': finished 2024-02-13T00:48:05.687 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.453348+0000 mon.a (mon.0) 2675 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [6, 0]}]': finished 2024-02-13T00:48:05.687 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.453430+0000 mon.a (mon.0) 2676 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.22", "id": [6, 4]}]': finished 2024-02-13T00:48:05.687 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.453504+0000 mon.a (mon.0) 2677 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.14", "id": [6, 1]}]': finished 2024-02-13T00:48:05.687 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:05.453569+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e481: 8 total, 8 up, 4 in 2024-02-13T00:48:05.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.453348+0000 mon.a (mon.0) 2675 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [6, 0]}]': finished 2024-02-13T00:48:05.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.453430+0000 mon.a (mon.0) 2676 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.22", "id": [6, 4]}]': finished 2024-02-13T00:48:05.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.453504+0000 mon.a (mon.0) 2677 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.14", "id": [6, 1]}]': finished 2024-02-13T00:48:05.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:05.453569+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e481: 8 total, 8 up, 4 in 2024-02-13T00:48:05.696 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:05.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:05.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-13T00:48:05.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:48:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:03.401058+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:03.401332+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:04.449445+0000 mgr.y (mgr.14544) 428 : cluster [DBG] pgmap v714: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 854 B/s rd, 854 B/s wr, 2 op/s 2024-02-13T00:48:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:04.482419+0000 mon.a (mon.0) 2667 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:04.491779+0000 mon.a (mon.0) 2668 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:48:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:04.491829+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e480: 8 total, 8 up, 4 in 2024-02-13T00:48:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.428308+0000 mon.c (mon.2) 400 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]: dispatch 2024-02-13T00:48:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.428528+0000 mon.c (mon.2) 401 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [6, 0]}]: dispatch 2024-02-13T00:48:05.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.428671+0000 mon.a (mon.0) 2670 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]: dispatch 2024-02-13T00:48:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.428773+0000 mon.c (mon.2) 402 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.22", "id": [6, 4]}]: dispatch 2024-02-13T00:48:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.428949+0000 mon.a (mon.0) 2671 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [6, 0]}]: dispatch 2024-02-13T00:48:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.428972+0000 mon.c (mon.2) 403 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.14", "id": [6, 1]}]: dispatch 2024-02-13T00:48:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.429128+0000 mon.a (mon.0) 2672 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.22", "id": [6, 4]}]: dispatch 2024-02-13T00:48:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.429303+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.14", "id": [6, 1]}]: dispatch 2024-02-13T00:48:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.453251+0000 mon.a (mon.0) 2674 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]': finished 2024-02-13T00:48:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.453348+0000 mon.a (mon.0) 2675 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [6, 0]}]': finished 2024-02-13T00:48:05.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.453430+0000 mon.a (mon.0) 2676 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.22", "id": [6, 4]}]': finished 2024-02-13T00:48:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.453504+0000 mon.a (mon.0) 2677 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.14", "id": [6, 1]}]': finished 2024-02-13T00:48:05.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:05.453569+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e481: 8 total, 8 up, 4 in 2024-02-13T00:48:05.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:05.829 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:06.042 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:48:06.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-13T00:48:06.489 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:48:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:48:06] "GET /metrics HTTP/1.1" 200 187995 "" "Prometheus/2.33.4" 2024-02-13T00:48:06.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:05.685327+0000 mgr.y (mgr.14544) 429 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:06.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:06.453220+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e482: 8 total, 8 up, 4 in 2024-02-13T00:48:06.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:05.685327+0000 mgr.y (mgr.14544) 429 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:06.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:06.453220+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e482: 8 total, 8 up, 4 in 2024-02-13T00:48:06.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:05.685327+0000 mgr.y (mgr.14544) 429 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:06.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:06.453220+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e482: 8 total, 8 up, 4 in 2024-02-13T00:48:06.875 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:07.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:06.449922+0000 mgr.y (mgr.14544) 430 : cluster [DBG] pgmap v717: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:07.681 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:06.449922+0000 mgr.y (mgr.14544) 430 : cluster [DBG] pgmap v717: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:07.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:06.449922+0000 mgr.y (mgr.14544) 430 : cluster [DBG] pgmap v717: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:48:08.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.589+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/761577704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe79c0fe580 msgr2=0x7fe79c0f7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.589+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/761577704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe79c0fe580 0x7fe79c0f7db0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 rx=0x7fe78c004870 tx=0x7fe78c009cb0).stop 2024-02-13T00:48:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.589+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/761577704 shutdown_connections 2024-02-13T00:48:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.589+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/761577704 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe79c0f8490 0x7fe79c104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.589+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/761577704 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe79c0fdc20 0x7fe79c0fe000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.589+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/761577704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe79c0fe580 0x7fe79c0f7db0 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.589+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/761577704 >> 172.21.15.162:0/761577704 conn(0x7fe79c0f3850 msgr2=0x7fe79c0f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.589+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/761577704 shutdown_connections 2024-02-13T00:48:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.589+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/761577704 wait complete. 2024-02-13T00:48:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.590+0000 7fe7a3fc0700 1 librados: starting msgr at 2024-02-13T00:48:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.590+0000 7fe7a3fc0700 1 librados: starting objecter 2024-02-13T00:48:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.590+0000 7fe7a3fc0700 1 Processor -- start 2024-02-13T00:48:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.590+0000 7fe7a3fc0700 1 -- start start 2024-02-13T00:48:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.590+0000 7fe7a3fc0700 1 librados: setting wanted keys 2024-02-13T00:48:08.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.590+0000 7fe7a3fc0700 1 librados: calling monclient init 2024-02-13T00:48:08.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.590+0000 7fe7a3fc0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe79c0f8490 0x7fe79c10e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:08.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.590+0000 7fe7a3fc0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe79c0fdc20 0x7fe79c10e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:08.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.590+0000 7fe7a3fc0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe79c0fe580 0x7fe79c1082b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:08.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.590+0000 7fe7a3fc0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c1087a0 con 0x7fe79c0fe580 2024-02-13T00:48:08.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.590+0000 7fe7a3fc0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c108910 con 0x7fe79c0fdc20 2024-02-13T00:48:08.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.590+0000 7fe7a3fc0700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c108ad0 con 0x7fe79c0f8490 2024-02-13T00:48:08.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a1d5c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe79c0f8490 0x7fe79c10e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:08.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a255d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe79c0fe580 0x7fe79c1082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:08.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a255d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe79c0fe580 0x7fe79c1082b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:54912/0 (socket says 172.21.15.162:54912) 2024-02-13T00:48:08.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a255d700 1 -- 172.21.15.162:0/1780262099 learned_addr learned my addr 172.21.15.162:0/1780262099 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:08.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a1d5c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe79c0f8490 0x7fe79c10e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:39500/0 (socket says 172.21.15.162:39500) 2024-02-13T00:48:08.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a155b700 1 --2- 172.21.15.162:0/1780262099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe79c0fdc20 0x7fe79c10e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:08.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a255d700 1 -- 172.21.15.162:0/1780262099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe79c0f8490 msgr2=0x7fe79c10e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a255d700 1 --2- 172.21.15.162:0/1780262099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe79c0f8490 0x7fe79c10e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a255d700 1 -- 172.21.15.162:0/1780262099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe79c0fdc20 msgr2=0x7fe79c10e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a255d700 1 --2- 172.21.15.162:0/1780262099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe79c0fdc20 0x7fe79c10e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a255d700 1 -- 172.21.15.162:0/1780262099 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7940097c0 con 0x7fe79c0fe580 2024-02-13T00:48:08.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a155b700 1 --2- 172.21.15.162:0/1780262099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe79c0fdc20 0x7fe79c10e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:48:08.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a255d700 1 --2- 172.21.15.162:0/1780262099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe79c0fe580 0x7fe79c1082b0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 rx=0x7fe794001df0 tx=0x7fe794004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:08.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe792ffd700 10 client.?.objecter ms_handle_connect 0x7fe79c0fe580 2024-02-13T00:48:08.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe792ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:08.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe794009a10 con 0x7fe79c0fe580 2024-02-13T00:48:08.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7940095a0 con 0x7fe79c0fe580 2024-02-13T00:48:08.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe78c005040 con 0x7fe79c0fe580 2024-02-13T00:48:08.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe794007da0 con 0x7fe79c0fe580 2024-02-13T00:48:08.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a3fc0700 10 client.27507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:08.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79c109110 con 0x7fe79c0fe580 2024-02-13T00:48:08.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.591+0000 7fe7a3fc0700 1 librados: init done 2024-02-13T00:48:08.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.592+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe79c0607d0 con 0x7fe79c0fe580 2024-02-13T00:48:08.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe7940090d0 con 0x7fe79c0fe580 2024-02-13T00:48:08.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe792ffd700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe788062c90 0x7fe788065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:08.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(483..483 src has 52..483) v4 ==== 8083+0+0 (secure 0 0 0) 0x7fe79407db20 con 0x7fe79c0fe580 2024-02-13T00:48:08.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 osd_map(483..483 src has 52..483) v4 2024-02-13T00:48:08.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe792ffd700 3 client.27507.objecter handle_osd_map got epochs [483,483] > 0 2024-02-13T00:48:08.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe792ffd700 3 client.27507.objecter handle_osd_map decoding full epoch 483 2024-02-13T00:48:08.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a1d5c700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe788062c90 0x7fe788065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:08.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe792ffd700 20 client.27507.objecter dump_active .. 0 homeless 2024-02-13T00:48:08.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v483) v1 ==== 54+0+16 (secure 0 0 0) 0x7fe794082220 con 0x7fe79c0fe580 2024-02-13T00:48:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe792ffd700 10 client.27507.objecter ms_handle_connect 0x7fe788062c90 2024-02-13T00:48:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a1d5c700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe788062c90 0x7fe788065150 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 rx=0x7fe798004e60 tx=0x7fe79800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 1 2024-02-13T00:48:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe79c028590 0x7fe79c0fe960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c0475e0 osd=0 3 2024-02-13T00:48:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c0475e0 osd=0 3 2024-02-13T00:48:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c0475e0 osd=0 3 2024-02-13T00:48:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 1 2024-02-13T00:48:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c0475e0 osd=0 3 2024-02-13T00:48:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 0 1 2024-02-13T00:48:08.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c0475e0 osd=0 4 2024-02-13T00:48:08.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 1 2024-02-13T00:48:08.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe79c107110 con 0x7fe79c028590 2024-02-13T00:48:08.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.593+0000 7fe7a155b700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe79c028590 0x7fe79c0fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:08.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.594+0000 7fe792ffd700 10 client.27507.objecter ms_handle_connect 0x7fe79c028590 2024-02-13T00:48:08.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.594+0000 7fe7a155b700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe79c028590 0x7fe79c0fe960 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:08.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.594+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe79c107110 con 0x7fe79c028590 2024-02-13T00:48:08.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.594+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:48:08.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.594+0000 7fe792ffd700 10 client.27507.objecter _finish_command 1 = system:0 2024-02-13T00:48:08.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.594+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c0475e0 osd=0 4 2024-02-13T00:48:08.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.594+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 0 1 2024-02-13T00:48:08.608 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:08.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:08.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.609+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:08.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.609+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 2 2024-02-13T00:48:08.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.609+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c0475e0 osd=0 3 2024-02-13T00:48:08.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.609+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c0475e0 osd=0 3 2024-02-13T00:48:08.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.609+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c0475e0 osd=0 3 2024-02-13T00:48:08.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.609+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 2 2024-02-13T00:48:08.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.609+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c0475e0 osd=0 3 2024-02-13T00:48:08.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.609+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 0 2 2024-02-13T00:48:08.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.609+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c0475e0 osd=0 4 2024-02-13T00:48:08.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.609+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 2 2024-02-13T00:48:08.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.609+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe79c05a0e0 con 0x7fe79c028590 2024-02-13T00:48:08.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe79c05a0e0 con 0x7fe79c028590 2024-02-13T00:48:08.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:08.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe792ffd700 10 client.27507.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c0475e0 osd=0 4 2024-02-13T00:48:08.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 0 2 2024-02-13T00:48:08.615 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:48:08.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:08.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 3 2024-02-13T00:48:08.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6805/150599780 conn(0x7fe79c027be0 0x7fe79c027fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:08.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c0fffd0 osd=1 3 2024-02-13T00:48:08.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c0fffd0 osd=1 3 2024-02-13T00:48:08.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c0fffd0 osd=1 3 2024-02-13T00:48:08.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 3 2024-02-13T00:48:08.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c0fffd0 osd=1 3 2024-02-13T00:48:08.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 1 3 2024-02-13T00:48:08.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c0fffd0 osd=1 4 2024-02-13T00:48:08.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 3 2024-02-13T00:48:08.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe79c100140 con 0x7fe79c027be0 2024-02-13T00:48:08.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a1d5c700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6805/150599780 conn(0x7fe79c027be0 0x7fe79c027fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:08.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe792ffd700 10 client.27507.objecter ms_handle_connect 0x7fe79c027be0 2024-02-13T00:48:08.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.611+0000 7fe7a1d5c700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6805/150599780 conn(0x7fe79c027be0 0x7fe79c027fc0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:08.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.612+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe79c100140 con 0x7fe79c027be0 2024-02-13T00:48:08.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.612+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:48:08.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.612+0000 7fe792ffd700 10 client.27507.objecter _finish_command 3 = system:0 2024-02-13T00:48:08.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.612+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c0fffd0 osd=1 4 2024-02-13T00:48:08.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.612+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 1 3 2024-02-13T00:48:08.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.627+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:08.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.627+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 4 2024-02-13T00:48:08.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.627+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c0fffd0 osd=1 3 2024-02-13T00:48:08.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.627+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c0fffd0 osd=1 3 2024-02-13T00:48:08.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.627+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c0fffd0 osd=1 3 2024-02-13T00:48:08.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.627+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 4 2024-02-13T00:48:08.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.627+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c0fffd0 osd=1 3 2024-02-13T00:48:08.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.627+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 1 4 2024-02-13T00:48:08.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.627+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c0fffd0 osd=1 4 2024-02-13T00:48:08.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.627+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 4 2024-02-13T00:48:08.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.627+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe79c027650 con 0x7fe79c027be0 2024-02-13T00:48:08.632 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:48:08.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe79c027650 con 0x7fe79c027be0 2024-02-13T00:48:08.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:08.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe792ffd700 10 client.27507.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c0fffd0 osd=1 4 2024-02-13T00:48:08.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 1 4 2024-02-13T00:48:08.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:08.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 5 2024-02-13T00:48:08.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe79c0ffb30 0x7fe79c11e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:08.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c027a60 osd=2 3 2024-02-13T00:48:08.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c027a60 osd=2 3 2024-02-13T00:48:08.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c027a60 osd=2 3 2024-02-13T00:48:08.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 5 2024-02-13T00:48:08.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c027a60 osd=2 3 2024-02-13T00:48:08.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 2 5 2024-02-13T00:48:08.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c027a60 osd=2 4 2024-02-13T00:48:08.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 5 2024-02-13T00:48:08.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.628+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe79c06d050 con 0x7fe79c0ffb30 2024-02-13T00:48:08.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.629+0000 7fe7a155b700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe79c0ffb30 0x7fe79c11e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:08.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.629+0000 7fe792ffd700 10 client.27507.objecter ms_handle_connect 0x7fe79c0ffb30 2024-02-13T00:48:08.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.629+0000 7fe7a155b700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe79c0ffb30 0x7fe79c11e550 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:08.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.629+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe79c06d050 con 0x7fe79c0ffb30 2024-02-13T00:48:08.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.629+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:48:08.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.629+0000 7fe792ffd700 10 client.27507.objecter _finish_command 5 = system:0 2024-02-13T00:48:08.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.629+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c027a60 osd=2 4 2024-02-13T00:48:08.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.629+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 2 5 2024-02-13T00:48:08.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.644+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:08.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.644+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 6 2024-02-13T00:48:08.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.644+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c027a60 osd=2 3 2024-02-13T00:48:08.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.644+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c027a60 osd=2 3 2024-02-13T00:48:08.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.644+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c027a60 osd=2 3 2024-02-13T00:48:08.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.644+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 6 2024-02-13T00:48:08.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.644+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c027a60 osd=2 3 2024-02-13T00:48:08.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.644+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 2 6 2024-02-13T00:48:08.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.644+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c027a60 osd=2 4 2024-02-13T00:48:08.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.644+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 6 2024-02-13T00:48:08.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.644+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe79c0ff0f0 con 0x7fe79c0ffb30 2024-02-13T00:48:08.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.645+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe79c0ff0f0 con 0x7fe79c0ffb30 2024-02-13T00:48:08.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.645+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:08.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.645+0000 7fe792ffd700 10 client.27507.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.645+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c027a60 osd=2 4 2024-02-13T00:48:08.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.645+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 2 6 2024-02-13T00:48:08.649 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:48:08.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:08.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 7 2024-02-13T00:48:08.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6813/604971011 conn(0x7fe79c11fbc0 0x7fe79c11ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:08.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c119790 osd=3 3 2024-02-13T00:48:08.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c119790 osd=3 3 2024-02-13T00:48:08.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c119790 osd=3 3 2024-02-13T00:48:08.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 7 2024-02-13T00:48:08.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c119790 osd=3 3 2024-02-13T00:48:08.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 3 7 2024-02-13T00:48:08.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c119790 osd=3 4 2024-02-13T00:48:08.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 7 2024-02-13T00:48:08.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe79c119ed0 con 0x7fe79c11fbc0 2024-02-13T00:48:08.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a255d700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6813/604971011 conn(0x7fe79c11fbc0 0x7fe79c11ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:08.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe792ffd700 10 client.27507.objecter ms_handle_connect 0x7fe79c11fbc0 2024-02-13T00:48:08.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe7a255d700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6813/604971011 conn(0x7fe79c11fbc0 0x7fe79c11ffe0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:08.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe79c119ed0 con 0x7fe79c11fbc0 2024-02-13T00:48:08.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:48:08.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe792ffd700 10 client.27507.objecter _finish_command 7 = system:0 2024-02-13T00:48:08.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c119790 osd=3 4 2024-02-13T00:48:08.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.646+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 3 7 2024-02-13T00:48:08.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.665+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:08.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.665+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 8 2024-02-13T00:48:08.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.665+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c119790 osd=3 3 2024-02-13T00:48:08.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.665+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c119790 osd=3 3 2024-02-13T00:48:08.668 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.665+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c119790 osd=3 3 2024-02-13T00:48:08.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.665+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 8 2024-02-13T00:48:08.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.665+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c119790 osd=3 3 2024-02-13T00:48:08.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.665+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 3 8 2024-02-13T00:48:08.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.665+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c119790 osd=3 4 2024-02-13T00:48:08.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.665+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 8 2024-02-13T00:48:08.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.665+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe79c120ca0 con 0x7fe79c11fbc0 2024-02-13T00:48:08.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.666+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe79c120ca0 con 0x7fe79c11fbc0 2024-02-13T00:48:08.669 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.666+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:08.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.666+0000 7fe792ffd700 10 client.27507.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.666+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c119790 osd=3 4 2024-02-13T00:48:08.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.666+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 3 8 2024-02-13T00:48:08.670 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:48:08.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.666+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:08.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.666+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 9 2024-02-13T00:48:08.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.666+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe79c123260 0x7fe79c125650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.667+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c1204d0 osd=4 3 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.667+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c1204d0 osd=4 3 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.667+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c1204d0 osd=4 3 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.667+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 9 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.667+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c1204d0 osd=4 3 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.667+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 4 9 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.667+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c1204d0 osd=4 4 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.667+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 9 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.667+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe79c120c80 con 0x7fe79c123260 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.667+0000 7fe7a1d5c700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe79c123260 0x7fe79c125650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.667+0000 7fe792ffd700 10 client.27507.objecter ms_handle_connect 0x7fe79c123260 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.667+0000 7fe7a1d5c700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe79c123260 0x7fe79c125650 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:08.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.668+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe79c120c80 con 0x7fe79c123260 2024-02-13T00:48:08.673 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.668+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:48:08.673 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.668+0000 7fe792ffd700 10 client.27507.objecter _finish_command 9 = system:0 2024-02-13T00:48:08.673 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.668+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c1204d0 osd=4 4 2024-02-13T00:48:08.673 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.668+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 4 9 2024-02-13T00:48:08.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.684+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:08.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.684+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 10 2024-02-13T00:48:08.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.684+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c1204d0 osd=4 3 2024-02-13T00:48:08.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.684+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c1204d0 osd=4 3 2024-02-13T00:48:08.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.684+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c1204d0 osd=4 3 2024-02-13T00:48:08.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.684+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 10 2024-02-13T00:48:08.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.684+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c1204d0 osd=4 3 2024-02-13T00:48:08.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.684+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 4 10 2024-02-13T00:48:08.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.684+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c1204d0 osd=4 4 2024-02-13T00:48:08.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.684+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 10 2024-02-13T00:48:08.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.684+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe79c120c80 con 0x7fe79c123260 2024-02-13T00:48:08.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe79c120c80 con 0x7fe79c123260 2024-02-13T00:48:08.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:08.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe792ffd700 10 client.27507.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c1204d0 osd=4 4 2024-02-13T00:48:08.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 4 10 2024-02-13T00:48:08.689 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:48:08.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:08.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 11 2024-02-13T00:48:08.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe79c126090 0x7fe79c1264f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:08.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c120c80 osd=5 3 2024-02-13T00:48:08.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c120c80 osd=5 3 2024-02-13T00:48:08.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c120c80 osd=5 3 2024-02-13T00:48:08.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 11 2024-02-13T00:48:08.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c120c80 osd=5 3 2024-02-13T00:48:08.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 5 11 2024-02-13T00:48:08.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c120c80 osd=5 4 2024-02-13T00:48:08.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 11 2024-02-13T00:48:08.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe79c12b510 con 0x7fe79c126090 2024-02-13T00:48:08.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a155b700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe79c126090 0x7fe79c1264f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:08.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe792ffd700 10 client.27507.objecter ms_handle_connect 0x7fe79c126090 2024-02-13T00:48:08.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.686+0000 7fe7a155b700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe79c126090 0x7fe79c1264f0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:08.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.687+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe79c12b510 con 0x7fe79c126090 2024-02-13T00:48:08.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.687+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:48:08.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.687+0000 7fe792ffd700 10 client.27507.objecter _finish_command 11 = system:0 2024-02-13T00:48:08.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.687+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c120c80 osd=5 4 2024-02-13T00:48:08.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.687+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 5 11 2024-02-13T00:48:08.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.701+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:08.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.701+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 12 2024-02-13T00:48:08.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.701+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c120c80 osd=5 3 2024-02-13T00:48:08.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.701+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c120c80 osd=5 3 2024-02-13T00:48:08.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.701+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c120c80 osd=5 3 2024-02-13T00:48:08.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.701+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 12 2024-02-13T00:48:08.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.701+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c120c80 osd=5 3 2024-02-13T00:48:08.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.701+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 5 12 2024-02-13T00:48:08.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.701+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c120c80 osd=5 4 2024-02-13T00:48:08.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.702+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 12 2024-02-13T00:48:08.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.702+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe79c126aa0 con 0x7fe79c126090 2024-02-13T00:48:08.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.703+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe79c126aa0 con 0x7fe79c126090 2024-02-13T00:48:08.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.703+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:08.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.703+0000 7fe792ffd700 10 client.27507.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.703+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c120c80 osd=5 4 2024-02-13T00:48:08.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.703+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 5 12 2024-02-13T00:48:08.707 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:48:08.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.703+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:08.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.703+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 13 2024-02-13T00:48:08.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe79c127180 0x7fe79c1275e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:08.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c126aa0 osd=6 3 2024-02-13T00:48:08.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c126aa0 osd=6 3 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c126aa0 osd=6 3 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 13 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c126aa0 osd=6 3 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 6 13 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c126aa0 osd=6 4 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 13 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe79c127b70 con 0x7fe79c127180 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe7a255d700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe79c127180 0x7fe79c1275e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe792ffd700 10 client.27507.objecter ms_handle_connect 0x7fe79c127180 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.704+0000 7fe7a255d700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe79c127180 0x7fe79c1275e0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.705+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe79c127b70 con 0x7fe79c127180 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.705+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.705+0000 7fe792ffd700 10 client.27507.objecter _finish_command 13 = system:0 2024-02-13T00:48:08.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.705+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c126aa0 osd=6 4 2024-02-13T00:48:08.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.705+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 6 13 2024-02-13T00:48:08.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:08.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.719+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:08.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.719+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 14 2024-02-13T00:48:08.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.719+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c126aa0 osd=6 3 2024-02-13T00:48:08.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.719+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c126aa0 osd=6 3 2024-02-13T00:48:08.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.719+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c126aa0 osd=6 3 2024-02-13T00:48:08.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.719+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 14 2024-02-13T00:48:08.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.719+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c126aa0 osd=6 3 2024-02-13T00:48:08.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.719+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 6 14 2024-02-13T00:48:08.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.719+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c126aa0 osd=6 4 2024-02-13T00:48:08.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.719+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 14 2024-02-13T00:48:08.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.719+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe79c127b70 con 0x7fe79c127180 2024-02-13T00:48:08.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.721+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe79c127b70 con 0x7fe79c127180 2024-02-13T00:48:08.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.721+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:08.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.721+0000 7fe792ffd700 10 client.27507.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.721+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c126aa0 osd=6 4 2024-02-13T00:48:08.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.721+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 6 14 2024-02-13T00:48:08.725 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:48:08.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.721+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:08.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.721+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 15 2024-02-13T00:48:08.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.721+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe79c12c8b0 0x7fe79c12eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:08.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.722+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c127b70 osd=7 3 2024-02-13T00:48:08.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.722+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c127b70 osd=7 3 2024-02-13T00:48:08.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.722+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c127b70 osd=7 3 2024-02-13T00:48:08.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.722+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 15 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.722+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c127b70 osd=7 3 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.722+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 7 15 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.722+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c127b70 osd=7 4 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.722+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 15 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.722+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe79c1318f0 con 0x7fe79c12c8b0 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.722+0000 7fe7a1d5c700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe79c12c8b0 0x7fe79c12eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.722+0000 7fe792ffd700 10 client.27507.objecter ms_handle_connect 0x7fe79c12c8b0 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.722+0000 7fe7a1d5c700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe79c12c8b0 0x7fe79c12eca0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.723+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe79c1318f0 con 0x7fe79c12c8b0 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.723+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.723+0000 7fe792ffd700 10 client.27507.objecter _finish_command 15 = system:0 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.723+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c127b70 osd=7 4 2024-02-13T00:48:08.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.723+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 7 15 2024-02-13T00:48:08.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.738+0000 7fe7a3fc0700 10 client.27507.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:08.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.738+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign -1 16 2024-02-13T00:48:08.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.738+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c127b70 osd=7 3 2024-02-13T00:48:08.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.738+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c127b70 osd=7 3 2024-02-13T00:48:08.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.738+0000 7fe7a3fc0700 20 client.27507.objecter _get_session s=0x7fe79c127b70 osd=7 3 2024-02-13T00:48:08.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.738+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_remove -1 16 2024-02-13T00:48:08.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.738+0000 7fe7a3fc0700 20 client.27507.objecter get_session s=0x7fe79c127b70 osd=7 3 2024-02-13T00:48:08.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.738+0000 7fe7a3fc0700 15 client.27507.objecter _session_command_op_assign 7 16 2024-02-13T00:48:08.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.738+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c127b70 osd=7 4 2024-02-13T00:48:08.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.738+0000 7fe7a3fc0700 10 client.27507.objecter _send_command 16 2024-02-13T00:48:08.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.738+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe79c12f190 con 0x7fe79c12c8b0 2024-02-13T00:48:08.742 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:08.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe792ffd700 1 -- 172.21.15.162:0/1780262099 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe79c12f190 con 0x7fe79c12c8b0 2024-02-13T00:48:08.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe792ffd700 10 client.27507.objecter ms_dispatch 0x7fe79c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:08.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe792ffd700 10 client.27507.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe792ffd700 20 client.27507.objecter put_session s=0x7fe79c127b70 osd=7 4 2024-02-13T00:48:08.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe792ffd700 15 client.27507.objecter _session_command_op_remove 7 16 2024-02-13T00:48:08.744 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:48:08.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 10 librados: watch_flush enter 2024-02-13T00:48:08.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 10 librados: watch_flush exit 2024-02-13T00:48:08.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 10 client.27507.objecter close_session for osd.0 2024-02-13T00:48:08.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe79c028590 msgr2=0x7fe79c0fe960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe79c028590 0x7fe79c0fe960 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c0475e0 osd=0 1 2024-02-13T00:48:08.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 10 client.27507.objecter close_session for osd.1 2024-02-13T00:48:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6805/150599780 conn(0x7fe79c027be0 msgr2=0x7fe79c027fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6805/150599780 conn(0x7fe79c027be0 0x7fe79c027fc0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c0fffd0 osd=1 1 2024-02-13T00:48:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 10 client.27507.objecter close_session for osd.2 2024-02-13T00:48:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe79c0ffb30 msgr2=0x7fe79c11e550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe79c0ffb30 0x7fe79c11e550 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c027a60 osd=2 1 2024-02-13T00:48:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 10 client.27507.objecter close_session for osd.3 2024-02-13T00:48:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6813/604971011 conn(0x7fe79c11fbc0 msgr2=0x7fe79c11ffe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6813/604971011 conn(0x7fe79c11fbc0 0x7fe79c11ffe0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c119790 osd=3 1 2024-02-13T00:48:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 10 client.27507.objecter close_session for osd.4 2024-02-13T00:48:08.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe79c123260 msgr2=0x7fe79c125650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe79c123260 0x7fe79c125650 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c1204d0 osd=4 1 2024-02-13T00:48:08.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 10 client.27507.objecter close_session for osd.5 2024-02-13T00:48:08.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe79c126090 msgr2=0x7fe79c1264f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe79c126090 0x7fe79c1264f0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c120c80 osd=5 1 2024-02-13T00:48:08.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 10 client.27507.objecter close_session for osd.6 2024-02-13T00:48:08.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe79c127180 msgr2=0x7fe79c1275e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe79c127180 0x7fe79c1275e0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c126aa0 osd=6 1 2024-02-13T00:48:08.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 10 client.27507.objecter close_session for osd.7 2024-02-13T00:48:08.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe79c12c8b0 msgr2=0x7fe79c12eca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe79c12c8b0 0x7fe79c12eca0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 20 client.27507.objecter put_session s=0x7fe79c127b70 osd=7 1 2024-02-13T00:48:08.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 20 client.27507.objecter shutdown clearing up homeless session... 2024-02-13T00:48:08.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 10 client.27507.objecter successfully canceled tick 2024-02-13T00:48:08.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a255d700 1 -- 172.21.15.162:0/1780262099 reap_dead start 2024-02-13T00:48:08.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a255d700 1 -- 172.21.15.162:0/1780262099 reap_dead start 2024-02-13T00:48:08.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a255d700 1 -- 172.21.15.162:0/1780262099 reap_dead start 2024-02-13T00:48:08.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a255d700 1 -- 172.21.15.162:0/1780262099 reap_dead start 2024-02-13T00:48:08.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a255d700 1 -- 172.21.15.162:0/1780262099 reap_dead start 2024-02-13T00:48:08.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe788062c90 msgr2=0x7fe788065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe788062c90 0x7fe788065150 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 rx=0x7fe798004e60 tx=0x7fe79800b040).stop 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe79c0fe580 msgr2=0x7fe79c1082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe79c0fe580 0x7fe79c1082b0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 rx=0x7fe794001df0 tx=0x7fe794004c50).stop 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 shutdown_connections 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe79c0fe580 0x7fe79c1082b0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 --2- 172.21.15.162:0/1780262099 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe788062c90 0x7fe788065150 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 >> 172.21.15.162:0/1780262099 conn(0x7fe79c0f3850 msgr2=0x7fe79c0fba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.740+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 shutdown_connections 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.741+0000 7fe7a3fc0700 1 -- 172.21.15.162:0/1780262099 wait complete. 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:08.741+0000 7fe7a3fc0700 1 librados: shutdown 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.750 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.750 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.750 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.750 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.750 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:48:08.765 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:07.325731+0000 osd.2 (osd.2) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:08.766 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:07.325919+0000 osd.2 (osd.2) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:08.766 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:07.490653+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e483: 8 total, 8 up, 4 in 2024-02-13T00:48:08.766 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:07.325731+0000 osd.2 (osd.2) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:08.766 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:07.325919+0000 osd.2 (osd.2) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:08.766 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:07.490653+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e483: 8 total, 8 up, 4 in 2024-02-13T00:48:08.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:07.325731+0000 osd.2 (osd.2) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:08.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:07.325919+0000 osd.2 (osd.2) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:08.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:07.490653+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e483: 8 total, 8 up, 4 in 2024-02-13T00:48:08.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:48:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:48:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:48:08.815 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:08.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:08.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:48:08.919 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:08.953 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:08.973 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:08.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:08.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:09.077 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:09.104 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:09.123 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:09.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:09.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:09.257 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:09.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:09.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:09.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:09.591 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:09.738 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:09.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:09.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:08.450499+0000 mgr.y (mgr.14544) 431 : cluster [DBG] pgmap v720: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:09.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:08.593949+0000 mon.a (mon.0) 2681 : audit [DBG] from='client.? 172.21.15.162:0/1780262099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:09.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:08.450499+0000 mgr.y (mgr.14544) 431 : cluster [DBG] pgmap v720: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:09.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:08.593949+0000 mon.a (mon.0) 2681 : audit [DBG] from='client.? 172.21.15.162:0/1780262099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:08.450499+0000 mgr.y (mgr.14544) 431 : cluster [DBG] pgmap v720: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:09.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:08.593949+0000 mon.a (mon.0) 2681 : audit [DBG] from='client.? 172.21.15.162:0/1780262099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:09.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:09.871 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:09.889 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:09.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:09.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:09.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:10.020 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:10.038 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:10.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:10.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:10.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:10.171 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:10.488 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:48:10.488 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:48:10.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14913.0:2 22.16 22:69efe4bc:test-rados-api-smithi162-84001-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.531523+0000", 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 428.53377201900003, 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.084235272999999999, 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14913", 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3501757973", 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.489 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531523+0000", 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531523+0000", 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531521+0000", 2024-02-13T00:48:10.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1880000000000008e-06 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531530+0000", 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0994e-05 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531541+0000", 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4500000000000004e-06 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531550+0000", 2024-02-13T00:48:10.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0084330910000000002 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.539983+0000", 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1860000000000003e-06 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.539990+0000", 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043643534999999997 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.583634+0000", 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011933 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.583753+0000", 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000156346 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,7", 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.583909+0000", 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026647634 2024-02-13T00:48:10.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.610557+0000", 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043537769999999996 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.614911+0000", 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00078932899999999996 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615700+0000", 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3908000000000001e-05 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615734+0000", 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4465e-05 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615758+0000", 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.084235272999999999 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.495 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24745.0:2 24.1b 24:de1a0751:test-rados-api-smithi162-84081-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.540986+0000", 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 428.524309647, 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12964152300000001, 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24745", 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/530375479", 2024-02-13T00:48:10.496 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540986+0000", 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540986+0000", 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540984+0000", 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.49e-05 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540999+0000", 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0326000000000001e-05 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.541009+0000", 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.926e-06 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.541017+0000", 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004928562 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.545945+0000", 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3000000000000001e-06 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.545951+0000", 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043217637000000003 2024-02-13T00:48:10.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589168+0000", 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011348799999999999 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589282+0000", 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0266999999999995e-05 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589372+0000", 2024-02-13T00:48:10.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020246106999999999 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.609618+0000", 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005197513 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.614816+0000", 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055733195999999999 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670549+0000", 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8872000000000002e-05 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670598+0000", 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9230999999999998e-05 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670627+0000", 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12964152300000001 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.502 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24773.0:2 11.2 11:42a49006:test-rados-api-smithi162-84105-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.602097+0000", 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 428.46319795699998, 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101585882, 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24773", 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4011549658", 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602097+0000", 2024-02-13T00:48:10.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602097+0000", 2024-02-13T00:48:10.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:48:10.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602095+0000", 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7520000000000002e-05 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602113+0000", 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1664999999999999e-05 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602124+0000", 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0750000000000004e-06 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602133+0000", 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044129360000000001 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.606546+0000", 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126408 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.606673+0000", 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002542182 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,3", 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.609215+0000", 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059528159999999997 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615168+0000", 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055524639000000001 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670692+0000", 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032890434000000003 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703583+0000", 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5461000000000003e-05 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703638+0000", 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4892000000000003e-05 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703683+0000", 2024-02-13T00:48:10.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101585882 2024-02-13T00:48:10.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.509 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.509 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24911.0:20 16.1 16:83282cb0:::1.obj2:head [write 0~90] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:48:10.509 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.474569+0000", 2024-02-13T00:48:10.509 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 427.59072661900001, 2024-02-13T00:48:10.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.158591963, 2024-02-13T00:48:10.509 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.509 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24911", 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1860840531", 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 20 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474569+0000", 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474569+0000", 2024-02-13T00:48:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474567+0000", 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9480000000000003e-06 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474577+0000", 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1072e-05 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474588+0000", 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8759999999999999e-06 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474595+0000", 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028252126999999998 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.502847+0000", 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013890300000000001 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.502986+0000", 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015514899999999999 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.503141+0000", 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016316878999999999 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.519458+0000", 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058076720999999998 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.577535+0000", 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055542411999999999 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.633077+0000", 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3886e-05 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.633121+0000", 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9544e-05 2024-02-13T00:48:10.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.633161+0000", 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.158591963 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14886.0:441 20.14 e107/105 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head v 107'198, mlcod=107'196)", 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 427.57035925899999, 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12436195799999999, 2024-02-13T00:48:10.515 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989999999999997e-06 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494942+0000", 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037274475000000001 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.532217+0000", 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2009999999999999e-06 2024-02-13T00:48:10.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.532225+0000", 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055477902000000003 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.587703+0000", 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6648999999999998e-05 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.587729+0000", 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031536963000000001 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.619266+0000", 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1473999999999997e-05 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.619298+0000", 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12436195799999999 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.519 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2064 20.19 20:9c8ffe76:test-rados-api-smithi162-84049-16::big.copy2:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.437684+0000", 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 425.62761168399999, 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088117519000000005, 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:48:10.520 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2064 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437684+0000", 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437684+0000", 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437682+0000", 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1730000000000001e-06 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437691+0000", 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4664999999999999e-05 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437706+0000", 2024-02-13T00:48:10.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6229999999999997e-06 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437713+0000", 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6378999999999994e-05 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437800+0000", 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2805000000000003e-05 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437893+0000", 2024-02-13T00:48:10.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.060281989000000001 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.498175+0000", 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034643450000000002 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.501639+0000", 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023806853999999999 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525446+0000", 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028838300000000002 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525734+0000", 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3445000000000002e-05 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525778+0000", 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3501000000000001e-05 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525801+0000", 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088117519000000005 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2069 20.1b 20:de4afda3:test-rados-api-smithi162-84049-17::big:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.630167+0000", 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 425.435128131, 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11190932200000001, 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.526 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2069 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.630167+0000", 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.630167+0000", 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.630164+0000", 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011350433 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641515+0000", 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3941e-05 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641539+0000", 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6942000000000001e-05 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641566+0000", 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0275e-05 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641646+0000", 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100653 2024-02-13T00:48:10.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641747+0000", 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137825 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,7", 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641884+0000", 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036914244999999998 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:10.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.678799+0000", 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.062888029999999998 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.741687+0000", 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032975399999999997 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742016+0000", 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4996000000000002e-05 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742051+0000", 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4997000000000001e-05 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742076+0000", 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11190932200000001 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2074 20.1b 20:de8a7420:test-rados-api-smithi162-84049-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:48:10.532 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.862849+0000", 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 425.202446302, 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101931788, 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2074 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.862849+0000", 2024-02-13T00:48:10.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.862849+0000", 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.862846+0000", 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070632639999999997 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869909+0000", 2024-02-13T00:48:10.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1322e-05 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869931+0000", 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0179e-05 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869941+0000", 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1819e-05 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869993+0000", 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010545 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.870098+0000", 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000146911 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,7", 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.870245+0000", 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029969216 2024-02-13T00:48:10.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.900214+0000", 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064237111 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964452+0000", 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025591999999999998 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964707+0000", 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8297999999999998e-05 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964746+0000", 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4975999999999999e-05 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964781+0000", 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101931788 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.538 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27739.0:2 68.6 68:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:21.788780+0000", 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 408.276515636, 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061004435000000003, 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27739", 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3559302368", 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.539 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788780+0000", 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788780+0000", 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788777+0000", 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2724999999999999e-05 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788789+0000", 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2472e-05 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788802+0000", 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3860000000000007e-06 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788810+0000", 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00058719999999999996 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.789398+0000", 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3919999999999997e-06 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.789405+0000", 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047759248999999997 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.837164+0000", 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118143 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.837282+0000", 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015534899999999999 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,7", 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.837438+0000", 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005545024 2024-02-13T00:48:10.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.842983+0000", 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059063199999999996 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.848889+0000", 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00079681100000000003 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:10.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.849686+0000", 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3878999999999997e-05 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.849740+0000", 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4291e-05 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.849784+0000", 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061004435000000003 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.545 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.5.0:281 66.6 e128/124 66:64b50987:test-rados-api-smithi162-84081-8::foo:3 v 128'5, mlcod=126'1)", 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:22.771607+0000", 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 407.29368823599998, 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033789000999999999, 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771607+0000", 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771607+0000", 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771606+0000", 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4549999999999998e-06 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771614+0000", 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7230000000000004e-06 2024-02-13T00:48:10.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771622+0000", 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2320000000000004e-06 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771629+0000", 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012838736 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784468+0000", 2024-02-13T00:48:10.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1084e-05 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784489+0000", 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020873023000000001 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.805362+0000", 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4317999999999999e-05 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.805396+0000", 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033789000999999999 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16275.0:9 66.6 e129/124 66:64b50987:test-rados-api-smithi162-84081-8::foo:head v 129'6, mlcod=127'3)", 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:23.831556+0000", 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 406.23373929100001, 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054147344, 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831556+0000", 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831556+0000", 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831554+0000", 2024-02-13T00:48:10.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.209e-05 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831576+0000", 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4198e-05 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831590+0000", 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1470000000000004e-06 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831597+0000", 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051779300000000001 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832115+0000", 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0034000000000002e-05 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832135+0000", 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053545497999999997 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.885681+0000", 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2527999999999998e-05 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.885703+0000", 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054147344 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25082.0:19 60.a e129/120 60:51f054f9:test-rados-api-smithi162-84073-8::foo:head v 129'7, mlcod=127'4)", 2024-02-13T00:48:10.554 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 406.23369091199999, 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048672976, 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:48:10.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0720000000000001e-06 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831608+0000", 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9790000000000001e-06 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831613+0000", 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.641e-06 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831617+0000", 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013029420000000001 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832920+0000", 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8600000000000001e-05 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832938+0000", 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047309385000000002 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.880247+0000", 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9788e-05 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.880277+0000", 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048672976 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16275.0:11 66.6 e130/124 66:629bc174:test-rados-api-smithi162-84081-9::foo:head v 130'7, mlcod=128'5)", 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:24.923953+0000", 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 405.14134234900001, 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.077278500999999999, 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923953+0000", 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923953+0000", 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923951+0000", 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1398e-05 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923963+0000", 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5424e-05 2024-02-13T00:48:10.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923978+0000", 2024-02-13T00:48:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4609999999999997e-06 2024-02-13T00:48:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923988+0000", 2024-02-13T00:48:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075993709000000007 2024-02-13T00:48:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.999981+0000", 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6506000000000001e-05 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.999998+0000", 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012042310000000001 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:25.001202+0000", 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9149000000000002e-05 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:25.001231+0000", 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.077278500999999999 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25162.0:109 61.1f 61:fec11214:test-rados-api-smithi162-84057-10::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:26.012720+0000", 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 404.05257511999997, 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075674577000000007, 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.563 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25162", 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 109 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.012720+0000", 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.012720+0000", 2024-02-13T00:48:10.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.012718+0000", 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6659999999999997e-06 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.012727+0000", 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042113080000000004 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.016938+0000", 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0062e-05 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.016948+0000", 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029517039999999999 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.019900+0000", 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010618700000000001 2024-02-13T00:48:10.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.020006+0000", 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130426 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.020136+0000", 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061208299999999998 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:10.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.026257+0000", 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030848077000000002 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.057105+0000", 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031192258 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.088298+0000", 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5677000000000002e-05 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.088353+0000", 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1408000000000001e-05 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.088395+0000", 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075674577000000007 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.5.0:293 66.6 e132/124 66:629bc174:test-rados-api-smithi162-84081-9::foo:7 v 132'15, mlcod=131'12)", 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:27.094916+0000", 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 402.970379708, 2024-02-13T00:48:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082889299, 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094916+0000", 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094916+0000", 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094915+0000", 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2129999999999999e-06 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094920+0000", 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1179999999999997e-06 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094927+0000", 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7989999999999996e-06 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094933+0000", 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6577999999999996e-05 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.095030+0000", 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1804e-05 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.095052+0000", 2024-02-13T00:48:10.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082724803 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.177777+0000", 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8288000000000001e-05 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.177805+0000", 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082889299 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.19047.0:9 78.0s1 78:1b3d1cfa:test-rados-api-smithi162-84073-12::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected e139)", 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:34.355949+0000", 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 395.70934580400001, 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056363352999999998, 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.19047", 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3520430640", 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 9 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355949+0000", 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355949+0000", 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999919 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355941+0000", 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4331000000000001e-05 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355956+0000", 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0973000000000001e-05 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355977+0000", 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2830000000000001e-06 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355985+0000", 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052008540000000004 2024-02-13T00:48:10.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.361186+0000", 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7719000000000005e-05 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.361274+0000", 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059102099999999995 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.361865+0000", 2024-02-13T00:48:10.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050108890000000003 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.411974+0000", 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7391000000000006e-05 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.412041+0000", 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027185499999999999 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.412313+0000", 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056363352999999998 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDECSubOpWriteReply(78.0s1 139/133 ECSubWriteReply(tid=284, last_complete=139'3, committed=1, applied=1))", 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:34.413870+0000", 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 395.65142525099998, 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033827619000000003, 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:48:10.579 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413870+0000", 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413870+0000", 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999967 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413867+0000", 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0979999999999996e-06 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413874+0000", 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3100000000000001e-06 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413882+0000", 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.49e-06 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413887+0000", 2024-02-13T00:48:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033771486000000003 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.447659+0000", 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8617e-05 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.447698+0000", 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033827619000000003 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.19047.0:10 78.0s1 78:1b3d1cfa:test-rados-api-smithi162-84073-12::foo:head [rollback 5] snapc 0=[] ondisk+write+known_if_redirected e139)", 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:34.415564+0000", 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 395.649731616, 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094033364999999994, 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.19047", 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3520430640", 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 10 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.583 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415564+0000", 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415564+0000", 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415561+0000", 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6379999999999999e-06 2024-02-13T00:48:10.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415571+0000", 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4296e-05 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415585+0000", 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6030000000000005e-06 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415594+0000", 2024-02-13T00:48:10.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032129234 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.447723+0000", 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010490200000000001 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.447828+0000", 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044360579999999997 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:48:10.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.452264+0000", 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00171576 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.453980+0000", 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055540957000000002 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.509521+0000", 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6073000000000002e-05 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.509597+0000", 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094033364999999994 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.37121.0:1 185.2s0 185:5c55be4c:::oid_RoundTripAppendPP_:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected e298)", 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:13.388163+0000", 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 236.677131858, 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0126707619999999, 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.588 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.37121", 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/695397089", 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.388163+0000", 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.388163+0000", 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.388162+0000", 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3216e-05 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.388175+0000", 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7540000000000001e-06 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.388185+0000", 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.123e-06 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.388193+0000", 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021198469999999998 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.390313+0000", 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.390316+0000", 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.007525633 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:14.397841+0000", 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010964299999999999 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:14.397951+0000", 2024-02-13T00:48:10.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037275699999999998 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:14.398324+0000", 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021763870000000001 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:14.400500+0000", 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3350999999999995e-05 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:14.400583+0000", 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025063499999999997 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:14.400834+0000", 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0126707619999999 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27024.0:1 187.6s1 187:72c88bc2:::oid_IsCompletePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e305)", 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:20.386298+0000", 2024-02-13T00:48:10.594 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 229.67899752299999, 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99356510300000001, 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27024", 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3998292204", 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386298+0000", 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:10.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386298+0000", 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386295+0000", 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7630000000000003e-06 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386302+0000", 2024-02-13T00:48:10.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6260000000000007e-06 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386312+0000", 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8469999999999996e-06 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386320+0000", 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1579000000000001e-05 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386381+0000", 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8029999999999998e-06 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386385+0000", 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99081751399999995 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:21.377203+0000", 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130801 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:21.377333+0000", 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00043861399999999999 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:21.377772+0000", 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00084405500000000002 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:21.378616+0000", 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00118315 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:21.379799+0000", 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3575e-05 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:21.379863+0000", 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99356510300000001 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:10.600 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:48:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:11.025 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:11.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:11.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-13T00:48:11.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:11.126 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:11.158 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:11.179 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:11.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:11.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:11.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:11.309 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:11.362 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:11.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:11.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:11.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:11.493 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:11.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:10.451766+0000 mgr.y (mgr.14544) 432 : cluster [DBG] pgmap v721: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:48:11.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:10.451766+0000 mgr.y (mgr.14544) 432 : cluster [DBG] pgmap v721: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:48:11.642 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:11.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:11.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:11.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:11.772 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:11.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:10.451766+0000 mgr.y (mgr.14544) 432 : cluster [DBG] pgmap v721: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:48:11.971 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:11.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:12.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:12.109 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:12.130 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:12.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:12.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:12.265 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:12.286 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:12.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:12.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:12.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:48:12.414 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:12.421 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:12.445 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:12.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:12.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:12.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:12.580 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:12.653 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:12.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:12.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:12.784 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:13.500 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6] out_osds: [3, 2, 7, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-13T00:48:13.501 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:48:13.501 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.control 2024-02-13T00:48:13.501 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.control by 10 min 8 2024-02-13T00:48:13.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:48:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.640+0000 7f1f28415700 1 -- 172.21.15.162:0/3224999595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f20104660 msgr2=0x7f1f20106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.640+0000 7f1f28415700 1 --2- 172.21.15.162:0/3224999595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f20104660 0x7f1f20106a50 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7f1f18009c60 tx=0x7f1f18004c50).stop 2024-02-13T00:48:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 -- 172.21.15.162:0/3224999595 shutdown_connections 2024-02-13T00:48:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 --2- 172.21.15.162:0/3224999595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f20104660 0x7f1f20106a50 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 --2- 172.21.15.162:0/3224999595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f200f6b00 0x7f1f200f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 --2- 172.21.15.162:0/3224999595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f200f73d0 0x7f1f20104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 -- 172.21.15.162:0/3224999595 >> 172.21.15.162:0/3224999595 conn(0x7f1f2006eb90 msgr2=0x7f1f200f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 -- 172.21.15.162:0/3224999595 shutdown_connections 2024-02-13T00:48:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 -- 172.21.15.162:0/3224999595 wait complete. 2024-02-13T00:48:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 librados: starting msgr at 2024-02-13T00:48:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 librados: starting objecter 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 Processor -- start 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 -- start start 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 librados: setting wanted keys 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 librados: calling monclient init 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.641+0000 7f1f28415700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f200f6b00 0x7f1f20110120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f28415700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f200f73d0 0x7f1f20110610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f28415700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f20104660 0x7f1f2010a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f28415700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2010a800 con 0x7f1f200f6b00 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f28415700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2010a970 con 0x7f1f200f73d0 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f28415700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2010ab30 con 0x7f1f20104660 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f261b1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f200f6b00 0x7f1f20110120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f269b2700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f20104660 0x7f1f2010a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:13.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f261b1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f200f6b00 0x7f1f20110120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:54916/0 (socket says 172.21.15.162:54916) 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f269b2700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f20104660 0x7f1f2010a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:39504/0 (socket says 172.21.15.162:39504) 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f261b1700 1 -- 172.21.15.162:0/3848313681 learned_addr learned my addr 172.21.15.162:0/3848313681 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f259b0700 1 --2- 172.21.15.162:0/3848313681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f200f73d0 0x7f1f20110610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f261b1700 1 -- 172.21.15.162:0/3848313681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f20104660 msgr2=0x7f1f2010a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f261b1700 1 --2- 172.21.15.162:0/3848313681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f20104660 0x7f1f2010a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f261b1700 1 -- 172.21.15.162:0/3848313681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f200f73d0 msgr2=0x7f1f20110610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f261b1700 1 --2- 172.21.15.162:0/3848313681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f200f73d0 0x7f1f20110610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f261b1700 1 -- 172.21.15.162:0/3848313681 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f10005040 con 0x7f1f200f6b00 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f269b2700 1 --2- 172.21.15.162:0/3848313681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f20104660 0x7f1f2010a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f261b1700 1 --2- 172.21.15.162:0/3848313681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f200f6b00 0x7f1f20110120 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7f1f1c00c7c0 tx=0x7f1f1c007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f177fe700 10 client.?.objecter ms_handle_connect 0x7f1f200f6b00 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f177fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:13.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f177fe700 1 -- 172.21.15.162:0/3848313681 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c004430 con 0x7f1f200f6b00 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f28415700 1 -- 172.21.15.162:0/3848313681 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f180097c0 con 0x7f1f200f6b00 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f177fe700 1 -- 172.21.15.162:0/3848313681 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f1c00e970 con 0x7f1f200f6b00 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f177fe700 1 -- 172.21.15.162:0/3848313681 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c013a50 con 0x7f1f200f6b00 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f28415700 10 client.27513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f28415700 1 -- 172.21.15.162:0/3848313681 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f20116f50 con 0x7f1f200f6b00 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.642+0000 7f1f28415700 1 librados: init done 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.643+0000 7f1f28415700 1 -- 172.21.15.162:0/3848313681 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f200475e0 con 0x7f1f200f6b00 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.645+0000 7f1f177fe700 1 -- 172.21.15.162:0/3848313681 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1f1c00e4a0 con 0x7f1f200f6b00 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.645+0000 7f1f177fe700 1 --2- 172.21.15.162:0/3848313681 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1f0c062c90 0x7f1f0c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.645+0000 7f1f177fe700 1 -- 172.21.15.162:0/3848313681 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(483..483 src has 52..483) v4 ==== 8083+0+0 (secure 0 0 0) 0x7f1f1c07d890 con 0x7f1f200f6b00 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.645+0000 7f1f177fe700 10 client.27513.objecter ms_dispatch 0x7f1f20001550 osd_map(483..483 src has 52..483) v4 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.645+0000 7f1f177fe700 3 client.27513.objecter handle_osd_map got epochs [483,483] > 0 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.645+0000 7f1f177fe700 3 client.27513.objecter handle_osd_map decoding full epoch 483 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.645+0000 7f1f259b0700 1 --2- 172.21.15.162:0/3848313681 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1f0c062c90 0x7f1f0c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:13.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.645+0000 7f1f177fe700 20 client.27513.objecter dump_active .. 0 homeless 2024-02-13T00:48:13.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.645+0000 7f1f177fe700 10 client.27513.objecter ms_handle_connect 0x7f1f0c062c90 2024-02-13T00:48:13.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.645+0000 7f1f259b0700 1 --2- 172.21.15.162:0/3848313681 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1f0c062c90 0x7f1f0c065150 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 rx=0x7f1f10009f20 tx=0x7f1f10009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:13.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.649+0000 7f1f177fe700 1 -- 172.21.15.162:0/3848313681 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1f1c050e20 con 0x7f1f200f6b00 2024-02-13T00:48:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:12.452148+0000 mgr.y (mgr.14544) 433 : cluster [DBG] pgmap v722: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 731 B/s rd, 0 op/s 2024-02-13T00:48:13.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:12.452148+0000 mgr.y (mgr.14544) 433 : cluster [DBG] pgmap v722: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 731 B/s rd, 0 op/s 2024-02-13T00:48:13.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:12.452148+0000 mgr.y (mgr.14544) 433 : cluster [DBG] pgmap v722: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 731 B/s rd, 0 op/s 2024-02-13T00:48:13.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:13.833+0000 7f1f28415700 1 -- 172.21.15.162:0/3848313681 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f1f20109370 con 0x7f1f200f6b00 2024-02-13T00:48:14.050 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:14.332 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:14.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:14.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:14.466 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:14.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.466+0000 7f1f177fe700 1 -- 172.21.15.162:0/3848313681 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v484) v1 ==== 91+0+0 (secure 0 0 0) 0x7f1f1c050a10 con 0x7f1f200f6b00 2024-02-13T00:48:14.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 10 librados: watch_flush enter 2024-02-13T00:48:14.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 10 librados: watch_flush exit 2024-02-13T00:48:14.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 20 client.27513.objecter shutdown clearing up homeless session... 2024-02-13T00:48:14.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 10 client.27513.objecter successfully canceled tick 2024-02-13T00:48:14.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 1 -- 172.21.15.162:0/3848313681 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1f0c062c90 msgr2=0x7f1f0c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:14.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 1 --2- 172.21.15.162:0/3848313681 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1f0c062c90 0x7f1f0c065150 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 rx=0x7f1f10009f20 tx=0x7f1f10009cb0).stop 2024-02-13T00:48:14.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 1 -- 172.21.15.162:0/3848313681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f200f6b00 msgr2=0x7f1f20110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:14.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 1 --2- 172.21.15.162:0/3848313681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f200f6b00 0x7f1f20110120 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7f1f1c00c7c0 tx=0x7f1f1c007ef0).stop 2024-02-13T00:48:14.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 1 -- 172.21.15.162:0/3848313681 shutdown_connections 2024-02-13T00:48:14.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 1 --2- 172.21.15.162:0/3848313681 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1f0c062c90 0x7f1f0c065150 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:14.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 1 --2- 172.21.15.162:0/3848313681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f20104660 0x7f1f2010a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:14.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 1 --2- 172.21.15.162:0/3848313681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f200f6b00 0x7f1f20110120 secure :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 rx=0x7f1f1c00c7c0 tx=0x7f1f1c007ef0).stop 2024-02-13T00:48:14.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 1 --2- 172.21.15.162:0/3848313681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f200f73d0 0x7f1f20110610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:14.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.469+0000 7f1f28415700 1 -- 172.21.15.162:0/3848313681 >> 172.21.15.162:0/3848313681 conn(0x7f1f2006eb90 msgr2=0x7f1f2006f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:14.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.474+0000 7f1f28415700 1 -- 172.21.15.162:0/3848313681 shutdown_connections 2024-02-13T00:48:14.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.474+0000 7f1f28415700 1 -- 172.21.15.162:0/3848313681 wait complete. 2024-02-13T00:48:14.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:14.474+0000 7f1f28415700 1 librados: shutdown 2024-02-13T00:48:14.478 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:48:14.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:13.834984+0000 mon.a (mon.0) 2682 : audit [INF] from='client.? 172.21.15.162:0/3848313681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:48:14.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:13.834984+0000 mon.a (mon.0) 2682 : audit [INF] from='client.? 172.21.15.162:0/3848313681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:48:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:13.834984+0000 mon.a (mon.0) 2682 : audit [INF] from='client.? 172.21.15.162:0/3848313681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:48:14.996 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:48:14.997 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:48:14.997 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-13T00:48:14.997 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:48:15.421 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:15.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:15.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:15.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:15.555 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:15.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:48:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:48:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:14.452537+0000 mgr.y (mgr.14544) 434 : cluster [DBG] pgmap v723: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:48:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:14.460760+0000 mon.a (mon.0) 2683 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:14.467613+0000 mon.a (mon.0) 2684 : audit [INF] from='client.? 172.21.15.162:0/3848313681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:48:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:14.467672+0000 mon.a (mon.0) 2685 : cluster [DBG] osdmap e484: 8 total, 8 up, 4 in 2024-02-13T00:48:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:15.453138+0000 mon.a (mon.0) 2686 : cluster [DBG] osdmap e485: 8 total, 8 up, 4 in 2024-02-13T00:48:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:14.452537+0000 mgr.y (mgr.14544) 434 : cluster [DBG] pgmap v723: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:48:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:14.460760+0000 mon.a (mon.0) 2683 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:14.467613+0000 mon.a (mon.0) 2684 : audit [INF] from='client.? 172.21.15.162:0/3848313681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:48:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:14.467672+0000 mon.a (mon.0) 2685 : cluster [DBG] osdmap e484: 8 total, 8 up, 4 in 2024-02-13T00:48:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:15.453138+0000 mon.a (mon.0) 2686 : cluster [DBG] osdmap e485: 8 total, 8 up, 4 in 2024-02-13T00:48:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:14.452537+0000 mgr.y (mgr.14544) 434 : cluster [DBG] pgmap v723: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:48:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:14.460760+0000 mon.a (mon.0) 2683 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:14.467613+0000 mon.a (mon.0) 2684 : audit [INF] from='client.? 172.21.15.162:0/3848313681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:48:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:14.467672+0000 mon.a (mon.0) 2685 : cluster [DBG] osdmap e484: 8 total, 8 up, 4 in 2024-02-13T00:48:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:15.453138+0000 mon.a (mon.0) 2686 : cluster [DBG] osdmap e485: 8 total, 8 up, 4 in 2024-02-13T00:48:16.471 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:48:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:48:16] "GET /metrics HTTP/1.1" 200 187995 "" "Prometheus/2.33.4" 2024-02-13T00:48:16.704 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:16.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:15.696122+0000 mgr.y (mgr.14544) 435 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:16.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:16.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:15.696122+0000 mgr.y (mgr.14544) 435 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:16.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:15.696122+0000 mgr.y (mgr.14544) 435 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:16.841 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:16.863 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:16.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:16.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:16.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:16.996 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:17.171 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:17.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-13T00:48:17.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:16.452972+0000 mgr.y (mgr.14544) 436 : cluster [DBG] pgmap v726: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:16.452972+0000 mgr.y (mgr.14544) 436 : cluster [DBG] pgmap v726: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:17.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:16.452972+0000 mgr.y (mgr.14544) 436 : cluster [DBG] pgmap v726: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:18.144 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:18.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:18.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:48:18.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:18.281 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:18.302 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:18.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:18.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:18.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:18.434 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:18.455 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:18.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:18.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:18.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:17.337893+0000 osd.6 (osd.6) 135 : cluster [DBG] 28.1a scrub starts 2024-02-13T00:48:18.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:17.340793+0000 osd.6 (osd.6) 136 : cluster [DBG] 28.1a scrub ok 2024-02-13T00:48:18.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:17.854113+0000 mon.a (mon.0) 2687 : audit [DBG] from='client.? 172.21.15.162:0/3052131972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:18.559 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:18.591 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:18.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 -- 172.21.15.162:0/2387973165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48940fd4a0 msgr2=0x7f48940f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 --2- 172.21.15.162:0/2387973165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48940fd4a0 0x7f48940f72f0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f48840047b0 tx=0x7f4884009cb0).stop 2024-02-13T00:48:18.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 -- 172.21.15.162:0/2387973165 shutdown_connections 2024-02-13T00:48:18.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 --2- 172.21.15.162:0/2387973165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48940f79d0 0x7f4894103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 --2- 172.21.15.162:0/2387973165 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48940fcb40 0x7f48940fcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 --2- 172.21.15.162:0/2387973165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48940fd4a0 0x7f48940f72f0 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 -- 172.21.15.162:0/2387973165 >> 172.21.15.162:0/2387973165 conn(0x7f4894070080 msgr2=0x7f48940f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 -- 172.21.15.162:0/2387973165 shutdown_connections 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 -- 172.21.15.162:0/2387973165 wait complete. 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 librados: starting msgr at 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 librados: starting objecter 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 Processor -- start 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 -- start start 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 librados: setting wanted keys 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.850+0000 7f48999f6700 1 librados: calling monclient init 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48999f6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48940f79d0 0x7f489410e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:18.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48999f6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48940fcb40 0x7f489410e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48999f6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48940fd4a0 0x7f4894108360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48999f6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4894108850 con 0x7f48940fd4a0 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48999f6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48941089c0 con 0x7f48940f79d0 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48999f6700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4894108b80 con 0x7f48940fcb40 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48937fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48940fd4a0 0x7f4894108360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48927fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48940fcb40 0x7f489410e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48937fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48940fd4a0 0x7f4894108360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:58290/0 (socket says 172.21.15.162:58290) 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48937fe700 1 -- 172.21.15.162:0/3052131972 learned_addr learned my addr 172.21.15.162:0/3052131972 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48927fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48940fcb40 0x7f489410e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:33274/0 (socket says 172.21.15.162:33274) 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f4892ffd700 1 --2- 172.21.15.162:0/3052131972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48940f79d0 0x7f489410e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48937fe700 1 -- 172.21.15.162:0/3052131972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48940fcb40 msgr2=0x7f489410e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48937fe700 1 --2- 172.21.15.162:0/3052131972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48940fcb40 0x7f489410e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48937fe700 1 -- 172.21.15.162:0/3052131972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48940f79d0 msgr2=0x7f489410e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48937fe700 1 --2- 172.21.15.162:0/3052131972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48940f79d0 0x7f489410e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48937fe700 1 -- 172.21.15.162:0/3052131972 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c0097c0 con 0x7f48940fd4a0 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48927fc700 1 --2- 172.21.15.162:0/3052131972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48940fcb40 0x7f489410e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48937fe700 1 --2- 172.21.15.162:0/3052131972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48940fd4a0 0x7f4894108360 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7f488c001df0 tx=0x7f488c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f487bfff700 10 client.?.objecter ms_handle_connect 0x7f48940fd4a0 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f487bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c009a10 con 0x7f48940fd4a0 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4884005040 con 0x7f48940fd4a0 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f488c0090d0 con 0x7f48940fd4a0 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c007d30 con 0x7f48940fd4a0 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48999f6700 10 client.27519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4894109170 con 0x7f48940fd4a0 2024-02-13T00:48:18.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.851+0000 7f48999f6700 1 librados: init done 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.852+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f489405fb80 con 0x7f48940fd4a0 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.853+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f488c00fb30 con 0x7f48940fd4a0 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.853+0000 7f487bfff700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6800/2866534053 conn(0x7f487c062c90 0x7f487c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.853+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(485..485 src has 52..485) v4 ==== 8083+0+0 (secure 0 0 0) 0x7f488c07da60 con 0x7f48940fd4a0 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.853+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 osd_map(485..485 src has 52..485) v4 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.853+0000 7f487bfff700 3 client.27519.objecter handle_osd_map got epochs [485,485] > 0 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.853+0000 7f487bfff700 3 client.27519.objecter handle_osd_map decoding full epoch 485 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.853+0000 7f4892ffd700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6800/2866534053 conn(0x7f487c062c90 0x7f487c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.853+0000 7f487bfff700 20 client.27519.objecter dump_active .. 0 homeless 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.853+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v485) v1 ==== 54+0+16 (secure 0 0 0) 0x7f488c082220 con 0x7f48940fd4a0 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.853+0000 7f487bfff700 10 client.27519.objecter ms_handle_connect 0x7f487c062c90 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.853+0000 7f4892ffd700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6800/2866534053 conn(0x7f487c062c90 0x7f487c065150 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 rx=0x7f48880041d0 tx=0x7f4888009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:18.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48999f6700 10 client.27519.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 1 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6801/2103743338 conn(0x7f48940ffe60 0x7f4894100240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f48940475e0 osd=0 3 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48940475e0 osd=0 3 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f48940475e0 osd=0 3 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 1 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f48940475e0 osd=0 3 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 0 1 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48940475e0 osd=0 4 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48999f6700 10 client.27519.objecter _send_command 1 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4894106080 con 0x7f48940ffe60 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48927fc700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6801/2103743338 conn(0x7f48940ffe60 0x7f4894100240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f487bfff700 10 client.27519.objecter ms_handle_connect 0x7f48940ffe60 2024-02-13T00:48:18.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f48927fc700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6801/2103743338 conn(0x7f48940ffe60 0x7f4894100240 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4894106080 con 0x7f48940ffe60 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f487bfff700 10 client.27519.objecter _finish_command 1 = system:0 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f48940475e0 osd=0 4 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.854+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 0 1 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.870+0000 7f48999f6700 10 client.27519.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.870+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 2 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.870+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f48940475e0 osd=0 3 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.870+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48940475e0 osd=0 3 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.870+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f48940475e0 osd=0 3 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.870+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 2 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.870+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f48940475e0 osd=0 3 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.870+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 0 2 2024-02-13T00:48:18.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.870+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48940475e0 osd=0 4 2024-02-13T00:48:18.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.870+0000 7f48999f6700 10 client.27519.objecter _send_command 2 2024-02-13T00:48:18.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.870+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4894027410 con 0x7f48940ffe60 2024-02-13T00:48:18.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4894027410 con 0x7f48940ffe60 2024-02-13T00:48:18.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:18.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f487bfff700 10 client.27519.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f48940475e0 osd=0 4 2024-02-13T00:48:18.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 0 2 2024-02-13T00:48:18.717 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:48:18.717 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:48:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f48999f6700 10 client.27519.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 3 2024-02-13T00:48:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6805/150599780 conn(0x7f4894059990 0x7f4894059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f4894028700 osd=1 3 2024-02-13T00:48:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894028700 osd=1 3 2024-02-13T00:48:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f4894028700 osd=1 3 2024-02-13T00:48:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 3 2024-02-13T00:48:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f4894028700 osd=1 3 2024-02-13T00:48:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 1 3 2024-02-13T00:48:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894028700 osd=1 4 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f48999f6700 10 client.27519.objecter _send_command 3 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.871+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4894028870 con 0x7f4894059990 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.872+0000 7f4892ffd700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6805/150599780 conn(0x7f4894059990 0x7f4894059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.872+0000 7f487bfff700 10 client.27519.objecter ms_handle_connect 0x7f4894059990 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.872+0000 7f4892ffd700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6805/150599780 conn(0x7f4894059990 0x7f4894059d70 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.872+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f488800fbf0 con 0x7f4894059990 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.872+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.872+0000 7f487bfff700 10 client.27519.objecter _finish_command 3 = system:0 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.872+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f4894028700 osd=1 4 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.872+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 1 3 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.888+0000 7f48999f6700 10 client.27519.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.888+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 4 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.888+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f4894028700 osd=1 3 2024-02-13T00:48:18.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.888+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894028700 osd=1 3 2024-02-13T00:48:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.888+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f4894028700 osd=1 3 2024-02-13T00:48:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.888+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 4 2024-02-13T00:48:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.888+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f4894028700 osd=1 3 2024-02-13T00:48:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.888+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 1 4 2024-02-13T00:48:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.888+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894028700 osd=1 4 2024-02-13T00:48:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.888+0000 7f48999f6700 10 client.27519.objecter _send_command 4 2024-02-13T00:48:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.888+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f48940fa7a0 con 0x7f4894059990 2024-02-13T00:48:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f48940fa7a0 con 0x7f4894059990 2024-02-13T00:48:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f487bfff700 10 client.27519.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f4894028700 osd=1 4 2024-02-13T00:48:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 1 4 2024-02-13T00:48:18.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48999f6700 10 client.27519.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:18.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 5 2024-02-13T00:48:18.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6809/3689719376 conn(0x7f48941181e0 0x7f489411ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:18.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f4894059640 osd=2 3 2024-02-13T00:48:18.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894059640 osd=2 3 2024-02-13T00:48:18.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f4894059640 osd=2 3 2024-02-13T00:48:18.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 5 2024-02-13T00:48:18.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f4894059640 osd=2 3 2024-02-13T00:48:18.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 2 5 2024-02-13T00:48:18.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894059640 osd=2 4 2024-02-13T00:48:18.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48999f6700 10 client.27519.objecter _send_command 5 2024-02-13T00:48:18.722 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:48:18.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48940597b0 con 0x7f48941181e0 2024-02-13T00:48:18.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.889+0000 7f48927fc700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6809/3689719376 conn(0x7f48941181e0 0x7f489411ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:18.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.890+0000 7f487bfff700 10 client.27519.objecter ms_handle_connect 0x7f48941181e0 2024-02-13T00:48:18.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.890+0000 7f48927fc700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6809/3689719376 conn(0x7f48941181e0 0x7f489411ea60 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:18.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.890+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f48940597b0 con 0x7f48941181e0 2024-02-13T00:48:18.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.890+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:48:18.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.890+0000 7f487bfff700 10 client.27519.objecter _finish_command 5 = system:0 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.890+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f4894059640 osd=2 4 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.890+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 2 5 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.905+0000 7f48999f6700 10 client.27519.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.905+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 6 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.905+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f4894059640 osd=2 3 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.905+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894059640 osd=2 3 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.905+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f4894059640 osd=2 3 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.905+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 6 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.905+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f4894059640 osd=2 3 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.905+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 2 6 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.905+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894059640 osd=2 4 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.905+0000 7f48999f6700 10 client.27519.objecter _send_command 6 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.905+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4894028580 con 0x7f48941181e0 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4884019040 con 0x7f48941181e0 2024-02-13T00:48:18.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f487bfff700 10 client.27519.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f4894059640 osd=2 4 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 2 6 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48999f6700 10 client.27519.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 7 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6813/604971011 conn(0x7f4894027930 0x7f48941200d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f48940277c0 osd=3 3 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48940277c0 osd=3 3 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f48940277c0 osd=3 3 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 7 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f48940277c0 osd=3 3 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 3 7 2024-02-13T00:48:18.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48940277c0 osd=3 4 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48999f6700 10 client.27519.objecter _send_command 7 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4894117ec0 con 0x7f4894027930 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.907+0000 7f48937fe700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6813/604971011 conn(0x7f4894027930 0x7f48941200d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.908+0000 7f487bfff700 10 client.27519.objecter ms_handle_connect 0x7f4894027930 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.908+0000 7f48937fe700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6813/604971011 conn(0x7f4894027930 0x7f48941200d0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.908+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4894117ec0 con 0x7f4894027930 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.908+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.908+0000 7f487bfff700 10 client.27519.objecter _finish_command 7 = system:0 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.909+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f48940277c0 osd=3 4 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.909+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 3 7 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.925+0000 7f48999f6700 10 client.27519.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.925+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 8 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.925+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f48940277c0 osd=3 3 2024-02-13T00:48:18.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.925+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48940277c0 osd=3 3 2024-02-13T00:48:18.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.925+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f48940277c0 osd=3 3 2024-02-13T00:48:18.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.925+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 8 2024-02-13T00:48:18.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.925+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f48940277c0 osd=3 3 2024-02-13T00:48:18.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.925+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 3 8 2024-02-13T00:48:18.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.925+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48940277c0 osd=3 4 2024-02-13T00:48:18.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.925+0000 7f48999f6700 10 client.27519.objecter _send_command 8 2024-02-13T00:48:18.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.925+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f48941172a0 con 0x7f4894027930 2024-02-13T00:48:18.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f48941172a0 con 0x7f4894027930 2024-02-13T00:48:18.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:18.726 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:48:18.726 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:48:18.727 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:48:18.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f487bfff700 10 client.27519.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f48940277c0 osd=3 4 2024-02-13T00:48:18.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 3 8 2024-02-13T00:48:18.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f48999f6700 10 client.27519.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:18.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 9 2024-02-13T00:48:18.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6800/3344263388 conn(0x7f48941212a0 0x7f4894125750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:18.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f48941172a0 osd=4 3 2024-02-13T00:48:18.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48941172a0 osd=4 3 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f48941172a0 osd=4 3 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 9 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f48941172a0 osd=4 3 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 4 9 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48941172a0 osd=4 4 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f48999f6700 10 client.27519.objecter _send_command 9 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4894117530 con 0x7f48941212a0 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.927+0000 7f4892ffd700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6800/3344263388 conn(0x7f48941212a0 0x7f4894125750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.928+0000 7f487bfff700 10 client.27519.objecter ms_handle_connect 0x7f48941212a0 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.928+0000 7f4892ffd700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6800/3344263388 conn(0x7f48941212a0 0x7f4894125750 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.928+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4894117530 con 0x7f48941212a0 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.928+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:48:18.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.928+0000 7f487bfff700 10 client.27519.objecter _finish_command 9 = system:0 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.928+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f48941172a0 osd=4 4 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.928+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 4 9 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.943+0000 7f48999f6700 10 client.27519.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.943+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 10 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.943+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f48941172a0 osd=4 3 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.943+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48941172a0 osd=4 3 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.943+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f48941172a0 osd=4 3 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.943+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 10 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.943+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f48941172a0 osd=4 3 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.943+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 4 10 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.943+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48941172a0 osd=4 4 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.943+0000 7f48999f6700 10 client.27519.objecter _send_command 10 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.943+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4894117530 con 0x7f48941212a0 2024-02-13T00:48:18.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4894117530 con 0x7f48941212a0 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f487bfff700 10 client.27519.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f48941172a0 osd=4 4 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 4 10 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48999f6700 10 client.27519.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 11 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4894126150 0x7f4894128580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f4894117530 osd=5 3 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894117530 osd=5 3 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f4894117530 osd=5 3 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 11 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f4894117530 osd=5 3 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 5 11 2024-02-13T00:48:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894117530 osd=5 4 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48999f6700 10 client.27519.objecter _send_command 11 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f489412b380 con 0x7f4894126150 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48927fc700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4894126150 0x7f4894128580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f487bfff700 10 client.27519.objecter ms_handle_connect 0x7f4894126150 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.946+0000 7f48927fc700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4894126150 0x7f4894128580 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.947+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f489412b380 con 0x7f4894126150 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.947+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.947+0000 7f487bfff700 10 client.27519.objecter _finish_command 11 = system:0 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.947+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f4894117530 osd=5 4 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.947+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 5 11 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.961+0000 7f48999f6700 10 client.27519.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.961+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 12 2024-02-13T00:48:18.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.961+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f4894117530 osd=5 3 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.961+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894117530 osd=5 3 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.961+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f4894117530 osd=5 3 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.961+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 12 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.961+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f4894117530 osd=5 3 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.961+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 5 12 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.961+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894117530 osd=5 4 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.961+0000 7f48999f6700 10 client.27519.objecter _send_command 12 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.961+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f489412b380 con 0x7f4894126150 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f489412b380 con 0x7f4894126150 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f487bfff700 10 client.27519.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f4894117530 osd=5 4 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 5 12 2024-02-13T00:48:18.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f48999f6700 10 client.27519.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:18.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 13 2024-02-13T00:48:18.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6808/1506640341 conn(0x7f489412a680 0x7f489412ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:18.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f489412b380 osd=6 3 2024-02-13T00:48:18.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f489412b380 osd=6 3 2024-02-13T00:48:18.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f489412b380 osd=6 3 2024-02-13T00:48:18.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 13 2024-02-13T00:48:18.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f489412b380 osd=6 3 2024-02-13T00:48:18.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 6 13 2024-02-13T00:48:18.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f489412b380 osd=6 4 2024-02-13T00:48:18.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f48999f6700 10 client.27519.objecter _send_command 13 2024-02-13T00:48:18.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.963+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f489412b120 con 0x7f489412a680 2024-02-13T00:48:18.734 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:48:18.734 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:48:18.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.964+0000 7f48937fe700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6808/1506640341 conn(0x7f489412a680 0x7f489412ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:18.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.964+0000 7f48937fe700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6808/1506640341 conn(0x7f489412a680 0x7f489412ab00 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:18.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.964+0000 7f487bfff700 10 client.27519.objecter ms_handle_connect 0x7f489412a680 2024-02-13T00:48:18.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.965+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f489412b120 con 0x7f489412a680 2024-02-13T00:48:18.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.965+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:48:18.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.965+0000 7f487bfff700 10 client.27519.objecter _finish_command 13 = system:0 2024-02-13T00:48:18.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.965+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f489412b380 osd=6 4 2024-02-13T00:48:18.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.965+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 6 13 2024-02-13T00:48:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.979+0000 7f48999f6700 10 client.27519.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.979+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 14 2024-02-13T00:48:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.979+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f489412b380 osd=6 3 2024-02-13T00:48:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.979+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f489412b380 osd=6 3 2024-02-13T00:48:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.979+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f489412b380 osd=6 3 2024-02-13T00:48:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.979+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 14 2024-02-13T00:48:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.979+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f489412b380 osd=6 3 2024-02-13T00:48:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.979+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 6 14 2024-02-13T00:48:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.979+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f489412b380 osd=6 4 2024-02-13T00:48:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.979+0000 7f48999f6700 10 client.27519.objecter _send_command 14 2024-02-13T00:48:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.979+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f489412b120 con 0x7f489412a680 2024-02-13T00:48:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.980+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f489412b120 con 0x7f489412a680 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.980+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.980+0000 7f487bfff700 10 client.27519.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.980+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f489412b380 osd=6 4 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.980+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 6 14 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f48999f6700 10 client.27519.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 15 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6812/1964192772 conn(0x7f489412dac0 0x7f489412df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f489412b120 osd=7 3 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f489412b120 osd=7 3 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f489412b120 osd=7 3 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 15 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f489412b120 osd=7 3 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 7 15 2024-02-13T00:48:18.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f489412b120 osd=7 4 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f48999f6700 10 client.27519.objecter _send_command 15 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f489412e590 con 0x7f489412dac0 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f4892ffd700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6812/1964192772 conn(0x7f489412dac0 0x7f489412df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f487bfff700 10 client.27519.objecter ms_handle_connect 0x7f489412dac0 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.981+0000 7f4892ffd700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6812/1964192772 conn(0x7f489412dac0 0x7f489412df20 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.982+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f489412e590 con 0x7f489412dac0 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.982+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.982+0000 7f487bfff700 10 client.27519.objecter _finish_command 15 = system:0 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.982+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f489412b120 osd=7 4 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.982+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 7 15 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.996+0000 7f48999f6700 10 client.27519.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.996+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign -1 16 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.996+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f489412b120 osd=7 3 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.996+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f489412b120 osd=7 3 2024-02-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.996+0000 7f48999f6700 20 client.27519.objecter _get_session s=0x7f489412b120 osd=7 3 2024-02-13T00:48:18.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.996+0000 7f48999f6700 15 client.27519.objecter _session_command_op_remove -1 16 2024-02-13T00:48:18.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.996+0000 7f48999f6700 20 client.27519.objecter get_session s=0x7f489412b120 osd=7 3 2024-02-13T00:48:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.996+0000 7f48999f6700 15 client.27519.objecter _session_command_op_assign 7 16 2024-02-13T00:48:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.996+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f489412b120 osd=7 4 2024-02-13T00:48:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.996+0000 7f48999f6700 10 client.27519.objecter _send_command 16 2024-02-13T00:48:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.996+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f489412e590 con 0x7f489412dac0 2024-02-13T00:48:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f487bfff700 1 -- 172.21.15.162:0/3052131972 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f489412e590 con 0x7f489412dac0 2024-02-13T00:48:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f487bfff700 10 client.27519.objecter ms_dispatch 0x7f4894001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f487bfff700 10 client.27519.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f487bfff700 20 client.27519.objecter put_session s=0x7f489412b120 osd=7 4 2024-02-13T00:48:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f487bfff700 15 client.27519.objecter _session_command_op_remove 7 16 2024-02-13T00:48:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 10 librados: watch_flush enter 2024-02-13T00:48:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 10 librados: watch_flush exit 2024-02-13T00:48:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 10 client.27519.objecter close_session for osd.0 2024-02-13T00:48:18.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6801/2103743338 conn(0x7f48940ffe60 msgr2=0x7f4894100240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6801/2103743338 conn(0x7f48940ffe60 0x7f4894100240 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48940475e0 osd=0 1 2024-02-13T00:48:18.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 10 client.27519.objecter close_session for osd.1 2024-02-13T00:48:18.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6805/150599780 conn(0x7f4894059990 msgr2=0x7f4894059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6805/150599780 conn(0x7f4894059990 0x7f4894059d70 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894028700 osd=1 1 2024-02-13T00:48:18.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 10 client.27519.objecter close_session for osd.2 2024-02-13T00:48:18.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6809/3689719376 conn(0x7f48941181e0 msgr2=0x7f489411ea60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6809/3689719376 conn(0x7f48941181e0 0x7f489411ea60 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894059640 osd=2 1 2024-02-13T00:48:18.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 10 client.27519.objecter close_session for osd.3 2024-02-13T00:48:18.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6813/604971011 conn(0x7f4894027930 msgr2=0x7f48941200d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6813/604971011 conn(0x7f4894027930 0x7f48941200d0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48937fe700 1 -- 172.21.15.162:0/3052131972 reap_dead start 2024-02-13T00:48:18.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48940277c0 osd=3 1 2024-02-13T00:48:18.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 10 client.27519.objecter close_session for osd.4 2024-02-13T00:48:18.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6800/3344263388 conn(0x7f48941212a0 msgr2=0x7f4894125750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6800/3344263388 conn(0x7f48941212a0 0x7f4894125750 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48937fe700 1 -- 172.21.15.162:0/3052131972 reap_dead start 2024-02-13T00:48:18.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f48941172a0 osd=4 1 2024-02-13T00:48:18.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 10 client.27519.objecter close_session for osd.5 2024-02-13T00:48:18.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4894126150 msgr2=0x7f4894128580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4894126150 0x7f4894128580 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f4894117530 osd=5 1 2024-02-13T00:48:18.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 10 client.27519.objecter close_session for osd.6 2024-02-13T00:48:18.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6808/1506640341 conn(0x7f489412a680 msgr2=0x7f489412ab00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6808/1506640341 conn(0x7f489412a680 0x7f489412ab00 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f489412b380 osd=6 1 2024-02-13T00:48:18.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 10 client.27519.objecter close_session for osd.7 2024-02-13T00:48:18.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6812/1964192772 conn(0x7f489412dac0 msgr2=0x7f489412df20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.181:6812/1964192772 conn(0x7f489412dac0 0x7f489412df20 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:18.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 20 client.27519.objecter put_session s=0x7f489412b120 osd=7 1 2024-02-13T00:48:18.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 20 client.27519.objecter shutdown clearing up homeless session... 2024-02-13T00:48:18.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 10 client.27519.objecter successfully canceled tick 2024-02-13T00:48:18.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6800/2866534053 conn(0x7f487c062c90 msgr2=0x7f487c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> v2:172.21.15.162:6800/2866534053 conn(0x7f487c062c90 0x7f487c065150 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 rx=0x7f48880041d0 tx=0x7f4888009040).stop 2024-02-13T00:48:18.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48940fd4a0 msgr2=0x7f4894108360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:18.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 --2- 172.21.15.162:0/3052131972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48940fd4a0 0x7f4894108360 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7f488c001df0 tx=0x7f488c004d10).stop 2024-02-13T00:48:18.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48937fe700 1 -- 172.21.15.162:0/3052131972 reap_dead start 2024-02-13T00:48:18.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48937fe700 1 -- 172.21.15.162:0/3052131972 reap_dead start 2024-02-13T00:48:18.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 shutdown_connections 2024-02-13T00:48:18.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.998+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 >> 172.21.15.162:0/3052131972 conn(0x7f4894070080 msgr2=0x7f48940f6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:18.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.999+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 shutdown_connections 2024-02-13T00:48:18.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.999+0000 7f48999f6700 1 -- 172.21.15.162:0/3052131972 wait complete. 2024-02-13T00:48:18.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:17.999+0000 7f48999f6700 1 librados: shutdown 2024-02-13T00:48:18.746 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.746 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.746 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.746 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.746 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.746 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.746 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.746 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:48:18.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:17.337893+0000 osd.6 (osd.6) 135 : cluster [DBG] 28.1a scrub starts 2024-02-13T00:48:18.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:17.340793+0000 osd.6 (osd.6) 136 : cluster [DBG] 28.1a scrub ok 2024-02-13T00:48:18.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:17.854113+0000 mon.a (mon.0) 2687 : audit [DBG] from='client.? 172.21.15.162:0/3052131972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:18.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:17.337893+0000 osd.6 (osd.6) 135 : cluster [DBG] 28.1a scrub starts 2024-02-13T00:48:18.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:17.340793+0000 osd.6 (osd.6) 136 : cluster [DBG] 28.1a scrub ok 2024-02-13T00:48:18.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:17.854113+0000 mon.a (mon.0) 2687 : audit [DBG] from='client.? 172.21.15.162:0/3052131972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:19.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:48:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:48:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:48:19.439 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:19.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:19.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:19.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:19.575 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:19.596 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:19.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:19.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:19.730 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:19.749 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:19.751 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:18.453312+0000 mgr.y (mgr.14544) 437 : cluster [DBG] pgmap v727: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:48:19.751 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:48:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:48:19.493+0000 7f8a56572700 -1 snap_mapper.add_oid found existing snaps mapped on 219:b9c3e6cd:test-rados-api-smithi162-84057-76::foo:2, removing 2024-02-13T00:48:19.752 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:48:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:48:19.493+0000 7f54b0765700 -1 snap_mapper.add_oid found existing snaps mapped on 219:b9c3e6cd:test-rados-api-smithi162-84057-76::foo:2, removing 2024-02-13T00:48:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:19.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:19.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 -- 172.21.15.162:0/2183400986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dac0fcb30 msgr2=0x7f8dac0fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:19.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 --2- 172.21.15.162:0/2183400986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dac0fcb30 0x7f8dac0fcf10 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 rx=0x7f8da8004870 tx=0x7f8da8009cb0).stop 2024-02-13T00:48:19.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 -- 172.21.15.162:0/2183400986 shutdown_connections 2024-02-13T00:48:19.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 --2- 172.21.15.162:0/2183400986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dac0f73a0 0x7f8dac1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:19.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 --2- 172.21.15.162:0/2183400986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dac0fcb30 0x7f8dac0fcf10 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:19.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 --2- 172.21.15.162:0/2183400986 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8dac0fd490 0x7f8dac0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:19.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 -- 172.21.15.162:0/2183400986 >> 172.21.15.162:0/2183400986 conn(0x7f8dac06eb90 msgr2=0x7f8dac0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:19.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 -- 172.21.15.162:0/2183400986 shutdown_connections 2024-02-13T00:48:19.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 -- 172.21.15.162:0/2183400986 wait complete. 2024-02-13T00:48:19.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 librados: starting msgr at 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 librados: starting objecter 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 Processor -- start 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 -- start start 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 librados: setting wanted keys 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.764+0000 7f8db32a7700 1 librados: calling monclient init 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db32a7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dac0f73a0 0x7f8dac0f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db32a7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dac0fcb30 0x7f8dac0f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db32a7700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8dac0fd490 0x7f8dac0f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db32a7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac1166a0 con 0x7f8dac0fcb30 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db32a7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac116810 con 0x7f8dac0f73a0 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db32a7700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac1169d0 con 0x7f8dac0fd490 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db0842700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dac0fcb30 0x7f8dac0f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db0842700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dac0fcb30 0x7f8dac0f8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:58306/0 (socket says 172.21.15.162:58306) 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db1844700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8dac0fd490 0x7f8dac0f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db0842700 1 -- 172.21.15.162:0/1318097025 learned_addr learned my addr 172.21.15.162:0/1318097025 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db1043700 1 --2- 172.21.15.162:0/1318097025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dac0f73a0 0x7f8dac0f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db1844700 1 -- 172.21.15.162:0/1318097025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dac0f73a0 msgr2=0x7f8dac0f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db1844700 1 --2- 172.21.15.162:0/1318097025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dac0f73a0 0x7f8dac0f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db1844700 1 -- 172.21.15.162:0/1318097025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dac0fcb30 msgr2=0x7f8dac0f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db1844700 1 --2- 172.21.15.162:0/1318097025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dac0fcb30 0x7f8dac0f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db1844700 1 -- 172.21.15.162:0/1318097025 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c005040 con 0x7f8dac0fd490 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db0842700 1 --2- 172.21.15.162:0/1318097025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dac0fcb30 0x7f8dac0f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db1844700 1 --2- 172.21.15.162:0/1318097025 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8dac0fd490 0x7f8dac0f9310 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7f8da400d940 tx=0x7f8da4004e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8da2777700 10 client.?.objecter ms_handle_connect 0x7f8dac0fd490 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8da2777700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:19.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8da2777700 1 -- 172.21.15.162:0/1318097025 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da400b4f0 con 0x7f8dac0fd490 2024-02-13T00:48:19.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db32a7700 1 -- 172.21.15.162:0/1318097025 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da8005040 con 0x7f8dac0fd490 2024-02-13T00:48:19.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8da2777700 1 -- 172.21.15.162:0/1318097025 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8da400f040 con 0x7f8dac0fd490 2024-02-13T00:48:19.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8da2777700 1 -- 172.21.15.162:0/1318097025 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da4013970 con 0x7f8dac0fd490 2024-02-13T00:48:19.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db32a7700 10 client.37625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:19.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db32a7700 1 -- 172.21.15.162:0/1318097025 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dac117010 con 0x7f8dac0fd490 2024-02-13T00:48:19.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.765+0000 7f8db32a7700 1 librados: init done 2024-02-13T00:48:19.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.766+0000 7f8db32a7700 1 -- 172.21.15.162:0/1318097025 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dac105fe0 con 0x7f8dac0fd490 2024-02-13T00:48:19.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.767+0000 7f8da2777700 1 -- 172.21.15.162:0/1318097025 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f8da400b650 con 0x7f8dac0fd490 2024-02-13T00:48:19.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.767+0000 7f8da2777700 1 --2- 172.21.15.162:0/1318097025 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8d98062c90 0x7f8d98065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:19.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.767+0000 7f8da2777700 1 -- 172.21.15.162:0/1318097025 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(485..485 src has 52..485) v4 ==== 8083+0+0 (secure 0 0 0) 0x7f8da407cc90 con 0x7f8dac0fd490 2024-02-13T00:48:19.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.767+0000 7f8da2777700 10 client.37625.objecter ms_dispatch 0x7f8dac001550 osd_map(485..485 src has 52..485) v4 2024-02-13T00:48:19.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.767+0000 7f8da2777700 3 client.37625.objecter handle_osd_map got epochs [485,485] > 0 2024-02-13T00:48:19.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.767+0000 7f8da2777700 3 client.37625.objecter handle_osd_map decoding full epoch 485 2024-02-13T00:48:19.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.767+0000 7f8db1043700 1 --2- 172.21.15.162:0/1318097025 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8d98062c90 0x7f8d98065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:19.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.767+0000 7f8da2777700 20 client.37625.objecter dump_active .. 0 homeless 2024-02-13T00:48:19.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.767+0000 7f8da2777700 10 client.37625.objecter ms_handle_connect 0x7f8d98062c90 2024-02-13T00:48:19.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.767+0000 7f8db1043700 1 --2- 172.21.15.162:0/1318097025 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8d98062c90 0x7f8d98065150 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7f8da8009f20 tx=0x7f8da800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:19.776 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.772+0000 7f8da2777700 1 -- 172.21.15.162:0/1318097025 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8da4050230 con 0x7f8dac0fd490 2024-02-13T00:48:19.793 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:19.793 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:19.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:18.453312+0000 mgr.y (mgr.14544) 437 : cluster [DBG] pgmap v727: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:48:19.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:18.453312+0000 mgr.y (mgr.14544) 437 : cluster [DBG] pgmap v727: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:48:19.830 INFO:journalctl@ceph.osd.1.smithi162.stdout:Feb 13 00:48:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-1[54323]: debug 2024-02-13T00:48:19.492+0000 7fe449bf1700 -1 snap_mapper.add_oid found existing snaps mapped on 219:b9c3e6cd:test-rados-api-smithi162-84057-76::foo:2, removing 2024-02-13T00:48:19.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:19.882 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:19.902 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:19.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:19.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:19.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.946+0000 7f8db32a7700 1 -- 172.21.15.162:0/1318097025 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8dac05fc90 con 0x7f8d98062c90 2024-02-13T00:48:19.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.961+0000 7f8da2777700 1 -- 172.21.15.162:0/1318097025 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+583763 (secure 0 0 0) 0x7f8dac05fc90 con 0x7f8d98062c90 2024-02-13T00:48:19.976 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:48:19.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 10 librados: watch_flush enter 2024-02-13T00:48:19.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 10 librados: watch_flush exit 2024-02-13T00:48:19.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 20 client.37625.objecter shutdown clearing up homeless session... 2024-02-13T00:48:19.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 10 client.37625.objecter successfully canceled tick 2024-02-13T00:48:19.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 -- 172.21.15.162:0/1318097025 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8d98062c90 msgr2=0x7f8d98065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:19.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 --2- 172.21.15.162:0/1318097025 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8d98062c90 0x7f8d98065150 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7f8da8009f20 tx=0x7f8da800b040).stop 2024-02-13T00:48:19.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 -- 172.21.15.162:0/1318097025 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8dac0fd490 msgr2=0x7f8dac0f9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:19.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 --2- 172.21.15.162:0/1318097025 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8dac0fd490 0x7f8dac0f9310 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7f8da400d940 tx=0x7f8da4004e30).stop 2024-02-13T00:48:19.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 -- 172.21.15.162:0/1318097025 shutdown_connections 2024-02-13T00:48:19.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 --2- 172.21.15.162:0/1318097025 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8d98062c90 0x7f8d98065150 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:19.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 --2- 172.21.15.162:0/1318097025 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8dac0fd490 0x7f8dac0f9310 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:19.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 --2- 172.21.15.162:0/1318097025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dac0f73a0 0x7f8dac0f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:19.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 --2- 172.21.15.162:0/1318097025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dac0fcb30 0x7f8dac0f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:19.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 -- 172.21.15.162:0/1318097025 >> 172.21.15.162:0/1318097025 conn(0x7f8dac06eb90 msgr2=0x7f8dac060e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:19.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 -- 172.21.15.162:0/1318097025 shutdown_connections 2024-02-13T00:48:19.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 -- 172.21.15.162:0/1318097025 wait complete. 2024-02-13T00:48:19.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:19.965+0000 7f8db32a7700 1 librados: shutdown 2024-02-13T00:48:19.986 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:48:20.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:20.034 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:20.053 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:20.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:20.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:20.187 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:18.749941+0000 osd.0 (osd.0) 181 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:48:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:18.751078+0000 osd.0 (osd.0) 182 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:48:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:19.494914+0000 mon.b (mon.1) 610 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:19.495296+0000 mon.a (mon.0) 2688 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:19.495702+0000 mon.b (mon.1) 611 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-75"}]: dispatch 2024-02-13T00:48:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:19.495997+0000 mon.a (mon.0) 2689 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-75"}]: dispatch 2024-02-13T00:48:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:19.948677+0000 mgr.y (mgr.14544) 438 : audit [DBG] from='client.37625 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:48:20.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:18.749941+0000 osd.0 (osd.0) 181 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:48:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:18.751078+0000 osd.0 (osd.0) 182 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:48:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:19.494914+0000 mon.b (mon.1) 610 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:19.495296+0000 mon.a (mon.0) 2688 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:19.495702+0000 mon.b (mon.1) 611 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-75"}]: dispatch 2024-02-13T00:48:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:19.495997+0000 mon.a (mon.0) 2689 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-75"}]: dispatch 2024-02-13T00:48:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:19.948677+0000 mgr.y (mgr.14544) 438 : audit [DBG] from='client.37625 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:48:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:18.749941+0000 osd.0 (osd.0) 181 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:48:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:18.751078+0000 osd.0 (osd.0) 182 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:48:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:19.494914+0000 mon.b (mon.1) 610 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:19.495296+0000 mon.a (mon.0) 2688 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:19.495702+0000 mon.b (mon.1) 611 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-75"}]: dispatch 2024-02-13T00:48:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:19.495997+0000 mon.a (mon.0) 2689 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-75"}]: dispatch 2024-02-13T00:48:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:19.948677+0000 mgr.y (mgr.14544) 438 : audit [DBG] from='client.37625 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:48:21.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:48:21.613 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:19.715226+0000 osd.0 (osd.0) 183 : cluster [DBG] 5.1b scrub starts 2024-02-13T00:48:21.614 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:19.718118+0000 osd.0 (osd.0) 184 : cluster [DBG] 5.1b scrub ok 2024-02-13T00:48:21.614 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:20.454227+0000 mgr.y (mgr.14544) 439 : cluster [DBG] pgmap v728: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.6 KiB/s rd, 127 B/s wr, 1 op/s 2024-02-13T00:48:21.614 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:20.545981+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e486: 8 total, 8 up, 4 in 2024-02-13T00:48:21.614 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:20.924546+0000 osd.1 (osd.1) 205 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:48:21.614 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:20.928154+0000 osd.1 (osd.1) 206 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:48:21.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:19.715226+0000 osd.0 (osd.0) 183 : cluster [DBG] 5.1b scrub starts 2024-02-13T00:48:21.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:19.718118+0000 osd.0 (osd.0) 184 : cluster [DBG] 5.1b scrub ok 2024-02-13T00:48:21.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:20.454227+0000 mgr.y (mgr.14544) 439 : cluster [DBG] pgmap v728: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.6 KiB/s rd, 127 B/s wr, 1 op/s 2024-02-13T00:48:21.615 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:20.545981+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e486: 8 total, 8 up, 4 in 2024-02-13T00:48:21.615 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:20.924546+0000 osd.1 (osd.1) 205 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:48:21.615 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:20.928154+0000 osd.1 (osd.1) 206 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:48:21.616 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":727,"stamp":"2024-02-13T00:48:18.453108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007449,"num_objects":401,"num_object_clones":11,"num_object_copies":1203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":2231,"num_read_kb":1851,"num_write":1547,"num_write_kb":4577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1852,"num_bytes_recovered":4017904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3053,"ondisk_log_size":3053,"up":714,"acting":714,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":714,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1277188,"kb_used_data":43208,"kb_used_omap":0,"kb_used_meta":1233920,"kb_avail":373703420,"statfs":{"total":383980142592,"available":382672302080,"internally_reserved":0,"allocated":44244992,"data_stored":29744498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1263534080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002442"},"pg_stats":[{"pgid":"219.1f","version":"0'0","reported_seq":17,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456601+0000","last_change":"2024-02-13T00:47:57.472443+0000","last_active":"2024-02-13T00:48:06.456601+0000","last_peered":"2024-02-13T00:48:06.456601+0000","last_clean":"2024-02-13T00:48:06.456601+0000","last_became_active":"2024-02-13T00:47:57.472238+0000","last_became_peered":"2024-02-13T00:47:57.472238+0000","last_unstale":"2024-02-13T00:48:06.456601+0000","last_undegraded":"2024-02-13T00:48:06.456601+0000","last_fullsized":"2024-02-13T00:48:06.456601+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"219.1e","version":"0'0","reported_seq":17,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458093+0000","last_change":"2024-02-13T00:47:57.735035+0000","last_active":"2024-02-13T00:48:06.458093+0000","last_peered":"2024-02-13T00:48:06.458093+0000","last_clean":"2024-02-13T00:48:06.458093+0000","last_became_active":"2024-02-13T00:47:57.734739+0000","last_became_peered":"2024-02-13T00:47:57.734739+0000","last_unstale":"2024-02-13T00:48:06.458093+0000","last_undegraded":"2024-02-13T00:48:06.458093+0000","last_fullsized":"2024-02-13T00:48:06.458093+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"219.1d","version":"477'8","reported_seq":29,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494200+0000","last_change":"2024-02-13T00:47:59.902625+0000","last_active":"2024-02-13T00:48:07.494200+0000","last_peered":"2024-02-13T00:48:07.494200+0000","last_clean":"2024-02-13T00:48:07.494200+0000","last_became_active":"2024-02-13T00:47:57.734597+0000","last_became_peered":"2024-02-13T00:47:57.734597+0000","last_unstale":"2024-02-13T00:48:07.494200+0000","last_undegraded":"2024-02-13T00:48:07.494200+0000","last_fullsized":"2024-02-13T00:48:07.494200+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'8","last_scrub_stamp":"2024-02-13T00:47:59.902555+0000","last_deep_scrub":"477'8","last_deep_scrub_stamp":"2024-02-13T00:47:59.902555+0000","last_clean_scrub_stamp":"2024-02-13T00:47:59.902555+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.1c","version":"0'0","reported_seq":17,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455497+0000","last_change":"2024-02-13T00:47:57.729750+0000","last_active":"2024-02-13T00:48:06.455497+0000","last_peered":"2024-02-13T00:48:06.455497+0000","last_clean":"2024-02-13T00:48:06.455497+0000","last_became_active":"2024-02-13T00:47:57.729521+0000","last_became_peered":"2024-02-13T00:47:57.729521+0000","last_unstale":"2024-02-13T00:48:06.455497+0000","last_undegraded":"2024-02-13T00:48:06.455497+0000","last_fullsized":"2024-02-13T00:48:06.455497+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"219.1b","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492911+0000","last_change":"2024-02-13T00:47:57.734793+0000","last_active":"2024-02-13T00:48:07.492911+0000","last_peered":"2024-02-13T00:48:07.492911+0000","last_clean":"2024-02-13T00:48:07.492911+0000","last_became_active":"2024-02-13T00:47:57.734444+0000","last_became_peered":"2024-02-13T00:47:57.734444+0000","last_unstale":"2024-02-13T00:48:07.492911+0000","last_undegraded":"2024-02-13T00:48:07.492911+0000","last_fullsized":"2024-02-13T00:48:07.492911+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.1a","version":"0'0","reported_seq":17,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.454762+0000","last_change":"2024-02-13T00:47:57.730456+0000","last_active":"2024-02-13T00:48:06.454762+0000","last_peered":"2024-02-13T00:48:06.454762+0000","last_clean":"2024-02-13T00:48:06.454762+0000","last_became_active":"2024-02-13T00:47:57.730182+0000","last_became_peered":"2024-02-13T00:47:57.730182+0000","last_unstale":"2024-02-13T00:48:06.454762+0000","last_undegraded":"2024-02-13T00:48:06.454762+0000","last_fullsized":"2024-02-13T00:48:06.454762+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"219.19","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722839+0000","last_change":"2024-02-13T00:47:57.478584+0000","last_active":"2024-02-13T00:48:07.722839+0000","last_peered":"2024-02-13T00:48:07.722839+0000","last_clean":"2024-02-13T00:48:07.722839+0000","last_became_active":"2024-02-13T00:47:57.478212+0000","last_became_peered":"2024-02-13T00:47:57.478212+0000","last_unstale":"2024-02-13T00:48:07.722839+0000","last_undegraded":"2024-02-13T00:48:07.722839+0000","last_fullsized":"2024-02-13T00:48:07.722839+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.18","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.491328+0000","last_change":"2024-02-13T00:47:57.734590+0000","last_active":"2024-02-13T00:48:07.491328+0000","last_peered":"2024-02-13T00:48:07.491328+0000","last_clean":"2024-02-13T00:48:07.491328+0000","last_became_active":"2024-02-13T00:47:57.734305+0000","last_became_peered":"2024-02-13T00:47:57.734305+0000","last_unstale":"2024-02-13T00:48:07.491328+0000","last_undegraded":"2024-02-13T00:48:07.491328+0000","last_fullsized":"2024-02-13T00:48:07.491328+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.17","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723630+0000","last_change":"2024-02-13T00:47:57.729686+0000","last_active":"2024-02-13T00:48:07.723630+0000","last_peered":"2024-02-13T00:48:07.723630+0000","last_clean":"2024-02-13T00:48:07.723630+0000","last_became_active":"2024-02-13T00:47:57.729382+0000","last_became_peered":"2024-02-13T00:47:57.729382+0000","last_unstale":"2024-02-13T00:48:07.723630+0000","last_undegraded":"2024-02-13T00:48:07.723630+0000","last_fullsized":"2024-02-13T00:48:07.723630+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.16","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723760+0000","last_change":"2024-02-13T00:47:57.731782+0000","last_active":"2024-02-13T00:48:07.723760+0000","last_peered":"2024-02-13T00:48:07.723760+0000","last_clean":"2024-02-13T00:48:07.723760+0000","last_became_active":"2024-02-13T00:47:57.731473+0000","last_became_peered":"2024-02-13T00:47:57.731473+0000","last_unstale":"2024-02-13T00:48:07.723760+0000","last_undegraded":"2024-02-13T00:48:07.723760+0000","last_fullsized":"2024-02-13T00:48:07.723760+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.15","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494678+0000","last_change":"2024-02-13T00:47:57.736402+0000","last_active":"2024-02-13T00:48:07.494678+0000","last_peered":"2024-02-13T00:48:07.494678+0000","last_clean":"2024-02-13T00:48:07.494678+0000","last_became_active":"2024-02-13T00:47:57.736124+0000","last_became_peered":"2024-02-13T00:47:57.736124+0000","last_unstale":"2024-02-13T00:48:07.494678+0000","last_undegraded":"2024-02-13T00:48:07.494678+0000","last_fullsized":"2024-02-13T00:48:07.494678+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.14","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722740+0000","last_change":"2024-02-13T00:47:57.471476+0000","last_active":"2024-02-13T00:48:07.722740+0000","last_peered":"2024-02-13T00:48:07.722740+0000","last_clean":"2024-02-13T00:48:07.722740+0000","last_became_active":"2024-02-13T00:47:57.471127+0000","last_became_peered":"2024-02-13T00:47:57.471127+0000","last_unstale":"2024-02-13T00:48:07.722740+0000","last_undegraded":"2024-02-13T00:48:07.722740+0000","last_fullsized":"2024-02-13T00:48:07.722740+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.13","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722357+0000","last_change":"2024-02-13T00:47:57.729727+0000","last_active":"2024-02-13T00:48:07.722357+0000","last_peered":"2024-02-13T00:48:07.722357+0000","last_clean":"2024-02-13T00:48:07.722357+0000","last_became_active":"2024-02-13T00:47:57.729449+0000","last_became_peered":"2024-02-13T00:47:57.729449+0000","last_unstale":"2024-02-13T00:48:07.722357+0000","last_undegraded":"2024-02-13T00:48:07.722357+0000","last_fullsized":"2024-02-13T00:48:07.722357+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.12","version":"0'0","reported_seq":17,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.454893+0000","last_change":"2024-02-13T00:47:57.730034+0000","last_active":"2024-02-13T00:48:06.454893+0000","last_peered":"2024-02-13T00:48:06.454893+0000","last_clean":"2024-02-13T00:48:06.454893+0000","last_became_active":"2024-02-13T00:47:57.729712+0000","last_became_peered":"2024-02-13T00:47:57.729712+0000","last_unstale":"2024-02-13T00:48:06.454893+0000","last_undegraded":"2024-02-13T00:48:06.454893+0000","last_fullsized":"2024-02-13T00:48:06.454893+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"219.11","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.491635+0000","last_change":"2024-02-13T00:47:57.479832+0000","last_active":"2024-02-13T00:48:07.491635+0000","last_peered":"2024-02-13T00:48:07.491635+0000","last_clean":"2024-02-13T00:48:07.491635+0000","last_became_active":"2024-02-13T00:47:57.479298+0000","last_became_peered":"2024-02-13T00:47:57.479298+0000","last_unstale":"2024-02-13T00:48:07.491635+0000","last_undegraded":"2024-02-13T00:48:07.491635+0000","last_fullsized":"2024-02-13T00:48:07.491635+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.10","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.721776+0000","last_change":"2024-02-13T00:47:57.735383+0000","last_active":"2024-02-13T00:48:07.721776+0000","last_peered":"2024-02-13T00:48:07.721776+0000","last_clean":"2024-02-13T00:48:07.721776+0000","last_became_active":"2024-02-13T00:47:57.735119+0000","last_became_peered":"2024-02-13T00:47:57.735119+0000","last_unstale":"2024-02-13T00:48:07.721776+0000","last_undegraded":"2024-02-13T00:48:07.721776+0000","last_fullsized":"2024-02-13T00:48:07.721776+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.f","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723696+0000","last_change":"2024-02-13T00:47:57.479412+0000","last_active":"2024-02-13T00:48:07.723696+0000","last_peered":"2024-02-13T00:48:07.723696+0000","last_clean":"2024-02-13T00:48:07.723696+0000","last_became_active":"2024-02-13T00:47:57.479120+0000","last_became_peered":"2024-02-13T00:47:57.479120+0000","last_unstale":"2024-02-13T00:48:07.723696+0000","last_undegraded":"2024-02-13T00:48:07.723696+0000","last_fullsized":"2024-02-13T00:48:07.723696+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.e","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494033+0000","last_change":"2024-02-13T00:47:57.477312+0000","last_active":"2024-02-13T00:48:07.494033+0000","last_peered":"2024-02-13T00:48:07.494033+0000","last_clean":"2024-02-13T00:48:07.494033+0000","last_became_active":"2024-02-13T00:47:57.477030+0000","last_became_peered":"2024-02-13T00:47:57.477030+0000","last_unstale":"2024-02-13T00:48:07.494033+0000","last_undegraded":"2024-02-13T00:48:07.494033+0000","last_fullsized":"2024-02-13T00:48:07.494033+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.d","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494931+0000","last_change":"2024-02-13T00:47:57.736324+0000","last_active":"2024-02-13T00:48:07.494931+0000","last_peered":"2024-02-13T00:48:07.494931+0000","last_clean":"2024-02-13T00:48:07.494931+0000","last_became_active":"2024-02-13T00:47:57.735867+0000","last_became_peered":"2024-02-13T00:47:57.735867+0000","last_unstale":"2024-02-13T00:48:07.494931+0000","last_undegraded":"2024-02-13T00:48:07.494931+0000","last_fullsized":"2024-02-13T00:48:07.494931+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.c","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722848+0000","last_change":"2024-02-13T00:47:57.731727+0000","last_active":"2024-02-13T00:48:07.722848+0000","last_peered":"2024-02-13T00:48:07.722848+0000","last_clean":"2024-02-13T00:48:07.722848+0000","last_became_active":"2024-02-13T00:47:57.731443+0000","last_became_peered":"2024-02-13T00:47:57.731443+0000","last_unstale":"2024-02-13T00:48:07.722848+0000","last_undegraded":"2024-02-13T00:48:07.722848+0000","last_fullsized":"2024-02-13T00:48:07.722848+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.b","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492986+0000","last_change":"2024-02-13T00:47:57.734656+0000","last_active":"2024-02-13T00:48:07.492986+0000","last_peered":"2024-02-13T00:48:07.492986+0000","last_clean":"2024-02-13T00:48:07.492986+0000","last_became_active":"2024-02-13T00:47:57.734376+0000","last_became_peered":"2024-02-13T00:47:57.734376+0000","last_unstale":"2024-02-13T00:48:07.492986+0000","last_undegraded":"2024-02-13T00:48:07.492986+0000","last_fullsized":"2024-02-13T00:48:07.492986+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.a","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722148+0000","last_change":"2024-02-13T00:47:57.731807+0000","last_active":"2024-02-13T00:48:07.722148+0000","last_peered":"2024-02-13T00:48:07.722148+0000","last_clean":"2024-02-13T00:48:07.722148+0000","last_became_active":"2024-02-13T00:47:57.731515+0000","last_became_peered":"2024-02-13T00:47:57.731515+0000","last_unstale":"2024-02-13T00:48:07.722148+0000","last_undegraded":"2024-02-13T00:48:07.722148+0000","last_fullsized":"2024-02-13T00:48:07.722148+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.9","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.491715+0000","last_change":"2024-02-13T00:47:57.479657+0000","last_active":"2024-02-13T00:48:07.491715+0000","last_peered":"2024-02-13T00:48:07.491715+0000","last_clean":"2024-02-13T00:48:07.491715+0000","last_became_active":"2024-02-13T00:47:57.479269+0000","last_became_peered":"2024-02-13T00:47:57.479269+0000","last_unstale":"2024-02-13T00:48:07.491715+0000","last_undegraded":"2024-02-13T00:48:07.491715+0000","last_fullsized":"2024-02-13T00:48:07.491715+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.8","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.490998+0000","last_change":"2024-02-13T00:47:57.736040+0000","last_active":"2024-02-13T00:48:07.490998+0000","last_peered":"2024-02-13T00:48:07.490998+0000","last_clean":"2024-02-13T00:48:07.490998+0000","last_became_active":"2024-02-13T00:47:57.735712+0000","last_became_peered":"2024-02-13T00:47:57.735712+0000","last_unstale":"2024-02-13T00:48:07.490998+0000","last_undegraded":"2024-02-13T00:48:07.490998+0000","last_fullsized":"2024-02-13T00:48:07.490998+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.7","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723719+0000","last_change":"2024-02-13T00:47:57.478774+0000","last_active":"2024-02-13T00:48:07.723719+0000","last_peered":"2024-02-13T00:48:07.723719+0000","last_clean":"2024-02-13T00:48:07.723719+0000","last_became_active":"2024-02-13T00:47:57.478494+0000","last_became_peered":"2024-02-13T00:47:57.478494+0000","last_unstale":"2024-02-13T00:48:07.723719+0000","last_undegraded":"2024-02-13T00:48:07.723719+0000","last_fullsized":"2024-02-13T00:48:07.723719+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.6","version":"0'0","reported_seq":17,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456883+0000","last_change":"2024-02-13T00:47:57.472811+0000","last_active":"2024-02-13T00:48:06.456883+0000","last_peered":"2024-02-13T00:48:06.456883+0000","last_clean":"2024-02-13T00:48:06.456883+0000","last_became_active":"2024-02-13T00:47:57.472522+0000","last_became_peered":"2024-02-13T00:47:57.472522+0000","last_unstale":"2024-02-13T00:48:06.456883+0000","last_undegraded":"2024-02-13T00:48:06.456883+0000","last_fullsized":"2024-02-13T00:48:06.456883+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"219.5","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.493591+0000","last_change":"2024-02-13T00:47:57.477033+0000","last_active":"2024-02-13T00:48:07.493591+0000","last_peered":"2024-02-13T00:48:07.493591+0000","last_clean":"2024-02-13T00:48:07.493591+0000","last_became_active":"2024-02-13T00:47:57.476794+0000","last_became_peered":"2024-02-13T00:47:57.476794+0000","last_unstale":"2024-02-13T00:48:07.493591+0000","last_undegraded":"2024-02-13T00:48:07.493591+0000","last_fullsized":"2024-02-13T00:48:07.493591+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.4","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722657+0000","last_change":"2024-02-13T00:47:57.731975+0000","last_active":"2024-02-13T00:48:07.722657+0000","last_peered":"2024-02-13T00:48:07.722657+0000","last_clean":"2024-02-13T00:48:07.722657+0000","last_became_active":"2024-02-13T00:47:57.731397+0000","last_became_peered":"2024-02-13T00:47:57.731397+0000","last_unstale":"2024-02-13T00:48:07.722657+0000","last_undegraded":"2024-02-13T00:48:07.722657+0000","last_fullsized":"2024-02-13T00:48:07.722657+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.3","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722376+0000","last_change":"2024-02-13T00:47:57.729657+0000","last_active":"2024-02-13T00:48:07.722376+0000","last_peered":"2024-02-13T00:48:07.722376+0000","last_clean":"2024-02-13T00:48:07.722376+0000","last_became_active":"2024-02-13T00:47:57.729323+0000","last_became_peered":"2024-02-13T00:47:57.729323+0000","last_unstale":"2024-02-13T00:48:07.722376+0000","last_undegraded":"2024-02-13T00:48:07.722376+0000","last_fullsized":"2024-02-13T00:48:07.722376+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.2","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722191+0000","last_change":"2024-02-13T00:47:57.731667+0000","last_active":"2024-02-13T00:48:07.722191+0000","last_peered":"2024-02-13T00:48:07.722191+0000","last_clean":"2024-02-13T00:48:07.722191+0000","last_became_active":"2024-02-13T00:47:57.731423+0000","last_became_peered":"2024-02-13T00:47:57.731423+0000","last_unstale":"2024-02-13T00:48:07.722191+0000","last_undegraded":"2024-02-13T00:48:07.722191+0000","last_fullsized":"2024-02-13T00:48:07.722191+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.1","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722248+0000","last_change":"2024-02-13T00:47:57.729785+0000","last_active":"2024-02-13T00:48:07.722248+0000","last_peered":"2024-02-13T00:48:07.722248+0000","last_clean":"2024-02-13T00:48:07.722248+0000","last_became_active":"2024-02-13T00:47:57.729499+0000","last_became_peered":"2024-02-13T00:47:57.729499+0000","last_unstale":"2024-02-13T00:48:07.722248+0000","last_undegraded":"2024-02-13T00:48:07.722248+0000","last_fullsized":"2024-02-13T00:48:07.722248+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"219.0","version":"0'0","reported_seq":18,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722080+0000","last_change":"2024-02-13T00:47:57.735133+0000","last_active":"2024-02-13T00:48:07.722080+0000","last_peered":"2024-02-13T00:48:07.722080+0000","last_clean":"2024-02-13T00:48:07.722080+0000","last_became_active":"2024-02-13T00:47:57.734850+0000","last_became_peered":"2024-02-13T00:47:57.734850+0000","last_unstale":"2024-02-13T00:48:07.722080+0000","last_undegraded":"2024-02-13T00:48:07.722080+0000","last_fullsized":"2024-02-13T00:48:07.722080+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:47:56.450097+0000","last_clean_scrub_stamp":"2024-02-13T00:47:56.450097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.2","version":"204'34","reported_seq":276,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456790+0000","last_change":"2024-02-13T00:46:53.301789+0000","last_active":"2024-02-13T00:48:06.456790+0000","last_peered":"2024-02-13T00:48:06.456790+0000","last_clean":"2024-02-13T00:48:06.456790+0000","last_became_active":"2024-02-13T00:46:01.448503+0000","last_became_peered":"2024-02-13T00:46:01.448503+0000","last_unstale":"2024-02-13T00:48:06.456790+0000","last_undegraded":"2024-02-13T00:48:06.456790+0000","last_fullsized":"2024-02-13T00:48:06.456790+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-02-13T00:46:19.267508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:19.267508+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.3","version":"375'104","reported_seq":685,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.493662+0000","last_change":"2024-02-13T00:46:52.541875+0000","last_active":"2024-02-13T00:48:07.493662+0000","last_peered":"2024-02-13T00:48:07.493662+0000","last_clean":"2024-02-13T00:48:07.493662+0000","last_became_active":"2024-02-13T00:45:22.426662+0000","last_became_peered":"2024-02-13T00:45:22.426662+0000","last_unstale":"2024-02-13T00:48:07.493662+0000","last_undegraded":"2024-02-13T00:48:07.493662+0000","last_fullsized":"2024-02-13T00:48:07.493662+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'104","last_scrub_stamp":"2024-02-13T00:45:34.225334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.225334+0000","log_size":104,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.0","version":"388'38","reported_seq":505,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722122+0000","last_change":"2024-02-13T00:46:52.599628+0000","last_active":"2024-02-13T00:48:07.722122+0000","last_peered":"2024-02-13T00:48:07.722122+0000","last_clean":"2024-02-13T00:48:07.722122+0000","last_became_active":"2024-02-13T00:46:01.436132+0000","last_became_peered":"2024-02-13T00:46:01.436132+0000","last_unstale":"2024-02-13T00:48:07.722122+0000","last_undegraded":"2024-02-13T00:48:07.722122+0000","last_fullsized":"2024-02-13T00:48:07.722122+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'38","last_scrub_stamp":"2024-02-13T00:45:43.553743+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-02-13T00:45:43.553743+0000","last_clean_scrub_stamp":"2024-02-13T00:45:43.553743+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000030,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4000030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.1","version":"375'35","reported_seq":499,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.491853+0000","last_change":"2024-02-13T00:46:52.539272+0000","last_active":"2024-02-13T00:48:07.491853+0000","last_peered":"2024-02-13T00:48:07.491853+0000","last_clean":"2024-02-13T00:48:07.491853+0000","last_became_active":"2024-02-13T00:45:15.420934+0000","last_became_peered":"2024-02-13T00:45:15.420934+0000","last_unstale":"2024-02-13T00:48:07.491853+0000","last_undegraded":"2024-02-13T00:48:07.491853+0000","last_fullsized":"2024-02-13T00:48:07.491853+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'34","last_scrub_stamp":"2024-02-13T00:45:29.241606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.241606+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.6","version":"400'20","reported_seq":427,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.496091+0000","last_change":"2024-02-13T00:46:52.544318+0000","last_active":"2024-02-13T00:48:07.496091+0000","last_peered":"2024-02-13T00:48:07.496091+0000","last_clean":"2024-02-13T00:48:07.496091+0000","last_became_active":"2024-02-13T00:46:01.452805+0000","last_became_peered":"2024-02-13T00:46:01.452805+0000","last_unstale":"2024-02-13T00:48:07.496091+0000","last_undegraded":"2024-02-13T00:48:07.496091+0000","last_fullsized":"2024-02-13T00:48:07.496091+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'20","last_scrub_stamp":"2024-02-13T00:46:20.166298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:20.166298+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.7","version":"270'85","reported_seq":419,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.724997+0000","last_change":"2024-02-13T00:46:52.602314+0000","last_active":"2024-02-13T00:48:07.724997+0000","last_peered":"2024-02-13T00:48:07.724997+0000","last_clean":"2024-02-13T00:48:07.724997+0000","last_became_active":"2024-02-13T00:46:01.458985+0000","last_became_peered":"2024-02-13T00:46:01.458985+0000","last_unstale":"2024-02-13T00:48:07.724997+0000","last_undegraded":"2024-02-13T00:48:07.724997+0000","last_fullsized":"2024-02-13T00:48:07.724997+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'85","last_scrub_stamp":"2024-02-13T00:45:02.230151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:02.230151+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.4","version":"445'64","reported_seq":231,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722909+0000","last_change":"2024-02-13T00:46:52.602005+0000","last_active":"2024-02-13T00:48:07.722909+0000","last_peered":"2024-02-13T00:48:07.722909+0000","last_clean":"2024-02-13T00:48:07.722909+0000","last_became_active":"2024-02-13T00:46:01.455621+0000","last_became_peered":"2024-02-13T00:46:01.455621+0000","last_unstale":"2024-02-13T00:48:07.722909+0000","last_undegraded":"2024-02-13T00:48:07.722909+0000","last_fullsized":"2024-02-13T00:48:07.722909+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'57","last_scrub_stamp":"2024-02-13T00:45:32.238130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.238130+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":64,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":84,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.5","version":"224'56","reported_seq":623,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495007+0000","last_change":"2024-02-13T00:46:52.544183+0000","last_active":"2024-02-13T00:48:07.495007+0000","last_peered":"2024-02-13T00:48:07.495007+0000","last_clean":"2024-02-13T00:48:07.495007+0000","last_became_active":"2024-02-13T00:46:01.456637+0000","last_became_peered":"2024-02-13T00:46:01.456637+0000","last_unstale":"2024-02-13T00:48:07.495007+0000","last_undegraded":"2024-02-13T00:48:07.495007+0000","last_fullsized":"2024-02-13T00:48:07.495007+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'56","last_scrub_stamp":"2024-02-13T00:45:55.071118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:55.071118+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.a","version":"472'107","reported_seq":579,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492479+0000","last_change":"2024-02-13T00:46:52.539511+0000","last_active":"2024-02-13T00:48:07.492479+0000","last_peered":"2024-02-13T00:48:07.492479+0000","last_clean":"2024-02-13T00:48:07.492479+0000","last_became_active":"2024-02-13T00:46:01.453135+0000","last_became_peered":"2024-02-13T00:46:01.453135+0000","last_unstale":"2024-02-13T00:48:07.492479+0000","last_undegraded":"2024-02-13T00:48:07.492479+0000","last_fullsized":"2024-02-13T00:48:07.492479+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'92","last_scrub_stamp":"2024-02-13T00:45:50.066059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:50.066059+0000","log_size":107,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106,"num_objects":11,"num_object_clones":2,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":38,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.b","version":"222'17","reported_seq":514,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.493543+0000","last_change":"2024-02-13T00:46:52.541543+0000","last_active":"2024-02-13T00:48:07.493543+0000","last_peered":"2024-02-13T00:48:07.493543+0000","last_clean":"2024-02-13T00:48:07.493543+0000","last_became_active":"2024-02-13T00:44:50.433096+0000","last_became_peered":"2024-02-13T00:44:50.433096+0000","last_unstale":"2024-02-13T00:48:07.493543+0000","last_undegraded":"2024-02-13T00:48:07.493543+0000","last_fullsized":"2024-02-13T00:48:07.493543+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'17","last_scrub_stamp":"2024-02-13T00:45:46.193088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.193088+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":74,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.8","version":"407'56","reported_seq":525,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722245+0000","last_change":"2024-02-13T00:46:52.600028+0000","last_active":"2024-02-13T00:48:07.722245+0000","last_peered":"2024-02-13T00:48:07.722245+0000","last_clean":"2024-02-13T00:48:07.722245+0000","last_became_active":"2024-02-13T00:46:01.445063+0000","last_became_peered":"2024-02-13T00:46:01.445063+0000","last_unstale":"2024-02-13T00:48:07.722245+0000","last_undegraded":"2024-02-13T00:48:07.722245+0000","last_fullsized":"2024-02-13T00:48:07.722245+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'53","last_scrub_stamp":"2024-02-13T00:44:30.046222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:30.046222+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.9","version":"343'36","reported_seq":504,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.491771+0000","last_change":"2024-02-13T00:46:52.539250+0000","last_active":"2024-02-13T00:48:07.491771+0000","last_peered":"2024-02-13T00:48:07.491771+0000","last_clean":"2024-02-13T00:48:07.491771+0000","last_became_active":"2024-02-13T00:45:15.421026+0000","last_became_peered":"2024-02-13T00:45:15.421026+0000","last_unstale":"2024-02-13T00:48:07.491771+0000","last_undegraded":"2024-02-13T00:48:07.491771+0000","last_fullsized":"2024-02-13T00:48:07.491771+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'36","last_scrub_stamp":"2024-02-13T00:45:42.196133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.196133+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.e","version":"434'129","reported_seq":608,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458192+0000","last_change":"2024-02-13T00:46:55.305903+0000","last_active":"2024-02-13T00:48:06.458192+0000","last_peered":"2024-02-13T00:48:06.458192+0000","last_clean":"2024-02-13T00:48:06.458192+0000","last_became_active":"2024-02-13T00:46:17.513379+0000","last_became_peered":"2024-02-13T00:46:17.513379+0000","last_unstale":"2024-02-13T00:48:06.458192+0000","last_undegraded":"2024-02-13T00:48:06.458192+0000","last_fullsized":"2024-02-13T00:48:06.458192+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'113","last_scrub_stamp":"2024-02-13T00:45:45.254008+0000","last_deep_scrub":"389'113","last_deep_scrub_stamp":"2024-02-13T00:45:45.254008+0000","last_clean_scrub_stamp":"2024-02-13T00:45:45.254008+0000","log_size":129,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":84,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.f","version":"252'51","reported_seq":487,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494413+0000","last_change":"2024-02-13T00:46:52.543890+0000","last_active":"2024-02-13T00:48:07.494413+0000","last_peered":"2024-02-13T00:48:07.494413+0000","last_clean":"2024-02-13T00:48:07.494413+0000","last_became_active":"2024-02-13T00:46:01.440972+0000","last_became_peered":"2024-02-13T00:46:01.440972+0000","last_unstale":"2024-02-13T00:48:07.494413+0000","last_undegraded":"2024-02-13T00:48:07.494413+0000","last_fullsized":"2024-02-13T00:48:07.494413+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'51","last_scrub_stamp":"2024-02-13T00:45:26.215792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.215792+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.c","version":"477'110","reported_seq":777,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456657+0000","last_change":"2024-02-13T00:46:53.303237+0000","last_active":"2024-02-13T00:48:06.456657+0000","last_peered":"2024-02-13T00:48:06.456657+0000","last_clean":"2024-02-13T00:48:06.456657+0000","last_became_active":"2024-02-13T00:46:17.514719+0000","last_became_peered":"2024-02-13T00:46:17.514719+0000","last_unstale":"2024-02-13T00:48:06.456657+0000","last_undegraded":"2024-02-13T00:48:06.456657+0000","last_fullsized":"2024-02-13T00:48:06.456657+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'106","last_scrub_stamp":"2024-02-13T00:45:31.255154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.255154+0000","log_size":110,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":37,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.d","version":"343'91","reported_seq":598,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494773+0000","last_change":"2024-02-13T00:46:52.543520+0000","last_active":"2024-02-13T00:48:07.494773+0000","last_peered":"2024-02-13T00:48:07.494773+0000","last_clean":"2024-02-13T00:48:07.494773+0000","last_became_active":"2024-02-13T00:46:01.442967+0000","last_became_peered":"2024-02-13T00:46:01.442967+0000","last_unstale":"2024-02-13T00:48:07.494773+0000","last_undegraded":"2024-02-13T00:48:07.494773+0000","last_fullsized":"2024-02-13T00:48:07.494773+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'90","last_scrub_stamp":"2024-02-13T00:44:31.080244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:31.080244+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":387,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722634+0000","last_change":"2024-02-13T00:46:01.448309+0000","last_active":"2024-02-13T00:48:07.722634+0000","last_peered":"2024-02-13T00:48:07.722634+0000","last_clean":"2024-02-13T00:48:07.722634+0000","last_became_active":"2024-02-13T00:46:01.445363+0000","last_became_peered":"2024-02-13T00:46:01.445363+0000","last_unstale":"2024-02-13T00:48:07.722634+0000","last_undegraded":"2024-02-13T00:48:07.722634+0000","last_fullsized":"2024-02-13T00:48:07.722634+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.12","version":"343'24","reported_seq":451,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723584+0000","last_change":"2024-02-13T00:46:52.600197+0000","last_active":"2024-02-13T00:48:07.723584+0000","last_peered":"2024-02-13T00:48:07.723584+0000","last_clean":"2024-02-13T00:48:07.723584+0000","last_became_active":"2024-02-13T00:46:01.433618+0000","last_became_peered":"2024-02-13T00:46:01.433618+0000","last_unstale":"2024-02-13T00:48:07.723584+0000","last_undegraded":"2024-02-13T00:48:07.723584+0000","last_fullsized":"2024-02-13T00:48:07.723584+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'22","last_scrub_stamp":"2024-02-13T00:44:34.629429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.629429+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":311,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455151+0000","last_change":"2024-02-13T00:46:01.449894+0000","last_active":"2024-02-13T00:48:06.455151+0000","last_peered":"2024-02-13T00:48:06.455151+0000","last_clean":"2024-02-13T00:48:06.455151+0000","last_became_active":"2024-02-13T00:46:01.449570+0000","last_became_peered":"2024-02-13T00:46:01.449570+0000","last_unstale":"2024-02-13T00:48:06.455151+0000","last_undegraded":"2024-02-13T00:48:06.455151+0000","last_fullsized":"2024-02-13T00:48:06.455151+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.13","version":"477'75","reported_seq":295,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457942+0000","last_change":"2024-02-13T00:46:53.304200+0000","last_active":"2024-02-13T00:48:06.457942+0000","last_peered":"2024-02-13T00:48:06.457942+0000","last_clean":"2024-02-13T00:48:06.457942+0000","last_became_active":"2024-02-13T00:45:15.447922+0000","last_became_peered":"2024-02-13T00:45:15.447922+0000","last_unstale":"2024-02-13T00:48:06.457942+0000","last_undegraded":"2024-02-13T00:48:06.457942+0000","last_fullsized":"2024-02-13T00:48:06.457942+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'73","last_scrub_stamp":"2024-02-13T00:46:22.291170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:22.291170+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":24,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.10","version":"463'91","reported_seq":678,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.454749+0000","last_change":"2024-02-13T00:46:53.300466+0000","last_active":"2024-02-13T00:48:06.454749+0000","last_peered":"2024-02-13T00:48:06.454749+0000","last_clean":"2024-02-13T00:48:06.454749+0000","last_became_active":"2024-02-13T00:46:01.440561+0000","last_became_peered":"2024-02-13T00:46:01.440561+0000","last_unstale":"2024-02-13T00:48:06.454749+0000","last_undegraded":"2024-02-13T00:48:06.454749+0000","last_fullsized":"2024-02-13T00:48:06.454749+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'73","last_scrub_stamp":"2024-02-13T00:45:27.173322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:27.173322+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":84,"num_objects":10,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.11","version":"343'37","reported_seq":383,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.491654+0000","last_change":"2024-02-13T00:46:52.538747+0000","last_active":"2024-02-13T00:48:07.491654+0000","last_peered":"2024-02-13T00:48:07.491654+0000","last_clean":"2024-02-13T00:48:07.491654+0000","last_became_active":"2024-02-13T00:44:50.443606+0000","last_became_peered":"2024-02-13T00:44:50.443606+0000","last_unstale":"2024-02-13T00:48:07.491654+0000","last_undegraded":"2024-02-13T00:48:07.491654+0000","last_fullsized":"2024-02-13T00:48:07.491654+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'37","last_scrub_stamp":"2024-02-13T00:46:22.170067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:22.170067+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.16","version":"442'81","reported_seq":590,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.724899+0000","last_change":"2024-02-13T00:46:52.603656+0000","last_active":"2024-02-13T00:48:07.724899+0000","last_peered":"2024-02-13T00:48:07.724899+0000","last_clean":"2024-02-13T00:48:07.724899+0000","last_became_active":"2024-02-13T00:46:01.449774+0000","last_became_peered":"2024-02-13T00:46:01.449774+0000","last_unstale":"2024-02-13T00:48:07.724899+0000","last_undegraded":"2024-02-13T00:48:07.724899+0000","last_fullsized":"2024-02-13T00:48:07.724899+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'75","last_scrub_stamp":"2024-02-13T00:45:46.157235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.157235+0000","log_size":81,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.17","version":"343'75","reported_seq":616,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725044+0000","last_change":"2024-02-13T00:46:52.602227+0000","last_active":"2024-02-13T00:48:07.725044+0000","last_peered":"2024-02-13T00:48:07.725044+0000","last_clean":"2024-02-13T00:48:07.725044+0000","last_became_active":"2024-02-13T00:46:01.447341+0000","last_became_peered":"2024-02-13T00:46:01.447341+0000","last_unstale":"2024-02-13T00:48:07.725044+0000","last_undegraded":"2024-02-13T00:48:07.725044+0000","last_fullsized":"2024-02-13T00:48:07.725044+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'75","last_scrub_stamp":"2024-02-13T00:45:36.518979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:36.518979+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.14","version":"200'17","reported_seq":496,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.724780+0000","last_change":"2024-02-13T00:46:52.602069+0000","last_active":"2024-02-13T00:48:07.724780+0000","last_peered":"2024-02-13T00:48:07.724780+0000","last_clean":"2024-02-13T00:48:07.724780+0000","last_became_active":"2024-02-13T00:46:01.455179+0000","last_became_peered":"2024-02-13T00:46:01.455179+0000","last_unstale":"2024-02-13T00:48:07.724780+0000","last_undegraded":"2024-02-13T00:48:07.724780+0000","last_fullsized":"2024-02-13T00:48:07.724780+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'17","last_scrub_stamp":"2024-02-13T00:45:37.548974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.548974+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.15","version":"423'99","reported_seq":355,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494379+0000","last_change":"2024-02-13T00:46:52.543515+0000","last_active":"2024-02-13T00:48:07.494379+0000","last_peered":"2024-02-13T00:48:07.494379+0000","last_clean":"2024-02-13T00:48:07.494379+0000","last_became_active":"2024-02-13T00:46:01.452384+0000","last_became_peered":"2024-02-13T00:46:01.452384+0000","last_unstale":"2024-02-13T00:48:07.494379+0000","last_undegraded":"2024-02-13T00:48:07.494379+0000","last_fullsized":"2024-02-13T00:48:07.494379+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'78","last_scrub_stamp":"2024-02-13T00:45:36.188466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:36.188466+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":177,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.491542+0000","last_change":"2024-02-13T00:46:01.445141+0000","last_active":"2024-02-13T00:48:07.491542+0000","last_peered":"2024-02-13T00:48:07.491542+0000","last_clean":"2024-02-13T00:48:07.491542+0000","last_became_active":"2024-02-13T00:46:01.444729+0000","last_became_peered":"2024-02-13T00:46:01.444729+0000","last_unstale":"2024-02-13T00:48:07.491542+0000","last_undegraded":"2024-02-13T00:48:07.491542+0000","last_fullsized":"2024-02-13T00:48:07.491542+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1a","version":"403'54","reported_seq":268,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456750+0000","last_change":"2024-02-13T00:46:53.301730+0000","last_active":"2024-02-13T00:48:06.456750+0000","last_peered":"2024-02-13T00:48:06.456750+0000","last_clean":"2024-02-13T00:48:06.456750+0000","last_became_active":"2024-02-13T00:45:15.436908+0000","last_became_peered":"2024-02-13T00:45:15.436908+0000","last_unstale":"2024-02-13T00:48:06.456750+0000","last_undegraded":"2024-02-13T00:48:06.456750+0000","last_fullsized":"2024-02-13T00:48:06.456750+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'54","last_scrub_stamp":"2024-02-13T00:46:26.235259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:26.235259+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":255,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":262,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.491407+0000","last_change":"2024-02-13T00:45:15.426658+0000","last_active":"2024-02-13T00:48:07.491407+0000","last_peered":"2024-02-13T00:48:07.491407+0000","last_clean":"2024-02-13T00:48:07.491407+0000","last_became_active":"2024-02-13T00:45:15.426109+0000","last_became_peered":"2024-02-13T00:45:15.426109+0000","last_unstale":"2024-02-13T00:48:07.491407+0000","last_undegraded":"2024-02-13T00:48:07.491407+0000","last_fullsized":"2024-02-13T00:48:07.491407+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1b","version":"397'54","reported_seq":275,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456716+0000","last_change":"2024-02-13T00:46:53.302763+0000","last_active":"2024-02-13T00:48:06.456716+0000","last_peered":"2024-02-13T00:48:06.456716+0000","last_clean":"2024-02-13T00:48:06.456716+0000","last_became_active":"2024-02-13T00:46:01.458733+0000","last_became_peered":"2024-02-13T00:46:01.458733+0000","last_unstale":"2024-02-13T00:48:06.456716+0000","last_undegraded":"2024-02-13T00:48:06.456716+0000","last_fullsized":"2024-02-13T00:48:06.456716+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'54","last_scrub_stamp":"2024-02-13T00:45:57.130510+0000","last_deep_scrub":"397'54","last_deep_scrub_stamp":"2024-02-13T00:45:57.130510+0000","last_clean_scrub_stamp":"2024-02-13T00:45:57.130510+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":16,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":189,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":167,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455889+0000","last_change":"2024-02-13T00:44:39.526065+0000","last_active":"2024-02-13T00:48:06.455889+0000","last_peered":"2024-02-13T00:48:06.455889+0000","last_clean":"2024-02-13T00:48:06.455889+0000","last_became_active":"2024-02-13T00:44:39.525161+0000","last_became_peered":"2024-02-13T00:44:39.525161+0000","last_unstale":"2024-02-13T00:48:06.455889+0000","last_undegraded":"2024-02-13T00:48:06.455889+0000","last_fullsized":"2024-02-13T00:48:06.455889+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.18","version":"204'34","reported_seq":277,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.491390+0000","last_change":"2024-02-13T00:46:52.538637+0000","last_active":"2024-02-13T00:48:07.491390+0000","last_peered":"2024-02-13T00:48:07.491390+0000","last_clean":"2024-02-13T00:48:07.491390+0000","last_became_active":"2024-02-13T00:46:01.438318+0000","last_became_peered":"2024-02-13T00:46:01.438318+0000","last_unstale":"2024-02-13T00:48:07.491390+0000","last_undegraded":"2024-02-13T00:48:07.491390+0000","last_fullsized":"2024-02-13T00:48:07.491390+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-02-13T00:45:29.218592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:29.218592+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":265,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492533+0000","last_change":"2024-02-13T00:48:06.705421+0000","last_active":"2024-02-13T00:48:07.492533+0000","last_peered":"2024-02-13T00:48:07.492533+0000","last_clean":"2024-02-13T00:48:07.492533+0000","last_became_active":"2024-02-13T00:48:06.705121+0000","last_became_peered":"2024-02-13T00:48:06.705121+0000","last_unstale":"2024-02-13T00:48:07.492533+0000","last_undegraded":"2024-02-13T00:48:07.492533+0000","last_fullsized":"2024-02-13T00:48:07.492533+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.19","version":"456'35","reported_seq":446,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722855+0000","last_change":"2024-02-13T00:46:52.599919+0000","last_active":"2024-02-13T00:48:07.722855+0000","last_peered":"2024-02-13T00:48:07.722855+0000","last_clean":"2024-02-13T00:48:07.722855+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:48:07.722855+0000","last_undegraded":"2024-02-13T00:48:07.722855+0000","last_fullsized":"2024-02-13T00:48:07.722855+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'18","last_scrub_stamp":"2024-02-13T00:45:51.098914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:51.098914+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":185,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458067+0000","last_change":"2024-02-13T00:46:01.459772+0000","last_active":"2024-02-13T00:48:06.458067+0000","last_peered":"2024-02-13T00:48:06.458067+0000","last_clean":"2024-02-13T00:48:06.458067+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:48:06.458067+0000","last_undegraded":"2024-02-13T00:48:06.458067+0000","last_fullsized":"2024-02-13T00:48:06.458067+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1e","version":"403'57","reported_seq":321,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458271+0000","last_change":"2024-02-13T00:46:53.304309+0000","last_active":"2024-02-13T00:48:06.458271+0000","last_peered":"2024-02-13T00:48:06.458271+0000","last_clean":"2024-02-13T00:48:06.458271+0000","last_became_active":"2024-02-13T00:45:15.446531+0000","last_became_peered":"2024-02-13T00:45:15.446531+0000","last_unstale":"2024-02-13T00:48:06.458271+0000","last_undegraded":"2024-02-13T00:48:06.458271+0000","last_fullsized":"2024-02-13T00:48:06.458271+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'57","last_scrub_stamp":"2024-02-13T00:46:33.286813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:33.286813+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":444,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456893+0000","last_change":"2024-02-13T00:46:01.463062+0000","last_active":"2024-02-13T00:48:06.456893+0000","last_peered":"2024-02-13T00:48:06.456893+0000","last_clean":"2024-02-13T00:48:06.456893+0000","last_became_active":"2024-02-13T00:46:01.462319+0000","last_became_peered":"2024-02-13T00:46:01.462319+0000","last_unstale":"2024-02-13T00:48:06.456893+0000","last_undegraded":"2024-02-13T00:48:06.456893+0000","last_fullsized":"2024-02-13T00:48:06.456893+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1f","version":"392'68","reported_seq":509,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725085+0000","last_change":"2024-02-13T00:46:52.602905+0000","last_active":"2024-02-13T00:48:07.725085+0000","last_peered":"2024-02-13T00:48:07.725085+0000","last_clean":"2024-02-13T00:48:07.725085+0000","last_became_active":"2024-02-13T00:44:50.441446+0000","last_became_peered":"2024-02-13T00:44:50.441446+0000","last_unstale":"2024-02-13T00:48:07.725085+0000","last_undegraded":"2024-02-13T00:48:07.725085+0000","last_fullsized":"2024-02-13T00:48:07.725085+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'68","last_scrub_stamp":"2024-02-13T00:45:51.651226+0000","last_deep_scrub":"392'68","last_deep_scrub_stamp":"2024-02-13T00:45:51.651226+0000","last_clean_scrub_stamp":"2024-02-13T00:45:51.651226+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":167,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457202+0000","last_change":"2024-02-13T00:44:39.525601+0000","last_active":"2024-02-13T00:48:06.457202+0000","last_peered":"2024-02-13T00:48:06.457202+0000","last_clean":"2024-02-13T00:48:06.457202+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:48:06.457202+0000","last_undegraded":"2024-02-13T00:48:06.457202+0000","last_fullsized":"2024-02-13T00:48:06.457202+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1c","version":"397'56","reported_seq":607,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456761+0000","last_change":"2024-02-13T00:46:53.304111+0000","last_active":"2024-02-13T00:48:06.456761+0000","last_peered":"2024-02-13T00:48:06.456761+0000","last_clean":"2024-02-13T00:48:06.456761+0000","last_became_active":"2024-02-13T00:46:01.462341+0000","last_became_peered":"2024-02-13T00:46:01.462341+0000","last_unstale":"2024-02-13T00:48:06.456761+0000","last_undegraded":"2024-02-13T00:48:06.456761+0000","last_fullsized":"2024-02-13T00:48:06.456761+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'52","last_scrub_stamp":"2024-02-13T00:45:44.205380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:45:44.205380+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":253,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.496077+0000","last_change":"2024-02-13T00:46:01.455629+0000","last_active":"2024-02-13T00:48:07.496077+0000","last_peered":"2024-02-13T00:48:07.496077+0000","last_clean":"2024-02-13T00:48:07.496077+0000","last_became_active":"2024-02-13T00:46:01.452801+0000","last_became_peered":"2024-02-13T00:46:01.452801+0000","last_unstale":"2024-02-13T00:48:07.496077+0000","last_undegraded":"2024-02-13T00:48:07.496077+0000","last_fullsized":"2024-02-13T00:48:07.496077+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1d","version":"343'69","reported_seq":599,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494859+0000","last_change":"2024-02-13T00:46:52.543895+0000","last_active":"2024-02-13T00:48:07.494859+0000","last_peered":"2024-02-13T00:48:07.494859+0000","last_clean":"2024-02-13T00:48:07.494859+0000","last_became_active":"2024-02-13T00:45:15.435242+0000","last_became_peered":"2024-02-13T00:45:15.435242+0000","last_unstale":"2024-02-13T00:48:07.494859+0000","last_undegraded":"2024-02-13T00:48:07.494859+0000","last_fullsized":"2024-02-13T00:48:07.494859+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'69","last_scrub_stamp":"2024-02-13T00:46:28.118117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:15.634020+0000","last_clean_scrub_stamp":"2024-02-13T00:46:28.118117+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":408,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.721609+0000","last_change":"2024-02-13T00:45:39.562095+0000","last_active":"2024-02-13T00:48:07.721609+0000","last_peered":"2024-02-13T00:48:07.721609+0000","last_clean":"2024-02-13T00:48:07.721609+0000","last_became_active":"2024-02-13T00:44:39.505121+0000","last_became_peered":"2024-02-13T00:44:39.505121+0000","last_unstale":"2024-02-13T00:48:07.721609+0000","last_undegraded":"2024-02-13T00:48:07.721609+0000","last_fullsized":"2024-02-13T00:48:07.721609+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:39.561994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:39.561994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"363'14","reported_seq":235,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:09.334245+0000","last_change":"2024-02-13T00:45:53.115355+0000","last_active":"2024-02-13T00:48:09.334245+0000","last_peered":"2024-02-13T00:48:09.334245+0000","last_clean":"2024-02-13T00:48:09.334245+0000","last_became_active":"2024-02-13T00:45:15.435476+0000","last_became_peered":"2024-02-13T00:45:15.435476+0000","last_unstale":"2024-02-13T00:48:09.334245+0000","last_undegraded":"2024-02-13T00:48:09.334245+0000","last_fullsized":"2024-02-13T00:48:09.334245+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:45:53.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.115260+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":511,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.454928+0000","last_change":"2024-02-13T00:47:03.491720+0000","last_active":"2024-02-13T00:48:06.454928+0000","last_peered":"2024-02-13T00:48:06.454928+0000","last_clean":"2024-02-13T00:48:06.454928+0000","last_became_active":"2024-02-13T00:46:01.438056+0000","last_became_peered":"2024-02-13T00:46:01.438056+0000","last_unstale":"2024-02-13T00:48:06.454928+0000","last_undegraded":"2024-02-13T00:48:06.454928+0000","last_fullsized":"2024-02-13T00:48:06.454928+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:47:03.491647+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":476,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458110+0000","last_change":"2024-02-13T00:46:01.464695+0000","last_active":"2024-02-13T00:48:06.458110+0000","last_peered":"2024-02-13T00:48:06.458110+0000","last_clean":"2024-02-13T00:48:06.458110+0000","last_became_active":"2024-02-13T00:46:01.460811+0000","last_became_peered":"2024-02-13T00:46:01.460811+0000","last_unstale":"2024-02-13T00:48:06.458110+0000","last_undegraded":"2024-02-13T00:48:06.458110+0000","last_fullsized":"2024-02-13T00:48:06.458110+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.340080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.340080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":387,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455457+0000","last_change":"2024-02-13T00:47:34.332791+0000","last_active":"2024-02-13T00:48:06.455457+0000","last_peered":"2024-02-13T00:48:06.455457+0000","last_clean":"2024-02-13T00:48:06.455457+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:48:06.455457+0000","last_undegraded":"2024-02-13T00:48:06.455457+0000","last_fullsized":"2024-02-13T00:48:06.455457+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:34.332704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:34.332704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":425,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455603+0000","last_change":"2024-02-13T00:46:01.448237+0000","last_active":"2024-02-13T00:48:06.455603+0000","last_peered":"2024-02-13T00:48:06.455603+0000","last_clean":"2024-02-13T00:48:06.455603+0000","last_became_active":"2024-02-13T00:46:01.446660+0000","last_became_peered":"2024-02-13T00:46:01.446660+0000","last_unstale":"2024-02-13T00:48:06.455603+0000","last_undegraded":"2024-02-13T00:48:06.455603+0000","last_fullsized":"2024-02-13T00:48:06.455603+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.260871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.260871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":487,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723881+0000","last_change":"2024-02-13T00:47:41.808333+0000","last_active":"2024-02-13T00:48:07.723881+0000","last_peered":"2024-02-13T00:48:07.723881+0000","last_clean":"2024-02-13T00:48:07.723881+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:48:07.723881+0000","last_undegraded":"2024-02-13T00:48:07.723881+0000","last_fullsized":"2024-02-13T00:48:07.723881+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:47:41.808259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:41.808259+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":463,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495641+0000","last_change":"2024-02-13T00:44:50.430435+0000","last_active":"2024-02-13T00:48:07.495641+0000","last_peered":"2024-02-13T00:48:07.495641+0000","last_clean":"2024-02-13T00:48:07.495641+0000","last_became_active":"2024-02-13T00:44:50.429994+0000","last_became_peered":"2024-02-13T00:44:50.429994+0000","last_unstale":"2024-02-13T00:48:07.495641+0000","last_undegraded":"2024-02-13T00:48:07.495641+0000","last_fullsized":"2024-02-13T00:48:07.495641+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:20.102471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:20.102471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.461844+0000","last_change":"2024-02-13T00:48:06.461618+0000","last_active":"2024-02-13T00:48:06.461844+0000","last_peered":"2024-02-13T00:48:06.461844+0000","last_clean":"2024-02-13T00:48:06.461844+0000","last_became_active":"2024-02-13T00:48:06.461426+0000","last_became_peered":"2024-02-13T00:48:06.461426+0000","last_unstale":"2024-02-13T00:48:06.461844+0000","last_undegraded":"2024-02-13T00:48:06.461844+0000","last_fullsized":"2024-02-13T00:48:06.461844+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.512181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.512181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":465,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723158+0000","last_change":"2024-02-13T00:45:15.435610+0000","last_active":"2024-02-13T00:48:07.723158+0000","last_peered":"2024-02-13T00:48:07.723158+0000","last_clean":"2024-02-13T00:48:07.723158+0000","last_became_active":"2024-02-13T00:45:15.434853+0000","last_became_peered":"2024-02-13T00:45:15.434853+0000","last_unstale":"2024-02-13T00:48:07.723158+0000","last_undegraded":"2024-02-13T00:48:07.723158+0000","last_fullsized":"2024-02-13T00:48:07.723158+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":91,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722459+0000","last_change":"2024-02-13T00:46:01.455201+0000","last_active":"2024-02-13T00:48:07.722459+0000","last_peered":"2024-02-13T00:48:07.722459+0000","last_clean":"2024-02-13T00:48:07.722459+0000","last_became_active":"2024-02-13T00:46:01.454882+0000","last_became_peered":"2024-02-13T00:46:01.454882+0000","last_unstale":"2024-02-13T00:48:07.722459+0000","last_undegraded":"2024-02-13T00:48:07.722459+0000","last_fullsized":"2024-02-13T00:48:07.722459+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:35.197045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:35.197045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":275,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455957+0000","last_change":"2024-02-13T00:46:09.256189+0000","last_active":"2024-02-13T00:48:06.455957+0000","last_peered":"2024-02-13T00:48:06.455957+0000","last_clean":"2024-02-13T00:48:06.455957+0000","last_became_active":"2024-02-13T00:46:01.459978+0000","last_became_peered":"2024-02-13T00:46:01.459978+0000","last_unstale":"2024-02-13T00:48:06.455957+0000","last_undegraded":"2024-02-13T00:48:06.455957+0000","last_fullsized":"2024-02-13T00:48:06.455957+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:44:24.196214+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:44:24.196214+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":101,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725020+0000","last_change":"2024-02-13T00:46:30.641539+0000","last_active":"2024-02-13T00:48:07.725020+0000","last_peered":"2024-02-13T00:48:07.725020+0000","last_clean":"2024-02-13T00:48:07.725020+0000","last_became_active":"2024-02-13T00:46:01.444548+0000","last_became_peered":"2024-02-13T00:46:01.444548+0000","last_unstale":"2024-02-13T00:48:07.725020+0000","last_undegraded":"2024-02-13T00:48:07.725020+0000","last_fullsized":"2024-02-13T00:48:07.725020+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.641460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.641460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":183,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722760+0000","last_change":"2024-02-13T00:46:01.455578+0000","last_active":"2024-02-13T00:48:07.722760+0000","last_peered":"2024-02-13T00:48:07.722760+0000","last_clean":"2024-02-13T00:48:07.722760+0000","last_became_active":"2024-02-13T00:46:01.454642+0000","last_became_peered":"2024-02-13T00:46:01.454642+0000","last_unstale":"2024-02-13T00:48:07.722760+0000","last_undegraded":"2024-02-13T00:48:07.722760+0000","last_fullsized":"2024-02-13T00:48:07.722760+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.592672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":303,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455056+0000","last_change":"2024-02-13T00:45:15.429026+0000","last_active":"2024-02-13T00:48:06.455056+0000","last_peered":"2024-02-13T00:48:06.455056+0000","last_clean":"2024-02-13T00:48:06.455056+0000","last_became_active":"2024-02-13T00:45:15.427945+0000","last_became_peered":"2024-02-13T00:45:15.427945+0000","last_unstale":"2024-02-13T00:48:06.455056+0000","last_undegraded":"2024-02-13T00:48:06.455056+0000","last_fullsized":"2024-02-13T00:48:06.455056+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":408,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495394+0000","last_change":"2024-02-13T00:45:26.255907+0000","last_active":"2024-02-13T00:48:07.495394+0000","last_peered":"2024-02-13T00:48:07.495394+0000","last_clean":"2024-02-13T00:48:07.495394+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:48:07.495394+0000","last_undegraded":"2024-02-13T00:48:07.495394+0000","last_fullsized":"2024-02-13T00:48:07.495394+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.255842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.255842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"467'31","reported_seq":240,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495473+0000","last_change":"2024-02-13T00:44:54.041282+0000","last_active":"2024-02-13T00:48:07.495473+0000","last_peered":"2024-02-13T00:48:07.495473+0000","last_clean":"2024-02-13T00:48:07.495473+0000","last_became_active":"2024-02-13T00:44:50.432807+0000","last_became_peered":"2024-02-13T00:44:50.432807+0000","last_unstale":"2024-02-13T00:48:07.495473+0000","last_undegraded":"2024-02-13T00:48:07.495473+0000","last_fullsized":"2024-02-13T00:48:07.495473+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'17","last_scrub_stamp":"2024-02-13T00:42:17.413826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:17.413826+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":258,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455260+0000","last_change":"2024-02-13T00:46:04.708510+0000","last_active":"2024-02-13T00:48:06.455260+0000","last_peered":"2024-02-13T00:48:06.455260+0000","last_clean":"2024-02-13T00:48:06.455260+0000","last_became_active":"2024-02-13T00:46:01.447801+0000","last_became_peered":"2024-02-13T00:46:01.447801+0000","last_unstale":"2024-02-13T00:48:06.455260+0000","last_undegraded":"2024-02-13T00:48:06.455260+0000","last_fullsized":"2024-02-13T00:48:06.455260+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:45:24.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:24.269595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":443,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494097+0000","last_change":"2024-02-13T00:42:28.037452+0000","last_active":"2024-02-13T00:48:07.494097+0000","last_peered":"2024-02-13T00:48:07.494097+0000","last_clean":"2024-02-13T00:48:07.494097+0000","last_became_active":"2024-02-13T00:37:45.242469+0000","last_became_peered":"2024-02-13T00:37:45.242469+0000","last_unstale":"2024-02-13T00:48:07.494097+0000","last_undegraded":"2024-02-13T00:48:07.494097+0000","last_fullsized":"2024-02-13T00:48:07.494097+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:28.037365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:28.037365+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":452,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456862+0000","last_change":"2024-02-13T00:44:50.441408+0000","last_active":"2024-02-13T00:48:06.456862+0000","last_peered":"2024-02-13T00:48:06.456862+0000","last_clean":"2024-02-13T00:48:06.456862+0000","last_became_active":"2024-02-13T00:44:50.440301+0000","last_became_peered":"2024-02-13T00:44:50.440301+0000","last_unstale":"2024-02-13T00:48:06.456862+0000","last_undegraded":"2024-02-13T00:48:06.456862+0000","last_fullsized":"2024-02-13T00:48:06.456862+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:25.502311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.502311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":108,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723925+0000","last_change":"2024-02-13T00:48:06.705079+0000","last_active":"2024-02-13T00:48:07.723925+0000","last_peered":"2024-02-13T00:48:07.723925+0000","last_clean":"2024-02-13T00:48:07.723925+0000","last_became_active":"2024-02-13T00:48:06.704718+0000","last_became_peered":"2024-02-13T00:48:06.704718+0000","last_unstale":"2024-02-13T00:48:07.723925+0000","last_undegraded":"2024-02-13T00:48:07.723925+0000","last_fullsized":"2024-02-13T00:48:07.723925+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.488353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.488353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":194,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457300+0000","last_change":"2024-02-13T00:44:46.481036+0000","last_active":"2024-02-13T00:48:06.457300+0000","last_peered":"2024-02-13T00:48:06.457300+0000","last_clean":"2024-02-13T00:48:06.457300+0000","last_became_active":"2024-02-13T00:44:39.523138+0000","last_became_peered":"2024-02-13T00:44:39.523138+0000","last_unstale":"2024-02-13T00:48:06.457300+0000","last_undegraded":"2024-02-13T00:48:06.457300+0000","last_fullsized":"2024-02-13T00:48:06.457300+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-02-13T00:43:00.776560+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.776560+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":472,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492888+0000","last_change":"2024-02-13T00:46:01.453216+0000","last_active":"2024-02-13T00:48:07.492888+0000","last_peered":"2024-02-13T00:48:07.492888+0000","last_clean":"2024-02-13T00:48:07.492888+0000","last_became_active":"2024-02-13T00:46:01.452674+0000","last_became_peered":"2024-02-13T00:46:01.452674+0000","last_unstale":"2024-02-13T00:48:07.492888+0000","last_undegraded":"2024-02-13T00:48:07.492888+0000","last_fullsized":"2024-02-13T00:48:07.492888+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:24.059313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:42:24.059313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"404'11","reported_seq":151,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:14.335291+0000","last_change":"2024-02-13T00:46:37.662485+0000","last_active":"2024-02-13T00:48:14.335291+0000","last_peered":"2024-02-13T00:48:14.335291+0000","last_clean":"2024-02-13T00:48:14.335291+0000","last_became_active":"2024-02-13T00:46:01.456187+0000","last_became_peered":"2024-02-13T00:46:01.456187+0000","last_unstale":"2024-02-13T00:48:14.335291+0000","last_undegraded":"2024-02-13T00:48:14.335291+0000","last_fullsized":"2024-02-13T00:48:14.335291+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'11","last_scrub_stamp":"2024-02-13T00:46:37.662413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.662413+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":444,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.496231+0000","last_change":"2024-02-13T00:46:05.839833+0000","last_active":"2024-02-13T00:48:07.496231+0000","last_peered":"2024-02-13T00:48:07.496231+0000","last_clean":"2024-02-13T00:48:07.496231+0000","last_became_active":"2024-02-13T00:46:01.455274+0000","last_became_peered":"2024-02-13T00:46:01.455274+0000","last_unstale":"2024-02-13T00:48:07.496231+0000","last_undegraded":"2024-02-13T00:48:07.496231+0000","last_fullsized":"2024-02-13T00:48:07.496231+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:44:25.230980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.230980+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":253,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494922+0000","last_change":"2024-02-13T00:46:01.457421+0000","last_active":"2024-02-13T00:48:07.494922+0000","last_peered":"2024-02-13T00:48:07.494922+0000","last_clean":"2024-02-13T00:48:07.494922+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:48:07.494922+0000","last_undegraded":"2024-02-13T00:48:07.494922+0000","last_fullsized":"2024-02-13T00:48:07.494922+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:18.089252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:18.089252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"467'27","reported_seq":561,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494876+0000","last_change":"2024-02-13T00:46:05.836637+0000","last_active":"2024-02-13T00:48:07.494876+0000","last_peered":"2024-02-13T00:48:07.494876+0000","last_clean":"2024-02-13T00:48:07.494876+0000","last_became_active":"2024-02-13T00:46:01.448705+0000","last_became_peered":"2024-02-13T00:46:01.448705+0000","last_unstale":"2024-02-13T00:48:07.494876+0000","last_undegraded":"2024-02-13T00:48:07.494876+0000","last_fullsized":"2024-02-13T00:48:07.494876+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:42:21.069482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:21.069482+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":504,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492134+0000","last_change":"2024-02-13T00:45:15.445834+0000","last_active":"2024-02-13T00:48:07.492134+0000","last_peered":"2024-02-13T00:48:07.492134+0000","last_clean":"2024-02-13T00:48:07.492134+0000","last_became_active":"2024-02-13T00:45:15.445228+0000","last_became_peered":"2024-02-13T00:45:15.445228+0000","last_unstale":"2024-02-13T00:48:07.492134+0000","last_undegraded":"2024-02-13T00:48:07.492134+0000","last_fullsized":"2024-02-13T00:48:07.492134+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:43:34.320637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.320637+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"467'11","reported_seq":149,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:14.335276+0000","last_change":"2024-02-13T00:47:49.687806+0000","last_active":"2024-02-13T00:48:14.335276+0000","last_peered":"2024-02-13T00:48:14.335276+0000","last_clean":"2024-02-13T00:48:14.335276+0000","last_became_active":"2024-02-13T00:46:01.448785+0000","last_became_peered":"2024-02-13T00:46:01.448785+0000","last_unstale":"2024-02-13T00:48:14.335276+0000","last_undegraded":"2024-02-13T00:48:14.335276+0000","last_fullsized":"2024-02-13T00:48:14.335276+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:47:49.687727+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":94,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495127+0000","last_change":"2024-02-13T00:46:01.457110+0000","last_active":"2024-02-13T00:48:07.495127+0000","last_peered":"2024-02-13T00:48:07.495127+0000","last_clean":"2024-02-13T00:48:07.495127+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:48:07.495127+0000","last_undegraded":"2024-02-13T00:48:07.495127+0000","last_fullsized":"2024-02-13T00:48:07.495127+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.092250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.092250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":408,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725800+0000","last_change":"2024-02-13T00:46:01.456531+0000","last_active":"2024-02-13T00:48:07.725800+0000","last_peered":"2024-02-13T00:48:07.725800+0000","last_clean":"2024-02-13T00:48:07.725800+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:48:07.725800+0000","last_undegraded":"2024-02-13T00:48:07.725800+0000","last_fullsized":"2024-02-13T00:48:07.725800+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:44:59.963616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":193,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458421+0000","last_change":"2024-02-13T00:44:45.962479+0000","last_active":"2024-02-13T00:48:06.458421+0000","last_peered":"2024-02-13T00:48:06.458421+0000","last_clean":"2024-02-13T00:48:06.458421+0000","last_became_active":"2024-02-13T00:44:39.521984+0000","last_became_peered":"2024-02-13T00:44:39.521984+0000","last_unstale":"2024-02-13T00:48:06.458421+0000","last_undegraded":"2024-02-13T00:48:06.458421+0000","last_fullsized":"2024-02-13T00:48:06.458421+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:42:52.245750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.245750+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":291,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.724726+0000","last_change":"2024-02-13T00:46:23.757417+0000","last_active":"2024-02-13T00:48:07.724726+0000","last_peered":"2024-02-13T00:48:07.724726+0000","last_clean":"2024-02-13T00:48:07.724726+0000","last_became_active":"2024-02-13T00:45:15.419239+0000","last_became_peered":"2024-02-13T00:45:15.419239+0000","last_unstale":"2024-02-13T00:48:07.724726+0000","last_undegraded":"2024-02-13T00:48:07.724726+0000","last_fullsized":"2024-02-13T00:48:07.724726+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:23.757351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:23.757351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":387,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456861+0000","last_change":"2024-02-13T00:46:01.460527+0000","last_active":"2024-02-13T00:48:06.456861+0000","last_peered":"2024-02-13T00:48:06.456861+0000","last_clean":"2024-02-13T00:48:06.456861+0000","last_became_active":"2024-02-13T00:46:01.459122+0000","last_became_peered":"2024-02-13T00:46:01.459122+0000","last_unstale":"2024-02-13T00:48:06.456861+0000","last_undegraded":"2024-02-13T00:48:06.456861+0000","last_fullsized":"2024-02-13T00:48:06.456861+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:57.546517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.546517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":185,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456990+0000","last_change":"2024-02-13T00:46:01.463229+0000","last_active":"2024-02-13T00:48:06.456990+0000","last_peered":"2024-02-13T00:48:06.456990+0000","last_clean":"2024-02-13T00:48:06.456990+0000","last_became_active":"2024-02-13T00:46:01.462717+0000","last_became_peered":"2024-02-13T00:46:01.462717+0000","last_unstale":"2024-02-13T00:48:06.456990+0000","last_undegraded":"2024-02-13T00:48:06.456990+0000","last_fullsized":"2024-02-13T00:48:06.456990+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.782345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.782345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":360,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722413+0000","last_change":"2024-02-13T00:46:01.457255+0000","last_active":"2024-02-13T00:48:07.722413+0000","last_peered":"2024-02-13T00:48:07.722413+0000","last_clean":"2024-02-13T00:48:07.722413+0000","last_became_active":"2024-02-13T00:46:01.456959+0000","last_became_peered":"2024-02-13T00:46:01.456959+0000","last_unstale":"2024-02-13T00:48:07.722413+0000","last_undegraded":"2024-02-13T00:48:07.722413+0000","last_fullsized":"2024-02-13T00:48:07.722413+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.281102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.281102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"263'10","reported_seq":462,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:14.335287+0000","last_change":"2024-02-13T00:47:02.949483+0000","last_active":"2024-02-13T00:48:14.335287+0000","last_peered":"2024-02-13T00:48:14.335287+0000","last_clean":"2024-02-13T00:48:14.335287+0000","last_became_active":"2024-02-13T00:43:35.454087+0000","last_became_peered":"2024-02-13T00:43:35.454087+0000","last_unstale":"2024-02-13T00:48:14.335287+0000","last_undegraded":"2024-02-13T00:48:14.335287+0000","last_fullsized":"2024-02-13T00:48:14.335287+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.949406+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":457,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455408+0000","last_change":"2024-02-13T00:46:08.351239+0000","last_active":"2024-02-13T00:48:06.455408+0000","last_peered":"2024-02-13T00:48:06.455408+0000","last_clean":"2024-02-13T00:48:06.455408+0000","last_became_active":"2024-02-13T00:46:01.449654+0000","last_became_peered":"2024-02-13T00:46:01.449654+0000","last_unstale":"2024-02-13T00:48:06.455408+0000","last_undegraded":"2024-02-13T00:48:06.455408+0000","last_fullsized":"2024-02-13T00:48:06.455408+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2024-02-13T00:43:00.636798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.636798+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":185,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457121+0000","last_change":"2024-02-13T00:45:15.446786+0000","last_active":"2024-02-13T00:48:06.457121+0000","last_peered":"2024-02-13T00:48:06.457121+0000","last_clean":"2024-02-13T00:48:06.457121+0000","last_became_active":"2024-02-13T00:45:15.446443+0000","last_became_peered":"2024-02-13T00:45:15.446443+0000","last_unstale":"2024-02-13T00:48:06.457121+0000","last_undegraded":"2024-02-13T00:48:06.457121+0000","last_fullsized":"2024-02-13T00:48:06.457121+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.557475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.557475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":185,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455778+0000","last_change":"2024-02-13T00:45:15.436286+0000","last_active":"2024-02-13T00:48:06.455778+0000","last_peered":"2024-02-13T00:48:06.455778+0000","last_clean":"2024-02-13T00:48:06.455778+0000","last_became_active":"2024-02-13T00:45:15.434559+0000","last_became_peered":"2024-02-13T00:45:15.434559+0000","last_unstale":"2024-02-13T00:48:06.455778+0000","last_undegraded":"2024-02-13T00:48:06.455778+0000","last_fullsized":"2024-02-13T00:48:06.455778+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":210,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723101+0000","last_change":"2024-02-13T00:46:01.445822+0000","last_active":"2024-02-13T00:48:07.723101+0000","last_peered":"2024-02-13T00:48:07.723101+0000","last_clean":"2024-02-13T00:48:07.723101+0000","last_became_active":"2024-02-13T00:46:01.444661+0000","last_became_peered":"2024-02-13T00:46:01.444661+0000","last_unstale":"2024-02-13T00:48:07.723101+0000","last_undegraded":"2024-02-13T00:48:07.723101+0000","last_fullsized":"2024-02-13T00:48:07.723101+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:43.049231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:43.049231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":251,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722633+0000","last_change":"2024-02-13T00:46:01.454510+0000","last_active":"2024-02-13T00:48:07.722633+0000","last_peered":"2024-02-13T00:48:07.722633+0000","last_clean":"2024-02-13T00:48:07.722633+0000","last_became_active":"2024-02-13T00:46:01.453842+0000","last_became_peered":"2024-02-13T00:46:01.453842+0000","last_unstale":"2024-02-13T00:48:07.722633+0000","last_undegraded":"2024-02-13T00:48:07.722633+0000","last_fullsized":"2024-02-13T00:48:07.722633+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:54.842056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:54.842056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"467'42","reported_seq":412,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723164+0000","last_change":"2024-02-13T00:46:04.744348+0000","last_active":"2024-02-13T00:48:07.723164+0000","last_peered":"2024-02-13T00:48:07.723164+0000","last_clean":"2024-02-13T00:48:07.723164+0000","last_became_active":"2024-02-13T00:46:01.445058+0000","last_became_peered":"2024-02-13T00:46:01.445058+0000","last_unstale":"2024-02-13T00:48:07.723164+0000","last_undegraded":"2024-02-13T00:48:07.723164+0000","last_fullsized":"2024-02-13T00:48:07.723164+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'30","last_scrub_stamp":"2024-02-13T00:42:53.881594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.881594+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":477,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.496122+0000","last_change":"2024-02-13T00:47:09.940673+0000","last_active":"2024-02-13T00:48:07.496122+0000","last_peered":"2024-02-13T00:48:07.496122+0000","last_clean":"2024-02-13T00:48:07.496122+0000","last_became_active":"2024-02-13T00:44:39.503590+0000","last_became_peered":"2024-02-13T00:44:39.503590+0000","last_unstale":"2024-02-13T00:48:07.496122+0000","last_undegraded":"2024-02-13T00:48:07.496122+0000","last_fullsized":"2024-02-13T00:48:07.496122+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:09.940605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:09.940605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":420,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455448+0000","last_change":"2024-02-13T00:46:01.448313+0000","last_active":"2024-02-13T00:48:06.455448+0000","last_peered":"2024-02-13T00:48:06.455448+0000","last_clean":"2024-02-13T00:48:06.455448+0000","last_became_active":"2024-02-13T00:46:01.447815+0000","last_became_peered":"2024-02-13T00:46:01.447815+0000","last_unstale":"2024-02-13T00:48:06.455448+0000","last_undegraded":"2024-02-13T00:48:06.455448+0000","last_fullsized":"2024-02-13T00:48:06.455448+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":407,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455522+0000","last_change":"2024-02-13T00:44:50.436745+0000","last_active":"2024-02-13T00:48:06.455522+0000","last_peered":"2024-02-13T00:48:06.455522+0000","last_clean":"2024-02-13T00:48:06.455522+0000","last_became_active":"2024-02-13T00:44:50.436058+0000","last_became_peered":"2024-02-13T00:44:50.436058+0000","last_unstale":"2024-02-13T00:48:06.455522+0000","last_undegraded":"2024-02-13T00:48:06.455522+0000","last_fullsized":"2024-02-13T00:48:06.455522+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:48.300400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.300400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":181,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456025+0000","last_change":"2024-02-13T00:46:01.451625+0000","last_active":"2024-02-13T00:48:06.456025+0000","last_peered":"2024-02-13T00:48:06.456025+0000","last_clean":"2024-02-13T00:48:06.456025+0000","last_became_active":"2024-02-13T00:46:01.450587+0000","last_became_peered":"2024-02-13T00:46:01.450587+0000","last_unstale":"2024-02-13T00:48:06.456025+0000","last_undegraded":"2024-02-13T00:48:06.456025+0000","last_fullsized":"2024-02-13T00:48:06.456025+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:33.204969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:33.204969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":580,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722937+0000","last_change":"2024-02-13T00:47:05.718055+0000","last_active":"2024-02-13T00:48:07.722937+0000","last_peered":"2024-02-13T00:48:07.722937+0000","last_clean":"2024-02-13T00:48:07.722937+0000","last_became_active":"2024-02-13T00:46:01.443777+0000","last_became_peered":"2024-02-13T00:46:01.443777+0000","last_unstale":"2024-02-13T00:48:07.722937+0000","last_undegraded":"2024-02-13T00:48:07.722937+0000","last_fullsized":"2024-02-13T00:48:07.722937+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:47:05.717987+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:47:05.717987+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.724739+0000","last_change":"2024-02-13T00:45:15.423607+0000","last_active":"2024-02-13T00:48:07.724739+0000","last_peered":"2024-02-13T00:48:07.724739+0000","last_clean":"2024-02-13T00:48:07.724739+0000","last_became_active":"2024-02-13T00:45:15.423415+0000","last_became_peered":"2024-02-13T00:45:15.423415+0000","last_unstale":"2024-02-13T00:48:07.724739+0000","last_undegraded":"2024-02-13T00:48:07.724739+0000","last_fullsized":"2024-02-13T00:48:07.724739+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:17.295459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:17.295459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":484,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492148+0000","last_change":"2024-02-13T00:45:15.445862+0000","last_active":"2024-02-13T00:48:07.492148+0000","last_peered":"2024-02-13T00:48:07.492148+0000","last_clean":"2024-02-13T00:48:07.492148+0000","last_became_active":"2024-02-13T00:45:15.445557+0000","last_became_peered":"2024-02-13T00:45:15.445557+0000","last_unstale":"2024-02-13T00:48:07.492148+0000","last_undegraded":"2024-02-13T00:48:07.492148+0000","last_fullsized":"2024-02-13T00:48:07.492148+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":97,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495998+0000","last_change":"2024-02-13T00:46:01.442629+0000","last_active":"2024-02-13T00:48:07.495998+0000","last_peered":"2024-02-13T00:48:07.495998+0000","last_clean":"2024-02-13T00:48:07.495998+0000","last_became_active":"2024-02-13T00:46:01.441421+0000","last_became_peered":"2024-02-13T00:46:01.441421+0000","last_unstale":"2024-02-13T00:48:07.495998+0000","last_undegraded":"2024-02-13T00:48:07.495998+0000","last_fullsized":"2024-02-13T00:48:07.495998+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:50.325063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:50.325063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":99,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495326+0000","last_change":"2024-02-13T00:47:43.903657+0000","last_active":"2024-02-13T00:48:07.495326+0000","last_peered":"2024-02-13T00:48:07.495326+0000","last_clean":"2024-02-13T00:48:07.495326+0000","last_became_active":"2024-02-13T00:46:01.449142+0000","last_became_peered":"2024-02-13T00:46:01.449142+0000","last_unstale":"2024-02-13T00:48:07.495326+0000","last_undegraded":"2024-02-13T00:48:07.495326+0000","last_fullsized":"2024-02-13T00:48:07.495326+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:43.903553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:43.903553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":575,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725246+0000","last_change":"2024-02-13T00:46:34.691109+0000","last_active":"2024-02-13T00:48:07.725246+0000","last_peered":"2024-02-13T00:48:07.725246+0000","last_clean":"2024-02-13T00:48:07.725246+0000","last_became_active":"2024-02-13T00:46:01.448163+0000","last_became_peered":"2024-02-13T00:46:01.448163+0000","last_unstale":"2024-02-13T00:48:07.725246+0000","last_undegraded":"2024-02-13T00:48:07.725246+0000","last_fullsized":"2024-02-13T00:48:07.725246+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:46:34.690980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.690980+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":185,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.491740+0000","last_change":"2024-02-13T00:46:37.056299+0000","last_active":"2024-02-13T00:48:07.491740+0000","last_peered":"2024-02-13T00:48:07.491740+0000","last_clean":"2024-02-13T00:48:07.491740+0000","last_became_active":"2024-02-13T00:46:01.446104+0000","last_became_peered":"2024-02-13T00:46:01.446104+0000","last_unstale":"2024-02-13T00:48:07.491740+0000","last_undegraded":"2024-02-13T00:48:07.491740+0000","last_fullsized":"2024-02-13T00:48:07.491740+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:37.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":467,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722941+0000","last_change":"2024-02-13T00:46:01.431899+0000","last_active":"2024-02-13T00:48:07.722941+0000","last_peered":"2024-02-13T00:48:07.722941+0000","last_clean":"2024-02-13T00:48:07.722941+0000","last_became_active":"2024-02-13T00:46:01.430623+0000","last_became_peered":"2024-02-13T00:46:01.430623+0000","last_unstale":"2024-02-13T00:48:07.722941+0000","last_undegraded":"2024-02-13T00:48:07.722941+0000","last_fullsized":"2024-02-13T00:48:07.722941+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:46.556229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.556229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":475,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494152+0000","last_change":"2024-02-13T00:46:01.450548+0000","last_active":"2024-02-13T00:48:07.494152+0000","last_peered":"2024-02-13T00:48:07.494152+0000","last_clean":"2024-02-13T00:48:07.494152+0000","last_became_active":"2024-02-13T00:46:01.450176+0000","last_became_peered":"2024-02-13T00:46:01.450176+0000","last_unstale":"2024-02-13T00:48:07.494152+0000","last_undegraded":"2024-02-13T00:48:07.494152+0000","last_fullsized":"2024-02-13T00:48:07.494152+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":327,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458170+0000","last_change":"2024-02-13T00:46:01.460470+0000","last_active":"2024-02-13T00:48:06.458170+0000","last_peered":"2024-02-13T00:48:06.458170+0000","last_clean":"2024-02-13T00:48:06.458170+0000","last_became_active":"2024-02-13T00:46:01.459859+0000","last_became_peered":"2024-02-13T00:46:01.459859+0000","last_unstale":"2024-02-13T00:48:06.458170+0000","last_undegraded":"2024-02-13T00:48:06.458170+0000","last_fullsized":"2024-02-13T00:48:06.458170+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:01.179031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:01.179031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":132,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723706+0000","last_change":"2024-02-13T00:47:33.932830+0000","last_active":"2024-02-13T00:48:07.723706+0000","last_peered":"2024-02-13T00:48:07.723706+0000","last_clean":"2024-02-13T00:48:07.723706+0000","last_became_active":"2024-02-13T00:46:01.455170+0000","last_became_peered":"2024-02-13T00:46:01.455170+0000","last_unstale":"2024-02-13T00:48:07.723706+0000","last_undegraded":"2024-02-13T00:48:07.723706+0000","last_fullsized":"2024-02-13T00:48:07.723706+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:33.932766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:33.932766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":231,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458207+0000","last_change":"2024-02-13T00:46:57.381602+0000","last_active":"2024-02-13T00:48:06.458207+0000","last_peered":"2024-02-13T00:48:06.458207+0000","last_clean":"2024-02-13T00:48:06.458207+0000","last_became_active":"2024-02-13T00:45:15.446969+0000","last_became_peered":"2024-02-13T00:45:15.446969+0000","last_unstale":"2024-02-13T00:48:06.458207+0000","last_undegraded":"2024-02-13T00:48:06.458207+0000","last_fullsized":"2024-02-13T00:48:06.458207+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:46:57.381540+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:46:57.381540+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":432,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722950+0000","last_change":"2024-02-13T00:47:39.822745+0000","last_active":"2024-02-13T00:48:07.722950+0000","last_peered":"2024-02-13T00:48:07.722950+0000","last_clean":"2024-02-13T00:48:07.722950+0000","last_became_active":"2024-02-13T00:45:15.434498+0000","last_became_peered":"2024-02-13T00:45:15.434498+0000","last_unstale":"2024-02-13T00:48:07.722950+0000","last_undegraded":"2024-02-13T00:48:07.722950+0000","last_fullsized":"2024-02-13T00:48:07.722950+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:39.822619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:47:39.822619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":429,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455428+0000","last_change":"2024-02-13T00:47:04.493911+0000","last_active":"2024-02-13T00:48:06.455428+0000","last_peered":"2024-02-13T00:48:06.455428+0000","last_clean":"2024-02-13T00:48:06.455428+0000","last_became_active":"2024-02-13T00:46:01.447823+0000","last_became_peered":"2024-02-13T00:46:01.447823+0000","last_unstale":"2024-02-13T00:48:06.455428+0000","last_undegraded":"2024-02-13T00:48:06.455428+0000","last_fullsized":"2024-02-13T00:48:06.455428+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:04.493841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:04.493841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":297,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455348+0000","last_change":"2024-02-13T00:46:01.460412+0000","last_active":"2024-02-13T00:48:06.455348+0000","last_peered":"2024-02-13T00:48:06.455348+0000","last_clean":"2024-02-13T00:48:06.455348+0000","last_became_active":"2024-02-13T00:46:01.460001+0000","last_became_peered":"2024-02-13T00:46:01.460001+0000","last_unstale":"2024-02-13T00:48:06.455348+0000","last_undegraded":"2024-02-13T00:48:06.455348+0000","last_fullsized":"2024-02-13T00:48:06.455348+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:27.544320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:42:27.544320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":323,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494691+0000","last_change":"2024-02-13T00:46:01.450879+0000","last_active":"2024-02-13T00:48:07.494691+0000","last_peered":"2024-02-13T00:48:07.494691+0000","last_clean":"2024-02-13T00:48:07.494691+0000","last_became_active":"2024-02-13T00:46:01.450088+0000","last_became_peered":"2024-02-13T00:46:01.450088+0000","last_unstale":"2024-02-13T00:48:07.494691+0000","last_undegraded":"2024-02-13T00:48:07.494691+0000","last_fullsized":"2024-02-13T00:48:07.494691+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":408,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725251+0000","last_change":"2024-02-13T00:46:01.459317+0000","last_active":"2024-02-13T00:48:07.725251+0000","last_peered":"2024-02-13T00:48:07.725251+0000","last_clean":"2024-02-13T00:48:07.725251+0000","last_became_active":"2024-02-13T00:46:01.458408+0000","last_became_peered":"2024-02-13T00:46:01.458408+0000","last_unstale":"2024-02-13T00:48:07.725251+0000","last_undegraded":"2024-02-13T00:48:07.725251+0000","last_fullsized":"2024-02-13T00:48:07.725251+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:16.256516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:16.256516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":131,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494648+0000","last_change":"2024-02-13T00:46:01.442001+0000","last_active":"2024-02-13T00:48:07.494648+0000","last_peered":"2024-02-13T00:48:07.494648+0000","last_clean":"2024-02-13T00:48:07.494648+0000","last_became_active":"2024-02-13T00:46:01.441013+0000","last_became_peered":"2024-02-13T00:46:01.441013+0000","last_unstale":"2024-02-13T00:48:07.494648+0000","last_undegraded":"2024-02-13T00:48:07.494648+0000","last_fullsized":"2024-02-13T00:48:07.494648+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:26.313580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:26.313580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"467'23","reported_seq":514,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495057+0000","last_change":"2024-02-13T00:46:04.714305+0000","last_active":"2024-02-13T00:48:07.495057+0000","last_peered":"2024-02-13T00:48:07.495057+0000","last_clean":"2024-02-13T00:48:07.495057+0000","last_became_active":"2024-02-13T00:46:01.448219+0000","last_became_peered":"2024-02-13T00:46:01.448219+0000","last_unstale":"2024-02-13T00:48:07.495057+0000","last_undegraded":"2024-02-13T00:48:07.495057+0000","last_fullsized":"2024-02-13T00:48:07.495057+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-13T00:42:54.174950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:54.174950+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":382,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722934+0000","last_change":"2024-02-13T00:44:50.426201+0000","last_active":"2024-02-13T00:48:07.722934+0000","last_peered":"2024-02-13T00:48:07.722934+0000","last_clean":"2024-02-13T00:48:07.722934+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:48:07.722934+0000","last_undegraded":"2024-02-13T00:48:07.722934+0000","last_fullsized":"2024-02-13T00:48:07.722934+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:20.481083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:44:20.481083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":1119,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-13T00:48:14.474086+0000","last_change":"2024-02-13T00:45:54.075270+0000","last_active":"2024-02-13T00:48:14.474086+0000","last_peered":"2024-02-13T00:48:14.474086+0000","last_clean":"2024-02-13T00:48:14.474086+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:48:14.474086+0000","last_undegraded":"2024-02-13T00:48:14.474086+0000","last_fullsized":"2024-02-13T00:48:14.474086+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:45:54.075194+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":621,"num_read_kb":616,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":194,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458110+0000","last_change":"2024-02-13T00:46:01.460282+0000","last_active":"2024-02-13T00:48:06.458110+0000","last_peered":"2024-02-13T00:48:06.458110+0000","last_clean":"2024-02-13T00:48:06.458110+0000","last_became_active":"2024-02-13T00:46:01.459473+0000","last_became_peered":"2024-02-13T00:46:01.459473+0000","last_unstale":"2024-02-13T00:48:06.458110+0000","last_undegraded":"2024-02-13T00:48:06.458110+0000","last_fullsized":"2024-02-13T00:48:06.458110+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":248,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.493873+0000","last_change":"2024-02-13T00:45:37.135398+0000","last_active":"2024-02-13T00:48:07.493873+0000","last_peered":"2024-02-13T00:48:07.493873+0000","last_clean":"2024-02-13T00:48:07.493873+0000","last_became_active":"2024-02-13T00:45:15.435588+0000","last_became_peered":"2024-02-13T00:45:15.435588+0000","last_unstale":"2024-02-13T00:48:07.493873+0000","last_undegraded":"2024-02-13T00:48:07.493873+0000","last_fullsized":"2024-02-13T00:48:07.493873+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.135332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.135332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":145,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457069+0000","last_change":"2024-02-13T00:45:31.183763+0000","last_active":"2024-02-13T00:48:06.457069+0000","last_peered":"2024-02-13T00:48:06.457069+0000","last_clean":"2024-02-13T00:48:06.457069+0000","last_became_active":"2024-02-13T00:45:06.421025+0000","last_became_peered":"2024-02-13T00:45:06.421025+0000","last_unstale":"2024-02-13T00:48:06.457069+0000","last_undegraded":"2024-02-13T00:48:06.457069+0000","last_fullsized":"2024-02-13T00:48:06.457069+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.183682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.183682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"467'65","reported_seq":161,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:14.335515+0000","last_change":"2024-02-13T00:46:04.950162+0000","last_active":"2024-02-13T00:48:14.335515+0000","last_peered":"2024-02-13T00:48:14.335515+0000","last_clean":"2024-02-13T00:48:14.335515+0000","last_became_active":"2024-02-13T00:46:01.449059+0000","last_became_peered":"2024-02-13T00:46:01.449059+0000","last_unstale":"2024-02-13T00:48:14.335515+0000","last_undegraded":"2024-02-13T00:48:14.335515+0000","last_fullsized":"2024-02-13T00:48:14.335515+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'48","last_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:43:48.212428+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":2148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":393,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492619+0000","last_change":"2024-02-13T00:46:01.453042+0000","last_active":"2024-02-13T00:48:07.492619+0000","last_peered":"2024-02-13T00:48:07.492619+0000","last_clean":"2024-02-13T00:48:07.492619+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:48:07.492619+0000","last_undegraded":"2024-02-13T00:48:07.492619+0000","last_fullsized":"2024-02-13T00:48:07.492619+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.045912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.045912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723424+0000","last_change":"2024-02-13T00:48:06.705030+0000","last_active":"2024-02-13T00:48:07.723424+0000","last_peered":"2024-02-13T00:48:07.723424+0000","last_clean":"2024-02-13T00:48:07.723424+0000","last_became_active":"2024-02-13T00:48:06.704651+0000","last_became_peered":"2024-02-13T00:48:06.704651+0000","last_unstale":"2024-02-13T00:48:07.723424+0000","last_undegraded":"2024-02-13T00:48:07.723424+0000","last_fullsized":"2024-02-13T00:48:07.723424+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":176,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456106+0000","last_change":"2024-02-13T00:44:50.439369+0000","last_active":"2024-02-13T00:48:06.456106+0000","last_peered":"2024-02-13T00:48:06.456106+0000","last_clean":"2024-02-13T00:48:06.456106+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:48:06.456106+0000","last_undegraded":"2024-02-13T00:48:06.456106+0000","last_fullsized":"2024-02-13T00:48:06.456106+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:19.407011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:42:19.407011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":629,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:14.335303+0000","last_change":"2024-02-13T00:46:11.962872+0000","last_active":"2024-02-13T00:48:14.335303+0000","last_peered":"2024-02-13T00:48:14.335303+0000","last_clean":"2024-02-13T00:48:14.335303+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:48:14.335303+0000","last_undegraded":"2024-02-13T00:48:14.335303+0000","last_fullsized":"2024-02-13T00:48:14.335303+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'11","last_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:43:02.423358+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":211,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.493072+0000","last_change":"2024-02-13T00:46:02.664116+0000","last_active":"2024-02-13T00:48:07.493072+0000","last_peered":"2024-02-13T00:48:07.493072+0000","last_clean":"2024-02-13T00:48:07.493072+0000","last_became_active":"2024-02-13T00:46:01.442003+0000","last_became_peered":"2024-02-13T00:46:01.442003+0000","last_unstale":"2024-02-13T00:48:07.493072+0000","last_undegraded":"2024-02-13T00:48:07.493072+0000","last_fullsized":"2024-02-13T00:48:07.493072+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:41:52.517392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:41:52.517392+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.493919+0000","last_change":"2024-02-13T00:45:15.423417+0000","last_active":"2024-02-13T00:48:07.493919+0000","last_peered":"2024-02-13T00:48:07.493919+0000","last_clean":"2024-02-13T00:48:07.493919+0000","last_became_active":"2024-02-13T00:45:15.422546+0000","last_became_peered":"2024-02-13T00:45:15.422546+0000","last_unstale":"2024-02-13T00:48:07.493919+0000","last_undegraded":"2024-02-13T00:48:07.493919+0000","last_fullsized":"2024-02-13T00:48:07.493919+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:41.223028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:43:41.223028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":474,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722396+0000","last_change":"2024-02-13T00:46:01.436997+0000","last_active":"2024-02-13T00:48:07.722396+0000","last_peered":"2024-02-13T00:48:07.722396+0000","last_clean":"2024-02-13T00:48:07.722396+0000","last_became_active":"2024-02-13T00:46:01.435178+0000","last_became_peered":"2024-02-13T00:46:01.435178+0000","last_unstale":"2024-02-13T00:48:07.722396+0000","last_undegraded":"2024-02-13T00:48:07.722396+0000","last_fullsized":"2024-02-13T00:48:07.722396+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":408,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492008+0000","last_change":"2024-02-13T00:45:15.445626+0000","last_active":"2024-02-13T00:48:07.492008+0000","last_peered":"2024-02-13T00:48:07.492008+0000","last_clean":"2024-02-13T00:48:07.492008+0000","last_became_active":"2024-02-13T00:45:15.445117+0000","last_became_peered":"2024-02-13T00:45:15.445117+0000","last_unstale":"2024-02-13T00:48:07.492008+0000","last_undegraded":"2024-02-13T00:48:07.492008+0000","last_fullsized":"2024-02-13T00:48:07.492008+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.171185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":174,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.724856+0000","last_change":"2024-02-13T00:46:01.436795+0000","last_active":"2024-02-13T00:48:07.724856+0000","last_peered":"2024-02-13T00:48:07.724856+0000","last_clean":"2024-02-13T00:48:07.724856+0000","last_became_active":"2024-02-13T00:46:01.434431+0000","last_became_peered":"2024-02-13T00:46:01.434431+0000","last_unstale":"2024-02-13T00:48:07.724856+0000","last_undegraded":"2024-02-13T00:48:07.724856+0000","last_fullsized":"2024-02-13T00:48:07.724856+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.592320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.592320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":529,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492012+0000","last_change":"2024-02-13T00:44:41.502069+0000","last_active":"2024-02-13T00:48:07.492012+0000","last_peered":"2024-02-13T00:48:07.492012+0000","last_clean":"2024-02-13T00:48:07.492012+0000","last_became_active":"2024-02-13T00:44:39.501886+0000","last_became_peered":"2024-02-13T00:44:39.501886+0000","last_unstale":"2024-02-13T00:48:07.492012+0000","last_undegraded":"2024-02-13T00:48:07.492012+0000","last_fullsized":"2024-02-13T00:48:07.492012+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'24","last_scrub_stamp":"2024-02-13T00:43:30.413372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.413372+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":448,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725128+0000","last_change":"2024-02-13T00:46:14.266858+0000","last_active":"2024-02-13T00:48:07.725128+0000","last_peered":"2024-02-13T00:48:07.725128+0000","last_clean":"2024-02-13T00:48:07.725128+0000","last_became_active":"2024-02-13T00:46:01.457926+0000","last_became_peered":"2024-02-13T00:46:01.457926+0000","last_unstale":"2024-02-13T00:48:07.725128+0000","last_undegraded":"2024-02-13T00:48:07.725128+0000","last_fullsized":"2024-02-13T00:48:07.725128+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:43:35.309308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:35.309308+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":259,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494149+0000","last_change":"2024-02-13T00:47:06.895693+0000","last_active":"2024-02-13T00:48:07.494149+0000","last_peered":"2024-02-13T00:48:07.494149+0000","last_clean":"2024-02-13T00:48:07.494149+0000","last_became_active":"2024-02-13T00:45:15.435404+0000","last_became_peered":"2024-02-13T00:45:15.435404+0000","last_unstale":"2024-02-13T00:48:07.494149+0000","last_undegraded":"2024-02-13T00:48:07.494149+0000","last_fullsized":"2024-02-13T00:48:07.494149+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:06.895612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:47:06.895612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":408,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725408+0000","last_change":"2024-02-13T00:46:01.450436+0000","last_active":"2024-02-13T00:48:07.725408+0000","last_peered":"2024-02-13T00:48:07.725408+0000","last_clean":"2024-02-13T00:48:07.725408+0000","last_became_active":"2024-02-13T00:46:01.449234+0000","last_became_peered":"2024-02-13T00:46:01.449234+0000","last_unstale":"2024-02-13T00:48:07.725408+0000","last_undegraded":"2024-02-13T00:48:07.725408+0000","last_fullsized":"2024-02-13T00:48:07.725408+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.110742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.110742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"467'27","reported_seq":594,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725328+0000","last_change":"2024-02-13T00:46:12.563543+0000","last_active":"2024-02-13T00:48:07.725328+0000","last_peered":"2024-02-13T00:48:07.725328+0000","last_clean":"2024-02-13T00:48:07.725328+0000","last_became_active":"2024-02-13T00:46:01.456876+0000","last_became_peered":"2024-02-13T00:46:01.456876+0000","last_unstale":"2024-02-13T00:48:07.725328+0000","last_undegraded":"2024-02-13T00:48:07.725328+0000","last_fullsized":"2024-02-13T00:48:07.725328+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:43:30.365536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.365536+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":166,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.491730+0000","last_change":"2024-02-13T00:46:30.095387+0000","last_active":"2024-02-13T00:48:07.491730+0000","last_peered":"2024-02-13T00:48:07.491730+0000","last_clean":"2024-02-13T00:48:07.491730+0000","last_became_active":"2024-02-13T00:44:50.441300+0000","last_became_peered":"2024-02-13T00:44:50.441300+0000","last_unstale":"2024-02-13T00:48:07.491730+0000","last_undegraded":"2024-02-13T00:48:07.491730+0000","last_fullsized":"2024-02-13T00:48:07.491730+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.095307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":245,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494966+0000","last_change":"2024-02-13T00:45:15.416946+0000","last_active":"2024-02-13T00:48:07.494966+0000","last_peered":"2024-02-13T00:48:07.494966+0000","last_clean":"2024-02-13T00:48:07.494966+0000","last_became_active":"2024-02-13T00:45:15.416364+0000","last_became_peered":"2024-02-13T00:45:15.416364+0000","last_unstale":"2024-02-13T00:48:07.494966+0000","last_undegraded":"2024-02-13T00:48:07.494966+0000","last_fullsized":"2024-02-13T00:48:07.494966+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:15.458752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:15.458752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":181,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723830+0000","last_change":"2024-02-13T00:46:01.454597+0000","last_active":"2024-02-13T00:48:07.723830+0000","last_peered":"2024-02-13T00:48:07.723830+0000","last_clean":"2024-02-13T00:48:07.723830+0000","last_became_active":"2024-02-13T00:46:01.454101+0000","last_became_peered":"2024-02-13T00:46:01.454101+0000","last_unstale":"2024-02-13T00:48:07.723830+0000","last_undegraded":"2024-02-13T00:48:07.723830+0000","last_fullsized":"2024-02-13T00:48:07.723830+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.651091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.651091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":176,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457056+0000","last_change":"2024-02-13T00:44:50.441196+0000","last_active":"2024-02-13T00:48:06.457056+0000","last_peered":"2024-02-13T00:48:06.457056+0000","last_clean":"2024-02-13T00:48:06.457056+0000","last_became_active":"2024-02-13T00:44:50.438274+0000","last_became_peered":"2024-02-13T00:44:50.438274+0000","last_unstale":"2024-02-13T00:48:06.457056+0000","last_undegraded":"2024-02-13T00:48:06.457056+0000","last_fullsized":"2024-02-13T00:48:06.457056+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.510219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.510219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":335,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495223+0000","last_change":"2024-02-13T00:46:34.141015+0000","last_active":"2024-02-13T00:48:07.495223+0000","last_peered":"2024-02-13T00:48:07.495223+0000","last_clean":"2024-02-13T00:48:07.495223+0000","last_became_active":"2024-02-13T00:45:15.436020+0000","last_became_peered":"2024-02-13T00:45:15.436020+0000","last_unstale":"2024-02-13T00:48:07.495223+0000","last_undegraded":"2024-02-13T00:48:07.495223+0000","last_fullsized":"2024-02-13T00:48:07.495223+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:46:34.140928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.140928+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":172,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455865+0000","last_change":"2024-02-13T00:45:00.311635+0000","last_active":"2024-02-13T00:48:06.455865+0000","last_peered":"2024-02-13T00:48:06.455865+0000","last_clean":"2024-02-13T00:48:06.455865+0000","last_became_active":"2024-02-13T00:44:39.512402+0000","last_became_peered":"2024-02-13T00:44:39.512402+0000","last_unstale":"2024-02-13T00:48:06.455865+0000","last_undegraded":"2024-02-13T00:48:06.455865+0000","last_fullsized":"2024-02-13T00:48:06.455865+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:00.311558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:45:00.311558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":389,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.496177+0000","last_change":"2024-02-13T00:46:01.455806+0000","last_active":"2024-02-13T00:48:07.496177+0000","last_peered":"2024-02-13T00:48:07.496177+0000","last_clean":"2024-02-13T00:48:07.496177+0000","last_became_active":"2024-02-13T00:46:01.453265+0000","last_became_peered":"2024-02-13T00:46:01.453265+0000","last_unstale":"2024-02-13T00:48:07.496177+0000","last_undegraded":"2024-02-13T00:48:07.496177+0000","last_fullsized":"2024-02-13T00:48:07.496177+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.106639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.106639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":185,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458115+0000","last_change":"2024-02-13T00:46:01.451840+0000","last_active":"2024-02-13T00:48:06.458115+0000","last_peered":"2024-02-13T00:48:06.458115+0000","last_clean":"2024-02-13T00:48:06.458115+0000","last_became_active":"2024-02-13T00:46:01.450312+0000","last_became_peered":"2024-02-13T00:46:01.450312+0000","last_unstale":"2024-02-13T00:48:06.458115+0000","last_undegraded":"2024-02-13T00:48:06.458115+0000","last_fullsized":"2024-02-13T00:48:06.458115+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.042291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.042291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":258,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.724936+0000","last_change":"2024-02-13T00:46:01.454858+0000","last_active":"2024-02-13T00:48:07.724936+0000","last_peered":"2024-02-13T00:48:07.724936+0000","last_clean":"2024-02-13T00:48:07.724936+0000","last_became_active":"2024-02-13T00:46:01.454284+0000","last_became_peered":"2024-02-13T00:46:01.454284+0000","last_unstale":"2024-02-13T00:48:07.724936+0000","last_undegraded":"2024-02-13T00:48:07.724936+0000","last_fullsized":"2024-02-13T00:48:07.724936+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:34.523594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.523594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":413,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494057+0000","last_change":"2024-02-13T00:46:04.434215+0000","last_active":"2024-02-13T00:48:07.494057+0000","last_peered":"2024-02-13T00:48:07.494057+0000","last_clean":"2024-02-13T00:48:07.494057+0000","last_became_active":"2024-02-13T00:46:01.450113+0000","last_became_peered":"2024-02-13T00:46:01.450113+0000","last_unstale":"2024-02-13T00:48:07.494057+0000","last_undegraded":"2024-02-13T00:48:07.494057+0000","last_fullsized":"2024-02-13T00:48:07.494057+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'29","last_scrub_stamp":"2024-02-13T00:42:48.774168+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.774168+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":428,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723816+0000","last_change":"2024-02-13T00:46:01.454689+0000","last_active":"2024-02-13T00:48:07.723816+0000","last_peered":"2024-02-13T00:48:07.723816+0000","last_clean":"2024-02-13T00:48:07.723816+0000","last_became_active":"2024-02-13T00:46:01.454310+0000","last_became_peered":"2024-02-13T00:46:01.454310+0000","last_unstale":"2024-02-13T00:48:07.723816+0000","last_undegraded":"2024-02-13T00:48:07.723816+0000","last_fullsized":"2024-02-13T00:48:07.723816+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.230330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.230330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"404'9","reported_seq":149,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:14.335344+0000","last_change":"2024-02-13T00:47:37.013003+0000","last_active":"2024-02-13T00:48:14.335344+0000","last_peered":"2024-02-13T00:48:14.335344+0000","last_clean":"2024-02-13T00:48:14.335344+0000","last_became_active":"2024-02-13T00:46:01.448144+0000","last_became_peered":"2024-02-13T00:46:01.448144+0000","last_unstale":"2024-02-13T00:48:14.335344+0000","last_undegraded":"2024-02-13T00:48:14.335344+0000","last_fullsized":"2024-02-13T00:48:14.335344+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9","last_scrub_stamp":"2024-02-13T00:47:37.012934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:37.012934+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":465,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725119+0000","last_change":"2024-02-13T00:46:01.449886+0000","last_active":"2024-02-13T00:48:07.725119+0000","last_peered":"2024-02-13T00:48:07.725119+0000","last_clean":"2024-02-13T00:48:07.725119+0000","last_became_active":"2024-02-13T00:46:01.447236+0000","last_became_peered":"2024-02-13T00:46:01.447236+0000","last_unstale":"2024-02-13T00:48:07.725119+0000","last_undegraded":"2024-02-13T00:48:07.725119+0000","last_fullsized":"2024-02-13T00:48:07.725119+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:52.391906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.391906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":136,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723679+0000","last_change":"2024-02-13T00:46:01.456618+0000","last_active":"2024-02-13T00:48:07.723679+0000","last_peered":"2024-02-13T00:48:07.723679+0000","last_clean":"2024-02-13T00:48:07.723679+0000","last_became_active":"2024-02-13T00:46:01.455989+0000","last_became_peered":"2024-02-13T00:46:01.455989+0000","last_unstale":"2024-02-13T00:48:07.723679+0000","last_undegraded":"2024-02-13T00:48:07.723679+0000","last_fullsized":"2024-02-13T00:48:07.723679+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:16.499780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:16.499780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"467'22","reported_seq":454,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457244+0000","last_change":"2024-02-13T00:47:35.369710+0000","last_active":"2024-02-13T00:48:06.457244+0000","last_peered":"2024-02-13T00:48:06.457244+0000","last_clean":"2024-02-13T00:48:06.457244+0000","last_became_active":"2024-02-13T00:46:01.460321+0000","last_became_peered":"2024-02-13T00:46:01.460321+0000","last_unstale":"2024-02-13T00:48:06.457244+0000","last_undegraded":"2024-02-13T00:48:06.457244+0000","last_fullsized":"2024-02-13T00:48:06.457244+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:47:35.369616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:35.369616+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":480,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722875+0000","last_change":"2024-02-13T00:46:01.432179+0000","last_active":"2024-02-13T00:48:07.722875+0000","last_peered":"2024-02-13T00:48:07.722875+0000","last_clean":"2024-02-13T00:48:07.722875+0000","last_became_active":"2024-02-13T00:46:01.431926+0000","last_became_peered":"2024-02-13T00:46:01.431926+0000","last_unstale":"2024-02-13T00:48:07.722875+0000","last_undegraded":"2024-02-13T00:48:07.722875+0000","last_fullsized":"2024-02-13T00:48:07.722875+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:40.335892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:40.335892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"363'14","reported_seq":292,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:14.335632+0000","last_change":"2024-02-13T00:47:07.942748+0000","last_active":"2024-02-13T00:48:14.335632+0000","last_peered":"2024-02-13T00:48:14.335632+0000","last_clean":"2024-02-13T00:48:14.335632+0000","last_became_active":"2024-02-13T00:45:15.423066+0000","last_became_peered":"2024-02-13T00:45:15.423066+0000","last_unstale":"2024-02-13T00:48:14.335632+0000","last_undegraded":"2024-02-13T00:48:14.335632+0000","last_fullsized":"2024-02-13T00:48:14.335632+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:47:07.942668+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:47:07.942668+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":104,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.724907+0000","last_change":"2024-02-13T00:47:02.789298+0000","last_active":"2024-02-13T00:48:07.724907+0000","last_peered":"2024-02-13T00:48:07.724907+0000","last_clean":"2024-02-13T00:48:07.724907+0000","last_became_active":"2024-02-13T00:46:01.446960+0000","last_became_peered":"2024-02-13T00:46:01.446960+0000","last_unstale":"2024-02-13T00:48:07.724907+0000","last_undegraded":"2024-02-13T00:48:07.724907+0000","last_fullsized":"2024-02-13T00:48:07.724907+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:02.789189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.789189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":341,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455602+0000","last_change":"2024-02-13T00:46:01.450308+0000","last_active":"2024-02-13T00:48:06.455602+0000","last_peered":"2024-02-13T00:48:06.455602+0000","last_clean":"2024-02-13T00:48:06.455602+0000","last_became_active":"2024-02-13T00:46:01.449694+0000","last_became_peered":"2024-02-13T00:46:01.449694+0000","last_unstale":"2024-02-13T00:48:06.455602+0000","last_undegraded":"2024-02-13T00:48:06.455602+0000","last_fullsized":"2024-02-13T00:48:06.455602+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:27.073034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:27.073034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":534,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492981+0000","last_change":"2024-02-13T00:44:54.926465+0000","last_active":"2024-02-13T00:48:07.492981+0000","last_peered":"2024-02-13T00:48:07.492981+0000","last_clean":"2024-02-13T00:48:07.492981+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:48:07.492981+0000","last_undegraded":"2024-02-13T00:48:07.492981+0000","last_fullsized":"2024-02-13T00:48:07.492981+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'13","last_scrub_stamp":"2024-02-13T00:43:40.244930+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:43:40.244930+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":185,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457071+0000","last_change":"2024-02-13T00:46:01.463310+0000","last_active":"2024-02-13T00:48:06.457071+0000","last_peered":"2024-02-13T00:48:06.457071+0000","last_clean":"2024-02-13T00:48:06.457071+0000","last_became_active":"2024-02-13T00:46:01.462898+0000","last_became_peered":"2024-02-13T00:46:01.462898+0000","last_unstale":"2024-02-13T00:48:06.457071+0000","last_undegraded":"2024-02-13T00:48:06.457071+0000","last_fullsized":"2024-02-13T00:48:06.457071+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.095888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":249,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722755+0000","last_change":"2024-02-13T00:46:32.638418+0000","last_active":"2024-02-13T00:48:07.722755+0000","last_peered":"2024-02-13T00:48:07.722755+0000","last_clean":"2024-02-13T00:48:07.722755+0000","last_became_active":"2024-02-13T00:46:01.443279+0000","last_became_peered":"2024-02-13T00:46:01.443279+0000","last_unstale":"2024-02-13T00:48:07.722755+0000","last_undegraded":"2024-02-13T00:48:07.722755+0000","last_fullsized":"2024-02-13T00:48:07.722755+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.638349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":447,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722300+0000","last_change":"2024-02-13T00:46:01.446071+0000","last_active":"2024-02-13T00:48:07.722300+0000","last_peered":"2024-02-13T00:48:07.722300+0000","last_clean":"2024-02-13T00:48:07.722300+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:48:07.722300+0000","last_undegraded":"2024-02-13T00:48:07.722300+0000","last_fullsized":"2024-02-13T00:48:07.722300+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.413326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":181,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456091+0000","last_change":"2024-02-13T00:45:40.233264+0000","last_active":"2024-02-13T00:48:06.456091+0000","last_peered":"2024-02-13T00:48:06.456091+0000","last_clean":"2024-02-13T00:48:06.456091+0000","last_became_active":"2024-02-13T00:45:15.447051+0000","last_became_peered":"2024-02-13T00:45:15.447051+0000","last_unstale":"2024-02-13T00:48:06.456091+0000","last_undegraded":"2024-02-13T00:48:06.456091+0000","last_fullsized":"2024-02-13T00:48:06.456091+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:40.233185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:40.233185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":212,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456000+0000","last_change":"2024-02-13T00:44:52.751765+0000","last_active":"2024-02-13T00:48:06.456000+0000","last_peered":"2024-02-13T00:48:06.456000+0000","last_clean":"2024-02-13T00:48:06.456000+0000","last_became_active":"2024-02-13T00:44:50.435203+0000","last_became_peered":"2024-02-13T00:44:50.435203+0000","last_unstale":"2024-02-13T00:48:06.456000+0000","last_undegraded":"2024-02-13T00:48:06.456000+0000","last_fullsized":"2024-02-13T00:48:06.456000+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-02-13T00:42:51.361489+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:42:51.361489+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":459,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495327+0000","last_change":"2024-02-13T00:46:01.457220+0000","last_active":"2024-02-13T00:48:07.495327+0000","last_peered":"2024-02-13T00:48:07.495327+0000","last_clean":"2024-02-13T00:48:07.495327+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:48:07.495327+0000","last_undegraded":"2024-02-13T00:48:07.495327+0000","last_fullsized":"2024-02-13T00:48:07.495327+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.097481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":473,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.493014+0000","last_change":"2024-02-13T00:46:01.453391+0000","last_active":"2024-02-13T00:48:07.493014+0000","last_peered":"2024-02-13T00:48:07.493014+0000","last_clean":"2024-02-13T00:48:07.493014+0000","last_became_active":"2024-02-13T00:46:01.452730+0000","last_became_peered":"2024-02-13T00:46:01.452730+0000","last_unstale":"2024-02-13T00:48:07.493014+0000","last_undegraded":"2024-02-13T00:48:07.493014+0000","last_fullsized":"2024-02-13T00:48:07.493014+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:26.218807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:44:26.218807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":253,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723041+0000","last_change":"2024-02-13T00:46:01.454624+0000","last_active":"2024-02-13T00:48:07.723041+0000","last_peered":"2024-02-13T00:48:07.723041+0000","last_clean":"2024-02-13T00:48:07.723041+0000","last_became_active":"2024-02-13T00:46:01.454287+0000","last_became_peered":"2024-02-13T00:46:01.454287+0000","last_unstale":"2024-02-13T00:48:07.723041+0000","last_undegraded":"2024-02-13T00:48:07.723041+0000","last_fullsized":"2024-02-13T00:48:07.723041+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:50.828264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:50.828264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":185,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455229+0000","last_change":"2024-02-13T00:46:01.449548+0000","last_active":"2024-02-13T00:48:06.455229+0000","last_peered":"2024-02-13T00:48:06.455229+0000","last_clean":"2024-02-13T00:48:06.455229+0000","last_became_active":"2024-02-13T00:46:01.448671+0000","last_became_peered":"2024-02-13T00:46:01.448671+0000","last_unstale":"2024-02-13T00:48:06.455229+0000","last_undegraded":"2024-02-13T00:48:06.455229+0000","last_fullsized":"2024-02-13T00:48:06.455229+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.820327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.820327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":286,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492065+0000","last_change":"2024-02-13T00:44:40.371566+0000","last_active":"2024-02-13T00:48:07.492065+0000","last_peered":"2024-02-13T00:48:07.492065+0000","last_clean":"2024-02-13T00:48:07.492065+0000","last_became_active":"2024-02-13T00:44:39.501040+0000","last_became_peered":"2024-02-13T00:44:39.501040+0000","last_unstale":"2024-02-13T00:48:07.492065+0000","last_undegraded":"2024-02-13T00:48:07.492065+0000","last_fullsized":"2024-02-13T00:48:07.492065+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:44:32.180113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:44:32.180113+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":411,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457267+0000","last_change":"2024-02-13T00:46:01.464539+0000","last_active":"2024-02-13T00:48:06.457267+0000","last_peered":"2024-02-13T00:48:06.457267+0000","last_clean":"2024-02-13T00:48:06.457267+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:48:06.457267+0000","last_undegraded":"2024-02-13T00:48:06.457267+0000","last_fullsized":"2024-02-13T00:48:06.457267+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.499576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":176,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455579+0000","last_change":"2024-02-13T00:45:15.436871+0000","last_active":"2024-02-13T00:48:06.455579+0000","last_peered":"2024-02-13T00:48:06.455579+0000","last_clean":"2024-02-13T00:48:06.455579+0000","last_became_active":"2024-02-13T00:45:15.435820+0000","last_became_peered":"2024-02-13T00:45:15.435820+0000","last_unstale":"2024-02-13T00:48:06.455579+0000","last_undegraded":"2024-02-13T00:48:06.455579+0000","last_fullsized":"2024-02-13T00:48:06.455579+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.646321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.646321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":168,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492353+0000","last_change":"2024-02-13T00:45:15.445131+0000","last_active":"2024-02-13T00:48:07.492353+0000","last_peered":"2024-02-13T00:48:07.492353+0000","last_clean":"2024-02-13T00:48:07.492353+0000","last_became_active":"2024-02-13T00:45:15.444818+0000","last_became_peered":"2024-02-13T00:45:15.444818+0000","last_unstale":"2024-02-13T00:48:07.492353+0000","last_undegraded":"2024-02-13T00:48:07.492353+0000","last_fullsized":"2024-02-13T00:48:07.492353+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:42:56.479450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":332,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455594+0000","last_change":"2024-02-13T00:46:01.439915+0000","last_active":"2024-02-13T00:48:06.455594+0000","last_peered":"2024-02-13T00:48:06.455594+0000","last_clean":"2024-02-13T00:48:06.455594+0000","last_became_active":"2024-02-13T00:46:01.438745+0000","last_became_peered":"2024-02-13T00:46:01.438745+0000","last_unstale":"2024-02-13T00:48:06.455594+0000","last_undegraded":"2024-02-13T00:48:06.455594+0000","last_fullsized":"2024-02-13T00:48:06.455594+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.104906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.104906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"467'25","reported_seq":244,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455675+0000","last_change":"2024-02-13T00:47:36.368967+0000","last_active":"2024-02-13T00:48:06.455675+0000","last_peered":"2024-02-13T00:48:06.455675+0000","last_clean":"2024-02-13T00:48:06.455675+0000","last_became_active":"2024-02-13T00:46:01.450079+0000","last_became_peered":"2024-02-13T00:46:01.450079+0000","last_unstale":"2024-02-13T00:48:06.455675+0000","last_undegraded":"2024-02-13T00:48:06.455675+0000","last_fullsized":"2024-02-13T00:48:06.455675+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:47:36.368863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.368863+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":190,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456973+0000","last_change":"2024-02-13T00:46:59.388809+0000","last_active":"2024-02-13T00:48:06.456973+0000","last_peered":"2024-02-13T00:48:06.456973+0000","last_clean":"2024-02-13T00:48:06.456973+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:48:06.456973+0000","last_undegraded":"2024-02-13T00:48:06.456973+0000","last_fullsized":"2024-02-13T00:48:06.456973+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:59.388745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:59.388745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":176,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.454964+0000","last_change":"2024-02-13T00:45:15.439752+0000","last_active":"2024-02-13T00:48:06.454964+0000","last_peered":"2024-02-13T00:48:06.454964+0000","last_clean":"2024-02-13T00:48:06.454964+0000","last_became_active":"2024-02-13T00:45:15.437943+0000","last_became_peered":"2024-02-13T00:45:15.437943+0000","last_unstale":"2024-02-13T00:48:06.454964+0000","last_undegraded":"2024-02-13T00:48:06.454964+0000","last_fullsized":"2024-02-13T00:48:06.454964+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.284740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.284740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":385,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.722673+0000","last_change":"2024-02-13T00:44:58.969806+0000","last_active":"2024-02-13T00:48:07.722673+0000","last_peered":"2024-02-13T00:48:07.722673+0000","last_clean":"2024-02-13T00:48:07.722673+0000","last_became_active":"2024-02-13T00:44:50.439512+0000","last_became_peered":"2024-02-13T00:44:50.439512+0000","last_unstale":"2024-02-13T00:48:07.722673+0000","last_undegraded":"2024-02-13T00:48:07.722673+0000","last_fullsized":"2024-02-13T00:48:07.722673+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:58.969692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:44:58.969692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":299,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458005+0000","last_change":"2024-02-13T00:45:15.431130+0000","last_active":"2024-02-13T00:48:06.458005+0000","last_peered":"2024-02-13T00:48:06.458005+0000","last_clean":"2024-02-13T00:48:06.458005+0000","last_became_active":"2024-02-13T00:45:15.430749+0000","last_became_peered":"2024-02-13T00:45:15.430749+0000","last_unstale":"2024-02-13T00:48:06.458005+0000","last_undegraded":"2024-02-13T00:48:06.458005+0000","last_fullsized":"2024-02-13T00:48:06.458005+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:07.605111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.605111+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":569,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725670+0000","last_change":"2024-02-13T00:46:02.034931+0000","last_active":"2024-02-13T00:48:07.725670+0000","last_peered":"2024-02-13T00:48:07.725670+0000","last_clean":"2024-02-13T00:48:07.725670+0000","last_became_active":"2024-02-13T00:46:01.443721+0000","last_became_peered":"2024-02-13T00:46:01.443721+0000","last_unstale":"2024-02-13T00:48:07.725670+0000","last_undegraded":"2024-02-13T00:48:07.725670+0000","last_fullsized":"2024-02-13T00:48:07.725670+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'22","last_scrub_stamp":"2024-02-13T00:43:07.393197+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:43:07.393197+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":418,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.454634+0000","last_change":"2024-02-13T00:46:32.298033+0000","last_active":"2024-02-13T00:48:06.454634+0000","last_peered":"2024-02-13T00:48:06.454634+0000","last_clean":"2024-02-13T00:48:06.454634+0000","last_became_active":"2024-02-13T00:46:01.442231+0000","last_became_peered":"2024-02-13T00:46:01.442231+0000","last_unstale":"2024-02-13T00:48:06.454634+0000","last_undegraded":"2024-02-13T00:48:06.454634+0000","last_fullsized":"2024-02-13T00:48:06.454634+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.297918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.297918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"363'14","reported_seq":230,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:09.334252+0000","last_change":"2024-02-13T00:45:18.328879+0000","last_active":"2024-02-13T00:48:09.334252+0000","last_peered":"2024-02-13T00:48:09.334252+0000","last_clean":"2024-02-13T00:48:09.334252+0000","last_became_active":"2024-02-13T00:45:15.455825+0000","last_became_peered":"2024-02-13T00:45:15.455825+0000","last_unstale":"2024-02-13T00:48:09.334252+0000","last_undegraded":"2024-02-13T00:48:09.334252+0000","last_fullsized":"2024-02-13T00:48:09.334252+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'10","last_scrub_stamp":"2024-02-13T00:43:05.604461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:05.604461+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":277,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455804+0000","last_change":"2024-02-13T00:46:03.966962+0000","last_active":"2024-02-13T00:48:06.455804+0000","last_peered":"2024-02-13T00:48:06.455804+0000","last_clean":"2024-02-13T00:48:06.455804+0000","last_became_active":"2024-02-13T00:46:01.439984+0000","last_became_peered":"2024-02-13T00:46:01.439984+0000","last_unstale":"2024-02-13T00:48:06.455804+0000","last_undegraded":"2024-02-13T00:48:06.455804+0000","last_fullsized":"2024-02-13T00:48:06.455804+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:42:55.481128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:55.481128+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":417,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495168+0000","last_change":"2024-02-13T00:45:38.084426+0000","last_active":"2024-02-13T00:48:07.495168+0000","last_peered":"2024-02-13T00:48:07.495168+0000","last_clean":"2024-02-13T00:48:07.495168+0000","last_became_active":"2024-02-13T00:45:15.435550+0000","last_became_peered":"2024-02-13T00:45:15.435550+0000","last_unstale":"2024-02-13T00:48:07.495168+0000","last_undegraded":"2024-02-13T00:48:07.495168+0000","last_fullsized":"2024-02-13T00:48:07.495168+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.084358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.084358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":216,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457198+0000","last_change":"2024-02-13T00:46:10.679144+0000","last_active":"2024-02-13T00:48:06.457198+0000","last_peered":"2024-02-13T00:48:06.457198+0000","last_clean":"2024-02-13T00:48:06.457198+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:48:06.457198+0000","last_undegraded":"2024-02-13T00:48:06.457198+0000","last_fullsized":"2024-02-13T00:48:06.457198+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:42:59.604002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:42:59.604002+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":190,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455829+0000","last_change":"2024-02-13T00:46:56.339969+0000","last_active":"2024-02-13T00:48:06.455829+0000","last_peered":"2024-02-13T00:48:06.455829+0000","last_clean":"2024-02-13T00:48:06.455829+0000","last_became_active":"2024-02-13T00:45:15.436239+0000","last_became_peered":"2024-02-13T00:45:15.436239+0000","last_unstale":"2024-02-13T00:48:06.455829+0000","last_undegraded":"2024-02-13T00:48:06.455829+0000","last_fullsized":"2024-02-13T00:48:06.455829+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:56.339885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.339885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":465,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725741+0000","last_change":"2024-02-13T00:47:51.641542+0000","last_active":"2024-02-13T00:48:07.725741+0000","last_peered":"2024-02-13T00:48:07.725741+0000","last_clean":"2024-02-13T00:48:07.725741+0000","last_became_active":"2024-02-13T00:45:15.422870+0000","last_became_peered":"2024-02-13T00:45:15.422870+0000","last_unstale":"2024-02-13T00:48:07.725741+0000","last_undegraded":"2024-02-13T00:48:07.725741+0000","last_fullsized":"2024-02-13T00:48:07.725741+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:51.641461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:51.641461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":453,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495202+0000","last_change":"2024-02-13T00:46:01.457036+0000","last_active":"2024-02-13T00:48:07.495202+0000","last_peered":"2024-02-13T00:48:07.495202+0000","last_clean":"2024-02-13T00:48:07.495202+0000","last_became_active":"2024-02-13T00:46:01.453269+0000","last_became_peered":"2024-02-13T00:46:01.453269+0000","last_unstale":"2024-02-13T00:48:07.495202+0000","last_undegraded":"2024-02-13T00:48:07.495202+0000","last_fullsized":"2024-02-13T00:48:07.495202+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.245237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.245237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":181,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457968+0000","last_change":"2024-02-13T00:45:37.280947+0000","last_active":"2024-02-13T00:48:06.457968+0000","last_peered":"2024-02-13T00:48:06.457968+0000","last_clean":"2024-02-13T00:48:06.457968+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:48:06.457968+0000","last_undegraded":"2024-02-13T00:48:06.457968+0000","last_fullsized":"2024-02-13T00:48:06.457968+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.280865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.280865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":232,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.494019+0000","last_change":"2024-02-13T00:44:55.939050+0000","last_active":"2024-02-13T00:48:07.494019+0000","last_peered":"2024-02-13T00:48:07.494019+0000","last_clean":"2024-02-13T00:48:07.494019+0000","last_became_active":"2024-02-13T00:44:50.442894+0000","last_became_peered":"2024-02-13T00:44:50.442894+0000","last_unstale":"2024-02-13T00:48:07.494019+0000","last_undegraded":"2024-02-13T00:48:07.494019+0000","last_fullsized":"2024-02-13T00:48:07.494019+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'15","last_scrub_stamp":"2024-02-13T00:43:30.578602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:30.578602+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":176,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455814+0000","last_change":"2024-02-13T00:45:15.446020+0000","last_active":"2024-02-13T00:48:06.455814+0000","last_peered":"2024-02-13T00:48:06.455814+0000","last_clean":"2024-02-13T00:48:06.455814+0000","last_became_active":"2024-02-13T00:45:15.445403+0000","last_became_peered":"2024-02-13T00:45:15.445403+0000","last_unstale":"2024-02-13T00:48:06.455814+0000","last_undegraded":"2024-02-13T00:48:06.455814+0000","last_fullsized":"2024-02-13T00:48:06.455814+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:06.651077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:06.651077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":441,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495806+0000","last_change":"2024-02-13T00:41:56.204734+0000","last_active":"2024-02-13T00:48:07.495806+0000","last_peered":"2024-02-13T00:48:07.495806+0000","last_clean":"2024-02-13T00:48:07.495806+0000","last_became_active":"2024-02-13T00:37:45.251445+0000","last_became_peered":"2024-02-13T00:37:45.251445+0000","last_unstale":"2024-02-13T00:48:07.495806+0000","last_undegraded":"2024-02-13T00:48:07.495806+0000","last_fullsized":"2024-02-13T00:48:07.495806+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:56.204671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:41:56.204671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":181,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458286+0000","last_change":"2024-02-13T00:47:42.329463+0000","last_active":"2024-02-13T00:48:06.458286+0000","last_peered":"2024-02-13T00:48:06.458286+0000","last_clean":"2024-02-13T00:48:06.458286+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:48:06.458286+0000","last_undegraded":"2024-02-13T00:48:06.458286+0000","last_fullsized":"2024-02-13T00:48:06.458286+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:42.329367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:42.329367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":90,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457939+0000","last_change":"2024-02-13T00:46:01.451610+0000","last_active":"2024-02-13T00:48:06.457939+0000","last_peered":"2024-02-13T00:48:06.457939+0000","last_clean":"2024-02-13T00:48:06.457939+0000","last_became_active":"2024-02-13T00:46:01.449873+0000","last_became_peered":"2024-02-13T00:46:01.449873+0000","last_unstale":"2024-02-13T00:48:06.457939+0000","last_undegraded":"2024-02-13T00:48:06.457939+0000","last_fullsized":"2024-02-13T00:48:06.457939+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.554065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.554065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":338,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.495725+0000","last_change":"2024-02-13T00:46:56.973071+0000","last_active":"2024-02-13T00:48:07.495725+0000","last_peered":"2024-02-13T00:48:07.495725+0000","last_clean":"2024-02-13T00:48:07.495725+0000","last_became_active":"2024-02-13T00:44:39.504670+0000","last_became_peered":"2024-02-13T00:44:39.504670+0000","last_unstale":"2024-02-13T00:48:07.495725+0000","last_undegraded":"2024-02-13T00:48:07.495725+0000","last_fullsized":"2024-02-13T00:48:07.495725+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:46:56.973000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.973000+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":185,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456831+0000","last_change":"2024-02-13T00:46:01.448134+0000","last_active":"2024-02-13T00:48:06.456831+0000","last_peered":"2024-02-13T00:48:06.456831+0000","last_clean":"2024-02-13T00:48:06.456831+0000","last_became_active":"2024-02-13T00:46:01.446635+0000","last_became_peered":"2024-02-13T00:46:01.446635+0000","last_unstale":"2024-02-13T00:48:06.456831+0000","last_undegraded":"2024-02-13T00:48:06.456831+0000","last_fullsized":"2024-02-13T00:48:06.456831+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:28.101254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:28.101254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":168,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.493649+0000","last_change":"2024-02-13T00:45:15.435977+0000","last_active":"2024-02-13T00:48:07.493649+0000","last_peered":"2024-02-13T00:48:07.493649+0000","last_clean":"2024-02-13T00:48:07.493649+0000","last_became_active":"2024-02-13T00:45:15.435206+0000","last_became_peered":"2024-02-13T00:45:15.435206+0000","last_unstale":"2024-02-13T00:48:07.493649+0000","last_undegraded":"2024-02-13T00:48:07.493649+0000","last_fullsized":"2024-02-13T00:48:07.493649+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:57.802417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.301282+0000","last_clean_scrub_stamp":"2024-02-13T00:42:57.802417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":181,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456967+0000","last_change":"2024-02-13T00:46:01.461188+0000","last_active":"2024-02-13T00:48:06.456967+0000","last_peered":"2024-02-13T00:48:06.456967+0000","last_clean":"2024-02-13T00:48:06.456967+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:48:06.456967+0000","last_undegraded":"2024-02-13T00:48:06.456967+0000","last_fullsized":"2024-02-13T00:48:06.456967+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.238152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.238152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":426,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723721+0000","last_change":"2024-02-13T00:46:01.443734+0000","last_active":"2024-02-13T00:48:07.723721+0000","last_peered":"2024-02-13T00:48:07.723721+0000","last_clean":"2024-02-13T00:48:07.723721+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:48:07.723721+0000","last_undegraded":"2024-02-13T00:48:07.723721+0000","last_fullsized":"2024-02-13T00:48:07.723721+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.621170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.621170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":331,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723691+0000","last_change":"2024-02-13T00:46:27.706096+0000","last_active":"2024-02-13T00:48:07.723691+0000","last_peered":"2024-02-13T00:48:07.723691+0000","last_clean":"2024-02-13T00:48:07.723691+0000","last_became_active":"2024-02-13T00:44:50.415894+0000","last_became_peered":"2024-02-13T00:44:50.415894+0000","last_unstale":"2024-02-13T00:48:07.723691+0000","last_undegraded":"2024-02-13T00:48:07.723691+0000","last_fullsized":"2024-02-13T00:48:07.723691+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:46:27.706006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:27.706006+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":260,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.456109+0000","last_change":"2024-02-13T00:46:01.452098+0000","last_active":"2024-02-13T00:48:06.456109+0000","last_peered":"2024-02-13T00:48:06.456109+0000","last_clean":"2024-02-13T00:48:06.456109+0000","last_became_active":"2024-02-13T00:46:01.451790+0000","last_became_peered":"2024-02-13T00:46:01.451790+0000","last_unstale":"2024-02-13T00:48:06.456109+0000","last_undegraded":"2024-02-13T00:48:06.456109+0000","last_fullsized":"2024-02-13T00:48:06.456109+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:03.665973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:03.665973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":415,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723807+0000","last_change":"2024-02-13T00:46:01.454932+0000","last_active":"2024-02-13T00:48:07.723807+0000","last_peered":"2024-02-13T00:48:07.723807+0000","last_clean":"2024-02-13T00:48:07.723807+0000","last_became_active":"2024-02-13T00:46:01.454163+0000","last_became_peered":"2024-02-13T00:46:01.454163+0000","last_unstale":"2024-02-13T00:48:07.723807+0000","last_undegraded":"2024-02-13T00:48:07.723807+0000","last_fullsized":"2024-02-13T00:48:07.723807+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:47.415414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:47.415414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":454,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.724807+0000","last_change":"2024-02-13T00:47:36.885611+0000","last_active":"2024-02-13T00:48:07.724807+0000","last_peered":"2024-02-13T00:48:07.724807+0000","last_clean":"2024-02-13T00:48:07.724807+0000","last_became_active":"2024-02-13T00:45:15.419668+0000","last_became_peered":"2024-02-13T00:45:15.419668+0000","last_unstale":"2024-02-13T00:48:07.724807+0000","last_undegraded":"2024-02-13T00:48:07.724807+0000","last_fullsized":"2024-02-13T00:48:07.724807+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.885510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.885510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":360,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723989+0000","last_change":"2024-02-13T00:46:01.457482+0000","last_active":"2024-02-13T00:48:07.723989+0000","last_peered":"2024-02-13T00:48:07.723989+0000","last_clean":"2024-02-13T00:48:07.723989+0000","last_became_active":"2024-02-13T00:46:01.455940+0000","last_became_peered":"2024-02-13T00:46:01.455940+0000","last_unstale":"2024-02-13T00:48:07.723989+0000","last_undegraded":"2024-02-13T00:48:07.723989+0000","last_fullsized":"2024-02-13T00:48:07.723989+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:21.087199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:21.087199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":346,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.493879+0000","last_change":"2024-02-13T00:46:36.089483+0000","last_active":"2024-02-13T00:48:07.493879+0000","last_peered":"2024-02-13T00:48:07.493879+0000","last_clean":"2024-02-13T00:48:07.493879+0000","last_became_active":"2024-02-13T00:45:15.419364+0000","last_became_peered":"2024-02-13T00:45:15.419364+0000","last_unstale":"2024-02-13T00:48:07.493879+0000","last_undegraded":"2024-02-13T00:48:07.493879+0000","last_fullsized":"2024-02-13T00:48:07.493879+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:46:36.089402+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:46:36.089402+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":190,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.458162+0000","last_change":"2024-02-13T00:47:47.284092+0000","last_active":"2024-02-13T00:48:06.458162+0000","last_peered":"2024-02-13T00:48:06.458162+0000","last_clean":"2024-02-13T00:48:06.458162+0000","last_became_active":"2024-02-13T00:46:01.451997+0000","last_became_peered":"2024-02-13T00:46:01.451997+0000","last_unstale":"2024-02-13T00:48:06.458162+0000","last_undegraded":"2024-02-13T00:48:06.458162+0000","last_fullsized":"2024-02-13T00:48:06.458162+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.284014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.284014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":286,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492659+0000","last_change":"2024-02-13T00:45:15.416632+0000","last_active":"2024-02-13T00:48:07.492659+0000","last_peered":"2024-02-13T00:48:07.492659+0000","last_clean":"2024-02-13T00:48:07.492659+0000","last_became_active":"2024-02-13T00:45:15.416307+0000","last_became_peered":"2024-02-13T00:45:15.416307+0000","last_unstale":"2024-02-13T00:48:07.492659+0000","last_undegraded":"2024-02-13T00:48:07.492659+0000","last_fullsized":"2024-02-13T00:48:07.492659+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:48.629024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:42:48.629024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":398,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492255+0000","last_change":"2024-02-13T00:46:01.445422+0000","last_active":"2024-02-13T00:48:07.492255+0000","last_peered":"2024-02-13T00:48:07.492255+0000","last_clean":"2024-02-13T00:48:07.492255+0000","last_became_active":"2024-02-13T00:46:01.445141+0000","last_became_peered":"2024-02-13T00:46:01.445141+0000","last_unstale":"2024-02-13T00:48:07.492255+0000","last_undegraded":"2024-02-13T00:48:07.492255+0000","last_fullsized":"2024-02-13T00:48:07.492255+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:52.447698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:42:52.447698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":172,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455253+0000","last_change":"2024-02-13T00:45:30.186208+0000","last_active":"2024-02-13T00:48:06.455253+0000","last_peered":"2024-02-13T00:48:06.455253+0000","last_clean":"2024-02-13T00:48:06.455253+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:48:06.455253+0000","last_undegraded":"2024-02-13T00:48:06.455253+0000","last_fullsized":"2024-02-13T00:48:06.455253+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:30.186129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:30.186129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":230,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455322+0000","last_change":"2024-02-13T00:46:06.251713+0000","last_active":"2024-02-13T00:48:06.455322+0000","last_peered":"2024-02-13T00:48:06.455322+0000","last_clean":"2024-02-13T00:48:06.455322+0000","last_became_active":"2024-02-13T00:46:01.442464+0000","last_became_peered":"2024-02-13T00:46:01.442464+0000","last_unstale":"2024-02-13T00:48:06.455322+0000","last_undegraded":"2024-02-13T00:48:06.455322+0000","last_fullsized":"2024-02-13T00:48:06.455322+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:43:44.241358+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:43:44.241358+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":267,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725483+0000","last_change":"2024-02-13T00:47:47.711551+0000","last_active":"2024-02-13T00:48:07.725483+0000","last_peered":"2024-02-13T00:48:07.725483+0000","last_clean":"2024-02-13T00:48:07.725483+0000","last_became_active":"2024-02-13T00:46:01.447654+0000","last_became_peered":"2024-02-13T00:46:01.447654+0000","last_unstale":"2024-02-13T00:48:07.725483+0000","last_undegraded":"2024-02-13T00:48:07.725483+0000","last_fullsized":"2024-02-13T00:48:07.725483+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.711486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.711486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":464,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492097+0000","last_change":"2024-02-13T00:46:55.979707+0000","last_active":"2024-02-13T00:48:07.492097+0000","last_peered":"2024-02-13T00:48:07.492097+0000","last_clean":"2024-02-13T00:48:07.492097+0000","last_became_active":"2024-02-13T00:46:01.444842+0000","last_became_peered":"2024-02-13T00:46:01.444842+0000","last_unstale":"2024-02-13T00:48:07.492097+0000","last_undegraded":"2024-02-13T00:48:07.492097+0000","last_fullsized":"2024-02-13T00:48:07.492097+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:55.979635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:46:55.979635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":102,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492577+0000","last_change":"2024-02-13T00:47:53.826039+0000","last_active":"2024-02-13T00:48:07.492577+0000","last_peered":"2024-02-13T00:48:07.492577+0000","last_clean":"2024-02-13T00:48:07.492577+0000","last_became_active":"2024-02-13T00:46:01.448971+0000","last_became_peered":"2024-02-13T00:46:01.448971+0000","last_unstale":"2024-02-13T00:48:07.492577+0000","last_undegraded":"2024-02-13T00:48:07.492577+0000","last_fullsized":"2024-02-13T00:48:07.492577+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:53.825971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:53.825971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":429,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.492350+0000","last_change":"2024-02-13T00:45:32.233265+0000","last_active":"2024-02-13T00:48:07.492350+0000","last_peered":"2024-02-13T00:48:07.492350+0000","last_clean":"2024-02-13T00:48:07.492350+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:48:07.492350+0000","last_undegraded":"2024-02-13T00:48:07.492350+0000","last_fullsized":"2024-02-13T00:48:07.492350+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:45:32.233197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.233197+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":366,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.723111+0000","last_change":"2024-02-13T00:47:44.757362+0000","last_active":"2024-02-13T00:48:07.723111+0000","last_peered":"2024-02-13T00:48:07.723111+0000","last_clean":"2024-02-13T00:48:07.723111+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:48:07.723111+0000","last_undegraded":"2024-02-13T00:48:07.723111+0000","last_fullsized":"2024-02-13T00:48:07.723111+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:47:44.757262+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:47:44.757262+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":453,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-13T00:48:07.725167+0000","last_change":"2024-02-13T00:45:15.434575+0000","last_active":"2024-02-13T00:48:07.725167+0000","last_peered":"2024-02-13T00:48:07.725167+0000","last_clean":"2024-02-13T00:48:07.725167+0000","last_became_active":"2024-02-13T00:45:15.434224+0000","last_became_peered":"2024-02-13T00:45:15.434224+0000","last_unstale":"2024-02-13T00:48:07.725167+0000","last_undegraded":"2024-02-13T00:48:07.725167+0000","last_fullsized":"2024-02-13T00:48:07.725167+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:43:03.464398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":172,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.455070+0000","last_change":"2024-02-13T00:47:40.322696+0000","last_active":"2024-02-13T00:48:06.455070+0000","last_peered":"2024-02-13T00:48:06.455070+0000","last_clean":"2024-02-13T00:48:06.455070+0000","last_became_active":"2024-02-13T00:44:39.508197+0000","last_became_peered":"2024-02-13T00:44:39.508197+0000","last_unstale":"2024-02-13T00:48:06.455070+0000","last_undegraded":"2024-02-13T00:48:06.455070+0000","last_fullsized":"2024-02-13T00:48:06.455070+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:40.322594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:23.273346+0000","last_clean_scrub_stamp":"2024-02-13T00:47:40.322594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":176,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-02-13T00:48:06.457993+0000","last_change":"2024-02-13T00:45:15.435955+0000","last_active":"2024-02-13T00:48:06.457993+0000","last_peered":"2024-02-13T00:48:06.457993+0000","last_clean":"2024-02-13T00:48:06.457993+0000","last_became_active":"2024-02-13T00:45:15.435259+0000","last_became_peered":"2024-02-13T00:45:15.435259+0000","last_unstale":"2024-02-13T00:48:06.457993+0000","last_undegraded":"2024-02-13T00:48:06.457993+0000","last_fullsized":"2024-02-13T00:48:06.457993+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.953989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.953989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":219,"num_pg":32,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":60,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":4},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":4001618,"num_objects":173,"num_object_clones":9,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":107,"num_read_kb":7,"num_write":630,"num_write_kb":4526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1266,"num_bytes_recovered":4004226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14249984,"data_stored":12005301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1959,"ondisk_log_size":1959,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1492,"num_read_kb":1217,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":9606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":11656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":970,"ondisk_log_size":970,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":621,"num_read_kb":616,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":227633266817,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":45,"seq":193273528455,"num_pgs":176,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309164,"kb_used_data":7832,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93435988,"statfs":{"total":95995035648,"available":95678451712,"internally_reserved":0,"allocated":8019968,"data_stored":4432093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47899999999999998,"15min":0.55400000000000005},"min":{"1min":0.28699999999999998,"5min":0.25600000000000001,"15min":0.19700000000000001},"max":{"1min":0.69799999999999995,"5min":0.77500000000000002,"15min":8.1340000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.52700000000000002,"15min":0.60599999999999998},"min":{"1min":0.312,"5min":0.23100000000000001,"15min":0.221},"max":{"1min":0.77900000000000003,"5min":4.2910000000000004,"15min":11.367000000000001},"last":0.41099999999999998}]},{"osd":1,"last update":"Tue Feb 13 00:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51600000000000001,"15min":0.88400000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.68400000000000005,"5min":4.5720000000000001,"15min":54.002000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.68100000000000005},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.69399999999999995,"5min":5.4989999999999997,"15min":12.403},"last":0.46200000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.70599999999999996},"min":{"1min":0.29399999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.72399999999999998,"5min":2.8399999999999999,"15min":12.477},"last":0.496},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.50800000000000001,"15min":0.67500000000000004},"min":{"1min":0.33200000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.66200000000000003,"5min":2.1659999999999999,"15min":11.702},"last":0.51100000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.61399999999999999,"15min":0.66600000000000004},"min":{"1min":0.33000000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71099999999999997,"5min":5.0990000000000002,"15min":8.0519999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.55300000000000005,"15min":0.77300000000000002},"min":{"1min":0.379,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":0.67100000000000004,"5min":2.8719999999999999,"15min":19.081},"last":0.54300000000000004}]},{"osd":4,"last update":"Tue Feb 13 00:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.42099999999999999,"15min":0.41099999999999998},"min":{"1min":0.26800000000000002,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":0.67400000000000004,"5min":0.71899999999999997,"15min":1.0649999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.435,"15min":0.42999999999999999},"min":{"1min":0.22800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.623,"5min":0.70799999999999996,"15min":0.83299999999999996},"last":0.56799999999999995}]},{"osd":5,"last update":"Tue Feb 13 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.434,"15min":0.41499999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.61199999999999999,"5min":0.66500000000000004,"15min":0.94399999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.48199999999999998,"15min":0.46999999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.86299999999999999},"last":0.59999999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64100000000000001,"5min":0.66800000000000004,"15min":1.173},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.51600000000000001,"15min":0.53300000000000003},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.26900000000000002},"max":{"1min":0.67200000000000004,"5min":0.77800000000000002,"15min":1.262},"last":0.626}]}]},{"osd":1,"up_from":13,"seq":55834575008,"num_pgs":194,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321356,"kb_used_data":11840,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93423796,"statfs":{"total":95995035648,"available":95665967104,"internally_reserved":0,"allocated":12124160,"data_stored":8440311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.38200000000000001,"15min":0.94699999999999995},"min":{"1min":0.24099999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.52200000000000002,"5min":5.7199999999999998,"15min":61.856000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.39500000000000002,"15min":0.66500000000000004},"min":{"1min":0.221,"5min":0.19400000000000001,"15min":0.16500000000000001},"max":{"1min":0.57399999999999995,"5min":7.1100000000000003,"15min":61.908999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:47:59 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.45600000000000002,"15min":0.84999999999999998},"min":{"1min":0.22900000000000001,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":0.53900000000000003,"5min":10.436999999999999,"15min":32.161000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.438,"15min":0.82899999999999996},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.55900000000000005,"5min":6.298,"15min":29.693000000000001},"last":0.41599999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.47099999999999997,"15min":1.28},"min":{"1min":0.27500000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.59099999999999997,"5min":10.478,"15min":131.23099999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.42699999999999999,"15min":0.875},"min":{"1min":0.28199999999999997,"5min":0.26800000000000002,"15min":0.246},"max":{"1min":0.52300000000000002,"5min":4.024,"15min":49.055},"last":0.439}]},{"osd":4,"last update":"Tue Feb 13 00:47:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58099999999999996,"15min":0.76200000000000001},"min":{"1min":0.36699999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.77800000000000002,"5min":4.0640000000000001,"15min":16.547000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61899999999999999,"15min":0.70699999999999996},"min":{"1min":0.38300000000000001,"5min":0.27800000000000002,"15min":0.24199999999999999},"max":{"1min":0.77200000000000002,"5min":7.0170000000000003,"15min":14.731999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.5680000000000001,"15min":0.90200000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.90500000000000003,"5min":26.561,"15min":26.561},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.137,"15min":0.76900000000000002},"min":{"1min":0.377,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.163,"5min":16.494,"15min":16.494},"last":0.55700000000000005}]},{"osd":6,"last update":"Tue Feb 13 00:47:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53500000000000003,"15min":0.72099999999999997},"min":{"1min":0.35599999999999998,"5min":0.29699999999999999,"15min":0.29099999999999998},"max":{"1min":0.74199999999999999,"5min":0.92700000000000005,"15min":14.772},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.65100000000000002,"15min":0.93300000000000005},"min":{"1min":0.38100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":10.565,"15min":26.600000000000001},"last":0.63}]},{"osd":7,"last update":"Tue Feb 13 00:47:59 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.61899999999999999,"15min":0.72099999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70999999999999996,"5min":7.048,"15min":14.794},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.67500000000000004,"15min":0.93100000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.32200000000000001},"max":{"1min":0.752,"5min":10.521000000000001,"15min":26.693999999999999},"last":0.60499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738532,"num_pgs":179,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325316,"kb_used_data":11700,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93419836,"statfs":{"total":95995035648,"available":95661912064,"internally_reserved":0,"allocated":11980800,"data_stored":8431717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.36799999999999999,"15min":0.61899999999999999},"min":{"1min":0.22600000000000001,"5min":0.192,"15min":0.189},"max":{"1min":0.45700000000000002,"5min":2.0379999999999998,"15min":30.184000000000001},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.49399999999999999,"15min":0.45000000000000001},"min":{"1min":0.24399999999999999,"5min":0.20300000000000001,"15min":0.17599999999999999},"max":{"1min":0.68600000000000005,"5min":5.6959999999999997,"15min":5.6959999999999997},"last":0.36499999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:48:02 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.52100000000000002,"15min":0.504},"min":{"1min":0.248,"5min":0.216,"15min":0.20999999999999999},"max":{"1min":1.137,"5min":7.0839999999999996,"15min":7.0839999999999996},"last":0.312},{"interface":"front","average":{"1min":0.376,"5min":0.47299999999999998,"15min":0.67200000000000004},"min":{"1min":0.26600000000000001,"5min":0.246,"15min":0.185},"max":{"1min":0.54200000000000004,"5min":3.6949999999999998,"15min":24.459},"last":0.33300000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.48799999999999999,"15min":0.49099999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":0.53300000000000003,"5min":4.8710000000000004,"15min":8.718},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.61699999999999999,"15min":0.54900000000000004},"min":{"1min":0.26900000000000002,"5min":0.222,"15min":0.222},"max":{"1min":1.2669999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"last":0.34599999999999997}]},{"osd":4,"last update":"Tue Feb 13 00:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.65300000000000002,"15min":0.79600000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.24399999999999999},"max":{"1min":0.77800000000000002,"5min":7.056,"15min":24.381},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.69999999999999996,"15min":0.65200000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.28699999999999998},"max":{"1min":0.76100000000000001,"5min":6.1859999999999999,"15min":8.6950000000000003},"last":0.59199999999999997}]},{"osd":5,"last update":"Tue Feb 13 00:46:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.749,"15min":0.63},"min":{"1min":0.36399999999999999,"5min":0.309,"15min":0.309},"max":{"1min":3.331,"5min":7.0990000000000002,"15min":7.0990000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.4330000000000001,"15min":0.85099999999999998},"min":{"1min":0.33900000000000002,"5min":0.307,"15min":0.30599999999999999},"max":{"1min":10.686,"5min":24.427,"15min":24.427},"last":0.60999999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.65800000000000003,"15min":0.622},"min":{"1min":0.38600000000000001,"5min":0.34100000000000003,"15min":0.31},"max":{"1min":0.78300000000000003,"5min":4.6600000000000001,"15min":8.6129999999999995},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.78900000000000003,"15min":0.64800000000000002},"min":{"1min":0.40500000000000003,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.206,"5min":14.263,"15min":14.263},"last":0.69799999999999995}]},{"osd":7,"last update":"Tue Feb 13 00:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.70099999999999996,"15min":0.66100000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.317},"max":{"1min":0.80800000000000005,"5min":7.7720000000000002,"15min":8.6679999999999993},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.747,"15min":0.66900000000000004},"min":{"1min":0.40400000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.6910000000000001,"5min":8.1609999999999996,"15min":8.1609999999999996},"last":0.67800000000000005}]}]},{"osd":2,"up_from":0,"seq":77309411483,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182549,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986321,"num_pgs":165,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321352,"kb_used_data":11836,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93423800,"statfs":{"total":95995035648,"available":95665971200,"internally_reserved":0,"allocated":12120064,"data_stored":8440377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.53100000000000003,"15min":0.59799999999999998},"min":{"1min":0.30399999999999999,"5min":0.23599999999999999,"15min":0.21199999999999999},"max":{"1min":0.60499999999999998,"5min":5.3319999999999999,"15min":9.3399999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.63300000000000001,"15min":0.61699999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.69999999999999996,"5min":11.532,"15min":13.833},"last":0.41999999999999998}]},{"osd":1,"last update":"Tue Feb 13 00:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.60299999999999998,"15min":0.75900000000000001},"min":{"1min":0.26800000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.625,"5min":7.5410000000000004,"15min":53.188000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.59599999999999997,"15min":0.70199999999999996},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.25600000000000001},"max":{"1min":0.67900000000000005,"5min":11.497999999999999,"15min":44.793999999999997},"last":0.497}]},{"osd":2,"last update":"Tue Feb 13 00:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.60399999999999998,"15min":0.59699999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.26900000000000002},"max":{"1min":0.67500000000000004,"5min":4.2199999999999998,"15min":8.6829999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.60699999999999998,"15min":0.63300000000000001},"min":{"1min":0.29099999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67300000000000004,"5min":4.0730000000000004,"15min":18.911000000000001},"last":0.45700000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.53900000000000003,"15min":0.65800000000000003},"min":{"1min":0.32000000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63200000000000001,"5min":4.0090000000000003,"15min":23.5},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.56499999999999995,"15min":0.71799999999999997},"min":{"1min":0.33700000000000002,"5min":0.32100000000000001,"15min":0.30599999999999999},"max":{"1min":0.66300000000000003,"5min":3.9729999999999999,"15min":36.152999999999999},"last":0.45300000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.42299999999999999,"15min":0.38500000000000001},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.20599999999999999},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.443,"15min":0.40500000000000003},"min":{"1min":0.27100000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.376}]},{"osd":6,"last update":"Tue Feb 13 00:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.47299999999999998,"15min":0.433},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.25700000000000001},"max":{"1min":0.69899999999999995,"5min":3.153,"15min":3.153},"last":0.438},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.51600000000000001,"15min":0.45600000000000002},"min":{"1min":0.32000000000000001,"5min":0.31,"15min":0.254},"max":{"1min":0.59999999999999998,"5min":4.9390000000000001,"15min":4.9390000000000001},"last":0.47699999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.49199999999999999,"15min":0.47499999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61499999999999999,"5min":1.575,"15min":1.575},"last":0.432},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.49399999999999999,"15min":0.46999999999999997},"min":{"1min":0.30199999999999999,"5min":0.29699999999999999,"15min":0.28100000000000003},"max":{"1min":0.63300000000000001,"5min":2.46,"15min":2.46},"last":0.51400000000000001}]}]},{"osd":5,"up_from":0,"seq":2057289334787,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":3328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4456448,"data_stored":4001089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4558848,"data_stored":4001312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4583424,"data_stored":4001356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":483328,"data_stored":1130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":219,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":219,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":219,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":219,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:48:21.627 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-13T00:48:21.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool set default.rgw.control pg_num 22 2024-02-13T00:48:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:19.715226+0000 osd.0 (osd.0) 183 : cluster [DBG] 5.1b scrub starts 2024-02-13T00:48:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:19.718118+0000 osd.0 (osd.0) 184 : cluster [DBG] 5.1b scrub ok 2024-02-13T00:48:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:20.454227+0000 mgr.y (mgr.14544) 439 : cluster [DBG] pgmap v728: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.6 KiB/s rd, 127 B/s wr, 1 op/s 2024-02-13T00:48:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:20.545981+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e486: 8 total, 8 up, 4 in 2024-02-13T00:48:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:20.924546+0000 osd.1 (osd.1) 205 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:48:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:20.928154+0000 osd.1 (osd.1) 206 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:48:22.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 -- 172.21.15.162:0/2268758958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49500f73a0 msgr2=0x7f49501039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:22.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 --2- 172.21.15.162:0/2268758958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49500f73a0 0x7f49501039a0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 rx=0x7f4948009be0 tx=0x7f4948004d10).stop 2024-02-13T00:48:22.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 -- 172.21.15.162:0/2268758958 shutdown_connections 2024-02-13T00:48:22.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 --2- 172.21.15.162:0/2268758958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49500f73a0 0x7f49501039a0 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:22.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 --2- 172.21.15.162:0/2268758958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49500fcb30 0x7f49500fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:22.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 --2- 172.21.15.162:0/2268758958 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49500fd490 0x7f49500f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:22.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 -- 172.21.15.162:0/2268758958 >> 172.21.15.162:0/2268758958 conn(0x7f495006eb90 msgr2=0x7f49500f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:22.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 -- 172.21.15.162:0/2268758958 shutdown_connections 2024-02-13T00:48:22.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 -- 172.21.15.162:0/2268758958 wait complete. 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 librados: starting msgr at 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 librados: starting objecter 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 Processor -- start 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 -- start start 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 librados: setting wanted keys 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.404+0000 7f4956e44700 1 librados: calling monclient init 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4956e44700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49500f73a0 0x7f495010ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4956e44700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49500fcb30 0x7f495010d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4956e44700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49500fd490 0x7f4950107010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4956e44700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4950107500 con 0x7f49500fd490 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4956e44700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4950107670 con 0x7f49500fcb30 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4956e44700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4950107830 con 0x7f49500f73a0 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4954be0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49500f73a0 0x7f495010ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:22.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4954be0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49500f73a0 0x7f495010ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:33312/0 (socket says 172.21.15.162:33312) 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4954be0700 1 -- 172.21.15.162:0/4099172468 learned_addr learned my addr 172.21.15.162:0/4099172468 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f49553e1700 1 --2- 172.21.15.162:0/4099172468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49500fd490 0x7f4950107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f494ffff700 1 --2- 172.21.15.162:0/4099172468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49500fcb30 0x7f495010d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4954be0700 1 -- 172.21.15.162:0/4099172468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49500fcb30 msgr2=0x7f495010d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4954be0700 1 --2- 172.21.15.162:0/4099172468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49500fcb30 0x7f495010d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4954be0700 1 -- 172.21.15.162:0/4099172468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49500fd490 msgr2=0x7f4950107010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4954be0700 1 --2- 172.21.15.162:0/4099172468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49500fd490 0x7f4950107010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4954be0700 1 -- 172.21.15.162:0/4099172468 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c005040 con 0x7f49500f73a0 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4954be0700 1 --2- 172.21.15.162:0/4099172468 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49500f73a0 0x7f495010ce70 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7f49440041d0 tx=0x7f4944009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f494dffb700 10 client.?.objecter ms_handle_connect 0x7f49500f73a0 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f494dffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f494dffb700 1 -- 172.21.15.162:0/4099172468 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944007b40 con 0x7f49500f73a0 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f494dffb700 1 -- 172.21.15.162:0/4099172468 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f494400f040 con 0x7f49500f73a0 2024-02-13T00:48:22.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f494dffb700 1 -- 172.21.15.162:0/4099172468 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944014810 con 0x7f49500f73a0 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.405+0000 7f4956e44700 1 -- 172.21.15.162:0/4099172468 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49480097c0 con 0x7f49500f73a0 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.406+0000 7f4956e44700 10 client.37631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.406+0000 7f4956e44700 1 -- 172.21.15.162:0/4099172468 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4950107e70 con 0x7f49500f73a0 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.406+0000 7f4956e44700 1 librados: init done 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.406+0000 7f4956e44700 1 -- 172.21.15.162:0/4099172468 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4950106090 con 0x7f49500f73a0 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.407+0000 7f494dffb700 1 -- 172.21.15.162:0/4099172468 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4944009110 con 0x7f49500f73a0 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.407+0000 7f494dffb700 1 --2- 172.21.15.162:0/4099172468 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4940062c90 0x7f4940065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.407+0000 7f494dffb700 1 -- 172.21.15.162:0/4099172468 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(487..487 src has 52..487) v4 ==== 8070+0+0 (secure 0 0 0) 0x7f494407d0e0 con 0x7f49500f73a0 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.407+0000 7f494dffb700 10 client.37631.objecter ms_dispatch 0x7f4950001550 osd_map(487..487 src has 52..487) v4 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.407+0000 7f494dffb700 3 client.37631.objecter handle_osd_map got epochs [487,487] > 0 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.407+0000 7f494dffb700 3 client.37631.objecter handle_osd_map decoding full epoch 487 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.407+0000 7f494ffff700 1 --2- 172.21.15.162:0/4099172468 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4940062c90 0x7f4940065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:22.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.407+0000 7f494dffb700 20 client.37631.objecter dump_active .. 0 homeless 2024-02-13T00:48:22.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.407+0000 7f494dffb700 10 client.37631.objecter ms_handle_connect 0x7f4940062c90 2024-02-13T00:48:22.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.407+0000 7f494ffff700 1 --2- 172.21.15.162:0/4099172468 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4940062c90 0x7f4940065150 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 rx=0x7f49501087b0 tx=0x7f493c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:22.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.411+0000 7f494dffb700 1 -- 172.21.15.162:0/4099172468 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4944050680 con 0x7f49500f73a0 2024-02-13T00:48:22.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:22.602+0000 7f4956e44700 1 -- 172.21.15.162:0/4099172468 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f495005fc50 con 0x7f49500f73a0 2024-02-13T00:48:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:21.560144+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e487: 8 total, 8 up, 4 in 2024-02-13T00:48:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:21.565320+0000 mon.b (mon.1) 612 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:21.565961+0000 mon.a (mon.0) 2692 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:21.877061+0000 osd.1 (osd.1) 207 : cluster [DBG] 5.1d scrub starts 2024-02-13T00:48:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:21.879944+0000 osd.1 (osd.1) 208 : cluster [DBG] 5.1d scrub ok 2024-02-13T00:48:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:21.560144+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e487: 8 total, 8 up, 4 in 2024-02-13T00:48:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:21.565320+0000 mon.b (mon.1) 612 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:21.565961+0000 mon.a (mon.0) 2692 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:22.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:21.877061+0000 osd.1 (osd.1) 207 : cluster [DBG] 5.1d scrub starts 2024-02-13T00:48:22.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:21.879944+0000 osd.1 (osd.1) 208 : cluster [DBG] 5.1d scrub ok 2024-02-13T00:48:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:21.560144+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e487: 8 total, 8 up, 4 in 2024-02-13T00:48:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:21.565320+0000 mon.b (mon.1) 612 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:21.565961+0000 mon.a (mon.0) 2692 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:21.877061+0000 osd.1 (osd.1) 207 : cluster [DBG] 5.1d scrub starts 2024-02-13T00:48:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:21.879944+0000 osd.1 (osd.1) 208 : cluster [DBG] 5.1d scrub ok 2024-02-13T00:48:23.547 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:23.548 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:23.567 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:23.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:23.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.570+0000 7f494dffb700 1 -- 172.21.15.162:0/4099172468 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v489) v1 ==== 101+0+0 (secure 0 0 0) 0x7f4944050270 con 0x7f49500f73a0 2024-02-13T00:48:23.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.573+0000 7f4956e44700 10 librados: watch_flush enter 2024-02-13T00:48:23.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.573+0000 7f4956e44700 10 librados: watch_flush exit 2024-02-13T00:48:23.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.573+0000 7f4956e44700 20 client.37631.objecter shutdown clearing up homeless session... 2024-02-13T00:48:23.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.573+0000 7f4956e44700 10 client.37631.objecter successfully canceled tick 2024-02-13T00:48:23.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.573+0000 7f4956e44700 1 -- 172.21.15.162:0/4099172468 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4940062c90 msgr2=0x7f4940065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:23.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.573+0000 7f4956e44700 1 --2- 172.21.15.162:0/4099172468 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4940062c90 0x7f4940065150 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 rx=0x7f49501087b0 tx=0x7f493c00b040).stop 2024-02-13T00:48:23.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.573+0000 7f4956e44700 1 -- 172.21.15.162:0/4099172468 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49500f73a0 msgr2=0x7f495010ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:23.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.573+0000 7f4956e44700 1 --2- 172.21.15.162:0/4099172468 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49500f73a0 0x7f495010ce70 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7f49440041d0 tx=0x7f4944009f60).stop 2024-02-13T00:48:23.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.575+0000 7f4956e44700 1 -- 172.21.15.162:0/4099172468 shutdown_connections 2024-02-13T00:48:23.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.575+0000 7f4956e44700 1 --2- 172.21.15.162:0/4099172468 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4940062c90 0x7f4940065150 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:23.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.575+0000 7f4956e44700 1 --2- 172.21.15.162:0/4099172468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49500fd490 0x7f4950107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:23.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.575+0000 7f4956e44700 1 --2- 172.21.15.162:0/4099172468 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49500f73a0 0x7f495010ce70 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:23.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.575+0000 7f4956e44700 1 --2- 172.21.15.162:0/4099172468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49500fcb30 0x7f495010d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:23.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.575+0000 7f4956e44700 1 -- 172.21.15.162:0/4099172468 >> 172.21.15.162:0/4099172468 conn(0x7f495006eb90 msgr2=0x7f495006fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:23.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.576+0000 7f4956e44700 1 -- 172.21.15.162:0/4099172468 shutdown_connections 2024-02-13T00:48:23.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.580+0000 7f4956e44700 1 -- 172.21.15.162:0/4099172468 wait complete. 2024-02-13T00:48:23.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:23.580+0000 7f4956e44700 1 librados: shutdown 2024-02-13T00:48:23.591 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:48:23.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:23.704 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:23.726 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:23.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:23.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:23.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:23.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:22.454671+0000 mgr.y (mgr.14544) 440 : cluster [DBG] pgmap v731: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:48:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:22.570003+0000 mon.a (mon.0) 2693 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:48:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:22.570086+0000 mon.a (mon.0) 2694 : cluster [DBG] osdmap e488: 8 total, 8 up, 4 in 2024-02-13T00:48:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:22.574597+0000 mon.b (mon.1) 613 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:48:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:22.578300+0000 mon.b (mon.1) 614 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:22.580069+0000 mon.a (mon.0) 2695 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:22.604825+0000 mon.c (mon.2) 404 : audit [INF] from='client.? 172.21.15.162:0/4099172468' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:22.605105+0000 mon.a (mon.0) 2696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:23.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:22.454671+0000 mgr.y (mgr.14544) 440 : cluster [DBG] pgmap v731: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:48:23.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:22.570003+0000 mon.a (mon.0) 2693 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:48:23.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:22.570086+0000 mon.a (mon.0) 2694 : cluster [DBG] osdmap e488: 8 total, 8 up, 4 in 2024-02-13T00:48:23.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:22.574597+0000 mon.b (mon.1) 613 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:48:23.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:22.578300+0000 mon.b (mon.1) 614 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:23.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:22.580069+0000 mon.a (mon.0) 2695 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:23.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:22.604825+0000 mon.c (mon.2) 404 : audit [INF] from='client.? 172.21.15.162:0/4099172468' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:23.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:22.605105+0000 mon.a (mon.0) 2696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:23.861 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:22.454671+0000 mgr.y (mgr.14544) 440 : cluster [DBG] pgmap v731: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:48:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:22.570003+0000 mon.a (mon.0) 2693 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:48:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:22.570086+0000 mon.a (mon.0) 2694 : cluster [DBG] osdmap e488: 8 total, 8 up, 4 in 2024-02-13T00:48:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:22.574597+0000 mon.b (mon.1) 613 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:48:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:22.578300+0000 mon.b (mon.1) 614 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:22.580069+0000 mon.a (mon.0) 2695 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:22.604825+0000 mon.c (mon.2) 404 : audit [INF] from='client.? 172.21.15.162:0/4099172468' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:22.605105+0000 mon.a (mon.0) 2696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:24.893 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:23.376863+0000 osd.6 (osd.6) 137 : cluster [DBG] 2.e scrub starts 2024-02-13T00:48:24.893 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:23.379769+0000 osd.6 (osd.6) 138 : cluster [DBG] 2.e scrub ok 2024-02-13T00:48:24.893 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:23.564797+0000 mon.a (mon.0) 2697 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:48:24.893 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:23.571963+0000 mon.a (mon.0) 2698 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:48:24.893 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:23.572048+0000 mon.a (mon.0) 2699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:48:24.893 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:23.572102+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e489: 8 total, 8 up, 4 in 2024-02-13T00:48:24.893 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:23.573033+0000 mon.b (mon.1) 615 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:24.893 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:23.573691+0000 mon.a (mon.0) 2701 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:24.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:24.574048+0000 mon.a (mon.0) 2702 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:48:24.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:24.574117+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e490: 8 total, 8 up, 4 in 2024-02-13T00:48:24.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:24.576508+0000 mon.b (mon.1) 616 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:24.894 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:24.576904+0000 mon.a (mon.0) 2704 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:24.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:23.376863+0000 osd.6 (osd.6) 137 : cluster [DBG] 2.e scrub starts 2024-02-13T00:48:24.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:23.379769+0000 osd.6 (osd.6) 138 : cluster [DBG] 2.e scrub ok 2024-02-13T00:48:24.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:23.564797+0000 mon.a (mon.0) 2697 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:48:24.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:23.571963+0000 mon.a (mon.0) 2698 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:48:24.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:23.572048+0000 mon.a (mon.0) 2699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:48:24.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:23.572102+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e489: 8 total, 8 up, 4 in 2024-02-13T00:48:24.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:23.573033+0000 mon.b (mon.1) 615 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:24.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:23.573691+0000 mon.a (mon.0) 2701 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:24.895 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:24.574048+0000 mon.a (mon.0) 2702 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:48:24.895 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:24.574117+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e490: 8 total, 8 up, 4 in 2024-02-13T00:48:24.895 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:24.576508+0000 mon.b (mon.1) 616 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:24.895 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:24.576904+0000 mon.a (mon.0) 2704 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:23.376863+0000 osd.6 (osd.6) 137 : cluster [DBG] 2.e scrub starts 2024-02-13T00:48:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:23.379769+0000 osd.6 (osd.6) 138 : cluster [DBG] 2.e scrub ok 2024-02-13T00:48:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:23.564797+0000 mon.a (mon.0) 2697 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:48:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:23.571963+0000 mon.a (mon.0) 2698 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:48:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:23.572048+0000 mon.a (mon.0) 2699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:48:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:23.572102+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e489: 8 total, 8 up, 4 in 2024-02-13T00:48:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:23.573033+0000 mon.b (mon.1) 615 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:23.573691+0000 mon.a (mon.0) 2701 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:24.574048+0000 mon.a (mon.0) 2702 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:48:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:24.574117+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e490: 8 total, 8 up, 4 in 2024-02-13T00:48:25.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:24.576508+0000 mon.b (mon.1) 616 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:25.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:24.576904+0000 mon.a (mon.0) 2704 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:25.863 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:24.455208+0000 mgr.y (mgr.14544) 441 : cluster [DBG] pgmap v734: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:48:25.863 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:24.671573+0000 osd.0 (osd.0) 185 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:48:25.863 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:24.675679+0000 osd.0 (osd.0) 186 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:48:25.863 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:24.820068+0000 osd.1 (osd.1) 209 : cluster [DBG] 2.d scrub starts 2024-02-13T00:48:25.863 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:24.823103+0000 osd.1 (osd.1) 210 : cluster [DBG] 2.d scrub ok 2024-02-13T00:48:25.863 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:25.356434+0000 osd.6 (osd.6) 139 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:48:25.863 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:25.359544+0000 osd.6 (osd.6) 140 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:48:25.863 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:25.457956+0000 mon.a (mon.0) 2705 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:48:25.863 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:25.458047+0000 mon.a (mon.0) 2706 : cluster [DBG] osdmap e491: 8 total, 8 up, 4 in 2024-02-13T00:48:25.863 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:25.459161+0000 mon.b (mon.1) 617 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:25.863 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:25.461308+0000 mon.a (mon.0) 2707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:25.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:24.455208+0000 mgr.y (mgr.14544) 441 : cluster [DBG] pgmap v734: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:48:25.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:24.671573+0000 osd.0 (osd.0) 185 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:48:25.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:24.675679+0000 osd.0 (osd.0) 186 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:48:25.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:24.820068+0000 osd.1 (osd.1) 209 : cluster [DBG] 2.d scrub starts 2024-02-13T00:48:25.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:24.823103+0000 osd.1 (osd.1) 210 : cluster [DBG] 2.d scrub ok 2024-02-13T00:48:25.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:25.356434+0000 osd.6 (osd.6) 139 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:48:25.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:25.359544+0000 osd.6 (osd.6) 140 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:48:25.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:25.457956+0000 mon.a (mon.0) 2705 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:48:25.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:25.458047+0000 mon.a (mon.0) 2706 : cluster [DBG] osdmap e491: 8 total, 8 up, 4 in 2024-02-13T00:48:25.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:25.459161+0000 mon.b (mon.1) 617 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:25.864 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:25.461308+0000 mon.a (mon.0) 2707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:26.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:48:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:48:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:24.455208+0000 mgr.y (mgr.14544) 441 : cluster [DBG] pgmap v734: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:48:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:24.671573+0000 osd.0 (osd.0) 185 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:48:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:24.675679+0000 osd.0 (osd.0) 186 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:48:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:24.820068+0000 osd.1 (osd.1) 209 : cluster [DBG] 2.d scrub starts 2024-02-13T00:48:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:24.823103+0000 osd.1 (osd.1) 210 : cluster [DBG] 2.d scrub ok 2024-02-13T00:48:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:25.356434+0000 osd.6 (osd.6) 139 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:48:26.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:25.359544+0000 osd.6 (osd.6) 140 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:48:26.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:25.457956+0000 mon.a (mon.0) 2705 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:48:26.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:25.458047+0000 mon.a (mon.0) 2706 : cluster [DBG] osdmap e491: 8 total, 8 up, 4 in 2024-02-13T00:48:26.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:25.459161+0000 mon.b (mon.1) 617 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:26.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:25.461308+0000 mon.a (mon.0) 2707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:26.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:48:26] "GET /metrics HTTP/1.1" 200 188011 "" "Prometheus/2.33.4" 2024-02-13T00:48:26.994 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:25.706911+0000 mgr.y (mgr.14544) 442 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:26.995 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:26.460037+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:26.995 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:26.460137+0000 mon.a (mon.0) 2709 : cluster [DBG] osdmap e492: 8 total, 8 up, 4 in 2024-02-13T00:48:26.995 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:26.481133+0000 mon.b (mon.1) 618 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-13T00:48:26.995 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:26.481629+0000 mon.a (mon.0) 2710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-13T00:48:26.995 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:26.489399+0000 osd.0 (osd.0) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:26.995 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:26.489580+0000 osd.0 (osd.0) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:26.995 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:25.706911+0000 mgr.y (mgr.14544) 442 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:26.995 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:26.460037+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:26.995 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:26.460137+0000 mon.a (mon.0) 2709 : cluster [DBG] osdmap e492: 8 total, 8 up, 4 in 2024-02-13T00:48:26.996 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:26.481133+0000 mon.b (mon.1) 618 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-13T00:48:26.996 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:26.481629+0000 mon.a (mon.0) 2710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-13T00:48:26.996 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:26.489399+0000 osd.0 (osd.0) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:26.996 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:26.489580+0000 osd.0 (osd.0) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:25.706911+0000 mgr.y (mgr.14544) 442 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:26.460037+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:26.460137+0000 mon.a (mon.0) 2709 : cluster [DBG] osdmap e492: 8 total, 8 up, 4 in 2024-02-13T00:48:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:26.481133+0000 mon.b (mon.1) 618 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-13T00:48:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:26.481629+0000 mon.a (mon.0) 2710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-13T00:48:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:26.489399+0000 osd.0 (osd.0) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:27.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:26.489580+0000 osd.0 (osd.0) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:27.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:48:27.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:27.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:27.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:27.573 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:48:27.574 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:48:27.581 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:27.603 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:27.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:27.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:27.735 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:27.834 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:27.859 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:26.455675+0000 mgr.y (mgr.14544) 443 : cluster [DBG] pgmap v737: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:27.859 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:26.644432+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:48:27.859 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:26.647362+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:48:27.859 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:27.080483+0000 osd.6 (osd.6) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:27.859 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:27.080660+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:27.859 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:27.466695+0000 mon.a (mon.0) 2711 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-13T00:48:27.859 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:27.466791+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e493: 8 total, 8 up, 4 in 2024-02-13T00:48:27.859 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:27.488780+0000 mon.b (mon.1) 619 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-13T00:48:27.859 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:27.490024+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-13T00:48:27.860 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:26.455675+0000 mgr.y (mgr.14544) 443 : cluster [DBG] pgmap v737: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:27.860 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:26.644432+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:48:27.860 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:26.647362+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:48:27.860 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:27.080483+0000 osd.6 (osd.6) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:27.860 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:27.080660+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:27.860 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:27.466695+0000 mon.a (mon.0) 2711 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-13T00:48:27.860 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:27.466791+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e493: 8 total, 8 up, 4 in 2024-02-13T00:48:27.860 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:27.488780+0000 mon.b (mon.1) 619 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-13T00:48:27.860 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:27.490024+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-13T00:48:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:27.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:27.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:27.989 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:26.455675+0000 mgr.y (mgr.14544) 443 : cluster [DBG] pgmap v737: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:26.644432+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:48:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:26.647362+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:48:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:27.080483+0000 osd.6 (osd.6) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:27.080660+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:27.466695+0000 mon.a (mon.0) 2711 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-13T00:48:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:27.466791+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e493: 8 total, 8 up, 4 in 2024-02-13T00:48:28.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:27.488780+0000 mon.b (mon.1) 619 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-13T00:48:28.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:27.490024+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-13T00:48:28.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.043+0000 7fe22f8ec700 1 -- 172.21.15.162:0/851018538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2280fd490 msgr2=0x7fe2280f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.043+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/851018538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2280fd490 0x7fe2280f6cc0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 rx=0x7fe21c004870 tx=0x7fe21c009cb0).stop 2024-02-13T00:48:28.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.043+0000 7fe22f8ec700 1 -- 172.21.15.162:0/851018538 shutdown_connections 2024-02-13T00:48:28.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.043+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/851018538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2280f73a0 0x7fe2281039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.043+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/851018538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2280fcb30 0x7fe2280fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.043+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/851018538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2280fd490 0x7fe2280f6cc0 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.043+0000 7fe22f8ec700 1 -- 172.21.15.162:0/851018538 >> 172.21.15.162:0/851018538 conn(0x7fe22806eb90 msgr2=0x7fe2280f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:28.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 -- 172.21.15.162:0/851018538 shutdown_connections 2024-02-13T00:48:28.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 -- 172.21.15.162:0/851018538 wait complete. 2024-02-13T00:48:28.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 librados: starting msgr at 2024-02-13T00:48:28.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 librados: starting objecter 2024-02-13T00:48:28.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 Processor -- start 2024-02-13T00:48:28.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 -- start start 2024-02-13T00:48:28.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 librados: setting wanted keys 2024-02-13T00:48:28.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 librados: calling monclient init 2024-02-13T00:48:28.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2280f73a0 0x7fe2280f9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:28.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2280fcb30 0x7fe2280f9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:28.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2280fd490 0x7fe2280f9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe228116630 con 0x7fe2280fcb30 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2281167a0 con 0x7fe2280f73a0 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22f8ec700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe228116960 con 0x7fe2280fd490 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.044+0000 7fe22ce87700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2280fcb30 0x7fe2280f9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22ce87700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2280fcb30 0x7fe2280f9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59740/0 (socket says 172.21.15.162:59740) 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22de89700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2280fd490 0x7fe2280f9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22ce87700 1 -- 172.21.15.162:0/1715814163 learned_addr learned my addr 172.21.15.162:0/1715814163 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22d688700 1 --2- 172.21.15.162:0/1715814163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2280f73a0 0x7fe2280f9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22ce87700 1 -- 172.21.15.162:0/1715814163 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2280fd490 msgr2=0x7fe2280f9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22ce87700 1 --2- 172.21.15.162:0/1715814163 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2280fd490 0x7fe2280f9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22ce87700 1 -- 172.21.15.162:0/1715814163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2280f73a0 msgr2=0x7fe2280f9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22ce87700 1 --2- 172.21.15.162:0/1715814163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2280f73a0 0x7fe2280f9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22ce87700 1 -- 172.21.15.162:0/1715814163 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe218005040 con 0x7fe2280fcb30 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22de89700 1 --2- 172.21.15.162:0/1715814163 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2280fd490 0x7fe2280f9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:28.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22d688700 1 --2- 172.21.15.162:0/1715814163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2280f73a0 0x7fe2280f9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22ce87700 1 --2- 172.21.15.162:0/1715814163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2280fcb30 0x7fe2280f9640 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7fe21c009eb0 tx=0x7fe21c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe2267fc700 10 client.?.objecter ms_handle_connect 0x7fe2280fcb30 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe2267fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c00e070 con 0x7fe2280fcb30 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe21c013da0 con 0x7fe2280fcb30 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c005040 con 0x7fe2280fcb30 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c019070 con 0x7fe2280fcb30 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22f8ec700 10 client.27537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe228116eb0 con 0x7fe2280fcb30 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.045+0000 7fe22f8ec700 1 librados: init done 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.046+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe2280475e0 con 0x7fe2280fcb30 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.046+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe21c01e420 con 0x7fe2280fcb30 2024-02-13T00:48:28.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe2267fc700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe210062c90 0x7fe210065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(493..493 src has 52..493) v4 ==== 8150+0+0 (secure 0 0 0) 0x7fe21c07dd40 con 0x7fe2280fcb30 2024-02-13T00:48:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 osd_map(493..493 src has 52..493) v4 2024-02-13T00:48:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe2267fc700 3 client.27537.objecter handle_osd_map got epochs [493,493] > 0 2024-02-13T00:48:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe2267fc700 3 client.27537.objecter handle_osd_map decoding full epoch 493 2024-02-13T00:48:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22d688700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe210062c90 0x7fe210065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe2267fc700 20 client.27537.objecter dump_active .. 0 homeless 2024-02-13T00:48:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v493) v1 ==== 54+0+16 (secure 0 0 0) 0x7fe21c080410 con 0x7fe2280fcb30 2024-02-13T00:48:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe2267fc700 10 client.27537.objecter ms_handle_connect 0x7fe210062c90 2024-02-13T00:48:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22d688700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe210062c90 0x7fe210065150 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 rx=0x7fe218007fd0 tx=0x7fe218009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 1 2024-02-13T00:48:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe2280ffdb0 0x7fe228100190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228105fe0 osd=0 3 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228105fe0 osd=0 3 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228105fe0 osd=0 3 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 1 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe228105fe0 osd=0 3 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 0 1 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228105fe0 osd=0 4 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22f8ec700 10 client.27537.objecter _send_command 1 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2280fa630 con 0x7fe2280ffdb0 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.047+0000 7fe22de89700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe2280ffdb0 0x7fe228100190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.048+0000 7fe22de89700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe2280ffdb0 0x7fe228100190 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.048+0000 7fe2267fc700 10 client.27537.objecter ms_handle_connect 0x7fe2280ffdb0 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.048+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe2280fa630 con 0x7fe2280ffdb0 2024-02-13T00:48:28.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.048+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:48:28.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.048+0000 7fe2267fc700 10 client.27537.objecter _finish_command 1 = system:0 2024-02-13T00:48:28.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.048+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe228105fe0 osd=0 4 2024-02-13T00:48:28.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.048+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 0 1 2024-02-13T00:48:28.062 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:28.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:28.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.065+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:28.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.065+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 2 2024-02-13T00:48:28.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.065+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228105fe0 osd=0 3 2024-02-13T00:48:28.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.065+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228105fe0 osd=0 3 2024-02-13T00:48:28.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.065+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228105fe0 osd=0 3 2024-02-13T00:48:28.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.065+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 2 2024-02-13T00:48:28.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.065+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe228105fe0 osd=0 3 2024-02-13T00:48:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.065+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 0 2 2024-02-13T00:48:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.065+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228105fe0 osd=0 4 2024-02-13T00:48:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.065+0000 7fe22f8ec700 10 client.27537.objecter _send_command 2 2024-02-13T00:48:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.065+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe228027410 con 0x7fe2280ffdb0 2024-02-13T00:48:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.066+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe228027410 con 0x7fe2280ffdb0 2024-02-13T00:48:28.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.066+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:28.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.066+0000 7fe2267fc700 10 client.27537.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.066+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe228105fe0 osd=0 4 2024-02-13T00:48:28.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.066+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 0 2 2024-02-13T00:48:28.071 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:48:28.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:28.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 3 2024-02-13T00:48:28.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6805/150599780 conn(0x7fe228059990 0x7fe228059d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:28.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228028760 osd=1 3 2024-02-13T00:48:28.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228028760 osd=1 3 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228028760 osd=1 3 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 3 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe228028760 osd=1 3 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 1 3 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228028760 osd=1 4 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22f8ec700 10 client.27537.objecter _send_command 3 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe22811d7c0 con 0x7fe228059990 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22d688700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6805/150599780 conn(0x7fe228059990 0x7fe228059d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe2267fc700 10 client.27537.objecter ms_handle_connect 0x7fe228059990 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.067+0000 7fe22d688700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6805/150599780 conn(0x7fe228059990 0x7fe228059d90 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.068+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe22811d7c0 con 0x7fe228059990 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.068+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.068+0000 7fe2267fc700 10 client.27537.objecter _finish_command 3 = system:0 2024-02-13T00:48:28.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.068+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe228028760 osd=1 4 2024-02-13T00:48:28.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.068+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 1 3 2024-02-13T00:48:28.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.082+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:28.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.082+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 4 2024-02-13T00:48:28.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.082+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228028760 osd=1 3 2024-02-13T00:48:28.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.082+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228028760 osd=1 3 2024-02-13T00:48:28.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.082+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228028760 osd=1 3 2024-02-13T00:48:28.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.082+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 4 2024-02-13T00:48:28.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.082+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe228028760 osd=1 3 2024-02-13T00:48:28.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.082+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 1 4 2024-02-13T00:48:28.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.082+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228028760 osd=1 4 2024-02-13T00:48:28.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.082+0000 7fe22f8ec700 10 client.27537.objecter _send_command 4 2024-02-13T00:48:28.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.082+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe228059400 con 0x7fe228059990 2024-02-13T00:48:28.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe228059400 con 0x7fe228059990 2024-02-13T00:48:28.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:28.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe2267fc700 10 client.27537.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe228028760 osd=1 4 2024-02-13T00:48:28.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 1 4 2024-02-13T00:48:28.088 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:48:28.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:28.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 5 2024-02-13T00:48:28.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe2280282c0 0x7fe22811c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228059810 osd=2 3 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228059810 osd=2 3 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228059810 osd=2 3 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 5 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe228059810 osd=2 3 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 2 5 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228059810 osd=2 4 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22f8ec700 10 client.27537.objecter _send_command 5 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe22811c760 con 0x7fe2280282c0 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22ce87700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe2280282c0 0x7fe22811c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe2267fc700 10 client.27537.objecter ms_handle_connect 0x7fe2280282c0 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.084+0000 7fe22ce87700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe2280282c0 0x7fe22811c1e0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.085+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe22811c760 con 0x7fe2280282c0 2024-02-13T00:48:28.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.085+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:48:28.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.085+0000 7fe2267fc700 10 client.27537.objecter _finish_command 5 = system:0 2024-02-13T00:48:28.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.085+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe228059810 osd=2 4 2024-02-13T00:48:28.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.085+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 2 5 2024-02-13T00:48:28.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.099+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:28.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.099+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 6 2024-02-13T00:48:28.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.099+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228059810 osd=2 3 2024-02-13T00:48:28.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.099+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228059810 osd=2 3 2024-02-13T00:48:28.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.099+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228059810 osd=2 3 2024-02-13T00:48:28.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.099+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 6 2024-02-13T00:48:28.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.099+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe228059810 osd=2 3 2024-02-13T00:48:28.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.099+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 2 6 2024-02-13T00:48:28.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.099+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228059810 osd=2 4 2024-02-13T00:48:28.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.099+0000 7fe22f8ec700 10 client.27537.objecter _send_command 6 2024-02-13T00:48:28.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.099+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe228027880 con 0x7fe2280282c0 2024-02-13T00:48:28.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.100+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe228027880 con 0x7fe2280282c0 2024-02-13T00:48:28.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.100+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:28.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.100+0000 7fe2267fc700 10 client.27537.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.100+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe228059810 osd=2 4 2024-02-13T00:48:28.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.100+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 2 6 2024-02-13T00:48:28.105 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:48:28.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.100+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:28.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 7 2024-02-13T00:48:28.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6813/604971011 conn(0x7fe22811d2c0 0x7fe228121e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:28.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22811d150 osd=3 3 2024-02-13T00:48:28.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811d150 osd=3 3 2024-02-13T00:48:28.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22811d150 osd=3 3 2024-02-13T00:48:28.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 7 2024-02-13T00:48:28.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe22811d150 osd=3 3 2024-02-13T00:48:28.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 3 7 2024-02-13T00:48:28.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811d150 osd=3 4 2024-02-13T00:48:28.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22f8ec700 10 client.27537.objecter _send_command 7 2024-02-13T00:48:28.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe228124ab0 con 0x7fe22811d2c0 2024-02-13T00:48:28.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22de89700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6813/604971011 conn(0x7fe22811d2c0 0x7fe228121e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:28.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe2267fc700 10 client.27537.objecter ms_handle_connect 0x7fe22811d2c0 2024-02-13T00:48:28.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe22de89700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6813/604971011 conn(0x7fe22811d2c0 0x7fe228121e60 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:28.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe228124ab0 con 0x7fe22811d2c0 2024-02-13T00:48:28.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:48:28.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.101+0000 7fe2267fc700 10 client.27537.objecter _finish_command 7 = system:0 2024-02-13T00:48:28.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.102+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe22811d150 osd=3 4 2024-02-13T00:48:28.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.102+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 3 7 2024-02-13T00:48:28.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.116+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:28.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.116+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 8 2024-02-13T00:48:28.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.116+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22811d150 osd=3 3 2024-02-13T00:48:28.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.116+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811d150 osd=3 3 2024-02-13T00:48:28.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.116+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22811d150 osd=3 3 2024-02-13T00:48:28.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.116+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 8 2024-02-13T00:48:28.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.116+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe22811d150 osd=3 3 2024-02-13T00:48:28.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.116+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 3 8 2024-02-13T00:48:28.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.116+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811d150 osd=3 4 2024-02-13T00:48:28.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.116+0000 7fe22f8ec700 10 client.27537.objecter _send_command 8 2024-02-13T00:48:28.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.116+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe22811cbc0 con 0x7fe22811d2c0 2024-02-13T00:48:28.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe22811cbc0 con 0x7fe22811d2c0 2024-02-13T00:48:28.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:28.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe2267fc700 10 client.27537.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe22811d150 osd=3 4 2024-02-13T00:48:28.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 3 8 2024-02-13T00:48:28.122 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:48:28.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:28.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 9 2024-02-13T00:48:28.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe22811bd40 0x7fe2281234d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:28.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22811cbc0 osd=4 3 2024-02-13T00:48:28.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811cbc0 osd=4 3 2024-02-13T00:48:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22811cbc0 osd=4 3 2024-02-13T00:48:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 9 2024-02-13T00:48:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe22811cbc0 osd=4 3 2024-02-13T00:48:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 4 9 2024-02-13T00:48:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811cbc0 osd=4 4 2024-02-13T00:48:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe22f8ec700 10 client.27537.objecter _send_command 9 2024-02-13T00:48:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.117+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe22811ce80 con 0x7fe22811bd40 2024-02-13T00:48:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.118+0000 7fe22d688700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe22811bd40 0x7fe2281234d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.118+0000 7fe2267fc700 10 client.27537.objecter ms_handle_connect 0x7fe22811bd40 2024-02-13T00:48:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.118+0000 7fe22d688700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe22811bd40 0x7fe2281234d0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.119+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe22811ce80 con 0x7fe22811bd40 2024-02-13T00:48:28.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.119+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:48:28.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.119+0000 7fe2267fc700 10 client.27537.objecter _finish_command 9 = system:0 2024-02-13T00:48:28.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.119+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe22811cbc0 osd=4 4 2024-02-13T00:48:28.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.119+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 4 9 2024-02-13T00:48:28.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.133+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:28.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.133+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 10 2024-02-13T00:48:28.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.133+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22811cbc0 osd=4 3 2024-02-13T00:48:28.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.133+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811cbc0 osd=4 3 2024-02-13T00:48:28.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.133+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22811cbc0 osd=4 3 2024-02-13T00:48:28.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.133+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 10 2024-02-13T00:48:28.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.133+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe22811cbc0 osd=4 3 2024-02-13T00:48:28.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.133+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 4 10 2024-02-13T00:48:28.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.133+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811cbc0 osd=4 4 2024-02-13T00:48:28.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.133+0000 7fe22f8ec700 10 client.27537.objecter _send_command 10 2024-02-13T00:48:28.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.133+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe22811ce80 con 0x7fe22811bd40 2024-02-13T00:48:28.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe22811ce80 con 0x7fe22811bd40 2024-02-13T00:48:28.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:28.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe2267fc700 10 client.27537.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe22811cbc0 osd=4 4 2024-02-13T00:48:28.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 4 10 2024-02-13T00:48:28.140 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:48:28.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:28.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 11 2024-02-13T00:48:28.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe228123f40 0x7fe228124360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:28.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22811ce80 osd=5 3 2024-02-13T00:48:28.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811ce80 osd=5 3 2024-02-13T00:48:28.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22811ce80 osd=5 3 2024-02-13T00:48:28.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 11 2024-02-13T00:48:28.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe22811ce80 osd=5 3 2024-02-13T00:48:28.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 5 11 2024-02-13T00:48:28.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811ce80 osd=5 4 2024-02-13T00:48:28.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22f8ec700 10 client.27537.objecter _send_command 11 2024-02-13T00:48:28.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe22812b420 con 0x7fe228123f40 2024-02-13T00:48:28.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22ce87700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe228123f40 0x7fe228124360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:28.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe2267fc700 10 client.27537.objecter ms_handle_connect 0x7fe228123f40 2024-02-13T00:48:28.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe22ce87700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe228123f40 0x7fe228124360 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:28.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe22812b420 con 0x7fe228123f40 2024-02-13T00:48:28.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:48:28.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe2267fc700 10 client.27537.objecter _finish_command 11 = system:0 2024-02-13T00:48:28.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe22811ce80 osd=5 4 2024-02-13T00:48:28.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.136+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 5 11 2024-02-13T00:48:28.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.151+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:28.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.151+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 12 2024-02-13T00:48:28.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.151+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22811ce80 osd=5 3 2024-02-13T00:48:28.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.151+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811ce80 osd=5 3 2024-02-13T00:48:28.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.151+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22811ce80 osd=5 3 2024-02-13T00:48:28.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.151+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 12 2024-02-13T00:48:28.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.151+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe22811ce80 osd=5 3 2024-02-13T00:48:28.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.151+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 5 12 2024-02-13T00:48:28.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.151+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811ce80 osd=5 4 2024-02-13T00:48:28.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.151+0000 7fe22f8ec700 10 client.27537.objecter _send_command 12 2024-02-13T00:48:28.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.151+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe228128cc0 con 0x7fe228123f40 2024-02-13T00:48:28.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe228128cc0 con 0x7fe228123f40 2024-02-13T00:48:28.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:28.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe2267fc700 10 client.27537.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe22811ce80 osd=5 4 2024-02-13T00:48:28.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 5 12 2024-02-13T00:48:28.158 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:48:28.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:28.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 13 2024-02-13T00:48:28.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe228129340 0x7fe22812b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:28.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228128cc0 osd=6 3 2024-02-13T00:48:28.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228128cc0 osd=6 3 2024-02-13T00:48:28.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228128cc0 osd=6 3 2024-02-13T00:48:28.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 13 2024-02-13T00:48:28.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe228128cc0 osd=6 3 2024-02-13T00:48:28.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 6 13 2024-02-13T00:48:28.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228128cc0 osd=6 4 2024-02-13T00:48:28.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22f8ec700 10 client.27537.objecter _send_command 13 2024-02-13T00:48:28.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe22812e580 con 0x7fe228129340 2024-02-13T00:48:28.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22de89700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe228129340 0x7fe22812b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:28.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe22de89700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe228129340 0x7fe22812b7b0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:28.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.153+0000 7fe2267fc700 10 client.27537.objecter ms_handle_connect 0x7fe228129340 2024-02-13T00:48:28.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.154+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe22812e580 con 0x7fe228129340 2024-02-13T00:48:28.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.154+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:48:28.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.154+0000 7fe2267fc700 10 client.27537.objecter _finish_command 13 = system:0 2024-02-13T00:48:28.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.154+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe228128cc0 osd=6 4 2024-02-13T00:48:28.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.154+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 6 13 2024-02-13T00:48:28.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:28.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.169+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:28.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.169+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 14 2024-02-13T00:48:28.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.169+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228128cc0 osd=6 3 2024-02-13T00:48:28.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.169+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228128cc0 osd=6 3 2024-02-13T00:48:28.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.169+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe228128cc0 osd=6 3 2024-02-13T00:48:28.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.169+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 14 2024-02-13T00:48:28.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.169+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe228128cc0 osd=6 3 2024-02-13T00:48:28.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.169+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 6 14 2024-02-13T00:48:28.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.169+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228128cc0 osd=6 4 2024-02-13T00:48:28.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.169+0000 7fe22f8ec700 10 client.27537.objecter _send_command 14 2024-02-13T00:48:28.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.169+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe22812e580 con 0x7fe228129340 2024-02-13T00:48:28.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe22812e580 con 0x7fe228129340 2024-02-13T00:48:28.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:28.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe2267fc700 10 client.27537.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe228128cc0 osd=6 4 2024-02-13T00:48:28.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 6 14 2024-02-13T00:48:28.176 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:48:28.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 15 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe22812d9f0 0x7fe22812de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22812d370 osd=7 3 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22812d370 osd=7 3 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22812d370 osd=7 3 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 15 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe22812d370 osd=7 3 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 7 15 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22812d370 osd=7 4 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22f8ec700 10 client.27537.objecter _send_command 15 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe228131960 con 0x7fe22812d9f0 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22d688700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe22812d9f0 0x7fe22812de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:28.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe2267fc700 10 client.27537.objecter ms_handle_connect 0x7fe22812d9f0 2024-02-13T00:48:28.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.171+0000 7fe22d688700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe22812d9f0 0x7fe22812de50 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:28.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.172+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe228131960 con 0x7fe22812d9f0 2024-02-13T00:48:28.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.172+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:48:28.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.172+0000 7fe2267fc700 10 client.27537.objecter _finish_command 15 = system:0 2024-02-13T00:48:28.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.172+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe22812d370 osd=7 4 2024-02-13T00:48:28.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.172+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 7 15 2024-02-13T00:48:28.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.186+0000 7fe22f8ec700 10 client.27537.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:28.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.186+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign -1 16 2024-02-13T00:48:28.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.186+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22812d370 osd=7 3 2024-02-13T00:48:28.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.186+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22812d370 osd=7 3 2024-02-13T00:48:28.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.186+0000 7fe22f8ec700 20 client.27537.objecter _get_session s=0x7fe22812d370 osd=7 3 2024-02-13T00:48:28.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.186+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_remove -1 16 2024-02-13T00:48:28.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.186+0000 7fe22f8ec700 20 client.27537.objecter get_session s=0x7fe22812d370 osd=7 3 2024-02-13T00:48:28.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.186+0000 7fe22f8ec700 15 client.27537.objecter _session_command_op_assign 7 16 2024-02-13T00:48:28.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.186+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22812d370 osd=7 4 2024-02-13T00:48:28.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.186+0000 7fe22f8ec700 10 client.27537.objecter _send_command 16 2024-02-13T00:48:28.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.186+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe22812bd60 con 0x7fe22812d9f0 2024-02-13T00:48:28.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe2267fc700 1 -- 172.21.15.162:0/1715814163 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe22812bd60 con 0x7fe22812d9f0 2024-02-13T00:48:28.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe2267fc700 10 client.27537.objecter ms_dispatch 0x7fe228001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:28.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe2267fc700 10 client.27537.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe2267fc700 20 client.27537.objecter put_session s=0x7fe22812d370 osd=7 4 2024-02-13T00:48:28.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe2267fc700 15 client.27537.objecter _session_command_op_remove 7 16 2024-02-13T00:48:28.193 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:48:28.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 10 librados: watch_flush enter 2024-02-13T00:48:28.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 10 librados: watch_flush exit 2024-02-13T00:48:28.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 10 client.27537.objecter close_session for osd.0 2024-02-13T00:48:28.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe2280ffdb0 msgr2=0x7fe228100190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe2280ffdb0 0x7fe228100190 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228105fe0 osd=0 1 2024-02-13T00:48:28.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 10 client.27537.objecter close_session for osd.1 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6805/150599780 conn(0x7fe228059990 msgr2=0x7fe228059d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6805/150599780 conn(0x7fe228059990 0x7fe228059d90 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228028760 osd=1 1 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 10 client.27537.objecter close_session for osd.2 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe2280282c0 msgr2=0x7fe22811c1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe2280282c0 0x7fe22811c1e0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228059810 osd=2 1 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 10 client.27537.objecter close_session for osd.3 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6813/604971011 conn(0x7fe22811d2c0 msgr2=0x7fe228121e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6813/604971011 conn(0x7fe22811d2c0 0x7fe228121e60 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811d150 osd=3 1 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 10 client.27537.objecter close_session for osd.4 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe22811bd40 msgr2=0x7fe2281234d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe22811bd40 0x7fe2281234d0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811cbc0 osd=4 1 2024-02-13T00:48:28.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 10 client.27537.objecter close_session for osd.5 2024-02-13T00:48:28.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe228123f40 msgr2=0x7fe228124360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe228123f40 0x7fe228124360 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22de89700 1 -- 172.21.15.162:0/1715814163 reap_dead start 2024-02-13T00:48:28.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22de89700 1 -- 172.21.15.162:0/1715814163 reap_dead start 2024-02-13T00:48:28.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22811ce80 osd=5 1 2024-02-13T00:48:28.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 10 client.27537.objecter close_session for osd.6 2024-02-13T00:48:28.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe228129340 msgr2=0x7fe22812b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe228129340 0x7fe22812b7b0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe228128cc0 osd=6 1 2024-02-13T00:48:28.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 10 client.27537.objecter close_session for osd.7 2024-02-13T00:48:28.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe22812d9f0 msgr2=0x7fe22812de50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe22812d9f0 0x7fe22812de50 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 20 client.27537.objecter put_session s=0x7fe22812d370 osd=7 1 2024-02-13T00:48:28.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 20 client.27537.objecter shutdown clearing up homeless session... 2024-02-13T00:48:28.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22f8ec700 10 client.27537.objecter successfully canceled tick 2024-02-13T00:48:28.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.188+0000 7fe22de89700 1 -- 172.21.15.162:0/1715814163 reap_dead start 2024-02-13T00:48:28.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.189+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe210062c90 msgr2=0x7fe210065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.189+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe210062c90 0x7fe210065150 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 rx=0x7fe218007fd0 tx=0x7fe218009cb0).stop 2024-02-13T00:48:28.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.189+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2280fcb30 msgr2=0x7fe2280f9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:28.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.189+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2280fcb30 0x7fe2280f9640 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7fe21c009eb0 tx=0x7fe21c00b040).stop 2024-02-13T00:48:28.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.189+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 shutdown_connections 2024-02-13T00:48:28.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.189+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2280fcb30 0x7fe2280f9640 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.189+0000 7fe22f8ec700 1 --2- 172.21.15.162:0/1715814163 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe210062c90 0x7fe210065150 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:28.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.189+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 >> 172.21.15.162:0/1715814163 conn(0x7fe22806eb90 msgr2=0x7fe228060e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:28.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.189+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 shutdown_connections 2024-02-13T00:48:28.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.189+0000 7fe22f8ec700 1 -- 172.21.15.162:0/1715814163 wait complete. 2024-02-13T00:48:28.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:28.189+0000 7fe22f8ec700 1 librados: shutdown 2024-02-13T00:48:28.198 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.198 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.198 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.198 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.199 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.199 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.199 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.199 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:48:28.199 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:28.220 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:28.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:28.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:28.352 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:28.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-13T00:48:29.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:48:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:48:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:48:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:27.657973+0000 osd.0 (osd.0) 191 : cluster [DBG] 6.10 scrub starts 2024-02-13T00:48:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:27.659193+0000 osd.0 (osd.0) 192 : cluster [DBG] 6.10 scrub ok 2024-02-13T00:48:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:27.748019+0000 osd.1 (osd.1) 211 : cluster [DBG] 5.17 scrub starts 2024-02-13T00:48:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:27.749337+0000 osd.1 (osd.1) 212 : cluster [DBG] 5.17 scrub ok 2024-02-13T00:48:29.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:28.048817+0000 mon.a (mon.0) 2714 : audit [DBG] from='client.? 172.21.15.162:0/1715814163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:29.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:28.350409+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:48:29.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:28.353337+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:48:29.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:28.468641+0000 mon.a (mon.0) 2715 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-13T00:48:29.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:28.468699+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e494: 8 total, 8 up, 4 in 2024-02-13T00:48:29.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:28.484316+0000 mon.b (mon.1) 620 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:29.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:28.484934+0000 mon.a (mon.0) 2717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:29.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:27.657973+0000 osd.0 (osd.0) 191 : cluster [DBG] 6.10 scrub starts 2024-02-13T00:48:29.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:27.659193+0000 osd.0 (osd.0) 192 : cluster [DBG] 6.10 scrub ok 2024-02-13T00:48:29.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:27.748019+0000 osd.1 (osd.1) 211 : cluster [DBG] 5.17 scrub starts 2024-02-13T00:48:29.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:27.749337+0000 osd.1 (osd.1) 212 : cluster [DBG] 5.17 scrub ok 2024-02-13T00:48:29.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:28.048817+0000 mon.a (mon.0) 2714 : audit [DBG] from='client.? 172.21.15.162:0/1715814163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:29.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:28.350409+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:48:29.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:28.353337+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:48:29.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:28.468641+0000 mon.a (mon.0) 2715 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-13T00:48:29.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:28.468699+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e494: 8 total, 8 up, 4 in 2024-02-13T00:48:29.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:28.484316+0000 mon.b (mon.1) 620 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:29.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:28.484934+0000 mon.a (mon.0) 2717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:29.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:27.657973+0000 osd.0 (osd.0) 191 : cluster [DBG] 6.10 scrub starts 2024-02-13T00:48:29.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:27.659193+0000 osd.0 (osd.0) 192 : cluster [DBG] 6.10 scrub ok 2024-02-13T00:48:29.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:27.748019+0000 osd.1 (osd.1) 211 : cluster [DBG] 5.17 scrub starts 2024-02-13T00:48:29.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:27.749337+0000 osd.1 (osd.1) 212 : cluster [DBG] 5.17 scrub ok 2024-02-13T00:48:29.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:28.048817+0000 mon.a (mon.0) 2714 : audit [DBG] from='client.? 172.21.15.162:0/1715814163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:29.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:28.350409+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:48:29.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:28.353337+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:48:29.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:28.468641+0000 mon.a (mon.0) 2715 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-13T00:48:29.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:28.468699+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e494: 8 total, 8 up, 4 in 2024-02-13T00:48:29.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:28.484316+0000 mon.b (mon.1) 620 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:29.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:28.484934+0000 mon.a (mon.0) 2717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:29.099 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:29.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:29.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:29.230 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:29.679 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:29.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:29.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:29.818 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:29.841 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:29.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:29.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 -- 172.21.15.162:0/3111162305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc0f9b70 msgr2=0x7f84fc0f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 --2- 172.21.15.162:0/3111162305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc0f9b70 0x7f84fc0f9f50 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 rx=0x7f84ec004870 tx=0x7f84ec009cb0).stop 2024-02-13T00:48:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 -- 172.21.15.162:0/3111162305 shutdown_connections 2024-02-13T00:48:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 --2- 172.21.15.162:0/3111162305 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84fc0f8500 0x7f84fc104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:29.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 --2- 172.21.15.162:0/3111162305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc0f9b70 0x7f84fc0f9f50 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 --2- 172.21.15.162:0/3111162305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84fc0fa440 0x7f84fc0f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 -- 172.21.15.162:0/3111162305 >> 172.21.15.162:0/3111162305 conn(0x7f84fc0f38c0 msgr2=0x7f84fc0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 -- 172.21.15.162:0/3111162305 shutdown_connections 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 -- 172.21.15.162:0/3111162305 wait complete. 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 librados: starting msgr at 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 librados: starting objecter 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 Processor -- start 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 -- start start 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 librados: setting wanted keys 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 librados: calling monclient init 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc0f8500 0x7f84fc06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84fc0f9b70 0x7f84fc06b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.916+0000 7f8502437700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84fc0fa440 0x7f84fc06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f8502437700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc116690 con 0x7f84fc0f8500 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f8502437700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc116800 con 0x7f84fc0f9b70 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f8502437700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc1169c0 con 0x7f84fc0fa440 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84fbfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc0f8500 0x7f84fc06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f85009d4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84fc0fa440 0x7f84fc06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84fbfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc0f8500 0x7f84fc06b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59756/0 (socket says 172.21.15.162:59756) 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84fbfff700 1 -- 172.21.15.162:0/1725080853 learned_addr learned my addr 172.21.15.162:0/1725080853 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f85009d4700 1 --2- 172.21.15.162:0/1725080853 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84fc0fa440 0x7f84fc06bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:34356/0 (socket says 172.21.15.162:34356) 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84f3fff700 1 --2- 172.21.15.162:0/1725080853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84fc0f9b70 0x7f84fc06b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84fbfff700 1 -- 172.21.15.162:0/1725080853 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84fc0fa440 msgr2=0x7f84fc06bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84fbfff700 1 --2- 172.21.15.162:0/1725080853 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84fc0fa440 0x7f84fc06bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84fbfff700 1 -- 172.21.15.162:0/1725080853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84fc0f9b70 msgr2=0x7f84fc06b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84fbfff700 1 --2- 172.21.15.162:0/1725080853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84fc0f9b70 0x7f84fc06b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:29.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84fbfff700 1 -- 172.21.15.162:0/1725080853 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f40097c0 con 0x7f84fc0f8500 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f85009d4700 1 --2- 172.21.15.162:0/1725080853 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84fc0fa440 0x7f84fc06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84fbfff700 1 --2- 172.21.15.162:0/1725080853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc0f8500 0x7f84fc06b170 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f84ec009eb0 tx=0x7f84ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84f9ffb700 10 client.?.objecter ms_handle_connect 0x7f84fc0f8500 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84f9ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84f9ffb700 1 -- 172.21.15.162:0/1725080853 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec00e070 con 0x7f84fc0f8500 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84f9ffb700 1 -- 172.21.15.162:0/1725080853 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84ec013da0 con 0x7f84fc0f8500 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f8502437700 1 -- 172.21.15.162:0/1725080853 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ec005040 con 0x7f84fc0f8500 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f84f9ffb700 1 -- 172.21.15.162:0/1725080853 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec019070 con 0x7f84fc0f8500 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f8502437700 10 client.27543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f8502437700 1 -- 172.21.15.162:0/1725080853 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84fc116f90 con 0x7f84fc0f8500 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.917+0000 7f8502437700 1 librados: init done 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.918+0000 7f8502437700 1 -- 172.21.15.162:0/1725080853 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84fc0475e0 con 0x7f84fc0f8500 2024-02-13T00:48:29.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.919+0000 7f84f9ffb700 1 -- 172.21.15.162:0/1725080853 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f84ec01e420 con 0x7f84fc0f8500 2024-02-13T00:48:29.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.919+0000 7f84f9ffb700 1 --2- 172.21.15.162:0/1725080853 >> v2:172.21.15.162:6800/2866534053 conn(0x7f84e8062c90 0x7f84e8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:29.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.919+0000 7f84f9ffb700 1 -- 172.21.15.162:0/1725080853 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(495..495 src has 52..495) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f84ec07ddf0 con 0x7f84fc0f8500 2024-02-13T00:48:29.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.919+0000 7f84f9ffb700 10 client.27543.objecter ms_dispatch 0x7f84fc001550 osd_map(495..495 src has 52..495) v4 2024-02-13T00:48:29.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.919+0000 7f84f9ffb700 3 client.27543.objecter handle_osd_map got epochs [495,495] > 0 2024-02-13T00:48:29.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.919+0000 7f84f9ffb700 3 client.27543.objecter handle_osd_map decoding full epoch 495 2024-02-13T00:48:29.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.919+0000 7f84f3fff700 1 --2- 172.21.15.162:0/1725080853 >> v2:172.21.15.162:6800/2866534053 conn(0x7f84e8062c90 0x7f84e8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:29.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.919+0000 7f84f9ffb700 20 client.27543.objecter dump_active .. 0 homeless 2024-02-13T00:48:29.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.919+0000 7f84f9ffb700 10 client.27543.objecter ms_handle_connect 0x7f84e8062c90 2024-02-13T00:48:29.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.919+0000 7f84f3fff700 1 --2- 172.21.15.162:0/1725080853 >> v2:172.21.15.162:6800/2866534053 conn(0x7f84e8062c90 0x7f84e8065150 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 rx=0x7f84fc06c8c0 tx=0x7f84e4009d60).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:29.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:29.924+0000 7f84f9ffb700 1 -- 172.21.15.162:0/1725080853 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f84ec0512a0 con 0x7f84fc0f8500 2024-02-13T00:48:29.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:29.973 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:28.456109+0000 mgr.y (mgr.14544) 444 : cluster [DBG] pgmap v740: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:29.973 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:28.639285+0000 osd.4 (osd.4) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:29.973 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:28.639551+0000 osd.4 (osd.4) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:29.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:29.469569+0000 mon.a (mon.0) 2718 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:29.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:29.469654+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e495: 8 total, 8 up, 4 in 2024-02-13T00:48:29.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:29.500459+0000 mon.b (mon.1) 621 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:29.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:29.500944+0000 mon.a (mon.0) 2720 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:29.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:29.501535+0000 mon.b (mon.1) 622 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-77"}]: dispatch 2024-02-13T00:48:29.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:29.501844+0000 mon.a (mon.0) 2721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-77"}]: dispatch 2024-02-13T00:48:29.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:28.456109+0000 mgr.y (mgr.14544) 444 : cluster [DBG] pgmap v740: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:29.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:28.639285+0000 osd.4 (osd.4) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:29.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:28.639551+0000 osd.4 (osd.4) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:29.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:29.469569+0000 mon.a (mon.0) 2718 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:29.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:29.469654+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e495: 8 total, 8 up, 4 in 2024-02-13T00:48:29.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:29.500459+0000 mon.b (mon.1) 621 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:29.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:29.500944+0000 mon.a (mon.0) 2720 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:29.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:29.501535+0000 mon.b (mon.1) 622 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-77"}]: dispatch 2024-02-13T00:48:29.975 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:29.501844+0000 mon.a (mon.0) 2721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-77"}]: dispatch 2024-02-13T00:48:29.975 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:29.975 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:29.977 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:30.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:30.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:30.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:28.456109+0000 mgr.y (mgr.14544) 444 : cluster [DBG] pgmap v740: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:28.639285+0000 osd.4 (osd.4) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:28.639551+0000 osd.4 (osd.4) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:29.469569+0000 mon.a (mon.0) 2718 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:29.469654+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e495: 8 total, 8 up, 4 in 2024-02-13T00:48:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:29.500459+0000 mon.b (mon.1) 621 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:29.500944+0000 mon.a (mon.0) 2720 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:29.501535+0000 mon.b (mon.1) 622 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-77"}]: dispatch 2024-02-13T00:48:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:29.501844+0000 mon.a (mon.0) 2721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-77"}]: dispatch 2024-02-13T00:48:30.104 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:30.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.109+0000 7f8502437700 1 -- 172.21.15.162:0/1725080853 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"} v 0) v1 -- 0x7f84fc06c600 con 0x7f84fc0f8500 2024-02-13T00:48:30.130 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:30.277 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:30.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:30.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:30.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:30.414 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:30.435 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:30.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:30.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.473+0000 7f84f9ffb700 1 -- 172.21.15.162:0/1725080853 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]=0 set pool 5 pg_num to 22 v496) v1 ==== 144+0+0 (secure 0 0 0) 0x7f84ec023030 con 0x7f84fc0f8500 2024-02-13T00:48:30.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.476+0000 7f8502437700 10 librados: watch_flush enter 2024-02-13T00:48:30.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.476+0000 7f8502437700 10 librados: watch_flush exit 2024-02-13T00:48:30.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.476+0000 7f8502437700 20 client.27543.objecter shutdown clearing up homeless session... 2024-02-13T00:48:30.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.476+0000 7f8502437700 10 client.27543.objecter successfully canceled tick 2024-02-13T00:48:30.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.476+0000 7f8502437700 1 -- 172.21.15.162:0/1725080853 >> v2:172.21.15.162:6800/2866534053 conn(0x7f84e8062c90 msgr2=0x7f84e8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:30.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.476+0000 7f8502437700 1 --2- 172.21.15.162:0/1725080853 >> v2:172.21.15.162:6800/2866534053 conn(0x7f84e8062c90 0x7f84e8065150 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 rx=0x7f84fc06c8c0 tx=0x7f84e4009d60).stop 2024-02-13T00:48:30.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.476+0000 7f8502437700 1 -- 172.21.15.162:0/1725080853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc0f8500 msgr2=0x7f84fc06b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:30.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.476+0000 7f8502437700 1 --2- 172.21.15.162:0/1725080853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc0f8500 0x7f84fc06b170 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f84ec009eb0 tx=0x7f84ec00b040).stop 2024-02-13T00:48:30.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.477+0000 7f8502437700 1 -- 172.21.15.162:0/1725080853 shutdown_connections 2024-02-13T00:48:30.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.477+0000 7f8502437700 1 --2- 172.21.15.162:0/1725080853 >> v2:172.21.15.162:6800/2866534053 conn(0x7f84e8062c90 0x7f84e8065150 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:30.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.477+0000 7f8502437700 1 --2- 172.21.15.162:0/1725080853 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f84fc0fa440 0x7f84fc06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:30.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.477+0000 7f8502437700 1 --2- 172.21.15.162:0/1725080853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc0f8500 0x7f84fc06b170 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:30.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.477+0000 7f8502437700 1 --2- 172.21.15.162:0/1725080853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84fc0f9b70 0x7f84fc06b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:30.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.477+0000 7f8502437700 1 -- 172.21.15.162:0/1725080853 >> 172.21.15.162:0/1725080853 conn(0x7f84fc0f38c0 msgr2=0x7f84fc0fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:30.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.488+0000 7f8502437700 1 -- 172.21.15.162:0/1725080853 shutdown_connections 2024-02-13T00:48:30.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.488+0000 7f8502437700 1 -- 172.21.15.162:0/1725080853 wait complete. 2024-02-13T00:48:30.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:30.488+0000 7f8502437700 1 librados: shutdown 2024-02-13T00:48:30.492 INFO:teuthology.orchestra.run.smithi162.stderr:set pool 5 pg_num to 22 2024-02-13T00:48:30.539 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:30.567 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:30.586 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:30.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:30.689 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:30.716 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:30.735 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:30.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:30.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:30.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:30.865 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:30.884 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:30.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:30.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:31.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:30.110543+0000 mon.a (mon.0) 2722 : audit [INF] from='client.? 172.21.15.162:0/1725080853' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-13T00:48:31.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:30.471853+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? 172.21.15.162:0/1725080853' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2024-02-13T00:48:31.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:30.471925+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e496: 8 total, 8 up, 4 in 2024-02-13T00:48:31.013 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:31.032 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:31.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:31.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:30.110543+0000 mon.a (mon.0) 2722 : audit [INF] from='client.? 172.21.15.162:0/1725080853' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-13T00:48:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:30.471853+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? 172.21.15.162:0/1725080853' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2024-02-13T00:48:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:30.471925+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e496: 8 total, 8 up, 4 in 2024-02-13T00:48:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:30.110543+0000 mon.a (mon.0) 2722 : audit [INF] from='client.? 172.21.15.162:0/1725080853' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2024-02-13T00:48:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:30.471853+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? 172.21.15.162:0/1725080853' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2024-02-13T00:48:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:30.471925+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e496: 8 total, 8 up, 4 in 2024-02-13T00:48:31.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:31.162 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:31.181 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:31.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:31.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:31.514 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:31.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:31.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:31.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:48:31.645 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:31.663 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:31.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:31.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:31.798 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:29.680354+0000 osd.0 (osd.0) 193 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:48:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:29.681410+0000 osd.0 (osd.0) 194 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:48:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:30.459620+0000 mgr.y (mgr.14544) 445 : cluster [DBG] pgmap v743: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 10 KiB/s rd, 15 KiB/s wr, 40 op/s 2024-02-13T00:48:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:31.460342+0000 osd.6 (osd.6) 145 : cluster [DBG] 6.c scrub starts 2024-02-13T00:48:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:31.463435+0000 osd.6 (osd.6) 146 : cluster [DBG] 6.c scrub ok 2024-02-13T00:48:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:31.472174+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e497: 8 total, 8 up, 4 in 2024-02-13T00:48:32.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:31.472927+0000 mon.b (mon.1) 623 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:32.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:31.473313+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:32.074 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:29.680354+0000 osd.0 (osd.0) 193 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:48:32.074 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:29.681410+0000 osd.0 (osd.0) 194 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:48:32.074 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:30.459620+0000 mgr.y (mgr.14544) 445 : cluster [DBG] pgmap v743: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 10 KiB/s rd, 15 KiB/s wr, 40 op/s 2024-02-13T00:48:32.074 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:31.460342+0000 osd.6 (osd.6) 145 : cluster [DBG] 6.c scrub starts 2024-02-13T00:48:32.074 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:31.463435+0000 osd.6 (osd.6) 146 : cluster [DBG] 6.c scrub ok 2024-02-13T00:48:32.074 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:31.472174+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e497: 8 total, 8 up, 4 in 2024-02-13T00:48:32.074 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:31.472927+0000 mon.b (mon.1) 623 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:32.074 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:31.473313+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:32.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:29.680354+0000 osd.0 (osd.0) 193 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:48:32.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:29.681410+0000 osd.0 (osd.0) 194 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:48:32.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:30.459620+0000 mgr.y (mgr.14544) 445 : cluster [DBG] pgmap v743: 238 pgs: 238 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 10 KiB/s rd, 15 KiB/s wr, 40 op/s 2024-02-13T00:48:32.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:31.460342+0000 osd.6 (osd.6) 145 : cluster [DBG] 6.c scrub starts 2024-02-13T00:48:32.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:31.463435+0000 osd.6 (osd.6) 146 : cluster [DBG] 6.c scrub ok 2024-02-13T00:48:32.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:31.472174+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e497: 8 total, 8 up, 4 in 2024-02-13T00:48:32.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:31.472927+0000 mon.b (mon.1) 623 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:32.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:31.473313+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:32.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:32.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:32.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:32.493 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:32.795 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:48:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:48:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:48:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24739.0:2 16.1a 16:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:40:57.475809+0000", 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 454.57419860599998, 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.081608487, 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24739", 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2737601038", 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:57.475809+0000", 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:57.475809+0000", 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:57.475807+0000", 2024-02-13T00:48:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6663999999999998e-05 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:57.475823+0000", 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3191e-05 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:57.475837+0000", 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3080000000000008e-06 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:57.475845+0000", 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7042351469999999 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.180080+0000", 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8939999999999993e-06 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.180088+0000", 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3713813859999999 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.551469+0000", 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8527999999999999e-05 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.551528+0000", 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3727999999999996e-05 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.551602+0000", 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000807076 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.552409+0000", 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048816069999999996 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557290+0000", 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4697000000000007e-05 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557365+0000", 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1501999999999998e-05 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557397+0000", 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0664000000000001e-05 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557417+0000", 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.081608487 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14862.0:2 12.1a 12:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:40:59.418169+0000", 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 452.63183830000003, 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1389136820000001, 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14862", 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4283548805", 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418169+0000", 2024-02-13T00:48:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418169+0000", 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418167+0000", 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0829999999999993e-06 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418176+0000", 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2118e-05 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418188+0000", 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.859e-06 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418198+0000", 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6501e-05 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418254+0000", 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4610000000000008e-06 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:40:59.418263+0000", 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.132368241 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.550631+0000", 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117495 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.550748+0000", 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128987 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.550877+0000", 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014619559999999999 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.552339+0000", 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003784141 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.556123+0000", 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087638699999999996 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557000+0000", 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3489999999999999e-05 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557043+0000", 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9342999999999999e-05 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.557083+0000", 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1389136820000001 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24739.0:4 16.16 16:681f27cd:::2.obj:head [write 0~105] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.697879+0000", 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 450.35212812399999, 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059591969000000002, 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24739", 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2737601038", 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 4 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697879+0000", 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697879+0000", 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697877+0000", 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5740000000000001e-06 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697884+0000", 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7660000000000004e-06 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697893+0000", 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5230000000000001e-06 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697899+0000", 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5326e-05 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697935+0000", 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7998999999999997e-05 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.698003+0000", 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.038e-05 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.698083+0000", 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053553109999999998 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703438+0000", 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.046529604000000002 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.749968+0000", 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007411946 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.757380+0000", 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8148e-05 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.757428+0000", 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3118000000000003e-05 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.757471+0000", 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059591969000000002 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14862.0:5 12.12 12:4a6d8757:::3.obj:head [write 0~115] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.736730+0000", 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 450.31327685000002, 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020875984, 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14862", 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4283548805", 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736730+0000", 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736730+0000", 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736729+0000", 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9850000000000003e-06 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736735+0000", 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5379999999999997e-06 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736743+0000", 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0959999999999997e-06 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736749+0000", 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010354899999999999 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736852+0000", 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118468 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.736971+0000", 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015531799999999999 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5", 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.737126+0000", 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043792969999999999 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.741505+0000", 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0087964570000000006 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.750302+0000", 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007230144 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.757532+0000", 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9459000000000003e-05 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.757572+0000", 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4918e-05 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.757606+0000", 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020875984 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24926.0:3 16.16 16:681f27cd:::2.obj:head [write 0~105] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.787450+0000", 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 449.26255688499998, 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0364658, 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24926", 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2404423687", 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787450+0000", 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787450+0000", 2024-02-13T00:48:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787448+0000", 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6799999999999999e-06 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787457+0000", 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0567999999999999e-05 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787468+0000", 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6150000000000002e-06 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787474+0000", 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0272999999999998e-05 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787525+0000", 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8916000000000001e-05 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787604+0000", 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012292999999999999 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.787726+0000", 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010954789999999999 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.788822+0000", 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0079658809999999993 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.796788+0000", 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027043886 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.823832+0000", 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6207e-05 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.823878+0000", 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8303999999999998e-05 2024-02-13T00:48:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.823916+0000", 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0364658 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14886.0:1809 20.14 e108/105 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head v 108'882, mlcod=108'880)", 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.004814+0000", 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 448.04519300499999, 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02470756, 2024-02-13T00:48:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.004814+0000", 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.004814+0000", 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.004814+0000", 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3890000000000004e-06 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.004819+0000", 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1920000000000002e-06 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.004826+0000", 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3960000000000003e-06 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.004832+0000", 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011069244000000001 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.015901+0000", 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1639000000000001e-05 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.015923+0000", 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013572752 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.029496+0000", 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6227000000000001e-05 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.029522+0000", 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02470756 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24770.0:1 33.7s0 33:f1dcbfa7:test-rados-api-smithi162-84113-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.018769+0000", 2024-02-13T00:48:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 448.03123798899998, 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0820363020000001, 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24770", 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2725867940", 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.018769+0000", 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.018769+0000", 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999909 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.018760+0000", 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7418000000000002e-05 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.018778+0000", 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5858e-05 2024-02-13T00:48:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.018793+0000", 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2320000000000001e-06 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.018802+0000", 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.516974135 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.535776+0000", 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2260000000000001e-06 2024-02-13T00:48:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.535779+0000", 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.52173712800000005 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.057516+0000", 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012353 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.057640+0000", 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000412645 2024-02-13T00:48:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.058052+0000", 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014702148999999999 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.072754+0000", 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027966924000000001 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.100721+0000", 2024-02-13T00:48:32.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4263000000000003e-05 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.100806+0000", 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0820363020000001 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14886.0:2059 20.6 e109/105 20:661c4854:test-rados-api-smithi162-84049-16::big:head v 109'1, mlcod=0'0)", 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.270780+0000", 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 447.77922758400001, 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059378140000000003, 2024-02-13T00:48:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.270780+0000", 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.270780+0000", 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.270777+0000", 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016126680000000001 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.286904+0000", 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3373000000000001e-05 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.286927+0000", 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4949999999999997e-06 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.286937+0000", 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1576999999999997e-05 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.286998+0000", 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1797000000000002e-05 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.287020+0000", 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043100139000000003 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.330120+0000", 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7494999999999999e-05 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.330158+0000", 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059378140000000003 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2061 20.17 20:e88892cc:test-rados-api-smithi162-84049-16::big.copy:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:48:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.334388+0000", 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 447.71561910499997, 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.071394319999999997, 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2061 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334388+0000", 2024-02-13T00:48:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334388+0000", 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334386+0000", 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1179999999999998e-06 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334395+0000", 2024-02-13T00:48:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3471e-05 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334408+0000", 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2110000000000001e-06 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334416+0000", 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5129999999999997e-05 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334451+0000", 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7012e-05 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.334538+0000", 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055863379999999997 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.390401+0000", 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010490476 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.400892+0000", 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000425387 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.401317+0000", 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004380353 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405697+0000", 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5701999999999998e-05 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405733+0000", 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9394999999999997e-05 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405783+0000", 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.071394319999999997 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2062 20.17 20:e88892cc:test-rados-api-smithi162-84049-16::big.copy:head [read 0~25165836] snapc 0=[] ondisk+read+known_if_redirected e109)", 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.405952+0000", 2024-02-13T00:48:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 447.64405578399999, 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015988108000000001, 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2062 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405952+0000", 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405952+0000", 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405950+0000", 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7400000000000001e-06 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405956+0000", 2024-02-13T00:48:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5199999999999997e-06 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405964+0000", 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6949999999999999e-06 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405969+0000", 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1183e-05 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.405990+0000", 2024-02-13T00:48:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7864e-05 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.406048+0000", 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015891445000000001 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.421940+0000", 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015988108000000001 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2067 20.1e 20:78e1aec5:test-rados-api-smithi162-84049-17::small:head [writefull 0~2800] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.582089+0000", 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 447.46791788500002, 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044450968, 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2067 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582089+0000", 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582089+0000", 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999962 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582086+0000", 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2827999999999999e-05 2024-02-13T00:48:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582109+0000", 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8418000000000001e-05 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582137+0000", 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4623999999999999e-05 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582152+0000", 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7082999999999995e-05 2024-02-13T00:48:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582219+0000", 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012813000000000001 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582347+0000", 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000140655 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,7", 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582487+0000", 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0084849650000000006 2024-02-13T00:48:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.590972+0000", 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034854675000000002 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.625827+0000", 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00063645100000000001 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.626464+0000", 2024-02-13T00:48:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4910000000000003e-05 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.626498+0000", 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1952000000000001e-05 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.626540+0000", 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044450968 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25009.0:2 37.6 37:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e110)", 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:05.024150+0000", 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 447.02585772100002, 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.087036812000000005, 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25009", 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/508499360", 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.024150+0000", 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.024150+0000", 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.024148+0000", 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.06e-05 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.024158+0000", 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033910860000000002 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.027549+0000", 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0696999999999999e-05 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.027560+0000", 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014665497 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.042225+0000", 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0219999999999999e-06 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.042232+0000", 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059524986000000002 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.101757+0000", 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116793 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.101874+0000", 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014743800000000001 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,2", 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.102022+0000", 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051785600000000002 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.102539+0000", 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0082938850000000008 2024-02-13T00:48:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.110833+0000", 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028565099999999999 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.111119+0000", 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8586e-05 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.111148+0000", 2024-02-13T00:48:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8788999999999997e-05 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:05.111186+0000", 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.087036812000000005 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "pg_update_log_missing(69.15 epoch 127/126 rep_tid 352 entries 127'3 (0'0) error 69:ad551702:::test:head by client.27745.0:6 0.000000 -4096 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 127'1)", 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:21.888669+0000", 2024-02-13T00:48:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 430.16133842800002, 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019065159000000002, 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888669+0000", 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888669+0000", 2024-02-13T00:48:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888668+0000", 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2411e-05 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888680+0000", 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8651000000000002e-05 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888699+0000", 2024-02-13T00:48:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8210000000000008e-06 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888708+0000", 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8349999999999994e-05 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.888786+0000", 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018947997000000001 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.907734+0000", 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019065159000000002 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.31189.0:2 75.1f 75:fd8eaea7:nspace::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:27.094388+0000", 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 424.955619291, 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.114165541, 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.868 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.31189", 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3145686446", 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094388+0000", 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094388+0000", 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:48:32.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094386+0000", 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8520000000000004e-06 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094394+0000", 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3560999999999999e-05 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094407+0000", 2024-02-13T00:48:32.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9789999999999993e-06 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094415+0000", 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0484e-05 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094466+0000", 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6290000000000004e-06 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094473+0000", 2024-02-13T00:48:32.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.083557333999999997 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.178030+0000", 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6599999999999995e-06 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.178038+0000", 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017103554 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.195141+0000", 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121271 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.195262+0000", 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014633300000000001 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,6", 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.195409+0000", 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013076450000000001 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.196716+0000", 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6741000000000001e-05 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.196733+0000", 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011723007000000001 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.208456+0000", 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5758999999999999e-05 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.208512+0000", 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1690000000000002e-05 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.208554+0000", 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.114165541 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.875 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDECSubOpWrite(78.4s1 134/133 ECSubWrite(tid=163, reqid=client.19047.0:1, at_version=0'0, trim_to=0'0, roll_forward_to=134'1))", 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:29.299708+0000", 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 422.75029922300001, 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024244241, 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.299708+0000", 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.299708+0000", 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.299707+0000", 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0579999999999999e-05 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.299727+0000", 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8301999999999999e-05 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.299745+0000", 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7370000000000006e-06 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.299753+0000", 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050936760000000001 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.304847+0000", 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.222e-05 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.304859+0000", 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019080481 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.323940+0000", 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2816000000000001e-05 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:29.323952+0000", 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024244241 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.5.0:515 61.8 e190/121 61:1196da48:test-rados-api-smithi162-84057-14::foo4:7 v 190'33, mlcod=188'29)", 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:25.341891+0000", 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 366.70811621899998, 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023575114000000001, 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:25.341891+0000", 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:25.341891+0000", 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:25.341890+0000", 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.754e-06 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:25.341897+0000", 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014015641000000001 2024-02-13T00:48:32.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:25.355913+0000", 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4300000000000006e-06 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:25.355921+0000", 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045060500000000002 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:25.360427+0000", 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8479e-05 2024-02-13T00:48:32.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:25.360446+0000", 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049940870000000004 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:25.365440+0000", 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.635e-05 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:25.365466+0000", 2024-02-13T00:48:32.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023575114000000001 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25162.0:1193 95.16 95:6ba2e39c:test-rados-api-smithi162-84057-14::foo79:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e191)", 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:26.393231+0000", 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 365.65677661000001, 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14628944699999999, 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25162", 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:48:32.884 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1193 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.393231+0000", 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.393231+0000", 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.393230+0000", 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720000000000003e-06 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.393235+0000", 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6970000000000005e-06 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.393244+0000", 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7929999999999997e-06 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.393249+0000", 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4258000000000002e-05 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.393293+0000", 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9697999999999994e-05 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.393383+0000", 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075393699999999999 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.394137+0000", 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6047999999999998e-05 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.394173+0000", 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14501647500000001 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.539189+0000", 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013342200000000001 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.539323+0000", 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000197269 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:26.539520+0000", 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14628944699999999 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.43843.0:1 130.6s0 130:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e192)", 2024-02-13T00:48:32.889 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:27.379907+0000", 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 364.67010012100002, 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0084759350000001, 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.43843", 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/977122138", 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.379907+0000", 2024-02-13T00:48:32.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.379907+0000", 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.379905+0000", 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6500000000000001e-06 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.379911+0000", 2024-02-13T00:48:32.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1198e-05 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.379922+0000", 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.878e-06 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.379929+0000", 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5877999999999995e-05 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.379995+0000", 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9489999999999999e-06 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:27.379997+0000", 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0018657550000001 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.381863+0000", 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000123087 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.381986+0000", 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000431512 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.382417+0000", 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055855330000000002 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.388003+0000", 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0426999999999997e-05 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.388053+0000", 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000329859 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:28.388383+0000", 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0084759350000001 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36773.0:2 141.6 141:67140490:::oid_RacingRemovePP_:head [delete] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:42.343006+0000", 2024-02-13T00:48:32.895 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 349.70700162700001, 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01921966, 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36773", 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3728040861", 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343006+0000", 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343006+0000", 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343004+0000", 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6169999999999998e-06 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343010+0000", 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0480000000000004e-06 2024-02-13T00:48:32.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343019+0000", 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6659999999999996e-06 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343025+0000", 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6413999999999999e-05 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343081+0000", 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8459999999999999e-06 2024-02-13T00:48:32.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343086+0000", 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001460346 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.344546+0000", 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1780000000000002e-06 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.344551+0000", 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014141512 2024-02-13T00:48:32.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.358693+0000", 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013063800000000001 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.358823+0000", 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034018970000000001 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.362225+0000", 2024-02-13T00:48:32.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01921966 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36773.0:3 141.6 141:67140490:::oid_RacingRemovePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e207)", 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:42.343042+0000", 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 349.70696532699998, 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019424446000000001, 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36773", 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3728040861", 2024-02-13T00:48:32.901 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343042+0000", 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343042+0000", 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:48:32.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343041+0000", 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4829999999999999e-06 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343045+0000", 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4410000000000001e-06 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343050+0000", 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.033e-06 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:48:32.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343053+0000", 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6760000000000004e-05 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343130+0000", 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7590000000000003e-06 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting_for_map not empty", 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.343136+0000", 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001427635 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.344564+0000", 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2280000000000001e-06 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.344566+0000", 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014486634999999999 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.359052+0000", 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4764999999999994e-05 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.359117+0000", 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147742 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5", 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.359265+0000", 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012793780000000001 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.360544+0000", 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00046092399999999998 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.361005+0000", 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014071559999999999 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.362412+0000", 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7472000000000001e-05 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.362440+0000", 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6619999999999999e-05 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:48:32.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:48:32.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.362466+0000", 2024-02-13T00:48:32.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019424446000000001 2024-02-13T00:48:32.908 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.908 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.908 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.908 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:48:32.908 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:48:32.908 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:48:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:31.778781+0000 osd.1 (osd.1) 213 : cluster [DBG] 4.d scrub starts 2024-02-13T00:48:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:31.782724+0000 osd.1 (osd.1) 214 : cluster [DBG] 4.d scrub ok 2024-02-13T00:48:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:32.438158+0000 osd.6 (osd.6) 147 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:48:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:32.441163+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:48:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:32.460362+0000 mon.c (mon.2) 405 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-13T00:48:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:32.461262+0000 mon.a (mon.0) 2727 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-13T00:48:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:32.474875+0000 mon.a (mon.0) 2728 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:48:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:32.475002+0000 mon.a (mon.0) 2729 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-13T00:48:33.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:32.475063+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e498: 8 total, 8 up, 4 in 2024-02-13T00:48:33.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:32.479071+0000 mon.b (mon.1) 624 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:48:33.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:32.489040+0000 mon.b (mon.1) 625 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:33.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:32.491575+0000 mon.a (mon.0) 2731 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:33.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:31.778781+0000 osd.1 (osd.1) 213 : cluster [DBG] 4.d scrub starts 2024-02-13T00:48:33.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:31.782724+0000 osd.1 (osd.1) 214 : cluster [DBG] 4.d scrub ok 2024-02-13T00:48:33.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:32.438158+0000 osd.6 (osd.6) 147 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:48:33.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:32.441163+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:48:33.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:32.460362+0000 mon.c (mon.2) 405 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-13T00:48:33.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:32.461262+0000 mon.a (mon.0) 2727 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-13T00:48:33.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:32.474875+0000 mon.a (mon.0) 2728 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:48:33.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:32.475002+0000 mon.a (mon.0) 2729 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-13T00:48:33.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:32.475063+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e498: 8 total, 8 up, 4 in 2024-02-13T00:48:33.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:32.479071+0000 mon.b (mon.1) 624 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:48:33.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:32.489040+0000 mon.b (mon.1) 625 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:33.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:32.491575+0000 mon.a (mon.0) 2731 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:33.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:31.778781+0000 osd.1 (osd.1) 213 : cluster [DBG] 4.d scrub starts 2024-02-13T00:48:33.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:31.782724+0000 osd.1 (osd.1) 214 : cluster [DBG] 4.d scrub ok 2024-02-13T00:48:33.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:32.438158+0000 osd.6 (osd.6) 147 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:48:33.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:32.441163+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:48:33.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:32.460362+0000 mon.c (mon.2) 405 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-13T00:48:33.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:32.461262+0000 mon.a (mon.0) 2727 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-13T00:48:33.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:32.474875+0000 mon.a (mon.0) 2728 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:48:33.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:32.475002+0000 mon.a (mon.0) 2729 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-13T00:48:33.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:32.475063+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e498: 8 total, 8 up, 4 in 2024-02-13T00:48:33.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:32.479071+0000 mon.b (mon.1) 624 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:48:33.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:32.489040+0000 mon.b (mon.1) 625 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:33.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:32.491575+0000 mon.a (mon.0) 2731 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:33.645 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:33.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:33.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:33.749 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:33.781 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:33.803 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:33.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:32.460021+0000 mgr.y (mgr.14544) 446 : cluster [DBG] pgmap v746: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 39 op/s 2024-02-13T00:48:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:32.665893+0000 osd.0 (osd.0) 195 : cluster [DBG] 5.3 scrub starts 2024-02-13T00:48:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:32.669402+0000 osd.0 (osd.0) 196 : cluster [DBG] 5.3 scrub ok 2024-02-13T00:48:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:32.790697+0000 osd.1 (osd.1) 215 : cluster [DBG] 5.c deep-scrub starts 2024-02-13T00:48:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:32.793629+0000 osd.1 (osd.1) 216 : cluster [DBG] 5.c deep-scrub ok 2024-02-13T00:48:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:33.456761+0000 osd.6 (osd.6) 149 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:48:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:33.460239+0000 osd.6 (osd.6) 150 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:48:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:33.479005+0000 mon.a (mon.0) 2732 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:48:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:33.479096+0000 mon.a (mon.0) 2733 : cluster [DBG] osdmap e499: 8 total, 8 up, 4 in 2024-02-13T00:48:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:33.480662+0000 mon.b (mon.1) 626 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:33.481153+0000 mon.a (mon.0) 2734 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:33.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:32.460021+0000 mgr.y (mgr.14544) 446 : cluster [DBG] pgmap v746: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 39 op/s 2024-02-13T00:48:33.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:32.665893+0000 osd.0 (osd.0) 195 : cluster [DBG] 5.3 scrub starts 2024-02-13T00:48:33.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:32.669402+0000 osd.0 (osd.0) 196 : cluster [DBG] 5.3 scrub ok 2024-02-13T00:48:33.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:32.790697+0000 osd.1 (osd.1) 215 : cluster [DBG] 5.c deep-scrub starts 2024-02-13T00:48:33.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:32.793629+0000 osd.1 (osd.1) 216 : cluster [DBG] 5.c deep-scrub ok 2024-02-13T00:48:33.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:33.456761+0000 osd.6 (osd.6) 149 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:48:33.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:33.460239+0000 osd.6 (osd.6) 150 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:48:33.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:33.479005+0000 mon.a (mon.0) 2732 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:48:33.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:33.479096+0000 mon.a (mon.0) 2733 : cluster [DBG] osdmap e499: 8 total, 8 up, 4 in 2024-02-13T00:48:33.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:33.480662+0000 mon.b (mon.1) 626 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:33.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:33.481153+0000 mon.a (mon.0) 2734 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:33.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:33.937 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:33.957 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:33.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:32.460021+0000 mgr.y (mgr.14544) 446 : cluster [DBG] pgmap v746: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 39 op/s 2024-02-13T00:48:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:32.665893+0000 osd.0 (osd.0) 195 : cluster [DBG] 5.3 scrub starts 2024-02-13T00:48:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:32.669402+0000 osd.0 (osd.0) 196 : cluster [DBG] 5.3 scrub ok 2024-02-13T00:48:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:32.790697+0000 osd.1 (osd.1) 215 : cluster [DBG] 5.c deep-scrub starts 2024-02-13T00:48:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:32.793629+0000 osd.1 (osd.1) 216 : cluster [DBG] 5.c deep-scrub ok 2024-02-13T00:48:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:33.456761+0000 osd.6 (osd.6) 149 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:48:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:33.460239+0000 osd.6 (osd.6) 150 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:48:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:33.479005+0000 mon.a (mon.0) 2732 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:48:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:33.479096+0000 mon.a (mon.0) 2733 : cluster [DBG] osdmap e499: 8 total, 8 up, 4 in 2024-02-13T00:48:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:33.480662+0000 mon.b (mon.1) 626 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:33.481153+0000 mon.a (mon.0) 2734 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:34.061 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:34.091 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:34.752 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:34.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.765+0000 7f4d67f78700 1 -- 172.21.15.162:0/2667860555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d600fcb30 msgr2=0x7f4d600fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:34.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.765+0000 7f4d67f78700 1 --2- 172.21.15.162:0/2667860555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d600fcb30 0x7f4d600fcf10 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7f4d5c004870 tx=0x7f4d5c009cb0).stop 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.765+0000 7f4d67f78700 1 -- 172.21.15.162:0/2667860555 shutdown_connections 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.765+0000 7f4d67f78700 1 --2- 172.21.15.162:0/2667860555 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d600f73a0 0x7f4d601039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.765+0000 7f4d67f78700 1 --2- 172.21.15.162:0/2667860555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d600fcb30 0x7f4d600fcf10 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.765+0000 7f4d67f78700 1 --2- 172.21.15.162:0/2667860555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d600fd490 0x7f4d600f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.765+0000 7f4d67f78700 1 -- 172.21.15.162:0/2667860555 >> 172.21.15.162:0/2667860555 conn(0x7f4d6006eb90 msgr2=0x7f4d600f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.765+0000 7f4d67f78700 1 -- 172.21.15.162:0/2667860555 shutdown_connections 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 -- 172.21.15.162:0/2667860555 wait complete. 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 librados: starting msgr at 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 librados: starting objecter 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 Processor -- start 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 -- start start 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 librados: setting wanted keys 2024-02-13T00:48:34.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 librados: calling monclient init 2024-02-13T00:48:34.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d600f73a0 0x7f4d601042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:34.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d600fcb30 0x7f4d601047c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:34.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d600fd490 0x7f4d60104cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:34.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60116690 con 0x7f4d600fcb30 2024-02-13T00:48:34.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60116800 con 0x7f4d600f73a0 2024-02-13T00:48:34.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.766+0000 7f4d67f78700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d601169c0 con 0x7f4d600fd490 2024-02-13T00:48:34.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d65513700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d600fcb30 0x7f4d601047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:34.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d65513700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d600fcb30 0x7f4d601047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:47346/0 (socket says 172.21.15.162:47346) 2024-02-13T00:48:34.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d65513700 1 -- 172.21.15.162:0/3695786122 learned_addr learned my addr 172.21.15.162:0/3695786122 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:34.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d66515700 1 --2- 172.21.15.162:0/3695786122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d600fd490 0x7f4d60104cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:34.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d65d14700 1 --2- 172.21.15.162:0/3695786122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d600f73a0 0x7f4d601042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:34.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d65513700 1 -- 172.21.15.162:0/3695786122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d600fd490 msgr2=0x7f4d60104cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d65513700 1 --2- 172.21.15.162:0/3695786122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d600fd490 0x7f4d60104cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d65513700 1 -- 172.21.15.162:0/3695786122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d600f73a0 msgr2=0x7f4d601042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d65513700 1 --2- 172.21.15.162:0/3695786122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d600f73a0 0x7f4d601042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d65513700 1 -- 172.21.15.162:0/3695786122 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d580097c0 con 0x7f4d600fcb30 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d65d14700 1 --2- 172.21.15.162:0/3695786122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d600f73a0 0x7f4d601042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d65513700 1 --2- 172.21.15.162:0/3695786122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d600fcb30 0x7f4d601047c0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f4d5000e970 tx=0x7f4d5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d56ffd700 10 client.?.objecter ms_handle_connect 0x7f4d600fcb30 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d56ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d56ffd700 1 -- 172.21.15.162:0/3695786122 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50004660 con 0x7f4d600fcb30 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d67f78700 1 -- 172.21.15.162:0/3695786122 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d5c005040 con 0x7f4d600fcb30 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d56ffd700 1 -- 172.21.15.162:0/3695786122 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d5000f040 con 0x7f4d600fcb30 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d56ffd700 1 -- 172.21.15.162:0/3695786122 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50014790 con 0x7f4d600fcb30 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d67f78700 10 client.27549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d67f78700 1 -- 172.21.15.162:0/3695786122 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d60117050 con 0x7f4d600fcb30 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.767+0000 7f4d67f78700 1 librados: init done 2024-02-13T00:48:34.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.768+0000 7f4d67f78700 1 -- 172.21.15.162:0/3695786122 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d60107190 con 0x7f4d600fcb30 2024-02-13T00:48:34.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.769+0000 7f4d56ffd700 1 -- 172.21.15.162:0/3695786122 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4d500090d0 con 0x7f4d600fcb30 2024-02-13T00:48:34.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.769+0000 7f4d56ffd700 1 --2- 172.21.15.162:0/3695786122 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4d4c062c90 0x7f4d4c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:34.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.769+0000 7f4d56ffd700 1 -- 172.21.15.162:0/3695786122 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(500..500 src has 52..500) v4 ==== 8115+0+0 (secure 0 0 0) 0x7f4d5007cd50 con 0x7f4d600fcb30 2024-02-13T00:48:34.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.769+0000 7f4d56ffd700 10 client.27549.objecter ms_dispatch 0x7f4d60001550 osd_map(500..500 src has 52..500) v4 2024-02-13T00:48:34.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.769+0000 7f4d56ffd700 3 client.27549.objecter handle_osd_map got epochs [500,500] > 0 2024-02-13T00:48:34.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.769+0000 7f4d56ffd700 3 client.27549.objecter handle_osd_map decoding full epoch 500 2024-02-13T00:48:34.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.770+0000 7f4d56ffd700 20 client.27549.objecter dump_active .. 0 homeless 2024-02-13T00:48:34.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.770+0000 7f4d65d14700 1 --2- 172.21.15.162:0/3695786122 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4d4c062c90 0x7f4d4c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:34.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.770+0000 7f4d56ffd700 10 client.27549.objecter ms_handle_connect 0x7f4d4c062c90 2024-02-13T00:48:34.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.770+0000 7f4d65d14700 1 --2- 172.21.15.162:0/3695786122 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4d4c062c90 0x7f4d4c065150 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 rx=0x7f4d5c004ee0 tx=0x7f4d5c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:34.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.775+0000 7f4d56ffd700 1 -- 172.21.15.162:0/3695786122 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4d5004af60 con 0x7f4d600fcb30 2024-02-13T00:48:34.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:34.959+0000 7f4d67f78700 1 -- 172.21.15.162:0/3695786122 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f4d601057b0 con 0x7f4d600fcb30 2024-02-13T00:48:35.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-13T00:48:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.460+0000 7f4d56ffd700 1 -- 172.21.15.162:0/3695786122 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v501) v1 ==== 87+0+0 (secure 0 0 0) 0x7f4d50019070 con 0x7f4d600fcb30 2024-02-13T00:48:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.463+0000 7f4d67f78700 10 librados: watch_flush enter 2024-02-13T00:48:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.463+0000 7f4d67f78700 10 librados: watch_flush exit 2024-02-13T00:48:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.463+0000 7f4d67f78700 20 client.27549.objecter shutdown clearing up homeless session... 2024-02-13T00:48:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.463+0000 7f4d67f78700 10 client.27549.objecter successfully canceled tick 2024-02-13T00:48:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 -- 172.21.15.162:0/3695786122 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4d4c062c90 msgr2=0x7f4d4c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 --2- 172.21.15.162:0/3695786122 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4d4c062c90 0x7f4d4c065150 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 rx=0x7f4d5c004ee0 tx=0x7f4d5c00b040).stop 2024-02-13T00:48:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 -- 172.21.15.162:0/3695786122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d600fcb30 msgr2=0x7f4d601047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 --2- 172.21.15.162:0/3695786122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d600fcb30 0x7f4d601047c0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f4d5000e970 tx=0x7f4d5000b040).stop 2024-02-13T00:48:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 -- 172.21.15.162:0/3695786122 shutdown_connections 2024-02-13T00:48:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 --2- 172.21.15.162:0/3695786122 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4d4c062c90 0x7f4d4c065150 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 --2- 172.21.15.162:0/3695786122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d600fd490 0x7f4d60104cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:35.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 --2- 172.21.15.162:0/3695786122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d600f73a0 0x7f4d601042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:35.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 --2- 172.21.15.162:0/3695786122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d600fcb30 0x7f4d601047c0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:35.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 -- 172.21.15.162:0/3695786122 >> 172.21.15.162:0/3695786122 conn(0x7f4d6006eb90 msgr2=0x7f4d600fa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:35.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 -- 172.21.15.162:0/3695786122 shutdown_connections 2024-02-13T00:48:35.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 -- 172.21.15.162:0/3695786122 wait complete. 2024-02-13T00:48:35.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:35.464+0000 7f4d67f78700 1 librados: shutdown 2024-02-13T00:48:35.469 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:48:35.560 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:34.338864+0000 mon.a (mon.0) 2735 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:48:35.560 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:34.454653+0000 osd.6 (osd.6) 151 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:48:35.560 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:34.458258+0000 osd.6 (osd.6) 152 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:48:35.560 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:34.460395+0000 mgr.y (mgr.14544) 447 : cluster [DBG] pgmap v749: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:48:35.560 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:34.460734+0000 mon.c (mon.2) 406 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-13T00:48:35.560 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:34.461599+0000 mon.a (mon.0) 2736 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-13T00:48:35.560 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:34.478405+0000 mon.a (mon.0) 2737 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:48:35.560 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:34.478480+0000 mon.a (mon.0) 2738 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-13T00:48:35.560 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:34.478539+0000 mon.a (mon.0) 2739 : cluster [DBG] osdmap e500: 8 total, 8 up, 4 in 2024-02-13T00:48:35.560 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:34.480139+0000 mon.b (mon.1) 627 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:35.561 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:34.480832+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:35.561 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:34.960688+0000 mon.a (mon.0) 2741 : audit [INF] from='client.? 172.21.15.162:0/3695786122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:48:35.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:34.338864+0000 mon.a (mon.0) 2735 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:48:35.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:34.454653+0000 osd.6 (osd.6) 151 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:48:35.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:34.458258+0000 osd.6 (osd.6) 152 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:48:35.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:34.460395+0000 mgr.y (mgr.14544) 447 : cluster [DBG] pgmap v749: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:48:35.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:34.460734+0000 mon.c (mon.2) 406 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-13T00:48:35.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:34.461599+0000 mon.a (mon.0) 2736 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-13T00:48:35.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:34.478405+0000 mon.a (mon.0) 2737 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:48:35.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:34.478480+0000 mon.a (mon.0) 2738 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-13T00:48:35.561 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:34.478539+0000 mon.a (mon.0) 2739 : cluster [DBG] osdmap e500: 8 total, 8 up, 4 in 2024-02-13T00:48:35.562 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:34.480139+0000 mon.b (mon.1) 627 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:35.562 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:34.480832+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:35.562 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:34.960688+0000 mon.a (mon.0) 2741 : audit [INF] from='client.? 172.21.15.162:0/3695786122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:48:35.589 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:48:35.712 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:34.338864+0000 mon.a (mon.0) 2735 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:48:35.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:34.454653+0000 osd.6 (osd.6) 151 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:48:35.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:34.458258+0000 osd.6 (osd.6) 152 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:48:35.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:34.460395+0000 mgr.y (mgr.14544) 447 : cluster [DBG] pgmap v749: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:48:35.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:34.460734+0000 mon.c (mon.2) 406 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-13T00:48:35.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:34.461599+0000 mon.a (mon.0) 2736 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-13T00:48:35.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:34.478405+0000 mon.a (mon.0) 2737 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:48:35.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:34.478480+0000 mon.a (mon.0) 2738 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-13T00:48:35.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:34.478539+0000 mon.a (mon.0) 2739 : cluster [DBG] osdmap e500: 8 total, 8 up, 4 in 2024-02-13T00:48:35.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:34.480139+0000 mon.b (mon.1) 627 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:35.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:34.480832+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:35.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:34.960688+0000 mon.a (mon.0) 2741 : audit [INF] from='client.? 172.21.15.162:0/3695786122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:48:36.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:48:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:48:36.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:35.431750+0000 osd.6 (osd.6) 153 : cluster [DBG] 28.1b deep-scrub starts 2024-02-13T00:48:36.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:35.434641+0000 osd.6 (osd.6) 154 : cluster [DBG] 28.1b deep-scrub ok 2024-02-13T00:48:36.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:35.456386+0000 mon.a (mon.0) 2742 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:36.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:35.461119+0000 mon.a (mon.0) 2743 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:48:36.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:35.461231+0000 mon.a (mon.0) 2744 : audit [INF] from='client.? 172.21.15.162:0/3695786122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:48:36.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:35.461287+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e501: 8 total, 8 up, 4 in 2024-02-13T00:48:36.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:35.463543+0000 mon.b (mon.1) 628 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:36.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:35.473275+0000 mon.a (mon.0) 2746 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:36.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:35.709540+0000 mgr.y (mgr.14544) 448 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:36.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:48:36] "GET /metrics HTTP/1.1" 200 188112 "" "Prometheus/2.33.4" 2024-02-13T00:48:36.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:35.431750+0000 osd.6 (osd.6) 153 : cluster [DBG] 28.1b deep-scrub starts 2024-02-13T00:48:36.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:35.434641+0000 osd.6 (osd.6) 154 : cluster [DBG] 28.1b deep-scrub ok 2024-02-13T00:48:36.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:35.456386+0000 mon.a (mon.0) 2742 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:36.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:35.461119+0000 mon.a (mon.0) 2743 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:48:36.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:35.461231+0000 mon.a (mon.0) 2744 : audit [INF] from='client.? 172.21.15.162:0/3695786122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:48:36.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:35.461287+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e501: 8 total, 8 up, 4 in 2024-02-13T00:48:36.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:35.463543+0000 mon.b (mon.1) 628 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:36.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:35.473275+0000 mon.a (mon.0) 2746 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:36.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:35.709540+0000 mgr.y (mgr.14544) 448 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:36.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:35.431750+0000 osd.6 (osd.6) 153 : cluster [DBG] 28.1b deep-scrub starts 2024-02-13T00:48:36.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:35.434641+0000 osd.6 (osd.6) 154 : cluster [DBG] 28.1b deep-scrub ok 2024-02-13T00:48:36.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:35.456386+0000 mon.a (mon.0) 2742 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:36.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:35.461119+0000 mon.a (mon.0) 2743 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:48:36.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:35.461231+0000 mon.a (mon.0) 2744 : audit [INF] from='client.? 172.21.15.162:0/3695786122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:48:36.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:35.461287+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e501: 8 total, 8 up, 4 in 2024-02-13T00:48:36.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:35.463543+0000 mon.b (mon.1) 628 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:36.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:35.473275+0000 mon.a (mon.0) 2746 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:36.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:35.709540+0000 mgr.y (mgr.14544) 448 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:36.770 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:36.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:36.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:36.903 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:37.030 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:37.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:37.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:37.161 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:37.235 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:37.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:37.339 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:37.357 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:48:37.357 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:48:37.357 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:48:37.357 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:48:37.368 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:37.388 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:37.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:37.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:37.519 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:36.461140+0000 mgr.y (mgr.14544) 449 : cluster [DBG] pgmap v752: 238 pgs: 2 peering, 236 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:37.519 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:36.470056+0000 mon.a (mon.0) 2747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:37.519 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:36.470136+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e502: 8 total, 8 up, 4 in 2024-02-13T00:48:37.519 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:36.461140+0000 mgr.y (mgr.14544) 449 : cluster [DBG] pgmap v752: 238 pgs: 2 peering, 236 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:37.519 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:36.470056+0000 mon.a (mon.0) 2747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:37.520 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:36.470136+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e502: 8 total, 8 up, 4 in 2024-02-13T00:48:37.522 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:37.543 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:37.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:37.646 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:37.672 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:36.461140+0000 mgr.y (mgr.14544) 449 : cluster [DBG] pgmap v752: 238 pgs: 2 peering, 236 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:37.672 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:36.470056+0000 mon.a (mon.0) 2747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-79","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:37.672 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:36.470136+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e502: 8 total, 8 up, 4 in 2024-02-13T00:48:37.673 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:37.711 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:37.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:37.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.774+0000 7facd9d83700 1 -- 172.21.15.162:0/3573802874 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7facd40f9540 msgr2=0x7facd4101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.774+0000 7facd9d83700 1 --2- 172.21.15.162:0/3573802874 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7facd40f9540 0x7facd4101ce0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 rx=0x7facc80047b0 tx=0x7facc8009cb0).stop 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.775+0000 7facd9d83700 1 -- 172.21.15.162:0/3573802874 shutdown_connections 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.775+0000 7facd9d83700 1 --2- 172.21.15.162:0/3573802874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facd41021d0 0x7facd41045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.775+0000 7facd9d83700 1 --2- 172.21.15.162:0/3573802874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facd40f8c70 0x7facd40f9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.775+0000 7facd9d83700 1 --2- 172.21.15.162:0/3573802874 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7facd40f9540 0x7facd4101ce0 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.775+0000 7facd9d83700 1 -- 172.21.15.162:0/3573802874 >> 172.21.15.162:0/3573802874 conn(0x7facd40f2740 msgr2=0x7facd40f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.775+0000 7facd9d83700 1 -- 172.21.15.162:0/3573802874 shutdown_connections 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.775+0000 7facd9d83700 1 -- 172.21.15.162:0/3573802874 wait complete. 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.775+0000 7facd9d83700 1 librados: starting msgr at 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.775+0000 7facd9d83700 1 librados: starting objecter 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.775+0000 7facd9d83700 1 Processor -- start 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd9d83700 1 -- start start 2024-02-13T00:48:37.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd9d83700 1 librados: setting wanted keys 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd9d83700 1 librados: calling monclient init 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd9d83700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7facd40f8c70 0x7facd410e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd9d83700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facd40f9540 0x7facd410e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd9d83700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facd41021d0 0x7facd4108380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd9d83700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd4108870 con 0x7facd41021d0 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd9d83700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd41089e0 con 0x7facd40f9540 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd9d83700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd4108ba0 con 0x7facd40f8c70 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd37fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7facd40f8c70 0x7facd410e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd37fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7facd40f8c70 0x7facd410e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:53770/0 (socket says 172.21.15.162:53770) 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd3fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facd41021d0 0x7facd4108380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd37fe700 1 -- 172.21.15.162:0/2092750269 learned_addr learned my addr 172.21.15.162:0/2092750269 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd2ffd700 1 --2- 172.21.15.162:0/2092750269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facd40f9540 0x7facd410e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:37.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd3fff700 1 -- 172.21.15.162:0/2092750269 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7facd40f8c70 msgr2=0x7facd410e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd3fff700 1 --2- 172.21.15.162:0/2092750269 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7facd40f8c70 0x7facd410e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd3fff700 1 -- 172.21.15.162:0/2092750269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facd40f9540 msgr2=0x7facd410e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd3fff700 1 --2- 172.21.15.162:0/2092750269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facd40f9540 0x7facd410e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd3fff700 1 -- 172.21.15.162:0/2092750269 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0097c0 con 0x7facd41021d0 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd2ffd700 1 --2- 172.21.15.162:0/2092750269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facd40f9540 0x7facd410e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.776+0000 7facd37fe700 1 --2- 172.21.15.162:0/2092750269 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7facd40f8c70 0x7facd410e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.777+0000 7facd3fff700 1 --2- 172.21.15.162:0/2092750269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facd41021d0 0x7facd4108380 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 rx=0x7faccc001dc0 tx=0x7faccc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.777+0000 7facd0ff9700 10 client.?.objecter ms_handle_connect 0x7facd41021d0 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.777+0000 7facd0ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.777+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc0099f0 con 0x7facd41021d0 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.777+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc8005040 con 0x7facd41021d0 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.777+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faccc0095a0 con 0x7facd41021d0 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.777+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc007cb0 con 0x7facd41021d0 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.777+0000 7facd9d83700 10 client.27555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:37.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.777+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd4115010 con 0x7facd41021d0 2024-02-13T00:48:37.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.777+0000 7facd9d83700 1 librados: init done 2024-02-13T00:48:37.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.777+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7facd40607d0 con 0x7facd41021d0 2024-02-13T00:48:37.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.778+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7faccc00fb80 con 0x7facd41021d0 2024-02-13T00:48:37.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.778+0000 7facd0ff9700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6800/2866534053 conn(0x7facbc062c90 0x7facbc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:37.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.778+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(503..503 src has 52..503) v4 ==== 8150+0+0 (secure 0 0 0) 0x7faccc07dac0 con 0x7facd41021d0 2024-02-13T00:48:37.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.778+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 osd_map(503..503 src has 52..503) v4 2024-02-13T00:48:37.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.778+0000 7facd0ff9700 3 client.27555.objecter handle_osd_map got epochs [503,503] > 0 2024-02-13T00:48:37.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.778+0000 7facd0ff9700 3 client.27555.objecter handle_osd_map decoding full epoch 503 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.778+0000 7facd37fe700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6800/2866534053 conn(0x7facbc062c90 0x7facbc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd0ff9700 20 client.27555.objecter dump_active .. 0 homeless 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v503) v1 ==== 54+0+16 (secure 0 0 0) 0x7faccc07de60 con 0x7facd41021d0 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd0ff9700 10 client.27555.objecter ms_handle_connect 0x7facbc062c90 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd37fe700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6800/2866534053 conn(0x7facbc062c90 0x7facbc065150 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 rx=0x7facc40041d0 tx=0x7facc4009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd9d83700 10 client.27555.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 1 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6801/2103743338 conn(0x7facd4028590 0x7facd40fbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd40475e0 osd=0 3 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd40475e0 osd=0 3 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd40475e0 osd=0 3 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 1 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd40475e0 osd=0 3 2024-02-13T00:48:37.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 0 1 2024-02-13T00:48:37.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd40475e0 osd=0 4 2024-02-13T00:48:37.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd9d83700 10 client.27555.objecter _send_command 1 2024-02-13T00:48:37.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7facd4107190 con 0x7facd4028590 2024-02-13T00:48:37.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd2ffd700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6801/2103743338 conn(0x7facd4028590 0x7facd40fbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:37.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd0ff9700 10 client.27555.objecter ms_handle_connect 0x7facd4028590 2024-02-13T00:48:37.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.779+0000 7facd2ffd700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6801/2103743338 conn(0x7facd4028590 0x7facd40fbc70 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:37.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.780+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facd4107190 con 0x7facd4028590 2024-02-13T00:48:37.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.780+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:48:37.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.780+0000 7facd0ff9700 10 client.27555.objecter _finish_command 1 = system:0 2024-02-13T00:48:37.809 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:48:37.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.780+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd40475e0 osd=0 4 2024-02-13T00:48:37.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.780+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 0 1 2024-02-13T00:48:37.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.794+0000 7facd9d83700 10 client.27555.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:37.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.794+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 2 2024-02-13T00:48:37.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.794+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd40475e0 osd=0 3 2024-02-13T00:48:37.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.794+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd40475e0 osd=0 3 2024-02-13T00:48:37.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.794+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd40475e0 osd=0 3 2024-02-13T00:48:37.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.794+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 2 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.794+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd40475e0 osd=0 3 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.794+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 0 2 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.794+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd40475e0 osd=0 4 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.794+0000 7facd9d83700 10 client.27555.objecter _send_command 2 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.794+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facd405a0e0 con 0x7facd4028590 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facd405a0e0 con 0x7facd4028590 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd0ff9700 10 client.27555.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd40475e0 osd=0 4 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 0 2 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd9d83700 10 client.27555.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 3 2024-02-13T00:48:37.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6805/150599780 conn(0x7facd4027be0 0x7facd4028000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:37.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd40fd2e0 osd=1 3 2024-02-13T00:48:37.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd40fd2e0 osd=1 3 2024-02-13T00:48:37.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd40fd2e0 osd=1 3 2024-02-13T00:48:37.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 3 2024-02-13T00:48:37.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd40fd2e0 osd=1 3 2024-02-13T00:48:37.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 1 3 2024-02-13T00:48:37.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd40fd2e0 osd=1 4 2024-02-13T00:48:37.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd9d83700 10 client.27555.objecter _send_command 3 2024-02-13T00:48:37.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7facd40fd450 con 0x7facd4027be0 2024-02-13T00:48:37.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd37fe700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6805/150599780 conn(0x7facd4027be0 0x7facd4028000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:37.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd37fe700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6805/150599780 conn(0x7facd4027be0 0x7facd4028000 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:37.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.796+0000 7facd0ff9700 10 client.27555.objecter ms_handle_connect 0x7facd4027be0 2024-02-13T00:48:37.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.797+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facd40fd450 con 0x7facd4027be0 2024-02-13T00:48:37.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.797+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:48:37.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.797+0000 7facd0ff9700 10 client.27555.objecter _finish_command 3 = system:0 2024-02-13T00:48:37.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.797+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd40fd2e0 osd=1 4 2024-02-13T00:48:37.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.797+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 1 3 2024-02-13T00:48:37.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.812+0000 7facd9d83700 10 client.27555.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:37.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.812+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 4 2024-02-13T00:48:37.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.812+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd40fd2e0 osd=1 3 2024-02-13T00:48:37.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.812+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd40fd2e0 osd=1 3 2024-02-13T00:48:37.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.812+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd40fd2e0 osd=1 3 2024-02-13T00:48:37.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.812+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 4 2024-02-13T00:48:37.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.812+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd40fd2e0 osd=1 3 2024-02-13T00:48:37.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.812+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 1 4 2024-02-13T00:48:37.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.812+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd40fd2e0 osd=1 4 2024-02-13T00:48:37.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.812+0000 7facd9d83700 10 client.27555.objecter _send_command 4 2024-02-13T00:48:37.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.812+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facd4027650 con 0x7facd4027be0 2024-02-13T00:48:37.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facd4027650 con 0x7facd4027be0 2024-02-13T00:48:37.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:37.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd0ff9700 10 client.27555.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd40fd2e0 osd=1 4 2024-02-13T00:48:37.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 1 4 2024-02-13T00:48:37.817 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:48:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd9d83700 10 client.27555.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 5 2024-02-13T00:48:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6809/3689719376 conn(0x7facd40fce40 0x7facd411c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4027a60 osd=2 3 2024-02-13T00:48:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4027a60 osd=2 3 2024-02-13T00:48:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4027a60 osd=2 3 2024-02-13T00:48:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 5 2024-02-13T00:48:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd4027a60 osd=2 3 2024-02-13T00:48:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 2 5 2024-02-13T00:48:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4027a60 osd=2 4 2024-02-13T00:48:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd9d83700 10 client.27555.objecter _send_command 5 2024-02-13T00:48:37.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7facd411efa0 con 0x7facd40fce40 2024-02-13T00:48:37.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.813+0000 7facd2ffd700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6809/3689719376 conn(0x7facd40fce40 0x7facd411c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:37.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.814+0000 7facd0ff9700 10 client.27555.objecter ms_handle_connect 0x7facd40fce40 2024-02-13T00:48:37.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.814+0000 7facd2ffd700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6809/3689719376 conn(0x7facd40fce40 0x7facd411c350 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:37.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.814+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facd411efa0 con 0x7facd40fce40 2024-02-13T00:48:37.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.814+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:48:37.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.814+0000 7facd0ff9700 10 client.27555.objecter _finish_command 5 = system:0 2024-02-13T00:48:37.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.814+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd4027a60 osd=2 4 2024-02-13T00:48:37.819 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.814+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 2 5 2024-02-13T00:48:37.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:37.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.829+0000 7facd9d83700 10 client.27555.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:37.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.829+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 6 2024-02-13T00:48:37.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.829+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4027a60 osd=2 3 2024-02-13T00:48:37.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.829+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4027a60 osd=2 3 2024-02-13T00:48:37.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.829+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4027a60 osd=2 3 2024-02-13T00:48:37.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.829+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 6 2024-02-13T00:48:37.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.829+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd4027a60 osd=2 3 2024-02-13T00:48:37.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.829+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 2 6 2024-02-13T00:48:37.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.829+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4027a60 osd=2 4 2024-02-13T00:48:37.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.829+0000 7facd9d83700 10 client.27555.objecter _send_command 6 2024-02-13T00:48:37.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.829+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facd40fc400 con 0x7facd40fce40 2024-02-13T00:48:37.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facd40fc400 con 0x7facd40fce40 2024-02-13T00:48:37.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:37.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd0ff9700 10 client.27555.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd4027a60 osd=2 4 2024-02-13T00:48:37.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 2 6 2024-02-13T00:48:37.834 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:48:37.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd9d83700 10 client.27555.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:37.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 7 2024-02-13T00:48:37.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6813/604971011 conn(0x7facd411d9c0 0x7facd411de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:37.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4119990 osd=3 3 2024-02-13T00:48:37.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4119990 osd=3 3 2024-02-13T00:48:37.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4119990 osd=3 3 2024-02-13T00:48:37.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 7 2024-02-13T00:48:37.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd4119990 osd=3 3 2024-02-13T00:48:37.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 3 7 2024-02-13T00:48:37.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4119990 osd=3 4 2024-02-13T00:48:37.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd9d83700 10 client.27555.objecter _send_command 7 2024-02-13T00:48:37.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7facd411a080 con 0x7facd411d9c0 2024-02-13T00:48:37.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.830+0000 7facd3fff700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6813/604971011 conn(0x7facd411d9c0 0x7facd411de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:37.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.831+0000 7facd0ff9700 10 client.27555.objecter ms_handle_connect 0x7facd411d9c0 2024-02-13T00:48:37.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.831+0000 7facd3fff700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6813/604971011 conn(0x7facd411d9c0 0x7facd411de00 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:37.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.831+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facd411a080 con 0x7facd411d9c0 2024-02-13T00:48:37.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.831+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:48:37.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.831+0000 7facd0ff9700 10 client.27555.objecter _finish_command 7 = system:0 2024-02-13T00:48:37.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.831+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd4119990 osd=3 4 2024-02-13T00:48:37.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.831+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 3 7 2024-02-13T00:48:37.846 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:37.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.845+0000 7facd9d83700 10 client.27555.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:37.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.845+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 8 2024-02-13T00:48:37.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.845+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4119990 osd=3 3 2024-02-13T00:48:37.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.845+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4119990 osd=3 3 2024-02-13T00:48:37.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.845+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4119990 osd=3 3 2024-02-13T00:48:37.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.845+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 8 2024-02-13T00:48:37.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.845+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd4119990 osd=3 3 2024-02-13T00:48:37.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.845+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 3 8 2024-02-13T00:48:37.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.845+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4119990 osd=3 4 2024-02-13T00:48:37.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.845+0000 7facd9d83700 10 client.27555.objecter _send_command 8 2024-02-13T00:48:37.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.845+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facd411eac0 con 0x7facd411d9c0 2024-02-13T00:48:37.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facd411eac0 con 0x7facd411d9c0 2024-02-13T00:48:37.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:37.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd0ff9700 10 client.27555.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd4119990 osd=3 4 2024-02-13T00:48:37.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 3 8 2024-02-13T00:48:37.852 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:48:37.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd9d83700 10 client.27555.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:37.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 9 2024-02-13T00:48:37.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6800/3344263388 conn(0x7facd41211d0 0x7facd41235c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd411e2f0 osd=4 3 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd411e2f0 osd=4 3 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd411e2f0 osd=4 3 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 9 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd411e2f0 osd=4 3 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 4 9 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd411e2f0 osd=4 4 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd9d83700 10 client.27555.objecter _send_command 9 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7facd4126210 con 0x7facd41211d0 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.847+0000 7facd37fe700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6800/3344263388 conn(0x7facd41211d0 0x7facd41235c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.848+0000 7facd0ff9700 10 client.27555.objecter ms_handle_connect 0x7facd41211d0 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.848+0000 7facd37fe700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6800/3344263388 conn(0x7facd41211d0 0x7facd41235c0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:37.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.849+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facd4126210 con 0x7facd41211d0 2024-02-13T00:48:37.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.849+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:48:37.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.849+0000 7facd0ff9700 10 client.27555.objecter _finish_command 9 = system:0 2024-02-13T00:48:37.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.849+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd411e2f0 osd=4 4 2024-02-13T00:48:37.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.849+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 4 9 2024-02-13T00:48:37.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.863+0000 7facd9d83700 10 client.27555.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:37.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.863+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 10 2024-02-13T00:48:37.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.863+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd411e2f0 osd=4 3 2024-02-13T00:48:37.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.863+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd411e2f0 osd=4 3 2024-02-13T00:48:37.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.863+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd411e2f0 osd=4 3 2024-02-13T00:48:37.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.863+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 10 2024-02-13T00:48:37.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.863+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd411e2f0 osd=4 3 2024-02-13T00:48:37.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.863+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 4 10 2024-02-13T00:48:37.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.863+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd411e2f0 osd=4 4 2024-02-13T00:48:37.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.863+0000 7facd9d83700 10 client.27555.objecter _send_command 10 2024-02-13T00:48:37.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.863+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facd4123ab0 con 0x7facd41211d0 2024-02-13T00:48:37.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facd4123ab0 con 0x7facd41211d0 2024-02-13T00:48:37.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:37.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd0ff9700 10 client.27555.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd411e2f0 osd=4 4 2024-02-13T00:48:37.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 4 10 2024-02-13T00:48:37.870 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:48:37.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd9d83700 10 client.27555.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:37.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 11 2024-02-13T00:48:37.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6804/2277917372 conn(0x7facd41241c0 0x7facd41265b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:37.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4123ab0 osd=5 3 2024-02-13T00:48:37.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4123ab0 osd=5 3 2024-02-13T00:48:37.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4123ab0 osd=5 3 2024-02-13T00:48:37.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 11 2024-02-13T00:48:37.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd4123ab0 osd=5 3 2024-02-13T00:48:37.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 5 11 2024-02-13T00:48:37.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4123ab0 osd=5 4 2024-02-13T00:48:37.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd9d83700 10 client.27555.objecter _send_command 11 2024-02-13T00:48:37.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7facd4129340 con 0x7facd41241c0 2024-02-13T00:48:37.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd2ffd700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6804/2277917372 conn(0x7facd41241c0 0x7facd41265b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:37.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd0ff9700 10 client.27555.objecter ms_handle_connect 0x7facd41241c0 2024-02-13T00:48:37.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.865+0000 7facd2ffd700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6804/2277917372 conn(0x7facd41241c0 0x7facd41265b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:37.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.866+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facd4129340 con 0x7facd41241c0 2024-02-13T00:48:37.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.866+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:48:37.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.866+0000 7facd0ff9700 10 client.27555.objecter _finish_command 11 = system:0 2024-02-13T00:48:37.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.866+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd4123ab0 osd=5 4 2024-02-13T00:48:37.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.866+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 5 11 2024-02-13T00:48:37.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.880+0000 7facd9d83700 10 client.27555.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:37.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.880+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 12 2024-02-13T00:48:37.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.880+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4123ab0 osd=5 3 2024-02-13T00:48:37.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.880+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4123ab0 osd=5 3 2024-02-13T00:48:37.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.880+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4123ab0 osd=5 3 2024-02-13T00:48:37.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.880+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 12 2024-02-13T00:48:37.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.880+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd4123ab0 osd=5 3 2024-02-13T00:48:37.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.880+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 5 12 2024-02-13T00:48:37.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.880+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4123ab0 osd=5 4 2024-02-13T00:48:37.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.880+0000 7facd9d83700 10 client.27555.objecter _send_command 12 2024-02-13T00:48:37.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.880+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facd4129340 con 0x7facd41241c0 2024-02-13T00:48:37.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facd4129340 con 0x7facd41241c0 2024-02-13T00:48:37.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:37.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd0ff9700 10 client.27555.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd4123ab0 osd=5 4 2024-02-13T00:48:37.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 5 12 2024-02-13T00:48:37.887 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:48:37.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd9d83700 10 client.27555.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:37.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 13 2024-02-13T00:48:37.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6808/1506640341 conn(0x7facd4128660 0x7facd4128ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:37.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4129340 osd=6 3 2024-02-13T00:48:37.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4129340 osd=6 3 2024-02-13T00:48:37.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4129340 osd=6 3 2024-02-13T00:48:37.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 13 2024-02-13T00:48:37.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd4129340 osd=6 3 2024-02-13T00:48:37.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 6 13 2024-02-13T00:48:37.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4129340 osd=6 4 2024-02-13T00:48:37.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd9d83700 10 client.27555.objecter _send_command 13 2024-02-13T00:48:37.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7facd4129160 con 0x7facd4128660 2024-02-13T00:48:37.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd3fff700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6808/1506640341 conn(0x7facd4128660 0x7facd4128ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:37.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd0ff9700 10 client.27555.objecter ms_handle_connect 0x7facd4128660 2024-02-13T00:48:37.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.882+0000 7facd3fff700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6808/1506640341 conn(0x7facd4128660 0x7facd4128ac0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:37.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.883+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facd4129160 con 0x7facd4128660 2024-02-13T00:48:37.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.883+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:48:37.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.883+0000 7facd0ff9700 10 client.27555.objecter _finish_command 13 = system:0 2024-02-13T00:48:37.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.883+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd4129340 osd=6 4 2024-02-13T00:48:37.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.883+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 6 13 2024-02-13T00:48:37.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.897+0000 7facd9d83700 10 client.27555.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:37.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.897+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 14 2024-02-13T00:48:37.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.897+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4129340 osd=6 3 2024-02-13T00:48:37.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.897+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4129340 osd=6 3 2024-02-13T00:48:37.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.897+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4129340 osd=6 3 2024-02-13T00:48:37.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.897+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 14 2024-02-13T00:48:37.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.897+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd4129340 osd=6 3 2024-02-13T00:48:37.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.897+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 6 14 2024-02-13T00:48:37.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.897+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4129340 osd=6 4 2024-02-13T00:48:37.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.897+0000 7facd9d83700 10 client.27555.objecter _send_command 14 2024-02-13T00:48:37.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.897+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facd4129160 con 0x7facd4128660 2024-02-13T00:48:37.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.898+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facd4129160 con 0x7facd4128660 2024-02-13T00:48:37.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.898+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:37.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.898+0000 7facd0ff9700 10 client.27555.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.898+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd4129340 osd=6 4 2024-02-13T00:48:37.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.898+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 6 14 2024-02-13T00:48:37.903 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:48:37.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd9d83700 10 client.27555.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:37.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 15 2024-02-13T00:48:37.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6812/1964192772 conn(0x7facd412ba80 0x7facd412bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:37.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4129160 osd=7 3 2024-02-13T00:48:37.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4129160 osd=7 3 2024-02-13T00:48:37.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4129160 osd=7 3 2024-02-13T00:48:37.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 15 2024-02-13T00:48:37.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd4129160 osd=7 3 2024-02-13T00:48:37.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 7 15 2024-02-13T00:48:37.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4129160 osd=7 4 2024-02-13T00:48:37.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd9d83700 10 client.27555.objecter _send_command 15 2024-02-13T00:48:37.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7facd412c580 con 0x7facd412ba80 2024-02-13T00:48:37.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd37fe700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6812/1964192772 conn(0x7facd412ba80 0x7facd412bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:37.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd0ff9700 10 client.27555.objecter ms_handle_connect 0x7facd412ba80 2024-02-13T00:48:37.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.899+0000 7facd37fe700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6812/1964192772 conn(0x7facd412ba80 0x7facd412bee0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:37.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.900+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facd412c580 con 0x7facd412ba80 2024-02-13T00:48:37.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.900+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:48:37.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.900+0000 7facd0ff9700 10 client.27555.objecter _finish_command 15 = system:0 2024-02-13T00:48:37.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.900+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd4129160 osd=7 4 2024-02-13T00:48:37.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.900+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 7 15 2024-02-13T00:48:37.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.914+0000 7facd9d83700 10 client.27555.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:37.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.914+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign -1 16 2024-02-13T00:48:37.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.914+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4129160 osd=7 3 2024-02-13T00:48:37.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.914+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4129160 osd=7 3 2024-02-13T00:48:37.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.914+0000 7facd9d83700 20 client.27555.objecter _get_session s=0x7facd4129160 osd=7 3 2024-02-13T00:48:37.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.914+0000 7facd9d83700 15 client.27555.objecter _session_command_op_remove -1 16 2024-02-13T00:48:37.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.914+0000 7facd9d83700 20 client.27555.objecter get_session s=0x7facd4129160 osd=7 3 2024-02-13T00:48:37.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.914+0000 7facd9d83700 15 client.27555.objecter _session_command_op_assign 7 16 2024-02-13T00:48:37.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.914+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4129160 osd=7 4 2024-02-13T00:48:37.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.914+0000 7facd9d83700 10 client.27555.objecter _send_command 16 2024-02-13T00:48:37.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.914+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facd412c580 con 0x7facd412ba80 2024-02-13T00:48:37.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.915+0000 7facd0ff9700 1 -- 172.21.15.162:0/2092750269 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facd412c580 con 0x7facd412ba80 2024-02-13T00:48:37.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.915+0000 7facd0ff9700 10 client.27555.objecter ms_dispatch 0x7facd4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:37.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.915+0000 7facd0ff9700 10 client.27555.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.915+0000 7facd0ff9700 20 client.27555.objecter put_session s=0x7facd4129160 osd=7 4 2024-02-13T00:48:37.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.915+0000 7facd0ff9700 15 client.27555.objecter _session_command_op_remove 7 16 2024-02-13T00:48:37.919 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:48:37.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 10 librados: watch_flush enter 2024-02-13T00:48:37.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 10 librados: watch_flush exit 2024-02-13T00:48:37.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 10 client.27555.objecter close_session for osd.0 2024-02-13T00:48:37.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6801/2103743338 conn(0x7facd4028590 msgr2=0x7facd40fbc70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6801/2103743338 conn(0x7facd4028590 0x7facd40fbc70 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd40475e0 osd=0 1 2024-02-13T00:48:37.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 10 client.27555.objecter close_session for osd.1 2024-02-13T00:48:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6805/150599780 conn(0x7facd4027be0 msgr2=0x7facd4028000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6805/150599780 conn(0x7facd4027be0 0x7facd4028000 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd40fd2e0 osd=1 1 2024-02-13T00:48:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 10 client.27555.objecter close_session for osd.2 2024-02-13T00:48:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6809/3689719376 conn(0x7facd40fce40 msgr2=0x7facd411c350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6809/3689719376 conn(0x7facd40fce40 0x7facd411c350 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4027a60 osd=2 1 2024-02-13T00:48:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 10 client.27555.objecter close_session for osd.3 2024-02-13T00:48:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6813/604971011 conn(0x7facd411d9c0 msgr2=0x7facd411de00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6813/604971011 conn(0x7facd411d9c0 0x7facd411de00 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd3fff700 1 -- 172.21.15.162:0/2092750269 reap_dead start 2024-02-13T00:48:37.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4119990 osd=3 1 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 10 client.27555.objecter close_session for osd.4 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6800/3344263388 conn(0x7facd41211d0 msgr2=0x7facd41235c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6800/3344263388 conn(0x7facd41211d0 0x7facd41235c0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd411e2f0 osd=4 1 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 10 client.27555.objecter close_session for osd.5 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd3fff700 1 -- 172.21.15.162:0/2092750269 reap_dead start 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6804/2277917372 conn(0x7facd41241c0 msgr2=0x7facd41265b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6804/2277917372 conn(0x7facd41241c0 0x7facd41265b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4123ab0 osd=5 1 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 10 client.27555.objecter close_session for osd.6 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6808/1506640341 conn(0x7facd4128660 msgr2=0x7facd4128ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6808/1506640341 conn(0x7facd4128660 0x7facd4128ac0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4129340 osd=6 1 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 10 client.27555.objecter close_session for osd.7 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6812/1964192772 conn(0x7facd412ba80 msgr2=0x7facd412bee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.181:6812/1964192772 conn(0x7facd412ba80 0x7facd412bee0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 20 client.27555.objecter put_session s=0x7facd4129160 osd=7 1 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 20 client.27555.objecter shutdown clearing up homeless session... 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 10 client.27555.objecter successfully canceled tick 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6800/2866534053 conn(0x7facbc062c90 msgr2=0x7facbc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> v2:172.21.15.162:6800/2866534053 conn(0x7facbc062c90 0x7facbc065150 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 rx=0x7facc40041d0 tx=0x7facc4009040).stop 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facd41021d0 msgr2=0x7facd4108380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 --2- 172.21.15.162:0/2092750269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facd41021d0 0x7facd4108380 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 rx=0x7faccc001dc0 tx=0x7faccc004d10).stop 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd3fff700 1 -- 172.21.15.162:0/2092750269 reap_dead start 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 shutdown_connections 2024-02-13T00:48:37.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 >> 172.21.15.162:0/2092750269 conn(0x7facd40f2740 msgr2=0x7facd40f33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:37.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 shutdown_connections 2024-02-13T00:48:37.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 -- 172.21.15.162:0/2092750269 wait complete. 2024-02-13T00:48:37.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:37.916+0000 7facd9d83700 1 librados: shutdown 2024-02-13T00:48:37.925 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.925 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.925 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.925 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.925 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.925 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.926 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:48:37.926 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:48:38.068 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-13T00:48:38.094 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:38.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:38.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:38.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:38.224 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:38.246 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:38.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:38.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:38.366 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:48:38.377 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:38.397 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:38.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:38.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:38.491 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:37.472607+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e503: 8 total, 8 up, 4 in 2024-02-13T00:48:38.491 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:37.779429+0000 mon.a (mon.0) 2750 : audit [DBG] from='client.? 172.21.15.162:0/2092750269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:38.492 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:38.461875+0000 mon.c (mon.2) 407 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-13T00:48:38.492 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:38.462756+0000 mon.a (mon.0) 2751 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-13T00:48:38.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:38.523 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6] out_osds: [3, 2, 7, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-13T00:48:38.523 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:48:38.524 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-02-13T00:48:38.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.0 reset_purged_snaps_last 2024-02-13T00:48:38.532 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:38.649 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:37.472607+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e503: 8 total, 8 up, 4 in 2024-02-13T00:48:38.649 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:37.779429+0000 mon.a (mon.0) 2750 : audit [DBG] from='client.? 172.21.15.162:0/2092750269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:38.649 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:38.461875+0000 mon.c (mon.2) 407 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-13T00:48:38.649 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:38.462756+0000 mon.a (mon.0) 2751 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-13T00:48:38.649 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:48:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:48:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:48:38.682 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:38.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:38.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:38.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:48:38.777 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:37.472607+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e503: 8 total, 8 up, 4 in 2024-02-13T00:48:38.777 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:37.779429+0000 mon.a (mon.0) 2750 : audit [DBG] from='client.? 172.21.15.162:0/2092750269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:38.777 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:38.461875+0000 mon.c (mon.2) 407 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-13T00:48:38.777 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:38.462756+0000 mon.a (mon.0) 2751 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-13T00:48:38.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:38.810 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:38.817 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:38.837 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:38.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:38.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:38.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:38.969 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:39.594 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:39.712 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:38.461589+0000 mgr.y (mgr.14544) 450 : cluster [DBG] pgmap v755: 237 pgs: 1 peering, 236 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:39.712 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:38.480513+0000 mon.a (mon.0) 2752 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-13T00:48:39.712 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:38.480587+0000 mon.a (mon.0) 2753 : cluster [DBG] osdmap e504: 8 total, 8 up, 4 in 2024-02-13T00:48:39.712 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:38.510432+0000 mon.b (mon.1) 629 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:39.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:38.510841+0000 mon.a (mon.0) 2754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:39.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:38.511297+0000 mon.b (mon.1) 630 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-79"}]: dispatch 2024-02-13T00:48:39.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:38.511535+0000 mon.a (mon.0) 2755 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-79"}]: dispatch 2024-02-13T00:48:39.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:38.979697+0000 osd.1 (osd.1) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:39.713 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:38.979891+0000 osd.1 (osd.1) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:39.717 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:39.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:39.743 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:38.461589+0000 mgr.y (mgr.14544) 450 : cluster [DBG] pgmap v755: 237 pgs: 1 peering, 236 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:39.743 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:38.480513+0000 mon.a (mon.0) 2752 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-13T00:48:39.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:38.480587+0000 mon.a (mon.0) 2753 : cluster [DBG] osdmap e504: 8 total, 8 up, 4 in 2024-02-13T00:48:39.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:38.510432+0000 mon.b (mon.1) 629 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:39.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:38.510841+0000 mon.a (mon.0) 2754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:39.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:38.511297+0000 mon.b (mon.1) 630 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-79"}]: dispatch 2024-02-13T00:48:39.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:38.511535+0000 mon.a (mon.0) 2755 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-79"}]: dispatch 2024-02-13T00:48:39.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:38.979697+0000 osd.1 (osd.1) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:39.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:38.979891+0000 osd.1 (osd.1) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:39.744 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:38.461589+0000 mgr.y (mgr.14544) 450 : cluster [DBG] pgmap v755: 237 pgs: 1 peering, 236 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:39.744 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:38.480513+0000 mon.a (mon.0) 2752 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-13T00:48:39.744 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:38.480587+0000 mon.a (mon.0) 2753 : cluster [DBG] osdmap e504: 8 total, 8 up, 4 in 2024-02-13T00:48:39.744 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:38.510432+0000 mon.b (mon.1) 629 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:39.745 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:38.510841+0000 mon.a (mon.0) 2754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:39.745 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:38.511297+0000 mon.b (mon.1) 630 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-79"}]: dispatch 2024-02-13T00:48:39.745 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:38.511535+0000 mon.a (mon.0) 2755 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-79"}]: dispatch 2024-02-13T00:48:39.745 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:38.979697+0000 osd.1 (osd.1) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:39.745 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:38.979891+0000 osd.1 (osd.1) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:39.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:39.850 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:40.218 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:48:40.219 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:48:40.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:40.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:40.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:40.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:40.542 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:40.656 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:39.497144+0000 mon.a (mon.0) 2756 : cluster [DBG] osdmap e505: 8 total, 8 up, 4 in 2024-02-13T00:48:40.656 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:39.497144+0000 mon.a (mon.0) 2756 : cluster [DBG] osdmap e505: 8 total, 8 up, 4 in 2024-02-13T00:48:40.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:48:40.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:39.497144+0000 mon.a (mon.0) 2756 : cluster [DBG] osdmap e505: 8 total, 8 up, 4 in 2024-02-13T00:48:41.039 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-13T00:48:41.090 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:41.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:41.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:41.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:41.223 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:41.305 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:48:41.321 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:41.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:41.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.331+0000 7fe06df23700 1 -- 172.21.15.162:0/3185947874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0680f7c80 msgr2=0x7fe068103a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:41.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.331+0000 7fe06df23700 1 --2- 172.21.15.162:0/3185947874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0680f7c80 0x7fe068103a10 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7fe060009c10 tx=0x7fe060004d10).stop 2024-02-13T00:48:41.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.331+0000 7fe06df23700 1 -- 172.21.15.162:0/3185947874 shutdown_connections 2024-02-13T00:48:41.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.331+0000 7fe06df23700 1 --2- 172.21.15.162:0/3185947874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0680f7c80 0x7fe068103a10 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:41.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.331+0000 7fe06df23700 1 --2- 172.21.15.162:0/3185947874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0680fcc30 0x7fe0680fd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:41.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.331+0000 7fe06df23700 1 --2- 172.21.15.162:0/3185947874 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0680fd590 0x7fe0680f75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:41.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.331+0000 7fe06df23700 1 -- 172.21.15.162:0/3185947874 >> 172.21.15.162:0/3185947874 conn(0x7fe06806c470 msgr2=0x7fe06806c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:41.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.331+0000 7fe06df23700 1 -- 172.21.15.162:0/3185947874 shutdown_connections 2024-02-13T00:48:41.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.331+0000 7fe06df23700 1 -- 172.21.15.162:0/3185947874 wait complete. 2024-02-13T00:48:41.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe06df23700 1 librados: starting msgr at 2024-02-13T00:48:41.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe06df23700 1 librados: starting objecter 2024-02-13T00:48:41.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe06df23700 1 Processor -- start 2024-02-13T00:48:41.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe06df23700 1 -- start start 2024-02-13T00:48:41.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe06df23700 1 librados: setting wanted keys 2024-02-13T00:48:41.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe06df23700 1 librados: calling monclient init 2024-02-13T00:48:41.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe06df23700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0680f7c80 0x7fe0680fdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:41.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe06df23700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0680fcc30 0x7fe0680fe050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:41.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe06df23700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0680fd590 0x7fe0680fe540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:41.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe06df23700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe068101c30 con 0x7fe0680fcc30 2024-02-13T00:48:41.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe06df23700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06811aaf0 con 0x7fe0680fd590 2024-02-13T00:48:41.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe06df23700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06811acb0 con 0x7fe0680f7c80 2024-02-13T00:48:41.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe066ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0680fcc30 0x7fe0680fe050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:41.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe066ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0680fcc30 0x7fe0680fe050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:47392/0 (socket says 172.21.15.162:47392) 2024-02-13T00:48:41.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe0677fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0680f7c80 0x7fe0680fdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.332+0000 7fe066ffd700 1 -- 172.21.15.162:0/637199619 learned_addr learned my addr 172.21.15.162:0/637199619 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe067fff700 1 --2- 172.21.15.162:0/637199619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0680fd590 0x7fe0680fe540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe066ffd700 1 -- 172.21.15.162:0/637199619 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0680f7c80 msgr2=0x7fe0680fdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe066ffd700 1 --2- 172.21.15.162:0/637199619 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0680f7c80 0x7fe0680fdb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe066ffd700 1 -- 172.21.15.162:0/637199619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0680fd590 msgr2=0x7fe0680fe540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe066ffd700 1 --2- 172.21.15.162:0/637199619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0680fd590 0x7fe0680fe540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe066ffd700 1 -- 172.21.15.162:0/637199619 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0600097c0 con 0x7fe0680fcc30 2024-02-13T00:48:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe066ffd700 1 --2- 172.21.15.162:0/637199619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0680fcc30 0x7fe0680fe050 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7fe05c00e910 tx=0x7fe05c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe064ff9700 10 client.?.objecter ms_handle_connect 0x7fe0680fcc30 2024-02-13T00:48:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe064ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe064ff9700 1 -- 172.21.15.162:0/637199619 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c004620 con 0x7fe0680fcc30 2024-02-13T00:48:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe06df23700 1 -- 172.21.15.162:0/637199619 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe06811af90 con 0x7fe0680fcc30 2024-02-13T00:48:41.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe064ff9700 1 -- 172.21.15.162:0/637199619 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe05c00f040 con 0x7fe0680fcc30 2024-02-13T00:48:41.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe064ff9700 1 -- 172.21.15.162:0/637199619 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c0146f0 con 0x7fe0680fcc30 2024-02-13T00:48:41.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe06df23700 10 client.27561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:41.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe06df23700 1 -- 172.21.15.162:0/637199619 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06811b440 con 0x7fe0680fcc30 2024-02-13T00:48:41.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.333+0000 7fe06df23700 1 librados: init done 2024-02-13T00:48:41.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.334+0000 7fe06df23700 10 client.27561.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:41.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.334+0000 7fe06df23700 15 client.27561.objecter _session_command_op_assign -1 1 2024-02-13T00:48:41.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.334+0000 7fe06df23700 20 client.27561.objecter _get_session osd=-1 returning homeless 2024-02-13T00:48:41.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.334+0000 7fe06df23700 10 client.27561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:41.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.334+0000 7fe06df23700 1 -- 172.21.15.162:0/637199619 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe06805f9a0 con 0x7fe0680fcc30 2024-02-13T00:48:41.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 1 -- 172.21.15.162:0/637199619 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe05c0090b0 con 0x7fe0680fcc30 2024-02-13T00:48:41.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 1 --2- 172.21.15.162:0/637199619 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe050062c90 0x7fe050065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:41.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 1 -- 172.21.15.162:0/637199619 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(506..506 src has 449..506) v4 ==== 8070+0+0 (secure 0 0 0) 0x7fe05c07cda0 con 0x7fe0680fcc30 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 10 client.27561.objecter ms_dispatch 0x7fe068001550 osd_map(506..506 src has 449..506) v4 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 3 client.27561.objecter handle_osd_map got epochs [506,506] > 0 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 3 client.27561.objecter handle_osd_map decoding full epoch 506 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe0677fe700 1 --2- 172.21.15.162:0/637199619 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe050062c90 0x7fe050065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 10 client.27561.objecter checking command 1 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 1 --2- 172.21.15.162:0/637199619 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe05006fa80 0x7fe050071f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 20 client.27561.objecter _get_session s=0x7fe05006f3d0 osd=0 3 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 20 client.27561.objecter put_session s=0x7fe05006f3d0 osd=0 3 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 15 client.27561.objecter _session_command_op_remove -1 1 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 20 client.27561.objecter _get_session s=0x7fe05006f3d0 osd=0 3 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 20 client.27561.objecter get_session s=0x7fe05006f3d0 osd=0 3 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 15 client.27561.objecter _session_command_op_assign 0 1 2024-02-13T00:48:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 20 client.27561.objecter put_session s=0x7fe05006f3d0 osd=0 4 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 10 client.27561.objecter _send_command 1 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 1 -- 172.21.15.162:0/637199619 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe050072560 con 0x7fe05006fa80 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 20 client.27561.objecter dump_active .. 0 homeless 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.335+0000 7fe064ff9700 1 -- 172.21.15.162:0/637199619 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=506) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe05c07d150 con 0x7fe0680fcc30 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.336+0000 7fe067fff700 1 --2- 172.21.15.162:0/637199619 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe05006fa80 0x7fe050071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.336+0000 7fe064ff9700 10 client.27561.objecter ms_handle_connect 0x7fe050062c90 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.336+0000 7fe0677fe700 1 --2- 172.21.15.162:0/637199619 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe050062c90 0x7fe050065150 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 rx=0x7fe0580048b0 tx=0x7fe05800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.336+0000 7fe064ff9700 10 client.27561.objecter ms_handle_connect 0x7fe05006fa80 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.336+0000 7fe067fff700 1 --2- 172.21.15.162:0/637199619 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe05006fa80 0x7fe050071f20 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.336+0000 7fe064ff9700 1 -- 172.21.15.162:0/637199619 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe050072560 con 0x7fe05006fa80 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.336+0000 7fe064ff9700 10 client.27561.objecter ms_dispatch 0x7fe068001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.336+0000 7fe064ff9700 10 client.27561.objecter _finish_command 1 = system:0 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.336+0000 7fe064ff9700 20 client.27561.objecter put_session s=0x7fe05006f3d0 osd=0 4 2024-02-13T00:48:41.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.336+0000 7fe064ff9700 15 client.27561.objecter _session_command_op_remove 0 1 2024-02-13T00:48:41.346 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:41.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.355+0000 7fe06df23700 10 client.27561.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-02-13T00:48:41.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.355+0000 7fe06df23700 15 client.27561.objecter _session_command_op_assign -1 2 2024-02-13T00:48:41.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.355+0000 7fe06df23700 20 client.27561.objecter _get_session s=0x7fe05006f3d0 osd=0 3 2024-02-13T00:48:41.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.355+0000 7fe06df23700 20 client.27561.objecter put_session s=0x7fe05006f3d0 osd=0 3 2024-02-13T00:48:41.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.355+0000 7fe06df23700 20 client.27561.objecter _get_session s=0x7fe05006f3d0 osd=0 3 2024-02-13T00:48:41.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.355+0000 7fe06df23700 15 client.27561.objecter _session_command_op_remove -1 2 2024-02-13T00:48:41.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.355+0000 7fe06df23700 20 client.27561.objecter get_session s=0x7fe05006f3d0 osd=0 3 2024-02-13T00:48:41.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.355+0000 7fe06df23700 15 client.27561.objecter _session_command_op_assign 0 2 2024-02-13T00:48:41.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.355+0000 7fe06df23700 20 client.27561.objecter put_session s=0x7fe05006f3d0 osd=0 4 2024-02-13T00:48:41.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.355+0000 7fe06df23700 10 client.27561.objecter _send_command 2 2024-02-13T00:48:41.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.355+0000 7fe06df23700 1 -- 172.21.15.162:0/637199619 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7fe0680475e0 con 0x7fe05006fa80 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe064ff9700 1 -- 172.21.15.162:0/637199619 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe0680475e0 con 0x7fe05006fa80 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe064ff9700 10 client.27561.objecter ms_dispatch 0x7fe068001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe064ff9700 10 client.27561.objecter _finish_command 2 = system:0 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe064ff9700 20 client.27561.objecter put_session s=0x7fe05006f3d0 osd=0 4 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe064ff9700 15 client.27561.objecter _session_command_op_remove 0 2 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 10 librados: watch_flush enter 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 10 librados: watch_flush exit 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 10 client.27561.objecter close_session for osd.0 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 1 -- 172.21.15.162:0/637199619 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe05006fa80 msgr2=0x7fe050071f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 1 --2- 172.21.15.162:0/637199619 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe05006fa80 0x7fe050071f20 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 20 client.27561.objecter put_session s=0x7fe05006f3d0 osd=0 1 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 20 client.27561.objecter shutdown clearing up homeless session... 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 10 client.27561.objecter successfully canceled tick 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 1 -- 172.21.15.162:0/637199619 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe050062c90 msgr2=0x7fe050065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 1 --2- 172.21.15.162:0/637199619 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe050062c90 0x7fe050065150 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 rx=0x7fe0580048b0 tx=0x7fe05800b040).stop 2024-02-13T00:48:41.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 1 -- 172.21.15.162:0/637199619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0680fcc30 msgr2=0x7fe0680fe050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:41.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 1 --2- 172.21.15.162:0/637199619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0680fcc30 0x7fe0680fe050 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7fe05c00e910 tx=0x7fe05c00b040).stop 2024-02-13T00:48:41.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe067fff700 1 -- 172.21.15.162:0/637199619 reap_dead start 2024-02-13T00:48:41.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 1 -- 172.21.15.162:0/637199619 shutdown_connections 2024-02-13T00:48:41.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.356+0000 7fe06df23700 1 -- 172.21.15.162:0/637199619 >> 172.21.15.162:0/637199619 conn(0x7fe06806c470 msgr2=0x7fe0680f6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:41.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.357+0000 7fe06df23700 1 -- 172.21.15.162:0/637199619 shutdown_connections 2024-02-13T00:48:41.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.357+0000 7fe06df23700 1 -- 172.21.15.162:0/637199619 wait complete. 2024-02-13T00:48:41.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:41.357+0000 7fe06df23700 1 librados: shutdown 2024-02-13T00:48:41.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:41.457 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:41.733 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:40.462657+0000 mgr.y (mgr.14544) 451 : cluster [DBG] pgmap v758: 205 pgs: 2 clean+premerge+peered, 203 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-02-13T00:48:41.733 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:40.501831+0000 mon.a (mon.0) 2757 : cluster [DBG] osdmap e506: 8 total, 8 up, 4 in 2024-02-13T00:48:41.733 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:40.504777+0000 mon.b (mon.1) 631 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:41.733 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:40.507615+0000 mon.a (mon.0) 2758 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:41.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:40.462657+0000 mgr.y (mgr.14544) 451 : cluster [DBG] pgmap v758: 205 pgs: 2 clean+premerge+peered, 203 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-02-13T00:48:41.734 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:40.501831+0000 mon.a (mon.0) 2757 : cluster [DBG] osdmap e506: 8 total, 8 up, 4 in 2024-02-13T00:48:41.734 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:40.504777+0000 mon.b (mon.1) 631 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:41.734 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:40.507615+0000 mon.a (mon.0) 2758 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:41.773 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:40.462657+0000 mgr.y (mgr.14544) 451 : cluster [DBG] pgmap v758: 205 pgs: 2 clean+premerge+peered, 203 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-02-13T00:48:41.773 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:40.501831+0000 mon.a (mon.0) 2757 : cluster [DBG] osdmap e506: 8 total, 8 up, 4 in 2024-02-13T00:48:41.773 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:40.504777+0000 mon.b (mon.1) 631 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:41.773 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:40.507615+0000 mon.a (mon.0) 2758 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:42.702 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:41.504064+0000 mon.a (mon.0) 2759 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:48:42.702 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:41.504160+0000 mon.a (mon.0) 2760 : cluster [DBG] osdmap e507: 8 total, 8 up, 4 in 2024-02-13T00:48:42.703 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:41.506347+0000 mon.b (mon.1) 632 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:48:42.703 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:41.515908+0000 mon.b (mon.1) 633 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:42.703 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:41.516374+0000 mon.a (mon.0) 2761 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:42.703 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:42.463296+0000 mon.c (mon.2) 408 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-13T00:48:42.703 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:42.464173+0000 mon.a (mon.0) 2762 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-13T00:48:42.703 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:41.504064+0000 mon.a (mon.0) 2759 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:48:42.703 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:41.504160+0000 mon.a (mon.0) 2760 : cluster [DBG] osdmap e507: 8 total, 8 up, 4 in 2024-02-13T00:48:42.703 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:41.506347+0000 mon.b (mon.1) 632 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:48:42.703 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:41.515908+0000 mon.b (mon.1) 633 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:42.703 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:41.516374+0000 mon.a (mon.0) 2761 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:42.703 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:42.463296+0000 mon.c (mon.2) 408 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-13T00:48:42.704 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:42.464173+0000 mon.a (mon.0) 2762 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-13T00:48:42.737 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:41.504064+0000 mon.a (mon.0) 2759 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:48:42.737 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:41.504160+0000 mon.a (mon.0) 2760 : cluster [DBG] osdmap e507: 8 total, 8 up, 4 in 2024-02-13T00:48:42.737 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:41.506347+0000 mon.b (mon.1) 632 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:48:42.738 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:41.515908+0000 mon.b (mon.1) 633 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:42.738 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:41.516374+0000 mon.a (mon.0) 2761 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:42.738 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:42.463296+0000 mon.c (mon.2) 408 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-13T00:48:42.738 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:42.464173+0000 mon.a (mon.0) 2762 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-13T00:48:42.812 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:42.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:42.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:42.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.902+0000 7fe9105c4700 1 -- 172.21.15.162:0/2904901150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9080fcb30 msgr2=0x7fe9080fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.902+0000 7fe9105c4700 1 --2- 172.21.15.162:0/2904901150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9080fcb30 0x7fe9080fcf10 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 rx=0x7fe8f80047b0 tx=0x7fe8f8009cb0).stop 2024-02-13T00:48:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.902+0000 7fe9105c4700 1 -- 172.21.15.162:0/2904901150 shutdown_connections 2024-02-13T00:48:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.902+0000 7fe9105c4700 1 --2- 172.21.15.162:0/2904901150 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9080f73a0 0x7fe9081039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.902+0000 7fe9105c4700 1 --2- 172.21.15.162:0/2904901150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9080fcb30 0x7fe9080fcf10 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.902+0000 7fe9105c4700 1 --2- 172.21.15.162:0/2904901150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9080fd490 0x7fe9080f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.902+0000 7fe9105c4700 1 -- 172.21.15.162:0/2904901150 >> 172.21.15.162:0/2904901150 conn(0x7fe90806eb90 msgr2=0x7fe9080f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.902+0000 7fe9105c4700 1 -- 172.21.15.162:0/2904901150 shutdown_connections 2024-02-13T00:48:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.902+0000 7fe9105c4700 1 -- 172.21.15.162:0/2904901150 wait complete. 2024-02-13T00:48:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe9105c4700 1 librados: starting msgr at 2024-02-13T00:48:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe9105c4700 1 librados: starting objecter 2024-02-13T00:48:42.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe9105c4700 1 Processor -- start 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe9105c4700 1 -- start start 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe9105c4700 1 librados: setting wanted keys 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe9105c4700 1 librados: calling monclient init 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe9105c4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9080f73a0 0x7fe90810d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe9105c4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9080fcb30 0x7fe90810d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe9105c4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9080fd490 0x7fe908107230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe9105c4700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe908107720 con 0x7fe9080f73a0 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe9105c4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe908107890 con 0x7fe9080fd490 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe9105c4700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe908107a50 con 0x7fe9080fcb30 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe90e360700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9080f73a0 0x7fe90810d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe90e360700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9080f73a0 0x7fe90810d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:47408/0 (socket says 172.21.15.162:47408) 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe90e360700 1 -- 172.21.15.162:0/1957566073 learned_addr learned my addr 172.21.15.162:0/1957566073 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:42.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.903+0000 7fe90db5f700 1 --2- 172.21.15.162:0/1957566073 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9080fcb30 0x7fe90810d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe90eb61700 1 --2- 172.21.15.162:0/1957566073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9080fd490 0x7fe908107230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe90e360700 1 -- 172.21.15.162:0/1957566073 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9080fcb30 msgr2=0x7fe90810d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe90e360700 1 --2- 172.21.15.162:0/1957566073 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9080fcb30 0x7fe90810d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe90e360700 1 -- 172.21.15.162:0/1957566073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9080fd490 msgr2=0x7fe908107230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe90e360700 1 --2- 172.21.15.162:0/1957566073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9080fd490 0x7fe908107230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe90e360700 1 -- 172.21.15.162:0/1957566073 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9000097c0 con 0x7fe9080f73a0 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe90e360700 1 --2- 172.21.15.162:0/1957566073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9080f73a0 0x7fe90810d090 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 rx=0x7fe8f80044e0 tx=0x7fe8f8007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe8ff7fe700 10 client.?.objecter ms_handle_connect 0x7fe9080f73a0 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe8ff7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe8ff7fe700 1 -- 172.21.15.162:0/1957566073 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8018070 con 0x7fe9080f73a0 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe9105c4700 1 -- 172.21.15.162:0/1957566073 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f8005040 con 0x7fe9080f73a0 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe8ff7fe700 1 -- 172.21.15.162:0/1957566073 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe8f80092e0 con 0x7fe9080f73a0 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe8ff7fe700 1 -- 172.21.15.162:0/1957566073 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f801dc50 con 0x7fe9080f73a0 2024-02-13T00:48:42.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe9105c4700 10 client.27567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe9105c4700 1 -- 172.21.15.162:0/1957566073 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe908107f70 con 0x7fe9080f73a0 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.904+0000 7fe9105c4700 1 librados: init done 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.905+0000 7fe9105c4700 1 -- 172.21.15.162:0/1957566073 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9080475e0 con 0x7fe9080f73a0 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.905+0000 7fe8ff7fe700 1 -- 172.21.15.162:0/1957566073 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe8f801d420 con 0x7fe9080f73a0 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.906+0000 7fe8ff7fe700 1 --2- 172.21.15.162:0/1957566073 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe8f4062c90 0x7fe8f4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.906+0000 7fe8ff7fe700 1 -- 172.21.15.162:0/1957566073 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(508..508 src has 449..508) v4 ==== 8099+0+0 (secure 0 0 0) 0x7fe8f807e4c0 con 0x7fe9080f73a0 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.906+0000 7fe8ff7fe700 10 client.27567.objecter ms_dispatch 0x7fe908001550 osd_map(508..508 src has 449..508) v4 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.906+0000 7fe8ff7fe700 3 client.27567.objecter handle_osd_map got epochs [508,508] > 0 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.906+0000 7fe8ff7fe700 3 client.27567.objecter handle_osd_map decoding full epoch 508 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.906+0000 7fe90db5f700 1 --2- 172.21.15.162:0/1957566073 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe8f4062c90 0x7fe8f4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.906+0000 7fe8ff7fe700 20 client.27567.objecter dump_active .. 0 homeless 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.906+0000 7fe8ff7fe700 10 client.27567.objecter ms_handle_connect 0x7fe8f4062c90 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.906+0000 7fe90db5f700 1 --2- 172.21.15.162:0/1957566073 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe8f4062c90 0x7fe8f4065150 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7fe9081089d0 tx=0x7fe90400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:42.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:42.910+0000 7fe8ff7fe700 1 -- 172.21.15.162:0/1957566073 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe8f80519a0 con 0x7fe9080f73a0 2024-02-13T00:48:42.947 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:42.967 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:42.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:42.990 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:48:42.990 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:48:43.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:43.098 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:43.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.104+0000 7fe9105c4700 1 -- 172.21.15.162:0/1957566073 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fe90805fb80 con 0x7fe9080f73a0 2024-02-13T00:48:43.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.518+0000 7fe8ff7fe700 1 -- 172.21.15.162:0/1957566073 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v509) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe8f8051590 con 0x7fe9080f73a0 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 10 librados: watch_flush enter 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 10 librados: watch_flush exit 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 20 client.27567.objecter shutdown clearing up homeless session... 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 10 client.27567.objecter successfully canceled tick 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 -- 172.21.15.162:0/1957566073 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe8f4062c90 msgr2=0x7fe8f4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 --2- 172.21.15.162:0/1957566073 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe8f4062c90 0x7fe8f4065150 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7fe9081089d0 tx=0x7fe90400b040).stop 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 -- 172.21.15.162:0/1957566073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9080f73a0 msgr2=0x7fe90810d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 --2- 172.21.15.162:0/1957566073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9080f73a0 0x7fe90810d090 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 rx=0x7fe8f80044e0 tx=0x7fe8f8007f90).stop 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 -- 172.21.15.162:0/1957566073 shutdown_connections 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 --2- 172.21.15.162:0/1957566073 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe8f4062c90 0x7fe8f4065150 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 --2- 172.21.15.162:0/1957566073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9080fd490 0x7fe908107230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 --2- 172.21.15.162:0/1957566073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9080f73a0 0x7fe90810d090 secure :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 rx=0x7fe8f80044e0 tx=0x7fe8f8007f90).stop 2024-02-13T00:48:43.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 --2- 172.21.15.162:0/1957566073 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9080fcb30 0x7fe90810d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:43.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 -- 172.21.15.162:0/1957566073 >> 172.21.15.162:0/1957566073 conn(0x7fe90806eb90 msgr2=0x7fe908070070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:43.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 -- 172.21.15.162:0/1957566073 shutdown_connections 2024-02-13T00:48:43.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 -- 172.21.15.162:0/1957566073 wait complete. 2024-02-13T00:48:43.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:43.523+0000 7fe9105c4700 1 librados: shutdown 2024-02-13T00:48:43.528 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:48:43.667 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:42.463027+0000 mgr.y (mgr.14544) 452 : cluster [DBG] pgmap v761: 236 pgs: 32 unknown, 1 clean+premerge+peered, 203 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:48:43.667 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:42.517415+0000 mon.a (mon.0) 2763 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:48:43.667 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:42.517511+0000 mon.a (mon.0) 2764 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-13T00:48:43.667 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:42.517571+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e508: 8 total, 8 up, 4 in 2024-02-13T00:48:43.667 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:42.519354+0000 mon.b (mon.1) 634 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:43.667 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:42.519939+0000 mon.a (mon.0) 2766 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:43.667 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:43.105092+0000 mon.a (mon.0) 2767 : audit [INF] from='client.? 172.21.15.162:0/1957566073' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:43.686 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-13T00:48:43.825 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:42.463027+0000 mgr.y (mgr.14544) 452 : cluster [DBG] pgmap v761: 236 pgs: 32 unknown, 1 clean+premerge+peered, 203 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:48:43.825 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:42.517415+0000 mon.a (mon.0) 2763 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:48:43.825 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:42.517511+0000 mon.a (mon.0) 2764 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-13T00:48:43.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:42.517571+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e508: 8 total, 8 up, 4 in 2024-02-13T00:48:43.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:42.519354+0000 mon.b (mon.1) 634 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:43.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:42.519939+0000 mon.a (mon.0) 2766 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:43.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:43.105092+0000 mon.a (mon.0) 2767 : audit [INF] from='client.? 172.21.15.162:0/1957566073' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:43.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:42.463027+0000 mgr.y (mgr.14544) 452 : cluster [DBG] pgmap v761: 236 pgs: 32 unknown, 1 clean+premerge+peered, 203 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:48:43.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:42.517415+0000 mon.a (mon.0) 2763 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:48:43.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:42.517511+0000 mon.a (mon.0) 2764 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-13T00:48:43.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:42.517571+0000 mon.a (mon.0) 2765 : cluster [DBG] osdmap e508: 8 total, 8 up, 4 in 2024-02-13T00:48:43.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:42.519354+0000 mon.b (mon.1) 634 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:43.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:42.519939+0000 mon.a (mon.0) 2766 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:43.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:43.105092+0000 mon.a (mon.0) 2767 : audit [INF] from='client.? 172.21.15.162:0/1957566073' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:48:43.951 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:48:44.031 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:44.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:44.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:44.162 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:44.214 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:44.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:44.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:44.345 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:44.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.1 reset_purged_snaps_last 2024-02-13T00:48:44.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:43.513466+0000 mon.a (mon.0) 2768 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:44.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:43.517435+0000 mon.a (mon.0) 2769 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:48:44.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:43.517566+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? 172.21.15.162:0/1957566073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:48:44.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:43.517619+0000 mon.a (mon.0) 2771 : cluster [DBG] osdmap e509: 8 total, 8 up, 4 in 2024-02-13T00:48:44.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:43.519501+0000 mon.b (mon.1) 635 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:44.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:43.521266+0000 osd.3 (osd.3) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:44.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:43.521540+0000 osd.3 (osd.3) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:44.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:43.524422+0000 mon.a (mon.0) 2772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:44.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:43.513466+0000 mon.a (mon.0) 2768 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:44.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:43.517435+0000 mon.a (mon.0) 2769 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:48:44.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:43.517566+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? 172.21.15.162:0/1957566073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:48:44.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:43.517619+0000 mon.a (mon.0) 2771 : cluster [DBG] osdmap e509: 8 total, 8 up, 4 in 2024-02-13T00:48:44.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:43.519501+0000 mon.b (mon.1) 635 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:44.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:43.521266+0000 osd.3 (osd.3) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:44.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:43.521540+0000 osd.3 (osd.3) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:44.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:43.524422+0000 mon.a (mon.0) 2772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:43.513466+0000 mon.a (mon.0) 2768 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:43.517435+0000 mon.a (mon.0) 2769 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_tier","val": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:48:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:43.517566+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? 172.21.15.162:0/1957566073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:48:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:43.517619+0000 mon.a (mon.0) 2771 : cluster [DBG] osdmap e509: 8 total, 8 up, 4 in 2024-02-13T00:48:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:43.519501+0000 mon.b (mon.1) 635 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:43.521266+0000 osd.3 (osd.3) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:48:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:43.521540+0000 osd.3 (osd.3) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:48:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:43.524422+0000 mon.a (mon.0) 2772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:45.093 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:45.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:45.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:45.225 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:45.351 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:45.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:45.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:45.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.478+0000 7f1e279d4700 1 -- 172.21.15.162:0/3244412122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e200fd460 msgr2=0x7f1e200f6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.478+0000 7f1e279d4700 1 --2- 172.21.15.162:0/3244412122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e200fd460 0x7f1e200f6180 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 rx=0x7f1e10004870 tx=0x7f1e10009cb0).stop 2024-02-13T00:48:45.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.478+0000 7f1e279d4700 1 -- 172.21.15.162:0/3244412122 shutdown_connections 2024-02-13T00:48:45.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.478+0000 7f1e279d4700 1 --2- 172.21.15.162:0/3244412122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e200f6860 0x7f1e200fa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.478+0000 7f1e279d4700 1 --2- 172.21.15.162:0/3244412122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e200fcb00 0x7f1e200fcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.478+0000 7f1e279d4700 1 --2- 172.21.15.162:0/3244412122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e200fd460 0x7f1e200f6180 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.478+0000 7f1e279d4700 1 -- 172.21.15.162:0/3244412122 >> 172.21.15.162:0/3244412122 conn(0x7f1e200700d0 msgr2=0x7f1e200f4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:45.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.478+0000 7f1e279d4700 1 -- 172.21.15.162:0/3244412122 shutdown_connections 2024-02-13T00:48:45.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.478+0000 7f1e279d4700 1 -- 172.21.15.162:0/3244412122 wait complete. 2024-02-13T00:48:45.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.479+0000 7f1e279d4700 1 librados: starting msgr at 2024-02-13T00:48:45.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.479+0000 7f1e279d4700 1 librados: starting objecter 2024-02-13T00:48:45.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.479+0000 7f1e279d4700 1 Processor -- start 2024-02-13T00:48:45.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.479+0000 7f1e279d4700 1 -- start start 2024-02-13T00:48:45.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.479+0000 7f1e279d4700 1 librados: setting wanted keys 2024-02-13T00:48:45.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.479+0000 7f1e279d4700 1 librados: calling monclient init 2024-02-13T00:48:45.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.479+0000 7f1e279d4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e200f6860 0x7f1e2006a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:45.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.479+0000 7f1e279d4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e200fcb00 0x7f1e2006a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:45.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.479+0000 7f1e279d4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e200fd460 0x7f1e20063320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:45.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.479+0000 7f1e279d4700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e20063810 con 0x7f1e200f6860 2024-02-13T00:48:45.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.479+0000 7f1e279d4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e20063950 con 0x7f1e200fcb00 2024-02-13T00:48:45.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.479+0000 7f1e279d4700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e20063b10 con 0x7f1e200fd460 2024-02-13T00:48:45.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e25770700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e200f6860 0x7f1e2006a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:45.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e25770700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e200f6860 0x7f1e2006a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35488/0 (socket says 172.21.15.162:35488) 2024-02-13T00:48:45.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e25770700 1 -- 172.21.15.162:0/1476601802 learned_addr learned my addr 172.21.15.162:0/1476601802 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:45.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e25f71700 1 --2- 172.21.15.162:0/1476601802 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e200fd460 0x7f1e20063320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:45.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e24f6f700 1 --2- 172.21.15.162:0/1476601802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e200fcb00 0x7f1e2006a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:45.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e25770700 1 -- 172.21.15.162:0/1476601802 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e200fd460 msgr2=0x7f1e20063320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e25770700 1 --2- 172.21.15.162:0/1476601802 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e200fd460 0x7f1e20063320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e25770700 1 -- 172.21.15.162:0/1476601802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e200fcb00 msgr2=0x7f1e2006a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e25770700 1 --2- 172.21.15.162:0/1476601802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e200fcb00 0x7f1e2006a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e25770700 1 -- 172.21.15.162:0/1476601802 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e10005040 con 0x7f1e200f6860 2024-02-13T00:48:45.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e24f6f700 1 --2- 172.21.15.162:0/1476601802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e200fcb00 0x7f1e2006a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:48:45.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e25f71700 1 --2- 172.21.15.162:0/1476601802 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e200fd460 0x7f1e20063320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:45.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e25770700 1 --2- 172.21.15.162:0/1476601802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e200f6860 0x7f1e2006a200 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 rx=0x7f1e1c00e970 tx=0x7f1e1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:45.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e167fc700 10 client.?.objecter ms_handle_connect 0x7f1e200f6860 2024-02-13T00:48:45.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.480+0000 7f1e167fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:45.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.481+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c00ee60 con 0x7f1e200f6860 2024-02-13T00:48:45.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.481+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e20063df0 con 0x7f1e200f6860 2024-02-13T00:48:45.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.481+0000 7f1e279d4700 10 client.27573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:45.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.481+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e20067db0 con 0x7f1e200f6860 2024-02-13T00:48:45.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.481+0000 7f1e279d4700 1 librados: init done 2024-02-13T00:48:45.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.482+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e1c00f040 con 0x7f1e200f6860 2024-02-13T00:48:45.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.482+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c007ea0 con 0x7f1e200f6860 2024-02-13T00:48:45.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.482+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1e2005f960 con 0x7f1e200f6860 2024-02-13T00:48:45.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.483+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1e1c00cb10 con 0x7f1e200f6860 2024-02-13T00:48:45.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.483+0000 7f1e167fc700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1e0c062c90 0x7f1e0c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:45.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.483+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(510..510 src has 449..510) v4 ==== 8134+0+0 (secure 0 0 0) 0x7f1e1c07dc90 con 0x7f1e200f6860 2024-02-13T00:48:45.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.483+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 osd_map(510..510 src has 449..510) v4 2024-02-13T00:48:45.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.483+0000 7f1e167fc700 3 client.27573.objecter handle_osd_map got epochs [510,510] > 0 2024-02-13T00:48:45.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.483+0000 7f1e167fc700 3 client.27573.objecter handle_osd_map decoding full epoch 510 2024-02-13T00:48:45.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.483+0000 7f1e167fc700 20 client.27573.objecter dump_active .. 0 homeless 2024-02-13T00:48:45.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v510) v1 ==== 54+0+16 (secure 0 0 0) 0x7f1e1c0512a0 con 0x7f1e200f6860 2024-02-13T00:48:45.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e279d4700 10 client.27573.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:45.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 1 2024-02-13T00:48:45.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6801/2103743338 conn(0x7f1e20068530 0x7f1e201060c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:45.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e200475e0 osd=0 3 2024-02-13T00:48:45.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e200475e0 osd=0 3 2024-02-13T00:48:45.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e200475e0 osd=0 3 2024-02-13T00:48:45.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 1 2024-02-13T00:48:45.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e200475e0 osd=0 3 2024-02-13T00:48:45.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 0 1 2024-02-13T00:48:45.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e200475e0 osd=0 4 2024-02-13T00:48:45.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e279d4700 10 client.27573.objecter _send_command 1 2024-02-13T00:48:45.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.484+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e2010d8c0 con 0x7f1e20068530 2024-02-13T00:48:45.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.485+0000 7f1e24f6f700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1e0c062c90 0x7f1e0c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:45.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.485+0000 7f1e25f71700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6801/2103743338 conn(0x7f1e20068530 0x7f1e201060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:45.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.485+0000 7f1e167fc700 10 client.27573.objecter ms_handle_connect 0x7f1e20068530 2024-02-13T00:48:45.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.485+0000 7f1e25f71700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6801/2103743338 conn(0x7f1e20068530 0x7f1e201060c0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:45.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.486+0000 7f1e167fc700 10 client.27573.objecter ms_handle_connect 0x7f1e0c062c90 2024-02-13T00:48:45.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.486+0000 7f1e24f6f700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1e0c062c90 0x7f1e0c065150 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 rx=0x7f1e10004ee0 tx=0x7f1e10009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:45.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.486+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1e2010d8c0 con 0x7f1e20068530 2024-02-13T00:48:45.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.486+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:48:45.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.486+0000 7f1e167fc700 10 client.27573.objecter _finish_command 1 = system:0 2024-02-13T00:48:45.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.486+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e200475e0 osd=0 4 2024-02-13T00:48:45.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.486+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 0 1 2024-02-13T00:48:45.493 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:45.516 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:45.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.514+0000 7f1e279d4700 10 client.27573.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:45.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.514+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 2 2024-02-13T00:48:45.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.514+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e200475e0 osd=0 3 2024-02-13T00:48:45.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.514+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e200475e0 osd=0 3 2024-02-13T00:48:45.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.514+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e200475e0 osd=0 3 2024-02-13T00:48:45.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.514+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 2 2024-02-13T00:48:45.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.514+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e200475e0 osd=0 3 2024-02-13T00:48:45.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.514+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 0 2 2024-02-13T00:48:45.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.514+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e200475e0 osd=0 4 2024-02-13T00:48:45.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.514+0000 7f1e279d4700 10 client.27573.objecter _send_command 2 2024-02-13T00:48:45.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.514+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1e20068b50 con 0x7f1e20068530 2024-02-13T00:48:45.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.515+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1e20068b50 con 0x7f1e20068530 2024-02-13T00:48:45.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.515+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:45.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.515+0000 7f1e167fc700 10 client.27573.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.515+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e200475e0 osd=0 4 2024-02-13T00:48:45.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.515+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 0 2 2024-02-13T00:48:45.521 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:48:45.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e279d4700 10 client.27573.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:45.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 3 2024-02-13T00:48:45.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6805/150599780 conn(0x7f1e20027410 0x7f1e200277f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:45.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20106d80 osd=1 3 2024-02-13T00:48:45.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20106d80 osd=1 3 2024-02-13T00:48:45.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20106d80 osd=1 3 2024-02-13T00:48:45.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 3 2024-02-13T00:48:45.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e20106d80 osd=1 3 2024-02-13T00:48:45.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 1 3 2024-02-13T00:48:45.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20106d80 osd=1 4 2024-02-13T00:48:45.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e279d4700 10 client.27573.objecter _send_command 3 2024-02-13T00:48:45.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e20106ef0 con 0x7f1e20027410 2024-02-13T00:48:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:45.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:45.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e25770700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6805/150599780 conn(0x7f1e20027410 0x7f1e200277f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:45.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e167fc700 10 client.27573.objecter ms_handle_connect 0x7f1e20027410 2024-02-13T00:48:45.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.516+0000 7f1e25770700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6805/150599780 conn(0x7f1e20027410 0x7f1e200277f0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:45.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.517+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1e20106ef0 con 0x7f1e20027410 2024-02-13T00:48:45.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.517+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:48:45.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.517+0000 7f1e167fc700 10 client.27573.objecter _finish_command 3 = system:0 2024-02-13T00:48:45.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.517+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e20106d80 osd=1 4 2024-02-13T00:48:45.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.517+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 1 3 2024-02-13T00:48:45.528 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:45.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.532+0000 7f1e279d4700 10 client.27573.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:45.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.532+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 4 2024-02-13T00:48:45.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.532+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20106d80 osd=1 3 2024-02-13T00:48:45.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.532+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20106d80 osd=1 3 2024-02-13T00:48:45.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.532+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20106d80 osd=1 3 2024-02-13T00:48:45.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.532+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 4 2024-02-13T00:48:45.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.532+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e20106d80 osd=1 3 2024-02-13T00:48:45.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.532+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 1 4 2024-02-13T00:48:45.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.532+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20106d80 osd=1 4 2024-02-13T00:48:45.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.532+0000 7f1e279d4700 10 client.27573.objecter _send_command 4 2024-02-13T00:48:45.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.532+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1e200284b0 con 0x7f1e20027410 2024-02-13T00:48:45.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1e200284b0 con 0x7f1e20027410 2024-02-13T00:48:45.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:45.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e167fc700 10 client.27573.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e20106d80 osd=1 4 2024-02-13T00:48:45.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 1 4 2024-02-13T00:48:45.539 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:48:45.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e279d4700 10 client.27573.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:45.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 5 2024-02-13T00:48:45.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6809/3689719376 conn(0x7f1e20106960 0x7f1e2012b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e201067f0 osd=2 3 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e201067f0 osd=2 3 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e201067f0 osd=2 3 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 5 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e201067f0 osd=2 3 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 2 5 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e201067f0 osd=2 4 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e279d4700 10 client.27573.objecter _send_command 5 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e2012e0b0 con 0x7f1e20106960 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.535+0000 7f1e24f6f700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6809/3689719376 conn(0x7f1e20106960 0x7f1e2012b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.536+0000 7f1e167fc700 10 client.27573.objecter ms_handle_connect 0x7f1e20106960 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.536+0000 7f1e24f6f700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6809/3689719376 conn(0x7f1e20106960 0x7f1e2012b460 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.536+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1e2012e0b0 con 0x7f1e20106960 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.536+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:48:45.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.536+0000 7f1e167fc700 10 client.27573.objecter _finish_command 5 = system:0 2024-02-13T00:48:45.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.536+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e201067f0 osd=2 4 2024-02-13T00:48:45.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.536+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 2 5 2024-02-13T00:48:45.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.552+0000 7f1e279d4700 10 client.27573.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:45.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.552+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 6 2024-02-13T00:48:45.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.552+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e201067f0 osd=2 3 2024-02-13T00:48:45.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.552+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e201067f0 osd=2 3 2024-02-13T00:48:45.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.552+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e201067f0 osd=2 3 2024-02-13T00:48:45.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.552+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 6 2024-02-13T00:48:45.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.552+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e201067f0 osd=2 3 2024-02-13T00:48:45.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.552+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 2 6 2024-02-13T00:48:45.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.552+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e201067f0 osd=2 4 2024-02-13T00:48:45.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.552+0000 7f1e279d4700 10 client.27573.objecter _send_command 6 2024-02-13T00:48:45.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.552+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1e20027f20 con 0x7f1e20106960 2024-02-13T00:48:45.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1e20027f20 con 0x7f1e20106960 2024-02-13T00:48:45.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:45.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e167fc700 10 client.27573.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e201067f0 osd=2 4 2024-02-13T00:48:45.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 2 6 2024-02-13T00:48:45.558 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:48:45.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e279d4700 10 client.27573.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:45.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 7 2024-02-13T00:48:45.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6813/604971011 conn(0x7f1e20059ea0 0x7f1e2012cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:45.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20028330 osd=3 3 2024-02-13T00:48:45.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20028330 osd=3 3 2024-02-13T00:48:45.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20028330 osd=3 3 2024-02-13T00:48:45.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 7 2024-02-13T00:48:45.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e20028330 osd=3 3 2024-02-13T00:48:45.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 3 7 2024-02-13T00:48:45.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20028330 osd=3 4 2024-02-13T00:48:45.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e279d4700 10 client.27573.objecter _send_command 7 2024-02-13T00:48:45.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e2012d080 con 0x7f1e20059ea0 2024-02-13T00:48:45.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.554+0000 7f1e25f71700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6813/604971011 conn(0x7f1e20059ea0 0x7f1e2012cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:45.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.555+0000 7f1e167fc700 10 client.27573.objecter ms_handle_connect 0x7f1e20059ea0 2024-02-13T00:48:45.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.555+0000 7f1e25f71700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6813/604971011 conn(0x7f1e20059ea0 0x7f1e2012cad0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:45.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.555+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1e2012d080 con 0x7f1e20059ea0 2024-02-13T00:48:45.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.555+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:48:45.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.555+0000 7f1e167fc700 10 client.27573.objecter _finish_command 7 = system:0 2024-02-13T00:48:45.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.555+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e20028330 osd=3 4 2024-02-13T00:48:45.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.555+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 3 7 2024-02-13T00:48:45.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.572+0000 7f1e279d4700 10 client.27573.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:45.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.572+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 8 2024-02-13T00:48:45.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.573+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20028330 osd=3 3 2024-02-13T00:48:45.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.573+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20028330 osd=3 3 2024-02-13T00:48:45.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.573+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20028330 osd=3 3 2024-02-13T00:48:45.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.573+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 8 2024-02-13T00:48:45.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.573+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e20028330 osd=3 3 2024-02-13T00:48:45.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.573+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 3 8 2024-02-13T00:48:45.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.573+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20028330 osd=3 4 2024-02-13T00:48:45.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.573+0000 7f1e279d4700 10 client.27573.objecter _send_command 8 2024-02-13T00:48:45.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.573+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1e20059400 con 0x7f1e20059ea0 2024-02-13T00:48:45.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1e20059400 con 0x7f1e20059ea0 2024-02-13T00:48:45.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:45.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e167fc700 10 client.27573.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e20028330 osd=3 4 2024-02-13T00:48:45.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 3 8 2024-02-13T00:48:45.578 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e279d4700 10 client.27573.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 9 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6800/3344263388 conn(0x7f1e20130230 0x7f1e20132620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e200595d0 osd=4 3 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e200595d0 osd=4 3 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e200595d0 osd=4 3 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 9 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e200595d0 osd=4 3 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 4 9 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e200595d0 osd=4 4 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e279d4700 10 client.27573.objecter _send_command 9 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.574+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e20059740 con 0x7f1e20130230 2024-02-13T00:48:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.575+0000 7f1e25770700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6800/3344263388 conn(0x7f1e20130230 0x7f1e20132620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:45.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.576+0000 7f1e167fc700 10 client.27573.objecter ms_handle_connect 0x7f1e20130230 2024-02-13T00:48:45.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.576+0000 7f1e25770700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6800/3344263388 conn(0x7f1e20130230 0x7f1e20132620 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:45.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.576+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1e20059740 con 0x7f1e20130230 2024-02-13T00:48:45.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.576+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:48:45.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.576+0000 7f1e167fc700 10 client.27573.objecter _finish_command 9 = system:0 2024-02-13T00:48:45.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.576+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e200595d0 osd=4 4 2024-02-13T00:48:45.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.576+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 4 9 2024-02-13T00:48:45.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.596+0000 7f1e279d4700 10 client.27573.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:45.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.597+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 10 2024-02-13T00:48:45.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.597+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e200595d0 osd=4 3 2024-02-13T00:48:45.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.597+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e200595d0 osd=4 3 2024-02-13T00:48:45.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.597+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e200595d0 osd=4 3 2024-02-13T00:48:45.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.597+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 10 2024-02-13T00:48:45.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.597+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e200595d0 osd=4 3 2024-02-13T00:48:45.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.597+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 4 10 2024-02-13T00:48:45.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.597+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e200595d0 osd=4 4 2024-02-13T00:48:45.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.597+0000 7f1e279d4700 10 client.27573.objecter _send_command 10 2024-02-13T00:48:45.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.597+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1e20059740 con 0x7f1e20130230 2024-02-13T00:48:45.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1e20059740 con 0x7f1e20130230 2024-02-13T00:48:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e167fc700 10 client.27573.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e200595d0 osd=4 4 2024-02-13T00:48:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 4 10 2024-02-13T00:48:45.602 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:48:45.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e279d4700 10 client.27573.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:45.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 11 2024-02-13T00:48:45.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6804/2277917372 conn(0x7f1e20133020 0x7f1e20135410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:45.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20059740 osd=5 3 2024-02-13T00:48:45.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20059740 osd=5 3 2024-02-13T00:48:45.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20059740 osd=5 3 2024-02-13T00:48:45.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 11 2024-02-13T00:48:45.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e20059740 osd=5 3 2024-02-13T00:48:45.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 5 11 2024-02-13T00:48:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20059740 osd=5 4 2024-02-13T00:48:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e279d4700 10 client.27573.objecter _send_command 11 2024-02-13T00:48:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.598+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e201381c0 con 0x7f1e20133020 2024-02-13T00:48:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.599+0000 7f1e24f6f700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6804/2277917372 conn(0x7f1e20133020 0x7f1e20135410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.599+0000 7f1e24f6f700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6804/2277917372 conn(0x7f1e20133020 0x7f1e20135410 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.599+0000 7f1e167fc700 10 client.27573.objecter ms_handle_connect 0x7f1e20133020 2024-02-13T00:48:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.600+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1e201381c0 con 0x7f1e20133020 2024-02-13T00:48:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.600+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:48:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.600+0000 7f1e167fc700 10 client.27573.objecter _finish_command 11 = system:0 2024-02-13T00:48:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.600+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e20059740 osd=5 4 2024-02-13T00:48:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.600+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 5 11 2024-02-13T00:48:45.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.614+0000 7f1e279d4700 10 client.27573.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:45.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.614+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 12 2024-02-13T00:48:45.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.614+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20059740 osd=5 3 2024-02-13T00:48:45.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.614+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20059740 osd=5 3 2024-02-13T00:48:45.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.614+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20059740 osd=5 3 2024-02-13T00:48:45.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.614+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 12 2024-02-13T00:48:45.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.614+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e20059740 osd=5 3 2024-02-13T00:48:45.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.614+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 5 12 2024-02-13T00:48:45.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.614+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20059740 osd=5 4 2024-02-13T00:48:45.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.614+0000 7f1e279d4700 10 client.27573.objecter _send_command 12 2024-02-13T00:48:45.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.614+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1e201381c0 con 0x7f1e20133020 2024-02-13T00:48:45.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.615+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1e201381c0 con 0x7f1e20133020 2024-02-13T00:48:45.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.615+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:45.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.615+0000 7f1e167fc700 10 client.27573.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.615+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e20059740 osd=5 4 2024-02-13T00:48:45.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.615+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 5 12 2024-02-13T00:48:45.619 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:48:45.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e279d4700 10 client.27573.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:45.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 13 2024-02-13T00:48:45.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6808/1506640341 conn(0x7f1e201374c0 0x7f1e20137940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:45.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e201381c0 osd=6 3 2024-02-13T00:48:45.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e201381c0 osd=6 3 2024-02-13T00:48:45.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e201381c0 osd=6 3 2024-02-13T00:48:45.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 13 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e201381c0 osd=6 3 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 6 13 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e201381c0 osd=6 4 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e279d4700 10 client.27573.objecter _send_command 13 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e20137fc0 con 0x7f1e201374c0 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e25f71700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6808/1506640341 conn(0x7f1e201374c0 0x7f1e20137940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e25f71700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6808/1506640341 conn(0x7f1e201374c0 0x7f1e20137940 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.616+0000 7f1e167fc700 10 client.27573.objecter ms_handle_connect 0x7f1e201374c0 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.617+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1e20137fc0 con 0x7f1e201374c0 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.617+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.617+0000 7f1e167fc700 10 client.27573.objecter _finish_command 13 = system:0 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.617+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e201381c0 osd=6 4 2024-02-13T00:48:45.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.617+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 6 13 2024-02-13T00:48:45.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:45.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.631+0000 7f1e279d4700 10 client.27573.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:45.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.631+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 14 2024-02-13T00:48:45.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.631+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e201381c0 osd=6 3 2024-02-13T00:48:45.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.631+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e201381c0 osd=6 3 2024-02-13T00:48:45.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.631+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e201381c0 osd=6 3 2024-02-13T00:48:45.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.631+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 14 2024-02-13T00:48:45.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.631+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e201381c0 osd=6 3 2024-02-13T00:48:45.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.631+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 6 14 2024-02-13T00:48:45.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.631+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e201381c0 osd=6 4 2024-02-13T00:48:45.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.631+0000 7f1e279d4700 10 client.27573.objecter _send_command 14 2024-02-13T00:48:45.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.631+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1e20137f80 con 0x7f1e201374c0 2024-02-13T00:48:45.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1e20137f80 con 0x7f1e201374c0 2024-02-13T00:48:45.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:45.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e167fc700 10 client.27573.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e201381c0 osd=6 4 2024-02-13T00:48:45.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 6 14 2024-02-13T00:48:45.637 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:48:45.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e279d4700 10 client.27573.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:45.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 15 2024-02-13T00:48:45.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6812/1964192772 conn(0x7f1e2013a900 0x7f1e2013ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:45.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20137f80 osd=7 3 2024-02-13T00:48:45.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20137f80 osd=7 3 2024-02-13T00:48:45.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20137f80 osd=7 3 2024-02-13T00:48:45.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 15 2024-02-13T00:48:45.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e20137f80 osd=7 3 2024-02-13T00:48:45.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 7 15 2024-02-13T00:48:45.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20137f80 osd=7 4 2024-02-13T00:48:45.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e279d4700 10 client.27573.objecter _send_command 15 2024-02-13T00:48:45.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e2013b3c0 con 0x7f1e2013a900 2024-02-13T00:48:45.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e25770700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6812/1964192772 conn(0x7f1e2013a900 0x7f1e2013ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:45.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e167fc700 10 client.27573.objecter ms_handle_connect 0x7f1e2013a900 2024-02-13T00:48:45.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.633+0000 7f1e25770700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6812/1964192772 conn(0x7f1e2013a900 0x7f1e2013ad20 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:45.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.634+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1e2013b3c0 con 0x7f1e2013a900 2024-02-13T00:48:45.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.634+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:48:45.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.634+0000 7f1e167fc700 10 client.27573.objecter _finish_command 15 = system:0 2024-02-13T00:48:45.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.634+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e20137f80 osd=7 4 2024-02-13T00:48:45.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.634+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 7 15 2024-02-13T00:48:45.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.649+0000 7f1e279d4700 10 client.27573.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:48:45.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.649+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign -1 16 2024-02-13T00:48:45.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.649+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20137f80 osd=7 3 2024-02-13T00:48:45.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.649+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20137f80 osd=7 3 2024-02-13T00:48:45.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.649+0000 7f1e279d4700 20 client.27573.objecter _get_session s=0x7f1e20137f80 osd=7 3 2024-02-13T00:48:45.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.649+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_remove -1 16 2024-02-13T00:48:45.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.649+0000 7f1e279d4700 20 client.27573.objecter get_session s=0x7f1e20137f80 osd=7 3 2024-02-13T00:48:45.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.649+0000 7f1e279d4700 15 client.27573.objecter _session_command_op_assign 7 16 2024-02-13T00:48:45.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.649+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20137f80 osd=7 4 2024-02-13T00:48:45.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.649+0000 7f1e279d4700 10 client.27573.objecter _send_command 16 2024-02-13T00:48:45.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.649+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1e2013b3c0 con 0x7f1e2013a900 2024-02-13T00:48:45.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.650+0000 7f1e167fc700 1 -- 172.21.15.162:0/1476601802 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1e2013b3c0 con 0x7f1e2013a900 2024-02-13T00:48:45.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.650+0000 7f1e167fc700 10 client.27573.objecter ms_dispatch 0x7f1e20001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:48:45.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.650+0000 7f1e167fc700 10 client.27573.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.650+0000 7f1e167fc700 20 client.27573.objecter put_session s=0x7f1e20137f80 osd=7 4 2024-02-13T00:48:45.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.650+0000 7f1e167fc700 15 client.27573.objecter _session_command_op_remove 7 16 2024-02-13T00:48:45.655 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:48:45.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 10 librados: watch_flush enter 2024-02-13T00:48:45.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 10 librados: watch_flush exit 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 10 client.27573.objecter close_session for osd.0 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6801/2103743338 conn(0x7f1e20068530 msgr2=0x7f1e201060c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6801/2103743338 conn(0x7f1e20068530 0x7f1e201060c0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e200475e0 osd=0 1 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 10 client.27573.objecter close_session for osd.1 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6805/150599780 conn(0x7f1e20027410 msgr2=0x7f1e200277f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6805/150599780 conn(0x7f1e20027410 0x7f1e200277f0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20106d80 osd=1 1 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 10 client.27573.objecter close_session for osd.2 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6809/3689719376 conn(0x7f1e20106960 msgr2=0x7f1e2012b460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6809/3689719376 conn(0x7f1e20106960 0x7f1e2012b460 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e201067f0 osd=2 1 2024-02-13T00:48:45.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 10 client.27573.objecter close_session for osd.3 2024-02-13T00:48:45.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6813/604971011 conn(0x7f1e20059ea0 msgr2=0x7f1e2012cad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6813/604971011 conn(0x7f1e20059ea0 0x7f1e2012cad0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20028330 osd=3 1 2024-02-13T00:48:45.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 10 client.27573.objecter close_session for osd.4 2024-02-13T00:48:45.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6800/3344263388 conn(0x7f1e20130230 msgr2=0x7f1e20132620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6800/3344263388 conn(0x7f1e20130230 0x7f1e20132620 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.657 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:45.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e200595d0 osd=4 1 2024-02-13T00:48:45.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 10 client.27573.objecter close_session for osd.5 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6804/2277917372 conn(0x7f1e20133020 msgr2=0x7f1e20135410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6804/2277917372 conn(0x7f1e20133020 0x7f1e20135410 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20059740 osd=5 1 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 10 client.27573.objecter close_session for osd.6 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6808/1506640341 conn(0x7f1e201374c0 msgr2=0x7f1e20137940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6808/1506640341 conn(0x7f1e201374c0 0x7f1e20137940 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e201381c0 osd=6 1 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 10 client.27573.objecter close_session for osd.7 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6812/1964192772 conn(0x7f1e2013a900 msgr2=0x7f1e2013ad20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.181:6812/1964192772 conn(0x7f1e2013a900 0x7f1e2013ad20 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 20 client.27573.objecter put_session s=0x7f1e20137f80 osd=7 1 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 20 client.27573.objecter shutdown clearing up homeless session... 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 10 client.27573.objecter successfully canceled tick 2024-02-13T00:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e25f71700 1 -- 172.21.15.162:0/1476601802 reap_dead start 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e25f71700 1 -- 172.21.15.162:0/1476601802 reap_dead start 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e25f71700 1 -- 172.21.15.162:0/1476601802 reap_dead start 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e25f71700 1 -- 172.21.15.162:0/1476601802 reap_dead start 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e25f71700 1 -- 172.21.15.162:0/1476601802 reap_dead start 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1e0c062c90 msgr2=0x7f1e0c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1e0c062c90 0x7f1e0c065150 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 rx=0x7f1e10004ee0 tx=0x7f1e10009cb0).stop 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e200f6860 msgr2=0x7f1e2006a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e200f6860 0x7f1e2006a200 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 rx=0x7f1e1c00e970 tx=0x7f1e1c00b040).stop 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 shutdown_connections 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e200f6860 0x7f1e2006a200 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 --2- 172.21.15.162:0/1476601802 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1e0c062c90 0x7f1e0c065150 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.651+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 >> 172.21.15.162:0/1476601802 conn(0x7f1e200700d0 msgr2=0x7f1e2006c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:45.660 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.652+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 shutdown_connections 2024-02-13T00:48:45.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.652+0000 7f1e279d4700 1 -- 172.21.15.162:0/1476601802 wait complete. 2024-02-13T00:48:45.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:45.652+0000 7f1e279d4700 1 librados: shutdown 2024-02-13T00:48:45.661 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.661 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.661 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.661 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.661 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.661 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.661 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.661 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:48:45.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:44.463500+0000 mgr.y (mgr.14544) 453 : cluster [DBG] pgmap v764: 236 pgs: 32 unknown, 1 clean+premerge+peered, 203 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:48:45.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:44.532364+0000 mon.a (mon.0) 2773 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:48:45.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:44.532456+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e510: 8 total, 8 up, 4 in 2024-02-13T00:48:45.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:44.534166+0000 mon.b (mon.1) 636 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:45.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:44.534534+0000 mon.a (mon.0) 2775 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:45.715 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:45.484627+0000 mon.a (mon.0) 2776 : audit [DBG] from='client.? 172.21.15.162:0/1476601802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:45.785 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:44.463500+0000 mgr.y (mgr.14544) 453 : cluster [DBG] pgmap v764: 236 pgs: 32 unknown, 1 clean+premerge+peered, 203 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:48:45.785 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:44.532364+0000 mon.a (mon.0) 2773 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:48:45.785 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:44.532456+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e510: 8 total, 8 up, 4 in 2024-02-13T00:48:45.785 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:44.534166+0000 mon.b (mon.1) 636 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:45.785 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:44.534534+0000 mon.a (mon.0) 2775 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:45.785 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:45.484627+0000 mon.a (mon.0) 2776 : audit [DBG] from='client.? 172.21.15.162:0/1476601802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:45.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:44.463500+0000 mgr.y (mgr.14544) 453 : cluster [DBG] pgmap v764: 236 pgs: 32 unknown, 1 clean+premerge+peered, 203 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:48:45.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:44.532364+0000 mon.a (mon.0) 2773 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:48:45.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:44.532456+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e510: 8 total, 8 up, 4 in 2024-02-13T00:48:45.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:44.534166+0000 mon.b (mon.1) 636 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:45.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:44.534534+0000 mon.a (mon.0) 2775 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:45.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:45.484627+0000 mon.a (mon.0) 2776 : audit [DBG] from='client.? 172.21.15.162:0/1476601802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:45.865 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:48:45.865 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:48:45.865 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:48:45.865 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:48:46.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:48:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:48:46.405 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:46.407 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:48:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:48:46] "GET /metrics HTTP/1.1" 200 188112 "" "Prometheus/2.33.4" 2024-02-13T00:48:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:46.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:46.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-13T00:48:46.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:46.539 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:46.779 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:48:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:45.578654+0000 mon.a (mon.0) 2777 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:45.578729+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e511: 8 total, 8 up, 4 in 2024-02-13T00:48:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:45.710944+0000 mgr.y (mgr.14544) 454 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:46.464256+0000 mon.c (mon.2) 409 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-13T00:48:46.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:46.465132+0000 mon.a (mon.0) 2779 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-13T00:48:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:45.578654+0000 mon.a (mon.0) 2777 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:45.578729+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e511: 8 total, 8 up, 4 in 2024-02-13T00:48:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:45.710944+0000 mgr.y (mgr.14544) 454 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:46.464256+0000 mon.c (mon.2) 409 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-13T00:48:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:46.465132+0000 mon.a (mon.0) 2779 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-13T00:48:46.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:45.578654+0000 mon.a (mon.0) 2777 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:46.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:45.578729+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e511: 8 total, 8 up, 4 in 2024-02-13T00:48:46.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:45.710944+0000 mgr.y (mgr.14544) 454 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:46.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:46.464256+0000 mon.c (mon.2) 409 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-13T00:48:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:46.465132+0000 mon.a (mon.0) 2779 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-13T00:48:47.196 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:48:47.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:47.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:47.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:47.522 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:47.541 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:47.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:47.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:47.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:47.676 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:47.695 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:47.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:47.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:47.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:46.463942+0000 mgr.y (mgr.14544) 455 : cluster [DBG] pgmap v767: 235 pgs: 1 peering, 234 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:46.581172+0000 mon.a (mon.0) 2780 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-13T00:48:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:46.581239+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e512: 8 total, 8 up, 4 in 2024-02-13T00:48:47.799 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:47.825 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:47.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.905+0000 7ff1eece1700 1 -- 172.21.15.162:0/958392498 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1e80fdc70 msgr2=0x7ff1e80fe050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:47.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.905+0000 7ff1eece1700 1 --2- 172.21.15.162:0/958392498 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1e80fdc70 0x7ff1e80fe050 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 rx=0x7ff1dc0047b0 tx=0x7ff1dc009cb0).stop 2024-02-13T00:48:47.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.905+0000 7ff1eece1700 1 -- 172.21.15.162:0/958392498 shutdown_connections 2024-02-13T00:48:47.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.905+0000 7ff1eece1700 1 --2- 172.21.15.162:0/958392498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1e80f6830 0x7ff1e8104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.905+0000 7ff1eece1700 1 --2- 172.21.15.162:0/958392498 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1e80fdc70 0x7ff1e80fe050 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.905+0000 7ff1eece1700 1 --2- 172.21.15.162:0/958392498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1e80fe5d0 0x7ff1e80f6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.905+0000 7ff1eece1700 1 -- 172.21.15.162:0/958392498 >> 172.21.15.162:0/958392498 conn(0x7ff1e8070060 msgr2=0x7ff1e80f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.905+0000 7ff1eece1700 1 -- 172.21.15.162:0/958392498 shutdown_connections 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.905+0000 7ff1eece1700 1 -- 172.21.15.162:0/958392498 wait complete. 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eece1700 1 librados: starting msgr at 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eece1700 1 librados: starting objecter 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eece1700 1 Processor -- start 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eece1700 1 -- start start 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eece1700 1 librados: setting wanted keys 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eece1700 1 librados: calling monclient init 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eece1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1e80f6830 0x7ff1e80f7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eece1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1e80fdc70 0x7ff1e80f7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eece1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1e80fe5d0 0x7ff1e80f7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eece1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e8116610 con 0x7ff1e80fe5d0 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eece1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e8116780 con 0x7ff1e80fdc70 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eece1700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e8116940 con 0x7ff1e80f6830 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eca7d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1e80f6830 0x7ff1e80f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eca7d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1e80f6830 0x7ff1e80f7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:42430/0 (socket says 172.21.15.162:42430) 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1eca7d700 1 -- 172.21.15.162:0/1518186778 learned_addr learned my addr 172.21.15.162:0/1518186778 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1ed27e700 1 --2- 172.21.15.162:0/1518186778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1e80fe5d0 0x7ff1e80f7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.906+0000 7ff1e7fff700 1 --2- 172.21.15.162:0/1518186778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1e80fdc70 0x7ff1e80f7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1eca7d700 1 -- 172.21.15.162:0/1518186778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1e80fdc70 msgr2=0x7ff1e80f7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1eca7d700 1 --2- 172.21.15.162:0/1518186778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1e80fdc70 0x7ff1e80f7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1eca7d700 1 -- 172.21.15.162:0/1518186778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1e80fe5d0 msgr2=0x7ff1e80f7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:47.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1eca7d700 1 --2- 172.21.15.162:0/1518186778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1e80fe5d0 0x7ff1e80f7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1eca7d700 1 -- 172.21.15.162:0/1518186778 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e00097c0 con 0x7ff1e80f6830 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1ed27e700 1 --2- 172.21.15.162:0/1518186778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1e80fe5d0 0x7ff1e80f7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1eca7d700 1 --2- 172.21.15.162:0/1518186778 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1e80f6830 0x7ff1e80f7420 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7ff1dc009b10 tx=0x7ff1dc009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1e5ffb700 10 client.?.objecter ms_handle_connect 0x7ff1e80f6830 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1e5ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1e5ffb700 1 -- 172.21.15.162:0/1518186778 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc018070 con 0x7ff1e80f6830 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1e5ffb700 1 -- 172.21.15.162:0/1518186778 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1dc013da0 con 0x7ff1e80f6830 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1eece1700 1 -- 172.21.15.162:0/1518186778 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1dc005040 con 0x7ff1e80f6830 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1e5ffb700 1 -- 172.21.15.162:0/1518186778 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc01dcc0 con 0x7ff1e80f6830 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1eece1700 10 client.37679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1eece1700 1 -- 172.21.15.162:0/1518186778 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e8116eb0 con 0x7ff1e80f6830 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.907+0000 7ff1eece1700 1 librados: init done 2024-02-13T00:48:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.908+0000 7ff1eece1700 10 client.37679.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.908+0000 7ff1eece1700 15 client.37679.objecter _session_command_op_assign -1 1 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.908+0000 7ff1eece1700 20 client.37679.objecter _get_session osd=-1 returning homeless 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.908+0000 7ff1eece1700 10 client.37679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.908+0000 7ff1eece1700 1 -- 172.21.15.162:0/1518186778 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff1e805f960 con 0x7ff1e80f6830 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.908+0000 7ff1e5ffb700 1 -- 172.21.15.162:0/1518186778 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff1dc009070 con 0x7ff1e80f6830 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 1 --2- 172.21.15.162:0/1518186778 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff1d8062c90 0x7ff1d8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 1 -- 172.21.15.162:0/1518186778 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(513..513 src has 449..513) v4 ==== 8150+0+0 (secure 0 0 0) 0x7ff1dc07e760 con 0x7ff1e80f6830 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 10 client.37679.objecter ms_dispatch 0x7ff1e8001550 osd_map(513..513 src has 449..513) v4 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 3 client.37679.objecter handle_osd_map got epochs [513,513] > 0 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 3 client.37679.objecter handle_osd_map decoding full epoch 513 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e7fff700 1 --2- 172.21.15.162:0/1518186778 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff1d8062c90 0x7ff1d8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 10 client.37679.objecter checking command 1 2024-02-13T00:48:47.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 1 --2- 172.21.15.162:0/1518186778 >> v2:172.21.15.162:6805/150599780 conn(0x7ff1d806fcb0 0x7ff1d8072150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 20 client.37679.objecter _get_session s=0x7ff1d806f600 osd=1 3 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 20 client.37679.objecter put_session s=0x7ff1d806f600 osd=1 3 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 15 client.37679.objecter _session_command_op_remove -1 1 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 20 client.37679.objecter _get_session s=0x7ff1d806f600 osd=1 3 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 20 client.37679.objecter get_session s=0x7ff1d806f600 osd=1 3 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 15 client.37679.objecter _session_command_op_assign 1 1 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 20 client.37679.objecter put_session s=0x7ff1d806f600 osd=1 4 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 10 client.37679.objecter _send_command 1 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 1 -- 172.21.15.162:0/1518186778 --> v2:172.21.15.162:6805/150599780 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1d8072790 con 0x7ff1d806fcb0 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 20 client.37679.objecter dump_active .. 0 homeless 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 1 -- 172.21.15.162:0/1518186778 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=513) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1dc080dd0 con 0x7ff1e80f6830 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1ed27e700 1 --2- 172.21.15.162:0/1518186778 >> v2:172.21.15.162:6805/150599780 conn(0x7ff1d806fcb0 0x7ff1d8072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 10 client.37679.objecter ms_handle_connect 0x7ff1d8062c90 2024-02-13T00:48:47.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e7fff700 1 --2- 172.21.15.162:0/1518186778 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff1d8062c90 0x7ff1d8065150 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7ff1e80f8bc0 tx=0x7ff1d400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:47.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1e5ffb700 10 client.37679.objecter ms_handle_connect 0x7ff1d806fcb0 2024-02-13T00:48:47.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.909+0000 7ff1ed27e700 1 --2- 172.21.15.162:0/1518186778 >> v2:172.21.15.162:6805/150599780 conn(0x7ff1d806fcb0 0x7ff1d8072150 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:47.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.910+0000 7ff1e5ffb700 1 -- 172.21.15.162:0/1518186778 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff1d8072790 con 0x7ff1d806fcb0 2024-02-13T00:48:47.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.910+0000 7ff1e5ffb700 10 client.37679.objecter ms_dispatch 0x7ff1e8001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:48:47.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.910+0000 7ff1e5ffb700 10 client.37679.objecter _finish_command 1 = system:0 2024-02-13T00:48:47.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.910+0000 7ff1e5ffb700 20 client.37679.objecter put_session s=0x7ff1d806f600 osd=1 4 2024-02-13T00:48:47.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.910+0000 7ff1e5ffb700 15 client.37679.objecter _session_command_op_remove 1 1 2024-02-13T00:48:47.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.929+0000 7ff1eece1700 10 client.37679.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-02-13T00:48:47.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.929+0000 7ff1eece1700 15 client.37679.objecter _session_command_op_assign -1 2 2024-02-13T00:48:47.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.929+0000 7ff1eece1700 20 client.37679.objecter _get_session s=0x7ff1d806f600 osd=1 3 2024-02-13T00:48:47.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.929+0000 7ff1eece1700 20 client.37679.objecter put_session s=0x7ff1d806f600 osd=1 3 2024-02-13T00:48:47.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.929+0000 7ff1eece1700 20 client.37679.objecter _get_session s=0x7ff1d806f600 osd=1 3 2024-02-13T00:48:47.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.929+0000 7ff1eece1700 15 client.37679.objecter _session_command_op_remove -1 2 2024-02-13T00:48:47.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.929+0000 7ff1eece1700 20 client.37679.objecter get_session s=0x7ff1d806f600 osd=1 3 2024-02-13T00:48:47.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.929+0000 7ff1eece1700 15 client.37679.objecter _session_command_op_assign 1 2 2024-02-13T00:48:47.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.929+0000 7ff1eece1700 20 client.37679.objecter put_session s=0x7ff1d806f600 osd=1 4 2024-02-13T00:48:47.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.929+0000 7ff1eece1700 10 client.37679.objecter _send_command 2 2024-02-13T00:48:47.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.929+0000 7ff1eece1700 1 -- 172.21.15.162:0/1518186778 --> v2:172.21.15.162:6805/150599780 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7ff1e80475e0 con 0x7ff1d806fcb0 2024-02-13T00:48:47.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1e5ffb700 1 -- 172.21.15.162:0/1518186778 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff1e80475e0 con 0x7ff1d806fcb0 2024-02-13T00:48:47.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1e5ffb700 10 client.37679.objecter ms_dispatch 0x7ff1e8001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:48:47.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1e5ffb700 10 client.37679.objecter _finish_command 2 = system:0 2024-02-13T00:48:47.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1e5ffb700 20 client.37679.objecter put_session s=0x7ff1d806f600 osd=1 4 2024-02-13T00:48:47.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1e5ffb700 15 client.37679.objecter _session_command_op_remove 1 2 2024-02-13T00:48:47.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 10 librados: watch_flush enter 2024-02-13T00:48:47.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 10 librados: watch_flush exit 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 10 client.37679.objecter close_session for osd.1 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 1 -- 172.21.15.162:0/1518186778 >> v2:172.21.15.162:6805/150599780 conn(0x7ff1d806fcb0 msgr2=0x7ff1d8072150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 1 --2- 172.21.15.162:0/1518186778 >> v2:172.21.15.162:6805/150599780 conn(0x7ff1d806fcb0 0x7ff1d8072150 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 20 client.37679.objecter put_session s=0x7ff1d806f600 osd=1 1 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 20 client.37679.objecter shutdown clearing up homeless session... 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 10 client.37679.objecter successfully canceled tick 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 1 -- 172.21.15.162:0/1518186778 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff1d8062c90 msgr2=0x7ff1d8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 1 --2- 172.21.15.162:0/1518186778 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff1d8062c90 0x7ff1d8065150 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7ff1e80f8bc0 tx=0x7ff1d400b040).stop 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 1 -- 172.21.15.162:0/1518186778 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1e80f6830 msgr2=0x7ff1e80f7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 1 --2- 172.21.15.162:0/1518186778 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1e80f6830 0x7ff1e80f7420 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7ff1dc009b10 tx=0x7ff1dc009cb0).stop 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1ed27e700 1 -- 172.21.15.162:0/1518186778 reap_dead start 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 1 -- 172.21.15.162:0/1518186778 shutdown_connections 2024-02-13T00:48:47.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 1 -- 172.21.15.162:0/1518186778 >> 172.21.15.162:0/1518186778 conn(0x7ff1e8070060 msgr2=0x7ff1e80fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:47.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 1 -- 172.21.15.162:0/1518186778 shutdown_connections 2024-02-13T00:48:47.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.930+0000 7ff1eece1700 1 -- 172.21.15.162:0/1518186778 wait complete. 2024-02-13T00:48:47.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:47.931+0000 7ff1eece1700 1 librados: shutdown 2024-02-13T00:48:47.955 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:47.960 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:46.463942+0000 mgr.y (mgr.14544) 455 : cluster [DBG] pgmap v767: 235 pgs: 1 peering, 234 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:47.960 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:46.581172+0000 mon.a (mon.0) 2780 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-13T00:48:47.960 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:46.581239+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e512: 8 total, 8 up, 4 in 2024-02-13T00:48:47.960 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:46.463942+0000 mgr.y (mgr.14544) 455 : cluster [DBG] pgmap v767: 235 pgs: 1 peering, 234 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:47.960 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:46.581172+0000 mon.a (mon.0) 2780 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-13T00:48:47.960 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:46.581239+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e512: 8 total, 8 up, 4 in 2024-02-13T00:48:47.973 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:47.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:47.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:48.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:48.105 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:48.407 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:48.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:48.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:48.484 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:48:48.484 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:48:48.484 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-02-13T00:48:48.485 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-02-13T00:48:48.485 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:48:48.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:48.542 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:48.715 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:47.582000+0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e513: 8 total, 8 up, 4 in 2024-02-13T00:48:48.715 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:47.621669+0000 mon.b (mon.1) 637 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:48.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:47.622078+0000 mon.a (mon.0) 2783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:48.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:47.622495+0000 mon.b (mon.1) 638 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-81"}]: dispatch 2024-02-13T00:48:48.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:47.622808+0000 mon.a (mon.0) 2784 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-81"}]: dispatch 2024-02-13T00:48:48.716 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:47.582000+0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e513: 8 total, 8 up, 4 in 2024-02-13T00:48:48.716 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:47.621669+0000 mon.b (mon.1) 637 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:48.716 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:47.622078+0000 mon.a (mon.0) 2783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:48.716 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:47.622495+0000 mon.b (mon.1) 638 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-81"}]: dispatch 2024-02-13T00:48:48.716 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:47.622808+0000 mon.a (mon.0) 2784 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-81"}]: dispatch 2024-02-13T00:48:48.741 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:48.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:48.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:48.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:48:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:48:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:48:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:47.582000+0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e513: 8 total, 8 up, 4 in 2024-02-13T00:48:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:47.621669+0000 mon.b (mon.1) 637 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:47.622078+0000 mon.a (mon.0) 2783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:47.622495+0000 mon.b (mon.1) 638 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-81"}]: dispatch 2024-02-13T00:48:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:47.622808+0000 mon.a (mon.0) 2784 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-81"}]: dispatch 2024-02-13T00:48:48.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:48.873 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:48.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.4 reset_purged_snaps_last 2024-02-13T00:48:48.997 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:48.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:49.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:49.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:49.127 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:49.163 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-13T00:48:49.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:48:49.428 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:48:49.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:48.464348+0000 mgr.y (mgr.14544) 456 : cluster [DBG] pgmap v770: 235 pgs: 1 peering, 234 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:49.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:48.597478+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e514: 8 total, 8 up, 4 in 2024-02-13T00:48:49.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:49.340265+0000 mon.a (mon.0) 2786 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:48:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:48.464348+0000 mgr.y (mgr.14544) 456 : cluster [DBG] pgmap v770: 235 pgs: 1 peering, 234 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:48.597478+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e514: 8 total, 8 up, 4 in 2024-02-13T00:48:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:49.340265+0000 mon.a (mon.0) 2786 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:48:49.876 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:49.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:49.980 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:50.001 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:48.464348+0000 mgr.y (mgr.14544) 456 : cluster [DBG] pgmap v770: 235 pgs: 1 peering, 234 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:50.001 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:48.597478+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e514: 8 total, 8 up, 4 in 2024-02-13T00:48:50.001 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:49.340265+0000 mon.a (mon.0) 2786 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:48:50.009 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:50.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.337+0000 7f75cd42b700 1 -- 172.21.15.162:0/2315245675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c80f6840 msgr2=0x7f75c8104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:50.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.337+0000 7f75cd42b700 1 --2- 172.21.15.162:0/2315245675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c80f6840 0x7f75c8104ac0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 rx=0x7f75c0009be0 tx=0x7f75c0004d10).stop 2024-02-13T00:48:50.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.337+0000 7f75cd42b700 1 -- 172.21.15.162:0/2315245675 shutdown_connections 2024-02-13T00:48:50.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.337+0000 7f75cd42b700 1 --2- 172.21.15.162:0/2315245675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c80f6840 0x7f75c8104ac0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:50.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.337+0000 7f75cd42b700 1 --2- 172.21.15.162:0/2315245675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75c80fdc50 0x7f75c80fe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:50.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.337+0000 7f75cd42b700 1 --2- 172.21.15.162:0/2315245675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c80fe5b0 0x7f75c80f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:50.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.337+0000 7f75cd42b700 1 -- 172.21.15.162:0/2315245675 >> 172.21.15.162:0/2315245675 conn(0x7f75c80700d0 msgr2=0x7f75c80f4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:50.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.337+0000 7f75cd42b700 1 -- 172.21.15.162:0/2315245675 shutdown_connections 2024-02-13T00:48:50.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.337+0000 7f75cd42b700 1 -- 172.21.15.162:0/2315245675 wait complete. 2024-02-13T00:48:50.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75cd42b700 1 librados: starting msgr at 2024-02-13T00:48:50.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75cd42b700 1 librados: starting objecter 2024-02-13T00:48:50.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75cd42b700 1 Processor -- start 2024-02-13T00:48:50.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75cd42b700 1 -- start start 2024-02-13T00:48:50.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75cd42b700 1 librados: setting wanted keys 2024-02-13T00:48:50.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75cd42b700 1 librados: calling monclient init 2024-02-13T00:48:50.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75cd42b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c80f6840 0x7f75c810e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:50.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75cd42b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75c80fdc50 0x7f75c810e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:50.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75cd42b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c80fe5b0 0x7f75c81083f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:50.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75cd42b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c81088e0 con 0x7f75c80f6840 2024-02-13T00:48:50.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75cd42b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c8108a50 con 0x7f75c80fdc50 2024-02-13T00:48:50.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75cd42b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c8108c10 con 0x7f75c80fe5b0 2024-02-13T00:48:50.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75c77fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c80fe5b0 0x7f75c81083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:50.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75c67fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75c80fdc50 0x7f75c810e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:50.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75c6ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c80f6840 0x7f75c810e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:50.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75c77fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c80fe5b0 0x7f75c81083f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:42450/0 (socket says 172.21.15.162:42450) 2024-02-13T00:48:50.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75c67fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75c80fdc50 0x7f75c810e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:33730/0 (socket says 172.21.15.162:33730) 2024-02-13T00:48:50.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.338+0000 7f75c77fe700 1 -- 172.21.15.162:0/4047904369 learned_addr learned my addr 172.21.15.162:0/4047904369 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:50.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75c77fe700 1 -- 172.21.15.162:0/4047904369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75c80fdc50 msgr2=0x7f75c810e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:50.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75c77fe700 1 --2- 172.21.15.162:0/4047904369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75c80fdc50 0x7f75c810e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:50.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75c77fe700 1 -- 172.21.15.162:0/4047904369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c80f6840 msgr2=0x7f75c810e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:50.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75c77fe700 1 --2- 172.21.15.162:0/4047904369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c80f6840 0x7f75c810e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:50.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75c77fe700 1 -- 172.21.15.162:0/4047904369 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b8005040 con 0x7f75c80fe5b0 2024-02-13T00:48:50.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75c6ffd700 1 --2- 172.21.15.162:0/4047904369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c80f6840 0x7f75c810e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:50.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75c77fe700 1 --2- 172.21.15.162:0/4047904369 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c80fe5b0 0x7f75c81083f0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 rx=0x7f75c0009fd0 tx=0x7f75c0004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75affff700 10 client.?.objecter ms_handle_connect 0x7f75c80fe5b0 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75affff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75affff700 1 -- 172.21.15.162:0/4047904369 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c000e040 con 0x7f75c80fe5b0 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75cd42b700 1 -- 172.21.15.162:0/4047904369 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c00097c0 con 0x7f75c80fe5b0 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75affff700 1 -- 172.21.15.162:0/4047904369 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75c0018040 con 0x7f75c80fe5b0 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75affff700 1 -- 172.21.15.162:0/4047904369 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0013e90 con 0x7f75c80fe5b0 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75cd42b700 10 client.37685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75cd42b700 1 -- 172.21.15.162:0/4047904369 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c81090b0 con 0x7f75c80fe5b0 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.339+0000 7f75cd42b700 1 librados: init done 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.340+0000 7f75cd42b700 1 -- 172.21.15.162:0/4047904369 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c81092a0 con 0x7f75c80fe5b0 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.341+0000 7f75affff700 1 -- 172.21.15.162:0/4047904369 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f75c0013590 con 0x7f75c80fe5b0 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.341+0000 7f75affff700 1 --2- 172.21.15.162:0/4047904369 >> v2:172.21.15.162:6800/2866534053 conn(0x7f75b0062c90 0x7f75b0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.341+0000 7f75affff700 1 -- 172.21.15.162:0/4047904369 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(515..515 src has 449..515) v4 ==== 8070+0+0 (secure 0 0 0) 0x7f75c007eff0 con 0x7f75c80fe5b0 2024-02-13T00:48:50.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.341+0000 7f75affff700 10 client.37685.objecter ms_dispatch 0x7f75c8001550 osd_map(515..515 src has 449..515) v4 2024-02-13T00:48:50.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.341+0000 7f75affff700 3 client.37685.objecter handle_osd_map got epochs [515,515] > 0 2024-02-13T00:48:50.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.341+0000 7f75affff700 3 client.37685.objecter handle_osd_map decoding full epoch 515 2024-02-13T00:48:50.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.341+0000 7f75c6ffd700 1 --2- 172.21.15.162:0/4047904369 >> v2:172.21.15.162:6800/2866534053 conn(0x7f75b0062c90 0x7f75b0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:50.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.341+0000 7f75affff700 20 client.37685.objecter dump_active .. 0 homeless 2024-02-13T00:48:50.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.342+0000 7f75affff700 10 client.37685.objecter ms_handle_connect 0x7f75b0062c90 2024-02-13T00:48:50.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.342+0000 7f75c6ffd700 1 --2- 172.21.15.162:0/4047904369 >> v2:172.21.15.162:6800/2866534053 conn(0x7f75b0062c90 0x7f75b0065150 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 rx=0x7f75c8060d60 tx=0x7f75bc00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:50.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.345+0000 7f75affff700 1 -- 172.21.15.162:0/4047904369 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f75c0052590 con 0x7f75c80fe5b0 2024-02-13T00:48:50.397 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:50.436 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:50.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:50.566+0000 7f75cd42b700 1 -- 172.21.15.162:0/4047904369 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f75c81071b0 con 0x7f75c80fe5b0 2024-02-13T00:48:50.959 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:50.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:50.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:50.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:49.601042+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e515: 8 total, 8 up, 4 in 2024-02-13T00:48:50.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:49.602281+0000 mon.b (mon.1) 639 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:50.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:49.608108+0000 mon.a (mon.0) 2788 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:50.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:50.465543+0000 mon.c (mon.2) 410 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-13T00:48:50.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:50.467793+0000 mon.a (mon.0) 2789 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-13T00:48:50.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:50.470873+0000 mon.a (mon.0) 2790 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:48:50.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:50.471036+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e516: 8 total, 8 up, 4 in 2024-02-13T00:48:50.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:50.472693+0000 mon.b (mon.1) 640 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:48:50.973 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:49.601042+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e515: 8 total, 8 up, 4 in 2024-02-13T00:48:50.973 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:49.602281+0000 mon.b (mon.1) 639 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:50.973 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:49.608108+0000 mon.a (mon.0) 2788 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:50.973 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:50.465543+0000 mon.c (mon.2) 410 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-13T00:48:50.973 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:50.467793+0000 mon.a (mon.0) 2789 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-13T00:48:50.973 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:50.470873+0000 mon.a (mon.0) 2790 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:48:50.973 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:50.471036+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e516: 8 total, 8 up, 4 in 2024-02-13T00:48:50.973 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:50.472693+0000 mon.b (mon.1) 640 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:48:50.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:49.601042+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e515: 8 total, 8 up, 4 in 2024-02-13T00:48:50.973 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:49.602281+0000 mon.b (mon.1) 639 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:50.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:49.608108+0000 mon.a (mon.0) 2788 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:50.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:50.465543+0000 mon.c (mon.2) 410 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-13T00:48:50.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:50.467793+0000 mon.a (mon.0) 2789 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-13T00:48:50.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:50.470873+0000 mon.a (mon.0) 2790 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:48:50.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:50.471036+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e516: 8 total, 8 up, 4 in 2024-02-13T00:48:50.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:50.472693+0000 mon.b (mon.1) 640 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:48:51.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:51.092 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:51.191 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:51.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:51.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:51.211 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:48:51.212 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:48:51.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:48:51.212 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:48:51.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.212 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.14862.0:3 12.f 12:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-13T00:48:51.212 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:00.557921+0000", 2024-02-13T00:48:51.212 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 470.64770046899997, 2024-02-13T00:48:51.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0933016659999999, 2024-02-13T00:48:51.213 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.213 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:51.213 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:48:51.213 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.14862", 2024-02-13T00:48:51.213 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/4283548805", 2024-02-13T00:48:51.213 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 3 2024-02-13T00:48:51.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.214 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557921+0000", 2024-02-13T00:48:51.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557921+0000", 2024-02-13T00:48:51.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:48:51.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557919+0000", 2024-02-13T00:48:51.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9099999999999999e-06 2024-02-13T00:48:51.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557926+0000", 2024-02-13T00:48:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4139e-05 2024-02-13T00:48:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557940+0000", 2024-02-13T00:48:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9110000000000004e-06 2024-02-13T00:48:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557947+0000", 2024-02-13T00:48:51.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4711e-05 2024-02-13T00:48:51.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557982+0000", 2024-02-13T00:48:51.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8269999999999997e-06 2024-02-13T00:48:51.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:48:51.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:00.557987+0000", 2024-02-13T00:48:51.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.060214974 2024-02-13T00:48:51.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.618202+0000", 2024-02-13T00:48:51.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016521299999999999 2024-02-13T00:48:51.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.618367+0000", 2024-02-13T00:48:51.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015337700000000001 2024-02-13T00:48:51.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,5", 2024-02-13T00:48:51.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.618520+0000", 2024-02-13T00:48:51.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083500600000000001 2024-02-13T00:48:51.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:48:51.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.619355+0000", 2024-02-13T00:48:51.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011477988999999999 2024-02-13T00:48:51.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:51.222 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.630833+0000", 2024-02-13T00:48:51.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020342135000000001 2024-02-13T00:48:51.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:51.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.651176+0000", 2024-02-13T00:48:51.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5749e-05 2024-02-13T00:48:51.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:48:51.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.651201+0000", 2024-02-13T00:48:51.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1682e-05 2024-02-13T00:48:51.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.651223+0000", 2024-02-13T00:48:51.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0933016659999999 2024-02-13T00:48:51.225 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.225 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.225 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.226 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24814.0:4 27.1f 27:ff7b1f36:::obj1:head [stat] snapc 0=[] ondisk+read+known_if_redirected e107)", 2024-02-13T00:48:51.226 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:01.629056+0000", 2024-02-13T00:48:51.226 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 469.57656564400003, 2024-02-13T00:48:51.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.033158676999999998, 2024-02-13T00:48:51.226 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.226 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:48:51.226 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:48:51.227 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24814", 2024-02-13T00:48:51.227 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/4028053558", 2024-02-13T00:48:51.227 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 4 2024-02-13T00:48:51.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.227 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629056+0000", 2024-02-13T00:48:51.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629056+0000", 2024-02-13T00:48:51.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:48:51.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629054+0000", 2024-02-13T00:48:51.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.032e-06 2024-02-13T00:48:51.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629060+0000", 2024-02-13T00:48:51.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4870000000000002e-06 2024-02-13T00:48:51.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629069+0000", 2024-02-13T00:48:51.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.793e-06 2024-02-13T00:48:51.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629076+0000", 2024-02-13T00:48:51.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012851900000000001 2024-02-13T00:48:51.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629204+0000", 2024-02-13T00:48:51.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5579999999999997e-06 2024-02-13T00:48:51.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:48:51.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.629211+0000", 2024-02-13T00:48:51.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.032924434000000002 2024-02-13T00:48:51.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.662135+0000", 2024-02-13T00:48:51.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9778999999999996e-05 2024-02-13T00:48:51.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:01.662215+0000", 2024-02-13T00:48:51.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.033158676999999998 2024-02-13T00:48:51.235 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.235 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.236 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.236 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24997.0:2 36.1a 36:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:48:51.236 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.022858+0000", 2024-02-13T00:48:51.236 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 467.18276414600001, 2024-02-13T00:48:51.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0671349779999999, 2024-02-13T00:48:51.236 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.236 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24997", 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/1225415695", 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022858+0000", 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022858+0000", 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022855+0000", 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9489999999999996e-06 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022863+0000", 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4700000000000008e-06 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022873+0000", 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2189999999999998e-06 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022880+0000", 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9550999999999999e-05 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022950+0000", 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0539999999999997e-06 2024-02-13T00:48:51.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:48:51.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.022957+0000", 2024-02-13T00:48:51.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0397542660000001 2024-02-13T00:48:51.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.062711+0000", 2024-02-13T00:48:51.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109751 2024-02-13T00:48:51.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.062821+0000", 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015550399999999999 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.062976+0000", 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001836599 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:48:51.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.064813+0000", 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004613102 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.069426+0000", 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020460633999999998 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.089887+0000", 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5318e-05 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.089952+0000", 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0765e-05 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:05.089993+0000", 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0671349779999999 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.14886.0:2070 20.b e109/105 20:d3a22371:test-rados-api-smithi162-84049-17::big2:head v 109'1, mlcod=0'0)", 2024-02-13T00:48:51.243 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:04.754351+0000", 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 466.45127073700002, 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088475559999999995, 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754351+0000", 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754351+0000", 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.754350+0000", 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.045778653000000002 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800129+0000", 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1767999999999999e-05 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800150+0000", 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6830000000000008e-06 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800158+0000", 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7021999999999998e-05 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800205+0000", 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.98e-05 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.800225+0000", 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.042577241000000002 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842802+0000", 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4448e-05 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:04.842827+0000", 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088475559999999995 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.247 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.248 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.248 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:1163 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94514175197056 gen 3] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-13T00:48:51.248 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:39.297056+0000", 2024-02-13T00:48:51.248 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 251.90856557999999, 2024-02-13T00:48:51.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.232174933, 2024-02-13T00:48:51.248 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.248 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.248 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:48:51.248 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:48:51.248 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1163 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297056+0000", 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297056+0000", 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999971 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297053+0000", 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3816e-05 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297067+0000", 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8560000000000001e-05 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297096+0000", 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9269000000000001e-05 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297115+0000", 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3229e-05 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297188+0000", 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9490000000000001e-06 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297194+0000", 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.225866915 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.523061+0000", 2024-02-13T00:48:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117611 2024-02-13T00:48:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for missing object", 2024-02-13T00:48:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.523179+0000", 2024-02-13T00:48:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044211959999999996 2024-02-13T00:48:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.527600+0000", 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3129999999999998e-05 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.527693+0000", 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00068676900000000005 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.528380+0000", 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2867999999999995e-05 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.528443+0000", 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00078838699999999999 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.529231+0000", 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.232174933 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:1165 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94514175188992 gen 3] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:39.297151+0000", 2024-02-13T00:48:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 251.908470548, 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23092579999999999, 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1165 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297151+0000", 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297151+0000", 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297151+0000", 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7009999999999998e-06 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297154+0000", 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.985e-06 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297161+0000", 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8110000000000002e-06 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297166+0000", 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2799000000000002e-05 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297199+0000", 2024-02-13T00:48:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9120000000000003e-06 2024-02-13T00:48:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:48:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.297205+0000", 2024-02-13T00:48:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21097220999999999 2024-02-13T00:48:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.508177+0000", 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000102681 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for missing object", 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.508280+0000", 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0052417039999999998 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.513521+0000", 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3107000000000002e-05 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.513604+0000", 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034221709999999999 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.517027+0000", 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4387000000000005e-05 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.517101+0000", 2024-02-13T00:48:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010976069999999999 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:39.528077+0000", 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23092579999999999 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25162.0:4733 191.6 191:602f83fe:::foo:head [write 0~1] snapc 0=[] ondisk+write+known_if_redirected e334)", 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:50.441367+0000", 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 240.76425506699999, 2024-02-13T00:48:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.053522410999999999, 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25162", 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 4733 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.441367+0000", 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.441367+0000", 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.441365+0000", 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0449999999999997e-06 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.441371+0000", 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8289999999999997e-06 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.441380+0000", 2024-02-13T00:48:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5399999999999998e-06 2024-02-13T00:48:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.441388+0000", 2024-02-13T00:48:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00138733 2024-02-13T00:48:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.442775+0000", 2024-02-13T00:48:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0977000000000002e-05 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.442836+0000", 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010878 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.442945+0000", 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.022457622 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.465402+0000", 2024-02-13T00:48:51.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002867521 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.468270+0000", 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.026506655 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.494777+0000", 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2019000000000004e-05 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:48:51.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.494839+0000", 2024-02-13T00:48:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0553999999999997e-05 2024-02-13T00:48:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.494889+0000", 2024-02-13T00:48:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.053522410999999999 2024-02-13T00:48:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.267 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.16 335/334 [PushOp(4:6dc3b799:reshard::reshard.0000000006:head, version: 64'4, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6dc3b799:reshard::reshard.0000000006:head@64'4, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:48:51.268 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:51.120970+0000", 2024-02-13T00:48:51.268 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 240.084651646, 2024-02-13T00:48:51.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063347392000000002, 2024-02-13T00:48:51.268 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.268 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.268 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.120970+0000", 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.120970+0000", 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.120969+0000", 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3338e-05 2024-02-13T00:48:51.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.120983+0000", 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4115e-05 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.120997+0000", 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7559999999999999e-06 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.121005+0000", 2024-02-13T00:48:51.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063207902999999996 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.184213+0000", 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.049e-06 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.184219+0000", 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8979000000000002e-05 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.184318+0000", 2024-02-13T00:48:51.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063347392000000002 2024-02-13T00:48:51.272 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.272 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.272 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.272 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.16 335/334 [PushOp(4:6849fd06:gc::gc.17:head, version: 64'5, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:6849fd06:gc::gc.17:head@64'5, size: 110, copy_subset: [0~110], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:110, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:48:51.272 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:51.206879+0000", 2024-02-13T00:48:51.273 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 239.998742435, 2024-02-13T00:48:51.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063759863, 2024-02-13T00:48:51.273 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.273 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.273 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.206879+0000", 2024-02-13T00:48:51.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.206879+0000", 2024-02-13T00:48:51.274 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:51.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.206879+0000", 2024-02-13T00:48:51.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3208e-05 2024-02-13T00:48:51.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.275 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.206892+0000", 2024-02-13T00:48:51.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3016999999999999e-05 2024-02-13T00:48:51.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.276 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.206905+0000", 2024-02-13T00:48:51.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.596e-06 2024-02-13T00:48:51.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.276 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.206912+0000", 2024-02-13T00:48:51.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063607217999999993 2024-02-13T00:48:51.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.270519+0000", 2024-02-13T00:48:51.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8039999999999999e-06 2024-02-13T00:48:51.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.270523+0000", 2024-02-13T00:48:51.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116678 2024-02-13T00:48:51.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.270639+0000", 2024-02-13T00:48:51.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063759863 2024-02-13T00:48:51.278 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.278 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.279 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.279 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.16 335/334 [PushOp(4:6a988a9b:reshard::reshard.0000000007:head, version: 64'7, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6a988a9b:reshard::reshard.0000000007:head@64'7, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:48:51.279 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:51.292782+0000", 2024-02-13T00:48:51.279 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 239.91284000600001, 2024-02-13T00:48:51.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068626035000000002, 2024-02-13T00:48:51.279 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.280 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.280 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.280 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.292782+0000", 2024-02-13T00:48:51.280 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.280 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.292782+0000", 2024-02-13T00:48:51.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:51.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.292781+0000", 2024-02-13T00:48:51.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0513000000000001e-05 2024-02-13T00:48:51.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.292792+0000", 2024-02-13T00:48:51.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3203999999999999e-05 2024-02-13T00:48:51.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.292805+0000", 2024-02-13T00:48:51.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8950000000000001e-06 2024-02-13T00:48:51.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.292812+0000", 2024-02-13T00:48:51.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068487231999999995 2024-02-13T00:48:51.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.284 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.361299+0000", 2024-02-13T00:48:51.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3909999999999997e-06 2024-02-13T00:48:51.284 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.284 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.361304+0000", 2024-02-13T00:48:51.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103461 2024-02-13T00:48:51.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.285 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.361408+0000", 2024-02-13T00:48:51.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068626035000000002 2024-02-13T00:48:51.285 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.285 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.285 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.285 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.16 335/334 [PushOp(4:6f6aba0e:::obj_delete_at_hint.0000000045:head, version: 64'17, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6f6aba0e:::obj_delete_at_hint.0000000045:head@64'17, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:51.378708+0000", 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 239.82691345000001, 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069001464999999998, 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.378708+0000", 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.378708+0000", 2024-02-13T00:48:51.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.378707+0000", 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4348999999999999e-05 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.378722+0000", 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5891000000000001e-05 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.378737+0000", 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4080000000000001e-06 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.378745+0000", 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068862920999999994 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.447608+0000", 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0780000000000003e-06 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.447613+0000", 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6967999999999997e-05 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.447710+0000", 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069001464999999998 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.289 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.16 336/334 [PushOp(4:6b0d4d95:lc::lc.20:head, version: 65'19, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6b0d4d95:lc::lc.20:head@65'19, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:51.464684+0000", 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 239.74093767100001, 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069328461999999993, 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.464684+0000", 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.464684+0000", 2024-02-13T00:48:51.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:51.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.464684+0000", 2024-02-13T00:48:51.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0465000000000001e-05 2024-02-13T00:48:51.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.464694+0000", 2024-02-13T00:48:51.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3033999999999999e-05 2024-02-13T00:48:51.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.464707+0000", 2024-02-13T00:48:51.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4029999999999996e-06 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.464716+0000", 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069196293000000006 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.533913+0000", 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9509999999999997e-06 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.533918+0000", 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4919000000000004e-05 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.534013+0000", 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069328461999999993 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.16 336/334 [PushOp(4:6d4431ed:lc::lc.7:head, version: 65'21, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d4431ed:lc::lc.7:head@65'21, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:48:51.294 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:51.550651+0000", 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 239.65497129799999, 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074139102999999998, 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.550651+0000", 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.550651+0000", 2024-02-13T00:48:51.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.550650+0000", 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1858999999999999e-05 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.550662+0000", 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3012e-05 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.550675+0000", 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.511e-06 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.550681+0000", 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074006764000000003 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.624688+0000", 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9620000000000004e-06 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.298 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.298 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.624692+0000", 2024-02-13T00:48:51.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7652999999999995e-05 2024-02-13T00:48:51.298 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.298 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.298 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.298 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:51.624790+0000", 2024-02-13T00:48:51.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074139102999999998 2024-02-13T00:48:51.298 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.298 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.298 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.298 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.27243.0:1 206.7s0 206:fbfde494:::oid_MultiWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:45:40.407348+0000", 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 190.79827416399999, 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.007028646, 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.27243", 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/879748064", 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407348+0000", 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407348+0000", 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407346+0000", 2024-02-13T00:48:51.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.246e-06 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407352+0000", 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7770000000000008e-06 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407361+0000", 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.96e-06 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407369+0000", 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3953000000000003e-05 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407423+0000", 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7290000000000002e-06 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407427+0000", 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0034671980000001 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:41.410894+0000", 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015416900000000001 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:41.411048+0000", 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000348709 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:41.411397+0000", 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000485553 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.303 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:41.411882+0000", 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002441473 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:41.414324+0000", 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2499e-05 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:41.414376+0000", 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.007028646 2024-02-13T00:48:51.304 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25162.0:5962 61.1e 61:78ef1149:test-rados-api-smithi162-84057-58::foo-chunk:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected e403)", 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:46:02.459220+0000", 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 168.74640182300001, 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.028868371, 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25162", 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:48:51.305 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 5962 2024-02-13T00:48:51.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.306 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.306 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459220+0000", 2024-02-13T00:48:51.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.306 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459220+0000", 2024-02-13T00:48:51.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:48:51.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459219+0000", 2024-02-13T00:48:51.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6969999999999998e-06 2024-02-13T00:48:51.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459224+0000", 2024-02-13T00:48:51.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2309999999999999e-06 2024-02-13T00:48:51.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.308 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.308 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459232+0000", 2024-02-13T00:48:51.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9619999999999996e-06 2024-02-13T00:48:51.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459239+0000", 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011526 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459354+0000", 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013904400000000001 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459493+0000", 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000156098 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459649+0000", 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019810800000000001 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.461630+0000", 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058996099999999996 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.462220+0000", 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025775693999999998 2024-02-13T00:48:51.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.487996+0000", 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2039000000000001e-05 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.488048+0000", 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0672e-05 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.488088+0000", 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.028868371 2024-02-13T00:48:51.311 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.0 404/401 [PushOp(4:02683654:reshard::reshard.0000000010:head, version: 64'10, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:02683654:reshard::reshard.0000000010:head@64'10, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:46:03.967764+0000", 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 167.23785799800001, 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084933228, 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:03.967764+0000", 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:03.967764+0000", 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:03.967763+0000", 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0074999999999999e-05 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.313 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:03.967773+0000", 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2969999999999999e-05 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:03.967786+0000", 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4960000000000002e-06 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:03.967796+0000", 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084775084000000001 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.052571+0000", 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1409999999999998e-06 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.052575+0000", 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000122024 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.052697+0000", 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084933228 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.315 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.0 404/401 [PushOp(4:013d6c33:gc::gc.30:head, version: 64'14, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:013d6c33:gc::gc.30:head@64'14, size: 110, copy_subset: [0~110], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:110, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:46:04.053490+0000", 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 167.15213161099999, 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085623471000000007, 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.053490+0000", 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.053490+0000", 2024-02-13T00:48:51.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:51.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.053490+0000", 2024-02-13T00:48:51.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0045e-05 2024-02-13T00:48:51.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.053500+0000", 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1745e-05 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.053511+0000", 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3350000000000003e-06 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.053518+0000", 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085375106000000006 2024-02-13T00:48:51.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.138893+0000", 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.75e-06 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.138899+0000", 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021510799999999999 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.139114+0000", 2024-02-13T00:48:51.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085623471000000007 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.0 404/401 [PushOp(4:03bbb29a:::obj_delete_at_hint.0000000026:head, version: 64'16, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:03bbb29a:::obj_delete_at_hint.0000000026:head@64'16, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:46:04.140167+0000", 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 167.06545484399999, 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085170970999999998, 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.140167+0000", 2024-02-13T00:48:51.321 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.140167+0000", 2024-02-13T00:48:51.321 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:48:51.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.140166+0000", 2024-02-13T00:48:51.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5435999999999999e-05 2024-02-13T00:48:51.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.140182+0000", 2024-02-13T00:48:51.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3654e-05 2024-02-13T00:48:51.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.323 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.323 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.140195+0000", 2024-02-13T00:48:51.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.579e-06 2024-02-13T00:48:51.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.140202+0000", 2024-02-13T00:48:51.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085002941999999998 2024-02-13T00:48:51.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.325 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.225205+0000", 2024-02-13T00:48:51.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0469999999999996e-06 2024-02-13T00:48:51.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.225212+0000", 2024-02-13T00:48:51.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012621999999999999 2024-02-13T00:48:51.326 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.225338+0000", 2024-02-13T00:48:51.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085170970999999998 2024-02-13T00:48:51.327 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.327 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.327 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.327 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.0 404/401 [PushOp(4:0705c68b:::obj_delete_at_hint.0000000028:head, version: 64'18, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:0705c68b:::obj_delete_at_hint.0000000028:head@64'18, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:48:51.328 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:46:04.226305+0000", 2024-02-13T00:48:51.328 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 166.97931716799999, 2024-02-13T00:48:51.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085333043999999997, 2024-02-13T00:48:51.328 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.328 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.328 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.226305+0000", 2024-02-13T00:48:51.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.226305+0000", 2024-02-13T00:48:51.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:48:51.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.226304+0000", 2024-02-13T00:48:51.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3427999999999999e-05 2024-02-13T00:48:51.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.226317+0000", 2024-02-13T00:48:51.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3625000000000001e-05 2024-02-13T00:48:51.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.226331+0000", 2024-02-13T00:48:51.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5089999999999999e-06 2024-02-13T00:48:51.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.226337+0000", 2024-02-13T00:48:51.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085176106000000001 2024-02-13T00:48:51.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.311514+0000", 2024-02-13T00:48:51.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.058e-06 2024-02-13T00:48:51.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.311521+0000", 2024-02-13T00:48:51.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117083 2024-02-13T00:48:51.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.311638+0000", 2024-02-13T00:48:51.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085333043999999997 2024-02-13T00:48:51.336 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.336 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.336 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.337 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.0 404/401 [PushOp(4:0667bb13:gc::gc.13:head, version: 64'20, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:0667bb13:gc::gc.13:head@64'20, size: 110, copy_subset: [0~110], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:110, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:48:51.337 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:46:04.323918+0000", 2024-02-13T00:48:51.337 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 166.88170404799999, 2024-02-13T00:48:51.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074098921999999998, 2024-02-13T00:48:51.337 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:48:51.338 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:48:51.338 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:48:51.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:48:51.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.323918+0000", 2024-02-13T00:48:51.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:48:51.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:48:51.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.323918+0000", 2024-02-13T00:48:51.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:48:51.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:48:51.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.323917+0000", 2024-02-13T00:48:51.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1127e-05 2024-02-13T00:48:51.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:48:51.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.323928+0000", 2024-02-13T00:48:51.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3138000000000001e-05 2024-02-13T00:48:51.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:48:51.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.323951+0000", 2024-02-13T00:48:51.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6029999999999994e-06 2024-02-13T00:48:51.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:48:51.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.323961+0000", 2024-02-13T00:48:51.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.073866717999999998 2024-02-13T00:48:51.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:48:51.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.397828+0000", 2024-02-13T00:48:51.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5600000000000004e-06 2024-02-13T00:48:51.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:48:51.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.397832+0000", 2024-02-13T00:48:51.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018444499999999999 2024-02-13T00:48:51.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:48:51.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:48:51.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:48:51.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.398017+0000", 2024-02-13T00:48:51.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074098921999999998 2024-02-13T00:48:51.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.344 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:48:51.345 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:48:51.345 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:48:51.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:48:51.374 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:51.426 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:48:51.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:51.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:48:51.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.474+0000 7f75affff700 1 -- 172.21.15.162:0/4047904369 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v517) v1 ==== 91+0+0 (secure 0 0 0) 0x7f75c0052180 con 0x7f75c80fe5b0 2024-02-13T00:48:51.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.478+0000 7f75cd42b700 10 librados: watch_flush enter 2024-02-13T00:48:51.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.479+0000 7f75cd42b700 10 librados: watch_flush exit 2024-02-13T00:48:51.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.479+0000 7f75cd42b700 20 client.37685.objecter shutdown clearing up homeless session... 2024-02-13T00:48:51.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.479+0000 7f75cd42b700 10 client.37685.objecter successfully canceled tick 2024-02-13T00:48:51.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.479+0000 7f75cd42b700 1 -- 172.21.15.162:0/4047904369 >> v2:172.21.15.162:6800/2866534053 conn(0x7f75b0062c90 msgr2=0x7f75b0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:51.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.479+0000 7f75cd42b700 1 --2- 172.21.15.162:0/4047904369 >> v2:172.21.15.162:6800/2866534053 conn(0x7f75b0062c90 0x7f75b0065150 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 rx=0x7f75c8060d60 tx=0x7f75bc00b040).stop 2024-02-13T00:48:51.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.479+0000 7f75cd42b700 1 -- 172.21.15.162:0/4047904369 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c80fe5b0 msgr2=0x7f75c81083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:51.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.479+0000 7f75cd42b700 1 --2- 172.21.15.162:0/4047904369 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c80fe5b0 0x7f75c81083f0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 rx=0x7f75c0009fd0 tx=0x7f75c0004d10).stop 2024-02-13T00:48:51.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.481+0000 7f75cd42b700 1 -- 172.21.15.162:0/4047904369 shutdown_connections 2024-02-13T00:48:51.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.481+0000 7f75cd42b700 1 --2- 172.21.15.162:0/4047904369 >> v2:172.21.15.162:6800/2866534053 conn(0x7f75b0062c90 0x7f75b0065150 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:51.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.481+0000 7f75cd42b700 1 --2- 172.21.15.162:0/4047904369 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c80fe5b0 0x7f75c81083f0 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:51.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.481+0000 7f75cd42b700 1 --2- 172.21.15.162:0/4047904369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c80f6840 0x7f75c810e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:51.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.481+0000 7f75cd42b700 1 --2- 172.21.15.162:0/4047904369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75c80fdc50 0x7f75c810e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:51.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.481+0000 7f75cd42b700 1 -- 172.21.15.162:0/4047904369 >> 172.21.15.162:0/4047904369 conn(0x7f75c80700d0 msgr2=0x7f75c806d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:51.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.481+0000 7f75cd42b700 1 -- 172.21.15.162:0/4047904369 shutdown_connections 2024-02-13T00:48:51.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.481+0000 7f75cd42b700 1 -- 172.21.15.162:0/4047904369 wait complete. 2024-02-13T00:48:51.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:51.481+0000 7f75cd42b700 1 librados: shutdown 2024-02-13T00:48:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:48:51.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:51.556 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:50.465216+0000 mgr.y (mgr.14544) 457 : cluster [DBG] pgmap v773: 234 pgs: 32 creating+peering, 202 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-02-13T00:48:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:50.567989+0000 mon.c (mon.2) 411 : audit [INF] from='client.? 172.21.15.162:0/4047904369' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:48:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:50.737863+0000 mon.a (mon.0) 2792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:48:51.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:51.468601+0000 mon.a (mon.0) 2793 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:51.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:51.472238+0000 mon.a (mon.0) 2794 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-13T00:48:51.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:51.472355+0000 mon.a (mon.0) 2795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:48:51.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:51.472408+0000 mon.a (mon.0) 2796 : cluster [DBG] osdmap e517: 8 total, 8 up, 4 in 2024-02-13T00:48:51.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:51.477152+0000 mon.b (mon.1) 641 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:51.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:51.485092+0000 mon.a (mon.0) 2797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:51.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-13T00:48:51.905 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:51.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:52.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:50.465216+0000 mgr.y (mgr.14544) 457 : cluster [DBG] pgmap v773: 234 pgs: 32 creating+peering, 202 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-02-13T00:48:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:50.567989+0000 mon.c (mon.2) 411 : audit [INF] from='client.? 172.21.15.162:0/4047904369' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:48:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:50.737863+0000 mon.a (mon.0) 2792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:48:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:51.468601+0000 mon.a (mon.0) 2793 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:51.472238+0000 mon.a (mon.0) 2794 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-13T00:48:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:51.472355+0000 mon.a (mon.0) 2795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:48:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:51.472408+0000 mon.a (mon.0) 2796 : cluster [DBG] osdmap e517: 8 total, 8 up, 4 in 2024-02-13T00:48:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:51.477152+0000 mon.b (mon.1) 641 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:51.485092+0000 mon.a (mon.0) 2797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:52.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:50.465216+0000 mgr.y (mgr.14544) 457 : cluster [DBG] pgmap v773: 234 pgs: 32 creating+peering, 202 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-02-13T00:48:52.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:50.567989+0000 mon.c (mon.2) 411 : audit [INF] from='client.? 172.21.15.162:0/4047904369' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:48:52.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:50.737863+0000 mon.a (mon.0) 2792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:48:52.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:51.468601+0000 mon.a (mon.0) 2793 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:48:52.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:51.472238+0000 mon.a (mon.0) 2794 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-13T00:48:52.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:51.472355+0000 mon.a (mon.0) 2795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:48:52.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:51.472408+0000 mon.a (mon.0) 2796 : cluster [DBG] osdmap e517: 8 total, 8 up, 4 in 2024-02-13T00:48:52.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:51.477152+0000 mon.b (mon.1) 641 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:52.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:51.485092+0000 mon.a (mon.0) 2797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:48:52.038 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:52.076 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:52.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:52.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:52.167 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:48:52.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:52.209 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:52.458 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:52.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:52.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:52.593 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:52.716 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:52.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:52.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:52.847 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:53.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.040+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/3868572675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3b40fd490 msgr2=0x7ff3b40f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:53.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.040+0000 7ff3bc0b8700 1 --2- 172.21.15.162:0/3868572675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3b40fd490 0x7ff3b40f6cc0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7ff3b0004870 tx=0x7ff3b0009cb0).stop 2024-02-13T00:48:53.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.040+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/3868572675 shutdown_connections 2024-02-13T00:48:53.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.040+0000 7ff3bc0b8700 1 --2- 172.21.15.162:0/3868572675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b40f73a0 0x7ff3b41039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:53.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.040+0000 7ff3bc0b8700 1 --2- 172.21.15.162:0/3868572675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3b40fcb30 0x7ff3b40fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:53.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.040+0000 7ff3bc0b8700 1 --2- 172.21.15.162:0/3868572675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3b40fd490 0x7ff3b40f6cc0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:53.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.040+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/3868572675 >> 172.21.15.162:0/3868572675 conn(0x7ff3b406eb90 msgr2=0x7ff3b40f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:53.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.040+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/3868572675 shutdown_connections 2024-02-13T00:48:53.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.040+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/3868572675 wait complete. 2024-02-13T00:48:53.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3bc0b8700 1 librados: starting msgr at 2024-02-13T00:48:53.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3bc0b8700 1 librados: starting objecter 2024-02-13T00:48:53.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3bc0b8700 1 Processor -- start 2024-02-13T00:48:53.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3bc0b8700 1 -- start start 2024-02-13T00:48:53.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3bc0b8700 1 librados: setting wanted keys 2024-02-13T00:48:53.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3bc0b8700 1 librados: calling monclient init 2024-02-13T00:48:53.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3bc0b8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3b40f73a0 0x7ff3b41091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:53.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3bc0b8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3b40fcb30 0x7ff3b41096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:53.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3bc0b8700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b40fd490 0x7ff3b4109c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:53.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3bc0b8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b41143a0 con 0x7ff3b40f73a0 2024-02-13T00:48:53.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3bc0b8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b4114510 con 0x7ff3b40fcb30 2024-02-13T00:48:53.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3bc0b8700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b41146d0 con 0x7ff3b40fd490 2024-02-13T00:48:53.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3b9e54700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3b40f73a0 0x7ff3b41091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:53.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3b9e54700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3b40f73a0 0x7ff3b41091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:35542/0 (socket says 172.21.15.162:35542) 2024-02-13T00:48:53.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3ba655700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b40fd490 0x7ff3b4109c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:53.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3b9e54700 1 -- 172.21.15.162:0/990017233 learned_addr learned my addr 172.21.15.162:0/990017233 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:53.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.041+0000 7ff3b9653700 1 --2- 172.21.15.162:0/990017233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3b40fcb30 0x7ff3b41096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3b9e54700 1 -- 172.21.15.162:0/990017233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b40fd490 msgr2=0x7ff3b4109c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3b9e54700 1 --2- 172.21.15.162:0/990017233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b40fd490 0x7ff3b4109c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3b9e54700 1 -- 172.21.15.162:0/990017233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3b40fcb30 msgr2=0x7ff3b41096a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3b9e54700 1 --2- 172.21.15.162:0/990017233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3b40fcb30 0x7ff3b41096a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3b9e54700 1 -- 172.21.15.162:0/990017233 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac0097c0 con 0x7ff3b40f73a0 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3ba655700 1 --2- 172.21.15.162:0/990017233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b40fd490 0x7ff3b4109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3b9653700 1 --2- 172.21.15.162:0/990017233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3b40fcb30 0x7ff3b41096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3b9e54700 1 --2- 172.21.15.162:0/990017233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3b40f73a0 0x7ff3b41091b0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 rx=0x7ff3a40041d0 tx=0x7ff3a4009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3aaffd700 10 client.?.objecter ms_handle_connect 0x7ff3b40f73a0 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3aaffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3aaffd700 1 -- 172.21.15.162:0/990017233 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4007b40 con 0x7ff3b40f73a0 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/990017233 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b0005040 con 0x7ff3b40f73a0 2024-02-13T00:48:53.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3aaffd700 1 -- 172.21.15.162:0/990017233 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff3a400f040 con 0x7ff3b40f73a0 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3aaffd700 1 -- 172.21.15.162:0/990017233 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4014810 con 0x7ff3b40f73a0 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3bc0b8700 10 client.27591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/990017233 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b4114d90 con 0x7ff3b40f73a0 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.042+0000 7ff3bc0b8700 1 librados: init done 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.043+0000 7ff3bc0b8700 10 client.27591.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.043+0000 7ff3bc0b8700 15 client.27591.objecter _session_command_op_assign -1 1 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.043+0000 7ff3bc0b8700 20 client.27591.objecter _get_session osd=-1 returning homeless 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.043+0000 7ff3bc0b8700 10 client.27591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.043+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/990017233 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff3b40475e0 con 0x7ff3b40f73a0 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.043+0000 7ff3aaffd700 1 -- 172.21.15.162:0/990017233 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff3a4009110 con 0x7ff3b40f73a0 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.043+0000 7ff3aaffd700 1 --2- 172.21.15.162:0/990017233 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff3a0062c90 0x7ff3a0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 1 -- 172.21.15.162:0/990017233 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(518..518 src has 449..518) v4 ==== 8482+0+0 (secure 0 0 0) 0x7ff3a407d390 con 0x7ff3b40f73a0 2024-02-13T00:48:53.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 10 client.27591.objecter ms_dispatch 0x7ff3b4001550 osd_map(518..518 src has 449..518) v4 2024-02-13T00:48:53.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 3 client.27591.objecter handle_osd_map got epochs [518,518] > 0 2024-02-13T00:48:53.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 3 client.27591.objecter handle_osd_map decoding full epoch 518 2024-02-13T00:48:53.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3b9653700 1 --2- 172.21.15.162:0/990017233 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff3a0062c90 0x7ff3a0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:53.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 10 client.27591.objecter checking command 1 2024-02-13T00:48:53.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 1 --2- 172.21.15.162:0/990017233 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff3a00707b0 0x7ff3a0072c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:53.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 20 client.27591.objecter _get_session s=0x7ff3a0070100 osd=4 3 2024-02-13T00:48:53.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 20 client.27591.objecter put_session s=0x7ff3a0070100 osd=4 3 2024-02-13T00:48:53.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 15 client.27591.objecter _session_command_op_remove -1 1 2024-02-13T00:48:53.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 20 client.27591.objecter _get_session s=0x7ff3a0070100 osd=4 3 2024-02-13T00:48:53.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 20 client.27591.objecter get_session s=0x7ff3a0070100 osd=4 3 2024-02-13T00:48:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 15 client.27591.objecter _session_command_op_assign 4 1 2024-02-13T00:48:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 20 client.27591.objecter put_session s=0x7ff3a0070100 osd=4 4 2024-02-13T00:48:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 10 client.27591.objecter _send_command 1 2024-02-13T00:48:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 1 -- 172.21.15.162:0/990017233 --> v2:172.21.15.181:6800/3344263388 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3a0073310 con 0x7ff3a00707b0 2024-02-13T00:48:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 20 client.27591.objecter dump_active .. 0 homeless 2024-02-13T00:48:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 1 -- 172.21.15.162:0/990017233 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=518) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3a407f970 con 0x7ff3b40f73a0 2024-02-13T00:48:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3ba655700 1 --2- 172.21.15.162:0/990017233 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff3a00707b0 0x7ff3a0072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 10 client.27591.objecter ms_handle_connect 0x7ff3a0062c90 2024-02-13T00:48:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3b9653700 1 --2- 172.21.15.162:0/990017233 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff3a0062c90 0x7ff3a0065150 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 rx=0x7ff3b00043d0 tx=0x7ff3b000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3aaffd700 10 client.27591.objecter ms_handle_connect 0x7ff3a00707b0 2024-02-13T00:48:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.044+0000 7ff3ba655700 1 --2- 172.21.15.162:0/990017233 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff3a00707b0 0x7ff3a0072c50 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.045+0000 7ff3aaffd700 1 -- 172.21.15.162:0/990017233 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff3a0073310 con 0x7ff3a00707b0 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.045+0000 7ff3aaffd700 10 client.27591.objecter ms_dispatch 0x7ff3b4001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.045+0000 7ff3aaffd700 10 client.27591.objecter _finish_command 1 = system:0 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.045+0000 7ff3aaffd700 20 client.27591.objecter put_session s=0x7ff3a0070100 osd=4 4 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.045+0000 7ff3aaffd700 15 client.27591.objecter _session_command_op_remove 4 1 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.061+0000 7ff3bc0b8700 10 client.27591.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.061+0000 7ff3bc0b8700 15 client.27591.objecter _session_command_op_assign -1 2 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.061+0000 7ff3bc0b8700 20 client.27591.objecter _get_session s=0x7ff3a0070100 osd=4 3 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.061+0000 7ff3bc0b8700 20 client.27591.objecter put_session s=0x7ff3a0070100 osd=4 3 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.061+0000 7ff3bc0b8700 20 client.27591.objecter _get_session s=0x7ff3a0070100 osd=4 3 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.061+0000 7ff3bc0b8700 15 client.27591.objecter _session_command_op_remove -1 2 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.061+0000 7ff3bc0b8700 20 client.27591.objecter get_session s=0x7ff3a0070100 osd=4 3 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.061+0000 7ff3bc0b8700 15 client.27591.objecter _session_command_op_assign 4 2 2024-02-13T00:48:53.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.062+0000 7ff3bc0b8700 20 client.27591.objecter put_session s=0x7ff3a0070100 osd=4 4 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.062+0000 7ff3bc0b8700 10 client.27591.objecter _send_command 2 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.062+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/990017233 --> v2:172.21.15.181:6800/3344263388 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7ff3b4106020 con 0x7ff3a00707b0 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.062+0000 7ff3aaffd700 1 -- 172.21.15.162:0/990017233 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff3b4106020 con 0x7ff3a00707b0 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.062+0000 7ff3aaffd700 10 client.27591.objecter ms_dispatch 0x7ff3b4001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.062+0000 7ff3aaffd700 10 client.27591.objecter _finish_command 2 = system:0 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.062+0000 7ff3aaffd700 20 client.27591.objecter put_session s=0x7ff3a0070100 osd=4 4 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.062+0000 7ff3aaffd700 15 client.27591.objecter _session_command_op_remove 4 2 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.062+0000 7ff3bc0b8700 10 librados: watch_flush enter 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.062+0000 7ff3bc0b8700 10 librados: watch_flush exit 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.062+0000 7ff3bc0b8700 10 client.27591.objecter close_session for osd.4 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.062+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/990017233 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff3a00707b0 msgr2=0x7ff3a0072c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 1 --2- 172.21.15.162:0/990017233 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff3a00707b0 0x7ff3a0072c50 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 20 client.27591.objecter put_session s=0x7ff3a0070100 osd=4 1 2024-02-13T00:48:53.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 20 client.27591.objecter shutdown clearing up homeless session... 2024-02-13T00:48:53.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 10 client.27591.objecter successfully canceled tick 2024-02-13T00:48:53.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/990017233 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff3a0062c90 msgr2=0x7ff3a0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:53.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 1 --2- 172.21.15.162:0/990017233 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff3a0062c90 0x7ff3a0065150 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 rx=0x7ff3b00043d0 tx=0x7ff3b000b040).stop 2024-02-13T00:48:53.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/990017233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3b40f73a0 msgr2=0x7ff3b41091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:53.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 1 --2- 172.21.15.162:0/990017233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3b40f73a0 0x7ff3b41091b0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 rx=0x7ff3a40041d0 tx=0x7ff3a4009f60).stop 2024-02-13T00:48:53.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3ba655700 1 -- 172.21.15.162:0/990017233 reap_dead start 2024-02-13T00:48:53.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/990017233 shutdown_connections 2024-02-13T00:48:53.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/990017233 >> 172.21.15.162:0/990017233 conn(0x7ff3b406eb90 msgr2=0x7ff3b40fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:53.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/990017233 shutdown_connections 2024-02-13T00:48:53.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 1 -- 172.21.15.162:0/990017233 wait complete. 2024-02-13T00:48:53.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:53.063+0000 7ff3bc0b8700 1 librados: shutdown 2024-02-13T00:48:53.696 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:53.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:53.700 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:52.465663+0000 mgr.y (mgr.14544) 458 : cluster [DBG] pgmap v776: 266 pgs: 32 unknown, 32 creating+peering, 202 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-02-13T00:48:53.700 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:52.484361+0000 mon.a (mon.0) 2798 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:48:53.700 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:52.484440+0000 mon.a (mon.0) 2799 : cluster [DBG] osdmap e518: 8 total, 8 up, 4 in 2024-02-13T00:48:53.700 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:52.490775+0000 mon.b (mon.1) 642 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_tier","val": "test-rados-api-smithi162-84057-85-test-flush"}]: dispatch 2024-02-13T00:48:53.700 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:52.491212+0000 mon.a (mon.0) 2800 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_tier","val": "test-rados-api-smithi162-84057-85-test-flush"}]: dispatch 2024-02-13T00:48:53.700 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:52.465663+0000 mgr.y (mgr.14544) 458 : cluster [DBG] pgmap v776: 266 pgs: 32 unknown, 32 creating+peering, 202 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-02-13T00:48:53.701 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:52.484361+0000 mon.a (mon.0) 2798 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:48:53.701 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:52.484440+0000 mon.a (mon.0) 2799 : cluster [DBG] osdmap e518: 8 total, 8 up, 4 in 2024-02-13T00:48:53.701 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:52.490775+0000 mon.b (mon.1) 642 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_tier","val": "test-rados-api-smithi162-84057-85-test-flush"}]: dispatch 2024-02-13T00:48:53.701 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:52.491212+0000 mon.a (mon.0) 2800 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_tier","val": "test-rados-api-smithi162-84057-85-test-flush"}]: dispatch 2024-02-13T00:48:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:52.465663+0000 mgr.y (mgr.14544) 458 : cluster [DBG] pgmap v776: 266 pgs: 32 unknown, 32 creating+peering, 202 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-02-13T00:48:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:52.484361+0000 mon.a (mon.0) 2798 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:48:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:52.484440+0000 mon.a (mon.0) 2799 : cluster [DBG] osdmap e518: 8 total, 8 up, 4 in 2024-02-13T00:48:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:52.490775+0000 mon.b (mon.1) 642 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_tier","val": "test-rados-api-smithi162-84057-85-test-flush"}]: dispatch 2024-02-13T00:48:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:52.491212+0000 mon.a (mon.0) 2800 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_tier","val": "test-rados-api-smithi162-84057-85-test-flush"}]: dispatch 2024-02-13T00:48:53.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:48:53.830 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:54.011 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:48:54.011 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:48:54.012 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:48:54.012 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:48:54.058 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.6 reset_purged_snaps_last 2024-02-13T00:48:54.079 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:48:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:54.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:48:54.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:54.214 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:54.236 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:54.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:54.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:54.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:54.368 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:54.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:52.822850+0000 osd.0 (osd.0) 197 : cluster [DBG] 5.1a scrub starts 2024-02-13T00:48:54.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:52.823680+0000 osd.0 (osd.0) 198 : cluster [DBG] 5.1a scrub ok 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:53.012354+0000 osd.1 (osd.1) 219 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:53.013099+0000 osd.1 (osd.1) 220 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:53.487558+0000 mon.a (mon.0) 2801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_tier","val": "test-rados-api-smithi162-84057-85-test-flush"}]': finished 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:53.487619+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e519: 8 total, 8 up, 4 in 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:53.489117+0000 mon.b (mon.1) 643 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:53.489445+0000 mon.a (mon.0) 2803 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:54.466400+0000 mon.c (mon.2) 412 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.a"}]: dispatch 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:54.466563+0000 mon.c (mon.2) 413 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.a"}]: dispatch 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:54.467462+0000 mon.a (mon.0) 2804 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.a"}]: dispatch 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:54.467704+0000 mon.a (mon.0) 2805 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.a"}]: dispatch 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:54.488700+0000 mon.a (mon.0) 2806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:54.488803+0000 mon.a (mon.0) 2807 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.a"}]': finished 2024-02-13T00:48:54.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:54.488853+0000 mon.a (mon.0) 2808 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.a"}]': finished 2024-02-13T00:48:54.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:54.488898+0000 mon.a (mon.0) 2809 : cluster [DBG] osdmap e520: 8 total, 8 up, 4 in 2024-02-13T00:48:54.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:52.822850+0000 osd.0 (osd.0) 197 : cluster [DBG] 5.1a scrub starts 2024-02-13T00:48:54.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:52.823680+0000 osd.0 (osd.0) 198 : cluster [DBG] 5.1a scrub ok 2024-02-13T00:48:54.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:53.012354+0000 osd.1 (osd.1) 219 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:48:54.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:53.013099+0000 osd.1 (osd.1) 220 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:48:54.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:53.487558+0000 mon.a (mon.0) 2801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_tier","val": "test-rados-api-smithi162-84057-85-test-flush"}]': finished 2024-02-13T00:48:54.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:53.487619+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e519: 8 total, 8 up, 4 in 2024-02-13T00:48:54.679 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:53.489117+0000 mon.b (mon.1) 643 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:54.680 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:53.489445+0000 mon.a (mon.0) 2803 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:54.680 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:54.466400+0000 mon.c (mon.2) 412 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.a"}]: dispatch 2024-02-13T00:48:54.680 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:54.466563+0000 mon.c (mon.2) 413 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.a"}]: dispatch 2024-02-13T00:48:54.680 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:54.467462+0000 mon.a (mon.0) 2804 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.a"}]: dispatch 2024-02-13T00:48:54.680 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:54.467704+0000 mon.a (mon.0) 2805 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.a"}]: dispatch 2024-02-13T00:48:54.680 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:54.488700+0000 mon.a (mon.0) 2806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:48:54.680 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:54.488803+0000 mon.a (mon.0) 2807 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.a"}]': finished 2024-02-13T00:48:54.680 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:54.488853+0000 mon.a (mon.0) 2808 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.a"}]': finished 2024-02-13T00:48:54.680 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:54.488898+0000 mon.a (mon.0) 2809 : cluster [DBG] osdmap e520: 8 total, 8 up, 4 in 2024-02-13T00:48:54.680 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:54.491438+0000 mon.b (mon.1) 644 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:54.680 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:54.492263+0000 mon.a (mon.0) 2810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:54.681 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:54.491438+0000 mon.b (mon.1) 644 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:54.681 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:54.492263+0000 mon.a (mon.0) 2810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:54.715 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:54.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:52.822850+0000 osd.0 (osd.0) 197 : cluster [DBG] 5.1a scrub starts 2024-02-13T00:48:54.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:52.823680+0000 osd.0 (osd.0) 198 : cluster [DBG] 5.1a scrub ok 2024-02-13T00:48:54.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:53.012354+0000 osd.1 (osd.1) 219 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:48:54.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:53.013099+0000 osd.1 (osd.1) 220 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:48:54.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:53.487558+0000 mon.a (mon.0) 2801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_tier","val": "test-rados-api-smithi162-84057-85-test-flush"}]': finished 2024-02-13T00:48:54.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:53.487619+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e519: 8 total, 8 up, 4 in 2024-02-13T00:48:54.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:53.489117+0000 mon.b (mon.1) 643 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:54.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:53.489445+0000 mon.a (mon.0) 2803 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:48:54.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:54.466400+0000 mon.c (mon.2) 412 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.a"}]: dispatch 2024-02-13T00:48:54.717 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:54.466563+0000 mon.c (mon.2) 413 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.a"}]: dispatch 2024-02-13T00:48:54.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:54.467462+0000 mon.a (mon.0) 2804 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.a"}]: dispatch 2024-02-13T00:48:54.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:54.467704+0000 mon.a (mon.0) 2805 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.a"}]: dispatch 2024-02-13T00:48:54.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:54.488700+0000 mon.a (mon.0) 2806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:48:54.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:54.488803+0000 mon.a (mon.0) 2807 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.a"}]': finished 2024-02-13T00:48:54.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:54.488853+0000 mon.a (mon.0) 2808 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.a"}]': finished 2024-02-13T00:48:54.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:54.488898+0000 mon.a (mon.0) 2809 : cluster [DBG] osdmap e520: 8 total, 8 up, 4 in 2024-02-13T00:48:54.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:54.491438+0000 mon.b (mon.1) 644 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:54.718 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:54.492263+0000 mon.a (mon.0) 2810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:48:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:54.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:54.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-13T00:48:54.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:48:54.848 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:54.866 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:48:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:54.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:48:54.970 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:48:54.998 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:55.007 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:48:55.018 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:48:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:55.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:48:55.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:55.151 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:55.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.171+0000 7f4cb7d10700 1 -- 172.21.15.162:0/4101513527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cb00f8d10 msgr2=0x7f4cb00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.171+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/4101513527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cb00f8d10 0x7f4cb00f90f0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 rx=0x7f4cac004870 tx=0x7f4cac009cb0).stop 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.171+0000 7f4cb7d10700 1 -- 172.21.15.162:0/4101513527 shutdown_connections 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.171+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/4101513527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cb0102270 0x7f4cb0104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.171+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/4101513527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cb00f8d10 0x7f4cb00f90f0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.171+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/4101513527 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4cb00f95e0 0x7f4cb0101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.171+0000 7f4cb7d10700 1 -- 172.21.15.162:0/4101513527 >> 172.21.15.162:0/4101513527 conn(0x7f4cb006eb90 msgr2=0x7f4cb00f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.171+0000 7f4cb7d10700 1 -- 172.21.15.162:0/4101513527 shutdown_connections 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.171+0000 7f4cb7d10700 1 -- 172.21.15.162:0/4101513527 wait complete. 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb7d10700 1 librados: starting msgr at 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb7d10700 1 librados: starting objecter 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb7d10700 1 Processor -- start 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb7d10700 1 -- start start 2024-02-13T00:48:55.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb7d10700 1 librados: setting wanted keys 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb7d10700 1 librados: calling monclient init 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb7d10700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4cb00f8d10 0x7f4cb00f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb7d10700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cb00f95e0 0x7f4cb00f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb7d10700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cb0102270 0x7f4cb00f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb7d10700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb01166e0 con 0x7f4cb0102270 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb7d10700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb0116850 con 0x7f4cb00f95e0 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb7d10700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb0116a10 con 0x7f4cb00f8d10 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb5aac700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4cb00f8d10 0x7f4cb00f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb5aac700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4cb00f8d10 0x7f4cb00f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:36364/0 (socket says 172.21.15.162:36364) 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb5aac700 1 -- 172.21.15.162:0/3425721080 learned_addr learned my addr 172.21.15.162:0/3425721080 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb62ad700 1 --2- 172.21.15.162:0/3425721080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cb0102270 0x7f4cb00f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.172+0000 7f4cb52ab700 1 --2- 172.21.15.162:0/3425721080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cb00f95e0 0x7f4cb00f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:55.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4cb5aac700 1 -- 172.21.15.162:0/3425721080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cb00f95e0 msgr2=0x7f4cb00f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4cb5aac700 1 --2- 172.21.15.162:0/3425721080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cb00f95e0 0x7f4cb00f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4cb5aac700 1 -- 172.21.15.162:0/3425721080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cb0102270 msgr2=0x7f4cb00f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4cb5aac700 1 --2- 172.21.15.162:0/3425721080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cb0102270 0x7f4cb00f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4cb5aac700 1 -- 172.21.15.162:0/3425721080 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca0005040 con 0x7f4cb00f8d10 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4cb52ab700 1 --2- 172.21.15.162:0/3425721080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cb00f95e0 0x7f4cb00f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4cb62ad700 1 --2- 172.21.15.162:0/3425721080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cb0102270 0x7f4cb00f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4cb5aac700 1 --2- 172.21.15.162:0/3425721080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4cb00f8d10 0x7f4cb00f6cc0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7f4cac0043a0 tx=0x7f4cac00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4ca6ffd700 10 client.?.objecter ms_handle_connect 0x7f4cb00f8d10 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4ca6ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac00e070 con 0x7f4cb00f8d10 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cac005040 con 0x7f4cb00f8d10 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4cac009710 con 0x7f4cb00f8d10 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac019070 con 0x7f4cb00f8d10 2024-02-13T00:48:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4cb7d10700 10 client.37697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:55.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb0116f00 con 0x7f4cb00f8d10 2024-02-13T00:48:55.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.173+0000 7f4cb7d10700 1 librados: init done 2024-02-13T00:48:55.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.174+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4cb0107170 con 0x7f4cb00f8d10 2024-02-13T00:48:55.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.175+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4cac0090d0 con 0x7f4cb00f8d10 2024-02-13T00:48:55.181 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:55.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.175+0000 7f4ca6ffd700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4c9c062c90 0x7f4c9c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.175+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(520..520 src has 449..520) v4 ==== 8488+0+0 (secure 0 0 0) 0x7f4cac07db60 con 0x7f4cb00f8d10 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.175+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 osd_map(520..520 src has 449..520) v4 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.175+0000 7f4ca6ffd700 3 client.37697.objecter handle_osd_map got epochs [520,520] > 0 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.175+0000 7f4ca6ffd700 3 client.37697.objecter handle_osd_map decoding full epoch 520 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.175+0000 7f4ca6ffd700 20 client.37697.objecter dump_active .. 0 homeless 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.175+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v520) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4cac050e80 con 0x7f4cb00f8d10 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb52ab700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4c9c062c90 0x7f4c9c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 1 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4cb0028590 0x7f4cb00fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb00f81a0 osd=0 3 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb00f81a0 osd=0 3 2024-02-13T00:48:55.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb00f81a0 osd=0 3 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 1 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb00f81a0 osd=0 3 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 0 1 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb00f81a0 osd=0 4 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb7d10700 10 client.37697.objecter _send_command 1 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4cb005a0e0 con 0x7f4cb0028590 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb62ad700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4cb0028590 0x7f4cb00fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4ca6ffd700 10 client.37697.objecter ms_handle_connect 0x7f4c9c062c90 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb52ab700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4c9c062c90 0x7f4c9c065150 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 rx=0x7f4ca0007fd0 tx=0x7f4ca0009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4ca6ffd700 10 client.37697.objecter ms_handle_connect 0x7f4cb0028590 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.176+0000 7f4cb62ad700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4cb0028590 0x7f4cb00fbd00 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.177+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4cb005a0e0 con 0x7f4cb0028590 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.177+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:48:55.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.177+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 1 = system:0 2024-02-13T00:48:55.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.177+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb00f81a0 osd=0 4 2024-02-13T00:48:55.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.177+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 0 1 2024-02-13T00:48:55.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:55.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:55.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.195+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:55.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.195+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 2 2024-02-13T00:48:55.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.195+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb00f81a0 osd=0 3 2024-02-13T00:48:55.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.195+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb00f81a0 osd=0 3 2024-02-13T00:48:55.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.195+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb00f81a0 osd=0 3 2024-02-13T00:48:55.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.195+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 2 2024-02-13T00:48:55.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.195+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb00f81a0 osd=0 3 2024-02-13T00:48:55.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.195+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 0 2 2024-02-13T00:48:55.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.195+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb00f81a0 osd=0 4 2024-02-13T00:48:55.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.195+0000 7f4cb7d10700 10 client.37697.objecter _send_command 2 2024-02-13T00:48:55.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.195+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4cb00fc1f0 con 0x7f4cb0028590 2024-02-13T00:48:55.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.196+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4cb00fc1f0 con 0x7f4cb0028590 2024-02-13T00:48:55.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.196+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:55.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.196+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.196+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb00f81a0 osd=0 4 2024-02-13T00:48:55.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.196+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 0 2 2024-02-13T00:48:55.200 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:48:55.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:55.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 3 2024-02-13T00:48:55.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6805/150599780 conn(0x7f4cb00597b0 0x7f4cb011b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:55.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0059640 osd=1 3 2024-02-13T00:48:55.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0059640 osd=1 3 2024-02-13T00:48:55.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0059640 osd=1 3 2024-02-13T00:48:55.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 3 2024-02-13T00:48:55.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb0059640 osd=1 3 2024-02-13T00:48:55.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 1 3 2024-02-13T00:48:55.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0059640 osd=1 4 2024-02-13T00:48:55.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb7d10700 10 client.37697.objecter _send_command 3 2024-02-13T00:48:55.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4cb00280f0 con 0x7f4cb00597b0 2024-02-13T00:48:55.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb5aac700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6805/150599780 conn(0x7f4cb00597b0 0x7f4cb011b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:55.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4ca6ffd700 10 client.37697.objecter ms_handle_connect 0x7f4cb00597b0 2024-02-13T00:48:55.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.197+0000 7f4cb5aac700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6805/150599780 conn(0x7f4cb00597b0 0x7f4cb011b0b0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:55.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.198+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4cb00280f0 con 0x7f4cb00597b0 2024-02-13T00:48:55.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.198+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:48:55.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.198+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 3 = system:0 2024-02-13T00:48:55.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.198+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb0059640 osd=1 4 2024-02-13T00:48:55.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.198+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 1 3 2024-02-13T00:48:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.213+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.214+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 4 2024-02-13T00:48:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.214+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0059640 osd=1 3 2024-02-13T00:48:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.214+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0059640 osd=1 3 2024-02-13T00:48:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.214+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0059640 osd=1 3 2024-02-13T00:48:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.214+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 4 2024-02-13T00:48:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.214+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb0059640 osd=1 3 2024-02-13T00:48:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.214+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 1 4 2024-02-13T00:48:55.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.214+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0059640 osd=1 4 2024-02-13T00:48:55.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.214+0000 7f4cb7d10700 10 client.37697.objecter _send_command 4 2024-02-13T00:48:55.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.214+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4cb011b5a0 con 0x7f4cb00597b0 2024-02-13T00:48:55.218 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4cb011b5a0 con 0x7f4cb00597b0 2024-02-13T00:48:55.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:55.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb0059640 osd=1 4 2024-02-13T00:48:55.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 1 4 2024-02-13T00:48:55.219 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:48:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 5 2024-02-13T00:48:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4cb00277c0 0x7f4cb00fd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0027650 osd=2 3 2024-02-13T00:48:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0027650 osd=2 3 2024-02-13T00:48:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0027650 osd=2 3 2024-02-13T00:48:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 5 2024-02-13T00:48:55.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb0027650 osd=2 3 2024-02-13T00:48:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 2 5 2024-02-13T00:48:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0027650 osd=2 4 2024-02-13T00:48:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4cb7d10700 10 client.37697.objecter _send_command 5 2024-02-13T00:48:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.215+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4cb0121060 con 0x7f4cb00277c0 2024-02-13T00:48:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.216+0000 7f4cb52ab700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4cb00277c0 0x7f4cb00fd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.216+0000 7f4ca6ffd700 10 client.37697.objecter ms_handle_connect 0x7f4cb00277c0 2024-02-13T00:48:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.216+0000 7f4cb52ab700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4cb00277c0 0x7f4cb00fd0a0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.217+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4cb0121060 con 0x7f4cb00277c0 2024-02-13T00:48:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.217+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:48:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.217+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 5 = system:0 2024-02-13T00:48:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.217+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb0027650 osd=2 4 2024-02-13T00:48:55.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.217+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 2 5 2024-02-13T00:48:55.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.240+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:55.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.240+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 6 2024-02-13T00:48:55.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.240+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0027650 osd=2 3 2024-02-13T00:48:55.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.240+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0027650 osd=2 3 2024-02-13T00:48:55.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.240+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0027650 osd=2 3 2024-02-13T00:48:55.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.240+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 6 2024-02-13T00:48:55.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.240+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb0027650 osd=2 3 2024-02-13T00:48:55.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.240+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 2 6 2024-02-13T00:48:55.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.240+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0027650 osd=2 4 2024-02-13T00:48:55.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.240+0000 7f4cb7d10700 10 client.37697.objecter _send_command 6 2024-02-13T00:48:55.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.240+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4cb00fc6c0 con 0x7f4cb00277c0 2024-02-13T00:48:55.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.241+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4cb00fc6c0 con 0x7f4cb00277c0 2024-02-13T00:48:55.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.241+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:55.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.241+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.241+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb0027650 osd=2 4 2024-02-13T00:48:55.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.241+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 2 6 2024-02-13T00:48:55.245 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:48:55.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:55.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 7 2024-02-13T00:48:55.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6813/604971011 conn(0x7f4cb011fa80 0x7f4cb011fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:55.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb00fc940 osd=3 3 2024-02-13T00:48:55.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb00fc940 osd=3 3 2024-02-13T00:48:55.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb00fc940 osd=3 3 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 7 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb00fc940 osd=3 3 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 3 7 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb00fc940 osd=3 4 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb7d10700 10 client.37697.objecter _send_command 7 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4cb011c730 con 0x7f4cb011fa80 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb62ad700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6813/604971011 conn(0x7f4cb011fa80 0x7f4cb011fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4ca6ffd700 10 client.37697.objecter ms_handle_connect 0x7f4cb011fa80 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.242+0000 7f4cb62ad700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6813/604971011 conn(0x7f4cb011fa80 0x7f4cb011fee0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.243+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4cb011c730 con 0x7f4cb011fa80 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.243+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.243+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 7 = system:0 2024-02-13T00:48:55.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.243+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb00fc940 osd=3 4 2024-02-13T00:48:55.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.243+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 3 7 2024-02-13T00:48:55.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.259+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:55.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.259+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 8 2024-02-13T00:48:55.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.259+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb00fc940 osd=3 3 2024-02-13T00:48:55.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.259+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb00fc940 osd=3 3 2024-02-13T00:48:55.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.259+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb00fc940 osd=3 3 2024-02-13T00:48:55.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.259+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 8 2024-02-13T00:48:55.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.259+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb00fc940 osd=3 3 2024-02-13T00:48:55.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.259+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 3 8 2024-02-13T00:48:55.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.259+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb00fc940 osd=3 4 2024-02-13T00:48:55.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.259+0000 7f4cb7d10700 10 client.37697.objecter _send_command 8 2024-02-13T00:48:55.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.259+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4cb0120c20 con 0x7f4cb011fa80 2024-02-13T00:48:55.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.260+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4cb0120c20 con 0x7f4cb011fa80 2024-02-13T00:48:55.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.260+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:55.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.260+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.260+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb00fc940 osd=3 4 2024-02-13T00:48:55.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.260+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 3 8 2024-02-13T00:48:55.264 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:48:55.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:55.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 9 2024-02-13T00:48:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4cb0123210 0x7f4cb0125600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0120450 osd=4 3 2024-02-13T00:48:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0120450 osd=4 3 2024-02-13T00:48:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0120450 osd=4 3 2024-02-13T00:48:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 9 2024-02-13T00:48:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb0120450 osd=4 3 2024-02-13T00:48:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 4 9 2024-02-13T00:48:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0120450 osd=4 4 2024-02-13T00:48:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb7d10700 10 client.37697.objecter _send_command 9 2024-02-13T00:48:55.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4cb0128250 con 0x7f4cb0123210 2024-02-13T00:48:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb5aac700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4cb0123210 0x7f4cb0125600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4ca6ffd700 10 client.37697.objecter ms_handle_connect 0x7f4cb0123210 2024-02-13T00:48:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.261+0000 7f4cb5aac700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4cb0123210 0x7f4cb0125600 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.262+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4cb0128250 con 0x7f4cb0123210 2024-02-13T00:48:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.262+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:48:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.262+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 9 = system:0 2024-02-13T00:48:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.262+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb0120450 osd=4 4 2024-02-13T00:48:55.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.262+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 4 9 2024-02-13T00:48:55.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.277+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:55.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.277+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 10 2024-02-13T00:48:55.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.277+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0120450 osd=4 3 2024-02-13T00:48:55.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.277+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0120450 osd=4 3 2024-02-13T00:48:55.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.277+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0120450 osd=4 3 2024-02-13T00:48:55.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.277+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 10 2024-02-13T00:48:55.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.277+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb0120450 osd=4 3 2024-02-13T00:48:55.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.277+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 4 10 2024-02-13T00:48:55.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.277+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0120450 osd=4 4 2024-02-13T00:48:55.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.277+0000 7f4cb7d10700 10 client.37697.objecter _send_command 10 2024-02-13T00:48:55.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.277+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4cb0125af0 con 0x7f4cb0123210 2024-02-13T00:48:55.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.278+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4cb0125af0 con 0x7f4cb0123210 2024-02-13T00:48:55.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.278+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:55.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb0120450 osd=4 4 2024-02-13T00:48:55.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 4 10 2024-02-13T00:48:55.283 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:48:55.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 11 2024-02-13T00:48:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4cb0126170 0x7f4cb0128560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0125af0 osd=5 3 2024-02-13T00:48:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0125af0 osd=5 3 2024-02-13T00:48:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0125af0 osd=5 3 2024-02-13T00:48:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 11 2024-02-13T00:48:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb0125af0 osd=5 3 2024-02-13T00:48:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 5 11 2024-02-13T00:48:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0125af0 osd=5 4 2024-02-13T00:48:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb7d10700 10 client.37697.objecter _send_command 11 2024-02-13T00:48:55.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4cb012b360 con 0x7f4cb0126170 2024-02-13T00:48:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb52ab700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4cb0126170 0x7f4cb0128560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4cb52ab700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4cb0126170 0x7f4cb0128560 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.279+0000 7f4ca6ffd700 10 client.37697.objecter ms_handle_connect 0x7f4cb0126170 2024-02-13T00:48:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.280+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4cb012b360 con 0x7f4cb0126170 2024-02-13T00:48:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.280+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:48:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.280+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 11 = system:0 2024-02-13T00:48:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.280+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb0125af0 osd=5 4 2024-02-13T00:48:55.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.280+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 5 11 2024-02-13T00:48:55.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:55.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.295+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:55.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.295+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 12 2024-02-13T00:48:55.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.295+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0125af0 osd=5 3 2024-02-13T00:48:55.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.295+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0125af0 osd=5 3 2024-02-13T00:48:55.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.295+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb0125af0 osd=5 3 2024-02-13T00:48:55.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.295+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 12 2024-02-13T00:48:55.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.295+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb0125af0 osd=5 3 2024-02-13T00:48:55.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.295+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 5 12 2024-02-13T00:48:55.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.295+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0125af0 osd=5 4 2024-02-13T00:48:55.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.295+0000 7f4cb7d10700 10 client.37697.objecter _send_command 12 2024-02-13T00:48:55.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.295+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4cb012b360 con 0x7f4cb0126170 2024-02-13T00:48:55.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.297+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4ca001f020 con 0x7f4cb0126170 2024-02-13T00:48:55.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.297+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:55.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.297+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.297+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb0125af0 osd=5 4 2024-02-13T00:48:55.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.297+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 5 12 2024-02-13T00:48:55.301 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:48:55.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:55.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 13 2024-02-13T00:48:55.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4cb012a7d0 0x7f4cb012ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:55.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb012a150 osd=6 3 2024-02-13T00:48:55.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb012a150 osd=6 3 2024-02-13T00:48:55.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb012a150 osd=6 3 2024-02-13T00:48:55.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 13 2024-02-13T00:48:55.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb012a150 osd=6 3 2024-02-13T00:48:55.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 6 13 2024-02-13T00:48:55.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb012a150 osd=6 4 2024-02-13T00:48:55.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb7d10700 10 client.37697.objecter _send_command 13 2024-02-13T00:48:55.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4cb012e790 con 0x7f4cb012a7d0 2024-02-13T00:48:55.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb62ad700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4cb012a7d0 0x7f4cb012ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:55.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4ca6ffd700 10 client.37697.objecter ms_handle_connect 0x7f4cb012a7d0 2024-02-13T00:48:55.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.298+0000 7f4cb62ad700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4cb012a7d0 0x7f4cb012ac30 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:55.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.299+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4cb012e790 con 0x7f4cb012a7d0 2024-02-13T00:48:55.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.299+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:48:55.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.299+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 13 = system:0 2024-02-13T00:48:55.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.299+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb012a150 osd=6 4 2024-02-13T00:48:55.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.299+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 6 13 2024-02-13T00:48:55.312 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:55.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.314+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:55.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.314+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 14 2024-02-13T00:48:55.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.314+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb012a150 osd=6 3 2024-02-13T00:48:55.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.314+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb012a150 osd=6 3 2024-02-13T00:48:55.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.314+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb012a150 osd=6 3 2024-02-13T00:48:55.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.314+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 14 2024-02-13T00:48:55.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.314+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb012a150 osd=6 3 2024-02-13T00:48:55.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.314+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 6 14 2024-02-13T00:48:55.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.314+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb012a150 osd=6 4 2024-02-13T00:48:55.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.314+0000 7f4cb7d10700 10 client.37697.objecter _send_command 14 2024-02-13T00:48:55.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.314+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4cb012d610 con 0x7f4cb012a7d0 2024-02-13T00:48:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4cb012d610 con 0x7f4cb012a7d0 2024-02-13T00:48:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb012a150 osd=6 4 2024-02-13T00:48:55.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 6 14 2024-02-13T00:48:55.320 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:48:55.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:55.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 15 2024-02-13T00:48:55.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4cb012dcd0 0x7f4cb012e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:55.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb012d610 osd=7 3 2024-02-13T00:48:55.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb012d610 osd=7 3 2024-02-13T00:48:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb012d610 osd=7 3 2024-02-13T00:48:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 15 2024-02-13T00:48:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb012d610 osd=7 3 2024-02-13T00:48:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 7 15 2024-02-13T00:48:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb012d610 osd=7 4 2024-02-13T00:48:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4cb7d10700 10 client.37697.objecter _send_command 15 2024-02-13T00:48:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.316+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4cb012e780 con 0x7f4cb012dcd0 2024-02-13T00:48:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.317+0000 7f4cb5aac700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4cb012dcd0 0x7f4cb012e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.317+0000 7f4ca6ffd700 10 client.37697.objecter ms_handle_connect 0x7f4cb012dcd0 2024-02-13T00:48:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.317+0000 7f4cb5aac700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4cb012dcd0 0x7f4cb012e110 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.318+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4cac07b130 con 0x7f4cb012dcd0 2024-02-13T00:48:55.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.318+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:48:55.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.318+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 15 = system:0 2024-02-13T00:48:55.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.318+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb012d610 osd=7 4 2024-02-13T00:48:55.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.318+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 7 15 2024-02-13T00:48:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.333+0000 7f4cb7d10700 10 client.37697.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:48:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.333+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign -1 16 2024-02-13T00:48:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.333+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb012d610 osd=7 3 2024-02-13T00:48:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.333+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb012d610 osd=7 3 2024-02-13T00:48:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.333+0000 7f4cb7d10700 20 client.37697.objecter _get_session s=0x7f4cb012d610 osd=7 3 2024-02-13T00:48:55.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.333+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_remove -1 16 2024-02-13T00:48:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.333+0000 7f4cb7d10700 20 client.37697.objecter get_session s=0x7f4cb012d610 osd=7 3 2024-02-13T00:48:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.333+0000 7f4cb7d10700 15 client.37697.objecter _session_command_op_assign 7 16 2024-02-13T00:48:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.333+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb012d610 osd=7 4 2024-02-13T00:48:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.333+0000 7f4cb7d10700 10 client.37697.objecter _send_command 16 2024-02-13T00:48:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.333+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4cb01309f0 con 0x7f4cb012dcd0 2024-02-13T00:48:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4ca6ffd700 1 -- 172.21.15.162:0/3425721080 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4cb01309f0 con 0x7f4cb012dcd0 2024-02-13T00:48:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4ca6ffd700 10 client.37697.objecter ms_dispatch 0x7f4cb0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:48:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4ca6ffd700 10 client.37697.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4ca6ffd700 20 client.37697.objecter put_session s=0x7f4cb012d610 osd=7 4 2024-02-13T00:48:55.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4ca6ffd700 15 client.37697.objecter _session_command_op_remove 7 16 2024-02-13T00:48:55.338 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:48:55.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 10 librados: watch_flush enter 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 10 librados: watch_flush exit 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 10 client.37697.objecter close_session for osd.0 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4cb0028590 msgr2=0x7f4cb00fbd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4cb0028590 0x7f4cb00fbd00 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb00f81a0 osd=0 1 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 10 client.37697.objecter close_session for osd.1 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6805/150599780 conn(0x7f4cb00597b0 msgr2=0x7f4cb011b0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6805/150599780 conn(0x7f4cb00597b0 0x7f4cb011b0b0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0059640 osd=1 1 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 10 client.37697.objecter close_session for osd.2 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4cb00277c0 msgr2=0x7f4cb00fd0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4cb00277c0 0x7f4cb00fd0a0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0027650 osd=2 1 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 10 client.37697.objecter close_session for osd.3 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6813/604971011 conn(0x7f4cb011fa80 msgr2=0x7f4cb011fee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6813/604971011 conn(0x7f4cb011fa80 0x7f4cb011fee0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb62ad700 1 -- 172.21.15.162:0/3425721080 reap_dead start 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb00fc940 osd=3 1 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 10 client.37697.objecter close_session for osd.4 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4cb0123210 msgr2=0x7f4cb0125600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4cb0123210 0x7f4cb0125600 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0120450 osd=4 1 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 10 client.37697.objecter close_session for osd.5 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4cb0126170 msgr2=0x7f4cb0128560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4cb0126170 0x7f4cb0128560 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb0125af0 osd=5 1 2024-02-13T00:48:55.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 10 client.37697.objecter close_session for osd.6 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4cb012a7d0 msgr2=0x7f4cb012ac30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4cb012a7d0 0x7f4cb012ac30 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb012a150 osd=6 1 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 10 client.37697.objecter close_session for osd.7 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4cb012dcd0 msgr2=0x7f4cb012e110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4cb012dcd0 0x7f4cb012e110 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 20 client.37697.objecter put_session s=0x7f4cb012d610 osd=7 1 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 20 client.37697.objecter shutdown clearing up homeless session... 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 10 client.37697.objecter successfully canceled tick 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb62ad700 1 -- 172.21.15.162:0/3425721080 reap_dead start 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4c9c062c90 msgr2=0x7f4c9c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4c9c062c90 0x7f4c9c065150 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 rx=0x7f4ca0007fd0 tx=0x7f4ca0009cb0).stop 2024-02-13T00:48:55.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4cb00f8d10 msgr2=0x7f4cb00f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4cb00f8d10 0x7f4cb00f6cc0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7f4cac0043a0 tx=0x7f4cac00b040).stop 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 shutdown_connections 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4cb00f8d10 0x7f4cb00f6cc0 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 --2- 172.21.15.162:0/3425721080 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4c9c062c90 0x7f4c9c065150 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 >> 172.21.15.162:0/3425721080 conn(0x7f4cb006eb90 msgr2=0x7f4cb0102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 shutdown_connections 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 -- 172.21.15.162:0/3425721080 wait complete. 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:55.335+0000 7f4cb7d10700 1 librados: shutdown 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.343 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.344 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:48:55.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:48:55.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:53.838193+0000 osd.0 (osd.0) 199 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:48:55.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:53.841791+0000 osd.0 (osd.0) 200 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:48:55.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:54.041295+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.f scrub starts 2024-02-13T00:48:55.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:54.043082+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.f scrub ok 2024-02-13T00:48:55.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:54.466085+0000 mgr.y (mgr.14544) 459 : cluster [DBG] pgmap v779: 265 pgs: 32 unknown, 32 creating+peering, 201 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:48:55.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:55.176033+0000 mon.c (mon.2) 414 : audit [DBG] from='client.? 172.21.15.162:0/3425721080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:55.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:55.472229+0000 mon.a (mon.0) 2811 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:55.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:55.472306+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e521: 8 total, 8 up, 4 in 2024-02-13T00:48:55.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:53.838193+0000 osd.0 (osd.0) 199 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:48:55.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:53.841791+0000 osd.0 (osd.0) 200 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:48:55.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:54.041295+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.f scrub starts 2024-02-13T00:48:55.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:54.043082+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.f scrub ok 2024-02-13T00:48:55.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:54.466085+0000 mgr.y (mgr.14544) 459 : cluster [DBG] pgmap v779: 265 pgs: 32 unknown, 32 creating+peering, 201 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:48:55.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:55.176033+0000 mon.c (mon.2) 414 : audit [DBG] from='client.? 172.21.15.162:0/3425721080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:55.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:55.472229+0000 mon.a (mon.0) 2811 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:55.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:55.472306+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e521: 8 total, 8 up, 4 in 2024-02-13T00:48:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:53.838193+0000 osd.0 (osd.0) 199 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:48:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:53.841791+0000 osd.0 (osd.0) 200 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:48:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:54.041295+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.f scrub starts 2024-02-13T00:48:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:54.043082+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.f scrub ok 2024-02-13T00:48:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:54.466085+0000 mgr.y (mgr.14544) 459 : cluster [DBG] pgmap v779: 265 pgs: 32 unknown, 32 creating+peering, 201 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:48:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:55.176033+0000 mon.c (mon.2) 414 : audit [DBG] from='client.? 172.21.15.162:0/3425721080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:48:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:55.472229+0000 mon.a (mon.0) 2811 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:48:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:55.472306+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e521: 8 total, 8 up, 4 in 2024-02-13T00:48:56.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:48:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:48:56.515 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:48:56] "GET /metrics HTTP/1.1" 200 185697 "" "Prometheus/2.33.4" 2024-02-13T00:48:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:54.815075+0000 osd.0 (osd.0) 201 : cluster [DBG] 2.1f scrub starts 2024-02-13T00:48:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:54.817988+0000 osd.0 (osd.0) 202 : cluster [DBG] 2.1f scrub ok 2024-02-13T00:48:56.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:55.049650+0000 osd.1 (osd.1) 223 : cluster [DBG] 28.1e scrub starts 2024-02-13T00:48:56.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:55.050407+0000 osd.1 (osd.1) 224 : cluster [DBG] 28.1e scrub ok 2024-02-13T00:48:56.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:55.422100+0000 osd.6 (osd.6) 155 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:48:56.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:55.423947+0000 osd.6 (osd.6) 156 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:48:56.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:55.716661+0000 mgr.y (mgr.14544) 460 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:56.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:56.481436+0000 mon.a (mon.0) 2813 : cluster [DBG] osdmap e522: 8 total, 8 up, 4 in 2024-02-13T00:48:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:54.815075+0000 osd.0 (osd.0) 201 : cluster [DBG] 2.1f scrub starts 2024-02-13T00:48:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:54.817988+0000 osd.0 (osd.0) 202 : cluster [DBG] 2.1f scrub ok 2024-02-13T00:48:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:55.049650+0000 osd.1 (osd.1) 223 : cluster [DBG] 28.1e scrub starts 2024-02-13T00:48:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:55.050407+0000 osd.1 (osd.1) 224 : cluster [DBG] 28.1e scrub ok 2024-02-13T00:48:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:55.422100+0000 osd.6 (osd.6) 155 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:48:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:55.423947+0000 osd.6 (osd.6) 156 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:48:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:55.716661+0000 mgr.y (mgr.14544) 460 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:56.481436+0000 mon.a (mon.0) 2813 : cluster [DBG] osdmap e522: 8 total, 8 up, 4 in 2024-02-13T00:48:56.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:54.815075+0000 osd.0 (osd.0) 201 : cluster [DBG] 2.1f scrub starts 2024-02-13T00:48:56.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:54.817988+0000 osd.0 (osd.0) 202 : cluster [DBG] 2.1f scrub ok 2024-02-13T00:48:56.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:55.049650+0000 osd.1 (osd.1) 223 : cluster [DBG] 28.1e scrub starts 2024-02-13T00:48:56.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:55.050407+0000 osd.1 (osd.1) 224 : cluster [DBG] 28.1e scrub ok 2024-02-13T00:48:56.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:55.422100+0000 osd.6 (osd.6) 155 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:48:56.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:55.423947+0000 osd.6 (osd.6) 156 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:48:56.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:55.716661+0000 mgr.y (mgr.14544) 460 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:48:56.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:56.481436+0000 mon.a (mon.0) 2813 : cluster [DBG] osdmap e522: 8 total, 8 up, 4 in 2024-02-13T00:48:56.863 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:48:56.863 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:48:56.962 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:56.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:57.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:57.097 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:57.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.167+0000 7f08c66e6700 1 -- 172.21.15.162:0/587581398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c00f8d10 msgr2=0x7f08c00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:57.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.167+0000 7f08c66e6700 1 --2- 172.21.15.162:0/587581398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c00f8d10 0x7f08c00f90f0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7f08b0004870 tx=0x7f08b0009cb0).stop 2024-02-13T00:48:57.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 -- 172.21.15.162:0/587581398 shutdown_connections 2024-02-13T00:48:57.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 --2- 172.21.15.162:0/587581398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08c0102270 0x7f08c0104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:57.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 --2- 172.21.15.162:0/587581398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c00f8d10 0x7f08c00f90f0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:57.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 --2- 172.21.15.162:0/587581398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08c00f95e0 0x7f08c0101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:57.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 -- 172.21.15.162:0/587581398 >> 172.21.15.162:0/587581398 conn(0x7f08c006eb90 msgr2=0x7f08c00f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:57.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 -- 172.21.15.162:0/587581398 shutdown_connections 2024-02-13T00:48:57.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 -- 172.21.15.162:0/587581398 wait complete. 2024-02-13T00:48:57.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 librados: starting msgr at 2024-02-13T00:48:57.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 librados: starting objecter 2024-02-13T00:48:57.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 Processor -- start 2024-02-13T00:48:57.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 -- start start 2024-02-13T00:48:57.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 librados: setting wanted keys 2024-02-13T00:48:57.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.168+0000 7f08c66e6700 1 librados: calling monclient init 2024-02-13T00:48:57.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08c66e6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c00f8d10 0x7f08c010e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:57.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08c66e6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08c00f95e0 0x7f08c010e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:57.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08c66e6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08c0102270 0x7f08c0108310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:57.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08c66e6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c0108800 con 0x7f08c00f8d10 2024-02-13T00:48:57.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08c66e6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c0108970 con 0x7f08c0102270 2024-02-13T00:48:57.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08c66e6700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c0108ae0 con 0x7f08c00f95e0 2024-02-13T00:48:57.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c00f8d10 0x7f08c010e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:57.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c00f8d10 0x7f08c010e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:50078/0 (socket says 172.21.15.162:50078) 2024-02-13T00:48:57.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bffff700 1 -- 172.21.15.162:0/1564933329 learned_addr learned my addr 172.21.15.162:0/1564933329 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:57.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08c4c83700 1 --2- 172.21.15.162:0/1564933329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08c0102270 0x7f08c0108310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:57.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bf7fe700 1 --2- 172.21.15.162:0/1564933329 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08c00f95e0 0x7f08c010e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:57.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bffff700 1 -- 172.21.15.162:0/1564933329 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08c00f95e0 msgr2=0x7f08c010e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:57.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bffff700 1 --2- 172.21.15.162:0/1564933329 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08c00f95e0 0x7f08c010e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:57.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bffff700 1 -- 172.21.15.162:0/1564933329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08c0102270 msgr2=0x7f08c0108310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:57.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bffff700 1 --2- 172.21.15.162:0/1564933329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08c0102270 0x7f08c0108310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:57.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bffff700 1 -- 172.21.15.162:0/1564933329 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b80097c0 con 0x7f08c00f8d10 2024-02-13T00:48:57.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bffff700 1 --2- 172.21.15.162:0/1564933329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c00f8d10 0x7f08c010e130 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f08b0009eb0 tx=0x7f08b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:57.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bd7fa700 10 client.?.objecter ms_handle_connect 0x7f08c00f8d10 2024-02-13T00:48:57.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bd7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:57.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bd7fa700 1 -- 172.21.15.162:0/1564933329 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b000e070 con 0x7f08c00f8d10 2024-02-13T00:48:57.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08c66e6700 1 -- 172.21.15.162:0/1564933329 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b0005040 con 0x7f08c00f8d10 2024-02-13T00:48:57.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bd7fa700 1 -- 172.21.15.162:0/1564933329 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08b0013da0 con 0x7f08c00f8d10 2024-02-13T00:48:57.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08bd7fa700 1 -- 172.21.15.162:0/1564933329 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b0019070 con 0x7f08c00f8d10 2024-02-13T00:48:57.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08c66e6700 10 client.27603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:57.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08c66e6700 1 -- 172.21.15.162:0/1564933329 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c0108fd0 con 0x7f08c00f8d10 2024-02-13T00:48:57.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.169+0000 7f08c66e6700 1 librados: init done 2024-02-13T00:48:57.178 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:57.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.171+0000 7f08bd7fa700 1 -- 172.21.15.162:0/1564933329 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f08b001e420 con 0x7f08c00f8d10 2024-02-13T00:48:57.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.171+0000 7f08c66e6700 10 client.27603.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:48:57.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.171+0000 7f08c66e6700 15 client.27603.objecter _session_command_op_assign -1 1 2024-02-13T00:48:57.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.171+0000 7f08c66e6700 20 client.27603.objecter _get_session osd=-1 returning homeless 2024-02-13T00:48:57.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.171+0000 7f08c66e6700 10 client.27603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:57.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.171+0000 7f08c66e6700 1 -- 172.21.15.162:0/1564933329 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f08c00475e0 con 0x7f08c00f8d10 2024-02-13T00:48:57.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.171+0000 7f08bd7fa700 1 --2- 172.21.15.162:0/1564933329 >> v2:172.21.15.162:6800/2866534053 conn(0x7f08a8062c90 0x7f08a8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:57.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.171+0000 7f08bd7fa700 1 -- 172.21.15.162:0/1564933329 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(522..522 src has 449..522) v4 ==== 8121+0+0 (secure 0 0 0) 0x7f08b007dc90 con 0x7f08c00f8d10 2024-02-13T00:48:57.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.171+0000 7f08bd7fa700 10 client.27603.objecter ms_dispatch 0x7f08c0001550 osd_map(522..522 src has 449..522) v4 2024-02-13T00:48:57.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.171+0000 7f08bd7fa700 3 client.27603.objecter handle_osd_map got epochs [522,522] > 0 2024-02-13T00:48:57.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.171+0000 7f08bd7fa700 3 client.27603.objecter handle_osd_map decoding full epoch 522 2024-02-13T00:48:57.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.171+0000 7f08bf7fe700 1 --2- 172.21.15.162:0/1564933329 >> v2:172.21.15.162:6800/2866534053 conn(0x7f08a8062c90 0x7f08a8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:57.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 10 client.27603.objecter checking command 1 2024-02-13T00:48:57.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 1 --2- 172.21.15.162:0/1564933329 >> v2:172.21.15.181:6808/1506640341 conn(0x7f08a806fb20 0x7f08a8071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:57.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 20 client.27603.objecter _get_session s=0x7f08a806f470 osd=6 3 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 20 client.27603.objecter put_session s=0x7f08a806f470 osd=6 3 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 15 client.27603.objecter _session_command_op_remove -1 1 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 20 client.27603.objecter _get_session s=0x7f08a806f470 osd=6 3 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 20 client.27603.objecter get_session s=0x7f08a806f470 osd=6 3 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 15 client.27603.objecter _session_command_op_assign 6 1 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 20 client.27603.objecter put_session s=0x7f08a806f470 osd=6 4 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 10 client.27603.objecter _send_command 1 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 1 -- 172.21.15.162:0/1564933329 --> v2:172.21.15.181:6808/1506640341 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08a8072600 con 0x7f08a806fb20 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 20 client.27603.objecter dump_active .. 0 homeless 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 1 -- 172.21.15.162:0/1564933329 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=522) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08b00802b0 con 0x7f08c00f8d10 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08c4c83700 1 --2- 172.21.15.162:0/1564933329 >> v2:172.21.15.181:6808/1506640341 conn(0x7f08a806fb20 0x7f08a8071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bf7fe700 1 --2- 172.21.15.162:0/1564933329 >> v2:172.21.15.162:6800/2866534053 conn(0x7f08a8062c90 0x7f08a8065150 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7f08c0109ab0 tx=0x7f08b400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:57.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 10 client.27603.objecter ms_handle_connect 0x7f08a8062c90 2024-02-13T00:48:57.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08bd7fa700 10 client.27603.objecter ms_handle_connect 0x7f08a806fb20 2024-02-13T00:48:57.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.172+0000 7f08c4c83700 1 --2- 172.21.15.162:0/1564933329 >> v2:172.21.15.181:6808/1506640341 conn(0x7f08a806fb20 0x7f08a8071fc0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:57.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.173+0000 7f08bd7fa700 1 -- 172.21.15.162:0/1564933329 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f08a8072600 con 0x7f08a806fb20 2024-02-13T00:48:57.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.173+0000 7f08bd7fa700 10 client.27603.objecter ms_dispatch 0x7f08c0001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:48:57.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.173+0000 7f08bd7fa700 10 client.27603.objecter _finish_command 1 = system:0 2024-02-13T00:48:57.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.173+0000 7f08bd7fa700 20 client.27603.objecter put_session s=0x7f08a806f470 osd=6 4 2024-02-13T00:48:57.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.173+0000 7f08bd7fa700 15 client.27603.objecter _session_command_op_remove 6 1 2024-02-13T00:48:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:57.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:57.193 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:48:57.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08c66e6700 10 client.27603.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-02-13T00:48:57.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08c66e6700 15 client.27603.objecter _session_command_op_assign -1 2 2024-02-13T00:48:57.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08c66e6700 20 client.27603.objecter _get_session s=0x7f08a806f470 osd=6 3 2024-02-13T00:48:57.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08c66e6700 20 client.27603.objecter put_session s=0x7f08a806f470 osd=6 3 2024-02-13T00:48:57.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08c66e6700 20 client.27603.objecter _get_session s=0x7f08a806f470 osd=6 3 2024-02-13T00:48:57.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08c66e6700 15 client.27603.objecter _session_command_op_remove -1 2 2024-02-13T00:48:57.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08c66e6700 20 client.27603.objecter get_session s=0x7f08a806f470 osd=6 3 2024-02-13T00:48:57.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08c66e6700 15 client.27603.objecter _session_command_op_assign 6 2 2024-02-13T00:48:57.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08c66e6700 20 client.27603.objecter put_session s=0x7f08a806f470 osd=6 4 2024-02-13T00:48:57.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08c66e6700 10 client.27603.objecter _send_command 2 2024-02-13T00:48:57.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08c66e6700 1 -- 172.21.15.162:0/1564933329 --> v2:172.21.15.181:6808/1506640341 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7f08c0107170 con 0x7f08a806fb20 2024-02-13T00:48:57.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08bd7fa700 1 -- 172.21.15.162:0/1564933329 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f08c0107170 con 0x7f08a806fb20 2024-02-13T00:48:57.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08bd7fa700 10 client.27603.objecter ms_dispatch 0x7f08c0001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:48:57.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08bd7fa700 10 client.27603.objecter _finish_command 2 = system:0 2024-02-13T00:48:57.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08bd7fa700 20 client.27603.objecter put_session s=0x7f08a806f470 osd=6 4 2024-02-13T00:48:57.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08bd7fa700 15 client.27603.objecter _session_command_op_remove 6 2 2024-02-13T00:48:57.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.194+0000 7f08c66e6700 10 librados: watch_flush enter 2024-02-13T00:48:57.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 10 librados: watch_flush exit 2024-02-13T00:48:57.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 10 client.27603.objecter close_session for osd.6 2024-02-13T00:48:57.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 1 -- 172.21.15.162:0/1564933329 >> v2:172.21.15.181:6808/1506640341 conn(0x7f08a806fb20 msgr2=0x7f08a8071fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:57.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 1 --2- 172.21.15.162:0/1564933329 >> v2:172.21.15.181:6808/1506640341 conn(0x7f08a806fb20 0x7f08a8071fc0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:57.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 20 client.27603.objecter put_session s=0x7f08a806f470 osd=6 1 2024-02-13T00:48:57.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 20 client.27603.objecter shutdown clearing up homeless session... 2024-02-13T00:48:57.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 10 client.27603.objecter successfully canceled tick 2024-02-13T00:48:57.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 1 -- 172.21.15.162:0/1564933329 >> v2:172.21.15.162:6800/2866534053 conn(0x7f08a8062c90 msgr2=0x7f08a8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:57.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 1 --2- 172.21.15.162:0/1564933329 >> v2:172.21.15.162:6800/2866534053 conn(0x7f08a8062c90 0x7f08a8065150 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7f08c0109ab0 tx=0x7f08b400b040).stop 2024-02-13T00:48:57.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 1 -- 172.21.15.162:0/1564933329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c00f8d10 msgr2=0x7f08c010e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:57.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 1 --2- 172.21.15.162:0/1564933329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c00f8d10 0x7f08c010e130 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f08b0009eb0 tx=0x7f08b000b040).stop 2024-02-13T00:48:57.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c4c83700 1 -- 172.21.15.162:0/1564933329 reap_dead start 2024-02-13T00:48:57.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 1 -- 172.21.15.162:0/1564933329 shutdown_connections 2024-02-13T00:48:57.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 1 -- 172.21.15.162:0/1564933329 >> 172.21.15.162:0/1564933329 conn(0x7f08c006eb90 msgr2=0x7f08c0102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:57.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 1 -- 172.21.15.162:0/1564933329 shutdown_connections 2024-02-13T00:48:57.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 1 -- 172.21.15.162:0/1564933329 wait complete. 2024-02-13T00:48:57.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:57.195+0000 7f08c66e6700 1 librados: shutdown 2024-02-13T00:48:57.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:48:57.312 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:57.525 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:56.466582+0000 mgr.y (mgr.14544) 461 : cluster [DBG] pgmap v782: 265 pgs: 1 peering, 264 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:57.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:56.517802+0000 mon.b (mon.1) 645 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:57.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:56.518112+0000 mon.a (mon.0) 2814 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:57.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:56.518584+0000 mon.b (mon.1) 646 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-83"}]: dispatch 2024-02-13T00:48:57.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:56.518824+0000 mon.a (mon.0) 2815 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-83"}]: dispatch 2024-02-13T00:48:57.527 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:56.466582+0000 mgr.y (mgr.14544) 461 : cluster [DBG] pgmap v782: 265 pgs: 1 peering, 264 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:57.527 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:56.517802+0000 mon.b (mon.1) 645 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:57.527 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:56.518112+0000 mon.a (mon.0) 2814 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:57.528 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:56.518584+0000 mon.b (mon.1) 646 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-83"}]: dispatch 2024-02-13T00:48:57.528 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:56.518824+0000 mon.a (mon.0) 2815 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-83"}]: dispatch 2024-02-13T00:48:57.565 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-13T00:48:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:56.466582+0000 mgr.y (mgr.14544) 461 : cluster [DBG] pgmap v782: 265 pgs: 1 peering, 264 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:56.517802+0000 mon.b (mon.1) 645 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:57.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:56.518112+0000 mon.a (mon.0) 2814 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:48:57.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:56.518584+0000 mon.b (mon.1) 646 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-83"}]: dispatch 2024-02-13T00:48:57.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:56.518824+0000 mon.a (mon.0) 2815 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-83"}]: dispatch 2024-02-13T00:48:57.873 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:48:57.960 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:48:57.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:57.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:48:58.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:48:58.096 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:58.118 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:48:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:58.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:48:58.221 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:58.250 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:58.534 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:57.382788+0000 osd.6 (osd.6) 157 : cluster [DBG] 4.e scrub starts 2024-02-13T00:48:58.534 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:57.386250+0000 osd.6 (osd.6) 158 : cluster [DBG] 4.e scrub ok 2024-02-13T00:48:58.534 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:57.517979+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e523: 8 total, 8 up, 4 in 2024-02-13T00:48:58.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:57.382788+0000 osd.6 (osd.6) 157 : cluster [DBG] 4.e scrub starts 2024-02-13T00:48:58.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:57.386250+0000 osd.6 (osd.6) 158 : cluster [DBG] 4.e scrub ok 2024-02-13T00:48:58.784 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:57.517979+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e523: 8 total, 8 up, 4 in 2024-02-13T00:48:58.784 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:48:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:48:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:48:58.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:57.382788+0000 osd.6 (osd.6) 157 : cluster [DBG] 4.e scrub starts 2024-02-13T00:48:58.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:57.386250+0000 osd.6 (osd.6) 158 : cluster [DBG] 4.e scrub ok 2024-02-13T00:48:58.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:57.517979+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e523: 8 total, 8 up, 4 in 2024-02-13T00:48:59.099 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:48:59.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:59.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:48:59.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:59.231 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:59.328 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:59.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:59.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:59.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:48:59.462 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:59.585 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:48:59.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:48:59.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:48:59.620 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:48:59.621 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:48:59.624 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:58.467036+0000 mgr.y (mgr.14544) 462 : cluster [DBG] pgmap v785: 201 pgs: 1 peering, 200 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:59.625 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:58.533467+0000 mon.a (mon.0) 2817 : cluster [DBG] osdmap e524: 8 total, 8 up, 4 in 2024-02-13T00:48:59.625 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:58.537348+0000 mon.b (mon.1) 647 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:59.625 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:48:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:58.538404+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:59.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.679+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3639366861 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fa40f73a0 msgr2=0x7f9fa41039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:59.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.679+0000 7f9fabcf9700 1 --2- 172.21.15.162:0/3639366861 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fa40f73a0 0x7f9fa41039a0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7f9f9c009be0 tx=0x7f9f9c004d10).stop 2024-02-13T00:48:59.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.679+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3639366861 shutdown_connections 2024-02-13T00:48:59.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.679+0000 7f9fabcf9700 1 --2- 172.21.15.162:0/3639366861 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fa40f73a0 0x7f9fa41039a0 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:59.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.679+0000 7f9fabcf9700 1 --2- 172.21.15.162:0/3639366861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fa40fcb30 0x7f9fa40fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:59.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.679+0000 7f9fabcf9700 1 --2- 172.21.15.162:0/3639366861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fa40fd490 0x7f9fa40f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:59.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.679+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3639366861 >> 172.21.15.162:0/3639366861 conn(0x7f9fa406eb90 msgr2=0x7f9fa40f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:48:59.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.679+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3639366861 shutdown_connections 2024-02-13T00:48:59.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.679+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3639366861 wait complete. 2024-02-13T00:48:59.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fabcf9700 1 librados: starting msgr at 2024-02-13T00:48:59.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fabcf9700 1 librados: starting objecter 2024-02-13T00:48:59.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fabcf9700 1 Processor -- start 2024-02-13T00:48:59.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fabcf9700 1 -- start start 2024-02-13T00:48:59.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fabcf9700 1 librados: setting wanted keys 2024-02-13T00:48:59.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fabcf9700 1 librados: calling monclient init 2024-02-13T00:48:59.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fabcf9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fa40f73a0 0x7f9fa410ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:59.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fabcf9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fa40fcb30 0x7f9fa410d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fabcf9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fa40fd490 0x7f9fa4107000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fabcf9700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa41074f0 con 0x7f9fa40fd490 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fabcf9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa4107660 con 0x7f9fa40fcb30 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fabcf9700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa41077d0 con 0x7f9fa40f73a0 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9faa296700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fa40fd490 0x7f9fa4107000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fa9a95700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fa40f73a0 0x7f9fa410ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9faa296700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fa40fd490 0x7f9fa4107000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:50094/0 (socket says 172.21.15.162:50094) 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9faa296700 1 -- 172.21.15.162:0/3273984515 learned_addr learned my addr 172.21.15.162:0/3273984515 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fa9294700 1 --2- 172.21.15.162:0/3273984515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fa40fcb30 0x7f9fa410d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fa9a95700 1 -- 172.21.15.162:0/3273984515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fa40fcb30 msgr2=0x7f9fa410d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.680+0000 7f9fa9a95700 1 --2- 172.21.15.162:0/3273984515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fa40fcb30 0x7f9fa410d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9fa9a95700 1 -- 172.21.15.162:0/3273984515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fa40fd490 msgr2=0x7f9fa4107000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:48:59.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9fa9a95700 1 --2- 172.21.15.162:0/3273984515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fa40fd490 0x7f9fa4107000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9fa9a95700 1 -- 172.21.15.162:0/3273984515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa0005040 con 0x7f9fa40f73a0 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9fa9294700 1 --2- 172.21.15.162:0/3273984515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fa40fcb30 0x7f9fa410d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9faa296700 1 --2- 172.21.15.162:0/3273984515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fa40fd490 0x7f9fa4107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9fa9a95700 1 --2- 172.21.15.162:0/3273984515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fa40f73a0 0x7f9fa410ce20 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 rx=0x7f9f9400c810 tx=0x7f9f94007ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9f9affd700 10 client.?.objecter ms_handle_connect 0x7f9fa40f73a0 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9f9affd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9f9affd700 1 -- 172.21.15.162:0/3273984515 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9400e4a0 con 0x7f9fa40f73a0 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9f9affd700 1 -- 172.21.15.162:0/3273984515 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f9400ead0 con 0x7f9fa40f73a0 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9f9affd700 1 -- 172.21.15.162:0/3273984515 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f94012a50 con 0x7f9fa40f73a0 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3273984515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c0097c0 con 0x7f9fa40f73a0 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9fabcf9700 10 client.37709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3273984515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa4113c90 con 0x7f9fa40f73a0 2024-02-13T00:48:59.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.681+0000 7f9fabcf9700 1 librados: init done 2024-02-13T00:48:59.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.682+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3273984515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa4106090 con 0x7f9fa40f73a0 2024-02-13T00:48:59.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.682+0000 7f9f9affd700 1 -- 172.21.15.162:0/3273984515 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f9f9400ec40 con 0x7f9fa40f73a0 2024-02-13T00:48:59.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.682+0000 7f9f9affd700 1 --2- 172.21.15.162:0/3273984515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9f90062c90 0x7f9f90065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:48:59.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.683+0000 7f9f9affd700 1 -- 172.21.15.162:0/3273984515 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(525..525 src has 449..525) v4 ==== 8054+0+0 (secure 0 0 0) 0x7f9f9407c8e0 con 0x7f9fa40f73a0 2024-02-13T00:48:59.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.683+0000 7f9f9affd700 10 client.37709.objecter ms_dispatch 0x7f9fa4001550 osd_map(525..525 src has 449..525) v4 2024-02-13T00:48:59.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.683+0000 7f9f9affd700 3 client.37709.objecter handle_osd_map got epochs [525,525] > 0 2024-02-13T00:48:59.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.683+0000 7f9f9affd700 3 client.37709.objecter handle_osd_map decoding full epoch 525 2024-02-13T00:48:59.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.683+0000 7f9fa9294700 1 --2- 172.21.15.162:0/3273984515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9f90062c90 0x7f9f90065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:48:59.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.683+0000 7f9f9affd700 20 client.37709.objecter dump_active .. 0 homeless 2024-02-13T00:48:59.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.683+0000 7f9f9affd700 10 client.37709.objecter ms_handle_connect 0x7f9f90062c90 2024-02-13T00:48:59.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.683+0000 7f9fa9294700 1 --2- 172.21.15.162:0/3273984515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9f90062c90 0x7f9f90065150 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 rx=0x7f9fa0007fd0 tx=0x7f9fa0009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:48:59.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:48:59.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.687+0000 7f9f9affd700 1 -- 172.21.15.162:0/3273984515 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9f94048f60 con 0x7f9fa40f73a0 2024-02-13T00:48:59.714 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:58.467036+0000 mgr.y (mgr.14544) 462 : cluster [DBG] pgmap v785: 201 pgs: 1 peering, 200 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:59.714 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:58.533467+0000 mon.a (mon.0) 2817 : cluster [DBG] osdmap e524: 8 total, 8 up, 4 in 2024-02-13T00:48:59.714 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:58.537348+0000 mon.b (mon.1) 647 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:59.714 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:48:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:58.538404+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:59.715 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:58.467036+0000 mgr.y (mgr.14544) 462 : cluster [DBG] pgmap v785: 201 pgs: 1 peering, 200 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:48:59.715 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:58.533467+0000 mon.a (mon.0) 2817 : cluster [DBG] osdmap e524: 8 total, 8 up, 4 in 2024-02-13T00:48:59.715 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:58.537348+0000 mon.b (mon.1) 647 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:59.715 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:48:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:58.538404+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:48:59.716 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:48:59.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:48:59.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:48:59.877+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3273984515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f9fa405fc50 con 0x7f9fa40f73a0 2024-02-13T00:49:00.065 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:00.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:00.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:49:00.072 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:00.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:00.200 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:00.336 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-13T00:49:00.548 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:00.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:00.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.561+0000 7f9f9affd700 1 -- 172.21.15.162:0/3273984515 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v526) v1 ==== 101+0+0 (secure 0 0 0) 0x7f9f94017030 con 0x7f9fa40f73a0 2024-02-13T00:49:00.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.564+0000 7f9fabcf9700 10 librados: watch_flush enter 2024-02-13T00:49:00.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.564+0000 7f9fabcf9700 10 librados: watch_flush exit 2024-02-13T00:49:00.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.564+0000 7f9fabcf9700 20 client.37709.objecter shutdown clearing up homeless session... 2024-02-13T00:49:00.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.564+0000 7f9fabcf9700 10 client.37709.objecter successfully canceled tick 2024-02-13T00:49:00.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.564+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3273984515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9f90062c90 msgr2=0x7f9f90065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:00.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.564+0000 7f9fabcf9700 1 --2- 172.21.15.162:0/3273984515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9f90062c90 0x7f9f90065150 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 rx=0x7f9fa0007fd0 tx=0x7f9fa0009cb0).stop 2024-02-13T00:49:00.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.564+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3273984515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fa40f73a0 msgr2=0x7f9fa410ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:00.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.564+0000 7f9fabcf9700 1 --2- 172.21.15.162:0/3273984515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fa40f73a0 0x7f9fa410ce20 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 rx=0x7f9f9400c810 tx=0x7f9f94007ee0).stop 2024-02-13T00:49:00.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.565+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3273984515 shutdown_connections 2024-02-13T00:49:00.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.565+0000 7f9fabcf9700 1 --2- 172.21.15.162:0/3273984515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9f90062c90 0x7f9f90065150 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:00.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.565+0000 7f9fabcf9700 1 --2- 172.21.15.162:0/3273984515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fa40fd490 0x7f9fa4107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:00.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.565+0000 7f9fabcf9700 1 --2- 172.21.15.162:0/3273984515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fa40f73a0 0x7f9fa410ce20 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:00.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.565+0000 7f9fabcf9700 1 --2- 172.21.15.162:0/3273984515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fa40fcb30 0x7f9fa410d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:00.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.565+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3273984515 >> 172.21.15.162:0/3273984515 conn(0x7f9fa406eb90 msgr2=0x7f9fa406fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:00.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.565+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3273984515 shutdown_connections 2024-02-13T00:49:00.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.565+0000 7f9fabcf9700 1 -- 172.21.15.162:0/3273984515 wait complete. 2024-02-13T00:49:00.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:00.565+0000 7f9fabcf9700 1 librados: shutdown 2024-02-13T00:49:00.569 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:49:00.618 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:49:00.652 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:00.678 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:58.857854+0000 osd.0 (osd.0) 203 : cluster [DBG] 2.f scrub starts 2024-02-13T00:49:00.678 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:58.861321+0000 osd.0 (osd.0) 204 : cluster [DBG] 2.f scrub ok 2024-02-13T00:49:00.678 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:59.108930+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.a scrub starts 2024-02-13T00:49:00.678 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:59.111823+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.a scrub ok 2024-02-13T00:49:00.678 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:59.536688+0000 mon.a (mon.0) 2819 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-86","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:00.679 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:48:59.536784+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e525: 8 total, 8 up, 4 in 2024-02-13T00:49:00.679 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:59.540117+0000 mon.b (mon.1) 648 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:49:00.679 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:59.552775+0000 mon.b (mon.1) 649 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:49:00.679 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:59.553162+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:49:00.679 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:59.878611+0000 mon.c (mon.2) 415 : audit [INF] from='client.? 172.21.15.162:0/3273984515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:00.679 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:48:59.878992+0000 mon.a (mon.0) 2822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:00.679 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:00.468185+0000 mon.c (mon.2) 416 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-13T00:49:00.679 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:00.470787+0000 mon.a (mon.0) 2823 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-13T00:49:00.679 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:58.857854+0000 osd.0 (osd.0) 203 : cluster [DBG] 2.f scrub starts 2024-02-13T00:49:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:58.861321+0000 osd.0 (osd.0) 204 : cluster [DBG] 2.f scrub ok 2024-02-13T00:49:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:59.108930+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.a scrub starts 2024-02-13T00:49:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:59.111823+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.a scrub ok 2024-02-13T00:49:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:59.536688+0000 mon.a (mon.0) 2819 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-86","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:48:59.536784+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e525: 8 total, 8 up, 4 in 2024-02-13T00:49:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:59.540117+0000 mon.b (mon.1) 648 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:49:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:59.552775+0000 mon.b (mon.1) 649 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:49:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:59.553162+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:49:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:59.878611+0000 mon.c (mon.2) 415 : audit [INF] from='client.? 172.21.15.162:0/3273984515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:48:59.878992+0000 mon.a (mon.0) 2822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:00.468185+0000 mon.c (mon.2) 416 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-13T00:49:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:00.470787+0000 mon.a (mon.0) 2823 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-13T00:49:00.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:58.857854+0000 osd.0 (osd.0) 203 : cluster [DBG] 2.f scrub starts 2024-02-13T00:49:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:58.861321+0000 osd.0 (osd.0) 204 : cluster [DBG] 2.f scrub ok 2024-02-13T00:49:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:59.108930+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.a scrub starts 2024-02-13T00:49:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:59.111823+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.a scrub ok 2024-02-13T00:49:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:59.536688+0000 mon.a (mon.0) 2819 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-86","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:48:59.536784+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e525: 8 total, 8 up, 4 in 2024-02-13T00:49:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:59.540117+0000 mon.b (mon.1) 648 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:49:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:59.552775+0000 mon.b (mon.1) 649 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:49:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:59.553162+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:49:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:59.878611+0000 mon.c (mon.2) 415 : audit [INF] from='client.? 172.21.15.162:0/3273984515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:48:59.878992+0000 mon.a (mon.0) 2822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:00.468185+0000 mon.c (mon.2) 416 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-13T00:49:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:00.470787+0000 mon.a (mon.0) 2823 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-13T00:49:00.928 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:00.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:01.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:01.063 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:00.467881+0000 mgr.y (mgr.14544) 463 : cluster [DBG] pgmap v788: 233 pgs: 233 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:49:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:00.547112+0000 mon.a (mon.0) 2824 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:49:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:00.557970+0000 mon.a (mon.0) 2825 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:49:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:00.558169+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:49:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:00.558241+0000 mon.a (mon.0) 2827 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-13T00:49:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:00.558306+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e526: 8 total, 8 up, 4 in 2024-02-13T00:49:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:00.880354+0000 osd.0 (osd.0) 205 : cluster [DBG] 28.19 scrub starts 2024-02-13T00:49:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:00.881428+0000 osd.0 (osd.0) 206 : cluster [DBG] 28.19 scrub ok 2024-02-13T00:49:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:00.957490+0000 mon.c (mon.2) 417 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:49:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.255367+0000 mon.c (mon.2) 418 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.255718+0000 mon.a (mon.0) 2829 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.256424+0000 mon.c (mon.2) 419 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.256621+0000 mon.a (mon.0) 2830 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.257257+0000 mon.c (mon.2) 420 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.257461+0000 mon.a (mon.0) 2831 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:00.467881+0000 mgr.y (mgr.14544) 463 : cluster [DBG] pgmap v788: 233 pgs: 233 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:49:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:00.547112+0000 mon.a (mon.0) 2824 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:49:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:00.557970+0000 mon.a (mon.0) 2825 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:49:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:00.558169+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:49:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:00.558241+0000 mon.a (mon.0) 2827 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-13T00:49:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:00.558306+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e526: 8 total, 8 up, 4 in 2024-02-13T00:49:01.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:00.880354+0000 osd.0 (osd.0) 205 : cluster [DBG] 28.19 scrub starts 2024-02-13T00:49:01.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:00.881428+0000 osd.0 (osd.0) 206 : cluster [DBG] 28.19 scrub ok 2024-02-13T00:49:01.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:00.957490+0000 mon.c (mon.2) 417 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:49:01.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.255367+0000 mon.c (mon.2) 418 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.255718+0000 mon.a (mon.0) 2829 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.256424+0000 mon.c (mon.2) 419 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.256621+0000 mon.a (mon.0) 2830 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.257257+0000 mon.c (mon.2) 420 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.257461+0000 mon.a (mon.0) 2831 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.258069+0000 mon.c (mon.2) 421 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.258296+0000 mon.a (mon.0) 2832 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.262797+0000 mon.a (mon.0) 2833 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.479465+0000 mon.c (mon.2) 422 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.479813+0000 mon.a (mon.0) 2834 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.480491+0000 mon.c (mon.2) 423 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.480714+0000 mon.a (mon.0) 2835 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.481406+0000 mon.c (mon.2) 424 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.481605+0000 mon.a (mon.0) 2836 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.482190+0000 mon.c (mon.2) 425 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.482384+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.487057+0000 mon.a (mon.0) 2838 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.488657+0000 mon.c (mon.2) 426 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.489225+0000 mon.c (mon.2) 427 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:49:01.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:01.493539+0000 mon.a (mon.0) 2839 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.258069+0000 mon.c (mon.2) 421 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.258296+0000 mon.a (mon.0) 2832 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.262797+0000 mon.a (mon.0) 2833 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.479465+0000 mon.c (mon.2) 422 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.479813+0000 mon.a (mon.0) 2834 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.480491+0000 mon.c (mon.2) 423 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.480714+0000 mon.a (mon.0) 2835 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.481406+0000 mon.c (mon.2) 424 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.481605+0000 mon.a (mon.0) 2836 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.482190+0000 mon.c (mon.2) 425 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.482384+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.487057+0000 mon.a (mon.0) 2838 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:01.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.488657+0000 mon.c (mon.2) 426 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:49:01.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.489225+0000 mon.c (mon.2) 427 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:49:01.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:01.493539+0000 mon.a (mon.0) 2839 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:01.911 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:01.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:01.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:00.467881+0000 mgr.y (mgr.14544) 463 : cluster [DBG] pgmap v788: 233 pgs: 233 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:49:01.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:00.547112+0000 mon.a (mon.0) 2824 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:49:01.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:00.557970+0000 mon.a (mon.0) 2825 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:49:01.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:00.558169+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:49:01.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:00.558241+0000 mon.a (mon.0) 2827 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-13T00:49:01.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:00.558306+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e526: 8 total, 8 up, 4 in 2024-02-13T00:49:01.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:00.880354+0000 osd.0 (osd.0) 205 : cluster [DBG] 28.19 scrub starts 2024-02-13T00:49:01.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:00.881428+0000 osd.0 (osd.0) 206 : cluster [DBG] 28.19 scrub ok 2024-02-13T00:49:01.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:00.957490+0000 mon.c (mon.2) 417 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:49:01.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.255367+0000 mon.c (mon.2) 418 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.255718+0000 mon.a (mon.0) 2829 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.256424+0000 mon.c (mon.2) 419 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.256621+0000 mon.a (mon.0) 2830 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.257257+0000 mon.c (mon.2) 420 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.257461+0000 mon.a (mon.0) 2831 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.258069+0000 mon.c (mon.2) 421 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.258296+0000 mon.a (mon.0) 2832 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.262797+0000 mon.a (mon.0) 2833 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.479465+0000 mon.c (mon.2) 422 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.479813+0000 mon.a (mon.0) 2834 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.480491+0000 mon.c (mon.2) 423 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.480714+0000 mon.a (mon.0) 2835 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.481406+0000 mon.c (mon.2) 424 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.481605+0000 mon.a (mon.0) 2836 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.482190+0000 mon.c (mon.2) 425 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.482384+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-13T00:49:01.989 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.487057+0000 mon.a (mon.0) 2838 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:01.990 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.488657+0000 mon.c (mon.2) 426 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:49:01.990 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.489225+0000 mon.c (mon.2) 427 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:49:01.990 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:01.493539+0000 mon.a (mon.0) 2839 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:02.015 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:02.044 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:02.585 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:49:02.585 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:49:02.593 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:02.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:02.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:02.702 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:01.130817+0000 osd.1 (osd.1) 227 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:49:02.702 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:01.133705+0000 osd.1 (osd.1) 228 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:49:02.702 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:49:01.258815+0000 mgr.y (mgr.14544) 464 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 3256M 2024-02-13T00:49:02.703 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cephadm 2024-02-13T00:49:01.482851+0000 mgr.y (mgr.14544) 465 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3256M 2024-02-13T00:49:02.703 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:01.557112+0000 mon.a (mon.0) 2840 : cluster [DBG] osdmap e527: 8 total, 8 up, 4 in 2024-02-13T00:49:02.703 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:01.130817+0000 osd.1 (osd.1) 227 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:49:02.703 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:01.133705+0000 osd.1 (osd.1) 228 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:49:02.703 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:49:01.258815+0000 mgr.y (mgr.14544) 464 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 3256M 2024-02-13T00:49:02.703 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cephadm 2024-02-13T00:49:01.482851+0000 mgr.y (mgr.14544) 465 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3256M 2024-02-13T00:49:02.703 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:01.557112+0000 mon.a (mon.0) 2840 : cluster [DBG] osdmap e527: 8 total, 8 up, 4 in 2024-02-13T00:49:02.726 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:02.824 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:02.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:02.908 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:01.130817+0000 osd.1 (osd.1) 227 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:49:02.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:01.133705+0000 osd.1 (osd.1) 228 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:49:02.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:49:01.258815+0000 mgr.y (mgr.14544) 464 : cephadm [INF] Adjusting osd_memory_target on smithi181 to 3256M 2024-02-13T00:49:02.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cephadm 2024-02-13T00:49:01.482851+0000 mgr.y (mgr.14544) 465 : cephadm [INF] Adjusting osd_memory_target on smithi162 to 3256M 2024-02-13T00:49:02.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:01.557112+0000 mon.a (mon.0) 2840 : cluster [DBG] osdmap e527: 8 total, 8 up, 4 in 2024-02-13T00:49:02.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:49:02.959 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.967+0000 7fb63abd0700 1 -- 172.21.15.162:0/452151943 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6340f9b70 msgr2=0x7fb6340f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.967+0000 7fb63abd0700 1 --2- 172.21.15.162:0/452151943 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6340f9b70 0x7fb6340f9f50 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 rx=0x7fb6280047b0 tx=0x7fb628009cb0).stop 2024-02-13T00:49:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.967+0000 7fb63abd0700 1 -- 172.21.15.162:0/452151943 shutdown_connections 2024-02-13T00:49:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.967+0000 7fb63abd0700 1 --2- 172.21.15.162:0/452151943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6340f8500 0x7fb634104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:02.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.967+0000 7fb63abd0700 1 --2- 172.21.15.162:0/452151943 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6340f9b70 0x7fb6340f9f50 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:02.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.967+0000 7fb63abd0700 1 --2- 172.21.15.162:0/452151943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6340fa440 0x7fb6340f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:02.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.967+0000 7fb63abd0700 1 -- 172.21.15.162:0/452151943 >> 172.21.15.162:0/452151943 conn(0x7fb6340f38c0 msgr2=0x7fb6340f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:02.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.968+0000 7fb63abd0700 1 -- 172.21.15.162:0/452151943 shutdown_connections 2024-02-13T00:49:02.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.968+0000 7fb63abd0700 1 -- 172.21.15.162:0/452151943 wait complete. 2024-02-13T00:49:02.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.968+0000 7fb63abd0700 1 librados: starting msgr at 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.968+0000 7fb63abd0700 1 librados: starting objecter 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.968+0000 7fb63abd0700 1 Processor -- start 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.968+0000 7fb63abd0700 1 -- start start 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.968+0000 7fb63abd0700 1 librados: setting wanted keys 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.968+0000 7fb63abd0700 1 librados: calling monclient init 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb63abd0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6340f8500 0x7fb63406b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb63abd0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6340f9b70 0x7fb63406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb63abd0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6340fa440 0x7fb63406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb63abd0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6341166a0 con 0x7fb6340fa440 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb63abd0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb634116810 con 0x7fb6340f8500 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb63abd0700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6341169d0 con 0x7fb6340f9b70 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb633fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6340f9b70 0x7fb63406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:02.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb63916d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6340fa440 0x7fb63406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:02.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb633fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6340f9b70 0x7fb63406b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:36406/0 (socket says 172.21.15.162:36406) 2024-02-13T00:49:02.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb63916d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6340fa440 0x7fb63406bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:50120/0 (socket says 172.21.15.162:50120) 2024-02-13T00:49:02.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb633fff700 1 -- 172.21.15.162:0/129739496 learned_addr learned my addr 172.21.15.162:0/129739496 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:02.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb63896c700 1 --2- 172.21.15.162:0/129739496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6340f8500 0x7fb63406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:02.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb633fff700 1 -- 172.21.15.162:0/129739496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6340f8500 msgr2=0x7fb63406b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:02.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb633fff700 1 --2- 172.21.15.162:0/129739496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6340f8500 0x7fb63406b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:02.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb633fff700 1 -- 172.21.15.162:0/129739496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6340fa440 msgr2=0x7fb63406bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:02.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb633fff700 1 --2- 172.21.15.162:0/129739496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6340fa440 0x7fb63406bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:02.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb633fff700 1 -- 172.21.15.162:0/129739496 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c0097c0 con 0x7fb6340f9b70 2024-02-13T00:49:02.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb63916d700 1 --2- 172.21.15.162:0/129739496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6340fa440 0x7fb63406bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:49:02.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb633fff700 1 --2- 172.21.15.162:0/129739496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6340f9b70 0x7fb63406b660 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7fb6200041d0 tx=0x7fb620009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:02.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb631ffb700 10 client.?.objecter ms_handle_connect 0x7fb6340f9b70 2024-02-13T00:49:02.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb631ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:02.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb620007b40 con 0x7fb6340f9b70 2024-02-13T00:49:02.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb628005040 con 0x7fb6340f9b70 2024-02-13T00:49:02.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb62000f040 con 0x7fb6340f9b70 2024-02-13T00:49:02.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.969+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb620014730 con 0x7fb6340f9b70 2024-02-13T00:49:02.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.970+0000 7fb63abd0700 10 client.37715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:02.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.970+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb634117060 con 0x7fb6340f9b70 2024-02-13T00:49:02.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.970+0000 7fb63abd0700 1 librados: init done 2024-02-13T00:49:02.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.970+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb6340475e0 con 0x7fb6340f9b70 2024-02-13T00:49:02.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.971+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fb620014b60 con 0x7fb6340f9b70 2024-02-13T00:49:02.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.971+0000 7fb631ffb700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb624062c90 0x7fb624065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:02.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.971+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(528..528 src has 449..528) v4 ==== 8054+0+0 (secure 0 0 0) 0x7fb62007d190 con 0x7fb6340f9b70 2024-02-13T00:49:02.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.971+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 osd_map(528..528 src has 449..528) v4 2024-02-13T00:49:02.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.971+0000 7fb631ffb700 3 client.37715.objecter handle_osd_map got epochs [528,528] > 0 2024-02-13T00:49:02.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.971+0000 7fb631ffb700 3 client.37715.objecter handle_osd_map decoding full epoch 528 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.971+0000 7fb631ffb700 20 client.37715.objecter dump_active .. 0 homeless 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.971+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v528) v1 ==== 54+0+16 (secure 0 0 0) 0x7fb620084070 con 0x7fb6340f9b70 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63abd0700 10 client.37715.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 1 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63896c700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb624062c90 0x7fb624065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb634028590 0x7fb6340fea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb6341072c0 osd=0 3 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6341072c0 osd=0 3 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb6341072c0 osd=0 3 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 1 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb6341072c0 osd=0 3 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 0 1 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6341072c0 osd=0 4 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63abd0700 10 client.37715.objecter _send_command 1 2024-02-13T00:49:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6341000d0 con 0x7fb634028590 2024-02-13T00:49:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63916d700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb634028590 0x7fb6340fea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb631ffb700 10 client.37715.objecter ms_handle_connect 0x7fb624062c90 2024-02-13T00:49:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63896c700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb624062c90 0x7fb624065150 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 rx=0x7fb628005010 tx=0x7fb628007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb631ffb700 10 client.37715.objecter ms_handle_connect 0x7fb634028590 2024-02-13T00:49:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.972+0000 7fb63916d700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb634028590 0x7fb6340fea60 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.973+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb6341000d0 con 0x7fb634028590 2024-02-13T00:49:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.973+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:49:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.973+0000 7fb631ffb700 10 client.37715.objecter _finish_command 1 = system:0 2024-02-13T00:49:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.973+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb6341072c0 osd=0 4 2024-02-13T00:49:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.973+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 0 1 2024-02-13T00:49:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:02.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:49:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.988+0000 7fb63abd0700 10 client.37715.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.988+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 2 2024-02-13T00:49:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.988+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb6341072c0 osd=0 3 2024-02-13T00:49:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.988+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6341072c0 osd=0 3 2024-02-13T00:49:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.988+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb6341072c0 osd=0 3 2024-02-13T00:49:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.988+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 2 2024-02-13T00:49:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.988+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb6341072c0 osd=0 3 2024-02-13T00:49:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.988+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 0 2 2024-02-13T00:49:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.988+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6341072c0 osd=0 4 2024-02-13T00:49:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.988+0000 7fb63abd0700 10 client.37715.objecter _send_command 2 2024-02-13T00:49:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.988+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb63405a0e0 con 0x7fb634028590 2024-02-13T00:49:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb63405a0e0 con 0x7fb634028590 2024-02-13T00:49:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb631ffb700 10 client.37715.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb6341072c0 osd=0 4 2024-02-13T00:49:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 0 2 2024-02-13T00:49:02.997 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:49:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63abd0700 10 client.37715.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 3 2024-02-13T00:49:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6805/150599780 conn(0x7fb634059570 0x7fb63411b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634059400 osd=1 3 2024-02-13T00:49:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634059400 osd=1 3 2024-02-13T00:49:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634059400 osd=1 3 2024-02-13T00:49:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 3 2024-02-13T00:49:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb634059400 osd=1 3 2024-02-13T00:49:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 1 3 2024-02-13T00:49:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634059400 osd=1 4 2024-02-13T00:49:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63abd0700 10 client.37715.objecter _send_command 3 2024-02-13T00:49:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6340280f0 con 0x7fb634059570 2024-02-13T00:49:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.990+0000 7fb63896c700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6805/150599780 conn(0x7fb634059570 0x7fb63411b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.991+0000 7fb631ffb700 10 client.37715.objecter ms_handle_connect 0x7fb634059570 2024-02-13T00:49:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.991+0000 7fb63896c700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6805/150599780 conn(0x7fb634059570 0x7fb63411b2d0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.991+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb6340280f0 con 0x7fb634059570 2024-02-13T00:49:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.991+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:49:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.991+0000 7fb631ffb700 10 client.37715.objecter _finish_command 3 = system:0 2024-02-13T00:49:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.991+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb634059400 osd=1 4 2024-02-13T00:49:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:02.991+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 1 3 2024-02-13T00:49:03.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.006+0000 7fb63abd0700 10 client.37715.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:03.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.006+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 4 2024-02-13T00:49:03.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.006+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634059400 osd=1 3 2024-02-13T00:49:03.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.006+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634059400 osd=1 3 2024-02-13T00:49:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.006+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634059400 osd=1 3 2024-02-13T00:49:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.006+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 4 2024-02-13T00:49:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.006+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb634059400 osd=1 3 2024-02-13T00:49:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.006+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 1 4 2024-02-13T00:49:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.006+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634059400 osd=1 4 2024-02-13T00:49:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.006+0000 7fb63abd0700 10 client.37715.objecter _send_command 4 2024-02-13T00:49:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.006+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb63411b7c0 con 0x7fb634059570 2024-02-13T00:49:03.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.007+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb63411b7c0 con 0x7fb634059570 2024-02-13T00:49:03.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.007+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:03.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.007+0000 7fb631ffb700 10 client.37715.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.007+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb634059400 osd=1 4 2024-02-13T00:49:03.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.007+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 1 4 2024-02-13T00:49:03.012 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:49:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb63abd0700 10 client.37715.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 5 2024-02-13T00:49:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb6340277c0 0x7fb63411eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634027650 osd=2 3 2024-02-13T00:49:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634027650 osd=2 3 2024-02-13T00:49:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634027650 osd=2 3 2024-02-13T00:49:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 5 2024-02-13T00:49:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb634027650 osd=2 3 2024-02-13T00:49:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 2 5 2024-02-13T00:49:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634027650 osd=2 4 2024-02-13T00:49:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb63abd0700 10 client.37715.objecter _send_command 5 2024-02-13T00:49:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6340ffc30 con 0x7fb6340277c0 2024-02-13T00:49:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb633fff700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb6340277c0 0x7fb63411eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb631ffb700 10 client.37715.objecter ms_handle_connect 0x7fb6340277c0 2024-02-13T00:49:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.008+0000 7fb633fff700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb6340277c0 0x7fb63411eb20 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.009+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb6340ffc30 con 0x7fb6340277c0 2024-02-13T00:49:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.009+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:49:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.009+0000 7fb631ffb700 10 client.37715.objecter _finish_command 5 = system:0 2024-02-13T00:49:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.009+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb634027650 osd=2 4 2024-02-13T00:49:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.009+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 2 5 2024-02-13T00:49:03.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.023+0000 7fb63abd0700 10 client.37715.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:03.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.023+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 6 2024-02-13T00:49:03.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.023+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634027650 osd=2 3 2024-02-13T00:49:03.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.023+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634027650 osd=2 3 2024-02-13T00:49:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.023+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634027650 osd=2 3 2024-02-13T00:49:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.023+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 6 2024-02-13T00:49:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.023+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb634027650 osd=2 3 2024-02-13T00:49:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.023+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 2 6 2024-02-13T00:49:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.023+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634027650 osd=2 4 2024-02-13T00:49:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.023+0000 7fb63abd0700 10 client.37715.objecter _send_command 6 2024-02-13T00:49:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.023+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb63411f010 con 0x7fb6340277c0 2024-02-13T00:49:03.028 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:49:03.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb63411f010 con 0x7fb6340277c0 2024-02-13T00:49:03.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:03.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb631ffb700 10 client.37715.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb634027650 osd=2 4 2024-02-13T00:49:03.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 2 6 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63abd0700 10 client.37715.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 7 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6813/604971011 conn(0x7fb6340ff300 0x7fb634122370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb6340ff190 osd=3 3 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6340ff190 osd=3 3 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb6340ff190 osd=3 3 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 7 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb6340ff190 osd=3 3 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 3 7 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6340ff190 osd=3 4 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63abd0700 10 client.37715.objecter _send_command 7 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb63411c6e0 con 0x7fb6340ff300 2024-02-13T00:49:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.024+0000 7fb63916d700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6813/604971011 conn(0x7fb6340ff300 0x7fb634122370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.025+0000 7fb631ffb700 10 client.37715.objecter ms_handle_connect 0x7fb6340ff300 2024-02-13T00:49:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.025+0000 7fb63916d700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6813/604971011 conn(0x7fb6340ff300 0x7fb634122370 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.025+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb63411c6e0 con 0x7fb6340ff300 2024-02-13T00:49:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.025+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:49:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.025+0000 7fb631ffb700 10 client.37715.objecter _finish_command 7 = system:0 2024-02-13T00:49:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.025+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb6340ff190 osd=3 4 2024-02-13T00:49:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.025+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 3 7 2024-02-13T00:49:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.039+0000 7fb63abd0700 10 client.37715.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.039+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 8 2024-02-13T00:49:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.039+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb6340ff190 osd=3 3 2024-02-13T00:49:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.039+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6340ff190 osd=3 3 2024-02-13T00:49:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.039+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb6340ff190 osd=3 3 2024-02-13T00:49:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.039+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 8 2024-02-13T00:49:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.039+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb6340ff190 osd=3 3 2024-02-13T00:49:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.039+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 3 8 2024-02-13T00:49:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.039+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6340ff190 osd=3 4 2024-02-13T00:49:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.039+0000 7fb63abd0700 10 client.37715.objecter _send_command 8 2024-02-13T00:49:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.039+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb63411c900 con 0x7fb6340ff300 2024-02-13T00:49:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb63411c900 con 0x7fb6340ff300 2024-02-13T00:49:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb631ffb700 10 client.37715.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb6340ff190 osd=3 4 2024-02-13T00:49:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 3 8 2024-02-13T00:49:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:49:03.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63abd0700 10 client.37715.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:03.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 9 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb63411f950 0x7fb63411fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb63411c900 osd=4 3 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb63411c900 osd=4 3 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb63411c900 osd=4 3 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 9 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb63411c900 osd=4 3 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 4 9 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb63411c900 osd=4 4 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63abd0700 10 client.37715.objecter _send_command 9 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb634128180 con 0x7fb63411f950 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.041+0000 7fb63896c700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb63411f950 0x7fb63411fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.042+0000 7fb631ffb700 10 client.37715.objecter ms_handle_connect 0x7fb63411f950 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.042+0000 7fb63896c700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb63411f950 0x7fb63411fdb0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.042+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb634128180 con 0x7fb63411f950 2024-02-13T00:49:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.042+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:49:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.042+0000 7fb631ffb700 10 client.37715.objecter _finish_command 9 = system:0 2024-02-13T00:49:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.042+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb63411c900 osd=4 4 2024-02-13T00:49:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.042+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 4 9 2024-02-13T00:49:03.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.057+0000 7fb63abd0700 10 client.37715.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:03.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.057+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 10 2024-02-13T00:49:03.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.057+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb63411c900 osd=4 3 2024-02-13T00:49:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.057+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb63411c900 osd=4 3 2024-02-13T00:49:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.057+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb63411c900 osd=4 3 2024-02-13T00:49:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.057+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 10 2024-02-13T00:49:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.057+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb63411c900 osd=4 3 2024-02-13T00:49:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.057+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 4 10 2024-02-13T00:49:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.057+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb63411c900 osd=4 4 2024-02-13T00:49:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.057+0000 7fb63abd0700 10 client.37715.objecter _send_command 10 2024-02-13T00:49:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.057+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb634128180 con 0x7fb63411f950 2024-02-13T00:49:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.058+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb634128180 con 0x7fb63411f950 2024-02-13T00:49:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.058+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.058+0000 7fb631ffb700 10 client.37715.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.058+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb63411c900 osd=4 4 2024-02-13T00:49:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.058+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 4 10 2024-02-13T00:49:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:49:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb63abd0700 10 client.37715.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 11 2024-02-13T00:49:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6804/2277917372 conn(0x7fb6341275d0 0x7fb6341279f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634128180 osd=5 3 2024-02-13T00:49:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634128180 osd=5 3 2024-02-13T00:49:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634128180 osd=5 3 2024-02-13T00:49:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 11 2024-02-13T00:49:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb634128180 osd=5 3 2024-02-13T00:49:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 5 11 2024-02-13T00:49:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634128180 osd=5 4 2024-02-13T00:49:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb63abd0700 10 client.37715.objecter _send_command 11 2024-02-13T00:49:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb63412b530 con 0x7fb6341275d0 2024-02-13T00:49:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb633fff700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6804/2277917372 conn(0x7fb6341275d0 0x7fb6341279f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb631ffb700 10 client.37715.objecter ms_handle_connect 0x7fb6341275d0 2024-02-13T00:49:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.059+0000 7fb633fff700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6804/2277917372 conn(0x7fb6341275d0 0x7fb6341279f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.060+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb63412b530 con 0x7fb6341275d0 2024-02-13T00:49:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.060+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:49:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.060+0000 7fb631ffb700 10 client.37715.objecter _finish_command 11 = system:0 2024-02-13T00:49:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.060+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb634128180 osd=5 4 2024-02-13T00:49:03.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.060+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 5 11 2024-02-13T00:49:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.074+0000 7fb63abd0700 10 client.37715.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.074+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 12 2024-02-13T00:49:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.074+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634128180 osd=5 3 2024-02-13T00:49:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.074+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634128180 osd=5 3 2024-02-13T00:49:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.074+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634128180 osd=5 3 2024-02-13T00:49:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.074+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 12 2024-02-13T00:49:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.074+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb634128180 osd=5 3 2024-02-13T00:49:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.074+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 5 12 2024-02-13T00:49:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.074+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634128180 osd=5 4 2024-02-13T00:49:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.074+0000 7fb63abd0700 10 client.37715.objecter _send_command 12 2024-02-13T00:49:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.074+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb6341259f0 con 0x7fb6341275d0 2024-02-13T00:49:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb6341259f0 con 0x7fb6341275d0 2024-02-13T00:49:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb631ffb700 10 client.37715.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb634128180 osd=5 4 2024-02-13T00:49:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 5 12 2024-02-13T00:49:03.084 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:49:03.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63abd0700 10 client.37715.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:03.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 13 2024-02-13T00:49:03.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb6341260d0 0x7fb634126530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:03.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb6341259f0 osd=6 3 2024-02-13T00:49:03.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6341259f0 osd=6 3 2024-02-13T00:49:03.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb6341259f0 osd=6 3 2024-02-13T00:49:03.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 13 2024-02-13T00:49:03.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb6341259f0 osd=6 3 2024-02-13T00:49:03.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 6 13 2024-02-13T00:49:03.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6341259f0 osd=6 4 2024-02-13T00:49:03.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63abd0700 10 client.37715.objecter _send_command 13 2024-02-13T00:49:03.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb634126ea0 con 0x7fb6341260d0 2024-02-13T00:49:03.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.076+0000 7fb63916d700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb6341260d0 0x7fb634126530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:03.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.077+0000 7fb631ffb700 10 client.37715.objecter ms_handle_connect 0x7fb6341260d0 2024-02-13T00:49:03.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.077+0000 7fb63916d700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb6341260d0 0x7fb634126530 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:03.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.077+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb634126ea0 con 0x7fb6341260d0 2024-02-13T00:49:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.077+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:49:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.077+0000 7fb631ffb700 10 client.37715.objecter _finish_command 13 = system:0 2024-02-13T00:49:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.077+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb6341259f0 osd=6 4 2024-02-13T00:49:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.077+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 6 13 2024-02-13T00:49:03.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:03.096 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.092+0000 7fb63abd0700 10 client.37715.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:03.096 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.092+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 14 2024-02-13T00:49:03.096 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.092+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb6341259f0 osd=6 3 2024-02-13T00:49:03.096 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.092+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6341259f0 osd=6 3 2024-02-13T00:49:03.096 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.092+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb6341259f0 osd=6 3 2024-02-13T00:49:03.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.092+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 14 2024-02-13T00:49:03.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.092+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb6341259f0 osd=6 3 2024-02-13T00:49:03.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.092+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 6 14 2024-02-13T00:49:03.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.092+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6341259f0 osd=6 4 2024-02-13T00:49:03.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.092+0000 7fb63abd0700 10 client.37715.objecter _send_command 14 2024-02-13T00:49:03.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.092+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb634126ea0 con 0x7fb6341260d0 2024-02-13T00:49:03.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.093+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb634126ea0 con 0x7fb6341260d0 2024-02-13T00:49:03.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.093+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:03.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.093+0000 7fb631ffb700 10 client.37715.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.093+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb6341259f0 osd=6 4 2024-02-13T00:49:03.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.093+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 6 14 2024-02-13T00:49:03.099 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:49:03.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63abd0700 10 client.37715.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:03.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 15 2024-02-13T00:49:03.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb63412c8d0 0x7fb63412ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:03.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634126ea0 osd=7 3 2024-02-13T00:49:03.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634126ea0 osd=7 3 2024-02-13T00:49:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634126ea0 osd=7 3 2024-02-13T00:49:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 15 2024-02-13T00:49:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb634126ea0 osd=7 3 2024-02-13T00:49:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 7 15 2024-02-13T00:49:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634126ea0 osd=7 4 2024-02-13T00:49:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63abd0700 10 client.37715.objecter _send_command 15 2024-02-13T00:49:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb634131910 con 0x7fb63412c8d0 2024-02-13T00:49:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63896c700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb63412c8d0 0x7fb63412ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:03.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb631ffb700 10 client.37715.objecter ms_handle_connect 0x7fb63412c8d0 2024-02-13T00:49:03.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.094+0000 7fb63896c700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb63412c8d0 0x7fb63412ecc0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:03.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.095+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb634131910 con 0x7fb63412c8d0 2024-02-13T00:49:03.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.095+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:49:03.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.095+0000 7fb631ffb700 10 client.37715.objecter _finish_command 15 = system:0 2024-02-13T00:49:03.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.095+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb634126ea0 osd=7 4 2024-02-13T00:49:03.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.095+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 7 15 2024-02-13T00:49:03.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.109+0000 7fb63abd0700 10 client.37715.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:03.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.109+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign -1 16 2024-02-13T00:49:03.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.109+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634126ea0 osd=7 3 2024-02-13T00:49:03.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.109+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634126ea0 osd=7 3 2024-02-13T00:49:03.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.109+0000 7fb63abd0700 20 client.37715.objecter _get_session s=0x7fb634126ea0 osd=7 3 2024-02-13T00:49:03.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.109+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_remove -1 16 2024-02-13T00:49:03.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.109+0000 7fb63abd0700 20 client.37715.objecter get_session s=0x7fb634126ea0 osd=7 3 2024-02-13T00:49:03.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.109+0000 7fb63abd0700 15 client.37715.objecter _session_command_op_assign 7 16 2024-02-13T00:49:03.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.109+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634126ea0 osd=7 4 2024-02-13T00:49:03.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.109+0000 7fb63abd0700 10 client.37715.objecter _send_command 16 2024-02-13T00:49:03.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.109+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb63412f1b0 con 0x7fb63412c8d0 2024-02-13T00:49:03.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.110+0000 7fb631ffb700 1 -- 172.21.15.162:0/129739496 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb63412f1b0 con 0x7fb63412c8d0 2024-02-13T00:49:03.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.110+0000 7fb631ffb700 10 client.37715.objecter ms_dispatch 0x7fb634001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:03.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.110+0000 7fb631ffb700 10 client.37715.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.110+0000 7fb631ffb700 20 client.37715.objecter put_session s=0x7fb634126ea0 osd=7 4 2024-02-13T00:49:03.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.110+0000 7fb631ffb700 15 client.37715.objecter _session_command_op_remove 7 16 2024-02-13T00:49:03.116 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:49:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 10 librados: watch_flush enter 2024-02-13T00:49:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 10 librados: watch_flush exit 2024-02-13T00:49:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 10 client.37715.objecter close_session for osd.0 2024-02-13T00:49:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb634028590 msgr2=0x7fb6340fea60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6801/2103743338 conn(0x7fb634028590 0x7fb6340fea60 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:03.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6341072c0 osd=0 1 2024-02-13T00:49:03.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 10 client.37715.objecter close_session for osd.1 2024-02-13T00:49:03.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6805/150599780 conn(0x7fb634059570 msgr2=0x7fb63411b2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:03.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6805/150599780 conn(0x7fb634059570 0x7fb63411b2d0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:03.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634059400 osd=1 1 2024-02-13T00:49:03.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 10 client.37715.objecter close_session for osd.2 2024-02-13T00:49:03.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb6340277c0 msgr2=0x7fb63411eb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:03.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6809/3689719376 conn(0x7fb6340277c0 0x7fb63411eb20 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:03.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634027650 osd=2 1 2024-02-13T00:49:03.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 10 client.37715.objecter close_session for osd.3 2024-02-13T00:49:03.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6813/604971011 conn(0x7fb6340ff300 msgr2=0x7fb634122370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:03.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6813/604971011 conn(0x7fb6340ff300 0x7fb634122370 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:03.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6340ff190 osd=3 1 2024-02-13T00:49:03.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 10 client.37715.objecter close_session for osd.4 2024-02-13T00:49:03.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb63411f950 msgr2=0x7fb63411fdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:03.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6800/3344263388 conn(0x7fb63411f950 0x7fb63411fdb0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:03.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63916d700 1 -- 172.21.15.162:0/129739496 reap_dead start 2024-02-13T00:49:03.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb63411c900 osd=4 1 2024-02-13T00:49:03.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 10 client.37715.objecter close_session for osd.5 2024-02-13T00:49:03.122 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6804/2277917372 conn(0x7fb6341275d0 msgr2=0x7fb6341279f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6804/2277917372 conn(0x7fb6341275d0 0x7fb6341279f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634128180 osd=5 1 2024-02-13T00:49:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 10 client.37715.objecter close_session for osd.6 2024-02-13T00:49:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb6341260d0 msgr2=0x7fb634126530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6808/1506640341 conn(0x7fb6341260d0 0x7fb634126530 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb6341259f0 osd=6 1 2024-02-13T00:49:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 10 client.37715.objecter close_session for osd.7 2024-02-13T00:49:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb63412c8d0 msgr2=0x7fb63412ecc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.181:6812/1964192772 conn(0x7fb63412c8d0 0x7fb63412ecc0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 20 client.37715.objecter put_session s=0x7fb634126ea0 osd=7 1 2024-02-13T00:49:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 20 client.37715.objecter shutdown clearing up homeless session... 2024-02-13T00:49:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 10 client.37715.objecter successfully canceled tick 2024-02-13T00:49:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63916d700 1 -- 172.21.15.162:0/129739496 reap_dead start 2024-02-13T00:49:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63916d700 1 -- 172.21.15.162:0/129739496 reap_dead start 2024-02-13T00:49:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63916d700 1 -- 172.21.15.162:0/129739496 reap_dead start 2024-02-13T00:49:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63916d700 1 -- 172.21.15.162:0/129739496 reap_dead start 2024-02-13T00:49:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb624062c90 msgr2=0x7fb624065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb624062c90 0x7fb624065150 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 rx=0x7fb628005010 tx=0x7fb628007f90).stop 2024-02-13T00:49:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6340f9b70 msgr2=0x7fb63406b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6340f9b70 0x7fb63406b660 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7fb6200041d0 tx=0x7fb620009f60).stop 2024-02-13T00:49:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 shutdown_connections 2024-02-13T00:49:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb6340f9b70 0x7fb63406b660 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 --2- 172.21.15.162:0/129739496 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb624062c90 0x7fb624065150 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 >> 172.21.15.162:0/129739496 conn(0x7fb6340f38c0 msgr2=0x7fb6340fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 shutdown_connections 2024-02-13T00:49:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 -- 172.21.15.162:0/129739496 wait complete. 2024-02-13T00:49:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:03.111+0000 7fb63abd0700 1 librados: shutdown 2024-02-13T00:49:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.128 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.128 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.128 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.128 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.128 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:49:03.129 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:03.247 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:03.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:03.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-13T00:49:03.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:03.380 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:03.563 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:49:03.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23048 ms) 2024-02-13T00:49:03.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-02-13T00:49:03.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3039 ms) 2024-02-13T00:49:03.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-02-13T00:49:03.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3017 ms) 2024-02-13T00:49:03.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-02-13T00:49:03.581 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (2848 ms) 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3023 ms) 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4030 ms) 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38988 ms) 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17132 ms) 2024-02-13T00:49:03.582 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (3872 ms) 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3061 ms) 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24934 ms) 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15084 ms) 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (4917 ms) 2024-02-13T00:49:03.583 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: waiting for scrubs... 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: done waiting 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25100 ms) 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9926 ms) 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9024 ms) 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.584 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9101 ms) 2024-02-13T00:49:03.585 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-02-13T00:49:03.585 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.585 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8922 ms) 2024-02-13T00:49:03.585 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-02-13T00:49:03.585 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:49:03.585 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6057 ms) 2024-02-13T00:49:03.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:02.257155+0000 osd.6 (osd.6) 159 : cluster [DBG] 28.8 scrub starts 2024-02-13T00:49:03.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:02.260042+0000 osd.6 (osd.6) 160 : cluster [DBG] 28.8 scrub ok 2024-02-13T00:49:03.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:02.468330+0000 mgr.y (mgr.14544) 466 : cluster [DBG] pgmap v791: 233 pgs: 233 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:49:03.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:02.571304+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e528: 8 total, 8 up, 4 in 2024-02-13T00:49:03.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:02.605647+0000 mon.b (mon.1) 650 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:03.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:02.606024+0000 mon.a (mon.0) 2842 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:03.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:02.606475+0000 mon.b (mon.1) 651 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-86"}]: dispatch 2024-02-13T00:49:03.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:02.606700+0000 mon.a (mon.0) 2843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-86"}]: dispatch 2024-02-13T00:49:03.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:02.972064+0000 mon.c (mon.2) 428 : audit [DBG] from='client.? 172.21.15.162:0/129739496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:02.257155+0000 osd.6 (osd.6) 159 : cluster [DBG] 28.8 scrub starts 2024-02-13T00:49:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:02.260042+0000 osd.6 (osd.6) 160 : cluster [DBG] 28.8 scrub ok 2024-02-13T00:49:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:02.468330+0000 mgr.y (mgr.14544) 466 : cluster [DBG] pgmap v791: 233 pgs: 233 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:49:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:02.571304+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e528: 8 total, 8 up, 4 in 2024-02-13T00:49:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:02.605647+0000 mon.b (mon.1) 650 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:02.606024+0000 mon.a (mon.0) 2842 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:02.606475+0000 mon.b (mon.1) 651 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-86"}]: dispatch 2024-02-13T00:49:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:02.606700+0000 mon.a (mon.0) 2843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-86"}]: dispatch 2024-02-13T00:49:03.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:02.972064+0000 mon.c (mon.2) 428 : audit [DBG] from='client.? 172.21.15.162:0/129739496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:03.855 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:02.257155+0000 osd.6 (osd.6) 159 : cluster [DBG] 28.8 scrub starts 2024-02-13T00:49:03.855 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:02.260042+0000 osd.6 (osd.6) 160 : cluster [DBG] 28.8 scrub ok 2024-02-13T00:49:03.855 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:02.468330+0000 mgr.y (mgr.14544) 466 : cluster [DBG] pgmap v791: 233 pgs: 233 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:49:03.855 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:02.571304+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e528: 8 total, 8 up, 4 in 2024-02-13T00:49:03.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:02.605647+0000 mon.b (mon.1) 650 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:03.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:02.606024+0000 mon.a (mon.0) 2842 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:03.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:02.606475+0000 mon.b (mon.1) 651 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-86"}]: dispatch 2024-02-13T00:49:03.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:02.606700+0000 mon.a (mon.0) 2843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-86"}]: dispatch 2024-02-13T00:49:03.856 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:02.972064+0000 mon.c (mon.2) 428 : audit [DBG] from='client.? 172.21.15.162:0/129739496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:03.875 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:49:03.930 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:03.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:03.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:04.034 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:04.062 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:04.145 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:03.290996+0000 osd.6 (osd.6) 161 : cluster [DBG] 4.c scrub starts 2024-02-13T00:49:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:03.294495+0000 osd.6 (osd.6) 162 : cluster [DBG] 4.c scrub ok 2024-02-13T00:49:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:03.575258+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e529: 8 total, 8 up, 4 in 2024-02-13T00:49:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:04.339637+0000 mon.a (mon.0) 2845 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:04.469142+0000 mon.c (mon.2) 429 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-13T00:49:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:04.470212+0000 mon.a (mon.0) 2846 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-13T00:49:04.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:03.290996+0000 osd.6 (osd.6) 161 : cluster [DBG] 4.c scrub starts 2024-02-13T00:49:04.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:03.294495+0000 osd.6 (osd.6) 162 : cluster [DBG] 4.c scrub ok 2024-02-13T00:49:04.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:03.575258+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e529: 8 total, 8 up, 4 in 2024-02-13T00:49:04.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:04.339637+0000 mon.a (mon.0) 2845 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:04.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:04.469142+0000 mon.c (mon.2) 429 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-13T00:49:04.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:04.470212+0000 mon.a (mon.0) 2846 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-13T00:49:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:03.290996+0000 osd.6 (osd.6) 161 : cluster [DBG] 4.c scrub starts 2024-02-13T00:49:04.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:03.294495+0000 osd.6 (osd.6) 162 : cluster [DBG] 4.c scrub ok 2024-02-13T00:49:04.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:03.575258+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e529: 8 total, 8 up, 4 in 2024-02-13T00:49:04.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:04.339637+0000 mon.a (mon.0) 2845 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:04.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:04.469142+0000 mon.c (mon.2) 429 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-13T00:49:04.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:04.470212+0000 mon.a (mon.0) 2846 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-13T00:49:05.020 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6] out_osds: [3, 2, 7, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-13T00:49:05.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-13T00:49:05.112 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:05.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:05.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:05.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:05.244 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:05.378 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:49:05.379 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:49:05.379 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-02-13T00:49:05.379 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-02-13T00:49:05.379 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:49:05.706 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:04.279675+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:49:05.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:04.282526+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:49:05.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:04.468800+0000 mgr.y (mgr.14544) 467 : cluster [DBG] pgmap v794: 200 pgs: 200 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:05.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:04.589227+0000 mon.a (mon.0) 2847 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-13T00:49:05.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:04.589307+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e530: 8 total, 8 up, 4 in 2024-02-13T00:49:05.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:04.594559+0000 mon.b (mon.1) 652 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:05.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:04.596017+0000 mon.a (mon.0) 2849 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:05.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:05.475144+0000 mon.a (mon.0) 2850 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-88","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:05.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:05.475235+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e531: 8 total, 8 up, 4 in 2024-02-13T00:49:05.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:05.497890+0000 mon.b (mon.1) 653 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:49:05.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:05.500378+0000 mon.a (mon.0) 2852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:49:05.900 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:49:05.927 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:04.279675+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:49:05.927 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:04.282526+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:49:05.927 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:04.468800+0000 mgr.y (mgr.14544) 467 : cluster [DBG] pgmap v794: 200 pgs: 200 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:05.928 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:04.589227+0000 mon.a (mon.0) 2847 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-13T00:49:05.928 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:04.589307+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e530: 8 total, 8 up, 4 in 2024-02-13T00:49:05.928 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:04.594559+0000 mon.b (mon.1) 652 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:05.928 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:04.596017+0000 mon.a (mon.0) 2849 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:05.928 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:05.475144+0000 mon.a (mon.0) 2850 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-88","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:05.928 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:05.475235+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e531: 8 total, 8 up, 4 in 2024-02-13T00:49:05.928 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:05.497890+0000 mon.b (mon.1) 653 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:49:05.928 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:05.500378+0000 mon.a (mon.0) 2852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:49:05.928 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:04.279675+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:49:05.928 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:04.282526+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:49:05.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:04.468800+0000 mgr.y (mgr.14544) 467 : cluster [DBG] pgmap v794: 200 pgs: 200 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:05.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:04.589227+0000 mon.a (mon.0) 2847 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-13T00:49:05.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:04.589307+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e530: 8 total, 8 up, 4 in 2024-02-13T00:49:05.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:04.594559+0000 mon.b (mon.1) 652 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:05.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:04.596017+0000 mon.a (mon.0) 2849 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:05.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:05.475144+0000 mon.a (mon.0) 2850 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-88","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:05.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:05.475235+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e531: 8 total, 8 up, 4 in 2024-02-13T00:49:05.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:05.497890+0000 mon.b (mon.1) 653 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:49:05.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:05.500378+0000 mon.a (mon.0) 2852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:49:06.020 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:49:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:49:06.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-13T00:49:06.304 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:49:06.330 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:49:06] "GET /metrics HTTP/1.1" 200 185678 "" "Prometheus/2.33.4" 2024-02-13T00:49:06.395 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:06.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:06.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:06.528 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:06.626 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:06.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:06.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:06.758 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:06.832 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:06.833 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:05.728193+0000 mgr.y (mgr.14544) 468 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:06.834 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:05.884784+0000 osd.0 (osd.0) 207 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:49:06.834 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:05.887660+0000 osd.0 (osd.0) 208 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:49:06.834 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:06.485461+0000 mon.a (mon.0) 2853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:49:06.834 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:06.485532+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e532: 8 total, 8 up, 4 in 2024-02-13T00:49:06.834 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:06.487236+0000 mon.b (mon.1) 654 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-88", "mode": "writeback"}]: dispatch 2024-02-13T00:49:06.834 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:06.487616+0000 mon.a (mon.0) 2855 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-88", "mode": "writeback"}]: dispatch 2024-02-13T00:49:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:06.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:06.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:06.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.954+0000 7f203885b700 1 -- 172.21.15.162:0/848855508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20340fd490 msgr2=0x7f20340f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:06.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.954+0000 7f203885b700 1 --2- 172.21.15.162:0/848855508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20340fd490 0x7f20340f6cc0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 rx=0x7f2028004870 tx=0x7f2028009cb0).stop 2024-02-13T00:49:06.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.954+0000 7f203885b700 1 -- 172.21.15.162:0/848855508 shutdown_connections 2024-02-13T00:49:06.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.954+0000 7f203885b700 1 --2- 172.21.15.162:0/848855508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20340f73a0 0x7f20341039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:06.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.954+0000 7f203885b700 1 --2- 172.21.15.162:0/848855508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20340fcb30 0x7f20340fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:06.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.954+0000 7f203885b700 1 --2- 172.21.15.162:0/848855508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20340fd490 0x7f20340f6cc0 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:06.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.954+0000 7f203885b700 1 -- 172.21.15.162:0/848855508 >> 172.21.15.162:0/848855508 conn(0x7f203406eb90 msgr2=0x7f20340f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:06.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.954+0000 7f203885b700 1 -- 172.21.15.162:0/848855508 shutdown_connections 2024-02-13T00:49:06.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.954+0000 7f203885b700 1 -- 172.21.15.162:0/848855508 wait complete. 2024-02-13T00:49:06.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.955+0000 7f203885b700 1 librados: starting msgr at 2024-02-13T00:49:06.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.955+0000 7f203885b700 1 librados: starting objecter 2024-02-13T00:49:06.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.955+0000 7f203885b700 1 Processor -- start 2024-02-13T00:49:06.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.955+0000 7f203885b700 1 -- start start 2024-02-13T00:49:06.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.955+0000 7f203885b700 1 librados: setting wanted keys 2024-02-13T00:49:06.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.955+0000 7f203885b700 1 librados: calling monclient init 2024-02-13T00:49:06.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f203885b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20340f73a0 0x7f2034109250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:06.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f203885b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20340fcb30 0x7f2034109740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:06.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f203885b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20340fd490 0x7f2034109c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:06.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f203885b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20341143e0 con 0x7f20340f73a0 2024-02-13T00:49:06.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f203885b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2034114550 con 0x7f20340fcb30 2024-02-13T00:49:06.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f203885b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2034114710 con 0x7f20340fd490 2024-02-13T00:49:06.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f2031d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20340fcb30 0x7f2034109740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:06.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f2032d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20340fd490 0x7f2034109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:06.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f2031d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20340fcb30 0x7f2034109740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:44776/0 (socket says 172.21.15.162:44776) 2024-02-13T00:49:06.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f2031d9b700 1 -- 172.21.15.162:0/2283452408 learned_addr learned my addr 172.21.15.162:0/2283452408 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:06.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f203259c700 1 --2- 172.21.15.162:0/2283452408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20340f73a0 0x7f2034109250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:06.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f2032d9d700 1 -- 172.21.15.162:0/2283452408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20340fcb30 msgr2=0x7f2034109740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:06.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f2032d9d700 1 --2- 172.21.15.162:0/2283452408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20340fcb30 0x7f2034109740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:06.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f2032d9d700 1 -- 172.21.15.162:0/2283452408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20340f73a0 msgr2=0x7f2034109250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:06.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f2032d9d700 1 --2- 172.21.15.162:0/2283452408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20340f73a0 0x7f2034109250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:06.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f2032d9d700 1 -- 172.21.15.162:0/2283452408 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202c0097c0 con 0x7f20340fd490 2024-02-13T00:49:06.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f2032d9d700 1 --2- 172.21.15.162:0/2283452408 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20340fd490 0x7f2034109c70 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7f202c001cb0 tx=0x7f202c004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:06.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f20237fe700 10 client.?.objecter ms_handle_connect 0x7f20340fd490 2024-02-13T00:49:06.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.956+0000 7f20237fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:06.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.957+0000 7f20237fe700 1 -- 172.21.15.162:0/2283452408 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0079b0 con 0x7f20340fd490 2024-02-13T00:49:06.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.957+0000 7f20237fe700 1 -- 172.21.15.162:0/2283452408 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f202c0090d0 con 0x7f20340fd490 2024-02-13T00:49:06.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.957+0000 7f203885b700 1 -- 172.21.15.162:0/2283452408 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2028005040 con 0x7f20340fd490 2024-02-13T00:49:06.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.957+0000 7f203885b700 10 client.37721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:06.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.957+0000 7f203885b700 1 -- 172.21.15.162:0/2283452408 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2034114dd0 con 0x7f20340fd490 2024-02-13T00:49:06.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.957+0000 7f203885b700 1 librados: init done 2024-02-13T00:49:06.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.957+0000 7f20237fe700 1 -- 172.21.15.162:0/2283452408 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c00fc30 con 0x7f20340fd490 2024-02-13T00:49:06.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.959+0000 7f203885b700 1 -- 172.21.15.162:0/2283452408 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2014004f50 con 0x7f20340fd490 2024-02-13T00:49:06.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.959+0000 7f20237fe700 1 -- 172.21.15.162:0/2283452408 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f202c00fd90 con 0x7f20340fd490 2024-02-13T00:49:06.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.959+0000 7f20237fe700 1 --2- 172.21.15.162:0/2283452408 >> v2:172.21.15.162:6800/2866534053 conn(0x7f201c0670f0 0x7f201c0695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:06.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.960+0000 7f20237fe700 1 -- 172.21.15.162:0/2283452408 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(532..532 src has 449..532) v4 ==== 8062+0+0 (secure 0 0 0) 0x7f202c07d870 con 0x7f20340fd490 2024-02-13T00:49:06.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.960+0000 7f20237fe700 10 client.37721.objecter ms_dispatch 0x7f2034001550 osd_map(532..532 src has 449..532) v4 2024-02-13T00:49:06.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.960+0000 7f20237fe700 3 client.37721.objecter handle_osd_map got epochs [532,532] > 0 2024-02-13T00:49:06.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.960+0000 7f20237fe700 3 client.37721.objecter handle_osd_map decoding full epoch 532 2024-02-13T00:49:06.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.960+0000 7f203259c700 1 --2- 172.21.15.162:0/2283452408 >> v2:172.21.15.162:6800/2866534053 conn(0x7f201c0670f0 0x7f201c0695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:06.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.960+0000 7f20237fe700 20 client.37721.objecter dump_active .. 0 homeless 2024-02-13T00:49:06.965 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:06.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.960+0000 7f20237fe700 10 client.37721.objecter ms_handle_connect 0x7f201c0670f0 2024-02-13T00:49:06.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.960+0000 7f203259c700 1 --2- 172.21.15.162:0/2283452408 >> v2:172.21.15.162:6800/2866534053 conn(0x7f201c0670f0 0x7f201c0695b0 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7f2024004de0 tx=0x7f202400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:06.965 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:06.966 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:05.728193+0000 mgr.y (mgr.14544) 468 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:06.966 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:05.884784+0000 osd.0 (osd.0) 207 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:49:06.966 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:05.887660+0000 osd.0 (osd.0) 208 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:49:06.966 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:06.485461+0000 mon.a (mon.0) 2853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:49:06.966 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:06.485532+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e532: 8 total, 8 up, 4 in 2024-02-13T00:49:06.966 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:06.487236+0000 mon.b (mon.1) 654 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-88", "mode": "writeback"}]: dispatch 2024-02-13T00:49:06.966 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:06.487616+0000 mon.a (mon.0) 2855 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-88", "mode": "writeback"}]: dispatch 2024-02-13T00:49:06.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:05.728193+0000 mgr.y (mgr.14544) 468 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:06.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:05.884784+0000 osd.0 (osd.0) 207 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:49:06.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:05.887660+0000 osd.0 (osd.0) 208 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:49:06.967 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:06.485461+0000 mon.a (mon.0) 2853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:49:06.967 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:06.485532+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e532: 8 total, 8 up, 4 in 2024-02-13T00:49:06.967 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:06.487236+0000 mon.b (mon.1) 654 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-88", "mode": "writeback"}]: dispatch 2024-02-13T00:49:06.967 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:06.487616+0000 mon.a (mon.0) 2855 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-88", "mode": "writeback"}]: dispatch 2024-02-13T00:49:06.968 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:06.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:06.967+0000 7f20237fe700 1 -- 172.21.15.162:0/2283452408 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f202c050e20 con 0x7f20340fd490 2024-02-13T00:49:06.991 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:06.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:07.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:07.121 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:07.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.160+0000 7f203885b700 1 -- 172.21.15.162:0/2283452408 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f2014005780 con 0x7f20340fd490 2024-02-13T00:49:07.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:07.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:07.392 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:07.419 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:07.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.488+0000 7f20237fe700 1 -- 172.21.15.162:0/2283452408 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v533) v1 ==== 87+0+0 (secure 0 0 0) 0x7f202c0191b0 con 0x7f20340fd490 2024-02-13T00:49:07.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.490+0000 7f203885b700 10 librados: watch_flush enter 2024-02-13T00:49:07.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.490+0000 7f203885b700 10 librados: watch_flush exit 2024-02-13T00:49:07.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.490+0000 7f203885b700 20 client.37721.objecter shutdown clearing up homeless session... 2024-02-13T00:49:07.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.490+0000 7f203885b700 10 client.37721.objecter successfully canceled tick 2024-02-13T00:49:07.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.490+0000 7f203885b700 1 -- 172.21.15.162:0/2283452408 >> v2:172.21.15.162:6800/2866534053 conn(0x7f201c0670f0 msgr2=0x7f201c0695b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:07.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.490+0000 7f203885b700 1 --2- 172.21.15.162:0/2283452408 >> v2:172.21.15.162:6800/2866534053 conn(0x7f201c0670f0 0x7f201c0695b0 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7f2024004de0 tx=0x7f202400b040).stop 2024-02-13T00:49:07.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.490+0000 7f203885b700 1 -- 172.21.15.162:0/2283452408 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20340fd490 msgr2=0x7f2034109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:07.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.490+0000 7f203885b700 1 --2- 172.21.15.162:0/2283452408 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20340fd490 0x7f2034109c70 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7f202c001cb0 tx=0x7f202c004c50).stop 2024-02-13T00:49:07.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.491+0000 7f203885b700 1 -- 172.21.15.162:0/2283452408 shutdown_connections 2024-02-13T00:49:07.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.491+0000 7f203885b700 1 --2- 172.21.15.162:0/2283452408 >> v2:172.21.15.162:6800/2866534053 conn(0x7f201c0670f0 0x7f201c0695b0 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:07.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.491+0000 7f203885b700 1 --2- 172.21.15.162:0/2283452408 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20340fd490 0x7f2034109c70 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:07.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.491+0000 7f203885b700 1 --2- 172.21.15.162:0/2283452408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20340f73a0 0x7f2034109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:07.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.491+0000 7f203885b700 1 --2- 172.21.15.162:0/2283452408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20340fcb30 0x7f2034109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:07.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.491+0000 7f203885b700 1 -- 172.21.15.162:0/2283452408 >> 172.21.15.162:0/2283452408 conn(0x7f203406eb90 msgr2=0x7f20340fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:07.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.493+0000 7f203885b700 1 -- 172.21.15.162:0/2283452408 shutdown_connections 2024-02-13T00:49:07.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.493+0000 7f203885b700 1 -- 172.21.15.162:0/2283452408 wait complete. 2024-02-13T00:49:07.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:07.495+0000 7f203885b700 1 librados: shutdown 2024-02-13T00:49:07.499 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:49:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:06.151463+0000 osd.1 (osd.1) 229 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:49:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:06.154926+0000 osd.1 (osd.1) 230 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:49:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:06.469237+0000 mgr.y (mgr.14544) 469 : cluster [DBG] pgmap v797: 232 pgs: 2 peering, 32 creating+peering, 198 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-13T00:49:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:07.161878+0000 mon.c (mon.2) 430 : audit [INF] from='client.? 172.21.15.162:0/2283452408' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:49:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:07.162203+0000 mon.a (mon.0) 2856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:49:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:07.480384+0000 mon.a (mon.0) 2857 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:49:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:07.480412+0000 mon.a (mon.0) 2858 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:07.486799+0000 mon.a (mon.0) 2859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-88", "mode": "writeback"}]': finished 2024-02-13T00:49:07.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:07.486901+0000 mon.a (mon.0) 2860 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:49:07.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:07.486952+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e533: 8 total, 8 up, 4 in 2024-02-13T00:49:07.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:07.488962+0000 mon.b (mon.1) 655 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-88"}]: dispatch 2024-02-13T00:49:07.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:07.489473+0000 mon.a (mon.0) 2862 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-88"}]: dispatch 2024-02-13T00:49:07.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:06.151463+0000 osd.1 (osd.1) 229 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:49:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:06.154926+0000 osd.1 (osd.1) 230 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:49:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:06.469237+0000 mgr.y (mgr.14544) 469 : cluster [DBG] pgmap v797: 232 pgs: 2 peering, 32 creating+peering, 198 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-13T00:49:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:07.161878+0000 mon.c (mon.2) 430 : audit [INF] from='client.? 172.21.15.162:0/2283452408' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:49:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:07.162203+0000 mon.a (mon.0) 2856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:49:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:07.480384+0000 mon.a (mon.0) 2857 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:49:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:07.480412+0000 mon.a (mon.0) 2858 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:07.486799+0000 mon.a (mon.0) 2859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-88", "mode": "writeback"}]': finished 2024-02-13T00:49:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:07.486901+0000 mon.a (mon.0) 2860 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:49:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:07.486952+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e533: 8 total, 8 up, 4 in 2024-02-13T00:49:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:07.488962+0000 mon.b (mon.1) 655 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-88"}]: dispatch 2024-02-13T00:49:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:07.489473+0000 mon.a (mon.0) 2862 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-88"}]: dispatch 2024-02-13T00:49:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:06.151463+0000 osd.1 (osd.1) 229 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:49:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:06.154926+0000 osd.1 (osd.1) 230 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:49:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:06.469237+0000 mgr.y (mgr.14544) 469 : cluster [DBG] pgmap v797: 232 pgs: 2 peering, 32 creating+peering, 198 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-13T00:49:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:07.161878+0000 mon.c (mon.2) 430 : audit [INF] from='client.? 172.21.15.162:0/2283452408' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:49:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:07.162203+0000 mon.a (mon.0) 2856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:49:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:07.480384+0000 mon.a (mon.0) 2857 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:49:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:07.480412+0000 mon.a (mon.0) 2858 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:07.486799+0000 mon.a (mon.0) 2859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-88", "mode": "writeback"}]': finished 2024-02-13T00:49:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:07.486901+0000 mon.a (mon.0) 2860 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:49:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:07.486952+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e533: 8 total, 8 up, 4 in 2024-02-13T00:49:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:07.488962+0000 mon.b (mon.1) 655 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-88"}]: dispatch 2024-02-13T00:49:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:07.489473+0000 mon.a (mon.0) 2862 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-88"}]: dispatch 2024-02-13T00:49:08.092 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:49:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:49:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:49:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [] 2024-02-13T00:49:08.092 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:49:08.793 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:49:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:49:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:49:08.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:07.191230+0000 osd.1 (osd.1) 231 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:49:08.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:07.192896+0000 osd.1 (osd.1) 232 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:49:08.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:07.864617+0000 osd.7 (osd.7) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:08.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:07.864814+0000 osd.7 (osd.7) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:08.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:08.470071+0000 mon.c (mon.2) 431 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-13T00:49:08.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:08.470967+0000 mon.a (mon.0) 2863 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-13T00:49:08.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:08.484730+0000 mon.a (mon.0) 2864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-88"}]': finished 2024-02-13T00:49:08.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:08.484807+0000 mon.a (mon.0) 2865 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-13T00:49:08.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:08.484868+0000 mon.a (mon.0) 2866 : cluster [DBG] osdmap e534: 8 total, 8 up, 4 in 2024-02-13T00:49:08.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:08.488217+0000 mon.b (mon.1) 656 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:49:08.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:08.489041+0000 mon.a (mon.0) 2867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:49:08.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-13T00:49:08.828 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:07.191230+0000 osd.1 (osd.1) 231 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:49:08.828 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:07.192896+0000 osd.1 (osd.1) 232 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:49:08.828 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:07.864617+0000 osd.7 (osd.7) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:08.828 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:07.864814+0000 osd.7 (osd.7) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:08.828 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:08.470071+0000 mon.c (mon.2) 431 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-13T00:49:08.828 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:08.470967+0000 mon.a (mon.0) 2863 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-13T00:49:08.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:08.484730+0000 mon.a (mon.0) 2864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-88"}]': finished 2024-02-13T00:49:08.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:08.484807+0000 mon.a (mon.0) 2865 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-13T00:49:08.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:08.484868+0000 mon.a (mon.0) 2866 : cluster [DBG] osdmap e534: 8 total, 8 up, 4 in 2024-02-13T00:49:08.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:08.488217+0000 mon.b (mon.1) 656 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:49:08.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:08.489041+0000 mon.a (mon.0) 2867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:49:08.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:07.191230+0000 osd.1 (osd.1) 231 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:49:08.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:07.192896+0000 osd.1 (osd.1) 232 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:49:08.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:07.864617+0000 osd.7 (osd.7) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:08.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:07.864814+0000 osd.7 (osd.7) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:08.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:08.470071+0000 mon.c (mon.2) 431 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-13T00:49:08.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:08.470967+0000 mon.a (mon.0) 2863 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-13T00:49:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:08.484730+0000 mon.a (mon.0) 2864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-88"}]': finished 2024-02-13T00:49:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:08.484807+0000 mon.a (mon.0) 2865 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-13T00:49:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:08.484868+0000 mon.a (mon.0) 2866 : cluster [DBG] osdmap e534: 8 total, 8 up, 4 in 2024-02-13T00:49:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:08.488217+0000 mon.b (mon.1) 656 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:49:08.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:08.489041+0000 mon.a (mon.0) 2867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:49:09.172 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:09.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:09.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:09.275 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:09.308 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:09.327 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:09.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:09.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:09.431 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:09.460 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:09.480 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:09.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:09.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:09.584 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:09.613 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:08.469716+0000 mgr.y (mgr.14544) 470 : cluster [DBG] pgmap v800: 231 pgs: 1 peering, 32 creating+peering, 198 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-13T00:49:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:09.482941+0000 mon.a (mon.0) 2868 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:49:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:09.486199+0000 mon.a (mon.0) 2869 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:49:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:09.486272+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e535: 8 total, 8 up, 4 in 2024-02-13T00:49:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:09.488792+0000 mon.b (mon.1) 657 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-13T00:49:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:09.489307+0000 mon.a (mon.0) 2871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-13T00:49:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:08.469716+0000 mgr.y (mgr.14544) 470 : cluster [DBG] pgmap v800: 231 pgs: 1 peering, 32 creating+peering, 198 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-13T00:49:09.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:09.482941+0000 mon.a (mon.0) 2868 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:49:09.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:09.486199+0000 mon.a (mon.0) 2869 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:49:09.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:09.486272+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e535: 8 total, 8 up, 4 in 2024-02-13T00:49:09.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:09.488792+0000 mon.b (mon.1) 657 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-13T00:49:09.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:09.489307+0000 mon.a (mon.0) 2871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-13T00:49:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:08.469716+0000 mgr.y (mgr.14544) 470 : cluster [DBG] pgmap v800: 231 pgs: 1 peering, 32 creating+peering, 198 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-13T00:49:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:09.482941+0000 mon.a (mon.0) 2868 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:49:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:09.486199+0000 mon.a (mon.0) 2869 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:49:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:09.486272+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e535: 8 total, 8 up, 4 in 2024-02-13T00:49:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:09.488792+0000 mon.b (mon.1) 657 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-13T00:49:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:09.489307+0000 mon.a (mon.0) 2871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-13T00:49:10.432 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:49:10.645 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:10.663 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:10.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:10.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.675+0000 7f371398f700 1 -- 172.21.15.162:0/613376062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f370c0f95e0 msgr2=0x7f370c101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:10.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.675+0000 7f371398f700 1 --2- 172.21.15.162:0/613376062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f370c0f95e0 0x7f370c101d80 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 rx=0x7f36fc004870 tx=0x7f36fc009cb0).stop 2024-02-13T00:49:10.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.675+0000 7f371398f700 1 -- 172.21.15.162:0/613376062 shutdown_connections 2024-02-13T00:49:10.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.675+0000 7f371398f700 1 --2- 172.21.15.162:0/613376062 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f370c102270 0x7f370c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:10.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.675+0000 7f371398f700 1 --2- 172.21.15.162:0/613376062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f370c0f8d10 0x7f370c0f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:10.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.675+0000 7f371398f700 1 --2- 172.21.15.162:0/613376062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f370c0f95e0 0x7f370c101d80 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:10.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.675+0000 7f371398f700 1 -- 172.21.15.162:0/613376062 >> 172.21.15.162:0/613376062 conn(0x7f370c06eb90 msgr2=0x7f370c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:10.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.675+0000 7f371398f700 1 -- 172.21.15.162:0/613376062 shutdown_connections 2024-02-13T00:49:10.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.675+0000 7f371398f700 1 -- 172.21.15.162:0/613376062 wait complete. 2024-02-13T00:49:10.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.675+0000 7f371398f700 1 librados: starting msgr at 2024-02-13T00:49:10.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.675+0000 7f371398f700 1 librados: starting objecter 2024-02-13T00:49:10.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f371398f700 1 Processor -- start 2024-02-13T00:49:10.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f371398f700 1 -- start start 2024-02-13T00:49:10.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f371398f700 1 librados: setting wanted keys 2024-02-13T00:49:10.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f371398f700 1 librados: calling monclient init 2024-02-13T00:49:10.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f371398f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f370c0f8d10 0x7f370c10e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:10.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f371398f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f370c0f95e0 0x7f370c10e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f371398f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f370c102270 0x7f370c1083b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f371398f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c1088a0 con 0x7f370c0f95e0 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f371398f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c108a10 con 0x7f370c0f8d10 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f371398f700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370c108b80 con 0x7f370c102270 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f3710f2a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f370c0f95e0 0x7f370c10e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f3710f2a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f370c0f95e0 0x7f370c10e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:42806/0 (socket says 172.21.15.162:42806) 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f3710f2a700 1 -- 172.21.15.162:0/258817619 learned_addr learned my addr 172.21.15.162:0/258817619 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f371172b700 1 --2- 172.21.15.162:0/258817619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f370c0f8d10 0x7f370c10e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f3711f2c700 1 --2- 172.21.15.162:0/258817619 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f370c102270 0x7f370c1083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f3710f2a700 1 -- 172.21.15.162:0/258817619 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f370c102270 msgr2=0x7f370c1083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f3710f2a700 1 --2- 172.21.15.162:0/258817619 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f370c102270 0x7f370c1083b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f3710f2a700 1 -- 172.21.15.162:0/258817619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f370c0f8d10 msgr2=0x7f370c10e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f3710f2a700 1 --2- 172.21.15.162:0/258817619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f370c0f8d10 0x7f370c10e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:10.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.676+0000 7f3710f2a700 1 -- 172.21.15.162:0/258817619 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36fc005040 con 0x7f370c0f95e0 2024-02-13T00:49:10.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.677+0000 7f3710f2a700 1 --2- 172.21.15.162:0/258817619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f370c0f95e0 0x7f370c10e6c0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f36fc013760 tx=0x7f36fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:10.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.677+0000 7f37027fc700 10 client.?.objecter ms_handle_connect 0x7f370c0f95e0 2024-02-13T00:49:10.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.677+0000 7f37027fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:10.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.677+0000 7f37027fc700 1 -- 172.21.15.162:0/258817619 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc00e070 con 0x7f370c0f95e0 2024-02-13T00:49:10.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.677+0000 7f371398f700 1 -- 172.21.15.162:0/258817619 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f370c108e00 con 0x7f370c0f95e0 2024-02-13T00:49:10.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.677+0000 7f37027fc700 1 -- 172.21.15.162:0/258817619 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36fc01f070 con 0x7f370c0f95e0 2024-02-13T00:49:10.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.677+0000 7f37027fc700 1 -- 172.21.15.162:0/258817619 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc01b9a0 con 0x7f370c0f95e0 2024-02-13T00:49:10.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.677+0000 7f371398f700 10 client.27627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:10.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.677+0000 7f371398f700 1 -- 172.21.15.162:0/258817619 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370c0f6b00 con 0x7f370c0f95e0 2024-02-13T00:49:10.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.677+0000 7f371398f700 1 librados: init done 2024-02-13T00:49:10.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.678+0000 7f371398f700 1 -- 172.21.15.162:0/258817619 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f0004f50 con 0x7f370c0f95e0 2024-02-13T00:49:10.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.678+0000 7f37027fc700 1 -- 172.21.15.162:0/258817619 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f36fc0090d0 con 0x7f370c0f95e0 2024-02-13T00:49:10.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.679+0000 7f37027fc700 1 --2- 172.21.15.162:0/258817619 >> v2:172.21.15.162:6800/2866534053 conn(0x7f36f8062c90 0x7f36f8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:10.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.679+0000 7f37027fc700 1 -- 172.21.15.162:0/258817619 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(536..536 src has 449..536) v4 ==== 8088+0+0 (secure 0 0 0) 0x7f36fc081df0 con 0x7f370c0f95e0 2024-02-13T00:49:10.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.679+0000 7f37027fc700 10 client.27627.objecter ms_dispatch 0x7f370c001550 osd_map(536..536 src has 449..536) v4 2024-02-13T00:49:10.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.679+0000 7f37027fc700 3 client.27627.objecter handle_osd_map got epochs [536,536] > 0 2024-02-13T00:49:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.679+0000 7f37027fc700 3 client.27627.objecter handle_osd_map decoding full epoch 536 2024-02-13T00:49:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.679+0000 7f371172b700 1 --2- 172.21.15.162:0/258817619 >> v2:172.21.15.162:6800/2866534053 conn(0x7f36f8062c90 0x7f36f8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.679+0000 7f37027fc700 20 client.27627.objecter dump_active .. 0 homeless 2024-02-13T00:49:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.679+0000 7f371172b700 1 --2- 172.21.15.162:0/258817619 >> v2:172.21.15.162:6800/2866534053 conn(0x7f36f8062c90 0x7f36f8065150 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 rx=0x7f3708004e60 tx=0x7f370800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.679+0000 7f37027fc700 10 client.27627.objecter ms_handle_connect 0x7f36f8062c90 2024-02-13T00:49:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.683+0000 7f37027fc700 1 -- 172.21.15.162:0/258817619 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f36fc055430 con 0x7f370c0f95e0 2024-02-13T00:49:10.771 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:10.802 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:10.822 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:10.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:10.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:10.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:10.864+0000 7f371398f700 1 -- 172.21.15.162:0/258817619 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f36f00054d0 con 0x7f370c0f95e0 2024-02-13T00:49:10.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:49:10.958 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:11.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.150+0000 7fab99b4b700 1 -- 172.21.15.162:0/2776641309 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab940fd4b0 msgr2=0x7fab940f6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.150+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2776641309 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab940fd4b0 0x7fab940f6ce0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7fab880047b0 tx=0x7fab88009cb0).stop 2024-02-13T00:49:11.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.150+0000 7fab99b4b700 1 -- 172.21.15.162:0/2776641309 shutdown_connections 2024-02-13T00:49:11.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.150+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2776641309 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab940f73c0 0x7fab941039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.150+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2776641309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab940fcb50 0x7fab940fcf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.150+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2776641309 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab940fd4b0 0x7fab940f6ce0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.150+0000 7fab99b4b700 1 -- 172.21.15.162:0/2776641309 >> 172.21.15.162:0/2776641309 conn(0x7fab940f2780 msgr2=0x7fab940f4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:11.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.150+0000 7fab99b4b700 1 -- 172.21.15.162:0/2776641309 shutdown_connections 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.150+0000 7fab99b4b700 1 -- 172.21.15.162:0/2776641309 wait complete. 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab99b4b700 1 librados: starting msgr at 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab99b4b700 1 librados: starting objecter 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab99b4b700 1 Processor -- start 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab99b4b700 1 -- start start 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab99b4b700 1 librados: setting wanted keys 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab99b4b700 1 librados: calling monclient init 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab99b4b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab940f73c0 0x7fab9410e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab99b4b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab940fcb50 0x7fab9410e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab99b4b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab940fd4b0 0x7fab94108330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab99b4b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab94108820 con 0x7fab940fcb50 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab99b4b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab94108990 con 0x7fab940f73c0 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab99b4b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab94108b50 con 0x7fab940fd4b0 2024-02-13T00:49:11.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab92ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab940fcb50 0x7fab9410e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:11.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab92ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab940fcb50 0x7fab9410e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:42830/0 (socket says 172.21.15.162:42830) 2024-02-13T00:49:11.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab92ffd700 1 -- 172.21.15.162:0/2852952005 learned_addr learned my addr 172.21.15.162:0/2852952005 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:11.158 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.151+0000 7fab93fff700 1 --2- 172.21.15.162:0/2852952005 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab940fd4b0 0x7fab94108330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab937fe700 1 --2- 172.21.15.162:0/2852952005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab940f73c0 0x7fab9410e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab92ffd700 1 -- 172.21.15.162:0/2852952005 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab940fd4b0 msgr2=0x7fab94108330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab92ffd700 1 --2- 172.21.15.162:0/2852952005 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab940fd4b0 0x7fab94108330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab92ffd700 1 -- 172.21.15.162:0/2852952005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab940f73c0 msgr2=0x7fab9410e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab92ffd700 1 --2- 172.21.15.162:0/2852952005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab940f73c0 0x7fab9410e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab92ffd700 1 -- 172.21.15.162:0/2852952005 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab84005040 con 0x7fab940fcb50 2024-02-13T00:49:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab937fe700 1 --2- 172.21.15.162:0/2852952005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab940f73c0 0x7fab9410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:49:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab92ffd700 1 --2- 172.21.15.162:0/2852952005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab940fcb50 0x7fab9410e680 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 rx=0x7fab880044e0 tx=0x7fab88007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab90ff9700 10 client.?.objecter ms_handle_connect 0x7fab940fcb50 2024-02-13T00:49:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab90ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab88018070 con 0x7fab940fcb50 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab88005040 con 0x7fab940fcb50 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab880093b0 con 0x7fab940fcb50 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8801dd30 con 0x7fab940fcb50 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.152+0000 7fab99b4b700 10 client.27633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.153+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab94109070 con 0x7fab940fcb50 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.153+0000 7fab99b4b700 1 librados: init done 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.154+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fab8801c5d0 con 0x7fab940fcb50 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.154+0000 7fab90ff9700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab7c062c90 0x7fab7c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.154+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fab94060790 con 0x7fab940fcb50 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.154+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(536..536 src has 449..536) v4 ==== 8088+0+0 (secure 0 0 0) 0x7fab880801e0 con 0x7fab940fcb50 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.154+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 osd_map(536..536 src has 449..536) v4 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.154+0000 7fab90ff9700 3 client.27633.objecter handle_osd_map got epochs [536,536] > 0 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.154+0000 7fab90ff9700 3 client.27633.objecter handle_osd_map decoding full epoch 536 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.154+0000 7fab937fe700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab7c062c90 0x7fab7c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.154+0000 7fab90ff9700 20 client.27633.objecter dump_active .. 0 homeless 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.154+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v536) v1 ==== 54+0+16 (secure 0 0 0) 0x7fab88051180 con 0x7fab940fcb50 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab90ff9700 10 client.27633.objecter ms_handle_connect 0x7fab7c062c90 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab937fe700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab7c062c90 0x7fab7c065150 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 rx=0x7fab84009f20 tx=0x7fab84009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab99b4b700 10 client.27633.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 1 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6801/2103743338 conn(0x7fab940ffdd0 0x7fab941001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab940475e0 osd=0 3 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab940475e0 osd=0 3 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab940475e0 osd=0 3 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 1 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab940475e0 osd=0 3 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 0 1 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab940475e0 osd=0 4 2024-02-13T00:49:11.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab99b4b700 10 client.27633.objecter _send_command 1 2024-02-13T00:49:11.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab940fa7a0 con 0x7fab940ffdd0 2024-02-13T00:49:11.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab93fff700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6801/2103743338 conn(0x7fab940ffdd0 0x7fab941001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:11.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab90ff9700 10 client.27633.objecter ms_handle_connect 0x7fab940ffdd0 2024-02-13T00:49:11.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.155+0000 7fab93fff700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6801/2103743338 conn(0x7fab940ffdd0 0x7fab941001b0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:11.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:49:11.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.156+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab940fa7a0 con 0x7fab940ffdd0 2024-02-13T00:49:11.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.156+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:49:11.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.156+0000 7fab90ff9700 10 client.27633.objecter _finish_command 1 = system:0 2024-02-13T00:49:11.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.156+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab940475e0 osd=0 4 2024-02-13T00:49:11.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.156+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 0 1 2024-02-13T00:49:11.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.181+0000 7fab99b4b700 10 client.27633.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:11.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.181+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 2 2024-02-13T00:49:11.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.181+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab940475e0 osd=0 3 2024-02-13T00:49:11.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.181+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab940475e0 osd=0 3 2024-02-13T00:49:11.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.181+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab940475e0 osd=0 3 2024-02-13T00:49:11.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.181+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 2 2024-02-13T00:49:11.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.181+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab940475e0 osd=0 3 2024-02-13T00:49:11.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.181+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 0 2 2024-02-13T00:49:11.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.181+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab940475e0 osd=0 4 2024-02-13T00:49:11.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.181+0000 7fab99b4b700 10 client.27633.objecter _send_command 2 2024-02-13T00:49:11.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.181+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab941060c0 con 0x7fab940ffdd0 2024-02-13T00:49:11.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab941060c0 con 0x7fab940ffdd0 2024-02-13T00:49:11.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:11.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab90ff9700 10 client.27633.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab940475e0 osd=0 4 2024-02-13T00:49:11.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 0 2 2024-02-13T00:49:11.188 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:49:11.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab99b4b700 10 client.27633.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:11.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 3 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6805/150599780 conn(0x7fab94059990 0x7fab94059db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94028590 osd=1 3 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94028590 osd=1 3 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94028590 osd=1 3 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 3 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab94028590 osd=1 3 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 1 3 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94028590 osd=1 4 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab99b4b700 10 client.27633.objecter _send_command 3 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab94028700 con 0x7fab94059990 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab937fe700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6805/150599780 conn(0x7fab94059990 0x7fab94059db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab90ff9700 10 client.27633.objecter ms_handle_connect 0x7fab94059990 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.183+0000 7fab937fe700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6805/150599780 conn(0x7fab94059990 0x7fab94059db0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.184+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab94028700 con 0x7fab94059990 2024-02-13T00:49:11.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.184+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:49:11.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.184+0000 7fab90ff9700 10 client.27633.objecter _finish_command 3 = system:0 2024-02-13T00:49:11.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.184+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab94028590 osd=1 4 2024-02-13T00:49:11.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.184+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 1 3 2024-02-13T00:49:11.197 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:11.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.200+0000 7fab99b4b700 10 client.27633.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:11.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.200+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 4 2024-02-13T00:49:11.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.200+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94028590 osd=1 3 2024-02-13T00:49:11.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.200+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94028590 osd=1 3 2024-02-13T00:49:11.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.200+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94028590 osd=1 3 2024-02-13T00:49:11.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.200+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 4 2024-02-13T00:49:11.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.200+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab94028590 osd=1 3 2024-02-13T00:49:11.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.200+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 1 4 2024-02-13T00:49:11.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.200+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94028590 osd=1 4 2024-02-13T00:49:11.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.200+0000 7fab99b4b700 10 client.27633.objecter _send_command 4 2024-02-13T00:49:11.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.200+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab94059400 con 0x7fab94059990 2024-02-13T00:49:11.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab94059400 con 0x7fab94059990 2024-02-13T00:49:11.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:11.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab90ff9700 10 client.27633.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab94028590 osd=1 4 2024-02-13T00:49:11.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 1 4 2024-02-13T00:49:11.207 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab99b4b700 10 client.27633.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 5 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6809/3689719376 conn(0x7fab9411c560 0x7fab9411e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94027be0 osd=2 3 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94027be0 osd=2 3 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94027be0 osd=2 3 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 5 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab94027be0 osd=2 3 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 2 5 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94027be0 osd=2 4 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab99b4b700 10 client.27633.objecter _send_command 5 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab94028260 con 0x7fab9411c560 2024-02-13T00:49:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.202+0000 7fab92ffd700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6809/3689719376 conn(0x7fab9411c560 0x7fab9411e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:11.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.203+0000 7fab90ff9700 10 client.27633.objecter ms_handle_connect 0x7fab9411c560 2024-02-13T00:49:11.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.203+0000 7fab92ffd700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6809/3689719376 conn(0x7fab9411c560 0x7fab9411e950 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:11.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.203+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab94028260 con 0x7fab9411c560 2024-02-13T00:49:11.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.203+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:49:11.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.203+0000 7fab90ff9700 10 client.27633.objecter _finish_command 5 = system:0 2024-02-13T00:49:11.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.203+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab94027be0 osd=2 4 2024-02-13T00:49:11.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.203+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 2 5 2024-02-13T00:49:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.219+0000 7fab99b4b700 10 client.27633.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.219+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 6 2024-02-13T00:49:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.219+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94027be0 osd=2 3 2024-02-13T00:49:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.219+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94027be0 osd=2 3 2024-02-13T00:49:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.219+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94027be0 osd=2 3 2024-02-13T00:49:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.219+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 6 2024-02-13T00:49:11.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.219+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab94027be0 osd=2 3 2024-02-13T00:49:11.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.219+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 2 6 2024-02-13T00:49:11.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.219+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94027be0 osd=2 4 2024-02-13T00:49:11.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.219+0000 7fab99b4b700 10 client.27633.objecter _send_command 6 2024-02-13T00:49:11.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.219+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab94027650 con 0x7fab9411c560 2024-02-13T00:49:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.220+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab94027650 con 0x7fab9411c560 2024-02-13T00:49:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.220+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.220+0000 7fab90ff9700 10 client.27633.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.220+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab94027be0 osd=2 4 2024-02-13T00:49:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.220+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 2 6 2024-02-13T00:49:11.225 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:49:11.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab99b4b700 10 client.27633.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:11.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 7 2024-02-13T00:49:11.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6813/604971011 conn(0x7fab9411c0c0 0x7fab94121fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:11.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94027a60 osd=3 3 2024-02-13T00:49:11.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94027a60 osd=3 3 2024-02-13T00:49:11.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94027a60 osd=3 3 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 7 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab94027a60 osd=3 3 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 3 7 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94027a60 osd=3 4 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab99b4b700 10 client.27633.objecter _send_command 7 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab94124c20 con 0x7fab9411c0c0 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab93fff700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6813/604971011 conn(0x7fab9411c0c0 0x7fab94121fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab90ff9700 10 client.27633.objecter ms_handle_connect 0x7fab9411c0c0 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.221+0000 7fab93fff700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6813/604971011 conn(0x7fab9411c0c0 0x7fab94121fd0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.222+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab94124c20 con 0x7fab9411c0c0 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.222+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.222+0000 7fab90ff9700 10 client.27633.objecter _finish_command 7 = system:0 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.222+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab94027a60 osd=3 4 2024-02-13T00:49:11.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.222+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 3 7 2024-02-13T00:49:11.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.236+0000 7fab99b4b700 10 client.27633.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:11.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.236+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 8 2024-02-13T00:49:11.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.236+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94027a60 osd=3 3 2024-02-13T00:49:11.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.236+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94027a60 osd=3 3 2024-02-13T00:49:11.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.236+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94027a60 osd=3 3 2024-02-13T00:49:11.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.236+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 8 2024-02-13T00:49:11.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.236+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab94027a60 osd=3 3 2024-02-13T00:49:11.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.236+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 3 8 2024-02-13T00:49:11.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.236+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94027a60 osd=3 4 2024-02-13T00:49:11.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.236+0000 7fab99b4b700 10 client.27633.objecter _send_command 8 2024-02-13T00:49:11.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.236+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab9411b620 con 0x7fab9411c0c0 2024-02-13T00:49:11.242 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:49:11.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab9411b620 con 0x7fab9411c0c0 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab90ff9700 10 client.27633.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab94027a60 osd=3 4 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 3 8 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab99b4b700 10 client.27633.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 9 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6800/3344263388 conn(0x7fab9411b790 0x7fab94123640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab9411b620 osd=4 3 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9411b620 osd=4 3 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab9411b620 osd=4 3 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 9 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab9411b620 osd=4 3 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 4 9 2024-02-13T00:49:11.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9411b620 osd=4 4 2024-02-13T00:49:11.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab99b4b700 10 client.27633.objecter _send_command 9 2024-02-13T00:49:11.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.237+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab9411fc20 con 0x7fab9411b790 2024-02-13T00:49:11.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.238+0000 7fab937fe700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6800/3344263388 conn(0x7fab9411b790 0x7fab94123640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:11.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.238+0000 7fab90ff9700 10 client.27633.objecter ms_handle_connect 0x7fab9411b790 2024-02-13T00:49:11.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.238+0000 7fab937fe700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6800/3344263388 conn(0x7fab9411b790 0x7fab94123640 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:11.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.239+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab9411fc20 con 0x7fab9411b790 2024-02-13T00:49:11.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.239+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:49:11.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.239+0000 7fab90ff9700 10 client.27633.objecter _finish_command 9 = system:0 2024-02-13T00:49:11.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.239+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab9411b620 osd=4 4 2024-02-13T00:49:11.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.239+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 4 9 2024-02-13T00:49:11.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.254+0000 7fab99b4b700 10 client.27633.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:11.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.254+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 10 2024-02-13T00:49:11.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.254+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab9411b620 osd=4 3 2024-02-13T00:49:11.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.254+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9411b620 osd=4 3 2024-02-13T00:49:11.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.254+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab9411b620 osd=4 3 2024-02-13T00:49:11.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.254+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 10 2024-02-13T00:49:11.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.254+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab9411b620 osd=4 3 2024-02-13T00:49:11.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.254+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 4 10 2024-02-13T00:49:11.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.254+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9411b620 osd=4 4 2024-02-13T00:49:11.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.254+0000 7fab99b4b700 10 client.27633.objecter _send_command 10 2024-02-13T00:49:11.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.254+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab9411fc20 con 0x7fab9411b790 2024-02-13T00:49:11.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab9411fc20 con 0x7fab9411b790 2024-02-13T00:49:11.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:11.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab90ff9700 10 client.27633.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab9411b620 osd=4 4 2024-02-13T00:49:11.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 4 10 2024-02-13T00:49:11.261 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:49:11.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab99b4b700 10 client.27633.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:11.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 11 2024-02-13T00:49:11.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6804/2277917372 conn(0x7fab94124040 0x7fab941244a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:11.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab9411fc20 osd=5 3 2024-02-13T00:49:11.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9411fc20 osd=5 3 2024-02-13T00:49:11.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab9411fc20 osd=5 3 2024-02-13T00:49:11.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 11 2024-02-13T00:49:11.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab9411fc20 osd=5 3 2024-02-13T00:49:11.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 5 11 2024-02-13T00:49:11.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9411fc20 osd=5 4 2024-02-13T00:49:11.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab99b4b700 10 client.27633.objecter _send_command 11 2024-02-13T00:49:11.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab94124a70 con 0x7fab94124040 2024-02-13T00:49:11.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.256+0000 7fab92ffd700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6804/2277917372 conn(0x7fab94124040 0x7fab941244a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:11.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.257+0000 7fab92ffd700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6804/2277917372 conn(0x7fab94124040 0x7fab941244a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:11.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.257+0000 7fab90ff9700 10 client.27633.objecter ms_handle_connect 0x7fab94124040 2024-02-13T00:49:11.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.258+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab94124a70 con 0x7fab94124040 2024-02-13T00:49:11.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.258+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:49:11.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.258+0000 7fab90ff9700 10 client.27633.objecter _finish_command 11 = system:0 2024-02-13T00:49:11.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.258+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab9411fc20 osd=5 4 2024-02-13T00:49:11.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.258+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 5 11 2024-02-13T00:49:11.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:11.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.272+0000 7fab99b4b700 10 client.27633.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:11.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.272+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 12 2024-02-13T00:49:11.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.272+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab9411fc20 osd=5 3 2024-02-13T00:49:11.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.272+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9411fc20 osd=5 3 2024-02-13T00:49:11.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.272+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab9411fc20 osd=5 3 2024-02-13T00:49:11.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.272+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 12 2024-02-13T00:49:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.272+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab9411fc20 osd=5 3 2024-02-13T00:49:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.272+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 5 12 2024-02-13T00:49:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.272+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9411fc20 osd=5 4 2024-02-13T00:49:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.272+0000 7fab99b4b700 10 client.27633.objecter _send_command 12 2024-02-13T00:49:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.272+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab94124a70 con 0x7fab94124040 2024-02-13T00:49:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab94124a70 con 0x7fab94124040 2024-02-13T00:49:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab90ff9700 10 client.27633.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab9411fc20 osd=5 4 2024-02-13T00:49:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 5 12 2024-02-13T00:49:11.279 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:49:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab99b4b700 10 client.27633.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 13 2024-02-13T00:49:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6808/1506640341 conn(0x7fab941292c0 0x7fab94129720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94124a70 osd=6 3 2024-02-13T00:49:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94124a70 osd=6 3 2024-02-13T00:49:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94124a70 osd=6 3 2024-02-13T00:49:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 13 2024-02-13T00:49:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab94124a70 osd=6 3 2024-02-13T00:49:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 6 13 2024-02-13T00:49:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94124a70 osd=6 4 2024-02-13T00:49:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab99b4b700 10 client.27633.objecter _send_command 13 2024-02-13T00:49:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab9412b380 con 0x7fab941292c0 2024-02-13T00:49:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.274+0000 7fab93fff700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6808/1506640341 conn(0x7fab941292c0 0x7fab94129720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.275+0000 7fab90ff9700 10 client.27633.objecter ms_handle_connect 0x7fab941292c0 2024-02-13T00:49:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.275+0000 7fab93fff700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6808/1506640341 conn(0x7fab941292c0 0x7fab94129720 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.275+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab9412b380 con 0x7fab941292c0 2024-02-13T00:49:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.275+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:49:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.275+0000 7fab90ff9700 10 client.27633.objecter _finish_command 13 = system:0 2024-02-13T00:49:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.275+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab94124a70 osd=6 4 2024-02-13T00:49:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.275+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 6 13 2024-02-13T00:49:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.290+0000 7fab99b4b700 10 client.27633.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.290+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 14 2024-02-13T00:49:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.290+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94124a70 osd=6 3 2024-02-13T00:49:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.290+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94124a70 osd=6 3 2024-02-13T00:49:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.290+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab94124a70 osd=6 3 2024-02-13T00:49:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.290+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 14 2024-02-13T00:49:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.290+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab94124a70 osd=6 3 2024-02-13T00:49:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.290+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 6 14 2024-02-13T00:49:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.290+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94124a70 osd=6 4 2024-02-13T00:49:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.290+0000 7fab99b4b700 10 client.27633.objecter _send_command 14 2024-02-13T00:49:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.290+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab9412b380 con 0x7fab941292c0 2024-02-13T00:49:11.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab9412b380 con 0x7fab941292c0 2024-02-13T00:49:11.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:11.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab90ff9700 10 client.27633.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab94124a70 osd=6 4 2024-02-13T00:49:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 6 14 2024-02-13T00:49:11.297 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:49:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab99b4b700 10 client.27633.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 15 2024-02-13T00:49:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6812/1964192772 conn(0x7fab9412dab0 0x7fab9412def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab9412b380 osd=7 3 2024-02-13T00:49:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9412b380 osd=7 3 2024-02-13T00:49:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab9412b380 osd=7 3 2024-02-13T00:49:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 15 2024-02-13T00:49:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab9412b380 osd=7 3 2024-02-13T00:49:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 7 15 2024-02-13T00:49:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9412b380 osd=7 4 2024-02-13T00:49:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab99b4b700 10 client.27633.objecter _send_command 15 2024-02-13T00:49:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab9412e510 con 0x7fab9412dab0 2024-02-13T00:49:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.292+0000 7fab937fe700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6812/1964192772 conn(0x7fab9412dab0 0x7fab9412def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.293+0000 7fab90ff9700 10 client.27633.objecter ms_handle_connect 0x7fab9412dab0 2024-02-13T00:49:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.293+0000 7fab937fe700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6812/1964192772 conn(0x7fab9412dab0 0x7fab9412def0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:11.299 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.294+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fab9412e510 con 0x7fab9412dab0 2024-02-13T00:49:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.294+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:49:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.294+0000 7fab90ff9700 10 client.27633.objecter _finish_command 15 = system:0 2024-02-13T00:49:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.294+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab9412b380 osd=7 4 2024-02-13T00:49:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.294+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 7 15 2024-02-13T00:49:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.308+0000 7fab99b4b700 10 client.27633.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.308+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign -1 16 2024-02-13T00:49:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.308+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab9412b380 osd=7 3 2024-02-13T00:49:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.308+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9412b380 osd=7 3 2024-02-13T00:49:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.308+0000 7fab99b4b700 20 client.27633.objecter _get_session s=0x7fab9412b380 osd=7 3 2024-02-13T00:49:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.308+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_remove -1 16 2024-02-13T00:49:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.308+0000 7fab99b4b700 20 client.27633.objecter get_session s=0x7fab9412b380 osd=7 3 2024-02-13T00:49:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.308+0000 7fab99b4b700 15 client.27633.objecter _session_command_op_assign 7 16 2024-02-13T00:49:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.308+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9412b380 osd=7 4 2024-02-13T00:49:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.308+0000 7fab99b4b700 10 client.27633.objecter _send_command 16 2024-02-13T00:49:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.308+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fab9412e510 con 0x7fab9412dab0 2024-02-13T00:49:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab90ff9700 1 -- 172.21.15.162:0/2852952005 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fab9412e510 con 0x7fab9412dab0 2024-02-13T00:49:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab90ff9700 10 client.27633.objecter ms_dispatch 0x7fab94001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab90ff9700 10 client.27633.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab90ff9700 20 client.27633.objecter put_session s=0x7fab9412b380 osd=7 4 2024-02-13T00:49:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab90ff9700 15 client.27633.objecter _session_command_op_remove 7 16 2024-02-13T00:49:11.316 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:49:11.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 10 librados: watch_flush enter 2024-02-13T00:49:11.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 10 librados: watch_flush exit 2024-02-13T00:49:11.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 10 client.27633.objecter close_session for osd.0 2024-02-13T00:49:11.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6801/2103743338 conn(0x7fab940ffdd0 msgr2=0x7fab941001b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6801/2103743338 conn(0x7fab940ffdd0 0x7fab941001b0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab940475e0 osd=0 1 2024-02-13T00:49:11.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 10 client.27633.objecter close_session for osd.1 2024-02-13T00:49:11.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6805/150599780 conn(0x7fab94059990 msgr2=0x7fab94059db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6805/150599780 conn(0x7fab94059990 0x7fab94059db0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94028590 osd=1 1 2024-02-13T00:49:11.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 10 client.27633.objecter close_session for osd.2 2024-02-13T00:49:11.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6809/3689719376 conn(0x7fab9411c560 msgr2=0x7fab9411e950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6809/3689719376 conn(0x7fab9411c560 0x7fab9411e950 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94027be0 osd=2 1 2024-02-13T00:49:11.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 10 client.27633.objecter close_session for osd.3 2024-02-13T00:49:11.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6813/604971011 conn(0x7fab9411c0c0 msgr2=0x7fab94121fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6813/604971011 conn(0x7fab9411c0c0 0x7fab94121fd0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab93fff700 1 -- 172.21.15.162:0/2852952005 reap_dead start 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94027a60 osd=3 1 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 10 client.27633.objecter close_session for osd.4 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6800/3344263388 conn(0x7fab9411b790 msgr2=0x7fab94123640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6800/3344263388 conn(0x7fab9411b790 0x7fab94123640 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab93fff700 1 -- 172.21.15.162:0/2852952005 reap_dead start 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9411b620 osd=4 1 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 10 client.27633.objecter close_session for osd.5 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6804/2277917372 conn(0x7fab94124040 msgr2=0x7fab941244a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6804/2277917372 conn(0x7fab94124040 0x7fab941244a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9411fc20 osd=5 1 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 10 client.27633.objecter close_session for osd.6 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6808/1506640341 conn(0x7fab941292c0 msgr2=0x7fab94129720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6808/1506640341 conn(0x7fab941292c0 0x7fab94129720 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab94124a70 osd=6 1 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 10 client.27633.objecter close_session for osd.7 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6812/1964192772 conn(0x7fab9412dab0 msgr2=0x7fab9412def0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.181:6812/1964192772 conn(0x7fab9412dab0 0x7fab9412def0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 20 client.27633.objecter put_session s=0x7fab9412b380 osd=7 1 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 20 client.27633.objecter shutdown clearing up homeless session... 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 10 client.27633.objecter successfully canceled tick 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab7c062c90 msgr2=0x7fab7c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab7c062c90 0x7fab7c065150 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 rx=0x7fab84009f20 tx=0x7fab84009cb0).stop 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab940fcb50 msgr2=0x7fab9410e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab99b4b700 1 --2- 172.21.15.162:0/2852952005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab940fcb50 0x7fab9410e680 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 rx=0x7fab880044e0 tx=0x7fab88007f90).stop 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.311+0000 7fab93fff700 1 -- 172.21.15.162:0/2852952005 reap_dead start 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.312+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 shutdown_connections 2024-02-13T00:49:11.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.312+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 >> 172.21.15.162:0/2852952005 conn(0x7fab940f2780 msgr2=0x7fab940f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:11.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.312+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 shutdown_connections 2024-02-13T00:49:11.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.312+0000 7fab99b4b700 1 -- 172.21.15.162:0/2852952005 wait complete. 2024-02-13T00:49:11.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.312+0000 7fab99b4b700 1 librados: shutdown 2024-02-13T00:49:11.321 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.321 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.321 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.321 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.321 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.321 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.321 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.321 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:49:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:11.322 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:11.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:11.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:11.453 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:11.475 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:11.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:11.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.493+0000 7f37027fc700 1 -- 172.21.15.162:0/258817619 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v537) v1 ==== 113+0+0 (secure 0 0 0) 0x7f36fc029030 con 0x7f370c0f95e0 2024-02-13T00:49:11.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 10 librados: watch_flush enter 2024-02-13T00:49:11.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 10 librados: watch_flush exit 2024-02-13T00:49:11.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 20 client.27627.objecter shutdown clearing up homeless session... 2024-02-13T00:49:11.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 10 client.27627.objecter successfully canceled tick 2024-02-13T00:49:11.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 1 -- 172.21.15.162:0/258817619 >> v2:172.21.15.162:6800/2866534053 conn(0x7f36f8062c90 msgr2=0x7f36f8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 1 --2- 172.21.15.162:0/258817619 >> v2:172.21.15.162:6800/2866534053 conn(0x7f36f8062c90 0x7f36f8065150 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 rx=0x7f3708004e60 tx=0x7f370800b040).stop 2024-02-13T00:49:11.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 1 -- 172.21.15.162:0/258817619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f370c0f95e0 msgr2=0x7f370c10e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:11.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 1 --2- 172.21.15.162:0/258817619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f370c0f95e0 0x7f370c10e6c0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f36fc013760 tx=0x7f36fc00b040).stop 2024-02-13T00:49:11.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 1 -- 172.21.15.162:0/258817619 shutdown_connections 2024-02-13T00:49:11.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 1 --2- 172.21.15.162:0/258817619 >> v2:172.21.15.162:6800/2866534053 conn(0x7f36f8062c90 0x7f36f8065150 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 1 --2- 172.21.15.162:0/258817619 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f370c102270 0x7f370c1083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 1 --2- 172.21.15.162:0/258817619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f370c0f8d10 0x7f370c10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 1 --2- 172.21.15.162:0/258817619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f370c0f95e0 0x7f370c10e6c0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:11.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 1 -- 172.21.15.162:0/258817619 >> 172.21.15.162:0/258817619 conn(0x7f370c06eb90 msgr2=0x7f370c102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:11.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.496+0000 7f371398f700 1 -- 172.21.15.162:0/258817619 shutdown_connections 2024-02-13T00:49:11.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.498+0000 7f371398f700 1 -- 172.21.15.162:0/258817619 wait complete. 2024-02-13T00:49:11.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:11.500+0000 7f371398f700 1 librados: shutdown 2024-02-13T00:49:11.505 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.0 to 1 (10000) 2024-02-13T00:49:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:10.470641+0000 mgr.y (mgr.14544) 471 : cluster [DBG] pgmap v803: 231 pgs: 2 activating, 229 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:49:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:10.489989+0000 mon.a (mon.0) 2872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_count","val": "1"}]': finished 2024-02-13T00:49:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:10.490098+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e536: 8 total, 8 up, 4 in 2024-02-13T00:49:11.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:10.500008+0000 mon.b (mon.1) 658 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:11.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:10.500461+0000 mon.a (mon.0) 2874 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:11.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:10.866203+0000 mon.a (mon.0) 2875 : audit [INF] from='client.? 172.21.15.162:0/258817619' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:49:11.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:11.156534+0000 mon.a (mon.0) 2876 : audit [DBG] from='client.? 172.21.15.162:0/2852952005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:11.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:11.606 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:11.625 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:11.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:11.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:11.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:10.470641+0000 mgr.y (mgr.14544) 471 : cluster [DBG] pgmap v803: 231 pgs: 2 activating, 229 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:49:11.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:10.489989+0000 mon.a (mon.0) 2872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_count","val": "1"}]': finished 2024-02-13T00:49:11.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:10.490098+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e536: 8 total, 8 up, 4 in 2024-02-13T00:49:11.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:10.500008+0000 mon.b (mon.1) 658 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:11.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:10.500461+0000 mon.a (mon.0) 2874 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:11.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:10.866203+0000 mon.a (mon.0) 2875 : audit [INF] from='client.? 172.21.15.162:0/258817619' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:49:11.757 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:11.156534+0000 mon.a (mon.0) 2876 : audit [DBG] from='client.? 172.21.15.162:0/2852952005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:11.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:10.470641+0000 mgr.y (mgr.14544) 471 : cluster [DBG] pgmap v803: 231 pgs: 2 activating, 229 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:49:11.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:10.489989+0000 mon.a (mon.0) 2872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_count","val": "1"}]': finished 2024-02-13T00:49:11.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:10.490098+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e536: 8 total, 8 up, 4 in 2024-02-13T00:49:11.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:10.500008+0000 mon.b (mon.1) 658 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:11.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:10.500461+0000 mon.a (mon.0) 2874 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:11.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:10.866203+0000 mon.a (mon.0) 2875 : audit [INF] from='client.? 172.21.15.162:0/258817619' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:49:11.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:11.156534+0000 mon.a (mon.0) 2876 : audit [DBG] from='client.? 172.21.15.162:0/2852952005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:11.758 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:11.832 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:11.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:11.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:11.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:11.971 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:11.990 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:11.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:12.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:12.121 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:12.140 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:12.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:12.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:12.272 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:12.291 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:12.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:12.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:12.423 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:11.494436+0000 mon.a (mon.0) 2877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:49:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:11.494508+0000 mon.a (mon.0) 2878 : audit [INF] from='client.? 172.21.15.162:0/258817619' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:49:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:11.494565+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e537: 8 total, 8 up, 4 in 2024-02-13T00:49:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:11.495181+0000 mon.b (mon.1) 659 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-13T00:49:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:11.495492+0000 mon.a (mon.0) 2880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-13T00:49:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:12.471449+0000 mon.c (mon.2) 432 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-13T00:49:12.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:12.472338+0000 mon.a (mon.0) 2881 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-13T00:49:12.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:11.494436+0000 mon.a (mon.0) 2877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:49:12.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:11.494508+0000 mon.a (mon.0) 2878 : audit [INF] from='client.? 172.21.15.162:0/258817619' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:49:12.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:11.494565+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e537: 8 total, 8 up, 4 in 2024-02-13T00:49:12.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:11.495181+0000 mon.b (mon.1) 659 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-13T00:49:12.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:11.495492+0000 mon.a (mon.0) 2880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-13T00:49:12.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:12.471449+0000 mon.c (mon.2) 432 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-13T00:49:12.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:12.472338+0000 mon.a (mon.0) 2881 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-13T00:49:12.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:11.494436+0000 mon.a (mon.0) 2877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:49:12.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:11.494508+0000 mon.a (mon.0) 2878 : audit [INF] from='client.? 172.21.15.162:0/258817619' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:49:12.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:11.494565+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e537: 8 total, 8 up, 4 in 2024-02-13T00:49:12.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:11.495181+0000 mon.b (mon.1) 659 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-13T00:49:12.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:11.495492+0000 mon.a (mon.0) 2880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-13T00:49:12.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:12.471449+0000 mon.c (mon.2) 432 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-13T00:49:12.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:12.472338+0000 mon.a (mon.0) 2881 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-13T00:49:12.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 1 1 2024-02-13T00:49:13.072 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:13.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:13.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:13.202 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:13.450 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:13.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:13.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:13.585 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:13.605 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:13.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:13.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:13.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:12.471124+0000 mgr.y (mgr.14544) 472 : cluster [DBG] pgmap v806: 230 pgs: 1 activating, 229 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:49:13.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:12.502626+0000 mon.a (mon.0) 2882 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "target_max_objects","val": "250"}]': finished 2024-02-13T00:49:13.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:12.502710+0000 mon.a (mon.0) 2883 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-13T00:49:13.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:12.502760+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e538: 8 total, 8 up, 4 in 2024-02-13T00:49:13.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:12.525245+0000 mon.b (mon.1) 660 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:13.734 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:12.525655+0000 mon.a (mon.0) 2885 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:13.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:12.471124+0000 mgr.y (mgr.14544) 472 : cluster [DBG] pgmap v806: 230 pgs: 1 activating, 229 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:49:13.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:12.502626+0000 mon.a (mon.0) 2882 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "target_max_objects","val": "250"}]': finished 2024-02-13T00:49:13.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:12.502710+0000 mon.a (mon.0) 2883 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-13T00:49:13.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:12.502760+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e538: 8 total, 8 up, 4 in 2024-02-13T00:49:13.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:12.525245+0000 mon.b (mon.1) 660 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:13.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:12.525655+0000 mon.a (mon.0) 2885 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:13.736 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:12.471124+0000 mgr.y (mgr.14544) 472 : cluster [DBG] pgmap v806: 230 pgs: 1 activating, 229 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:49:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:12.502626+0000 mon.a (mon.0) 2882 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-88","var": "target_max_objects","val": "250"}]': finished 2024-02-13T00:49:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:12.502710+0000 mon.a (mon.0) 2883 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-13T00:49:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:12.502760+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e538: 8 total, 8 up, 4 in 2024-02-13T00:49:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:12.525245+0000 mon.b (mon.1) 660 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:12.525655+0000 mon.a (mon.0) 2885 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:14.187 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:14.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:14.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:14.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:49:14.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:14.323 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:14.343 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:14.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:14.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:14.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:49:14.474 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:14.722 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:14.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:13.502514+0000 mon.a (mon.0) 2886 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:49:14.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:13.502574+0000 mon.a (mon.0) 2887 : cluster [DBG] osdmap e539: 8 total, 8 up, 4 in 2024-02-13T00:49:14.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:13.505183+0000 mon.b (mon.1) 661 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88"}]: dispatch 2024-02-13T00:49:14.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:13.505587+0000 mon.a (mon.0) 2888 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88"}]: dispatch 2024-02-13T00:49:14.723 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:14.504164+0000 mon.a (mon.0) 2889 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88"}]': finished 2024-02-13T00:49:14.724 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:14.504228+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e540: 8 total, 8 up, 4 in 2024-02-13T00:49:14.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:13.502514+0000 mon.a (mon.0) 2886 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:49:14.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:13.502574+0000 mon.a (mon.0) 2887 : cluster [DBG] osdmap e539: 8 total, 8 up, 4 in 2024-02-13T00:49:14.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:13.505183+0000 mon.b (mon.1) 661 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88"}]: dispatch 2024-02-13T00:49:14.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:13.505587+0000 mon.a (mon.0) 2888 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88"}]: dispatch 2024-02-13T00:49:14.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:14.504164+0000 mon.a (mon.0) 2889 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88"}]': finished 2024-02-13T00:49:14.724 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:14.504228+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e540: 8 total, 8 up, 4 in 2024-02-13T00:49:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:14.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:49:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:13.502514+0000 mon.a (mon.0) 2886 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:49:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:13.502574+0000 mon.a (mon.0) 2887 : cluster [DBG] osdmap e539: 8 total, 8 up, 4 in 2024-02-13T00:49:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:13.505183+0000 mon.b (mon.1) 661 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88"}]: dispatch 2024-02-13T00:49:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:13.505587+0000 mon.a (mon.0) 2888 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88"}]: dispatch 2024-02-13T00:49:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:14.504164+0000 mon.a (mon.0) 2889 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-88"}]': finished 2024-02-13T00:49:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:14.504228+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e540: 8 total, 8 up, 4 in 2024-02-13T00:49:14.826 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:14.858 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:14.879 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:14.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:14.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:15.011 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:15.133 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:15.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:15.237 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:15.270 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:15.291 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:15.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:15.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:15.395 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:15.425 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:15.447 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:15.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:15.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:15.580 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:15.728 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:15.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:15.768 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:49:15.768 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:49:15.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:15.861 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:15.881 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:15.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:15.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:15.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:49:16.010 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:14.471555+0000 mgr.y (mgr.14544) 473 : cluster [DBG] pgmap v809: 230 pgs: 1 activating, 229 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:16.010 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:15.483991+0000 mon.a (mon.0) 2891 : cluster [DBG] osdmap e541: 8 total, 8 up, 4 in 2024-02-13T00:49:16.011 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:14.471555+0000 mgr.y (mgr.14544) 473 : cluster [DBG] pgmap v809: 230 pgs: 1 activating, 229 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:16.011 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:15.483991+0000 mon.a (mon.0) 2891 : cluster [DBG] osdmap e541: 8 total, 8 up, 4 in 2024-02-13T00:49:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.009+0000 7f4013f8d700 1 -- 172.21.15.162:0/1373755820 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f400c0fdd70 msgr2=0x7f400c0fe150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.009+0000 7f4013f8d700 1 --2- 172.21.15.162:0/1373755820 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f400c0fdd70 0x7f400c0fe150 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 rx=0x7f4008004870 tx=0x7f4008009cb0).stop 2024-02-13T00:49:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.010+0000 7f4013f8d700 1 -- 172.21.15.162:0/1373755820 shutdown_connections 2024-02-13T00:49:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.010+0000 7f4013f8d700 1 --2- 172.21.15.162:0/1373755820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f400c0f6b70 0x7f400c104bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.010+0000 7f4013f8d700 1 --2- 172.21.15.162:0/1373755820 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f400c0fdd70 0x7f400c0fe150 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.010+0000 7f4013f8d700 1 --2- 172.21.15.162:0/1373755820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f400c0fe6d0 0x7f400c0f64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:16.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.010+0000 7f4013f8d700 1 -- 172.21.15.162:0/1373755820 >> 172.21.15.162:0/1373755820 conn(0x7f400c06c410 msgr2=0x7f400c06c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.010+0000 7f4013f8d700 1 -- 172.21.15.162:0/1373755820 shutdown_connections 2024-02-13T00:49:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.010+0000 7f4013f8d700 1 -- 172.21.15.162:0/1373755820 wait complete. 2024-02-13T00:49:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.010+0000 7f4013f8d700 1 librados: starting msgr at 2024-02-13T00:49:16.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.010+0000 7f4013f8d700 1 librados: starting objecter 2024-02-13T00:49:16.016 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.010+0000 7f4013f8d700 1 Processor -- start 2024-02-13T00:49:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4013f8d700 1 -- start start 2024-02-13T00:49:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4013f8d700 1 librados: setting wanted keys 2024-02-13T00:49:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4013f8d700 1 librados: calling monclient init 2024-02-13T00:49:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4013f8d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f400c0f6b70 0x7f400c0f7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4013f8d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f400c0fdd70 0x7f400c0f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4013f8d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f400c0fe6d0 0x7f400c0f8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4013f8d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c1167c0 con 0x7f400c0fdd70 2024-02-13T00:49:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4013f8d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c116930 con 0x7f400c0f6b70 2024-02-13T00:49:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4013f8d700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400c116af0 con 0x7f400c0fe6d0 2024-02-13T00:49:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4011528700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f400c0fdd70 0x7f400c0f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f401252a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f400c0fe6d0 0x7f400c0f8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4011528700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f400c0fdd70 0x7f400c0f7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:39754/0 (socket says 172.21.15.162:39754) 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4011528700 1 -- 172.21.15.162:0/403628515 learned_addr learned my addr 172.21.15.162:0/403628515 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f401252a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f400c0fe6d0 0x7f400c0f8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:55412/0 (socket says 172.21.15.162:55412) 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f4011d29700 1 --2- 172.21.15.162:0/403628515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f400c0f6b70 0x7f400c0f7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f401252a700 1 -- 172.21.15.162:0/403628515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f400c0f6b70 msgr2=0x7f400c0f7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f401252a700 1 --2- 172.21.15.162:0/403628515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f400c0f6b70 0x7f400c0f7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f401252a700 1 -- 172.21.15.162:0/403628515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f400c0fdd70 msgr2=0x7f400c0f7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f401252a700 1 --2- 172.21.15.162:0/403628515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f400c0fdd70 0x7f400c0f7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.011+0000 7f401252a700 1 -- 172.21.15.162:0/403628515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4008005040 con 0x7f400c0fe6d0 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.012+0000 7f4011d29700 1 --2- 172.21.15.162:0/403628515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f400c0f6b70 0x7f400c0f7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.012+0000 7f401252a700 1 --2- 172.21.15.162:0/403628515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f400c0fe6d0 0x7f400c0f8160 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7f400400d8d0 tx=0x7f4004004db0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:16.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.012+0000 7f4002ffd700 10 client.?.objecter ms_handle_connect 0x7f400c0fe6d0 2024-02-13T00:49:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.012+0000 7f4002ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.012+0000 7f4002ffd700 1 -- 172.21.15.162:0/403628515 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004004020 con 0x7f400c0fe6d0 2024-02-13T00:49:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.012+0000 7f4002ffd700 1 -- 172.21.15.162:0/403628515 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f400400f040 con 0x7f400c0fe6d0 2024-02-13T00:49:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.012+0000 7f4013f8d700 1 -- 172.21.15.162:0/403628515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400c116dd0 con 0x7f400c0fe6d0 2024-02-13T00:49:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.012+0000 7f4002ffd700 1 -- 172.21.15.162:0/403628515 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004003430 con 0x7f400c0fe6d0 2024-02-13T00:49:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.012+0000 7f4013f8d700 10 client.37739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.012+0000 7f4013f8d700 1 -- 172.21.15.162:0/403628515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400c1172c0 con 0x7f400c0fe6d0 2024-02-13T00:49:16.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.012+0000 7f4013f8d700 1 librados: init done 2024-02-13T00:49:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.013+0000 7f4013f8d700 1 -- 172.21.15.162:0/403628515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400c05f9a0 con 0x7f400c0fe6d0 2024-02-13T00:49:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.014+0000 7f4002ffd700 1 -- 172.21.15.162:0/403628515 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4004004180 con 0x7f400c0fe6d0 2024-02-13T00:49:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.014+0000 7f4002ffd700 1 --2- 172.21.15.162:0/403628515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ff8062c90 0x7f3ff8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.014+0000 7f4002ffd700 1 -- 172.21.15.162:0/403628515 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(541..541 src has 449..541) v4 ==== 7669+0+0 (secure 0 0 0) 0x7f400407ca30 con 0x7f400c0fe6d0 2024-02-13T00:49:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.014+0000 7f4002ffd700 10 client.37739.objecter ms_dispatch 0x7f400c001550 osd_map(541..541 src has 449..541) v4 2024-02-13T00:49:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.014+0000 7f4002ffd700 3 client.37739.objecter handle_osd_map got epochs [541,541] > 0 2024-02-13T00:49:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.014+0000 7f4002ffd700 3 client.37739.objecter handle_osd_map decoding full epoch 541 2024-02-13T00:49:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.014+0000 7f4011d29700 1 --2- 172.21.15.162:0/403628515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ff8062c90 0x7f3ff8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:16.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.014+0000 7f4002ffd700 20 client.37739.objecter dump_active .. 0 homeless 2024-02-13T00:49:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.015+0000 7f4002ffd700 10 client.37739.objecter ms_handle_connect 0x7f3ff8062c90 2024-02-13T00:49:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.015+0000 7f4011d29700 1 --2- 172.21.15.162:0/403628515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ff8062c90 0x7f3ff8065150 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 rx=0x7f4008004ee0 tx=0x7f400800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.019+0000 7f4002ffd700 1 -- 172.21.15.162:0/403628515 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4004048f60 con 0x7f400c0fe6d0 2024-02-13T00:49:16.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:49:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:49:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:14.471555+0000 mgr.y (mgr.14544) 473 : cluster [DBG] pgmap v809: 230 pgs: 1 activating, 229 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:15.483991+0000 mon.a (mon.0) 2891 : cluster [DBG] osdmap e541: 8 total, 8 up, 4 in 2024-02-13T00:49:16.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.204+0000 7f4013f8d700 1 -- 172.21.15.162:0/403628515 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f400c0475e0 con 0x7f400c0fe6d0 2024-02-13T00:49:16.305 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:49:16] "GET /metrics HTTP/1.1" 200 185678 "" "Prometheus/2.33.4" 2024-02-13T00:49:16.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.492+0000 7f4002ffd700 1 -- 172.21.15.162:0/403628515 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v542) v1 ==== 97+0+0 (secure 0 0 0) 0x7f40040500c0 con 0x7f400c0fe6d0 2024-02-13T00:49:16.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.496+0000 7f4013f8d700 10 librados: watch_flush enter 2024-02-13T00:49:16.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.496+0000 7f4013f8d700 10 librados: watch_flush exit 2024-02-13T00:49:16.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.496+0000 7f4013f8d700 20 client.37739.objecter shutdown clearing up homeless session... 2024-02-13T00:49:16.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.496+0000 7f4013f8d700 10 client.37739.objecter successfully canceled tick 2024-02-13T00:49:16.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.496+0000 7f4013f8d700 1 -- 172.21.15.162:0/403628515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ff8062c90 msgr2=0x7f3ff8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:16.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.496+0000 7f4013f8d700 1 --2- 172.21.15.162:0/403628515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ff8062c90 0x7f3ff8065150 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 rx=0x7f4008004ee0 tx=0x7f400800b040).stop 2024-02-13T00:49:16.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.496+0000 7f4013f8d700 1 -- 172.21.15.162:0/403628515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f400c0fe6d0 msgr2=0x7f400c0f8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:16.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.496+0000 7f4013f8d700 1 --2- 172.21.15.162:0/403628515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f400c0fe6d0 0x7f400c0f8160 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7f400400d8d0 tx=0x7f4004004db0).stop 2024-02-13T00:49:16.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.497+0000 7f4013f8d700 1 -- 172.21.15.162:0/403628515 shutdown_connections 2024-02-13T00:49:16.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.497+0000 7f4013f8d700 1 --2- 172.21.15.162:0/403628515 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ff8062c90 0x7f3ff8065150 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:16.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.497+0000 7f4013f8d700 1 --2- 172.21.15.162:0/403628515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f400c0fe6d0 0x7f400c0f8160 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:16.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.497+0000 7f4013f8d700 1 --2- 172.21.15.162:0/403628515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f400c0f6b70 0x7f400c0f7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:16.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.497+0000 7f4013f8d700 1 --2- 172.21.15.162:0/403628515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f400c0fdd70 0x7f400c0f7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:16.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.497+0000 7f4013f8d700 1 -- 172.21.15.162:0/403628515 >> 172.21.15.162:0/403628515 conn(0x7f400c06c410 msgr2=0x7f400c0fbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:16.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.498+0000 7f4013f8d700 1 -- 172.21.15.162:0/403628515 shutdown_connections 2024-02-13T00:49:16.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.498+0000 7f4013f8d700 1 -- 172.21.15.162:0/403628515 wait complete. 2024-02-13T00:49:16.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:16.498+0000 7f4013f8d700 1 librados: shutdown 2024-02-13T00:49:16.507 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:49:16.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-13T00:49:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:15.738409+0000 mgr.y (mgr.14544) 474 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:17.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:16.206292+0000 mon.c (mon.2) 433 : audit [INF] from='client.? 172.21.15.162:0/403628515' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:17.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:16.206648+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:17.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:16.472134+0000 mon.c (mon.2) 434 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-13T00:49:17.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:16.472922+0000 mon.a (mon.0) 2893 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-13T00:49:17.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:16.488869+0000 mon.a (mon.0) 2894 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:17.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:16.493403+0000 mon.a (mon.0) 2895 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:49:17.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:16.493472+0000 mon.a (mon.0) 2896 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-13T00:49:17.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:16.493529+0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e542: 8 total, 8 up, 4 in 2024-02-13T00:49:17.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:16.499646+0000 mon.b (mon.1) 662 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:17.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:16.500255+0000 mon.a (mon.0) 2898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:15.738409+0000 mgr.y (mgr.14544) 474 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:16.206292+0000 mon.c (mon.2) 433 : audit [INF] from='client.? 172.21.15.162:0/403628515' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:16.206648+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:16.472134+0000 mon.c (mon.2) 434 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-13T00:49:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:16.472922+0000 mon.a (mon.0) 2893 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-13T00:49:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:16.488869+0000 mon.a (mon.0) 2894 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:16.493403+0000 mon.a (mon.0) 2895 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:49:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:16.493472+0000 mon.a (mon.0) 2896 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-13T00:49:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:16.493529+0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e542: 8 total, 8 up, 4 in 2024-02-13T00:49:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:16.499646+0000 mon.b (mon.1) 662 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:16.500255+0000 mon.a (mon.0) 2898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:17.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:15.738409+0000 mgr.y (mgr.14544) 474 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:17.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:16.206292+0000 mon.c (mon.2) 433 : audit [INF] from='client.? 172.21.15.162:0/403628515' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:17.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:16.206648+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:17.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:16.472134+0000 mon.c (mon.2) 434 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-13T00:49:17.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:16.472922+0000 mon.a (mon.0) 2893 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-13T00:49:17.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:16.488869+0000 mon.a (mon.0) 2894 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:17.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:16.493403+0000 mon.a (mon.0) 2895 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:49:17.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:16.493472+0000 mon.a (mon.0) 2896 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-13T00:49:17.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:16.493529+0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e542: 8 total, 8 up, 4 in 2024-02-13T00:49:17.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:16.499646+0000 mon.b (mon.1) 662 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:17.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:16.500255+0000 mon.a (mon.0) 2898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:17.467 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:17.470 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:17.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:49:17.485 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:17.518 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:17.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:17.603 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:17.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:17.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:17.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:17.890 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:17.910 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:17.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:18.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:49:18.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:16.471903+0000 mgr.y (mgr.14544) 475 : cluster [DBG] pgmap v812: 197 pgs: 1 clean+premerge+peered, 196 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:18.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:17.226246+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:18.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:17.226447+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:18.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:17.494324+0000 mon.a (mon.0) 2899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:18.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:17.494398+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e543: 8 total, 8 up, 4 in 2024-02-13T00:49:18.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:17.497770+0000 mon.b (mon.1) 663 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:49:18.020 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:17.498182+0000 mon.a (mon.0) 2901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:49:18.023 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:16.471903+0000 mgr.y (mgr.14544) 475 : cluster [DBG] pgmap v812: 197 pgs: 1 clean+premerge+peered, 196 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:18.023 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:17.226246+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:18.024 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:17.226447+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:18.024 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:17.494324+0000 mon.a (mon.0) 2899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:18.024 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:17.494398+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e543: 8 total, 8 up, 4 in 2024-02-13T00:49:18.024 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:17.497770+0000 mon.b (mon.1) 663 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:49:18.024 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:17.498182+0000 mon.a (mon.0) 2901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:49:18.043 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:16.471903+0000 mgr.y (mgr.14544) 475 : cluster [DBG] pgmap v812: 197 pgs: 1 clean+premerge+peered, 196 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:17.226246+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:17.226447+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:17.494324+0000 mon.a (mon.0) 2899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:17.494398+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e543: 8 total, 8 up, 4 in 2024-02-13T00:49:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:17.497770+0000 mon.b (mon.1) 663 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:49:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:17.498182+0000 mon.a (mon.0) 2901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:49:18.992 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:18.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:49:19.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:49:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:49:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:49:19.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:19.124 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:19.257 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:19.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:19.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:19.393 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:19.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:49:19.415 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:19.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:19.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:19.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:19.546 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:18.472308+0000 mgr.y (mgr.14544) 476 : cluster [DBG] pgmap v815: 229 pgs: 32 unknown, 1 clean+premerge+peered, 196 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:18.496233+0000 mon.a (mon.0) 2902 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:49:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:18.496323+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e544: 8 total, 8 up, 4 in 2024-02-13T00:49:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:18.498299+0000 mon.b (mon.1) 664 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-90"}]: dispatch 2024-02-13T00:49:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:18.500034+0000 mon.a (mon.0) 2904 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-90"}]: dispatch 2024-02-13T00:49:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:19.340163+0000 mon.a (mon.0) 2905 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:19.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:18.472308+0000 mgr.y (mgr.14544) 476 : cluster [DBG] pgmap v815: 229 pgs: 32 unknown, 1 clean+premerge+peered, 196 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:19.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:18.496233+0000 mon.a (mon.0) 2902 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:49:19.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:18.496323+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e544: 8 total, 8 up, 4 in 2024-02-13T00:49:19.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:18.498299+0000 mon.b (mon.1) 664 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-90"}]: dispatch 2024-02-13T00:49:19.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:18.500034+0000 mon.a (mon.0) 2904 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-90"}]: dispatch 2024-02-13T00:49:19.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:19.340163+0000 mon.a (mon.0) 2905 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:19.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:18.472308+0000 mgr.y (mgr.14544) 476 : cluster [DBG] pgmap v815: 229 pgs: 32 unknown, 1 clean+premerge+peered, 196 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:19.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:18.496233+0000 mon.a (mon.0) 2902 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:49:19.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:18.496323+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e544: 8 total, 8 up, 4 in 2024-02-13T00:49:19.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:18.498299+0000 mon.b (mon.1) 664 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-90"}]: dispatch 2024-02-13T00:49:19.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:18.500034+0000 mon.a (mon.0) 2904 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-90"}]: dispatch 2024-02-13T00:49:19.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:19.340163+0000 mon.a (mon.0) 2905 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:49:20.032 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:20.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:20.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:20.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:20.168 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:20.189 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:20.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:20.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:20.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:20.325 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:20.561 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:19.498240+0000 mon.a (mon.0) 2906 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-90"}]': finished 2024-02-13T00:49:20.561 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:19.498327+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e545: 8 total, 8 up, 4 in 2024-02-13T00:49:20.561 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:19.500170+0000 mon.b (mon.1) 665 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-90", "mode": "writeback"}]: dispatch 2024-02-13T00:49:20.561 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:19.501056+0000 mon.a (mon.0) 2908 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-90", "mode": "writeback"}]: dispatch 2024-02-13T00:49:20.561 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:20.495316+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:49:20.561 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:20.499992+0000 mon.a (mon.0) 2910 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-90", "mode": "writeback"}]': finished 2024-02-13T00:49:20.562 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:20.500079+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e546: 8 total, 8 up, 4 in 2024-02-13T00:49:20.562 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:19.498240+0000 mon.a (mon.0) 2906 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-90"}]': finished 2024-02-13T00:49:20.562 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:19.498327+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e545: 8 total, 8 up, 4 in 2024-02-13T00:49:20.562 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:19.500170+0000 mon.b (mon.1) 665 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-90", "mode": "writeback"}]: dispatch 2024-02-13T00:49:20.562 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:19.501056+0000 mon.a (mon.0) 2908 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-90", "mode": "writeback"}]: dispatch 2024-02-13T00:49:20.562 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:20.495316+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:49:20.562 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:20.499992+0000 mon.a (mon.0) 2910 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-90", "mode": "writeback"}]': finished 2024-02-13T00:49:20.563 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:20.500079+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e546: 8 total, 8 up, 4 in 2024-02-13T00:49:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:19.498240+0000 mon.a (mon.0) 2906 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-6", "overlaypool": "test-rados-api-smithi162-84057-90"}]': finished 2024-02-13T00:49:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:19.498327+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e545: 8 total, 8 up, 4 in 2024-02-13T00:49:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:19.500170+0000 mon.b (mon.1) 665 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-90", "mode": "writeback"}]: dispatch 2024-02-13T00:49:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:19.501056+0000 mon.a (mon.0) 2908 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-90", "mode": "writeback"}]: dispatch 2024-02-13T00:49:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:20.495316+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:49:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:20.499992+0000 mon.a (mon.0) 2910 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-90", "mode": "writeback"}]': finished 2024-02-13T00:49:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:20.500079+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e546: 8 total, 8 up, 4 in 2024-02-13T00:49:20.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.817+0000 7fcd77c42700 1 -- 172.21.15.162:0/441650556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd700fd470 msgr2=0x7fcd700f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:20.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.817+0000 7fcd77c42700 1 --2- 172.21.15.162:0/441650556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd700fd470 0x7fcd700f6ca0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 rx=0x7fcd6c004870 tx=0x7fcd6c009cb0).stop 2024-02-13T00:49:20.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.817+0000 7fcd77c42700 1 -- 172.21.15.162:0/441650556 shutdown_connections 2024-02-13T00:49:20.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.817+0000 7fcd77c42700 1 --2- 172.21.15.162:0/441650556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd700f7380 0x7fcd70103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:20.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.817+0000 7fcd77c42700 1 --2- 172.21.15.162:0/441650556 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd700fcb10 0x7fcd700fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:20.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.817+0000 7fcd77c42700 1 --2- 172.21.15.162:0/441650556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd700fd470 0x7fcd700f6ca0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:20.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.817+0000 7fcd77c42700 1 -- 172.21.15.162:0/441650556 >> 172.21.15.162:0/441650556 conn(0x7fcd700f2720 msgr2=0x7fcd700f4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:20.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.817+0000 7fcd77c42700 1 -- 172.21.15.162:0/441650556 shutdown_connections 2024-02-13T00:49:20.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.817+0000 7fcd77c42700 1 -- 172.21.15.162:0/441650556 wait complete. 2024-02-13T00:49:20.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd77c42700 1 librados: starting msgr at 2024-02-13T00:49:20.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd77c42700 1 librados: starting objecter 2024-02-13T00:49:20.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd77c42700 1 Processor -- start 2024-02-13T00:49:20.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd77c42700 1 -- start start 2024-02-13T00:49:20.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd77c42700 1 librados: setting wanted keys 2024-02-13T00:49:20.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd77c42700 1 librados: calling monclient init 2024-02-13T00:49:20.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd77c42700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd700f7380 0x7fcd70109240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:20.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd77c42700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd700fcb10 0x7fcd70109730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:20.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd77c42700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd700fd470 0x7fcd70109d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:20.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd77c42700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd70114420 con 0x7fcd700fcb10 2024-02-13T00:49:20.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd77c42700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd70114590 con 0x7fcd700f7380 2024-02-13T00:49:20.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd77c42700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd70114750 con 0x7fcd700fd470 2024-02-13T00:49:20.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd761df700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd700fd470 0x7fcd70109d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:20.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd751dd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd700fcb10 0x7fcd70109730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:20.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd761df700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd700fd470 0x7fcd70109d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:55430/0 (socket says 172.21.15.162:55430) 2024-02-13T00:49:20.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd751dd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd700fcb10 0x7fcd70109730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:39786/0 (socket says 172.21.15.162:39786) 2024-02-13T00:49:20.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.818+0000 7fcd761df700 1 -- 172.21.15.162:0/4050087500 learned_addr learned my addr 172.21.15.162:0/4050087500 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:20.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd759de700 1 --2- 172.21.15.162:0/4050087500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd700f7380 0x7fcd70109240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:20.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd751dd700 1 -- 172.21.15.162:0/4050087500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd700fd470 msgr2=0x7fcd70109d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:20.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd751dd700 1 --2- 172.21.15.162:0/4050087500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd700fd470 0x7fcd70109d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:20.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd751dd700 1 -- 172.21.15.162:0/4050087500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd700f7380 msgr2=0x7fcd70109240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:20.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd751dd700 1 --2- 172.21.15.162:0/4050087500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd700f7380 0x7fcd70109240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:20.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd751dd700 1 -- 172.21.15.162:0/4050087500 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd60005040 con 0x7fcd700fcb10 2024-02-13T00:49:20.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd761df700 1 --2- 172.21.15.162:0/4050087500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd700fd470 0x7fcd70109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:49:20.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd751dd700 1 --2- 172.21.15.162:0/4050087500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd700fcb10 0x7fcd70109730 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 rx=0x7fcd6c004870 tx=0x7fcd6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:20.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd66ffd700 10 client.?.objecter ms_handle_connect 0x7fcd700fcb10 2024-02-13T00:49:20.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd66ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:20.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd66ffd700 1 -- 172.21.15.162:0/4050087500 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c00e070 con 0x7fcd700fcb10 2024-02-13T00:49:20.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd66ffd700 1 -- 172.21.15.162:0/4050087500 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd6c013da0 con 0x7fcd700fcb10 2024-02-13T00:49:20.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd77c42700 1 -- 172.21.15.162:0/4050087500 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd6c005040 con 0x7fcd700fcb10 2024-02-13T00:49:20.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd66ffd700 1 -- 172.21.15.162:0/4050087500 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c019070 con 0x7fcd700fcb10 2024-02-13T00:49:20.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd77c42700 10 client.27645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:20.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd77c42700 1 -- 172.21.15.162:0/4050087500 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd70114d20 con 0x7fcd700fcb10 2024-02-13T00:49:20.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.819+0000 7fcd77c42700 1 librados: init done 2024-02-13T00:49:20.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.820+0000 7fcd77c42700 1 -- 172.21.15.162:0/4050087500 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd700607d0 con 0x7fcd700fcb10 2024-02-13T00:49:20.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.821+0000 7fcd66ffd700 1 -- 172.21.15.162:0/4050087500 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fcd6c01e420 con 0x7fcd700fcb10 2024-02-13T00:49:20.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.821+0000 7fcd66ffd700 1 --2- 172.21.15.162:0/4050087500 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcd5c062c90 0x7fcd5c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:20.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.821+0000 7fcd66ffd700 1 -- 172.21.15.162:0/4050087500 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(546..546 src has 449..546) v4 ==== 8062+0+0 (secure 0 0 0) 0x7fcd6c07da10 con 0x7fcd700fcb10 2024-02-13T00:49:20.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.821+0000 7fcd759de700 1 --2- 172.21.15.162:0/4050087500 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcd5c062c90 0x7fcd5c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:20.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.821+0000 7fcd66ffd700 10 client.27645.objecter ms_dispatch 0x7fcd70001550 osd_map(546..546 src has 449..546) v4 2024-02-13T00:49:20.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.821+0000 7fcd66ffd700 3 client.27645.objecter handle_osd_map got epochs [546,546] > 0 2024-02-13T00:49:20.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.821+0000 7fcd66ffd700 3 client.27645.objecter handle_osd_map decoding full epoch 546 2024-02-13T00:49:20.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.821+0000 7fcd66ffd700 20 client.27645.objecter dump_active .. 0 homeless 2024-02-13T00:49:20.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.821+0000 7fcd66ffd700 10 client.27645.objecter ms_handle_connect 0x7fcd5c062c90 2024-02-13T00:49:20.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.821+0000 7fcd759de700 1 --2- 172.21.15.162:0/4050087500 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcd5c062c90 0x7fcd5c065150 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 rx=0x7fcd60009350 tx=0x7fcd6000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:20.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:20.826+0000 7fcd66ffd700 1 -- 172.21.15.162:0/4050087500 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fcd6c051000 con 0x7fcd700fcb10 2024-02-13T00:49:21.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.005+0000 7fcd77c42700 1 -- 172.21.15.162:0/4050087500 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fcd70105ff0 con 0x7fcd700fcb10 2024-02-13T00:49:21.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.500+0000 7fcd66ffd700 1 -- 172.21.15.162:0/4050087500 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v547) v1 ==== 113+0+0 (secure 0 0 0) 0x7fcd6c050bf0 con 0x7fcd700fcb10 2024-02-13T00:49:21.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 10 librados: watch_flush enter 2024-02-13T00:49:21.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 10 librados: watch_flush exit 2024-02-13T00:49:21.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 20 client.27645.objecter shutdown clearing up homeless session... 2024-02-13T00:49:21.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 10 client.27645.objecter successfully canceled tick 2024-02-13T00:49:21.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 1 -- 172.21.15.162:0/4050087500 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcd5c062c90 msgr2=0x7fcd5c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:21.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 1 --2- 172.21.15.162:0/4050087500 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcd5c062c90 0x7fcd5c065150 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 rx=0x7fcd60009350 tx=0x7fcd6000b040).stop 2024-02-13T00:49:21.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 1 -- 172.21.15.162:0/4050087500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd700fcb10 msgr2=0x7fcd70109730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:21.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 1 --2- 172.21.15.162:0/4050087500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd700fcb10 0x7fcd70109730 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 rx=0x7fcd6c004870 tx=0x7fcd6c00b040).stop 2024-02-13T00:49:21.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 1 -- 172.21.15.162:0/4050087500 shutdown_connections 2024-02-13T00:49:21.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 1 --2- 172.21.15.162:0/4050087500 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcd5c062c90 0x7fcd5c065150 unknown :-1 s=CLOSED pgs=2954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:21.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 1 --2- 172.21.15.162:0/4050087500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd700fd470 0x7fcd70109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:21.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 1 --2- 172.21.15.162:0/4050087500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd700f7380 0x7fcd70109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:21.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 1 --2- 172.21.15.162:0/4050087500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd700fcb10 0x7fcd70109730 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:21.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.504+0000 7fcd77c42700 1 -- 172.21.15.162:0/4050087500 >> 172.21.15.162:0/4050087500 conn(0x7fcd700f2720 msgr2=0x7fcd700fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:21.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.505+0000 7fcd77c42700 1 -- 172.21.15.162:0/4050087500 shutdown_connections 2024-02-13T00:49:21.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.505+0000 7fcd77c42700 1 -- 172.21.15.162:0/4050087500 wait complete. 2024-02-13T00:49:21.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.505+0000 7fcd77c42700 1 librados: shutdown 2024-02-13T00:49:21.509 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.1 to 1 (10000) 2024-02-13T00:49:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:20.473324+0000 mgr.y (mgr.14544) 477 : cluster [DBG] pgmap v818: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:20.501242+0000 mon.b (mon.1) 666 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:49:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:20.502544+0000 mon.a (mon.0) 2912 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:49:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:21.006218+0000 mon.a (mon.0) 2913 : audit [INF] from='client.? 172.21.15.162:0/4050087500' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:49:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:21.501221+0000 mon.a (mon.0) 2914 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:49:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:21.501307+0000 mon.a (mon.0) 2915 : audit [INF] from='client.? 172.21.15.162:0/4050087500' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:49:21.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:21.501358+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e547: 8 total, 8 up, 4 in 2024-02-13T00:49:21.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:21.505498+0000 mon.b (mon.1) 667 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:21.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:21.505907+0000 mon.a (mon.0) 2917 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:20.473324+0000 mgr.y (mgr.14544) 477 : cluster [DBG] pgmap v818: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:20.501242+0000 mon.b (mon.1) 666 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:49:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:20.502544+0000 mon.a (mon.0) 2912 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:49:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:21.006218+0000 mon.a (mon.0) 2913 : audit [INF] from='client.? 172.21.15.162:0/4050087500' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:49:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:21.501221+0000 mon.a (mon.0) 2914 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:49:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:21.501307+0000 mon.a (mon.0) 2915 : audit [INF] from='client.? 172.21.15.162:0/4050087500' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:49:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:21.501358+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e547: 8 total, 8 up, 4 in 2024-02-13T00:49:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:21.505498+0000 mon.b (mon.1) 667 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:21.505907+0000 mon.a (mon.0) 2917 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:21.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:20.473324+0000 mgr.y (mgr.14544) 477 : cluster [DBG] pgmap v818: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:21.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:20.501242+0000 mon.b (mon.1) 666 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:49:21.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:20.502544+0000 mon.a (mon.0) 2912 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:49:21.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:21.006218+0000 mon.a (mon.0) 2913 : audit [INF] from='client.? 172.21.15.162:0/4050087500' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:49:21.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:21.501221+0000 mon.a (mon.0) 2914 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:49:21.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:21.501307+0000 mon.a (mon.0) 2915 : audit [INF] from='client.? 172.21.15.162:0/4050087500' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:49:21.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:21.501358+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e547: 8 total, 8 up, 4 in 2024-02-13T00:49:21.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:21.505498+0000 mon.b (mon.1) 667 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:21.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:21.505907+0000 mon.a (mon.0) 2917 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:22.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.937+0000 7f6f71d90700 1 -- 172.21.15.162:0/1861808897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f6c0f7380 msgr2=0x7f6f6c103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.937+0000 7f6f71d90700 1 --2- 172.21.15.162:0/1861808897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f6c0f7380 0x7f6f6c103980 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7f6f64009c30 tx=0x7f6f64004c50).stop 2024-02-13T00:49:22.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.937+0000 7f6f71d90700 1 -- 172.21.15.162:0/1861808897 shutdown_connections 2024-02-13T00:49:22.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.937+0000 7f6f71d90700 1 --2- 172.21.15.162:0/1861808897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f6c0f7380 0x7f6f6c103980 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.937+0000 7f6f71d90700 1 --2- 172.21.15.162:0/1861808897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f6c0fcb10 0x7f6f6c0fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.937+0000 7f6f71d90700 1 --2- 172.21.15.162:0/1861808897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f6c0fd470 0x7f6f6c0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.937+0000 7f6f71d90700 1 -- 172.21.15.162:0/1861808897 >> 172.21.15.162:0/1861808897 conn(0x7f6f6c0f2720 msgr2=0x7f6f6c0f4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:22.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.937+0000 7f6f71d90700 1 -- 172.21.15.162:0/1861808897 shutdown_connections 2024-02-13T00:49:22.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.937+0000 7f6f71d90700 1 -- 172.21.15.162:0/1861808897 wait complete. 2024-02-13T00:49:22.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f71d90700 1 librados: starting msgr at 2024-02-13T00:49:22.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f71d90700 1 librados: starting objecter 2024-02-13T00:49:22.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f71d90700 1 Processor -- start 2024-02-13T00:49:22.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f71d90700 1 -- start start 2024-02-13T00:49:22.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f71d90700 1 librados: setting wanted keys 2024-02-13T00:49:22.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f71d90700 1 librados: calling monclient init 2024-02-13T00:49:22.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f71d90700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f6c0f7380 0x7f6f6c10ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f71d90700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f6c0fcb10 0x7f6f6c10d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f71d90700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f6c0fd470 0x7f6f6c107050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f71d90700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c107540 con 0x7f6f6c0fcb10 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f71d90700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1076b0 con 0x7f6f6c0fd470 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f71d90700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c107870 con 0x7f6f6c0f7380 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f6b7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f6c0f7380 0x7f6f6c10ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f6affd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f6c0fcb10 0x7f6f6c10d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f6b7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f6c0f7380 0x7f6f6c10ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:55450/0 (socket says 172.21.15.162:55450) 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.938+0000 7f6f6b7fe700 1 -- 172.21.15.162:0/2170996107 learned_addr learned my addr 172.21.15.162:0/2170996107 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f6bfff700 1 --2- 172.21.15.162:0/2170996107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f6c0fd470 0x7f6f6c107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f6b7fe700 1 -- 172.21.15.162:0/2170996107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f6c0fd470 msgr2=0x7f6f6c107050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f6b7fe700 1 --2- 172.21.15.162:0/2170996107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f6c0fd470 0x7f6f6c107050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f6b7fe700 1 -- 172.21.15.162:0/2170996107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f6c0fcb10 msgr2=0x7f6f6c10d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f6b7fe700 1 --2- 172.21.15.162:0/2170996107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f6c0fcb10 0x7f6f6c10d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f6b7fe700 1 -- 172.21.15.162:0/2170996107 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c005040 con 0x7f6f6c0f7380 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f6affd700 1 --2- 172.21.15.162:0/2170996107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f6c0fcb10 0x7f6f6c10d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f6bfff700 1 --2- 172.21.15.162:0/2170996107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f6c0fd470 0x7f6f6c107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f6b7fe700 1 --2- 172.21.15.162:0/2170996107 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f6c0f7380 0x7f6f6c10ceb0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7f6f5c009400 tx=0x7f6f5c009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f68ff9700 10 client.?.objecter ms_handle_connect 0x7f6f6c0f7380 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f68ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c007a40 con 0x7f6f6c0f7380 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f5c00f040 con 0x7f6f6c0f7380 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f640097c0 con 0x7f6f6c0f7380 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c014aa0 con 0x7f6f6c0f7380 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f71d90700 10 client.37751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6c107eb0 con 0x7f6f6c0f7380 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.939+0000 7f6f71d90700 1 librados: init done 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.940+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6f6c0607d0 con 0x7f6f6c0f7380 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.940+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6f5c00e630 con 0x7f6f6c0f7380 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f68ff9700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6f54062c90 0x7f6f54065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(547..547 src has 449..547) v4 ==== 8062+0+0 (secure 0 0 0) 0x7f6f5c07cdf0 con 0x7f6f6c0f7380 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 osd_map(547..547 src has 449..547) v4 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f68ff9700 3 client.37751.objecter handle_osd_map got epochs [547,547] > 0 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f68ff9700 3 client.37751.objecter handle_osd_map decoding full epoch 547 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f6affd700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6f54062c90 0x7f6f54065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f68ff9700 20 client.37751.objecter dump_active .. 0 homeless 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v547) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6f5c07f4c0 con 0x7f6f6c0f7380 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f68ff9700 10 client.37751.objecter ms_handle_connect 0x7f6f54062c90 2024-02-13T00:49:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f6affd700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6f54062c90 0x7f6f54065150 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 rx=0x7f6f6c1087f0 tx=0x7f6f6000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:22.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f71d90700 10 client.37751.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:22.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 1 2024-02-13T00:49:22.138 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:49:22.138 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:49:22.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6f6c0fdde0 0x7f6f6c0fe1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:22.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c106070 osd=0 3 2024-02-13T00:49:22.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c106070 osd=0 3 2024-02-13T00:49:22.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c106070 osd=0 3 2024-02-13T00:49:22.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 1 2024-02-13T00:49:22.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c106070 osd=0 3 2024-02-13T00:49:22.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 0 1 2024-02-13T00:49:22.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c106070 osd=0 4 2024-02-13T00:49:22.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f71d90700 10 client.37751.objecter _send_command 1 2024-02-13T00:49:22.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f6c0fe6b0 con 0x7f6f6c0fdde0 2024-02-13T00:49:22.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.941+0000 7f6f6bfff700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6f6c0fdde0 0x7f6f6c0fe1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:22.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.942+0000 7f6f68ff9700 10 client.37751.objecter ms_handle_connect 0x7f6f6c0fdde0 2024-02-13T00:49:22.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.942+0000 7f6f6bfff700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6f6c0fdde0 0x7f6f6c0fe1c0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.942+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6f6c0fe6b0 con 0x7f6f6c0fdde0 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.942+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.942+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 1 = system:0 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.942+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c106070 osd=0 4 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.942+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 0 1 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.958+0000 7f6f71d90700 10 client.37751.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.958+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 2 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.958+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c106070 osd=0 3 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.958+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c106070 osd=0 3 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.958+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c106070 osd=0 3 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.958+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 2 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.958+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c106070 osd=0 3 2024-02-13T00:49:22.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.958+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 0 2 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.958+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c106070 osd=0 4 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.958+0000 7f6f71d90700 10 client.37751.objecter _send_command 2 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.958+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6f6c0fe880 con 0x7f6f6c0fdde0 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6f6c0fe880 con 0x7f6f6c0fdde0 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c106070 osd=0 4 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 0 2 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f71d90700 10 client.37751.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 3 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6805/150599780 conn(0x7f6f6c028590 0x7f6f6c117d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c0f4c60 osd=1 3 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c0f4c60 osd=1 3 2024-02-13T00:49:22.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c0f4c60 osd=1 3 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 3 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c0f4c60 osd=1 3 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 1 3 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c0f4c60 osd=1 4 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f71d90700 10 client.37751.objecter _send_command 3 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f6c0f4dd0 con 0x7f6f6c028590 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.960+0000 7f6f6b7fe700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6805/150599780 conn(0x7f6f6c028590 0x7f6f6c117d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.961+0000 7f6f68ff9700 10 client.37751.objecter ms_handle_connect 0x7f6f6c028590 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.961+0000 7f6f6b7fe700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6805/150599780 conn(0x7f6f6c028590 0x7f6f6c117d90 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.961+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6f6c0f4dd0 con 0x7f6f6c028590 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.961+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.961+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 3 = system:0 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.961+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c0f4c60 osd=1 4 2024-02-13T00:49:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.961+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 1 3 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.976+0000 7f6f71d90700 10 client.37751.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.976+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 4 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.976+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c0f4c60 osd=1 3 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.976+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c0f4c60 osd=1 3 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.976+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c0f4c60 osd=1 3 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.976+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 4 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.976+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c0f4c60 osd=1 3 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.976+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 1 4 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.976+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c0f4c60 osd=1 4 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.976+0000 7f6f71d90700 10 client.37751.objecter _send_command 4 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.976+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6f6c05a0e0 con 0x7f6f6c028590 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.977+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6f6c05a0e0 con 0x7f6f6c028590 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:22.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c0f4c60 osd=1 4 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 1 4 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f71d90700 10 client.37751.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 5 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6f6c059570 0x7f6f6c11e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c059400 osd=2 3 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c059400 osd=2 3 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c059400 osd=2 3 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 5 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c059400 osd=2 3 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 2 5 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c059400 osd=2 4 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f71d90700 10 client.37751.objecter _send_command 5 2024-02-13T00:49:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f6c11a290 con 0x7f6f6c059570 2024-02-13T00:49:22.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f6affd700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6f6c059570 0x7f6f6c11e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:22.145 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:49:22.145 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:49:22.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f68ff9700 10 client.37751.objecter ms_handle_connect 0x7f6f6c059570 2024-02-13T00:49:22.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.978+0000 7f6f6affd700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6f6c059570 0x7f6f6c11e8a0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:22.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.979+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6f6c11a290 con 0x7f6f6c059570 2024-02-13T00:49:22.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.979+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.979+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 5 = system:0 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.979+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c059400 osd=2 4 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.979+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 2 5 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.993+0000 7f6f71d90700 10 client.37751.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.993+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 6 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.993+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c059400 osd=2 3 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.993+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c059400 osd=2 3 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.993+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c059400 osd=2 3 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.993+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 6 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.993+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c059400 osd=2 3 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.993+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 2 6 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.993+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c059400 osd=2 4 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.993+0000 7f6f71d90700 10 client.37751.objecter _send_command 6 2024-02-13T00:49:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.993+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6f6c061f00 con 0x7f6f6c059570 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.994+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6f6c061f00 con 0x7f6f6c059570 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.994+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.994+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.994+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c059400 osd=2 4 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.994+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 2 6 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f71d90700 10 client.37751.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 7 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6813/604971011 conn(0x7f6f6c061330 0x7f6f6c11ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c0611c0 osd=3 3 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c0611c0 osd=3 3 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c0611c0 osd=3 3 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 7 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c0611c0 osd=3 3 2024-02-13T00:49:22.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 3 7 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c0611c0 osd=3 4 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f71d90700 10 client.37751.objecter _send_command 7 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f6c0282f0 con 0x7f6f6c061330 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f6bfff700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6813/604971011 conn(0x7f6f6c061330 0x7f6f6c11ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f6bfff700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6813/604971011 conn(0x7f6f6c061330 0x7f6f6c11ff70 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f68ff9700 10 client.37751.objecter ms_handle_connect 0x7f6f6c061330 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6f6c0282f0 con 0x7f6f6c061330 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 7 = system:0 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c0611c0 osd=3 4 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:21.995+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 3 7 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.010+0000 7f6f71d90700 10 client.37751.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.010+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 8 2024-02-13T00:49:22.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.010+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c0611c0 osd=3 3 2024-02-13T00:49:22.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.010+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c0611c0 osd=3 3 2024-02-13T00:49:22.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.010+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c0611c0 osd=3 3 2024-02-13T00:49:22.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.010+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 8 2024-02-13T00:49:22.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.010+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c0611c0 osd=3 3 2024-02-13T00:49:22.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.010+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 3 8 2024-02-13T00:49:22.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.010+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c0611c0 osd=3 4 2024-02-13T00:49:22.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.010+0000 7f6f71d90700 10 client.37751.objecter _send_command 8 2024-02-13T00:49:22.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.010+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6f6c027650 con 0x7f6f6c061330 2024-02-13T00:49:22.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6f6c027650 con 0x7f6f6c061330 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c0611c0 osd=3 4 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 3 8 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f71d90700 10 client.37751.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 9 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6f6c0277f0 0x7f6f6c1255a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c027650 osd=4 3 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c027650 osd=4 3 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c027650 osd=4 3 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 9 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c027650 osd=4 3 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 4 9 2024-02-13T00:49:22.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c027650 osd=4 4 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f71d90700 10 client.37751.objecter _send_command 9 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.011+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f6c121240 con 0x7f6f6c0277f0 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.012+0000 7f6f6b7fe700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6f6c0277f0 0x7f6f6c1255a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.012+0000 7f6f68ff9700 10 client.37751.objecter ms_handle_connect 0x7f6f6c0277f0 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.012+0000 7f6f6b7fe700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6f6c0277f0 0x7f6f6c1255a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.013+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6f6c121240 con 0x7f6f6c0277f0 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.013+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.013+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 9 = system:0 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.013+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c027650 osd=4 4 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.013+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 4 9 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.027+0000 7f6f71d90700 10 client.37751.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.027+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 10 2024-02-13T00:49:22.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.027+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c027650 osd=4 3 2024-02-13T00:49:22.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.027+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c027650 osd=4 3 2024-02-13T00:49:22.152 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:49:22.152 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:49:22.152 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:49:22.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.027+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c027650 osd=4 3 2024-02-13T00:49:22.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.027+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 10 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.027+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c027650 osd=4 3 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.027+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 4 10 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.027+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c027650 osd=4 4 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.027+0000 7f6f71d90700 10 client.37751.objecter _send_command 10 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.027+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6f6c121240 con 0x7f6f6c0277f0 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6f6c121240 con 0x7f6f6c0277f0 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c027650 osd=4 4 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 4 10 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f71d90700 10 client.37751.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 11 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6f6c125fa0 0x7f6f6c128390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c121240 osd=5 3 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c121240 osd=5 3 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c121240 osd=5 3 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 11 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c121240 osd=5 3 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 5 11 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c121240 osd=5 4 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f71d90700 10 client.37751.objecter _send_command 11 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.029+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f6c12b110 con 0x7f6f6c125fa0 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.030+0000 7f6f6affd700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6f6c125fa0 0x7f6f6c128390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.030+0000 7f6f6affd700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6f6c125fa0 0x7f6f6c128390 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.030+0000 7f6f68ff9700 10 client.37751.objecter ms_handle_connect 0x7f6f6c125fa0 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.031+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6f6c12b110 con 0x7f6f6c125fa0 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.031+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:49:22.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.031+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 11 = system:0 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.031+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c121240 osd=5 4 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.031+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 5 11 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.044+0000 7f6f71d90700 10 client.37751.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.044+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 12 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.044+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c121240 osd=5 3 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.044+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c121240 osd=5 3 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.044+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c121240 osd=5 3 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.044+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 12 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.044+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c121240 osd=5 3 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.044+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 5 12 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.044+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c121240 osd=5 4 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.044+0000 7f6f71d90700 10 client.37751.objecter _send_command 12 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.044+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6f6c12b110 con 0x7f6f6c125fa0 2024-02-13T00:49:22.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6f6c12b110 con 0x7f6f6c125fa0 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c121240 osd=5 4 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 5 12 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f71d90700 10 client.37751.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 13 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6f6c128e70 0x7f6f6c1292d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c12b110 osd=6 3 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c12b110 osd=6 3 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c12b110 osd=6 3 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 13 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c12b110 osd=6 3 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 6 13 2024-02-13T00:49:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c12b110 osd=6 4 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f71d90700 10 client.37751.objecter _send_command 13 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f6c12e540 con 0x7f6f6c128e70 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.047+0000 7f6f6bfff700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6f6c128e70 0x7f6f6c1292d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.048+0000 7f6f68ff9700 10 client.37751.objecter ms_handle_connect 0x7f6f6c128e70 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.048+0000 7f6f6bfff700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6f6c128e70 0x7f6f6c1292d0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.048+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6f6c12e540 con 0x7f6f6c128e70 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.048+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.048+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 13 = system:0 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.048+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c12b110 osd=6 4 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.048+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 6 13 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.062+0000 7f6f71d90700 10 client.37751.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.062+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 14 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.062+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c12b110 osd=6 3 2024-02-13T00:49:22.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.062+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c12b110 osd=6 3 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.062+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c12b110 osd=6 3 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.062+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 14 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.062+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c12b110 osd=6 3 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.062+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 6 14 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.062+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c12b110 osd=6 4 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.062+0000 7f6f71d90700 10 client.37751.objecter _send_command 14 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.062+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6f6c1298b0 con 0x7f6f6c128e70 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.064+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6f6c1298b0 con 0x7f6f6c128e70 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.064+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.064+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.064+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c12b110 osd=6 4 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.064+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 6 14 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f71d90700 10 client.37751.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:22.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 15 2024-02-13T00:49:22.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6f6c129f30 0x7f6f6c12a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:22.159 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:49:22.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c1298b0 osd=7 3 2024-02-13T00:49:22.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c1298b0 osd=7 3 2024-02-13T00:49:22.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c1298b0 osd=7 3 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 15 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c1298b0 osd=7 3 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 7 15 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c1298b0 osd=7 4 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f71d90700 10 client.37751.objecter _send_command 15 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f6c12a940 con 0x7f6f6c129f30 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f6b7fe700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6f6c129f30 0x7f6f6c12a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f6b7fe700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6f6c129f30 0x7f6f6c12a370 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.065+0000 7f6f68ff9700 10 client.37751.objecter ms_handle_connect 0x7f6f6c129f30 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.066+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6f6c12a940 con 0x7f6f6c129f30 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.066+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.066+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 15 = system:0 2024-02-13T00:49:22.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.066+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c1298b0 osd=7 4 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.066+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 7 15 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.080+0000 7f6f71d90700 10 client.37751.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.080+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign -1 16 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.080+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c1298b0 osd=7 3 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.080+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c1298b0 osd=7 3 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.080+0000 7f6f71d90700 20 client.37751.objecter _get_session s=0x7f6f6c1298b0 osd=7 3 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.080+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_remove -1 16 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.080+0000 7f6f71d90700 20 client.37751.objecter get_session s=0x7f6f6c1298b0 osd=7 3 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.080+0000 7f6f71d90700 15 client.37751.objecter _session_command_op_assign 7 16 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.080+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c1298b0 osd=7 4 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.080+0000 7f6f71d90700 10 client.37751.objecter _send_command 16 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.080+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6f6c12a940 con 0x7f6f6c129f30 2024-02-13T00:49:22.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f68ff9700 1 -- 172.21.15.162:0/2170996107 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6f6c12a940 con 0x7f6f6c129f30 2024-02-13T00:49:22.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f68ff9700 10 client.37751.objecter ms_dispatch 0x7f6f6c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:22.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f68ff9700 10 client.37751.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f68ff9700 20 client.37751.objecter put_session s=0x7f6f6c1298b0 osd=7 4 2024-02-13T00:49:22.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f68ff9700 15 client.37751.objecter _session_command_op_remove 7 16 2024-02-13T00:49:22.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 10 librados: watch_flush enter 2024-02-13T00:49:22.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 10 librados: watch_flush exit 2024-02-13T00:49:22.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 10 client.37751.objecter close_session for osd.0 2024-02-13T00:49:22.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6f6c0fdde0 msgr2=0x7f6f6c0fe1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6f6c0fdde0 0x7f6f6c0fe1c0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c106070 osd=0 1 2024-02-13T00:49:22.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 10 client.37751.objecter close_session for osd.1 2024-02-13T00:49:22.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6805/150599780 conn(0x7f6f6c028590 msgr2=0x7f6f6c117d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6805/150599780 conn(0x7f6f6c028590 0x7f6f6c117d90 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c0f4c60 osd=1 1 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 10 client.37751.objecter close_session for osd.2 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6f6c059570 msgr2=0x7f6f6c11e8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6f6c059570 0x7f6f6c11e8a0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c059400 osd=2 1 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 10 client.37751.objecter close_session for osd.3 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6813/604971011 conn(0x7f6f6c061330 msgr2=0x7f6f6c11ff70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6813/604971011 conn(0x7f6f6c061330 0x7f6f6c11ff70 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f6bfff700 1 -- 172.21.15.162:0/2170996107 reap_dead start 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c0611c0 osd=3 1 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 10 client.37751.objecter close_session for osd.4 2024-02-13T00:49:22.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6f6c0277f0 msgr2=0x7f6f6c1255a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6f6c0277f0 0x7f6f6c1255a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c027650 osd=4 1 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 10 client.37751.objecter close_session for osd.5 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6f6c125fa0 msgr2=0x7f6f6c128390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f6bfff700 1 -- 172.21.15.162:0/2170996107 reap_dead start 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6f6c125fa0 0x7f6f6c128390 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c121240 osd=5 1 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 10 client.37751.objecter close_session for osd.6 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6f6c128e70 msgr2=0x7f6f6c1292d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6f6c128e70 0x7f6f6c1292d0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c12b110 osd=6 1 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 10 client.37751.objecter close_session for osd.7 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6f6c129f30 msgr2=0x7f6f6c12a370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.164 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6f6c129f30 0x7f6f6c12a370 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 20 client.37751.objecter put_session s=0x7f6f6c1298b0 osd=7 1 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 20 client.37751.objecter shutdown clearing up homeless session... 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 10 client.37751.objecter successfully canceled tick 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6f54062c90 msgr2=0x7f6f54065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6f54062c90 0x7f6f54065150 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 rx=0x7f6f6c1087f0 tx=0x7f6f6000b040).stop 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f6c0f7380 msgr2=0x7f6f6c10ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 --2- 172.21.15.162:0/2170996107 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f6c0f7380 0x7f6f6c10ceb0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7f6f5c009400 tx=0x7f6f5c009cb0).stop 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f6bfff700 1 -- 172.21.15.162:0/2170996107 reap_dead start 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 shutdown_connections 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.082+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 >> 172.21.15.162:0/2170996107 conn(0x7f6f6c0f2720 msgr2=0x7f6f6c0f40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.083+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 shutdown_connections 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.083+0000 7f6f71d90700 1 -- 172.21.15.162:0/2170996107 wait complete. 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:22.083+0000 7f6f71d90700 1 librados: shutdown 2024-02-13T00:49:22.165 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.166 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.166 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.166 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.166 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.166 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.166 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.166 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:49:22.166 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:22.375 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:22.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:22.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:22.513 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:22.535 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:22.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:22.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:22.669 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:22.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:22.747 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:21.941737+0000 mon.c (mon.2) 435 : audit [DBG] from='client.? 172.21.15.162:0/2170996107' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:22.747 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:22.505668+0000 mon.a (mon.0) 2918 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:49:22.747 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:22.505739+0000 mon.a (mon.0) 2919 : cluster [DBG] osdmap e548: 8 total, 8 up, 4 in 2024-02-13T00:49:22.747 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:22.507136+0000 mon.b (mon.1) 668 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:49:22.748 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:22.507562+0000 mon.a (mon.0) 2920 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:49:22.748 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:21.941737+0000 mon.c (mon.2) 435 : audit [DBG] from='client.? 172.21.15.162:0/2170996107' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:22.748 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:22.505668+0000 mon.a (mon.0) 2918 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:49:22.748 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:22.505739+0000 mon.a (mon.0) 2919 : cluster [DBG] osdmap e548: 8 total, 8 up, 4 in 2024-02-13T00:49:22.748 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:22.507136+0000 mon.b (mon.1) 668 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:49:22.748 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:22.507562+0000 mon.a (mon.0) 2920 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:49:22.769 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 4 1 2024-02-13T00:49:22.792 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:22.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:21.941737+0000 mon.c (mon.2) 435 : audit [DBG] from='client.? 172.21.15.162:0/2170996107' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:22.505668+0000 mon.a (mon.0) 2918 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:49:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:22.505739+0000 mon.a (mon.0) 2919 : cluster [DBG] osdmap e548: 8 total, 8 up, 4 in 2024-02-13T00:49:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:22.507136+0000 mon.b (mon.1) 668 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:49:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:22.507562+0000 mon.a (mon.0) 2920 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:49:22.838 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:22.990 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:22.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:23.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:23.126 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:23.148 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:23.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:23.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:23.251 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:23.280 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:22.473789+0000 mgr.y (mgr.14544) 478 : cluster [DBG] pgmap v821: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:23.500288+0000 mon.a (mon.0) 2921 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:49:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:23.507610+0000 mon.a (mon.0) 2922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-13T00:49:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:23.507688+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e549: 8 total, 8 up, 4 in 2024-02-13T00:49:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:23.509082+0000 mon.b (mon.1) 669 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-13T00:49:23.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:23.510978+0000 mon.a (mon.0) 2924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-13T00:49:23.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:22.473789+0000 mgr.y (mgr.14544) 478 : cluster [DBG] pgmap v821: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:23.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:23.500288+0000 mon.a (mon.0) 2921 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:49:23.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:23.507610+0000 mon.a (mon.0) 2922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-13T00:49:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:23.507688+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e549: 8 total, 8 up, 4 in 2024-02-13T00:49:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:23.509082+0000 mon.b (mon.1) 669 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-13T00:49:23.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:23.510978+0000 mon.a (mon.0) 2924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-13T00:49:23.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:23.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:23.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:49:23.760 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:22.473789+0000 mgr.y (mgr.14544) 478 : cluster [DBG] pgmap v821: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:23.500288+0000 mon.a (mon.0) 2921 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:49:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:23.507610+0000 mon.a (mon.0) 2922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-13T00:49:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:23.507688+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e549: 8 total, 8 up, 4 in 2024-02-13T00:49:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:23.509082+0000 mon.b (mon.1) 669 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-13T00:49:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:23.510978+0000 mon.a (mon.0) 2924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-13T00:49:23.858 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:23.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:49:23.881 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:23.893 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:49:23.893 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:49:23.893 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:49:23.893 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:49:23.893 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:49:23.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:23.995 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:24.017 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:24.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:24.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:24.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:24.152 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:24.173 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:24.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:24.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:24.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:24.305 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:24.324 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:24.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:24.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:24.455 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:24.474 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:24.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:24.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:24.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:24.604 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:24.621 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:24.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:24.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:24.752 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:24.770 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:24.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:24.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:24.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:24.900 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:24.918 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:24.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:25.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:25.052 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:25.175 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:25.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:25.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:25.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-13T00:49:25.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:25.306 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:25.325 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:25.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:25.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:25.460 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:25.479 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:25.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:25.584 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:25.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:24.474175+0000 mgr.y (mgr.14544) 479 : cluster [DBG] pgmap v824: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:25.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:24.506449+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-13T00:49:25.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:24.506523+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e550: 8 total, 8 up, 4 in 2024-02-13T00:49:25.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:24.530023+0000 mon.b (mon.1) 670 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:25.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:24.530351+0000 mon.a (mon.0) 2927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:25.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:25.482125+0000 mon.a (mon.0) 2928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:49:25.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:25.482197+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e551: 8 total, 8 up, 4 in 2024-02-13T00:49:25.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:25.485204+0000 mon.b (mon.1) 671 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90"}]: dispatch 2024-02-13T00:49:25.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:25.488778+0000 mon.a (mon.0) 2930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90"}]: dispatch 2024-02-13T00:49:25.610 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:25.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:49:25.629 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:25.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:25.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:25.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:25.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:24.474175+0000 mgr.y (mgr.14544) 479 : cluster [DBG] pgmap v824: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:25.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:24.506449+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-13T00:49:25.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:24.506523+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e550: 8 total, 8 up, 4 in 2024-02-13T00:49:25.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:24.530023+0000 mon.b (mon.1) 670 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:25.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:24.530351+0000 mon.a (mon.0) 2927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:25.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:25.482125+0000 mon.a (mon.0) 2928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:49:25.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:25.482197+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e551: 8 total, 8 up, 4 in 2024-02-13T00:49:25.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:25.485204+0000 mon.b (mon.1) 671 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90"}]: dispatch 2024-02-13T00:49:25.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:25.488778+0000 mon.a (mon.0) 2930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90"}]: dispatch 2024-02-13T00:49:25.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:24.474175+0000 mgr.y (mgr.14544) 479 : cluster [DBG] pgmap v824: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:25.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:24.506449+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-90","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-13T00:49:25.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:24.506523+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e550: 8 total, 8 up, 4 in 2024-02-13T00:49:25.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:24.530023+0000 mon.b (mon.1) 670 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:25.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:24.530351+0000 mon.a (mon.0) 2927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:25.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:25.482125+0000 mon.a (mon.0) 2928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]': finished 2024-02-13T00:49:25.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:25.482197+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e551: 8 total, 8 up, 4 in 2024-02-13T00:49:25.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:25.485204+0000 mon.b (mon.1) 671 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90"}]: dispatch 2024-02-13T00:49:25.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:25.488778+0000 mon.a (mon.0) 2930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90"}]: dispatch 2024-02-13T00:49:25.759 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:26.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:49:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:49:26.330 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:49:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:49:26] "GET /metrics HTTP/1.1" 200 188198 "" "Prometheus/2.33.4" 2024-02-13T00:49:26.507 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:26.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:26.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:26.648 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:26.668 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:26.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:26.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:26.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:25.746353+0000 mgr.y (mgr.14544) 480 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:26.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:26.493317+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90"}]': finished 2024-02-13T00:49:26.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:26.493379+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e552: 8 total, 8 up, 4 in 2024-02-13T00:49:26.802 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:26.822 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:26.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:26.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:26.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:25.746353+0000 mgr.y (mgr.14544) 480 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:26.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:26.493317+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90"}]': finished 2024-02-13T00:49:26.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:26.493379+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e552: 8 total, 8 up, 4 in 2024-02-13T00:49:26.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:25.746353+0000 mgr.y (mgr.14544) 480 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:26.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:26.493317+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-90"}]': finished 2024-02-13T00:49:26.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:26.493379+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e552: 8 total, 8 up, 4 in 2024-02-13T00:49:26.926 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:26.953 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:26.973 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:26.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:26.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:27.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:27.103 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:27.122 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:27.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:27.148 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:27.148 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:27.226 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:27.256 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:27.404 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:27.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:27.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:27.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:27.535 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:27.642 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:27.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:27.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:27.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:27.778 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:26.474604+0000 mgr.y (mgr.14544) 481 : cluster [DBG] pgmap v827: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-13T00:49:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:26.506868+0000 osd.2 (osd.2) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:26.507179+0000 osd.2 (osd.2) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:27.491136+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e553: 8 total, 8 up, 4 in 2024-02-13T00:49:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:26.474604+0000 mgr.y (mgr.14544) 481 : cluster [DBG] pgmap v827: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-13T00:49:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:26.506868+0000 osd.2 (osd.2) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:26.507179+0000 osd.2 (osd.2) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:27.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:27.491136+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e553: 8 total, 8 up, 4 in 2024-02-13T00:49:28.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.977+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/1479873842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f480f73a0 msgr2=0x7f0f481039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:28.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.977+0000 7f0f4ee7f700 1 --2- 172.21.15.162:0/1479873842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f480f73a0 0x7f0f481039a0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 rx=0x7f0f40009c30 tx=0x7f0f40004c50).stop 2024-02-13T00:49:28.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.977+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/1479873842 shutdown_connections 2024-02-13T00:49:28.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.977+0000 7f0f4ee7f700 1 --2- 172.21.15.162:0/1479873842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f480f73a0 0x7f0f481039a0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.977+0000 7f0f4ee7f700 1 --2- 172.21.15.162:0/1479873842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f480fcb30 0x7f0f480fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.977+0000 7f0f4ee7f700 1 --2- 172.21.15.162:0/1479873842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f480fd490 0x7f0f480f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.977+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/1479873842 >> 172.21.15.162:0/1479873842 conn(0x7f0f4806eb90 msgr2=0x7f0f480f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:28.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.977+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/1479873842 shutdown_connections 2024-02-13T00:49:28.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.977+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/1479873842 wait complete. 2024-02-13T00:49:28.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4ee7f700 1 librados: starting msgr at 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4ee7f700 1 librados: starting objecter 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4ee7f700 1 Processor -- start 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4ee7f700 1 -- start start 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4ee7f700 1 librados: setting wanted keys 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4ee7f700 1 librados: calling monclient init 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4ee7f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f480f73a0 0x7f0f48106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4ee7f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f480fcb30 0x7f0f48107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4ee7f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f480fd490 0x7f0f48107780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4ee7f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f48116420 con 0x7f0f480f73a0 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4ee7f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f48116560 con 0x7f0f480fcb30 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4ee7f700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f481166d0 con 0x7f0f480fd490 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4d41c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f480fd490 0x7f0f48107780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:28.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4cc1b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f480f73a0 0x7f0f48106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4d41c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f480fd490 0x7f0f48107780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:50632/0 (socket says 172.21.15.162:50632) 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4d41c700 1 -- 172.21.15.162:0/2989219503 learned_addr learned my addr 172.21.15.162:0/2989219503 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f4cc1b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f480f73a0 0x7f0f48106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:47820/0 (socket says 172.21.15.162:47820) 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.978+0000 7f0f47fff700 1 --2- 172.21.15.162:0/2989219503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f480fcb30 0x7f0f48107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f4d41c700 1 -- 172.21.15.162:0/2989219503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f480fcb30 msgr2=0x7f0f48107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f4d41c700 1 --2- 172.21.15.162:0/2989219503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f480fcb30 0x7f0f48107290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f4d41c700 1 -- 172.21.15.162:0/2989219503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f480f73a0 msgr2=0x7f0f48106da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f4d41c700 1 --2- 172.21.15.162:0/2989219503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f480f73a0 0x7f0f48106da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f4d41c700 1 -- 172.21.15.162:0/2989219503 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c005040 con 0x7f0f480fd490 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f4cc1b700 1 --2- 172.21.15.162:0/2989219503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f480f73a0 0x7f0f48106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f4d41c700 1 --2- 172.21.15.162:0/2989219503 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f480fd490 0x7f0f48107780 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7f0f40004860 tx=0x7f0f40004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f45ffb700 10 client.?.objecter ms_handle_connect 0x7f0f480fd490 2024-02-13T00:49:28.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f45ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:28.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f45ffb700 1 -- 172.21.15.162:0/2989219503 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4000e040 con 0x7f0f480fd490 2024-02-13T00:49:28.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f45ffb700 1 -- 172.21.15.162:0/2989219503 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f40018040 con 0x7f0f480fd490 2024-02-13T00:49:28.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f45ffb700 1 -- 172.21.15.162:0/2989219503 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4001c3f0 con 0x7f0f480fd490 2024-02-13T00:49:28.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/2989219503 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f400097c0 con 0x7f0f480fd490 2024-02-13T00:49:28.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f4ee7f700 10 client.37757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:28.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/2989219503 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f48116c40 con 0x7f0f480fd490 2024-02-13T00:49:28.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.979+0000 7f0f4ee7f700 1 librados: init done 2024-02-13T00:49:28.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.980+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/2989219503 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f480475e0 con 0x7f0f480fd490 2024-02-13T00:49:28.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.980+0000 7f0f45ffb700 1 -- 172.21.15.162:0/2989219503 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0f40013420 con 0x7f0f480fd490 2024-02-13T00:49:28.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.981+0000 7f0f45ffb700 1 --2- 172.21.15.162:0/2989219503 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0f38062c90 0x7f0f38065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:28.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.981+0000 7f0f45ffb700 1 -- 172.21.15.162:0/2989219503 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(553..553 src has 449..553) v4 ==== 7669+0+0 (secure 0 0 0) 0x7f0f4007e0c0 con 0x7f0f480fd490 2024-02-13T00:49:28.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.981+0000 7f0f45ffb700 10 client.37757.objecter ms_dispatch 0x7f0f48001550 osd_map(553..553 src has 449..553) v4 2024-02-13T00:49:28.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.981+0000 7f0f45ffb700 3 client.37757.objecter handle_osd_map got epochs [553,553] > 0 2024-02-13T00:49:28.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.981+0000 7f0f45ffb700 3 client.37757.objecter handle_osd_map decoding full epoch 553 2024-02-13T00:49:28.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.981+0000 7f0f4cc1b700 1 --2- 172.21.15.162:0/2989219503 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0f38062c90 0x7f0f38065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:28.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.981+0000 7f0f45ffb700 20 client.37757.objecter dump_active .. 0 homeless 2024-02-13T00:49:28.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.981+0000 7f0f45ffb700 10 client.37757.objecter ms_handle_connect 0x7f0f38062c90 2024-02-13T00:49:28.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.981+0000 7f0f4cc1b700 1 --2- 172.21.15.162:0/2989219503 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0f38062c90 0x7f0f38065150 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 rx=0x7f0f3c004c80 tx=0x7f0f3c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:28.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:27.985+0000 7f0f45ffb700 1 -- 172.21.15.162:0/2989219503 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0f400517c0 con 0x7f0f480fd490 2024-02-13T00:49:28.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:28.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:28.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:26.474604+0000 mgr.y (mgr.14544) 481 : cluster [DBG] pgmap v827: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-13T00:49:28.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:26.506868+0000 osd.2 (osd.2) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:28.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:26.507179+0000 osd.2 (osd.2) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:28.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:27.491136+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e553: 8 total, 8 up, 4 in 2024-02-13T00:49:28.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:28.157 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:28.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.167+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/2989219503 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f0f4805fb80 con 0x7f0f480fd490 2024-02-13T00:49:28.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.220+0000 7f8f89597700 1 -- 172.21.15.162:0/506801796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f840fd470 msgr2=0x7f8f840f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:28.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.220+0000 7f8f89597700 1 --2- 172.21.15.162:0/506801796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f840fd470 0x7f8f840f6ca0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 rx=0x7f8f6c0047b0 tx=0x7f8f6c009cb0).stop 2024-02-13T00:49:28.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.220+0000 7f8f89597700 1 -- 172.21.15.162:0/506801796 shutdown_connections 2024-02-13T00:49:28.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.220+0000 7f8f89597700 1 --2- 172.21.15.162:0/506801796 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f840f7380 0x7f8f84103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.220+0000 7f8f89597700 1 --2- 172.21.15.162:0/506801796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f840fcb10 0x7f8f840fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.220+0000 7f8f89597700 1 --2- 172.21.15.162:0/506801796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f840fd470 0x7f8f840f6ca0 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.220+0000 7f8f89597700 1 -- 172.21.15.162:0/506801796 >> 172.21.15.162:0/506801796 conn(0x7f8f840f2740 msgr2=0x7f8f840f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:28.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.220+0000 7f8f89597700 1 -- 172.21.15.162:0/506801796 shutdown_connections 2024-02-13T00:49:28.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.220+0000 7f8f89597700 1 -- 172.21.15.162:0/506801796 wait complete. 2024-02-13T00:49:28.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f89597700 1 librados: starting msgr at 2024-02-13T00:49:28.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f89597700 1 librados: starting objecter 2024-02-13T00:49:28.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f89597700 1 Processor -- start 2024-02-13T00:49:28.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f89597700 1 -- start start 2024-02-13T00:49:28.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f89597700 1 librados: setting wanted keys 2024-02-13T00:49:28.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f89597700 1 librados: calling monclient init 2024-02-13T00:49:28.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f89597700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f840f7380 0x7f8f840f8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:28.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f89597700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f840fcb10 0x7f8f840f8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:28.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f89597700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f840fd470 0x7f8f840f92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:28.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f89597700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f84116600 con 0x7f8f840fcb10 2024-02-13T00:49:28.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f89597700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f84116770 con 0x7f8f840f7380 2024-02-13T00:49:28.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f89597700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f84116930 con 0x7f8f840fd470 2024-02-13T00:49:28.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f827fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f840fcb10 0x7f8f840f8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:28.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f827fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f840fcb10 0x7f8f840f8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:47840/0 (socket says 172.21.15.162:47840) 2024-02-13T00:49:28.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f827fc700 1 -- 172.21.15.162:0/2241850441 learned_addr learned my addr 172.21.15.162:0/2241850441 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f837fe700 1 --2- 172.21.15.162:0/2241850441 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f840fd470 0x7f8f840f92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.221+0000 7f8f82ffd700 1 --2- 172.21.15.162:0/2241850441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f840f7380 0x7f8f840f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f827fc700 1 -- 172.21.15.162:0/2241850441 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f840fd470 msgr2=0x7f8f840f92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f827fc700 1 --2- 172.21.15.162:0/2241850441 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f840fd470 0x7f8f840f92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f827fc700 1 -- 172.21.15.162:0/2241850441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f840f7380 msgr2=0x7f8f840f8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f827fc700 1 --2- 172.21.15.162:0/2241850441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f840f7380 0x7f8f840f8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f827fc700 1 -- 172.21.15.162:0/2241850441 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f74005040 con 0x7f8f840fcb10 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f82ffd700 1 --2- 172.21.15.162:0/2241850441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f840f7380 0x7f8f840f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f827fc700 1 --2- 172.21.15.162:0/2241850441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f840fcb10 0x7f8f840f8e00 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 rx=0x7f8f6c0044e0 tx=0x7f8f6c007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f7bfff700 10 client.?.objecter ms_handle_connect 0x7f8f840fcb10 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f7bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f7bfff700 1 -- 172.21.15.162:0/2241850441 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c018070 con 0x7f8f840fcb10 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f89597700 1 -- 172.21.15.162:0/2241850441 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c005040 con 0x7f8f840fcb10 2024-02-13T00:49:28.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f7bfff700 1 -- 172.21.15.162:0/2241850441 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f6c0093b0 con 0x7f8f840fcb10 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f7bfff700 1 -- 172.21.15.162:0/2241850441 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c01de40 con 0x7f8f840fcb10 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f89597700 10 client.27663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f89597700 1 -- 172.21.15.162:0/2241850441 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f84116e50 con 0x7f8f840fcb10 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.222+0000 7f8f89597700 1 librados: init done 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.223+0000 7f8f89597700 1 -- 172.21.15.162:0/2241850441 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f840607b0 con 0x7f8f840fcb10 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.223+0000 7f8f7bfff700 1 -- 172.21.15.162:0/2241850441 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f8f6c013ad0 con 0x7f8f840fcb10 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.224+0000 7f8f7bfff700 1 --2- 172.21.15.162:0/2241850441 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8f70062c90 0x7f8f70065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.224+0000 7f8f7bfff700 1 -- 172.21.15.162:0/2241850441 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(553..553 src has 449..553) v4 ==== 7669+0+0 (secure 0 0 0) 0x7f8f6c07e4c0 con 0x7f8f840fcb10 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.224+0000 7f8f7bfff700 10 client.27663.objecter ms_dispatch 0x7f8f84001550 osd_map(553..553 src has 449..553) v4 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.224+0000 7f8f7bfff700 3 client.27663.objecter handle_osd_map got epochs [553,553] > 0 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.224+0000 7f8f7bfff700 3 client.27663.objecter handle_osd_map decoding full epoch 553 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.224+0000 7f8f82ffd700 1 --2- 172.21.15.162:0/2241850441 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8f70062c90 0x7f8f70065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.224+0000 7f8f7bfff700 20 client.27663.objecter dump_active .. 0 homeless 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.224+0000 7f8f7bfff700 10 client.27663.objecter ms_handle_connect 0x7f8f70062c90 2024-02-13T00:49:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.224+0000 7f8f82ffd700 1 --2- 172.21.15.162:0/2241850441 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8f70062c90 0x7f8f70065150 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 rx=0x7f8f7c007ed0 tx=0x7f8f7c004d00).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:28.231 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:28.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.229+0000 7f8f7bfff700 1 -- 172.21.15.162:0/2241850441 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8f6c051b50 con 0x7f8f840fcb10 2024-02-13T00:49:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:28.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:28.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:28.362 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:28.384 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:28.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:28.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:28.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.419+0000 7f8f89597700 1 -- 172.21.15.162:0/2241850441 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f8f840f9da0 con 0x7f8f840fcb10 2024-02-13T00:49:28.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:28.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.494+0000 7f0f45ffb700 1 -- 172.21.15.162:0/2989219503 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v554) v1 ==== 113+0+0 (secure 0 0 0) 0x7f0f40021070 con 0x7f0f480fd490 2024-02-13T00:49:28.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.494+0000 7f8f7bfff700 1 -- 172.21.15.162:0/2241850441 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v554) v1 ==== 91+0+0 (secure 0 0 0) 0x7f8f6c051740 con 0x7f8f840fcb10 2024-02-13T00:49:28.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 10 librados: watch_flush enter 2024-02-13T00:49:28.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 10 librados: watch_flush exit 2024-02-13T00:49:28.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 20 client.27663.objecter shutdown clearing up homeless session... 2024-02-13T00:49:28.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 10 client.27663.objecter successfully canceled tick 2024-02-13T00:49:28.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 1 -- 172.21.15.162:0/2241850441 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8f70062c90 msgr2=0x7f8f70065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:28.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 1 --2- 172.21.15.162:0/2241850441 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8f70062c90 0x7f8f70065150 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 rx=0x7f8f7c007ed0 tx=0x7f8f7c004d00).stop 2024-02-13T00:49:28.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 1 -- 172.21.15.162:0/2241850441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f840fcb10 msgr2=0x7f8f840f8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:28.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 1 --2- 172.21.15.162:0/2241850441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f840fcb10 0x7f8f840f8e00 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 rx=0x7f8f6c0044e0 tx=0x7f8f6c007f90).stop 2024-02-13T00:49:28.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 1 -- 172.21.15.162:0/2241850441 shutdown_connections 2024-02-13T00:49:28.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 1 --2- 172.21.15.162:0/2241850441 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8f70062c90 0x7f8f70065150 secure :-1 s=CLOSED pgs=2957 cs=0 l=1 rev1=1 rx=0x7f8f7c007ed0 tx=0x7f8f7c004d00).stop 2024-02-13T00:49:28.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 1 --2- 172.21.15.162:0/2241850441 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f840fd470 0x7f8f840f92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 1 --2- 172.21.15.162:0/2241850441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f840f7380 0x7f8f840f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 1 --2- 172.21.15.162:0/2241850441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f840fcb10 0x7f8f840f8e00 secure :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 rx=0x7f8f6c0044e0 tx=0x7f8f6c007f90).stop 2024-02-13T00:49:28.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.498+0000 7f8f89597700 1 -- 172.21.15.162:0/2241850441 >> 172.21.15.162:0/2241850441 conn(0x7f8f840f2740 msgr2=0x7f8f840f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:28.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.500+0000 7f8f89597700 1 -- 172.21.15.162:0/2241850441 shutdown_connections 2024-02-13T00:49:28.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.502+0000 7f8f89597700 1 -- 172.21.15.162:0/2241850441 wait complete. 2024-02-13T00:49:28.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.501+0000 7f0f4ee7f700 10 librados: watch_flush enter 2024-02-13T00:49:28.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.501+0000 7f0f4ee7f700 10 librados: watch_flush exit 2024-02-13T00:49:28.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.501+0000 7f0f4ee7f700 20 client.37757.objecter shutdown clearing up homeless session... 2024-02-13T00:49:28.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.501+0000 7f0f4ee7f700 10 client.37757.objecter successfully canceled tick 2024-02-13T00:49:28.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.501+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/2989219503 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0f38062c90 msgr2=0x7f0f38065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:28.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.501+0000 7f0f4ee7f700 1 --2- 172.21.15.162:0/2989219503 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0f38062c90 0x7f0f38065150 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 rx=0x7f0f3c004c80 tx=0x7f0f3c00b040).stop 2024-02-13T00:49:28.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.501+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/2989219503 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f480fd490 msgr2=0x7f0f48107780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:28.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.501+0000 7f0f4ee7f700 1 --2- 172.21.15.162:0/2989219503 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f480fd490 0x7f0f48107780 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7f0f40004860 tx=0x7f0f40004c50).stop 2024-02-13T00:49:28.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.502+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/2989219503 shutdown_connections 2024-02-13T00:49:28.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.502+0000 7f0f4ee7f700 1 --2- 172.21.15.162:0/2989219503 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0f38062c90 0x7f0f38065150 unknown :-1 s=CLOSED pgs=2956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.502+0000 7f0f4ee7f700 1 --2- 172.21.15.162:0/2989219503 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f480fd490 0x7f0f48107780 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.502+0000 7f0f4ee7f700 1 --2- 172.21.15.162:0/2989219503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f480f73a0 0x7f0f48106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.502+0000 7f0f4ee7f700 1 --2- 172.21.15.162:0/2989219503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f480fcb30 0x7f0f48107290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:28.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.502+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/2989219503 >> 172.21.15.162:0/2989219503 conn(0x7f0f4806eb90 msgr2=0x7f0f4806fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:28.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.502+0000 7f8f89597700 1 librados: shutdown 2024-02-13T00:49:28.509 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:49:28.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.506+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/2989219503 shutdown_connections 2024-02-13T00:49:28.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.507+0000 7f0f4ee7f700 1 -- 172.21.15.162:0/2989219503 wait complete. 2024-02-13T00:49:28.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:28.507+0000 7f0f4ee7f700 1 librados: shutdown 2024-02-13T00:49:28.512 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.4 to 1 (10000) 2024-02-13T00:49:28.529 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:28.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:28.168620+0000 mon.c (mon.2) 436 : audit [INF] from='client.? 172.21.15.162:0/2989219503' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:49:28.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:28.169057+0000 mon.a (mon.0) 2934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:49:28.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:28.420740+0000 mon.a (mon.0) 2935 : audit [INF] from='client.? 172.21.15.162:0/2241850441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:49:28.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:28.489276+0000 mon.a (mon.0) 2936 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:28.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:28.494697+0000 mon.a (mon.0) 2937 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-13T00:49:28.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:28.494788+0000 mon.a (mon.0) 2938 : audit [INF] from='client.? 172.21.15.162:0/2241850441' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:49:28.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:28.494838+0000 mon.a (mon.0) 2939 : cluster [DBG] osdmap e554: 8 total, 8 up, 4 in 2024-02-13T00:49:28.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:28.500966+0000 mon.b (mon.1) 672 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:28.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:28.501454+0000 mon.a (mon.0) 2940 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:28.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:28.168620+0000 mon.c (mon.2) 436 : audit [INF] from='client.? 172.21.15.162:0/2989219503' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:49:28.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:28.169057+0000 mon.a (mon.0) 2934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:49:28.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:28.420740+0000 mon.a (mon.0) 2935 : audit [INF] from='client.? 172.21.15.162:0/2241850441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:49:28.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:28.489276+0000 mon.a (mon.0) 2936 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:28.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:28.494697+0000 mon.a (mon.0) 2937 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-13T00:49:28.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:28.494788+0000 mon.a (mon.0) 2938 : audit [INF] from='client.? 172.21.15.162:0/2241850441' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:49:28.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:28.494838+0000 mon.a (mon.0) 2939 : cluster [DBG] osdmap e554: 8 total, 8 up, 4 in 2024-02-13T00:49:28.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:28.500966+0000 mon.b (mon.1) 672 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:28.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:28.501454+0000 mon.a (mon.0) 2940 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:29.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:49:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:49:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:49:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:28.168620+0000 mon.c (mon.2) 436 : audit [INF] from='client.? 172.21.15.162:0/2989219503' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:49:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:28.169057+0000 mon.a (mon.0) 2934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:49:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:28.420740+0000 mon.a (mon.0) 2935 : audit [INF] from='client.? 172.21.15.162:0/2241850441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:49:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:28.489276+0000 mon.a (mon.0) 2936 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:28.494697+0000 mon.a (mon.0) 2937 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-13T00:49:29.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:28.494788+0000 mon.a (mon.0) 2938 : audit [INF] from='client.? 172.21.15.162:0/2241850441' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:49:29.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:28.494838+0000 mon.a (mon.0) 2939 : cluster [DBG] osdmap e554: 8 total, 8 up, 4 in 2024-02-13T00:49:29.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:28.500966+0000 mon.b (mon.1) 672 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:29.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:28.501454+0000 mon.a (mon.0) 2940 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:29.780 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:29.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:29.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:29.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:29.915 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:28.475190+0000 mgr.y (mgr.14544) 482 : cluster [DBG] pgmap v830: 196 pgs: 196 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:49:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:29.497286+0000 mon.a (mon.0) 2941 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:29.497363+0000 mon.a (mon.0) 2942 : cluster [DBG] osdmap e555: 8 total, 8 up, 4 in 2024-02-13T00:49:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:29.498437+0000 mon.b (mon.1) 673 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:49:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:29.501581+0000 mon.b (mon.1) 674 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:49:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:29.502017+0000 mon.a (mon.0) 2943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:49:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:28.475190+0000 mgr.y (mgr.14544) 482 : cluster [DBG] pgmap v830: 196 pgs: 196 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:49:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:29.497286+0000 mon.a (mon.0) 2941 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:29.497363+0000 mon.a (mon.0) 2942 : cluster [DBG] osdmap e555: 8 total, 8 up, 4 in 2024-02-13T00:49:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:29.498437+0000 mon.b (mon.1) 673 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:49:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:29.501581+0000 mon.b (mon.1) 674 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:49:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:29.502017+0000 mon.a (mon.0) 2943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:49:30.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:28.475190+0000 mgr.y (mgr.14544) 482 : cluster [DBG] pgmap v830: 196 pgs: 196 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:49:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:29.497286+0000 mon.a (mon.0) 2941 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:29.497363+0000 mon.a (mon.0) 2942 : cluster [DBG] osdmap e555: 8 total, 8 up, 4 in 2024-02-13T00:49:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:29.498437+0000 mon.b (mon.1) 673 : audit [DBG] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-13T00:49:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:29.501581+0000 mon.b (mon.1) 674 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:49:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:29.502017+0000 mon.a (mon.0) 2943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-13T00:49:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:29.049515+0000 osd.1 (osd.1) 233 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:49:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:29.052637+0000 osd.1 (osd.1) 234 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:49:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:29.914838+0000 osd.0 (osd.0) 209 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:49:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:29.916148+0000 osd.0 (osd.0) 210 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:49:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:30.497835+0000 mon.a (mon.0) 2944 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:49:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:30.497914+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e556: 8 total, 8 up, 4 in 2024-02-13T00:49:31.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:30.506290+0000 mon.b (mon.1) 675 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:49:31.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:30.506621+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:49:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:29.049515+0000 osd.1 (osd.1) 233 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:49:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:29.052637+0000 osd.1 (osd.1) 234 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:49:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:29.914838+0000 osd.0 (osd.0) 209 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:49:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:29.916148+0000 osd.0 (osd.0) 210 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:49:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:30.497835+0000 mon.a (mon.0) 2944 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:49:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:30.497914+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e556: 8 total, 8 up, 4 in 2024-02-13T00:49:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:30.506290+0000 mon.b (mon.1) 675 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:49:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:30.506621+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:49:31.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:29.049515+0000 osd.1 (osd.1) 233 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:49:31.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:29.052637+0000 osd.1 (osd.1) 234 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:49:31.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:29.914838+0000 osd.0 (osd.0) 209 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:49:31.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:29.916148+0000 osd.0 (osd.0) 210 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:49:31.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:30.497835+0000 mon.a (mon.0) 2944 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-13T00:49:31.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:30.497914+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e556: 8 total, 8 up, 4 in 2024-02-13T00:49:31.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:30.506290+0000 mon.b (mon.1) 675 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:49:31.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:30.506621+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-13T00:49:31.364 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:31.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:31.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:31.471 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:31.507 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:31.531 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:31.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:31.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:31.664 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:31.684 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:31.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:31.759 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 6 1 2024-02-13T00:49:31.788 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:30.083157+0000 osd.1 (osd.1) 235 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:49:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:30.086852+0000 osd.1 (osd.1) 236 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:49:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:30.476214+0000 mgr.y (mgr.14544) 483 : cluster [DBG] pgmap v833: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:31.498752+0000 mon.a (mon.0) 2947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:49:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:31.498850+0000 mon.a (mon.0) 2948 : cluster [DBG] osdmap e557: 8 total, 8 up, 4 in 2024-02-13T00:49:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:31.501388+0000 mon.b (mon.1) 676 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:49:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:31.501818+0000 mon.a (mon.0) 2949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:49:32.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:30.083157+0000 osd.1 (osd.1) 235 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:49:32.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:30.086852+0000 osd.1 (osd.1) 236 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:49:32.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:30.476214+0000 mgr.y (mgr.14544) 483 : cluster [DBG] pgmap v833: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:32.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:31.498752+0000 mon.a (mon.0) 2947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:49:32.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:31.498850+0000 mon.a (mon.0) 2948 : cluster [DBG] osdmap e557: 8 total, 8 up, 4 in 2024-02-13T00:49:32.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:31.501388+0000 mon.b (mon.1) 676 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:49:32.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:31.501818+0000 mon.a (mon.0) 2949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:49:32.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:30.083157+0000 osd.1 (osd.1) 235 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:49:32.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:30.086852+0000 osd.1 (osd.1) 236 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:49:32.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:30.476214+0000 mgr.y (mgr.14544) 483 : cluster [DBG] pgmap v833: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:32.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:31.498752+0000 mon.a (mon.0) 2947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-13T00:49:32.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:31.498850+0000 mon.a (mon.0) 2948 : cluster [DBG] osdmap e557: 8 total, 8 up, 4 in 2024-02-13T00:49:32.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:31.501388+0000 mon.b (mon.1) 676 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:49:32.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:31.501818+0000 mon.a (mon.0) 2949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-13T00:49:32.382 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:32.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:32.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:32.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:32.525 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:32.622 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:49:32.622 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:49:32.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:49:32.622 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:49:32.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.622 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24739.0:3 16.f 16:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-13T00:49:32.622 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:00.558130+0000", 2024-02-13T00:49:32.622 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 511.95209993399999, 2024-02-13T00:49:32.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.081761433, 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24739", 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2737601038", 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558130+0000", 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558130+0000", 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558128+0000", 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2910000000000002e-06 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558135+0000", 2024-02-13T00:49:32.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0115999999999999e-05 2024-02-13T00:49:32.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558145+0000", 2024-02-13T00:49:32.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2560000000000002e-06 2024-02-13T00:49:32.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558153+0000", 2024-02-13T00:49:32.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0846e-05 2024-02-13T00:49:32.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558214+0000", 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8210000000000008e-06 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:00.558223+0000", 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.017115333 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.575338+0000", 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122046 2024-02-13T00:49:32.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.575460+0000", 2024-02-13T00:49:32.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001568549 2024-02-13T00:49:32.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-02-13T00:49:32.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.577029+0000", 2024-02-13T00:49:32.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012430968000000001 2024-02-13T00:49:32.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:49:32.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589460+0000", 2024-02-13T00:49:32.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015196783 2024-02-13T00:49:32.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.604657+0000", 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034770550999999997 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.639427+0000", 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040860799999999997 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.639836+0000", 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5262e-05 2024-02-13T00:49:32.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.639891+0000", 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.081761433 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24763.0:2 23.d 23:b36015c1:test-rados-api-smithi162-84089-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.495169+0000", 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 511.01506047100003, 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22330878300000001, 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.630 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.631 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.631 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24763", 2024-02-13T00:49:32.631 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/90739862", 2024-02-13T00:49:32.631 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:49:32.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.631 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495169+0000", 2024-02-13T00:49:32.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495169+0000", 2024-02-13T00:49:32.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:49:32.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495167+0000", 2024-02-13T00:49:32.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1899999999999995e-06 2024-02-13T00:49:32.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495175+0000", 2024-02-13T00:49:32.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0474e-05 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495186+0000", 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9990000000000008e-06 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495194+0000", 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3149000000000002e-05 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495247+0000", 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.533e-06 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.495254+0000", 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031521518999999998 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.526775+0000", 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0169999999999996e-06 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.526783+0000", 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.170767906 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697551+0000", 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3193000000000003e-05 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697614+0000", 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8979000000000002e-05 2024-02-13T00:49:32.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,4", 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.697713+0000", 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012975179999999999 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.699011+0000", 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003049412 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:49:32.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.702060+0000", 2024-02-13T00:49:32.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016330172 2024-02-13T00:49:32.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.718390+0000", 2024-02-13T00:49:32.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8174999999999999e-05 2024-02-13T00:49:32.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.718438+0000", 2024-02-13T00:49:32.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9382999999999999e-05 2024-02-13T00:49:32.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.718478+0000", 2024-02-13T00:49:32.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22330878300000001 2024-02-13T00:49:32.640 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.640 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.640 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.641 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24767.0:2 14.1f 14:feb32ed4:test-rados-api-smithi162-84161-3::testobj:head [write 0~8] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:49:32.641 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.597458+0000", 2024-02-13T00:49:32.641 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 510.91277115999998, 2024-02-13T00:49:32.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.074387914999999999, 2024-02-13T00:49:32.641 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.641 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.641 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.641 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24767", 2024-02-13T00:49:32.641 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2472368574", 2024-02-13T00:49:32.641 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:49:32.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.642 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597458+0000", 2024-02-13T00:49:32.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597458+0000", 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597456+0000", 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.696e-06 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597466+0000", 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0883999999999999e-05 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597477+0000", 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7740000000000004e-06 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597485+0000", 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029994400000000001 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597785+0000", 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113497 2024-02-13T00:49:32.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.597898+0000", 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015973699999999999 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,6", 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.598058+0000", 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020561050000000001 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.618619+0000", 2024-02-13T00:49:32.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033024080000000002 2024-02-13T00:49:32.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:49:32.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.621921+0000", 2024-02-13T00:49:32.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049844082999999997 2024-02-13T00:49:32.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.671766+0000", 2024-02-13T00:49:32.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4239000000000001e-05 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.671810+0000", 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6464999999999997e-05 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.671846+0000", 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.074387914999999999 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.648 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:441 20.14 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head [setxattr a196 in=4b] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:49:32.648 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.493977+0000", 2024-02-13T00:49:32.648 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 510.01625207799998, 2024-02-13T00:49:32.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16342337300000001, 2024-02-13T00:49:32.648 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.648 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.648 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.648 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:49:32.648 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:49:32.648 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 441 2024-02-13T00:49:32.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.649 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.493977+0000", 2024-02-13T00:49:32.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.493977+0000", 2024-02-13T00:49:32.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:49:32.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.493975+0000", 2024-02-13T00:49:32.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3498e-05 2024-02-13T00:49:32.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.493988+0000", 2024-02-13T00:49:32.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5116999999999999e-05 2024-02-13T00:49:32.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494003+0000", 2024-02-13T00:49:32.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3450000000000006e-06 2024-02-13T00:49:32.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494012+0000", 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0686999999999998e-05 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494062+0000", 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6800999999999998e-05 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494149+0000", 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147002 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:49:32.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494296+0000", 2024-02-13T00:49:32.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001189708 2024-02-13T00:49:32.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:49:32.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.495486+0000", 2024-02-13T00:49:32.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000325693 2024-02-13T00:49:32.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.495812+0000", 2024-02-13T00:49:32.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.161531377 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.657343+0000", 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1778e-05 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.657375+0000", 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5942999999999999e-05 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.657401+0000", 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16342337300000001 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:1809 20.14 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head [setxattr a880 in=4b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:03.990416+0000", 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 508.51981335699998, 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055387081999999997, 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.656 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1809 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990416+0000", 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990416+0000", 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990415+0000", 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7190000000000004e-06 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990422+0000", 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5320000000000006e-06 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990430+0000", 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0110000000000003e-06 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.990437+0000", 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043852149999999996 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.994823+0000", 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7717000000000001e-05 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.994900+0000", 2024-02-13T00:49:32.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013310199999999999 2024-02-13T00:49:32.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-02-13T00:49:32.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.995033+0000", 2024-02-13T00:49:32.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089106199999999997 2024-02-13T00:49:32.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:49:32.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:03.995924+0000", 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047824802 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.043749+0000", 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00194904 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.045698+0000", 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8034e-05 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.045756+0000", 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6875000000000001e-05 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.045803+0000", 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055387081999999997 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14886.0:2067 20.1e e109/105 20:78e1aec5:test-rados-api-smithi162-84049-17::small:head v 109'3, mlcod=109'1)", 2024-02-13T00:49:32.662 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.582878+0000", 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 507.92735187800002, 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043476111999999997, 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582878+0000", 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582878+0000", 2024-02-13T00:49:32.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582876+0000", 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2574e-05 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582889+0000", 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5733999999999999e-05 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582905+0000", 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4379999999999998e-06 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582912+0000", 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9762999999999997e-05 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582972+0000", 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3986999999999999e-05 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.582996+0000", 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043311409000000002 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.626307+0000", 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6532000000000001e-05 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.626354+0000", 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043476111999999997 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.666 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2070 20.b 20:d3a22371:test-rados-api-smithi162-84049-17::big2:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.742315+0000", 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 507.76791454200003, 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.100851074, 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2070 2024-02-13T00:49:32.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742315+0000", 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742315+0000", 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742313+0000", 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011733019000000001 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754046+0000", 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3737e-05 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754070+0000", 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0747999999999999e-05 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754081+0000", 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1600000000000001e-05 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754132+0000", 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6428999999999994e-05 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754229+0000", 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106959 2024-02-13T00:49:32.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,6", 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.754336+0000", 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036365882000000002 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.790702+0000", 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.052175179000000002 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.842877+0000", 2024-02-13T00:49:32.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000231101 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.843108+0000", 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3439000000000003e-05 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.843141+0000", 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4810000000000001e-05 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.843166+0000", 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.100851074 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16275.0:35 66.8 66:110c41bf:test-rados-api-smithi162-84081-10::foo:head [write 128~128] snapc b=[b,a] ondisk+write+known_if_redirected e135)", 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:30.279104+0000", 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 482.23112502999999, 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044119924999999997, 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.673 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16275", 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/238911824", 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 35 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279104+0000", 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279104+0000", 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:49:32.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279103+0000", 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1609999999999997e-06 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279110+0000", 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9039999999999998e-06 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279119+0000", 2024-02-13T00:49:32.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8399999999999997e-06 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.279126+0000", 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026694828 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.305820+0000", 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2825999999999995e-05 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.305913+0000", 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018201799999999999 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.306095+0000", 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025887029999999999 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.308684+0000", 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013424475999999999 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.322109+0000", 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010389150000000001 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.323147+0000", 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0759999999999997e-05 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.323178+0000", 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6211000000000002e-05 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:30.323224+0000", 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044119924999999997 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16275.0:42 66.8 66:110c41bf:test-rados-api-smithi162-84081-10::foo:head [write 128~128] snapc c=[c,b,a] ondisk+write+known_if_redirected e136)", 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:31.308000+0000", 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 481.20222967299998, 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075133532000000003, 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.679 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16275", 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/238911824", 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 42 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308000+0000", 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308000+0000", 2024-02-13T00:49:32.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.307997+0000", 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5245999999999999e-05 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308012+0000", 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8859e-05 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308031+0000", 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0020999999999999e-05 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.308041+0000", 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014614509999999999 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.309503+0000", 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8609999999999998e-06 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:49:32.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.309511+0000", 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043170606 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.352681+0000", 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010319699999999999 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.352784+0000", 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016862800000000001 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.352953+0000", 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002530109 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.355483+0000", 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0083680739999999997 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.363851+0000", 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019179723999999999 2024-02-13T00:49:32.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383031+0000", 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6895000000000003e-05 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383088+0000", 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5457000000000002e-05 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:31.383133+0000", 2024-02-13T00:49:32.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075133532000000003 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.35941.0:2 84.6 e139/138 84:602f83fe:::foo:head v 139'1, mlcod=0'0)", 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:34.483127+0000", 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 478.02710217499998, 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041127036999999998, 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483127+0000", 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483127+0000", 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483127+0000", 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3149999999999999e-06 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483132+0000", 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0050000000000004e-06 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483137+0000", 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1290000000000001e-06 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.483143+0000", 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022933839999999999 2024-02-13T00:49:32.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.485437+0000", 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6535e-05 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.485453+0000", 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038765455999999997 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.524219+0000", 2024-02-13T00:49:32.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5552000000000001e-05 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.524254+0000", 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041127036999999998 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.43732.0:2 112.6 112:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e172)", 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:07.426136+0000", 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 445.08409323500001, 2024-02-13T00:49:32.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058639376999999999, 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.43732", 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/981068687", 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426136+0000", 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426136+0000", 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426134+0000", 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5360000000000004e-06 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426142+0000", 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0329999999999999e-05 2024-02-13T00:49:32.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426152+0000", 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5750000000000003e-06 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426160+0000", 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4138999999999998e-05 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426214+0000", 2024-02-13T00:49:32.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6590000000000007e-06 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.426222+0000", 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014917939 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.441140+0000", 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109622 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.441249+0000", 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015227799999999999 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.441401+0000", 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028880260000000001 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.444289+0000", 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033112480000000001 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:49:32.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.447601+0000", 2024-02-13T00:49:32.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037075416 2024-02-13T00:49:32.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:49:32.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.484676+0000", 2024-02-13T00:49:32.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2821999999999997e-05 2024-02-13T00:49:32.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.484729+0000", 2024-02-13T00:49:32.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6717e-05 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:07.484776+0000", 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058639376999999999 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36767.0:1 140.6s0 140:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:41.361654+0000", 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 411.14857557900001, 2024-02-13T00:49:32.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99434686900000002, 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36767", 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1509178592", 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361654+0000", 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361654+0000", 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361653+0000", 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2099999999999998e-06 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361659+0000", 2024-02-13T00:49:32.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1799999999999996e-06 2024-02-13T00:49:32.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361667+0000", 2024-02-13T00:49:32.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8100000000000003e-06 2024-02-13T00:49:32.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361673+0000", 2024-02-13T00:49:32.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0318000000000002e-05 2024-02-13T00:49:32.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361723+0000", 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5279999999999999e-06 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:41.361727+0000", 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98580988800000002 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.347536+0000", 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149586 2024-02-13T00:49:32.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.347686+0000", 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038543399999999998 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.348071+0000", 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0075773510000000004 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.355649+0000", 2024-02-13T00:49:32.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027853600000000002 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.355927+0000", 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3433e-05 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:42.356001+0000", 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99434686900000002 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.36824.0:1 148.6s1 148:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e220)", 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:42:55.339285+0000", 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 397.170944593, 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0318101019999999, 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.36824", 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4039137531", 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339285+0000", 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339285+0000", 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339283+0000", 2024-02-13T00:49:32.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8839999999999994e-06 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339291+0000", 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1019000000000001e-05 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339302+0000", 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6029999999999994e-06 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.339312+0000", 2024-02-13T00:49:32.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061423129999999999 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.345454+0000", 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3289999999999998e-06 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:55.345457+0000", 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.020285004 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.365742+0000", 2024-02-13T00:49:32.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137575 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.365880+0000", 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040855500000000001 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.366289+0000", 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051522999999999998 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.366804+0000", 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004224088 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.371028+0000", 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7168999999999993e-05 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:42:56.371095+0000", 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0318101019999999 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.709 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26868.0:1 170.7s1 170:fd8eaea7:nspace::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:43:35.394195+0000", 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 357.11603474499998, 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.199555962, 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26868", 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1602322330", 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.710 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394195+0000", 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394195+0000", 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394193+0000", 2024-02-13T00:49:32.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1409999999999999e-06 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394200+0000", 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1063000000000001e-05 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394211+0000", 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0160000000000008e-06 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394219+0000", 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4419999999999996e-05 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394284+0000", 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2720000000000002e-06 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:35.394287+0000", 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1938998459999999 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:36.588187+0000", 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011178799999999999 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:36.588298+0000", 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038829199999999999 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:36.588687+0000", 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050951100000000004 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:36.589196+0000", 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044877179999999999 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:36.593684+0000", 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6774999999999997e-05 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:43:36.593751+0000", 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.199555962 2024-02-13T00:49:32.715 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27057.0:1 190.5s2 190:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e319)", 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:34.404917+0000", 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 298.105312884, 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.331389991, 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27057", 2024-02-13T00:49:32.716 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4276929747", 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404917+0000", 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404917+0000", 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404915+0000", 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7039999999999997e-06 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404922+0000", 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0203999999999999e-05 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404932+0000", 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9459999999999998e-06 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404940+0000", 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022950620000000001 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.407235+0000", 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5379999999999998e-06 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.407238+0000", 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3289484380000001 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:35.736187+0000", 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011990899999999999 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:35.736307+0000", 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.331389991 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.720 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24329.0:1153 5.3 5:c609908c:::notify.5:head [watch ping cookie 94514175194752 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:39.296113+0000", 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 293.21411655399999, 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223829202, 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24329", 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1153 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296113+0000", 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296113+0000", 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296110+0000", 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3057000000000001e-05 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296123+0000", 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0857999999999999e-05 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296144+0000", 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1107999999999999e-05 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296155+0000", 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112284 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296268+0000", 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3674e-05 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296281+0000", 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.197784131 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.494065+0000", 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034288400000000001 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.494408+0000", 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025363974000000001 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.519772+0000", 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5661999999999994e-05 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.519858+0000", 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4176000000000002e-05 2024-02-13T00:49:32.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.519942+0000", 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223829202 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24329.0:1162 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94514175194752 gen 2] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:39.297175+0000", 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 293.21305461200001, 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22574032099999999, 2024-02-13T00:49:32.726 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24329", 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1162 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297175+0000", 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297175+0000", 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297174+0000", 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4430000000000002e-06 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297179+0000", 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0529e-05 2024-02-13T00:49:32.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297190+0000", 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2519999999999999e-06 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297195+0000", 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2176999999999999e-05 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297247+0000", 2024-02-13T00:49:32.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0348e-05 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297258+0000", 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19719434299999999 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.494452+0000", 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1651999999999999e-05 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-02-13T00:49:32.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.494514+0000", 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025459631 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.519973+0000", 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0639000000000002e-05 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.520014+0000", 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002901299 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.522915+0000", 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22574032099999999 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25162.0:4733 191.6 e335/323 191:602f83fe:::foo:head v 335'11, mlcod=333'9)", 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:50.443597+0000", 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 282.06663294499998, 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051008379, 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.732 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443597+0000", 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443597+0000", 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443596+0000", 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2199999999999992e-06 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443604+0000", 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3690000000000003e-06 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443611+0000", 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4969999999999998e-06 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443617+0000", 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7141999999999999e-05 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443714+0000", 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9049999999999999e-05 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443733+0000", 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050843781999999997 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.494577+0000", 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7654e-05 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.494605+0000", 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051008379 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.736 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24748.0:1139 18.4 18:20000000::::head [pgnls start_epoch 348] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e348)", 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:45:03.458723+0000", 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 269.05150603599998, 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94893485399999999, 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24748", 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2831164779", 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1139 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458723+0000", 2024-02-13T00:49:32.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458723+0000", 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458723+0000", 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9849999999999997e-06 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458728+0000", 2024-02-13T00:49:32.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1950000000000003e-06 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458736+0000", 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4649999999999996e-06 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458740+0000", 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9974999999999993e-05 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458810+0000", 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0842999999999999e-05 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458821+0000", 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94869083399999998 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:04.407512+0000", 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0757999999999998e-05 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:04.407553+0000", 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105543 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:04.407658+0000", 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94893485399999999 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.741 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.37256.0:1 197.2s1 197:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:45:05.418265+0000", 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 267.09196494700001, 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.004095116, 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.37256", 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1003782898", 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:49:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418265+0000", 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418265+0000", 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418262+0000", 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0108999999999999e-05 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418272+0000", 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1304000000000001e-05 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418283+0000", 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7669999999999992e-06 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418292+0000", 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00073741699999999998 2024-02-13T00:49:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.419030+0000", 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9870000000000001e-06 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.419034+0000", 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99819390399999997 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.417227+0000", 2024-02-13T00:49:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013077100000000001 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.417358+0000", 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00046316100000000002 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.417821+0000", 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041701299999999999 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.418238+0000", 2024-02-13T00:49:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040563220000000002 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.422295+0000", 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4969999999999996e-05 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.422360+0000", 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.004095116 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:49:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:49:32.748 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:49:33.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.016+0000 7faedbf8b700 1 -- 172.21.15.162:0/2642870257 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faed40f8500 msgr2=0x7faed4104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.016+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2642870257 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faed40f8500 0x7faed4104bd0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 rx=0x7faecc009c30 tx=0x7faecc004c50).stop 2024-02-13T00:49:33.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.016+0000 7faedbf8b700 1 -- 172.21.15.162:0/2642870257 shutdown_connections 2024-02-13T00:49:33.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.016+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2642870257 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faed40f8500 0x7faed4104bd0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.016+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2642870257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed40f9b70 0x7faed40f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.016+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2642870257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faed40fa440 0x7faed40f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.016+0000 7faedbf8b700 1 -- 172.21.15.162:0/2642870257 >> 172.21.15.162:0/2642870257 conn(0x7faed40f38c0 msgr2=0x7faed40f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:33.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 -- 172.21.15.162:0/2642870257 shutdown_connections 2024-02-13T00:49:33.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 -- 172.21.15.162:0/2642870257 wait complete. 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 librados: starting msgr at 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 librados: starting objecter 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 Processor -- start 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 -- start start 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 librados: setting wanted keys 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 librados: calling monclient init 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed40f8500 0x7faed410e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faed40f9b70 0x7faed410e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faed40fa440 0x7faed41081f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed41086e0 con 0x7faed40f8500 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed4108850 con 0x7faed40fa440 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faedbf8b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed41089c0 con 0x7faed40f9b70 2024-02-13T00:49:33.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.017+0000 7faed9d27700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed40f8500 0x7faed410e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faed9526700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faed40f9b70 0x7faed410e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faed9d27700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed40f8500 0x7faed410e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:47854/0 (socket says 172.21.15.162:47854) 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faed9d27700 1 -- 172.21.15.162:0/2862611407 learned_addr learned my addr 172.21.15.162:0/2862611407 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faeda528700 1 --2- 172.21.15.162:0/2862611407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faed40fa440 0x7faed41081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faed9d27700 1 -- 172.21.15.162:0/2862611407 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faed40f9b70 msgr2=0x7faed410e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faed9d27700 1 --2- 172.21.15.162:0/2862611407 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faed40f9b70 0x7faed410e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faed9d27700 1 -- 172.21.15.162:0/2862611407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faed40fa440 msgr2=0x7faed41081f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faed9d27700 1 --2- 172.21.15.162:0/2862611407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faed40fa440 0x7faed41081f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faed9d27700 1 -- 172.21.15.162:0/2862611407 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec4005040 con 0x7faed40f8500 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faed9526700 1 --2- 172.21.15.162:0/2862611407 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faed40f9b70 0x7faed410e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faed9d27700 1 --2- 172.21.15.162:0/2862611407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed40f8500 0x7faed410e010 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 rx=0x7faec4009400 tx=0x7faec4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faecaffd700 10 client.?.objecter ms_handle_connect 0x7faed40f8500 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faecaffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4007a40 con 0x7faed40f8500 2024-02-13T00:49:33.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc0097c0 con 0x7faed40f8500 2024-02-13T00:49:33.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faec400f040 con 0x7faed40f8500 2024-02-13T00:49:33.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4014a70 con 0x7faed40f8500 2024-02-13T00:49:33.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faedbf8b700 10 client.27669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:33.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed4114e80 con 0x7faed40f8500 2024-02-13T00:49:33.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.018+0000 7faedbf8b700 1 librados: init done 2024-02-13T00:49:33.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.019+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7faed41090d0 con 0x7faed40f8500 2024-02-13T00:49:33.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7faec4010530 con 0x7faed40f8500 2024-02-13T00:49:33.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faecaffd700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6800/2866534053 conn(0x7faec0062c90 0x7faec0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:33.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(558..558 src has 449..558) v4 ==== 8105+0+0 (secure 0 0 0) 0x7faec407d250 con 0x7faed40f8500 2024-02-13T00:49:33.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 osd_map(558..558 src has 449..558) v4 2024-02-13T00:49:33.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faecaffd700 3 client.27669.objecter handle_osd_map got epochs [558,558] > 0 2024-02-13T00:49:33.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faecaffd700 3 client.27669.objecter handle_osd_map decoding full epoch 558 2024-02-13T00:49:33.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faed9526700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6800/2866534053 conn(0x7faec0062c90 0x7faec0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:33.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faecaffd700 20 client.27669.objecter dump_active .. 0 homeless 2024-02-13T00:49:33.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v558) v1 ==== 54+0+16 (secure 0 0 0) 0x7faec407d730 con 0x7faed40f8500 2024-02-13T00:49:33.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faecaffd700 10 client.27669.objecter ms_handle_connect 0x7faec0062c90 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faed9526700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6800/2866534053 conn(0x7faec0062c90 0x7faec0065150 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 rx=0x7faed4109990 tx=0x7faed000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faedbf8b700 10 client.27669.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 1 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6801/2103743338 conn(0x7faed40ff070 0x7faed40ff470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed40475e0 osd=0 3 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed40475e0 osd=0 3 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed40475e0 osd=0 3 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 1 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed40475e0 osd=0 3 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 0 1 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed40475e0 osd=0 4 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faedbf8b700 10 client.27669.objecter _send_command 1 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.020+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faed41072c0 con 0x7faed40ff070 2024-02-13T00:49:33.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.021+0000 7faeda528700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6801/2103743338 conn(0x7faed40ff070 0x7faed40ff470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:33.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.021+0000 7faecaffd700 10 client.27669.objecter ms_handle_connect 0x7faed40ff070 2024-02-13T00:49:33.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.021+0000 7faeda528700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6801/2103743338 conn(0x7faed40ff070 0x7faed40ff470 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:33.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.021+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faed41072c0 con 0x7faed40ff070 2024-02-13T00:49:33.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.021+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:49:33.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.021+0000 7faecaffd700 10 client.27669.objecter _finish_command 1 = system:0 2024-02-13T00:49:33.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.021+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed40475e0 osd=0 4 2024-02-13T00:49:33.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.021+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 0 1 2024-02-13T00:49:33.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.036+0000 7faedbf8b700 10 client.27669.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:33.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.036+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 2 2024-02-13T00:49:33.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.036+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed40475e0 osd=0 3 2024-02-13T00:49:33.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.036+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed40475e0 osd=0 3 2024-02-13T00:49:33.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.036+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed40475e0 osd=0 3 2024-02-13T00:49:33.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.036+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 2 2024-02-13T00:49:33.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.036+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed40475e0 osd=0 3 2024-02-13T00:49:33.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.036+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 0 2 2024-02-13T00:49:33.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.036+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed40475e0 osd=0 4 2024-02-13T00:49:33.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.036+0000 7faedbf8b700 10 client.27669.objecter _send_command 2 2024-02-13T00:49:33.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.036+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faed4027410 con 0x7faed40ff070 2024-02-13T00:49:33.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faed4027410 con 0x7faed40ff070 2024-02-13T00:49:33.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:33.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faecaffd700 10 client.27669.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed40475e0 osd=0 4 2024-02-13T00:49:33.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 0 2 2024-02-13T00:49:33.041 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:49:33.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faedbf8b700 10 client.27669.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:33.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 3 2024-02-13T00:49:33.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6805/150599780 conn(0x7faed4059990 0x7faed4059dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:33.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4028760 osd=1 3 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4028760 osd=1 3 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4028760 osd=1 3 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 3 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed4028760 osd=1 3 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 1 3 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4028760 osd=1 4 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faedbf8b700 10 client.27669.objecter _send_command 3 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7faed411b790 con 0x7faed4059990 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faed9d27700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6805/150599780 conn(0x7faed4059990 0x7faed4059dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faecaffd700 10 client.27669.objecter ms_handle_connect 0x7faed4059990 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.038+0000 7faed9d27700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6805/150599780 conn(0x7faed4059990 0x7faed4059dd0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.039+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faed411b790 con 0x7faed4059990 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.039+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:49:33.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.039+0000 7faecaffd700 10 client.27669.objecter _finish_command 3 = system:0 2024-02-13T00:49:33.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.039+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed4028760 osd=1 4 2024-02-13T00:49:33.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.039+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 1 3 2024-02-13T00:49:33.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:31.069094+0000 osd.1 (osd.1) 237 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:49:33.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:31.072854+0000 osd.1 (osd.1) 238 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:49:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:32.501056+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:49:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:32.501129+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e558: 8 total, 8 up, 4 in 2024-02-13T00:49:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:32.531242+0000 mon.b (mon.1) 677 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:32.531655+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:32.532438+0000 mon.b (mon.1) 678 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-92"}]: dispatch 2024-02-13T00:49:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:32.532754+0000 mon.a (mon.0) 2953 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-92"}]: dispatch 2024-02-13T00:49:33.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.053+0000 7faedbf8b700 10 client.27669.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:33.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.053+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 4 2024-02-13T00:49:33.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.053+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4028760 osd=1 3 2024-02-13T00:49:33.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.053+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4028760 osd=1 3 2024-02-13T00:49:33.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.053+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4028760 osd=1 3 2024-02-13T00:49:33.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.053+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 4 2024-02-13T00:49:33.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.053+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed4028760 osd=1 3 2024-02-13T00:49:33.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.053+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 1 4 2024-02-13T00:49:33.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.053+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4028760 osd=1 4 2024-02-13T00:49:33.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.053+0000 7faedbf8b700 10 client.27669.objecter _send_command 4 2024-02-13T00:49:33.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.053+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faed4059400 con 0x7faed4059990 2024-02-13T00:49:33.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faed4059400 con 0x7faed4059990 2024-02-13T00:49:33.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:33.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faecaffd700 10 client.27669.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed4028760 osd=1 4 2024-02-13T00:49:33.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 1 4 2024-02-13T00:49:33.059 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:49:33.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faedbf8b700 10 client.27669.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:33.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 5 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6809/3689719376 conn(0x7faed40282c0 0x7faed411a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4059810 osd=2 3 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4059810 osd=2 3 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4059810 osd=2 3 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 5 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed4059810 osd=2 3 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 2 5 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4059810 osd=2 4 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faedbf8b700 10 client.27669.objecter _send_command 5 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7faed411a6a0 con 0x7faed40282c0 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faed9526700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6809/3689719376 conn(0x7faed40282c0 0x7faed411a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faecaffd700 10 client.27669.objecter ms_handle_connect 0x7faed40282c0 2024-02-13T00:49:33.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.055+0000 7faed9526700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6809/3689719376 conn(0x7faed40282c0 0x7faed411a1b0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:33.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.056+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faed411a6a0 con 0x7faed40282c0 2024-02-13T00:49:33.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.056+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:49:33.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.056+0000 7faecaffd700 10 client.27669.objecter _finish_command 5 = system:0 2024-02-13T00:49:33.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.056+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed4059810 osd=2 4 2024-02-13T00:49:33.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.056+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 2 5 2024-02-13T00:49:33.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.070+0000 7faedbf8b700 10 client.27669.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:33.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.070+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 6 2024-02-13T00:49:33.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.070+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4059810 osd=2 3 2024-02-13T00:49:33.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.070+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4059810 osd=2 3 2024-02-13T00:49:33.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.070+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4059810 osd=2 3 2024-02-13T00:49:33.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.070+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 6 2024-02-13T00:49:33.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.070+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed4059810 osd=2 3 2024-02-13T00:49:33.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.070+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 2 6 2024-02-13T00:49:33.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.070+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4059810 osd=2 4 2024-02-13T00:49:33.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.070+0000 7faedbf8b700 10 client.27669.objecter _send_command 6 2024-02-13T00:49:33.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.070+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faed4027820 con 0x7faed40282c0 2024-02-13T00:49:33.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faed4027820 con 0x7faed40282c0 2024-02-13T00:49:33.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:33.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faecaffd700 10 client.27669.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed4059810 osd=2 4 2024-02-13T00:49:33.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 2 6 2024-02-13T00:49:33.076 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:49:33.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faedbf8b700 10 client.27669.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:33.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 7 2024-02-13T00:49:33.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6813/604971011 conn(0x7faed411d910 0x7faed411fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:33.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4027bd0 osd=3 3 2024-02-13T00:49:33.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4027bd0 osd=3 3 2024-02-13T00:49:33.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4027bd0 osd=3 3 2024-02-13T00:49:33.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 7 2024-02-13T00:49:33.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed4027bd0 osd=3 3 2024-02-13T00:49:33.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 3 7 2024-02-13T00:49:33.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4027bd0 osd=3 4 2024-02-13T00:49:33.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faedbf8b700 10 client.27669.objecter _send_command 7 2024-02-13T00:49:33.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7faed411b5a0 con 0x7faed411d910 2024-02-13T00:49:33.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.072+0000 7faeda528700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6813/604971011 conn(0x7faed411d910 0x7faed411fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:33.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.073+0000 7faecaffd700 10 client.27669.objecter ms_handle_connect 0x7faed411d910 2024-02-13T00:49:33.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.073+0000 7faeda528700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6813/604971011 conn(0x7faed411d910 0x7faed411fd00 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:33.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.073+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faed411b5a0 con 0x7faed411d910 2024-02-13T00:49:33.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.073+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:49:33.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.073+0000 7faecaffd700 10 client.27669.objecter _finish_command 7 = system:0 2024-02-13T00:49:33.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.073+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed4027bd0 osd=3 4 2024-02-13T00:49:33.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.073+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 3 7 2024-02-13T00:49:33.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:31.069094+0000 osd.1 (osd.1) 237 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:49:33.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:31.072854+0000 osd.1 (osd.1) 238 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:49:33.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:32.501056+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:49:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:32.501129+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e558: 8 total, 8 up, 4 in 2024-02-13T00:49:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:32.531242+0000 mon.b (mon.1) 677 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:32.531655+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:32.532438+0000 mon.b (mon.1) 678 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-92"}]: dispatch 2024-02-13T00:49:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:32.532754+0000 mon.a (mon.0) 2953 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-92"}]: dispatch 2024-02-13T00:49:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:31.069094+0000 osd.1 (osd.1) 237 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:49:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:31.072854+0000 osd.1 (osd.1) 238 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:49:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:32.501056+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-92","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-13T00:49:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:32.501129+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e558: 8 total, 8 up, 4 in 2024-02-13T00:49:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:32.531242+0000 mon.b (mon.1) 677 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:32.531655+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-6"}]: dispatch 2024-02-13T00:49:33.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:32.532438+0000 mon.b (mon.1) 678 : audit [INF] from='client.? 172.21.15.162:0/868737482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-92"}]: dispatch 2024-02-13T00:49:33.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:32.532754+0000 mon.a (mon.0) 2953 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-6", "tierpool": "test-rados-api-smithi162-84057-92"}]: dispatch 2024-02-13T00:49:33.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.088+0000 7faedbf8b700 10 client.27669.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:33.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.088+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 8 2024-02-13T00:49:33.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.088+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4027bd0 osd=3 3 2024-02-13T00:49:33.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.088+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4027bd0 osd=3 3 2024-02-13T00:49:33.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.088+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4027bd0 osd=3 3 2024-02-13T00:49:33.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.088+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 8 2024-02-13T00:49:33.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.088+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed4027bd0 osd=3 3 2024-02-13T00:49:33.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.088+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 3 8 2024-02-13T00:49:33.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.088+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4027bd0 osd=3 4 2024-02-13T00:49:33.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.088+0000 7faedbf8b700 10 client.27669.objecter _send_command 8 2024-02-13T00:49:33.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.088+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faed411ab00 con 0x7faed411d910 2024-02-13T00:49:33.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faed411ab00 con 0x7faed411d910 2024-02-13T00:49:33.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:33.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faecaffd700 10 client.27669.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed4027bd0 osd=3 4 2024-02-13T00:49:33.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 3 8 2024-02-13T00:49:33.093 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:49:33.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faedbf8b700 10 client.27669.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 9 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6800/3344263388 conn(0x7faed411ac70 0x7faed4123380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed411ab00 osd=4 3 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed411ab00 osd=4 3 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed411ab00 osd=4 3 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 9 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed411ab00 osd=4 3 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 4 9 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed411ab00 osd=4 4 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faedbf8b700 10 client.27669.objecter _send_command 9 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.089+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7faed4119e10 con 0x7faed411ac70 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.090+0000 7faed9d27700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6800/3344263388 conn(0x7faed411ac70 0x7faed4123380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:33.094 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.090+0000 7faecaffd700 10 client.27669.objecter ms_handle_connect 0x7faed411ac70 2024-02-13T00:49:33.095 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.090+0000 7faed9d27700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6800/3344263388 conn(0x7faed411ac70 0x7faed4123380 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:33.095 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.091+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faed4119e10 con 0x7faed411ac70 2024-02-13T00:49:33.095 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.091+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:49:33.095 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.091+0000 7faecaffd700 10 client.27669.objecter _finish_command 9 = system:0 2024-02-13T00:49:33.095 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.091+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed411ab00 osd=4 4 2024-02-13T00:49:33.095 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.091+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 4 9 2024-02-13T00:49:33.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:49:33.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.105+0000 7faedbf8b700 10 client.27669.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:33.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.105+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 10 2024-02-13T00:49:33.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.105+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed411ab00 osd=4 3 2024-02-13T00:49:33.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.105+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed411ab00 osd=4 3 2024-02-13T00:49:33.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.105+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed411ab00 osd=4 3 2024-02-13T00:49:33.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.105+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 10 2024-02-13T00:49:33.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.105+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed411ab00 osd=4 3 2024-02-13T00:49:33.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.105+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 4 10 2024-02-13T00:49:33.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.105+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed411ab00 osd=4 4 2024-02-13T00:49:33.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.105+0000 7faedbf8b700 10 client.27669.objecter _send_command 10 2024-02-13T00:49:33.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.105+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faed4119e10 con 0x7faed411ac70 2024-02-13T00:49:33.112 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:49:33.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faed4119e10 con 0x7faed411ac70 2024-02-13T00:49:33.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:33.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faecaffd700 10 client.27669.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed411ab00 osd=4 4 2024-02-13T00:49:33.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 4 10 2024-02-13T00:49:33.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faedbf8b700 10 client.27669.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:33.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 11 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6804/2277917372 conn(0x7faed4123d80 0x7faed4126170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4119e10 osd=5 3 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4119e10 osd=5 3 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4119e10 osd=5 3 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 11 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed4119e10 osd=5 3 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 5 11 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4119e10 osd=5 4 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faedbf8b700 10 client.27669.objecter _send_command 11 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.107+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7faed4128e80 con 0x7faed4123d80 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.108+0000 7faed9526700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6804/2277917372 conn(0x7faed4123d80 0x7faed4126170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.108+0000 7faed9526700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6804/2277917372 conn(0x7faed4123d80 0x7faed4126170 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.108+0000 7faecaffd700 10 client.27669.objecter ms_handle_connect 0x7faed4123d80 2024-02-13T00:49:33.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.109+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faed4128e80 con 0x7faed4123d80 2024-02-13T00:49:33.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.109+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:49:33.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.109+0000 7faecaffd700 10 client.27669.objecter _finish_command 11 = system:0 2024-02-13T00:49:33.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.109+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed4119e10 osd=5 4 2024-02-13T00:49:33.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.109+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 5 11 2024-02-13T00:49:33.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.126+0000 7faedbf8b700 10 client.27669.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:33.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.126+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 12 2024-02-13T00:49:33.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.126+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4119e10 osd=5 3 2024-02-13T00:49:33.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.126+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4119e10 osd=5 3 2024-02-13T00:49:33.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.126+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4119e10 osd=5 3 2024-02-13T00:49:33.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.126+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 12 2024-02-13T00:49:33.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.126+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed4119e10 osd=5 3 2024-02-13T00:49:33.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.126+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 5 12 2024-02-13T00:49:33.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.126+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4119e10 osd=5 4 2024-02-13T00:49:33.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.126+0000 7faedbf8b700 10 client.27669.objecter _send_command 12 2024-02-13T00:49:33.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.126+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faed4128e80 con 0x7faed4123d80 2024-02-13T00:49:33.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.128+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faed4128e80 con 0x7faed4123d80 2024-02-13T00:49:33.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.128+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:33.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.128+0000 7faecaffd700 10 client.27669.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.128+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed4119e10 osd=5 4 2024-02-13T00:49:33.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.128+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 5 12 2024-02-13T00:49:33.132 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:49:33.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faedbf8b700 10 client.27669.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:33.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 13 2024-02-13T00:49:33.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6808/1506640341 conn(0x7faed41281c0 0x7faed4128620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:33.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4128e80 osd=6 3 2024-02-13T00:49:33.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4128e80 osd=6 3 2024-02-13T00:49:33.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4128e80 osd=6 3 2024-02-13T00:49:33.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 13 2024-02-13T00:49:33.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed4128e80 osd=6 3 2024-02-13T00:49:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 6 13 2024-02-13T00:49:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4128e80 osd=6 4 2024-02-13T00:49:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faedbf8b700 10 client.27669.objecter _send_command 13 2024-02-13T00:49:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7faed4128c90 con 0x7faed41281c0 2024-02-13T00:49:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faeda528700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6808/1506640341 conn(0x7faed41281c0 0x7faed4128620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faecaffd700 10 client.27669.objecter ms_handle_connect 0x7faed41281c0 2024-02-13T00:49:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.129+0000 7faeda528700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6808/1506640341 conn(0x7faed41281c0 0x7faed4128620 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.130+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faecc018040 con 0x7faed41281c0 2024-02-13T00:49:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.130+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:49:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.130+0000 7faecaffd700 10 client.27669.objecter _finish_command 13 = system:0 2024-02-13T00:49:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.130+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed4128e80 osd=6 4 2024-02-13T00:49:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.130+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 6 13 2024-02-13T00:49:33.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.145+0000 7faedbf8b700 10 client.27669.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:33.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.145+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 14 2024-02-13T00:49:33.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.145+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4128e80 osd=6 3 2024-02-13T00:49:33.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.145+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4128e80 osd=6 3 2024-02-13T00:49:33.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.145+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed4128e80 osd=6 3 2024-02-13T00:49:33.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.145+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 14 2024-02-13T00:49:33.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.145+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed4128e80 osd=6 3 2024-02-13T00:49:33.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.145+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 6 14 2024-02-13T00:49:33.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.145+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4128e80 osd=6 4 2024-02-13T00:49:33.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.145+0000 7faedbf8b700 10 client.27669.objecter _send_command 14 2024-02-13T00:49:33.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.145+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faed412b130 con 0x7faed41281c0 2024-02-13T00:49:33.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faed412b130 con 0x7faed41281c0 2024-02-13T00:49:33.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:33.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faecaffd700 10 client.27669.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed4128e80 osd=6 4 2024-02-13T00:49:33.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 6 14 2024-02-13T00:49:33.151 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:49:33.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faedbf8b700 10 client.27669.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:33.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 15 2024-02-13T00:49:33.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6812/1964192772 conn(0x7faed412b7e0 0x7faed412bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:33.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed412b130 osd=7 3 2024-02-13T00:49:33.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed412b130 osd=7 3 2024-02-13T00:49:33.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed412b130 osd=7 3 2024-02-13T00:49:33.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 15 2024-02-13T00:49:33.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed412b130 osd=7 3 2024-02-13T00:49:33.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 7 15 2024-02-13T00:49:33.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed412b130 osd=7 4 2024-02-13T00:49:33.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faedbf8b700 10 client.27669.objecter _send_command 15 2024-02-13T00:49:33.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.147+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7faed412f4f0 con 0x7faed412b7e0 2024-02-13T00:49:33.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.148+0000 7faed9d27700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6812/1964192772 conn(0x7faed412b7e0 0x7faed412bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:33.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.148+0000 7faed9d27700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6812/1964192772 conn(0x7faed412b7e0 0x7faed412bc40 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:33.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.148+0000 7faecaffd700 10 client.27669.objecter ms_handle_connect 0x7faed412b7e0 2024-02-13T00:49:33.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.149+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faed412f4f0 con 0x7faed412b7e0 2024-02-13T00:49:33.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.149+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:49:33.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.149+0000 7faecaffd700 10 client.27669.objecter _finish_command 15 = system:0 2024-02-13T00:49:33.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.149+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed412b130 osd=7 4 2024-02-13T00:49:33.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.149+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 7 15 2024-02-13T00:49:33.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.163+0000 7faedbf8b700 10 client.27669.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:33.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.163+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign -1 16 2024-02-13T00:49:33.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.163+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed412b130 osd=7 3 2024-02-13T00:49:33.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.163+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed412b130 osd=7 3 2024-02-13T00:49:33.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.163+0000 7faedbf8b700 20 client.27669.objecter _get_session s=0x7faed412b130 osd=7 3 2024-02-13T00:49:33.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.163+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_remove -1 16 2024-02-13T00:49:33.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.163+0000 7faedbf8b700 20 client.27669.objecter get_session s=0x7faed412b130 osd=7 3 2024-02-13T00:49:33.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.163+0000 7faedbf8b700 15 client.27669.objecter _session_command_op_assign 7 16 2024-02-13T00:49:33.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.163+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed412b130 osd=7 4 2024-02-13T00:49:33.167 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.163+0000 7faedbf8b700 10 client.27669.objecter _send_command 16 2024-02-13T00:49:33.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.163+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faed412e370 con 0x7faed412b7e0 2024-02-13T00:49:33.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faecaffd700 1 -- 172.21.15.162:0/2862611407 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faed412e370 con 0x7faed412b7e0 2024-02-13T00:49:33.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faecaffd700 10 client.27669.objecter ms_dispatch 0x7faed4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:33.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faecaffd700 10 client.27669.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faecaffd700 20 client.27669.objecter put_session s=0x7faed412b130 osd=7 4 2024-02-13T00:49:33.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faecaffd700 15 client.27669.objecter _session_command_op_remove 7 16 2024-02-13T00:49:33.169 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 10 librados: watch_flush enter 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 10 librados: watch_flush exit 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 10 client.27669.objecter close_session for osd.0 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6801/2103743338 conn(0x7faed40ff070 msgr2=0x7faed40ff470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6801/2103743338 conn(0x7faed40ff070 0x7faed40ff470 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed40475e0 osd=0 1 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 10 client.27669.objecter close_session for osd.1 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6805/150599780 conn(0x7faed4059990 msgr2=0x7faed4059dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6805/150599780 conn(0x7faed4059990 0x7faed4059dd0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4028760 osd=1 1 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 10 client.27669.objecter close_session for osd.2 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6809/3689719376 conn(0x7faed40282c0 msgr2=0x7faed411a1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6809/3689719376 conn(0x7faed40282c0 0x7faed411a1b0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4059810 osd=2 1 2024-02-13T00:49:33.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 10 client.27669.objecter close_session for osd.3 2024-02-13T00:49:33.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6813/604971011 conn(0x7faed411d910 msgr2=0x7faed411fd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6813/604971011 conn(0x7faed411d910 0x7faed411fd00 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faeda528700 1 -- 172.21.15.162:0/2862611407 reap_dead start 2024-02-13T00:49:33.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4027bd0 osd=3 1 2024-02-13T00:49:33.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 10 client.27669.objecter close_session for osd.4 2024-02-13T00:49:33.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6800/3344263388 conn(0x7faed411ac70 msgr2=0x7faed4123380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6800/3344263388 conn(0x7faed411ac70 0x7faed4123380 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed411ab00 osd=4 1 2024-02-13T00:49:33.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 10 client.27669.objecter close_session for osd.5 2024-02-13T00:49:33.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6804/2277917372 conn(0x7faed4123d80 msgr2=0x7faed4126170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6804/2277917372 conn(0x7faed4123d80 0x7faed4126170 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4119e10 osd=5 1 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 10 client.27669.objecter close_session for osd.6 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6808/1506640341 conn(0x7faed41281c0 msgr2=0x7faed4128620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6808/1506640341 conn(0x7faed41281c0 0x7faed4128620 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed4128e80 osd=6 1 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 10 client.27669.objecter close_session for osd.7 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6812/1964192772 conn(0x7faed412b7e0 msgr2=0x7faed412bc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.181:6812/1964192772 conn(0x7faed412b7e0 0x7faed412bc40 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 20 client.27669.objecter put_session s=0x7faed412b130 osd=7 1 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 20 client.27669.objecter shutdown clearing up homeless session... 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.165+0000 7faedbf8b700 10 client.27669.objecter successfully canceled tick 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6800/2866534053 conn(0x7faec0062c90 msgr2=0x7faec0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> v2:172.21.15.162:6800/2866534053 conn(0x7faec0062c90 0x7faec0065150 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 rx=0x7faed4109990 tx=0x7faed000b040).stop 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed40f8500 msgr2=0x7faed410e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faedbf8b700 1 --2- 172.21.15.162:0/2862611407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faed40f8500 0x7faed410e010 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 rx=0x7faec4009400 tx=0x7faec4009cb0).stop 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faeda528700 1 -- 172.21.15.162:0/2862611407 reap_dead start 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faeda528700 1 -- 172.21.15.162:0/2862611407 reap_dead start 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faeda528700 1 -- 172.21.15.162:0/2862611407 reap_dead start 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faeda528700 1 -- 172.21.15.162:0/2862611407 reap_dead start 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 shutdown_connections 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 >> 172.21.15.162:0/2862611407 conn(0x7faed40f38c0 msgr2=0x7faed40f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 shutdown_connections 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faedbf8b700 1 -- 172.21.15.162:0/2862611407 wait complete. 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:33.166+0000 7faedbf8b700 1 librados: shutdown 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.174 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.175 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.175 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.175 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.175 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.175 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.175 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:49:33.673 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-13T00:49:33.966 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:33.975 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:33.977 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:33.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:32.476546+0000 mgr.y (mgr.14544) 484 : cluster [DBG] pgmap v836: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:33.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:33.020435+0000 mon.a (mon.0) 2954 : audit [DBG] from='client.? 172.21.15.162:0/2862611407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:33.983 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:33.502272+0000 mon.a (mon.0) 2955 : cluster [DBG] osdmap e559: 8 total, 8 up, 4 in 2024-02-13T00:49:33.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:32.476546+0000 mgr.y (mgr.14544) 484 : cluster [DBG] pgmap v836: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:33.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:33.020435+0000 mon.a (mon.0) 2954 : audit [DBG] from='client.? 172.21.15.162:0/2862611407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:33.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:33.502272+0000 mon.a (mon.0) 2955 : cluster [DBG] osdmap e559: 8 total, 8 up, 4 in 2024-02-13T00:49:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:33.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:32.476546+0000 mgr.y (mgr.14544) 484 : cluster [DBG] pgmap v836: 228 pgs: 228 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:33.020435+0000 mon.a (mon.0) 2954 : audit [DBG] from='client.? 172.21.15.162:0/2862611407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:33.502272+0000 mon.a (mon.0) 2955 : cluster [DBG] osdmap e559: 8 total, 8 up, 4 in 2024-02-13T00:49:34.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:34.114 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:35.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:33.037457+0000 osd.1 (osd.1) 239 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:49:35.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:33.040508+0000 osd.1 (osd.1) 240 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:49:35.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:34.508635+0000 mon.a (mon.0) 2956 : cluster [DBG] osdmap e560: 8 total, 8 up, 4 in 2024-02-13T00:49:35.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:34.527893+0000 mon.c (mon.2) 437 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:35.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:34.528174+0000 mon.a (mon.0) 2957 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:35.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:34.528876+0000 mon.c (mon.2) 438 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:35.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:34.529103+0000 mon.a (mon.0) 2958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:35.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:34.529478+0000 mon.c (mon.2) 439 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:49:35.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:34.529704+0000 mon.a (mon.0) 2959 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:49:35.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:33.037457+0000 osd.1 (osd.1) 239 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:49:35.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:33.040508+0000 osd.1 (osd.1) 240 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:49:35.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:34.508635+0000 mon.a (mon.0) 2956 : cluster [DBG] osdmap e560: 8 total, 8 up, 4 in 2024-02-13T00:49:35.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:34.527893+0000 mon.c (mon.2) 437 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:35.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:34.528174+0000 mon.a (mon.0) 2957 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:35.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:34.528876+0000 mon.c (mon.2) 438 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:35.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:34.529103+0000 mon.a (mon.0) 2958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:35.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:34.529478+0000 mon.c (mon.2) 439 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:49:35.039 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:34.529704+0000 mon.a (mon.0) 2959 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:49:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:33.037457+0000 osd.1 (osd.1) 239 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:49:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:33.040508+0000 osd.1 (osd.1) 240 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:49:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:34.508635+0000 mon.a (mon.0) 2956 : cluster [DBG] osdmap e560: 8 total, 8 up, 4 in 2024-02-13T00:49:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:34.527893+0000 mon.c (mon.2) 437 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:34.528174+0000 mon.a (mon.0) 2957 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:35.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:34.528876+0000 mon.c (mon.2) 438 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:35.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:34.529103+0000 mon.a (mon.0) 2958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:35.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:34.529478+0000 mon.c (mon.2) 439 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:49:35.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:34.529704+0000 mon.a (mon.0) 2959 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:49:36.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:49:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:49:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:34.477106+0000 mgr.y (mgr.14544) 485 : cluster [DBG] pgmap v839: 196 pgs: 196 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:35.098305+0000 osd.6 (osd.6) 165 : cluster [DBG] 28.6 scrub starts 2024-02-13T00:49:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:35.101290+0000 osd.6 (osd.6) 166 : cluster [DBG] 28.6 scrub ok 2024-02-13T00:49:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:35.486522+0000 mon.a (mon.0) 2960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:49:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:35.486600+0000 mon.a (mon.0) 2961 : cluster [DBG] osdmap e561: 8 total, 8 up, 4 in 2024-02-13T00:49:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:35.487833+0000 mon.c (mon.2) 440 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-84057-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:35.491894+0000 mon.a (mon.0) 2962 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-84057-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:34.477106+0000 mgr.y (mgr.14544) 485 : cluster [DBG] pgmap v839: 196 pgs: 196 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:35.098305+0000 osd.6 (osd.6) 165 : cluster [DBG] 28.6 scrub starts 2024-02-13T00:49:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:35.101290+0000 osd.6 (osd.6) 166 : cluster [DBG] 28.6 scrub ok 2024-02-13T00:49:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:35.486522+0000 mon.a (mon.0) 2960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:49:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:35.486600+0000 mon.a (mon.0) 2961 : cluster [DBG] osdmap e561: 8 total, 8 up, 4 in 2024-02-13T00:49:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:35.487833+0000 mon.c (mon.2) 440 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-84057-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:35.491894+0000 mon.a (mon.0) 2962 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-84057-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:36.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:34.477106+0000 mgr.y (mgr.14544) 485 : cluster [DBG] pgmap v839: 196 pgs: 196 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:36.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:35.098305+0000 osd.6 (osd.6) 165 : cluster [DBG] 28.6 scrub starts 2024-02-13T00:49:36.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:35.101290+0000 osd.6 (osd.6) 166 : cluster [DBG] 28.6 scrub ok 2024-02-13T00:49:36.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:35.486522+0000 mon.a (mon.0) 2960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:49:36.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:35.486600+0000 mon.a (mon.0) 2961 : cluster [DBG] osdmap e561: 8 total, 8 up, 4 in 2024-02-13T00:49:36.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:35.487833+0000 mon.c (mon.2) 440 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-84057-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:36.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:35.491894+0000 mon.a (mon.0) 2962 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-84057-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:49:36.263 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:36.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:36.273 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:36.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:49:36.398 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:36.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:49:36.458 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:49:36] "GET /metrics HTTP/1.1" 200 185774 "" "Prometheus/2.33.4" 2024-02-13T00:49:36.547 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:36.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:36.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:49:36.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:36.685 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:36.706 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:36.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:36.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:36.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.765+0000 7f9ffc860700 1 -- 172.21.15.162:0/1780037595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ff80f73a0 msgr2=0x7f9ff81039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:36.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.765+0000 7f9ffc860700 1 --2- 172.21.15.162:0/1780037595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ff80f73a0 0x7f9ff81039a0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7f9ff0009c30 tx=0x7f9ff0004c50).stop 2024-02-13T00:49:36.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.765+0000 7f9ffc860700 1 -- 172.21.15.162:0/1780037595 shutdown_connections 2024-02-13T00:49:36.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.765+0000 7f9ffc860700 1 --2- 172.21.15.162:0/1780037595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ff80f73a0 0x7f9ff81039a0 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:36.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.765+0000 7f9ffc860700 1 --2- 172.21.15.162:0/1780037595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ff80fcb30 0x7f9ff80fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:36.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.765+0000 7f9ffc860700 1 --2- 172.21.15.162:0/1780037595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ff80fd490 0x7f9ff80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:36.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.765+0000 7f9ffc860700 1 -- 172.21.15.162:0/1780037595 >> 172.21.15.162:0/1780037595 conn(0x7f9ff806eb90 msgr2=0x7f9ff80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:36.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.765+0000 7f9ffc860700 1 -- 172.21.15.162:0/1780037595 shutdown_connections 2024-02-13T00:49:36.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.765+0000 7f9ffc860700 1 -- 172.21.15.162:0/1780037595 wait complete. 2024-02-13T00:49:36.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.765+0000 7f9ffc860700 1 librados: starting msgr at 2024-02-13T00:49:36.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.765+0000 7f9ffc860700 1 librados: starting objecter 2024-02-13T00:49:36.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ffc860700 1 Processor -- start 2024-02-13T00:49:36.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ffc860700 1 -- start start 2024-02-13T00:49:36.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ffc860700 1 librados: setting wanted keys 2024-02-13T00:49:36.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ffc860700 1 librados: calling monclient init 2024-02-13T00:49:36.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ffc860700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ff80f73a0 0x7f9ff8109020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:36.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ffc860700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ff80fcb30 0x7f9ff8109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:36.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ffc860700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ff80fd490 0x7f9ff8109b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:36.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ffc860700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff8114280 con 0x7f9ff80fcb30 2024-02-13T00:49:36.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ffc860700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff81143f0 con 0x7f9ff80f73a0 2024-02-13T00:49:36.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ffc860700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff81145b0 con 0x7f9ff80fd490 2024-02-13T00:49:36.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ff5d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ff80fcb30 0x7f9ff8109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:36.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ff5d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ff80fcb30 0x7f9ff8109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:58734/0 (socket says 172.21.15.162:58734) 2024-02-13T00:49:36.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ff6d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ff80fd490 0x7f9ff8109b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:36.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ff5d9b700 1 -- 172.21.15.162:0/2100317237 learned_addr learned my addr 172.21.15.162:0/2100317237 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:36.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.766+0000 7f9ff659c700 1 --2- 172.21.15.162:0/2100317237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ff80f73a0 0x7f9ff8109020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9ff5d9b700 1 -- 172.21.15.162:0/2100317237 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ff80fd490 msgr2=0x7f9ff8109b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9ff5d9b700 1 --2- 172.21.15.162:0/2100317237 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ff80fd490 0x7f9ff8109b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9ff5d9b700 1 -- 172.21.15.162:0/2100317237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ff80f73a0 msgr2=0x7f9ff8109020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9ff5d9b700 1 --2- 172.21.15.162:0/2100317237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ff80f73a0 0x7f9ff8109020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9ff5d9b700 1 -- 172.21.15.162:0/2100317237 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe8005040 con 0x7f9ff80fcb30 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9ff5d9b700 1 --2- 172.21.15.162:0/2100317237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ff80fcb30 0x7f9ff8109510 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7f9fec0041d0 tx=0x7f9fec009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9fe77fe700 10 client.?.objecter ms_handle_connect 0x7f9ff80fcb30 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9fe77fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9fe77fe700 1 -- 172.21.15.162:0/2100317237 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec007b40 con 0x7f9ff80fcb30 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9ffc860700 1 -- 172.21.15.162:0/2100317237 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff00097c0 con 0x7f9ff80fcb30 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9fe77fe700 1 -- 172.21.15.162:0/2100317237 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fec00f040 con 0x7f9ff80fcb30 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9fe77fe700 1 -- 172.21.15.162:0/2100317237 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec014730 con 0x7f9ff80fcb30 2024-02-13T00:49:36.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9ffc860700 10 client.27681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:36.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9ffc860700 1 -- 172.21.15.162:0/2100317237 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff8114c40 con 0x7f9ff80fcb30 2024-02-13T00:49:36.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.767+0000 7f9ffc860700 1 librados: init done 2024-02-13T00:49:36.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.768+0000 7f9ffc860700 1 -- 172.21.15.162:0/2100317237 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff80475e0 con 0x7f9ff80fcb30 2024-02-13T00:49:36.774 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.769+0000 7f9fe77fe700 1 -- 172.21.15.162:0/2100317237 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f9fec009110 con 0x7f9ff80fcb30 2024-02-13T00:49:36.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.769+0000 7f9fe77fe700 1 --2- 172.21.15.162:0/2100317237 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9fe0062c90 0x7f9fe0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:36.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.769+0000 7f9fe77fe700 1 -- 172.21.15.162:0/2100317237 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(562..562 src has 449..562) v4 ==== 7672+0+0 (secure 0 0 0) 0x7f9fec07cfb0 con 0x7f9ff80fcb30 2024-02-13T00:49:36.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.770+0000 7f9fe77fe700 10 client.27681.objecter ms_dispatch 0x7f9ff8001550 osd_map(562..562 src has 449..562) v4 2024-02-13T00:49:36.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.770+0000 7f9fe77fe700 3 client.27681.objecter handle_osd_map got epochs [562,562] > 0 2024-02-13T00:49:36.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.770+0000 7f9fe77fe700 3 client.27681.objecter handle_osd_map decoding full epoch 562 2024-02-13T00:49:36.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.770+0000 7f9ff659c700 1 --2- 172.21.15.162:0/2100317237 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9fe0062c90 0x7f9fe0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:36.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.770+0000 7f9fe77fe700 20 client.27681.objecter dump_active .. 0 homeless 2024-02-13T00:49:36.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.770+0000 7f9fe77fe700 10 client.27681.objecter ms_handle_connect 0x7f9fe0062c90 2024-02-13T00:49:36.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.770+0000 7f9ff659c700 1 --2- 172.21.15.162:0/2100317237 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9fe0062c90 0x7f9fe0065150 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 rx=0x7f9fe8004870 tx=0x7f9fe800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:36.777 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.774+0000 7f9fe77fe700 1 -- 172.21.15.162:0/2100317237 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9fec0506e0 con 0x7f9ff80fcb30 2024-02-13T00:49:36.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:49:36.847 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:36.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:36.965+0000 7f9ffc860700 1 -- 172.21.15.162:0/2100317237 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f9ff8106090 con 0x7f9ff80fcb30 2024-02-13T00:49:36.992 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:37.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:49:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:35.748918+0000 mgr.y (mgr.14544) 486 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:35.830930+0000 osd.0 (osd.0) 211 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:49:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:35.834661+0000 osd.0 (osd.0) 212 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:49:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:36.286236+0000 osd.6 (osd.6) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:36.286489+0000 osd.6 (osd.6) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:36.742538+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e562: 8 total, 8 up, 4 in 2024-02-13T00:49:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:35.748918+0000 mgr.y (mgr.14544) 486 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:35.830930+0000 osd.0 (osd.0) 211 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:49:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:35.834661+0000 osd.0 (osd.0) 212 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:49:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:36.286236+0000 osd.6 (osd.6) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:36.286489+0000 osd.6 (osd.6) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:36.742538+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e562: 8 total, 8 up, 4 in 2024-02-13T00:49:37.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:35.748918+0000 mgr.y (mgr.14544) 486 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:37.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:35.830930+0000 osd.0 (osd.0) 211 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:49:37.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:35.834661+0000 osd.0 (osd.0) 212 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:49:37.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:36.286236+0000 osd.6 (osd.6) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:37.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:36.286489+0000 osd.6 (osd.6) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:37.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:36.742538+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e562: 8 total, 8 up, 4 in 2024-02-13T00:49:37.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:37.139 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:37.337 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:37.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:37.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:37.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:37.472 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:37.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.497+0000 7f9fe77fe700 1 -- 172.21.15.162:0/2100317237 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v563) v1 ==== 101+0+0 (secure 0 0 0) 0x7f9fec00d360 con 0x7f9ff80fcb30 2024-02-13T00:49:37.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.500+0000 7f9ffc860700 10 librados: watch_flush enter 2024-02-13T00:49:37.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 10 librados: watch_flush exit 2024-02-13T00:49:37.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 20 client.27681.objecter shutdown clearing up homeless session... 2024-02-13T00:49:37.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 10 client.27681.objecter successfully canceled tick 2024-02-13T00:49:37.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 -- 172.21.15.162:0/2100317237 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9fe0062c90 msgr2=0x7f9fe0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:37.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 --2- 172.21.15.162:0/2100317237 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9fe0062c90 0x7f9fe0065150 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 rx=0x7f9fe8004870 tx=0x7f9fe800b040).stop 2024-02-13T00:49:37.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 -- 172.21.15.162:0/2100317237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ff80fcb30 msgr2=0x7f9ff8109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:37.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 --2- 172.21.15.162:0/2100317237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ff80fcb30 0x7f9ff8109510 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7f9fec0041d0 tx=0x7f9fec009f60).stop 2024-02-13T00:49:37.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 -- 172.21.15.162:0/2100317237 shutdown_connections 2024-02-13T00:49:37.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 --2- 172.21.15.162:0/2100317237 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9fe0062c90 0x7f9fe0065150 unknown :-1 s=CLOSED pgs=2960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:37.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 --2- 172.21.15.162:0/2100317237 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ff80fd490 0x7f9ff8109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:37.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 --2- 172.21.15.162:0/2100317237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ff80f73a0 0x7f9ff8109020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:37.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 --2- 172.21.15.162:0/2100317237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ff80fcb30 0x7f9ff8109510 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:37.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 -- 172.21.15.162:0/2100317237 >> 172.21.15.162:0/2100317237 conn(0x7f9ff806eb90 msgr2=0x7f9ff80fab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:37.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 -- 172.21.15.162:0/2100317237 shutdown_connections 2024-02-13T00:49:37.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 -- 172.21.15.162:0/2100317237 wait complete. 2024-02-13T00:49:37.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:37.501+0000 7f9ffc860700 1 librados: shutdown 2024-02-13T00:49:37.508 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:49:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:36.477456+0000 mgr.y (mgr.14544) 487 : cluster [DBG] pgmap v842: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:36.839265+0000 osd.0 (osd.0) 213 : cluster [DBG] 28.18 scrub starts 2024-02-13T00:49:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:36.841499+0000 osd.0 (osd.0) 214 : cluster [DBG] 28.18 scrub ok 2024-02-13T00:49:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:36.966646+0000 mon.a (mon.0) 2964 : audit [INF] from='client.? 172.21.15.162:0/2100317237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:37.491117+0000 mon.a (mon.0) 2965 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:49:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:37.497950+0000 mon.a (mon.0) 2966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-84057-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-84057-94"}]': finished 2024-02-13T00:49:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:37.498025+0000 mon.a (mon.0) 2967 : audit [INF] from='client.? 172.21.15.162:0/2100317237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:49:38.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:37.498081+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e563: 8 total, 8 up, 4 in 2024-02-13T00:49:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:36.477456+0000 mgr.y (mgr.14544) 487 : cluster [DBG] pgmap v842: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:36.839265+0000 osd.0 (osd.0) 213 : cluster [DBG] 28.18 scrub starts 2024-02-13T00:49:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:36.841499+0000 osd.0 (osd.0) 214 : cluster [DBG] 28.18 scrub ok 2024-02-13T00:49:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:36.966646+0000 mon.a (mon.0) 2964 : audit [INF] from='client.? 172.21.15.162:0/2100317237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:37.491117+0000 mon.a (mon.0) 2965 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:49:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:37.497950+0000 mon.a (mon.0) 2966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-84057-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-84057-94"}]': finished 2024-02-13T00:49:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:37.498025+0000 mon.a (mon.0) 2967 : audit [INF] from='client.? 172.21.15.162:0/2100317237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:49:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:37.498081+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e563: 8 total, 8 up, 4 in 2024-02-13T00:49:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:36.477456+0000 mgr.y (mgr.14544) 487 : cluster [DBG] pgmap v842: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:36.839265+0000 osd.0 (osd.0) 213 : cluster [DBG] 28.18 scrub starts 2024-02-13T00:49:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:36.841499+0000 osd.0 (osd.0) 214 : cluster [DBG] 28.18 scrub ok 2024-02-13T00:49:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:36.966646+0000 mon.a (mon.0) 2964 : audit [INF] from='client.? 172.21.15.162:0/2100317237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:37.491117+0000 mon.a (mon.0) 2965 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:49:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:37.497950+0000 mon.a (mon.0) 2966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-84057-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-84057-94"}]': finished 2024-02-13T00:49:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:37.498025+0000 mon.a (mon.0) 2967 : audit [INF] from='client.? 172.21.15.162:0/2100317237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:49:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:37.498081+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e563: 8 total, 8 up, 4 in 2024-02-13T00:49:38.221 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:38.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:38.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:38.358 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:38.379 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:38.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:38.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:38.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:38.519 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:38.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:49:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:49:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:49:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:37.049717+0000 osd.1 (osd.1) 241 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:49:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:37.053394+0000 osd.1 (osd.1) 242 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:49:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:38.003413+0000 osd.1 (osd.1) 243 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:49:39.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:38.502038+0000 mon.a (mon.0) 2969 : cluster [DBG] osdmap e564: 8 total, 8 up, 4 in 2024-02-13T00:49:39.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:37.049717+0000 osd.1 (osd.1) 241 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:49:39.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:37.053394+0000 osd.1 (osd.1) 242 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:49:39.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:38.003413+0000 osd.1 (osd.1) 243 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:49:39.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:38.502038+0000 mon.a (mon.0) 2969 : cluster [DBG] osdmap e564: 8 total, 8 up, 4 in 2024-02-13T00:49:39.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:37.049717+0000 osd.1 (osd.1) 241 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:49:39.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:37.053394+0000 osd.1 (osd.1) 242 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:49:39.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:38.003413+0000 osd.1 (osd.1) 243 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:49:39.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:38.502038+0000 mon.a (mon.0) 2969 : cluster [DBG] osdmap e564: 8 total, 8 up, 4 in 2024-02-13T00:49:39.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.890+0000 7efe96793700 1 -- 172.21.15.162:0/2503413133 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efe900fcb30 msgr2=0x7efe900fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:39.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.890+0000 7efe96793700 1 --2- 172.21.15.162:0/2503413133 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efe900fcb30 0x7efe900fcf10 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 rx=0x7efe800047b0 tx=0x7efe80009cb0).stop 2024-02-13T00:49:39.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.890+0000 7efe96793700 1 -- 172.21.15.162:0/2503413133 shutdown_connections 2024-02-13T00:49:39.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.890+0000 7efe96793700 1 --2- 172.21.15.162:0/2503413133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe900f73a0 0x7efe901039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:39.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.890+0000 7efe96793700 1 --2- 172.21.15.162:0/2503413133 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efe900fcb30 0x7efe900fcf10 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:39.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.890+0000 7efe96793700 1 --2- 172.21.15.162:0/2503413133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe900fd490 0x7efe900f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:39.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.890+0000 7efe96793700 1 -- 172.21.15.162:0/2503413133 >> 172.21.15.162:0/2503413133 conn(0x7efe9006eb90 msgr2=0x7efe900f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:39.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.890+0000 7efe96793700 1 -- 172.21.15.162:0/2503413133 shutdown_connections 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.890+0000 7efe96793700 1 -- 172.21.15.162:0/2503413133 wait complete. 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe96793700 1 librados: starting msgr at 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe96793700 1 librados: starting objecter 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe96793700 1 Processor -- start 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe96793700 1 -- start start 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe96793700 1 librados: setting wanted keys 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe96793700 1 librados: calling monclient init 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe96793700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe900f73a0 0x7efe90110380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe96793700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe900fcb30 0x7efe90110870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe96793700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efe900fd490 0x7efe9010a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe96793700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9010aa10 con 0x7efe900f73a0 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe96793700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9010ab80 con 0x7efe900fcb30 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe96793700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9010ad40 con 0x7efe900fd490 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe8ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe900f73a0 0x7efe90110380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe94d30700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efe900fd490 0x7efe9010a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe8ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe900f73a0 0x7efe90110380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:58772/0 (socket says 172.21.15.162:58772) 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe94d30700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efe900fd490 0x7efe9010a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:56228/0 (socket says 172.21.15.162:56228) 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe8ffff700 1 -- 172.21.15.162:0/841146475 learned_addr learned my addr 172.21.15.162:0/841146475 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe87fff700 1 --2- 172.21.15.162:0/841146475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe900fcb30 0x7efe90110870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe8ffff700 1 -- 172.21.15.162:0/841146475 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efe900fd490 msgr2=0x7efe9010a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe8ffff700 1 --2- 172.21.15.162:0/841146475 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efe900fd490 0x7efe9010a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe8ffff700 1 -- 172.21.15.162:0/841146475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe900fcb30 msgr2=0x7efe90110870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe8ffff700 1 --2- 172.21.15.162:0/841146475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe900fcb30 0x7efe90110870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe8ffff700 1 -- 172.21.15.162:0/841146475 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe880097c0 con 0x7efe900f73a0 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.891+0000 7efe87fff700 1 --2- 172.21.15.162:0/841146475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe900fcb30 0x7efe90110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.892+0000 7efe8ffff700 1 --2- 172.21.15.162:0/841146475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe900f73a0 0x7efe90110380 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 rx=0x7efe800044e0 tx=0x7efe80007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.892+0000 7efe8dffb700 10 client.?.objecter ms_handle_connect 0x7efe900f73a0 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.892+0000 7efe8dffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:39.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.892+0000 7efe8dffb700 1 -- 172.21.15.162:0/841146475 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe80018070 con 0x7efe900f73a0 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.892+0000 7efe96793700 1 -- 172.21.15.162:0/841146475 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe80005040 con 0x7efe900f73a0 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.892+0000 7efe96793700 10 client.27693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.892+0000 7efe96793700 1 -- 172.21.15.162:0/841146475 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9010b1e0 con 0x7efe900f73a0 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.892+0000 7efe96793700 1 librados: init done 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.892+0000 7efe8dffb700 1 -- 172.21.15.162:0/841146475 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe800093b0 con 0x7efe900f73a0 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.892+0000 7efe8dffb700 1 -- 172.21.15.162:0/841146475 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8001de00 con 0x7efe900f73a0 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.893+0000 7efe96793700 1 -- 172.21.15.162:0/841146475 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe70004f50 con 0x7efe900f73a0 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.893+0000 7efe8dffb700 1 -- 172.21.15.162:0/841146475 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7efe80013ce0 con 0x7efe900f73a0 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.894+0000 7efe8dffb700 1 --2- 172.21.15.162:0/841146475 >> v2:172.21.15.162:6800/2866534053 conn(0x7efe7c062c90 0x7efe7c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.894+0000 7efe87fff700 1 --2- 172.21.15.162:0/841146475 >> v2:172.21.15.162:6800/2866534053 conn(0x7efe7c062c90 0x7efe7c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.894+0000 7efe8dffb700 1 -- 172.21.15.162:0/841146475 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(564..564 src has 449..564) v4 ==== 8093+0+0 (secure 0 0 0) 0x7efe8007e890 con 0x7efe900f73a0 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.894+0000 7efe8dffb700 10 client.27693.objecter ms_dispatch 0x7efe90001550 osd_map(564..564 src has 449..564) v4 2024-02-13T00:49:39.920 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.894+0000 7efe8dffb700 3 client.27693.objecter handle_osd_map got epochs [564,564] > 0 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.894+0000 7efe8dffb700 3 client.27693.objecter handle_osd_map decoding full epoch 564 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.894+0000 7efe8dffb700 20 client.27693.objecter dump_active .. 0 homeless 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.894+0000 7efe8dffb700 10 client.27693.objecter ms_handle_connect 0x7efe7c062c90 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.894+0000 7efe87fff700 1 --2- 172.21.15.162:0/841146475 >> v2:172.21.15.162:6800/2866534053 conn(0x7efe7c062c90 0x7efe7c065150 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 rx=0x7efe9010bcc0 tx=0x7efe7800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:38.898+0000 7efe8dffb700 1 -- 172.21.15.162:0/841146475 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7efe80051ce0 con 0x7efe900f73a0 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.084+0000 7efe96793700 1 -- 172.21.15.162:0/841146475 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7efe700054d0 con 0x7efe900f73a0 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.498+0000 7efe8dffb700 1 -- 172.21.15.162:0/841146475 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v565) v1 ==== 113+0+0 (secure 0 0 0) 0x7efe800518d0 con 0x7efe900f73a0 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.501+0000 7efe96793700 10 librados: watch_flush enter 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.501+0000 7efe96793700 10 librados: watch_flush exit 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.501+0000 7efe96793700 20 client.27693.objecter shutdown clearing up homeless session... 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.501+0000 7efe96793700 10 client.27693.objecter successfully canceled tick 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.502+0000 7efe96793700 1 -- 172.21.15.162:0/841146475 >> v2:172.21.15.162:6800/2866534053 conn(0x7efe7c062c90 msgr2=0x7efe7c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.502+0000 7efe96793700 1 --2- 172.21.15.162:0/841146475 >> v2:172.21.15.162:6800/2866534053 conn(0x7efe7c062c90 0x7efe7c065150 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 rx=0x7efe9010bcc0 tx=0x7efe7800b040).stop 2024-02-13T00:49:39.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.502+0000 7efe96793700 1 -- 172.21.15.162:0/841146475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe900f73a0 msgr2=0x7efe90110380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:39.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.502+0000 7efe96793700 1 --2- 172.21.15.162:0/841146475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe900f73a0 0x7efe90110380 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 rx=0x7efe800044e0 tx=0x7efe80007f90).stop 2024-02-13T00:49:39.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.502+0000 7efe96793700 1 -- 172.21.15.162:0/841146475 shutdown_connections 2024-02-13T00:49:39.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.502+0000 7efe96793700 1 --2- 172.21.15.162:0/841146475 >> v2:172.21.15.162:6800/2866534053 conn(0x7efe7c062c90 0x7efe7c065150 secure :-1 s=CLOSED pgs=2962 cs=0 l=1 rev1=1 rx=0x7efe9010bcc0 tx=0x7efe7800b040).stop 2024-02-13T00:49:39.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.502+0000 7efe96793700 1 --2- 172.21.15.162:0/841146475 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efe900fd490 0x7efe9010a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:39.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.502+0000 7efe96793700 1 --2- 172.21.15.162:0/841146475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe900f73a0 0x7efe90110380 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:39.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.502+0000 7efe96793700 1 --2- 172.21.15.162:0/841146475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe900fcb30 0x7efe90110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:39.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.502+0000 7efe96793700 1 -- 172.21.15.162:0/841146475 >> 172.21.15.162:0/841146475 conn(0x7efe9006eb90 msgr2=0x7efe9006ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:39.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.505+0000 7efe96793700 1 -- 172.21.15.162:0/841146475 shutdown_connections 2024-02-13T00:49:39.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.505+0000 7efe96793700 1 -- 172.21.15.162:0/841146475 wait complete. 2024-02-13T00:49:39.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:39.505+0000 7efe96793700 1 librados: shutdown 2024-02-13T00:49:39.922 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.6 to 1 (10000) 2024-02-13T00:49:40.228 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:38.007200+0000 osd.1 (osd.1) 244 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:49:40.228 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:38.477877+0000 mgr.y (mgr.14544) 488 : cluster [DBG] pgmap v845: 172 pgs: 8 unknown, 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:40.228 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:39.085904+0000 mon.a (mon.0) 2970 : audit [INF] from='client.? 172.21.15.162:0/841146475' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:49:40.228 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:39.498738+0000 mon.a (mon.0) 2971 : audit [INF] from='client.? 172.21.15.162:0/841146475' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:49:40.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:39.498804+0000 mon.a (mon.0) 2972 : cluster [DBG] osdmap e565: 8 total, 8 up, 4 in 2024-02-13T00:49:40.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:39.500503+0000 mon.c (mon.2) 441 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:40.229 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:39.501500+0000 mon.a (mon.0) 2973 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:40.229 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:38.007200+0000 osd.1 (osd.1) 244 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:49:40.229 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:38.477877+0000 mgr.y (mgr.14544) 488 : cluster [DBG] pgmap v845: 172 pgs: 8 unknown, 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:40.229 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:39.085904+0000 mon.a (mon.0) 2970 : audit [INF] from='client.? 172.21.15.162:0/841146475' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:49:40.229 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:39.498738+0000 mon.a (mon.0) 2971 : audit [INF] from='client.? 172.21.15.162:0/841146475' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:49:40.229 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:39.498804+0000 mon.a (mon.0) 2972 : cluster [DBG] osdmap e565: 8 total, 8 up, 4 in 2024-02-13T00:49:40.229 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:39.500503+0000 mon.c (mon.2) 441 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:40.229 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:39.501500+0000 mon.a (mon.0) 2973 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:38.007200+0000 osd.1 (osd.1) 244 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:49:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:38.477877+0000 mgr.y (mgr.14544) 488 : cluster [DBG] pgmap v845: 172 pgs: 8 unknown, 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:39.085904+0000 mon.a (mon.0) 2970 : audit [INF] from='client.? 172.21.15.162:0/841146475' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:49:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:39.498738+0000 mon.a (mon.0) 2971 : audit [INF] from='client.? 172.21.15.162:0/841146475' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:49:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:39.498804+0000 mon.a (mon.0) 2972 : cluster [DBG] osdmap e565: 8 total, 8 up, 4 in 2024-02-13T00:49:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:39.500503+0000 mon.c (mon.2) 441 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:39.501500+0000 mon.a (mon.0) 2973 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:40.604 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-13T00:49:40.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:49:41.050 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:41.069 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:41.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:41.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:41.208 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:41.228 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:41.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:41.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:41.362 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:41.382 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:41.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:41.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:49:41.486 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:41.519 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:41.538 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:41.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:41.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:39.787874+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.a scrub starts 2024-02-13T00:49:41.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:39.788779+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.a scrub ok 2024-02-13T00:49:41.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:40.478846+0000 mgr.y (mgr.14544) 489 : cluster [DBG] pgmap v848: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:49:41.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:40.502806+0000 mon.a (mon.0) 2974 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:41.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:40.502946+0000 mon.a (mon.0) 2975 : cluster [DBG] osdmap e566: 8 total, 8 up, 4 in 2024-02-13T00:49:41.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:40.847427+0000 mon.a (mon.0) 2976 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:49:41.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:39.787874+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.a scrub starts 2024-02-13T00:49:41.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:39.788779+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.a scrub ok 2024-02-13T00:49:41.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:40.478846+0000 mgr.y (mgr.14544) 489 : cluster [DBG] pgmap v848: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:49:41.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:40.502806+0000 mon.a (mon.0) 2974 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:41.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:40.502946+0000 mon.a (mon.0) 2975 : cluster [DBG] osdmap e566: 8 total, 8 up, 4 in 2024-02-13T00:49:41.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:40.847427+0000 mon.a (mon.0) 2976 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:49:41.642 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:41.671 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:41.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:39.787874+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.a scrub starts 2024-02-13T00:49:41.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:39.788779+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.a scrub ok 2024-02-13T00:49:41.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:40.478846+0000 mgr.y (mgr.14544) 489 : cluster [DBG] pgmap v848: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:49:41.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:40.502806+0000 mon.a (mon.0) 2974 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:41.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:40.502946+0000 mon.a (mon.0) 2975 : cluster [DBG] osdmap e566: 8 total, 8 up, 4 in 2024-02-13T00:49:41.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:40.847427+0000 mon.a (mon.0) 2976 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:49:41.869 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:41.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:41.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:41.891 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:49:41.892 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:49:41.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:42.002 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:42.021 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:42.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:42.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:42.154 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:42.253 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:42.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:42.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:42.383 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:42.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:41.394663+0000 osd.0 (osd.0) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:42.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:41.394913+0000 osd.0 (osd.0) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:42.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:41.502917+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e567: 8 total, 8 up, 4 in 2024-02-13T00:49:42.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:41.520120+0000 mon.c (mon.2) 442 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:42.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:41.520400+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:42.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:42.507591+0000 mon.a (mon.0) 2979 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:42.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:42.507674+0000 mon.a (mon.0) 2980 : cluster [DBG] osdmap e568: 8 total, 8 up, 4 in 2024-02-13T00:49:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:41.394663+0000 osd.0 (osd.0) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:41.394913+0000 osd.0 (osd.0) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:41.502917+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e567: 8 total, 8 up, 4 in 2024-02-13T00:49:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:41.520120+0000 mon.c (mon.2) 442 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:41.520400+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:42.507591+0000 mon.a (mon.0) 2979 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:42.507674+0000 mon.a (mon.0) 2980 : cluster [DBG] osdmap e568: 8 total, 8 up, 4 in 2024-02-13T00:49:42.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:41.394663+0000 osd.0 (osd.0) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:42.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:41.394913+0000 osd.0 (osd.0) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:41.502917+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e567: 8 total, 8 up, 4 in 2024-02-13T00:49:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:41.520120+0000 mon.c (mon.2) 442 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:41.520400+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:42.507591+0000 mon.a (mon.0) 2979 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-97-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:42.507674+0000 mon.a (mon.0) 2980 : cluster [DBG] osdmap e568: 8 total, 8 up, 4 in 2024-02-13T00:49:42.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-13T00:49:42.930 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:42.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:42.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:43.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:43.060 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:43.229 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:41.836808+0000 osd.0 (osd.0) 219 : cluster [DBG] 6.9 scrub starts 2024-02-13T00:49:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:41.837806+0000 osd.0 (osd.0) 220 : cluster [DBG] 6.9 scrub ok 2024-02-13T00:49:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:42.479248+0000 mgr.y (mgr.14544) 490 : cluster [DBG] pgmap v851: 236 pgs: 32 unknown, 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:49:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:42.515063+0000 mon.c (mon.2) 443 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:42.516298+0000 mon.a (mon.0) 2981 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:43.507148+0000 mon.a (mon.0) 2982 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]': finished 2024-02-13T00:49:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:43.507221+0000 mon.a (mon.0) 2983 : cluster [DBG] osdmap e569: 8 total, 8 up, 4 in 2024-02-13T00:49:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:43.508669+0000 mon.c (mon.2) 444 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-97", "overlaypool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:43.509513+0000 mon.a (mon.0) 2984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-97", "overlaypool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:43.797 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:41.836808+0000 osd.0 (osd.0) 219 : cluster [DBG] 6.9 scrub starts 2024-02-13T00:49:43.797 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:41.837806+0000 osd.0 (osd.0) 220 : cluster [DBG] 6.9 scrub ok 2024-02-13T00:49:43.797 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:42.479248+0000 mgr.y (mgr.14544) 490 : cluster [DBG] pgmap v851: 236 pgs: 32 unknown, 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:49:43.797 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:42.515063+0000 mon.c (mon.2) 443 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:43.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:42.516298+0000 mon.a (mon.0) 2981 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:43.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:43.507148+0000 mon.a (mon.0) 2982 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]': finished 2024-02-13T00:49:43.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:43.507221+0000 mon.a (mon.0) 2983 : cluster [DBG] osdmap e569: 8 total, 8 up, 4 in 2024-02-13T00:49:43.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:43.508669+0000 mon.c (mon.2) 444 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-97", "overlaypool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:43.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:43.509513+0000 mon.a (mon.0) 2984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-97", "overlaypool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:43.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:41.836808+0000 osd.0 (osd.0) 219 : cluster [DBG] 6.9 scrub starts 2024-02-13T00:49:43.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:41.837806+0000 osd.0 (osd.0) 220 : cluster [DBG] 6.9 scrub ok 2024-02-13T00:49:43.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:42.479248+0000 mgr.y (mgr.14544) 490 : cluster [DBG] pgmap v851: 236 pgs: 32 unknown, 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:49:43.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:42.515063+0000 mon.c (mon.2) 443 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:43.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:42.516298+0000 mon.a (mon.0) 2981 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:43.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:43.507148+0000 mon.a (mon.0) 2982 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]': finished 2024-02-13T00:49:43.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:43.507221+0000 mon.a (mon.0) 2983 : cluster [DBG] osdmap e569: 8 total, 8 up, 4 in 2024-02-13T00:49:43.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:43.508669+0000 mon.c (mon.2) 444 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-97", "overlaypool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:43.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:43.509513+0000 mon.a (mon.0) 2984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-97", "overlaypool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:43.799 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:44.067 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:44.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:44.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:44.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:44.205 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:44.227 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:44.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:44.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:44.360 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:44.509 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:44.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:44.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.611+0000 7ff394b84700 1 -- 172.21.15.162:0/3022436094 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3900f95a0 msgr2=0x7ff390101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.611+0000 7ff394b84700 1 --2- 172.21.15.162:0/3022436094 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3900f95a0 0x7ff390101cb0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 rx=0x7ff3840047b0 tx=0x7ff384009cb0).stop 2024-02-13T00:49:44.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.611+0000 7ff394b84700 1 -- 172.21.15.162:0/3022436094 shutdown_connections 2024-02-13T00:49:44.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.611+0000 7ff394b84700 1 --2- 172.21.15.162:0/3022436094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3901021a0 0x7ff390104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.611+0000 7ff394b84700 1 --2- 172.21.15.162:0/3022436094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3900f8c40 0x7ff3900f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.611+0000 7ff394b84700 1 --2- 172.21.15.162:0/3022436094 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3900f95a0 0x7ff390101cb0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.611+0000 7ff394b84700 1 -- 172.21.15.162:0/3022436094 >> 172.21.15.162:0/3022436094 conn(0x7ff3900f2750 msgr2=0x7ff3900f4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.611+0000 7ff394b84700 1 -- 172.21.15.162:0/3022436094 shutdown_connections 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.611+0000 7ff394b84700 1 -- 172.21.15.162:0/3022436094 wait complete. 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.611+0000 7ff394b84700 1 librados: starting msgr at 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.611+0000 7ff394b84700 1 librados: starting objecter 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.611+0000 7ff394b84700 1 Processor -- start 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff394b84700 1 -- start start 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff394b84700 1 librados: setting wanted keys 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff394b84700 1 librados: calling monclient init 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff394b84700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3900f8c40 0x7ff39010e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff394b84700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3900f95a0 0x7ff39010e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff394b84700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3901021a0 0x7ff390108400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:44.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff394b84700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3901088f0 con 0x7ff3901021a0 2024-02-13T00:49:44.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff394b84700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff390108a60 con 0x7ff3900f95a0 2024-02-13T00:49:44.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff394b84700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff390108c20 con 0x7ff3900f8c40 2024-02-13T00:49:44.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff38e59c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3900f8c40 0x7ff39010e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:44.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff38ed9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3901021a0 0x7ff390108400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:44.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff38e59c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3900f8c40 0x7ff39010e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:42898/0 (socket says 172.21.15.162:42898) 2024-02-13T00:49:44.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff38e59c700 1 -- 172.21.15.162:0/2859439938 learned_addr learned my addr 172.21.15.162:0/2859439938 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:44.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff38dd9b700 1 --2- 172.21.15.162:0/2859439938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3900f95a0 0x7ff39010e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:44.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:44.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff38e59c700 1 -- 172.21.15.162:0/2859439938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3900f95a0 msgr2=0x7ff39010e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff38e59c700 1 --2- 172.21.15.162:0/2859439938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3900f95a0 0x7ff39010e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff38e59c700 1 -- 172.21.15.162:0/2859439938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3901021a0 msgr2=0x7ff390108400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff38e59c700 1 --2- 172.21.15.162:0/2859439938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3901021a0 0x7ff390108400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff38e59c700 1 -- 172.21.15.162:0/2859439938 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3880097c0 con 0x7ff3900f8c40 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff38ed9d700 1 --2- 172.21.15.162:0/2859439938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3901021a0 0x7ff390108400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff38e59c700 1 --2- 172.21.15.162:0/2859439938 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3900f8c40 0x7ff39010e210 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7ff380002820 tx=0x7ff380009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff37f7fe700 10 client.?.objecter ms_handle_connect 0x7ff3900f8c40 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff37f7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff380007b40 con 0x7ff3900f8c40 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff38000f040 con 0x7ff3900f8c40 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.612+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff380014840 con 0x7ff3900f8c40 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.613+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff384005040 con 0x7ff3900f8c40 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.613+0000 7ff394b84700 10 client.37799.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.613+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff390109210 con 0x7ff3900f8c40 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.613+0000 7ff394b84700 1 librados: init done 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.613+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff390060790 con 0x7ff3900f8c40 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.614+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff380007ca0 con 0x7ff3900f8c40 2024-02-13T00:49:44.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.614+0000 7ff37f7fe700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff378062c90 0x7ff378065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.614+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(570..570 src has 449..570) v4 ==== 8877+0+0 (secure 0 0 0) 0x7ff38007d070 con 0x7ff3900f8c40 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.614+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 osd_map(570..570 src has 449..570) v4 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.614+0000 7ff37f7fe700 3 client.37799.objecter handle_osd_map got epochs [570,570] > 0 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.614+0000 7ff37f7fe700 3 client.37799.objecter handle_osd_map decoding full epoch 570 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.614+0000 7ff38dd9b700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff378062c90 0x7ff378065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.614+0000 7ff37f7fe700 20 client.37799.objecter dump_active .. 0 homeless 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.614+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v570) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff38007f5a0 con 0x7ff3900f8c40 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff37f7fe700 10 client.37799.objecter ms_handle_connect 0x7ff378062c90 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff38dd9b700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff378062c90 0x7ff378065150 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7ff384005010 tx=0x7ff384009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff394b84700 10 client.37799.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 1 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff390028590 0x7ff3900fbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff3900475e0 osd=0 3 2024-02-13T00:49:44.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900475e0 osd=0 3 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff3900475e0 osd=0 3 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 1 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff3900475e0 osd=0 3 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 0 1 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900475e0 osd=0 4 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff394b84700 10 client.37799.objecter _send_command 1 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3901071d0 con 0x7ff390028590 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff38ed9d700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff390028590 0x7ff3900fbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff38ed9d700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff390028590 0x7ff3900fbc60 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.615+0000 7ff37f7fe700 10 client.37799.objecter ms_handle_connect 0x7ff390028590 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.616+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff3901071d0 con 0x7ff390028590 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.616+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.616+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 1 = system:0 2024-02-13T00:49:44.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.616+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff3900475e0 osd=0 4 2024-02-13T00:49:44.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.616+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 0 1 2024-02-13T00:49:44.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.631+0000 7ff394b84700 10 client.37799.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:44.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.631+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 2 2024-02-13T00:49:44.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.631+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff3900475e0 osd=0 3 2024-02-13T00:49:44.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.631+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900475e0 osd=0 3 2024-02-13T00:49:44.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.631+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff3900475e0 osd=0 3 2024-02-13T00:49:44.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.631+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 2 2024-02-13T00:49:44.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.631+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff3900475e0 osd=0 3 2024-02-13T00:49:44.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.631+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 0 2 2024-02-13T00:49:44.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.631+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900475e0 osd=0 4 2024-02-13T00:49:44.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.631+0000 7ff394b84700 10 client.37799.objecter _send_command 2 2024-02-13T00:49:44.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.631+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff39005a0e0 con 0x7ff390028590 2024-02-13T00:49:44.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff39005a0e0 con 0x7ff390028590 2024-02-13T00:49:44.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:44.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff3900475e0 osd=0 4 2024-02-13T00:49:44.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 0 2 2024-02-13T00:49:44.636 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:49:44.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff394b84700 10 client.37799.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:44.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 3 2024-02-13T00:49:44.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6805/150599780 conn(0x7ff390027be0 0x7ff390027fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:44.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff3900fd2d0 osd=1 3 2024-02-13T00:49:44.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900fd2d0 osd=1 3 2024-02-13T00:49:44.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff3900fd2d0 osd=1 3 2024-02-13T00:49:44.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 3 2024-02-13T00:49:44.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff3900fd2d0 osd=1 3 2024-02-13T00:49:44.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 1 3 2024-02-13T00:49:44.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900fd2d0 osd=1 4 2024-02-13T00:49:44.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff394b84700 10 client.37799.objecter _send_command 3 2024-02-13T00:49:44.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3900fd440 con 0x7ff390027be0 2024-02-13T00:49:44.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff38e59c700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6805/150599780 conn(0x7ff390027be0 0x7ff390027fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:44.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff37f7fe700 10 client.37799.objecter ms_handle_connect 0x7ff390027be0 2024-02-13T00:49:44.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.633+0000 7ff38e59c700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6805/150599780 conn(0x7ff390027be0 0x7ff390027fc0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:44.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.634+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff3900fd440 con 0x7ff390027be0 2024-02-13T00:49:44.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.634+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:49:44.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.634+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 3 = system:0 2024-02-13T00:49:44.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.634+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff3900fd2d0 osd=1 4 2024-02-13T00:49:44.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.634+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 1 3 2024-02-13T00:49:44.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.649+0000 7ff394b84700 10 client.37799.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:44.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.649+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 4 2024-02-13T00:49:44.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.649+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff3900fd2d0 osd=1 3 2024-02-13T00:49:44.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.649+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900fd2d0 osd=1 3 2024-02-13T00:49:44.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.649+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff3900fd2d0 osd=1 3 2024-02-13T00:49:44.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.649+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 4 2024-02-13T00:49:44.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.649+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff3900fd2d0 osd=1 3 2024-02-13T00:49:44.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.649+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 1 4 2024-02-13T00:49:44.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.649+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900fd2d0 osd=1 4 2024-02-13T00:49:44.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.649+0000 7ff394b84700 10 client.37799.objecter _send_command 4 2024-02-13T00:49:44.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.649+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff3900f8a40 con 0x7ff390027be0 2024-02-13T00:49:44.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff3900f8a40 con 0x7ff390027be0 2024-02-13T00:49:44.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:44.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff3900fd2d0 osd=1 4 2024-02-13T00:49:44.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 1 4 2024-02-13T00:49:44.655 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:49:44.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff394b84700 10 client.37799.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:44.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 5 2024-02-13T00:49:44.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff3900fce30 0x7ff39011e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:44.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff390027890 osd=2 3 2024-02-13T00:49:44.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff390027890 osd=2 3 2024-02-13T00:49:44.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff390027890 osd=2 3 2024-02-13T00:49:44.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 5 2024-02-13T00:49:44.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff390027890 osd=2 3 2024-02-13T00:49:44.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 2 5 2024-02-13T00:49:44.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff390027890 osd=2 4 2024-02-13T00:49:44.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff394b84700 10 client.37799.objecter _send_command 5 2024-02-13T00:49:44.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3901212a0 con 0x7ff3900fce30 2024-02-13T00:49:44.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.651+0000 7ff38dd9b700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff3900fce30 0x7ff39011e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:44.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.652+0000 7ff37f7fe700 10 client.37799.objecter ms_handle_connect 0x7ff3900fce30 2024-02-13T00:49:44.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.652+0000 7ff38dd9b700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff3900fce30 0x7ff39011e650 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:44.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.653+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff3901212a0 con 0x7ff3900fce30 2024-02-13T00:49:44.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.653+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:49:44.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.653+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 5 = system:0 2024-02-13T00:49:44.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.653+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff390027890 osd=2 4 2024-02-13T00:49:44.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.653+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 2 5 2024-02-13T00:49:44.659 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:44.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.668+0000 7ff394b84700 10 client.37799.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:44.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.668+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 6 2024-02-13T00:49:44.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.668+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff390027890 osd=2 3 2024-02-13T00:49:44.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.668+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff390027890 osd=2 3 2024-02-13T00:49:44.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.668+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff390027890 osd=2 3 2024-02-13T00:49:44.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.668+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 6 2024-02-13T00:49:44.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.668+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff390027890 osd=2 3 2024-02-13T00:49:44.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.668+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 2 6 2024-02-13T00:49:44.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.668+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff390027890 osd=2 4 2024-02-13T00:49:44.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.668+0000 7ff394b84700 10 client.37799.objecter _send_command 6 2024-02-13T00:49:44.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.668+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff3900fc390 con 0x7ff3900fce30 2024-02-13T00:49:44.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff3900fc390 con 0x7ff3900fce30 2024-02-13T00:49:44.673 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:44.673 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.673 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff390027890 osd=2 4 2024-02-13T00:49:44.673 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 2 6 2024-02-13T00:49:44.673 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:49:44.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff394b84700 10 client.37799.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:44.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 7 2024-02-13T00:49:44.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6813/604971011 conn(0x7ff39011c1a0 0x7ff39011fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:44.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff3900fc770 osd=3 3 2024-02-13T00:49:44.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900fc770 osd=3 3 2024-02-13T00:49:44.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff3900fc770 osd=3 3 2024-02-13T00:49:44.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 7 2024-02-13T00:49:44.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff3900fc770 osd=3 3 2024-02-13T00:49:44.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 3 7 2024-02-13T00:49:44.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900fc770 osd=3 4 2024-02-13T00:49:44.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff394b84700 10 client.37799.objecter _send_command 7 2024-02-13T00:49:44.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.669+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3901201b0 con 0x7ff39011c1a0 2024-02-13T00:49:44.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.670+0000 7ff38ed9d700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6813/604971011 conn(0x7ff39011c1a0 0x7ff39011fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:44.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.670+0000 7ff38ed9d700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6813/604971011 conn(0x7ff39011c1a0 0x7ff39011fcc0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:44.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.670+0000 7ff37f7fe700 10 client.37799.objecter ms_handle_connect 0x7ff39011c1a0 2024-02-13T00:49:44.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.670+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff3901201b0 con 0x7ff39011c1a0 2024-02-13T00:49:44.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.670+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:49:44.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.670+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 7 = system:0 2024-02-13T00:49:44.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.670+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff3900fc770 osd=3 4 2024-02-13T00:49:44.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.670+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 3 7 2024-02-13T00:49:44.676 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:44.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:44.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.685+0000 7ff394b84700 10 client.37799.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:44.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.685+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 8 2024-02-13T00:49:44.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.685+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff3900fc770 osd=3 3 2024-02-13T00:49:44.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.685+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900fc770 osd=3 3 2024-02-13T00:49:44.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.685+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff3900fc770 osd=3 3 2024-02-13T00:49:44.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.685+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 8 2024-02-13T00:49:44.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.685+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff3900fc770 osd=3 3 2024-02-13T00:49:44.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.685+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 3 8 2024-02-13T00:49:44.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.685+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900fc770 osd=3 4 2024-02-13T00:49:44.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.685+0000 7ff394b84700 10 client.37799.objecter _send_command 8 2024-02-13T00:49:44.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.685+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff39011b700 con 0x7ff39011c1a0 2024-02-13T00:49:44.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff39011b700 con 0x7ff39011c1a0 2024-02-13T00:49:44.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:44.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff3900fc770 osd=3 4 2024-02-13T00:49:44.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 3 8 2024-02-13T00:49:44.690 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:49:44.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff394b84700 10 client.37799.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:44.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 9 2024-02-13T00:49:44.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff390123420 0x7ff390125810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:44.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff39011b8d0 osd=4 3 2024-02-13T00:49:44.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39011b8d0 osd=4 3 2024-02-13T00:49:44.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff39011b8d0 osd=4 3 2024-02-13T00:49:44.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 9 2024-02-13T00:49:44.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff39011b8d0 osd=4 3 2024-02-13T00:49:44.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 4 9 2024-02-13T00:49:44.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39011b8d0 osd=4 4 2024-02-13T00:49:44.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff394b84700 10 client.37799.objecter _send_command 9 2024-02-13T00:49:44.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.686+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff390121060 con 0x7ff390123420 2024-02-13T00:49:44.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.687+0000 7ff38e59c700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff390123420 0x7ff390125810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:44.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.687+0000 7ff38e59c700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff390123420 0x7ff390125810 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:44.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.687+0000 7ff37f7fe700 10 client.37799.objecter ms_handle_connect 0x7ff390123420 2024-02-13T00:49:44.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.688+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff390121060 con 0x7ff390123420 2024-02-13T00:49:44.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.688+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:49:44.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.688+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 9 = system:0 2024-02-13T00:49:44.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.688+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff39011b8d0 osd=4 4 2024-02-13T00:49:44.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.688+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 4 9 2024-02-13T00:49:44.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.702+0000 7ff394b84700 10 client.37799.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:44.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.702+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 10 2024-02-13T00:49:44.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.702+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff39011b8d0 osd=4 3 2024-02-13T00:49:44.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.702+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39011b8d0 osd=4 3 2024-02-13T00:49:44.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.702+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff39011b8d0 osd=4 3 2024-02-13T00:49:44.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.702+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 10 2024-02-13T00:49:44.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.702+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff39011b8d0 osd=4 3 2024-02-13T00:49:44.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.702+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 4 10 2024-02-13T00:49:44.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.702+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39011b8d0 osd=4 4 2024-02-13T00:49:44.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.702+0000 7ff394b84700 10 client.37799.objecter _send_command 10 2024-02-13T00:49:44.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.702+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff390121060 con 0x7ff390123420 2024-02-13T00:49:44.708 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:49:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.704+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff390121060 con 0x7ff390123420 2024-02-13T00:49:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.704+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.704+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.704+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff39011b8d0 osd=4 4 2024-02-13T00:49:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.704+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 4 10 2024-02-13T00:49:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff394b84700 10 client.37799.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 11 2024-02-13T00:49:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6804/2277917372 conn(0x7ff390126210 0x7ff390128600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff390121060 osd=5 3 2024-02-13T00:49:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff390121060 osd=5 3 2024-02-13T00:49:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff390121060 osd=5 3 2024-02-13T00:49:44.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 11 2024-02-13T00:49:44.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff390121060 osd=5 3 2024-02-13T00:49:44.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 5 11 2024-02-13T00:49:44.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff390121060 osd=5 4 2024-02-13T00:49:44.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff394b84700 10 client.37799.objecter _send_command 11 2024-02-13T00:49:44.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff39012b2c0 con 0x7ff390126210 2024-02-13T00:49:44.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff38dd9b700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6804/2277917372 conn(0x7ff390126210 0x7ff390128600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:44.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff37f7fe700 10 client.37799.objecter ms_handle_connect 0x7ff390126210 2024-02-13T00:49:44.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.705+0000 7ff38dd9b700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6804/2277917372 conn(0x7ff390126210 0x7ff390128600 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:44.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.706+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff39012b2c0 con 0x7ff390126210 2024-02-13T00:49:44.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.706+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:49:44.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.706+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 11 = system:0 2024-02-13T00:49:44.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.706+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff390121060 osd=5 4 2024-02-13T00:49:44.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.706+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 5 11 2024-02-13T00:49:44.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.720+0000 7ff394b84700 10 client.37799.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:44.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.720+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 12 2024-02-13T00:49:44.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.720+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff390121060 osd=5 3 2024-02-13T00:49:44.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.720+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff390121060 osd=5 3 2024-02-13T00:49:44.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.720+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff390121060 osd=5 3 2024-02-13T00:49:44.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.720+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 12 2024-02-13T00:49:44.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.720+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff390121060 osd=5 3 2024-02-13T00:49:44.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.720+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 5 12 2024-02-13T00:49:44.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.720+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff390121060 osd=5 4 2024-02-13T00:49:44.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.720+0000 7ff394b84700 10 client.37799.objecter _send_command 12 2024-02-13T00:49:44.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.720+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff39012b2c0 con 0x7ff390126210 2024-02-13T00:49:44.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff39012b2c0 con 0x7ff390126210 2024-02-13T00:49:44.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:44.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff390121060 osd=5 4 2024-02-13T00:49:44.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 5 12 2024-02-13T00:49:44.726 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:49:44.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff394b84700 10 client.37799.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:44.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 13 2024-02-13T00:49:44.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff390129000 0x7ff390129480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:44.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff39012b2c0 osd=6 3 2024-02-13T00:49:44.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39012b2c0 osd=6 3 2024-02-13T00:49:44.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff39012b2c0 osd=6 3 2024-02-13T00:49:44.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 13 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff39012b2c0 osd=6 3 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 6 13 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39012b2c0 osd=6 4 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff394b84700 10 client.37799.objecter _send_command 13 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.722+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff39012b040 con 0x7ff390129000 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.723+0000 7ff38ed9d700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff390129000 0x7ff390129480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.723+0000 7ff37f7fe700 10 client.37799.objecter ms_handle_connect 0x7ff390129000 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.723+0000 7ff38ed9d700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff390129000 0x7ff390129480 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.724+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff39012b040 con 0x7ff390129000 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.724+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.724+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 13 = system:0 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.724+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff39012b2c0 osd=6 4 2024-02-13T00:49:44.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.724+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 6 13 2024-02-13T00:49:44.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.738+0000 7ff394b84700 10 client.37799.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:44.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.738+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 14 2024-02-13T00:49:44.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.738+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff39012b2c0 osd=6 3 2024-02-13T00:49:44.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.738+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39012b2c0 osd=6 3 2024-02-13T00:49:44.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.738+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff39012b2c0 osd=6 3 2024-02-13T00:49:44.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.738+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 14 2024-02-13T00:49:44.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.738+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff39012b2c0 osd=6 3 2024-02-13T00:49:44.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.738+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 6 14 2024-02-13T00:49:44.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.738+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39012b2c0 osd=6 4 2024-02-13T00:49:44.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.738+0000 7ff394b84700 10 client.37799.objecter _send_command 14 2024-02-13T00:49:44.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.738+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff39012b040 con 0x7ff390129000 2024-02-13T00:49:44.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.740+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff39012b040 con 0x7ff390129000 2024-02-13T00:49:44.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.740+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:44.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.740+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.740+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff39012b2c0 osd=6 4 2024-02-13T00:49:44.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.740+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 6 14 2024-02-13T00:49:44.745 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:49:44.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff394b84700 10 client.37799.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:44.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 15 2024-02-13T00:49:44.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff39012da00 0x7ff39012de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:44.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff39012b040 osd=7 3 2024-02-13T00:49:44.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39012b040 osd=7 3 2024-02-13T00:49:44.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff39012b040 osd=7 3 2024-02-13T00:49:44.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 15 2024-02-13T00:49:44.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff39012b040 osd=7 3 2024-02-13T00:49:44.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 7 15 2024-02-13T00:49:44.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39012b040 osd=7 4 2024-02-13T00:49:44.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff394b84700 10 client.37799.objecter _send_command 15 2024-02-13T00:49:44.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff39012e4d0 con 0x7ff39012da00 2024-02-13T00:49:44.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff38e59c700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff39012da00 0x7ff39012de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:44.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff37f7fe700 10 client.37799.objecter ms_handle_connect 0x7ff39012da00 2024-02-13T00:49:44.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.741+0000 7ff38e59c700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff39012da00 0x7ff39012de60 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:44.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.742+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff39012e4d0 con 0x7ff39012da00 2024-02-13T00:49:44.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.742+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:49:44.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.742+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 15 = system:0 2024-02-13T00:49:44.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.742+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff39012b040 osd=7 4 2024-02-13T00:49:44.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.742+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 7 15 2024-02-13T00:49:44.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.757+0000 7ff394b84700 10 client.37799.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:49:44.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.757+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign -1 16 2024-02-13T00:49:44.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.757+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff39012b040 osd=7 3 2024-02-13T00:49:44.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.757+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39012b040 osd=7 3 2024-02-13T00:49:44.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.757+0000 7ff394b84700 20 client.37799.objecter _get_session s=0x7ff39012b040 osd=7 3 2024-02-13T00:49:44.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.757+0000 7ff394b84700 15 client.37799.objecter _session_command_op_remove -1 16 2024-02-13T00:49:44.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.757+0000 7ff394b84700 20 client.37799.objecter get_session s=0x7ff39012b040 osd=7 3 2024-02-13T00:49:44.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.757+0000 7ff394b84700 15 client.37799.objecter _session_command_op_assign 7 16 2024-02-13T00:49:44.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.757+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39012b040 osd=7 4 2024-02-13T00:49:44.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.757+0000 7ff394b84700 10 client.37799.objecter _send_command 16 2024-02-13T00:49:44.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.757+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff39012e4d0 con 0x7ff39012da00 2024-02-13T00:49:44.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff37f7fe700 1 -- 172.21.15.162:0/2859439938 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff39012e4d0 con 0x7ff39012da00 2024-02-13T00:49:44.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff37f7fe700 10 client.37799.objecter ms_dispatch 0x7ff390001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:49:44.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff37f7fe700 10 client.37799.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff37f7fe700 20 client.37799.objecter put_session s=0x7ff39012b040 osd=7 4 2024-02-13T00:49:44.763 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff37f7fe700 15 client.37799.objecter _session_command_op_remove 7 16 2024-02-13T00:49:44.763 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:49:44.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 10 librados: watch_flush enter 2024-02-13T00:49:44.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 10 librados: watch_flush exit 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 10 client.37799.objecter close_session for osd.0 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff390028590 msgr2=0x7ff3900fbc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6801/2103743338 conn(0x7ff390028590 0x7ff3900fbc60 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900475e0 osd=0 1 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 10 client.37799.objecter close_session for osd.1 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6805/150599780 conn(0x7ff390027be0 msgr2=0x7ff390027fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6805/150599780 conn(0x7ff390027be0 0x7ff390027fc0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900fd2d0 osd=1 1 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 10 client.37799.objecter close_session for osd.2 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff3900fce30 msgr2=0x7ff39011e650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6809/3689719376 conn(0x7ff3900fce30 0x7ff39011e650 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff390027890 osd=2 1 2024-02-13T00:49:44.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 10 client.37799.objecter close_session for osd.3 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff38ed9d700 1 -- 172.21.15.162:0/2859439938 reap_dead start 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6813/604971011 conn(0x7ff39011c1a0 msgr2=0x7ff39011fcc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6813/604971011 conn(0x7ff39011c1a0 0x7ff39011fcc0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff3900fc770 osd=3 1 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 10 client.37799.objecter close_session for osd.4 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff390123420 msgr2=0x7ff390125810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6800/3344263388 conn(0x7ff390123420 0x7ff390125810 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39011b8d0 osd=4 1 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 10 client.37799.objecter close_session for osd.5 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6804/2277917372 conn(0x7ff390126210 msgr2=0x7ff390128600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6804/2277917372 conn(0x7ff390126210 0x7ff390128600 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff390121060 osd=5 1 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 10 client.37799.objecter close_session for osd.6 2024-02-13T00:49:44.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff390129000 msgr2=0x7ff390129480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6808/1506640341 conn(0x7ff390129000 0x7ff390129480 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39012b2c0 osd=6 1 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 10 client.37799.objecter close_session for osd.7 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff39012da00 msgr2=0x7ff39012de60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.181:6812/1964192772 conn(0x7ff39012da00 0x7ff39012de60 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 20 client.37799.objecter put_session s=0x7ff39012b040 osd=7 1 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 20 client.37799.objecter shutdown clearing up homeless session... 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 10 client.37799.objecter successfully canceled tick 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff38ed9d700 1 -- 172.21.15.162:0/2859439938 reap_dead start 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff378062c90 msgr2=0x7ff378065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff378062c90 0x7ff378065150 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7ff384005010 tx=0x7ff384009cb0).stop 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3900f8c40 msgr2=0x7ff39010e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.759+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3900f8c40 0x7ff39010e210 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7ff380002820 tx=0x7ff380009f60).stop 2024-02-13T00:49:44.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.760+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 shutdown_connections 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.760+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3900f8c40 0x7ff39010e210 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.760+0000 7ff394b84700 1 --2- 172.21.15.162:0/2859439938 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff378062c90 0x7ff378065150 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.760+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 >> 172.21.15.162:0/2859439938 conn(0x7ff3900f2750 msgr2=0x7ff3900f33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.760+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 shutdown_connections 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.760+0000 7ff394b84700 1 -- 172.21.15.162:0/2859439938 wait complete. 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:44.760+0000 7ff394b84700 1 librados: shutdown 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.768 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:49:44.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:44.788 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:42.850275+0000 osd.0 (osd.0) 221 : cluster [DBG] 28.1f scrub starts 2024-02-13T00:49:44.789 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:42.854577+0000 osd.0 (osd.0) 222 : cluster [DBG] 28.1f scrub ok 2024-02-13T00:49:44.789 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:43.064450+0000 osd.6 (osd.6) 169 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:49:44.789 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:43.067611+0000 osd.6 (osd.6) 170 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:49:44.789 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:44.509031+0000 mon.a (mon.0) 2985 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-97", "overlaypool": "test-rados-api-smithi162-84057-97-cache"}]': finished 2024-02-13T00:49:44.789 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:44.509125+0000 mon.a (mon.0) 2986 : cluster [DBG] osdmap e570: 8 total, 8 up, 4 in 2024-02-13T00:49:44.789 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:44.509794+0000 mon.c (mon.2) 445 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-97-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:49:44.789 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:44.510140+0000 mon.a (mon.0) 2987 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-97-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:49:44.789 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:42.850275+0000 osd.0 (osd.0) 221 : cluster [DBG] 28.1f scrub starts 2024-02-13T00:49:44.789 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:42.854577+0000 osd.0 (osd.0) 222 : cluster [DBG] 28.1f scrub ok 2024-02-13T00:49:44.789 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:43.064450+0000 osd.6 (osd.6) 169 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:49:44.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:43.067611+0000 osd.6 (osd.6) 170 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:49:44.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:44.509031+0000 mon.a (mon.0) 2985 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-97", "overlaypool": "test-rados-api-smithi162-84057-97-cache"}]': finished 2024-02-13T00:49:44.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:44.509125+0000 mon.a (mon.0) 2986 : cluster [DBG] osdmap e570: 8 total, 8 up, 4 in 2024-02-13T00:49:44.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:44.509794+0000 mon.c (mon.2) 445 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-97-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:49:44.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:44.510140+0000 mon.a (mon.0) 2987 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-97-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:49:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:42.850275+0000 osd.0 (osd.0) 221 : cluster [DBG] 28.1f scrub starts 2024-02-13T00:49:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:42.854577+0000 osd.0 (osd.0) 222 : cluster [DBG] 28.1f scrub ok 2024-02-13T00:49:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:43.064450+0000 osd.6 (osd.6) 169 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:49:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:43.067611+0000 osd.6 (osd.6) 170 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:49:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:44.509031+0000 mon.a (mon.0) 2985 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-97", "overlaypool": "test-rados-api-smithi162-84057-97-cache"}]': finished 2024-02-13T00:49:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:44.509125+0000 mon.a (mon.0) 2986 : cluster [DBG] osdmap e570: 8 total, 8 up, 4 in 2024-02-13T00:49:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:44.509794+0000 mon.c (mon.2) 445 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-97-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:49:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:44.510140+0000 mon.a (mon.0) 2987 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-97-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:49:44.809 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:44.957 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:44.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:45.062 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:45.094 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:45.115 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:45.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:45.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:45.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:45.248 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:45.268 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:45.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:45.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:45.400 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:45.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:49:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:49:45.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:44.479695+0000 mgr.y (mgr.14544) 491 : cluster [DBG] pgmap v854: 236 pgs: 32 unknown, 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:45.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:44.615255+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? 172.21.15.162:0/2859439938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:45.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:45.484742+0000 mon.a (mon.0) 2988 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:49:45.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:45.490371+0000 mon.a (mon.0) 2989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-97-cache", "mode": "writeback"}]': finished 2024-02-13T00:49:45.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:45.490441+0000 mon.a (mon.0) 2990 : cluster [DBG] osdmap e571: 8 total, 8 up, 4 in 2024-02-13T00:49:45.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:45.517253+0000 mon.c (mon.2) 447 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-97"}]: dispatch 2024-02-13T00:49:45.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:45.517578+0000 mon.a (mon.0) 2991 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-97"}]: dispatch 2024-02-13T00:49:45.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:44.479695+0000 mgr.y (mgr.14544) 491 : cluster [DBG] pgmap v854: 236 pgs: 32 unknown, 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:45.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:44.615255+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? 172.21.15.162:0/2859439938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:45.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:45.484742+0000 mon.a (mon.0) 2988 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:49:45.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:45.490371+0000 mon.a (mon.0) 2989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-97-cache", "mode": "writeback"}]': finished 2024-02-13T00:49:45.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:45.490441+0000 mon.a (mon.0) 2990 : cluster [DBG] osdmap e571: 8 total, 8 up, 4 in 2024-02-13T00:49:45.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:45.517253+0000 mon.c (mon.2) 447 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-97"}]: dispatch 2024-02-13T00:49:45.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:45.517578+0000 mon.a (mon.0) 2991 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-97"}]: dispatch 2024-02-13T00:49:45.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:44.479695+0000 mgr.y (mgr.14544) 491 : cluster [DBG] pgmap v854: 236 pgs: 32 unknown, 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:45.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:44.615255+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? 172.21.15.162:0/2859439938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:45.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:45.484742+0000 mon.a (mon.0) 2988 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:49:45.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:45.490371+0000 mon.a (mon.0) 2989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-97-cache", "mode": "writeback"}]': finished 2024-02-13T00:49:45.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:45.490441+0000 mon.a (mon.0) 2990 : cluster [DBG] osdmap e571: 8 total, 8 up, 4 in 2024-02-13T00:49:45.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:45.517253+0000 mon.c (mon.2) 447 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-97"}]: dispatch 2024-02-13T00:49:45.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:45.517578+0000 mon.a (mon.0) 2991 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-97"}]: dispatch 2024-02-13T00:49:46.455 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:49:46] "GET /metrics HTTP/1.1" 200 185774 "" "Prometheus/2.33.4" 2024-02-13T00:49:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:45.050281+0000 osd.6 (osd.6) 171 : cluster [DBG] 5.f scrub starts 2024-02-13T00:49:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:45.051533+0000 osd.6 (osd.6) 172 : cluster [DBG] 5.f scrub ok 2024-02-13T00:49:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:45.759939+0000 mgr.y (mgr.14544) 492 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:46.497734+0000 mon.a (mon.0) 2992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-97"}]': finished 2024-02-13T00:49:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:46.497822+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e572: 8 total, 8 up, 4 in 2024-02-13T00:49:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:46.500792+0000 mon.c (mon.2) 448 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:46.501133+0000 mon.a (mon.0) 2994 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:45.050281+0000 osd.6 (osd.6) 171 : cluster [DBG] 5.f scrub starts 2024-02-13T00:49:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:45.051533+0000 osd.6 (osd.6) 172 : cluster [DBG] 5.f scrub ok 2024-02-13T00:49:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:45.759939+0000 mgr.y (mgr.14544) 492 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:46.497734+0000 mon.a (mon.0) 2992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-97"}]': finished 2024-02-13T00:49:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:46.497822+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e572: 8 total, 8 up, 4 in 2024-02-13T00:49:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:46.500792+0000 mon.c (mon.2) 448 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:46.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:46.501133+0000 mon.a (mon.0) 2994 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:45.050281+0000 osd.6 (osd.6) 171 : cluster [DBG] 5.f scrub starts 2024-02-13T00:49:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:45.051533+0000 osd.6 (osd.6) 172 : cluster [DBG] 5.f scrub ok 2024-02-13T00:49:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:45.759939+0000 mgr.y (mgr.14544) 492 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:46.497734+0000 mon.a (mon.0) 2992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-97"}]': finished 2024-02-13T00:49:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:46.497822+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e572: 8 total, 8 up, 4 in 2024-02-13T00:49:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:46.500792+0000 mon.c (mon.2) 448 : audit [INF] from='client.? 172.21.15.162:0/3528749676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:46.501133+0000 mon.a (mon.0) 2994 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]: dispatch 2024-02-13T00:49:47.350 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:47.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:47.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:47.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:49:47.515 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:47.567 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:47.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:49:47.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:46.098349+0000 osd.6 (osd.6) 173 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:49:47.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:46.101868+0000 osd.6 (osd.6) 174 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:49:47.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:46.480156+0000 mgr.y (mgr.14544) 493 : cluster [DBG] pgmap v857: 236 pgs: 236 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:47.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:46.850570+0000 osd.0 (osd.0) 223 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:49:47.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:46.851481+0000 osd.0 (osd.0) 224 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:49:47.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:46.925591+0000 osd.1 (osd.1) 245 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:49:47.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:46.928647+0000 osd.1 (osd.1) 246 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:49:47.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:47.495136+0000 mon.a (mon.0) 2995 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:49:47.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:47.501180+0000 mon.a (mon.0) 2996 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]': finished 2024-02-13T00:49:47.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:47.501248+0000 mon.a (mon.0) 2997 : cluster [DBG] osdmap e573: 8 total, 8 up, 4 in 2024-02-13T00:49:47.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:46.098349+0000 osd.6 (osd.6) 173 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:49:47.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:46.101868+0000 osd.6 (osd.6) 174 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:49:47.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:46.480156+0000 mgr.y (mgr.14544) 493 : cluster [DBG] pgmap v857: 236 pgs: 236 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:47.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:46.850570+0000 osd.0 (osd.0) 223 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:49:47.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:46.851481+0000 osd.0 (osd.0) 224 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:49:47.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:46.925591+0000 osd.1 (osd.1) 245 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:49:47.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:46.928647+0000 osd.1 (osd.1) 246 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:49:47.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:47.495136+0000 mon.a (mon.0) 2995 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:49:47.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:47.501180+0000 mon.a (mon.0) 2996 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]': finished 2024-02-13T00:49:47.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:47.501248+0000 mon.a (mon.0) 2997 : cluster [DBG] osdmap e573: 8 total, 8 up, 4 in 2024-02-13T00:49:47.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:47.704 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:47.726 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:47.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:47.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:47.858 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:46.098349+0000 osd.6 (osd.6) 173 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:49:47.858 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:46.101868+0000 osd.6 (osd.6) 174 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:49:47.858 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:46.480156+0000 mgr.y (mgr.14544) 493 : cluster [DBG] pgmap v857: 236 pgs: 236 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:47.858 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:46.850570+0000 osd.0 (osd.0) 223 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:49:47.858 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:46.851481+0000 osd.0 (osd.0) 224 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:49:47.858 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:46.925591+0000 osd.1 (osd.1) 245 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:49:47.858 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:46.928647+0000 osd.1 (osd.1) 246 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:49:47.858 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:47.495136+0000 mon.a (mon.0) 2995 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:49:47.858 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:47.501180+0000 mon.a (mon.0) 2996 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-97", "tierpool": "test-rados-api-smithi162-84057-97-cache"}]': finished 2024-02-13T00:49:47.858 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:47.501248+0000 mon.a (mon.0) 2997 : cluster [DBG] osdmap e573: 8 total, 8 up, 4 in 2024-02-13T00:49:47.859 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:47.879 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:47.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:47.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:47.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:48.013 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:48.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:49:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.068+0000 7f6dc3648700 1 -- 172.21.15.162:0/2613127051 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6dbc0f8d10 msgr2=0x7f6dbc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.068+0000 7f6dc3648700 1 --2- 172.21.15.162:0/2613127051 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6dbc0f8d10 0x7f6dbc0f90f0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7f6db8004870 tx=0x7f6db8009cb0).stop 2024-02-13T00:49:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.068+0000 7f6dc3648700 1 -- 172.21.15.162:0/2613127051 shutdown_connections 2024-02-13T00:49:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.068+0000 7f6dc3648700 1 --2- 172.21.15.162:0/2613127051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dbc102270 0x7f6dbc104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.068+0000 7f6dc3648700 1 --2- 172.21.15.162:0/2613127051 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6dbc0f8d10 0x7f6dbc0f90f0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.068+0000 7f6dc3648700 1 --2- 172.21.15.162:0/2613127051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6dbc0f95e0 0x7f6dbc101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.068+0000 7f6dc3648700 1 -- 172.21.15.162:0/2613127051 >> 172.21.15.162:0/2613127051 conn(0x7f6dbc06eb90 msgr2=0x7f6dbc0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.068+0000 7f6dc3648700 1 -- 172.21.15.162:0/2613127051 shutdown_connections 2024-02-13T00:49:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.068+0000 7f6dc3648700 1 -- 172.21.15.162:0/2613127051 wait complete. 2024-02-13T00:49:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.068+0000 7f6dc3648700 1 librados: starting msgr at 2024-02-13T00:49:48.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.068+0000 7f6dc3648700 1 librados: starting objecter 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc3648700 1 Processor -- start 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc3648700 1 -- start start 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc3648700 1 librados: setting wanted keys 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc3648700 1 librados: calling monclient init 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc3648700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6dbc0f8d10 0x7f6dbc1103e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc3648700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6dbc0f95e0 0x7f6dbc1108d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc3648700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dbc102270 0x7f6dbc10a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc3648700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc10aa70 con 0x7f6dbc102270 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc3648700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc10abe0 con 0x7f6dbc0f95e0 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc3648700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc10ada0 con 0x7f6dbc0f8d10 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc13e4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6dbc0f8d10 0x7f6dbc1103e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc13e4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6dbc0f8d10 0x7f6dbc1103e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:42912/0 (socket says 172.21.15.162:42912) 2024-02-13T00:49:48.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc13e4700 1 -- 172.21.15.162:0/2090214977 learned_addr learned my addr 172.21.15.162:0/2090214977 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc1be5700 1 --2- 172.21.15.162:0/2090214977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dbc102270 0x7f6dbc10a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc0be3700 1 --2- 172.21.15.162:0/2090214977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6dbc0f95e0 0x7f6dbc1108d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc13e4700 1 -- 172.21.15.162:0/2090214977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6dbc0f95e0 msgr2=0x7f6dbc1108d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc13e4700 1 --2- 172.21.15.162:0/2090214977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6dbc0f95e0 0x7f6dbc1108d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc13e4700 1 -- 172.21.15.162:0/2090214977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dbc102270 msgr2=0x7f6dbc10a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc13e4700 1 --2- 172.21.15.162:0/2090214977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dbc102270 0x7f6dbc10a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc13e4700 1 -- 172.21.15.162:0/2090214977 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db40097c0 con 0x7f6dbc0f8d10 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.069+0000 7f6dc0be3700 1 --2- 172.21.15.162:0/2090214977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6dbc0f95e0 0x7f6dbc1108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.070+0000 7f6dc13e4700 1 --2- 172.21.15.162:0/2090214977 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6dbc0f8d10 0x7f6dbc1103e0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 rx=0x7f6db8009eb0 tx=0x7f6db800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.070+0000 7f6db27fc700 10 client.?.objecter ms_handle_connect 0x7f6dbc0f8d10 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.070+0000 7f6db27fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.070+0000 7f6db27fc700 1 -- 172.21.15.162:0/2090214977 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db800e070 con 0x7f6dbc0f8d10 2024-02-13T00:49:48.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.070+0000 7f6db27fc700 1 -- 172.21.15.162:0/2090214977 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6db8013da0 con 0x7f6dbc0f8d10 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.070+0000 7f6dc3648700 1 -- 172.21.15.162:0/2090214977 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db8005040 con 0x7f6dbc0f8d10 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.070+0000 7f6db27fc700 1 -- 172.21.15.162:0/2090214977 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db8019070 con 0x7f6dbc0f8d10 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.070+0000 7f6dc3648700 10 client.37805.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.070+0000 7f6dc3648700 1 -- 172.21.15.162:0/2090214977 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dbc10b240 con 0x7f6dbc0f8d10 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.070+0000 7f6dc3648700 1 librados: init done 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.071+0000 7f6dc3648700 1 -- 172.21.15.162:0/2090214977 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dbc0475e0 con 0x7f6dbc0f8d10 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.071+0000 7f6db27fc700 1 -- 172.21.15.162:0/2090214977 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6db801e420 con 0x7f6dbc0f8d10 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.071+0000 7f6db27fc700 1 --2- 172.21.15.162:0/2090214977 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6da8062c90 0x7f6da8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.072+0000 7f6db27fc700 1 -- 172.21.15.162:0/2090214977 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(573..573 src has 449..573) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f6db807e110 con 0x7f6dbc0f8d10 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.072+0000 7f6db27fc700 10 client.37805.objecter ms_dispatch 0x7f6dbc001550 osd_map(573..573 src has 449..573) v4 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.072+0000 7f6db27fc700 3 client.37805.objecter handle_osd_map got epochs [573,573] > 0 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.072+0000 7f6db27fc700 3 client.37805.objecter handle_osd_map decoding full epoch 573 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.072+0000 7f6dc0be3700 1 --2- 172.21.15.162:0/2090214977 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6da8062c90 0x7f6da8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:48.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.072+0000 7f6db27fc700 20 client.37805.objecter dump_active .. 0 homeless 2024-02-13T00:49:48.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.073+0000 7f6db27fc700 10 client.37805.objecter ms_handle_connect 0x7f6da8062c90 2024-02-13T00:49:48.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.073+0000 7f6dc0be3700 1 --2- 172.21.15.162:0/2090214977 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6da8062c90 0x7f6da8065150 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7f6dac007fd0 tx=0x7f6dac009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:48.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.076+0000 7f6db27fc700 1 -- 172.21.15.162:0/2090214977 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6db80512b0 con 0x7f6dbc0f8d10 2024-02-13T00:49:48.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.246+0000 7f6dc3648700 1 -- 172.21.15.162:0/2090214977 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6dbc10bba0 con 0x7f6da8062c90 2024-02-13T00:49:48.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.262+0000 7f6db27fc700 1 -- 172.21.15.162:0/2090214977 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+580081 (secure 0 0 0) 0x7f6dac00e690 con 0x7f6da8062c90 2024-02-13T00:49:48.274 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:49:48.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 10 librados: watch_flush enter 2024-02-13T00:49:48.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 10 librados: watch_flush exit 2024-02-13T00:49:48.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 20 client.37805.objecter shutdown clearing up homeless session... 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 10 client.37805.objecter successfully canceled tick 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 -- 172.21.15.162:0/2090214977 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6da8062c90 msgr2=0x7f6da8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 --2- 172.21.15.162:0/2090214977 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6da8062c90 0x7f6da8065150 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7f6dac007fd0 tx=0x7f6dac009040).stop 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 -- 172.21.15.162:0/2090214977 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6dbc0f8d10 msgr2=0x7f6dbc1103e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 --2- 172.21.15.162:0/2090214977 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6dbc0f8d10 0x7f6dbc1103e0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 rx=0x7f6db8009eb0 tx=0x7f6db800b040).stop 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 -- 172.21.15.162:0/2090214977 shutdown_connections 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 --2- 172.21.15.162:0/2090214977 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6da8062c90 0x7f6da8065150 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 --2- 172.21.15.162:0/2090214977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6dbc102270 0x7f6dbc10a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 --2- 172.21.15.162:0/2090214977 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6dbc0f8d10 0x7f6dbc1103e0 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 --2- 172.21.15.162:0/2090214977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6dbc0f95e0 0x7f6dbc1108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 -- 172.21.15.162:0/2090214977 >> 172.21.15.162:0/2090214977 conn(0x7f6dbc06eb90 msgr2=0x7f6dbc0f7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 -- 172.21.15.162:0/2090214977 shutdown_connections 2024-02-13T00:49:48.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 -- 172.21.15.162:0/2090214977 wait complete. 2024-02-13T00:49:48.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.268+0000 7f6dc3648700 1 librados: shutdown 2024-02-13T00:49:48.281 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:49:48.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.188+0000 7fed0b48d700 1 -- 172.21.15.162:0/922766918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed040f6880 msgr2=0x7fed04104cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:48.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.188+0000 7fed0b48d700 1 --2- 172.21.15.162:0/922766918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed040f6880 0x7fed04104cc0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 rx=0x7fecfc009be0 tx=0x7fecfc004d10).stop 2024-02-13T00:49:48.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.188+0000 7fed0b48d700 1 -- 172.21.15.162:0/922766918 shutdown_connections 2024-02-13T00:49:48.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.188+0000 7fed0b48d700 1 --2- 172.21.15.162:0/922766918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed040f6880 0x7fed04104cc0 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.188+0000 7fed0b48d700 1 --2- 172.21.15.162:0/922766918 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed040fde50 0x7fed040fe230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.188+0000 7fed0b48d700 1 --2- 172.21.15.162:0/922766918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed040fe7b0 0x7fed040f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.188+0000 7fed0b48d700 1 -- 172.21.15.162:0/922766918 >> 172.21.15.162:0/922766918 conn(0x7fed04070080 msgr2=0x7fed040f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:48.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.188+0000 7fed0b48d700 1 -- 172.21.15.162:0/922766918 shutdown_connections 2024-02-13T00:49:48.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.188+0000 7fed0b48d700 1 -- 172.21.15.162:0/922766918 wait complete. 2024-02-13T00:49:48.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.188+0000 7fed0b48d700 1 librados: starting msgr at 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.188+0000 7fed0b48d700 1 librados: starting objecter 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed0b48d700 1 Processor -- start 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed0b48d700 1 -- start start 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed0b48d700 1 librados: setting wanted keys 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed0b48d700 1 librados: calling monclient init 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed0b48d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed040f6880 0x7fed0410e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed0b48d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed040fde50 0x7fed0410e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed0b48d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed040fe7b0 0x7fed04108420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed0b48d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed04108910 con 0x7fed040f6880 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed0b48d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed04108a80 con 0x7fed040fe7b0 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed0b48d700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed04108c40 con 0x7fed040fde50 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed09229700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed040f6880 0x7fed0410e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:48.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed09229700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed040f6880 0x7fed0410e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:57610/0 (socket says 172.21.15.162:57610) 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed09229700 1 -- 172.21.15.162:0/1258124582 learned_addr learned my addr 172.21.15.162:0/1258124582 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed08a28700 1 --2- 172.21.15.162:0/1258124582 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed040fde50 0x7fed0410e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed09a2a700 1 --2- 172.21.15.162:0/1258124582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed040fe7b0 0x7fed04108420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed09229700 1 -- 172.21.15.162:0/1258124582 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed040fde50 msgr2=0x7fed0410e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed09229700 1 --2- 172.21.15.162:0/1258124582 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed040fde50 0x7fed0410e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed09229700 1 -- 172.21.15.162:0/1258124582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed040fe7b0 msgr2=0x7fed04108420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed09229700 1 --2- 172.21.15.162:0/1258124582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed040fe7b0 0x7fed04108420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed09229700 1 -- 172.21.15.162:0/1258124582 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0097c0 con 0x7fed040f6880 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed09a2a700 1 --2- 172.21.15.162:0/1258124582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed040fe7b0 0x7fed04108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.189+0000 7fed08a28700 1 --2- 172.21.15.162:0/1258124582 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed040fde50 0x7fed0410e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.190+0000 7fed09229700 1 --2- 172.21.15.162:0/1258124582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed040f6880 0x7fed0410e280 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7fed0000e970 tx=0x7fed0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.190+0000 7fecfa7fc700 10 client.?.objecter ms_handle_connect 0x7fed040f6880 2024-02-13T00:49:48.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.190+0000 7fecfa7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.190+0000 7fecfa7fc700 1 -- 172.21.15.162:0/1258124582 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0000ee60 con 0x7fed040f6880 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.190+0000 7fecfa7fc700 1 -- 172.21.15.162:0/1258124582 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed0000f040 con 0x7fed040f6880 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.190+0000 7fed0b48d700 1 -- 172.21.15.162:0/1258124582 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed04108f20 con 0x7fed040f6880 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.190+0000 7fecfa7fc700 1 -- 172.21.15.162:0/1258124582 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00007ea0 con 0x7fed040f6880 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.190+0000 7fed0b48d700 10 client.27711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.190+0000 7fed0b48d700 1 -- 172.21.15.162:0/1258124582 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed041092a0 con 0x7fed040f6880 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.190+0000 7fed0b48d700 1 librados: init done 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.191+0000 7fed0b48d700 1 -- 172.21.15.162:0/1258124582 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0405fb80 con 0x7fed040f6880 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.191+0000 7fecfa7fc700 1 -- 172.21.15.162:0/1258124582 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fed0000cb10 con 0x7fed040f6880 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.192+0000 7fecfa7fc700 1 --2- 172.21.15.162:0/1258124582 >> v2:172.21.15.162:6800/2866534053 conn(0x7fecf0062c90 0x7fecf0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.192+0000 7fecfa7fc700 1 -- 172.21.15.162:0/1258124582 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(573..573 src has 449..573) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fed0007d130 con 0x7fed040f6880 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.192+0000 7fecfa7fc700 10 client.27711.objecter ms_dispatch 0x7fed04001550 osd_map(573..573 src has 449..573) v4 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.192+0000 7fecfa7fc700 3 client.27711.objecter handle_osd_map got epochs [573,573] > 0 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.192+0000 7fecfa7fc700 3 client.27711.objecter handle_osd_map decoding full epoch 573 2024-02-13T00:49:48.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.192+0000 7fed08a28700 1 --2- 172.21.15.162:0/1258124582 >> v2:172.21.15.162:6800/2866534053 conn(0x7fecf0062c90 0x7fecf0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:48.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.192+0000 7fecfa7fc700 20 client.27711.objecter dump_active .. 0 homeless 2024-02-13T00:49:48.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.192+0000 7fecfa7fc700 10 client.27711.objecter ms_handle_connect 0x7fecf0062c90 2024-02-13T00:49:48.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.192+0000 7fed08a28700 1 --2- 172.21.15.162:0/1258124582 >> v2:172.21.15.162:6800/2866534053 conn(0x7fecf0062c90 0x7fecf0065150 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 rx=0x7fed04109bc0 tx=0x7fecf400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:48.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.196+0000 7fecfa7fc700 1 -- 172.21.15.162:0/1258124582 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fed0004af60 con 0x7fed040f6880 2024-02-13T00:49:48.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.390+0000 7fed0b48d700 1 -- 172.21.15.162:0/1258124582 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fed040475e0 con 0x7fed040f6880 2024-02-13T00:49:48.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.505+0000 7fecfa7fc700 1 -- 172.21.15.162:0/1258124582 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v574) v1 ==== 87+0+0 (secure 0 0 0) 0x7fed00019020 con 0x7fed040f6880 2024-02-13T00:49:48.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.507+0000 7fed0b48d700 10 librados: watch_flush enter 2024-02-13T00:49:48.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.507+0000 7fed0b48d700 10 librados: watch_flush exit 2024-02-13T00:49:48.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.507+0000 7fed0b48d700 20 client.27711.objecter shutdown clearing up homeless session... 2024-02-13T00:49:48.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.507+0000 7fed0b48d700 10 client.27711.objecter successfully canceled tick 2024-02-13T00:49:48.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 -- 172.21.15.162:0/1258124582 >> v2:172.21.15.162:6800/2866534053 conn(0x7fecf0062c90 msgr2=0x7fecf0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 --2- 172.21.15.162:0/1258124582 >> v2:172.21.15.162:6800/2866534053 conn(0x7fecf0062c90 0x7fecf0065150 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 rx=0x7fed04109bc0 tx=0x7fecf400b040).stop 2024-02-13T00:49:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 -- 172.21.15.162:0/1258124582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed040f6880 msgr2=0x7fed0410e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 --2- 172.21.15.162:0/1258124582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed040f6880 0x7fed0410e280 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7fed0000e970 tx=0x7fed0000b040).stop 2024-02-13T00:49:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 -- 172.21.15.162:0/1258124582 shutdown_connections 2024-02-13T00:49:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 --2- 172.21.15.162:0/1258124582 >> v2:172.21.15.162:6800/2866534053 conn(0x7fecf0062c90 0x7fecf0065150 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 --2- 172.21.15.162:0/1258124582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed040fe7b0 0x7fed04108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 --2- 172.21.15.162:0/1258124582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed040f6880 0x7fed0410e280 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 --2- 172.21.15.162:0/1258124582 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed040fde50 0x7fed0410e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 -- 172.21.15.162:0/1258124582 >> 172.21.15.162:0/1258124582 conn(0x7fed04070080 msgr2=0x7fed0406d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 -- 172.21.15.162:0/1258124582 shutdown_connections 2024-02-13T00:49:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 -- 172.21.15.162:0/1258124582 wait complete. 2024-02-13T00:49:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:48.508+0000 7fed0b48d700 1 librados: shutdown 2024-02-13T00:49:48.514 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:49:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:47.074579+0000 osd.6 (osd.6) 175 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:49:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:47.075732+0000 osd.6 (osd.6) 176 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:49:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:47.922491+0000 osd.1 (osd.1) 247 : cluster [DBG] 4.4 scrub starts 2024-02-13T00:49:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:47.928544+0000 osd.1 (osd.1) 248 : cluster [DBG] 4.4 scrub ok 2024-02-13T00:49:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:48.391204+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? 172.21.15.162:0/1258124582' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:49:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:48.496407+0000 mon.a (mon.0) 2999 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:48.505660+0000 mon.a (mon.0) 3000 : audit [INF] from='client.? 172.21.15.162:0/1258124582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:49:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:48.505713+0000 mon.a (mon.0) 3001 : cluster [DBG] osdmap e574: 8 total, 8 up, 4 in 2024-02-13T00:49:48.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:47.074579+0000 osd.6 (osd.6) 175 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:49:48.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:47.075732+0000 osd.6 (osd.6) 176 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:49:48.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:47.922491+0000 osd.1 (osd.1) 247 : cluster [DBG] 4.4 scrub starts 2024-02-13T00:49:48.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:47.928544+0000 osd.1 (osd.1) 248 : cluster [DBG] 4.4 scrub ok 2024-02-13T00:49:48.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:48.391204+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? 172.21.15.162:0/1258124582' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:49:48.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:48.496407+0000 mon.a (mon.0) 2999 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:48.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:48.505660+0000 mon.a (mon.0) 3000 : audit [INF] from='client.? 172.21.15.162:0/1258124582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:49:48.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:48.505713+0000 mon.a (mon.0) 3001 : cluster [DBG] osdmap e574: 8 total, 8 up, 4 in 2024-02-13T00:49:48.862 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:48.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:48.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:48.994 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:49.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:49:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:49:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:49:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:47.074579+0000 osd.6 (osd.6) 175 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:49:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:47.075732+0000 osd.6 (osd.6) 176 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:49:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:47.922491+0000 osd.1 (osd.1) 247 : cluster [DBG] 4.4 scrub starts 2024-02-13T00:49:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:47.928544+0000 osd.1 (osd.1) 248 : cluster [DBG] 4.4 scrub ok 2024-02-13T00:49:49.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:48.391204+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? 172.21.15.162:0/1258124582' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:49:49.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:48.496407+0000 mon.a (mon.0) 2999 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:49.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:48.505660+0000 mon.a (mon.0) 3000 : audit [INF] from='client.? 172.21.15.162:0/1258124582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:49:49.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:48.505713+0000 mon.a (mon.0) 3001 : cluster [DBG] osdmap e574: 8 total, 8 up, 4 in 2024-02-13T00:49:49.409 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":859,"stamp":"2024-02-13T00:49:47.499336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":2216,"num_read_kb":1934,"num_write":914,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":13678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1048,"ondisk_log_size":1048,"up":708,"acting":708,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":675,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1215352,"kb_used_data":10072,"kb_used_omap":0,"kb_used_meta":1205248,"kb_avail":373765256,"statfs":{"total":383980142592,"available":382735622144,"internally_reserved":0,"allocated":10313728,"data_stored":4622016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1234173952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.994562"},"pg_stats":[{"pgid":"231.1e","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.524017+0000","last_change":"2024-02-13T00:49:42.523029+0000","last_active":"2024-02-13T00:49:42.524017+0000","last_peered":"2024-02-13T00:49:42.524017+0000","last_clean":"2024-02-13T00:49:42.524017+0000","last_became_active":"2024-02-13T00:49:42.522751+0000","last_became_peered":"2024-02-13T00:49:42.522751+0000","last_unstale":"2024-02-13T00:49:42.524017+0000","last_undegraded":"2024-02-13T00:49:42.524017+0000","last_fullsized":"2024-02-13T00:49:42.524017+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.1f","version":"0'0","reported_seq":13,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507636+0000","last_change":"2024-02-13T00:49:40.686062+0000","last_active":"2024-02-13T00:49:43.507636+0000","last_peered":"2024-02-13T00:49:43.507636+0000","last_clean":"2024-02-13T00:49:43.507636+0000","last_became_active":"2024-02-13T00:49:40.685787+0000","last_became_peered":"2024-02-13T00:49:40.685787+0000","last_unstale":"2024-02-13T00:49:43.507636+0000","last_undegraded":"2024-02-13T00:49:43.507636+0000","last_fullsized":"2024-02-13T00:49:43.507636+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"231.1f","version":"0'0","reported_seq":11,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508485+0000","last_change":"2024-02-13T00:49:42.526921+0000","last_active":"2024-02-13T00:49:43.508485+0000","last_peered":"2024-02-13T00:49:43.508485+0000","last_clean":"2024-02-13T00:49:43.508485+0000","last_became_active":"2024-02-13T00:49:42.526409+0000","last_became_peered":"2024-02-13T00:49:42.526409+0000","last_unstale":"2024-02-13T00:49:43.508485+0000","last_undegraded":"2024-02-13T00:49:43.508485+0000","last_fullsized":"2024-02-13T00:49:43.508485+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"230.1e","version":"0'0","reported_seq":13,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.510099+0000","last_change":"2024-02-13T00:49:40.678589+0000","last_active":"2024-02-13T00:49:43.510099+0000","last_peered":"2024-02-13T00:49:43.510099+0000","last_clean":"2024-02-13T00:49:43.510099+0000","last_became_active":"2024-02-13T00:49:40.678333+0000","last_became_peered":"2024-02-13T00:49:40.678333+0000","last_unstale":"2024-02-13T00:49:43.510099+0000","last_undegraded":"2024-02-13T00:49:43.510099+0000","last_fullsized":"2024-02-13T00:49:43.510099+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"231.1c","version":"0'0","reported_seq":11,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508642+0000","last_change":"2024-02-13T00:49:42.525949+0000","last_active":"2024-02-13T00:49:43.508642+0000","last_peered":"2024-02-13T00:49:43.508642+0000","last_clean":"2024-02-13T00:49:43.508642+0000","last_became_active":"2024-02-13T00:49:42.525658+0000","last_became_peered":"2024-02-13T00:49:42.525658+0000","last_unstale":"2024-02-13T00:49:43.508642+0000","last_undegraded":"2024-02-13T00:49:43.508642+0000","last_fullsized":"2024-02-13T00:49:43.508642+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"230.1d","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.515405+0000","last_change":"2024-02-13T00:49:40.518298+0000","last_active":"2024-02-13T00:49:42.515405+0000","last_peered":"2024-02-13T00:49:42.515405+0000","last_clean":"2024-02-13T00:49:42.515405+0000","last_became_active":"2024-02-13T00:49:40.517063+0000","last_became_peered":"2024-02-13T00:49:40.517063+0000","last_unstale":"2024-02-13T00:49:42.515405+0000","last_undegraded":"2024-02-13T00:49:42.515405+0000","last_fullsized":"2024-02-13T00:49:42.515405+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.1d","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.524907+0000","last_change":"2024-02-13T00:49:42.524516+0000","last_active":"2024-02-13T00:49:42.524907+0000","last_peered":"2024-02-13T00:49:42.524907+0000","last_clean":"2024-02-13T00:49:42.524907+0000","last_became_active":"2024-02-13T00:49:42.524221+0000","last_became_peered":"2024-02-13T00:49:42.524221+0000","last_unstale":"2024-02-13T00:49:42.524907+0000","last_undegraded":"2024-02-13T00:49:42.524907+0000","last_fullsized":"2024-02-13T00:49:42.524907+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"230.1c","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.513401+0000","last_change":"2024-02-13T00:49:40.679655+0000","last_active":"2024-02-13T00:49:42.513401+0000","last_peered":"2024-02-13T00:49:42.513401+0000","last_clean":"2024-02-13T00:49:42.513401+0000","last_became_active":"2024-02-13T00:49:40.679173+0000","last_became_peered":"2024-02-13T00:49:40.679173+0000","last_unstale":"2024-02-13T00:49:42.513401+0000","last_undegraded":"2024-02-13T00:49:42.513401+0000","last_fullsized":"2024-02-13T00:49:42.513401+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.1a","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.519885+0000","last_change":"2024-02-13T00:49:42.519378+0000","last_active":"2024-02-13T00:49:42.519885+0000","last_peered":"2024-02-13T00:49:42.519885+0000","last_clean":"2024-02-13T00:49:42.519885+0000","last_became_active":"2024-02-13T00:49:42.519075+0000","last_became_peered":"2024-02-13T00:49:42.519075+0000","last_unstale":"2024-02-13T00:49:42.519885+0000","last_undegraded":"2024-02-13T00:49:42.519885+0000","last_fullsized":"2024-02-13T00:49:42.519885+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"230.1b","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.511334+0000","last_change":"2024-02-13T00:49:40.683964+0000","last_active":"2024-02-13T00:49:42.511334+0000","last_peered":"2024-02-13T00:49:42.511334+0000","last_clean":"2024-02-13T00:49:42.511334+0000","last_became_active":"2024-02-13T00:49:40.683647+0000","last_became_peered":"2024-02-13T00:49:40.683647+0000","last_unstale":"2024-02-13T00:49:42.511334+0000","last_undegraded":"2024-02-13T00:49:42.511334+0000","last_fullsized":"2024-02-13T00:49:42.511334+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.1b","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.523601+0000","last_change":"2024-02-13T00:49:42.523115+0000","last_active":"2024-02-13T00:49:42.523601+0000","last_peered":"2024-02-13T00:49:42.523601+0000","last_clean":"2024-02-13T00:49:42.523601+0000","last_became_active":"2024-02-13T00:49:42.522774+0000","last_became_peered":"2024-02-13T00:49:42.522774+0000","last_unstale":"2024-02-13T00:49:42.523601+0000","last_undegraded":"2024-02-13T00:49:42.523601+0000","last_fullsized":"2024-02-13T00:49:42.523601+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"230.1a","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512222+0000","last_change":"2024-02-13T00:49:40.684399+0000","last_active":"2024-02-13T00:49:42.512222+0000","last_peered":"2024-02-13T00:49:42.512222+0000","last_clean":"2024-02-13T00:49:42.512222+0000","last_became_active":"2024-02-13T00:49:40.684043+0000","last_became_peered":"2024-02-13T00:49:40.684043+0000","last_unstale":"2024-02-13T00:49:42.512222+0000","last_undegraded":"2024-02-13T00:49:42.512222+0000","last_fullsized":"2024-02-13T00:49:42.512222+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.18","version":"0'0","reported_seq":11,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506514+0000","last_change":"2024-02-13T00:49:42.516979+0000","last_active":"2024-02-13T00:49:43.506514+0000","last_peered":"2024-02-13T00:49:43.506514+0000","last_clean":"2024-02-13T00:49:43.506514+0000","last_became_active":"2024-02-13T00:49:42.516207+0000","last_became_peered":"2024-02-13T00:49:42.516207+0000","last_unstale":"2024-02-13T00:49:43.506514+0000","last_undegraded":"2024-02-13T00:49:43.506514+0000","last_fullsized":"2024-02-13T00:49:43.506514+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"230.19","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.515623+0000","last_change":"2024-02-13T00:49:40.684776+0000","last_active":"2024-02-13T00:49:42.515623+0000","last_peered":"2024-02-13T00:49:42.515623+0000","last_clean":"2024-02-13T00:49:42.515623+0000","last_became_active":"2024-02-13T00:49:40.684485+0000","last_became_peered":"2024-02-13T00:49:40.684485+0000","last_unstale":"2024-02-13T00:49:42.515623+0000","last_undegraded":"2024-02-13T00:49:42.515623+0000","last_fullsized":"2024-02-13T00:49:42.515623+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.19","version":"0'0","reported_seq":11,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506007+0000","last_change":"2024-02-13T00:49:42.526779+0000","last_active":"2024-02-13T00:49:43.506007+0000","last_peered":"2024-02-13T00:49:43.506007+0000","last_clean":"2024-02-13T00:49:43.506007+0000","last_became_active":"2024-02-13T00:49:42.526476+0000","last_became_peered":"2024-02-13T00:49:42.526476+0000","last_unstale":"2024-02-13T00:49:43.506007+0000","last_undegraded":"2024-02-13T00:49:43.506007+0000","last_fullsized":"2024-02-13T00:49:43.506007+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"230.18","version":"0'0","reported_seq":13,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.505851+0000","last_change":"2024-02-13T00:49:40.513810+0000","last_active":"2024-02-13T00:49:43.505851+0000","last_peered":"2024-02-13T00:49:43.505851+0000","last_clean":"2024-02-13T00:49:43.505851+0000","last_became_active":"2024-02-13T00:49:40.513148+0000","last_became_peered":"2024-02-13T00:49:40.513148+0000","last_unstale":"2024-02-13T00:49:43.505851+0000","last_undegraded":"2024-02-13T00:49:43.505851+0000","last_fullsized":"2024-02-13T00:49:43.505851+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"231.16","version":"0'0","reported_seq":11,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508831+0000","last_change":"2024-02-13T00:49:42.519134+0000","last_active":"2024-02-13T00:49:43.508831+0000","last_peered":"2024-02-13T00:49:43.508831+0000","last_clean":"2024-02-13T00:49:43.508831+0000","last_became_active":"2024-02-13T00:49:42.517961+0000","last_became_peered":"2024-02-13T00:49:42.517961+0000","last_unstale":"2024-02-13T00:49:43.508831+0000","last_undegraded":"2024-02-13T00:49:43.508831+0000","last_fullsized":"2024-02-13T00:49:43.508831+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"230.17","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.519216+0000","last_change":"2024-02-13T00:49:40.683878+0000","last_active":"2024-02-13T00:49:42.519216+0000","last_peered":"2024-02-13T00:49:42.519216+0000","last_clean":"2024-02-13T00:49:42.519216+0000","last_became_active":"2024-02-13T00:49:40.683552+0000","last_became_peered":"2024-02-13T00:49:40.683552+0000","last_unstale":"2024-02-13T00:49:42.519216+0000","last_undegraded":"2024-02-13T00:49:42.519216+0000","last_fullsized":"2024-02-13T00:49:42.519216+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.17","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.524553+0000","last_change":"2024-02-13T00:49:42.523659+0000","last_active":"2024-02-13T00:49:42.524553+0000","last_peered":"2024-02-13T00:49:42.524553+0000","last_clean":"2024-02-13T00:49:42.524553+0000","last_became_active":"2024-02-13T00:49:42.523380+0000","last_became_peered":"2024-02-13T00:49:42.523380+0000","last_unstale":"2024-02-13T00:49:42.524553+0000","last_undegraded":"2024-02-13T00:49:42.524553+0000","last_fullsized":"2024-02-13T00:49:42.524553+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.16","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.513589+0000","last_change":"2024-02-13T00:49:40.683970+0000","last_active":"2024-02-13T00:49:42.513589+0000","last_peered":"2024-02-13T00:49:42.513589+0000","last_clean":"2024-02-13T00:49:42.513589+0000","last_became_active":"2024-02-13T00:49:40.683379+0000","last_became_peered":"2024-02-13T00:49:40.683379+0000","last_unstale":"2024-02-13T00:49:42.513589+0000","last_undegraded":"2024-02-13T00:49:42.513589+0000","last_fullsized":"2024-02-13T00:49:42.513589+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.14","version":"0'0","reported_seq":11,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508580+0000","last_change":"2024-02-13T00:49:42.514605+0000","last_active":"2024-02-13T00:49:43.508580+0000","last_peered":"2024-02-13T00:49:43.508580+0000","last_clean":"2024-02-13T00:49:43.508580+0000","last_became_active":"2024-02-13T00:49:42.513956+0000","last_became_peered":"2024-02-13T00:49:42.513956+0000","last_unstale":"2024-02-13T00:49:43.508580+0000","last_undegraded":"2024-02-13T00:49:43.508580+0000","last_fullsized":"2024-02-13T00:49:43.508580+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"230.15","version":"0'0","reported_seq":13,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507522+0000","last_change":"2024-02-13T00:49:40.515522+0000","last_active":"2024-02-13T00:49:43.507522+0000","last_peered":"2024-02-13T00:49:43.507522+0000","last_clean":"2024-02-13T00:49:43.507522+0000","last_became_active":"2024-02-13T00:49:40.515161+0000","last_became_peered":"2024-02-13T00:49:40.515161+0000","last_unstale":"2024-02-13T00:49:43.507522+0000","last_undegraded":"2024-02-13T00:49:43.507522+0000","last_fullsized":"2024-02-13T00:49:43.507522+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"231.15","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.523592+0000","last_change":"2024-02-13T00:49:42.523210+0000","last_active":"2024-02-13T00:49:42.523592+0000","last_peered":"2024-02-13T00:49:42.523592+0000","last_clean":"2024-02-13T00:49:42.523592+0000","last_became_active":"2024-02-13T00:49:42.522880+0000","last_became_peered":"2024-02-13T00:49:42.522880+0000","last_unstale":"2024-02-13T00:49:42.523592+0000","last_undegraded":"2024-02-13T00:49:42.523592+0000","last_fullsized":"2024-02-13T00:49:42.523592+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"230.14","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.513461+0000","last_change":"2024-02-13T00:49:40.516826+0000","last_active":"2024-02-13T00:49:42.513461+0000","last_peered":"2024-02-13T00:49:42.513461+0000","last_clean":"2024-02-13T00:49:42.513461+0000","last_became_active":"2024-02-13T00:49:40.516546+0000","last_became_peered":"2024-02-13T00:49:40.516546+0000","last_unstale":"2024-02-13T00:49:42.513461+0000","last_undegraded":"2024-02-13T00:49:42.513461+0000","last_fullsized":"2024-02-13T00:49:42.513461+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.12","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.523921+0000","last_change":"2024-02-13T00:49:42.520401+0000","last_active":"2024-02-13T00:49:42.523921+0000","last_peered":"2024-02-13T00:49:42.523921+0000","last_clean":"2024-02-13T00:49:42.523921+0000","last_became_active":"2024-02-13T00:49:42.519697+0000","last_became_peered":"2024-02-13T00:49:42.519697+0000","last_unstale":"2024-02-13T00:49:42.523921+0000","last_undegraded":"2024-02-13T00:49:42.523921+0000","last_fullsized":"2024-02-13T00:49:42.523921+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.13","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512394+0000","last_change":"2024-02-13T00:49:40.680913+0000","last_active":"2024-02-13T00:49:42.512394+0000","last_peered":"2024-02-13T00:49:42.512394+0000","last_clean":"2024-02-13T00:49:42.512394+0000","last_became_active":"2024-02-13T00:49:40.680630+0000","last_became_peered":"2024-02-13T00:49:40.680630+0000","last_unstale":"2024-02-13T00:49:42.512394+0000","last_undegraded":"2024-02-13T00:49:42.512394+0000","last_fullsized":"2024-02-13T00:49:42.512394+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.13","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.523933+0000","last_change":"2024-02-13T00:49:42.519912+0000","last_active":"2024-02-13T00:49:42.523933+0000","last_peered":"2024-02-13T00:49:42.523933+0000","last_clean":"2024-02-13T00:49:42.523933+0000","last_became_active":"2024-02-13T00:49:42.519515+0000","last_became_peered":"2024-02-13T00:49:42.519515+0000","last_unstale":"2024-02-13T00:49:42.523933+0000","last_undegraded":"2024-02-13T00:49:42.523933+0000","last_fullsized":"2024-02-13T00:49:42.523933+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.12","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.511718+0000","last_change":"2024-02-13T00:49:40.684238+0000","last_active":"2024-02-13T00:49:42.511718+0000","last_peered":"2024-02-13T00:49:42.511718+0000","last_clean":"2024-02-13T00:49:42.511718+0000","last_became_active":"2024-02-13T00:49:40.683361+0000","last_became_peered":"2024-02-13T00:49:40.683361+0000","last_unstale":"2024-02-13T00:49:42.511718+0000","last_undegraded":"2024-02-13T00:49:42.511718+0000","last_fullsized":"2024-02-13T00:49:42.511718+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.10","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.524881+0000","last_change":"2024-02-13T00:49:42.520672+0000","last_active":"2024-02-13T00:49:42.524881+0000","last_peered":"2024-02-13T00:49:42.524881+0000","last_clean":"2024-02-13T00:49:42.524881+0000","last_became_active":"2024-02-13T00:49:42.520252+0000","last_became_peered":"2024-02-13T00:49:42.520252+0000","last_unstale":"2024-02-13T00:49:42.524881+0000","last_undegraded":"2024-02-13T00:49:42.524881+0000","last_fullsized":"2024-02-13T00:49:42.524881+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.11","version":"0'0","reported_seq":13,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506117+0000","last_change":"2024-02-13T00:49:40.678969+0000","last_active":"2024-02-13T00:49:43.506117+0000","last_peered":"2024-02-13T00:49:43.506117+0000","last_clean":"2024-02-13T00:49:43.506117+0000","last_became_active":"2024-02-13T00:49:40.678648+0000","last_became_peered":"2024-02-13T00:49:40.678648+0000","last_unstale":"2024-02-13T00:49:43.506117+0000","last_undegraded":"2024-02-13T00:49:43.506117+0000","last_fullsized":"2024-02-13T00:49:43.506117+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"231.11","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.515998+0000","last_change":"2024-02-13T00:49:42.514793+0000","last_active":"2024-02-13T00:49:42.515998+0000","last_peered":"2024-02-13T00:49:42.515998+0000","last_clean":"2024-02-13T00:49:42.515998+0000","last_became_active":"2024-02-13T00:49:42.514085+0000","last_became_peered":"2024-02-13T00:49:42.514085+0000","last_unstale":"2024-02-13T00:49:42.515998+0000","last_undegraded":"2024-02-13T00:49:42.515998+0000","last_fullsized":"2024-02-13T00:49:42.515998+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"230.10","version":"0'0","reported_seq":13,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.505927+0000","last_change":"2024-02-13T00:49:40.677156+0000","last_active":"2024-02-13T00:49:43.505927+0000","last_peered":"2024-02-13T00:49:43.505927+0000","last_clean":"2024-02-13T00:49:43.505927+0000","last_became_active":"2024-02-13T00:49:40.676844+0000","last_became_peered":"2024-02-13T00:49:40.676844+0000","last_unstale":"2024-02-13T00:49:43.505927+0000","last_undegraded":"2024-02-13T00:49:43.505927+0000","last_fullsized":"2024-02-13T00:49:43.505927+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"231.e","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.524002+0000","last_change":"2024-02-13T00:49:42.519439+0000","last_active":"2024-02-13T00:49:42.524002+0000","last_peered":"2024-02-13T00:49:42.524002+0000","last_clean":"2024-02-13T00:49:42.524002+0000","last_became_active":"2024-02-13T00:49:42.519178+0000","last_became_peered":"2024-02-13T00:49:42.519178+0000","last_unstale":"2024-02-13T00:49:42.524002+0000","last_undegraded":"2024-02-13T00:49:42.524002+0000","last_fullsized":"2024-02-13T00:49:42.524002+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.f","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.519308+0000","last_change":"2024-02-13T00:49:40.683877+0000","last_active":"2024-02-13T00:49:42.519308+0000","last_peered":"2024-02-13T00:49:42.519308+0000","last_clean":"2024-02-13T00:49:42.519308+0000","last_became_active":"2024-02-13T00:49:40.683525+0000","last_became_peered":"2024-02-13T00:49:40.683525+0000","last_unstale":"2024-02-13T00:49:42.519308+0000","last_undegraded":"2024-02-13T00:49:42.519308+0000","last_fullsized":"2024-02-13T00:49:42.519308+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.f","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.526148+0000","last_change":"2024-02-13T00:49:42.524775+0000","last_active":"2024-02-13T00:49:42.526148+0000","last_peered":"2024-02-13T00:49:42.526148+0000","last_clean":"2024-02-13T00:49:42.526148+0000","last_became_active":"2024-02-13T00:49:42.524384+0000","last_became_peered":"2024-02-13T00:49:42.524384+0000","last_unstale":"2024-02-13T00:49:42.526148+0000","last_undegraded":"2024-02-13T00:49:42.526148+0000","last_fullsized":"2024-02-13T00:49:42.526148+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.e","version":"0'0","reported_seq":13,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.509087+0000","last_change":"2024-02-13T00:49:40.678931+0000","last_active":"2024-02-13T00:49:43.509087+0000","last_peered":"2024-02-13T00:49:43.509087+0000","last_clean":"2024-02-13T00:49:43.509087+0000","last_became_active":"2024-02-13T00:49:40.678553+0000","last_became_peered":"2024-02-13T00:49:40.678553+0000","last_unstale":"2024-02-13T00:49:43.509087+0000","last_undegraded":"2024-02-13T00:49:43.509087+0000","last_fullsized":"2024-02-13T00:49:43.509087+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"231.c","version":"0'0","reported_seq":11,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506459+0000","last_change":"2024-02-13T00:49:42.514769+0000","last_active":"2024-02-13T00:49:43.506459+0000","last_peered":"2024-02-13T00:49:43.506459+0000","last_clean":"2024-02-13T00:49:43.506459+0000","last_became_active":"2024-02-13T00:49:42.514334+0000","last_became_peered":"2024-02-13T00:49:42.514334+0000","last_unstale":"2024-02-13T00:49:43.506459+0000","last_undegraded":"2024-02-13T00:49:43.506459+0000","last_fullsized":"2024-02-13T00:49:43.506459+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"230.d","version":"0'0","reported_seq":13,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507563+0000","last_change":"2024-02-13T00:49:40.679120+0000","last_active":"2024-02-13T00:49:43.507563+0000","last_peered":"2024-02-13T00:49:43.507563+0000","last_clean":"2024-02-13T00:49:43.507563+0000","last_became_active":"2024-02-13T00:49:40.678545+0000","last_became_peered":"2024-02-13T00:49:40.678545+0000","last_unstale":"2024-02-13T00:49:43.507563+0000","last_undegraded":"2024-02-13T00:49:43.507563+0000","last_fullsized":"2024-02-13T00:49:43.507563+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"231.d","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.524930+0000","last_change":"2024-02-13T00:49:42.524594+0000","last_active":"2024-02-13T00:49:42.524930+0000","last_peered":"2024-02-13T00:49:42.524930+0000","last_clean":"2024-02-13T00:49:42.524930+0000","last_became_active":"2024-02-13T00:49:42.524312+0000","last_became_peered":"2024-02-13T00:49:42.524312+0000","last_unstale":"2024-02-13T00:49:42.524930+0000","last_undegraded":"2024-02-13T00:49:42.524930+0000","last_fullsized":"2024-02-13T00:49:42.524930+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"230.c","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.513520+0000","last_change":"2024-02-13T00:49:40.679341+0000","last_active":"2024-02-13T00:49:42.513520+0000","last_peered":"2024-02-13T00:49:42.513520+0000","last_clean":"2024-02-13T00:49:42.513520+0000","last_became_active":"2024-02-13T00:49:40.678619+0000","last_became_peered":"2024-02-13T00:49:40.678619+0000","last_unstale":"2024-02-13T00:49:42.513520+0000","last_undegraded":"2024-02-13T00:49:42.513520+0000","last_fullsized":"2024-02-13T00:49:42.513520+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.a","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.523957+0000","last_change":"2024-02-13T00:49:42.520567+0000","last_active":"2024-02-13T00:49:42.523957+0000","last_peered":"2024-02-13T00:49:42.523957+0000","last_clean":"2024-02-13T00:49:42.523957+0000","last_became_active":"2024-02-13T00:49:42.520197+0000","last_became_peered":"2024-02-13T00:49:42.520197+0000","last_unstale":"2024-02-13T00:49:42.523957+0000","last_undegraded":"2024-02-13T00:49:42.523957+0000","last_fullsized":"2024-02-13T00:49:42.523957+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.b","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512530+0000","last_change":"2024-02-13T00:49:40.681020+0000","last_active":"2024-02-13T00:49:42.512530+0000","last_peered":"2024-02-13T00:49:42.512530+0000","last_clean":"2024-02-13T00:49:42.512530+0000","last_became_active":"2024-02-13T00:49:40.680770+0000","last_became_peered":"2024-02-13T00:49:40.680770+0000","last_unstale":"2024-02-13T00:49:42.512530+0000","last_undegraded":"2024-02-13T00:49:42.512530+0000","last_fullsized":"2024-02-13T00:49:42.512530+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.b","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.523951+0000","last_change":"2024-02-13T00:49:42.519915+0000","last_active":"2024-02-13T00:49:42.523951+0000","last_peered":"2024-02-13T00:49:42.523951+0000","last_clean":"2024-02-13T00:49:42.523951+0000","last_became_active":"2024-02-13T00:49:42.519600+0000","last_became_peered":"2024-02-13T00:49:42.519600+0000","last_unstale":"2024-02-13T00:49:42.523951+0000","last_undegraded":"2024-02-13T00:49:42.523951+0000","last_fullsized":"2024-02-13T00:49:42.523951+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.a","version":"0'0","reported_seq":13,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506386+0000","last_change":"2024-02-13T00:49:40.516373+0000","last_active":"2024-02-13T00:49:43.506386+0000","last_peered":"2024-02-13T00:49:43.506386+0000","last_clean":"2024-02-13T00:49:43.506386+0000","last_became_active":"2024-02-13T00:49:40.516023+0000","last_became_peered":"2024-02-13T00:49:40.516023+0000","last_unstale":"2024-02-13T00:49:43.506386+0000","last_undegraded":"2024-02-13T00:49:43.506386+0000","last_fullsized":"2024-02-13T00:49:43.506386+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"231.8","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.524863+0000","last_change":"2024-02-13T00:49:42.520507+0000","last_active":"2024-02-13T00:49:42.524863+0000","last_peered":"2024-02-13T00:49:42.524863+0000","last_clean":"2024-02-13T00:49:42.524863+0000","last_became_active":"2024-02-13T00:49:42.519873+0000","last_became_peered":"2024-02-13T00:49:42.519873+0000","last_unstale":"2024-02-13T00:49:42.524863+0000","last_undegraded":"2024-02-13T00:49:42.524863+0000","last_fullsized":"2024-02-13T00:49:42.524863+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.9","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.511620+0000","last_change":"2024-02-13T00:49:40.678987+0000","last_active":"2024-02-13T00:49:42.511620+0000","last_peered":"2024-02-13T00:49:42.511620+0000","last_clean":"2024-02-13T00:49:42.511620+0000","last_became_active":"2024-02-13T00:49:40.678593+0000","last_became_peered":"2024-02-13T00:49:40.678593+0000","last_unstale":"2024-02-13T00:49:42.511620+0000","last_undegraded":"2024-02-13T00:49:42.511620+0000","last_fullsized":"2024-02-13T00:49:42.511620+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.9","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.528063+0000","last_change":"2024-02-13T00:49:42.526666+0000","last_active":"2024-02-13T00:49:42.528063+0000","last_peered":"2024-02-13T00:49:42.528063+0000","last_clean":"2024-02-13T00:49:42.528063+0000","last_became_active":"2024-02-13T00:49:42.526334+0000","last_became_peered":"2024-02-13T00:49:42.526334+0000","last_unstale":"2024-02-13T00:49:42.528063+0000","last_undegraded":"2024-02-13T00:49:42.528063+0000","last_fullsized":"2024-02-13T00:49:42.528063+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"230.8","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.511735+0000","last_change":"2024-02-13T00:49:40.677221+0000","last_active":"2024-02-13T00:49:42.511735+0000","last_peered":"2024-02-13T00:49:42.511735+0000","last_clean":"2024-02-13T00:49:42.511735+0000","last_became_active":"2024-02-13T00:49:40.676826+0000","last_became_peered":"2024-02-13T00:49:40.676826+0000","last_unstale":"2024-02-13T00:49:42.511735+0000","last_undegraded":"2024-02-13T00:49:42.511735+0000","last_fullsized":"2024-02-13T00:49:42.511735+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.5","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.524098+0000","last_change":"2024-02-13T00:49:42.523705+0000","last_active":"2024-02-13T00:49:42.524098+0000","last_peered":"2024-02-13T00:49:42.524098+0000","last_clean":"2024-02-13T00:49:42.524098+0000","last_became_active":"2024-02-13T00:49:42.523434+0000","last_became_peered":"2024-02-13T00:49:42.523434+0000","last_unstale":"2024-02-13T00:49:42.524098+0000","last_undegraded":"2024-02-13T00:49:42.524098+0000","last_fullsized":"2024-02-13T00:49:42.524098+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.4","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.513553+0000","last_change":"2024-02-13T00:49:40.679539+0000","last_active":"2024-02-13T00:49:42.513553+0000","last_peered":"2024-02-13T00:49:42.513553+0000","last_clean":"2024-02-13T00:49:42.513553+0000","last_became_active":"2024-02-13T00:49:40.678939+0000","last_became_peered":"2024-02-13T00:49:40.678939+0000","last_unstale":"2024-02-13T00:49:42.513553+0000","last_undegraded":"2024-02-13T00:49:42.513553+0000","last_fullsized":"2024-02-13T00:49:42.513553+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"229.7","version":"0'0","reported_seq":14,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516660+0000","last_change":"2024-02-13T00:49:38.508094+0000","last_active":"2024-02-13T00:49:42.516660+0000","last_peered":"2024-02-13T00:49:42.516660+0000","last_clean":"2024-02-13T00:49:42.516660+0000","last_became_active":"2024-02-13T00:49:38.507750+0000","last_became_peered":"2024-02-13T00:49:38.507750+0000","last_unstale":"2024-02-13T00:49:42.516660+0000","last_undegraded":"2024-02-13T00:49:42.516660+0000","last_fullsized":"2024-02-13T00:49:42.516660+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.4","version":"0'0","reported_seq":11,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506672+0000","last_change":"2024-02-13T00:49:42.525732+0000","last_active":"2024-02-13T00:49:43.506672+0000","last_peered":"2024-02-13T00:49:43.506672+0000","last_clean":"2024-02-13T00:49:43.506672+0000","last_became_active":"2024-02-13T00:49:42.525268+0000","last_became_peered":"2024-02-13T00:49:42.525268+0000","last_unstale":"2024-02-13T00:49:43.506672+0000","last_undegraded":"2024-02-13T00:49:43.506672+0000","last_fullsized":"2024-02-13T00:49:43.506672+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"230.5","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.515492+0000","last_change":"2024-02-13T00:49:40.683660+0000","last_active":"2024-02-13T00:49:42.515492+0000","last_peered":"2024-02-13T00:49:42.515492+0000","last_clean":"2024-02-13T00:49:42.515492+0000","last_became_active":"2024-02-13T00:49:40.683343+0000","last_became_peered":"2024-02-13T00:49:40.683343+0000","last_unstale":"2024-02-13T00:49:42.515492+0000","last_undegraded":"2024-02-13T00:49:42.515492+0000","last_fullsized":"2024-02-13T00:49:42.515492+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"229.6","version":"0'0","reported_seq":15,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.509169+0000","last_change":"2024-02-13T00:49:38.569413+0000","last_active":"2024-02-13T00:49:43.509169+0000","last_peered":"2024-02-13T00:49:43.509169+0000","last_clean":"2024-02-13T00:49:43.509169+0000","last_became_active":"2024-02-13T00:49:38.569040+0000","last_became_peered":"2024-02-13T00:49:38.569040+0000","last_unstale":"2024-02-13T00:49:43.509169+0000","last_undegraded":"2024-02-13T00:49:43.509169+0000","last_fullsized":"2024-02-13T00:49:43.509169+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"231.7","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.524835+0000","last_change":"2024-02-13T00:49:42.524674+0000","last_active":"2024-02-13T00:49:42.524835+0000","last_peered":"2024-02-13T00:49:42.524835+0000","last_clean":"2024-02-13T00:49:42.524835+0000","last_became_active":"2024-02-13T00:49:42.524071+0000","last_became_peered":"2024-02-13T00:49:42.524071+0000","last_unstale":"2024-02-13T00:49:42.524835+0000","last_undegraded":"2024-02-13T00:49:42.524835+0000","last_fullsized":"2024-02-13T00:49:42.524835+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.6","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.513131+0000","last_change":"2024-02-13T00:49:40.684109+0000","last_active":"2024-02-13T00:49:42.513131+0000","last_peered":"2024-02-13T00:49:42.513131+0000","last_clean":"2024-02-13T00:49:42.513131+0000","last_became_active":"2024-02-13T00:49:40.683713+0000","last_became_peered":"2024-02-13T00:49:40.683713+0000","last_unstale":"2024-02-13T00:49:42.513131+0000","last_undegraded":"2024-02-13T00:49:42.513131+0000","last_fullsized":"2024-02-13T00:49:42.513131+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"229.5","version":"564'4","reported_seq":21,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507721+0000","last_change":"2024-02-13T00:49:38.570168+0000","last_active":"2024-02-13T00:49:43.507721+0000","last_peered":"2024-02-13T00:49:43.507721+0000","last_clean":"2024-02-13T00:49:43.507721+0000","last_became_active":"2024-02-13T00:49:38.569352+0000","last_became_peered":"2024-02-13T00:49:38.569352+0000","last_unstale":"2024-02-13T00:49:43.507721+0000","last_undegraded":"2024-02-13T00:49:43.507721+0000","last_fullsized":"2024-02-13T00:49:43.507721+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"231.6","version":"0'0","reported_seq":11,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508929+0000","last_change":"2024-02-13T00:49:42.519416+0000","last_active":"2024-02-13T00:49:43.508929+0000","last_peered":"2024-02-13T00:49:43.508929+0000","last_clean":"2024-02-13T00:49:43.508929+0000","last_became_active":"2024-02-13T00:49:42.517977+0000","last_became_peered":"2024-02-13T00:49:42.517977+0000","last_unstale":"2024-02-13T00:49:43.508929+0000","last_undegraded":"2024-02-13T00:49:43.508929+0000","last_fullsized":"2024-02-13T00:49:43.508929+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"230.7","version":"0'0","reported_seq":13,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507667+0000","last_change":"2024-02-13T00:49:40.512625+0000","last_active":"2024-02-13T00:49:43.507667+0000","last_peered":"2024-02-13T00:49:43.507667+0000","last_clean":"2024-02-13T00:49:43.507667+0000","last_became_active":"2024-02-13T00:49:40.512339+0000","last_became_peered":"2024-02-13T00:49:40.512339+0000","last_unstale":"2024-02-13T00:49:43.507667+0000","last_undegraded":"2024-02-13T00:49:43.507667+0000","last_fullsized":"2024-02-13T00:49:43.507667+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"229.4","version":"0'0","reported_seq":14,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.515560+0000","last_change":"2024-02-13T00:49:38.507477+0000","last_active":"2024-02-13T00:49:42.515560+0000","last_peered":"2024-02-13T00:49:42.515560+0000","last_clean":"2024-02-13T00:49:42.515560+0000","last_became_active":"2024-02-13T00:49:38.507129+0000","last_became_peered":"2024-02-13T00:49:38.507129+0000","last_unstale":"2024-02-13T00:49:42.515560+0000","last_undegraded":"2024-02-13T00:49:42.515560+0000","last_fullsized":"2024-02-13T00:49:42.515560+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.1","version":"0'0","reported_seq":11,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506046+0000","last_change":"2024-02-13T00:49:42.512561+0000","last_active":"2024-02-13T00:49:43.506046+0000","last_peered":"2024-02-13T00:49:43.506046+0000","last_clean":"2024-02-13T00:49:43.506046+0000","last_became_active":"2024-02-13T00:49:42.511894+0000","last_became_peered":"2024-02-13T00:49:42.511894+0000","last_unstale":"2024-02-13T00:49:43.506046+0000","last_undegraded":"2024-02-13T00:49:43.506046+0000","last_fullsized":"2024-02-13T00:49:43.506046+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"230.0","version":"0'0","reported_seq":13,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.505950+0000","last_change":"2024-02-13T00:49:40.513996+0000","last_active":"2024-02-13T00:49:43.505950+0000","last_peered":"2024-02-13T00:49:43.505950+0000","last_clean":"2024-02-13T00:49:43.505950+0000","last_became_active":"2024-02-13T00:49:40.513519+0000","last_became_peered":"2024-02-13T00:49:40.513519+0000","last_unstale":"2024-02-13T00:49:43.505950+0000","last_undegraded":"2024-02-13T00:49:43.505950+0000","last_fullsized":"2024-02-13T00:49:43.505950+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"229.3","version":"0'0","reported_seq":14,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512487+0000","last_change":"2024-02-13T00:49:38.570001+0000","last_active":"2024-02-13T00:49:42.512487+0000","last_peered":"2024-02-13T00:49:42.512487+0000","last_clean":"2024-02-13T00:49:42.512487+0000","last_became_active":"2024-02-13T00:49:38.569665+0000","last_became_peered":"2024-02-13T00:49:38.569665+0000","last_unstale":"2024-02-13T00:49:42.512487+0000","last_undegraded":"2024-02-13T00:49:42.512487+0000","last_fullsized":"2024-02-13T00:49:42.512487+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.0","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.519566+0000","last_change":"2024-02-13T00:49:42.518879+0000","last_active":"2024-02-13T00:49:42.519566+0000","last_peered":"2024-02-13T00:49:42.519566+0000","last_clean":"2024-02-13T00:49:42.519566+0000","last_became_active":"2024-02-13T00:49:42.518565+0000","last_became_peered":"2024-02-13T00:49:42.518565+0000","last_unstale":"2024-02-13T00:49:42.519566+0000","last_undegraded":"2024-02-13T00:49:42.519566+0000","last_fullsized":"2024-02-13T00:49:42.519566+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"230.1","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.515696+0000","last_change":"2024-02-13T00:49:40.684264+0000","last_active":"2024-02-13T00:49:42.515696+0000","last_peered":"2024-02-13T00:49:42.515696+0000","last_clean":"2024-02-13T00:49:42.515696+0000","last_became_active":"2024-02-13T00:49:40.683566+0000","last_became_peered":"2024-02-13T00:49:40.683566+0000","last_unstale":"2024-02-13T00:49:42.515696+0000","last_undegraded":"2024-02-13T00:49:42.515696+0000","last_fullsized":"2024-02-13T00:49:42.515696+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"229.2","version":"0'0","reported_seq":14,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.511714+0000","last_change":"2024-02-13T00:49:38.569965+0000","last_active":"2024-02-13T00:49:42.511714+0000","last_peered":"2024-02-13T00:49:42.511714+0000","last_clean":"2024-02-13T00:49:42.511714+0000","last_became_active":"2024-02-13T00:49:38.569583+0000","last_became_peered":"2024-02-13T00:49:38.569583+0000","last_unstale":"2024-02-13T00:49:42.511714+0000","last_undegraded":"2024-02-13T00:49:42.511714+0000","last_fullsized":"2024-02-13T00:49:42.511714+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.3","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.523967+0000","last_change":"2024-02-13T00:49:42.521620+0000","last_active":"2024-02-13T00:49:42.523967+0000","last_peered":"2024-02-13T00:49:42.523967+0000","last_clean":"2024-02-13T00:49:42.523967+0000","last_became_active":"2024-02-13T00:49:42.521346+0000","last_became_peered":"2024-02-13T00:49:42.521346+0000","last_unstale":"2024-02-13T00:49:42.523967+0000","last_undegraded":"2024-02-13T00:49:42.523967+0000","last_fullsized":"2024-02-13T00:49:42.523967+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.2","version":"0'0","reported_seq":12,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.511803+0000","last_change":"2024-02-13T00:49:40.684305+0000","last_active":"2024-02-13T00:49:42.511803+0000","last_peered":"2024-02-13T00:49:42.511803+0000","last_clean":"2024-02-13T00:49:42.511803+0000","last_became_active":"2024-02-13T00:49:40.683963+0000","last_became_peered":"2024-02-13T00:49:40.683963+0000","last_unstale":"2024-02-13T00:49:42.511803+0000","last_undegraded":"2024-02-13T00:49:42.511803+0000","last_fullsized":"2024-02-13T00:49:42.511803+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"229.1","version":"0'0","reported_seq":15,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506135+0000","last_change":"2024-02-13T00:49:38.508927+0000","last_active":"2024-02-13T00:49:43.506135+0000","last_peered":"2024-02-13T00:49:43.506135+0000","last_clean":"2024-02-13T00:49:43.506135+0000","last_became_active":"2024-02-13T00:49:38.508528+0000","last_became_peered":"2024-02-13T00:49:38.508528+0000","last_unstale":"2024-02-13T00:49:43.506135+0000","last_undegraded":"2024-02-13T00:49:43.506135+0000","last_fullsized":"2024-02-13T00:49:43.506135+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"231.2","version":"0'0","reported_seq":10,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.523940+0000","last_change":"2024-02-13T00:49:42.520507+0000","last_active":"2024-02-13T00:49:42.523940+0000","last_peered":"2024-02-13T00:49:42.523940+0000","last_clean":"2024-02-13T00:49:42.523940+0000","last_became_active":"2024-02-13T00:49:42.520109+0000","last_became_peered":"2024-02-13T00:49:42.520109+0000","last_unstale":"2024-02-13T00:49:42.523940+0000","last_undegraded":"2024-02-13T00:49:42.523940+0000","last_fullsized":"2024-02-13T00:49:42.523940+0000","mapping_epoch":567,"log_start":"0'0","ondisk_log_start":"0'0","created":567,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:41.498606+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.498606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.3","version":"0'0","reported_seq":13,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507458+0000","last_change":"2024-02-13T00:49:40.680222+0000","last_active":"2024-02-13T00:49:43.507458+0000","last_peered":"2024-02-13T00:49:43.507458+0000","last_clean":"2024-02-13T00:49:43.507458+0000","last_became_active":"2024-02-13T00:49:40.679941+0000","last_became_peered":"2024-02-13T00:49:40.679941+0000","last_unstale":"2024-02-13T00:49:43.507458+0000","last_undegraded":"2024-02-13T00:49:43.507458+0000","last_fullsized":"2024-02-13T00:49:43.507458+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:39.494030+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.494030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"229.0","version":"0'0","reported_seq":15,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506029+0000","last_change":"2024-02-13T00:49:38.569473+0000","last_active":"2024-02-13T00:49:43.506029+0000","last_peered":"2024-02-13T00:49:43.506029+0000","last_clean":"2024-02-13T00:49:43.506029+0000","last_became_active":"2024-02-13T00:49:38.569185+0000","last_became_peered":"2024-02-13T00:49:38.569185+0000","last_unstale":"2024-02-13T00:49:43.506029+0000","last_undegraded":"2024-02-13T00:49:43.506029+0000","last_fullsized":"2024-02-13T00:49:43.506029+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":472,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506585+0000","last_change":"2024-02-13T00:46:01.448309+0000","last_active":"2024-02-13T00:49:43.506585+0000","last_peered":"2024-02-13T00:49:43.506585+0000","last_clean":"2024-02-13T00:49:43.506585+0000","last_became_active":"2024-02-13T00:46:01.445363+0000","last_became_peered":"2024-02-13T00:46:01.445363+0000","last_unstale":"2024-02-13T00:49:43.506585+0000","last_undegraded":"2024-02-13T00:49:43.506585+0000","last_fullsized":"2024-02-13T00:49:43.506585+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":397,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512303+0000","last_change":"2024-02-13T00:46:01.449894+0000","last_active":"2024-02-13T00:49:42.512303+0000","last_peered":"2024-02-13T00:49:42.512303+0000","last_clean":"2024-02-13T00:49:42.512303+0000","last_became_active":"2024-02-13T00:46:01.449570+0000","last_became_peered":"2024-02-13T00:46:01.449570+0000","last_unstale":"2024-02-13T00:49:42.512303+0000","last_undegraded":"2024-02-13T00:49:42.512303+0000","last_fullsized":"2024-02-13T00:49:42.512303+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":261,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516154+0000","last_change":"2024-02-13T00:46:01.445141+0000","last_active":"2024-02-13T00:49:42.516154+0000","last_peered":"2024-02-13T00:49:42.516154+0000","last_clean":"2024-02-13T00:49:42.516154+0000","last_became_active":"2024-02-13T00:46:01.444729+0000","last_became_peered":"2024-02-13T00:46:01.444729+0000","last_unstale":"2024-02-13T00:49:42.516154+0000","last_undegraded":"2024-02-13T00:49:42.516154+0000","last_fullsized":"2024-02-13T00:49:42.516154+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":346,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.511566+0000","last_change":"2024-02-13T00:45:15.426658+0000","last_active":"2024-02-13T00:49:42.511566+0000","last_peered":"2024-02-13T00:49:42.511566+0000","last_clean":"2024-02-13T00:49:42.511566+0000","last_became_active":"2024-02-13T00:45:15.426109+0000","last_became_peered":"2024-02-13T00:45:15.426109+0000","last_unstale":"2024-02-13T00:49:42.511566+0000","last_undegraded":"2024-02-13T00:49:42.511566+0000","last_fullsized":"2024-02-13T00:49:42.511566+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":253,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516987+0000","last_change":"2024-02-13T00:44:39.526065+0000","last_active":"2024-02-13T00:49:42.516987+0000","last_peered":"2024-02-13T00:49:42.516987+0000","last_clean":"2024-02-13T00:49:42.516987+0000","last_became_active":"2024-02-13T00:44:39.525161+0000","last_became_peered":"2024-02-13T00:44:39.525161+0000","last_unstale":"2024-02-13T00:49:42.516987+0000","last_undegraded":"2024-02-13T00:49:42.516987+0000","last_fullsized":"2024-02-13T00:49:42.516987+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":349,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512363+0000","last_change":"2024-02-13T00:48:06.705421+0000","last_active":"2024-02-13T00:49:42.512363+0000","last_peered":"2024-02-13T00:49:42.512363+0000","last_clean":"2024-02-13T00:49:42.512363+0000","last_became_active":"2024-02-13T00:48:06.705121+0000","last_became_peered":"2024-02-13T00:48:06.705121+0000","last_unstale":"2024-02-13T00:49:42.512363+0000","last_undegraded":"2024-02-13T00:49:42.512363+0000","last_fullsized":"2024-02-13T00:49:42.512363+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":271,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.518539+0000","last_change":"2024-02-13T00:46:01.459772+0000","last_active":"2024-02-13T00:49:42.518539+0000","last_peered":"2024-02-13T00:49:42.518539+0000","last_clean":"2024-02-13T00:49:42.518539+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:49:42.518539+0000","last_undegraded":"2024-02-13T00:49:42.518539+0000","last_fullsized":"2024-02-13T00:49:42.518539+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":530,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.515324+0000","last_change":"2024-02-13T00:46:01.463062+0000","last_active":"2024-02-13T00:49:42.515324+0000","last_peered":"2024-02-13T00:49:42.515324+0000","last_clean":"2024-02-13T00:49:42.515324+0000","last_became_active":"2024-02-13T00:46:01.462319+0000","last_became_peered":"2024-02-13T00:46:01.462319+0000","last_unstale":"2024-02-13T00:49:42.515324+0000","last_undegraded":"2024-02-13T00:49:42.515324+0000","last_fullsized":"2024-02-13T00:49:42.515324+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":253,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.520744+0000","last_change":"2024-02-13T00:44:39.525601+0000","last_active":"2024-02-13T00:49:42.520744+0000","last_peered":"2024-02-13T00:49:42.520744+0000","last_clean":"2024-02-13T00:49:42.520744+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:49:42.520744+0000","last_undegraded":"2024-02-13T00:49:42.520744+0000","last_fullsized":"2024-02-13T00:49:42.520744+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":337,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.514097+0000","last_change":"2024-02-13T00:46:01.455629+0000","last_active":"2024-02-13T00:49:42.514097+0000","last_peered":"2024-02-13T00:49:42.514097+0000","last_clean":"2024-02-13T00:49:42.514097+0000","last_became_active":"2024-02-13T00:46:01.452801+0000","last_became_peered":"2024-02-13T00:46:01.452801+0000","last_unstale":"2024-02-13T00:49:42.514097+0000","last_undegraded":"2024-02-13T00:49:42.514097+0000","last_fullsized":"2024-02-13T00:49:42.514097+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":493,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507298+0000","last_change":"2024-02-13T00:45:39.562095+0000","last_active":"2024-02-13T00:49:43.507298+0000","last_peered":"2024-02-13T00:49:43.507298+0000","last_clean":"2024-02-13T00:49:43.507298+0000","last_became_active":"2024-02-13T00:44:39.505121+0000","last_became_peered":"2024-02-13T00:44:39.505121+0000","last_unstale":"2024-02-13T00:49:43.507298+0000","last_undegraded":"2024-02-13T00:49:43.507298+0000","last_fullsized":"2024-02-13T00:49:43.507298+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:39.561994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:39.561994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"534'16","reported_seq":358,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.636306+0000","last_change":"2024-02-13T00:49:07.494629+0000","last_active":"2024-02-13T00:49:43.636306+0000","last_peered":"2024-02-13T00:49:43.636306+0000","last_clean":"2024-02-13T00:49:43.636306+0000","last_became_active":"2024-02-13T00:49:07.494388+0000","last_became_peered":"2024-02-13T00:49:07.494388+0000","last_unstale":"2024-02-13T00:49:43.636306+0000","last_undegraded":"2024-02-13T00:49:43.636306+0000","last_fullsized":"2024-02-13T00:49:43.636306+0000","mapping_epoch":359,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:45:53.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.115260+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":597,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.511985+0000","last_change":"2024-02-13T00:47:03.491720+0000","last_active":"2024-02-13T00:49:42.511985+0000","last_peered":"2024-02-13T00:49:42.511985+0000","last_clean":"2024-02-13T00:49:42.511985+0000","last_became_active":"2024-02-13T00:46:01.438056+0000","last_became_peered":"2024-02-13T00:46:01.438056+0000","last_unstale":"2024-02-13T00:49:42.511985+0000","last_undegraded":"2024-02-13T00:49:42.511985+0000","last_fullsized":"2024-02-13T00:49:42.511985+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:47:03.491647+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":567,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517843+0000","last_change":"2024-02-13T00:48:23.379872+0000","last_active":"2024-02-13T00:49:42.517843+0000","last_peered":"2024-02-13T00:49:42.517843+0000","last_clean":"2024-02-13T00:49:42.517843+0000","last_became_active":"2024-02-13T00:46:01.460811+0000","last_became_peered":"2024-02-13T00:46:01.460811+0000","last_unstale":"2024-02-13T00:49:42.517843+0000","last_undegraded":"2024-02-13T00:49:42.517843+0000","last_fullsized":"2024-02-13T00:49:42.517843+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:23.379773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:23.379773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":473,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512450+0000","last_change":"2024-02-13T00:47:34.332791+0000","last_active":"2024-02-13T00:49:42.512450+0000","last_peered":"2024-02-13T00:49:42.512450+0000","last_clean":"2024-02-13T00:49:42.512450+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:49:42.512450+0000","last_undegraded":"2024-02-13T00:49:42.512450+0000","last_fullsized":"2024-02-13T00:49:42.512450+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:34.332704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:34.332704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":511,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.514356+0000","last_change":"2024-02-13T00:46:01.448237+0000","last_active":"2024-02-13T00:49:42.514356+0000","last_peered":"2024-02-13T00:49:42.514356+0000","last_clean":"2024-02-13T00:49:42.514356+0000","last_became_active":"2024-02-13T00:46:01.446660+0000","last_became_peered":"2024-02-13T00:46:01.446660+0000","last_unstale":"2024-02-13T00:49:42.514356+0000","last_undegraded":"2024-02-13T00:49:42.514356+0000","last_fullsized":"2024-02-13T00:49:42.514356+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.260871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.260871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":572,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.509070+0000","last_change":"2024-02-13T00:47:41.808333+0000","last_active":"2024-02-13T00:49:43.509070+0000","last_peered":"2024-02-13T00:49:43.509070+0000","last_clean":"2024-02-13T00:49:43.509070+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:49:43.509070+0000","last_undegraded":"2024-02-13T00:49:43.509070+0000","last_fullsized":"2024-02-13T00:49:43.509070+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:47:41.808259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:41.808259+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":552,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516573+0000","last_change":"2024-02-13T00:48:24.823190+0000","last_active":"2024-02-13T00:49:42.516573+0000","last_peered":"2024-02-13T00:49:42.516573+0000","last_clean":"2024-02-13T00:49:42.516573+0000","last_became_active":"2024-02-13T00:44:50.429994+0000","last_became_peered":"2024-02-13T00:44:50.429994+0000","last_unstale":"2024-02-13T00:49:42.516573+0000","last_undegraded":"2024-02-13T00:49:42.516573+0000","last_fullsized":"2024-02-13T00:49:42.516573+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:24.823107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.823107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":456,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507201+0000","last_change":"2024-02-13T00:49:39.788846+0000","last_active":"2024-02-13T00:49:43.507201+0000","last_peered":"2024-02-13T00:49:43.507201+0000","last_clean":"2024-02-13T00:49:43.507201+0000","last_became_active":"2024-02-13T00:49:03.971407+0000","last_became_peered":"2024-02-13T00:49:03.971407+0000","last_unstale":"2024-02-13T00:49:43.507201+0000","last_undegraded":"2024-02-13T00:49:43.507201+0000","last_fullsized":"2024-02-13T00:49:43.507201+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.788783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.788783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":555,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506883+0000","last_change":"2024-02-13T00:49:41.837871+0000","last_active":"2024-02-13T00:49:43.506883+0000","last_peered":"2024-02-13T00:49:43.506883+0000","last_clean":"2024-02-13T00:49:43.506883+0000","last_became_active":"2024-02-13T00:45:15.434853+0000","last_became_peered":"2024-02-13T00:45:15.434853+0000","last_unstale":"2024-02-13T00:49:43.506883+0000","last_undegraded":"2024-02-13T00:49:43.506883+0000","last_fullsized":"2024-02-13T00:49:43.506883+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.837810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.837810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":176,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506780+0000","last_change":"2024-02-13T00:46:01.455201+0000","last_active":"2024-02-13T00:49:43.506780+0000","last_peered":"2024-02-13T00:49:43.506780+0000","last_clean":"2024-02-13T00:49:43.506780+0000","last_became_active":"2024-02-13T00:46:01.454882+0000","last_became_peered":"2024-02-13T00:46:01.454882+0000","last_unstale":"2024-02-13T00:49:43.506780+0000","last_undegraded":"2024-02-13T00:49:43.506780+0000","last_fullsized":"2024-02-13T00:49:43.506780+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:35.197045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:35.197045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":361,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512760+0000","last_change":"2024-02-13T00:46:09.256189+0000","last_active":"2024-02-13T00:49:42.512760+0000","last_peered":"2024-02-13T00:49:42.512760+0000","last_clean":"2024-02-13T00:49:42.512760+0000","last_became_active":"2024-02-13T00:46:01.459978+0000","last_became_peered":"2024-02-13T00:46:01.459978+0000","last_unstale":"2024-02-13T00:49:42.512760+0000","last_undegraded":"2024-02-13T00:49:42.512760+0000","last_fullsized":"2024-02-13T00:49:42.512760+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:44:24.196214+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:44:24.196214+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":186,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508504+0000","last_change":"2024-02-13T00:46:30.641539+0000","last_active":"2024-02-13T00:49:43.508504+0000","last_peered":"2024-02-13T00:49:43.508504+0000","last_clean":"2024-02-13T00:49:43.508504+0000","last_became_active":"2024-02-13T00:46:01.444548+0000","last_became_peered":"2024-02-13T00:46:01.444548+0000","last_unstale":"2024-02-13T00:49:43.508504+0000","last_undegraded":"2024-02-13T00:49:43.508504+0000","last_fullsized":"2024-02-13T00:49:43.508504+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.641460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.641460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":286,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.509284+0000","last_change":"2024-02-13T00:48:54.498301+0000","last_active":"2024-02-13T00:49:43.509284+0000","last_peered":"2024-02-13T00:49:43.509284+0000","last_clean":"2024-02-13T00:49:43.509284+0000","last_became_active":"2024-02-13T00:48:54.497977+0000","last_became_peered":"2024-02-13T00:48:54.497977+0000","last_unstale":"2024-02-13T00:49:43.509284+0000","last_undegraded":"2024-02-13T00:49:43.509284+0000","last_fullsized":"2024-02-13T00:49:43.509284+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.592672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":389,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512204+0000","last_change":"2024-02-13T00:45:15.429026+0000","last_active":"2024-02-13T00:49:42.512204+0000","last_peered":"2024-02-13T00:49:42.512204+0000","last_clean":"2024-02-13T00:49:42.512204+0000","last_became_active":"2024-02-13T00:45:15.427945+0000","last_became_peered":"2024-02-13T00:45:15.427945+0000","last_unstale":"2024-02-13T00:49:42.512204+0000","last_undegraded":"2024-02-13T00:49:42.512204+0000","last_fullsized":"2024-02-13T00:49:42.512204+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":492,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516297+0000","last_change":"2024-02-13T00:45:26.255907+0000","last_active":"2024-02-13T00:49:42.516297+0000","last_peered":"2024-02-13T00:49:42.516297+0000","last_clean":"2024-02-13T00:49:42.516297+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:49:42.516297+0000","last_undegraded":"2024-02-13T00:49:42.516297+0000","last_fullsized":"2024-02-13T00:49:42.516297+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.255842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.255842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"467'31","reported_seq":329,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516388+0000","last_change":"2024-02-13T00:48:31.782821+0000","last_active":"2024-02-13T00:49:42.516388+0000","last_peered":"2024-02-13T00:49:42.516388+0000","last_clean":"2024-02-13T00:49:42.516388+0000","last_became_active":"2024-02-13T00:44:50.432807+0000","last_became_peered":"2024-02-13T00:44:50.432807+0000","last_unstale":"2024-02-13T00:49:42.516388+0000","last_undegraded":"2024-02-13T00:49:42.516388+0000","last_fullsized":"2024-02-13T00:49:42.516388+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'31","last_scrub_stamp":"2024-02-13T00:48:31.782729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.782729+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":344,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.513367+0000","last_change":"2024-02-13T00:46:04.708510+0000","last_active":"2024-02-13T00:49:42.513367+0000","last_peered":"2024-02-13T00:49:42.513367+0000","last_clean":"2024-02-13T00:49:42.513367+0000","last_became_active":"2024-02-13T00:46:01.447801+0000","last_became_peered":"2024-02-13T00:46:01.447801+0000","last_unstale":"2024-02-13T00:49:42.513367+0000","last_undegraded":"2024-02-13T00:49:42.513367+0000","last_fullsized":"2024-02-13T00:49:42.513367+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:45:24.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:24.269595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":550,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517094+0000","last_change":"2024-02-13T00:48:49.612735+0000","last_active":"2024-02-13T00:49:42.517094+0000","last_peered":"2024-02-13T00:49:42.517094+0000","last_clean":"2024-02-13T00:49:42.517094+0000","last_became_active":"2024-02-13T00:48:49.612437+0000","last_became_peered":"2024-02-13T00:48:49.612437+0000","last_unstale":"2024-02-13T00:49:42.517094+0000","last_undegraded":"2024-02-13T00:49:42.517094+0000","last_fullsized":"2024-02-13T00:49:42.517094+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.793634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":538,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.520243+0000","last_change":"2024-02-13T00:44:50.441408+0000","last_active":"2024-02-13T00:49:42.520243+0000","last_peered":"2024-02-13T00:49:42.520243+0000","last_clean":"2024-02-13T00:49:42.520243+0000","last_became_active":"2024-02-13T00:44:50.440301+0000","last_became_peered":"2024-02-13T00:44:50.440301+0000","last_unstale":"2024-02-13T00:49:42.520243+0000","last_undegraded":"2024-02-13T00:49:42.520243+0000","last_fullsized":"2024-02-13T00:49:42.520243+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:25.502311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.502311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":193,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507470+0000","last_change":"2024-02-13T00:48:06.705079+0000","last_active":"2024-02-13T00:49:43.507470+0000","last_peered":"2024-02-13T00:49:43.507470+0000","last_clean":"2024-02-13T00:49:43.507470+0000","last_became_active":"2024-02-13T00:48:06.704718+0000","last_became_peered":"2024-02-13T00:48:06.704718+0000","last_unstale":"2024-02-13T00:49:43.507470+0000","last_undegraded":"2024-02-13T00:49:43.507470+0000","last_fullsized":"2024-02-13T00:49:43.507470+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.488353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.488353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":285,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.515446+0000","last_change":"2024-02-13T00:49:03.294585+0000","last_active":"2024-02-13T00:49:42.515446+0000","last_peered":"2024-02-13T00:49:42.515446+0000","last_clean":"2024-02-13T00:49:42.515446+0000","last_became_active":"2024-02-13T00:44:39.523138+0000","last_became_peered":"2024-02-13T00:44:39.523138+0000","last_unstale":"2024-02-13T00:49:42.515446+0000","last_undegraded":"2024-02-13T00:49:42.515446+0000","last_fullsized":"2024-02-13T00:49:42.515446+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:49:03.294499+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:49:03.294499+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":561,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512702+0000","last_change":"2024-02-13T00:48:59.111910+0000","last_active":"2024-02-13T00:49:42.512702+0000","last_peered":"2024-02-13T00:49:42.512702+0000","last_clean":"2024-02-13T00:49:42.512702+0000","last_became_active":"2024-02-13T00:46:01.452674+0000","last_became_peered":"2024-02-13T00:46:01.452674+0000","last_unstale":"2024-02-13T00:49:42.512702+0000","last_undegraded":"2024-02-13T00:49:42.512702+0000","last_fullsized":"2024-02-13T00:49:42.512702+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:59.111828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:59.111828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"511'13","reported_seq":273,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:44.351537+0000","last_change":"2024-02-13T00:48:45.592523+0000","last_active":"2024-02-13T00:49:44.351537+0000","last_peered":"2024-02-13T00:49:44.351537+0000","last_clean":"2024-02-13T00:49:44.351537+0000","last_became_active":"2024-02-13T00:48:45.591249+0000","last_became_peered":"2024-02-13T00:48:45.591249+0000","last_unstale":"2024-02-13T00:49:44.351537+0000","last_undegraded":"2024-02-13T00:49:44.351537+0000","last_fullsized":"2024-02-13T00:49:44.351537+0000","mapping_epoch":401,"log_start":"404'11","ondisk_log_start":"404'11","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'11","last_scrub_stamp":"2024-02-13T00:46:37.662413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.662413+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":528,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.514412+0000","last_change":"2024-02-13T00:46:05.839833+0000","last_active":"2024-02-13T00:49:42.514412+0000","last_peered":"2024-02-13T00:49:42.514412+0000","last_clean":"2024-02-13T00:49:42.514412+0000","last_became_active":"2024-02-13T00:46:01.455274+0000","last_became_peered":"2024-02-13T00:46:01.455274+0000","last_unstale":"2024-02-13T00:49:42.514412+0000","last_undegraded":"2024-02-13T00:49:42.514412+0000","last_fullsized":"2024-02-13T00:49:42.514412+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:44:25.230980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.230980+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":337,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517364+0000","last_change":"2024-02-13T00:46:01.457421+0000","last_active":"2024-02-13T00:49:42.517364+0000","last_peered":"2024-02-13T00:49:42.517364+0000","last_clean":"2024-02-13T00:49:42.517364+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:49:42.517364+0000","last_undegraded":"2024-02-13T00:49:42.517364+0000","last_fullsized":"2024-02-13T00:49:42.517364+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:18.089252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:18.089252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"467'27","reported_seq":650,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517285+0000","last_change":"2024-02-13T00:48:54.043163+0000","last_active":"2024-02-13T00:49:42.517285+0000","last_peered":"2024-02-13T00:49:42.517285+0000","last_clean":"2024-02-13T00:49:42.517285+0000","last_became_active":"2024-02-13T00:46:01.448705+0000","last_became_peered":"2024-02-13T00:46:01.448705+0000","last_unstale":"2024-02-13T00:49:42.517285+0000","last_undegraded":"2024-02-13T00:49:42.517285+0000","last_fullsized":"2024-02-13T00:49:42.517285+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:48:54.043088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.043088+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":593,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516418+0000","last_change":"2024-02-13T00:49:37.053462+0000","last_active":"2024-02-13T00:49:42.516418+0000","last_peered":"2024-02-13T00:49:42.516418+0000","last_clean":"2024-02-13T00:49:42.516418+0000","last_became_active":"2024-02-13T00:45:15.445228+0000","last_became_peered":"2024-02-13T00:45:15.445228+0000","last_unstale":"2024-02-13T00:49:42.516418+0000","last_undegraded":"2024-02-13T00:49:42.516418+0000","last_fullsized":"2024-02-13T00:49:42.516418+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:49:37.053398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.053398+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"507'13","reported_seq":271,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:44.351408+0000","last_change":"2024-02-13T00:48:41.526587+0000","last_active":"2024-02-13T00:49:44.351408+0000","last_peered":"2024-02-13T00:49:44.351408+0000","last_clean":"2024-02-13T00:49:44.351408+0000","last_became_active":"2024-02-13T00:48:41.525262+0000","last_became_peered":"2024-02-13T00:48:41.525262+0000","last_unstale":"2024-02-13T00:49:44.351408+0000","last_undegraded":"2024-02-13T00:49:44.351408+0000","last_fullsized":"2024-02-13T00:49:44.351408+0000","mapping_epoch":401,"log_start":"467'11","ondisk_log_start":"467'11","created":60,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:47:49.687727+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":178,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516817+0000","last_change":"2024-02-13T00:46:01.457110+0000","last_active":"2024-02-13T00:49:42.516817+0000","last_peered":"2024-02-13T00:49:42.516817+0000","last_clean":"2024-02-13T00:49:42.516817+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:49:42.516817+0000","last_undegraded":"2024-02-13T00:49:42.516817+0000","last_fullsized":"2024-02-13T00:49:42.516817+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.092250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.092250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":493,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.510189+0000","last_change":"2024-02-13T00:46:01.456531+0000","last_active":"2024-02-13T00:49:43.510189+0000","last_peered":"2024-02-13T00:49:43.510189+0000","last_clean":"2024-02-13T00:49:43.510189+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:49:43.510189+0000","last_undegraded":"2024-02-13T00:49:43.510189+0000","last_fullsized":"2024-02-13T00:49:43.510189+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:44:59.963616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":284,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.518116+0000","last_change":"2024-02-13T00:48:57.386335+0000","last_active":"2024-02-13T00:49:42.518116+0000","last_peered":"2024-02-13T00:49:42.518116+0000","last_clean":"2024-02-13T00:49:42.518116+0000","last_became_active":"2024-02-13T00:44:39.521984+0000","last_became_peered":"2024-02-13T00:44:39.521984+0000","last_unstale":"2024-02-13T00:49:42.518116+0000","last_undegraded":"2024-02-13T00:49:42.518116+0000","last_fullsized":"2024-02-13T00:49:42.518116+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:57.386254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:57.386254+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":376,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506915+0000","last_change":"2024-02-13T00:46:23.757417+0000","last_active":"2024-02-13T00:49:43.506915+0000","last_peered":"2024-02-13T00:49:43.506915+0000","last_clean":"2024-02-13T00:49:43.506915+0000","last_became_active":"2024-02-13T00:45:15.419239+0000","last_became_peered":"2024-02-13T00:45:15.419239+0000","last_unstale":"2024-02-13T00:49:43.506915+0000","last_undegraded":"2024-02-13T00:49:43.506915+0000","last_fullsized":"2024-02-13T00:49:43.506915+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:23.757351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:23.757351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"504'16","reported_seq":511,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:45.051624+0000","last_change":"2024-02-13T00:49:45.051624+0000","last_active":"2024-02-13T00:49:45.051624+0000","last_peered":"2024-02-13T00:49:45.051624+0000","last_clean":"2024-02-13T00:49:45.051624+0000","last_became_active":"2024-02-13T00:48:37.480010+0000","last_became_peered":"2024-02-13T00:48:37.480010+0000","last_unstale":"2024-02-13T00:49:45.051624+0000","last_undegraded":"2024-02-13T00:49:45.051624+0000","last_fullsized":"2024-02-13T00:49:45.051624+0000","mapping_epoch":401,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'16","last_scrub_stamp":"2024-02-13T00:49:45.051539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:49:45.051539+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":276,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.515347+0000","last_change":"2024-02-13T00:48:31.463534+0000","last_active":"2024-02-13T00:49:42.515347+0000","last_peered":"2024-02-13T00:49:42.515347+0000","last_clean":"2024-02-13T00:49:42.515347+0000","last_became_active":"2024-02-13T00:46:01.462717+0000","last_became_peered":"2024-02-13T00:46:01.462717+0000","last_unstale":"2024-02-13T00:49:42.515347+0000","last_undegraded":"2024-02-13T00:49:42.515347+0000","last_fullsized":"2024-02-13T00:49:42.515347+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:31.463442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.463442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":450,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506486+0000","last_change":"2024-02-13T00:49:00.881498+0000","last_active":"2024-02-13T00:49:43.506486+0000","last_peered":"2024-02-13T00:49:43.506486+0000","last_clean":"2024-02-13T00:49:43.506486+0000","last_became_active":"2024-02-13T00:46:01.456959+0000","last_became_peered":"2024-02-13T00:46:01.456959+0000","last_unstale":"2024-02-13T00:49:43.506486+0000","last_undegraded":"2024-02-13T00:49:43.506486+0000","last_fullsized":"2024-02-13T00:49:43.506486+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:00.881431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:00.881431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"263'10","reported_seq":564,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.618921+0000","last_change":"2024-02-13T00:47:02.949483+0000","last_active":"2024-02-13T00:49:43.618921+0000","last_peered":"2024-02-13T00:49:43.618921+0000","last_clean":"2024-02-13T00:49:43.618921+0000","last_became_active":"2024-02-13T00:43:35.454087+0000","last_became_peered":"2024-02-13T00:43:35.454087+0000","last_unstale":"2024-02-13T00:49:43.618921+0000","last_undegraded":"2024-02-13T00:49:43.618921+0000","last_fullsized":"2024-02-13T00:49:43.618921+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.949406+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":543,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512457+0000","last_change":"2024-02-13T00:46:08.351239+0000","last_active":"2024-02-13T00:49:42.512457+0000","last_peered":"2024-02-13T00:49:42.512457+0000","last_clean":"2024-02-13T00:49:42.512457+0000","last_became_active":"2024-02-13T00:46:01.449654+0000","last_became_peered":"2024-02-13T00:46:01.449654+0000","last_unstale":"2024-02-13T00:49:42.512457+0000","last_undegraded":"2024-02-13T00:49:42.512457+0000","last_fullsized":"2024-02-13T00:49:42.512457+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2024-02-13T00:43:00.636798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:43:00.636798+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":271,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.520622+0000","last_change":"2024-02-13T00:45:15.446786+0000","last_active":"2024-02-13T00:49:42.520622+0000","last_peered":"2024-02-13T00:49:42.520622+0000","last_clean":"2024-02-13T00:49:42.520622+0000","last_became_active":"2024-02-13T00:45:15.446443+0000","last_became_peered":"2024-02-13T00:45:15.446443+0000","last_unstale":"2024-02-13T00:49:42.520622+0000","last_undegraded":"2024-02-13T00:49:42.520622+0000","last_fullsized":"2024-02-13T00:49:42.520622+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.557475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.557475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":271,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516178+0000","last_change":"2024-02-13T00:45:15.436286+0000","last_active":"2024-02-13T00:49:42.516178+0000","last_peered":"2024-02-13T00:49:42.516178+0000","last_clean":"2024-02-13T00:49:42.516178+0000","last_became_active":"2024-02-13T00:45:15.434559+0000","last_became_peered":"2024-02-13T00:45:15.434559+0000","last_unstale":"2024-02-13T00:49:42.516178+0000","last_undegraded":"2024-02-13T00:49:42.516178+0000","last_fullsized":"2024-02-13T00:49:42.516178+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":300,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506660+0000","last_change":"2024-02-13T00:49:36.841674+0000","last_active":"2024-02-13T00:49:43.506660+0000","last_peered":"2024-02-13T00:49:43.506660+0000","last_clean":"2024-02-13T00:49:43.506660+0000","last_became_active":"2024-02-13T00:46:01.444661+0000","last_became_peered":"2024-02-13T00:46:01.444661+0000","last_unstale":"2024-02-13T00:49:43.506660+0000","last_undegraded":"2024-02-13T00:49:43.506660+0000","last_fullsized":"2024-02-13T00:49:43.506660+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:36.841508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:36.841508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":341,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506495+0000","last_change":"2024-02-13T00:48:18.751148+0000","last_active":"2024-02-13T00:49:43.506495+0000","last_peered":"2024-02-13T00:49:43.506495+0000","last_clean":"2024-02-13T00:49:43.506495+0000","last_became_active":"2024-02-13T00:46:01.453842+0000","last_became_peered":"2024-02-13T00:46:01.453842+0000","last_unstale":"2024-02-13T00:49:43.506495+0000","last_undegraded":"2024-02-13T00:49:43.506495+0000","last_fullsized":"2024-02-13T00:49:43.506495+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:18.751081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:18.751081+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"467'42","reported_seq":502,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506758+0000","last_change":"2024-02-13T00:48:24.675807+0000","last_active":"2024-02-13T00:49:43.506758+0000","last_peered":"2024-02-13T00:49:43.506758+0000","last_clean":"2024-02-13T00:49:43.506758+0000","last_became_active":"2024-02-13T00:46:01.445058+0000","last_became_peered":"2024-02-13T00:46:01.445058+0000","last_unstale":"2024-02-13T00:49:43.506758+0000","last_undegraded":"2024-02-13T00:49:43.506758+0000","last_fullsized":"2024-02-13T00:49:43.506758+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'42","last_scrub_stamp":"2024-02-13T00:48:24.675687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.675687+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":561,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.514171+0000","last_change":"2024-02-13T00:47:09.940673+0000","last_active":"2024-02-13T00:49:42.514171+0000","last_peered":"2024-02-13T00:49:42.514171+0000","last_clean":"2024-02-13T00:49:42.514171+0000","last_became_active":"2024-02-13T00:44:39.503590+0000","last_became_peered":"2024-02-13T00:44:39.503590+0000","last_unstale":"2024-02-13T00:49:42.514171+0000","last_undegraded":"2024-02-13T00:49:42.514171+0000","last_fullsized":"2024-02-13T00:49:42.514171+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:09.940605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:09.940605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":506,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512496+0000","last_change":"2024-02-13T00:46:01.448313+0000","last_active":"2024-02-13T00:49:42.512496+0000","last_peered":"2024-02-13T00:49:42.512496+0000","last_clean":"2024-02-13T00:49:42.512496+0000","last_became_active":"2024-02-13T00:46:01.447815+0000","last_became_peered":"2024-02-13T00:46:01.447815+0000","last_unstale":"2024-02-13T00:49:42.512496+0000","last_undegraded":"2024-02-13T00:49:42.512496+0000","last_fullsized":"2024-02-13T00:49:42.512496+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":498,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516087+0000","last_change":"2024-02-13T00:48:35.434713+0000","last_active":"2024-02-13T00:49:42.516087+0000","last_peered":"2024-02-13T00:49:42.516087+0000","last_clean":"2024-02-13T00:49:42.516087+0000","last_became_active":"2024-02-13T00:44:50.436058+0000","last_became_peered":"2024-02-13T00:44:50.436058+0000","last_unstale":"2024-02-13T00:49:42.516087+0000","last_undegraded":"2024-02-13T00:49:42.516087+0000","last_fullsized":"2024-02-13T00:49:42.516087+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_clean_scrub_stamp":"2024-02-13T00:48:35.434645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":267,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512954+0000","last_change":"2024-02-13T00:46:01.451625+0000","last_active":"2024-02-13T00:49:42.512954+0000","last_peered":"2024-02-13T00:49:42.512954+0000","last_clean":"2024-02-13T00:49:42.512954+0000","last_became_active":"2024-02-13T00:46:01.450587+0000","last_became_peered":"2024-02-13T00:46:01.450587+0000","last_unstale":"2024-02-13T00:49:42.512954+0000","last_undegraded":"2024-02-13T00:49:42.512954+0000","last_fullsized":"2024-02-13T00:49:42.512954+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:33.204969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:33.204969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":665,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508751+0000","last_change":"2024-02-13T00:47:05.718055+0000","last_active":"2024-02-13T00:49:43.508751+0000","last_peered":"2024-02-13T00:49:43.508751+0000","last_clean":"2024-02-13T00:49:43.508751+0000","last_became_active":"2024-02-13T00:46:01.443777+0000","last_became_peered":"2024-02-13T00:46:01.443777+0000","last_unstale":"2024-02-13T00:49:43.508751+0000","last_undegraded":"2024-02-13T00:49:43.508751+0000","last_fullsized":"2024-02-13T00:49:43.508751+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:47:05.717987+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:47:05.717987+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":376,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.509087+0000","last_change":"2024-02-13T00:45:15.423607+0000","last_active":"2024-02-13T00:49:43.509087+0000","last_peered":"2024-02-13T00:49:43.509087+0000","last_clean":"2024-02-13T00:49:43.509087+0000","last_became_active":"2024-02-13T00:45:15.423415+0000","last_became_peered":"2024-02-13T00:45:15.423415+0000","last_unstale":"2024-02-13T00:49:43.509087+0000","last_undegraded":"2024-02-13T00:49:43.509087+0000","last_fullsized":"2024-02-13T00:49:43.509087+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:17.295459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:17.295459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":568,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516464+0000","last_change":"2024-02-13T00:45:15.445862+0000","last_active":"2024-02-13T00:49:42.516464+0000","last_peered":"2024-02-13T00:49:42.516464+0000","last_clean":"2024-02-13T00:49:42.516464+0000","last_became_active":"2024-02-13T00:45:15.445557+0000","last_became_peered":"2024-02-13T00:45:15.445557+0000","last_unstale":"2024-02-13T00:49:42.516464+0000","last_undegraded":"2024-02-13T00:49:42.516464+0000","last_fullsized":"2024-02-13T00:49:42.516464+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":186,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.513894+0000","last_change":"2024-02-13T00:48:55.050483+0000","last_active":"2024-02-13T00:49:42.513894+0000","last_peered":"2024-02-13T00:49:42.513894+0000","last_clean":"2024-02-13T00:49:42.513894+0000","last_became_active":"2024-02-13T00:46:01.441421+0000","last_became_peered":"2024-02-13T00:46:01.441421+0000","last_unstale":"2024-02-13T00:49:42.513894+0000","last_undegraded":"2024-02-13T00:49:42.513894+0000","last_fullsized":"2024-02-13T00:49:42.513894+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:55.050411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.050411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":201,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517639+0000","last_change":"2024-02-13T00:49:16.505082+0000","last_active":"2024-02-13T00:49:42.517639+0000","last_peered":"2024-02-13T00:49:42.517639+0000","last_clean":"2024-02-13T00:49:42.517639+0000","last_became_active":"2024-02-13T00:49:16.504828+0000","last_became_peered":"2024-02-13T00:49:16.504828+0000","last_unstale":"2024-02-13T00:49:42.517639+0000","last_undegraded":"2024-02-13T00:49:42.517639+0000","last_fullsized":"2024-02-13T00:49:42.517639+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:43.903553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:43.903553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":660,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.509995+0000","last_change":"2024-02-13T00:46:34.691109+0000","last_active":"2024-02-13T00:49:43.509995+0000","last_peered":"2024-02-13T00:49:43.509995+0000","last_clean":"2024-02-13T00:49:43.509995+0000","last_became_active":"2024-02-13T00:46:01.448163+0000","last_became_peered":"2024-02-13T00:46:01.448163+0000","last_unstale":"2024-02-13T00:49:43.509995+0000","last_undegraded":"2024-02-13T00:49:43.509995+0000","last_fullsized":"2024-02-13T00:49:43.509995+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:46:34.690980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.690980+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":269,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512092+0000","last_change":"2024-02-13T00:46:37.056299+0000","last_active":"2024-02-13T00:49:42.512092+0000","last_peered":"2024-02-13T00:49:42.512092+0000","last_clean":"2024-02-13T00:49:42.512092+0000","last_became_active":"2024-02-13T00:46:01.446104+0000","last_became_peered":"2024-02-13T00:46:01.446104+0000","last_unstale":"2024-02-13T00:49:42.512092+0000","last_undegraded":"2024-02-13T00:49:42.512092+0000","last_fullsized":"2024-02-13T00:49:42.512092+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:37.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":552,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506679+0000","last_change":"2024-02-13T00:46:01.431899+0000","last_active":"2024-02-13T00:49:43.506679+0000","last_peered":"2024-02-13T00:49:43.506679+0000","last_clean":"2024-02-13T00:49:43.506679+0000","last_became_active":"2024-02-13T00:46:01.430623+0000","last_became_peered":"2024-02-13T00:46:01.430623+0000","last_unstale":"2024-02-13T00:49:43.506679+0000","last_undegraded":"2024-02-13T00:49:43.506679+0000","last_fullsized":"2024-02-13T00:49:43.506679+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:46.556229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.556229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":559,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517186+0000","last_change":"2024-02-13T00:46:01.450548+0000","last_active":"2024-02-13T00:49:42.517186+0000","last_peered":"2024-02-13T00:49:42.517186+0000","last_clean":"2024-02-13T00:49:42.517186+0000","last_became_active":"2024-02-13T00:46:01.450176+0000","last_became_peered":"2024-02-13T00:46:01.450176+0000","last_unstale":"2024-02-13T00:49:42.517186+0000","last_undegraded":"2024-02-13T00:49:42.517186+0000","last_fullsized":"2024-02-13T00:49:42.517186+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":413,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.518729+0000","last_change":"2024-02-13T00:46:01.460470+0000","last_active":"2024-02-13T00:49:42.518729+0000","last_peered":"2024-02-13T00:49:42.518729+0000","last_clean":"2024-02-13T00:49:42.518729+0000","last_became_active":"2024-02-13T00:46:01.459859+0000","last_became_peered":"2024-02-13T00:46:01.459859+0000","last_unstale":"2024-02-13T00:49:42.518729+0000","last_undegraded":"2024-02-13T00:49:42.518729+0000","last_fullsized":"2024-02-13T00:49:42.518729+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:01.179031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:01.179031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":217,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507258+0000","last_change":"2024-02-13T00:47:33.932830+0000","last_active":"2024-02-13T00:49:43.507258+0000","last_peered":"2024-02-13T00:49:43.507258+0000","last_clean":"2024-02-13T00:49:43.507258+0000","last_became_active":"2024-02-13T00:46:01.455170+0000","last_became_peered":"2024-02-13T00:46:01.455170+0000","last_unstale":"2024-02-13T00:49:43.507258+0000","last_undegraded":"2024-02-13T00:49:43.507258+0000","last_fullsized":"2024-02-13T00:49:43.507258+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:33.932766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:33.932766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":317,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.518744+0000","last_change":"2024-02-13T00:46:57.381602+0000","last_active":"2024-02-13T00:49:42.518744+0000","last_peered":"2024-02-13T00:49:42.518744+0000","last_clean":"2024-02-13T00:49:42.518744+0000","last_became_active":"2024-02-13T00:45:15.446969+0000","last_became_peered":"2024-02-13T00:45:15.446969+0000","last_unstale":"2024-02-13T00:49:42.518744+0000","last_undegraded":"2024-02-13T00:49:42.518744+0000","last_fullsized":"2024-02-13T00:49:42.518744+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:46:57.381540+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:46:57.381540+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":517,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506494+0000","last_change":"2024-02-13T00:47:39.822745+0000","last_active":"2024-02-13T00:49:43.506494+0000","last_peered":"2024-02-13T00:49:43.506494+0000","last_clean":"2024-02-13T00:49:43.506494+0000","last_became_active":"2024-02-13T00:45:15.434498+0000","last_became_peered":"2024-02-13T00:45:15.434498+0000","last_unstale":"2024-02-13T00:49:43.506494+0000","last_undegraded":"2024-02-13T00:49:43.506494+0000","last_fullsized":"2024-02-13T00:49:43.506494+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:39.822619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:47:39.822619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.513464+0000","last_change":"2024-02-13T00:47:04.493911+0000","last_active":"2024-02-13T00:49:42.513464+0000","last_peered":"2024-02-13T00:49:42.513464+0000","last_clean":"2024-02-13T00:49:42.513464+0000","last_became_active":"2024-02-13T00:46:01.447823+0000","last_became_peered":"2024-02-13T00:46:01.447823+0000","last_unstale":"2024-02-13T00:49:42.513464+0000","last_undegraded":"2024-02-13T00:49:42.513464+0000","last_fullsized":"2024-02-13T00:49:42.513464+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:04.493841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:04.493841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":388,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512277+0000","last_change":"2024-02-13T00:48:25.359625+0000","last_active":"2024-02-13T00:49:42.512277+0000","last_peered":"2024-02-13T00:49:42.512277+0000","last_clean":"2024-02-13T00:49:42.512277+0000","last_became_active":"2024-02-13T00:46:01.460001+0000","last_became_peered":"2024-02-13T00:46:01.460001+0000","last_unstale":"2024-02-13T00:49:42.512277+0000","last_undegraded":"2024-02-13T00:49:42.512277+0000","last_fullsized":"2024-02-13T00:49:42.512277+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:25.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:48:25.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":407,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517063+0000","last_change":"2024-02-13T00:46:01.450879+0000","last_active":"2024-02-13T00:49:42.517063+0000","last_peered":"2024-02-13T00:49:42.517063+0000","last_clean":"2024-02-13T00:49:42.517063+0000","last_became_active":"2024-02-13T00:46:01.450088+0000","last_became_peered":"2024-02-13T00:46:01.450088+0000","last_unstale":"2024-02-13T00:49:42.517063+0000","last_undegraded":"2024-02-13T00:49:42.517063+0000","last_fullsized":"2024-02-13T00:49:42.517063+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":498,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508959+0000","last_change":"2024-02-13T00:49:42.854702+0000","last_active":"2024-02-13T00:49:43.508959+0000","last_peered":"2024-02-13T00:49:43.508959+0000","last_clean":"2024-02-13T00:49:43.508959+0000","last_became_active":"2024-02-13T00:46:01.458408+0000","last_became_peered":"2024-02-13T00:46:01.458408+0000","last_unstale":"2024-02-13T00:49:43.508959+0000","last_undegraded":"2024-02-13T00:49:43.508959+0000","last_fullsized":"2024-02-13T00:49:43.508959+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:42.854585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:42.854585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":238,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.513672+0000","last_change":"2024-02-13T00:49:19.638381+0000","last_active":"2024-02-13T00:49:42.513672+0000","last_peered":"2024-02-13T00:49:42.513672+0000","last_clean":"2024-02-13T00:49:42.513672+0000","last_became_active":"2024-02-13T00:49:19.638083+0000","last_became_peered":"2024-02-13T00:49:19.638083+0000","last_unstale":"2024-02-13T00:49:42.513672+0000","last_undegraded":"2024-02-13T00:49:42.513672+0000","last_fullsized":"2024-02-13T00:49:42.513672+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:53.013103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.013103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"467'23","reported_seq":603,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517396+0000","last_change":"2024-02-13T00:49:07.192972+0000","last_active":"2024-02-13T00:49:42.517396+0000","last_peered":"2024-02-13T00:49:42.517396+0000","last_clean":"2024-02-13T00:49:42.517396+0000","last_became_active":"2024-02-13T00:46:01.448219+0000","last_became_peered":"2024-02-13T00:46:01.448219+0000","last_unstale":"2024-02-13T00:49:42.517396+0000","last_undegraded":"2024-02-13T00:49:42.517396+0000","last_fullsized":"2024-02-13T00:49:42.517396+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'23","last_scrub_stamp":"2024-02-13T00:49:07.192902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:07.192902+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":467,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506597+0000","last_change":"2024-02-13T00:44:50.426201+0000","last_active":"2024-02-13T00:49:43.506597+0000","last_peered":"2024-02-13T00:49:43.506597+0000","last_clean":"2024-02-13T00:49:43.506597+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:49:43.506597+0000","last_undegraded":"2024-02-13T00:49:43.506597+0000","last_fullsized":"2024-02-13T00:49:43.506597+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:20.481083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:44:20.481083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":1293,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.923692+0000","last_change":"2024-02-13T00:45:54.075270+0000","last_active":"2024-02-13T00:49:43.923692+0000","last_peered":"2024-02-13T00:49:43.923692+0000","last_clean":"2024-02-13T00:49:43.923692+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:49:43.923692+0000","last_undegraded":"2024-02-13T00:49:43.923692+0000","last_fullsized":"2024-02-13T00:49:43.923692+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:45:54.075194+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":711,"num_read_kb":706,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":280,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.518598+0000","last_change":"2024-02-13T00:46:01.460282+0000","last_active":"2024-02-13T00:49:42.518598+0000","last_peered":"2024-02-13T00:49:42.518598+0000","last_clean":"2024-02-13T00:49:42.518598+0000","last_became_active":"2024-02-13T00:46:01.459473+0000","last_became_peered":"2024-02-13T00:46:01.459473+0000","last_unstale":"2024-02-13T00:49:42.518598+0000","last_undegraded":"2024-02-13T00:49:42.518598+0000","last_fullsized":"2024-02-13T00:49:42.518598+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":332,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517141+0000","last_change":"2024-02-13T00:45:37.135398+0000","last_active":"2024-02-13T00:49:42.517141+0000","last_peered":"2024-02-13T00:49:42.517141+0000","last_clean":"2024-02-13T00:49:42.517141+0000","last_became_active":"2024-02-13T00:45:15.435588+0000","last_became_peered":"2024-02-13T00:45:15.435588+0000","last_unstale":"2024-02-13T00:49:42.517141+0000","last_undegraded":"2024-02-13T00:49:42.517141+0000","last_fullsized":"2024-02-13T00:49:42.517141+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.135332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.135332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":231,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.518000+0000","last_change":"2024-02-13T00:45:31.183763+0000","last_active":"2024-02-13T00:49:42.518000+0000","last_peered":"2024-02-13T00:49:42.518000+0000","last_clean":"2024-02-13T00:49:42.518000+0000","last_became_active":"2024-02-13T00:45:06.421025+0000","last_became_peered":"2024-02-13T00:45:06.421025+0000","last_unstale":"2024-02-13T00:49:42.518000+0000","last_undegraded":"2024-02-13T00:49:42.518000+0000","last_fullsized":"2024-02-13T00:49:42.518000+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.183682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.183682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"467'65","reported_seq":263,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.621397+0000","last_change":"2024-02-13T00:46:04.950162+0000","last_active":"2024-02-13T00:49:43.621397+0000","last_peered":"2024-02-13T00:49:43.621397+0000","last_clean":"2024-02-13T00:49:43.621397+0000","last_became_active":"2024-02-13T00:46:01.449059+0000","last_became_peered":"2024-02-13T00:46:01.449059+0000","last_unstale":"2024-02-13T00:49:43.621397+0000","last_undegraded":"2024-02-13T00:49:43.621397+0000","last_fullsized":"2024-02-13T00:49:43.621397+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'48","last_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:43:48.212428+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":2148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":477,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512537+0000","last_change":"2024-02-13T00:46:01.453042+0000","last_active":"2024-02-13T00:49:42.512537+0000","last_peered":"2024-02-13T00:49:42.512537+0000","last_clean":"2024-02-13T00:49:42.512537+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:49:42.512537+0000","last_undegraded":"2024-02-13T00:49:42.512537+0000","last_fullsized":"2024-02-13T00:49:42.512537+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.045912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.045912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":97,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.509193+0000","last_change":"2024-02-13T00:48:06.705030+0000","last_active":"2024-02-13T00:49:43.509193+0000","last_peered":"2024-02-13T00:49:43.509193+0000","last_clean":"2024-02-13T00:49:43.509193+0000","last_became_active":"2024-02-13T00:48:06.704651+0000","last_became_peered":"2024-02-13T00:48:06.704651+0000","last_unstale":"2024-02-13T00:49:43.509193+0000","last_undegraded":"2024-02-13T00:49:43.509193+0000","last_fullsized":"2024-02-13T00:49:43.509193+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":267,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512872+0000","last_change":"2024-02-13T00:48:17.340891+0000","last_active":"2024-02-13T00:49:42.512872+0000","last_peered":"2024-02-13T00:49:42.512872+0000","last_clean":"2024-02-13T00:49:42.512872+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:49:42.512872+0000","last_undegraded":"2024-02-13T00:49:42.512872+0000","last_fullsized":"2024-02-13T00:49:42.512872+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:17.340797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:17.340797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":737,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:44.351418+0000","last_change":"2024-02-13T00:48:32.669477+0000","last_active":"2024-02-13T00:49:44.351418+0000","last_peered":"2024-02-13T00:49:44.351418+0000","last_clean":"2024-02-13T00:49:44.351418+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:49:44.351418+0000","last_undegraded":"2024-02-13T00:49:44.351418+0000","last_fullsized":"2024-02-13T00:49:44.351418+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14","last_scrub_stamp":"2024-02-13T00:48:32.669406+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.669406+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":300,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.513001+0000","last_change":"2024-02-13T00:48:20.928236+0000","last_active":"2024-02-13T00:49:42.513001+0000","last_peered":"2024-02-13T00:49:42.513001+0000","last_clean":"2024-02-13T00:49:42.513001+0000","last_became_active":"2024-02-13T00:46:01.442003+0000","last_became_peered":"2024-02-13T00:46:01.442003+0000","last_unstale":"2024-02-13T00:49:42.513001+0000","last_undegraded":"2024-02-13T00:49:42.513001+0000","last_fullsized":"2024-02-13T00:49:42.513001+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:48:20.928158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:20.928158+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":548,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516770+0000","last_change":"2024-02-13T00:49:33.040619+0000","last_active":"2024-02-13T00:49:42.516770+0000","last_peered":"2024-02-13T00:49:42.516770+0000","last_clean":"2024-02-13T00:49:42.516770+0000","last_became_active":"2024-02-13T00:45:15.422546+0000","last_became_peered":"2024-02-13T00:45:15.422546+0000","last_unstale":"2024-02-13T00:49:42.516770+0000","last_undegraded":"2024-02-13T00:49:42.516770+0000","last_fullsized":"2024-02-13T00:49:42.516770+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:33.040516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:49:33.040516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":559,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506412+0000","last_change":"2024-02-13T00:46:01.436997+0000","last_active":"2024-02-13T00:49:43.506412+0000","last_peered":"2024-02-13T00:49:43.506412+0000","last_clean":"2024-02-13T00:49:43.506412+0000","last_became_active":"2024-02-13T00:46:01.435178+0000","last_became_peered":"2024-02-13T00:46:01.435178+0000","last_unstale":"2024-02-13T00:49:43.506412+0000","last_undegraded":"2024-02-13T00:49:43.506412+0000","last_fullsized":"2024-02-13T00:49:43.506412+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":492,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516368+0000","last_change":"2024-02-13T00:45:15.445626+0000","last_active":"2024-02-13T00:49:42.516368+0000","last_peered":"2024-02-13T00:49:42.516368+0000","last_clean":"2024-02-13T00:49:42.516368+0000","last_became_active":"2024-02-13T00:45:15.445117+0000","last_became_peered":"2024-02-13T00:45:15.445117+0000","last_unstale":"2024-02-13T00:49:42.516368+0000","last_undegraded":"2024-02-13T00:49:42.516368+0000","last_fullsized":"2024-02-13T00:49:42.516368+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.171185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":277,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507218+0000","last_change":"2024-02-13T00:49:12.252240+0000","last_active":"2024-02-13T00:49:43.507218+0000","last_peered":"2024-02-13T00:49:43.507218+0000","last_clean":"2024-02-13T00:49:43.507218+0000","last_became_active":"2024-02-13T00:49:12.251980+0000","last_became_peered":"2024-02-13T00:49:12.251980+0000","last_unstale":"2024-02-13T00:49:43.507218+0000","last_undegraded":"2024-02-13T00:49:43.507218+0000","last_fullsized":"2024-02-13T00:49:43.507218+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.592320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.592320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":618,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516321+0000","last_change":"2024-02-13T00:49:31.072963+0000","last_active":"2024-02-13T00:49:42.516321+0000","last_peered":"2024-02-13T00:49:42.516321+0000","last_clean":"2024-02-13T00:49:42.516321+0000","last_became_active":"2024-02-13T00:44:39.501886+0000","last_became_peered":"2024-02-13T00:44:39.501886+0000","last_unstale":"2024-02-13T00:49:42.516321+0000","last_undegraded":"2024-02-13T00:49:42.516321+0000","last_fullsized":"2024-02-13T00:49:42.516321+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'30","last_scrub_stamp":"2024-02-13T00:49:31.072862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:31.072862+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":538,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508827+0000","last_change":"2024-02-13T00:48:58.861408+0000","last_active":"2024-02-13T00:49:43.508827+0000","last_peered":"2024-02-13T00:49:43.508827+0000","last_clean":"2024-02-13T00:49:43.508827+0000","last_became_active":"2024-02-13T00:46:01.457926+0000","last_became_peered":"2024-02-13T00:46:01.457926+0000","last_unstale":"2024-02-13T00:49:43.508827+0000","last_undegraded":"2024-02-13T00:49:43.508827+0000","last_fullsized":"2024-02-13T00:49:43.508827+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:48:58.861325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:58.861325+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":343,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512690+0000","last_change":"2024-02-13T00:47:06.895693+0000","last_active":"2024-02-13T00:49:42.512690+0000","last_peered":"2024-02-13T00:49:42.512690+0000","last_clean":"2024-02-13T00:49:42.512690+0000","last_became_active":"2024-02-13T00:45:15.435404+0000","last_became_peered":"2024-02-13T00:45:15.435404+0000","last_unstale":"2024-02-13T00:49:42.512690+0000","last_undegraded":"2024-02-13T00:49:42.512690+0000","last_fullsized":"2024-02-13T00:49:42.512690+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:06.895612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:47:06.895612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":493,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.510043+0000","last_change":"2024-02-13T00:46:01.450436+0000","last_active":"2024-02-13T00:49:43.510043+0000","last_peered":"2024-02-13T00:49:43.510043+0000","last_clean":"2024-02-13T00:49:43.510043+0000","last_became_active":"2024-02-13T00:46:01.449234+0000","last_became_peered":"2024-02-13T00:46:01.449234+0000","last_unstale":"2024-02-13T00:49:43.510043+0000","last_undegraded":"2024-02-13T00:49:43.510043+0000","last_fullsized":"2024-02-13T00:49:43.510043+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.110742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.110742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"467'27","reported_seq":686,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.510030+0000","last_change":"2024-02-13T00:49:35.834793+0000","last_active":"2024-02-13T00:49:43.510030+0000","last_peered":"2024-02-13T00:49:43.510030+0000","last_clean":"2024-02-13T00:49:43.510030+0000","last_became_active":"2024-02-13T00:46:01.456876+0000","last_became_peered":"2024-02-13T00:46:01.456876+0000","last_unstale":"2024-02-13T00:49:43.510030+0000","last_undegraded":"2024-02-13T00:49:43.510030+0000","last_fullsized":"2024-02-13T00:49:43.510030+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:35.834669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.834669+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":250,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.511958+0000","last_change":"2024-02-13T00:46:30.095387+0000","last_active":"2024-02-13T00:49:42.511958+0000","last_peered":"2024-02-13T00:49:42.511958+0000","last_clean":"2024-02-13T00:49:42.511958+0000","last_became_active":"2024-02-13T00:44:50.441300+0000","last_became_peered":"2024-02-13T00:44:50.441300+0000","last_unstale":"2024-02-13T00:49:42.511958+0000","last_undegraded":"2024-02-13T00:49:42.511958+0000","last_fullsized":"2024-02-13T00:49:42.511958+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.095307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":266,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507350+0000","last_change":"2024-02-13T00:46:01.454597+0000","last_active":"2024-02-13T00:49:43.507350+0000","last_peered":"2024-02-13T00:49:43.507350+0000","last_clean":"2024-02-13T00:49:43.507350+0000","last_became_active":"2024-02-13T00:46:01.454101+0000","last_became_peered":"2024-02-13T00:46:01.454101+0000","last_unstale":"2024-02-13T00:49:43.507350+0000","last_undegraded":"2024-02-13T00:49:43.507350+0000","last_fullsized":"2024-02-13T00:49:43.507350+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.651091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.651091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":262,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.520564+0000","last_change":"2024-02-13T00:44:50.441196+0000","last_active":"2024-02-13T00:49:42.520564+0000","last_peered":"2024-02-13T00:49:42.520564+0000","last_clean":"2024-02-13T00:49:42.520564+0000","last_became_active":"2024-02-13T00:44:50.438274+0000","last_became_peered":"2024-02-13T00:44:50.438274+0000","last_unstale":"2024-02-13T00:49:42.520564+0000","last_undegraded":"2024-02-13T00:49:42.520564+0000","last_fullsized":"2024-02-13T00:49:42.520564+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.510219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.510219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":419,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517558+0000","last_change":"2024-02-13T00:46:34.141015+0000","last_active":"2024-02-13T00:49:42.517558+0000","last_peered":"2024-02-13T00:49:42.517558+0000","last_clean":"2024-02-13T00:49:42.517558+0000","last_became_active":"2024-02-13T00:45:15.436020+0000","last_became_peered":"2024-02-13T00:45:15.436020+0000","last_unstale":"2024-02-13T00:49:42.517558+0000","last_undegraded":"2024-02-13T00:49:42.517558+0000","last_fullsized":"2024-02-13T00:49:42.517558+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:46:34.140928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.140928+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":258,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512847+0000","last_change":"2024-02-13T00:45:00.311635+0000","last_active":"2024-02-13T00:49:42.512847+0000","last_peered":"2024-02-13T00:49:42.512847+0000","last_clean":"2024-02-13T00:49:42.512847+0000","last_became_active":"2024-02-13T00:44:39.512402+0000","last_became_peered":"2024-02-13T00:44:39.512402+0000","last_unstale":"2024-02-13T00:49:42.512847+0000","last_undegraded":"2024-02-13T00:49:42.512847+0000","last_fullsized":"2024-02-13T00:49:42.512847+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:00.311558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:45:00.311558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":271,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.518635+0000","last_change":"2024-02-13T00:46:01.451840+0000","last_active":"2024-02-13T00:49:42.518635+0000","last_peered":"2024-02-13T00:49:42.518635+0000","last_clean":"2024-02-13T00:49:42.518635+0000","last_became_active":"2024-02-13T00:46:01.450312+0000","last_became_peered":"2024-02-13T00:46:01.450312+0000","last_unstale":"2024-02-13T00:49:42.518635+0000","last_undegraded":"2024-02-13T00:49:42.518635+0000","last_fullsized":"2024-02-13T00:49:42.518635+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.042291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.042291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":343,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507814+0000","last_change":"2024-02-13T00:46:01.454858+0000","last_active":"2024-02-13T00:49:43.507814+0000","last_peered":"2024-02-13T00:49:43.507814+0000","last_clean":"2024-02-13T00:49:43.507814+0000","last_became_active":"2024-02-13T00:46:01.454284+0000","last_became_peered":"2024-02-13T00:46:01.454284+0000","last_unstale":"2024-02-13T00:49:43.507814+0000","last_undegraded":"2024-02-13T00:49:43.507814+0000","last_fullsized":"2024-02-13T00:49:43.507814+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:34.523594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.523594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":502,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517015+0000","last_change":"2024-02-13T00:49:06.154997+0000","last_active":"2024-02-13T00:49:42.517015+0000","last_peered":"2024-02-13T00:49:42.517015+0000","last_clean":"2024-02-13T00:49:42.517015+0000","last_became_active":"2024-02-13T00:46:01.450113+0000","last_became_peered":"2024-02-13T00:46:01.450113+0000","last_unstale":"2024-02-13T00:49:42.517015+0000","last_undegraded":"2024-02-13T00:49:42.517015+0000","last_fullsized":"2024-02-13T00:49:42.517015+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'39","last_scrub_stamp":"2024-02-13T00:49:06.154931+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:49:06.154931+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":518,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508602+0000","last_change":"2024-02-13T00:49:29.916234+0000","last_active":"2024-02-13T00:49:43.508602+0000","last_peered":"2024-02-13T00:49:43.508602+0000","last_clean":"2024-02-13T00:49:43.508602+0000","last_became_active":"2024-02-13T00:46:01.454310+0000","last_became_peered":"2024-02-13T00:46:01.454310+0000","last_unstale":"2024-02-13T00:49:43.508602+0000","last_undegraded":"2024-02-13T00:49:43.508602+0000","last_fullsized":"2024-02-13T00:49:43.508602+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.916152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"404'9","reported_seq":251,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.621141+0000","last_change":"2024-02-13T00:47:37.013003+0000","last_active":"2024-02-13T00:49:43.621141+0000","last_peered":"2024-02-13T00:49:43.621141+0000","last_clean":"2024-02-13T00:49:43.621141+0000","last_became_active":"2024-02-13T00:46:01.448144+0000","last_became_peered":"2024-02-13T00:46:01.448144+0000","last_unstale":"2024-02-13T00:49:43.621141+0000","last_undegraded":"2024-02-13T00:49:43.621141+0000","last_fullsized":"2024-02-13T00:49:43.621141+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9","last_scrub_stamp":"2024-02-13T00:47:37.012934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:37.012934+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":555,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.509963+0000","last_change":"2024-02-13T00:48:26.647460+0000","last_active":"2024-02-13T00:49:43.509963+0000","last_peered":"2024-02-13T00:49:43.509963+0000","last_clean":"2024-02-13T00:49:43.509963+0000","last_became_active":"2024-02-13T00:46:01.447236+0000","last_became_peered":"2024-02-13T00:46:01.447236+0000","last_unstale":"2024-02-13T00:49:43.509963+0000","last_undegraded":"2024-02-13T00:49:43.509963+0000","last_fullsized":"2024-02-13T00:49:43.509963+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:26.647367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:26.647367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":221,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508860+0000","last_change":"2024-02-13T00:46:01.456618+0000","last_active":"2024-02-13T00:49:43.508860+0000","last_peered":"2024-02-13T00:49:43.508860+0000","last_clean":"2024-02-13T00:49:43.508860+0000","last_became_active":"2024-02-13T00:46:01.455989+0000","last_became_peered":"2024-02-13T00:46:01.455989+0000","last_unstale":"2024-02-13T00:49:43.508860+0000","last_undegraded":"2024-02-13T00:49:43.508860+0000","last_fullsized":"2024-02-13T00:49:43.508860+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:16.499780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:16.499780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"467'22","reported_seq":540,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.518232+0000","last_change":"2024-02-13T00:47:35.369710+0000","last_active":"2024-02-13T00:49:42.518232+0000","last_peered":"2024-02-13T00:49:42.518232+0000","last_clean":"2024-02-13T00:49:42.518232+0000","last_became_active":"2024-02-13T00:46:01.460321+0000","last_became_peered":"2024-02-13T00:46:01.460321+0000","last_unstale":"2024-02-13T00:49:42.518232+0000","last_undegraded":"2024-02-13T00:49:42.518232+0000","last_fullsized":"2024-02-13T00:49:42.518232+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:47:35.369616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:35.369616+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":570,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508849+0000","last_change":"2024-02-13T00:49:05.887739+0000","last_active":"2024-02-13T00:49:43.508849+0000","last_peered":"2024-02-13T00:49:43.508849+0000","last_clean":"2024-02-13T00:49:43.508849+0000","last_became_active":"2024-02-13T00:46:01.431926+0000","last_became_peered":"2024-02-13T00:46:01.431926+0000","last_unstale":"2024-02-13T00:49:43.508849+0000","last_undegraded":"2024-02-13T00:49:43.508849+0000","last_fullsized":"2024-02-13T00:49:43.508849+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:05.887665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:05.887665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"363'14","reported_seq":394,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.621286+0000","last_change":"2024-02-13T00:47:07.942748+0000","last_active":"2024-02-13T00:49:43.621286+0000","last_peered":"2024-02-13T00:49:43.621286+0000","last_clean":"2024-02-13T00:49:43.621286+0000","last_became_active":"2024-02-13T00:45:15.423066+0000","last_became_peered":"2024-02-13T00:45:15.423066+0000","last_unstale":"2024-02-13T00:49:43.621286+0000","last_undegraded":"2024-02-13T00:49:43.621286+0000","last_fullsized":"2024-02-13T00:49:43.621286+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:47:07.942668+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:47:07.942668+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":189,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508725+0000","last_change":"2024-02-13T00:47:02.789298+0000","last_active":"2024-02-13T00:49:43.508725+0000","last_peered":"2024-02-13T00:49:43.508725+0000","last_clean":"2024-02-13T00:49:43.508725+0000","last_became_active":"2024-02-13T00:46:01.446960+0000","last_became_peered":"2024-02-13T00:46:01.446960+0000","last_unstale":"2024-02-13T00:49:43.508725+0000","last_undegraded":"2024-02-13T00:49:43.508725+0000","last_fullsized":"2024-02-13T00:49:43.508725+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:02.789189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.789189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":427,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512626+0000","last_change":"2024-02-13T00:46:01.450308+0000","last_active":"2024-02-13T00:49:42.512626+0000","last_peered":"2024-02-13T00:49:42.512626+0000","last_clean":"2024-02-13T00:49:42.512626+0000","last_became_active":"2024-02-13T00:46:01.449694+0000","last_became_peered":"2024-02-13T00:46:01.449694+0000","last_unstale":"2024-02-13T00:49:42.512626+0000","last_undegraded":"2024-02-13T00:49:42.512626+0000","last_fullsized":"2024-02-13T00:49:42.512626+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:27.073034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:27.073034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":623,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512842+0000","last_change":"2024-02-13T00:49:38.007273+0000","last_active":"2024-02-13T00:49:42.512842+0000","last_peered":"2024-02-13T00:49:42.512842+0000","last_clean":"2024-02-13T00:49:42.512842+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:49:42.512842+0000","last_undegraded":"2024-02-13T00:49:42.512842+0000","last_fullsized":"2024-02-13T00:49:42.512842+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'19","last_scrub_stamp":"2024-02-13T00:49:38.007204+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:49:38.007204+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":271,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.515428+0000","last_change":"2024-02-13T00:46:01.463310+0000","last_active":"2024-02-13T00:49:42.515428+0000","last_peered":"2024-02-13T00:49:42.515428+0000","last_clean":"2024-02-13T00:49:42.515428+0000","last_became_active":"2024-02-13T00:46:01.462898+0000","last_became_peered":"2024-02-13T00:46:01.462898+0000","last_unstale":"2024-02-13T00:49:42.515428+0000","last_undegraded":"2024-02-13T00:49:42.515428+0000","last_fullsized":"2024-02-13T00:49:42.515428+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.095888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":334,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508531+0000","last_change":"2024-02-13T00:46:32.638418+0000","last_active":"2024-02-13T00:49:43.508531+0000","last_peered":"2024-02-13T00:49:43.508531+0000","last_clean":"2024-02-13T00:49:43.508531+0000","last_became_active":"2024-02-13T00:46:01.443279+0000","last_became_peered":"2024-02-13T00:46:01.443279+0000","last_unstale":"2024-02-13T00:49:43.508531+0000","last_undegraded":"2024-02-13T00:49:43.508531+0000","last_fullsized":"2024-02-13T00:49:43.508531+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.638349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":537,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506303+0000","last_change":"2024-02-13T00:48:27.659266+0000","last_active":"2024-02-13T00:49:43.506303+0000","last_peered":"2024-02-13T00:49:43.506303+0000","last_clean":"2024-02-13T00:49:43.506303+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:49:43.506303+0000","last_undegraded":"2024-02-13T00:49:43.506303+0000","last_fullsized":"2024-02-13T00:49:43.506303+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:27.659197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:48:27.659197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":267,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516334+0000","last_change":"2024-02-13T00:45:40.233264+0000","last_active":"2024-02-13T00:49:42.516334+0000","last_peered":"2024-02-13T00:49:42.516334+0000","last_clean":"2024-02-13T00:49:42.516334+0000","last_became_active":"2024-02-13T00:45:15.447051+0000","last_became_peered":"2024-02-13T00:45:15.447051+0000","last_unstale":"2024-02-13T00:49:42.516334+0000","last_undegraded":"2024-02-13T00:49:42.516334+0000","last_fullsized":"2024-02-13T00:49:42.516334+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:40.233185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:40.233185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":303,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516266+0000","last_change":"2024-02-13T00:48:34.458358+0000","last_active":"2024-02-13T00:49:42.516266+0000","last_peered":"2024-02-13T00:49:42.516266+0000","last_clean":"2024-02-13T00:49:42.516266+0000","last_became_active":"2024-02-13T00:44:50.435203+0000","last_became_peered":"2024-02-13T00:44:50.435203+0000","last_unstale":"2024-02-13T00:49:42.516266+0000","last_undegraded":"2024-02-13T00:49:42.516266+0000","last_fullsized":"2024-02-13T00:49:42.516266+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:48:34.458264+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:48:34.458264+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":543,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516221+0000","last_change":"2024-02-13T00:46:01.457220+0000","last_active":"2024-02-13T00:49:42.516221+0000","last_peered":"2024-02-13T00:49:42.516221+0000","last_clean":"2024-02-13T00:49:42.516221+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:49:42.516221+0000","last_undegraded":"2024-02-13T00:49:42.516221+0000","last_fullsized":"2024-02-13T00:49:42.516221+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.097481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":557,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512919+0000","last_change":"2024-02-13T00:46:01.453391+0000","last_active":"2024-02-13T00:49:42.512919+0000","last_peered":"2024-02-13T00:49:42.512919+0000","last_clean":"2024-02-13T00:49:42.512919+0000","last_became_active":"2024-02-13T00:46:01.452730+0000","last_became_peered":"2024-02-13T00:46:01.452730+0000","last_unstale":"2024-02-13T00:49:42.512919+0000","last_undegraded":"2024-02-13T00:49:42.512919+0000","last_fullsized":"2024-02-13T00:49:42.512919+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:26.218807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:44:26.218807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":343,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506692+0000","last_change":"2024-02-13T00:48:29.681480+0000","last_active":"2024-02-13T00:49:43.506692+0000","last_peered":"2024-02-13T00:49:43.506692+0000","last_clean":"2024-02-13T00:49:43.506692+0000","last_became_active":"2024-02-13T00:46:01.454287+0000","last_became_peered":"2024-02-13T00:46:01.454287+0000","last_unstale":"2024-02-13T00:49:43.506692+0000","last_undegraded":"2024-02-13T00:49:43.506692+0000","last_fullsized":"2024-02-13T00:49:43.506692+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:29.681414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:29.681414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":276,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512484+0000","last_change":"2024-02-13T00:49:02.260116+0000","last_active":"2024-02-13T00:49:42.512484+0000","last_peered":"2024-02-13T00:49:42.512484+0000","last_clean":"2024-02-13T00:49:42.512484+0000","last_became_active":"2024-02-13T00:46:01.448671+0000","last_became_peered":"2024-02-13T00:46:01.448671+0000","last_unstale":"2024-02-13T00:49:42.512484+0000","last_undegraded":"2024-02-13T00:49:42.512484+0000","last_fullsized":"2024-02-13T00:49:42.512484+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:02.260045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:02.260045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":370,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512403+0000","last_change":"2024-02-13T00:44:40.371566+0000","last_active":"2024-02-13T00:49:42.512403+0000","last_peered":"2024-02-13T00:49:42.512403+0000","last_clean":"2024-02-13T00:49:42.512403+0000","last_became_active":"2024-02-13T00:44:39.501040+0000","last_became_peered":"2024-02-13T00:44:39.501040+0000","last_unstale":"2024-02-13T00:49:42.512403+0000","last_undegraded":"2024-02-13T00:49:42.512403+0000","last_fullsized":"2024-02-13T00:49:42.512403+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:44:32.180113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:44:32.180113+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":502,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517636+0000","last_change":"2024-02-13T00:48:32.441279+0000","last_active":"2024-02-13T00:49:42.517636+0000","last_peered":"2024-02-13T00:49:42.517636+0000","last_clean":"2024-02-13T00:49:42.517636+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:49:42.517636+0000","last_undegraded":"2024-02-13T00:49:42.517636+0000","last_fullsized":"2024-02-13T00:49:42.517636+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.441170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.441170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":267,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512604+0000","last_change":"2024-02-13T00:48:28.353456+0000","last_active":"2024-02-13T00:49:42.512604+0000","last_peered":"2024-02-13T00:49:42.512604+0000","last_clean":"2024-02-13T00:49:42.512604+0000","last_became_active":"2024-02-13T00:45:15.435820+0000","last_became_peered":"2024-02-13T00:45:15.435820+0000","last_unstale":"2024-02-13T00:49:42.512604+0000","last_undegraded":"2024-02-13T00:49:42.512604+0000","last_fullsized":"2024-02-13T00:49:42.512604+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:28.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:28.353347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":257,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512147+0000","last_change":"2024-02-13T00:49:29.052728+0000","last_active":"2024-02-13T00:49:42.512147+0000","last_peered":"2024-02-13T00:49:42.512147+0000","last_clean":"2024-02-13T00:49:42.512147+0000","last_became_active":"2024-02-13T00:45:15.444818+0000","last_became_peered":"2024-02-13T00:45:15.444818+0000","last_unstale":"2024-02-13T00:49:42.512147+0000","last_undegraded":"2024-02-13T00:49:42.512147+0000","last_fullsized":"2024-02-13T00:49:42.512147+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.052640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.052640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":418,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512671+0000","last_change":"2024-02-13T00:46:01.439915+0000","last_active":"2024-02-13T00:49:42.512671+0000","last_peered":"2024-02-13T00:49:42.512671+0000","last_clean":"2024-02-13T00:49:42.512671+0000","last_became_active":"2024-02-13T00:46:01.438745+0000","last_became_peered":"2024-02-13T00:46:01.438745+0000","last_unstale":"2024-02-13T00:49:42.512671+0000","last_undegraded":"2024-02-13T00:49:42.512671+0000","last_fullsized":"2024-02-13T00:49:42.512671+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.104906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.104906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"467'25","reported_seq":330,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512732+0000","last_change":"2024-02-13T00:47:36.368967+0000","last_active":"2024-02-13T00:49:42.512732+0000","last_peered":"2024-02-13T00:49:42.512732+0000","last_clean":"2024-02-13T00:49:42.512732+0000","last_became_active":"2024-02-13T00:46:01.450079+0000","last_became_peered":"2024-02-13T00:46:01.450079+0000","last_unstale":"2024-02-13T00:49:42.512732+0000","last_undegraded":"2024-02-13T00:49:42.512732+0000","last_fullsized":"2024-02-13T00:49:42.512732+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:47:36.368863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.368863+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":276,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.520411+0000","last_change":"2024-02-13T00:46:59.388809+0000","last_active":"2024-02-13T00:49:42.520411+0000","last_peered":"2024-02-13T00:49:42.520411+0000","last_clean":"2024-02-13T00:49:42.520411+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:49:42.520411+0000","last_undegraded":"2024-02-13T00:49:42.520411+0000","last_fullsized":"2024-02-13T00:49:42.520411+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:59.388745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:59.388745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":262,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512083+0000","last_change":"2024-02-13T00:45:15.439752+0000","last_active":"2024-02-13T00:49:42.512083+0000","last_peered":"2024-02-13T00:49:42.512083+0000","last_clean":"2024-02-13T00:49:42.512083+0000","last_became_active":"2024-02-13T00:45:15.437943+0000","last_became_peered":"2024-02-13T00:45:15.437943+0000","last_unstale":"2024-02-13T00:49:42.512083+0000","last_undegraded":"2024-02-13T00:49:42.512083+0000","last_fullsized":"2024-02-13T00:49:42.512083+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.284740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.284740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":470,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508540+0000","last_change":"2024-02-13T00:44:58.969806+0000","last_active":"2024-02-13T00:49:43.508540+0000","last_peered":"2024-02-13T00:49:43.508540+0000","last_clean":"2024-02-13T00:49:43.508540+0000","last_became_active":"2024-02-13T00:44:50.439512+0000","last_became_peered":"2024-02-13T00:44:50.439512+0000","last_unstale":"2024-02-13T00:49:43.508540+0000","last_undegraded":"2024-02-13T00:49:43.508540+0000","last_fullsized":"2024-02-13T00:49:43.508540+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:58.969692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:44:58.969692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":390,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517755+0000","last_change":"2024-02-13T00:49:35.101413+0000","last_active":"2024-02-13T00:49:42.517755+0000","last_peered":"2024-02-13T00:49:42.517755+0000","last_clean":"2024-02-13T00:49:42.517755+0000","last_became_active":"2024-02-13T00:45:15.430749+0000","last_became_peered":"2024-02-13T00:45:15.430749+0000","last_unstale":"2024-02-13T00:49:42.517755+0000","last_undegraded":"2024-02-13T00:49:42.517755+0000","last_fullsized":"2024-02-13T00:49:42.517755+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:35.101298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.101298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":659,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.510192+0000","last_change":"2024-02-13T00:48:53.841888+0000","last_active":"2024-02-13T00:49:43.510192+0000","last_peered":"2024-02-13T00:49:43.510192+0000","last_clean":"2024-02-13T00:49:43.510192+0000","last_became_active":"2024-02-13T00:46:01.443721+0000","last_became_peered":"2024-02-13T00:46:01.443721+0000","last_unstale":"2024-02-13T00:49:43.510192+0000","last_undegraded":"2024-02-13T00:49:43.510192+0000","last_fullsized":"2024-02-13T00:49:43.510192+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'32","last_scrub_stamp":"2024-02-13T00:48:53.841797+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.841797+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":504,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.511436+0000","last_change":"2024-02-13T00:46:32.298033+0000","last_active":"2024-02-13T00:49:42.511436+0000","last_peered":"2024-02-13T00:49:42.511436+0000","last_clean":"2024-02-13T00:49:42.511436+0000","last_became_active":"2024-02-13T00:46:01.442231+0000","last_became_peered":"2024-02-13T00:46:01.442231+0000","last_unstale":"2024-02-13T00:49:42.511436+0000","last_undegraded":"2024-02-13T00:49:42.511436+0000","last_fullsized":"2024-02-13T00:49:42.511436+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.297918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.297918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":368,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.513818+0000","last_change":"2024-02-13T00:48:33.460377+0000","last_active":"2024-02-13T00:49:42.513818+0000","last_peered":"2024-02-13T00:49:42.513818+0000","last_clean":"2024-02-13T00:49:42.513818+0000","last_became_active":"2024-02-13T00:46:01.439984+0000","last_became_peered":"2024-02-13T00:46:01.439984+0000","last_unstale":"2024-02-13T00:49:42.513818+0000","last_undegraded":"2024-02-13T00:49:42.513818+0000","last_fullsized":"2024-02-13T00:49:42.513818+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:48:33.460247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:33.460247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":501,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517504+0000","last_change":"2024-02-13T00:45:38.084426+0000","last_active":"2024-02-13T00:49:42.517504+0000","last_peered":"2024-02-13T00:49:42.517504+0000","last_clean":"2024-02-13T00:49:42.517504+0000","last_became_active":"2024-02-13T00:45:15.435550+0000","last_became_peered":"2024-02-13T00:45:15.435550+0000","last_unstale":"2024-02-13T00:49:42.517504+0000","last_undegraded":"2024-02-13T00:49:42.517504+0000","last_fullsized":"2024-02-13T00:49:42.517504+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.084358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.084358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":307,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.520697+0000","last_change":"2024-02-13T00:48:55.424021+0000","last_active":"2024-02-13T00:49:42.520697+0000","last_peered":"2024-02-13T00:49:42.520697+0000","last_clean":"2024-02-13T00:49:42.520697+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:49:42.520697+0000","last_undegraded":"2024-02-13T00:49:42.520697+0000","last_fullsized":"2024-02-13T00:49:42.520697+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:55.423952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.423952+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":276,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512833+0000","last_change":"2024-02-13T00:46:56.339969+0000","last_active":"2024-02-13T00:49:42.512833+0000","last_peered":"2024-02-13T00:49:42.512833+0000","last_clean":"2024-02-13T00:49:42.512833+0000","last_became_active":"2024-02-13T00:45:15.436239+0000","last_became_peered":"2024-02-13T00:45:15.436239+0000","last_unstale":"2024-02-13T00:49:42.512833+0000","last_undegraded":"2024-02-13T00:49:42.512833+0000","last_fullsized":"2024-02-13T00:49:42.512833+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:56.339885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.339885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":537,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516089+0000","last_change":"2024-02-13T00:46:01.457036+0000","last_active":"2024-02-13T00:49:42.516089+0000","last_peered":"2024-02-13T00:49:42.516089+0000","last_clean":"2024-02-13T00:49:42.516089+0000","last_became_active":"2024-02-13T00:46:01.453269+0000","last_became_peered":"2024-02-13T00:46:01.453269+0000","last_unstale":"2024-02-13T00:49:42.516089+0000","last_undegraded":"2024-02-13T00:49:42.516089+0000","last_fullsized":"2024-02-13T00:49:42.516089+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.245237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.245237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":267,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.515529+0000","last_change":"2024-02-13T00:45:37.280947+0000","last_active":"2024-02-13T00:49:42.515529+0000","last_peered":"2024-02-13T00:49:42.515529+0000","last_clean":"2024-02-13T00:49:42.515529+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:49:42.515529+0000","last_undegraded":"2024-02-13T00:49:42.515529+0000","last_fullsized":"2024-02-13T00:49:42.515529+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.280865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.280865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":321,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516924+0000","last_change":"2024-02-13T00:49:30.086932+0000","last_active":"2024-02-13T00:49:42.516924+0000","last_peered":"2024-02-13T00:49:42.516924+0000","last_clean":"2024-02-13T00:49:42.516924+0000","last_became_active":"2024-02-13T00:44:50.442894+0000","last_became_peered":"2024-02-13T00:44:50.442894+0000","last_unstale":"2024-02-13T00:49:42.516924+0000","last_undegraded":"2024-02-13T00:49:42.516924+0000","last_fullsized":"2024-02-13T00:49:42.516924+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:30.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:30.086856+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":267,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512696+0000","last_change":"2024-02-13T00:49:04.282594+0000","last_active":"2024-02-13T00:49:42.512696+0000","last_peered":"2024-02-13T00:49:42.512696+0000","last_clean":"2024-02-13T00:49:42.512696+0000","last_became_active":"2024-02-13T00:45:15.445403+0000","last_became_peered":"2024-02-13T00:45:15.445403+0000","last_unstale":"2024-02-13T00:49:42.512696+0000","last_undegraded":"2024-02-13T00:49:42.512696+0000","last_fullsized":"2024-02-13T00:49:42.512696+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:04.282529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:04.282529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":267,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.517992+0000","last_change":"2024-02-13T00:47:42.329463+0000","last_active":"2024-02-13T00:49:42.517992+0000","last_peered":"2024-02-13T00:49:42.517992+0000","last_clean":"2024-02-13T00:49:42.517992+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:49:42.517992+0000","last_undegraded":"2024-02-13T00:49:42.517992+0000","last_fullsized":"2024-02-13T00:49:42.517992+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:42.329367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:42.329367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":176,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.518364+0000","last_change":"2024-02-13T00:46:01.451610+0000","last_active":"2024-02-13T00:49:42.518364+0000","last_peered":"2024-02-13T00:49:42.518364+0000","last_clean":"2024-02-13T00:49:42.518364+0000","last_became_active":"2024-02-13T00:46:01.449873+0000","last_became_peered":"2024-02-13T00:46:01.449873+0000","last_unstale":"2024-02-13T00:49:42.518364+0000","last_undegraded":"2024-02-13T00:49:42.518364+0000","last_fullsized":"2024-02-13T00:49:42.518364+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.554065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.554065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":422,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516662+0000","last_change":"2024-02-13T00:46:56.973071+0000","last_active":"2024-02-13T00:49:42.516662+0000","last_peered":"2024-02-13T00:49:42.516662+0000","last_clean":"2024-02-13T00:49:42.516662+0000","last_became_active":"2024-02-13T00:44:39.504670+0000","last_became_peered":"2024-02-13T00:44:39.504670+0000","last_unstale":"2024-02-13T00:49:42.516662+0000","last_undegraded":"2024-02-13T00:49:42.516662+0000","last_fullsized":"2024-02-13T00:49:42.516662+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:46:56.973000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.973000+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":271,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516393+0000","last_change":"2024-02-13T00:46:01.448134+0000","last_active":"2024-02-13T00:49:42.516393+0000","last_peered":"2024-02-13T00:49:42.516393+0000","last_clean":"2024-02-13T00:49:42.516393+0000","last_became_active":"2024-02-13T00:46:01.446635+0000","last_became_peered":"2024-02-13T00:46:01.446635+0000","last_unstale":"2024-02-13T00:49:42.516393+0000","last_undegraded":"2024-02-13T00:49:42.516393+0000","last_fullsized":"2024-02-13T00:49:42.516393+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:28.101254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:28.101254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":267,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.520324+0000","last_change":"2024-02-13T00:46:01.461188+0000","last_active":"2024-02-13T00:49:42.520324+0000","last_peered":"2024-02-13T00:49:42.520324+0000","last_clean":"2024-02-13T00:49:42.520324+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:49:42.520324+0000","last_undegraded":"2024-02-13T00:49:42.520324+0000","last_fullsized":"2024-02-13T00:49:42.520324+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.238152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.238152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":511,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507398+0000","last_change":"2024-02-13T00:46:01.443734+0000","last_active":"2024-02-13T00:49:43.507398+0000","last_peered":"2024-02-13T00:49:43.507398+0000","last_clean":"2024-02-13T00:49:43.507398+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:49:43.507398+0000","last_undegraded":"2024-02-13T00:49:43.507398+0000","last_fullsized":"2024-02-13T00:49:43.507398+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.621170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.621170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":416,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507382+0000","last_change":"2024-02-13T00:46:27.706096+0000","last_active":"2024-02-13T00:49:43.507382+0000","last_peered":"2024-02-13T00:49:43.507382+0000","last_clean":"2024-02-13T00:49:43.507382+0000","last_became_active":"2024-02-13T00:44:50.415894+0000","last_became_peered":"2024-02-13T00:44:50.415894+0000","last_unstale":"2024-02-13T00:49:43.507382+0000","last_undegraded":"2024-02-13T00:49:43.507382+0000","last_fullsized":"2024-02-13T00:49:43.507382+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:46:27.706006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:27.706006+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":351,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.067761+0000","last_change":"2024-02-13T00:49:43.067713+0000","last_active":"2024-02-13T00:49:43.067761+0000","last_peered":"2024-02-13T00:49:43.067761+0000","last_clean":"2024-02-13T00:49:43.067761+0000","last_became_active":"2024-02-13T00:46:01.451790+0000","last_became_peered":"2024-02-13T00:46:01.451790+0000","last_unstale":"2024-02-13T00:49:43.067761+0000","last_undegraded":"2024-02-13T00:49:43.067761+0000","last_fullsized":"2024-02-13T00:49:43.067761+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:43.067615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:43.067615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":539,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.507137+0000","last_change":"2024-02-13T00:47:36.885611+0000","last_active":"2024-02-13T00:49:43.507137+0000","last_peered":"2024-02-13T00:49:43.507137+0000","last_clean":"2024-02-13T00:49:43.507137+0000","last_became_active":"2024-02-13T00:45:15.419668+0000","last_became_peered":"2024-02-13T00:45:15.419668+0000","last_unstale":"2024-02-13T00:49:43.507137+0000","last_undegraded":"2024-02-13T00:49:43.507137+0000","last_fullsized":"2024-02-13T00:49:43.507137+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.885510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.885510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":445,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.509899+0000","last_change":"2024-02-13T00:46:01.457482+0000","last_active":"2024-02-13T00:49:43.509899+0000","last_peered":"2024-02-13T00:49:43.509899+0000","last_clean":"2024-02-13T00:49:43.509899+0000","last_became_active":"2024-02-13T00:46:01.455940+0000","last_became_peered":"2024-02-13T00:46:01.455940+0000","last_unstale":"2024-02-13T00:49:43.509899+0000","last_undegraded":"2024-02-13T00:49:43.509899+0000","last_fullsized":"2024-02-13T00:49:43.509899+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:21.087199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:21.087199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":430,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512574+0000","last_change":"2024-02-13T00:46:36.089483+0000","last_active":"2024-02-13T00:49:42.512574+0000","last_peered":"2024-02-13T00:49:42.512574+0000","last_clean":"2024-02-13T00:49:42.512574+0000","last_became_active":"2024-02-13T00:45:15.419364+0000","last_became_peered":"2024-02-13T00:45:15.419364+0000","last_unstale":"2024-02-13T00:49:42.512574+0000","last_undegraded":"2024-02-13T00:49:42.512574+0000","last_fullsized":"2024-02-13T00:49:42.512574+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:46:36.089402+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:46:36.089402+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":276,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.518685+0000","last_change":"2024-02-13T00:47:47.284092+0000","last_active":"2024-02-13T00:49:42.518685+0000","last_peered":"2024-02-13T00:49:42.518685+0000","last_clean":"2024-02-13T00:49:42.518685+0000","last_became_active":"2024-02-13T00:46:01.451997+0000","last_became_peered":"2024-02-13T00:46:01.451997+0000","last_unstale":"2024-02-13T00:49:42.518685+0000","last_undegraded":"2024-02-13T00:49:42.518685+0000","last_fullsized":"2024-02-13T00:49:42.518685+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.284014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.284014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":487,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516510+0000","last_change":"2024-02-13T00:49:01.133798+0000","last_active":"2024-02-13T00:49:42.516510+0000","last_peered":"2024-02-13T00:49:42.516510+0000","last_clean":"2024-02-13T00:49:42.516510+0000","last_became_active":"2024-02-13T00:46:01.445141+0000","last_became_peered":"2024-02-13T00:46:01.445141+0000","last_unstale":"2024-02-13T00:49:42.516510+0000","last_undegraded":"2024-02-13T00:49:42.516510+0000","last_fullsized":"2024-02-13T00:49:42.516510+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:01.133709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:49:01.133709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":258,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512618+0000","last_change":"2024-02-13T00:45:30.186208+0000","last_active":"2024-02-13T00:49:42.512618+0000","last_peered":"2024-02-13T00:49:42.512618+0000","last_clean":"2024-02-13T00:49:42.512618+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:49:42.512618+0000","last_undegraded":"2024-02-13T00:49:42.512618+0000","last_fullsized":"2024-02-13T00:49:42.512618+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:30.186129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:30.186129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":316,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512732+0000","last_change":"2024-02-13T00:46:06.251713+0000","last_active":"2024-02-13T00:49:42.512732+0000","last_peered":"2024-02-13T00:49:42.512732+0000","last_clean":"2024-02-13T00:49:42.512732+0000","last_became_active":"2024-02-13T00:46:01.442464+0000","last_became_peered":"2024-02-13T00:46:01.442464+0000","last_unstale":"2024-02-13T00:49:42.512732+0000","last_undegraded":"2024-02-13T00:49:42.512732+0000","last_fullsized":"2024-02-13T00:49:42.512732+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:43:44.241358+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:43:44.241358+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":352,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.510085+0000","last_change":"2024-02-13T00:47:47.711551+0000","last_active":"2024-02-13T00:49:43.510085+0000","last_peered":"2024-02-13T00:49:43.510085+0000","last_clean":"2024-02-13T00:49:43.510085+0000","last_became_active":"2024-02-13T00:46:01.447654+0000","last_became_peered":"2024-02-13T00:46:01.447654+0000","last_unstale":"2024-02-13T00:49:43.510085+0000","last_undegraded":"2024-02-13T00:49:43.510085+0000","last_fullsized":"2024-02-13T00:49:43.510085+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.711486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.711486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":186,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.512443+0000","last_change":"2024-02-13T00:47:53.826039+0000","last_active":"2024-02-13T00:49:42.512443+0000","last_peered":"2024-02-13T00:49:42.512443+0000","last_clean":"2024-02-13T00:49:42.512443+0000","last_became_active":"2024-02-13T00:46:01.448971+0000","last_became_peered":"2024-02-13T00:46:01.448971+0000","last_unstale":"2024-02-13T00:49:42.512443+0000","last_undegraded":"2024-02-13T00:49:42.512443+0000","last_fullsized":"2024-02-13T00:49:42.512443+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:53.825971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:53.825971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":513,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516791+0000","last_change":"2024-02-13T00:45:32.233265+0000","last_active":"2024-02-13T00:49:42.516791+0000","last_peered":"2024-02-13T00:49:42.516791+0000","last_clean":"2024-02-13T00:49:42.516791+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:49:42.516791+0000","last_undegraded":"2024-02-13T00:49:42.516791+0000","last_fullsized":"2024-02-13T00:49:42.516791+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:45:32.233197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.233197+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":451,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.506797+0000","last_change":"2024-02-13T00:47:44.757362+0000","last_active":"2024-02-13T00:49:43.506797+0000","last_peered":"2024-02-13T00:49:43.506797+0000","last_clean":"2024-02-13T00:49:43.506797+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:49:43.506797+0000","last_undegraded":"2024-02-13T00:49:43.506797+0000","last_fullsized":"2024-02-13T00:49:43.506797+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:47:44.757262+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:47:44.757262+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":543,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-02-13T00:49:43.508832+0000","last_change":"2024-02-13T00:48:54.818104+0000","last_active":"2024-02-13T00:49:43.508832+0000","last_peered":"2024-02-13T00:49:43.508832+0000","last_clean":"2024-02-13T00:49:43.508832+0000","last_became_active":"2024-02-13T00:45:15.434224+0000","last_became_peered":"2024-02-13T00:45:15.434224+0000","last_unstale":"2024-02-13T00:49:43.508832+0000","last_undegraded":"2024-02-13T00:49:43.508832+0000","last_fullsized":"2024-02-13T00:49:43.508832+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:54.817999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.817999+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":262,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-02-13T00:49:42.516443+0000","last_change":"2024-02-13T00:45:15.435955+0000","last_active":"2024-02-13T00:49:42.516443+0000","last_peered":"2024-02-13T00:49:42.516443+0000","last_clean":"2024-02-13T00:49:42.516443+0000","last_became_active":"2024-02-13T00:45:15.435259+0000","last_became_peered":"2024-02-13T00:45:15.435259+0000","last_unstale":"2024-02-13T00:49:42.516443+0000","last_undegraded":"2024-02-13T00:49:42.516443+0000","last_fullsized":"2024-02-13T00:49:42.516443+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.953989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.953989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":231,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":4},{"poolid":230,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":4},{"poolid":229,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":24,"acting":24,"num_store_stats":4},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1492,"num_read_kb":1217,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":9606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":11656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":970,"ondisk_log_size":970,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":711,"num_read_kb":706,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":227633266835,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":45,"seq":193273528473,"num_pgs":168,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307932,"kb_used_data":2516,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437220,"statfs":{"total":95995035648,"available":95679713280,"internally_reserved":0,"allocated":2576384,"data_stored":1153415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.46700000000000003,"15min":0.54600000000000004},"min":{"1min":0.316,"5min":0.25600000000000001,"15min":0.19700000000000001},"max":{"1min":0.65600000000000003,"5min":0.77500000000000002,"15min":8.1340000000000003},"last":0.434},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52100000000000002,"15min":0.60999999999999999},"min":{"1min":0.34100000000000003,"5min":0.23100000000000001,"15min":0.221},"max":{"1min":0.72799999999999998,"5min":4.2910000000000004,"15min":11.367000000000001},"last":0.50600000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51200000000000001,"15min":0.88400000000000001},"min":{"1min":0.34899999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.63200000000000001,"5min":4.5720000000000001,"15min":54.002000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52300000000000002,"15min":0.68100000000000005},"min":{"1min":0.28999999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66800000000000004,"5min":5.4989999999999997,"15min":12.403},"last":0.441}]},{"osd":2,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52300000000000002,"15min":0.71099999999999997},"min":{"1min":0.31900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":2.8399999999999999,"15min":12.477},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50700000000000001,"15min":0.68000000000000005},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.63200000000000001,"5min":2.1659999999999999,"15min":11.702},"last":0.45100000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.61499999999999999,"15min":0.68100000000000005},"min":{"1min":0.35299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.65700000000000003,"5min":5.931,"15min":8.0519999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54400000000000004,"15min":0.77200000000000002},"min":{"1min":0.44,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":0.65400000000000003,"5min":2.8719999999999999,"15min":19.081},"last":0.52300000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.41499999999999998},"min":{"1min":0.29099999999999998,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":0.51400000000000001,"5min":0.71899999999999997,"15min":1.0649999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.441,"15min":0.438},"min":{"1min":0.315,"5min":0.22800000000000001,"15min":0.221},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.32100000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.33000000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45800000000000002,"15min":0.48299999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61399999999999999,"5min":0.66800000000000004,"15min":1.173},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50800000000000001,"15min":0.53000000000000003},"min":{"1min":0.30099999999999999,"5min":0.28299999999999997,"15min":0.26900000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":1.262},"last":0.34999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575026,"num_pgs":186,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303852,"kb_used_data":2532,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441300,"statfs":{"total":95995035648,"available":95683891200,"internally_reserved":0,"allocated":2592768,"data_stored":1153176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34399999999999997,"15min":0.95099999999999996},"min":{"1min":0.20699999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.52700000000000002,"5min":1.1080000000000001,"15min":61.856000000000002},"last":0.372},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33600000000000002,"15min":0.66800000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.16500000000000001},"max":{"1min":0.70199999999999996,"5min":1.052,"15min":61.908999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34200000000000003,"15min":0.84699999999999998},"min":{"1min":0.251,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":0.40799999999999997,"5min":0.90100000000000002,"15min":32.161000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.377,"15min":0.82799999999999996},"min":{"1min":0.249,"5min":0.187,"15min":0.187},"max":{"1min":0.54000000000000004,"5min":1.9350000000000001,"15min":29.693000000000001},"last":0.34200000000000003}]},{"osd":3,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.376,"15min":1.2669999999999999},"min":{"1min":0.28100000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.55400000000000005,"5min":1.0900000000000001,"15min":131.23099999999999},"last":0.312},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.39700000000000002,"15min":0.86099999999999999},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.48099999999999998,"5min":1.214,"15min":49.055},"last":0.28699999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:48:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.53100000000000003,"15min":0.75800000000000001},"min":{"1min":0.35999999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76700000000000002,"5min":1.246,"15min":16.547000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.54800000000000004,"15min":0.70699999999999996},"min":{"1min":0.34699999999999998,"5min":0.27800000000000002,"15min":0.24199999999999999},"max":{"1min":0.79500000000000004,"5min":0.89200000000000002,"15min":14.731999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Tue Feb 13 00:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.60799999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54700000000000004,"15min":0.71799999999999997},"min":{"1min":0.41299999999999998,"5min":0.29699999999999999,"15min":0.29099999999999998},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":14.772},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.54600000000000004,"15min":0.92500000000000004},"min":{"1min":0.40500000000000003,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":26.600000000000001},"last":0.43099999999999999}]},{"osd":7,"last update":"Tue Feb 13 00:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54800000000000004,"15min":0.71999999999999997},"min":{"1min":0.42999999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76900000000000002,"5min":0.95899999999999996,"15min":14.794},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56200000000000006,"15min":0.92800000000000005},"min":{"1min":0.38500000000000001,"5min":0.33000000000000002,"15min":0.32200000000000001},"max":{"1min":0.77900000000000003,"5min":0.80100000000000005,"15min":26.693999999999999},"last":0.45600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738550,"num_pgs":158,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303844,"kb_used_data":2524,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441308,"statfs":{"total":95995035648,"available":95683899392,"internally_reserved":0,"allocated":2584576,"data_stored":1162542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.378,"15min":0.621},"min":{"1min":0.25,"5min":0.191,"15min":0.189},"max":{"1min":0.59999999999999998,"5min":2.0379999999999998,"15min":30.184000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.31,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.17599999999999999},"max":{"1min":0.52300000000000002,"5min":5.6959999999999997,"15min":5.6959999999999997},"last":0.42099999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.51800000000000002,"15min":0.499},"min":{"1min":0.251,"5min":0.245,"15min":0.20999999999999999},"max":{"1min":0.46999999999999997,"5min":7.0839999999999996,"15min":7.0839999999999996},"last":0.376},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.47399999999999998,"15min":0.66900000000000004},"min":{"1min":0.27200000000000002,"5min":0.248,"15min":0.185},"max":{"1min":0.48599999999999999,"5min":3.6949999999999998,"15min":24.459},"last":0.31900000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.47499999999999998,"15min":0.48599999999999999},"min":{"1min":0.28699999999999998,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":0.628,"5min":4.8710000000000004,"15min":8.718},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.28299999999999997,"5min":0.26900000000000002,"15min":0.222},"max":{"1min":0.67200000000000004,"5min":7.1139999999999999,"15min":7.1139999999999999},"last":0.45400000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.64200000000000002,"15min":0.79400000000000004},"min":{"1min":0.38500000000000001,"5min":0.29899999999999999,"15min":0.24399999999999999},"max":{"1min":0.67300000000000004,"5min":7.056,"15min":24.381},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.68999999999999995,"15min":0.65300000000000002},"min":{"1min":0.40899999999999997,"5min":0.31,"15min":0.28699999999999998},"max":{"1min":0.63100000000000001,"5min":6.1859999999999999,"15min":8.6950000000000003},"last":0.624}]},{"osd":5,"last update":"Tue Feb 13 00:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.59499999999999997}]},{"osd":6,"last update":"Tue Feb 13 00:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.35999999999999999,"5min":0.34100000000000003,"15min":0.31},"max":{"1min":0.79500000000000004,"5min":3.302,"15min":8.6129999999999995},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.70999999999999996,"15min":0.65100000000000002},"min":{"1min":0.34000000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.89200000000000002,"5min":14.263,"15min":14.263},"last":0.63200000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.69099999999999995,"15min":0.65800000000000003},"min":{"1min":0.39700000000000002,"5min":0.35999999999999999,"15min":0.317},"max":{"1min":0.68600000000000005,"5min":7.7720000000000002,"15min":8.6679999999999993},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.747,"15min":0.66900000000000004},"min":{"1min":0.40400000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3700000000000001,"5min":8.1609999999999996,"15min":8.1609999999999996},"last":0.58299999999999996}]}]},{"osd":2,"up_from":0,"seq":77309411501,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182567,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986339,"num_pgs":163,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299724,"kb_used_data":2500,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445428,"statfs":{"total":95995035648,"available":95688118272,"internally_reserved":0,"allocated":2560000,"data_stored":1152883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.52400000000000002,"15min":0.59499999999999997},"min":{"1min":0.36499999999999999,"5min":0.23899999999999999,"15min":0.21199999999999999},"max":{"1min":0.66400000000000003,"5min":5.3319999999999999,"15min":9.3399999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.63100000000000001,"15min":0.61199999999999999},"min":{"1min":0.35799999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.65200000000000002,"5min":11.532,"15min":13.833},"last":0.54500000000000004}]},{"osd":1,"last update":"Tue Feb 13 00:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.59599999999999997,"15min":0.75700000000000001},"min":{"1min":0.40799999999999997,"5min":0.25,"15min":0.25},"max":{"1min":0.61099999999999999,"5min":7.5410000000000004,"15min":53.188000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.59499999999999997,"15min":0.69899999999999995},"min":{"1min":0.36399999999999999,"5min":0.28599999999999998,"15min":0.25600000000000001},"max":{"1min":0.66300000000000003,"5min":11.497999999999999,"15min":44.793999999999997},"last":0.63}]},{"osd":2,"last update":"Tue Feb 13 00:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.60299999999999998,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.31900000000000001,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":4.2199999999999998,"15min":8.6829999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.60399999999999998,"15min":0.629},"min":{"1min":0.35699999999999998,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.57499999999999996,"5min":4.0730000000000004,"15min":18.911000000000001},"last":0.52200000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54000000000000004,"15min":0.65600000000000003},"min":{"1min":0.38200000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68999999999999995,"5min":4.0090000000000003,"15min":23.5},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.55300000000000005,"15min":0.71399999999999997},"min":{"1min":0.35599999999999998,"5min":0.32100000000000001,"15min":0.30599999999999999},"max":{"1min":0.60699999999999998,"5min":3.9729999999999999,"15min":36.152999999999999},"last":0.47299999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.69399999999999995}]},{"osd":6,"last update":"Tue Feb 13 00:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.46000000000000002,"15min":0.436},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.25700000000000001},"max":{"1min":0.60999999999999999,"5min":3.153,"15min":3.153},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.497,"15min":0.46300000000000002},"min":{"1min":0.32700000000000001,"5min":0.25800000000000001,"15min":0.254},"max":{"1min":0.65900000000000003,"5min":4.9390000000000001,"15min":4.9390000000000001},"last":0.65900000000000003}]},{"osd":7,"last update":"Tue Feb 13 00:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48699999999999999,"15min":0.47699999999999998},"min":{"1min":0.374,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70099999999999996,"5min":1.575,"15min":1.575},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.495,"15min":0.47099999999999997},"min":{"1min":0.308,"5min":0.29699999999999999,"15min":0.28100000000000003},"max":{"1min":0.71399999999999997,"5min":2.46,"15min":2.46},"last":0.60899999999999999}]}]},{"osd":5,"up_from":0,"seq":2057289334805,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":3328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":230,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":230,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":230,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":230,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":231,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":231,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":231,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":231,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:49:49.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:49:49.421 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:49.442 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:49.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:49.448 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:49.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:49.579 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:49.600 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:49.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:49.651 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:48.069070+0000 osd.6 (osd.6) 177 : cluster [DBG] 6.27 scrub starts 2024-02-13T00:49:49.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:48.070191+0000 osd.6 (osd.6) 178 : cluster [DBG] 6.27 scrub ok 2024-02-13T00:49:49.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:48.247574+0000 mgr.y (mgr.14544) 494 : audit [DBG] from='client.37805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:49:49.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:48.480524+0000 mgr.y (mgr.14544) 495 : cluster [DBG] pgmap v860: 236 pgs: 236 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:49.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:49.502711+0000 mon.a (mon.0) 3002 : cluster [DBG] osdmap e575: 8 total, 8 up, 4 in 2024-02-13T00:49:49.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:49.524408+0000 mon.c (mon.2) 449 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:49.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:49.524774+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:49.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:49.525362+0000 mon.c (mon.2) 450 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:49.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:49.525637+0000 mon.a (mon.0) 3004 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:49.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:49.526073+0000 mon.c (mon.2) 451 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:49:49.652 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:49.526324+0000 mon.a (mon.0) 3005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:49:49.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:48.069070+0000 osd.6 (osd.6) 177 : cluster [DBG] 6.27 scrub starts 2024-02-13T00:49:49.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:48.070191+0000 osd.6 (osd.6) 178 : cluster [DBG] 6.27 scrub ok 2024-02-13T00:49:49.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:48.247574+0000 mgr.y (mgr.14544) 494 : audit [DBG] from='client.37805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:49:49.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:48.480524+0000 mgr.y (mgr.14544) 495 : cluster [DBG] pgmap v860: 236 pgs: 236 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:49.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:49.502711+0000 mon.a (mon.0) 3002 : cluster [DBG] osdmap e575: 8 total, 8 up, 4 in 2024-02-13T00:49:49.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:49.524408+0000 mon.c (mon.2) 449 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:49.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:49.524774+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:49.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:49.525362+0000 mon.c (mon.2) 450 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:49.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:49.525637+0000 mon.a (mon.0) 3004 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:49.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:49.526073+0000 mon.c (mon.2) 451 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:49:49.653 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:49.526324+0000 mon.a (mon.0) 3005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:49:49.704 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:49.732 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:49.751 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:49.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:49.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:49.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:48.069070+0000 osd.6 (osd.6) 177 : cluster [DBG] 6.27 scrub starts 2024-02-13T00:49:49.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:48.070191+0000 osd.6 (osd.6) 178 : cluster [DBG] 6.27 scrub ok 2024-02-13T00:49:49.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:48.247574+0000 mgr.y (mgr.14544) 494 : audit [DBG] from='client.37805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:49:49.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:48.480524+0000 mgr.y (mgr.14544) 495 : cluster [DBG] pgmap v860: 236 pgs: 236 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:49.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:49.502711+0000 mon.a (mon.0) 3002 : cluster [DBG] osdmap e575: 8 total, 8 up, 4 in 2024-02-13T00:49:49.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:49.524408+0000 mon.c (mon.2) 449 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:49.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:49.524774+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:49.883 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:49.525362+0000 mon.c (mon.2) 450 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:49.883 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:49.525637+0000 mon.a (mon.0) 3004 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:49.883 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:49.526073+0000 mon.c (mon.2) 451 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:49:49.883 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:49.526324+0000 mon.a (mon.0) 3005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:49:49.884 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:49.903 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:49.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:49.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:50.007 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:50.033 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:50.051 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:50.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:50.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:50.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:50.183 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:50.280 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:50.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:50.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:50.410 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:50.481501+0000 mgr.y (mgr.14544) 496 : cluster [DBG] pgmap v863: 172 pgs: 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:50.507477+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:49:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:50.507564+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e576: 8 total, 8 up, 4 in 2024-02-13T00:49:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:50.512350+0000 mon.c (mon.2) 452 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:50.512679+0000 mon.a (mon.0) 3008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:51.827 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:50.481501+0000 mgr.y (mgr.14544) 496 : cluster [DBG] pgmap v863: 172 pgs: 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:51.827 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:50.507477+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:49:51.827 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:50.507564+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e576: 8 total, 8 up, 4 in 2024-02-13T00:49:51.827 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:50.512350+0000 mon.c (mon.2) 452 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:51.828 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:50.512679+0000 mon.a (mon.0) 3008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:51.828 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:50.481501+0000 mgr.y (mgr.14544) 496 : cluster [DBG] pgmap v863: 172 pgs: 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:51.828 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:50.507477+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:49:51.828 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:50.507564+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e576: 8 total, 8 up, 4 in 2024-02-13T00:49:51.828 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:50.512350+0000 mon.c (mon.2) 452 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:51.828 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:50.512679+0000 mon.a (mon.0) 3008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:49:52.489 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:52.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:52.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:49:52.624 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:52.744 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:51.759534+0000 mon.a (mon.0) 3009 : cluster [DBG] osdmap e577: 8 total, 8 up, 4 in 2024-02-13T00:49:52.744 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:51.947730+0000 osd.1 (osd.1) 249 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:52.745 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:51.947961+0000 osd.1 (osd.1) 250 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:52.745 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:51.759534+0000 mon.a (mon.0) 3009 : cluster [DBG] osdmap e577: 8 total, 8 up, 4 in 2024-02-13T00:49:52.745 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:51.947730+0000 osd.1 (osd.1) 249 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:52.745 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:51.947961+0000 osd.1 (osd.1) 250 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:52.750 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:52.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:49:52.759 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:49:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:51.759534+0000 mon.a (mon.0) 3009 : cluster [DBG] osdmap e577: 8 total, 8 up, 4 in 2024-02-13T00:49:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:51.947730+0000 osd.1 (osd.1) 249 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:51.947961+0000 osd.1 (osd.1) 250 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:52.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:52.881 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.990+0000 7f4b8d23f700 1 -- 172.21.15.162:0/951832355 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4b880f73a0 msgr2=0x7f4b881039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.990+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/951832355 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4b880f73a0 0x7f4b881039a0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f4b80009c30 tx=0x7f4b80004c50).stop 2024-02-13T00:49:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.990+0000 7f4b8d23f700 1 -- 172.21.15.162:0/951832355 shutdown_connections 2024-02-13T00:49:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.990+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/951832355 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4b880f73a0 0x7f4b881039a0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.990+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/951832355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b880fcb30 0x7f4b880fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.990+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/951832355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b880fd490 0x7f4b880f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.990+0000 7f4b8d23f700 1 -- 172.21.15.162:0/951832355 >> 172.21.15.162:0/951832355 conn(0x7f4b8806eb90 msgr2=0x7f4b880f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.990+0000 7f4b8d23f700 1 -- 172.21.15.162:0/951832355 shutdown_connections 2024-02-13T00:49:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.990+0000 7f4b8d23f700 1 -- 172.21.15.162:0/951832355 wait complete. 2024-02-13T00:49:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.990+0000 7f4b8d23f700 1 librados: starting msgr at 2024-02-13T00:49:53.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.990+0000 7f4b8d23f700 1 librados: starting objecter 2024-02-13T00:49:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8d23f700 1 Processor -- start 2024-02-13T00:49:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8d23f700 1 -- start start 2024-02-13T00:49:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8d23f700 1 librados: setting wanted keys 2024-02-13T00:49:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8d23f700 1 librados: calling monclient init 2024-02-13T00:49:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8d23f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b880f73a0 0x7f4b880f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8d23f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b880fcb30 0x7f4b880f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8d23f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4b880fd490 0x7f4b880f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8d23f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b88116440 con 0x7f4b880f73a0 2024-02-13T00:49:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8d23f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b881165b0 con 0x7f4b880fcb30 2024-02-13T00:49:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8d23f700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b88116770 con 0x7f4b880fd490 2024-02-13T00:49:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8759e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4b880fd490 0x7f4b880f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b86d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b880f73a0 0x7f4b880f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b86d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b880f73a0 0x7f4b880f8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:57658/0 (socket says 172.21.15.162:57658) 2024-02-13T00:49:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8759e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4b880fd490 0x7f4b880f9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:42972/0 (socket says 172.21.15.162:42972) 2024-02-13T00:49:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b86d9d700 1 -- 172.21.15.162:0/2990219646 learned_addr learned my addr 172.21.15.162:0/2990219646 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.991+0000 7f4b8659c700 1 --2- 172.21.15.162:0/2990219646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b880fcb30 0x7f4b880f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b8759e700 1 -- 172.21.15.162:0/2990219646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b880fcb30 msgr2=0x7f4b880f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b8759e700 1 --2- 172.21.15.162:0/2990219646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b880fcb30 0x7f4b880f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b8759e700 1 -- 172.21.15.162:0/2990219646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b880f73a0 msgr2=0x7f4b880f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b8759e700 1 --2- 172.21.15.162:0/2990219646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b880f73a0 0x7f4b880f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b8759e700 1 -- 172.21.15.162:0/2990219646 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b800097c0 con 0x7f4b880fd490 2024-02-13T00:49:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b86d9d700 1 --2- 172.21.15.162:0/2990219646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b880f73a0 0x7f4b880f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:49:53.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b8759e700 1 --2- 172.21.15.162:0/2990219646 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4b880fd490 0x7f4b880f9310 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 rx=0x7f4b80009c30 tx=0x7f4b80004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b6ffff700 10 client.?.objecter ms_handle_connect 0x7f4b880fd490 2024-02-13T00:49:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b6ffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8000e040 con 0x7f4b880fd490 2024-02-13T00:49:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b881169f0 con 0x7f4b880fd490 2024-02-13T00:49:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b80018040 con 0x7f4b880fd490 2024-02-13T00:49:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80003b90 con 0x7f4b880fd490 2024-02-13T00:49:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b8d23f700 10 client.37823.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b88116e10 con 0x7f4b880fd490 2024-02-13T00:49:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.992+0000 7f4b8d23f700 1 librados: init done 2024-02-13T00:49:53.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.994+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4b88106090 con 0x7f4b880fd490 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.994+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4b80007bf0 con 0x7f4b880fd490 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.994+0000 7f4b6ffff700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4b70062c90 0x7f4b70065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.994+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(578..578 src has 449..578) v4 ==== 8909+0+0 (secure 0 0 0) 0x7f4b8007e570 con 0x7f4b880fd490 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.994+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 osd_map(578..578 src has 449..578) v4 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.994+0000 7f4b6ffff700 3 client.37823.objecter handle_osd_map got epochs [578,578] > 0 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.994+0000 7f4b6ffff700 3 client.37823.objecter handle_osd_map decoding full epoch 578 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.994+0000 7f4b86d9d700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4b70062c90 0x7f4b70065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.994+0000 7f4b6ffff700 20 client.37823.objecter dump_active .. 0 homeless 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.994+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v578) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4b80085070 con 0x7f4b880fd490 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b6ffff700 10 client.37823.objecter ms_handle_connect 0x7f4b70062c90 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b86d9d700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4b70062c90 0x7f4b70065150 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 rx=0x7f4b78004fd0 tx=0x7f4b7800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 1 2024-02-13T00:49:53.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4b880ffdf0 0x7f4b88100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:53.032 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:49:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b880f9e10 osd=0 3 2024-02-13T00:49:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b880f9e10 osd=0 3 2024-02-13T00:49:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b880f9e10 osd=0 3 2024-02-13T00:49:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 1 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b880f9e10 osd=0 3 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 0 1 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b880f9e10 osd=0 4 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8d23f700 10 client.37823.objecter _send_command 1 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b88100700 con 0x7f4b880ffdf0 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8659c700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4b880ffdf0 0x7f4b88100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b6ffff700 10 client.37823.objecter ms_handle_connect 0x7f4b880ffdf0 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.995+0000 7f4b8659c700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4b880ffdf0 0x7f4b88100210 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.996+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4b88100700 con 0x7f4b880ffdf0 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.996+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.996+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 1 = system:0 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.996+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b880f9e10 osd=0 4 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:52.996+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 0 1 2024-02-13T00:49:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.017+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.017+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 2 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.017+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b880f9e10 osd=0 3 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.017+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b880f9e10 osd=0 3 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.017+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b880f9e10 osd=0 3 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.017+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 2 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.017+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b880f9e10 osd=0 3 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.017+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 0 2 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.017+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b880f9e10 osd=0 4 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.017+0000 7f4b8d23f700 10 client.37823.objecter _send_command 2 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.017+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4b88100870 con 0x7f4b880ffdf0 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4b88100870 con 0x7f4b880ffdf0 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b880f9e10 osd=0 4 2024-02-13T00:49:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 0 2 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 3 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6805/150599780 conn(0x7f4b88028590 0x7f4b8811b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88061a10 osd=1 3 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88061a10 osd=1 3 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88061a10 osd=1 3 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 3 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b88061a10 osd=1 3 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 1 3 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88061a10 osd=1 4 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b8d23f700 10 client.37823.objecter _send_command 3 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b88061b80 con 0x7f4b88028590 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b86d9d700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6805/150599780 conn(0x7f4b88028590 0x7f4b8811b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b6ffff700 10 client.37823.objecter ms_handle_connect 0x7f4b88028590 2024-02-13T00:49:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.019+0000 7f4b86d9d700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6805/150599780 conn(0x7f4b88028590 0x7f4b8811b0b0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.020+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4b88061b80 con 0x7f4b88028590 2024-02-13T00:49:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.020+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:49:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.020+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 3 = system:0 2024-02-13T00:49:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.020+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b88061a10 osd=1 4 2024-02-13T00:49:53.036 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.020+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 1 3 2024-02-13T00:49:53.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:53.037 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:53.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:53.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.036+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:53.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.036+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 4 2024-02-13T00:49:53.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.036+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88061a10 osd=1 3 2024-02-13T00:49:53.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.036+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88061a10 osd=1 3 2024-02-13T00:49:53.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.036+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88061a10 osd=1 3 2024-02-13T00:49:53.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.036+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 4 2024-02-13T00:49:53.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.036+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b88061a10 osd=1 3 2024-02-13T00:49:53.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.036+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 1 4 2024-02-13T00:49:53.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.036+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88061a10 osd=1 4 2024-02-13T00:49:53.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.036+0000 7f4b8d23f700 10 client.37823.objecter _send_command 4 2024-02-13T00:49:53.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.036+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4b8805a0e0 con 0x7f4b88028590 2024-02-13T00:49:53.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4b8805a0e0 con 0x7f4b88028590 2024-02-13T00:49:53.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:53.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b88061a10 osd=1 4 2024-02-13T00:49:53.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 1 4 2024-02-13T00:49:53.043 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 5 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4b88059570 0x7f4b8811e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88059400 osd=2 3 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88059400 osd=2 3 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88059400 osd=2 3 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 5 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b88059400 osd=2 3 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 2 5 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88059400 osd=2 4 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8d23f700 10 client.37823.objecter _send_command 5 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b88028210 con 0x7f4b88059570 2024-02-13T00:49:53.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8659c700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4b88059570 0x7f4b8811e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:53.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b6ffff700 10 client.37823.objecter ms_handle_connect 0x7f4b88059570 2024-02-13T00:49:53.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.039+0000 7f4b8659c700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4b88059570 0x7f4b8811e730 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:53.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.040+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4b88028210 con 0x7f4b88059570 2024-02-13T00:49:53.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.040+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:49:53.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.040+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 5 = system:0 2024-02-13T00:49:53.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.040+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b88059400 osd=2 4 2024-02-13T00:49:53.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.040+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 2 5 2024-02-13T00:49:53.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.056+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:53.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.056+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 6 2024-02-13T00:49:53.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.056+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88059400 osd=2 3 2024-02-13T00:49:53.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.056+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88059400 osd=2 3 2024-02-13T00:49:53.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.056+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88059400 osd=2 3 2024-02-13T00:49:53.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.056+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 6 2024-02-13T00:49:53.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.056+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b88059400 osd=2 3 2024-02-13T00:49:53.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.056+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 2 6 2024-02-13T00:49:53.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.056+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88059400 osd=2 4 2024-02-13T00:49:53.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.056+0000 7f4b8d23f700 10 client.37823.objecter _send_command 6 2024-02-13T00:49:53.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.056+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4b88027650 con 0x7f4b88059570 2024-02-13T00:49:53.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.057+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4b88027650 con 0x7f4b88059570 2024-02-13T00:49:53.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.057+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:53.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.057+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.057+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b88059400 osd=2 4 2024-02-13T00:49:53.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.057+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 2 6 2024-02-13T00:49:53.062 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:49:53.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:53.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 7 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6813/604971011 conn(0x7f4b8811c280 0x7f4b88121db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88027a60 osd=3 3 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88027a60 osd=3 3 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88027a60 osd=3 3 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 7 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b88027a60 osd=3 3 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 3 7 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88027a60 osd=3 4 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8d23f700 10 client.37823.objecter _send_command 7 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b88124a40 con 0x7f4b8811c280 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8759e700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6813/604971011 conn(0x7f4b8811c280 0x7f4b88121db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b6ffff700 10 client.37823.objecter ms_handle_connect 0x7f4b8811c280 2024-02-13T00:49:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.058+0000 7f4b8759e700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6813/604971011 conn(0x7f4b8811c280 0x7f4b88121db0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:53.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.059+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4b88124a40 con 0x7f4b8811c280 2024-02-13T00:49:53.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.059+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:49:53.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.059+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 7 = system:0 2024-02-13T00:49:53.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.059+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b88027a60 osd=3 4 2024-02-13T00:49:53.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.059+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 3 7 2024-02-13T00:49:53.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.074+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:53.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.074+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 8 2024-02-13T00:49:53.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.074+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88027a60 osd=3 3 2024-02-13T00:49:53.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.074+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88027a60 osd=3 3 2024-02-13T00:49:53.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.074+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88027a60 osd=3 3 2024-02-13T00:49:53.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.074+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 8 2024-02-13T00:49:53.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.074+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b88027a60 osd=3 3 2024-02-13T00:49:53.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.074+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 3 8 2024-02-13T00:49:53.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.074+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88027a60 osd=3 4 2024-02-13T00:49:53.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.074+0000 7f4b8d23f700 10 client.37823.objecter _send_command 8 2024-02-13T00:49:53.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.074+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4b8811b7e0 con 0x7f4b8811c280 2024-02-13T00:49:53.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4b8811b7e0 con 0x7f4b8811c280 2024-02-13T00:49:53.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:53.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b88027a60 osd=3 4 2024-02-13T00:49:53.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 3 8 2024-02-13T00:49:53.082 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:49:53.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:53.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 9 2024-02-13T00:49:53.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4b8811f900 0x7f4b88123420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:53.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b8811b5a0 osd=4 3 2024-02-13T00:49:53.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b8811b5a0 osd=4 3 2024-02-13T00:49:53.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b8811b5a0 osd=4 3 2024-02-13T00:49:53.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 9 2024-02-13T00:49:53.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b8811b5a0 osd=4 3 2024-02-13T00:49:53.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 4 9 2024-02-13T00:49:53.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b8811b5a0 osd=4 4 2024-02-13T00:49:53.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b8d23f700 10 client.37823.objecter _send_command 9 2024-02-13T00:49:53.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.076+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b8811b780 con 0x7f4b8811f900 2024-02-13T00:49:53.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.077+0000 7f4b86d9d700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4b8811f900 0x7f4b88123420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:53.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.077+0000 7f4b6ffff700 10 client.37823.objecter ms_handle_connect 0x7f4b8811f900 2024-02-13T00:49:53.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.077+0000 7f4b86d9d700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4b8811f900 0x7f4b88123420 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:53.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.078+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4b8811b780 con 0x7f4b8811f900 2024-02-13T00:49:53.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.078+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:49:53.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.078+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 9 = system:0 2024-02-13T00:49:53.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.078+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b8811b5a0 osd=4 4 2024-02-13T00:49:53.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.078+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 4 9 2024-02-13T00:49:53.096 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.092+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:53.096 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.092+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 10 2024-02-13T00:49:53.096 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.092+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b8811b5a0 osd=4 3 2024-02-13T00:49:53.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.092+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b8811b5a0 osd=4 3 2024-02-13T00:49:53.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.092+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b8811b5a0 osd=4 3 2024-02-13T00:49:53.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.092+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 10 2024-02-13T00:49:53.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.092+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b8811b5a0 osd=4 3 2024-02-13T00:49:53.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.092+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 4 10 2024-02-13T00:49:53.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.092+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b8811b5a0 osd=4 4 2024-02-13T00:49:53.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.092+0000 7f4b8d23f700 10 client.37823.objecter _send_command 10 2024-02-13T00:49:53.098 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.092+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4b8811b780 con 0x7f4b8811f900 2024-02-13T00:49:53.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.094+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4b8811b780 con 0x7f4b8811f900 2024-02-13T00:49:53.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.094+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:53.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.094+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.094+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b8811b5a0 osd=4 4 2024-02-13T00:49:53.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.094+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 4 10 2024-02-13T00:49:53.100 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:49:53.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:53.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 11 2024-02-13T00:49:53.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4b88124010 0x7f4b88124450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:53.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88123990 osd=5 3 2024-02-13T00:49:53.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88123990 osd=5 3 2024-02-13T00:49:53.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88123990 osd=5 3 2024-02-13T00:49:53.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 11 2024-02-13T00:49:53.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b88123990 osd=5 3 2024-02-13T00:49:53.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 5 11 2024-02-13T00:49:53.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88123990 osd=5 4 2024-02-13T00:49:53.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8d23f700 10 client.37823.objecter _send_command 11 2024-02-13T00:49:53.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b8812b3e0 con 0x7f4b88124010 2024-02-13T00:49:53.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8659c700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4b88124010 0x7f4b88124450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:53.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b6ffff700 10 client.37823.objecter ms_handle_connect 0x7f4b88124010 2024-02-13T00:49:53.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.095+0000 7f4b8659c700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4b88124010 0x7f4b88124450 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:53.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.096+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4b8812b3e0 con 0x7f4b88124010 2024-02-13T00:49:53.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.096+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:49:53.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.096+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 11 = system:0 2024-02-13T00:49:53.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.096+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b88123990 osd=5 4 2024-02-13T00:49:53.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.096+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 5 11 2024-02-13T00:49:53.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.110+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:53.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.110+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 12 2024-02-13T00:49:53.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.110+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88123990 osd=5 3 2024-02-13T00:49:53.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.110+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88123990 osd=5 3 2024-02-13T00:49:53.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.110+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88123990 osd=5 3 2024-02-13T00:49:53.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.110+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 12 2024-02-13T00:49:53.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.110+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b88123990 osd=5 3 2024-02-13T00:49:53.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.110+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 5 12 2024-02-13T00:49:53.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.110+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88123990 osd=5 4 2024-02-13T00:49:53.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.110+0000 7f4b8d23f700 10 client.37823.objecter _send_command 12 2024-02-13T00:49:53.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.110+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4b88128c80 con 0x7f4b88124010 2024-02-13T00:49:53.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4b88128c80 con 0x7f4b88124010 2024-02-13T00:49:53.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:53.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b88123990 osd=5 4 2024-02-13T00:49:53.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 5 12 2024-02-13T00:49:53.116 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:49:53.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:53.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 13 2024-02-13T00:49:53.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4b88129300 0x7f4b88129740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:53.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88128c80 osd=6 3 2024-02-13T00:49:53.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88128c80 osd=6 3 2024-02-13T00:49:53.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88128c80 osd=6 3 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 13 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b88128c80 osd=6 3 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 6 13 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88128c80 osd=6 4 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b8d23f700 10 client.37823.objecter _send_command 13 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.112+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b8812b3c0 con 0x7f4b88129300 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.113+0000 7f4b8759e700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4b88129300 0x7f4b88129740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.113+0000 7f4b8759e700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4b88129300 0x7f4b88129740 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.113+0000 7f4b6ffff700 10 client.37823.objecter ms_handle_connect 0x7f4b88129300 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.114+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4b8812b3c0 con 0x7f4b88129300 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.114+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.114+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 13 = system:0 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.114+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b88128c80 osd=6 4 2024-02-13T00:49:53.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.114+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 6 13 2024-02-13T00:49:53.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.128+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:53.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.128+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 14 2024-02-13T00:49:53.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.128+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88128c80 osd=6 3 2024-02-13T00:49:53.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.128+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88128c80 osd=6 3 2024-02-13T00:49:53.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.128+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b88128c80 osd=6 3 2024-02-13T00:49:53.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.128+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 14 2024-02-13T00:49:53.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.128+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b88128c80 osd=6 3 2024-02-13T00:49:53.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.128+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 6 14 2024-02-13T00:49:53.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.128+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88128c80 osd=6 4 2024-02-13T00:49:53.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.128+0000 7f4b8d23f700 10 client.37823.objecter _send_command 14 2024-02-13T00:49:53.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.128+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4b8812b3c0 con 0x7f4b88129300 2024-02-13T00:49:53.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4b8812b3c0 con 0x7f4b88129300 2024-02-13T00:49:53.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:53.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b88128c80 osd=6 4 2024-02-13T00:49:53.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 6 14 2024-02-13T00:49:53.134 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:49:53.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:49:53.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 15 2024-02-13T00:49:53.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4b8812db20 0x7f4b8812df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:53.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b8812b3c0 osd=7 3 2024-02-13T00:49:53.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b8812b3c0 osd=7 3 2024-02-13T00:49:53.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b8812b3c0 osd=7 3 2024-02-13T00:49:53.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 15 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b8812b3c0 osd=7 3 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 7 15 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b8812b3c0 osd=7 4 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b8d23f700 10 client.37823.objecter _send_command 15 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.130+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b8812e5d0 con 0x7f4b8812db20 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.131+0000 7f4b86d9d700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4b8812db20 0x7f4b8812df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.131+0000 7f4b6ffff700 10 client.37823.objecter ms_handle_connect 0x7f4b8812db20 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.131+0000 7f4b86d9d700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4b8812db20 0x7f4b8812df60 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.132+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4b8812e5d0 con 0x7f4b8812db20 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.132+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.132+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 15 = system:0 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.132+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b8812b3c0 osd=7 4 2024-02-13T00:49:53.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.132+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 7 15 2024-02-13T00:49:53.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:53.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.146+0000 7f4b8d23f700 10 client.37823.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:49:53.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.146+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign -1 16 2024-02-13T00:49:53.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.146+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b8812b3c0 osd=7 3 2024-02-13T00:49:53.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.146+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b8812b3c0 osd=7 3 2024-02-13T00:49:53.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.146+0000 7f4b8d23f700 20 client.37823.objecter _get_session s=0x7f4b8812b3c0 osd=7 3 2024-02-13T00:49:53.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.146+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_remove -1 16 2024-02-13T00:49:53.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.146+0000 7f4b8d23f700 20 client.37823.objecter get_session s=0x7f4b8812b3c0 osd=7 3 2024-02-13T00:49:53.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.146+0000 7f4b8d23f700 15 client.37823.objecter _session_command_op_assign 7 16 2024-02-13T00:49:53.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.146+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b8812b3c0 osd=7 4 2024-02-13T00:49:53.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.146+0000 7f4b8d23f700 10 client.37823.objecter _send_command 16 2024-02-13T00:49:53.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.146+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4b8812e5d0 con 0x7f4b8812db20 2024-02-13T00:49:53.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b6ffff700 1 -- 172.21.15.162:0/2990219646 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4b8812e5d0 con 0x7f4b8812db20 2024-02-13T00:49:53.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b6ffff700 10 client.37823.objecter ms_dispatch 0x7f4b88001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:49:53.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b6ffff700 10 client.37823.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b6ffff700 20 client.37823.objecter put_session s=0x7f4b8812b3c0 osd=7 4 2024-02-13T00:49:53.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b6ffff700 15 client.37823.objecter _session_command_op_remove 7 16 2024-02-13T00:49:53.152 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:49:53.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 10 librados: watch_flush enter 2024-02-13T00:49:53.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 10 librados: watch_flush exit 2024-02-13T00:49:53.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 10 client.37823.objecter close_session for osd.0 2024-02-13T00:49:53.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4b880ffdf0 msgr2=0x7f4b88100210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4b880ffdf0 0x7f4b88100210 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b880f9e10 osd=0 1 2024-02-13T00:49:53.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 10 client.37823.objecter close_session for osd.1 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6805/150599780 conn(0x7f4b88028590 msgr2=0x7f4b8811b0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6805/150599780 conn(0x7f4b88028590 0x7f4b8811b0b0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88061a10 osd=1 1 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 10 client.37823.objecter close_session for osd.2 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4b88059570 msgr2=0x7f4b8811e730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4b88059570 0x7f4b8811e730 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88059400 osd=2 1 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 10 client.37823.objecter close_session for osd.3 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6813/604971011 conn(0x7f4b8811c280 msgr2=0x7f4b88121db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6813/604971011 conn(0x7f4b8811c280 0x7f4b88121db0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88027a60 osd=3 1 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8759e700 1 -- 172.21.15.162:0/2990219646 reap_dead start 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 10 client.37823.objecter close_session for osd.4 2024-02-13T00:49:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4b8811f900 msgr2=0x7f4b88123420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4b8811f900 0x7f4b88123420 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b8811b5a0 osd=4 1 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 10 client.37823.objecter close_session for osd.5 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8759e700 1 -- 172.21.15.162:0/2990219646 reap_dead start 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4b88124010 msgr2=0x7f4b88124450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4b88124010 0x7f4b88124450 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88123990 osd=5 1 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 10 client.37823.objecter close_session for osd.6 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4b88129300 msgr2=0x7f4b88129740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4b88129300 0x7f4b88129740 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b88128c80 osd=6 1 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 10 client.37823.objecter close_session for osd.7 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4b8812db20 msgr2=0x7f4b8812df60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4b8812db20 0x7f4b8812df60 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 20 client.37823.objecter put_session s=0x7f4b8812b3c0 osd=7 1 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 20 client.37823.objecter shutdown clearing up homeless session... 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.148+0000 7f4b8d23f700 10 client.37823.objecter successfully canceled tick 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.149+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4b70062c90 msgr2=0x7f4b70065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.149+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4b70062c90 0x7f4b70065150 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 rx=0x7f4b78004fd0 tx=0x7f4b7800b040).stop 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.149+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4b880fd490 msgr2=0x7f4b880f9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.149+0000 7f4b8d23f700 1 --2- 172.21.15.162:0/2990219646 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4b880fd490 0x7f4b880f9310 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 rx=0x7f4b80009c30 tx=0x7f4b80004c50).stop 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.149+0000 7f4b8759e700 1 -- 172.21.15.162:0/2990219646 reap_dead start 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.149+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 shutdown_connections 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.149+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 >> 172.21.15.162:0/2990219646 conn(0x7f4b8806eb90 msgr2=0x7f4b88060e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.149+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 shutdown_connections 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.149+0000 7f4b8d23f700 1 -- 172.21.15.162:0/2990219646 wait complete. 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:53.149+0000 7f4b8d23f700 1 librados: shutdown 2024-02-13T00:49:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.157 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.157 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.157 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.157 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.157 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.157 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.157 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:49:53.170 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:53.190 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:53.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:53.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:53.247 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:49:53.247 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:49:53.247 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:49:53.247 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:49:53.247 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:49:53.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:49:53.325 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:53.345 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:49:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:53.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:49:53.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:53.481 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:53.501 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:53.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:53.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:53.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:53.633 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:53.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:53.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:53.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:52.481978+0000 mgr.y (mgr.14544) 497 : cluster [DBG] pgmap v866: 172 pgs: 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:52.526055+0000 mon.a (mon.0) 3010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-99"}]': finished 2024-02-13T00:49:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:52.526185+0000 mon.a (mon.0) 3011 : cluster [DBG] osdmap e578: 8 total, 8 up, 4 in 2024-02-13T00:49:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:52.995128+0000 mon.c (mon.2) 453 : audit [DBG] from='client.? 172.21.15.162:0/2990219646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:53.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:52.481978+0000 mgr.y (mgr.14544) 497 : cluster [DBG] pgmap v866: 172 pgs: 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:53.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:52.526055+0000 mon.a (mon.0) 3010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-99"}]': finished 2024-02-13T00:49:53.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:52.526185+0000 mon.a (mon.0) 3011 : cluster [DBG] osdmap e578: 8 total, 8 up, 4 in 2024-02-13T00:49:53.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:52.995128+0000 mon.c (mon.2) 453 : audit [DBG] from='client.? 172.21.15.162:0/2990219646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:52.481978+0000 mgr.y (mgr.14544) 497 : cluster [DBG] pgmap v866: 172 pgs: 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:52.526055+0000 mon.a (mon.0) 3010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-99"}]': finished 2024-02-13T00:49:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:52.526185+0000 mon.a (mon.0) 3011 : cluster [DBG] osdmap e578: 8 total, 8 up, 4 in 2024-02-13T00:49:53.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:52.995128+0000 mon.c (mon.2) 453 : audit [DBG] from='client.? 172.21.15.162:0/2990219646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:49:53.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:53.889 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:53.909 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:53.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:53.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:54.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:49:54.039 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:54.693 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:53.525188+0000 mon.a (mon.0) 3012 : cluster [DBG] osdmap e579: 8 total, 8 up, 4 in 2024-02-13T00:49:54.693 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:53.528086+0000 mon.c (mon.2) 454 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:54.693 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:53.534013+0000 mon.a (mon.0) 3013 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:54.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:53.525188+0000 mon.a (mon.0) 3012 : cluster [DBG] osdmap e579: 8 total, 8 up, 4 in 2024-02-13T00:49:54.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:53.528086+0000 mon.c (mon.2) 454 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:54.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:53.534013+0000 mon.a (mon.0) 3013 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:54.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-13T00:49:54.718 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:49:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:54.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:49:54.746 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:54.747 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:53.525188+0000 mon.a (mon.0) 3012 : cluster [DBG] osdmap e579: 8 total, 8 up, 4 in 2024-02-13T00:49:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:53.528086+0000 mon.c (mon.2) 454 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:53.534013+0000 mon.a (mon.0) 3013 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:49:54.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:49:54.855 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:54.876 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:49:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:54.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:49:54.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:49:55.006 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:55.059 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:49:55.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:55.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:49:55.163 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:49:55.195 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:55.216 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:49:55.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:55.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:49:55.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:55.350 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:55.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:49:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:49:55.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:54.482403+0000 mgr.y (mgr.14544) 498 : cluster [DBG] pgmap v869: 212 pgs: 40 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:55.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:54.531804+0000 mon.a (mon.0) 3014 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-99-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:55.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:54.531903+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e580: 8 total, 8 up, 4 in 2024-02-13T00:49:55.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:54.534123+0000 mon.c (mon.2) 455 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:49:55.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:54.534548+0000 mon.a (mon.0) 3016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:49:55.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:54.836845+0000 osd.4 (osd.4) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:55.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:54.837169+0000 osd.4 (osd.4) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:55.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:55.494136+0000 mon.a (mon.0) 3017 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]': finished 2024-02-13T00:49:55.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:55.494212+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e581: 8 total, 8 up, 4 in 2024-02-13T00:49:55.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:55.495524+0000 mon.c (mon.2) 456 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-99", "overlaypool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:49:55.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:55.496596+0000 mon.a (mon.0) 3019 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-99", "overlaypool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:49:55.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:54.482403+0000 mgr.y (mgr.14544) 498 : cluster [DBG] pgmap v869: 212 pgs: 40 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:54.531804+0000 mon.a (mon.0) 3014 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-99-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:54.531903+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e580: 8 total, 8 up, 4 in 2024-02-13T00:49:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:54.534123+0000 mon.c (mon.2) 455 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:49:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:54.534548+0000 mon.a (mon.0) 3016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:49:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:54.836845+0000 osd.4 (osd.4) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:55.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:54.837169+0000 osd.4 (osd.4) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:55.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:55.494136+0000 mon.a (mon.0) 3017 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]': finished 2024-02-13T00:49:55.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:55.494212+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e581: 8 total, 8 up, 4 in 2024-02-13T00:49:55.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:55.495524+0000 mon.c (mon.2) 456 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-99", "overlaypool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:49:55.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:55.496596+0000 mon.a (mon.0) 3019 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-99", "overlaypool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:49:55.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:54.482403+0000 mgr.y (mgr.14544) 498 : cluster [DBG] pgmap v869: 212 pgs: 40 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:49:55.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:54.531804+0000 mon.a (mon.0) 3014 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-99-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:49:55.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:54.531903+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e580: 8 total, 8 up, 4 in 2024-02-13T00:49:55.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:54.534123+0000 mon.c (mon.2) 455 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:49:55.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:54.534548+0000 mon.a (mon.0) 3016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:49:55.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:54.836845+0000 osd.4 (osd.4) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:55.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:54.837169+0000 osd.4 (osd.4) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:55.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:55.494136+0000 mon.a (mon.0) 3017 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]': finished 2024-02-13T00:49:55.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:55.494212+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e581: 8 total, 8 up, 4 in 2024-02-13T00:49:55.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:55.495524+0000 mon.c (mon.2) 456 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-99", "overlaypool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:49:55.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:55.496596+0000 mon.a (mon.0) 3019 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-99", "overlaypool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:49:56.135 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:56.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:49:56.417 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:49:56] "GET /metrics HTTP/1.1" 200 188442 "" "Prometheus/2.33.4" 2024-02-13T00:49:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:54.589743+0000 osd.3 (osd.3) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:54.589900+0000 osd.3 (osd.3) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:55.770798+0000 mgr.y (mgr.14544) 499 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:56.494184+0000 mon.a (mon.0) 3020 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-99", "overlaypool": "test-rados-api-smithi162-84057-99-cache"}]': finished 2024-02-13T00:49:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:56.494261+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e582: 8 total, 8 up, 4 in 2024-02-13T00:49:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:56.495249+0000 mon.c (mon.2) 457 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-99-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:49:56.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:56.495623+0000 mon.a (mon.0) 3022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-99-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:49:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:54.589743+0000 osd.3 (osd.3) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:54.589900+0000 osd.3 (osd.3) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:55.770798+0000 mgr.y (mgr.14544) 499 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:56.494184+0000 mon.a (mon.0) 3020 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-99", "overlaypool": "test-rados-api-smithi162-84057-99-cache"}]': finished 2024-02-13T00:49:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:56.494261+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e582: 8 total, 8 up, 4 in 2024-02-13T00:49:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:56.495249+0000 mon.c (mon.2) 457 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-99-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:49:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:56.495623+0000 mon.a (mon.0) 3022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-99-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:49:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:54.589743+0000 osd.3 (osd.3) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:49:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:54.589900+0000 osd.3 (osd.3) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:49:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:55.770798+0000 mgr.y (mgr.14544) 499 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:49:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:56.494184+0000 mon.a (mon.0) 3020 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-99", "overlaypool": "test-rados-api-smithi162-84057-99-cache"}]': finished 2024-02-13T00:49:56.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:56.494261+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e582: 8 total, 8 up, 4 in 2024-02-13T00:49:56.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:56.495249+0000 mon.c (mon.2) 457 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-99-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:49:56.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:56.495623+0000 mon.a (mon.0) 3022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-99-cache", "mode": "writeback"}]: dispatch 2024-02-13T00:49:57.601 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:57.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:57.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:57.734 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:56.482823+0000 mgr.y (mgr.14544) 500 : cluster [DBG] pgmap v872: 212 pgs: 20 creating+activating, 192 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:57.491234+0000 mon.a (mon.0) 3023 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:49:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:57.495440+0000 mon.a (mon.0) 3024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-99-cache", "mode": "writeback"}]': finished 2024-02-13T00:49:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:57.495508+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e583: 8 total, 8 up, 4 in 2024-02-13T00:49:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:57.497282+0000 mon.c (mon.2) 458 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:49:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:57.497556+0000 mon.a (mon.0) 3026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:49:57.807 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:57.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:56.482823+0000 mgr.y (mgr.14544) 500 : cluster [DBG] pgmap v872: 212 pgs: 20 creating+activating, 192 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:57.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:57.491234+0000 mon.a (mon.0) 3023 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:49:57.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:57.495440+0000 mon.a (mon.0) 3024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-99-cache", "mode": "writeback"}]': finished 2024-02-13T00:49:57.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:57.495508+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e583: 8 total, 8 up, 4 in 2024-02-13T00:49:57.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:57.497282+0000 mon.c (mon.2) 458 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:49:57.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:57.497556+0000 mon.a (mon.0) 3026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:49:57.810 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:56.482823+0000 mgr.y (mgr.14544) 500 : cluster [DBG] pgmap v872: 212 pgs: 20 creating+activating, 192 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:57.810 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:57.491234+0000 mon.a (mon.0) 3023 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:49:57.810 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:57.495440+0000 mon.a (mon.0) 3024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-99-cache", "mode": "writeback"}]': finished 2024-02-13T00:49:57.810 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:57.495508+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e583: 8 total, 8 up, 4 in 2024-02-13T00:49:57.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:57.497282+0000 mon.c (mon.2) 458 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:49:57.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:57.497556+0000 mon.a (mon.0) 3026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:49:57.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:57.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:57.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.897+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1489344261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b00f8490 msgr2=0x7ff5b0104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:57.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.897+0000 7ff5b6af1700 1 --2- 172.21.15.162:0/1489344261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b00f8490 0x7ff5b0104a90 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 rx=0x7ff5a8009be0 tx=0x7ff5a8004d10).stop 2024-02-13T00:49:57.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.897+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1489344261 shutdown_connections 2024-02-13T00:49:57.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.897+0000 7ff5b6af1700 1 --2- 172.21.15.162:0/1489344261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b00f8490 0x7ff5b0104a90 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:57.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.897+0000 7ff5b6af1700 1 --2- 172.21.15.162:0/1489344261 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5b00fdc20 0x7ff5b00fe000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:57.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.897+0000 7ff5b6af1700 1 --2- 172.21.15.162:0/1489344261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5b00fe580 0x7ff5b00f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:57.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.897+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1489344261 >> 172.21.15.162:0/1489344261 conn(0x7ff5b00f3850 msgr2=0x7ff5b00f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:57.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.897+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1489344261 shutdown_connections 2024-02-13T00:49:57.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.897+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1489344261 wait complete. 2024-02-13T00:49:57.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b6af1700 1 librados: starting msgr at 2024-02-13T00:49:57.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b6af1700 1 librados: starting objecter 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b6af1700 1 Processor -- start 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b6af1700 1 -- start start 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b6af1700 1 librados: setting wanted keys 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b6af1700 1 librados: calling monclient init 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b6af1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5b00f8490 0x7ff5b006b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b6af1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5b00fdc20 0x7ff5b006b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b6af1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b00fe580 0x7ff5b006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b6af1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b01164b0 con 0x7ff5b00fe580 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b6af1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b0116620 con 0x7ff5b00fdc20 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b6af1700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b01167e0 con 0x7ff5b00f8490 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b488d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5b00f8490 0x7ff5b006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b488d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5b00f8490 0x7ff5b006b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:45536/0 (socket says 172.21.15.162:45536) 2024-02-13T00:49:57.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b508e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b00fe580 0x7ff5b006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b488d700 1 -- 172.21.15.162:0/1366176352 learned_addr learned my addr 172.21.15.162:0/1366176352 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b488d700 1 -- 172.21.15.162:0/1366176352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5b00fdc20 msgr2=0x7ff5b006b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5affff700 1 --2- 172.21.15.162:0/1366176352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5b00fdc20 0x7ff5b006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.898+0000 7ff5b488d700 1 --2- 172.21.15.162:0/1366176352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5b00fdc20 0x7ff5b006b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5b488d700 1 -- 172.21.15.162:0/1366176352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b00fe580 msgr2=0x7ff5b006bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5b488d700 1 --2- 172.21.15.162:0/1366176352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b00fe580 0x7ff5b006bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5b488d700 1 -- 172.21.15.162:0/1366176352 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a4005040 con 0x7ff5b00f8490 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5b508e700 1 --2- 172.21.15.162:0/1366176352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b00fe580 0x7ff5b006bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5b488d700 1 --2- 172.21.15.162:0/1366176352 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5b00f8490 0x7ff5b006b170 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7ff5a40048f0 tx=0x7ff5a4009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5adffb700 10 client.?.objecter ms_handle_connect 0x7ff5b00f8490 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5adffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5adffb700 1 -- 172.21.15.162:0/1366176352 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4007b00 con 0x7ff5b00f8490 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1366176352 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a80097c0 con 0x7ff5b00f8490 2024-02-13T00:49:57.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5adffb700 1 -- 172.21.15.162:0/1366176352 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5a400f040 con 0x7ff5b00f8490 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5adffb700 1 -- 172.21.15.162:0/1366176352 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4014a00 con 0x7ff5b00f8490 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5b6af1700 10 client.37829.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1366176352 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b0116e20 con 0x7ff5b00f8490 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.899+0000 7ff5b6af1700 1 librados: init done 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.900+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1366176352 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b00607b0 con 0x7ff5b00f8490 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.900+0000 7ff5adffb700 1 -- 172.21.15.162:0/1366176352 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff5a400e580 con 0x7ff5b00f8490 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.900+0000 7ff5adffb700 1 --2- 172.21.15.162:0/1366176352 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff5a0062c90 0x7ff5a0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.900+0000 7ff5adffb700 1 -- 172.21.15.162:0/1366176352 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(583..583 src has 449..583) v4 ==== 9295+0+0 (secure 0 0 0) 0x7ff5a407d250 con 0x7ff5b00f8490 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.900+0000 7ff5adffb700 10 client.37829.objecter ms_dispatch 0x7ff5b0001550 osd_map(583..583 src has 449..583) v4 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.900+0000 7ff5adffb700 3 client.37829.objecter handle_osd_map got epochs [583,583] > 0 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.900+0000 7ff5adffb700 3 client.37829.objecter handle_osd_map decoding full epoch 583 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.900+0000 7ff5affff700 1 --2- 172.21.15.162:0/1366176352 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff5a0062c90 0x7ff5a0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.901+0000 7ff5adffb700 20 client.37829.objecter dump_active .. 0 homeless 2024-02-13T00:49:57.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.901+0000 7ff5adffb700 10 client.37829.objecter ms_handle_connect 0x7ff5a0062c90 2024-02-13T00:49:57.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.901+0000 7ff5affff700 1 --2- 172.21.15.162:0/1366176352 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff5a0062c90 0x7ff5a0065150 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 rx=0x7ff5b006c910 tx=0x7ff59c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:57.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:57.905+0000 7ff5adffb700 1 -- 172.21.15.162:0/1366176352 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff5a4049f60 con 0x7ff5b00f8490 2024-02-13T00:49:57.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:49:57.946 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:58.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.078+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1366176352 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff5b006c6e0 con 0x7ff5a0062c90 2024-02-13T00:49:58.097 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.092+0000 7ff5adffb700 1 -- 172.21.15.162:0/1366176352 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525721 (secure 0 0 0) 0x7ff5b006c6e0 con 0x7ff5a0062c90 2024-02-13T00:49:58.103 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:49:58.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.096+0000 7ff5b6af1700 10 librados: watch_flush enter 2024-02-13T00:49:58.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.096+0000 7ff5b6af1700 10 librados: watch_flush exit 2024-02-13T00:49:58.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 20 client.37829.objecter shutdown clearing up homeless session... 2024-02-13T00:49:58.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 10 client.37829.objecter successfully canceled tick 2024-02-13T00:49:58.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1366176352 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff5a0062c90 msgr2=0x7ff5a0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:58.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 --2- 172.21.15.162:0/1366176352 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff5a0062c90 0x7ff5a0065150 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 rx=0x7ff5b006c910 tx=0x7ff59c00b040).stop 2024-02-13T00:49:58.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1366176352 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5b00f8490 msgr2=0x7ff5b006b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:58.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 --2- 172.21.15.162:0/1366176352 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5b00f8490 0x7ff5b006b170 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7ff5a40048f0 tx=0x7ff5a4009cb0).stop 2024-02-13T00:49:58.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1366176352 shutdown_connections 2024-02-13T00:49:58.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 --2- 172.21.15.162:0/1366176352 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff5a0062c90 0x7ff5a0065150 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:58.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 --2- 172.21.15.162:0/1366176352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5b00fe580 0x7ff5b006bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:58.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 --2- 172.21.15.162:0/1366176352 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5b00f8490 0x7ff5b006b170 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:58.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 --2- 172.21.15.162:0/1366176352 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5b00fdc20 0x7ff5b006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:58.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1366176352 >> 172.21.15.162:0/1366176352 conn(0x7ff5b00f3850 msgr2=0x7ff5b00f5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:58.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1366176352 shutdown_connections 2024-02-13T00:49:58.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 -- 172.21.15.162:0/1366176352 wait complete. 2024-02-13T00:49:58.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:58.097+0000 7ff5b6af1700 1 librados: shutdown 2024-02-13T00:49:58.110 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:49:58.396 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:49:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:58.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:49:58.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:49:58.529 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:58.551 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:49:58.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:58.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:49:58.618 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:49:58.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:49:58.681 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:58.080624+0000 mgr.y (mgr.14544) 501 : audit [DBG] from='client.37829 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:49:58.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:58.497948+0000 mon.a (mon.0) 3027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:49:58.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:58.498025+0000 mon.a (mon.0) 3028 : cluster [DBG] osdmap e584: 8 total, 8 up, 4 in 2024-02-13T00:49:58.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:58.500583+0000 mon.c (mon.2) 459 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:58.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:58.500919+0000 mon.a (mon.0) 3029 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:58.682 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:58.080624+0000 mgr.y (mgr.14544) 501 : audit [DBG] from='client.37829 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:49:58.682 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:58.497948+0000 mon.a (mon.0) 3027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:49:58.682 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:58.498025+0000 mon.a (mon.0) 3028 : cluster [DBG] osdmap e584: 8 total, 8 up, 4 in 2024-02-13T00:49:58.682 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:58.500583+0000 mon.c (mon.2) 459 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:58.682 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:58.500919+0000 mon.a (mon.0) 3029 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:58.683 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:58.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:49:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:49:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:49:58.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:58.080624+0000 mgr.y (mgr.14544) 501 : audit [DBG] from='client.37829 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:49:58.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:58.497948+0000 mon.a (mon.0) 3027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:49:58.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:58.498025+0000 mon.a (mon.0) 3028 : cluster [DBG] osdmap e584: 8 total, 8 up, 4 in 2024-02-13T00:49:58.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:58.500583+0000 mon.c (mon.2) 459 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:58.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:58.500919+0000 mon.a (mon.0) 3029 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:49:58.882 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:58.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:58.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:49:58.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:49:59.019 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:49:59.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.276+0000 7faf26a16700 1 -- 172.21.15.162:0/262458936 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf200f8d10 msgr2=0x7faf200f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:59.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.276+0000 7faf26a16700 1 --2- 172.21.15.162:0/262458936 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf200f8d10 0x7faf200f90f0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7faf10004870 tx=0x7faf10009cb0).stop 2024-02-13T00:49:59.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.276+0000 7faf26a16700 1 -- 172.21.15.162:0/262458936 shutdown_connections 2024-02-13T00:49:59.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.276+0000 7faf26a16700 1 --2- 172.21.15.162:0/262458936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf20102270 0x7faf20104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:59.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.276+0000 7faf26a16700 1 --2- 172.21.15.162:0/262458936 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf200f8d10 0x7faf200f90f0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:59.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.276+0000 7faf26a16700 1 --2- 172.21.15.162:0/262458936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf200f95e0 0x7faf20101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:59.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.276+0000 7faf26a16700 1 -- 172.21.15.162:0/262458936 >> 172.21.15.162:0/262458936 conn(0x7faf2006eb90 msgr2=0x7faf200f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:59.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.276+0000 7faf26a16700 1 -- 172.21.15.162:0/262458936 shutdown_connections 2024-02-13T00:49:59.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.276+0000 7faf26a16700 1 -- 172.21.15.162:0/262458936 wait complete. 2024-02-13T00:49:59.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf26a16700 1 librados: starting msgr at 2024-02-13T00:49:59.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf26a16700 1 librados: starting objecter 2024-02-13T00:49:59.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf26a16700 1 Processor -- start 2024-02-13T00:49:59.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf26a16700 1 -- start start 2024-02-13T00:49:59.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf26a16700 1 librados: setting wanted keys 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf26a16700 1 librados: calling monclient init 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf26a16700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf200f8d10 0x7faf200f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf26a16700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf200f95e0 0x7faf200f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf26a16700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf20102270 0x7faf200f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf26a16700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf20116720 con 0x7faf20102270 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf26a16700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf20116890 con 0x7faf200f95e0 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf26a16700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf20116a50 con 0x7faf200f8d10 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.277+0000 7faf1ffff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf200f8d10 0x7faf200f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf24fb3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf20102270 0x7faf200f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1ffff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf200f8d10 0x7faf200f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:45554/0 (socket says 172.21.15.162:45554) 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1ffff700 1 -- 172.21.15.162:0/2298352761 learned_addr learned my addr 172.21.15.162:0/2298352761 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1f7fe700 1 --2- 172.21.15.162:0/2298352761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf200f95e0 0x7faf200f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:59.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1ffff700 1 -- 172.21.15.162:0/2298352761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf200f95e0 msgr2=0x7faf200f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:59.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1ffff700 1 --2- 172.21.15.162:0/2298352761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf200f95e0 0x7faf200f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:59.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1ffff700 1 -- 172.21.15.162:0/2298352761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf20102270 msgr2=0x7faf200f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:59.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1ffff700 1 --2- 172.21.15.162:0/2298352761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf20102270 0x7faf200f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:59.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1ffff700 1 -- 172.21.15.162:0/2298352761 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf14005040 con 0x7faf200f8d10 2024-02-13T00:49:59.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf24fb3700 1 --2- 172.21.15.162:0/2298352761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf20102270 0x7faf200f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:49:59.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1ffff700 1 --2- 172.21.15.162:0/2298352761 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf200f8d10 0x7faf200f6cc0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 rx=0x7faf10009eb0 tx=0x7faf1000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:59.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1d7fa700 10 client.?.objecter ms_handle_connect 0x7faf200f8d10 2024-02-13T00:49:59.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:49:59.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1d7fa700 1 -- 172.21.15.162:0/2298352761 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1000e070 con 0x7faf200f8d10 2024-02-13T00:49:59.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf26a16700 1 -- 172.21.15.162:0/2298352761 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf10005040 con 0x7faf200f8d10 2024-02-13T00:49:59.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1d7fa700 1 -- 172.21.15.162:0/2298352761 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf10013da0 con 0x7faf200f8d10 2024-02-13T00:49:59.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf1d7fa700 1 -- 172.21.15.162:0/2298352761 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf10019070 con 0x7faf200f8d10 2024-02-13T00:49:59.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf26a16700 10 client.37835.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:49:59.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf26a16700 1 -- 172.21.15.162:0/2298352761 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf20116f40 con 0x7faf200f8d10 2024-02-13T00:49:59.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.278+0000 7faf26a16700 1 librados: init done 2024-02-13T00:49:59.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.279+0000 7faf26a16700 1 -- 172.21.15.162:0/2298352761 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf20107170 con 0x7faf200f8d10 2024-02-13T00:49:59.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.279+0000 7faf1d7fa700 1 -- 172.21.15.162:0/2298352761 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7faf1001e420 con 0x7faf200f8d10 2024-02-13T00:49:59.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.280+0000 7faf1d7fa700 1 --2- 172.21.15.162:0/2298352761 >> v2:172.21.15.162:6800/2866534053 conn(0x7faf08062c90 0x7faf08065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:49:59.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.280+0000 7faf1f7fe700 1 --2- 172.21.15.162:0/2298352761 >> v2:172.21.15.162:6800/2866534053 conn(0x7faf08062c90 0x7faf08065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:49:59.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.280+0000 7faf1d7fa700 1 -- 172.21.15.162:0/2298352761 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(584..584 src has 449..584) v4 ==== 9295+0+0 (secure 0 0 0) 0x7faf1007e190 con 0x7faf200f8d10 2024-02-13T00:49:59.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.280+0000 7faf1d7fa700 10 client.37835.objecter ms_dispatch 0x7faf20001550 osd_map(584..584 src has 449..584) v4 2024-02-13T00:49:59.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.280+0000 7faf1d7fa700 3 client.37835.objecter handle_osd_map got epochs [584,584] > 0 2024-02-13T00:49:59.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.280+0000 7faf1d7fa700 3 client.37835.objecter handle_osd_map decoding full epoch 584 2024-02-13T00:49:59.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.280+0000 7faf1d7fa700 20 client.37835.objecter dump_active .. 0 homeless 2024-02-13T00:49:59.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.280+0000 7faf1d7fa700 10 client.37835.objecter ms_handle_connect 0x7faf08062c90 2024-02-13T00:49:59.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.280+0000 7faf1f7fe700 1 --2- 172.21.15.162:0/2298352761 >> v2:172.21.15.162:6800/2866534053 conn(0x7faf08062c90 0x7faf08065150 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7faf200f8460 tx=0x7faf14009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:49:59.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.284+0000 7faf1d7fa700 1 -- 172.21.15.162:0/2298352761 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7faf10051270 con 0x7faf200f8d10 2024-02-13T00:49:59.469 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.464+0000 7faf26a16700 1 -- 172.21.15.162:0/2298352761 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7faf2005fb80 con 0x7faf200f8d10 2024-02-13T00:49:59.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.497+0000 7faf1d7fa700 1 -- 172.21.15.162:0/2298352761 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v585) v1 ==== 97+0+0 (secure 0 0 0) 0x7faf10023030 con 0x7faf200f8d10 2024-02-13T00:49:59.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 10 librados: watch_flush enter 2024-02-13T00:49:59.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 10 librados: watch_flush exit 2024-02-13T00:49:59.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 20 client.37835.objecter shutdown clearing up homeless session... 2024-02-13T00:49:59.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 10 client.37835.objecter successfully canceled tick 2024-02-13T00:49:59.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 -- 172.21.15.162:0/2298352761 >> v2:172.21.15.162:6800/2866534053 conn(0x7faf08062c90 msgr2=0x7faf08065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:59.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 --2- 172.21.15.162:0/2298352761 >> v2:172.21.15.162:6800/2866534053 conn(0x7faf08062c90 0x7faf08065150 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7faf200f8460 tx=0x7faf14009cb0).stop 2024-02-13T00:49:59.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 -- 172.21.15.162:0/2298352761 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf200f8d10 msgr2=0x7faf200f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:49:59.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 --2- 172.21.15.162:0/2298352761 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf200f8d10 0x7faf200f6cc0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 rx=0x7faf10009eb0 tx=0x7faf1000b040).stop 2024-02-13T00:49:59.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 -- 172.21.15.162:0/2298352761 shutdown_connections 2024-02-13T00:49:59.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 --2- 172.21.15.162:0/2298352761 >> v2:172.21.15.162:6800/2866534053 conn(0x7faf08062c90 0x7faf08065150 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:59.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 --2- 172.21.15.162:0/2298352761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf20102270 0x7faf200f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:59.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 --2- 172.21.15.162:0/2298352761 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf200f8d10 0x7faf200f6cc0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:59.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 --2- 172.21.15.162:0/2298352761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf200f95e0 0x7faf200f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:49:59.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 -- 172.21.15.162:0/2298352761 >> 172.21.15.162:0/2298352761 conn(0x7faf2006eb90 msgr2=0x7faf20102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:49:59.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 -- 172.21.15.162:0/2298352761 shutdown_connections 2024-02-13T00:49:59.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 -- 172.21.15.162:0/2298352761 wait complete. 2024-02-13T00:49:59.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:49:59.500+0000 7faf26a16700 1 librados: shutdown 2024-02-13T00:49:59.506 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:49:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:58.483296+0000 mgr.y (mgr.14544) 502 : cluster [DBG] pgmap v875: 212 pgs: 20 creating+activating, 192 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:59.466996+0000 mon.c (mon.2) 460 : audit [INF] from='client.? 172.21.15.162:0/2298352761' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:59.467309+0000 mon.a (mon.0) 3030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:59.494395+0000 mon.a (mon.0) 3031 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:59.498516+0000 mon.a (mon.0) 3032 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:49:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:59.498595+0000 mon.a (mon.0) 3033 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:49:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:49:59.498623+0000 mon.a (mon.0) 3034 : cluster [DBG] osdmap e585: 8 total, 8 up, 4 in 2024-02-13T00:49:59.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:59.500023+0000 mon.c (mon.2) 461 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:49:59.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:49:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:49:59.500751+0000 mon.a (mon.0) 3035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:49:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:58.483296+0000 mgr.y (mgr.14544) 502 : cluster [DBG] pgmap v875: 212 pgs: 20 creating+activating, 192 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:59.466996+0000 mon.c (mon.2) 460 : audit [INF] from='client.? 172.21.15.162:0/2298352761' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:59.467309+0000 mon.a (mon.0) 3030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:59.494395+0000 mon.a (mon.0) 3031 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:59.498516+0000 mon.a (mon.0) 3032 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:49:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:59.498595+0000 mon.a (mon.0) 3033 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:49:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:49:59.498623+0000 mon.a (mon.0) 3034 : cluster [DBG] osdmap e585: 8 total, 8 up, 4 in 2024-02-13T00:49:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:59.500023+0000 mon.c (mon.2) 461 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:49:59.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:49:59.500751+0000 mon.a (mon.0) 3035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:49:59.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:58.483296+0000 mgr.y (mgr.14544) 502 : cluster [DBG] pgmap v875: 212 pgs: 20 creating+activating, 192 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:49:59.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:59.466996+0000 mon.c (mon.2) 460 : audit [INF] from='client.? 172.21.15.162:0/2298352761' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:59.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:59.467309+0000 mon.a (mon.0) 3030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:49:59.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:59.494395+0000 mon.a (mon.0) 3031 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:49:59.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:59.498516+0000 mon.a (mon.0) 3032 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:49:59.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:59.498595+0000 mon.a (mon.0) 3033 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:49:59.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:49:59.498623+0000 mon.a (mon.0) 3034 : cluster [DBG] osdmap e585: 8 total, 8 up, 4 in 2024-02-13T00:49:59.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:59.500023+0000 mon.c (mon.2) 461 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:49:59.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:49:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:49:59.500751+0000 mon.a (mon.0) 3035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:49:59.969 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:49:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:49:59.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:00.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:00.106 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:00.128 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:00.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:00.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:00.231 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:00.263 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:00.285 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:00.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:00.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:00.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:00.417 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:00.436 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:00.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:00.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:00.538 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":874,"stamp":"2024-02-13T00:49:57.492960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":2226,"num_read_kb":1944,"num_write":914,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":13678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1048,"ondisk_log_size":1048,"up":636,"acting":636,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":520,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1216024,"kb_used_data":10744,"kb_used_omap":0,"kb_used_meta":1205248,"kb_avail":373764584,"statfs":{"total":383980142592,"available":382734934016,"internally_reserved":0,"allocated":11001856,"data_stored":5010999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1234173952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.969294"},"pg_stats":[{"pgid":"233.1f","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.531562+0000","last_change":"2024-02-13T00:49:54.531562+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.531562+0000","last_undegraded":"2024-02-13T00:49:54.531562+0000","last_fullsized":"2024-02-13T00:49:54.531562+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["0","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.1e","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.531540+0000","last_change":"2024-02-13T00:49:54.531540+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.531540+0000","last_undegraded":"2024-02-13T00:49:54.531540+0000","last_fullsized":"2024-02-13T00:49:54.531540+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":["0","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.1d","version":"0'0","reported_seq":9,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.543530+0000","last_change":"2024-02-13T00:49:54.543530+0000","last_active":"2024-02-13T00:49:54.543530+0000","last_peered":"2024-02-13T00:49:54.543530+0000","last_clean":"2024-02-13T00:49:54.543530+0000","last_became_active":"2024-02-13T00:49:54.543203+0000","last_became_peered":"2024-02-13T00:49:54.543203+0000","last_unstale":"2024-02-13T00:49:54.543530+0000","last_undegraded":"2024-02-13T00:49:54.543530+0000","last_fullsized":"2024-02-13T00:49:54.543530+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"233.1c","version":"0'0","reported_seq":9,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.546413+0000","last_change":"2024-02-13T00:49:54.546413+0000","last_active":"2024-02-13T00:49:54.546413+0000","last_peered":"2024-02-13T00:49:54.546413+0000","last_clean":"2024-02-13T00:49:54.546413+0000","last_became_active":"2024-02-13T00:49:54.546229+0000","last_became_peered":"2024-02-13T00:49:54.546229+0000","last_unstale":"2024-02-13T00:49:54.546413+0000","last_undegraded":"2024-02-13T00:49:54.546413+0000","last_fullsized":"2024-02-13T00:49:54.546413+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.1b","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.530810+0000","last_change":"2024-02-13T00:49:54.530810+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.530810+0000","last_undegraded":"2024-02-13T00:49:54.530810+0000","last_fullsized":"2024-02-13T00:49:54.530810+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["0","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.1a","version":"0'0","reported_seq":9,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.543339+0000","last_change":"2024-02-13T00:49:54.543339+0000","last_active":"2024-02-13T00:49:54.543339+0000","last_peered":"2024-02-13T00:49:54.543339+0000","last_clean":"2024-02-13T00:49:54.543339+0000","last_became_active":"2024-02-13T00:49:54.543069+0000","last_became_peered":"2024-02-13T00:49:54.543069+0000","last_unstale":"2024-02-13T00:49:54.543339+0000","last_undegraded":"2024-02-13T00:49:54.543339+0000","last_fullsized":"2024-02-13T00:49:54.543339+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.19","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.536262+0000","last_change":"2024-02-13T00:49:54.536262+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.536262+0000","last_undegraded":"2024-02-13T00:49:54.536262+0000","last_fullsized":"2024-02-13T00:49:54.536262+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":["1","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"233.18","version":"0'0","reported_seq":9,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.543549+0000","last_change":"2024-02-13T00:49:54.543549+0000","last_active":"2024-02-13T00:49:54.543549+0000","last_peered":"2024-02-13T00:49:54.543549+0000","last_clean":"2024-02-13T00:49:54.543549+0000","last_became_active":"2024-02-13T00:49:54.543246+0000","last_became_peered":"2024-02-13T00:49:54.543246+0000","last_unstale":"2024-02-13T00:49:54.543549+0000","last_undegraded":"2024-02-13T00:49:54.543549+0000","last_fullsized":"2024-02-13T00:49:54.543549+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"233.17","version":"0'0","reported_seq":9,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.545174+0000","last_change":"2024-02-13T00:49:54.545174+0000","last_active":"2024-02-13T00:49:54.545174+0000","last_peered":"2024-02-13T00:49:54.545174+0000","last_clean":"2024-02-13T00:49:54.545174+0000","last_became_active":"2024-02-13T00:49:54.544275+0000","last_became_peered":"2024-02-13T00:49:54.544275+0000","last_unstale":"2024-02-13T00:49:54.545174+0000","last_undegraded":"2024-02-13T00:49:54.545174+0000","last_fullsized":"2024-02-13T00:49:54.545174+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"233.16","version":"0'0","reported_seq":9,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.543835+0000","last_change":"2024-02-13T00:49:54.543835+0000","last_active":"2024-02-13T00:49:54.543835+0000","last_peered":"2024-02-13T00:49:54.543835+0000","last_clean":"2024-02-13T00:49:54.543835+0000","last_became_active":"2024-02-13T00:49:54.543538+0000","last_became_peered":"2024-02-13T00:49:54.543538+0000","last_unstale":"2024-02-13T00:49:54.543835+0000","last_undegraded":"2024-02-13T00:49:54.543835+0000","last_fullsized":"2024-02-13T00:49:54.543835+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"233.15","version":"0'0","reported_seq":9,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.542464+0000","last_change":"2024-02-13T00:49:54.542464+0000","last_active":"2024-02-13T00:49:54.542464+0000","last_peered":"2024-02-13T00:49:54.542464+0000","last_clean":"2024-02-13T00:49:54.542464+0000","last_became_active":"2024-02-13T00:49:54.542246+0000","last_became_peered":"2024-02-13T00:49:54.542246+0000","last_unstale":"2024-02-13T00:49:54.542464+0000","last_undegraded":"2024-02-13T00:49:54.542464+0000","last_fullsized":"2024-02-13T00:49:54.542464+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.14","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.539458+0000","last_change":"2024-02-13T00:49:54.539458+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.539458+0000","last_undegraded":"2024-02-13T00:49:54.539458+0000","last_fullsized":"2024-02-13T00:49:54.539458+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":["1","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"233.13","version":"0'0","reported_seq":9,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.542457+0000","last_change":"2024-02-13T00:49:54.542457+0000","last_active":"2024-02-13T00:49:54.542457+0000","last_peered":"2024-02-13T00:49:54.542457+0000","last_clean":"2024-02-13T00:49:54.542457+0000","last_became_active":"2024-02-13T00:49:54.542155+0000","last_became_peered":"2024-02-13T00:49:54.542155+0000","last_unstale":"2024-02-13T00:49:54.542457+0000","last_undegraded":"2024-02-13T00:49:54.542457+0000","last_fullsized":"2024-02-13T00:49:54.542457+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.12","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.535277+0000","last_change":"2024-02-13T00:49:54.535277+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.535277+0000","last_undegraded":"2024-02-13T00:49:54.535277+0000","last_fullsized":"2024-02-13T00:49:54.535277+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":["6","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"233.11","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.536592+0000","last_change":"2024-02-13T00:49:54.536592+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.536592+0000","last_undegraded":"2024-02-13T00:49:54.536592+0000","last_fullsized":"2024-02-13T00:49:54.536592+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"233.10","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.534200+0000","last_change":"2024-02-13T00:49:54.534200+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.534200+0000","last_undegraded":"2024-02-13T00:49:54.534200+0000","last_fullsized":"2024-02-13T00:49:54.534200+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":["6","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"233.6","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.536691+0000","last_change":"2024-02-13T00:49:54.536691+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.536691+0000","last_undegraded":"2024-02-13T00:49:54.536691+0000","last_fullsized":"2024-02-13T00:49:54.536691+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"232.7","version":"0'0","reported_seq":10,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.538195+0000","last_change":"2024-02-13T00:49:53.537094+0000","last_active":"2024-02-13T00:49:53.538195+0000","last_peered":"2024-02-13T00:49:53.538195+0000","last_clean":"2024-02-13T00:49:53.538195+0000","last_became_active":"2024-02-13T00:49:53.536833+0000","last_became_peered":"2024-02-13T00:49:53.536833+0000","last_unstale":"2024-02-13T00:49:53.538195+0000","last_undegraded":"2024-02-13T00:49:53.538195+0000","last_fullsized":"2024-02-13T00:49:53.538195+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":578,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_clean_scrub_stamp":"2024-02-13T00:49:52.509490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.7","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.537995+0000","last_change":"2024-02-13T00:49:54.537995+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.537995+0000","last_undegraded":"2024-02-13T00:49:54.537995+0000","last_fullsized":"2024-02-13T00:49:54.537995+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":["1","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"232.6","version":"0'0","reported_seq":10,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.537435+0000","last_change":"2024-02-13T00:49:53.535731+0000","last_active":"2024-02-13T00:49:53.537435+0000","last_peered":"2024-02-13T00:49:53.537435+0000","last_clean":"2024-02-13T00:49:53.537435+0000","last_became_active":"2024-02-13T00:49:53.535467+0000","last_became_peered":"2024-02-13T00:49:53.535467+0000","last_unstale":"2024-02-13T00:49:53.537435+0000","last_undegraded":"2024-02-13T00:49:53.537435+0000","last_fullsized":"2024-02-13T00:49:53.537435+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":578,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_clean_scrub_stamp":"2024-02-13T00:49:52.509490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.4","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.539528+0000","last_change":"2024-02-13T00:49:54.539528+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.539528+0000","last_undegraded":"2024-02-13T00:49:54.539528+0000","last_fullsized":"2024-02-13T00:49:54.539528+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"232.5","version":"0'0","reported_seq":10,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.533452+0000","last_change":"2024-02-13T00:49:53.533266+0000","last_active":"2024-02-13T00:49:53.533452+0000","last_peered":"2024-02-13T00:49:53.533452+0000","last_clean":"2024-02-13T00:49:53.533452+0000","last_became_active":"2024-02-13T00:49:53.533015+0000","last_became_peered":"2024-02-13T00:49:53.533015+0000","last_unstale":"2024-02-13T00:49:53.533452+0000","last_undegraded":"2024-02-13T00:49:53.533452+0000","last_fullsized":"2024-02-13T00:49:53.533452+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":578,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_clean_scrub_stamp":"2024-02-13T00:49:52.509490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"233.5","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.535741+0000","last_change":"2024-02-13T00:49:54.535741+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.535741+0000","last_undegraded":"2024-02-13T00:49:54.535741+0000","last_fullsized":"2024-02-13T00:49:54.535741+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":["6","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"232.4","version":"0'0","reported_seq":10,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.536924+0000","last_change":"2024-02-13T00:49:53.536636+0000","last_active":"2024-02-13T00:49:53.536924+0000","last_peered":"2024-02-13T00:49:53.536924+0000","last_clean":"2024-02-13T00:49:53.536924+0000","last_became_active":"2024-02-13T00:49:53.536379+0000","last_became_peered":"2024-02-13T00:49:53.536379+0000","last_unstale":"2024-02-13T00:49:53.536924+0000","last_undegraded":"2024-02-13T00:49:53.536924+0000","last_fullsized":"2024-02-13T00:49:53.536924+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":578,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_clean_scrub_stamp":"2024-02-13T00:49:52.509490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"233.2","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.537108+0000","last_change":"2024-02-13T00:49:54.537108+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.537108+0000","last_undegraded":"2024-02-13T00:49:54.537108+0000","last_fullsized":"2024-02-13T00:49:54.537108+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":["1","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"232.3","version":"0'0","reported_seq":10,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.532546+0000","last_change":"2024-02-13T00:49:53.532232+0000","last_active":"2024-02-13T00:49:53.532546+0000","last_peered":"2024-02-13T00:49:53.532546+0000","last_clean":"2024-02-13T00:49:53.532546+0000","last_became_active":"2024-02-13T00:49:53.531918+0000","last_became_peered":"2024-02-13T00:49:53.531918+0000","last_unstale":"2024-02-13T00:49:53.532546+0000","last_undegraded":"2024-02-13T00:49:53.532546+0000","last_fullsized":"2024-02-13T00:49:53.532546+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":578,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_clean_scrub_stamp":"2024-02-13T00:49:52.509490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"233.3","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.534707+0000","last_change":"2024-02-13T00:49:54.534707+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.534707+0000","last_undegraded":"2024-02-13T00:49:54.534707+0000","last_fullsized":"2024-02-13T00:49:54.534707+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":["6","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"232.2","version":"0'0","reported_seq":10,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.537895+0000","last_change":"2024-02-13T00:49:53.537592+0000","last_active":"2024-02-13T00:49:53.537895+0000","last_peered":"2024-02-13T00:49:53.537895+0000","last_clean":"2024-02-13T00:49:53.537895+0000","last_became_active":"2024-02-13T00:49:53.536429+0000","last_became_peered":"2024-02-13T00:49:53.536429+0000","last_unstale":"2024-02-13T00:49:53.537895+0000","last_undegraded":"2024-02-13T00:49:53.537895+0000","last_fullsized":"2024-02-13T00:49:53.537895+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":578,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_clean_scrub_stamp":"2024-02-13T00:49:52.509490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"233.0","version":"0'0","reported_seq":9,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.545570+0000","last_change":"2024-02-13T00:49:54.545570+0000","last_active":"2024-02-13T00:49:54.545570+0000","last_peered":"2024-02-13T00:49:54.545570+0000","last_clean":"2024-02-13T00:49:54.545570+0000","last_became_active":"2024-02-13T00:49:54.545289+0000","last_became_peered":"2024-02-13T00:49:54.545289+0000","last_unstale":"2024-02-13T00:49:54.545570+0000","last_undegraded":"2024-02-13T00:49:54.545570+0000","last_fullsized":"2024-02-13T00:49:54.545570+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"232.1","version":"0'0","reported_seq":10,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.537463+0000","last_change":"2024-02-13T00:49:53.535810+0000","last_active":"2024-02-13T00:49:53.537463+0000","last_peered":"2024-02-13T00:49:53.537463+0000","last_clean":"2024-02-13T00:49:53.537463+0000","last_became_active":"2024-02-13T00:49:53.534973+0000","last_became_peered":"2024-02-13T00:49:53.534973+0000","last_unstale":"2024-02-13T00:49:53.537463+0000","last_undegraded":"2024-02-13T00:49:53.537463+0000","last_fullsized":"2024-02-13T00:49:53.537463+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":578,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_clean_scrub_stamp":"2024-02-13T00:49:52.509490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.1","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.531126+0000","last_change":"2024-02-13T00:49:54.531126+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.531126+0000","last_undegraded":"2024-02-13T00:49:54.531126+0000","last_fullsized":"2024-02-13T00:49:54.531126+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":["0","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"232.0","version":"0'0","reported_seq":10,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.533308+0000","last_change":"2024-02-13T00:49:53.533112+0000","last_active":"2024-02-13T00:49:53.533308+0000","last_peered":"2024-02-13T00:49:53.533308+0000","last_clean":"2024-02-13T00:49:53.533308+0000","last_became_active":"2024-02-13T00:49:53.532515+0000","last_became_peered":"2024-02-13T00:49:53.532515+0000","last_unstale":"2024-02-13T00:49:53.533308+0000","last_undegraded":"2024-02-13T00:49:53.533308+0000","last_fullsized":"2024-02-13T00:49:53.533308+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":578,"last_epoch_clean":579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:52.509490+0000","last_clean_scrub_stamp":"2024-02-13T00:49:52.509490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"233.b","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.536870+0000","last_change":"2024-02-13T00:49:54.536870+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.536870+0000","last_undegraded":"2024-02-13T00:49:54.536870+0000","last_fullsized":"2024-02-13T00:49:54.536870+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":["1","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"229.7","version":"0'0","reported_seq":24,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528788+0000","last_change":"2024-02-13T00:49:38.508094+0000","last_active":"2024-02-13T00:49:53.528788+0000","last_peered":"2024-02-13T00:49:53.528788+0000","last_clean":"2024-02-13T00:49:53.528788+0000","last_became_active":"2024-02-13T00:49:38.507750+0000","last_became_peered":"2024-02-13T00:49:38.507750+0000","last_unstale":"2024-02-13T00:49:53.528788+0000","last_undegraded":"2024-02-13T00:49:53.528788+0000","last_fullsized":"2024-02-13T00:49:53.528788+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"233.a","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.537586+0000","last_change":"2024-02-13T00:49:54.537586+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.537586+0000","last_undegraded":"2024-02-13T00:49:54.537586+0000","last_fullsized":"2024-02-13T00:49:54.537586+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"229.6","version":"0'0","reported_seq":25,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.539974+0000","last_change":"2024-02-13T00:49:38.569413+0000","last_active":"2024-02-13T00:49:53.539974+0000","last_peered":"2024-02-13T00:49:53.539974+0000","last_clean":"2024-02-13T00:49:53.539974+0000","last_became_active":"2024-02-13T00:49:38.569040+0000","last_became_peered":"2024-02-13T00:49:38.569040+0000","last_unstale":"2024-02-13T00:49:53.539974+0000","last_undegraded":"2024-02-13T00:49:53.539974+0000","last_fullsized":"2024-02-13T00:49:53.539974+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.9","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.533735+0000","last_change":"2024-02-13T00:49:54.533735+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.533735+0000","last_undegraded":"2024-02-13T00:49:54.533735+0000","last_fullsized":"2024-02-13T00:49:54.533735+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":["6","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"229.5","version":"564'4","reported_seq":31,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.539885+0000","last_change":"2024-02-13T00:49:38.570168+0000","last_active":"2024-02-13T00:49:53.539885+0000","last_peered":"2024-02-13T00:49:53.539885+0000","last_clean":"2024-02-13T00:49:53.539885+0000","last_became_active":"2024-02-13T00:49:38.569352+0000","last_became_peered":"2024-02-13T00:49:38.569352+0000","last_unstale":"2024-02-13T00:49:53.539885+0000","last_undegraded":"2024-02-13T00:49:53.539885+0000","last_fullsized":"2024-02-13T00:49:53.539885+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.8","version":"0'0","reported_seq":9,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.543062+0000","last_change":"2024-02-13T00:49:54.543062+0000","last_active":"2024-02-13T00:49:54.543062+0000","last_peered":"2024-02-13T00:49:54.543062+0000","last_clean":"2024-02-13T00:49:54.543062+0000","last_became_active":"2024-02-13T00:49:54.542797+0000","last_became_peered":"2024-02-13T00:49:54.542797+0000","last_unstale":"2024-02-13T00:49:54.543062+0000","last_undegraded":"2024-02-13T00:49:54.543062+0000","last_fullsized":"2024-02-13T00:49:54.543062+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"229.4","version":"0'0","reported_seq":24,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528509+0000","last_change":"2024-02-13T00:49:38.507477+0000","last_active":"2024-02-13T00:49:53.528509+0000","last_peered":"2024-02-13T00:49:53.528509+0000","last_clean":"2024-02-13T00:49:53.528509+0000","last_became_active":"2024-02-13T00:49:38.507129+0000","last_became_peered":"2024-02-13T00:49:38.507129+0000","last_unstale":"2024-02-13T00:49:53.528509+0000","last_undegraded":"2024-02-13T00:49:53.528509+0000","last_fullsized":"2024-02-13T00:49:53.528509+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"233.f","version":"0'0","reported_seq":9,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.543122+0000","last_change":"2024-02-13T00:49:54.543122+0000","last_active":"2024-02-13T00:49:54.543122+0000","last_peered":"2024-02-13T00:49:54.543122+0000","last_clean":"2024-02-13T00:49:54.543122+0000","last_became_active":"2024-02-13T00:49:54.542864+0000","last_became_peered":"2024-02-13T00:49:54.542864+0000","last_unstale":"2024-02-13T00:49:54.543122+0000","last_undegraded":"2024-02-13T00:49:54.543122+0000","last_fullsized":"2024-02-13T00:49:54.543122+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"229.3","version":"0'0","reported_seq":24,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527822+0000","last_change":"2024-02-13T00:49:38.570001+0000","last_active":"2024-02-13T00:49:53.527822+0000","last_peered":"2024-02-13T00:49:53.527822+0000","last_clean":"2024-02-13T00:49:53.527822+0000","last_became_active":"2024-02-13T00:49:38.569665+0000","last_became_peered":"2024-02-13T00:49:38.569665+0000","last_unstale":"2024-02-13T00:49:53.527822+0000","last_undegraded":"2024-02-13T00:49:53.527822+0000","last_fullsized":"2024-02-13T00:49:53.527822+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"233.e","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.536409+0000","last_change":"2024-02-13T00:49:54.536409+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.536409+0000","last_undegraded":"2024-02-13T00:49:54.536409+0000","last_fullsized":"2024-02-13T00:49:54.536409+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":["6","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"229.2","version":"0'0","reported_seq":24,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527237+0000","last_change":"2024-02-13T00:49:38.569965+0000","last_active":"2024-02-13T00:49:53.527237+0000","last_peered":"2024-02-13T00:49:53.527237+0000","last_clean":"2024-02-13T00:49:53.527237+0000","last_became_active":"2024-02-13T00:49:38.569583+0000","last_became_peered":"2024-02-13T00:49:38.569583+0000","last_unstale":"2024-02-13T00:49:53.527237+0000","last_undegraded":"2024-02-13T00:49:53.527237+0000","last_fullsized":"2024-02-13T00:49:53.527237+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"233.d","version":"0'0","reported_seq":9,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.544678+0000","last_change":"2024-02-13T00:49:54.544678+0000","last_active":"2024-02-13T00:49:54.544678+0000","last_peered":"2024-02-13T00:49:54.544678+0000","last_clean":"2024-02-13T00:49:54.544678+0000","last_became_active":"2024-02-13T00:49:54.544290+0000","last_became_peered":"2024-02-13T00:49:54.544290+0000","last_unstale":"2024-02-13T00:49:54.544678+0000","last_undegraded":"2024-02-13T00:49:54.544678+0000","last_fullsized":"2024-02-13T00:49:54.544678+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"229.1","version":"0'0","reported_seq":25,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535227+0000","last_change":"2024-02-13T00:49:38.508927+0000","last_active":"2024-02-13T00:49:53.535227+0000","last_peered":"2024-02-13T00:49:53.535227+0000","last_clean":"2024-02-13T00:49:53.535227+0000","last_became_active":"2024-02-13T00:49:38.508528+0000","last_became_peered":"2024-02-13T00:49:38.508528+0000","last_unstale":"2024-02-13T00:49:53.535227+0000","last_undegraded":"2024-02-13T00:49:53.535227+0000","last_fullsized":"2024-02-13T00:49:53.535227+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"233.c","version":"0'0","reported_seq":3,"reported_epoch":580,"state":"creating+activating","last_fresh":"2024-02-13T00:49:54.534548+0000","last_change":"2024-02-13T00:49:54.534548+0000","last_active":"2024-02-13T00:49:53.517154+0000","last_peered":"2024-02-13T00:49:53.517154+0000","last_clean":"2024-02-13T00:49:53.517154+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:49:54.534548+0000","last_undegraded":"2024-02-13T00:49:54.534548+0000","last_fullsized":"2024-02-13T00:49:54.534548+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:53.517154+0000","last_clean_scrub_stamp":"2024-02-13T00:49:53.517154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":["6","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"229.0","version":"0'0","reported_seq":25,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.533626+0000","last_change":"2024-02-13T00:49:38.569473+0000","last_active":"2024-02-13T00:49:53.533626+0000","last_peered":"2024-02-13T00:49:53.533626+0000","last_clean":"2024-02-13T00:49:53.533626+0000","last_became_active":"2024-02-13T00:49:38.569185+0000","last_became_peered":"2024-02-13T00:49:38.569185+0000","last_unstale":"2024-02-13T00:49:53.533626+0000","last_undegraded":"2024-02-13T00:49:53.533626+0000","last_fullsized":"2024-02-13T00:49:53.533626+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:49:37.490792+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.490792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":482,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535438+0000","last_change":"2024-02-13T00:46:01.448309+0000","last_active":"2024-02-13T00:49:53.535438+0000","last_peered":"2024-02-13T00:49:53.535438+0000","last_clean":"2024-02-13T00:49:53.535438+0000","last_became_active":"2024-02-13T00:46:01.445363+0000","last_became_peered":"2024-02-13T00:46:01.445363+0000","last_unstale":"2024-02-13T00:49:53.535438+0000","last_undegraded":"2024-02-13T00:49:53.535438+0000","last_fullsized":"2024-02-13T00:49:53.535438+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":407,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527476+0000","last_change":"2024-02-13T00:46:01.449894+0000","last_active":"2024-02-13T00:49:53.527476+0000","last_peered":"2024-02-13T00:49:53.527476+0000","last_clean":"2024-02-13T00:49:53.527476+0000","last_became_active":"2024-02-13T00:46:01.449570+0000","last_became_peered":"2024-02-13T00:46:01.449570+0000","last_unstale":"2024-02-13T00:49:53.527476+0000","last_undegraded":"2024-02-13T00:49:53.527476+0000","last_fullsized":"2024-02-13T00:49:53.527476+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":271,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.526877+0000","last_change":"2024-02-13T00:46:01.445141+0000","last_active":"2024-02-13T00:49:53.526877+0000","last_peered":"2024-02-13T00:49:53.526877+0000","last_clean":"2024-02-13T00:49:53.526877+0000","last_became_active":"2024-02-13T00:46:01.444729+0000","last_became_peered":"2024-02-13T00:46:01.444729+0000","last_unstale":"2024-02-13T00:49:53.526877+0000","last_undegraded":"2024-02-13T00:49:53.526877+0000","last_fullsized":"2024-02-13T00:49:53.526877+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":356,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.526806+0000","last_change":"2024-02-13T00:45:15.426658+0000","last_active":"2024-02-13T00:49:53.526806+0000","last_peered":"2024-02-13T00:49:53.526806+0000","last_clean":"2024-02-13T00:49:53.526806+0000","last_became_active":"2024-02-13T00:45:15.426109+0000","last_became_peered":"2024-02-13T00:45:15.426109+0000","last_unstale":"2024-02-13T00:49:53.526806+0000","last_undegraded":"2024-02-13T00:49:53.526806+0000","last_fullsized":"2024-02-13T00:49:53.526806+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":263,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528430+0000","last_change":"2024-02-13T00:44:39.526065+0000","last_active":"2024-02-13T00:49:53.528430+0000","last_peered":"2024-02-13T00:49:53.528430+0000","last_clean":"2024-02-13T00:49:53.528430+0000","last_became_active":"2024-02-13T00:44:39.525161+0000","last_became_peered":"2024-02-13T00:44:39.525161+0000","last_unstale":"2024-02-13T00:49:53.528430+0000","last_undegraded":"2024-02-13T00:49:53.528430+0000","last_fullsized":"2024-02-13T00:49:53.528430+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":359,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527404+0000","last_change":"2024-02-13T00:48:06.705421+0000","last_active":"2024-02-13T00:49:53.527404+0000","last_peered":"2024-02-13T00:49:53.527404+0000","last_clean":"2024-02-13T00:49:53.527404+0000","last_became_active":"2024-02-13T00:48:06.705121+0000","last_became_peered":"2024-02-13T00:48:06.705121+0000","last_unstale":"2024-02-13T00:49:53.527404+0000","last_undegraded":"2024-02-13T00:49:53.527404+0000","last_fullsized":"2024-02-13T00:49:53.527404+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":281,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529683+0000","last_change":"2024-02-13T00:46:01.459772+0000","last_active":"2024-02-13T00:49:53.529683+0000","last_peered":"2024-02-13T00:49:53.529683+0000","last_clean":"2024-02-13T00:49:53.529683+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:49:53.529683+0000","last_undegraded":"2024-02-13T00:49:53.529683+0000","last_fullsized":"2024-02-13T00:49:53.529683+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":540,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529829+0000","last_change":"2024-02-13T00:46:01.463062+0000","last_active":"2024-02-13T00:49:53.529829+0000","last_peered":"2024-02-13T00:49:53.529829+0000","last_clean":"2024-02-13T00:49:53.529829+0000","last_became_active":"2024-02-13T00:46:01.462319+0000","last_became_peered":"2024-02-13T00:46:01.462319+0000","last_unstale":"2024-02-13T00:49:53.529829+0000","last_undegraded":"2024-02-13T00:49:53.529829+0000","last_fullsized":"2024-02-13T00:49:53.529829+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":268,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.531478+0000","last_change":"2024-02-13T00:49:48.070260+0000","last_active":"2024-02-13T00:49:53.531478+0000","last_peered":"2024-02-13T00:49:53.531478+0000","last_clean":"2024-02-13T00:49:53.531478+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:49:53.531478+0000","last_undegraded":"2024-02-13T00:49:53.531478+0000","last_fullsized":"2024-02-13T00:49:53.531478+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:48.070195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:49:48.070195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":347,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530419+0000","last_change":"2024-02-13T00:46:01.455629+0000","last_active":"2024-02-13T00:49:53.530419+0000","last_peered":"2024-02-13T00:49:53.530419+0000","last_clean":"2024-02-13T00:49:53.530419+0000","last_became_active":"2024-02-13T00:46:01.452801+0000","last_became_peered":"2024-02-13T00:46:01.452801+0000","last_unstale":"2024-02-13T00:49:53.530419+0000","last_undegraded":"2024-02-13T00:49:53.530419+0000","last_fullsized":"2024-02-13T00:49:53.530419+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":503,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535291+0000","last_change":"2024-02-13T00:45:39.562095+0000","last_active":"2024-02-13T00:49:53.535291+0000","last_peered":"2024-02-13T00:49:53.535291+0000","last_clean":"2024-02-13T00:49:53.535291+0000","last_became_active":"2024-02-13T00:44:39.505121+0000","last_became_peered":"2024-02-13T00:44:39.505121+0000","last_unstale":"2024-02-13T00:49:53.535291+0000","last_undegraded":"2024-02-13T00:49:53.535291+0000","last_fullsized":"2024-02-13T00:49:53.535291+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:39.561994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:39.561994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"534'16","reported_seq":370,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.353251+0000","last_change":"2024-02-13T00:49:07.494629+0000","last_active":"2024-02-13T00:49:54.353251+0000","last_peered":"2024-02-13T00:49:54.353251+0000","last_clean":"2024-02-13T00:49:54.353251+0000","last_became_active":"2024-02-13T00:49:07.494388+0000","last_became_peered":"2024-02-13T00:49:07.494388+0000","last_unstale":"2024-02-13T00:49:54.353251+0000","last_undegraded":"2024-02-13T00:49:54.353251+0000","last_fullsized":"2024-02-13T00:49:54.353251+0000","mapping_epoch":359,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:45:53.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.115260+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":607,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527153+0000","last_change":"2024-02-13T00:47:03.491720+0000","last_active":"2024-02-13T00:49:53.527153+0000","last_peered":"2024-02-13T00:49:53.527153+0000","last_clean":"2024-02-13T00:49:53.527153+0000","last_became_active":"2024-02-13T00:46:01.438056+0000","last_became_peered":"2024-02-13T00:46:01.438056+0000","last_unstale":"2024-02-13T00:49:53.527153+0000","last_undegraded":"2024-02-13T00:49:53.527153+0000","last_fullsized":"2024-02-13T00:49:53.527153+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:47:03.491647+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":577,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530216+0000","last_change":"2024-02-13T00:48:23.379872+0000","last_active":"2024-02-13T00:49:53.530216+0000","last_peered":"2024-02-13T00:49:53.530216+0000","last_clean":"2024-02-13T00:49:53.530216+0000","last_became_active":"2024-02-13T00:46:01.460811+0000","last_became_peered":"2024-02-13T00:46:01.460811+0000","last_unstale":"2024-02-13T00:49:53.530216+0000","last_undegraded":"2024-02-13T00:49:53.530216+0000","last_fullsized":"2024-02-13T00:49:53.530216+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:23.379773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:23.379773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":483,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529180+0000","last_change":"2024-02-13T00:47:34.332791+0000","last_active":"2024-02-13T00:49:53.529180+0000","last_peered":"2024-02-13T00:49:53.529180+0000","last_clean":"2024-02-13T00:49:53.529180+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:49:53.529180+0000","last_undegraded":"2024-02-13T00:49:53.529180+0000","last_fullsized":"2024-02-13T00:49:53.529180+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:34.332704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:34.332704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":521,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528191+0000","last_change":"2024-02-13T00:46:01.448237+0000","last_active":"2024-02-13T00:49:53.528191+0000","last_peered":"2024-02-13T00:49:53.528191+0000","last_clean":"2024-02-13T00:49:53.528191+0000","last_became_active":"2024-02-13T00:46:01.446660+0000","last_became_peered":"2024-02-13T00:46:01.446660+0000","last_unstale":"2024-02-13T00:49:53.528191+0000","last_undegraded":"2024-02-13T00:49:53.528191+0000","last_fullsized":"2024-02-13T00:49:53.528191+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.260871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.260871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":582,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.538388+0000","last_change":"2024-02-13T00:47:41.808333+0000","last_active":"2024-02-13T00:49:53.538388+0000","last_peered":"2024-02-13T00:49:53.538388+0000","last_clean":"2024-02-13T00:49:53.538388+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:49:53.538388+0000","last_undegraded":"2024-02-13T00:49:53.538388+0000","last_fullsized":"2024-02-13T00:49:53.538388+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:47:41.808259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:41.808259+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":562,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529706+0000","last_change":"2024-02-13T00:48:24.823190+0000","last_active":"2024-02-13T00:49:53.529706+0000","last_peered":"2024-02-13T00:49:53.529706+0000","last_clean":"2024-02-13T00:49:53.529706+0000","last_became_active":"2024-02-13T00:44:50.429994+0000","last_became_peered":"2024-02-13T00:44:50.429994+0000","last_unstale":"2024-02-13T00:49:53.529706+0000","last_undegraded":"2024-02-13T00:49:53.529706+0000","last_fullsized":"2024-02-13T00:49:53.529706+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:24.823107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.823107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":466,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.537428+0000","last_change":"2024-02-13T00:49:39.788846+0000","last_active":"2024-02-13T00:49:53.537428+0000","last_peered":"2024-02-13T00:49:53.537428+0000","last_clean":"2024-02-13T00:49:53.537428+0000","last_became_active":"2024-02-13T00:49:03.971407+0000","last_became_peered":"2024-02-13T00:49:03.971407+0000","last_unstale":"2024-02-13T00:49:53.537428+0000","last_undegraded":"2024-02-13T00:49:53.537428+0000","last_fullsized":"2024-02-13T00:49:53.537428+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.788783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.788783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":565,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535662+0000","last_change":"2024-02-13T00:49:41.837871+0000","last_active":"2024-02-13T00:49:53.535662+0000","last_peered":"2024-02-13T00:49:53.535662+0000","last_clean":"2024-02-13T00:49:53.535662+0000","last_became_active":"2024-02-13T00:45:15.434853+0000","last_became_peered":"2024-02-13T00:45:15.434853+0000","last_unstale":"2024-02-13T00:49:53.535662+0000","last_undegraded":"2024-02-13T00:49:53.535662+0000","last_fullsized":"2024-02-13T00:49:53.535662+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.837810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.837810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":186,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.536954+0000","last_change":"2024-02-13T00:46:01.455201+0000","last_active":"2024-02-13T00:49:53.536954+0000","last_peered":"2024-02-13T00:49:53.536954+0000","last_clean":"2024-02-13T00:49:53.536954+0000","last_became_active":"2024-02-13T00:46:01.454882+0000","last_became_peered":"2024-02-13T00:46:01.454882+0000","last_unstale":"2024-02-13T00:49:53.536954+0000","last_undegraded":"2024-02-13T00:49:53.536954+0000","last_fullsized":"2024-02-13T00:49:53.536954+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:35.197045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:35.197045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":371,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529871+0000","last_change":"2024-02-13T00:46:09.256189+0000","last_active":"2024-02-13T00:49:53.529871+0000","last_peered":"2024-02-13T00:49:53.529871+0000","last_clean":"2024-02-13T00:49:53.529871+0000","last_became_active":"2024-02-13T00:46:01.459978+0000","last_became_peered":"2024-02-13T00:46:01.459978+0000","last_unstale":"2024-02-13T00:49:53.529871+0000","last_undegraded":"2024-02-13T00:49:53.529871+0000","last_fullsized":"2024-02-13T00:49:53.529871+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:44:24.196214+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:44:24.196214+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":196,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.538423+0000","last_change":"2024-02-13T00:46:30.641539+0000","last_active":"2024-02-13T00:49:53.538423+0000","last_peered":"2024-02-13T00:49:53.538423+0000","last_clean":"2024-02-13T00:49:53.538423+0000","last_became_active":"2024-02-13T00:46:01.444548+0000","last_became_peered":"2024-02-13T00:46:01.444548+0000","last_unstale":"2024-02-13T00:49:53.538423+0000","last_undegraded":"2024-02-13T00:49:53.538423+0000","last_fullsized":"2024-02-13T00:49:53.538423+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.641460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.641460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":296,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.537463+0000","last_change":"2024-02-13T00:48:54.498301+0000","last_active":"2024-02-13T00:49:53.537463+0000","last_peered":"2024-02-13T00:49:53.537463+0000","last_clean":"2024-02-13T00:49:53.537463+0000","last_became_active":"2024-02-13T00:48:54.497977+0000","last_became_peered":"2024-02-13T00:48:54.497977+0000","last_unstale":"2024-02-13T00:49:53.537463+0000","last_undegraded":"2024-02-13T00:49:53.537463+0000","last_fullsized":"2024-02-13T00:49:53.537463+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.592672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":399,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527444+0000","last_change":"2024-02-13T00:45:15.429026+0000","last_active":"2024-02-13T00:49:53.527444+0000","last_peered":"2024-02-13T00:49:53.527444+0000","last_clean":"2024-02-13T00:49:53.527444+0000","last_became_active":"2024-02-13T00:45:15.427945+0000","last_became_peered":"2024-02-13T00:45:15.427945+0000","last_unstale":"2024-02-13T00:49:53.527444+0000","last_undegraded":"2024-02-13T00:49:53.527444+0000","last_fullsized":"2024-02-13T00:49:53.527444+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":502,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529512+0000","last_change":"2024-02-13T00:45:26.255907+0000","last_active":"2024-02-13T00:49:53.529512+0000","last_peered":"2024-02-13T00:49:53.529512+0000","last_clean":"2024-02-13T00:49:53.529512+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:49:53.529512+0000","last_undegraded":"2024-02-13T00:49:53.529512+0000","last_fullsized":"2024-02-13T00:49:53.529512+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.255842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.255842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"467'31","reported_seq":339,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529569+0000","last_change":"2024-02-13T00:48:31.782821+0000","last_active":"2024-02-13T00:49:53.529569+0000","last_peered":"2024-02-13T00:49:53.529569+0000","last_clean":"2024-02-13T00:49:53.529569+0000","last_became_active":"2024-02-13T00:44:50.432807+0000","last_became_peered":"2024-02-13T00:44:50.432807+0000","last_unstale":"2024-02-13T00:49:53.529569+0000","last_undegraded":"2024-02-13T00:49:53.529569+0000","last_fullsized":"2024-02-13T00:49:53.529569+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'31","last_scrub_stamp":"2024-02-13T00:48:31.782729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.782729+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":354,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528004+0000","last_change":"2024-02-13T00:46:04.708510+0000","last_active":"2024-02-13T00:49:53.528004+0000","last_peered":"2024-02-13T00:49:53.528004+0000","last_clean":"2024-02-13T00:49:53.528004+0000","last_became_active":"2024-02-13T00:46:01.447801+0000","last_became_peered":"2024-02-13T00:46:01.447801+0000","last_unstale":"2024-02-13T00:49:53.528004+0000","last_undegraded":"2024-02-13T00:49:53.528004+0000","last_fullsized":"2024-02-13T00:49:53.528004+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:45:24.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:24.269595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":560,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529708+0000","last_change":"2024-02-13T00:48:49.612735+0000","last_active":"2024-02-13T00:49:53.529708+0000","last_peered":"2024-02-13T00:49:53.529708+0000","last_clean":"2024-02-13T00:49:53.529708+0000","last_became_active":"2024-02-13T00:48:49.612437+0000","last_became_peered":"2024-02-13T00:48:49.612437+0000","last_unstale":"2024-02-13T00:49:53.529708+0000","last_undegraded":"2024-02-13T00:49:53.529708+0000","last_fullsized":"2024-02-13T00:49:53.529708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.793634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":548,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.531131+0000","last_change":"2024-02-13T00:44:50.441408+0000","last_active":"2024-02-13T00:49:53.531131+0000","last_peered":"2024-02-13T00:49:53.531131+0000","last_clean":"2024-02-13T00:49:53.531131+0000","last_became_active":"2024-02-13T00:44:50.440301+0000","last_became_peered":"2024-02-13T00:44:50.440301+0000","last_unstale":"2024-02-13T00:49:53.531131+0000","last_undegraded":"2024-02-13T00:49:53.531131+0000","last_fullsized":"2024-02-13T00:49:53.531131+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:25.502311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.502311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":203,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.537504+0000","last_change":"2024-02-13T00:48:06.705079+0000","last_active":"2024-02-13T00:49:53.537504+0000","last_peered":"2024-02-13T00:49:53.537504+0000","last_clean":"2024-02-13T00:49:53.537504+0000","last_became_active":"2024-02-13T00:48:06.704718+0000","last_became_peered":"2024-02-13T00:48:06.704718+0000","last_unstale":"2024-02-13T00:49:53.537504+0000","last_undegraded":"2024-02-13T00:49:53.537504+0000","last_fullsized":"2024-02-13T00:49:53.537504+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.488353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.488353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":295,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530100+0000","last_change":"2024-02-13T00:49:03.294585+0000","last_active":"2024-02-13T00:49:53.530100+0000","last_peered":"2024-02-13T00:49:53.530100+0000","last_clean":"2024-02-13T00:49:53.530100+0000","last_became_active":"2024-02-13T00:44:39.523138+0000","last_became_peered":"2024-02-13T00:44:39.523138+0000","last_unstale":"2024-02-13T00:49:53.530100+0000","last_undegraded":"2024-02-13T00:49:53.530100+0000","last_fullsized":"2024-02-13T00:49:53.530100+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:49:03.294499+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:49:03.294499+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":571,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527689+0000","last_change":"2024-02-13T00:48:59.111910+0000","last_active":"2024-02-13T00:49:53.527689+0000","last_peered":"2024-02-13T00:49:53.527689+0000","last_clean":"2024-02-13T00:49:53.527689+0000","last_became_active":"2024-02-13T00:46:01.452674+0000","last_became_peered":"2024-02-13T00:46:01.452674+0000","last_unstale":"2024-02-13T00:49:53.527689+0000","last_undegraded":"2024-02-13T00:49:53.527689+0000","last_fullsized":"2024-02-13T00:49:53.527689+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:59.111828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:59.111828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"511'13","reported_seq":285,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.352999+0000","last_change":"2024-02-13T00:48:45.592523+0000","last_active":"2024-02-13T00:49:54.352999+0000","last_peered":"2024-02-13T00:49:54.352999+0000","last_clean":"2024-02-13T00:49:54.352999+0000","last_became_active":"2024-02-13T00:48:45.591249+0000","last_became_peered":"2024-02-13T00:48:45.591249+0000","last_unstale":"2024-02-13T00:49:54.352999+0000","last_undegraded":"2024-02-13T00:49:54.352999+0000","last_fullsized":"2024-02-13T00:49:54.352999+0000","mapping_epoch":401,"log_start":"404'11","ondisk_log_start":"404'11","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'11","last_scrub_stamp":"2024-02-13T00:46:37.662413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.662413+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":538,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530576+0000","last_change":"2024-02-13T00:46:05.839833+0000","last_active":"2024-02-13T00:49:53.530576+0000","last_peered":"2024-02-13T00:49:53.530576+0000","last_clean":"2024-02-13T00:49:53.530576+0000","last_became_active":"2024-02-13T00:46:01.455274+0000","last_became_peered":"2024-02-13T00:46:01.455274+0000","last_unstale":"2024-02-13T00:49:53.530576+0000","last_undegraded":"2024-02-13T00:49:53.530576+0000","last_fullsized":"2024-02-13T00:49:53.530576+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:44:25.230980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.230980+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":347,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.531067+0000","last_change":"2024-02-13T00:46:01.457421+0000","last_active":"2024-02-13T00:49:53.531067+0000","last_peered":"2024-02-13T00:49:53.531067+0000","last_clean":"2024-02-13T00:49:53.531067+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:49:53.531067+0000","last_undegraded":"2024-02-13T00:49:53.531067+0000","last_fullsized":"2024-02-13T00:49:53.531067+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:18.089252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:18.089252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"467'27","reported_seq":660,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530981+0000","last_change":"2024-02-13T00:48:54.043163+0000","last_active":"2024-02-13T00:49:53.530981+0000","last_peered":"2024-02-13T00:49:53.530981+0000","last_clean":"2024-02-13T00:49:53.530981+0000","last_became_active":"2024-02-13T00:46:01.448705+0000","last_became_peered":"2024-02-13T00:46:01.448705+0000","last_unstale":"2024-02-13T00:49:53.530981+0000","last_undegraded":"2024-02-13T00:49:53.530981+0000","last_fullsized":"2024-02-13T00:49:53.530981+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:48:54.043088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.043088+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":603,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527305+0000","last_change":"2024-02-13T00:49:37.053462+0000","last_active":"2024-02-13T00:49:53.527305+0000","last_peered":"2024-02-13T00:49:53.527305+0000","last_clean":"2024-02-13T00:49:53.527305+0000","last_became_active":"2024-02-13T00:45:15.445228+0000","last_became_peered":"2024-02-13T00:45:15.445228+0000","last_unstale":"2024-02-13T00:49:53.527305+0000","last_undegraded":"2024-02-13T00:49:53.527305+0000","last_fullsized":"2024-02-13T00:49:53.527305+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:49:37.053398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.053398+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"507'13","reported_seq":283,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.352866+0000","last_change":"2024-02-13T00:48:41.526587+0000","last_active":"2024-02-13T00:49:54.352866+0000","last_peered":"2024-02-13T00:49:54.352866+0000","last_clean":"2024-02-13T00:49:54.352866+0000","last_became_active":"2024-02-13T00:48:41.525262+0000","last_became_peered":"2024-02-13T00:48:41.525262+0000","last_unstale":"2024-02-13T00:49:54.352866+0000","last_undegraded":"2024-02-13T00:49:54.352866+0000","last_fullsized":"2024-02-13T00:49:54.352866+0000","mapping_epoch":401,"log_start":"467'11","ondisk_log_start":"467'11","created":60,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:47:49.687727+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":188,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529226+0000","last_change":"2024-02-13T00:46:01.457110+0000","last_active":"2024-02-13T00:49:53.529226+0000","last_peered":"2024-02-13T00:49:53.529226+0000","last_clean":"2024-02-13T00:49:53.529226+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:49:53.529226+0000","last_undegraded":"2024-02-13T00:49:53.529226+0000","last_fullsized":"2024-02-13T00:49:53.529226+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.092250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.092250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":503,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540584+0000","last_change":"2024-02-13T00:46:01.456531+0000","last_active":"2024-02-13T00:49:53.540584+0000","last_peered":"2024-02-13T00:49:53.540584+0000","last_clean":"2024-02-13T00:49:53.540584+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:49:53.540584+0000","last_undegraded":"2024-02-13T00:49:53.540584+0000","last_fullsized":"2024-02-13T00:49:53.540584+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:44:59.963616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":294,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530170+0000","last_change":"2024-02-13T00:48:57.386335+0000","last_active":"2024-02-13T00:49:53.530170+0000","last_peered":"2024-02-13T00:49:53.530170+0000","last_clean":"2024-02-13T00:49:53.530170+0000","last_became_active":"2024-02-13T00:44:39.521984+0000","last_became_peered":"2024-02-13T00:44:39.521984+0000","last_unstale":"2024-02-13T00:49:53.530170+0000","last_undegraded":"2024-02-13T00:49:53.530170+0000","last_fullsized":"2024-02-13T00:49:53.530170+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:57.386254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:57.386254+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":386,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535114+0000","last_change":"2024-02-13T00:46:23.757417+0000","last_active":"2024-02-13T00:49:53.535114+0000","last_peered":"2024-02-13T00:49:53.535114+0000","last_clean":"2024-02-13T00:49:53.535114+0000","last_became_active":"2024-02-13T00:45:15.419239+0000","last_became_peered":"2024-02-13T00:45:15.419239+0000","last_unstale":"2024-02-13T00:49:53.535114+0000","last_undegraded":"2024-02-13T00:49:53.535114+0000","last_fullsized":"2024-02-13T00:49:53.535114+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:23.757351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:23.757351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"504'16","reported_seq":523,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.353234+0000","last_change":"2024-02-13T00:49:45.051624+0000","last_active":"2024-02-13T00:49:54.353234+0000","last_peered":"2024-02-13T00:49:54.353234+0000","last_clean":"2024-02-13T00:49:54.353234+0000","last_became_active":"2024-02-13T00:48:37.480010+0000","last_became_peered":"2024-02-13T00:48:37.480010+0000","last_unstale":"2024-02-13T00:49:54.353234+0000","last_undegraded":"2024-02-13T00:49:54.353234+0000","last_fullsized":"2024-02-13T00:49:54.353234+0000","mapping_epoch":401,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'16","last_scrub_stamp":"2024-02-13T00:49:45.051539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:49:45.051539+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":286,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529916+0000","last_change":"2024-02-13T00:48:31.463534+0000","last_active":"2024-02-13T00:49:53.529916+0000","last_peered":"2024-02-13T00:49:53.529916+0000","last_clean":"2024-02-13T00:49:53.529916+0000","last_became_active":"2024-02-13T00:46:01.462717+0000","last_became_peered":"2024-02-13T00:46:01.462717+0000","last_unstale":"2024-02-13T00:49:53.529916+0000","last_undegraded":"2024-02-13T00:49:53.529916+0000","last_fullsized":"2024-02-13T00:49:53.529916+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:31.463442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.463442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":460,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535340+0000","last_change":"2024-02-13T00:49:00.881498+0000","last_active":"2024-02-13T00:49:53.535340+0000","last_peered":"2024-02-13T00:49:53.535340+0000","last_clean":"2024-02-13T00:49:53.535340+0000","last_became_active":"2024-02-13T00:46:01.456959+0000","last_became_peered":"2024-02-13T00:46:01.456959+0000","last_unstale":"2024-02-13T00:49:53.535340+0000","last_undegraded":"2024-02-13T00:49:53.535340+0000","last_fullsized":"2024-02-13T00:49:53.535340+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:00.881431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:00.881431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"263'10","reported_seq":576,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.352782+0000","last_change":"2024-02-13T00:47:02.949483+0000","last_active":"2024-02-13T00:49:54.352782+0000","last_peered":"2024-02-13T00:49:54.352782+0000","last_clean":"2024-02-13T00:49:54.352782+0000","last_became_active":"2024-02-13T00:43:35.454087+0000","last_became_peered":"2024-02-13T00:43:35.454087+0000","last_unstale":"2024-02-13T00:49:54.352782+0000","last_undegraded":"2024-02-13T00:49:54.352782+0000","last_fullsized":"2024-02-13T00:49:54.352782+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.949406+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":558,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528230+0000","last_change":"2024-02-13T00:49:46.101991+0000","last_active":"2024-02-13T00:49:53.528230+0000","last_peered":"2024-02-13T00:49:53.528230+0000","last_clean":"2024-02-13T00:49:53.528230+0000","last_became_active":"2024-02-13T00:46:01.449654+0000","last_became_peered":"2024-02-13T00:46:01.449654+0000","last_unstale":"2024-02-13T00:49:53.528230+0000","last_undegraded":"2024-02-13T00:49:53.528230+0000","last_fullsized":"2024-02-13T00:49:53.528230+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'34","last_scrub_stamp":"2024-02-13T00:49:46.101874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.101874+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":286,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.531376+0000","last_change":"2024-02-13T00:49:47.075845+0000","last_active":"2024-02-13T00:49:53.531376+0000","last_peered":"2024-02-13T00:49:53.531376+0000","last_clean":"2024-02-13T00:49:53.531376+0000","last_became_active":"2024-02-13T00:45:15.446443+0000","last_became_peered":"2024-02-13T00:45:15.446443+0000","last_unstale":"2024-02-13T00:49:53.531376+0000","last_undegraded":"2024-02-13T00:49:53.531376+0000","last_fullsized":"2024-02-13T00:49:53.531376+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:47.075741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.075741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":281,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528497+0000","last_change":"2024-02-13T00:45:15.436286+0000","last_active":"2024-02-13T00:49:53.528497+0000","last_peered":"2024-02-13T00:49:53.528497+0000","last_clean":"2024-02-13T00:49:53.528497+0000","last_became_active":"2024-02-13T00:45:15.434559+0000","last_became_peered":"2024-02-13T00:45:15.434559+0000","last_unstale":"2024-02-13T00:49:53.528497+0000","last_undegraded":"2024-02-13T00:49:53.528497+0000","last_fullsized":"2024-02-13T00:49:53.528497+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":310,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535030+0000","last_change":"2024-02-13T00:49:36.841674+0000","last_active":"2024-02-13T00:49:53.535030+0000","last_peered":"2024-02-13T00:49:53.535030+0000","last_clean":"2024-02-13T00:49:53.535030+0000","last_became_active":"2024-02-13T00:46:01.444661+0000","last_became_peered":"2024-02-13T00:46:01.444661+0000","last_unstale":"2024-02-13T00:49:53.535030+0000","last_undegraded":"2024-02-13T00:49:53.535030+0000","last_fullsized":"2024-02-13T00:49:53.535030+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:36.841508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:36.841508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":351,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535418+0000","last_change":"2024-02-13T00:48:18.751148+0000","last_active":"2024-02-13T00:49:53.535418+0000","last_peered":"2024-02-13T00:49:53.535418+0000","last_clean":"2024-02-13T00:49:53.535418+0000","last_became_active":"2024-02-13T00:46:01.453842+0000","last_became_peered":"2024-02-13T00:46:01.453842+0000","last_unstale":"2024-02-13T00:49:53.535418+0000","last_undegraded":"2024-02-13T00:49:53.535418+0000","last_fullsized":"2024-02-13T00:49:53.535418+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:18.751081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:18.751081+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"467'42","reported_seq":512,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535080+0000","last_change":"2024-02-13T00:48:24.675807+0000","last_active":"2024-02-13T00:49:53.535080+0000","last_peered":"2024-02-13T00:49:53.535080+0000","last_clean":"2024-02-13T00:49:53.535080+0000","last_became_active":"2024-02-13T00:46:01.445058+0000","last_became_peered":"2024-02-13T00:46:01.445058+0000","last_unstale":"2024-02-13T00:49:53.535080+0000","last_undegraded":"2024-02-13T00:49:53.535080+0000","last_fullsized":"2024-02-13T00:49:53.535080+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'42","last_scrub_stamp":"2024-02-13T00:48:24.675687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.675687+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":571,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530462+0000","last_change":"2024-02-13T00:47:09.940673+0000","last_active":"2024-02-13T00:49:53.530462+0000","last_peered":"2024-02-13T00:49:53.530462+0000","last_clean":"2024-02-13T00:49:53.530462+0000","last_became_active":"2024-02-13T00:44:39.503590+0000","last_became_peered":"2024-02-13T00:44:39.503590+0000","last_unstale":"2024-02-13T00:49:53.530462+0000","last_undegraded":"2024-02-13T00:49:53.530462+0000","last_fullsized":"2024-02-13T00:49:53.530462+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:09.940605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:09.940605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":516,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529273+0000","last_change":"2024-02-13T00:46:01.448313+0000","last_active":"2024-02-13T00:49:53.529273+0000","last_peered":"2024-02-13T00:49:53.529273+0000","last_clean":"2024-02-13T00:49:53.529273+0000","last_became_active":"2024-02-13T00:46:01.447815+0000","last_became_peered":"2024-02-13T00:46:01.447815+0000","last_unstale":"2024-02-13T00:49:53.529273+0000","last_undegraded":"2024-02-13T00:49:53.529273+0000","last_fullsized":"2024-02-13T00:49:53.529273+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":508,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528103+0000","last_change":"2024-02-13T00:48:35.434713+0000","last_active":"2024-02-13T00:49:53.528103+0000","last_peered":"2024-02-13T00:49:53.528103+0000","last_clean":"2024-02-13T00:49:53.528103+0000","last_became_active":"2024-02-13T00:44:50.436058+0000","last_became_peered":"2024-02-13T00:44:50.436058+0000","last_unstale":"2024-02-13T00:49:53.528103+0000","last_undegraded":"2024-02-13T00:49:53.528103+0000","last_fullsized":"2024-02-13T00:49:53.528103+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_clean_scrub_stamp":"2024-02-13T00:48:35.434645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":277,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529970+0000","last_change":"2024-02-13T00:46:01.451625+0000","last_active":"2024-02-13T00:49:53.529970+0000","last_peered":"2024-02-13T00:49:53.529970+0000","last_clean":"2024-02-13T00:49:53.529970+0000","last_became_active":"2024-02-13T00:46:01.450587+0000","last_became_peered":"2024-02-13T00:46:01.450587+0000","last_unstale":"2024-02-13T00:49:53.529970+0000","last_undegraded":"2024-02-13T00:49:53.529970+0000","last_fullsized":"2024-02-13T00:49:53.529970+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:33.204969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:33.204969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":675,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.538217+0000","last_change":"2024-02-13T00:47:05.718055+0000","last_active":"2024-02-13T00:49:53.538217+0000","last_peered":"2024-02-13T00:49:53.538217+0000","last_clean":"2024-02-13T00:49:53.538217+0000","last_became_active":"2024-02-13T00:46:01.443777+0000","last_became_peered":"2024-02-13T00:46:01.443777+0000","last_unstale":"2024-02-13T00:49:53.538217+0000","last_undegraded":"2024-02-13T00:49:53.538217+0000","last_fullsized":"2024-02-13T00:49:53.538217+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:47:05.717987+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:47:05.717987+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":386,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540310+0000","last_change":"2024-02-13T00:45:15.423607+0000","last_active":"2024-02-13T00:49:53.540310+0000","last_peered":"2024-02-13T00:49:53.540310+0000","last_clean":"2024-02-13T00:49:53.540310+0000","last_became_active":"2024-02-13T00:45:15.423415+0000","last_became_peered":"2024-02-13T00:45:15.423415+0000","last_unstale":"2024-02-13T00:49:53.540310+0000","last_undegraded":"2024-02-13T00:49:53.540310+0000","last_fullsized":"2024-02-13T00:49:53.540310+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:17.295459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:17.295459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":578,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527391+0000","last_change":"2024-02-13T00:45:15.445862+0000","last_active":"2024-02-13T00:49:53.527391+0000","last_peered":"2024-02-13T00:49:53.527391+0000","last_clean":"2024-02-13T00:49:53.527391+0000","last_became_active":"2024-02-13T00:45:15.445557+0000","last_became_peered":"2024-02-13T00:45:15.445557+0000","last_unstale":"2024-02-13T00:49:53.527391+0000","last_undegraded":"2024-02-13T00:49:53.527391+0000","last_fullsized":"2024-02-13T00:49:53.527391+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":196,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530315+0000","last_change":"2024-02-13T00:48:55.050483+0000","last_active":"2024-02-13T00:49:53.530315+0000","last_peered":"2024-02-13T00:49:53.530315+0000","last_clean":"2024-02-13T00:49:53.530315+0000","last_became_active":"2024-02-13T00:46:01.441421+0000","last_became_peered":"2024-02-13T00:46:01.441421+0000","last_unstale":"2024-02-13T00:49:53.530315+0000","last_undegraded":"2024-02-13T00:49:53.530315+0000","last_fullsized":"2024-02-13T00:49:53.530315+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:55.050411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.050411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":211,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.531392+0000","last_change":"2024-02-13T00:49:16.505082+0000","last_active":"2024-02-13T00:49:53.531392+0000","last_peered":"2024-02-13T00:49:53.531392+0000","last_clean":"2024-02-13T00:49:53.531392+0000","last_became_active":"2024-02-13T00:49:16.504828+0000","last_became_peered":"2024-02-13T00:49:16.504828+0000","last_unstale":"2024-02-13T00:49:53.531392+0000","last_undegraded":"2024-02-13T00:49:53.531392+0000","last_fullsized":"2024-02-13T00:49:53.531392+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:43.903553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:43.903553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":670,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540335+0000","last_change":"2024-02-13T00:46:34.691109+0000","last_active":"2024-02-13T00:49:53.540335+0000","last_peered":"2024-02-13T00:49:53.540335+0000","last_clean":"2024-02-13T00:49:53.540335+0000","last_became_active":"2024-02-13T00:46:01.448163+0000","last_became_peered":"2024-02-13T00:46:01.448163+0000","last_unstale":"2024-02-13T00:49:53.540335+0000","last_undegraded":"2024-02-13T00:49:53.540335+0000","last_fullsized":"2024-02-13T00:49:53.540335+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:46:34.690980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.690980+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":279,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527089+0000","last_change":"2024-02-13T00:46:37.056299+0000","last_active":"2024-02-13T00:49:53.527089+0000","last_peered":"2024-02-13T00:49:53.527089+0000","last_clean":"2024-02-13T00:49:53.527089+0000","last_became_active":"2024-02-13T00:46:01.446104+0000","last_became_peered":"2024-02-13T00:46:01.446104+0000","last_unstale":"2024-02-13T00:49:53.527089+0000","last_undegraded":"2024-02-13T00:49:53.527089+0000","last_fullsized":"2024-02-13T00:49:53.527089+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:37.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":562,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535488+0000","last_change":"2024-02-13T00:46:01.431899+0000","last_active":"2024-02-13T00:49:53.535488+0000","last_peered":"2024-02-13T00:49:53.535488+0000","last_clean":"2024-02-13T00:49:53.535488+0000","last_became_active":"2024-02-13T00:46:01.430623+0000","last_became_peered":"2024-02-13T00:46:01.430623+0000","last_unstale":"2024-02-13T00:49:53.535488+0000","last_undegraded":"2024-02-13T00:49:53.535488+0000","last_fullsized":"2024-02-13T00:49:53.535488+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:46.556229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.556229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":569,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529801+0000","last_change":"2024-02-13T00:46:01.450548+0000","last_active":"2024-02-13T00:49:53.529801+0000","last_peered":"2024-02-13T00:49:53.529801+0000","last_clean":"2024-02-13T00:49:53.529801+0000","last_became_active":"2024-02-13T00:46:01.450176+0000","last_became_peered":"2024-02-13T00:46:01.450176+0000","last_unstale":"2024-02-13T00:49:53.529801+0000","last_undegraded":"2024-02-13T00:49:53.529801+0000","last_fullsized":"2024-02-13T00:49:53.529801+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":423,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529871+0000","last_change":"2024-02-13T00:46:01.460470+0000","last_active":"2024-02-13T00:49:53.529871+0000","last_peered":"2024-02-13T00:49:53.529871+0000","last_clean":"2024-02-13T00:49:53.529871+0000","last_became_active":"2024-02-13T00:46:01.459859+0000","last_became_peered":"2024-02-13T00:46:01.459859+0000","last_unstale":"2024-02-13T00:49:53.529871+0000","last_undegraded":"2024-02-13T00:49:53.529871+0000","last_fullsized":"2024-02-13T00:49:53.529871+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:01.179031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:01.179031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":227,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.537401+0000","last_change":"2024-02-13T00:47:33.932830+0000","last_active":"2024-02-13T00:49:53.537401+0000","last_peered":"2024-02-13T00:49:53.537401+0000","last_clean":"2024-02-13T00:49:53.537401+0000","last_became_active":"2024-02-13T00:46:01.455170+0000","last_became_peered":"2024-02-13T00:46:01.455170+0000","last_unstale":"2024-02-13T00:49:53.537401+0000","last_undegraded":"2024-02-13T00:49:53.537401+0000","last_fullsized":"2024-02-13T00:49:53.537401+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:33.932766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:33.932766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":327,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529941+0000","last_change":"2024-02-13T00:46:57.381602+0000","last_active":"2024-02-13T00:49:53.529941+0000","last_peered":"2024-02-13T00:49:53.529941+0000","last_clean":"2024-02-13T00:49:53.529941+0000","last_became_active":"2024-02-13T00:45:15.446969+0000","last_became_peered":"2024-02-13T00:45:15.446969+0000","last_unstale":"2024-02-13T00:49:53.529941+0000","last_undegraded":"2024-02-13T00:49:53.529941+0000","last_fullsized":"2024-02-13T00:49:53.529941+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:46:57.381540+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:46:57.381540+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":527,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.534942+0000","last_change":"2024-02-13T00:47:39.822745+0000","last_active":"2024-02-13T00:49:53.534942+0000","last_peered":"2024-02-13T00:49:53.534942+0000","last_clean":"2024-02-13T00:49:53.534942+0000","last_became_active":"2024-02-13T00:45:15.434498+0000","last_became_peered":"2024-02-13T00:45:15.434498+0000","last_unstale":"2024-02-13T00:49:53.534942+0000","last_undegraded":"2024-02-13T00:49:53.534942+0000","last_fullsized":"2024-02-13T00:49:53.534942+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:39.822619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:47:39.822619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":525,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527998+0000","last_change":"2024-02-13T00:47:04.493911+0000","last_active":"2024-02-13T00:49:53.527998+0000","last_peered":"2024-02-13T00:49:53.527998+0000","last_clean":"2024-02-13T00:49:53.527998+0000","last_became_active":"2024-02-13T00:46:01.447823+0000","last_became_peered":"2024-02-13T00:46:01.447823+0000","last_unstale":"2024-02-13T00:49:53.527998+0000","last_undegraded":"2024-02-13T00:49:53.527998+0000","last_fullsized":"2024-02-13T00:49:53.527998+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:04.493841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:04.493841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":398,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529021+0000","last_change":"2024-02-13T00:48:25.359625+0000","last_active":"2024-02-13T00:49:53.529021+0000","last_peered":"2024-02-13T00:49:53.529021+0000","last_clean":"2024-02-13T00:49:53.529021+0000","last_became_active":"2024-02-13T00:46:01.460001+0000","last_became_peered":"2024-02-13T00:46:01.460001+0000","last_unstale":"2024-02-13T00:49:53.529021+0000","last_undegraded":"2024-02-13T00:49:53.529021+0000","last_fullsized":"2024-02-13T00:49:53.529021+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:25.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:48:25.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":417,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.532058+0000","last_change":"2024-02-13T00:46:01.450879+0000","last_active":"2024-02-13T00:49:53.532058+0000","last_peered":"2024-02-13T00:49:53.532058+0000","last_clean":"2024-02-13T00:49:53.532058+0000","last_became_active":"2024-02-13T00:46:01.450088+0000","last_became_peered":"2024-02-13T00:46:01.450088+0000","last_unstale":"2024-02-13T00:49:53.532058+0000","last_undegraded":"2024-02-13T00:49:53.532058+0000","last_fullsized":"2024-02-13T00:49:53.532058+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":508,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540479+0000","last_change":"2024-02-13T00:49:42.854702+0000","last_active":"2024-02-13T00:49:53.540479+0000","last_peered":"2024-02-13T00:49:53.540479+0000","last_clean":"2024-02-13T00:49:53.540479+0000","last_became_active":"2024-02-13T00:46:01.458408+0000","last_became_peered":"2024-02-13T00:46:01.458408+0000","last_unstale":"2024-02-13T00:49:53.540479+0000","last_undegraded":"2024-02-13T00:49:53.540479+0000","last_fullsized":"2024-02-13T00:49:53.540479+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:42.854585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:42.854585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":248,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530240+0000","last_change":"2024-02-13T00:49:19.638381+0000","last_active":"2024-02-13T00:49:53.530240+0000","last_peered":"2024-02-13T00:49:53.530240+0000","last_clean":"2024-02-13T00:49:53.530240+0000","last_became_active":"2024-02-13T00:49:19.638083+0000","last_became_peered":"2024-02-13T00:49:19.638083+0000","last_unstale":"2024-02-13T00:49:53.530240+0000","last_undegraded":"2024-02-13T00:49:53.530240+0000","last_fullsized":"2024-02-13T00:49:53.530240+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:53.013103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.013103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"467'23","reported_seq":613,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.531144+0000","last_change":"2024-02-13T00:49:07.192972+0000","last_active":"2024-02-13T00:49:53.531144+0000","last_peered":"2024-02-13T00:49:53.531144+0000","last_clean":"2024-02-13T00:49:53.531144+0000","last_became_active":"2024-02-13T00:46:01.448219+0000","last_became_peered":"2024-02-13T00:46:01.448219+0000","last_unstale":"2024-02-13T00:49:53.531144+0000","last_undegraded":"2024-02-13T00:49:53.531144+0000","last_fullsized":"2024-02-13T00:49:53.531144+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'23","last_scrub_stamp":"2024-02-13T00:49:07.192902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:07.192902+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":482,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535499+0000","last_change":"2024-02-13T00:49:46.851558+0000","last_active":"2024-02-13T00:49:53.535499+0000","last_peered":"2024-02-13T00:49:53.535499+0000","last_clean":"2024-02-13T00:49:53.535499+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:49:53.535499+0000","last_undegraded":"2024-02-13T00:49:53.535499+0000","last_fullsized":"2024-02-13T00:49:53.535499+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.851486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.851486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":1313,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.539430+0000","last_change":"2024-02-13T00:45:54.075270+0000","last_active":"2024-02-13T00:49:54.539430+0000","last_peered":"2024-02-13T00:49:54.539430+0000","last_clean":"2024-02-13T00:49:54.539430+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:49:54.539430+0000","last_undegraded":"2024-02-13T00:49:54.539430+0000","last_fullsized":"2024-02-13T00:49:54.539430+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:45:54.075194+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":721,"num_read_kb":716,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":290,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529767+0000","last_change":"2024-02-13T00:46:01.460282+0000","last_active":"2024-02-13T00:49:53.529767+0000","last_peered":"2024-02-13T00:49:53.529767+0000","last_clean":"2024-02-13T00:49:53.529767+0000","last_became_active":"2024-02-13T00:46:01.459473+0000","last_became_peered":"2024-02-13T00:46:01.459473+0000","last_unstale":"2024-02-13T00:49:53.529767+0000","last_undegraded":"2024-02-13T00:49:53.529767+0000","last_fullsized":"2024-02-13T00:49:53.529767+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":342,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529280+0000","last_change":"2024-02-13T00:45:37.135398+0000","last_active":"2024-02-13T00:49:53.529280+0000","last_peered":"2024-02-13T00:49:53.529280+0000","last_clean":"2024-02-13T00:49:53.529280+0000","last_became_active":"2024-02-13T00:45:15.435588+0000","last_became_peered":"2024-02-13T00:45:15.435588+0000","last_unstale":"2024-02-13T00:49:53.529280+0000","last_undegraded":"2024-02-13T00:49:53.529280+0000","last_fullsized":"2024-02-13T00:49:53.529280+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.135332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.135332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":241,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529281+0000","last_change":"2024-02-13T00:45:31.183763+0000","last_active":"2024-02-13T00:49:53.529281+0000","last_peered":"2024-02-13T00:49:53.529281+0000","last_clean":"2024-02-13T00:49:53.529281+0000","last_became_active":"2024-02-13T00:45:06.421025+0000","last_became_peered":"2024-02-13T00:45:06.421025+0000","last_unstale":"2024-02-13T00:49:53.529281+0000","last_undegraded":"2024-02-13T00:49:53.529281+0000","last_fullsized":"2024-02-13T00:49:53.529281+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.183682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.183682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"467'65","reported_seq":280,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.352793+0000","last_change":"2024-02-13T00:49:47.928628+0000","last_active":"2024-02-13T00:49:54.352793+0000","last_peered":"2024-02-13T00:49:54.352793+0000","last_clean":"2024-02-13T00:49:54.352793+0000","last_became_active":"2024-02-13T00:46:01.449059+0000","last_became_peered":"2024-02-13T00:46:01.449059+0000","last_unstale":"2024-02-13T00:49:54.352793+0000","last_undegraded":"2024-02-13T00:49:54.352793+0000","last_fullsized":"2024-02-13T00:49:54.352793+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'65","last_scrub_stamp":"2024-02-13T00:49:47.928548+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.928548+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":2148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":487,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527540+0000","last_change":"2024-02-13T00:46:01.453042+0000","last_active":"2024-02-13T00:49:53.527540+0000","last_peered":"2024-02-13T00:49:53.527540+0000","last_clean":"2024-02-13T00:49:53.527540+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:49:53.527540+0000","last_undegraded":"2024-02-13T00:49:53.527540+0000","last_fullsized":"2024-02-13T00:49:53.527540+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.045912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.045912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":107,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.539972+0000","last_change":"2024-02-13T00:48:06.705030+0000","last_active":"2024-02-13T00:49:53.539972+0000","last_peered":"2024-02-13T00:49:53.539972+0000","last_clean":"2024-02-13T00:49:53.539972+0000","last_became_active":"2024-02-13T00:48:06.704651+0000","last_became_peered":"2024-02-13T00:48:06.704651+0000","last_unstale":"2024-02-13T00:49:53.539972+0000","last_undegraded":"2024-02-13T00:49:53.539972+0000","last_fullsized":"2024-02-13T00:49:53.539972+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":277,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530051+0000","last_change":"2024-02-13T00:48:17.340891+0000","last_active":"2024-02-13T00:49:53.530051+0000","last_peered":"2024-02-13T00:49:53.530051+0000","last_clean":"2024-02-13T00:49:53.530051+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:49:53.530051+0000","last_undegraded":"2024-02-13T00:49:53.530051+0000","last_fullsized":"2024-02-13T00:49:53.530051+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:17.340797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:17.340797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":749,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.353019+0000","last_change":"2024-02-13T00:48:32.669477+0000","last_active":"2024-02-13T00:49:54.353019+0000","last_peered":"2024-02-13T00:49:54.353019+0000","last_clean":"2024-02-13T00:49:54.353019+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:49:54.353019+0000","last_undegraded":"2024-02-13T00:49:54.353019+0000","last_fullsized":"2024-02-13T00:49:54.353019+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14","last_scrub_stamp":"2024-02-13T00:48:32.669406+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.669406+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":310,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527987+0000","last_change":"2024-02-13T00:48:20.928236+0000","last_active":"2024-02-13T00:49:53.527987+0000","last_peered":"2024-02-13T00:49:53.527987+0000","last_clean":"2024-02-13T00:49:53.527987+0000","last_became_active":"2024-02-13T00:46:01.442003+0000","last_became_peered":"2024-02-13T00:46:01.442003+0000","last_unstale":"2024-02-13T00:49:53.527987+0000","last_undegraded":"2024-02-13T00:49:53.527987+0000","last_fullsized":"2024-02-13T00:49:53.527987+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:48:20.928158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:20.928158+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":558,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529358+0000","last_change":"2024-02-13T00:49:33.040619+0000","last_active":"2024-02-13T00:49:53.529358+0000","last_peered":"2024-02-13T00:49:53.529358+0000","last_clean":"2024-02-13T00:49:53.529358+0000","last_became_active":"2024-02-13T00:45:15.422546+0000","last_became_peered":"2024-02-13T00:45:15.422546+0000","last_unstale":"2024-02-13T00:49:53.529358+0000","last_undegraded":"2024-02-13T00:49:53.529358+0000","last_fullsized":"2024-02-13T00:49:53.529358+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:33.040516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:49:33.040516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":569,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.534806+0000","last_change":"2024-02-13T00:46:01.436997+0000","last_active":"2024-02-13T00:49:53.534806+0000","last_peered":"2024-02-13T00:49:53.534806+0000","last_clean":"2024-02-13T00:49:53.534806+0000","last_became_active":"2024-02-13T00:46:01.435178+0000","last_became_peered":"2024-02-13T00:46:01.435178+0000","last_unstale":"2024-02-13T00:49:53.534806+0000","last_undegraded":"2024-02-13T00:49:53.534806+0000","last_fullsized":"2024-02-13T00:49:53.534806+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":502,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527194+0000","last_change":"2024-02-13T00:45:15.445626+0000","last_active":"2024-02-13T00:49:53.527194+0000","last_peered":"2024-02-13T00:49:53.527194+0000","last_clean":"2024-02-13T00:49:53.527194+0000","last_became_active":"2024-02-13T00:45:15.445117+0000","last_became_peered":"2024-02-13T00:45:15.445117+0000","last_unstale":"2024-02-13T00:49:53.527194+0000","last_undegraded":"2024-02-13T00:49:53.527194+0000","last_fullsized":"2024-02-13T00:49:53.527194+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.171185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":287,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535213+0000","last_change":"2024-02-13T00:49:12.252240+0000","last_active":"2024-02-13T00:49:53.535213+0000","last_peered":"2024-02-13T00:49:53.535213+0000","last_clean":"2024-02-13T00:49:53.535213+0000","last_became_active":"2024-02-13T00:49:12.251980+0000","last_became_peered":"2024-02-13T00:49:12.251980+0000","last_unstale":"2024-02-13T00:49:53.535213+0000","last_undegraded":"2024-02-13T00:49:53.535213+0000","last_fullsized":"2024-02-13T00:49:53.535213+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.592320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.592320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":628,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527094+0000","last_change":"2024-02-13T00:49:31.072963+0000","last_active":"2024-02-13T00:49:53.527094+0000","last_peered":"2024-02-13T00:49:53.527094+0000","last_clean":"2024-02-13T00:49:53.527094+0000","last_became_active":"2024-02-13T00:44:39.501886+0000","last_became_peered":"2024-02-13T00:44:39.501886+0000","last_unstale":"2024-02-13T00:49:53.527094+0000","last_undegraded":"2024-02-13T00:49:53.527094+0000","last_fullsized":"2024-02-13T00:49:53.527094+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'30","last_scrub_stamp":"2024-02-13T00:49:31.072862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:31.072862+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":548,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540410+0000","last_change":"2024-02-13T00:48:58.861408+0000","last_active":"2024-02-13T00:49:53.540410+0000","last_peered":"2024-02-13T00:49:53.540410+0000","last_clean":"2024-02-13T00:49:53.540410+0000","last_became_active":"2024-02-13T00:46:01.457926+0000","last_became_peered":"2024-02-13T00:46:01.457926+0000","last_unstale":"2024-02-13T00:49:53.540410+0000","last_undegraded":"2024-02-13T00:49:53.540410+0000","last_fullsized":"2024-02-13T00:49:53.540410+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:48:58.861325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:58.861325+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":353,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528747+0000","last_change":"2024-02-13T00:47:06.895693+0000","last_active":"2024-02-13T00:49:53.528747+0000","last_peered":"2024-02-13T00:49:53.528747+0000","last_clean":"2024-02-13T00:49:53.528747+0000","last_became_active":"2024-02-13T00:45:15.435404+0000","last_became_peered":"2024-02-13T00:45:15.435404+0000","last_unstale":"2024-02-13T00:49:53.528747+0000","last_undegraded":"2024-02-13T00:49:53.528747+0000","last_fullsized":"2024-02-13T00:49:53.528747+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:06.895612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:47:06.895612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":503,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540429+0000","last_change":"2024-02-13T00:46:01.450436+0000","last_active":"2024-02-13T00:49:53.540429+0000","last_peered":"2024-02-13T00:49:53.540429+0000","last_clean":"2024-02-13T00:49:53.540429+0000","last_became_active":"2024-02-13T00:46:01.449234+0000","last_became_peered":"2024-02-13T00:46:01.449234+0000","last_unstale":"2024-02-13T00:49:53.540429+0000","last_undegraded":"2024-02-13T00:49:53.540429+0000","last_fullsized":"2024-02-13T00:49:53.540429+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.110742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.110742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"467'27","reported_seq":696,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540352+0000","last_change":"2024-02-13T00:49:35.834793+0000","last_active":"2024-02-13T00:49:53.540352+0000","last_peered":"2024-02-13T00:49:53.540352+0000","last_clean":"2024-02-13T00:49:53.540352+0000","last_became_active":"2024-02-13T00:46:01.456876+0000","last_became_peered":"2024-02-13T00:46:01.456876+0000","last_unstale":"2024-02-13T00:49:53.540352+0000","last_undegraded":"2024-02-13T00:49:53.540352+0000","last_fullsized":"2024-02-13T00:49:53.540352+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:35.834669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.834669+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":260,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527034+0000","last_change":"2024-02-13T00:46:30.095387+0000","last_active":"2024-02-13T00:49:53.527034+0000","last_peered":"2024-02-13T00:49:53.527034+0000","last_clean":"2024-02-13T00:49:53.527034+0000","last_became_active":"2024-02-13T00:44:50.441300+0000","last_became_peered":"2024-02-13T00:44:50.441300+0000","last_unstale":"2024-02-13T00:49:53.527034+0000","last_undegraded":"2024-02-13T00:49:53.527034+0000","last_fullsized":"2024-02-13T00:49:53.527034+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.095307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":276,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.537446+0000","last_change":"2024-02-13T00:46:01.454597+0000","last_active":"2024-02-13T00:49:53.537446+0000","last_peered":"2024-02-13T00:49:53.537446+0000","last_clean":"2024-02-13T00:49:53.537446+0000","last_became_active":"2024-02-13T00:46:01.454101+0000","last_became_peered":"2024-02-13T00:46:01.454101+0000","last_unstale":"2024-02-13T00:49:53.537446+0000","last_undegraded":"2024-02-13T00:49:53.537446+0000","last_fullsized":"2024-02-13T00:49:53.537446+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.651091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.651091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":272,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.531306+0000","last_change":"2024-02-13T00:44:50.441196+0000","last_active":"2024-02-13T00:49:53.531306+0000","last_peered":"2024-02-13T00:49:53.531306+0000","last_clean":"2024-02-13T00:49:53.531306+0000","last_became_active":"2024-02-13T00:44:50.438274+0000","last_became_peered":"2024-02-13T00:44:50.438274+0000","last_unstale":"2024-02-13T00:49:53.531306+0000","last_undegraded":"2024-02-13T00:49:53.531306+0000","last_fullsized":"2024-02-13T00:49:53.531306+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.510219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.510219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":429,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.531269+0000","last_change":"2024-02-13T00:46:34.141015+0000","last_active":"2024-02-13T00:49:53.531269+0000","last_peered":"2024-02-13T00:49:53.531269+0000","last_clean":"2024-02-13T00:49:53.531269+0000","last_became_active":"2024-02-13T00:45:15.436020+0000","last_became_peered":"2024-02-13T00:45:15.436020+0000","last_unstale":"2024-02-13T00:49:53.531269+0000","last_undegraded":"2024-02-13T00:49:53.531269+0000","last_fullsized":"2024-02-13T00:49:53.531269+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:46:34.140928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.140928+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":268,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528740+0000","last_change":"2024-02-13T00:45:00.311635+0000","last_active":"2024-02-13T00:49:53.528740+0000","last_peered":"2024-02-13T00:49:53.528740+0000","last_clean":"2024-02-13T00:49:53.528740+0000","last_became_active":"2024-02-13T00:44:39.512402+0000","last_became_peered":"2024-02-13T00:44:39.512402+0000","last_unstale":"2024-02-13T00:49:53.528740+0000","last_undegraded":"2024-02-13T00:49:53.528740+0000","last_fullsized":"2024-02-13T00:49:53.528740+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:00.311558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:45:00.311558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":281,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529796+0000","last_change":"2024-02-13T00:46:01.451840+0000","last_active":"2024-02-13T00:49:53.529796+0000","last_peered":"2024-02-13T00:49:53.529796+0000","last_clean":"2024-02-13T00:49:53.529796+0000","last_became_active":"2024-02-13T00:46:01.450312+0000","last_became_peered":"2024-02-13T00:46:01.450312+0000","last_unstale":"2024-02-13T00:49:53.529796+0000","last_undegraded":"2024-02-13T00:49:53.529796+0000","last_fullsized":"2024-02-13T00:49:53.529796+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.042291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.042291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":353,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.538335+0000","last_change":"2024-02-13T00:46:01.454858+0000","last_active":"2024-02-13T00:49:53.538335+0000","last_peered":"2024-02-13T00:49:53.538335+0000","last_clean":"2024-02-13T00:49:53.538335+0000","last_became_active":"2024-02-13T00:46:01.454284+0000","last_became_peered":"2024-02-13T00:46:01.454284+0000","last_unstale":"2024-02-13T00:49:53.538335+0000","last_undegraded":"2024-02-13T00:49:53.538335+0000","last_fullsized":"2024-02-13T00:49:53.538335+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:34.523594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.523594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":512,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529621+0000","last_change":"2024-02-13T00:49:06.154997+0000","last_active":"2024-02-13T00:49:53.529621+0000","last_peered":"2024-02-13T00:49:53.529621+0000","last_clean":"2024-02-13T00:49:53.529621+0000","last_became_active":"2024-02-13T00:46:01.450113+0000","last_became_peered":"2024-02-13T00:46:01.450113+0000","last_unstale":"2024-02-13T00:49:53.529621+0000","last_undegraded":"2024-02-13T00:49:53.529621+0000","last_fullsized":"2024-02-13T00:49:53.529621+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'39","last_scrub_stamp":"2024-02-13T00:49:06.154931+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:49:06.154931+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":528,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.538449+0000","last_change":"2024-02-13T00:49:29.916234+0000","last_active":"2024-02-13T00:49:53.538449+0000","last_peered":"2024-02-13T00:49:53.538449+0000","last_clean":"2024-02-13T00:49:53.538449+0000","last_became_active":"2024-02-13T00:46:01.454310+0000","last_became_peered":"2024-02-13T00:46:01.454310+0000","last_unstale":"2024-02-13T00:49:53.538449+0000","last_undegraded":"2024-02-13T00:49:53.538449+0000","last_fullsized":"2024-02-13T00:49:53.538449+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.916152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"404'9","reported_seq":263,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.352839+0000","last_change":"2024-02-13T00:47:37.013003+0000","last_active":"2024-02-13T00:49:54.352839+0000","last_peered":"2024-02-13T00:49:54.352839+0000","last_clean":"2024-02-13T00:49:54.352839+0000","last_became_active":"2024-02-13T00:46:01.448144+0000","last_became_peered":"2024-02-13T00:46:01.448144+0000","last_unstale":"2024-02-13T00:49:54.352839+0000","last_undegraded":"2024-02-13T00:49:54.352839+0000","last_fullsized":"2024-02-13T00:49:54.352839+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9","last_scrub_stamp":"2024-02-13T00:47:37.012934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:37.012934+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":565,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540236+0000","last_change":"2024-02-13T00:48:26.647460+0000","last_active":"2024-02-13T00:49:53.540236+0000","last_peered":"2024-02-13T00:49:53.540236+0000","last_clean":"2024-02-13T00:49:53.540236+0000","last_became_active":"2024-02-13T00:46:01.447236+0000","last_became_peered":"2024-02-13T00:46:01.447236+0000","last_unstale":"2024-02-13T00:49:53.540236+0000","last_undegraded":"2024-02-13T00:49:53.540236+0000","last_fullsized":"2024-02-13T00:49:53.540236+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:26.647367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:26.647367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":231,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540122+0000","last_change":"2024-02-13T00:46:01.456618+0000","last_active":"2024-02-13T00:49:53.540122+0000","last_peered":"2024-02-13T00:49:53.540122+0000","last_clean":"2024-02-13T00:49:53.540122+0000","last_became_active":"2024-02-13T00:46:01.455989+0000","last_became_peered":"2024-02-13T00:46:01.455989+0000","last_unstale":"2024-02-13T00:49:53.540122+0000","last_undegraded":"2024-02-13T00:49:53.540122+0000","last_fullsized":"2024-02-13T00:49:53.540122+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:16.499780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:16.499780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"467'22","reported_seq":550,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529375+0000","last_change":"2024-02-13T00:47:35.369710+0000","last_active":"2024-02-13T00:49:53.529375+0000","last_peered":"2024-02-13T00:49:53.529375+0000","last_clean":"2024-02-13T00:49:53.529375+0000","last_became_active":"2024-02-13T00:46:01.460321+0000","last_became_peered":"2024-02-13T00:46:01.460321+0000","last_unstale":"2024-02-13T00:49:53.529375+0000","last_undegraded":"2024-02-13T00:49:53.529375+0000","last_fullsized":"2024-02-13T00:49:53.529375+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:47:35.369616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:35.369616+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":580,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.538297+0000","last_change":"2024-02-13T00:49:05.887739+0000","last_active":"2024-02-13T00:49:53.538297+0000","last_peered":"2024-02-13T00:49:53.538297+0000","last_clean":"2024-02-13T00:49:53.538297+0000","last_became_active":"2024-02-13T00:46:01.431926+0000","last_became_peered":"2024-02-13T00:46:01.431926+0000","last_unstale":"2024-02-13T00:49:53.538297+0000","last_undegraded":"2024-02-13T00:49:53.538297+0000","last_fullsized":"2024-02-13T00:49:53.538297+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:05.887665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:05.887665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"363'14","reported_seq":406,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:54.352802+0000","last_change":"2024-02-13T00:47:07.942748+0000","last_active":"2024-02-13T00:49:54.352802+0000","last_peered":"2024-02-13T00:49:54.352802+0000","last_clean":"2024-02-13T00:49:54.352802+0000","last_became_active":"2024-02-13T00:45:15.423066+0000","last_became_peered":"2024-02-13T00:45:15.423066+0000","last_unstale":"2024-02-13T00:49:54.352802+0000","last_undegraded":"2024-02-13T00:49:54.352802+0000","last_fullsized":"2024-02-13T00:49:54.352802+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:47:07.942668+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:47:07.942668+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":199,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540328+0000","last_change":"2024-02-13T00:47:02.789298+0000","last_active":"2024-02-13T00:49:53.540328+0000","last_peered":"2024-02-13T00:49:53.540328+0000","last_clean":"2024-02-13T00:49:53.540328+0000","last_became_active":"2024-02-13T00:46:01.446960+0000","last_became_peered":"2024-02-13T00:46:01.446960+0000","last_unstale":"2024-02-13T00:49:53.540328+0000","last_undegraded":"2024-02-13T00:49:53.540328+0000","last_fullsized":"2024-02-13T00:49:53.540328+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:02.789189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.789189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":437,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529503+0000","last_change":"2024-02-13T00:46:01.450308+0000","last_active":"2024-02-13T00:49:53.529503+0000","last_peered":"2024-02-13T00:49:53.529503+0000","last_clean":"2024-02-13T00:49:53.529503+0000","last_became_active":"2024-02-13T00:46:01.449694+0000","last_became_peered":"2024-02-13T00:46:01.449694+0000","last_unstale":"2024-02-13T00:49:53.529503+0000","last_undegraded":"2024-02-13T00:49:53.529503+0000","last_fullsized":"2024-02-13T00:49:53.529503+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:27.073034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:27.073034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":633,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527849+0000","last_change":"2024-02-13T00:49:38.007273+0000","last_active":"2024-02-13T00:49:53.527849+0000","last_peered":"2024-02-13T00:49:53.527849+0000","last_clean":"2024-02-13T00:49:53.527849+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:49:53.527849+0000","last_undegraded":"2024-02-13T00:49:53.527849+0000","last_fullsized":"2024-02-13T00:49:53.527849+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'19","last_scrub_stamp":"2024-02-13T00:49:38.007204+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:49:38.007204+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":281,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530005+0000","last_change":"2024-02-13T00:46:01.463310+0000","last_active":"2024-02-13T00:49:53.530005+0000","last_peered":"2024-02-13T00:49:53.530005+0000","last_clean":"2024-02-13T00:49:53.530005+0000","last_became_active":"2024-02-13T00:46:01.462898+0000","last_became_peered":"2024-02-13T00:46:01.462898+0000","last_unstale":"2024-02-13T00:49:53.530005+0000","last_undegraded":"2024-02-13T00:49:53.530005+0000","last_fullsized":"2024-02-13T00:49:53.530005+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.095888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":344,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.537390+0000","last_change":"2024-02-13T00:46:32.638418+0000","last_active":"2024-02-13T00:49:53.537390+0000","last_peered":"2024-02-13T00:49:53.537390+0000","last_clean":"2024-02-13T00:49:53.537390+0000","last_became_active":"2024-02-13T00:46:01.443279+0000","last_became_peered":"2024-02-13T00:46:01.443279+0000","last_unstale":"2024-02-13T00:49:53.537390+0000","last_undegraded":"2024-02-13T00:49:53.537390+0000","last_fullsized":"2024-02-13T00:49:53.537390+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.638349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":547,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.533953+0000","last_change":"2024-02-13T00:48:27.659266+0000","last_active":"2024-02-13T00:49:53.533953+0000","last_peered":"2024-02-13T00:49:53.533953+0000","last_clean":"2024-02-13T00:49:53.533953+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:49:53.533953+0000","last_undegraded":"2024-02-13T00:49:53.533953+0000","last_fullsized":"2024-02-13T00:49:53.533953+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:27.659197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:48:27.659197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":277,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528627+0000","last_change":"2024-02-13T00:45:40.233264+0000","last_active":"2024-02-13T00:49:53.528627+0000","last_peered":"2024-02-13T00:49:53.528627+0000","last_clean":"2024-02-13T00:49:53.528627+0000","last_became_active":"2024-02-13T00:45:15.447051+0000","last_became_peered":"2024-02-13T00:45:15.447051+0000","last_unstale":"2024-02-13T00:49:53.528627+0000","last_undegraded":"2024-02-13T00:49:53.528627+0000","last_fullsized":"2024-02-13T00:49:53.528627+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:40.233185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:40.233185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":313,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528566+0000","last_change":"2024-02-13T00:48:34.458358+0000","last_active":"2024-02-13T00:49:53.528566+0000","last_peered":"2024-02-13T00:49:53.528566+0000","last_clean":"2024-02-13T00:49:53.528566+0000","last_became_active":"2024-02-13T00:44:50.435203+0000","last_became_peered":"2024-02-13T00:44:50.435203+0000","last_unstale":"2024-02-13T00:49:53.528566+0000","last_undegraded":"2024-02-13T00:49:53.528566+0000","last_fullsized":"2024-02-13T00:49:53.528566+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:48:34.458264+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:48:34.458264+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":553,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529441+0000","last_change":"2024-02-13T00:46:01.457220+0000","last_active":"2024-02-13T00:49:53.529441+0000","last_peered":"2024-02-13T00:49:53.529441+0000","last_clean":"2024-02-13T00:49:53.529441+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:49:53.529441+0000","last_undegraded":"2024-02-13T00:49:53.529441+0000","last_fullsized":"2024-02-13T00:49:53.529441+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.097481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":567,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527924+0000","last_change":"2024-02-13T00:46:01.453391+0000","last_active":"2024-02-13T00:49:53.527924+0000","last_peered":"2024-02-13T00:49:53.527924+0000","last_clean":"2024-02-13T00:49:53.527924+0000","last_became_active":"2024-02-13T00:46:01.452730+0000","last_became_peered":"2024-02-13T00:46:01.452730+0000","last_unstale":"2024-02-13T00:49:53.527924+0000","last_undegraded":"2024-02-13T00:49:53.527924+0000","last_fullsized":"2024-02-13T00:49:53.527924+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:26.218807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:44:26.218807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":353,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535536+0000","last_change":"2024-02-13T00:48:29.681480+0000","last_active":"2024-02-13T00:49:53.535536+0000","last_peered":"2024-02-13T00:49:53.535536+0000","last_clean":"2024-02-13T00:49:53.535536+0000","last_became_active":"2024-02-13T00:46:01.454287+0000","last_became_peered":"2024-02-13T00:46:01.454287+0000","last_unstale":"2024-02-13T00:49:53.535536+0000","last_undegraded":"2024-02-13T00:49:53.535536+0000","last_fullsized":"2024-02-13T00:49:53.535536+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:29.681414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:29.681414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":286,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527559+0000","last_change":"2024-02-13T00:49:02.260116+0000","last_active":"2024-02-13T00:49:53.527559+0000","last_peered":"2024-02-13T00:49:53.527559+0000","last_clean":"2024-02-13T00:49:53.527559+0000","last_became_active":"2024-02-13T00:46:01.448671+0000","last_became_peered":"2024-02-13T00:46:01.448671+0000","last_unstale":"2024-02-13T00:49:53.527559+0000","last_undegraded":"2024-02-13T00:49:53.527559+0000","last_fullsized":"2024-02-13T00:49:53.527559+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:02.260045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:02.260045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":380,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527307+0000","last_change":"2024-02-13T00:44:40.371566+0000","last_active":"2024-02-13T00:49:53.527307+0000","last_peered":"2024-02-13T00:49:53.527307+0000","last_clean":"2024-02-13T00:49:53.527307+0000","last_became_active":"2024-02-13T00:44:39.501040+0000","last_became_peered":"2024-02-13T00:44:39.501040+0000","last_unstale":"2024-02-13T00:49:53.527307+0000","last_undegraded":"2024-02-13T00:49:53.527307+0000","last_fullsized":"2024-02-13T00:49:53.527307+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:44:32.180113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:44:32.180113+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":512,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529711+0000","last_change":"2024-02-13T00:48:32.441279+0000","last_active":"2024-02-13T00:49:53.529711+0000","last_peered":"2024-02-13T00:49:53.529711+0000","last_clean":"2024-02-13T00:49:53.529711+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:49:53.529711+0000","last_undegraded":"2024-02-13T00:49:53.529711+0000","last_fullsized":"2024-02-13T00:49:53.529711+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.441170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.441170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":277,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528419+0000","last_change":"2024-02-13T00:48:28.353456+0000","last_active":"2024-02-13T00:49:53.528419+0000","last_peered":"2024-02-13T00:49:53.528419+0000","last_clean":"2024-02-13T00:49:53.528419+0000","last_became_active":"2024-02-13T00:45:15.435820+0000","last_became_peered":"2024-02-13T00:45:15.435820+0000","last_unstale":"2024-02-13T00:49:53.528419+0000","last_undegraded":"2024-02-13T00:49:53.528419+0000","last_fullsized":"2024-02-13T00:49:53.528419+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:28.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:28.353347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":267,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527275+0000","last_change":"2024-02-13T00:49:29.052728+0000","last_active":"2024-02-13T00:49:53.527275+0000","last_peered":"2024-02-13T00:49:53.527275+0000","last_clean":"2024-02-13T00:49:53.527275+0000","last_became_active":"2024-02-13T00:45:15.444818+0000","last_became_peered":"2024-02-13T00:45:15.444818+0000","last_unstale":"2024-02-13T00:49:53.527275+0000","last_undegraded":"2024-02-13T00:49:53.527275+0000","last_fullsized":"2024-02-13T00:49:53.527275+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.052640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.052640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":428,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528496+0000","last_change":"2024-02-13T00:46:01.439915+0000","last_active":"2024-02-13T00:49:53.528496+0000","last_peered":"2024-02-13T00:49:53.528496+0000","last_clean":"2024-02-13T00:49:53.528496+0000","last_became_active":"2024-02-13T00:46:01.438745+0000","last_became_peered":"2024-02-13T00:46:01.438745+0000","last_unstale":"2024-02-13T00:49:53.528496+0000","last_undegraded":"2024-02-13T00:49:53.528496+0000","last_fullsized":"2024-02-13T00:49:53.528496+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.104906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.104906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"467'25","reported_seq":340,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528584+0000","last_change":"2024-02-13T00:47:36.368967+0000","last_active":"2024-02-13T00:49:53.528584+0000","last_peered":"2024-02-13T00:49:53.528584+0000","last_clean":"2024-02-13T00:49:53.528584+0000","last_became_active":"2024-02-13T00:46:01.450079+0000","last_became_peered":"2024-02-13T00:46:01.450079+0000","last_unstale":"2024-02-13T00:49:53.528584+0000","last_undegraded":"2024-02-13T00:49:53.528584+0000","last_fullsized":"2024-02-13T00:49:53.528584+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:47:36.368863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.368863+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":286,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.531218+0000","last_change":"2024-02-13T00:46:59.388809+0000","last_active":"2024-02-13T00:49:53.531218+0000","last_peered":"2024-02-13T00:49:53.531218+0000","last_clean":"2024-02-13T00:49:53.531218+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:49:53.531218+0000","last_undegraded":"2024-02-13T00:49:53.531218+0000","last_fullsized":"2024-02-13T00:49:53.531218+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:59.388745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:59.388745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":272,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527385+0000","last_change":"2024-02-13T00:45:15.439752+0000","last_active":"2024-02-13T00:49:53.527385+0000","last_peered":"2024-02-13T00:49:53.527385+0000","last_clean":"2024-02-13T00:49:53.527385+0000","last_became_active":"2024-02-13T00:45:15.437943+0000","last_became_peered":"2024-02-13T00:45:15.437943+0000","last_unstale":"2024-02-13T00:49:53.527385+0000","last_undegraded":"2024-02-13T00:49:53.527385+0000","last_fullsized":"2024-02-13T00:49:53.527385+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.284740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.284740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":480,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.537363+0000","last_change":"2024-02-13T00:44:58.969806+0000","last_active":"2024-02-13T00:49:53.537363+0000","last_peered":"2024-02-13T00:49:53.537363+0000","last_clean":"2024-02-13T00:49:53.537363+0000","last_became_active":"2024-02-13T00:44:50.439512+0000","last_became_peered":"2024-02-13T00:44:50.439512+0000","last_unstale":"2024-02-13T00:49:53.537363+0000","last_undegraded":"2024-02-13T00:49:53.537363+0000","last_fullsized":"2024-02-13T00:49:53.537363+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:58.969692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:44:58.969692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":400,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529831+0000","last_change":"2024-02-13T00:49:35.101413+0000","last_active":"2024-02-13T00:49:53.529831+0000","last_peered":"2024-02-13T00:49:53.529831+0000","last_clean":"2024-02-13T00:49:53.529831+0000","last_became_active":"2024-02-13T00:45:15.430749+0000","last_became_peered":"2024-02-13T00:45:15.430749+0000","last_unstale":"2024-02-13T00:49:53.529831+0000","last_undegraded":"2024-02-13T00:49:53.529831+0000","last_fullsized":"2024-02-13T00:49:53.529831+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:35.101298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.101298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":669,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540535+0000","last_change":"2024-02-13T00:48:53.841888+0000","last_active":"2024-02-13T00:49:53.540535+0000","last_peered":"2024-02-13T00:49:53.540535+0000","last_clean":"2024-02-13T00:49:53.540535+0000","last_became_active":"2024-02-13T00:46:01.443721+0000","last_became_peered":"2024-02-13T00:46:01.443721+0000","last_unstale":"2024-02-13T00:49:53.540535+0000","last_undegraded":"2024-02-13T00:49:53.540535+0000","last_fullsized":"2024-02-13T00:49:53.540535+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'32","last_scrub_stamp":"2024-02-13T00:48:53.841797+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.841797+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":514,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.526725+0000","last_change":"2024-02-13T00:46:32.298033+0000","last_active":"2024-02-13T00:49:53.526725+0000","last_peered":"2024-02-13T00:49:53.526725+0000","last_clean":"2024-02-13T00:49:53.526725+0000","last_became_active":"2024-02-13T00:46:01.442231+0000","last_became_peered":"2024-02-13T00:46:01.442231+0000","last_unstale":"2024-02-13T00:49:53.526725+0000","last_undegraded":"2024-02-13T00:49:53.526725+0000","last_fullsized":"2024-02-13T00:49:53.526725+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.297918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.297918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":378,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528919+0000","last_change":"2024-02-13T00:48:33.460377+0000","last_active":"2024-02-13T00:49:53.528919+0000","last_peered":"2024-02-13T00:49:53.528919+0000","last_clean":"2024-02-13T00:49:53.528919+0000","last_became_active":"2024-02-13T00:46:01.439984+0000","last_became_peered":"2024-02-13T00:46:01.439984+0000","last_unstale":"2024-02-13T00:49:53.528919+0000","last_undegraded":"2024-02-13T00:49:53.528919+0000","last_fullsized":"2024-02-13T00:49:53.528919+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:48:33.460247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:33.460247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":511,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.531221+0000","last_change":"2024-02-13T00:45:38.084426+0000","last_active":"2024-02-13T00:49:53.531221+0000","last_peered":"2024-02-13T00:49:53.531221+0000","last_clean":"2024-02-13T00:49:53.531221+0000","last_became_active":"2024-02-13T00:45:15.435550+0000","last_became_peered":"2024-02-13T00:45:15.435550+0000","last_unstale":"2024-02-13T00:49:53.531221+0000","last_undegraded":"2024-02-13T00:49:53.531221+0000","last_fullsized":"2024-02-13T00:49:53.531221+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.084358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.084358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":317,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.531441+0000","last_change":"2024-02-13T00:48:55.424021+0000","last_active":"2024-02-13T00:49:53.531441+0000","last_peered":"2024-02-13T00:49:53.531441+0000","last_clean":"2024-02-13T00:49:53.531441+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:49:53.531441+0000","last_undegraded":"2024-02-13T00:49:53.531441+0000","last_fullsized":"2024-02-13T00:49:53.531441+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:55.423952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.423952+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":286,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528664+0000","last_change":"2024-02-13T00:46:56.339969+0000","last_active":"2024-02-13T00:49:53.528664+0000","last_peered":"2024-02-13T00:49:53.528664+0000","last_clean":"2024-02-13T00:49:53.528664+0000","last_became_active":"2024-02-13T00:45:15.436239+0000","last_became_peered":"2024-02-13T00:45:15.436239+0000","last_unstale":"2024-02-13T00:49:53.528664+0000","last_undegraded":"2024-02-13T00:49:53.528664+0000","last_fullsized":"2024-02-13T00:49:53.528664+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:56.339885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.339885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":552,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529309+0000","last_change":"2024-02-13T00:49:46.928723+0000","last_active":"2024-02-13T00:49:53.529309+0000","last_peered":"2024-02-13T00:49:53.529309+0000","last_clean":"2024-02-13T00:49:53.529309+0000","last_became_active":"2024-02-13T00:46:01.453269+0000","last_became_peered":"2024-02-13T00:46:01.453269+0000","last_unstale":"2024-02-13T00:49:53.529309+0000","last_undegraded":"2024-02-13T00:49:53.529309+0000","last_fullsized":"2024-02-13T00:49:53.529309+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.928651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.928651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":277,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530185+0000","last_change":"2024-02-13T00:45:37.280947+0000","last_active":"2024-02-13T00:49:53.530185+0000","last_peered":"2024-02-13T00:49:53.530185+0000","last_clean":"2024-02-13T00:49:53.530185+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:49:53.530185+0000","last_undegraded":"2024-02-13T00:49:53.530185+0000","last_fullsized":"2024-02-13T00:49:53.530185+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.280865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.280865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":331,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529507+0000","last_change":"2024-02-13T00:49:30.086932+0000","last_active":"2024-02-13T00:49:53.529507+0000","last_peered":"2024-02-13T00:49:53.529507+0000","last_clean":"2024-02-13T00:49:53.529507+0000","last_became_active":"2024-02-13T00:44:50.442894+0000","last_became_peered":"2024-02-13T00:44:50.442894+0000","last_unstale":"2024-02-13T00:49:53.529507+0000","last_undegraded":"2024-02-13T00:49:53.529507+0000","last_fullsized":"2024-02-13T00:49:53.529507+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:30.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:30.086856+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":277,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529718+0000","last_change":"2024-02-13T00:49:04.282594+0000","last_active":"2024-02-13T00:49:53.529718+0000","last_peered":"2024-02-13T00:49:53.529718+0000","last_clean":"2024-02-13T00:49:53.529718+0000","last_became_active":"2024-02-13T00:45:15.445403+0000","last_became_peered":"2024-02-13T00:45:15.445403+0000","last_unstale":"2024-02-13T00:49:53.529718+0000","last_undegraded":"2024-02-13T00:49:53.529718+0000","last_fullsized":"2024-02-13T00:49:53.529718+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:04.282529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:04.282529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":277,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.530033+0000","last_change":"2024-02-13T00:47:42.329463+0000","last_active":"2024-02-13T00:49:53.530033+0000","last_peered":"2024-02-13T00:49:53.530033+0000","last_clean":"2024-02-13T00:49:53.530033+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:49:53.530033+0000","last_undegraded":"2024-02-13T00:49:53.530033+0000","last_fullsized":"2024-02-13T00:49:53.530033+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:42.329367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:42.329367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":186,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529424+0000","last_change":"2024-02-13T00:46:01.451610+0000","last_active":"2024-02-13T00:49:53.529424+0000","last_peered":"2024-02-13T00:49:53.529424+0000","last_clean":"2024-02-13T00:49:53.529424+0000","last_became_active":"2024-02-13T00:46:01.449873+0000","last_became_peered":"2024-02-13T00:46:01.449873+0000","last_unstale":"2024-02-13T00:49:53.529424+0000","last_undegraded":"2024-02-13T00:49:53.529424+0000","last_fullsized":"2024-02-13T00:49:53.529424+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.554065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.554065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":432,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529754+0000","last_change":"2024-02-13T00:46:56.973071+0000","last_active":"2024-02-13T00:49:53.529754+0000","last_peered":"2024-02-13T00:49:53.529754+0000","last_clean":"2024-02-13T00:49:53.529754+0000","last_became_active":"2024-02-13T00:44:39.504670+0000","last_became_peered":"2024-02-13T00:44:39.504670+0000","last_unstale":"2024-02-13T00:49:53.529754+0000","last_undegraded":"2024-02-13T00:49:53.529754+0000","last_fullsized":"2024-02-13T00:49:53.529754+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:46:56.973000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.973000+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":281,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528617+0000","last_change":"2024-02-13T00:46:01.448134+0000","last_active":"2024-02-13T00:49:53.528617+0000","last_peered":"2024-02-13T00:49:53.528617+0000","last_clean":"2024-02-13T00:49:53.528617+0000","last_became_active":"2024-02-13T00:46:01.446635+0000","last_became_peered":"2024-02-13T00:46:01.446635+0000","last_unstale":"2024-02-13T00:49:53.528617+0000","last_undegraded":"2024-02-13T00:49:53.528617+0000","last_fullsized":"2024-02-13T00:49:53.528617+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:28.101254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:28.101254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":277,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.531194+0000","last_change":"2024-02-13T00:46:01.461188+0000","last_active":"2024-02-13T00:49:53.531194+0000","last_peered":"2024-02-13T00:49:53.531194+0000","last_clean":"2024-02-13T00:49:53.531194+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:49:53.531194+0000","last_undegraded":"2024-02-13T00:49:53.531194+0000","last_fullsized":"2024-02-13T00:49:53.531194+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.238152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.238152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":521,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.538306+0000","last_change":"2024-02-13T00:46:01.443734+0000","last_active":"2024-02-13T00:49:53.538306+0000","last_peered":"2024-02-13T00:49:53.538306+0000","last_clean":"2024-02-13T00:49:53.538306+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:49:53.538306+0000","last_undegraded":"2024-02-13T00:49:53.538306+0000","last_fullsized":"2024-02-13T00:49:53.538306+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.621170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.621170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":426,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.538233+0000","last_change":"2024-02-13T00:46:27.706096+0000","last_active":"2024-02-13T00:49:53.538233+0000","last_peered":"2024-02-13T00:49:53.538233+0000","last_clean":"2024-02-13T00:49:53.538233+0000","last_became_active":"2024-02-13T00:44:50.415894+0000","last_became_peered":"2024-02-13T00:44:50.415894+0000","last_unstale":"2024-02-13T00:49:53.538233+0000","last_undegraded":"2024-02-13T00:49:53.538233+0000","last_fullsized":"2024-02-13T00:49:53.538233+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:46:27.706006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:27.706006+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":361,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529545+0000","last_change":"2024-02-13T00:49:43.067713+0000","last_active":"2024-02-13T00:49:53.529545+0000","last_peered":"2024-02-13T00:49:53.529545+0000","last_clean":"2024-02-13T00:49:53.529545+0000","last_became_active":"2024-02-13T00:46:01.451790+0000","last_became_peered":"2024-02-13T00:46:01.451790+0000","last_unstale":"2024-02-13T00:49:53.529545+0000","last_undegraded":"2024-02-13T00:49:53.529545+0000","last_fullsized":"2024-02-13T00:49:53.529545+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:43.067615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:43.067615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":549,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535210+0000","last_change":"2024-02-13T00:47:36.885611+0000","last_active":"2024-02-13T00:49:53.535210+0000","last_peered":"2024-02-13T00:49:53.535210+0000","last_clean":"2024-02-13T00:49:53.535210+0000","last_became_active":"2024-02-13T00:45:15.419668+0000","last_became_peered":"2024-02-13T00:45:15.419668+0000","last_unstale":"2024-02-13T00:49:53.535210+0000","last_undegraded":"2024-02-13T00:49:53.535210+0000","last_fullsized":"2024-02-13T00:49:53.535210+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.885510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.885510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":455,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.538553+0000","last_change":"2024-02-13T00:46:01.457482+0000","last_active":"2024-02-13T00:49:53.538553+0000","last_peered":"2024-02-13T00:49:53.538553+0000","last_clean":"2024-02-13T00:49:53.538553+0000","last_became_active":"2024-02-13T00:46:01.455940+0000","last_became_peered":"2024-02-13T00:46:01.455940+0000","last_unstale":"2024-02-13T00:49:53.538553+0000","last_undegraded":"2024-02-13T00:49:53.538553+0000","last_fullsized":"2024-02-13T00:49:53.538553+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:21.087199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:21.087199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":440,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528568+0000","last_change":"2024-02-13T00:46:36.089483+0000","last_active":"2024-02-13T00:49:53.528568+0000","last_peered":"2024-02-13T00:49:53.528568+0000","last_clean":"2024-02-13T00:49:53.528568+0000","last_became_active":"2024-02-13T00:45:15.419364+0000","last_became_peered":"2024-02-13T00:45:15.419364+0000","last_unstale":"2024-02-13T00:49:53.528568+0000","last_undegraded":"2024-02-13T00:49:53.528568+0000","last_fullsized":"2024-02-13T00:49:53.528568+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:46:36.089402+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:46:36.089402+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":286,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.529848+0000","last_change":"2024-02-13T00:47:47.284092+0000","last_active":"2024-02-13T00:49:53.529848+0000","last_peered":"2024-02-13T00:49:53.529848+0000","last_clean":"2024-02-13T00:49:53.529848+0000","last_became_active":"2024-02-13T00:46:01.451997+0000","last_became_peered":"2024-02-13T00:46:01.451997+0000","last_unstale":"2024-02-13T00:49:53.529848+0000","last_undegraded":"2024-02-13T00:49:53.529848+0000","last_fullsized":"2024-02-13T00:49:53.529848+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.284014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.284014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":497,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527470+0000","last_change":"2024-02-13T00:49:01.133798+0000","last_active":"2024-02-13T00:49:53.527470+0000","last_peered":"2024-02-13T00:49:53.527470+0000","last_clean":"2024-02-13T00:49:53.527470+0000","last_became_active":"2024-02-13T00:46:01.445141+0000","last_became_peered":"2024-02-13T00:46:01.445141+0000","last_unstale":"2024-02-13T00:49:53.527470+0000","last_undegraded":"2024-02-13T00:49:53.527470+0000","last_fullsized":"2024-02-13T00:49:53.527470+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:01.133709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:49:01.133709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":268,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527582+0000","last_change":"2024-02-13T00:45:30.186208+0000","last_active":"2024-02-13T00:49:53.527582+0000","last_peered":"2024-02-13T00:49:53.527582+0000","last_clean":"2024-02-13T00:49:53.527582+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:49:53.527582+0000","last_undegraded":"2024-02-13T00:49:53.527582+0000","last_fullsized":"2024-02-13T00:49:53.527582+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:30.186129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:30.186129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":326,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527649+0000","last_change":"2024-02-13T00:46:06.251713+0000","last_active":"2024-02-13T00:49:53.527649+0000","last_peered":"2024-02-13T00:49:53.527649+0000","last_clean":"2024-02-13T00:49:53.527649+0000","last_became_active":"2024-02-13T00:46:01.442464+0000","last_became_peered":"2024-02-13T00:46:01.442464+0000","last_unstale":"2024-02-13T00:49:53.527649+0000","last_undegraded":"2024-02-13T00:49:53.527649+0000","last_fullsized":"2024-02-13T00:49:53.527649+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:43:44.241358+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:43:44.241358+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":362,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540441+0000","last_change":"2024-02-13T00:47:47.711551+0000","last_active":"2024-02-13T00:49:53.540441+0000","last_peered":"2024-02-13T00:49:53.540441+0000","last_clean":"2024-02-13T00:49:53.540441+0000","last_became_active":"2024-02-13T00:46:01.447654+0000","last_became_peered":"2024-02-13T00:46:01.447654+0000","last_unstale":"2024-02-13T00:49:53.540441+0000","last_undegraded":"2024-02-13T00:49:53.540441+0000","last_fullsized":"2024-02-13T00:49:53.540441+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.711486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.711486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":196,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527469+0000","last_change":"2024-02-13T00:47:53.826039+0000","last_active":"2024-02-13T00:49:53.527469+0000","last_peered":"2024-02-13T00:49:53.527469+0000","last_clean":"2024-02-13T00:49:53.527469+0000","last_became_active":"2024-02-13T00:46:01.448971+0000","last_became_peered":"2024-02-13T00:46:01.448971+0000","last_unstale":"2024-02-13T00:49:53.527469+0000","last_undegraded":"2024-02-13T00:49:53.527469+0000","last_fullsized":"2024-02-13T00:49:53.527469+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:53.825971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:53.825971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":523,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.527700+0000","last_change":"2024-02-13T00:45:32.233265+0000","last_active":"2024-02-13T00:49:53.527700+0000","last_peered":"2024-02-13T00:49:53.527700+0000","last_clean":"2024-02-13T00:49:53.527700+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:49:53.527700+0000","last_undegraded":"2024-02-13T00:49:53.527700+0000","last_fullsized":"2024-02-13T00:49:53.527700+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:45:32.233197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.233197+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":461,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.535588+0000","last_change":"2024-02-13T00:47:44.757362+0000","last_active":"2024-02-13T00:49:53.535588+0000","last_peered":"2024-02-13T00:49:53.535588+0000","last_clean":"2024-02-13T00:49:53.535588+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:49:53.535588+0000","last_undegraded":"2024-02-13T00:49:53.535588+0000","last_fullsized":"2024-02-13T00:49:53.535588+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:47:44.757262+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:47:44.757262+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":553,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.540430+0000","last_change":"2024-02-13T00:48:54.818104+0000","last_active":"2024-02-13T00:49:53.540430+0000","last_peered":"2024-02-13T00:49:53.540430+0000","last_clean":"2024-02-13T00:49:53.540430+0000","last_became_active":"2024-02-13T00:45:15.434224+0000","last_became_peered":"2024-02-13T00:45:15.434224+0000","last_unstale":"2024-02-13T00:49:53.540430+0000","last_undegraded":"2024-02-13T00:49:53.540430+0000","last_fullsized":"2024-02-13T00:49:53.540430+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:54.817999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.817999+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":272,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-02-13T00:49:53.528664+0000","last_change":"2024-02-13T00:45:15.435955+0000","last_active":"2024-02-13T00:49:53.528664+0000","last_peered":"2024-02-13T00:49:53.528664+0000","last_clean":"2024-02-13T00:49:53.528664+0000","last_became_active":"2024-02-13T00:45:15.435259+0000","last_became_peered":"2024-02-13T00:45:15.435259+0000","last_unstale":"2024-02-13T00:49:53.528664+0000","last_undegraded":"2024-02-13T00:49:53.528664+0000","last_fullsized":"2024-02-13T00:49:53.528664+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.953989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.953989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":233,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":3},{"poolid":232,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":229,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":24,"acting":24,"num_store_stats":4},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1492,"num_read_kb":1217,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":9606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":11656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":970,"ondisk_log_size":970,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":721,"num_read_kb":716,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":227633266837,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":45,"seq":193273528475,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308096,"kb_used_data":2680,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437056,"statfs":{"total":95995035648,"available":95679545344,"internally_reserved":0,"allocated":2744320,"data_stored":1247986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.46700000000000003,"15min":0.54600000000000004},"min":{"1min":0.316,"5min":0.25600000000000001,"15min":0.19700000000000001},"max":{"1min":0.65600000000000003,"5min":0.77500000000000002,"15min":8.1340000000000003},"last":0.373},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52100000000000002,"15min":0.60999999999999999},"min":{"1min":0.34100000000000003,"5min":0.23100000000000001,"15min":0.221},"max":{"1min":0.72799999999999998,"5min":4.2910000000000004,"15min":11.367000000000001},"last":0.42999999999999999}]},{"osd":1,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51200000000000001,"15min":0.88400000000000001},"min":{"1min":0.34899999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.63200000000000001,"5min":4.5720000000000001,"15min":54.002000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52300000000000002,"15min":0.68100000000000005},"min":{"1min":0.28999999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66800000000000004,"5min":5.4989999999999997,"15min":12.403},"last":0.53300000000000003}]},{"osd":2,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52300000000000002,"15min":0.71099999999999997},"min":{"1min":0.31900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":2.8399999999999999,"15min":12.477},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50700000000000001,"15min":0.68000000000000005},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.63200000000000001,"5min":2.1659999999999999,"15min":11.702},"last":0.39000000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.61499999999999999,"15min":0.68100000000000005},"min":{"1min":0.35299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.65700000000000003,"5min":5.931,"15min":8.0519999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54400000000000004,"15min":0.77200000000000002},"min":{"1min":0.44,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":0.65400000000000003,"5min":2.8719999999999999,"15min":19.081},"last":0.58999999999999997}]},{"osd":4,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.41499999999999998},"min":{"1min":0.29099999999999998,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":0.51400000000000001,"5min":0.71899999999999997,"15min":1.0649999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.441,"15min":0.438},"min":{"1min":0.315,"5min":0.22800000000000001,"15min":0.221},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.51400000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.69899999999999995}]},{"osd":7,"last update":"Tue Feb 13 00:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45800000000000002,"15min":0.48299999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61399999999999999,"5min":0.66800000000000004,"15min":1.173},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50800000000000001,"15min":0.53000000000000003},"min":{"1min":0.30099999999999999,"5min":0.28299999999999997,"15min":0.26900000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":1.262},"last":0.61299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575028,"num_pgs":138,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304032,"kb_used_data":2712,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441120,"statfs":{"total":95995035648,"available":95683706880,"internally_reserved":0,"allocated":2777088,"data_stored":1257975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34399999999999997,"15min":0.95099999999999996},"min":{"1min":0.20699999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.52700000000000002,"5min":1.1080000000000001,"15min":61.856000000000002},"last":0.442},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33600000000000002,"15min":0.66800000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.16500000000000001},"max":{"1min":0.70199999999999996,"5min":1.052,"15min":61.908999999999999},"last":0.38400000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34200000000000003,"15min":0.84699999999999998},"min":{"1min":0.251,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":0.40799999999999997,"5min":0.90100000000000002,"15min":32.161000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.377,"15min":0.82799999999999996},"min":{"1min":0.249,"5min":0.187,"15min":0.187},"max":{"1min":0.54000000000000004,"5min":1.9350000000000001,"15min":29.693000000000001},"last":0.41699999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.376,"15min":1.2669999999999999},"min":{"1min":0.28100000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.55400000000000005,"5min":1.0900000000000001,"15min":131.23099999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.39700000000000002,"15min":0.86099999999999999},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.48099999999999998,"5min":1.214,"15min":49.055},"last":0.36499999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.75800000000000001},"min":{"1min":0.36199999999999999,"5min":0.33500000000000002,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":1.246,"15min":16.547000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.56399999999999995,"15min":0.71199999999999997},"min":{"1min":0.42499999999999999,"5min":0.34100000000000003,"15min":0.24199999999999999},"max":{"1min":0.85199999999999998,"5min":0.89200000000000002,"15min":14.731999999999999},"last":0.621}]},{"osd":5,"last update":"Tue Feb 13 00:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.66100000000000003}]},{"osd":6,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54700000000000004,"15min":0.71799999999999997},"min":{"1min":0.41299999999999998,"5min":0.29699999999999999,"15min":0.29099999999999998},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":14.772},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.54600000000000004,"15min":0.92500000000000004},"min":{"1min":0.40500000000000003,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":26.600000000000001},"last":0.60699999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54800000000000004,"15min":0.71999999999999997},"min":{"1min":0.42999999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76900000000000002,"5min":0.95899999999999996,"15min":14.794},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56200000000000006,"15min":0.92800000000000005},"min":{"1min":0.38500000000000001,"5min":0.33000000000000002,"15min":0.32200000000000001},"max":{"1min":0.77900000000000003,"5min":0.80100000000000005,"15min":26.693999999999999},"last":0.64100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738552,"num_pgs":132,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304008,"kb_used_data":2688,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441144,"statfs":{"total":95995035648,"available":95683731456,"internally_reserved":0,"allocated":2752512,"data_stored":1257584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.378,"15min":0.621},"min":{"1min":0.25,"5min":0.191,"15min":0.189},"max":{"1min":0.59999999999999998,"5min":2.0379999999999998,"15min":30.184000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.31,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.17599999999999999},"max":{"1min":0.52300000000000002,"5min":5.6959999999999997,"15min":5.6959999999999997},"last":0.45500000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.51800000000000002,"15min":0.499},"min":{"1min":0.251,"5min":0.245,"15min":0.20999999999999999},"max":{"1min":0.46999999999999997,"5min":7.0839999999999996,"15min":7.0839999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.47399999999999998,"15min":0.66900000000000004},"min":{"1min":0.27200000000000002,"5min":0.248,"15min":0.185},"max":{"1min":0.48599999999999999,"5min":3.6949999999999998,"15min":24.459},"last":0.442}]},{"osd":3,"last update":"Tue Feb 13 00:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.47499999999999998,"15min":0.48599999999999999},"min":{"1min":0.28699999999999998,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":0.628,"5min":4.8710000000000004,"15min":8.718},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.28299999999999997,"5min":0.26900000000000002,"15min":0.222},"max":{"1min":0.67200000000000004,"5min":7.1139999999999999,"15min":7.1139999999999999},"last":0.432}]},{"osd":4,"last update":"Tue Feb 13 00:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.64200000000000002,"15min":0.79400000000000004},"min":{"1min":0.38500000000000001,"5min":0.29899999999999999,"15min":0.24399999999999999},"max":{"1min":0.67300000000000004,"5min":7.056,"15min":24.381},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.68999999999999995,"15min":0.65300000000000002},"min":{"1min":0.40899999999999997,"5min":0.31,"15min":0.28699999999999998},"max":{"1min":0.63100000000000001,"5min":6.1859999999999999,"15min":8.6950000000000003},"last":0.63800000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.66500000000000004}]},{"osd":6,"last update":"Tue Feb 13 00:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.35999999999999999,"5min":0.34100000000000003,"15min":0.31},"max":{"1min":0.79500000000000004,"5min":3.302,"15min":8.6129999999999995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.70999999999999996,"15min":0.65100000000000002},"min":{"1min":0.34000000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.89200000000000002,"5min":14.263,"15min":14.263},"last":0.67500000000000004}]},{"osd":7,"last update":"Tue Feb 13 00:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.69099999999999995,"15min":0.65800000000000003},"min":{"1min":0.39700000000000002,"5min":0.35999999999999999,"15min":0.317},"max":{"1min":0.68600000000000005,"5min":7.7720000000000002,"15min":8.6679999999999993},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.747,"15min":0.66900000000000004},"min":{"1min":0.40400000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3700000000000001,"5min":8.1609999999999996,"15min":8.1609999999999996},"last":0.64900000000000002}]}]},{"osd":2,"up_from":0,"seq":77309411503,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182569,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986341,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299888,"kb_used_data":2664,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445264,"statfs":{"total":95995035648,"available":95687950336,"internally_reserved":0,"allocated":2727936,"data_stored":1247454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.52400000000000002,"15min":0.59499999999999997},"min":{"1min":0.36499999999999999,"5min":0.23899999999999999,"15min":0.21199999999999999},"max":{"1min":0.66400000000000003,"5min":5.3319999999999999,"15min":9.3399999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.63100000000000001,"15min":0.61199999999999999},"min":{"1min":0.35799999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.65200000000000002,"5min":11.532,"15min":13.833},"last":0.443}]},{"osd":1,"last update":"Tue Feb 13 00:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.59599999999999997,"15min":0.75700000000000001},"min":{"1min":0.40799999999999997,"5min":0.25,"15min":0.25},"max":{"1min":0.61099999999999999,"5min":7.5410000000000004,"15min":53.188000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.59499999999999997,"15min":0.69899999999999995},"min":{"1min":0.36399999999999999,"5min":0.28599999999999998,"15min":0.25600000000000001},"max":{"1min":0.66300000000000003,"5min":11.497999999999999,"15min":44.793999999999997},"last":0.52500000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.60299999999999998,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.31900000000000001,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":4.2199999999999998,"15min":8.6829999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.60399999999999998,"15min":0.629},"min":{"1min":0.35699999999999998,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.57499999999999996,"5min":4.0730000000000004,"15min":18.911000000000001},"last":0.53900000000000003}]},{"osd":3,"last update":"Tue Feb 13 00:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54000000000000004,"15min":0.65600000000000003},"min":{"1min":0.38200000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68999999999999995,"5min":4.0090000000000003,"15min":23.5},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.55300000000000005,"15min":0.71399999999999997},"min":{"1min":0.35599999999999998,"5min":0.32100000000000001,"15min":0.30599999999999999},"max":{"1min":0.60699999999999998,"5min":3.9729999999999999,"15min":36.152999999999999},"last":0.68600000000000005}]},{"osd":5,"last update":"Tue Feb 13 00:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.64700000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.46000000000000002,"15min":0.436},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.25700000000000001},"max":{"1min":0.60999999999999999,"5min":3.153,"15min":3.153},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.497,"15min":0.46300000000000002},"min":{"1min":0.32700000000000001,"5min":0.25800000000000001,"15min":0.254},"max":{"1min":0.65900000000000003,"5min":4.9390000000000001,"15min":4.9390000000000001},"last":0.56999999999999995}]},{"osd":7,"last update":"Tue Feb 13 00:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48699999999999999,"15min":0.47699999999999998},"min":{"1min":0.374,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70099999999999996,"5min":1.575,"15min":1.575},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.495,"15min":0.47099999999999997},"min":{"1min":0.308,"5min":0.29699999999999999,"15min":0.28100000000000003},"max":{"1min":0.71399999999999997,"5min":2.46,"15min":2.46},"last":0.72499999999999998}]}]},{"osd":5,"up_from":0,"seq":2057289334807,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":3328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":232,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":232,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":232,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":232,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":233,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":233,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":233,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:50:00.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph status --format=json 2024-02-13T00:50:00.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:00.576 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:00.595 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:00.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:00.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:00.726 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:00.000138+0000 mon.a (mon.0) 3036 : cluster [WRN] Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; noscrub,nodeep-scrub flag(s) set; 2 pool(s) do not have an application enabled 2024-02-13T00:50:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:00.000184+0000 mon.a (mon.0) 3037 : cluster [WRN] [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-02-13T00:50:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:00.000207+0000 mon.a (mon.0) 3038 : cluster [WRN] pool 'test-rados-api-smithi162-84057-99-cache' with cache_mode writeback needs hit_set_type to be set but it is not 2024-02-13T00:50:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:00.000236+0000 mon.a (mon.0) 3039 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-13T00:50:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:00.000254+0000 mon.a (mon.0) 3040 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-02-13T00:50:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:00.000275+0000 mon.a (mon.0) 3041 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-13T00:50:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:00.000296+0000 mon.a (mon.0) 3042 : cluster [WRN] application not enabled on pool 'LibRadosTierECPP_smithi162-84057-94' 2024-02-13T00:50:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:00.000315+0000 mon.a (mon.0) 3043 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-13T00:50:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:00.495992+0000 mon.a (mon.0) 3044 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:50:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:00.501235+0000 mon.a (mon.0) 3045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-13T00:50:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:00.501301+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e586: 8 total, 8 up, 4 in 2024-02-13T00:50:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:00.506590+0000 mon.c (mon.2) 462 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-13T00:50:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:00.507174+0000 mon.a (mon.0) 3047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-13T00:50:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:00.000138+0000 mon.a (mon.0) 3036 : cluster [WRN] Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; noscrub,nodeep-scrub flag(s) set; 2 pool(s) do not have an application enabled 2024-02-13T00:50:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:00.000184+0000 mon.a (mon.0) 3037 : cluster [WRN] [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-02-13T00:50:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:00.000207+0000 mon.a (mon.0) 3038 : cluster [WRN] pool 'test-rados-api-smithi162-84057-99-cache' with cache_mode writeback needs hit_set_type to be set but it is not 2024-02-13T00:50:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:00.000236+0000 mon.a (mon.0) 3039 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-13T00:50:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:00.000254+0000 mon.a (mon.0) 3040 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-02-13T00:50:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:00.000275+0000 mon.a (mon.0) 3041 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-13T00:50:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:00.000296+0000 mon.a (mon.0) 3042 : cluster [WRN] application not enabled on pool 'LibRadosTierECPP_smithi162-84057-94' 2024-02-13T00:50:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:00.000315+0000 mon.a (mon.0) 3043 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-13T00:50:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:00.495992+0000 mon.a (mon.0) 3044 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:50:00.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:00.501235+0000 mon.a (mon.0) 3045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-13T00:50:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:00.501301+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e586: 8 total, 8 up, 4 in 2024-02-13T00:50:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:00.506590+0000 mon.c (mon.2) 462 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-13T00:50:00.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:00.507174+0000 mon.a (mon.0) 3047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-13T00:50:00.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:00.000138+0000 mon.a (mon.0) 3036 : cluster [WRN] Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; noscrub,nodeep-scrub flag(s) set; 2 pool(s) do not have an application enabled 2024-02-13T00:50:00.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:00.000184+0000 mon.a (mon.0) 3037 : cluster [WRN] [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-02-13T00:50:00.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:00.000207+0000 mon.a (mon.0) 3038 : cluster [WRN] pool 'test-rados-api-smithi162-84057-99-cache' with cache_mode writeback needs hit_set_type to be set but it is not 2024-02-13T00:50:00.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:00.000236+0000 mon.a (mon.0) 3039 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-13T00:50:00.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:00.000254+0000 mon.a (mon.0) 3040 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-02-13T00:50:00.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:00.000275+0000 mon.a (mon.0) 3041 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-13T00:50:00.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:00.000296+0000 mon.a (mon.0) 3042 : cluster [WRN] application not enabled on pool 'LibRadosTierECPP_smithi162-84057-94' 2024-02-13T00:50:00.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:00.000315+0000 mon.a (mon.0) 3043 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-13T00:50:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:00.495992+0000 mon.a (mon.0) 3044 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:50:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:00.501235+0000 mon.a (mon.0) 3045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-13T00:50:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:00.501301+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e586: 8 total, 8 up, 4 in 2024-02-13T00:50:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:00.506590+0000 mon.c (mon.2) 462 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-13T00:50:00.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:00.507174+0000 mon.a (mon.0) 3047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-13T00:50:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:00.484331+0000 mgr.y (mgr.14544) 503 : cluster [DBG] pgmap v878: 212 pgs: 212 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:01.498689+0000 mon.c (mon.2) 463 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:50:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:01.503893+0000 mon.a (mon.0) 3048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-13T00:50:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:01.503970+0000 mon.a (mon.0) 3049 : cluster [DBG] osdmap e587: 8 total, 8 up, 4 in 2024-02-13T00:50:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:01.540994+0000 mon.c (mon.2) 464 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:01.541454+0000 mon.a (mon.0) 3050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:01.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:00.484331+0000 mgr.y (mgr.14544) 503 : cluster [DBG] pgmap v878: 212 pgs: 212 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:01.498689+0000 mon.c (mon.2) 463 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:50:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:01.503893+0000 mon.a (mon.0) 3048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-13T00:50:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:01.503970+0000 mon.a (mon.0) 3049 : cluster [DBG] osdmap e587: 8 total, 8 up, 4 in 2024-02-13T00:50:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:01.540994+0000 mon.c (mon.2) 464 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:01.541454+0000 mon.a (mon.0) 3050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:00.484331+0000 mgr.y (mgr.14544) 503 : cluster [DBG] pgmap v878: 212 pgs: 212 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:01.498689+0000 mon.c (mon.2) 463 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:50:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:01.503893+0000 mon.a (mon.0) 3048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-99-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-13T00:50:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:01.503970+0000 mon.a (mon.0) 3049 : cluster [DBG] osdmap e587: 8 total, 8 up, 4 in 2024-02-13T00:50:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:01.540994+0000 mon.c (mon.2) 464 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:01.541454+0000 mon.a (mon.0) 3050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:02.846 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:50:02.846 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:50:03.057 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:03.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.786+0000 7fd265ec3700 1 -- 172.21.15.162:0/4123784332 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd26006ba60 msgr2=0x7fd2601040b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.786+0000 7fd265ec3700 1 --2- 172.21.15.162:0/4123784332 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd26006ba60 0x7fd2601040b0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7fd248004870 tx=0x7fd248009cb0).stop 2024-02-13T00:50:03.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.786+0000 7fd265ec3700 1 -- 172.21.15.162:0/4123784332 shutdown_connections 2024-02-13T00:50:03.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.786+0000 7fd265ec3700 1 --2- 172.21.15.162:0/4123784332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2601045a0 0x7fd260106990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.786+0000 7fd265ec3700 1 --2- 172.21.15.162:0/4123784332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd26006b190 0x7fd26006b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.786+0000 7fd265ec3700 1 --2- 172.21.15.162:0/4123784332 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd26006ba60 0x7fd2601040b0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.786+0000 7fd265ec3700 1 -- 172.21.15.162:0/4123784332 >> 172.21.15.162:0/4123784332 conn(0x7fd2600f38c0 msgr2=0x7fd2600f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:03.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.786+0000 7fd265ec3700 1 -- 172.21.15.162:0/4123784332 shutdown_connections 2024-02-13T00:50:03.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.786+0000 7fd265ec3700 1 -- 172.21.15.162:0/4123784332 wait complete. 2024-02-13T00:50:03.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.787+0000 7fd265ec3700 1 librados: starting msgr at 2024-02-13T00:50:03.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.787+0000 7fd265ec3700 1 librados: starting objecter 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.787+0000 7fd265ec3700 1 Processor -- start 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.787+0000 7fd265ec3700 1 -- start start 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.787+0000 7fd265ec3700 1 librados: setting wanted keys 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.787+0000 7fd265ec3700 1 librados: calling monclient init 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.787+0000 7fd265ec3700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd26006b190 0x7fd260110600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.787+0000 7fd265ec3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26006ba60 0x7fd260110af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.787+0000 7fd265ec3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2601045a0 0x7fd26010a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.787+0000 7fd265ec3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26010acd0 con 0x7fd2601045a0 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.787+0000 7fd265ec3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26010ae40 con 0x7fd26006ba60 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.787+0000 7fd265ec3700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd26010b000 con 0x7fd26006b190 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd26006b190 0x7fd260110600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2601045a0 0x7fd26010a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd26006b190 0x7fd260110600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:45570/0 (socket says 172.21.15.162:45570) 2024-02-13T00:50:03.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25f7fe700 1 -- 172.21.15.162:0/1103499511 learned_addr learned my addr 172.21.15.162:0/1103499511 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd256ffd700 1 --2- 172.21.15.162:0/1103499511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26006ba60 0x7fd260110af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25ffff700 1 -- 172.21.15.162:0/1103499511 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd26006b190 msgr2=0x7fd260110600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25ffff700 1 --2- 172.21.15.162:0/1103499511 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd26006b190 0x7fd260110600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25ffff700 1 -- 172.21.15.162:0/1103499511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26006ba60 msgr2=0x7fd260110af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25ffff700 1 --2- 172.21.15.162:0/1103499511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd26006ba60 0x7fd260110af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25ffff700 1 -- 172.21.15.162:0/1103499511 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd250005040 con 0x7fd2601045a0 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25f7fe700 1 --2- 172.21.15.162:0/1103499511 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd26006b190 0x7fd260110600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25ffff700 1 --2- 172.21.15.162:0/1103499511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2601045a0 0x7fd26010a7e0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 rx=0x7fd26006c170 tx=0x7fd258007d30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25d7fa700 10 client.?.objecter ms_handle_connect 0x7fd2601045a0 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25800e4a0 con 0x7fd2601045a0 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd248005040 con 0x7fd2601045a0 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd25800ead0 con 0x7fd2601045a0 2024-02-13T00:50:03.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258012a90 con 0x7fd2601045a0 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd265ec3700 10 client.27741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26010b640 con 0x7fd2601045a0 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.788+0000 7fd265ec3700 1 librados: init done 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.789+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd2600475e0 con 0x7fd2601045a0 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.789+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fd25800ec40 con 0x7fd2601045a0 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd25d7fa700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd24c062c90 0x7fd24c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(588..588 src has 449..588) v4 ==== 9295+0+0 (secure 0 0 0) 0x7fd25807ccf0 con 0x7fd2601045a0 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 osd_map(588..588 src has 449..588) v4 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd25d7fa700 3 client.27741.objecter handle_osd_map got epochs [588,588] > 0 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd25d7fa700 3 client.27741.objecter handle_osd_map decoding full epoch 588 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd25f7fe700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd24c062c90 0x7fd24c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd25d7fa700 20 client.27741.objecter dump_active .. 0 homeless 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v588) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd25807f380 con 0x7fd2601045a0 2024-02-13T00:50:03.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd25d7fa700 10 client.27741.objecter ms_handle_connect 0x7fd24c062c90 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd25f7fe700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd24c062c90 0x7fd24c065150 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7fd250009810 tx=0x7fd25000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd265ec3700 10 client.27741.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 1 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd260028590 0x7fd2600fe060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260109480 osd=0 3 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260109480 osd=0 3 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260109480 osd=0 3 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 1 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd260109480 osd=0 3 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 0 1 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260109480 osd=0 4 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd265ec3700 10 client.27741.objecter _send_command 1 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2600ff6d0 con 0x7fd260028590 2024-02-13T00:50:03.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.790+0000 7fd256ffd700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd260028590 0x7fd2600fe060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:03.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.791+0000 7fd25d7fa700 10 client.27741.objecter ms_handle_connect 0x7fd260028590 2024-02-13T00:50:03.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.791+0000 7fd256ffd700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd260028590 0x7fd2600fe060 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:03.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.791+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd2600ff6d0 con 0x7fd260028590 2024-02-13T00:50:03.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.791+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:50:03.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.791+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 1 = system:0 2024-02-13T00:50:03.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.791+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd260109480 osd=0 4 2024-02-13T00:50:03.406 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:50:03.406 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.791+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 0 1 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.806+0000 7fd265ec3700 10 client.27741.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.806+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 2 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.806+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260109480 osd=0 3 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.806+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260109480 osd=0 3 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.806+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260109480 osd=0 3 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.806+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 2 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.806+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd260109480 osd=0 3 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.806+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 0 2 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.806+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260109480 osd=0 4 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.806+0000 7fd265ec3700 10 client.27741.objecter _send_command 2 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.806+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd26005a0e0 con 0x7fd260028590 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.807+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd26005a0e0 con 0x7fd260028590 2024-02-13T00:50:03.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.807+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.807+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.807+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd260109480 osd=0 4 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.807+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 0 2 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd265ec3700 10 client.27741.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 3 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6805/150599780 conn(0x7fd260059570 0x7fd26011b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260059400 osd=1 3 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260059400 osd=1 3 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260059400 osd=1 3 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 3 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd260059400 osd=1 3 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 1 3 2024-02-13T00:50:03.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260059400 osd=1 4 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd265ec3700 10 client.27741.objecter _send_command 3 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2600280f0 con 0x7fd260059570 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd25f7fe700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6805/150599780 conn(0x7fd260059570 0x7fd26011b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd25d7fa700 10 client.27741.objecter ms_handle_connect 0x7fd260059570 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.808+0000 7fd25f7fe700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6805/150599780 conn(0x7fd260059570 0x7fd26011b550 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.809+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd2600280f0 con 0x7fd260059570 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.809+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.809+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 3 = system:0 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.809+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd260059400 osd=1 4 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.809+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 1 3 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.823+0000 7fd265ec3700 10 client.27741.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.823+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 4 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.823+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260059400 osd=1 3 2024-02-13T00:50:03.409 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.823+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260059400 osd=1 3 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.823+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260059400 osd=1 3 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.823+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 4 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.823+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd260059400 osd=1 3 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.823+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 1 4 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.823+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260059400 osd=1 4 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.823+0000 7fd265ec3700 10 client.27741.objecter _send_command 4 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.823+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd26011ba40 con 0x7fd260059570 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd26011ba40 con 0x7fd260059570 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd260059400 osd=1 4 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 1 4 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd265ec3700 10 client.27741.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 5 2024-02-13T00:50:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd2600277c0 0x7fd26011eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260027650 osd=2 3 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260027650 osd=2 3 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260027650 osd=2 3 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 5 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd260027650 osd=2 3 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 2 5 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260027650 osd=2 4 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd265ec3700 10 client.27741.objecter _send_command 5 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2600ff230 con 0x7fd2600277c0 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.824+0000 7fd256ffd700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd2600277c0 0x7fd26011eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.825+0000 7fd25d7fa700 10 client.27741.objecter ms_handle_connect 0x7fd2600277c0 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.825+0000 7fd256ffd700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd2600277c0 0x7fd26011eda0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:03.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.825+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd2600ff230 con 0x7fd2600277c0 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.825+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.825+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 5 = system:0 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.825+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd260027650 osd=2 4 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.825+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 2 5 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.840+0000 7fd265ec3700 10 client.27741.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.840+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 6 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.840+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260027650 osd=2 3 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.840+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260027650 osd=2 3 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.840+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260027650 osd=2 3 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.840+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 6 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.840+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd260027650 osd=2 3 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.840+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 2 6 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.840+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260027650 osd=2 4 2024-02-13T00:50:03.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.840+0000 7fd265ec3700 10 client.27741.objecter _send_command 6 2024-02-13T00:50:03.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.840+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd26011f290 con 0x7fd2600277c0 2024-02-13T00:50:03.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd26011f290 con 0x7fd2600277c0 2024-02-13T00:50:03.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:03.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd260027650 osd=2 4 2024-02-13T00:50:03.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 2 6 2024-02-13T00:50:03.413 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:50:03.413 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:50:03.413 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd265ec3700 10 client.27741.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 7 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6813/604971011 conn(0x7fd2600fe900 0x7fd260122620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd2600fe790 osd=3 3 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd2600fe790 osd=3 3 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd2600fe790 osd=3 3 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 7 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd2600fe790 osd=3 3 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 3 7 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd2600fe790 osd=3 4 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd265ec3700 10 client.27741.objecter _send_command 7 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd26011c8f0 con 0x7fd2600fe900 2024-02-13T00:50:03.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.842+0000 7fd25ffff700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6813/604971011 conn(0x7fd2600fe900 0x7fd260122620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.843+0000 7fd25d7fa700 10 client.27741.objecter ms_handle_connect 0x7fd2600fe900 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.843+0000 7fd25ffff700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6813/604971011 conn(0x7fd2600fe900 0x7fd260122620 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.843+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd26011c8f0 con 0x7fd2600fe900 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.843+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.843+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 7 = system:0 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.843+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd2600fe790 osd=3 4 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.843+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 3 7 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.858+0000 7fd265ec3700 10 client.27741.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.858+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 8 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.858+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd2600fe790 osd=3 3 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.858+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd2600fe790 osd=3 3 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.858+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd2600fe790 osd=3 3 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.858+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 8 2024-02-13T00:50:03.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.858+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd2600fe790 osd=3 3 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.858+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 3 8 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.858+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd2600fe790 osd=3 4 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.858+0000 7fd265ec3700 10 client.27741.objecter _send_command 8 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.858+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd260122b10 con 0x7fd2600fe900 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd260122b10 con 0x7fd2600fe900 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd2600fe790 osd=3 4 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 3 8 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd265ec3700 10 client.27741.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 9 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd2601231f0 0x7fd2601255e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260122b10 osd=4 3 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260122b10 osd=4 3 2024-02-13T00:50:03.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260122b10 osd=4 3 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 9 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd260122b10 osd=4 3 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 4 9 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260122b10 osd=4 4 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd265ec3700 10 client.27741.objecter _send_command 9 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.859+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd260128260 con 0x7fd2601231f0 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.860+0000 7fd25f7fe700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd2601231f0 0x7fd2601255e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.860+0000 7fd25f7fe700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd2601231f0 0x7fd2601255e0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.860+0000 7fd25d7fa700 10 client.27741.objecter ms_handle_connect 0x7fd2601231f0 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.861+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd260128260 con 0x7fd2601231f0 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.861+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.861+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 9 = system:0 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.861+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd260122b10 osd=4 4 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.861+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 4 9 2024-02-13T00:50:03.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.875+0000 7fd265ec3700 10 client.27741.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.875+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 10 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.875+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260122b10 osd=4 3 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.875+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260122b10 osd=4 3 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.875+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260122b10 osd=4 3 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.875+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 10 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.875+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd260122b10 osd=4 3 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.875+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 4 10 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.875+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260122b10 osd=4 4 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.875+0000 7fd265ec3700 10 client.27741.objecter _send_command 10 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.875+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd260128260 con 0x7fd2601231f0 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd260128260 con 0x7fd2601231f0 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd260122b10 osd=4 4 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 4 10 2024-02-13T00:50:03.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd265ec3700 10 client.27741.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:03.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 11 2024-02-13T00:50:03.419 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:50:03.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6804/2277917372 conn(0x7fd260125fe0 0x7fd260126440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:03.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260128260 osd=5 3 2024-02-13T00:50:03.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260128260 osd=5 3 2024-02-13T00:50:03.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260128260 osd=5 3 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 11 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd260128260 osd=5 3 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 5 11 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260128260 osd=5 4 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd265ec3700 10 client.27741.objecter _send_command 11 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd260128090 con 0x7fd260125fe0 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd256ffd700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6804/2277917372 conn(0x7fd260125fe0 0x7fd260126440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd256ffd700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6804/2277917372 conn(0x7fd260125fe0 0x7fd260126440 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.877+0000 7fd25d7fa700 10 client.27741.objecter ms_handle_connect 0x7fd260125fe0 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.878+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd260128090 con 0x7fd260125fe0 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.878+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.878+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 11 = system:0 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.878+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd260128260 osd=5 4 2024-02-13T00:50:03.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.878+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 5 11 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.894+0000 7fd265ec3700 10 client.27741.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.894+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 12 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.894+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260128260 osd=5 3 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.894+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260128260 osd=5 3 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.894+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260128260 osd=5 3 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.894+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 12 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.894+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd260128260 osd=5 3 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.894+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 5 12 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.894+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260128260 osd=5 4 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.894+0000 7fd265ec3700 10 client.27741.objecter _send_command 12 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.894+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd260128090 con 0x7fd260125fe0 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.895+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd260128090 con 0x7fd260125fe0 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.895+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:03.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.895+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.895+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd260128260 osd=5 4 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.895+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 5 12 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.895+0000 7fd265ec3700 10 client.27741.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.895+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 13 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.895+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd26012aa60 0x7fd26012aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.895+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260128090 osd=6 3 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.896+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260128090 osd=6 3 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.896+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260128090 osd=6 3 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.896+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 13 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.896+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd260128090 osd=6 3 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.896+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 6 13 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.896+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260128090 osd=6 4 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.896+0000 7fd265ec3700 10 client.27741.objecter _send_command 13 2024-02-13T00:50:03.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.896+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd26012b510 con 0x7fd26012aa60 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.896+0000 7fd25ffff700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd26012aa60 0x7fd26012aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.896+0000 7fd25d7fa700 10 client.27741.objecter ms_handle_connect 0x7fd26012aa60 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.896+0000 7fd25ffff700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd26012aa60 0x7fd26012aea0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.897+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd25804ff20 con 0x7fd26012aa60 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.897+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.897+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 13 = system:0 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.897+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd260128090 osd=6 4 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.897+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 6 13 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.911+0000 7fd265ec3700 10 client.27741.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.911+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 14 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.911+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260128090 osd=6 3 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.911+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260128090 osd=6 3 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.911+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd260128090 osd=6 3 2024-02-13T00:50:03.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.911+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 14 2024-02-13T00:50:03.424 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:50:03.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.911+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd260128090 osd=6 3 2024-02-13T00:50:03.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.911+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 6 14 2024-02-13T00:50:03.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.911+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260128090 osd=6 4 2024-02-13T00:50:03.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.911+0000 7fd265ec3700 10 client.27741.objecter _send_command 14 2024-02-13T00:50:03.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.911+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd26012d720 con 0x7fd26012aa60 2024-02-13T00:50:03.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd26012d720 con 0x7fd26012aa60 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd260128090 osd=6 4 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 6 14 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd265ec3700 10 client.27741.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 15 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd26012ddd0 0x7fd26012e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd26012d720 osd=7 3 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd26012d720 osd=7 3 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd26012d720 osd=7 3 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 15 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd26012d720 osd=7 3 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 7 15 2024-02-13T00:50:03.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd26012d720 osd=7 4 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd265ec3700 10 client.27741.objecter _send_command 15 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd260131ae0 con 0x7fd26012ddd0 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd25f7fe700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd26012ddd0 0x7fd26012e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd25d7fa700 10 client.27741.objecter ms_handle_connect 0x7fd26012ddd0 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.914+0000 7fd25f7fe700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd26012ddd0 0x7fd26012e230 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.915+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd260131ae0 con 0x7fd26012ddd0 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.915+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.915+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 15 = system:0 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.915+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd26012d720 osd=7 4 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.915+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 7 15 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.929+0000 7fd265ec3700 10 client.27741.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.929+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign -1 16 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.929+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd26012d720 osd=7 3 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.929+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd26012d720 osd=7 3 2024-02-13T00:50:03.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.929+0000 7fd265ec3700 20 client.27741.objecter _get_session s=0x7fd26012d720 osd=7 3 2024-02-13T00:50:03.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.929+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_remove -1 16 2024-02-13T00:50:03.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.929+0000 7fd265ec3700 20 client.27741.objecter get_session s=0x7fd26012d720 osd=7 3 2024-02-13T00:50:03.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.929+0000 7fd265ec3700 15 client.27741.objecter _session_command_op_assign 7 16 2024-02-13T00:50:03.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.929+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd26012d720 osd=7 4 2024-02-13T00:50:03.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.929+0000 7fd265ec3700 10 client.27741.objecter _send_command 16 2024-02-13T00:50:03.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.929+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd260130960 con 0x7fd26012ddd0 2024-02-13T00:50:03.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd25d7fa700 1 -- 172.21.15.162:0/1103499511 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd260130960 con 0x7fd26012ddd0 2024-02-13T00:50:03.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd25d7fa700 10 client.27741.objecter ms_dispatch 0x7fd260001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:03.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd25d7fa700 10 client.27741.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd25d7fa700 20 client.27741.objecter put_session s=0x7fd26012d720 osd=7 4 2024-02-13T00:50:03.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd25d7fa700 15 client.27741.objecter _session_command_op_remove 7 16 2024-02-13T00:50:03.428 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:50:03.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 10 librados: watch_flush enter 2024-02-13T00:50:03.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 10 librados: watch_flush exit 2024-02-13T00:50:03.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 10 client.27741.objecter close_session for osd.0 2024-02-13T00:50:03.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd260028590 msgr2=0x7fd2600fe060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6801/2103743338 conn(0x7fd260028590 0x7fd2600fe060 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260109480 osd=0 1 2024-02-13T00:50:03.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 10 client.27741.objecter close_session for osd.1 2024-02-13T00:50:03.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6805/150599780 conn(0x7fd260059570 msgr2=0x7fd26011b550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6805/150599780 conn(0x7fd260059570 0x7fd26011b550 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260059400 osd=1 1 2024-02-13T00:50:03.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 10 client.27741.objecter close_session for osd.2 2024-02-13T00:50:03.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd2600277c0 msgr2=0x7fd26011eda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6809/3689719376 conn(0x7fd2600277c0 0x7fd26011eda0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260027650 osd=2 1 2024-02-13T00:50:03.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 10 client.27741.objecter close_session for osd.3 2024-02-13T00:50:03.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6813/604971011 conn(0x7fd2600fe900 msgr2=0x7fd260122620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6813/604971011 conn(0x7fd2600fe900 0x7fd260122620 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd25ffff700 1 -- 172.21.15.162:0/1103499511 reap_dead start 2024-02-13T00:50:03.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd2600fe790 osd=3 1 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 10 client.27741.objecter close_session for osd.4 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd2601231f0 msgr2=0x7fd2601255e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6800/3344263388 conn(0x7fd2601231f0 0x7fd2601255e0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260122b10 osd=4 1 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 10 client.27741.objecter close_session for osd.5 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6804/2277917372 conn(0x7fd260125fe0 msgr2=0x7fd260126440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6804/2277917372 conn(0x7fd260125fe0 0x7fd260126440 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260128260 osd=5 1 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd25ffff700 1 -- 172.21.15.162:0/1103499511 reap_dead start 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 10 client.27741.objecter close_session for osd.6 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd26012aa60 msgr2=0x7fd26012aea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6808/1506640341 conn(0x7fd26012aa60 0x7fd26012aea0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd260128090 osd=6 1 2024-02-13T00:50:03.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 10 client.27741.objecter close_session for osd.7 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd26012ddd0 msgr2=0x7fd26012e230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.181:6812/1964192772 conn(0x7fd26012ddd0 0x7fd26012e230 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd25ffff700 1 -- 172.21.15.162:0/1103499511 reap_dead start 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 20 client.27741.objecter put_session s=0x7fd26012d720 osd=7 1 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 20 client.27741.objecter shutdown clearing up homeless session... 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 10 client.27741.objecter successfully canceled tick 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd24c062c90 msgr2=0x7fd24c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd24c062c90 0x7fd24c065150 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7fd250009810 tx=0x7fd25000b040).stop 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2601045a0 msgr2=0x7fd26010a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2601045a0 0x7fd26010a7e0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 rx=0x7fd26006c170 tx=0x7fd258007d30).stop 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.931+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 shutdown_connections 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.932+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2601045a0 0x7fd26010a7e0 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.932+0000 7fd265ec3700 1 --2- 172.21.15.162:0/1103499511 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd24c062c90 0x7fd24c065150 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.932+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 >> 172.21.15.162:0/1103499511 conn(0x7fd2600f38c0 msgr2=0x7fd2600f4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.932+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 shutdown_connections 2024-02-13T00:50:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.932+0000 7fd265ec3700 1 -- 172.21.15.162:0/1103499511 wait complete. 2024-02-13T00:50:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:02.932+0000 7fd265ec3700 1 librados: shutdown 2024-02-13T00:50:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.432 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:50:03.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:50:03.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:02.484992+0000 mgr.y (mgr.14544) 504 : cluster [DBG] pgmap v881: 212 pgs: 212 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:03.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:02.502885+0000 mon.a (mon.0) 3051 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-99"}]': finished 2024-02-13T00:50:03.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:02.502960+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e588: 8 total, 8 up, 4 in 2024-02-13T00:50:03.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:02.505275+0000 mon.c (mon.2) 465 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:50:03.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:02.505795+0000 mon.a (mon.0) 3053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:50:03.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:02.791919+0000 mon.a (mon.0) 3054 : audit [DBG] from='client.? 172.21.15.162:0/1103499511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:03.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:02.484992+0000 mgr.y (mgr.14544) 504 : cluster [DBG] pgmap v881: 212 pgs: 212 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:03.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:02.502885+0000 mon.a (mon.0) 3051 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-99"}]': finished 2024-02-13T00:50:03.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:02.502960+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e588: 8 total, 8 up, 4 in 2024-02-13T00:50:03.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:02.505275+0000 mon.c (mon.2) 465 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:50:03.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:02.505795+0000 mon.a (mon.0) 3053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:50:03.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:02.791919+0000 mon.a (mon.0) 3054 : audit [DBG] from='client.? 172.21.15.162:0/1103499511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:03.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:02.484992+0000 mgr.y (mgr.14544) 504 : cluster [DBG] pgmap v881: 212 pgs: 212 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:03.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:02.502885+0000 mon.a (mon.0) 3051 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-99"}]': finished 2024-02-13T00:50:03.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:02.502960+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e588: 8 total, 8 up, 4 in 2024-02-13T00:50:03.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:02.505275+0000 mon.c (mon.2) 465 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:50:03.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:02.505795+0000 mon.a (mon.0) 3053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]: dispatch 2024-02-13T00:50:03.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:02.791919+0000 mon.a (mon.0) 3054 : audit [DBG] from='client.? 172.21.15.162:0/1103499511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:03.504532+0000 mon.a (mon.0) 3055 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]': finished 2024-02-13T00:50:04.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:03.504605+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e589: 8 total, 8 up, 4 in 2024-02-13T00:50:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:03.504532+0000 mon.a (mon.0) 3055 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]': finished 2024-02-13T00:50:04.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:03.504605+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e589: 8 total, 8 up, 4 in 2024-02-13T00:50:04.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:03.504532+0000 mon.a (mon.0) 3055 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-99", "tierpool": "test-rados-api-smithi162-84057-99-cache"}]': finished 2024-02-13T00:50:04.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:03.504605+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e589: 8 total, 8 up, 4 in 2024-02-13T00:50:05.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-13T00:50:05.780 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:04.485304+0000 mgr.y (mgr.14544) 505 : cluster [DBG] pgmap v884: 212 pgs: 212 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:05.780 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:04.513922+0000 mon.a (mon.0) 3057 : cluster [DBG] osdmap e590: 8 total, 8 up, 4 in 2024-02-13T00:50:05.781 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:05.495017+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e591: 8 total, 8 up, 4 in 2024-02-13T00:50:05.781 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:05.496562+0000 mon.c (mon.2) 466 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:05.781 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:05.502443+0000 mon.a (mon.0) 3059 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:05.814 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:04.485304+0000 mgr.y (mgr.14544) 505 : cluster [DBG] pgmap v884: 212 pgs: 212 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:05.814 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:04.513922+0000 mon.a (mon.0) 3057 : cluster [DBG] osdmap e590: 8 total, 8 up, 4 in 2024-02-13T00:50:05.814 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:05.495017+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e591: 8 total, 8 up, 4 in 2024-02-13T00:50:05.814 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:05.496562+0000 mon.c (mon.2) 466 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:05.814 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:05.502443+0000 mon.a (mon.0) 3059 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:04.485304+0000 mgr.y (mgr.14544) 505 : cluster [DBG] pgmap v884: 212 pgs: 212 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:04.513922+0000 mon.a (mon.0) 3057 : cluster [DBG] osdmap e590: 8 total, 8 up, 4 in 2024-02-13T00:50:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:05.495017+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e591: 8 total, 8 up, 4 in 2024-02-13T00:50:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:05.496562+0000 mon.c (mon.2) 466 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:05.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:05.502443+0000 mon.a (mon.0) 3059 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:05.862 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:05.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:05.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:06.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:50:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:50:06.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:06.126 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:06.170 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:50:06.411 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:50:06] "GET /metrics HTTP/1.1" 200 191106 "" "Prometheus/2.33.4" 2024-02-13T00:50:06.576 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:06.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:06.680 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:06.713 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:06.734 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:06.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:06.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:06.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:05.640999+0000 mon.a (mon.0) 3060 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:06.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:05.777362+0000 mgr.y (mgr.14544) 506 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:06.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:06.501199+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]': finished 2024-02-13T00:50:06.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:06.501270+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e592: 8 total, 8 up, 4 in 2024-02-13T00:50:06.765 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:06.502748+0000 mon.c (mon.2) 467 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:06.765 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:06.503039+0000 mon.a (mon.0) 3063 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:06.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:05.640999+0000 mon.a (mon.0) 3060 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:06.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:05.777362+0000 mgr.y (mgr.14544) 506 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:06.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:06.501199+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]': finished 2024-02-13T00:50:06.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:06.501270+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e592: 8 total, 8 up, 4 in 2024-02-13T00:50:06.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:06.502748+0000 mon.c (mon.2) 467 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:06.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:06.503039+0000 mon.a (mon.0) 3063 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:06.838 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:06.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.777+0000 7fab4e23e700 1 -- 172.21.15.162:0/999120430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab480fd490 msgr2=0x7fab480f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:06.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.777+0000 7fab4e23e700 1 --2- 172.21.15.162:0/999120430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab480fd490 0x7fab480f6cc0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7fab3c0047b0 tx=0x7fab3c009cb0).stop 2024-02-13T00:50:06.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.777+0000 7fab4e23e700 1 -- 172.21.15.162:0/999120430 shutdown_connections 2024-02-13T00:50:06.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.777+0000 7fab4e23e700 1 --2- 172.21.15.162:0/999120430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab480f73a0 0x7fab481039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:06.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.777+0000 7fab4e23e700 1 --2- 172.21.15.162:0/999120430 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab480fcb30 0x7fab480fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:06.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.777+0000 7fab4e23e700 1 --2- 172.21.15.162:0/999120430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab480fd490 0x7fab480f6cc0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:06.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.777+0000 7fab4e23e700 1 -- 172.21.15.162:0/999120430 >> 172.21.15.162:0/999120430 conn(0x7fab4806eb90 msgr2=0x7fab480f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:06.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.777+0000 7fab4e23e700 1 -- 172.21.15.162:0/999120430 shutdown_connections 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.777+0000 7fab4e23e700 1 -- 172.21.15.162:0/999120430 wait complete. 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.777+0000 7fab4e23e700 1 librados: starting msgr at 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.777+0000 7fab4e23e700 1 librados: starting objecter 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab4e23e700 1 Processor -- start 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab4e23e700 1 -- start start 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab4e23e700 1 librados: setting wanted keys 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab4e23e700 1 librados: calling monclient init 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab4e23e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab480f73a0 0x7fab4810d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab4e23e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab480fcb30 0x7fab4810d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab4e23e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab480fd490 0x7fab48107210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab4e23e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab48107700 con 0x7fab480fd490 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab4e23e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab48107870 con 0x7fab480fcb30 2024-02-13T00:50:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab4e23e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab48107a30 con 0x7fab480f73a0 2024-02-13T00:50:06.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab477fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab480f73a0 0x7fab4810d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:06.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab47fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab480fd490 0x7fab48107210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:06.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab477fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab480f73a0 0x7fab4810d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:35692/0 (socket says 172.21.15.162:35692) 2024-02-13T00:50:06.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab477fe700 1 -- 172.21.15.162:0/1907305243 learned_addr learned my addr 172.21.15.162:0/1907305243 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:06.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.778+0000 7fab46ffd700 1 --2- 172.21.15.162:0/1907305243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab480fcb30 0x7fab4810d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:06.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab477fe700 1 -- 172.21.15.162:0/1907305243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab480fcb30 msgr2=0x7fab4810d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:06.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab477fe700 1 --2- 172.21.15.162:0/1907305243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab480fcb30 0x7fab4810d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:06.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab477fe700 1 -- 172.21.15.162:0/1907305243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab480fd490 msgr2=0x7fab48107210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:06.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab477fe700 1 --2- 172.21.15.162:0/1907305243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab480fd490 0x7fab48107210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:06.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab477fe700 1 -- 172.21.15.162:0/1907305243 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab38005040 con 0x7fab480f73a0 2024-02-13T00:50:06.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab477fe700 1 --2- 172.21.15.162:0/1907305243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab480f73a0 0x7fab4810d070 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7fab38009400 tx=0x7fab38009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:06.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab44ff9700 10 client.?.objecter ms_handle_connect 0x7fab480f73a0 2024-02-13T00:50:06.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab44ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:06.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab44ff9700 1 -- 172.21.15.162:0/1907305243 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38007a40 con 0x7fab480f73a0 2024-02-13T00:50:06.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab44ff9700 1 -- 172.21.15.162:0/1907305243 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab3800f040 con 0x7fab480f73a0 2024-02-13T00:50:06.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab4e23e700 1 -- 172.21.15.162:0/1907305243 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3c005040 con 0x7fab480f73a0 2024-02-13T00:50:06.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab44ff9700 1 -- 172.21.15.162:0/1907305243 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38014a70 con 0x7fab480f73a0 2024-02-13T00:50:06.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab4e23e700 10 client.37847.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:06.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab4e23e700 1 -- 172.21.15.162:0/1907305243 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab48108070 con 0x7fab480f73a0 2024-02-13T00:50:06.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.779+0000 7fab4e23e700 1 librados: init done 2024-02-13T00:50:06.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.780+0000 7fab4e23e700 1 -- 172.21.15.162:0/1907305243 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab48106050 con 0x7fab480f73a0 2024-02-13T00:50:06.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.780+0000 7fab44ff9700 1 -- 172.21.15.162:0/1907305243 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fab38010530 con 0x7fab480f73a0 2024-02-13T00:50:06.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.780+0000 7fab44ff9700 1 --2- 172.21.15.162:0/1907305243 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab30062c90 0x7fab30065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:06.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.781+0000 7fab44ff9700 1 -- 172.21.15.162:0/1907305243 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(592..592 src has 449..592) v4 ==== 8202+0+0 (secure 0 0 0) 0x7fab3807ce30 con 0x7fab480f73a0 2024-02-13T00:50:06.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.781+0000 7fab44ff9700 10 client.37847.objecter ms_dispatch 0x7fab48001550 osd_map(592..592 src has 449..592) v4 2024-02-13T00:50:06.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.781+0000 7fab44ff9700 3 client.37847.objecter handle_osd_map got epochs [592,592] > 0 2024-02-13T00:50:06.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.781+0000 7fab44ff9700 3 client.37847.objecter handle_osd_map decoding full epoch 592 2024-02-13T00:50:06.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.781+0000 7fab46ffd700 1 --2- 172.21.15.162:0/1907305243 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab30062c90 0x7fab30065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:06.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.781+0000 7fab44ff9700 20 client.37847.objecter dump_active .. 0 homeless 2024-02-13T00:50:06.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.781+0000 7fab44ff9700 10 client.37847.objecter ms_handle_connect 0x7fab30062c90 2024-02-13T00:50:06.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.781+0000 7fab46ffd700 1 --2- 172.21.15.162:0/1907305243 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab30062c90 0x7fab30065150 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 rx=0x7fab481089b0 tx=0x7fab3c007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:06.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:06.785+0000 7fab44ff9700 1 -- 172.21.15.162:0/1907305243 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fab38049f60 con 0x7fab480f73a0 2024-02-13T00:50:06.866 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:06.885 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:06.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:06.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:06.989 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:07.019 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:07.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.023+0000 7fab4e23e700 1 -- 172.21.15.162:0/1907305243 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fab4805fbb0 con 0x7fab480f73a0 2024-02-13T00:50:07.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.024+0000 7fab44ff9700 1 -- 172.21.15.162:0/1907305243 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2456 (secure 0 0 0) 0x7fab38050270 con 0x7fab480f73a0 2024-02-13T00:50:07.029 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:50:07.029 INFO:teuthology.orchestra.run.smithi162.stdout:{"fsid":"d4ee01fc-ca06-11ee-95b9-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) do not have an application enabled","count":2},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":959,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":592,"num_osds":8,"num_up_osds":8,"osd_up_since":1707785283,"num_in_osds":4,"osd_in_since":1707785160,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":172}],"num_pgs":172,"num_pools":8,"num_objects":226,"data_bytes":5801,"bytes_used":1245892608,"bytes_avail":382734249984,"bytes_total":383980142592,"read_bytes_sec":1279,"read_op_per_sec":1,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.162:8443/","prometheus":"http://172.21.15.162:9283/"}},"servicemap":{"epoch":21,"modified":"2024-02-13T00:49:04.468548+0000","services":{"rgw":{"daemons":{"summary":"","24377":{"start_epoch":12,"start_stamp":"2024-02-13T00:37:48.349946+0000","gid":24377,"addr":"172.21.15.162:0/2326056861","metadata":{"arch":"x86_64","ceph_release":"pacific","ceph_version":"ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable)","ceph_version_short":"16.2.14-521-g0e714d9a","container_hostname":"smithi162","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi162","id":"foo.a","kernel_description":"#1 SMP Thu Jan 18 15:15:29 UTC 2024","kernel_version":"4.18.0-536.el8.x86_64","mem_swap_kb":"0","mem_total_kb":"32537672","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"65231d82-160b-42d4-a07d-27856f74fab0","zone_name":"default","zonegroup_id":"7c7caf1b-a652-4162-8eb9-09cc67b471b1","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-02-13T00:50:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 10 librados: watch_flush enter 2024-02-13T00:50:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 10 librados: watch_flush exit 2024-02-13T00:50:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 20 client.37847.objecter shutdown clearing up homeless session... 2024-02-13T00:50:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 10 client.37847.objecter successfully canceled tick 2024-02-13T00:50:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 -- 172.21.15.162:0/1907305243 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab30062c90 msgr2=0x7fab30065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 --2- 172.21.15.162:0/1907305243 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab30062c90 0x7fab30065150 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 rx=0x7fab481089b0 tx=0x7fab3c007f90).stop 2024-02-13T00:50:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 -- 172.21.15.162:0/1907305243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab480f73a0 msgr2=0x7fab4810d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 --2- 172.21.15.162:0/1907305243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab480f73a0 0x7fab4810d070 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7fab38009400 tx=0x7fab38009cb0).stop 2024-02-13T00:50:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 -- 172.21.15.162:0/1907305243 shutdown_connections 2024-02-13T00:50:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 --2- 172.21.15.162:0/1907305243 >> v2:172.21.15.162:6800/2866534053 conn(0x7fab30062c90 0x7fab30065150 unknown :-1 s=CLOSED pgs=2971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 --2- 172.21.15.162:0/1907305243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab480fd490 0x7fab48107210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 --2- 172.21.15.162:0/1907305243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab480f73a0 0x7fab4810d070 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 --2- 172.21.15.162:0/1907305243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab480fcb30 0x7fab4810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 -- 172.21.15.162:0/1907305243 >> 172.21.15.162:0/1907305243 conn(0x7fab4806eb90 msgr2=0x7fab480fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 -- 172.21.15.162:0/1907305243 shutdown_connections 2024-02-13T00:50:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 -- 172.21.15.162:0/1907305243 wait complete. 2024-02-13T00:50:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:07.026+0000 7fab4e23e700 1 librados: shutdown 2024-02-13T00:50:07.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:05.640999+0000 mon.a (mon.0) 3060 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:07.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:05.777362+0000 mgr.y (mgr.14544) 506 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:06.501199+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-99"}]': finished 2024-02-13T00:50:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:06.501270+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e592: 8 total, 8 up, 4 in 2024-02-13T00:50:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:06.502748+0000 mon.c (mon.2) 467 : audit [INF] from='client.? 172.21.15.162:0/334137482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:06.503039+0000 mon.a (mon.0) 3063 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]: dispatch 2024-02-13T00:50:07.219 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:07.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:07.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:07.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:07.353 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:07.376 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:07.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:07.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:07.478 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:07.506 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:07.545 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:07.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:07.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:07.680 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:07.700 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:07.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:07.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:07.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:06.485724+0000 mgr.y (mgr.14544) 507 : cluster [DBG] pgmap v887: 172 pgs: 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:07.901 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:07.025974+0000 mon.c (mon.2) 468 : audit [DBG] from='client.? 172.21.15.162:0/1907305243' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-13T00:50:07.902 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:07.506627+0000 mon.a (mon.0) 3064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]': finished 2024-02-13T00:50:07.902 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:07.506696+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e593: 8 total, 8 up, 4 in 2024-02-13T00:50:07.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:06.485724+0000 mgr.y (mgr.14544) 507 : cluster [DBG] pgmap v887: 172 pgs: 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:07.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:07.025974+0000 mon.c (mon.2) 468 : audit [DBG] from='client.? 172.21.15.162:0/1907305243' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-13T00:50:07.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:07.506627+0000 mon.a (mon.0) 3064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]': finished 2024-02-13T00:50:07.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:07.506696+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e593: 8 total, 8 up, 4 in 2024-02-13T00:50:07.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:07.904 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:07.904 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:07.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:07.997 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-02-13T00:50:07.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:50:08.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:08.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:06.485724+0000 mgr.y (mgr.14544) 507 : cluster [DBG] pgmap v887: 172 pgs: 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:08.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:07.025974+0000 mon.c (mon.2) 468 : audit [DBG] from='client.? 172.21.15.162:0/1907305243' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-13T00:50:08.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:07.506627+0000 mon.a (mon.0) 3064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-99"}]': finished 2024-02-13T00:50:08.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:07.506696+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e593: 8 total, 8 up, 4 in 2024-02-13T00:50:08.032 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:08.050 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:08.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:08.154 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:08.180 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:08.198 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:08.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:08.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:08.328 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:08.476 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:08.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:08.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:08.531 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:08.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:08.606 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:08.626 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:08.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:08.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:08.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:08.756 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:08.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:09.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:50:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:50:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:50:09.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:09.084 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:09.103 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:09.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:09.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:09.232 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:09.251 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:09.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:09.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:09.380 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:09.398 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:09.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:09.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:09.532 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:09.570 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:09.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:09.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:09.674 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:09.701 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:08.486183+0000 mgr.y (mgr.14544) 508 : cluster [DBG] pgmap v890: 172 pgs: 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:08.507088+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e594: 8 total, 8 up, 4 in 2024-02-13T00:50:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:08.508365+0000 mon.c (mon.2) 469 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:50:09.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:08.509516+0000 mon.a (mon.0) 3067 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:50:09.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:08.486183+0000 mgr.y (mgr.14544) 508 : cluster [DBG] pgmap v890: 172 pgs: 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:08.507088+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e594: 8 total, 8 up, 4 in 2024-02-13T00:50:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:08.508365+0000 mon.c (mon.2) 469 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:50:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:08.509516+0000 mon.a (mon.0) 3067 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:50:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:08.486183+0000 mgr.y (mgr.14544) 508 : cluster [DBG] pgmap v890: 172 pgs: 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:08.507088+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e594: 8 total, 8 up, 4 in 2024-02-13T00:50:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:08.508365+0000 mon.c (mon.2) 469 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:50:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:08.509516+0000 mon.a (mon.0) 3067 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:50:09.859 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:09.860 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:09.915 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:50:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:50:09.916 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-13T00:50:09.916 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:50:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:09.505973+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]': finished 2024-02-13T00:50:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:09.506038+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e595: 8 total, 8 up, 4 in 2024-02-13T00:50:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:09.508637+0000 mon.c (mon.2) 470 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:50:10.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:09.513529+0000 mon.a (mon.0) 3070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:50:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:09.505973+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]': finished 2024-02-13T00:50:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:09.506038+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e595: 8 total, 8 up, 4 in 2024-02-13T00:50:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:09.508637+0000 mon.c (mon.2) 470 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:50:10.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:09.513529+0000 mon.a (mon.0) 3070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:50:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:09.505973+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-84057-94"}]': finished 2024-02-13T00:50:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:09.506038+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e595: 8 total, 8 up, 4 in 2024-02-13T00:50:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:09.508637+0000 mon.c (mon.2) 470 : audit [INF] from='client.? 172.21.15.162:0/4140841795' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:50:10.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:09.513529+0000 mon.a (mon.0) 3070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]: dispatch 2024-02-13T00:50:10.851 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:10.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:10.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:10.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:10.984 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:11.333 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:11.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:11.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:11.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:11.464 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:10.486964+0000 mgr.y (mgr.14544) 509 : cluster [DBG] pgmap v893: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-13T00:50:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:10.509168+0000 mon.a (mon.0) 3071 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:50:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:10.520400+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]': finished 2024-02-13T00:50:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:10.520470+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e596: 8 total, 8 up, 4 in 2024-02-13T00:50:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:10.543385+0000 mon.a (mon.0) 3074 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:10.543999+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:10.544312+0000 mon.a (mon.0) 3076 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:50:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:10.486964+0000 mgr.y (mgr.14544) 509 : cluster [DBG] pgmap v893: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-13T00:50:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:10.509168+0000 mon.a (mon.0) 3071 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:50:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:10.520400+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]': finished 2024-02-13T00:50:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:10.520470+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e596: 8 total, 8 up, 4 in 2024-02-13T00:50:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:10.543385+0000 mon.a (mon.0) 3074 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:10.543999+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:10.544312+0000 mon.a (mon.0) 3076 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:50:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:10.486964+0000 mgr.y (mgr.14544) 509 : cluster [DBG] pgmap v893: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-13T00:50:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:10.509168+0000 mon.a (mon.0) 3071 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:50:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:10.520400+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-84057-94"}]': finished 2024-02-13T00:50:11.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:10.520470+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e596: 8 total, 8 up, 4 in 2024-02-13T00:50:11.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:10.543385+0000 mon.a (mon.0) 3074 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:11.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:10.543999+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:11.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:10.544312+0000 mon.a (mon.0) 3076 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-13T00:50:12.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:11.525309+0000 mon.a (mon.0) 3077 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:50:12.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:11.525362+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e597: 8 total, 8 up, 4 in 2024-02-13T00:50:12.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:11.525822+0000 mon.a (mon.0) 3079 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:12.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:11.525309+0000 mon.a (mon.0) 3077 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:50:12.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:11.525362+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e597: 8 total, 8 up, 4 in 2024-02-13T00:50:12.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:11.525822+0000 mon.a (mon.0) 3079 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:12.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:11.525309+0000 mon.a (mon.0) 3077 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-84057-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-13T00:50:12.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:11.525362+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e597: 8 total, 8 up, 4 in 2024-02-13T00:50:12.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:11.525822+0000 mon.a (mon.0) 3079 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:13.414 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:13.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:13.518 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:13.553 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:13.574 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:13.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:13.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:13.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.670+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/1854399390 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fa40f8500 msgr2=0x7f5fa4104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:13.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.670+0000 7f5fa8aa9700 1 --2- 172.21.15.162:0/1854399390 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fa40f8500 0x7f5fa4104bd0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 rx=0x7f5f9c009be0 tx=0x7f5f9c004d10).stop 2024-02-13T00:50:13.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.670+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/1854399390 shutdown_connections 2024-02-13T00:50:13.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.670+0000 7f5fa8aa9700 1 --2- 172.21.15.162:0/1854399390 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fa40f8500 0x7f5fa4104bd0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:13.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.670+0000 7f5fa8aa9700 1 --2- 172.21.15.162:0/1854399390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fa40f9b70 0x7f5fa40f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:13.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.670+0000 7f5fa8aa9700 1 --2- 172.21.15.162:0/1854399390 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fa40fa440 0x7f5fa40f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:13.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.670+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/1854399390 >> 172.21.15.162:0/1854399390 conn(0x7f5fa40f38c0 msgr2=0x7f5fa40f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:13.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.670+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/1854399390 shutdown_connections 2024-02-13T00:50:13.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.670+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/1854399390 wait complete. 2024-02-13T00:50:13.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.671+0000 7f5fa8aa9700 1 librados: starting msgr at 2024-02-13T00:50:13.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.671+0000 7f5fa8aa9700 1 librados: starting objecter 2024-02-13T00:50:13.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.671+0000 7f5fa8aa9700 1 Processor -- start 2024-02-13T00:50:13.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.671+0000 7f5fa8aa9700 1 -- start start 2024-02-13T00:50:13.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.671+0000 7f5fa8aa9700 1 librados: setting wanted keys 2024-02-13T00:50:13.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.671+0000 7f5fa8aa9700 1 librados: calling monclient init 2024-02-13T00:50:13.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa8aa9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fa40f8500 0x7f5fa406b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:13.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa8aa9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fa40f9b70 0x7f5fa406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:13.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa8aa9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fa40fa440 0x7f5fa406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:13.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa8aa9700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa41165d0 con 0x7f5fa40f9b70 2024-02-13T00:50:13.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa8aa9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa4116740 con 0x7f5fa40fa440 2024-02-13T00:50:13.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa8aa9700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa4116900 con 0x7f5fa40f8500 2024-02-13T00:50:13.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa259c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fa40f8500 0x7f5fa406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:13.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa1d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fa40f9b70 0x7f5fa406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:13.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa259c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fa40f8500 0x7f5fa406b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:35760/0 (socket says 172.21.15.162:35760) 2024-02-13T00:50:13.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa259c700 1 -- 172.21.15.162:0/2250589568 learned_addr learned my addr 172.21.15.162:0/2250589568 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:13.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa2d9d700 1 --2- 172.21.15.162:0/2250589568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fa40fa440 0x7f5fa406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:13.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa259c700 1 -- 172.21.15.162:0/2250589568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fa40fa440 msgr2=0x7f5fa406bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa259c700 1 --2- 172.21.15.162:0/2250589568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fa40fa440 0x7f5fa406bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa259c700 1 -- 172.21.15.162:0/2250589568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fa40f9b70 msgr2=0x7f5fa406b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa259c700 1 --2- 172.21.15.162:0/2250589568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fa40f9b70 0x7f5fa406b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa259c700 1 -- 172.21.15.162:0/2250589568 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c0097c0 con 0x7f5fa40f8500 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa259c700 1 --2- 172.21.15.162:0/2250589568 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fa40f8500 0x7f5fa406b170 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7f5f9400e970 tx=0x7f5f9400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5f937fe700 10 client.?.objecter ms_handle_connect 0x7f5fa40f8500 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5f937fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5f937fe700 1 -- 172.21.15.162:0/2250589568 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94004660 con 0x7f5fa40f8500 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/2250589568 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa4116be0 con 0x7f5fa40f8500 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5f937fe700 1 -- 172.21.15.162:0/2250589568 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f9400f040 con 0x7f5fa40f8500 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5f937fe700 1 -- 172.21.15.162:0/2250589568 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94014790 con 0x7f5fa40f8500 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa8aa9700 10 client.37865.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/2250589568 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa4117120 con 0x7f5fa40f8500 2024-02-13T00:50:13.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.672+0000 7f5fa8aa9700 1 librados: init done 2024-02-13T00:50:13.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.673+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/2250589568 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa40475e0 con 0x7f5fa40f8500 2024-02-13T00:50:13.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.674+0000 7f5f937fe700 1 -- 172.21.15.162:0/2250589568 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f5f940090d0 con 0x7f5fa40f8500 2024-02-13T00:50:13.680 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:13.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.674+0000 7f5f937fe700 1 --2- 172.21.15.162:0/2250589568 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5f8c062c90 0x7f5f8c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:13.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.674+0000 7f5f937fe700 1 -- 172.21.15.162:0/2250589568 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(599..599 src has 449..599) v4 ==== 8089+0+0 (secure 0 0 0) 0x7f5f9407cd30 con 0x7f5fa40f8500 2024-02-13T00:50:13.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.674+0000 7f5f937fe700 10 client.37865.objecter ms_dispatch 0x7f5fa4001550 osd_map(599..599 src has 449..599) v4 2024-02-13T00:50:13.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.674+0000 7f5f937fe700 3 client.37865.objecter handle_osd_map got epochs [599,599] > 0 2024-02-13T00:50:13.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.674+0000 7f5f937fe700 3 client.37865.objecter handle_osd_map decoding full epoch 599 2024-02-13T00:50:13.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.674+0000 7f5fa1d9b700 1 --2- 172.21.15.162:0/2250589568 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5f8c062c90 0x7f5f8c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:13.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.674+0000 7f5f937fe700 20 client.37865.objecter dump_active .. 0 homeless 2024-02-13T00:50:13.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.675+0000 7f5f937fe700 10 client.37865.objecter ms_handle_connect 0x7f5f8c062c90 2024-02-13T00:50:13.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.675+0000 7f5fa1d9b700 1 --2- 172.21.15.162:0/2250589568 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5f8c062c90 0x7f5f8c065150 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 rx=0x7f5fa406c910 tx=0x7f5f9800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:13.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.679+0000 7f5f937fe700 1 -- 172.21.15.162:0/2250589568 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5f9404af60 con 0x7f5fa40f8500 2024-02-13T00:50:13.712 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:13.732 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:13.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:13.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:13.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:12.487311+0000 mgr.y (mgr.14544) 510 : cluster [DBG] pgmap v896: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-13T00:50:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:12.696208+0000 mon.c (mon.2) 471 : audit [DBG] from='client.? 172.21.15.162:0/1027175019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:12.778019+0000 mon.a (mon.0) 3080 : cluster [DBG] osdmap e598: 8 total, 8 up, 4 in 2024-02-13T00:50:13.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:12.487311+0000 mgr.y (mgr.14544) 510 : cluster [DBG] pgmap v896: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-13T00:50:13.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:12.696208+0000 mon.c (mon.2) 471 : audit [DBG] from='client.? 172.21.15.162:0/1027175019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:13.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:12.778019+0000 mon.a (mon.0) 3080 : cluster [DBG] osdmap e598: 8 total, 8 up, 4 in 2024-02-13T00:50:13.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:12.487311+0000 mgr.y (mgr.14544) 510 : cluster [DBG] pgmap v896: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-13T00:50:13.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:12.696208+0000 mon.c (mon.2) 471 : audit [DBG] from='client.? 172.21.15.162:0/1027175019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:13.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:12.778019+0000 mon.a (mon.0) 3080 : cluster [DBG] osdmap e598: 8 total, 8 up, 4 in 2024-02-13T00:50:13.836 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:13.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:13.863+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/2250589568 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f5fa406c650 con 0x7f5fa40f8500 2024-02-13T00:50:13.868 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:13.992 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:13.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:13.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:14.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:14.123 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:14.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 -- 172.21.15.162:0/2041277533 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f04f40fe5b0 msgr2=0x7f04f40f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/2041277533 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f04f40fe5b0 0x7f04f40f7de0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 rx=0x7f04e8004870 tx=0x7f04e8009cb0).stop 2024-02-13T00:50:14.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 -- 172.21.15.162:0/2041277533 shutdown_connections 2024-02-13T00:50:14.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/2041277533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04f40f84c0 0x7f04f4104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/2041277533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04f40fdc50 0x7f04f40fe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/2041277533 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f04f40fe5b0 0x7f04f40f7de0 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 -- 172.21.15.162:0/2041277533 >> 172.21.15.162:0/2041277533 conn(0x7f04f40f38a0 msgr2=0x7f04f40f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:14.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 -- 172.21.15.162:0/2041277533 shutdown_connections 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 -- 172.21.15.162:0/2041277533 wait complete. 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 librados: starting msgr at 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 librados: starting objecter 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 Processor -- start 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 -- start start 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 librados: setting wanted keys 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.692+0000 7f04fa7c6700 1 librados: calling monclient init 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04fa7c6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f04f40f84c0 0x7f04f406b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04fa7c6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04f40fdc50 0x7f04f406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04fa7c6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04f40fe5b0 0x7f04f406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04fa7c6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f41165b0 con 0x7f04f40fdc50 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04fa7c6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f4116720 con 0x7f04f40fe5b0 2024-02-13T00:50:14.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04fa7c6700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f41168e0 con 0x7f04f40f84c0 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f3fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f04f40f84c0 0x7f04f406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f3fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f04f40f84c0 0x7f04f406b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:35736/0 (socket says 172.21.15.162:35736) 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f3fff700 1 -- 172.21.15.162:0/1027175019 learned_addr learned my addr 172.21.15.162:0/1027175019 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f37fe700 1 --2- 172.21.15.162:0/1027175019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04f40fdc50 0x7f04f406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f8d63700 1 --2- 172.21.15.162:0/1027175019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04f40fe5b0 0x7f04f406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f3fff700 1 -- 172.21.15.162:0/1027175019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04f40fe5b0 msgr2=0x7f04f406bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f3fff700 1 --2- 172.21.15.162:0/1027175019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04f40fe5b0 0x7f04f406bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f3fff700 1 -- 172.21.15.162:0/1027175019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04f40fdc50 msgr2=0x7f04f406b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f3fff700 1 --2- 172.21.15.162:0/1027175019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04f40fdc50 0x7f04f406b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f3fff700 1 -- 172.21.15.162:0/1027175019 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e8005040 con 0x7f04f40f84c0 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f3fff700 1 --2- 172.21.15.162:0/1027175019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f04f40f84c0 0x7f04f406b170 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7f04e40041d0 tx=0x7f04e4009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f17fa700 10 client.?.objecter ms_handle_connect 0x7f04f40f84c0 2024-02-13T00:50:14.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f17fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e4007b40 con 0x7f04f40f84c0 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f04e400f040 con 0x7f04f40f84c0 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e4014860 con 0x7f04f40f84c0 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.693+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f4116bc0 con 0x7f04f40f84c0 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.694+0000 7f04fa7c6700 10 client.37859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.694+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f4117060 con 0x7f04f40f84c0 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.694+0000 7f04fa7c6700 1 librados: init done 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.694+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f04f4107140 con 0x7f04f40f84c0 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.695+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f04e4014c50 con 0x7f04f40f84c0 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.695+0000 7f04f17fa700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6800/2866534053 conn(0x7f04dc062c90 0x7f04dc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.695+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(598..598 src has 449..598) v4 ==== 7670+0+0 (secure 0 0 0) 0x7f04e407cb20 con 0x7f04f40f84c0 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.695+0000 7f04f37fe700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6800/2866534053 conn(0x7f04dc062c90 0x7f04dc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.695+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 osd_map(598..598 src has 449..598) v4 2024-02-13T00:50:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.695+0000 7f04f17fa700 3 client.37859.objecter handle_osd_map got epochs [598,598] > 0 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.695+0000 7f04f17fa700 3 client.37859.objecter handle_osd_map decoding full epoch 598 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04f17fa700 20 client.37859.objecter dump_active .. 0 homeless 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v598) v1 ==== 54+0+16 (secure 0 0 0) 0x7f04e4081220 con 0x7f04f40f84c0 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04f17fa700 10 client.37859.objecter ms_handle_connect 0x7f04dc062c90 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04f37fe700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6800/2866534053 conn(0x7f04dc062c90 0x7f04dc065150 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 rx=0x7f04e8009a00 tx=0x7f04e800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 1 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6801/2103743338 conn(0x7f04f4028590 0x7f04f40fe990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f406c650 osd=0 3 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f406c650 osd=0 3 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f406c650 osd=0 3 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 1 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f406c650 osd=0 3 2024-02-13T00:50:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 0 1 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f406c650 osd=0 4 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04fa7c6700 10 client.37859.objecter _send_command 1 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f4100000 con 0x7f04f4028590 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04f8d63700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6801/2103743338 conn(0x7f04f4028590 0x7f04f40fe990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04f17fa700 10 client.37859.objecter ms_handle_connect 0x7f04f4028590 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.696+0000 7f04f8d63700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6801/2103743338 conn(0x7f04f4028590 0x7f04f40fe990 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.697+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f04f4100000 con 0x7f04f4028590 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.697+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.697+0000 7f04f17fa700 10 client.37859.objecter _finish_command 1 = system:0 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.697+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f406c650 osd=0 4 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.697+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 0 1 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.714+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.714+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 2 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.714+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f406c650 osd=0 3 2024-02-13T00:50:14.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.714+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f406c650 osd=0 3 2024-02-13T00:50:14.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.714+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f406c650 osd=0 3 2024-02-13T00:50:14.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.714+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 2 2024-02-13T00:50:14.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.714+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f406c650 osd=0 3 2024-02-13T00:50:14.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.714+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 0 2 2024-02-13T00:50:14.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.714+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f406c650 osd=0 4 2024-02-13T00:50:14.239 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:50:14.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.714+0000 7f04fa7c6700 10 client.37859.objecter _send_command 2 2024-02-13T00:50:14.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.714+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04f405a0e0 con 0x7f04f4028590 2024-02-13T00:50:14.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f04f405a0e0 con 0x7f04f4028590 2024-02-13T00:50:14.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:14.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04f17fa700 10 client.37859.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f406c650 osd=0 4 2024-02-13T00:50:14.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 0 2 2024-02-13T00:50:14.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:14.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 3 2024-02-13T00:50:14.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6805/150599780 conn(0x7f04f4059570 0x7f04f411b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:14.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4059400 osd=1 3 2024-02-13T00:50:14.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4059400 osd=1 3 2024-02-13T00:50:14.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4059400 osd=1 3 2024-02-13T00:50:14.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 3 2024-02-13T00:50:14.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f4059400 osd=1 3 2024-02-13T00:50:14.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 1 3 2024-02-13T00:50:14.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4059400 osd=1 4 2024-02-13T00:50:14.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04fa7c6700 10 client.37859.objecter _send_command 3 2024-02-13T00:50:14.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f40280f0 con 0x7f04f4059570 2024-02-13T00:50:14.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04f3fff700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6805/150599780 conn(0x7f04f4059570 0x7f04f411b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:14.241 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04f17fa700 10 client.37859.objecter ms_handle_connect 0x7f04f4059570 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.716+0000 7f04f3fff700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6805/150599780 conn(0x7f04f4059570 0x7f04f411b2d0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.717+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f04f40280f0 con 0x7f04f4059570 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.717+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.717+0000 7f04f17fa700 10 client.37859.objecter _finish_command 3 = system:0 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.717+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f4059400 osd=1 4 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.717+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 1 3 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.733+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.733+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 4 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.733+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4059400 osd=1 3 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.733+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4059400 osd=1 3 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.733+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4059400 osd=1 3 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.733+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 4 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.733+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f4059400 osd=1 3 2024-02-13T00:50:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.733+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 1 4 2024-02-13T00:50:14.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.733+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4059400 osd=1 4 2024-02-13T00:50:14.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.733+0000 7f04fa7c6700 10 client.37859.objecter _send_command 4 2024-02-13T00:50:14.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.733+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04f411b7c0 con 0x7f04f4059570 2024-02-13T00:50:14.243 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:50:14.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f04f411b7c0 con 0x7f04f4059570 2024-02-13T00:50:14.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:14.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04f17fa700 10 client.37859.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f4059400 osd=1 4 2024-02-13T00:50:14.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 1 4 2024-02-13T00:50:14.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:14.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 5 2024-02-13T00:50:14.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6809/3689719376 conn(0x7f04f40277c0 0x7f04f411eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:14.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4027650 osd=2 3 2024-02-13T00:50:14.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4027650 osd=2 3 2024-02-13T00:50:14.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4027650 osd=2 3 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 5 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f4027650 osd=2 3 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 2 5 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4027650 osd=2 4 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04fa7c6700 10 client.37859.objecter _send_command 5 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f40ffb60 con 0x7f04f40277c0 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04f37fe700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6809/3689719376 conn(0x7f04f40277c0 0x7f04f411eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04f17fa700 10 client.37859.objecter ms_handle_connect 0x7f04f40277c0 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.735+0000 7f04f37fe700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6809/3689719376 conn(0x7f04f40277c0 0x7f04f411eb20 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.736+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f04f40ffb60 con 0x7f04f40277c0 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.736+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.736+0000 7f04f17fa700 10 client.37859.objecter _finish_command 5 = system:0 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.736+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f4027650 osd=2 4 2024-02-13T00:50:14.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.736+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 2 5 2024-02-13T00:50:14.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.750+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:14.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.750+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 6 2024-02-13T00:50:14.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.750+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4027650 osd=2 3 2024-02-13T00:50:14.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.750+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4027650 osd=2 3 2024-02-13T00:50:14.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.750+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4027650 osd=2 3 2024-02-13T00:50:14.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.750+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 6 2024-02-13T00:50:14.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.750+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f4027650 osd=2 3 2024-02-13T00:50:14.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.750+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 2 6 2024-02-13T00:50:14.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.750+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4027650 osd=2 4 2024-02-13T00:50:14.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.750+0000 7f04fa7c6700 10 client.37859.objecter _send_command 6 2024-02-13T00:50:14.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.750+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04f411f010 con 0x7f04f40277c0 2024-02-13T00:50:14.246 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:50:14.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f04f411f010 con 0x7f04f40277c0 2024-02-13T00:50:14.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:14.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04f17fa700 10 client.37859.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f4027650 osd=2 4 2024-02-13T00:50:14.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 2 6 2024-02-13T00:50:14.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:14.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 7 2024-02-13T00:50:14.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6813/604971011 conn(0x7f04f40ff230 0x7f04f4120330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:14.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f40ff0c0 osd=3 3 2024-02-13T00:50:14.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f40ff0c0 osd=3 3 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f40ff0c0 osd=3 3 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 7 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f40ff0c0 osd=3 3 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 3 7 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f40ff0c0 osd=3 4 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04fa7c6700 10 client.37859.objecter _send_command 7 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f411c730 con 0x7f04f40ff230 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04f8d63700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6813/604971011 conn(0x7f04f40ff230 0x7f04f4120330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04f17fa700 10 client.37859.objecter ms_handle_connect 0x7f04f40ff230 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.752+0000 7f04f8d63700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6813/604971011 conn(0x7f04f40ff230 0x7f04f4120330 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.753+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f04f411c730 con 0x7f04f40ff230 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.753+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.753+0000 7f04f17fa700 10 client.37859.objecter _finish_command 7 = system:0 2024-02-13T00:50:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.753+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f40ff0c0 osd=3 4 2024-02-13T00:50:14.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.753+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 3 7 2024-02-13T00:50:14.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.767+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:14.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.767+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 8 2024-02-13T00:50:14.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.767+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f40ff0c0 osd=3 3 2024-02-13T00:50:14.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.767+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f40ff0c0 osd=3 3 2024-02-13T00:50:14.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.767+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f40ff0c0 osd=3 3 2024-02-13T00:50:14.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.767+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 8 2024-02-13T00:50:14.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.767+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f40ff0c0 osd=3 3 2024-02-13T00:50:14.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.767+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 3 8 2024-02-13T00:50:14.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.767+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f40ff0c0 osd=3 4 2024-02-13T00:50:14.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.767+0000 7f04fa7c6700 10 client.37859.objecter _send_command 8 2024-02-13T00:50:14.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.767+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04f411c980 con 0x7f04f40ff230 2024-02-13T00:50:14.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.768+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f04f411c980 con 0x7f04f40ff230 2024-02-13T00:50:14.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.768+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:14.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.768+0000 7f04f17fa700 10 client.37859.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.768+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f40ff0c0 osd=3 4 2024-02-13T00:50:14.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.768+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 3 8 2024-02-13T00:50:14.250 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:50:14.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:14.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 9 2024-02-13T00:50:14.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6800/3344263388 conn(0x7f04f4121500 0x7f04f411bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:14.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f411c980 osd=4 3 2024-02-13T00:50:14.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f411c980 osd=4 3 2024-02-13T00:50:14.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f411c980 osd=4 3 2024-02-13T00:50:14.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 9 2024-02-13T00:50:14.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f411c980 osd=4 3 2024-02-13T00:50:14.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 4 9 2024-02-13T00:50:14.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f411c980 osd=4 4 2024-02-13T00:50:14.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04fa7c6700 10 client.37859.objecter _send_command 9 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f4128240 con 0x7f04f4121500 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04f3fff700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6800/3344263388 conn(0x7f04f4121500 0x7f04f411bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04f17fa700 10 client.37859.objecter ms_handle_connect 0x7f04f4121500 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.769+0000 7f04f3fff700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6800/3344263388 conn(0x7f04f4121500 0x7f04f411bc60 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.770+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f04f4128240 con 0x7f04f4121500 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.770+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.770+0000 7f04f17fa700 10 client.37859.objecter _finish_command 9 = system:0 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.770+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f411c980 osd=4 4 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.770+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 4 9 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.785+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.785+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 10 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.785+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f411c980 osd=4 3 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.785+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f411c980 osd=4 3 2024-02-13T00:50:14.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.785+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f411c980 osd=4 3 2024-02-13T00:50:14.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.785+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 10 2024-02-13T00:50:14.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.785+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f411c980 osd=4 3 2024-02-13T00:50:14.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.785+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 4 10 2024-02-13T00:50:14.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.785+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f411c980 osd=4 4 2024-02-13T00:50:14.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.785+0000 7f04fa7c6700 10 client.37859.objecter _send_command 10 2024-02-13T00:50:14.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.785+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04f4128240 con 0x7f04f4121500 2024-02-13T00:50:14.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f04f4128240 con 0x7f04f4121500 2024-02-13T00:50:14.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:14.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04f17fa700 10 client.37859.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f411c980 osd=4 4 2024-02-13T00:50:14.253 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:50:14.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 4 10 2024-02-13T00:50:14.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:14.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 11 2024-02-13T00:50:14.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6804/2277917372 conn(0x7f04f41275d0 0x7f04f4127a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:14.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4128240 osd=5 3 2024-02-13T00:50:14.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4128240 osd=5 3 2024-02-13T00:50:14.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4128240 osd=5 3 2024-02-13T00:50:14.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 11 2024-02-13T00:50:14.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f4128240 osd=5 3 2024-02-13T00:50:14.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 5 11 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4128240 osd=5 4 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04fa7c6700 10 client.37859.objecter _send_command 11 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.787+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f412b5f0 con 0x7f04f41275d0 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.788+0000 7f04f37fe700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6804/2277917372 conn(0x7f04f41275d0 0x7f04f4127a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.788+0000 7f04f17fa700 10 client.37859.objecter ms_handle_connect 0x7f04f41275d0 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.788+0000 7f04f37fe700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6804/2277917372 conn(0x7f04f41275d0 0x7f04f4127a50 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.789+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f04f412b5f0 con 0x7f04f41275d0 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.789+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.789+0000 7f04f17fa700 10 client.37859.objecter _finish_command 11 = system:0 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.789+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f4128240 osd=5 4 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.789+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 5 11 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.804+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.804+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 12 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.804+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4128240 osd=5 3 2024-02-13T00:50:14.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.804+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4128240 osd=5 3 2024-02-13T00:50:14.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.804+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4128240 osd=5 3 2024-02-13T00:50:14.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.804+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 12 2024-02-13T00:50:14.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.804+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f4128240 osd=5 3 2024-02-13T00:50:14.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.804+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 5 12 2024-02-13T00:50:14.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.804+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4128240 osd=5 4 2024-02-13T00:50:14.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.804+0000 7f04fa7c6700 10 client.37859.objecter _send_command 12 2024-02-13T00:50:14.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.804+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04f4125a20 con 0x7f04f41275d0 2024-02-13T00:50:14.256 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:50:14.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f04f4125a20 con 0x7f04f41275d0 2024-02-13T00:50:14.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:14.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04f17fa700 10 client.37859.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f4128240 osd=5 4 2024-02-13T00:50:14.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 5 12 2024-02-13T00:50:14.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:14.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 13 2024-02-13T00:50:14.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6808/1506640341 conn(0x7f04f41260d0 0x7f04f4126530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:14.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4125a20 osd=6 3 2024-02-13T00:50:14.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4125a20 osd=6 3 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4125a20 osd=6 3 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 13 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f4125a20 osd=6 3 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 6 13 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4125a20 osd=6 4 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04fa7c6700 10 client.37859.objecter _send_command 13 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f4126ef0 con 0x7f04f41260d0 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.806+0000 7f04f8d63700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6808/1506640341 conn(0x7f04f41260d0 0x7f04f4126530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.807+0000 7f04f17fa700 10 client.37859.objecter ms_handle_connect 0x7f04f41260d0 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.807+0000 7f04f8d63700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6808/1506640341 conn(0x7f04f41260d0 0x7f04f4126530 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.808+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f04f4126ef0 con 0x7f04f41260d0 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.808+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.808+0000 7f04f17fa700 10 client.37859.objecter _finish_command 13 = system:0 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.808+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f4125a20 osd=6 4 2024-02-13T00:50:14.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.808+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 6 13 2024-02-13T00:50:14.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.825+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:14.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.825+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 14 2024-02-13T00:50:14.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.825+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4125a20 osd=6 3 2024-02-13T00:50:14.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.825+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4125a20 osd=6 3 2024-02-13T00:50:14.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.825+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4125a20 osd=6 3 2024-02-13T00:50:14.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.825+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 14 2024-02-13T00:50:14.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.825+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f4125a20 osd=6 3 2024-02-13T00:50:14.259 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.825+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 6 14 2024-02-13T00:50:14.259 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:50:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.825+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4125a20 osd=6 4 2024-02-13T00:50:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.825+0000 7f04fa7c6700 10 client.37859.objecter _send_command 14 2024-02-13T00:50:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.825+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04f4126ef0 con 0x7f04f41260d0 2024-02-13T00:50:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f04f4126ef0 con 0x7f04f41260d0 2024-02-13T00:50:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04f17fa700 10 client.37859.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f4125a20 osd=6 4 2024-02-13T00:50:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 6 14 2024-02-13T00:50:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 15 2024-02-13T00:50:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6812/1964192772 conn(0x7f04f412c990 0x7f04f412ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:14.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4126ef0 osd=7 3 2024-02-13T00:50:14.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4126ef0 osd=7 3 2024-02-13T00:50:14.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4126ef0 osd=7 3 2024-02-13T00:50:14.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 15 2024-02-13T00:50:14.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f4126ef0 osd=7 3 2024-02-13T00:50:14.261 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 7 15 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4126ef0 osd=7 4 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04fa7c6700 10 client.37859.objecter _send_command 15 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.827+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f41319d0 con 0x7f04f412c990 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.828+0000 7f04f3fff700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6812/1964192772 conn(0x7f04f412c990 0x7f04f412ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.828+0000 7f04f17fa700 10 client.37859.objecter ms_handle_connect 0x7f04f412c990 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.828+0000 7f04f3fff700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6812/1964192772 conn(0x7f04f412c990 0x7f04f412ed80 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.829+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f04f41319d0 con 0x7f04f412c990 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.829+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.829+0000 7f04f17fa700 10 client.37859.objecter _finish_command 15 = system:0 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.829+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f4126ef0 osd=7 4 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.829+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 7 15 2024-02-13T00:50:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.845+0000 7f04fa7c6700 10 client.37859.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.845+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign -1 16 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.845+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4126ef0 osd=7 3 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.845+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4126ef0 osd=7 3 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.845+0000 7f04fa7c6700 20 client.37859.objecter _get_session s=0x7f04f4126ef0 osd=7 3 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.845+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_remove -1 16 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.845+0000 7f04fa7c6700 20 client.37859.objecter get_session s=0x7f04f4126ef0 osd=7 3 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.845+0000 7f04fa7c6700 15 client.37859.objecter _session_command_op_assign 7 16 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.845+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4126ef0 osd=7 4 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.845+0000 7f04fa7c6700 10 client.37859.objecter _send_command 16 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.845+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04f412f270 con 0x7f04f412c990 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04f17fa700 1 -- 172.21.15.162:0/1027175019 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f04f412f270 con 0x7f04f412c990 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04f17fa700 10 client.37859.objecter ms_dispatch 0x7f04f4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04f17fa700 10 client.37859.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04f17fa700 20 client.37859.objecter put_session s=0x7f04f4126ef0 osd=7 4 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04f17fa700 15 client.37859.objecter _session_command_op_remove 7 16 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 10 librados: watch_flush enter 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 10 librados: watch_flush exit 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 10 client.37859.objecter close_session for osd.0 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6801/2103743338 conn(0x7f04f4028590 msgr2=0x7f04f40fe990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6801/2103743338 conn(0x7f04f4028590 0x7f04f40fe990 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f406c650 osd=0 1 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 10 client.37859.objecter close_session for osd.1 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6805/150599780 conn(0x7f04f4059570 msgr2=0x7f04f411b2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6805/150599780 conn(0x7f04f4059570 0x7f04f411b2d0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4059400 osd=1 1 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 10 client.37859.objecter close_session for osd.2 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6809/3689719376 conn(0x7f04f40277c0 msgr2=0x7f04f411eb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6809/3689719376 conn(0x7f04f40277c0 0x7f04f411eb20 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4027650 osd=2 1 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 10 client.37859.objecter close_session for osd.3 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6813/604971011 conn(0x7f04f40ff230 msgr2=0x7f04f4120330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6813/604971011 conn(0x7f04f40ff230 0x7f04f4120330 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f40ff0c0 osd=3 1 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 10 client.37859.objecter close_session for osd.4 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6800/3344263388 conn(0x7f04f4121500 msgr2=0x7f04f411bc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6800/3344263388 conn(0x7f04f4121500 0x7f04f411bc60 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f411c980 osd=4 1 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 10 client.37859.objecter close_session for osd.5 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6804/2277917372 conn(0x7f04f41275d0 msgr2=0x7f04f4127a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6804/2277917372 conn(0x7f04f41275d0 0x7f04f4127a50 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4128240 osd=5 1 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04f8d63700 1 -- 172.21.15.162:0/1027175019 reap_dead start 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 10 client.37859.objecter close_session for osd.6 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6808/1506640341 conn(0x7f04f41260d0 msgr2=0x7f04f4126530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6808/1506640341 conn(0x7f04f41260d0 0x7f04f4126530 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04f8d63700 1 -- 172.21.15.162:0/1027175019 reap_dead start 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4125a20 osd=6 1 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 10 client.37859.objecter close_session for osd.7 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6812/1964192772 conn(0x7f04f412c990 msgr2=0x7f04f412ed80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.181:6812/1964192772 conn(0x7f04f412c990 0x7f04f412ed80 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 20 client.37859.objecter put_session s=0x7f04f4126ef0 osd=7 1 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 20 client.37859.objecter shutdown clearing up homeless session... 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 10 client.37859.objecter successfully canceled tick 2024-02-13T00:50:14.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04f8d63700 1 -- 172.21.15.162:0/1027175019 reap_dead start 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04f8d63700 1 -- 172.21.15.162:0/1027175019 reap_dead start 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6800/2866534053 conn(0x7f04dc062c90 msgr2=0x7f04dc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6800/2866534053 conn(0x7f04dc062c90 0x7f04dc065150 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 rx=0x7f04e8009a00 tx=0x7f04e800b040).stop 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f04f40f84c0 msgr2=0x7f04f406b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f04f40f84c0 0x7f04f406b170 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7f04e40041d0 tx=0x7f04e4009f60).stop 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 shutdown_connections 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f04f40f84c0 0x7f04f406b170 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 --2- 172.21.15.162:0/1027175019 >> v2:172.21.15.162:6800/2866534053 conn(0x7f04dc062c90 0x7f04dc065150 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.847+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 >> 172.21.15.162:0/1027175019 conn(0x7f04f40f38a0 msgr2=0x7f04f40fba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.848+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 shutdown_connections 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.848+0000 7f04fa7c6700 1 -- 172.21.15.162:0/1027175019 wait complete. 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:12.848+0000 7f04fa7c6700 1 librados: shutdown 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.267 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.268 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.268 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.268 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.268 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.268 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:50:14.472 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:14.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:14.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.540+0000 7f5f937fe700 1 -- 172.21.15.162:0/2250589568 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v600) v1 ==== 91+0+0 (secure 0 0 0) 0x7f5f94019070 con 0x7f5fa40f8500 2024-02-13T00:50:14.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.544+0000 7f5fa8aa9700 10 librados: watch_flush enter 2024-02-13T00:50:14.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.544+0000 7f5fa8aa9700 10 librados: watch_flush exit 2024-02-13T00:50:14.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.544+0000 7f5fa8aa9700 20 client.37865.objecter shutdown clearing up homeless session... 2024-02-13T00:50:14.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.544+0000 7f5fa8aa9700 10 client.37865.objecter successfully canceled tick 2024-02-13T00:50:14.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.544+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/2250589568 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5f8c062c90 msgr2=0x7f5f8c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.544+0000 7f5fa8aa9700 1 --2- 172.21.15.162:0/2250589568 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5f8c062c90 0x7f5f8c065150 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 rx=0x7f5fa406c910 tx=0x7f5f9800b040).stop 2024-02-13T00:50:14.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.544+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/2250589568 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fa40f8500 msgr2=0x7f5fa406b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:14.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.544+0000 7f5fa8aa9700 1 --2- 172.21.15.162:0/2250589568 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fa40f8500 0x7f5fa406b170 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7f5f9400e970 tx=0x7f5f9400b040).stop 2024-02-13T00:50:14.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.547+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/2250589568 shutdown_connections 2024-02-13T00:50:14.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.547+0000 7f5fa8aa9700 1 --2- 172.21.15.162:0/2250589568 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5f8c062c90 0x7f5f8c065150 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.547+0000 7f5fa8aa9700 1 --2- 172.21.15.162:0/2250589568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fa40fa440 0x7f5fa406bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.547+0000 7f5fa8aa9700 1 --2- 172.21.15.162:0/2250589568 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fa40f8500 0x7f5fa406b170 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.547+0000 7f5fa8aa9700 1 --2- 172.21.15.162:0/2250589568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fa40f9b70 0x7f5fa406b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:14.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.547+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/2250589568 >> 172.21.15.162:0/2250589568 conn(0x7f5fa40f38c0 msgr2=0x7f5fa40f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:14.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.552+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/2250589568 shutdown_connections 2024-02-13T00:50:14.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.552+0000 7f5fa8aa9700 1 -- 172.21.15.162:0/2250589568 wait complete. 2024-02-13T00:50:14.556 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:14.553+0000 7f5fa8aa9700 1 librados: shutdown 2024-02-13T00:50:14.559 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:50:14.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:14.604 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:13.537337+0000 mon.a (mon.0) 3081 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:50:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:13.537418+0000 mon.a (mon.0) 3082 : cluster [DBG] osdmap e599: 8 total, 8 up, 4 in 2024-02-13T00:50:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:13.830979+0000 osd.7 (osd.7) 110 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:50:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:13.831363+0000 osd.7 (osd.7) 111 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:50:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:13.864667+0000 mon.c (mon.2) 472 : audit [INF] from='client.? 172.21.15.162:0/2250589568' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:50:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:13.865049+0000 mon.a (mon.0) 3083 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:50:14.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:13.537337+0000 mon.a (mon.0) 3081 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:50:14.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:13.537418+0000 mon.a (mon.0) 3082 : cluster [DBG] osdmap e599: 8 total, 8 up, 4 in 2024-02-13T00:50:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:13.830979+0000 osd.7 (osd.7) 110 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:50:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:13.831363+0000 osd.7 (osd.7) 111 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:50:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:13.864667+0000 mon.c (mon.2) 472 : audit [INF] from='client.? 172.21.15.162:0/2250589568' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:50:14.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:13.865049+0000 mon.a (mon.0) 3083 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:50:14.958 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:14.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:15.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:13.537337+0000 mon.a (mon.0) 3081 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-84057-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:50:15.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:13.537418+0000 mon.a (mon.0) 3082 : cluster [DBG] osdmap e599: 8 total, 8 up, 4 in 2024-02-13T00:50:15.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:13.830979+0000 osd.7 (osd.7) 110 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:50:15.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:13.831363+0000 osd.7 (osd.7) 111 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:50:15.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:13.864667+0000 mon.c (mon.2) 472 : audit [INF] from='client.? 172.21.15.162:0/2250589568' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:50:15.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:13.865049+0000 mon.a (mon.0) 3083 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:50:15.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:15.092 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:15.557 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:14.487698+0000 mgr.y (mgr.14544) 511 : cluster [DBG] pgmap v899: 172 pgs: 8 unknown, 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:15.557 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:14.534897+0000 mon.a (mon.0) 3084 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:50:15.557 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:14.539555+0000 mon.a (mon.0) 3085 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:50:15.557 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:14.539632+0000 mon.a (mon.0) 3086 : cluster [DBG] osdmap e600: 8 total, 8 up, 4 in 2024-02-13T00:50:15.557 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:14.542135+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:50:15.557 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:14.723751+0000 osd.1 (osd.1) 251 : cluster [DBG] 6.1 scrub starts 2024-02-13T00:50:15.558 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:14.726657+0000 osd.1 (osd.1) 252 : cluster [DBG] 6.1 scrub ok 2024-02-13T00:50:15.558 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:14.846858+0000 osd.0 (osd.0) 225 : cluster [DBG] 28.d scrub starts 2024-02-13T00:50:15.558 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:14.849684+0000 osd.0 (osd.0) 226 : cluster [DBG] 28.d scrub ok 2024-02-13T00:50:15.558 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:15.500505+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:50:15.740 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:15.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:15.500599+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e601: 8 total, 8 up, 4 in 2024-02-13T00:50:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:15.531284+0000 mon.a (mon.0) 3090 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:50:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:14.487698+0000 mgr.y (mgr.14544) 511 : cluster [DBG] pgmap v899: 172 pgs: 8 unknown, 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:14.534897+0000 mon.a (mon.0) 3084 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:50:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:14.539555+0000 mon.a (mon.0) 3085 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:50:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:14.539632+0000 mon.a (mon.0) 3086 : cluster [DBG] osdmap e600: 8 total, 8 up, 4 in 2024-02-13T00:50:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:14.542135+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:50:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:14.723751+0000 osd.1 (osd.1) 251 : cluster [DBG] 6.1 scrub starts 2024-02-13T00:50:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:14.726657+0000 osd.1 (osd.1) 252 : cluster [DBG] 6.1 scrub ok 2024-02-13T00:50:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:14.846858+0000 osd.0 (osd.0) 225 : cluster [DBG] 28.d scrub starts 2024-02-13T00:50:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:14.849684+0000 osd.0 (osd.0) 226 : cluster [DBG] 28.d scrub ok 2024-02-13T00:50:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:15.500505+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:50:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:15.500599+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e601: 8 total, 8 up, 4 in 2024-02-13T00:50:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:15.531284+0000 mon.a (mon.0) 3090 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:50:15.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:15.874 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:16.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:50:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:14.487698+0000 mgr.y (mgr.14544) 511 : cluster [DBG] pgmap v899: 172 pgs: 8 unknown, 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:14.534897+0000 mon.a (mon.0) 3084 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:50:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:14.539555+0000 mon.a (mon.0) 3085 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:50:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:14.539632+0000 mon.a (mon.0) 3086 : cluster [DBG] osdmap e600: 8 total, 8 up, 4 in 2024-02-13T00:50:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:14.542135+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:50:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:14.723751+0000 osd.1 (osd.1) 251 : cluster [DBG] 6.1 scrub starts 2024-02-13T00:50:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:14.726657+0000 osd.1 (osd.1) 252 : cluster [DBG] 6.1 scrub ok 2024-02-13T00:50:16.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:14.846858+0000 osd.0 (osd.0) 225 : cluster [DBG] 28.d scrub starts 2024-02-13T00:50:16.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:14.849684+0000 osd.0 (osd.0) 226 : cluster [DBG] 28.d scrub ok 2024-02-13T00:50:16.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:15.500505+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:50:16.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:15.500599+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e601: 8 total, 8 up, 4 in 2024-02-13T00:50:16.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:15.531284+0000 mon.a (mon.0) 3090 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:50:16.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.083+0000 7f615fd1e700 1 -- 172.21.15.162:0/91446279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6158108cf0 msgr2=0x7f615810b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 --2- 172.21.15.162:0/91446279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6158108cf0 0x7f615810b0e0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 rx=0x7f6150009be0 tx=0x7f6150004d10).stop 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 -- 172.21.15.162:0/91446279 shutdown_connections 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 --2- 172.21.15.162:0/91446279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6158108cf0 0x7f615810b0e0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 --2- 172.21.15.162:0/91446279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f615806b210 0x7f615806b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 --2- 172.21.15.162:0/91446279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f615806bae0 0x7f6158108800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 -- 172.21.15.162:0/91446279 >> 172.21.15.162:0/91446279 conn(0x7f61580f38c0 msgr2=0x7f61580f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 -- 172.21.15.162:0/91446279 shutdown_connections 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 -- 172.21.15.162:0/91446279 wait complete. 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 librados: starting msgr at 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 librados: starting objecter 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 Processor -- start 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 -- start start 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 librados: setting wanted keys 2024-02-13T00:50:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 librados: calling monclient init 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f615806b210 0x7f6158114770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f615806bae0 0x7f6158114c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6158108cf0 0x7f615810e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615810ee00 con 0x7f615806bae0 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615810ef40 con 0x7f6158108cf0 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.084+0000 7f615fd1e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615810f100 con 0x7f615806b210 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615daba700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f615806b210 0x7f6158114770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615d2b9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f615806bae0 0x7f6158114c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615daba700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f615806b210 0x7f6158114770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:39924/0 (socket says 172.21.15.162:39924) 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615daba700 1 -- 172.21.15.162:0/1183861962 learned_addr learned my addr 172.21.15.162:0/1183861962 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615e2bb700 1 --2- 172.21.15.162:0/1183861962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6158108cf0 0x7f615810e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615daba700 1 -- 172.21.15.162:0/1183861962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6158108cf0 msgr2=0x7f615810e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615daba700 1 --2- 172.21.15.162:0/1183861962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6158108cf0 0x7f615810e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615daba700 1 -- 172.21.15.162:0/1183861962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f615806bae0 msgr2=0x7f6158114c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615daba700 1 --2- 172.21.15.162:0/1183861962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f615806bae0 0x7f6158114c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615daba700 1 -- 172.21.15.162:0/1183861962 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6148005040 con 0x7f615806b210 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615d2b9700 1 --2- 172.21.15.162:0/1183861962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f615806bae0 0x7f6158114c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615daba700 1 --2- 172.21.15.162:0/1183861962 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f615806b210 0x7f6158114770 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f615400c810 tx=0x7f6154007f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f614effd700 10 client.?.objecter ms_handle_connect 0x7f615806b210 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f614effd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f614effd700 1 -- 172.21.15.162:0/1183861962 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615400e4a0 con 0x7f615806b210 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f614effd700 1 -- 172.21.15.162:0/1183861962 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f615400ead0 con 0x7f615806b210 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615fd1e700 1 -- 172.21.15.162:0/1183861962 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61500097c0 con 0x7f615806b210 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f614effd700 1 -- 172.21.15.162:0/1183861962 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6154012a00 con 0x7f615806b210 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615fd1e700 10 client.37871.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615fd1e700 1 -- 172.21.15.162:0/1183861962 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f615810f740 con 0x7f615806b210 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.085+0000 7f615fd1e700 1 librados: init done 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.086+0000 7f615fd1e700 1 -- 172.21.15.162:0/1183861962 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61580475e0 con 0x7f615806b210 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.087+0000 7f614effd700 1 -- 172.21.15.162:0/1183861962 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f615400ec40 con 0x7f615806b210 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.087+0000 7f614effd700 1 --2- 172.21.15.162:0/1183861962 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6144062c90 0x7f6144065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.087+0000 7f614effd700 1 -- 172.21.15.162:0/1183861962 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(601..601 src has 449..601) v4 ==== 8475+0+0 (secure 0 0 0) 0x7f615407c9e0 con 0x7f615806b210 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.087+0000 7f614effd700 10 client.37871.objecter ms_dispatch 0x7f6158001550 osd_map(601..601 src has 449..601) v4 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.087+0000 7f614effd700 3 client.37871.objecter handle_osd_map got epochs [601,601] > 0 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.087+0000 7f614effd700 3 client.37871.objecter handle_osd_map decoding full epoch 601 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.087+0000 7f615d2b9700 1 --2- 172.21.15.162:0/1183861962 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6144062c90 0x7f6144065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.087+0000 7f614effd700 20 client.37871.objecter dump_active .. 0 homeless 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.087+0000 7f614effd700 10 client.37871.objecter ms_handle_connect 0x7f6144062c90 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.087+0000 7f615d2b9700 1 --2- 172.21.15.162:0/1183861962 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6144062c90 0x7f6144065150 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7f6158110040 tx=0x7f614800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.091+0000 7f614effd700 1 -- 172.21.15.162:0/1183861962 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6154047f60 con 0x7f615806b210 2024-02-13T00:50:16.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.257+0000 7f615fd1e700 1 -- 172.21.15.162:0/1183861962 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f615810d8b0 con 0x7f6144062c90 2024-02-13T00:50:16.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.271+0000 7f614effd700 1 -- 172.21.15.162:0/1183861962 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+505067 (secure 0 0 0) 0x7f615810d8b0 con 0x7f6144062c90 2024-02-13T00:50:16.282 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:50:16.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 10 librados: watch_flush enter 2024-02-13T00:50:16.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 10 librados: watch_flush exit 2024-02-13T00:50:16.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 20 client.37871.objecter shutdown clearing up homeless session... 2024-02-13T00:50:16.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 10 client.37871.objecter successfully canceled tick 2024-02-13T00:50:16.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 -- 172.21.15.162:0/1183861962 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6144062c90 msgr2=0x7f6144065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:16.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 --2- 172.21.15.162:0/1183861962 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6144062c90 0x7f6144065150 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7f6158110040 tx=0x7f614800b040).stop 2024-02-13T00:50:16.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 -- 172.21.15.162:0/1183861962 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f615806b210 msgr2=0x7f6158114770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:16.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 --2- 172.21.15.162:0/1183861962 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f615806b210 0x7f6158114770 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f615400c810 tx=0x7f6154007f70).stop 2024-02-13T00:50:16.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 -- 172.21.15.162:0/1183861962 shutdown_connections 2024-02-13T00:50:16.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 --2- 172.21.15.162:0/1183861962 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6144062c90 0x7f6144065150 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:16.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 --2- 172.21.15.162:0/1183861962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6158108cf0 0x7f615810e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:16.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 --2- 172.21.15.162:0/1183861962 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f615806b210 0x7f6158114770 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:16.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 --2- 172.21.15.162:0/1183861962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f615806bae0 0x7f6158114c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:16.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 -- 172.21.15.162:0/1183861962 >> 172.21.15.162:0/1183861962 conn(0x7f61580f38c0 msgr2=0x7f61580f44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:16.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 -- 172.21.15.162:0/1183861962 shutdown_connections 2024-02-13T00:50:16.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 -- 172.21.15.162:0/1183861962 wait complete. 2024-02-13T00:50:16.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:16.275+0000 7f615fd1e700 1 librados: shutdown 2024-02-13T00:50:16.292 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:50:16.554 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:50:16] "GET /metrics HTTP/1.1" 200 191106 "" "Prometheus/2.33.4" 2024-02-13T00:50:16.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-13T00:50:16.823 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:16.825 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:15.788215+0000 mgr.y (mgr.14544) 512 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:16.825 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:15.842756+0000 osd.0 (osd.0) 227 : cluster [DBG] 28.3 scrub starts 2024-02-13T00:50:16.825 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:15.846456+0000 osd.0 (osd.0) 228 : cluster [DBG] 28.3 scrub ok 2024-02-13T00:50:16.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:16.511658+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-102", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:50:16.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:16.511732+0000 mon.a (mon.0) 3092 : cluster [DBG] osdmap e602: 8 total, 8 up, 4 in 2024-02-13T00:50:16.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:16.513810+0000 mon.a (mon.0) 3093 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-102"}]: dispatch 2024-02-13T00:50:16.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:15.788215+0000 mgr.y (mgr.14544) 512 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:16.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:15.842756+0000 osd.0 (osd.0) 227 : cluster [DBG] 28.3 scrub starts 2024-02-13T00:50:16.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:15.846456+0000 osd.0 (osd.0) 228 : cluster [DBG] 28.3 scrub ok 2024-02-13T00:50:16.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:16.511658+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-102", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:50:16.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:16.511732+0000 mon.a (mon.0) 3092 : cluster [DBG] osdmap e602: 8 total, 8 up, 4 in 2024-02-13T00:50:16.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:16.513810+0000 mon.a (mon.0) 3093 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-102"}]: dispatch 2024-02-13T00:50:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:16.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:16.928 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:16.958 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:15.788215+0000 mgr.y (mgr.14544) 512 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:16.958 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:15.842756+0000 osd.0 (osd.0) 227 : cluster [DBG] 28.3 scrub starts 2024-02-13T00:50:16.958 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:15.846456+0000 osd.0 (osd.0) 228 : cluster [DBG] 28.3 scrub ok 2024-02-13T00:50:16.959 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:16.511658+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-102", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:50:16.959 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:16.511732+0000 mon.a (mon.0) 3092 : cluster [DBG] osdmap e602: 8 total, 8 up, 4 in 2024-02-13T00:50:16.959 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:16.513810+0000 mon.a (mon.0) 3093 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-102"}]: dispatch 2024-02-13T00:50:16.960 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:16.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:50:16.981 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:16.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:16.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:17.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:17.115 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:17.135 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:17.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:17.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:17.268 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:17.287 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:17.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:17.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:17.418 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:17.437 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:17.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:17.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:17.570 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:16.258666+0000 mgr.y (mgr.14544) 513 : audit [DBG] from='client.37871 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:50:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:16.488037+0000 mgr.y (mgr.14544) 514 : cluster [DBG] pgmap v902: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:16.674057+0000 osd.1 (osd.1) 253 : cluster [DBG] 6.20 scrub starts 2024-02-13T00:50:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:16.677425+0000 osd.1 (osd.1) 254 : cluster [DBG] 6.20 scrub ok 2024-02-13T00:50:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:17.514661+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-102"}]': finished 2024-02-13T00:50:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:17.514728+0000 mon.a (mon.0) 3095 : cluster [DBG] osdmap e603: 8 total, 8 up, 4 in 2024-02-13T00:50:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:17.535110+0000 mon.a (mon.0) 3096 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:17.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:16.258666+0000 mgr.y (mgr.14544) 513 : audit [DBG] from='client.37871 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:50:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:16.488037+0000 mgr.y (mgr.14544) 514 : cluster [DBG] pgmap v902: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:16.674057+0000 osd.1 (osd.1) 253 : cluster [DBG] 6.20 scrub starts 2024-02-13T00:50:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:16.677425+0000 osd.1 (osd.1) 254 : cluster [DBG] 6.20 scrub ok 2024-02-13T00:50:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:17.514661+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-102"}]': finished 2024-02-13T00:50:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:17.514728+0000 mon.a (mon.0) 3095 : cluster [DBG] osdmap e603: 8 total, 8 up, 4 in 2024-02-13T00:50:17.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:17.535110+0000 mon.a (mon.0) 3096 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:18.026 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:18.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:16.258666+0000 mgr.y (mgr.14544) 513 : audit [DBG] from='client.37871 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:50:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:16.488037+0000 mgr.y (mgr.14544) 514 : cluster [DBG] pgmap v902: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:16.674057+0000 osd.1 (osd.1) 253 : cluster [DBG] 6.20 scrub starts 2024-02-13T00:50:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:16.677425+0000 osd.1 (osd.1) 254 : cluster [DBG] 6.20 scrub ok 2024-02-13T00:50:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:17.514661+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-102"}]': finished 2024-02-13T00:50:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:17.514728+0000 mon.a (mon.0) 3095 : cluster [DBG] osdmap e603: 8 total, 8 up, 4 in 2024-02-13T00:50:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:17.535110+0000 mon.a (mon.0) 3096 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:18.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:18.158 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:18.177 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:18.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:18.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:18.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:18.308 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:18.416 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:18.430 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:18.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:18.440 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":901,"stamp":"2024-02-13T00:50:15.501404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":2244,"num_read_kb":1964,"num_write":913,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":13678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1044,"ondisk_log_size":1044,"up":612,"acting":612,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":498,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1217224,"kb_used_data":11944,"kb_used_omap":0,"kb_used_meta":1205248,"kb_avail":373763384,"statfs":{"total":383980142592,"available":382733705216,"internally_reserved":0,"allocated":12230656,"data_stored":5756829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1234173952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.974822"},"pg_stats":[{"pgid":"235.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.539825+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"235.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.540576+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"235.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.545316+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.542688+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.539419+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"235.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.542089+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"235.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.542130+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"235.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.546118+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.544815+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.545675+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.543076+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"235.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.539373+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"235.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.542695+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.547137+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.538964+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"235.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.546406+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.544593+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"235.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.544568+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"235.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.539318+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"235.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.544931+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.542383+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.542505+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"235.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.539548+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"235.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.546757+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.540243+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.7","version":"0'0","reported_seq":9,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.554059+0000","last_change":"2024-02-13T00:50:14.554059+0000","last_active":"2024-02-13T00:50:14.554059+0000","last_peered":"2024-02-13T00:50:14.554059+0000","last_clean":"2024-02-13T00:50:14.554059+0000","last_became_active":"2024-02-13T00:50:14.553744+0000","last_became_peered":"2024-02-13T00:50:14.553744+0000","last_unstale":"2024-02-13T00:50:14.554059+0000","last_undegraded":"2024-02-13T00:50:14.554059+0000","last_fullsized":"2024-02-13T00:50:14.554059+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.544438+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.6","version":"0'0","reported_seq":9,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.553031+0000","last_change":"2024-02-13T00:50:14.553031+0000","last_active":"2024-02-13T00:50:14.553031+0000","last_peered":"2024-02-13T00:50:14.553031+0000","last_clean":"2024-02-13T00:50:14.553031+0000","last_became_active":"2024-02-13T00:50:14.552756+0000","last_became_peered":"2024-02-13T00:50:14.552756+0000","last_unstale":"2024-02-13T00:50:14.553031+0000","last_undegraded":"2024-02-13T00:50:14.553031+0000","last_fullsized":"2024-02-13T00:50:14.553031+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"235.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.542273+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.5","version":"0'0","reported_seq":9,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.551022+0000","last_change":"2024-02-13T00:50:14.551022+0000","last_active":"2024-02-13T00:50:14.551022+0000","last_peered":"2024-02-13T00:50:14.551022+0000","last_clean":"2024-02-13T00:50:14.551022+0000","last_became_active":"2024-02-13T00:50:14.550644+0000","last_became_peered":"2024-02-13T00:50:14.550644+0000","last_unstale":"2024-02-13T00:50:14.551022+0000","last_undegraded":"2024-02-13T00:50:14.551022+0000","last_fullsized":"2024-02-13T00:50:14.551022+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"235.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.539019+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.4","version":"0'0","reported_seq":9,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.550087+0000","last_change":"2024-02-13T00:50:14.550087+0000","last_active":"2024-02-13T00:50:14.550087+0000","last_peered":"2024-02-13T00:50:14.550087+0000","last_clean":"2024-02-13T00:50:14.550087+0000","last_became_active":"2024-02-13T00:50:14.549625+0000","last_became_peered":"2024-02-13T00:50:14.549625+0000","last_unstale":"2024-02-13T00:50:14.550087+0000","last_undegraded":"2024-02-13T00:50:14.550087+0000","last_fullsized":"2024-02-13T00:50:14.550087+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"235.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.538912+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.3","version":"0'0","reported_seq":9,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.550875+0000","last_change":"2024-02-13T00:50:14.550875+0000","last_active":"2024-02-13T00:50:14.550875+0000","last_peered":"2024-02-13T00:50:14.550875+0000","last_clean":"2024-02-13T00:50:14.550875+0000","last_became_active":"2024-02-13T00:50:14.549053+0000","last_became_peered":"2024-02-13T00:50:14.549053+0000","last_unstale":"2024-02-13T00:50:14.550875+0000","last_undegraded":"2024-02-13T00:50:14.550875+0000","last_fullsized":"2024-02-13T00:50:14.550875+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"235.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.538954+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.2","version":"0'0","reported_seq":9,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.552504+0000","last_change":"2024-02-13T00:50:14.552504+0000","last_active":"2024-02-13T00:50:14.552504+0000","last_peered":"2024-02-13T00:50:14.552504+0000","last_clean":"2024-02-13T00:50:14.552504+0000","last_became_active":"2024-02-13T00:50:14.552162+0000","last_became_peered":"2024-02-13T00:50:14.552162+0000","last_unstale":"2024-02-13T00:50:14.552504+0000","last_undegraded":"2024-02-13T00:50:14.552504+0000","last_fullsized":"2024-02-13T00:50:14.552504+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"235.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.539953+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.1","version":"0'0","reported_seq":9,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.553804+0000","last_change":"2024-02-13T00:50:14.553804+0000","last_active":"2024-02-13T00:50:14.553804+0000","last_peered":"2024-02-13T00:50:14.553804+0000","last_clean":"2024-02-13T00:50:14.553804+0000","last_became_active":"2024-02-13T00:50:14.553460+0000","last_became_peered":"2024-02-13T00:50:14.553460+0000","last_unstale":"2024-02-13T00:50:14.553804+0000","last_undegraded":"2024-02-13T00:50:14.553804+0000","last_fullsized":"2024-02-13T00:50:14.553804+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"235.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-13T00:50:14.534216+0000","last_change":"2024-02-13T00:50:14.542595+0000","last_active":"2024-02-13T00:50:14.534216+0000","last_peered":"2024-02-13T00:50:14.534216+0000","last_clean":"2024-02-13T00:50:14.534216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:50:14.534216+0000","last_undegraded":"2024-02-13T00:50:14.534216+0000","last_fullsized":"2024-02-13T00:50:14.534216+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:14.534216+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.534216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.0","version":"0'0","reported_seq":9,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.551368+0000","last_change":"2024-02-13T00:50:14.551368+0000","last_active":"2024-02-13T00:50:14.551368+0000","last_peered":"2024-02-13T00:50:14.551368+0000","last_clean":"2024-02-13T00:50:14.551368+0000","last_became_active":"2024-02-13T00:50:14.551016+0000","last_became_peered":"2024-02-13T00:50:14.551016+0000","last_unstale":"2024-02-13T00:50:14.551368+0000","last_undegraded":"2024-02-13T00:50:14.551368+0000","last_fullsized":"2024-02-13T00:50:14.551368+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":500,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541192+0000","last_change":"2024-02-13T00:46:01.448309+0000","last_active":"2024-02-13T00:50:13.541192+0000","last_peered":"2024-02-13T00:50:13.541192+0000","last_clean":"2024-02-13T00:50:13.541192+0000","last_became_active":"2024-02-13T00:46:01.445363+0000","last_became_peered":"2024-02-13T00:46:01.445363+0000","last_unstale":"2024-02-13T00:50:13.541192+0000","last_undegraded":"2024-02-13T00:50:13.541192+0000","last_fullsized":"2024-02-13T00:50:13.541192+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":425,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540060+0000","last_change":"2024-02-13T00:46:01.449894+0000","last_active":"2024-02-13T00:50:13.540060+0000","last_peered":"2024-02-13T00:50:13.540060+0000","last_clean":"2024-02-13T00:50:13.540060+0000","last_became_active":"2024-02-13T00:46:01.449570+0000","last_became_peered":"2024-02-13T00:46:01.449570+0000","last_unstale":"2024-02-13T00:50:13.540060+0000","last_undegraded":"2024-02-13T00:50:13.540060+0000","last_fullsized":"2024-02-13T00:50:13.540060+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":291,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538236+0000","last_change":"2024-02-13T00:46:01.445141+0000","last_active":"2024-02-13T00:50:13.538236+0000","last_peered":"2024-02-13T00:50:13.538236+0000","last_clean":"2024-02-13T00:50:13.538236+0000","last_became_active":"2024-02-13T00:46:01.444729+0000","last_became_peered":"2024-02-13T00:46:01.444729+0000","last_unstale":"2024-02-13T00:50:13.538236+0000","last_undegraded":"2024-02-13T00:50:13.538236+0000","last_fullsized":"2024-02-13T00:50:13.538236+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":376,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.537305+0000","last_change":"2024-02-13T00:45:15.426658+0000","last_active":"2024-02-13T00:50:13.537305+0000","last_peered":"2024-02-13T00:50:13.537305+0000","last_clean":"2024-02-13T00:50:13.537305+0000","last_became_active":"2024-02-13T00:45:15.426109+0000","last_became_peered":"2024-02-13T00:45:15.426109+0000","last_unstale":"2024-02-13T00:50:13.537305+0000","last_undegraded":"2024-02-13T00:50:13.537305+0000","last_fullsized":"2024-02-13T00:50:13.537305+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":281,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541244+0000","last_change":"2024-02-13T00:44:39.526065+0000","last_active":"2024-02-13T00:50:13.541244+0000","last_peered":"2024-02-13T00:50:13.541244+0000","last_clean":"2024-02-13T00:50:13.541244+0000","last_became_active":"2024-02-13T00:44:39.525161+0000","last_became_peered":"2024-02-13T00:44:39.525161+0000","last_unstale":"2024-02-13T00:50:13.541244+0000","last_undegraded":"2024-02-13T00:50:13.541244+0000","last_fullsized":"2024-02-13T00:50:13.541244+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":379,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538477+0000","last_change":"2024-02-13T00:48:06.705421+0000","last_active":"2024-02-13T00:50:13.538477+0000","last_peered":"2024-02-13T00:50:13.538477+0000","last_clean":"2024-02-13T00:50:13.538477+0000","last_became_active":"2024-02-13T00:48:06.705121+0000","last_became_peered":"2024-02-13T00:48:06.705121+0000","last_unstale":"2024-02-13T00:50:13.538477+0000","last_undegraded":"2024-02-13T00:50:13.538477+0000","last_fullsized":"2024-02-13T00:50:13.538477+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":299,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542962+0000","last_change":"2024-02-13T00:46:01.459772+0000","last_active":"2024-02-13T00:50:13.542962+0000","last_peered":"2024-02-13T00:50:13.542962+0000","last_clean":"2024-02-13T00:50:13.542962+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:50:13.542962+0000","last_undegraded":"2024-02-13T00:50:13.542962+0000","last_fullsized":"2024-02-13T00:50:13.542962+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":558,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542768+0000","last_change":"2024-02-13T00:46:01.463062+0000","last_active":"2024-02-13T00:50:13.542768+0000","last_peered":"2024-02-13T00:50:13.542768+0000","last_clean":"2024-02-13T00:50:13.542768+0000","last_became_active":"2024-02-13T00:46:01.462319+0000","last_became_peered":"2024-02-13T00:46:01.462319+0000","last_unstale":"2024-02-13T00:50:13.542768+0000","last_undegraded":"2024-02-13T00:50:13.542768+0000","last_fullsized":"2024-02-13T00:50:13.542768+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":286,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543105+0000","last_change":"2024-02-13T00:49:48.070260+0000","last_active":"2024-02-13T00:50:13.543105+0000","last_peered":"2024-02-13T00:50:13.543105+0000","last_clean":"2024-02-13T00:50:13.543105+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:50:13.543105+0000","last_undegraded":"2024-02-13T00:50:13.543105+0000","last_fullsized":"2024-02-13T00:50:13.543105+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:48.070195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:49:48.070195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":367,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540199+0000","last_change":"2024-02-13T00:46:01.455629+0000","last_active":"2024-02-13T00:50:13.540199+0000","last_peered":"2024-02-13T00:50:13.540199+0000","last_clean":"2024-02-13T00:50:13.540199+0000","last_became_active":"2024-02-13T00:46:01.452801+0000","last_became_peered":"2024-02-13T00:46:01.452801+0000","last_unstale":"2024-02-13T00:50:13.540199+0000","last_undegraded":"2024-02-13T00:50:13.540199+0000","last_fullsized":"2024-02-13T00:50:13.540199+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":521,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540838+0000","last_change":"2024-02-13T00:45:39.562095+0000","last_active":"2024-02-13T00:50:13.540838+0000","last_peered":"2024-02-13T00:50:13.540838+0000","last_clean":"2024-02-13T00:50:13.540838+0000","last_became_active":"2024-02-13T00:44:39.505121+0000","last_became_peered":"2024-02-13T00:44:39.505121+0000","last_unstale":"2024-02-13T00:50:13.540838+0000","last_undegraded":"2024-02-13T00:50:13.540838+0000","last_fullsized":"2024-02-13T00:50:13.540838+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:39.561994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:39.561994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"534'16","reported_seq":392,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.357847+0000","last_change":"2024-02-13T00:49:07.494629+0000","last_active":"2024-02-13T00:50:14.357847+0000","last_peered":"2024-02-13T00:50:14.357847+0000","last_clean":"2024-02-13T00:50:14.357847+0000","last_became_active":"2024-02-13T00:49:07.494388+0000","last_became_peered":"2024-02-13T00:49:07.494388+0000","last_unstale":"2024-02-13T00:50:14.357847+0000","last_undegraded":"2024-02-13T00:50:14.357847+0000","last_fullsized":"2024-02-13T00:50:14.357847+0000","mapping_epoch":359,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:45:53.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.115260+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":625,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539911+0000","last_change":"2024-02-13T00:47:03.491720+0000","last_active":"2024-02-13T00:50:13.539911+0000","last_peered":"2024-02-13T00:50:13.539911+0000","last_clean":"2024-02-13T00:50:13.539911+0000","last_became_active":"2024-02-13T00:46:01.438056+0000","last_became_peered":"2024-02-13T00:46:01.438056+0000","last_unstale":"2024-02-13T00:50:13.539911+0000","last_undegraded":"2024-02-13T00:50:13.539911+0000","last_fullsized":"2024-02-13T00:50:13.539911+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:47:03.491647+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":595,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541803+0000","last_change":"2024-02-13T00:48:23.379872+0000","last_active":"2024-02-13T00:50:13.541803+0000","last_peered":"2024-02-13T00:50:13.541803+0000","last_clean":"2024-02-13T00:50:13.541803+0000","last_became_active":"2024-02-13T00:46:01.460811+0000","last_became_peered":"2024-02-13T00:46:01.460811+0000","last_unstale":"2024-02-13T00:50:13.541803+0000","last_undegraded":"2024-02-13T00:50:13.541803+0000","last_fullsized":"2024-02-13T00:50:13.541803+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:23.379773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:23.379773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":501,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541697+0000","last_change":"2024-02-13T00:47:34.332791+0000","last_active":"2024-02-13T00:50:13.541697+0000","last_peered":"2024-02-13T00:50:13.541697+0000","last_clean":"2024-02-13T00:50:13.541697+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:50:13.541697+0000","last_undegraded":"2024-02-13T00:50:13.541697+0000","last_fullsized":"2024-02-13T00:50:13.541697+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:34.332704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:34.332704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":539,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541042+0000","last_change":"2024-02-13T00:46:01.448237+0000","last_active":"2024-02-13T00:50:13.541042+0000","last_peered":"2024-02-13T00:50:13.541042+0000","last_clean":"2024-02-13T00:50:13.541042+0000","last_became_active":"2024-02-13T00:46:01.446660+0000","last_became_peered":"2024-02-13T00:46:01.446660+0000","last_unstale":"2024-02-13T00:50:13.541042+0000","last_undegraded":"2024-02-13T00:50:13.541042+0000","last_fullsized":"2024-02-13T00:50:13.541042+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.260871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.260871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":600,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541962+0000","last_change":"2024-02-13T00:47:41.808333+0000","last_active":"2024-02-13T00:50:13.541962+0000","last_peered":"2024-02-13T00:50:13.541962+0000","last_clean":"2024-02-13T00:50:13.541962+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:50:13.541962+0000","last_undegraded":"2024-02-13T00:50:13.541962+0000","last_fullsized":"2024-02-13T00:50:13.541962+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:47:41.808259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:41.808259+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":582,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539870+0000","last_change":"2024-02-13T00:48:24.823190+0000","last_active":"2024-02-13T00:50:13.539870+0000","last_peered":"2024-02-13T00:50:13.539870+0000","last_clean":"2024-02-13T00:50:13.539870+0000","last_became_active":"2024-02-13T00:44:50.429994+0000","last_became_peered":"2024-02-13T00:44:50.429994+0000","last_unstale":"2024-02-13T00:50:13.539870+0000","last_undegraded":"2024-02-13T00:50:13.539870+0000","last_fullsized":"2024-02-13T00:50:13.539870+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:24.823107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.823107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":484,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541589+0000","last_change":"2024-02-13T00:49:39.788846+0000","last_active":"2024-02-13T00:50:13.541589+0000","last_peered":"2024-02-13T00:50:13.541589+0000","last_clean":"2024-02-13T00:50:13.541589+0000","last_became_active":"2024-02-13T00:49:03.971407+0000","last_became_peered":"2024-02-13T00:49:03.971407+0000","last_unstale":"2024-02-13T00:50:13.541589+0000","last_undegraded":"2024-02-13T00:50:13.541589+0000","last_fullsized":"2024-02-13T00:50:13.541589+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.788783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.788783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":583,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541466+0000","last_change":"2024-02-13T00:49:41.837871+0000","last_active":"2024-02-13T00:50:13.541466+0000","last_peered":"2024-02-13T00:50:13.541466+0000","last_clean":"2024-02-13T00:50:13.541466+0000","last_became_active":"2024-02-13T00:45:15.434853+0000","last_became_peered":"2024-02-13T00:45:15.434853+0000","last_unstale":"2024-02-13T00:50:13.541466+0000","last_undegraded":"2024-02-13T00:50:13.541466+0000","last_fullsized":"2024-02-13T00:50:13.541466+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.837810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.837810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":204,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541213+0000","last_change":"2024-02-13T00:46:01.455201+0000","last_active":"2024-02-13T00:50:13.541213+0000","last_peered":"2024-02-13T00:50:13.541213+0000","last_clean":"2024-02-13T00:50:13.541213+0000","last_became_active":"2024-02-13T00:46:01.454882+0000","last_became_peered":"2024-02-13T00:46:01.454882+0000","last_unstale":"2024-02-13T00:50:13.541213+0000","last_undegraded":"2024-02-13T00:50:13.541213+0000","last_fullsized":"2024-02-13T00:50:13.541213+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:35.197045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:35.197045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":389,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542020+0000","last_change":"2024-02-13T00:46:09.256189+0000","last_active":"2024-02-13T00:50:13.542020+0000","last_peered":"2024-02-13T00:50:13.542020+0000","last_clean":"2024-02-13T00:50:13.542020+0000","last_became_active":"2024-02-13T00:46:01.459978+0000","last_became_peered":"2024-02-13T00:46:01.459978+0000","last_unstale":"2024-02-13T00:50:13.542020+0000","last_undegraded":"2024-02-13T00:50:13.542020+0000","last_fullsized":"2024-02-13T00:50:13.542020+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:44:24.196214+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:44:24.196214+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":214,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542855+0000","last_change":"2024-02-13T00:46:30.641539+0000","last_active":"2024-02-13T00:50:13.542855+0000","last_peered":"2024-02-13T00:50:13.542855+0000","last_clean":"2024-02-13T00:50:13.542855+0000","last_became_active":"2024-02-13T00:46:01.444548+0000","last_became_peered":"2024-02-13T00:46:01.444548+0000","last_unstale":"2024-02-13T00:50:13.542855+0000","last_undegraded":"2024-02-13T00:50:13.542855+0000","last_fullsized":"2024-02-13T00:50:13.542855+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.641460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.641460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":314,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541810+0000","last_change":"2024-02-13T00:48:54.498301+0000","last_active":"2024-02-13T00:50:13.541810+0000","last_peered":"2024-02-13T00:50:13.541810+0000","last_clean":"2024-02-13T00:50:13.541810+0000","last_became_active":"2024-02-13T00:48:54.497977+0000","last_became_peered":"2024-02-13T00:48:54.497977+0000","last_unstale":"2024-02-13T00:50:13.541810+0000","last_undegraded":"2024-02-13T00:50:13.541810+0000","last_fullsized":"2024-02-13T00:50:13.541810+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.592672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":417,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540052+0000","last_change":"2024-02-13T00:45:15.429026+0000","last_active":"2024-02-13T00:50:13.540052+0000","last_peered":"2024-02-13T00:50:13.540052+0000","last_clean":"2024-02-13T00:50:13.540052+0000","last_became_active":"2024-02-13T00:45:15.427945+0000","last_became_peered":"2024-02-13T00:45:15.427945+0000","last_unstale":"2024-02-13T00:50:13.540052+0000","last_undegraded":"2024-02-13T00:50:13.540052+0000","last_fullsized":"2024-02-13T00:50:13.540052+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":522,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539741+0000","last_change":"2024-02-13T00:45:26.255907+0000","last_active":"2024-02-13T00:50:13.539741+0000","last_peered":"2024-02-13T00:50:13.539741+0000","last_clean":"2024-02-13T00:50:13.539741+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:50:13.539741+0000","last_undegraded":"2024-02-13T00:50:13.539741+0000","last_fullsized":"2024-02-13T00:50:13.539741+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.255842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.255842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"467'31","reported_seq":359,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539802+0000","last_change":"2024-02-13T00:48:31.782821+0000","last_active":"2024-02-13T00:50:13.539802+0000","last_peered":"2024-02-13T00:50:13.539802+0000","last_clean":"2024-02-13T00:50:13.539802+0000","last_became_active":"2024-02-13T00:44:50.432807+0000","last_became_peered":"2024-02-13T00:44:50.432807+0000","last_unstale":"2024-02-13T00:50:13.539802+0000","last_undegraded":"2024-02-13T00:50:13.539802+0000","last_fullsized":"2024-02-13T00:50:13.539802+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'31","last_scrub_stamp":"2024-02-13T00:48:31.782729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.782729+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":372,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540712+0000","last_change":"2024-02-13T00:46:04.708510+0000","last_active":"2024-02-13T00:50:13.540712+0000","last_peered":"2024-02-13T00:50:13.540712+0000","last_clean":"2024-02-13T00:50:13.540712+0000","last_became_active":"2024-02-13T00:46:01.447801+0000","last_became_peered":"2024-02-13T00:46:01.447801+0000","last_unstale":"2024-02-13T00:50:13.540712+0000","last_undegraded":"2024-02-13T00:50:13.540712+0000","last_fullsized":"2024-02-13T00:50:13.540712+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:45:24.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:24.269595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":580,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540608+0000","last_change":"2024-02-13T00:48:49.612735+0000","last_active":"2024-02-13T00:50:13.540608+0000","last_peered":"2024-02-13T00:50:13.540608+0000","last_clean":"2024-02-13T00:50:13.540608+0000","last_became_active":"2024-02-13T00:48:49.612437+0000","last_became_peered":"2024-02-13T00:48:49.612437+0000","last_unstale":"2024-02-13T00:50:13.540608+0000","last_undegraded":"2024-02-13T00:50:13.540608+0000","last_fullsized":"2024-02-13T00:50:13.540608+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.793634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":566,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542705+0000","last_change":"2024-02-13T00:44:50.441408+0000","last_active":"2024-02-13T00:50:13.542705+0000","last_peered":"2024-02-13T00:50:13.542705+0000","last_clean":"2024-02-13T00:50:13.542705+0000","last_became_active":"2024-02-13T00:44:50.440301+0000","last_became_peered":"2024-02-13T00:44:50.440301+0000","last_unstale":"2024-02-13T00:50:13.542705+0000","last_undegraded":"2024-02-13T00:50:13.542705+0000","last_fullsized":"2024-02-13T00:50:13.542705+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:25.502311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.502311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":221,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542416+0000","last_change":"2024-02-13T00:48:06.705079+0000","last_active":"2024-02-13T00:50:13.542416+0000","last_peered":"2024-02-13T00:50:13.542416+0000","last_clean":"2024-02-13T00:50:13.542416+0000","last_became_active":"2024-02-13T00:48:06.704718+0000","last_became_peered":"2024-02-13T00:48:06.704718+0000","last_unstale":"2024-02-13T00:50:13.542416+0000","last_undegraded":"2024-02-13T00:50:13.542416+0000","last_fullsized":"2024-02-13T00:50:13.542416+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.488353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.488353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":313,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543219+0000","last_change":"2024-02-13T00:49:03.294585+0000","last_active":"2024-02-13T00:50:13.543219+0000","last_peered":"2024-02-13T00:50:13.543219+0000","last_clean":"2024-02-13T00:50:13.543219+0000","last_became_active":"2024-02-13T00:44:39.523138+0000","last_became_peered":"2024-02-13T00:44:39.523138+0000","last_unstale":"2024-02-13T00:50:13.543219+0000","last_undegraded":"2024-02-13T00:50:13.543219+0000","last_fullsized":"2024-02-13T00:50:13.543219+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:49:03.294499+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:49:03.294499+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":591,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538718+0000","last_change":"2024-02-13T00:48:59.111910+0000","last_active":"2024-02-13T00:50:13.538718+0000","last_peered":"2024-02-13T00:50:13.538718+0000","last_clean":"2024-02-13T00:50:13.538718+0000","last_became_active":"2024-02-13T00:46:01.452674+0000","last_became_peered":"2024-02-13T00:46:01.452674+0000","last_unstale":"2024-02-13T00:50:13.538718+0000","last_undegraded":"2024-02-13T00:50:13.538718+0000","last_fullsized":"2024-02-13T00:50:13.538718+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:59.111828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:59.111828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"511'13","reported_seq":307,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.357976+0000","last_change":"2024-02-13T00:48:45.592523+0000","last_active":"2024-02-13T00:50:14.357976+0000","last_peered":"2024-02-13T00:50:14.357976+0000","last_clean":"2024-02-13T00:50:14.357976+0000","last_became_active":"2024-02-13T00:48:45.591249+0000","last_became_peered":"2024-02-13T00:48:45.591249+0000","last_unstale":"2024-02-13T00:50:14.357976+0000","last_undegraded":"2024-02-13T00:50:14.357976+0000","last_fullsized":"2024-02-13T00:50:14.357976+0000","mapping_epoch":401,"log_start":"404'11","ondisk_log_start":"404'11","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'11","last_scrub_stamp":"2024-02-13T00:46:37.662413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.662413+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":558,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540368+0000","last_change":"2024-02-13T00:46:05.839833+0000","last_active":"2024-02-13T00:50:13.540368+0000","last_peered":"2024-02-13T00:50:13.540368+0000","last_clean":"2024-02-13T00:50:13.540368+0000","last_became_active":"2024-02-13T00:46:01.455274+0000","last_became_peered":"2024-02-13T00:46:01.455274+0000","last_unstale":"2024-02-13T00:50:13.540368+0000","last_undegraded":"2024-02-13T00:50:13.540368+0000","last_fullsized":"2024-02-13T00:50:13.540368+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:44:25.230980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.230980+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":367,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540888+0000","last_change":"2024-02-13T00:46:01.457421+0000","last_active":"2024-02-13T00:50:13.540888+0000","last_peered":"2024-02-13T00:50:13.540888+0000","last_clean":"2024-02-13T00:50:13.540888+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:50:13.540888+0000","last_undegraded":"2024-02-13T00:50:13.540888+0000","last_fullsized":"2024-02-13T00:50:13.540888+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:18.089252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:18.089252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"467'27","reported_seq":680,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540799+0000","last_change":"2024-02-13T00:48:54.043163+0000","last_active":"2024-02-13T00:50:13.540799+0000","last_peered":"2024-02-13T00:50:13.540799+0000","last_clean":"2024-02-13T00:50:13.540799+0000","last_became_active":"2024-02-13T00:46:01.448705+0000","last_became_peered":"2024-02-13T00:46:01.448705+0000","last_unstale":"2024-02-13T00:50:13.540799+0000","last_undegraded":"2024-02-13T00:50:13.540799+0000","last_fullsized":"2024-02-13T00:50:13.540799+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:48:54.043088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.043088+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":623,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538421+0000","last_change":"2024-02-13T00:49:37.053462+0000","last_active":"2024-02-13T00:50:13.538421+0000","last_peered":"2024-02-13T00:50:13.538421+0000","last_clean":"2024-02-13T00:50:13.538421+0000","last_became_active":"2024-02-13T00:45:15.445228+0000","last_became_peered":"2024-02-13T00:45:15.445228+0000","last_unstale":"2024-02-13T00:50:13.538421+0000","last_undegraded":"2024-02-13T00:50:13.538421+0000","last_fullsized":"2024-02-13T00:50:13.538421+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:49:37.053398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.053398+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"507'13","reported_seq":305,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.357922+0000","last_change":"2024-02-13T00:48:41.526587+0000","last_active":"2024-02-13T00:50:14.357922+0000","last_peered":"2024-02-13T00:50:14.357922+0000","last_clean":"2024-02-13T00:50:14.357922+0000","last_became_active":"2024-02-13T00:48:41.525262+0000","last_became_peered":"2024-02-13T00:48:41.525262+0000","last_unstale":"2024-02-13T00:50:14.357922+0000","last_undegraded":"2024-02-13T00:50:14.357922+0000","last_fullsized":"2024-02-13T00:50:14.357922+0000","mapping_epoch":401,"log_start":"467'11","ondisk_log_start":"467'11","created":60,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:47:49.687727+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":208,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539452+0000","last_change":"2024-02-13T00:46:01.457110+0000","last_active":"2024-02-13T00:50:13.539452+0000","last_peered":"2024-02-13T00:50:13.539452+0000","last_clean":"2024-02-13T00:50:13.539452+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:50:13.539452+0000","last_undegraded":"2024-02-13T00:50:13.539452+0000","last_fullsized":"2024-02-13T00:50:13.539452+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:47.092250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:43:47.092250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":521,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543098+0000","last_change":"2024-02-13T00:46:01.456531+0000","last_active":"2024-02-13T00:50:13.543098+0000","last_peered":"2024-02-13T00:50:13.543098+0000","last_clean":"2024-02-13T00:50:13.543098+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:50:13.543098+0000","last_undegraded":"2024-02-13T00:50:13.543098+0000","last_fullsized":"2024-02-13T00:50:13.543098+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:44:59.963616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":312,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542195+0000","last_change":"2024-02-13T00:48:57.386335+0000","last_active":"2024-02-13T00:50:13.542195+0000","last_peered":"2024-02-13T00:50:13.542195+0000","last_clean":"2024-02-13T00:50:13.542195+0000","last_became_active":"2024-02-13T00:44:39.521984+0000","last_became_peered":"2024-02-13T00:44:39.521984+0000","last_unstale":"2024-02-13T00:50:13.542195+0000","last_undegraded":"2024-02-13T00:50:13.542195+0000","last_fullsized":"2024-02-13T00:50:13.542195+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:57.386254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:57.386254+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":404,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540586+0000","last_change":"2024-02-13T00:46:23.757417+0000","last_active":"2024-02-13T00:50:13.540586+0000","last_peered":"2024-02-13T00:50:13.540586+0000","last_clean":"2024-02-13T00:50:13.540586+0000","last_became_active":"2024-02-13T00:45:15.419239+0000","last_became_peered":"2024-02-13T00:45:15.419239+0000","last_unstale":"2024-02-13T00:50:13.540586+0000","last_undegraded":"2024-02-13T00:50:13.540586+0000","last_fullsized":"2024-02-13T00:50:13.540586+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:23.757351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:23.757351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"504'16","reported_seq":545,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.357833+0000","last_change":"2024-02-13T00:49:45.051624+0000","last_active":"2024-02-13T00:50:14.357833+0000","last_peered":"2024-02-13T00:50:14.357833+0000","last_clean":"2024-02-13T00:50:14.357833+0000","last_became_active":"2024-02-13T00:48:37.480010+0000","last_became_peered":"2024-02-13T00:48:37.480010+0000","last_unstale":"2024-02-13T00:50:14.357833+0000","last_undegraded":"2024-02-13T00:50:14.357833+0000","last_fullsized":"2024-02-13T00:50:14.357833+0000","mapping_epoch":401,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'16","last_scrub_stamp":"2024-02-13T00:49:45.051539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:49:45.051539+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":304,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542865+0000","last_change":"2024-02-13T00:48:31.463534+0000","last_active":"2024-02-13T00:50:13.542865+0000","last_peered":"2024-02-13T00:50:13.542865+0000","last_clean":"2024-02-13T00:50:13.542865+0000","last_became_active":"2024-02-13T00:46:01.462717+0000","last_became_peered":"2024-02-13T00:46:01.462717+0000","last_unstale":"2024-02-13T00:50:13.542865+0000","last_undegraded":"2024-02-13T00:50:13.542865+0000","last_fullsized":"2024-02-13T00:50:13.542865+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:31.463442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.463442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":478,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541043+0000","last_change":"2024-02-13T00:49:00.881498+0000","last_active":"2024-02-13T00:50:13.541043+0000","last_peered":"2024-02-13T00:50:13.541043+0000","last_clean":"2024-02-13T00:50:13.541043+0000","last_became_active":"2024-02-13T00:46:01.456959+0000","last_became_peered":"2024-02-13T00:46:01.456959+0000","last_unstale":"2024-02-13T00:50:13.541043+0000","last_undegraded":"2024-02-13T00:50:13.541043+0000","last_fullsized":"2024-02-13T00:50:13.541043+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:00.881431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:00.881431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"263'10","reported_seq":600,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.357396+0000","last_change":"2024-02-13T00:47:02.949483+0000","last_active":"2024-02-13T00:50:14.357396+0000","last_peered":"2024-02-13T00:50:14.357396+0000","last_clean":"2024-02-13T00:50:14.357396+0000","last_became_active":"2024-02-13T00:43:35.454087+0000","last_became_peered":"2024-02-13T00:43:35.454087+0000","last_unstale":"2024-02-13T00:50:14.357396+0000","last_undegraded":"2024-02-13T00:50:14.357396+0000","last_fullsized":"2024-02-13T00:50:14.357396+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.949406+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":576,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541123+0000","last_change":"2024-02-13T00:49:46.101991+0000","last_active":"2024-02-13T00:50:13.541123+0000","last_peered":"2024-02-13T00:50:13.541123+0000","last_clean":"2024-02-13T00:50:13.541123+0000","last_became_active":"2024-02-13T00:46:01.449654+0000","last_became_peered":"2024-02-13T00:46:01.449654+0000","last_unstale":"2024-02-13T00:50:13.541123+0000","last_undegraded":"2024-02-13T00:50:13.541123+0000","last_fullsized":"2024-02-13T00:50:13.541123+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'34","last_scrub_stamp":"2024-02-13T00:49:46.101874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.101874+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":304,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543008+0000","last_change":"2024-02-13T00:49:47.075845+0000","last_active":"2024-02-13T00:50:13.543008+0000","last_peered":"2024-02-13T00:50:13.543008+0000","last_clean":"2024-02-13T00:50:13.543008+0000","last_became_active":"2024-02-13T00:45:15.446443+0000","last_became_peered":"2024-02-13T00:45:15.446443+0000","last_unstale":"2024-02-13T00:50:13.543008+0000","last_undegraded":"2024-02-13T00:50:13.543008+0000","last_fullsized":"2024-02-13T00:50:13.543008+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:47.075741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.075741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":299,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541333+0000","last_change":"2024-02-13T00:45:15.436286+0000","last_active":"2024-02-13T00:50:13.541333+0000","last_peered":"2024-02-13T00:50:13.541333+0000","last_clean":"2024-02-13T00:50:13.541333+0000","last_became_active":"2024-02-13T00:45:15.434559+0000","last_became_peered":"2024-02-13T00:45:15.434559+0000","last_unstale":"2024-02-13T00:50:13.541333+0000","last_undegraded":"2024-02-13T00:50:13.541333+0000","last_fullsized":"2024-02-13T00:50:13.541333+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":328,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540448+0000","last_change":"2024-02-13T00:49:36.841674+0000","last_active":"2024-02-13T00:50:13.540448+0000","last_peered":"2024-02-13T00:50:13.540448+0000","last_clean":"2024-02-13T00:50:13.540448+0000","last_became_active":"2024-02-13T00:46:01.444661+0000","last_became_peered":"2024-02-13T00:46:01.444661+0000","last_unstale":"2024-02-13T00:50:13.540448+0000","last_undegraded":"2024-02-13T00:50:13.540448+0000","last_fullsized":"2024-02-13T00:50:13.540448+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:36.841508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:36.841508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":369,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541138+0000","last_change":"2024-02-13T00:48:18.751148+0000","last_active":"2024-02-13T00:50:13.541138+0000","last_peered":"2024-02-13T00:50:13.541138+0000","last_clean":"2024-02-13T00:50:13.541138+0000","last_became_active":"2024-02-13T00:46:01.453842+0000","last_became_peered":"2024-02-13T00:46:01.453842+0000","last_unstale":"2024-02-13T00:50:13.541138+0000","last_undegraded":"2024-02-13T00:50:13.541138+0000","last_fullsized":"2024-02-13T00:50:13.541138+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:18.751081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:18.751081+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"467'42","reported_seq":530,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540523+0000","last_change":"2024-02-13T00:48:24.675807+0000","last_active":"2024-02-13T00:50:13.540523+0000","last_peered":"2024-02-13T00:50:13.540523+0000","last_clean":"2024-02-13T00:50:13.540523+0000","last_became_active":"2024-02-13T00:46:01.445058+0000","last_became_peered":"2024-02-13T00:46:01.445058+0000","last_unstale":"2024-02-13T00:50:13.540523+0000","last_undegraded":"2024-02-13T00:50:13.540523+0000","last_fullsized":"2024-02-13T00:50:13.540523+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'42","last_scrub_stamp":"2024-02-13T00:48:24.675687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.675687+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":591,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540275+0000","last_change":"2024-02-13T00:47:09.940673+0000","last_active":"2024-02-13T00:50:13.540275+0000","last_peered":"2024-02-13T00:50:13.540275+0000","last_clean":"2024-02-13T00:50:13.540275+0000","last_became_active":"2024-02-13T00:44:39.503590+0000","last_became_peered":"2024-02-13T00:44:39.503590+0000","last_unstale":"2024-02-13T00:50:13.540275+0000","last_undegraded":"2024-02-13T00:50:13.540275+0000","last_fullsized":"2024-02-13T00:50:13.540275+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:09.940605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:09.940605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":534,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541742+0000","last_change":"2024-02-13T00:46:01.448313+0000","last_active":"2024-02-13T00:50:13.541742+0000","last_peered":"2024-02-13T00:50:13.541742+0000","last_clean":"2024-02-13T00:50:13.541742+0000","last_became_active":"2024-02-13T00:46:01.447815+0000","last_became_peered":"2024-02-13T00:46:01.447815+0000","last_unstale":"2024-02-13T00:50:13.541742+0000","last_undegraded":"2024-02-13T00:50:13.541742+0000","last_fullsized":"2024-02-13T00:50:13.541742+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":526,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541028+0000","last_change":"2024-02-13T00:48:35.434713+0000","last_active":"2024-02-13T00:50:13.541028+0000","last_peered":"2024-02-13T00:50:13.541028+0000","last_clean":"2024-02-13T00:50:13.541028+0000","last_became_active":"2024-02-13T00:44:50.436058+0000","last_became_peered":"2024-02-13T00:44:50.436058+0000","last_unstale":"2024-02-13T00:50:13.541028+0000","last_undegraded":"2024-02-13T00:50:13.541028+0000","last_fullsized":"2024-02-13T00:50:13.541028+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_clean_scrub_stamp":"2024-02-13T00:48:35.434645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":295,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542055+0000","last_change":"2024-02-13T00:46:01.451625+0000","last_active":"2024-02-13T00:50:13.542055+0000","last_peered":"2024-02-13T00:50:13.542055+0000","last_clean":"2024-02-13T00:50:13.542055+0000","last_became_active":"2024-02-13T00:46:01.450587+0000","last_became_peered":"2024-02-13T00:46:01.450587+0000","last_unstale":"2024-02-13T00:50:13.542055+0000","last_undegraded":"2024-02-13T00:50:13.542055+0000","last_fullsized":"2024-02-13T00:50:13.542055+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:33.204969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:33.204969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":693,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541873+0000","last_change":"2024-02-13T00:47:05.718055+0000","last_active":"2024-02-13T00:50:13.541873+0000","last_peered":"2024-02-13T00:50:13.541873+0000","last_clean":"2024-02-13T00:50:13.541873+0000","last_became_active":"2024-02-13T00:46:01.443777+0000","last_became_peered":"2024-02-13T00:46:01.443777+0000","last_unstale":"2024-02-13T00:50:13.541873+0000","last_undegraded":"2024-02-13T00:50:13.541873+0000","last_fullsized":"2024-02-13T00:50:13.541873+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:47:05.717987+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:47:05.717987+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":404,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542663+0000","last_change":"2024-02-13T00:45:15.423607+0000","last_active":"2024-02-13T00:50:13.542663+0000","last_peered":"2024-02-13T00:50:13.542663+0000","last_clean":"2024-02-13T00:50:13.542663+0000","last_became_active":"2024-02-13T00:45:15.423415+0000","last_became_peered":"2024-02-13T00:45:15.423415+0000","last_unstale":"2024-02-13T00:50:13.542663+0000","last_undegraded":"2024-02-13T00:50:13.542663+0000","last_fullsized":"2024-02-13T00:50:13.542663+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:17.295459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:17.295459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":603,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.726735+0000","last_change":"2024-02-13T00:50:14.726735+0000","last_active":"2024-02-13T00:50:14.726735+0000","last_peered":"2024-02-13T00:50:14.726735+0000","last_clean":"2024-02-13T00:50:14.726735+0000","last_became_active":"2024-02-13T00:45:15.445557+0000","last_became_peered":"2024-02-13T00:45:15.445557+0000","last_unstale":"2024-02-13T00:50:14.726735+0000","last_undegraded":"2024-02-13T00:50:14.726735+0000","last_fullsized":"2024-02-13T00:50:14.726735+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.726660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.726660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":216,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540051+0000","last_change":"2024-02-13T00:48:55.050483+0000","last_active":"2024-02-13T00:50:13.540051+0000","last_peered":"2024-02-13T00:50:13.540051+0000","last_clean":"2024-02-13T00:50:13.540051+0000","last_became_active":"2024-02-13T00:46:01.441421+0000","last_became_peered":"2024-02-13T00:46:01.441421+0000","last_unstale":"2024-02-13T00:50:13.540051+0000","last_undegraded":"2024-02-13T00:50:13.540051+0000","last_fullsized":"2024-02-13T00:50:13.540051+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:55.050411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.050411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":231,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541328+0000","last_change":"2024-02-13T00:49:16.505082+0000","last_active":"2024-02-13T00:50:13.541328+0000","last_peered":"2024-02-13T00:50:13.541328+0000","last_clean":"2024-02-13T00:50:13.541328+0000","last_became_active":"2024-02-13T00:49:16.504828+0000","last_became_peered":"2024-02-13T00:49:16.504828+0000","last_unstale":"2024-02-13T00:50:13.541328+0000","last_undegraded":"2024-02-13T00:50:13.541328+0000","last_fullsized":"2024-02-13T00:50:13.541328+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:43.903553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:43.903553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":688,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542903+0000","last_change":"2024-02-13T00:46:34.691109+0000","last_active":"2024-02-13T00:50:13.542903+0000","last_peered":"2024-02-13T00:50:13.542903+0000","last_clean":"2024-02-13T00:50:13.542903+0000","last_became_active":"2024-02-13T00:46:01.448163+0000","last_became_peered":"2024-02-13T00:46:01.448163+0000","last_unstale":"2024-02-13T00:50:13.542903+0000","last_undegraded":"2024-02-13T00:50:13.542903+0000","last_fullsized":"2024-02-13T00:50:13.542903+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:46:34.690980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.690980+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":299,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.537684+0000","last_change":"2024-02-13T00:46:37.056299+0000","last_active":"2024-02-13T00:50:13.537684+0000","last_peered":"2024-02-13T00:50:13.537684+0000","last_clean":"2024-02-13T00:50:13.537684+0000","last_became_active":"2024-02-13T00:46:01.446104+0000","last_became_peered":"2024-02-13T00:46:01.446104+0000","last_unstale":"2024-02-13T00:50:13.537684+0000","last_undegraded":"2024-02-13T00:50:13.537684+0000","last_fullsized":"2024-02-13T00:50:13.537684+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:37.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":580,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541267+0000","last_change":"2024-02-13T00:46:01.431899+0000","last_active":"2024-02-13T00:50:13.541267+0000","last_peered":"2024-02-13T00:50:13.541267+0000","last_clean":"2024-02-13T00:50:13.541267+0000","last_became_active":"2024-02-13T00:46:01.430623+0000","last_became_peered":"2024-02-13T00:46:01.430623+0000","last_unstale":"2024-02-13T00:50:13.541267+0000","last_undegraded":"2024-02-13T00:50:13.541267+0000","last_fullsized":"2024-02-13T00:50:13.541267+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:46.556229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.556229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":589,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540699+0000","last_change":"2024-02-13T00:46:01.450548+0000","last_active":"2024-02-13T00:50:13.540699+0000","last_peered":"2024-02-13T00:50:13.540699+0000","last_clean":"2024-02-13T00:50:13.540699+0000","last_became_active":"2024-02-13T00:46:01.450176+0000","last_became_peered":"2024-02-13T00:46:01.450176+0000","last_unstale":"2024-02-13T00:50:13.540699+0000","last_undegraded":"2024-02-13T00:50:13.540699+0000","last_fullsized":"2024-02-13T00:50:13.540699+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":441,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543123+0000","last_change":"2024-02-13T00:46:01.460470+0000","last_active":"2024-02-13T00:50:13.543123+0000","last_peered":"2024-02-13T00:50:13.543123+0000","last_clean":"2024-02-13T00:50:13.543123+0000","last_became_active":"2024-02-13T00:46:01.459859+0000","last_became_peered":"2024-02-13T00:46:01.459859+0000","last_unstale":"2024-02-13T00:50:13.543123+0000","last_undegraded":"2024-02-13T00:50:13.543123+0000","last_fullsized":"2024-02-13T00:50:13.543123+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:01.179031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:01.179031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":245,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542313+0000","last_change":"2024-02-13T00:47:33.932830+0000","last_active":"2024-02-13T00:50:13.542313+0000","last_peered":"2024-02-13T00:50:13.542313+0000","last_clean":"2024-02-13T00:50:13.542313+0000","last_became_active":"2024-02-13T00:46:01.455170+0000","last_became_peered":"2024-02-13T00:46:01.455170+0000","last_unstale":"2024-02-13T00:50:13.542313+0000","last_undegraded":"2024-02-13T00:50:13.542313+0000","last_fullsized":"2024-02-13T00:50:13.542313+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:33.932766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:33.932766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":345,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543151+0000","last_change":"2024-02-13T00:46:57.381602+0000","last_active":"2024-02-13T00:50:13.543151+0000","last_peered":"2024-02-13T00:50:13.543151+0000","last_clean":"2024-02-13T00:50:13.543151+0000","last_became_active":"2024-02-13T00:45:15.446969+0000","last_became_peered":"2024-02-13T00:45:15.446969+0000","last_unstale":"2024-02-13T00:50:13.543151+0000","last_undegraded":"2024-02-13T00:50:13.543151+0000","last_fullsized":"2024-02-13T00:50:13.543151+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:46:57.381540+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:46:57.381540+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":545,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540326+0000","last_change":"2024-02-13T00:47:39.822745+0000","last_active":"2024-02-13T00:50:13.540326+0000","last_peered":"2024-02-13T00:50:13.540326+0000","last_clean":"2024-02-13T00:50:13.540326+0000","last_became_active":"2024-02-13T00:45:15.434498+0000","last_became_peered":"2024-02-13T00:45:15.434498+0000","last_unstale":"2024-02-13T00:50:13.540326+0000","last_undegraded":"2024-02-13T00:50:13.540326+0000","last_fullsized":"2024-02-13T00:50:13.540326+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:39.822619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:47:39.822619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":543,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540828+0000","last_change":"2024-02-13T00:47:04.493911+0000","last_active":"2024-02-13T00:50:13.540828+0000","last_peered":"2024-02-13T00:50:13.540828+0000","last_clean":"2024-02-13T00:50:13.540828+0000","last_became_active":"2024-02-13T00:46:01.447823+0000","last_became_peered":"2024-02-13T00:46:01.447823+0000","last_unstale":"2024-02-13T00:50:13.540828+0000","last_undegraded":"2024-02-13T00:50:13.540828+0000","last_fullsized":"2024-02-13T00:50:13.540828+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:04.493841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:04.493841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":416,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541525+0000","last_change":"2024-02-13T00:48:25.359625+0000","last_active":"2024-02-13T00:50:13.541525+0000","last_peered":"2024-02-13T00:50:13.541525+0000","last_clean":"2024-02-13T00:50:13.541525+0000","last_became_active":"2024-02-13T00:46:01.460001+0000","last_became_peered":"2024-02-13T00:46:01.460001+0000","last_unstale":"2024-02-13T00:50:13.541525+0000","last_undegraded":"2024-02-13T00:50:13.541525+0000","last_fullsized":"2024-02-13T00:50:13.541525+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:25.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:48:25.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":437,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540570+0000","last_change":"2024-02-13T00:46:01.450879+0000","last_active":"2024-02-13T00:50:13.540570+0000","last_peered":"2024-02-13T00:50:13.540570+0000","last_clean":"2024-02-13T00:50:13.540570+0000","last_became_active":"2024-02-13T00:46:01.450088+0000","last_became_peered":"2024-02-13T00:46:01.450088+0000","last_unstale":"2024-02-13T00:50:13.540570+0000","last_undegraded":"2024-02-13T00:50:13.540570+0000","last_fullsized":"2024-02-13T00:50:13.540570+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":526,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543273+0000","last_change":"2024-02-13T00:49:42.854702+0000","last_active":"2024-02-13T00:50:13.543273+0000","last_peered":"2024-02-13T00:50:13.543273+0000","last_clean":"2024-02-13T00:50:13.543273+0000","last_became_active":"2024-02-13T00:46:01.458408+0000","last_became_peered":"2024-02-13T00:46:01.458408+0000","last_unstale":"2024-02-13T00:50:13.543273+0000","last_undegraded":"2024-02-13T00:50:13.543273+0000","last_fullsized":"2024-02-13T00:50:13.543273+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:42.854585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:42.854585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":268,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539903+0000","last_change":"2024-02-13T00:49:19.638381+0000","last_active":"2024-02-13T00:50:13.539903+0000","last_peered":"2024-02-13T00:50:13.539903+0000","last_clean":"2024-02-13T00:50:13.539903+0000","last_became_active":"2024-02-13T00:49:19.638083+0000","last_became_peered":"2024-02-13T00:49:19.638083+0000","last_unstale":"2024-02-13T00:50:13.539903+0000","last_undegraded":"2024-02-13T00:50:13.539903+0000","last_fullsized":"2024-02-13T00:50:13.539903+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:53.013103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.013103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"467'23","reported_seq":633,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540975+0000","last_change":"2024-02-13T00:49:07.192972+0000","last_active":"2024-02-13T00:50:13.540975+0000","last_peered":"2024-02-13T00:50:13.540975+0000","last_clean":"2024-02-13T00:50:13.540975+0000","last_became_active":"2024-02-13T00:46:01.448219+0000","last_became_peered":"2024-02-13T00:46:01.448219+0000","last_unstale":"2024-02-13T00:50:13.540975+0000","last_undegraded":"2024-02-13T00:50:13.540975+0000","last_fullsized":"2024-02-13T00:50:13.540975+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'23","last_scrub_stamp":"2024-02-13T00:49:07.192902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:07.192902+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":500,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541213+0000","last_change":"2024-02-13T00:49:46.851558+0000","last_active":"2024-02-13T00:50:13.541213+0000","last_peered":"2024-02-13T00:50:13.541213+0000","last_clean":"2024-02-13T00:50:13.541213+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:50:13.541213+0000","last_undegraded":"2024-02-13T00:50:13.541213+0000","last_fullsized":"2024-02-13T00:50:13.541213+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.851486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.851486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":1353,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.547741+0000","last_change":"2024-02-13T00:45:54.075270+0000","last_active":"2024-02-13T00:50:14.547741+0000","last_peered":"2024-02-13T00:50:14.547741+0000","last_clean":"2024-02-13T00:50:14.547741+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:50:14.547741+0000","last_undegraded":"2024-02-13T00:50:14.547741+0000","last_fullsized":"2024-02-13T00:50:14.547741+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:45:54.075194+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":741,"num_read_kb":736,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":308,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542982+0000","last_change":"2024-02-13T00:46:01.460282+0000","last_active":"2024-02-13T00:50:13.542982+0000","last_peered":"2024-02-13T00:50:13.542982+0000","last_clean":"2024-02-13T00:50:13.542982+0000","last_became_active":"2024-02-13T00:46:01.459473+0000","last_became_peered":"2024-02-13T00:46:01.459473+0000","last_unstale":"2024-02-13T00:50:13.542982+0000","last_undegraded":"2024-02-13T00:50:13.542982+0000","last_fullsized":"2024-02-13T00:50:13.542982+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":362,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540111+0000","last_change":"2024-02-13T00:45:37.135398+0000","last_active":"2024-02-13T00:50:13.540111+0000","last_peered":"2024-02-13T00:50:13.540111+0000","last_clean":"2024-02-13T00:50:13.540111+0000","last_became_active":"2024-02-13T00:45:15.435588+0000","last_became_peered":"2024-02-13T00:45:15.435588+0000","last_unstale":"2024-02-13T00:50:13.540111+0000","last_undegraded":"2024-02-13T00:50:13.540111+0000","last_fullsized":"2024-02-13T00:50:13.540111+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.135332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.135332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":259,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542228+0000","last_change":"2024-02-13T00:45:31.183763+0000","last_active":"2024-02-13T00:50:13.542228+0000","last_peered":"2024-02-13T00:50:13.542228+0000","last_clean":"2024-02-13T00:50:13.542228+0000","last_became_active":"2024-02-13T00:45:06.421025+0000","last_became_peered":"2024-02-13T00:45:06.421025+0000","last_unstale":"2024-02-13T00:50:13.542228+0000","last_undegraded":"2024-02-13T00:50:13.542228+0000","last_fullsized":"2024-02-13T00:50:13.542228+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.183682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.183682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"467'65","reported_seq":304,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.357515+0000","last_change":"2024-02-13T00:49:47.928628+0000","last_active":"2024-02-13T00:50:14.357515+0000","last_peered":"2024-02-13T00:50:14.357515+0000","last_clean":"2024-02-13T00:50:14.357515+0000","last_became_active":"2024-02-13T00:46:01.449059+0000","last_became_peered":"2024-02-13T00:46:01.449059+0000","last_unstale":"2024-02-13T00:50:14.357515+0000","last_undegraded":"2024-02-13T00:50:14.357515+0000","last_fullsized":"2024-02-13T00:50:14.357515+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'65","last_scrub_stamp":"2024-02-13T00:49:47.928548+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.928548+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":2148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":507,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538601+0000","last_change":"2024-02-13T00:46:01.453042+0000","last_active":"2024-02-13T00:50:13.538601+0000","last_peered":"2024-02-13T00:50:13.538601+0000","last_clean":"2024-02-13T00:50:13.538601+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:50:13.538601+0000","last_undegraded":"2024-02-13T00:50:13.538601+0000","last_fullsized":"2024-02-13T00:50:13.538601+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.045912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.045912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":125,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542413+0000","last_change":"2024-02-13T00:48:06.705030+0000","last_active":"2024-02-13T00:50:13.542413+0000","last_peered":"2024-02-13T00:50:13.542413+0000","last_clean":"2024-02-13T00:50:13.542413+0000","last_became_active":"2024-02-13T00:48:06.704651+0000","last_became_peered":"2024-02-13T00:48:06.704651+0000","last_unstale":"2024-02-13T00:50:13.542413+0000","last_undegraded":"2024-02-13T00:50:13.542413+0000","last_fullsized":"2024-02-13T00:50:13.542413+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":295,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542117+0000","last_change":"2024-02-13T00:48:17.340891+0000","last_active":"2024-02-13T00:50:13.542117+0000","last_peered":"2024-02-13T00:50:13.542117+0000","last_clean":"2024-02-13T00:50:13.542117+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:50:13.542117+0000","last_undegraded":"2024-02-13T00:50:13.542117+0000","last_fullsized":"2024-02-13T00:50:13.542117+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:17.340797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:17.340797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":771,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.357989+0000","last_change":"2024-02-13T00:48:32.669477+0000","last_active":"2024-02-13T00:50:14.357989+0000","last_peered":"2024-02-13T00:50:14.357989+0000","last_clean":"2024-02-13T00:50:14.357989+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:50:14.357989+0000","last_undegraded":"2024-02-13T00:50:14.357989+0000","last_fullsized":"2024-02-13T00:50:14.357989+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14","last_scrub_stamp":"2024-02-13T00:48:32.669406+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.669406+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":330,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539037+0000","last_change":"2024-02-13T00:48:20.928236+0000","last_active":"2024-02-13T00:50:13.539037+0000","last_peered":"2024-02-13T00:50:13.539037+0000","last_clean":"2024-02-13T00:50:13.539037+0000","last_became_active":"2024-02-13T00:46:01.442003+0000","last_became_peered":"2024-02-13T00:46:01.442003+0000","last_unstale":"2024-02-13T00:50:13.539037+0000","last_undegraded":"2024-02-13T00:50:13.539037+0000","last_fullsized":"2024-02-13T00:50:13.539037+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:48:20.928158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:20.928158+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":578,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540206+0000","last_change":"2024-02-13T00:49:33.040619+0000","last_active":"2024-02-13T00:50:13.540206+0000","last_peered":"2024-02-13T00:50:13.540206+0000","last_clean":"2024-02-13T00:50:13.540206+0000","last_became_active":"2024-02-13T00:45:15.422546+0000","last_became_peered":"2024-02-13T00:45:15.422546+0000","last_unstale":"2024-02-13T00:50:13.540206+0000","last_undegraded":"2024-02-13T00:50:13.540206+0000","last_fullsized":"2024-02-13T00:50:13.540206+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:33.040516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:49:33.040516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":587,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540244+0000","last_change":"2024-02-13T00:46:01.436997+0000","last_active":"2024-02-13T00:50:13.540244+0000","last_peered":"2024-02-13T00:50:13.540244+0000","last_clean":"2024-02-13T00:50:13.540244+0000","last_became_active":"2024-02-13T00:46:01.435178+0000","last_became_peered":"2024-02-13T00:46:01.435178+0000","last_unstale":"2024-02-13T00:50:13.540244+0000","last_undegraded":"2024-02-13T00:50:13.540244+0000","last_fullsized":"2024-02-13T00:50:13.540244+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":522,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538375+0000","last_change":"2024-02-13T00:45:15.445626+0000","last_active":"2024-02-13T00:50:13.538375+0000","last_peered":"2024-02-13T00:50:13.538375+0000","last_clean":"2024-02-13T00:50:13.538375+0000","last_became_active":"2024-02-13T00:45:15.445117+0000","last_became_peered":"2024-02-13T00:45:15.445117+0000","last_unstale":"2024-02-13T00:50:13.538375+0000","last_undegraded":"2024-02-13T00:50:13.538375+0000","last_fullsized":"2024-02-13T00:50:13.538375+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.171185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":305,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540776+0000","last_change":"2024-02-13T00:49:12.252240+0000","last_active":"2024-02-13T00:50:13.540776+0000","last_peered":"2024-02-13T00:50:13.540776+0000","last_clean":"2024-02-13T00:50:13.540776+0000","last_became_active":"2024-02-13T00:49:12.251980+0000","last_became_peered":"2024-02-13T00:49:12.251980+0000","last_unstale":"2024-02-13T00:50:13.540776+0000","last_undegraded":"2024-02-13T00:50:13.540776+0000","last_fullsized":"2024-02-13T00:50:13.540776+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.592320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.592320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":648,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538333+0000","last_change":"2024-02-13T00:49:31.072963+0000","last_active":"2024-02-13T00:50:13.538333+0000","last_peered":"2024-02-13T00:50:13.538333+0000","last_clean":"2024-02-13T00:50:13.538333+0000","last_became_active":"2024-02-13T00:44:39.501886+0000","last_became_peered":"2024-02-13T00:44:39.501886+0000","last_unstale":"2024-02-13T00:50:13.538333+0000","last_undegraded":"2024-02-13T00:50:13.538333+0000","last_fullsized":"2024-02-13T00:50:13.538333+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'30","last_scrub_stamp":"2024-02-13T00:49:31.072862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:31.072862+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":566,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543051+0000","last_change":"2024-02-13T00:48:58.861408+0000","last_active":"2024-02-13T00:50:13.543051+0000","last_peered":"2024-02-13T00:50:13.543051+0000","last_clean":"2024-02-13T00:50:13.543051+0000","last_became_active":"2024-02-13T00:46:01.457926+0000","last_became_peered":"2024-02-13T00:46:01.457926+0000","last_unstale":"2024-02-13T00:50:13.543051+0000","last_undegraded":"2024-02-13T00:50:13.543051+0000","last_fullsized":"2024-02-13T00:50:13.543051+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:48:58.861325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:58.861325+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":373,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539209+0000","last_change":"2024-02-13T00:47:06.895693+0000","last_active":"2024-02-13T00:50:13.539209+0000","last_peered":"2024-02-13T00:50:13.539209+0000","last_clean":"2024-02-13T00:50:13.539209+0000","last_became_active":"2024-02-13T00:45:15.435404+0000","last_became_peered":"2024-02-13T00:45:15.435404+0000","last_unstale":"2024-02-13T00:50:13.539209+0000","last_undegraded":"2024-02-13T00:50:13.539209+0000","last_fullsized":"2024-02-13T00:50:13.539209+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:06.895612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:47:06.895612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":521,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542952+0000","last_change":"2024-02-13T00:46:01.450436+0000","last_active":"2024-02-13T00:50:13.542952+0000","last_peered":"2024-02-13T00:50:13.542952+0000","last_clean":"2024-02-13T00:50:13.542952+0000","last_became_active":"2024-02-13T00:46:01.449234+0000","last_became_peered":"2024-02-13T00:46:01.449234+0000","last_unstale":"2024-02-13T00:50:13.542952+0000","last_undegraded":"2024-02-13T00:50:13.542952+0000","last_fullsized":"2024-02-13T00:50:13.542952+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.110742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.110742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"467'27","reported_seq":714,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542903+0000","last_change":"2024-02-13T00:49:35.834793+0000","last_active":"2024-02-13T00:50:13.542903+0000","last_peered":"2024-02-13T00:50:13.542903+0000","last_clean":"2024-02-13T00:50:13.542903+0000","last_became_active":"2024-02-13T00:46:01.456876+0000","last_became_peered":"2024-02-13T00:46:01.456876+0000","last_unstale":"2024-02-13T00:50:13.542903+0000","last_undegraded":"2024-02-13T00:50:13.542903+0000","last_fullsized":"2024-02-13T00:50:13.542903+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:35.834669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.834669+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":280,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.537623+0000","last_change":"2024-02-13T00:46:30.095387+0000","last_active":"2024-02-13T00:50:13.537623+0000","last_peered":"2024-02-13T00:50:13.537623+0000","last_clean":"2024-02-13T00:50:13.537623+0000","last_became_active":"2024-02-13T00:44:50.441300+0000","last_became_peered":"2024-02-13T00:44:50.441300+0000","last_unstale":"2024-02-13T00:50:13.537623+0000","last_undegraded":"2024-02-13T00:50:13.537623+0000","last_fullsized":"2024-02-13T00:50:13.537623+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.095307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":294,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542590+0000","last_change":"2024-02-13T00:46:01.454597+0000","last_active":"2024-02-13T00:50:13.542590+0000","last_peered":"2024-02-13T00:50:13.542590+0000","last_clean":"2024-02-13T00:50:13.542590+0000","last_became_active":"2024-02-13T00:46:01.454101+0000","last_became_peered":"2024-02-13T00:46:01.454101+0000","last_unstale":"2024-02-13T00:50:13.542590+0000","last_undegraded":"2024-02-13T00:50:13.542590+0000","last_fullsized":"2024-02-13T00:50:13.542590+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.651091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.651091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":290,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542892+0000","last_change":"2024-02-13T00:44:50.441196+0000","last_active":"2024-02-13T00:50:13.542892+0000","last_peered":"2024-02-13T00:50:13.542892+0000","last_clean":"2024-02-13T00:50:13.542892+0000","last_became_active":"2024-02-13T00:44:50.438274+0000","last_became_peered":"2024-02-13T00:44:50.438274+0000","last_unstale":"2024-02-13T00:50:13.542892+0000","last_undegraded":"2024-02-13T00:50:13.542892+0000","last_fullsized":"2024-02-13T00:50:13.542892+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.510219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.510219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":449,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541197+0000","last_change":"2024-02-13T00:46:34.141015+0000","last_active":"2024-02-13T00:50:13.541197+0000","last_peered":"2024-02-13T00:50:13.541197+0000","last_clean":"2024-02-13T00:50:13.541197+0000","last_became_active":"2024-02-13T00:45:15.436020+0000","last_became_peered":"2024-02-13T00:45:15.436020+0000","last_unstale":"2024-02-13T00:50:13.541197+0000","last_undegraded":"2024-02-13T00:50:13.541197+0000","last_fullsized":"2024-02-13T00:50:13.541197+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:46:34.140928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.140928+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":286,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541734+0000","last_change":"2024-02-13T00:45:00.311635+0000","last_active":"2024-02-13T00:50:13.541734+0000","last_peered":"2024-02-13T00:50:13.541734+0000","last_clean":"2024-02-13T00:50:13.541734+0000","last_became_active":"2024-02-13T00:44:39.512402+0000","last_became_peered":"2024-02-13T00:44:39.512402+0000","last_unstale":"2024-02-13T00:50:13.541734+0000","last_undegraded":"2024-02-13T00:50:13.541734+0000","last_fullsized":"2024-02-13T00:50:13.541734+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:00.311558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:45:00.311558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":299,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543049+0000","last_change":"2024-02-13T00:46:01.451840+0000","last_active":"2024-02-13T00:50:13.543049+0000","last_peered":"2024-02-13T00:50:13.543049+0000","last_clean":"2024-02-13T00:50:13.543049+0000","last_became_active":"2024-02-13T00:46:01.450312+0000","last_became_peered":"2024-02-13T00:46:01.450312+0000","last_unstale":"2024-02-13T00:50:13.543049+0000","last_undegraded":"2024-02-13T00:50:13.543049+0000","last_fullsized":"2024-02-13T00:50:13.543049+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.042291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.042291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":371,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542750+0000","last_change":"2024-02-13T00:46:01.454858+0000","last_active":"2024-02-13T00:50:13.542750+0000","last_peered":"2024-02-13T00:50:13.542750+0000","last_clean":"2024-02-13T00:50:13.542750+0000","last_became_active":"2024-02-13T00:46:01.454284+0000","last_became_peered":"2024-02-13T00:46:01.454284+0000","last_unstale":"2024-02-13T00:50:13.542750+0000","last_undegraded":"2024-02-13T00:50:13.542750+0000","last_fullsized":"2024-02-13T00:50:13.542750+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:34.523594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.523594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":532,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540523+0000","last_change":"2024-02-13T00:49:06.154997+0000","last_active":"2024-02-13T00:50:13.540523+0000","last_peered":"2024-02-13T00:50:13.540523+0000","last_clean":"2024-02-13T00:50:13.540523+0000","last_became_active":"2024-02-13T00:46:01.450113+0000","last_became_peered":"2024-02-13T00:46:01.450113+0000","last_unstale":"2024-02-13T00:50:13.540523+0000","last_undegraded":"2024-02-13T00:50:13.540523+0000","last_fullsized":"2024-02-13T00:50:13.540523+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'39","last_scrub_stamp":"2024-02-13T00:49:06.154931+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:49:06.154931+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":546,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541774+0000","last_change":"2024-02-13T00:49:29.916234+0000","last_active":"2024-02-13T00:50:13.541774+0000","last_peered":"2024-02-13T00:50:13.541774+0000","last_clean":"2024-02-13T00:50:13.541774+0000","last_became_active":"2024-02-13T00:46:01.454310+0000","last_became_peered":"2024-02-13T00:46:01.454310+0000","last_unstale":"2024-02-13T00:50:13.541774+0000","last_undegraded":"2024-02-13T00:50:13.541774+0000","last_fullsized":"2024-02-13T00:50:13.541774+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.916152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"404'9","reported_seq":287,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.357405+0000","last_change":"2024-02-13T00:47:37.013003+0000","last_active":"2024-02-13T00:50:14.357405+0000","last_peered":"2024-02-13T00:50:14.357405+0000","last_clean":"2024-02-13T00:50:14.357405+0000","last_became_active":"2024-02-13T00:46:01.448144+0000","last_became_peered":"2024-02-13T00:46:01.448144+0000","last_unstale":"2024-02-13T00:50:14.357405+0000","last_undegraded":"2024-02-13T00:50:14.357405+0000","last_fullsized":"2024-02-13T00:50:14.357405+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9","last_scrub_stamp":"2024-02-13T00:47:37.012934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:37.012934+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":583,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542757+0000","last_change":"2024-02-13T00:48:26.647460+0000","last_active":"2024-02-13T00:50:13.542757+0000","last_peered":"2024-02-13T00:50:13.542757+0000","last_clean":"2024-02-13T00:50:13.542757+0000","last_became_active":"2024-02-13T00:46:01.447236+0000","last_became_peered":"2024-02-13T00:46:01.447236+0000","last_unstale":"2024-02-13T00:50:13.542757+0000","last_undegraded":"2024-02-13T00:50:13.542757+0000","last_fullsized":"2024-02-13T00:50:13.542757+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:26.647367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:26.647367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":254,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.849751+0000","last_change":"2024-02-13T00:50:14.849751+0000","last_active":"2024-02-13T00:50:14.849751+0000","last_peered":"2024-02-13T00:50:14.849751+0000","last_clean":"2024-02-13T00:50:14.849751+0000","last_became_active":"2024-02-13T00:46:01.455989+0000","last_became_peered":"2024-02-13T00:46:01.455989+0000","last_unstale":"2024-02-13T00:50:14.849751+0000","last_undegraded":"2024-02-13T00:50:14.849751+0000","last_fullsized":"2024-02-13T00:50:14.849751+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.849687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.849687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"467'22","reported_seq":568,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542679+0000","last_change":"2024-02-13T00:47:35.369710+0000","last_active":"2024-02-13T00:50:13.542679+0000","last_peered":"2024-02-13T00:50:13.542679+0000","last_clean":"2024-02-13T00:50:13.542679+0000","last_became_active":"2024-02-13T00:46:01.460321+0000","last_became_peered":"2024-02-13T00:46:01.460321+0000","last_unstale":"2024-02-13T00:50:13.542679+0000","last_undegraded":"2024-02-13T00:50:13.542679+0000","last_fullsized":"2024-02-13T00:50:13.542679+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:47:35.369616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:35.369616+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":598,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541876+0000","last_change":"2024-02-13T00:49:05.887739+0000","last_active":"2024-02-13T00:50:13.541876+0000","last_peered":"2024-02-13T00:50:13.541876+0000","last_clean":"2024-02-13T00:50:13.541876+0000","last_became_active":"2024-02-13T00:46:01.431926+0000","last_became_peered":"2024-02-13T00:46:01.431926+0000","last_unstale":"2024-02-13T00:50:13.541876+0000","last_undegraded":"2024-02-13T00:50:13.541876+0000","last_fullsized":"2024-02-13T00:50:13.541876+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:05.887665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:05.887665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"363'14","reported_seq":430,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:14.357489+0000","last_change":"2024-02-13T00:47:07.942748+0000","last_active":"2024-02-13T00:50:14.357489+0000","last_peered":"2024-02-13T00:50:14.357489+0000","last_clean":"2024-02-13T00:50:14.357489+0000","last_became_active":"2024-02-13T00:45:15.423066+0000","last_became_peered":"2024-02-13T00:45:15.423066+0000","last_unstale":"2024-02-13T00:50:14.357489+0000","last_undegraded":"2024-02-13T00:50:14.357489+0000","last_fullsized":"2024-02-13T00:50:14.357489+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:47:07.942668+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:47:07.942668+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":217,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542909+0000","last_change":"2024-02-13T00:47:02.789298+0000","last_active":"2024-02-13T00:50:13.542909+0000","last_peered":"2024-02-13T00:50:13.542909+0000","last_clean":"2024-02-13T00:50:13.542909+0000","last_became_active":"2024-02-13T00:46:01.446960+0000","last_became_peered":"2024-02-13T00:46:01.446960+0000","last_unstale":"2024-02-13T00:50:13.542909+0000","last_undegraded":"2024-02-13T00:50:13.542909+0000","last_fullsized":"2024-02-13T00:50:13.542909+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:02.789189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.789189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":455,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541842+0000","last_change":"2024-02-13T00:46:01.450308+0000","last_active":"2024-02-13T00:50:13.541842+0000","last_peered":"2024-02-13T00:50:13.541842+0000","last_clean":"2024-02-13T00:50:13.541842+0000","last_became_active":"2024-02-13T00:46:01.449694+0000","last_became_peered":"2024-02-13T00:46:01.449694+0000","last_unstale":"2024-02-13T00:50:13.541842+0000","last_undegraded":"2024-02-13T00:50:13.541842+0000","last_fullsized":"2024-02-13T00:50:13.541842+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:27.073034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:27.073034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":653,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538944+0000","last_change":"2024-02-13T00:49:38.007273+0000","last_active":"2024-02-13T00:50:13.538944+0000","last_peered":"2024-02-13T00:50:13.538944+0000","last_clean":"2024-02-13T00:50:13.538944+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:50:13.538944+0000","last_undegraded":"2024-02-13T00:50:13.538944+0000","last_fullsized":"2024-02-13T00:50:13.538944+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'19","last_scrub_stamp":"2024-02-13T00:49:38.007204+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:49:38.007204+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":299,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542961+0000","last_change":"2024-02-13T00:46:01.463310+0000","last_active":"2024-02-13T00:50:13.542961+0000","last_peered":"2024-02-13T00:50:13.542961+0000","last_clean":"2024-02-13T00:50:13.542961+0000","last_became_active":"2024-02-13T00:46:01.462898+0000","last_became_peered":"2024-02-13T00:46:01.462898+0000","last_unstale":"2024-02-13T00:50:13.542961+0000","last_undegraded":"2024-02-13T00:50:13.542961+0000","last_fullsized":"2024-02-13T00:50:13.542961+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.095888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":362,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541760+0000","last_change":"2024-02-13T00:46:32.638418+0000","last_active":"2024-02-13T00:50:13.541760+0000","last_peered":"2024-02-13T00:50:13.541760+0000","last_clean":"2024-02-13T00:50:13.541760+0000","last_became_active":"2024-02-13T00:46:01.443279+0000","last_became_peered":"2024-02-13T00:46:01.443279+0000","last_unstale":"2024-02-13T00:50:13.541760+0000","last_undegraded":"2024-02-13T00:50:13.541760+0000","last_fullsized":"2024-02-13T00:50:13.541760+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.638349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":565,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540161+0000","last_change":"2024-02-13T00:48:27.659266+0000","last_active":"2024-02-13T00:50:13.540161+0000","last_peered":"2024-02-13T00:50:13.540161+0000","last_clean":"2024-02-13T00:50:13.540161+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:50:13.540161+0000","last_undegraded":"2024-02-13T00:50:13.540161+0000","last_fullsized":"2024-02-13T00:50:13.540161+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:27.659197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:48:27.659197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":295,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541433+0000","last_change":"2024-02-13T00:45:40.233264+0000","last_active":"2024-02-13T00:50:13.541433+0000","last_peered":"2024-02-13T00:50:13.541433+0000","last_clean":"2024-02-13T00:50:13.541433+0000","last_became_active":"2024-02-13T00:45:15.447051+0000","last_became_peered":"2024-02-13T00:45:15.447051+0000","last_unstale":"2024-02-13T00:50:13.541433+0000","last_undegraded":"2024-02-13T00:50:13.541433+0000","last_fullsized":"2024-02-13T00:50:13.541433+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:40.233185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:40.233185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":331,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541345+0000","last_change":"2024-02-13T00:48:34.458358+0000","last_active":"2024-02-13T00:50:13.541345+0000","last_peered":"2024-02-13T00:50:13.541345+0000","last_clean":"2024-02-13T00:50:13.541345+0000","last_became_active":"2024-02-13T00:44:50.435203+0000","last_became_peered":"2024-02-13T00:44:50.435203+0000","last_unstale":"2024-02-13T00:50:13.541345+0000","last_undegraded":"2024-02-13T00:50:13.541345+0000","last_fullsized":"2024-02-13T00:50:13.541345+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:48:34.458264+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:48:34.458264+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":573,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539691+0000","last_change":"2024-02-13T00:46:01.457220+0000","last_active":"2024-02-13T00:50:13.539691+0000","last_peered":"2024-02-13T00:50:13.539691+0000","last_clean":"2024-02-13T00:50:13.539691+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:50:13.539691+0000","last_undegraded":"2024-02-13T00:50:13.539691+0000","last_fullsized":"2024-02-13T00:50:13.539691+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.097481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":587,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538974+0000","last_change":"2024-02-13T00:46:01.453391+0000","last_active":"2024-02-13T00:50:13.538974+0000","last_peered":"2024-02-13T00:50:13.538974+0000","last_clean":"2024-02-13T00:50:13.538974+0000","last_became_active":"2024-02-13T00:46:01.452730+0000","last_became_peered":"2024-02-13T00:46:01.452730+0000","last_unstale":"2024-02-13T00:50:13.538974+0000","last_undegraded":"2024-02-13T00:50:13.538974+0000","last_fullsized":"2024-02-13T00:50:13.538974+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:26.218807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:44:26.218807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":371,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541288+0000","last_change":"2024-02-13T00:48:29.681480+0000","last_active":"2024-02-13T00:50:13.541288+0000","last_peered":"2024-02-13T00:50:13.541288+0000","last_clean":"2024-02-13T00:50:13.541288+0000","last_became_active":"2024-02-13T00:46:01.454287+0000","last_became_peered":"2024-02-13T00:46:01.454287+0000","last_unstale":"2024-02-13T00:50:13.541288+0000","last_undegraded":"2024-02-13T00:50:13.541288+0000","last_fullsized":"2024-02-13T00:50:13.541288+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:29.681414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:29.681414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":304,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540160+0000","last_change":"2024-02-13T00:49:02.260116+0000","last_active":"2024-02-13T00:50:13.540160+0000","last_peered":"2024-02-13T00:50:13.540160+0000","last_clean":"2024-02-13T00:50:13.540160+0000","last_became_active":"2024-02-13T00:46:01.448671+0000","last_became_peered":"2024-02-13T00:46:01.448671+0000","last_unstale":"2024-02-13T00:50:13.540160+0000","last_undegraded":"2024-02-13T00:50:13.540160+0000","last_fullsized":"2024-02-13T00:50:13.540160+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:02.260045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:02.260045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":400,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.537949+0000","last_change":"2024-02-13T00:44:40.371566+0000","last_active":"2024-02-13T00:50:13.537949+0000","last_peered":"2024-02-13T00:50:13.537949+0000","last_clean":"2024-02-13T00:50:13.537949+0000","last_became_active":"2024-02-13T00:44:39.501040+0000","last_became_peered":"2024-02-13T00:44:39.501040+0000","last_unstale":"2024-02-13T00:50:13.537949+0000","last_undegraded":"2024-02-13T00:50:13.537949+0000","last_fullsized":"2024-02-13T00:50:13.537949+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:44:32.180113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:44:32.180113+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":530,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541646+0000","last_change":"2024-02-13T00:48:32.441279+0000","last_active":"2024-02-13T00:50:13.541646+0000","last_peered":"2024-02-13T00:50:13.541646+0000","last_clean":"2024-02-13T00:50:13.541646+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:50:13.541646+0000","last_undegraded":"2024-02-13T00:50:13.541646+0000","last_fullsized":"2024-02-13T00:50:13.541646+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.441170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.441170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":295,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541343+0000","last_change":"2024-02-13T00:48:28.353456+0000","last_active":"2024-02-13T00:50:13.541343+0000","last_peered":"2024-02-13T00:50:13.541343+0000","last_clean":"2024-02-13T00:50:13.541343+0000","last_became_active":"2024-02-13T00:45:15.435820+0000","last_became_peered":"2024-02-13T00:45:15.435820+0000","last_unstale":"2024-02-13T00:50:13.541343+0000","last_undegraded":"2024-02-13T00:50:13.541343+0000","last_fullsized":"2024-02-13T00:50:13.541343+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:28.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:28.353347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":287,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538400+0000","last_change":"2024-02-13T00:49:29.052728+0000","last_active":"2024-02-13T00:50:13.538400+0000","last_peered":"2024-02-13T00:50:13.538400+0000","last_clean":"2024-02-13T00:50:13.538400+0000","last_became_active":"2024-02-13T00:45:15.444818+0000","last_became_peered":"2024-02-13T00:45:15.444818+0000","last_unstale":"2024-02-13T00:50:13.538400+0000","last_undegraded":"2024-02-13T00:50:13.538400+0000","last_fullsized":"2024-02-13T00:50:13.538400+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.052640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.052640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":446,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541433+0000","last_change":"2024-02-13T00:46:01.439915+0000","last_active":"2024-02-13T00:50:13.541433+0000","last_peered":"2024-02-13T00:50:13.541433+0000","last_clean":"2024-02-13T00:50:13.541433+0000","last_became_active":"2024-02-13T00:46:01.438745+0000","last_became_peered":"2024-02-13T00:46:01.438745+0000","last_unstale":"2024-02-13T00:50:13.541433+0000","last_undegraded":"2024-02-13T00:50:13.541433+0000","last_fullsized":"2024-02-13T00:50:13.541433+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.104906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.104906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"467'25","reported_seq":358,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541542+0000","last_change":"2024-02-13T00:47:36.368967+0000","last_active":"2024-02-13T00:50:13.541542+0000","last_peered":"2024-02-13T00:50:13.541542+0000","last_clean":"2024-02-13T00:50:13.541542+0000","last_became_active":"2024-02-13T00:46:01.450079+0000","last_became_peered":"2024-02-13T00:46:01.450079+0000","last_unstale":"2024-02-13T00:50:13.541542+0000","last_undegraded":"2024-02-13T00:50:13.541542+0000","last_fullsized":"2024-02-13T00:50:13.541542+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:47:36.368863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.368863+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":304,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542797+0000","last_change":"2024-02-13T00:46:59.388809+0000","last_active":"2024-02-13T00:50:13.542797+0000","last_peered":"2024-02-13T00:50:13.542797+0000","last_clean":"2024-02-13T00:50:13.542797+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:50:13.542797+0000","last_undegraded":"2024-02-13T00:50:13.542797+0000","last_fullsized":"2024-02-13T00:50:13.542797+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:59.388745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:59.388745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":290,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539959+0000","last_change":"2024-02-13T00:45:15.439752+0000","last_active":"2024-02-13T00:50:13.539959+0000","last_peered":"2024-02-13T00:50:13.539959+0000","last_clean":"2024-02-13T00:50:13.539959+0000","last_became_active":"2024-02-13T00:45:15.437943+0000","last_became_peered":"2024-02-13T00:45:15.437943+0000","last_unstale":"2024-02-13T00:50:13.539959+0000","last_undegraded":"2024-02-13T00:50:13.539959+0000","last_fullsized":"2024-02-13T00:50:13.539959+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.284740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.284740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":498,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541725+0000","last_change":"2024-02-13T00:44:58.969806+0000","last_active":"2024-02-13T00:50:13.541725+0000","last_peered":"2024-02-13T00:50:13.541725+0000","last_clean":"2024-02-13T00:50:13.541725+0000","last_became_active":"2024-02-13T00:44:50.439512+0000","last_became_peered":"2024-02-13T00:44:50.439512+0000","last_unstale":"2024-02-13T00:50:13.541725+0000","last_undegraded":"2024-02-13T00:50:13.541725+0000","last_fullsized":"2024-02-13T00:50:13.541725+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:58.969692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:44:58.969692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":418,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541724+0000","last_change":"2024-02-13T00:49:35.101413+0000","last_active":"2024-02-13T00:50:13.541724+0000","last_peered":"2024-02-13T00:50:13.541724+0000","last_clean":"2024-02-13T00:50:13.541724+0000","last_became_active":"2024-02-13T00:45:15.430749+0000","last_became_peered":"2024-02-13T00:45:15.430749+0000","last_unstale":"2024-02-13T00:50:13.541724+0000","last_undegraded":"2024-02-13T00:50:13.541724+0000","last_fullsized":"2024-02-13T00:50:13.541724+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:35.101298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.101298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":687,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543096+0000","last_change":"2024-02-13T00:48:53.841888+0000","last_active":"2024-02-13T00:50:13.543096+0000","last_peered":"2024-02-13T00:50:13.543096+0000","last_clean":"2024-02-13T00:50:13.543096+0000","last_became_active":"2024-02-13T00:46:01.443721+0000","last_became_peered":"2024-02-13T00:46:01.443721+0000","last_unstale":"2024-02-13T00:50:13.543096+0000","last_undegraded":"2024-02-13T00:50:13.543096+0000","last_fullsized":"2024-02-13T00:50:13.543096+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'32","last_scrub_stamp":"2024-02-13T00:48:53.841797+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.841797+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":532,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539575+0000","last_change":"2024-02-13T00:46:32.298033+0000","last_active":"2024-02-13T00:50:13.539575+0000","last_peered":"2024-02-13T00:50:13.539575+0000","last_clean":"2024-02-13T00:50:13.539575+0000","last_became_active":"2024-02-13T00:46:01.442231+0000","last_became_peered":"2024-02-13T00:46:01.442231+0000","last_unstale":"2024-02-13T00:50:13.539575+0000","last_undegraded":"2024-02-13T00:50:13.539575+0000","last_fullsized":"2024-02-13T00:50:13.539575+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.297918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.297918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":396,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540890+0000","last_change":"2024-02-13T00:48:33.460377+0000","last_active":"2024-02-13T00:50:13.540890+0000","last_peered":"2024-02-13T00:50:13.540890+0000","last_clean":"2024-02-13T00:50:13.540890+0000","last_became_active":"2024-02-13T00:46:01.439984+0000","last_became_peered":"2024-02-13T00:46:01.439984+0000","last_unstale":"2024-02-13T00:50:13.540890+0000","last_undegraded":"2024-02-13T00:50:13.540890+0000","last_fullsized":"2024-02-13T00:50:13.540890+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:48:33.460247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:33.460247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":531,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541110+0000","last_change":"2024-02-13T00:45:38.084426+0000","last_active":"2024-02-13T00:50:13.541110+0000","last_peered":"2024-02-13T00:50:13.541110+0000","last_clean":"2024-02-13T00:50:13.541110+0000","last_became_active":"2024-02-13T00:45:15.435550+0000","last_became_peered":"2024-02-13T00:45:15.435550+0000","last_unstale":"2024-02-13T00:50:13.541110+0000","last_undegraded":"2024-02-13T00:50:13.541110+0000","last_fullsized":"2024-02-13T00:50:13.541110+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.084358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.084358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":335,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543067+0000","last_change":"2024-02-13T00:48:55.424021+0000","last_active":"2024-02-13T00:50:13.543067+0000","last_peered":"2024-02-13T00:50:13.543067+0000","last_clean":"2024-02-13T00:50:13.543067+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:50:13.543067+0000","last_undegraded":"2024-02-13T00:50:13.543067+0000","last_fullsized":"2024-02-13T00:50:13.543067+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:55.423952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.423952+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":304,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541632+0000","last_change":"2024-02-13T00:46:56.339969+0000","last_active":"2024-02-13T00:50:13.541632+0000","last_peered":"2024-02-13T00:50:13.541632+0000","last_clean":"2024-02-13T00:50:13.541632+0000","last_became_active":"2024-02-13T00:45:15.436239+0000","last_became_peered":"2024-02-13T00:45:15.436239+0000","last_unstale":"2024-02-13T00:50:13.541632+0000","last_undegraded":"2024-02-13T00:50:13.541632+0000","last_fullsized":"2024-02-13T00:50:13.541632+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:56.339885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.339885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":572,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539552+0000","last_change":"2024-02-13T00:49:46.928723+0000","last_active":"2024-02-13T00:50:13.539552+0000","last_peered":"2024-02-13T00:50:13.539552+0000","last_clean":"2024-02-13T00:50:13.539552+0000","last_became_active":"2024-02-13T00:46:01.453269+0000","last_became_peered":"2024-02-13T00:46:01.453269+0000","last_unstale":"2024-02-13T00:50:13.539552+0000","last_undegraded":"2024-02-13T00:50:13.539552+0000","last_fullsized":"2024-02-13T00:50:13.539552+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.928651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.928651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":295,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543235+0000","last_change":"2024-02-13T00:45:37.280947+0000","last_active":"2024-02-13T00:50:13.543235+0000","last_peered":"2024-02-13T00:50:13.543235+0000","last_clean":"2024-02-13T00:50:13.543235+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:50:13.543235+0000","last_undegraded":"2024-02-13T00:50:13.543235+0000","last_fullsized":"2024-02-13T00:50:13.543235+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.280865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.280865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":351,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540429+0000","last_change":"2024-02-13T00:49:30.086932+0000","last_active":"2024-02-13T00:50:13.540429+0000","last_peered":"2024-02-13T00:50:13.540429+0000","last_clean":"2024-02-13T00:50:13.540429+0000","last_became_active":"2024-02-13T00:44:50.442894+0000","last_became_peered":"2024-02-13T00:44:50.442894+0000","last_unstale":"2024-02-13T00:50:13.540429+0000","last_undegraded":"2024-02-13T00:50:13.540429+0000","last_fullsized":"2024-02-13T00:50:13.540429+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:30.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:30.086856+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":295,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541908+0000","last_change":"2024-02-13T00:49:04.282594+0000","last_active":"2024-02-13T00:50:13.541908+0000","last_peered":"2024-02-13T00:50:13.541908+0000","last_clean":"2024-02-13T00:50:13.541908+0000","last_became_active":"2024-02-13T00:45:15.445403+0000","last_became_peered":"2024-02-13T00:45:15.445403+0000","last_unstale":"2024-02-13T00:50:13.541908+0000","last_undegraded":"2024-02-13T00:50:13.541908+0000","last_fullsized":"2024-02-13T00:50:13.541908+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:04.282529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:04.282529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":295,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541989+0000","last_change":"2024-02-13T00:47:42.329463+0000","last_active":"2024-02-13T00:50:13.541989+0000","last_peered":"2024-02-13T00:50:13.541989+0000","last_clean":"2024-02-13T00:50:13.541989+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:50:13.541989+0000","last_undegraded":"2024-02-13T00:50:13.541989+0000","last_fullsized":"2024-02-13T00:50:13.541989+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:42.329367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:42.329367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":204,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542841+0000","last_change":"2024-02-13T00:46:01.451610+0000","last_active":"2024-02-13T00:50:13.542841+0000","last_peered":"2024-02-13T00:50:13.542841+0000","last_clean":"2024-02-13T00:50:13.542841+0000","last_became_active":"2024-02-13T00:46:01.449873+0000","last_became_peered":"2024-02-13T00:46:01.449873+0000","last_unstale":"2024-02-13T00:50:13.542841+0000","last_undegraded":"2024-02-13T00:50:13.542841+0000","last_fullsized":"2024-02-13T00:50:13.542841+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.554065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.554065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":452,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.539951+0000","last_change":"2024-02-13T00:46:56.973071+0000","last_active":"2024-02-13T00:50:13.539951+0000","last_peered":"2024-02-13T00:50:13.539951+0000","last_clean":"2024-02-13T00:50:13.539951+0000","last_became_active":"2024-02-13T00:44:39.504670+0000","last_became_peered":"2024-02-13T00:44:39.504670+0000","last_unstale":"2024-02-13T00:50:13.539951+0000","last_undegraded":"2024-02-13T00:50:13.539951+0000","last_fullsized":"2024-02-13T00:50:13.539951+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:46:56.973000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.973000+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":299,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541446+0000","last_change":"2024-02-13T00:46:01.448134+0000","last_active":"2024-02-13T00:50:13.541446+0000","last_peered":"2024-02-13T00:50:13.541446+0000","last_clean":"2024-02-13T00:50:13.541446+0000","last_became_active":"2024-02-13T00:46:01.446635+0000","last_became_peered":"2024-02-13T00:46:01.446635+0000","last_unstale":"2024-02-13T00:50:13.541446+0000","last_undegraded":"2024-02-13T00:50:13.541446+0000","last_fullsized":"2024-02-13T00:50:13.541446+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:28.101254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:28.101254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":295,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542783+0000","last_change":"2024-02-13T00:46:01.461188+0000","last_active":"2024-02-13T00:50:13.542783+0000","last_peered":"2024-02-13T00:50:13.542783+0000","last_clean":"2024-02-13T00:50:13.542783+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:50:13.542783+0000","last_undegraded":"2024-02-13T00:50:13.542783+0000","last_fullsized":"2024-02-13T00:50:13.542783+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.238152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.238152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":539,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541673+0000","last_change":"2024-02-13T00:46:01.443734+0000","last_active":"2024-02-13T00:50:13.541673+0000","last_peered":"2024-02-13T00:50:13.541673+0000","last_clean":"2024-02-13T00:50:13.541673+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:50:13.541673+0000","last_undegraded":"2024-02-13T00:50:13.541673+0000","last_fullsized":"2024-02-13T00:50:13.541673+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.621170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.621170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":444,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541670+0000","last_change":"2024-02-13T00:46:27.706096+0000","last_active":"2024-02-13T00:50:13.541670+0000","last_peered":"2024-02-13T00:50:13.541670+0000","last_clean":"2024-02-13T00:50:13.541670+0000","last_became_active":"2024-02-13T00:44:50.415894+0000","last_became_peered":"2024-02-13T00:44:50.415894+0000","last_unstale":"2024-02-13T00:50:13.541670+0000","last_undegraded":"2024-02-13T00:50:13.541670+0000","last_fullsized":"2024-02-13T00:50:13.541670+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:46:27.706006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:27.706006+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":379,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541136+0000","last_change":"2024-02-13T00:49:43.067713+0000","last_active":"2024-02-13T00:50:13.541136+0000","last_peered":"2024-02-13T00:50:13.541136+0000","last_clean":"2024-02-13T00:50:13.541136+0000","last_became_active":"2024-02-13T00:46:01.451790+0000","last_became_peered":"2024-02-13T00:46:01.451790+0000","last_unstale":"2024-02-13T00:50:13.541136+0000","last_undegraded":"2024-02-13T00:50:13.541136+0000","last_fullsized":"2024-02-13T00:50:13.541136+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:43.067615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:43.067615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":567,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540705+0000","last_change":"2024-02-13T00:47:36.885611+0000","last_active":"2024-02-13T00:50:13.540705+0000","last_peered":"2024-02-13T00:50:13.540705+0000","last_clean":"2024-02-13T00:50:13.540705+0000","last_became_active":"2024-02-13T00:45:15.419668+0000","last_became_peered":"2024-02-13T00:45:15.419668+0000","last_unstale":"2024-02-13T00:50:13.540705+0000","last_undegraded":"2024-02-13T00:50:13.540705+0000","last_fullsized":"2024-02-13T00:50:13.540705+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.885510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.885510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":473,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542058+0000","last_change":"2024-02-13T00:46:01.457482+0000","last_active":"2024-02-13T00:50:13.542058+0000","last_peered":"2024-02-13T00:50:13.542058+0000","last_clean":"2024-02-13T00:50:13.542058+0000","last_became_active":"2024-02-13T00:46:01.455940+0000","last_became_peered":"2024-02-13T00:46:01.455940+0000","last_unstale":"2024-02-13T00:50:13.542058+0000","last_undegraded":"2024-02-13T00:50:13.542058+0000","last_fullsized":"2024-02-13T00:50:13.542058+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:21.087199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:21.087199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":460,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538972+0000","last_change":"2024-02-13T00:46:36.089483+0000","last_active":"2024-02-13T00:50:13.538972+0000","last_peered":"2024-02-13T00:50:13.538972+0000","last_clean":"2024-02-13T00:50:13.538972+0000","last_became_active":"2024-02-13T00:45:15.419364+0000","last_became_peered":"2024-02-13T00:45:15.419364+0000","last_unstale":"2024-02-13T00:50:13.538972+0000","last_undegraded":"2024-02-13T00:50:13.538972+0000","last_fullsized":"2024-02-13T00:50:13.538972+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:46:36.089402+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:46:36.089402+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":304,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543068+0000","last_change":"2024-02-13T00:47:47.284092+0000","last_active":"2024-02-13T00:50:13.543068+0000","last_peered":"2024-02-13T00:50:13.543068+0000","last_clean":"2024-02-13T00:50:13.543068+0000","last_became_active":"2024-02-13T00:46:01.451997+0000","last_became_peered":"2024-02-13T00:46:01.451997+0000","last_unstale":"2024-02-13T00:50:13.543068+0000","last_undegraded":"2024-02-13T00:50:13.543068+0000","last_fullsized":"2024-02-13T00:50:13.543068+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.284014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.284014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":517,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538540+0000","last_change":"2024-02-13T00:49:01.133798+0000","last_active":"2024-02-13T00:50:13.538540+0000","last_peered":"2024-02-13T00:50:13.538540+0000","last_clean":"2024-02-13T00:50:13.538540+0000","last_became_active":"2024-02-13T00:46:01.445141+0000","last_became_peered":"2024-02-13T00:46:01.445141+0000","last_unstale":"2024-02-13T00:50:13.538540+0000","last_undegraded":"2024-02-13T00:50:13.538540+0000","last_fullsized":"2024-02-13T00:50:13.538540+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:01.133709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:49:01.133709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":286,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540200+0000","last_change":"2024-02-13T00:45:30.186208+0000","last_active":"2024-02-13T00:50:13.540200+0000","last_peered":"2024-02-13T00:50:13.540200+0000","last_clean":"2024-02-13T00:50:13.540200+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:50:13.540200+0000","last_undegraded":"2024-02-13T00:50:13.540200+0000","last_fullsized":"2024-02-13T00:50:13.540200+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:30.186129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:30.186129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":344,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.540264+0000","last_change":"2024-02-13T00:46:06.251713+0000","last_active":"2024-02-13T00:50:13.540264+0000","last_peered":"2024-02-13T00:50:13.540264+0000","last_clean":"2024-02-13T00:50:13.540264+0000","last_became_active":"2024-02-13T00:46:01.442464+0000","last_became_peered":"2024-02-13T00:46:01.442464+0000","last_unstale":"2024-02-13T00:50:13.540264+0000","last_undegraded":"2024-02-13T00:50:13.540264+0000","last_fullsized":"2024-02-13T00:50:13.540264+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:43:44.241358+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:43:44.241358+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":380,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.542980+0000","last_change":"2024-02-13T00:47:47.711551+0000","last_active":"2024-02-13T00:50:13.542980+0000","last_peered":"2024-02-13T00:50:13.542980+0000","last_clean":"2024-02-13T00:50:13.542980+0000","last_became_active":"2024-02-13T00:46:01.447654+0000","last_became_peered":"2024-02-13T00:46:01.447654+0000","last_unstale":"2024-02-13T00:50:13.542980+0000","last_undegraded":"2024-02-13T00:50:13.542980+0000","last_fullsized":"2024-02-13T00:50:13.542980+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.711486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.711486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":216,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538484+0000","last_change":"2024-02-13T00:47:53.826039+0000","last_active":"2024-02-13T00:50:13.538484+0000","last_peered":"2024-02-13T00:50:13.538484+0000","last_clean":"2024-02-13T00:50:13.538484+0000","last_became_active":"2024-02-13T00:46:01.448971+0000","last_became_peered":"2024-02-13T00:46:01.448971+0000","last_unstale":"2024-02-13T00:50:13.538484+0000","last_undegraded":"2024-02-13T00:50:13.538484+0000","last_fullsized":"2024-02-13T00:50:13.538484+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:53.825971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:53.825971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":543,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.538727+0000","last_change":"2024-02-13T00:45:32.233265+0000","last_active":"2024-02-13T00:50:13.538727+0000","last_peered":"2024-02-13T00:50:13.538727+0000","last_clean":"2024-02-13T00:50:13.538727+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:50:13.538727+0000","last_undegraded":"2024-02-13T00:50:13.538727+0000","last_fullsized":"2024-02-13T00:50:13.538727+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:45:32.233197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.233197+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":479,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541384+0000","last_change":"2024-02-13T00:47:44.757362+0000","last_active":"2024-02-13T00:50:13.541384+0000","last_peered":"2024-02-13T00:50:13.541384+0000","last_clean":"2024-02-13T00:50:13.541384+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:50:13.541384+0000","last_undegraded":"2024-02-13T00:50:13.541384+0000","last_fullsized":"2024-02-13T00:50:13.541384+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:47:44.757262+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:47:44.757262+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":571,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.543132+0000","last_change":"2024-02-13T00:48:54.818104+0000","last_active":"2024-02-13T00:50:13.543132+0000","last_peered":"2024-02-13T00:50:13.543132+0000","last_clean":"2024-02-13T00:50:13.543132+0000","last_became_active":"2024-02-13T00:45:15.434224+0000","last_became_peered":"2024-02-13T00:45:15.434224+0000","last_unstale":"2024-02-13T00:50:13.543132+0000","last_undegraded":"2024-02-13T00:50:13.543132+0000","last_fullsized":"2024-02-13T00:50:13.543132+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:54.817999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.817999+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":290,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-02-13T00:50:13.541518+0000","last_change":"2024-02-13T00:45:15.435955+0000","last_active":"2024-02-13T00:50:13.541518+0000","last_peered":"2024-02-13T00:50:13.541518+0000","last_clean":"2024-02-13T00:50:13.541518+0000","last_became_active":"2024-02-13T00:45:15.435259+0000","last_became_peered":"2024-02-13T00:45:15.435259+0000","last_unstale":"2024-02-13T00:50:13.541518+0000","last_undegraded":"2024-02-13T00:50:13.541518+0000","last_fullsized":"2024-02-13T00:50:13.541518+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.953989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.953989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":235,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":3},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1492,"num_read_kb":1217,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":9606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":11656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":970,"ondisk_log_size":970,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":741,"num_read_kb":736,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":227633266841,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":45,"seq":193273528479,"num_pgs":128,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308396,"kb_used_data":2980,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436756,"statfs":{"total":95995035648,"available":95679238144,"internally_reserved":0,"allocated":3051520,"data_stored":1435007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.46700000000000003,"15min":0.54600000000000004},"min":{"1min":0.316,"5min":0.25600000000000001,"15min":0.19700000000000001},"max":{"1min":0.65600000000000003,"5min":0.77500000000000002,"15min":8.1340000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52100000000000002,"15min":0.60999999999999999},"min":{"1min":0.34100000000000003,"5min":0.23100000000000001,"15min":0.221},"max":{"1min":0.72799999999999998,"5min":4.2910000000000004,"15min":11.367000000000001},"last":0.34799999999999998}]},{"osd":1,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51200000000000001,"15min":0.88400000000000001},"min":{"1min":0.34899999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.63200000000000001,"5min":4.5720000000000001,"15min":54.002000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52300000000000002,"15min":0.68100000000000005},"min":{"1min":0.28999999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66800000000000004,"5min":5.4989999999999997,"15min":12.403},"last":0.50600000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52300000000000002,"15min":0.71099999999999997},"min":{"1min":0.31900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":2.8399999999999999,"15min":12.477},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50700000000000001,"15min":0.68000000000000005},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.63200000000000001,"5min":2.1659999999999999,"15min":11.702},"last":0.44900000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.61499999999999999,"15min":0.68100000000000005},"min":{"1min":0.35299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.65700000000000003,"5min":5.931,"15min":8.0519999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54400000000000004,"15min":0.77200000000000002},"min":{"1min":0.44,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":0.65400000000000003,"5min":2.8719999999999999,"15min":19.081},"last":0.61699999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.41499999999999998},"min":{"1min":0.29099999999999998,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":0.51400000000000001,"5min":0.71899999999999997,"15min":1.0649999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.441,"15min":0.438},"min":{"1min":0.315,"5min":0.22800000000000001,"15min":0.221},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.57199999999999995}]},{"osd":5,"last update":"Tue Feb 13 00:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66400000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.497},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69899999999999995,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.59099999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45600000000000002,"15min":0.48099999999999998},"min":{"1min":0.36699999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60899999999999999,"5min":0.66800000000000004,"15min":1.173},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.50900000000000001,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.28299999999999997,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.80700000000000005,"15min":1.262},"last":0.60399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":131,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304332,"kb_used_data":3012,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440820,"statfs":{"total":95995035648,"available":95683399680,"internally_reserved":0,"allocated":3084288,"data_stored":1443869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34399999999999997,"15min":0.95099999999999996},"min":{"1min":0.20699999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.52700000000000002,"5min":1.1080000000000001,"15min":61.856000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33600000000000002,"15min":0.66800000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.16500000000000001},"max":{"1min":0.70199999999999996,"5min":1.052,"15min":61.908999999999999},"last":0.307}]},{"osd":2,"last update":"Tue Feb 13 00:50:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33700000000000002,"15min":0.84599999999999997},"min":{"1min":0.27300000000000002,"5min":0.22900000000000001,"15min":0.20599999999999999},"max":{"1min":0.53000000000000003,"5min":0.80000000000000004,"15min":32.161000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.82699999999999996},"min":{"1min":0.253,"5min":0.187,"15min":0.187},"max":{"1min":0.504,"5min":1.21,"15min":29.693000000000001},"last":0.34999999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.376,"15min":1.2669999999999999},"min":{"1min":0.28100000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.55400000000000005,"5min":1.0900000000000001,"15min":131.23099999999999},"last":0.497},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.39700000000000002,"15min":0.86099999999999999},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.48099999999999998,"5min":1.214,"15min":49.055},"last":0.45200000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.75800000000000001},"min":{"1min":0.36199999999999999,"5min":0.33500000000000002,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":1.246,"15min":16.547000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.56399999999999995,"15min":0.71199999999999997},"min":{"1min":0.42499999999999999,"5min":0.34100000000000003,"15min":0.24199999999999999},"max":{"1min":0.85199999999999998,"5min":0.89200000000000002,"15min":14.731999999999999},"last":0.46300000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.091},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59899999999999998,"15min":0.58699999999999997},"min":{"1min":0.437,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":1.127}]},{"osd":6,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54700000000000004,"15min":0.71799999999999997},"min":{"1min":0.41299999999999998,"5min":0.29699999999999999,"15min":0.29099999999999998},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":14.772},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.54600000000000004,"15min":0.92500000000000004},"min":{"1min":0.40500000000000003,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":26.600000000000001},"last":1.2709999999999999}]},{"osd":7,"last update":"Tue Feb 13 00:50:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.56299999999999994,"15min":0.72299999999999998},"min":{"1min":0.41499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":14.794},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55900000000000005,"15min":0.92500000000000004},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.32200000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":26.693999999999999},"last":1.246}]}]},{"osd":0,"up_from":8,"seq":34359738556,"num_pgs":124,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304308,"kb_used_data":2988,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440844,"statfs":{"total":95995035648,"available":95683424256,"internally_reserved":0,"allocated":3059712,"data_stored":1443478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.378,"15min":0.621},"min":{"1min":0.25,"5min":0.191,"15min":0.189},"max":{"1min":0.59999999999999998,"5min":2.0379999999999998,"15min":30.184000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.17599999999999999},"max":{"1min":0.52300000000000002,"5min":5.6959999999999997,"15min":5.6959999999999997},"last":0.31}]},{"osd":2,"last update":"Tue Feb 13 00:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40300000000000002,"15min":0.498},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.20999999999999999},"max":{"1min":0.64300000000000002,"5min":1.3700000000000001,"15min":7.0839999999999996},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42499999999999999,"15min":0.66900000000000004},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.185},"max":{"1min":0.53600000000000003,"5min":3.6949999999999998,"15min":24.459},"last":0.33300000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.47499999999999998,"15min":0.48599999999999999},"min":{"1min":0.28699999999999998,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":0.628,"5min":4.8710000000000004,"15min":8.718},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.28299999999999997,"5min":0.26900000000000002,"15min":0.222},"max":{"1min":0.67200000000000004,"5min":7.1139999999999999,"15min":7.1139999999999999},"last":0.35399999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.63300000000000001,"15min":0.79900000000000004},"min":{"1min":0.372,"5min":0.29899999999999999,"15min":0.24399999999999999},"max":{"1min":0.874,"5min":7.056,"15min":24.381},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.40500000000000003,"5min":0.31,"15min":0.28699999999999998},"max":{"1min":0.88200000000000001,"5min":6.1859999999999999,"15min":8.6950000000000003},"last":0.56499999999999995}]},{"osd":5,"last update":"Tue Feb 13 00:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.5,"15min":0.495},"min":{"1min":0.40799999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.45400000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.35999999999999999,"5min":0.34100000000000003,"15min":0.31},"max":{"1min":0.79500000000000004,"5min":3.302,"15min":8.6129999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.70999999999999996,"15min":0.65100000000000002},"min":{"1min":0.34000000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.89200000000000002,"5min":14.263,"15min":14.263},"last":0.47899999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.44700000000000001,"5min":0.35999999999999999,"15min":0.317},"max":{"1min":0.78400000000000003,"5min":7.7720000000000002,"15min":8.6679999999999993},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58299999999999996,"15min":0.66900000000000004},"min":{"1min":0.44,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91700000000000004,"5min":1.6910000000000001,"15min":8.1609999999999996},"last":0.46000000000000002}]}]},{"osd":2,"up_from":0,"seq":77309411507,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182573,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986345,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300188,"kb_used_data":2964,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444964,"statfs":{"total":95995035648,"available":95687643136,"internally_reserved":0,"allocated":3035136,"data_stored":1434475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.59399999999999997},"min":{"1min":0.34200000000000003,"5min":0.248,"15min":0.21199999999999999},"max":{"1min":0.71999999999999997,"5min":5.3319999999999999,"15min":9.3399999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.63500000000000001,"15min":0.60999999999999999},"min":{"1min":0.34499999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.65800000000000003,"5min":11.532,"15min":13.833},"last":0.65700000000000003}]},{"osd":1,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.59799999999999998,"15min":0.753},"min":{"1min":0.373,"5min":0.25,"15min":0.25},"max":{"1min":0.76900000000000002,"5min":7.5410000000000004,"15min":53.188000000000002},"last":0.63},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.60899999999999999,"15min":0.69999999999999996},"min":{"1min":0.32300000000000001,"5min":0.28999999999999998,"15min":0.25600000000000001},"max":{"1min":0.71099999999999997,"5min":11.497999999999999,"15min":44.793999999999997},"last":0.50700000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58499999999999996,"15min":0.59699999999999998},"min":{"1min":0.37,"5min":0.31900000000000001,"15min":0.26900000000000002},"max":{"1min":0.78900000000000003,"5min":4.2199999999999998,"15min":8.6829999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.59299999999999997,"15min":0.629},"min":{"1min":0.38300000000000001,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.74299999999999999,"5min":4.0730000000000004,"15min":18.911000000000001},"last":0.53300000000000003}]},{"osd":3,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53800000000000003,"15min":0.65400000000000003},"min":{"1min":0.34599999999999997,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64000000000000001,"5min":4.0090000000000003,"15min":23.5},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55100000000000005,"15min":0.71299999999999997},"min":{"1min":0.39400000000000002,"5min":0.32100000000000001,"15min":0.30599999999999999},"max":{"1min":0.68899999999999995,"5min":3.9729999999999999,"15min":36.152999999999999},"last":0.46300000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:50:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.33700000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.69399999999999995,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.38700000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.46000000000000002,"15min":0.436},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.25700000000000001},"max":{"1min":0.60999999999999999,"5min":3.153,"15min":3.153},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.497,"15min":0.46300000000000002},"min":{"1min":0.32700000000000001,"5min":0.25800000000000001,"15min":0.254},"max":{"1min":0.65900000000000003,"5min":4.9390000000000001,"15min":4.9390000000000001},"last":0.63900000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.49299999999999999,"15min":0.48099999999999998},"min":{"1min":0.34000000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.76500000000000001,"5min":1.575,"15min":1.575},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.5,"15min":0.47699999999999998},"min":{"1min":0.309,"5min":0.29699999999999999,"15min":0.28100000000000003},"max":{"1min":0.72499999999999998,"5min":2.46,"15min":2.46},"last":0.41299999999999998}]}]},{"osd":5,"up_from":0,"seq":2057289334811,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":3328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":235,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":235,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":235,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:50:18.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:18.565 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:18.584 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:18.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:18.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:18.716 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:18.735 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:18.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:18.789 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:18.838 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:18.865 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:50:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:50:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:50:18.866 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:18.886 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:18.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:18.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:19.016 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:19.035 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:19.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:19.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:19.167 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:19.290 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:19.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:19.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:19.393 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:19.421 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:19.770 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:19.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:18.488448+0000 mgr.y (mgr.14544) 515 : cluster [DBG] pgmap v905: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:19.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:18.515829+0000 mon.a (mon.0) 3097 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:50:19.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:18.515889+0000 mon.a (mon.0) 3098 : cluster [DBG] osdmap e604: 8 total, 8 up, 4 in 2024-02-13T00:50:19.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:18.517095+0000 mon.a (mon.0) 3099 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-102"}]: dispatch 2024-02-13T00:50:19.773 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:18.488448+0000 mgr.y (mgr.14544) 515 : cluster [DBG] pgmap v905: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:19.773 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:18.515829+0000 mon.a (mon.0) 3097 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:50:19.773 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:18.515889+0000 mon.a (mon.0) 3098 : cluster [DBG] osdmap e604: 8 total, 8 up, 4 in 2024-02-13T00:50:19.773 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:18.517095+0000 mon.a (mon.0) 3099 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-102"}]: dispatch 2024-02-13T00:50:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:19.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:18.488448+0000 mgr.y (mgr.14544) 515 : cluster [DBG] pgmap v905: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:18.515829+0000 mon.a (mon.0) 3097 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:50:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:18.515889+0000 mon.a (mon.0) 3098 : cluster [DBG] osdmap e604: 8 total, 8 up, 4 in 2024-02-13T00:50:19.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:18.517095+0000 mon.a (mon.0) 3099 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-102"}]: dispatch 2024-02-13T00:50:19.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:19.903 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:19.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:50:20.251 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:20.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:20.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:20.387 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:20.408 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:20.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:20.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:20.542 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:20.562 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:20.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:20.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:20.694 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:19.517100+0000 mon.a (mon.0) 3100 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-102"}]': finished 2024-02-13T00:50:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:19.517167+0000 mon.a (mon.0) 3101 : cluster [DBG] osdmap e605: 8 total, 8 up, 4 in 2024-02-13T00:50:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:19.731314+0000 osd.1 (osd.1) 255 : cluster [DBG] 6.d scrub starts 2024-02-13T00:50:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:19.734313+0000 osd.1 (osd.1) 256 : cluster [DBG] 6.d scrub ok 2024-02-13T00:50:20.862 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:19.517100+0000 mon.a (mon.0) 3100 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-102"}]': finished 2024-02-13T00:50:20.862 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:19.517167+0000 mon.a (mon.0) 3101 : cluster [DBG] osdmap e605: 8 total, 8 up, 4 in 2024-02-13T00:50:20.862 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:19.731314+0000 osd.1 (osd.1) 255 : cluster [DBG] 6.d scrub starts 2024-02-13T00:50:20.862 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:19.734313+0000 osd.1 (osd.1) 256 : cluster [DBG] 6.d scrub ok 2024-02-13T00:50:20.863 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:19.517100+0000 mon.a (mon.0) 3100 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-102"}]': finished 2024-02-13T00:50:20.863 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:19.517167+0000 mon.a (mon.0) 3101 : cluster [DBG] osdmap e605: 8 total, 8 up, 4 in 2024-02-13T00:50:20.863 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:19.731314+0000 osd.1 (osd.1) 255 : cluster [DBG] 6.d scrub starts 2024-02-13T00:50:20.863 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:19.734313+0000 osd.1 (osd.1) 256 : cluster [DBG] 6.d scrub ok 2024-02-13T00:50:21.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:50:21.644 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:21.646 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:21.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:21.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:21.652 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:50:21.652 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:50:21.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:21.785 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:21.806 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:21.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:20.489557+0000 mgr.y (mgr.14544) 516 : cluster [DBG] pgmap v908: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-13T00:50:21.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:20.519802+0000 mon.a (mon.0) 3102 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:50:21.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:20.589328+0000 mon.a (mon.0) 3103 : cluster [DBG] osdmap e606: 8 total, 8 up, 4 in 2024-02-13T00:50:21.808 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:20.845174+0000 mon.a (mon.0) 3104 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:21.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:21.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:21.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:20.489557+0000 mgr.y (mgr.14544) 516 : cluster [DBG] pgmap v908: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-13T00:50:21.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:20.519802+0000 mon.a (mon.0) 3102 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:50:21.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:20.589328+0000 mon.a (mon.0) 3103 : cluster [DBG] osdmap e606: 8 total, 8 up, 4 in 2024-02-13T00:50:21.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:20.845174+0000 mon.a (mon.0) 3104 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:20.489557+0000 mgr.y (mgr.14544) 516 : cluster [DBG] pgmap v908: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-13T00:50:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:20.519802+0000 mon.a (mon.0) 3102 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:50:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:20.589328+0000 mon.a (mon.0) 3103 : cluster [DBG] osdmap e606: 8 total, 8 up, 4 in 2024-02-13T00:50:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:20.845174+0000 mon.a (mon.0) 3104 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:21.910 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:21.940 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:22.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:22.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:22.623 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:22.642 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:22.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:22.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:22.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:22.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:21.560483+0000 mon.a (mon.0) 3105 : cluster [DBG] osdmap e607: 8 total, 8 up, 4 in 2024-02-13T00:50:22.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:21.565392+0000 mon.a (mon.0) 3106 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:50:22.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:21.876912+0000 osd.0 (osd.0) 229 : cluster [DBG] 28.e scrub starts 2024-02-13T00:50:22.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:21.879989+0000 osd.0 (osd.0) 230 : cluster [DBG] 28.e scrub ok 2024-02-13T00:50:22.772 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:21.560483+0000 mon.a (mon.0) 3105 : cluster [DBG] osdmap e607: 8 total, 8 up, 4 in 2024-02-13T00:50:22.772 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:21.565392+0000 mon.a (mon.0) 3106 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:50:22.772 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:21.876912+0000 osd.0 (osd.0) 229 : cluster [DBG] 28.e scrub starts 2024-02-13T00:50:22.772 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:21.879989+0000 osd.0 (osd.0) 230 : cluster [DBG] 28.e scrub ok 2024-02-13T00:50:22.773 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:22.811 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:22.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:22.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:22.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:22.942 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:21.560483+0000 mon.a (mon.0) 3105 : cluster [DBG] osdmap e607: 8 total, 8 up, 4 in 2024-02-13T00:50:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:21.565392+0000 mon.a (mon.0) 3106 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:50:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:21.876912+0000 osd.0 (osd.0) 229 : cluster [DBG] 28.e scrub starts 2024-02-13T00:50:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:21.879989+0000 osd.0 (osd.0) 230 : cluster [DBG] 28.e scrub ok 2024-02-13T00:50:23.140 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:23.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:23.246 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:23.277 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:23.298 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:23.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:23.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.335+0000 7fc978e10700 1 -- 172.21.15.162:0/807998308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9740fdc50 msgr2=0x7fc9740fe030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.335+0000 7fc978e10700 1 --2- 172.21.15.162:0/807998308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9740fdc50 0x7fc9740fe030 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 rx=0x7fc9680047b0 tx=0x7fc968009cb0).stop 2024-02-13T00:50:23.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.335+0000 7fc978e10700 1 -- 172.21.15.162:0/807998308 shutdown_connections 2024-02-13T00:50:23.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.335+0000 7fc978e10700 1 --2- 172.21.15.162:0/807998308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9740f6840 0x7fc974104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.335+0000 7fc978e10700 1 --2- 172.21.15.162:0/807998308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9740fdc50 0x7fc9740fe030 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.335+0000 7fc978e10700 1 --2- 172.21.15.162:0/807998308 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9740fe5b0 0x7fc9740f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.335+0000 7fc978e10700 1 -- 172.21.15.162:0/807998308 >> 172.21.15.162:0/807998308 conn(0x7fc9740700d0 msgr2=0x7fc9740f4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:23.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.335+0000 7fc978e10700 1 -- 172.21.15.162:0/807998308 shutdown_connections 2024-02-13T00:50:23.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.335+0000 7fc978e10700 1 -- 172.21.15.162:0/807998308 wait complete. 2024-02-13T00:50:23.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.335+0000 7fc978e10700 1 librados: starting msgr at 2024-02-13T00:50:23.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.335+0000 7fc978e10700 1 librados: starting objecter 2024-02-13T00:50:23.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc978e10700 1 Processor -- start 2024-02-13T00:50:23.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc978e10700 1 -- start start 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc978e10700 1 librados: setting wanted keys 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc978e10700 1 librados: calling monclient init 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc978e10700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9740f6840 0x7fc97410e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc978e10700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9740fdc50 0x7fc97410e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc978e10700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9740fe5b0 0x7fc9741084c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc978e10700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9741089b0 con 0x7fc9740fdc50 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc978e10700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc974108b20 con 0x7fc9740f6840 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc978e10700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc974108ce0 con 0x7fc9740fe5b0 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc971d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9740fdc50 0x7fc97410e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc972d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9740fe5b0 0x7fc9741084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc971d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9740fdc50 0x7fc97410e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45232/0 (socket says 172.21.15.162:45232) 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc971d9b700 1 -- 172.21.15.162:0/2300717534 learned_addr learned my addr 172.21.15.162:0/2300717534 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.336+0000 7fc97259c700 1 --2- 172.21.15.162:0/2300717534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9740f6840 0x7fc97410e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:23.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc971d9b700 1 -- 172.21.15.162:0/2300717534 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9740fe5b0 msgr2=0x7fc9741084c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc971d9b700 1 --2- 172.21.15.162:0/2300717534 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9740fe5b0 0x7fc9741084c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc971d9b700 1 -- 172.21.15.162:0/2300717534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9740f6840 msgr2=0x7fc97410e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc971d9b700 1 --2- 172.21.15.162:0/2300717534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9740f6840 0x7fc97410e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc971d9b700 1 -- 172.21.15.162:0/2300717534 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc964005040 con 0x7fc9740fdc50 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc971d9b700 1 --2- 172.21.15.162:0/2300717534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9740fdc50 0x7fc97410e810 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 rx=0x7fc964009400 tx=0x7fc964009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc9637fe700 10 client.?.objecter ms_handle_connect 0x7fc9740fdc50 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc9637fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964007a40 con 0x7fc9740fdc50 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc968005040 con 0x7fc9740fdc50 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc96400f040 con 0x7fc9740fdc50 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964014a70 con 0x7fc9740fdc50 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc978e10700 10 client.27777.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc974109320 con 0x7fc9740fdc50 2024-02-13T00:50:23.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.337+0000 7fc978e10700 1 librados: init done 2024-02-13T00:50:23.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.338+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc97405f960 con 0x7fc9740fdc50 2024-02-13T00:50:23.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc964010530 con 0x7fc9740fdc50 2024-02-13T00:50:23.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc9637fe700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc95c062c90 0x7fc95c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:23.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(608..608 src has 449..608) v4 ==== 8475+0+0 (secure 0 0 0) 0x7fc96407d410 con 0x7fc9740fdc50 2024-02-13T00:50:23.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 osd_map(608..608 src has 449..608) v4 2024-02-13T00:50:23.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc9637fe700 3 client.27777.objecter handle_osd_map got epochs [608,608] > 0 2024-02-13T00:50:23.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc9637fe700 3 client.27777.objecter handle_osd_map decoding full epoch 608 2024-02-13T00:50:23.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc97259c700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc95c062c90 0x7fc95c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:23.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc9637fe700 20 client.27777.objecter dump_active .. 0 homeless 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v608) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc96407d8b0 con 0x7fc9740fdc50 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc9637fe700 10 client.27777.objecter ms_handle_connect 0x7fc95c062c90 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc97259c700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc95c062c90 0x7fc95c065150 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 rx=0x7fc968005010 tx=0x7fc968009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc978e10700 10 client.27777.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 1 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc974028590 0x7fc9740fea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9740475e0 osd=0 3 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740475e0 osd=0 3 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9740475e0 osd=0 3 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 1 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc9740475e0 osd=0 3 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 0 1 2024-02-13T00:50:23.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740475e0 osd=0 4 2024-02-13T00:50:23.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc978e10700 10 client.27777.objecter _send_command 1 2024-02-13T00:50:23.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc974107170 con 0x7fc974028590 2024-02-13T00:50:23.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.339+0000 7fc972d9d700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc974028590 0x7fc9740fea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:23.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.340+0000 7fc9637fe700 10 client.27777.objecter ms_handle_connect 0x7fc974028590 2024-02-13T00:50:23.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.340+0000 7fc972d9d700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc974028590 0x7fc9740fea30 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:23.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.340+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc974107170 con 0x7fc974028590 2024-02-13T00:50:23.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.340+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:50:23.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.340+0000 7fc9637fe700 10 client.27777.objecter _finish_command 1 = system:0 2024-02-13T00:50:23.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.340+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc9740475e0 osd=0 4 2024-02-13T00:50:23.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.340+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 0 1 2024-02-13T00:50:23.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.355+0000 7fc978e10700 10 client.27777.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:23.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.355+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 2 2024-02-13T00:50:23.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.355+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9740475e0 osd=0 3 2024-02-13T00:50:23.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.355+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740475e0 osd=0 3 2024-02-13T00:50:23.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.355+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9740475e0 osd=0 3 2024-02-13T00:50:23.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.355+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 2 2024-02-13T00:50:23.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.355+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc9740475e0 osd=0 3 2024-02-13T00:50:23.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.355+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 0 2 2024-02-13T00:50:23.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.355+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740475e0 osd=0 4 2024-02-13T00:50:23.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.355+0000 7fc978e10700 10 client.27777.objecter _send_command 2 2024-02-13T00:50:23.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.355+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc97405a0e0 con 0x7fc974028590 2024-02-13T00:50:23.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc97405a0e0 con 0x7fc974028590 2024-02-13T00:50:23.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:23.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc9637fe700 10 client.27777.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc9740475e0 osd=0 4 2024-02-13T00:50:23.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 0 2 2024-02-13T00:50:23.360 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:50:23.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc978e10700 10 client.27777.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:23.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 3 2024-02-13T00:50:23.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6805/150599780 conn(0x7fc974027be0 0x7fc974027fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:23.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9741000a0 osd=1 3 2024-02-13T00:50:23.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9741000a0 osd=1 3 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9741000a0 osd=1 3 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 3 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc9741000a0 osd=1 3 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 1 3 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9741000a0 osd=1 4 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc978e10700 10 client.27777.objecter _send_command 3 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc974100210 con 0x7fc974027be0 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc97259c700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6805/150599780 conn(0x7fc974027be0 0x7fc974027fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc9637fe700 10 client.27777.objecter ms_handle_connect 0x7fc974027be0 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.357+0000 7fc97259c700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6805/150599780 conn(0x7fc974027be0 0x7fc974027fe0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.358+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc974100210 con 0x7fc974027be0 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.358+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.358+0000 7fc9637fe700 10 client.27777.objecter _finish_command 3 = system:0 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.358+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc9741000a0 osd=1 4 2024-02-13T00:50:23.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.358+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 1 3 2024-02-13T00:50:23.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.372+0000 7fc978e10700 10 client.27777.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:23.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.372+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 4 2024-02-13T00:50:23.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.372+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9741000a0 osd=1 3 2024-02-13T00:50:23.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.372+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9741000a0 osd=1 3 2024-02-13T00:50:23.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.372+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9741000a0 osd=1 3 2024-02-13T00:50:23.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.372+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 4 2024-02-13T00:50:23.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.372+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc9741000a0 osd=1 3 2024-02-13T00:50:23.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.372+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 1 4 2024-02-13T00:50:23.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.372+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9741000a0 osd=1 4 2024-02-13T00:50:23.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.372+0000 7fc978e10700 10 client.27777.objecter _send_command 4 2024-02-13T00:50:23.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.372+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc974027650 con 0x7fc974027be0 2024-02-13T00:50:23.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc974027650 con 0x7fc974027be0 2024-02-13T00:50:23.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:23.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc9637fe700 10 client.27777.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc9741000a0 osd=1 4 2024-02-13T00:50:23.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 1 4 2024-02-13T00:50:23.378 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc978e10700 10 client.27777.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 5 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc9740ffc00 0x7fc97411c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc974027a00 osd=2 3 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc974027a00 osd=2 3 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc974027a00 osd=2 3 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 5 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc974027a00 osd=2 3 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 2 5 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc974027a00 osd=2 4 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc978e10700 10 client.27777.objecter _send_command 5 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc97411efb0 con 0x7fc9740ffc00 2024-02-13T00:50:23.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc971d9b700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc9740ffc00 0x7fc97411c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:23.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc9637fe700 10 client.27777.objecter ms_handle_connect 0x7fc9740ffc00 2024-02-13T00:50:23.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.374+0000 7fc971d9b700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc9740ffc00 0x7fc97411c360 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:23.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.375+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc97411efb0 con 0x7fc9740ffc00 2024-02-13T00:50:23.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.375+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:50:23.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.375+0000 7fc9637fe700 10 client.27777.objecter _finish_command 5 = system:0 2024-02-13T00:50:23.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.375+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc974027a00 osd=2 4 2024-02-13T00:50:23.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.375+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 2 5 2024-02-13T00:50:23.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.389+0000 7fc978e10700 10 client.27777.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:23.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.389+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 6 2024-02-13T00:50:23.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.389+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc974027a00 osd=2 3 2024-02-13T00:50:23.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.389+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc974027a00 osd=2 3 2024-02-13T00:50:23.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.389+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc974027a00 osd=2 3 2024-02-13T00:50:23.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.389+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 6 2024-02-13T00:50:23.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.389+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc974027a00 osd=2 3 2024-02-13T00:50:23.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.389+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 2 6 2024-02-13T00:50:23.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.389+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc974027a00 osd=2 4 2024-02-13T00:50:23.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.389+0000 7fc978e10700 10 client.27777.objecter _send_command 6 2024-02-13T00:50:23.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.389+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc9740ff1c0 con 0x7fc9740ffc00 2024-02-13T00:50:23.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.390+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc9740ff1c0 con 0x7fc9740ffc00 2024-02-13T00:50:23.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:23.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc9637fe700 10 client.27777.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc974027a00 osd=2 4 2024-02-13T00:50:23.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 2 6 2024-02-13T00:50:23.395 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:50:23.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc978e10700 10 client.27777.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:23.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 7 2024-02-13T00:50:23.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6813/604971011 conn(0x7fc97411d9d0 0x7fc97411de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:23.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9741199a0 osd=3 3 2024-02-13T00:50:23.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9741199a0 osd=3 3 2024-02-13T00:50:23.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9741199a0 osd=3 3 2024-02-13T00:50:23.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 7 2024-02-13T00:50:23.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc9741199a0 osd=3 3 2024-02-13T00:50:23.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 3 7 2024-02-13T00:50:23.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9741199a0 osd=3 4 2024-02-13T00:50:23.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc978e10700 10 client.27777.objecter _send_command 7 2024-02-13T00:50:23.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc97411a050 con 0x7fc97411d9d0 2024-02-13T00:50:23.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc972d9d700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6813/604971011 conn(0x7fc97411d9d0 0x7fc97411de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:23.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc9637fe700 10 client.27777.objecter ms_handle_connect 0x7fc97411d9d0 2024-02-13T00:50:23.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.391+0000 7fc972d9d700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6813/604971011 conn(0x7fc97411d9d0 0x7fc97411de50 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:23.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.392+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc97411a050 con 0x7fc97411d9d0 2024-02-13T00:50:23.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.392+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:50:23.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.392+0000 7fc9637fe700 10 client.27777.objecter _finish_command 7 = system:0 2024-02-13T00:50:23.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.392+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc9741199a0 osd=3 4 2024-02-13T00:50:23.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.392+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 3 7 2024-02-13T00:50:23.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:23.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.406+0000 7fc978e10700 10 client.27777.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:23.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.406+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 8 2024-02-13T00:50:23.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.406+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9741199a0 osd=3 3 2024-02-13T00:50:23.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.406+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9741199a0 osd=3 3 2024-02-13T00:50:23.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.406+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9741199a0 osd=3 3 2024-02-13T00:50:23.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.406+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 8 2024-02-13T00:50:23.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.406+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc9741199a0 osd=3 3 2024-02-13T00:50:23.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.406+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 3 8 2024-02-13T00:50:23.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.406+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9741199a0 osd=3 4 2024-02-13T00:50:23.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.406+0000 7fc978e10700 10 client.27777.objecter _send_command 8 2024-02-13T00:50:23.410 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.406+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc97411eb10 con 0x7fc97411d9d0 2024-02-13T00:50:23.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc97411eb10 con 0x7fc97411d9d0 2024-02-13T00:50:23.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:23.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc9637fe700 10 client.27777.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc9741199a0 osd=3 4 2024-02-13T00:50:23.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 3 8 2024-02-13T00:50:23.412 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:50:23.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc978e10700 10 client.27777.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:23.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 9 2024-02-13T00:50:23.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc974121130 0x7fc974123520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:23.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc97411eb10 osd=4 3 2024-02-13T00:50:23.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc97411eb10 osd=4 3 2024-02-13T00:50:23.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc97411eb10 osd=4 3 2024-02-13T00:50:23.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 9 2024-02-13T00:50:23.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc97411eb10 osd=4 3 2024-02-13T00:50:23.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 4 9 2024-02-13T00:50:23.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc97411eb10 osd=4 4 2024-02-13T00:50:23.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc978e10700 10 client.27777.objecter _send_command 9 2024-02-13T00:50:23.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc97411eda0 con 0x7fc974121130 2024-02-13T00:50:23.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.408+0000 7fc97259c700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc974121130 0x7fc974123520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:23.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.409+0000 7fc9637fe700 10 client.27777.objecter ms_handle_connect 0x7fc974121130 2024-02-13T00:50:23.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.409+0000 7fc97259c700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc974121130 0x7fc974123520 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:23.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.410+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc97411eda0 con 0x7fc974121130 2024-02-13T00:50:23.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.410+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:50:23.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.410+0000 7fc9637fe700 10 client.27777.objecter _finish_command 9 = system:0 2024-02-13T00:50:23.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.410+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc97411eb10 osd=4 4 2024-02-13T00:50:23.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.410+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 4 9 2024-02-13T00:50:23.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.425+0000 7fc978e10700 10 client.27777.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:23.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.425+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 10 2024-02-13T00:50:23.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.425+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc97411eb10 osd=4 3 2024-02-13T00:50:23.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.425+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc97411eb10 osd=4 3 2024-02-13T00:50:23.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.425+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc97411eb10 osd=4 3 2024-02-13T00:50:23.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.425+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 10 2024-02-13T00:50:23.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.425+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc97411eb10 osd=4 3 2024-02-13T00:50:23.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.425+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 4 10 2024-02-13T00:50:23.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.425+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc97411eb10 osd=4 4 2024-02-13T00:50:23.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.425+0000 7fc978e10700 10 client.27777.objecter _send_command 10 2024-02-13T00:50:23.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.425+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc97411eda0 con 0x7fc974121130 2024-02-13T00:50:23.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc97411eda0 con 0x7fc974121130 2024-02-13T00:50:23.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:23.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc9637fe700 10 client.27777.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc97411eb10 osd=4 4 2024-02-13T00:50:23.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 4 10 2024-02-13T00:50:23.431 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:50:23.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc978e10700 10 client.27777.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:23.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 11 2024-02-13T00:50:23.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc9740f77a0 0x7fc974125a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:23.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc97411eda0 osd=5 3 2024-02-13T00:50:23.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc97411eda0 osd=5 3 2024-02-13T00:50:23.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc97411eda0 osd=5 3 2024-02-13T00:50:23.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 11 2024-02-13T00:50:23.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc97411eda0 osd=5 3 2024-02-13T00:50:23.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 5 11 2024-02-13T00:50:23.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc97411eda0 osd=5 4 2024-02-13T00:50:23.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc978e10700 10 client.27777.objecter _send_command 11 2024-02-13T00:50:23.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9740f91c0 con 0x7fc9740f77a0 2024-02-13T00:50:23.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc971d9b700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc9740f77a0 0x7fc974125a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:23.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc9637fe700 10 client.27777.objecter ms_handle_connect 0x7fc9740f77a0 2024-02-13T00:50:23.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.427+0000 7fc971d9b700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc9740f77a0 0x7fc974125a20 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:23.433 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:23.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.428+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc9740f91c0 con 0x7fc9740f77a0 2024-02-13T00:50:23.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.428+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:50:23.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.428+0000 7fc9637fe700 10 client.27777.objecter _finish_command 11 = system:0 2024-02-13T00:50:23.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.428+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc97411eda0 osd=5 4 2024-02-13T00:50:23.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.428+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 5 11 2024-02-13T00:50:23.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.453+0000 7fc978e10700 10 client.27777.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:23.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.453+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 12 2024-02-13T00:50:23.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.453+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc97411eda0 osd=5 3 2024-02-13T00:50:23.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.453+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc97411eda0 osd=5 3 2024-02-13T00:50:23.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.453+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc97411eda0 osd=5 3 2024-02-13T00:50:23.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.453+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 12 2024-02-13T00:50:23.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.453+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc97411eda0 osd=5 3 2024-02-13T00:50:23.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.453+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 5 12 2024-02-13T00:50:23.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.453+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc97411eda0 osd=5 4 2024-02-13T00:50:23.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.453+0000 7fc978e10700 10 client.27777.objecter _send_command 12 2024-02-13T00:50:23.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.453+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc9740f7be0 con 0x7fc9740f77a0 2024-02-13T00:50:23.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc9740f7be0 con 0x7fc9740f77a0 2024-02-13T00:50:23.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:23.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc9637fe700 10 client.27777.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc97411eda0 osd=5 4 2024-02-13T00:50:23.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 5 12 2024-02-13T00:50:23.459 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:50:23.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc978e10700 10 client.27777.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:23.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 13 2024-02-13T00:50:23.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc9740f82f0 0x7fc9740f8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:23.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9740f7be0 osd=6 3 2024-02-13T00:50:23.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740f7be0 osd=6 3 2024-02-13T00:50:23.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9740f7be0 osd=6 3 2024-02-13T00:50:23.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 13 2024-02-13T00:50:23.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc9740f7be0 osd=6 3 2024-02-13T00:50:23.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 6 13 2024-02-13T00:50:23.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740f7be0 osd=6 4 2024-02-13T00:50:23.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc978e10700 10 client.27777.objecter _send_command 13 2024-02-13T00:50:23.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9740f8c40 con 0x7fc9740f82f0 2024-02-13T00:50:23.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.455+0000 7fc972d9d700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc9740f82f0 0x7fc9740f8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:23.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.456+0000 7fc9637fe700 10 client.27777.objecter ms_handle_connect 0x7fc9740f82f0 2024-02-13T00:50:23.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.456+0000 7fc972d9d700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc9740f82f0 0x7fc9740f8750 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:23.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.456+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc9740f8c40 con 0x7fc9740f82f0 2024-02-13T00:50:23.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.456+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:50:23.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.456+0000 7fc9637fe700 10 client.27777.objecter _finish_command 13 = system:0 2024-02-13T00:50:23.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.456+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc9740f7be0 osd=6 4 2024-02-13T00:50:23.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.456+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 6 13 2024-02-13T00:50:23.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.471+0000 7fc978e10700 10 client.27777.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:23.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.471+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 14 2024-02-13T00:50:23.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.471+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9740f7be0 osd=6 3 2024-02-13T00:50:23.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.471+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740f7be0 osd=6 3 2024-02-13T00:50:23.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.471+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9740f7be0 osd=6 3 2024-02-13T00:50:23.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.471+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 14 2024-02-13T00:50:23.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.471+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc9740f7be0 osd=6 3 2024-02-13T00:50:23.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.471+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 6 14 2024-02-13T00:50:23.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.471+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740f7be0 osd=6 4 2024-02-13T00:50:23.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.471+0000 7fc978e10700 10 client.27777.objecter _send_command 14 2024-02-13T00:50:23.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.471+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc9740f8c40 con 0x7fc9740f82f0 2024-02-13T00:50:23.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc9740f8c40 con 0x7fc9740f82f0 2024-02-13T00:50:23.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:23.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc9637fe700 10 client.27777.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc9740f7be0 osd=6 4 2024-02-13T00:50:23.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 6 14 2024-02-13T00:50:23.477 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:50:23.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc978e10700 10 client.27777.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 15 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc9740f8e60 0x7fc97412eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9740f8c40 osd=7 3 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740f8c40 osd=7 3 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9740f8c40 osd=7 3 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 15 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc9740f8c40 osd=7 3 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 7 15 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740f8c40 osd=7 4 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc978e10700 10 client.27777.objecter _send_command 15 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc974131830 con 0x7fc9740f8e60 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.473+0000 7fc97259c700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc9740f8e60 0x7fc97412eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:23.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.474+0000 7fc9637fe700 10 client.27777.objecter ms_handle_connect 0x7fc9740f8e60 2024-02-13T00:50:23.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.474+0000 7fc97259c700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc9740f8e60 0x7fc97412eb10 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:23.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.474+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc974131830 con 0x7fc9740f8e60 2024-02-13T00:50:23.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.474+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:50:23.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.474+0000 7fc9637fe700 10 client.27777.objecter _finish_command 15 = system:0 2024-02-13T00:50:23.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.474+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc9740f8c40 osd=7 4 2024-02-13T00:50:23.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.474+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 7 15 2024-02-13T00:50:23.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.488+0000 7fc978e10700 10 client.27777.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:23.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.488+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign -1 16 2024-02-13T00:50:23.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.488+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9740f8c40 osd=7 3 2024-02-13T00:50:23.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.488+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740f8c40 osd=7 3 2024-02-13T00:50:23.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.488+0000 7fc978e10700 20 client.27777.objecter _get_session s=0x7fc9740f8c40 osd=7 3 2024-02-13T00:50:23.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.488+0000 7fc978e10700 15 client.27777.objecter _session_command_op_remove -1 16 2024-02-13T00:50:23.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.488+0000 7fc978e10700 20 client.27777.objecter get_session s=0x7fc9740f8c40 osd=7 3 2024-02-13T00:50:23.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.488+0000 7fc978e10700 15 client.27777.objecter _session_command_op_assign 7 16 2024-02-13T00:50:23.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.488+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740f8c40 osd=7 4 2024-02-13T00:50:23.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.488+0000 7fc978e10700 10 client.27777.objecter _send_command 16 2024-02-13T00:50:23.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.488+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc974131830 con 0x7fc9740f8e60 2024-02-13T00:50:23.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.490+0000 7fc9637fe700 1 -- 172.21.15.162:0/2300717534 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc974131830 con 0x7fc9740f8e60 2024-02-13T00:50:23.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.490+0000 7fc9637fe700 10 client.27777.objecter ms_dispatch 0x7fc974001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:23.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.490+0000 7fc9637fe700 10 client.27777.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.490+0000 7fc9637fe700 20 client.27777.objecter put_session s=0x7fc9740f8c40 osd=7 4 2024-02-13T00:50:23.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.490+0000 7fc9637fe700 15 client.27777.objecter _session_command_op_remove 7 16 2024-02-13T00:50:23.494 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:50:23.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 10 librados: watch_flush enter 2024-02-13T00:50:23.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 10 librados: watch_flush exit 2024-02-13T00:50:23.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 10 client.27777.objecter close_session for osd.0 2024-02-13T00:50:23.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc974028590 msgr2=0x7fc9740fea30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc974028590 0x7fc9740fea30 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740475e0 osd=0 1 2024-02-13T00:50:23.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 10 client.27777.objecter close_session for osd.1 2024-02-13T00:50:23.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6805/150599780 conn(0x7fc974027be0 msgr2=0x7fc974027fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6805/150599780 conn(0x7fc974027be0 0x7fc974027fe0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9741000a0 osd=1 1 2024-02-13T00:50:23.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 10 client.27777.objecter close_session for osd.2 2024-02-13T00:50:23.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc9740ffc00 msgr2=0x7fc97411c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc9740ffc00 0x7fc97411c360 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc974027a00 osd=2 1 2024-02-13T00:50:23.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 10 client.27777.objecter close_session for osd.3 2024-02-13T00:50:23.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc972d9d700 1 -- 172.21.15.162:0/2300717534 reap_dead start 2024-02-13T00:50:23.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6813/604971011 conn(0x7fc97411d9d0 msgr2=0x7fc97411de50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6813/604971011 conn(0x7fc97411d9d0 0x7fc97411de50 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9741199a0 osd=3 1 2024-02-13T00:50:23.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 10 client.27777.objecter close_session for osd.4 2024-02-13T00:50:23.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc974121130 msgr2=0x7fc974123520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc974121130 0x7fc974123520 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc97411eb10 osd=4 1 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 10 client.27777.objecter close_session for osd.5 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc9740f77a0 msgr2=0x7fc974125a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc9740f77a0 0x7fc974125a20 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc97411eda0 osd=5 1 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 10 client.27777.objecter close_session for osd.6 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc9740f82f0 msgr2=0x7fc9740f8750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc9740f82f0 0x7fc9740f8750 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740f7be0 osd=6 1 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 10 client.27777.objecter close_session for osd.7 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc9740f8e60 msgr2=0x7fc97412eb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc9740f8e60 0x7fc97412eb10 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 20 client.27777.objecter put_session s=0x7fc9740f8c40 osd=7 1 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 20 client.27777.objecter shutdown clearing up homeless session... 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 10 client.27777.objecter successfully canceled tick 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc972d9d700 1 -- 172.21.15.162:0/2300717534 reap_dead start 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc95c062c90 msgr2=0x7fc95c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc95c062c90 0x7fc95c065150 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 rx=0x7fc968005010 tx=0x7fc968009cb0).stop 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9740fdc50 msgr2=0x7fc97410e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9740fdc50 0x7fc97410e810 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 rx=0x7fc964009400 tx=0x7fc964009cb0).stop 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 shutdown_connections 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9740fdc50 0x7fc97410e810 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 --2- 172.21.15.162:0/2300717534 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc95c062c90 0x7fc95c065150 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 >> 172.21.15.162:0/2300717534 conn(0x7fc9740700d0 msgr2=0x7fc9740fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 shutdown_connections 2024-02-13T00:50:23.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 -- 172.21.15.162:0/2300717534 wait complete. 2024-02-13T00:50:23.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:23.491+0000 7fc978e10700 1 librados: shutdown 2024-02-13T00:50:23.500 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.500 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.500 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.500 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.500 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.500 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.500 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.500 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:50:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:22.466395+0000 osd.6 (osd.6) 179 : cluster [DBG] 6.23 scrub starts 2024-02-13T00:50:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:22.469425+0000 osd.6 (osd.6) 180 : cluster [DBG] 6.23 scrub ok 2024-02-13T00:50:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:22.489982+0000 mgr.y (mgr.14544) 517 : cluster [DBG] pgmap v911: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:50:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:22.567086+0000 mon.a (mon.0) 3107 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-104","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:50:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:22.567203+0000 mon.a (mon.0) 3108 : cluster [DBG] osdmap e608: 8 total, 8 up, 4 in 2024-02-13T00:50:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:22.578558+0000 mon.a (mon.0) 3109 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:50:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:22.922670+0000 osd.0 (osd.0) 231 : cluster [DBG] 28.14 scrub starts 2024-02-13T00:50:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:22.925599+0000 osd.0 (osd.0) 232 : cluster [DBG] 28.14 scrub ok 2024-02-13T00:50:23.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:23.339298+0000 mon.a (mon.0) 3110 : audit [DBG] from='client.? 172.21.15.162:0/2300717534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:22.466395+0000 osd.6 (osd.6) 179 : cluster [DBG] 6.23 scrub starts 2024-02-13T00:50:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:22.469425+0000 osd.6 (osd.6) 180 : cluster [DBG] 6.23 scrub ok 2024-02-13T00:50:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:22.489982+0000 mgr.y (mgr.14544) 517 : cluster [DBG] pgmap v911: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:50:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:22.567086+0000 mon.a (mon.0) 3107 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-104","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:50:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:22.567203+0000 mon.a (mon.0) 3108 : cluster [DBG] osdmap e608: 8 total, 8 up, 4 in 2024-02-13T00:50:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:22.578558+0000 mon.a (mon.0) 3109 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:50:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:22.922670+0000 osd.0 (osd.0) 231 : cluster [DBG] 28.14 scrub starts 2024-02-13T00:50:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:22.925599+0000 osd.0 (osd.0) 232 : cluster [DBG] 28.14 scrub ok 2024-02-13T00:50:23.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:23.339298+0000 mon.a (mon.0) 3110 : audit [DBG] from='client.? 172.21.15.162:0/2300717534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:22.466395+0000 osd.6 (osd.6) 179 : cluster [DBG] 6.23 scrub starts 2024-02-13T00:50:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:22.469425+0000 osd.6 (osd.6) 180 : cluster [DBG] 6.23 scrub ok 2024-02-13T00:50:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:22.489982+0000 mgr.y (mgr.14544) 517 : cluster [DBG] pgmap v911: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:50:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:22.567086+0000 mon.a (mon.0) 3107 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-104","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:50:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:22.567203+0000 mon.a (mon.0) 3108 : cluster [DBG] osdmap e608: 8 total, 8 up, 4 in 2024-02-13T00:50:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:22.578558+0000 mon.a (mon.0) 3109 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:50:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:22.922670+0000 osd.0 (osd.0) 231 : cluster [DBG] 28.14 scrub starts 2024-02-13T00:50:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:22.925599+0000 osd.0 (osd.0) 232 : cluster [DBG] 28.14 scrub ok 2024-02-13T00:50:24.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:23.339298+0000 mon.a (mon.0) 3110 : audit [DBG] from='client.? 172.21.15.162:0/2300717534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:24.287 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:24.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:24.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-13T00:50:24.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:24.424 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:24.445 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:24.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:24.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:24.582 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:24.603 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:24.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:24.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:24.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:24.735 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:23.427119+0000 osd.6 (osd.6) 181 : cluster [DBG] 28.5 scrub starts 2024-02-13T00:50:24.735 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:23.430391+0000 osd.6 (osd.6) 182 : cluster [DBG] 28.5 scrub ok 2024-02-13T00:50:24.735 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:23.568626+0000 mon.a (mon.0) 3111 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-104", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:50:24.735 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:23.568692+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e609: 8 total, 8 up, 4 in 2024-02-13T00:50:24.735 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:23.570844+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-104"}]: dispatch 2024-02-13T00:50:24.735 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:23.878897+0000 mon.c (mon.2) 473 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:50:24.735 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:23.879584+0000 mon.c (mon.2) 474 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:50:24.735 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:23.884084+0000 mon.a (mon.0) 3114 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:24.736 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:24.756 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:24.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:23.427119+0000 osd.6 (osd.6) 181 : cluster [DBG] 28.5 scrub starts 2024-02-13T00:50:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:23.430391+0000 osd.6 (osd.6) 182 : cluster [DBG] 28.5 scrub ok 2024-02-13T00:50:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:23.568626+0000 mon.a (mon.0) 3111 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-104", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:50:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:23.568692+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e609: 8 total, 8 up, 4 in 2024-02-13T00:50:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:23.570844+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-104"}]: dispatch 2024-02-13T00:50:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:23.878897+0000 mon.c (mon.2) 473 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:50:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:23.879584+0000 mon.c (mon.2) 474 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:50:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:23.884084+0000 mon.a (mon.0) 3114 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:23.427119+0000 osd.6 (osd.6) 181 : cluster [DBG] 28.5 scrub starts 2024-02-13T00:50:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:23.430391+0000 osd.6 (osd.6) 182 : cluster [DBG] 28.5 scrub ok 2024-02-13T00:50:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:23.568626+0000 mon.a (mon.0) 3111 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-104", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:50:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:23.568692+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e609: 8 total, 8 up, 4 in 2024-02-13T00:50:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:23.570844+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-104"}]: dispatch 2024-02-13T00:50:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:23.878897+0000 mon.c (mon.2) 473 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:50:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:23.879584+0000 mon.c (mon.2) 474 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:50:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:23.884084+0000 mon.a (mon.0) 3114 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:24.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:24.890 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:24.910 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:24.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:25.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:25.370 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:25.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:25.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:25.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:25.523 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:25.674 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:25.676 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:24.490476+0000 mgr.y (mgr.14544) 518 : cluster [DBG] pgmap v914: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:25.676 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:24.574827+0000 mon.a (mon.0) 3115 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-104"}]': finished 2024-02-13T00:50:25.676 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:24.574897+0000 mon.a (mon.0) 3116 : cluster [DBG] osdmap e610: 8 total, 8 up, 4 in 2024-02-13T00:50:25.676 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:24.576470+0000 mon.a (mon.0) 3117 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-104", "mode": "writeback"}]: dispatch 2024-02-13T00:50:25.676 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:24.997776+0000 osd.0 (osd.0) 233 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:50:25.676 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:25.001213+0000 osd.0 (osd.0) 234 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:50:25.676 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:25.498183+0000 mon.a (mon.0) 3118 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:50:25.676 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:25.502510+0000 mon.a (mon.0) 3119 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-104", "mode": "writeback"}]': finished 2024-02-13T00:50:25.676 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:25.502568+0000 mon.a (mon.0) 3120 : cluster [DBG] osdmap e611: 8 total, 8 up, 4 in 2024-02-13T00:50:25.676 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:25.545340+0000 mon.a (mon.0) 3121 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:25.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:24.490476+0000 mgr.y (mgr.14544) 518 : cluster [DBG] pgmap v914: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:25.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:24.574827+0000 mon.a (mon.0) 3115 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-104"}]': finished 2024-02-13T00:50:25.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:24.574897+0000 mon.a (mon.0) 3116 : cluster [DBG] osdmap e610: 8 total, 8 up, 4 in 2024-02-13T00:50:25.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:24.576470+0000 mon.a (mon.0) 3117 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-104", "mode": "writeback"}]: dispatch 2024-02-13T00:50:25.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:24.997776+0000 osd.0 (osd.0) 233 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:50:25.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:25.001213+0000 osd.0 (osd.0) 234 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:50:25.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:25.498183+0000 mon.a (mon.0) 3118 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:50:25.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:25.502510+0000 mon.a (mon.0) 3119 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-104", "mode": "writeback"}]': finished 2024-02-13T00:50:25.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:25.502568+0000 mon.a (mon.0) 3120 : cluster [DBG] osdmap e611: 8 total, 8 up, 4 in 2024-02-13T00:50:25.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:25.545340+0000 mon.a (mon.0) 3121 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:25.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:25.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:25.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:25.809 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:25.829 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:25.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:25.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:25.960 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:50:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:50:25.960 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:24.490476+0000 mgr.y (mgr.14544) 518 : cluster [DBG] pgmap v914: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:25.960 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:24.574827+0000 mon.a (mon.0) 3115 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-104"}]': finished 2024-02-13T00:50:25.960 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:24.574897+0000 mon.a (mon.0) 3116 : cluster [DBG] osdmap e610: 8 total, 8 up, 4 in 2024-02-13T00:50:25.960 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:24.576470+0000 mon.a (mon.0) 3117 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-104", "mode": "writeback"}]: dispatch 2024-02-13T00:50:25.960 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:24.997776+0000 osd.0 (osd.0) 233 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:50:25.960 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:25.001213+0000 osd.0 (osd.0) 234 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:50:25.960 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:25.498183+0000 mon.a (mon.0) 3118 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:50:25.961 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:25.502510+0000 mon.a (mon.0) 3119 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-104", "mode": "writeback"}]': finished 2024-02-13T00:50:25.961 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:25.502568+0000 mon.a (mon.0) 3120 : cluster [DBG] osdmap e611: 8 total, 8 up, 4 in 2024-02-13T00:50:25.961 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:25.545340+0000 mon.a (mon.0) 3121 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:25.961 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:25.980 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:25.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:26.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:26.113 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:26.211 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:26.212 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:50:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:50:26] "GET /metrics HTTP/1.1" 200 188926 "" "Prometheus/2.33.4" 2024-02-13T00:50:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:26.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:26.314 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:26.340 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:26.367 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:26.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:26.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:26.503 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:26.525 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:26.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:26.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:26.654 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:26.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:26.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:26.788 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:26.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:25.798251+0000 mgr.y (mgr.14544) 519 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:26.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:26.510855+0000 mon.a (mon.0) 3122 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:50:26.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:26.510918+0000 mon.a (mon.0) 3123 : cluster [DBG] osdmap e612: 8 total, 8 up, 4 in 2024-02-13T00:50:26.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:26.512945+0000 mon.a (mon.0) 3124 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-104"}]: dispatch 2024-02-13T00:50:26.815 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:26.853 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:26.854 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:25.798251+0000 mgr.y (mgr.14544) 519 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:26.854 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:26.510855+0000 mon.a (mon.0) 3122 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:50:26.854 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:26.510918+0000 mon.a (mon.0) 3123 : cluster [DBG] osdmap e612: 8 total, 8 up, 4 in 2024-02-13T00:50:26.855 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:26.512945+0000 mon.a (mon.0) 3124 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-104"}]: dispatch 2024-02-13T00:50:26.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:25.798251+0000 mgr.y (mgr.14544) 519 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:26.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:26.510855+0000 mon.a (mon.0) 3122 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:50:26.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:26.510918+0000 mon.a (mon.0) 3123 : cluster [DBG] osdmap e612: 8 total, 8 up, 4 in 2024-02-13T00:50:26.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:26.512945+0000 mon.a (mon.0) 3124 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-104"}]: dispatch 2024-02-13T00:50:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:26.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:26.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:26.959 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:50:26.986 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:27.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.286+0000 7f9588395700 1 -- 172.21.15.162:0/1301844267 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95800fd490 msgr2=0x7f95800f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:27.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.286+0000 7f9588395700 1 --2- 172.21.15.162:0/1301844267 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95800fd490 0x7f95800f6cc0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 rx=0x7f95700047b0 tx=0x7f9570009cb0).stop 2024-02-13T00:50:27.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.286+0000 7f9588395700 1 -- 172.21.15.162:0/1301844267 shutdown_connections 2024-02-13T00:50:27.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.286+0000 7f9588395700 1 --2- 172.21.15.162:0/1301844267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95800f73a0 0x7f95801039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.286+0000 7f9588395700 1 --2- 172.21.15.162:0/1301844267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95800fcb30 0x7f95800fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:50:27.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.286+0000 7f9588395700 1 --2- 172.21.15.162:0/1301844267 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95800fd490 0x7f95800f6cc0 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.286+0000 7f9588395700 1 -- 172.21.15.162:0/1301844267 >> 172.21.15.162:0/1301844267 conn(0x7f958006eb90 msgr2=0x7f95800f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:27.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.286+0000 7f9588395700 1 -- 172.21.15.162:0/1301844267 shutdown_connections 2024-02-13T00:50:27.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.286+0000 7f9588395700 1 -- 172.21.15.162:0/1301844267 wait complete. 2024-02-13T00:50:27.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.286+0000 7f9588395700 1 librados: starting msgr at 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.286+0000 7f9588395700 1 librados: starting objecter 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.286+0000 7f9588395700 1 Processor -- start 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9588395700 1 -- start start 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9588395700 1 librados: setting wanted keys 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9588395700 1 librados: calling monclient init 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9588395700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95800f73a0 0x7f9580109280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9588395700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95800fcb30 0x7f9580109770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9588395700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95800fd490 0x7f9580109da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9588395700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95801144b0 con 0x7f95800fcb30 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9588395700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9580114620 con 0x7f95800f73a0 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9588395700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95801147e0 con 0x7f95800fd490 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9586932700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95800fd490 0x7f9580109da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9585930700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95800fcb30 0x7f9580109770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9586932700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95800fd490 0x7f9580109da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:50616/0 (socket says 172.21.15.162:50616) 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9585930700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95800fcb30 0x7f9580109770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60984/0 (socket says 172.21.15.162:60984) 2024-02-13T00:50:27.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9586932700 1 -- 172.21.15.162:0/3275029841 learned_addr learned my addr 172.21.15.162:0/3275029841 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9586131700 1 --2- 172.21.15.162:0/3275029841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95800f73a0 0x7f9580109280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9585930700 1 -- 172.21.15.162:0/3275029841 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95800fd490 msgr2=0x7f9580109da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9585930700 1 --2- 172.21.15.162:0/3275029841 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95800fd490 0x7f9580109da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9585930700 1 -- 172.21.15.162:0/3275029841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95800f73a0 msgr2=0x7f9580109280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9585930700 1 --2- 172.21.15.162:0/3275029841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95800f73a0 0x7f9580109280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9585930700 1 -- 172.21.15.162:0/3275029841 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95780097c0 con 0x7f95800fcb30 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9586131700 1 --2- 172.21.15.162:0/3275029841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95800f73a0 0x7f9580109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9586932700 1 --2- 172.21.15.162:0/3275029841 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95800fd490 0x7f9580109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f9585930700 1 --2- 172.21.15.162:0/3275029841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95800fcb30 0x7f9580109770 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7f9570004650 tx=0x7f9570007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f95777fe700 10 client.?.objecter ms_handle_connect 0x7f95800fcb30 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.287+0000 7f95777fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.288+0000 7f95777fe700 1 -- 172.21.15.162:0/3275029841 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9570018070 con 0x7f95800fcb30 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.288+0000 7f95777fe700 1 -- 172.21.15.162:0/3275029841 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95700093b0 con 0x7f95800fcb30 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.288+0000 7f9588395700 1 -- 172.21.15.162:0/3275029841 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9570005040 con 0x7f95800fcb30 2024-02-13T00:50:27.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.288+0000 7f95777fe700 1 -- 172.21.15.162:0/3275029841 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957001ddd0 con 0x7f95800fcb30 2024-02-13T00:50:27.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.288+0000 7f9588395700 10 client.27783.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:27.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.288+0000 7f9588395700 1 -- 172.21.15.162:0/3275029841 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9580114c80 con 0x7f95800fcb30 2024-02-13T00:50:27.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.288+0000 7f9588395700 1 librados: init done 2024-02-13T00:50:27.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.289+0000 7f95777fe700 1 -- 172.21.15.162:0/3275029841 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f957001d420 con 0x7f95800fcb30 2024-02-13T00:50:27.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.289+0000 7f9588395700 1 -- 172.21.15.162:0/3275029841 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95800475e0 con 0x7f95800fcb30 2024-02-13T00:50:27.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.289+0000 7f95777fe700 1 --2- 172.21.15.162:0/3275029841 >> v2:172.21.15.162:6800/2866534053 conn(0x7f956c062c90 0x7f956c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:27.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.289+0000 7f95777fe700 1 -- 172.21.15.162:0/3275029841 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(612..612 src has 449..612) v4 ==== 8470+0+0 (secure 0 0 0) 0x7f957007e910 con 0x7f95800fcb30 2024-02-13T00:50:27.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.289+0000 7f95777fe700 10 client.27783.objecter ms_dispatch 0x7f9580001550 osd_map(612..612 src has 449..612) v4 2024-02-13T00:50:27.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.289+0000 7f95777fe700 3 client.27783.objecter handle_osd_map got epochs [612,612] > 0 2024-02-13T00:50:27.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.289+0000 7f9586131700 1 --2- 172.21.15.162:0/3275029841 >> v2:172.21.15.162:6800/2866534053 conn(0x7f956c062c90 0x7f956c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:27.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.289+0000 7f95777fe700 3 client.27783.objecter handle_osd_map decoding full epoch 612 2024-02-13T00:50:27.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.290+0000 7f95777fe700 20 client.27783.objecter dump_active .. 0 homeless 2024-02-13T00:50:27.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.290+0000 7f95777fe700 10 client.27783.objecter ms_handle_connect 0x7f956c062c90 2024-02-13T00:50:27.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.290+0000 7f9586131700 1 --2- 172.21.15.162:0/3275029841 >> v2:172.21.15.162:6800/2866534053 conn(0x7f956c062c90 0x7f956c065150 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 rx=0x7f957c009990 tx=0x7f957c008040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:27.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.294+0000 7f95777fe700 1 -- 172.21.15.162:0/3275029841 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9570051c40 con 0x7f95800fcb30 2024-02-13T00:50:27.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.460+0000 7f9588395700 1 -- 172.21.15.162:0/3275029841 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f958005fb80 con 0x7f956c062c90 2024-02-13T00:50:27.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.474+0000 7f95777fe700 1 -- 172.21.15.162:0/3275029841 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+506701 (secure 0 0 0) 0x7f958005fb80 con 0x7f956c062c90 2024-02-13T00:50:27.483 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:50:27.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.477+0000 7f9588395700 10 librados: watch_flush enter 2024-02-13T00:50:27.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 10 librados: watch_flush exit 2024-02-13T00:50:27.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 20 client.27783.objecter shutdown clearing up homeless session... 2024-02-13T00:50:27.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 10 client.27783.objecter successfully canceled tick 2024-02-13T00:50:27.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 -- 172.21.15.162:0/3275029841 >> v2:172.21.15.162:6800/2866534053 conn(0x7f956c062c90 msgr2=0x7f956c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:27.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 --2- 172.21.15.162:0/3275029841 >> v2:172.21.15.162:6800/2866534053 conn(0x7f956c062c90 0x7f956c065150 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 rx=0x7f957c009990 tx=0x7f957c008040).stop 2024-02-13T00:50:27.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 -- 172.21.15.162:0/3275029841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95800fcb30 msgr2=0x7f9580109770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:27.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 --2- 172.21.15.162:0/3275029841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95800fcb30 0x7f9580109770 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7f9570004650 tx=0x7f9570007f90).stop 2024-02-13T00:50:27.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 -- 172.21.15.162:0/3275029841 shutdown_connections 2024-02-13T00:50:27.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 --2- 172.21.15.162:0/3275029841 >> v2:172.21.15.162:6800/2866534053 conn(0x7f956c062c90 0x7f956c065150 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 --2- 172.21.15.162:0/3275029841 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95800fd490 0x7f9580109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 --2- 172.21.15.162:0/3275029841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95800f73a0 0x7f9580109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 --2- 172.21.15.162:0/3275029841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95800fcb30 0x7f9580109770 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 -- 172.21.15.162:0/3275029841 >> 172.21.15.162:0/3275029841 conn(0x7f958006eb90 msgr2=0x7f95800fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:27.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 -- 172.21.15.162:0/3275029841 shutdown_connections 2024-02-13T00:50:27.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 -- 172.21.15.162:0/3275029841 wait complete. 2024-02-13T00:50:27.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.478+0000 7f9588395700 1 librados: shutdown 2024-02-13T00:50:27.489 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:50:27.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.522+0000 7f5feeb98700 1 -- 172.21.15.162:0/382207245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe80fcb30 msgr2=0x7f5fe80fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:27.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.522+0000 7f5feeb98700 1 --2- 172.21.15.162:0/382207245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe80fcb30 0x7f5fe80fcf10 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7f5fdc004870 tx=0x7f5fdc009cb0).stop 2024-02-13T00:50:27.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.522+0000 7f5feeb98700 1 -- 172.21.15.162:0/382207245 shutdown_connections 2024-02-13T00:50:27.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.522+0000 7f5feeb98700 1 --2- 172.21.15.162:0/382207245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe80f73a0 0x7f5fe81039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.522+0000 7f5feeb98700 1 --2- 172.21.15.162:0/382207245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe80fcb30 0x7f5fe80fcf10 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.522+0000 7f5feeb98700 1 --2- 172.21.15.162:0/382207245 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fe80fd490 0x7f5fe80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.522+0000 7f5feeb98700 1 -- 172.21.15.162:0/382207245 >> 172.21.15.162:0/382207245 conn(0x7f5fe806eb90 msgr2=0x7f5fe80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:27.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.522+0000 7f5feeb98700 1 -- 172.21.15.162:0/382207245 shutdown_connections 2024-02-13T00:50:27.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.523+0000 7f5feeb98700 1 -- 172.21.15.162:0/382207245 wait complete. 2024-02-13T00:50:27.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.523+0000 7f5feeb98700 1 librados: starting msgr at 2024-02-13T00:50:27.569 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.523+0000 7f5feeb98700 1 librados: starting objecter 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.523+0000 7f5feeb98700 1 Processor -- start 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.523+0000 7f5feeb98700 1 -- start start 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.523+0000 7f5feeb98700 1 librados: setting wanted keys 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.523+0000 7f5feeb98700 1 librados: calling monclient init 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5feeb98700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fe80f73a0 0x7f5fe806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5feeb98700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe80fcb30 0x7f5fe8062960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5feeb98700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe80fd490 0x7f5fe8062e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5feeb98700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80634c0 con 0x7f5fe80fd490 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5feeb98700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe8063630 con 0x7f5fe80fcb30 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5feeb98700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80637f0 con 0x7f5fe80f73a0 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fec934700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fe80f73a0 0x7f5fe806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fed135700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe80fd490 0x7f5fe8062e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fec934700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fe80f73a0 0x7f5fe806a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:50630/0 (socket says 172.21.15.162:50630) 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fec934700 1 -- 172.21.15.162:0/4133962918 learned_addr learned my addr 172.21.15.162:0/4133962918 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:27.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fe7fff700 1 --2- 172.21.15.162:0/4133962918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe80fcb30 0x7f5fe8062960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fed135700 1 -- 172.21.15.162:0/4133962918 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fe80f73a0 msgr2=0x7f5fe806a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fed135700 1 --2- 172.21.15.162:0/4133962918 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fe80f73a0 0x7f5fe806a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fed135700 1 -- 172.21.15.162:0/4133962918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe80fcb30 msgr2=0x7f5fe8062960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fed135700 1 --2- 172.21.15.162:0/4133962918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe80fcb30 0x7f5fe8062960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fed135700 1 -- 172.21.15.162:0/4133962918 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc005040 con 0x7f5fe80fd490 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fec934700 1 --2- 172.21.15.162:0/4133962918 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fe80f73a0 0x7f5fe806a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fe7fff700 1 --2- 172.21.15.162:0/4133962918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe80fcb30 0x7f5fe8062960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fed135700 1 --2- 172.21.15.162:0/4133962918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe80fd490 0x7f5fe8062e50 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 rx=0x7f5fe0030bb0 tx=0x7f5fe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fe5ffb700 10 client.?.objecter ms_handle_connect 0x7f5fe80fd490 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fe5ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.524+0000 7f5fe5ffb700 1 -- 172.21.15.162:0/4133962918 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0009870 con 0x7f5fe80fd490 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.525+0000 7f5fe5ffb700 1 -- 172.21.15.162:0/4133962918 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fe0004020 con 0x7f5fe80fd490 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.525+0000 7f5fe5ffb700 1 -- 172.21.15.162:0/4133962918 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe002e560 con 0x7f5fe80fd490 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.525+0000 7f5feeb98700 1 -- 172.21.15.162:0/4133962918 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe810a3b0 con 0x7f5fe80fd490 2024-02-13T00:50:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.525+0000 7f5feeb98700 10 client.27789.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.525+0000 7f5feeb98700 1 -- 172.21.15.162:0/4133962918 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe810a850 con 0x7f5fe80fd490 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.525+0000 7f5feeb98700 1 librados: init done 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.526+0000 7f5feeb98700 1 -- 172.21.15.162:0/4133962918 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe80475e0 con 0x7f5fe80fd490 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.528+0000 7f5fe5ffb700 1 -- 172.21.15.162:0/4133962918 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f5fe002e990 con 0x7f5fe80fd490 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.528+0000 7f5fe5ffb700 1 --2- 172.21.15.162:0/4133962918 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5fd8062c90 0x7f5fd8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.528+0000 7f5fe5ffb700 1 -- 172.21.15.162:0/4133962918 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(613..613 src has 449..613) v4 ==== 8462+0+0 (secure 0 0 0) 0x7f5fe009fc50 con 0x7f5fe80fd490 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.528+0000 7f5fe5ffb700 10 client.27789.objecter ms_dispatch 0x7f5fe8001550 osd_map(613..613 src has 449..613) v4 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.528+0000 7f5fe5ffb700 3 client.27789.objecter handle_osd_map got epochs [613,613] > 0 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.528+0000 7f5fe5ffb700 3 client.27789.objecter handle_osd_map decoding full epoch 613 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.529+0000 7f5fe5ffb700 20 client.27789.objecter dump_active .. 0 homeless 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.529+0000 7f5fec934700 1 --2- 172.21.15.162:0/4133962918 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5fd8062c90 0x7f5fd8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.529+0000 7f5fec934700 1 --2- 172.21.15.162:0/4133962918 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5fd8062c90 0x7f5fd8065150 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7f5fdc009a00 tx=0x7f5fdc00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.529+0000 7f5fe5ffb700 10 client.27789.objecter ms_handle_connect 0x7f5fd8062c90 2024-02-13T00:50:27.573 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.532+0000 7f5fe5ffb700 1 -- 172.21.15.162:0/4133962918 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5fe0073010 con 0x7f5fe80fd490 2024-02-13T00:50:27.577 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:27.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:27.722+0000 7f5feeb98700 1 -- 172.21.15.162:0/4133962918 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f5fe8106020 con 0x7f5fe80fd490 2024-02-13T00:50:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:26.491116+0000 mgr.y (mgr.14544) 520 : cluster [DBG] pgmap v917: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:50:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:26.828070+0000 osd.1 (osd.1) 257 : cluster [DBG] 28.17 scrub starts 2024-02-13T00:50:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:26.830265+0000 osd.1 (osd.1) 258 : cluster [DBG] 28.17 scrub ok 2024-02-13T00:50:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:27.507999+0000 mon.a (mon.0) 3125 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:50:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:27.512522+0000 mon.a (mon.0) 3126 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-104"}]': finished 2024-02-13T00:50:27.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:27.512589+0000 mon.a (mon.0) 3127 : cluster [DBG] osdmap e613: 8 total, 8 up, 4 in 2024-02-13T00:50:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:26.491116+0000 mgr.y (mgr.14544) 520 : cluster [DBG] pgmap v917: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:50:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:26.828070+0000 osd.1 (osd.1) 257 : cluster [DBG] 28.17 scrub starts 2024-02-13T00:50:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:26.830265+0000 osd.1 (osd.1) 258 : cluster [DBG] 28.17 scrub ok 2024-02-13T00:50:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:27.507999+0000 mon.a (mon.0) 3125 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:50:27.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:27.512522+0000 mon.a (mon.0) 3126 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-104"}]': finished 2024-02-13T00:50:27.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:27.512589+0000 mon.a (mon.0) 3127 : cluster [DBG] osdmap e613: 8 total, 8 up, 4 in 2024-02-13T00:50:27.837 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:27.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:27.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:27.971 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:26.491116+0000 mgr.y (mgr.14544) 520 : cluster [DBG] pgmap v917: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:50:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:26.828070+0000 osd.1 (osd.1) 257 : cluster [DBG] 28.17 scrub starts 2024-02-13T00:50:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:26.830265+0000 osd.1 (osd.1) 258 : cluster [DBG] 28.17 scrub ok 2024-02-13T00:50:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:27.507999+0000 mon.a (mon.0) 3125 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:50:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:27.512522+0000 mon.a (mon.0) 3126 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-104"}]': finished 2024-02-13T00:50:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:27.512589+0000 mon.a (mon.0) 3127 : cluster [DBG] osdmap e613: 8 total, 8 up, 4 in 2024-02-13T00:50:28.068 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:28.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:28.081 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:28.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:28.204 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:28.224 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:28.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:28.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:28.359 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:28.379 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:28.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:28.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:28.512 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:28.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.519+0000 7f5fe5ffb700 1 -- 172.21.15.162:0/4133962918 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v614) v1 ==== 101+0+0 (secure 0 0 0) 0x7f5fe0072c00 con 0x7f5fe80fd490 2024-02-13T00:50:28.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.522+0000 7f5feeb98700 10 librados: watch_flush enter 2024-02-13T00:50:28.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 10 librados: watch_flush exit 2024-02-13T00:50:28.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 20 client.27789.objecter shutdown clearing up homeless session... 2024-02-13T00:50:28.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 10 client.27789.objecter successfully canceled tick 2024-02-13T00:50:28.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 1 -- 172.21.15.162:0/4133962918 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5fd8062c90 msgr2=0x7f5fd8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:28.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 1 --2- 172.21.15.162:0/4133962918 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5fd8062c90 0x7f5fd8065150 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7f5fdc009a00 tx=0x7f5fdc00b040).stop 2024-02-13T00:50:28.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 1 -- 172.21.15.162:0/4133962918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe80fd490 msgr2=0x7f5fe8062e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:28.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 1 --2- 172.21.15.162:0/4133962918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe80fd490 0x7f5fe8062e50 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 rx=0x7f5fe0030bb0 tx=0x7f5fe000b040).stop 2024-02-13T00:50:28.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 1 -- 172.21.15.162:0/4133962918 shutdown_connections 2024-02-13T00:50:28.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 1 --2- 172.21.15.162:0/4133962918 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5fd8062c90 0x7f5fd8065150 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:28.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 1 --2- 172.21.15.162:0/4133962918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fe80fd490 0x7f5fe8062e50 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:28.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 1 --2- 172.21.15.162:0/4133962918 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5fe80f73a0 0x7f5fe806a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:28.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 1 --2- 172.21.15.162:0/4133962918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fe80fcb30 0x7f5fe8062960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:28.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 1 -- 172.21.15.162:0/4133962918 >> 172.21.15.162:0/4133962918 conn(0x7f5fe806eb90 msgr2=0x7f5fe806f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:28.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.523+0000 7f5feeb98700 1 -- 172.21.15.162:0/4133962918 shutdown_connections 2024-02-13T00:50:28.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.524+0000 7f5feeb98700 1 -- 172.21.15.162:0/4133962918 wait complete. 2024-02-13T00:50:28.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:28.524+0000 7f5feeb98700 1 librados: shutdown 2024-02-13T00:50:28.532 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:50:28.913 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":918,"stamp":"2024-02-13T00:50:26.508655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5813,"num_objects":227,"num_object_clones":0,"num_object_copies":681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":227,"num_whiteouts":0,"num_read":2255,"num_read_kb":1975,"num_write":915,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":13678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1046,"ondisk_log_size":1046,"up":612,"acting":612,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":612,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1221968,"kb_used_data":12592,"kb_used_omap":0,"kb_used_meta":1209344,"kb_avail":373758640,"statfs":{"total":383980142592,"available":382728847360,"internally_reserved":0,"allocated":12894208,"data_stored":6140761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1238368256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.943768"},"pg_stats":[{"pgid":"236.1f","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956795+0000","last_change":"2024-02-13T00:50:22.581898+0000","last_active":"2024-02-13T00:50:23.956795+0000","last_peered":"2024-02-13T00:50:23.956795+0000","last_clean":"2024-02-13T00:50:23.956795+0000","last_became_active":"2024-02-13T00:50:22.581583+0000","last_became_peered":"2024-02-13T00:50:22.581583+0000","last_unstale":"2024-02-13T00:50:23.956795+0000","last_undegraded":"2024-02-13T00:50:23.956795+0000","last_fullsized":"2024-02-13T00:50:23.956795+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"236.1e","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956405+0000","last_change":"2024-02-13T00:50:22.583899+0000","last_active":"2024-02-13T00:50:23.956405+0000","last_peered":"2024-02-13T00:50:23.956405+0000","last_clean":"2024-02-13T00:50:23.956405+0000","last_became_active":"2024-02-13T00:50:22.583622+0000","last_became_peered":"2024-02-13T00:50:22.583622+0000","last_unstale":"2024-02-13T00:50:23.956405+0000","last_undegraded":"2024-02-13T00:50:23.956405+0000","last_fullsized":"2024-02-13T00:50:23.956405+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"236.1d","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570544+0000","last_change":"2024-02-13T00:50:22.582244+0000","last_active":"2024-02-13T00:50:23.570544+0000","last_peered":"2024-02-13T00:50:23.570544+0000","last_clean":"2024-02-13T00:50:23.570544+0000","last_became_active":"2024-02-13T00:50:22.581906+0000","last_became_peered":"2024-02-13T00:50:22.581906+0000","last_unstale":"2024-02-13T00:50:23.570544+0000","last_undegraded":"2024-02-13T00:50:23.570544+0000","last_fullsized":"2024-02-13T00:50:23.570544+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"236.1c","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569692+0000","last_change":"2024-02-13T00:50:22.583303+0000","last_active":"2024-02-13T00:50:23.569692+0000","last_peered":"2024-02-13T00:50:23.569692+0000","last_clean":"2024-02-13T00:50:23.569692+0000","last_became_active":"2024-02-13T00:50:22.583030+0000","last_became_peered":"2024-02-13T00:50:22.583030+0000","last_unstale":"2024-02-13T00:50:23.569692+0000","last_undegraded":"2024-02-13T00:50:23.569692+0000","last_fullsized":"2024-02-13T00:50:23.569692+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"236.1b","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568477+0000","last_change":"2024-02-13T00:50:22.584860+0000","last_active":"2024-02-13T00:50:23.568477+0000","last_peered":"2024-02-13T00:50:23.568477+0000","last_clean":"2024-02-13T00:50:23.568477+0000","last_became_active":"2024-02-13T00:50:22.584457+0000","last_became_peered":"2024-02-13T00:50:22.584457+0000","last_unstale":"2024-02-13T00:50:23.568477+0000","last_undegraded":"2024-02-13T00:50:23.568477+0000","last_fullsized":"2024-02-13T00:50:23.568477+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.1a","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568146+0000","last_change":"2024-02-13T00:50:22.585888+0000","last_active":"2024-02-13T00:50:23.568146+0000","last_peered":"2024-02-13T00:50:23.568146+0000","last_clean":"2024-02-13T00:50:23.568146+0000","last_became_active":"2024-02-13T00:50:22.584518+0000","last_became_peered":"2024-02-13T00:50:22.584518+0000","last_unstale":"2024-02-13T00:50:23.568146+0000","last_undegraded":"2024-02-13T00:50:23.568146+0000","last_fullsized":"2024-02-13T00:50:23.568146+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.19","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955315+0000","last_change":"2024-02-13T00:50:22.588015+0000","last_active":"2024-02-13T00:50:23.955315+0000","last_peered":"2024-02-13T00:50:23.955315+0000","last_clean":"2024-02-13T00:50:23.955315+0000","last_became_active":"2024-02-13T00:50:22.587733+0000","last_became_peered":"2024-02-13T00:50:22.587733+0000","last_unstale":"2024-02-13T00:50:23.955315+0000","last_undegraded":"2024-02-13T00:50:23.955315+0000","last_fullsized":"2024-02-13T00:50:23.955315+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"236.18","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955144+0000","last_change":"2024-02-13T00:50:22.579596+0000","last_active":"2024-02-13T00:50:23.955144+0000","last_peered":"2024-02-13T00:50:23.955144+0000","last_clean":"2024-02-13T00:50:23.955144+0000","last_became_active":"2024-02-13T00:50:22.579319+0000","last_became_peered":"2024-02-13T00:50:22.579319+0000","last_unstale":"2024-02-13T00:50:23.955144+0000","last_undegraded":"2024-02-13T00:50:23.955144+0000","last_fullsized":"2024-02-13T00:50:23.955144+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"236.17","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571021+0000","last_change":"2024-02-13T00:50:22.582920+0000","last_active":"2024-02-13T00:50:23.571021+0000","last_peered":"2024-02-13T00:50:23.571021+0000","last_clean":"2024-02-13T00:50:23.571021+0000","last_became_active":"2024-02-13T00:50:22.582621+0000","last_became_peered":"2024-02-13T00:50:22.582621+0000","last_unstale":"2024-02-13T00:50:23.571021+0000","last_undegraded":"2024-02-13T00:50:23.571021+0000","last_fullsized":"2024-02-13T00:50:23.571021+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"236.16","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956600+0000","last_change":"2024-02-13T00:50:22.583710+0000","last_active":"2024-02-13T00:50:23.956600+0000","last_peered":"2024-02-13T00:50:23.956600+0000","last_clean":"2024-02-13T00:50:23.956600+0000","last_became_active":"2024-02-13T00:50:22.583423+0000","last_became_peered":"2024-02-13T00:50:22.583423+0000","last_unstale":"2024-02-13T00:50:23.956600+0000","last_undegraded":"2024-02-13T00:50:23.956600+0000","last_fullsized":"2024-02-13T00:50:23.956600+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"236.15","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956716+0000","last_change":"2024-02-13T00:50:22.582959+0000","last_active":"2024-02-13T00:50:23.956716+0000","last_peered":"2024-02-13T00:50:23.956716+0000","last_clean":"2024-02-13T00:50:23.956716+0000","last_became_active":"2024-02-13T00:50:22.582673+0000","last_became_peered":"2024-02-13T00:50:22.582673+0000","last_unstale":"2024-02-13T00:50:23.956716+0000","last_undegraded":"2024-02-13T00:50:23.956716+0000","last_fullsized":"2024-02-13T00:50:23.956716+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"236.14","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569060+0000","last_change":"2024-02-13T00:50:22.585095+0000","last_active":"2024-02-13T00:50:23.569060+0000","last_peered":"2024-02-13T00:50:23.569060+0000","last_clean":"2024-02-13T00:50:23.569060+0000","last_became_active":"2024-02-13T00:50:22.584825+0000","last_became_peered":"2024-02-13T00:50:22.584825+0000","last_unstale":"2024-02-13T00:50:23.569060+0000","last_undegraded":"2024-02-13T00:50:23.569060+0000","last_fullsized":"2024-02-13T00:50:23.569060+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.13","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568594+0000","last_change":"2024-02-13T00:50:22.584761+0000","last_active":"2024-02-13T00:50:23.568594+0000","last_peered":"2024-02-13T00:50:23.568594+0000","last_clean":"2024-02-13T00:50:23.568594+0000","last_became_active":"2024-02-13T00:50:22.584363+0000","last_became_peered":"2024-02-13T00:50:22.584363+0000","last_unstale":"2024-02-13T00:50:23.568594+0000","last_undegraded":"2024-02-13T00:50:23.568594+0000","last_fullsized":"2024-02-13T00:50:23.568594+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.12","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568275+0000","last_change":"2024-02-13T00:50:22.583186+0000","last_active":"2024-02-13T00:50:23.568275+0000","last_peered":"2024-02-13T00:50:23.568275+0000","last_clean":"2024-02-13T00:50:23.568275+0000","last_became_active":"2024-02-13T00:50:22.582926+0000","last_became_peered":"2024-02-13T00:50:22.582926+0000","last_unstale":"2024-02-13T00:50:23.568275+0000","last_undegraded":"2024-02-13T00:50:23.568275+0000","last_fullsized":"2024-02-13T00:50:23.568275+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"236.11","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955365+0000","last_change":"2024-02-13T00:50:22.579693+0000","last_active":"2024-02-13T00:50:23.955365+0000","last_peered":"2024-02-13T00:50:23.955365+0000","last_clean":"2024-02-13T00:50:23.955365+0000","last_became_active":"2024-02-13T00:50:22.579421+0000","last_became_peered":"2024-02-13T00:50:22.579421+0000","last_unstale":"2024-02-13T00:50:23.955365+0000","last_undegraded":"2024-02-13T00:50:23.955365+0000","last_fullsized":"2024-02-13T00:50:23.955365+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"236.10","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.567546+0000","last_change":"2024-02-13T00:50:22.583687+0000","last_active":"2024-02-13T00:50:23.567546+0000","last_peered":"2024-02-13T00:50:23.567546+0000","last_clean":"2024-02-13T00:50:23.567546+0000","last_became_active":"2024-02-13T00:50:22.583327+0000","last_became_peered":"2024-02-13T00:50:22.583327+0000","last_unstale":"2024-02-13T00:50:23.567546+0000","last_undegraded":"2024-02-13T00:50:23.567546+0000","last_fullsized":"2024-02-13T00:50:23.567546+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.f","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570308+0000","last_change":"2024-02-13T00:50:22.583990+0000","last_active":"2024-02-13T00:50:23.570308+0000","last_peered":"2024-02-13T00:50:23.570308+0000","last_clean":"2024-02-13T00:50:23.570308+0000","last_became_active":"2024-02-13T00:50:22.583243+0000","last_became_peered":"2024-02-13T00:50:22.583243+0000","last_unstale":"2024-02-13T00:50:23.570308+0000","last_undegraded":"2024-02-13T00:50:23.570308+0000","last_fullsized":"2024-02-13T00:50:23.570308+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.e","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956606+0000","last_change":"2024-02-13T00:50:22.581999+0000","last_active":"2024-02-13T00:50:23.956606+0000","last_peered":"2024-02-13T00:50:23.956606+0000","last_clean":"2024-02-13T00:50:23.956606+0000","last_became_active":"2024-02-13T00:50:22.581754+0000","last_became_peered":"2024-02-13T00:50:22.581754+0000","last_unstale":"2024-02-13T00:50:23.956606+0000","last_undegraded":"2024-02-13T00:50:23.956606+0000","last_fullsized":"2024-02-13T00:50:23.956606+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"236.d","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570590+0000","last_change":"2024-02-13T00:50:22.582125+0000","last_active":"2024-02-13T00:50:23.570590+0000","last_peered":"2024-02-13T00:50:23.570590+0000","last_clean":"2024-02-13T00:50:23.570590+0000","last_became_active":"2024-02-13T00:50:22.581505+0000","last_became_peered":"2024-02-13T00:50:22.581505+0000","last_unstale":"2024-02-13T00:50:23.570590+0000","last_undegraded":"2024-02-13T00:50:23.570590+0000","last_fullsized":"2024-02-13T00:50:23.570590+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"236.c","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956122+0000","last_change":"2024-02-13T00:50:22.582962+0000","last_active":"2024-02-13T00:50:23.956122+0000","last_peered":"2024-02-13T00:50:23.956122+0000","last_clean":"2024-02-13T00:50:23.956122+0000","last_became_active":"2024-02-13T00:50:22.582687+0000","last_became_peered":"2024-02-13T00:50:22.582687+0000","last_unstale":"2024-02-13T00:50:23.956122+0000","last_undegraded":"2024-02-13T00:50:23.956122+0000","last_fullsized":"2024-02-13T00:50:23.956122+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"236.b","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956588+0000","last_change":"2024-02-13T00:50:22.580438+0000","last_active":"2024-02-13T00:50:23.956588+0000","last_peered":"2024-02-13T00:50:23.956588+0000","last_clean":"2024-02-13T00:50:23.956588+0000","last_became_active":"2024-02-13T00:50:22.580230+0000","last_became_peered":"2024-02-13T00:50:22.580230+0000","last_unstale":"2024-02-13T00:50:23.956588+0000","last_undegraded":"2024-02-13T00:50:23.956588+0000","last_fullsized":"2024-02-13T00:50:23.956588+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"236.a","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568341+0000","last_change":"2024-02-13T00:50:22.583188+0000","last_active":"2024-02-13T00:50:23.568341+0000","last_peered":"2024-02-13T00:50:23.568341+0000","last_clean":"2024-02-13T00:50:23.568341+0000","last_became_active":"2024-02-13T00:50:22.582547+0000","last_became_peered":"2024-02-13T00:50:22.582547+0000","last_unstale":"2024-02-13T00:50:23.568341+0000","last_undegraded":"2024-02-13T00:50:23.568341+0000","last_fullsized":"2024-02-13T00:50:23.568341+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"236.9","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.567740+0000","last_change":"2024-02-13T00:50:22.580602+0000","last_active":"2024-02-13T00:50:23.567740+0000","last_peered":"2024-02-13T00:50:23.567740+0000","last_clean":"2024-02-13T00:50:23.567740+0000","last_became_active":"2024-02-13T00:50:22.580335+0000","last_became_peered":"2024-02-13T00:50:22.580335+0000","last_unstale":"2024-02-13T00:50:23.567740+0000","last_undegraded":"2024-02-13T00:50:23.567740+0000","last_fullsized":"2024-02-13T00:50:23.567740+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.8","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.567590+0000","last_change":"2024-02-13T00:50:22.584610+0000","last_active":"2024-02-13T00:50:23.567590+0000","last_peered":"2024-02-13T00:50:23.567590+0000","last_clean":"2024-02-13T00:50:23.567590+0000","last_became_active":"2024-02-13T00:50:22.584063+0000","last_became_peered":"2024-02-13T00:50:22.584063+0000","last_unstale":"2024-02-13T00:50:23.567590+0000","last_undegraded":"2024-02-13T00:50:23.567590+0000","last_fullsized":"2024-02-13T00:50:23.567590+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.1","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955407+0000","last_change":"2024-02-13T00:50:22.579616+0000","last_active":"2024-02-13T00:50:23.955407+0000","last_peered":"2024-02-13T00:50:23.955407+0000","last_clean":"2024-02-13T00:50:23.955407+0000","last_became_active":"2024-02-13T00:50:22.579350+0000","last_became_peered":"2024-02-13T00:50:22.579350+0000","last_unstale":"2024-02-13T00:50:23.955407+0000","last_undegraded":"2024-02-13T00:50:23.955407+0000","last_fullsized":"2024-02-13T00:50:23.955407+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.7","version":"601'1","reported_seq":21,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571148+0000","last_change":"2024-02-13T00:50:14.554059+0000","last_active":"2024-02-13T00:50:23.571148+0000","last_peered":"2024-02-13T00:50:23.571148+0000","last_clean":"2024-02-13T00:50:23.571148+0000","last_became_active":"2024-02-13T00:50:14.553744+0000","last_became_peered":"2024-02-13T00:50:14.553744+0000","last_unstale":"2024-02-13T00:50:23.571148+0000","last_undegraded":"2024-02-13T00:50:23.571148+0000","last_fullsized":"2024-02-13T00:50:23.571148+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"236.0","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955230+0000","last_change":"2024-02-13T00:50:22.579595+0000","last_active":"2024-02-13T00:50:23.955230+0000","last_peered":"2024-02-13T00:50:23.955230+0000","last_clean":"2024-02-13T00:50:23.955230+0000","last_became_active":"2024-02-13T00:50:22.579302+0000","last_became_peered":"2024-02-13T00:50:22.579302+0000","last_unstale":"2024-02-13T00:50:23.955230+0000","last_undegraded":"2024-02-13T00:50:23.955230+0000","last_fullsized":"2024-02-13T00:50:23.955230+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.6","version":"0'0","reported_seq":19,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569602+0000","last_change":"2024-02-13T00:50:14.553031+0000","last_active":"2024-02-13T00:50:23.569602+0000","last_peered":"2024-02-13T00:50:23.569602+0000","last_clean":"2024-02-13T00:50:23.569602+0000","last_became_active":"2024-02-13T00:50:14.552756+0000","last_became_peered":"2024-02-13T00:50:14.552756+0000","last_unstale":"2024-02-13T00:50:23.569602+0000","last_undegraded":"2024-02-13T00:50:23.569602+0000","last_fullsized":"2024-02-13T00:50:23.569602+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.3","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568903+0000","last_change":"2024-02-13T00:50:22.584680+0000","last_active":"2024-02-13T00:50:23.568903+0000","last_peered":"2024-02-13T00:50:23.568903+0000","last_clean":"2024-02-13T00:50:23.568903+0000","last_became_active":"2024-02-13T00:50:22.583912+0000","last_became_peered":"2024-02-13T00:50:22.583912+0000","last_unstale":"2024-02-13T00:50:23.568903+0000","last_undegraded":"2024-02-13T00:50:23.568903+0000","last_fullsized":"2024-02-13T00:50:23.568903+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.5","version":"0'0","reported_seq":19,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569586+0000","last_change":"2024-02-13T00:50:14.551022+0000","last_active":"2024-02-13T00:50:23.569586+0000","last_peered":"2024-02-13T00:50:23.569586+0000","last_clean":"2024-02-13T00:50:23.569586+0000","last_became_active":"2024-02-13T00:50:14.550644+0000","last_became_peered":"2024-02-13T00:50:14.550644+0000","last_unstale":"2024-02-13T00:50:23.569586+0000","last_undegraded":"2024-02-13T00:50:23.569586+0000","last_fullsized":"2024-02-13T00:50:23.569586+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.2","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568610+0000","last_change":"2024-02-13T00:50:22.580030+0000","last_active":"2024-02-13T00:50:23.568610+0000","last_peered":"2024-02-13T00:50:23.568610+0000","last_clean":"2024-02-13T00:50:23.568610+0000","last_became_active":"2024-02-13T00:50:22.579699+0000","last_became_peered":"2024-02-13T00:50:22.579699+0000","last_unstale":"2024-02-13T00:50:23.568610+0000","last_undegraded":"2024-02-13T00:50:23.568610+0000","last_fullsized":"2024-02-13T00:50:23.568610+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.4","version":"0'0","reported_seq":19,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956217+0000","last_change":"2024-02-13T00:50:14.550087+0000","last_active":"2024-02-13T00:50:23.956217+0000","last_peered":"2024-02-13T00:50:23.956217+0000","last_clean":"2024-02-13T00:50:23.956217+0000","last_became_active":"2024-02-13T00:50:14.549625+0000","last_became_peered":"2024-02-13T00:50:14.549625+0000","last_unstale":"2024-02-13T00:50:23.956217+0000","last_undegraded":"2024-02-13T00:50:23.956217+0000","last_fullsized":"2024-02-13T00:50:23.956217+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"236.5","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956811+0000","last_change":"2024-02-13T00:50:22.581683+0000","last_active":"2024-02-13T00:50:23.956811+0000","last_peered":"2024-02-13T00:50:23.956811+0000","last_clean":"2024-02-13T00:50:23.956811+0000","last_became_active":"2024-02-13T00:50:22.581401+0000","last_became_peered":"2024-02-13T00:50:22.581401+0000","last_unstale":"2024-02-13T00:50:23.956811+0000","last_undegraded":"2024-02-13T00:50:23.956811+0000","last_fullsized":"2024-02-13T00:50:23.956811+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.3","version":"0'0","reported_seq":19,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568985+0000","last_change":"2024-02-13T00:50:14.550875+0000","last_active":"2024-02-13T00:50:23.568985+0000","last_peered":"2024-02-13T00:50:23.568985+0000","last_clean":"2024-02-13T00:50:23.568985+0000","last_became_active":"2024-02-13T00:50:14.549053+0000","last_became_peered":"2024-02-13T00:50:14.549053+0000","last_unstale":"2024-02-13T00:50:23.568985+0000","last_undegraded":"2024-02-13T00:50:23.568985+0000","last_fullsized":"2024-02-13T00:50:23.568985+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.4","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569122+0000","last_change":"2024-02-13T00:50:22.580364+0000","last_active":"2024-02-13T00:50:23.569122+0000","last_peered":"2024-02-13T00:50:23.569122+0000","last_clean":"2024-02-13T00:50:23.569122+0000","last_became_active":"2024-02-13T00:50:22.579688+0000","last_became_peered":"2024-02-13T00:50:22.579688+0000","last_unstale":"2024-02-13T00:50:23.569122+0000","last_undegraded":"2024-02-13T00:50:23.569122+0000","last_fullsized":"2024-02-13T00:50:23.569122+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.2","version":"0'0","reported_seq":19,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568102+0000","last_change":"2024-02-13T00:50:14.552504+0000","last_active":"2024-02-13T00:50:23.568102+0000","last_peered":"2024-02-13T00:50:23.568102+0000","last_clean":"2024-02-13T00:50:23.568102+0000","last_became_active":"2024-02-13T00:50:14.552162+0000","last_became_peered":"2024-02-13T00:50:14.552162+0000","last_unstale":"2024-02-13T00:50:23.568102+0000","last_undegraded":"2024-02-13T00:50:23.568102+0000","last_fullsized":"2024-02-13T00:50:23.568102+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"236.7","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956898+0000","last_change":"2024-02-13T00:50:22.579856+0000","last_active":"2024-02-13T00:50:23.956898+0000","last_peered":"2024-02-13T00:50:23.956898+0000","last_clean":"2024-02-13T00:50:23.956898+0000","last_became_active":"2024-02-13T00:50:22.579622+0000","last_became_peered":"2024-02-13T00:50:22.579622+0000","last_unstale":"2024-02-13T00:50:23.956898+0000","last_undegraded":"2024-02-13T00:50:23.956898+0000","last_fullsized":"2024-02-13T00:50:23.956898+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.1","version":"0'0","reported_seq":19,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568388+0000","last_change":"2024-02-13T00:50:14.553804+0000","last_active":"2024-02-13T00:50:23.568388+0000","last_peered":"2024-02-13T00:50:23.568388+0000","last_clean":"2024-02-13T00:50:23.568388+0000","last_became_active":"2024-02-13T00:50:14.553460+0000","last_became_peered":"2024-02-13T00:50:14.553460+0000","last_unstale":"2024-02-13T00:50:23.568388+0000","last_undegraded":"2024-02-13T00:50:23.568388+0000","last_fullsized":"2024-02-13T00:50:23.568388+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"236.6","version":"0'0","reported_seq":11,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956834+0000","last_change":"2024-02-13T00:50:22.581493+0000","last_active":"2024-02-13T00:50:23.956834+0000","last_peered":"2024-02-13T00:50:23.956834+0000","last_clean":"2024-02-13T00:50:23.956834+0000","last_became_active":"2024-02-13T00:50:22.581168+0000","last_became_peered":"2024-02-13T00:50:22.581168+0000","last_unstale":"2024-02-13T00:50:23.956834+0000","last_undegraded":"2024-02-13T00:50:23.956834+0000","last_fullsized":"2024-02-13T00:50:23.956834+0000","mapping_epoch":607,"log_start":"0'0","ondisk_log_start":"0'0","created":607,"last_epoch_clean":608,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:21.552056+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.552056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.0","version":"608'1","reported_seq":20,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.567673+0000","last_change":"2024-02-13T00:50:14.551368+0000","last_active":"2024-02-13T00:50:23.567673+0000","last_peered":"2024-02-13T00:50:23.567673+0000","last_clean":"2024-02-13T00:50:23.567673+0000","last_became_active":"2024-02-13T00:50:14.551016+0000","last_became_peered":"2024-02-13T00:50:14.551016+0000","last_unstale":"2024-02-13T00:50:23.567673+0000","last_undegraded":"2024-02-13T00:50:23.567673+0000","last_fullsized":"2024-02-13T00:50:23.567673+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":510,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955828+0000","last_change":"2024-02-13T00:46:01.448309+0000","last_active":"2024-02-13T00:50:23.955828+0000","last_peered":"2024-02-13T00:50:23.955828+0000","last_clean":"2024-02-13T00:50:23.955828+0000","last_became_active":"2024-02-13T00:46:01.445363+0000","last_became_peered":"2024-02-13T00:46:01.445363+0000","last_unstale":"2024-02-13T00:50:23.955828+0000","last_undegraded":"2024-02-13T00:50:23.955828+0000","last_fullsized":"2024-02-13T00:50:23.955828+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:41.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:41:41.476172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":435,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568260+0000","last_change":"2024-02-13T00:46:01.449894+0000","last_active":"2024-02-13T00:50:23.568260+0000","last_peered":"2024-02-13T00:50:23.568260+0000","last_clean":"2024-02-13T00:50:23.568260+0000","last_became_active":"2024-02-13T00:46:01.449570+0000","last_became_peered":"2024-02-13T00:46:01.449570+0000","last_unstale":"2024-02-13T00:50:23.568260+0000","last_undegraded":"2024-02-13T00:50:23.568260+0000","last_fullsized":"2024-02-13T00:50:23.568260+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":301,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568742+0000","last_change":"2024-02-13T00:46:01.445141+0000","last_active":"2024-02-13T00:50:23.568742+0000","last_peered":"2024-02-13T00:50:23.568742+0000","last_clean":"2024-02-13T00:50:23.568742+0000","last_became_active":"2024-02-13T00:46:01.444729+0000","last_became_peered":"2024-02-13T00:46:01.444729+0000","last_unstale":"2024-02-13T00:50:23.568742+0000","last_undegraded":"2024-02-13T00:50:23.568742+0000","last_fullsized":"2024-02-13T00:50:23.568742+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":391,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568323+0000","last_change":"2024-02-13T00:50:16.677491+0000","last_active":"2024-02-13T00:50:23.568323+0000","last_peered":"2024-02-13T00:50:23.568323+0000","last_clean":"2024-02-13T00:50:23.568323+0000","last_became_active":"2024-02-13T00:45:15.426109+0000","last_became_peered":"2024-02-13T00:45:15.426109+0000","last_unstale":"2024-02-13T00:50:23.568323+0000","last_undegraded":"2024-02-13T00:50:23.568323+0000","last_fullsized":"2024-02-13T00:50:23.568323+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:16.677431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:16.677431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":296,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569670+0000","last_change":"2024-02-13T00:50:22.469511+0000","last_active":"2024-02-13T00:50:23.569670+0000","last_peered":"2024-02-13T00:50:23.569670+0000","last_clean":"2024-02-13T00:50:23.569670+0000","last_became_active":"2024-02-13T00:44:39.525161+0000","last_became_peered":"2024-02-13T00:44:39.525161+0000","last_unstale":"2024-02-13T00:50:23.569670+0000","last_undegraded":"2024-02-13T00:50:23.569670+0000","last_fullsized":"2024-02-13T00:50:23.569670+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.469431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.469431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":389,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568697+0000","last_change":"2024-02-13T00:48:06.705421+0000","last_active":"2024-02-13T00:50:23.568697+0000","last_peered":"2024-02-13T00:50:23.568697+0000","last_clean":"2024-02-13T00:50:23.568697+0000","last_became_active":"2024-02-13T00:48:06.705121+0000","last_became_peered":"2024-02-13T00:48:06.705121+0000","last_unstale":"2024-02-13T00:50:23.568697+0000","last_undegraded":"2024-02-13T00:50:23.568697+0000","last_fullsized":"2024-02-13T00:50:23.568697+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":309,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570677+0000","last_change":"2024-02-13T00:46:01.459772+0000","last_active":"2024-02-13T00:50:23.570677+0000","last_peered":"2024-02-13T00:50:23.570677+0000","last_clean":"2024-02-13T00:50:23.570677+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:50:23.570677+0000","last_undegraded":"2024-02-13T00:50:23.570677+0000","last_fullsized":"2024-02-13T00:50:23.570677+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":568,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569864+0000","last_change":"2024-02-13T00:46:01.463062+0000","last_active":"2024-02-13T00:50:23.569864+0000","last_peered":"2024-02-13T00:50:23.569864+0000","last_clean":"2024-02-13T00:50:23.569864+0000","last_became_active":"2024-02-13T00:46:01.462319+0000","last_became_peered":"2024-02-13T00:46:01.462319+0000","last_unstale":"2024-02-13T00:50:23.569864+0000","last_undegraded":"2024-02-13T00:50:23.569864+0000","last_fullsized":"2024-02-13T00:50:23.569864+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":296,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571519+0000","last_change":"2024-02-13T00:49:48.070260+0000","last_active":"2024-02-13T00:50:23.571519+0000","last_peered":"2024-02-13T00:50:23.571519+0000","last_clean":"2024-02-13T00:50:23.571519+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:50:23.571519+0000","last_undegraded":"2024-02-13T00:50:23.571519+0000","last_fullsized":"2024-02-13T00:50:23.571519+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:48.070195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:49:48.070195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":377,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571432+0000","last_change":"2024-02-13T00:46:01.455629+0000","last_active":"2024-02-13T00:50:23.571432+0000","last_peered":"2024-02-13T00:50:23.571432+0000","last_clean":"2024-02-13T00:50:23.571432+0000","last_became_active":"2024-02-13T00:46:01.452801+0000","last_became_peered":"2024-02-13T00:46:01.452801+0000","last_unstale":"2024-02-13T00:50:23.571432+0000","last_undegraded":"2024-02-13T00:50:23.571432+0000","last_fullsized":"2024-02-13T00:50:23.571432+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":531,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956041+0000","last_change":"2024-02-13T00:45:39.562095+0000","last_active":"2024-02-13T00:50:23.956041+0000","last_peered":"2024-02-13T00:50:23.956041+0000","last_clean":"2024-02-13T00:50:23.956041+0000","last_became_active":"2024-02-13T00:44:39.505121+0000","last_became_peered":"2024-02-13T00:44:39.505121+0000","last_unstale":"2024-02-13T00:50:23.956041+0000","last_undegraded":"2024-02-13T00:50:23.956041+0000","last_fullsized":"2024-02-13T00:50:23.956041+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:39.561994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:39.561994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"534'16","reported_seq":404,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:24.359577+0000","last_change":"2024-02-13T00:49:07.494629+0000","last_active":"2024-02-13T00:50:24.359577+0000","last_peered":"2024-02-13T00:50:24.359577+0000","last_clean":"2024-02-13T00:50:24.359577+0000","last_became_active":"2024-02-13T00:49:07.494388+0000","last_became_peered":"2024-02-13T00:49:07.494388+0000","last_unstale":"2024-02-13T00:50:24.359577+0000","last_undegraded":"2024-02-13T00:50:24.359577+0000","last_fullsized":"2024-02-13T00:50:24.359577+0000","mapping_epoch":359,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:45:53.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.115260+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":635,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568108+0000","last_change":"2024-02-13T00:47:03.491720+0000","last_active":"2024-02-13T00:50:23.568108+0000","last_peered":"2024-02-13T00:50:23.568108+0000","last_clean":"2024-02-13T00:50:23.568108+0000","last_became_active":"2024-02-13T00:46:01.438056+0000","last_became_peered":"2024-02-13T00:46:01.438056+0000","last_unstale":"2024-02-13T00:50:23.568108+0000","last_undegraded":"2024-02-13T00:50:23.568108+0000","last_fullsized":"2024-02-13T00:50:23.568108+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:47:03.491647+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":605,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570154+0000","last_change":"2024-02-13T00:48:23.379872+0000","last_active":"2024-02-13T00:50:23.570154+0000","last_peered":"2024-02-13T00:50:23.570154+0000","last_clean":"2024-02-13T00:50:23.570154+0000","last_became_active":"2024-02-13T00:46:01.460811+0000","last_became_peered":"2024-02-13T00:46:01.460811+0000","last_unstale":"2024-02-13T00:50:23.570154+0000","last_undegraded":"2024-02-13T00:50:23.570154+0000","last_fullsized":"2024-02-13T00:50:23.570154+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:23.379773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:23.379773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":511,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568836+0000","last_change":"2024-02-13T00:47:34.332791+0000","last_active":"2024-02-13T00:50:23.568836+0000","last_peered":"2024-02-13T00:50:23.568836+0000","last_clean":"2024-02-13T00:50:23.568836+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:50:23.568836+0000","last_undegraded":"2024-02-13T00:50:23.568836+0000","last_fullsized":"2024-02-13T00:50:23.568836+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:34.332704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:34.332704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":549,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569408+0000","last_change":"2024-02-13T00:46:01.448237+0000","last_active":"2024-02-13T00:50:23.569408+0000","last_peered":"2024-02-13T00:50:23.569408+0000","last_clean":"2024-02-13T00:50:23.569408+0000","last_became_active":"2024-02-13T00:46:01.446660+0000","last_became_peered":"2024-02-13T00:46:01.446660+0000","last_unstale":"2024-02-13T00:50:23.569408+0000","last_undegraded":"2024-02-13T00:50:23.569408+0000","last_fullsized":"2024-02-13T00:50:23.569408+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.260871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.260871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":610,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956686+0000","last_change":"2024-02-13T00:47:41.808333+0000","last_active":"2024-02-13T00:50:23.956686+0000","last_peered":"2024-02-13T00:50:23.956686+0000","last_clean":"2024-02-13T00:50:23.956686+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:50:23.956686+0000","last_undegraded":"2024-02-13T00:50:23.956686+0000","last_fullsized":"2024-02-13T00:50:23.956686+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:47:41.808259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:41.808259+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":592,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571089+0000","last_change":"2024-02-13T00:48:24.823190+0000","last_active":"2024-02-13T00:50:23.571089+0000","last_peered":"2024-02-13T00:50:23.571089+0000","last_clean":"2024-02-13T00:50:23.571089+0000","last_became_active":"2024-02-13T00:44:50.429994+0000","last_became_peered":"2024-02-13T00:44:50.429994+0000","last_unstale":"2024-02-13T00:50:23.571089+0000","last_undegraded":"2024-02-13T00:50:23.571089+0000","last_fullsized":"2024-02-13T00:50:23.571089+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:24.823107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.823107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":494,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956271+0000","last_change":"2024-02-13T00:49:39.788846+0000","last_active":"2024-02-13T00:50:23.956271+0000","last_peered":"2024-02-13T00:50:23.956271+0000","last_clean":"2024-02-13T00:50:23.956271+0000","last_became_active":"2024-02-13T00:49:03.971407+0000","last_became_peered":"2024-02-13T00:49:03.971407+0000","last_unstale":"2024-02-13T00:50:23.956271+0000","last_undegraded":"2024-02-13T00:50:23.956271+0000","last_fullsized":"2024-02-13T00:50:23.956271+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.788783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.788783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":593,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956150+0000","last_change":"2024-02-13T00:49:41.837871+0000","last_active":"2024-02-13T00:50:23.956150+0000","last_peered":"2024-02-13T00:50:23.956150+0000","last_clean":"2024-02-13T00:50:23.956150+0000","last_became_active":"2024-02-13T00:45:15.434853+0000","last_became_peered":"2024-02-13T00:45:15.434853+0000","last_unstale":"2024-02-13T00:50:23.956150+0000","last_undegraded":"2024-02-13T00:50:23.956150+0000","last_fullsized":"2024-02-13T00:50:23.956150+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.837810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.837810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":214,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955912+0000","last_change":"2024-02-13T00:46:01.455201+0000","last_active":"2024-02-13T00:50:23.955912+0000","last_peered":"2024-02-13T00:50:23.955912+0000","last_clean":"2024-02-13T00:50:23.955912+0000","last_became_active":"2024-02-13T00:46:01.454882+0000","last_became_peered":"2024-02-13T00:46:01.454882+0000","last_unstale":"2024-02-13T00:50:23.955912+0000","last_undegraded":"2024-02-13T00:50:23.955912+0000","last_fullsized":"2024-02-13T00:50:23.955912+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:35.197045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:35.197045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":399,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569281+0000","last_change":"2024-02-13T00:46:09.256189+0000","last_active":"2024-02-13T00:50:23.569281+0000","last_peered":"2024-02-13T00:50:23.569281+0000","last_clean":"2024-02-13T00:50:23.569281+0000","last_became_active":"2024-02-13T00:46:01.459978+0000","last_became_peered":"2024-02-13T00:46:01.459978+0000","last_unstale":"2024-02-13T00:50:23.569281+0000","last_undegraded":"2024-02-13T00:50:23.569281+0000","last_fullsized":"2024-02-13T00:50:23.569281+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:44:24.196214+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:44:24.196214+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":224,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.957040+0000","last_change":"2024-02-13T00:46:30.641539+0000","last_active":"2024-02-13T00:50:23.957040+0000","last_peered":"2024-02-13T00:50:23.957040+0000","last_clean":"2024-02-13T00:50:23.957040+0000","last_became_active":"2024-02-13T00:46:01.444548+0000","last_became_peered":"2024-02-13T00:46:01.444548+0000","last_unstale":"2024-02-13T00:50:23.957040+0000","last_undegraded":"2024-02-13T00:50:23.957040+0000","last_fullsized":"2024-02-13T00:50:23.957040+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.641460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.641460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":324,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956315+0000","last_change":"2024-02-13T00:48:54.498301+0000","last_active":"2024-02-13T00:50:23.956315+0000","last_peered":"2024-02-13T00:50:23.956315+0000","last_clean":"2024-02-13T00:50:23.956315+0000","last_became_active":"2024-02-13T00:48:54.497977+0000","last_became_peered":"2024-02-13T00:48:54.497977+0000","last_unstale":"2024-02-13T00:50:23.956315+0000","last_undegraded":"2024-02-13T00:50:23.956315+0000","last_fullsized":"2024-02-13T00:50:23.956315+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.592672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":427,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568255+0000","last_change":"2024-02-13T00:45:15.429026+0000","last_active":"2024-02-13T00:50:23.568255+0000","last_peered":"2024-02-13T00:50:23.568255+0000","last_clean":"2024-02-13T00:50:23.568255+0000","last_became_active":"2024-02-13T00:45:15.427945+0000","last_became_peered":"2024-02-13T00:45:15.427945+0000","last_unstale":"2024-02-13T00:50:23.568255+0000","last_undegraded":"2024-02-13T00:50:23.568255+0000","last_fullsized":"2024-02-13T00:50:23.568255+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":532,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570955+0000","last_change":"2024-02-13T00:45:26.255907+0000","last_active":"2024-02-13T00:50:23.570955+0000","last_peered":"2024-02-13T00:50:23.570955+0000","last_clean":"2024-02-13T00:50:23.570955+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:50:23.570955+0000","last_undegraded":"2024-02-13T00:50:23.570955+0000","last_fullsized":"2024-02-13T00:50:23.570955+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.255842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.255842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"467'31","reported_seq":369,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571024+0000","last_change":"2024-02-13T00:48:31.782821+0000","last_active":"2024-02-13T00:50:23.571024+0000","last_peered":"2024-02-13T00:50:23.571024+0000","last_clean":"2024-02-13T00:50:23.571024+0000","last_became_active":"2024-02-13T00:44:50.432807+0000","last_became_peered":"2024-02-13T00:44:50.432807+0000","last_unstale":"2024-02-13T00:50:23.571024+0000","last_undegraded":"2024-02-13T00:50:23.571024+0000","last_fullsized":"2024-02-13T00:50:23.571024+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'31","last_scrub_stamp":"2024-02-13T00:48:31.782729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.782729+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":382,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569217+0000","last_change":"2024-02-13T00:46:04.708510+0000","last_active":"2024-02-13T00:50:23.569217+0000","last_peered":"2024-02-13T00:50:23.569217+0000","last_clean":"2024-02-13T00:50:23.569217+0000","last_became_active":"2024-02-13T00:46:01.447801+0000","last_became_peered":"2024-02-13T00:46:01.447801+0000","last_unstale":"2024-02-13T00:50:23.569217+0000","last_undegraded":"2024-02-13T00:50:23.569217+0000","last_fullsized":"2024-02-13T00:50:23.569217+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:45:24.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:24.269595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":590,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570391+0000","last_change":"2024-02-13T00:48:49.612735+0000","last_active":"2024-02-13T00:50:23.570391+0000","last_peered":"2024-02-13T00:50:23.570391+0000","last_clean":"2024-02-13T00:50:23.570391+0000","last_became_active":"2024-02-13T00:48:49.612437+0000","last_became_peered":"2024-02-13T00:48:49.612437+0000","last_unstale":"2024-02-13T00:50:23.570391+0000","last_undegraded":"2024-02-13T00:50:23.570391+0000","last_fullsized":"2024-02-13T00:50:23.570391+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.793634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":576,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570967+0000","last_change":"2024-02-13T00:44:50.441408+0000","last_active":"2024-02-13T00:50:23.570967+0000","last_peered":"2024-02-13T00:50:23.570967+0000","last_clean":"2024-02-13T00:50:23.570967+0000","last_became_active":"2024-02-13T00:44:50.440301+0000","last_became_peered":"2024-02-13T00:44:50.440301+0000","last_unstale":"2024-02-13T00:50:23.570967+0000","last_undegraded":"2024-02-13T00:50:23.570967+0000","last_fullsized":"2024-02-13T00:50:23.570967+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:25.502311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.502311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":236,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956668+0000","last_change":"2024-02-13T00:50:22.925676+0000","last_active":"2024-02-13T00:50:23.956668+0000","last_peered":"2024-02-13T00:50:23.956668+0000","last_clean":"2024-02-13T00:50:23.956668+0000","last_became_active":"2024-02-13T00:48:06.704718+0000","last_became_peered":"2024-02-13T00:48:06.704718+0000","last_unstale":"2024-02-13T00:50:23.956668+0000","last_undegraded":"2024-02-13T00:50:23.956668+0000","last_fullsized":"2024-02-13T00:50:23.956668+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.925604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.925604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":323,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570066+0000","last_change":"2024-02-13T00:49:03.294585+0000","last_active":"2024-02-13T00:50:23.570066+0000","last_peered":"2024-02-13T00:50:23.570066+0000","last_clean":"2024-02-13T00:50:23.570066+0000","last_became_active":"2024-02-13T00:44:39.523138+0000","last_became_peered":"2024-02-13T00:44:39.523138+0000","last_unstale":"2024-02-13T00:50:23.570066+0000","last_undegraded":"2024-02-13T00:50:23.570066+0000","last_fullsized":"2024-02-13T00:50:23.570066+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:49:03.294499+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:49:03.294499+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":601,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569304+0000","last_change":"2024-02-13T00:48:59.111910+0000","last_active":"2024-02-13T00:50:23.569304+0000","last_peered":"2024-02-13T00:50:23.569304+0000","last_clean":"2024-02-13T00:50:23.569304+0000","last_became_active":"2024-02-13T00:46:01.452674+0000","last_became_peered":"2024-02-13T00:46:01.452674+0000","last_unstale":"2024-02-13T00:50:23.569304+0000","last_undegraded":"2024-02-13T00:50:23.569304+0000","last_fullsized":"2024-02-13T00:50:23.569304+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:59.111828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:59.111828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"511'13","reported_seq":319,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:24.359459+0000","last_change":"2024-02-13T00:48:45.592523+0000","last_active":"2024-02-13T00:50:24.359459+0000","last_peered":"2024-02-13T00:50:24.359459+0000","last_clean":"2024-02-13T00:50:24.359459+0000","last_became_active":"2024-02-13T00:48:45.591249+0000","last_became_peered":"2024-02-13T00:48:45.591249+0000","last_unstale":"2024-02-13T00:50:24.359459+0000","last_undegraded":"2024-02-13T00:50:24.359459+0000","last_fullsized":"2024-02-13T00:50:24.359459+0000","mapping_epoch":401,"log_start":"404'11","ondisk_log_start":"404'11","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'11","last_scrub_stamp":"2024-02-13T00:46:37.662413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.662413+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":568,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571591+0000","last_change":"2024-02-13T00:46:05.839833+0000","last_active":"2024-02-13T00:50:23.571591+0000","last_peered":"2024-02-13T00:50:23.571591+0000","last_clean":"2024-02-13T00:50:23.571591+0000","last_became_active":"2024-02-13T00:46:01.455274+0000","last_became_peered":"2024-02-13T00:46:01.455274+0000","last_unstale":"2024-02-13T00:50:23.571591+0000","last_undegraded":"2024-02-13T00:50:23.571591+0000","last_fullsized":"2024-02-13T00:50:23.571591+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:44:25.230980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.230980+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":377,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571463+0000","last_change":"2024-02-13T00:46:01.457421+0000","last_active":"2024-02-13T00:50:23.571463+0000","last_peered":"2024-02-13T00:50:23.571463+0000","last_clean":"2024-02-13T00:50:23.571463+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:50:23.571463+0000","last_undegraded":"2024-02-13T00:50:23.571463+0000","last_fullsized":"2024-02-13T00:50:23.571463+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:18.089252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:18.089252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"467'27","reported_seq":690,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571433+0000","last_change":"2024-02-13T00:48:54.043163+0000","last_active":"2024-02-13T00:50:23.571433+0000","last_peered":"2024-02-13T00:50:23.571433+0000","last_clean":"2024-02-13T00:50:23.571433+0000","last_became_active":"2024-02-13T00:46:01.448705+0000","last_became_peered":"2024-02-13T00:46:01.448705+0000","last_unstale":"2024-02-13T00:50:23.571433+0000","last_undegraded":"2024-02-13T00:50:23.571433+0000","last_fullsized":"2024-02-13T00:50:23.571433+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:48:54.043088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.043088+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":633,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569102+0000","last_change":"2024-02-13T00:49:37.053462+0000","last_active":"2024-02-13T00:50:23.569102+0000","last_peered":"2024-02-13T00:50:23.569102+0000","last_clean":"2024-02-13T00:50:23.569102+0000","last_became_active":"2024-02-13T00:45:15.445228+0000","last_became_peered":"2024-02-13T00:45:15.445228+0000","last_unstale":"2024-02-13T00:50:23.569102+0000","last_undegraded":"2024-02-13T00:50:23.569102+0000","last_fullsized":"2024-02-13T00:50:23.569102+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:49:37.053398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.053398+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"507'13","reported_seq":317,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:24.359328+0000","last_change":"2024-02-13T00:48:41.526587+0000","last_active":"2024-02-13T00:50:24.359328+0000","last_peered":"2024-02-13T00:50:24.359328+0000","last_clean":"2024-02-13T00:50:24.359328+0000","last_became_active":"2024-02-13T00:48:41.525262+0000","last_became_peered":"2024-02-13T00:48:41.525262+0000","last_unstale":"2024-02-13T00:50:24.359328+0000","last_undegraded":"2024-02-13T00:50:24.359328+0000","last_fullsized":"2024-02-13T00:50:24.359328+0000","mapping_epoch":401,"log_start":"467'11","ondisk_log_start":"467'11","created":60,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:47:49.687727+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":223,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570845+0000","last_change":"2024-02-13T00:50:19.734411+0000","last_active":"2024-02-13T00:50:23.570845+0000","last_peered":"2024-02-13T00:50:23.570845+0000","last_clean":"2024-02-13T00:50:23.570845+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:50:23.570845+0000","last_undegraded":"2024-02-13T00:50:23.570845+0000","last_fullsized":"2024-02-13T00:50:23.570845+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:19.734320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:50:19.734320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":531,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.958181+0000","last_change":"2024-02-13T00:46:01.456531+0000","last_active":"2024-02-13T00:50:23.958181+0000","last_peered":"2024-02-13T00:50:23.958181+0000","last_clean":"2024-02-13T00:50:23.958181+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:50:23.958181+0000","last_undegraded":"2024-02-13T00:50:23.958181+0000","last_fullsized":"2024-02-13T00:50:23.958181+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:44:59.963616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":322,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570412+0000","last_change":"2024-02-13T00:48:57.386335+0000","last_active":"2024-02-13T00:50:23.570412+0000","last_peered":"2024-02-13T00:50:23.570412+0000","last_clean":"2024-02-13T00:50:23.570412+0000","last_became_active":"2024-02-13T00:44:39.521984+0000","last_became_peered":"2024-02-13T00:44:39.521984+0000","last_unstale":"2024-02-13T00:50:23.570412+0000","last_undegraded":"2024-02-13T00:50:23.570412+0000","last_fullsized":"2024-02-13T00:50:23.570412+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:57.386254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:57.386254+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":414,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955846+0000","last_change":"2024-02-13T00:46:23.757417+0000","last_active":"2024-02-13T00:50:23.955846+0000","last_peered":"2024-02-13T00:50:23.955846+0000","last_clean":"2024-02-13T00:50:23.955846+0000","last_became_active":"2024-02-13T00:45:15.419239+0000","last_became_peered":"2024-02-13T00:45:15.419239+0000","last_unstale":"2024-02-13T00:50:23.955846+0000","last_undegraded":"2024-02-13T00:50:23.955846+0000","last_fullsized":"2024-02-13T00:50:23.955846+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:23.757351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:23.757351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"504'16","reported_seq":557,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:24.359571+0000","last_change":"2024-02-13T00:49:45.051624+0000","last_active":"2024-02-13T00:50:24.359571+0000","last_peered":"2024-02-13T00:50:24.359571+0000","last_clean":"2024-02-13T00:50:24.359571+0000","last_became_active":"2024-02-13T00:48:37.480010+0000","last_became_peered":"2024-02-13T00:48:37.480010+0000","last_unstale":"2024-02-13T00:50:24.359571+0000","last_undegraded":"2024-02-13T00:50:24.359571+0000","last_fullsized":"2024-02-13T00:50:24.359571+0000","mapping_epoch":401,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'16","last_scrub_stamp":"2024-02-13T00:49:45.051539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:49:45.051539+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":314,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569925+0000","last_change":"2024-02-13T00:48:31.463534+0000","last_active":"2024-02-13T00:50:23.569925+0000","last_peered":"2024-02-13T00:50:23.569925+0000","last_clean":"2024-02-13T00:50:23.569925+0000","last_became_active":"2024-02-13T00:46:01.462717+0000","last_became_peered":"2024-02-13T00:46:01.462717+0000","last_unstale":"2024-02-13T00:50:23.569925+0000","last_undegraded":"2024-02-13T00:50:23.569925+0000","last_fullsized":"2024-02-13T00:50:23.569925+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:31.463442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.463442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":488,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955629+0000","last_change":"2024-02-13T00:49:00.881498+0000","last_active":"2024-02-13T00:50:23.955629+0000","last_peered":"2024-02-13T00:50:23.955629+0000","last_clean":"2024-02-13T00:50:23.955629+0000","last_became_active":"2024-02-13T00:46:01.456959+0000","last_became_peered":"2024-02-13T00:46:01.456959+0000","last_unstale":"2024-02-13T00:50:23.955629+0000","last_undegraded":"2024-02-13T00:50:23.955629+0000","last_fullsized":"2024-02-13T00:50:23.955629+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:00.881431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:00.881431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"263'10","reported_seq":612,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:24.359297+0000","last_change":"2024-02-13T00:47:02.949483+0000","last_active":"2024-02-13T00:50:24.359297+0000","last_peered":"2024-02-13T00:50:24.359297+0000","last_clean":"2024-02-13T00:50:24.359297+0000","last_became_active":"2024-02-13T00:43:35.454087+0000","last_became_peered":"2024-02-13T00:43:35.454087+0000","last_unstale":"2024-02-13T00:50:24.359297+0000","last_undegraded":"2024-02-13T00:50:24.359297+0000","last_fullsized":"2024-02-13T00:50:24.359297+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.949406+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":586,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568462+0000","last_change":"2024-02-13T00:49:46.101991+0000","last_active":"2024-02-13T00:50:23.568462+0000","last_peered":"2024-02-13T00:50:23.568462+0000","last_clean":"2024-02-13T00:50:23.568462+0000","last_became_active":"2024-02-13T00:46:01.449654+0000","last_became_peered":"2024-02-13T00:46:01.449654+0000","last_unstale":"2024-02-13T00:50:23.568462+0000","last_undegraded":"2024-02-13T00:50:23.568462+0000","last_fullsized":"2024-02-13T00:50:23.568462+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'34","last_scrub_stamp":"2024-02-13T00:49:46.101874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.101874+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":314,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571445+0000","last_change":"2024-02-13T00:49:47.075845+0000","last_active":"2024-02-13T00:50:23.571445+0000","last_peered":"2024-02-13T00:50:23.571445+0000","last_clean":"2024-02-13T00:50:23.571445+0000","last_became_active":"2024-02-13T00:45:15.446443+0000","last_became_peered":"2024-02-13T00:45:15.446443+0000","last_unstale":"2024-02-13T00:50:23.571445+0000","last_undegraded":"2024-02-13T00:50:23.571445+0000","last_fullsized":"2024-02-13T00:50:23.571445+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:47.075741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.075741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":309,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569904+0000","last_change":"2024-02-13T00:45:15.436286+0000","last_active":"2024-02-13T00:50:23.569904+0000","last_peered":"2024-02-13T00:50:23.569904+0000","last_clean":"2024-02-13T00:50:23.569904+0000","last_became_active":"2024-02-13T00:45:15.434559+0000","last_became_peered":"2024-02-13T00:45:15.434559+0000","last_unstale":"2024-02-13T00:50:23.569904+0000","last_undegraded":"2024-02-13T00:50:23.569904+0000","last_fullsized":"2024-02-13T00:50:23.569904+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":338,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955704+0000","last_change":"2024-02-13T00:49:36.841674+0000","last_active":"2024-02-13T00:50:23.955704+0000","last_peered":"2024-02-13T00:50:23.955704+0000","last_clean":"2024-02-13T00:50:23.955704+0000","last_became_active":"2024-02-13T00:46:01.444661+0000","last_became_peered":"2024-02-13T00:46:01.444661+0000","last_unstale":"2024-02-13T00:50:23.955704+0000","last_undegraded":"2024-02-13T00:50:23.955704+0000","last_fullsized":"2024-02-13T00:50:23.955704+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:36.841508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:36.841508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":379,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955791+0000","last_change":"2024-02-13T00:48:18.751148+0000","last_active":"2024-02-13T00:50:23.955791+0000","last_peered":"2024-02-13T00:50:23.955791+0000","last_clean":"2024-02-13T00:50:23.955791+0000","last_became_active":"2024-02-13T00:46:01.453842+0000","last_became_peered":"2024-02-13T00:46:01.453842+0000","last_unstale":"2024-02-13T00:50:23.955791+0000","last_undegraded":"2024-02-13T00:50:23.955791+0000","last_fullsized":"2024-02-13T00:50:23.955791+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:18.751081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:18.751081+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"467'42","reported_seq":540,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955748+0000","last_change":"2024-02-13T00:48:24.675807+0000","last_active":"2024-02-13T00:50:23.955748+0000","last_peered":"2024-02-13T00:50:23.955748+0000","last_clean":"2024-02-13T00:50:23.955748+0000","last_became_active":"2024-02-13T00:46:01.445058+0000","last_became_peered":"2024-02-13T00:46:01.445058+0000","last_unstale":"2024-02-13T00:50:23.955748+0000","last_undegraded":"2024-02-13T00:50:23.955748+0000","last_fullsized":"2024-02-13T00:50:23.955748+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'42","last_scrub_stamp":"2024-02-13T00:48:24.675687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.675687+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":601,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571431+0000","last_change":"2024-02-13T00:47:09.940673+0000","last_active":"2024-02-13T00:50:23.571431+0000","last_peered":"2024-02-13T00:50:23.571431+0000","last_clean":"2024-02-13T00:50:23.571431+0000","last_became_active":"2024-02-13T00:44:39.503590+0000","last_became_peered":"2024-02-13T00:44:39.503590+0000","last_unstale":"2024-02-13T00:50:23.571431+0000","last_undegraded":"2024-02-13T00:50:23.571431+0000","last_fullsized":"2024-02-13T00:50:23.571431+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:09.940605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:09.940605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":544,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568825+0000","last_change":"2024-02-13T00:46:01.448313+0000","last_active":"2024-02-13T00:50:23.568825+0000","last_peered":"2024-02-13T00:50:23.568825+0000","last_clean":"2024-02-13T00:50:23.568825+0000","last_became_active":"2024-02-13T00:46:01.447815+0000","last_became_peered":"2024-02-13T00:46:01.447815+0000","last_unstale":"2024-02-13T00:50:23.568825+0000","last_undegraded":"2024-02-13T00:50:23.568825+0000","last_fullsized":"2024-02-13T00:50:23.568825+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":536,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569385+0000","last_change":"2024-02-13T00:48:35.434713+0000","last_active":"2024-02-13T00:50:23.569385+0000","last_peered":"2024-02-13T00:50:23.569385+0000","last_clean":"2024-02-13T00:50:23.569385+0000","last_became_active":"2024-02-13T00:44:50.436058+0000","last_became_peered":"2024-02-13T00:44:50.436058+0000","last_unstale":"2024-02-13T00:50:23.569385+0000","last_undegraded":"2024-02-13T00:50:23.569385+0000","last_fullsized":"2024-02-13T00:50:23.569385+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_clean_scrub_stamp":"2024-02-13T00:48:35.434645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":305,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569275+0000","last_change":"2024-02-13T00:46:01.451625+0000","last_active":"2024-02-13T00:50:23.569275+0000","last_peered":"2024-02-13T00:50:23.569275+0000","last_clean":"2024-02-13T00:50:23.569275+0000","last_became_active":"2024-02-13T00:46:01.450587+0000","last_became_peered":"2024-02-13T00:46:01.450587+0000","last_unstale":"2024-02-13T00:50:23.569275+0000","last_undegraded":"2024-02-13T00:50:23.569275+0000","last_fullsized":"2024-02-13T00:50:23.569275+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:33.204969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:33.204969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":703,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956412+0000","last_change":"2024-02-13T00:47:05.718055+0000","last_active":"2024-02-13T00:50:23.956412+0000","last_peered":"2024-02-13T00:50:23.956412+0000","last_clean":"2024-02-13T00:50:23.956412+0000","last_became_active":"2024-02-13T00:46:01.443777+0000","last_became_peered":"2024-02-13T00:46:01.443777+0000","last_unstale":"2024-02-13T00:50:23.956412+0000","last_undegraded":"2024-02-13T00:50:23.956412+0000","last_fullsized":"2024-02-13T00:50:23.956412+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:47:05.717987+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:47:05.717987+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":419,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-02-13T00:50:25.001319+0000","last_change":"2024-02-13T00:50:25.001319+0000","last_active":"2024-02-13T00:50:25.001319+0000","last_peered":"2024-02-13T00:50:25.001319+0000","last_clean":"2024-02-13T00:50:25.001319+0000","last_became_active":"2024-02-13T00:45:15.423415+0000","last_became_peered":"2024-02-13T00:45:15.423415+0000","last_unstale":"2024-02-13T00:50:25.001319+0000","last_undegraded":"2024-02-13T00:50:25.001319+0000","last_fullsized":"2024-02-13T00:50:25.001319+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:25.001219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:25.001219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":613,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569116+0000","last_change":"2024-02-13T00:50:14.726735+0000","last_active":"2024-02-13T00:50:23.569116+0000","last_peered":"2024-02-13T00:50:23.569116+0000","last_clean":"2024-02-13T00:50:23.569116+0000","last_became_active":"2024-02-13T00:45:15.445557+0000","last_became_peered":"2024-02-13T00:45:15.445557+0000","last_unstale":"2024-02-13T00:50:23.569116+0000","last_undegraded":"2024-02-13T00:50:23.569116+0000","last_fullsized":"2024-02-13T00:50:23.569116+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.726660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.726660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":226,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571303+0000","last_change":"2024-02-13T00:48:55.050483+0000","last_active":"2024-02-13T00:50:23.571303+0000","last_peered":"2024-02-13T00:50:23.571303+0000","last_clean":"2024-02-13T00:50:23.571303+0000","last_became_active":"2024-02-13T00:46:01.441421+0000","last_became_peered":"2024-02-13T00:46:01.441421+0000","last_unstale":"2024-02-13T00:50:23.571303+0000","last_undegraded":"2024-02-13T00:50:23.571303+0000","last_fullsized":"2024-02-13T00:50:23.571303+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:55.050411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.050411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":241,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571691+0000","last_change":"2024-02-13T00:49:16.505082+0000","last_active":"2024-02-13T00:50:23.571691+0000","last_peered":"2024-02-13T00:50:23.571691+0000","last_clean":"2024-02-13T00:50:23.571691+0000","last_became_active":"2024-02-13T00:49:16.504828+0000","last_became_peered":"2024-02-13T00:49:16.504828+0000","last_unstale":"2024-02-13T00:50:23.571691+0000","last_undegraded":"2024-02-13T00:50:23.571691+0000","last_fullsized":"2024-02-13T00:50:23.571691+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:43.903553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:43.903553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":698,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.957982+0000","last_change":"2024-02-13T00:46:34.691109+0000","last_active":"2024-02-13T00:50:23.957982+0000","last_peered":"2024-02-13T00:50:23.957982+0000","last_clean":"2024-02-13T00:50:23.957982+0000","last_became_active":"2024-02-13T00:46:01.448163+0000","last_became_peered":"2024-02-13T00:46:01.448163+0000","last_unstale":"2024-02-13T00:50:23.957982+0000","last_undegraded":"2024-02-13T00:50:23.957982+0000","last_fullsized":"2024-02-13T00:50:23.957982+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:46:34.690980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.690980+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":309,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568588+0000","last_change":"2024-02-13T00:46:37.056299+0000","last_active":"2024-02-13T00:50:23.568588+0000","last_peered":"2024-02-13T00:50:23.568588+0000","last_clean":"2024-02-13T00:50:23.568588+0000","last_became_active":"2024-02-13T00:46:01.446104+0000","last_became_peered":"2024-02-13T00:46:01.446104+0000","last_unstale":"2024-02-13T00:50:23.568588+0000","last_undegraded":"2024-02-13T00:50:23.568588+0000","last_fullsized":"2024-02-13T00:50:23.568588+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:37.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":590,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955918+0000","last_change":"2024-02-13T00:46:01.431899+0000","last_active":"2024-02-13T00:50:23.955918+0000","last_peered":"2024-02-13T00:50:23.955918+0000","last_clean":"2024-02-13T00:50:23.955918+0000","last_became_active":"2024-02-13T00:46:01.430623+0000","last_became_peered":"2024-02-13T00:46:01.430623+0000","last_unstale":"2024-02-13T00:50:23.955918+0000","last_undegraded":"2024-02-13T00:50:23.955918+0000","last_fullsized":"2024-02-13T00:50:23.955918+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:46.556229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.556229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":599,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570408+0000","last_change":"2024-02-13T00:46:01.450548+0000","last_active":"2024-02-13T00:50:23.570408+0000","last_peered":"2024-02-13T00:50:23.570408+0000","last_clean":"2024-02-13T00:50:23.570408+0000","last_became_active":"2024-02-13T00:46:01.450176+0000","last_became_peered":"2024-02-13T00:46:01.450176+0000","last_unstale":"2024-02-13T00:50:23.570408+0000","last_undegraded":"2024-02-13T00:50:23.570408+0000","last_fullsized":"2024-02-13T00:50:23.570408+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":451,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570847+0000","last_change":"2024-02-13T00:46:01.460470+0000","last_active":"2024-02-13T00:50:23.570847+0000","last_peered":"2024-02-13T00:50:23.570847+0000","last_clean":"2024-02-13T00:50:23.570847+0000","last_became_active":"2024-02-13T00:46:01.459859+0000","last_became_peered":"2024-02-13T00:46:01.459859+0000","last_unstale":"2024-02-13T00:50:23.570847+0000","last_undegraded":"2024-02-13T00:50:23.570847+0000","last_fullsized":"2024-02-13T00:50:23.570847+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:01.179031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:01.179031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":255,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956948+0000","last_change":"2024-02-13T00:47:33.932830+0000","last_active":"2024-02-13T00:50:23.956948+0000","last_peered":"2024-02-13T00:50:23.956948+0000","last_clean":"2024-02-13T00:50:23.956948+0000","last_became_active":"2024-02-13T00:46:01.455170+0000","last_became_peered":"2024-02-13T00:46:01.455170+0000","last_unstale":"2024-02-13T00:50:23.956948+0000","last_undegraded":"2024-02-13T00:50:23.956948+0000","last_fullsized":"2024-02-13T00:50:23.956948+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:33.932766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:33.932766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":355,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570881+0000","last_change":"2024-02-13T00:46:57.381602+0000","last_active":"2024-02-13T00:50:23.570881+0000","last_peered":"2024-02-13T00:50:23.570881+0000","last_clean":"2024-02-13T00:50:23.570881+0000","last_became_active":"2024-02-13T00:45:15.446969+0000","last_became_peered":"2024-02-13T00:45:15.446969+0000","last_unstale":"2024-02-13T00:50:23.570881+0000","last_undegraded":"2024-02-13T00:50:23.570881+0000","last_fullsized":"2024-02-13T00:50:23.570881+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:46:57.381540+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:46:57.381540+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":555,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955651+0000","last_change":"2024-02-13T00:47:39.822745+0000","last_active":"2024-02-13T00:50:23.955651+0000","last_peered":"2024-02-13T00:50:23.955651+0000","last_clean":"2024-02-13T00:50:23.955651+0000","last_became_active":"2024-02-13T00:45:15.434498+0000","last_became_peered":"2024-02-13T00:45:15.434498+0000","last_unstale":"2024-02-13T00:50:23.955651+0000","last_undegraded":"2024-02-13T00:50:23.955651+0000","last_fullsized":"2024-02-13T00:50:23.955651+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:39.822619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:47:39.822619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":553,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569190+0000","last_change":"2024-02-13T00:47:04.493911+0000","last_active":"2024-02-13T00:50:23.569190+0000","last_peered":"2024-02-13T00:50:23.569190+0000","last_clean":"2024-02-13T00:50:23.569190+0000","last_became_active":"2024-02-13T00:46:01.447823+0000","last_became_peered":"2024-02-13T00:46:01.447823+0000","last_unstale":"2024-02-13T00:50:23.569190+0000","last_undegraded":"2024-02-13T00:50:23.569190+0000","last_fullsized":"2024-02-13T00:50:23.569190+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:04.493841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:04.493841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":426,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568696+0000","last_change":"2024-02-13T00:48:25.359625+0000","last_active":"2024-02-13T00:50:23.568696+0000","last_peered":"2024-02-13T00:50:23.568696+0000","last_clean":"2024-02-13T00:50:23.568696+0000","last_became_active":"2024-02-13T00:46:01.460001+0000","last_became_peered":"2024-02-13T00:46:01.460001+0000","last_unstale":"2024-02-13T00:50:23.568696+0000","last_undegraded":"2024-02-13T00:50:23.568696+0000","last_fullsized":"2024-02-13T00:50:23.568696+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:25.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:48:25.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":447,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571328+0000","last_change":"2024-02-13T00:46:01.450879+0000","last_active":"2024-02-13T00:50:23.571328+0000","last_peered":"2024-02-13T00:50:23.571328+0000","last_clean":"2024-02-13T00:50:23.571328+0000","last_became_active":"2024-02-13T00:46:01.450088+0000","last_became_peered":"2024-02-13T00:46:01.450088+0000","last_unstale":"2024-02-13T00:50:23.571328+0000","last_undegraded":"2024-02-13T00:50:23.571328+0000","last_fullsized":"2024-02-13T00:50:23.571328+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":536,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.957997+0000","last_change":"2024-02-13T00:49:42.854702+0000","last_active":"2024-02-13T00:50:23.957997+0000","last_peered":"2024-02-13T00:50:23.957997+0000","last_clean":"2024-02-13T00:50:23.957997+0000","last_became_active":"2024-02-13T00:46:01.458408+0000","last_became_peered":"2024-02-13T00:46:01.458408+0000","last_unstale":"2024-02-13T00:50:23.957997+0000","last_undegraded":"2024-02-13T00:50:23.957997+0000","last_fullsized":"2024-02-13T00:50:23.957997+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:42.854585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:42.854585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":278,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571205+0000","last_change":"2024-02-13T00:49:19.638381+0000","last_active":"2024-02-13T00:50:23.571205+0000","last_peered":"2024-02-13T00:50:23.571205+0000","last_clean":"2024-02-13T00:50:23.571205+0000","last_became_active":"2024-02-13T00:49:19.638083+0000","last_became_peered":"2024-02-13T00:49:19.638083+0000","last_unstale":"2024-02-13T00:50:23.571205+0000","last_undegraded":"2024-02-13T00:50:23.571205+0000","last_fullsized":"2024-02-13T00:50:23.571205+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:53.013103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.013103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"467'23","reported_seq":643,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571519+0000","last_change":"2024-02-13T00:49:07.192972+0000","last_active":"2024-02-13T00:50:23.571519+0000","last_peered":"2024-02-13T00:50:23.571519+0000","last_clean":"2024-02-13T00:50:23.571519+0000","last_became_active":"2024-02-13T00:46:01.448219+0000","last_became_peered":"2024-02-13T00:46:01.448219+0000","last_unstale":"2024-02-13T00:50:23.571519+0000","last_undegraded":"2024-02-13T00:50:23.571519+0000","last_fullsized":"2024-02-13T00:50:23.571519+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'23","last_scrub_stamp":"2024-02-13T00:49:07.192902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:07.192902+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":510,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955881+0000","last_change":"2024-02-13T00:49:46.851558+0000","last_active":"2024-02-13T00:50:23.955881+0000","last_peered":"2024-02-13T00:50:23.955881+0000","last_clean":"2024-02-13T00:50:23.955881+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:50:23.955881+0000","last_undegraded":"2024-02-13T00:50:23.955881+0000","last_fullsized":"2024-02-13T00:50:23.955881+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.851486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.851486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":1373,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-02-13T00:50:24.576800+0000","last_change":"2024-02-13T00:45:54.075270+0000","last_active":"2024-02-13T00:50:24.576800+0000","last_peered":"2024-02-13T00:50:24.576800+0000","last_clean":"2024-02-13T00:50:24.576800+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:50:24.576800+0000","last_undegraded":"2024-02-13T00:50:24.576800+0000","last_fullsized":"2024-02-13T00:50:24.576800+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:45:54.075194+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":751,"num_read_kb":746,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":318,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570667+0000","last_change":"2024-02-13T00:46:01.460282+0000","last_active":"2024-02-13T00:50:23.570667+0000","last_peered":"2024-02-13T00:50:23.570667+0000","last_clean":"2024-02-13T00:50:23.570667+0000","last_became_active":"2024-02-13T00:46:01.459473+0000","last_became_peered":"2024-02-13T00:46:01.459473+0000","last_unstale":"2024-02-13T00:50:23.570667+0000","last_undegraded":"2024-02-13T00:50:23.570667+0000","last_fullsized":"2024-02-13T00:50:23.570667+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":372,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570179+0000","last_change":"2024-02-13T00:45:37.135398+0000","last_active":"2024-02-13T00:50:23.570179+0000","last_peered":"2024-02-13T00:50:23.570179+0000","last_clean":"2024-02-13T00:50:23.570179+0000","last_became_active":"2024-02-13T00:45:15.435588+0000","last_became_peered":"2024-02-13T00:45:15.435588+0000","last_unstale":"2024-02-13T00:50:23.570179+0000","last_undegraded":"2024-02-13T00:50:23.570179+0000","last_fullsized":"2024-02-13T00:50:23.570179+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.135332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.135332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":269,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570389+0000","last_change":"2024-02-13T00:45:31.183763+0000","last_active":"2024-02-13T00:50:23.570389+0000","last_peered":"2024-02-13T00:50:23.570389+0000","last_clean":"2024-02-13T00:50:23.570389+0000","last_became_active":"2024-02-13T00:45:06.421025+0000","last_became_peered":"2024-02-13T00:45:06.421025+0000","last_unstale":"2024-02-13T00:50:23.570389+0000","last_undegraded":"2024-02-13T00:50:23.570389+0000","last_fullsized":"2024-02-13T00:50:23.570389+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.183682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.183682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"467'65","reported_seq":316,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:24.359387+0000","last_change":"2024-02-13T00:49:47.928628+0000","last_active":"2024-02-13T00:50:24.359387+0000","last_peered":"2024-02-13T00:50:24.359387+0000","last_clean":"2024-02-13T00:50:24.359387+0000","last_became_active":"2024-02-13T00:46:01.449059+0000","last_became_peered":"2024-02-13T00:46:01.449059+0000","last_unstale":"2024-02-13T00:50:24.359387+0000","last_undegraded":"2024-02-13T00:50:24.359387+0000","last_fullsized":"2024-02-13T00:50:24.359387+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'65","last_scrub_stamp":"2024-02-13T00:49:47.928548+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.928548+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":2148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":517,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568820+0000","last_change":"2024-02-13T00:46:01.453042+0000","last_active":"2024-02-13T00:50:23.568820+0000","last_peered":"2024-02-13T00:50:23.568820+0000","last_clean":"2024-02-13T00:50:23.568820+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:50:23.568820+0000","last_undegraded":"2024-02-13T00:50:23.568820+0000","last_fullsized":"2024-02-13T00:50:23.568820+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:22.045912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:22.045912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":135,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956898+0000","last_change":"2024-02-13T00:48:06.705030+0000","last_active":"2024-02-13T00:50:23.956898+0000","last_peered":"2024-02-13T00:50:23.956898+0000","last_clean":"2024-02-13T00:50:23.956898+0000","last_became_active":"2024-02-13T00:48:06.704651+0000","last_became_peered":"2024-02-13T00:48:06.704651+0000","last_unstale":"2024-02-13T00:50:23.956898+0000","last_undegraded":"2024-02-13T00:50:23.956898+0000","last_fullsized":"2024-02-13T00:50:23.956898+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":305,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569399+0000","last_change":"2024-02-13T00:48:17.340891+0000","last_active":"2024-02-13T00:50:23.569399+0000","last_peered":"2024-02-13T00:50:23.569399+0000","last_clean":"2024-02-13T00:50:23.569399+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:50:23.569399+0000","last_undegraded":"2024-02-13T00:50:23.569399+0000","last_fullsized":"2024-02-13T00:50:23.569399+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:17.340797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:17.340797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":783,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:24.359463+0000","last_change":"2024-02-13T00:48:32.669477+0000","last_active":"2024-02-13T00:50:24.359463+0000","last_peered":"2024-02-13T00:50:24.359463+0000","last_clean":"2024-02-13T00:50:24.359463+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:50:24.359463+0000","last_undegraded":"2024-02-13T00:50:24.359463+0000","last_fullsized":"2024-02-13T00:50:24.359463+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14","last_scrub_stamp":"2024-02-13T00:48:32.669406+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.669406+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":340,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569509+0000","last_change":"2024-02-13T00:48:20.928236+0000","last_active":"2024-02-13T00:50:23.569509+0000","last_peered":"2024-02-13T00:50:23.569509+0000","last_clean":"2024-02-13T00:50:23.569509+0000","last_became_active":"2024-02-13T00:46:01.442003+0000","last_became_peered":"2024-02-13T00:46:01.442003+0000","last_unstale":"2024-02-13T00:50:23.569509+0000","last_undegraded":"2024-02-13T00:50:23.569509+0000","last_fullsized":"2024-02-13T00:50:23.569509+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:48:20.928158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:20.928158+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":588,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570229+0000","last_change":"2024-02-13T00:49:33.040619+0000","last_active":"2024-02-13T00:50:23.570229+0000","last_peered":"2024-02-13T00:50:23.570229+0000","last_clean":"2024-02-13T00:50:23.570229+0000","last_became_active":"2024-02-13T00:45:15.422546+0000","last_became_peered":"2024-02-13T00:45:15.422546+0000","last_unstale":"2024-02-13T00:50:23.570229+0000","last_undegraded":"2024-02-13T00:50:23.570229+0000","last_fullsized":"2024-02-13T00:50:23.570229+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:33.040516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:49:33.040516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":597,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955532+0000","last_change":"2024-02-13T00:46:01.436997+0000","last_active":"2024-02-13T00:50:23.955532+0000","last_peered":"2024-02-13T00:50:23.955532+0000","last_clean":"2024-02-13T00:50:23.955532+0000","last_became_active":"2024-02-13T00:46:01.435178+0000","last_became_peered":"2024-02-13T00:46:01.435178+0000","last_unstale":"2024-02-13T00:50:23.955532+0000","last_undegraded":"2024-02-13T00:50:23.955532+0000","last_fullsized":"2024-02-13T00:50:23.955532+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:48.645014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:48.645014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":532,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569036+0000","last_change":"2024-02-13T00:45:15.445626+0000","last_active":"2024-02-13T00:50:23.569036+0000","last_peered":"2024-02-13T00:50:23.569036+0000","last_clean":"2024-02-13T00:50:23.569036+0000","last_became_active":"2024-02-13T00:45:15.445117+0000","last_became_peered":"2024-02-13T00:45:15.445117+0000","last_unstale":"2024-02-13T00:50:23.569036+0000","last_undegraded":"2024-02-13T00:50:23.569036+0000","last_fullsized":"2024-02-13T00:50:23.569036+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.171185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":315,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955940+0000","last_change":"2024-02-13T00:49:12.252240+0000","last_active":"2024-02-13T00:50:23.955940+0000","last_peered":"2024-02-13T00:50:23.955940+0000","last_clean":"2024-02-13T00:50:23.955940+0000","last_became_active":"2024-02-13T00:49:12.251980+0000","last_became_peered":"2024-02-13T00:49:12.251980+0000","last_unstale":"2024-02-13T00:50:23.955940+0000","last_undegraded":"2024-02-13T00:50:23.955940+0000","last_fullsized":"2024-02-13T00:50:23.955940+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.592320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.592320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":658,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568953+0000","last_change":"2024-02-13T00:49:31.072963+0000","last_active":"2024-02-13T00:50:23.568953+0000","last_peered":"2024-02-13T00:50:23.568953+0000","last_clean":"2024-02-13T00:50:23.568953+0000","last_became_active":"2024-02-13T00:44:39.501886+0000","last_became_peered":"2024-02-13T00:44:39.501886+0000","last_unstale":"2024-02-13T00:50:23.568953+0000","last_undegraded":"2024-02-13T00:50:23.568953+0000","last_fullsized":"2024-02-13T00:50:23.568953+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'30","last_scrub_stamp":"2024-02-13T00:49:31.072862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:31.072862+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":576,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.957887+0000","last_change":"2024-02-13T00:48:58.861408+0000","last_active":"2024-02-13T00:50:23.957887+0000","last_peered":"2024-02-13T00:50:23.957887+0000","last_clean":"2024-02-13T00:50:23.957887+0000","last_became_active":"2024-02-13T00:46:01.457926+0000","last_became_peered":"2024-02-13T00:46:01.457926+0000","last_unstale":"2024-02-13T00:50:23.957887+0000","last_undegraded":"2024-02-13T00:50:23.957887+0000","last_fullsized":"2024-02-13T00:50:23.957887+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:48:58.861325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:58.861325+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":383,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569932+0000","last_change":"2024-02-13T00:47:06.895693+0000","last_active":"2024-02-13T00:50:23.569932+0000","last_peered":"2024-02-13T00:50:23.569932+0000","last_clean":"2024-02-13T00:50:23.569932+0000","last_became_active":"2024-02-13T00:45:15.435404+0000","last_became_peered":"2024-02-13T00:45:15.435404+0000","last_unstale":"2024-02-13T00:50:23.569932+0000","last_undegraded":"2024-02-13T00:50:23.569932+0000","last_fullsized":"2024-02-13T00:50:23.569932+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:06.895612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:47:06.895612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":536,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.958047+0000","last_change":"2024-02-13T00:50:21.880120+0000","last_active":"2024-02-13T00:50:23.958047+0000","last_peered":"2024-02-13T00:50:23.958047+0000","last_clean":"2024-02-13T00:50:23.958047+0000","last_became_active":"2024-02-13T00:46:01.449234+0000","last_became_peered":"2024-02-13T00:46:01.449234+0000","last_unstale":"2024-02-13T00:50:23.958047+0000","last_undegraded":"2024-02-13T00:50:23.958047+0000","last_fullsized":"2024-02-13T00:50:23.958047+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.880001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.880001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"467'27","reported_seq":724,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.958000+0000","last_change":"2024-02-13T00:49:35.834793+0000","last_active":"2024-02-13T00:50:23.958000+0000","last_peered":"2024-02-13T00:50:23.958000+0000","last_clean":"2024-02-13T00:50:23.958000+0000","last_became_active":"2024-02-13T00:46:01.456876+0000","last_became_peered":"2024-02-13T00:46:01.456876+0000","last_unstale":"2024-02-13T00:50:23.958000+0000","last_undegraded":"2024-02-13T00:50:23.958000+0000","last_fullsized":"2024-02-13T00:50:23.958000+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:35.834669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.834669+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":290,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568545+0000","last_change":"2024-02-13T00:46:30.095387+0000","last_active":"2024-02-13T00:50:23.568545+0000","last_peered":"2024-02-13T00:50:23.568545+0000","last_clean":"2024-02-13T00:50:23.568545+0000","last_became_active":"2024-02-13T00:44:50.441300+0000","last_became_peered":"2024-02-13T00:44:50.441300+0000","last_unstale":"2024-02-13T00:50:23.568545+0000","last_undegraded":"2024-02-13T00:50:23.568545+0000","last_fullsized":"2024-02-13T00:50:23.568545+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.095307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":304,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956579+0000","last_change":"2024-02-13T00:46:01.454597+0000","last_active":"2024-02-13T00:50:23.956579+0000","last_peered":"2024-02-13T00:50:23.956579+0000","last_clean":"2024-02-13T00:50:23.956579+0000","last_became_active":"2024-02-13T00:46:01.454101+0000","last_became_peered":"2024-02-13T00:46:01.454101+0000","last_unstale":"2024-02-13T00:50:23.956579+0000","last_undegraded":"2024-02-13T00:50:23.956579+0000","last_fullsized":"2024-02-13T00:50:23.956579+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.651091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.651091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":300,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571418+0000","last_change":"2024-02-13T00:44:50.441196+0000","last_active":"2024-02-13T00:50:23.571418+0000","last_peered":"2024-02-13T00:50:23.571418+0000","last_clean":"2024-02-13T00:50:23.571418+0000","last_became_active":"2024-02-13T00:44:50.438274+0000","last_became_peered":"2024-02-13T00:44:50.438274+0000","last_unstale":"2024-02-13T00:50:23.571418+0000","last_undegraded":"2024-02-13T00:50:23.571418+0000","last_fullsized":"2024-02-13T00:50:23.571418+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.510219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.510219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":459,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571616+0000","last_change":"2024-02-13T00:46:34.141015+0000","last_active":"2024-02-13T00:50:23.571616+0000","last_peered":"2024-02-13T00:50:23.571616+0000","last_clean":"2024-02-13T00:50:23.571616+0000","last_became_active":"2024-02-13T00:45:15.436020+0000","last_became_peered":"2024-02-13T00:45:15.436020+0000","last_unstale":"2024-02-13T00:50:23.571616+0000","last_undegraded":"2024-02-13T00:50:23.571616+0000","last_fullsized":"2024-02-13T00:50:23.571616+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:46:34.140928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.140928+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":296,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568780+0000","last_change":"2024-02-13T00:45:00.311635+0000","last_active":"2024-02-13T00:50:23.568780+0000","last_peered":"2024-02-13T00:50:23.568780+0000","last_clean":"2024-02-13T00:50:23.568780+0000","last_became_active":"2024-02-13T00:44:39.512402+0000","last_became_peered":"2024-02-13T00:44:39.512402+0000","last_unstale":"2024-02-13T00:50:23.568780+0000","last_undegraded":"2024-02-13T00:50:23.568780+0000","last_fullsized":"2024-02-13T00:50:23.568780+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:00.311558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:45:00.311558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":309,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570765+0000","last_change":"2024-02-13T00:46:01.451840+0000","last_active":"2024-02-13T00:50:23.570765+0000","last_peered":"2024-02-13T00:50:23.570765+0000","last_clean":"2024-02-13T00:50:23.570765+0000","last_became_active":"2024-02-13T00:46:01.450312+0000","last_became_peered":"2024-02-13T00:46:01.450312+0000","last_unstale":"2024-02-13T00:50:23.570765+0000","last_undegraded":"2024-02-13T00:50:23.570765+0000","last_fullsized":"2024-02-13T00:50:23.570765+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.042291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.042291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":381,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.957073+0000","last_change":"2024-02-13T00:46:01.454858+0000","last_active":"2024-02-13T00:50:23.957073+0000","last_peered":"2024-02-13T00:50:23.957073+0000","last_clean":"2024-02-13T00:50:23.957073+0000","last_became_active":"2024-02-13T00:46:01.454284+0000","last_became_peered":"2024-02-13T00:46:01.454284+0000","last_unstale":"2024-02-13T00:50:23.957073+0000","last_undegraded":"2024-02-13T00:50:23.957073+0000","last_fullsized":"2024-02-13T00:50:23.957073+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:34.523594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.523594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":542,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570319+0000","last_change":"2024-02-13T00:49:06.154997+0000","last_active":"2024-02-13T00:50:23.570319+0000","last_peered":"2024-02-13T00:50:23.570319+0000","last_clean":"2024-02-13T00:50:23.570319+0000","last_became_active":"2024-02-13T00:46:01.450113+0000","last_became_peered":"2024-02-13T00:46:01.450113+0000","last_unstale":"2024-02-13T00:50:23.570319+0000","last_undegraded":"2024-02-13T00:50:23.570319+0000","last_fullsized":"2024-02-13T00:50:23.570319+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'39","last_scrub_stamp":"2024-02-13T00:49:06.154931+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:49:06.154931+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":556,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956624+0000","last_change":"2024-02-13T00:49:29.916234+0000","last_active":"2024-02-13T00:50:23.956624+0000","last_peered":"2024-02-13T00:50:23.956624+0000","last_clean":"2024-02-13T00:50:23.956624+0000","last_became_active":"2024-02-13T00:46:01.454310+0000","last_became_peered":"2024-02-13T00:46:01.454310+0000","last_unstale":"2024-02-13T00:50:23.956624+0000","last_undegraded":"2024-02-13T00:50:23.956624+0000","last_fullsized":"2024-02-13T00:50:23.956624+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.916152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"404'9","reported_seq":299,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:24.359667+0000","last_change":"2024-02-13T00:47:37.013003+0000","last_active":"2024-02-13T00:50:24.359667+0000","last_peered":"2024-02-13T00:50:24.359667+0000","last_clean":"2024-02-13T00:50:24.359667+0000","last_became_active":"2024-02-13T00:46:01.448144+0000","last_became_peered":"2024-02-13T00:46:01.448144+0000","last_unstale":"2024-02-13T00:50:24.359667+0000","last_undegraded":"2024-02-13T00:50:24.359667+0000","last_fullsized":"2024-02-13T00:50:24.359667+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9","last_scrub_stamp":"2024-02-13T00:47:37.012934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:37.012934+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":593,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.957889+0000","last_change":"2024-02-13T00:48:26.647460+0000","last_active":"2024-02-13T00:50:23.957889+0000","last_peered":"2024-02-13T00:50:23.957889+0000","last_clean":"2024-02-13T00:50:23.957889+0000","last_became_active":"2024-02-13T00:46:01.447236+0000","last_became_peered":"2024-02-13T00:46:01.447236+0000","last_unstale":"2024-02-13T00:50:23.957889+0000","last_undegraded":"2024-02-13T00:50:23.957889+0000","last_fullsized":"2024-02-13T00:50:23.957889+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:26.647367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:26.647367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":264,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.957154+0000","last_change":"2024-02-13T00:50:14.849751+0000","last_active":"2024-02-13T00:50:23.957154+0000","last_peered":"2024-02-13T00:50:23.957154+0000","last_clean":"2024-02-13T00:50:23.957154+0000","last_became_active":"2024-02-13T00:46:01.455989+0000","last_became_peered":"2024-02-13T00:46:01.455989+0000","last_unstale":"2024-02-13T00:50:23.957154+0000","last_undegraded":"2024-02-13T00:50:23.957154+0000","last_fullsized":"2024-02-13T00:50:23.957154+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.849687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.849687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"467'22","reported_seq":578,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570502+0000","last_change":"2024-02-13T00:47:35.369710+0000","last_active":"2024-02-13T00:50:23.570502+0000","last_peered":"2024-02-13T00:50:23.570502+0000","last_clean":"2024-02-13T00:50:23.570502+0000","last_became_active":"2024-02-13T00:46:01.460321+0000","last_became_peered":"2024-02-13T00:46:01.460321+0000","last_unstale":"2024-02-13T00:50:23.570502+0000","last_undegraded":"2024-02-13T00:50:23.570502+0000","last_fullsized":"2024-02-13T00:50:23.570502+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:47:35.369616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:35.369616+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":608,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956599+0000","last_change":"2024-02-13T00:49:05.887739+0000","last_active":"2024-02-13T00:50:23.956599+0000","last_peered":"2024-02-13T00:50:23.956599+0000","last_clean":"2024-02-13T00:50:23.956599+0000","last_became_active":"2024-02-13T00:46:01.431926+0000","last_became_peered":"2024-02-13T00:46:01.431926+0000","last_unstale":"2024-02-13T00:50:23.956599+0000","last_undegraded":"2024-02-13T00:50:23.956599+0000","last_fullsized":"2024-02-13T00:50:23.956599+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:05.887665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:05.887665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"363'14","reported_seq":442,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:24.359532+0000","last_change":"2024-02-13T00:47:07.942748+0000","last_active":"2024-02-13T00:50:24.359532+0000","last_peered":"2024-02-13T00:50:24.359532+0000","last_clean":"2024-02-13T00:50:24.359532+0000","last_became_active":"2024-02-13T00:45:15.423066+0000","last_became_peered":"2024-02-13T00:45:15.423066+0000","last_unstale":"2024-02-13T00:50:24.359532+0000","last_undegraded":"2024-02-13T00:50:24.359532+0000","last_fullsized":"2024-02-13T00:50:24.359532+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:47:07.942668+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:47:07.942668+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":227,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.957822+0000","last_change":"2024-02-13T00:47:02.789298+0000","last_active":"2024-02-13T00:50:23.957822+0000","last_peered":"2024-02-13T00:50:23.957822+0000","last_clean":"2024-02-13T00:50:23.957822+0000","last_became_active":"2024-02-13T00:46:01.446960+0000","last_became_peered":"2024-02-13T00:46:01.446960+0000","last_unstale":"2024-02-13T00:50:23.957822+0000","last_undegraded":"2024-02-13T00:50:23.957822+0000","last_fullsized":"2024-02-13T00:50:23.957822+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:02.789189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.789189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":465,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569076+0000","last_change":"2024-02-13T00:46:01.450308+0000","last_active":"2024-02-13T00:50:23.569076+0000","last_peered":"2024-02-13T00:50:23.569076+0000","last_clean":"2024-02-13T00:50:23.569076+0000","last_became_active":"2024-02-13T00:46:01.449694+0000","last_became_peered":"2024-02-13T00:46:01.449694+0000","last_unstale":"2024-02-13T00:50:23.569076+0000","last_undegraded":"2024-02-13T00:50:23.569076+0000","last_fullsized":"2024-02-13T00:50:23.569076+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:27.073034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:27.073034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":663,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569403+0000","last_change":"2024-02-13T00:49:38.007273+0000","last_active":"2024-02-13T00:50:23.569403+0000","last_peered":"2024-02-13T00:50:23.569403+0000","last_clean":"2024-02-13T00:50:23.569403+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:50:23.569403+0000","last_undegraded":"2024-02-13T00:50:23.569403+0000","last_fullsized":"2024-02-13T00:50:23.569403+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'19","last_scrub_stamp":"2024-02-13T00:49:38.007204+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:49:38.007204+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":309,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569987+0000","last_change":"2024-02-13T00:46:01.463310+0000","last_active":"2024-02-13T00:50:23.569987+0000","last_peered":"2024-02-13T00:50:23.569987+0000","last_clean":"2024-02-13T00:50:23.569987+0000","last_became_active":"2024-02-13T00:46:01.462898+0000","last_became_peered":"2024-02-13T00:46:01.462898+0000","last_unstale":"2024-02-13T00:50:23.569987+0000","last_undegraded":"2024-02-13T00:50:23.569987+0000","last_fullsized":"2024-02-13T00:50:23.569987+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.095888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":372,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956213+0000","last_change":"2024-02-13T00:46:32.638418+0000","last_active":"2024-02-13T00:50:23.956213+0000","last_peered":"2024-02-13T00:50:23.956213+0000","last_clean":"2024-02-13T00:50:23.956213+0000","last_became_active":"2024-02-13T00:46:01.443279+0000","last_became_peered":"2024-02-13T00:46:01.443279+0000","last_unstale":"2024-02-13T00:50:23.956213+0000","last_undegraded":"2024-02-13T00:50:23.956213+0000","last_fullsized":"2024-02-13T00:50:23.956213+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.638349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":575,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955523+0000","last_change":"2024-02-13T00:48:27.659266+0000","last_active":"2024-02-13T00:50:23.955523+0000","last_peered":"2024-02-13T00:50:23.955523+0000","last_clean":"2024-02-13T00:50:23.955523+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:50:23.955523+0000","last_undegraded":"2024-02-13T00:50:23.955523+0000","last_fullsized":"2024-02-13T00:50:23.955523+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:27.659197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:48:27.659197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":305,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570106+0000","last_change":"2024-02-13T00:45:40.233264+0000","last_active":"2024-02-13T00:50:23.570106+0000","last_peered":"2024-02-13T00:50:23.570106+0000","last_clean":"2024-02-13T00:50:23.570106+0000","last_became_active":"2024-02-13T00:45:15.447051+0000","last_became_peered":"2024-02-13T00:45:15.447051+0000","last_unstale":"2024-02-13T00:50:23.570106+0000","last_undegraded":"2024-02-13T00:50:23.570106+0000","last_fullsized":"2024-02-13T00:50:23.570106+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:40.233185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:40.233185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":341,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570065+0000","last_change":"2024-02-13T00:48:34.458358+0000","last_active":"2024-02-13T00:50:23.570065+0000","last_peered":"2024-02-13T00:50:23.570065+0000","last_clean":"2024-02-13T00:50:23.570065+0000","last_became_active":"2024-02-13T00:44:50.435203+0000","last_became_peered":"2024-02-13T00:44:50.435203+0000","last_unstale":"2024-02-13T00:50:23.570065+0000","last_undegraded":"2024-02-13T00:50:23.570065+0000","last_fullsized":"2024-02-13T00:50:23.570065+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:48:34.458264+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:48:34.458264+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":583,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570959+0000","last_change":"2024-02-13T00:46:01.457220+0000","last_active":"2024-02-13T00:50:23.570959+0000","last_peered":"2024-02-13T00:50:23.570959+0000","last_clean":"2024-02-13T00:50:23.570959+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:50:23.570959+0000","last_undegraded":"2024-02-13T00:50:23.570959+0000","last_fullsized":"2024-02-13T00:50:23.570959+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.097481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":597,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569406+0000","last_change":"2024-02-13T00:46:01.453391+0000","last_active":"2024-02-13T00:50:23.569406+0000","last_peered":"2024-02-13T00:50:23.569406+0000","last_clean":"2024-02-13T00:50:23.569406+0000","last_became_active":"2024-02-13T00:46:01.452730+0000","last_became_peered":"2024-02-13T00:46:01.452730+0000","last_unstale":"2024-02-13T00:50:23.569406+0000","last_undegraded":"2024-02-13T00:50:23.569406+0000","last_fullsized":"2024-02-13T00:50:23.569406+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:26.218807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:44:26.218807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":381,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.955987+0000","last_change":"2024-02-13T00:48:29.681480+0000","last_active":"2024-02-13T00:50:23.955987+0000","last_peered":"2024-02-13T00:50:23.955987+0000","last_clean":"2024-02-13T00:50:23.955987+0000","last_became_active":"2024-02-13T00:46:01.454287+0000","last_became_peered":"2024-02-13T00:46:01.454287+0000","last_unstale":"2024-02-13T00:50:23.955987+0000","last_undegraded":"2024-02-13T00:50:23.955987+0000","last_fullsized":"2024-02-13T00:50:23.955987+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:29.681414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:29.681414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":314,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568375+0000","last_change":"2024-02-13T00:49:02.260116+0000","last_active":"2024-02-13T00:50:23.568375+0000","last_peered":"2024-02-13T00:50:23.568375+0000","last_clean":"2024-02-13T00:50:23.568375+0000","last_became_active":"2024-02-13T00:46:01.448671+0000","last_became_peered":"2024-02-13T00:46:01.448671+0000","last_unstale":"2024-02-13T00:50:23.568375+0000","last_undegraded":"2024-02-13T00:50:23.568375+0000","last_fullsized":"2024-02-13T00:50:23.568375+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:02.260045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:02.260045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":410,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568843+0000","last_change":"2024-02-13T00:44:40.371566+0000","last_active":"2024-02-13T00:50:23.568843+0000","last_peered":"2024-02-13T00:50:23.568843+0000","last_clean":"2024-02-13T00:50:23.568843+0000","last_became_active":"2024-02-13T00:44:39.501040+0000","last_became_peered":"2024-02-13T00:44:39.501040+0000","last_unstale":"2024-02-13T00:50:23.568843+0000","last_undegraded":"2024-02-13T00:50:23.568843+0000","last_fullsized":"2024-02-13T00:50:23.568843+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'4","last_scrub_stamp":"2024-02-13T00:44:32.180113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:44:32.180113+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":540,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569902+0000","last_change":"2024-02-13T00:48:32.441279+0000","last_active":"2024-02-13T00:50:23.569902+0000","last_peered":"2024-02-13T00:50:23.569902+0000","last_clean":"2024-02-13T00:50:23.569902+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:50:23.569902+0000","last_undegraded":"2024-02-13T00:50:23.569902+0000","last_fullsized":"2024-02-13T00:50:23.569902+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.441170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.441170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":305,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568559+0000","last_change":"2024-02-13T00:48:28.353456+0000","last_active":"2024-02-13T00:50:23.568559+0000","last_peered":"2024-02-13T00:50:23.568559+0000","last_clean":"2024-02-13T00:50:23.568559+0000","last_became_active":"2024-02-13T00:45:15.435820+0000","last_became_peered":"2024-02-13T00:45:15.435820+0000","last_unstale":"2024-02-13T00:50:23.568559+0000","last_undegraded":"2024-02-13T00:50:23.568559+0000","last_fullsized":"2024-02-13T00:50:23.568559+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:28.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:28.353347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":297,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568601+0000","last_change":"2024-02-13T00:49:29.052728+0000","last_active":"2024-02-13T00:50:23.568601+0000","last_peered":"2024-02-13T00:50:23.568601+0000","last_clean":"2024-02-13T00:50:23.568601+0000","last_became_active":"2024-02-13T00:45:15.444818+0000","last_became_peered":"2024-02-13T00:45:15.444818+0000","last_unstale":"2024-02-13T00:50:23.568601+0000","last_undegraded":"2024-02-13T00:50:23.568601+0000","last_fullsized":"2024-02-13T00:50:23.568601+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.052640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.052640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":456,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568651+0000","last_change":"2024-02-13T00:46:01.439915+0000","last_active":"2024-02-13T00:50:23.568651+0000","last_peered":"2024-02-13T00:50:23.568651+0000","last_clean":"2024-02-13T00:50:23.568651+0000","last_became_active":"2024-02-13T00:46:01.438745+0000","last_became_peered":"2024-02-13T00:46:01.438745+0000","last_unstale":"2024-02-13T00:50:23.568651+0000","last_undegraded":"2024-02-13T00:50:23.568651+0000","last_fullsized":"2024-02-13T00:50:23.568651+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.104906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.104906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"467'25","reported_seq":368,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568674+0000","last_change":"2024-02-13T00:47:36.368967+0000","last_active":"2024-02-13T00:50:23.568674+0000","last_peered":"2024-02-13T00:50:23.568674+0000","last_clean":"2024-02-13T00:50:23.568674+0000","last_became_active":"2024-02-13T00:46:01.450079+0000","last_became_peered":"2024-02-13T00:46:01.450079+0000","last_unstale":"2024-02-13T00:50:23.568674+0000","last_undegraded":"2024-02-13T00:50:23.568674+0000","last_fullsized":"2024-02-13T00:50:23.568674+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:47:36.368863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.368863+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":314,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571030+0000","last_change":"2024-02-13T00:46:59.388809+0000","last_active":"2024-02-13T00:50:23.571030+0000","last_peered":"2024-02-13T00:50:23.571030+0000","last_clean":"2024-02-13T00:50:23.571030+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:50:23.571030+0000","last_undegraded":"2024-02-13T00:50:23.571030+0000","last_fullsized":"2024-02-13T00:50:23.571030+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:59.388745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:59.388745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":300,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568138+0000","last_change":"2024-02-13T00:45:15.439752+0000","last_active":"2024-02-13T00:50:23.568138+0000","last_peered":"2024-02-13T00:50:23.568138+0000","last_clean":"2024-02-13T00:50:23.568138+0000","last_became_active":"2024-02-13T00:45:15.437943+0000","last_became_peered":"2024-02-13T00:45:15.437943+0000","last_unstale":"2024-02-13T00:50:23.568138+0000","last_undegraded":"2024-02-13T00:50:23.568138+0000","last_fullsized":"2024-02-13T00:50:23.568138+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.284740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.284740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":508,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956135+0000","last_change":"2024-02-13T00:44:58.969806+0000","last_active":"2024-02-13T00:50:23.956135+0000","last_peered":"2024-02-13T00:50:23.956135+0000","last_clean":"2024-02-13T00:50:23.956135+0000","last_became_active":"2024-02-13T00:44:50.439512+0000","last_became_peered":"2024-02-13T00:44:50.439512+0000","last_unstale":"2024-02-13T00:50:23.956135+0000","last_undegraded":"2024-02-13T00:50:23.956135+0000","last_fullsized":"2024-02-13T00:50:23.956135+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:58.969692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:44:58.969692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":428,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569981+0000","last_change":"2024-02-13T00:49:35.101413+0000","last_active":"2024-02-13T00:50:23.569981+0000","last_peered":"2024-02-13T00:50:23.569981+0000","last_clean":"2024-02-13T00:50:23.569981+0000","last_became_active":"2024-02-13T00:45:15.430749+0000","last_became_peered":"2024-02-13T00:45:15.430749+0000","last_unstale":"2024-02-13T00:50:23.569981+0000","last_undegraded":"2024-02-13T00:50:23.569981+0000","last_fullsized":"2024-02-13T00:50:23.569981+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:35.101298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.101298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":697,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.958200+0000","last_change":"2024-02-13T00:48:53.841888+0000","last_active":"2024-02-13T00:50:23.958200+0000","last_peered":"2024-02-13T00:50:23.958200+0000","last_clean":"2024-02-13T00:50:23.958200+0000","last_became_active":"2024-02-13T00:46:01.443721+0000","last_became_peered":"2024-02-13T00:46:01.443721+0000","last_unstale":"2024-02-13T00:50:23.958200+0000","last_undegraded":"2024-02-13T00:50:23.958200+0000","last_fullsized":"2024-02-13T00:50:23.958200+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'32","last_scrub_stamp":"2024-02-13T00:48:53.841797+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.841797+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":542,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.567785+0000","last_change":"2024-02-13T00:46:32.298033+0000","last_active":"2024-02-13T00:50:23.567785+0000","last_peered":"2024-02-13T00:50:23.567785+0000","last_clean":"2024-02-13T00:50:23.567785+0000","last_became_active":"2024-02-13T00:46:01.442231+0000","last_became_peered":"2024-02-13T00:46:01.442231+0000","last_unstale":"2024-02-13T00:50:23.567785+0000","last_undegraded":"2024-02-13T00:50:23.567785+0000","last_fullsized":"2024-02-13T00:50:23.567785+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.297918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.297918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":406,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569376+0000","last_change":"2024-02-13T00:48:33.460377+0000","last_active":"2024-02-13T00:50:23.569376+0000","last_peered":"2024-02-13T00:50:23.569376+0000","last_clean":"2024-02-13T00:50:23.569376+0000","last_became_active":"2024-02-13T00:46:01.439984+0000","last_became_peered":"2024-02-13T00:46:01.439984+0000","last_unstale":"2024-02-13T00:50:23.569376+0000","last_undegraded":"2024-02-13T00:50:23.569376+0000","last_fullsized":"2024-02-13T00:50:23.569376+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:48:33.460247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:33.460247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":541,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571600+0000","last_change":"2024-02-13T00:45:38.084426+0000","last_active":"2024-02-13T00:50:23.571600+0000","last_peered":"2024-02-13T00:50:23.571600+0000","last_clean":"2024-02-13T00:50:23.571600+0000","last_became_active":"2024-02-13T00:45:15.435550+0000","last_became_peered":"2024-02-13T00:45:15.435550+0000","last_unstale":"2024-02-13T00:50:23.571600+0000","last_undegraded":"2024-02-13T00:50:23.571600+0000","last_fullsized":"2024-02-13T00:50:23.571600+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.084358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.084358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":345,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571513+0000","last_change":"2024-02-13T00:48:55.424021+0000","last_active":"2024-02-13T00:50:23.571513+0000","last_peered":"2024-02-13T00:50:23.571513+0000","last_clean":"2024-02-13T00:50:23.571513+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:50:23.571513+0000","last_undegraded":"2024-02-13T00:50:23.571513+0000","last_fullsized":"2024-02-13T00:50:23.571513+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:55.423952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.423952+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":314,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568786+0000","last_change":"2024-02-13T00:46:56.339969+0000","last_active":"2024-02-13T00:50:23.568786+0000","last_peered":"2024-02-13T00:50:23.568786+0000","last_clean":"2024-02-13T00:50:23.568786+0000","last_became_active":"2024-02-13T00:45:15.436239+0000","last_became_peered":"2024-02-13T00:45:15.436239+0000","last_unstale":"2024-02-13T00:50:23.568786+0000","last_undegraded":"2024-02-13T00:50:23.568786+0000","last_fullsized":"2024-02-13T00:50:23.568786+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:56.339885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.339885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":582,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570874+0000","last_change":"2024-02-13T00:49:46.928723+0000","last_active":"2024-02-13T00:50:23.570874+0000","last_peered":"2024-02-13T00:50:23.570874+0000","last_clean":"2024-02-13T00:50:23.570874+0000","last_became_active":"2024-02-13T00:46:01.453269+0000","last_became_peered":"2024-02-13T00:46:01.453269+0000","last_unstale":"2024-02-13T00:50:23.570874+0000","last_undegraded":"2024-02-13T00:50:23.570874+0000","last_fullsized":"2024-02-13T00:50:23.570874+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.928651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.928651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":305,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570116+0000","last_change":"2024-02-13T00:45:37.280947+0000","last_active":"2024-02-13T00:50:23.570116+0000","last_peered":"2024-02-13T00:50:23.570116+0000","last_clean":"2024-02-13T00:50:23.570116+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:50:23.570116+0000","last_undegraded":"2024-02-13T00:50:23.570116+0000","last_fullsized":"2024-02-13T00:50:23.570116+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.280865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.280865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":361,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570316+0000","last_change":"2024-02-13T00:49:30.086932+0000","last_active":"2024-02-13T00:50:23.570316+0000","last_peered":"2024-02-13T00:50:23.570316+0000","last_clean":"2024-02-13T00:50:23.570316+0000","last_became_active":"2024-02-13T00:44:50.442894+0000","last_became_peered":"2024-02-13T00:44:50.442894+0000","last_unstale":"2024-02-13T00:50:23.570316+0000","last_undegraded":"2024-02-13T00:50:23.570316+0000","last_fullsized":"2024-02-13T00:50:23.570316+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:30.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:30.086856+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":305,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569149+0000","last_change":"2024-02-13T00:49:04.282594+0000","last_active":"2024-02-13T00:50:23.569149+0000","last_peered":"2024-02-13T00:50:23.569149+0000","last_clean":"2024-02-13T00:50:23.569149+0000","last_became_active":"2024-02-13T00:45:15.445403+0000","last_became_peered":"2024-02-13T00:45:15.445403+0000","last_unstale":"2024-02-13T00:50:23.569149+0000","last_undegraded":"2024-02-13T00:50:23.569149+0000","last_fullsized":"2024-02-13T00:50:23.569149+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:04.282529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:04.282529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":305,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570301+0000","last_change":"2024-02-13T00:47:42.329463+0000","last_active":"2024-02-13T00:50:23.570301+0000","last_peered":"2024-02-13T00:50:23.570301+0000","last_clean":"2024-02-13T00:50:23.570301+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:50:23.570301+0000","last_undegraded":"2024-02-13T00:50:23.570301+0000","last_fullsized":"2024-02-13T00:50:23.570301+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:42.329367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:42.329367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":219,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570509+0000","last_change":"2024-02-13T00:50:23.430479+0000","last_active":"2024-02-13T00:50:23.570509+0000","last_peered":"2024-02-13T00:50:23.570509+0000","last_clean":"2024-02-13T00:50:23.570509+0000","last_became_active":"2024-02-13T00:46:01.449873+0000","last_became_peered":"2024-02-13T00:46:01.449873+0000","last_unstale":"2024-02-13T00:50:23.570509+0000","last_undegraded":"2024-02-13T00:50:23.570509+0000","last_fullsized":"2024-02-13T00:50:23.570509+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:23.430396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:23.430396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":462,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571240+0000","last_change":"2024-02-13T00:46:56.973071+0000","last_active":"2024-02-13T00:50:23.571240+0000","last_peered":"2024-02-13T00:50:23.571240+0000","last_clean":"2024-02-13T00:50:23.571240+0000","last_became_active":"2024-02-13T00:44:39.504670+0000","last_became_peered":"2024-02-13T00:44:39.504670+0000","last_unstale":"2024-02-13T00:50:23.571240+0000","last_undegraded":"2024-02-13T00:50:23.571240+0000","last_fullsized":"2024-02-13T00:50:23.571240+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:46:56.973000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.973000+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":309,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570192+0000","last_change":"2024-02-13T00:46:01.448134+0000","last_active":"2024-02-13T00:50:23.570192+0000","last_peered":"2024-02-13T00:50:23.570192+0000","last_clean":"2024-02-13T00:50:23.570192+0000","last_became_active":"2024-02-13T00:46:01.446635+0000","last_became_peered":"2024-02-13T00:46:01.446635+0000","last_unstale":"2024-02-13T00:50:23.570192+0000","last_undegraded":"2024-02-13T00:50:23.570192+0000","last_fullsized":"2024-02-13T00:50:23.570192+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:28.101254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:28.101254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":305,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.571036+0000","last_change":"2024-02-13T00:46:01.461188+0000","last_active":"2024-02-13T00:50:23.571036+0000","last_peered":"2024-02-13T00:50:23.571036+0000","last_clean":"2024-02-13T00:50:23.571036+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:50:23.571036+0000","last_undegraded":"2024-02-13T00:50:23.571036+0000","last_fullsized":"2024-02-13T00:50:23.571036+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.238152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.238152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":549,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956528+0000","last_change":"2024-02-13T00:46:01.443734+0000","last_active":"2024-02-13T00:50:23.956528+0000","last_peered":"2024-02-13T00:50:23.956528+0000","last_clean":"2024-02-13T00:50:23.956528+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:50:23.956528+0000","last_undegraded":"2024-02-13T00:50:23.956528+0000","last_fullsized":"2024-02-13T00:50:23.956528+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.621170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.621170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":454,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956526+0000","last_change":"2024-02-13T00:46:27.706096+0000","last_active":"2024-02-13T00:50:23.956526+0000","last_peered":"2024-02-13T00:50:23.956526+0000","last_clean":"2024-02-13T00:50:23.956526+0000","last_became_active":"2024-02-13T00:44:50.415894+0000","last_became_peered":"2024-02-13T00:44:50.415894+0000","last_unstale":"2024-02-13T00:50:23.956526+0000","last_undegraded":"2024-02-13T00:50:23.956526+0000","last_fullsized":"2024-02-13T00:50:23.956526+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:46:27.706006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:27.706006+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":389,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569494+0000","last_change":"2024-02-13T00:49:43.067713+0000","last_active":"2024-02-13T00:50:23.569494+0000","last_peered":"2024-02-13T00:50:23.569494+0000","last_clean":"2024-02-13T00:50:23.569494+0000","last_became_active":"2024-02-13T00:46:01.451790+0000","last_became_peered":"2024-02-13T00:46:01.451790+0000","last_unstale":"2024-02-13T00:50:23.569494+0000","last_undegraded":"2024-02-13T00:50:23.569494+0000","last_fullsized":"2024-02-13T00:50:23.569494+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:43.067615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:43.067615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":577,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956030+0000","last_change":"2024-02-13T00:47:36.885611+0000","last_active":"2024-02-13T00:50:23.956030+0000","last_peered":"2024-02-13T00:50:23.956030+0000","last_clean":"2024-02-13T00:50:23.956030+0000","last_became_active":"2024-02-13T00:45:15.419668+0000","last_became_peered":"2024-02-13T00:45:15.419668+0000","last_unstale":"2024-02-13T00:50:23.956030+0000","last_undegraded":"2024-02-13T00:50:23.956030+0000","last_fullsized":"2024-02-13T00:50:23.956030+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.885510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.885510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":488,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956868+0000","last_change":"2024-02-13T00:50:15.846534+0000","last_active":"2024-02-13T00:50:23.956868+0000","last_peered":"2024-02-13T00:50:23.956868+0000","last_clean":"2024-02-13T00:50:23.956868+0000","last_became_active":"2024-02-13T00:46:01.455940+0000","last_became_peered":"2024-02-13T00:46:01.455940+0000","last_unstale":"2024-02-13T00:50:23.956868+0000","last_undegraded":"2024-02-13T00:50:23.956868+0000","last_fullsized":"2024-02-13T00:50:23.956868+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:15.846461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:15.846461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":470,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569829+0000","last_change":"2024-02-13T00:46:36.089483+0000","last_active":"2024-02-13T00:50:23.569829+0000","last_peered":"2024-02-13T00:50:23.569829+0000","last_clean":"2024-02-13T00:50:23.569829+0000","last_became_active":"2024-02-13T00:45:15.419364+0000","last_became_peered":"2024-02-13T00:45:15.419364+0000","last_unstale":"2024-02-13T00:50:23.569829+0000","last_undegraded":"2024-02-13T00:50:23.569829+0000","last_fullsized":"2024-02-13T00:50:23.569829+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:46:36.089402+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:46:36.089402+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":314,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570782+0000","last_change":"2024-02-13T00:47:47.284092+0000","last_active":"2024-02-13T00:50:23.570782+0000","last_peered":"2024-02-13T00:50:23.570782+0000","last_clean":"2024-02-13T00:50:23.570782+0000","last_became_active":"2024-02-13T00:46:01.451997+0000","last_became_peered":"2024-02-13T00:46:01.451997+0000","last_unstale":"2024-02-13T00:50:23.570782+0000","last_undegraded":"2024-02-13T00:50:23.570782+0000","last_fullsized":"2024-02-13T00:50:23.570782+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.284014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.284014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":527,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569188+0000","last_change":"2024-02-13T00:49:01.133798+0000","last_active":"2024-02-13T00:50:23.569188+0000","last_peered":"2024-02-13T00:50:23.569188+0000","last_clean":"2024-02-13T00:50:23.569188+0000","last_became_active":"2024-02-13T00:46:01.445141+0000","last_became_peered":"2024-02-13T00:46:01.445141+0000","last_unstale":"2024-02-13T00:50:23.569188+0000","last_undegraded":"2024-02-13T00:50:23.569188+0000","last_fullsized":"2024-02-13T00:50:23.569188+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:01.133709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:49:01.133709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":296,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568462+0000","last_change":"2024-02-13T00:45:30.186208+0000","last_active":"2024-02-13T00:50:23.568462+0000","last_peered":"2024-02-13T00:50:23.568462+0000","last_clean":"2024-02-13T00:50:23.568462+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:50:23.568462+0000","last_undegraded":"2024-02-13T00:50:23.568462+0000","last_fullsized":"2024-02-13T00:50:23.568462+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:30.186129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:30.186129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":354,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568538+0000","last_change":"2024-02-13T00:46:06.251713+0000","last_active":"2024-02-13T00:50:23.568538+0000","last_peered":"2024-02-13T00:50:23.568538+0000","last_clean":"2024-02-13T00:50:23.568538+0000","last_became_active":"2024-02-13T00:46:01.442464+0000","last_became_peered":"2024-02-13T00:46:01.442464+0000","last_unstale":"2024-02-13T00:50:23.568538+0000","last_undegraded":"2024-02-13T00:50:23.568538+0000","last_fullsized":"2024-02-13T00:50:23.568538+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:43:44.241358+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:43:44.241358+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":390,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.958106+0000","last_change":"2024-02-13T00:47:47.711551+0000","last_active":"2024-02-13T00:50:23.958106+0000","last_peered":"2024-02-13T00:50:23.958106+0000","last_clean":"2024-02-13T00:50:23.958106+0000","last_became_active":"2024-02-13T00:46:01.447654+0000","last_became_peered":"2024-02-13T00:46:01.447654+0000","last_unstale":"2024-02-13T00:50:23.958106+0000","last_undegraded":"2024-02-13T00:50:23.958106+0000","last_fullsized":"2024-02-13T00:50:23.958106+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.711486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.711486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":226,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.568706+0000","last_change":"2024-02-13T00:47:53.826039+0000","last_active":"2024-02-13T00:50:23.568706+0000","last_peered":"2024-02-13T00:50:23.568706+0000","last_clean":"2024-02-13T00:50:23.568706+0000","last_became_active":"2024-02-13T00:46:01.448971+0000","last_became_peered":"2024-02-13T00:46:01.448971+0000","last_unstale":"2024-02-13T00:50:23.568706+0000","last_undegraded":"2024-02-13T00:50:23.568706+0000","last_fullsized":"2024-02-13T00:50:23.568706+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:53.825971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:53.825971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":553,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.569269+0000","last_change":"2024-02-13T00:45:32.233265+0000","last_active":"2024-02-13T00:50:23.569269+0000","last_peered":"2024-02-13T00:50:23.569269+0000","last_clean":"2024-02-13T00:50:23.569269+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:50:23.569269+0000","last_undegraded":"2024-02-13T00:50:23.569269+0000","last_fullsized":"2024-02-13T00:50:23.569269+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:45:32.233197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.233197+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":489,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.956077+0000","last_change":"2024-02-13T00:47:44.757362+0000","last_active":"2024-02-13T00:50:23.956077+0000","last_peered":"2024-02-13T00:50:23.956077+0000","last_clean":"2024-02-13T00:50:23.956077+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:50:23.956077+0000","last_undegraded":"2024-02-13T00:50:23.956077+0000","last_fullsized":"2024-02-13T00:50:23.956077+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:47:44.757262+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:47:44.757262+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":581,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.957929+0000","last_change":"2024-02-13T00:48:54.818104+0000","last_active":"2024-02-13T00:50:23.957929+0000","last_peered":"2024-02-13T00:50:23.957929+0000","last_clean":"2024-02-13T00:50:23.957929+0000","last_became_active":"2024-02-13T00:45:15.434224+0000","last_became_peered":"2024-02-13T00:45:15.434224+0000","last_unstale":"2024-02-13T00:50:23.957929+0000","last_undegraded":"2024-02-13T00:50:23.957929+0000","last_fullsized":"2024-02-13T00:50:23.957929+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:54.817999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.817999+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":300,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-02-13T00:50:23.570225+0000","last_change":"2024-02-13T00:45:15.435955+0000","last_active":"2024-02-13T00:50:23.570225+0000","last_peered":"2024-02-13T00:50:23.570225+0000","last_clean":"2024-02-13T00:50:23.570225+0000","last_became_active":"2024-02-13T00:45:15.435259+0000","last_became_peered":"2024-02-13T00:45:15.435259+0000","last_unstale":"2024-02-13T00:50:23.570225+0000","last_undegraded":"2024-02-13T00:50:23.570225+0000","last_fullsized":"2024-02-13T00:50:23.570225+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.953989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.953989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":236,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":4},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":12,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1492,"num_read_kb":1217,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":9606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":11656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":970,"ondisk_log_size":970,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":751,"num_read_kb":746,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":227633266843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":45,"seq":193273528481,"num_pgs":156,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308564,"kb_used_data":3148,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436588,"statfs":{"total":95995035648,"available":95679066112,"internally_reserved":0,"allocated":3223552,"data_stored":1535246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.46700000000000003,"15min":0.54600000000000004},"min":{"1min":0.316,"5min":0.25600000000000001,"15min":0.19700000000000001},"max":{"1min":0.65600000000000003,"5min":0.77500000000000002,"15min":8.1340000000000003},"last":0.437},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52100000000000002,"15min":0.60999999999999999},"min":{"1min":0.34100000000000003,"5min":0.23100000000000001,"15min":0.221},"max":{"1min":0.72799999999999998,"5min":4.2910000000000004,"15min":11.367000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51200000000000001,"15min":0.88400000000000001},"min":{"1min":0.34899999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.63200000000000001,"5min":4.5720000000000001,"15min":54.002000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52300000000000002,"15min":0.68100000000000005},"min":{"1min":0.28999999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66800000000000004,"5min":5.4989999999999997,"15min":12.403},"last":0.66200000000000003}]},{"osd":2,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52300000000000002,"15min":0.71099999999999997},"min":{"1min":0.31900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":2.8399999999999999,"15min":12.477},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50700000000000001,"15min":0.68000000000000005},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.63200000000000001,"5min":2.1659999999999999,"15min":11.702},"last":0.54100000000000004}]},{"osd":3,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.61499999999999999,"15min":0.68100000000000005},"min":{"1min":0.35299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.65700000000000003,"5min":5.931,"15min":8.0519999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54400000000000004,"15min":0.77200000000000002},"min":{"1min":0.44,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":0.65400000000000003,"5min":2.8719999999999999,"15min":19.081},"last":0.62}]},{"osd":4,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.41499999999999998},"min":{"1min":0.29099999999999998,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":0.51400000000000001,"5min":0.71899999999999997,"15min":1.0649999999999999},"last":0.497},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.441,"15min":0.438},"min":{"1min":0.315,"5min":0.22800000000000001,"15min":0.221},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.65000000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66400000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69899999999999995,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.71299999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45600000000000002,"15min":0.48099999999999998},"min":{"1min":0.36699999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60899999999999999,"5min":0.66800000000000004,"15min":1.173},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.50900000000000001,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.28299999999999997,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.80700000000000005,"15min":1.262},"last":0.67400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575034,"num_pgs":161,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304500,"kb_used_data":3180,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440652,"statfs":{"total":95995035648,"available":95683227648,"internally_reserved":0,"allocated":3256320,"data_stored":1544377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34399999999999997,"15min":0.95099999999999996},"min":{"1min":0.20699999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.52700000000000002,"5min":1.1080000000000001,"15min":61.856000000000002},"last":0.251},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33600000000000002,"15min":0.66800000000000004},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.16500000000000001},"max":{"1min":0.70199999999999996,"5min":1.052,"15min":61.908999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:50:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33700000000000002,"15min":0.84599999999999997},"min":{"1min":0.27300000000000002,"5min":0.22900000000000001,"15min":0.20599999999999999},"max":{"1min":0.53000000000000003,"5min":0.80000000000000004,"15min":32.161000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.82699999999999996},"min":{"1min":0.253,"5min":0.187,"15min":0.187},"max":{"1min":0.504,"5min":1.21,"15min":29.693000000000001},"last":0.28699999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.376,"15min":1.2669999999999999},"min":{"1min":0.28100000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.55400000000000005,"5min":1.0900000000000001,"15min":131.23099999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.39700000000000002,"15min":0.86099999999999999},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.48099999999999998,"5min":1.214,"15min":49.055},"last":0.35399999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.75800000000000001},"min":{"1min":0.36199999999999999,"5min":0.33500000000000002,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":1.246,"15min":16.547000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.56399999999999995,"15min":0.71199999999999997},"min":{"1min":0.42499999999999999,"5min":0.34100000000000003,"15min":0.24199999999999999},"max":{"1min":0.85199999999999998,"5min":0.89200000000000002,"15min":14.731999999999999},"last":0.70599999999999996}]},{"osd":5,"last update":"Tue Feb 13 00:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.748},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59899999999999998,"15min":0.58699999999999997},"min":{"1min":0.437,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.67200000000000004}]},{"osd":6,"last update":"Tue Feb 13 00:49:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54700000000000004,"15min":0.71799999999999997},"min":{"1min":0.41299999999999998,"5min":0.29699999999999999,"15min":0.29099999999999998},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":14.772},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.54600000000000004,"15min":0.92500000000000004},"min":{"1min":0.40500000000000003,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":26.600000000000001},"last":0.57999999999999996}]},{"osd":7,"last update":"Tue Feb 13 00:50:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.56299999999999994,"15min":0.72299999999999998},"min":{"1min":0.41499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":14.794},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55900000000000005,"15min":0.92500000000000004},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.32200000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":26.693999999999999},"last":0.55700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738558,"num_pgs":151,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304452,"kb_used_data":3132,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440700,"statfs":{"total":95995035648,"available":95683276800,"internally_reserved":0,"allocated":3207168,"data_stored":1526424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.378,"15min":0.621},"min":{"1min":0.25,"5min":0.191,"15min":0.189},"max":{"1min":0.59999999999999998,"5min":2.0379999999999998,"15min":30.184000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.17599999999999999},"max":{"1min":0.52300000000000002,"5min":5.6959999999999997,"15min":5.6959999999999997},"last":0.35999999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40300000000000002,"15min":0.498},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.20999999999999999},"max":{"1min":0.64300000000000002,"5min":1.3700000000000001,"15min":7.0839999999999996},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42499999999999999,"15min":0.66900000000000004},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.185},"max":{"1min":0.53600000000000003,"5min":3.6949999999999998,"15min":24.459},"last":0.32800000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:50:21 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.42799999999999999,"15min":0.48499999999999999},"min":{"1min":0.32600000000000001,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":0.48499999999999999,"5min":4.8710000000000004,"15min":8.718},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.442,"15min":0.53400000000000003},"min":{"1min":0.315,"5min":0.26900000000000002,"15min":0.222},"max":{"1min":0.47099999999999997,"5min":2.3140000000000001,"15min":7.1139999999999999},"last":0.42399999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.63300000000000001,"15min":0.79900000000000004},"min":{"1min":0.372,"5min":0.29899999999999999,"15min":0.24399999999999999},"max":{"1min":0.874,"5min":7.056,"15min":24.381},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.40500000000000003,"5min":0.31,"15min":0.28699999999999998},"max":{"1min":0.88200000000000001,"5min":6.1859999999999999,"15min":8.6950000000000003},"last":0.59399999999999997}]},{"osd":5,"last update":"Tue Feb 13 00:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.5,"15min":0.495},"min":{"1min":0.40799999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.65400000000000003}]},{"osd":6,"last update":"Tue Feb 13 00:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.35999999999999999,"5min":0.34100000000000003,"15min":0.31},"max":{"1min":0.79500000000000004,"5min":3.302,"15min":8.6129999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.70999999999999996,"15min":0.65100000000000002},"min":{"1min":0.34000000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.89200000000000002,"5min":14.263,"15min":14.263},"last":0.57599999999999996}]},{"osd":7,"last update":"Tue Feb 13 00:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.44700000000000001,"5min":0.35999999999999999,"15min":0.317},"max":{"1min":0.78400000000000003,"5min":7.7720000000000002,"15min":8.6679999999999993},"last":0.628},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58299999999999996,"15min":0.66900000000000004},"min":{"1min":0.44,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91700000000000004,"5min":1.6910000000000001,"15min":8.1609999999999996},"last":0.61699999999999999}]}]},{"osd":2,"up_from":0,"seq":77309411509,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182575,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986347,"num_pgs":144,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304452,"kb_used_data":3132,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440700,"statfs":{"total":95995035648,"available":95683276800,"internally_reserved":0,"allocated":3207168,"data_stored":1534714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.59399999999999997},"min":{"1min":0.34200000000000003,"5min":0.248,"15min":0.21199999999999999},"max":{"1min":0.71999999999999997,"5min":5.3319999999999999,"15min":9.3399999999999999},"last":0.435},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.63500000000000001,"15min":0.60999999999999999},"min":{"1min":0.34499999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.65800000000000003,"5min":11.532,"15min":13.833},"last":0.56999999999999995}]},{"osd":1,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.59799999999999998,"15min":0.753},"min":{"1min":0.373,"5min":0.25,"15min":0.25},"max":{"1min":0.76900000000000002,"5min":7.5410000000000004,"15min":53.188000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.60899999999999999,"15min":0.69999999999999996},"min":{"1min":0.32300000000000001,"5min":0.28999999999999998,"15min":0.25600000000000001},"max":{"1min":0.71099999999999997,"5min":11.497999999999999,"15min":44.793999999999997},"last":0.58599999999999997}]},{"osd":2,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58499999999999996,"15min":0.59699999999999998},"min":{"1min":0.37,"5min":0.31900000000000001,"15min":0.26900000000000002},"max":{"1min":0.78900000000000003,"5min":4.2199999999999998,"15min":8.6829999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.59299999999999997,"15min":0.629},"min":{"1min":0.38300000000000001,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.74299999999999999,"5min":4.0730000000000004,"15min":18.911000000000001},"last":0.46200000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53800000000000003,"15min":0.65400000000000003},"min":{"1min":0.34599999999999997,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64000000000000001,"5min":4.0090000000000003,"15min":23.5},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55100000000000005,"15min":0.71299999999999997},"min":{"1min":0.39400000000000002,"5min":0.32100000000000001,"15min":0.30599999999999999},"max":{"1min":0.68899999999999995,"5min":3.9729999999999999,"15min":36.152999999999999},"last":0.64200000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:50:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.33700000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.69399999999999995,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53300000000000003}]},{"osd":6,"last update":"Tue Feb 13 00:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.46000000000000002,"15min":0.436},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.25700000000000001},"max":{"1min":0.60999999999999999,"5min":3.153,"15min":3.153},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.497,"15min":0.46300000000000002},"min":{"1min":0.32700000000000001,"5min":0.25800000000000001,"15min":0.254},"max":{"1min":0.65900000000000003,"5min":4.9390000000000001,"15min":4.9390000000000001},"last":0.628}]},{"osd":7,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.49299999999999999,"15min":0.48099999999999998},"min":{"1min":0.34000000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.76500000000000001,"5min":1.575,"15min":1.575},"last":0.499},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.5,"15min":0.47699999999999998},"min":{"1min":0.309,"5min":0.29699999999999999,"15min":0.28100000000000003},"max":{"1min":0.72499999999999998,"5min":2.46,"15min":2.46},"last":0.60099999999999998}]}]},{"osd":5,"up_from":0,"seq":2057289334813,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":3328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":236,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":236,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":236,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":236,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:50:28.921 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-13T00:50:28.921 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:50:28.922 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-13T00:50:28.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-13T00:50:29.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:50:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:50:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:50:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:27.461227+0000 mgr.y (mgr.14544) 521 : audit [DBG] from='client.27783 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:50:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:27.723451+0000 mon.a (mon.0) 3128 : audit [INF] from='client.? 172.21.15.162:0/4133962918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:50:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:28.509559+0000 mon.a (mon.0) 3129 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:50:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:28.519352+0000 mon.a (mon.0) 3130 : audit [INF] from='client.? 172.21.15.162:0/4133962918' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:50:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:28.519408+0000 mon.a (mon.0) 3131 : cluster [DBG] osdmap e614: 8 total, 8 up, 4 in 2024-02-13T00:50:29.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:27.461227+0000 mgr.y (mgr.14544) 521 : audit [DBG] from='client.27783 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:50:29.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:27.723451+0000 mon.a (mon.0) 3128 : audit [INF] from='client.? 172.21.15.162:0/4133962918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:50:29.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:28.509559+0000 mon.a (mon.0) 3129 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:50:29.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:28.519352+0000 mon.a (mon.0) 3130 : audit [INF] from='client.? 172.21.15.162:0/4133962918' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:50:29.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:28.519408+0000 mon.a (mon.0) 3131 : cluster [DBG] osdmap e614: 8 total, 8 up, 4 in 2024-02-13T00:50:29.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:27.461227+0000 mgr.y (mgr.14544) 521 : audit [DBG] from='client.27783 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:50:29.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:27.723451+0000 mon.a (mon.0) 3128 : audit [INF] from='client.? 172.21.15.162:0/4133962918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:50:29.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:28.509559+0000 mon.a (mon.0) 3129 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:50:29.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:28.519352+0000 mon.a (mon.0) 3130 : audit [INF] from='client.? 172.21.15.162:0/4133962918' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:50:29.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:28.519408+0000 mon.a (mon.0) 3131 : cluster [DBG] osdmap e614: 8 total, 8 up, 4 in 2024-02-13T00:50:29.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:29.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:29.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:29.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:29.634 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:29.902 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:28.491484+0000 mgr.y (mgr.14544) 522 : cluster [DBG] pgmap v920: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:50:29.903 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:29.516548+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e615: 8 total, 8 up, 4 in 2024-02-13T00:50:29.903 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:29.519028+0000 mon.a (mon.0) 3133 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:50:29.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:28.491484+0000 mgr.y (mgr.14544) 522 : cluster [DBG] pgmap v920: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:50:29.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:29.516548+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e615: 8 total, 8 up, 4 in 2024-02-13T00:50:29.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:29.519028+0000 mon.a (mon.0) 3133 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:50:30.010 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:30.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:28.491484+0000 mgr.y (mgr.14544) 522 : cluster [DBG] pgmap v920: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:50:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:29.516548+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e615: 8 total, 8 up, 4 in 2024-02-13T00:50:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:29.519028+0000 mon.a (mon.0) 3133 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:50:30.114 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:30.149 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:30.170 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:30.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:30.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:30.304 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:30.324 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:30.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:30.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:30.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:30.455 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:31.104 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:31.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:31.207 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:31.239 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:31.260 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:31.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:31.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:31.393 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:31.415 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:31.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:31.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:31.551 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:31.572 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:31.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:31.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:31.669 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:50:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:50:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:50:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:50:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14913.0:2 22.16 22:69efe4bc:test-rados-api-smithi162-84001-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:50:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.531523+0000", 2024-02-13T00:50:31.669 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 570.13227167800005, 2024-02-13T00:50:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.084235272999999999, 2024-02-13T00:50:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14913", 2024-02-13T00:50:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3501757973", 2024-02-13T00:50:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:50:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531523+0000", 2024-02-13T00:50:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531523+0000", 2024-02-13T00:50:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:50:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531521+0000", 2024-02-13T00:50:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1880000000000008e-06 2024-02-13T00:50:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531530+0000", 2024-02-13T00:50:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0994e-05 2024-02-13T00:50:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531541+0000", 2024-02-13T00:50:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4500000000000004e-06 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.531550+0000", 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0084330910000000002 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.539983+0000", 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1860000000000003e-06 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:50:31.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.539990+0000", 2024-02-13T00:50:31.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043643534999999997 2024-02-13T00:50:31.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.583634+0000", 2024-02-13T00:50:31.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011933 2024-02-13T00:50:31.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.583753+0000", 2024-02-13T00:50:31.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000156346 2024-02-13T00:50:31.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,7", 2024-02-13T00:50:31.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.583909+0000", 2024-02-13T00:50:31.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026647634 2024-02-13T00:50:31.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:50:31.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.610557+0000", 2024-02-13T00:50:31.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043537769999999996 2024-02-13T00:50:31.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.614911+0000", 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00078932899999999996 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615700+0000", 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3908000000000001e-05 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615734+0000", 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4465e-05 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615758+0000", 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.084235272999999999 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24745.0:2 24.1b 24:de1a0751:test-rados-api-smithi162-84081-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.540986+0000", 2024-02-13T00:50:31.677 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 570.12280930600002, 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12964152300000001, 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24745", 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/530375479", 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540986+0000", 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540986+0000", 2024-02-13T00:50:31.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:50:31.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540984+0000", 2024-02-13T00:50:31.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.49e-05 2024-02-13T00:50:31.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.540999+0000", 2024-02-13T00:50:31.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0326000000000001e-05 2024-02-13T00:50:31.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.541009+0000", 2024-02-13T00:50:31.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.926e-06 2024-02-13T00:50:31.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.541017+0000", 2024-02-13T00:50:31.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004928562 2024-02-13T00:50:31.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.545945+0000", 2024-02-13T00:50:31.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3000000000000001e-06 2024-02-13T00:50:31.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:50:31.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.545951+0000", 2024-02-13T00:50:31.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.043217637000000003 2024-02-13T00:50:31.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589168+0000", 2024-02-13T00:50:31.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011348799999999999 2024-02-13T00:50:31.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589282+0000", 2024-02-13T00:50:31.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0266999999999995e-05 2024-02-13T00:50:31.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-13T00:50:31.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.589372+0000", 2024-02-13T00:50:31.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020246106999999999 2024-02-13T00:50:31.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:50:31.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.609618+0000", 2024-02-13T00:50:31.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005197513 2024-02-13T00:50:31.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.614816+0000", 2024-02-13T00:50:31.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055733195999999999 2024-02-13T00:50:31.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670549+0000", 2024-02-13T00:50:31.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8872000000000002e-05 2024-02-13T00:50:31.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670598+0000", 2024-02-13T00:50:31.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9230999999999998e-05 2024-02-13T00:50:31.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670627+0000", 2024-02-13T00:50:31.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12964152300000001 2024-02-13T00:50:31.685 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.685 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.685 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.685 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24773.0:2 11.2 11:42a49006:test-rados-api-smithi162-84105-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:50:31.685 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:01.602097+0000", 2024-02-13T00:50:31.686 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 570.06169761599995, 2024-02-13T00:50:31.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101585882, 2024-02-13T00:50:31.686 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.686 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.686 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.686 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24773", 2024-02-13T00:50:31.686 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4011549658", 2024-02-13T00:50:31.686 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:50:31.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.686 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602097+0000", 2024-02-13T00:50:31.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602097+0000", 2024-02-13T00:50:31.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:50:31.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602095+0000", 2024-02-13T00:50:31.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7520000000000002e-05 2024-02-13T00:50:31.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602113+0000", 2024-02-13T00:50:31.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1664999999999999e-05 2024-02-13T00:50:31.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602124+0000", 2024-02-13T00:50:31.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0750000000000004e-06 2024-02-13T00:50:31.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.602133+0000", 2024-02-13T00:50:31.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044129360000000001 2024-02-13T00:50:31.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.606546+0000", 2024-02-13T00:50:31.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126408 2024-02-13T00:50:31.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.606673+0000", 2024-02-13T00:50:31.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002542182 2024-02-13T00:50:31.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,3", 2024-02-13T00:50:31.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.609215+0000", 2024-02-13T00:50:31.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059528159999999997 2024-02-13T00:50:31.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:50:31.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.615168+0000", 2024-02-13T00:50:31.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055524639000000001 2024-02-13T00:50:31.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.670692+0000", 2024-02-13T00:50:31.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032890434000000003 2024-02-13T00:50:31.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703583+0000", 2024-02-13T00:50:31.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5461000000000003e-05 2024-02-13T00:50:31.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703638+0000", 2024-02-13T00:50:31.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4892000000000003e-05 2024-02-13T00:50:31.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:01.703683+0000", 2024-02-13T00:50:31.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101585882 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24911.0:20 16.1 16:83282cb0:::1.obj2:head [write 0~90] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.474569+0000", 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 569.18922627799998, 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.158591963, 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.693 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24911", 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1860840531", 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 20 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474569+0000", 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474569+0000", 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:50:31.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474567+0000", 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9480000000000003e-06 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474577+0000", 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1072e-05 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474588+0000", 2024-02-13T00:50:31.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8759999999999999e-06 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.474595+0000", 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028252126999999998 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.502847+0000", 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013890300000000001 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.502986+0000", 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015514899999999999 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.503141+0000", 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016316878999999999 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.519458+0000", 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058076720999999998 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.577535+0000", 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055542411999999999 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.633077+0000", 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3886e-05 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.633121+0000", 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9544e-05 2024-02-13T00:50:31.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.633161+0000", 2024-02-13T00:50:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.158591963 2024-02-13T00:50:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14886.0:441 20.14 e107/105 20:2c0dcf68:test-rados-api-smithi162-84049-15::foo:head v 107'198, mlcod=107'196)", 2024-02-13T00:50:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:50:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 569.16885891799996, 2024-02-13T00:50:31.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12436195799999999, 2024-02-13T00:50:31.700 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.700 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.700 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:50:31.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:50:31.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:50:31.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494936+0000", 2024-02-13T00:50:31.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989999999999997e-06 2024-02-13T00:50:31.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.494942+0000", 2024-02-13T00:50:31.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037274475000000001 2024-02-13T00:50:31.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.532217+0000", 2024-02-13T00:50:31.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2009999999999999e-06 2024-02-13T00:50:31.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.532225+0000", 2024-02-13T00:50:31.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055477902000000003 2024-02-13T00:50:31.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.587703+0000", 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6648999999999998e-05 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.587729+0000", 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031536963000000001 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.619266+0000", 2024-02-13T00:50:31.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1473999999999997e-05 2024-02-13T00:50:31.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:02.619298+0000", 2024-02-13T00:50:31.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12436195799999999 2024-02-13T00:50:31.704 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.704 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.704 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.704 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2064 20.19 20:9c8ffe76:test-rados-api-smithi162-84049-16::big.copy2:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:50:31.705 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.437684+0000", 2024-02-13T00:50:31.705 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 567.22611134299996, 2024-02-13T00:50:31.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088117519000000005, 2024-02-13T00:50:31.705 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.705 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.705 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.705 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:50:31.705 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:50:31.705 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2064 2024-02-13T00:50:31.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.705 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437684+0000", 2024-02-13T00:50:31.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437684+0000", 2024-02-13T00:50:31.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:50:31.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437682+0000", 2024-02-13T00:50:31.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1730000000000001e-06 2024-02-13T00:50:31.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437691+0000", 2024-02-13T00:50:31.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4664999999999999e-05 2024-02-13T00:50:31.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437706+0000", 2024-02-13T00:50:31.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6229999999999997e-06 2024-02-13T00:50:31.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437713+0000", 2024-02-13T00:50:31.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6378999999999994e-05 2024-02-13T00:50:31.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437800+0000", 2024-02-13T00:50:31.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2805000000000003e-05 2024-02-13T00:50:31.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.437893+0000", 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.060281989000000001 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,5", 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.498175+0000", 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034643450000000002 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.501639+0000", 2024-02-13T00:50:31.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023806853999999999 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525446+0000", 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028838300000000002 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525734+0000", 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3445000000000002e-05 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525778+0000", 2024-02-13T00:50:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3501000000000001e-05 2024-02-13T00:50:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.525801+0000", 2024-02-13T00:50:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088117519000000005 2024-02-13T00:50:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.711 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2069 20.1b 20:de4afda3:test-rados-api-smithi162-84049-17::big:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:50:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.630167+0000", 2024-02-13T00:50:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 567.03362778999997, 2024-02-13T00:50:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11190932200000001, 2024-02-13T00:50:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:50:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:50:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2069 2024-02-13T00:50:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.630167+0000", 2024-02-13T00:50:31.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.630167+0000", 2024-02-13T00:50:31.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:50:31.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.630164+0000", 2024-02-13T00:50:31.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011350433 2024-02-13T00:50:31.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641515+0000", 2024-02-13T00:50:31.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3941e-05 2024-02-13T00:50:31.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641539+0000", 2024-02-13T00:50:31.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6942000000000001e-05 2024-02-13T00:50:31.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641566+0000", 2024-02-13T00:50:31.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0275e-05 2024-02-13T00:50:31.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641646+0000", 2024-02-13T00:50:31.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100653 2024-02-13T00:50:31.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641747+0000", 2024-02-13T00:50:31.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137825 2024-02-13T00:50:31.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,7", 2024-02-13T00:50:31.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.641884+0000", 2024-02-13T00:50:31.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036914244999999998 2024-02-13T00:50:31.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:50:31.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.678799+0000", 2024-02-13T00:50:31.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.062888029999999998 2024-02-13T00:50:31.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.741687+0000", 2024-02-13T00:50:31.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032975399999999997 2024-02-13T00:50:31.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742016+0000", 2024-02-13T00:50:31.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4996000000000002e-05 2024-02-13T00:50:31.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742051+0000", 2024-02-13T00:50:31.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4997000000000001e-05 2024-02-13T00:50:31.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.742076+0000", 2024-02-13T00:50:31.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11190932200000001 2024-02-13T00:50:31.720 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.720 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.720 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.720 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14886.0:2074 20.1b 20:de8a7420:test-rados-api-smithi162-84049-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-13T00:50:31.720 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:04.862849+0000", 2024-02-13T00:50:31.720 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 566.80094596100002, 2024-02-13T00:50:31.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101931788, 2024-02-13T00:50:31.720 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.720 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.720 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.721 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14886", 2024-02-13T00:50:31.721 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3870171401", 2024-02-13T00:50:31.721 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2074 2024-02-13T00:50:31.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.721 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.862849+0000", 2024-02-13T00:50:31.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.862849+0000", 2024-02-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.862846+0000", 2024-02-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070632639999999997 2024-02-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869909+0000", 2024-02-13T00:50:31.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1322e-05 2024-02-13T00:50:31.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869931+0000", 2024-02-13T00:50:31.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0179e-05 2024-02-13T00:50:31.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869941+0000", 2024-02-13T00:50:31.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1819e-05 2024-02-13T00:50:31.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.869993+0000", 2024-02-13T00:50:31.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010545 2024-02-13T00:50:31.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.870098+0000", 2024-02-13T00:50:31.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000146911 2024-02-13T00:50:31.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,7", 2024-02-13T00:50:31.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.870245+0000", 2024-02-13T00:50:31.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029969216 2024-02-13T00:50:31.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:50:31.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.900214+0000", 2024-02-13T00:50:31.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064237111 2024-02-13T00:50:31.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964452+0000", 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025591999999999998 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964707+0000", 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8297999999999998e-05 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964746+0000", 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4975999999999999e-05 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:04.964781+0000", 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101931788 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27739.0:2 68.6 68:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:21.788780+0000", 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 549.87501529500003, 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061004435000000003, 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27739", 2024-02-13T00:50:31.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3559302368", 2024-02-13T00:50:31.728 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2 2024-02-13T00:50:31.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.728 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788780+0000", 2024-02-13T00:50:31.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788780+0000", 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788777+0000", 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2724999999999999e-05 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788789+0000", 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2472e-05 2024-02-13T00:50:31.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788802+0000", 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3860000000000007e-06 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.788810+0000", 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00058719999999999996 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.789398+0000", 2024-02-13T00:50:31.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3919999999999997e-06 2024-02-13T00:50:31.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:50:31.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.789405+0000", 2024-02-13T00:50:31.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047759248999999997 2024-02-13T00:50:31.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.837164+0000", 2024-02-13T00:50:31.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118143 2024-02-13T00:50:31.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.837282+0000", 2024-02-13T00:50:31.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015534899999999999 2024-02-13T00:50:31.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,7", 2024-02-13T00:50:31.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.837438+0000", 2024-02-13T00:50:31.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005545024 2024-02-13T00:50:31.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.842983+0000", 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059063199999999996 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.848889+0000", 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00079681100000000003 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.849686+0000", 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3878999999999997e-05 2024-02-13T00:50:31.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.849740+0000", 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4291e-05 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:21.849784+0000", 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061004435000000003 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.5.0:281 66.6 e128/124 66:64b50987:test-rados-api-smithi162-84081-8::foo:3 v 128'5, mlcod=126'1)", 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:22.771607+0000", 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 548.89218789500001, 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033789000999999999, 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771607+0000", 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771607+0000", 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771606+0000", 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4549999999999998e-06 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771614+0000", 2024-02-13T00:50:31.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7230000000000004e-06 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771622+0000", 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2320000000000004e-06 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.771629+0000", 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012838736 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784468+0000", 2024-02-13T00:50:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1084e-05 2024-02-13T00:50:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.784489+0000", 2024-02-13T00:50:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020873023000000001 2024-02-13T00:50:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.805362+0000", 2024-02-13T00:50:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4317999999999999e-05 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:22.805396+0000", 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033789000999999999 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16275.0:9 66.6 e129/124 66:64b50987:test-rados-api-smithi162-84081-8::foo:head v 129'6, mlcod=127'3)", 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:23.831556+0000", 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 547.83223895000003, 2024-02-13T00:50:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054147344, 2024-02-13T00:50:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831556+0000", 2024-02-13T00:50:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831556+0000", 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831554+0000", 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.209e-05 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831576+0000", 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4198e-05 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831590+0000", 2024-02-13T00:50:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1470000000000004e-06 2024-02-13T00:50:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831597+0000", 2024-02-13T00:50:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051779300000000001 2024-02-13T00:50:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832115+0000", 2024-02-13T00:50:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0034000000000002e-05 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832135+0000", 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053545497999999997 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.885681+0000", 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2527999999999998e-05 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.885703+0000", 2024-02-13T00:50:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054147344 2024-02-13T00:50:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25082.0:19 60.a e129/120 60:51f054f9:test-rados-api-smithi162-84073-8::foo:head v 129'7, mlcod=127'4)", 2024-02-13T00:50:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:50:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 547.83219057099996, 2024-02-13T00:50:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048672976, 2024-02-13T00:50:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:50:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:50:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:50:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831604+0000", 2024-02-13T00:50:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0720000000000001e-06 2024-02-13T00:50:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831608+0000", 2024-02-13T00:50:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9790000000000001e-06 2024-02-13T00:50:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831613+0000", 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.641e-06 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.831617+0000", 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013029420000000001 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832920+0000", 2024-02-13T00:50:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8600000000000001e-05 2024-02-13T00:50:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.832938+0000", 2024-02-13T00:50:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047309385000000002 2024-02-13T00:50:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.880247+0000", 2024-02-13T00:50:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9788e-05 2024-02-13T00:50:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:23.880277+0000", 2024-02-13T00:50:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048672976 2024-02-13T00:50:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16275.0:11 66.6 e130/124 66:629bc174:test-rados-api-smithi162-84081-9::foo:head v 130'7, mlcod=128'5)", 2024-02-13T00:50:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:24.923953+0000", 2024-02-13T00:50:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 546.73984200799998, 2024-02-13T00:50:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.077278500999999999, 2024-02-13T00:50:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923953+0000", 2024-02-13T00:50:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923953+0000", 2024-02-13T00:50:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:50:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923951+0000", 2024-02-13T00:50:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1398e-05 2024-02-13T00:50:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923963+0000", 2024-02-13T00:50:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5424e-05 2024-02-13T00:50:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923978+0000", 2024-02-13T00:50:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4609999999999997e-06 2024-02-13T00:50:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.923988+0000", 2024-02-13T00:50:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075993709000000007 2024-02-13T00:50:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.999981+0000", 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6506000000000001e-05 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:24.999998+0000", 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012042310000000001 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:25.001202+0000", 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9149000000000002e-05 2024-02-13T00:50:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:25.001231+0000", 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.077278500999999999 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25162.0:109 61.1f 61:fec11214:test-rados-api-smithi162-84057-10::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:26.012720+0000", 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 545.65107477900006, 2024-02-13T00:50:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075674577000000007, 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25162", 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 109 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.012720+0000", 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.012720+0000", 2024-02-13T00:50:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:50:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.012718+0000", 2024-02-13T00:50:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6659999999999997e-06 2024-02-13T00:50:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.012727+0000", 2024-02-13T00:50:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042113080000000004 2024-02-13T00:50:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.016938+0000", 2024-02-13T00:50:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0062e-05 2024-02-13T00:50:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.016948+0000", 2024-02-13T00:50:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029517039999999999 2024-02-13T00:50:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.019900+0000", 2024-02-13T00:50:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010618700000000001 2024-02-13T00:50:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.020006+0000", 2024-02-13T00:50:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130426 2024-02-13T00:50:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.020136+0000", 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061208299999999998 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.026257+0000", 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030848077000000002 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.057105+0000", 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031192258 2024-02-13T00:50:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:50:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.088298+0000", 2024-02-13T00:50:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5677000000000002e-05 2024-02-13T00:50:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.088353+0000", 2024-02-13T00:50:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1408000000000001e-05 2024-02-13T00:50:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:26.088395+0000", 2024-02-13T00:50:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075674577000000007 2024-02-13T00:50:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(osd.5.0:293 66.6 e132/124 66:629bc174:test-rados-api-smithi162-84081-9::foo:7 v 132'15, mlcod=131'12)", 2024-02-13T00:50:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:27.094916+0000", 2024-02-13T00:50:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 544.56887936700002, 2024-02-13T00:50:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082889299, 2024-02-13T00:50:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094916+0000", 2024-02-13T00:50:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094916+0000", 2024-02-13T00:50:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:50:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094915+0000", 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2129999999999999e-06 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094920+0000", 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1179999999999997e-06 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094927+0000", 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7989999999999996e-06 2024-02-13T00:50:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.094933+0000", 2024-02-13T00:50:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6577999999999996e-05 2024-02-13T00:50:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.095030+0000", 2024-02-13T00:50:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1804e-05 2024-02-13T00:50:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.095052+0000", 2024-02-13T00:50:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082724803 2024-02-13T00:50:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.177777+0000", 2024-02-13T00:50:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8288000000000001e-05 2024-02-13T00:50:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:27.177805+0000", 2024-02-13T00:50:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082889299 2024-02-13T00:50:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.19047.0:9 78.0s1 78:1b3d1cfa:test-rados-api-smithi162-84073-12::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected e139)", 2024-02-13T00:50:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:34.355949+0000", 2024-02-13T00:50:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 537.30784546300004, 2024-02-13T00:50:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056363352999999998, 2024-02-13T00:50:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.19047", 2024-02-13T00:50:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3520430640", 2024-02-13T00:50:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 9 2024-02-13T00:50:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355949+0000", 2024-02-13T00:50:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355949+0000", 2024-02-13T00:50:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999919 2024-02-13T00:50:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355941+0000", 2024-02-13T00:50:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4331000000000001e-05 2024-02-13T00:50:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355956+0000", 2024-02-13T00:50:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0973000000000001e-05 2024-02-13T00:50:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355977+0000", 2024-02-13T00:50:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2830000000000001e-06 2024-02-13T00:50:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.355985+0000", 2024-02-13T00:50:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052008540000000004 2024-02-13T00:50:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.361186+0000", 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7719000000000005e-05 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.361274+0000", 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059102099999999995 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.361865+0000", 2024-02-13T00:50:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050108890000000003 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.411974+0000", 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7391000000000006e-05 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.412041+0000", 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027185499999999999 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.412313+0000", 2024-02-13T00:50:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056363352999999998 2024-02-13T00:50:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDECSubOpWriteReply(78.0s1 139/133 ECSubWriteReply(tid=284, last_complete=139'3, committed=1, applied=1))", 2024-02-13T00:50:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:34.413870+0000", 2024-02-13T00:50:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 537.24992491, 2024-02-13T00:50:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033827619000000003, 2024-02-13T00:50:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413870+0000", 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413870+0000", 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999967 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413867+0000", 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0979999999999996e-06 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413874+0000", 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3100000000000001e-06 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413882+0000", 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.49e-06 2024-02-13T00:50:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.413887+0000", 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033771486000000003 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.447659+0000", 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8617e-05 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.447698+0000", 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033827619000000003 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.19047.0:10 78.0s1 78:1b3d1cfa:test-rados-api-smithi162-84073-12::foo:head [rollback 5] snapc 0=[] ondisk+write+known_if_redirected e139)", 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:41:34.415564+0000", 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 537.24823127499997, 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094033364999999994, 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.19047", 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3520430640", 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 10 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415564+0000", 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415564+0000", 2024-02-13T00:50:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415561+0000", 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6379999999999999e-06 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415571+0000", 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4296e-05 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415585+0000", 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6030000000000005e-06 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.415594+0000", 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032129234 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.447723+0000", 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010490200000000001 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.447828+0000", 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044360579999999997 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.452264+0000", 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00171576 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.453980+0000", 2024-02-13T00:50:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055540957000000002 2024-02-13T00:50:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.509521+0000", 2024-02-13T00:50:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6073000000000002e-05 2024-02-13T00:50:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:41:34.509597+0000", 2024-02-13T00:50:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094033364999999994 2024-02-13T00:50:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.37121.0:1 185.2s0 185:5c55be4c:::oid_RoundTripAppendPP_:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected e298)", 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:13.388163+0000", 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 378.27563151700002, 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0126707619999999, 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.37121", 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/695397089", 2024-02-13T00:50:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.388163+0000", 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.388163+0000", 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.388162+0000", 2024-02-13T00:50:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3216e-05 2024-02-13T00:50:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.388175+0000", 2024-02-13T00:50:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7540000000000001e-06 2024-02-13T00:50:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.388185+0000", 2024-02-13T00:50:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.123e-06 2024-02-13T00:50:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.388193+0000", 2024-02-13T00:50:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021198469999999998 2024-02-13T00:50:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.390313+0000", 2024-02-13T00:50:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0290000000000001e-06 2024-02-13T00:50:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:13.390316+0000", 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.007525633 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:14.397841+0000", 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010964299999999999 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:14.397951+0000", 2024-02-13T00:50:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037275699999999998 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:14.398324+0000", 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021763870000000001 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:14.400500+0000", 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3350999999999995e-05 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:14.400583+0000", 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025063499999999997 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:14.400834+0000", 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0126707619999999 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27024.0:1 187.6s1 187:72c88bc2:::oid_IsCompletePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e305)", 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:20.386298+0000", 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 371.27749718199999, 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99356510300000001, 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27024", 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3998292204", 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386298+0000", 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386298+0000", 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386295+0000", 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7630000000000003e-06 2024-02-13T00:50:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:50:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386302+0000", 2024-02-13T00:50:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6260000000000007e-06 2024-02-13T00:50:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:50:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386312+0000", 2024-02-13T00:50:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8469999999999996e-06 2024-02-13T00:50:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:50:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386320+0000", 2024-02-13T00:50:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1579000000000001e-05 2024-02-13T00:50:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386381+0000", 2024-02-13T00:50:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8029999999999998e-06 2024-02-13T00:50:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:50:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:20.386385+0000", 2024-02-13T00:50:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99081751399999995 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:21.377203+0000", 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130801 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:21.377333+0000", 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00043861399999999999 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:50:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:21.377772+0000", 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00084405500000000002 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:21.378616+0000", 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00118315 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:21.379799+0000", 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3575e-05 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:50:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:21.379863+0000", 2024-02-13T00:50:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99356510300000001 2024-02-13T00:50:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:31.795 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:50:31.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:31.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:30.492309+0000 mgr.y (mgr.14544) 523 : cluster [DBG] pgmap v923: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:31.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:30.517549+0000 mon.a (mon.0) 3134 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-106","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:50:31.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:30.517649+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e616: 8 total, 8 up, 4 in 2024-02-13T00:50:31.797 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:30.492309+0000 mgr.y (mgr.14544) 523 : cluster [DBG] pgmap v923: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:31.797 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:30.517549+0000 mon.a (mon.0) 3134 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-106","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:50:31.797 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:30.517649+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e616: 8 total, 8 up, 4 in 2024-02-13T00:50:31.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:30.492309+0000 mgr.y (mgr.14544) 523 : cluster [DBG] pgmap v923: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:31.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:30.517549+0000 mon.a (mon.0) 3134 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-106","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:50:31.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:30.517649+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e616: 8 total, 8 up, 4 in 2024-02-13T00:50:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:32.722 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:32.792 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:31.519345+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e617: 8 total, 8 up, 4 in 2024-02-13T00:50:32.792 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:31.535348+0000 mon.a (mon.0) 3137 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:50:32.792 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:31.519345+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e617: 8 total, 8 up, 4 in 2024-02-13T00:50:32.792 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:31.535348+0000 mon.a (mon.0) 3137 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:50:32.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:31.519345+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e617: 8 total, 8 up, 4 in 2024-02-13T00:50:32.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:31.535348+0000 mon.a (mon.0) 3137 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:50:33.104 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:50:33.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-13T00:50:33.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.673+0000 7efd7a4cd700 1 -- 172.21.15.162:0/68292134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd740f73a0 msgr2=0x7efd741039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.673+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/68292134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd740f73a0 0x7efd741039a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7efd6c009c30 tx=0x7efd6c004c50).stop 2024-02-13T00:50:33.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.673+0000 7efd7a4cd700 1 -- 172.21.15.162:0/68292134 shutdown_connections 2024-02-13T00:50:33.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.673+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/68292134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd740f73a0 0x7efd741039a0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.673+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/68292134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd740fcb30 0x7efd740fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.673+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/68292134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd740fd490 0x7efd740f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.673+0000 7efd7a4cd700 1 -- 172.21.15.162:0/68292134 >> 172.21.15.162:0/68292134 conn(0x7efd7406eb90 msgr2=0x7efd740f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.673+0000 7efd7a4cd700 1 -- 172.21.15.162:0/68292134 shutdown_connections 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.673+0000 7efd7a4cd700 1 -- 172.21.15.162:0/68292134 wait complete. 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.674+0000 7efd7a4cd700 1 librados: starting msgr at 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.674+0000 7efd7a4cd700 1 librados: starting objecter 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.674+0000 7efd7a4cd700 1 Processor -- start 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.674+0000 7efd7a4cd700 1 -- start start 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.674+0000 7efd7a4cd700 1 librados: setting wanted keys 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.674+0000 7efd7a4cd700 1 librados: calling monclient init 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.674+0000 7efd7a4cd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd740f73a0 0x7efd740f8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.674+0000 7efd7a4cd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd740fcb30 0x7efd740f91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.674+0000 7efd7a4cd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd740fd490 0x7efd740f96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.674+0000 7efd7a4cd700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74116490 con 0x7efd740fd490 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.674+0000 7efd7a4cd700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74116600 con 0x7efd740fcb30 2024-02-13T00:50:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.674+0000 7efd7a4cd700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd741167c0 con 0x7efd740f73a0 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd73fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd740f73a0 0x7efd740f8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd73fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd740f73a0 0x7efd740f8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:50638/0 (socket says 172.21.15.162:50638) 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd73fff700 1 -- 172.21.15.162:0/2917148586 learned_addr learned my addr 172.21.15.162:0/2917148586 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd78a6a700 1 --2- 172.21.15.162:0/2917148586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd740fd490 0x7efd740f96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd737fe700 1 --2- 172.21.15.162:0/2917148586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd740fcb30 0x7efd740f91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd73fff700 1 -- 172.21.15.162:0/2917148586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd740fcb30 msgr2=0x7efd740f91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd73fff700 1 --2- 172.21.15.162:0/2917148586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd740fcb30 0x7efd740f91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd73fff700 1 -- 172.21.15.162:0/2917148586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd740fd490 msgr2=0x7efd740f96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd73fff700 1 --2- 172.21.15.162:0/2917148586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd740fd490 0x7efd740f96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd73fff700 1 -- 172.21.15.162:0/2917148586 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd68005040 con 0x7efd740f73a0 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd73fff700 1 --2- 172.21.15.162:0/2917148586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd740f73a0 0x7efd740f8cc0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 rx=0x7efd6400e970 tx=0x7efd6400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd717fa700 10 client.?.objecter ms_handle_connect 0x7efd740f73a0 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd717fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6400ee60 con 0x7efd740f73a0 2024-02-13T00:50:33.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd6400f040 con 0x7efd740f73a0 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd6c0097c0 con 0x7efd740f73a0 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64007ea0 con 0x7efd740f73a0 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd7a4cd700 10 client.37895.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd74116ed0 con 0x7efd740f73a0 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.675+0000 7efd7a4cd700 1 librados: init done 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.676+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7efd74106090 con 0x7efd740f73a0 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.676+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7efd6400cb10 con 0x7efd740f73a0 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd717fa700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6800/2866534053 conn(0x7efd5c062c90 0x7efd5c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(619..619 src has 449..619) v4 ==== 8470+0+0 (secure 0 0 0) 0x7efd6407cf90 con 0x7efd740f73a0 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd737fe700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6800/2866534053 conn(0x7efd5c062c90 0x7efd5c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 osd_map(619..619 src has 449..619) v4 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd717fa700 3 client.37895.objecter handle_osd_map got epochs [619,619] > 0 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd717fa700 3 client.37895.objecter handle_osd_map decoding full epoch 619 2024-02-13T00:50:33.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd717fa700 20 client.37895.objecter dump_active .. 0 homeless 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v619) v1 ==== 54+0+16 (secure 0 0 0) 0x7efd6407f620 con 0x7efd740f73a0 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd717fa700 10 client.37895.objecter ms_handle_connect 0x7efd5c062c90 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd737fe700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6800/2866534053 conn(0x7efd5c062c90 0x7efd5c065150 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 rx=0x7efd740fa460 tx=0x7efd68009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 1 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6801/2103743338 conn(0x7efd74059990 0x7efd74059db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd740fa1a0 osd=0 3 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd740fa1a0 osd=0 3 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd740fa1a0 osd=0 3 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 1 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd740fa1a0 osd=0 3 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 0 1 2024-02-13T00:50:33.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd740fa1a0 osd=0 4 2024-02-13T00:50:33.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd7a4cd700 10 client.37895.objecter _send_command 1 2024-02-13T00:50:33.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd7405fb80 con 0x7efd74059990 2024-02-13T00:50:33.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.677+0000 7efd78a6a700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6801/2103743338 conn(0x7efd74059990 0x7efd74059db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:33.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.678+0000 7efd717fa700 10 client.37895.objecter ms_handle_connect 0x7efd74059990 2024-02-13T00:50:33.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.678+0000 7efd78a6a700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6801/2103743338 conn(0x7efd74059990 0x7efd74059db0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:33.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.678+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd7405fb80 con 0x7efd74059990 2024-02-13T00:50:33.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.678+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:50:33.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.678+0000 7efd717fa700 10 client.37895.objecter _finish_command 1 = system:0 2024-02-13T00:50:33.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.678+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd740fa1a0 osd=0 4 2024-02-13T00:50:33.693 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.678+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 0 1 2024-02-13T00:50:33.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.693+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:33.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.693+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 2 2024-02-13T00:50:33.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.693+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd740fa1a0 osd=0 3 2024-02-13T00:50:33.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.693+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd740fa1a0 osd=0 3 2024-02-13T00:50:33.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.693+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd740fa1a0 osd=0 3 2024-02-13T00:50:33.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.693+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 2 2024-02-13T00:50:33.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.693+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd740fa1a0 osd=0 3 2024-02-13T00:50:33.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.693+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 0 2 2024-02-13T00:50:33.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.693+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd740fa1a0 osd=0 4 2024-02-13T00:50:33.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.693+0000 7efd7a4cd700 10 client.37895.objecter _send_command 2 2024-02-13T00:50:33.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.693+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd74028590 con 0x7efd74059990 2024-02-13T00:50:33.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.694+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd74028590 con 0x7efd74059990 2024-02-13T00:50:33.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.694+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:33.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.694+0000 7efd717fa700 10 client.37895.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.694+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd740fa1a0 osd=0 4 2024-02-13T00:50:33.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.694+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 0 2 2024-02-13T00:50:33.698 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:50:33.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:33.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 3 2024-02-13T00:50:33.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6805/150599780 conn(0x7efd74059570 0x7efd7411b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:33.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd74059400 osd=1 3 2024-02-13T00:50:33.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74059400 osd=1 3 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd74059400 osd=1 3 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 3 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd74059400 osd=1 3 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 1 3 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74059400 osd=1 4 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd7a4cd700 10 client.37895.objecter _send_command 3 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd741005c0 con 0x7efd74059570 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd73fff700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6805/150599780 conn(0x7efd74059570 0x7efd7411b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd717fa700 10 client.37895.objecter ms_handle_connect 0x7efd74059570 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.695+0000 7efd73fff700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6805/150599780 conn(0x7efd74059570 0x7efd7411b080 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.696+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd741005c0 con 0x7efd74059570 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.696+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:50:33.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.696+0000 7efd717fa700 10 client.37895.objecter _finish_command 3 = system:0 2024-02-13T00:50:33.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.696+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd74059400 osd=1 4 2024-02-13T00:50:33.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.696+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 1 3 2024-02-13T00:50:33.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.710+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:33.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.710+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 4 2024-02-13T00:50:33.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.710+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd74059400 osd=1 3 2024-02-13T00:50:33.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.710+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74059400 osd=1 3 2024-02-13T00:50:33.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.710+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd74059400 osd=1 3 2024-02-13T00:50:33.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.710+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 4 2024-02-13T00:50:33.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.710+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd74059400 osd=1 3 2024-02-13T00:50:33.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.710+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 1 4 2024-02-13T00:50:33.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.710+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74059400 osd=1 4 2024-02-13T00:50:33.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.710+0000 7efd7a4cd700 10 client.37895.objecter _send_command 4 2024-02-13T00:50:33.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.710+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd7411b570 con 0x7efd74059570 2024-02-13T00:50:33.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd7411b570 con 0x7efd74059570 2024-02-13T00:50:33.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:33.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd717fa700 10 client.37895.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd74059400 osd=1 4 2024-02-13T00:50:33.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 1 4 2024-02-13T00:50:33.716 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 5 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6809/3689719376 conn(0x7efd740ffc90 0x7efd7411e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd740ffb20 osd=2 3 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd740ffb20 osd=2 3 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd740ffb20 osd=2 3 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 5 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd740ffb20 osd=2 3 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 2 5 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd740ffb20 osd=2 4 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd7a4cd700 10 client.37895.objecter _send_command 5 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd740280f0 con 0x7efd740ffc90 2024-02-13T00:50:33.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd737fe700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6809/3689719376 conn(0x7efd740ffc90 0x7efd7411e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:33.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd717fa700 10 client.37895.objecter ms_handle_connect 0x7efd740ffc90 2024-02-13T00:50:33.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.712+0000 7efd737fe700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6809/3689719376 conn(0x7efd740ffc90 0x7efd7411e8d0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:33.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.713+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd740280f0 con 0x7efd740ffc90 2024-02-13T00:50:33.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.713+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:50:33.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.713+0000 7efd717fa700 10 client.37895.objecter _finish_command 5 = system:0 2024-02-13T00:50:33.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.713+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd740ffb20 osd=2 4 2024-02-13T00:50:33.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.713+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 2 5 2024-02-13T00:50:33.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.727+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:33.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.727+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 6 2024-02-13T00:50:33.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.727+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd740ffb20 osd=2 3 2024-02-13T00:50:33.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.727+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd740ffb20 osd=2 3 2024-02-13T00:50:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.727+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd740ffb20 osd=2 3 2024-02-13T00:50:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.727+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 6 2024-02-13T00:50:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.727+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd740ffb20 osd=2 3 2024-02-13T00:50:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.727+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 2 6 2024-02-13T00:50:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.727+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd740ffb20 osd=2 4 2024-02-13T00:50:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.727+0000 7efd7a4cd700 10 client.37895.objecter _send_command 6 2024-02-13T00:50:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.727+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd7411edc0 con 0x7efd740ffc90 2024-02-13T00:50:33.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd7411edc0 con 0x7efd740ffc90 2024-02-13T00:50:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd717fa700 10 client.37895.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd740ffb20 osd=2 4 2024-02-13T00:50:33.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 2 6 2024-02-13T00:50:33.733 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:50:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 7 2024-02-13T00:50:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6813/604971011 conn(0x7efd740277c0 0x7efd74122120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd74027650 osd=3 3 2024-02-13T00:50:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74027650 osd=3 3 2024-02-13T00:50:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd74027650 osd=3 3 2024-02-13T00:50:33.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 7 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd74027650 osd=3 3 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 3 7 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74027650 osd=3 4 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd7a4cd700 10 client.37895.objecter _send_command 7 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd7411c5e0 con 0x7efd740277c0 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd78a6a700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6813/604971011 conn(0x7efd740277c0 0x7efd74122120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd717fa700 10 client.37895.objecter ms_handle_connect 0x7efd740277c0 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.729+0000 7efd78a6a700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6813/604971011 conn(0x7efd740277c0 0x7efd74122120 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.730+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd7411c5e0 con 0x7efd740277c0 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.730+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.730+0000 7efd717fa700 10 client.37895.objecter _finish_command 7 = system:0 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.730+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd74027650 osd=3 4 2024-02-13T00:50:33.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.730+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 3 7 2024-02-13T00:50:33.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.744+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:33.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.744+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 8 2024-02-13T00:50:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.744+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd74027650 osd=3 3 2024-02-13T00:50:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.744+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74027650 osd=3 3 2024-02-13T00:50:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.744+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd74027650 osd=3 3 2024-02-13T00:50:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.744+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 8 2024-02-13T00:50:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.744+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd74027650 osd=3 3 2024-02-13T00:50:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.744+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 3 8 2024-02-13T00:50:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.744+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74027650 osd=3 4 2024-02-13T00:50:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.744+0000 7efd7a4cd700 10 client.37895.objecter _send_command 8 2024-02-13T00:50:33.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.744+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd7411b980 con 0x7efd740277c0 2024-02-13T00:50:33.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd7411b980 con 0x7efd740277c0 2024-02-13T00:50:33.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:33.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd717fa700 10 client.37895.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd74027650 osd=3 4 2024-02-13T00:50:33.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 3 8 2024-02-13T00:50:33.749 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:50:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 9 2024-02-13T00:50:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6800/3344263388 conn(0x7efd7411fc10 0x7efd741257a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:33.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd7411b980 osd=4 3 2024-02-13T00:50:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411b980 osd=4 3 2024-02-13T00:50:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd7411b980 osd=4 3 2024-02-13T00:50:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 9 2024-02-13T00:50:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd7411b980 osd=4 3 2024-02-13T00:50:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 4 9 2024-02-13T00:50:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411b980 osd=4 4 2024-02-13T00:50:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd7a4cd700 10 client.37895.objecter _send_command 9 2024-02-13T00:50:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd7411bb50 con 0x7efd7411fc10 2024-02-13T00:50:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd73fff700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6800/3344263388 conn(0x7efd7411fc10 0x7efd741257a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd73fff700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6800/3344263388 conn(0x7efd7411fc10 0x7efd741257a0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:33.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.746+0000 7efd717fa700 10 client.37895.objecter ms_handle_connect 0x7efd7411fc10 2024-02-13T00:50:33.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.747+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd7411bb50 con 0x7efd7411fc10 2024-02-13T00:50:33.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.747+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:50:33.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.747+0000 7efd717fa700 10 client.37895.objecter _finish_command 9 = system:0 2024-02-13T00:50:33.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.747+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd7411b980 osd=4 4 2024-02-13T00:50:33.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.747+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 4 9 2024-02-13T00:50:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.762+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.762+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 10 2024-02-13T00:50:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.762+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd7411b980 osd=4 3 2024-02-13T00:50:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.762+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411b980 osd=4 3 2024-02-13T00:50:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.762+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd7411b980 osd=4 3 2024-02-13T00:50:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.762+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 10 2024-02-13T00:50:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.762+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd7411b980 osd=4 3 2024-02-13T00:50:33.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.762+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 4 10 2024-02-13T00:50:33.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.762+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411b980 osd=4 4 2024-02-13T00:50:33.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.762+0000 7efd7a4cd700 10 client.37895.objecter _send_command 10 2024-02-13T00:50:33.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.762+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd7411bb50 con 0x7efd7411fc10 2024-02-13T00:50:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd7411bb50 con 0x7efd7411fc10 2024-02-13T00:50:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd717fa700 10 client.37895.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd7411b980 osd=4 4 2024-02-13T00:50:33.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 4 10 2024-02-13T00:50:33.768 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:50:33.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:33.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 11 2024-02-13T00:50:33.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6804/2277917372 conn(0x7efd741261a0 0x7efd741285f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd7411bda0 osd=5 3 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411bda0 osd=5 3 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd7411bda0 osd=5 3 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 11 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd7411bda0 osd=5 3 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 5 11 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411bda0 osd=5 4 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd7a4cd700 10 client.37895.objecter _send_command 11 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd7411bb50 con 0x7efd741261a0 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd737fe700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6804/2277917372 conn(0x7efd741261a0 0x7efd741285f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.764+0000 7efd737fe700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6804/2277917372 conn(0x7efd741261a0 0x7efd741285f0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.765+0000 7efd717fa700 10 client.37895.objecter ms_handle_connect 0x7efd741261a0 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.765+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd7411bb50 con 0x7efd741261a0 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.765+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:50:33.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.765+0000 7efd717fa700 10 client.37895.objecter _finish_command 11 = system:0 2024-02-13T00:50:33.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.765+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd7411bda0 osd=5 4 2024-02-13T00:50:33.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.765+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 5 11 2024-02-13T00:50:33.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.781+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:33.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.781+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 12 2024-02-13T00:50:33.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.781+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd7411bda0 osd=5 3 2024-02-13T00:50:33.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.781+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411bda0 osd=5 3 2024-02-13T00:50:33.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.781+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd7411bda0 osd=5 3 2024-02-13T00:50:33.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.781+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 12 2024-02-13T00:50:33.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.781+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd7411bda0 osd=5 3 2024-02-13T00:50:33.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.781+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 5 12 2024-02-13T00:50:33.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.781+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411bda0 osd=5 4 2024-02-13T00:50:33.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.781+0000 7efd7a4cd700 10 client.37895.objecter _send_command 12 2024-02-13T00:50:33.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.781+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd7411bb50 con 0x7efd741261a0 2024-02-13T00:50:33.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd7411bb50 con 0x7efd741261a0 2024-02-13T00:50:33.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:33.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd717fa700 10 client.37895.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd7411bda0 osd=5 4 2024-02-13T00:50:33.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 5 12 2024-02-13T00:50:33.787 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:50:33.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:33.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 13 2024-02-13T00:50:33.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6808/1506640341 conn(0x7efd7412a700 0x7efd7412ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:33.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd7411bb50 osd=6 3 2024-02-13T00:50:33.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411bb50 osd=6 3 2024-02-13T00:50:33.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd7411bb50 osd=6 3 2024-02-13T00:50:33.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 13 2024-02-13T00:50:33.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd7411bb50 osd=6 3 2024-02-13T00:50:33.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 6 13 2024-02-13T00:50:33.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411bb50 osd=6 4 2024-02-13T00:50:33.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd7a4cd700 10 client.37895.objecter _send_command 13 2024-02-13T00:50:33.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.783+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd7412e630 con 0x7efd7412a700 2024-02-13T00:50:33.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.784+0000 7efd78a6a700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6808/1506640341 conn(0x7efd7412a700 0x7efd7412ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:33.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.784+0000 7efd717fa700 10 client.37895.objecter ms_handle_connect 0x7efd7412a700 2024-02-13T00:50:33.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.784+0000 7efd78a6a700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6808/1506640341 conn(0x7efd7412a700 0x7efd7412ab80 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:33.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.785+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd7412e630 con 0x7efd7412a700 2024-02-13T00:50:33.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.785+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:50:33.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.785+0000 7efd717fa700 10 client.37895.objecter _finish_command 13 = system:0 2024-02-13T00:50:33.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.785+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd7411bb50 osd=6 4 2024-02-13T00:50:33.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.785+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 6 13 2024-02-13T00:50:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:32.099640+0000 osd.0 (osd.0) 235 : cluster [DBG] 6.29 scrub starts 2024-02-13T00:50:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:32.100603+0000 osd.0 (osd.0) 236 : cluster [DBG] 6.29 scrub ok 2024-02-13T00:50:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:32.492737+0000 mgr.y (mgr.14544) 524 : cluster [DBG] pgmap v926: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:32.533629+0000 mon.a (mon.0) 3138 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-106", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:50:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:32.533701+0000 mon.a (mon.0) 3139 : cluster [DBG] osdmap e618: 8 total, 8 up, 4 in 2024-02-13T00:50:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:32.535262+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-106"}]: dispatch 2024-02-13T00:50:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:32.771192+0000 osd.1 (osd.1) 259 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:50:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:32.774816+0000 osd.1 (osd.1) 260 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:50:33.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.800+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:33.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.800+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 14 2024-02-13T00:50:33.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.800+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd7411bb50 osd=6 3 2024-02-13T00:50:33.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.800+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411bb50 osd=6 3 2024-02-13T00:50:33.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.800+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd7411bb50 osd=6 3 2024-02-13T00:50:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.800+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 14 2024-02-13T00:50:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.800+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd7411bb50 osd=6 3 2024-02-13T00:50:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.800+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 6 14 2024-02-13T00:50:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.800+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411bb50 osd=6 4 2024-02-13T00:50:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.800+0000 7efd7a4cd700 10 client.37895.objecter _send_command 14 2024-02-13T00:50:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.800+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd74128ba0 con 0x7efd7412a700 2024-02-13T00:50:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd74128ba0 con 0x7efd7412a700 2024-02-13T00:50:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd717fa700 10 client.37895.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd7411bb50 osd=6 4 2024-02-13T00:50:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 6 14 2024-02-13T00:50:33.806 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:50:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 15 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6812/1964192772 conn(0x7efd74129250 0x7efd741296b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd74128ba0 osd=7 3 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74128ba0 osd=7 3 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd74128ba0 osd=7 3 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 15 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd74128ba0 osd=7 3 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 7 15 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74128ba0 osd=7 4 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd7a4cd700 10 client.37895.objecter _send_command 15 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.802+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd74129ca0 con 0x7efd74129250 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.803+0000 7efd73fff700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6812/1964192772 conn(0x7efd74129250 0x7efd741296b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.803+0000 7efd73fff700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6812/1964192772 conn(0x7efd74129250 0x7efd741296b0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.803+0000 7efd717fa700 10 client.37895.objecter ms_handle_connect 0x7efd74129250 2024-02-13T00:50:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.804+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efd74129ca0 con 0x7efd74129250 2024-02-13T00:50:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.804+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:50:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.804+0000 7efd717fa700 10 client.37895.objecter _finish_command 15 = system:0 2024-02-13T00:50:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.804+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd74128ba0 osd=7 4 2024-02-13T00:50:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.804+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 7 15 2024-02-13T00:50:33.810 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:32.099640+0000 osd.0 (osd.0) 235 : cluster [DBG] 6.29 scrub starts 2024-02-13T00:50:33.811 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:32.100603+0000 osd.0 (osd.0) 236 : cluster [DBG] 6.29 scrub ok 2024-02-13T00:50:33.811 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:32.492737+0000 mgr.y (mgr.14544) 524 : cluster [DBG] pgmap v926: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:33.811 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:32.533629+0000 mon.a (mon.0) 3138 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-106", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:50:33.811 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:32.533701+0000 mon.a (mon.0) 3139 : cluster [DBG] osdmap e618: 8 total, 8 up, 4 in 2024-02-13T00:50:33.811 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:32.535262+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-106"}]: dispatch 2024-02-13T00:50:33.811 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:32.771192+0000 osd.1 (osd.1) 259 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:50:33.811 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:32.774816+0000 osd.1 (osd.1) 260 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:50:33.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:32.099640+0000 osd.0 (osd.0) 235 : cluster [DBG] 6.29 scrub starts 2024-02-13T00:50:33.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:32.100603+0000 osd.0 (osd.0) 236 : cluster [DBG] 6.29 scrub ok 2024-02-13T00:50:33.811 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:32.492737+0000 mgr.y (mgr.14544) 524 : cluster [DBG] pgmap v926: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:33.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:32.533629+0000 mon.a (mon.0) 3138 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-106", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:50:33.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:32.533701+0000 mon.a (mon.0) 3139 : cluster [DBG] osdmap e618: 8 total, 8 up, 4 in 2024-02-13T00:50:33.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:32.535262+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-106"}]: dispatch 2024-02-13T00:50:33.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:32.771192+0000 osd.1 (osd.1) 259 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:50:33.812 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:32.774816+0000 osd.1 (osd.1) 260 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:50:33.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.818+0000 7efd7a4cd700 10 client.37895.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:33.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.818+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign -1 16 2024-02-13T00:50:33.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.818+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd74128ba0 osd=7 3 2024-02-13T00:50:33.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.818+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74128ba0 osd=7 3 2024-02-13T00:50:33.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.818+0000 7efd7a4cd700 20 client.37895.objecter _get_session s=0x7efd74128ba0 osd=7 3 2024-02-13T00:50:33.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.818+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_remove -1 16 2024-02-13T00:50:33.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.818+0000 7efd7a4cd700 20 client.37895.objecter get_session s=0x7efd74128ba0 osd=7 3 2024-02-13T00:50:33.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.818+0000 7efd7a4cd700 15 client.37895.objecter _session_command_op_assign 7 16 2024-02-13T00:50:33.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.818+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74128ba0 osd=7 4 2024-02-13T00:50:33.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.818+0000 7efd7a4cd700 10 client.37895.objecter _send_command 16 2024-02-13T00:50:33.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.818+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efd74129ca0 con 0x7efd74129250 2024-02-13T00:50:33.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.820+0000 7efd717fa700 1 -- 172.21.15.162:0/2917148586 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7efd74129ca0 con 0x7efd74129250 2024-02-13T00:50:33.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.820+0000 7efd717fa700 10 client.37895.objecter ms_dispatch 0x7efd74001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:33.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.820+0000 7efd717fa700 10 client.37895.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.820+0000 7efd717fa700 20 client.37895.objecter put_session s=0x7efd74128ba0 osd=7 4 2024-02-13T00:50:33.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.820+0000 7efd717fa700 15 client.37895.objecter _session_command_op_remove 7 16 2024-02-13T00:50:33.825 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:50:33.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 10 librados: watch_flush enter 2024-02-13T00:50:33.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 10 librados: watch_flush exit 2024-02-13T00:50:33.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 10 client.37895.objecter close_session for osd.0 2024-02-13T00:50:33.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6801/2103743338 conn(0x7efd74059990 msgr2=0x7efd74059db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6801/2103743338 conn(0x7efd74059990 0x7efd74059db0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd740fa1a0 osd=0 1 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 10 client.37895.objecter close_session for osd.1 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6805/150599780 conn(0x7efd74059570 msgr2=0x7efd7411b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6805/150599780 conn(0x7efd74059570 0x7efd7411b080 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74059400 osd=1 1 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 10 client.37895.objecter close_session for osd.2 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6809/3689719376 conn(0x7efd740ffc90 msgr2=0x7efd7411e8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6809/3689719376 conn(0x7efd740ffc90 0x7efd7411e8d0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd740ffb20 osd=2 1 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 10 client.37895.objecter close_session for osd.3 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6813/604971011 conn(0x7efd740277c0 msgr2=0x7efd74122120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6813/604971011 conn(0x7efd740277c0 0x7efd74122120 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74027650 osd=3 1 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 10 client.37895.objecter close_session for osd.4 2024-02-13T00:50:33.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6800/3344263388 conn(0x7efd7411fc10 msgr2=0x7efd741257a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6800/3344263388 conn(0x7efd7411fc10 0x7efd741257a0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd78a6a700 1 -- 172.21.15.162:0/2917148586 reap_dead start 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411b980 osd=4 1 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 10 client.37895.objecter close_session for osd.5 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6804/2277917372 conn(0x7efd741261a0 msgr2=0x7efd741285f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6804/2277917372 conn(0x7efd741261a0 0x7efd741285f0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd78a6a700 1 -- 172.21.15.162:0/2917148586 reap_dead start 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411bda0 osd=5 1 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 10 client.37895.objecter close_session for osd.6 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6808/1506640341 conn(0x7efd7412a700 msgr2=0x7efd7412ab80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6808/1506640341 conn(0x7efd7412a700 0x7efd7412ab80 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd7411bb50 osd=6 1 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 10 client.37895.objecter close_session for osd.7 2024-02-13T00:50:33.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6812/1964192772 conn(0x7efd74129250 msgr2=0x7efd741296b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.181:6812/1964192772 conn(0x7efd74129250 0x7efd741296b0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 20 client.37895.objecter put_session s=0x7efd74128ba0 osd=7 1 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 20 client.37895.objecter shutdown clearing up homeless session... 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 10 client.37895.objecter successfully canceled tick 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd78a6a700 1 -- 172.21.15.162:0/2917148586 reap_dead start 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6800/2866534053 conn(0x7efd5c062c90 msgr2=0x7efd5c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6800/2866534053 conn(0x7efd5c062c90 0x7efd5c065150 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 rx=0x7efd740fa460 tx=0x7efd68009cb0).stop 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd740f73a0 msgr2=0x7efd740f8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd740f73a0 0x7efd740f8cc0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 rx=0x7efd6400e970 tx=0x7efd6400b040).stop 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 shutdown_connections 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd740f73a0 0x7efd740f8cc0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 --2- 172.21.15.162:0/2917148586 >> v2:172.21.15.162:6800/2866534053 conn(0x7efd5c062c90 0x7efd5c065150 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 >> 172.21.15.162:0/2917148586 conn(0x7efd7406eb90 msgr2=0x7efd740fad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 shutdown_connections 2024-02-13T00:50:33.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 -- 172.21.15.162:0/2917148586 wait complete. 2024-02-13T00:50:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:33.821+0000 7efd7a4cd700 1 librados: shutdown 2024-02-13T00:50:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:50:33.830 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:50:34.176 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:34.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:34.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:34.313 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:34.333 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:34.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:34.437 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:34.466 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:34.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:33.533753+0000 mon.a (mon.0) 3141 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-106"}]': finished 2024-02-13T00:50:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:33.533828+0000 mon.a (mon.0) 3142 : cluster [DBG] osdmap e619: 8 total, 8 up, 4 in 2024-02-13T00:50:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:33.535920+0000 mon.a (mon.0) 3143 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-106", "mode": "writeback"}]: dispatch 2024-02-13T00:50:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:33.677861+0000 mon.c (mon.2) 475 : audit [DBG] from='client.? 172.21.15.162:0/2917148586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:34.132976+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:50:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:34.133211+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:50:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:34.340680+0000 mon.a (mon.0) 3144 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:34.531290+0000 mon.a (mon.0) 3145 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:50:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:34.537926+0000 mon.a (mon.0) 3146 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-106", "mode": "writeback"}]': finished 2024-02-13T00:50:34.716 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:34.537991+0000 mon.a (mon.0) 3147 : cluster [DBG] osdmap e620: 8 total, 8 up, 4 in 2024-02-13T00:50:34.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:33.533753+0000 mon.a (mon.0) 3141 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-106"}]': finished 2024-02-13T00:50:34.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:33.533828+0000 mon.a (mon.0) 3142 : cluster [DBG] osdmap e619: 8 total, 8 up, 4 in 2024-02-13T00:50:34.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:33.535920+0000 mon.a (mon.0) 3143 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-106", "mode": "writeback"}]: dispatch 2024-02-13T00:50:34.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:33.677861+0000 mon.c (mon.2) 475 : audit [DBG] from='client.? 172.21.15.162:0/2917148586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:34.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:34.132976+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:50:34.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:34.133211+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:50:34.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:34.340680+0000 mon.a (mon.0) 3144 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:34.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:34.531290+0000 mon.a (mon.0) 3145 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:50:34.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:34.537926+0000 mon.a (mon.0) 3146 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-106", "mode": "writeback"}]': finished 2024-02-13T00:50:34.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:34.537991+0000 mon.a (mon.0) 3147 : cluster [DBG] osdmap e620: 8 total, 8 up, 4 in 2024-02-13T00:50:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:34.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:33.533753+0000 mon.a (mon.0) 3141 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-106"}]': finished 2024-02-13T00:50:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:33.533828+0000 mon.a (mon.0) 3142 : cluster [DBG] osdmap e619: 8 total, 8 up, 4 in 2024-02-13T00:50:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:33.535920+0000 mon.a (mon.0) 3143 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-106", "mode": "writeback"}]: dispatch 2024-02-13T00:50:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:33.677861+0000 mon.c (mon.2) 475 : audit [DBG] from='client.? 172.21.15.162:0/2917148586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:34.132976+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:50:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:34.133211+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:50:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:34.340680+0000 mon.a (mon.0) 3144 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:34.531290+0000 mon.a (mon.0) 3145 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:50:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:34.537926+0000 mon.a (mon.0) 3146 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-106", "mode": "writeback"}]': finished 2024-02-13T00:50:34.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:34.537991+0000 mon.a (mon.0) 3147 : cluster [DBG] osdmap e620: 8 total, 8 up, 4 in 2024-02-13T00:50:34.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:34.851 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:34.872 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:34.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:34.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:35.006 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:35.026 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:35.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:35.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:35.160 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:35.180 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:35.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:35.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:35.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:35.313 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:35.333 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:35.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:35.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:35.437 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:35.465 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:35.714 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:35.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:35.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:34.493316+0000 mgr.y (mgr.14544) 525 : cluster [DBG] pgmap v929: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:35.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:34.550922+0000 mon.a (mon.0) 3148 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "237.4"}]: dispatch 2024-02-13T00:50:35.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:34.551074+0000 mgr.y (mgr.14544) 526 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "237.4"}]: dispatch 2024-02-13T00:50:35.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:35.510315+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e621: 8 total, 8 up, 4 in 2024-02-13T00:50:35.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:34.493316+0000 mgr.y (mgr.14544) 525 : cluster [DBG] pgmap v929: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:34.550922+0000 mon.a (mon.0) 3148 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "237.4"}]: dispatch 2024-02-13T00:50:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:34.551074+0000 mgr.y (mgr.14544) 526 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "237.4"}]: dispatch 2024-02-13T00:50:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:35.510315+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e621: 8 total, 8 up, 4 in 2024-02-13T00:50:35.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:34.493316+0000 mgr.y (mgr.14544) 525 : cluster [DBG] pgmap v929: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:35.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:34.550922+0000 mon.a (mon.0) 3148 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "237.4"}]: dispatch 2024-02-13T00:50:35.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:34.551074+0000 mgr.y (mgr.14544) 526 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "237.4"}]: dispatch 2024-02-13T00:50:35.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:35.510315+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e621: 8 total, 8 up, 4 in 2024-02-13T00:50:35.845 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:35.969 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:35.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:36.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:36.103 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:50:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:50:36.105 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:36.126 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:36.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:36.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:36.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:36.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 -- 172.21.15.162:0/1721312412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b400fe650 msgr2=0x7f9b400f6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:36.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 --2- 172.21.15.162:0/1721312412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b400fe650 0x7f9b400f6270 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 rx=0x7f9b340047b0 tx=0x7f9b34009cb0).stop 2024-02-13T00:50:36.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 -- 172.21.15.162:0/1721312412 shutdown_connections 2024-02-13T00:50:36.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 --2- 172.21.15.162:0/1721312412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b400f6950 0x7f9b40104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:36.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 --2- 172.21.15.162:0/1721312412 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9b400fdcf0 0x7f9b400fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:36.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 --2- 172.21.15.162:0/1721312412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b400fe650 0x7f9b400f6270 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:36.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 -- 172.21.15.162:0/1721312412 >> 172.21.15.162:0/1721312412 conn(0x7f9b40070100 msgr2=0x7f9b400f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:36.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 -- 172.21.15.162:0/1721312412 shutdown_connections 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 -- 172.21.15.162:0/1721312412 wait complete. 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 librados: starting msgr at 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 librados: starting objecter 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 Processor -- start 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 -- start start 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 librados: setting wanted keys 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.228+0000 7f9b46453700 1 librados: calling monclient init 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b46453700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b400f6950 0x7f9b400f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b46453700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b400fdcf0 0x7f9b400f7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b46453700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9b400fe650 0x7f9b400f7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b46453700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40116750 con 0x7f9b400fdcf0 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b46453700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b401168c0 con 0x7f9b400f6950 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b46453700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40116a80 con 0x7f9b400fe650 2024-02-13T00:50:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b3f7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b400fdcf0 0x7f9b400f7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b3f7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b400fdcf0 0x7f9b400f7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:56252/0 (socket says 172.21.15.162:56252) 2024-02-13T00:50:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b3f7fe700 1 -- 172.21.15.162:0/3726199774 learned_addr learned my addr 172.21.15.162:0/3726199774 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b449f0700 1 --2- 172.21.15.162:0/3726199774 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9b400fe650 0x7f9b400f7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b3ffff700 1 --2- 172.21.15.162:0/3726199774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b400f6950 0x7f9b400f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b3f7fe700 1 -- 172.21.15.162:0/3726199774 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9b400fe650 msgr2=0x7f9b400f7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b3f7fe700 1 --2- 172.21.15.162:0/3726199774 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9b400fe650 0x7f9b400f7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b3f7fe700 1 -- 172.21.15.162:0/3726199774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b400f6950 msgr2=0x7f9b400f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b3f7fe700 1 --2- 172.21.15.162:0/3726199774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b400f6950 0x7f9b400f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:36.238 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b3f7fe700 1 -- 172.21.15.162:0/3726199774 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b30005040 con 0x7f9b400fdcf0 2024-02-13T00:50:36.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b3f7fe700 1 --2- 172.21.15.162:0/3726199774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b400fdcf0 0x7f9b400f7a30 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7f9b340099f0 tx=0x7f9b34009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:36.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b3d7fa700 10 client.?.objecter ms_handle_connect 0x7f9b400fdcf0 2024-02-13T00:50:36.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.229+0000 7f9b3d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:36.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.230+0000 7f9b3d7fa700 1 -- 172.21.15.162:0/3726199774 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34018070 con 0x7f9b400fdcf0 2024-02-13T00:50:36.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.230+0000 7f9b3d7fa700 1 -- 172.21.15.162:0/3726199774 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b34013da0 con 0x7f9b400fdcf0 2024-02-13T00:50:36.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.230+0000 7f9b46453700 1 -- 172.21.15.162:0/3726199774 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b34005040 con 0x7f9b400fdcf0 2024-02-13T00:50:36.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.230+0000 7f9b3d7fa700 1 -- 172.21.15.162:0/3726199774 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3401dcb0 con 0x7f9b400fdcf0 2024-02-13T00:50:36.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.230+0000 7f9b46453700 10 client.27801.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:36.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.230+0000 7f9b46453700 1 -- 172.21.15.162:0/3726199774 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b40116f20 con 0x7f9b400fdcf0 2024-02-13T00:50:36.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.230+0000 7f9b46453700 1 librados: init done 2024-02-13T00:50:36.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.231+0000 7f9b46453700 1 -- 172.21.15.162:0/3726199774 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b4005f980 con 0x7f9b400fdcf0 2024-02-13T00:50:36.239 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.231+0000 7f9b3d7fa700 1 -- 172.21.15.162:0/3726199774 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f9b3401d420 con 0x7f9b400fdcf0 2024-02-13T00:50:36.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.231+0000 7f9b3d7fa700 1 --2- 172.21.15.162:0/3726199774 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9b28062c90 0x7f9b28065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:36.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.232+0000 7f9b3d7fa700 1 -- 172.21.15.162:0/3726199774 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(621..621 src has 449..621) v4 ==== 8470+0+0 (secure 0 0 0) 0x7f9b3407e8d0 con 0x7f9b400fdcf0 2024-02-13T00:50:36.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.232+0000 7f9b3d7fa700 10 client.27801.objecter ms_dispatch 0x7f9b40001550 osd_map(621..621 src has 449..621) v4 2024-02-13T00:50:36.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.232+0000 7f9b3d7fa700 3 client.27801.objecter handle_osd_map got epochs [621,621] > 0 2024-02-13T00:50:36.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.232+0000 7f9b3d7fa700 3 client.27801.objecter handle_osd_map decoding full epoch 621 2024-02-13T00:50:36.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.232+0000 7f9b3ffff700 1 --2- 172.21.15.162:0/3726199774 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9b28062c90 0x7f9b28065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:36.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.232+0000 7f9b3d7fa700 20 client.27801.objecter dump_active .. 0 homeless 2024-02-13T00:50:36.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.232+0000 7f9b3d7fa700 10 client.27801.objecter ms_handle_connect 0x7f9b28062c90 2024-02-13T00:50:36.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.232+0000 7f9b3ffff700 1 --2- 172.21.15.162:0/3726199774 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9b28062c90 0x7f9b28065150 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 rx=0x7f9b40060a70 tx=0x7f9b3000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:36.240 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.237+0000 7f9b3d7fa700 1 -- 172.21.15.162:0/3726199774 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9b34051c00 con 0x7f9b400fdcf0 2024-02-13T00:50:36.264 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:36.265 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:36.266 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:36.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:36.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:36.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:36.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:36.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.419+0000 7f9b46453700 1 -- 172.21.15.162:0/3726199774 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f9b400f8a20 con 0x7f9b400fdcf0 2024-02-13T00:50:36.423 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:36.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.420+0000 7f9b3d7fa700 1 -- 172.21.15.162:0/3726199774 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v621) v1 ==== 81+0+48599 (secure 0 0 0) 0x7f9b34019270 con 0x7f9b400fdcf0 2024-02-13T00:50:36.425 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:50:36.425 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:50:36.425 INFO:teuthology.orchestra.run.smithi162.stdout: "epoch": 621, 2024-02-13T00:50:36.425 INFO:teuthology.orchestra.run.smithi162.stdout: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:50:36.425 INFO:teuthology.orchestra.run.smithi162.stdout: "created": "2024-02-13T00:31:51.797017+0000", 2024-02-13T00:50:36.425 INFO:teuthology.orchestra.run.smithi162.stdout: "modified": "2024-02-13T00:50:35.501912+0000", 2024-02-13T00:50:36.425 INFO:teuthology.orchestra.run.smithi162.stdout: "last_up_change": "2024-02-13T00:48:03.474219+0000", 2024-02-13T00:50:36.425 INFO:teuthology.orchestra.run.smithi162.stdout: "last_in_change": "2024-02-13T00:46:00.399226+0000", 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_num": 5799936, 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_set": [ 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: "pglog_hardlimit", 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: "purged_snapdirs", 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_deletes", 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: "sortbitwise" 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_version": 109, 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: "full_ratio": 0.94999998807907104, 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-13T00:50:36.426 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_snapshot": "", 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_max": 237, 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "max_osd": 8, 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "require_min_compat_client": "luminous", 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "min_compat_client": "luminous", 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "require_osd_release": "pacific", 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "pools": [ 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 1, 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "device_health_metrics", 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:35:34.545390+0000", 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:50:36.427 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 1, 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 1, 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 1, 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 1, 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 1, 2024-02-13T00:50:36.428 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "71", 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:50:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:50:36.430 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:50:36.431 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_max": 32, 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_min": 1 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_devicehealth": {} 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 2, 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": ".rgw.root", 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.459980+0000", 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:50:36.433 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:50:36.434 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "58", 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:50:36.435 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:50:36.436 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:50:36.436 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:50:36.436 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:50:36.436 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:50:36.436 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:50:36.436 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:50:36.436 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:50:36.436 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:50:36.436 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:50:36.437 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 3, 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "iscsi", 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.740219+0000", 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:50:36.438 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 3, 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 3, 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 3, 2024-02-13T00:50:36.439 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 3, 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 3, 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "102", 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:50:36.440 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:50:36.441 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:50:36.442 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd": {} 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.443 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 4, 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.log", 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:41.489292+0000", 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:50:36.444 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:50:36.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "60", 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:50:36.446 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:50:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:50:36.448 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 5, 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.control", 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:43.466304+0000", 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:50:36.449 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 22, 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 22, 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 22, 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 22, 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 22, 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "5.16", 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 544, 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 543, 2024-02-13T00:50:36.450 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 543, 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "544", 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "544", 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "542", 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:50:36.451 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:50:36.452 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:50:36.453 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 6, 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.meta", 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:45.542442+0000", 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:50:36.454 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 42, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 42, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 42, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 42, 2024-02-13T00:50:36.455 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 42, 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "248", 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "246", 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:50:36.456 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:50:36.457 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:50:36.458 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_bias": 4 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 28, 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:00.190777+0000", 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:50:36.460 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.461 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "108", 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:50:36.462 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.463 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 234, 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-101", 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:50:12.777962+0000", 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8193, 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 3, 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 1, 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:50:36.465 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 8, 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 8, 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 8, 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 8, 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 8, 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:50:36.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "619", 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "619", 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "619", 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "619", 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 2, 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 618, 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [ 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: 237 2024-02-13T00:50:36.467 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": 237, 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": 237, 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi162-84057-101", 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:50:36.468 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 8192, 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:50:36.469 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 237, 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-106", 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:50:28.765176+0000", 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8201, 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:50:36.470 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:50:36.471 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "620", 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "619", 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "619", 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "619", 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 2, 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 618, 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:50:36.472 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": 234, 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "writeback", 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:50:36.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:50:36.474 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99", 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 8, 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 615, 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:50:36.475 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6801", 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.476 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6802", 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6804", 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6803", 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6801/2103743338", 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6802/2103743338", 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6804/2103743338", 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6803/2103743338", 2024-02-13T00:50:36.478 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886", 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 13, 2024-02-13T00:50:36.479 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 615, 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6805", 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:50:36.480 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6806", 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6808", 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.481 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6807", 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6805/150599780", 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6806/150599780", 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6808/150599780", 2024-02-13T00:50:36.482 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6807/150599780", 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e", 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:50:36.483 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 18, 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 333, 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6809", 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.484 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6810", 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.485 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6812", 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6811", 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6809/3689719376", 2024-02-13T00:50:36.486 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6810/3689719376", 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6812/3689719376", 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6811/3689719376", 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001", 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:50:36.487 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 25, 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 255, 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6813", 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.488 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6814", 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.489 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6816", 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6815", 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.490 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6813/604971011", 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6814/604971011", 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6816/604971011", 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6815/604971011", 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3", 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:50:36.491 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 31, 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 81, 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6800", 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:50:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6801", 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6803", 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6802", 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:50:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6800/3344263388", 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6801/3344263388", 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6803/3344263388", 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6802/3344263388", 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:50:36.495 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723", 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 39, 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 436, 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 479, 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 399, 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 437, 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6804", 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6805", 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.497 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6807", 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.498 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6806", 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6804/2277917372", 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6805/2277917372", 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2277917372", 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2277917372", 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474", 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 45, 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 615, 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:50:36.500 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6808", 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6809", 2024-02-13T00:50:36.501 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6811", 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:50:36.502 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6810", 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6808/1506640341", 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6809/1506640341", 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1506640341", 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1506640341", 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:50:36.503 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b", 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 53, 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 215, 2024-02-13T00:50:36.504 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6812", 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.505 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6813", 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6815", 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6814", 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6812/1964192772", 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6813/1964192772", 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6815/1964192772", 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6814/1964192772", 2024-02-13T00:50:36.507 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_xinfo": [ 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:50:36.508 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:49:41.394915+0000", 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:49:51.947964+0000", 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:49:26.507182+0000", 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:50:36.510 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:49:54.589902+0000", 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:49:54.837171+0000", 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:50:36.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:50:34.133215+0000", 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 436 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:50:36.512 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:49:36.286492+0000", 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:50:36.513 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:50:13.831366+0000", 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap": [], 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap_items": [ 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.4", 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 0 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "5.5", 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 6 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "6.6", 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.515 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 0 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "6.22", 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 4 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "28.12", 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 4 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "28.14", 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:50:36.517 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 1 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_temp": [], 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_temp": [], 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: "blocklist": { 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/1909610488": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2370739667": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/1502590187": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/4121673848": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2412382714": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3041562701": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:50:36.518 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/2775563314": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3135632190": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/197307636": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2653247729": "2024-02-14T00:32:24.380625+0000" 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: "range_blocklist": {}, 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profiles": { 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: "default": { 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van" 2024-02-13T00:50:36.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: "testprofile-test-rados-api-smithi162-84057-101": { 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-device-class": "", 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-root": "default", 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van", 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: "w": "8" 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.520 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps_queue": [ 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 24, 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 9, 2024-02-13T00:50:36.521 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 25, 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 60, 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.523 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 78, 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:50:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 85, 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 94, 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:50:36.526 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 95, 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 4, 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 4 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 107, 2024-02-13T00:50:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 113, 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 125, 2024-02-13T00:50:36.529 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 10, 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 133, 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 12, 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 154, 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 14, 2024-02-13T00:50:36.531 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 166, 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 17, 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.532 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 176, 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 20, 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: "new_removed_snaps": [], 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: "new_purged_snaps": [], 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_node_flags": {}, 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: "device_class_flags": {}, 2024-02-13T00:50:36.533 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode": { 2024-02-13T00:50:36.534 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_enabled": false, 2024-02-13T00:50:36.534 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_bucket_count": 0, 2024-02-13T00:50:36.534 INFO:teuthology.orchestra.run.smithi162.stdout: "degraded_stretch_mode": 0, 2024-02-13T00:50:36.534 INFO:teuthology.orchestra.run.smithi162.stdout: "recovering_stretch_mode": 0, 2024-02-13T00:50:36.534 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_bucket": 0 2024-02-13T00:50:36.534 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:50:36.534 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:50:36.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:36.536 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:36.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.422+0000 7f9b46453700 10 librados: watch_flush enter 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.422+0000 7f9b46453700 10 librados: watch_flush exit 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.422+0000 7f9b46453700 20 client.27801.objecter shutdown clearing up homeless session... 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.422+0000 7f9b46453700 10 client.27801.objecter successfully canceled tick 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.422+0000 7f9b46453700 1 -- 172.21.15.162:0/3726199774 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9b28062c90 msgr2=0x7f9b28065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.422+0000 7f9b46453700 1 --2- 172.21.15.162:0/3726199774 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9b28062c90 0x7f9b28065150 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 rx=0x7f9b40060a70 tx=0x7f9b3000b040).stop 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.422+0000 7f9b46453700 1 -- 172.21.15.162:0/3726199774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b400fdcf0 msgr2=0x7f9b400f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.422+0000 7f9b46453700 1 --2- 172.21.15.162:0/3726199774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b400fdcf0 0x7f9b400f7a30 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7f9b340099f0 tx=0x7f9b34009cb0).stop 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.423+0000 7f9b46453700 1 -- 172.21.15.162:0/3726199774 shutdown_connections 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.423+0000 7f9b46453700 1 --2- 172.21.15.162:0/3726199774 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9b28062c90 0x7f9b28065150 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.423+0000 7f9b46453700 1 --2- 172.21.15.162:0/3726199774 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9b400fe650 0x7f9b400f7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.423+0000 7f9b46453700 1 --2- 172.21.15.162:0/3726199774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b400f6950 0x7f9b400f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.423+0000 7f9b46453700 1 --2- 172.21.15.162:0/3726199774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b400fdcf0 0x7f9b400f7a30 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:36.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.423+0000 7f9b46453700 1 -- 172.21.15.162:0/3726199774 >> 172.21.15.162:0/3726199774 conn(0x7f9b40070100 msgr2=0x7f9b400fba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:36.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.423+0000 7f9b46453700 1 -- 172.21.15.162:0/3726199774 shutdown_connections 2024-02-13T00:50:36.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.423+0000 7f9b46453700 1 -- 172.21.15.162:0/3726199774 wait complete. 2024-02-13T00:50:36.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:36.423+0000 7f9b46453700 1 librados: shutdown 2024-02-13T00:50:36.538 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:50:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:50:36] "GET /metrics HTTP/1.1" 200 189081 "" "Prometheus/2.33.4" 2024-02-13T00:50:36.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:36.665 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:36.788 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:36.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:35.800720+0000 mgr.y (mgr.14544) 527 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:36.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:36.420497+0000 mon.a (mon.0) 3150 : audit [DBG] from='client.? 172.21.15.162:0/3726199774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:50:36.791 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:35.800720+0000 mgr.y (mgr.14544) 527 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:36.791 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:36.420497+0000 mon.a (mon.0) 3150 : audit [DBG] from='client.? 172.21.15.162:0/3726199774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:50:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:36.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:36.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:35.800720+0000 mgr.y (mgr.14544) 527 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:36.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:36.420497+0000 mon.a (mon.0) 3150 : audit [DBG] from='client.? 172.21.15.162:0/3726199774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:50:36.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:36.920 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:37.469 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:37.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:37.573 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:37.606 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:37.627 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:37.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:37.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:37.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:50:37.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:36.086302+0000 osd.0 (osd.0) 237 : cluster [DBG] 237.4 scrub starts 2024-02-13T00:50:37.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:36.089192+0000 osd.0 (osd.0) 238 : cluster [DBG] 237.4 scrub ok 2024-02-13T00:50:37.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:36.493731+0000 mgr.y (mgr.14544) 528 : cluster [DBG] pgmap v932: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-13T00:50:37.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:36.784381+0000 osd.1 (osd.1) 261 : cluster [DBG] 5.12 scrub starts 2024-02-13T00:50:37.722 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:36.787600+0000 osd.1 (osd.1) 262 : cluster [DBG] 5.12 scrub ok 2024-02-13T00:50:37.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:36.086302+0000 osd.0 (osd.0) 237 : cluster [DBG] 237.4 scrub starts 2024-02-13T00:50:37.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:36.089192+0000 osd.0 (osd.0) 238 : cluster [DBG] 237.4 scrub ok 2024-02-13T00:50:37.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:36.493731+0000 mgr.y (mgr.14544) 528 : cluster [DBG] pgmap v932: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-13T00:50:37.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:36.784381+0000 osd.1 (osd.1) 261 : cluster [DBG] 5.12 scrub starts 2024-02-13T00:50:37.723 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:36.787600+0000 osd.1 (osd.1) 262 : cluster [DBG] 5.12 scrub ok 2024-02-13T00:50:37.731 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:37.757 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:38.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:36.086302+0000 osd.0 (osd.0) 237 : cluster [DBG] 237.4 scrub starts 2024-02-13T00:50:38.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:36.089192+0000 osd.0 (osd.0) 238 : cluster [DBG] 237.4 scrub ok 2024-02-13T00:50:38.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:36.493731+0000 mgr.y (mgr.14544) 528 : cluster [DBG] pgmap v932: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-13T00:50:38.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:36.784381+0000 osd.1 (osd.1) 261 : cluster [DBG] 5.12 scrub starts 2024-02-13T00:50:38.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:36.787600+0000 osd.1 (osd.1) 262 : cluster [DBG] 5.12 scrub ok 2024-02-13T00:50:38.206 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:38.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:38.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:38.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:38.342 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:38.363 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:38.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:38.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:38.496 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:38.516 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:38.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:38.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:38.647 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:50:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:50:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:50:38.648 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:37.774908+0000 osd.1 (osd.1) 263 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:50:38.648 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:37.777887+0000 osd.1 (osd.1) 264 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:50:38.648 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:38.050605+0000 osd.0 (osd.0) 239 : cluster [DBG] 6.0 deep-scrub starts 2024-02-13T00:50:38.648 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:38.051464+0000 osd.0 (osd.0) 240 : cluster [DBG] 6.0 deep-scrub ok 2024-02-13T00:50:38.649 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:38.671 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:38.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:38.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:38.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:38.802 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:38.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:37.774908+0000 osd.1 (osd.1) 263 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:50:38.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:37.777887+0000 osd.1 (osd.1) 264 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:50:38.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:38.050605+0000 osd.0 (osd.0) 239 : cluster [DBG] 6.0 deep-scrub starts 2024-02-13T00:50:38.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:38.051464+0000 osd.0 (osd.0) 240 : cluster [DBG] 6.0 deep-scrub ok 2024-02-13T00:50:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:37.774908+0000 osd.1 (osd.1) 263 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:50:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:37.777887+0000 osd.1 (osd.1) 264 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:50:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:38.050605+0000 osd.0 (osd.0) 239 : cluster [DBG] 6.0 deep-scrub starts 2024-02-13T00:50:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:38.051464+0000 osd.0 (osd.0) 240 : cluster [DBG] 6.0 deep-scrub ok 2024-02-13T00:50:38.899 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:38.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:38.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:39.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:39.033 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:39.052 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:39.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:39.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:39.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:39.185 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:39.439 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:39.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:39.449 INFO:tasks.thrashosds.thrasher:j is {'epoch': 621, 'fsid': 'd4ee01fc-ca06-11ee-95b9-87774f69a715', 'created': '2024-02-13T00:31:51.797017+0000', 'modified': '2024-02-13T00:50:35.501912+0000', 'last_up_change': '2024-02-13T00:48:03.474219+0000', 'last_in_change': '2024-02-13T00:46:00.399226+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 109, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 237, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-13T00:35:34.545390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-13T00:37:39.459980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-13T00:37:39.740219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-13T00:37:41.489292+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-13T00:37:43.466304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 544, 'last_epoch_started': 543, 'last_epoch_clean': 543, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '544', 'last_force_op_resend_preluminous': '542', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-13T00:37:45.542442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '246', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 28, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-13T00:41:00.190777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 234, 'pool_name': 'test-rados-api-smithi162-84057-101', 'create_time': '2024-02-13T00:50:12.777962+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '619', 'last_force_op_resend': '619', 'last_force_op_resend_prenautilus': '619', 'last_force_op_resend_preluminous': '619', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 618, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [237], 'tier_of': -1, 'read_tier': 237, 'write_tier': 237, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi162-84057-101', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 237, 'pool_name': 'test-rados-api-smithi162-84057-106', 'create_time': '2024-02-13T00:50:28.765176+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '620', 'last_force_op_resend': '619', 'last_force_op_resend_prenautilus': '619', 'last_force_op_resend_preluminous': '619', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 618, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 234, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 615, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6801', 'nonce': 2103743338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 2103743338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 2103743338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6803', 'nonce': 2103743338}]}, 'public_addr': '172.21.15.162:6801/2103743338', 'cluster_addr': '172.21.15.162:6802/2103743338', 'heartbeat_back_addr': '172.21.15.162:6804/2103743338', 'heartbeat_front_addr': '172.21.15.162:6803/2103743338', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '39e9d2ef-9aa4-45bb-8a47-b0e0ec222886', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 615, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6805', 'nonce': 150599780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 150599780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 150599780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6807', 'nonce': 150599780}]}, 'public_addr': '172.21.15.162:6805/150599780', 'cluster_addr': '172.21.15.162:6806/150599780', 'heartbeat_back_addr': '172.21.15.162:6808/150599780', 'heartbeat_front_addr': '172.21.15.162:6807/150599780', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 333, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6809', 'nonce': 3689719376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3689719376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3689719376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6811', 'nonce': 3689719376}]}, 'public_addr': '172.21.15.162:6809/3689719376', 'cluster_addr': '172.21.15.162:6810/3689719376', 'heartbeat_back_addr': '172.21.15.162:6812/3689719376', 'heartbeat_front_addr': '172.21.15.162:6811/3689719376', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e6b9f0e-7a18-4cf8-9331-1a973fadc001', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 255, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6813', 'nonce': 604971011}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 604971011}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 604971011}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6815', 'nonce': 604971011}]}, 'public_addr': '172.21.15.162:6813/604971011', 'cluster_addr': '172.21.15.162:6814/604971011', 'heartbeat_back_addr': '172.21.15.162:6816/604971011', 'heartbeat_front_addr': '172.21.15.162:6815/604971011', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 3344263388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 3344263388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 3344263388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 3344263388}]}, 'public_addr': '172.21.15.181:6800/3344263388', 'cluster_addr': '172.21.15.181:6801/3344263388', 'heartbeat_back_addr': '172.21.15.181:6803/3344263388', 'heartbeat_front_addr': '172.21.15.181:6802/3344263388', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6aa2bd68-7f4b-4973-ac49-7fbc57c1a723', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 436, 'up_from': 479, 'up_thru': 399, 'down_at': 437, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2277917372}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2277917372}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2277917372}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2277917372}]}, 'public_addr': '172.21.15.181:6804/2277917372', 'cluster_addr': '172.21.15.181:6805/2277917372', 'heartbeat_back_addr': '172.21.15.181:6807/2277917372', 'heartbeat_front_addr': '172.21.15.181:6806/2277917372', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '791ba814-5284-432e-addc-d6b7f5f3d474', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 615, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1506640341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1506640341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1506640341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1506640341}]}, 'public_addr': '172.21.15.181:6808/1506640341', 'cluster_addr': '172.21.15.181:6809/1506640341', 'heartbeat_back_addr': '172.21.15.181:6811/1506640341', 'heartbeat_front_addr': '172.21.15.181:6810/1506640341', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f369c65-e663-4eb2-87a5-f2900f15b35b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 215, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 1964192772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 1964192772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 1964192772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 1964192772}]}, 'public_addr': '172.21.15.181:6812/1964192772', 'cluster_addr': '172.21.15.181:6813/1964192772', 'heartbeat_back_addr': '172.21.15.181:6815/1964192772', 'heartbeat_front_addr': '172.21.15.181:6814/1964192772', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:49:41.394915+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:49:51.947964+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:49:26.507182+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:49:54.589902+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:49:54.837171+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:50:34.133215+0000', 'dead_epoch': 436}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:49:36.286492+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:50:13.831366+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '5.5', 'mappings': [{'from': 0, 'to': 6}]}, {'pgid': '6.6', 'mappings': [{'from': 6, 'to': 0}]}, {'pgid': '6.22', 'mappings': [{'from': 6, 'to': 4}]}, {'pgid': '28.12', 'mappings': [{'from': 0, 'to': 4}]}, {'pgid': '28.14', 'mappings': [{'from': 6, 'to': 1}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.162:6800/1909610488': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2370739667': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/1502590187': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/4121673848': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:0/2412382714': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/3041562701': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/2775563314': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/3135632190': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/197307636': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2653247729': '2024-02-14T00:32:24.380625+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi162-84057-101': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 24, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 60, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 78, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 94, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 113, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 166, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-13T00:50:39.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:50:39.543 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:39.575 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:39.596 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:39.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:39.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.655+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1858307546 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf180f8d10 msgr2=0x7fbf180f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:39.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.655+0000 7fbf1cd3a700 1 --2- 172.21.15.162:0/1858307546 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf180f8d10 0x7fbf180f90f0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 rx=0x7fbf08004870 tx=0x7fbf08009cb0).stop 2024-02-13T00:50:39.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.655+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1858307546 shutdown_connections 2024-02-13T00:50:39.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.655+0000 7fbf1cd3a700 1 --2- 172.21.15.162:0/1858307546 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf18102270 0x7fbf18104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:39.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.655+0000 7fbf1cd3a700 1 --2- 172.21.15.162:0/1858307546 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf180f8d10 0x7fbf180f90f0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:39.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.655+0000 7fbf1cd3a700 1 --2- 172.21.15.162:0/1858307546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf180f95e0 0x7fbf18101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:39.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.655+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1858307546 >> 172.21.15.162:0/1858307546 conn(0x7fbf1806eb90 msgr2=0x7fbf180f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:39.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.655+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1858307546 shutdown_connections 2024-02-13T00:50:39.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.655+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1858307546 wait complete. 2024-02-13T00:50:39.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf1cd3a700 1 librados: starting msgr at 2024-02-13T00:50:39.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf1cd3a700 1 librados: starting objecter 2024-02-13T00:50:39.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf1cd3a700 1 Processor -- start 2024-02-13T00:50:39.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf1cd3a700 1 -- start start 2024-02-13T00:50:39.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf1cd3a700 1 librados: setting wanted keys 2024-02-13T00:50:39.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf1cd3a700 1 librados: calling monclient init 2024-02-13T00:50:39.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf1cd3a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf180f8d10 0x7fbf1810e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:39.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf1cd3a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf180f95e0 0x7fbf1810e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:39.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf1cd3a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf18102270 0x7fbf18108400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:39.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf1cd3a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf181088f0 con 0x7fbf180f95e0 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf1cd3a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf18108a60 con 0x7fbf18102270 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf1cd3a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf18108bd0 con 0x7fbf180f8d10 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.656+0000 7fbf15d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf180f95e0 0x7fbf1810e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf15d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf180f95e0 0x7fbf1810e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:56274/0 (socket says 172.21.15.162:56274) 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf15d9b700 1 -- 172.21.15.162:0/1469239515 learned_addr learned my addr 172.21.15.162:0/1469239515 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf16d9d700 1 --2- 172.21.15.162:0/1469239515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf18102270 0x7fbf18108400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf1659c700 1 --2- 172.21.15.162:0/1469239515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf180f8d10 0x7fbf1810e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf15d9b700 1 -- 172.21.15.162:0/1469239515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf180f8d10 msgr2=0x7fbf1810e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf15d9b700 1 --2- 172.21.15.162:0/1469239515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf180f8d10 0x7fbf1810e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf15d9b700 1 -- 172.21.15.162:0/1469239515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf18102270 msgr2=0x7fbf18108400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf15d9b700 1 --2- 172.21.15.162:0/1469239515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf18102270 0x7fbf18108400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf15d9b700 1 -- 172.21.15.162:0/1469239515 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c005040 con 0x7fbf180f95e0 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf1659c700 1 --2- 172.21.15.162:0/1469239515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf180f8d10 0x7fbf1810e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:50:39.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf15d9b700 1 --2- 172.21.15.162:0/1469239515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf180f95e0 0x7fbf1810e710 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 rx=0x7fbf0c0048f0 tx=0x7fbf0c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:39.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf077fe700 10 client.?.objecter ms_handle_connect 0x7fbf180f95e0 2024-02-13T00:50:39.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf077fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:39.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf077fe700 1 -- 172.21.15.162:0/1469239515 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c007b00 con 0x7fbf180f95e0 2024-02-13T00:50:39.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1469239515 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf08005040 con 0x7fbf180f95e0 2024-02-13T00:50:39.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf077fe700 1 -- 172.21.15.162:0/1469239515 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf0c00f040 con 0x7fbf180f95e0 2024-02-13T00:50:39.664 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf077fe700 1 -- 172.21.15.162:0/1469239515 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c014a00 con 0x7fbf180f95e0 2024-02-13T00:50:39.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf1cd3a700 10 client.27807.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:39.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1469239515 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf18109260 con 0x7fbf180f95e0 2024-02-13T00:50:39.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.657+0000 7fbf1cd3a700 1 librados: init done 2024-02-13T00:50:39.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.658+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1469239515 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf181071b0 con 0x7fbf180f95e0 2024-02-13T00:50:39.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.659+0000 7fbf077fe700 1 -- 172.21.15.162:0/1469239515 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fbf0c00e580 con 0x7fbf180f95e0 2024-02-13T00:50:39.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.659+0000 7fbf077fe700 1 --2- 172.21.15.162:0/1469239515 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbf00062c90 0x7fbf00065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:39.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.659+0000 7fbf077fe700 1 -- 172.21.15.162:0/1469239515 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(621..621 src has 449..621) v4 ==== 8470+0+0 (secure 0 0 0) 0x7fbf0c07cf20 con 0x7fbf180f95e0 2024-02-13T00:50:39.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.659+0000 7fbf077fe700 10 client.27807.objecter ms_dispatch 0x7fbf18001550 osd_map(621..621 src has 449..621) v4 2024-02-13T00:50:39.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.659+0000 7fbf077fe700 3 client.27807.objecter handle_osd_map got epochs [621,621] > 0 2024-02-13T00:50:39.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.659+0000 7fbf077fe700 3 client.27807.objecter handle_osd_map decoding full epoch 621 2024-02-13T00:50:39.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.659+0000 7fbf1659c700 1 --2- 172.21.15.162:0/1469239515 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbf00062c90 0x7fbf00065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:39.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.659+0000 7fbf077fe700 20 client.27807.objecter dump_active .. 0 homeless 2024-02-13T00:50:39.666 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.660+0000 7fbf077fe700 10 client.27807.objecter ms_handle_connect 0x7fbf00062c90 2024-02-13T00:50:39.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.660+0000 7fbf1659c700 1 --2- 172.21.15.162:0/1469239515 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbf00062c90 0x7fbf00065150 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7fbf08009a00 tx=0x7fbf0800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:39.667 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.664+0000 7fbf077fe700 1 -- 172.21.15.162:0/1469239515 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fbf0c049f60 con 0x7fbf180f95e0 2024-02-13T00:50:39.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:39.729 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:39.749 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:39.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:38.494196+0000 mgr.y (mgr.14544) 529 : cluster [DBG] pgmap v933: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:50:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:38.494196+0000 mgr.y (mgr.14544) 529 : cluster [DBG] pgmap v933: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:50:39.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:39.845+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1469239515 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fbf1805fb80 con 0x7fbf180f95e0 2024-02-13T00:50:39.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:39.880 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:39.977 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:39.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:40.004 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:38.494196+0000 mgr.y (mgr.14544) 529 : cluster [DBG] pgmap v933: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:50:40.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:40.110 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:40.258 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:40.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:40.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:40.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:40.393 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:40.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.568+0000 7fbf077fe700 1 -- 172.21.15.162:0/1469239515 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v622) v1 ==== 87+0+0 (secure 0 0 0) 0x7fbf0c050250 con 0x7fbf180f95e0 2024-02-13T00:50:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.570+0000 7fbf1cd3a700 10 librados: watch_flush enter 2024-02-13T00:50:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 10 librados: watch_flush exit 2024-02-13T00:50:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 20 client.27807.objecter shutdown clearing up homeless session... 2024-02-13T00:50:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 10 client.27807.objecter successfully canceled tick 2024-02-13T00:50:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1469239515 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbf00062c90 msgr2=0x7fbf00065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 --2- 172.21.15.162:0/1469239515 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbf00062c90 0x7fbf00065150 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7fbf08009a00 tx=0x7fbf0800b040).stop 2024-02-13T00:50:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1469239515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf180f95e0 msgr2=0x7fbf1810e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 --2- 172.21.15.162:0/1469239515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf180f95e0 0x7fbf1810e710 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 rx=0x7fbf0c0048f0 tx=0x7fbf0c009cb0).stop 2024-02-13T00:50:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1469239515 shutdown_connections 2024-02-13T00:50:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 --2- 172.21.15.162:0/1469239515 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbf00062c90 0x7fbf00065150 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 --2- 172.21.15.162:0/1469239515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf18102270 0x7fbf18108400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:40.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 --2- 172.21.15.162:0/1469239515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf180f8d10 0x7fbf1810e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:40.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 --2- 172.21.15.162:0/1469239515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf180f95e0 0x7fbf1810e710 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:40.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1469239515 >> 172.21.15.162:0/1469239515 conn(0x7fbf1806eb90 msgr2=0x7fbf18102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:40.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1469239515 shutdown_connections 2024-02-13T00:50:40.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 -- 172.21.15.162:0/1469239515 wait complete. 2024-02-13T00:50:40.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:40.571+0000 7fbf1cd3a700 1 librados: shutdown 2024-02-13T00:50:40.609 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:50:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:39.597155+0000 osd.6 (osd.6) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:50:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:39.597251+0000 osd.6 (osd.6) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:50:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:39.846113+0000 mon.a (mon.0) 3151 : audit [INF] from='client.? 172.21.15.162:0/1469239515' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:50:40.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:39.597155+0000 osd.6 (osd.6) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:50:40.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:39.597251+0000 osd.6 (osd.6) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:50:40.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:39.846113+0000 mon.a (mon.0) 3151 : audit [INF] from='client.? 172.21.15.162:0/1469239515' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:50:41.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:39.597155+0000 osd.6 (osd.6) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:50:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:39.597251+0000 osd.6 (osd.6) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:50:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:39.846113+0000 mon.a (mon.0) 3151 : audit [INF] from='client.? 172.21.15.162:0/1469239515' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:50:41.144 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:41.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:41.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:41.281 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:41.302 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:41.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:41.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:41.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:41.434 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:41.456 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:41.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:41.560 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:41.591 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:41.790 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:41.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:41.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:40.381477+0000 osd.6 (osd.6) 185 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:50:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:40.384423+0000 osd.6 (osd.6) 186 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:50:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:40.495071+0000 mgr.y (mgr.14544) 530 : cluster [DBG] pgmap v934: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-02-13T00:50:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:40.561401+0000 mon.a (mon.0) 3152 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:50:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:40.569401+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? 172.21.15.162:0/1469239515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:50:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:40.569445+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e622: 8 total, 8 up, 4 in 2024-02-13T00:50:41.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:40.381477+0000 osd.6 (osd.6) 185 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:50:41.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:40.384423+0000 osd.6 (osd.6) 186 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:50:41.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:40.495071+0000 mgr.y (mgr.14544) 530 : cluster [DBG] pgmap v934: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-02-13T00:50:41.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:40.561401+0000 mon.a (mon.0) 3152 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:50:41.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:40.569401+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? 172.21.15.162:0/1469239515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:50:41.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:40.569445+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e622: 8 total, 8 up, 4 in 2024-02-13T00:50:41.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:41.922 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:40.381477+0000 osd.6 (osd.6) 185 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:50:41.922 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:40.384423+0000 osd.6 (osd.6) 186 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:50:41.922 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:40.495071+0000 mgr.y (mgr.14544) 530 : cluster [DBG] pgmap v934: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-02-13T00:50:41.922 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:40.561401+0000 mon.a (mon.0) 3152 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:50:41.922 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:40.569401+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? 172.21.15.162:0/1469239515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:50:41.923 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:40.569445+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e622: 8 total, 8 up, 4 in 2024-02-13T00:50:41.924 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:41.945 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:41.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:41.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:41.988 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:50:41.989 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:50:42.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:42.077 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:42.278 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:42.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:42.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:42.409 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:42.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-13T00:50:43.057 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:43.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:43.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:43.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:43.194 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:43.215 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:43.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:43.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:43.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:43.349 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:43.369 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:43.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:43.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:43.500 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:43.552 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:43.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:43.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:43.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:43.683 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:42.495571+0000 mgr.y (mgr.14544) 531 : cluster [DBG] pgmap v936: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 900 B/s wr, 2 op/s 2024-02-13T00:50:43.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:42.495571+0000 mgr.y (mgr.14544) 531 : cluster [DBG] pgmap v936: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 900 B/s wr, 2 op/s 2024-02-13T00:50:43.831 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:43.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:43.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:43.967 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:42.495571+0000 mgr.y (mgr.14544) 531 : cluster [DBG] pgmap v936: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 900 B/s wr, 2 op/s 2024-02-13T00:50:43.969 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:43.990 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:43.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:44.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:44.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.080+0000 7f35a1fba700 1 -- 172.21.15.162:0/3946575389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f359c0f95e0 msgr2=0x7f359c101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.080+0000 7f35a1fba700 1 --2- 172.21.15.162:0/3946575389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f359c0f95e0 0x7f359c101d80 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7f3590004870 tx=0x7f3590009cb0).stop 2024-02-13T00:50:44.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.080+0000 7f35a1fba700 1 -- 172.21.15.162:0/3946575389 shutdown_connections 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.080+0000 7f35a1fba700 1 --2- 172.21.15.162:0/3946575389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f359c102270 0x7f359c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.080+0000 7f35a1fba700 1 --2- 172.21.15.162:0/3946575389 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f359c0f8d10 0x7f359c0f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.080+0000 7f35a1fba700 1 --2- 172.21.15.162:0/3946575389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f359c0f95e0 0x7f359c101d80 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.080+0000 7f35a1fba700 1 -- 172.21.15.162:0/3946575389 >> 172.21.15.162:0/3946575389 conn(0x7f359c06eb90 msgr2=0x7f359c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.081+0000 7f35a1fba700 1 -- 172.21.15.162:0/3946575389 shutdown_connections 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.081+0000 7f35a1fba700 1 -- 172.21.15.162:0/3946575389 wait complete. 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.081+0000 7f35a1fba700 1 librados: starting msgr at 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.081+0000 7f35a1fba700 1 librados: starting objecter 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.081+0000 7f35a1fba700 1 Processor -- start 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.081+0000 7f35a1fba700 1 -- start start 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.081+0000 7f35a1fba700 1 librados: setting wanted keys 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.081+0000 7f35a1fba700 1 librados: calling monclient init 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.081+0000 7f35a1fba700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f359c0f8d10 0x7f359c10e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:44.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.081+0000 7f35a1fba700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f359c0f95e0 0x7f359c10e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f35a1fba700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f359c102270 0x7f359c108400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f35a1fba700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c1088f0 con 0x7f359c102270 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f35a1fba700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c108a60 con 0x7f359c0f95e0 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f35a1fba700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c108bd0 con 0x7f359c0f8d10 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359b7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f359c0f8d10 0x7f359c10e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359b7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f359c0f8d10 0x7f359c10e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:34962/0 (socket says 172.21.15.162:34962) 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359b7fe700 1 -- 172.21.15.162:0/82149337 learned_addr learned my addr 172.21.15.162:0/82149337 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359bfff700 1 --2- 172.21.15.162:0/82149337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f359c102270 0x7f359c108400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359affd700 1 --2- 172.21.15.162:0/82149337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f359c0f95e0 0x7f359c10e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359bfff700 1 -- 172.21.15.162:0/82149337 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f359c0f8d10 msgr2=0x7f359c10e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359bfff700 1 --2- 172.21.15.162:0/82149337 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f359c0f8d10 0x7f359c10e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359bfff700 1 -- 172.21.15.162:0/82149337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f359c0f95e0 msgr2=0x7f359c10e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359bfff700 1 --2- 172.21.15.162:0/82149337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f359c0f95e0 0x7f359c10e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359bfff700 1 -- 172.21.15.162:0/82149337 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c005040 con 0x7f359c102270 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359b7fe700 1 --2- 172.21.15.162:0/82149337 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f359c0f8d10 0x7f359c10e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359affd700 1 --2- 172.21.15.162:0/82149337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f359c0f95e0 0x7f359c10e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f359bfff700 1 --2- 172.21.15.162:0/82149337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f359c102270 0x7f359c108400 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 rx=0x7f359400d8a0 tx=0x7f3594004db0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f3598ff9700 10 client.?.objecter ms_handle_connect 0x7f359c102270 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f3598ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594004020 con 0x7f359c102270 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f359400f040 con 0x7f359c102270 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3590005040 con 0x7f359c102270 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35940036c0 con 0x7f359c102270 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f35a1fba700 10 client.27813.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c109210 con 0x7f359c102270 2024-02-13T00:50:44.113 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.082+0000 7f35a1fba700 1 librados: init done 2024-02-13T00:50:44.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.083+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f359c107170 con 0x7f359c102270 2024-02-13T00:50:44.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.084+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f3594004180 con 0x7f359c102270 2024-02-13T00:50:44.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.084+0000 7f3598ff9700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3584062c90 0x7f3584065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:44.114 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:50:44.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.084+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(622..622 src has 449..622) v4 ==== 8470+0+0 (secure 0 0 0) 0x7f359407d9c0 con 0x7f359c102270 2024-02-13T00:50:44.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.084+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 osd_map(622..622 src has 449..622) v4 2024-02-13T00:50:44.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.084+0000 7f3598ff9700 3 client.27813.objecter handle_osd_map got epochs [622,622] > 0 2024-02-13T00:50:44.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.084+0000 7f3598ff9700 3 client.27813.objecter handle_osd_map decoding full epoch 622 2024-02-13T00:50:44.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.084+0000 7f359b7fe700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3584062c90 0x7f3584065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:44.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.084+0000 7f3598ff9700 20 client.27813.objecter dump_active .. 0 homeless 2024-02-13T00:50:44.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.084+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v622) v1 ==== 54+0+16 (secure 0 0 0) 0x7f359407e0a0 con 0x7f359c102270 2024-02-13T00:50:44.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f359b7fe700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3584062c90 0x7f3584065150 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 rx=0x7f358c009810 tx=0x7f358c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:44.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f3598ff9700 10 client.27813.objecter ms_handle_connect 0x7f3584062c90 2024-02-13T00:50:44.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f35a1fba700 10 client.27813.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:44.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 1 2024-02-13T00:50:44.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6801/2103743338 conn(0x7f359c028590 0x7f359c0fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:44.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c1098e0 osd=0 3 2024-02-13T00:50:44.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c1098e0 osd=0 3 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c1098e0 osd=0 3 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 1 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c1098e0 osd=0 3 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 0 1 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c1098e0 osd=0 4 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f35a1fba700 10 client.27813.objecter _send_command 1 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f359c0fd370 con 0x7f359c028590 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f359affd700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6801/2103743338 conn(0x7f359c028590 0x7f359c0fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f3598ff9700 10 client.27813.objecter ms_handle_connect 0x7f359c028590 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.085+0000 7f359affd700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6801/2103743338 conn(0x7f359c028590 0x7f359c0fbd00 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.086+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f359c0fd370 con 0x7f359c028590 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.086+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.086+0000 7f3598ff9700 10 client.27813.objecter _finish_command 1 = system:0 2024-02-13T00:50:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.086+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c1098e0 osd=0 4 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.086+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 0 1 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.100+0000 7f35a1fba700 10 client.27813.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.100+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 2 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.100+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c1098e0 osd=0 3 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.100+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c1098e0 osd=0 3 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.100+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c1098e0 osd=0 3 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.100+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 2 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.100+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c1098e0 osd=0 3 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.100+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 0 2 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.100+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c1098e0 osd=0 4 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.100+0000 7f35a1fba700 10 client.27813.objecter _send_command 2 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.100+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f359c05a0e0 con 0x7f359c028590 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.102+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f359c05a0e0 con 0x7f359c028590 2024-02-13T00:50:44.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.102+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.102+0000 7f3598ff9700 10 client.27813.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.102+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c1098e0 osd=0 4 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.102+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 0 2 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.102+0000 7f35a1fba700 10 client.27813.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.102+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 3 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.102+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6805/150599780 conn(0x7f359c059570 0x7f359c119110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c059400 osd=1 3 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c059400 osd=1 3 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c059400 osd=1 3 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 3 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c059400 osd=1 3 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 1 3 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c059400 osd=1 4 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f35a1fba700 10 client.27813.objecter _send_command 3 2024-02-13T00:50:44.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f359c0280f0 con 0x7f359c059570 2024-02-13T00:50:44.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f359b7fe700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6805/150599780 conn(0x7f359c059570 0x7f359c119110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:44.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f3598ff9700 10 client.27813.objecter ms_handle_connect 0x7f359c059570 2024-02-13T00:50:44.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f359b7fe700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6805/150599780 conn(0x7f359c059570 0x7f359c119110 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:44.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f359c0280f0 con 0x7f359c059570 2024-02-13T00:50:44.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:50:44.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f3598ff9700 10 client.27813.objecter _finish_command 3 = system:0 2024-02-13T00:50:44.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c059400 osd=1 4 2024-02-13T00:50:44.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.103+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 1 3 2024-02-13T00:50:44.120 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:44.120 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:44.128 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:44.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.125+0000 7f35a1fba700 10 client.27813.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:44.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.125+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 4 2024-02-13T00:50:44.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.125+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c059400 osd=1 3 2024-02-13T00:50:44.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.125+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c059400 osd=1 3 2024-02-13T00:50:44.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.125+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c059400 osd=1 3 2024-02-13T00:50:44.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.125+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 4 2024-02-13T00:50:44.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.125+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c059400 osd=1 3 2024-02-13T00:50:44.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.125+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 1 4 2024-02-13T00:50:44.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.125+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c059400 osd=1 4 2024-02-13T00:50:44.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.125+0000 7f35a1fba700 10 client.27813.objecter _send_command 4 2024-02-13T00:50:44.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.125+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f359c119600 con 0x7f359c059570 2024-02-13T00:50:44.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f359c119600 con 0x7f359c059570 2024-02-13T00:50:44.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:44.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f3598ff9700 10 client.27813.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c059400 osd=1 4 2024-02-13T00:50:44.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 1 4 2024-02-13T00:50:44.132 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:50:44.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f35a1fba700 10 client.27813.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:44.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 5 2024-02-13T00:50:44.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6809/3689719376 conn(0x7f359c0277c0 0x7f359c11b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:44.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c027650 osd=2 3 2024-02-13T00:50:44.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c027650 osd=2 3 2024-02-13T00:50:44.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c027650 osd=2 3 2024-02-13T00:50:44.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 5 2024-02-13T00:50:44.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c027650 osd=2 3 2024-02-13T00:50:44.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 2 5 2024-02-13T00:50:44.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c027650 osd=2 4 2024-02-13T00:50:44.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f35a1fba700 10 client.27813.objecter _send_command 5 2024-02-13T00:50:44.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.128+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f359c11bcd0 con 0x7f359c0277c0 2024-02-13T00:50:44.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.129+0000 7f359affd700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6809/3689719376 conn(0x7f359c0277c0 0x7f359c11b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:44.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.129+0000 7f3598ff9700 10 client.27813.objecter ms_handle_connect 0x7f359c0277c0 2024-02-13T00:50:44.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.129+0000 7f359affd700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6809/3689719376 conn(0x7f359c0277c0 0x7f359c11b7e0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:44.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.130+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f359c11bcd0 con 0x7f359c0277c0 2024-02-13T00:50:44.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.130+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:50:44.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.130+0000 7f3598ff9700 10 client.27813.objecter _finish_command 5 = system:0 2024-02-13T00:50:44.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.130+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c027650 osd=2 4 2024-02-13T00:50:44.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.130+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 2 5 2024-02-13T00:50:44.150 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:44.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.149+0000 7f35a1fba700 10 client.27813.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:44.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.149+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 6 2024-02-13T00:50:44.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.149+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c027650 osd=2 3 2024-02-13T00:50:44.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.149+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c027650 osd=2 3 2024-02-13T00:50:44.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.149+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c027650 osd=2 3 2024-02-13T00:50:44.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.149+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 6 2024-02-13T00:50:44.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.149+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c027650 osd=2 3 2024-02-13T00:50:44.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.149+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 2 6 2024-02-13T00:50:44.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.149+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c027650 osd=2 4 2024-02-13T00:50:44.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.149+0000 7f35a1fba700 10 client.27813.objecter _send_command 6 2024-02-13T00:50:44.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.149+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f359c0fd110 con 0x7f359c0277c0 2024-02-13T00:50:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:44.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:44.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.151+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f359c0fd110 con 0x7f359c0277c0 2024-02-13T00:50:44.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.151+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:44.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.151+0000 7f3598ff9700 10 client.27813.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.151+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c027650 osd=2 4 2024-02-13T00:50:44.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.151+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 2 6 2024-02-13T00:50:44.155 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:50:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f35a1fba700 10 client.27813.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 7 2024-02-13T00:50:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6813/604971011 conn(0x7f359c0fc5a0 0x7f359c122350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c0fc430 osd=3 3 2024-02-13T00:50:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c0fc430 osd=3 3 2024-02-13T00:50:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c0fc430 osd=3 3 2024-02-13T00:50:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 7 2024-02-13T00:50:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c0fc430 osd=3 3 2024-02-13T00:50:44.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 3 7 2024-02-13T00:50:44.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c0fc430 osd=3 4 2024-02-13T00:50:44.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f35a1fba700 10 client.27813.objecter _send_command 7 2024-02-13T00:50:44.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f359c0f8960 con 0x7f359c0fc5a0 2024-02-13T00:50:44.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f359bfff700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6813/604971011 conn(0x7f359c0fc5a0 0x7f359c122350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:44.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f3598ff9700 10 client.27813.objecter ms_handle_connect 0x7f359c0fc5a0 2024-02-13T00:50:44.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.152+0000 7f359bfff700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6813/604971011 conn(0x7f359c0fc5a0 0x7f359c122350 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:44.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.153+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f359c0f8960 con 0x7f359c0fc5a0 2024-02-13T00:50:44.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.153+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:50:44.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.153+0000 7f3598ff9700 10 client.27813.objecter _finish_command 7 = system:0 2024-02-13T00:50:44.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.153+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c0fc430 osd=3 4 2024-02-13T00:50:44.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.153+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 3 7 2024-02-13T00:50:44.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.171+0000 7f35a1fba700 10 client.27813.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:44.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.171+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 8 2024-02-13T00:50:44.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.171+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c0fc430 osd=3 3 2024-02-13T00:50:44.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.171+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c0fc430 osd=3 3 2024-02-13T00:50:44.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.171+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c0fc430 osd=3 3 2024-02-13T00:50:44.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.171+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 8 2024-02-13T00:50:44.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.171+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c0fc430 osd=3 3 2024-02-13T00:50:44.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.171+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 3 8 2024-02-13T00:50:44.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.171+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c0fc430 osd=3 4 2024-02-13T00:50:44.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.171+0000 7f35a1fba700 10 client.27813.objecter _send_command 8 2024-02-13T00:50:44.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.171+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f359c0f8b80 con 0x7f359c0fc5a0 2024-02-13T00:50:44.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f359c0f8b80 con 0x7f359c0fc5a0 2024-02-13T00:50:44.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:44.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f3598ff9700 10 client.27813.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c0fc430 osd=3 4 2024-02-13T00:50:44.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 3 8 2024-02-13T00:50:44.177 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:50:44.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f35a1fba700 10 client.27813.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:44.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 9 2024-02-13T00:50:44.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6800/3344263388 conn(0x7f359c0f7a20 0x7f359c0f7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:44.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c0f8b80 osd=4 3 2024-02-13T00:50:44.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c0f8b80 osd=4 3 2024-02-13T00:50:44.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c0f8b80 osd=4 3 2024-02-13T00:50:44.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 9 2024-02-13T00:50:44.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c0f8b80 osd=4 3 2024-02-13T00:50:44.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 4 9 2024-02-13T00:50:44.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c0f8b80 osd=4 4 2024-02-13T00:50:44.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f35a1fba700 10 client.27813.objecter _send_command 9 2024-02-13T00:50:44.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.173+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f359c128130 con 0x7f359c0f7a20 2024-02-13T00:50:44.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.174+0000 7f359b7fe700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6800/3344263388 conn(0x7f359c0f7a20 0x7f359c0f7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:44.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.174+0000 7f359b7fe700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6800/3344263388 conn(0x7f359c0f7a20 0x7f359c0f7e20 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:44.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.174+0000 7f3598ff9700 10 client.27813.objecter ms_handle_connect 0x7f359c0f7a20 2024-02-13T00:50:44.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.175+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f359c128130 con 0x7f359c0f7a20 2024-02-13T00:50:44.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.175+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:50:44.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.175+0000 7f3598ff9700 10 client.27813.objecter _finish_command 9 = system:0 2024-02-13T00:50:44.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.175+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c0f8b80 osd=4 4 2024-02-13T00:50:44.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.175+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 4 9 2024-02-13T00:50:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.190+0000 7f35a1fba700 10 client.27813.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.190+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 10 2024-02-13T00:50:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.190+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c0f8b80 osd=4 3 2024-02-13T00:50:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.190+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c0f8b80 osd=4 3 2024-02-13T00:50:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.190+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c0f8b80 osd=4 3 2024-02-13T00:50:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.190+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 10 2024-02-13T00:50:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.190+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c0f8b80 osd=4 3 2024-02-13T00:50:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.190+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 4 10 2024-02-13T00:50:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.190+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c0f8b80 osd=4 4 2024-02-13T00:50:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.190+0000 7f35a1fba700 10 client.27813.objecter _send_command 10 2024-02-13T00:50:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.190+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f359c1259d0 con 0x7f359c0f7a20 2024-02-13T00:50:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f359c1259d0 con 0x7f359c0f7a20 2024-02-13T00:50:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f3598ff9700 10 client.27813.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c0f8b80 osd=4 4 2024-02-13T00:50:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 4 10 2024-02-13T00:50:44.196 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:50:44.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f35a1fba700 10 client.27813.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:44.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 11 2024-02-13T00:50:44.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6804/2277917372 conn(0x7f359c126080 0x7f359c1284f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c1259d0 osd=5 3 2024-02-13T00:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c1259d0 osd=5 3 2024-02-13T00:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c1259d0 osd=5 3 2024-02-13T00:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 11 2024-02-13T00:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c1259d0 osd=5 3 2024-02-13T00:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 5 11 2024-02-13T00:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c1259d0 osd=5 4 2024-02-13T00:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f35a1fba700 10 client.27813.objecter _send_command 11 2024-02-13T00:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f359c12b2c0 con 0x7f359c126080 2024-02-13T00:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.192+0000 7f359affd700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6804/2277917372 conn(0x7f359c126080 0x7f359c1284f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.193+0000 7f3598ff9700 10 client.27813.objecter ms_handle_connect 0x7f359c126080 2024-02-13T00:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.193+0000 7f359affd700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6804/2277917372 conn(0x7f359c126080 0x7f359c1284f0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:44.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.193+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f359c12b2c0 con 0x7f359c126080 2024-02-13T00:50:44.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.194+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:50:44.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.194+0000 7f3598ff9700 10 client.27813.objecter _finish_command 11 = system:0 2024-02-13T00:50:44.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.194+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c1259d0 osd=5 4 2024-02-13T00:50:44.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.194+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 5 11 2024-02-13T00:50:44.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.208+0000 7f35a1fba700 10 client.27813.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:44.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.208+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 12 2024-02-13T00:50:44.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.208+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c1259d0 osd=5 3 2024-02-13T00:50:44.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.208+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c1259d0 osd=5 3 2024-02-13T00:50:44.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.208+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c1259d0 osd=5 3 2024-02-13T00:50:44.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.208+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 12 2024-02-13T00:50:44.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.208+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c1259d0 osd=5 3 2024-02-13T00:50:44.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.208+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 5 12 2024-02-13T00:50:44.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.208+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c1259d0 osd=5 4 2024-02-13T00:50:44.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.208+0000 7f35a1fba700 10 client.27813.objecter _send_command 12 2024-02-13T00:50:44.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.208+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f359c12b2c0 con 0x7f359c126080 2024-02-13T00:50:44.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f359c12b2c0 con 0x7f359c126080 2024-02-13T00:50:44.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:44.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f3598ff9700 10 client.27813.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c1259d0 osd=5 4 2024-02-13T00:50:44.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 5 12 2024-02-13T00:50:44.214 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:50:44.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f35a1fba700 10 client.27813.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:44.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 13 2024-02-13T00:50:44.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6808/1506640341 conn(0x7f359c12a5c0 0x7f359c12aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:44.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c12b2c0 osd=6 3 2024-02-13T00:50:44.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c12b2c0 osd=6 3 2024-02-13T00:50:44.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c12b2c0 osd=6 3 2024-02-13T00:50:44.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 13 2024-02-13T00:50:44.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c12b2c0 osd=6 3 2024-02-13T00:50:44.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 6 13 2024-02-13T00:50:44.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c12b2c0 osd=6 4 2024-02-13T00:50:44.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f35a1fba700 10 client.27813.objecter _send_command 13 2024-02-13T00:50:44.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.210+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f359c12b090 con 0x7f359c12a5c0 2024-02-13T00:50:44.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.211+0000 7f359bfff700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6808/1506640341 conn(0x7f359c12a5c0 0x7f359c12aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:44.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.211+0000 7f3598ff9700 10 client.27813.objecter ms_handle_connect 0x7f359c12a5c0 2024-02-13T00:50:44.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.211+0000 7f359bfff700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6808/1506640341 conn(0x7f359c12a5c0 0x7f359c12aa20 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:44.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.212+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f359c12b090 con 0x7f359c12a5c0 2024-02-13T00:50:44.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.212+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:50:44.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.212+0000 7f3598ff9700 10 client.27813.objecter _finish_command 13 = system:0 2024-02-13T00:50:44.216 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.212+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c12b2c0 osd=6 4 2024-02-13T00:50:44.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.212+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 6 13 2024-02-13T00:50:44.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.226+0000 7f35a1fba700 10 client.27813.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:44.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.226+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 14 2024-02-13T00:50:44.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.226+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c12b2c0 osd=6 3 2024-02-13T00:50:44.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.226+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c12b2c0 osd=6 3 2024-02-13T00:50:44.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.226+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c12b2c0 osd=6 3 2024-02-13T00:50:44.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.226+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 14 2024-02-13T00:50:44.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.226+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c12b2c0 osd=6 3 2024-02-13T00:50:44.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.226+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 6 14 2024-02-13T00:50:44.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.226+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c12b2c0 osd=6 4 2024-02-13T00:50:44.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.226+0000 7f35a1fba700 10 client.27813.objecter _send_command 14 2024-02-13T00:50:44.230 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.226+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f359c12b090 con 0x7f359c12a5c0 2024-02-13T00:50:44.231 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:50:44.231 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.227+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f359c12b090 con 0x7f359c12a5c0 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.227+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.227+0000 7f3598ff9700 10 client.27813.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.227+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c12b2c0 osd=6 4 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.227+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 6 14 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f35a1fba700 10 client.27813.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 15 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6812/1964192772 conn(0x7f359c12db00 0x7f359c12df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c12b090 osd=7 3 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c12b090 osd=7 3 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c12b090 osd=7 3 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 15 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c12b090 osd=7 3 2024-02-13T00:50:44.232 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 7 15 2024-02-13T00:50:44.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c12b090 osd=7 4 2024-02-13T00:50:44.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f35a1fba700 10 client.27813.objecter _send_command 15 2024-02-13T00:50:44.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f359c1318b0 con 0x7f359c12db00 2024-02-13T00:50:44.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f359b7fe700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6812/1964192772 conn(0x7f359c12db00 0x7f359c12df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:44.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f3598ff9700 10 client.27813.objecter ms_handle_connect 0x7f359c12db00 2024-02-13T00:50:44.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.228+0000 7f359b7fe700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6812/1964192772 conn(0x7f359c12db00 0x7f359c12df40 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:44.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.229+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f359c1318b0 con 0x7f359c12db00 2024-02-13T00:50:44.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.229+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:50:44.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.229+0000 7f3598ff9700 10 client.27813.objecter _finish_command 15 = system:0 2024-02-13T00:50:44.233 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.229+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c12b090 osd=7 4 2024-02-13T00:50:44.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.229+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 7 15 2024-02-13T00:50:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.243+0000 7f35a1fba700 10 client.27813.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:50:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.243+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign -1 16 2024-02-13T00:50:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.243+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c12b090 osd=7 3 2024-02-13T00:50:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.243+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c12b090 osd=7 3 2024-02-13T00:50:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.243+0000 7f35a1fba700 20 client.27813.objecter _get_session s=0x7f359c12b090 osd=7 3 2024-02-13T00:50:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.243+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_remove -1 16 2024-02-13T00:50:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.243+0000 7f35a1fba700 20 client.27813.objecter get_session s=0x7f359c12b090 osd=7 3 2024-02-13T00:50:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.243+0000 7f35a1fba700 15 client.27813.objecter _session_command_op_assign 7 16 2024-02-13T00:50:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.243+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c12b090 osd=7 4 2024-02-13T00:50:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.243+0000 7f35a1fba700 10 client.27813.objecter _send_command 16 2024-02-13T00:50:44.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.243+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f359c128aa0 con 0x7f359c12db00 2024-02-13T00:50:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f3598ff9700 1 -- 172.21.15.162:0/82149337 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f359c128aa0 con 0x7f359c12db00 2024-02-13T00:50:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f3598ff9700 10 client.27813.objecter ms_dispatch 0x7f359c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:50:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f3598ff9700 10 client.27813.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f3598ff9700 20 client.27813.objecter put_session s=0x7f359c12b090 osd=7 4 2024-02-13T00:50:44.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f3598ff9700 15 client.27813.objecter _session_command_op_remove 7 16 2024-02-13T00:50:44.250 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:50:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 10 librados: watch_flush enter 2024-02-13T00:50:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 10 librados: watch_flush exit 2024-02-13T00:50:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 10 client.27813.objecter close_session for osd.0 2024-02-13T00:50:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6801/2103743338 conn(0x7f359c028590 msgr2=0x7f359c0fbd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6801/2103743338 conn(0x7f359c028590 0x7f359c0fbd00 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c1098e0 osd=0 1 2024-02-13T00:50:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 10 client.27813.objecter close_session for osd.1 2024-02-13T00:50:44.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6805/150599780 conn(0x7f359c059570 msgr2=0x7f359c119110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6805/150599780 conn(0x7f359c059570 0x7f359c119110 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c059400 osd=1 1 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 10 client.27813.objecter close_session for osd.2 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6809/3689719376 conn(0x7f359c0277c0 msgr2=0x7f359c11b7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6809/3689719376 conn(0x7f359c0277c0 0x7f359c11b7e0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c027650 osd=2 1 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 10 client.27813.objecter close_session for osd.3 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6813/604971011 conn(0x7f359c0fc5a0 msgr2=0x7f359c122350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6813/604971011 conn(0x7f359c0fc5a0 0x7f359c122350 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c0fc430 osd=3 1 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 10 client.27813.objecter close_session for osd.4 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6800/3344263388 conn(0x7f359c0f7a20 msgr2=0x7f359c0f7e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f359bfff700 1 -- 172.21.15.162:0/82149337 reap_dead start 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6800/3344263388 conn(0x7f359c0f7a20 0x7f359c0f7e20 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f359bfff700 1 -- 172.21.15.162:0/82149337 reap_dead start 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c0f8b80 osd=4 1 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 10 client.27813.objecter close_session for osd.5 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6804/2277917372 conn(0x7f359c126080 msgr2=0x7f359c1284f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6804/2277917372 conn(0x7f359c126080 0x7f359c1284f0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c1259d0 osd=5 1 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 10 client.27813.objecter close_session for osd.6 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6808/1506640341 conn(0x7f359c12a5c0 msgr2=0x7f359c12aa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6808/1506640341 conn(0x7f359c12a5c0 0x7f359c12aa20 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c12b2c0 osd=6 1 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 10 client.27813.objecter close_session for osd.7 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6812/1964192772 conn(0x7f359c12db00 msgr2=0x7f359c12df40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.181:6812/1964192772 conn(0x7f359c12db00 0x7f359c12df40 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 20 client.27813.objecter put_session s=0x7f359c12b090 osd=7 1 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 20 client.27813.objecter shutdown clearing up homeless session... 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 10 client.27813.objecter successfully canceled tick 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3584062c90 msgr2=0x7f3584065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3584062c90 0x7f3584065150 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 rx=0x7f358c009810 tx=0x7f358c00b040).stop 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f359c102270 msgr2=0x7f359c108400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 --2- 172.21.15.162:0/82149337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f359c102270 0x7f359c108400 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 rx=0x7f359400d8a0 tx=0x7f3594004db0).stop 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f359bfff700 1 -- 172.21.15.162:0/82149337 reap_dead start 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 shutdown_connections 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 >> 172.21.15.162:0/82149337 conn(0x7f359c06eb90 msgr2=0x7f359c102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 shutdown_connections 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 -- 172.21.15.162:0/82149337 wait complete. 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:44.246+0000 7f35a1fba700 1 librados: shutdown 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.254 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.255 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.255 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.255 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.255 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.255 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.255 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:50:44.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:44.283 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:44.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:44.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:44.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:44.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:44.440 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:44.460 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:44.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:44.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:44.599 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:44.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:44.084674+0000 mon.a (mon.0) 3155 : audit [DBG] from='client.? 172.21.15.162:0/82149337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:44.736 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:44.084674+0000 mon.a (mon.0) 3155 : audit [DBG] from='client.? 172.21.15.162:0/82149337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:44.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:44.084674+0000 mon.a (mon.0) 3155 : audit [DBG] from='client.? 172.21.15.162:0/82149337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:45.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:50:45.548 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:45.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:45.653 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:45.682 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:45.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:44.496183+0000 mgr.y (mgr.14544) 532 : cluster [DBG] pgmap v937: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 569 B/s rd, 0 op/s 2024-02-13T00:50:45.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:45.510249+0000 mon.a (mon.0) 3156 : cluster [DBG] osdmap e623: 8 total, 8 up, 4 in 2024-02-13T00:50:45.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:44.496183+0000 mgr.y (mgr.14544) 532 : cluster [DBG] pgmap v937: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 569 B/s rd, 0 op/s 2024-02-13T00:50:45.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:45.510249+0000 mon.a (mon.0) 3156 : cluster [DBG] osdmap e623: 8 total, 8 up, 4 in 2024-02-13T00:50:46.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:50:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:50:46.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:44.496183+0000 mgr.y (mgr.14544) 532 : cluster [DBG] pgmap v937: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 569 B/s rd, 0 op/s 2024-02-13T00:50:46.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:45.510249+0000 mon.a (mon.0) 3156 : cluster [DBG] osdmap e623: 8 total, 8 up, 4 in 2024-02-13T00:50:46.247 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:50:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:50:46] "GET /metrics HTTP/1.1" 200 189081 "" "Prometheus/2.33.4" 2024-02-13T00:50:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.479+0000 7f0b1fb67700 1 -- 172.21.15.162:0/3959651115 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b180fa440 msgr2=0x7f0b180f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.479+0000 7f0b1fb67700 1 --2- 172.21.15.162:0/3959651115 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b180fa440 0x7f0b180f7e20 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 rx=0x7f0b0c004870 tx=0x7f0b0c009cb0).stop 2024-02-13T00:50:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.479+0000 7f0b1fb67700 1 -- 172.21.15.162:0/3959651115 shutdown_connections 2024-02-13T00:50:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.479+0000 7f0b1fb67700 1 --2- 172.21.15.162:0/3959651115 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b180f8500 0x7f0b18104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.479+0000 7f0b1fb67700 1 --2- 172.21.15.162:0/3959651115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b180f9b70 0x7f0b180f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.479+0000 7f0b1fb67700 1 --2- 172.21.15.162:0/3959651115 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b180fa440 0x7f0b180f7e20 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.479+0000 7f0b1fb67700 1 -- 172.21.15.162:0/3959651115 >> 172.21.15.162:0/3959651115 conn(0x7f0b180f38c0 msgr2=0x7f0b180f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 -- 172.21.15.162:0/3959651115 shutdown_connections 2024-02-13T00:50:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 -- 172.21.15.162:0/3959651115 wait complete. 2024-02-13T00:50:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 librados: starting msgr at 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 librados: starting objecter 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 Processor -- start 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 -- start start 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 librados: setting wanted keys 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 librados: calling monclient init 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b180f8500 0x7f0b1806b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b180f9b70 0x7f0b1806b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b180fa440 0x7f0b1806bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b18116670 con 0x7f0b180f8500 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b181167e0 con 0x7f0b180fa440 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1fb67700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b181169a0 con 0x7f0b180f9b70 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1d903700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b180f8500 0x7f0b1806b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1d102700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b180f9b70 0x7f0b1806b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1d903700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b180f8500 0x7f0b1806b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:55716/0 (socket says 172.21.15.162:55716) 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1d903700 1 -- 172.21.15.162:0/2615289385 learned_addr learned my addr 172.21.15.162:0/2615289385 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.480+0000 7f0b1d102700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b180f9b70 0x7f0b1806b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:54102/0 (socket says 172.21.15.162:54102) 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1e104700 1 --2- 172.21.15.162:0/2615289385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b180fa440 0x7f0b1806bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1d903700 1 -- 172.21.15.162:0/2615289385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b180f9b70 msgr2=0x7f0b1806b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1d903700 1 --2- 172.21.15.162:0/2615289385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b180f9b70 0x7f0b1806b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1d903700 1 -- 172.21.15.162:0/2615289385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b180fa440 msgr2=0x7f0b1806bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1d903700 1 --2- 172.21.15.162:0/2615289385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b180fa440 0x7f0b1806bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1d903700 1 -- 172.21.15.162:0/2615289385 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b080097c0 con 0x7f0b180f8500 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1d102700 1 --2- 172.21.15.162:0/2615289385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b180f9b70 0x7f0b1806b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1d903700 1 --2- 172.21.15.162:0/2615289385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b180f8500 0x7f0b1806b170 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 rx=0x7f0b08001e70 tx=0x7f0b08004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b16ffd700 10 client.?.objecter ms_handle_connect 0x7f0b180f8500 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b16ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b16ffd700 1 -- 172.21.15.162:0/2615289385 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b080099f0 con 0x7f0b180f8500 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b16ffd700 1 -- 172.21.15.162:0/2615289385 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b080090d0 con 0x7f0b180f8500 2024-02-13T00:50:46.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b16ffd700 1 -- 172.21.15.162:0/2615289385 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b08007c70 con 0x7f0b180f8500 2024-02-13T00:50:46.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1fb67700 1 -- 172.21.15.162:0/2615289385 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0c005040 con 0x7f0b180f8500 2024-02-13T00:50:46.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1fb67700 10 client.27819.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:46.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1fb67700 1 -- 172.21.15.162:0/2615289385 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b18116f90 con 0x7f0b180f8500 2024-02-13T00:50:46.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.481+0000 7f0b1fb67700 1 librados: init done 2024-02-13T00:50:46.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.482+0000 7f0b1fb67700 1 -- 172.21.15.162:0/2615289385 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b180475e0 con 0x7f0b180f8500 2024-02-13T00:50:46.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.482+0000 7f0b16ffd700 1 -- 172.21.15.162:0/2615289385 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0b0800fad0 con 0x7f0b180f8500 2024-02-13T00:50:46.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.483+0000 7f0b16ffd700 1 --2- 172.21.15.162:0/2615289385 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0b00062c90 0x7f0b00065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:46.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.483+0000 7f0b16ffd700 1 -- 172.21.15.162:0/2615289385 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(623..623 src has 449..623) v4 ==== 8470+0+0 (secure 0 0 0) 0x7f0b0807dc90 con 0x7f0b180f8500 2024-02-13T00:50:46.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.483+0000 7f0b16ffd700 10 client.27819.objecter ms_dispatch 0x7f0b18001550 osd_map(623..623 src has 449..623) v4 2024-02-13T00:50:46.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.483+0000 7f0b16ffd700 3 client.27819.objecter handle_osd_map got epochs [623,623] > 0 2024-02-13T00:50:46.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.483+0000 7f0b1d102700 1 --2- 172.21.15.162:0/2615289385 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0b00062c90 0x7f0b00065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:46.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.483+0000 7f0b16ffd700 3 client.27819.objecter handle_osd_map decoding full epoch 623 2024-02-13T00:50:46.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.483+0000 7f0b16ffd700 20 client.27819.objecter dump_active .. 0 homeless 2024-02-13T00:50:46.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.483+0000 7f0b16ffd700 10 client.27819.objecter ms_handle_connect 0x7f0b00062c90 2024-02-13T00:50:46.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.483+0000 7f0b1d102700 1 --2- 172.21.15.162:0/2615289385 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0b00062c90 0x7f0b00065150 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 rx=0x7f0b0c0043d0 tx=0x7f0b0c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:46.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.487+0000 7f0b16ffd700 1 -- 172.21.15.162:0/2615289385 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0b08083020 con 0x7f0b180f8500 2024-02-13T00:50:46.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:46.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:46.574 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:46.639 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:46.667 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:46.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.667+0000 7f0b1fb67700 1 -- 172.21.15.162:0/2615289385 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b1806c650 con 0x7f0b00062c90 2024-02-13T00:50:46.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.681+0000 7f0b16ffd700 1 -- 172.21.15.162:0/2615289385 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+506710 (secure 0 0 0) 0x7f0b1806c650 con 0x7f0b00062c90 2024-02-13T00:50:46.691 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:50:46.695 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.684+0000 7f0b1fb67700 10 librados: watch_flush enter 2024-02-13T00:50:46.695 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.684+0000 7f0b1fb67700 10 librados: watch_flush exit 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.684+0000 7f0b1fb67700 20 client.27819.objecter shutdown clearing up homeless session... 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 10 client.27819.objecter successfully canceled tick 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 -- 172.21.15.162:0/2615289385 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0b00062c90 msgr2=0x7f0b00065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 --2- 172.21.15.162:0/2615289385 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0b00062c90 0x7f0b00065150 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 rx=0x7f0b0c0043d0 tx=0x7f0b0c00b040).stop 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 -- 172.21.15.162:0/2615289385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b180f8500 msgr2=0x7f0b1806b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 --2- 172.21.15.162:0/2615289385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b180f8500 0x7f0b1806b170 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 rx=0x7f0b08001e70 tx=0x7f0b08004c50).stop 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 -- 172.21.15.162:0/2615289385 shutdown_connections 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 --2- 172.21.15.162:0/2615289385 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0b00062c90 0x7f0b00065150 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 --2- 172.21.15.162:0/2615289385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b180fa440 0x7f0b1806bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 --2- 172.21.15.162:0/2615289385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b180f8500 0x7f0b1806b170 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 --2- 172.21.15.162:0/2615289385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0b180f9b70 0x7f0b1806b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 -- 172.21.15.162:0/2615289385 >> 172.21.15.162:0/2615289385 conn(0x7f0b180f38c0 msgr2=0x7f0b180f4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 -- 172.21.15.162:0/2615289385 shutdown_connections 2024-02-13T00:50:46.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 -- 172.21.15.162:0/2615289385 wait complete. 2024-02-13T00:50:46.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:46.685+0000 7f0b1fb67700 1 librados: shutdown 2024-02-13T00:50:46.697 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:50:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:45.808613+0000 mgr.y (mgr.14544) 533 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:46.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:45.808613+0000 mgr.y (mgr.14544) 533 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:46.916 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:46.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:47.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:45.808613+0000 mgr.y (mgr.14544) 533 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:47.050 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:47.148 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:47.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:47.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:47.284 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:47.306 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:47.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:47.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:47.409 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:47.439 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:47.570 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:50:47.788 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:47.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:47.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:47.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:47.918 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:46.496534+0000 mgr.y (mgr.14544) 534 : cluster [DBG] pgmap v939: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:47.918 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:46.668313+0000 mgr.y (mgr.14544) 535 : audit [DBG] from='client.27819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:50:47.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:46.496534+0000 mgr.y (mgr.14544) 534 : cluster [DBG] pgmap v939: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:47.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:46.668313+0000 mgr.y (mgr.14544) 535 : audit [DBG] from='client.27819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:50:47.920 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:47.938 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:47.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:48.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:48.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:46.496534+0000 mgr.y (mgr.14544) 534 : cluster [DBG] pgmap v939: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:50:48.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:46.668313+0000 mgr.y (mgr.14544) 535 : audit [DBG] from='client.27819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:50:48.074 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:48.096 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:48.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:48.200 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:48.227 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:48.299 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:48.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:48.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:48.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:48.432 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:48.453 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:48.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:48.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:48.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:48.584 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:48.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:50:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:50:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:50:48.830 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":939,"stamp":"2024-02-13T00:50:46.496343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5870,"num_objects":237,"num_object_clones":4,"num_object_copies":711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":235,"num_whiteouts":1,"num_read":2280,"num_read_kb":1997,"num_write":923,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":13678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1060,"ondisk_log_size":1060,"up":612,"acting":612,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":612,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1226988,"kb_used_data":13516,"kb_used_omap":0,"kb_used_meta":1213440,"kb_avail":373753620,"statfs":{"total":383980142592,"available":382723706880,"internally_reserved":0,"allocated":13840384,"data_stored":6690733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1242562560},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002376"},"pg_stats":[{"pgid":"237.1f","version":"620'1","reported_seq":17,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.566212+0000","last_change":"2024-02-13T00:50:30.533460+0000","last_active":"2024-02-13T00:50:35.566212+0000","last_peered":"2024-02-13T00:50:35.566212+0000","last_clean":"2024-02-13T00:50:35.566212+0000","last_became_active":"2024-02-13T00:50:30.533228+0000","last_became_peered":"2024-02-13T00:50:30.533228+0000","last_unstale":"2024-02-13T00:50:35.566212+0000","last_undegraded":"2024-02-13T00:50:35.566212+0000","last_fullsized":"2024-02-13T00:50:35.566212+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.1e","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511684+0000","last_change":"2024-02-13T00:50:30.608142+0000","last_active":"2024-02-13T00:50:35.511684+0000","last_peered":"2024-02-13T00:50:35.511684+0000","last_clean":"2024-02-13T00:50:35.511684+0000","last_became_active":"2024-02-13T00:50:30.607836+0000","last_became_peered":"2024-02-13T00:50:30.607836+0000","last_unstale":"2024-02-13T00:50:35.511684+0000","last_undegraded":"2024-02-13T00:50:35.511684+0000","last_fullsized":"2024-02-13T00:50:35.511684+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.1d","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565641+0000","last_change":"2024-02-13T00:50:30.608813+0000","last_active":"2024-02-13T00:50:35.565641+0000","last_peered":"2024-02-13T00:50:35.565641+0000","last_clean":"2024-02-13T00:50:35.565641+0000","last_became_active":"2024-02-13T00:50:30.608515+0000","last_became_peered":"2024-02-13T00:50:30.608515+0000","last_unstale":"2024-02-13T00:50:35.565641+0000","last_undegraded":"2024-02-13T00:50:35.565641+0000","last_fullsized":"2024-02-13T00:50:35.565641+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.1c","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564027+0000","last_change":"2024-02-13T00:50:30.613473+0000","last_active":"2024-02-13T00:50:35.564027+0000","last_peered":"2024-02-13T00:50:35.564027+0000","last_clean":"2024-02-13T00:50:35.564027+0000","last_became_active":"2024-02-13T00:50:30.613193+0000","last_became_peered":"2024-02-13T00:50:30.613193+0000","last_unstale":"2024-02-13T00:50:35.564027+0000","last_undegraded":"2024-02-13T00:50:35.564027+0000","last_fullsized":"2024-02-13T00:50:35.564027+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.1b","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563264+0000","last_change":"2024-02-13T00:50:30.530812+0000","last_active":"2024-02-13T00:50:35.563264+0000","last_peered":"2024-02-13T00:50:35.563264+0000","last_clean":"2024-02-13T00:50:35.563264+0000","last_became_active":"2024-02-13T00:50:30.530525+0000","last_became_peered":"2024-02-13T00:50:30.530525+0000","last_unstale":"2024-02-13T00:50:35.563264+0000","last_undegraded":"2024-02-13T00:50:35.563264+0000","last_fullsized":"2024-02-13T00:50:35.563264+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.1a","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509410+0000","last_change":"2024-02-13T00:50:30.608653+0000","last_active":"2024-02-13T00:50:35.509410+0000","last_peered":"2024-02-13T00:50:35.509410+0000","last_clean":"2024-02-13T00:50:35.509410+0000","last_became_active":"2024-02-13T00:50:30.607930+0000","last_became_peered":"2024-02-13T00:50:30.607930+0000","last_unstale":"2024-02-13T00:50:35.509410+0000","last_undegraded":"2024-02-13T00:50:35.509410+0000","last_fullsized":"2024-02-13T00:50:35.509410+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.19","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563356+0000","last_change":"2024-02-13T00:50:30.606323+0000","last_active":"2024-02-13T00:50:35.563356+0000","last_peered":"2024-02-13T00:50:35.563356+0000","last_clean":"2024-02-13T00:50:35.563356+0000","last_became_active":"2024-02-13T00:50:30.606019+0000","last_became_peered":"2024-02-13T00:50:30.606019+0000","last_unstale":"2024-02-13T00:50:35.563356+0000","last_undegraded":"2024-02-13T00:50:35.563356+0000","last_fullsized":"2024-02-13T00:50:35.563356+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.18","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.562813+0000","last_change":"2024-02-13T00:50:30.607191+0000","last_active":"2024-02-13T00:50:35.562813+0000","last_peered":"2024-02-13T00:50:35.562813+0000","last_clean":"2024-02-13T00:50:35.562813+0000","last_became_active":"2024-02-13T00:50:30.606897+0000","last_became_peered":"2024-02-13T00:50:30.606897+0000","last_unstale":"2024-02-13T00:50:35.562813+0000","last_undegraded":"2024-02-13T00:50:35.562813+0000","last_fullsized":"2024-02-13T00:50:35.562813+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.17","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565844+0000","last_change":"2024-02-13T00:50:30.553975+0000","last_active":"2024-02-13T00:50:35.565844+0000","last_peered":"2024-02-13T00:50:35.565844+0000","last_clean":"2024-02-13T00:50:35.565844+0000","last_became_active":"2024-02-13T00:50:30.553668+0000","last_became_peered":"2024-02-13T00:50:30.553668+0000","last_unstale":"2024-02-13T00:50:35.565844+0000","last_undegraded":"2024-02-13T00:50:35.565844+0000","last_fullsized":"2024-02-13T00:50:35.565844+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.16","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564333+0000","last_change":"2024-02-13T00:50:30.610120+0000","last_active":"2024-02-13T00:50:35.564333+0000","last_peered":"2024-02-13T00:50:35.564333+0000","last_clean":"2024-02-13T00:50:35.564333+0000","last_became_active":"2024-02-13T00:50:30.609833+0000","last_became_peered":"2024-02-13T00:50:30.609833+0000","last_unstale":"2024-02-13T00:50:35.564333+0000","last_undegraded":"2024-02-13T00:50:35.564333+0000","last_fullsized":"2024-02-13T00:50:35.564333+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.15","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511257+0000","last_change":"2024-02-13T00:50:30.615758+0000","last_active":"2024-02-13T00:50:35.511257+0000","last_peered":"2024-02-13T00:50:35.511257+0000","last_clean":"2024-02-13T00:50:35.511257+0000","last_became_active":"2024-02-13T00:50:30.615466+0000","last_became_peered":"2024-02-13T00:50:30.615466+0000","last_unstale":"2024-02-13T00:50:35.511257+0000","last_undegraded":"2024-02-13T00:50:35.511257+0000","last_fullsized":"2024-02-13T00:50:35.511257+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.14","version":"620'1","reported_seq":17,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510099+0000","last_change":"2024-02-13T00:50:30.558706+0000","last_active":"2024-02-13T00:50:35.510099+0000","last_peered":"2024-02-13T00:50:35.510099+0000","last_clean":"2024-02-13T00:50:35.510099+0000","last_became_active":"2024-02-13T00:50:30.558406+0000","last_became_peered":"2024-02-13T00:50:30.558406+0000","last_unstale":"2024-02-13T00:50:35.510099+0000","last_undegraded":"2024-02-13T00:50:35.510099+0000","last_fullsized":"2024-02-13T00:50:35.510099+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.13","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564296+0000","last_change":"2024-02-13T00:50:30.533512+0000","last_active":"2024-02-13T00:50:35.564296+0000","last_peered":"2024-02-13T00:50:35.564296+0000","last_clean":"2024-02-13T00:50:35.564296+0000","last_became_active":"2024-02-13T00:50:30.533230+0000","last_became_peered":"2024-02-13T00:50:30.533230+0000","last_unstale":"2024-02-13T00:50:35.564296+0000","last_undegraded":"2024-02-13T00:50:35.564296+0000","last_fullsized":"2024-02-13T00:50:35.564296+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.12","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509456+0000","last_change":"2024-02-13T00:50:30.528844+0000","last_active":"2024-02-13T00:50:35.509456+0000","last_peered":"2024-02-13T00:50:35.509456+0000","last_clean":"2024-02-13T00:50:35.509456+0000","last_became_active":"2024-02-13T00:50:30.528547+0000","last_became_peered":"2024-02-13T00:50:30.528547+0000","last_unstale":"2024-02-13T00:50:35.509456+0000","last_undegraded":"2024-02-13T00:50:35.509456+0000","last_fullsized":"2024-02-13T00:50:35.509456+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.11","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564904+0000","last_change":"2024-02-13T00:50:30.606668+0000","last_active":"2024-02-13T00:50:35.564904+0000","last_peered":"2024-02-13T00:50:35.564904+0000","last_clean":"2024-02-13T00:50:35.564904+0000","last_became_active":"2024-02-13T00:50:30.606382+0000","last_became_peered":"2024-02-13T00:50:30.606382+0000","last_unstale":"2024-02-13T00:50:35.564904+0000","last_undegraded":"2024-02-13T00:50:35.564904+0000","last_fullsized":"2024-02-13T00:50:35.564904+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.10","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509127+0000","last_change":"2024-02-13T00:50:30.612711+0000","last_active":"2024-02-13T00:50:35.509127+0000","last_peered":"2024-02-13T00:50:35.509127+0000","last_clean":"2024-02-13T00:50:35.509127+0000","last_became_active":"2024-02-13T00:50:30.609229+0000","last_became_peered":"2024-02-13T00:50:30.609229+0000","last_unstale":"2024-02-13T00:50:35.509127+0000","last_undegraded":"2024-02-13T00:50:35.509127+0000","last_fullsized":"2024-02-13T00:50:35.509127+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.f","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510934+0000","last_change":"2024-02-13T00:50:30.584984+0000","last_active":"2024-02-13T00:50:35.510934+0000","last_peered":"2024-02-13T00:50:35.510934+0000","last_clean":"2024-02-13T00:50:35.510934+0000","last_became_active":"2024-02-13T00:50:30.584690+0000","last_became_peered":"2024-02-13T00:50:30.584690+0000","last_unstale":"2024-02-13T00:50:35.510934+0000","last_undegraded":"2024-02-13T00:50:35.510934+0000","last_fullsized":"2024-02-13T00:50:35.510934+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.e","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511753+0000","last_change":"2024-02-13T00:50:30.558567+0000","last_active":"2024-02-13T00:50:35.511753+0000","last_peered":"2024-02-13T00:50:35.511753+0000","last_clean":"2024-02-13T00:50:35.511753+0000","last_became_active":"2024-02-13T00:50:30.558318+0000","last_became_peered":"2024-02-13T00:50:30.558318+0000","last_unstale":"2024-02-13T00:50:35.511753+0000","last_undegraded":"2024-02-13T00:50:35.511753+0000","last_fullsized":"2024-02-13T00:50:35.511753+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.d","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564229+0000","last_change":"2024-02-13T00:50:30.613333+0000","last_active":"2024-02-13T00:50:35.564229+0000","last_peered":"2024-02-13T00:50:35.564229+0000","last_clean":"2024-02-13T00:50:35.564229+0000","last_became_active":"2024-02-13T00:50:30.613083+0000","last_became_peered":"2024-02-13T00:50:30.613083+0000","last_unstale":"2024-02-13T00:50:35.564229+0000","last_undegraded":"2024-02-13T00:50:35.564229+0000","last_fullsized":"2024-02-13T00:50:35.564229+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.c","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564100+0000","last_change":"2024-02-13T00:50:30.606190+0000","last_active":"2024-02-13T00:50:35.564100+0000","last_peered":"2024-02-13T00:50:35.564100+0000","last_clean":"2024-02-13T00:50:35.564100+0000","last_became_active":"2024-02-13T00:50:30.605896+0000","last_became_peered":"2024-02-13T00:50:30.605896+0000","last_unstale":"2024-02-13T00:50:35.564100+0000","last_undegraded":"2024-02-13T00:50:35.564100+0000","last_fullsized":"2024-02-13T00:50:35.564100+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.b","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563356+0000","last_change":"2024-02-13T00:50:30.530658+0000","last_active":"2024-02-13T00:50:35.563356+0000","last_peered":"2024-02-13T00:50:35.563356+0000","last_clean":"2024-02-13T00:50:35.563356+0000","last_became_active":"2024-02-13T00:50:30.530360+0000","last_became_peered":"2024-02-13T00:50:30.530360+0000","last_unstale":"2024-02-13T00:50:35.563356+0000","last_undegraded":"2024-02-13T00:50:35.563356+0000","last_fullsized":"2024-02-13T00:50:35.563356+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.a","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564108+0000","last_change":"2024-02-13T00:50:30.606562+0000","last_active":"2024-02-13T00:50:35.564108+0000","last_peered":"2024-02-13T00:50:35.564108+0000","last_clean":"2024-02-13T00:50:35.564108+0000","last_became_active":"2024-02-13T00:50:30.606264+0000","last_became_peered":"2024-02-13T00:50:30.606264+0000","last_unstale":"2024-02-13T00:50:35.564108+0000","last_undegraded":"2024-02-13T00:50:35.564108+0000","last_fullsized":"2024-02-13T00:50:35.564108+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.9","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.562911+0000","last_change":"2024-02-13T00:50:30.526358+0000","last_active":"2024-02-13T00:50:35.562911+0000","last_peered":"2024-02-13T00:50:35.562911+0000","last_clean":"2024-02-13T00:50:35.562911+0000","last_became_active":"2024-02-13T00:50:30.526114+0000","last_became_peered":"2024-02-13T00:50:30.526114+0000","last_unstale":"2024-02-13T00:50:35.562911+0000","last_undegraded":"2024-02-13T00:50:35.562911+0000","last_fullsized":"2024-02-13T00:50:35.562911+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.8","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.562813+0000","last_change":"2024-02-13T00:50:30.528377+0000","last_active":"2024-02-13T00:50:35.562813+0000","last_peered":"2024-02-13T00:50:35.562813+0000","last_clean":"2024-02-13T00:50:35.562813+0000","last_became_active":"2024-02-13T00:50:30.528089+0000","last_became_peered":"2024-02-13T00:50:30.528089+0000","last_unstale":"2024-02-13T00:50:35.562813+0000","last_undegraded":"2024-02-13T00:50:35.562813+0000","last_fullsized":"2024-02-13T00:50:35.562813+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.0","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.562901+0000","last_change":"2024-02-13T00:50:30.606190+0000","last_active":"2024-02-13T00:50:35.562901+0000","last_peered":"2024-02-13T00:50:35.562901+0000","last_clean":"2024-02-13T00:50:35.562901+0000","last_became_active":"2024-02-13T00:50:30.605850+0000","last_became_peered":"2024-02-13T00:50:30.605850+0000","last_unstale":"2024-02-13T00:50:35.562901+0000","last_undegraded":"2024-02-13T00:50:35.562901+0000","last_fullsized":"2024-02-13T00:50:35.562901+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.7","version":"617'4","reported_seq":37,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564869+0000","last_change":"2024-02-13T00:50:14.554059+0000","last_active":"2024-02-13T00:50:35.564869+0000","last_peered":"2024-02-13T00:50:35.564869+0000","last_clean":"2024-02-13T00:50:35.564869+0000","last_became_active":"2024-02-13T00:50:14.553744+0000","last_became_peered":"2024-02-13T00:50:14.553744+0000","last_unstale":"2024-02-13T00:50:35.564869+0000","last_undegraded":"2024-02-13T00:50:35.564869+0000","last_fullsized":"2024-02-13T00:50:35.564869+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.1","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509303+0000","last_change":"2024-02-13T00:50:30.609838+0000","last_active":"2024-02-13T00:50:35.509303+0000","last_peered":"2024-02-13T00:50:35.509303+0000","last_clean":"2024-02-13T00:50:35.509303+0000","last_became_active":"2024-02-13T00:50:30.608853+0000","last_became_peered":"2024-02-13T00:50:30.608853+0000","last_unstale":"2024-02-13T00:50:35.509303+0000","last_undegraded":"2024-02-13T00:50:35.509303+0000","last_fullsized":"2024-02-13T00:50:35.509303+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.6","version":"0'0","reported_seq":31,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564276+0000","last_change":"2024-02-13T00:50:14.553031+0000","last_active":"2024-02-13T00:50:35.564276+0000","last_peered":"2024-02-13T00:50:35.564276+0000","last_clean":"2024-02-13T00:50:35.564276+0000","last_became_active":"2024-02-13T00:50:14.552756+0000","last_became_peered":"2024-02-13T00:50:14.552756+0000","last_unstale":"2024-02-13T00:50:35.564276+0000","last_undegraded":"2024-02-13T00:50:35.564276+0000","last_fullsized":"2024-02-13T00:50:35.564276+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.2","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509496+0000","last_change":"2024-02-13T00:50:30.608848+0000","last_active":"2024-02-13T00:50:35.509496+0000","last_peered":"2024-02-13T00:50:35.509496+0000","last_clean":"2024-02-13T00:50:35.509496+0000","last_became_active":"2024-02-13T00:50:30.608355+0000","last_became_peered":"2024-02-13T00:50:30.608355+0000","last_unstale":"2024-02-13T00:50:35.509496+0000","last_undegraded":"2024-02-13T00:50:35.509496+0000","last_fullsized":"2024-02-13T00:50:35.509496+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.5","version":"617'3","reported_seq":33,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564348+0000","last_change":"2024-02-13T00:50:14.551022+0000","last_active":"2024-02-13T00:50:35.564348+0000","last_peered":"2024-02-13T00:50:35.564348+0000","last_clean":"2024-02-13T00:50:35.564348+0000","last_became_active":"2024-02-13T00:50:14.550644+0000","last_became_peered":"2024-02-13T00:50:14.550644+0000","last_unstale":"2024-02-13T00:50:35.564348+0000","last_undegraded":"2024-02-13T00:50:35.564348+0000","last_fullsized":"2024-02-13T00:50:35.564348+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.3","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563698+0000","last_change":"2024-02-13T00:50:30.606212+0000","last_active":"2024-02-13T00:50:35.563698+0000","last_peered":"2024-02-13T00:50:35.563698+0000","last_clean":"2024-02-13T00:50:35.563698+0000","last_became_active":"2024-02-13T00:50:30.605890+0000","last_became_peered":"2024-02-13T00:50:30.605890+0000","last_unstale":"2024-02-13T00:50:35.563698+0000","last_undegraded":"2024-02-13T00:50:35.563698+0000","last_fullsized":"2024-02-13T00:50:35.563698+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.4","version":"617'3","reported_seq":34,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510287+0000","last_change":"2024-02-13T00:50:14.550087+0000","last_active":"2024-02-13T00:50:35.510287+0000","last_peered":"2024-02-13T00:50:35.510287+0000","last_clean":"2024-02-13T00:50:35.510287+0000","last_became_active":"2024-02-13T00:50:14.549625+0000","last_became_peered":"2024-02-13T00:50:14.549625+0000","last_unstale":"2024-02-13T00:50:35.510287+0000","last_undegraded":"2024-02-13T00:50:35.510287+0000","last_fullsized":"2024-02-13T00:50:35.510287+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.4","version":"0'0","reported_seq":20,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:36.089300+0000","last_change":"2024-02-13T00:50:36.089274+0000","last_active":"2024-02-13T00:50:36.089300+0000","last_peered":"2024-02-13T00:50:36.089300+0000","last_clean":"2024-02-13T00:50:36.089300+0000","last_became_active":"2024-02-13T00:50:30.609300+0000","last_became_peered":"2024-02-13T00:50:30.609300+0000","last_unstale":"2024-02-13T00:50:36.089300+0000","last_undegraded":"2024-02-13T00:50:36.089300+0000","last_fullsized":"2024-02-13T00:50:36.089300+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:36.089196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:36.089196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.3","version":"0'0","reported_seq":31,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564571+0000","last_change":"2024-02-13T00:50:14.550875+0000","last_active":"2024-02-13T00:50:35.564571+0000","last_peered":"2024-02-13T00:50:35.564571+0000","last_clean":"2024-02-13T00:50:35.564571+0000","last_became_active":"2024-02-13T00:50:14.549053+0000","last_became_peered":"2024-02-13T00:50:14.549053+0000","last_unstale":"2024-02-13T00:50:35.564571+0000","last_undegraded":"2024-02-13T00:50:35.564571+0000","last_fullsized":"2024-02-13T00:50:35.564571+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.5","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510542+0000","last_change":"2024-02-13T00:50:30.615076+0000","last_active":"2024-02-13T00:50:35.510542+0000","last_peered":"2024-02-13T00:50:35.510542+0000","last_clean":"2024-02-13T00:50:35.510542+0000","last_became_active":"2024-02-13T00:50:30.610265+0000","last_became_peered":"2024-02-13T00:50:30.610265+0000","last_unstale":"2024-02-13T00:50:35.510542+0000","last_undegraded":"2024-02-13T00:50:35.510542+0000","last_fullsized":"2024-02-13T00:50:35.510542+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.2","version":"617'3","reported_seq":33,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564213+0000","last_change":"2024-02-13T00:50:14.552504+0000","last_active":"2024-02-13T00:50:35.564213+0000","last_peered":"2024-02-13T00:50:35.564213+0000","last_clean":"2024-02-13T00:50:35.564213+0000","last_became_active":"2024-02-13T00:50:14.552162+0000","last_became_peered":"2024-02-13T00:50:14.552162+0000","last_unstale":"2024-02-13T00:50:35.564213+0000","last_undegraded":"2024-02-13T00:50:35.564213+0000","last_fullsized":"2024-02-13T00:50:35.564213+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.6","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564171+0000","last_change":"2024-02-13T00:50:30.607665+0000","last_active":"2024-02-13T00:50:35.564171+0000","last_peered":"2024-02-13T00:50:35.564171+0000","last_clean":"2024-02-13T00:50:35.564171+0000","last_became_active":"2024-02-13T00:50:30.607391+0000","last_became_peered":"2024-02-13T00:50:30.607391+0000","last_unstale":"2024-02-13T00:50:35.564171+0000","last_undegraded":"2024-02-13T00:50:35.564171+0000","last_fullsized":"2024-02-13T00:50:35.564171+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.1","version":"0'0","reported_seq":31,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.562945+0000","last_change":"2024-02-13T00:50:14.553804+0000","last_active":"2024-02-13T00:50:35.562945+0000","last_peered":"2024-02-13T00:50:35.562945+0000","last_clean":"2024-02-13T00:50:35.562945+0000","last_became_active":"2024-02-13T00:50:14.553460+0000","last_became_peered":"2024-02-13T00:50:14.553460+0000","last_unstale":"2024-02-13T00:50:35.562945+0000","last_undegraded":"2024-02-13T00:50:35.562945+0000","last_fullsized":"2024-02-13T00:50:35.562945+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.7","version":"0'0","reported_seq":15,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564773+0000","last_change":"2024-02-13T00:50:30.554034+0000","last_active":"2024-02-13T00:50:35.564773+0000","last_peered":"2024-02-13T00:50:35.564773+0000","last_clean":"2024-02-13T00:50:35.564773+0000","last_became_active":"2024-02-13T00:50:30.553731+0000","last_became_peered":"2024-02-13T00:50:30.553731+0000","last_unstale":"2024-02-13T00:50:35.564773+0000","last_undegraded":"2024-02-13T00:50:35.564773+0000","last_fullsized":"2024-02-13T00:50:35.564773+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":615,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:29.510954+0000","last_clean_scrub_stamp":"2024-02-13T00:50:29.510954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.0","version":"608'1","reported_seq":34,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563376+0000","last_change":"2024-02-13T00:50:14.551368+0000","last_active":"2024-02-13T00:50:35.563376+0000","last_peered":"2024-02-13T00:50:35.563376+0000","last_clean":"2024-02-13T00:50:35.563376+0000","last_became_active":"2024-02-13T00:50:14.551016+0000","last_became_peered":"2024-02-13T00:50:14.551016+0000","last_unstale":"2024-02-13T00:50:35.563376+0000","last_undegraded":"2024-02-13T00:50:35.563376+0000","last_fullsized":"2024-02-13T00:50:35.563376+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":528,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509978+0000","last_change":"2024-02-13T00:50:32.100672+0000","last_active":"2024-02-13T00:50:35.509978+0000","last_peered":"2024-02-13T00:50:35.509978+0000","last_clean":"2024-02-13T00:50:35.509978+0000","last_became_active":"2024-02-13T00:46:01.445363+0000","last_became_peered":"2024-02-13T00:46:01.445363+0000","last_unstale":"2024-02-13T00:50:35.509978+0000","last_undegraded":"2024-02-13T00:50:35.509978+0000","last_fullsized":"2024-02-13T00:50:35.509978+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:32.100607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:50:32.100607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":447,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563911+0000","last_change":"2024-02-13T00:46:01.449894+0000","last_active":"2024-02-13T00:50:35.563911+0000","last_peered":"2024-02-13T00:50:35.563911+0000","last_clean":"2024-02-13T00:50:35.563911+0000","last_became_active":"2024-02-13T00:46:01.449570+0000","last_became_peered":"2024-02-13T00:46:01.449570+0000","last_unstale":"2024-02-13T00:50:35.563911+0000","last_undegraded":"2024-02-13T00:50:35.563911+0000","last_fullsized":"2024-02-13T00:50:35.563911+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":313,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563127+0000","last_change":"2024-02-13T00:46:01.445141+0000","last_active":"2024-02-13T00:50:35.563127+0000","last_peered":"2024-02-13T00:50:35.563127+0000","last_clean":"2024-02-13T00:50:35.563127+0000","last_became_active":"2024-02-13T00:46:01.444729+0000","last_became_peered":"2024-02-13T00:46:01.444729+0000","last_unstale":"2024-02-13T00:50:35.563127+0000","last_undegraded":"2024-02-13T00:50:35.563127+0000","last_fullsized":"2024-02-13T00:50:35.563127+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":403,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563208+0000","last_change":"2024-02-13T00:50:16.677491+0000","last_active":"2024-02-13T00:50:35.563208+0000","last_peered":"2024-02-13T00:50:35.563208+0000","last_clean":"2024-02-13T00:50:35.563208+0000","last_became_active":"2024-02-13T00:45:15.426109+0000","last_became_peered":"2024-02-13T00:45:15.426109+0000","last_unstale":"2024-02-13T00:50:35.563208+0000","last_undegraded":"2024-02-13T00:50:35.563208+0000","last_fullsized":"2024-02-13T00:50:35.563208+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:16.677431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:16.677431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":308,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567020+0000","last_change":"2024-02-13T00:50:22.469511+0000","last_active":"2024-02-13T00:50:35.567020+0000","last_peered":"2024-02-13T00:50:35.567020+0000","last_clean":"2024-02-13T00:50:35.567020+0000","last_became_active":"2024-02-13T00:44:39.525161+0000","last_became_peered":"2024-02-13T00:44:39.525161+0000","last_unstale":"2024-02-13T00:50:35.567020+0000","last_undegraded":"2024-02-13T00:50:35.567020+0000","last_fullsized":"2024-02-13T00:50:35.567020+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.469431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.469431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":401,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563422+0000","last_change":"2024-02-13T00:48:06.705421+0000","last_active":"2024-02-13T00:50:35.563422+0000","last_peered":"2024-02-13T00:50:35.563422+0000","last_clean":"2024-02-13T00:50:35.563422+0000","last_became_active":"2024-02-13T00:48:06.705121+0000","last_became_peered":"2024-02-13T00:48:06.705121+0000","last_unstale":"2024-02-13T00:50:35.563422+0000","last_undegraded":"2024-02-13T00:50:35.563422+0000","last_fullsized":"2024-02-13T00:50:35.563422+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":321,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565549+0000","last_change":"2024-02-13T00:46:01.459772+0000","last_active":"2024-02-13T00:50:35.565549+0000","last_peered":"2024-02-13T00:50:35.565549+0000","last_clean":"2024-02-13T00:50:35.565549+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:50:35.565549+0000","last_undegraded":"2024-02-13T00:50:35.565549+0000","last_fullsized":"2024-02-13T00:50:35.565549+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":580,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565690+0000","last_change":"2024-02-13T00:46:01.463062+0000","last_active":"2024-02-13T00:50:35.565690+0000","last_peered":"2024-02-13T00:50:35.565690+0000","last_clean":"2024-02-13T00:50:35.565690+0000","last_became_active":"2024-02-13T00:46:01.462319+0000","last_became_peered":"2024-02-13T00:46:01.462319+0000","last_unstale":"2024-02-13T00:50:35.565690+0000","last_undegraded":"2024-02-13T00:50:35.565690+0000","last_fullsized":"2024-02-13T00:50:35.565690+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":308,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567072+0000","last_change":"2024-02-13T00:49:48.070260+0000","last_active":"2024-02-13T00:50:35.567072+0000","last_peered":"2024-02-13T00:50:35.567072+0000","last_clean":"2024-02-13T00:50:35.567072+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:50:35.567072+0000","last_undegraded":"2024-02-13T00:50:35.567072+0000","last_fullsized":"2024-02-13T00:50:35.567072+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:48.070195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:49:48.070195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":389,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565057+0000","last_change":"2024-02-13T00:46:01.455629+0000","last_active":"2024-02-13T00:50:35.565057+0000","last_peered":"2024-02-13T00:50:35.565057+0000","last_clean":"2024-02-13T00:50:35.565057+0000","last_became_active":"2024-02-13T00:46:01.452801+0000","last_became_peered":"2024-02-13T00:46:01.452801+0000","last_unstale":"2024-02-13T00:50:35.565057+0000","last_undegraded":"2024-02-13T00:50:35.565057+0000","last_fullsized":"2024-02-13T00:50:35.565057+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":542,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509831+0000","last_change":"2024-02-13T00:45:39.562095+0000","last_active":"2024-02-13T00:50:35.509831+0000","last_peered":"2024-02-13T00:50:35.509831+0000","last_clean":"2024-02-13T00:50:35.509831+0000","last_became_active":"2024-02-13T00:44:39.505121+0000","last_became_peered":"2024-02-13T00:44:39.505121+0000","last_unstale":"2024-02-13T00:50:35.509831+0000","last_undegraded":"2024-02-13T00:50:35.509831+0000","last_fullsized":"2024-02-13T00:50:35.509831+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:39.561994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:39.561994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"534'16","reported_seq":420,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-13T00:50:40.684898+0000","last_change":"2024-02-13T00:49:07.494629+0000","last_active":"2024-02-13T00:50:40.684898+0000","last_peered":"2024-02-13T00:50:40.684898+0000","last_clean":"2024-02-13T00:50:40.684898+0000","last_became_active":"2024-02-13T00:49:07.494388+0000","last_became_peered":"2024-02-13T00:49:07.494388+0000","last_unstale":"2024-02-13T00:50:40.684898+0000","last_undegraded":"2024-02-13T00:50:40.684898+0000","last_fullsized":"2024-02-13T00:50:40.684898+0000","mapping_epoch":359,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:45:53.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.115260+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":647,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563783+0000","last_change":"2024-02-13T00:47:03.491720+0000","last_active":"2024-02-13T00:50:35.563783+0000","last_peered":"2024-02-13T00:50:35.563783+0000","last_clean":"2024-02-13T00:50:35.563783+0000","last_became_active":"2024-02-13T00:46:01.438056+0000","last_became_peered":"2024-02-13T00:46:01.438056+0000","last_unstale":"2024-02-13T00:50:35.563783+0000","last_undegraded":"2024-02-13T00:50:35.563783+0000","last_fullsized":"2024-02-13T00:50:35.563783+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:47:03.491647+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":617,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564799+0000","last_change":"2024-02-13T00:48:23.379872+0000","last_active":"2024-02-13T00:50:35.564799+0000","last_peered":"2024-02-13T00:50:35.564799+0000","last_clean":"2024-02-13T00:50:35.564799+0000","last_became_active":"2024-02-13T00:46:01.460811+0000","last_became_peered":"2024-02-13T00:46:01.460811+0000","last_unstale":"2024-02-13T00:50:35.564799+0000","last_undegraded":"2024-02-13T00:50:35.564799+0000","last_fullsized":"2024-02-13T00:50:35.564799+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:23.379773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:23.379773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":523,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564784+0000","last_change":"2024-02-13T00:47:34.332791+0000","last_active":"2024-02-13T00:50:35.564784+0000","last_peered":"2024-02-13T00:50:35.564784+0000","last_clean":"2024-02-13T00:50:35.564784+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:50:35.564784+0000","last_undegraded":"2024-02-13T00:50:35.564784+0000","last_fullsized":"2024-02-13T00:50:35.564784+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:34.332704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:34.332704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":561,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565389+0000","last_change":"2024-02-13T00:46:01.448237+0000","last_active":"2024-02-13T00:50:35.565389+0000","last_peered":"2024-02-13T00:50:35.565389+0000","last_clean":"2024-02-13T00:50:35.565389+0000","last_became_active":"2024-02-13T00:46:01.446660+0000","last_became_peered":"2024-02-13T00:46:01.446660+0000","last_unstale":"2024-02-13T00:50:35.565389+0000","last_undegraded":"2024-02-13T00:50:35.565389+0000","last_fullsized":"2024-02-13T00:50:35.565389+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.260871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.260871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":621,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511117+0000","last_change":"2024-02-13T00:47:41.808333+0000","last_active":"2024-02-13T00:50:35.511117+0000","last_peered":"2024-02-13T00:50:35.511117+0000","last_clean":"2024-02-13T00:50:35.511117+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:50:35.511117+0000","last_undegraded":"2024-02-13T00:50:35.511117+0000","last_fullsized":"2024-02-13T00:50:35.511117+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:47:41.808259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:41.808259+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":604,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564996+0000","last_change":"2024-02-13T00:48:24.823190+0000","last_active":"2024-02-13T00:50:35.564996+0000","last_peered":"2024-02-13T00:50:35.564996+0000","last_clean":"2024-02-13T00:50:35.564996+0000","last_became_active":"2024-02-13T00:44:50.429994+0000","last_became_peered":"2024-02-13T00:44:50.429994+0000","last_unstale":"2024-02-13T00:50:35.564996+0000","last_undegraded":"2024-02-13T00:50:35.564996+0000","last_fullsized":"2024-02-13T00:50:35.564996+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:24.823107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.823107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":505,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509986+0000","last_change":"2024-02-13T00:49:39.788846+0000","last_active":"2024-02-13T00:50:35.509986+0000","last_peered":"2024-02-13T00:50:35.509986+0000","last_clean":"2024-02-13T00:50:35.509986+0000","last_became_active":"2024-02-13T00:49:03.971407+0000","last_became_peered":"2024-02-13T00:49:03.971407+0000","last_unstale":"2024-02-13T00:50:35.509986+0000","last_undegraded":"2024-02-13T00:50:35.509986+0000","last_fullsized":"2024-02-13T00:50:35.509986+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.788783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.788783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":604,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510334+0000","last_change":"2024-02-13T00:49:41.837871+0000","last_active":"2024-02-13T00:50:35.510334+0000","last_peered":"2024-02-13T00:50:35.510334+0000","last_clean":"2024-02-13T00:50:35.510334+0000","last_became_active":"2024-02-13T00:45:15.434853+0000","last_became_peered":"2024-02-13T00:45:15.434853+0000","last_unstale":"2024-02-13T00:50:35.510334+0000","last_undegraded":"2024-02-13T00:50:35.510334+0000","last_fullsized":"2024-02-13T00:50:35.510334+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.837810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.837810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":225,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509708+0000","last_change":"2024-02-13T00:46:01.455201+0000","last_active":"2024-02-13T00:50:35.509708+0000","last_peered":"2024-02-13T00:50:35.509708+0000","last_clean":"2024-02-13T00:50:35.509708+0000","last_became_active":"2024-02-13T00:46:01.454882+0000","last_became_peered":"2024-02-13T00:46:01.454882+0000","last_unstale":"2024-02-13T00:50:35.509708+0000","last_undegraded":"2024-02-13T00:50:35.509708+0000","last_fullsized":"2024-02-13T00:50:35.509708+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:35.197045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:35.197045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":411,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567227+0000","last_change":"2024-02-13T00:46:09.256189+0000","last_active":"2024-02-13T00:50:35.567227+0000","last_peered":"2024-02-13T00:50:35.567227+0000","last_clean":"2024-02-13T00:50:35.567227+0000","last_became_active":"2024-02-13T00:46:01.459978+0000","last_became_peered":"2024-02-13T00:46:01.459978+0000","last_unstale":"2024-02-13T00:50:35.567227+0000","last_undegraded":"2024-02-13T00:50:35.567227+0000","last_fullsized":"2024-02-13T00:50:35.567227+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:44:24.196214+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:44:24.196214+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":235,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511707+0000","last_change":"2024-02-13T00:46:30.641539+0000","last_active":"2024-02-13T00:50:35.511707+0000","last_peered":"2024-02-13T00:50:35.511707+0000","last_clean":"2024-02-13T00:50:35.511707+0000","last_became_active":"2024-02-13T00:46:01.444548+0000","last_became_peered":"2024-02-13T00:46:01.444548+0000","last_unstale":"2024-02-13T00:50:35.511707+0000","last_undegraded":"2024-02-13T00:50:35.511707+0000","last_fullsized":"2024-02-13T00:50:35.511707+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.641460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.641460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":335,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510637+0000","last_change":"2024-02-13T00:48:54.498301+0000","last_active":"2024-02-13T00:50:35.510637+0000","last_peered":"2024-02-13T00:50:35.510637+0000","last_clean":"2024-02-13T00:50:35.510637+0000","last_became_active":"2024-02-13T00:48:54.497977+0000","last_became_peered":"2024-02-13T00:48:54.497977+0000","last_unstale":"2024-02-13T00:50:35.510637+0000","last_undegraded":"2024-02-13T00:50:35.510637+0000","last_fullsized":"2024-02-13T00:50:35.510637+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.592672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":439,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563880+0000","last_change":"2024-02-13T00:45:15.429026+0000","last_active":"2024-02-13T00:50:35.563880+0000","last_peered":"2024-02-13T00:50:35.563880+0000","last_clean":"2024-02-13T00:50:35.563880+0000","last_became_active":"2024-02-13T00:45:15.427945+0000","last_became_peered":"2024-02-13T00:45:15.427945+0000","last_unstale":"2024-02-13T00:50:35.563880+0000","last_undegraded":"2024-02-13T00:50:35.563880+0000","last_fullsized":"2024-02-13T00:50:35.563880+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":544,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564884+0000","last_change":"2024-02-13T00:45:26.255907+0000","last_active":"2024-02-13T00:50:35.564884+0000","last_peered":"2024-02-13T00:50:35.564884+0000","last_clean":"2024-02-13T00:50:35.564884+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:50:35.564884+0000","last_undegraded":"2024-02-13T00:50:35.564884+0000","last_fullsized":"2024-02-13T00:50:35.564884+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.255842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.255842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"467'31","reported_seq":381,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564905+0000","last_change":"2024-02-13T00:48:31.782821+0000","last_active":"2024-02-13T00:50:35.564905+0000","last_peered":"2024-02-13T00:50:35.564905+0000","last_clean":"2024-02-13T00:50:35.564905+0000","last_became_active":"2024-02-13T00:44:50.432807+0000","last_became_peered":"2024-02-13T00:44:50.432807+0000","last_unstale":"2024-02-13T00:50:35.564905+0000","last_undegraded":"2024-02-13T00:50:35.564905+0000","last_fullsized":"2024-02-13T00:50:35.564905+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'31","last_scrub_stamp":"2024-02-13T00:48:31.782729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.782729+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":394,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564915+0000","last_change":"2024-02-13T00:46:04.708510+0000","last_active":"2024-02-13T00:50:35.564915+0000","last_peered":"2024-02-13T00:50:35.564915+0000","last_clean":"2024-02-13T00:50:35.564915+0000","last_became_active":"2024-02-13T00:46:01.447801+0000","last_became_peered":"2024-02-13T00:46:01.447801+0000","last_unstale":"2024-02-13T00:50:35.564915+0000","last_undegraded":"2024-02-13T00:50:35.564915+0000","last_fullsized":"2024-02-13T00:50:35.564915+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:45:24.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:24.269595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":602,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565082+0000","last_change":"2024-02-13T00:48:49.612735+0000","last_active":"2024-02-13T00:50:35.565082+0000","last_peered":"2024-02-13T00:50:35.565082+0000","last_clean":"2024-02-13T00:50:35.565082+0000","last_became_active":"2024-02-13T00:48:49.612437+0000","last_became_peered":"2024-02-13T00:48:49.612437+0000","last_unstale":"2024-02-13T00:50:35.565082+0000","last_undegraded":"2024-02-13T00:50:35.565082+0000","last_fullsized":"2024-02-13T00:50:35.565082+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.793634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":588,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565864+0000","last_change":"2024-02-13T00:44:50.441408+0000","last_active":"2024-02-13T00:50:35.565864+0000","last_peered":"2024-02-13T00:50:35.565864+0000","last_clean":"2024-02-13T00:50:35.565864+0000","last_became_active":"2024-02-13T00:44:50.440301+0000","last_became_peered":"2024-02-13T00:44:50.440301+0000","last_unstale":"2024-02-13T00:50:35.565864+0000","last_undegraded":"2024-02-13T00:50:35.565864+0000","last_fullsized":"2024-02-13T00:50:35.565864+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:25.502311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.502311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":247,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511523+0000","last_change":"2024-02-13T00:50:22.925676+0000","last_active":"2024-02-13T00:50:35.511523+0000","last_peered":"2024-02-13T00:50:35.511523+0000","last_clean":"2024-02-13T00:50:35.511523+0000","last_became_active":"2024-02-13T00:48:06.704718+0000","last_became_peered":"2024-02-13T00:48:06.704718+0000","last_unstale":"2024-02-13T00:50:35.511523+0000","last_undegraded":"2024-02-13T00:50:35.511523+0000","last_fullsized":"2024-02-13T00:50:35.511523+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.925604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.925604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":335,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565958+0000","last_change":"2024-02-13T00:49:03.294585+0000","last_active":"2024-02-13T00:50:35.565958+0000","last_peered":"2024-02-13T00:50:35.565958+0000","last_clean":"2024-02-13T00:50:35.565958+0000","last_became_active":"2024-02-13T00:44:39.523138+0000","last_became_peered":"2024-02-13T00:44:39.523138+0000","last_unstale":"2024-02-13T00:50:35.565958+0000","last_undegraded":"2024-02-13T00:50:35.565958+0000","last_fullsized":"2024-02-13T00:50:35.565958+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:49:03.294499+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:49:03.294499+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":613,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563615+0000","last_change":"2024-02-13T00:48:59.111910+0000","last_active":"2024-02-13T00:50:35.563615+0000","last_peered":"2024-02-13T00:50:35.563615+0000","last_clean":"2024-02-13T00:50:35.563615+0000","last_became_active":"2024-02-13T00:46:01.452674+0000","last_became_peered":"2024-02-13T00:46:01.452674+0000","last_unstale":"2024-02-13T00:50:35.563615+0000","last_undegraded":"2024-02-13T00:50:35.563615+0000","last_fullsized":"2024-02-13T00:50:35.563615+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:59.111828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:59.111828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"511'13","reported_seq":334,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-13T00:50:41.365080+0000","last_change":"2024-02-13T00:48:45.592523+0000","last_active":"2024-02-13T00:50:41.365080+0000","last_peered":"2024-02-13T00:50:41.365080+0000","last_clean":"2024-02-13T00:50:41.365080+0000","last_became_active":"2024-02-13T00:48:45.591249+0000","last_became_peered":"2024-02-13T00:48:45.591249+0000","last_unstale":"2024-02-13T00:50:41.365080+0000","last_undegraded":"2024-02-13T00:50:41.365080+0000","last_fullsized":"2024-02-13T00:50:41.365080+0000","mapping_epoch":401,"log_start":"404'11","ondisk_log_start":"404'11","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'11","last_scrub_stamp":"2024-02-13T00:46:37.662413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.662413+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":580,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565210+0000","last_change":"2024-02-13T00:46:05.839833+0000","last_active":"2024-02-13T00:50:35.565210+0000","last_peered":"2024-02-13T00:50:35.565210+0000","last_clean":"2024-02-13T00:50:35.565210+0000","last_became_active":"2024-02-13T00:46:01.455274+0000","last_became_peered":"2024-02-13T00:46:01.455274+0000","last_unstale":"2024-02-13T00:50:35.565210+0000","last_undegraded":"2024-02-13T00:50:35.565210+0000","last_fullsized":"2024-02-13T00:50:35.565210+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:44:25.230980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.230980+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":394,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565443+0000","last_change":"2024-02-13T00:50:26.830352+0000","last_active":"2024-02-13T00:50:35.565443+0000","last_peered":"2024-02-13T00:50:35.565443+0000","last_clean":"2024-02-13T00:50:35.565443+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:50:35.565443+0000","last_undegraded":"2024-02-13T00:50:35.565443+0000","last_fullsized":"2024-02-13T00:50:35.565443+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:26.830270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:26.830270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"467'27","reported_seq":702,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565400+0000","last_change":"2024-02-13T00:48:54.043163+0000","last_active":"2024-02-13T00:50:35.565400+0000","last_peered":"2024-02-13T00:50:35.565400+0000","last_clean":"2024-02-13T00:50:35.565400+0000","last_became_active":"2024-02-13T00:46:01.448705+0000","last_became_peered":"2024-02-13T00:46:01.448705+0000","last_unstale":"2024-02-13T00:50:35.565400+0000","last_undegraded":"2024-02-13T00:50:35.565400+0000","last_fullsized":"2024-02-13T00:50:35.565400+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:48:54.043088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.043088+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":645,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563344+0000","last_change":"2024-02-13T00:49:37.053462+0000","last_active":"2024-02-13T00:50:35.563344+0000","last_peered":"2024-02-13T00:50:35.563344+0000","last_clean":"2024-02-13T00:50:35.563344+0000","last_became_active":"2024-02-13T00:45:15.445228+0000","last_became_peered":"2024-02-13T00:45:15.445228+0000","last_unstale":"2024-02-13T00:50:35.563344+0000","last_undegraded":"2024-02-13T00:50:35.563344+0000","last_fullsized":"2024-02-13T00:50:35.563344+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:49:37.053398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.053398+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"507'13","reported_seq":332,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-13T00:50:41.366352+0000","last_change":"2024-02-13T00:48:41.526587+0000","last_active":"2024-02-13T00:50:41.366352+0000","last_peered":"2024-02-13T00:50:41.366352+0000","last_clean":"2024-02-13T00:50:41.366352+0000","last_became_active":"2024-02-13T00:48:41.525262+0000","last_became_peered":"2024-02-13T00:48:41.525262+0000","last_unstale":"2024-02-13T00:50:41.366352+0000","last_undegraded":"2024-02-13T00:50:41.366352+0000","last_fullsized":"2024-02-13T00:50:41.366352+0000","mapping_epoch":401,"log_start":"467'11","ondisk_log_start":"467'11","created":60,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:47:49.687727+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":235,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564689+0000","last_change":"2024-02-13T00:50:19.734411+0000","last_active":"2024-02-13T00:50:35.564689+0000","last_peered":"2024-02-13T00:50:35.564689+0000","last_clean":"2024-02-13T00:50:35.564689+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:50:35.564689+0000","last_undegraded":"2024-02-13T00:50:35.564689+0000","last_fullsized":"2024-02-13T00:50:35.564689+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:19.734320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:50:19.734320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":542,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.512514+0000","last_change":"2024-02-13T00:46:01.456531+0000","last_active":"2024-02-13T00:50:35.512514+0000","last_peered":"2024-02-13T00:50:35.512514+0000","last_clean":"2024-02-13T00:50:35.512514+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:50:35.512514+0000","last_undegraded":"2024-02-13T00:50:35.512514+0000","last_fullsized":"2024-02-13T00:50:35.512514+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:44:59.963616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":334,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565116+0000","last_change":"2024-02-13T00:48:57.386335+0000","last_active":"2024-02-13T00:50:35.565116+0000","last_peered":"2024-02-13T00:50:35.565116+0000","last_clean":"2024-02-13T00:50:35.565116+0000","last_became_active":"2024-02-13T00:44:39.521984+0000","last_became_peered":"2024-02-13T00:44:39.521984+0000","last_unstale":"2024-02-13T00:50:35.565116+0000","last_undegraded":"2024-02-13T00:50:35.565116+0000","last_fullsized":"2024-02-13T00:50:35.565116+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:57.386254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:57.386254+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":425,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509651+0000","last_change":"2024-02-13T00:46:23.757417+0000","last_active":"2024-02-13T00:50:35.509651+0000","last_peered":"2024-02-13T00:50:35.509651+0000","last_clean":"2024-02-13T00:50:35.509651+0000","last_became_active":"2024-02-13T00:45:15.419239+0000","last_became_peered":"2024-02-13T00:45:15.419239+0000","last_unstale":"2024-02-13T00:50:35.509651+0000","last_undegraded":"2024-02-13T00:50:35.509651+0000","last_fullsized":"2024-02-13T00:50:35.509651+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:23.757351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:23.757351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"504'16","reported_seq":573,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-13T00:50:40.686505+0000","last_change":"2024-02-13T00:49:45.051624+0000","last_active":"2024-02-13T00:50:40.686505+0000","last_peered":"2024-02-13T00:50:40.686505+0000","last_clean":"2024-02-13T00:50:40.686505+0000","last_became_active":"2024-02-13T00:48:37.480010+0000","last_became_peered":"2024-02-13T00:48:37.480010+0000","last_unstale":"2024-02-13T00:50:40.686505+0000","last_undegraded":"2024-02-13T00:50:40.686505+0000","last_fullsized":"2024-02-13T00:50:40.686505+0000","mapping_epoch":401,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'16","last_scrub_stamp":"2024-02-13T00:49:45.051539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:49:45.051539+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":326,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565775+0000","last_change":"2024-02-13T00:48:31.463534+0000","last_active":"2024-02-13T00:50:35.565775+0000","last_peered":"2024-02-13T00:50:35.565775+0000","last_clean":"2024-02-13T00:50:35.565775+0000","last_became_active":"2024-02-13T00:46:01.462717+0000","last_became_peered":"2024-02-13T00:46:01.462717+0000","last_unstale":"2024-02-13T00:50:35.565775+0000","last_undegraded":"2024-02-13T00:50:35.565775+0000","last_fullsized":"2024-02-13T00:50:35.565775+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:31.463442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.463442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":499,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509718+0000","last_change":"2024-02-13T00:49:00.881498+0000","last_active":"2024-02-13T00:50:35.509718+0000","last_peered":"2024-02-13T00:50:35.509718+0000","last_clean":"2024-02-13T00:50:35.509718+0000","last_became_active":"2024-02-13T00:46:01.456959+0000","last_became_peered":"2024-02-13T00:46:01.456959+0000","last_unstale":"2024-02-13T00:50:35.509718+0000","last_undegraded":"2024-02-13T00:50:35.509718+0000","last_fullsized":"2024-02-13T00:50:35.509718+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:00.881431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:00.881431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"263'10","reported_seq":628,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-13T00:50:40.568031+0000","last_change":"2024-02-13T00:47:02.949483+0000","last_active":"2024-02-13T00:50:40.568031+0000","last_peered":"2024-02-13T00:50:40.568031+0000","last_clean":"2024-02-13T00:50:40.568031+0000","last_became_active":"2024-02-13T00:43:35.454087+0000","last_became_peered":"2024-02-13T00:43:35.454087+0000","last_unstale":"2024-02-13T00:50:40.568031+0000","last_undegraded":"2024-02-13T00:50:40.568031+0000","last_fullsized":"2024-02-13T00:50:40.568031+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.949406+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":598,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567006+0000","last_change":"2024-02-13T00:49:46.101991+0000","last_active":"2024-02-13T00:50:35.567006+0000","last_peered":"2024-02-13T00:50:35.567006+0000","last_clean":"2024-02-13T00:50:35.567006+0000","last_became_active":"2024-02-13T00:46:01.449654+0000","last_became_peered":"2024-02-13T00:46:01.449654+0000","last_unstale":"2024-02-13T00:50:35.567006+0000","last_undegraded":"2024-02-13T00:50:35.567006+0000","last_fullsized":"2024-02-13T00:50:35.567006+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'34","last_scrub_stamp":"2024-02-13T00:49:46.101874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.101874+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":326,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.566232+0000","last_change":"2024-02-13T00:49:47.075845+0000","last_active":"2024-02-13T00:50:35.566232+0000","last_peered":"2024-02-13T00:50:35.566232+0000","last_clean":"2024-02-13T00:50:35.566232+0000","last_became_active":"2024-02-13T00:45:15.446443+0000","last_became_peered":"2024-02-13T00:45:15.446443+0000","last_unstale":"2024-02-13T00:50:35.566232+0000","last_undegraded":"2024-02-13T00:50:35.566232+0000","last_fullsized":"2024-02-13T00:50:35.566232+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:47.075741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.075741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":321,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565636+0000","last_change":"2024-02-13T00:45:15.436286+0000","last_active":"2024-02-13T00:50:35.565636+0000","last_peered":"2024-02-13T00:50:35.565636+0000","last_clean":"2024-02-13T00:50:35.565636+0000","last_became_active":"2024-02-13T00:45:15.434559+0000","last_became_peered":"2024-02-13T00:45:15.434559+0000","last_unstale":"2024-02-13T00:50:35.565636+0000","last_undegraded":"2024-02-13T00:50:35.565636+0000","last_fullsized":"2024-02-13T00:50:35.565636+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":349,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509569+0000","last_change":"2024-02-13T00:49:36.841674+0000","last_active":"2024-02-13T00:50:35.509569+0000","last_peered":"2024-02-13T00:50:35.509569+0000","last_clean":"2024-02-13T00:50:35.509569+0000","last_became_active":"2024-02-13T00:46:01.444661+0000","last_became_peered":"2024-02-13T00:46:01.444661+0000","last_unstale":"2024-02-13T00:50:35.509569+0000","last_undegraded":"2024-02-13T00:50:35.509569+0000","last_fullsized":"2024-02-13T00:50:35.509569+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:36.841508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:36.841508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":390,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509948+0000","last_change":"2024-02-13T00:48:18.751148+0000","last_active":"2024-02-13T00:50:35.509948+0000","last_peered":"2024-02-13T00:50:35.509948+0000","last_clean":"2024-02-13T00:50:35.509948+0000","last_became_active":"2024-02-13T00:46:01.453842+0000","last_became_peered":"2024-02-13T00:46:01.453842+0000","last_unstale":"2024-02-13T00:50:35.509948+0000","last_undegraded":"2024-02-13T00:50:35.509948+0000","last_fullsized":"2024-02-13T00:50:35.509948+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:18.751081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:18.751081+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"467'42","reported_seq":551,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509597+0000","last_change":"2024-02-13T00:48:24.675807+0000","last_active":"2024-02-13T00:50:35.509597+0000","last_peered":"2024-02-13T00:50:35.509597+0000","last_clean":"2024-02-13T00:50:35.509597+0000","last_became_active":"2024-02-13T00:46:01.445058+0000","last_became_peered":"2024-02-13T00:46:01.445058+0000","last_unstale":"2024-02-13T00:50:35.509597+0000","last_undegraded":"2024-02-13T00:50:35.509597+0000","last_fullsized":"2024-02-13T00:50:35.509597+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'42","last_scrub_stamp":"2024-02-13T00:48:24.675687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.675687+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":613,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565094+0000","last_change":"2024-02-13T00:47:09.940673+0000","last_active":"2024-02-13T00:50:35.565094+0000","last_peered":"2024-02-13T00:50:35.565094+0000","last_clean":"2024-02-13T00:50:35.565094+0000","last_became_active":"2024-02-13T00:44:39.503590+0000","last_became_peered":"2024-02-13T00:44:39.503590+0000","last_unstale":"2024-02-13T00:50:35.565094+0000","last_undegraded":"2024-02-13T00:50:35.565094+0000","last_fullsized":"2024-02-13T00:50:35.565094+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:09.940605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:09.940605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":556,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564866+0000","last_change":"2024-02-13T00:46:01.448313+0000","last_active":"2024-02-13T00:50:35.564866+0000","last_peered":"2024-02-13T00:50:35.564866+0000","last_clean":"2024-02-13T00:50:35.564866+0000","last_became_active":"2024-02-13T00:46:01.447815+0000","last_became_peered":"2024-02-13T00:46:01.447815+0000","last_unstale":"2024-02-13T00:50:35.564866+0000","last_undegraded":"2024-02-13T00:50:35.564866+0000","last_fullsized":"2024-02-13T00:50:35.564866+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":548,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565295+0000","last_change":"2024-02-13T00:48:35.434713+0000","last_active":"2024-02-13T00:50:35.565295+0000","last_peered":"2024-02-13T00:50:35.565295+0000","last_clean":"2024-02-13T00:50:35.565295+0000","last_became_active":"2024-02-13T00:44:50.436058+0000","last_became_peered":"2024-02-13T00:44:50.436058+0000","last_unstale":"2024-02-13T00:50:35.565295+0000","last_undegraded":"2024-02-13T00:50:35.565295+0000","last_fullsized":"2024-02-13T00:50:35.565295+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_clean_scrub_stamp":"2024-02-13T00:48:35.434645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":317,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567276+0000","last_change":"2024-02-13T00:46:01.451625+0000","last_active":"2024-02-13T00:50:35.567276+0000","last_peered":"2024-02-13T00:50:35.567276+0000","last_clean":"2024-02-13T00:50:35.567276+0000","last_became_active":"2024-02-13T00:46:01.450587+0000","last_became_peered":"2024-02-13T00:46:01.450587+0000","last_unstale":"2024-02-13T00:50:35.567276+0000","last_undegraded":"2024-02-13T00:50:35.567276+0000","last_fullsized":"2024-02-13T00:50:35.567276+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:33.204969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:33.204969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":714,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510742+0000","last_change":"2024-02-13T00:47:05.718055+0000","last_active":"2024-02-13T00:50:35.510742+0000","last_peered":"2024-02-13T00:50:35.510742+0000","last_clean":"2024-02-13T00:50:35.510742+0000","last_became_active":"2024-02-13T00:46:01.443777+0000","last_became_peered":"2024-02-13T00:46:01.443777+0000","last_unstale":"2024-02-13T00:50:35.510742+0000","last_undegraded":"2024-02-13T00:50:35.510742+0000","last_fullsized":"2024-02-13T00:50:35.510742+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:47:05.717987+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:47:05.717987+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":430,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511545+0000","last_change":"2024-02-13T00:50:25.001319+0000","last_active":"2024-02-13T00:50:35.511545+0000","last_peered":"2024-02-13T00:50:35.511545+0000","last_clean":"2024-02-13T00:50:35.511545+0000","last_became_active":"2024-02-13T00:45:15.423415+0000","last_became_peered":"2024-02-13T00:45:15.423415+0000","last_unstale":"2024-02-13T00:50:35.511545+0000","last_undegraded":"2024-02-13T00:50:35.511545+0000","last_fullsized":"2024-02-13T00:50:35.511545+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:25.001219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:25.001219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":625,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563461+0000","last_change":"2024-02-13T00:50:14.726735+0000","last_active":"2024-02-13T00:50:35.563461+0000","last_peered":"2024-02-13T00:50:35.563461+0000","last_clean":"2024-02-13T00:50:35.563461+0000","last_became_active":"2024-02-13T00:45:15.445557+0000","last_became_peered":"2024-02-13T00:45:15.445557+0000","last_unstale":"2024-02-13T00:50:35.563461+0000","last_undegraded":"2024-02-13T00:50:35.563461+0000","last_fullsized":"2024-02-13T00:50:35.563461+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.726660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.726660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":238,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564937+0000","last_change":"2024-02-13T00:48:55.050483+0000","last_active":"2024-02-13T00:50:35.564937+0000","last_peered":"2024-02-13T00:50:35.564937+0000","last_clean":"2024-02-13T00:50:35.564937+0000","last_became_active":"2024-02-13T00:46:01.441421+0000","last_became_peered":"2024-02-13T00:46:01.441421+0000","last_unstale":"2024-02-13T00:50:35.564937+0000","last_undegraded":"2024-02-13T00:50:35.564937+0000","last_fullsized":"2024-02-13T00:50:35.564937+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:55.050411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.050411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":253,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565971+0000","last_change":"2024-02-13T00:49:16.505082+0000","last_active":"2024-02-13T00:50:35.565971+0000","last_peered":"2024-02-13T00:50:35.565971+0000","last_clean":"2024-02-13T00:50:35.565971+0000","last_became_active":"2024-02-13T00:49:16.504828+0000","last_became_peered":"2024-02-13T00:49:16.504828+0000","last_unstale":"2024-02-13T00:50:35.565971+0000","last_undegraded":"2024-02-13T00:50:35.565971+0000","last_fullsized":"2024-02-13T00:50:35.565971+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:43.903553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:43.903553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":709,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.512369+0000","last_change":"2024-02-13T00:46:34.691109+0000","last_active":"2024-02-13T00:50:35.512369+0000","last_peered":"2024-02-13T00:50:35.512369+0000","last_clean":"2024-02-13T00:50:35.512369+0000","last_became_active":"2024-02-13T00:46:01.448163+0000","last_became_peered":"2024-02-13T00:46:01.448163+0000","last_unstale":"2024-02-13T00:50:35.512369+0000","last_undegraded":"2024-02-13T00:50:35.512369+0000","last_fullsized":"2024-02-13T00:50:35.512369+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:46:34.690980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.690980+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":321,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563509+0000","last_change":"2024-02-13T00:46:37.056299+0000","last_active":"2024-02-13T00:50:35.563509+0000","last_peered":"2024-02-13T00:50:35.563509+0000","last_clean":"2024-02-13T00:50:35.563509+0000","last_became_active":"2024-02-13T00:46:01.446104+0000","last_became_peered":"2024-02-13T00:46:01.446104+0000","last_unstale":"2024-02-13T00:50:35.563509+0000","last_undegraded":"2024-02-13T00:50:35.563509+0000","last_fullsized":"2024-02-13T00:50:35.563509+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:37.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":601,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510064+0000","last_change":"2024-02-13T00:46:01.431899+0000","last_active":"2024-02-13T00:50:35.510064+0000","last_peered":"2024-02-13T00:50:35.510064+0000","last_clean":"2024-02-13T00:50:35.510064+0000","last_became_active":"2024-02-13T00:46:01.430623+0000","last_became_peered":"2024-02-13T00:46:01.430623+0000","last_unstale":"2024-02-13T00:50:35.510064+0000","last_undegraded":"2024-02-13T00:50:35.510064+0000","last_fullsized":"2024-02-13T00:50:35.510064+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:46.556229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.556229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":611,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565123+0000","last_change":"2024-02-13T00:46:01.450548+0000","last_active":"2024-02-13T00:50:35.565123+0000","last_peered":"2024-02-13T00:50:35.565123+0000","last_clean":"2024-02-13T00:50:35.565123+0000","last_became_active":"2024-02-13T00:46:01.450176+0000","last_became_peered":"2024-02-13T00:46:01.450176+0000","last_unstale":"2024-02-13T00:50:35.565123+0000","last_undegraded":"2024-02-13T00:50:35.565123+0000","last_fullsized":"2024-02-13T00:50:35.565123+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":463,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.566010+0000","last_change":"2024-02-13T00:46:01.460470+0000","last_active":"2024-02-13T00:50:35.566010+0000","last_peered":"2024-02-13T00:50:35.566010+0000","last_clean":"2024-02-13T00:50:35.566010+0000","last_became_active":"2024-02-13T00:46:01.459859+0000","last_became_peered":"2024-02-13T00:46:01.459859+0000","last_unstale":"2024-02-13T00:50:35.566010+0000","last_undegraded":"2024-02-13T00:50:35.566010+0000","last_fullsized":"2024-02-13T00:50:35.566010+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:01.179031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:01.179031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":266,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510768+0000","last_change":"2024-02-13T00:47:33.932830+0000","last_active":"2024-02-13T00:50:35.510768+0000","last_peered":"2024-02-13T00:50:35.510768+0000","last_clean":"2024-02-13T00:50:35.510768+0000","last_became_active":"2024-02-13T00:46:01.455170+0000","last_became_peered":"2024-02-13T00:46:01.455170+0000","last_unstale":"2024-02-13T00:50:35.510768+0000","last_undegraded":"2024-02-13T00:50:35.510768+0000","last_fullsized":"2024-02-13T00:50:35.510768+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:33.932766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:33.932766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":367,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.566105+0000","last_change":"2024-02-13T00:46:57.381602+0000","last_active":"2024-02-13T00:50:35.566105+0000","last_peered":"2024-02-13T00:50:35.566105+0000","last_clean":"2024-02-13T00:50:35.566105+0000","last_became_active":"2024-02-13T00:45:15.446969+0000","last_became_peered":"2024-02-13T00:45:15.446969+0000","last_unstale":"2024-02-13T00:50:35.566105+0000","last_undegraded":"2024-02-13T00:50:35.566105+0000","last_fullsized":"2024-02-13T00:50:35.566105+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:46:57.381540+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:46:57.381540+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":566,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509499+0000","last_change":"2024-02-13T00:47:39.822745+0000","last_active":"2024-02-13T00:50:35.509499+0000","last_peered":"2024-02-13T00:50:35.509499+0000","last_clean":"2024-02-13T00:50:35.509499+0000","last_became_active":"2024-02-13T00:45:15.434498+0000","last_became_peered":"2024-02-13T00:45:15.434498+0000","last_unstale":"2024-02-13T00:50:35.509499+0000","last_undegraded":"2024-02-13T00:50:35.509499+0000","last_fullsized":"2024-02-13T00:50:35.509499+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:39.822619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:47:39.822619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":565,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565124+0000","last_change":"2024-02-13T00:47:04.493911+0000","last_active":"2024-02-13T00:50:35.565124+0000","last_peered":"2024-02-13T00:50:35.565124+0000","last_clean":"2024-02-13T00:50:35.565124+0000","last_became_active":"2024-02-13T00:46:01.447823+0000","last_became_peered":"2024-02-13T00:46:01.447823+0000","last_unstale":"2024-02-13T00:50:35.565124+0000","last_undegraded":"2024-02-13T00:50:35.565124+0000","last_fullsized":"2024-02-13T00:50:35.565124+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:04.493841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:04.493841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":438,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564649+0000","last_change":"2024-02-13T00:48:25.359625+0000","last_active":"2024-02-13T00:50:35.564649+0000","last_peered":"2024-02-13T00:50:35.564649+0000","last_clean":"2024-02-13T00:50:35.564649+0000","last_became_active":"2024-02-13T00:46:01.460001+0000","last_became_peered":"2024-02-13T00:46:01.460001+0000","last_unstale":"2024-02-13T00:50:35.564649+0000","last_undegraded":"2024-02-13T00:50:35.564649+0000","last_fullsized":"2024-02-13T00:50:35.564649+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:25.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:48:25.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":459,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565286+0000","last_change":"2024-02-13T00:46:01.450879+0000","last_active":"2024-02-13T00:50:35.565286+0000","last_peered":"2024-02-13T00:50:35.565286+0000","last_clean":"2024-02-13T00:50:35.565286+0000","last_became_active":"2024-02-13T00:46:01.450088+0000","last_became_peered":"2024-02-13T00:46:01.450088+0000","last_unstale":"2024-02-13T00:50:35.565286+0000","last_undegraded":"2024-02-13T00:50:35.565286+0000","last_fullsized":"2024-02-13T00:50:35.565286+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":547,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511370+0000","last_change":"2024-02-13T00:49:42.854702+0000","last_active":"2024-02-13T00:50:35.511370+0000","last_peered":"2024-02-13T00:50:35.511370+0000","last_clean":"2024-02-13T00:50:35.511370+0000","last_became_active":"2024-02-13T00:46:01.458408+0000","last_became_peered":"2024-02-13T00:46:01.458408+0000","last_unstale":"2024-02-13T00:50:35.511370+0000","last_undegraded":"2024-02-13T00:50:35.511370+0000","last_fullsized":"2024-02-13T00:50:35.511370+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:42.854585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:42.854585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":290,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564850+0000","last_change":"2024-02-13T00:49:19.638381+0000","last_active":"2024-02-13T00:50:35.564850+0000","last_peered":"2024-02-13T00:50:35.564850+0000","last_clean":"2024-02-13T00:50:35.564850+0000","last_became_active":"2024-02-13T00:49:19.638083+0000","last_became_peered":"2024-02-13T00:49:19.638083+0000","last_unstale":"2024-02-13T00:50:35.564850+0000","last_undegraded":"2024-02-13T00:50:35.564850+0000","last_fullsized":"2024-02-13T00:50:35.564850+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:53.013103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.013103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"467'23","reported_seq":655,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565813+0000","last_change":"2024-02-13T00:49:07.192972+0000","last_active":"2024-02-13T00:50:35.565813+0000","last_peered":"2024-02-13T00:50:35.565813+0000","last_clean":"2024-02-13T00:50:35.565813+0000","last_became_active":"2024-02-13T00:46:01.448219+0000","last_became_peered":"2024-02-13T00:46:01.448219+0000","last_unstale":"2024-02-13T00:50:35.565813+0000","last_undegraded":"2024-02-13T00:50:35.565813+0000","last_fullsized":"2024-02-13T00:50:35.565813+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'23","last_scrub_stamp":"2024-02-13T00:49:07.192902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:07.192902+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510039+0000","last_change":"2024-02-13T00:49:46.851558+0000","last_active":"2024-02-13T00:50:35.510039+0000","last_peered":"2024-02-13T00:50:35.510039+0000","last_clean":"2024-02-13T00:50:35.510039+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:50:35.510039+0000","last_undegraded":"2024-02-13T00:50:35.510039+0000","last_fullsized":"2024-02-13T00:50:35.510039+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.851486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.851486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":1405,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-13T00:50:44.024562+0000","last_change":"2024-02-13T00:45:54.075270+0000","last_active":"2024-02-13T00:50:44.024562+0000","last_peered":"2024-02-13T00:50:44.024562+0000","last_clean":"2024-02-13T00:50:44.024562+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:50:44.024562+0000","last_undegraded":"2024-02-13T00:50:44.024562+0000","last_fullsized":"2024-02-13T00:50:44.024562+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:45:54.075194+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":771,"num_read_kb":766,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":330,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565733+0000","last_change":"2024-02-13T00:46:01.460282+0000","last_active":"2024-02-13T00:50:35.565733+0000","last_peered":"2024-02-13T00:50:35.565733+0000","last_clean":"2024-02-13T00:50:35.565733+0000","last_became_active":"2024-02-13T00:46:01.459473+0000","last_became_peered":"2024-02-13T00:46:01.459473+0000","last_unstale":"2024-02-13T00:50:35.565733+0000","last_undegraded":"2024-02-13T00:50:35.565733+0000","last_fullsized":"2024-02-13T00:50:35.565733+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":384,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564832+0000","last_change":"2024-02-13T00:45:37.135398+0000","last_active":"2024-02-13T00:50:35.564832+0000","last_peered":"2024-02-13T00:50:35.564832+0000","last_clean":"2024-02-13T00:50:35.564832+0000","last_became_active":"2024-02-13T00:45:15.435588+0000","last_became_peered":"2024-02-13T00:45:15.435588+0000","last_unstale":"2024-02-13T00:50:35.564832+0000","last_undegraded":"2024-02-13T00:50:35.564832+0000","last_fullsized":"2024-02-13T00:50:35.564832+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.135332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.135332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":281,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564778+0000","last_change":"2024-02-13T00:45:31.183763+0000","last_active":"2024-02-13T00:50:35.564778+0000","last_peered":"2024-02-13T00:50:35.564778+0000","last_clean":"2024-02-13T00:50:35.564778+0000","last_became_active":"2024-02-13T00:45:06.421025+0000","last_became_peered":"2024-02-13T00:45:06.421025+0000","last_unstale":"2024-02-13T00:50:35.564778+0000","last_undegraded":"2024-02-13T00:50:35.564778+0000","last_fullsized":"2024-02-13T00:50:35.564778+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.183682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.183682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"467'65","reported_seq":332,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-13T00:50:40.606042+0000","last_change":"2024-02-13T00:49:47.928628+0000","last_active":"2024-02-13T00:50:40.606042+0000","last_peered":"2024-02-13T00:50:40.606042+0000","last_clean":"2024-02-13T00:50:40.606042+0000","last_became_active":"2024-02-13T00:46:01.449059+0000","last_became_peered":"2024-02-13T00:46:01.449059+0000","last_unstale":"2024-02-13T00:50:40.606042+0000","last_undegraded":"2024-02-13T00:50:40.606042+0000","last_fullsized":"2024-02-13T00:50:40.606042+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'65","last_scrub_stamp":"2024-02-13T00:49:47.928548+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.928548+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":2148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":534,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:37.777987+0000","last_change":"2024-02-13T00:50:37.777963+0000","last_active":"2024-02-13T00:50:37.777987+0000","last_peered":"2024-02-13T00:50:37.777987+0000","last_clean":"2024-02-13T00:50:37.777987+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:50:37.777987+0000","last_undegraded":"2024-02-13T00:50:37.777987+0000","last_fullsized":"2024-02-13T00:50:37.777987+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:37.777891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:37.777891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":146,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511794+0000","last_change":"2024-02-13T00:48:06.705030+0000","last_active":"2024-02-13T00:50:35.511794+0000","last_peered":"2024-02-13T00:50:35.511794+0000","last_clean":"2024-02-13T00:50:35.511794+0000","last_became_active":"2024-02-13T00:48:06.704651+0000","last_became_peered":"2024-02-13T00:48:06.704651+0000","last_unstale":"2024-02-13T00:50:35.511794+0000","last_undegraded":"2024-02-13T00:50:35.511794+0000","last_fullsized":"2024-02-13T00:50:35.511794+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":317,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567321+0000","last_change":"2024-02-13T00:48:17.340891+0000","last_active":"2024-02-13T00:50:35.567321+0000","last_peered":"2024-02-13T00:50:35.567321+0000","last_clean":"2024-02-13T00:50:35.567321+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:50:35.567321+0000","last_undegraded":"2024-02-13T00:50:35.567321+0000","last_fullsized":"2024-02-13T00:50:35.567321+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:17.340797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:17.340797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":798,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-13T00:50:41.366504+0000","last_change":"2024-02-13T00:48:32.669477+0000","last_active":"2024-02-13T00:50:41.366504+0000","last_peered":"2024-02-13T00:50:41.366504+0000","last_clean":"2024-02-13T00:50:41.366504+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:50:41.366504+0000","last_undegraded":"2024-02-13T00:50:41.366504+0000","last_fullsized":"2024-02-13T00:50:41.366504+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14","last_scrub_stamp":"2024-02-13T00:48:32.669406+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.669406+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":352,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563750+0000","last_change":"2024-02-13T00:48:20.928236+0000","last_active":"2024-02-13T00:50:35.563750+0000","last_peered":"2024-02-13T00:50:35.563750+0000","last_clean":"2024-02-13T00:50:35.563750+0000","last_became_active":"2024-02-13T00:46:01.442003+0000","last_became_peered":"2024-02-13T00:46:01.442003+0000","last_unstale":"2024-02-13T00:50:35.563750+0000","last_undegraded":"2024-02-13T00:50:35.563750+0000","last_fullsized":"2024-02-13T00:50:35.563750+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:48:20.928158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:20.928158+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":600,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564899+0000","last_change":"2024-02-13T00:49:33.040619+0000","last_active":"2024-02-13T00:50:35.564899+0000","last_peered":"2024-02-13T00:50:35.564899+0000","last_clean":"2024-02-13T00:50:35.564899+0000","last_became_active":"2024-02-13T00:45:15.422546+0000","last_became_peered":"2024-02-13T00:45:15.422546+0000","last_unstale":"2024-02-13T00:50:35.564899+0000","last_undegraded":"2024-02-13T00:50:35.564899+0000","last_fullsized":"2024-02-13T00:50:35.564899+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:33.040516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:49:33.040516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":613,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:38.051554+0000","last_change":"2024-02-13T00:50:38.051528+0000","last_active":"2024-02-13T00:50:38.051554+0000","last_peered":"2024-02-13T00:50:38.051554+0000","last_clean":"2024-02-13T00:50:38.051554+0000","last_became_active":"2024-02-13T00:46:01.435178+0000","last_became_peered":"2024-02-13T00:46:01.435178+0000","last_unstale":"2024-02-13T00:50:38.051554+0000","last_undegraded":"2024-02-13T00:50:38.051554+0000","last_fullsized":"2024-02-13T00:50:38.051554+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:38.051468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:38.051468+0000","last_clean_scrub_stamp":"2024-02-13T00:50:38.051468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":544,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563271+0000","last_change":"2024-02-13T00:45:15.445626+0000","last_active":"2024-02-13T00:50:35.563271+0000","last_peered":"2024-02-13T00:50:35.563271+0000","last_clean":"2024-02-13T00:50:35.563271+0000","last_became_active":"2024-02-13T00:45:15.445117+0000","last_became_peered":"2024-02-13T00:45:15.445117+0000","last_unstale":"2024-02-13T00:50:35.563271+0000","last_undegraded":"2024-02-13T00:50:35.563271+0000","last_fullsized":"2024-02-13T00:50:35.563271+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:44:34.171185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":326,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509752+0000","last_change":"2024-02-13T00:49:12.252240+0000","last_active":"2024-02-13T00:50:35.509752+0000","last_peered":"2024-02-13T00:50:35.509752+0000","last_clean":"2024-02-13T00:50:35.509752+0000","last_became_active":"2024-02-13T00:49:12.251980+0000","last_became_peered":"2024-02-13T00:49:12.251980+0000","last_unstale":"2024-02-13T00:50:35.509752+0000","last_undegraded":"2024-02-13T00:50:35.509752+0000","last_fullsized":"2024-02-13T00:50:35.509752+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.592320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.592320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":670,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563237+0000","last_change":"2024-02-13T00:49:31.072963+0000","last_active":"2024-02-13T00:50:35.563237+0000","last_peered":"2024-02-13T00:50:35.563237+0000","last_clean":"2024-02-13T00:50:35.563237+0000","last_became_active":"2024-02-13T00:44:39.501886+0000","last_became_peered":"2024-02-13T00:44:39.501886+0000","last_unstale":"2024-02-13T00:50:35.563237+0000","last_undegraded":"2024-02-13T00:50:35.563237+0000","last_fullsized":"2024-02-13T00:50:35.563237+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'30","last_scrub_stamp":"2024-02-13T00:49:31.072862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:31.072862+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":587,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511258+0000","last_change":"2024-02-13T00:48:58.861408+0000","last_active":"2024-02-13T00:50:35.511258+0000","last_peered":"2024-02-13T00:50:35.511258+0000","last_clean":"2024-02-13T00:50:35.511258+0000","last_became_active":"2024-02-13T00:46:01.457926+0000","last_became_peered":"2024-02-13T00:46:01.457926+0000","last_unstale":"2024-02-13T00:50:35.511258+0000","last_undegraded":"2024-02-13T00:50:35.511258+0000","last_fullsized":"2024-02-13T00:50:35.511258+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:48:58.861325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:58.861325+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":395,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564212+0000","last_change":"2024-02-13T00:47:06.895693+0000","last_active":"2024-02-13T00:50:35.564212+0000","last_peered":"2024-02-13T00:50:35.564212+0000","last_clean":"2024-02-13T00:50:35.564212+0000","last_became_active":"2024-02-13T00:45:15.435404+0000","last_became_peered":"2024-02-13T00:45:15.435404+0000","last_unstale":"2024-02-13T00:50:35.564212+0000","last_undegraded":"2024-02-13T00:50:35.564212+0000","last_fullsized":"2024-02-13T00:50:35.564212+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:06.895612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:47:06.895612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":547,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.512417+0000","last_change":"2024-02-13T00:50:21.880120+0000","last_active":"2024-02-13T00:50:35.512417+0000","last_peered":"2024-02-13T00:50:35.512417+0000","last_clean":"2024-02-13T00:50:35.512417+0000","last_became_active":"2024-02-13T00:46:01.449234+0000","last_became_peered":"2024-02-13T00:46:01.449234+0000","last_unstale":"2024-02-13T00:50:35.512417+0000","last_undegraded":"2024-02-13T00:50:35.512417+0000","last_fullsized":"2024-02-13T00:50:35.512417+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.880001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.880001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"467'27","reported_seq":735,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.512376+0000","last_change":"2024-02-13T00:49:35.834793+0000","last_active":"2024-02-13T00:50:35.512376+0000","last_peered":"2024-02-13T00:50:35.512376+0000","last_clean":"2024-02-13T00:50:35.512376+0000","last_became_active":"2024-02-13T00:46:01.456876+0000","last_became_peered":"2024-02-13T00:46:01.456876+0000","last_unstale":"2024-02-13T00:50:35.512376+0000","last_undegraded":"2024-02-13T00:50:35.512376+0000","last_fullsized":"2024-02-13T00:50:35.512376+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:35.834669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.834669+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":302,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563481+0000","last_change":"2024-02-13T00:46:30.095387+0000","last_active":"2024-02-13T00:50:35.563481+0000","last_peered":"2024-02-13T00:50:35.563481+0000","last_clean":"2024-02-13T00:50:35.563481+0000","last_became_active":"2024-02-13T00:44:50.441300+0000","last_became_peered":"2024-02-13T00:44:50.441300+0000","last_unstale":"2024-02-13T00:50:35.563481+0000","last_undegraded":"2024-02-13T00:50:35.563481+0000","last_fullsized":"2024-02-13T00:50:35.563481+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.095307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":315,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511457+0000","last_change":"2024-02-13T00:46:01.454597+0000","last_active":"2024-02-13T00:50:35.511457+0000","last_peered":"2024-02-13T00:50:35.511457+0000","last_clean":"2024-02-13T00:50:35.511457+0000","last_became_active":"2024-02-13T00:46:01.454101+0000","last_became_peered":"2024-02-13T00:46:01.454101+0000","last_unstale":"2024-02-13T00:50:35.511457+0000","last_undegraded":"2024-02-13T00:50:35.511457+0000","last_fullsized":"2024-02-13T00:50:35.511457+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.651091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.651091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":312,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.566151+0000","last_change":"2024-02-13T00:44:50.441196+0000","last_active":"2024-02-13T00:50:35.566151+0000","last_peered":"2024-02-13T00:50:35.566151+0000","last_clean":"2024-02-13T00:50:35.566151+0000","last_became_active":"2024-02-13T00:44:50.438274+0000","last_became_peered":"2024-02-13T00:44:50.438274+0000","last_unstale":"2024-02-13T00:50:35.566151+0000","last_undegraded":"2024-02-13T00:50:35.566151+0000","last_fullsized":"2024-02-13T00:50:35.566151+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:34.510219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:43:34.510219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":471,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565972+0000","last_change":"2024-02-13T00:46:34.141015+0000","last_active":"2024-02-13T00:50:35.565972+0000","last_peered":"2024-02-13T00:50:35.565972+0000","last_clean":"2024-02-13T00:50:35.565972+0000","last_became_active":"2024-02-13T00:45:15.436020+0000","last_became_peered":"2024-02-13T00:45:15.436020+0000","last_unstale":"2024-02-13T00:50:35.565972+0000","last_undegraded":"2024-02-13T00:50:35.565972+0000","last_fullsized":"2024-02-13T00:50:35.565972+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:46:34.140928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.140928+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":308,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567297+0000","last_change":"2024-02-13T00:45:00.311635+0000","last_active":"2024-02-13T00:50:35.567297+0000","last_peered":"2024-02-13T00:50:35.567297+0000","last_clean":"2024-02-13T00:50:35.567297+0000","last_became_active":"2024-02-13T00:44:39.512402+0000","last_became_peered":"2024-02-13T00:44:39.512402+0000","last_unstale":"2024-02-13T00:50:35.567297+0000","last_undegraded":"2024-02-13T00:50:35.567297+0000","last_fullsized":"2024-02-13T00:50:35.567297+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:00.311558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:45:00.311558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":326,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:40.384536+0000","last_change":"2024-02-13T00:50:40.384509+0000","last_active":"2024-02-13T00:50:40.384536+0000","last_peered":"2024-02-13T00:50:40.384536+0000","last_clean":"2024-02-13T00:50:40.384536+0000","last_became_active":"2024-02-13T00:46:01.450312+0000","last_became_peered":"2024-02-13T00:46:01.450312+0000","last_unstale":"2024-02-13T00:50:40.384536+0000","last_undegraded":"2024-02-13T00:50:40.384536+0000","last_fullsized":"2024-02-13T00:50:40.384536+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:40.384428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:40.384428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":392,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511688+0000","last_change":"2024-02-13T00:46:01.454858+0000","last_active":"2024-02-13T00:50:35.511688+0000","last_peered":"2024-02-13T00:50:35.511688+0000","last_clean":"2024-02-13T00:50:35.511688+0000","last_became_active":"2024-02-13T00:46:01.454284+0000","last_became_peered":"2024-02-13T00:46:01.454284+0000","last_unstale":"2024-02-13T00:50:35.511688+0000","last_undegraded":"2024-02-13T00:50:35.511688+0000","last_fullsized":"2024-02-13T00:50:35.511688+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:34.523594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.523594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":554,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565030+0000","last_change":"2024-02-13T00:49:06.154997+0000","last_active":"2024-02-13T00:50:35.565030+0000","last_peered":"2024-02-13T00:50:35.565030+0000","last_clean":"2024-02-13T00:50:35.565030+0000","last_became_active":"2024-02-13T00:46:01.450113+0000","last_became_peered":"2024-02-13T00:46:01.450113+0000","last_unstale":"2024-02-13T00:50:35.565030+0000","last_undegraded":"2024-02-13T00:50:35.565030+0000","last_fullsized":"2024-02-13T00:50:35.565030+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'39","last_scrub_stamp":"2024-02-13T00:49:06.154931+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:49:06.154931+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":567,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510390+0000","last_change":"2024-02-13T00:49:29.916234+0000","last_active":"2024-02-13T00:50:35.510390+0000","last_peered":"2024-02-13T00:50:35.510390+0000","last_clean":"2024-02-13T00:50:35.510390+0000","last_became_active":"2024-02-13T00:46:01.454310+0000","last_became_peered":"2024-02-13T00:46:01.454310+0000","last_unstale":"2024-02-13T00:50:35.510390+0000","last_undegraded":"2024-02-13T00:50:35.510390+0000","last_fullsized":"2024-02-13T00:50:35.510390+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.916152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"404'9","reported_seq":315,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-13T00:50:40.607252+0000","last_change":"2024-02-13T00:47:37.013003+0000","last_active":"2024-02-13T00:50:40.607252+0000","last_peered":"2024-02-13T00:50:40.607252+0000","last_clean":"2024-02-13T00:50:40.607252+0000","last_became_active":"2024-02-13T00:46:01.448144+0000","last_became_peered":"2024-02-13T00:46:01.448144+0000","last_unstale":"2024-02-13T00:50:40.607252+0000","last_undegraded":"2024-02-13T00:50:40.607252+0000","last_fullsized":"2024-02-13T00:50:40.607252+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9","last_scrub_stamp":"2024-02-13T00:47:37.012934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:37.012934+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":604,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.512323+0000","last_change":"2024-02-13T00:48:26.647460+0000","last_active":"2024-02-13T00:50:35.512323+0000","last_peered":"2024-02-13T00:50:35.512323+0000","last_clean":"2024-02-13T00:50:35.512323+0000","last_became_active":"2024-02-13T00:46:01.447236+0000","last_became_peered":"2024-02-13T00:46:01.447236+0000","last_unstale":"2024-02-13T00:50:35.512323+0000","last_undegraded":"2024-02-13T00:50:35.512323+0000","last_fullsized":"2024-02-13T00:50:35.512323+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:26.647367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:26.647367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":275,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511182+0000","last_change":"2024-02-13T00:50:14.849751+0000","last_active":"2024-02-13T00:50:35.511182+0000","last_peered":"2024-02-13T00:50:35.511182+0000","last_clean":"2024-02-13T00:50:35.511182+0000","last_became_active":"2024-02-13T00:46:01.455989+0000","last_became_peered":"2024-02-13T00:46:01.455989+0000","last_unstale":"2024-02-13T00:50:35.511182+0000","last_undegraded":"2024-02-13T00:50:35.511182+0000","last_fullsized":"2024-02-13T00:50:35.511182+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.849687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.849687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"467'22","reported_seq":590,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565108+0000","last_change":"2024-02-13T00:47:35.369710+0000","last_active":"2024-02-13T00:50:35.565108+0000","last_peered":"2024-02-13T00:50:35.565108+0000","last_clean":"2024-02-13T00:50:35.565108+0000","last_became_active":"2024-02-13T00:46:01.460321+0000","last_became_peered":"2024-02-13T00:46:01.460321+0000","last_unstale":"2024-02-13T00:50:35.565108+0000","last_undegraded":"2024-02-13T00:50:35.565108+0000","last_fullsized":"2024-02-13T00:50:35.565108+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:47:35.369616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:35.369616+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":619,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510967+0000","last_change":"2024-02-13T00:49:05.887739+0000","last_active":"2024-02-13T00:50:35.510967+0000","last_peered":"2024-02-13T00:50:35.510967+0000","last_clean":"2024-02-13T00:50:35.510967+0000","last_became_active":"2024-02-13T00:46:01.431926+0000","last_became_peered":"2024-02-13T00:46:01.431926+0000","last_unstale":"2024-02-13T00:50:35.510967+0000","last_undegraded":"2024-02-13T00:50:35.510967+0000","last_fullsized":"2024-02-13T00:50:35.510967+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:05.887665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:05.887665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"363'14","reported_seq":458,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-13T00:50:40.606023+0000","last_change":"2024-02-13T00:47:07.942748+0000","last_active":"2024-02-13T00:50:40.606023+0000","last_peered":"2024-02-13T00:50:40.606023+0000","last_clean":"2024-02-13T00:50:40.606023+0000","last_became_active":"2024-02-13T00:45:15.423066+0000","last_became_peered":"2024-02-13T00:45:15.423066+0000","last_unstale":"2024-02-13T00:50:40.606023+0000","last_undegraded":"2024-02-13T00:50:40.606023+0000","last_fullsized":"2024-02-13T00:50:40.606023+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:47:07.942668+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:47:07.942668+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":238,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511155+0000","last_change":"2024-02-13T00:47:02.789298+0000","last_active":"2024-02-13T00:50:35.511155+0000","last_peered":"2024-02-13T00:50:35.511155+0000","last_clean":"2024-02-13T00:50:35.511155+0000","last_became_active":"2024-02-13T00:46:01.446960+0000","last_became_peered":"2024-02-13T00:46:01.446960+0000","last_unstale":"2024-02-13T00:50:35.511155+0000","last_undegraded":"2024-02-13T00:50:35.511155+0000","last_fullsized":"2024-02-13T00:50:35.511155+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:02.789189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.789189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":477,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567338+0000","last_change":"2024-02-13T00:46:01.450308+0000","last_active":"2024-02-13T00:50:35.567338+0000","last_peered":"2024-02-13T00:50:35.567338+0000","last_clean":"2024-02-13T00:50:35.567338+0000","last_became_active":"2024-02-13T00:46:01.449694+0000","last_became_peered":"2024-02-13T00:46:01.449694+0000","last_unstale":"2024-02-13T00:50:35.567338+0000","last_undegraded":"2024-02-13T00:50:35.567338+0000","last_fullsized":"2024-02-13T00:50:35.567338+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:27.073034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:27.073034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":675,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563697+0000","last_change":"2024-02-13T00:49:38.007273+0000","last_active":"2024-02-13T00:50:35.563697+0000","last_peered":"2024-02-13T00:50:35.563697+0000","last_clean":"2024-02-13T00:50:35.563697+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:50:35.563697+0000","last_undegraded":"2024-02-13T00:50:35.563697+0000","last_fullsized":"2024-02-13T00:50:35.563697+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'19","last_scrub_stamp":"2024-02-13T00:49:38.007204+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:49:38.007204+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":321,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565857+0000","last_change":"2024-02-13T00:46:01.463310+0000","last_active":"2024-02-13T00:50:35.565857+0000","last_peered":"2024-02-13T00:50:35.565857+0000","last_clean":"2024-02-13T00:50:35.565857+0000","last_became_active":"2024-02-13T00:46:01.462898+0000","last_became_peered":"2024-02-13T00:46:01.462898+0000","last_unstale":"2024-02-13T00:50:35.565857+0000","last_undegraded":"2024-02-13T00:50:35.565857+0000","last_fullsized":"2024-02-13T00:50:35.565857+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.095888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":383,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510556+0000","last_change":"2024-02-13T00:46:32.638418+0000","last_active":"2024-02-13T00:50:35.510556+0000","last_peered":"2024-02-13T00:50:35.510556+0000","last_clean":"2024-02-13T00:50:35.510556+0000","last_became_active":"2024-02-13T00:46:01.443279+0000","last_became_peered":"2024-02-13T00:46:01.443279+0000","last_unstale":"2024-02-13T00:50:35.510556+0000","last_undegraded":"2024-02-13T00:50:35.510556+0000","last_fullsized":"2024-02-13T00:50:35.510556+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.638349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":586,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509418+0000","last_change":"2024-02-13T00:48:27.659266+0000","last_active":"2024-02-13T00:50:35.509418+0000","last_peered":"2024-02-13T00:50:35.509418+0000","last_clean":"2024-02-13T00:50:35.509418+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:50:35.509418+0000","last_undegraded":"2024-02-13T00:50:35.509418+0000","last_fullsized":"2024-02-13T00:50:35.509418+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:27.659197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:48:27.659197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":317,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565812+0000","last_change":"2024-02-13T00:45:40.233264+0000","last_active":"2024-02-13T00:50:35.565812+0000","last_peered":"2024-02-13T00:50:35.565812+0000","last_clean":"2024-02-13T00:50:35.565812+0000","last_became_active":"2024-02-13T00:45:15.447051+0000","last_became_peered":"2024-02-13T00:45:15.447051+0000","last_unstale":"2024-02-13T00:50:35.565812+0000","last_undegraded":"2024-02-13T00:50:35.565812+0000","last_fullsized":"2024-02-13T00:50:35.565812+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:40.233185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:40.233185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":353,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565725+0000","last_change":"2024-02-13T00:48:34.458358+0000","last_active":"2024-02-13T00:50:35.565725+0000","last_peered":"2024-02-13T00:50:35.565725+0000","last_clean":"2024-02-13T00:50:35.565725+0000","last_became_active":"2024-02-13T00:44:50.435203+0000","last_became_peered":"2024-02-13T00:44:50.435203+0000","last_unstale":"2024-02-13T00:50:35.565725+0000","last_undegraded":"2024-02-13T00:50:35.565725+0000","last_fullsized":"2024-02-13T00:50:35.565725+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:48:34.458264+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:48:34.458264+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":595,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564791+0000","last_change":"2024-02-13T00:46:01.457220+0000","last_active":"2024-02-13T00:50:35.564791+0000","last_peered":"2024-02-13T00:50:35.564791+0000","last_clean":"2024-02-13T00:50:35.564791+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:50:35.564791+0000","last_undegraded":"2024-02-13T00:50:35.564791+0000","last_fullsized":"2024-02-13T00:50:35.564791+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.097481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":614,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:36.787714+0000","last_change":"2024-02-13T00:50:36.787685+0000","last_active":"2024-02-13T00:50:36.787714+0000","last_peered":"2024-02-13T00:50:36.787714+0000","last_clean":"2024-02-13T00:50:36.787714+0000","last_became_active":"2024-02-13T00:46:01.452730+0000","last_became_peered":"2024-02-13T00:46:01.452730+0000","last_unstale":"2024-02-13T00:50:36.787714+0000","last_undegraded":"2024-02-13T00:50:36.787714+0000","last_fullsized":"2024-02-13T00:50:36.787714+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:36.787604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:50:36.787604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":392,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510126+0000","last_change":"2024-02-13T00:48:29.681480+0000","last_active":"2024-02-13T00:50:35.510126+0000","last_peered":"2024-02-13T00:50:35.510126+0000","last_clean":"2024-02-13T00:50:35.510126+0000","last_became_active":"2024-02-13T00:46:01.454287+0000","last_became_peered":"2024-02-13T00:46:01.454287+0000","last_unstale":"2024-02-13T00:50:35.510126+0000","last_undegraded":"2024-02-13T00:50:35.510126+0000","last_fullsized":"2024-02-13T00:50:35.510126+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:29.681414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:29.681414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":326,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564035+0000","last_change":"2024-02-13T00:49:02.260116+0000","last_active":"2024-02-13T00:50:35.564035+0000","last_peered":"2024-02-13T00:50:35.564035+0000","last_clean":"2024-02-13T00:50:35.564035+0000","last_became_active":"2024-02-13T00:46:01.448671+0000","last_became_peered":"2024-02-13T00:46:01.448671+0000","last_unstale":"2024-02-13T00:50:35.564035+0000","last_undegraded":"2024-02-13T00:50:35.564035+0000","last_fullsized":"2024-02-13T00:50:35.564035+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:02.260045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:02.260045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":427,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563690+0000","last_change":"2024-02-13T00:50:32.774892+0000","last_active":"2024-02-13T00:50:35.563690+0000","last_peered":"2024-02-13T00:50:35.563690+0000","last_clean":"2024-02-13T00:50:35.563690+0000","last_became_active":"2024-02-13T00:44:39.501040+0000","last_became_peered":"2024-02-13T00:44:39.501040+0000","last_unstale":"2024-02-13T00:50:35.563690+0000","last_undegraded":"2024-02-13T00:50:35.563690+0000","last_fullsized":"2024-02-13T00:50:35.563690+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'8","last_scrub_stamp":"2024-02-13T00:50:32.774820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:50:32.774820+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":552,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564563+0000","last_change":"2024-02-13T00:48:32.441279+0000","last_active":"2024-02-13T00:50:35.564563+0000","last_peered":"2024-02-13T00:50:35.564563+0000","last_clean":"2024-02-13T00:50:35.564563+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:50:35.564563+0000","last_undegraded":"2024-02-13T00:50:35.564563+0000","last_fullsized":"2024-02-13T00:50:35.564563+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.441170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.441170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":317,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567105+0000","last_change":"2024-02-13T00:48:28.353456+0000","last_active":"2024-02-13T00:50:35.567105+0000","last_peered":"2024-02-13T00:50:35.567105+0000","last_clean":"2024-02-13T00:50:35.567105+0000","last_became_active":"2024-02-13T00:45:15.435820+0000","last_became_peered":"2024-02-13T00:45:15.435820+0000","last_unstale":"2024-02-13T00:50:35.567105+0000","last_undegraded":"2024-02-13T00:50:35.567105+0000","last_fullsized":"2024-02-13T00:50:35.567105+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:28.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:28.353347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":309,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563318+0000","last_change":"2024-02-13T00:49:29.052728+0000","last_active":"2024-02-13T00:50:35.563318+0000","last_peered":"2024-02-13T00:50:35.563318+0000","last_clean":"2024-02-13T00:50:35.563318+0000","last_became_active":"2024-02-13T00:45:15.444818+0000","last_became_peered":"2024-02-13T00:45:15.444818+0000","last_unstale":"2024-02-13T00:50:35.563318+0000","last_undegraded":"2024-02-13T00:50:35.563318+0000","last_fullsized":"2024-02-13T00:50:35.563318+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.052640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.052640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":468,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567167+0000","last_change":"2024-02-13T00:46:01.439915+0000","last_active":"2024-02-13T00:50:35.567167+0000","last_peered":"2024-02-13T00:50:35.567167+0000","last_clean":"2024-02-13T00:50:35.567167+0000","last_became_active":"2024-02-13T00:46:01.438745+0000","last_became_peered":"2024-02-13T00:46:01.438745+0000","last_unstale":"2024-02-13T00:50:35.567167+0000","last_undegraded":"2024-02-13T00:50:35.567167+0000","last_fullsized":"2024-02-13T00:50:35.567167+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.104906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.104906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"467'25","reported_seq":380,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567204+0000","last_change":"2024-02-13T00:47:36.368967+0000","last_active":"2024-02-13T00:50:35.567204+0000","last_peered":"2024-02-13T00:50:35.567204+0000","last_clean":"2024-02-13T00:50:35.567204+0000","last_became_active":"2024-02-13T00:46:01.450079+0000","last_became_peered":"2024-02-13T00:46:01.450079+0000","last_unstale":"2024-02-13T00:50:35.567204+0000","last_undegraded":"2024-02-13T00:50:35.567204+0000","last_fullsized":"2024-02-13T00:50:35.567204+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:47:36.368863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.368863+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":326,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.566094+0000","last_change":"2024-02-13T00:46:59.388809+0000","last_active":"2024-02-13T00:50:35.566094+0000","last_peered":"2024-02-13T00:50:35.566094+0000","last_clean":"2024-02-13T00:50:35.566094+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:50:35.566094+0000","last_undegraded":"2024-02-13T00:50:35.566094+0000","last_fullsized":"2024-02-13T00:50:35.566094+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:59.388745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:59.388745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":312,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563802+0000","last_change":"2024-02-13T00:45:15.439752+0000","last_active":"2024-02-13T00:50:35.563802+0000","last_peered":"2024-02-13T00:50:35.563802+0000","last_clean":"2024-02-13T00:50:35.563802+0000","last_became_active":"2024-02-13T00:45:15.437943+0000","last_became_peered":"2024-02-13T00:45:15.437943+0000","last_unstale":"2024-02-13T00:50:35.563802+0000","last_undegraded":"2024-02-13T00:50:35.563802+0000","last_fullsized":"2024-02-13T00:50:35.563802+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:46.284740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:43:46.284740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":519,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510470+0000","last_change":"2024-02-13T00:44:58.969806+0000","last_active":"2024-02-13T00:50:35.510470+0000","last_peered":"2024-02-13T00:50:35.510470+0000","last_clean":"2024-02-13T00:50:35.510470+0000","last_became_active":"2024-02-13T00:44:50.439512+0000","last_became_peered":"2024-02-13T00:44:50.439512+0000","last_unstale":"2024-02-13T00:50:35.510470+0000","last_undegraded":"2024-02-13T00:50:35.510470+0000","last_fullsized":"2024-02-13T00:50:35.510470+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:58.969692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:44:58.969692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":440,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564690+0000","last_change":"2024-02-13T00:49:35.101413+0000","last_active":"2024-02-13T00:50:35.564690+0000","last_peered":"2024-02-13T00:50:35.564690+0000","last_clean":"2024-02-13T00:50:35.564690+0000","last_became_active":"2024-02-13T00:45:15.430749+0000","last_became_peered":"2024-02-13T00:45:15.430749+0000","last_unstale":"2024-02-13T00:50:35.564690+0000","last_undegraded":"2024-02-13T00:50:35.564690+0000","last_fullsized":"2024-02-13T00:50:35.564690+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:35.101298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.101298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":708,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.512511+0000","last_change":"2024-02-13T00:48:53.841888+0000","last_active":"2024-02-13T00:50:35.512511+0000","last_peered":"2024-02-13T00:50:35.512511+0000","last_clean":"2024-02-13T00:50:35.512511+0000","last_became_active":"2024-02-13T00:46:01.443721+0000","last_became_peered":"2024-02-13T00:46:01.443721+0000","last_unstale":"2024-02-13T00:50:35.512511+0000","last_undegraded":"2024-02-13T00:50:35.512511+0000","last_fullsized":"2024-02-13T00:50:35.512511+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'32","last_scrub_stamp":"2024-02-13T00:48:53.841797+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.841797+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":554,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563502+0000","last_change":"2024-02-13T00:46:32.298033+0000","last_active":"2024-02-13T00:50:35.563502+0000","last_peered":"2024-02-13T00:50:35.563502+0000","last_clean":"2024-02-13T00:50:35.563502+0000","last_became_active":"2024-02-13T00:46:01.442231+0000","last_became_peered":"2024-02-13T00:46:01.442231+0000","last_unstale":"2024-02-13T00:50:35.563502+0000","last_undegraded":"2024-02-13T00:50:35.563502+0000","last_fullsized":"2024-02-13T00:50:35.563502+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.297918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.297918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":418,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565104+0000","last_change":"2024-02-13T00:48:33.460377+0000","last_active":"2024-02-13T00:50:35.565104+0000","last_peered":"2024-02-13T00:50:35.565104+0000","last_clean":"2024-02-13T00:50:35.565104+0000","last_became_active":"2024-02-13T00:46:01.439984+0000","last_became_peered":"2024-02-13T00:46:01.439984+0000","last_unstale":"2024-02-13T00:50:35.565104+0000","last_undegraded":"2024-02-13T00:50:35.565104+0000","last_fullsized":"2024-02-13T00:50:35.565104+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:48:33.460247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:33.460247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":553,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565900+0000","last_change":"2024-02-13T00:45:38.084426+0000","last_active":"2024-02-13T00:50:35.565900+0000","last_peered":"2024-02-13T00:50:35.565900+0000","last_clean":"2024-02-13T00:50:35.565900+0000","last_became_active":"2024-02-13T00:45:15.435550+0000","last_became_peered":"2024-02-13T00:45:15.435550+0000","last_unstale":"2024-02-13T00:50:35.565900+0000","last_undegraded":"2024-02-13T00:50:35.565900+0000","last_fullsized":"2024-02-13T00:50:35.565900+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.084358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.084358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":357,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.566318+0000","last_change":"2024-02-13T00:48:55.424021+0000","last_active":"2024-02-13T00:50:35.566318+0000","last_peered":"2024-02-13T00:50:35.566318+0000","last_clean":"2024-02-13T00:50:35.566318+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:50:35.566318+0000","last_undegraded":"2024-02-13T00:50:35.566318+0000","last_fullsized":"2024-02-13T00:50:35.566318+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:55.423952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.423952+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":326,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567253+0000","last_change":"2024-02-13T00:46:56.339969+0000","last_active":"2024-02-13T00:50:35.567253+0000","last_peered":"2024-02-13T00:50:35.567253+0000","last_clean":"2024-02-13T00:50:35.567253+0000","last_became_active":"2024-02-13T00:45:15.436239+0000","last_became_peered":"2024-02-13T00:45:15.436239+0000","last_unstale":"2024-02-13T00:50:35.567253+0000","last_undegraded":"2024-02-13T00:50:35.567253+0000","last_fullsized":"2024-02-13T00:50:35.567253+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:56.339885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.339885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":594,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564696+0000","last_change":"2024-02-13T00:49:46.928723+0000","last_active":"2024-02-13T00:50:35.564696+0000","last_peered":"2024-02-13T00:50:35.564696+0000","last_clean":"2024-02-13T00:50:35.564696+0000","last_became_active":"2024-02-13T00:46:01.453269+0000","last_became_peered":"2024-02-13T00:46:01.453269+0000","last_unstale":"2024-02-13T00:50:35.564696+0000","last_undegraded":"2024-02-13T00:50:35.564696+0000","last_fullsized":"2024-02-13T00:50:35.564696+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.928651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.928651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":317,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.566051+0000","last_change":"2024-02-13T00:45:37.280947+0000","last_active":"2024-02-13T00:50:35.566051+0000","last_peered":"2024-02-13T00:50:35.566051+0000","last_clean":"2024-02-13T00:50:35.566051+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:50:35.566051+0000","last_undegraded":"2024-02-13T00:50:35.566051+0000","last_fullsized":"2024-02-13T00:50:35.566051+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.280865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.280865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":373,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565000+0000","last_change":"2024-02-13T00:49:30.086932+0000","last_active":"2024-02-13T00:50:35.565000+0000","last_peered":"2024-02-13T00:50:35.565000+0000","last_clean":"2024-02-13T00:50:35.565000+0000","last_became_active":"2024-02-13T00:44:50.442894+0000","last_became_peered":"2024-02-13T00:44:50.442894+0000","last_unstale":"2024-02-13T00:50:35.565000+0000","last_undegraded":"2024-02-13T00:50:35.565000+0000","last_fullsized":"2024-02-13T00:50:35.565000+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:30.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:30.086856+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":317,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.567137+0000","last_change":"2024-02-13T00:49:04.282594+0000","last_active":"2024-02-13T00:50:35.567137+0000","last_peered":"2024-02-13T00:50:35.567137+0000","last_clean":"2024-02-13T00:50:35.567137+0000","last_became_active":"2024-02-13T00:45:15.445403+0000","last_became_peered":"2024-02-13T00:45:15.445403+0000","last_unstale":"2024-02-13T00:50:35.567137+0000","last_undegraded":"2024-02-13T00:50:35.567137+0000","last_fullsized":"2024-02-13T00:50:35.567137+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:04.282529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:04.282529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":317,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564952+0000","last_change":"2024-02-13T00:47:42.329463+0000","last_active":"2024-02-13T00:50:35.564952+0000","last_peered":"2024-02-13T00:50:35.564952+0000","last_clean":"2024-02-13T00:50:35.564952+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:50:35.564952+0000","last_undegraded":"2024-02-13T00:50:35.564952+0000","last_fullsized":"2024-02-13T00:50:35.564952+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:42.329367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:42.329367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":231,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565225+0000","last_change":"2024-02-13T00:50:23.430479+0000","last_active":"2024-02-13T00:50:35.565225+0000","last_peered":"2024-02-13T00:50:35.565225+0000","last_clean":"2024-02-13T00:50:35.565225+0000","last_became_active":"2024-02-13T00:46:01.449873+0000","last_became_peered":"2024-02-13T00:46:01.449873+0000","last_unstale":"2024-02-13T00:50:35.565225+0000","last_undegraded":"2024-02-13T00:50:35.565225+0000","last_fullsized":"2024-02-13T00:50:35.565225+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:23.430396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:23.430396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":474,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565063+0000","last_change":"2024-02-13T00:46:56.973071+0000","last_active":"2024-02-13T00:50:35.565063+0000","last_peered":"2024-02-13T00:50:35.565063+0000","last_clean":"2024-02-13T00:50:35.565063+0000","last_became_active":"2024-02-13T00:44:39.504670+0000","last_became_peered":"2024-02-13T00:44:39.504670+0000","last_unstale":"2024-02-13T00:50:35.565063+0000","last_undegraded":"2024-02-13T00:50:35.565063+0000","last_fullsized":"2024-02-13T00:50:35.565063+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:46:56.973000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.973000+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":321,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565902+0000","last_change":"2024-02-13T00:46:01.448134+0000","last_active":"2024-02-13T00:50:35.565902+0000","last_peered":"2024-02-13T00:50:35.565902+0000","last_clean":"2024-02-13T00:50:35.565902+0000","last_became_active":"2024-02-13T00:46:01.446635+0000","last_became_peered":"2024-02-13T00:46:01.446635+0000","last_unstale":"2024-02-13T00:50:35.565902+0000","last_undegraded":"2024-02-13T00:50:35.565902+0000","last_fullsized":"2024-02-13T00:50:35.565902+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:28.101254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:28.101254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":317,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565952+0000","last_change":"2024-02-13T00:46:01.461188+0000","last_active":"2024-02-13T00:50:35.565952+0000","last_peered":"2024-02-13T00:50:35.565952+0000","last_clean":"2024-02-13T00:50:35.565952+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:50:35.565952+0000","last_undegraded":"2024-02-13T00:50:35.565952+0000","last_fullsized":"2024-02-13T00:50:35.565952+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.238152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.238152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":560,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510419+0000","last_change":"2024-02-13T00:46:01.443734+0000","last_active":"2024-02-13T00:50:35.510419+0000","last_peered":"2024-02-13T00:50:35.510419+0000","last_clean":"2024-02-13T00:50:35.510419+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:50:35.510419+0000","last_undegraded":"2024-02-13T00:50:35.510419+0000","last_fullsized":"2024-02-13T00:50:35.510419+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.621170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.621170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":465,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510135+0000","last_change":"2024-02-13T00:46:27.706096+0000","last_active":"2024-02-13T00:50:35.510135+0000","last_peered":"2024-02-13T00:50:35.510135+0000","last_clean":"2024-02-13T00:50:35.510135+0000","last_became_active":"2024-02-13T00:44:50.415894+0000","last_became_peered":"2024-02-13T00:44:50.415894+0000","last_unstale":"2024-02-13T00:50:35.510135+0000","last_undegraded":"2024-02-13T00:50:35.510135+0000","last_fullsized":"2024-02-13T00:50:35.510135+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:46:27.706006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:27.706006+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":401,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565410+0000","last_change":"2024-02-13T00:49:43.067713+0000","last_active":"2024-02-13T00:50:35.565410+0000","last_peered":"2024-02-13T00:50:35.565410+0000","last_clean":"2024-02-13T00:50:35.565410+0000","last_became_active":"2024-02-13T00:46:01.451790+0000","last_became_peered":"2024-02-13T00:46:01.451790+0000","last_unstale":"2024-02-13T00:50:35.565410+0000","last_undegraded":"2024-02-13T00:50:35.565410+0000","last_fullsized":"2024-02-13T00:50:35.565410+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:43.067615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:43.067615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":588,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.509756+0000","last_change":"2024-02-13T00:47:36.885611+0000","last_active":"2024-02-13T00:50:35.509756+0000","last_peered":"2024-02-13T00:50:35.509756+0000","last_clean":"2024-02-13T00:50:35.509756+0000","last_became_active":"2024-02-13T00:45:15.419668+0000","last_became_peered":"2024-02-13T00:45:15.419668+0000","last_unstale":"2024-02-13T00:50:35.509756+0000","last_undegraded":"2024-02-13T00:50:35.509756+0000","last_fullsized":"2024-02-13T00:50:35.509756+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.885510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.885510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":499,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511562+0000","last_change":"2024-02-13T00:50:15.846534+0000","last_active":"2024-02-13T00:50:35.511562+0000","last_peered":"2024-02-13T00:50:35.511562+0000","last_clean":"2024-02-13T00:50:35.511562+0000","last_became_active":"2024-02-13T00:46:01.455940+0000","last_became_peered":"2024-02-13T00:46:01.455940+0000","last_unstale":"2024-02-13T00:50:35.511562+0000","last_undegraded":"2024-02-13T00:50:35.511562+0000","last_fullsized":"2024-02-13T00:50:35.511562+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:15.846461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:15.846461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":482,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564069+0000","last_change":"2024-02-13T00:46:36.089483+0000","last_active":"2024-02-13T00:50:35.564069+0000","last_peered":"2024-02-13T00:50:35.564069+0000","last_clean":"2024-02-13T00:50:35.564069+0000","last_became_active":"2024-02-13T00:45:15.419364+0000","last_became_peered":"2024-02-13T00:45:15.419364+0000","last_unstale":"2024-02-13T00:50:35.564069+0000","last_undegraded":"2024-02-13T00:50:35.564069+0000","last_fullsized":"2024-02-13T00:50:35.564069+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:46:36.089402+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:46:36.089402+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":326,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.565912+0000","last_change":"2024-02-13T00:47:47.284092+0000","last_active":"2024-02-13T00:50:35.565912+0000","last_peered":"2024-02-13T00:50:35.565912+0000","last_clean":"2024-02-13T00:50:35.565912+0000","last_became_active":"2024-02-13T00:46:01.451997+0000","last_became_peered":"2024-02-13T00:46:01.451997+0000","last_unstale":"2024-02-13T00:50:35.565912+0000","last_undegraded":"2024-02-13T00:50:35.565912+0000","last_fullsized":"2024-02-13T00:50:35.565912+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.284014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.284014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":539,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563461+0000","last_change":"2024-02-13T00:49:01.133798+0000","last_active":"2024-02-13T00:50:35.563461+0000","last_peered":"2024-02-13T00:50:35.563461+0000","last_clean":"2024-02-13T00:50:35.563461+0000","last_became_active":"2024-02-13T00:46:01.445141+0000","last_became_peered":"2024-02-13T00:46:01.445141+0000","last_unstale":"2024-02-13T00:50:35.563461+0000","last_undegraded":"2024-02-13T00:50:35.563461+0000","last_fullsized":"2024-02-13T00:50:35.563461+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:01.133709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:49:01.133709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":308,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564079+0000","last_change":"2024-02-13T00:45:30.186208+0000","last_active":"2024-02-13T00:50:35.564079+0000","last_peered":"2024-02-13T00:50:35.564079+0000","last_clean":"2024-02-13T00:50:35.564079+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:50:35.564079+0000","last_undegraded":"2024-02-13T00:50:35.564079+0000","last_fullsized":"2024-02-13T00:50:35.564079+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:30.186129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:30.186129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":366,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.564130+0000","last_change":"2024-02-13T00:46:06.251713+0000","last_active":"2024-02-13T00:50:35.564130+0000","last_peered":"2024-02-13T00:50:35.564130+0000","last_clean":"2024-02-13T00:50:35.564130+0000","last_became_active":"2024-02-13T00:46:01.442464+0000","last_became_peered":"2024-02-13T00:46:01.442464+0000","last_unstale":"2024-02-13T00:50:35.564130+0000","last_undegraded":"2024-02-13T00:50:35.564130+0000","last_fullsized":"2024-02-13T00:50:35.564130+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:43:44.241358+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:43:44.241358+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":401,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.512427+0000","last_change":"2024-02-13T00:47:47.711551+0000","last_active":"2024-02-13T00:50:35.512427+0000","last_peered":"2024-02-13T00:50:35.512427+0000","last_clean":"2024-02-13T00:50:35.512427+0000","last_became_active":"2024-02-13T00:46:01.447654+0000","last_became_peered":"2024-02-13T00:46:01.447654+0000","last_unstale":"2024-02-13T00:50:35.512427+0000","last_undegraded":"2024-02-13T00:50:35.512427+0000","last_fullsized":"2024-02-13T00:50:35.512427+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.711486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.711486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":238,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563423+0000","last_change":"2024-02-13T00:47:53.826039+0000","last_active":"2024-02-13T00:50:35.563423+0000","last_peered":"2024-02-13T00:50:35.563423+0000","last_clean":"2024-02-13T00:50:35.563423+0000","last_became_active":"2024-02-13T00:46:01.448971+0000","last_became_peered":"2024-02-13T00:46:01.448971+0000","last_unstale":"2024-02-13T00:50:35.563423+0000","last_undegraded":"2024-02-13T00:50:35.563423+0000","last_fullsized":"2024-02-13T00:50:35.563423+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:53.825971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:53.825971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":565,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.563630+0000","last_change":"2024-02-13T00:45:32.233265+0000","last_active":"2024-02-13T00:50:35.563630+0000","last_peered":"2024-02-13T00:50:35.563630+0000","last_clean":"2024-02-13T00:50:35.563630+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:50:35.563630+0000","last_undegraded":"2024-02-13T00:50:35.563630+0000","last_fullsized":"2024-02-13T00:50:35.563630+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:45:32.233197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.233197+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":500,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.510232+0000","last_change":"2024-02-13T00:47:44.757362+0000","last_active":"2024-02-13T00:50:35.510232+0000","last_peered":"2024-02-13T00:50:35.510232+0000","last_clean":"2024-02-13T00:50:35.510232+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:50:35.510232+0000","last_undegraded":"2024-02-13T00:50:35.510232+0000","last_fullsized":"2024-02-13T00:50:35.510232+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:47:44.757262+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:47:44.757262+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":592,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.511295+0000","last_change":"2024-02-13T00:48:54.818104+0000","last_active":"2024-02-13T00:50:35.511295+0000","last_peered":"2024-02-13T00:50:35.511295+0000","last_clean":"2024-02-13T00:50:35.511295+0000","last_became_active":"2024-02-13T00:45:15.434224+0000","last_became_peered":"2024-02-13T00:45:15.434224+0000","last_unstale":"2024-02-13T00:50:35.511295+0000","last_undegraded":"2024-02-13T00:50:35.511295+0000","last_fullsized":"2024-02-13T00:50:35.511295+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:54.817999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.817999+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":312,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-13T00:50:35.566006+0000","last_change":"2024-02-13T00:45:15.435955+0000","last_active":"2024-02-13T00:50:35.566006+0000","last_peered":"2024-02-13T00:50:35.566006+0000","last_clean":"2024-02-13T00:50:35.566006+0000","last_became_active":"2024-02-13T00:45:15.435259+0000","last_became_peered":"2024-02-13T00:45:15.435259+0000","last_unstale":"2024-02-13T00:50:35.566006+0000","last_undegraded":"2024-02-13T00:50:35.566006+0000","last_fullsized":"2024-02-13T00:50:35.566006+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:31.953989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:43:31.953989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":237,"num_pg":32,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":4},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":59,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":24,"acting":24,"num_store_stats":4},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1492,"num_read_kb":1217,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":9606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":11656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":970,"ondisk_log_size":970,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":771,"num_read_kb":766,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":227633266847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":45,"seq":193273528485,"num_pgs":160,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308808,"kb_used_data":3392,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436344,"statfs":{"total":95995035648,"available":95678816256,"internally_reserved":0,"allocated":3473408,"data_stored":1682252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.47599999999999998,"15min":0.54400000000000004},"min":{"1min":0.33900000000000002,"5min":0.25600000000000001,"15min":0.19700000000000001},"max":{"1min":0.76100000000000001,"5min":0.76900000000000002,"15min":8.1340000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.503,"15min":0.61199999999999999},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.221},"max":{"1min":0.72399999999999998,"5min":1.0309999999999999,"15min":11.367000000000001},"last":0.65000000000000002}]},{"osd":1,"last update":"Tue Feb 13 00:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48399999999999999,"15min":0.88300000000000001},"min":{"1min":0.30099999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.68799999999999994,"5min":1.1080000000000001,"15min":54.002000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.498,"15min":0.68100000000000005},"min":{"1min":0.38400000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.77700000000000002,"5min":0.95199999999999996,"15min":12.403},"last":0.67800000000000005}]},{"osd":2,"last update":"Tue Feb 13 00:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51500000000000001,"15min":0.71099999999999997},"min":{"1min":0.379,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66700000000000004,"5min":1.587,"15min":12.477},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.495,"15min":0.67900000000000005},"min":{"1min":0.35399999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.70499999999999996,"5min":1.1899999999999999,"15min":11.702},"last":0.66100000000000003}]},{"osd":3,"last update":"Tue Feb 13 00:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54900000000000004,"15min":0.67800000000000005},"min":{"1min":0.31900000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66400000000000003,"5min":5.931,"15min":8.0519999999999996},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.77200000000000002},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.67300000000000004,"5min":1.2330000000000001,"15min":19.081},"last":0.58699999999999997}]},{"osd":4,"last update":"Tue Feb 13 00:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.41899999999999998,"15min":0.41699999999999998},"min":{"1min":0.316,"5min":0.26100000000000001,"15min":0.24199999999999999},"max":{"1min":0.67000000000000004,"5min":0.71899999999999997,"15min":1.0649999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44},"min":{"1min":0.29899999999999999,"5min":0.22800000000000001,"15min":0.221},"max":{"1min":0.66900000000000004,"5min":1.008,"15min":1.008},"last":0.63200000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66400000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69899999999999995,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.71399999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45600000000000002,"15min":0.48099999999999998},"min":{"1min":0.36699999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60899999999999999,"5min":0.66800000000000004,"15min":1.173},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.50900000000000001,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.28299999999999997,"15min":0.26900000000000002},"max":{"1min":0.69999999999999996,"5min":0.80700000000000005,"15min":1.262},"last":0.70699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575038,"num_pgs":161,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304724,"kb_used_data":3404,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440428,"statfs":{"total":95995035648,"available":95682998272,"internally_reserved":0,"allocated":3485696,"data_stored":1677917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.33200000000000002,"15min":0.95299999999999996},"min":{"1min":0.19,"5min":0.187,"15min":0.187},"max":{"1min":0.60499999999999998,"5min":0.68200000000000005,"15min":61.856000000000002},"last":0.311},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.67000000000000004},"min":{"1min":0.26400000000000001,"5min":0.19900000000000001,"15min":0.16500000000000001},"max":{"1min":0.48199999999999998,"5min":0.70199999999999996,"15min":61.908999999999999},"last":0.38300000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:50:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33700000000000002,"15min":0.84599999999999997},"min":{"1min":0.27300000000000002,"5min":0.22900000000000001,"15min":0.20599999999999999},"max":{"1min":0.53000000000000003,"5min":0.80000000000000004,"15min":32.161000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.82699999999999996},"min":{"1min":0.253,"5min":0.187,"15min":0.187},"max":{"1min":0.504,"5min":1.21,"15min":29.693000000000001},"last":0.26300000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":1.2609999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":0.497,"5min":0.59099999999999997,"15min":131.23099999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.379,"15min":0.85499999999999998},"min":{"1min":0.22800000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.48799999999999999,"5min":0.61799999999999999,"15min":49.055},"last":0.35699999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.75800000000000001},"min":{"1min":0.36199999999999999,"5min":0.33500000000000002,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":1.246,"15min":16.547000000000001},"last":0.495},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.56399999999999995,"15min":0.71199999999999997},"min":{"1min":0.42499999999999999,"5min":0.34100000000000003,"15min":0.24199999999999999},"max":{"1min":0.85199999999999998,"5min":0.89200000000000002,"15min":14.731999999999999},"last":0.53100000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59899999999999998,"15min":0.58699999999999997},"min":{"1min":0.437,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.51100000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56200000000000006,"15min":0.71699999999999997},"min":{"1min":0.38600000000000001,"5min":0.32300000000000001,"15min":0.29099999999999998},"max":{"1min":1.0700000000000001,"5min":1.9279999999999999,"15min":14.772},"last":0.434},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.92200000000000004},"min":{"1min":0.41199999999999998,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":26.600000000000001},"last":0.48199999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:50:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.56299999999999994,"15min":0.72299999999999998},"min":{"1min":0.41499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":14.794},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55900000000000005,"15min":0.92500000000000004},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.32200000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":26.693999999999999},"last":0.51700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738562,"num_pgs":156,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308784,"kb_used_data":3368,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436368,"statfs":{"total":95995035648,"available":95678840832,"internally_reserved":0,"allocated":3448832,"data_stored":1665238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.35199999999999998,"15min":0.622},"min":{"1min":0.24199999999999999,"5min":0.191,"15min":0.189},"max":{"1min":0.44500000000000001,"5min":1.7370000000000001,"15min":30.184000000000001},"last":0.308},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45700000000000002},"min":{"1min":0.26400000000000001,"5min":0.20499999999999999,"15min":0.17599999999999999},"max":{"1min":0.58899999999999997,"5min":2.839,"15min":5.6959999999999997},"last":0.35299999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40300000000000002,"15min":0.498},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.20999999999999999},"max":{"1min":0.64300000000000002,"5min":1.3700000000000001,"15min":7.0839999999999996},"last":0.438},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42499999999999999,"15min":0.66900000000000004},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.185},"max":{"1min":0.53600000000000003,"5min":3.6949999999999998,"15min":24.459},"last":0.33600000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:50:21 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.42799999999999999,"15min":0.48499999999999999},"min":{"1min":0.32600000000000001,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":0.48499999999999999,"5min":4.8710000000000004,"15min":8.718},"last":0.371},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.442,"15min":0.53400000000000003},"min":{"1min":0.315,"5min":0.26900000000000002,"15min":0.222},"max":{"1min":0.47099999999999997,"5min":2.3140000000000001,"15min":7.1139999999999999},"last":0.39500000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.63300000000000001,"15min":0.79900000000000004},"min":{"1min":0.372,"5min":0.29899999999999999,"15min":0.24399999999999999},"max":{"1min":0.874,"5min":7.056,"15min":24.381},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.40500000000000003,"5min":0.31,"15min":0.28699999999999998},"max":{"1min":0.88200000000000001,"5min":6.1859999999999999,"15min":8.6950000000000003},"last":0.49399999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.5,"15min":0.495},"min":{"1min":0.40799999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.42699999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53300000000000003,"15min":0.628},"min":{"1min":0.40500000000000003,"5min":0.34100000000000003,"15min":0.31},"max":{"1min":0.77200000000000002,"5min":1.4990000000000001,"15min":8.6129999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.65500000000000003},"min":{"1min":0.42899999999999999,"5min":0.34000000000000002,"15min":0.32400000000000001},"max":{"1min":0.72099999999999997,"5min":2.206,"15min":14.263},"last":0.59199999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.44700000000000001,"5min":0.35999999999999999,"15min":0.317},"max":{"1min":0.78400000000000003,"5min":7.7720000000000002,"15min":8.6679999999999993},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58299999999999996,"15min":0.66900000000000004},"min":{"1min":0.44,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91700000000000004,"5min":1.6910000000000001,"15min":8.1609999999999996},"last":0.61199999999999999}]}]},{"osd":2,"up_from":0,"seq":77309411513,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182579,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986351,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304672,"kb_used_data":3352,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440480,"statfs":{"total":95995035648,"available":95683051520,"internally_reserved":0,"allocated":3432448,"data_stored":1665326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.59399999999999997},"min":{"1min":0.34200000000000003,"5min":0.248,"15min":0.21199999999999999},"max":{"1min":0.71999999999999997,"5min":5.3319999999999999,"15min":9.3399999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.63500000000000001,"15min":0.60999999999999999},"min":{"1min":0.34499999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.65800000000000003,"5min":11.532,"15min":13.833},"last":0.35599999999999998}]},{"osd":1,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.59799999999999998,"15min":0.753},"min":{"1min":0.373,"5min":0.25,"15min":0.25},"max":{"1min":0.76900000000000002,"5min":7.5410000000000004,"15min":53.188000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.60899999999999999,"15min":0.69999999999999996},"min":{"1min":0.32300000000000001,"5min":0.28999999999999998,"15min":0.25600000000000001},"max":{"1min":0.71099999999999997,"5min":11.497999999999999,"15min":44.793999999999997},"last":0.34699999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58499999999999996,"15min":0.59699999999999998},"min":{"1min":0.37,"5min":0.31900000000000001,"15min":0.26900000000000002},"max":{"1min":0.78900000000000003,"5min":4.2199999999999998,"15min":8.6829999999999998},"last":0.379},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.59299999999999997,"15min":0.629},"min":{"1min":0.38300000000000001,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.74299999999999999,"5min":4.0730000000000004,"15min":18.911000000000001},"last":0.42999999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53800000000000003,"15min":0.65400000000000003},"min":{"1min":0.34599999999999997,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64000000000000001,"5min":4.0090000000000003,"15min":23.5},"last":0.373},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55100000000000005,"15min":0.71299999999999997},"min":{"1min":0.39400000000000002,"5min":0.32100000000000001,"15min":0.30599999999999999},"max":{"1min":0.68899999999999995,"5min":3.9729999999999999,"15min":36.152999999999999},"last":0.41999999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:50:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.33700000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.69399999999999995,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.317}]},{"osd":6,"last update":"Tue Feb 13 00:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.435,"15min":0.443},"min":{"1min":0.378,"5min":0.26800000000000002,"15min":0.25700000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":3.153},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.46400000000000002,"15min":0.47199999999999998},"min":{"1min":0.34499999999999997,"5min":0.25800000000000001,"15min":0.254},"max":{"1min":0.65900000000000003,"5min":0.73099999999999998,"15min":4.9390000000000001},"last":0.44}]},{"osd":7,"last update":"Tue Feb 13 00:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.49299999999999999,"15min":0.48099999999999998},"min":{"1min":0.34000000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.76500000000000001,"5min":1.575,"15min":1.575},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.5,"15min":0.47699999999999998},"min":{"1min":0.309,"5min":0.29699999999999999,"15min":0.28100000000000003},"max":{"1min":0.72499999999999998,"5min":2.46,"15min":2.46},"last":0.33300000000000002}]}]},{"osd":5,"up_from":0,"seq":2057289334817,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":3328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:50:48.838 INFO:tasks.thrashosds.thrasher:Setting 4.b to [3, 6, 5] 2024-02-13T00:50:48.838 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.b', '3', '6', '5'] 2024-02-13T00:50:48.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pg-upmap 4.b 3 6 5 2024-02-13T00:50:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:48.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:48.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:48.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:48.973 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:48.994 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:48.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:49.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:49.124 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:49.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:48.496892+0000 mgr.y (mgr.14544) 536 : cluster [DBG] pgmap v940: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:50:49.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:49.340678+0000 mon.a (mon.0) 3157 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:49.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:48.496892+0000 mgr.y (mgr.14544) 536 : cluster [DBG] pgmap v940: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:50:49.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:49.340678+0000 mon.a (mon.0) 3157 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:49.787 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:48.496892+0000 mgr.y (mgr.14544) 536 : cluster [DBG] pgmap v940: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:50:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:49.340678+0000 mon.a (mon.0) 3157 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:50:50.904 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:50.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:51.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:51.040 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:51.289 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:51.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:51.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:51.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:51.425 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:51.447 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:51.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:51.533 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:50:51.533 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:50:51.533 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:50:51.533 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:50:51.534 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:50:51.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:51.554 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:50.348779+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e624: 8 total, 8 up, 4 in 2024-02-13T00:50:51.554 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:50.373502+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:51.554 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:50.497265+0000 mgr.y (mgr.14544) 537 : cluster [DBG] pgmap v942: 204 pgs: 204 active+clean; 5.8 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-02-13T00:50:51.554 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:50.516140+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:50:51.554 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:50.516202+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e625: 8 total, 8 up, 4 in 2024-02-13T00:50:51.554 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:50.517630+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-106"}]: dispatch 2024-02-13T00:50:51.554 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:50.348779+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e624: 8 total, 8 up, 4 in 2024-02-13T00:50:51.554 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:50.373502+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:51.555 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:50.497265+0000 mgr.y (mgr.14544) 537 : cluster [DBG] pgmap v942: 204 pgs: 204 active+clean; 5.8 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-02-13T00:50:51.555 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:50.516140+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:50:51.555 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:50.516202+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e625: 8 total, 8 up, 4 in 2024-02-13T00:50:51.555 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:50.517630+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-106"}]: dispatch 2024-02-13T00:50:51.579 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:50.348779+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e624: 8 total, 8 up, 4 in 2024-02-13T00:50:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:50.373502+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:50:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:50.497265+0000 mgr.y (mgr.14544) 537 : cluster [DBG] pgmap v942: 204 pgs: 204 active+clean; 5.8 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-02-13T00:50:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:50.516140+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:50:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:50.516202+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e625: 8 total, 8 up, 4 in 2024-02-13T00:50:51.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:50.517630+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-106"}]: dispatch 2024-02-13T00:50:52.229 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:52.229 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:52.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:52.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.117+0000 7f7be1c7d700 1 -- 172.21.15.162:0/2781589569 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7bdc0f73a0 msgr2=0x7f7bdc1039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:52.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.117+0000 7f7be1c7d700 1 --2- 172.21.15.162:0/2781589569 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7bdc0f73a0 0x7f7bdc1039a0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7f7bd4009be0 tx=0x7f7bd4004d10).stop 2024-02-13T00:50:52.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.117+0000 7f7be1c7d700 1 -- 172.21.15.162:0/2781589569 shutdown_connections 2024-02-13T00:50:52.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.117+0000 7f7be1c7d700 1 --2- 172.21.15.162:0/2781589569 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7bdc0f73a0 0x7f7bdc1039a0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:52.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.117+0000 7f7be1c7d700 1 --2- 172.21.15.162:0/2781589569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bdc0fcb30 0x7f7bdc0fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.117+0000 7f7be1c7d700 1 --2- 172.21.15.162:0/2781589569 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bdc0fd490 0x7f7bdc0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.117+0000 7f7be1c7d700 1 -- 172.21.15.162:0/2781589569 >> 172.21.15.162:0/2781589569 conn(0x7f7bdc06eb90 msgr2=0x7f7bdc0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.117+0000 7f7be1c7d700 1 -- 172.21.15.162:0/2781589569 shutdown_connections 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.117+0000 7f7be1c7d700 1 -- 172.21.15.162:0/2781589569 wait complete. 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.117+0000 7f7be1c7d700 1 librados: starting msgr at 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.117+0000 7f7be1c7d700 1 librados: starting objecter 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.117+0000 7f7be1c7d700 1 Processor -- start 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7be1c7d700 1 -- start start 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7be1c7d700 1 librados: setting wanted keys 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7be1c7d700 1 librados: calling monclient init 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7be1c7d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bdc0f73a0 0x7f7bdc0f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7be1c7d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bdc0fcb30 0x7f7bdc0f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7be1c7d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7bdc0fd490 0x7f7bdc0f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:52.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7be1c7d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc116470 con 0x7f7bdc0f73a0 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7be1c7d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc1165e0 con 0x7f7bdc0fcb30 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7be1c7d700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc1167a0 con 0x7f7bdc0fd490 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdb7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bdc0f73a0 0x7f7bdc0f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdbfff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7bdc0fd490 0x7f7bdc0f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdb7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bdc0f73a0 0x7f7bdc0f8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:55742/0 (socket says 172.21.15.162:55742) 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdbfff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7bdc0fd490 0x7f7bdc0f9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:54124/0 (socket says 172.21.15.162:54124) 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdb7fe700 1 -- 172.21.15.162:0/926262831 learned_addr learned my addr 172.21.15.162:0/926262831 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdaffd700 1 --2- 172.21.15.162:0/926262831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bdc0fcb30 0x7f7bdc0f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdbfff700 1 -- 172.21.15.162:0/926262831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bdc0fcb30 msgr2=0x7f7bdc0f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdbfff700 1 --2- 172.21.15.162:0/926262831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bdc0fcb30 0x7f7bdc0f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdbfff700 1 -- 172.21.15.162:0/926262831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bdc0f73a0 msgr2=0x7f7bdc0f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdbfff700 1 --2- 172.21.15.162:0/926262831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bdc0f73a0 0x7f7bdc0f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdbfff700 1 -- 172.21.15.162:0/926262831 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd0005040 con 0x7f7bdc0fd490 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdb7fe700 1 --2- 172.21.15.162:0/926262831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bdc0f73a0 0x7f7bdc0f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bdbfff700 1 --2- 172.21.15.162:0/926262831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7bdc0fd490 0x7f7bdc0f9310 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 rx=0x7f7bd400b6d0 tx=0x7f7bd4004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:52.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bd8ff9700 10 client.?.objecter ms_handle_connect 0x7f7bdc0fd490 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.118+0000 7f7bd8ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.119+0000 7f7bd8ff9700 1 -- 172.21.15.162:0/926262831 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd400e040 con 0x7f7bdc0fd490 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.119+0000 7f7bd8ff9700 1 -- 172.21.15.162:0/926262831 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7bd4018040 con 0x7f7bdc0fd490 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.119+0000 7f7be1c7d700 1 -- 172.21.15.162:0/926262831 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd40097c0 con 0x7f7bdc0fd490 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.119+0000 7f7bd8ff9700 1 -- 172.21.15.162:0/926262831 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4013e90 con 0x7f7bdc0fd490 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.119+0000 7f7be1c7d700 10 client.37922.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.119+0000 7f7be1c7d700 1 -- 172.21.15.162:0/926262831 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bdc116d10 con 0x7f7bdc0fd490 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.119+0000 7f7be1c7d700 1 librados: init done 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.119+0000 7f7be1c7d700 1 -- 172.21.15.162:0/926262831 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bdc106090 con 0x7f7bdc0fd490 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.120+0000 7f7bd8ff9700 1 -- 172.21.15.162:0/926262831 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f7bd4013590 con 0x7f7bdc0fd490 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.121+0000 7f7bd8ff9700 1 --2- 172.21.15.162:0/926262831 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7bc4062c90 0x7f7bc4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.121+0000 7f7bd8ff9700 1 -- 172.21.15.162:0/926262831 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(626..626 src has 449..626) v4 ==== 8518+0+0 (secure 0 0 0) 0x7f7bd407eed0 con 0x7f7bdc0fd490 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.121+0000 7f7bd8ff9700 10 client.37922.objecter ms_dispatch 0x7f7bdc001550 osd_map(626..626 src has 449..626) v4 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.121+0000 7f7bd8ff9700 3 client.37922.objecter handle_osd_map got epochs [626,626] > 0 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.121+0000 7f7bd8ff9700 3 client.37922.objecter handle_osd_map decoding full epoch 626 2024-02-13T00:50:52.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.121+0000 7f7bdb7fe700 1 --2- 172.21.15.162:0/926262831 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7bc4062c90 0x7f7bc4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:52.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.121+0000 7f7bd8ff9700 20 client.37922.objecter dump_active .. 0 homeless 2024-02-13T00:50:52.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.121+0000 7f7bd8ff9700 10 client.37922.objecter ms_handle_connect 0x7f7bc4062c90 2024-02-13T00:50:52.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.121+0000 7f7bdb7fe700 1 --2- 172.21.15.162:0/926262831 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7bc4062c90 0x7f7bc4065150 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 rx=0x7f7bd0009810 tx=0x7f7bd000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:52.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.124+0000 7f7bd8ff9700 1 -- 172.21.15.162:0/926262831 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7bd4052250 con 0x7f7bdc0fd490 2024-02-13T00:50:52.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.306+0000 7f7be1c7d700 1 -- 172.21.15.162:0/926262831 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f7bdc05fc20 con 0x7f7bdc0fd490 2024-02-13T00:50:52.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:52.365 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:52.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.514+0000 7f7bd8ff9700 1 -- 172.21.15.162:0/926262831 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v627) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7bd4021070 con 0x7f7bdc0fd490 2024-02-13T00:50:52.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.517+0000 7f7be1c7d700 10 librados: watch_flush enter 2024-02-13T00:50:52.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.517+0000 7f7be1c7d700 10 librados: watch_flush exit 2024-02-13T00:50:52.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.517+0000 7f7be1c7d700 20 client.37922.objecter shutdown clearing up homeless session... 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.517+0000 7f7be1c7d700 10 client.37922.objecter successfully canceled tick 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.517+0000 7f7be1c7d700 1 -- 172.21.15.162:0/926262831 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7bc4062c90 msgr2=0x7f7bc4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.517+0000 7f7be1c7d700 1 --2- 172.21.15.162:0/926262831 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7bc4062c90 0x7f7bc4065150 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 rx=0x7f7bd0009810 tx=0x7f7bd000b040).stop 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.517+0000 7f7be1c7d700 1 -- 172.21.15.162:0/926262831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7bdc0fd490 msgr2=0x7f7bdc0f9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.517+0000 7f7be1c7d700 1 --2- 172.21.15.162:0/926262831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7bdc0fd490 0x7f7bdc0f9310 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 rx=0x7f7bd400b6d0 tx=0x7f7bd4004d10).stop 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.518+0000 7f7be1c7d700 1 -- 172.21.15.162:0/926262831 shutdown_connections 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.518+0000 7f7be1c7d700 1 --2- 172.21.15.162:0/926262831 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7bc4062c90 0x7f7bc4065150 unknown :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.518+0000 7f7be1c7d700 1 --2- 172.21.15.162:0/926262831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7bdc0fd490 0x7f7bdc0f9310 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.518+0000 7f7be1c7d700 1 --2- 172.21.15.162:0/926262831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bdc0f73a0 0x7f7bdc0f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.518+0000 7f7be1c7d700 1 --2- 172.21.15.162:0/926262831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bdc0fcb30 0x7f7bdc0f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.518+0000 7f7be1c7d700 1 -- 172.21.15.162:0/926262831 >> 172.21.15.162:0/926262831 conn(0x7f7bdc06eb90 msgr2=0x7f7bdc060e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.518+0000 7f7be1c7d700 1 -- 172.21.15.162:0/926262831 shutdown_connections 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.518+0000 7f7be1c7d700 1 -- 172.21.15.162:0/926262831 wait complete. 2024-02-13T00:50:52.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:52.518+0000 7f7be1c7d700 1 librados: shutdown 2024-02-13T00:50:52.524 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:50:52.548 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:51.149165+0000 osd.2 (osd.2) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:50:52.548 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:51.149333+0000 osd.2 (osd.2) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:50:52.548 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:51.508748+0000 mon.a (mon.0) 3163 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:50:52.548 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:51.512964+0000 mon.a (mon.0) 3164 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-106"}]': finished 2024-02-13T00:50:52.548 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:51.513029+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e626: 8 total, 8 up, 4 in 2024-02-13T00:50:52.548 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:52.309064+0000 mon.c (mon.2) 476 : audit [INF] from='client.? 172.21.15.162:0/926262831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:50:52.548 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:52.309375+0000 mon.a (mon.0) 3166 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:50:52.548 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:51.149165+0000 osd.2 (osd.2) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:50:52.549 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:51.149333+0000 osd.2 (osd.2) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:50:52.549 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:51.508748+0000 mon.a (mon.0) 3163 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:50:52.549 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:51.512964+0000 mon.a (mon.0) 3164 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-106"}]': finished 2024-02-13T00:50:52.549 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:51.513029+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e626: 8 total, 8 up, 4 in 2024-02-13T00:50:52.549 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:52.309064+0000 mon.c (mon.2) 476 : audit [INF] from='client.? 172.21.15.162:0/926262831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:50:52.549 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:52.309375+0000 mon.a (mon.0) 3166 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:50:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:51.149165+0000 osd.2 (osd.2) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:50:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:51.149333+0000 osd.2 (osd.2) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:50:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:51.508748+0000 mon.a (mon.0) 3163 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:50:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:51.512964+0000 mon.a (mon.0) 3164 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-106"}]': finished 2024-02-13T00:50:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:51.513029+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e626: 8 total, 8 up, 4 in 2024-02-13T00:50:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:52.309064+0000 mon.c (mon.2) 476 : audit [INF] from='client.? 172.21.15.162:0/926262831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:50:52.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:52.309375+0000 mon.a (mon.0) 3166 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:50:53.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:52.497724+0000 mgr.y (mgr.14544) 538 : cluster [DBG] pgmap v945: 204 pgs: 204 active+clean; 5.8 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:50:53.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:52.510305+0000 mon.a (mon.0) 3167 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:50:53.471 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:52.515156+0000 mon.a (mon.0) 3168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:50:53.472 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:52.515224+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e627: 8 total, 8 up, 4 in 2024-02-13T00:50:53.472 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:52.497724+0000 mgr.y (mgr.14544) 538 : cluster [DBG] pgmap v945: 204 pgs: 204 active+clean; 5.8 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:50:53.472 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:52.510305+0000 mon.a (mon.0) 3167 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:50:53.472 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:52.515156+0000 mon.a (mon.0) 3168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:50:53.472 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:52.515224+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e627: 8 total, 8 up, 4 in 2024-02-13T00:50:53.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-13T00:50:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:52.497724+0000 mgr.y (mgr.14544) 538 : cluster [DBG] pgmap v945: 204 pgs: 204 active+clean; 5.8 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:50:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:52.510305+0000 mon.a (mon.0) 3167 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:50:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:52.515156+0000 mon.a (mon.0) 3168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:50:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:52.515224+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e627: 8 total, 8 up, 4 in 2024-02-13T00:50:53.944 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:53.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:54.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:54.104 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:54.178 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:54.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:54.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:54.309 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:54.331 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:54.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:54.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:54.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:54.488 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:54.509 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:54.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:54.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:53.347038+0000 mon.a (mon.0) 3170 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:50:54.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:53.516635+0000 mon.a (mon.0) 3171 : cluster [DBG] osdmap e628: 8 total, 8 up, 4 in 2024-02-13T00:50:54.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:53.521165+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:50:54.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:53.347038+0000 mon.a (mon.0) 3170 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:50:54.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:53.516635+0000 mon.a (mon.0) 3171 : cluster [DBG] osdmap e628: 8 total, 8 up, 4 in 2024-02-13T00:50:54.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:53.521165+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:50:54.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:54.639 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:53.347038+0000 mon.a (mon.0) 3170 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:50:54.639 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:53.516635+0000 mon.a (mon.0) 3171 : cluster [DBG] osdmap e628: 8 total, 8 up, 4 in 2024-02-13T00:50:54.639 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:53.521165+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:50:54.641 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:54.660 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:54.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:54.764 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:54.792 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:54.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.848+0000 7fafd9d03700 1 -- 172.21.15.162:0/806106808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd40f9570 msgr2=0x7fafd4101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:54.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.848+0000 7fafd9d03700 1 --2- 172.21.15.162:0/806106808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd40f9570 0x7fafd4101c80 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7fafbc004870 tx=0x7fafbc009cb0).stop 2024-02-13T00:50:54.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.848+0000 7fafd9d03700 1 -- 172.21.15.162:0/806106808 shutdown_connections 2024-02-13T00:50:54.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.848+0000 7fafd9d03700 1 --2- 172.21.15.162:0/806106808 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafd4102170 0x7fafd4104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:54.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.848+0000 7fafd9d03700 1 --2- 172.21.15.162:0/806106808 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafd40f8c10 0x7fafd40f8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:54.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.848+0000 7fafd9d03700 1 --2- 172.21.15.162:0/806106808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd40f9570 0x7fafd4101c80 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.848+0000 7fafd9d03700 1 -- 172.21.15.162:0/806106808 >> 172.21.15.162:0/806106808 conn(0x7fafd40f2720 msgr2=0x7fafd40f4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.848+0000 7fafd9d03700 1 -- 172.21.15.162:0/806106808 shutdown_connections 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.848+0000 7fafd9d03700 1 -- 172.21.15.162:0/806106808 wait complete. 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.848+0000 7fafd9d03700 1 librados: starting msgr at 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.848+0000 7fafd9d03700 1 librados: starting objecter 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.848+0000 7fafd9d03700 1 Processor -- start 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd9d03700 1 -- start start 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd9d03700 1 librados: setting wanted keys 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd9d03700 1 librados: calling monclient init 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd9d03700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafd40f8c10 0x7fafd40f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd9d03700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd40f9570 0x7fafd40f70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd9d03700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafd4102170 0x7fafd40f75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd9d03700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd41166d0 con 0x7fafd40f9570 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd9d03700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd4116840 con 0x7fafd40f8c10 2024-02-13T00:50:54.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd9d03700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd4116a00 con 0x7fafd4102170 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd3fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafd4102170 0x7fafd40f75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd40f9570 0x7fafd40f70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd40f9570 0x7fafd40f70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44220/0 (socket says 172.21.15.162:44220) 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd3fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafd4102170 0x7fafd40f75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:38996/0 (socket says 172.21.15.162:38996) 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd2ffd700 1 -- 172.21.15.162:0/1726523762 learned_addr learned my addr 172.21.15.162:0/1726523762 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd37fe700 1 --2- 172.21.15.162:0/1726523762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafd40f8c10 0x7fafd40f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd2ffd700 1 -- 172.21.15.162:0/1726523762 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafd4102170 msgr2=0x7fafd40f75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd2ffd700 1 --2- 172.21.15.162:0/1726523762 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafd4102170 0x7fafd40f75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd2ffd700 1 -- 172.21.15.162:0/1726523762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafd40f8c10 msgr2=0x7fafd40f6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd2ffd700 1 --2- 172.21.15.162:0/1726523762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafd40f8c10 0x7fafd40f6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd2ffd700 1 -- 172.21.15.162:0/1726523762 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc40097c0 con 0x7fafd40f9570 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd3fff700 1 --2- 172.21.15.162:0/1726523762 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafd4102170 0x7fafd40f75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd2ffd700 1 --2- 172.21.15.162:0/1726523762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd40f9570 0x7fafd40f70b0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 rx=0x7fafbc009eb0 tx=0x7fafbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd0ff9700 10 client.?.objecter ms_handle_connect 0x7fafd40f9570 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.849+0000 7fafd0ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:54.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.850+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc00e070 con 0x7fafd40f9570 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.850+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafbc005040 con 0x7fafd40f9570 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.850+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafbc013da0 con 0x7fafd40f9570 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.850+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc019070 con 0x7fafd40f9570 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.850+0000 7fafd9d03700 10 client.27831.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.850+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd4116ef0 con 0x7fafd40f9570 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.850+0000 7fafd9d03700 1 librados: init done 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.850+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fafd40607d0 con 0x7fafd40f9570 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.851+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fafbc01e420 con 0x7fafd40f9570 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.851+0000 7fafd0ff9700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6800/2866534053 conn(0x7fafc0062c90 0x7fafc0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.851+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(629..629 src has 449..629) v4 ==== 8531+0+0 (secure 0 0 0) 0x7fafbc07e380 con 0x7fafd40f9570 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.851+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 osd_map(629..629 src has 449..629) v4 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.851+0000 7fafd0ff9700 3 client.27831.objecter handle_osd_map got epochs [629,629] > 0 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.851+0000 7fafd0ff9700 3 client.27831.objecter handle_osd_map decoding full epoch 629 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.851+0000 7fafd37fe700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6800/2866534053 conn(0x7fafc0062c90 0x7fafc0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd0ff9700 20 client.27831.objecter dump_active .. 0 homeless 2024-02-13T00:50:54.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v629) v1 ==== 54+0+16 (secure 0 0 0) 0x7fafbc07e820 con 0x7fafd40f9570 2024-02-13T00:50:54.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd9d03700 10 client.27831.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:54.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 1 2024-02-13T00:50:54.879 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:50:54.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6801/2103743338 conn(0x7fafd40fc0b0 0x7fafd40fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:54.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd37fe700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6800/2866534053 conn(0x7fafc0062c90 0x7fafc0065150 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 rx=0x7fafc4001530 tx=0x7fafc4004c50).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:54.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd0ff9700 10 client.27831.objecter ms_handle_connect 0x7fafc0062c90 2024-02-13T00:50:54.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd40475e0 osd=0 3 2024-02-13T00:50:54.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd40475e0 osd=0 3 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd40475e0 osd=0 3 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 1 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd40475e0 osd=0 3 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 0 1 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd40475e0 osd=0 4 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd9d03700 10 client.27831.objecter _send_command 1 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafd41071a0 con 0x7fafd40fc0b0 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd3fff700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6801/2103743338 conn(0x7fafd40fc0b0 0x7fafd40fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd0ff9700 10 client.27831.objecter ms_handle_connect 0x7fafd40fc0b0 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.852+0000 7fafd3fff700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6801/2103743338 conn(0x7fafd40fc0b0 0x7fafd40fc4d0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.853+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fafd41071a0 con 0x7fafd40fc0b0 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.853+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.853+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 1 = system:0 2024-02-13T00:50:54.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.853+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd40475e0 osd=0 4 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.853+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 0 1 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.867+0000 7fafd9d03700 10 client.27831.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.867+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 2 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.867+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd40475e0 osd=0 3 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.867+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd40475e0 osd=0 3 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.867+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd40475e0 osd=0 3 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.867+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 2 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.867+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd40475e0 osd=0 3 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.867+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 0 2 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.867+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd40475e0 osd=0 4 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.867+0000 7fafd9d03700 10 client.27831.objecter _send_command 2 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.867+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fafd4027410 con 0x7fafd40fc0b0 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fafd4027410 con 0x7fafd40fc0b0 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:54.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd40475e0 osd=0 4 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 0 2 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd9d03700 10 client.27831.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 3 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6805/150599780 conn(0x7fafd4059990 0x7fafd4059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4028760 osd=1 3 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4028760 osd=1 3 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4028760 osd=1 3 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 3 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd4028760 osd=1 3 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 1 3 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4028760 osd=1 4 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd9d03700 10 client.27831.objecter _send_command 3 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafd411d800 con 0x7fafd4059990 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd37fe700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6805/150599780 conn(0x7fafd4059990 0x7fafd4059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:54.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd0ff9700 10 client.27831.objecter ms_handle_connect 0x7fafd4059990 2024-02-13T00:50:54.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.869+0000 7fafd37fe700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6805/150599780 conn(0x7fafd4059990 0x7fafd4059d70 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:54.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.870+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fafd411d800 con 0x7fafd4059990 2024-02-13T00:50:54.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.870+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:50:54.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.870+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 3 = system:0 2024-02-13T00:50:54.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.870+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd4028760 osd=1 4 2024-02-13T00:50:54.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.870+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 1 3 2024-02-13T00:50:54.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.885+0000 7fafd9d03700 10 client.27831.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:54.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.885+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 4 2024-02-13T00:50:54.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.885+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4028760 osd=1 3 2024-02-13T00:50:54.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.885+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4028760 osd=1 3 2024-02-13T00:50:54.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.885+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4028760 osd=1 3 2024-02-13T00:50:54.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.885+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 4 2024-02-13T00:50:54.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.885+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd4028760 osd=1 3 2024-02-13T00:50:54.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.885+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 1 4 2024-02-13T00:50:54.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.885+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4028760 osd=1 4 2024-02-13T00:50:54.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.885+0000 7fafd9d03700 10 client.27831.objecter _send_command 4 2024-02-13T00:50:54.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.885+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fafd4059400 con 0x7fafd4059990 2024-02-13T00:50:54.890 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:54.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.886+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fafd4059400 con 0x7fafd4059990 2024-02-13T00:50:54.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.886+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:54.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.886+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:54.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.886+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd4028760 osd=1 4 2024-02-13T00:50:54.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.886+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 1 4 2024-02-13T00:50:54.891 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:50:54.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd9d03700 10 client.27831.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:54.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 5 2024-02-13T00:50:54.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6809/3689719376 conn(0x7fafd411c220 0x7fafd411c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4027db0 osd=2 3 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4027db0 osd=2 3 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4027db0 osd=2 3 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 5 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd4027db0 osd=2 3 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 2 5 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4027db0 osd=2 4 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd9d03700 10 client.27831.objecter _send_command 5 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafd4028430 con 0x7fafd411c220 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd2ffd700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6809/3689719376 conn(0x7fafd411c220 0x7fafd411c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd0ff9700 10 client.27831.objecter ms_handle_connect 0x7fafd411c220 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.887+0000 7fafd2ffd700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6809/3689719376 conn(0x7fafd411c220 0x7fafd411c620 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.888+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fafd4028430 con 0x7fafd411c220 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.888+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.888+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 5 = system:0 2024-02-13T00:50:54.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.888+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd4027db0 osd=2 4 2024-02-13T00:50:54.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.888+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 2 5 2024-02-13T00:50:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:54.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:54.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.903+0000 7fafd9d03700 10 client.27831.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:54.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.903+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 6 2024-02-13T00:50:54.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.903+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4027db0 osd=2 3 2024-02-13T00:50:54.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.903+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4027db0 osd=2 3 2024-02-13T00:50:54.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.903+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4027db0 osd=2 3 2024-02-13T00:50:54.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.903+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 6 2024-02-13T00:50:54.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.903+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd4027db0 osd=2 3 2024-02-13T00:50:54.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.903+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 2 6 2024-02-13T00:50:54.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.903+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4027db0 osd=2 4 2024-02-13T00:50:54.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.903+0000 7fafd9d03700 10 client.27831.objecter _send_command 6 2024-02-13T00:50:54.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.903+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fafd411d2e0 con 0x7fafd411c220 2024-02-13T00:50:54.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fafd411d2e0 con 0x7fafd411c220 2024-02-13T00:50:54.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:54.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:54.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd4027db0 osd=2 4 2024-02-13T00:50:54.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 2 6 2024-02-13T00:50:54.909 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:50:54.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd9d03700 10 client.27831.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:54.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 7 2024-02-13T00:50:54.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6813/604971011 conn(0x7fafd4027990 0x7fafd4121ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:54.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4027820 osd=3 3 2024-02-13T00:50:54.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4027820 osd=3 3 2024-02-13T00:50:54.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4027820 osd=3 3 2024-02-13T00:50:54.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 7 2024-02-13T00:50:54.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd4027820 osd=3 3 2024-02-13T00:50:54.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 3 7 2024-02-13T00:50:54.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4027820 osd=3 4 2024-02-13T00:50:54.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd9d03700 10 client.27831.objecter _send_command 7 2024-02-13T00:50:54.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.904+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafd4124af0 con 0x7fafd4027990 2024-02-13T00:50:54.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.905+0000 7fafd3fff700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6813/604971011 conn(0x7fafd4027990 0x7fafd4121ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:54.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.905+0000 7fafd0ff9700 10 client.27831.objecter ms_handle_connect 0x7fafd4027990 2024-02-13T00:50:54.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.905+0000 7fafd3fff700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6813/604971011 conn(0x7fafd4027990 0x7fafd4121ea0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:54.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.905+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fafd4124af0 con 0x7fafd4027990 2024-02-13T00:50:54.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.905+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:50:54.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.905+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 7 = system:0 2024-02-13T00:50:54.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.905+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd4027820 osd=3 4 2024-02-13T00:50:54.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.905+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 3 7 2024-02-13T00:50:54.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.920+0000 7fafd9d03700 10 client.27831.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:54.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.920+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 8 2024-02-13T00:50:54.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.920+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4027820 osd=3 3 2024-02-13T00:50:54.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.920+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4027820 osd=3 3 2024-02-13T00:50:54.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.920+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4027820 osd=3 3 2024-02-13T00:50:54.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.920+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 8 2024-02-13T00:50:54.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.920+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd4027820 osd=3 3 2024-02-13T00:50:54.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.920+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 3 8 2024-02-13T00:50:54.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.920+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4027820 osd=3 4 2024-02-13T00:50:54.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.920+0000 7fafd9d03700 10 client.27831.objecter _send_command 8 2024-02-13T00:50:54.925 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.920+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fafd411cd50 con 0x7fafd4027990 2024-02-13T00:50:54.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fafd411cd50 con 0x7fafd4027990 2024-02-13T00:50:54.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:54.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:54.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd4027820 osd=3 4 2024-02-13T00:50:54.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 3 8 2024-02-13T00:50:54.927 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:50:54.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd9d03700 10 client.27831.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:54.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 9 2024-02-13T00:50:54.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6800/3344263388 conn(0x7fafd411cec0 0x7fafd4123510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:54.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd411cd50 osd=4 3 2024-02-13T00:50:54.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd411cd50 osd=4 3 2024-02-13T00:50:54.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd411cd50 osd=4 3 2024-02-13T00:50:54.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 9 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd411cd50 osd=4 3 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 4 9 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd411cd50 osd=4 4 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd9d03700 10 client.27831.objecter _send_command 9 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.922+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafd411be20 con 0x7fafd411cec0 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.923+0000 7fafd37fe700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6800/3344263388 conn(0x7fafd411cec0 0x7fafd4123510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.923+0000 7fafd0ff9700 10 client.27831.objecter ms_handle_connect 0x7fafd411cec0 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.923+0000 7fafd37fe700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6800/3344263388 conn(0x7fafd411cec0 0x7fafd4123510 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.924+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fafd411be20 con 0x7fafd411cec0 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.924+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.924+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 9 = system:0 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.924+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd411cd50 osd=4 4 2024-02-13T00:50:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.924+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 4 9 2024-02-13T00:50:54.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.938+0000 7fafd9d03700 10 client.27831.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:54.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.938+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 10 2024-02-13T00:50:54.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.938+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd411cd50 osd=4 3 2024-02-13T00:50:54.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.938+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd411cd50 osd=4 3 2024-02-13T00:50:54.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.938+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd411cd50 osd=4 3 2024-02-13T00:50:54.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.938+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 10 2024-02-13T00:50:54.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.938+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd411cd50 osd=4 3 2024-02-13T00:50:54.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.938+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 4 10 2024-02-13T00:50:54.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.938+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd411cd50 osd=4 4 2024-02-13T00:50:54.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.938+0000 7fafd9d03700 10 client.27831.objecter _send_command 10 2024-02-13T00:50:54.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.938+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fafd411be20 con 0x7fafd411cec0 2024-02-13T00:50:54.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fafd411be20 con 0x7fafd411cec0 2024-02-13T00:50:54.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:54.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:54.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd411cd50 osd=4 4 2024-02-13T00:50:54.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 4 10 2024-02-13T00:50:54.945 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:50:54.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd9d03700 10 client.27831.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:54.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 11 2024-02-13T00:50:54.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6804/2277917372 conn(0x7fafd4123f10 0x7fafd4124370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:54.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd411be20 osd=5 3 2024-02-13T00:50:54.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd411be20 osd=5 3 2024-02-13T00:50:54.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd411be20 osd=5 3 2024-02-13T00:50:54.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 11 2024-02-13T00:50:54.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd411be20 osd=5 3 2024-02-13T00:50:54.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 5 11 2024-02-13T00:50:54.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd411be20 osd=5 4 2024-02-13T00:50:54.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd9d03700 10 client.27831.objecter _send_command 11 2024-02-13T00:50:54.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.940+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafd4124890 con 0x7fafd4123f10 2024-02-13T00:50:54.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.941+0000 7fafd2ffd700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6804/2277917372 conn(0x7fafd4123f10 0x7fafd4124370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:54.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.941+0000 7fafd0ff9700 10 client.27831.objecter ms_handle_connect 0x7fafd4123f10 2024-02-13T00:50:54.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.941+0000 7fafd2ffd700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6804/2277917372 conn(0x7fafd4123f10 0x7fafd4124370 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:54.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.942+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fafd4124890 con 0x7fafd4123f10 2024-02-13T00:50:54.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.942+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:50:54.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.942+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 11 = system:0 2024-02-13T00:50:54.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.942+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd411be20 osd=5 4 2024-02-13T00:50:54.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.942+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 5 11 2024-02-13T00:50:54.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.956+0000 7fafd9d03700 10 client.27831.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:54.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.956+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 12 2024-02-13T00:50:54.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.956+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd411be20 osd=5 3 2024-02-13T00:50:54.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.956+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd411be20 osd=5 3 2024-02-13T00:50:54.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.956+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd411be20 osd=5 3 2024-02-13T00:50:54.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.956+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 12 2024-02-13T00:50:54.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.956+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd411be20 osd=5 3 2024-02-13T00:50:54.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.956+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 5 12 2024-02-13T00:50:54.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.956+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd411be20 osd=5 4 2024-02-13T00:50:54.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.956+0000 7fafd9d03700 10 client.27831.objecter _send_command 12 2024-02-13T00:50:54.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.956+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fafd4124890 con 0x7fafd4123f10 2024-02-13T00:50:54.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.958+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fafd4124890 con 0x7fafd4123f10 2024-02-13T00:50:54.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.958+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:54.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.958+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:54.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.958+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd411be20 osd=5 4 2024-02-13T00:50:54.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.958+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 5 12 2024-02-13T00:50:54.964 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:50:54.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd9d03700 10 client.27831.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:54.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 13 2024-02-13T00:50:54.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6808/1506640341 conn(0x7fafd4129190 0x7fafd41295f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:54.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4124890 osd=6 3 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4124890 osd=6 3 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4124890 osd=6 3 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 13 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd4124890 osd=6 3 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 6 13 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4124890 osd=6 4 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd9d03700 10 client.27831.objecter _send_command 13 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafd412b240 con 0x7fafd4129190 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd3fff700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6808/1506640341 conn(0x7fafd4129190 0x7fafd41295f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd0ff9700 10 client.27831.objecter ms_handle_connect 0x7fafd4129190 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.959+0000 7fafd3fff700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6808/1506640341 conn(0x7fafd4129190 0x7fafd41295f0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.960+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fafd412b240 con 0x7fafd4129190 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.960+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:50:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.960+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 13 = system:0 2024-02-13T00:50:54.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.960+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd4124890 osd=6 4 2024-02-13T00:50:54.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.960+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 6 13 2024-02-13T00:50:54.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.974+0000 7fafd9d03700 10 client.27831.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:54.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.974+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 14 2024-02-13T00:50:54.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.974+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4124890 osd=6 3 2024-02-13T00:50:54.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.974+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4124890 osd=6 3 2024-02-13T00:50:54.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.974+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd4124890 osd=6 3 2024-02-13T00:50:54.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.974+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 14 2024-02-13T00:50:54.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.974+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd4124890 osd=6 3 2024-02-13T00:50:54.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.974+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 6 14 2024-02-13T00:50:54.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.974+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4124890 osd=6 4 2024-02-13T00:50:54.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.974+0000 7fafd9d03700 10 client.27831.objecter _send_command 14 2024-02-13T00:50:54.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.974+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fafd412b240 con 0x7fafd4129190 2024-02-13T00:50:54.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.976+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fafd412b240 con 0x7fafd4129190 2024-02-13T00:50:54.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.976+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:54.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.976+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:54.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.976+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd4124890 osd=6 4 2024-02-13T00:50:54.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.976+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 6 14 2024-02-13T00:50:54.981 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:50:54.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd9d03700 10 client.27831.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:50:54.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 15 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6812/1964192772 conn(0x7fafd412d970 0x7fafd412ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd412b240 osd=7 3 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd412b240 osd=7 3 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd412b240 osd=7 3 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 15 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd412b240 osd=7 3 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 7 15 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd412b240 osd=7 4 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd9d03700 10 client.27831.objecter _send_command 15 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafd41317e0 con 0x7fafd412d970 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd37fe700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6812/1964192772 conn(0x7fafd412d970 0x7fafd412ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd0ff9700 10 client.27831.objecter ms_handle_connect 0x7fafd412d970 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.977+0000 7fafd37fe700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6812/1964192772 conn(0x7fafd412d970 0x7fafd412ddd0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.978+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fafd41317e0 con 0x7fafd412d970 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.978+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:50:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.978+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 15 = system:0 2024-02-13T00:50:54.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.978+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd412b240 osd=7 4 2024-02-13T00:50:54.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.978+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 7 15 2024-02-13T00:50:54.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:54.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.992+0000 7fafd9d03700 10 client.27831.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:50:54.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.992+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign -1 16 2024-02-13T00:50:54.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.992+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd412b240 osd=7 3 2024-02-13T00:50:54.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.992+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd412b240 osd=7 3 2024-02-13T00:50:54.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.992+0000 7fafd9d03700 20 client.27831.objecter _get_session s=0x7fafd412b240 osd=7 3 2024-02-13T00:50:54.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.992+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_remove -1 16 2024-02-13T00:50:54.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.992+0000 7fafd9d03700 20 client.27831.objecter get_session s=0x7fafd412b240 osd=7 3 2024-02-13T00:50:54.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.992+0000 7fafd9d03700 15 client.27831.objecter _session_command_op_assign 7 16 2024-02-13T00:50:54.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.992+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd412b240 osd=7 4 2024-02-13T00:50:54.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.992+0000 7fafd9d03700 10 client.27831.objecter _send_command 16 2024-02-13T00:50:54.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.992+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fafd4129ba0 con 0x7fafd412d970 2024-02-13T00:50:54.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd0ff9700 1 -- 172.21.15.162:0/1726523762 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fafd4129ba0 con 0x7fafd412d970 2024-02-13T00:50:54.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd0ff9700 10 client.27831.objecter ms_dispatch 0x7fafd4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:50:54.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd0ff9700 10 client.27831.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:50:54.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd0ff9700 20 client.27831.objecter put_session s=0x7fafd412b240 osd=7 4 2024-02-13T00:50:54.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd0ff9700 15 client.27831.objecter _session_command_op_remove 7 16 2024-02-13T00:50:54.999 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:50:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 10 librados: watch_flush enter 2024-02-13T00:50:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 10 librados: watch_flush exit 2024-02-13T00:50:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 10 client.27831.objecter close_session for osd.0 2024-02-13T00:50:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6801/2103743338 conn(0x7fafd40fc0b0 msgr2=0x7fafd40fc4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6801/2103743338 conn(0x7fafd40fc0b0 0x7fafd40fc4d0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd40475e0 osd=0 1 2024-02-13T00:50:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 10 client.27831.objecter close_session for osd.1 2024-02-13T00:50:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6805/150599780 conn(0x7fafd4059990 msgr2=0x7fafd4059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6805/150599780 conn(0x7fafd4059990 0x7fafd4059d70 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4028760 osd=1 1 2024-02-13T00:50:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 10 client.27831.objecter close_session for osd.2 2024-02-13T00:50:55.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6809/3689719376 conn(0x7fafd411c220 msgr2=0x7fafd411c620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6809/3689719376 conn(0x7fafd411c220 0x7fafd411c620 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4027db0 osd=2 1 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 10 client.27831.objecter close_session for osd.3 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd3fff700 1 -- 172.21.15.162:0/1726523762 reap_dead start 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.994+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6813/604971011 conn(0x7fafd4027990 msgr2=0x7fafd4121ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6813/604971011 conn(0x7fafd4027990 0x7fafd4121ea0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4027820 osd=3 1 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 10 client.27831.objecter close_session for osd.4 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6800/3344263388 conn(0x7fafd411cec0 msgr2=0x7fafd4123510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6800/3344263388 conn(0x7fafd411cec0 0x7fafd4123510 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd411cd50 osd=4 1 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 10 client.27831.objecter close_session for osd.5 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6804/2277917372 conn(0x7fafd4123f10 msgr2=0x7fafd4124370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6804/2277917372 conn(0x7fafd4123f10 0x7fafd4124370 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd411be20 osd=5 1 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 10 client.27831.objecter close_session for osd.6 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6808/1506640341 conn(0x7fafd4129190 msgr2=0x7fafd41295f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6808/1506640341 conn(0x7fafd4129190 0x7fafd41295f0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd4124890 osd=6 1 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 10 client.27831.objecter close_session for osd.7 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6812/1964192772 conn(0x7fafd412d970 msgr2=0x7fafd412ddd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.181:6812/1964192772 conn(0x7fafd412d970 0x7fafd412ddd0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 20 client.27831.objecter put_session s=0x7fafd412b240 osd=7 1 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 20 client.27831.objecter shutdown clearing up homeless session... 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 10 client.27831.objecter successfully canceled tick 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd3fff700 1 -- 172.21.15.162:0/1726523762 reap_dead start 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6800/2866534053 conn(0x7fafc0062c90 msgr2=0x7fafc0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6800/2866534053 conn(0x7fafc0062c90 0x7fafc0065150 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 rx=0x7fafc4001530 tx=0x7fafc4004c50).stop 2024-02-13T00:50:55.003 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd40f9570 msgr2=0x7fafd40f70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd40f9570 0x7fafd40f70b0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 rx=0x7fafbc009eb0 tx=0x7fafbc00b040).stop 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 shutdown_connections 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd40f9570 0x7fafd40f70b0 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 --2- 172.21.15.162:0/1726523762 >> v2:172.21.15.162:6800/2866534053 conn(0x7fafc0062c90 0x7fafc0065150 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 >> 172.21.15.162:0/1726523762 conn(0x7fafd40f2720 msgr2=0x7fafd40f3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 shutdown_connections 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 -- 172.21.15.162:0/1726523762 wait complete. 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:54.995+0000 7fafd9d03700 1 librados: shutdown 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:50:55.004 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:50:55.005 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:50:55.025 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:55.045 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:55.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:55.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:55.177 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:55.196 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:55.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:55.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:55.328 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:55.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:54.498057+0000 mgr.y (mgr.14544) 539 : cluster [DBG] pgmap v948: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:55.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:54.518089+0000 mon.a (mon.0) 3173 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-108","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:50:55.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:54.518153+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e629: 8 total, 8 up, 4 in 2024-02-13T00:50:55.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:54.852970+0000 mon.a (mon.0) 3175 : audit [DBG] from='client.? 172.21.15.162:0/1726523762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:55.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:54.498057+0000 mgr.y (mgr.14544) 539 : cluster [DBG] pgmap v948: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:55.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:54.518089+0000 mon.a (mon.0) 3173 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-108","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:50:55.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:54.518153+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e629: 8 total, 8 up, 4 in 2024-02-13T00:50:55.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:54.852970+0000 mon.a (mon.0) 3175 : audit [DBG] from='client.? 172.21.15.162:0/1726523762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:55.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.587+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/4145003901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a780f8d10 msgr2=0x7f6a780f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.587+0000 7f6a7d6d3700 1 --2- 172.21.15.162:0/4145003901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a780f8d10 0x7f6a780f90f0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 rx=0x7f6a70004870 tx=0x7f6a70009cb0).stop 2024-02-13T00:50:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.587+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/4145003901 shutdown_connections 2024-02-13T00:50:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.587+0000 7f6a7d6d3700 1 --2- 172.21.15.162:0/4145003901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a78102270 0x7f6a78104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.587+0000 7f6a7d6d3700 1 --2- 172.21.15.162:0/4145003901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a780f8d10 0x7f6a780f90f0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.587+0000 7f6a7d6d3700 1 --2- 172.21.15.162:0/4145003901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a780f95e0 0x7f6a78101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.587+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/4145003901 >> 172.21.15.162:0/4145003901 conn(0x7f6a7806eb90 msgr2=0x7f6a780f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.587+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/4145003901 shutdown_connections 2024-02-13T00:50:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.587+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/4145003901 wait complete. 2024-02-13T00:50:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.587+0000 7f6a7d6d3700 1 librados: starting msgr at 2024-02-13T00:50:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.587+0000 7f6a7d6d3700 1 librados: starting objecter 2024-02-13T00:50:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.587+0000 7f6a7d6d3700 1 Processor -- start 2024-02-13T00:50:55.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a7d6d3700 1 -- start start 2024-02-13T00:50:55.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a7d6d3700 1 librados: setting wanted keys 2024-02-13T00:50:55.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a7d6d3700 1 librados: calling monclient init 2024-02-13T00:50:55.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a7d6d3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a780f8d10 0x7f6a7810e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:55.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a7d6d3700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a780f95e0 0x7f6a7810e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:55.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a7d6d3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a78102270 0x7f6a781083d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:55.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a7d6d3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a781088c0 con 0x7f6a780f8d10 2024-02-13T00:50:55.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a7d6d3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a78108a30 con 0x7f6a78102270 2024-02-13T00:50:55.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a7d6d3700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a78108bf0 con 0x7f6a780f95e0 2024-02-13T00:50:55.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a76ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a780f8d10 0x7f6a7810e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:55.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a76ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a780f8d10 0x7f6a7810e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44232/0 (socket says 172.21.15.162:44232) 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a76ffd700 1 -- 172.21.15.162:0/1247994401 learned_addr learned my addr 172.21.15.162:0/1247994401 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a767fc700 1 --2- 172.21.15.162:0/1247994401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a780f95e0 0x7f6a7810e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a777fe700 1 --2- 172.21.15.162:0/1247994401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a78102270 0x7f6a781083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a76ffd700 1 -- 172.21.15.162:0/1247994401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a780f95e0 msgr2=0x7f6a7810e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a76ffd700 1 --2- 172.21.15.162:0/1247994401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a780f95e0 0x7f6a7810e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a76ffd700 1 -- 172.21.15.162:0/1247994401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a78102270 msgr2=0x7f6a781083d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a76ffd700 1 --2- 172.21.15.162:0/1247994401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a78102270 0x7f6a781083d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a76ffd700 1 -- 172.21.15.162:0/1247994401 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60005040 con 0x7f6a780f8d10 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a76ffd700 1 --2- 172.21.15.162:0/1247994401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a780f8d10 0x7f6a7810e230 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f6a70009eb0 tx=0x7f6a7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a6ffff700 10 client.?.objecter ms_handle_connect 0x7f6a780f8d10 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.588+0000 7f6a6ffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.589+0000 7f6a6ffff700 1 -- 172.21.15.162:0/1247994401 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7000e070 con 0x7f6a780f8d10 2024-02-13T00:50:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.589+0000 7f6a6ffff700 1 -- 172.21.15.162:0/1247994401 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a70013da0 con 0x7f6a780f8d10 2024-02-13T00:50:55.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.589+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/1247994401 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a70005040 con 0x7f6a780f8d10 2024-02-13T00:50:55.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.589+0000 7f6a6ffff700 1 -- 172.21.15.162:0/1247994401 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a70019070 con 0x7f6a780f8d10 2024-02-13T00:50:55.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.589+0000 7f6a7d6d3700 10 client.27837.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:50:55.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.589+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/1247994401 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a78109090 con 0x7f6a780f8d10 2024-02-13T00:50:55.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.589+0000 7f6a7d6d3700 1 librados: init done 2024-02-13T00:50:55.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.590+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/1247994401 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a780475e0 con 0x7f6a780f8d10 2024-02-13T00:50:55.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.590+0000 7f6a6ffff700 1 -- 172.21.15.162:0/1247994401 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6a7001e420 con 0x7f6a780f8d10 2024-02-13T00:50:55.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.590+0000 7f6a6ffff700 1 --2- 172.21.15.162:0/1247994401 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6a64062c90 0x7f6a64065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:50:55.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.590+0000 7f6a6ffff700 1 -- 172.21.15.162:0/1247994401 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(630..630 src has 449..630) v4 ==== 8531+0+0 (secure 0 0 0) 0x7f6a7007ddc0 con 0x7f6a780f8d10 2024-02-13T00:50:55.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.590+0000 7f6a6ffff700 10 client.27837.objecter ms_dispatch 0x7f6a78001550 osd_map(630..630 src has 449..630) v4 2024-02-13T00:50:55.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.590+0000 7f6a6ffff700 3 client.27837.objecter handle_osd_map got epochs [630,630] > 0 2024-02-13T00:50:55.612 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.590+0000 7f6a6ffff700 3 client.27837.objecter handle_osd_map decoding full epoch 630 2024-02-13T00:50:55.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.590+0000 7f6a767fc700 1 --2- 172.21.15.162:0/1247994401 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6a64062c90 0x7f6a64065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:50:55.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.591+0000 7f6a6ffff700 20 client.27837.objecter dump_active .. 0 homeless 2024-02-13T00:50:55.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.591+0000 7f6a6ffff700 10 client.27837.objecter ms_handle_connect 0x7f6a64062c90 2024-02-13T00:50:55.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.591+0000 7f6a767fc700 1 --2- 172.21.15.162:0/1247994401 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6a64062c90 0x7f6a64065150 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 rx=0x7f6a78109b70 tx=0x7f6a6000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:50:55.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.595+0000 7f6a6ffff700 1 -- 172.21.15.162:0/1247994401 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6a70051130 con 0x7f6a780f8d10 2024-02-13T00:50:55.651 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:55.677 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:55.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:55.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:55.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:54.498057+0000 mgr.y (mgr.14544) 539 : cluster [DBG] pgmap v948: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:50:55.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:54.518089+0000 mon.a (mon.0) 3173 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-108","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:50:55.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:54.518153+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e629: 8 total, 8 up, 4 in 2024-02-13T00:50:55.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:54.852970+0000 mon.a (mon.0) 3175 : audit [DBG] from='client.? 172.21.15.162:0/1726523762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:50:55.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:55.797+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/1247994401 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "4.b", "id": [3, 6, 5]} v 0) v1 -- 0x7f6a78107170 con 0x7f6a780f8d10 2024-02-13T00:50:55.810 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:55.909 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:55.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:56.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:50:56.042 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:56.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:50:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:50:56.330 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:50:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:50:56] "GET /metrics HTTP/1.1" 200 189114 "" "Prometheus/2.33.4" 2024-02-13T00:50:56.470 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:50:56.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:56.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:50:56.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.533+0000 7f6a6ffff700 1 -- 172.21.15.162:0/1247994401 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [3, 6, 5]}]=0 set 4.b pg_upmap mapping to [3,6,5] v631) v1 ==== 127+0+0 (secure 0 0 0) 0x7f6a70050d20 con 0x7f6a780f8d10 2024-02-13T00:50:56.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 10 librados: watch_flush enter 2024-02-13T00:50:56.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 10 librados: watch_flush exit 2024-02-13T00:50:56.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 20 client.27837.objecter shutdown clearing up homeless session... 2024-02-13T00:50:56.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 10 client.27837.objecter successfully canceled tick 2024-02-13T00:50:56.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/1247994401 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6a64062c90 msgr2=0x7f6a64065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:56.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 --2- 172.21.15.162:0/1247994401 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6a64062c90 0x7f6a64065150 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 rx=0x7f6a78109b70 tx=0x7f6a6000b040).stop 2024-02-13T00:50:56.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/1247994401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a780f8d10 msgr2=0x7f6a7810e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:50:56.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 --2- 172.21.15.162:0/1247994401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a780f8d10 0x7f6a7810e230 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f6a70009eb0 tx=0x7f6a7000b040).stop 2024-02-13T00:50:56.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/1247994401 shutdown_connections 2024-02-13T00:50:56.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 --2- 172.21.15.162:0/1247994401 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6a64062c90 0x7f6a64065150 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:56.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 --2- 172.21.15.162:0/1247994401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a78102270 0x7f6a781083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:56.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 --2- 172.21.15.162:0/1247994401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a780f8d10 0x7f6a7810e230 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:56.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 --2- 172.21.15.162:0/1247994401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a780f95e0 0x7f6a7810e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:50:56.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/1247994401 >> 172.21.15.162:0/1247994401 conn(0x7f6a7806eb90 msgr2=0x7f6a78102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:50:56.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/1247994401 shutdown_connections 2024-02-13T00:50:56.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 -- 172.21.15.162:0/1247994401 wait complete. 2024-02-13T00:50:56.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:50:56.535+0000 7f6a7d6d3700 1 librados: shutdown 2024-02-13T00:50:56.542 INFO:teuthology.orchestra.run.smithi162.stderr:set 4.b pg_upmap mapping to [3,6,5] 2024-02-13T00:50:56.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:56.607 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:56.629 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:56.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:56.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:56.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:56.763 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:56.783 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:56.784 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:55.527389+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e630: 8 total, 8 up, 4 in 2024-02-13T00:50:56.784 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:55.536437+0000 mon.a (mon.0) 3177 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:50:56.784 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:55.799962+0000 mon.a (mon.0) 3178 : audit [INF] from='client.? 172.21.15.162:0/1247994401' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [3, 6, 5]}]: dispatch 2024-02-13T00:50:56.784 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:55.813318+0000 mgr.y (mgr.14544) 540 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:56.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:56.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:55.527389+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e630: 8 total, 8 up, 4 in 2024-02-13T00:50:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:55.536437+0000 mon.a (mon.0) 3177 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:50:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:55.799962+0000 mon.a (mon.0) 3178 : audit [INF] from='client.? 172.21.15.162:0/1247994401' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [3, 6, 5]}]: dispatch 2024-02-13T00:50:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:55.813318+0000 mgr.y (mgr.14544) 540 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:55.527389+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e630: 8 total, 8 up, 4 in 2024-02-13T00:50:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:55.536437+0000 mon.a (mon.0) 3177 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:50:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:55.799962+0000 mon.a (mon.0) 3178 : audit [INF] from='client.? 172.21.15.162:0/1247994401' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [3, 6, 5]}]: dispatch 2024-02-13T00:50:56.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:55.813318+0000 mgr.y (mgr.14544) 540 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:50:56.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:50:56.917 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:57.166 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:50:57.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:57.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:50:57.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:57.302 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:57.323 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:57.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:57.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:57.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:57.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:50:57.456 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:57.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:56.498473+0000 mgr.y (mgr.14544) 541 : cluster [DBG] pgmap v951: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 255 B/s wr, 0 op/s 2024-02-13T00:50:57.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:56.532982+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-108", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:50:57.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:56.533064+0000 mon.a (mon.0) 3180 : audit [INF] from='client.? 172.21.15.162:0/1247994401' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [3, 6, 5]}]': finished 2024-02-13T00:50:57.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:56.533114+0000 mon.a (mon.0) 3181 : cluster [DBG] osdmap e631: 8 total, 8 up, 4 in 2024-02-13T00:50:57.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:56.536113+0000 mon.a (mon.0) 3182 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-108"}]: dispatch 2024-02-13T00:50:57.740 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:56.498473+0000 mgr.y (mgr.14544) 541 : cluster [DBG] pgmap v951: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 255 B/s wr, 0 op/s 2024-02-13T00:50:57.741 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:56.532982+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-108", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:50:57.741 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:56.533064+0000 mon.a (mon.0) 3180 : audit [INF] from='client.? 172.21.15.162:0/1247994401' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [3, 6, 5]}]': finished 2024-02-13T00:50:57.741 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:56.533114+0000 mon.a (mon.0) 3181 : cluster [DBG] osdmap e631: 8 total, 8 up, 4 in 2024-02-13T00:50:57.741 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:56.536113+0000 mon.a (mon.0) 3182 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-108"}]: dispatch 2024-02-13T00:50:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:56.498473+0000 mgr.y (mgr.14544) 541 : cluster [DBG] pgmap v951: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 255 B/s wr, 0 op/s 2024-02-13T00:50:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:56.532982+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-108", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:50:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:56.533064+0000 mon.a (mon.0) 3180 : audit [INF] from='client.? 172.21.15.162:0/1247994401' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [3, 6, 5]}]': finished 2024-02-13T00:50:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:56.533114+0000 mon.a (mon.0) 3181 : cluster [DBG] osdmap e631: 8 total, 8 up, 4 in 2024-02-13T00:50:57.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:56.536113+0000 mon.a (mon.0) 3182 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-108"}]: dispatch 2024-02-13T00:50:58.110 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:58.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:58.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:50:58.249 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:58.269 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:50:58.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:58.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:50:58.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:50:58.404 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:58.424 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:50:58.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:58.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:50:58.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:58.556 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:58.576 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:58.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:58.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:58.698 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:50:58.698 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:50:58.700 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:50:58.707 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:57.537092+0000 mon.a (mon.0) 3183 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-108"}]': finished 2024-02-13T00:50:58.707 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:57.537160+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e632: 8 total, 8 up, 4 in 2024-02-13T00:50:58.707 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:57.539020+0000 mon.a (mon.0) 3185 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-108", "mode": "writeback"}]: dispatch 2024-02-13T00:50:58.707 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:58.532103+0000 mon.a (mon.0) 3186 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:50:58.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:58.538401+0000 mon.a (mon.0) 3187 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-108", "mode": "writeback"}]': finished 2024-02-13T00:50:58.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:58.538469+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e633: 8 total, 8 up, 4 in 2024-02-13T00:50:58.708 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:57.537092+0000 mon.a (mon.0) 3183 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-108"}]': finished 2024-02-13T00:50:58.708 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:57.537160+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e632: 8 total, 8 up, 4 in 2024-02-13T00:50:58.708 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:57.539020+0000 mon.a (mon.0) 3185 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-108", "mode": "writeback"}]: dispatch 2024-02-13T00:50:58.708 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:58.532103+0000 mon.a (mon.0) 3186 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:50:58.708 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:58.538401+0000 mon.a (mon.0) 3187 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-108", "mode": "writeback"}]': finished 2024-02-13T00:50:58.708 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:58.538469+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e633: 8 total, 8 up, 4 in 2024-02-13T00:50:58.711 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:58.731 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:58.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:58.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:58.861 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:50:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:50:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:50:58.861 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:57.537092+0000 mon.a (mon.0) 3183 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-108"}]': finished 2024-02-13T00:50:58.861 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:57.537160+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e632: 8 total, 8 up, 4 in 2024-02-13T00:50:58.861 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:57.539020+0000 mon.a (mon.0) 3185 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-108", "mode": "writeback"}]: dispatch 2024-02-13T00:50:58.861 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:58.532103+0000 mon.a (mon.0) 3186 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:50:58.861 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:58.538401+0000 mon.a (mon.0) 3187 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-108", "mode": "writeback"}]': finished 2024-02-13T00:50:58.861 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:58.538469+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e633: 8 total, 8 up, 4 in 2024-02-13T00:50:58.862 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:58.881 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:58.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:58.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:58.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:50:58.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:50:59.011 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:59.029 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:50:59.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:59.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:50:59.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:59.160 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:59.179 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:59.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:50:59.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:50:59.309 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:59.327 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:50:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:59.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:50:59.431 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:50:59.458 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:59.606 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:50:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:59.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:50:59.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:50:59.737 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:50:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:58.498900+0000 mgr.y (mgr.14544) 542 : cluster [DBG] pgmap v954: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:50:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:50:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:59.538401+0000 mon.a (mon.0) 3189 : cluster [DBG] osdmap e634: 8 total, 8 up, 4 in 2024-02-13T00:50:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:58.498900+0000 mgr.y (mgr.14544) 542 : cluster [DBG] pgmap v954: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:50:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:50:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:59.538401+0000 mon.a (mon.0) 3189 : cluster [DBG] osdmap e634: 8 total, 8 up, 4 in 2024-02-13T00:50:59.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:58.498900+0000 mgr.y (mgr.14544) 542 : cluster [DBG] pgmap v954: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:50:59.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:50:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:59.538401+0000 mon.a (mon.0) 3189 : cluster [DBG] osdmap e634: 8 total, 8 up, 4 in 2024-02-13T00:50:59.984 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:50:59.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:50:59.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:00.006 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-13T00:51:00.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:00.116 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:00.213 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:00.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:00.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:00.267 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:00.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:00.352 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:00.551 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:00.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:00.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:00.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:51:00.660 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:00.684 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:00.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:50:59.564008+0000 mon.a (mon.0) 3190 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:00.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:59.963055+0000 osd.4 (osd.4) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:00.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:50:59.963293+0000 osd.4 (osd.4) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:00.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:00.509580+0000 mon.a (mon.0) 3191 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:51:00.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:00.515082+0000 mon.a (mon.0) 3192 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:00.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:00.515119+0000 mon.a (mon.0) 3193 : cluster [DBG] osdmap e635: 8 total, 8 up, 4 in 2024-02-13T00:51:00.874 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:00.516102+0000 mon.a (mon.0) 3194 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-108"}]: dispatch 2024-02-13T00:51:00.874 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:50:59.564008+0000 mon.a (mon.0) 3190 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:00.874 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:59.963055+0000 osd.4 (osd.4) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:00.874 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:50:59.963293+0000 osd.4 (osd.4) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:00.874 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:00.509580+0000 mon.a (mon.0) 3191 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:51:00.874 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:00.515082+0000 mon.a (mon.0) 3192 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:00.874 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:00.515119+0000 mon.a (mon.0) 3193 : cluster [DBG] osdmap e635: 8 total, 8 up, 4 in 2024-02-13T00:51:00.874 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:00.516102+0000 mon.a (mon.0) 3194 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-108"}]: dispatch 2024-02-13T00:51:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:50:59.564008+0000 mon.a (mon.0) 3190 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:59.963055+0000 osd.4 (osd.4) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:50:59.963293+0000 osd.4 (osd.4) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:00.509580+0000 mon.a (mon.0) 3191 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:51:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:00.515082+0000 mon.a (mon.0) 3192 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:00.515119+0000 mon.a (mon.0) 3193 : cluster [DBG] osdmap e635: 8 total, 8 up, 4 in 2024-02-13T00:51:01.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:00.516102+0000 mon.a (mon.0) 3194 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-108"}]: dispatch 2024-02-13T00:51:01.634 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:51:01.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:01.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:51:01.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.342+0000 7f20c762a700 1 -- 172.21.15.162:0/716562523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20c00f84c0 msgr2=0x7f20c0104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:01.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.342+0000 7f20c762a700 1 --2- 172.21.15.162:0/716562523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20c00f84c0 0x7f20c0104ac0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 rx=0x7f20b8009c30 tx=0x7f20b8004c50).stop 2024-02-13T00:51:01.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.342+0000 7f20c762a700 1 -- 172.21.15.162:0/716562523 shutdown_connections 2024-02-13T00:51:01.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.342+0000 7f20c762a700 1 --2- 172.21.15.162:0/716562523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20c00f84c0 0x7f20c0104ac0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:01.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.342+0000 7f20c762a700 1 --2- 172.21.15.162:0/716562523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c00fdc50 0x7f20c00fe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:01.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.342+0000 7f20c762a700 1 --2- 172.21.15.162:0/716562523 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20c00fe5b0 0x7f20c00f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:01.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.342+0000 7f20c762a700 1 -- 172.21.15.162:0/716562523 >> 172.21.15.162:0/716562523 conn(0x7f20c00f38a0 msgr2=0x7f20c00f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:01.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.343+0000 7f20c762a700 1 -- 172.21.15.162:0/716562523 shutdown_connections 2024-02-13T00:51:01.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.343+0000 7f20c762a700 1 -- 172.21.15.162:0/716562523 wait complete. 2024-02-13T00:51:01.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.343+0000 7f20c762a700 1 librados: starting msgr at 2024-02-13T00:51:01.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.343+0000 7f20c762a700 1 librados: starting objecter 2024-02-13T00:51:01.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.343+0000 7f20c762a700 1 Processor -- start 2024-02-13T00:51:01.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.343+0000 7f20c762a700 1 -- start start 2024-02-13T00:51:01.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.343+0000 7f20c762a700 1 librados: setting wanted keys 2024-02-13T00:51:01.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.343+0000 7f20c762a700 1 librados: calling monclient init 2024-02-13T00:51:01.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c762a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20c00f84c0 0x7f20c010df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:01.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c762a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c00fdc50 0x7f20c010e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:01.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c762a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20c00fe5b0 0x7f20c01080d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:01.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c762a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c01085c0 con 0x7f20c00f84c0 2024-02-13T00:51:01.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c762a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c0108730 con 0x7f20c00fdc50 2024-02-13T00:51:01.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c762a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c01088f0 con 0x7f20c00fe5b0 2024-02-13T00:51:01.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c53c6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20c00f84c0 0x7f20c010df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:01.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c53c6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20c00f84c0 0x7f20c010df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44256/0 (socket says 172.21.15.162:44256) 2024-02-13T00:51:01.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c53c6700 1 -- 172.21.15.162:0/518470681 learned_addr learned my addr 172.21.15.162:0/518470681 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:01.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c5bc7700 1 --2- 172.21.15.162:0/518470681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20c00fe5b0 0x7f20c01080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:01.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c4bc5700 1 --2- 172.21.15.162:0/518470681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c00fdc50 0x7f20c010e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:01.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c53c6700 1 -- 172.21.15.162:0/518470681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20c00fe5b0 msgr2=0x7f20c01080d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:01.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c53c6700 1 --2- 172.21.15.162:0/518470681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20c00fe5b0 0x7f20c01080d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:01.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c53c6700 1 -- 172.21.15.162:0/518470681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c00fdc50 msgr2=0x7f20c010e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:01.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c53c6700 1 --2- 172.21.15.162:0/518470681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c00fdc50 0x7f20c010e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:01.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c53c6700 1 -- 172.21.15.162:0/518470681 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20bc005040 con 0x7f20c00f84c0 2024-02-13T00:51:01.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c5bc7700 1 --2- 172.21.15.162:0/518470681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20c00fe5b0 0x7f20c01080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:51:01.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c53c6700 1 --2- 172.21.15.162:0/518470681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20c00f84c0 0x7f20c010df30 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 rx=0x7f20b40041d0 tx=0x7f20b4009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:01.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20b27fc700 10 client.?.objecter ms_handle_connect 0x7f20c00f84c0 2024-02-13T00:51:01.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20b27fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:01.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20b27fc700 1 -- 172.21.15.162:0/518470681 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b4007b40 con 0x7f20c00f84c0 2024-02-13T00:51:01.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20b27fc700 1 -- 172.21.15.162:0/518470681 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20b400f040 con 0x7f20c00f84c0 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20b27fc700 1 -- 172.21.15.162:0/518470681 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b4014810 con 0x7f20c00f84c0 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c762a700 1 -- 172.21.15.162:0/518470681 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b80097c0 con 0x7f20c00f84c0 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c762a700 10 client.27843.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c762a700 1 -- 172.21.15.162:0/518470681 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c0108f30 con 0x7f20c00f84c0 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.344+0000 7f20c762a700 1 librados: init done 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.345+0000 7f20c762a700 1 -- 172.21.15.162:0/518470681 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c01071b0 con 0x7f20c00f84c0 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.346+0000 7f20b27fc700 1 -- 172.21.15.162:0/518470681 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f20b4009110 con 0x7f20c00f84c0 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.346+0000 7f20b27fc700 1 --2- 172.21.15.162:0/518470681 >> v2:172.21.15.162:6800/2866534053 conn(0x7f20ac062c90 0x7f20ac065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.346+0000 7f20b27fc700 1 -- 172.21.15.162:0/518470681 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(635..635 src has 449..635) v4 ==== 8615+0+0 (secure 0 0 0) 0x7f20b407d3e0 con 0x7f20c00f84c0 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.346+0000 7f20b27fc700 10 client.27843.objecter ms_dispatch 0x7f20c0001550 osd_map(635..635 src has 449..635) v4 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.346+0000 7f20b27fc700 3 client.27843.objecter handle_osd_map got epochs [635,635] > 0 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.346+0000 7f20b27fc700 3 client.27843.objecter handle_osd_map decoding full epoch 635 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.346+0000 7f20c4bc5700 1 --2- 172.21.15.162:0/518470681 >> v2:172.21.15.162:6800/2866534053 conn(0x7f20ac062c90 0x7f20ac065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.346+0000 7f20b27fc700 20 client.27843.objecter dump_active .. 0 homeless 2024-02-13T00:51:01.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.347+0000 7f20b27fc700 10 client.27843.objecter ms_handle_connect 0x7f20ac062c90 2024-02-13T00:51:01.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.347+0000 7f20c4bc5700 1 --2- 172.21.15.162:0/518470681 >> v2:172.21.15.162:6800/2866534053 conn(0x7f20ac062c90 0x7f20ac065150 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f20c0109870 tx=0x7f20bc00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:01.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.350+0000 7f20b27fc700 1 -- 172.21.15.162:0/518470681 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f20b4050680 con 0x7f20c00f84c0 2024-02-13T00:51:01.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:01.534+0000 7f20c762a700 1 -- 172.21.15.162:0/518470681 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f20c005fc10 con 0x7f20c00f84c0 2024-02-13T00:51:01.740 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:01.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:00.499798+0000 mgr.y (mgr.14544) 543 : cluster [DBG] pgmap v957: 204 pgs: 1 active+clean+snaptrim, 203 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:01.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:01.512304+0000 mon.a (mon.0) 3195 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:01.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:01.517123+0000 mon.a (mon.0) 3196 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-108"}]': finished 2024-02-13T00:51:01.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:01.517193+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e636: 8 total, 8 up, 4 in 2024-02-13T00:51:01.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:01.535709+0000 mon.a (mon.0) 3198 : audit [INF] from='client.? 172.21.15.162:0/518470681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:51:01.769 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:00.499798+0000 mgr.y (mgr.14544) 543 : cluster [DBG] pgmap v957: 204 pgs: 1 active+clean+snaptrim, 203 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:01.770 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:01.512304+0000 mon.a (mon.0) 3195 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:01.770 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:01.517123+0000 mon.a (mon.0) 3196 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-108"}]': finished 2024-02-13T00:51:01.770 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:01.517193+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e636: 8 total, 8 up, 4 in 2024-02-13T00:51:01.770 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:01.535709+0000 mon.a (mon.0) 3198 : audit [INF] from='client.? 172.21.15.162:0/518470681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:51:01.771 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:01.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:00.499798+0000 mgr.y (mgr.14544) 543 : cluster [DBG] pgmap v957: 204 pgs: 1 active+clean+snaptrim, 203 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:01.512304+0000 mon.a (mon.0) 3195 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:01.517123+0000 mon.a (mon.0) 3196 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-108"}]': finished 2024-02-13T00:51:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:01.517193+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e636: 8 total, 8 up, 4 in 2024-02-13T00:51:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:01.535709+0000 mon.a (mon.0) 3198 : audit [INF] from='client.? 172.21.15.162:0/518470681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:51:02.173 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:51:02.173 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:51:02.220 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:02.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:02.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:02.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:02.353 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:02.391 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:02.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:02.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:02.522 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:02.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.518+0000 7f20b27fc700 1 -- 172.21.15.162:0/518470681 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v637) v1 ==== 91+0+0 (secure 0 0 0) 0x7f20b4050270 con 0x7f20c00f84c0 2024-02-13T00:51:02.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 10 librados: watch_flush enter 2024-02-13T00:51:02.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 10 librados: watch_flush exit 2024-02-13T00:51:02.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 20 client.27843.objecter shutdown clearing up homeless session... 2024-02-13T00:51:02.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 10 client.27843.objecter successfully canceled tick 2024-02-13T00:51:02.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 1 -- 172.21.15.162:0/518470681 >> v2:172.21.15.162:6800/2866534053 conn(0x7f20ac062c90 msgr2=0x7f20ac065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:02.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 1 --2- 172.21.15.162:0/518470681 >> v2:172.21.15.162:6800/2866534053 conn(0x7f20ac062c90 0x7f20ac065150 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f20c0109870 tx=0x7f20bc00b040).stop 2024-02-13T00:51:02.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 1 -- 172.21.15.162:0/518470681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20c00f84c0 msgr2=0x7f20c010df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:02.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 1 --2- 172.21.15.162:0/518470681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20c00f84c0 0x7f20c010df30 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 rx=0x7f20b40041d0 tx=0x7f20b4009f60).stop 2024-02-13T00:51:02.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 1 -- 172.21.15.162:0/518470681 shutdown_connections 2024-02-13T00:51:02.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 1 --2- 172.21.15.162:0/518470681 >> v2:172.21.15.162:6800/2866534053 conn(0x7f20ac062c90 0x7f20ac065150 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:02.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 1 --2- 172.21.15.162:0/518470681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20c00fe5b0 0x7f20c01080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:02.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 1 --2- 172.21.15.162:0/518470681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20c00f84c0 0x7f20c010df30 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:02.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 1 --2- 172.21.15.162:0/518470681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c00fdc50 0x7f20c010e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:02.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.521+0000 7f20c762a700 1 -- 172.21.15.162:0/518470681 >> 172.21.15.162:0/518470681 conn(0x7f20c00f38a0 msgr2=0x7f20c00f50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:02.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.523+0000 7f20c762a700 1 -- 172.21.15.162:0/518470681 shutdown_connections 2024-02-13T00:51:02.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.523+0000 7f20c762a700 1 -- 172.21.15.162:0/518470681 wait complete. 2024-02-13T00:51:02.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.523+0000 7f20c762a700 1 librados: shutdown 2024-02-13T00:51:02.528 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:51:02.899 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:02.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:02.919 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-13T00:51:03.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:03.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:02.513821+0000 mon.a (mon.0) 3199 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:03.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:02.517808+0000 mon.a (mon.0) 3200 : audit [INF] from='client.? 172.21.15.162:0/518470681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:51:03.032 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:02.517887+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e637: 8 total, 8 up, 4 in 2024-02-13T00:51:03.032 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:02.513821+0000 mon.a (mon.0) 3199 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:03.032 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:02.517808+0000 mon.a (mon.0) 3200 : audit [INF] from='client.? 172.21.15.162:0/518470681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:51:03.032 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:02.517887+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e637: 8 total, 8 up, 4 in 2024-02-13T00:51:03.034 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:03.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:02.513821+0000 mon.a (mon.0) 3199 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:03.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:02.517808+0000 mon.a (mon.0) 3200 : audit [INF] from='client.? 172.21.15.162:0/518470681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:51:03.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:02.517887+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e637: 8 total, 8 up, 4 in 2024-02-13T00:51:03.222 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:03.382 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:03.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:03.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:03.437 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6] out_osds: [3, 2, 7, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-13T00:51:03.437 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:51:03.437 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-13T00:51:03.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-13T00:51:03.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:03.512 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:03.514 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:51:03.514 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:51:03.514 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:51:03.514 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:51:03.514 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:51:03.514 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:51:03.514 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:51:03.514 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:51:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.784+0000 7fec69797700 1 -- 172.21.15.162:0/282371268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec640f9b70 msgr2=0x7fec640f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.784+0000 7fec69797700 1 --2- 172.21.15.162:0/282371268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec640f9b70 0x7fec640f9f50 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 rx=0x7fec54009be0 tx=0x7fec54004d10).stop 2024-02-13T00:51:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.784+0000 7fec69797700 1 -- 172.21.15.162:0/282371268 shutdown_connections 2024-02-13T00:51:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.784+0000 7fec69797700 1 --2- 172.21.15.162:0/282371268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec640f8500 0x7fec64104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.784+0000 7fec69797700 1 --2- 172.21.15.162:0/282371268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec640f9b70 0x7fec640f9f50 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.784+0000 7fec69797700 1 --2- 172.21.15.162:0/282371268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec640fa440 0x7fec640f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.784+0000 7fec69797700 1 -- 172.21.15.162:0/282371268 >> 172.21.15.162:0/282371268 conn(0x7fec640f38c0 msgr2=0x7fec640f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.784+0000 7fec69797700 1 -- 172.21.15.162:0/282371268 shutdown_connections 2024-02-13T00:51:03.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.784+0000 7fec69797700 1 -- 172.21.15.162:0/282371268 wait complete. 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.784+0000 7fec69797700 1 librados: starting msgr at 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.784+0000 7fec69797700 1 librados: starting objecter 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.784+0000 7fec69797700 1 Processor -- start 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec69797700 1 -- start start 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec69797700 1 librados: setting wanted keys 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec69797700 1 librados: calling monclient init 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec69797700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec640f8500 0x7fec6410e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec69797700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec640f9b70 0x7fec6410e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec69797700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec640fa440 0x7fec641084c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec69797700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec641089b0 con 0x7fec640f9b70 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec69797700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec64108b20 con 0x7fec640f8500 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec69797700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec64108ce0 con 0x7fec640fa440 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec637fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec640fa440 0x7fec641084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:03.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec627fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec640f9b70 0x7fec6410e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec627fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec640f9b70 0x7fec6410e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44274/0 (socket says 172.21.15.162:44274) 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec637fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec640fa440 0x7fec641084c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:39046/0 (socket says 172.21.15.162:39046) 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec627fc700 1 -- 172.21.15.162:0/386879398 learned_addr learned my addr 172.21.15.162:0/386879398 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec62ffd700 1 --2- 172.21.15.162:0/386879398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec640f8500 0x7fec6410e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec627fc700 1 -- 172.21.15.162:0/386879398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec640fa440 msgr2=0x7fec641084c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec627fc700 1 --2- 172.21.15.162:0/386879398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec640fa440 0x7fec641084c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec627fc700 1 -- 172.21.15.162:0/386879398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec640f8500 msgr2=0x7fec6410e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec627fc700 1 --2- 172.21.15.162:0/386879398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec640f8500 0x7fec6410e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec627fc700 1 -- 172.21.15.162:0/386879398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c005040 con 0x7fec640f9b70 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec637fe700 1 --2- 172.21.15.162:0/386879398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec640fa440 0x7fec641084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec627fc700 1 --2- 172.21.15.162:0/386879398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec640f9b70 0x7fec6410e7c0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 rx=0x7fec4c0048f0 tx=0x7fec4c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec5bfff700 10 client.?.objecter ms_handle_connect 0x7fec640f9b70 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec5bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c007b00 con 0x7fec640f9b70 2024-02-13T00:51:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.785+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec540097c0 con 0x7fec640f9b70 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.786+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec4c00f040 con 0x7fec640f9b70 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.786+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c014990 con 0x7fec640f9b70 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.786+0000 7fec69797700 10 client.27849.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.786+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec64109320 con 0x7fec640f9b70 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.786+0000 7fec69797700 1 librados: init done 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.786+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fec640475e0 con 0x7fec640f9b70 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.787+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fec4c010500 con 0x7fec640f9b70 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.787+0000 7fec5bfff700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6800/2866534053 conn(0x7fec50062c90 0x7fec50065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.787+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(637..637 src has 449..637) v4 ==== 8206+0+0 (secure 0 0 0) 0x7fec4c07d0a0 con 0x7fec640f9b70 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.787+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 osd_map(637..637 src has 449..637) v4 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.787+0000 7fec5bfff700 3 client.27849.objecter handle_osd_map got epochs [637,637] > 0 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.787+0000 7fec5bfff700 3 client.27849.objecter handle_osd_map decoding full epoch 637 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.787+0000 7fec62ffd700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6800/2866534053 conn(0x7fec50062c90 0x7fec50065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.787+0000 7fec5bfff700 20 client.27849.objecter dump_active .. 0 homeless 2024-02-13T00:51:03.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.787+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v637) v1 ==== 54+0+16 (secure 0 0 0) 0x7fec4c07d540 con 0x7fec640f9b70 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec69797700 10 client.27849.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 1 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec5bfff700 10 client.27849.objecter ms_handle_connect 0x7fec50062c90 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec62ffd700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6800/2866534053 conn(0x7fec50062c90 0x7fec50065150 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 rx=0x7fec5400b3b0 tx=0x7fec54004d10).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6801/2103743338 conn(0x7fec64028590 0x7fec640fea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64107280 osd=0 3 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64107280 osd=0 3 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64107280 osd=0 3 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 1 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64107280 osd=0 3 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 0 1 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64107280 osd=0 4 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec69797700 10 client.27849.objecter _send_command 1 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec641000d0 con 0x7fec64028590 2024-02-13T00:51:03.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec637fe700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6801/2103743338 conn(0x7fec64028590 0x7fec640fea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec637fe700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6801/2103743338 conn(0x7fec64028590 0x7fec640fea60 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.788+0000 7fec5bfff700 10 client.27849.objecter ms_handle_connect 0x7fec64028590 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.789+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fec641000d0 con 0x7fec64028590 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.789+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.789+0000 7fec5bfff700 10 client.27849.objecter _finish_command 1 = system:0 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.789+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64107280 osd=0 4 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.789+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 0 1 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.803+0000 7fec69797700 10 client.27849.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.803+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 2 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.803+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64107280 osd=0 3 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.803+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64107280 osd=0 3 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.803+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64107280 osd=0 3 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.803+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 2 2024-02-13T00:51:03.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.803+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64107280 osd=0 3 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.803+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 0 2 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.803+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64107280 osd=0 4 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.803+0000 7fec69797700 10 client.27849.objecter _send_command 2 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.803+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fec6405a0e0 con 0x7fec64028590 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.804+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fec6405a0e0 con 0x7fec64028590 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.804+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.804+0000 7fec5bfff700 10 client.27849.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.804+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64107280 osd=0 4 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.804+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 0 2 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec69797700 10 client.27849.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 3 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6805/150599780 conn(0x7fec64059570 0x7fec64119210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64059400 osd=1 3 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64059400 osd=1 3 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64059400 osd=1 3 2024-02-13T00:51:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 3 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64059400 osd=1 3 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 1 3 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64059400 osd=1 4 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec69797700 10 client.27849.objecter _send_command 3 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec640280f0 con 0x7fec64059570 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec62ffd700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6805/150599780 conn(0x7fec64059570 0x7fec64119210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec5bfff700 10 client.27849.objecter ms_handle_connect 0x7fec64059570 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.805+0000 7fec62ffd700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6805/150599780 conn(0x7fec64059570 0x7fec64119210 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.806+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fec640280f0 con 0x7fec64059570 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.806+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.806+0000 7fec5bfff700 10 client.27849.objecter _finish_command 3 = system:0 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.806+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64059400 osd=1 4 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.806+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 1 3 2024-02-13T00:51:03.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.820+0000 7fec69797700 10 client.27849.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.820+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 4 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.820+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64059400 osd=1 3 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.820+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64059400 osd=1 3 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.820+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64059400 osd=1 3 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.820+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 4 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.820+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64059400 osd=1 3 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.820+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 1 4 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.820+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64059400 osd=1 4 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.820+0000 7fec69797700 10 client.27849.objecter _send_command 4 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.820+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fec64119700 con 0x7fec64059570 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.821+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fec64119700 con 0x7fec64059570 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec5bfff700 10 client.27849.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64059400 osd=1 4 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 1 4 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec69797700 10 client.27849.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 5 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6809/3689719376 conn(0x7fec640277c0 0x7fec6411ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64027650 osd=2 3 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64027650 osd=2 3 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64027650 osd=2 3 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 5 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64027650 osd=2 3 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 2 5 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64027650 osd=2 4 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec69797700 10 client.27849.objecter _send_command 5 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec640ffc30 con 0x7fec640277c0 2024-02-13T00:51:03.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec627fc700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6809/3689719376 conn(0x7fec640277c0 0x7fec6411ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec5bfff700 10 client.27849.objecter ms_handle_connect 0x7fec640277c0 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.822+0000 7fec627fc700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6809/3689719376 conn(0x7fec640277c0 0x7fec6411ca60 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.823+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fec640ffc30 con 0x7fec640277c0 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.823+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.823+0000 7fec5bfff700 10 client.27849.objecter _finish_command 5 = system:0 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.823+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64027650 osd=2 4 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.823+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 2 5 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.837+0000 7fec69797700 10 client.27849.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.837+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 6 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.837+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64027650 osd=2 3 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.837+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64027650 osd=2 3 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.837+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64027650 osd=2 3 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.837+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 6 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.837+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64027650 osd=2 3 2024-02-13T00:51:03.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.837+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 2 6 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.837+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64027650 osd=2 4 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.837+0000 7fec69797700 10 client.27849.objecter _send_command 6 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.837+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fec6411cf50 con 0x7fec640277c0 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fec6411cf50 con 0x7fec640277c0 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec5bfff700 10 client.27849.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64027650 osd=2 4 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 2 6 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec69797700 10 client.27849.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 7 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6813/604971011 conn(0x7fec640ff300 0x7fec64120250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec640ff190 osd=3 3 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec640ff190 osd=3 3 2024-02-13T00:51:03.526 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec640ff190 osd=3 3 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 7 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec640ff190 osd=3 3 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 3 7 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec640ff190 osd=3 4 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec69797700 10 client.27849.objecter _send_command 7 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec6411a550 con 0x7fec640ff300 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec637fe700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6813/604971011 conn(0x7fec640ff300 0x7fec64120250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec5bfff700 10 client.27849.objecter ms_handle_connect 0x7fec640ff300 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.839+0000 7fec637fe700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6813/604971011 conn(0x7fec640ff300 0x7fec64120250 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.840+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fec6411a550 con 0x7fec640ff300 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.840+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.840+0000 7fec5bfff700 10 client.27849.objecter _finish_command 7 = system:0 2024-02-13T00:51:03.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.840+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec640ff190 osd=3 4 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.840+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 3 7 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.854+0000 7fec69797700 10 client.27849.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.854+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 8 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.854+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec640ff190 osd=3 3 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.854+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec640ff190 osd=3 3 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.854+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec640ff190 osd=3 3 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.854+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 8 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.854+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec640ff190 osd=3 3 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.854+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 3 8 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.854+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec640ff190 osd=3 4 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.854+0000 7fec69797700 10 client.27849.objecter _send_command 8 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.854+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fec64120740 con 0x7fec640ff300 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.855+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fec64120740 con 0x7fec640ff300 2024-02-13T00:51:03.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.855+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.855+0000 7fec5bfff700 10 client.27849.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.855+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec640ff190 osd=3 4 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.855+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 3 8 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec69797700 10 client.27849.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 9 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6800/3344263388 conn(0x7fec64120e20 0x7fec64123210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64120740 osd=4 3 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64120740 osd=4 3 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64120740 osd=4 3 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 9 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64120740 osd=4 3 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 4 9 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64120740 osd=4 4 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec69797700 10 client.27849.objecter _send_command 9 2024-02-13T00:51:03.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec64125e90 con 0x7fec64120e20 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec62ffd700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6800/3344263388 conn(0x7fec64120e20 0x7fec64123210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec5bfff700 10 client.27849.objecter ms_handle_connect 0x7fec64120e20 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.856+0000 7fec62ffd700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6800/3344263388 conn(0x7fec64120e20 0x7fec64123210 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.857+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fec64125e90 con 0x7fec64120e20 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.857+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.857+0000 7fec5bfff700 10 client.27849.objecter _finish_command 9 = system:0 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.857+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64120740 osd=4 4 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.857+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 4 9 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.871+0000 7fec69797700 10 client.27849.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.871+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 10 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.871+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64120740 osd=4 3 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.871+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64120740 osd=4 3 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.871+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64120740 osd=4 3 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.871+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 10 2024-02-13T00:51:03.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.871+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64120740 osd=4 3 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.871+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 4 10 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.871+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64120740 osd=4 4 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.871+0000 7fec69797700 10 client.27849.objecter _send_command 10 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.871+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fec64125e90 con 0x7fec64120e20 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fec64125e90 con 0x7fec64120e20 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec5bfff700 10 client.27849.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64120740 osd=4 4 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 4 10 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec69797700 10 client.27849.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 11 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6804/2277917372 conn(0x7fec64123c10 0x7fec64124070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:03.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64125e90 osd=5 3 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64125e90 osd=5 3 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64125e90 osd=5 3 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 11 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64125e90 osd=5 3 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 5 11 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64125e90 osd=5 4 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec69797700 10 client.27849.objecter _send_command 11 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec64125ca0 con 0x7fec64123c10 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.873+0000 7fec627fc700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6804/2277917372 conn(0x7fec64123c10 0x7fec64124070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.874+0000 7fec5bfff700 10 client.27849.objecter ms_handle_connect 0x7fec64123c10 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.874+0000 7fec627fc700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6804/2277917372 conn(0x7fec64123c10 0x7fec64124070 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.874+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fec64125ca0 con 0x7fec64123c10 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.874+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:51:03.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.874+0000 7fec5bfff700 10 client.27849.objecter _finish_command 11 = system:0 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.874+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64125e90 osd=5 4 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.874+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 5 11 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.888+0000 7fec69797700 10 client.27849.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.888+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 12 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.888+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64125e90 osd=5 3 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.888+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64125e90 osd=5 3 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.888+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64125e90 osd=5 3 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.888+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 12 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.888+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64125e90 osd=5 3 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.888+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 5 12 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.888+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64125e90 osd=5 4 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.888+0000 7fec69797700 10 client.27849.objecter _send_command 12 2024-02-13T00:51:03.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.888+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fec64125ca0 con 0x7fec64123c10 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fec64125ca0 con 0x7fec64123c10 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec5bfff700 10 client.27849.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64125e90 osd=5 4 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 5 12 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec69797700 10 client.27849.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 13 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6808/1506640341 conn(0x7fec641285d0 0x7fec64128a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64125ca0 osd=6 3 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64125ca0 osd=6 3 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64125ca0 osd=6 3 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 13 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64125ca0 osd=6 3 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 6 13 2024-02-13T00:51:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64125ca0 osd=6 4 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec69797700 10 client.27849.objecter _send_command 13 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.890+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec64129030 con 0x7fec641285d0 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.891+0000 7fec637fe700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6808/1506640341 conn(0x7fec641285d0 0x7fec64128a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.891+0000 7fec637fe700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6808/1506640341 conn(0x7fec641285d0 0x7fec64128a10 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.891+0000 7fec5bfff700 10 client.27849.objecter ms_handle_connect 0x7fec641285d0 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.892+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fec64129030 con 0x7fec641285d0 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.892+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.892+0000 7fec5bfff700 10 client.27849.objecter _finish_command 13 = system:0 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.892+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64125ca0 osd=6 4 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.892+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 6 13 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.906+0000 7fec69797700 10 client.27849.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.906+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 14 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.906+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64125ca0 osd=6 3 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.906+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64125ca0 osd=6 3 2024-02-13T00:51:03.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.906+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64125ca0 osd=6 3 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.906+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 14 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.906+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64125ca0 osd=6 3 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.906+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 6 14 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.906+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64125ca0 osd=6 4 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.906+0000 7fec69797700 10 client.27849.objecter _send_command 14 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.906+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fec64129030 con 0x7fec641285d0 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fec64129030 con 0x7fec641285d0 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec5bfff700 10 client.27849.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64125ca0 osd=6 4 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 6 14 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec69797700 10 client.27849.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 15 2024-02-13T00:51:03.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6812/1964192772 conn(0x7fec6412b870 0x7fec6412bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64129030 osd=7 3 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64129030 osd=7 3 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64129030 osd=7 3 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 15 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64129030 osd=7 3 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 7 15 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64129030 osd=7 4 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec69797700 10 client.27849.objecter _send_command 15 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.908+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec6412c1a0 con 0x7fec6412b870 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.909+0000 7fec62ffd700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6812/1964192772 conn(0x7fec6412b870 0x7fec6412bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.909+0000 7fec5bfff700 10 client.27849.objecter ms_handle_connect 0x7fec6412b870 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.909+0000 7fec62ffd700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6812/1964192772 conn(0x7fec6412b870 0x7fec6412bcb0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.910+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fec6412c1a0 con 0x7fec6412b870 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.910+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:51:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.910+0000 7fec5bfff700 10 client.27849.objecter _finish_command 15 = system:0 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.910+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64129030 osd=7 4 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.910+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 7 15 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.924+0000 7fec69797700 10 client.27849.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.924+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign -1 16 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.924+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64129030 osd=7 3 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.924+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64129030 osd=7 3 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.924+0000 7fec69797700 20 client.27849.objecter _get_session s=0x7fec64129030 osd=7 3 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.924+0000 7fec69797700 15 client.27849.objecter _session_command_op_remove -1 16 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.924+0000 7fec69797700 20 client.27849.objecter get_session s=0x7fec64129030 osd=7 3 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.924+0000 7fec69797700 15 client.27849.objecter _session_command_op_assign 7 16 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.924+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64129030 osd=7 4 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.924+0000 7fec69797700 10 client.27849.objecter _send_command 16 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.924+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fec6412c1a0 con 0x7fec6412b870 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.926+0000 7fec5bfff700 1 -- 172.21.15.162:0/386879398 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fec6412c1a0 con 0x7fec6412b870 2024-02-13T00:51:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.926+0000 7fec5bfff700 10 client.27849.objecter ms_dispatch 0x7fec64001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.926+0000 7fec5bfff700 10 client.27849.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.926+0000 7fec5bfff700 20 client.27849.objecter put_session s=0x7fec64129030 osd=7 4 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.926+0000 7fec5bfff700 15 client.27849.objecter _session_command_op_remove 7 16 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 10 librados: watch_flush enter 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 10 librados: watch_flush exit 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 10 client.27849.objecter close_session for osd.0 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6801/2103743338 conn(0x7fec64028590 msgr2=0x7fec640fea60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6801/2103743338 conn(0x7fec64028590 0x7fec640fea60 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64107280 osd=0 1 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 10 client.27849.objecter close_session for osd.1 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6805/150599780 conn(0x7fec64059570 msgr2=0x7fec64119210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6805/150599780 conn(0x7fec64059570 0x7fec64119210 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64059400 osd=1 1 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 10 client.27849.objecter close_session for osd.2 2024-02-13T00:51:03.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6809/3689719376 conn(0x7fec640277c0 msgr2=0x7fec6411ca60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6809/3689719376 conn(0x7fec640277c0 0x7fec6411ca60 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64027650 osd=2 1 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 10 client.27849.objecter close_session for osd.3 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6813/604971011 conn(0x7fec640ff300 msgr2=0x7fec64120250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6813/604971011 conn(0x7fec640ff300 0x7fec64120250 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec640ff190 osd=3 1 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec637fe700 1 -- 172.21.15.162:0/386879398 reap_dead start 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 10 client.27849.objecter close_session for osd.4 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6800/3344263388 conn(0x7fec64120e20 msgr2=0x7fec64123210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6800/3344263388 conn(0x7fec64120e20 0x7fec64123210 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec637fe700 1 -- 172.21.15.162:0/386879398 reap_dead start 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64120740 osd=4 1 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 10 client.27849.objecter close_session for osd.5 2024-02-13T00:51:03.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6804/2277917372 conn(0x7fec64123c10 msgr2=0x7fec64124070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6804/2277917372 conn(0x7fec64123c10 0x7fec64124070 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64125e90 osd=5 1 2024-02-13T00:51:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 10 client.27849.objecter close_session for osd.6 2024-02-13T00:51:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6808/1506640341 conn(0x7fec641285d0 msgr2=0x7fec64128a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6808/1506640341 conn(0x7fec641285d0 0x7fec64128a10 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64125ca0 osd=6 1 2024-02-13T00:51:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 10 client.27849.objecter close_session for osd.7 2024-02-13T00:51:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6812/1964192772 conn(0x7fec6412b870 msgr2=0x7fec6412bcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.181:6812/1964192772 conn(0x7fec6412b870 0x7fec6412bcb0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 20 client.27849.objecter put_session s=0x7fec64129030 osd=7 1 2024-02-13T00:51:03.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 20 client.27849.objecter shutdown clearing up homeless session... 2024-02-13T00:51:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 10 client.27849.objecter successfully canceled tick 2024-02-13T00:51:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6800/2866534053 conn(0x7fec50062c90 msgr2=0x7fec50065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> v2:172.21.15.162:6800/2866534053 conn(0x7fec50062c90 0x7fec50065150 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 rx=0x7fec5400b3b0 tx=0x7fec54004d10).stop 2024-02-13T00:51:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec640f9b70 msgr2=0x7fec6410e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 --2- 172.21.15.162:0/386879398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec640f9b70 0x7fec6410e7c0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 rx=0x7fec4c0048f0 tx=0x7fec4c009cb0).stop 2024-02-13T00:51:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec637fe700 1 -- 172.21.15.162:0/386879398 reap_dead start 2024-02-13T00:51:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 shutdown_connections 2024-02-13T00:51:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 >> 172.21.15.162:0/386879398 conn(0x7fec640f38c0 msgr2=0x7fec640fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 shutdown_connections 2024-02-13T00:51:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 -- 172.21.15.162:0/386879398 wait complete. 2024-02-13T00:51:03.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:02.927+0000 7fec69797700 1 librados: shutdown 2024-02-13T00:51:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:51:03.543 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:51:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:02.500196+0000 mgr.y (mgr.14544) 544 : cluster [DBG] pgmap v960: 204 pgs: 1 active+clean+snaptrim, 203 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:02.787698+0000 mon.a (mon.0) 3202 : audit [DBG] from='client.? 172.21.15.162:0/386879398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:03.380758+0000 osd.6 (osd.6) 187 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:51:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:03.383663+0000 osd.6 (osd.6) 188 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:51:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:03.519326+0000 mon.a (mon.0) 3203 : cluster [DBG] osdmap e638: 8 total, 8 up, 4 in 2024-02-13T00:51:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:03.520617+0000 mon.a (mon.0) 3204 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:04.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:02.500196+0000 mgr.y (mgr.14544) 544 : cluster [DBG] pgmap v960: 204 pgs: 1 active+clean+snaptrim, 203 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:04.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:02.787698+0000 mon.a (mon.0) 3202 : audit [DBG] from='client.? 172.21.15.162:0/386879398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:04.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:03.380758+0000 osd.6 (osd.6) 187 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:51:04.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:03.383663+0000 osd.6 (osd.6) 188 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:51:04.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:03.519326+0000 mon.a (mon.0) 3203 : cluster [DBG] osdmap e638: 8 total, 8 up, 4 in 2024-02-13T00:51:04.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:03.520617+0000 mon.a (mon.0) 3204 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:04.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:02.500196+0000 mgr.y (mgr.14544) 544 : cluster [DBG] pgmap v960: 204 pgs: 1 active+clean+snaptrim, 203 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:04.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:02.787698+0000 mon.a (mon.0) 3202 : audit [DBG] from='client.? 172.21.15.162:0/386879398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:04.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:03.380758+0000 osd.6 (osd.6) 187 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:51:04.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:03.383663+0000 osd.6 (osd.6) 188 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:51:04.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:03.519326+0000 mon.a (mon.0) 3203 : cluster [DBG] osdmap e638: 8 total, 8 up, 4 in 2024-02-13T00:51:04.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:03.520617+0000 mon.a (mon.0) 3204 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:04.170 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:04.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:04.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:04.259 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:04.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:04.313 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:04.515 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:04.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:04.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:04.645 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:04.777 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:04.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:04.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:04.913 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:04.987 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:04.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:04.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:05.037 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:51:05.037 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:51:05.037 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-02-13T00:51:05.038 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-02-13T00:51:05.038 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:51:05.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:05.119 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:05.317 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:05.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:05.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:05.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:51:05.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:05.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:04.341004+0000 mon.a (mon.0) 3205 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:05.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:04.500738+0000 mgr.y (mgr.14544) 545 : cluster [DBG] pgmap v963: 204 pgs: 32 unknown, 1 active+clean+snaptrim, 171 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:05.447 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:04.520379+0000 mon.a (mon.0) 3206 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-110","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:05.448 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:04.520482+0000 mon.a (mon.0) 3207 : cluster [DBG] osdmap e639: 8 total, 8 up, 4 in 2024-02-13T00:51:05.448 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:04.538913+0000 mon.a (mon.0) 3208 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:05.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:04.341004+0000 mon.a (mon.0) 3205 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:05.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:04.500738+0000 mgr.y (mgr.14544) 545 : cluster [DBG] pgmap v963: 204 pgs: 32 unknown, 1 active+clean+snaptrim, 171 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:05.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:04.520379+0000 mon.a (mon.0) 3206 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-110","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:05.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:04.520482+0000 mon.a (mon.0) 3207 : cluster [DBG] osdmap e639: 8 total, 8 up, 4 in 2024-02-13T00:51:05.448 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:04.538913+0000 mon.a (mon.0) 3208 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:05.450 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:05.653 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:05.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:05.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:05.765 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:04.341004+0000 mon.a (mon.0) 3205 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:05.765 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:04.500738+0000 mgr.y (mgr.14544) 545 : cluster [DBG] pgmap v963: 204 pgs: 32 unknown, 1 active+clean+snaptrim, 171 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:05.765 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:04.520379+0000 mon.a (mon.0) 3206 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-110","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:05.765 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:04.520482+0000 mon.a (mon.0) 3207 : cluster [DBG] osdmap e639: 8 total, 8 up, 4 in 2024-02-13T00:51:05.765 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:04.538913+0000 mon.a (mon.0) 3208 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:05.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-13T00:51:05.787 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:05.809 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:05.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:05.913 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:05.942 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:05.962 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:05.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:06.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:51:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:51:06.050 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:06.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:06.096 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:06.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:51:06.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.186+0000 7fadbf51b700 1 -- 172.21.15.162:0/2653371787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadb80f84a0 msgr2=0x7fadb8104aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:06.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.186+0000 7fadbf51b700 1 --2- 172.21.15.162:0/2653371787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadb80f84a0 0x7fadb8104aa0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7fadb0009c30 tx=0x7fadb0004c50).stop 2024-02-13T00:51:06.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.186+0000 7fadbf51b700 1 -- 172.21.15.162:0/2653371787 shutdown_connections 2024-02-13T00:51:06.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.186+0000 7fadbf51b700 1 --2- 172.21.15.162:0/2653371787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadb80f84a0 0x7fadb8104aa0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:06.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.186+0000 7fadbf51b700 1 --2- 172.21.15.162:0/2653371787 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadb80fdc30 0x7fadb80fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:06.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.186+0000 7fadbf51b700 1 --2- 172.21.15.162:0/2653371787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fadb80fe590 0x7fadb80f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:06.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.186+0000 7fadbf51b700 1 -- 172.21.15.162:0/2653371787 >> 172.21.15.162:0/2653371787 conn(0x7fadb80f3880 msgr2=0x7fadb80f5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:06.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.186+0000 7fadbf51b700 1 -- 172.21.15.162:0/2653371787 shutdown_connections 2024-02-13T00:51:06.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.186+0000 7fadbf51b700 1 -- 172.21.15.162:0/2653371787 wait complete. 2024-02-13T00:51:06.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.187+0000 7fadbf51b700 1 librados: starting msgr at 2024-02-13T00:51:06.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.187+0000 7fadbf51b700 1 librados: starting objecter 2024-02-13T00:51:06.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.187+0000 7fadbf51b700 1 Processor -- start 2024-02-13T00:51:06.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.187+0000 7fadbf51b700 1 -- start start 2024-02-13T00:51:06.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.187+0000 7fadbf51b700 1 librados: setting wanted keys 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.187+0000 7fadbf51b700 1 librados: calling monclient init 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.187+0000 7fadbf51b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fadb80f84a0 0x7fadb806b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.187+0000 7fadbf51b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadb80fdc30 0x7fadb806b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.187+0000 7fadbf51b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadb80fe590 0x7fadb806bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.187+0000 7fadbf51b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb81164e0 con 0x7fadb80fdc30 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.187+0000 7fadbf51b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb8116650 con 0x7fadb80f84a0 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.187+0000 7fadbf51b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb8116810 con 0x7fadb80fe590 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbcab6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadb80fdc30 0x7fadb806b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbcab6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadb80fdc30 0x7fadb806b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:36330/0 (socket says 172.21.15.162:36330) 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbcab6700 1 -- 172.21.15.162:0/994913832 learned_addr learned my addr 172.21.15.162:0/994913832 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbdab8700 1 --2- 172.21.15.162:0/994913832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadb80fe590 0x7fadb806bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbd2b7700 1 --2- 172.21.15.162:0/994913832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fadb80f84a0 0x7fadb806b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:06.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbcab6700 1 -- 172.21.15.162:0/994913832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadb80fe590 msgr2=0x7fadb806bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbcab6700 1 --2- 172.21.15.162:0/994913832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadb80fe590 0x7fadb806bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbcab6700 1 -- 172.21.15.162:0/994913832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fadb80f84a0 msgr2=0x7fadb806b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbcab6700 1 --2- 172.21.15.162:0/994913832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fadb80f84a0 0x7fadb806b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbcab6700 1 -- 172.21.15.162:0/994913832 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb4005040 con 0x7fadb80fdc30 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbcab6700 1 --2- 172.21.15.162:0/994913832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadb80fdc30 0x7fadb806b660 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 rx=0x7fada800e970 tx=0x7fada800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadae7fc700 10 client.?.objecter ms_handle_connect 0x7fadb80fdc30 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadae7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadae7fc700 1 -- 172.21.15.162:0/994913832 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada8004660 con 0x7fadb80fdc30 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbf51b700 1 -- 172.21.15.162:0/994913832 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb00097c0 con 0x7fadb80fdc30 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadae7fc700 1 -- 172.21.15.162:0/994913832 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fada800f040 con 0x7fadb80fdc30 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadae7fc700 1 -- 172.21.15.162:0/994913832 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada8014790 con 0x7fadb80fdc30 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbf51b700 10 client.27855.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbf51b700 1 -- 172.21.15.162:0/994913832 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb8116f20 con 0x7fadb80fdc30 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.188+0000 7fadbf51b700 1 librados: init done 2024-02-13T00:51:06.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.189+0000 7fadbf51b700 1 -- 172.21.15.162:0/994913832 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb805fae0 con 0x7fadb80fdc30 2024-02-13T00:51:06.196 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:06.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.191+0000 7fadae7fc700 1 -- 172.21.15.162:0/994913832 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fada80090d0 con 0x7fadb80fdc30 2024-02-13T00:51:06.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.191+0000 7fadae7fc700 1 --2- 172.21.15.162:0/994913832 >> v2:172.21.15.162:6800/2866534053 conn(0x7fada4062c90 0x7fada4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:06.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.191+0000 7fadae7fc700 1 -- 172.21.15.162:0/994913832 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(640..640 src has 449..640) v4 ==== 8587+0+0 (secure 0 0 0) 0x7fada807d170 con 0x7fadb80fdc30 2024-02-13T00:51:06.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.191+0000 7fadae7fc700 10 client.27855.objecter ms_dispatch 0x7fadb8001550 osd_map(640..640 src has 449..640) v4 2024-02-13T00:51:06.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.191+0000 7fadae7fc700 3 client.27855.objecter handle_osd_map got epochs [640,640] > 0 2024-02-13T00:51:06.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.191+0000 7fadae7fc700 3 client.27855.objecter handle_osd_map decoding full epoch 640 2024-02-13T00:51:06.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.191+0000 7fadbd2b7700 1 --2- 172.21.15.162:0/994913832 >> v2:172.21.15.162:6800/2866534053 conn(0x7fada4062c90 0x7fada4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:06.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.191+0000 7fadae7fc700 20 client.27855.objecter dump_active .. 0 homeless 2024-02-13T00:51:06.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.192+0000 7fadae7fc700 10 client.27855.objecter ms_handle_connect 0x7fada4062c90 2024-02-13T00:51:06.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.192+0000 7fadbd2b7700 1 --2- 172.21.15.162:0/994913832 >> v2:172.21.15.162:6800/2866534053 conn(0x7fada4062c90 0x7fada4065150 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7fadb4004870 tx=0x7fadb400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:06.197 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:51:06] "GET /metrics HTTP/1.1" 200 189212 "" "Prometheus/2.33.4" 2024-02-13T00:51:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:06.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:06.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.195+0000 7fadae7fc700 1 -- 172.21.15.162:0/994913832 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fada8050560 con 0x7fadb80fdc30 2024-02-13T00:51:06.248 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:06.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:06.327 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:06.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.386+0000 7fadbf51b700 1 -- 172.21.15.162:0/994913832 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fadb806c720 con 0x7fadb80fdc30 2024-02-13T00:51:06.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.387+0000 7fadae7fc700 1 -- 172.21.15.162:0/994913832 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v640) v1 ==== 81+0+49298 (secure 0 0 0) 0x7fada8019070 con 0x7fadb80fdc30 2024-02-13T00:51:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:51:06.392 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:51:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "epoch": 640, 2024-02-13T00:51:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:51:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "created": "2024-02-13T00:31:51.797017+0000", 2024-02-13T00:51:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "modified": "2024-02-13T00:51:05.517992+0000", 2024-02-13T00:51:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "last_up_change": "2024-02-13T00:48:03.474219+0000", 2024-02-13T00:51:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "last_in_change": "2024-02-13T00:46:00.399226+0000", 2024-02-13T00:51:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-13T00:51:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_num": 5804032, 2024-02-13T00:51:06.392 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_set": [ 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "nodeep-scrub", 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "pglog_hardlimit", 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "purged_snapdirs", 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_deletes", 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "sortbitwise" 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_version": 109, 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "full_ratio": 0.94999998807907104, 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_snapshot": "", 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_max": 239, 2024-02-13T00:51:06.393 INFO:teuthology.orchestra.run.smithi162.stdout: "max_osd": 8, 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "require_min_compat_client": "luminous", 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "min_compat_client": "luminous", 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "require_osd_release": "pacific", 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "pools": [ 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 1, 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "device_health_metrics", 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:35:34.545390+0000", 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:06.394 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 1, 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 1, 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 1, 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 1, 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 1, 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:06.395 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "71", 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:06.396 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:06.397 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:06.398 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_max": 32, 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_min": 1 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_devicehealth": {} 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 2, 2024-02-13T00:51:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": ".rgw.root", 2024-02-13T00:51:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.459980+0000", 2024-02-13T00:51:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:06.400 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:06.401 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "58", 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:06.402 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:06.403 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 3, 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "iscsi", 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.740219+0000", 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:06.405 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 3, 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 3, 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 3, 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 3, 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 3, 2024-02-13T00:51:06.406 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "102", 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:06.407 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:06.408 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd": {} 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 4, 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.log", 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:41.489292+0000", 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:06.410 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:51:06.411 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "60", 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:06.413 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:06.414 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 5, 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.control", 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:43.466304+0000", 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:06.415 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 22, 2024-02-13T00:51:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 22, 2024-02-13T00:51:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 22, 2024-02-13T00:51:06.416 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 22, 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 22, 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "5.16", 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 544, 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 543, 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 543, 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "544", 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "544", 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "542", 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:06.417 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:06.418 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:06.419 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 6, 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.meta", 2024-02-13T00:51:06.420 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:45.542442+0000", 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 42, 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 42, 2024-02-13T00:51:06.421 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 42, 2024-02-13T00:51:06.422 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 42, 2024-02-13T00:51:06.422 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 42, 2024-02-13T00:51:06.422 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:06.422 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:06.422 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:06.422 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:06.422 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:06.422 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:06.422 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:06.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.422 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "248", 2024-02-13T00:51:06.422 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "246", 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:06.423 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:06.424 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_bias": 4 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 28, 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:00.190777+0000", 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "108", 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:06.429 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:06.430 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 234, 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-101", 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:50:12.777962+0000", 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8193, 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 3, 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 1, 2024-02-13T00:51:06.431 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 8, 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 8, 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 8, 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 8, 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 8, 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:06.432 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "640", 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "635", 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "635", 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "635", 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 5, 2024-02-13T00:51:06.433 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 640, 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [ 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: 239 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:06.434 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi162-84057-101", 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:06.435 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 8192, 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 239, 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-110", 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:51:02.770538+0000", 2024-02-13T00:51:06.436 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8193, 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:51:06.437 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "640", 2024-02-13T00:51:06.438 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 5, 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 640, 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": 234, 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:06.439 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.440 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:06.441 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99", 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 8, 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 638, 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:06.442 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6801", 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.443 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.444 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6802", 2024-02-13T00:51:06.444 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:51:06.444 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.444 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.444 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:06.444 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.444 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.444 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6804", 2024-02-13T00:51:06.444 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:51:06.444 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6803", 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6801/2103743338", 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6802/2103743338", 2024-02-13T00:51:06.445 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6804/2103743338", 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6803/2103743338", 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886", 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:51:06.446 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 13, 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 638, 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6805", 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:51:06.447 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6806", 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:06.448 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6808", 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6807", 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:51:06.449 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6805/150599780", 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6806/150599780", 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6808/150599780", 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6807/150599780", 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.450 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e", 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 18, 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 333, 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6809", 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6810", 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.452 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6812", 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6811", 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6809/3689719376", 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6810/3689719376", 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6812/3689719376", 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6811/3689719376", 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:06.454 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001", 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 25, 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 255, 2024-02-13T00:51:06.455 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6813", 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6814", 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6816", 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.457 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6815", 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6813/604971011", 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6814/604971011", 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6816/604971011", 2024-02-13T00:51:06.458 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6815/604971011", 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3", 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:06.459 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 31, 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 81, 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6800", 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:06.460 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6801", 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6803", 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:51:06.461 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6802", 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6800/3344263388", 2024-02-13T00:51:06.462 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6801/3344263388", 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6803/3344263388", 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6802/3344263388", 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723", 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:51:06.463 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 39, 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 436, 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 479, 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 399, 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 437, 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6804", 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:51:06.464 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6805", 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6807", 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6806", 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.466 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6804/2277917372", 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6805/2277917372", 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2277917372", 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2277917372", 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474", 2024-02-13T00:51:06.467 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 45, 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 638, 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.468 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6808", 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6809", 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.469 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6811", 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6810", 2024-02-13T00:51:06.470 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6808/1506640341", 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6809/1506640341", 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1506640341", 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1506640341", 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:51:06.471 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b", 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 53, 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 215, 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.472 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6812", 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6813", 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.473 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6815", 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6814", 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:51:06.474 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6812/1964192772", 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6813/1964192772", 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6815/1964192772", 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6814/1964192772", 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_xinfo": [ 2024-02-13T00:51:06.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:51:05.019053+0000", 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:06.476 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:49:51.947964+0000", 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:51:06.477 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:50:51.149335+0000", 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:49:54.589902+0000", 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:51:06.478 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:50:59.963296+0000", 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:51:06.479 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:50:34.133215+0000", 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 436 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:50:39.597253+0000", 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:50:13.831366+0000", 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap": [ 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.481 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.b", 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: 3, 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: 6, 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: 5 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap_items": [ 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.4", 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.482 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 0 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "5.5", 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 6 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "6.6", 2024-02-13T00:51:06.483 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 0 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "6.22", 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 4 2024-02-13T00:51:06.484 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "28.12", 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 4 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "28.14", 2024-02-13T00:51:06.485 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 1 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_temp": [], 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_temp": [], 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "blocklist": { 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/1909610488": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2370739667": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/1502590187": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/4121673848": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:51:06.486 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2412382714": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3041562701": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/2775563314": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3135632190": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/197307636": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2653247729": "2024-02-14T00:32:24.380625+0000" 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "range_blocklist": {}, 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profiles": { 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "default": { 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:51:06.487 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van" 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "testprofile-test-rados-api-smithi162-84057-101": { 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-device-class": "", 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-root": "default", 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van", 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: "w": "8" 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps_queue": [ 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 24, 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 9, 2024-02-13T00:51:06.489 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 25, 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.490 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 60, 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.491 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 78, 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.492 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.493 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 85, 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 94, 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.494 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 95, 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 4, 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 4 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.495 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 107, 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 113, 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 125, 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 10, 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 133, 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 12, 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 154, 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 14, 2024-02-13T00:51:06.499 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 166, 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 17, 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.500 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 176, 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 20, 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 234, 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 4, 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.501 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 237, 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 238, 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:06.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 4, 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "new_removed_snaps": [], 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "new_purged_snaps": [], 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_node_flags": {}, 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "device_class_flags": {}, 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode": { 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_enabled": false, 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_bucket_count": 0, 2024-02-13T00:51:06.503 INFO:teuthology.orchestra.run.smithi162.stdout: "degraded_stretch_mode": 0, 2024-02-13T00:51:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: "recovering_stretch_mode": 0, 2024-02-13T00:51:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_bucket": 0 2024-02-13T00:51:06.504 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:06.504 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:51:06.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 10 librados: watch_flush enter 2024-02-13T00:51:06.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 10 librados: watch_flush exit 2024-02-13T00:51:06.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 20 client.27855.objecter shutdown clearing up homeless session... 2024-02-13T00:51:06.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 10 client.27855.objecter successfully canceled tick 2024-02-13T00:51:06.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 1 -- 172.21.15.162:0/994913832 >> v2:172.21.15.162:6800/2866534053 conn(0x7fada4062c90 msgr2=0x7fada4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:06.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 1 --2- 172.21.15.162:0/994913832 >> v2:172.21.15.162:6800/2866534053 conn(0x7fada4062c90 0x7fada4065150 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7fadb4004870 tx=0x7fadb400b040).stop 2024-02-13T00:51:06.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 1 -- 172.21.15.162:0/994913832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadb80fdc30 msgr2=0x7fadb806b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:06.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 1 --2- 172.21.15.162:0/994913832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadb80fdc30 0x7fadb806b660 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 rx=0x7fada800e970 tx=0x7fada800b040).stop 2024-02-13T00:51:06.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 1 -- 172.21.15.162:0/994913832 shutdown_connections 2024-02-13T00:51:06.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 1 --2- 172.21.15.162:0/994913832 >> v2:172.21.15.162:6800/2866534053 conn(0x7fada4062c90 0x7fada4065150 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:06.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 1 --2- 172.21.15.162:0/994913832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadb80fe590 0x7fadb806bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:06.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 1 --2- 172.21.15.162:0/994913832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fadb80f84a0 0x7fadb806b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:06.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 1 --2- 172.21.15.162:0/994913832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadb80fdc30 0x7fadb806b660 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:06.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 1 -- 172.21.15.162:0/994913832 >> 172.21.15.162:0/994913832 conn(0x7fadb80f3880 msgr2=0x7fadb80f50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:06.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 1 -- 172.21.15.162:0/994913832 shutdown_connections 2024-02-13T00:51:06.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.389+0000 7fadbf51b700 1 -- 172.21.15.162:0/994913832 wait complete. 2024-02-13T00:51:06.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:06.390+0000 7fadbf51b700 1 librados: shutdown 2024-02-13T00:51:06.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:05.018870+0000 osd.0 (osd.0) 241 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:06.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:05.019051+0000 osd.0 (osd.0) 242 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:06.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:05.522558+0000 mon.a (mon.0) 3209 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-110", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:06.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:05.522630+0000 mon.a (mon.0) 3210 : cluster [DBG] osdmap e640: 8 total, 8 up, 4 in 2024-02-13T00:51:06.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:05.525695+0000 mon.a (mon.0) 3211 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-110"}]: dispatch 2024-02-13T00:51:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:05.643014+0000 osd.1 (osd.1) 265 : cluster [DBG] 6.e scrub starts 2024-02-13T00:51:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:05.644553+0000 osd.1 (osd.1) 266 : cluster [DBG] 6.e scrub ok 2024-02-13T00:51:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:05.823170+0000 mgr.y (mgr.14544) 546 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:06.387328+0000 mon.a (mon.0) 3212 : audit [DBG] from='client.? 172.21.15.162:0/994913832' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:51:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:05.018870+0000 osd.0 (osd.0) 241 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:05.019051+0000 osd.0 (osd.0) 242 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:05.522558+0000 mon.a (mon.0) 3209 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-110", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:05.522630+0000 mon.a (mon.0) 3210 : cluster [DBG] osdmap e640: 8 total, 8 up, 4 in 2024-02-13T00:51:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:05.525695+0000 mon.a (mon.0) 3211 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-110"}]: dispatch 2024-02-13T00:51:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:05.643014+0000 osd.1 (osd.1) 265 : cluster [DBG] 6.e scrub starts 2024-02-13T00:51:06.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:05.644553+0000 osd.1 (osd.1) 266 : cluster [DBG] 6.e scrub ok 2024-02-13T00:51:06.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:05.823170+0000 mgr.y (mgr.14544) 546 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:06.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:06.387328+0000 mon.a (mon.0) 3212 : audit [DBG] from='client.? 172.21.15.162:0/994913832' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:51:06.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:05.018870+0000 osd.0 (osd.0) 241 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:06.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:05.019051+0000 osd.0 (osd.0) 242 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:06.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:05.522558+0000 mon.a (mon.0) 3209 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-110", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:06.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:05.522630+0000 mon.a (mon.0) 3210 : cluster [DBG] osdmap e640: 8 total, 8 up, 4 in 2024-02-13T00:51:06.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:05.525695+0000 mon.a (mon.0) 3211 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-110"}]: dispatch 2024-02-13T00:51:06.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:05.643014+0000 osd.1 (osd.1) 265 : cluster [DBG] 6.e scrub starts 2024-02-13T00:51:06.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:05.644553+0000 osd.1 (osd.1) 266 : cluster [DBG] 6.e scrub ok 2024-02-13T00:51:06.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:05.823170+0000 mgr.y (mgr.14544) 546 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:06.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:06.387328+0000 mon.a (mon.0) 3212 : audit [DBG] from='client.? 172.21.15.162:0/994913832' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:51:07.075 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:07.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:07.182 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:07.210 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:07.509 INFO:tasks.thrashosds.thrasher:j is {'epoch': 640, 'fsid': 'd4ee01fc-ca06-11ee-95b9-87774f69a715', 'created': '2024-02-13T00:31:51.797017+0000', 'modified': '2024-02-13T00:51:05.517992+0000', 'last_up_change': '2024-02-13T00:48:03.474219+0000', 'last_in_change': '2024-02-13T00:46:00.399226+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 109, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 239, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-13T00:35:34.545390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-13T00:37:39.459980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-13T00:37:39.740219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-13T00:37:41.489292+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-13T00:37:43.466304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 544, 'last_epoch_started': 543, 'last_epoch_clean': 543, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '544', 'last_force_op_resend_preluminous': '542', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-13T00:37:45.542442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '246', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 28, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-13T00:41:00.190777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 234, 'pool_name': 'test-rados-api-smithi162-84057-101', 'create_time': '2024-02-13T00:50:12.777962+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '640', 'last_force_op_resend': '635', 'last_force_op_resend_prenautilus': '635', 'last_force_op_resend_preluminous': '635', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 640, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [239], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi162-84057-101', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 239, 'pool_name': 'test-rados-api-smithi162-84057-110', 'create_time': '2024-02-13T00:51:02.770538+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '640', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 640, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 234, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 638, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6801', 'nonce': 2103743338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 2103743338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 2103743338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6803', 'nonce': 2103743338}]}, 'public_addr': '172.21.15.162:6801/2103743338', 'cluster_addr': '172.21.15.162:6802/2103743338', 'heartbeat_back_addr': '172.21.15.162:6804/2103743338', 'heartbeat_front_addr': '172.21.15.162:6803/2103743338', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '39e9d2ef-9aa4-45bb-8a47-b0e0ec222886', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 638, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6805', 'nonce': 150599780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 150599780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 150599780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6807', 'nonce': 150599780}]}, 'public_addr': '172.21.15.162:6805/150599780', 'cluster_addr': '172.21.15.162:6806/150599780', 'heartbeat_back_addr': '172.21.15.162:6808/150599780', 'heartbeat_front_addr': '172.21.15.162:6807/150599780', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 333, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6809', 'nonce': 3689719376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3689719376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3689719376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6811', 'nonce': 3689719376}]}, 'public_addr': '172.21.15.162:6809/3689719376', 'cluster_addr': '172.21.15.162:6810/3689719376', 'heartbeat_back_addr': '172.21.15.162:6812/3689719376', 'heartbeat_front_addr': '172.21.15.162:6811/3689719376', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e6b9f0e-7a18-4cf8-9331-1a973fadc001', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 255, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6813', 'nonce': 604971011}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 604971011}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 604971011}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6815', 'nonce': 604971011}]}, 'public_addr': '172.21.15.162:6813/604971011', 'cluster_addr': '172.21.15.162:6814/604971011', 'heartbeat_back_addr': '172.21.15.162:6816/604971011', 'heartbeat_front_addr': '172.21.15.162:6815/604971011', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 3344263388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 3344263388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 3344263388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 3344263388}]}, 'public_addr': '172.21.15.181:6800/3344263388', 'cluster_addr': '172.21.15.181:6801/3344263388', 'heartbeat_back_addr': '172.21.15.181:6803/3344263388', 'heartbeat_front_addr': '172.21.15.181:6802/3344263388', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6aa2bd68-7f4b-4973-ac49-7fbc57c1a723', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 436, 'up_from': 479, 'up_thru': 399, 'down_at': 437, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2277917372}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2277917372}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2277917372}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2277917372}]}, 'public_addr': '172.21.15.181:6804/2277917372', 'cluster_addr': '172.21.15.181:6805/2277917372', 'heartbeat_back_addr': '172.21.15.181:6807/2277917372', 'heartbeat_front_addr': '172.21.15.181:6806/2277917372', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '791ba814-5284-432e-addc-d6b7f5f3d474', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 638, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1506640341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1506640341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1506640341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1506640341}]}, 'public_addr': '172.21.15.181:6808/1506640341', 'cluster_addr': '172.21.15.181:6809/1506640341', 'heartbeat_back_addr': '172.21.15.181:6811/1506640341', 'heartbeat_front_addr': '172.21.15.181:6810/1506640341', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f369c65-e663-4eb2-87a5-f2900f15b35b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 215, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 1964192772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 1964192772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 1964192772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 1964192772}]}, 'public_addr': '172.21.15.181:6812/1964192772', 'cluster_addr': '172.21.15.181:6813/1964192772', 'heartbeat_back_addr': '172.21.15.181:6815/1964192772', 'heartbeat_front_addr': '172.21.15.181:6814/1964192772', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:51:05.019053+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:49:51.947964+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:50:51.149335+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:49:54.589902+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:50:59.963296+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:50:34.133215+0000', 'dead_epoch': 436}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:50:39.597253+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:50:13.831366+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.b', 'osds': [3, 6, 5]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '5.5', 'mappings': [{'from': 0, 'to': 6}]}, {'pgid': '6.6', 'mappings': [{'from': 6, 'to': 0}]}, {'pgid': '6.22', 'mappings': [{'from': 6, 'to': 4}]}, {'pgid': '28.12', 'mappings': [{'from': 0, 'to': 4}]}, {'pgid': '28.14', 'mappings': [{'from': 6, 'to': 1}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.162:6800/1909610488': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2370739667': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/1502590187': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/4121673848': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:0/2412382714': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/3041562701': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/2775563314': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/3135632190': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/197307636': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2653247729': '2024-02-14T00:32:24.380625+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi162-84057-101': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 24, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 60, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 78, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 94, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 113, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 166, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 4, 'length': 1}]}, {'pool': 237, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 238, 'snaps': [{'begin': 4, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-13T00:51:07.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:51:07.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:06.261476+0000 osd.0 (osd.0) 243 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:51:07.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:06.264563+0000 osd.0 (osd.0) 244 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:51:07.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:06.501098+0000 mgr.y (mgr.14544) 547 : cluster [DBG] pgmap v966: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:51:07.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:06.529032+0000 mon.a (mon.0) 3213 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-110"}]': finished 2024-02-13T00:51:07.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:06.529100+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e641: 8 total, 8 up, 4 in 2024-02-13T00:51:07.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:06.531068+0000 mon.a (mon.0) 3215 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-110", "mode": "writeback"}]: dispatch 2024-02-13T00:51:07.797 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:51:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:51:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:51:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:51:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.5.0:282 60.a e128/120 60:51f054f9:test-rados-api-smithi162-84073-8::foo:7 v 128'6, mlcod=127'3)", 2024-02-13T00:51:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:22.774235+0000", 2024-02-13T00:51:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 585.017271876, 2024-02-13T00:51:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.007417252, 2024-02-13T00:51:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:51:07.797 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774235+0000", 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774235+0000", 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774234+0000", 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4309999999999996e-06 2024-02-13T00:51:07.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774239+0000", 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3290000000000001e-06 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774244+0000", 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8940000000000003e-06 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774248+0000", 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000423449 2024-02-13T00:51:07.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774671+0000", 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7167e-05 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.774689+0000", 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0069276140000000003 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:51:07.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.781616+0000", 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5657000000000001e-05 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:22.781652+0000", 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.007417252 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.16275.0:11 66.6 e130/124 66:629bc174:test-rados-api-smithi162-84081-9::foo:head v 130'7, mlcod=128'5)", 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:24.903032+0000", 2024-02-13T00:51:07.801 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 582.88847459800002, 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041090170000000004, 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903032+0000", 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903032+0000", 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:51:07.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903031+0000", 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3899999999999999e-06 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903040+0000", 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0533999999999999e-05 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903051+0000", 2024-02-13T00:51:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0879999999999997e-06 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903058+0000", 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6895000000000002e-05 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903135+0000", 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2158999999999999e-05 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.903157+0000", 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039450989999999997 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.907102+0000", 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9041000000000001e-05 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.907141+0000", 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041090170000000004 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.5.0:293 66.6 e132/124 66:629bc174:test-rados-api-smithi162-84081-9::foo:7 v 132'15, mlcod=131'12)", 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:27.091552+0000", 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 580.69995460099994, 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084929270000000008, 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091552+0000", 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091552+0000", 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091550+0000", 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1979e-05 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091562+0000", 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2990000000000002e-06 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091572+0000", 2024-02-13T00:51:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1720000000000001e-06 2024-02-13T00:51:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.091579+0000", 2024-02-13T00:51:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000886285 2024-02-13T00:51:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.092465+0000", 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2421000000000001e-05 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.092488+0000", 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075259120000000001 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.100013+0000", 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1429e-05 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:27.100045+0000", 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084929270000000008 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.2.0:365 61.4 e188/121 61:21c2c914:test-rados-api-smithi162-84057-14::foo8:7 v 188'36, mlcod=154'30)", 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:42:23.329308+0000", 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 524.46219898799995, 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0089267670000000004, 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:51:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329308+0000", 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329308+0000", 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329307+0000", 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3189999999999996e-06 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329312+0000", 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.648e-06 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329317+0000", 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3749999999999996e-06 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329322+0000", 2024-02-13T00:51:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00029372900000000001 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329615+0000", 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7045999999999999e-05 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.329632+0000", 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0085733790000000008 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:51:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.338206+0000", 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8561e-05 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.338234+0000", 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0089267670000000004 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.5.0:512 61.19 e188/121 61:9a2cc200:test-rados-api-smithi162-84057-14::foo19:7 v 188'11, mlcod=153'7)", 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:42:23.330027+0000", 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 524.46147932199995, 2024-02-13T00:51:07.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073812160000000003, 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330027+0000", 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330027+0000", 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330026+0000", 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5390000000000002e-06 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330034+0000", 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0818999999999999e-05 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330045+0000", 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1320000000000002e-06 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330050+0000", 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000152163 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330202+0000", 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9652000000000001e-05 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.330222+0000", 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071553379999999998 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.337377+0000", 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1399999999999998e-05 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:23.337408+0000", 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073812160000000003 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.25162.0:3128 184.0 e300/296 184:00adb854:test-rados-api-smithi162-84057-38::foo:head v 300'2, mlcod=0'0)", 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:15.383766+0000", 2024-02-13T00:51:07.818 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 412.40774030799997, 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013839536, 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.383766+0000", 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.383766+0000", 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.383765+0000", 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002561158 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386326+0000", 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7059e-05 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386344+0000", 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.259e-06 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386352+0000", 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014103499999999999 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386493+0000", 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2059999999999999e-05 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386515+0000", 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011046402 2024-02-13T00:51:07.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.397561+0000", 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4487000000000003e-05 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.397606+0000", 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013839536 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.822 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(28.c REQUEST e379)", 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:45:34.516696+0000", 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 333.274810436, 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003757631, 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516696+0000", 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516696+0000", 2024-02-13T00:51:07.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516695+0000", 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1699999999999997e-06 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516703+0000", 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0827000000000001e-05 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516734+0000", 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5639000000000001e-05 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516750+0000", 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011122500000000001 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516861+0000", 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6915999999999999e-05 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516878+0000", 2024-02-13T00:51:07.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035454750000000002 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.520424+0000", 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0348e-05 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.520434+0000", 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9860999999999998e-05 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.520454+0000", 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003757631 2024-02-13T00:51:07.826 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(6.17 REQUEST e438)", 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:47:02.781512+0000", 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 245.009994431, 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043540109999999996, 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781512+0000", 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781512+0000", 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781512+0000", 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4500000000000003e-06 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781517+0000", 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4729999999999999e-05 2024-02-13T00:51:07.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781542+0000", 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3523000000000001e-05 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781555+0000", 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010741799999999999 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781663+0000", 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9942999999999999e-05 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781683+0000", 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041526920000000004 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.785835+0000", 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2148e-05 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.785848+0000", 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8539999999999999e-05 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.785866+0000", 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043540109999999996 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(28.17 REQUEST e612)", 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:50:26.821965+0000", 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.969541735999996, 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049237200000000004, 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821965+0000", 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821965+0000", 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821964+0000", 2024-02-13T00:51:07.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1099999999999999e-06 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821970+0000", 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6339999999999999e-05 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821997+0000", 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4263e-05 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.822011+0000", 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012683799999999999 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.822138+0000", 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8040000000000002e-06 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.822146+0000", 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046927059999999996 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.826838+0000", 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9426e-05 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.826858+0000", 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0803000000000003e-05 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.826889+0000", 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049237200000000004 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.5s1 639/599 ECSubWrite(tid=705, reqid=client.27753.0:474, at_version=639'6, trim_to=0'0, roll_forward_to=626'5))", 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:04.529881+0000", 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.2616254169999999, 2024-02-13T00:51:07.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00053443499999999999, 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.529881+0000", 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.529881+0000", 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.529881+0000", 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6520999999999999e-05 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.529907+0000", 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2522e-05 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.529920+0000", 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0662000000000001e-05 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.529930+0000", 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6884000000000005e-05 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.530007+0000", 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2383e-05 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.530020+0000", 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00037475099999999999 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:51:07.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.530394+0000", 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1158999999999999e-05 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.530416+0000", 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00053443499999999999 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.5s1 639/599 ECSubWrite(tid=706, reqid=client.27753.0:474, at_version=0'0, trim_to=0'0, roll_forward_to=639'6))", 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:04.536955+0000", 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.254551384, 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012597979999999999, 2024-02-13T00:51:07.839 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536955+0000", 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536955+0000", 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536955+0000", 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2790000000000004e-06 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536964+0000", 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8669999999999996e-06 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536971+0000", 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1420000000000004e-06 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536978+0000", 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010723200000000001 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.537085+0000", 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0047999999999999e-05 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.537095+0000", 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011055310000000001 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:51:07.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.538201+0000", 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.396e-05 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.538215+0000", 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012597979999999999 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(6.e REQUEST e640)", 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:05.640107+0000", 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.1513998559999998, 2024-02-13T00:51:07.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00283475, 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640107+0000", 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640107+0000", 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640106+0000", 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4880000000000001e-06 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640109+0000", 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.359e-05 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640123+0000", 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6920000000000002e-06 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640130+0000", 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0739000000000003e-05 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640181+0000", 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8570000000000007e-06 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640191+0000", 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026984040000000002 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.642889+0000", 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6985000000000001e-05 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.642906+0000", 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5238999999999999e-05 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.642941+0000", 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00283475 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(6.13 REQUEST e640)", 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:06.261204+0000", 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.530302756, 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000153428, 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:06.261204+0000", 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:06.261204+0000", 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:06.261201+0000", 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4116e-05 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:06.261216+0000", 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4530000000000001e-05 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:06.261240+0000", 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3603e-05 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:06.261254+0000", 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6573000000000001e-05 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:06.261320+0000", 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8431999999999999e-05 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:06.261339+0000", 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8510000000000001e-05 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:06.261357+0000", 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000153428 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(234.5s1 642/599 ECSubRead(tid=707, to_read={234:ab1ab8e8:test-rados-api-smithi162-84057-111::foo:head=0,4096,40}, subchunks={234:ab1ab8e8:test-rados-api-smithi162-84057-111::foo:head=[0,1]}, attrs_to_read=))", 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:07.541504+0000", 2024-02-13T00:51:07.851 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.250002475, 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025868499999999998, 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.541504+0000", 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.541504+0000", 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:51:07.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.541504+0000", 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5439999999999999e-06 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.541512+0000", 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0583e-05 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.541523+0000", 2024-02-13T00:51:07.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7729999999999999e-06 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.541531+0000", 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9815000000000003e-05 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.541581+0000", 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000182227 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.541763+0000", 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025868499999999998 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.6.0:708 239.18 e642/638 239:1ea42aa8:test-rados-api-smithi162-84057-111::bar:head v 642'1, mlcod=0'0)", 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:07.544825+0000", 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.246681714, 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00041323500000000002, 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.544825+0000", 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.544825+0000", 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.544824+0000", 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8110000000000003e-06 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.544832+0000", 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0380000000000005e-06 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.544841+0000", 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5860000000000004e-06 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.544846+0000", 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3731999999999999e-05 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.544870+0000", 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5998000000000002e-05 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.544886+0000", 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031266099999999999 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.545199+0000", 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9196999999999998e-05 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.545238+0000", 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00041323500000000002 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(239.18 epoch 642/638 rep_tid 709 entries 642'2 (0'0) error 239:1ea42aa8:test-rados-api-smithi162-84057-111::bar:head by client.27753.0:476 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:07.545288+0000", 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.24621876100000001, 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030454800000000003, 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.545288+0000", 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.545288+0000", 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.545287+0000", 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2650000000000002e-06 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.545296+0000", 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2559e-05 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.545309+0000", 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0590000000000003e-06 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.545317+0000", 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5179000000000001e-05 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.545372+0000", 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022008800000000001 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.545592+0000", 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030454800000000003 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(239.18 epoch 642/638 rep_tid 710 entries 642'3 (0'0) error 239:1ea42aa8:test-rados-api-smithi162-84057-111::bar:head by client.27753.0:477 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 642'1)", 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:07.546122+0000", 2024-02-13T00:51:07.862 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.24538404899999999, 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00028282900000000001, 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.546122+0000", 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.546122+0000", 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.546122+0000", 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8690000000000004e-06 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.546130+0000", 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0229e-05 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.546140+0000", 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1329999999999998e-06 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.546145+0000", 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0871000000000001e-05 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.546196+0000", 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020947399999999999 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.546405+0000", 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00028282900000000001 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.865 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.27753.0:511 239.18 e642/638 239:1ea42aa8:test-rados-api-smithi162-84057-111::bar:head v 642'4, mlcod=642'2)", 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:07.556875+0000", 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.234631116, 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000364271, 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.556875+0000", 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.556875+0000", 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.556875+0000", 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4520000000000002e-06 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.556881+0000", 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7909999999999993e-06 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.556890+0000", 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.578e-06 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.556894+0000", 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5923999999999997e-05 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.556930+0000", 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.885e-05 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.556959+0000", 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025731300000000001 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.557216+0000", 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3215999999999999e-05 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.557240+0000", 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000364271 2024-02-13T00:51:07.869 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.5s1 642/599 ECSubWrite(tid=712, reqid=osd.1.13:197, at_version=642'7, trim_to=0'0, roll_forward_to=639'6))", 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:07.578652+0000", 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.21285420699999999, 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000428169, 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.578652+0000", 2024-02-13T00:51:07.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.578652+0000", 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.578652+0000", 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3946000000000001e-05 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.578676+0000", 2024-02-13T00:51:07.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3489999999999998e-06 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.578685+0000", 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1919999999999996e-06 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.578690+0000", 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3256000000000001e-05 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.578723+0000", 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.409e-06 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.578730+0000", 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033615799999999998 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.579066+0000", 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4508999999999999e-05 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.579080+0000", 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000428169 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.5s1 642/599 ECSubWrite(tid=713, reqid=osd.1.13:197, at_version=0'0, trim_to=0'0, roll_forward_to=642'7))", 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:07.579483+0000", 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.21202309499999999, 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031665399999999998, 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.579483+0000", 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.579483+0000", 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.579483+0000", 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5200000000000003e-06 2024-02-13T00:51:07.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.579492+0000", 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.85e-06 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.579501+0000", 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2020000000000001e-06 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.579507+0000", 2024-02-13T00:51:07.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5074000000000002e-05 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.579552+0000", 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7200000000000001e-06 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.579562+0000", 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022289 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.579785+0000", 2024-02-13T00:51:07.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4978e-05 2024-02-13T00:51:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:07.579800+0000", 2024-02-13T00:51:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031665399999999998 2024-02-13T00:51:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:07.878 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:07.878 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:51:07.879 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:06.261476+0000 osd.0 (osd.0) 243 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:51:07.879 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:06.264563+0000 osd.0 (osd.0) 244 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:51:07.879 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:06.501098+0000 mgr.y (mgr.14544) 547 : cluster [DBG] pgmap v966: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:51:07.879 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:06.529032+0000 mon.a (mon.0) 3213 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-110"}]': finished 2024-02-13T00:51:07.879 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:06.529100+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e641: 8 total, 8 up, 4 in 2024-02-13T00:51:07.879 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:06.531068+0000 mon.a (mon.0) 3215 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-110", "mode": "writeback"}]: dispatch 2024-02-13T00:51:07.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:06.261476+0000 osd.0 (osd.0) 243 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:51:07.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:06.264563+0000 osd.0 (osd.0) 244 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:51:07.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:06.501098+0000 mgr.y (mgr.14544) 547 : cluster [DBG] pgmap v966: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:51:07.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:06.529032+0000 mon.a (mon.0) 3213 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-110"}]': finished 2024-02-13T00:51:07.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:06.529100+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e641: 8 total, 8 up, 4 in 2024-02-13T00:51:07.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:06.531068+0000 mon.a (mon.0) 3215 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-110", "mode": "writeback"}]: dispatch 2024-02-13T00:51:07.959 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:07.959 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:07.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:07.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:08.064 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:08.093 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:08.422 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-13T00:51:08.441 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:08.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:08.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:08.550 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:07.302661+0000 osd.6 (osd.6) 189 : cluster [DBG] 28.f scrub starts 2024-02-13T00:51:08.550 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:07.304571+0000 osd.6 (osd.6) 190 : cluster [DBG] 28.f scrub ok 2024-02-13T00:51:08.551 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:07.526093+0000 mon.a (mon.0) 3216 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:08.551 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:07.532578+0000 mon.a (mon.0) 3217 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-110", "mode": "writeback"}]': finished 2024-02-13T00:51:08.551 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:07.532645+0000 mon.a (mon.0) 3218 : cluster [DBG] osdmap e642: 8 total, 8 up, 4 in 2024-02-13T00:51:08.551 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:07.581141+0000 mon.a (mon.0) 3219 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:08.551 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:07.302661+0000 osd.6 (osd.6) 189 : cluster [DBG] 28.f scrub starts 2024-02-13T00:51:08.551 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:07.304571+0000 osd.6 (osd.6) 190 : cluster [DBG] 28.f scrub ok 2024-02-13T00:51:08.551 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:07.526093+0000 mon.a (mon.0) 3216 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:08.551 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:07.532578+0000 mon.a (mon.0) 3217 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-110", "mode": "writeback"}]': finished 2024-02-13T00:51:08.551 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:07.532645+0000 mon.a (mon.0) 3218 : cluster [DBG] osdmap e642: 8 total, 8 up, 4 in 2024-02-13T00:51:08.551 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:07.581141+0000 mon.a (mon.0) 3219 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:08.576 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:08.713 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:08.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:51:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:51:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:51:08.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:07.302661+0000 osd.6 (osd.6) 189 : cluster [DBG] 28.f scrub starts 2024-02-13T00:51:08.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:07.304571+0000 osd.6 (osd.6) 190 : cluster [DBG] 28.f scrub ok 2024-02-13T00:51:08.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:07.526093+0000 mon.a (mon.0) 3216 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:08.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:07.532578+0000 mon.a (mon.0) 3217 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-110", "mode": "writeback"}]': finished 2024-02-13T00:51:08.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:07.532645+0000 mon.a (mon.0) 3218 : cluster [DBG] osdmap e642: 8 total, 8 up, 4 in 2024-02-13T00:51:08.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:07.581141+0000 mon.a (mon.0) 3219 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:09.176 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:09.428 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:09.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:09.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:51:09.567 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:09.691 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:08.342735+0000 osd.6 (osd.6) 191 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:51:09.691 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:08.345630+0000 osd.6 (osd.6) 192 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:51:09.691 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:08.501467+0000 mgr.y (mgr.14544) 548 : cluster [DBG] pgmap v969: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:09.691 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:08.539067+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:09.691 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:08.539106+0000 mon.a (mon.0) 3221 : cluster [DBG] osdmap e643: 8 total, 8 up, 4 in 2024-02-13T00:51:09.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:08.540070+0000 mon.a (mon.0) 3222 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-110"}]: dispatch 2024-02-13T00:51:09.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:08.603154+0000 osd.1 (osd.1) 267 : cluster [DBG] 28.11 scrub starts 2024-02-13T00:51:09.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:08.606259+0000 osd.1 (osd.1) 268 : cluster [DBG] 28.11 scrub ok 2024-02-13T00:51:09.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:09.537045+0000 mon.a (mon.0) 3223 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:09.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:09.540183+0000 mon.a (mon.0) 3224 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-110"}]': finished 2024-02-13T00:51:09.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:09.540247+0000 mon.a (mon.0) 3225 : cluster [DBG] osdmap e644: 8 total, 8 up, 4 in 2024-02-13T00:51:09.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:08.342735+0000 osd.6 (osd.6) 191 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:51:09.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:08.345630+0000 osd.6 (osd.6) 192 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:51:09.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:08.501467+0000 mgr.y (mgr.14544) 548 : cluster [DBG] pgmap v969: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:09.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:08.539067+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:09.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:08.539106+0000 mon.a (mon.0) 3221 : cluster [DBG] osdmap e643: 8 total, 8 up, 4 in 2024-02-13T00:51:09.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:08.540070+0000 mon.a (mon.0) 3222 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-110"}]: dispatch 2024-02-13T00:51:09.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:08.603154+0000 osd.1 (osd.1) 267 : cluster [DBG] 28.11 scrub starts 2024-02-13T00:51:09.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:08.606259+0000 osd.1 (osd.1) 268 : cluster [DBG] 28.11 scrub ok 2024-02-13T00:51:09.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:09.537045+0000 mon.a (mon.0) 3223 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:09.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:09.540183+0000 mon.a (mon.0) 3224 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-110"}]': finished 2024-02-13T00:51:09.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:09.540247+0000 mon.a (mon.0) 3225 : cluster [DBG] osdmap e644: 8 total, 8 up, 4 in 2024-02-13T00:51:09.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:08.342735+0000 osd.6 (osd.6) 191 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:51:09.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:08.345630+0000 osd.6 (osd.6) 192 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:51:09.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:08.501467+0000 mgr.y (mgr.14544) 548 : cluster [DBG] pgmap v969: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:09.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:08.539067+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:09.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:08.539106+0000 mon.a (mon.0) 3221 : cluster [DBG] osdmap e643: 8 total, 8 up, 4 in 2024-02-13T00:51:09.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:08.540070+0000 mon.a (mon.0) 3222 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-110"}]: dispatch 2024-02-13T00:51:09.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:08.603154+0000 osd.1 (osd.1) 267 : cluster [DBG] 28.11 scrub starts 2024-02-13T00:51:09.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:08.606259+0000 osd.1 (osd.1) 268 : cluster [DBG] 28.11 scrub ok 2024-02-13T00:51:09.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:09.537045+0000 mon.a (mon.0) 3223 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:09.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:09.540183+0000 mon.a (mon.0) 3224 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-110"}]': finished 2024-02-13T00:51:09.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:09.540247+0000 mon.a (mon.0) 3225 : cluster [DBG] osdmap e644: 8 total, 8 up, 4 in 2024-02-13T00:51:09.845 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:51:09.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:09.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:51:09.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:09.980 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:10.394 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:51:10.394 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:51:10.394 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:51:10.395 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:51:10.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.711+0000 7f08df47b700 1 -- 172.21.15.162:0/1006997417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08d80f9b70 msgr2=0x7f08d80f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:10.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.711+0000 7f08df47b700 1 --2- 172.21.15.162:0/1006997417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08d80f9b70 0x7f08d80f9f50 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7f08cc004870 tx=0x7f08cc009cb0).stop 2024-02-13T00:51:10.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.711+0000 7f08df47b700 1 -- 172.21.15.162:0/1006997417 shutdown_connections 2024-02-13T00:51:10.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.711+0000 7f08df47b700 1 --2- 172.21.15.162:0/1006997417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08d80f8500 0x7f08d8104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:10.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.711+0000 7f08df47b700 1 --2- 172.21.15.162:0/1006997417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08d80f9b70 0x7f08d80f9f50 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:10.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.711+0000 7f08df47b700 1 --2- 172.21.15.162:0/1006997417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08d80fa440 0x7f08d80f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:10.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.711+0000 7f08df47b700 1 -- 172.21.15.162:0/1006997417 >> 172.21.15.162:0/1006997417 conn(0x7f08d80f38c0 msgr2=0x7f08d80f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:10.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 -- 172.21.15.162:0/1006997417 shutdown_connections 2024-02-13T00:51:10.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 -- 172.21.15.162:0/1006997417 wait complete. 2024-02-13T00:51:10.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 librados: starting msgr at 2024-02-13T00:51:10.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 librados: starting objecter 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 Processor -- start 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 -- start start 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 librados: setting wanted keys 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 librados: calling monclient init 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08d80f8500 0x7f08d806b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08d80f9b70 0x7f08d806b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08d80fa440 0x7f08d806bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d81166c0 con 0x7f08d80f9b70 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d8116830 con 0x7f08d80f8500 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08df47b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d81169f0 con 0x7f08d80fa440 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.712+0000 7f08dda18700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08d80fa440 0x7f08d806bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08dca16700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08d80f9b70 0x7f08d806b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:10.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08dd217700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08d80f8500 0x7f08d806b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:10.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08dda18700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08d80fa440 0x7f08d806bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:38378/0 (socket says 172.21.15.162:38378) 2024-02-13T00:51:10.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08dda18700 1 -- 172.21.15.162:0/4017395790 learned_addr learned my addr 172.21.15.162:0/4017395790 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:10.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08dda18700 1 -- 172.21.15.162:0/4017395790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08d80f8500 msgr2=0x7f08d806b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:10.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08dda18700 1 --2- 172.21.15.162:0/4017395790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08d80f8500 0x7f08d806b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:10.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08dda18700 1 -- 172.21.15.162:0/4017395790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08d80f9b70 msgr2=0x7f08d806b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:10.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08dda18700 1 --2- 172.21.15.162:0/4017395790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08d80f9b70 0x7f08d806b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:10.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08dda18700 1 -- 172.21.15.162:0/4017395790 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc005040 con 0x7f08d80fa440 2024-02-13T00:51:10.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08dda18700 1 --2- 172.21.15.162:0/4017395790 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08d80fa440 0x7f08d806bb50 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 rx=0x7f08d000ec20 tx=0x7f08d000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:10.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08ca7fc700 10 client.?.objecter ms_handle_connect 0x7f08d80fa440 2024-02-13T00:51:10.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08ca7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:10.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08ca7fc700 1 -- 172.21.15.162:0/4017395790 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0009950 con 0x7f08d80fa440 2024-02-13T00:51:10.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08ca7fc700 1 -- 172.21.15.162:0/4017395790 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08d000f040 con 0x7f08d80fa440 2024-02-13T00:51:10.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08df47b700 1 -- 172.21.15.162:0/4017395790 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d8116c70 con 0x7f08d80fa440 2024-02-13T00:51:10.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08ca7fc700 1 -- 172.21.15.162:0/4017395790 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0007df0 con 0x7f08d80fa440 2024-02-13T00:51:10.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08df47b700 10 client.37958.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:10.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08df47b700 1 -- 172.21.15.162:0/4017395790 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d8117190 con 0x7f08d80fa440 2024-02-13T00:51:10.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.713+0000 7f08df47b700 1 librados: init done 2024-02-13T00:51:10.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.714+0000 7f08df47b700 1 -- 172.21.15.162:0/4017395790 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d80475e0 con 0x7f08d80fa440 2024-02-13T00:51:10.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.714+0000 7f08ca7fc700 1 -- 172.21.15.162:0/4017395790 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f08d00092e0 con 0x7f08d80fa440 2024-02-13T00:51:10.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.715+0000 7f08ca7fc700 1 --2- 172.21.15.162:0/4017395790 >> v2:172.21.15.162:6800/2866534053 conn(0x7f08c4062c90 0x7f08c4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:10.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.715+0000 7f08ca7fc700 1 -- 172.21.15.162:0/4017395790 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(645..645 src has 449..645) v4 ==== 8206+0+0 (secure 0 0 0) 0x7f08d007cf60 con 0x7f08d80fa440 2024-02-13T00:51:10.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.715+0000 7f08ca7fc700 10 client.37958.objecter ms_dispatch 0x7f08d8001550 osd_map(645..645 src has 449..645) v4 2024-02-13T00:51:10.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.715+0000 7f08ca7fc700 3 client.37958.objecter handle_osd_map got epochs [645,645] > 0 2024-02-13T00:51:10.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.715+0000 7f08ca7fc700 3 client.37958.objecter handle_osd_map decoding full epoch 645 2024-02-13T00:51:10.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.715+0000 7f08dd217700 1 --2- 172.21.15.162:0/4017395790 >> v2:172.21.15.162:6800/2866534053 conn(0x7f08c4062c90 0x7f08c4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:10.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.715+0000 7f08ca7fc700 20 client.37958.objecter dump_active .. 0 homeless 2024-02-13T00:51:10.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.715+0000 7f08dd217700 1 --2- 172.21.15.162:0/4017395790 >> v2:172.21.15.162:6800/2866534053 conn(0x7f08c4062c90 0x7f08c4065150 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 rx=0x7f08cc009a00 tx=0x7f08cc009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:10.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.715+0000 7f08ca7fc700 10 client.37958.objecter ms_handle_connect 0x7f08c4062c90 2024-02-13T00:51:10.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.722+0000 7f08ca7fc700 1 -- 172.21.15.162:0/4017395790 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f08d0049f60 con 0x7f08d80fa440 2024-02-13T00:51:10.728 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:10.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:10.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:10.862 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:10.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:10.908+0000 7f08df47b700 1 -- 172.21.15.162:0/4017395790 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f08d806c650 con 0x7f08d80fa440 2024-02-13T00:51:10.986 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:10.987 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:08.962206+0000 osd.3 (osd.3) 161 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:10.987 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:08.962347+0000 osd.3 (osd.3) 162 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:10.988 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:10.523697+0000 mon.a (mon.0) 3226 : cluster [DBG] osdmap e645: 8 total, 8 up, 4 in 2024-02-13T00:51:10.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:08.962206+0000 osd.3 (osd.3) 161 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:10.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:08.962347+0000 osd.3 (osd.3) 162 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:10.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:10.523697+0000 mon.a (mon.0) 3226 : cluster [DBG] osdmap e645: 8 total, 8 up, 4 in 2024-02-13T00:51:10.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:10.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:11.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:08.962206+0000 osd.3 (osd.3) 161 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:11.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:08.962347+0000 osd.3 (osd.3) 162 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:11.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:10.523697+0000 mon.a (mon.0) 3226 : cluster [DBG] osdmap e645: 8 total, 8 up, 4 in 2024-02-13T00:51:11.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:11.098 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-13T00:51:11.118 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:11.216 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:11.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:11.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.303+0000 7faa04ee0700 1 -- 172.21.15.162:0/2539858563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa000fdc50 msgr2=0x7faa000fe030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.303+0000 7faa04ee0700 1 --2- 172.21.15.162:0/2539858563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa000fdc50 0x7faa000fe030 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 rx=0x7fa9f00047b0 tx=0x7fa9f0009cb0).stop 2024-02-13T00:51:11.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.303+0000 7faa04ee0700 1 -- 172.21.15.162:0/2539858563 shutdown_connections 2024-02-13T00:51:11.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.303+0000 7faa04ee0700 1 --2- 172.21.15.162:0/2539858563 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faa000f84c0 0x7faa00104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.303+0000 7faa04ee0700 1 --2- 172.21.15.162:0/2539858563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa000fdc50 0x7faa000fe030 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.303+0000 7faa04ee0700 1 --2- 172.21.15.162:0/2539858563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa000fe5b0 0x7faa000f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.303+0000 7faa04ee0700 1 -- 172.21.15.162:0/2539858563 >> 172.21.15.162:0/2539858563 conn(0x7faa000f38a0 msgr2=0x7faa000f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.303+0000 7faa04ee0700 1 -- 172.21.15.162:0/2539858563 shutdown_connections 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.303+0000 7faa04ee0700 1 -- 172.21.15.162:0/2539858563 wait complete. 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.303+0000 7faa04ee0700 1 librados: starting msgr at 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.303+0000 7faa04ee0700 1 librados: starting objecter 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7faa04ee0700 1 Processor -- start 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7faa04ee0700 1 -- start start 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7faa04ee0700 1 librados: setting wanted keys 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7faa04ee0700 1 librados: calling monclient init 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7faa04ee0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faa000f84c0 0x7faa0010e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7faa04ee0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa000fdc50 0x7faa0010e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7faa04ee0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa000fe5b0 0x7faa00108260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:11.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7faa04ee0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa00108750 con 0x7faa000fe5b0 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7faa04ee0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa001088c0 con 0x7faa000fdc50 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7faa04ee0700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa00108a80 con 0x7faa000f84c0 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fed9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa000fe5b0 0x7faa00108260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fe59c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faa000f84c0 0x7faa0010e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fe59c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faa000f84c0 0x7faa0010e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:38388/0 (socket says 172.21.15.162:38388) 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fe59c700 1 -- 172.21.15.162:0/3227401986 learned_addr learned my addr 172.21.15.162:0/3227401986 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fed9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa000fe5b0 0x7faa00108260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:36366/0 (socket says 172.21.15.162:36366) 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fdd9b700 1 --2- 172.21.15.162:0/3227401986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa000fdc50 0x7faa0010e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fe59c700 1 -- 172.21.15.162:0/3227401986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa000fdc50 msgr2=0x7faa0010e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fe59c700 1 --2- 172.21.15.162:0/3227401986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa000fdc50 0x7faa0010e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fe59c700 1 -- 172.21.15.162:0/3227401986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa000fe5b0 msgr2=0x7faa00108260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fe59c700 1 --2- 172.21.15.162:0/3227401986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa000fe5b0 0x7faa00108260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fe59c700 1 -- 172.21.15.162:0/3227401986 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f80097c0 con 0x7faa000f84c0 2024-02-13T00:51:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fed9d700 1 --2- 172.21.15.162:0/3227401986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa000fe5b0 0x7faa00108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9fe59c700 1 --2- 172.21.15.162:0/3227401986 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faa000f84c0 0x7faa0010e0c0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7fa9f00044e0 tx=0x7fa9f0007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9ef7fe700 10 client.?.objecter ms_handle_connect 0x7faa000f84c0 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.304+0000 7fa9ef7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.305+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0018070 con 0x7faa000f84c0 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.305+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f0005040 con 0x7faa000f84c0 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.305+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa9f00093b0 con 0x7faa000f84c0 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.305+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f001dd90 con 0x7faa000f84c0 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.305+0000 7faa04ee0700 10 client.37964.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.305+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa00108f20 con 0x7faa000f84c0 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.305+0000 7faa04ee0700 1 librados: init done 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.305+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7faa000475e0 con 0x7faa000f84c0 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.306+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fa9f001d420 con 0x7faa000f84c0 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.306+0000 7fa9ef7fe700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa9e8062c90 0x7fa9e8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:11.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.306+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(645..645 src has 449..645) v4 ==== 8206+0+0 (secure 0 0 0) 0x7fa9f007e770 con 0x7faa000f84c0 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.306+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 osd_map(645..645 src has 449..645) v4 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.306+0000 7fa9ef7fe700 3 client.37964.objecter handle_osd_map got epochs [645,645] > 0 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.306+0000 7fa9ef7fe700 3 client.37964.objecter handle_osd_map decoding full epoch 645 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.306+0000 7fa9fdd9b700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa9e8062c90 0x7fa9e8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.306+0000 7fa9ef7fe700 20 client.37964.objecter dump_active .. 0 homeless 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.306+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v645) v1 ==== 54+0+16 (secure 0 0 0) 0x7fa9f0085070 con 0x7faa000f84c0 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7fa9ef7fe700 10 client.37964.objecter ms_handle_connect 0x7fa9e8062c90 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7fa9fdd9b700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa9e8062c90 0x7fa9e8065150 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 rx=0x7faa00109a00 tx=0x7fa9f400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7faa04ee0700 10 client.37964.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 1 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6801/2103743338 conn(0x7faa000fef20 0x7faa000ff300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa001071b0 osd=0 3 2024-02-13T00:51:11.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa001071b0 osd=0 3 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa001071b0 osd=0 3 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 1 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa001071b0 osd=0 3 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 0 1 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa001071b0 osd=0 4 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7faa04ee0700 10 client.37964.objecter _send_command 1 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa00109740 con 0x7faa000fef20 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7fa9fed9d700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6801/2103743338 conn(0x7faa000fef20 0x7faa000ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7fa9ef7fe700 10 client.37964.objecter ms_handle_connect 0x7faa000fef20 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.307+0000 7fa9fed9d700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6801/2103743338 conn(0x7faa000fef20 0x7faa000ff300 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.308+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faa00109740 con 0x7faa000fef20 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.308+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.308+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 1 = system:0 2024-02-13T00:51:11.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.308+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa001071b0 osd=0 4 2024-02-13T00:51:11.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.308+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 0 1 2024-02-13T00:51:11.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:11.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.323+0000 7faa04ee0700 10 client.37964.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:11.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.323+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 2 2024-02-13T00:51:11.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.323+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa001071b0 osd=0 3 2024-02-13T00:51:11.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.323+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa001071b0 osd=0 3 2024-02-13T00:51:11.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.323+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa001071b0 osd=0 3 2024-02-13T00:51:11.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.323+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 2 2024-02-13T00:51:11.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.323+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa001071b0 osd=0 3 2024-02-13T00:51:11.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.323+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 0 2 2024-02-13T00:51:11.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.323+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa001071b0 osd=0 4 2024-02-13T00:51:11.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.323+0000 7faa04ee0700 10 client.37964.objecter _send_command 2 2024-02-13T00:51:11.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.323+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa00027410 con 0x7faa000fef20 2024-02-13T00:51:11.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faa00027410 con 0x7faa000fef20 2024-02-13T00:51:11.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:11.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa001071b0 osd=0 4 2024-02-13T00:51:11.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 0 2 2024-02-13T00:51:11.329 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:51:11.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7faa04ee0700 10 client.37964.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 3 2024-02-13T00:51:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6805/150599780 conn(0x7faa00059990 0x7faa00059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00028760 osd=1 3 2024-02-13T00:51:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00028760 osd=1 3 2024-02-13T00:51:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00028760 osd=1 3 2024-02-13T00:51:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 3 2024-02-13T00:51:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa00028760 osd=1 3 2024-02-13T00:51:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 1 3 2024-02-13T00:51:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00028760 osd=1 4 2024-02-13T00:51:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7faa04ee0700 10 client.37964.objecter _send_command 3 2024-02-13T00:51:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa0011b670 con 0x7faa00059990 2024-02-13T00:51:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7fa9fe59c700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6805/150599780 conn(0x7faa00059990 0x7faa00059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7fa9ef7fe700 10 client.37964.objecter ms_handle_connect 0x7faa00059990 2024-02-13T00:51:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.325+0000 7fa9fe59c700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6805/150599780 conn(0x7faa00059990 0x7faa00059d70 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.326+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faa0011b670 con 0x7faa00059990 2024-02-13T00:51:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.326+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:51:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.326+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 3 = system:0 2024-02-13T00:51:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.326+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa00028760 osd=1 4 2024-02-13T00:51:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.326+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 1 3 2024-02-13T00:51:11.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.343+0000 7faa04ee0700 10 client.37964.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:11.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.343+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 4 2024-02-13T00:51:11.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.343+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00028760 osd=1 3 2024-02-13T00:51:11.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.343+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00028760 osd=1 3 2024-02-13T00:51:11.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.343+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00028760 osd=1 3 2024-02-13T00:51:11.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.343+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 4 2024-02-13T00:51:11.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.343+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa00028760 osd=1 3 2024-02-13T00:51:11.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.343+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 1 4 2024-02-13T00:51:11.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.343+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00028760 osd=1 4 2024-02-13T00:51:11.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.343+0000 7faa04ee0700 10 client.37964.objecter _send_command 4 2024-02-13T00:51:11.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.343+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa00059400 con 0x7faa00059990 2024-02-13T00:51:11.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.344+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faa00059400 con 0x7faa00059990 2024-02-13T00:51:11.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.344+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:11.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.344+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.344+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa00028760 osd=1 4 2024-02-13T00:51:11.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.344+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 1 4 2024-02-13T00:51:11.348 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7faa04ee0700 10 client.37964.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 5 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6809/3689719376 conn(0x7faa0011a090 0x7faa0011a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00027db0 osd=2 3 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00027db0 osd=2 3 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00027db0 osd=2 3 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 5 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa00027db0 osd=2 3 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 2 5 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00027db0 osd=2 4 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7faa04ee0700 10 client.37964.objecter _send_command 5 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa00028430 con 0x7faa0011a090 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7fa9fdd9b700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6809/3689719376 conn(0x7faa0011a090 0x7faa0011a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:11.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7fa9ef7fe700 10 client.37964.objecter ms_handle_connect 0x7faa0011a090 2024-02-13T00:51:11.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.345+0000 7fa9fdd9b700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6809/3689719376 conn(0x7faa0011a090 0x7faa0011a490 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:11.351 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:11.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.346+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faa00028430 con 0x7faa0011a090 2024-02-13T00:51:11.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.346+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:51:11.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.346+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 5 = system:0 2024-02-13T00:51:11.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.346+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa00027db0 osd=2 4 2024-02-13T00:51:11.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.346+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 2 5 2024-02-13T00:51:11.362 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:11.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.361+0000 7faa04ee0700 10 client.37964.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:11.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.361+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 6 2024-02-13T00:51:11.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.361+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00027db0 osd=2 3 2024-02-13T00:51:11.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.361+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00027db0 osd=2 3 2024-02-13T00:51:11.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.361+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00027db0 osd=2 3 2024-02-13T00:51:11.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.361+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 6 2024-02-13T00:51:11.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.361+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa00027db0 osd=2 3 2024-02-13T00:51:11.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.361+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 2 6 2024-02-13T00:51:11.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.361+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00027db0 osd=2 4 2024-02-13T00:51:11.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.361+0000 7faa04ee0700 10 client.37964.objecter _send_command 6 2024-02-13T00:51:11.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.361+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa0011b150 con 0x7faa0011a090 2024-02-13T00:51:11.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faa0011b150 con 0x7faa0011a090 2024-02-13T00:51:11.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:11.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa00027db0 osd=2 4 2024-02-13T00:51:11.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 2 6 2024-02-13T00:51:11.367 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:51:11.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7faa04ee0700 10 client.37964.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:11.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 7 2024-02-13T00:51:11.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6813/604971011 conn(0x7faa00027990 0x7faa0011fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:11.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00027820 osd=3 3 2024-02-13T00:51:11.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00027820 osd=3 3 2024-02-13T00:51:11.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00027820 osd=3 3 2024-02-13T00:51:11.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 7 2024-02-13T00:51:11.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa00027820 osd=3 3 2024-02-13T00:51:11.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 3 7 2024-02-13T00:51:11.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00027820 osd=3 4 2024-02-13T00:51:11.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7faa04ee0700 10 client.37964.objecter _send_command 7 2024-02-13T00:51:11.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa00122960 con 0x7faa00027990 2024-02-13T00:51:11.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7fa9fed9d700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6813/604971011 conn(0x7faa00027990 0x7faa0011fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:11.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7fa9ef7fe700 10 client.37964.objecter ms_handle_connect 0x7faa00027990 2024-02-13T00:51:11.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.363+0000 7fa9fed9d700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6813/604971011 conn(0x7faa00027990 0x7faa0011fd10 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:11.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.364+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faa00122960 con 0x7faa00027990 2024-02-13T00:51:11.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.364+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:51:11.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.364+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 7 = system:0 2024-02-13T00:51:11.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.364+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa00027820 osd=3 4 2024-02-13T00:51:11.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.364+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 3 7 2024-02-13T00:51:11.382 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.378+0000 7faa04ee0700 10 client.37964.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:11.382 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.378+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 8 2024-02-13T00:51:11.382 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.378+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00027820 osd=3 3 2024-02-13T00:51:11.382 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.378+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00027820 osd=3 3 2024-02-13T00:51:11.382 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.378+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00027820 osd=3 3 2024-02-13T00:51:11.382 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.378+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 8 2024-02-13T00:51:11.382 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.378+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa00027820 osd=3 3 2024-02-13T00:51:11.382 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.378+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 3 8 2024-02-13T00:51:11.382 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.378+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00027820 osd=3 4 2024-02-13T00:51:11.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.378+0000 7faa04ee0700 10 client.37964.objecter _send_command 8 2024-02-13T00:51:11.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.378+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa0011abc0 con 0x7faa00027990 2024-02-13T00:51:11.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faa0011abc0 con 0x7faa00027990 2024-02-13T00:51:11.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:11.383 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa00027820 osd=3 4 2024-02-13T00:51:11.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 3 8 2024-02-13T00:51:11.384 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:51:11.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7faa04ee0700 10 client.37964.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:11.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 9 2024-02-13T00:51:11.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6800/3344263388 conn(0x7faa0011ad60 0x7faa00121380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:11.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa0011abc0 osd=4 3 2024-02-13T00:51:11.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa0011abc0 osd=4 3 2024-02-13T00:51:11.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa0011abc0 osd=4 3 2024-02-13T00:51:11.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 9 2024-02-13T00:51:11.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa0011abc0 osd=4 3 2024-02-13T00:51:11.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 4 9 2024-02-13T00:51:11.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa0011abc0 osd=4 4 2024-02-13T00:51:11.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7faa04ee0700 10 client.37964.objecter _send_command 9 2024-02-13T00:51:11.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.380+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa00119c80 con 0x7faa0011ad60 2024-02-13T00:51:11.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.381+0000 7fa9fe59c700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6800/3344263388 conn(0x7faa0011ad60 0x7faa00121380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:11.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.381+0000 7fa9fe59c700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6800/3344263388 conn(0x7faa0011ad60 0x7faa00121380 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:11.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.381+0000 7fa9ef7fe700 10 client.37964.objecter ms_handle_connect 0x7faa0011ad60 2024-02-13T00:51:11.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.382+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faa00119c80 con 0x7faa0011ad60 2024-02-13T00:51:11.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.382+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:51:11.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.382+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 9 = system:0 2024-02-13T00:51:11.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.382+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa0011abc0 osd=4 4 2024-02-13T00:51:11.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.382+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 4 9 2024-02-13T00:51:11.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.396+0000 7faa04ee0700 10 client.37964.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:11.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.396+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 10 2024-02-13T00:51:11.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.396+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa0011abc0 osd=4 3 2024-02-13T00:51:11.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.396+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa0011abc0 osd=4 3 2024-02-13T00:51:11.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.396+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa0011abc0 osd=4 3 2024-02-13T00:51:11.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.396+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 10 2024-02-13T00:51:11.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.396+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa0011abc0 osd=4 3 2024-02-13T00:51:11.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.396+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 4 10 2024-02-13T00:51:11.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.396+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa0011abc0 osd=4 4 2024-02-13T00:51:11.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.396+0000 7faa04ee0700 10 client.37964.objecter _send_command 10 2024-02-13T00:51:11.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.396+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa00119c80 con 0x7faa0011ad60 2024-02-13T00:51:11.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faa00119c80 con 0x7faa0011ad60 2024-02-13T00:51:11.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:11.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa0011abc0 osd=4 4 2024-02-13T00:51:11.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 4 10 2024-02-13T00:51:11.402 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:51:11.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7faa04ee0700 10 client.37964.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:11.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 11 2024-02-13T00:51:11.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6804/2277917372 conn(0x7faa00121d80 0x7faa001221e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:11.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00119c80 osd=5 3 2024-02-13T00:51:11.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00119c80 osd=5 3 2024-02-13T00:51:11.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00119c80 osd=5 3 2024-02-13T00:51:11.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 11 2024-02-13T00:51:11.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa00119c80 osd=5 3 2024-02-13T00:51:11.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 5 11 2024-02-13T00:51:11.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00119c80 osd=5 4 2024-02-13T00:51:11.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7faa04ee0700 10 client.37964.objecter _send_command 11 2024-02-13T00:51:11.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa00122700 con 0x7faa00121d80 2024-02-13T00:51:11.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.398+0000 7fa9fdd9b700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6804/2277917372 conn(0x7faa00121d80 0x7faa001221e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:11.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.399+0000 7fa9fdd9b700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6804/2277917372 conn(0x7faa00121d80 0x7faa001221e0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:11.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.399+0000 7fa9ef7fe700 10 client.37964.objecter ms_handle_connect 0x7faa00121d80 2024-02-13T00:51:11.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.400+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faa00122700 con 0x7faa00121d80 2024-02-13T00:51:11.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.400+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:51:11.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.400+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 11 = system:0 2024-02-13T00:51:11.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.400+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa00119c80 osd=5 4 2024-02-13T00:51:11.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.400+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 5 11 2024-02-13T00:51:11.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.414+0000 7faa04ee0700 10 client.37964.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:11.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.414+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 12 2024-02-13T00:51:11.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.414+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00119c80 osd=5 3 2024-02-13T00:51:11.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.414+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00119c80 osd=5 3 2024-02-13T00:51:11.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.414+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00119c80 osd=5 3 2024-02-13T00:51:11.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.414+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 12 2024-02-13T00:51:11.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.414+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa00119c80 osd=5 3 2024-02-13T00:51:11.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.414+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 5 12 2024-02-13T00:51:11.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.414+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00119c80 osd=5 4 2024-02-13T00:51:11.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.414+0000 7faa04ee0700 10 client.37964.objecter _send_command 12 2024-02-13T00:51:11.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.414+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa00122700 con 0x7faa00121d80 2024-02-13T00:51:11.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.415+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faa00122700 con 0x7faa00121d80 2024-02-13T00:51:11.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.415+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:11.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.415+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.415+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa00119c80 osd=5 4 2024-02-13T00:51:11.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.415+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 5 12 2024-02-13T00:51:11.419 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:51:11.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.415+0000 7faa04ee0700 10 client.37964.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:11.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 13 2024-02-13T00:51:11.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6808/1506640341 conn(0x7faa00127000 0x7faa00127440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:11.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00122700 osd=6 3 2024-02-13T00:51:11.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00122700 osd=6 3 2024-02-13T00:51:11.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00122700 osd=6 3 2024-02-13T00:51:11.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 13 2024-02-13T00:51:11.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa00122700 osd=6 3 2024-02-13T00:51:11.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 6 13 2024-02-13T00:51:11.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00122700 osd=6 4 2024-02-13T00:51:11.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7faa04ee0700 10 client.37964.objecter _send_command 13 2024-02-13T00:51:11.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa00129090 con 0x7faa00127000 2024-02-13T00:51:11.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7fa9fed9d700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6808/1506640341 conn(0x7faa00127000 0x7faa00127440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:11.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7fa9ef7fe700 10 client.37964.objecter ms_handle_connect 0x7faa00127000 2024-02-13T00:51:11.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.416+0000 7fa9fed9d700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6808/1506640341 conn(0x7faa00127000 0x7faa00127440 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:11.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.417+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faa00129090 con 0x7faa00127000 2024-02-13T00:51:11.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.417+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:51:11.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.417+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 13 = system:0 2024-02-13T00:51:11.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.417+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa00122700 osd=6 4 2024-02-13T00:51:11.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.417+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 6 13 2024-02-13T00:51:11.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.431+0000 7faa04ee0700 10 client.37964.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.431+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 14 2024-02-13T00:51:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.431+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00122700 osd=6 3 2024-02-13T00:51:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.431+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00122700 osd=6 3 2024-02-13T00:51:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.431+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00122700 osd=6 3 2024-02-13T00:51:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.431+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 14 2024-02-13T00:51:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.431+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa00122700 osd=6 3 2024-02-13T00:51:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.431+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 6 14 2024-02-13T00:51:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.431+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00122700 osd=6 4 2024-02-13T00:51:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.431+0000 7faa04ee0700 10 client.37964.objecter _send_command 14 2024-02-13T00:51:11.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.431+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa00129090 con 0x7faa00127000 2024-02-13T00:51:11.436 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:51:11.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faa00129090 con 0x7faa00127000 2024-02-13T00:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa00122700 osd=6 4 2024-02-13T00:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 6 14 2024-02-13T00:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7faa04ee0700 10 client.37964.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 15 2024-02-13T00:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6812/1964192772 conn(0x7faa0012b7f0 0x7faa0012bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00129090 osd=7 3 2024-02-13T00:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00129090 osd=7 3 2024-02-13T00:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00129090 osd=7 3 2024-02-13T00:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 15 2024-02-13T00:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa00129090 osd=7 3 2024-02-13T00:51:11.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 7 15 2024-02-13T00:51:11.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00129090 osd=7 4 2024-02-13T00:51:11.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7faa04ee0700 10 client.37964.objecter _send_command 15 2024-02-13T00:51:11.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa0012c280 con 0x7faa0012b7f0 2024-02-13T00:51:11.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.433+0000 7fa9fe59c700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6812/1964192772 conn(0x7faa0012b7f0 0x7faa0012bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:11.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.434+0000 7fa9ef7fe700 10 client.37964.objecter ms_handle_connect 0x7faa0012b7f0 2024-02-13T00:51:11.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.434+0000 7fa9fe59c700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6812/1964192772 conn(0x7faa0012b7f0 0x7faa0012bc10 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:11.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.434+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faa0012c280 con 0x7faa0012b7f0 2024-02-13T00:51:11.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.434+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:51:11.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.434+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 15 = system:0 2024-02-13T00:51:11.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.434+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa00129090 osd=7 4 2024-02-13T00:51:11.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.434+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 7 15 2024-02-13T00:51:11.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.448+0000 7faa04ee0700 10 client.37964.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:11.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.448+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign -1 16 2024-02-13T00:51:11.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.448+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00129090 osd=7 3 2024-02-13T00:51:11.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.448+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00129090 osd=7 3 2024-02-13T00:51:11.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.448+0000 7faa04ee0700 20 client.37964.objecter _get_session s=0x7faa00129090 osd=7 3 2024-02-13T00:51:11.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.448+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_remove -1 16 2024-02-13T00:51:11.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.448+0000 7faa04ee0700 20 client.37964.objecter get_session s=0x7faa00129090 osd=7 3 2024-02-13T00:51:11.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.448+0000 7faa04ee0700 15 client.37964.objecter _session_command_op_assign 7 16 2024-02-13T00:51:11.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.448+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00129090 osd=7 4 2024-02-13T00:51:11.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.448+0000 7faa04ee0700 10 client.37964.objecter _send_command 16 2024-02-13T00:51:11.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.448+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa0012c280 con 0x7faa0012b7f0 2024-02-13T00:51:11.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.450+0000 7fa9ef7fe700 1 -- 172.21.15.162:0/3227401986 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faa0012c280 con 0x7faa0012b7f0 2024-02-13T00:51:11.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.450+0000 7fa9ef7fe700 10 client.37964.objecter ms_dispatch 0x7faa00001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:11.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.450+0000 7fa9ef7fe700 10 client.37964.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.450+0000 7fa9ef7fe700 20 client.37964.objecter put_session s=0x7faa00129090 osd=7 4 2024-02-13T00:51:11.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.450+0000 7fa9ef7fe700 15 client.37964.objecter _session_command_op_remove 7 16 2024-02-13T00:51:11.455 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:51:11.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 10 librados: watch_flush enter 2024-02-13T00:51:11.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 10 librados: watch_flush exit 2024-02-13T00:51:11.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 10 client.37964.objecter close_session for osd.0 2024-02-13T00:51:11.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6801/2103743338 conn(0x7faa000fef20 msgr2=0x7faa000ff300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6801/2103743338 conn(0x7faa000fef20 0x7faa000ff300 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa001071b0 osd=0 1 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 10 client.37964.objecter close_session for osd.1 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6805/150599780 conn(0x7faa00059990 msgr2=0x7faa00059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6805/150599780 conn(0x7faa00059990 0x7faa00059d70 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00028760 osd=1 1 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 10 client.37964.objecter close_session for osd.2 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6809/3689719376 conn(0x7faa0011a090 msgr2=0x7faa0011a490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6809/3689719376 conn(0x7faa0011a090 0x7faa0011a490 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00027db0 osd=2 1 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 10 client.37964.objecter close_session for osd.3 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6813/604971011 conn(0x7faa00027990 msgr2=0x7faa0011fd10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6813/604971011 conn(0x7faa00027990 0x7faa0011fd10 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7fa9fed9d700 1 -- 172.21.15.162:0/3227401986 reap_dead start 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00027820 osd=3 1 2024-02-13T00:51:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 10 client.37964.objecter close_session for osd.4 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6800/3344263388 conn(0x7faa0011ad60 msgr2=0x7faa00121380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6800/3344263388 conn(0x7faa0011ad60 0x7faa00121380 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa0011abc0 osd=4 1 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 10 client.37964.objecter close_session for osd.5 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6804/2277917372 conn(0x7faa00121d80 msgr2=0x7faa001221e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6804/2277917372 conn(0x7faa00121d80 0x7faa001221e0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00119c80 osd=5 1 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 10 client.37964.objecter close_session for osd.6 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6808/1506640341 conn(0x7faa00127000 msgr2=0x7faa00127440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6808/1506640341 conn(0x7faa00127000 0x7faa00127440 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00122700 osd=6 1 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 10 client.37964.objecter close_session for osd.7 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6812/1964192772 conn(0x7faa0012b7f0 msgr2=0x7faa0012bc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.181:6812/1964192772 conn(0x7faa0012b7f0 0x7faa0012bc10 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 20 client.37964.objecter put_session s=0x7faa00129090 osd=7 1 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 20 client.37964.objecter shutdown clearing up homeless session... 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 10 client.37964.objecter successfully canceled tick 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7fa9fed9d700 1 -- 172.21.15.162:0/3227401986 reap_dead start 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa9e8062c90 msgr2=0x7fa9e8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa9e8062c90 0x7fa9e8065150 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 rx=0x7faa00109a00 tx=0x7fa9f400b040).stop 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faa000f84c0 msgr2=0x7faa0010e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faa000f84c0 0x7faa0010e0c0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7fa9f00044e0 tx=0x7fa9f0007f90).stop 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 shutdown_connections 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faa000f84c0 0x7faa0010e0c0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 --2- 172.21.15.162:0/3227401986 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa9e8062c90 0x7fa9e8065150 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 >> 172.21.15.162:0/3227401986 conn(0x7faa000f38a0 msgr2=0x7faa000f5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 shutdown_connections 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 -- 172.21.15.162:0/3227401986 wait complete. 2024-02-13T00:51:11.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.451+0000 7faa04ee0700 1 librados: shutdown 2024-02-13T00:51:11.460 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.460 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.460 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.460 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.460 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.460 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.460 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.460 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:51:11.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.524+0000 7f08ca7fc700 1 -- 172.21.15.162:0/4017395790 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v646) v1 ==== 101+0+0 (secure 0 0 0) 0x7f08d00503a0 con 0x7f08d80fa440 2024-02-13T00:51:11.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 10 librados: watch_flush enter 2024-02-13T00:51:11.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 10 librados: watch_flush exit 2024-02-13T00:51:11.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 20 client.37958.objecter shutdown clearing up homeless session... 2024-02-13T00:51:11.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 10 client.37958.objecter successfully canceled tick 2024-02-13T00:51:11.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 1 -- 172.21.15.162:0/4017395790 >> v2:172.21.15.162:6800/2866534053 conn(0x7f08c4062c90 msgr2=0x7f08c4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 1 --2- 172.21.15.162:0/4017395790 >> v2:172.21.15.162:6800/2866534053 conn(0x7f08c4062c90 0x7f08c4065150 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 rx=0x7f08cc009a00 tx=0x7f08cc009cb0).stop 2024-02-13T00:51:11.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 1 -- 172.21.15.162:0/4017395790 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08d80fa440 msgr2=0x7f08d806bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:11.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 1 --2- 172.21.15.162:0/4017395790 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08d80fa440 0x7f08d806bb50 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 rx=0x7f08d000ec20 tx=0x7f08d000b040).stop 2024-02-13T00:51:11.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 1 -- 172.21.15.162:0/4017395790 shutdown_connections 2024-02-13T00:51:11.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 1 --2- 172.21.15.162:0/4017395790 >> v2:172.21.15.162:6800/2866534053 conn(0x7f08c4062c90 0x7f08c4065150 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 1 --2- 172.21.15.162:0/4017395790 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08d80fa440 0x7f08d806bb50 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 1 --2- 172.21.15.162:0/4017395790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08d80f8500 0x7f08d806b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 1 --2- 172.21.15.162:0/4017395790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08d80f9b70 0x7f08d806b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:11.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.533+0000 7f08df47b700 1 -- 172.21.15.162:0/4017395790 >> 172.21.15.162:0/4017395790 conn(0x7f08d80f38c0 msgr2=0x7f08d80fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:11.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.534+0000 7f08df47b700 1 -- 172.21.15.162:0/4017395790 shutdown_connections 2024-02-13T00:51:11.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.534+0000 7f08df47b700 1 -- 172.21.15.162:0/4017395790 wait complete. 2024-02-13T00:51:11.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:11.534+0000 7f08df47b700 1 librados: shutdown 2024-02-13T00:51:11.539 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:51:11.895 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:10.294764+0000 osd.6 (osd.6) 193 : cluster [DBG] 4.18 scrub starts 2024-02-13T00:51:11.895 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:10.298229+0000 osd.6 (osd.6) 194 : cluster [DBG] 4.18 scrub ok 2024-02-13T00:51:11.895 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:10.502344+0000 mgr.y (mgr.14544) 549 : cluster [DBG] pgmap v972: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-13T00:51:11.895 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:10.909404+0000 mon.c (mon.2) 477 : audit [INF] from='client.? 172.21.15.162:0/4017395790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:11.895 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:10.909708+0000 mon.a (mon.0) 3227 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:11.896 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:11.235934+0000 osd.0 (osd.0) 245 : cluster [DBG] 28.16 scrub starts 2024-02-13T00:51:11.896 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:11.238902+0000 osd.0 (osd.0) 246 : cluster [DBG] 28.16 scrub ok 2024-02-13T00:51:11.896 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:11.307325+0000 mon.c (mon.2) 478 : audit [DBG] from='client.? 172.21.15.162:0/3227401986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:11.896 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:11.515984+0000 mon.a (mon.0) 3228 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:51:11.896 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:11.523190+0000 mon.a (mon.0) 3229 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:51:11.896 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:11.523251+0000 mon.a (mon.0) 3230 : cluster [DBG] osdmap e646: 8 total, 8 up, 4 in 2024-02-13T00:51:11.896 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:11.532640+0000 mon.a (mon.0) 3231 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:10.294764+0000 osd.6 (osd.6) 193 : cluster [DBG] 4.18 scrub starts 2024-02-13T00:51:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:10.298229+0000 osd.6 (osd.6) 194 : cluster [DBG] 4.18 scrub ok 2024-02-13T00:51:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:10.502344+0000 mgr.y (mgr.14544) 549 : cluster [DBG] pgmap v972: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-13T00:51:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:10.909404+0000 mon.c (mon.2) 477 : audit [INF] from='client.? 172.21.15.162:0/4017395790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:10.909708+0000 mon.a (mon.0) 3227 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:11.235934+0000 osd.0 (osd.0) 245 : cluster [DBG] 28.16 scrub starts 2024-02-13T00:51:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:11.238902+0000 osd.0 (osd.0) 246 : cluster [DBG] 28.16 scrub ok 2024-02-13T00:51:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:11.307325+0000 mon.c (mon.2) 478 : audit [DBG] from='client.? 172.21.15.162:0/3227401986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:11.515984+0000 mon.a (mon.0) 3228 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:51:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:11.523190+0000 mon.a (mon.0) 3229 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:51:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:11.523251+0000 mon.a (mon.0) 3230 : cluster [DBG] osdmap e646: 8 total, 8 up, 4 in 2024-02-13T00:51:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:11.532640+0000 mon.a (mon.0) 3231 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:12.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:10.294764+0000 osd.6 (osd.6) 193 : cluster [DBG] 4.18 scrub starts 2024-02-13T00:51:12.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:10.298229+0000 osd.6 (osd.6) 194 : cluster [DBG] 4.18 scrub ok 2024-02-13T00:51:12.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:10.502344+0000 mgr.y (mgr.14544) 549 : cluster [DBG] pgmap v972: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-13T00:51:12.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:10.909404+0000 mon.c (mon.2) 477 : audit [INF] from='client.? 172.21.15.162:0/4017395790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:10.909708+0000 mon.a (mon.0) 3227 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:11.235934+0000 osd.0 (osd.0) 245 : cluster [DBG] 28.16 scrub starts 2024-02-13T00:51:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:11.238902+0000 osd.0 (osd.0) 246 : cluster [DBG] 28.16 scrub ok 2024-02-13T00:51:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:11.307325+0000 mon.c (mon.2) 478 : audit [DBG] from='client.? 172.21.15.162:0/3227401986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:11.515984+0000 mon.a (mon.0) 3228 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:51:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:11.523190+0000 mon.a (mon.0) 3229 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:51:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:11.523251+0000 mon.a (mon.0) 3230 : cluster [DBG] osdmap e646: 8 total, 8 up, 4 in 2024-02-13T00:51:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:11.532640+0000 mon.a (mon.0) 3231 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:13.099 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:13.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:13.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:51:13.235 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:13.335 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:51:13.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:13.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:51:13.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:13.453 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:51:13.453 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:51:13.472 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:13.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:12.502905+0000 mgr.y (mgr.14544) 550 : cluster [DBG] pgmap v975: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:12.525829+0000 mon.a (mon.0) 3232 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:12.525940+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e647: 8 total, 8 up, 4 in 2024-02-13T00:51:13.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:12.540143+0000 mon.a (mon.0) 3234 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:13.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:12.502905+0000 mgr.y (mgr.14544) 550 : cluster [DBG] pgmap v975: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:13.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:12.525829+0000 mon.a (mon.0) 3232 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:13.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:12.525940+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e647: 8 total, 8 up, 4 in 2024-02-13T00:51:13.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:12.540143+0000 mon.a (mon.0) 3234 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:13.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:12.502905+0000 mgr.y (mgr.14544) 550 : cluster [DBG] pgmap v975: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:13.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:12.525829+0000 mon.a (mon.0) 3232 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:13.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:12.525940+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e647: 8 total, 8 up, 4 in 2024-02-13T00:51:13.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:12.540143+0000 mon.a (mon.0) 3234 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:13.950 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:13.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:14.056 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:14.086 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:14.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-13T00:51:14.211 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:14.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:14.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:14.313 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:14.340 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:14.358 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:14.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:14.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:14.490 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:14.524 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:14.690 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:14.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:14.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:13.313620+0000 osd.6 (osd.6) 195 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:51:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:13.316723+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:51:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:13.527128+0000 mon.a (mon.0) 3235 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:13.527217+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e648: 8 total, 8 up, 4 in 2024-02-13T00:51:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:13.528005+0000 mon.a (mon.0) 3237 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-112"}]: dispatch 2024-02-13T00:51:14.822 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:14.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:13.313620+0000 osd.6 (osd.6) 195 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:51:14.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:13.316723+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:51:14.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:13.527128+0000 mon.a (mon.0) 3235 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:14.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:13.527217+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e648: 8 total, 8 up, 4 in 2024-02-13T00:51:14.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:13.528005+0000 mon.a (mon.0) 3237 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-112"}]: dispatch 2024-02-13T00:51:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:13.313620+0000 osd.6 (osd.6) 195 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:51:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:13.316723+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:51:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:13.527128+0000 mon.a (mon.0) 3235 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:13.527217+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e648: 8 total, 8 up, 4 in 2024-02-13T00:51:14.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:13.528005+0000 mon.a (mon.0) 3237 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-112"}]: dispatch 2024-02-13T00:51:14.842 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:14.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:14.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.873+0000 7f1ae900c700 1 -- 172.21.15.162:0/807998348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ae40fcb30 msgr2=0x7f1ae40fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:14.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.873+0000 7f1ae900c700 1 --2- 172.21.15.162:0/807998348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ae40fcb30 0x7f1ae40fcf10 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7f1ad4004870 tx=0x7f1ad4009cb0).stop 2024-02-13T00:51:14.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.873+0000 7f1ae900c700 1 -- 172.21.15.162:0/807998348 shutdown_connections 2024-02-13T00:51:14.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.873+0000 7f1ae900c700 1 --2- 172.21.15.162:0/807998348 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ae40f73a0 0x7f1ae41039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:14.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.873+0000 7f1ae900c700 1 --2- 172.21.15.162:0/807998348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ae40fcb30 0x7f1ae40fcf10 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.873+0000 7f1ae900c700 1 --2- 172.21.15.162:0/807998348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ae40fd490 0x7f1ae40f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.873+0000 7f1ae900c700 1 -- 172.21.15.162:0/807998348 >> 172.21.15.162:0/807998348 conn(0x7f1ae406eb90 msgr2=0x7f1ae40f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.873+0000 7f1ae900c700 1 -- 172.21.15.162:0/807998348 shutdown_connections 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.873+0000 7f1ae900c700 1 -- 172.21.15.162:0/807998348 wait complete. 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.874+0000 7f1ae900c700 1 librados: starting msgr at 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.874+0000 7f1ae900c700 1 librados: starting objecter 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.874+0000 7f1ae900c700 1 Processor -- start 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.874+0000 7f1ae900c700 1 -- start start 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.874+0000 7f1ae900c700 1 librados: setting wanted keys 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.874+0000 7f1ae900c700 1 librados: calling monclient init 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae900c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ae40f73a0 0x7f1ae4109150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae900c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ae40fcb30 0x7f1ae4109640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:14.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae900c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ae40fd490 0x7f1ae4109c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae900c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae4114380 con 0x7f1ae40fcb30 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae900c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae41144f0 con 0x7f1ae40f73a0 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae900c700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae41146b0 con 0x7f1ae40fd490 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae259c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ae40fcb30 0x7f1ae4109640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae359e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ae40fd490 0x7f1ae4109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae259c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ae40fcb30 0x7f1ae4109640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46934/0 (socket says 172.21.15.162:46934) 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae259c700 1 -- 172.21.15.162:0/538428692 learned_addr learned my addr 172.21.15.162:0/538428692 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae2d9d700 1 --2- 172.21.15.162:0/538428692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ae40f73a0 0x7f1ae4109150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae359e700 1 -- 172.21.15.162:0/538428692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ae40f73a0 msgr2=0x7f1ae4109150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae359e700 1 --2- 172.21.15.162:0/538428692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ae40f73a0 0x7f1ae4109150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae359e700 1 -- 172.21.15.162:0/538428692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ae40fcb30 msgr2=0x7f1ae4109640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae359e700 1 --2- 172.21.15.162:0/538428692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ae40fcb30 0x7f1ae4109640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae359e700 1 -- 172.21.15.162:0/538428692 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1adc0097c0 con 0x7f1ae40fd490 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae259c700 1 --2- 172.21.15.162:0/538428692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ae40fcb30 0x7f1ae4109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae359e700 1 --2- 172.21.15.162:0/538428692 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ae40fd490 0x7f1ae4109c70 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7f1adc001cb0 tx=0x7f1adc004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1acbfff700 10 client.?.objecter ms_handle_connect 0x7f1ae40fd490 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1acbfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1acbfff700 1 -- 172.21.15.162:0/538428692 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc0079b0 con 0x7f1ae40fd490 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1acbfff700 1 -- 172.21.15.162:0/538428692 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1adc0090d0 con 0x7f1ae40fd490 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae900c700 1 -- 172.21.15.162:0/538428692 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad4005040 con 0x7f1ae40fd490 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1acbfff700 1 -- 172.21.15.162:0/538428692 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc00fc20 con 0x7f1ae40fd490 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae900c700 10 client.37970.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae900c700 1 -- 172.21.15.162:0/538428692 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae4114ca0 con 0x7f1ae40fd490 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.875+0000 7f1ae900c700 1 librados: init done 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.877+0000 7f1ae900c700 1 -- 172.21.15.162:0/538428692 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae4106020 con 0x7f1ae40fd490 2024-02-13T00:51:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.877+0000 7f1acbfff700 1 -- 172.21.15.162:0/538428692 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1adc00fd80 con 0x7f1ae40fd490 2024-02-13T00:51:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.877+0000 7f1acbfff700 1 --2- 172.21.15.162:0/538428692 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1acc062c90 0x7f1acc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.877+0000 7f1acbfff700 1 -- 172.21.15.162:0/538428692 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(649..649 src has 449..649) v4 ==== 8559+0+0 (secure 0 0 0) 0x7f1adc07dc30 con 0x7f1ae40fd490 2024-02-13T00:51:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.877+0000 7f1acbfff700 10 client.37970.objecter ms_dispatch 0x7f1ae4001550 osd_map(649..649 src has 449..649) v4 2024-02-13T00:51:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.877+0000 7f1acbfff700 3 client.37970.objecter handle_osd_map got epochs [649,649] > 0 2024-02-13T00:51:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.877+0000 7f1acbfff700 3 client.37970.objecter handle_osd_map decoding full epoch 649 2024-02-13T00:51:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.877+0000 7f1ae2d9d700 1 --2- 172.21.15.162:0/538428692 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1acc062c90 0x7f1acc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.878+0000 7f1acbfff700 20 client.37970.objecter dump_active .. 0 homeless 2024-02-13T00:51:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.878+0000 7f1acbfff700 10 client.37970.objecter ms_handle_connect 0x7f1acc062c90 2024-02-13T00:51:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.878+0000 7f1ae2d9d700 1 --2- 172.21.15.162:0/538428692 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1acc062c90 0x7f1acc065150 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 rx=0x7f1ad40043d0 tx=0x7f1ad400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:14.882+0000 7f1acbfff700 1 -- 172.21.15.162:0/538428692 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1adc050f10 con 0x7f1ae40fd490 2024-02-13T00:51:14.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:14.975 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:14.998 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:15.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:15.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.056+0000 7f1ae900c700 1 -- 172.21.15.162:0/538428692 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1ae405fb80 con 0x7f1acc062c90 2024-02-13T00:51:15.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.070+0000 7f1acbfff700 1 -- 172.21.15.162:0/538428692 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+503853 (secure 0 0 0) 0x7f1ae405fb80 con 0x7f1acc062c90 2024-02-13T00:51:15.080 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:51:15.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 10 librados: watch_flush enter 2024-02-13T00:51:15.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 10 librados: watch_flush exit 2024-02-13T00:51:15.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 20 client.37970.objecter shutdown clearing up homeless session... 2024-02-13T00:51:15.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 10 client.37970.objecter successfully canceled tick 2024-02-13T00:51:15.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 -- 172.21.15.162:0/538428692 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1acc062c90 msgr2=0x7f1acc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:15.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 --2- 172.21.15.162:0/538428692 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1acc062c90 0x7f1acc065150 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 rx=0x7f1ad40043d0 tx=0x7f1ad400b040).stop 2024-02-13T00:51:15.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 -- 172.21.15.162:0/538428692 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ae40fd490 msgr2=0x7f1ae4109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:15.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 --2- 172.21.15.162:0/538428692 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ae40fd490 0x7f1ae4109c70 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7f1adc001cb0 tx=0x7f1adc004c50).stop 2024-02-13T00:51:15.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 -- 172.21.15.162:0/538428692 shutdown_connections 2024-02-13T00:51:15.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 --2- 172.21.15.162:0/538428692 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1acc062c90 0x7f1acc065150 unknown :-1 s=CLOSED pgs=2992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:15.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 --2- 172.21.15.162:0/538428692 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ae40fd490 0x7f1ae4109c70 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:15.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 --2- 172.21.15.162:0/538428692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ae40f73a0 0x7f1ae4109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:15.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 --2- 172.21.15.162:0/538428692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ae40fcb30 0x7f1ae4109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:15.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 -- 172.21.15.162:0/538428692 >> 172.21.15.162:0/538428692 conn(0x7f1ae406eb90 msgr2=0x7f1ae40fa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:15.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 -- 172.21.15.162:0/538428692 shutdown_connections 2024-02-13T00:51:15.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 -- 172.21.15.162:0/538428692 wait complete. 2024-02-13T00:51:15.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:15.074+0000 7f1ae900c700 1 librados: shutdown 2024-02-13T00:51:15.086 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:51:15.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:15.128 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:15.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:14.095137+0000 osd.1 (osd.1) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:15.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:14.095518+0000 osd.1 (osd.1) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:15.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:14.241676+0000 osd.0 (osd.0) 247 : cluster [DBG] 5.b scrub starts 2024-02-13T00:51:15.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:14.244942+0000 osd.0 (osd.0) 248 : cluster [DBG] 5.b scrub ok 2024-02-13T00:51:15.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:14.503341+0000 mgr.y (mgr.14544) 551 : cluster [DBG] pgmap v978: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:15.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:14.520430+0000 osd.1 (osd.1) 271 : cluster [DBG] 6.26 scrub starts 2024-02-13T00:51:15.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:14.521235+0000 osd.1 (osd.1) 272 : cluster [DBG] 6.26 scrub ok 2024-02-13T00:51:15.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:14.534035+0000 mon.a (mon.0) 3238 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-112"}]': finished 2024-02-13T00:51:15.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:14.534107+0000 mon.a (mon.0) 3239 : cluster [DBG] osdmap e649: 8 total, 8 up, 4 in 2024-02-13T00:51:15.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:14.535387+0000 mon.a (mon.0) 3240 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-112", "mode": "writeback"}]: dispatch 2024-02-13T00:51:15.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:15.057802+0000 mgr.y (mgr.14544) 552 : audit [DBG] from='client.37970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:51:15.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:15.526338+0000 mon.a (mon.0) 3241 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:15.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:15.533668+0000 mon.a (mon.0) 3242 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-112", "mode": "writeback"}]': finished 2024-02-13T00:51:15.800 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:15.533748+0000 mon.a (mon.0) 3243 : cluster [DBG] osdmap e650: 8 total, 8 up, 4 in 2024-02-13T00:51:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:14.095137+0000 osd.1 (osd.1) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:14.095518+0000 osd.1 (osd.1) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:14.241676+0000 osd.0 (osd.0) 247 : cluster [DBG] 5.b scrub starts 2024-02-13T00:51:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:14.244942+0000 osd.0 (osd.0) 248 : cluster [DBG] 5.b scrub ok 2024-02-13T00:51:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:14.503341+0000 mgr.y (mgr.14544) 551 : cluster [DBG] pgmap v978: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:14.520430+0000 osd.1 (osd.1) 271 : cluster [DBG] 6.26 scrub starts 2024-02-13T00:51:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:14.521235+0000 osd.1 (osd.1) 272 : cluster [DBG] 6.26 scrub ok 2024-02-13T00:51:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:14.534035+0000 mon.a (mon.0) 3238 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-112"}]': finished 2024-02-13T00:51:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:14.534107+0000 mon.a (mon.0) 3239 : cluster [DBG] osdmap e649: 8 total, 8 up, 4 in 2024-02-13T00:51:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:14.535387+0000 mon.a (mon.0) 3240 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-112", "mode": "writeback"}]: dispatch 2024-02-13T00:51:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:15.057802+0000 mgr.y (mgr.14544) 552 : audit [DBG] from='client.37970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:51:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:15.526338+0000 mon.a (mon.0) 3241 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:15.533668+0000 mon.a (mon.0) 3242 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-112", "mode": "writeback"}]': finished 2024-02-13T00:51:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:15.533748+0000 mon.a (mon.0) 3243 : cluster [DBG] osdmap e650: 8 total, 8 up, 4 in 2024-02-13T00:51:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:14.095137+0000 osd.1 (osd.1) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:14.095518+0000 osd.1 (osd.1) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:14.241676+0000 osd.0 (osd.0) 247 : cluster [DBG] 5.b scrub starts 2024-02-13T00:51:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:14.244942+0000 osd.0 (osd.0) 248 : cluster [DBG] 5.b scrub ok 2024-02-13T00:51:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:14.503341+0000 mgr.y (mgr.14544) 551 : cluster [DBG] pgmap v978: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:14.520430+0000 osd.1 (osd.1) 271 : cluster [DBG] 6.26 scrub starts 2024-02-13T00:51:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:14.521235+0000 osd.1 (osd.1) 272 : cluster [DBG] 6.26 scrub ok 2024-02-13T00:51:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:14.534035+0000 mon.a (mon.0) 3238 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-112"}]': finished 2024-02-13T00:51:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:14.534107+0000 mon.a (mon.0) 3239 : cluster [DBG] osdmap e649: 8 total, 8 up, 4 in 2024-02-13T00:51:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:14.535387+0000 mon.a (mon.0) 3240 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-112", "mode": "writeback"}]: dispatch 2024-02-13T00:51:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:15.057802+0000 mgr.y (mgr.14544) 552 : audit [DBG] from='client.37970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:51:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:15.526338+0000 mon.a (mon.0) 3241 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:15.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:15.533668+0000 mon.a (mon.0) 3242 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-112", "mode": "writeback"}]': finished 2024-02-13T00:51:15.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:15.533748+0000 mon.a (mon.0) 3243 : cluster [DBG] osdmap e650: 8 total, 8 up, 4 in 2024-02-13T00:51:15.865 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":979,"stamp":"2024-02-13T00:51:14.531528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5841,"num_objects":232,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":2307,"num_read_kb":2020,"num_write":927,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":13678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1073,"ondisk_log_size":1073,"up":516,"acting":516,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":612,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1232412,"kb_used_data":14844,"kb_used_omap":0,"kb_used_meta":1217536,"kb_avail":373748196,"statfs":{"total":383980142592,"available":382718152704,"internally_reserved":0,"allocated":15200256,"data_stored":7464297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1246756864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010538"},"pg_stats":[{"pgid":"240.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"240.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.7","version":"624'6","reported_seq":70,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544799+0000","last_change":"2024-02-13T00:50:59.556434+0000","last_active":"2024-02-13T00:51:08.544799+0000","last_peered":"2024-02-13T00:51:08.544799+0000","last_clean":"2024-02-13T00:51:08.544799+0000","last_became_active":"2024-02-13T00:50:14.553744+0000","last_became_peered":"2024-02-13T00:50:14.553744+0000","last_unstale":"2024-02-13T00:51:08.544799+0000","last_undegraded":"2024-02-13T00:51:08.544799+0000","last_fullsized":"2024-02-13T00:51:08.544799+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"240.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.6","version":"0'0","reported_seq":61,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603569+0000","last_change":"2024-02-13T00:50:59.547240+0000","last_active":"2024-02-13T00:51:08.603569+0000","last_peered":"2024-02-13T00:51:08.603569+0000","last_clean":"2024-02-13T00:51:08.603569+0000","last_became_active":"2024-02-13T00:50:14.552756+0000","last_became_peered":"2024-02-13T00:50:14.552756+0000","last_unstale":"2024-02-13T00:51:08.603569+0000","last_undegraded":"2024-02-13T00:51:08.603569+0000","last_fullsized":"2024-02-13T00:51:08.603569+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"240.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.5","version":"642'7","reported_seq":71,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603880+0000","last_change":"2024-02-13T00:50:59.546762+0000","last_active":"2024-02-13T00:51:08.603880+0000","last_peered":"2024-02-13T00:51:08.603880+0000","last_clean":"2024-02-13T00:51:08.603880+0000","last_became_active":"2024-02-13T00:50:14.550644+0000","last_became_peered":"2024-02-13T00:50:14.550644+0000","last_unstale":"2024-02-13T00:51:08.603880+0000","last_undegraded":"2024-02-13T00:51:08.603880+0000","last_fullsized":"2024-02-13T00:51:08.603880+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"240.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.4","version":"624'5","reported_seq":67,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602608+0000","last_change":"2024-02-13T00:50:59.540704+0000","last_active":"2024-02-13T00:51:08.602608+0000","last_peered":"2024-02-13T00:51:08.602608+0000","last_clean":"2024-02-13T00:51:08.602608+0000","last_became_active":"2024-02-13T00:50:14.549625+0000","last_became_peered":"2024-02-13T00:50:14.549625+0000","last_unstale":"2024-02-13T00:51:08.602608+0000","last_undegraded":"2024-02-13T00:51:08.602608+0000","last_fullsized":"2024-02-13T00:51:08.602608+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"240.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.3","version":"0'0","reported_seq":61,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602453+0000","last_change":"2024-02-13T00:50:59.545667+0000","last_active":"2024-02-13T00:51:08.602453+0000","last_peered":"2024-02-13T00:51:08.602453+0000","last_clean":"2024-02-13T00:51:08.602453+0000","last_became_active":"2024-02-13T00:50:14.549053+0000","last_became_peered":"2024-02-13T00:50:14.549053+0000","last_unstale":"2024-02-13T00:51:08.602453+0000","last_undegraded":"2024-02-13T00:51:08.602453+0000","last_fullsized":"2024-02-13T00:51:08.602453+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"240.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.2","version":"634'10","reported_seq":75,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602349+0000","last_change":"2024-02-13T00:51:01.548780+0000","last_active":"2024-02-13T00:51:08.602349+0000","last_peered":"2024-02-13T00:51:08.602349+0000","last_clean":"2024-02-13T00:51:08.602349+0000","last_became_active":"2024-02-13T00:50:14.552162+0000","last_became_peered":"2024-02-13T00:50:14.552162+0000","last_unstale":"2024-02-13T00:51:08.602349+0000","last_undegraded":"2024-02-13T00:51:08.602349+0000","last_fullsized":"2024-02-13T00:51:08.602349+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"240.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.1","version":"0'0","reported_seq":62,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.543759+0000","last_change":"2024-02-13T00:50:59.553878+0000","last_active":"2024-02-13T00:51:08.543759+0000","last_peered":"2024-02-13T00:51:08.543759+0000","last_clean":"2024-02-13T00:51:08.543759+0000","last_became_active":"2024-02-13T00:50:14.553460+0000","last_became_peered":"2024-02-13T00:50:14.553460+0000","last_unstale":"2024-02-13T00:51:08.543759+0000","last_undegraded":"2024-02-13T00:51:08.543759+0000","last_fullsized":"2024-02-13T00:51:08.543759+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"240.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:51:11.515400+0000","last_change":"2024-02-13T00:51:11.515400+0000","last_active":"2024-02-13T00:51:11.515400+0000","last_peered":"2024-02-13T00:51:11.515400+0000","last_clean":"2024-02-13T00:51:11.515400+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:51:11.515400+0000","last_undegraded":"2024-02-13T00:51:11.515400+0000","last_fullsized":"2024-02-13T00:51:11.515400+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:11.515400+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.515400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.0","version":"608'1","reported_seq":64,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602163+0000","last_change":"2024-02-13T00:50:59.542973+0000","last_active":"2024-02-13T00:51:08.602163+0000","last_peered":"2024-02-13T00:51:08.602163+0000","last_clean":"2024-02-13T00:51:08.602163+0000","last_became_active":"2024-02-13T00:50:14.551016+0000","last_became_peered":"2024-02-13T00:50:14.551016+0000","last_unstale":"2024-02-13T00:51:08.602163+0000","last_undegraded":"2024-02-13T00:51:08.602163+0000","last_fullsized":"2024-02-13T00:51:08.602163+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"6.29","version":"0'0","reported_seq":549,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602213+0000","last_change":"2024-02-13T00:50:32.100672+0000","last_active":"2024-02-13T00:51:08.602213+0000","last_peered":"2024-02-13T00:51:08.602213+0000","last_clean":"2024-02-13T00:51:08.602213+0000","last_became_active":"2024-02-13T00:46:01.445363+0000","last_became_peered":"2024-02-13T00:46:01.445363+0000","last_unstale":"2024-02-13T00:51:08.602213+0000","last_undegraded":"2024-02-13T00:51:08.602213+0000","last_fullsized":"2024-02-13T00:51:08.602213+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:32.100607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:50:32.100607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":468,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602947+0000","last_change":"2024-02-13T00:46:01.449894+0000","last_active":"2024-02-13T00:51:08.602947+0000","last_peered":"2024-02-13T00:51:08.602947+0000","last_clean":"2024-02-13T00:51:08.602947+0000","last_became_active":"2024-02-13T00:46:01.449570+0000","last_became_peered":"2024-02-13T00:46:01.449570+0000","last_unstale":"2024-02-13T00:51:08.602947+0000","last_undegraded":"2024-02-13T00:51:08.602947+0000","last_fullsized":"2024-02-13T00:51:08.602947+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":335,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544746+0000","last_change":"2024-02-13T00:46:01.445141+0000","last_active":"2024-02-13T00:51:08.544746+0000","last_peered":"2024-02-13T00:51:08.544746+0000","last_clean":"2024-02-13T00:51:08.544746+0000","last_became_active":"2024-02-13T00:46:01.444729+0000","last_became_peered":"2024-02-13T00:46:01.444729+0000","last_unstale":"2024-02-13T00:51:08.544746+0000","last_undegraded":"2024-02-13T00:51:08.544746+0000","last_fullsized":"2024-02-13T00:51:08.544746+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.198293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.198293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":425,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.543846+0000","last_change":"2024-02-13T00:50:16.677491+0000","last_active":"2024-02-13T00:51:08.543846+0000","last_peered":"2024-02-13T00:51:08.543846+0000","last_clean":"2024-02-13T00:51:08.543846+0000","last_became_active":"2024-02-13T00:45:15.426109+0000","last_became_peered":"2024-02-13T00:45:15.426109+0000","last_unstale":"2024-02-13T00:51:08.543846+0000","last_undegraded":"2024-02-13T00:51:08.543846+0000","last_fullsized":"2024-02-13T00:51:08.543846+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:16.677431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:16.677431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":329,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604947+0000","last_change":"2024-02-13T00:50:22.469511+0000","last_active":"2024-02-13T00:51:08.604947+0000","last_peered":"2024-02-13T00:51:08.604947+0000","last_clean":"2024-02-13T00:51:08.604947+0000","last_became_active":"2024-02-13T00:44:39.525161+0000","last_became_peered":"2024-02-13T00:44:39.525161+0000","last_unstale":"2024-02-13T00:51:08.604947+0000","last_undegraded":"2024-02-13T00:51:08.604947+0000","last_fullsized":"2024-02-13T00:51:08.604947+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.469431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.469431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":423,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544152+0000","last_change":"2024-02-13T00:48:06.705421+0000","last_active":"2024-02-13T00:51:08.544152+0000","last_peered":"2024-02-13T00:51:08.544152+0000","last_clean":"2024-02-13T00:51:08.544152+0000","last_became_active":"2024-02-13T00:48:06.705121+0000","last_became_peered":"2024-02-13T00:48:06.705121+0000","last_unstale":"2024-02-13T00:51:08.544152+0000","last_undegraded":"2024-02-13T00:51:08.544152+0000","last_fullsized":"2024-02-13T00:51:08.544152+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":342,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605363+0000","last_change":"2024-02-13T00:46:01.459772+0000","last_active":"2024-02-13T00:51:08.605363+0000","last_peered":"2024-02-13T00:51:08.605363+0000","last_clean":"2024-02-13T00:51:08.605363+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:51:08.605363+0000","last_undegraded":"2024-02-13T00:51:08.605363+0000","last_fullsized":"2024-02-13T00:51:08.605363+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":601,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604742+0000","last_change":"2024-02-13T00:46:01.463062+0000","last_active":"2024-02-13T00:51:08.604742+0000","last_peered":"2024-02-13T00:51:08.604742+0000","last_clean":"2024-02-13T00:51:08.604742+0000","last_became_active":"2024-02-13T00:46:01.462319+0000","last_became_peered":"2024-02-13T00:46:01.462319+0000","last_unstale":"2024-02-13T00:51:08.604742+0000","last_undegraded":"2024-02-13T00:51:08.604742+0000","last_fullsized":"2024-02-13T00:51:08.604742+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":329,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605212+0000","last_change":"2024-02-13T00:49:48.070260+0000","last_active":"2024-02-13T00:51:08.605212+0000","last_peered":"2024-02-13T00:51:08.605212+0000","last_clean":"2024-02-13T00:51:08.605212+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:51:08.605212+0000","last_undegraded":"2024-02-13T00:51:08.605212+0000","last_fullsized":"2024-02-13T00:51:08.605212+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:48.070195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:49:48.070195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":411,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546146+0000","last_change":"2024-02-13T00:46:01.455629+0000","last_active":"2024-02-13T00:51:08.546146+0000","last_peered":"2024-02-13T00:51:08.546146+0000","last_clean":"2024-02-13T00:51:08.546146+0000","last_became_active":"2024-02-13T00:46:01.452801+0000","last_became_peered":"2024-02-13T00:46:01.452801+0000","last_unstale":"2024-02-13T00:51:08.546146+0000","last_undegraded":"2024-02-13T00:51:08.546146+0000","last_fullsized":"2024-02-13T00:51:08.546146+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":563,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602393+0000","last_change":"2024-02-13T00:45:39.562095+0000","last_active":"2024-02-13T00:51:08.602393+0000","last_peered":"2024-02-13T00:51:08.602393+0000","last_clean":"2024-02-13T00:51:08.602393+0000","last_became_active":"2024-02-13T00:44:39.505121+0000","last_became_peered":"2024-02-13T00:44:39.505121+0000","last_unstale":"2024-02-13T00:51:08.602393+0000","last_undegraded":"2024-02-13T00:51:08.602393+0000","last_fullsized":"2024-02-13T00:51:08.602393+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:39.561994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:39.561994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"534'16","reported_seq":446,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:09.367747+0000","last_change":"2024-02-13T00:49:07.494629+0000","last_active":"2024-02-13T00:51:09.367747+0000","last_peered":"2024-02-13T00:51:09.367747+0000","last_clean":"2024-02-13T00:51:09.367747+0000","last_became_active":"2024-02-13T00:49:07.494388+0000","last_became_peered":"2024-02-13T00:49:07.494388+0000","last_unstale":"2024-02-13T00:51:09.367747+0000","last_undegraded":"2024-02-13T00:51:09.367747+0000","last_fullsized":"2024-02-13T00:51:09.367747+0000","mapping_epoch":359,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:45:53.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.115260+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":668,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602691+0000","last_change":"2024-02-13T00:47:03.491720+0000","last_active":"2024-02-13T00:51:08.602691+0000","last_peered":"2024-02-13T00:51:08.602691+0000","last_clean":"2024-02-13T00:51:08.602691+0000","last_became_active":"2024-02-13T00:46:01.438056+0000","last_became_peered":"2024-02-13T00:46:01.438056+0000","last_unstale":"2024-02-13T00:51:08.602691+0000","last_undegraded":"2024-02-13T00:51:08.602691+0000","last_fullsized":"2024-02-13T00:51:08.602691+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:47:03.491647+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":638,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604757+0000","last_change":"2024-02-13T00:48:23.379872+0000","last_active":"2024-02-13T00:51:08.604757+0000","last_peered":"2024-02-13T00:51:08.604757+0000","last_clean":"2024-02-13T00:51:08.604757+0000","last_became_active":"2024-02-13T00:46:01.460811+0000","last_became_peered":"2024-02-13T00:46:01.460811+0000","last_unstale":"2024-02-13T00:51:08.604757+0000","last_undegraded":"2024-02-13T00:51:08.604757+0000","last_fullsized":"2024-02-13T00:51:08.604757+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:23.379773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:23.379773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":544,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602806+0000","last_change":"2024-02-13T00:47:34.332791+0000","last_active":"2024-02-13T00:51:08.602806+0000","last_peered":"2024-02-13T00:51:08.602806+0000","last_clean":"2024-02-13T00:51:08.602806+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:51:08.602806+0000","last_undegraded":"2024-02-13T00:51:08.602806+0000","last_fullsized":"2024-02-13T00:51:08.602806+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:34.332704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:34.332704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":582,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603082+0000","last_change":"2024-02-13T00:46:01.448237+0000","last_active":"2024-02-13T00:51:08.603082+0000","last_peered":"2024-02-13T00:51:08.603082+0000","last_clean":"2024-02-13T00:51:08.603082+0000","last_became_active":"2024-02-13T00:46:01.446660+0000","last_became_peered":"2024-02-13T00:46:01.446660+0000","last_unstale":"2024-02-13T00:51:08.603082+0000","last_undegraded":"2024-02-13T00:51:08.603082+0000","last_fullsized":"2024-02-13T00:51:08.603082+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.260871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.260871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":642,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603181+0000","last_change":"2024-02-13T00:47:41.808333+0000","last_active":"2024-02-13T00:51:08.603181+0000","last_peered":"2024-02-13T00:51:08.603181+0000","last_clean":"2024-02-13T00:51:08.603181+0000","last_became_active":"2024-02-13T00:37:43.233821+0000","last_became_peered":"2024-02-13T00:37:43.233821+0000","last_unstale":"2024-02-13T00:51:08.603181+0000","last_undegraded":"2024-02-13T00:51:08.603181+0000","last_fullsized":"2024-02-13T00:51:08.603181+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-13T00:47:41.808259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:41.808259+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":626,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546750+0000","last_change":"2024-02-13T00:48:24.823190+0000","last_active":"2024-02-13T00:51:08.546750+0000","last_peered":"2024-02-13T00:51:08.546750+0000","last_clean":"2024-02-13T00:51:08.546750+0000","last_became_active":"2024-02-13T00:44:50.429994+0000","last_became_peered":"2024-02-13T00:44:50.429994+0000","last_unstale":"2024-02-13T00:51:08.546750+0000","last_undegraded":"2024-02-13T00:51:08.546750+0000","last_fullsized":"2024-02-13T00:51:08.546750+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:24.823107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.823107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":526,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602535+0000","last_change":"2024-02-13T00:49:39.788846+0000","last_active":"2024-02-13T00:51:08.602535+0000","last_peered":"2024-02-13T00:51:08.602535+0000","last_clean":"2024-02-13T00:51:08.602535+0000","last_became_active":"2024-02-13T00:49:03.971407+0000","last_became_peered":"2024-02-13T00:49:03.971407+0000","last_unstale":"2024-02-13T00:51:08.602535+0000","last_undegraded":"2024-02-13T00:51:08.602535+0000","last_fullsized":"2024-02-13T00:51:08.602535+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.788783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.788783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":625,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602546+0000","last_change":"2024-02-13T00:49:41.837871+0000","last_active":"2024-02-13T00:51:08.602546+0000","last_peered":"2024-02-13T00:51:08.602546+0000","last_clean":"2024-02-13T00:51:08.602546+0000","last_became_active":"2024-02-13T00:45:15.434853+0000","last_became_peered":"2024-02-13T00:45:15.434853+0000","last_unstale":"2024-02-13T00:51:08.602546+0000","last_undegraded":"2024-02-13T00:51:08.602546+0000","last_fullsized":"2024-02-13T00:51:08.602546+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.837810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.837810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":246,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602081+0000","last_change":"2024-02-13T00:46:01.455201+0000","last_active":"2024-02-13T00:51:08.602081+0000","last_peered":"2024-02-13T00:51:08.602081+0000","last_clean":"2024-02-13T00:51:08.602081+0000","last_became_active":"2024-02-13T00:46:01.454882+0000","last_became_peered":"2024-02-13T00:46:01.454882+0000","last_unstale":"2024-02-13T00:51:08.602081+0000","last_undegraded":"2024-02-13T00:51:08.602081+0000","last_fullsized":"2024-02-13T00:51:08.602081+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:35.197045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:35.197045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":432,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603441+0000","last_change":"2024-02-13T00:46:09.256189+0000","last_active":"2024-02-13T00:51:08.603441+0000","last_peered":"2024-02-13T00:51:08.603441+0000","last_clean":"2024-02-13T00:51:08.603441+0000","last_became_active":"2024-02-13T00:46:01.459978+0000","last_became_peered":"2024-02-13T00:46:01.459978+0000","last_unstale":"2024-02-13T00:51:08.603441+0000","last_undegraded":"2024-02-13T00:51:08.603441+0000","last_fullsized":"2024-02-13T00:51:08.603441+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'35","last_scrub_stamp":"2024-02-13T00:44:24.196214+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:44:24.196214+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":256,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603512+0000","last_change":"2024-02-13T00:46:30.641539+0000","last_active":"2024-02-13T00:51:08.603512+0000","last_peered":"2024-02-13T00:51:08.603512+0000","last_clean":"2024-02-13T00:51:08.603512+0000","last_became_active":"2024-02-13T00:46:01.444548+0000","last_became_peered":"2024-02-13T00:46:01.444548+0000","last_unstale":"2024-02-13T00:51:08.603512+0000","last_undegraded":"2024-02-13T00:51:08.603512+0000","last_fullsized":"2024-02-13T00:51:08.603512+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.641460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.641460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":356,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602366+0000","last_change":"2024-02-13T00:48:54.498301+0000","last_active":"2024-02-13T00:51:08.602366+0000","last_peered":"2024-02-13T00:51:08.602366+0000","last_clean":"2024-02-13T00:51:08.602366+0000","last_became_active":"2024-02-13T00:48:54.497977+0000","last_became_peered":"2024-02-13T00:48:54.497977+0000","last_unstale":"2024-02-13T00:51:08.602366+0000","last_undegraded":"2024-02-13T00:51:08.602366+0000","last_fullsized":"2024-02-13T00:51:08.602366+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.592672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":460,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602853+0000","last_change":"2024-02-13T00:45:15.429026+0000","last_active":"2024-02-13T00:51:08.602853+0000","last_peered":"2024-02-13T00:51:08.602853+0000","last_clean":"2024-02-13T00:51:08.602853+0000","last_became_active":"2024-02-13T00:45:15.427945+0000","last_became_peered":"2024-02-13T00:45:15.427945+0000","last_unstale":"2024-02-13T00:51:08.602853+0000","last_undegraded":"2024-02-13T00:51:08.602853+0000","last_fullsized":"2024-02-13T00:51:08.602853+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:55.626874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:55.626874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":566,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546608+0000","last_change":"2024-02-13T00:45:26.255907+0000","last_active":"2024-02-13T00:51:08.546608+0000","last_peered":"2024-02-13T00:51:08.546608+0000","last_clean":"2024-02-13T00:51:08.546608+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:51:08.546608+0000","last_undegraded":"2024-02-13T00:51:08.546608+0000","last_fullsized":"2024-02-13T00:51:08.546608+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.255842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.255842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"467'31","reported_seq":403,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546666+0000","last_change":"2024-02-13T00:48:31.782821+0000","last_active":"2024-02-13T00:51:08.546666+0000","last_peered":"2024-02-13T00:51:08.546666+0000","last_clean":"2024-02-13T00:51:08.546666+0000","last_became_active":"2024-02-13T00:44:50.432807+0000","last_became_peered":"2024-02-13T00:44:50.432807+0000","last_unstale":"2024-02-13T00:51:08.546666+0000","last_undegraded":"2024-02-13T00:51:08.546666+0000","last_fullsized":"2024-02-13T00:51:08.546666+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'31","last_scrub_stamp":"2024-02-13T00:48:31.782729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.782729+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":415,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602670+0000","last_change":"2024-02-13T00:46:04.708510+0000","last_active":"2024-02-13T00:51:08.602670+0000","last_peered":"2024-02-13T00:51:08.602670+0000","last_clean":"2024-02-13T00:51:08.602670+0000","last_became_active":"2024-02-13T00:46:01.447801+0000","last_became_peered":"2024-02-13T00:46:01.447801+0000","last_unstale":"2024-02-13T00:51:08.602670+0000","last_undegraded":"2024-02-13T00:51:08.602670+0000","last_fullsized":"2024-02-13T00:51:08.602670+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:45:24.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:24.269595+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":624,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546817+0000","last_change":"2024-02-13T00:48:49.612735+0000","last_active":"2024-02-13T00:51:08.546817+0000","last_peered":"2024-02-13T00:51:08.546817+0000","last_clean":"2024-02-13T00:51:08.546817+0000","last_became_active":"2024-02-13T00:48:49.612437+0000","last_became_peered":"2024-02-13T00:48:49.612437+0000","last_unstale":"2024-02-13T00:51:08.546817+0000","last_undegraded":"2024-02-13T00:51:08.546817+0000","last_fullsized":"2024-02-13T00:51:08.546817+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.793634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":609,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604815+0000","last_change":"2024-02-13T00:44:50.441408+0000","last_active":"2024-02-13T00:51:08.604815+0000","last_peered":"2024-02-13T00:51:08.604815+0000","last_clean":"2024-02-13T00:51:08.604815+0000","last_became_active":"2024-02-13T00:44:50.440301+0000","last_became_peered":"2024-02-13T00:44:50.440301+0000","last_unstale":"2024-02-13T00:51:08.604815+0000","last_undegraded":"2024-02-13T00:51:08.604815+0000","last_fullsized":"2024-02-13T00:51:08.604815+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:25.502311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:44:25.502311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":268,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603274+0000","last_change":"2024-02-13T00:50:22.925676+0000","last_active":"2024-02-13T00:51:08.603274+0000","last_peered":"2024-02-13T00:51:08.603274+0000","last_clean":"2024-02-13T00:51:08.603274+0000","last_became_active":"2024-02-13T00:48:06.704718+0000","last_became_peered":"2024-02-13T00:48:06.704718+0000","last_unstale":"2024-02-13T00:51:08.603274+0000","last_undegraded":"2024-02-13T00:51:08.603274+0000","last_fullsized":"2024-02-13T00:51:08.603274+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.925604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.925604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":356,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604880+0000","last_change":"2024-02-13T00:49:03.294585+0000","last_active":"2024-02-13T00:51:08.604880+0000","last_peered":"2024-02-13T00:51:08.604880+0000","last_clean":"2024-02-13T00:51:08.604880+0000","last_became_active":"2024-02-13T00:44:39.523138+0000","last_became_peered":"2024-02-13T00:44:39.523138+0000","last_unstale":"2024-02-13T00:51:08.604880+0000","last_undegraded":"2024-02-13T00:51:08.604880+0000","last_fullsized":"2024-02-13T00:51:08.604880+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:49:03.294499+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:49:03.294499+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":635,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544860+0000","last_change":"2024-02-13T00:48:59.111910+0000","last_active":"2024-02-13T00:51:08.544860+0000","last_peered":"2024-02-13T00:51:08.544860+0000","last_clean":"2024-02-13T00:51:08.544860+0000","last_became_active":"2024-02-13T00:46:01.452674+0000","last_became_peered":"2024-02-13T00:46:01.452674+0000","last_unstale":"2024-02-13T00:51:08.544860+0000","last_undegraded":"2024-02-13T00:51:08.544860+0000","last_fullsized":"2024-02-13T00:51:08.544860+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:59.111828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:59.111828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"511'13","reported_seq":360,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:09.367653+0000","last_change":"2024-02-13T00:48:45.592523+0000","last_active":"2024-02-13T00:51:09.367653+0000","last_peered":"2024-02-13T00:51:09.367653+0000","last_clean":"2024-02-13T00:51:09.367653+0000","last_became_active":"2024-02-13T00:48:45.591249+0000","last_became_peered":"2024-02-13T00:48:45.591249+0000","last_unstale":"2024-02-13T00:51:09.367653+0000","last_undegraded":"2024-02-13T00:51:09.367653+0000","last_fullsized":"2024-02-13T00:51:09.367653+0000","mapping_epoch":401,"log_start":"404'11","ondisk_log_start":"404'11","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'11","last_scrub_stamp":"2024-02-13T00:46:37.662413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.662413+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":607,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546279+0000","last_change":"2024-02-13T00:51:05.644627+0000","last_active":"2024-02-13T00:51:08.546279+0000","last_peered":"2024-02-13T00:51:08.546279+0000","last_clean":"2024-02-13T00:51:08.546279+0000","last_became_active":"2024-02-13T00:46:01.455274+0000","last_became_peered":"2024-02-13T00:46:01.455274+0000","last_unstale":"2024-02-13T00:51:08.546279+0000","last_undegraded":"2024-02-13T00:51:08.546279+0000","last_fullsized":"2024-02-13T00:51:08.546279+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:51:05.644558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:05.644558+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":416,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.547301+0000","last_change":"2024-02-13T00:50:26.830352+0000","last_active":"2024-02-13T00:51:08.547301+0000","last_peered":"2024-02-13T00:51:08.547301+0000","last_clean":"2024-02-13T00:51:08.547301+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:51:08.547301+0000","last_undegraded":"2024-02-13T00:51:08.547301+0000","last_fullsized":"2024-02-13T00:51:08.547301+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:26.830270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:26.830270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"467'27","reported_seq":724,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.547224+0000","last_change":"2024-02-13T00:48:54.043163+0000","last_active":"2024-02-13T00:51:08.547224+0000","last_peered":"2024-02-13T00:51:08.547224+0000","last_clean":"2024-02-13T00:51:08.547224+0000","last_became_active":"2024-02-13T00:46:01.448705+0000","last_became_peered":"2024-02-13T00:46:01.448705+0000","last_unstale":"2024-02-13T00:51:08.547224+0000","last_undegraded":"2024-02-13T00:51:08.547224+0000","last_fullsized":"2024-02-13T00:51:08.547224+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:48:54.043088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.043088+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":667,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.545264+0000","last_change":"2024-02-13T00:49:37.053462+0000","last_active":"2024-02-13T00:51:08.545264+0000","last_peered":"2024-02-13T00:51:08.545264+0000","last_clean":"2024-02-13T00:51:08.545264+0000","last_became_active":"2024-02-13T00:45:15.445228+0000","last_became_peered":"2024-02-13T00:45:15.445228+0000","last_unstale":"2024-02-13T00:51:08.545264+0000","last_undegraded":"2024-02-13T00:51:08.545264+0000","last_fullsized":"2024-02-13T00:51:08.545264+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:49:37.053398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.053398+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"507'13","reported_seq":358,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:09.367696+0000","last_change":"2024-02-13T00:48:41.526587+0000","last_active":"2024-02-13T00:51:09.367696+0000","last_peered":"2024-02-13T00:51:09.367696+0000","last_clean":"2024-02-13T00:51:09.367696+0000","last_became_active":"2024-02-13T00:48:41.525262+0000","last_became_peered":"2024-02-13T00:48:41.525262+0000","last_unstale":"2024-02-13T00:51:09.367696+0000","last_undegraded":"2024-02-13T00:51:09.367696+0000","last_fullsized":"2024-02-13T00:51:09.367696+0000","mapping_epoch":401,"log_start":"467'11","ondisk_log_start":"467'11","created":60,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:47:49.687727+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":257,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546486+0000","last_change":"2024-02-13T00:50:19.734411+0000","last_active":"2024-02-13T00:51:08.546486+0000","last_peered":"2024-02-13T00:51:08.546486+0000","last_clean":"2024-02-13T00:51:08.546486+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:51:08.546486+0000","last_undegraded":"2024-02-13T00:51:08.546486+0000","last_fullsized":"2024-02-13T00:51:08.546486+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:19.734320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:50:19.734320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":563,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605103+0000","last_change":"2024-02-13T00:46:01.456531+0000","last_active":"2024-02-13T00:51:08.605103+0000","last_peered":"2024-02-13T00:51:08.605103+0000","last_clean":"2024-02-13T00:51:08.605103+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:51:08.605103+0000","last_undegraded":"2024-02-13T00:51:08.605103+0000","last_fullsized":"2024-02-13T00:51:08.605103+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:44:59.963616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":355,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604841+0000","last_change":"2024-02-13T00:48:57.386335+0000","last_active":"2024-02-13T00:51:08.604841+0000","last_peered":"2024-02-13T00:51:08.604841+0000","last_clean":"2024-02-13T00:51:08.604841+0000","last_became_active":"2024-02-13T00:44:39.521984+0000","last_became_peered":"2024-02-13T00:44:39.521984+0000","last_unstale":"2024-02-13T00:51:08.604841+0000","last_undegraded":"2024-02-13T00:51:08.604841+0000","last_fullsized":"2024-02-13T00:51:08.604841+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:57.386254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:57.386254+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":446,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602112+0000","last_change":"2024-02-13T00:46:23.757417+0000","last_active":"2024-02-13T00:51:08.602112+0000","last_peered":"2024-02-13T00:51:08.602112+0000","last_clean":"2024-02-13T00:51:08.602112+0000","last_became_active":"2024-02-13T00:45:15.419239+0000","last_became_peered":"2024-02-13T00:45:15.419239+0000","last_unstale":"2024-02-13T00:51:08.602112+0000","last_undegraded":"2024-02-13T00:51:08.602112+0000","last_fullsized":"2024-02-13T00:51:08.602112+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:23.757351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:23.757351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"504'16","reported_seq":599,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:09.367742+0000","last_change":"2024-02-13T00:49:45.051624+0000","last_active":"2024-02-13T00:51:09.367742+0000","last_peered":"2024-02-13T00:51:09.367742+0000","last_clean":"2024-02-13T00:51:09.367742+0000","last_became_active":"2024-02-13T00:48:37.480010+0000","last_became_peered":"2024-02-13T00:48:37.480010+0000","last_unstale":"2024-02-13T00:51:09.367742+0000","last_undegraded":"2024-02-13T00:51:09.367742+0000","last_fullsized":"2024-02-13T00:51:09.367742+0000","mapping_epoch":401,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'16","last_scrub_stamp":"2024-02-13T00:49:45.051539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:49:45.051539+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":347,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604795+0000","last_change":"2024-02-13T00:48:31.463534+0000","last_active":"2024-02-13T00:51:08.604795+0000","last_peered":"2024-02-13T00:51:08.604795+0000","last_clean":"2024-02-13T00:51:08.604795+0000","last_became_active":"2024-02-13T00:46:01.462717+0000","last_became_peered":"2024-02-13T00:46:01.462717+0000","last_unstale":"2024-02-13T00:51:08.604795+0000","last_undegraded":"2024-02-13T00:51:08.604795+0000","last_fullsized":"2024-02-13T00:51:08.604795+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:31.463442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.463442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":520,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602030+0000","last_change":"2024-02-13T00:49:00.881498+0000","last_active":"2024-02-13T00:51:08.602030+0000","last_peered":"2024-02-13T00:51:08.602030+0000","last_clean":"2024-02-13T00:51:08.602030+0000","last_became_active":"2024-02-13T00:46:01.456959+0000","last_became_peered":"2024-02-13T00:46:01.456959+0000","last_unstale":"2024-02-13T00:51:08.602030+0000","last_undegraded":"2024-02-13T00:51:08.602030+0000","last_fullsized":"2024-02-13T00:51:08.602030+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:00.881431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:00.881431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"263'10","reported_seq":655,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:09.367461+0000","last_change":"2024-02-13T00:47:02.949483+0000","last_active":"2024-02-13T00:51:09.367461+0000","last_peered":"2024-02-13T00:51:09.367461+0000","last_clean":"2024-02-13T00:51:09.367461+0000","last_became_active":"2024-02-13T00:43:35.454087+0000","last_became_peered":"2024-02-13T00:43:35.454087+0000","last_unstale":"2024-02-13T00:51:09.367461+0000","last_undegraded":"2024-02-13T00:51:09.367461+0000","last_fullsized":"2024-02-13T00:51:09.367461+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.949406+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":619,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603128+0000","last_change":"2024-02-13T00:49:46.101991+0000","last_active":"2024-02-13T00:51:08.603128+0000","last_peered":"2024-02-13T00:51:08.603128+0000","last_clean":"2024-02-13T00:51:08.603128+0000","last_became_active":"2024-02-13T00:46:01.449654+0000","last_became_peered":"2024-02-13T00:46:01.449654+0000","last_unstale":"2024-02-13T00:51:08.603128+0000","last_undegraded":"2024-02-13T00:51:08.603128+0000","last_fullsized":"2024-02-13T00:51:08.603128+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'34","last_scrub_stamp":"2024-02-13T00:49:46.101874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.101874+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":347,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605083+0000","last_change":"2024-02-13T00:49:47.075845+0000","last_active":"2024-02-13T00:51:08.605083+0000","last_peered":"2024-02-13T00:51:08.605083+0000","last_clean":"2024-02-13T00:51:08.605083+0000","last_became_active":"2024-02-13T00:45:15.446443+0000","last_became_peered":"2024-02-13T00:45:15.446443+0000","last_unstale":"2024-02-13T00:51:08.605083+0000","last_undegraded":"2024-02-13T00:51:08.605083+0000","last_fullsized":"2024-02-13T00:51:08.605083+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:47.075741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.075741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":342,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605000+0000","last_change":"2024-02-13T00:45:15.436286+0000","last_active":"2024-02-13T00:51:08.605000+0000","last_peered":"2024-02-13T00:51:08.605000+0000","last_clean":"2024-02-13T00:51:08.605000+0000","last_became_active":"2024-02-13T00:45:15.434559+0000","last_became_peered":"2024-02-13T00:45:15.434559+0000","last_unstale":"2024-02-13T00:51:08.605000+0000","last_undegraded":"2024-02-13T00:51:08.605000+0000","last_fullsized":"2024-02-13T00:51:08.605000+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:53.443280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:53.443280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":370,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.601983+0000","last_change":"2024-02-13T00:49:36.841674+0000","last_active":"2024-02-13T00:51:08.601983+0000","last_peered":"2024-02-13T00:51:08.601983+0000","last_clean":"2024-02-13T00:51:08.601983+0000","last_became_active":"2024-02-13T00:46:01.444661+0000","last_became_peered":"2024-02-13T00:46:01.444661+0000","last_unstale":"2024-02-13T00:51:08.601983+0000","last_undegraded":"2024-02-13T00:51:08.601983+0000","last_fullsized":"2024-02-13T00:51:08.601983+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:36.841508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:36.841508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":411,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602172+0000","last_change":"2024-02-13T00:48:18.751148+0000","last_active":"2024-02-13T00:51:08.602172+0000","last_peered":"2024-02-13T00:51:08.602172+0000","last_clean":"2024-02-13T00:51:08.602172+0000","last_became_active":"2024-02-13T00:46:01.453842+0000","last_became_peered":"2024-02-13T00:46:01.453842+0000","last_unstale":"2024-02-13T00:51:08.602172+0000","last_undegraded":"2024-02-13T00:51:08.602172+0000","last_fullsized":"2024-02-13T00:51:08.602172+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:18.751081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:18.751081+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"467'42","reported_seq":572,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602008+0000","last_change":"2024-02-13T00:48:24.675807+0000","last_active":"2024-02-13T00:51:08.602008+0000","last_peered":"2024-02-13T00:51:08.602008+0000","last_clean":"2024-02-13T00:51:08.602008+0000","last_became_active":"2024-02-13T00:46:01.445058+0000","last_became_peered":"2024-02-13T00:46:01.445058+0000","last_unstale":"2024-02-13T00:51:08.602008+0000","last_undegraded":"2024-02-13T00:51:08.602008+0000","last_fullsized":"2024-02-13T00:51:08.602008+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'42","last_scrub_stamp":"2024-02-13T00:48:24.675687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.675687+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":635,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546174+0000","last_change":"2024-02-13T00:47:09.940673+0000","last_active":"2024-02-13T00:51:08.546174+0000","last_peered":"2024-02-13T00:51:08.546174+0000","last_clean":"2024-02-13T00:51:08.546174+0000","last_became_active":"2024-02-13T00:44:39.503590+0000","last_became_peered":"2024-02-13T00:44:39.503590+0000","last_unstale":"2024-02-13T00:51:08.546174+0000","last_undegraded":"2024-02-13T00:51:08.546174+0000","last_fullsized":"2024-02-13T00:51:08.546174+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:09.940605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:09.940605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":577,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602852+0000","last_change":"2024-02-13T00:46:01.448313+0000","last_active":"2024-02-13T00:51:08.602852+0000","last_peered":"2024-02-13T00:51:08.602852+0000","last_clean":"2024-02-13T00:51:08.602852+0000","last_became_active":"2024-02-13T00:46:01.447815+0000","last_became_peered":"2024-02-13T00:46:01.447815+0000","last_unstale":"2024-02-13T00:51:08.602852+0000","last_undegraded":"2024-02-13T00:51:08.602852+0000","last_fullsized":"2024-02-13T00:51:08.602852+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":569,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602912+0000","last_change":"2024-02-13T00:48:35.434713+0000","last_active":"2024-02-13T00:51:08.602912+0000","last_peered":"2024-02-13T00:51:08.602912+0000","last_clean":"2024-02-13T00:51:08.602912+0000","last_became_active":"2024-02-13T00:44:50.436058+0000","last_became_peered":"2024-02-13T00:44:50.436058+0000","last_unstale":"2024-02-13T00:51:08.602912+0000","last_undegraded":"2024-02-13T00:51:08.602912+0000","last_fullsized":"2024-02-13T00:51:08.602912+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_clean_scrub_stamp":"2024-02-13T00:48:35.434645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":338,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603531+0000","last_change":"2024-02-13T00:46:01.451625+0000","last_active":"2024-02-13T00:51:08.603531+0000","last_peered":"2024-02-13T00:51:08.603531+0000","last_clean":"2024-02-13T00:51:08.603531+0000","last_became_active":"2024-02-13T00:46:01.450587+0000","last_became_peered":"2024-02-13T00:46:01.450587+0000","last_unstale":"2024-02-13T00:51:08.603531+0000","last_undegraded":"2024-02-13T00:51:08.603531+0000","last_fullsized":"2024-02-13T00:51:08.603531+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:33.204969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:33.204969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":735,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602618+0000","last_change":"2024-02-13T00:47:05.718055+0000","last_active":"2024-02-13T00:51:08.602618+0000","last_peered":"2024-02-13T00:51:08.602618+0000","last_clean":"2024-02-13T00:51:08.602618+0000","last_became_active":"2024-02-13T00:46:01.443777+0000","last_became_peered":"2024-02-13T00:46:01.443777+0000","last_unstale":"2024-02-13T00:51:08.602618+0000","last_undegraded":"2024-02-13T00:51:08.602618+0000","last_fullsized":"2024-02-13T00:51:08.602618+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'34","last_scrub_stamp":"2024-02-13T00:47:05.717987+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:47:05.717987+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":451,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604890+0000","last_change":"2024-02-13T00:50:25.001319+0000","last_active":"2024-02-13T00:51:08.604890+0000","last_peered":"2024-02-13T00:51:08.604890+0000","last_clean":"2024-02-13T00:51:08.604890+0000","last_became_active":"2024-02-13T00:45:15.423415+0000","last_became_peered":"2024-02-13T00:45:15.423415+0000","last_unstale":"2024-02-13T00:51:08.604890+0000","last_undegraded":"2024-02-13T00:51:08.604890+0000","last_fullsized":"2024-02-13T00:51:08.604890+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:25.001219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:25.001219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":647,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.545111+0000","last_change":"2024-02-13T00:50:14.726735+0000","last_active":"2024-02-13T00:51:08.545111+0000","last_peered":"2024-02-13T00:51:08.545111+0000","last_clean":"2024-02-13T00:51:08.545111+0000","last_became_active":"2024-02-13T00:45:15.445557+0000","last_became_peered":"2024-02-13T00:45:15.445557+0000","last_unstale":"2024-02-13T00:51:08.545111+0000","last_undegraded":"2024-02-13T00:51:08.545111+0000","last_fullsized":"2024-02-13T00:51:08.545111+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.726660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.726660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":260,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.545909+0000","last_change":"2024-02-13T00:48:55.050483+0000","last_active":"2024-02-13T00:51:08.545909+0000","last_peered":"2024-02-13T00:51:08.545909+0000","last_clean":"2024-02-13T00:51:08.545909+0000","last_became_active":"2024-02-13T00:46:01.441421+0000","last_became_peered":"2024-02-13T00:46:01.441421+0000","last_unstale":"2024-02-13T00:51:08.545909+0000","last_undegraded":"2024-02-13T00:51:08.545909+0000","last_fullsized":"2024-02-13T00:51:08.545909+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:55.050411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.050411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":275,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.547772+0000","last_change":"2024-02-13T00:49:16.505082+0000","last_active":"2024-02-13T00:51:08.547772+0000","last_peered":"2024-02-13T00:51:08.547772+0000","last_clean":"2024-02-13T00:51:08.547772+0000","last_became_active":"2024-02-13T00:49:16.504828+0000","last_became_peered":"2024-02-13T00:49:16.504828+0000","last_unstale":"2024-02-13T00:51:08.547772+0000","last_undegraded":"2024-02-13T00:51:08.547772+0000","last_fullsized":"2024-02-13T00:51:08.547772+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:43.903553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:43.903553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":730,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604845+0000","last_change":"2024-02-13T00:46:34.691109+0000","last_active":"2024-02-13T00:51:08.604845+0000","last_peered":"2024-02-13T00:51:08.604845+0000","last_clean":"2024-02-13T00:51:08.604845+0000","last_became_active":"2024-02-13T00:46:01.448163+0000","last_became_peered":"2024-02-13T00:46:01.448163+0000","last_unstale":"2024-02-13T00:51:08.604845+0000","last_undegraded":"2024-02-13T00:51:08.604845+0000","last_fullsized":"2024-02-13T00:51:08.604845+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'12","last_scrub_stamp":"2024-02-13T00:46:34.690980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.690980+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":343,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544469+0000","last_change":"2024-02-13T00:46:37.056299+0000","last_active":"2024-02-13T00:51:08.544469+0000","last_peered":"2024-02-13T00:51:08.544469+0000","last_clean":"2024-02-13T00:51:08.544469+0000","last_became_active":"2024-02-13T00:46:01.446104+0000","last_became_peered":"2024-02-13T00:46:01.446104+0000","last_unstale":"2024-02-13T00:51:08.544469+0000","last_undegraded":"2024-02-13T00:51:08.544469+0000","last_fullsized":"2024-02-13T00:51:08.544469+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:37.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:37.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":622,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602314+0000","last_change":"2024-02-13T00:46:01.431899+0000","last_active":"2024-02-13T00:51:08.602314+0000","last_peered":"2024-02-13T00:51:08.602314+0000","last_clean":"2024-02-13T00:51:08.602314+0000","last_became_active":"2024-02-13T00:46:01.430623+0000","last_became_peered":"2024-02-13T00:46:01.430623+0000","last_unstale":"2024-02-13T00:51:08.602314+0000","last_undegraded":"2024-02-13T00:51:08.602314+0000","last_fullsized":"2024-02-13T00:51:08.602314+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:46.556229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:45:46.556229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":633,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546897+0000","last_change":"2024-02-13T00:46:01.450548+0000","last_active":"2024-02-13T00:51:08.546897+0000","last_peered":"2024-02-13T00:51:08.546897+0000","last_clean":"2024-02-13T00:51:08.546897+0000","last_became_active":"2024-02-13T00:46:01.450176+0000","last_became_peered":"2024-02-13T00:46:01.450176+0000","last_unstale":"2024-02-13T00:51:08.546897+0000","last_undegraded":"2024-02-13T00:51:08.546897+0000","last_fullsized":"2024-02-13T00:51:08.546897+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":484,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605519+0000","last_change":"2024-02-13T00:46:01.460470+0000","last_active":"2024-02-13T00:51:08.605519+0000","last_peered":"2024-02-13T00:51:08.605519+0000","last_clean":"2024-02-13T00:51:08.605519+0000","last_became_active":"2024-02-13T00:46:01.459859+0000","last_became_peered":"2024-02-13T00:46:01.459859+0000","last_unstale":"2024-02-13T00:51:08.605519+0000","last_undegraded":"2024-02-13T00:51:08.605519+0000","last_fullsized":"2024-02-13T00:51:08.605519+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:01.179031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:01.179031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":287,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603182+0000","last_change":"2024-02-13T00:47:33.932830+0000","last_active":"2024-02-13T00:51:08.603182+0000","last_peered":"2024-02-13T00:51:08.603182+0000","last_clean":"2024-02-13T00:51:08.603182+0000","last_became_active":"2024-02-13T00:46:01.455170+0000","last_became_peered":"2024-02-13T00:46:01.455170+0000","last_unstale":"2024-02-13T00:51:08.603182+0000","last_undegraded":"2024-02-13T00:51:08.603182+0000","last_fullsized":"2024-02-13T00:51:08.603182+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:33.932766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:33.932766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":388,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605520+0000","last_change":"2024-02-13T00:46:57.381602+0000","last_active":"2024-02-13T00:51:08.605520+0000","last_peered":"2024-02-13T00:51:08.605520+0000","last_clean":"2024-02-13T00:51:08.605520+0000","last_became_active":"2024-02-13T00:45:15.446969+0000","last_became_peered":"2024-02-13T00:45:15.446969+0000","last_unstale":"2024-02-13T00:51:08.605520+0000","last_undegraded":"2024-02-13T00:51:08.605520+0000","last_fullsized":"2024-02-13T00:51:08.605520+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:46:57.381540+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:46:57.381540+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":587,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.601822+0000","last_change":"2024-02-13T00:47:39.822745+0000","last_active":"2024-02-13T00:51:08.601822+0000","last_peered":"2024-02-13T00:51:08.601822+0000","last_clean":"2024-02-13T00:51:08.601822+0000","last_became_active":"2024-02-13T00:45:15.434498+0000","last_became_peered":"2024-02-13T00:45:15.434498+0000","last_unstale":"2024-02-13T00:51:08.601822+0000","last_undegraded":"2024-02-13T00:51:08.601822+0000","last_fullsized":"2024-02-13T00:51:08.601822+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:39.822619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:47:39.822619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":586,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603300+0000","last_change":"2024-02-13T00:47:04.493911+0000","last_active":"2024-02-13T00:51:08.603300+0000","last_peered":"2024-02-13T00:51:08.603300+0000","last_clean":"2024-02-13T00:51:08.603300+0000","last_became_active":"2024-02-13T00:46:01.447823+0000","last_became_peered":"2024-02-13T00:46:01.447823+0000","last_unstale":"2024-02-13T00:51:08.603300+0000","last_undegraded":"2024-02-13T00:51:08.603300+0000","last_fullsized":"2024-02-13T00:51:08.603300+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:04.493841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:04.493841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":459,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602704+0000","last_change":"2024-02-13T00:48:25.359625+0000","last_active":"2024-02-13T00:51:08.602704+0000","last_peered":"2024-02-13T00:51:08.602704+0000","last_clean":"2024-02-13T00:51:08.602704+0000","last_became_active":"2024-02-13T00:46:01.460001+0000","last_became_peered":"2024-02-13T00:46:01.460001+0000","last_unstale":"2024-02-13T00:51:08.602704+0000","last_undegraded":"2024-02-13T00:51:08.602704+0000","last_fullsized":"2024-02-13T00:51:08.602704+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:25.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:48:25.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":481,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.548809+0000","last_change":"2024-02-13T00:46:01.450879+0000","last_active":"2024-02-13T00:51:08.548809+0000","last_peered":"2024-02-13T00:51:08.548809+0000","last_clean":"2024-02-13T00:51:08.548809+0000","last_became_active":"2024-02-13T00:46:01.450088+0000","last_became_peered":"2024-02-13T00:46:01.450088+0000","last_unstale":"2024-02-13T00:51:08.548809+0000","last_undegraded":"2024-02-13T00:51:08.548809+0000","last_fullsized":"2024-02-13T00:51:08.548809+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:51.414093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:41:51.414093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":568,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605018+0000","last_change":"2024-02-13T00:49:42.854702+0000","last_active":"2024-02-13T00:51:08.605018+0000","last_peered":"2024-02-13T00:51:08.605018+0000","last_clean":"2024-02-13T00:51:08.605018+0000","last_became_active":"2024-02-13T00:46:01.458408+0000","last_became_peered":"2024-02-13T00:46:01.458408+0000","last_unstale":"2024-02-13T00:51:08.605018+0000","last_undegraded":"2024-02-13T00:51:08.605018+0000","last_fullsized":"2024-02-13T00:51:08.605018+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:42.854585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:42.854585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":312,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.545181+0000","last_change":"2024-02-13T00:49:19.638381+0000","last_active":"2024-02-13T00:51:08.545181+0000","last_peered":"2024-02-13T00:51:08.545181+0000","last_clean":"2024-02-13T00:51:08.545181+0000","last_became_active":"2024-02-13T00:49:19.638083+0000","last_became_peered":"2024-02-13T00:49:19.638083+0000","last_unstale":"2024-02-13T00:51:08.545181+0000","last_undegraded":"2024-02-13T00:51:08.545181+0000","last_fullsized":"2024-02-13T00:51:08.545181+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:53.013103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.013103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"467'23","reported_seq":677,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.547381+0000","last_change":"2024-02-13T00:49:07.192972+0000","last_active":"2024-02-13T00:51:08.547381+0000","last_peered":"2024-02-13T00:51:08.547381+0000","last_clean":"2024-02-13T00:51:08.547381+0000","last_became_active":"2024-02-13T00:46:01.448219+0000","last_became_peered":"2024-02-13T00:46:01.448219+0000","last_unstale":"2024-02-13T00:51:08.547381+0000","last_undegraded":"2024-02-13T00:51:08.547381+0000","last_fullsized":"2024-02-13T00:51:08.547381+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'23","last_scrub_stamp":"2024-02-13T00:49:07.192902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:07.192902+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":542,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602257+0000","last_change":"2024-02-13T00:49:46.851558+0000","last_active":"2024-02-13T00:51:08.602257+0000","last_peered":"2024-02-13T00:51:08.602257+0000","last_clean":"2024-02-13T00:51:08.602257+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:51:08.602257+0000","last_undegraded":"2024-02-13T00:51:08.602257+0000","last_fullsized":"2024-02-13T00:51:08.602257+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.851486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.851486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":1452,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-02-13T00:51:09.544754+0000","last_change":"2024-02-13T00:45:54.075270+0000","last_active":"2024-02-13T00:51:09.544754+0000","last_peered":"2024-02-13T00:51:09.544754+0000","last_clean":"2024-02-13T00:51:09.544754+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:51:09.544754+0000","last_undegraded":"2024-02-13T00:51:09.544754+0000","last_fullsized":"2024-02-13T00:51:09.544754+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:45:54.075194+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":796,"num_read_kb":791,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":351,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605440+0000","last_change":"2024-02-13T00:46:01.460282+0000","last_active":"2024-02-13T00:51:08.605440+0000","last_peered":"2024-02-13T00:51:08.605440+0000","last_clean":"2024-02-13T00:51:08.605440+0000","last_became_active":"2024-02-13T00:46:01.459473+0000","last_became_peered":"2024-02-13T00:46:01.459473+0000","last_unstale":"2024-02-13T00:51:08.605440+0000","last_undegraded":"2024-02-13T00:51:08.605440+0000","last_fullsized":"2024-02-13T00:51:08.605440+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":406,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546443+0000","last_change":"2024-02-13T00:45:37.135398+0000","last_active":"2024-02-13T00:51:08.546443+0000","last_peered":"2024-02-13T00:51:08.546443+0000","last_clean":"2024-02-13T00:51:08.546443+0000","last_became_active":"2024-02-13T00:45:15.435588+0000","last_became_peered":"2024-02-13T00:45:15.435588+0000","last_unstale":"2024-02-13T00:51:08.546443+0000","last_undegraded":"2024-02-13T00:51:08.546443+0000","last_fullsized":"2024-02-13T00:51:08.546443+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.135332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.135332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":302,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604732+0000","last_change":"2024-02-13T00:45:31.183763+0000","last_active":"2024-02-13T00:51:08.604732+0000","last_peered":"2024-02-13T00:51:08.604732+0000","last_clean":"2024-02-13T00:51:08.604732+0000","last_became_active":"2024-02-13T00:45:06.421025+0000","last_became_peered":"2024-02-13T00:45:06.421025+0000","last_unstale":"2024-02-13T00:51:08.604732+0000","last_undegraded":"2024-02-13T00:51:08.604732+0000","last_fullsized":"2024-02-13T00:51:08.604732+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.183682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.183682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"467'65","reported_seq":359,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:09.367560+0000","last_change":"2024-02-13T00:49:47.928628+0000","last_active":"2024-02-13T00:51:09.367560+0000","last_peered":"2024-02-13T00:51:09.367560+0000","last_clean":"2024-02-13T00:51:09.367560+0000","last_became_active":"2024-02-13T00:46:01.449059+0000","last_became_peered":"2024-02-13T00:46:01.449059+0000","last_unstale":"2024-02-13T00:51:09.367560+0000","last_undegraded":"2024-02-13T00:51:09.367560+0000","last_fullsized":"2024-02-13T00:51:09.367560+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'65","last_scrub_stamp":"2024-02-13T00:49:47.928548+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.928548+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":2148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":556,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544676+0000","last_change":"2024-02-13T00:50:37.777963+0000","last_active":"2024-02-13T00:51:08.544676+0000","last_peered":"2024-02-13T00:51:08.544676+0000","last_clean":"2024-02-13T00:51:08.544676+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:51:08.544676+0000","last_undegraded":"2024-02-13T00:51:08.544676+0000","last_fullsized":"2024-02-13T00:51:08.544676+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:37.777891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:37.777891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":167,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603500+0000","last_change":"2024-02-13T00:48:06.705030+0000","last_active":"2024-02-13T00:51:08.603500+0000","last_peered":"2024-02-13T00:51:08.603500+0000","last_clean":"2024-02-13T00:51:08.603500+0000","last_became_active":"2024-02-13T00:48:06.704651+0000","last_became_peered":"2024-02-13T00:48:06.704651+0000","last_unstale":"2024-02-13T00:51:08.603500+0000","last_undegraded":"2024-02-13T00:51:08.603500+0000","last_fullsized":"2024-02-13T00:51:08.603500+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":338,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603629+0000","last_change":"2024-02-13T00:48:17.340891+0000","last_active":"2024-02-13T00:51:08.603629+0000","last_peered":"2024-02-13T00:51:08.603629+0000","last_clean":"2024-02-13T00:51:08.603629+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:51:08.603629+0000","last_undegraded":"2024-02-13T00:51:08.603629+0000","last_fullsized":"2024-02-13T00:51:08.603629+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:17.340797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:17.340797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":824,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:09.367483+0000","last_change":"2024-02-13T00:48:32.669477+0000","last_active":"2024-02-13T00:51:09.367483+0000","last_peered":"2024-02-13T00:51:09.367483+0000","last_clean":"2024-02-13T00:51:09.367483+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:51:09.367483+0000","last_undegraded":"2024-02-13T00:51:09.367483+0000","last_fullsized":"2024-02-13T00:51:09.367483+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14","last_scrub_stamp":"2024-02-13T00:48:32.669406+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.669406+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":374,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.545064+0000","last_change":"2024-02-13T00:48:20.928236+0000","last_active":"2024-02-13T00:51:08.545064+0000","last_peered":"2024-02-13T00:51:08.545064+0000","last_clean":"2024-02-13T00:51:08.545064+0000","last_became_active":"2024-02-13T00:46:01.442003+0000","last_became_peered":"2024-02-13T00:46:01.442003+0000","last_unstale":"2024-02-13T00:51:08.545064+0000","last_undegraded":"2024-02-13T00:51:08.545064+0000","last_fullsized":"2024-02-13T00:51:08.545064+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:48:20.928158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:20.928158+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":622,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546516+0000","last_change":"2024-02-13T00:49:33.040619+0000","last_active":"2024-02-13T00:51:08.546516+0000","last_peered":"2024-02-13T00:51:08.546516+0000","last_clean":"2024-02-13T00:51:08.546516+0000","last_became_active":"2024-02-13T00:45:15.422546+0000","last_became_peered":"2024-02-13T00:45:15.422546+0000","last_unstale":"2024-02-13T00:51:08.546516+0000","last_undegraded":"2024-02-13T00:51:08.546516+0000","last_fullsized":"2024-02-13T00:51:08.546516+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:33.040516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:49:33.040516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":634,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.601770+0000","last_change":"2024-02-13T00:50:38.051528+0000","last_active":"2024-02-13T00:51:08.601770+0000","last_peered":"2024-02-13T00:51:08.601770+0000","last_clean":"2024-02-13T00:51:08.601770+0000","last_became_active":"2024-02-13T00:46:01.435178+0000","last_became_peered":"2024-02-13T00:46:01.435178+0000","last_unstale":"2024-02-13T00:51:08.601770+0000","last_undegraded":"2024-02-13T00:51:08.601770+0000","last_fullsized":"2024-02-13T00:51:08.601770+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:38.051468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:38.051468+0000","last_clean_scrub_stamp":"2024-02-13T00:50:38.051468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":571,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.606342+0000","last_change":"2024-02-13T00:51:08.606320+0000","last_active":"2024-02-13T00:51:08.606342+0000","last_peered":"2024-02-13T00:51:08.606342+0000","last_clean":"2024-02-13T00:51:08.606342+0000","last_became_active":"2024-02-13T00:45:15.445117+0000","last_became_peered":"2024-02-13T00:45:15.445117+0000","last_unstale":"2024-02-13T00:51:08.606342+0000","last_undegraded":"2024-02-13T00:51:08.606342+0000","last_fullsized":"2024-02-13T00:51:08.606342+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:08.606262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:51:08.606262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":347,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602329+0000","last_change":"2024-02-13T00:49:12.252240+0000","last_active":"2024-02-13T00:51:08.602329+0000","last_peered":"2024-02-13T00:51:08.602329+0000","last_clean":"2024-02-13T00:51:08.602329+0000","last_became_active":"2024-02-13T00:49:12.251980+0000","last_became_peered":"2024-02-13T00:49:12.251980+0000","last_unstale":"2024-02-13T00:51:08.602329+0000","last_undegraded":"2024-02-13T00:51:08.602329+0000","last_fullsized":"2024-02-13T00:51:08.602329+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.592320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.592320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":692,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544105+0000","last_change":"2024-02-13T00:49:31.072963+0000","last_active":"2024-02-13T00:51:08.544105+0000","last_peered":"2024-02-13T00:51:08.544105+0000","last_clean":"2024-02-13T00:51:08.544105+0000","last_became_active":"2024-02-13T00:44:39.501886+0000","last_became_peered":"2024-02-13T00:44:39.501886+0000","last_unstale":"2024-02-13T00:51:08.544105+0000","last_undegraded":"2024-02-13T00:51:08.544105+0000","last_fullsized":"2024-02-13T00:51:08.544105+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'30","last_scrub_stamp":"2024-02-13T00:49:31.072862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:31.072862+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":608,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604817+0000","last_change":"2024-02-13T00:48:58.861408+0000","last_active":"2024-02-13T00:51:08.604817+0000","last_peered":"2024-02-13T00:51:08.604817+0000","last_clean":"2024-02-13T00:51:08.604817+0000","last_became_active":"2024-02-13T00:46:01.457926+0000","last_became_peered":"2024-02-13T00:46:01.457926+0000","last_unstale":"2024-02-13T00:51:08.604817+0000","last_undegraded":"2024-02-13T00:51:08.604817+0000","last_fullsized":"2024-02-13T00:51:08.604817+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:48:58.861325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:58.861325+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":417,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.545005+0000","last_change":"2024-02-13T00:47:06.895693+0000","last_active":"2024-02-13T00:51:08.545005+0000","last_peered":"2024-02-13T00:51:08.545005+0000","last_clean":"2024-02-13T00:51:08.545005+0000","last_became_active":"2024-02-13T00:45:15.435404+0000","last_became_peered":"2024-02-13T00:45:15.435404+0000","last_unstale":"2024-02-13T00:51:08.545005+0000","last_undegraded":"2024-02-13T00:51:08.545005+0000","last_fullsized":"2024-02-13T00:51:08.545005+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:06.895612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:47:06.895612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":568,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604927+0000","last_change":"2024-02-13T00:50:21.880120+0000","last_active":"2024-02-13T00:51:08.604927+0000","last_peered":"2024-02-13T00:51:08.604927+0000","last_clean":"2024-02-13T00:51:08.604927+0000","last_became_active":"2024-02-13T00:46:01.449234+0000","last_became_peered":"2024-02-13T00:46:01.449234+0000","last_unstale":"2024-02-13T00:51:08.604927+0000","last_undegraded":"2024-02-13T00:51:08.604927+0000","last_fullsized":"2024-02-13T00:51:08.604927+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.880001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.880001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"467'27","reported_seq":756,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604854+0000","last_change":"2024-02-13T00:49:35.834793+0000","last_active":"2024-02-13T00:51:08.604854+0000","last_peered":"2024-02-13T00:51:08.604854+0000","last_clean":"2024-02-13T00:51:08.604854+0000","last_became_active":"2024-02-13T00:46:01.456876+0000","last_became_peered":"2024-02-13T00:46:01.456876+0000","last_unstale":"2024-02-13T00:51:08.604854+0000","last_undegraded":"2024-02-13T00:51:08.604854+0000","last_fullsized":"2024-02-13T00:51:08.604854+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:35.834669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.834669+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":324,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544382+0000","last_change":"2024-02-13T00:46:30.095387+0000","last_active":"2024-02-13T00:51:08.544382+0000","last_peered":"2024-02-13T00:51:08.544382+0000","last_clean":"2024-02-13T00:51:08.544382+0000","last_became_active":"2024-02-13T00:44:50.441300+0000","last_became_peered":"2024-02-13T00:44:50.441300+0000","last_unstale":"2024-02-13T00:51:08.544382+0000","last_undegraded":"2024-02-13T00:51:08.544382+0000","last_fullsized":"2024-02-13T00:51:08.544382+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:46:30.095307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":336,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603248+0000","last_change":"2024-02-13T00:46:01.454597+0000","last_active":"2024-02-13T00:51:08.603248+0000","last_peered":"2024-02-13T00:51:08.603248+0000","last_clean":"2024-02-13T00:51:08.603248+0000","last_became_active":"2024-02-13T00:46:01.454101+0000","last_became_peered":"2024-02-13T00:46:01.454101+0000","last_unstale":"2024-02-13T00:51:08.603248+0000","last_undegraded":"2024-02-13T00:51:08.603248+0000","last_fullsized":"2024-02-13T00:51:08.603248+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:26.651091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:26.651091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":338,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605079+0000","last_change":"2024-02-13T00:51:07.304642+0000","last_active":"2024-02-13T00:51:08.605079+0000","last_peered":"2024-02-13T00:51:08.605079+0000","last_clean":"2024-02-13T00:51:08.605079+0000","last_became_active":"2024-02-13T00:44:50.438274+0000","last_became_peered":"2024-02-13T00:44:50.438274+0000","last_unstale":"2024-02-13T00:51:08.605079+0000","last_undegraded":"2024-02-13T00:51:08.605079+0000","last_fullsized":"2024-02-13T00:51:08.605079+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:07.304576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:07.304576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":493,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.547641+0000","last_change":"2024-02-13T00:46:34.141015+0000","last_active":"2024-02-13T00:51:08.547641+0000","last_peered":"2024-02-13T00:51:08.547641+0000","last_clean":"2024-02-13T00:51:08.547641+0000","last_became_active":"2024-02-13T00:45:15.436020+0000","last_became_peered":"2024-02-13T00:45:15.436020+0000","last_unstale":"2024-02-13T00:51:08.547641+0000","last_undegraded":"2024-02-13T00:51:08.547641+0000","last_fullsized":"2024-02-13T00:51:08.547641+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-13T00:46:34.140928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:34.140928+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":329,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603643+0000","last_change":"2024-02-13T00:45:00.311635+0000","last_active":"2024-02-13T00:51:08.603643+0000","last_peered":"2024-02-13T00:51:08.603643+0000","last_clean":"2024-02-13T00:51:08.603643+0000","last_became_active":"2024-02-13T00:44:39.512402+0000","last_became_peered":"2024-02-13T00:44:39.512402+0000","last_unstale":"2024-02-13T00:51:08.603643+0000","last_undegraded":"2024-02-13T00:51:08.603643+0000","last_fullsized":"2024-02-13T00:51:08.603643+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:00.311558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:45:00.311558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":347,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605418+0000","last_change":"2024-02-13T00:50:40.384509+0000","last_active":"2024-02-13T00:51:08.605418+0000","last_peered":"2024-02-13T00:51:08.605418+0000","last_clean":"2024-02-13T00:51:08.605418+0000","last_became_active":"2024-02-13T00:46:01.450312+0000","last_became_peered":"2024-02-13T00:46:01.450312+0000","last_unstale":"2024-02-13T00:51:08.605418+0000","last_undegraded":"2024-02-13T00:51:08.605418+0000","last_fullsized":"2024-02-13T00:51:08.605418+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:40.384428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:40.384428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":413,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603529+0000","last_change":"2024-02-13T00:46:01.454858+0000","last_active":"2024-02-13T00:51:08.603529+0000","last_peered":"2024-02-13T00:51:08.603529+0000","last_clean":"2024-02-13T00:51:08.603529+0000","last_became_active":"2024-02-13T00:46:01.454284+0000","last_became_peered":"2024-02-13T00:46:01.454284+0000","last_unstale":"2024-02-13T00:51:08.603529+0000","last_undegraded":"2024-02-13T00:51:08.603529+0000","last_fullsized":"2024-02-13T00:51:08.603529+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:34.523594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:34.523594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":576,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546733+0000","last_change":"2024-02-13T00:49:06.154997+0000","last_active":"2024-02-13T00:51:08.546733+0000","last_peered":"2024-02-13T00:51:08.546733+0000","last_clean":"2024-02-13T00:51:08.546733+0000","last_became_active":"2024-02-13T00:46:01.450113+0000","last_became_peered":"2024-02-13T00:46:01.450113+0000","last_unstale":"2024-02-13T00:51:08.546733+0000","last_undegraded":"2024-02-13T00:51:08.546733+0000","last_fullsized":"2024-02-13T00:51:08.546733+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'39","last_scrub_stamp":"2024-02-13T00:49:06.154931+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:49:06.154931+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":588,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603006+0000","last_change":"2024-02-13T00:49:29.916234+0000","last_active":"2024-02-13T00:51:08.603006+0000","last_peered":"2024-02-13T00:51:08.603006+0000","last_clean":"2024-02-13T00:51:08.603006+0000","last_became_active":"2024-02-13T00:46:01.454310+0000","last_became_peered":"2024-02-13T00:46:01.454310+0000","last_unstale":"2024-02-13T00:51:08.603006+0000","last_undegraded":"2024-02-13T00:51:08.603006+0000","last_fullsized":"2024-02-13T00:51:08.603006+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.916152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"404'9","reported_seq":342,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:09.367462+0000","last_change":"2024-02-13T00:47:37.013003+0000","last_active":"2024-02-13T00:51:09.367462+0000","last_peered":"2024-02-13T00:51:09.367462+0000","last_clean":"2024-02-13T00:51:09.367462+0000","last_became_active":"2024-02-13T00:46:01.448144+0000","last_became_peered":"2024-02-13T00:46:01.448144+0000","last_unstale":"2024-02-13T00:51:09.367462+0000","last_undegraded":"2024-02-13T00:51:09.367462+0000","last_fullsized":"2024-02-13T00:51:09.367462+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9","last_scrub_stamp":"2024-02-13T00:47:37.012934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:37.012934+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":625,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604752+0000","last_change":"2024-02-13T00:48:26.647460+0000","last_active":"2024-02-13T00:51:08.604752+0000","last_peered":"2024-02-13T00:51:08.604752+0000","last_clean":"2024-02-13T00:51:08.604752+0000","last_became_active":"2024-02-13T00:46:01.447236+0000","last_became_peered":"2024-02-13T00:46:01.447236+0000","last_unstale":"2024-02-13T00:51:08.604752+0000","last_undegraded":"2024-02-13T00:51:08.604752+0000","last_fullsized":"2024-02-13T00:51:08.604752+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:26.647367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:26.647367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":296,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603713+0000","last_change":"2024-02-13T00:50:14.849751+0000","last_active":"2024-02-13T00:51:08.603713+0000","last_peered":"2024-02-13T00:51:08.603713+0000","last_clean":"2024-02-13T00:51:08.603713+0000","last_became_active":"2024-02-13T00:46:01.455989+0000","last_became_peered":"2024-02-13T00:46:01.455989+0000","last_unstale":"2024-02-13T00:51:08.603713+0000","last_undegraded":"2024-02-13T00:51:08.603713+0000","last_fullsized":"2024-02-13T00:51:08.603713+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.849687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.849687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"467'22","reported_seq":611,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605142+0000","last_change":"2024-02-13T00:47:35.369710+0000","last_active":"2024-02-13T00:51:08.605142+0000","last_peered":"2024-02-13T00:51:08.605142+0000","last_clean":"2024-02-13T00:51:08.605142+0000","last_became_active":"2024-02-13T00:46:01.460321+0000","last_became_peered":"2024-02-13T00:46:01.460321+0000","last_unstale":"2024-02-13T00:51:08.605142+0000","last_undegraded":"2024-02-13T00:51:08.605142+0000","last_fullsized":"2024-02-13T00:51:08.605142+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:47:35.369616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:35.369616+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":640,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602706+0000","last_change":"2024-02-13T00:49:05.887739+0000","last_active":"2024-02-13T00:51:08.602706+0000","last_peered":"2024-02-13T00:51:08.602706+0000","last_clean":"2024-02-13T00:51:08.602706+0000","last_became_active":"2024-02-13T00:46:01.431926+0000","last_became_peered":"2024-02-13T00:46:01.431926+0000","last_unstale":"2024-02-13T00:51:08.602706+0000","last_undegraded":"2024-02-13T00:51:08.602706+0000","last_fullsized":"2024-02-13T00:51:08.602706+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:05.887665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:05.887665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"363'14","reported_seq":485,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:09.367776+0000","last_change":"2024-02-13T00:47:07.942748+0000","last_active":"2024-02-13T00:51:09.367776+0000","last_peered":"2024-02-13T00:51:09.367776+0000","last_clean":"2024-02-13T00:51:09.367776+0000","last_became_active":"2024-02-13T00:45:15.423066+0000","last_became_peered":"2024-02-13T00:45:15.423066+0000","last_unstale":"2024-02-13T00:51:09.367776+0000","last_undegraded":"2024-02-13T00:51:09.367776+0000","last_fullsized":"2024-02-13T00:51:09.367776+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:47:07.942668+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:47:07.942668+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":259,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604784+0000","last_change":"2024-02-13T00:47:02.789298+0000","last_active":"2024-02-13T00:51:08.604784+0000","last_peered":"2024-02-13T00:51:08.604784+0000","last_clean":"2024-02-13T00:51:08.604784+0000","last_became_active":"2024-02-13T00:46:01.446960+0000","last_became_peered":"2024-02-13T00:46:01.446960+0000","last_unstale":"2024-02-13T00:51:08.604784+0000","last_undegraded":"2024-02-13T00:51:08.604784+0000","last_fullsized":"2024-02-13T00:51:08.604784+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:02.789189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.789189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":498,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602975+0000","last_change":"2024-02-13T00:46:01.450308+0000","last_active":"2024-02-13T00:51:08.602975+0000","last_peered":"2024-02-13T00:51:08.602975+0000","last_clean":"2024-02-13T00:51:08.602975+0000","last_became_active":"2024-02-13T00:46:01.449694+0000","last_became_peered":"2024-02-13T00:46:01.449694+0000","last_unstale":"2024-02-13T00:51:08.602975+0000","last_undegraded":"2024-02-13T00:51:08.602975+0000","last_fullsized":"2024-02-13T00:51:08.602975+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:27.073034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:27.073034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":697,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544962+0000","last_change":"2024-02-13T00:49:38.007273+0000","last_active":"2024-02-13T00:51:08.544962+0000","last_peered":"2024-02-13T00:51:08.544962+0000","last_clean":"2024-02-13T00:51:08.544962+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:51:08.544962+0000","last_undegraded":"2024-02-13T00:51:08.544962+0000","last_fullsized":"2024-02-13T00:51:08.544962+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'19","last_scrub_stamp":"2024-02-13T00:49:38.007204+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:49:38.007204+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":342,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604822+0000","last_change":"2024-02-13T00:46:01.463310+0000","last_active":"2024-02-13T00:51:08.604822+0000","last_peered":"2024-02-13T00:51:08.604822+0000","last_clean":"2024-02-13T00:51:08.604822+0000","last_became_active":"2024-02-13T00:46:01.462898+0000","last_became_peered":"2024-02-13T00:46:01.462898+0000","last_unstale":"2024-02-13T00:51:08.604822+0000","last_undegraded":"2024-02-13T00:51:08.604822+0000","last_fullsized":"2024-02-13T00:51:08.604822+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:43:49.095888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":404,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602275+0000","last_change":"2024-02-13T00:46:32.638418+0000","last_active":"2024-02-13T00:51:08.602275+0000","last_peered":"2024-02-13T00:51:08.602275+0000","last_clean":"2024-02-13T00:51:08.602275+0000","last_became_active":"2024-02-13T00:46:01.443279+0000","last_became_peered":"2024-02-13T00:46:01.443279+0000","last_unstale":"2024-02-13T00:51:08.602275+0000","last_undegraded":"2024-02-13T00:51:08.602275+0000","last_fullsized":"2024-02-13T00:51:08.602275+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.638349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":607,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.601725+0000","last_change":"2024-02-13T00:48:27.659266+0000","last_active":"2024-02-13T00:51:08.601725+0000","last_peered":"2024-02-13T00:51:08.601725+0000","last_clean":"2024-02-13T00:51:08.601725+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:51:08.601725+0000","last_undegraded":"2024-02-13T00:51:08.601725+0000","last_fullsized":"2024-02-13T00:51:08.601725+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:27.659197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:48:27.659197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":338,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605084+0000","last_change":"2024-02-13T00:45:40.233264+0000","last_active":"2024-02-13T00:51:08.605084+0000","last_peered":"2024-02-13T00:51:08.605084+0000","last_clean":"2024-02-13T00:51:08.605084+0000","last_became_active":"2024-02-13T00:45:15.447051+0000","last_became_peered":"2024-02-13T00:45:15.447051+0000","last_unstale":"2024-02-13T00:51:08.605084+0000","last_undegraded":"2024-02-13T00:51:08.605084+0000","last_fullsized":"2024-02-13T00:51:08.605084+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:40.233185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:40.233185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":374,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605028+0000","last_change":"2024-02-13T00:48:34.458358+0000","last_active":"2024-02-13T00:51:08.605028+0000","last_peered":"2024-02-13T00:51:08.605028+0000","last_clean":"2024-02-13T00:51:08.605028+0000","last_became_active":"2024-02-13T00:44:50.435203+0000","last_became_peered":"2024-02-13T00:44:50.435203+0000","last_unstale":"2024-02-13T00:51:08.605028+0000","last_undegraded":"2024-02-13T00:51:08.605028+0000","last_fullsized":"2024-02-13T00:51:08.605028+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:48:34.458264+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:48:34.458264+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":617,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546586+0000","last_change":"2024-02-13T00:46:01.457220+0000","last_active":"2024-02-13T00:51:08.546586+0000","last_peered":"2024-02-13T00:51:08.546586+0000","last_clean":"2024-02-13T00:51:08.546586+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:51:08.546586+0000","last_undegraded":"2024-02-13T00:51:08.546586+0000","last_fullsized":"2024-02-13T00:51:08.546586+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.097481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":636,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.545883+0000","last_change":"2024-02-13T00:50:36.787685+0000","last_active":"2024-02-13T00:51:08.545883+0000","last_peered":"2024-02-13T00:51:08.545883+0000","last_clean":"2024-02-13T00:51:08.545883+0000","last_became_active":"2024-02-13T00:46:01.452730+0000","last_became_peered":"2024-02-13T00:46:01.452730+0000","last_unstale":"2024-02-13T00:51:08.545883+0000","last_undegraded":"2024-02-13T00:51:08.545883+0000","last_fullsized":"2024-02-13T00:51:08.545883+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:36.787604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:50:36.787604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":413,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602345+0000","last_change":"2024-02-13T00:48:29.681480+0000","last_active":"2024-02-13T00:51:08.602345+0000","last_peered":"2024-02-13T00:51:08.602345+0000","last_clean":"2024-02-13T00:51:08.602345+0000","last_became_active":"2024-02-13T00:46:01.454287+0000","last_became_peered":"2024-02-13T00:46:01.454287+0000","last_unstale":"2024-02-13T00:51:08.602345+0000","last_undegraded":"2024-02-13T00:51:08.602345+0000","last_fullsized":"2024-02-13T00:51:08.602345+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:29.681414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:29.681414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":347,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603286+0000","last_change":"2024-02-13T00:49:02.260116+0000","last_active":"2024-02-13T00:51:08.603286+0000","last_peered":"2024-02-13T00:51:08.603286+0000","last_clean":"2024-02-13T00:51:08.603286+0000","last_became_active":"2024-02-13T00:46:01.448671+0000","last_became_peered":"2024-02-13T00:46:01.448671+0000","last_unstale":"2024-02-13T00:51:08.603286+0000","last_undegraded":"2024-02-13T00:51:08.603286+0000","last_fullsized":"2024-02-13T00:51:08.603286+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:02.260045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:02.260045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":449,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544931+0000","last_change":"2024-02-13T00:50:32.774892+0000","last_active":"2024-02-13T00:51:08.544931+0000","last_peered":"2024-02-13T00:51:08.544931+0000","last_clean":"2024-02-13T00:51:08.544931+0000","last_became_active":"2024-02-13T00:44:39.501040+0000","last_became_peered":"2024-02-13T00:44:39.501040+0000","last_unstale":"2024-02-13T00:51:08.544931+0000","last_undegraded":"2024-02-13T00:51:08.544931+0000","last_fullsized":"2024-02-13T00:51:08.544931+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'8","last_scrub_stamp":"2024-02-13T00:50:32.774820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:50:32.774820+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":573,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603759+0000","last_change":"2024-02-13T00:48:32.441279+0000","last_active":"2024-02-13T00:51:08.603759+0000","last_peered":"2024-02-13T00:51:08.603759+0000","last_clean":"2024-02-13T00:51:08.603759+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:51:08.603759+0000","last_undegraded":"2024-02-13T00:51:08.603759+0000","last_fullsized":"2024-02-13T00:51:08.603759+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.441170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.441170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":338,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603357+0000","last_change":"2024-02-13T00:48:28.353456+0000","last_active":"2024-02-13T00:51:08.603357+0000","last_peered":"2024-02-13T00:51:08.603357+0000","last_clean":"2024-02-13T00:51:08.603357+0000","last_became_active":"2024-02-13T00:45:15.435820+0000","last_became_peered":"2024-02-13T00:45:15.435820+0000","last_unstale":"2024-02-13T00:51:08.603357+0000","last_undegraded":"2024-02-13T00:51:08.603357+0000","last_fullsized":"2024-02-13T00:51:08.603357+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:28.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:28.353347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":331,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544066+0000","last_change":"2024-02-13T00:49:29.052728+0000","last_active":"2024-02-13T00:51:08.544066+0000","last_peered":"2024-02-13T00:51:08.544066+0000","last_clean":"2024-02-13T00:51:08.544066+0000","last_became_active":"2024-02-13T00:45:15.444818+0000","last_became_peered":"2024-02-13T00:45:15.444818+0000","last_unstale":"2024-02-13T00:51:08.544066+0000","last_undegraded":"2024-02-13T00:51:08.544066+0000","last_fullsized":"2024-02-13T00:51:08.544066+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.052640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.052640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":489,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603386+0000","last_change":"2024-02-13T00:46:01.439915+0000","last_active":"2024-02-13T00:51:08.603386+0000","last_peered":"2024-02-13T00:51:08.603386+0000","last_clean":"2024-02-13T00:51:08.603386+0000","last_became_active":"2024-02-13T00:46:01.438745+0000","last_became_peered":"2024-02-13T00:46:01.438745+0000","last_unstale":"2024-02-13T00:51:08.603386+0000","last_undegraded":"2024-02-13T00:51:08.603386+0000","last_fullsized":"2024-02-13T00:51:08.603386+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:33.104906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:44:33.104906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"467'25","reported_seq":401,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603477+0000","last_change":"2024-02-13T00:47:36.368967+0000","last_active":"2024-02-13T00:51:08.603477+0000","last_peered":"2024-02-13T00:51:08.603477+0000","last_clean":"2024-02-13T00:51:08.603477+0000","last_became_active":"2024-02-13T00:46:01.450079+0000","last_became_peered":"2024-02-13T00:46:01.450079+0000","last_unstale":"2024-02-13T00:51:08.603477+0000","last_undegraded":"2024-02-13T00:51:08.603477+0000","last_fullsized":"2024-02-13T00:51:08.603477+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:47:36.368863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.368863+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":347,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604986+0000","last_change":"2024-02-13T00:46:59.388809+0000","last_active":"2024-02-13T00:51:08.604986+0000","last_peered":"2024-02-13T00:51:08.604986+0000","last_clean":"2024-02-13T00:51:08.604986+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:51:08.604986+0000","last_undegraded":"2024-02-13T00:51:08.604986+0000","last_fullsized":"2024-02-13T00:51:08.604986+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:59.388745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:59.388745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":338,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602768+0000","last_change":"2024-02-13T00:51:08.345742+0000","last_active":"2024-02-13T00:51:08.602768+0000","last_peered":"2024-02-13T00:51:08.602768+0000","last_clean":"2024-02-13T00:51:08.602768+0000","last_became_active":"2024-02-13T00:45:15.437943+0000","last_became_peered":"2024-02-13T00:45:15.437943+0000","last_unstale":"2024-02-13T00:51:08.602768+0000","last_undegraded":"2024-02-13T00:51:08.602768+0000","last_fullsized":"2024-02-13T00:51:08.602768+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:08.345652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:51:08.345652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":545,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603424+0000","last_change":"2024-02-13T00:51:06.264706+0000","last_active":"2024-02-13T00:51:08.603424+0000","last_peered":"2024-02-13T00:51:08.603424+0000","last_clean":"2024-02-13T00:51:08.603424+0000","last_became_active":"2024-02-13T00:44:50.439512+0000","last_became_peered":"2024-02-13T00:44:50.439512+0000","last_unstale":"2024-02-13T00:51:08.603424+0000","last_undegraded":"2024-02-13T00:51:08.603424+0000","last_fullsized":"2024-02-13T00:51:08.603424+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:06.264572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:51:06.264572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":461,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603912+0000","last_change":"2024-02-13T00:49:35.101413+0000","last_active":"2024-02-13T00:51:08.603912+0000","last_peered":"2024-02-13T00:51:08.603912+0000","last_clean":"2024-02-13T00:51:08.603912+0000","last_became_active":"2024-02-13T00:45:15.430749+0000","last_became_peered":"2024-02-13T00:45:15.430749+0000","last_unstale":"2024-02-13T00:51:08.603912+0000","last_undegraded":"2024-02-13T00:51:08.603912+0000","last_fullsized":"2024-02-13T00:51:08.603912+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:35.101298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.101298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":729,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605105+0000","last_change":"2024-02-13T00:48:53.841888+0000","last_active":"2024-02-13T00:51:08.605105+0000","last_peered":"2024-02-13T00:51:08.605105+0000","last_clean":"2024-02-13T00:51:08.605105+0000","last_became_active":"2024-02-13T00:46:01.443721+0000","last_became_peered":"2024-02-13T00:46:01.443721+0000","last_unstale":"2024-02-13T00:51:08.605105+0000","last_undegraded":"2024-02-13T00:51:08.605105+0000","last_fullsized":"2024-02-13T00:51:08.605105+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'32","last_scrub_stamp":"2024-02-13T00:48:53.841797+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.841797+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":575,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602330+0000","last_change":"2024-02-13T00:46:32.298033+0000","last_active":"2024-02-13T00:51:08.602330+0000","last_peered":"2024-02-13T00:51:08.602330+0000","last_clean":"2024-02-13T00:51:08.602330+0000","last_became_active":"2024-02-13T00:46:01.442231+0000","last_became_peered":"2024-02-13T00:46:01.442231+0000","last_unstale":"2024-02-13T00:51:08.602330+0000","last_undegraded":"2024-02-13T00:51:08.602330+0000","last_fullsized":"2024-02-13T00:51:08.602330+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.297918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.297918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":439,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603683+0000","last_change":"2024-02-13T00:48:33.460377+0000","last_active":"2024-02-13T00:51:08.603683+0000","last_peered":"2024-02-13T00:51:08.603683+0000","last_clean":"2024-02-13T00:51:08.603683+0000","last_became_active":"2024-02-13T00:46:01.439984+0000","last_became_peered":"2024-02-13T00:46:01.439984+0000","last_unstale":"2024-02-13T00:51:08.603683+0000","last_undegraded":"2024-02-13T00:51:08.603683+0000","last_fullsized":"2024-02-13T00:51:08.603683+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:48:33.460247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:33.460247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":575,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.547559+0000","last_change":"2024-02-13T00:45:38.084426+0000","last_active":"2024-02-13T00:51:08.547559+0000","last_peered":"2024-02-13T00:51:08.547559+0000","last_clean":"2024-02-13T00:51:08.547559+0000","last_became_active":"2024-02-13T00:45:15.435550+0000","last_became_peered":"2024-02-13T00:45:15.435550+0000","last_unstale":"2024-02-13T00:51:08.547559+0000","last_undegraded":"2024-02-13T00:51:08.547559+0000","last_fullsized":"2024-02-13T00:51:08.547559+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:38.084358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:38.084358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":378,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605158+0000","last_change":"2024-02-13T00:48:55.424021+0000","last_active":"2024-02-13T00:51:08.605158+0000","last_peered":"2024-02-13T00:51:08.605158+0000","last_clean":"2024-02-13T00:51:08.605158+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:51:08.605158+0000","last_undegraded":"2024-02-13T00:51:08.605158+0000","last_fullsized":"2024-02-13T00:51:08.605158+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:55.423952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.423952+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":347,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603553+0000","last_change":"2024-02-13T00:46:56.339969+0000","last_active":"2024-02-13T00:51:08.603553+0000","last_peered":"2024-02-13T00:51:08.603553+0000","last_clean":"2024-02-13T00:51:08.603553+0000","last_became_active":"2024-02-13T00:45:15.436239+0000","last_became_peered":"2024-02-13T00:45:15.436239+0000","last_unstale":"2024-02-13T00:51:08.603553+0000","last_undegraded":"2024-02-13T00:51:08.603553+0000","last_fullsized":"2024-02-13T00:51:08.603553+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:56.339885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.339885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":616,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546509+0000","last_change":"2024-02-13T00:49:46.928723+0000","last_active":"2024-02-13T00:51:08.546509+0000","last_peered":"2024-02-13T00:51:08.546509+0000","last_clean":"2024-02-13T00:51:08.546509+0000","last_became_active":"2024-02-13T00:46:01.453269+0000","last_became_peered":"2024-02-13T00:46:01.453269+0000","last_unstale":"2024-02-13T00:51:08.546509+0000","last_undegraded":"2024-02-13T00:51:08.546509+0000","last_fullsized":"2024-02-13T00:51:08.546509+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.928651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.928651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":338,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604900+0000","last_change":"2024-02-13T00:45:37.280947+0000","last_active":"2024-02-13T00:51:08.604900+0000","last_peered":"2024-02-13T00:51:08.604900+0000","last_clean":"2024-02-13T00:51:08.604900+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:51:08.604900+0000","last_undegraded":"2024-02-13T00:51:08.604900+0000","last_fullsized":"2024-02-13T00:51:08.604900+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:37.280865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:37.280865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":395,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546646+0000","last_change":"2024-02-13T00:49:30.086932+0000","last_active":"2024-02-13T00:51:08.546646+0000","last_peered":"2024-02-13T00:51:08.546646+0000","last_clean":"2024-02-13T00:51:08.546646+0000","last_became_active":"2024-02-13T00:44:50.442894+0000","last_became_peered":"2024-02-13T00:44:50.442894+0000","last_unstale":"2024-02-13T00:51:08.546646+0000","last_undegraded":"2024-02-13T00:51:08.546646+0000","last_fullsized":"2024-02-13T00:51:08.546646+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:30.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:30.086856+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":338,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603124+0000","last_change":"2024-02-13T00:49:04.282594+0000","last_active":"2024-02-13T00:51:08.603124+0000","last_peered":"2024-02-13T00:51:08.603124+0000","last_clean":"2024-02-13T00:51:08.603124+0000","last_became_active":"2024-02-13T00:45:15.445403+0000","last_became_peered":"2024-02-13T00:45:15.445403+0000","last_unstale":"2024-02-13T00:51:08.603124+0000","last_undegraded":"2024-02-13T00:51:08.603124+0000","last_fullsized":"2024-02-13T00:51:08.603124+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:04.282529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:04.282529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":338,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604706+0000","last_change":"2024-02-13T00:47:42.329463+0000","last_active":"2024-02-13T00:51:08.604706+0000","last_peered":"2024-02-13T00:51:08.604706+0000","last_clean":"2024-02-13T00:51:08.604706+0000","last_became_active":"2024-02-13T00:46:01.450395+0000","last_became_peered":"2024-02-13T00:46:01.450395+0000","last_unstale":"2024-02-13T00:51:08.604706+0000","last_undegraded":"2024-02-13T00:51:08.604706+0000","last_fullsized":"2024-02-13T00:51:08.604706+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:42.329367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:42.329367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":252,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605216+0000","last_change":"2024-02-13T00:50:23.430479+0000","last_active":"2024-02-13T00:51:08.605216+0000","last_peered":"2024-02-13T00:51:08.605216+0000","last_clean":"2024-02-13T00:51:08.605216+0000","last_became_active":"2024-02-13T00:46:01.449873+0000","last_became_peered":"2024-02-13T00:46:01.449873+0000","last_unstale":"2024-02-13T00:51:08.605216+0000","last_undegraded":"2024-02-13T00:51:08.605216+0000","last_fullsized":"2024-02-13T00:51:08.605216+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:23.430396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:23.430396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":496,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.546782+0000","last_change":"2024-02-13T00:46:56.973071+0000","last_active":"2024-02-13T00:51:08.546782+0000","last_peered":"2024-02-13T00:51:08.546782+0000","last_clean":"2024-02-13T00:51:08.546782+0000","last_became_active":"2024-02-13T00:44:39.504670+0000","last_became_peered":"2024-02-13T00:44:39.504670+0000","last_unstale":"2024-02-13T00:51:08.546782+0000","last_undegraded":"2024-02-13T00:51:08.546782+0000","last_fullsized":"2024-02-13T00:51:08.546782+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-13T00:46:56.973000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.973000+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":342,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605109+0000","last_change":"2024-02-13T00:46:01.448134+0000","last_active":"2024-02-13T00:51:08.605109+0000","last_peered":"2024-02-13T00:51:08.605109+0000","last_clean":"2024-02-13T00:51:08.605109+0000","last_became_active":"2024-02-13T00:46:01.446635+0000","last_became_peered":"2024-02-13T00:46:01.446635+0000","last_unstale":"2024-02-13T00:51:08.605109+0000","last_undegraded":"2024-02-13T00:51:08.605109+0000","last_fullsized":"2024-02-13T00:51:08.605109+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:44:28.101254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:44:28.101254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":338,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604924+0000","last_change":"2024-02-13T00:46:01.461188+0000","last_active":"2024-02-13T00:51:08.604924+0000","last_peered":"2024-02-13T00:51:08.604924+0000","last_clean":"2024-02-13T00:51:08.604924+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:51:08.604924+0000","last_undegraded":"2024-02-13T00:51:08.604924+0000","last_fullsized":"2024-02-13T00:51:08.604924+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.238152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.238152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":581,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602645+0000","last_change":"2024-02-13T00:46:01.443734+0000","last_active":"2024-02-13T00:51:08.602645+0000","last_peered":"2024-02-13T00:51:08.602645+0000","last_clean":"2024-02-13T00:51:08.602645+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:51:08.602645+0000","last_undegraded":"2024-02-13T00:51:08.602645+0000","last_fullsized":"2024-02-13T00:51:08.602645+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:31.621170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:31.621170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":486,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602634+0000","last_change":"2024-02-13T00:46:27.706096+0000","last_active":"2024-02-13T00:51:08.602634+0000","last_peered":"2024-02-13T00:51:08.602634+0000","last_clean":"2024-02-13T00:51:08.602634+0000","last_became_active":"2024-02-13T00:44:50.415894+0000","last_became_peered":"2024-02-13T00:44:50.415894+0000","last_unstale":"2024-02-13T00:51:08.602634+0000","last_undegraded":"2024-02-13T00:51:08.602634+0000","last_fullsized":"2024-02-13T00:51:08.602634+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'21","last_scrub_stamp":"2024-02-13T00:46:27.706006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:46:27.706006+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":422,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604704+0000","last_change":"2024-02-13T00:49:43.067713+0000","last_active":"2024-02-13T00:51:08.604704+0000","last_peered":"2024-02-13T00:51:08.604704+0000","last_clean":"2024-02-13T00:51:08.604704+0000","last_became_active":"2024-02-13T00:46:01.451790+0000","last_became_peered":"2024-02-13T00:46:01.451790+0000","last_unstale":"2024-02-13T00:51:08.604704+0000","last_undegraded":"2024-02-13T00:51:08.604704+0000","last_fullsized":"2024-02-13T00:51:08.604704+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:43.067615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:43.067615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":609,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602300+0000","last_change":"2024-02-13T00:47:36.885611+0000","last_active":"2024-02-13T00:51:08.602300+0000","last_peered":"2024-02-13T00:51:08.602300+0000","last_clean":"2024-02-13T00:51:08.602300+0000","last_became_active":"2024-02-13T00:45:15.419668+0000","last_became_peered":"2024-02-13T00:45:15.419668+0000","last_unstale":"2024-02-13T00:51:08.602300+0000","last_undegraded":"2024-02-13T00:51:08.602300+0000","last_fullsized":"2024-02-13T00:51:08.602300+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:36.885510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.885510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":520,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603613+0000","last_change":"2024-02-13T00:50:15.846534+0000","last_active":"2024-02-13T00:51:08.603613+0000","last_peered":"2024-02-13T00:51:08.603613+0000","last_clean":"2024-02-13T00:51:08.603613+0000","last_became_active":"2024-02-13T00:46:01.455940+0000","last_became_peered":"2024-02-13T00:46:01.455940+0000","last_unstale":"2024-02-13T00:51:08.603613+0000","last_undegraded":"2024-02-13T00:51:08.603613+0000","last_fullsized":"2024-02-13T00:51:08.603613+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:15.846461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:15.846461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":504,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544704+0000","last_change":"2024-02-13T00:46:36.089483+0000","last_active":"2024-02-13T00:51:08.544704+0000","last_peered":"2024-02-13T00:51:08.544704+0000","last_clean":"2024-02-13T00:51:08.544704+0000","last_became_active":"2024-02-13T00:45:15.419364+0000","last_became_peered":"2024-02-13T00:45:15.419364+0000","last_unstale":"2024-02-13T00:51:08.544704+0000","last_undegraded":"2024-02-13T00:51:08.544704+0000","last_fullsized":"2024-02-13T00:51:08.544704+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'14","last_scrub_stamp":"2024-02-13T00:46:36.089402+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:46:36.089402+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":347,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605462+0000","last_change":"2024-02-13T00:47:47.284092+0000","last_active":"2024-02-13T00:51:08.605462+0000","last_peered":"2024-02-13T00:51:08.605462+0000","last_clean":"2024-02-13T00:51:08.605462+0000","last_became_active":"2024-02-13T00:46:01.451997+0000","last_became_peered":"2024-02-13T00:46:01.451997+0000","last_unstale":"2024-02-13T00:51:08.605462+0000","last_undegraded":"2024-02-13T00:51:08.605462+0000","last_fullsized":"2024-02-13T00:51:08.605462+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.284014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.284014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":561,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.545213+0000","last_change":"2024-02-13T00:49:01.133798+0000","last_active":"2024-02-13T00:51:08.545213+0000","last_peered":"2024-02-13T00:51:08.545213+0000","last_clean":"2024-02-13T00:51:08.545213+0000","last_became_active":"2024-02-13T00:46:01.445141+0000","last_became_peered":"2024-02-13T00:46:01.445141+0000","last_unstale":"2024-02-13T00:51:08.545213+0000","last_undegraded":"2024-02-13T00:51:08.545213+0000","last_fullsized":"2024-02-13T00:51:08.545213+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:01.133709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:49:01.133709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":329,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603375+0000","last_change":"2024-02-13T00:45:30.186208+0000","last_active":"2024-02-13T00:51:08.603375+0000","last_peered":"2024-02-13T00:51:08.603375+0000","last_clean":"2024-02-13T00:51:08.603375+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:51:08.603375+0000","last_undegraded":"2024-02-13T00:51:08.603375+0000","last_fullsized":"2024-02-13T00:51:08.603375+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:30.186129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:30.186129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":387,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.603456+0000","last_change":"2024-02-13T00:46:06.251713+0000","last_active":"2024-02-13T00:51:08.603456+0000","last_peered":"2024-02-13T00:51:08.603456+0000","last_clean":"2024-02-13T00:51:08.603456+0000","last_became_active":"2024-02-13T00:46:01.442464+0000","last_became_peered":"2024-02-13T00:46:01.442464+0000","last_unstale":"2024-02-13T00:51:08.603456+0000","last_undegraded":"2024-02-13T00:51:08.603456+0000","last_fullsized":"2024-02-13T00:51:08.603456+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'20","last_scrub_stamp":"2024-02-13T00:43:44.241358+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:43:44.241358+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":422,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604971+0000","last_change":"2024-02-13T00:47:47.711551+0000","last_active":"2024-02-13T00:51:08.604971+0000","last_peered":"2024-02-13T00:51:08.604971+0000","last_clean":"2024-02-13T00:51:08.604971+0000","last_became_active":"2024-02-13T00:46:01.447654+0000","last_became_peered":"2024-02-13T00:46:01.447654+0000","last_unstale":"2024-02-13T00:51:08.604971+0000","last_undegraded":"2024-02-13T00:51:08.604971+0000","last_fullsized":"2024-02-13T00:51:08.604971+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.711486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.711486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":260,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.544241+0000","last_change":"2024-02-13T00:47:53.826039+0000","last_active":"2024-02-13T00:51:08.544241+0000","last_peered":"2024-02-13T00:51:08.544241+0000","last_clean":"2024-02-13T00:51:08.544241+0000","last_became_active":"2024-02-13T00:46:01.448971+0000","last_became_peered":"2024-02-13T00:46:01.448971+0000","last_unstale":"2024-02-13T00:51:08.544241+0000","last_undegraded":"2024-02-13T00:51:08.544241+0000","last_fullsized":"2024-02-13T00:51:08.544241+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:53.825971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:53.825971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":587,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.545376+0000","last_change":"2024-02-13T00:45:32.233265+0000","last_active":"2024-02-13T00:51:08.545376+0000","last_peered":"2024-02-13T00:51:08.545376+0000","last_clean":"2024-02-13T00:51:08.545376+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:51:08.545376+0000","last_undegraded":"2024-02-13T00:51:08.545376+0000","last_fullsized":"2024-02-13T00:51:08.545376+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:45:32.233197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:32.233197+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":521,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.602448+0000","last_change":"2024-02-13T00:47:44.757362+0000","last_active":"2024-02-13T00:51:08.602448+0000","last_peered":"2024-02-13T00:51:08.602448+0000","last_clean":"2024-02-13T00:51:08.602448+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:51:08.602448+0000","last_undegraded":"2024-02-13T00:51:08.602448+0000","last_fullsized":"2024-02-13T00:51:08.602448+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:47:44.757262+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:47:44.757262+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":613,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.604911+0000","last_change":"2024-02-13T00:48:54.818104+0000","last_active":"2024-02-13T00:51:08.604911+0000","last_peered":"2024-02-13T00:51:08.604911+0000","last_clean":"2024-02-13T00:51:08.604911+0000","last_became_active":"2024-02-13T00:45:15.434224+0000","last_became_peered":"2024-02-13T00:45:15.434224+0000","last_unstale":"2024-02-13T00:51:08.604911+0000","last_undegraded":"2024-02-13T00:51:08.604911+0000","last_fullsized":"2024-02-13T00:51:08.604911+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:54.817999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.817999+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":338,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-02-13T00:51:08.605175+0000","last_change":"2024-02-13T00:51:03.383732+0000","last_active":"2024-02-13T00:51:08.605175+0000","last_peered":"2024-02-13T00:51:08.605175+0000","last_clean":"2024-02-13T00:51:08.605175+0000","last_became_active":"2024-02-13T00:45:15.435259+0000","last_became_peered":"2024-02-13T00:45:15.435259+0000","last_unstale":"2024-02-13T00:51:08.605175+0000","last_undegraded":"2024-02-13T00:51:08.605175+0000","last_fullsized":"2024-02-13T00:51:08.605175+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:03.383666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:03.383666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":240,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":40,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":14,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":24,"acting":24,"num_store_stats":4},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1492,"num_read_kb":1217,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":9606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":11656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":970,"ondisk_log_size":970,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":796,"num_read_kb":791,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":227633266852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":45,"seq":193273528490,"num_pgs":161,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313224,"kb_used_data":3712,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431928,"statfs":{"total":95995035648,"available":95674294272,"internally_reserved":0,"allocated":3801088,"data_stored":1868832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.47599999999999998,"15min":0.54400000000000004},"min":{"1min":0.33900000000000002,"5min":0.25600000000000001,"15min":0.19700000000000001},"max":{"1min":0.76100000000000001,"5min":0.76900000000000002,"15min":8.1340000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.503,"15min":0.61199999999999999},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.221},"max":{"1min":0.72399999999999998,"5min":1.0309999999999999,"15min":11.367000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Tue Feb 13 00:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48399999999999999,"15min":0.88300000000000001},"min":{"1min":0.30099999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.68799999999999994,"5min":1.1080000000000001,"15min":54.002000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.498,"15min":0.68100000000000005},"min":{"1min":0.38400000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.77700000000000002,"5min":0.95199999999999996,"15min":12.403},"last":0.436}]},{"osd":2,"last update":"Tue Feb 13 00:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51500000000000001,"15min":0.71099999999999997},"min":{"1min":0.379,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66700000000000004,"5min":1.587,"15min":12.477},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.495,"15min":0.67900000000000005},"min":{"1min":0.35399999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.70499999999999996,"5min":1.1899999999999999,"15min":11.702},"last":0.45900000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54900000000000004,"15min":0.67800000000000005},"min":{"1min":0.31900000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66400000000000003,"5min":5.931,"15min":8.0519999999999996},"last":0.499},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.77200000000000002},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.67300000000000004,"5min":1.2330000000000001,"15min":19.081},"last":0.52000000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.41899999999999998,"15min":0.41699999999999998},"min":{"1min":0.316,"5min":0.26100000000000001,"15min":0.24199999999999999},"max":{"1min":0.67000000000000004,"5min":0.71899999999999997,"15min":1.0649999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44},"min":{"1min":0.29899999999999999,"5min":0.22800000000000001,"15min":0.221},"max":{"1min":0.66900000000000004,"5min":1.008,"15min":1.008},"last":0.38300000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.50800000000000001,"15min":0.51600000000000001},"min":{"1min":0.33600000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67400000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.71699999999999997,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.39800000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.46999999999999997,"15min":0.48099999999999998},"min":{"1min":0.372,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":1.173},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.40600000000000003,"5min":0.28299999999999997,"15min":0.26900000000000002},"max":{"1min":0.73099999999999998,"5min":0.80700000000000005,"15min":1.262},"last":0.44600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":158,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305060,"kb_used_data":3740,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440092,"statfs":{"total":95995035648,"available":95682654208,"internally_reserved":0,"allocated":3829760,"data_stored":1874018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.33200000000000002,"15min":0.95299999999999996},"min":{"1min":0.19,"5min":0.187,"15min":0.187},"max":{"1min":0.60499999999999998,"5min":0.68200000000000005,"15min":61.856000000000002},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.67000000000000004},"min":{"1min":0.26400000000000001,"5min":0.19900000000000001,"15min":0.16500000000000001},"max":{"1min":0.48199999999999998,"5min":0.70199999999999996,"15min":61.908999999999999},"last":0.316}]},{"osd":2,"last update":"Tue Feb 13 00:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.84499999999999997},"min":{"1min":0.26400000000000001,"5min":0.22900000000000001,"15min":0.20599999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":32.161000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.35599999999999998,"15min":0.82399999999999995},"min":{"1min":0.245,"5min":0.187,"15min":0.187},"max":{"1min":0.441,"5min":0.56200000000000006,"15min":29.693000000000001},"last":0.28399999999999997}]},{"osd":3,"last update":"Tue Feb 13 00:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":1.2609999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":0.497,"5min":0.59099999999999997,"15min":131.23099999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.379,"15min":0.85499999999999998},"min":{"1min":0.22800000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.48799999999999999,"5min":0.61799999999999999,"15min":49.055},"last":0.33300000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:50:53 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.753},"min":{"1min":0.36499999999999999,"5min":0.33500000000000002,"15min":0.27900000000000003},"max":{"1min":0.68700000000000006,"5min":0.77800000000000002,"15min":16.547000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57399999999999995,"15min":0.71199999999999997},"min":{"1min":0.39300000000000002,"5min":0.34699999999999998,"15min":0.24199999999999999},"max":{"1min":0.73699999999999999,"5min":0.85199999999999998,"15min":14.731999999999999},"last":0.53600000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.57099999999999995},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59899999999999998,"15min":0.58699999999999997},"min":{"1min":0.437,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.42799999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56200000000000006,"15min":0.71699999999999997},"min":{"1min":0.38600000000000001,"5min":0.32300000000000001,"15min":0.29099999999999998},"max":{"1min":1.0700000000000001,"5min":1.9279999999999999,"15min":14.772},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.92200000000000004},"min":{"1min":0.41199999999999998,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":26.600000000000001},"last":0.51800000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56699999999999995,"15min":0.72199999999999998},"min":{"1min":0.38200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.0349999999999999,"5min":1.8680000000000001,"15min":14.794},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56699999999999995,"15min":0.92400000000000004},"min":{"1min":0.45900000000000002,"5min":0.33000000000000002,"15min":0.32200000000000001},"max":{"1min":1.246,"5min":1.246,"15min":26.693999999999999},"last":0.48899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738567,"num_pgs":156,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309124,"kb_used_data":3708,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436028,"statfs":{"total":95995035648,"available":95678492672,"internally_reserved":0,"allocated":3796992,"data_stored":1865435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.35199999999999998,"15min":0.622},"min":{"1min":0.24199999999999999,"5min":0.191,"15min":0.189},"max":{"1min":0.44500000000000001,"5min":1.7370000000000001,"15min":30.184000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45700000000000002},"min":{"1min":0.26400000000000001,"5min":0.20499999999999999,"15min":0.17599999999999999},"max":{"1min":0.58899999999999997,"5min":2.839,"15min":5.6959999999999997},"last":0.34499999999999997}]},{"osd":2,"last update":"Tue Feb 13 00:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40300000000000002,"15min":0.498},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.20999999999999999},"max":{"1min":0.64300000000000002,"5min":1.3700000000000001,"15min":7.0839999999999996},"last":0.376},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42499999999999999,"15min":0.66900000000000004},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.185},"max":{"1min":0.53600000000000003,"5min":3.6949999999999998,"15min":24.459},"last":0.40699999999999997}]},{"osd":3,"last update":"Tue Feb 13 00:50:21 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.42799999999999999,"15min":0.48499999999999999},"min":{"1min":0.32600000000000001,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":0.48499999999999999,"5min":4.8710000000000004,"15min":8.718},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.442,"15min":0.53400000000000003},"min":{"1min":0.315,"5min":0.26900000000000002,"15min":0.222},"max":{"1min":0.47099999999999997,"5min":2.3140000000000001,"15min":7.1139999999999999},"last":0.41599999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:51:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55900000000000005,"15min":0.80300000000000005},"min":{"1min":0.371,"5min":0.29899999999999999,"15min":0.24399999999999999},"max":{"1min":0.79600000000000004,"5min":1.639,"15min":24.381},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.56899999999999995,"15min":0.66700000000000004},"min":{"1min":0.48399999999999999,"5min":0.31,"15min":0.28699999999999998},"max":{"1min":0.78900000000000003,"5min":0.88200000000000001,"15min":8.6950000000000003},"last":0.45800000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.5,"15min":0.495},"min":{"1min":0.40799999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.48599999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53300000000000003,"15min":0.628},"min":{"1min":0.40500000000000003,"5min":0.34100000000000003,"15min":0.31},"max":{"1min":0.77200000000000002,"5min":1.4990000000000001,"15min":8.6129999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.65500000000000003},"min":{"1min":0.42899999999999999,"5min":0.34000000000000002,"15min":0.32400000000000001},"max":{"1min":0.72099999999999997,"5min":2.206,"15min":14.263},"last":0.53800000000000003}]},{"osd":7,"last update":"Tue Feb 13 00:51:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.54800000000000004,"15min":0.66400000000000003},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.317},"max":{"1min":0.79500000000000004,"5min":0.80800000000000005,"15min":8.6679999999999993},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57899999999999996,"15min":0.67000000000000004},"min":{"1min":0.45300000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86899999999999999,"5min":1.6910000000000001,"15min":8.1609999999999996},"last":0.57199999999999995}]}]},{"osd":2,"up_from":0,"seq":77309411518,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182584,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986356,"num_pgs":137,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305004,"kb_used_data":3684,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440148,"statfs":{"total":95995035648,"available":95682711552,"internally_reserved":0,"allocated":3772416,"data_stored":1856012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47399999999999998,"15min":0.59199999999999997},"min":{"1min":0.32800000000000001,"5min":0.248,"15min":0.21199999999999999},"max":{"1min":0.63700000000000001,"5min":0.71999999999999997,"15min":9.3399999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.58999999999999997,"15min":0.60899999999999999},"min":{"1min":0.35599999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.745,"5min":11.532,"15min":13.833},"last":0.40999999999999998}]},{"osd":1,"last update":"Tue Feb 13 00:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.47499999999999998,"15min":0.752},"min":{"1min":0.38300000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.67300000000000004,"5min":0.76900000000000002,"15min":53.188000000000002},"last":0.435},{"interface":"front","average":{"1min":0.5,"5min":0.59499999999999997,"15min":0.69699999999999995},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.25600000000000001},"max":{"1min":0.68600000000000005,"5min":11.497999999999999,"15min":44.793999999999997},"last":0.51700000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.379,"5min":0.31900000000000001,"15min":0.26900000000000002},"max":{"1min":0.66600000000000004,"5min":4.2199999999999998,"15min":8.6829999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.51800000000000002,"15min":0.627},"min":{"1min":0.36399999999999999,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.64000000000000001,"5min":4.0730000000000004,"15min":18.911000000000001},"last":0.44700000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.52400000000000002,"15min":0.65300000000000002},"min":{"1min":0.35899999999999999,"5min":0.312,"15min":0.30599999999999999},"max":{"1min":0.69299999999999995,"5min":4.0090000000000003,"15min":23.5},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.71199999999999997},"min":{"1min":0.40799999999999997,"5min":0.33700000000000002,"15min":0.30599999999999999},"max":{"1min":0.69399999999999995,"5min":3.9729999999999999,"15min":36.152999999999999},"last":0.501}]},{"osd":5,"last update":"Tue Feb 13 00:50:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.33700000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.69399999999999995,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.33600000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.435,"15min":0.443},"min":{"1min":0.378,"5min":0.26800000000000002,"15min":0.25700000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":3.153},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.46400000000000002,"15min":0.47199999999999998},"min":{"1min":0.34499999999999997,"5min":0.25800000000000001,"15min":0.254},"max":{"1min":0.65900000000000003,"5min":0.73099999999999998,"15min":4.9390000000000001},"last":0.28199999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.39300000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":1.575},"last":0.371},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.48799999999999999,"15min":0.48399999999999999},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.28100000000000003},"max":{"1min":0.877,"5min":0.877,"15min":2.46},"last":0.35399999999999998}]}]},{"osd":5,"up_from":0,"seq":2057289334823,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":3328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:51:15.874 INFO:tasks.thrashosds.thrasher:Setting 6.23 to [3, 6, 7] 2024-02-13T00:51:15.874 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.23', '3', '6', '7'] 2024-02-13T00:51:15.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pg-upmap 6.23 3 6 7 2024-02-13T00:51:15.877 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:15.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:15.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:51:15.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:16.042 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:16.061 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:16.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:16.098 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:51:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:51:16.153 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:16.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:16.194 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:16.364 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:51:16.364 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:51:16.393 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:16.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:16.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:16.417 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:51:16] "GET /metrics HTTP/1.1" 200 189212 "" "Prometheus/2.33.4" 2024-02-13T00:51:16.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:51:16.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:16.527 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:16.650 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:16.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:15.105258+0000 osd.7 (osd.7) 112 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:15.105474+0000 osd.7 (osd.7) 113 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:15.282566+0000 osd.6 (osd.6) 197 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:51:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:15.285529+0000 osd.6 (osd.6) 198 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:51:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:15.527040+0000 osd.1 (osd.1) 273 : cluster [DBG] 6.21 scrub starts 2024-02-13T00:51:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:15.530049+0000 osd.1 (osd.1) 274 : cluster [DBG] 6.21 scrub ok 2024-02-13T00:51:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:15.629673+0000 mon.a (mon.0) 3244 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:15.833190+0000 mgr.y (mgr.14544) 553 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:16.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:16.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:16.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:16.779 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:15.105258+0000 osd.7 (osd.7) 112 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:16.779 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:15.105474+0000 osd.7 (osd.7) 113 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:16.779 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:15.282566+0000 osd.6 (osd.6) 197 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:51:16.779 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:15.285529+0000 osd.6 (osd.6) 198 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:51:16.779 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:15.527040+0000 osd.1 (osd.1) 273 : cluster [DBG] 6.21 scrub starts 2024-02-13T00:51:16.779 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:15.530049+0000 osd.1 (osd.1) 274 : cluster [DBG] 6.21 scrub ok 2024-02-13T00:51:16.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:15.629673+0000 mon.a (mon.0) 3244 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:16.780 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:15.833190+0000 mgr.y (mgr.14544) 553 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:16.780 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:15.105258+0000 osd.7 (osd.7) 112 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:16.780 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:15.105474+0000 osd.7 (osd.7) 113 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:16.780 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:15.282566+0000 osd.6 (osd.6) 197 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:51:16.780 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:15.285529+0000 osd.6 (osd.6) 198 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:51:16.780 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:15.527040+0000 osd.1 (osd.1) 273 : cluster [DBG] 6.21 scrub starts 2024-02-13T00:51:16.780 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:15.530049+0000 osd.1 (osd.1) 274 : cluster [DBG] 6.21 scrub ok 2024-02-13T00:51:16.780 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:15.629673+0000 mon.a (mon.0) 3244 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:16.780 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:15.833190+0000 mgr.y (mgr.14544) 553 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:16.781 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:16.980 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:16.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:16.998 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:17.067 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-13T00:51:17.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:17.111 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:17.332 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:17.458 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:17.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:17.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:17.562 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:17.593 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:17.785 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:16.503743+0000 mgr.y (mgr.14544) 554 : cluster [DBG] pgmap v981: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:17.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:16.532951+0000 osd.1 (osd.1) 275 : cluster [DBG] 6.4 scrub starts 2024-02-13T00:51:17.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:16.535957+0000 osd.1 (osd.1) 276 : cluster [DBG] 6.4 scrub ok 2024-02-13T00:51:17.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:16.547806+0000 mon.a (mon.0) 3245 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:17.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:16.547889+0000 mon.a (mon.0) 3246 : cluster [DBG] osdmap e651: 8 total, 8 up, 4 in 2024-02-13T00:51:17.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:16.550361+0000 mon.a (mon.0) 3247 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-112"}]: dispatch 2024-02-13T00:51:17.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:16.503743+0000 mgr.y (mgr.14544) 554 : cluster [DBG] pgmap v981: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:17.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:16.532951+0000 osd.1 (osd.1) 275 : cluster [DBG] 6.4 scrub starts 2024-02-13T00:51:17.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:16.535957+0000 osd.1 (osd.1) 276 : cluster [DBG] 6.4 scrub ok 2024-02-13T00:51:17.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:16.547806+0000 mon.a (mon.0) 3245 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:17.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:16.547889+0000 mon.a (mon.0) 3246 : cluster [DBG] osdmap e651: 8 total, 8 up, 4 in 2024-02-13T00:51:17.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:16.550361+0000 mon.a (mon.0) 3247 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-112"}]: dispatch 2024-02-13T00:51:17.818 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:16.503743+0000 mgr.y (mgr.14544) 554 : cluster [DBG] pgmap v981: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:17.818 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:16.532951+0000 osd.1 (osd.1) 275 : cluster [DBG] 6.4 scrub starts 2024-02-13T00:51:17.818 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:16.535957+0000 osd.1 (osd.1) 276 : cluster [DBG] 6.4 scrub ok 2024-02-13T00:51:17.818 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:16.547806+0000 mon.a (mon.0) 3245 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:17.818 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:16.547889+0000 mon.a (mon.0) 3246 : cluster [DBG] osdmap e651: 8 total, 8 up, 4 in 2024-02-13T00:51:17.818 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:16.550361+0000 mon.a (mon.0) 3247 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-112"}]: dispatch 2024-02-13T00:51:18.141 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:18.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:18.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:18.275 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:18.523 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:18.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:18.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:18.653 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:18.804 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:51:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:51:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:51:18.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:17.265010+0000 osd.6 (osd.6) 199 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:51:18.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:17.268185+0000 osd.6 (osd.6) 200 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:51:18.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:17.544787+0000 mon.a (mon.0) 3248 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:18.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:17.549319+0000 mon.a (mon.0) 3249 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-112"}]': finished 2024-02-13T00:51:18.805 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:17.549386+0000 mon.a (mon.0) 3250 : cluster [DBG] osdmap e652: 8 total, 8 up, 4 in 2024-02-13T00:51:18.817 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:17.265010+0000 osd.6 (osd.6) 199 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:51:18.817 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:17.268185+0000 osd.6 (osd.6) 200 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:51:18.817 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:17.544787+0000 mon.a (mon.0) 3248 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:18.817 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:17.549319+0000 mon.a (mon.0) 3249 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-112"}]': finished 2024-02-13T00:51:18.817 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:17.549386+0000 mon.a (mon.0) 3250 : cluster [DBG] osdmap e652: 8 total, 8 up, 4 in 2024-02-13T00:51:18.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:17.265010+0000 osd.6 (osd.6) 199 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:51:18.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:17.268185+0000 osd.6 (osd.6) 200 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:51:18.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:17.544787+0000 mon.a (mon.0) 3248 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:18.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:17.549319+0000 mon.a (mon.0) 3249 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-112"}]': finished 2024-02-13T00:51:18.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:17.549386+0000 mon.a (mon.0) 3250 : cluster [DBG] osdmap e652: 8 total, 8 up, 4 in 2024-02-13T00:51:18.854 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:18.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:18.959 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:18.988 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:19.077 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:51:19.077 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:51:19.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 -- 172.21.15.162:0/3627463437 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51e00fcb30 msgr2=0x7f51e00fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:19.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 --2- 172.21.15.162:0/3627463437 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51e00fcb30 0x7f51e00fcf10 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 rx=0x7f51d0004870 tx=0x7f51d0009cb0).stop 2024-02-13T00:51:19.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 -- 172.21.15.162:0/3627463437 shutdown_connections 2024-02-13T00:51:19.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 --2- 172.21.15.162:0/3627463437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51e00f73a0 0x7f51e01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:19.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 --2- 172.21.15.162:0/3627463437 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51e00fcb30 0x7f51e00fcf10 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 --2- 172.21.15.162:0/3627463437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51e00fd490 0x7f51e00f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 -- 172.21.15.162:0/3627463437 >> 172.21.15.162:0/3627463437 conn(0x7f51e006eb90 msgr2=0x7f51e00f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 -- 172.21.15.162:0/3627463437 shutdown_connections 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 -- 172.21.15.162:0/3627463437 wait complete. 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 librados: starting msgr at 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 librados: starting objecter 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 Processor -- start 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 -- start start 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 librados: setting wanted keys 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.071+0000 7f51e6360700 1 librados: calling monclient init 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51e6360700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51e00f73a0 0x7f51e0109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51e6360700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51e00fcb30 0x7f51e0109780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51e6360700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51e00fd490 0x7f51e0109d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:19.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51e6360700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e0114470 con 0x7f51e00fcb30 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51e6360700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e01145e0 con 0x7f51e00fd490 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51e6360700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e01147a0 con 0x7f51e00f73a0 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dffff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51e00f73a0 0x7f51e0109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dffff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51e00f73a0 0x7f51e0109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:52338/0 (socket says 172.21.15.162:52338) 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dffff700 1 -- 172.21.15.162:0/1189441742 learned_addr learned my addr 172.21.15.162:0/1189441742 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51df7fe700 1 --2- 172.21.15.162:0/1189441742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51e00fcb30 0x7f51e0109780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51e48fd700 1 --2- 172.21.15.162:0/1189441742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51e00fd490 0x7f51e0109d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dffff700 1 -- 172.21.15.162:0/1189441742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51e00fd490 msgr2=0x7f51e0109d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dffff700 1 --2- 172.21.15.162:0/1189441742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51e00fd490 0x7f51e0109d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dffff700 1 -- 172.21.15.162:0/1189441742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51e00fcb30 msgr2=0x7f51e0109780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dffff700 1 --2- 172.21.15.162:0/1189441742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51e00fcb30 0x7f51e0109780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dffff700 1 -- 172.21.15.162:0/1189441742 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d0005040 con 0x7f51e00f73a0 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dffff700 1 --2- 172.21.15.162:0/1189441742 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51e00f73a0 0x7f51e0109290 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 rx=0x7f51d0009eb0 tx=0x7f51d000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dd7fa700 10 client.?.objecter ms_handle_connect 0x7f51e00f73a0 2024-02-13T00:51:19.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dd7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dd7fa700 1 -- 172.21.15.162:0/1189441742 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d000e070 con 0x7f51e00f73a0 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dd7fa700 1 -- 172.21.15.162:0/1189441742 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51d0013da0 con 0x7f51e00f73a0 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51e6360700 1 -- 172.21.15.162:0/1189441742 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e0114a20 con 0x7f51e00f73a0 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51dd7fa700 1 -- 172.21.15.162:0/1189441742 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0019070 con 0x7f51e00f73a0 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.072+0000 7f51e6360700 10 client.37976.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.073+0000 7f51e6360700 1 -- 172.21.15.162:0/1189441742 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e0114f00 con 0x7f51e00f73a0 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.073+0000 7f51e6360700 1 librados: init done 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.073+0000 7f51e6360700 1 -- 172.21.15.162:0/1189441742 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e0105fe0 con 0x7f51e00f73a0 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.074+0000 7f51dd7fa700 1 -- 172.21.15.162:0/1189441742 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f51d001e420 con 0x7f51e00f73a0 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.074+0000 7f51dd7fa700 1 --2- 172.21.15.162:0/1189441742 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51c8062c90 0x7f51c8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.074+0000 7f51dd7fa700 1 -- 172.21.15.162:0/1189441742 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(653..653 src has 449..653) v4 ==== 8178+0+0 (secure 0 0 0) 0x7f51d007dcd0 con 0x7f51e00f73a0 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.074+0000 7f51dd7fa700 10 client.37976.objecter ms_dispatch 0x7f51e0001550 osd_map(653..653 src has 449..653) v4 2024-02-13T00:51:19.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.074+0000 7f51dd7fa700 3 client.37976.objecter handle_osd_map got epochs [653,653] > 0 2024-02-13T00:51:19.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.074+0000 7f51dd7fa700 3 client.37976.objecter handle_osd_map decoding full epoch 653 2024-02-13T00:51:19.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.074+0000 7f51df7fe700 1 --2- 172.21.15.162:0/1189441742 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51c8062c90 0x7f51c8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:19.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.075+0000 7f51dd7fa700 20 client.37976.objecter dump_active .. 0 homeless 2024-02-13T00:51:19.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.075+0000 7f51dd7fa700 10 client.37976.objecter ms_handle_connect 0x7f51c8062c90 2024-02-13T00:51:19.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.075+0000 7f51df7fe700 1 --2- 172.21.15.162:0/1189441742 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51c8062c90 0x7f51c8065150 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 rx=0x7f51e00617b0 tx=0x7f51d400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:19.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.079+0000 7f51dd7fa700 1 -- 172.21.15.162:0/1189441742 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f51d0051120 con 0x7f51e00f73a0 2024-02-13T00:51:19.085 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:19.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:19.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:19.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:19.217 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:19.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.263+0000 7f51e6360700 1 -- 172.21.15.162:0/1189441742 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f51e005fc30 con 0x7f51e00f73a0 2024-02-13T00:51:19.366 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:19.367 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:19.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:19.471 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:19.502 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:19.570 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.566+0000 7f51dd7fa700 1 -- 172.21.15.162:0/1189441742 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v654) v1 ==== 87+0+0 (secure 0 0 0) 0x7f51d0023030 con 0x7f51e00f73a0 2024-02-13T00:51:19.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.571+0000 7f51e6360700 10 librados: watch_flush enter 2024-02-13T00:51:19.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 10 librados: watch_flush exit 2024-02-13T00:51:19.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 20 client.37976.objecter shutdown clearing up homeless session... 2024-02-13T00:51:19.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 10 client.37976.objecter successfully canceled tick 2024-02-13T00:51:19.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 1 -- 172.21.15.162:0/1189441742 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51c8062c90 msgr2=0x7f51c8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:19.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 1 --2- 172.21.15.162:0/1189441742 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51c8062c90 0x7f51c8065150 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 rx=0x7f51e00617b0 tx=0x7f51d400b040).stop 2024-02-13T00:51:19.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 1 -- 172.21.15.162:0/1189441742 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51e00f73a0 msgr2=0x7f51e0109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:19.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 1 --2- 172.21.15.162:0/1189441742 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51e00f73a0 0x7f51e0109290 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 rx=0x7f51d0009eb0 tx=0x7f51d000b040).stop 2024-02-13T00:51:19.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 1 -- 172.21.15.162:0/1189441742 shutdown_connections 2024-02-13T00:51:19.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 1 --2- 172.21.15.162:0/1189441742 >> v2:172.21.15.162:6800/2866534053 conn(0x7f51c8062c90 0x7f51c8065150 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:19.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 1 --2- 172.21.15.162:0/1189441742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51e00fd490 0x7f51e0109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:19.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 1 --2- 172.21.15.162:0/1189441742 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51e00f73a0 0x7f51e0109290 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:19.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 1 --2- 172.21.15.162:0/1189441742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51e00fcb30 0x7f51e0109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:19.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.572+0000 7f51e6360700 1 -- 172.21.15.162:0/1189441742 >> 172.21.15.162:0/1189441742 conn(0x7f51e006eb90 msgr2=0x7f51e00fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:19.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.573+0000 7f51e6360700 1 -- 172.21.15.162:0/1189441742 shutdown_connections 2024-02-13T00:51:19.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.574+0000 7f51e6360700 1 -- 172.21.15.162:0/1189441742 wait complete. 2024-02-13T00:51:19.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:19.574+0000 7f51e6360700 1 librados: shutdown 2024-02-13T00:51:19.578 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:51:19.630 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:18.261831+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.1b deep-scrub starts 2024-02-13T00:51:19.630 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:18.264731+0000 osd.6 (osd.6) 202 : cluster [DBG] 2.1b deep-scrub ok 2024-02-13T00:51:19.630 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:18.504155+0000 mgr.y (mgr.14544) 555 : cluster [DBG] pgmap v984: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:19.631 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:18.560803+0000 mon.a (mon.0) 3251 : cluster [DBG] osdmap e653: 8 total, 8 up, 4 in 2024-02-13T00:51:19.631 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:19.264027+0000 mon.c (mon.2) 479 : audit [INF] from='client.? 172.21.15.162:0/1189441742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:51:19.631 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:19.264373+0000 mon.a (mon.0) 3252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:51:19.631 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:19.283703+0000 osd.0 (osd.0) 249 : cluster [DBG] 28.c scrub starts 2024-02-13T00:51:19.631 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:19.286607+0000 osd.0 (osd.0) 250 : cluster [DBG] 28.c scrub ok 2024-02-13T00:51:19.631 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:19.341610+0000 mon.a (mon.0) 3253 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:19.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:18.261831+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.1b deep-scrub starts 2024-02-13T00:51:19.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:18.264731+0000 osd.6 (osd.6) 202 : cluster [DBG] 2.1b deep-scrub ok 2024-02-13T00:51:19.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:18.504155+0000 mgr.y (mgr.14544) 555 : cluster [DBG] pgmap v984: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:19.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:18.560803+0000 mon.a (mon.0) 3251 : cluster [DBG] osdmap e653: 8 total, 8 up, 4 in 2024-02-13T00:51:19.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:19.264027+0000 mon.c (mon.2) 479 : audit [INF] from='client.? 172.21.15.162:0/1189441742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:51:19.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:19.264373+0000 mon.a (mon.0) 3252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:51:19.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:19.283703+0000 osd.0 (osd.0) 249 : cluster [DBG] 28.c scrub starts 2024-02-13T00:51:19.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:19.286607+0000 osd.0 (osd.0) 250 : cluster [DBG] 28.c scrub ok 2024-02-13T00:51:19.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:19.341610+0000 mon.a (mon.0) 3253 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:19.721 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:18.261831+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.1b deep-scrub starts 2024-02-13T00:51:19.721 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:18.264731+0000 osd.6 (osd.6) 202 : cluster [DBG] 2.1b deep-scrub ok 2024-02-13T00:51:19.721 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:18.504155+0000 mgr.y (mgr.14544) 555 : cluster [DBG] pgmap v984: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:19.721 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:18.560803+0000 mon.a (mon.0) 3251 : cluster [DBG] osdmap e653: 8 total, 8 up, 4 in 2024-02-13T00:51:19.721 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:19.264027+0000 mon.c (mon.2) 479 : audit [INF] from='client.? 172.21.15.162:0/1189441742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:51:19.721 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:19.264373+0000 mon.a (mon.0) 3252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:51:19.721 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:19.283703+0000 osd.0 (osd.0) 249 : cluster [DBG] 28.c scrub starts 2024-02-13T00:51:19.722 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:19.286607+0000 osd.0 (osd.0) 250 : cluster [DBG] 28.c scrub ok 2024-02-13T00:51:19.722 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:19.341610+0000 mon.a (mon.0) 3253 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:19.729 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:19.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:19.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:19.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-13T00:51:19.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:19.862 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:20.040 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:20.511 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:20.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:20.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:20.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:20.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.620+0000 7ffb9abe6700 1 -- 172.21.15.162:0/1422955811 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb940fcb30 msgr2=0x7ffb940fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:20.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.620+0000 7ffb9abe6700 1 --2- 172.21.15.162:0/1422955811 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb940fcb30 0x7ffb940fcf10 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 rx=0x7ffb880047b0 tx=0x7ffb88009cb0).stop 2024-02-13T00:51:20.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.620+0000 7ffb9abe6700 1 -- 172.21.15.162:0/1422955811 shutdown_connections 2024-02-13T00:51:20.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.620+0000 7ffb9abe6700 1 --2- 172.21.15.162:0/1422955811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb940f73a0 0x7ffb941039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:20.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.620+0000 7ffb9abe6700 1 --2- 172.21.15.162:0/1422955811 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb940fcb30 0x7ffb940fcf10 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:20.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.620+0000 7ffb9abe6700 1 --2- 172.21.15.162:0/1422955811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb940fd490 0x7ffb940f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:20.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.620+0000 7ffb9abe6700 1 -- 172.21.15.162:0/1422955811 >> 172.21.15.162:0/1422955811 conn(0x7ffb9406eb90 msgr2=0x7ffb940f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:20.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.620+0000 7ffb9abe6700 1 -- 172.21.15.162:0/1422955811 shutdown_connections 2024-02-13T00:51:20.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.620+0000 7ffb9abe6700 1 -- 172.21.15.162:0/1422955811 wait complete. 2024-02-13T00:51:20.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb9abe6700 1 librados: starting msgr at 2024-02-13T00:51:20.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb9abe6700 1 librados: starting objecter 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb9abe6700 1 Processor -- start 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb9abe6700 1 -- start start 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb9abe6700 1 librados: setting wanted keys 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb9abe6700 1 librados: calling monclient init 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb9abe6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb940f73a0 0x7ffb9410d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb9abe6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb940fcb30 0x7ffb9410d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb9abe6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb940fd490 0x7ffb94107210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb9abe6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb94107700 con 0x7ffb940fcb30 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb9abe6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb94107870 con 0x7ffb940fd490 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb9abe6700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb94107a30 con 0x7ffb940f73a0 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb93fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb940fcb30 0x7ffb9410d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb98982700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb940f73a0 0x7ffb9410d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb93fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb940fcb30 0x7ffb9410d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:46976/0 (socket says 172.21.15.162:46976) 2024-02-13T00:51:20.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb93fff700 1 -- 172.21.15.162:0/125392486 learned_addr learned my addr 172.21.15.162:0/125392486 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.621+0000 7ffb99183700 1 --2- 172.21.15.162:0/125392486 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb940fd490 0x7ffb94107210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb98982700 1 -- 172.21.15.162:0/125392486 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb940fd490 msgr2=0x7ffb94107210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb98982700 1 --2- 172.21.15.162:0/125392486 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb940fd490 0x7ffb94107210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb98982700 1 -- 172.21.15.162:0/125392486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb940fcb30 msgr2=0x7ffb9410d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb98982700 1 --2- 172.21.15.162:0/125392486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb940fcb30 0x7ffb9410d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb98982700 1 -- 172.21.15.162:0/125392486 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb80005040 con 0x7ffb940f73a0 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb93fff700 1 --2- 172.21.15.162:0/125392486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb940fcb30 0x7ffb9410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb98982700 1 --2- 172.21.15.162:0/125392486 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb940f73a0 0x7ffb9410d070 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 rx=0x7ffb88004650 tx=0x7ffb88007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb91ffb700 10 client.?.objecter ms_handle_connect 0x7ffb940f73a0 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb91ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb91ffb700 1 -- 172.21.15.162:0/125392486 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb88018070 con 0x7ffb940f73a0 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb9abe6700 1 -- 172.21.15.162:0/125392486 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb88005040 con 0x7ffb940f73a0 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb91ffb700 1 -- 172.21.15.162:0/125392486 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb880092e0 con 0x7ffb940f73a0 2024-02-13T00:51:20.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb91ffb700 1 -- 172.21.15.162:0/125392486 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8801dbd0 con 0x7ffb940f73a0 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb9abe6700 10 client.37982.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb9abe6700 1 -- 172.21.15.162:0/125392486 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb94107f50 con 0x7ffb940f73a0 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.622+0000 7ffb9abe6700 1 librados: init done 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.623+0000 7ffb9abe6700 1 -- 172.21.15.162:0/125392486 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb940475e0 con 0x7ffb940f73a0 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.624+0000 7ffb91ffb700 1 -- 172.21.15.162:0/125392486 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ffb8801d420 con 0x7ffb940f73a0 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.624+0000 7ffb91ffb700 1 --2- 172.21.15.162:0/125392486 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffb84062c90 0x7ffb84065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.624+0000 7ffb91ffb700 1 -- 172.21.15.162:0/125392486 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(655..655 src has 449..655) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ffb8807e700 con 0x7ffb940f73a0 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.624+0000 7ffb91ffb700 10 client.37982.objecter ms_dispatch 0x7ffb94001550 osd_map(655..655 src has 449..655) v4 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.624+0000 7ffb91ffb700 3 client.37982.objecter handle_osd_map got epochs [655,655] > 0 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.624+0000 7ffb91ffb700 3 client.37982.objecter handle_osd_map decoding full epoch 655 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.624+0000 7ffb93fff700 1 --2- 172.21.15.162:0/125392486 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffb84062c90 0x7ffb84065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.624+0000 7ffb91ffb700 20 client.37982.objecter dump_active .. 0 homeless 2024-02-13T00:51:20.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.624+0000 7ffb91ffb700 10 client.37982.objecter ms_handle_connect 0x7ffb84062c90 2024-02-13T00:51:20.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.624+0000 7ffb93fff700 1 --2- 172.21.15.162:0/125392486 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffb84062c90 0x7ffb84065150 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 rx=0x7ffb941089b0 tx=0x7ffb8000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:20.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.628+0000 7ffb91ffb700 1 -- 172.21.15.162:0/125392486 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ffb880519d0 con 0x7ffb940f73a0 2024-02-13T00:51:20.644 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:20.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:19.520380+0000 osd.1 (osd.1) 277 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:51:20.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:19.521316+0000 osd.1 (osd.1) 278 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:51:20.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:19.558493+0000 mon.a (mon.0) 3254 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:20.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:19.565909+0000 mon.a (mon.0) 3255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:51:20.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:19.566013+0000 mon.a (mon.0) 3256 : cluster [DBG] osdmap e654: 8 total, 8 up, 4 in 2024-02-13T00:51:20.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:19.566797+0000 mon.a (mon.0) 3257 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:20.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:20.527684+0000 mon.a (mon.0) 3258 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:20.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:20.527814+0000 mon.a (mon.0) 3259 : cluster [DBG] osdmap e655: 8 total, 8 up, 4 in 2024-02-13T00:51:20.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:19.520380+0000 osd.1 (osd.1) 277 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:51:20.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:19.521316+0000 osd.1 (osd.1) 278 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:51:20.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:19.558493+0000 mon.a (mon.0) 3254 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:20.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:19.565909+0000 mon.a (mon.0) 3255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:51:20.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:19.566013+0000 mon.a (mon.0) 3256 : cluster [DBG] osdmap e654: 8 total, 8 up, 4 in 2024-02-13T00:51:20.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:19.566797+0000 mon.a (mon.0) 3257 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:20.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:20.527684+0000 mon.a (mon.0) 3258 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:20.787 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:20.527814+0000 mon.a (mon.0) 3259 : cluster [DBG] osdmap e655: 8 total, 8 up, 4 in 2024-02-13T00:51:20.791 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:20.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:20.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:20.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:20.812+0000 7ffb9abe6700 1 -- 172.21.15.162:0/125392486 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "6.23", "id": [3, 6, 7]} v 0) v1 -- 0x7ffb94106090 con 0x7ffb940f73a0 2024-02-13T00:51:20.889 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:19.520380+0000 osd.1 (osd.1) 277 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:51:20.890 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:19.521316+0000 osd.1 (osd.1) 278 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:51:20.890 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:19.558493+0000 mon.a (mon.0) 3254 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:20.890 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:19.565909+0000 mon.a (mon.0) 3255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:51:20.890 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:19.566013+0000 mon.a (mon.0) 3256 : cluster [DBG] osdmap e654: 8 total, 8 up, 4 in 2024-02-13T00:51:20.890 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:19.566797+0000 mon.a (mon.0) 3257 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:20.890 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:20.527684+0000 mon.a (mon.0) 3258 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:20.890 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:20.527814+0000 mon.a (mon.0) 3259 : cluster [DBG] osdmap e655: 8 total, 8 up, 4 in 2024-02-13T00:51:20.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:20.925 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:21.173 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:21.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:21.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:21.305 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:21.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:21.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:21.462 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:21.496 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:21.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.546+0000 7ffb91ffb700 1 -- 172.21.15.162:0/125392486 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "6.23", "id": [3, 6, 7]}]=0 set 6.23 pg_upmap mapping to [3,6,7] v656) v1 ==== 129+0+0 (secure 0 0 0) 0x7ffb94106090 con 0x7ffb940f73a0 2024-02-13T00:51:21.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 10 librados: watch_flush enter 2024-02-13T00:51:21.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 10 librados: watch_flush exit 2024-02-13T00:51:21.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 20 client.37982.objecter shutdown clearing up homeless session... 2024-02-13T00:51:21.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 10 client.37982.objecter successfully canceled tick 2024-02-13T00:51:21.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 -- 172.21.15.162:0/125392486 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffb84062c90 msgr2=0x7ffb84065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:21.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 --2- 172.21.15.162:0/125392486 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffb84062c90 0x7ffb84065150 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 rx=0x7ffb941089b0 tx=0x7ffb8000b040).stop 2024-02-13T00:51:21.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 -- 172.21.15.162:0/125392486 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb940f73a0 msgr2=0x7ffb9410d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:21.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 --2- 172.21.15.162:0/125392486 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb940f73a0 0x7ffb9410d070 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 rx=0x7ffb88004650 tx=0x7ffb88007f90).stop 2024-02-13T00:51:21.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 -- 172.21.15.162:0/125392486 shutdown_connections 2024-02-13T00:51:21.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 --2- 172.21.15.162:0/125392486 >> v2:172.21.15.162:6800/2866534053 conn(0x7ffb84062c90 0x7ffb84065150 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:21.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 --2- 172.21.15.162:0/125392486 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb940fd490 0x7ffb94107210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:21.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 --2- 172.21.15.162:0/125392486 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb940f73a0 0x7ffb9410d070 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:21.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 --2- 172.21.15.162:0/125392486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb940fcb30 0x7ffb9410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:21.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 -- 172.21.15.162:0/125392486 >> 172.21.15.162:0/125392486 conn(0x7ffb9406eb90 msgr2=0x7ffb940fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:21.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 -- 172.21.15.162:0/125392486 shutdown_connections 2024-02-13T00:51:21.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 -- 172.21.15.162:0/125392486 wait complete. 2024-02-13T00:51:21.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.549+0000 7ffb9abe6700 1 librados: shutdown 2024-02-13T00:51:21.554 INFO:teuthology.orchestra.run.smithi162.stderr:set 6.23 pg_upmap mapping to [3,6,7] 2024-02-13T00:51:21.620 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:21.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:21.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:21.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:21.750 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:20.504938+0000 mgr.y (mgr.14544) 556 : cluster [DBG] pgmap v987: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:21.750 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:20.813604+0000 mon.c (mon.2) 480 : audit [INF] from='client.? 172.21.15.162:0/125392486' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.23", "id": [3, 6, 7]}]: dispatch 2024-02-13T00:51:21.750 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:20.813906+0000 mon.a (mon.0) 3260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.23", "id": [3, 6, 7]}]: dispatch 2024-02-13T00:51:21.750 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:21.546664+0000 mon.a (mon.0) 3261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.23", "id": [3, 6, 7]}]': finished 2024-02-13T00:51:21.750 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:21.546743+0000 mon.a (mon.0) 3262 : cluster [DBG] osdmap e656: 8 total, 8 up, 4 in 2024-02-13T00:51:21.750 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:21.560811+0000 mon.a (mon.0) 3263 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:21.751 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:20.504938+0000 mgr.y (mgr.14544) 556 : cluster [DBG] pgmap v987: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:21.751 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:20.813604+0000 mon.c (mon.2) 480 : audit [INF] from='client.? 172.21.15.162:0/125392486' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.23", "id": [3, 6, 7]}]: dispatch 2024-02-13T00:51:21.751 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:20.813906+0000 mon.a (mon.0) 3260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.23", "id": [3, 6, 7]}]: dispatch 2024-02-13T00:51:21.751 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:21.546664+0000 mon.a (mon.0) 3261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.23", "id": [3, 6, 7]}]': finished 2024-02-13T00:51:21.751 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:21.546743+0000 mon.a (mon.0) 3262 : cluster [DBG] osdmap e656: 8 total, 8 up, 4 in 2024-02-13T00:51:21.751 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:21.560811+0000 mon.a (mon.0) 3263 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:21.752 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:21.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:20.504938+0000 mgr.y (mgr.14544) 556 : cluster [DBG] pgmap v987: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:21.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:20.813604+0000 mon.c (mon.2) 480 : audit [INF] from='client.? 172.21.15.162:0/125392486' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.23", "id": [3, 6, 7]}]: dispatch 2024-02-13T00:51:21.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:20.813906+0000 mon.a (mon.0) 3260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.23", "id": [3, 6, 7]}]: dispatch 2024-02-13T00:51:21.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:21.546664+0000 mon.a (mon.0) 3261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.23", "id": [3, 6, 7]}]': finished 2024-02-13T00:51:21.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:21.546743+0000 mon.a (mon.0) 3262 : cluster [DBG] osdmap e656: 8 total, 8 up, 4 in 2024-02-13T00:51:21.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:21.560811+0000 mon.a (mon.0) 3263 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:21.759 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:51:21.759 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:51:21.829 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:21.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:21.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:21.933 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:21.961 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:22.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-13T00:51:22.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.735+0000 7f53b4eae700 1 -- 172.21.15.162:0/4207380260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b00fcb30 msgr2=0x7f53b00fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.735+0000 7f53b4eae700 1 --2- 172.21.15.162:0/4207380260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b00fcb30 0x7f53b00fcf10 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7f53a00047b0 tx=0x7f53a0009cb0).stop 2024-02-13T00:51:22.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.735+0000 7f53b4eae700 1 -- 172.21.15.162:0/4207380260 shutdown_connections 2024-02-13T00:51:22.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.735+0000 7f53b4eae700 1 --2- 172.21.15.162:0/4207380260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b00f73a0 0x7f53b01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.735+0000 7f53b4eae700 1 --2- 172.21.15.162:0/4207380260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b00fcb30 0x7f53b00fcf10 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.735+0000 7f53b4eae700 1 --2- 172.21.15.162:0/4207380260 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53b00fd490 0x7f53b00f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.735+0000 7f53b4eae700 1 -- 172.21.15.162:0/4207380260 >> 172.21.15.162:0/4207380260 conn(0x7f53b006eb90 msgr2=0x7f53b00f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:22.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.735+0000 7f53b4eae700 1 -- 172.21.15.162:0/4207380260 shutdown_connections 2024-02-13T00:51:22.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.735+0000 7f53b4eae700 1 -- 172.21.15.162:0/4207380260 wait complete. 2024-02-13T00:51:22.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.736+0000 7f53b4eae700 1 librados: starting msgr at 2024-02-13T00:51:22.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.736+0000 7f53b4eae700 1 librados: starting objecter 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.736+0000 7f53b4eae700 1 Processor -- start 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.736+0000 7f53b4eae700 1 -- start start 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.736+0000 7f53b4eae700 1 librados: setting wanted keys 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.736+0000 7f53b4eae700 1 librados: calling monclient init 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53b4eae700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b00f73a0 0x7f53b010d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53b4eae700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b00fcb30 0x7f53b010d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53b4eae700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53b00fd490 0x7f53b0107250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53b4eae700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b0107740 con 0x7f53b00f73a0 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53b4eae700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01078b0 con 0x7f53b00fcb30 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53b4eae700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b0107a70 con 0x7f53b00fd490 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53aed9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53b00fd490 0x7f53b0107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53a5d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b00fcb30 0x7f53b010d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:22.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53a5d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b00fcb30 0x7f53b010d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:57792/0 (socket says 172.21.15.162:57792) 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53aed9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53b00fd490 0x7f53b0107250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:52362/0 (socket says 172.21.15.162:52362) 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53a5d9b700 1 -- 172.21.15.162:0/3575743705 learned_addr learned my addr 172.21.15.162:0/3575743705 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53ae59c700 1 --2- 172.21.15.162:0/3575743705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b00f73a0 0x7f53b010d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53a5d9b700 1 -- 172.21.15.162:0/3575743705 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53b00fd490 msgr2=0x7f53b0107250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53a5d9b700 1 --2- 172.21.15.162:0/3575743705 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53b00fd490 0x7f53b0107250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53a5d9b700 1 -- 172.21.15.162:0/3575743705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b00f73a0 msgr2=0x7f53b010d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53a5d9b700 1 --2- 172.21.15.162:0/3575743705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b00f73a0 0x7f53b010d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53a5d9b700 1 -- 172.21.15.162:0/3575743705 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a0005040 con 0x7f53b00fcb30 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53ae59c700 1 --2- 172.21.15.162:0/3575743705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b00f73a0 0x7f53b010d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53a5d9b700 1 --2- 172.21.15.162:0/3575743705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b00fcb30 0x7f53b010d5a0 secure :-1 s=READY pgs=4859 cs=0 l=1 rev1=1 rx=0x7f539800e990 tx=0x7f539800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53a7fff700 10 client.?.objecter ms_handle_connect 0x7f53b00fcb30 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.737+0000 7f53a7fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:22.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.738+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53980046e0 con 0x7f53b00fcb30 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.738+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b0107cf0 con 0x7f53b00fcb30 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.738+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f539800f040 con 0x7f53b00fcb30 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.738+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53980146b0 con 0x7f53b00fcb30 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.738+0000 7f53b4eae700 10 client.44899.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.738+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b0113ee0 con 0x7f53b00fcb30 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.738+0000 7f53b4eae700 1 librados: init done 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.739+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f53b0106090 con 0x7f53b00fcb30 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.739+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f53980090d0 con 0x7f53b00fcb30 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.739+0000 7f53a7fff700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6800/2866534053 conn(0x7f539c062c90 0x7f539c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.739+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(656..656 src has 449..656) v4 ==== 8597+0+0 (secure 0 0 0) 0x7f539807cfe0 con 0x7f53b00fcb30 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.739+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 osd_map(656..656 src has 449..656) v4 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.739+0000 7f53a7fff700 3 client.44899.objecter handle_osd_map got epochs [656,656] > 0 2024-02-13T00:51:22.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.739+0000 7f53a7fff700 3 client.44899.objecter handle_osd_map decoding full epoch 656 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.739+0000 7f53ae59c700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6800/2866534053 conn(0x7f539c062c90 0x7f539c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53a7fff700 20 client.44899.objecter dump_active .. 0 homeless 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v656) v1 ==== 54+0+16 (secure 0 0 0) 0x7f539807f610 con 0x7f53b00fcb30 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53a7fff700 10 client.44899.objecter ms_handle_connect 0x7f539c062c90 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53ae59c700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6800/2866534053 conn(0x7f539c062c90 0x7f539c065150 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7f53a0005010 tx=0x7f53a0007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53b4eae700 10 client.44899.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 1 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6801/2103743338 conn(0x7f53b00fde00 0x7f53b00fe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0108730 osd=0 3 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0108730 osd=0 3 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0108730 osd=0 3 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 1 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b0108730 osd=0 3 2024-02-13T00:51:22.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 0 1 2024-02-13T00:51:22.484 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:51:22.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0108730 osd=0 4 2024-02-13T00:51:22.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53b4eae700 10 client.44899.objecter _send_command 1 2024-02-13T00:51:22.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53b00fe6d0 con 0x7f53b00fde00 2024-02-13T00:51:22.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53aed9d700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6801/2103743338 conn(0x7f53b00fde00 0x7f53b00fe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:22.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53a7fff700 10 client.44899.objecter ms_handle_connect 0x7f53b00fde00 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.740+0000 7f53aed9d700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6801/2103743338 conn(0x7f53b00fde00 0x7f53b00fe1e0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.741+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f53b00fe6d0 con 0x7f53b00fde00 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.741+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.741+0000 7f53a7fff700 10 client.44899.objecter _finish_command 1 = system:0 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.741+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b0108730 osd=0 4 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.741+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 0 1 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.756+0000 7f53b4eae700 10 client.44899.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.756+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 2 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.756+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0108730 osd=0 3 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.756+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0108730 osd=0 3 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.756+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0108730 osd=0 3 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.756+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 2 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.756+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b0108730 osd=0 3 2024-02-13T00:51:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.756+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 0 2 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.756+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0108730 osd=0 4 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.756+0000 7f53b4eae700 10 client.44899.objecter _send_command 2 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.756+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f53b00fe890 con 0x7f53b00fde00 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f53b00fe890 con 0x7f53b00fde00 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53a7fff700 10 client.44899.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b0108730 osd=0 4 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 0 2 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53b4eae700 10 client.44899.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 3 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6805/150599780 conn(0x7f53b0061b40 0x7f53b0116080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0070c70 osd=1 3 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0070c70 osd=1 3 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0070c70 osd=1 3 2024-02-13T00:51:22.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 3 2024-02-13T00:51:22.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b0070c70 osd=1 3 2024-02-13T00:51:22.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 1 3 2024-02-13T00:51:22.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0070c70 osd=1 4 2024-02-13T00:51:22.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53b4eae700 10 client.44899.objecter _send_command 3 2024-02-13T00:51:22.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.758+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53b0070de0 con 0x7f53b0061b40 2024-02-13T00:51:22.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.759+0000 7f53ae59c700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6805/150599780 conn(0x7f53b0061b40 0x7f53b0116080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:22.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.759+0000 7f53a7fff700 10 client.44899.objecter ms_handle_connect 0x7f53b0061b40 2024-02-13T00:51:22.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.759+0000 7f53ae59c700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6805/150599780 conn(0x7f53b0061b40 0x7f53b0116080 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:22.487 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.759+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f53b0070de0 con 0x7f53b0061b40 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.759+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.759+0000 7f53a7fff700 10 client.44899.objecter _finish_command 3 = system:0 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.759+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b0070c70 osd=1 4 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.759+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 1 3 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.779+0000 7f53b4eae700 10 client.44899.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.779+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 4 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.779+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0070c70 osd=1 3 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.779+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0070c70 osd=1 3 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.779+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0070c70 osd=1 3 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.779+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 4 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.779+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b0070c70 osd=1 3 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.779+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 1 4 2024-02-13T00:51:22.488 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.779+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0070c70 osd=1 4 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.779+0000 7f53b4eae700 10 client.44899.objecter _send_command 4 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.779+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f53b00610a0 con 0x7f53b0061b40 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f53b00610a0 con 0x7f53b0061b40 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53a7fff700 10 client.44899.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b0070c70 osd=1 4 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 1 4 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53b4eae700 10 client.44899.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 5 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6809/3689719376 conn(0x7f53b0028590 0x7f53b0118580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b00614b0 osd=2 3 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b00614b0 osd=2 3 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b00614b0 osd=2 3 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 5 2024-02-13T00:51:22.489 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b00614b0 osd=2 3 2024-02-13T00:51:22.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 2 5 2024-02-13T00:51:22.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b00614b0 osd=2 4 2024-02-13T00:51:22.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53b4eae700 10 client.44899.objecter _send_command 5 2024-02-13T00:51:22.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53b005a000 con 0x7f53b0028590 2024-02-13T00:51:22.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53a5d9b700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6809/3689719376 conn(0x7f53b0028590 0x7f53b0118580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:22.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53a7fff700 10 client.44899.objecter ms_handle_connect 0x7f53b0028590 2024-02-13T00:51:22.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.781+0000 7f53a5d9b700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6809/3689719376 conn(0x7f53b0028590 0x7f53b0118580 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:22.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.782+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f53b005a000 con 0x7f53b0028590 2024-02-13T00:51:22.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.782+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:51:22.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.782+0000 7f53a7fff700 10 client.44899.objecter _finish_command 5 = system:0 2024-02-13T00:51:22.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.782+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b00614b0 osd=2 4 2024-02-13T00:51:22.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.782+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 2 5 2024-02-13T00:51:22.491 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:51:22.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.796+0000 7f53b4eae700 10 client.44899.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:22.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.796+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 6 2024-02-13T00:51:22.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.796+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b00614b0 osd=2 3 2024-02-13T00:51:22.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.796+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b00614b0 osd=2 3 2024-02-13T00:51:22.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.796+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b00614b0 osd=2 3 2024-02-13T00:51:22.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.796+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 6 2024-02-13T00:51:22.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.796+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b00614b0 osd=2 3 2024-02-13T00:51:22.491 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.796+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 2 6 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.796+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b00614b0 osd=2 4 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.796+0000 7f53b4eae700 10 client.44899.objecter _send_command 6 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.796+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f53b0119240 con 0x7f53b0028590 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f53b0119240 con 0x7f53b0028590 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53a7fff700 10 client.44899.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b00614b0 osd=2 4 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 2 6 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53b4eae700 10 client.44899.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 7 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6813/604971011 conn(0x7f53b011fd40 0x7f53b0122130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0059640 osd=3 3 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0059640 osd=3 3 2024-02-13T00:51:22.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0059640 osd=3 3 2024-02-13T00:51:22.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 7 2024-02-13T00:51:22.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b0059640 osd=3 3 2024-02-13T00:51:22.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 3 7 2024-02-13T00:51:22.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0059640 osd=3 4 2024-02-13T00:51:22.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53b4eae700 10 client.44899.objecter _send_command 7 2024-02-13T00:51:22.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53b0124d80 con 0x7f53b011fd40 2024-02-13T00:51:22.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53aed9d700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6813/604971011 conn(0x7f53b011fd40 0x7f53b0122130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:22.493 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:51:22.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53a7fff700 10 client.44899.objecter ms_handle_connect 0x7f53b011fd40 2024-02-13T00:51:22.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.798+0000 7f53aed9d700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6813/604971011 conn(0x7f53b011fd40 0x7f53b0122130 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:22.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.799+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f53b0124d80 con 0x7f53b011fd40 2024-02-13T00:51:22.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.799+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:51:22.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.799+0000 7f53a7fff700 10 client.44899.objecter _finish_command 7 = system:0 2024-02-13T00:51:22.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.799+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b0059640 osd=3 4 2024-02-13T00:51:22.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.799+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 3 7 2024-02-13T00:51:22.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.813+0000 7f53b4eae700 10 client.44899.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:22.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.813+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 8 2024-02-13T00:51:22.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.813+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0059640 osd=3 3 2024-02-13T00:51:22.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.813+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0059640 osd=3 3 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.813+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0059640 osd=3 3 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.813+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 8 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.813+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b0059640 osd=3 3 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.813+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 3 8 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.813+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0059640 osd=3 4 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.813+0000 7f53b4eae700 10 client.44899.objecter _send_command 8 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.813+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f53b0118cb0 con 0x7f53b011fd40 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f53b0118cb0 con 0x7f53b011fd40 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53a7fff700 10 client.44899.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b0059640 osd=3 4 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 3 8 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53b4eae700 10 client.44899.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:22.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 9 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6800/3344263388 conn(0x7f53b00280f0 0x7f53b01237a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0118e80 osd=4 3 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0118e80 osd=4 3 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0118e80 osd=4 3 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 9 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b0118e80 osd=4 3 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 4 9 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0118e80 osd=4 4 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53b4eae700 10 client.44899.objecter _send_command 9 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53b0118ff0 con 0x7f53b00280f0 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53ae59c700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6800/3344263388 conn(0x7f53b00280f0 0x7f53b01237a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53a7fff700 10 client.44899.objecter ms_handle_connect 0x7f53b00280f0 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.815+0000 7f53ae59c700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6800/3344263388 conn(0x7f53b00280f0 0x7f53b01237a0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.816+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f53b0118ff0 con 0x7f53b00280f0 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.816+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.816+0000 7f53a7fff700 10 client.44899.objecter _finish_command 9 = system:0 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.816+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b0118e80 osd=4 4 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.816+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 4 9 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.832+0000 7f53b4eae700 10 client.44899.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.832+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 10 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.832+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0118e80 osd=4 3 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.832+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0118e80 osd=4 3 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.832+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0118e80 osd=4 3 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.832+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 10 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.832+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b0118e80 osd=4 3 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.832+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 4 10 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.832+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0118e80 osd=4 4 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.832+0000 7f53b4eae700 10 client.44899.objecter _send_command 10 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.832+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f53b0118ff0 con 0x7f53b00280f0 2024-02-13T00:51:22.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f53b0118ff0 con 0x7f53b00280f0 2024-02-13T00:51:22.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:22.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53a7fff700 10 client.44899.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b0118e80 osd=4 4 2024-02-13T00:51:22.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 4 10 2024-02-13T00:51:22.498 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:51:22.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53b4eae700 10 client.44899.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:22.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 11 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6804/2277917372 conn(0x7f53b01241a0 0x7f53b01245e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0118ff0 osd=5 3 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0118ff0 osd=5 3 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0118ff0 osd=5 3 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 11 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b0118ff0 osd=5 3 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 5 11 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0118ff0 osd=5 4 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53b4eae700 10 client.44899.objecter _send_command 11 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.834+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53b012b670 con 0x7f53b01241a0 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.835+0000 7f53a5d9b700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6804/2277917372 conn(0x7f53b01241a0 0x7f53b01245e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.835+0000 7f53a7fff700 10 client.44899.objecter ms_handle_connect 0x7f53b01241a0 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.835+0000 7f53a5d9b700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6804/2277917372 conn(0x7f53b01241a0 0x7f53b01245e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.836+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f53b012b670 con 0x7f53b01241a0 2024-02-13T00:51:22.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.836+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:51:22.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.836+0000 7f53a7fff700 10 client.44899.objecter _finish_command 11 = system:0 2024-02-13T00:51:22.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.836+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b0118ff0 osd=5 4 2024-02-13T00:51:22.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.836+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 5 11 2024-02-13T00:51:22.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.850+0000 7f53b4eae700 10 client.44899.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:22.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.850+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 12 2024-02-13T00:51:22.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.850+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0118ff0 osd=5 3 2024-02-13T00:51:22.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.850+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0118ff0 osd=5 3 2024-02-13T00:51:22.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.850+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0118ff0 osd=5 3 2024-02-13T00:51:22.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.850+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 12 2024-02-13T00:51:22.503 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.850+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b0118ff0 osd=5 3 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.850+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 5 12 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.850+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0118ff0 osd=5 4 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.850+0000 7f53b4eae700 10 client.44899.objecter _send_command 12 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.850+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f53b0128f10 con 0x7f53b01241a0 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.851+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f53b0128f10 con 0x7f53b01241a0 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.851+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.851+0000 7f53a7fff700 10 client.44899.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.851+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b0118ff0 osd=5 4 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.851+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 5 12 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53b4eae700 10 client.44899.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 13 2024-02-13T00:51:22.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6808/1506640341 conn(0x7f53b0129590 0x7f53b012b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0128f10 osd=6 3 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0128f10 osd=6 3 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0128f10 osd=6 3 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 13 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b0128f10 osd=6 3 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 6 13 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0128f10 osd=6 4 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53b4eae700 10 client.44899.objecter _send_command 13 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53b012e750 con 0x7f53b0129590 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53aed9d700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6808/1506640341 conn(0x7f53b0129590 0x7f53b012b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53a7fff700 10 client.44899.objecter ms_handle_connect 0x7f53b0129590 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.852+0000 7f53aed9d700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6808/1506640341 conn(0x7f53b0129590 0x7f53b012b980 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.853+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f53b012e750 con 0x7f53b0129590 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.853+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:51:22.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.853+0000 7f53a7fff700 10 client.44899.objecter _finish_command 13 = system:0 2024-02-13T00:51:22.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.853+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b0128f10 osd=6 4 2024-02-13T00:51:22.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.853+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 6 13 2024-02-13T00:51:22.506 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:51:22.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.867+0000 7f53b4eae700 10 client.44899.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:22.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.867+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 14 2024-02-13T00:51:22.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.867+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0128f10 osd=6 3 2024-02-13T00:51:22.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.867+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0128f10 osd=6 3 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.867+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b0128f10 osd=6 3 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.867+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 14 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.867+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b0128f10 osd=6 3 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.867+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 6 14 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.867+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0128f10 osd=6 4 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.867+0000 7f53b4eae700 10 client.44899.objecter _send_command 14 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.867+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f53b012e750 con 0x7f53b0129590 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f53b012e750 con 0x7f53b0129590 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53a7fff700 10 client.44899.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b0128f10 osd=6 4 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 6 14 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53b4eae700 10 client.44899.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:22.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 15 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6812/1964192772 conn(0x7f53b012db70 0x7f53b012dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b012e750 osd=7 3 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b012e750 osd=7 3 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b012e750 osd=7 3 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 15 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b012e750 osd=7 3 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 7 15 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b012e750 osd=7 4 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53b4eae700 10 client.44899.objecter _send_command 15 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.869+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53b0131b00 con 0x7f53b012db70 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.870+0000 7f53ae59c700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6812/1964192772 conn(0x7f53b012db70 0x7f53b012dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.870+0000 7f53a7fff700 10 client.44899.objecter ms_handle_connect 0x7f53b012db70 2024-02-13T00:51:22.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.870+0000 7f53ae59c700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6812/1964192772 conn(0x7f53b012db70 0x7f53b012dfd0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:22.509 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:51:22.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.871+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f53b0131b00 con 0x7f53b012db70 2024-02-13T00:51:22.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.871+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:51:22.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.871+0000 7f53a7fff700 10 client.44899.objecter _finish_command 15 = system:0 2024-02-13T00:51:22.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.871+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b012e750 osd=7 4 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.871+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 7 15 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.884+0000 7f53b4eae700 10 client.44899.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.884+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign -1 16 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.884+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b012e750 osd=7 3 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.884+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b012e750 osd=7 3 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.884+0000 7f53b4eae700 20 client.44899.objecter _get_session s=0x7f53b012e750 osd=7 3 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.884+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_remove -1 16 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.884+0000 7f53b4eae700 20 client.44899.objecter get_session s=0x7f53b012e750 osd=7 3 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.884+0000 7f53b4eae700 15 client.44899.objecter _session_command_op_assign 7 16 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.884+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b012e750 osd=7 4 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.884+0000 7f53b4eae700 10 client.44899.objecter _send_command 16 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.884+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f53b0130980 con 0x7f53b012db70 2024-02-13T00:51:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53a7fff700 1 -- 172.21.15.162:0/3575743705 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f53b0130980 con 0x7f53b012db70 2024-02-13T00:51:22.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53a7fff700 10 client.44899.objecter ms_dispatch 0x7f53b0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:22.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53a7fff700 10 client.44899.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53a7fff700 20 client.44899.objecter put_session s=0x7f53b012e750 osd=7 4 2024-02-13T00:51:22.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53a7fff700 15 client.44899.objecter _session_command_op_remove 7 16 2024-02-13T00:51:22.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 10 librados: watch_flush enter 2024-02-13T00:51:22.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 10 librados: watch_flush exit 2024-02-13T00:51:22.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 10 client.44899.objecter close_session for osd.0 2024-02-13T00:51:22.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6801/2103743338 conn(0x7f53b00fde00 msgr2=0x7f53b00fe1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6801/2103743338 conn(0x7f53b00fde00 0x7f53b00fe1e0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0108730 osd=0 1 2024-02-13T00:51:22.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 10 client.44899.objecter close_session for osd.1 2024-02-13T00:51:22.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6805/150599780 conn(0x7f53b0061b40 msgr2=0x7f53b0116080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6805/150599780 conn(0x7f53b0061b40 0x7f53b0116080 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0070c70 osd=1 1 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 10 client.44899.objecter close_session for osd.2 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6809/3689719376 conn(0x7f53b0028590 msgr2=0x7f53b0118580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6809/3689719376 conn(0x7f53b0028590 0x7f53b0118580 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b00614b0 osd=2 1 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 10 client.44899.objecter close_session for osd.3 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6813/604971011 conn(0x7f53b011fd40 msgr2=0x7f53b0122130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6813/604971011 conn(0x7f53b011fd40 0x7f53b0122130 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53aed9d700 1 -- 172.21.15.162:0/3575743705 reap_dead start 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0059640 osd=3 1 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 10 client.44899.objecter close_session for osd.4 2024-02-13T00:51:22.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6800/3344263388 conn(0x7f53b00280f0 msgr2=0x7f53b01237a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6800/3344263388 conn(0x7f53b00280f0 0x7f53b01237a0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0118e80 osd=4 1 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 10 client.44899.objecter close_session for osd.5 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6804/2277917372 conn(0x7f53b01241a0 msgr2=0x7f53b01245e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6804/2277917372 conn(0x7f53b01241a0 0x7f53b01245e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0118ff0 osd=5 1 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 10 client.44899.objecter close_session for osd.6 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6808/1506640341 conn(0x7f53b0129590 msgr2=0x7f53b012b980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6808/1506640341 conn(0x7f53b0129590 0x7f53b012b980 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b0128f10 osd=6 1 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 10 client.44899.objecter close_session for osd.7 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6812/1964192772 conn(0x7f53b012db70 msgr2=0x7f53b012dfd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.181:6812/1964192772 conn(0x7f53b012db70 0x7f53b012dfd0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 20 client.44899.objecter put_session s=0x7f53b012e750 osd=7 1 2024-02-13T00:51:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 20 client.44899.objecter shutdown clearing up homeless session... 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53aed9d700 1 -- 172.21.15.162:0/3575743705 reap_dead start 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 10 client.44899.objecter successfully canceled tick 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6800/2866534053 conn(0x7f539c062c90 msgr2=0x7f539c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6800/2866534053 conn(0x7f539c062c90 0x7f539c065150 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7f53a0005010 tx=0x7f53a0007f90).stop 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b00fcb30 msgr2=0x7f53b010d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b00fcb30 0x7f53b010d5a0 secure :-1 s=READY pgs=4859 cs=0 l=1 rev1=1 rx=0x7f539800e990 tx=0x7f539800b040).stop 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 shutdown_connections 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b00fcb30 0x7f53b010d5a0 unknown :-1 s=CLOSED pgs=4859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 --2- 172.21.15.162:0/3575743705 >> v2:172.21.15.162:6800/2866534053 conn(0x7f539c062c90 0x7f539c065150 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 >> 172.21.15.162:0/3575743705 conn(0x7f53b006eb90 msgr2=0x7f53b00700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 shutdown_connections 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.887+0000 7f53b4eae700 1 -- 172.21.15.162:0/3575743705 wait complete. 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:21.888+0000 7f53b4eae700 1 librados: shutdown 2024-02-13T00:51:22.514 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.515 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.515 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.515 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.515 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.515 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.515 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.515 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:51:22.705 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:22.711 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:22.713 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:21.740570+0000 mon.b (mon.1) 679 : audit [DBG] from='client.? 172.21.15.162:0/3575743705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:22.713 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:22.544784+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:22.713 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:22.544831+0000 mon.a (mon.0) 3265 : cluster [DBG] osdmap e657: 8 total, 8 up, 4 in 2024-02-13T00:51:22.714 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:22.545707+0000 mon.a (mon.0) 3266 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-114"}]: dispatch 2024-02-13T00:51:22.714 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:21.740570+0000 mon.b (mon.1) 679 : audit [DBG] from='client.? 172.21.15.162:0/3575743705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:22.714 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:22.544784+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:22.714 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:22.544831+0000 mon.a (mon.0) 3265 : cluster [DBG] osdmap e657: 8 total, 8 up, 4 in 2024-02-13T00:51:22.714 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:22.545707+0000 mon.a (mon.0) 3266 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-114"}]: dispatch 2024-02-13T00:51:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:22.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:21.740570+0000 mon.b (mon.1) 679 : audit [DBG] from='client.? 172.21.15.162:0/3575743705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:22.544784+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:22.544831+0000 mon.a (mon.0) 3265 : cluster [DBG] osdmap e657: 8 total, 8 up, 4 in 2024-02-13T00:51:22.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:22.545707+0000 mon.a (mon.0) 3266 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-114"}]: dispatch 2024-02-13T00:51:22.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:22.843 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:23.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:22.505544+0000 mgr.y (mgr.14544) 557 : cluster [DBG] pgmap v990: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:23.547403+0000 mon.a (mon.0) 3267 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-114"}]': finished 2024-02-13T00:51:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:23.547474+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e658: 8 total, 8 up, 4 in 2024-02-13T00:51:23.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:23.548822+0000 mon.a (mon.0) 3269 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-114", "mode": "writeback"}]: dispatch 2024-02-13T00:51:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:23.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:23.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:22.505544+0000 mgr.y (mgr.14544) 557 : cluster [DBG] pgmap v990: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:23.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:23.547403+0000 mon.a (mon.0) 3267 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-114"}]': finished 2024-02-13T00:51:23.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:23.547474+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e658: 8 total, 8 up, 4 in 2024-02-13T00:51:23.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:23.548822+0000 mon.a (mon.0) 3269 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-114", "mode": "writeback"}]: dispatch 2024-02-13T00:51:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:22.505544+0000 mgr.y (mgr.14544) 557 : cluster [DBG] pgmap v990: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:23.547403+0000 mon.a (mon.0) 3267 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-114"}]': finished 2024-02-13T00:51:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:23.547474+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e658: 8 total, 8 up, 4 in 2024-02-13T00:51:23.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:23.548822+0000 mon.a (mon.0) 3269 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-114", "mode": "writeback"}]: dispatch 2024-02-13T00:51:23.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:23.931 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:24.376 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:51:24.478 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:24.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:24.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:24.613 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:24.651 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:24.757 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:51:24.758 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:51:24.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:51:24.758 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:51:24.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.758 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.29851.0:2 72.6 72:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-02-13T00:51:24.758 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:24.983112+0000", 2024-02-13T00:51:24.758 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 599.77096823199997, 2024-02-13T00:51:24.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.056797175999999998, 2024-02-13T00:51:24.758 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:24.758 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.29851", 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3524561227", 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983112+0000", 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:24.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983112+0000", 2024-02-13T00:51:24.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:51:24.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:24.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983111+0000", 2024-02-13T00:51:24.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.57e-06 2024-02-13T00:51:24.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:24.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983116+0000", 2024-02-13T00:51:24.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8830000000000006e-06 2024-02-13T00:51:24.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:24.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983124+0000", 2024-02-13T00:51:24.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5900000000000002e-06 2024-02-13T00:51:24.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:24.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983132+0000", 2024-02-13T00:51:24.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8680000000000001e-05 2024-02-13T00:51:24.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:24.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983180+0000", 2024-02-13T00:51:24.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1040000000000001e-06 2024-02-13T00:51:24.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:51:24.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:24.983187+0000", 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.037057128000000002 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.020244+0000", 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108541 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.020353+0000", 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000146265 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-02-13T00:51:24.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.020499+0000", 2024-02-13T00:51:24.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001090929 2024-02-13T00:51:24.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:51:24.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.021590+0000", 2024-02-13T00:51:24.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000159708 2024-02-13T00:51:24.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:51:24.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.021750+0000", 2024-02-13T00:51:24.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.018075356000000001 2024-02-13T00:51:24.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:51:24.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.039825+0000", 2024-02-13T00:51:24.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2278000000000003e-05 2024-02-13T00:51:24.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:51:24.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.039878+0000", 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1665000000000003e-05 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:25.039909+0000", 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.056797175999999998 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.19047.0:1 78.4s0 78:2fee9e45:test-rados-api-smithi162-84073-10::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-02-13T00:51:24.766 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:28.235955+0000", 2024-02-13T00:51:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 596.51812510299999, 2024-02-13T00:51:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0634874270000001, 2024-02-13T00:51:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:51:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:51:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.19047", 2024-02-13T00:51:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3520430640", 2024-02-13T00:51:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-02-13T00:51:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.235955+0000", 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.235955+0000", 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999967 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.235952+0000", 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3636e-05 2024-02-13T00:51:24.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.235965+0000", 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8223000000000002e-05 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.235984+0000", 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2216e-05 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.235996+0000", 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012535500000000001 2024-02-13T00:51:24.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.236121+0000", 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.976e-06 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:28.236127+0000", 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0553561279999999 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:29.291483+0000", 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000100157 2024-02-13T00:51:24.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:29.291583+0000", 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027292499999999998 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:29.291856+0000", 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00043762800000000002 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:29.292294+0000", 2024-02-13T00:51:24.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071099309999999999 2024-02-13T00:51:24.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:51:24.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:29.299404+0000", 2024-02-13T00:51:24.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8661999999999997e-05 2024-02-13T00:51:24.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:24.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:29.299443+0000", 2024-02-13T00:51:24.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0634874270000001 2024-02-13T00:51:24.772 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25162.0:110 73.1f 73:fec11214:test-rados-api-smithi162-84057-10::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e135)", 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:30.310984+0000", 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 594.44309616199996, 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.050975678000000003, 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25162", 2024-02-13T00:51:24.773 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:51:24.774 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 110 2024-02-13T00:51:24.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.774 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:24.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:24.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.310984+0000", 2024-02-13T00:51:24.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:24.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:24.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.310984+0000", 2024-02-13T00:51:24.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:51:24.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:24.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.310982+0000", 2024-02-13T00:51:24.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4608e-05 2024-02-13T00:51:24.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:24.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.310997+0000", 2024-02-13T00:51:24.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3023999999999999e-05 2024-02-13T00:51:24.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:24.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.311010+0000", 2024-02-13T00:51:24.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8660000000000002e-06 2024-02-13T00:51:24.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:24.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.311016+0000", 2024-02-13T00:51:24.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7796999999999997e-05 2024-02-13T00:51:24.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:24.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.311054+0000", 2024-02-13T00:51:24.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020388700000000001 2024-02-13T00:51:24.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-02-13T00:51:24.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.311258+0000", 2024-02-13T00:51:24.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.045311125000000001 2024-02-13T00:51:24.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.356569+0000", 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9762000000000001e-05 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.356639+0000", 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030073029999999998 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.359646+0000", 2024-02-13T00:51:24.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7300999999999999e-05 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.359703+0000", 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022566830000000002 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:30.361960+0000", 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.050975678000000003 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.43639.0:2 102.d 102:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:41:54.329646+0000", 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 570.42443470399996, 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.034329884999999997, 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.43639", 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/891139537", 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329646+0000", 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329646+0000", 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:24.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329644+0000", 2024-02-13T00:51:24.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6475000000000001e-05 2024-02-13T00:51:24.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:24.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329660+0000", 2024-02-13T00:51:24.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0299000000000001e-05 2024-02-13T00:51:24.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:24.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329671+0000", 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.25e-06 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329678+0000", 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3374000000000001e-05 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329731+0000", 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000122748 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.329854+0000", 2024-02-13T00:51:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000161397 2024-02-13T00:51:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,6", 2024-02-13T00:51:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.330015+0000", 2024-02-13T00:51:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039438199999999998 2024-02-13T00:51:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:51:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.330410+0000", 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014673799999999999 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.330556+0000", 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.033313427999999999 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.363870+0000", 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2893000000000003e-05 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:51:24.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.363923+0000", 2024-02-13T00:51:24.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2667999999999998e-05 2024-02-13T00:51:24.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:24.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:41:54.363975+0000", 2024-02-13T00:51:24.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.034329884999999997 2024-02-13T00:51:24.786 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.786 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:24.786 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.787 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPushReply(20.2 172/167 [PushReplyOp(20:40ede249:test-rados-api-smithi162-84049-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head)])", 2024-02-13T00:51:24.787 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:42:07.547345+0000", 2024-02-13T00:51:24.787 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 557.20673523200003, 2024-02-13T00:51:24.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.082099599999999995, 2024-02-13T00:51:24.788 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:24.788 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:51:24.788 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:24.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:24.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.547345+0000", 2024-02-13T00:51:24.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:24.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.547345+0000", 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.547344+0000", 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4939999999999992e-06 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.547354+0000", 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9225999999999999e-05 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:24.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.547373+0000", 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0322999999999999e-05 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.547383+0000", 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.081862927000000002 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.629246+0000", 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019853000000000001 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:42:07.629445+0000", 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.082099599999999995 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.25162.0:4733 191.6 e335/323 191:602f83fe:::foo:head v 335'11, mlcod=333'9)", 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:50.443481+0000", 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 394.31059926699999, 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.024653034000000001, 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:51:24.791 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443481+0000", 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443481+0000", 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443481+0000", 2024-02-13T00:51:24.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7215999999999999e-05 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443498+0000", 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1439999999999997e-06 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443505+0000", 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.409e-06 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443511+0000", 2024-02-13T00:51:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039672500000000001 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443908+0000", 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9888999999999999e-05 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443928+0000", 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.024168541000000002 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:51:24.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.468097+0000", 2024-02-13T00:51:24.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7477e-05 2024-02-13T00:51:24.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:51:24.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:51:24.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:51:24.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.468134+0000", 2024-02-13T00:51:24.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.024653034000000001 2024-02-13T00:51:24.795 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.795 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:24.795 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.795 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:51:24.795 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:51:24.795 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:51:24.796 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:24.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:23.889293+0000 mon.c (mon.2) 481 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:51:24.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:24.317079+0000 mon.c (mon.2) 482 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:51:24.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:24.317788+0000 mon.c (mon.2) 483 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:51:24.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:24.322434+0000 mon.a (mon.0) 3270 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:24.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:24.544354+0000 mon.a (mon.0) 3271 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:24.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:24.549286+0000 mon.a (mon.0) 3272 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-114", "mode": "writeback"}]': finished 2024-02-13T00:51:24.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:24.549351+0000 mon.a (mon.0) 3273 : cluster [DBG] osdmap e659: 8 total, 8 up, 4 in 2024-02-13T00:51:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:24.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:23.889293+0000 mon.c (mon.2) 481 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:51:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:24.317079+0000 mon.c (mon.2) 482 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:51:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:24.317788+0000 mon.c (mon.2) 483 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:51:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:24.322434+0000 mon.a (mon.0) 3270 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:24.544354+0000 mon.a (mon.0) 3271 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:24.549286+0000 mon.a (mon.0) 3272 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-114", "mode": "writeback"}]': finished 2024-02-13T00:51:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:24.549351+0000 mon.a (mon.0) 3273 : cluster [DBG] osdmap e659: 8 total, 8 up, 4 in 2024-02-13T00:51:24.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:23.889293+0000 mon.c (mon.2) 481 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:51:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:24.317079+0000 mon.c (mon.2) 482 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:51:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:24.317788+0000 mon.c (mon.2) 483 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:51:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:24.322434+0000 mon.a (mon.0) 3270 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:24.544354+0000 mon.a (mon.0) 3271 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:24.549286+0000 mon.a (mon.0) 3272 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-114", "mode": "writeback"}]': finished 2024-02-13T00:51:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:24.549351+0000 mon.a (mon.0) 3273 : cluster [DBG] osdmap e659: 8 total, 8 up, 4 in 2024-02-13T00:51:24.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:24.925 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:25.048 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:25.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:25.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:25.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:25.183 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:25.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-13T00:51:25.739 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:25.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:51:25.786 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:24.505961+0000 mgr.y (mgr.14544) 558 : cluster [DBG] pgmap v993: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:25.786 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:24.505961+0000 mgr.y (mgr.14544) 558 : cluster [DBG] pgmap v993: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:25.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:24.505961+0000 mgr.y (mgr.14544) 558 : cluster [DBG] pgmap v993: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:25.832 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:25.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:25.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:25.964 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:26.062 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:26.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:26.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:26.168 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:26.196 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:26.215 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:26.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:26.264 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:51:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:51:26.319 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:26.330 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:51:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:51:26] "GET /metrics HTTP/1.1" 200 186790 "" "Prometheus/2.33.4" 2024-02-13T00:51:26.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.326+0000 7efc86879700 1 -- 172.21.15.162:0/669284900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc800f68c0 msgr2=0x7efc80104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:26.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.326+0000 7efc86879700 1 --2- 172.21.15.162:0/669284900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc800f68c0 0x7efc80104ac0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 rx=0x7efc78009be0 tx=0x7efc78004d10).stop 2024-02-13T00:51:26.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.326+0000 7efc86879700 1 -- 172.21.15.162:0/669284900 shutdown_connections 2024-02-13T00:51:26.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.326+0000 7efc86879700 1 --2- 172.21.15.162:0/669284900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc800f68c0 0x7efc80104ac0 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:26.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.326+0000 7efc86879700 1 --2- 172.21.15.162:0/669284900 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efc800fdcb0 0x7efc800fe090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:26.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.326+0000 7efc86879700 1 --2- 172.21.15.162:0/669284900 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc800fe610 0x7efc800f61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:26.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.326+0000 7efc86879700 1 -- 172.21.15.162:0/669284900 >> 172.21.15.162:0/669284900 conn(0x7efc80070080 msgr2=0x7efc800f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:26.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.326+0000 7efc86879700 1 -- 172.21.15.162:0/669284900 shutdown_connections 2024-02-13T00:51:26.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.326+0000 7efc86879700 1 -- 172.21.15.162:0/669284900 wait complete. 2024-02-13T00:51:26.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.327+0000 7efc86879700 1 librados: starting msgr at 2024-02-13T00:51:26.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.327+0000 7efc86879700 1 librados: starting objecter 2024-02-13T00:51:26.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.327+0000 7efc86879700 1 Processor -- start 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.327+0000 7efc86879700 1 -- start start 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.327+0000 7efc86879700 1 librados: setting wanted keys 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.327+0000 7efc86879700 1 librados: calling monclient init 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc86879700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc800f68c0 0x7efc80103840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc86879700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efc800fdcb0 0x7efc80103d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc86879700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc800fe610 0x7efc80104310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc86879700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc80123230 con 0x7efc800f68c0 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc86879700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc801233a0 con 0x7efc800fe610 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc86879700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc80123560 con 0x7efc800fdcb0 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc84e16700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc800fe610 0x7efc80104310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc7f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efc800fdcb0 0x7efc80103d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc7f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efc800fdcb0 0x7efc80103d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:49364/0 (socket says 172.21.15.162:49364) 2024-02-13T00:51:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc84e16700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc800fe610 0x7efc80104310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:60666/0 (socket says 172.21.15.162:60666) 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc7f7fe700 1 -- 172.21.15.162:0/2133653754 learned_addr learned my addr 172.21.15.162:0/2133653754 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc7ffff700 1 --2- 172.21.15.162:0/2133653754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc800f68c0 0x7efc80103840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc84e16700 1 -- 172.21.15.162:0/2133653754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efc800fdcb0 msgr2=0x7efc80103d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc84e16700 1 --2- 172.21.15.162:0/2133653754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efc800fdcb0 0x7efc80103d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc84e16700 1 -- 172.21.15.162:0/2133653754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc800f68c0 msgr2=0x7efc80103840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc84e16700 1 --2- 172.21.15.162:0/2133653754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc800f68c0 0x7efc80103840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc84e16700 1 -- 172.21.15.162:0/2133653754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc780097c0 con 0x7efc800fe610 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc84e16700 1 --2- 172.21.15.162:0/2133653754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc800fe610 0x7efc80104310 secure :-1 s=READY pgs=4860 cs=0 l=1 rev1=1 rx=0x7efc78004e00 tx=0x7efc78004d10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc7d7fa700 10 client.?.objecter ms_handle_connect 0x7efc800fe610 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.328+0000 7efc7d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.329+0000 7efc7d7fa700 1 -- 172.21.15.162:0/2133653754 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7800e040 con 0x7efc800fe610 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.329+0000 7efc7d7fa700 1 -- 172.21.15.162:0/2133653754 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc78018040 con 0x7efc800fe610 2024-02-13T00:51:26.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.329+0000 7efc7d7fa700 1 -- 172.21.15.162:0/2133653754 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc78013cf0 con 0x7efc800fe610 2024-02-13T00:51:26.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.329+0000 7efc86879700 1 -- 172.21.15.162:0/2133653754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc801237e0 con 0x7efc800fe610 2024-02-13T00:51:26.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.329+0000 7efc86879700 10 client.44905.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:26.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.329+0000 7efc86879700 1 -- 172.21.15.162:0/2133653754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc80123b90 con 0x7efc800fe610 2024-02-13T00:51:26.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.329+0000 7efc86879700 1 librados: init done 2024-02-13T00:51:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.330+0000 7efc86879700 1 -- 172.21.15.162:0/2133653754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc8005fb80 con 0x7efc800fe610 2024-02-13T00:51:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.330+0000 7efc7d7fa700 1 -- 172.21.15.162:0/2133653754 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7efc78007b20 con 0x7efc800fe610 2024-02-13T00:51:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.330+0000 7efc7d7fa700 1 --2- 172.21.15.162:0/2133653754 >> v2:172.21.15.162:6800/2866534053 conn(0x7efc68062c90 0x7efc68065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.331+0000 7efc7d7fa700 1 -- 172.21.15.162:0/2133653754 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(660..660 src has 449..660) v4 ==== 8704+0+0 (secure 0 0 0) 0x7efc7807dfc0 con 0x7efc800fe610 2024-02-13T00:51:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.331+0000 7efc7d7fa700 10 client.44905.objecter ms_dispatch 0x7efc80001550 osd_map(660..660 src has 449..660) v4 2024-02-13T00:51:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.331+0000 7efc7d7fa700 3 client.44905.objecter handle_osd_map got epochs [660,660] > 0 2024-02-13T00:51:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.331+0000 7efc7d7fa700 3 client.44905.objecter handle_osd_map decoding full epoch 660 2024-02-13T00:51:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.331+0000 7efc7ffff700 1 --2- 172.21.15.162:0/2133653754 >> v2:172.21.15.162:6800/2866534053 conn(0x7efc68062c90 0x7efc68065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.331+0000 7efc7d7fa700 20 client.44905.objecter dump_active .. 0 homeless 2024-02-13T00:51:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.331+0000 7efc7d7fa700 10 client.44905.objecter ms_handle_connect 0x7efc68062c90 2024-02-13T00:51:26.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.331+0000 7efc7ffff700 1 --2- 172.21.15.162:0/2133653754 >> v2:172.21.15.162:6800/2866534053 conn(0x7efc68062c90 0x7efc68065150 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7efc800f9270 tx=0x7efc7000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:26.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.335+0000 7efc7d7fa700 1 -- 172.21.15.162:0/2133653754 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7efc780512e0 con 0x7efc800fe610 2024-02-13T00:51:26.349 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:26.351 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:26.377 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:26.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:26.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:26.512 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:26.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.532+0000 7efc86879700 1 -- 172.21.15.162:0/2133653754 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7efc80104c80 con 0x7efc800fe610 2024-02-13T00:51:26.610 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:26.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:26.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.619+0000 7efc7d7fa700 1 -- 172.21.15.162:0/2133653754 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v661) v1 ==== 97+0+0 (secure 0 0 0) 0x7efc78021030 con 0x7efc800fe610 2024-02-13T00:51:26.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.622+0000 7efc86879700 10 librados: watch_flush enter 2024-02-13T00:51:26.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.622+0000 7efc86879700 10 librados: watch_flush exit 2024-02-13T00:51:26.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.622+0000 7efc86879700 20 client.44905.objecter shutdown clearing up homeless session... 2024-02-13T00:51:26.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.622+0000 7efc86879700 10 client.44905.objecter successfully canceled tick 2024-02-13T00:51:26.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.622+0000 7efc86879700 1 -- 172.21.15.162:0/2133653754 >> v2:172.21.15.162:6800/2866534053 conn(0x7efc68062c90 msgr2=0x7efc68065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:26.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.622+0000 7efc86879700 1 --2- 172.21.15.162:0/2133653754 >> v2:172.21.15.162:6800/2866534053 conn(0x7efc68062c90 0x7efc68065150 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7efc800f9270 tx=0x7efc7000b040).stop 2024-02-13T00:51:26.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.622+0000 7efc86879700 1 -- 172.21.15.162:0/2133653754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc800fe610 msgr2=0x7efc80104310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:26.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.622+0000 7efc86879700 1 --2- 172.21.15.162:0/2133653754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc800fe610 0x7efc80104310 secure :-1 s=READY pgs=4860 cs=0 l=1 rev1=1 rx=0x7efc78004e00 tx=0x7efc78004d10).stop 2024-02-13T00:51:26.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.623+0000 7efc86879700 1 -- 172.21.15.162:0/2133653754 shutdown_connections 2024-02-13T00:51:26.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.623+0000 7efc86879700 1 --2- 172.21.15.162:0/2133653754 >> v2:172.21.15.162:6800/2866534053 conn(0x7efc68062c90 0x7efc68065150 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:26.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.623+0000 7efc86879700 1 --2- 172.21.15.162:0/2133653754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc800fe610 0x7efc80104310 unknown :-1 s=CLOSED pgs=4860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:26.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.623+0000 7efc86879700 1 --2- 172.21.15.162:0/2133653754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc800f68c0 0x7efc80103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:26.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.623+0000 7efc86879700 1 --2- 172.21.15.162:0/2133653754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efc800fdcb0 0x7efc80103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:26.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.623+0000 7efc86879700 1 -- 172.21.15.162:0/2133653754 >> 172.21.15.162:0/2133653754 conn(0x7efc80070080 msgr2=0x7efc8006d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:26.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.623+0000 7efc86879700 1 -- 172.21.15.162:0/2133653754 shutdown_connections 2024-02-13T00:51:26.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.623+0000 7efc86879700 1 -- 172.21.15.162:0/2133653754 wait complete. 2024-02-13T00:51:26.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:26.623+0000 7efc86879700 1 librados: shutdown 2024-02-13T00:51:26.629 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:51:26.717 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:26.746 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:26.939 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:25.615411+0000 mon.a (mon.0) 3274 : cluster [DBG] osdmap e660: 8 total, 8 up, 4 in 2024-02-13T00:51:26.940 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:25.641365+0000 mon.a (mon.0) 3275 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:26.940 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:25.843939+0000 mgr.y (mgr.14544) 559 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:26.940 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:26.533828+0000 mon.b (mon.1) 680 : audit [INF] from='client.? 172.21.15.162:0/2133653754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:26.940 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:26.534151+0000 mon.a (mon.0) 3276 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:26.940 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:25.615411+0000 mon.a (mon.0) 3274 : cluster [DBG] osdmap e660: 8 total, 8 up, 4 in 2024-02-13T00:51:26.940 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:25.641365+0000 mon.a (mon.0) 3275 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:26.940 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:25.843939+0000 mgr.y (mgr.14544) 559 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:26.940 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:26.533828+0000 mon.b (mon.1) 680 : audit [INF] from='client.? 172.21.15.162:0/2133653754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:26.940 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:26.534151+0000 mon.a (mon.0) 3276 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:26.965 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:25.615411+0000 mon.a (mon.0) 3274 : cluster [DBG] osdmap e660: 8 total, 8 up, 4 in 2024-02-13T00:51:26.965 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:25.641365+0000 mon.a (mon.0) 3275 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:26.965 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:25.843939+0000 mgr.y (mgr.14544) 559 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:26.965 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:26.533828+0000 mon.b (mon.1) 680 : audit [INF] from='client.? 172.21.15.162:0/2133653754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:26.966 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:26.534151+0000 mon.a (mon.0) 3276 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:26.994 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:26.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:27.098 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:27.125 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:27.222 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:27.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:27.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:27.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:27.354 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:27.382 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:27.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:27.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:27.522 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:27.538 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:51:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:51:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:51:27.539 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:51:27.870 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:27.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:26.506417+0000 mgr.y (mgr.14544) 560 : cluster [DBG] pgmap v996: 204 pgs: 204 active+clean; 5.8 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-13T00:51:27.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:26.608144+0000 mon.a (mon.0) 3277 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:27.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:26.619365+0000 mon.a (mon.0) 3278 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:27.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:26.619429+0000 mon.a (mon.0) 3279 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:51:27.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:26.619487+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e661: 8 total, 8 up, 4 in 2024-02-13T00:51:27.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:26.620494+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-114"}]: dispatch 2024-02-13T00:51:27.873 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:26.506417+0000 mgr.y (mgr.14544) 560 : cluster [DBG] pgmap v996: 204 pgs: 204 active+clean; 5.8 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-13T00:51:27.873 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:26.608144+0000 mon.a (mon.0) 3277 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:27.873 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:26.619365+0000 mon.a (mon.0) 3278 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:27.873 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:26.619429+0000 mon.a (mon.0) 3279 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:51:27.873 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:26.619487+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e661: 8 total, 8 up, 4 in 2024-02-13T00:51:27.873 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:26.620494+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-114"}]: dispatch 2024-02-13T00:51:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:27.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:27.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:26.506417+0000 mgr.y (mgr.14544) 560 : cluster [DBG] pgmap v996: 204 pgs: 204 active+clean; 5.8 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-13T00:51:27.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:26.608144+0000 mon.a (mon.0) 3277 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:27.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:26.619365+0000 mon.a (mon.0) 3278 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:27.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:26.619429+0000 mon.a (mon.0) 3279 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:51:27.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:26.619487+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e661: 8 total, 8 up, 4 in 2024-02-13T00:51:27.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:26.620494+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-114"}]: dispatch 2024-02-13T00:51:27.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:28.001 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:28.132 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6] out_osds: [3, 2, 7, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-13T00:51:28.132 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:51:28.132 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-02-13T00:51:28.132 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-13T00:51:28.132 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-13T00:51:28.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-02-13T00:51:28.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-13T00:51:28.428 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:28.663 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:27.614221+0000 mon.a (mon.0) 3282 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:28.663 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:27.619963+0000 mon.a (mon.0) 3283 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-114"}]': finished 2024-02-13T00:51:28.663 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:27.620029+0000 mon.a (mon.0) 3284 : cluster [DBG] osdmap e662: 8 total, 8 up, 4 in 2024-02-13T00:51:28.664 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:27.614221+0000 mon.a (mon.0) 3282 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:28.664 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:27.619963+0000 mon.a (mon.0) 3283 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-114"}]': finished 2024-02-13T00:51:28.664 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:27.620029+0000 mon.a (mon.0) 3284 : cluster [DBG] osdmap e662: 8 total, 8 up, 4 in 2024-02-13T00:51:28.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:27.614221+0000 mon.a (mon.0) 3282 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:28.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:27.619963+0000 mon.a (mon.0) 3283 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-114"}]': finished 2024-02-13T00:51:28.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:27.620029+0000 mon.a (mon.0) 3284 : cluster [DBG] osdmap e662: 8 total, 8 up, 4 in 2024-02-13T00:51:28.898 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:51:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:51:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:51:28.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.918+0000 7f9f72650700 1 -- 172.21.15.162:0/2266459556 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6c0f73a0 msgr2=0x7f9f6c1039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:28.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.918+0000 7f9f72650700 1 --2- 172.21.15.162:0/2266459556 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6c0f73a0 0x7f9f6c1039a0 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 rx=0x7f9f64009c30 tx=0x7f9f64004c50).stop 2024-02-13T00:51:28.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.918+0000 7f9f72650700 1 -- 172.21.15.162:0/2266459556 shutdown_connections 2024-02-13T00:51:28.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.918+0000 7f9f72650700 1 --2- 172.21.15.162:0/2266459556 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6c0f73a0 0x7f9f6c1039a0 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:28.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.918+0000 7f9f72650700 1 --2- 172.21.15.162:0/2266459556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f6c0fcb30 0x7f9f6c0fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:51:28.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.918+0000 7f9f72650700 1 --2- 172.21.15.162:0/2266459556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f6c0fd490 0x7f9f6c0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:28.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.918+0000 7f9f72650700 1 -- 172.21.15.162:0/2266459556 >> 172.21.15.162:0/2266459556 conn(0x7f9f6c06eb90 msgr2=0x7f9f6c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:28.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.918+0000 7f9f72650700 1 -- 172.21.15.162:0/2266459556 shutdown_connections 2024-02-13T00:51:28.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.918+0000 7f9f72650700 1 -- 172.21.15.162:0/2266459556 wait complete. 2024-02-13T00:51:28.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f72650700 1 librados: starting msgr at 2024-02-13T00:51:28.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f72650700 1 librados: starting objecter 2024-02-13T00:51:28.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f72650700 1 Processor -- start 2024-02-13T00:51:28.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f72650700 1 -- start start 2024-02-13T00:51:28.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f72650700 1 librados: setting wanted keys 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f72650700 1 librados: calling monclient init 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f72650700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6c0f73a0 0x7f9f6c10cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f72650700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f6c0fcb30 0x7f9f6c10d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f72650700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f6c0fd490 0x7f9f6c1070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f72650700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c1075d0 con 0x7f9f6c0fcb30 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f72650700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c107740 con 0x7f9f6c0fd490 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f72650700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c107900 con 0x7f9f6c0f73a0 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f6b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f6c0fcb30 0x7f9f6c10d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.919+0000 7f9f6b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f6c0fcb30 0x7f9f6c10d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44960/0 (socket says 172.21.15.162:44960) 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f6b7fe700 1 -- 172.21.15.162:0/1116176153 learned_addr learned my addr 172.21.15.162:0/1116176153 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f6bfff700 1 --2- 172.21.15.162:0/1116176153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6c0f73a0 0x7f9f6c10cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f70bed700 1 --2- 172.21.15.162:0/1116176153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f6c0fd490 0x7f9f6c1070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:28.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f6b7fe700 1 -- 172.21.15.162:0/1116176153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6c0f73a0 msgr2=0x7f9f6c10cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f6b7fe700 1 --2- 172.21.15.162:0/1116176153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6c0f73a0 0x7f9f6c10cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f6b7fe700 1 -- 172.21.15.162:0/1116176153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f6c0fd490 msgr2=0x7f9f6c1070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f6b7fe700 1 --2- 172.21.15.162:0/1116176153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f6c0fd490 0x7f9f6c1070e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f6b7fe700 1 -- 172.21.15.162:0/1116176153 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f54005040 con 0x7f9f6c0fcb30 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f6bfff700 1 --2- 172.21.15.162:0/1116176153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6c0f73a0 0x7f9f6c10cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f6b7fe700 1 --2- 172.21.15.162:0/1116176153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f6c0fcb30 0x7f9f6c10d430 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 rx=0x7f9f54009400 tx=0x7f9f54009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f697fa700 10 client.?.objecter ms_handle_connect 0x7f9f6c0fcb30 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f697fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f54007a40 con 0x7f9f6c0fcb30 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f640097c0 con 0x7f9f6c0fcb30 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f72650700 10 client.27903.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6c107f30 con 0x7f9f6c0fcb30 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f72650700 1 librados: init done 2024-02-13T00:51:28.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f54012e20 con 0x7f9f6c0fcb30 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.920+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5401bd20 con 0x7f9f6c0fcb30 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.921+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9f4c004f30 con 0x7f9f6c0fcb30 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f9f54012950 con 0x7f9f6c0fcb30 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f697fa700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9f58062c90 0x7f9f58065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(663..663 src has 449..663) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f9f5407fc90 con 0x7f9f6c0fcb30 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 osd_map(663..663 src has 449..663) v4 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f697fa700 3 client.27903.objecter handle_osd_map got epochs [663,663] > 0 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f697fa700 3 client.27903.objecter handle_osd_map decoding full epoch 663 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f6bfff700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9f58062c90 0x7f9f58065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f697fa700 20 client.27903.objecter dump_active .. 0 homeless 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v663) v1 ==== 54+0+16 (secure 0 0 0) 0x7f9f54080130 con 0x7f9f6c0fcb30 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f72650700 10 client.27903.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f697fa700 10 client.27903.objecter ms_handle_connect 0x7f9f58062c90 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f6bfff700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9f58062c90 0x7f9f58065150 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 rx=0x7f9f5c006fd0 tx=0x7f9f5c009060).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 1 2024-02-13T00:51:28.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9f4c004560 0x7f9f4c004980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:28.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c0051d0 osd=0 3 2024-02-13T00:51:28.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0051d0 osd=0 3 2024-02-13T00:51:28.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c0051d0 osd=0 3 2024-02-13T00:51:28.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 1 2024-02-13T00:51:28.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c0051d0 osd=0 3 2024-02-13T00:51:28.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 0 1 2024-02-13T00:51:28.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0051d0 osd=0 4 2024-02-13T00:51:28.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f72650700 10 client.27903.objecter _send_command 1 2024-02-13T00:51:28.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.922+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f4c008960 con 0x7f9f4c004560 2024-02-13T00:51:28.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.923+0000 7f9f70bed700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9f4c004560 0x7f9f4c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:28.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.923+0000 7f9f697fa700 10 client.27903.objecter ms_handle_connect 0x7f9f4c004560 2024-02-13T00:51:28.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.923+0000 7f9f70bed700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9f4c004560 0x7f9f4c004980 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:28.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.923+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9f4c008960 con 0x7f9f4c004560 2024-02-13T00:51:28.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.923+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:51:28.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.923+0000 7f9f697fa700 10 client.27903.objecter _finish_command 1 = system:0 2024-02-13T00:51:28.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.923+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c0051d0 osd=0 4 2024-02-13T00:51:28.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.923+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 0 1 2024-02-13T00:51:28.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.938+0000 7f9f72650700 10 client.27903.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:28.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.938+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 2 2024-02-13T00:51:28.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.938+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c0051d0 osd=0 3 2024-02-13T00:51:28.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.938+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0051d0 osd=0 3 2024-02-13T00:51:28.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.938+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c0051d0 osd=0 3 2024-02-13T00:51:28.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.938+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 2 2024-02-13T00:51:28.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.938+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c0051d0 osd=0 3 2024-02-13T00:51:28.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.938+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 0 2 2024-02-13T00:51:28.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.938+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0051d0 osd=0 4 2024-02-13T00:51:28.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.938+0000 7f9f72650700 10 client.27903.objecter _send_command 2 2024-02-13T00:51:28.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.938+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9f4c0081f0 con 0x7f9f4c004560 2024-02-13T00:51:28.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.939+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9f4c0081f0 con 0x7f9f4c004560 2024-02-13T00:51:28.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.939+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:28.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.939+0000 7f9f697fa700 10 client.27903.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:28.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.939+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c0051d0 osd=0 4 2024-02-13T00:51:28.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.939+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 0 2 2024-02-13T00:51:28.944 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f72650700 10 client.27903.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 3 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6805/150599780 conn(0x7f9f4c002c90 0x7f9f4c003120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c0085a0 osd=1 3 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0085a0 osd=1 3 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c0085a0 osd=1 3 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 3 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c0085a0 osd=1 3 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 1 3 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0085a0 osd=1 4 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f72650700 10 client.27903.objecter _send_command 3 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f4c003610 con 0x7f9f4c002c90 2024-02-13T00:51:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f6bfff700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6805/150599780 conn(0x7f9f4c002c90 0x7f9f4c003120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:28.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f697fa700 10 client.27903.objecter ms_handle_connect 0x7f9f4c002c90 2024-02-13T00:51:28.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f6bfff700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6805/150599780 conn(0x7f9f4c002c90 0x7f9f4c003120 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:28.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.940+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9f4c003610 con 0x7f9f4c002c90 2024-02-13T00:51:28.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.941+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:51:28.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.941+0000 7f9f697fa700 10 client.27903.objecter _finish_command 3 = system:0 2024-02-13T00:51:28.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.941+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c0085a0 osd=1 4 2024-02-13T00:51:28.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.941+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 1 3 2024-02-13T00:51:28.949 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:28.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:28.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:28.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.956+0000 7f9f72650700 10 client.27903.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:28.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.956+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 4 2024-02-13T00:51:28.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.956+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c0085a0 osd=1 3 2024-02-13T00:51:28.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.956+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0085a0 osd=1 3 2024-02-13T00:51:28.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.956+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c0085a0 osd=1 3 2024-02-13T00:51:28.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.956+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 4 2024-02-13T00:51:28.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.956+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c0085a0 osd=1 3 2024-02-13T00:51:28.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.956+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 1 4 2024-02-13T00:51:28.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.956+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0085a0 osd=1 4 2024-02-13T00:51:28.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.956+0000 7f9f72650700 10 client.27903.objecter _send_command 4 2024-02-13T00:51:28.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.956+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9f4c007c60 con 0x7f9f4c002c90 2024-02-13T00:51:28.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9f4c007c60 con 0x7f9f4c002c90 2024-02-13T00:51:28.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:28.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f697fa700 10 client.27903.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:28.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c0085a0 osd=1 4 2024-02-13T00:51:28.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 1 4 2024-02-13T00:51:28.962 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:51:28.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f72650700 10 client.27903.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:28.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 5 2024-02-13T00:51:28.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6809/3689719376 conn(0x7f9f4c004120 0x7f9f4c011410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c003fb0 osd=2 3 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c003fb0 osd=2 3 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c003fb0 osd=2 3 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 5 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c003fb0 osd=2 3 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 2 5 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c003fb0 osd=2 4 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f72650700 10 client.27903.objecter _send_command 5 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f4c014260 con 0x7f9f4c004120 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f6b7fe700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6809/3689719376 conn(0x7f9f4c004120 0x7f9f4c011410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f697fa700 10 client.27903.objecter ms_handle_connect 0x7f9f4c004120 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.958+0000 7f9f6b7fe700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6809/3689719376 conn(0x7f9f4c004120 0x7f9f4c011410 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.959+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9f4c014260 con 0x7f9f4c004120 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.959+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:51:28.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.959+0000 7f9f697fa700 10 client.27903.objecter _finish_command 5 = system:0 2024-02-13T00:51:28.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.959+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c003fb0 osd=2 4 2024-02-13T00:51:28.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.959+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 2 5 2024-02-13T00:51:28.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.973+0000 7f9f72650700 10 client.27903.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:28.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.973+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 6 2024-02-13T00:51:28.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.973+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c003fb0 osd=2 3 2024-02-13T00:51:28.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.973+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c003fb0 osd=2 3 2024-02-13T00:51:28.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.973+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c003fb0 osd=2 3 2024-02-13T00:51:28.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.973+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 6 2024-02-13T00:51:28.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.973+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c003fb0 osd=2 3 2024-02-13T00:51:28.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.973+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 2 6 2024-02-13T00:51:28.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.973+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c003fb0 osd=2 4 2024-02-13T00:51:28.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.973+0000 7f9f72650700 10 client.27903.objecter _send_command 6 2024-02-13T00:51:28.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.973+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9f4c003a20 con 0x7f9f4c004120 2024-02-13T00:51:28.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.974+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9f4c003a20 con 0x7f9f4c004120 2024-02-13T00:51:28.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.974+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:28.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.974+0000 7f9f697fa700 10 client.27903.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:28.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.974+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c003fb0 osd=2 4 2024-02-13T00:51:28.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.974+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 2 6 2024-02-13T00:51:28.978 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:51:28.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f72650700 10 client.27903.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:28.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 7 2024-02-13T00:51:28.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6813/604971011 conn(0x7f9f4c0119d0 0x7f9f4c011e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:28.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c001b90 osd=3 3 2024-02-13T00:51:28.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c001b90 osd=3 3 2024-02-13T00:51:28.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c001b90 osd=3 3 2024-02-13T00:51:28.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 7 2024-02-13T00:51:28.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c001b90 osd=3 3 2024-02-13T00:51:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 3 7 2024-02-13T00:51:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c001b90 osd=3 4 2024-02-13T00:51:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f72650700 10 client.27903.objecter _send_command 7 2024-02-13T00:51:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f4c002380 con 0x7f9f4c0119d0 2024-02-13T00:51:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f70bed700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6813/604971011 conn(0x7f9f4c0119d0 0x7f9f4c011e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f697fa700 10 client.27903.objecter ms_handle_connect 0x7f9f4c0119d0 2024-02-13T00:51:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.975+0000 7f9f70bed700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6813/604971011 conn(0x7f9f4c0119d0 0x7f9f4c011e10 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.976+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9f4c002380 con 0x7f9f4c0119d0 2024-02-13T00:51:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.976+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:51:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.976+0000 7f9f697fa700 10 client.27903.objecter _finish_command 7 = system:0 2024-02-13T00:51:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.976+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c001b90 osd=3 4 2024-02-13T00:51:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.976+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 3 7 2024-02-13T00:51:28.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.990+0000 7f9f72650700 10 client.27903.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:28.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.990+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 8 2024-02-13T00:51:28.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.990+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c001b90 osd=3 3 2024-02-13T00:51:28.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.990+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c001b90 osd=3 3 2024-02-13T00:51:28.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.990+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c001b90 osd=3 3 2024-02-13T00:51:28.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.990+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 8 2024-02-13T00:51:28.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.990+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c001b90 osd=3 3 2024-02-13T00:51:28.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.990+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 3 8 2024-02-13T00:51:28.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.990+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c001b90 osd=3 4 2024-02-13T00:51:28.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.990+0000 7f9f72650700 10 client.27903.objecter _send_command 8 2024-02-13T00:51:28.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.990+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9f4c013c80 con 0x7f9f4c0119d0 2024-02-13T00:51:28.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9f4c013c80 con 0x7f9f4c0119d0 2024-02-13T00:51:28.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:28.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f697fa700 10 client.27903.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:28.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c001b90 osd=3 4 2024-02-13T00:51:28.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 3 8 2024-02-13T00:51:28.995 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:51:28.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f72650700 10 client.27903.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:28.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 9 2024-02-13T00:51:28.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9f4c016440 0x7f9f4c018830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:28.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c0134b0 osd=4 3 2024-02-13T00:51:28.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0134b0 osd=4 3 2024-02-13T00:51:28.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c0134b0 osd=4 3 2024-02-13T00:51:28.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 9 2024-02-13T00:51:28.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c0134b0 osd=4 3 2024-02-13T00:51:28.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 4 9 2024-02-13T00:51:28.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0134b0 osd=4 4 2024-02-13T00:51:28.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f72650700 10 client.27903.objecter _send_command 9 2024-02-13T00:51:28.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.991+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f4c01b480 con 0x7f9f4c016440 2024-02-13T00:51:28.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.992+0000 7f9f6bfff700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9f4c016440 0x7f9f4c018830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:28.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.992+0000 7f9f697fa700 10 client.27903.objecter ms_handle_connect 0x7f9f4c016440 2024-02-13T00:51:28.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.992+0000 7f9f6bfff700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9f4c016440 0x7f9f4c018830 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:28.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.993+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9f4c01b480 con 0x7f9f4c016440 2024-02-13T00:51:28.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.993+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:51:28.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.993+0000 7f9f697fa700 10 client.27903.objecter _finish_command 9 = system:0 2024-02-13T00:51:28.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.993+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c0134b0 osd=4 4 2024-02-13T00:51:28.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:28.993+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 4 9 2024-02-13T00:51:29.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.007+0000 7f9f72650700 10 client.27903.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:29.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.007+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 10 2024-02-13T00:51:29.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.007+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c0134b0 osd=4 3 2024-02-13T00:51:29.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.007+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0134b0 osd=4 3 2024-02-13T00:51:29.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.007+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c0134b0 osd=4 3 2024-02-13T00:51:29.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.007+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 10 2024-02-13T00:51:29.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.007+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c0134b0 osd=4 3 2024-02-13T00:51:29.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.007+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 4 10 2024-02-13T00:51:29.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.007+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0134b0 osd=4 4 2024-02-13T00:51:29.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.007+0000 7f9f72650700 10 client.27903.objecter _send_command 10 2024-02-13T00:51:29.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.007+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9f4c018d20 con 0x7f9f4c016440 2024-02-13T00:51:29.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9f4c018d20 con 0x7f9f4c016440 2024-02-13T00:51:29.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:29.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f697fa700 10 client.27903.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:29.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c0134b0 osd=4 4 2024-02-13T00:51:29.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 4 10 2024-02-13T00:51:29.013 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:51:29.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f72650700 10 client.27903.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:29.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 11 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6804/2277917372 conn(0x7f9f4c019400 0x7f9f4c01b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c018d20 osd=5 3 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c018d20 osd=5 3 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c018d20 osd=5 3 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 11 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c018d20 osd=5 3 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 5 11 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c018d20 osd=5 4 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f72650700 10 client.27903.objecter _send_command 11 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f4c01e5f0 con 0x7f9f4c019400 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f6b7fe700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6804/2277917372 conn(0x7f9f4c019400 0x7f9f4c01b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f697fa700 10 client.27903.objecter ms_handle_connect 0x7f9f4c019400 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.009+0000 7f9f6b7fe700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6804/2277917372 conn(0x7f9f4c019400 0x7f9f4c01b7f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.010+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9f4c01e5f0 con 0x7f9f4c019400 2024-02-13T00:51:29.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.010+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:51:29.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.010+0000 7f9f697fa700 10 client.27903.objecter _finish_command 11 = system:0 2024-02-13T00:51:29.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.010+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c018d20 osd=5 4 2024-02-13T00:51:29.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.010+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 5 11 2024-02-13T00:51:29.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.024+0000 7f9f72650700 10 client.27903.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:29.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.024+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 12 2024-02-13T00:51:29.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.024+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c018d20 osd=5 3 2024-02-13T00:51:29.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.024+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c018d20 osd=5 3 2024-02-13T00:51:29.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.024+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c018d20 osd=5 3 2024-02-13T00:51:29.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.024+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 12 2024-02-13T00:51:29.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.024+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c018d20 osd=5 3 2024-02-13T00:51:29.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.024+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 5 12 2024-02-13T00:51:29.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.024+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c018d20 osd=5 4 2024-02-13T00:51:29.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.024+0000 7f9f72650700 10 client.27903.objecter _send_command 12 2024-02-13T00:51:29.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.024+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9f4c01e5f0 con 0x7f9f4c019400 2024-02-13T00:51:29.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9f4c01e5f0 con 0x7f9f4c019400 2024-02-13T00:51:29.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:29.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f697fa700 10 client.27903.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:29.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c018d20 osd=5 4 2024-02-13T00:51:29.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 5 12 2024-02-13T00:51:29.030 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f72650700 10 client.27903.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 13 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9f4c01d8f0 0x7f9f4c01dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c01e5f0 osd=6 3 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c01e5f0 osd=6 3 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c01e5f0 osd=6 3 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 13 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c01e5f0 osd=6 3 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 6 13 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c01e5f0 osd=6 4 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f72650700 10 client.27903.objecter _send_command 13 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f4c01e3c0 con 0x7f9f4c01d8f0 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.026+0000 7f9f70bed700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9f4c01d8f0 0x7f9f4c01dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.027+0000 7f9f697fa700 10 client.27903.objecter ms_handle_connect 0x7f9f4c01d8f0 2024-02-13T00:51:29.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.027+0000 7f9f70bed700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9f4c01d8f0 0x7f9f4c01dd50 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:29.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.027+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9f4c01e3c0 con 0x7f9f4c01d8f0 2024-02-13T00:51:29.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.027+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:51:29.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.027+0000 7f9f697fa700 10 client.27903.objecter _finish_command 13 = system:0 2024-02-13T00:51:29.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.027+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c01e5f0 osd=6 4 2024-02-13T00:51:29.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.027+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 6 13 2024-02-13T00:51:29.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.041+0000 7f9f72650700 10 client.27903.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:29.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.041+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 14 2024-02-13T00:51:29.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.041+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c01e5f0 osd=6 3 2024-02-13T00:51:29.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.041+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c01e5f0 osd=6 3 2024-02-13T00:51:29.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.041+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c01e5f0 osd=6 3 2024-02-13T00:51:29.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.041+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 14 2024-02-13T00:51:29.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.041+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c01e5f0 osd=6 3 2024-02-13T00:51:29.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.041+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 6 14 2024-02-13T00:51:29.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.041+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c01e5f0 osd=6 4 2024-02-13T00:51:29.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.041+0000 7f9f72650700 10 client.27903.objecter _send_command 14 2024-02-13T00:51:29.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.041+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9f4c01e3c0 con 0x7f9f4c01d8f0 2024-02-13T00:51:29.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9f4c01e3c0 con 0x7f9f4c01d8f0 2024-02-13T00:51:29.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:29.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f697fa700 10 client.27903.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:29.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c01e5f0 osd=6 4 2024-02-13T00:51:29.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 6 14 2024-02-13T00:51:29.047 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:51:29.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f72650700 10 client.27903.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:29.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 15 2024-02-13T00:51:29.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6812/1964192772 conn(0x7f9f4c020da0 0x7f9f4c0211a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:29.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c01e3c0 osd=7 3 2024-02-13T00:51:29.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c01e3c0 osd=7 3 2024-02-13T00:51:29.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c01e3c0 osd=7 3 2024-02-13T00:51:29.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 15 2024-02-13T00:51:29.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c01e3c0 osd=7 3 2024-02-13T00:51:29.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 7 15 2024-02-13T00:51:29.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c01e3c0 osd=7 4 2024-02-13T00:51:29.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f72650700 10 client.27903.objecter _send_command 15 2024-02-13T00:51:29.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f4c021810 con 0x7f9f4c020da0 2024-02-13T00:51:29.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.043+0000 7f9f6bfff700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6812/1964192772 conn(0x7f9f4c020da0 0x7f9f4c0211a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:29.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.044+0000 7f9f697fa700 10 client.27903.objecter ms_handle_connect 0x7f9f4c020da0 2024-02-13T00:51:29.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.044+0000 7f9f6bfff700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6812/1964192772 conn(0x7f9f4c020da0 0x7f9f4c0211a0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:29.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.044+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9f4c021810 con 0x7f9f4c020da0 2024-02-13T00:51:29.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.044+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:51:29.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.044+0000 7f9f697fa700 10 client.27903.objecter _finish_command 15 = system:0 2024-02-13T00:51:29.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.044+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c01e3c0 osd=7 4 2024-02-13T00:51:29.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.044+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 7 15 2024-02-13T00:51:29.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:29.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.059+0000 7f9f72650700 10 client.27903.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:29.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.059+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign -1 16 2024-02-13T00:51:29.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.059+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c01e3c0 osd=7 3 2024-02-13T00:51:29.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.059+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c01e3c0 osd=7 3 2024-02-13T00:51:29.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.059+0000 7f9f72650700 20 client.27903.objecter _get_session s=0x7f9f4c01e3c0 osd=7 3 2024-02-13T00:51:29.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.059+0000 7f9f72650700 15 client.27903.objecter _session_command_op_remove -1 16 2024-02-13T00:51:29.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.059+0000 7f9f72650700 20 client.27903.objecter get_session s=0x7f9f4c01e3c0 osd=7 3 2024-02-13T00:51:29.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.059+0000 7f9f72650700 15 client.27903.objecter _session_command_op_assign 7 16 2024-02-13T00:51:29.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.059+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c01e3c0 osd=7 4 2024-02-13T00:51:29.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.059+0000 7f9f72650700 10 client.27903.objecter _send_command 16 2024-02-13T00:51:29.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.059+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9f4c021810 con 0x7f9f4c020da0 2024-02-13T00:51:29.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.060+0000 7f9f697fa700 1 -- 172.21.15.162:0/1116176153 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9f4c021810 con 0x7f9f4c020da0 2024-02-13T00:51:29.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.060+0000 7f9f697fa700 10 client.27903.objecter ms_dispatch 0x7f9f6c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:29.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.060+0000 7f9f697fa700 10 client.27903.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:29.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.060+0000 7f9f697fa700 20 client.27903.objecter put_session s=0x7f9f4c01e3c0 osd=7 4 2024-02-13T00:51:29.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.060+0000 7f9f697fa700 15 client.27903.objecter _session_command_op_remove 7 16 2024-02-13T00:51:29.064 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:51:29.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 10 librados: watch_flush enter 2024-02-13T00:51:29.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 10 librados: watch_flush exit 2024-02-13T00:51:29.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 10 client.27903.objecter close_session for osd.0 2024-02-13T00:51:29.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9f4c004560 msgr2=0x7f9f4c004980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:29.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9f4c004560 0x7f9f4c004980 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:29.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0051d0 osd=0 1 2024-02-13T00:51:29.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 10 client.27903.objecter close_session for osd.1 2024-02-13T00:51:29.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6805/150599780 conn(0x7f9f4c002c90 msgr2=0x7f9f4c003120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:29.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6805/150599780 conn(0x7f9f4c002c90 0x7f9f4c003120 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:29.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0085a0 osd=1 1 2024-02-13T00:51:29.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 10 client.27903.objecter close_session for osd.2 2024-02-13T00:51:29.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6809/3689719376 conn(0x7f9f4c004120 msgr2=0x7f9f4c011410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:29.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6809/3689719376 conn(0x7f9f4c004120 0x7f9f4c011410 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c003fb0 osd=2 1 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 10 client.27903.objecter close_session for osd.3 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6813/604971011 conn(0x7f9f4c0119d0 msgr2=0x7f9f4c011e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6813/604971011 conn(0x7f9f4c0119d0 0x7f9f4c011e10 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f70bed700 1 -- 172.21.15.162:0/1116176153 reap_dead start 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c001b90 osd=3 1 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 10 client.27903.objecter close_session for osd.4 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9f4c016440 msgr2=0x7f9f4c018830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6800/3344263388 conn(0x7f9f4c016440 0x7f9f4c018830 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c0134b0 osd=4 1 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 10 client.27903.objecter close_session for osd.5 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6804/2277917372 conn(0x7f9f4c019400 msgr2=0x7f9f4c01b7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6804/2277917372 conn(0x7f9f4c019400 0x7f9f4c01b7f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c018d20 osd=5 1 2024-02-13T00:51:29.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 10 client.27903.objecter close_session for osd.6 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9f4c01d8f0 msgr2=0x7f9f4c01dd50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9f4c01d8f0 0x7f9f4c01dd50 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c01e5f0 osd=6 1 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 10 client.27903.objecter close_session for osd.7 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6812/1964192772 conn(0x7f9f4c020da0 msgr2=0x7f9f4c0211a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.181:6812/1964192772 conn(0x7f9f4c020da0 0x7f9f4c0211a0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 20 client.27903.objecter put_session s=0x7f9f4c01e3c0 osd=7 1 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 20 client.27903.objecter shutdown clearing up homeless session... 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 10 client.27903.objecter successfully canceled tick 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f70bed700 1 -- 172.21.15.162:0/1116176153 reap_dead start 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9f58062c90 msgr2=0x7f9f58065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9f58062c90 0x7f9f58065150 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 rx=0x7f9f5c006fd0 tx=0x7f9f5c009060).stop 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f6c0fcb30 msgr2=0x7f9f6c10d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:29.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f6c0fcb30 0x7f9f6c10d430 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 rx=0x7f9f54009400 tx=0x7f9f54009cb0).stop 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 shutdown_connections 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f6c0fcb30 0x7f9f6c10d430 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 --2- 172.21.15.162:0/1116176153 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9f58062c90 0x7f9f58065150 unknown :-1 s=CLOSED pgs=2997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 >> 172.21.15.162:0/1116176153 conn(0x7f9f6c06eb90 msgr2=0x7f9f6c0fab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 shutdown_connections 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 -- 172.21.15.162:0/1116176153 wait complete. 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:29.061+0000 7f9f72650700 1 librados: shutdown 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:51:29.069 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:51:29.082 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:29.101 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:29.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:29.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:29.233 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:29.307 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:29.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:29.410 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:29.440 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:29.643 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:29.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:29.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:29.775 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:29.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:29.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:29.846 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:51:29.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:28.506833+0000 mgr.y (mgr.14544) 561 : cluster [DBG] pgmap v999: 204 pgs: 204 active+clean; 5.8 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-13T00:51:29.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:28.615056+0000 mon.a (mon.0) 3285 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:51:29.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:28.624667+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e663: 8 total, 8 up, 4 in 2024-02-13T00:51:29.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:28.922510+0000 mon.a (mon.0) 3287 : audit [DBG] from='client.? 172.21.15.162:0/1116176153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:29.907 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:28.506833+0000 mgr.y (mgr.14544) 561 : cluster [DBG] pgmap v999: 204 pgs: 204 active+clean; 5.8 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-13T00:51:29.907 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:28.615056+0000 mon.a (mon.0) 3285 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:51:29.907 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:28.624667+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e663: 8 total, 8 up, 4 in 2024-02-13T00:51:29.907 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:28.922510+0000 mon.a (mon.0) 3287 : audit [DBG] from='client.? 172.21.15.162:0/1116176153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:29.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:28.506833+0000 mgr.y (mgr.14544) 561 : cluster [DBG] pgmap v999: 204 pgs: 204 active+clean; 5.8 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-02-13T00:51:29.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:28.615056+0000 mon.a (mon.0) 3285 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:51:29.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:28.624667+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e663: 8 total, 8 up, 4 in 2024-02-13T00:51:29.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:28.922510+0000 mon.a (mon.0) 3287 : audit [DBG] from='client.? 172.21.15.162:0/1116176153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:29.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:29.959 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:29.982 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:29.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:30.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:30.115 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:30.136 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:30.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:30.155 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:51:30.155 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "" 2024-02-13T00:51:30.155 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:51:30.163 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:30.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:30.266 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:30.415 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:30.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:30.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:30.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:30.555 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:29.627770+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e664: 8 total, 8 up, 4 in 2024-02-13T00:51:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:29.628533+0000 mon.a (mon.0) 3289 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:30.525581+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:30.525673+0000 mon.a (mon.0) 3291 : cluster [DBG] osdmap e665: 8 total, 8 up, 4 in 2024-02-13T00:51:30.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:30.526825+0000 mon.a (mon.0) 3292 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:30.845 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:29.627770+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e664: 8 total, 8 up, 4 in 2024-02-13T00:51:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:29.628533+0000 mon.a (mon.0) 3289 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:30.525581+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:30.525673+0000 mon.a (mon.0) 3291 : cluster [DBG] osdmap e665: 8 total, 8 up, 4 in 2024-02-13T00:51:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:30.526825+0000 mon.a (mon.0) 3292 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:29.627770+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e664: 8 total, 8 up, 4 in 2024-02-13T00:51:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:29.628533+0000 mon.a (mon.0) 3289 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:31.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:30.525581+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:31.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:30.525673+0000 mon.a (mon.0) 3291 : cluster [DBG] osdmap e665: 8 total, 8 up, 4 in 2024-02-13T00:51:31.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:30.526825+0000 mon.a (mon.0) 3292 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:31.104 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:31.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:31.208 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:31.237 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:31.256 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:31.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:31.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:31.388 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:31.636 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:31.638 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:30.507629+0000 mgr.y (mgr.14544) 562 : cluster [DBG] pgmap v1002: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:31.638 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:31.528616+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:31.638 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:31.528693+0000 mon.a (mon.0) 3294 : cluster [DBG] osdmap e666: 8 total, 8 up, 4 in 2024-02-13T00:51:31.638 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:31.530584+0000 mon.a (mon.0) 3295 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-116"}]: dispatch 2024-02-13T00:51:31.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:31.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:31.740 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:31.772 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:31.776 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:30.507629+0000 mgr.y (mgr.14544) 562 : cluster [DBG] pgmap v1002: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:31.776 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:31.528616+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:31.776 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:31.528693+0000 mon.a (mon.0) 3294 : cluster [DBG] osdmap e666: 8 total, 8 up, 4 in 2024-02-13T00:51:31.776 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:31.530584+0000 mon.a (mon.0) 3295 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-116"}]: dispatch 2024-02-13T00:51:31.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:30.507629+0000 mgr.y (mgr.14544) 562 : cluster [DBG] pgmap v1002: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:31.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:31.528616+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:31.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:31.528693+0000 mon.a (mon.0) 3294 : cluster [DBG] osdmap e666: 8 total, 8 up, 4 in 2024-02-13T00:51:31.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:31.530584+0000 mon.a (mon.0) 3295 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-116"}]: dispatch 2024-02-13T00:51:31.793 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:31.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:31.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:51:31.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:31.926 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.032+0000 7f7449426700 1 -- 172.21.15.162:0/1889053497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7444104580 msgr2=0x7f7444106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.032+0000 7f7449426700 1 --2- 172.21.15.162:0/1889053497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7444104580 0x7f7444106970 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 rx=0x7f743c009be0 tx=0x7f743c004d10).stop 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.032+0000 7f7449426700 1 -- 172.21.15.162:0/1889053497 shutdown_connections 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.032+0000 7f7449426700 1 --2- 172.21.15.162:0/1889053497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7444104580 0x7f7444106970 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.032+0000 7f7449426700 1 --2- 172.21.15.162:0/1889053497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f744406b1b0 0x7f744406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.032+0000 7f7449426700 1 --2- 172.21.15.162:0/1889053497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f744406ba80 0x7f7444104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.032+0000 7f7449426700 1 -- 172.21.15.162:0/1889053497 >> 172.21.15.162:0/1889053497 conn(0x7f74440f38c0 msgr2=0x7f74440f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.032+0000 7f7449426700 1 -- 172.21.15.162:0/1889053497 shutdown_connections 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.032+0000 7f7449426700 1 -- 172.21.15.162:0/1889053497 wait complete. 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7449426700 1 librados: starting msgr at 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7449426700 1 librados: starting objecter 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7449426700 1 Processor -- start 2024-02-13T00:51:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7449426700 1 -- start start 2024-02-13T00:51:32.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7449426700 1 librados: setting wanted keys 2024-02-13T00:51:32.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7449426700 1 librados: calling monclient init 2024-02-13T00:51:32.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7449426700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f744406b1b0 0x7f74441103b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:32.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7449426700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f744406ba80 0x7f74441108a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:32.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7449426700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7444104580 0x7f744410a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:32.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7449426700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744410aa80 con 0x7f7444104580 2024-02-13T00:51:32.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7449426700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744410abf0 con 0x7f744406ba80 2024-02-13T00:51:32.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7449426700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f744410ad60 con 0x7f744406b1b0 2024-02-13T00:51:32.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.033+0000 7f7442ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f744406b1b0 0x7f74441103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:32.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f7442ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f744406b1b0 0x7f74441103b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:49388/0 (socket says 172.21.15.162:49388) 2024-02-13T00:51:32.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f7442ffd700 1 -- 172.21.15.162:0/2099215233 learned_addr learned my addr 172.21.15.162:0/2099215233 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:32.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f74437fe700 1 --2- 172.21.15.162:0/2099215233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7444104580 0x7f744410a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:32.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f74427fc700 1 --2- 172.21.15.162:0/2099215233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f744406ba80 0x7f74441108a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:32.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f7442ffd700 1 -- 172.21.15.162:0/2099215233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f744406ba80 msgr2=0x7f74441108a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:32.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f7442ffd700 1 --2- 172.21.15.162:0/2099215233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f744406ba80 0x7f74441108a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:32.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f7442ffd700 1 -- 172.21.15.162:0/2099215233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7444104580 msgr2=0x7f744410a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:32.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f7442ffd700 1 --2- 172.21.15.162:0/2099215233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7444104580 0x7f744410a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:32.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f7442ffd700 1 -- 172.21.15.162:0/2099215233 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743c0097c0 con 0x7f744406b1b0 2024-02-13T00:51:32.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f74437fe700 1 --2- 172.21.15.162:0/2099215233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7444104580 0x7f744410a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:51:32.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f7442ffd700 1 --2- 172.21.15.162:0/2099215233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f744406b1b0 0x7f74441103b0 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 rx=0x7f743400e970 tx=0x7f743400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f742bfff700 10 client.?.objecter ms_handle_connect 0x7f744406b1b0 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f742bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f742bfff700 1 -- 172.21.15.162:0/2099215233 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434004660 con 0x7f744406b1b0 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f7449426700 1 -- 172.21.15.162:0/2099215233 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744410b040 con 0x7f744406b1b0 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f742bfff700 1 -- 172.21.15.162:0/2099215233 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f743400f040 con 0x7f744406b1b0 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f742bfff700 1 -- 172.21.15.162:0/2099215233 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434014790 con 0x7f744406b1b0 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f7449426700 10 client.38006.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f7449426700 1 -- 172.21.15.162:0/2099215233 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7444117220 con 0x7f744406b1b0 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.034+0000 7f7449426700 1 librados: init done 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.035+0000 7f7449426700 1 -- 172.21.15.162:0/2099215233 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74440475e0 con 0x7f744406b1b0 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.036+0000 7f742bfff700 1 -- 172.21.15.162:0/2099215233 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f74340090d0 con 0x7f744406b1b0 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.036+0000 7f742bfff700 1 --2- 172.21.15.162:0/2099215233 >> v2:172.21.15.162:6800/2866534053 conn(0x7f742c062c90 0x7f742c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.036+0000 7f742bfff700 1 -- 172.21.15.162:0/2099215233 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(666..666 src has 449..666) v4 ==== 8648+0+0 (secure 0 0 0) 0x7f743407cfa0 con 0x7f744406b1b0 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.036+0000 7f742bfff700 10 client.38006.objecter ms_dispatch 0x7f7444001550 osd_map(666..666 src has 449..666) v4 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.036+0000 7f742bfff700 3 client.38006.objecter handle_osd_map got epochs [666,666] > 0 2024-02-13T00:51:32.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.036+0000 7f742bfff700 3 client.38006.objecter handle_osd_map decoding full epoch 666 2024-02-13T00:51:32.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.036+0000 7f74427fc700 1 --2- 172.21.15.162:0/2099215233 >> v2:172.21.15.162:6800/2866534053 conn(0x7f742c062c90 0x7f742c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:32.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.036+0000 7f742bfff700 20 client.38006.objecter dump_active .. 0 homeless 2024-02-13T00:51:32.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.037+0000 7f742bfff700 10 client.38006.objecter ms_handle_connect 0x7f742c062c90 2024-02-13T00:51:32.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.037+0000 7f74427fc700 1 --2- 172.21.15.162:0/2099215233 >> v2:172.21.15.162:6800/2866534053 conn(0x7f742c062c90 0x7f742c065150 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 rx=0x7f744410bd30 tx=0x7f743800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:32.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.041+0000 7f742bfff700 1 -- 172.21.15.162:0/2099215233 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f743404af60 con 0x7f744406b1b0 2024-02-13T00:51:32.118 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:32.135 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:32.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:32.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.231+0000 7f7449426700 1 -- 172.21.15.162:0/2099215233 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f744410ba70 con 0x7f744406b1b0 2024-02-13T00:51:32.240 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:32.266 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:32.389 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:32.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:32.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:32.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:32.520 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:32.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.529+0000 7f742bfff700 1 -- 172.21.15.162:0/2099215233 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v667) v1 ==== 91+0+0 (secure 0 0 0) 0x7f7434019070 con 0x7f744406b1b0 2024-02-13T00:51:32.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 10 librados: watch_flush enter 2024-02-13T00:51:32.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 10 librados: watch_flush exit 2024-02-13T00:51:32.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 20 client.38006.objecter shutdown clearing up homeless session... 2024-02-13T00:51:32.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 10 client.38006.objecter successfully canceled tick 2024-02-13T00:51:32.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 -- 172.21.15.162:0/2099215233 >> v2:172.21.15.162:6800/2866534053 conn(0x7f742c062c90 msgr2=0x7f742c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:32.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 --2- 172.21.15.162:0/2099215233 >> v2:172.21.15.162:6800/2866534053 conn(0x7f742c062c90 0x7f742c065150 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 rx=0x7f744410bd30 tx=0x7f743800b040).stop 2024-02-13T00:51:32.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 -- 172.21.15.162:0/2099215233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f744406b1b0 msgr2=0x7f74441103b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:32.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 --2- 172.21.15.162:0/2099215233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f744406b1b0 0x7f74441103b0 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 rx=0x7f743400e970 tx=0x7f743400b040).stop 2024-02-13T00:51:32.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 -- 172.21.15.162:0/2099215233 shutdown_connections 2024-02-13T00:51:32.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 --2- 172.21.15.162:0/2099215233 >> v2:172.21.15.162:6800/2866534053 conn(0x7f742c062c90 0x7f742c065150 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:32.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 --2- 172.21.15.162:0/2099215233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7444104580 0x7f744410a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:32.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 --2- 172.21.15.162:0/2099215233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f744406b1b0 0x7f74441103b0 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:32.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 --2- 172.21.15.162:0/2099215233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f744406ba80 0x7f74441108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:32.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 -- 172.21.15.162:0/2099215233 >> 172.21.15.162:0/2099215233 conn(0x7f74440f38c0 msgr2=0x7f74440f44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:32.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 -- 172.21.15.162:0/2099215233 shutdown_connections 2024-02-13T00:51:32.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 -- 172.21.15.162:0/2099215233 wait complete. 2024-02-13T00:51:32.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:32.532+0000 7f7449426700 1 librados: shutdown 2024-02-13T00:51:32.538 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:51:32.638 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:32.232526+0000 mon.c (mon.2) 484 : audit [INF] from='client.? 172.21.15.162:0/2099215233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:51:32.638 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:32.232856+0000 mon.a (mon.0) 3296 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:51:32.638 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:32.525375+0000 mon.a (mon.0) 3297 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:32.638 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:32.528710+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-116"}]': finished 2024-02-13T00:51:32.639 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:32.528791+0000 mon.a (mon.0) 3299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:51:32.639 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:32.528845+0000 mon.a (mon.0) 3300 : cluster [DBG] osdmap e667: 8 total, 8 up, 4 in 2024-02-13T00:51:32.639 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:32.530448+0000 mon.a (mon.0) 3301 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-116", "mode": "writeback"}]: dispatch 2024-02-13T00:51:32.654 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:51:32.654 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:51:32.669 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:32.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:32.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:32.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:32.232526+0000 mon.c (mon.2) 484 : audit [INF] from='client.? 172.21.15.162:0/2099215233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:51:32.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:32.232856+0000 mon.a (mon.0) 3296 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:51:32.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:32.525375+0000 mon.a (mon.0) 3297 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:32.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:32.528710+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-116"}]': finished 2024-02-13T00:51:32.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:32.528791+0000 mon.a (mon.0) 3299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:51:32.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:32.528845+0000 mon.a (mon.0) 3300 : cluster [DBG] osdmap e667: 8 total, 8 up, 4 in 2024-02-13T00:51:32.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:32.530448+0000 mon.a (mon.0) 3301 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-116", "mode": "writeback"}]: dispatch 2024-02-13T00:51:32.801 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:32.924 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:32.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:32.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:33.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:33.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:32.232526+0000 mon.c (mon.2) 484 : audit [INF] from='client.? 172.21.15.162:0/2099215233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:51:33.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:32.232856+0000 mon.a (mon.0) 3296 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:51:33.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:32.525375+0000 mon.a (mon.0) 3297 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:33.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:32.528710+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-116"}]': finished 2024-02-13T00:51:33.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:32.528791+0000 mon.a (mon.0) 3299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:51:33.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:32.528845+0000 mon.a (mon.0) 3300 : cluster [DBG] osdmap e667: 8 total, 8 up, 4 in 2024-02-13T00:51:33.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:32.530448+0000 mon.a (mon.0) 3301 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-116", "mode": "writeback"}]: dispatch 2024-02-13T00:51:33.059 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:33.086 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:33.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:33.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:33.218 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:33.270 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:33.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:33.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:33.412 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:33.424 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-13T00:51:33.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:32.508135+0000 mgr.y (mgr.14544) 563 : cluster [DBG] pgmap v1005: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:33.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:33.096748+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.a scrub starts 2024-02-13T00:51:33.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:33.100279+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.a scrub ok 2024-02-13T00:51:33.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:33.526646+0000 mon.a (mon.0) 3302 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:33.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:33.533207+0000 mon.a (mon.0) 3303 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-116", "mode": "writeback"}]': finished 2024-02-13T00:51:33.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:33.533273+0000 mon.a (mon.0) 3304 : cluster [DBG] osdmap e668: 8 total, 8 up, 4 in 2024-02-13T00:51:33.685 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:33.580218+0000 mon.a (mon.0) 3305 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:33.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:32.508135+0000 mgr.y (mgr.14544) 563 : cluster [DBG] pgmap v1005: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:33.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:33.096748+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.a scrub starts 2024-02-13T00:51:33.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:33.100279+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.a scrub ok 2024-02-13T00:51:33.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:33.526646+0000 mon.a (mon.0) 3302 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:33.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:33.533207+0000 mon.a (mon.0) 3303 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-116", "mode": "writeback"}]': finished 2024-02-13T00:51:33.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:33.533273+0000 mon.a (mon.0) 3304 : cluster [DBG] osdmap e668: 8 total, 8 up, 4 in 2024-02-13T00:51:33.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:33.580218+0000 mon.a (mon.0) 3305 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:33.697 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:51:33.761 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:33.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:33.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:32.508135+0000 mgr.y (mgr.14544) 563 : cluster [DBG] pgmap v1005: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:33.096748+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.a scrub starts 2024-02-13T00:51:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:33.100279+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.a scrub ok 2024-02-13T00:51:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:33.526646+0000 mon.a (mon.0) 3302 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:33.533207+0000 mon.a (mon.0) 3303 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-116", "mode": "writeback"}]': finished 2024-02-13T00:51:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:33.533273+0000 mon.a (mon.0) 3304 : cluster [DBG] osdmap e668: 8 total, 8 up, 4 in 2024-02-13T00:51:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:33.580218+0000 mon.a (mon.0) 3305 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:33.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:33.891 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:34.539 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:34.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:34.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:34.678 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:34.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.805+0000 7fa6b8133700 1 -- 172.21.15.162:0/190655572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b00fe580 msgr2=0x7fa6b00f7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.805+0000 7fa6b8133700 1 --2- 172.21.15.162:0/190655572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b00fe580 0x7fa6b00f7db0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 rx=0x7fa6ac004870 tx=0x7fa6ac009cb0).stop 2024-02-13T00:51:34.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.805+0000 7fa6b8133700 1 -- 172.21.15.162:0/190655572 shutdown_connections 2024-02-13T00:51:34.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.805+0000 7fa6b8133700 1 --2- 172.21.15.162:0/190655572 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b00f8490 0x7fa6b0104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.805+0000 7fa6b8133700 1 --2- 172.21.15.162:0/190655572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6b00fdc20 0x7fa6b00fe000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.805+0000 7fa6b8133700 1 --2- 172.21.15.162:0/190655572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b00fe580 0x7fa6b00f7db0 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.805+0000 7fa6b8133700 1 -- 172.21.15.162:0/190655572 >> 172.21.15.162:0/190655572 conn(0x7fa6b00f3850 msgr2=0x7fa6b00f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:34.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.806+0000 7fa6b8133700 1 -- 172.21.15.162:0/190655572 shutdown_connections 2024-02-13T00:51:34.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.806+0000 7fa6b8133700 1 -- 172.21.15.162:0/190655572 wait complete. 2024-02-13T00:51:34.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.806+0000 7fa6b8133700 1 librados: starting msgr at 2024-02-13T00:51:34.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.806+0000 7fa6b8133700 1 librados: starting objecter 2024-02-13T00:51:34.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b8133700 1 Processor -- start 2024-02-13T00:51:34.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b8133700 1 -- start start 2024-02-13T00:51:34.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b8133700 1 librados: setting wanted keys 2024-02-13T00:51:34.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b8133700 1 librados: calling monclient init 2024-02-13T00:51:34.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b8133700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6b00f8490 0x7fa6b010e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:34.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b8133700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b00fdc20 0x7fa6b010e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:34.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b8133700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b00fe580 0x7fa6b01082a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:34.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b8133700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b0108790 con 0x7fa6b00f8490 2024-02-13T00:51:34.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b8133700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b0108900 con 0x7fa6b00fdc20 2024-02-13T00:51:34.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b8133700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b0108ac0 con 0x7fa6b00fe580 2024-02-13T00:51:34.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b5ecf700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6b00f8490 0x7fa6b010e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b5ecf700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6b00f8490 0x7fa6b010e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34712/0 (socket says 172.21.15.162:34712) 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b5ecf700 1 -- 172.21.15.162:0/3002456398 learned_addr learned my addr 172.21.15.162:0/3002456398 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b66d0700 1 --2- 172.21.15.162:0/3002456398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b00fe580 0x7fa6b01082a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b56ce700 1 --2- 172.21.15.162:0/3002456398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b00fdc20 0x7fa6b010e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b5ecf700 1 -- 172.21.15.162:0/3002456398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b00fe580 msgr2=0x7fa6b01082a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b5ecf700 1 --2- 172.21.15.162:0/3002456398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b00fe580 0x7fa6b01082a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b5ecf700 1 -- 172.21.15.162:0/3002456398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b00fdc20 msgr2=0x7fa6b010e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b5ecf700 1 --2- 172.21.15.162:0/3002456398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b00fdc20 0x7fa6b010e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b5ecf700 1 -- 172.21.15.162:0/3002456398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a0005040 con 0x7fa6b00f8490 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.807+0000 7fa6b56ce700 1 --2- 172.21.15.162:0/3002456398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b00fdc20 0x7fa6b010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.808+0000 7fa6b5ecf700 1 --2- 172.21.15.162:0/3002456398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6b00f8490 0x7fa6b010e100 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 rx=0x7fa6a00048f0 tx=0x7fa6a0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.808+0000 7fa6a6ffd700 10 client.?.objecter ms_handle_connect 0x7fa6b00f8490 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.808+0000 7fa6a6ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.808+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0007b00 con 0x7fa6b00f8490 2024-02-13T00:51:34.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.808+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6a000f040 con 0x7fa6b00f8490 2024-02-13T00:51:34.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.808+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac005040 con 0x7fa6b00f8490 2024-02-13T00:51:34.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.808+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0014a00 con 0x7fa6b00f8490 2024-02-13T00:51:34.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.808+0000 7fa6b8133700 10 client.27915.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:34.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.808+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b0109100 con 0x7fa6b00f8490 2024-02-13T00:51:34.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.808+0000 7fa6b8133700 1 librados: init done 2024-02-13T00:51:34.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.808+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa6b00607f0 con 0x7fa6b00f8490 2024-02-13T00:51:34.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.809+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fa6a000e580 con 0x7fa6b00f8490 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.809+0000 7fa6a6ffd700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa69c062c90 0x7fa69c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(669..669 src has 449..669) v4 ==== 8648+0+0 (secure 0 0 0) 0x7fa6a007d330 con 0x7fa6b00f8490 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 osd_map(669..669 src has 449..669) v4 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6a6ffd700 3 client.27915.objecter handle_osd_map got epochs [669,669] > 0 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6a6ffd700 3 client.27915.objecter handle_osd_map decoding full epoch 669 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b56ce700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa69c062c90 0x7fa69c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6a6ffd700 20 client.27915.objecter dump_active .. 0 homeless 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v669) v1 ==== 54+0+16 (secure 0 0 0) 0x7fa6a007d7d0 con 0x7fa6b00f8490 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b56ce700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa69c062c90 0x7fa69c065150 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7fa6ac0043d0 tx=0x7fa6ac00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6a6ffd700 10 client.27915.objecter ms_handle_connect 0x7fa69c062c90 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b8133700 10 client.27915.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 1 2024-02-13T00:51:34.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa6b0028590 0x7fa6b00fe960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b00475e0 osd=0 3 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b00475e0 osd=0 3 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b00475e0 osd=0 3 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 1 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b00475e0 osd=0 3 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 0 1 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b00475e0 osd=0 4 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b8133700 10 client.27915.objecter _send_command 1 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6b01070d0 con 0x7fa6b0028590 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b66d0700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa6b0028590 0x7fa6b00fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6a6ffd700 10 client.27915.objecter ms_handle_connect 0x7fa6b0028590 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.810+0000 7fa6b66d0700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa6b0028590 0x7fa6b00fe960 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.811+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa6b01070d0 con 0x7fa6b0028590 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.811+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.811+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 1 = system:0 2024-02-13T00:51:34.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.811+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b00475e0 osd=0 4 2024-02-13T00:51:34.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.811+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 0 1 2024-02-13T00:51:34.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.825+0000 7fa6b8133700 10 client.27915.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:34.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.825+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 2 2024-02-13T00:51:34.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.825+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b00475e0 osd=0 3 2024-02-13T00:51:34.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.825+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b00475e0 osd=0 3 2024-02-13T00:51:34.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.825+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b00475e0 osd=0 3 2024-02-13T00:51:34.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.825+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 2 2024-02-13T00:51:34.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.825+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b00475e0 osd=0 3 2024-02-13T00:51:34.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.825+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 0 2 2024-02-13T00:51:34.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.825+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b00475e0 osd=0 4 2024-02-13T00:51:34.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.825+0000 7fa6b8133700 10 client.27915.objecter _send_command 2 2024-02-13T00:51:34.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.825+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa6b0109780 con 0x7fa6b0028590 2024-02-13T00:51:34.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa6b0109780 con 0x7fa6b0028590 2024-02-13T00:51:34.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:34.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b00475e0 osd=0 4 2024-02-13T00:51:34.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 0 2 2024-02-13T00:51:34.831 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6b8133700 10 client.27915.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 3 2024-02-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6805/150599780 conn(0x7fa6b0027be0 0x7fa6b0027fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b00fffd0 osd=1 3 2024-02-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b00fffd0 osd=1 3 2024-02-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b00fffd0 osd=1 3 2024-02-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 3 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b00fffd0 osd=1 3 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 1 3 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b00fffd0 osd=1 4 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6b8133700 10 client.27915.objecter _send_command 3 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.827+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6b0100140 con 0x7fa6b0027be0 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.828+0000 7fa6b5ecf700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6805/150599780 conn(0x7fa6b0027be0 0x7fa6b0027fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.828+0000 7fa6a6ffd700 10 client.27915.objecter ms_handle_connect 0x7fa6b0027be0 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.828+0000 7fa6b5ecf700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6805/150599780 conn(0x7fa6b0027be0 0x7fa6b0027fc0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.828+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa6b0100140 con 0x7fa6b0027be0 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.828+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.828+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 3 = system:0 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.828+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b00fffd0 osd=1 4 2024-02-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.828+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 1 3 2024-02-13T00:51:34.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.843+0000 7fa6b8133700 10 client.27915.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:34.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.843+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 4 2024-02-13T00:51:34.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.843+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b00fffd0 osd=1 3 2024-02-13T00:51:34.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.843+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b00fffd0 osd=1 3 2024-02-13T00:51:34.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.843+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b00fffd0 osd=1 3 2024-02-13T00:51:34.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.843+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 4 2024-02-13T00:51:34.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.843+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b00fffd0 osd=1 3 2024-02-13T00:51:34.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.843+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 1 4 2024-02-13T00:51:34.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.843+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b00fffd0 osd=1 4 2024-02-13T00:51:34.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.843+0000 7fa6b8133700 10 client.27915.objecter _send_command 4 2024-02-13T00:51:34.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.843+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa6b0027650 con 0x7fa6b0027be0 2024-02-13T00:51:34.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa6b0027650 con 0x7fa6b0027be0 2024-02-13T00:51:34.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:34.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b00fffd0 osd=1 4 2024-02-13T00:51:34.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 1 4 2024-02-13T00:51:34.893 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:51:34.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b8133700 10 client.27915.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:34.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 5 2024-02-13T00:51:34.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa6b00ffb30 0x7fa6b011c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:34.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b0027a60 osd=2 3 2024-02-13T00:51:34.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b0027a60 osd=2 3 2024-02-13T00:51:34.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b0027a60 osd=2 3 2024-02-13T00:51:34.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 5 2024-02-13T00:51:34.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b0027a60 osd=2 3 2024-02-13T00:51:34.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 2 5 2024-02-13T00:51:34.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b0027a60 osd=2 4 2024-02-13T00:51:34.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b8133700 10 client.27915.objecter _send_command 5 2024-02-13T00:51:34.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6b011ee50 con 0x7fa6b00ffb30 2024-02-13T00:51:34.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b56ce700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa6b00ffb30 0x7fa6b011c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:34.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6a6ffd700 10 client.27915.objecter ms_handle_connect 0x7fa6b00ffb30 2024-02-13T00:51:34.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.845+0000 7fa6b56ce700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa6b00ffb30 0x7fa6b011c170 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:34.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.846+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa6b011ee50 con 0x7fa6b00ffb30 2024-02-13T00:51:34.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.846+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:51:34.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.846+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 5 = system:0 2024-02-13T00:51:34.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.846+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b0027a60 osd=2 4 2024-02-13T00:51:34.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.846+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 2 5 2024-02-13T00:51:34.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.860+0000 7fa6b8133700 10 client.27915.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:34.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.860+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 6 2024-02-13T00:51:34.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.860+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b0027a60 osd=2 3 2024-02-13T00:51:34.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.860+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b0027a60 osd=2 3 2024-02-13T00:51:34.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.860+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b0027a60 osd=2 3 2024-02-13T00:51:34.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.860+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 6 2024-02-13T00:51:34.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.860+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b0027a60 osd=2 3 2024-02-13T00:51:34.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.860+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 2 6 2024-02-13T00:51:34.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.860+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b0027a60 osd=2 4 2024-02-13T00:51:34.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.860+0000 7fa6b8133700 10 client.27915.objecter _send_command 6 2024-02-13T00:51:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.860+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa6b00ff0f0 con 0x7fa6b00ffb30 2024-02-13T00:51:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.861+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa6b00ff0f0 con 0x7fa6b00ffb30 2024-02-13T00:51:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.861+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.861+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.861+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b0027a60 osd=2 4 2024-02-13T00:51:34.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.861+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 2 6 2024-02-13T00:51:34.899 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:51:34.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b8133700 10 client.27915.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:34.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 7 2024-02-13T00:51:34.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6813/604971011 conn(0x7fa6b011d7e0 0x7fa6b011dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:34.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b01197b0 osd=3 3 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b01197b0 osd=3 3 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b01197b0 osd=3 3 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 7 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b01197b0 osd=3 3 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 3 7 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b01197b0 osd=3 4 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b8133700 10 client.27915.objecter _send_command 7 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6b0119ec0 con 0x7fa6b011d7e0 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b66d0700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6813/604971011 conn(0x7fa6b011d7e0 0x7fa6b011dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6a6ffd700 10 client.27915.objecter ms_handle_connect 0x7fa6b011d7e0 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6b66d0700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6813/604971011 conn(0x7fa6b011d7e0 0x7fa6b011dbc0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa6b0119ec0 con 0x7fa6b011d7e0 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 7 = system:0 2024-02-13T00:51:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b01197b0 osd=3 4 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.862+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 3 7 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.877+0000 7fa6b8133700 10 client.27915.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.877+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 8 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.877+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b01197b0 osd=3 3 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.877+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b01197b0 osd=3 3 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.877+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b01197b0 osd=3 3 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.877+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 8 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.877+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b01197b0 osd=3 3 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.877+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 3 8 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.877+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b01197b0 osd=3 4 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.877+0000 7fa6b8133700 10 client.27915.objecter _send_command 8 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.877+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa6b011e880 con 0x7fa6b011d7e0 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa6b011e880 con 0x7fa6b011d7e0 2024-02-13T00:51:34.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:34.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b01197b0 osd=3 4 2024-02-13T00:51:34.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 3 8 2024-02-13T00:51:34.903 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:51:34.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6b8133700 10 client.27915.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:34.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 9 2024-02-13T00:51:34.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa6b0121080 0x7fa6b0123470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:34.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b011e0b0 osd=4 3 2024-02-13T00:51:34.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b011e0b0 osd=4 3 2024-02-13T00:51:34.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b011e0b0 osd=4 3 2024-02-13T00:51:34.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 9 2024-02-13T00:51:34.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b011e0b0 osd=4 3 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 4 9 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b011e0b0 osd=4 4 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6b8133700 10 client.27915.objecter _send_command 9 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.878+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6b011e7a0 con 0x7fa6b0121080 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.879+0000 7fa6b5ecf700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa6b0121080 0x7fa6b0123470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.879+0000 7fa6a6ffd700 10 client.27915.objecter ms_handle_connect 0x7fa6b0121080 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.879+0000 7fa6b5ecf700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa6b0121080 0x7fa6b0123470 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.880+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa6b011e7a0 con 0x7fa6b0121080 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.880+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.880+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 9 = system:0 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.880+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b011e0b0 osd=4 4 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.880+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 4 9 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.894+0000 7fa6b8133700 10 client.27915.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:34.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.894+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 10 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.894+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b011e0b0 osd=4 3 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.894+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b011e0b0 osd=4 3 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.894+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b011e0b0 osd=4 3 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.894+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 10 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.894+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b011e0b0 osd=4 3 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.894+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 4 10 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.894+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b011e0b0 osd=4 4 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.894+0000 7fa6b8133700 10 client.27915.objecter _send_command 10 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.894+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa6b011e7a0 con 0x7fa6b0121080 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa6b011e7a0 con 0x7fa6b0121080 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b011e0b0 osd=4 4 2024-02-13T00:51:34.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 4 10 2024-02-13T00:51:34.907 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:51:34.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b8133700 10 client.27915.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:34.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 11 2024-02-13T00:51:34.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6804/2277917372 conn(0x7fa6b0123e70 0x7fa6b0126300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b011e7a0 osd=5 3 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b011e7a0 osd=5 3 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b011e7a0 osd=5 3 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 11 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b011e7a0 osd=5 3 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 5 11 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b011e7a0 osd=5 4 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b8133700 10 client.27915.objecter _send_command 11 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6b01290a0 con 0x7fa6b0123e70 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b56ce700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6804/2277917372 conn(0x7fa6b0123e70 0x7fa6b0126300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6a6ffd700 10 client.27915.objecter ms_handle_connect 0x7fa6b0123e70 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.896+0000 7fa6b56ce700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6804/2277917372 conn(0x7fa6b0123e70 0x7fa6b0126300 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:34.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.897+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa6b01290a0 con 0x7fa6b0123e70 2024-02-13T00:51:34.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.897+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:51:34.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.897+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 11 = system:0 2024-02-13T00:51:34.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.897+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b011e7a0 osd=5 4 2024-02-13T00:51:34.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.897+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 5 11 2024-02-13T00:51:34.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.911+0000 7fa6b8133700 10 client.27915.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:34.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.911+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 12 2024-02-13T00:51:34.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.911+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b011e7a0 osd=5 3 2024-02-13T00:51:34.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.911+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b011e7a0 osd=5 3 2024-02-13T00:51:34.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.911+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b011e7a0 osd=5 3 2024-02-13T00:51:34.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.911+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 12 2024-02-13T00:51:34.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.911+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b011e7a0 osd=5 3 2024-02-13T00:51:34.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.911+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 5 12 2024-02-13T00:51:34.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.911+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b011e7a0 osd=5 4 2024-02-13T00:51:34.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.911+0000 7fa6b8133700 10 client.27915.objecter _send_command 12 2024-02-13T00:51:34.915 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.911+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa6b01290a0 con 0x7fa6b0123e70 2024-02-13T00:51:34.916 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:51:34.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa6b01290a0 con 0x7fa6b0123e70 2024-02-13T00:51:34.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:34.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b011e7a0 osd=5 4 2024-02-13T00:51:34.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 5 12 2024-02-13T00:51:34.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b8133700 10 client.27915.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:34.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 13 2024-02-13T00:51:34.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa6b01283d0 0x7fa6b0128830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:34.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b01290a0 osd=6 3 2024-02-13T00:51:34.917 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b01290a0 osd=6 3 2024-02-13T00:51:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b01290a0 osd=6 3 2024-02-13T00:51:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 13 2024-02-13T00:51:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b01290a0 osd=6 3 2024-02-13T00:51:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 6 13 2024-02-13T00:51:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b01290a0 osd=6 4 2024-02-13T00:51:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b8133700 10 client.27915.objecter _send_command 13 2024-02-13T00:51:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6b0128ea0 con 0x7fa6b01283d0 2024-02-13T00:51:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.913+0000 7fa6b66d0700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa6b01283d0 0x7fa6b0128830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.914+0000 7fa6a6ffd700 10 client.27915.objecter ms_handle_connect 0x7fa6b01283d0 2024-02-13T00:51:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.914+0000 7fa6b66d0700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa6b01283d0 0x7fa6b0128830 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.914+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa6b0128ea0 con 0x7fa6b01283d0 2024-02-13T00:51:34.918 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.914+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:51:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.914+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 13 = system:0 2024-02-13T00:51:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.914+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b01290a0 osd=6 4 2024-02-13T00:51:34.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.914+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 6 13 2024-02-13T00:51:34.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.928+0000 7fa6b8133700 10 client.27915.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:34.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.928+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 14 2024-02-13T00:51:34.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.928+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b01290a0 osd=6 3 2024-02-13T00:51:34.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.928+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b01290a0 osd=6 3 2024-02-13T00:51:34.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.928+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b01290a0 osd=6 3 2024-02-13T00:51:34.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.928+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 14 2024-02-13T00:51:34.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.928+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b01290a0 osd=6 3 2024-02-13T00:51:34.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.928+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 6 14 2024-02-13T00:51:34.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.928+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b01290a0 osd=6 4 2024-02-13T00:51:34.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.928+0000 7fa6b8133700 10 client.27915.objecter _send_command 14 2024-02-13T00:51:34.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.928+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa6b0128ea0 con 0x7fa6b01283d0 2024-02-13T00:51:34.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa6b0128ea0 con 0x7fa6b01283d0 2024-02-13T00:51:34.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:34.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b01290a0 osd=6 4 2024-02-13T00:51:34.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 6 14 2024-02-13T00:51:34.934 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:51:34.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b8133700 10 client.27915.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:34.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 15 2024-02-13T00:51:34.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa6b012b7e0 0x7fa6b012bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:34.934 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b0128ea0 osd=7 3 2024-02-13T00:51:34.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b0128ea0 osd=7 3 2024-02-13T00:51:34.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b0128ea0 osd=7 3 2024-02-13T00:51:34.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 15 2024-02-13T00:51:34.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b0128ea0 osd=7 3 2024-02-13T00:51:34.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 7 15 2024-02-13T00:51:34.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b0128ea0 osd=7 4 2024-02-13T00:51:34.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b8133700 10 client.27915.objecter _send_command 15 2024-02-13T00:51:34.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6b012c130 con 0x7fa6b012b7e0 2024-02-13T00:51:34.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.930+0000 7fa6b5ecf700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa6b012b7e0 0x7fa6b012bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:34.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.931+0000 7fa6a6ffd700 10 client.27915.objecter ms_handle_connect 0x7fa6b012b7e0 2024-02-13T00:51:34.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.931+0000 7fa6b5ecf700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa6b012b7e0 0x7fa6b012bc40 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:34.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.931+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa6b012c130 con 0x7fa6b012b7e0 2024-02-13T00:51:34.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.931+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:51:34.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.931+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 15 = system:0 2024-02-13T00:51:34.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.931+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b0128ea0 osd=7 4 2024-02-13T00:51:34.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.931+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 7 15 2024-02-13T00:51:34.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.945+0000 7fa6b8133700 10 client.27915.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:51:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.945+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign -1 16 2024-02-13T00:51:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.945+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b0128ea0 osd=7 3 2024-02-13T00:51:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.945+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b0128ea0 osd=7 3 2024-02-13T00:51:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.945+0000 7fa6b8133700 20 client.27915.objecter _get_session s=0x7fa6b0128ea0 osd=7 3 2024-02-13T00:51:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.945+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_remove -1 16 2024-02-13T00:51:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.945+0000 7fa6b8133700 20 client.27915.objecter get_session s=0x7fa6b0128ea0 osd=7 3 2024-02-13T00:51:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.945+0000 7fa6b8133700 15 client.27915.objecter _session_command_op_assign 7 16 2024-02-13T00:51:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.945+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b0128ea0 osd=7 4 2024-02-13T00:51:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.945+0000 7fa6b8133700 10 client.27915.objecter _send_command 16 2024-02-13T00:51:34.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.945+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa6b012c130 con 0x7fa6b012b7e0 2024-02-13T00:51:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6a6ffd700 1 -- 172.21.15.162:0/3002456398 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fa6b012c130 con 0x7fa6b012b7e0 2024-02-13T00:51:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6a6ffd700 10 client.27915.objecter ms_dispatch 0x7fa6b0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:51:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6a6ffd700 10 client.27915.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6a6ffd700 20 client.27915.objecter put_session s=0x7fa6b0128ea0 osd=7 4 2024-02-13T00:51:34.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6a6ffd700 15 client.27915.objecter _session_command_op_remove 7 16 2024-02-13T00:51:34.951 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:51:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 10 librados: watch_flush enter 2024-02-13T00:51:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 10 librados: watch_flush exit 2024-02-13T00:51:34.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 10 client.27915.objecter close_session for osd.0 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa6b0028590 msgr2=0x7fa6b00fe960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6801/2103743338 conn(0x7fa6b0028590 0x7fa6b00fe960 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b00475e0 osd=0 1 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 10 client.27915.objecter close_session for osd.1 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6805/150599780 conn(0x7fa6b0027be0 msgr2=0x7fa6b0027fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6805/150599780 conn(0x7fa6b0027be0 0x7fa6b0027fc0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b00fffd0 osd=1 1 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 10 client.27915.objecter close_session for osd.2 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa6b00ffb30 msgr2=0x7fa6b011c170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6809/3689719376 conn(0x7fa6b00ffb30 0x7fa6b011c170 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b0027a60 osd=2 1 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 10 client.27915.objecter close_session for osd.3 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6813/604971011 conn(0x7fa6b011d7e0 msgr2=0x7fa6b011dbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b66d0700 1 -- 172.21.15.162:0/3002456398 reap_dead start 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6813/604971011 conn(0x7fa6b011d7e0 0x7fa6b011dbc0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b01197b0 osd=3 1 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 10 client.27915.objecter close_session for osd.4 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa6b0121080 msgr2=0x7fa6b0123470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa6b0121080 0x7fa6b0123470 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b011e0b0 osd=4 1 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 10 client.27915.objecter close_session for osd.5 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6804/2277917372 conn(0x7fa6b0123e70 msgr2=0x7fa6b0126300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6804/2277917372 conn(0x7fa6b0123e70 0x7fa6b0126300 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b011e7a0 osd=5 1 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 10 client.27915.objecter close_session for osd.6 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa6b01283d0 msgr2=0x7fa6b0128830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6808/1506640341 conn(0x7fa6b01283d0 0x7fa6b0128830 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b01290a0 osd=6 1 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 10 client.27915.objecter close_session for osd.7 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa6b012b7e0 msgr2=0x7fa6b012bc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.181:6812/1964192772 conn(0x7fa6b012b7e0 0x7fa6b012bc40 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 20 client.27915.objecter put_session s=0x7fa6b0128ea0 osd=7 1 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 20 client.27915.objecter shutdown clearing up homeless session... 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 10 client.27915.objecter successfully canceled tick 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b66d0700 1 -- 172.21.15.162:0/3002456398 reap_dead start 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa69c062c90 msgr2=0x7fa69c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa69c062c90 0x7fa69c065150 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7fa6ac0043d0 tx=0x7fa6ac00b040).stop 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6b00f8490 msgr2=0x7fa6b010e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6b00f8490 0x7fa6b010e100 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 rx=0x7fa6a00048f0 tx=0x7fa6a0009cb0).stop 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 shutdown_connections 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6b00f8490 0x7fa6b010e100 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 --2- 172.21.15.162:0/3002456398 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa69c062c90 0x7fa69c065150 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 >> 172.21.15.162:0/3002456398 conn(0x7fa6b00f3850 msgr2=0x7fa6b00fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 shutdown_connections 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 -- 172.21.15.162:0/3002456398 wait complete. 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:34.948+0000 7fa6b8133700 1 librados: shutdown 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.955 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.956 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.956 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.956 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.956 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.956 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:51:34.956 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:51:35.026 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:35.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:35.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:35.161 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:35.292 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:35.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:35.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:35.426 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:34.124359+0000 osd.6 (osd.6) 205 : cluster [DBG] 28.a scrub starts 2024-02-13T00:51:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:34.127248+0000 osd.6 (osd.6) 206 : cluster [DBG] 28.a scrub ok 2024-02-13T00:51:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:34.183308+0000 osd.0 (osd.0) 251 : cluster [DBG] 28.12 scrub starts 2024-02-13T00:51:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:34.184087+0000 osd.0 (osd.0) 252 : cluster [DBG] 28.12 scrub ok 2024-02-13T00:51:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:34.341596+0000 mon.a (mon.0) 3306 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:34.508543+0000 mgr.y (mgr.14544) 564 : cluster [DBG] pgmap v1008: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:35.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:34.539401+0000 mon.a (mon.0) 3307 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:35.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:34.539465+0000 mon.a (mon.0) 3308 : cluster [DBG] osdmap e669: 8 total, 8 up, 4 in 2024-02-13T00:51:35.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:34.541586+0000 mon.a (mon.0) 3309 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-116"}]: dispatch 2024-02-13T00:51:35.581 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:34.809927+0000 mon.a (mon.0) 3310 : audit [DBG] from='client.? 172.21.15.162:0/3002456398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:35.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:34.124359+0000 osd.6 (osd.6) 205 : cluster [DBG] 28.a scrub starts 2024-02-13T00:51:35.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:34.127248+0000 osd.6 (osd.6) 206 : cluster [DBG] 28.a scrub ok 2024-02-13T00:51:35.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:34.183308+0000 osd.0 (osd.0) 251 : cluster [DBG] 28.12 scrub starts 2024-02-13T00:51:35.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:34.184087+0000 osd.0 (osd.0) 252 : cluster [DBG] 28.12 scrub ok 2024-02-13T00:51:35.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:34.341596+0000 mon.a (mon.0) 3306 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:35.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:34.508543+0000 mgr.y (mgr.14544) 564 : cluster [DBG] pgmap v1008: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:35.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:34.539401+0000 mon.a (mon.0) 3307 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:35.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:34.539465+0000 mon.a (mon.0) 3308 : cluster [DBG] osdmap e669: 8 total, 8 up, 4 in 2024-02-13T00:51:35.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:34.541586+0000 mon.a (mon.0) 3309 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-116"}]: dispatch 2024-02-13T00:51:35.582 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:34.809927+0000 mon.a (mon.0) 3310 : audit [DBG] from='client.? 172.21.15.162:0/3002456398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:35.613 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:51:35.614 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:51:35.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:51:35.614 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [] 2024-02-13T00:51:35.614 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:51:35.635 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:34.124359+0000 osd.6 (osd.6) 205 : cluster [DBG] 28.a scrub starts 2024-02-13T00:51:35.635 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:34.127248+0000 osd.6 (osd.6) 206 : cluster [DBG] 28.a scrub ok 2024-02-13T00:51:35.635 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:34.183308+0000 osd.0 (osd.0) 251 : cluster [DBG] 28.12 scrub starts 2024-02-13T00:51:35.635 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:34.184087+0000 osd.0 (osd.0) 252 : cluster [DBG] 28.12 scrub ok 2024-02-13T00:51:35.635 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:34.341596+0000 mon.a (mon.0) 3306 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:35.635 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:34.508543+0000 mgr.y (mgr.14544) 564 : cluster [DBG] pgmap v1008: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:51:35.635 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:34.539401+0000 mon.a (mon.0) 3307 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:35.635 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:34.539465+0000 mon.a (mon.0) 3308 : cluster [DBG] osdmap e669: 8 total, 8 up, 4 in 2024-02-13T00:51:35.636 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:34.541586+0000 mon.a (mon.0) 3309 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-116"}]: dispatch 2024-02-13T00:51:35.636 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:34.809927+0000 mon.a (mon.0) 3310 : audit [DBG] from='client.? 172.21.15.162:0/3002456398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:35.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:51:35.773 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:35.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:35.852 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6] out_osds: [3, 2, 7, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-13T00:51:35.852 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:51:35.853 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-02-13T00:51:35.853 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd in 7 2024-02-13T00:51:35.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:51:35.903 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:51:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:51:35.906 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:35.985 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:36.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-13T00:51:36.449 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:51:36] "GET /metrics HTTP/1.1" 200 189336 "" "Prometheus/2.33.4" 2024-02-13T00:51:36.449 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:35.189104+0000 osd.0 (osd.0) 253 : cluster [DBG] 28.2 scrub starts 2024-02-13T00:51:36.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:35.190413+0000 osd.0 (osd.0) 254 : cluster [DBG] 28.2 scrub ok 2024-02-13T00:51:36.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:35.502409+0000 osd.1 (osd.1) 279 : cluster [DBG] 28.15 scrub starts 2024-02-13T00:51:36.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:35.505299+0000 osd.1 (osd.1) 280 : cluster [DBG] 28.15 scrub ok 2024-02-13T00:51:36.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:35.523157+0000 mon.a (mon.0) 3311 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:51:36.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:35.523753+0000 mon.a (mon.0) 3312 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:36.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:35.529314+0000 mon.a (mon.0) 3313 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-116"}]': finished 2024-02-13T00:51:36.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:35.529381+0000 mon.a (mon.0) 3314 : cluster [DBG] osdmap e670: 8 total, 8 up, 4 in 2024-02-13T00:51:36.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:35.854790+0000 mgr.y (mgr.14544) 565 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:36.450 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:35.189104+0000 osd.0 (osd.0) 253 : cluster [DBG] 28.2 scrub starts 2024-02-13T00:51:36.450 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:35.190413+0000 osd.0 (osd.0) 254 : cluster [DBG] 28.2 scrub ok 2024-02-13T00:51:36.450 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:35.502409+0000 osd.1 (osd.1) 279 : cluster [DBG] 28.15 scrub starts 2024-02-13T00:51:36.451 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:35.505299+0000 osd.1 (osd.1) 280 : cluster [DBG] 28.15 scrub ok 2024-02-13T00:51:36.451 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:35.523157+0000 mon.a (mon.0) 3311 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:51:36.451 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:35.523753+0000 mon.a (mon.0) 3312 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:36.451 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:35.529314+0000 mon.a (mon.0) 3313 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-116"}]': finished 2024-02-13T00:51:36.451 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:35.529381+0000 mon.a (mon.0) 3314 : cluster [DBG] osdmap e670: 8 total, 8 up, 4 in 2024-02-13T00:51:36.451 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:35.854790+0000 mgr.y (mgr.14544) 565 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:36.456 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:51:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:36.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:51:36.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:36.588 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:36.607 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:36.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:35.189104+0000 osd.0 (osd.0) 253 : cluster [DBG] 28.2 scrub starts 2024-02-13T00:51:36.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:35.190413+0000 osd.0 (osd.0) 254 : cluster [DBG] 28.2 scrub ok 2024-02-13T00:51:36.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:35.502409+0000 osd.1 (osd.1) 279 : cluster [DBG] 28.15 scrub starts 2024-02-13T00:51:36.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:35.505299+0000 osd.1 (osd.1) 280 : cluster [DBG] 28.15 scrub ok 2024-02-13T00:51:36.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:35.523157+0000 mon.a (mon.0) 3311 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:51:36.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:35.523753+0000 mon.a (mon.0) 3312 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:36.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:35.529314+0000 mon.a (mon.0) 3313 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-116"}]': finished 2024-02-13T00:51:36.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:35.529381+0000 mon.a (mon.0) 3314 : cluster [DBG] osdmap e670: 8 total, 8 up, 4 in 2024-02-13T00:51:36.609 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:35.854790+0000 mgr.y (mgr.14544) 565 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:36.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:36.711 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:36.738 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:36.836 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:36.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:36.851 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:36.939 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:36.968 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:36.987 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:36.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:37.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:37.117 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:37.136 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:37.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:37.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:37.268 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:37.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:51:37.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:36.149806+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:51:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:36.152695+0000 osd.0 (osd.0) 256 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:51:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:36.508938+0000 mgr.y (mgr.14544) 566 : cluster [DBG] pgmap v1011: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-13T00:51:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:36.518606+0000 osd.1 (osd.1) 281 : cluster [DBG] 28.1 scrub starts 2024-02-13T00:51:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:36.522204+0000 osd.1 (osd.1) 282 : cluster [DBG] 28.1 scrub ok 2024-02-13T00:51:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:36.538360+0000 mon.a (mon.0) 3315 : cluster [DBG] osdmap e671: 8 total, 8 up, 4 in 2024-02-13T00:51:37.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:36.149806+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:51:37.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:36.152695+0000 osd.0 (osd.0) 256 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:51:37.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:36.508938+0000 mgr.y (mgr.14544) 566 : cluster [DBG] pgmap v1011: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-13T00:51:37.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:36.518606+0000 osd.1 (osd.1) 281 : cluster [DBG] 28.1 scrub starts 2024-02-13T00:51:37.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:36.522204+0000 osd.1 (osd.1) 282 : cluster [DBG] 28.1 scrub ok 2024-02-13T00:51:37.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:36.538360+0000 mon.a (mon.0) 3315 : cluster [DBG] osdmap e671: 8 total, 8 up, 4 in 2024-02-13T00:51:37.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:36.149806+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:51:37.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:36.152695+0000 osd.0 (osd.0) 256 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:51:37.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:36.508938+0000 mgr.y (mgr.14544) 566 : cluster [DBG] pgmap v1011: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-13T00:51:37.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:36.518606+0000 osd.1 (osd.1) 281 : cluster [DBG] 28.1 scrub starts 2024-02-13T00:51:37.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:36.522204+0000 osd.1 (osd.1) 282 : cluster [DBG] 28.1 scrub ok 2024-02-13T00:51:37.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:36.538360+0000 mon.a (mon.0) 3315 : cluster [DBG] osdmap e671: 8 total, 8 up, 4 in 2024-02-13T00:51:38.059 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:38.559 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:37.542195+0000 osd.1 (osd.1) 283 : cluster [DBG] 28.1c scrub starts 2024-02-13T00:51:38.559 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:37.545084+0000 osd.1 (osd.1) 284 : cluster [DBG] 28.1c scrub ok 2024-02-13T00:51:38.559 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:37.551053+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e672: 8 total, 8 up, 4 in 2024-02-13T00:51:38.559 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:37.554568+0000 mon.a (mon.0) 3317 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:38.595 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:38.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:38.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:38.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:38.730 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:38.751 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:38.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:37.542195+0000 osd.1 (osd.1) 283 : cluster [DBG] 28.1c scrub starts 2024-02-13T00:51:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:37.545084+0000 osd.1 (osd.1) 284 : cluster [DBG] 28.1c scrub ok 2024-02-13T00:51:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:37.551053+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e672: 8 total, 8 up, 4 in 2024-02-13T00:51:38.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:37.554568+0000 mon.a (mon.0) 3317 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:38.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:38.881 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:39.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:51:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:51:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:51:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:37.542195+0000 osd.1 (osd.1) 283 : cluster [DBG] 28.1c scrub starts 2024-02-13T00:51:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:37.545084+0000 osd.1 (osd.1) 284 : cluster [DBG] 28.1c scrub ok 2024-02-13T00:51:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:37.551053+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e672: 8 total, 8 up, 4 in 2024-02-13T00:51:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:37.554568+0000 mon.a (mon.0) 3317 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:39.129 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:39.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:39.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:39.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:39.264 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:39.285 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:39.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:39.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:39.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.364+0000 7f775dee1700 1 -- 172.21.15.162:0/3572821097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77580f95e0 msgr2=0x7f7758101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:39.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.364+0000 7f775dee1700 1 --2- 172.21.15.162:0/3572821097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77580f95e0 0x7f7758101d80 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 rx=0x7f774c004870 tx=0x7f774c009cb0).stop 2024-02-13T00:51:39.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.364+0000 7f775dee1700 1 -- 172.21.15.162:0/3572821097 shutdown_connections 2024-02-13T00:51:39.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.364+0000 7f775dee1700 1 --2- 172.21.15.162:0/3572821097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7758102270 0x7f7758104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:39.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.364+0000 7f775dee1700 1 --2- 172.21.15.162:0/3572821097 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77580f8d10 0x7f77580f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:39.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.364+0000 7f775dee1700 1 --2- 172.21.15.162:0/3572821097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77580f95e0 0x7f7758101d80 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:39.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.364+0000 7f775dee1700 1 -- 172.21.15.162:0/3572821097 >> 172.21.15.162:0/3572821097 conn(0x7f775806eb90 msgr2=0x7f77580f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:39.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.364+0000 7f775dee1700 1 -- 172.21.15.162:0/3572821097 shutdown_connections 2024-02-13T00:51:39.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.364+0000 7f775dee1700 1 -- 172.21.15.162:0/3572821097 wait complete. 2024-02-13T00:51:39.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.365+0000 7f775dee1700 1 librados: starting msgr at 2024-02-13T00:51:39.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.365+0000 7f775dee1700 1 librados: starting objecter 2024-02-13T00:51:39.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.365+0000 7f775dee1700 1 Processor -- start 2024-02-13T00:51:39.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.365+0000 7f775dee1700 1 -- start start 2024-02-13T00:51:39.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.365+0000 7f775dee1700 1 librados: setting wanted keys 2024-02-13T00:51:39.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.365+0000 7f775dee1700 1 librados: calling monclient init 2024-02-13T00:51:39.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.365+0000 7f775dee1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77580f8d10 0x7f77580f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:39.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.365+0000 7f775dee1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77580f95e0 0x7f77580f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:39.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.365+0000 7f775dee1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7758102270 0x7f77580f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.365+0000 7f775dee1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77581166d0 con 0x7f77580f95e0 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.365+0000 7f775dee1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7758116840 con 0x7f77580f8d10 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.365+0000 7f775dee1700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7758116a00 con 0x7f7758102270 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7756ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77580f95e0 0x7f77580f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7756ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77580f95e0 0x7f77580f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34730/0 (socket says 172.21.15.162:34730) 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7756ffd700 1 -- 172.21.15.162:0/1787218616 learned_addr learned my addr 172.21.15.162:0/1787218616 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f77577fe700 1 --2- 172.21.15.162:0/1787218616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77580f8d10 0x7f77580f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7757fff700 1 --2- 172.21.15.162:0/1787218616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7758102270 0x7f77580f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7756ffd700 1 -- 172.21.15.162:0/1787218616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7758102270 msgr2=0x7f77580f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7756ffd700 1 --2- 172.21.15.162:0/1787218616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7758102270 0x7f77580f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7756ffd700 1 -- 172.21.15.162:0/1787218616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77580f8d10 msgr2=0x7f77580f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7756ffd700 1 --2- 172.21.15.162:0/1787218616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77580f8d10 0x7f77580f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:39.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7756ffd700 1 -- 172.21.15.162:0/1787218616 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f774c005040 con 0x7f77580f95e0 2024-02-13T00:51:39.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7757fff700 1 --2- 172.21.15.162:0/1787218616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7758102270 0x7f77580f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:51:39.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7756ffd700 1 --2- 172.21.15.162:0/1787218616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77580f95e0 0x7f77580f71b0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7f774c009eb0 tx=0x7f774c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:39.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7754ff9700 10 client.?.objecter ms_handle_connect 0x7f77580f95e0 2024-02-13T00:51:39.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7754ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:39.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7754ff9700 1 -- 172.21.15.162:0/1787218616 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c00e070 con 0x7f77580f95e0 2024-02-13T00:51:39.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7754ff9700 1 -- 172.21.15.162:0/1787218616 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f774c013da0 con 0x7f77580f95e0 2024-02-13T00:51:39.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f775dee1700 1 -- 172.21.15.162:0/1787218616 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7758116c80 con 0x7f77580f95e0 2024-02-13T00:51:39.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f7754ff9700 1 -- 172.21.15.162:0/1787218616 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c019070 con 0x7f77580f95e0 2024-02-13T00:51:39.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f775dee1700 10 client.27921.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:39.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f775dee1700 1 -- 172.21.15.162:0/1787218616 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7758117080 con 0x7f77580f95e0 2024-02-13T00:51:39.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.366+0000 7f775dee1700 1 librados: init done 2024-02-13T00:51:39.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.367+0000 7f775dee1700 1 -- 172.21.15.162:0/1787218616 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77581071b0 con 0x7f77580f95e0 2024-02-13T00:51:39.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.368+0000 7f7754ff9700 1 -- 172.21.15.162:0/1787218616 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f774c01e420 con 0x7f77580f95e0 2024-02-13T00:51:39.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.368+0000 7f7754ff9700 1 --2- 172.21.15.162:0/1787218616 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7740062c90 0x7f7740065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:39.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.368+0000 7f7754ff9700 1 -- 172.21.15.162:0/1787218616 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(673..673 src has 449..673) v4 ==== 8625+0+0 (secure 0 0 0) 0x7f774c07de00 con 0x7f77580f95e0 2024-02-13T00:51:39.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.368+0000 7f7754ff9700 10 client.27921.objecter ms_dispatch 0x7f7758001550 osd_map(673..673 src has 449..673) v4 2024-02-13T00:51:39.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.368+0000 7f7754ff9700 3 client.27921.objecter handle_osd_map got epochs [673,673] > 0 2024-02-13T00:51:39.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.368+0000 7f7754ff9700 3 client.27921.objecter handle_osd_map decoding full epoch 673 2024-02-13T00:51:39.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.368+0000 7f77577fe700 1 --2- 172.21.15.162:0/1787218616 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7740062c90 0x7f7740065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:39.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.368+0000 7f7754ff9700 20 client.27921.objecter dump_active .. 0 homeless 2024-02-13T00:51:39.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.368+0000 7f7754ff9700 10 client.27921.objecter ms_handle_connect 0x7f7740062c90 2024-02-13T00:51:39.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.368+0000 7f77577fe700 1 --2- 172.21.15.162:0/1787218616 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7740062c90 0x7f7740065150 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f77480048b0 tx=0x7f774800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:39.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.373+0000 7f7754ff9700 1 -- 172.21.15.162:0/1787218616 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f774c051110 con 0x7f77580f95e0 2024-02-13T00:51:39.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:39.392 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:39.421 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:39.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:39.576+0000 7f775dee1700 1 -- 172.21.15.162:0/1787218616 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd in", "ids": ["7"]} v 0) v1 -- 0x7f77580f81a0 con 0x7f77580f95e0 2024-02-13T00:51:39.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:38.509266+0000 mgr.y (mgr.14544) 567 : cluster [DBG] pgmap v1014: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:39.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:38.523439+0000 osd.1 (osd.1) 285 : cluster [DBG] 3.0 scrub starts 2024-02-13T00:51:39.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:38.527079+0000 osd.1 (osd.1) 286 : cluster [DBG] 3.0 scrub ok 2024-02-13T00:51:39.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:38.549952+0000 mon.a (mon.0) 3318 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:39.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:38.550036+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e673: 8 total, 8 up, 4 in 2024-02-13T00:51:39.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:38.552368+0000 mon.a (mon.0) 3320 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:39.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:39.551862+0000 mon.a (mon.0) 3321 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:39.680 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:39.551921+0000 mon.a (mon.0) 3322 : cluster [DBG] osdmap e674: 8 total, 8 up, 4 in 2024-02-13T00:51:39.681 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:39.553465+0000 mon.a (mon.0) 3323 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-118"}]: dispatch 2024-02-13T00:51:39.681 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:38.509266+0000 mgr.y (mgr.14544) 567 : cluster [DBG] pgmap v1014: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:39.681 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:38.523439+0000 osd.1 (osd.1) 285 : cluster [DBG] 3.0 scrub starts 2024-02-13T00:51:39.681 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:38.527079+0000 osd.1 (osd.1) 286 : cluster [DBG] 3.0 scrub ok 2024-02-13T00:51:39.681 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:38.549952+0000 mon.a (mon.0) 3318 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:39.681 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:38.550036+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e673: 8 total, 8 up, 4 in 2024-02-13T00:51:39.681 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:38.552368+0000 mon.a (mon.0) 3320 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:39.681 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:39.551862+0000 mon.a (mon.0) 3321 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:39.681 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:39.551921+0000 mon.a (mon.0) 3322 : cluster [DBG] osdmap e674: 8 total, 8 up, 4 in 2024-02-13T00:51:39.681 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:39.553465+0000 mon.a (mon.0) 3323 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-118"}]: dispatch 2024-02-13T00:51:39.681 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:51:39.577+0000 7f5f6486f700 -1 mon.a@0(leader).osd e674 definitely_dead 0 2024-02-13T00:51:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:38.509266+0000 mgr.y (mgr.14544) 567 : cluster [DBG] pgmap v1014: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:38.523439+0000 osd.1 (osd.1) 285 : cluster [DBG] 3.0 scrub starts 2024-02-13T00:51:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:38.527079+0000 osd.1 (osd.1) 286 : cluster [DBG] 3.0 scrub ok 2024-02-13T00:51:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:38.549952+0000 mon.a (mon.0) 3318 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:38.550036+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e673: 8 total, 8 up, 4 in 2024-02-13T00:51:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:38.552368+0000 mon.a (mon.0) 3320 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:39.551862+0000 mon.a (mon.0) 3321 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:39.551921+0000 mon.a (mon.0) 3322 : cluster [DBG] osdmap e674: 8 total, 8 up, 4 in 2024-02-13T00:51:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:39.553465+0000 mon.a (mon.0) 3323 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-118"}]: dispatch 2024-02-13T00:51:40.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.529+0000 7f7754ff9700 1 -- 172.21.15.162:0/1787218616 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["7"]}]=0 marked in osd.7. v675) v1 ==== 85+0+0 (secure 0 0 0) 0x7f774c050d00 con 0x7f77580f95e0 2024-02-13T00:51:40.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.534+0000 7f775dee1700 10 librados: watch_flush enter 2024-02-13T00:51:40.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.536+0000 7f775dee1700 10 librados: watch_flush exit 2024-02-13T00:51:40.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.536+0000 7f775dee1700 20 client.27921.objecter shutdown clearing up homeless session... 2024-02-13T00:51:40.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.536+0000 7f775dee1700 10 client.27921.objecter successfully canceled tick 2024-02-13T00:51:40.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.537+0000 7f775dee1700 1 -- 172.21.15.162:0/1787218616 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7740062c90 msgr2=0x7f7740065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:40.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.537+0000 7f775dee1700 1 --2- 172.21.15.162:0/1787218616 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7740062c90 0x7f7740065150 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f77480048b0 tx=0x7f774800b040).stop 2024-02-13T00:51:40.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.537+0000 7f775dee1700 1 -- 172.21.15.162:0/1787218616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77580f95e0 msgr2=0x7f77580f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:40.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.537+0000 7f775dee1700 1 --2- 172.21.15.162:0/1787218616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77580f95e0 0x7f77580f71b0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7f774c009eb0 tx=0x7f774c00b040).stop 2024-02-13T00:51:40.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.537+0000 7f775dee1700 1 -- 172.21.15.162:0/1787218616 shutdown_connections 2024-02-13T00:51:40.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.537+0000 7f775dee1700 1 --2- 172.21.15.162:0/1787218616 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7740062c90 0x7f7740065150 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:40.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.537+0000 7f775dee1700 1 --2- 172.21.15.162:0/1787218616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7758102270 0x7f77580f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:40.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.537+0000 7f775dee1700 1 --2- 172.21.15.162:0/1787218616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77580f8d10 0x7f77580f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:40.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.537+0000 7f775dee1700 1 --2- 172.21.15.162:0/1787218616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77580f95e0 0x7f77580f71b0 secure :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 rx=0x7f774c009eb0 tx=0x7f774c00b040).stop 2024-02-13T00:51:40.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.537+0000 7f775dee1700 1 -- 172.21.15.162:0/1787218616 >> 172.21.15.162:0/1787218616 conn(0x7f775806eb90 msgr2=0x7f7758102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:40.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.540+0000 7f775dee1700 1 -- 172.21.15.162:0/1787218616 shutdown_connections 2024-02-13T00:51:40.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.540+0000 7f775dee1700 1 -- 172.21.15.162:0/1787218616 wait complete. 2024-02-13T00:51:40.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:40.540+0000 7f775dee1700 1 librados: shutdown 2024-02-13T00:51:40.555 INFO:teuthology.orchestra.run.smithi162.stderr:marked in osd.7. 2024-02-13T00:51:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:38.995052+0000 osd.6 (osd.6) 207 : cluster [DBG] 6.f scrub starts 2024-02-13T00:51:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:38.996282+0000 osd.6 (osd.6) 208 : cluster [DBG] 6.f scrub ok 2024-02-13T00:51:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:39.568437+0000 osd.1 (osd.1) 287 : cluster [DBG] 28.7 scrub starts 2024-02-13T00:51:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:39.570598+0000 osd.1 (osd.1) 288 : cluster [DBG] 28.7 scrub ok 2024-02-13T00:51:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:39.578933+0000 mon.a (mon.0) 3324 : audit [DBG] from='client.? 172.21.15.162:0/1787218616' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-13T00:51:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:40.530544+0000 mon.a (mon.0) 3325 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-118"}]': finished 2024-02-13T00:51:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:40.530624+0000 mon.a (mon.0) 3326 : audit [INF] from='client.? 172.21.15.162:0/1787218616' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-13T00:51:40.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:40.530667+0000 mon.a (mon.0) 3327 : cluster [DBG] osdmap e675: 8 total, 8 up, 5 in 2024-02-13T00:51:40.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:40.533222+0000 mon.a (mon.0) 3328 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-118", "mode": "writeback"}]: dispatch 2024-02-13T00:51:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:38.995052+0000 osd.6 (osd.6) 207 : cluster [DBG] 6.f scrub starts 2024-02-13T00:51:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:38.996282+0000 osd.6 (osd.6) 208 : cluster [DBG] 6.f scrub ok 2024-02-13T00:51:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:39.568437+0000 osd.1 (osd.1) 287 : cluster [DBG] 28.7 scrub starts 2024-02-13T00:51:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:39.570598+0000 osd.1 (osd.1) 288 : cluster [DBG] 28.7 scrub ok 2024-02-13T00:51:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:39.578933+0000 mon.a (mon.0) 3324 : audit [DBG] from='client.? 172.21.15.162:0/1787218616' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-13T00:51:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:40.530544+0000 mon.a (mon.0) 3325 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-118"}]': finished 2024-02-13T00:51:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:40.530624+0000 mon.a (mon.0) 3326 : audit [INF] from='client.? 172.21.15.162:0/1787218616' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-13T00:51:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:40.530667+0000 mon.a (mon.0) 3327 : cluster [DBG] osdmap e675: 8 total, 8 up, 5 in 2024-02-13T00:51:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:40.533222+0000 mon.a (mon.0) 3328 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-118", "mode": "writeback"}]: dispatch 2024-02-13T00:51:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:38.995052+0000 osd.6 (osd.6) 207 : cluster [DBG] 6.f scrub starts 2024-02-13T00:51:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:38.996282+0000 osd.6 (osd.6) 208 : cluster [DBG] 6.f scrub ok 2024-02-13T00:51:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:39.568437+0000 osd.1 (osd.1) 287 : cluster [DBG] 28.7 scrub starts 2024-02-13T00:51:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:39.570598+0000 osd.1 (osd.1) 288 : cluster [DBG] 28.7 scrub ok 2024-02-13T00:51:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:39.578933+0000 mon.a (mon.0) 3324 : audit [DBG] from='client.? 172.21.15.162:0/1787218616' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-13T00:51:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:40.530544+0000 mon.a (mon.0) 3325 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-118"}]': finished 2024-02-13T00:51:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:40.530624+0000 mon.a (mon.0) 3326 : audit [INF] from='client.? 172.21.15.162:0/1787218616' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-13T00:51:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:40.530667+0000 mon.a (mon.0) 3327 : cluster [DBG] osdmap e675: 8 total, 8 up, 5 in 2024-02-13T00:51:41.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:40.533222+0000 mon.a (mon.0) 3328 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-118", "mode": "writeback"}]: dispatch 2024-02-13T00:51:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:39.960709+0000 osd.6 (osd.6) 209 : cluster [DBG] 6.28 scrub starts 2024-02-13T00:51:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:39.963621+0000 osd.6 (osd.6) 210 : cluster [DBG] 6.28 scrub ok 2024-02-13T00:51:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:40.510086+0000 mgr.y (mgr.14544) 568 : cluster [DBG] pgmap v1017: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-13T00:51:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:41.528025+0000 mon.a (mon.0) 3329 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:41.532314+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-118", "mode": "writeback"}]': finished 2024-02-13T00:51:41.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:41.532469+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e676: 8 total, 8 up, 5 in 2024-02-13T00:51:41.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:39.960709+0000 osd.6 (osd.6) 209 : cluster [DBG] 6.28 scrub starts 2024-02-13T00:51:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:39.963621+0000 osd.6 (osd.6) 210 : cluster [DBG] 6.28 scrub ok 2024-02-13T00:51:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:40.510086+0000 mgr.y (mgr.14544) 568 : cluster [DBG] pgmap v1017: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-13T00:51:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:41.528025+0000 mon.a (mon.0) 3329 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:41.532314+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-118", "mode": "writeback"}]': finished 2024-02-13T00:51:41.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:41.532469+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e676: 8 total, 8 up, 5 in 2024-02-13T00:51:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:39.960709+0000 osd.6 (osd.6) 209 : cluster [DBG] 6.28 scrub starts 2024-02-13T00:51:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:39.963621+0000 osd.6 (osd.6) 210 : cluster [DBG] 6.28 scrub ok 2024-02-13T00:51:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:40.510086+0000 mgr.y (mgr.14544) 568 : cluster [DBG] pgmap v1017: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-13T00:51:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:41.528025+0000 mon.a (mon.0) 3329 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:41.532314+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-118", "mode": "writeback"}]': finished 2024-02-13T00:51:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:41.532469+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e676: 8 total, 8 up, 5 in 2024-02-13T00:51:42.064 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-02-13T00:51:42.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.651+0000 7f7780a4e700 1 -- 172.21.15.162:0/4076515042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f777c0fdc10 msgr2=0x7f777c0fdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:42.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.651+0000 7f7780a4e700 1 --2- 172.21.15.162:0/4076515042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f777c0fdc10 0x7f777c0fdff0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 rx=0x7f776c004870 tx=0x7f776c009cb0).stop 2024-02-13T00:51:42.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.651+0000 7f7780a4e700 1 -- 172.21.15.162:0/4076515042 shutdown_connections 2024-02-13T00:51:42.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.651+0000 7f7780a4e700 1 --2- 172.21.15.162:0/4076515042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f777c0f8480 0x7f777c104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:42.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.651+0000 7f7780a4e700 1 --2- 172.21.15.162:0/4076515042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f777c0fdc10 0x7f777c0fdff0 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:42.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.651+0000 7f7780a4e700 1 --2- 172.21.15.162:0/4076515042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f777c0fe570 0x7f777c0f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:42.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.651+0000 7f7780a4e700 1 -- 172.21.15.162:0/4076515042 >> 172.21.15.162:0/4076515042 conn(0x7f777c0f3800 msgr2=0x7f777c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.651+0000 7f7780a4e700 1 -- 172.21.15.162:0/4076515042 shutdown_connections 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.651+0000 7f7780a4e700 1 -- 172.21.15.162:0/4076515042 wait complete. 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.652+0000 7f7780a4e700 1 librados: starting msgr at 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.652+0000 7f7780a4e700 1 librados: starting objecter 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.652+0000 7f7780a4e700 1 Processor -- start 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.652+0000 7f7780a4e700 1 -- start start 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.652+0000 7f7780a4e700 1 librados: setting wanted keys 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.652+0000 7f7780a4e700 1 librados: calling monclient init 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7780a4e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f777c0f8480 0x7f777c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7780a4e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f777c0fdc10 0x7f777c06b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7780a4e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f777c0fe570 0x7f777c06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7780a4e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c116630 con 0x7f777c0f8480 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7780a4e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c1167a0 con 0x7f777c0fdc10 2024-02-13T00:51:42.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7780a4e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777c116960 con 0x7f777c0fe570 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7779d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f777c0fdc10 0x7f777c06b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7779d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f777c0fdc10 0x7f777c06b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:48208/0 (socket says 172.21.15.162:48208) 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7779d9b700 1 -- 172.21.15.162:0/2721104158 learned_addr learned my addr 172.21.15.162:0/2721104158 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f777ad9d700 1 --2- 172.21.15.162:0/2721104158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f777c0fe570 0x7f777c06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f777a59c700 1 --2- 172.21.15.162:0/2721104158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f777c0f8480 0x7f777c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7779d9b700 1 -- 172.21.15.162:0/2721104158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f777c0fe570 msgr2=0x7f777c06bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7779d9b700 1 --2- 172.21.15.162:0/2721104158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f777c0fe570 0x7f777c06bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7779d9b700 1 -- 172.21.15.162:0/2721104158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f777c0f8480 msgr2=0x7f777c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7779d9b700 1 --2- 172.21.15.162:0/2721104158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f777c0f8480 0x7f777c06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7779d9b700 1 -- 172.21.15.162:0/2721104158 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77740097c0 con 0x7f777c0fdc10 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7779d9b700 1 --2- 172.21.15.162:0/2721104158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f777c0fdc10 0x7f777c06b660 secure :-1 s=READY pgs=4861 cs=0 l=1 rev1=1 rx=0x7f77700048f0 tx=0x7f7770009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f776b7fe700 10 client.?.objecter ms_handle_connect 0x7f777c0fdc10 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f776b7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f776b7fe700 1 -- 172.21.15.162:0/2721104158 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770007b00 con 0x7f777c0fdc10 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7780a4e700 1 -- 172.21.15.162:0/2721104158 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7770005040 con 0x7f777c0fdc10 2024-02-13T00:51:42.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f776b7fe700 1 -- 172.21.15.162:0/2721104158 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f777000f040 con 0x7f777c0fdc10 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f776b7fe700 1 -- 172.21.15.162:0/2721104158 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770014990 con 0x7f777c0fdc10 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7780a4e700 10 client.44926.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7780a4e700 1 -- 172.21.15.162:0/2721104158 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776c005040 con 0x7f777c0fdc10 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.653+0000 7f7780a4e700 1 librados: init done 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.654+0000 7f7780a4e700 1 -- 172.21.15.162:0/2721104158 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777c05fb20 con 0x7f777c0fdc10 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.655+0000 7f776b7fe700 1 -- 172.21.15.162:0/2721104158 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f7770010500 con 0x7f777c0fdc10 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.655+0000 7f776b7fe700 1 --2- 172.21.15.162:0/2721104158 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7764062c90 0x7f7764065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.655+0000 7f776b7fe700 1 -- 172.21.15.162:0/2721104158 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(676..676 src has 449..676) v4 ==== 8690+0+0 (secure 0 0 0) 0x7f777007cf30 con 0x7f777c0fdc10 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.655+0000 7f776b7fe700 10 client.44926.objecter ms_dispatch 0x7f777c001550 osd_map(676..676 src has 449..676) v4 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.655+0000 7f776b7fe700 3 client.44926.objecter handle_osd_map got epochs [676,676] > 0 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.655+0000 7f776b7fe700 3 client.44926.objecter handle_osd_map decoding full epoch 676 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.655+0000 7f777a59c700 1 --2- 172.21.15.162:0/2721104158 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7764062c90 0x7f7764065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.655+0000 7f776b7fe700 20 client.44926.objecter dump_active .. 0 homeless 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.655+0000 7f776b7fe700 10 client.44926.objecter ms_handle_connect 0x7f7764062c90 2024-02-13T00:51:42.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.655+0000 7f777a59c700 1 --2- 172.21.15.162:0/2721104158 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7764062c90 0x7f7764065150 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 rx=0x7f776c0043d0 tx=0x7f776c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:42.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.660+0000 7f776b7fe700 1 -- 172.21.15.162:0/2721104158 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7770049f60 con 0x7f777c0fdc10 2024-02-13T00:51:42.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:41.839+0000 7f7780a4e700 1 -- 172.21.15.162:0/2721104158 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f777c06c650 con 0x7f777c0fdc10 2024-02-13T00:51:42.173 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:42.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:42.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:42.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:51:42.304 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:42.322 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:51:42.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:42.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:51:42.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:42.456 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:42.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.534+0000 7f776b7fe700 1 -- 172.21.15.162:0/2721104158 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v677) v1 ==== 101+0+0 (secure 0 0 0) 0x7f77700501f0 con 0x7f777c0fdc10 2024-02-13T00:51:42.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.537+0000 7f7780a4e700 10 librados: watch_flush enter 2024-02-13T00:51:42.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.538+0000 7f7780a4e700 10 librados: watch_flush exit 2024-02-13T00:51:42.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.538+0000 7f7780a4e700 20 client.44926.objecter shutdown clearing up homeless session... 2024-02-13T00:51:42.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.538+0000 7f7780a4e700 10 client.44926.objecter successfully canceled tick 2024-02-13T00:51:42.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 -- 172.21.15.162:0/2721104158 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7764062c90 msgr2=0x7f7764065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:42.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 --2- 172.21.15.162:0/2721104158 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7764062c90 0x7f7764065150 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 rx=0x7f776c0043d0 tx=0x7f776c00b040).stop 2024-02-13T00:51:42.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 -- 172.21.15.162:0/2721104158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f777c0fdc10 msgr2=0x7f777c06b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:42.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 --2- 172.21.15.162:0/2721104158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f777c0fdc10 0x7f777c06b660 secure :-1 s=READY pgs=4861 cs=0 l=1 rev1=1 rx=0x7f77700048f0 tx=0x7f7770009cb0).stop 2024-02-13T00:51:42.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 -- 172.21.15.162:0/2721104158 shutdown_connections 2024-02-13T00:51:42.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 --2- 172.21.15.162:0/2721104158 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7764062c90 0x7f7764065150 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:42.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 --2- 172.21.15.162:0/2721104158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f777c0fe570 0x7f777c06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:42.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 --2- 172.21.15.162:0/2721104158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f777c0f8480 0x7f777c06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:42.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 --2- 172.21.15.162:0/2721104158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f777c0fdc10 0x7f777c06b660 unknown :-1 s=CLOSED pgs=4861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:42.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 -- 172.21.15.162:0/2721104158 >> 172.21.15.162:0/2721104158 conn(0x7f777c0f3800 msgr2=0x7f777c0fb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:42.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 -- 172.21.15.162:0/2721104158 shutdown_connections 2024-02-13T00:51:42.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 -- 172.21.15.162:0/2721104158 wait complete. 2024-02-13T00:51:42.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:42.539+0000 7f7780a4e700 1 librados: shutdown 2024-02-13T00:51:42.545 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:51:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:40.924502+0000 osd.6 (osd.6) 211 : cluster [DBG] 6.25 scrub starts 2024-02-13T00:51:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:40.927450+0000 osd.6 (osd.6) 212 : cluster [DBG] 6.25 scrub ok 2024-02-13T00:51:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:41.841377+0000 mon.b (mon.1) 681 : audit [INF] from='client.? 172.21.15.162:0/2721104158' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:41.841710+0000 mon.a (mon.0) 3332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:42.529567+0000 mon.a (mon.0) 3333 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:51:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:42.534644+0000 mon.a (mon.0) 3334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:51:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:42.534708+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e677: 8 total, 8 up, 5 in 2024-02-13T00:51:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:42.538132+0000 osd.0 (osd.0) 257 : cluster [DBG] 5.d starting backfill to osd.7 from (0'0,0'0] MAX to 511'13 2024-02-13T00:51:42.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:42.538143+0000 osd.0 (osd.0) 258 : cluster [DBG] 5.e starting backfill to osd.7 from (0'0,0'0] MAX to 507'13 2024-02-13T00:51:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:40.924502+0000 osd.6 (osd.6) 211 : cluster [DBG] 6.25 scrub starts 2024-02-13T00:51:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:40.927450+0000 osd.6 (osd.6) 212 : cluster [DBG] 6.25 scrub ok 2024-02-13T00:51:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:41.841377+0000 mon.b (mon.1) 681 : audit [INF] from='client.? 172.21.15.162:0/2721104158' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:41.841710+0000 mon.a (mon.0) 3332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:42.529567+0000 mon.a (mon.0) 3333 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:51:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:42.534644+0000 mon.a (mon.0) 3334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:51:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:42.534708+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e677: 8 total, 8 up, 5 in 2024-02-13T00:51:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:42.538132+0000 osd.0 (osd.0) 257 : cluster [DBG] 5.d starting backfill to osd.7 from (0'0,0'0] MAX to 511'13 2024-02-13T00:51:42.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:42.538143+0000 osd.0 (osd.0) 258 : cluster [DBG] 5.e starting backfill to osd.7 from (0'0,0'0] MAX to 507'13 2024-02-13T00:51:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:40.924502+0000 osd.6 (osd.6) 211 : cluster [DBG] 6.25 scrub starts 2024-02-13T00:51:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:40.927450+0000 osd.6 (osd.6) 212 : cluster [DBG] 6.25 scrub ok 2024-02-13T00:51:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:41.841377+0000 mon.b (mon.1) 681 : audit [INF] from='client.? 172.21.15.162:0/2721104158' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:41.841710+0000 mon.a (mon.0) 3332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:51:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:42.529567+0000 mon.a (mon.0) 3333 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:51:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:42.534644+0000 mon.a (mon.0) 3334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:51:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:42.534708+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e677: 8 total, 8 up, 5 in 2024-02-13T00:51:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:42.538132+0000 osd.0 (osd.0) 257 : cluster [DBG] 5.d starting backfill to osd.7 from (0'0,0'0] MAX to 511'13 2024-02-13T00:51:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:42.538143+0000 osd.0 (osd.0) 258 : cluster [DBG] 5.e starting backfill to osd.7 from (0'0,0'0] MAX to 507'13 2024-02-13T00:51:43.412 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:43.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:43.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:43.422 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:51:43.422 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:51:43.423 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-13T00:51:43.423 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:51:43.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:43.541 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:43.559 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:43.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:43.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:43.689 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:43.707 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:43.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:43.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:42.510470+0000 mgr.y (mgr.14544) 569 : cluster [DBG] pgmap v1020: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:51:43.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:42.536469+0000 osd.6 (osd.6) 213 : cluster [DBG] 5.9 starting backfill to osd.7 from (0'0,0'0] MAX to 534'16 2024-02-13T00:51:43.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:43.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:42.510470+0000 mgr.y (mgr.14544) 569 : cluster [DBG] pgmap v1020: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:51:43.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:42.536469+0000 osd.6 (osd.6) 213 : cluster [DBG] 5.9 starting backfill to osd.7 from (0'0,0'0] MAX to 534'16 2024-02-13T00:51:43.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:42.510470+0000 mgr.y (mgr.14544) 569 : cluster [DBG] pgmap v1020: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:51:43.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:42.536469+0000 osd.6 (osd.6) 213 : cluster [DBG] 5.9 starting backfill to osd.7 from (0'0,0'0] MAX to 534'16 2024-02-13T00:51:43.836 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:43.854 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:43.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:43.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:43.986 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:44.014 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:44.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:44.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:44.145 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:44.164 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:44.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:44.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:44.267 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:44.292 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:44.311 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:44.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:44.414 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:44.440 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:44.461 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:44.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:44.565 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:51:44.590 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:44.614 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:51:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:44.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:51:44.718 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:44.744 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:44.762 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:44.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:44.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:44.894 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:45.149 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:45.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:45.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:45.279 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:45.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:44.510850+0000 mgr.y (mgr.14544) 570 : cluster [DBG] pgmap v1022: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:51:45.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:45.532526+0000 mon.a (mon.0) 3336 : cluster [DBG] osdmap e678: 8 total, 8 up, 5 in 2024-02-13T00:51:45.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:44.510850+0000 mgr.y (mgr.14544) 570 : cluster [DBG] pgmap v1022: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:51:45.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:45.532526+0000 mon.a (mon.0) 3336 : cluster [DBG] osdmap e678: 8 total, 8 up, 5 in 2024-02-13T00:51:45.868 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:44.510850+0000 mgr.y (mgr.14544) 570 : cluster [DBG] pgmap v1022: 204 pgs: 204 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:51:45.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:45.532526+0000 mon.a (mon.0) 3336 : cluster [DBG] osdmap e678: 8 total, 8 up, 5 in 2024-02-13T00:51:45.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-13T00:51:46.158 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:46.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:46.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:46.288 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:51:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:51:46.290 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:46.309 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:46.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:46.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:46.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:46.442 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 -- 172.21.15.162:0/806252382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf780f95e0 msgr2=0x7fcf78101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 --2- 172.21.15.162:0/806252382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf780f95e0 0x7fcf78101d80 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 rx=0x7fcf64004870 tx=0x7fcf64009cb0).stop 2024-02-13T00:51:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 -- 172.21.15.162:0/806252382 shutdown_connections 2024-02-13T00:51:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 --2- 172.21.15.162:0/806252382 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcf78102270 0x7fcf78104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 --2- 172.21.15.162:0/806252382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf780f8d10 0x7fcf780f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 --2- 172.21.15.162:0/806252382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf780f95e0 0x7fcf78101d80 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 -- 172.21.15.162:0/806252382 >> 172.21.15.162:0/806252382 conn(0x7fcf7806eb90 msgr2=0x7fcf780f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 -- 172.21.15.162:0/806252382 shutdown_connections 2024-02-13T00:51:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 -- 172.21.15.162:0/806252382 wait complete. 2024-02-13T00:51:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 librados: starting msgr at 2024-02-13T00:51:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 librados: starting objecter 2024-02-13T00:51:46.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 Processor -- start 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 -- start start 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 librados: setting wanted keys 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 librados: calling monclient init 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf780f8d10 0x7fcf780f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcf780f95e0 0x7fcf780f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf78102270 0x7fcf780f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.435+0000 7fcf7f155700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf781166d0 con 0x7fcf780f8d10 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7f155700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf78116840 con 0x7fcf78102270 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7f155700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf78116a00 con 0x7fcf780f95e0 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf77fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcf780f95e0 0x7fcf780f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7cef1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf780f8d10 0x7fcf780f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7cef1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf780f8d10 0x7fcf780f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:36188/0 (socket says 172.21.15.162:36188) 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf77fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcf780f95e0 0x7fcf780f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:46808/0 (socket says 172.21.15.162:46808) 2024-02-13T00:51:46.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7cef1700 1 -- 172.21.15.162:0/2300331042 learned_addr learned my addr 172.21.15.162:0/2300331042 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7d6f2700 1 --2- 172.21.15.162:0/2300331042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf78102270 0x7fcf780f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7cef1700 1 -- 172.21.15.162:0/2300331042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcf780f95e0 msgr2=0x7fcf780f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7cef1700 1 --2- 172.21.15.162:0/2300331042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcf780f95e0 0x7fcf780f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7cef1700 1 -- 172.21.15.162:0/2300331042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf78102270 msgr2=0x7fcf780f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7cef1700 1 --2- 172.21.15.162:0/2300331042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf78102270 0x7fcf780f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7cef1700 1 -- 172.21.15.162:0/2300331042 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c005040 con 0x7fcf780f8d10 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf77fff700 1 --2- 172.21.15.162:0/2300331042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcf780f95e0 0x7fcf780f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7cef1700 1 --2- 172.21.15.162:0/2300331042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf780f8d10 0x7fcf780f6cc0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7fcf6c0048f0 tx=0x7fcf6c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf75ffb700 10 client.?.objecter ms_handle_connect 0x7fcf780f8d10 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf75ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c007b00 con 0x7fcf780f8d10 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf700097c0 con 0x7fcf780f8d10 2024-02-13T00:51:46.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf6c00f040 con 0x7fcf780f8d10 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c014990 con 0x7fcf780f8d10 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7f155700 10 client.27933.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf64005040 con 0x7fcf780f8d10 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.436+0000 7fcf7f155700 1 librados: init done 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.437+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fcf780475e0 con 0x7fcf780f8d10 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fcf6c010500 con 0x7fcf780f8d10 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf75ffb700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcf68062c90 0x7fcf68065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(678..678 src has 449..678) v4 ==== 8591+0+0 (secure 0 0 0) 0x7fcf6c07cf20 con 0x7fcf780f8d10 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 osd_map(678..678 src has 449..678) v4 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf75ffb700 3 client.27933.objecter handle_osd_map got epochs [678,678] > 0 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf75ffb700 3 client.27933.objecter handle_osd_map decoding full epoch 678 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf77fff700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcf68062c90 0x7fcf68065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf75ffb700 20 client.27933.objecter dump_active .. 0 homeless 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v678) v1 ==== 54+0+16 (secure 0 0 0) 0x7fcf6c07f5b0 con 0x7fcf780f8d10 2024-02-13T00:51:46.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf75ffb700 10 client.27933.objecter ms_handle_connect 0x7fcf68062c90 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf77fff700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcf68062c90 0x7fcf68065150 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 rx=0x7fcf64004ee0 tx=0x7fcf6400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf7f155700 10 client.27933.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 1 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6801/2103743338 conn(0x7fcf78028590 0x7fcf780fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf781071b0 osd=0 3 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf781071b0 osd=0 3 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf781071b0 osd=0 3 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 1 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf781071b0 osd=0 3 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 0 1 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf781071b0 osd=0 4 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf7f155700 10 client.27933.objecter _send_command 1 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.438+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf780f81a0 con 0x7fcf78028590 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.439+0000 7fcf7d6f2700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6801/2103743338 conn(0x7fcf78028590 0x7fcf780fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:46.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.439+0000 7fcf75ffb700 10 client.27933.objecter ms_handle_connect 0x7fcf78028590 2024-02-13T00:51:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.439+0000 7fcf7d6f2700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6801/2103743338 conn(0x7fcf78028590 0x7fcf780fbd00 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.439+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf780f81a0 con 0x7fcf78028590 2024-02-13T00:51:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.439+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:51:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.439+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 1 = system:0 2024-02-13T00:51:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.439+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf781071b0 osd=0 4 2024-02-13T00:51:46.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.439+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 0 1 2024-02-13T00:51:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.456+0000 7fcf7f155700 10 client.27933.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.456+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 2 2024-02-13T00:51:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.456+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf781071b0 osd=0 3 2024-02-13T00:51:46.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.456+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf781071b0 osd=0 3 2024-02-13T00:51:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.456+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf781071b0 osd=0 3 2024-02-13T00:51:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.456+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 2 2024-02-13T00:51:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.456+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf781071b0 osd=0 3 2024-02-13T00:51:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.456+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 0 2 2024-02-13T00:51:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.456+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf781071b0 osd=0 4 2024-02-13T00:51:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.456+0000 7fcf7f155700 10 client.27933.objecter _send_command 2 2024-02-13T00:51:46.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.456+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcf7805a0e0 con 0x7fcf78028590 2024-02-13T00:51:46.461 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcf7805a0e0 con 0x7fcf78028590 2024-02-13T00:51:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf781071b0 osd=0 4 2024-02-13T00:51:46.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 0 2 2024-02-13T00:51:46.463 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:51:46.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf7f155700 10 client.27933.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:46.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 3 2024-02-13T00:51:46.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6805/150599780 conn(0x7fcf78027be0 0x7fcf78027fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:46.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf780fd370 osd=1 3 2024-02-13T00:51:46.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf780fd370 osd=1 3 2024-02-13T00:51:46.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf780fd370 osd=1 3 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 3 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf780fd370 osd=1 3 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 1 3 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf780fd370 osd=1 4 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf7f155700 10 client.27933.objecter _send_command 3 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.458+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf780fd4e0 con 0x7fcf78027be0 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.459+0000 7fcf7cef1700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6805/150599780 conn(0x7fcf78027be0 0x7fcf78027fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.459+0000 7fcf75ffb700 10 client.27933.objecter ms_handle_connect 0x7fcf78027be0 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.459+0000 7fcf7cef1700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6805/150599780 conn(0x7fcf78027be0 0x7fcf78027fc0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.460+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf780fd4e0 con 0x7fcf78027be0 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.460+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.460+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 3 = system:0 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.460+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf780fd370 osd=1 4 2024-02-13T00:51:46.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.460+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 1 3 2024-02-13T00:51:46.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:46.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:46.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.477+0000 7fcf7f155700 10 client.27933.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:46.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.477+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 4 2024-02-13T00:51:46.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.477+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf780fd370 osd=1 3 2024-02-13T00:51:46.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.477+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf780fd370 osd=1 3 2024-02-13T00:51:46.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.477+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf780fd370 osd=1 3 2024-02-13T00:51:46.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.477+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 4 2024-02-13T00:51:46.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.477+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf780fd370 osd=1 3 2024-02-13T00:51:46.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.477+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 1 4 2024-02-13T00:51:46.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.477+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf780fd370 osd=1 4 2024-02-13T00:51:46.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.477+0000 7fcf7f155700 10 client.27933.objecter _send_command 4 2024-02-13T00:51:46.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.477+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcf78027650 con 0x7fcf78027be0 2024-02-13T00:51:46.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcf78027650 con 0x7fcf78027be0 2024-02-13T00:51:46.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:46.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf780fd370 osd=1 4 2024-02-13T00:51:46.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 1 4 2024-02-13T00:51:46.484 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:51:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf7f155700 10 client.27933.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 5 2024-02-13T00:51:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6809/3689719376 conn(0x7fcf780fced0 0x7fcf7811e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78027a60 osd=2 3 2024-02-13T00:51:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78027a60 osd=2 3 2024-02-13T00:51:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78027a60 osd=2 3 2024-02-13T00:51:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 5 2024-02-13T00:51:46.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf78027a60 osd=2 3 2024-02-13T00:51:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 2 5 2024-02-13T00:51:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78027a60 osd=2 4 2024-02-13T00:51:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf7f155700 10 client.27933.objecter _send_command 5 2024-02-13T00:51:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf78120fc0 con 0x7fcf780fced0 2024-02-13T00:51:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.479+0000 7fcf77fff700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6809/3689719376 conn(0x7fcf780fced0 0x7fcf7811e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.480+0000 7fcf75ffb700 10 client.27933.objecter ms_handle_connect 0x7fcf780fced0 2024-02-13T00:51:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.480+0000 7fcf77fff700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6809/3689719376 conn(0x7fcf780fced0 0x7fcf7811e2a0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.480+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf78120fc0 con 0x7fcf780fced0 2024-02-13T00:51:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.480+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:51:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.480+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 5 = system:0 2024-02-13T00:51:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.480+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf78027a60 osd=2 4 2024-02-13T00:51:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.480+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 2 5 2024-02-13T00:51:46.490 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:46.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.495+0000 7fcf7f155700 10 client.27933.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:46.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.495+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 6 2024-02-13T00:51:46.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.495+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78027a60 osd=2 3 2024-02-13T00:51:46.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.495+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78027a60 osd=2 3 2024-02-13T00:51:46.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.495+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78027a60 osd=2 3 2024-02-13T00:51:46.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.495+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 6 2024-02-13T00:51:46.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.495+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf78027a60 osd=2 3 2024-02-13T00:51:46.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.495+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 2 6 2024-02-13T00:51:46.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.495+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78027a60 osd=2 4 2024-02-13T00:51:46.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.495+0000 7fcf7f155700 10 client.27933.objecter _send_command 6 2024-02-13T00:51:46.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.495+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcf780fc490 con 0x7fcf780fced0 2024-02-13T00:51:46.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcf780fc490 con 0x7fcf780fced0 2024-02-13T00:51:46.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:46.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf78027a60 osd=2 4 2024-02-13T00:51:46.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 2 6 2024-02-13T00:51:46.503 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7f155700 10 client.27933.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 7 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6813/604971011 conn(0x7fcf7811f910 0x7fcf7811fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf7811b8e0 osd=3 3 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf7811b8e0 osd=3 3 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf7811b8e0 osd=3 3 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 7 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf7811b8e0 osd=3 3 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 3 7 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf7811b8e0 osd=3 4 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7f155700 10 client.27933.objecter _send_command 7 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf7811bfd0 con 0x7fcf7811f910 2024-02-13T00:51:46.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7d6f2700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6813/604971011 conn(0x7fcf7811f910 0x7fcf7811fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:46.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf75ffb700 10 client.27933.objecter ms_handle_connect 0x7fcf7811f910 2024-02-13T00:51:46.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.498+0000 7fcf7d6f2700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6813/604971011 conn(0x7fcf7811f910 0x7fcf7811fd50 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:46.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.499+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf7811bfd0 con 0x7fcf7811f910 2024-02-13T00:51:46.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.499+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:51:46.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.499+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 7 = system:0 2024-02-13T00:51:46.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.499+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf7811b8e0 osd=3 4 2024-02-13T00:51:46.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.499+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 3 7 2024-02-13T00:51:46.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.515+0000 7fcf7f155700 10 client.27933.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:46.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.515+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 8 2024-02-13T00:51:46.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.515+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf7811b8e0 osd=3 3 2024-02-13T00:51:46.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.515+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf7811b8e0 osd=3 3 2024-02-13T00:51:46.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.515+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf7811b8e0 osd=3 3 2024-02-13T00:51:46.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.515+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 8 2024-02-13T00:51:46.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.515+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf7811b8e0 osd=3 3 2024-02-13T00:51:46.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.515+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 3 8 2024-02-13T00:51:46.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.515+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf7811b8e0 osd=3 4 2024-02-13T00:51:46.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.515+0000 7fcf7f155700 10 client.27933.objecter _send_command 8 2024-02-13T00:51:46.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.515+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcf78120a10 con 0x7fcf7811f910 2024-02-13T00:51:46.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcf78120a10 con 0x7fcf7811f910 2024-02-13T00:51:46.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:46.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf7811b8e0 osd=3 4 2024-02-13T00:51:46.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 3 8 2024-02-13T00:51:46.521 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:51:46.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf7f155700 10 client.27933.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:46.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 9 2024-02-13T00:51:46.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6800/3344263388 conn(0x7fcf781231f0 0x7fcf781255e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:46.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78120a10 osd=4 3 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78120a10 osd=4 3 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78120a10 osd=4 3 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 9 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf78120a10 osd=4 3 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 4 9 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78120a10 osd=4 4 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf7f155700 10 client.27933.objecter _send_command 9 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.516+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf78120bf0 con 0x7fcf781231f0 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.517+0000 7fcf7cef1700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6800/3344263388 conn(0x7fcf781231f0 0x7fcf781255e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.517+0000 7fcf75ffb700 10 client.27933.objecter ms_handle_connect 0x7fcf781231f0 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.517+0000 7fcf7cef1700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6800/3344263388 conn(0x7fcf781231f0 0x7fcf781255e0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.518+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf78120bf0 con 0x7fcf781231f0 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.518+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.518+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 9 = system:0 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.518+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf78120a10 osd=4 4 2024-02-13T00:51:46.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.518+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 4 9 2024-02-13T00:51:46.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.537+0000 7fcf7f155700 10 client.27933.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:46.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.537+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 10 2024-02-13T00:51:46.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.537+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78120a10 osd=4 3 2024-02-13T00:51:46.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.537+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78120a10 osd=4 3 2024-02-13T00:51:46.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.537+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78120a10 osd=4 3 2024-02-13T00:51:46.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.537+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 10 2024-02-13T00:51:46.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.537+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf78120a10 osd=4 3 2024-02-13T00:51:46.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.537+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 4 10 2024-02-13T00:51:46.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.537+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78120a10 osd=4 4 2024-02-13T00:51:46.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.537+0000 7fcf7f155700 10 client.27933.objecter _send_command 10 2024-02-13T00:51:46.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.537+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcf78120bf0 con 0x7fcf781231f0 2024-02-13T00:51:46.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.540+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcf78120bf0 con 0x7fcf781231f0 2024-02-13T00:51:46.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.540+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:46.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.540+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.540+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf78120a10 osd=4 4 2024-02-13T00:51:46.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.540+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 4 10 2024-02-13T00:51:46.545 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:51:46.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf7f155700 10 client.27933.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:46.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 11 2024-02-13T00:51:46.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6804/2277917372 conn(0x7fcf78125fe0 0x7fcf781283d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:46.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78120bf0 osd=5 3 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78120bf0 osd=5 3 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78120bf0 osd=5 3 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 11 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf78120bf0 osd=5 3 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 5 11 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78120bf0 osd=5 4 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf7f155700 10 client.27933.objecter _send_command 11 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf7812b170 con 0x7fcf78125fe0 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf77fff700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6804/2277917372 conn(0x7fcf78125fe0 0x7fcf781283d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf75ffb700 10 client.27933.objecter ms_handle_connect 0x7fcf78125fe0 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.541+0000 7fcf77fff700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6804/2277917372 conn(0x7fcf78125fe0 0x7fcf781283d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.542+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf7812b170 con 0x7fcf78125fe0 2024-02-13T00:51:46.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.542+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:51:46.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.542+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 11 = system:0 2024-02-13T00:51:46.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.542+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf78120bf0 osd=5 4 2024-02-13T00:51:46.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.542+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 5 11 2024-02-13T00:51:46.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.556+0000 7fcf7f155700 10 client.27933.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:46.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.556+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 12 2024-02-13T00:51:46.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.556+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78120bf0 osd=5 3 2024-02-13T00:51:46.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.556+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78120bf0 osd=5 3 2024-02-13T00:51:46.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.556+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78120bf0 osd=5 3 2024-02-13T00:51:46.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.556+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 12 2024-02-13T00:51:46.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.556+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf78120bf0 osd=5 3 2024-02-13T00:51:46.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.556+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 5 12 2024-02-13T00:51:46.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.556+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78120bf0 osd=5 4 2024-02-13T00:51:46.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.556+0000 7fcf7f155700 10 client.27933.objecter _send_command 12 2024-02-13T00:51:46.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.556+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcf7812b170 con 0x7fcf78125fe0 2024-02-13T00:51:46.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcf7812b170 con 0x7fcf78125fe0 2024-02-13T00:51:46.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:46.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf78120bf0 osd=5 4 2024-02-13T00:51:46.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 5 12 2024-02-13T00:51:46.563 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:51:46.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf7f155700 10 client.27933.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:46.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 13 2024-02-13T00:51:46.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6808/1506640341 conn(0x7fcf7812a4e0 0x7fcf7812a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:46.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf7812b170 osd=6 3 2024-02-13T00:51:46.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf7812b170 osd=6 3 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf7812b170 osd=6 3 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 13 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf7812b170 osd=6 3 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 6 13 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf7812b170 osd=6 4 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf7f155700 10 client.27933.objecter _send_command 13 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.558+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf7812e520 con 0x7fcf7812a4e0 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.559+0000 7fcf7d6f2700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6808/1506640341 conn(0x7fcf7812a4e0 0x7fcf7812a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.559+0000 7fcf7d6f2700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6808/1506640341 conn(0x7fcf7812a4e0 0x7fcf7812a960 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.559+0000 7fcf75ffb700 10 client.27933.objecter ms_handle_connect 0x7fcf7812a4e0 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.560+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf7812e520 con 0x7fcf7812a4e0 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.560+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:51:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.560+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 13 = system:0 2024-02-13T00:51:46.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.560+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf7812b170 osd=6 4 2024-02-13T00:51:46.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.560+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 6 13 2024-02-13T00:51:46.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:46.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:51:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:51:46] "GET /metrics HTTP/1.1" 200 189336 "" "Prometheus/2.33.4" 2024-02-13T00:51:46.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.576+0000 7fcf7f155700 10 client.27933.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:46.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.576+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 14 2024-02-13T00:51:46.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.576+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf7812b170 osd=6 3 2024-02-13T00:51:46.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.576+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf7812b170 osd=6 3 2024-02-13T00:51:46.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.576+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf7812b170 osd=6 3 2024-02-13T00:51:46.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.576+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 14 2024-02-13T00:51:46.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.576+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf7812b170 osd=6 3 2024-02-13T00:51:46.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.576+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 6 14 2024-02-13T00:51:46.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.576+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf7812b170 osd=6 4 2024-02-13T00:51:46.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.576+0000 7fcf7f155700 10 client.27933.objecter _send_command 14 2024-02-13T00:51:46.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.576+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcf78128980 con 0x7fcf7812a4e0 2024-02-13T00:51:46.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcf78128980 con 0x7fcf7812a4e0 2024-02-13T00:51:46.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:46.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf7812b170 osd=6 4 2024-02-13T00:51:46.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 6 14 2024-02-13T00:51:46.583 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7f155700 10 client.27933.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 15 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6812/1964192772 conn(0x7fcf78129000 0x7fcf78129440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78128980 osd=7 3 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78128980 osd=7 3 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78128980 osd=7 3 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 15 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf78128980 osd=7 3 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 7 15 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78128980 osd=7 4 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7f155700 10 client.27933.objecter _send_command 15 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf78129a40 con 0x7fcf78129000 2024-02-13T00:51:46.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.578+0000 7fcf7cef1700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6812/1964192772 conn(0x7fcf78129000 0x7fcf78129440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:46.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.579+0000 7fcf75ffb700 10 client.27933.objecter ms_handle_connect 0x7fcf78129000 2024-02-13T00:51:46.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.579+0000 7fcf7cef1700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6812/1964192772 conn(0x7fcf78129000 0x7fcf78129440 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:46.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.579+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf78129a40 con 0x7fcf78129000 2024-02-13T00:51:46.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.579+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:51:46.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.579+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 15 = system:0 2024-02-13T00:51:46.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.579+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf78128980 osd=7 4 2024-02-13T00:51:46.585 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.579+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 7 15 2024-02-13T00:51:46.597 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:46.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.597+0000 7fcf7f155700 10 client.27933.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:51:46.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.597+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign -1 16 2024-02-13T00:51:46.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.597+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78128980 osd=7 3 2024-02-13T00:51:46.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.597+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78128980 osd=7 3 2024-02-13T00:51:46.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.597+0000 7fcf7f155700 20 client.27933.objecter _get_session s=0x7fcf78128980 osd=7 3 2024-02-13T00:51:46.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.597+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_remove -1 16 2024-02-13T00:51:46.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.597+0000 7fcf7f155700 20 client.27933.objecter get_session s=0x7fcf78128980 osd=7 3 2024-02-13T00:51:46.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.597+0000 7fcf7f155700 15 client.27933.objecter _session_command_op_assign 7 16 2024-02-13T00:51:46.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.597+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78128980 osd=7 4 2024-02-13T00:51:46.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.597+0000 7fcf7f155700 10 client.27933.objecter _send_command 16 2024-02-13T00:51:46.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.597+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fcf78129a40 con 0x7fcf78129000 2024-02-13T00:51:46.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf75ffb700 1 -- 172.21.15.162:0/2300331042 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fcf78129a40 con 0x7fcf78129000 2024-02-13T00:51:46.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf75ffb700 10 client.27933.objecter ms_dispatch 0x7fcf78001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:51:46.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf75ffb700 10 client.27933.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf75ffb700 20 client.27933.objecter put_session s=0x7fcf78128980 osd=7 4 2024-02-13T00:51:46.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf75ffb700 15 client.27933.objecter _session_command_op_remove 7 16 2024-02-13T00:51:46.604 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:51:46.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 10 librados: watch_flush enter 2024-02-13T00:51:46.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 10 librados: watch_flush exit 2024-02-13T00:51:46.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 10 client.27933.objecter close_session for osd.0 2024-02-13T00:51:46.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6801/2103743338 conn(0x7fcf78028590 msgr2=0x7fcf780fbd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6801/2103743338 conn(0x7fcf78028590 0x7fcf780fbd00 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf781071b0 osd=0 1 2024-02-13T00:51:46.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 10 client.27933.objecter close_session for osd.1 2024-02-13T00:51:46.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6805/150599780 conn(0x7fcf78027be0 msgr2=0x7fcf78027fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6805/150599780 conn(0x7fcf78027be0 0x7fcf78027fc0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf780fd370 osd=1 1 2024-02-13T00:51:46.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 10 client.27933.objecter close_session for osd.2 2024-02-13T00:51:46.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6809/3689719376 conn(0x7fcf780fced0 msgr2=0x7fcf7811e2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6809/3689719376 conn(0x7fcf780fced0 0x7fcf7811e2a0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78027a60 osd=2 1 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 10 client.27933.objecter close_session for osd.3 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6813/604971011 conn(0x7fcf7811f910 msgr2=0x7fcf7811fd50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6813/604971011 conn(0x7fcf7811f910 0x7fcf7811fd50 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf7811b8e0 osd=3 1 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 10 client.27933.objecter close_session for osd.4 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7d6f2700 1 -- 172.21.15.162:0/2300331042 reap_dead start 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6800/3344263388 conn(0x7fcf781231f0 msgr2=0x7fcf781255e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6800/3344263388 conn(0x7fcf781231f0 0x7fcf781255e0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7d6f2700 1 -- 172.21.15.162:0/2300331042 reap_dead start 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78120a10 osd=4 1 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 10 client.27933.objecter close_session for osd.5 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6804/2277917372 conn(0x7fcf78125fe0 msgr2=0x7fcf781283d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6804/2277917372 conn(0x7fcf78125fe0 0x7fcf781283d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78120bf0 osd=5 1 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 10 client.27933.objecter close_session for osd.6 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6808/1506640341 conn(0x7fcf7812a4e0 msgr2=0x7fcf7812a960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6808/1506640341 conn(0x7fcf7812a4e0 0x7fcf7812a960 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf7812b170 osd=6 1 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 10 client.27933.objecter close_session for osd.7 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6812/1964192772 conn(0x7fcf78129000 msgr2=0x7fcf78129440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.181:6812/1964192772 conn(0x7fcf78129000 0x7fcf78129440 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 20 client.27933.objecter put_session s=0x7fcf78128980 osd=7 1 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.599+0000 7fcf7f155700 20 client.27933.objecter shutdown clearing up homeless session... 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.600+0000 7fcf7f155700 10 client.27933.objecter successfully canceled tick 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.600+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcf68062c90 msgr2=0x7fcf68065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.600+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcf68062c90 0x7fcf68065150 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 rx=0x7fcf64004ee0 tx=0x7fcf6400b040).stop 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.600+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf780f8d10 msgr2=0x7fcf780f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.600+0000 7fcf7f155700 1 --2- 172.21.15.162:0/2300331042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf780f8d10 0x7fcf780f6cc0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7fcf6c0048f0 tx=0x7fcf6c009cb0).stop 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.600+0000 7fcf7d6f2700 1 -- 172.21.15.162:0/2300331042 reap_dead start 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.600+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 shutdown_connections 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.600+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 >> 172.21.15.162:0/2300331042 conn(0x7fcf7806eb90 msgr2=0x7fcf78102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.600+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 shutdown_connections 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.600+0000 7fcf7f155700 1 -- 172.21.15.162:0/2300331042 wait complete. 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:46.600+0000 7fcf7f155700 1 librados: shutdown 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.609 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:51:46.610 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:51:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:45.865534+0000 mgr.y (mgr.14544) 571 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:46.118531+0000 osd.0 (osd.0) 259 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:51:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:46.119427+0000 osd.0 (osd.0) 260 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:51:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:46.439984+0000 mon.a (mon.0) 3337 : audit [DBG] from='client.? 172.21.15.162:0/2300331042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:46.533563+0000 mon.a (mon.0) 3338 : cluster [DBG] osdmap e679: 8 total, 8 up, 5 in 2024-02-13T00:51:47.051 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:47.053 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:45.865534+0000 mgr.y (mgr.14544) 571 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:47.053 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:46.118531+0000 osd.0 (osd.0) 259 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:51:47.053 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:46.119427+0000 osd.0 (osd.0) 260 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:51:47.053 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:46.439984+0000 mon.a (mon.0) 3337 : audit [DBG] from='client.? 172.21.15.162:0/2300331042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:47.054 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:46.533563+0000 mon.a (mon.0) 3338 : cluster [DBG] osdmap e679: 8 total, 8 up, 5 in 2024-02-13T00:51:47.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:45.865534+0000 mgr.y (mgr.14544) 571 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:47.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:46.118531+0000 osd.0 (osd.0) 259 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:51:47.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:46.119427+0000 osd.0 (osd.0) 260 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:51:47.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:46.439984+0000 mon.a (mon.0) 3337 : audit [DBG] from='client.? 172.21.15.162:0/2300331042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:51:47.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:46.533563+0000 mon.a (mon.0) 3338 : cluster [DBG] osdmap e679: 8 total, 8 up, 5 in 2024-02-13T00:51:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:47.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:47.064 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6, 7] out_osds: [3, 2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-13T00:51:47.064 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:51:47.064 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-13T00:51:47.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-13T00:51:47.155 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:47.183 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:47.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:51:48.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:46.511209+0000 mgr.y (mgr.14544) 572 : cluster [DBG] pgmap v1024: 204 pgs: 2 active+clean+remapped, 1 active+remapped+backfilling, 1 active+recovering, 200 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 855 B/s rd, 342 B/s wr, 1 op/s; 3/717 objects misplaced (0.418%); 652 B/s, 22 objects/s recovering 2024-02-13T00:51:48.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:46.637076+0000 mon.a (mon.0) 3339 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:48.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:46.511209+0000 mgr.y (mgr.14544) 572 : cluster [DBG] pgmap v1024: 204 pgs: 2 active+clean+remapped, 1 active+remapped+backfilling, 1 active+recovering, 200 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 855 B/s rd, 342 B/s wr, 1 op/s; 3/717 objects misplaced (0.418%); 652 B/s, 22 objects/s recovering 2024-02-13T00:51:48.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:46.637076+0000 mon.a (mon.0) 3339 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:48.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:46.511209+0000 mgr.y (mgr.14544) 572 : cluster [DBG] pgmap v1024: 204 pgs: 2 active+clean+remapped, 1 active+remapped+backfilling, 1 active+recovering, 200 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 855 B/s rd, 342 B/s wr, 1 op/s; 3/717 objects misplaced (0.418%); 652 B/s, 22 objects/s recovering 2024-02-13T00:51:48.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:46.637076+0000 mon.a (mon.0) 3339 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:48.632 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:48.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:48.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:48.765 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:48.950 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:47.535200+0000 osd.1 (osd.1) 289 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:51:48.950 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:47.538350+0000 osd.1 (osd.1) 290 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:51:48.950 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:47.599565+0000 mon.a (mon.0) 3340 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:48.950 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:47.599627+0000 mon.a (mon.0) 3341 : cluster [DBG] osdmap e680: 8 total, 8 up, 5 in 2024-02-13T00:51:48.950 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:47.601203+0000 mon.a (mon.0) 3342 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-118"}]: dispatch 2024-02-13T00:51:48.950 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:48.087024+0000 osd.0 (osd.0) 261 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:51:48.951 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:48.089413+0000 osd.0 (osd.0) 262 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:51:48.951 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:47.535200+0000 osd.1 (osd.1) 289 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:51:48.951 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:47.538350+0000 osd.1 (osd.1) 290 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:51:48.951 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:47.599565+0000 mon.a (mon.0) 3340 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:48.951 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:47.599627+0000 mon.a (mon.0) 3341 : cluster [DBG] osdmap e680: 8 total, 8 up, 5 in 2024-02-13T00:51:48.951 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:47.601203+0000 mon.a (mon.0) 3342 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-118"}]: dispatch 2024-02-13T00:51:48.951 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:48.087024+0000 osd.0 (osd.0) 261 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:51:48.951 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:48.089413+0000 osd.0 (osd.0) 262 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:51:48.963 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:48.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:49.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:51:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:51:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:51:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:47.535200+0000 osd.1 (osd.1) 289 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:51:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:47.538350+0000 osd.1 (osd.1) 290 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:51:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:47.599565+0000 mon.a (mon.0) 3340 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:47.599627+0000 mon.a (mon.0) 3341 : cluster [DBG] osdmap e680: 8 total, 8 up, 5 in 2024-02-13T00:51:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:47.601203+0000 mon.a (mon.0) 3342 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-118"}]: dispatch 2024-02-13T00:51:49.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:48.087024+0000 osd.0 (osd.0) 261 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:51:49.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:48.089413+0000 osd.0 (osd.0) 262 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:51:49.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:49.095 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:49.218 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:51:49.218 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:51:49.222 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:49.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:49.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:49.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:49.354 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:49.477 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:49.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:49.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:49.518 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:49.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:49.616 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:49.674 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:51:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:48.511562+0000 mgr.y (mgr.14544) 573 : cluster [DBG] pgmap v1027: 204 pgs: 2 active+clean+remapped, 1 active+remapped+backfilling, 1 active+recovering, 200 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 856 B/s rd, 342 B/s wr, 1 op/s; 3/717 objects misplaced (0.418%); 653 B/s, 22 objects/s recovering 2024-02-13T00:51:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:48.518692+0000 osd.1 (osd.1) 291 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:51:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:48.521651+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:51:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:48.594782+0000 mon.a (mon.0) 3343 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:48.604726+0000 mon.a (mon.0) 3344 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-118"}]': finished 2024-02-13T00:51:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:48.604793+0000 mon.a (mon.0) 3345 : cluster [DBG] osdmap e681: 8 total, 8 up, 5 in 2024-02-13T00:51:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:49.067778+0000 osd.0 (osd.0) 263 : cluster [DBG] 28.10 scrub starts 2024-02-13T00:51:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:49.069009+0000 osd.0 (osd.0) 264 : cluster [DBG] 28.10 scrub ok 2024-02-13T00:51:49.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:49.342742+0000 mon.a (mon.0) 3346 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:48.511562+0000 mgr.y (mgr.14544) 573 : cluster [DBG] pgmap v1027: 204 pgs: 2 active+clean+remapped, 1 active+remapped+backfilling, 1 active+recovering, 200 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 856 B/s rd, 342 B/s wr, 1 op/s; 3/717 objects misplaced (0.418%); 653 B/s, 22 objects/s recovering 2024-02-13T00:51:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:48.518692+0000 osd.1 (osd.1) 291 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:51:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:48.521651+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:51:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:48.594782+0000 mon.a (mon.0) 3343 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:48.604726+0000 mon.a (mon.0) 3344 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-118"}]': finished 2024-02-13T00:51:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:48.604793+0000 mon.a (mon.0) 3345 : cluster [DBG] osdmap e681: 8 total, 8 up, 5 in 2024-02-13T00:51:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:49.067778+0000 osd.0 (osd.0) 263 : cluster [DBG] 28.10 scrub starts 2024-02-13T00:51:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:49.069009+0000 osd.0 (osd.0) 264 : cluster [DBG] 28.10 scrub ok 2024-02-13T00:51:49.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:49.342742+0000 mon.a (mon.0) 3346 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:50.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-13T00:51:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:48.511562+0000 mgr.y (mgr.14544) 573 : cluster [DBG] pgmap v1027: 204 pgs: 2 active+clean+remapped, 1 active+remapped+backfilling, 1 active+recovering, 200 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 856 B/s rd, 342 B/s wr, 1 op/s; 3/717 objects misplaced (0.418%); 653 B/s, 22 objects/s recovering 2024-02-13T00:51:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:48.518692+0000 osd.1 (osd.1) 291 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:51:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:48.521651+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:51:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:48.594782+0000 mon.a (mon.0) 3343 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:48.604726+0000 mon.a (mon.0) 3344 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-118"}]': finished 2024-02-13T00:51:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:48.604793+0000 mon.a (mon.0) 3345 : cluster [DBG] osdmap e681: 8 total, 8 up, 5 in 2024-02-13T00:51:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:49.067778+0000 osd.0 (osd.0) 263 : cluster [DBG] 28.10 scrub starts 2024-02-13T00:51:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:49.069009+0000 osd.0 (osd.0) 264 : cluster [DBG] 28.10 scrub ok 2024-02-13T00:51:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:49.342742+0000 mon.a (mon.0) 3346 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:51:50.365 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:50.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:50.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:50.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:50.502 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:50.523 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:50.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:50.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:50.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:50.656 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:49.487658+0000 osd.1 (osd.1) 293 : cluster [DBG] 4.1b scrub starts 2024-02-13T00:51:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:49.491164+0000 osd.1 (osd.1) 294 : cluster [DBG] 4.1b scrub ok 2024-02-13T00:51:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:49.611308+0000 mon.a (mon.0) 3347 : cluster [DBG] osdmap e682: 8 total, 8 up, 5 in 2024-02-13T00:51:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:50.536943+0000 mon.a (mon.0) 3348 : cluster [DBG] osdmap e683: 8 total, 8 up, 5 in 2024-02-13T00:51:50.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:50.545845+0000 mon.a (mon.0) 3349 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:50.866 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:50.866 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:50.904 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:50.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:49.487658+0000 osd.1 (osd.1) 293 : cluster [DBG] 4.1b scrub starts 2024-02-13T00:51:50.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:49.491164+0000 osd.1 (osd.1) 294 : cluster [DBG] 4.1b scrub ok 2024-02-13T00:51:50.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:49.611308+0000 mon.a (mon.0) 3347 : cluster [DBG] osdmap e682: 8 total, 8 up, 5 in 2024-02-13T00:51:50.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:50.536943+0000 mon.a (mon.0) 3348 : cluster [DBG] osdmap e683: 8 total, 8 up, 5 in 2024-02-13T00:51:50.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:50.545845+0000 mon.a (mon.0) 3349 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:50.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:49.487658+0000 osd.1 (osd.1) 293 : cluster [DBG] 4.1b scrub starts 2024-02-13T00:51:50.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:49.491164+0000 osd.1 (osd.1) 294 : cluster [DBG] 4.1b scrub ok 2024-02-13T00:51:50.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:49.611308+0000 mon.a (mon.0) 3347 : cluster [DBG] osdmap e682: 8 total, 8 up, 5 in 2024-02-13T00:51:50.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:50.536943+0000 mon.a (mon.0) 3348 : cluster [DBG] osdmap e683: 8 total, 8 up, 5 in 2024-02-13T00:51:50.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:50.545845+0000 mon.a (mon.0) 3349 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:50.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:51.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:51.039 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:51.137 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:51.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:51.142 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:51.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:51:51.273 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:51.294 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:51:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:51.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:51:51.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:51.427 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:51.447 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:51.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:51.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:51.577 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:51.597 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:51.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:51.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:51.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:51:51.730 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:51.750 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:51:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:51.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:51:51.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:50.512373+0000 mgr.y (mgr.14544) 574 : cluster [DBG] pgmap v1030: 172 pgs: 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:51:51.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:51.049189+0000 osd.0 (osd.0) 265 : cluster [DBG] 2.c scrub starts 2024-02-13T00:51:51.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:51.052135+0000 osd.0 (osd.0) 266 : cluster [DBG] 2.c scrub ok 2024-02-13T00:51:51.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:51.536168+0000 mon.a (mon.0) 3350 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:51.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:51.536256+0000 mon.a (mon.0) 3351 : cluster [DBG] osdmap e684: 8 total, 8 up, 5 in 2024-02-13T00:51:51.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:51.538137+0000 mon.a (mon.0) 3352 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:51.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:50.512373+0000 mgr.y (mgr.14544) 574 : cluster [DBG] pgmap v1030: 172 pgs: 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:51:51.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:51.049189+0000 osd.0 (osd.0) 265 : cluster [DBG] 2.c scrub starts 2024-02-13T00:51:51.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:51.052135+0000 osd.0 (osd.0) 266 : cluster [DBG] 2.c scrub ok 2024-02-13T00:51:51.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:51.536168+0000 mon.a (mon.0) 3350 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:51.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:51.536256+0000 mon.a (mon.0) 3351 : cluster [DBG] osdmap e684: 8 total, 8 up, 5 in 2024-02-13T00:51:51.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:51.538137+0000 mon.a (mon.0) 3352 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:51.853 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:51:51.881 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:50.512373+0000 mgr.y (mgr.14544) 574 : cluster [DBG] pgmap v1030: 172 pgs: 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:51:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:51.049189+0000 osd.0 (osd.0) 265 : cluster [DBG] 2.c scrub starts 2024-02-13T00:51:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:51.052135+0000 osd.0 (osd.0) 266 : cluster [DBG] 2.c scrub ok 2024-02-13T00:51:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:51.536168+0000 mon.a (mon.0) 3350 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:51:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:51.536256+0000 mon.a (mon.0) 3351 : cluster [DBG] osdmap e684: 8 total, 8 up, 5 in 2024-02-13T00:51:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:51.538137+0000 mon.a (mon.0) 3352 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:51:53.531 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:51:53.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:53.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:51:53.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:53.666 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:53.687 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:53.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:53.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:53.792 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:51:53.819 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:52.512810+0000 mgr.y (mgr.14544) 575 : cluster [DBG] pgmap v1033: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:51:53.819 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:52.534566+0000 osd.1 (osd.1) 295 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:51:53.819 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:52.536515+0000 osd.1 (osd.1) 296 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:51:53.819 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:52.537901+0000 mon.a (mon.0) 3353 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:53.819 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:52.538003+0000 mon.a (mon.0) 3354 : cluster [DBG] osdmap e685: 8 total, 8 up, 5 in 2024-02-13T00:51:53.819 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:52.538848+0000 mon.a (mon.0) 3355 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-120"}]: dispatch 2024-02-13T00:51:53.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:52.512810+0000 mgr.y (mgr.14544) 575 : cluster [DBG] pgmap v1033: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:51:53.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:52.534566+0000 osd.1 (osd.1) 295 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:51:53.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:52.536515+0000 osd.1 (osd.1) 296 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:51:53.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:52.537901+0000 mon.a (mon.0) 3353 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:53.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:52.538003+0000 mon.a (mon.0) 3354 : cluster [DBG] osdmap e685: 8 total, 8 up, 5 in 2024-02-13T00:51:53.820 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:52.538848+0000 mon.a (mon.0) 3355 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-120"}]: dispatch 2024-02-13T00:51:53.822 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:53.918 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:51:53.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:53.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:51:54.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:54.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:52.512810+0000 mgr.y (mgr.14544) 575 : cluster [DBG] pgmap v1033: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:51:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:52.534566+0000 osd.1 (osd.1) 295 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:51:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:52.536515+0000 osd.1 (osd.1) 296 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:51:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:52.537901+0000 mon.a (mon.0) 3353 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:51:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:52.538003+0000 mon.a (mon.0) 3354 : cluster [DBG] osdmap e685: 8 total, 8 up, 5 in 2024-02-13T00:51:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:52.538848+0000 mon.a (mon.0) 3355 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-120"}]: dispatch 2024-02-13T00:51:54.058 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:54.079 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:54.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:54.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 -- 172.21.15.162:0/2747844637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8c0f8500 msgr2=0x7fdf8c104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:54.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 --2- 172.21.15.162:0/2747844637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8c0f8500 0x7fdf8c104bd0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7fdf84009be0 tx=0x7fdf84004d10).stop 2024-02-13T00:51:54.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 -- 172.21.15.162:0/2747844637 shutdown_connections 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 --2- 172.21.15.162:0/2747844637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8c0f8500 0x7fdf8c104bd0 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 --2- 172.21.15.162:0/2747844637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf8c0f9b70 0x7fdf8c0f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 --2- 172.21.15.162:0/2747844637 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8c0fa440 0x7fdf8c0f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 -- 172.21.15.162:0/2747844637 >> 172.21.15.162:0/2747844637 conn(0x7fdf8c0f38c0 msgr2=0x7fdf8c0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 -- 172.21.15.162:0/2747844637 shutdown_connections 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 -- 172.21.15.162:0/2747844637 wait complete. 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 librados: starting msgr at 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 librados: starting objecter 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 Processor -- start 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 -- start start 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 librados: setting wanted keys 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.147+0000 7fdf9206f700 1 librados: calling monclient init 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf9206f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8c0f8500 0x7fdf8c10e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf9206f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8c0f9b70 0x7fdf8c10e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:54.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf9206f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf8c0fa440 0x7fdf8c1081d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf9206f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c1086c0 con 0x7fdf8c0f8500 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf9206f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c108830 con 0x7fdf8c0fa440 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf9206f700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c1089f0 con 0x7fdf8c0f9b70 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf8b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8c0f8500 0x7fdf8c10e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf8affd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8c0f9b70 0x7fdf8c10e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf8b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8c0f8500 0x7fdf8c10e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:36212/0 (socket says 172.21.15.162:36212) 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf8affd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8c0f9b70 0x7fdf8c10e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:46818/0 (socket says 172.21.15.162:46818) 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf8b7fe700 1 -- 172.21.15.162:0/4231875472 learned_addr learned my addr 172.21.15.162:0/4231875472 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf8bfff700 1 --2- 172.21.15.162:0/4231875472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf8c0fa440 0x7fdf8c1081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf8b7fe700 1 -- 172.21.15.162:0/4231875472 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8c0f9b70 msgr2=0x7fdf8c10e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf8b7fe700 1 --2- 172.21.15.162:0/4231875472 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8c0f9b70 0x7fdf8c10e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf8b7fe700 1 -- 172.21.15.162:0/4231875472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf8c0fa440 msgr2=0x7fdf8c1081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:54.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf8b7fe700 1 --2- 172.21.15.162:0/4231875472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf8c0fa440 0x7fdf8c1081d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf8b7fe700 1 -- 172.21.15.162:0/4231875472 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c005040 con 0x7fdf8c0f8500 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf8b7fe700 1 --2- 172.21.15.162:0/4231875472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8c0f8500 0x7fdf8c10e030 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 rx=0x7fdf800041d0 tx=0x7fdf80009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf88ff9700 10 client.?.objecter ms_handle_connect 0x7fdf8c0f8500 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf88ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf88ff9700 1 -- 172.21.15.162:0/4231875472 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80007b40 con 0x7fdf8c0f8500 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf9206f700 1 -- 172.21.15.162:0/4231875472 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf840097c0 con 0x7fdf8c0f8500 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf88ff9700 1 -- 172.21.15.162:0/4231875472 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf8000f040 con 0x7fdf8c0f8500 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf88ff9700 1 -- 172.21.15.162:0/4231875472 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80014810 con 0x7fdf8c0f8500 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf9206f700 10 client.27939.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf9206f700 1 -- 172.21.15.162:0/4231875472 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8c109030 con 0x7fdf8c0f8500 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.148+0000 7fdf9206f700 1 librados: init done 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.149+0000 7fdf9206f700 1 -- 172.21.15.162:0/4231875472 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8c0475e0 con 0x7fdf8c0f8500 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.150+0000 7fdf88ff9700 1 -- 172.21.15.162:0/4231875472 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fdf80009110 con 0x7fdf8c0f8500 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.150+0000 7fdf88ff9700 1 --2- 172.21.15.162:0/4231875472 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf74062c90 0x7fdf74065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:54.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.150+0000 7fdf88ff9700 1 -- 172.21.15.162:0/4231875472 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(686..686 src has 449..686) v4 ==== 8591+0+0 (secure 0 0 0) 0x7fdf8007d410 con 0x7fdf8c0f8500 2024-02-13T00:51:54.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.150+0000 7fdf88ff9700 10 client.27939.objecter ms_dispatch 0x7fdf8c001550 osd_map(686..686 src has 449..686) v4 2024-02-13T00:51:54.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.150+0000 7fdf88ff9700 3 client.27939.objecter handle_osd_map got epochs [686,686] > 0 2024-02-13T00:51:54.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.150+0000 7fdf88ff9700 3 client.27939.objecter handle_osd_map decoding full epoch 686 2024-02-13T00:51:54.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.150+0000 7fdf8affd700 1 --2- 172.21.15.162:0/4231875472 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf74062c90 0x7fdf74065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:54.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.150+0000 7fdf88ff9700 20 client.27939.objecter dump_active .. 0 homeless 2024-02-13T00:51:54.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.150+0000 7fdf8affd700 1 --2- 172.21.15.162:0/4231875472 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf74062c90 0x7fdf74065150 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 rx=0x7fdf8c109970 tx=0x7fdf7c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:54.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.150+0000 7fdf88ff9700 10 client.27939.objecter ms_handle_connect 0x7fdf74062c90 2024-02-13T00:51:54.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.155+0000 7fdf88ff9700 1 -- 172.21.15.162:0/4231875472 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fdf800506d0 con 0x7fdf8c0f8500 2024-02-13T00:51:54.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:54.210 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:54.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.338+0000 7fdf9206f700 1 -- 172.21.15.162:0/4231875472 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fdf8c1096b0 con 0x7fdf8c0f8500 2024-02-13T00:51:54.407 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:51:54.407 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:51:54.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.543+0000 7fdf88ff9700 1 -- 172.21.15.162:0/4231875472 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v687) v1 ==== 87+0+0 (secure 0 0 0) 0x7fdf800502c0 con 0x7fdf8c0f8500 2024-02-13T00:51:54.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.548+0000 7fdf9206f700 10 librados: watch_flush enter 2024-02-13T00:51:54.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.549+0000 7fdf9206f700 10 librados: watch_flush exit 2024-02-13T00:51:54.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.549+0000 7fdf9206f700 20 client.27939.objecter shutdown clearing up homeless session... 2024-02-13T00:51:54.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.549+0000 7fdf9206f700 10 client.27939.objecter successfully canceled tick 2024-02-13T00:51:54.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.549+0000 7fdf9206f700 1 -- 172.21.15.162:0/4231875472 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf74062c90 msgr2=0x7fdf74065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:54.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.549+0000 7fdf9206f700 1 --2- 172.21.15.162:0/4231875472 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf74062c90 0x7fdf74065150 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 rx=0x7fdf8c109970 tx=0x7fdf7c00b040).stop 2024-02-13T00:51:54.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.549+0000 7fdf9206f700 1 -- 172.21.15.162:0/4231875472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8c0f8500 msgr2=0x7fdf8c10e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:54.553 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.549+0000 7fdf9206f700 1 --2- 172.21.15.162:0/4231875472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8c0f8500 0x7fdf8c10e030 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 rx=0x7fdf800041d0 tx=0x7fdf80009f60).stop 2024-02-13T00:51:54.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.550+0000 7fdf9206f700 1 -- 172.21.15.162:0/4231875472 shutdown_connections 2024-02-13T00:51:54.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.550+0000 7fdf9206f700 1 --2- 172.21.15.162:0/4231875472 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf74062c90 0x7fdf74065150 unknown :-1 s=CLOSED pgs=3003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:54.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.550+0000 7fdf9206f700 1 --2- 172.21.15.162:0/4231875472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf8c0fa440 0x7fdf8c1081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:54.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.550+0000 7fdf9206f700 1 --2- 172.21.15.162:0/4231875472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8c0f8500 0x7fdf8c10e030 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:54.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.550+0000 7fdf9206f700 1 --2- 172.21.15.162:0/4231875472 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8c0f9b70 0x7fdf8c10e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:54.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.550+0000 7fdf9206f700 1 -- 172.21.15.162:0/4231875472 >> 172.21.15.162:0/4231875472 conn(0x7fdf8c0f38c0 msgr2=0x7fdf8c0f40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:54.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.550+0000 7fdf9206f700 1 -- 172.21.15.162:0/4231875472 shutdown_connections 2024-02-13T00:51:54.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.550+0000 7fdf9206f700 1 -- 172.21.15.162:0/4231875472 wait complete. 2024-02-13T00:51:54.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:54.550+0000 7fdf9206f700 1 librados: shutdown 2024-02-13T00:51:54.555 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:51:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:53.089016+0000 osd.0 (osd.0) 267 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:51:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:53.089943+0000 osd.0 (osd.0) 268 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:51:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:53.540937+0000 mon.a (mon.0) 3356 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-120"}]': finished 2024-02-13T00:51:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:53.540997+0000 mon.a (mon.0) 3357 : cluster [DBG] osdmap e686: 8 total, 8 up, 5 in 2024-02-13T00:51:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:53.541541+0000 mon.a (mon.0) 3358 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-120", "mode": "writeback"}]: dispatch 2024-02-13T00:51:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:53.571072+0000 osd.1 (osd.1) 297 : cluster [DBG] 2.0 deep-scrub starts 2024-02-13T00:51:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:53.574600+0000 osd.1 (osd.1) 298 : cluster [DBG] 2.0 deep-scrub ok 2024-02-13T00:51:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:54.339475+0000 mon.a (mon.0) 3359 : audit [INF] from='client.? 172.21.15.162:0/4231875472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:51:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:54.537070+0000 mon.a (mon.0) 3360 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:54.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:54.537096+0000 mon.a (mon.0) 3361 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:53.089016+0000 osd.0 (osd.0) 267 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:51:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:53.089943+0000 osd.0 (osd.0) 268 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:51:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:53.540937+0000 mon.a (mon.0) 3356 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-120"}]': finished 2024-02-13T00:51:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:53.540997+0000 mon.a (mon.0) 3357 : cluster [DBG] osdmap e686: 8 total, 8 up, 5 in 2024-02-13T00:51:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:53.541541+0000 mon.a (mon.0) 3358 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-120", "mode": "writeback"}]: dispatch 2024-02-13T00:51:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:53.571072+0000 osd.1 (osd.1) 297 : cluster [DBG] 2.0 deep-scrub starts 2024-02-13T00:51:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:53.574600+0000 osd.1 (osd.1) 298 : cluster [DBG] 2.0 deep-scrub ok 2024-02-13T00:51:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:54.339475+0000 mon.a (mon.0) 3359 : audit [INF] from='client.? 172.21.15.162:0/4231875472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:51:54.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:54.537070+0000 mon.a (mon.0) 3360 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:54.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:54.537096+0000 mon.a (mon.0) 3361 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:55.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:53.089016+0000 osd.0 (osd.0) 267 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:51:55.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:53.089943+0000 osd.0 (osd.0) 268 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:51:55.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:53.540937+0000 mon.a (mon.0) 3356 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-120"}]': finished 2024-02-13T00:51:55.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:53.540997+0000 mon.a (mon.0) 3357 : cluster [DBG] osdmap e686: 8 total, 8 up, 5 in 2024-02-13T00:51:55.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:53.541541+0000 mon.a (mon.0) 3358 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-120", "mode": "writeback"}]: dispatch 2024-02-13T00:51:55.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:53.571072+0000 osd.1 (osd.1) 297 : cluster [DBG] 2.0 deep-scrub starts 2024-02-13T00:51:55.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:53.574600+0000 osd.1 (osd.1) 298 : cluster [DBG] 2.0 deep-scrub ok 2024-02-13T00:51:55.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:54.339475+0000 mon.a (mon.0) 3359 : audit [INF] from='client.? 172.21.15.162:0/4231875472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:51:55.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:54.537070+0000 mon.a (mon.0) 3360 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:51:55.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:54.537096+0000 mon.a (mon.0) 3361 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:51:55.748 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-13T00:51:55.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:54.109500+0000 osd.6 (osd.6) 214 : cluster [DBG] 28.9 scrub starts 2024-02-13T00:51:55.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:54.114111+0000 osd.6 (osd.6) 215 : cluster [DBG] 28.9 scrub ok 2024-02-13T00:51:55.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:54.513225+0000 mgr.y (mgr.14544) 576 : cluster [DBG] pgmap v1036: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:51:55.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:54.538904+0000 osd.1 (osd.1) 299 : cluster [DBG] 6.22 scrub starts 2024-02-13T00:51:55.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:54.542193+0000 osd.1 (osd.1) 300 : cluster [DBG] 6.22 scrub ok 2024-02-13T00:51:55.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:54.543197+0000 mon.a (mon.0) 3362 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-120", "mode": "writeback"}]': finished 2024-02-13T00:51:55.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:54.543266+0000 mon.a (mon.0) 3363 : audit [INF] from='client.? 172.21.15.162:0/4231875472' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:51:55.815 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:54.543318+0000 mon.a (mon.0) 3364 : cluster [DBG] osdmap e687: 8 total, 8 up, 5 in 2024-02-13T00:51:55.816 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:54.619671+0000 mon.a (mon.0) 3365 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:55.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:54.109500+0000 osd.6 (osd.6) 214 : cluster [DBG] 28.9 scrub starts 2024-02-13T00:51:55.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:54.114111+0000 osd.6 (osd.6) 215 : cluster [DBG] 28.9 scrub ok 2024-02-13T00:51:55.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:54.513225+0000 mgr.y (mgr.14544) 576 : cluster [DBG] pgmap v1036: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:51:55.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:54.538904+0000 osd.1 (osd.1) 299 : cluster [DBG] 6.22 scrub starts 2024-02-13T00:51:55.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:54.542193+0000 osd.1 (osd.1) 300 : cluster [DBG] 6.22 scrub ok 2024-02-13T00:51:55.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:54.543197+0000 mon.a (mon.0) 3362 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-120", "mode": "writeback"}]': finished 2024-02-13T00:51:55.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:54.543266+0000 mon.a (mon.0) 3363 : audit [INF] from='client.? 172.21.15.162:0/4231875472' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:51:55.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:54.543318+0000 mon.a (mon.0) 3364 : cluster [DBG] osdmap e687: 8 total, 8 up, 5 in 2024-02-13T00:51:55.816 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:54.619671+0000 mon.a (mon.0) 3365 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:55.872 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:54.109500+0000 osd.6 (osd.6) 214 : cluster [DBG] 28.9 scrub starts 2024-02-13T00:51:55.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:54.114111+0000 osd.6 (osd.6) 215 : cluster [DBG] 28.9 scrub ok 2024-02-13T00:51:55.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:54.513225+0000 mgr.y (mgr.14544) 576 : cluster [DBG] pgmap v1036: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:51:55.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:54.538904+0000 osd.1 (osd.1) 299 : cluster [DBG] 6.22 scrub starts 2024-02-13T00:51:55.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:54.542193+0000 osd.1 (osd.1) 300 : cluster [DBG] 6.22 scrub ok 2024-02-13T00:51:55.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:54.543197+0000 mon.a (mon.0) 3362 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-120", "mode": "writeback"}]': finished 2024-02-13T00:51:55.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:54.543266+0000 mon.a (mon.0) 3363 : audit [INF] from='client.? 172.21.15.162:0/4231875472' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:51:55.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:54.543318+0000 mon.a (mon.0) 3364 : cluster [DBG] osdmap e687: 8 total, 8 up, 5 in 2024-02-13T00:51:55.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:54.619671+0000 mon.a (mon.0) 3365 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:51:56.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:51:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:51:56.564 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:51:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:51:56] "GET /metrics HTTP/1.1" 200 189500 "" "Prometheus/2.33.4" 2024-02-13T00:51:56.740 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:56.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:55.559805+0000 mon.a (mon.0) 3366 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:55.559887+0000 mon.a (mon.0) 3367 : cluster [DBG] osdmap e688: 8 total, 8 up, 5 in 2024-02-13T00:51:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:55.561080+0000 mon.a (mon.0) 3368 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-120"}]: dispatch 2024-02-13T00:51:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:55.869437+0000 mgr.y (mgr.14544) 577 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:55.559805+0000 mon.a (mon.0) 3366 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:55.559887+0000 mon.a (mon.0) 3367 : cluster [DBG] osdmap e688: 8 total, 8 up, 5 in 2024-02-13T00:51:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:55.561080+0000 mon.a (mon.0) 3368 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-120"}]: dispatch 2024-02-13T00:51:56.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:55.869437+0000 mgr.y (mgr.14544) 577 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:56.844 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:56.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:55.559805+0000 mon.a (mon.0) 3366 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:51:56.874 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:55.559887+0000 mon.a (mon.0) 3367 : cluster [DBG] osdmap e688: 8 total, 8 up, 5 in 2024-02-13T00:51:56.874 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:55.561080+0000 mon.a (mon.0) 3368 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-120"}]: dispatch 2024-02-13T00:51:56.874 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:55.869437+0000 mgr.y (mgr.14544) 577 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:51:56.876 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:56.897 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:56.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:57.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:51:57.031 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:57.051 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:51:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:57.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:51:57.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:51:57.183 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:57.202 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:51:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:57.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:51:57.227 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:51:57.305 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:51:57.332 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:57.456 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:51:57.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:57.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:51:57.561 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:51:57.595 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:57.614 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:51:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:57.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:51:57.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:51:57.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:56.513839+0000 mgr.y (mgr.14544) 578 : cluster [DBG] pgmap v1039: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-13T00:51:57.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:56.557478+0000 mon.a (mon.0) 3369 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:57.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:56.564355+0000 mon.a (mon.0) 3370 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-120"}]': finished 2024-02-13T00:51:57.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:56.564428+0000 mon.a (mon.0) 3371 : cluster [DBG] osdmap e689: 8 total, 8 up, 5 in 2024-02-13T00:51:57.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:56.513839+0000 mgr.y (mgr.14544) 578 : cluster [DBG] pgmap v1039: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-13T00:51:57.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:56.557478+0000 mon.a (mon.0) 3369 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:57.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:56.564355+0000 mon.a (mon.0) 3370 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-120"}]': finished 2024-02-13T00:51:57.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:56.564428+0000 mon.a (mon.0) 3371 : cluster [DBG] osdmap e689: 8 total, 8 up, 5 in 2024-02-13T00:51:57.748 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:57.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/3092481620 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc340f8d10 msgr2=0x7fbc340f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:57.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 --2- 172.21.15.162:0/3092481620 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc340f8d10 0x7fbc340f90f0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 rx=0x7fbc24004870 tx=0x7fbc24009cb0).stop 2024-02-13T00:51:57.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/3092481620 shutdown_connections 2024-02-13T00:51:57.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 --2- 172.21.15.162:0/3092481620 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc34102270 0x7fbc34104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:57.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 --2- 172.21.15.162:0/3092481620 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc340f8d10 0x7fbc340f90f0 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:57.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 --2- 172.21.15.162:0/3092481620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc340f95e0 0x7fbc34101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:57.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/3092481620 >> 172.21.15.162:0/3092481620 conn(0x7fbc3406eb90 msgr2=0x7fbc340f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:57.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/3092481620 shutdown_connections 2024-02-13T00:51:57.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/3092481620 wait complete. 2024-02-13T00:51:57.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 librados: starting msgr at 2024-02-13T00:51:57.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 librados: starting objecter 2024-02-13T00:51:57.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 Processor -- start 2024-02-13T00:51:57.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 -- start start 2024-02-13T00:51:57.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 librados: setting wanted keys 2024-02-13T00:51:57.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.928+0000 7fbc3a5f4700 1 librados: calling monclient init 2024-02-13T00:51:57.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc3a5f4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc340f8d10 0x7fbc340f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:57.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc3a5f4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc340f95e0 0x7fbc340f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:57.935 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc3a5f4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc34102270 0x7fbc340f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc3a5f4700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341167a0 con 0x7fbc340f8d10 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc3a5f4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc34116910 con 0x7fbc34102270 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc3a5f4700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc34116ad0 con 0x7fbc340f95e0 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc33fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc340f8d10 0x7fbc340f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc33fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc340f8d10 0x7fbc340f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44278/0 (socket says 172.21.15.162:44278) 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc33fff700 1 -- 172.21.15.162:0/1659217068 learned_addr learned my addr 172.21.15.162:0/1659217068 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc337fe700 1 --2- 172.21.15.162:0/1659217068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc340f95e0 0x7fbc340f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc38b91700 1 --2- 172.21.15.162:0/1659217068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc34102270 0x7fbc340f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc337fe700 1 -- 172.21.15.162:0/1659217068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc34102270 msgr2=0x7fbc340f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc337fe700 1 --2- 172.21.15.162:0/1659217068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc34102270 0x7fbc340f76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc337fe700 1 -- 172.21.15.162:0/1659217068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc340f8d10 msgr2=0x7fbc340f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc337fe700 1 --2- 172.21.15.162:0/1659217068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc340f8d10 0x7fbc340f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc337fe700 1 -- 172.21.15.162:0/1659217068 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc24005040 con 0x7fbc340f95e0 2024-02-13T00:51:57.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc33fff700 1 --2- 172.21.15.162:0/1659217068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc340f8d10 0x7fbc340f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc337fe700 1 --2- 172.21.15.162:0/1659217068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc340f95e0 0x7fbc340f71b0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 rx=0x7fbc1c00e8f0 tx=0x7fbc1c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc317fa700 10 client.?.objecter ms_handle_connect 0x7fbc340f95e0 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.929+0000 7fbc317fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.930+0000 7fbc317fa700 1 -- 172.21.15.162:0/1659217068 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c007b40 con 0x7fbc340f95e0 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.930+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/1659217068 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc34116db0 con 0x7fbc340f95e0 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.930+0000 7fbc317fa700 1 -- 172.21.15.162:0/1659217068 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc1c00f040 con 0x7fbc340f95e0 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.930+0000 7fbc317fa700 1 -- 172.21.15.162:0/1659217068 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c014710 con 0x7fbc340f95e0 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.930+0000 7fbc3a5f4700 10 client.38042.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.930+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/1659217068 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc341171f0 con 0x7fbc340f95e0 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.930+0000 7fbc3a5f4700 1 librados: init done 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.931+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/1659217068 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc340475e0 con 0x7fbc340f95e0 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.931+0000 7fbc317fa700 1 -- 172.21.15.162:0/1659217068 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fbc1c014b40 con 0x7fbc340f95e0 2024-02-13T00:51:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.931+0000 7fbc317fa700 1 --2- 172.21.15.162:0/1659217068 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbc20062c90 0x7fbc20065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:51:57.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.931+0000 7fbc317fa700 1 -- 172.21.15.162:0/1659217068 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(690..690 src has 449..690) v4 ==== 8210+0+0 (secure 0 0 0) 0x7fbc1c07cde0 con 0x7fbc340f95e0 2024-02-13T00:51:57.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.931+0000 7fbc317fa700 10 client.38042.objecter ms_dispatch 0x7fbc34001550 osd_map(690..690 src has 449..690) v4 2024-02-13T00:51:57.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.931+0000 7fbc317fa700 3 client.38042.objecter handle_osd_map got epochs [690,690] > 0 2024-02-13T00:51:57.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.931+0000 7fbc33fff700 1 --2- 172.21.15.162:0/1659217068 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbc20062c90 0x7fbc20065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:51:57.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.931+0000 7fbc317fa700 3 client.38042.objecter handle_osd_map decoding full epoch 690 2024-02-13T00:51:57.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.932+0000 7fbc317fa700 20 client.38042.objecter dump_active .. 0 homeless 2024-02-13T00:51:57.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.932+0000 7fbc317fa700 10 client.38042.objecter ms_handle_connect 0x7fbc20062c90 2024-02-13T00:51:57.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.932+0000 7fbc33fff700 1 --2- 172.21.15.162:0/1659217068 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbc20062c90 0x7fbc20065150 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7fbc24009a00 tx=0x7fbc2400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:51:57.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:57.936+0000 7fbc317fa700 1 -- 172.21.15.162:0/1659217068 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fbc1c049f60 con 0x7fbc340f95e0 2024-02-13T00:51:58.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:56.513839+0000 mgr.y (mgr.14544) 578 : cluster [DBG] pgmap v1039: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-13T00:51:58.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:56.557478+0000 mon.a (mon.0) 3369 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:51:58.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:56.564355+0000 mon.a (mon.0) 3370 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-120"}]': finished 2024-02-13T00:51:58.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:56.564428+0000 mon.a (mon.0) 3371 : cluster [DBG] osdmap e689: 8 total, 8 up, 5 in 2024-02-13T00:51:58.110 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.107+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/1659217068 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fbc3405fb80 con 0x7fbc340f95e0 2024-02-13T00:51:58.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.108+0000 7fbc317fa700 1 -- 172.21.15.162:0/1659217068 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v690) v1 ==== 81+0+46626 (secure 0 0 0) 0x7fbc1c050210 con 0x7fbc340f95e0 2024-02-13T00:51:58.113 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:51:58.113 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:51:58.113 INFO:teuthology.orchestra.run.smithi162.stdout: "epoch": 690, 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "created": "2024-02-13T00:31:51.797017+0000", 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "modified": "2024-02-13T00:51:57.561636+0000", 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "last_up_change": "2024-02-13T00:48:03.474219+0000", 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "last_in_change": "2024-02-13T00:51:40.525589+0000", 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_num": 5801984, 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_set": [ 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "noscrub", 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "pglog_hardlimit", 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "purged_snapdirs", 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_deletes", 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: "sortbitwise" 2024-02-13T00:51:58.114 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_version": 109, 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "full_ratio": 0.94999998807907104, 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_snapshot": "", 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_max": 244, 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "max_osd": 8, 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "require_min_compat_client": "luminous", 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "min_compat_client": "luminous", 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "require_osd_release": "pacific", 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "pools": [ 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 1, 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "device_health_metrics", 2024-02-13T00:51:58.115 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:35:34.545390+0000", 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 1, 2024-02-13T00:51:58.116 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 1, 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 1, 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 1, 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 1, 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "71", 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:58.118 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:58.119 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_max": 32, 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_min": 1 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_devicehealth": {} 2024-02-13T00:51:58.120 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 2, 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": ".rgw.root", 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.459980+0000", 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:58.121 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:58.122 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "58", 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:58.123 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:58.124 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:58.125 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 3, 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "iscsi", 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.740219+0000", 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:58.127 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 3, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 3, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 3, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 3, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 3, 2024-02-13T00:51:58.128 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "102", 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:58.129 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:58.130 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:58.131 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd": {} 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 4, 2024-02-13T00:51:58.132 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.log", 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:41.489292+0000", 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:58.133 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:58.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "60", 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:58.135 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:58.136 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:58.137 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 5, 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.control", 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:43.466304+0000", 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:58.138 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 22, 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 22, 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 22, 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 22, 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 22, 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:58.139 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "5.16", 2024-02-13T00:51:58.140 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 544, 2024-02-13T00:51:58.140 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 543, 2024-02-13T00:51:58.140 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 543, 2024-02-13T00:51:58.140 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:58.140 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:58.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.140 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "544", 2024-02-13T00:51:58.140 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:58.140 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "544", 2024-02-13T00:51:58.140 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "542", 2024-02-13T00:51:58.140 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:58.140 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:58.141 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:58.142 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 6, 2024-02-13T00:51:58.143 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.meta", 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:45.542442+0000", 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:58.144 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 42, 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 42, 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 42, 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 42, 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 42, 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:58.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "248", 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "246", 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:58.146 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:58.147 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_bias": 4 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.148 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 28, 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:00.190777+0000", 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:58.149 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:58.150 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "108", 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:58.151 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:58.152 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:58.152 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:58.152 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:58.152 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:58.152 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:58.152 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:58.152 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:58.152 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:58.152 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:58.152 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:58.152 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:58.152 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:51:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 234, 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-101", 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:50:12.777962+0000", 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8193, 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 3, 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:51:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 1, 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 8, 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 8, 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 8, 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 8, 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 8, 2024-02-13T00:51:58.155 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "689", 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "688", 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "688", 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "688", 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:51:58.156 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 7, 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 685, 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:51:58.157 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi162-84057-101", 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:51:58.158 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 8192, 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99", 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:51:58.159 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 8, 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 683, 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6801", 2024-02-13T00:51:58.160 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6802", 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:58.161 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6804", 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6803", 2024-02-13T00:51:58.162 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6801/2103743338", 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6802/2103743338", 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6804/2103743338", 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6803/2103743338", 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886", 2024-02-13T00:51:58.163 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 13, 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 683, 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6805", 2024-02-13T00:51:58.164 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6806", 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6808", 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6807", 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6805/150599780", 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6806/150599780", 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6808/150599780", 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6807/150599780", 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e", 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:58.167 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 18, 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 333, 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.168 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6809", 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6810", 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.169 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6812", 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6811", 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:51:58.170 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6809/3689719376", 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6810/3689719376", 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6812/3689719376", 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6811/3689719376", 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001", 2024-02-13T00:51:58.171 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 25, 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 255, 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.172 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6813", 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6814", 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6816", 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6815", 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.174 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6813/604971011", 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6814/604971011", 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6816/604971011", 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6815/604971011", 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3", 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:51:58.175 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 31, 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 81, 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6800", 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6801", 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6803", 2024-02-13T00:51:58.177 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6802", 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6800/3344263388", 2024-02-13T00:51:58.178 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6801/3344263388", 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6803/3344263388", 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6802/3344263388", 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723", 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 39, 2024-02-13T00:51:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 436, 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 479, 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 399, 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 437, 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6804", 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:58.180 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6805", 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6807", 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6806", 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6804/2277917372", 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6805/2277917372", 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2277917372", 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2277917372", 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:58.182 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474", 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 45, 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 683, 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:58.183 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6808", 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6809", 2024-02-13T00:51:58.184 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6811", 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6810", 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6808/1506640341", 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6809/1506640341", 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1506640341", 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1506640341", 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b", 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 53, 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 215, 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:51:58.187 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6812", 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6813", 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.188 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6815", 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:51:58.189 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6814", 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6812/1964192772", 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6813/1964192772", 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6815/1964192772", 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6814/1964192772", 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.190 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_xinfo": [ 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:51:05.019053+0000", 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:51:58.191 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:51:14.095523+0000", 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:58.192 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:50:51.149335+0000", 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:51:08.962349+0000", 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:51:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:50:59.963296+0000", 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:50:34.133215+0000", 2024-02-13T00:51:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 436 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:51:57.475514+0000", 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:51:58.195 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:51:15.105476+0000", 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap": [ 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.b", 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: 3, 2024-02-13T00:51:58.196 INFO:teuthology.orchestra.run.smithi162.stdout: 6, 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: 5 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "6.23", 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: 3, 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: 6, 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: 7 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap_items": [ 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.197 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.4", 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 1, 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 0 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "5.5", 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 6 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "6.6", 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 0 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "6.22", 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 4 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.199 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "28.12", 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 4 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_temp": [], 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_temp": [], 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: "blocklist": { 2024-02-13T00:51:58.200 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/1909610488": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2370739667": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/1502590187": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/4121673848": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2412382714": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3041562701": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/2775563314": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3135632190": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/197307636": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2653247729": "2024-02-14T00:32:24.380625+0000" 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "range_blocklist": {}, 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profiles": { 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "default": { 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:51:58.201 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van" 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "testprofile-test-rados-api-smithi162-84057-101": { 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-device-class": "", 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-root": "default", 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:51:58.202 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van", 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: "w": "8" 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps_queue": [ 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 24, 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 9, 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 25, 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.204 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 60, 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 78, 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 85, 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.207 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 94, 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:58.208 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 95, 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 4, 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 4 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 107, 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 113, 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:51:58.210 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 125, 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 10, 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 133, 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 12, 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 154, 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 14, 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.213 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 166, 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 17, 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 176, 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 20, 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 237, 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 238, 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 4, 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 241, 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:51:58.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: "new_removed_snaps": [], 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: "new_purged_snaps": [], 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_node_flags": {}, 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: "device_class_flags": {}, 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode": { 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_enabled": false, 2024-02-13T00:51:58.217 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_bucket_count": 0, 2024-02-13T00:51:58.218 INFO:teuthology.orchestra.run.smithi162.stdout: "degraded_stretch_mode": 0, 2024-02-13T00:51:58.218 INFO:teuthology.orchestra.run.smithi162.stdout: "recovering_stretch_mode": 0, 2024-02-13T00:51:58.218 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_bucket": 0 2024-02-13T00:51:58.218 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:51:58.218 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:51:58.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 10 librados: watch_flush enter 2024-02-13T00:51:58.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 10 librados: watch_flush exit 2024-02-13T00:51:58.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 20 client.38042.objecter shutdown clearing up homeless session... 2024-02-13T00:51:58.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 10 client.38042.objecter successfully canceled tick 2024-02-13T00:51:58.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/1659217068 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbc20062c90 msgr2=0x7fbc20065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:58.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 --2- 172.21.15.162:0/1659217068 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbc20062c90 0x7fbc20065150 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7fbc24009a00 tx=0x7fbc2400b040).stop 2024-02-13T00:51:58.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/1659217068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc340f95e0 msgr2=0x7fbc340f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:51:58.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 --2- 172.21.15.162:0/1659217068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc340f95e0 0x7fbc340f71b0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 rx=0x7fbc1c00e8f0 tx=0x7fbc1c00b040).stop 2024-02-13T00:51:58.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/1659217068 shutdown_connections 2024-02-13T00:51:58.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 --2- 172.21.15.162:0/1659217068 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbc20062c90 0x7fbc20065150 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:58.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 --2- 172.21.15.162:0/1659217068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc34102270 0x7fbc340f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:58.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 --2- 172.21.15.162:0/1659217068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc340f8d10 0x7fbc340f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:58.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 --2- 172.21.15.162:0/1659217068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc340f95e0 0x7fbc340f71b0 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:51:58.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/1659217068 >> 172.21.15.162:0/1659217068 conn(0x7fbc3406eb90 msgr2=0x7fbc34102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:51:58.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/1659217068 shutdown_connections 2024-02-13T00:51:58.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 -- 172.21.15.162:0/1659217068 wait complete. 2024-02-13T00:51:58.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:51:58.111+0000 7fbc3a5f4700 1 librados: shutdown 2024-02-13T00:51:58.711 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:51:58.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:57.475269+0000 osd.6 (osd.6) 216 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:58.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:57.475511+0000 osd.6 (osd.6) 217 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:58.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:57.570148+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e690: 8 total, 8 up, 5 in 2024-02-13T00:51:58.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:58.108588+0000 mon.c (mon.2) 485 : audit [DBG] from='client.? 172.21.15.162:0/1659217068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:51:58.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:57.475269+0000 osd.6 (osd.6) 216 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:58.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:57.475511+0000 osd.6 (osd.6) 217 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:58.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:57.570148+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e690: 8 total, 8 up, 5 in 2024-02-13T00:51:58.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:58.108588+0000 mon.c (mon.2) 485 : audit [DBG] from='client.? 172.21.15.162:0/1659217068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:51:59.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:57.475269+0000 osd.6 (osd.6) 216 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:59.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:57.475511+0000 osd.6 (osd.6) 217 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:59.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:57.570148+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e690: 8 total, 8 up, 5 in 2024-02-13T00:51:59.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:58.108588+0000 mon.c (mon.2) 485 : audit [DBG] from='client.? 172.21.15.162:0/1659217068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:51:59.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:51:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:51:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:51:59.701 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:51:59.704 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:58.514218+0000 mgr.y (mgr.14544) 579 : cluster [DBG] pgmap v1042: 172 pgs: 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:59.704 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:58.575752+0000 mon.a (mon.0) 3373 : cluster [DBG] osdmap e691: 8 total, 8 up, 5 in 2024-02-13T00:51:59.704 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:58.577610+0000 mon.a (mon.0) 3374 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:59.704 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:58.951018+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:59.704 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:51:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:58.951248+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:59.704 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:58.514218+0000 mgr.y (mgr.14544) 579 : cluster [DBG] pgmap v1042: 172 pgs: 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:59.704 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:58.575752+0000 mon.a (mon.0) 3373 : cluster [DBG] osdmap e691: 8 total, 8 up, 5 in 2024-02-13T00:51:59.705 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:58.577610+0000 mon.a (mon.0) 3374 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:59.705 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:58.951018+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:59.705 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:51:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:58.951248+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:59.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:51:59.806 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:59.836 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:58.514218+0000 mgr.y (mgr.14544) 579 : cluster [DBG] pgmap v1042: 172 pgs: 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:51:59.836 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:58.575752+0000 mon.a (mon.0) 3373 : cluster [DBG] osdmap e691: 8 total, 8 up, 5 in 2024-02-13T00:51:59.836 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:58.577610+0000 mon.a (mon.0) 3374 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:51:59.836 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:58.951018+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:51:59.836 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:51:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:58.951248+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:51:59.838 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:51:59.859 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:51:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:51:59.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:51:59.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:51:59.993 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:00.013 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:00.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:00.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:00.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:00.146 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:00.270 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:00.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:00.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:00.279 INFO:tasks.thrashosds.thrasher:j is {'epoch': 690, 'fsid': 'd4ee01fc-ca06-11ee-95b9-87774f69a715', 'created': '2024-02-13T00:31:51.797017+0000', 'modified': '2024-02-13T00:51:57.561636+0000', 'last_up_change': '2024-02-13T00:48:03.474219+0000', 'last_in_change': '2024-02-13T00:51:40.525589+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 109, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 244, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-13T00:35:34.545390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-13T00:37:39.459980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-13T00:37:39.740219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-13T00:37:41.489292+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-13T00:37:43.466304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 544, 'last_epoch_started': 543, 'last_epoch_clean': 543, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '544', 'last_force_op_resend_preluminous': '542', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-13T00:37:45.542442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '246', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 28, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-13T00:41:00.190777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 234, 'pool_name': 'test-rados-api-smithi162-84057-101', 'create_time': '2024-02-13T00:50:12.777962+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '689', 'last_force_op_resend': '688', 'last_force_op_resend_prenautilus': '688', 'last_force_op_resend_preluminous': '688', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 685, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi162-84057-101', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 683, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6801', 'nonce': 2103743338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 2103743338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 2103743338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6803', 'nonce': 2103743338}]}, 'public_addr': '172.21.15.162:6801/2103743338', 'cluster_addr': '172.21.15.162:6802/2103743338', 'heartbeat_back_addr': '172.21.15.162:6804/2103743338', 'heartbeat_front_addr': '172.21.15.162:6803/2103743338', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '39e9d2ef-9aa4-45bb-8a47-b0e0ec222886', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 683, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6805', 'nonce': 150599780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 150599780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 150599780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6807', 'nonce': 150599780}]}, 'public_addr': '172.21.15.162:6805/150599780', 'cluster_addr': '172.21.15.162:6806/150599780', 'heartbeat_back_addr': '172.21.15.162:6808/150599780', 'heartbeat_front_addr': '172.21.15.162:6807/150599780', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 333, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6809', 'nonce': 3689719376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3689719376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3689719376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6811', 'nonce': 3689719376}]}, 'public_addr': '172.21.15.162:6809/3689719376', 'cluster_addr': '172.21.15.162:6810/3689719376', 'heartbeat_back_addr': '172.21.15.162:6812/3689719376', 'heartbeat_front_addr': '172.21.15.162:6811/3689719376', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e6b9f0e-7a18-4cf8-9331-1a973fadc001', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 255, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6813', 'nonce': 604971011}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 604971011}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 604971011}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6815', 'nonce': 604971011}]}, 'public_addr': '172.21.15.162:6813/604971011', 'cluster_addr': '172.21.15.162:6814/604971011', 'heartbeat_back_addr': '172.21.15.162:6816/604971011', 'heartbeat_front_addr': '172.21.15.162:6815/604971011', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 3344263388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 3344263388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 3344263388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 3344263388}]}, 'public_addr': '172.21.15.181:6800/3344263388', 'cluster_addr': '172.21.15.181:6801/3344263388', 'heartbeat_back_addr': '172.21.15.181:6803/3344263388', 'heartbeat_front_addr': '172.21.15.181:6802/3344263388', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6aa2bd68-7f4b-4973-ac49-7fbc57c1a723', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 436, 'up_from': 479, 'up_thru': 399, 'down_at': 437, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2277917372}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2277917372}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2277917372}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2277917372}]}, 'public_addr': '172.21.15.181:6804/2277917372', 'cluster_addr': '172.21.15.181:6805/2277917372', 'heartbeat_back_addr': '172.21.15.181:6807/2277917372', 'heartbeat_front_addr': '172.21.15.181:6806/2277917372', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '791ba814-5284-432e-addc-d6b7f5f3d474', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 683, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1506640341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1506640341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1506640341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1506640341}]}, 'public_addr': '172.21.15.181:6808/1506640341', 'cluster_addr': '172.21.15.181:6809/1506640341', 'heartbeat_back_addr': '172.21.15.181:6811/1506640341', 'heartbeat_front_addr': '172.21.15.181:6810/1506640341', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f369c65-e663-4eb2-87a5-f2900f15b35b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 215, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 1964192772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 1964192772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 1964192772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 1964192772}]}, 'public_addr': '172.21.15.181:6812/1964192772', 'cluster_addr': '172.21.15.181:6813/1964192772', 'heartbeat_back_addr': '172.21.15.181:6815/1964192772', 'heartbeat_front_addr': '172.21.15.181:6814/1964192772', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:51:05.019053+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:51:14.095523+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:50:51.149335+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:51:08.962349+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:50:59.963296+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:50:34.133215+0000', 'dead_epoch': 436}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:51:57.475514+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:51:15.105476+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.b', 'osds': [3, 6, 5]}, {'pgid': '6.23', 'osds': [3, 6, 7]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '5.5', 'mappings': [{'from': 0, 'to': 6}]}, {'pgid': '6.6', 'mappings': [{'from': 6, 'to': 0}]}, {'pgid': '6.22', 'mappings': [{'from': 6, 'to': 4}]}, {'pgid': '28.12', 'mappings': [{'from': 0, 'to': 4}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.162:6800/1909610488': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2370739667': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/1502590187': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/4121673848': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:0/2412382714': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/3041562701': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/2775563314': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/3135632190': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/197307636': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2653247729': '2024-02-14T00:32:24.380625+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi162-84057-101': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 24, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 60, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 78, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 94, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 113, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 166, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}, {'pool': 237, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 238, 'snaps': [{'begin': 4, 'length': 1}]}, {'pool': 241, 'snaps': [{'begin': 6, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-13T00:52:00.279 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.4 2024-02-13T00:52:00.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd rm-pg-upmap-items 4.4 2024-02-13T00:52:00.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:00.402 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:00.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.510+0000 7fe524e49700 1 -- 172.21.15.162:0/3313567159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe52006bca0 msgr2=0x7fe520104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.510+0000 7fe524e49700 1 --2- 172.21.15.162:0/3313567159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe52006bca0 0x7fe520104030 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7fe514004870 tx=0x7fe514009cb0).stop 2024-02-13T00:52:00.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.510+0000 7fe524e49700 1 -- 172.21.15.162:0/3313567159 shutdown_connections 2024-02-13T00:52:00.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.510+0000 7fe524e49700 1 --2- 172.21.15.162:0/3313567159 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe520104520 0x7fe520106910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.510+0000 7fe524e49700 1 --2- 172.21.15.162:0/3313567159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe52006b3d0 0x7fe52006b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.510+0000 7fe524e49700 1 --2- 172.21.15.162:0/3313567159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe52006bca0 0x7fe520104030 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.510+0000 7fe524e49700 1 -- 172.21.15.162:0/3313567159 >> 172.21.15.162:0/3313567159 conn(0x7fe5200f38c0 msgr2=0x7fe5200f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.510+0000 7fe524e49700 1 -- 172.21.15.162:0/3313567159 shutdown_connections 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.510+0000 7fe524e49700 1 -- 172.21.15.162:0/3313567159 wait complete. 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.511+0000 7fe524e49700 1 librados: starting msgr at 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.511+0000 7fe524e49700 1 librados: starting objecter 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.511+0000 7fe524e49700 1 Processor -- start 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.511+0000 7fe524e49700 1 -- start start 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.511+0000 7fe524e49700 1 librados: setting wanted keys 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.511+0000 7fe524e49700 1 librados: calling monclient init 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe524e49700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe52006b3d0 0x7fe520110520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe524e49700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe52006bca0 0x7fe520110a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:00.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe524e49700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe520104520 0x7fe52010a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe524e49700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52010abb0 con 0x7fe52006bca0 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe524e49700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52010acf0 con 0x7fe520104520 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe524e49700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52010aeb0 con 0x7fe52006b3d0 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51e59c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe52006b3d0 0x7fe520110520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51e59c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe52006b3d0 0x7fe520110520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:43366/0 (socket says 172.21.15.162:43366) 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51dd9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe52006bca0 0x7fe520110a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51e59c700 1 -- 172.21.15.162:0/3653708805 learned_addr learned my addr 172.21.15.162:0/3653708805 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51ed9d700 1 --2- 172.21.15.162:0/3653708805 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe520104520 0x7fe52010a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51e59c700 1 -- 172.21.15.162:0/3653708805 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe520104520 msgr2=0x7fe52010a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51e59c700 1 --2- 172.21.15.162:0/3653708805 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe520104520 0x7fe52010a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51e59c700 1 -- 172.21.15.162:0/3653708805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe52006bca0 msgr2=0x7fe520110a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51e59c700 1 --2- 172.21.15.162:0/3653708805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe52006bca0 0x7fe520110a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51e59c700 1 -- 172.21.15.162:0/3653708805 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe514005040 con 0x7fe52006b3d0 2024-02-13T00:52:00.522 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51ed9d700 1 --2- 172.21.15.162:0/3653708805 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe520104520 0x7fe52010a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51dd9b700 1 --2- 172.21.15.162:0/3653708805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe52006bca0 0x7fe520110a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe51e59c700 1 --2- 172.21.15.162:0/3653708805 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe52006b3d0 0x7fe520110520 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 rx=0x7fe5100041d0 tx=0x7fe510009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe50f7fe700 10 client.?.objecter ms_handle_connect 0x7fe52006b3d0 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe50f7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510007b40 con 0x7fe52006b3d0 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52010b190 con 0x7fe52006b3d0 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe51000f040 con 0x7fe52006b3d0 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510014770 con 0x7fe52006b3d0 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe524e49700 10 client.38048.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe520117320 con 0x7fe52006b3d0 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.512+0000 7fe524e49700 1 librados: init done 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.513+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe5200475e0 con 0x7fe52006b3d0 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.514+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe510009110 con 0x7fe52006b3d0 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.514+0000 7fe50f7fe700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe508062c90 0x7fe508065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:00.523 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(692..692 src has 449..692) v4 ==== 8596+0+0 (secure 0 0 0) 0x7fe51007d4d0 con 0x7fe52006b3d0 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 osd_map(692..692 src has 449..692) v4 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe50f7fe700 3 client.38048.objecter handle_osd_map got epochs [692,692] > 0 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe50f7fe700 3 client.38048.objecter handle_osd_map decoding full epoch 692 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe51dd9b700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe508062c90 0x7fe508065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe50f7fe700 20 client.38048.objecter dump_active .. 0 homeless 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v692) v1 ==== 54+0+16 (secure 0 0 0) 0x7fe510084070 con 0x7fe52006b3d0 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe50f7fe700 10 client.38048.objecter ms_handle_connect 0x7fe508062c90 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe51dd9b700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe508062c90 0x7fe508065150 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 rx=0x7fe52010be60 tx=0x7fe514016040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe524e49700 10 client.38048.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 1 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe520028590 0x7fe5200fdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe520109470 osd=0 3 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520109470 osd=0 3 2024-02-13T00:52:00.524 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe520109470 osd=0 3 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 1 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe520109470 osd=0 3 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 0 1 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520109470 osd=0 4 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe524e49700 10 client.38048.objecter _send_command 1 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5200ff650 con 0x7fe520028590 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.515+0000 7fe51ed9d700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe520028590 0x7fe5200fdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.516+0000 7fe50f7fe700 10 client.38048.objecter ms_handle_connect 0x7fe520028590 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.516+0000 7fe51ed9d700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe520028590 0x7fe5200fdfe0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.516+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe5200ff650 con 0x7fe520028590 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.516+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.516+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 1 = system:0 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.516+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe520109470 osd=0 4 2024-02-13T00:52:00.525 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.516+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 0 1 2024-02-13T00:52:00.530 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:00.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:00.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:00.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.534+0000 7fe524e49700 10 client.38048.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:00.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.534+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 2 2024-02-13T00:52:00.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.534+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe520109470 osd=0 3 2024-02-13T00:52:00.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.534+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520109470 osd=0 3 2024-02-13T00:52:00.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.534+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe520109470 osd=0 3 2024-02-13T00:52:00.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.534+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 2 2024-02-13T00:52:00.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.534+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe520109470 osd=0 3 2024-02-13T00:52:00.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.534+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 0 2 2024-02-13T00:52:00.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.534+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520109470 osd=0 4 2024-02-13T00:52:00.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.534+0000 7fe524e49700 10 client.38048.objecter _send_command 2 2024-02-13T00:52:00.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.534+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe5200ff7c0 con 0x7fe520028590 2024-02-13T00:52:00.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.536+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe5200ff7c0 con 0x7fe520028590 2024-02-13T00:52:00.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.536+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:00.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.536+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.536+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe520109470 osd=0 4 2024-02-13T00:52:00.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.536+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 0 2 2024-02-13T00:52:00.544 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:52:00.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe524e49700 10 client.38048.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:00.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 3 2024-02-13T00:52:00.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6805/150599780 conn(0x7fe5200feb50 0x7fe5200fef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:00.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe5200fe4d0 osd=1 3 2024-02-13T00:52:00.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe5200fe4d0 osd=1 3 2024-02-13T00:52:00.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe5200fe4d0 osd=1 3 2024-02-13T00:52:00.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 3 2024-02-13T00:52:00.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe5200fe4d0 osd=1 3 2024-02-13T00:52:00.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 1 3 2024-02-13T00:52:00.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe5200fe4d0 osd=1 4 2024-02-13T00:52:00.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe524e49700 10 client.38048.objecter _send_command 3 2024-02-13T00:52:00.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5200ff420 con 0x7fe5200feb50 2024-02-13T00:52:00.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.539+0000 7fe51e59c700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6805/150599780 conn(0x7fe5200feb50 0x7fe5200fef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:00.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.542+0000 7fe51e59c700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6805/150599780 conn(0x7fe5200feb50 0x7fe5200fef30 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:00.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.542+0000 7fe50f7fe700 10 client.38048.objecter ms_handle_connect 0x7fe5200feb50 2024-02-13T00:52:00.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.543+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe5200ff420 con 0x7fe5200feb50 2024-02-13T00:52:00.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.543+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:52:00.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.543+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 3 = system:0 2024-02-13T00:52:00.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.543+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe5200fe4d0 osd=1 4 2024-02-13T00:52:00.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.543+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 1 3 2024-02-13T00:52:00.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.558+0000 7fe524e49700 10 client.38048.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:00.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.558+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 4 2024-02-13T00:52:00.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.558+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe5200fe4d0 osd=1 3 2024-02-13T00:52:00.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.558+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe5200fe4d0 osd=1 3 2024-02-13T00:52:00.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.558+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe5200fe4d0 osd=1 3 2024-02-13T00:52:00.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.558+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 4 2024-02-13T00:52:00.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.558+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe5200fe4d0 osd=1 3 2024-02-13T00:52:00.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.558+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 1 4 2024-02-13T00:52:00.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.558+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe5200fe4d0 osd=1 4 2024-02-13T00:52:00.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.558+0000 7fe524e49700 10 client.38048.objecter _send_command 4 2024-02-13T00:52:00.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.558+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe520059400 con 0x7fe5200feb50 2024-02-13T00:52:00.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe520059400 con 0x7fe5200feb50 2024-02-13T00:52:00.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:00.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe5200fe4d0 osd=1 4 2024-02-13T00:52:00.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 1 4 2024-02-13T00:52:00.564 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:52:00.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe524e49700 10 client.38048.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:00.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 5 2024-02-13T00:52:00.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe5200280f0 0x7fe52011e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:00.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe520059810 osd=2 3 2024-02-13T00:52:00.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520059810 osd=2 3 2024-02-13T00:52:00.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe520059810 osd=2 3 2024-02-13T00:52:00.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 5 2024-02-13T00:52:00.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe520059810 osd=2 3 2024-02-13T00:52:00.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 2 5 2024-02-13T00:52:00.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520059810 osd=2 4 2024-02-13T00:52:00.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe524e49700 10 client.38048.objecter _send_command 5 2024-02-13T00:52:00.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.560+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5201213a0 con 0x7fe5200280f0 2024-02-13T00:52:00.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.561+0000 7fe51dd9b700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe5200280f0 0x7fe52011e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:00.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.561+0000 7fe50f7fe700 10 client.38048.objecter ms_handle_connect 0x7fe5200280f0 2024-02-13T00:52:00.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.561+0000 7fe51dd9b700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe5200280f0 0x7fe52011e750 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:00.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.561+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe5201213a0 con 0x7fe5200280f0 2024-02-13T00:52:00.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.561+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:52:00.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.561+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 5 = system:0 2024-02-13T00:52:00.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.561+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe520059810 osd=2 4 2024-02-13T00:52:00.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.561+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 2 5 2024-02-13T00:52:00.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.576+0000 7fe524e49700 10 client.38048.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:00.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.576+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 6 2024-02-13T00:52:00.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.576+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe520059810 osd=2 3 2024-02-13T00:52:00.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.576+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520059810 osd=2 3 2024-02-13T00:52:00.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.576+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe520059810 osd=2 3 2024-02-13T00:52:00.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.576+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 6 2024-02-13T00:52:00.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.576+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe520059810 osd=2 3 2024-02-13T00:52:00.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.576+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 2 6 2024-02-13T00:52:00.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.576+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520059810 osd=2 4 2024-02-13T00:52:00.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.576+0000 7fe524e49700 10 client.38048.objecter _send_command 6 2024-02-13T00:52:00.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.576+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe520027650 con 0x7fe5200280f0 2024-02-13T00:52:00.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.577+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe520027650 con 0x7fe5200280f0 2024-02-13T00:52:00.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.577+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:00.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.577+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.577+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe520059810 osd=2 4 2024-02-13T00:52:00.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.577+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 2 6 2024-02-13T00:52:00.581 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:52:00.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe524e49700 10 client.38048.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:00.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 7 2024-02-13T00:52:00.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6813/604971011 conn(0x7fe52011c2a0 0x7fe52011fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:00.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe520027a60 osd=3 3 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520027a60 osd=3 3 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe520027a60 osd=3 3 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 7 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe520027a60 osd=3 3 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 3 7 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520027a60 osd=3 4 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe524e49700 10 client.38048.objecter _send_command 7 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe520120370 con 0x7fe52011c2a0 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe51ed9d700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6813/604971011 conn(0x7fe52011c2a0 0x7fe52011fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe50f7fe700 10 client.38048.objecter ms_handle_connect 0x7fe52011c2a0 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe51ed9d700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6813/604971011 conn(0x7fe52011c2a0 0x7fe52011fdc0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe520120370 con 0x7fe52011c2a0 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 7 = system:0 2024-02-13T00:52:00.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe520027a60 osd=3 4 2024-02-13T00:52:00.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.578+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 3 7 2024-02-13T00:52:00.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.595+0000 7fe524e49700 10 client.38048.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:00.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.595+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 8 2024-02-13T00:52:00.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.595+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe520027a60 osd=3 3 2024-02-13T00:52:00.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.595+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520027a60 osd=3 3 2024-02-13T00:52:00.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.595+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe520027a60 osd=3 3 2024-02-13T00:52:00.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.595+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 8 2024-02-13T00:52:00.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.595+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe520027a60 osd=3 3 2024-02-13T00:52:00.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.595+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 3 8 2024-02-13T00:52:00.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.595+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520027a60 osd=3 4 2024-02-13T00:52:00.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.595+0000 7fe524e49700 10 client.38048.objecter _send_command 8 2024-02-13T00:52:00.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.595+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe52011b800 con 0x7fe52011c2a0 2024-02-13T00:52:00.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe52011b800 con 0x7fe52011c2a0 2024-02-13T00:52:00.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:00.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe520027a60 osd=3 4 2024-02-13T00:52:00.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 3 8 2024-02-13T00:52:00.600 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:52:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe524e49700 10 client.38048.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 9 2024-02-13T00:52:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe520123520 0x7fe520125910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52011b800 osd=4 3 2024-02-13T00:52:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52011b800 osd=4 3 2024-02-13T00:52:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52011b800 osd=4 3 2024-02-13T00:52:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 9 2024-02-13T00:52:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe52011b800 osd=4 3 2024-02-13T00:52:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 4 9 2024-02-13T00:52:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52011b800 osd=4 4 2024-02-13T00:52:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe524e49700 10 client.38048.objecter _send_command 9 2024-02-13T00:52:00.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.596+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe52011baf0 con 0x7fe520123520 2024-02-13T00:52:00.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.597+0000 7fe51e59c700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe520123520 0x7fe520125910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:00.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.597+0000 7fe50f7fe700 10 client.38048.objecter ms_handle_connect 0x7fe520123520 2024-02-13T00:52:00.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.597+0000 7fe51e59c700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe520123520 0x7fe520125910 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:00.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.598+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe52011baf0 con 0x7fe520123520 2024-02-13T00:52:00.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.598+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:52:00.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.598+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 9 = system:0 2024-02-13T00:52:00.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.598+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe52011b800 osd=4 4 2024-02-13T00:52:00.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.598+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 4 9 2024-02-13T00:52:00.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.612+0000 7fe524e49700 10 client.38048.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:00.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.612+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 10 2024-02-13T00:52:00.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.612+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52011b800 osd=4 3 2024-02-13T00:52:00.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.612+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52011b800 osd=4 3 2024-02-13T00:52:00.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.612+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52011b800 osd=4 3 2024-02-13T00:52:00.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.612+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 10 2024-02-13T00:52:00.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.612+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe52011b800 osd=4 3 2024-02-13T00:52:00.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.612+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 4 10 2024-02-13T00:52:00.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.612+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52011b800 osd=4 4 2024-02-13T00:52:00.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.612+0000 7fe524e49700 10 client.38048.objecter _send_command 10 2024-02-13T00:52:00.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.612+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe52011baf0 con 0x7fe520123520 2024-02-13T00:52:00.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe52011baf0 con 0x7fe520123520 2024-02-13T00:52:00.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:00.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe52011b800 osd=4 4 2024-02-13T00:52:00.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 4 10 2024-02-13T00:52:00.619 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:52:00.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe524e49700 10 client.38048.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:00.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 11 2024-02-13T00:52:00.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe520126310 0x7fe520128720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52011baf0 osd=5 3 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52011baf0 osd=5 3 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52011baf0 osd=5 3 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 11 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe52011baf0 osd=5 3 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 5 11 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52011baf0 osd=5 4 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe524e49700 10 client.38048.objecter _send_command 11 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe52012b4f0 con 0x7fe520126310 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe51dd9b700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe520126310 0x7fe520128720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe50f7fe700 10 client.38048.objecter ms_handle_connect 0x7fe520126310 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.615+0000 7fe51dd9b700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe520126310 0x7fe520128720 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.616+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe52012b4f0 con 0x7fe520126310 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.616+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:52:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.616+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 11 = system:0 2024-02-13T00:52:00.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.616+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe52011baf0 osd=5 4 2024-02-13T00:52:00.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.616+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 5 11 2024-02-13T00:52:00.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.630+0000 7fe524e49700 10 client.38048.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:00.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.630+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 12 2024-02-13T00:52:00.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.630+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52011baf0 osd=5 3 2024-02-13T00:52:00.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.630+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52011baf0 osd=5 3 2024-02-13T00:52:00.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.630+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52011baf0 osd=5 3 2024-02-13T00:52:00.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.630+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 12 2024-02-13T00:52:00.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.630+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe52011baf0 osd=5 3 2024-02-13T00:52:00.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.630+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 5 12 2024-02-13T00:52:00.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.630+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52011baf0 osd=5 4 2024-02-13T00:52:00.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.630+0000 7fe524e49700 10 client.38048.objecter _send_command 12 2024-02-13T00:52:00.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.630+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe52012b4f0 con 0x7fe520126310 2024-02-13T00:52:00.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:00.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe52012b4f0 con 0x7fe520126310 2024-02-13T00:52:00.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:00.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe52011baf0 osd=5 4 2024-02-13T00:52:00.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 5 12 2024-02-13T00:52:00.636 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe524e49700 10 client.38048.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 13 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe52012a7f0 0x7fe52012ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52012b4f0 osd=6 3 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52012b4f0 osd=6 3 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52012b4f0 osd=6 3 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 13 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe52012b4f0 osd=6 3 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 6 13 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52012b4f0 osd=6 4 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe524e49700 10 client.38048.objecter _send_command 13 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.632+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe52012b300 con 0x7fe52012a7f0 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.633+0000 7fe51ed9d700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe52012a7f0 0x7fe52012ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.633+0000 7fe50f7fe700 10 client.38048.objecter ms_handle_connect 0x7fe52012a7f0 2024-02-13T00:52:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.633+0000 7fe51ed9d700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe52012a7f0 0x7fe52012ac50 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:00.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.634+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe51800f040 con 0x7fe52012a7f0 2024-02-13T00:52:00.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.634+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:52:00.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.634+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 13 = system:0 2024-02-13T00:52:00.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.634+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe52012b4f0 osd=6 4 2024-02-13T00:52:00.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.634+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 6 13 2024-02-13T00:52:00.639 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:52:00.639 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:52:00.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.649+0000 7fe524e49700 10 client.38048.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:00.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.649+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 14 2024-02-13T00:52:00.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.649+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52012b4f0 osd=6 3 2024-02-13T00:52:00.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.649+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52012b4f0 osd=6 3 2024-02-13T00:52:00.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.649+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52012b4f0 osd=6 3 2024-02-13T00:52:00.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.649+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 14 2024-02-13T00:52:00.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.649+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe52012b4f0 osd=6 3 2024-02-13T00:52:00.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.649+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 6 14 2024-02-13T00:52:00.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.649+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52012b4f0 osd=6 4 2024-02-13T00:52:00.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.649+0000 7fe524e49700 10 client.38048.objecter _send_command 14 2024-02-13T00:52:00.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.649+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe52012d720 con 0x7fe52012a7f0 2024-02-13T00:52:00.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe52012d720 con 0x7fe52012a7f0 2024-02-13T00:52:00.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:00.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe52012b4f0 osd=6 4 2024-02-13T00:52:00.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 6 14 2024-02-13T00:52:00.655 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:52:00.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe524e49700 10 client.38048.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:00.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 15 2024-02-13T00:52:00.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe52012de20 0x7fe52012e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:00.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52012d760 osd=7 3 2024-02-13T00:52:00.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52012d760 osd=7 3 2024-02-13T00:52:00.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52012d760 osd=7 3 2024-02-13T00:52:00.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 15 2024-02-13T00:52:00.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe52012d760 osd=7 3 2024-02-13T00:52:00.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 7 15 2024-02-13T00:52:00.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52012d760 osd=7 4 2024-02-13T00:52:00.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe524e49700 10 client.38048.objecter _send_command 15 2024-02-13T00:52:00.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe520131a90 con 0x7fe52012de20 2024-02-13T00:52:00.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.651+0000 7fe51e59c700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe52012de20 0x7fe52012e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:00.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.652+0000 7fe50f7fe700 10 client.38048.objecter ms_handle_connect 0x7fe52012de20 2024-02-13T00:52:00.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.652+0000 7fe51e59c700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe52012de20 0x7fe52012e260 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:00.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.652+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe520131a90 con 0x7fe52012de20 2024-02-13T00:52:00.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.652+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:52:00.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.652+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 15 = system:0 2024-02-13T00:52:00.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.652+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe52012d760 osd=7 4 2024-02-13T00:52:00.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.652+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 7 15 2024-02-13T00:52:00.668 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:00.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.666+0000 7fe524e49700 10 client.38048.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:00.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.666+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign -1 16 2024-02-13T00:52:00.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.666+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52012d760 osd=7 3 2024-02-13T00:52:00.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.666+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52012d760 osd=7 3 2024-02-13T00:52:00.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.666+0000 7fe524e49700 20 client.38048.objecter _get_session s=0x7fe52012d760 osd=7 3 2024-02-13T00:52:00.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.666+0000 7fe524e49700 15 client.38048.objecter _session_command_op_remove -1 16 2024-02-13T00:52:00.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.666+0000 7fe524e49700 20 client.38048.objecter get_session s=0x7fe52012d760 osd=7 3 2024-02-13T00:52:00.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.666+0000 7fe524e49700 15 client.38048.objecter _session_command_op_assign 7 16 2024-02-13T00:52:00.670 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.666+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52012d760 osd=7 4 2024-02-13T00:52:00.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.666+0000 7fe524e49700 10 client.38048.objecter _send_command 16 2024-02-13T00:52:00.671 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.666+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe520130910 con 0x7fe52012de20 2024-02-13T00:52:00.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.668+0000 7fe50f7fe700 1 -- 172.21.15.162:0/3653708805 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe520130910 con 0x7fe52012de20 2024-02-13T00:52:00.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe50f7fe700 10 client.38048.objecter ms_dispatch 0x7fe520001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:00.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe50f7fe700 10 client.38048.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe50f7fe700 20 client.38048.objecter put_session s=0x7fe52012d760 osd=7 4 2024-02-13T00:52:00.672 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe50f7fe700 15 client.38048.objecter _session_command_op_remove 7 16 2024-02-13T00:52:00.673 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:52:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 10 librados: watch_flush enter 2024-02-13T00:52:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 10 librados: watch_flush exit 2024-02-13T00:52:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 10 client.38048.objecter close_session for osd.0 2024-02-13T00:52:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe520028590 msgr2=0x7fe5200fdfe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6801/2103743338 conn(0x7fe520028590 0x7fe5200fdfe0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520109470 osd=0 1 2024-02-13T00:52:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 10 client.38048.objecter close_session for osd.1 2024-02-13T00:52:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6805/150599780 conn(0x7fe5200feb50 msgr2=0x7fe5200fef30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6805/150599780 conn(0x7fe5200feb50 0x7fe5200fef30 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe5200fe4d0 osd=1 1 2024-02-13T00:52:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 10 client.38048.objecter close_session for osd.2 2024-02-13T00:52:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe5200280f0 msgr2=0x7fe52011e750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6809/3689719376 conn(0x7fe5200280f0 0x7fe52011e750 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520059810 osd=2 1 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 10 client.38048.objecter close_session for osd.3 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6813/604971011 conn(0x7fe52011c2a0 msgr2=0x7fe52011fdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe51ed9d700 1 -- 172.21.15.162:0/3653708805 reap_dead start 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6813/604971011 conn(0x7fe52011c2a0 0x7fe52011fdc0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe520027a60 osd=3 1 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 10 client.38048.objecter close_session for osd.4 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe520123520 msgr2=0x7fe520125910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6800/3344263388 conn(0x7fe520123520 0x7fe520125910 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52011b800 osd=4 1 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 10 client.38048.objecter close_session for osd.5 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe520126310 msgr2=0x7fe520128720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6804/2277917372 conn(0x7fe520126310 0x7fe520128720 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52011baf0 osd=5 1 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 10 client.38048.objecter close_session for osd.6 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe52012a7f0 msgr2=0x7fe52012ac50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6808/1506640341 conn(0x7fe52012a7f0 0x7fe52012ac50 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52012b4f0 osd=6 1 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 10 client.38048.objecter close_session for osd.7 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe52012de20 msgr2=0x7fe52012e260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.181:6812/1964192772 conn(0x7fe52012de20 0x7fe52012e260 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 20 client.38048.objecter put_session s=0x7fe52012d760 osd=7 1 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 20 client.38048.objecter shutdown clearing up homeless session... 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 10 client.38048.objecter successfully canceled tick 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe51ed9d700 1 -- 172.21.15.162:0/3653708805 reap_dead start 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe508062c90 msgr2=0x7fe508065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe508062c90 0x7fe508065150 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 rx=0x7fe52010be60 tx=0x7fe514016040).stop 2024-02-13T00:52:00.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe52006b3d0 msgr2=0x7fe520110520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe52006b3d0 0x7fe520110520 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 rx=0x7fe5100041d0 tx=0x7fe510009f60).stop 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 shutdown_connections 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe52006b3d0 0x7fe520110520 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 --2- 172.21.15.162:0/3653708805 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe508062c90 0x7fe508065150 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 >> 172.21.15.162:0/3653708805 conn(0x7fe5200f38c0 msgr2=0x7fe5200f4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 shutdown_connections 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 -- 172.21.15.162:0/3653708805 wait complete. 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:00.669+0000 7fe524e49700 1 librados: shutdown 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.677 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.678 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:52:00.689 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:00.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:00.699 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:59.586442+0000 mon.a (mon.0) 3375 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:00.700 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:51:59.586515+0000 mon.a (mon.0) 3376 : cluster [DBG] osdmap e692: 8 total, 8 up, 5 in 2024-02-13T00:52:00.700 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:51:59.587266+0000 mon.a (mon.0) 3377 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:00.700 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:00.332693+0000 mon.a (mon.0) 3378 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:00.700 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:00.515580+0000 mon.c (mon.2) 486 : audit [DBG] from='client.? 172.21.15.162:0/3653708805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:00.700 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:00.538408+0000 mon.a (mon.0) 3379 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:00.700 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:00.538483+0000 mon.a (mon.0) 3380 : cluster [DBG] osdmap e693: 8 total, 8 up, 5 in 2024-02-13T00:52:00.700 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:00.541110+0000 mon.a (mon.0) 3381 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-122"}]: dispatch 2024-02-13T00:52:00.700 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:59.586442+0000 mon.a (mon.0) 3375 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:00.700 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:51:59.586515+0000 mon.a (mon.0) 3376 : cluster [DBG] osdmap e692: 8 total, 8 up, 5 in 2024-02-13T00:52:00.700 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:51:59.587266+0000 mon.a (mon.0) 3377 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:00.703 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:00.332693+0000 mon.a (mon.0) 3378 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:00.703 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:00.515580+0000 mon.c (mon.2) 486 : audit [DBG] from='client.? 172.21.15.162:0/3653708805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:00.703 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:00.538408+0000 mon.a (mon.0) 3379 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:00.704 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:00.538483+0000 mon.a (mon.0) 3380 : cluster [DBG] osdmap e693: 8 total, 8 up, 5 in 2024-02-13T00:52:00.704 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:00.541110+0000 mon.a (mon.0) 3381 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-122"}]: dispatch 2024-02-13T00:52:00.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:00.819 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:00.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:00.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:00.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:00.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:01.002 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:59.586442+0000 mon.a (mon.0) 3375 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:51:59.586515+0000 mon.a (mon.0) 3376 : cluster [DBG] osdmap e692: 8 total, 8 up, 5 in 2024-02-13T00:52:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:51:59.587266+0000 mon.a (mon.0) 3377 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:00.332693+0000 mon.a (mon.0) 3378 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:00.515580+0000 mon.c (mon.2) 486 : audit [DBG] from='client.? 172.21.15.162:0/3653708805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:00.538408+0000 mon.a (mon.0) 3379 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:00.538483+0000 mon.a (mon.0) 3380 : cluster [DBG] osdmap e693: 8 total, 8 up, 5 in 2024-02-13T00:52:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:00.541110+0000 mon.a (mon.0) 3381 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-122"}]: dispatch 2024-02-13T00:52:01.101 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:01.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:01.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:01.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:01.230 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:00.515153+0000 mgr.y (mgr.14544) 580 : cluster [DBG] pgmap v1045: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:52:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:01.540420+0000 mon.a (mon.0) 3382 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-122"}]': finished 2024-02-13T00:52:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:01.540488+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e694: 8 total, 8 up, 5 in 2024-02-13T00:52:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:01.541124+0000 mon.a (mon.0) 3384 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-122", "mode": "writeback"}]: dispatch 2024-02-13T00:52:02.076 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:00.515153+0000 mgr.y (mgr.14544) 580 : cluster [DBG] pgmap v1045: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:52:02.076 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:01.540420+0000 mon.a (mon.0) 3382 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-122"}]': finished 2024-02-13T00:52:02.077 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:01.540488+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e694: 8 total, 8 up, 5 in 2024-02-13T00:52:02.077 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:01.541124+0000 mon.a (mon.0) 3384 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-122", "mode": "writeback"}]: dispatch 2024-02-13T00:52:02.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:00.515153+0000 mgr.y (mgr.14544) 580 : cluster [DBG] pgmap v1045: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:52:02.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:01.540420+0000 mon.a (mon.0) 3382 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-122"}]': finished 2024-02-13T00:52:02.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:01.540488+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e694: 8 total, 8 up, 5 in 2024-02-13T00:52:02.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:01.541124+0000 mon.a (mon.0) 3384 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-122", "mode": "writeback"}]: dispatch 2024-02-13T00:52:02.078 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:02.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:02.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:02.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-13T00:52:02.129 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:02.167 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:02.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:02.215 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:02.237 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:02.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:02.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:02.371 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:02.391 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:02.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:02.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:02.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:52:02.524 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:02.544 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:52:02.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:02.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:52:02.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:02.537119+0000 mon.a (mon.0) 3385 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:52:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:02.540932+0000 mon.a (mon.0) 3386 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-122", "mode": "writeback"}]': finished 2024-02-13T00:52:02.678 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:02.541015+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e695: 8 total, 8 up, 5 in 2024-02-13T00:52:02.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:02.537119+0000 mon.a (mon.0) 3385 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:52:02.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:02.540932+0000 mon.a (mon.0) 3386 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-122", "mode": "writeback"}]': finished 2024-02-13T00:52:02.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:02.541015+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e695: 8 total, 8 up, 5 in 2024-02-13T00:52:02.679 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:02.696 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:02.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:02.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:02.537119+0000 mon.a (mon.0) 3385 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:52:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:02.540932+0000 mon.a (mon.0) 3386 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-122", "mode": "writeback"}]': finished 2024-02-13T00:52:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:02.541015+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e695: 8 total, 8 up, 5 in 2024-02-13T00:52:02.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:02.827 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:03.476 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:03.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:03.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:03.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:03.610 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:03.630 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:03.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:03.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:03.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:52:03.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:03.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:02.515568+0000 mgr.y (mgr.14544) 581 : cluster [DBG] pgmap v1048: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:03.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:03.542380+0000 mon.a (mon.0) 3388 : cluster [DBG] osdmap e696: 8 total, 8 up, 5 in 2024-02-13T00:52:03.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:02.515568+0000 mgr.y (mgr.14544) 581 : cluster [DBG] pgmap v1048: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:03.758 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:03.542380+0000 mon.a (mon.0) 3388 : cluster [DBG] osdmap e696: 8 total, 8 up, 5 in 2024-02-13T00:52:03.760 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:04.009 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:04.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:04.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:04.033 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:04.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:02.515568+0000 mgr.y (mgr.14544) 581 : cluster [DBG] pgmap v1048: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:03.542380+0000 mon.a (mon.0) 3388 : cluster [DBG] osdmap e696: 8 total, 8 up, 5 in 2024-02-13T00:52:04.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:04.139 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:04.262 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:04.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:04.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:04.398 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:04.419 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:04.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:04.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:04.553 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:04.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.621+0000 7ff324886700 1 -- 172.21.15.162:0/3480064723 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fdcb0 msgr2=0x7ff3200fe090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.621+0000 7ff324886700 1 --2- 172.21.15.162:0/3480064723 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fdcb0 0x7ff3200fe090 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7ff314004870 tx=0x7ff314009cb0).stop 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.621+0000 7ff324886700 1 -- 172.21.15.162:0/3480064723 shutdown_connections 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.621+0000 7ff324886700 1 --2- 172.21.15.162:0/3480064723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200f6920 0x7ff320104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.621+0000 7ff324886700 1 --2- 172.21.15.162:0/3480064723 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fdcb0 0x7ff3200fe090 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.621+0000 7ff324886700 1 --2- 172.21.15.162:0/3480064723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3200fe610 0x7ff3200f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.621+0000 7ff324886700 1 -- 172.21.15.162:0/3480064723 >> 172.21.15.162:0/3480064723 conn(0x7ff320070080 msgr2=0x7ff3200f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.621+0000 7ff324886700 1 -- 172.21.15.162:0/3480064723 shutdown_connections 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.621+0000 7ff324886700 1 -- 172.21.15.162:0/3480064723 wait complete. 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff324886700 1 librados: starting msgr at 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff324886700 1 librados: starting objecter 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff324886700 1 Processor -- start 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff324886700 1 -- start start 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff324886700 1 librados: setting wanted keys 2024-02-13T00:52:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff324886700 1 librados: calling monclient init 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff324886700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200f6920 0x7ff3200f7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff324886700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fdcb0 0x7ff3200f7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff324886700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3200fe610 0x7ff3200f7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff324886700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3201166f0 con 0x7ff3200f6920 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff324886700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff320116860 con 0x7ff3200fe610 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff324886700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff320116a20 con 0x7ff3200fdcb0 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff31e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200f6920 0x7ff3200f7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff31dd9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fdcb0 0x7ff3200f7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff31e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200f6920 0x7ff3200f7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:55314/0 (socket says 172.21.15.162:55314) 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.622+0000 7ff31e59c700 1 -- 172.21.15.162:0/2180056885 learned_addr learned my addr 172.21.15.162:0/2180056885 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff31ed9d700 1 --2- 172.21.15.162:0/2180056885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3200fe610 0x7ff3200f7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff31e59c700 1 -- 172.21.15.162:0/2180056885 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fdcb0 msgr2=0x7ff3200f7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff31e59c700 1 --2- 172.21.15.162:0/2180056885 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fdcb0 0x7ff3200f7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:04.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff31e59c700 1 -- 172.21.15.162:0/2180056885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3200fe610 msgr2=0x7ff3200f7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff31e59c700 1 --2- 172.21.15.162:0/2180056885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3200fe610 0x7ff3200f7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff31e59c700 1 -- 172.21.15.162:0/2180056885 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3180097c0 con 0x7ff3200f6920 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff31dd9b700 1 --2- 172.21.15.162:0/2180056885 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fdcb0 0x7ff3200f7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff31e59c700 1 --2- 172.21.15.162:0/2180056885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200f6920 0x7ff3200f7510 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 rx=0x7ff3140043a0 tx=0x7ff31400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff30f7fe700 10 client.?.objecter ms_handle_connect 0x7ff3200f6920 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff30f7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff30f7fe700 1 -- 172.21.15.162:0/2180056885 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31400e070 con 0x7ff3200f6920 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff324886700 1 -- 172.21.15.162:0/2180056885 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff314005040 con 0x7ff3200f6920 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff30f7fe700 1 -- 172.21.15.162:0/2180056885 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff314009710 con 0x7ff3200f6920 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff324886700 10 client.27957.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff30f7fe700 1 -- 172.21.15.162:0/2180056885 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314019070 con 0x7ff3200f6920 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff324886700 1 -- 172.21.15.162:0/2180056885 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff320116f10 con 0x7ff3200f6920 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.623+0000 7ff324886700 1 librados: init done 2024-02-13T00:52:04.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.624+0000 7ff324886700 1 -- 172.21.15.162:0/2180056885 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32005fb80 con 0x7ff3200f6920 2024-02-13T00:52:04.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.624+0000 7ff30f7fe700 1 -- 172.21.15.162:0/2180056885 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff3140090d0 con 0x7ff3200f6920 2024-02-13T00:52:04.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.625+0000 7ff30f7fe700 1 --2- 172.21.15.162:0/2180056885 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff308062c90 0x7ff308065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:04.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.625+0000 7ff30f7fe700 1 -- 172.21.15.162:0/2180056885 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(697..697 src has 449..697) v4 ==== 8591+0+0 (secure 0 0 0) 0x7ff31407de20 con 0x7ff3200f6920 2024-02-13T00:52:04.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.625+0000 7ff30f7fe700 10 client.27957.objecter ms_dispatch 0x7ff320001550 osd_map(697..697 src has 449..697) v4 2024-02-13T00:52:04.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.625+0000 7ff30f7fe700 3 client.27957.objecter handle_osd_map got epochs [697,697] > 0 2024-02-13T00:52:04.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.625+0000 7ff30f7fe700 3 client.27957.objecter handle_osd_map decoding full epoch 697 2024-02-13T00:52:04.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.625+0000 7ff31dd9b700 1 --2- 172.21.15.162:0/2180056885 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff308062c90 0x7ff308065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:04.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.625+0000 7ff30f7fe700 20 client.27957.objecter dump_active .. 0 homeless 2024-02-13T00:52:04.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.625+0000 7ff30f7fe700 10 client.27957.objecter ms_handle_connect 0x7ff308062c90 2024-02-13T00:52:04.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.625+0000 7ff31dd9b700 1 --2- 172.21.15.162:0/2180056885 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff308062c90 0x7ff308065150 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 rx=0x7ff3200f8cb0 tx=0x7ff31000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:04.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.630+0000 7ff30f7fe700 1 -- 172.21.15.162:0/2180056885 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff314051230 con 0x7ff3200f6920 2024-02-13T00:52:04.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:04.809+0000 7ff324886700 1 -- 172.21.15.162:0/2180056885 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7ff3200475e0 con 0x7ff3200f6920 2024-02-13T00:52:04.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:04.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:05.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:05.044 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:05.066 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:05.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:05.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:05.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:05.199 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:05.219 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:05.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:05.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:05.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:05.355 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:05.408 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:05.409 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:04.342215+0000 mon.a (mon.0) 3389 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:05.410 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:04.515947+0000 mgr.y (mgr.14544) 582 : cluster [DBG] pgmap v1051: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:52:05.410 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:04.543548+0000 mon.a (mon.0) 3390 : cluster [DBG] osdmap e697: 8 total, 8 up, 5 in 2024-02-13T00:52:05.410 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:04.588915+0000 mon.a (mon.0) 3391 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:05.410 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:04.810969+0000 mon.a (mon.0) 3392 : audit [INF] from='client.? 172.21.15.162:0/2180056885' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:05.410 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:04.342215+0000 mon.a (mon.0) 3389 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:05.410 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:04.515947+0000 mgr.y (mgr.14544) 582 : cluster [DBG] pgmap v1051: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:52:05.410 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:04.543548+0000 mon.a (mon.0) 3390 : cluster [DBG] osdmap e697: 8 total, 8 up, 5 in 2024-02-13T00:52:05.410 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:04.588915+0000 mon.a (mon.0) 3391 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:05.410 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:04.810969+0000 mon.a (mon.0) 3392 : audit [INF] from='client.? 172.21.15.162:0/2180056885' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:05.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:05.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:05.545 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:05.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.554+0000 7ff30f7fe700 1 -- 172.21.15.162:0/2180056885 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v698) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff314023030 con 0x7ff3200f6920 2024-02-13T00:52:05.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.560+0000 7ff324886700 10 librados: watch_flush enter 2024-02-13T00:52:05.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 10 librados: watch_flush exit 2024-02-13T00:52:05.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 20 client.27957.objecter shutdown clearing up homeless session... 2024-02-13T00:52:05.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 10 client.27957.objecter successfully canceled tick 2024-02-13T00:52:05.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 -- 172.21.15.162:0/2180056885 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff308062c90 msgr2=0x7ff308065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 --2- 172.21.15.162:0/2180056885 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff308062c90 0x7ff308065150 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 rx=0x7ff3200f8cb0 tx=0x7ff31000b040).stop 2024-02-13T00:52:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 -- 172.21.15.162:0/2180056885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200f6920 msgr2=0x7ff3200f7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 --2- 172.21.15.162:0/2180056885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200f6920 0x7ff3200f7510 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 rx=0x7ff3140043a0 tx=0x7ff31400b040).stop 2024-02-13T00:52:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 -- 172.21.15.162:0/2180056885 shutdown_connections 2024-02-13T00:52:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 --2- 172.21.15.162:0/2180056885 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff308062c90 0x7ff308065150 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 --2- 172.21.15.162:0/2180056885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3200fe610 0x7ff3200f7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 --2- 172.21.15.162:0/2180056885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3200f6920 0x7ff3200f7510 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 --2- 172.21.15.162:0/2180056885 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3200fdcb0 0x7ff3200f7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 -- 172.21.15.162:0/2180056885 >> 172.21.15.162:0/2180056885 conn(0x7ff320070080 msgr2=0x7ff3200fba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 -- 172.21.15.162:0/2180056885 shutdown_connections 2024-02-13T00:52:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 -- 172.21.15.162:0/2180056885 wait complete. 2024-02-13T00:52:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.562+0000 7ff324886700 1 librados: shutdown 2024-02-13T00:52:05.615 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:52:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:05.615 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:05.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:05.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 -- 172.21.15.162:0/213873205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd1bc0f95e0 msgr2=0x7fd1bc101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:05.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 --2- 172.21.15.162:0/213873205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd1bc0f95e0 0x7fd1bc101d80 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 rx=0x7fd1a80047b0 tx=0x7fd1a8009cb0).stop 2024-02-13T00:52:05.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 -- 172.21.15.162:0/213873205 shutdown_connections 2024-02-13T00:52:05.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 --2- 172.21.15.162:0/213873205 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1bc102270 0x7fd1bc104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:05.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 --2- 172.21.15.162:0/213873205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1bc0f8d10 0x7fd1bc0f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:05.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 --2- 172.21.15.162:0/213873205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd1bc0f95e0 0x7fd1bc101d80 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 -- 172.21.15.162:0/213873205 >> 172.21.15.162:0/213873205 conn(0x7fd1bc06eb90 msgr2=0x7fd1bc0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 -- 172.21.15.162:0/213873205 shutdown_connections 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 -- 172.21.15.162:0/213873205 wait complete. 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 librados: starting msgr at 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 librados: starting objecter 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 Processor -- start 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 -- start start 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 librados: setting wanted keys 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.698+0000 7fd1c2c33700 1 librados: calling monclient init 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1c2c33700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1bc0f8d10 0x7fd1bc0f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1c2c33700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd1bc0f95e0 0x7fd1bc0f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1c2c33700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1bc102270 0x7fd1bc0f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1c2c33700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc116740 con 0x7fd1bc0f8d10 2024-02-13T00:52:05.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1c2c33700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc1168b0 con 0x7fd1bc102270 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1c2c33700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc116a70 con 0x7fd1bc0f95e0 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1bbfff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd1bc0f95e0 0x7fd1bc0f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1bbfff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd1bc0f95e0 0x7fd1bc0f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:50394/0 (socket says 172.21.15.162:50394) 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1bbfff700 1 -- 172.21.15.162:0/810333527 learned_addr learned my addr 172.21.15.162:0/810333527 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1c11d0700 1 --2- 172.21.15.162:0/810333527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1bc102270 0x7fd1bc0f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1c09cf700 1 --2- 172.21.15.162:0/810333527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1bc0f8d10 0x7fd1bc0f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1bbfff700 1 -- 172.21.15.162:0/810333527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1bc102270 msgr2=0x7fd1bc0f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1bbfff700 1 --2- 172.21.15.162:0/810333527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1bc102270 0x7fd1bc0f76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1bbfff700 1 -- 172.21.15.162:0/810333527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1bc0f8d10 msgr2=0x7fd1bc0f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1bbfff700 1 --2- 172.21.15.162:0/810333527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1bc0f8d10 0x7fd1bc0f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1bbfff700 1 -- 172.21.15.162:0/810333527 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b0005040 con 0x7fd1bc0f95e0 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1c11d0700 1 --2- 172.21.15.162:0/810333527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1bc102270 0x7fd1bc0f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1bbfff700 1 --2- 172.21.15.162:0/810333527 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd1bc0f95e0 0x7fd1bc0f71b0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 rx=0x7fd1a8009b10 tx=0x7fd1a8009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1b9ffb700 10 client.?.objecter ms_handle_connect 0x7fd1bc0f95e0 2024-02-13T00:52:05.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1b9ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:05.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1b9ffb700 1 -- 172.21.15.162:0/810333527 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8018070 con 0x7fd1bc0f95e0 2024-02-13T00:52:05.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1b9ffb700 1 -- 172.21.15.162:0/810333527 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1a8013da0 con 0x7fd1bc0f95e0 2024-02-13T00:52:05.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1c2c33700 1 -- 172.21.15.162:0/810333527 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a8005040 con 0x7fd1bc0f95e0 2024-02-13T00:52:05.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.699+0000 7fd1b9ffb700 1 -- 172.21.15.162:0/810333527 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a801dc10 con 0x7fd1bc0f95e0 2024-02-13T00:52:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.700+0000 7fd1c2c33700 10 client.38060.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.700+0000 7fd1c2c33700 1 -- 172.21.15.162:0/810333527 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc116f10 con 0x7fd1bc0f95e0 2024-02-13T00:52:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.700+0000 7fd1c2c33700 1 librados: init done 2024-02-13T00:52:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.701+0000 7fd1c2c33700 1 -- 172.21.15.162:0/810333527 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc0475e0 con 0x7fd1bc0f95e0 2024-02-13T00:52:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.701+0000 7fd1b9ffb700 1 -- 172.21.15.162:0/810333527 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fd1a801d420 con 0x7fd1bc0f95e0 2024-02-13T00:52:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.701+0000 7fd1b9ffb700 1 --2- 172.21.15.162:0/810333527 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd1ac062c90 0x7fd1ac065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.701+0000 7fd1b9ffb700 1 -- 172.21.15.162:0/810333527 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(698..698 src has 449..698) v4 ==== 8736+0+0 (secure 0 0 0) 0x7fd1a807e8f0 con 0x7fd1bc0f95e0 2024-02-13T00:52:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.701+0000 7fd1b9ffb700 10 client.38060.objecter ms_dispatch 0x7fd1bc001550 osd_map(698..698 src has 449..698) v4 2024-02-13T00:52:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.701+0000 7fd1b9ffb700 3 client.38060.objecter handle_osd_map got epochs [698,698] > 0 2024-02-13T00:52:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.701+0000 7fd1b9ffb700 3 client.38060.objecter handle_osd_map decoding full epoch 698 2024-02-13T00:52:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.701+0000 7fd1c09cf700 1 --2- 172.21.15.162:0/810333527 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd1ac062c90 0x7fd1ac065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.702+0000 7fd1b9ffb700 20 client.38060.objecter dump_active .. 0 homeless 2024-02-13T00:52:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.702+0000 7fd1b9ffb700 10 client.38060.objecter ms_handle_connect 0x7fd1ac062c90 2024-02-13T00:52:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.702+0000 7fd1c09cf700 1 --2- 172.21.15.162:0/810333527 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd1ac062c90 0x7fd1ac065150 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7fd1b00047e0 tx=0x7fd1b000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:05.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.706+0000 7fd1b9ffb700 1 -- 172.21.15.162:0/810333527 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd1a8051b20 con 0x7fd1bc0f95e0 2024-02-13T00:52:05.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:05.745 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:05.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:04.342215+0000 mon.a (mon.0) 3389 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:04.515947+0000 mgr.y (mgr.14544) 582 : cluster [DBG] pgmap v1051: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:52:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:04.543548+0000 mon.a (mon.0) 3390 : cluster [DBG] osdmap e697: 8 total, 8 up, 5 in 2024-02-13T00:52:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:04.588915+0000 mon.a (mon.0) 3391 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:04.810969+0000 mon.a (mon.0) 3392 : audit [INF] from='client.? 172.21.15.162:0/2180056885' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:05.876 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:05.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:05.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:05.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:05.894+0000 7fd1c2c33700 1 -- 172.21.15.162:0/810333527 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"} v 0) v1 -- 0x7fd1bc05fb80 con 0x7fd1bc0f95e0 2024-02-13T00:52:05.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:06.013 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:06.034 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:06.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:06.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:06.166 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:52:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:52:06.169 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:06.189 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:06.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:06.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:06.322 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:06.341 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:06.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:06.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:06.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:06.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.473436+0000 mon.c (mon.2) 487 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [0, 4]}]: dispatch 2024-02-13T00:52:06.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.473683+0000 mon.c (mon.2) 488 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 6]}]: dispatch 2024-02-13T00:52:06.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.473853+0000 mon.a (mon.0) 3393 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [0, 4]}]: dispatch 2024-02-13T00:52:06.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.473951+0000 mon.c (mon.2) 489 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [6, 1]}]: dispatch 2024-02-13T00:52:06.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.474169+0000 mon.c (mon.2) 490 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]: dispatch 2024-02-13T00:52:06.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.474380+0000 mon.c (mon.2) 491 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "245.9", "id": [4, 0]}]: dispatch 2024-02-13T00:52:06.456 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.474390+0000 mon.a (mon.0) 3394 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 6]}]: dispatch 2024-02-13T00:52:06.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.474715+0000 mon.a (mon.0) 3395 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [6, 1]}]: dispatch 2024-02-13T00:52:06.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.474894+0000 mon.a (mon.0) 3396 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]: dispatch 2024-02-13T00:52:06.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.475057+0000 mon.a (mon.0) 3397 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "245.9", "id": [4, 0]}]: dispatch 2024-02-13T00:52:06.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:05.541755+0000 mon.a (mon.0) 3398 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:52:06.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.554821+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:52:06.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.554893+0000 mon.a (mon.0) 3400 : audit [INF] from='client.? 172.21.15.162:0/2180056885' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:52:06.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.554954+0000 mon.a (mon.0) 3401 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [0, 4]}]': finished 2024-02-13T00:52:06.457 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:52:06] "GET /metrics HTTP/1.1" 200 189585 "" "Prometheus/2.33.4" 2024-02-13T00:52:06.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.473436+0000 mon.c (mon.2) 487 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [0, 4]}]: dispatch 2024-02-13T00:52:06.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.473683+0000 mon.c (mon.2) 488 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 6]}]: dispatch 2024-02-13T00:52:06.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.473853+0000 mon.a (mon.0) 3393 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [0, 4]}]: dispatch 2024-02-13T00:52:06.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.473951+0000 mon.c (mon.2) 489 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [6, 1]}]: dispatch 2024-02-13T00:52:06.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.474169+0000 mon.c (mon.2) 490 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]: dispatch 2024-02-13T00:52:06.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.474380+0000 mon.c (mon.2) 491 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "245.9", "id": [4, 0]}]: dispatch 2024-02-13T00:52:06.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.474390+0000 mon.a (mon.0) 3394 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 6]}]: dispatch 2024-02-13T00:52:06.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.474715+0000 mon.a (mon.0) 3395 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [6, 1]}]: dispatch 2024-02-13T00:52:06.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.474894+0000 mon.a (mon.0) 3396 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]: dispatch 2024-02-13T00:52:06.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.475057+0000 mon.a (mon.0) 3397 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "245.9", "id": [4, 0]}]: dispatch 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:05.541755+0000 mon.a (mon.0) 3398 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.554821+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.554893+0000 mon.a (mon.0) 3400 : audit [INF] from='client.? 172.21.15.162:0/2180056885' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.554954+0000 mon.a (mon.0) 3401 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [0, 4]}]': finished 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.555025+0000 mon.a (mon.0) 3402 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 6]}]': finished 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.555094+0000 mon.a (mon.0) 3403 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [6, 1]}]': finished 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.555157+0000 mon.a (mon.0) 3404 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]': finished 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.555211+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "245.9", "id": [4, 0]}]': finished 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:05.555261+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e698: 8 total, 8 up, 5 in 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.584489+0000 mon.a (mon.0) 3407 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-122"}]: dispatch 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.874179+0000 mgr.y (mgr.14544) 583 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.895136+0000 mon.c (mon.2) 492 : audit [INF] from='client.? 172.21.15.162:0/810333527' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-02-13T00:52:06.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:05.895468+0000 mon.a (mon.0) 3408 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-02-13T00:52:06.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.555025+0000 mon.a (mon.0) 3402 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 6]}]': finished 2024-02-13T00:52:06.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.555094+0000 mon.a (mon.0) 3403 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [6, 1]}]': finished 2024-02-13T00:52:06.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.555157+0000 mon.a (mon.0) 3404 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]': finished 2024-02-13T00:52:06.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.555211+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "245.9", "id": [4, 0]}]': finished 2024-02-13T00:52:06.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:05.555261+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e698: 8 total, 8 up, 5 in 2024-02-13T00:52:06.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.584489+0000 mon.a (mon.0) 3407 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-122"}]: dispatch 2024-02-13T00:52:06.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.874179+0000 mgr.y (mgr.14544) 583 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:06.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.895136+0000 mon.c (mon.2) 492 : audit [INF] from='client.? 172.21.15.162:0/810333527' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-02-13T00:52:06.460 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:05.895468+0000 mon.a (mon.0) 3408 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-02-13T00:52:06.474 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:06.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.473436+0000 mon.c (mon.2) 487 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [0, 4]}]: dispatch 2024-02-13T00:52:06.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.473683+0000 mon.c (mon.2) 488 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 6]}]: dispatch 2024-02-13T00:52:06.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.473853+0000 mon.a (mon.0) 3393 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [0, 4]}]: dispatch 2024-02-13T00:52:06.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.473951+0000 mon.c (mon.2) 489 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [6, 1]}]: dispatch 2024-02-13T00:52:06.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.474169+0000 mon.c (mon.2) 490 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]: dispatch 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.474380+0000 mon.c (mon.2) 491 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "245.9", "id": [4, 0]}]: dispatch 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.474390+0000 mon.a (mon.0) 3394 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 6]}]: dispatch 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.474715+0000 mon.a (mon.0) 3395 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [6, 1]}]: dispatch 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.474894+0000 mon.a (mon.0) 3396 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]: dispatch 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.475057+0000 mon.a (mon.0) 3397 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "245.9", "id": [4, 0]}]: dispatch 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:05.541755+0000 mon.a (mon.0) 3398 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.554821+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.554893+0000 mon.a (mon.0) 3400 : audit [INF] from='client.? 172.21.15.162:0/2180056885' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.554954+0000 mon.a (mon.0) 3401 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [0, 4]}]': finished 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.555025+0000 mon.a (mon.0) 3402 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 6]}]': finished 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.555094+0000 mon.a (mon.0) 3403 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [6, 1]}]': finished 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.555157+0000 mon.a (mon.0) 3404 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [4, 6]}]': finished 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.555211+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "245.9", "id": [4, 0]}]': finished 2024-02-13T00:52:06.546 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:05.555261+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e698: 8 total, 8 up, 5 in 2024-02-13T00:52:06.547 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.584489+0000 mon.a (mon.0) 3407 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-122"}]: dispatch 2024-02-13T00:52:06.547 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.874179+0000 mgr.y (mgr.14544) 583 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:06.547 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.895136+0000 mon.c (mon.2) 492 : audit [INF] from='client.? 172.21.15.162:0/810333527' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-02-13T00:52:06.547 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:05.895468+0000 mon.a (mon.0) 3408 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-02-13T00:52:06.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.559+0000 7fd1b9ffb700 1 -- 172.21.15.162:0/810333527 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]=0 clear 4.4 pg_upmap_items mapping v699) v1 ==== 116+0+0 (secure 0 0 0) 0x7fd1a8051710 con 0x7fd1bc0f95e0 2024-02-13T00:52:06.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.561+0000 7fd1c2c33700 10 librados: watch_flush enter 2024-02-13T00:52:06.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.561+0000 7fd1c2c33700 10 librados: watch_flush exit 2024-02-13T00:52:06.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.561+0000 7fd1c2c33700 20 client.38060.objecter shutdown clearing up homeless session... 2024-02-13T00:52:06.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.561+0000 7fd1c2c33700 10 client.38060.objecter successfully canceled tick 2024-02-13T00:52:06.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.562+0000 7fd1c2c33700 1 -- 172.21.15.162:0/810333527 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd1ac062c90 msgr2=0x7fd1ac065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:06.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.562+0000 7fd1c2c33700 1 --2- 172.21.15.162:0/810333527 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd1ac062c90 0x7fd1ac065150 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7fd1b00047e0 tx=0x7fd1b000b040).stop 2024-02-13T00:52:06.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.562+0000 7fd1c2c33700 1 -- 172.21.15.162:0/810333527 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd1bc0f95e0 msgr2=0x7fd1bc0f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:06.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.562+0000 7fd1c2c33700 1 --2- 172.21.15.162:0/810333527 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd1bc0f95e0 0x7fd1bc0f71b0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 rx=0x7fd1a8009b10 tx=0x7fd1a8009cb0).stop 2024-02-13T00:52:06.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.563+0000 7fd1c2c33700 1 -- 172.21.15.162:0/810333527 shutdown_connections 2024-02-13T00:52:06.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.563+0000 7fd1c2c33700 1 --2- 172.21.15.162:0/810333527 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd1ac062c90 0x7fd1ac065150 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:06.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.563+0000 7fd1c2c33700 1 --2- 172.21.15.162:0/810333527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1bc102270 0x7fd1bc0f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:06.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.563+0000 7fd1c2c33700 1 --2- 172.21.15.162:0/810333527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1bc0f8d10 0x7fd1bc0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:06.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.563+0000 7fd1c2c33700 1 --2- 172.21.15.162:0/810333527 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd1bc0f95e0 0x7fd1bc0f71b0 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:06.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.563+0000 7fd1c2c33700 1 -- 172.21.15.162:0/810333527 >> 172.21.15.162:0/810333527 conn(0x7fd1bc06eb90 msgr2=0x7fd1bc102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:06.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.563+0000 7fd1c2c33700 1 -- 172.21.15.162:0/810333527 shutdown_connections 2024-02-13T00:52:06.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.564+0000 7fd1c2c33700 1 -- 172.21.15.162:0/810333527 wait complete. 2024-02-13T00:52:06.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:06.564+0000 7fd1c2c33700 1 librados: shutdown 2024-02-13T00:52:06.568 INFO:teuthology.orchestra.run.smithi162.stderr:clear 4.4 pg_upmap_items mapping 2024-02-13T00:52:06.573 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:06.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:06.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:06.708 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:06.728 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:06.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:06.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:06.858 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:06.930 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:06.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:06.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:07.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:07.063 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:07.083 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:07.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:07.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:07.115 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:07.187 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:07.214 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:06.516352+0000 mgr.y (mgr.14544) 584 : cluster [DBG] pgmap v1054: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-13T00:52:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:06.558270+0000 mon.a (mon.0) 3409 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-122"}]': finished 2024-02-13T00:52:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:06.558345+0000 mon.a (mon.0) 3410 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]': finished 2024-02-13T00:52:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:06.558398+0000 mon.a (mon.0) 3411 : cluster [DBG] osdmap e699: 8 total, 8 up, 5 in 2024-02-13T00:52:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:06.516352+0000 mgr.y (mgr.14544) 584 : cluster [DBG] pgmap v1054: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-13T00:52:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:06.558270+0000 mon.a (mon.0) 3409 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-122"}]': finished 2024-02-13T00:52:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:06.558345+0000 mon.a (mon.0) 3410 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]': finished 2024-02-13T00:52:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:06.558398+0000 mon.a (mon.0) 3411 : cluster [DBG] osdmap e699: 8 total, 8 up, 5 in 2024-02-13T00:52:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:06.516352+0000 mgr.y (mgr.14544) 584 : cluster [DBG] pgmap v1054: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-13T00:52:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:06.558270+0000 mon.a (mon.0) 3409 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-122"}]': finished 2024-02-13T00:52:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:06.558345+0000 mon.a (mon.0) 3410 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]': finished 2024-02-13T00:52:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:06.558398+0000 mon.a (mon.0) 3411 : cluster [DBG] osdmap e699: 8 total, 8 up, 5 in 2024-02-13T00:52:08.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:52:08.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:07.554230+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e700: 8 total, 8 up, 5 in 2024-02-13T00:52:08.684 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:07.577229+0000 mon.a (mon.0) 3413 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:08.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:07.554230+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e700: 8 total, 8 up, 5 in 2024-02-13T00:52:08.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:07.577229+0000 mon.a (mon.0) 3413 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:09.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:07.554230+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e700: 8 total, 8 up, 5 in 2024-02-13T00:52:09.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:07.577229+0000 mon.a (mon.0) 3413 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:09.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:52:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:52:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:52:09.327 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:52:09.327 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:52:09.327 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:52:09.327 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:52:09.327 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:52:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:08.516712+0000 mgr.y (mgr.14544) 585 : cluster [DBG] pgmap v1057: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-13T00:52:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:08.570152+0000 mon.a (mon.0) 3414 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:52:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:08.570211+0000 mon.a (mon.0) 3415 : cluster [DBG] osdmap e701: 8 total, 8 up, 5 in 2024-02-13T00:52:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:08.570656+0000 mon.a (mon.0) 3416 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-122"}]: dispatch 2024-02-13T00:52:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:08.516712+0000 mgr.y (mgr.14544) 585 : cluster [DBG] pgmap v1057: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-13T00:52:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:08.570152+0000 mon.a (mon.0) 3414 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:52:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:08.570211+0000 mon.a (mon.0) 3415 : cluster [DBG] osdmap e701: 8 total, 8 up, 5 in 2024-02-13T00:52:09.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:08.570656+0000 mon.a (mon.0) 3416 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-122"}]: dispatch 2024-02-13T00:52:09.966 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:09.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:09.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-13T00:52:10.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:08.516712+0000 mgr.y (mgr.14544) 585 : cluster [DBG] pgmap v1057: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-13T00:52:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:08.570152+0000 mon.a (mon.0) 3414 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:52:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:08.570211+0000 mon.a (mon.0) 3415 : cluster [DBG] osdmap e701: 8 total, 8 up, 5 in 2024-02-13T00:52:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:08.570656+0000 mon.a (mon.0) 3416 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-122"}]: dispatch 2024-02-13T00:52:10.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:10.102 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:10.123 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:10.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:10.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:10.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:10.255 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:10.275 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:10.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:10.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:10.407 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:10.426 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:10.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:10.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:10.561 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:10.579 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:10.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:10.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:10.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:10.712 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:10.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:09.567111+0000 mon.a (mon.0) 3417 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:52:10.967 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:09.578326+0000 mon.a (mon.0) 3418 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-122"}]': finished 2024-02-13T00:52:10.967 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:09.578395+0000 mon.a (mon.0) 3419 : cluster [DBG] osdmap e702: 8 total, 8 up, 5 in 2024-02-13T00:52:10.967 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:10.541591+0000 mon.a (mon.0) 3420 : cluster [DBG] osdmap e703: 8 total, 8 up, 5 in 2024-02-13T00:52:10.967 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:09.567111+0000 mon.a (mon.0) 3417 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:52:10.967 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:09.578326+0000 mon.a (mon.0) 3418 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-122"}]': finished 2024-02-13T00:52:10.967 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:09.578395+0000 mon.a (mon.0) 3419 : cluster [DBG] osdmap e702: 8 total, 8 up, 5 in 2024-02-13T00:52:10.967 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:10.541591+0000 mon.a (mon.0) 3420 : cluster [DBG] osdmap e703: 8 total, 8 up, 5 in 2024-02-13T00:52:11.160 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:11.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:11.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:11.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.788+0000 7fdc47536700 1 -- 172.21.15.162:0/2494948547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc400693c0 msgr2=0x7fdc40069840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.788+0000 7fdc47536700 1 --2- 172.21.15.162:0/2494948547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc400693c0 0x7fdc40069840 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7fdc3c0047b0 tx=0x7fdc3c009cb0).stop 2024-02-13T00:52:11.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.788+0000 7fdc47536700 1 -- 172.21.15.162:0/2494948547 shutdown_connections 2024-02-13T00:52:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.788+0000 7fdc47536700 1 --2- 172.21.15.162:0/2494948547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc40069d30 0x7fdc4010d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.788+0000 7fdc47536700 1 --2- 172.21.15.162:0/2494948547 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc40100220 0x7fdc40100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.788+0000 7fdc47536700 1 --2- 172.21.15.162:0/2494948547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc400693c0 0x7fdc40069840 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.788+0000 7fdc47536700 1 -- 172.21.15.162:0/2494948547 >> 172.21.15.162:0/2494948547 conn(0x7fdc40063df0 msgr2=0x7fdc400676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.788+0000 7fdc47536700 1 -- 172.21.15.162:0/2494948547 shutdown_connections 2024-02-13T00:52:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.788+0000 7fdc47536700 1 -- 172.21.15.162:0/2494948547 wait complete. 2024-02-13T00:52:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.788+0000 7fdc47536700 1 librados: starting msgr at 2024-02-13T00:52:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.788+0000 7fdc47536700 1 librados: starting objecter 2024-02-13T00:52:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.788+0000 7fdc47536700 1 Processor -- start 2024-02-13T00:52:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc47536700 1 -- start start 2024-02-13T00:52:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc47536700 1 librados: setting wanted keys 2024-02-13T00:52:11.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc47536700 1 librados: calling monclient init 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc47536700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc400693c0 0x7fdc40116b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc47536700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc40069d30 0x7fdc40117030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc47536700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc40100220 0x7fdc40110ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc47536700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc401111d0 con 0x7fdc400693c0 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc47536700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc40111340 con 0x7fdc40100220 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc47536700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc40111500 con 0x7fdc40069d30 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc452d2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc400693c0 0x7fdc40116b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc452d2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc400693c0 0x7fdc40116b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:55344/0 (socket says 172.21.15.162:55344) 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc44ad1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc40069d30 0x7fdc40117030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc452d2700 1 -- 172.21.15.162:0/2490891584 learned_addr learned my addr 172.21.15.162:0/2490891584 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc45ad3700 1 --2- 172.21.15.162:0/2490891584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc40100220 0x7fdc40110ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc452d2700 1 -- 172.21.15.162:0/2490891584 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc40069d30 msgr2=0x7fdc40117030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc452d2700 1 --2- 172.21.15.162:0/2490891584 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc40069d30 0x7fdc40117030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc452d2700 1 -- 172.21.15.162:0/2490891584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc40100220 msgr2=0x7fdc40110ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc452d2700 1 --2- 172.21.15.162:0/2490891584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc40100220 0x7fdc40110ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc452d2700 1 -- 172.21.15.162:0/2490891584 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc30005040 con 0x7fdc400693c0 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc44ad1700 1 --2- 172.21.15.162:0/2490891584 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc40069d30 0x7fdc40117030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.789+0000 7fdc45ad3700 1 --2- 172.21.15.162:0/2490891584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc40100220 0x7fdc40110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.790+0000 7fdc452d2700 1 --2- 172.21.15.162:0/2490891584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc400693c0 0x7fdc40116b40 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 rx=0x7fdc30009980 tx=0x7fdc30009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.790+0000 7fdc367fc700 10 client.?.objecter ms_handle_connect 0x7fdc400693c0 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.790+0000 7fdc367fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.790+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30007b00 con 0x7fdc400693c0 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.790+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3c005040 con 0x7fdc400693c0 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.790+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc30017070 con 0x7fdc400693c0 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.790+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3001aaf0 con 0x7fdc400693c0 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.791+0000 7fdc47536700 10 client.27969.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:11.274 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.791+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc40111ae0 con 0x7fdc400693c0 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.791+0000 7fdc47536700 1 librados: init done 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.792+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fdc300138a0 con 0x7fdc400693c0 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.793+0000 7fdc367fc700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdc2c062c90 0x7fdc2c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.793+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(703..703 src has 449..703) v4 ==== 8353+0+0 (secure 0 0 0) 0x7fdc30080eb0 con 0x7fdc400693c0 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.793+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 osd_map(703..703 src has 449..703) v4 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.793+0000 7fdc367fc700 3 client.27969.objecter handle_osd_map got epochs [703,703] > 0 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.793+0000 7fdc367fc700 3 client.27969.objecter handle_osd_map decoding full epoch 703 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.793+0000 7fdc44ad1700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdc2c062c90 0x7fdc2c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.793+0000 7fdc367fc700 20 client.27969.objecter dump_active .. 0 homeless 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.794+0000 7fdc367fc700 10 client.27969.objecter ms_handle_connect 0x7fdc2c062c90 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.794+0000 7fdc44ad1700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdc2c062c90 0x7fdc2c065150 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 rx=0x7fdc3c009da0 tx=0x7fdc3c009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.799+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fdc400603a0 con 0x7fdc400693c0 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v703) v1 ==== 54+0+16 (secure 0 0 0) 0x7fdc30054230 con 0x7fdc400693c0 2024-02-13T00:52:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc47536700 10 client.27969.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 1 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6801/2103743338 conn(0x7fdc40028590 0x7fdc40106ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc400475e0 osd=0 3 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc400475e0 osd=0 3 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc400475e0 osd=0 3 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 1 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc400475e0 osd=0 3 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 0 1 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc400475e0 osd=0 4 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc47536700 10 client.27969.objecter _send_command 1 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc4010f950 con 0x7fdc40028590 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.800+0000 7fdc45ad3700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6801/2103743338 conn(0x7fdc40028590 0x7fdc40106ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.801+0000 7fdc367fc700 10 client.27969.objecter ms_handle_connect 0x7fdc40028590 2024-02-13T00:52:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.801+0000 7fdc45ad3700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6801/2103743338 conn(0x7fdc40028590 0x7fdc40106ff0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:11.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.801+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fdc4010f950 con 0x7fdc40028590 2024-02-13T00:52:11.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.801+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:52:11.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.801+0000 7fdc367fc700 10 client.27969.objecter _finish_command 1 = system:0 2024-02-13T00:52:11.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.801+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc400475e0 osd=0 4 2024-02-13T00:52:11.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.801+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 0 1 2024-02-13T00:52:11.277 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:52:11.277 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.817+0000 7fdc47536700 10 client.27969.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.817+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 2 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.817+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc400475e0 osd=0 3 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.817+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc400475e0 osd=0 3 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.817+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc400475e0 osd=0 3 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.817+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 2 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.817+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc400475e0 osd=0 3 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.817+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 0 2 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.817+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc400475e0 osd=0 4 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.817+0000 7fdc47536700 10 client.27969.objecter _send_command 2 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.817+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc4005a0e0 con 0x7fdc40028590 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.818+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fdc4005a0e0 con 0x7fdc40028590 2024-02-13T00:52:11.278 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.818+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.818+0000 7fdc367fc700 10 client.27969.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.818+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc400475e0 osd=0 4 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.818+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 0 2 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc47536700 10 client.27969.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 3 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6805/150599780 conn(0x7fdc40027be0 0x7fdc40028020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc40108660 osd=1 3 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40108660 osd=1 3 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc40108660 osd=1 3 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 3 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc40108660 osd=1 3 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 1 3 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40108660 osd=1 4 2024-02-13T00:52:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc47536700 10 client.27969.objecter _send_command 3 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc401087d0 con 0x7fdc40027be0 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc452d2700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6805/150599780 conn(0x7fdc40027be0 0x7fdc40028020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc367fc700 10 client.27969.objecter ms_handle_connect 0x7fdc40027be0 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.819+0000 7fdc452d2700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6805/150599780 conn(0x7fdc40027be0 0x7fdc40028020 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.820+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fdc401087d0 con 0x7fdc40027be0 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.820+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.820+0000 7fdc367fc700 10 client.27969.objecter _finish_command 3 = system:0 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.820+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc40108660 osd=1 4 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.820+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 1 3 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.834+0000 7fdc47536700 10 client.27969.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.834+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 4 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.834+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc40108660 osd=1 3 2024-02-13T00:52:11.280 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.834+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40108660 osd=1 3 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.834+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc40108660 osd=1 3 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.834+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 4 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.834+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc40108660 osd=1 3 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.834+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 1 4 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.834+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40108660 osd=1 4 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.834+0000 7fdc47536700 10 client.27969.objecter _send_command 4 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.834+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc40027650 con 0x7fdc40027be0 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fdc40027650 con 0x7fdc40027be0 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc367fc700 10 client.27969.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc40108660 osd=1 4 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 1 4 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc47536700 10 client.27969.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:11.281 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 5 2024-02-13T00:52:11.282 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:52:11.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6809/3689719376 conn(0x7fdc401081c0 0x7fdc40122b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:11.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc40027a60 osd=2 3 2024-02-13T00:52:11.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40027a60 osd=2 3 2024-02-13T00:52:11.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc40027a60 osd=2 3 2024-02-13T00:52:11.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 5 2024-02-13T00:52:11.282 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc40027a60 osd=2 3 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 2 5 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40027a60 osd=2 4 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc47536700 10 client.27969.objecter _send_command 5 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc40125760 con 0x7fdc401081c0 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc44ad1700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6809/3689719376 conn(0x7fdc401081c0 0x7fdc40122b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc367fc700 10 client.27969.objecter ms_handle_connect 0x7fdc401081c0 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.836+0000 7fdc44ad1700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6809/3689719376 conn(0x7fdc401081c0 0x7fdc40122b10 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.837+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fdc40125760 con 0x7fdc401081c0 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.837+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.837+0000 7fdc367fc700 10 client.27969.objecter _finish_command 5 = system:0 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.837+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc40027a60 osd=2 4 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.837+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 2 5 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.851+0000 7fdc47536700 10 client.27969.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.851+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 6 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.851+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc40027a60 osd=2 3 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.851+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40027a60 osd=2 3 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.851+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc40027a60 osd=2 3 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.851+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 6 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.851+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc40027a60 osd=2 3 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.851+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 2 6 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.851+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40027a60 osd=2 4 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.851+0000 7fdc47536700 10 client.27969.objecter _send_command 6 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.851+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc40107720 con 0x7fdc401081c0 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fdc40107720 con 0x7fdc401081c0 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc367fc700 10 client.27969.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.284 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc40027a60 osd=2 4 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 2 6 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc47536700 10 client.27969.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 7 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6813/604971011 conn(0x7fdc40120660 0x7fdc40124180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc40107b30 osd=3 3 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40107b30 osd=3 3 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc40107b30 osd=3 3 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 7 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc40107b30 osd=3 3 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 3 7 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40107b30 osd=3 4 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc47536700 10 client.27969.objecter _send_command 7 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc401246f0 con 0x7fdc40120660 2024-02-13T00:52:11.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc45ad3700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6813/604971011 conn(0x7fdc40120660 0x7fdc40124180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:11.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc367fc700 10 client.27969.objecter ms_handle_connect 0x7fdc40120660 2024-02-13T00:52:11.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.853+0000 7fdc45ad3700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6813/604971011 conn(0x7fdc40120660 0x7fdc40124180 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:11.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.854+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fdc401246f0 con 0x7fdc40120660 2024-02-13T00:52:11.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.854+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:52:11.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.854+0000 7fdc367fc700 10 client.27969.objecter _finish_command 7 = system:0 2024-02-13T00:52:11.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.854+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc40107b30 osd=3 4 2024-02-13T00:52:11.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.854+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 3 7 2024-02-13T00:52:11.286 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:52:11.286 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:52:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.869+0000 7fdc47536700 10 client.27969.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.869+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 8 2024-02-13T00:52:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.869+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc40107b30 osd=3 3 2024-02-13T00:52:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.869+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40107b30 osd=3 3 2024-02-13T00:52:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.869+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc40107b30 osd=3 3 2024-02-13T00:52:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.869+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 8 2024-02-13T00:52:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.869+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc40107b30 osd=3 3 2024-02-13T00:52:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.869+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 3 8 2024-02-13T00:52:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.869+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40107b30 osd=3 4 2024-02-13T00:52:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.869+0000 7fdc47536700 10 client.27969.objecter _send_command 8 2024-02-13T00:52:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.869+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc4011fbc0 con 0x7fdc40120660 2024-02-13T00:52:11.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fdc4011fbc0 con 0x7fdc40120660 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc367fc700 10 client.27969.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc40107b30 osd=3 4 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 3 8 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc47536700 10 client.27969.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 9 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6800/3344263388 conn(0x7fdc401278e0 0x7fdc40129cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4011fd60 osd=4 3 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4011fd60 osd=4 3 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4011fd60 osd=4 3 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 9 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc4011fd60 osd=4 3 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 4 9 2024-02-13T00:52:11.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4011fd60 osd=4 4 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc47536700 10 client.27969.objecter _send_command 9 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.870+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc4011fed0 con 0x7fdc401278e0 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.871+0000 7fdc452d2700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6800/3344263388 conn(0x7fdc401278e0 0x7fdc40129cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.871+0000 7fdc367fc700 10 client.27969.objecter ms_handle_connect 0x7fdc401278e0 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.871+0000 7fdc452d2700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6800/3344263388 conn(0x7fdc401278e0 0x7fdc40129cd0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.872+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fdc4011fed0 con 0x7fdc401278e0 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.872+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.872+0000 7fdc367fc700 10 client.27969.objecter _finish_command 9 = system:0 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.872+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc4011fd60 osd=4 4 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.872+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 4 9 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.886+0000 7fdc47536700 10 client.27969.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.886+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 10 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.886+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4011fd60 osd=4 3 2024-02-13T00:52:11.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.886+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4011fd60 osd=4 3 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.886+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4011fd60 osd=4 3 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.886+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 10 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.886+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc4011fd60 osd=4 3 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.886+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 4 10 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.886+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4011fd60 osd=4 4 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.886+0000 7fdc47536700 10 client.27969.objecter _send_command 10 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.886+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc4011fed0 con 0x7fdc401278e0 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fdc4011fed0 con 0x7fdc401278e0 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc367fc700 10 client.27969.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc4011fd60 osd=4 4 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 4 10 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc47536700 10 client.27969.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 11 2024-02-13T00:52:11.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6804/2277917372 conn(0x7fdc4012a6d0 0x7fdc4012cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:11.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4011fed0 osd=5 3 2024-02-13T00:52:11.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4011fed0 osd=5 3 2024-02-13T00:52:11.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4011fed0 osd=5 3 2024-02-13T00:52:11.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 11 2024-02-13T00:52:11.291 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:52:11.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc4011fed0 osd=5 3 2024-02-13T00:52:11.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 5 11 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4011fed0 osd=5 4 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc47536700 10 client.27969.objecter _send_command 11 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.888+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc4012f820 con 0x7fdc4012a6d0 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.889+0000 7fdc44ad1700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6804/2277917372 conn(0x7fdc4012a6d0 0x7fdc4012cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.889+0000 7fdc367fc700 10 client.27969.objecter ms_handle_connect 0x7fdc4012a6d0 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.889+0000 7fdc44ad1700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6804/2277917372 conn(0x7fdc4012a6d0 0x7fdc4012cac0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.890+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fdc4012f820 con 0x7fdc4012a6d0 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.890+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.890+0000 7fdc367fc700 10 client.27969.objecter _finish_command 11 = system:0 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.890+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc4011fed0 osd=5 4 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.890+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 5 11 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.904+0000 7fdc47536700 10 client.27969.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:11.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.904+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 12 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.904+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4011fed0 osd=5 3 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.904+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4011fed0 osd=5 3 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.904+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4011fed0 osd=5 3 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.904+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 12 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.904+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc4011fed0 osd=5 3 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.904+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 5 12 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.904+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4011fed0 osd=5 4 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.904+0000 7fdc47536700 10 client.27969.objecter _send_command 12 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.904+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc4012f820 con 0x7fdc4012a6d0 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fdc4012f820 con 0x7fdc4012a6d0 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc367fc700 10 client.27969.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc4011fed0 osd=5 4 2024-02-13T00:52:11.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 5 12 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc47536700 10 client.27969.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 13 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6808/1506640341 conn(0x7fdc4012d560 0x7fdc4012d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4012f820 osd=6 3 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4012f820 osd=6 3 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4012f820 osd=6 3 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 13 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc4012f820 osd=6 3 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 6 13 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4012f820 osd=6 4 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc47536700 10 client.27969.objecter _send_command 13 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc4012f5a0 con 0x7fdc4012d560 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.906+0000 7fdc45ad3700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6808/1506640341 conn(0x7fdc4012d560 0x7fdc4012d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.907+0000 7fdc367fc700 10 client.27969.objecter ms_handle_connect 0x7fdc4012d560 2024-02-13T00:52:11.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.907+0000 7fdc45ad3700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6808/1506640341 conn(0x7fdc4012d560 0x7fdc4012d9e0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.907+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fdc4012f5a0 con 0x7fdc4012d560 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.907+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.907+0000 7fdc367fc700 10 client.27969.objecter _finish_command 13 = system:0 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.907+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc4012f820 osd=6 4 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.907+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 6 13 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.921+0000 7fdc47536700 10 client.27969.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.921+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 14 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.921+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4012f820 osd=6 3 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.921+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4012f820 osd=6 3 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.921+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4012f820 osd=6 3 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.921+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 14 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.921+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc4012f820 osd=6 3 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.921+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 6 14 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.921+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4012f820 osd=6 4 2024-02-13T00:52:11.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.921+0000 7fdc47536700 10 client.27969.objecter _send_command 14 2024-02-13T00:52:11.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.921+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc4012f5a0 con 0x7fdc4012d560 2024-02-13T00:52:11.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fdc4012f5a0 con 0x7fdc4012d560 2024-02-13T00:52:11.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:11.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc367fc700 10 client.27969.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc4012f820 osd=6 4 2024-02-13T00:52:11.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 6 14 2024-02-13T00:52:11.296 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:52:11.296 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:52:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc47536700 10 client.27969.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 15 2024-02-13T00:52:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6812/1964192772 conn(0x7fdc40131ff0 0x7fdc40132450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4012f5a0 osd=7 3 2024-02-13T00:52:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4012f5a0 osd=7 3 2024-02-13T00:52:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4012f5a0 osd=7 3 2024-02-13T00:52:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 15 2024-02-13T00:52:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc4012f5a0 osd=7 3 2024-02-13T00:52:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 7 15 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4012f5a0 osd=7 4 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc47536700 10 client.27969.objecter _send_command 15 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.923+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc40135d90 con 0x7fdc40131ff0 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.924+0000 7fdc452d2700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6812/1964192772 conn(0x7fdc40131ff0 0x7fdc40132450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.924+0000 7fdc367fc700 10 client.27969.objecter ms_handle_connect 0x7fdc40131ff0 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.924+0000 7fdc452d2700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6812/1964192772 conn(0x7fdc40131ff0 0x7fdc40132450 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.925+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fdc40135d90 con 0x7fdc40131ff0 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.925+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.925+0000 7fdc367fc700 10 client.27969.objecter _finish_command 15 = system:0 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.925+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc4012f5a0 osd=7 4 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.925+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 7 15 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.939+0000 7fdc47536700 10 client.27969.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.939+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign -1 16 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.939+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4012f5a0 osd=7 3 2024-02-13T00:52:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.939+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4012f5a0 osd=7 3 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.939+0000 7fdc47536700 20 client.27969.objecter _get_session s=0x7fdc4012f5a0 osd=7 3 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.939+0000 7fdc47536700 15 client.27969.objecter _session_command_op_remove -1 16 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.939+0000 7fdc47536700 20 client.27969.objecter get_session s=0x7fdc4012f5a0 osd=7 3 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.939+0000 7fdc47536700 15 client.27969.objecter _session_command_op_assign 7 16 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.939+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4012f5a0 osd=7 4 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.939+0000 7fdc47536700 10 client.27969.objecter _send_command 16 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.939+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdc4012df90 con 0x7fdc40131ff0 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.940+0000 7fdc367fc700 1 -- 172.21.15.162:0/2490891584 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fdc4012df90 con 0x7fdc40131ff0 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.940+0000 7fdc367fc700 10 client.27969.objecter ms_dispatch 0x7fdc40001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.940+0000 7fdc367fc700 10 client.27969.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.940+0000 7fdc367fc700 20 client.27969.objecter put_session s=0x7fdc4012f5a0 osd=7 4 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.940+0000 7fdc367fc700 15 client.27969.objecter _session_command_op_remove 7 16 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 10 librados: watch_flush enter 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 10 librados: watch_flush exit 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 10 client.27969.objecter close_session for osd.0 2024-02-13T00:52:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6801/2103743338 conn(0x7fdc40028590 msgr2=0x7fdc40106ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6801/2103743338 conn(0x7fdc40028590 0x7fdc40106ff0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc400475e0 osd=0 1 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 10 client.27969.objecter close_session for osd.1 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6805/150599780 conn(0x7fdc40027be0 msgr2=0x7fdc40028020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6805/150599780 conn(0x7fdc40027be0 0x7fdc40028020 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40108660 osd=1 1 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 10 client.27969.objecter close_session for osd.2 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6809/3689719376 conn(0x7fdc401081c0 msgr2=0x7fdc40122b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6809/3689719376 conn(0x7fdc401081c0 0x7fdc40122b10 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40027a60 osd=2 1 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 10 client.27969.objecter close_session for osd.3 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6813/604971011 conn(0x7fdc40120660 msgr2=0x7fdc40124180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6813/604971011 conn(0x7fdc40120660 0x7fdc40124180 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc40107b30 osd=3 1 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 10 client.27969.objecter close_session for osd.4 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6800/3344263388 conn(0x7fdc401278e0 msgr2=0x7fdc40129cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6800/3344263388 conn(0x7fdc401278e0 0x7fdc40129cd0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc45ad3700 1 -- 172.21.15.162:0/2490891584 reap_dead start 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4011fd60 osd=4 1 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 10 client.27969.objecter close_session for osd.5 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6804/2277917372 conn(0x7fdc4012a6d0 msgr2=0x7fdc4012cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6804/2277917372 conn(0x7fdc4012a6d0 0x7fdc4012cac0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc45ad3700 1 -- 172.21.15.162:0/2490891584 reap_dead start 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4011fed0 osd=5 1 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 10 client.27969.objecter close_session for osd.6 2024-02-13T00:52:11.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6808/1506640341 conn(0x7fdc4012d560 msgr2=0x7fdc4012d9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6808/1506640341 conn(0x7fdc4012d560 0x7fdc4012d9e0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4012f820 osd=6 1 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 10 client.27969.objecter close_session for osd.7 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6812/1964192772 conn(0x7fdc40131ff0 msgr2=0x7fdc40132450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.181:6812/1964192772 conn(0x7fdc40131ff0 0x7fdc40132450 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 20 client.27969.objecter put_session s=0x7fdc4012f5a0 osd=7 1 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 20 client.27969.objecter shutdown clearing up homeless session... 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 10 client.27969.objecter successfully canceled tick 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdc2c062c90 msgr2=0x7fdc2c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdc2c062c90 0x7fdc2c065150 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 rx=0x7fdc3c009da0 tx=0x7fdc3c009cb0).stop 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc400693c0 msgr2=0x7fdc40116b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 --2- 172.21.15.162:0/2490891584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc400693c0 0x7fdc40116b40 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 rx=0x7fdc30009980 tx=0x7fdc30009cb0).stop 2024-02-13T00:52:11.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc45ad3700 1 -- 172.21.15.162:0/2490891584 reap_dead start 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 shutdown_connections 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 >> 172.21.15.162:0/2490891584 conn(0x7fdc40063df0 msgr2=0x7fdc401058d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 shutdown_connections 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 -- 172.21.15.162:0/2490891584 wait complete. 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:10.941+0000 7fdc47536700 1 librados: shutdown 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.303 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:52:11.304 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:11.304 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:09.567111+0000 mon.a (mon.0) 3417 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:52:11.304 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:09.578326+0000 mon.a (mon.0) 3418 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-122"}]': finished 2024-02-13T00:52:11.304 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:09.578395+0000 mon.a (mon.0) 3419 : cluster [DBG] osdmap e702: 8 total, 8 up, 5 in 2024-02-13T00:52:11.304 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:10.541591+0000 mon.a (mon.0) 3420 : cluster [DBG] osdmap e703: 8 total, 8 up, 5 in 2024-02-13T00:52:11.888 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:11.891 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:10.517558+0000 mgr.y (mgr.14544) 586 : cluster [DBG] pgmap v1060: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 112 B/s, 3 objects/s recovering 2024-02-13T00:52:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:10.787182+0000 mon.a (mon.0) 3421 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:52:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:10.800720+0000 mon.a (mon.0) 3422 : audit [DBG] from='client.? 172.21.15.162:0/2490891584' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:11.251191+0000 osd.0 (osd.0) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:11.251370+0000 osd.0 (osd.0) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:11.546632+0000 mon.a (mon.0) 3423 : cluster [DBG] osdmap e704: 8 total, 8 up, 5 in 2024-02-13T00:52:12.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:11.551436+0000 mon.a (mon.0) 3424 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:12.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:10.517558+0000 mgr.y (mgr.14544) 586 : cluster [DBG] pgmap v1060: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 112 B/s, 3 objects/s recovering 2024-02-13T00:52:12.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:10.787182+0000 mon.a (mon.0) 3421 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:52:12.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:10.800720+0000 mon.a (mon.0) 3422 : audit [DBG] from='client.? 172.21.15.162:0/2490891584' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:12.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:11.251191+0000 osd.0 (osd.0) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:12.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:11.251370+0000 osd.0 (osd.0) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:12.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:11.546632+0000 mon.a (mon.0) 3423 : cluster [DBG] osdmap e704: 8 total, 8 up, 5 in 2024-02-13T00:52:12.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:11.551436+0000 mon.a (mon.0) 3424 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:10.517558+0000 mgr.y (mgr.14544) 586 : cluster [DBG] pgmap v1060: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 112 B/s, 3 objects/s recovering 2024-02-13T00:52:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:10.787182+0000 mon.a (mon.0) 3421 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:52:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:10.800720+0000 mon.a (mon.0) 3422 : audit [DBG] from='client.? 172.21.15.162:0/2490891584' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:11.251191+0000 osd.0 (osd.0) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:11.251370+0000 osd.0 (osd.0) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:11.546632+0000 mon.a (mon.0) 3423 : cluster [DBG] osdmap e704: 8 total, 8 up, 5 in 2024-02-13T00:52:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:11.551436+0000 mon.a (mon.0) 3424 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:13.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:12.219531+0000 osd.2 (osd.2) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:13.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:12.219847+0000 osd.2 (osd.2) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:13.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:12.518168+0000 mgr.y (mgr.14544) 587 : cluster [DBG] pgmap v1063: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 112 B/s, 3 objects/s recovering 2024-02-13T00:52:13.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:12.545195+0000 mon.a (mon.0) 3425 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:13.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:12.545277+0000 mon.a (mon.0) 3426 : cluster [DBG] osdmap e705: 8 total, 8 up, 5 in 2024-02-13T00:52:13.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:12.547683+0000 mon.a (mon.0) 3427 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:13.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:12.219531+0000 osd.2 (osd.2) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:13.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:12.219847+0000 osd.2 (osd.2) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:13.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:12.518168+0000 mgr.y (mgr.14544) 587 : cluster [DBG] pgmap v1063: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 112 B/s, 3 objects/s recovering 2024-02-13T00:52:13.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:12.545195+0000 mon.a (mon.0) 3425 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:13.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:12.545277+0000 mon.a (mon.0) 3426 : cluster [DBG] osdmap e705: 8 total, 8 up, 5 in 2024-02-13T00:52:13.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:12.547683+0000 mon.a (mon.0) 3427 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:13.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:52:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:12.219531+0000 osd.2 (osd.2) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:12.219847+0000 osd.2 (osd.2) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:12.518168+0000 mgr.y (mgr.14544) 587 : cluster [DBG] pgmap v1063: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 112 B/s, 3 objects/s recovering 2024-02-13T00:52:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:12.545195+0000 mon.a (mon.0) 3425 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:12.545277+0000 mon.a (mon.0) 3426 : cluster [DBG] osdmap e705: 8 total, 8 up, 5 in 2024-02-13T00:52:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:12.547683+0000 mon.a (mon.0) 3427 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:14.545 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:14.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:14.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:14.648 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:14.678 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:14.776 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:14.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:13.547814+0000 mon.a (mon.0) 3428 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:14.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:13.547905+0000 mon.a (mon.0) 3429 : cluster [DBG] osdmap e706: 8 total, 8 up, 5 in 2024-02-13T00:52:14.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:13.549521+0000 mon.a (mon.0) 3430 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-124"}]: dispatch 2024-02-13T00:52:14.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:14.549584+0000 mon.a (mon.0) 3431 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-124"}]': finished 2024-02-13T00:52:14.778 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:14.549664+0000 mon.a (mon.0) 3432 : cluster [DBG] osdmap e707: 8 total, 8 up, 5 in 2024-02-13T00:52:14.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:13.547814+0000 mon.a (mon.0) 3428 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:14.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:13.547905+0000 mon.a (mon.0) 3429 : cluster [DBG] osdmap e706: 8 total, 8 up, 5 in 2024-02-13T00:52:14.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:13.549521+0000 mon.a (mon.0) 3430 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-124"}]: dispatch 2024-02-13T00:52:14.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:14.549584+0000 mon.a (mon.0) 3431 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-124"}]': finished 2024-02-13T00:52:14.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:14.549664+0000 mon.a (mon.0) 3432 : cluster [DBG] osdmap e707: 8 total, 8 up, 5 in 2024-02-13T00:52:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:14.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:14.783 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:52:14.783 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:52:14.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:14.910 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:13.547814+0000 mon.a (mon.0) 3428 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:14.910 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:13.547905+0000 mon.a (mon.0) 3429 : cluster [DBG] osdmap e706: 8 total, 8 up, 5 in 2024-02-13T00:52:14.910 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:13.549521+0000 mon.a (mon.0) 3430 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-124"}]: dispatch 2024-02-13T00:52:14.911 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:14.549584+0000 mon.a (mon.0) 3431 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-124"}]': finished 2024-02-13T00:52:14.911 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:14.549664+0000 mon.a (mon.0) 3432 : cluster [DBG] osdmap e707: 8 total, 8 up, 5 in 2024-02-13T00:52:14.912 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:14.934 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:14.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:14.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:15.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:15.067 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:15.087 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:15.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:15.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:15.220 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:15.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-13T00:52:15.571 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:15.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:15.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:15.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:15.701 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:15.828 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:15.828 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:14.518548+0000 mgr.y (mgr.14544) 588 : cluster [DBG] pgmap v1066: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:52:15.828 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:14.551137+0000 mon.a (mon.0) 3433 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-124", "mode": "writeback"}]: dispatch 2024-02-13T00:52:15.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:15.539125+0000 mon.a (mon.0) 3434 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:52:15.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:15.543814+0000 mon.a (mon.0) 3435 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-124", "mode": "writeback"}]': finished 2024-02-13T00:52:15.829 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:15.543887+0000 mon.a (mon.0) 3436 : cluster [DBG] osdmap e708: 8 total, 8 up, 5 in 2024-02-13T00:52:15.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:14.518548+0000 mgr.y (mgr.14544) 588 : cluster [DBG] pgmap v1066: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:52:15.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:14.551137+0000 mon.a (mon.0) 3433 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-124", "mode": "writeback"}]: dispatch 2024-02-13T00:52:15.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:15.539125+0000 mon.a (mon.0) 3434 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:52:15.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:15.543814+0000 mon.a (mon.0) 3435 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-124", "mode": "writeback"}]': finished 2024-02-13T00:52:15.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:15.543887+0000 mon.a (mon.0) 3436 : cluster [DBG] osdmap e708: 8 total, 8 up, 5 in 2024-02-13T00:52:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:15.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:15.888 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:14.518548+0000 mgr.y (mgr.14544) 588 : cluster [DBG] pgmap v1066: 204 pgs: 32 unknown, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:52:15.888 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:14.551137+0000 mon.a (mon.0) 3433 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-124", "mode": "writeback"}]: dispatch 2024-02-13T00:52:15.888 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:15.539125+0000 mon.a (mon.0) 3434 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:52:15.888 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:15.543814+0000 mon.a (mon.0) 3435 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-124", "mode": "writeback"}]': finished 2024-02-13T00:52:15.889 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:15.543887+0000 mon.a (mon.0) 3436 : cluster [DBG] osdmap e708: 8 total, 8 up, 5 in 2024-02-13T00:52:15.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:15.959 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:16.100 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.099+0000 7fac1a6f8700 1 -- 172.21.15.162:0/2638234448 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fac140fcb30 msgr2=0x7fac140fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.099+0000 7fac1a6f8700 1 --2- 172.21.15.162:0/2638234448 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fac140fcb30 0x7fac140fcf10 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 rx=0x7fac040047b0 tx=0x7fac04009cb0).stop 2024-02-13T00:52:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.099+0000 7fac1a6f8700 1 -- 172.21.15.162:0/2638234448 shutdown_connections 2024-02-13T00:52:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.099+0000 7fac1a6f8700 1 --2- 172.21.15.162:0/2638234448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac140f73a0 0x7fac141039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.099+0000 7fac1a6f8700 1 --2- 172.21.15.162:0/2638234448 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fac140fcb30 0x7fac140fcf10 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.099+0000 7fac1a6f8700 1 --2- 172.21.15.162:0/2638234448 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac140fd490 0x7fac140f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.099+0000 7fac1a6f8700 1 -- 172.21.15.162:0/2638234448 >> 172.21.15.162:0/2638234448 conn(0x7fac1406eb90 msgr2=0x7fac140f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.099+0000 7fac1a6f8700 1 -- 172.21.15.162:0/2638234448 shutdown_connections 2024-02-13T00:52:16.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.099+0000 7fac1a6f8700 1 -- 172.21.15.162:0/2638234448 wait complete. 2024-02-13T00:52:16.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.100+0000 7fac1a6f8700 1 librados: starting msgr at 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.100+0000 7fac1a6f8700 1 librados: starting objecter 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.100+0000 7fac1a6f8700 1 Processor -- start 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.100+0000 7fac1a6f8700 1 -- start start 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.100+0000 7fac1a6f8700 1 librados: setting wanted keys 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.100+0000 7fac1a6f8700 1 librados: calling monclient init 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac1a6f8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac140f73a0 0x7fac141092a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac1a6f8700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fac140fcb30 0x7fac14109790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac1a6f8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac140fd490 0x7fac14109d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac1a6f8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac14114480 con 0x7fac140f73a0 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac1a6f8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac141145f0 con 0x7fac140fd490 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac1a6f8700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac141147b0 con 0x7fac140fcb30 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac137fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fac140fcb30 0x7fac14109790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac137fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fac140fcb30 0x7fac14109790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:42518/0 (socket says 172.21.15.162:42518) 2024-02-13T00:52:16.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac137fe700 1 -- 172.21.15.162:0/1708145204 learned_addr learned my addr 172.21.15.162:0/1708145204 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:16.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac13fff700 1 --2- 172.21.15.162:0/1708145204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac140f73a0 0x7fac141092a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:16.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac18c95700 1 --2- 172.21.15.162:0/1708145204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac140fd490 0x7fac14109d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:16.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac137fe700 1 -- 172.21.15.162:0/1708145204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac140fd490 msgr2=0x7fac14109d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:16.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac137fe700 1 --2- 172.21.15.162:0/1708145204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac140fd490 0x7fac14109d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac137fe700 1 -- 172.21.15.162:0/1708145204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac140f73a0 msgr2=0x7fac141092a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac137fe700 1 --2- 172.21.15.162:0/1708145204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac140f73a0 0x7fac141092a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac137fe700 1 -- 172.21.15.162:0/1708145204 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c0097c0 con 0x7fac140fcb30 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac18c95700 1 --2- 172.21.15.162:0/1708145204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac140fd490 0x7fac14109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac13fff700 1 --2- 172.21.15.162:0/1708145204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac140f73a0 0x7fac141092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac137fe700 1 --2- 172.21.15.162:0/1708145204 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fac140fcb30 0x7fac14109790 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 rx=0x7fac0800e990 tx=0x7fac0800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac117fa700 10 client.?.objecter ms_handle_connect 0x7fac140fcb30 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.101+0000 7fac117fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.102+0000 7fac117fa700 1 -- 172.21.15.162:0/1708145204 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0800ee80 con 0x7fac140fcb30 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.102+0000 7fac117fa700 1 -- 172.21.15.162:0/1708145204 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac0800f040 con 0x7fac140fcb30 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.102+0000 7fac1a6f8700 1 -- 172.21.15.162:0/1708145204 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac04005040 con 0x7fac140fcb30 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.102+0000 7fac1a6f8700 10 client.38072.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.102+0000 7fac117fa700 1 -- 172.21.15.162:0/1708145204 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0800c840 con 0x7fac140fcb30 2024-02-13T00:52:16.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.102+0000 7fac1a6f8700 1 -- 172.21.15.162:0/1708145204 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac14107420 con 0x7fac140fcb30 2024-02-13T00:52:16.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.102+0000 7fac1a6f8700 1 librados: init done 2024-02-13T00:52:16.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.103+0000 7fac117fa700 1 -- 172.21.15.162:0/1708145204 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fac0800cc70 con 0x7fac140fcb30 2024-02-13T00:52:16.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.103+0000 7fac1a6f8700 1 -- 172.21.15.162:0/1708145204 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac00004f50 con 0x7fac140fcb30 2024-02-13T00:52:16.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:16.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.104+0000 7fac117fa700 1 --2- 172.21.15.162:0/1708145204 >> v2:172.21.15.162:6800/2866534053 conn(0x7fabfc062c90 0x7fabfc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:16.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.104+0000 7fac117fa700 1 -- 172.21.15.162:0/1708145204 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(708..708 src has 449..708) v4 ==== 8734+0+0 (secure 0 0 0) 0x7fac0807e060 con 0x7fac140fcb30 2024-02-13T00:52:16.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.104+0000 7fac117fa700 10 client.38072.objecter ms_dispatch 0x7fac14001550 osd_map(708..708 src has 449..708) v4 2024-02-13T00:52:16.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.104+0000 7fac117fa700 3 client.38072.objecter handle_osd_map got epochs [708,708] > 0 2024-02-13T00:52:16.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.104+0000 7fac117fa700 3 client.38072.objecter handle_osd_map decoding full epoch 708 2024-02-13T00:52:16.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.104+0000 7fac117fa700 20 client.38072.objecter dump_active .. 0 homeless 2024-02-13T00:52:16.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:16.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:16.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.112+0000 7fac13fff700 1 --2- 172.21.15.162:0/1708145204 >> v2:172.21.15.162:6800/2866534053 conn(0x7fabfc062c90 0x7fabfc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:16.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.112+0000 7fac117fa700 1 -- 172.21.15.162:0/1708145204 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fac08051290 con 0x7fac140fcb30 2024-02-13T00:52:16.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.112+0000 7fac117fa700 10 client.38072.objecter ms_handle_connect 0x7fabfc062c90 2024-02-13T00:52:16.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.112+0000 7fac13fff700 1 --2- 172.21.15.162:0/1708145204 >> v2:172.21.15.162:6800/2866534053 conn(0x7fabfc062c90 0x7fabfc065150 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 rx=0x7fac04004e20 tx=0x7fac04007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:16.144 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:16.154 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 6, 7] out_osds: [3, 2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-13T00:52:16.154 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:52:16.155 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 4, 6, 7] 2024-02-13T00:52:16.155 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd out 4 2024-02-13T00:52:16.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:16.240 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:16.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:52:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:52:16.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.311+0000 7fac1a6f8700 1 -- 172.21.15.162:0/1708145204 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fac00005200 con 0x7fac140fcb30 2024-02-13T00:52:16.564 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:52:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:52:16] "GET /metrics HTTP/1.1" 200 189585 "" "Prometheus/2.33.4" 2024-02-13T00:52:16.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.571+0000 7fac117fa700 1 -- 172.21.15.162:0/1708145204 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v709) v1 ==== 91+0+0 (secure 0 0 0) 0x7fac08050e80 con 0x7fac140fcb30 2024-02-13T00:52:16.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 10 librados: watch_flush enter 2024-02-13T00:52:16.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 10 librados: watch_flush exit 2024-02-13T00:52:16.577 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 20 client.38072.objecter shutdown clearing up homeless session... 2024-02-13T00:52:16.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 10 client.38072.objecter successfully canceled tick 2024-02-13T00:52:16.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 -- 172.21.15.162:0/1708145204 >> v2:172.21.15.162:6800/2866534053 conn(0x7fabfc062c90 msgr2=0x7fabfc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:16.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 --2- 172.21.15.162:0/1708145204 >> v2:172.21.15.162:6800/2866534053 conn(0x7fabfc062c90 0x7fabfc065150 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 rx=0x7fac04004e20 tx=0x7fac04007f90).stop 2024-02-13T00:52:16.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 -- 172.21.15.162:0/1708145204 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fac140fcb30 msgr2=0x7fac14109790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:16.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 --2- 172.21.15.162:0/1708145204 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fac140fcb30 0x7fac14109790 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 rx=0x7fac0800e990 tx=0x7fac0800b040).stop 2024-02-13T00:52:16.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 -- 172.21.15.162:0/1708145204 shutdown_connections 2024-02-13T00:52:16.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 --2- 172.21.15.162:0/1708145204 >> v2:172.21.15.162:6800/2866534053 conn(0x7fabfc062c90 0x7fabfc065150 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:16.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 --2- 172.21.15.162:0/1708145204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac140fd490 0x7fac14109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:16.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 --2- 172.21.15.162:0/1708145204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac140f73a0 0x7fac141092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:16.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 --2- 172.21.15.162:0/1708145204 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fac140fcb30 0x7fac14109790 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:16.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 -- 172.21.15.162:0/1708145204 >> 172.21.15.162:0/1708145204 conn(0x7fac1406eb90 msgr2=0x7fac140700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:16.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 -- 172.21.15.162:0/1708145204 shutdown_connections 2024-02-13T00:52:16.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 -- 172.21.15.162:0/1708145204 wait complete. 2024-02-13T00:52:16.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:16.573+0000 7fac1a6f8700 1 librados: shutdown 2024-02-13T00:52:16.579 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:52:16.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:15.593291+0000 mon.a (mon.0) 3437 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:16.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:15.885097+0000 mgr.y (mgr.14544) 589 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:16.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:16.312913+0000 mon.c (mon.2) 493 : audit [INF] from='client.? 172.21.15.162:0/1708145204' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:52:16.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:16.313210+0000 mon.a (mon.0) 3438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:52:16.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:15.593291+0000 mon.a (mon.0) 3437 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:16.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:15.885097+0000 mgr.y (mgr.14544) 589 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:16.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:16.312913+0000 mon.c (mon.2) 493 : audit [INF] from='client.? 172.21.15.162:0/1708145204' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:52:16.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:16.313210+0000 mon.a (mon.0) 3438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:52:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:15.593291+0000 mon.a (mon.0) 3437 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:15.885097+0000 mgr.y (mgr.14544) 589 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:16.312913+0000 mon.c (mon.2) 493 : audit [INF] from='client.? 172.21.15.162:0/1708145204' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:52:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:16.313210+0000 mon.a (mon.0) 3438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:52:17.089 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:17.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:17.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:17.225 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:17.246 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:17.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:17.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:17.380 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:17.400 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:17.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:17.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:17.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:17.534 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:17.825 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:16.073950+0000 osd.1 (osd.1) 301 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:17.825 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:16.074155+0000 osd.1 (osd.1) 302 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:17.825 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:16.519166+0000 mgr.y (mgr.14544) 590 : cluster [DBG] pgmap v1069: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:17.825 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:16.561210+0000 mon.a (mon.0) 3439 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:52:17.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:16.571974+0000 mon.a (mon.0) 3440 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:52:17.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:16.572038+0000 mon.a (mon.0) 3441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:52:17.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:16.572085+0000 mon.a (mon.0) 3442 : cluster [DBG] osdmap e709: 8 total, 8 up, 5 in 2024-02-13T00:52:17.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:16.572878+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-124"}]: dispatch 2024-02-13T00:52:17.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:16.782245+0000 osd.4 (osd.4) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:17.826 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:16.782537+0000 osd.4 (osd.4) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:17.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:16.073950+0000 osd.1 (osd.1) 301 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:17.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:16.074155+0000 osd.1 (osd.1) 302 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:17.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:16.519166+0000 mgr.y (mgr.14544) 590 : cluster [DBG] pgmap v1069: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:17.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:16.561210+0000 mon.a (mon.0) 3439 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:52:17.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:16.571974+0000 mon.a (mon.0) 3440 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:52:17.826 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:16.572038+0000 mon.a (mon.0) 3441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:52:17.827 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:16.572085+0000 mon.a (mon.0) 3442 : cluster [DBG] osdmap e709: 8 total, 8 up, 5 in 2024-02-13T00:52:17.827 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:16.572878+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-124"}]: dispatch 2024-02-13T00:52:17.827 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:16.782245+0000 osd.4 (osd.4) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:17.827 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:16.782537+0000 osd.4 (osd.4) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:16.073950+0000 osd.1 (osd.1) 301 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:16.074155+0000 osd.1 (osd.1) 302 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:16.519166+0000 mgr.y (mgr.14544) 590 : cluster [DBG] pgmap v1069: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:16.561210+0000 mon.a (mon.0) 3439 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:52:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:16.571974+0000 mon.a (mon.0) 3440 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:52:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:16.572038+0000 mon.a (mon.0) 3441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:52:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:16.572085+0000 mon.a (mon.0) 3442 : cluster [DBG] osdmap e709: 8 total, 8 up, 5 in 2024-02-13T00:52:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:16.572878+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-124"}]: dispatch 2024-02-13T00:52:18.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:16.782245+0000 osd.4 (osd.4) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:18.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:16.782537+0000 osd.4 (osd.4) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:18.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:16.973846+0000 osd.0 (osd.0) 271 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:52:18.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:16.976044+0000 osd.0 (osd.0) 272 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:52:18.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:17.568971+0000 mon.a (mon.0) 3444 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:52:18.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:17.572422+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-124"}]': finished 2024-02-13T00:52:18.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:17.572487+0000 mon.a (mon.0) 3446 : cluster [DBG] osdmap e710: 8 total, 8 up, 5 in 2024-02-13T00:52:18.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:16.973846+0000 osd.0 (osd.0) 271 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:52:18.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:16.976044+0000 osd.0 (osd.0) 272 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:52:18.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:17.568971+0000 mon.a (mon.0) 3444 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:52:18.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:17.572422+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-124"}]': finished 2024-02-13T00:52:18.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:17.572487+0000 mon.a (mon.0) 3446 : cluster [DBG] osdmap e710: 8 total, 8 up, 5 in 2024-02-13T00:52:18.913 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:18.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:18.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:19.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:19.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:52:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:52:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:52:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:16.973846+0000 osd.0 (osd.0) 271 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:52:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:16.976044+0000 osd.0 (osd.0) 272 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:52:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:17.568971+0000 mon.a (mon.0) 3444 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:52:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:17.572422+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-124"}]': finished 2024-02-13T00:52:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:17.572487+0000 mon.a (mon.0) 3446 : cluster [DBG] osdmap e710: 8 total, 8 up, 5 in 2024-02-13T00:52:19.046 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:19.193 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:19.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:19.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:19.239 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:52:19.239 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:52:19.297 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:19.326 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:19.774 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:19.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:19.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:19.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:19.908 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:18.519594+0000 mgr.y (mgr.14544) 591 : cluster [DBG] pgmap v1072: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:19.908 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:18.587163+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e711: 8 total, 8 up, 5 in 2024-02-13T00:52:19.908 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:19.343478+0000 mon.a (mon.0) 3448 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:19.910 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:19.931 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:19.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:20.035 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:20.064 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:20.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:18.519594+0000 mgr.y (mgr.14544) 591 : cluster [DBG] pgmap v1072: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:20.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:18.587163+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e711: 8 total, 8 up, 5 in 2024-02-13T00:52:20.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:19.343478+0000 mon.a (mon.0) 3448 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:18.519594+0000 mgr.y (mgr.14544) 591 : cluster [DBG] pgmap v1072: 204 pgs: 204 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:18.587163+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e711: 8 total, 8 up, 5 in 2024-02-13T00:52:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:19.343478+0000 mon.a (mon.0) 3448 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:20.085 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:20.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:20.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:20.189 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:20.217 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:20.236 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:20.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:20.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:20.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:20.367 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:20.407 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:20.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:20.509 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:20.537 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:20.559 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:20.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:20.637 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:19.590280+0000 mon.a (mon.0) 3449 : cluster [DBG] osdmap e712: 8 total, 8 up, 5 in 2024-02-13T00:52:20.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:20.689 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:20.708 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:20.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:19.590280+0000 mon.a (mon.0) 3449 : cluster [DBG] osdmap e712: 8 total, 8 up, 5 in 2024-02-13T00:52:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:19.591768+0000 mon.a (mon.0) 3450 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:20.540272+0000 mon.a (mon.0) 3451 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:52:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:20.548371+0000 mon.a (mon.0) 3452 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:20.548467+0000 mon.a (mon.0) 3453 : cluster [DBG] osdmap e713: 8 total, 8 up, 5 in 2024-02-13T00:52:20.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:20.550758+0000 mon.a (mon.0) 3454 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:20.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:20.843 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:20.933 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:19.591768+0000 mon.a (mon.0) 3450 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:20.933 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:20.540272+0000 mon.a (mon.0) 3451 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:52:20.933 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:20.548371+0000 mon.a (mon.0) 3452 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:20.933 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:20.548467+0000 mon.a (mon.0) 3453 : cluster [DBG] osdmap e713: 8 total, 8 up, 5 in 2024-02-13T00:52:20.933 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:20.550758+0000 mon.a (mon.0) 3454 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:20.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:19.590280+0000 mon.a (mon.0) 3449 : cluster [DBG] osdmap e712: 8 total, 8 up, 5 in 2024-02-13T00:52:20.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:19.591768+0000 mon.a (mon.0) 3450 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:20.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:20.540272+0000 mon.a (mon.0) 3451 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:52:20.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:20.548371+0000 mon.a (mon.0) 3452 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:20.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:20.548467+0000 mon.a (mon.0) 3453 : cluster [DBG] osdmap e713: 8 total, 8 up, 5 in 2024-02-13T00:52:20.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:20.550758+0000 mon.a (mon.0) 3454 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:21.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.922+0000 7f08428f7700 1 -- 172.21.15.162:0/3901098780 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f083c0f6b50 msgr2=0x7f083c103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.922+0000 7f08428f7700 1 --2- 172.21.15.162:0/3901098780 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f083c0f6b50 0x7f083c103150 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 rx=0x7f0834009be0 tx=0x7f0834004d10).stop 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.922+0000 7f08428f7700 1 -- 172.21.15.162:0/3901098780 shutdown_connections 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.922+0000 7f08428f7700 1 --2- 172.21.15.162:0/3901098780 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f083c0f6b50 0x7f083c103150 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.922+0000 7f08428f7700 1 --2- 172.21.15.162:0/3901098780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f083c0fc2e0 0x7f083c0fc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.922+0000 7f08428f7700 1 --2- 172.21.15.162:0/3901098780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f083c0fcc40 0x7f083c0f6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.922+0000 7f08428f7700 1 -- 172.21.15.162:0/3901098780 >> 172.21.15.162:0/3901098780 conn(0x7f083c070060 msgr2=0x7f083c0f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.922+0000 7f08428f7700 1 -- 172.21.15.162:0/3901098780 shutdown_connections 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.922+0000 7f08428f7700 1 -- 172.21.15.162:0/3901098780 wait complete. 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.922+0000 7f08428f7700 1 librados: starting msgr at 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.922+0000 7f08428f7700 1 librados: starting objecter 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f08428f7700 1 Processor -- start 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f08428f7700 1 -- start start 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f08428f7700 1 librados: setting wanted keys 2024-02-13T00:52:21.050 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f08428f7700 1 librados: calling monclient init 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f08428f7700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f083c0f6b50 0x7f083c10bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f08428f7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f083c0fc2e0 0x7f083c10c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f08428f7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f083c0fcc40 0x7f083c105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f08428f7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c1063a0 con 0x7f083c0fcc40 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f08428f7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c106510 con 0x7f083c0fc2e0 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f08428f7700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083c1066d0 con 0x7f083c0f6b50 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f083bfff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f083c0f6b50 0x7f083c10bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f083bfff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f083c0f6b50 0x7f083c10bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:42528/0 (socket says 172.21.15.162:42528) 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f083bfff700 1 -- 172.21.15.162:0/1093685901 learned_addr learned my addr 172.21.15.162:0/1093685901 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.923+0000 7f0840e94700 1 --2- 172.21.15.162:0/1093685901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f083c0fcc40 0x7f083c105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f083b7fe700 1 --2- 172.21.15.162:0/1093685901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f083c0fc2e0 0x7f083c10c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f083bfff700 1 -- 172.21.15.162:0/1093685901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f083c0fc2e0 msgr2=0x7f083c10c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.051 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f083bfff700 1 --2- 172.21.15.162:0/1093685901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f083c0fc2e0 0x7f083c10c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f083bfff700 1 -- 172.21.15.162:0/1093685901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f083c0fcc40 msgr2=0x7f083c105eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f083bfff700 1 --2- 172.21.15.162:0/1093685901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f083c0fcc40 0x7f083c105eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f083bfff700 1 -- 172.21.15.162:0/1093685901 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c005040 con 0x7f083c0f6b50 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f083b7fe700 1 --2- 172.21.15.162:0/1093685901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f083c0fc2e0 0x7f083c10c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f0840e94700 1 --2- 172.21.15.162:0/1093685901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f083c0fcc40 0x7f083c105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f083bfff700 1 --2- 172.21.15.162:0/1093685901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f083c0f6b50 0x7f083c10bd10 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 rx=0x7f082c00a780 tx=0x7f082c00adc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f08397fa700 10 client.?.objecter ms_handle_connect 0x7f083c0f6b50 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f08397fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c008c10 con 0x7f083c0f6b50 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f082c010040 con 0x7f083c0f6b50 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08340097c0 con 0x7f083c0f6b50 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c015a50 con 0x7f083c0f6b50 2024-02-13T00:52:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f08428f7700 10 client.38078.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:21.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083c112b40 con 0x7f083c0f6b50 2024-02-13T00:52:21.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.924+0000 7f08428f7700 1 librados: init done 2024-02-13T00:52:21.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.925+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f083c05f960 con 0x7f083c0f6b50 2024-02-13T00:52:21.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.925+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f082c00f840 con 0x7f083c0f6b50 2024-02-13T00:52:21.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08397fa700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0824062c90 0x7f0824065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:21.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(713..713 src has 449..713) v4 ==== 8711+0+0 (secure 0 0 0) 0x7f082c07e210 con 0x7f083c0f6b50 2024-02-13T00:52:21.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 osd_map(713..713 src has 449..713) v4 2024-02-13T00:52:21.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08397fa700 3 client.38078.objecter handle_osd_map got epochs [713,713] > 0 2024-02-13T00:52:21.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08397fa700 3 client.38078.objecter handle_osd_map decoding full epoch 713 2024-02-13T00:52:21.053 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:52:21.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f083b7fe700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0824062c90 0x7f0824065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:21.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08397fa700 20 client.38078.objecter dump_active .. 0 homeless 2024-02-13T00:52:21.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v713) v1 ==== 54+0+16 (secure 0 0 0) 0x7f082c085070 con 0x7f083c0f6b50 2024-02-13T00:52:21.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08428f7700 10 client.38078.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:21.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 1 2024-02-13T00:52:21.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6801/2103743338 conn(0x7f083c0fd5b0 0x7f083c0fd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:21.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08397fa700 10 client.38078.objecter ms_handle_connect 0x7f0824062c90 2024-02-13T00:52:21.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f083b7fe700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0824062c90 0x7f0824065150 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 rx=0x7f083c107650 tx=0x7f083000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:21.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c0475e0 osd=0 3 2024-02-13T00:52:21.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c0475e0 osd=0 3 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c0475e0 osd=0 3 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 1 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c0475e0 osd=0 3 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 0 1 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c0475e0 osd=0 4 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08428f7700 10 client.38078.objecter _send_command 1 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f083c104fb0 con 0x7f083c0fd5b0 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.926+0000 7f0840e94700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6801/2103743338 conn(0x7f083c0fd5b0 0x7f083c0fd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.927+0000 7f08397fa700 10 client.38078.objecter ms_handle_connect 0x7f083c0fd5b0 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.927+0000 7f0840e94700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6801/2103743338 conn(0x7f083c0fd5b0 0x7f083c0fd990 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.927+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f083c104fb0 con 0x7f083c0fd5b0 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.927+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:52:21.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.927+0000 7f08397fa700 10 client.38078.objecter _finish_command 1 = system:0 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.927+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c0475e0 osd=0 4 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.927+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 0 1 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.944+0000 7f08428f7700 10 client.38078.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.944+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 2 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.944+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c0475e0 osd=0 3 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.944+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c0475e0 osd=0 3 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.944+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c0475e0 osd=0 3 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.944+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 2 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.944+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c0475e0 osd=0 3 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.944+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 0 2 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.944+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c0475e0 osd=0 4 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.944+0000 7f08428f7700 10 client.38078.objecter _send_command 2 2024-02-13T00:52:21.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.944+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f083c027410 con 0x7f083c0fd5b0 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f083c027410 con 0x7f083c0fd5b0 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08397fa700 10 client.38078.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c0475e0 osd=0 4 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 0 2 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08428f7700 10 client.38078.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 3 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6805/150599780 conn(0x7f083c059990 0x7f083c059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c028760 osd=1 3 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c028760 osd=1 3 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c028760 osd=1 3 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 3 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c028760 osd=1 3 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 1 3 2024-02-13T00:52:21.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c028760 osd=1 4 2024-02-13T00:52:21.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08428f7700 10 client.38078.objecter _send_command 3 2024-02-13T00:52:21.058 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:52:21.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f083c117e70 con 0x7f083c059990 2024-02-13T00:52:21.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.946+0000 7f083bfff700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6805/150599780 conn(0x7f083c059990 0x7f083c059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:21.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.947+0000 7f083bfff700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6805/150599780 conn(0x7f083c059990 0x7f083c059d70 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:21.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.951+0000 7f08397fa700 10 client.38078.objecter ms_handle_connect 0x7f083c059990 2024-02-13T00:52:21.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.951+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f083c117e70 con 0x7f083c059990 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.951+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.951+0000 7f08397fa700 10 client.38078.objecter _finish_command 3 = system:0 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.951+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c028760 osd=1 4 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.951+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 1 3 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.966+0000 7f08428f7700 10 client.38078.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.966+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 4 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.966+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c028760 osd=1 3 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.966+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c028760 osd=1 3 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.966+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c028760 osd=1 3 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.966+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 4 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.966+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c028760 osd=1 3 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.966+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 1 4 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.966+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c028760 osd=1 4 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.966+0000 7f08428f7700 10 client.38078.objecter _send_command 4 2024-02-13T00:52:21.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.966+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f083c059400 con 0x7f083c059990 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f083c059400 con 0x7f083c059990 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08397fa700 10 client.38078.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c028760 osd=1 4 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 1 4 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08428f7700 10 client.38078.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 5 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6809/3689719376 conn(0x7f083c0282c0 0x7f083c11e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c059810 osd=2 3 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c059810 osd=2 3 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c059810 osd=2 3 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 5 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c059810 osd=2 3 2024-02-13T00:52:21.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 2 5 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c059810 osd=2 4 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08428f7700 10 client.38078.objecter _send_command 5 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.967+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f083c1210e0 con 0x7f083c0282c0 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.968+0000 7f083b7fe700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6809/3689719376 conn(0x7f083c0282c0 0x7f083c11e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.968+0000 7f08397fa700 10 client.38078.objecter ms_handle_connect 0x7f083c0282c0 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.968+0000 7f083b7fe700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6809/3689719376 conn(0x7f083c0282c0 0x7f083c11e490 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.968+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f083c1210e0 con 0x7f083c0282c0 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.968+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.968+0000 7f08397fa700 10 client.38078.objecter _finish_command 5 = system:0 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.968+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c059810 osd=2 4 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.968+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 2 5 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.983+0000 7f08428f7700 10 client.38078.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.983+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 6 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.983+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c059810 osd=2 3 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.983+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c059810 osd=2 3 2024-02-13T00:52:21.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.983+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c059810 osd=2 3 2024-02-13T00:52:21.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.983+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 6 2024-02-13T00:52:21.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.983+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c059810 osd=2 3 2024-02-13T00:52:21.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.983+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 2 6 2024-02-13T00:52:21.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.983+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c059810 osd=2 4 2024-02-13T00:52:21.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.983+0000 7f08428f7700 10 client.38078.objecter _send_command 6 2024-02-13T00:52:21.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.983+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f083c027820 con 0x7f083c0282c0 2024-02-13T00:52:21.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f083c027820 con 0x7f083c0282c0 2024-02-13T00:52:21.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:21.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08397fa700 10 client.38078.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.062 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:52:21.062 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:52:21.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c059810 osd=2 4 2024-02-13T00:52:21.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 2 6 2024-02-13T00:52:21.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08428f7700 10 client.38078.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:21.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 7 2024-02-13T00:52:21.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6813/604971011 conn(0x7f083c1179d0 0x7f083c11fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:21.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c027c30 osd=3 3 2024-02-13T00:52:21.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c027c30 osd=3 3 2024-02-13T00:52:21.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c027c30 osd=3 3 2024-02-13T00:52:21.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 7 2024-02-13T00:52:21.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c027c30 osd=3 3 2024-02-13T00:52:21.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 3 7 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c027c30 osd=3 4 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08428f7700 10 client.38078.objecter _send_command 7 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.984+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f083c11fff0 con 0x7f083c1179d0 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.985+0000 7f0840e94700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6813/604971011 conn(0x7f083c1179d0 0x7f083c11fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.985+0000 7f08397fa700 10 client.38078.objecter ms_handle_connect 0x7f083c1179d0 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.985+0000 7f0840e94700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6813/604971011 conn(0x7f083c1179d0 0x7f083c11fb00 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.985+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f083c11fff0 con 0x7f083c1179d0 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.985+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.985+0000 7f08397fa700 10 client.38078.objecter _finish_command 7 = system:0 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.985+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c027c30 osd=3 4 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:20.985+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 3 7 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.000+0000 7f08428f7700 10 client.38078.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.000+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 8 2024-02-13T00:52:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.000+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c027c30 osd=3 3 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.000+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c027c30 osd=3 3 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.000+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c027c30 osd=3 3 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.000+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 8 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.000+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c027c30 osd=3 3 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.000+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 3 8 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.000+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c027c30 osd=3 4 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.000+0000 7f08428f7700 10 client.38078.objecter _send_command 8 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.000+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f083c116f30 con 0x7f083c1179d0 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f083c116f30 con 0x7f083c1179d0 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08397fa700 10 client.38078.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c027c30 osd=3 4 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 3 8 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08428f7700 10 client.38078.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 9 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6800/3344263388 conn(0x7f083c1170a0 0x7f083c125270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c116f30 osd=4 3 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c116f30 osd=4 3 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c116f30 osd=4 3 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 9 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c116f30 osd=4 3 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 4 9 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c116f30 osd=4 4 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08428f7700 10 client.38078.objecter _send_command 9 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.001+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f083c127ec0 con 0x7f083c1170a0 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.002+0000 7f083bfff700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6800/3344263388 conn(0x7f083c1170a0 0x7f083c125270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.002+0000 7f08397fa700 10 client.38078.objecter ms_handle_connect 0x7f083c1170a0 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.002+0000 7f083bfff700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6800/3344263388 conn(0x7f083c1170a0 0x7f083c125270 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.003+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f083c127ec0 con 0x7f083c1170a0 2024-02-13T00:52:21.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.003+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.003+0000 7f08397fa700 10 client.38078.objecter _finish_command 9 = system:0 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.003+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c116f30 osd=4 4 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.003+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 4 9 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.018+0000 7f08428f7700 10 client.38078.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.018+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 10 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.018+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c116f30 osd=4 3 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.018+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c116f30 osd=4 3 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.018+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c116f30 osd=4 3 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.018+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 10 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.018+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c116f30 osd=4 3 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.018+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 4 10 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.018+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c116f30 osd=4 4 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.018+0000 7f08428f7700 10 client.38078.objecter _send_command 10 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.018+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f083c125760 con 0x7f083c1170a0 2024-02-13T00:52:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f083c125760 con 0x7f083c1170a0 2024-02-13T00:52:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08397fa700 10 client.38078.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c116f30 osd=4 4 2024-02-13T00:52:21.068 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:52:21.068 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 4 10 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08428f7700 10 client.38078.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 11 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6804/2277917372 conn(0x7f083c125e70 0x7f083c128260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c125760 osd=5 3 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c125760 osd=5 3 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c125760 osd=5 3 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 11 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c125760 osd=5 3 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 5 11 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c125760 osd=5 4 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08428f7700 10 client.38078.objecter _send_command 11 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.019+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f083c12afc0 con 0x7f083c125e70 2024-02-13T00:52:21.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.020+0000 7f083b7fe700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6804/2277917372 conn(0x7f083c125e70 0x7f083c128260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.020+0000 7f08397fa700 10 client.38078.objecter ms_handle_connect 0x7f083c125e70 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.020+0000 7f083b7fe700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6804/2277917372 conn(0x7f083c125e70 0x7f083c128260 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.021+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f083c12afc0 con 0x7f083c125e70 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.021+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.021+0000 7f08397fa700 10 client.38078.objecter _finish_command 11 = system:0 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.021+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c125760 osd=5 4 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.021+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 5 11 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.035+0000 7f08428f7700 10 client.38078.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.035+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 12 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.035+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c125760 osd=5 3 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.035+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c125760 osd=5 3 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.035+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c125760 osd=5 3 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.035+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 12 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.035+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c125760 osd=5 3 2024-02-13T00:52:21.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.035+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 5 12 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.035+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c125760 osd=5 4 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.035+0000 7f08428f7700 10 client.38078.objecter _send_command 12 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.035+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f083c12afc0 con 0x7f083c125e70 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f083c12afc0 con 0x7f083c125e70 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08397fa700 10 client.38078.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c125760 osd=5 4 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 5 12 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08428f7700 10 client.38078.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 13 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6808/1506640341 conn(0x7f083c128d00 0x7f083c129160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c12afc0 osd=6 3 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c12afc0 osd=6 3 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c12afc0 osd=6 3 2024-02-13T00:52:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 13 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c12afc0 osd=6 3 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 6 13 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c12afc0 osd=6 4 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08428f7700 10 client.38078.objecter _send_command 13 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.037+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f083c12adb0 con 0x7f083c128d00 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.038+0000 7f0840e94700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6808/1506640341 conn(0x7f083c128d00 0x7f083c129160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.038+0000 7f0840e94700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6808/1506640341 conn(0x7f083c128d00 0x7f083c129160 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.038+0000 7f08397fa700 10 client.38078.objecter ms_handle_connect 0x7f083c128d00 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.039+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f083c12adb0 con 0x7f083c128d00 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.039+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.039+0000 7f08397fa700 10 client.38078.objecter _finish_command 13 = system:0 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.039+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c12afc0 osd=6 4 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.039+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 6 13 2024-02-13T00:52:21.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.054+0000 7f08428f7700 10 client.38078.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.054+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 14 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.054+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c12afc0 osd=6 3 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.054+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c12afc0 osd=6 3 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.054+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c12afc0 osd=6 3 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.054+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 14 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.054+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c12afc0 osd=6 3 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.054+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 6 14 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.054+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c12afc0 osd=6 4 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.054+0000 7f08428f7700 10 client.38078.objecter _send_command 14 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.054+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f083c12adb0 con 0x7f083c128d00 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.055+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f083c12adb0 con 0x7f083c128d00 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.055+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.055+0000 7f08397fa700 10 client.38078.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.055+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c12afc0 osd=6 4 2024-02-13T00:52:21.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.055+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 6 14 2024-02-13T00:52:21.074 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:52:21.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08428f7700 10 client.38078.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:21.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 15 2024-02-13T00:52:21.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6812/1964192772 conn(0x7f083c12d7c0 0x7f083c12dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:21.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c12adb0 osd=7 3 2024-02-13T00:52:21.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c12adb0 osd=7 3 2024-02-13T00:52:21.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c12adb0 osd=7 3 2024-02-13T00:52:21.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 15 2024-02-13T00:52:21.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c12adb0 osd=7 3 2024-02-13T00:52:21.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 7 15 2024-02-13T00:52:21.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c12adb0 osd=7 4 2024-02-13T00:52:21.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08428f7700 10 client.38078.objecter _send_command 15 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f083c12e270 con 0x7f083c12d7c0 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f083bfff700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6812/1964192772 conn(0x7f083c12d7c0 0x7f083c12dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f08397fa700 10 client.38078.objecter ms_handle_connect 0x7f083c12d7c0 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.056+0000 7f083bfff700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6812/1964192772 conn(0x7f083c12d7c0 0x7f083c12dc00 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.057+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f082c04fb00 con 0x7f083c12d7c0 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.057+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.057+0000 7f08397fa700 10 client.38078.objecter _finish_command 15 = system:0 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.057+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c12adb0 osd=7 4 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.057+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 7 15 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.071+0000 7f08428f7700 10 client.38078.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.071+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign -1 16 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.071+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c12adb0 osd=7 3 2024-02-13T00:52:21.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.071+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c12adb0 osd=7 3 2024-02-13T00:52:21.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.071+0000 7f08428f7700 20 client.38078.objecter _get_session s=0x7f083c12adb0 osd=7 3 2024-02-13T00:52:21.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.071+0000 7f08428f7700 15 client.38078.objecter _session_command_op_remove -1 16 2024-02-13T00:52:21.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.071+0000 7f08428f7700 20 client.38078.objecter get_session s=0x7f083c12adb0 osd=7 3 2024-02-13T00:52:21.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.071+0000 7f08428f7700 15 client.38078.objecter _session_command_op_assign 7 16 2024-02-13T00:52:21.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.071+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c12adb0 osd=7 4 2024-02-13T00:52:21.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.071+0000 7f08428f7700 10 client.38078.objecter _send_command 16 2024-02-13T00:52:21.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.071+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f083c130480 con 0x7f083c12d7c0 2024-02-13T00:52:21.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08397fa700 1 -- 172.21.15.162:0/1093685901 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f083c130480 con 0x7f083c12d7c0 2024-02-13T00:52:21.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08397fa700 10 client.38078.objecter ms_dispatch 0x7f083c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:21.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08397fa700 10 client.38078.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08397fa700 20 client.38078.objecter put_session s=0x7f083c12adb0 osd=7 4 2024-02-13T00:52:21.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08397fa700 15 client.38078.objecter _session_command_op_remove 7 16 2024-02-13T00:52:21.078 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:52:21.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 10 librados: watch_flush enter 2024-02-13T00:52:21.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 10 librados: watch_flush exit 2024-02-13T00:52:21.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 10 client.38078.objecter close_session for osd.0 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6801/2103743338 conn(0x7f083c0fd5b0 msgr2=0x7f083c0fd990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6801/2103743338 conn(0x7f083c0fd5b0 0x7f083c0fd990 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c0475e0 osd=0 1 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 10 client.38078.objecter close_session for osd.1 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6805/150599780 conn(0x7f083c059990 msgr2=0x7f083c059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6805/150599780 conn(0x7f083c059990 0x7f083c059d70 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c028760 osd=1 1 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 10 client.38078.objecter close_session for osd.2 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6809/3689719376 conn(0x7f083c0282c0 msgr2=0x7f083c11e490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6809/3689719376 conn(0x7f083c0282c0 0x7f083c11e490 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c059810 osd=2 1 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 10 client.38078.objecter close_session for osd.3 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.073+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6813/604971011 conn(0x7f083c1179d0 msgr2=0x7f083c11fb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6813/604971011 conn(0x7f083c1179d0 0x7f083c11fb00 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c027c30 osd=3 1 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 10 client.38078.objecter close_session for osd.4 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6800/3344263388 conn(0x7f083c1170a0 msgr2=0x7f083c125270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6800/3344263388 conn(0x7f083c1170a0 0x7f083c125270 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c116f30 osd=4 1 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 10 client.38078.objecter close_session for osd.5 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6804/2277917372 conn(0x7f083c125e70 msgr2=0x7f083c128260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6804/2277917372 conn(0x7f083c125e70 0x7f083c128260 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c125760 osd=5 1 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 10 client.38078.objecter close_session for osd.6 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6808/1506640341 conn(0x7f083c128d00 msgr2=0x7f083c129160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6808/1506640341 conn(0x7f083c128d00 0x7f083c129160 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c12afc0 osd=6 1 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 10 client.38078.objecter close_session for osd.7 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6812/1964192772 conn(0x7f083c12d7c0 msgr2=0x7f083c12dc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.181:6812/1964192772 conn(0x7f083c12d7c0 0x7f083c12dc00 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 20 client.38078.objecter put_session s=0x7f083c12adb0 osd=7 1 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 20 client.38078.objecter shutdown clearing up homeless session... 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 10 client.38078.objecter successfully canceled tick 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0824062c90 msgr2=0x7f0824065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0824062c90 0x7f0824065150 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 rx=0x7f083c107650 tx=0x7f083000b040).stop 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f0840e94700 1 -- 172.21.15.162:0/1093685901 reap_dead start 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f083c0f6b50 msgr2=0x7f083c10bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 --2- 172.21.15.162:0/1093685901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f083c0f6b50 0x7f083c10bd10 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 rx=0x7f082c00a780 tx=0x7f082c00adc0).stop 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f0840e94700 1 -- 172.21.15.162:0/1093685901 reap_dead start 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f0840e94700 1 -- 172.21.15.162:0/1093685901 reap_dead start 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f0840e94700 1 -- 172.21.15.162:0/1093685901 reap_dead start 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f0840e94700 1 -- 172.21.15.162:0/1093685901 reap_dead start 2024-02-13T00:52:21.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f0840e94700 1 -- 172.21.15.162:0/1093685901 reap_dead start 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 shutdown_connections 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 >> 172.21.15.162:0/1093685901 conn(0x7f083c070060 msgr2=0x7f083c06d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 shutdown_connections 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 -- 172.21.15.162:0/1093685901 wait complete. 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:21.074+0000 7f08428f7700 1 librados: shutdown 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:52:21.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-13T00:52:21.789 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:52:22.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:20.090376+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:52:22.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:20.093276+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:52:22.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:20.520404+0000 mgr.y (mgr.14544) 592 : cluster [DBG] pgmap v1075: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:52:22.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:20.926855+0000 mon.c (mon.2) 494 : audit [DBG] from='client.? 172.21.15.162:0/1093685901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:22.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:21.547406+0000 mon.a (mon.0) 3455 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:22.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:21.547476+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e714: 8 total, 8 up, 5 in 2024-02-13T00:52:22.048 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:21.550170+0000 mon.a (mon.0) 3457 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-126"}]: dispatch 2024-02-13T00:52:22.048 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:20.090376+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:52:22.049 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:20.093276+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:52:22.049 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:20.520404+0000 mgr.y (mgr.14544) 592 : cluster [DBG] pgmap v1075: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:52:22.049 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:20.926855+0000 mon.c (mon.2) 494 : audit [DBG] from='client.? 172.21.15.162:0/1093685901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:22.049 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:21.547406+0000 mon.a (mon.0) 3455 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:22.049 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:21.547476+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e714: 8 total, 8 up, 5 in 2024-02-13T00:52:22.049 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:21.550170+0000 mon.a (mon.0) 3457 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-126"}]: dispatch 2024-02-13T00:52:22.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:20.090376+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:52:22.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:20.093276+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:52:22.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:20.520404+0000 mgr.y (mgr.14544) 592 : cluster [DBG] pgmap v1075: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:52:22.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:20.926855+0000 mon.c (mon.2) 494 : audit [DBG] from='client.? 172.21.15.162:0/1093685901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:22.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:21.547406+0000 mon.a (mon.0) 3455 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:22.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:21.547476+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e714: 8 total, 8 up, 5 in 2024-02-13T00:52:22.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:21.550170+0000 mon.a (mon.0) 3457 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-126"}]: dispatch 2024-02-13T00:52:22.092 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:22.096 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:22.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:22.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:22.137 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:22.146 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:22.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:22.230 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:22.250 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:22.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:22.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:22.386 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:22.405 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:22.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:22.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:22.537 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:22.613 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:22.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:22.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:22.647 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:21.308913+0000 osd.1 (osd.1) 303 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:52:22.647 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:21.312566+0000 osd.1 (osd.1) 304 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:52:22.647 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:22.548563+0000 mon.a (mon.0) 3458 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-126"}]': finished 2024-02-13T00:52:22.647 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:22.548633+0000 mon.a (mon.0) 3459 : cluster [DBG] osdmap e715: 8 total, 8 up, 5 in 2024-02-13T00:52:22.647 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:22.549850+0000 mon.a (mon.0) 3460 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-126", "mode": "writeback"}]: dispatch 2024-02-13T00:52:22.648 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:21.308913+0000 osd.1 (osd.1) 303 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:52:22.648 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:21.312566+0000 osd.1 (osd.1) 304 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:52:22.648 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:22.548563+0000 mon.a (mon.0) 3458 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-126"}]': finished 2024-02-13T00:52:22.648 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:22.548633+0000 mon.a (mon.0) 3459 : cluster [DBG] osdmap e715: 8 total, 8 up, 5 in 2024-02-13T00:52:22.648 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:22.549850+0000 mon.a (mon.0) 3460 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-126", "mode": "writeback"}]: dispatch 2024-02-13T00:52:22.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:22.746 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:21.308913+0000 osd.1 (osd.1) 303 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:52:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:21.312566+0000 osd.1 (osd.1) 304 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:52:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:22.548563+0000 mon.a (mon.0) 3458 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-126"}]': finished 2024-02-13T00:52:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:22.548633+0000 mon.a (mon.0) 3459 : cluster [DBG] osdmap e715: 8 total, 8 up, 5 in 2024-02-13T00:52:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:22.549850+0000 mon.a (mon.0) 3460 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-126", "mode": "writeback"}]: dispatch 2024-02-13T00:52:23.394 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:23.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:23.498 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:23.527 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:23.992 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:22.520793+0000 mgr.y (mgr.14544) 593 : cluster [DBG] pgmap v1078: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:52:23.992 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:22.985279+0000 osd.0 (osd.0) 273 : cluster [DBG] 5.d scrub starts 2024-02-13T00:52:23.992 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:22.988899+0000 osd.0 (osd.0) 274 : cluster [DBG] 5.d scrub ok 2024-02-13T00:52:23.992 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:23.120658+0000 osd.6 (osd.6) 220 : cluster [DBG] 28.1d scrub starts 2024-02-13T00:52:23.992 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:23.124079+0000 osd.6 (osd.6) 221 : cluster [DBG] 28.1d scrub ok 2024-02-13T00:52:23.993 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:23.546167+0000 mon.a (mon.0) 3461 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:52:23.993 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:23.550479+0000 mon.a (mon.0) 3462 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-126", "mode": "writeback"}]': finished 2024-02-13T00:52:23.993 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:23.550538+0000 mon.a (mon.0) 3463 : cluster [DBG] osdmap e716: 8 total, 8 up, 5 in 2024-02-13T00:52:23.993 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:23.609147+0000 mon.a (mon.0) 3464 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:23.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:22.520793+0000 mgr.y (mgr.14544) 593 : cluster [DBG] pgmap v1078: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:52:23.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:22.985279+0000 osd.0 (osd.0) 273 : cluster [DBG] 5.d scrub starts 2024-02-13T00:52:23.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:22.988899+0000 osd.0 (osd.0) 274 : cluster [DBG] 5.d scrub ok 2024-02-13T00:52:23.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:23.120658+0000 osd.6 (osd.6) 220 : cluster [DBG] 28.1d scrub starts 2024-02-13T00:52:23.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:23.124079+0000 osd.6 (osd.6) 221 : cluster [DBG] 28.1d scrub ok 2024-02-13T00:52:23.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:23.546167+0000 mon.a (mon.0) 3461 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:52:23.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:23.550479+0000 mon.a (mon.0) 3462 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-126", "mode": "writeback"}]': finished 2024-02-13T00:52:23.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:23.550538+0000 mon.a (mon.0) 3463 : cluster [DBG] osdmap e716: 8 total, 8 up, 5 in 2024-02-13T00:52:23.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:23.609147+0000 mon.a (mon.0) 3464 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:22.520793+0000 mgr.y (mgr.14544) 593 : cluster [DBG] pgmap v1078: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-13T00:52:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:22.985279+0000 osd.0 (osd.0) 273 : cluster [DBG] 5.d scrub starts 2024-02-13T00:52:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:22.988899+0000 osd.0 (osd.0) 274 : cluster [DBG] 5.d scrub ok 2024-02-13T00:52:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:23.120658+0000 osd.6 (osd.6) 220 : cluster [DBG] 28.1d scrub starts 2024-02-13T00:52:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:23.124079+0000 osd.6 (osd.6) 221 : cluster [DBG] 28.1d scrub ok 2024-02-13T00:52:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:23.546167+0000 mon.a (mon.0) 3461 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:52:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:23.550479+0000 mon.a (mon.0) 3462 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-126", "mode": "writeback"}]': finished 2024-02-13T00:52:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:23.550538+0000 mon.a (mon.0) 3463 : cluster [DBG] osdmap e716: 8 total, 8 up, 5 in 2024-02-13T00:52:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:23.609147+0000 mon.a (mon.0) 3464 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:24.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:52:24.276 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:24.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:24.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:24.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:24.410 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:24.431 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:24.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:24.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:24.563 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:24.589 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:24.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:24.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:24.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:24.724 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:24.753 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:24.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:23.947745+0000 osd.0 (osd.0) 275 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:52:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:23.951202+0000 osd.0 (osd.0) 276 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:52:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:24.170518+0000 osd.6 (osd.6) 222 : cluster [DBG] 28.4 scrub starts 2024-02-13T00:52:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:24.173647+0000 osd.6 (osd.6) 223 : cluster [DBG] 28.4 scrub ok 2024-02-13T00:52:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:24.326668+0000 mon.c (mon.2) 495 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:52:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:24.552315+0000 mon.a (mon.0) 3465 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:52:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:24.552371+0000 mon.a (mon.0) 3466 : cluster [DBG] osdmap e717: 8 total, 8 up, 5 in 2024-02-13T00:52:24.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:24.552986+0000 mon.a (mon.0) 3467 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-126"}]: dispatch 2024-02-13T00:52:24.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:24.980 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:23.947745+0000 osd.0 (osd.0) 275 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:52:24.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:23.951202+0000 osd.0 (osd.0) 276 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:52:24.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:24.170518+0000 osd.6 (osd.6) 222 : cluster [DBG] 28.4 scrub starts 2024-02-13T00:52:24.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:24.173647+0000 osd.6 (osd.6) 223 : cluster [DBG] 28.4 scrub ok 2024-02-13T00:52:24.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:24.326668+0000 mon.c (mon.2) 495 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:52:24.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:24.552315+0000 mon.a (mon.0) 3465 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:52:24.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:24.552371+0000 mon.a (mon.0) 3466 : cluster [DBG] osdmap e717: 8 total, 8 up, 5 in 2024-02-13T00:52:24.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:24.552986+0000 mon.a (mon.0) 3467 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-126"}]: dispatch 2024-02-13T00:52:24.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:23.947745+0000 osd.0 (osd.0) 275 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:52:24.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:23.951202+0000 osd.0 (osd.0) 276 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:52:24.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:24.170518+0000 osd.6 (osd.6) 222 : cluster [DBG] 28.4 scrub starts 2024-02-13T00:52:24.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:24.173647+0000 osd.6 (osd.6) 223 : cluster [DBG] 28.4 scrub ok 2024-02-13T00:52:24.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:24.326668+0000 mon.c (mon.2) 495 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:52:24.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:24.552315+0000 mon.a (mon.0) 3465 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:52:24.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:24.552371+0000 mon.a (mon.0) 3466 : cluster [DBG] osdmap e717: 8 total, 8 up, 5 in 2024-02-13T00:52:24.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:24.552986+0000 mon.a (mon.0) 3467 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-126"}]: dispatch 2024-02-13T00:52:25.737 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:25.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:25.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:25.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:25.872 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:25.894 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:25.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:25.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:25.923 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:25.957 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:52:25.957 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:52:25.957 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:52:25.957 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:52:25.957 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:52:25.988 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:24.521173+0000 mgr.y (mgr.14544) 594 : cluster [DBG] pgmap v1081: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:52:25.988 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:24.803926+0000 mon.c (mon.2) 496 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:52:25.988 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:24.804618+0000 mon.c (mon.2) 497 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:24.808971+0000 mon.a (mon.0) 3468 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:24.912071+0000 osd.0 (osd.0) 277 : cluster [DBG] 5.13 scrub starts 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:24.916452+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.13 scrub ok 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:25.146473+0000 osd.6 (osd.6) 224 : cluster [DBG] 2.b scrub starts 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:25.149938+0000 osd.6 (osd.6) 225 : cluster [DBG] 2.b scrub ok 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:25.548927+0000 mon.a (mon.0) 3469 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:25.556919+0000 mon.a (mon.0) 3470 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-126"}]': finished 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:25.556959+0000 mon.a (mon.0) 3471 : cluster [DBG] osdmap e718: 8 total, 8 up, 5 in 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:24.521173+0000 mgr.y (mgr.14544) 594 : cluster [DBG] pgmap v1081: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:24.803926+0000 mon.c (mon.2) 496 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:24.804618+0000 mon.c (mon.2) 497 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:24.808971+0000 mon.a (mon.0) 3468 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:25.989 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:24.912071+0000 osd.0 (osd.0) 277 : cluster [DBG] 5.13 scrub starts 2024-02-13T00:52:25.990 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:24.916452+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.13 scrub ok 2024-02-13T00:52:25.990 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:25.146473+0000 osd.6 (osd.6) 224 : cluster [DBG] 2.b scrub starts 2024-02-13T00:52:25.990 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:25.149938+0000 osd.6 (osd.6) 225 : cluster [DBG] 2.b scrub ok 2024-02-13T00:52:25.990 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:25.548927+0000 mon.a (mon.0) 3469 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:52:25.990 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:25.556919+0000 mon.a (mon.0) 3470 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-126"}]': finished 2024-02-13T00:52:25.990 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:25.556959+0000 mon.a (mon.0) 3471 : cluster [DBG] osdmap e718: 8 total, 8 up, 5 in 2024-02-13T00:52:25.997 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:26.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:52:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:52:26.024 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:24.521173+0000 mgr.y (mgr.14544) 594 : cluster [DBG] pgmap v1081: 204 pgs: 32 creating+peering, 172 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail 2024-02-13T00:52:26.024 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:24.803926+0000 mon.c (mon.2) 496 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:52:26.024 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:24.804618+0000 mon.c (mon.2) 497 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:52:26.024 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:24.808971+0000 mon.a (mon.0) 3468 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:26.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:24.912071+0000 osd.0 (osd.0) 277 : cluster [DBG] 5.13 scrub starts 2024-02-13T00:52:26.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:24.916452+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.13 scrub ok 2024-02-13T00:52:26.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:25.146473+0000 osd.6 (osd.6) 224 : cluster [DBG] 2.b scrub starts 2024-02-13T00:52:26.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:25.149938+0000 osd.6 (osd.6) 225 : cluster [DBG] 2.b scrub ok 2024-02-13T00:52:26.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:25.548927+0000 mon.a (mon.0) 3469 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:52:26.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:25.556919+0000 mon.a (mon.0) 3470 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-126"}]': finished 2024-02-13T00:52:26.025 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:25.556959+0000 mon.a (mon.0) 3471 : cluster [DBG] osdmap e718: 8 total, 8 up, 5 in 2024-02-13T00:52:26.026 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:26.047 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:26.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:26.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:26.179 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:26.198 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:26.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:26.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:26.271 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:52:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:52:26] "GET /metrics HTTP/1.1" 200 187162 "" "Prometheus/2.33.4" 2024-02-13T00:52:26.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:26.333 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:26.430 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:26.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:26.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:26.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:26.563 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:26.693 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:26.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:26.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:26.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:26.824 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:25.895277+0000 mgr.y (mgr.14544) 595 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:26.110208+0000 osd.6 (osd.6) 226 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:52:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:26.113115+0000 osd.6 (osd.6) 227 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:52:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:26.557371+0000 mon.a (mon.0) 3472 : cluster [DBG] osdmap e719: 8 total, 8 up, 5 in 2024-02-13T00:52:27.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:25.895277+0000 mgr.y (mgr.14544) 595 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:27.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:26.110208+0000 osd.6 (osd.6) 226 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:52:27.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:26.113115+0000 osd.6 (osd.6) 227 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:52:27.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:26.557371+0000 mon.a (mon.0) 3472 : cluster [DBG] osdmap e719: 8 total, 8 up, 5 in 2024-02-13T00:52:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:25.895277+0000 mgr.y (mgr.14544) 595 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:26.110208+0000 osd.6 (osd.6) 226 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:52:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:26.113115+0000 osd.6 (osd.6) 227 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:52:27.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:26.557371+0000 mon.a (mon.0) 3472 : cluster [DBG] osdmap e719: 8 total, 8 up, 5 in 2024-02-13T00:52:27.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.668+0000 7fc451539700 1 -- 172.21.15.162:0/2057906929 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc44c0f9b70 msgr2=0x7fc44c0f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:27.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.668+0000 7fc451539700 1 --2- 172.21.15.162:0/2057906929 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc44c0f9b70 0x7fc44c0f9f50 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 rx=0x7fc43c004870 tx=0x7fc43c009cb0).stop 2024-02-13T00:52:27.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.668+0000 7fc451539700 1 -- 172.21.15.162:0/2057906929 shutdown_connections 2024-02-13T00:52:27.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.668+0000 7fc451539700 1 --2- 172.21.15.162:0/2057906929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc44c0f8500 0x7fc44c104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:27.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.668+0000 7fc451539700 1 --2- 172.21.15.162:0/2057906929 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc44c0f9b70 0x7fc44c0f9f50 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:27.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.668+0000 7fc451539700 1 --2- 172.21.15.162:0/2057906929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc44c0fa440 0x7fc44c0f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:27.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.668+0000 7fc451539700 1 -- 172.21.15.162:0/2057906929 >> 172.21.15.162:0/2057906929 conn(0x7fc44c0f38c0 msgr2=0x7fc44c0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.668+0000 7fc451539700 1 -- 172.21.15.162:0/2057906929 shutdown_connections 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.668+0000 7fc451539700 1 -- 172.21.15.162:0/2057906929 wait complete. 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc451539700 1 librados: starting msgr at 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc451539700 1 librados: starting objecter 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc451539700 1 Processor -- start 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc451539700 1 -- start start 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc451539700 1 librados: setting wanted keys 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc451539700 1 librados: calling monclient init 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc451539700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc44c0f8500 0x7fc44c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc451539700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc44c0f9b70 0x7fc44c06b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc451539700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc44c0fa440 0x7fc44c06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc451539700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c116720 con 0x7fc44c0fa440 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc451539700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c116890 con 0x7fc44c0f8500 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc451539700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c116a50 con 0x7fc44c0f9b70 2024-02-13T00:52:27.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc44b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc44c0fa440 0x7fc44c06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc44a7fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc44c0f9b70 0x7fc44c06b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc44b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc44c0fa440 0x7fc44c06bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45104/0 (socket says 172.21.15.162:45104) 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.669+0000 7fc44b7fe700 1 -- 172.21.15.162:0/1193919010 learned_addr learned my addr 172.21.15.162:0/1193919010 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc44affd700 1 --2- 172.21.15.162:0/1193919010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc44c0f8500 0x7fc44c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc44a7fc700 1 -- 172.21.15.162:0/1193919010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc44c0f8500 msgr2=0x7fc44c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc44a7fc700 1 --2- 172.21.15.162:0/1193919010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc44c0f8500 0x7fc44c06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc44a7fc700 1 -- 172.21.15.162:0/1193919010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc44c0fa440 msgr2=0x7fc44c06bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc44a7fc700 1 --2- 172.21.15.162:0/1193919010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc44c0fa440 0x7fc44c06bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc44a7fc700 1 -- 172.21.15.162:0/1193919010 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c005040 con 0x7fc44c0f9b70 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc44a7fc700 1 --2- 172.21.15.162:0/1193919010 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc44c0f9b70 0x7fc44c06b660 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 rx=0x7fc44000e8f0 tx=0x7fc44000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc433fff700 10 client.?.objecter ms_handle_connect 0x7fc44c0f9b70 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc433fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc433fff700 1 -- 172.21.15.162:0/1193919010 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc440004620 con 0x7fc44c0f9b70 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc433fff700 1 -- 172.21.15.162:0/1193919010 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc44000f040 con 0x7fc44c0f9b70 2024-02-13T00:52:27.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc451539700 1 -- 172.21.15.162:0/1193919010 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c116d30 con 0x7fc44c0f9b70 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc433fff700 1 -- 172.21.15.162:0/1193919010 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc440014830 con 0x7fc44c0f9b70 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc451539700 10 client.38084.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc451539700 1 -- 172.21.15.162:0/1193919010 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44c117290 con 0x7fc44c0f9b70 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.670+0000 7fc451539700 1 librados: init done 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.671+0000 7fc451539700 1 -- 172.21.15.162:0/1193919010 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44c0475e0 con 0x7fc44c0f9b70 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.671+0000 7fc433fff700 1 -- 172.21.15.162:0/1193919010 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc4400090d0 con 0x7fc44c0f9b70 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.672+0000 7fc433fff700 1 --2- 172.21.15.162:0/1193919010 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc434062c90 0x7fc434065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.672+0000 7fc433fff700 1 -- 172.21.15.162:0/1193919010 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(719..719 src has 449..719) v4 ==== 8325+0+0 (secure 0 0 0) 0x7fc44007cbd0 con 0x7fc44c0f9b70 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.672+0000 7fc433fff700 10 client.38084.objecter ms_dispatch 0x7fc44c001550 osd_map(719..719 src has 449..719) v4 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.672+0000 7fc433fff700 3 client.38084.objecter handle_osd_map got epochs [719,719] > 0 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.672+0000 7fc433fff700 3 client.38084.objecter handle_osd_map decoding full epoch 719 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.672+0000 7fc44affd700 1 --2- 172.21.15.162:0/1193919010 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc434062c90 0x7fc434065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.672+0000 7fc433fff700 20 client.38084.objecter dump_active .. 0 homeless 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.672+0000 7fc433fff700 10 client.38084.objecter ms_handle_connect 0x7fc434062c90 2024-02-13T00:52:27.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.672+0000 7fc44affd700 1 --2- 172.21.15.162:0/1193919010 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc434062c90 0x7fc434065150 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 rx=0x7fc43c0043d0 tx=0x7fc43c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:27.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.676+0000 7fc433fff700 1 -- 172.21.15.162:0/1193919010 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc440050130 con 0x7fc44c0f9b70 2024-02-13T00:52:27.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:26.869+0000 7fc451539700 1 -- 172.21.15.162:0/1193919010 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc44c06c650 con 0x7fc44c0f9b70 2024-02-13T00:52:27.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.563+0000 7fc433fff700 1 -- 172.21.15.162:0/1193919010 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v720) v1 ==== 101+0+0 (secure 0 0 0) 0x7fc440019070 con 0x7fc44c0f9b70 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 10 librados: watch_flush enter 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 10 librados: watch_flush exit 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 20 client.38084.objecter shutdown clearing up homeless session... 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 10 client.38084.objecter successfully canceled tick 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 -- 172.21.15.162:0/1193919010 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc434062c90 msgr2=0x7fc434065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 --2- 172.21.15.162:0/1193919010 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc434062c90 0x7fc434065150 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 rx=0x7fc43c0043d0 tx=0x7fc43c00b040).stop 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 -- 172.21.15.162:0/1193919010 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc44c0f9b70 msgr2=0x7fc44c06b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 --2- 172.21.15.162:0/1193919010 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc44c0f9b70 0x7fc44c06b660 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 rx=0x7fc44000e8f0 tx=0x7fc44000b040).stop 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 -- 172.21.15.162:0/1193919010 shutdown_connections 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 --2- 172.21.15.162:0/1193919010 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc434062c90 0x7fc434065150 unknown :-1 s=CLOSED pgs=3011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 --2- 172.21.15.162:0/1193919010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc44c0fa440 0x7fc44c06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 --2- 172.21.15.162:0/1193919010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc44c0f8500 0x7fc44c06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:27.571 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 --2- 172.21.15.162:0/1193919010 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc44c0f9b70 0x7fc44c06b660 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 -- 172.21.15.162:0/1193919010 >> 172.21.15.162:0/1193919010 conn(0x7fc44c0f38c0 msgr2=0x7fc44c0fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 -- 172.21.15.162:0/1193919010 shutdown_connections 2024-02-13T00:52:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 -- 172.21.15.162:0/1193919010 wait complete. 2024-02-13T00:52:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:27.566+0000 7fc451539700 1 librados: shutdown 2024-02-13T00:52:27.572 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:52:27.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-13T00:52:28.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:26.521535+0000 mgr.y (mgr.14544) 596 : cluster [DBG] pgmap v1084: 204 pgs: 204 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-02-13T00:52:28.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:26.870228+0000 mon.c (mon.2) 498 : audit [INF] from='client.? 172.21.15.162:0/1193919010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:28.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:26.870554+0000 mon.a (mon.0) 3473 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:28.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:26.946800+0000 osd.0 (osd.0) 279 : cluster [DBG] 4.3 scrub starts 2024-02-13T00:52:28.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:26.954377+0000 osd.0 (osd.0) 280 : cluster [DBG] 4.3 scrub ok 2024-02-13T00:52:28.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:27.554789+0000 mon.a (mon.0) 3474 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:52:28.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:27.559932+0000 mon.a (mon.0) 3475 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:52:28.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:27.559999+0000 mon.a (mon.0) 3476 : cluster [DBG] osdmap e720: 8 total, 8 up, 5 in 2024-02-13T00:52:28.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:27.564993+0000 mon.a (mon.0) 3477 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:28.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:26.521535+0000 mgr.y (mgr.14544) 596 : cluster [DBG] pgmap v1084: 204 pgs: 204 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-02-13T00:52:28.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:26.870228+0000 mon.c (mon.2) 498 : audit [INF] from='client.? 172.21.15.162:0/1193919010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:28.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:26.870554+0000 mon.a (mon.0) 3473 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:28.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:26.946800+0000 osd.0 (osd.0) 279 : cluster [DBG] 4.3 scrub starts 2024-02-13T00:52:28.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:26.954377+0000 osd.0 (osd.0) 280 : cluster [DBG] 4.3 scrub ok 2024-02-13T00:52:28.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:27.554789+0000 mon.a (mon.0) 3474 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:52:28.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:27.559932+0000 mon.a (mon.0) 3475 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:52:28.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:27.559999+0000 mon.a (mon.0) 3476 : cluster [DBG] osdmap e720: 8 total, 8 up, 5 in 2024-02-13T00:52:28.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:27.564993+0000 mon.a (mon.0) 3477 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:28.276 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:28.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:26.521535+0000 mgr.y (mgr.14544) 596 : cluster [DBG] pgmap v1084: 204 pgs: 204 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-02-13T00:52:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:26.870228+0000 mon.c (mon.2) 498 : audit [INF] from='client.? 172.21.15.162:0/1193919010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:26.870554+0000 mon.a (mon.0) 3473 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:26.946800+0000 osd.0 (osd.0) 279 : cluster [DBG] 4.3 scrub starts 2024-02-13T00:52:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:26.954377+0000 osd.0 (osd.0) 280 : cluster [DBG] 4.3 scrub ok 2024-02-13T00:52:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:27.554789+0000 mon.a (mon.0) 3474 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:52:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:27.559932+0000 mon.a (mon.0) 3475 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:52:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:27.559999+0000 mon.a (mon.0) 3476 : cluster [DBG] osdmap e720: 8 total, 8 up, 5 in 2024-02-13T00:52:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:27.564993+0000 mon.a (mon.0) 3477 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:28.381 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:28.413 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:28.437 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:28.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:28.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:28.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 -- 172.21.15.162:0/338666435 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00700f73a0 msgr2=0x7f00701039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:28.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 --2- 172.21.15.162:0/338666435 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00700f73a0 0x7f00701039a0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 rx=0x7f0068009be0 tx=0x7f0068004d10).stop 2024-02-13T00:52:28.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 -- 172.21.15.162:0/338666435 shutdown_connections 2024-02-13T00:52:28.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 --2- 172.21.15.162:0/338666435 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00700f73a0 0x7f00701039a0 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:28.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 --2- 172.21.15.162:0/338666435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00700fcb30 0x7f00700fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:28.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 --2- 172.21.15.162:0/338666435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00700fd490 0x7f00700f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 -- 172.21.15.162:0/338666435 >> 172.21.15.162:0/338666435 conn(0x7f007006eb90 msgr2=0x7f00700f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 -- 172.21.15.162:0/338666435 shutdown_connections 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 -- 172.21.15.162:0/338666435 wait complete. 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 librados: starting msgr at 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 librados: starting objecter 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 Processor -- start 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 -- start start 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 librados: setting wanted keys 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.487+0000 7f0076944700 1 librados: calling monclient init 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f0076944700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00700f73a0 0x7f00700f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f0076944700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00700fcb30 0x7f00700f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f0076944700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00700fd490 0x7f00700f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f0076944700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0070116470 con 0x7f00700f73a0 2024-02-13T00:52:28.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f0076944700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00701165e0 con 0x7f00700fcb30 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f0076944700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00701167a0 con 0x7f00700fd490 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00700f73a0 0x7f00700f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00700f73a0 0x7f00700f8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45122/0 (socket says 172.21.15.162:45122) 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006ffff700 1 -- 172.21.15.162:0/1005027324 learned_addr learned my addr 172.21.15.162:0/1005027324 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f0074ee1700 1 --2- 172.21.15.162:0/1005027324 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00700fd490 0x7f00700f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006f7fe700 1 --2- 172.21.15.162:0/1005027324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00700fcb30 0x7f00700f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006ffff700 1 -- 172.21.15.162:0/1005027324 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00700fd490 msgr2=0x7f00700f9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006ffff700 1 --2- 172.21.15.162:0/1005027324 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00700fd490 0x7f00700f9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006ffff700 1 -- 172.21.15.162:0/1005027324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00700fcb30 msgr2=0x7f00700f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006ffff700 1 --2- 172.21.15.162:0/1005027324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00700fcb30 0x7f00700f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006ffff700 1 -- 172.21.15.162:0/1005027324 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00680097c0 con 0x7f00700f73a0 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f0074ee1700 1 --2- 172.21.15.162:0/1005027324 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00700fd490 0x7f00700f9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006ffff700 1 --2- 172.21.15.162:0/1005027324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00700f73a0 0x7f00700f8930 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 rx=0x7f006000e970 tx=0x7f006000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006d7fa700 10 client.?.objecter ms_handle_connect 0x7f00700f73a0 2024-02-13T00:52:28.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:28.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006d7fa700 1 -- 172.21.15.162:0/1005027324 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060004660 con 0x7f00700f73a0 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f006d7fa700 1 -- 172.21.15.162:0/1005027324 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f006000f040 con 0x7f00700f73a0 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.488+0000 7f0076944700 1 -- 172.21.15.162:0/1005027324 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0070116a80 con 0x7f00700f73a0 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.489+0000 7f006d7fa700 1 -- 172.21.15.162:0/1005027324 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060014790 con 0x7f00700f73a0 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.489+0000 7f0076944700 10 client.27993.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.489+0000 7f0076944700 1 -- 172.21.15.162:0/1005027324 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0070116fc0 con 0x7f00700f73a0 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.489+0000 7f0076944700 1 librados: init done 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.489+0000 7f0076944700 1 -- 172.21.15.162:0/1005027324 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0070106090 con 0x7f00700f73a0 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.490+0000 7f006d7fa700 1 -- 172.21.15.162:0/1005027324 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f00600090d0 con 0x7f00700f73a0 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.490+0000 7f006d7fa700 1 --2- 172.21.15.162:0/1005027324 >> v2:172.21.15.162:6800/2866534053 conn(0x7f005c062c90 0x7f005c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.490+0000 7f006d7fa700 1 -- 172.21.15.162:0/1005027324 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(720..720 src has 449..720) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f006007cfd0 con 0x7f00700f73a0 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.490+0000 7f006d7fa700 10 client.27993.objecter ms_dispatch 0x7f0070001550 osd_map(720..720 src has 449..720) v4 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.490+0000 7f006d7fa700 3 client.27993.objecter handle_osd_map got epochs [720,720] > 0 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.490+0000 7f006d7fa700 3 client.27993.objecter handle_osd_map decoding full epoch 720 2024-02-13T00:52:28.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.490+0000 7f006f7fe700 1 --2- 172.21.15.162:0/1005027324 >> v2:172.21.15.162:6800/2866534053 conn(0x7f005c062c90 0x7f005c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:28.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.491+0000 7f006d7fa700 20 client.27993.objecter dump_active .. 0 homeless 2024-02-13T00:52:28.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.491+0000 7f006d7fa700 10 client.27993.objecter ms_handle_connect 0x7f005c062c90 2024-02-13T00:52:28.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.491+0000 7f006f7fe700 1 --2- 172.21.15.162:0/1005027324 >> v2:172.21.15.162:6800/2866534053 conn(0x7f005c062c90 0x7f005c065150 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7f00700fa0d0 tx=0x7f0058009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:28.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.495+0000 7f006d7fa700 1 -- 172.21.15.162:0/1005027324 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f006004af60 con 0x7f00700f73a0 2024-02-13T00:52:28.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:28.574 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:28.594 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:28.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:28.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:28.677+0000 7f0076944700 1 -- 172.21.15.162:0/1005027324 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd out", "ids": ["4"]} v 0) v1 -- 0x7f0070117210 con 0x7f00700f73a0 2024-02-13T00:52:28.698 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:28.724 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:52:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:52:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:52:28.726 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:28.745 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:28.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:28.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:28.849 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:52:28.876 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:28.895 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:52:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:28.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:52:28.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:52:28.678+0000 7f5f6486f700 -1 mon.a@0(leader).osd e721 definitely_dead 0 2024-02-13T00:52:28.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:29.027 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:29.078 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:29.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:29.189 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:29.217 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:29.238 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:29.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:29.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:29.368 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:29.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:29.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:29.473 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:29.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:29.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.566+0000 7f006d7fa700 1 -- 172.21.15.162:0/1005027324 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["4"]}]=0 marked out osd.4. v722) v1 ==== 87+0+0 (secure 0 0 0) 0x7f0060019070 con 0x7f00700f73a0 2024-02-13T00:52:29.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.572+0000 7f0076944700 10 librados: watch_flush enter 2024-02-13T00:52:29.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.575+0000 7f0076944700 10 librados: watch_flush exit 2024-02-13T00:52:29.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.575+0000 7f0076944700 20 client.27993.objecter shutdown clearing up homeless session... 2024-02-13T00:52:29.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.575+0000 7f0076944700 10 client.27993.objecter successfully canceled tick 2024-02-13T00:52:29.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.576+0000 7f0076944700 1 -- 172.21.15.162:0/1005027324 >> v2:172.21.15.162:6800/2866534053 conn(0x7f005c062c90 msgr2=0x7f005c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:29.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.576+0000 7f0076944700 1 --2- 172.21.15.162:0/1005027324 >> v2:172.21.15.162:6800/2866534053 conn(0x7f005c062c90 0x7f005c065150 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7f00700fa0d0 tx=0x7f0058009040).stop 2024-02-13T00:52:29.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.576+0000 7f0076944700 1 -- 172.21.15.162:0/1005027324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00700f73a0 msgr2=0x7f00700f8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:29.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.576+0000 7f0076944700 1 --2- 172.21.15.162:0/1005027324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00700f73a0 0x7f00700f8930 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 rx=0x7f006000e970 tx=0x7f006000b040).stop 2024-02-13T00:52:29.582 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:29.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.587+0000 7f0076944700 1 -- 172.21.15.162:0/1005027324 shutdown_connections 2024-02-13T00:52:29.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.587+0000 7f0076944700 1 --2- 172.21.15.162:0/1005027324 >> v2:172.21.15.162:6800/2866534053 conn(0x7f005c062c90 0x7f005c065150 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:29.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.587+0000 7f0076944700 1 --2- 172.21.15.162:0/1005027324 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00700fd490 0x7f00700f9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:29.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.587+0000 7f0076944700 1 --2- 172.21.15.162:0/1005027324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00700f73a0 0x7f00700f8930 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:29.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.587+0000 7f0076944700 1 --2- 172.21.15.162:0/1005027324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00700fcb30 0x7f00700f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:29.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.587+0000 7f0076944700 1 -- 172.21.15.162:0/1005027324 >> 172.21.15.162:0/1005027324 conn(0x7f007006eb90 msgr2=0x7f0070060e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.594+0000 7f0076944700 1 -- 172.21.15.162:0/1005027324 shutdown_connections 2024-02-13T00:52:29.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.594+0000 7f0076944700 1 -- 172.21.15.162:0/1005027324 wait complete. 2024-02-13T00:52:29.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:29.594+0000 7f0076944700 1 librados: shutdown 2024-02-13T00:52:29.606 INFO:teuthology.orchestra.run.smithi162.stderr:marked out osd.4. 2024-02-13T00:52:29.612 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:29.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:29.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:29.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:27.930247+0000 osd.0 (osd.0) 281 : cluster [DBG] 6.17 deep-scrub starts 2024-02-13T00:52:29.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:27.933635+0000 osd.0 (osd.0) 282 : cluster [DBG] 6.17 deep-scrub ok 2024-02-13T00:52:29.632 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:28.521919+0000 mgr.y (mgr.14544) 597 : cluster [DBG] pgmap v1087: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:28.561187+0000 mon.a (mon.0) 3478 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:28.561249+0000 mon.a (mon.0) 3479 : cluster [DBG] osdmap e721: 8 total, 8 up, 5 in 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:28.567718+0000 mon.a (mon.0) 3480 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:28.679964+0000 mon.a (mon.0) 3481 : audit [INF] from='client.? 172.21.15.162:0/1005027324' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:28.680050+0000 mon.a (mon.0) 3482 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:27.930247+0000 osd.0 (osd.0) 281 : cluster [DBG] 6.17 deep-scrub starts 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:27.933635+0000 osd.0 (osd.0) 282 : cluster [DBG] 6.17 deep-scrub ok 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:28.521919+0000 mgr.y (mgr.14544) 597 : cluster [DBG] pgmap v1087: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:28.561187+0000 mon.a (mon.0) 3478 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:28.561249+0000 mon.a (mon.0) 3479 : cluster [DBG] osdmap e721: 8 total, 8 up, 5 in 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:28.567718+0000 mon.a (mon.0) 3480 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:28.679964+0000 mon.a (mon.0) 3481 : audit [INF] from='client.? 172.21.15.162:0/1005027324' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-13T00:52:29.633 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:28.680050+0000 mon.a (mon.0) 3482 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-02-13T00:52:29.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:29.745 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:27.930247+0000 osd.0 (osd.0) 281 : cluster [DBG] 6.17 deep-scrub starts 2024-02-13T00:52:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:27.933635+0000 osd.0 (osd.0) 282 : cluster [DBG] 6.17 deep-scrub ok 2024-02-13T00:52:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:28.521919+0000 mgr.y (mgr.14544) 597 : cluster [DBG] pgmap v1087: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-13T00:52:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:28.561187+0000 mon.a (mon.0) 3478 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:28.561249+0000 mon.a (mon.0) 3479 : cluster [DBG] osdmap e721: 8 total, 8 up, 5 in 2024-02-13T00:52:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:28.567718+0000 mon.a (mon.0) 3480 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:28.679964+0000 mon.a (mon.0) 3481 : audit [INF] from='client.? 172.21.15.162:0/1005027324' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-13T00:52:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:28.680050+0000 mon.a (mon.0) 3482 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-02-13T00:52:30.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:28.985013+0000 osd.0 (osd.0) 283 : cluster [DBG] 4.b scrub starts 2024-02-13T00:52:30.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:28.988455+0000 osd.0 (osd.0) 284 : cluster [DBG] 4.b scrub ok 2024-02-13T00:52:30.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:29.411485+0000 osd.1 (osd.1) 305 : cluster [DBG] 2.6 scrub starts 2024-02-13T00:52:30.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:29.412332+0000 osd.1 (osd.1) 306 : cluster [DBG] 2.6 scrub ok 2024-02-13T00:52:30.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:29.565346+0000 mon.a (mon.0) 3483 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:30.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:29.565426+0000 mon.a (mon.0) 3484 : audit [INF] from='client.? 172.21.15.162:0/1005027324' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-13T00:52:30.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:29.565479+0000 mon.a (mon.0) 3485 : cluster [DBG] osdmap e722: 8 total, 8 up, 4 in 2024-02-13T00:52:30.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:29.576306+0000 mon.a (mon.0) 3486 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:52:30.879 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:30.548384+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:52:30.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:30.548560+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e723: 8 total, 8 up, 4 in 2024-02-13T00:52:30.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:30.551379+0000 mon.a (mon.0) 3489 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:52:30.880 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:28.985013+0000 osd.0 (osd.0) 283 : cluster [DBG] 4.b scrub starts 2024-02-13T00:52:30.880 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:28.988455+0000 osd.0 (osd.0) 284 : cluster [DBG] 4.b scrub ok 2024-02-13T00:52:30.880 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:29.411485+0000 osd.1 (osd.1) 305 : cluster [DBG] 2.6 scrub starts 2024-02-13T00:52:30.880 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:29.412332+0000 osd.1 (osd.1) 306 : cluster [DBG] 2.6 scrub ok 2024-02-13T00:52:30.880 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:29.565346+0000 mon.a (mon.0) 3483 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:30.881 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:29.565426+0000 mon.a (mon.0) 3484 : audit [INF] from='client.? 172.21.15.162:0/1005027324' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-13T00:52:30.881 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:29.565479+0000 mon.a (mon.0) 3485 : cluster [DBG] osdmap e722: 8 total, 8 up, 4 in 2024-02-13T00:52:30.881 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:29.576306+0000 mon.a (mon.0) 3486 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:52:30.881 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:30.548384+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:52:30.881 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:30.548560+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e723: 8 total, 8 up, 4 in 2024-02-13T00:52:30.881 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:30.551379+0000 mon.a (mon.0) 3489 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:52:30.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:52:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:28.985013+0000 osd.0 (osd.0) 283 : cluster [DBG] 4.b scrub starts 2024-02-13T00:52:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:28.988455+0000 osd.0 (osd.0) 284 : cluster [DBG] 4.b scrub ok 2024-02-13T00:52:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:29.411485+0000 osd.1 (osd.1) 305 : cluster [DBG] 2.6 scrub starts 2024-02-13T00:52:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:29.412332+0000 osd.1 (osd.1) 306 : cluster [DBG] 2.6 scrub ok 2024-02-13T00:52:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:29.565346+0000 mon.a (mon.0) 3483 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:29.565426+0000 mon.a (mon.0) 3484 : audit [INF] from='client.? 172.21.15.162:0/1005027324' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-13T00:52:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:29.565479+0000 mon.a (mon.0) 3485 : cluster [DBG] osdmap e722: 8 total, 8 up, 4 in 2024-02-13T00:52:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:29.576306+0000 mon.a (mon.0) 3486 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:52:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:30.548384+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:52:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:30.548560+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e723: 8 total, 8 up, 4 in 2024-02-13T00:52:31.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:30.551379+0000 mon.a (mon.0) 3489 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:52:31.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.851+0000 7f2921b39700 1 -- 172.21.15.162:0/2279076797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f291c0f8d10 msgr2=0x7f291c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.851+0000 7f2921b39700 1 --2- 172.21.15.162:0/2279076797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f291c0f8d10 0x7f291c0f90f0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 rx=0x7f290c004870 tx=0x7f290c009cb0).stop 2024-02-13T00:52:31.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.851+0000 7f2921b39700 1 -- 172.21.15.162:0/2279076797 shutdown_connections 2024-02-13T00:52:31.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.851+0000 7f2921b39700 1 --2- 172.21.15.162:0/2279076797 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f291c102270 0x7f291c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.851+0000 7f2921b39700 1 --2- 172.21.15.162:0/2279076797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f291c0f8d10 0x7f291c0f90f0 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.851+0000 7f2921b39700 1 --2- 172.21.15.162:0/2279076797 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f291c0f95e0 0x7f291c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.851+0000 7f2921b39700 1 -- 172.21.15.162:0/2279076797 >> 172.21.15.162:0/2279076797 conn(0x7f291c06eb90 msgr2=0x7f291c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.851+0000 7f2921b39700 1 -- 172.21.15.162:0/2279076797 shutdown_connections 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.851+0000 7f2921b39700 1 -- 172.21.15.162:0/2279076797 wait complete. 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.851+0000 7f2921b39700 1 librados: starting msgr at 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.851+0000 7f2921b39700 1 librados: starting objecter 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.851+0000 7f2921b39700 1 Processor -- start 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f2921b39700 1 -- start start 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f2921b39700 1 librados: setting wanted keys 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f2921b39700 1 librados: calling monclient init 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f2921b39700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f291c0f8d10 0x7f291c10e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:31.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f2921b39700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f291c0f95e0 0x7f291c10e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f2921b39700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f291c102270 0x7f291c1084e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f2921b39700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1089d0 con 0x7f291c0f95e0 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f2921b39700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c108b40 con 0x7f291c102270 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f2921b39700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291c108d00 con 0x7f291c0f8d10 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f291affd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f291c0f95e0 0x7f291c10e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f291affd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f291c0f95e0 0x7f291c10e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45128/0 (socket says 172.21.15.162:45128) 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f291affd700 1 -- 172.21.15.162:0/2984064873 learned_addr learned my addr 172.21.15.162:0/2984064873 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f291b7fe700 1 --2- 172.21.15.162:0/2984064873 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f291c0f8d10 0x7f291c10e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f291affd700 1 -- 172.21.15.162:0/2984064873 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f291c0f8d10 msgr2=0x7f291c10e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f291affd700 1 --2- 172.21.15.162:0/2984064873 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f291c0f8d10 0x7f291c10e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f291affd700 1 -- 172.21.15.162:0/2984064873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f291c102270 msgr2=0x7f291c1084e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f291affd700 1 --2- 172.21.15.162:0/2984064873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f291c102270 0x7f291c1084e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:52:31.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.852+0000 7f291affd700 1 -- 172.21.15.162:0/2984064873 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c005040 con 0x7f291c0f95e0 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.853+0000 7f291affd700 1 --2- 172.21.15.162:0/2984064873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f291c0f95e0 0x7f291c10e830 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7f291000e970 tx=0x7f291000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.853+0000 7f2918ff9700 10 client.?.objecter ms_handle_connect 0x7f291c0f95e0 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.853+0000 7f2918ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.853+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291000ee60 con 0x7f291c0f95e0 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.853+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291c108f60 con 0x7f291c0f95e0 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.853+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f291000f040 con 0x7f291c0f95e0 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.853+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2910007ea0 con 0x7f291c0f95e0 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.853+0000 7f2921b39700 10 client.27999.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.853+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c0f6b00 con 0x7f291c0f95e0 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.853+0000 7f2921b39700 1 librados: init done 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.853+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f291c1071b0 con 0x7f291c0f95e0 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.855+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f291000cb10 con 0x7f291c0f95e0 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.855+0000 7f2918ff9700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2904062c90 0x7f2904065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.855+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(723..723 src has 449..723) v4 ==== 8722+0+0 (secure 0 0 0) 0x7f291007d410 con 0x7f291c0f95e0 2024-02-13T00:52:31.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.855+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 osd_map(723..723 src has 449..723) v4 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.855+0000 7f2918ff9700 3 client.27999.objecter handle_osd_map got epochs [723,723] > 0 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.855+0000 7f2918ff9700 3 client.27999.objecter handle_osd_map decoding full epoch 723 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.855+0000 7f291b7fe700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2904062c90 0x7f2904065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.855+0000 7f2918ff9700 20 client.27999.objecter dump_active .. 0 homeless 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.855+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v723) v1 ==== 54+0+16 (secure 0 0 0) 0x7f291007d8b0 con 0x7f291c0f95e0 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.856+0000 7f2918ff9700 10 client.27999.objecter ms_handle_connect 0x7f2904062c90 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.856+0000 7f291b7fe700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2904062c90 0x7f2904065150 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7f290c009f20 tx=0x7f290c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.857+0000 7f2921b39700 10 client.27999.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.857+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 1 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.857+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6801/2103743338 conn(0x7f291c028590 0x7f291c0fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.857+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c1099c0 osd=0 3 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.857+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c1099c0 osd=0 3 2024-02-13T00:52:31.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.857+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c1099c0 osd=0 3 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.857+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 1 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.857+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c1099c0 osd=0 3 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.857+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 0 1 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.857+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c1099c0 osd=0 4 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.857+0000 7f2921b39700 10 client.27999.objecter _send_command 1 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.857+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f291c05a0e0 con 0x7f291c028590 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.859+0000 7f291bfff700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6801/2103743338 conn(0x7f291c028590 0x7f291c0fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.859+0000 7f2918ff9700 10 client.27999.objecter ms_handle_connect 0x7f291c028590 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.859+0000 7f291bfff700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6801/2103743338 conn(0x7f291c028590 0x7f291c0fbd00 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.860+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f291c05a0e0 con 0x7f291c028590 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.860+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.860+0000 7f2918ff9700 10 client.27999.objecter _finish_command 1 = system:0 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.860+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c1099c0 osd=0 4 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.860+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 0 1 2024-02-13T00:52:31.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.876+0000 7f2921b39700 10 client.27999.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.876+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 2 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.876+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c1099c0 osd=0 3 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.876+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c1099c0 osd=0 3 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.876+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c1099c0 osd=0 3 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.876+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 2 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.876+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c1099c0 osd=0 3 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.876+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 0 2 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.876+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c1099c0 osd=0 4 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.876+0000 7f2921b39700 10 client.27999.objecter _send_command 2 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.876+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f291c0fc1f0 con 0x7f291c028590 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f291c0fc1f0 con 0x7f291c028590 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2918ff9700 10 client.27999.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c1099c0 osd=0 4 2024-02-13T00:52:31.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 0 2 2024-02-13T00:52:31.303 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:52:31.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2921b39700 10 client.27999.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:31.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 3 2024-02-13T00:52:31.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6805/150599780 conn(0x7f291c0597b0 0x7f291c0f6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:31.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c059640 osd=1 3 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c059640 osd=1 3 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c059640 osd=1 3 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 3 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c059640 osd=1 3 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 1 3 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c059640 osd=1 4 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2921b39700 10 client.27999.objecter _send_command 3 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f291c0280f0 con 0x7f291c0597b0 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.878+0000 7f291b7fe700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6805/150599780 conn(0x7f291c0597b0 0x7f291c0f6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.879+0000 7f2918ff9700 10 client.27999.objecter ms_handle_connect 0x7f291c0597b0 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.879+0000 7f291b7fe700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6805/150599780 conn(0x7f291c0597b0 0x7f291c0f6d20 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.879+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f291c0280f0 con 0x7f291c0597b0 2024-02-13T00:52:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.879+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:52:31.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.879+0000 7f2918ff9700 10 client.27999.objecter _finish_command 3 = system:0 2024-02-13T00:52:31.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.879+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c059640 osd=1 4 2024-02-13T00:52:31.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.879+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 1 3 2024-02-13T00:52:31.305 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:52:31.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.907+0000 7f2921b39700 10 client.27999.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:31.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.907+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 4 2024-02-13T00:52:31.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.907+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c059640 osd=1 3 2024-02-13T00:52:31.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.907+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c059640 osd=1 3 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.907+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c059640 osd=1 3 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.907+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 4 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.907+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c059640 osd=1 3 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.907+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 1 4 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.907+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c059640 osd=1 4 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.907+0000 7f2921b39700 10 client.27999.objecter _send_command 4 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.907+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f291c0f7210 con 0x7f291c0597b0 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f291c0f7210 con 0x7f291c0597b0 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2918ff9700 10 client.27999.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c059640 osd=1 4 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 1 4 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2921b39700 10 client.27999.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 5 2024-02-13T00:52:31.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6809/3689719376 conn(0x7f291c0277c0 0x7f291c0fcb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:31.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c027650 osd=2 3 2024-02-13T00:52:31.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c027650 osd=2 3 2024-02-13T00:52:31.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c027650 osd=2 3 2024-02-13T00:52:31.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 5 2024-02-13T00:52:31.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c027650 osd=2 3 2024-02-13T00:52:31.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 2 5 2024-02-13T00:52:31.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c027650 osd=2 4 2024-02-13T00:52:31.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2921b39700 10 client.27999.objecter _send_command 5 2024-02-13T00:52:31.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.909+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f291c0f8a70 con 0x7f291c0277c0 2024-02-13T00:52:31.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.911+0000 7f291affd700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6809/3689719376 conn(0x7f291c0277c0 0x7f291c0fcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:31.307 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:52:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.911+0000 7f291affd700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6809/3689719376 conn(0x7f291c0277c0 0x7f291c0fcb90 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.911+0000 7f2918ff9700 10 client.27999.objecter ms_handle_connect 0x7f291c0277c0 2024-02-13T00:52:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.912+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f291c0f8a70 con 0x7f291c0277c0 2024-02-13T00:52:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.912+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:52:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.912+0000 7f2918ff9700 10 client.27999.objecter _finish_command 5 = system:0 2024-02-13T00:52:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.912+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c027650 osd=2 4 2024-02-13T00:52:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.912+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 2 5 2024-02-13T00:52:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.939+0000 7f2921b39700 10 client.27999.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.939+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 6 2024-02-13T00:52:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.939+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c027650 osd=2 3 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.939+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c027650 osd=2 3 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.939+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c027650 osd=2 3 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.939+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 6 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.939+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c027650 osd=2 3 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.939+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 2 6 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.939+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c027650 osd=2 4 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.939+0000 7f2921b39700 10 client.27999.objecter _send_command 6 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.939+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f291c0fd2c0 con 0x7f291c0277c0 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f291c0fd2c0 con 0x7f291c0277c0 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2918ff9700 10 client.27999.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c027650 osd=2 4 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 2 6 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2921b39700 10 client.27999.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 7 2024-02-13T00:52:31.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6813/604971011 conn(0x7f291c0f7bb0 0x7f291c0f7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c0fd490 osd=3 3 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0fd490 osd=3 3 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c0fd490 osd=3 3 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 7 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c0fd490 osd=3 3 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 3 7 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0fd490 osd=3 4 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2921b39700 10 client.27999.objecter _send_command 7 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.942+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f291c124ea0 con 0x7f291c0f7bb0 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.945+0000 7f291bfff700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6813/604971011 conn(0x7f291c0f7bb0 0x7f291c0f7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.945+0000 7f291bfff700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6813/604971011 conn(0x7f291c0f7bb0 0x7f291c0f7fd0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.945+0000 7f2918ff9700 10 client.27999.objecter ms_handle_connect 0x7f291c0f7bb0 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.947+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f291c124ea0 con 0x7f291c0f7bb0 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.947+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:52:31.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.947+0000 7f2918ff9700 10 client.27999.objecter _finish_command 7 = system:0 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.947+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c0fd490 osd=3 4 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.947+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 3 7 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.975+0000 7f2921b39700 10 client.27999.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.975+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 8 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.975+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c0fd490 osd=3 3 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.975+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0fd490 osd=3 3 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.975+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c0fd490 osd=3 3 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.975+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 8 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.975+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c0fd490 osd=3 3 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.975+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 3 8 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.975+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0fd490 osd=3 4 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.975+0000 7f2921b39700 10 client.27999.objecter _send_command 8 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.975+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f291c0f7620 con 0x7f291c0f7bb0 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f291c0f7620 con 0x7f291c0f7bb0 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:31.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2918ff9700 10 client.27999.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c0fd490 osd=3 4 2024-02-13T00:52:31.312 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:52:31.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 3 8 2024-02-13T00:52:31.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2921b39700 10 client.27999.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:31.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 9 2024-02-13T00:52:31.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6800/3344263388 conn(0x7f291c120290 0x7f291c1238c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c0f73e0 osd=4 3 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0f73e0 osd=4 3 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c0f73e0 osd=4 3 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 9 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c0f73e0 osd=4 3 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 4 9 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0f73e0 osd=4 4 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2921b39700 10 client.27999.objecter _send_command 9 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.977+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f291c0f7550 con 0x7f291c120290 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.978+0000 7f291b7fe700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6800/3344263388 conn(0x7f291c120290 0x7f291c1238c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.978+0000 7f2918ff9700 10 client.27999.objecter ms_handle_connect 0x7f291c120290 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.978+0000 7f291b7fe700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6800/3344263388 conn(0x7f291c120290 0x7f291c1238c0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.979+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f291c0f7550 con 0x7f291c120290 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.979+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:52:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.979+0000 7f2918ff9700 10 client.27999.objecter _finish_command 9 = system:0 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.979+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c0f73e0 osd=4 4 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:30.979+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 4 9 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.008+0000 7f2921b39700 10 client.27999.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.008+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 10 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.008+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c0f73e0 osd=4 3 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.008+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0f73e0 osd=4 3 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.008+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c0f73e0 osd=4 3 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.008+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 10 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.008+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c0f73e0 osd=4 3 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.008+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 4 10 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.008+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0f73e0 osd=4 4 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.008+0000 7f2921b39700 10 client.27999.objecter _send_command 10 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.008+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f291c0f7550 con 0x7f291c120290 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f291c0f7550 con 0x7f291c120290 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:31.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2918ff9700 10 client.27999.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c0f73e0 osd=4 4 2024-02-13T00:52:31.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 4 10 2024-02-13T00:52:31.315 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:52:31.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2921b39700 10 client.27999.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:31.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 11 2024-02-13T00:52:31.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6804/2277917372 conn(0x7f291c1242c0 0x7f291c1246e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c0f7550 osd=5 3 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0f7550 osd=5 3 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c0f7550 osd=5 3 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 11 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c0f7550 osd=5 3 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 5 11 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0f7550 osd=5 4 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2921b39700 10 client.27999.objecter _send_command 11 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.011+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f291c124d30 con 0x7f291c1242c0 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.012+0000 7f291affd700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6804/2277917372 conn(0x7f291c1242c0 0x7f291c1246e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.012+0000 7f2918ff9700 10 client.27999.objecter ms_handle_connect 0x7f291c1242c0 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.012+0000 7f291affd700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6804/2277917372 conn(0x7f291c1242c0 0x7f291c1246e0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:31.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.012+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f291c124d30 con 0x7f291c1242c0 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.012+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.012+0000 7f2918ff9700 10 client.27999.objecter _finish_command 11 = system:0 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.012+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c0f7550 osd=5 4 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.012+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 5 11 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.042+0000 7f2921b39700 10 client.27999.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.042+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 12 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.042+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c0f7550 osd=5 3 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.042+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0f7550 osd=5 3 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.042+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c0f7550 osd=5 3 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.042+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 12 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.042+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c0f7550 osd=5 3 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.042+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 5 12 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.042+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0f7550 osd=5 4 2024-02-13T00:52:31.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.042+0000 7f2921b39700 10 client.27999.objecter _send_command 12 2024-02-13T00:52:31.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.042+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f291c124d30 con 0x7f291c1242c0 2024-02-13T00:52:31.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f291c124d30 con 0x7f291c1242c0 2024-02-13T00:52:31.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:31.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2918ff9700 10 client.27999.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c0f7550 osd=5 4 2024-02-13T00:52:31.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 5 12 2024-02-13T00:52:31.318 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:52:31.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2921b39700 10 client.27999.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 13 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6808/1506640341 conn(0x7f291c12abf0 0x7f291c12b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c124d30 osd=6 3 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c124d30 osd=6 3 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c124d30 osd=6 3 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 13 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c124d30 osd=6 3 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 6 13 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c124d30 osd=6 4 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2921b39700 10 client.27999.objecter _send_command 13 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f291c12e9d0 con 0x7f291c12abf0 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f291bfff700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6808/1506640341 conn(0x7f291c12abf0 0x7f291c12b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:31.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f2918ff9700 10 client.27999.objecter ms_handle_connect 0x7f291c12abf0 2024-02-13T00:52:31.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.044+0000 7f291bfff700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6808/1506640341 conn(0x7f291c12abf0 0x7f291c12b050 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:31.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.045+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f291c12e9d0 con 0x7f291c12abf0 2024-02-13T00:52:31.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.045+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:52:31.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.045+0000 7f2918ff9700 10 client.27999.objecter _finish_command 13 = system:0 2024-02-13T00:52:31.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.045+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c124d30 osd=6 4 2024-02-13T00:52:31.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.045+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 6 13 2024-02-13T00:52:31.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.060+0000 7f2921b39700 10 client.27999.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:31.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.060+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 14 2024-02-13T00:52:31.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.060+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c124d30 osd=6 3 2024-02-13T00:52:31.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.060+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c124d30 osd=6 3 2024-02-13T00:52:31.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.060+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c124d30 osd=6 3 2024-02-13T00:52:31.321 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:52:31.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.060+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 14 2024-02-13T00:52:31.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.060+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c124d30 osd=6 3 2024-02-13T00:52:31.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.060+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 6 14 2024-02-13T00:52:31.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.060+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c124d30 osd=6 4 2024-02-13T00:52:31.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.060+0000 7f2921b39700 10 client.27999.objecter _send_command 14 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.060+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f291c129030 con 0x7f291c12abf0 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f291c129030 con 0x7f291c12abf0 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2918ff9700 10 client.27999.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c124d30 osd=6 4 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 6 14 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2921b39700 10 client.27999.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 15 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6812/1964192772 conn(0x7f291c129740 0x7f291c129b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c129030 osd=7 3 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c129030 osd=7 3 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c129030 osd=7 3 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 15 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c129030 osd=7 3 2024-02-13T00:52:31.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 7 15 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c129030 osd=7 4 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2921b39700 10 client.27999.objecter _send_command 15 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f291c12a070 con 0x7f291c129740 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f291b7fe700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6812/1964192772 conn(0x7f291c129740 0x7f291c129b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f2918ff9700 10 client.27999.objecter ms_handle_connect 0x7f291c129740 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.063+0000 7f291b7fe700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6812/1964192772 conn(0x7f291c129740 0x7f291c129b80 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.064+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f291c12a070 con 0x7f291c129740 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.064+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.064+0000 7f2918ff9700 10 client.27999.objecter _finish_command 15 = system:0 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.064+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c129030 osd=7 4 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.064+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 7 15 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.095+0000 7f2921b39700 10 client.27999.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.095+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign -1 16 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.095+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c129030 osd=7 3 2024-02-13T00:52:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.095+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c129030 osd=7 3 2024-02-13T00:52:31.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.095+0000 7f2921b39700 20 client.27999.objecter _get_session s=0x7f291c129030 osd=7 3 2024-02-13T00:52:31.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.095+0000 7f2921b39700 15 client.27999.objecter _session_command_op_remove -1 16 2024-02-13T00:52:31.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.095+0000 7f2921b39700 20 client.27999.objecter get_session s=0x7f291c129030 osd=7 3 2024-02-13T00:52:31.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.095+0000 7f2921b39700 15 client.27999.objecter _session_command_op_assign 7 16 2024-02-13T00:52:31.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.095+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c129030 osd=7 4 2024-02-13T00:52:31.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.095+0000 7f2921b39700 10 client.27999.objecter _send_command 16 2024-02-13T00:52:31.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.095+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f291c12a070 con 0x7f291c129740 2024-02-13T00:52:31.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.096+0000 7f2918ff9700 1 -- 172.21.15.162:0/2984064873 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f291c12a070 con 0x7f291c129740 2024-02-13T00:52:31.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.096+0000 7f2918ff9700 10 client.27999.objecter ms_dispatch 0x7f291c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:31.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.096+0000 7f2918ff9700 10 client.27999.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.096+0000 7f2918ff9700 20 client.27999.objecter put_session s=0x7f291c129030 osd=7 4 2024-02-13T00:52:31.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.096+0000 7f2918ff9700 15 client.27999.objecter _session_command_op_remove 7 16 2024-02-13T00:52:31.325 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:52:31.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 10 librados: watch_flush enter 2024-02-13T00:52:31.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 10 librados: watch_flush exit 2024-02-13T00:52:31.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 10 client.27999.objecter close_session for osd.0 2024-02-13T00:52:31.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6801/2103743338 conn(0x7f291c028590 msgr2=0x7f291c0fbd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6801/2103743338 conn(0x7f291c028590 0x7f291c0fbd00 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c1099c0 osd=0 1 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 10 client.27999.objecter close_session for osd.1 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6805/150599780 conn(0x7f291c0597b0 msgr2=0x7f291c0f6d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6805/150599780 conn(0x7f291c0597b0 0x7f291c0f6d20 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c059640 osd=1 1 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 10 client.27999.objecter close_session for osd.2 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6809/3689719376 conn(0x7f291c0277c0 msgr2=0x7f291c0fcb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6809/3689719376 conn(0x7f291c0277c0 0x7f291c0fcb90 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c027650 osd=2 1 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 10 client.27999.objecter close_session for osd.3 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6813/604971011 conn(0x7f291c0f7bb0 msgr2=0x7f291c0f7fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6813/604971011 conn(0x7f291c0f7bb0 0x7f291c0f7fd0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0fd490 osd=3 1 2024-02-13T00:52:31.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 10 client.27999.objecter close_session for osd.4 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6800/3344263388 conn(0x7f291c120290 msgr2=0x7f291c1238c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6800/3344263388 conn(0x7f291c120290 0x7f291c1238c0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0f73e0 osd=4 1 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 10 client.27999.objecter close_session for osd.5 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6804/2277917372 conn(0x7f291c1242c0 msgr2=0x7f291c1246e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6804/2277917372 conn(0x7f291c1242c0 0x7f291c1246e0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f291bfff700 1 -- 172.21.15.162:0/2984064873 reap_dead start 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c0f7550 osd=5 1 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 10 client.27999.objecter close_session for osd.6 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6808/1506640341 conn(0x7f291c12abf0 msgr2=0x7f291c12b050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6808/1506640341 conn(0x7f291c12abf0 0x7f291c12b050 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f291bfff700 1 -- 172.21.15.162:0/2984064873 reap_dead start 2024-02-13T00:52:31.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c124d30 osd=6 1 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 10 client.27999.objecter close_session for osd.7 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6812/1964192772 conn(0x7f291c129740 msgr2=0x7f291c129b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.181:6812/1964192772 conn(0x7f291c129740 0x7f291c129b80 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 20 client.27999.objecter put_session s=0x7f291c129030 osd=7 1 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 20 client.27999.objecter shutdown clearing up homeless session... 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 10 client.27999.objecter successfully canceled tick 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f291bfff700 1 -- 172.21.15.162:0/2984064873 reap_dead start 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f291bfff700 1 -- 172.21.15.162:0/2984064873 reap_dead start 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2904062c90 msgr2=0x7f2904065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2904062c90 0x7f2904065150 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7f290c009f20 tx=0x7f290c00b040).stop 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f291c0f95e0 msgr2=0x7f291c10e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f291c0f95e0 0x7f291c10e830 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7f291000e970 tx=0x7f291000b040).stop 2024-02-13T00:52:31.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 shutdown_connections 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f291c0f95e0 0x7f291c10e830 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 --2- 172.21.15.162:0/2984064873 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2904062c90 0x7f2904065150 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 >> 172.21.15.162:0/2984064873 conn(0x7f291c06eb90 msgr2=0x7f291c102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 shutdown_connections 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 -- 172.21.15.162:0/2984064873 wait complete. 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:31.097+0000 7f2921b39700 1 librados: shutdown 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.329 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:52:31.496 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:31.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:31.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:31.638 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:31.668 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:31.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:31.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:31.797 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:30.014641+0000 osd.0 (osd.0) 285 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:52:31.797 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:30.017613+0000 osd.0 (osd.0) 286 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:52:31.797 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:30.186462+0000 osd.6 (osd.6) 228 : cluster [DBG] 28.13 scrub starts 2024-02-13T00:52:31.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:30.189921+0000 osd.6 (osd.6) 229 : cluster [DBG] 28.13 scrub ok 2024-02-13T00:52:31.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:30.522902+0000 mgr.y (mgr.14544) 598 : cluster [DBG] pgmap v1090: 204 pgs: 115 peering, 89 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:52:31.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:30.856187+0000 mon.a (mon.0) 3490 : audit [DBG] from='client.? 172.21.15.162:0/2984064873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:31.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:31.552191+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:52:31.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:31.552281+0000 mon.a (mon.0) 3492 : cluster [DBG] osdmap e724: 8 total, 8 up, 4 in 2024-02-13T00:52:31.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:31.552836+0000 mon.a (mon.0) 3493 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:52:31.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:31.554304+0000 osd.0 (osd.0) 287 : cluster [DBG] 5.e starting backfill to osd.1 from (0'0,0'0] MAX to 681'16 2024-02-13T00:52:31.798 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:31.554413+0000 osd.0 (osd.0) 288 : cluster [DBG] 5.d starting backfill to osd.6 from (0'0,0'0] MAX to 681'16 2024-02-13T00:52:31.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:30.014641+0000 osd.0 (osd.0) 285 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:52:31.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:30.017613+0000 osd.0 (osd.0) 286 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:52:31.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:30.186462+0000 osd.6 (osd.6) 228 : cluster [DBG] 28.13 scrub starts 2024-02-13T00:52:31.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:30.189921+0000 osd.6 (osd.6) 229 : cluster [DBG] 28.13 scrub ok 2024-02-13T00:52:31.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:30.522902+0000 mgr.y (mgr.14544) 598 : cluster [DBG] pgmap v1090: 204 pgs: 115 peering, 89 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:52:31.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:30.856187+0000 mon.a (mon.0) 3490 : audit [DBG] from='client.? 172.21.15.162:0/2984064873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:31.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:31.552191+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:52:31.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:31.552281+0000 mon.a (mon.0) 3492 : cluster [DBG] osdmap e724: 8 total, 8 up, 4 in 2024-02-13T00:52:31.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:31.552836+0000 mon.a (mon.0) 3493 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:52:31.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:31.554304+0000 osd.0 (osd.0) 287 : cluster [DBG] 5.e starting backfill to osd.1 from (0'0,0'0] MAX to 681'16 2024-02-13T00:52:31.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:31.554413+0000 osd.0 (osd.0) 288 : cluster [DBG] 5.d starting backfill to osd.6 from (0'0,0'0] MAX to 681'16 2024-02-13T00:52:31.799 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:30.014641+0000 osd.0 (osd.0) 285 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:52:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:30.017613+0000 osd.0 (osd.0) 286 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:52:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:30.186462+0000 osd.6 (osd.6) 228 : cluster [DBG] 28.13 scrub starts 2024-02-13T00:52:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:30.189921+0000 osd.6 (osd.6) 229 : cluster [DBG] 28.13 scrub ok 2024-02-13T00:52:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:30.522902+0000 mgr.y (mgr.14544) 598 : cluster [DBG] pgmap v1090: 204 pgs: 115 peering, 89 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:52:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:30.856187+0000 mon.a (mon.0) 3490 : audit [DBG] from='client.? 172.21.15.162:0/2984064873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:31.552191+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:52:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:31.552281+0000 mon.a (mon.0) 3492 : cluster [DBG] osdmap e724: 8 total, 8 up, 4 in 2024-02-13T00:52:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:31.552836+0000 mon.a (mon.0) 3493 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-13T00:52:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:31.554304+0000 osd.0 (osd.0) 287 : cluster [DBG] 5.e starting backfill to osd.1 from (0'0,0'0] MAX to 681'16 2024-02-13T00:52:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:31.554413+0000 osd.0 (osd.0) 288 : cluster [DBG] 5.d starting backfill to osd.6 from (0'0,0'0] MAX to 681'16 2024-02-13T00:52:32.655 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:32.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:32.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:32.786 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:32.805 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:32.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:32.890 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:31.550317+0000 osd.6 (osd.6) 230 : cluster [DBG] 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 681'19 2024-02-13T00:52:32.890 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:31.554946+0000 osd.6 (osd.6) 231 : cluster [DBG] 5.f starting backfill to osd.0 from (0'0,0'0] MAX to 504'16 2024-02-13T00:52:32.890 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:32.554010+0000 mon.a (mon.0) 3494 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-13T00:52:32.890 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:32.554092+0000 mon.a (mon.0) 3495 : cluster [DBG] osdmap e725: 8 total, 8 up, 4 in 2024-02-13T00:52:32.890 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:31.550317+0000 osd.6 (osd.6) 230 : cluster [DBG] 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 681'19 2024-02-13T00:52:32.890 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:31.554946+0000 osd.6 (osd.6) 231 : cluster [DBG] 5.f starting backfill to osd.0 from (0'0,0'0] MAX to 504'16 2024-02-13T00:52:32.890 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:32.554010+0000 mon.a (mon.0) 3494 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-13T00:52:32.890 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:32.554092+0000 mon.a (mon.0) 3495 : cluster [DBG] osdmap e725: 8 total, 8 up, 4 in 2024-02-13T00:52:32.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:32.937 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:33.022 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:52:33.023 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:52:33.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:31.550317+0000 osd.6 (osd.6) 230 : cluster [DBG] 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 681'19 2024-02-13T00:52:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:31.554946+0000 osd.6 (osd.6) 231 : cluster [DBG] 5.f starting backfill to osd.0 from (0'0,0'0] MAX to 504'16 2024-02-13T00:52:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:32.554010+0000 mon.a (mon.0) 3494 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-128","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-13T00:52:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:32.554092+0000 mon.a (mon.0) 3495 : cluster [DBG] osdmap e725: 8 total, 8 up, 4 in 2024-02-13T00:52:33.289 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:33.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:33.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:33.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:33.418 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:33.436 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:33.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:33.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:33.571 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:32.523264+0000 mgr.y (mgr.14544) 599 : cluster [DBG] pgmap v1093: 204 pgs: 115 peering, 89 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:52:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:33.552702+0000 mon.a (mon.0) 3496 : cluster [DBG] osdmap e726: 8 total, 8 up, 4 in 2024-02-13T00:52:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:33.572571+0000 mon.a (mon.0) 3497 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:33.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:33.572949+0000 mon.a (mon.0) 3498 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-128"}]: dispatch 2024-02-13T00:52:33.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:32.523264+0000 mgr.y (mgr.14544) 599 : cluster [DBG] pgmap v1093: 204 pgs: 115 peering, 89 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:52:33.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:33.552702+0000 mon.a (mon.0) 3496 : cluster [DBG] osdmap e726: 8 total, 8 up, 4 in 2024-02-13T00:52:33.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:33.572571+0000 mon.a (mon.0) 3497 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:33.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:33.572949+0000 mon.a (mon.0) 3498 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-128"}]: dispatch 2024-02-13T00:52:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:32.523264+0000 mgr.y (mgr.14544) 599 : cluster [DBG] pgmap v1093: 204 pgs: 115 peering, 89 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:52:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:33.552702+0000 mon.a (mon.0) 3496 : cluster [DBG] osdmap e726: 8 total, 8 up, 4 in 2024-02-13T00:52:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:33.572571+0000 mon.a (mon.0) 3497 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:33.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:33.572949+0000 mon.a (mon.0) 3498 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-128"}]: dispatch 2024-02-13T00:52:33.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-13T00:52:33.921 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:33.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:33.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:34.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:52:34.050 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:34.074 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:52:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:34.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:52:34.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:34.203 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:34.221 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:34.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:34.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:34.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:52:34.351 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:34.363 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:34.370 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:52:34.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:34.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:52:34.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:34.500 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:34.624 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:34.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:34.728 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:34.755 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:34.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:34.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:34.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:34.958 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:35.196 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:35.206 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:35.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:35.311 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:35.341 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:35.364 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:35.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:35.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:34.342422+0000 mon.a (mon.0) 3499 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:35.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:34.523656+0000 mgr.y (mgr.14544) 600 : cluster [DBG] pgmap v1096: 204 pgs: 115 peering, 89 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:52:35.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:34.559415+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-128"}]': finished 2024-02-13T00:52:35.462 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:34.559515+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e727: 8 total, 8 up, 4 in 2024-02-13T00:52:35.462 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:34.342422+0000 mon.a (mon.0) 3499 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:35.463 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:34.523656+0000 mgr.y (mgr.14544) 600 : cluster [DBG] pgmap v1096: 204 pgs: 115 peering, 89 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:52:35.463 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:34.559415+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-128"}]': finished 2024-02-13T00:52:35.463 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:34.559515+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e727: 8 total, 8 up, 4 in 2024-02-13T00:52:35.468 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:35.497 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:35.516 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:35.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11908 ms) 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 00000010 69 73 |is| 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 00000012 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12007 ms) 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = pacific 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6012 ms) 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 43 tests from LibRadosTwoPoolsPP (496814 ms total) 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-02-13T00:52:35.574 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1074 ms) 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10931 ms) 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18001 ms) 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30008 ms total) 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7053 ms) 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-02-13T00:52:35.575 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (7929 ms) 2024-02-13T00:52:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-02-13T00:52:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: waiting for scrub... 2024-02-13T00:52:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: done waiting 2024-02-13T00:52:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (23997 ms) 2024-02-13T00:52:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-02-13T00:52:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10002 ms) 2024-02-13T00:52:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-02-13T00:52:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8006 ms) 2024-02-13T00:52:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-02-13T00:52:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8037 ms) 2024-02-13T00:52:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-02-13T00:52:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10065 ms) 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7915 ms) 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13071 ms) 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7958 ms) 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12972 ms) 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8045 ms) 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (7973 ms) 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-02-13T00:52:35.577 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: hmm, no HitSet yet 2024-02-13T00:52:35.578 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: ok, hit_set contains 248:602f83fe:::foo:head 2024-02-13T00:52:35.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:35.644 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:34.342422+0000 mon.a (mon.0) 3499 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:35.644 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:34.523656+0000 mgr.y (mgr.14544) 600 : cluster [DBG] pgmap v1096: 204 pgs: 115 peering, 89 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:52:35.645 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:34.559415+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-128"}]': finished 2024-02-13T00:52:35.645 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:34.559515+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e727: 8 total, 8 up, 4 in 2024-02-13T00:52:35.646 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:35.664 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:35.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:35.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:35.797 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:36.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:36.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:36.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:36.131 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:36.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:52:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:52:36.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:52:36.394 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:52:36] "GET /metrics HTTP/1.1" 200 189720 "" "Prometheus/2.33.4" 2024-02-13T00:52:36.687 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 6, 7] out_osds: [3, 2, 5, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-13T00:52:36.687 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:52:36.687 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-13T00:52:36.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-13T00:52:36.712 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:34.637438+0000 osd.3 (osd.3) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:36.712 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:34.637589+0000 osd.3 (osd.3) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:36.713 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:35.022195+0000 osd.0 (osd.0) 289 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:52:36.713 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:35.022987+0000 osd.0 (osd.0) 290 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:52:36.713 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:35.570132+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e728: 8 total, 8 up, 4 in 2024-02-13T00:52:36.713 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:35.906270+0000 mgr.y (mgr.14544) 601 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:36.713 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:34.637438+0000 osd.3 (osd.3) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:36.713 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:34.637589+0000 osd.3 (osd.3) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:36.713 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:35.022195+0000 osd.0 (osd.0) 289 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:52:36.713 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:35.022987+0000 osd.0 (osd.0) 290 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:52:36.713 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:35.570132+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e728: 8 total, 8 up, 4 in 2024-02-13T00:52:36.713 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:35.906270+0000 mgr.y (mgr.14544) 601 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:36.780 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:36.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:36.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:52:36.923 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:34.637438+0000 osd.3 (osd.3) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:36.923 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:34.637589+0000 osd.3 (osd.3) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:36.923 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:35.022195+0000 osd.0 (osd.0) 289 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:52:36.924 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:35.022987+0000 osd.0 (osd.0) 290 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:52:36.924 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:35.570132+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e728: 8 total, 8 up, 4 in 2024-02-13T00:52:36.924 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:35.906270+0000 mgr.y (mgr.14544) 601 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:36.925 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:36.946 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:52:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:36.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:52:37.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:37.079 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:37.099 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:37.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:37.203 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:37.231 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:37.250 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:37.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:37.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:37.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:37.380 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:37.399 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:37.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:37.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.412+0000 7f082ade2700 1 -- 172.21.15.162:0/3057336088 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08240fa440 msgr2=0x7f08240f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:37.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.412+0000 7f082ade2700 1 --2- 172.21.15.162:0/3057336088 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08240fa440 0x7f08240f7e20 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7f0810004870 tx=0x7f0810009cb0).stop 2024-02-13T00:52:37.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.412+0000 7f082ade2700 1 -- 172.21.15.162:0/3057336088 shutdown_connections 2024-02-13T00:52:37.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.412+0000 7f082ade2700 1 --2- 172.21.15.162:0/3057336088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08240f8500 0x7f0824104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:37.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.412+0000 7f082ade2700 1 --2- 172.21.15.162:0/3057336088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08240f9b70 0x7f08240f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:37.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.412+0000 7f082ade2700 1 --2- 172.21.15.162:0/3057336088 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08240fa440 0x7f08240f7e20 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:37.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.412+0000 7f082ade2700 1 -- 172.21.15.162:0/3057336088 >> 172.21.15.162:0/3057336088 conn(0x7f08240f38c0 msgr2=0x7f08240f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:37.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.412+0000 7f082ade2700 1 -- 172.21.15.162:0/3057336088 shutdown_connections 2024-02-13T00:52:37.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.412+0000 7f082ade2700 1 -- 172.21.15.162:0/3057336088 wait complete. 2024-02-13T00:52:37.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082ade2700 1 librados: starting msgr at 2024-02-13T00:52:37.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082ade2700 1 librados: starting objecter 2024-02-13T00:52:37.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082ade2700 1 Processor -- start 2024-02-13T00:52:37.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082ade2700 1 -- start start 2024-02-13T00:52:37.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082ade2700 1 librados: setting wanted keys 2024-02-13T00:52:37.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082ade2700 1 librados: calling monclient init 2024-02-13T00:52:37.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082ade2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08240f8500 0x7f082410e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:37.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082ade2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08240f9b70 0x7f082410e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:37.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082ade2700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08240fa440 0x7f08241083e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:37.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082ade2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08241088d0 con 0x7f08240f9b70 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082ade2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0824108a40 con 0x7f08240f8500 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082ade2700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0824108c00 con 0x7f08240fa440 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082937f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08240fa440 0x7f08241083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f0823fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08240f9b70 0x7f082410e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f0823fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08240f9b70 0x7f082410e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:58640/0 (socket says 172.21.15.162:58640) 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f082937f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08240fa440 0x7f08241083e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48906/0 (socket says 172.21.15.162:48906) 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f0823fff700 1 -- 172.21.15.162:0/3312504713 learned_addr learned my addr 172.21.15.162:0/3312504713 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f0828b7e700 1 --2- 172.21.15.162:0/3312504713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08240f8500 0x7f082410e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.413+0000 7f0823fff700 1 -- 172.21.15.162:0/3312504713 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08240fa440 msgr2=0x7f08241083e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f0823fff700 1 --2- 172.21.15.162:0/3312504713 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08240fa440 0x7f08241083e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f0823fff700 1 -- 172.21.15.162:0/3312504713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08240f8500 msgr2=0x7f082410e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f0823fff700 1 --2- 172.21.15.162:0/3312504713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08240f8500 0x7f082410e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f0823fff700 1 -- 172.21.15.162:0/3312504713 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081c0097c0 con 0x7f08240f9b70 2024-02-13T00:52:37.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f082937f700 1 --2- 172.21.15.162:0/3312504713 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08240fa440 0x7f08241083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:52:37.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f0823fff700 1 --2- 172.21.15.162:0/3312504713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08240f9b70 0x7f082410e6e0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 rx=0x7f0810009eb0 tx=0x7f081000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:37.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f0821ffb700 10 client.?.objecter ms_handle_connect 0x7f08240f9b70 2024-02-13T00:52:37.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f0821ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:37.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f0821ffb700 1 -- 172.21.15.162:0/3312504713 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081000e070 con 0x7f08240f9b70 2024-02-13T00:52:37.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f082ade2700 1 -- 172.21.15.162:0/3312504713 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0810005040 con 0x7f08240f9b70 2024-02-13T00:52:37.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f0821ffb700 1 -- 172.21.15.162:0/3312504713 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0810013da0 con 0x7f08240f9b70 2024-02-13T00:52:37.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f082ade2700 10 client.28005.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:37.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f082ade2700 1 -- 172.21.15.162:0/3312504713 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0824109170 con 0x7f08240f9b70 2024-02-13T00:52:37.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f082ade2700 1 librados: init done 2024-02-13T00:52:37.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.414+0000 7f0821ffb700 1 -- 172.21.15.162:0/3312504713 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0810019070 con 0x7f08240f9b70 2024-02-13T00:52:37.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.415+0000 7f082ade2700 1 -- 172.21.15.162:0/3312504713 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08240475e0 con 0x7f08240f9b70 2024-02-13T00:52:37.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.415+0000 7f0821ffb700 1 -- 172.21.15.162:0/3312504713 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f081001e420 con 0x7f08240f9b70 2024-02-13T00:52:37.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.416+0000 7f0821ffb700 1 --2- 172.21.15.162:0/3312504713 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0814062c90 0x7f0814065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:37.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.416+0000 7f0821ffb700 1 -- 172.21.15.162:0/3312504713 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(729..729 src has 449..729) v4 ==== 8582+0+0 (secure 0 0 0) 0x7f081007df60 con 0x7f08240f9b70 2024-02-13T00:52:37.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.416+0000 7f0821ffb700 10 client.28005.objecter ms_dispatch 0x7f0824001550 osd_map(729..729 src has 449..729) v4 2024-02-13T00:52:37.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.416+0000 7f0821ffb700 3 client.28005.objecter handle_osd_map got epochs [729,729] > 0 2024-02-13T00:52:37.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.416+0000 7f0821ffb700 3 client.28005.objecter handle_osd_map decoding full epoch 729 2024-02-13T00:52:37.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.416+0000 7f0828b7e700 1 --2- 172.21.15.162:0/3312504713 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0814062c90 0x7f0814065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:37.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.416+0000 7f0821ffb700 20 client.28005.objecter dump_active .. 0 homeless 2024-02-13T00:52:37.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.416+0000 7f0821ffb700 10 client.28005.objecter ms_handle_connect 0x7f0814062c90 2024-02-13T00:52:37.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.416+0000 7f0828b7e700 1 --2- 172.21.15.162:0/3312504713 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0814062c90 0x7f0814065150 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 rx=0x7f0818004de0 tx=0x7f081800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:37.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.420+0000 7f0821ffb700 1 -- 172.21.15.162:0/3312504713 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0810051290 con 0x7f08240f9b70 2024-02-13T00:52:37.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:37.529 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:37.547 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:37.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:37.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:37.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:37.613+0000 7f082ade2700 1 -- 172.21.15.162:0/3312504713 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f0824109870 con 0x7f08240f9b70 2024-02-13T00:52:37.651 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:37.677 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:37.695 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:37.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:36.041324+0000 osd.0 (osd.0) 291 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:52:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:36.042218+0000 osd.0 (osd.0) 292 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:52:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:36.524063+0000 mgr.y (mgr.14544) 602 : cluster [DBG] pgmap v1099: 172 pgs: 2 peering, 170 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 977 KiB/s, 33 objects/s recovering 2024-02-13T00:52:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:36.574229+0000 mon.a (mon.0) 3503 : cluster [DBG] osdmap e729: 8 total, 8 up, 4 in 2024-02-13T00:52:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:36.584689+0000 mon.a (mon.0) 3504 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:37.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:37.829 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:37.907 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:37.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:36.041324+0000 osd.0 (osd.0) 291 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:52:37.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:36.042218+0000 osd.0 (osd.0) 292 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:52:37.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:36.524063+0000 mgr.y (mgr.14544) 602 : cluster [DBG] pgmap v1099: 172 pgs: 2 peering, 170 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 977 KiB/s, 33 objects/s recovering 2024-02-13T00:52:37.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:36.574229+0000 mon.a (mon.0) 3503 : cluster [DBG] osdmap e729: 8 total, 8 up, 4 in 2024-02-13T00:52:37.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:36.584689+0000 mon.a (mon.0) 3504 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:37.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:36.041324+0000 osd.0 (osd.0) 291 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:52:37.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:36.042218+0000 osd.0 (osd.0) 292 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:52:37.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:36.524063+0000 mgr.y (mgr.14544) 602 : cluster [DBG] pgmap v1099: 172 pgs: 2 peering, 170 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 977 KiB/s, 33 objects/s recovering 2024-02-13T00:52:37.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:36.574229+0000 mon.a (mon.0) 3503 : cluster [DBG] osdmap e729: 8 total, 8 up, 4 in 2024-02-13T00:52:37.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:36.584689+0000 mon.a (mon.0) 3504 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:37.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:37.913 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:37.916 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:37.927 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:52:37.928 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:52:38.011 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:38.038 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:38.057 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:38.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:38.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:38.186 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:38.205 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:38.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:38.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:52:38.334 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:38.352 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:52:38.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:38.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:52:38.456 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:38.485 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:38.596 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:52:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:52:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:52:38.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.599+0000 7f0821ffb700 1 -- 172.21.15.162:0/3312504713 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v731) v1 ==== 87+0+0 (secure 0 0 0) 0x7f0810023030 con 0x7f08240f9b70 2024-02-13T00:52:38.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 10 librados: watch_flush enter 2024-02-13T00:52:38.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 10 librados: watch_flush exit 2024-02-13T00:52:38.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 20 client.28005.objecter shutdown clearing up homeless session... 2024-02-13T00:52:38.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 10 client.28005.objecter successfully canceled tick 2024-02-13T00:52:38.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 1 -- 172.21.15.162:0/3312504713 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0814062c90 msgr2=0x7f0814065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:38.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 1 --2- 172.21.15.162:0/3312504713 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0814062c90 0x7f0814065150 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 rx=0x7f0818004de0 tx=0x7f081800b040).stop 2024-02-13T00:52:38.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 1 -- 172.21.15.162:0/3312504713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08240f9b70 msgr2=0x7f082410e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:38.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 1 --2- 172.21.15.162:0/3312504713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08240f9b70 0x7f082410e6e0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 rx=0x7f0810009eb0 tx=0x7f081000b040).stop 2024-02-13T00:52:38.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 1 -- 172.21.15.162:0/3312504713 shutdown_connections 2024-02-13T00:52:38.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 1 --2- 172.21.15.162:0/3312504713 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0814062c90 0x7f0814065150 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:38.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 1 --2- 172.21.15.162:0/3312504713 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08240fa440 0x7f08241083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:38.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 1 --2- 172.21.15.162:0/3312504713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08240f8500 0x7f082410e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:38.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 1 --2- 172.21.15.162:0/3312504713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08240f9b70 0x7f082410e6e0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:38.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.603+0000 7f082ade2700 1 -- 172.21.15.162:0/3312504713 >> 172.21.15.162:0/3312504713 conn(0x7f08240f38c0 msgr2=0x7f08240f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:38.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.604+0000 7f082ade2700 1 -- 172.21.15.162:0/3312504713 shutdown_connections 2024-02-13T00:52:38.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.604+0000 7f082ade2700 1 -- 172.21.15.162:0/3312504713 wait complete. 2024-02-13T00:52:38.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:38.604+0000 7f082ade2700 1 librados: shutdown 2024-02-13T00:52:38.611 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:52:38.775 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:37.574506+0000 mon.a (mon.0) 3505 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:38.776 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:37.574573+0000 mon.a (mon.0) 3506 : cluster [DBG] osdmap e730: 8 total, 8 up, 4 in 2024-02-13T00:52:38.776 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:37.587296+0000 mon.a (mon.0) 3507 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:38.776 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:37.615617+0000 mon.a (mon.0) 3508 : audit [INF] from='client.? 172.21.15.162:0/3312504713' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:52:38.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:37.574506+0000 mon.a (mon.0) 3505 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:38.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:37.574573+0000 mon.a (mon.0) 3506 : cluster [DBG] osdmap e730: 8 total, 8 up, 4 in 2024-02-13T00:52:38.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:37.587296+0000 mon.a (mon.0) 3507 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:38.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:37.615617+0000 mon.a (mon.0) 3508 : audit [INF] from='client.? 172.21.15.162:0/3312504713' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:52:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:37.574506+0000 mon.a (mon.0) 3505 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:37.574573+0000 mon.a (mon.0) 3506 : cluster [DBG] osdmap e730: 8 total, 8 up, 4 in 2024-02-13T00:52:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:37.587296+0000 mon.a (mon.0) 3507 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:37.615617+0000 mon.a (mon.0) 3508 : audit [INF] from='client.? 172.21.15.162:0/3312504713' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:52:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:38.233597+0000 osd.6 (osd.6) 232 : cluster [DBG] 28.0 scrub starts 2024-02-13T00:52:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:38.236651+0000 osd.6 (osd.6) 233 : cluster [DBG] 28.0 scrub ok 2024-02-13T00:52:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:38.524519+0000 mgr.y (mgr.14544) 603 : cluster [DBG] pgmap v1102: 204 pgs: 32 unknown, 2 peering, 170 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 977 KiB/s, 33 objects/s recovering 2024-02-13T00:52:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:38.584101+0000 mon.a (mon.0) 3509 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:52:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:38.598101+0000 mon.a (mon.0) 3510 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:38.598169+0000 mon.a (mon.0) 3511 : audit [INF] from='client.? 172.21.15.162:0/3312504713' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:52:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:38.598208+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e731: 8 total, 8 up, 4 in 2024-02-13T00:52:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:38.602066+0000 mon.a (mon.0) 3513 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-13T00:52:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:38.233597+0000 osd.6 (osd.6) 232 : cluster [DBG] 28.0 scrub starts 2024-02-13T00:52:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:38.236651+0000 osd.6 (osd.6) 233 : cluster [DBG] 28.0 scrub ok 2024-02-13T00:52:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:38.524519+0000 mgr.y (mgr.14544) 603 : cluster [DBG] pgmap v1102: 204 pgs: 32 unknown, 2 peering, 170 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 977 KiB/s, 33 objects/s recovering 2024-02-13T00:52:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:38.584101+0000 mon.a (mon.0) 3509 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:52:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:38.598101+0000 mon.a (mon.0) 3510 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:38.598169+0000 mon.a (mon.0) 3511 : audit [INF] from='client.? 172.21.15.162:0/3312504713' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:52:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:38.598208+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e731: 8 total, 8 up, 4 in 2024-02-13T00:52:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:38.602066+0000 mon.a (mon.0) 3513 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-13T00:52:40.049 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:38.233597+0000 osd.6 (osd.6) 232 : cluster [DBG] 28.0 scrub starts 2024-02-13T00:52:40.049 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:38.236651+0000 osd.6 (osd.6) 233 : cluster [DBG] 28.0 scrub ok 2024-02-13T00:52:40.049 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:38.524519+0000 mgr.y (mgr.14544) 603 : cluster [DBG] pgmap v1102: 204 pgs: 32 unknown, 2 peering, 170 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 977 KiB/s, 33 objects/s recovering 2024-02-13T00:52:40.049 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:38.584101+0000 mon.a (mon.0) 3509 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:52:40.049 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:38.598101+0000 mon.a (mon.0) 3510 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:52:40.049 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:38.598169+0000 mon.a (mon.0) 3511 : audit [INF] from='client.? 172.21.15.162:0/3312504713' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:52:40.049 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:38.598208+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e731: 8 total, 8 up, 4 in 2024-02-13T00:52:40.049 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:38.602066+0000 mon.a (mon.0) 3513 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-13T00:52:40.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-13T00:52:40.165 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:40.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:40.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:40.295 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:39.209416+0000 osd.6 (osd.6) 234 : cluster [DBG] 28.b scrub starts 2024-02-13T00:52:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:39.596586+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_count","val": "3"}]': finished 2024-02-13T00:52:40.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:39.596653+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e732: 8 total, 8 up, 4 in 2024-02-13T00:52:40.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:39.597573+0000 mon.a (mon.0) 3516 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-13T00:52:40.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:40.552242+0000 mon.a (mon.0) 3517 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_period","val": "3"}]': finished 2024-02-13T00:52:40.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:40.552318+0000 mon.a (mon.0) 3518 : cluster [DBG] osdmap e733: 8 total, 8 up, 4 in 2024-02-13T00:52:40.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:40.554624+0000 mon.a (mon.0) 3519 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:52:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:39.209416+0000 osd.6 (osd.6) 234 : cluster [DBG] 28.b scrub starts 2024-02-13T00:52:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:39.596586+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_count","val": "3"}]': finished 2024-02-13T00:52:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:39.596653+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e732: 8 total, 8 up, 4 in 2024-02-13T00:52:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:39.597573+0000 mon.a (mon.0) 3516 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-13T00:52:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:40.552242+0000 mon.a (mon.0) 3517 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_period","val": "3"}]': finished 2024-02-13T00:52:40.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:40.552318+0000 mon.a (mon.0) 3518 : cluster [DBG] osdmap e733: 8 total, 8 up, 4 in 2024-02-13T00:52:40.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:40.554624+0000 mon.a (mon.0) 3519 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:52:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:39.209416+0000 osd.6 (osd.6) 234 : cluster [DBG] 28.b scrub starts 2024-02-13T00:52:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:39.596586+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_count","val": "3"}]': finished 2024-02-13T00:52:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:39.596653+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e732: 8 total, 8 up, 4 in 2024-02-13T00:52:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:39.597573+0000 mon.a (mon.0) 3516 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-13T00:52:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:40.552242+0000 mon.a (mon.0) 3517 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_period","val": "3"}]': finished 2024-02-13T00:52:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:40.552318+0000 mon.a (mon.0) 3518 : cluster [DBG] osdmap e733: 8 total, 8 up, 4 in 2024-02-13T00:52:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:40.554624+0000 mon.a (mon.0) 3519 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:52:41.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:40.525346+0000 mgr.y (mgr.14544) 604 : cluster [DBG] pgmap v1105: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:41.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:40.814322+0000 osd.7 (osd.7) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:41.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:40.814496+0000 osd.7 (osd.7) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:41.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:41.555090+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:52:41.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:41.555140+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e734: 8 total, 8 up, 4 in 2024-02-13T00:52:41.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:41.556696+0000 mon.a (mon.0) 3522 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-13T00:52:41.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:40.525346+0000 mgr.y (mgr.14544) 604 : cluster [DBG] pgmap v1105: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:41.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:40.814322+0000 osd.7 (osd.7) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:41.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:40.814496+0000 osd.7 (osd.7) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:41.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:41.555090+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:52:41.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:41.555140+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e734: 8 total, 8 up, 4 in 2024-02-13T00:52:41.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:41.556696+0000 mon.a (mon.0) 3522 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-13T00:52:41.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.536+0000 7fc9da25b700 1 -- 172.21.15.162:0/2088496093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d40fdc00 msgr2=0x7fc9d40fdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:41.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.536+0000 7fc9da25b700 1 --2- 172.21.15.162:0/2088496093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d40fdc00 0x7fc9d40fdfe0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 rx=0x7fc9c4004870 tx=0x7fc9c4009cb0).stop 2024-02-13T00:52:41.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.536+0000 7fc9da25b700 1 -- 172.21.15.162:0/2088496093 shutdown_connections 2024-02-13T00:52:41.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.536+0000 7fc9da25b700 1 --2- 172.21.15.162:0/2088496093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9d40f8470 0x7fc9d4104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:41.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.536+0000 7fc9da25b700 1 --2- 172.21.15.162:0/2088496093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d40fdc00 0x7fc9d40fdfe0 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:41.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.536+0000 7fc9da25b700 1 --2- 172.21.15.162:0/2088496093 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9d40fe560 0x7fc9d40f7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:41.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.536+0000 7fc9da25b700 1 -- 172.21.15.162:0/2088496093 >> 172.21.15.162:0/2088496093 conn(0x7fc9d40f3850 msgr2=0x7fc9d40f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:41.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.536+0000 7fc9da25b700 1 -- 172.21.15.162:0/2088496093 shutdown_connections 2024-02-13T00:52:41.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.536+0000 7fc9da25b700 1 -- 172.21.15.162:0/2088496093 wait complete. 2024-02-13T00:52:41.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.536+0000 7fc9da25b700 1 librados: starting msgr at 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.536+0000 7fc9da25b700 1 librados: starting objecter 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9da25b700 1 Processor -- start 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9da25b700 1 -- start start 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9da25b700 1 librados: setting wanted keys 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9da25b700 1 librados: calling monclient init 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9da25b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d40f8470 0x7fc9d410e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9da25b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9d40fdc00 0x7fc9d410e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9da25b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9d40fe560 0x7fc9d41082d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9da25b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d41087c0 con 0x7fc9d40f8470 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9da25b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d4108930 con 0x7fc9d40fe560 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9da25b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d4108af0 con 0x7fc9d40fdc00 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9caffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9d40fdc00 0x7fc9d410e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:41.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9d37fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d40f8470 0x7fc9d410e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:41.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9caffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9d40fdc00 0x7fc9d410e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48928/0 (socket says 172.21.15.162:48928) 2024-02-13T00:52:41.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9d37fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d40f8470 0x7fc9d410e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:58656/0 (socket says 172.21.15.162:58656) 2024-02-13T00:52:41.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9caffd700 1 -- 172.21.15.162:0/3932743531 learned_addr learned my addr 172.21.15.162:0/3932743531 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:41.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9d3fff700 1 --2- 172.21.15.162:0/3932743531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9d40fe560 0x7fc9d41082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:41.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9d37fe700 1 -- 172.21.15.162:0/3932743531 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9d40fdc00 msgr2=0x7fc9d410e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:41.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.537+0000 7fc9d37fe700 1 --2- 172.21.15.162:0/3932743531 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9d40fdc00 0x7fc9d410e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:41.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9d37fe700 1 -- 172.21.15.162:0/3932743531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9d40fe560 msgr2=0x7fc9d41082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:41.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9d37fe700 1 --2- 172.21.15.162:0/3932743531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9d40fe560 0x7fc9d41082d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:41.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9d37fe700 1 -- 172.21.15.162:0/3932743531 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc005040 con 0x7fc9d40f8470 2024-02-13T00:52:41.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9d37fe700 1 --2- 172.21.15.162:0/3932743531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d40f8470 0x7fc9d410e130 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7fc9c4004870 tx=0x7fc9c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:41.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9d17fa700 10 client.?.objecter ms_handle_connect 0x7fc9d40f8470 2024-02-13T00:52:41.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9d17fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:41.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9d17fa700 1 -- 172.21.15.162:0/3932743531 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c400e070 con 0x7fc9d40f8470 2024-02-13T00:52:41.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9d17fa700 1 -- 172.21.15.162:0/3932743531 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9c4013da0 con 0x7fc9d40f8470 2024-02-13T00:52:41.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9da25b700 1 -- 172.21.15.162:0/3932743531 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c4005040 con 0x7fc9d40f8470 2024-02-13T00:52:41.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9d17fa700 1 -- 172.21.15.162:0/3932743531 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4019070 con 0x7fc9d40f8470 2024-02-13T00:52:41.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9da25b700 10 client.28011.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:41.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9da25b700 1 -- 172.21.15.162:0/3932743531 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d4109060 con 0x7fc9d40f8470 2024-02-13T00:52:41.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.538+0000 7fc9da25b700 1 librados: init done 2024-02-13T00:52:41.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.539+0000 7fc9d17fa700 1 -- 172.21.15.162:0/3932743531 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc9c401e420 con 0x7fc9d40f8470 2024-02-13T00:52:41.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.539+0000 7fc9da25b700 1 -- 172.21.15.162:0/3932743531 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d4060380 con 0x7fc9d40f8470 2024-02-13T00:52:41.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.540+0000 7fc9d17fa700 1 --2- 172.21.15.162:0/3932743531 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc9c0062c90 0x7fc9c0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:41.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.540+0000 7fc9d17fa700 1 -- 172.21.15.162:0/3932743531 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(733..733 src has 449..733) v4 ==== 8590+0+0 (secure 0 0 0) 0x7fc9c407dc60 con 0x7fc9d40f8470 2024-02-13T00:52:41.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.540+0000 7fc9d17fa700 10 client.28011.objecter ms_dispatch 0x7fc9d4001550 osd_map(733..733 src has 449..733) v4 2024-02-13T00:52:41.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.540+0000 7fc9d17fa700 3 client.28011.objecter handle_osd_map got epochs [733,733] > 0 2024-02-13T00:52:41.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.540+0000 7fc9d17fa700 3 client.28011.objecter handle_osd_map decoding full epoch 733 2024-02-13T00:52:41.885 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:52:41.885 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:52:41.885 INFO:teuthology.orchestra.run.smithi162.stdout: "epoch": 734, 2024-02-13T00:52:41.885 INFO:teuthology.orchestra.run.smithi162.stdout: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:52:41.885 INFO:teuthology.orchestra.run.smithi162.stdout: "created": "2024-02-13T00:31:51.797017+0000", 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: "modified": "2024-02-13T00:52:41.549801+0000", 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: "last_up_change": "2024-02-13T00:48:03.474219+0000", 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: "last_in_change": "2024-02-13T00:52:29.558263+0000", 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_num": 5801984, 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_set": [ 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: "noscrub", 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: "pglog_hardlimit", 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: "purged_snapdirs", 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_deletes", 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: "sortbitwise" 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:52:41.886 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_version": 109, 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "full_ratio": 0.94999998807907104, 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_snapshot": "", 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_max": 249, 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "max_osd": 8, 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "require_min_compat_client": "luminous", 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "min_compat_client": "luminous", 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "require_osd_release": "pacific", 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "pools": [ 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 1, 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "device_health_metrics", 2024-02-13T00:52:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:35:34.545390+0000", 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:52:41.888 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 1, 2024-02-13T00:52:41.889 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 1, 2024-02-13T00:52:41.889 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 1, 2024-02-13T00:52:41.891 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 1, 2024-02-13T00:52:41.891 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 1, 2024-02-13T00:52:41.891 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:52:41.891 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:52:41.891 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:52:41.891 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:52:41.891 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:52:41.891 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:52:41.891 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "71", 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:52:41.892 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:52:41.893 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:52:41.894 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:52:41.894 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:52:41.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.894 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:52:41.894 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:52:41.894 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:52:41.894 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:52:41.894 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:52:41.894 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_max": 32, 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_min": 1 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_devicehealth": {} 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 2, 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": ".rgw.root", 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.459980+0000", 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:52:41.896 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:52:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "58", 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:52:41.898 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:52:41.899 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:52:41.900 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 3, 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "iscsi", 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.740219+0000", 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:52:41.901 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 3, 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 3, 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 3, 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 3, 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 3, 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:52:41.902 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "102", 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:52:41.903 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:52:41.904 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:52:41.904 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:52:41.904 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:52:41.904 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:52:41.904 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:52:41.904 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:52:41.904 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:52:41.904 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:52:41.904 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:52:41.904 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:52:41.904 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:52:41.905 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:52:41.905 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:52:41.905 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:52:41.905 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:52:41.905 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:52:41.905 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:52:41.905 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:52:41.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.905 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:52:41.905 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd": {} 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 4, 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.log", 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:41.489292+0000", 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:52:41.907 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:52:41.908 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "60", 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:52:41.909 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:52:41.910 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:52:41.911 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 5, 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.control", 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:43.466304+0000", 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:52:41.912 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 22, 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 22, 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 22, 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 22, 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 22, 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "5.16", 2024-02-13T00:52:41.913 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 544, 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 543, 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 543, 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "544", 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "544", 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "542", 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:52:41.914 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:52:41.915 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:52:41.915 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:52:41.915 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:52:41.915 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:52:41.915 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:52:41.915 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:52:41.915 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:52:41.915 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:52:41.915 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:52:41.915 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:52:41.916 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.917 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 6, 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.meta", 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:45.542442+0000", 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:52:41.918 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 42, 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 42, 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 42, 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 42, 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 42, 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:52:41.919 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "248", 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "246", 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:52:41.920 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:52:41.921 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:52:41.922 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:52:41.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.540+0000 7fc9caffd700 1 --2- 172.21.15.162:0/3932743531 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc9c0062c90 0x7fc9c0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:41.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.540+0000 7fc9d17fa700 20 client.28011.objecter dump_active .. 0 homeless 2024-02-13T00:52:41.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.540+0000 7fc9caffd700 1 --2- 172.21.15.162:0/3932743531 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc9c0062c90 0x7fc9c0065150 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 rx=0x7fc9bc007fd0 tx=0x7fc9bc00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:41.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.540+0000 7fc9d17fa700 10 client.28011.objecter ms_handle_connect 0x7fc9c0062c90 2024-02-13T00:52:41.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.544+0000 7fc9d17fa700 1 -- 172.21.15.162:0/3932743531 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc9c4051000 con 0x7fc9d40f8470 2024-02-13T00:52:41.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.722+0000 7fc9da25b700 1 -- 172.21.15.162:0/3932743531 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fc9d41097b0 con 0x7fc9d40f8470 2024-02-13T00:52:41.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.724+0000 7fc9d17fa700 1 -- 172.21.15.162:0/3932743531 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v734) v1 ==== 81+0+49461 (secure 0 0 0) 0x7fc9c4050bf0 con 0x7fc9d40f8470 2024-02-13T00:52:41.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.725+0000 7fc9da25b700 10 librados: watch_flush enter 2024-02-13T00:52:41.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.725+0000 7fc9da25b700 10 librados: watch_flush exit 2024-02-13T00:52:41.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.725+0000 7fc9da25b700 20 client.28011.objecter shutdown clearing up homeless session... 2024-02-13T00:52:41.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.725+0000 7fc9da25b700 10 client.28011.objecter successfully canceled tick 2024-02-13T00:52:41.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 -- 172.21.15.162:0/3932743531 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc9c0062c90 msgr2=0x7fc9c0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:41.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 --2- 172.21.15.162:0/3932743531 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc9c0062c90 0x7fc9c0065150 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 rx=0x7fc9bc007fd0 tx=0x7fc9bc00b040).stop 2024-02-13T00:52:41.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 -- 172.21.15.162:0/3932743531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d40f8470 msgr2=0x7fc9d410e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:41.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 --2- 172.21.15.162:0/3932743531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d40f8470 0x7fc9d410e130 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7fc9c4004870 tx=0x7fc9c400b040).stop 2024-02-13T00:52:41.924 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_bias": 4 2024-02-13T00:52:41.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.925 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:52:41.925 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:52:41.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.925 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 28, 2024-02-13T00:52:41.925 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-13T00:52:41.925 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:00.190777+0000", 2024-02-13T00:52:41.925 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:52:41.925 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:52:41.925 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:52:41.925 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:52:41.926 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:52:41.926 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:52:41.926 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:52:41.926 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:52:41.926 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:52:41.926 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:52:41.926 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:52:41.926 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:52:41.926 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:52:41.926 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:52:41.926 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "108", 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:52:41.927 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:52:41.928 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:52:41.928 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:52:41.928 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:52:41.928 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:52:41.928 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:52:41.928 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:52:41.928 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:52:41.928 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:52:41.929 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:52:41.929 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:52:41.929 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:52:41.929 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:52:41.929 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:52:41.929 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:52:41.929 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:52:41.930 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:52:41.930 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:52:41.930 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:52:41.930 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:52:41.930 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:52:41.930 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:52:41.931 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:52:41.931 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:52:41.931 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:52:41.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.931 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:52:41.931 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:52:41.931 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:52:41.932 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:52:41.932 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:52:41.932 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:52:41.932 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:52:41.932 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:52:41.932 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:52:41.933 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:52:41.933 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:52:41.933 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:52:41.933 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:52:41.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.933 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 234, 2024-02-13T00:52:41.934 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-101", 2024-02-13T00:52:41.934 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:50:12.777962+0000", 2024-02-13T00:52:41.934 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8193, 2024-02-13T00:52:41.934 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-13T00:52:41.934 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 3, 2024-02-13T00:52:41.934 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:52:41.934 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:52:41.935 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 1, 2024-02-13T00:52:41.935 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:52:41.935 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:52:41.935 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:52:41.935 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:52:41.935 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:52:41.935 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:52:41.935 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 8, 2024-02-13T00:52:41.936 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 8, 2024-02-13T00:52:41.936 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 8, 2024-02-13T00:52:41.936 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 8, 2024-02-13T00:52:41.936 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 8, 2024-02-13T00:52:41.936 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:52:41.936 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:52:41.936 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:52:41.937 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:52:41.937 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:52:41.937 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:52:41.937 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:52:41.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.937 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "731", 2024-02-13T00:52:41.938 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "717", 2024-02-13T00:52:41.938 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "717", 2024-02-13T00:52:41.938 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "717", 2024-02-13T00:52:41.938 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:52:41.938 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:52:41.938 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 10, 2024-02-13T00:52:41.938 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 731, 2024-02-13T00:52:41.938 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:52:41.938 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:52:41.939 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:52:41.939 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:52:41.939 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [ 2024-02-13T00:52:41.939 INFO:teuthology.orchestra.run.smithi162.stdout: 249 2024-02-13T00:52:41.939 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:52:41.939 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:52:41.939 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:52:41.939 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:52:41.939 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:52:41.939 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:52:41.939 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:52:41.939 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi162-84057-101", 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:52:41.940 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 8192, 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 249, 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-130", 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:52:35.828383+0000", 2024-02-13T00:52:41.941 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8193, 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:52:41.942 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "734", 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:52:41.943 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:52:41.944 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:52:41.944 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:52:41.944 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:52:41.944 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 10, 2024-02-13T00:52:41.944 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 731, 2024-02-13T00:52:41.944 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:52:41.944 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:52:41.944 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:52:41.944 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:52:41.944 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:52:41.944 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": 234, 2024-02-13T00:52:41.944 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "bloom", 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "false_positive_probability": 0.050000000000000003, 2024-02-13T00:52:41.945 INFO:teuthology.orchestra.run.smithi162.stdout: "target_size": 0, 2024-02-13T00:52:41.946 INFO:teuthology.orchestra.run.smithi162.stdout: "seed": 0 2024-02-13T00:52:41.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.946 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 3, 2024-02-13T00:52:41.946 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 3, 2024-02-13T00:52:41.946 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:52:41.946 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:52:41.946 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:52:41.946 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:52:41.946 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:52:41.946 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [ 2024-02-13T00:52:41.946 INFO:teuthology.orchestra.run.smithi162.stdout: 1000000, 2024-02-13T00:52:41.947 INFO:teuthology.orchestra.run.smithi162.stdout: 1000000, 2024-02-13T00:52:41.947 INFO:teuthology.orchestra.run.smithi162.stdout: 1000000 2024-02-13T00:52:41.947 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:52:41.947 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:52:41.947 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:52:41.947 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:52:41.947 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:52:41.947 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:52:41.947 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.947 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:52:41.947 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:52:41.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.948 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:52:41.948 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99", 2024-02-13T00:52:41.948 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:52:41.948 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:52:41.948 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:52:41.948 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:52:41.948 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:52:41.948 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:52:41.948 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 8, 2024-02-13T00:52:41.948 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 729, 2024-02-13T00:52:41.948 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:52:41.949 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:52:41.949 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:52:41.949 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.949 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.949 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6801", 2024-02-13T00:52:41.949 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:52:41.949 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.949 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.949 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:52:41.950 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.950 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.950 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6802", 2024-02-13T00:52:41.950 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:52:41.950 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.950 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6804", 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.952 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.952 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6803", 2024-02-13T00:52:41.952 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:52:41.952 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.952 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.952 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6801/2103743338", 2024-02-13T00:52:41.952 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6802/2103743338", 2024-02-13T00:52:41.952 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6804/2103743338", 2024-02-13T00:52:41.952 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6803/2103743338", 2024-02-13T00:52:41.952 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:52:41.952 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886", 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 13, 2024-02-13T00:52:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 729, 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6805", 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6806", 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6808", 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.956 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:52:41.956 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.956 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.956 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6807", 2024-02-13T00:52:41.956 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:52:41.956 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.956 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.956 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6805/150599780", 2024-02-13T00:52:41.956 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6806/150599780", 2024-02-13T00:52:41.956 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6808/150599780", 2024-02-13T00:52:41.957 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6807/150599780", 2024-02-13T00:52:41.957 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:52:41.957 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:52:41.957 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:52:41.957 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.957 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:52:41.957 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e", 2024-02-13T00:52:41.957 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 18, 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 333, 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.958 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6809", 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6810", 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.959 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6812", 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.960 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6811", 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6809/3689719376", 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6810/3689719376", 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6812/3689719376", 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6811/3689719376", 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001", 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 25, 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 255, 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:52:41.962 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6813", 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.963 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6814", 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6816", 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6815", 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6813/604971011", 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6814/604971011", 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6816/604971011", 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6815/604971011", 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:52:41.965 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3", 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 31, 2024-02-13T00:52:41.966 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 81, 2024-02-13T00:52:41.967 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:52:41.967 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:52:41.967 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:52:41.967 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.967 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.967 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6800", 2024-02-13T00:52:41.967 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:52:41.967 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.967 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.967 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:52:41.968 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.968 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.968 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6801", 2024-02-13T00:52:41.968 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:52:41.968 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.968 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 -- 172.21.15.162:0/3932743531 shutdown_connections 2024-02-13T00:52:41.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 --2- 172.21.15.162:0/3932743531 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc9c0062c90 0x7fc9c0065150 unknown :-1 s=CLOSED pgs=3015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:41.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 --2- 172.21.15.162:0/3932743531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9d40fe560 0x7fc9d41082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:41.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 --2- 172.21.15.162:0/3932743531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d40f8470 0x7fc9d410e130 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:41.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 --2- 172.21.15.162:0/3932743531 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9d40fdc00 0x7fc9d410e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:41.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 -- 172.21.15.162:0/3932743531 >> 172.21.15.162:0/3932743531 conn(0x7fc9d40f3850 msgr2=0x7fc9d40fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:41.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 -- 172.21.15.162:0/3932743531 shutdown_connections 2024-02-13T00:52:41.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 -- 172.21.15.162:0/3932743531 wait complete. 2024-02-13T00:52:41.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:41.726+0000 7fc9da25b700 1 librados: shutdown 2024-02-13T00:52:41.970 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:52:41.970 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.970 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.970 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6803", 2024-02-13T00:52:41.970 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:52:41.970 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.970 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.970 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:52:41.970 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6802", 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6800/3344263388", 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6801/3344263388", 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6803/3344263388", 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6802/3344263388", 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.972 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:52:41.972 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723", 2024-02-13T00:52:41.972 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:52:41.972 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:52:41.972 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:52:41.972 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:52:41.972 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 39, 2024-02-13T00:52:41.972 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 436, 2024-02-13T00:52:41.972 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 479, 2024-02-13T00:52:41.972 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 399, 2024-02-13T00:52:41.972 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 437, 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6804", 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6805", 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6807", 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.974 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6806", 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6804/2277917372", 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6805/2277917372", 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2277917372", 2024-02-13T00:52:41.975 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2277917372", 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474", 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:52:41.976 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 45, 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 729, 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6808", 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.977 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.978 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:52:41.978 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.978 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.978 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6809", 2024-02-13T00:52:41.978 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:52:41.978 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.978 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.979 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:52:41.979 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.979 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.979 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6811", 2024-02-13T00:52:41.979 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:52:41.979 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.979 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.979 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:52:41.979 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6810", 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6808/1506640341", 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6809/1506640341", 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1506640341", 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1506640341", 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:52:41.980 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b", 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 53, 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 215, 2024-02-13T00:52:41.981 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6812", 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.982 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6813", 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6815", 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.983 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6814", 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6812/1964192772", 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6813/1964192772", 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6815/1964192772", 2024-02-13T00:52:41.984 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6814/1964192772", 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_xinfo": [ 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:52:41.985 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:52:11.251373+0000", 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:52:16.074157+0000", 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:52:12.219850+0000", 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:52:41.987 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:52:41.988 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:52:41.988 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:52:34.637591+0000", 2024-02-13T00:52:41.988 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:52:41.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.988 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:52:41.988 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:52:41.988 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:52:41.988 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:52:16.782540+0000", 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:51:58.951251+0000", 2024-02-13T00:52:41.989 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 436 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:51:57.475514+0000", 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:52:41.990 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:52:40.814498+0000", 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap": [ 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.b", 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: 3, 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: 6, 2024-02-13T00:52:41.991 INFO:teuthology.orchestra.run.smithi162.stdout: 5 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "6.23", 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: 3, 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: 6, 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: 7 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap_items": [ 2024-02-13T00:52:41.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.993 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "2.f", 2024-02-13T00:52:41.993 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:52:41.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.993 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 7, 2024-02-13T00:52:41.993 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 6 2024-02-13T00:52:41.993 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.993 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.993 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "2.1a", 2024-02-13T00:52:41.993 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:52:41.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 1 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "5.5", 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 6 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.994 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "6.6", 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 0 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_temp": [], 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_temp": [], 2024-02-13T00:52:41.995 INFO:teuthology.orchestra.run.smithi162.stdout: "blocklist": { 2024-02-13T00:52:41.996 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/1909610488": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:52:41.996 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2370739667": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:52:41.996 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/1502590187": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:52:41.996 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/4121673848": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:52:41.996 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2412382714": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:52:41.996 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3041562701": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:52:41.996 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/2775563314": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:52:41.996 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3135632190": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:52:41.996 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/197307636": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:52:41.997 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2653247729": "2024-02-14T00:32:24.380625+0000" 2024-02-13T00:52:41.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.997 INFO:teuthology.orchestra.run.smithi162.stdout: "range_blocklist": {}, 2024-02-13T00:52:41.997 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profiles": { 2024-02-13T00:52:41.997 INFO:teuthology.orchestra.run.smithi162.stdout: "default": { 2024-02-13T00:52:41.997 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:52:41.997 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:52:41.997 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:52:41.997 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:52:41.997 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:52:41.997 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van" 2024-02-13T00:52:41.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.998 INFO:teuthology.orchestra.run.smithi162.stdout: "testprofile-test-rados-api-smithi162-84057-101": { 2024-02-13T00:52:41.998 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-device-class": "", 2024-02-13T00:52:41.998 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:52:41.998 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-root": "default", 2024-02-13T00:52:41.998 INFO:teuthology.orchestra.run.smithi162.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-13T00:52:41.998 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:52:41.998 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:52:41.998 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:52:41.998 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:52:41.998 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van", 2024-02-13T00:52:41.998 INFO:teuthology.orchestra.run.smithi162.stdout: "w": "8" 2024-02-13T00:52:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps_queue": [ 2024-02-13T00:52:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 24, 2024-02-13T00:52:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:52:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:41.999 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 9, 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 25, 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.000 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 60, 2024-02-13T00:52:42.001 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 78, 2024-02-13T00:52:42.002 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.003 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 85, 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 94, 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 95, 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 4, 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 4 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 107, 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:52:42.006 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 113, 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 125, 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.009 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:52:42.009 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:52:42.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.009 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:52:42.009 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:52:42.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.009 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 10, 2024-02-13T00:52:42.009 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 133, 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 12, 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 154, 2024-02-13T00:52:42.010 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 14, 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 166, 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 17, 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 176, 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 20, 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 237, 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 238, 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 4, 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.014 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 241, 2024-02-13T00:52:42.014 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.014 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:52:42.014 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.014 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:52:42.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.014 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 245, 2024-02-13T00:52:42.014 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:52:42.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:52:42.015 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 9, 2024-02-13T00:52:42.015 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:52:42.015 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.015 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:52:42.015 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.015 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:52:42.015 INFO:teuthology.orchestra.run.smithi162.stdout: "new_removed_snaps": [], 2024-02-13T00:52:42.015 INFO:teuthology.orchestra.run.smithi162.stdout: "new_purged_snaps": [], 2024-02-13T00:52:42.015 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_node_flags": {}, 2024-02-13T00:52:42.015 INFO:teuthology.orchestra.run.smithi162.stdout: "device_class_flags": {}, 2024-02-13T00:52:42.015 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode": { 2024-02-13T00:52:42.015 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_enabled": false, 2024-02-13T00:52:42.016 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_bucket_count": 0, 2024-02-13T00:52:42.016 INFO:teuthology.orchestra.run.smithi162.stdout: "degraded_stretch_mode": 0, 2024-02-13T00:52:42.016 INFO:teuthology.orchestra.run.smithi162.stdout: "recovering_stretch_mode": 0, 2024-02-13T00:52:42.016 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_bucket": 0 2024-02-13T00:52:42.016 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:52:42.016 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:52:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:40.525346+0000 mgr.y (mgr.14544) 604 : cluster [DBG] pgmap v1105: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:40.814322+0000 osd.7 (osd.7) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:52:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:40.814496+0000 osd.7 (osd.7) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:52:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:41.555090+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:52:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:41.555140+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e734: 8 total, 8 up, 4 in 2024-02-13T00:52:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:41.556696+0000 mon.a (mon.0) 3522 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-13T00:52:42.125 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:42.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:42.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:42.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:42.260 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:42.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:41.723807+0000 mon.a (mon.0) 3523 : audit [DBG] from='client.? 172.21.15.162:0/3932743531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:52:42.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:42.555071+0000 mon.a (mon.0) 3524 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-13T00:52:42.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:42.555135+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e735: 8 total, 8 up, 4 in 2024-02-13T00:52:42.991 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:41.723807+0000 mon.a (mon.0) 3523 : audit [DBG] from='client.? 172.21.15.162:0/3932743531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:52:42.991 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:42.555071+0000 mon.a (mon.0) 3524 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-13T00:52:42.991 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:42.555135+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e735: 8 total, 8 up, 4 in 2024-02-13T00:52:43.011 INFO:tasks.thrashosds.thrasher:j is {'epoch': 734, 'fsid': 'd4ee01fc-ca06-11ee-95b9-87774f69a715', 'created': '2024-02-13T00:31:51.797017+0000', 'modified': '2024-02-13T00:52:41.549801+0000', 'last_up_change': '2024-02-13T00:48:03.474219+0000', 'last_in_change': '2024-02-13T00:52:29.558263+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 109, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 249, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-13T00:35:34.545390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-13T00:37:39.459980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-13T00:37:39.740219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-13T00:37:41.489292+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-13T00:37:43.466304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 544, 'last_epoch_started': 543, 'last_epoch_clean': 543, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '544', 'last_force_op_resend_preluminous': '542', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-13T00:37:45.542442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '246', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 28, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-13T00:41:00.190777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 234, 'pool_name': 'test-rados-api-smithi162-84057-101', 'create_time': '2024-02-13T00:50:12.777962+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '731', 'last_force_op_resend': '717', 'last_force_op_resend_prenautilus': '717', 'last_force_op_resend_preluminous': '717', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 10, 'snap_epoch': 731, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [249], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi162-84057-101', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 249, 'pool_name': 'test-rados-api-smithi162-84057-130', 'create_time': '2024-02-13T00:52:35.828383+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '734', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 10, 'snap_epoch': 731, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 234, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 3, 'hit_set_count': 3, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 729, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6801', 'nonce': 2103743338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 2103743338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 2103743338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6803', 'nonce': 2103743338}]}, 'public_addr': '172.21.15.162:6801/2103743338', 'cluster_addr': '172.21.15.162:6802/2103743338', 'heartbeat_back_addr': '172.21.15.162:6804/2103743338', 'heartbeat_front_addr': '172.21.15.162:6803/2103743338', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '39e9d2ef-9aa4-45bb-8a47-b0e0ec222886', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 729, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6805', 'nonce': 150599780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 150599780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 150599780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6807', 'nonce': 150599780}]}, 'public_addr': '172.21.15.162:6805/150599780', 'cluster_addr': '172.21.15.162:6806/150599780', 'heartbeat_back_addr': '172.21.15.162:6808/150599780', 'heartbeat_front_addr': '172.21.15.162:6807/150599780', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 333, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6809', 'nonce': 3689719376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3689719376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3689719376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6811', 'nonce': 3689719376}]}, 'public_addr': '172.21.15.162:6809/3689719376', 'cluster_addr': '172.21.15.162:6810/3689719376', 'heartbeat_back_addr': '172.21.15.162:6812/3689719376', 'heartbeat_front_addr': '172.21.15.162:6811/3689719376', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e6b9f0e-7a18-4cf8-9331-1a973fadc001', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 255, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6813', 'nonce': 604971011}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 604971011}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 604971011}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6815', 'nonce': 604971011}]}, 'public_addr': '172.21.15.162:6813/604971011', 'cluster_addr': '172.21.15.162:6814/604971011', 'heartbeat_back_addr': '172.21.15.162:6816/604971011', 'heartbeat_front_addr': '172.21.15.162:6815/604971011', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 3344263388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 3344263388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 3344263388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 3344263388}]}, 'public_addr': '172.21.15.181:6800/3344263388', 'cluster_addr': '172.21.15.181:6801/3344263388', 'heartbeat_back_addr': '172.21.15.181:6803/3344263388', 'heartbeat_front_addr': '172.21.15.181:6802/3344263388', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6aa2bd68-7f4b-4973-ac49-7fbc57c1a723', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 436, 'up_from': 479, 'up_thru': 399, 'down_at': 437, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2277917372}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2277917372}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2277917372}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2277917372}]}, 'public_addr': '172.21.15.181:6804/2277917372', 'cluster_addr': '172.21.15.181:6805/2277917372', 'heartbeat_back_addr': '172.21.15.181:6807/2277917372', 'heartbeat_front_addr': '172.21.15.181:6806/2277917372', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '791ba814-5284-432e-addc-d6b7f5f3d474', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 729, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1506640341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1506640341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1506640341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1506640341}]}, 'public_addr': '172.21.15.181:6808/1506640341', 'cluster_addr': '172.21.15.181:6809/1506640341', 'heartbeat_back_addr': '172.21.15.181:6811/1506640341', 'heartbeat_front_addr': '172.21.15.181:6810/1506640341', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f369c65-e663-4eb2-87a5-f2900f15b35b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 215, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 1964192772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 1964192772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 1964192772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 1964192772}]}, 'public_addr': '172.21.15.181:6812/1964192772', 'cluster_addr': '172.21.15.181:6813/1964192772', 'heartbeat_back_addr': '172.21.15.181:6815/1964192772', 'heartbeat_front_addr': '172.21.15.181:6814/1964192772', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:52:11.251373+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:52:16.074157+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:52:12.219850+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:52:34.637591+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:52:16.782540+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:51:58.951251+0000', 'dead_epoch': 436}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:51:57.475514+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:52:40.814498+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.b', 'osds': [3, 6, 5]}, {'pgid': '6.23', 'osds': [3, 6, 7]}], 'pg_upmap_items': [{'pgid': '2.f', 'mappings': [{'from': 7, 'to': 6}]}, {'pgid': '2.1a', 'mappings': [{'from': 6, 'to': 1}]}, {'pgid': '5.5', 'mappings': [{'from': 0, 'to': 6}]}, {'pgid': '6.6', 'mappings': [{'from': 6, 'to': 0}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.162:6800/1909610488': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2370739667': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/1502590187': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/4121673848': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:0/2412382714': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/3041562701': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/2775563314': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/3135632190': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/197307636': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2653247729': '2024-02-14T00:32:24.380625+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi162-84057-101': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 24, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 60, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 78, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 94, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 113, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 166, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}, {'pool': 237, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 238, 'snaps': [{'begin': 4, 'length': 1}]}, {'pool': 241, 'snaps': [{'begin': 6, 'length': 1}]}, {'pool': 245, 'snaps': [{'begin': 9, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-13T00:52:43.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:52:43.011 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:43.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:43.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:41.723807+0000 mon.a (mon.0) 3523 : audit [DBG] from='client.? 172.21.15.162:0/3932743531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:52:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:42.555071+0000 mon.a (mon.0) 3524 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-130","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-13T00:52:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:42.555135+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e735: 8 total, 8 up, 4 in 2024-02-13T00:52:43.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:43.140 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:43.470 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:43.474 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:43.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:43.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:52:43.613 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:43.634 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:52:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:43.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:52:43.717 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:42.525981+0000 mgr.y (mgr.14544) 605 : cluster [DBG] pgmap v1108: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:43.717 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:42.525981+0000 mgr.y (mgr.14544) 605 : cluster [DBG] pgmap v1108: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:43.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:43.767 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:43.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:52:43.787 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:43.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:43.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:43.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:43.918 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:44.022 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:44.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:44.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:42.525981+0000 mgr.y (mgr.14544) 605 : cluster [DBG] pgmap v1108: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:52:44.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:44.156 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:44.176 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:44.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:44.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:44.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.279+0000 7f4ef678a700 1 -- 172.21.15.162:0/2110984823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ef00f73a0 msgr2=0x7f4ef01039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.279+0000 7f4ef678a700 1 --2- 172.21.15.162:0/2110984823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ef00f73a0 0x7f4ef01039a0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 rx=0x7f4ee8009be0 tx=0x7f4ee8004d10).stop 2024-02-13T00:52:44.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.279+0000 7f4ef678a700 1 -- 172.21.15.162:0/2110984823 shutdown_connections 2024-02-13T00:52:44.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.279+0000 7f4ef678a700 1 --2- 172.21.15.162:0/2110984823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ef00f73a0 0x7f4ef01039a0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.279+0000 7f4ef678a700 1 --2- 172.21.15.162:0/2110984823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ef00fcb30 0x7f4ef00fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.279+0000 7f4ef678a700 1 --2- 172.21.15.162:0/2110984823 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ef00fd490 0x7f4ef00f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.279+0000 7f4ef678a700 1 -- 172.21.15.162:0/2110984823 >> 172.21.15.162:0/2110984823 conn(0x7f4ef006eb90 msgr2=0x7f4ef00f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.279+0000 7f4ef678a700 1 -- 172.21.15.162:0/2110984823 shutdown_connections 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.279+0000 7f4ef678a700 1 -- 172.21.15.162:0/2110984823 wait complete. 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.280+0000 7f4ef678a700 1 librados: starting msgr at 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.280+0000 7f4ef678a700 1 librados: starting objecter 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.280+0000 7f4ef678a700 1 Processor -- start 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.280+0000 7f4ef678a700 1 -- start start 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.280+0000 7f4ef678a700 1 librados: setting wanted keys 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.280+0000 7f4ef678a700 1 librados: calling monclient init 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.280+0000 7f4ef678a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ef00f73a0 0x7f4ef01102c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.280+0000 7f4ef678a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ef00fcb30 0x7f4ef01107b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.280+0000 7f4ef678a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ef00fd490 0x7f4ef010a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.280+0000 7f4ef678a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef010a950 con 0x7f4ef00f73a0 2024-02-13T00:52:44.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.280+0000 7f4ef678a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef010aac0 con 0x7f4ef00fcb30 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.280+0000 7f4ef678a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef010ac80 con 0x7f4ef00fd490 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef4d27700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ef00fd490 0x7f4ef010a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4eef7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ef00fcb30 0x7f4ef01107b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef4d27700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ef00fd490 0x7f4ef010a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48946/0 (socket says 172.21.15.162:48946) 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef4d27700 1 -- 172.21.15.162:0/91274364 learned_addr learned my addr 172.21.15.162:0/91274364 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef4d27700 1 -- 172.21.15.162:0/91274364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ef00fcb30 msgr2=0x7f4ef01107b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef4d27700 1 --2- 172.21.15.162:0/91274364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ef00fcb30 0x7f4ef01107b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef4d27700 1 -- 172.21.15.162:0/91274364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ef00f73a0 msgr2=0x7f4ef01102c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef4d27700 1 --2- 172.21.15.162:0/91274364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ef00f73a0 0x7f4ef01102c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef4d27700 1 -- 172.21.15.162:0/91274364 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee4005040 con 0x7f4ef00fd490 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4eef7fe700 1 --2- 172.21.15.162:0/91274364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ef00fcb30 0x7f4ef01107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef4d27700 1 --2- 172.21.15.162:0/91274364 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ef00fd490 0x7f4ef010a460 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 rx=0x7f4ee8004b70 tx=0x7f4ee8004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:44.287 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4eed7fa700 10 client.?.objecter ms_handle_connect 0x7f4ef00fd490 2024-02-13T00:52:44.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4eed7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:44.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee800e030 con 0x7f4ef00fd490 2024-02-13T00:52:44.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee80097c0 con 0x7f4ef00fd490 2024-02-13T00:52:44.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef678a700 10 client.38111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:44.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef010b1a0 con 0x7f4ef00fd490 2024-02-13T00:52:44.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4ef678a700 1 librados: init done 2024-02-13T00:52:44.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ee8007d30 con 0x7f4ef00fd490 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.281+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8018070 con 0x7f4ef00fd490 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.282+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4edc004f30 con 0x7f4ef00fd490 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.283+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4ee801d420 con 0x7f4ef00fd490 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.283+0000 7f4eed7fa700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4ed8062c90 0x7f4ed8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.283+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(735..735 src has 449..735) v4 ==== 8616+0+0 (secure 0 0 0) 0x7f4ee807eb60 con 0x7f4ef00fd490 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.283+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 osd_map(735..735 src has 449..735) v4 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.283+0000 7f4eed7fa700 3 client.38111.objecter handle_osd_map got epochs [735,735] > 0 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.283+0000 7f4eed7fa700 3 client.38111.objecter handle_osd_map decoding full epoch 735 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.283+0000 7f4eeffff700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4ed8062c90 0x7f4ed8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4eed7fa700 20 client.38111.objecter dump_active .. 0 homeless 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v735) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4ee8083220 con 0x7f4ef00fd490 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4eed7fa700 10 client.38111.objecter ms_handle_connect 0x7f4ed8062c90 2024-02-13T00:52:44.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4eeffff700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4ed8062c90 0x7f4ed8065150 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 rx=0x7f4ee0004c60 tx=0x7f4ee000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4ef678a700 10 client.38111.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 1 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4edc004560 0x7f4edc0049a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc0051d0 osd=0 3 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc0051d0 osd=0 3 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc0051d0 osd=0 3 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 1 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc0051d0 osd=0 3 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 0 1 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc0051d0 osd=0 4 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4ef678a700 10 client.38111.objecter _send_command 1 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4edc008960 con 0x7f4edc004560 2024-02-13T00:52:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4eef7fe700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4edc004560 0x7f4edc0049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:44.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4eed7fa700 10 client.38111.objecter ms_handle_connect 0x7f4edc004560 2024-02-13T00:52:44.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.284+0000 7f4eef7fe700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4edc004560 0x7f4edc0049a0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:44.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.285+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4edc008960 con 0x7f4edc004560 2024-02-13T00:52:44.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.285+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:52:44.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.285+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 1 = system:0 2024-02-13T00:52:44.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.285+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc0051d0 osd=0 4 2024-02-13T00:52:44.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.285+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 0 1 2024-02-13T00:52:44.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.301+0000 7f4ef678a700 10 client.38111.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:44.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.301+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 2 2024-02-13T00:52:44.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.301+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc0051d0 osd=0 3 2024-02-13T00:52:44.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.301+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc0051d0 osd=0 3 2024-02-13T00:52:44.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.301+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc0051d0 osd=0 3 2024-02-13T00:52:44.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.301+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 2 2024-02-13T00:52:44.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.301+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc0051d0 osd=0 3 2024-02-13T00:52:44.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.301+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 0 2 2024-02-13T00:52:44.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.301+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc0051d0 osd=0 4 2024-02-13T00:52:44.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.301+0000 7f4ef678a700 10 client.38111.objecter _send_command 2 2024-02-13T00:52:44.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.301+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4edc0081f0 con 0x7f4edc004560 2024-02-13T00:52:44.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4edc0081f0 con 0x7f4edc004560 2024-02-13T00:52:44.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:44.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc0051d0 osd=0 4 2024-02-13T00:52:44.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 0 2 2024-02-13T00:52:44.307 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:52:44.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4ef678a700 10 client.38111.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 3 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6805/150599780 conn(0x7f4edc002c90 0x7f4edc003120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc008650 osd=1 3 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc008650 osd=1 3 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc008650 osd=1 3 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 3 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc008650 osd=1 3 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 1 3 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc008650 osd=1 4 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4ef678a700 10 client.38111.objecter _send_command 3 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4edc0036a0 con 0x7f4edc002c90 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4eeffff700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6805/150599780 conn(0x7f4edc002c90 0x7f4edc003120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:44.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4eed7fa700 10 client.38111.objecter ms_handle_connect 0x7f4edc002c90 2024-02-13T00:52:44.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.303+0000 7f4eeffff700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6805/150599780 conn(0x7f4edc002c90 0x7f4edc003120 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:44.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.304+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4edc0036a0 con 0x7f4edc002c90 2024-02-13T00:52:44.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.304+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:52:44.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.304+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 3 = system:0 2024-02-13T00:52:44.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.304+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc008650 osd=1 4 2024-02-13T00:52:44.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.304+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 1 3 2024-02-13T00:52:44.309 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.320+0000 7f4ef678a700 10 client.38111.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:44.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.320+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 4 2024-02-13T00:52:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.320+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc008650 osd=1 3 2024-02-13T00:52:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.320+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc008650 osd=1 3 2024-02-13T00:52:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.320+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc008650 osd=1 3 2024-02-13T00:52:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.320+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 4 2024-02-13T00:52:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.320+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc008650 osd=1 3 2024-02-13T00:52:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.320+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 1 4 2024-02-13T00:52:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.320+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc008650 osd=1 4 2024-02-13T00:52:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.320+0000 7f4ef678a700 10 client.38111.objecter _send_command 4 2024-02-13T00:52:44.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.320+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4edc004090 con 0x7f4edc002c90 2024-02-13T00:52:44.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4edc004090 con 0x7f4edc002c90 2024-02-13T00:52:44.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:44.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc008650 osd=1 4 2024-02-13T00:52:44.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 1 4 2024-02-13T00:52:44.326 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4ef678a700 10 client.38111.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 5 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4edc00aff0 0x7f4edc00d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc007d20 osd=2 3 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc007d20 osd=2 3 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc007d20 osd=2 3 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 5 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc007d20 osd=2 3 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 2 5 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc007d20 osd=2 4 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4ef678a700 10 client.38111.objecter _send_command 5 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4edc010030 con 0x7f4edc00aff0 2024-02-13T00:52:44.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4eef7fe700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4edc00aff0 0x7f4edc00d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:44.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4eed7fa700 10 client.38111.objecter ms_handle_connect 0x7f4edc00aff0 2024-02-13T00:52:44.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.322+0000 7f4eef7fe700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4edc00aff0 0x7f4edc00d3e0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:44.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.323+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4edc010030 con 0x7f4edc00aff0 2024-02-13T00:52:44.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.323+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:52:44.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.323+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 5 = system:0 2024-02-13T00:52:44.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.323+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc007d20 osd=2 4 2024-02-13T00:52:44.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.323+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 2 5 2024-02-13T00:52:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.337+0000 7f4ef678a700 10 client.38111.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.337+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 6 2024-02-13T00:52:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.337+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc007d20 osd=2 3 2024-02-13T00:52:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.337+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc007d20 osd=2 3 2024-02-13T00:52:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.337+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc007d20 osd=2 3 2024-02-13T00:52:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.337+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 6 2024-02-13T00:52:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.337+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc007d20 osd=2 3 2024-02-13T00:52:44.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.337+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 2 6 2024-02-13T00:52:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.337+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc007d20 osd=2 4 2024-02-13T00:52:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.337+0000 7f4ef678a700 10 client.38111.objecter _send_command 6 2024-02-13T00:52:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.337+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4edc003b00 con 0x7f4edc00aff0 2024-02-13T00:52:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4edc003b00 con 0x7f4edc00aff0 2024-02-13T00:52:44.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:44.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc007d20 osd=2 4 2024-02-13T00:52:44.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 2 6 2024-02-13T00:52:44.343 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:52:44.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef678a700 10 client.38111.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:44.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 7 2024-02-13T00:52:44.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6813/604971011 conn(0x7f4edc00ea50 0x7f4edc00ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:44.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc001bb0 osd=3 3 2024-02-13T00:52:44.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc001bb0 osd=3 3 2024-02-13T00:52:44.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc001bb0 osd=3 3 2024-02-13T00:52:44.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 7 2024-02-13T00:52:44.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc001bb0 osd=3 3 2024-02-13T00:52:44.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 3 7 2024-02-13T00:52:44.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc001bb0 osd=3 4 2024-02-13T00:52:44.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef678a700 10 client.38111.objecter _send_command 7 2024-02-13T00:52:44.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4edc002230 con 0x7f4edc00ea50 2024-02-13T00:52:44.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.339+0000 7f4ef4d27700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6813/604971011 conn(0x7f4edc00ea50 0x7f4edc00ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:44.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.340+0000 7f4eed7fa700 10 client.38111.objecter ms_handle_connect 0x7f4edc00ea50 2024-02-13T00:52:44.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.340+0000 7f4ef4d27700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6813/604971011 conn(0x7f4edc00ea50 0x7f4edc00ef00 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:44.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.340+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4edc002230 con 0x7f4edc00ea50 2024-02-13T00:52:44.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.340+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:52:44.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.340+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 7 = system:0 2024-02-13T00:52:44.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.340+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc001bb0 osd=3 4 2024-02-13T00:52:44.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.340+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 3 7 2024-02-13T00:52:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.354+0000 7f4ef678a700 10 client.38111.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.354+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 8 2024-02-13T00:52:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.354+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc001bb0 osd=3 3 2024-02-13T00:52:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.354+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc001bb0 osd=3 3 2024-02-13T00:52:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.354+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc001bb0 osd=3 3 2024-02-13T00:52:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.354+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 8 2024-02-13T00:52:44.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.354+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc001bb0 osd=3 3 2024-02-13T00:52:44.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.354+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 3 8 2024-02-13T00:52:44.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.354+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc001bb0 osd=3 4 2024-02-13T00:52:44.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.354+0000 7f4ef678a700 10 client.38111.objecter _send_command 8 2024-02-13T00:52:44.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.354+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4edc00fbc0 con 0x7f4edc00ea50 2024-02-13T00:52:44.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4edc00fbc0 con 0x7f4edc00ea50 2024-02-13T00:52:44.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:44.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc001bb0 osd=3 4 2024-02-13T00:52:44.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 3 8 2024-02-13T00:52:44.360 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:52:44.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4ef678a700 10 client.38111.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:44.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 9 2024-02-13T00:52:44.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4edc0121e0 0x7f4edc0145d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:44.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc00fbc0 osd=4 3 2024-02-13T00:52:44.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc00fbc0 osd=4 3 2024-02-13T00:52:44.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc00fbc0 osd=4 3 2024-02-13T00:52:44.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 9 2024-02-13T00:52:44.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc00fbc0 osd=4 3 2024-02-13T00:52:44.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 4 9 2024-02-13T00:52:44.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc00fbc0 osd=4 4 2024-02-13T00:52:44.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4ef678a700 10 client.38111.objecter _send_command 9 2024-02-13T00:52:44.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.356+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4edc017260 con 0x7f4edc0121e0 2024-02-13T00:52:44.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.357+0000 7f4eeffff700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4edc0121e0 0x7f4edc0145d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:44.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.357+0000 7f4eed7fa700 10 client.38111.objecter ms_handle_connect 0x7f4edc0121e0 2024-02-13T00:52:44.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.357+0000 7f4eeffff700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4edc0121e0 0x7f4edc0145d0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:44.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.358+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4edc017260 con 0x7f4edc0121e0 2024-02-13T00:52:44.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.358+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:52:44.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.358+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 9 = system:0 2024-02-13T00:52:44.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.358+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc00fbc0 osd=4 4 2024-02-13T00:52:44.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.358+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 4 9 2024-02-13T00:52:44.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.372+0000 7f4ef678a700 10 client.38111.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:44.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.372+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 10 2024-02-13T00:52:44.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.372+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc00fbc0 osd=4 3 2024-02-13T00:52:44.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.372+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc00fbc0 osd=4 3 2024-02-13T00:52:44.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.372+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc00fbc0 osd=4 3 2024-02-13T00:52:44.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.372+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 10 2024-02-13T00:52:44.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.372+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc00fbc0 osd=4 3 2024-02-13T00:52:44.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.372+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 4 10 2024-02-13T00:52:44.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.372+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc00fbc0 osd=4 4 2024-02-13T00:52:44.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.372+0000 7f4ef678a700 10 client.38111.objecter _send_command 10 2024-02-13T00:52:44.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.372+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4edc017260 con 0x7f4edc0121e0 2024-02-13T00:52:44.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4edc017260 con 0x7f4edc0121e0 2024-02-13T00:52:44.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:44.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc00fbc0 osd=4 4 2024-02-13T00:52:44.378 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:52:44.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 4 10 2024-02-13T00:52:44.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4ef678a700 10 client.38111.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:44.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 11 2024-02-13T00:52:44.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4edc014fd0 0x7f4edc015410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:44.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc017260 osd=5 3 2024-02-13T00:52:44.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc017260 osd=5 3 2024-02-13T00:52:44.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc017260 osd=5 3 2024-02-13T00:52:44.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 11 2024-02-13T00:52:44.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc017260 osd=5 3 2024-02-13T00:52:44.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 5 11 2024-02-13T00:52:44.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc017260 osd=5 4 2024-02-13T00:52:44.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4ef678a700 10 client.38111.objecter _send_command 11 2024-02-13T00:52:44.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4edc017060 con 0x7f4edc014fd0 2024-02-13T00:52:44.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4eef7fe700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4edc014fd0 0x7f4edc015410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:44.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4eed7fa700 10 client.38111.objecter ms_handle_connect 0x7f4edc014fd0 2024-02-13T00:52:44.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.374+0000 7f4eef7fe700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4edc014fd0 0x7f4edc015410 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:44.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.375+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4edc017060 con 0x7f4edc014fd0 2024-02-13T00:52:44.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.375+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:52:44.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.375+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 11 = system:0 2024-02-13T00:52:44.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.375+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc017260 osd=5 4 2024-02-13T00:52:44.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.375+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 5 11 2024-02-13T00:52:44.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.390+0000 7f4ef678a700 10 client.38111.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:44.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.390+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 12 2024-02-13T00:52:44.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.390+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc017260 osd=5 3 2024-02-13T00:52:44.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.390+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc017260 osd=5 3 2024-02-13T00:52:44.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.390+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc017260 osd=5 3 2024-02-13T00:52:44.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.390+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 12 2024-02-13T00:52:44.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.390+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc017260 osd=5 3 2024-02-13T00:52:44.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.390+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 5 12 2024-02-13T00:52:44.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.390+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc017260 osd=5 4 2024-02-13T00:52:44.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.390+0000 7f4ef678a700 10 client.38111.objecter _send_command 12 2024-02-13T00:52:44.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.390+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4edc017060 con 0x7f4edc014fd0 2024-02-13T00:52:44.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4edc017060 con 0x7f4edc014fd0 2024-02-13T00:52:44.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:44.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc017260 osd=5 4 2024-02-13T00:52:44.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 5 12 2024-02-13T00:52:44.396 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:52:44.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4ef678a700 10 client.38111.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:44.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 13 2024-02-13T00:52:44.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4edc019ac0 0x7f4edc019f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:44.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc017060 osd=6 3 2024-02-13T00:52:44.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc017060 osd=6 3 2024-02-13T00:52:44.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc017060 osd=6 3 2024-02-13T00:52:44.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 13 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc017060 osd=6 3 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 6 13 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc017060 osd=6 4 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4ef678a700 10 client.38111.objecter _send_command 13 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.392+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4edc01d820 con 0x7f4edc019ac0 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.393+0000 7f4ef4d27700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4edc019ac0 0x7f4edc019f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.393+0000 7f4eed7fa700 10 client.38111.objecter ms_handle_connect 0x7f4edc019ac0 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.393+0000 7f4ef4d27700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4edc019ac0 0x7f4edc019f00 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.393+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4edc01d820 con 0x7f4edc019ac0 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.393+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.393+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 13 = system:0 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.394+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc017060 osd=6 4 2024-02-13T00:52:44.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.394+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 6 13 2024-02-13T00:52:44.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.408+0000 7f4ef678a700 10 client.38111.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:44.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.408+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 14 2024-02-13T00:52:44.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.408+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc017060 osd=6 3 2024-02-13T00:52:44.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.408+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc017060 osd=6 3 2024-02-13T00:52:44.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.408+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc017060 osd=6 3 2024-02-13T00:52:44.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.408+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 14 2024-02-13T00:52:44.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.408+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc017060 osd=6 3 2024-02-13T00:52:44.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.408+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 6 14 2024-02-13T00:52:44.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.408+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc017060 osd=6 4 2024-02-13T00:52:44.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.408+0000 7f4ef678a700 10 client.38111.objecter _send_command 14 2024-02-13T00:52:44.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.408+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4edc01c6a0 con 0x7f4edc019ac0 2024-02-13T00:52:44.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4edc01c6a0 con 0x7f4edc019ac0 2024-02-13T00:52:44.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:44.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc017060 osd=6 4 2024-02-13T00:52:44.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 6 14 2024-02-13T00:52:44.414 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:52:44.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4ef678a700 10 client.38111.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:44.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 15 2024-02-13T00:52:44.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4edc01cd80 0x7f4edc01d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:44.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc01c6a0 osd=7 3 2024-02-13T00:52:44.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc01c6a0 osd=7 3 2024-02-13T00:52:44.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc01c6a0 osd=7 3 2024-02-13T00:52:44.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 15 2024-02-13T00:52:44.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc01c6a0 osd=7 3 2024-02-13T00:52:44.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 7 15 2024-02-13T00:52:44.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc01c6a0 osd=7 4 2024-02-13T00:52:44.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4ef678a700 10 client.38111.objecter _send_command 15 2024-02-13T00:52:44.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4edc01d7c0 con 0x7f4edc01cd80 2024-02-13T00:52:44.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.410+0000 7f4eeffff700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4edc01cd80 0x7f4edc01d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:44.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.411+0000 7f4eed7fa700 10 client.38111.objecter ms_handle_connect 0x7f4edc01cd80 2024-02-13T00:52:44.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.411+0000 7f4eeffff700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4edc01cd80 0x7f4edc01d1e0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:44.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.411+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4edc01d7c0 con 0x7f4edc01cd80 2024-02-13T00:52:44.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.411+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:52:44.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.411+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 15 = system:0 2024-02-13T00:52:44.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.411+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc01c6a0 osd=7 4 2024-02-13T00:52:44.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.411+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 7 15 2024-02-13T00:52:44.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.426+0000 7f4ef678a700 10 client.38111.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:52:44.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.426+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign -1 16 2024-02-13T00:52:44.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.426+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc01c6a0 osd=7 3 2024-02-13T00:52:44.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.426+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc01c6a0 osd=7 3 2024-02-13T00:52:44.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.426+0000 7f4ef678a700 20 client.38111.objecter _get_session s=0x7f4edc01c6a0 osd=7 3 2024-02-13T00:52:44.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.426+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_remove -1 16 2024-02-13T00:52:44.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.426+0000 7f4ef678a700 20 client.38111.objecter get_session s=0x7f4edc01c6a0 osd=7 3 2024-02-13T00:52:44.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.426+0000 7f4ef678a700 15 client.38111.objecter _session_command_op_assign 7 16 2024-02-13T00:52:44.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.426+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc01c6a0 osd=7 4 2024-02-13T00:52:44.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.426+0000 7f4ef678a700 10 client.38111.objecter _send_command 16 2024-02-13T00:52:44.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.426+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4edc01d7c0 con 0x7f4edc01cd80 2024-02-13T00:52:44.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4eed7fa700 1 -- 172.21.15.162:0/91274364 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4edc01d7c0 con 0x7f4edc01cd80 2024-02-13T00:52:44.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4eed7fa700 10 client.38111.objecter ms_dispatch 0x7f4ef0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:52:44.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4eed7fa700 10 client.38111.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4eed7fa700 20 client.38111.objecter put_session s=0x7f4edc01c6a0 osd=7 4 2024-02-13T00:52:44.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4eed7fa700 15 client.38111.objecter _session_command_op_remove 7 16 2024-02-13T00:52:44.432 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:52:44.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 10 librados: watch_flush enter 2024-02-13T00:52:44.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 10 librados: watch_flush exit 2024-02-13T00:52:44.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 10 client.38111.objecter close_session for osd.0 2024-02-13T00:52:44.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4edc004560 msgr2=0x7f4edc0049a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6801/2103743338 conn(0x7f4edc004560 0x7f4edc0049a0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc0051d0 osd=0 1 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 10 client.38111.objecter close_session for osd.1 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6805/150599780 conn(0x7f4edc002c90 msgr2=0x7f4edc003120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6805/150599780 conn(0x7f4edc002c90 0x7f4edc003120 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc008650 osd=1 1 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 10 client.38111.objecter close_session for osd.2 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4edc00aff0 msgr2=0x7f4edc00d3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6809/3689719376 conn(0x7f4edc00aff0 0x7f4edc00d3e0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc007d20 osd=2 1 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 10 client.38111.objecter close_session for osd.3 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6813/604971011 conn(0x7f4edc00ea50 msgr2=0x7f4edc00ef00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6813/604971011 conn(0x7f4edc00ea50 0x7f4edc00ef00 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc001bb0 osd=3 1 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 10 client.38111.objecter close_session for osd.4 2024-02-13T00:52:44.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4edc0121e0 msgr2=0x7f4edc0145d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6800/3344263388 conn(0x7f4edc0121e0 0x7f4edc0145d0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc00fbc0 osd=4 1 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 10 client.38111.objecter close_session for osd.5 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4edc014fd0 msgr2=0x7f4edc015410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6804/2277917372 conn(0x7f4edc014fd0 0x7f4edc015410 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc017260 osd=5 1 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 10 client.38111.objecter close_session for osd.6 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4edc019ac0 msgr2=0x7f4edc019f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6808/1506640341 conn(0x7f4edc019ac0 0x7f4edc019f00 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc017060 osd=6 1 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 10 client.38111.objecter close_session for osd.7 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4edc01cd80 msgr2=0x7f4edc01d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.181:6812/1964192772 conn(0x7f4edc01cd80 0x7f4edc01d1e0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.435 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 20 client.38111.objecter put_session s=0x7f4edc01c6a0 osd=7 1 2024-02-13T00:52:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 20 client.38111.objecter shutdown clearing up homeless session... 2024-02-13T00:52:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef678a700 10 client.38111.objecter successfully canceled tick 2024-02-13T00:52:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef4d27700 1 -- 172.21.15.162:0/91274364 reap_dead start 2024-02-13T00:52:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.428+0000 7f4ef4d27700 1 -- 172.21.15.162:0/91274364 reap_dead start 2024-02-13T00:52:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef4d27700 1 -- 172.21.15.162:0/91274364 reap_dead start 2024-02-13T00:52:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef4d27700 1 -- 172.21.15.162:0/91274364 reap_dead start 2024-02-13T00:52:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef4d27700 1 -- 172.21.15.162:0/91274364 reap_dead start 2024-02-13T00:52:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4ed8062c90 msgr2=0x7f4ed8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4ed8062c90 0x7f4ed8065150 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 rx=0x7f4ee0004c60 tx=0x7f4ee000b040).stop 2024-02-13T00:52:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ef00fd490 msgr2=0x7f4ef010a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ef00fd490 0x7f4ef010a460 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 rx=0x7f4ee8004b70 tx=0x7f4ee8004d10).stop 2024-02-13T00:52:44.437 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:44.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 shutdown_connections 2024-02-13T00:52:44.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ef00fd490 0x7f4ef010a460 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef678a700 1 --2- 172.21.15.162:0/91274364 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4ed8062c90 0x7f4ed8065150 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 >> 172.21.15.162:0/91274364 conn(0x7f4ef006eb90 msgr2=0x7f4ef006fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 shutdown_connections 2024-02-13T00:52:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef678a700 1 -- 172.21.15.162:0/91274364 wait complete. 2024-02-13T00:52:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:44.429+0000 7f4ef678a700 1 librados: shutdown 2024-02-13T00:52:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.438 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:52:44.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:44.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:44.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:44.566 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:44.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:44.284285+0000 mon.c (mon.2) 499 : audit [DBG] from='client.? 172.21.15.162:0/91274364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:44.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:44.284285+0000 mon.c (mon.2) 499 : audit [DBG] from='client.? 172.21.15.162:0/91274364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:44.284285+0000 mon.c (mon.2) 499 : audit [DBG] from='client.? 172.21.15.162:0/91274364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:45.517 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:45.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:45.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:45.650 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:45.724 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:45.724 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:45.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:45.822 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:44.526365+0000 mgr.y (mgr.14544) 606 : cluster [DBG] pgmap v1110: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:52:45.823 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:44.526365+0000 mgr.y (mgr.14544) 606 : cluster [DBG] pgmap v1110: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:52:45.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:52:45.860 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:45.881 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:52:45.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:44.526365+0000 mgr.y (mgr.14544) 606 : cluster [DBG] pgmap v1110: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:52:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:45.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:52:45.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:46.012 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:46.237 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:46.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:46.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:52:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:52:46.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:46.369 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:46.456 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:46.468 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:52:46.469 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:52:46.469 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:52:46.469 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:52:46.469 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:52:46.470 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:46.473 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:52:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:52:46] "GET /metrics HTTP/1.1" 200 189720 "" "Prometheus/2.33.4" 2024-02-13T00:52:46.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:46.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:46.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:46.609 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:46.629 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:46.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:46.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:45.917119+0000 mgr.y (mgr.14544) 607 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:46.726 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:45.917119+0000 mgr.y (mgr.14544) 607 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:46.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:46.763 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:45.917119+0000 mgr.y (mgr.14544) 607 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:47.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:47.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:47.446 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:48.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:46.527195+0000 mgr.y (mgr.14544) 608 : cluster [DBG] pgmap v1111: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-13T00:52:48.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:46.527195+0000 mgr.y (mgr.14544) 608 : cluster [DBG] pgmap v1111: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-13T00:52:48.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:46.527195+0000 mgr.y (mgr.14544) 608 : cluster [DBG] pgmap v1111: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-13T00:52:48.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:52:48.287 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:48.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:48.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:48.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:48.419 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:48.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-13T00:52:49.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:52:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:52:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:52:49.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:49.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:49.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:49.301 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:48.527726+0000 mgr.y (mgr.14544) 609 : cluster [DBG] pgmap v1112: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 641 B/s rd, 3.0 KiB/s wr, 1 op/s 2024-02-13T00:52:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:49.343271+0000 mon.a (mon.0) 3526 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:50.074 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:48.527726+0000 mgr.y (mgr.14544) 609 : cluster [DBG] pgmap v1112: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 641 B/s rd, 3.0 KiB/s wr, 1 op/s 2024-02-13T00:52:50.074 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:49.343271+0000 mon.a (mon.0) 3526 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:50.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:48.527726+0000 mgr.y (mgr.14544) 609 : cluster [DBG] pgmap v1112: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 641 B/s rd, 3.0 KiB/s wr, 1 op/s 2024-02-13T00:52:50.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:49.343271+0000 mon.a (mon.0) 3526 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:50.350 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:50.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:50.364 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:50.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:50.485 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:50.506 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:50.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:50.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:50.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:50.702 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:50.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:50.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:50.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:50.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.798+0000 7fe487836700 1 -- 172.21.15.162:0/2852305059 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4800fa440 msgr2=0x7fe4800f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:50.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.798+0000 7fe487836700 1 --2- 172.21.15.162:0/2852305059 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4800fa440 0x7fe4800f7e20 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 rx=0x7fe47c004870 tx=0x7fe47c009cb0).stop 2024-02-13T00:52:50.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.798+0000 7fe487836700 1 -- 172.21.15.162:0/2852305059 shutdown_connections 2024-02-13T00:52:50.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.798+0000 7fe487836700 1 --2- 172.21.15.162:0/2852305059 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4800f8500 0x7fe480104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:50.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.798+0000 7fe487836700 1 --2- 172.21.15.162:0/2852305059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4800f9b70 0x7fe4800f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:50.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.798+0000 7fe487836700 1 --2- 172.21.15.162:0/2852305059 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4800fa440 0x7fe4800f7e20 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:50.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.798+0000 7fe487836700 1 -- 172.21.15.162:0/2852305059 >> 172.21.15.162:0/2852305059 conn(0x7fe4800f38c0 msgr2=0x7fe4800f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:50.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.798+0000 7fe487836700 1 -- 172.21.15.162:0/2852305059 shutdown_connections 2024-02-13T00:52:50.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.798+0000 7fe487836700 1 -- 172.21.15.162:0/2852305059 wait complete. 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.798+0000 7fe487836700 1 librados: starting msgr at 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.798+0000 7fe487836700 1 librados: starting objecter 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.798+0000 7fe487836700 1 Processor -- start 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 1 -- start start 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 1 librados: setting wanted keys 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 1 librados: calling monclient init 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4800f8500 0x7fe48010e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4800f9b70 0x7fe48010e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4800fa440 0x7fe4801083d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4801088c0 con 0x7fe4800fa440 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe480108a30 con 0x7fe4800f8500 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe480108bf0 con 0x7fe4800f9b70 2024-02-13T00:52:50.811 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe484dd1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4800f9b70 0x7fe48010e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:50.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe485dd3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4800fa440 0x7fe4801083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:50.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe484dd1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4800f9b70 0x7fe48010e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:42126/0 (socket says 172.21.15.162:42126) 2024-02-13T00:52:50.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe484dd1700 1 -- 172.21.15.162:0/3324659107 learned_addr learned my addr 172.21.15.162:0/3324659107 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:50.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe4855d2700 1 --2- 172.21.15.162:0/3324659107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4800f8500 0x7fe48010e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:50.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe484dd1700 1 -- 172.21.15.162:0/3324659107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4800f8500 msgr2=0x7fe48010e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:50.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe484dd1700 1 --2- 172.21.15.162:0/3324659107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4800f8500 0x7fe48010e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:50.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe484dd1700 1 -- 172.21.15.162:0/3324659107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4800fa440 msgr2=0x7fe4801083d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:50.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe484dd1700 1 --2- 172.21.15.162:0/3324659107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4800fa440 0x7fe4801083d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:50.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe484dd1700 1 -- 172.21.15.162:0/3324659107 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe470005040 con 0x7fe4800f9b70 2024-02-13T00:52:50.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe485dd3700 1 --2- 172.21.15.162:0/3324659107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4800fa440 0x7fe4801083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:52:50.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe484dd1700 1 --2- 172.21.15.162:0/3324659107 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4800f9b70 0x7fe48010e720 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7fe47c009eb0 tx=0x7fe47c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe4767fc700 10 client.?.objecter ms_handle_connect 0x7fe4800f9b70 2024-02-13T00:52:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe4767fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe4767fc700 1 -- 172.21.15.162:0/3324659107 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00e070 con 0x7fe4800f9b70 2024-02-13T00:52:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe4767fc700 1 -- 172.21.15.162:0/3324659107 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe47c013da0 con 0x7fe4800f9b70 2024-02-13T00:52:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 1 -- 172.21.15.162:0/3324659107 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe47c005040 con 0x7fe4800f9b70 2024-02-13T00:52:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe4767fc700 1 -- 172.21.15.162:0/3324659107 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c019070 con 0x7fe4800f9b70 2024-02-13T00:52:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 10 client.38117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 1 -- 172.21.15.162:0/3324659107 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe480109160 con 0x7fe4800f9b70 2024-02-13T00:52:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.799+0000 7fe487836700 1 librados: init done 2024-02-13T00:52:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.800+0000 7fe487836700 1 -- 172.21.15.162:0/3324659107 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4800475e0 con 0x7fe4800f9b70 2024-02-13T00:52:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.801+0000 7fe4767fc700 1 -- 172.21.15.162:0/3324659107 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe47c01e420 con 0x7fe4800f9b70 2024-02-13T00:52:50.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.801+0000 7fe4767fc700 1 --2- 172.21.15.162:0/3324659107 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe46c062c90 0x7fe46c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:50.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.801+0000 7fe4767fc700 1 -- 172.21.15.162:0/3324659107 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(735..735 src has 449..735) v4 ==== 8616+0+0 (secure 0 0 0) 0x7fe47c07df10 con 0x7fe4800f9b70 2024-02-13T00:52:50.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.801+0000 7fe4767fc700 10 client.38117.objecter ms_dispatch 0x7fe480001550 osd_map(735..735 src has 449..735) v4 2024-02-13T00:52:50.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.801+0000 7fe4767fc700 3 client.38117.objecter handle_osd_map got epochs [735,735] > 0 2024-02-13T00:52:50.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.801+0000 7fe4767fc700 3 client.38117.objecter handle_osd_map decoding full epoch 735 2024-02-13T00:52:50.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.801+0000 7fe4855d2700 1 --2- 172.21.15.162:0/3324659107 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe46c062c90 0x7fe46c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:50.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.801+0000 7fe4767fc700 20 client.38117.objecter dump_active .. 0 homeless 2024-02-13T00:52:50.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.802+0000 7fe4855d2700 1 --2- 172.21.15.162:0/3324659107 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe46c062c90 0x7fe46c065150 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7fe4700092f0 tx=0x7fe470009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:50.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.802+0000 7fe4767fc700 10 client.38117.objecter ms_handle_connect 0x7fe46c062c90 2024-02-13T00:52:50.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.805+0000 7fe4767fc700 1 -- 172.21.15.162:0/3324659107 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe47c0511b0 con 0x7fe4800f9b70 2024-02-13T00:52:50.842 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:50.861 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:50.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:50.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:50.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:50.988+0000 7fe487836700 1 -- 172.21.15.162:0/3324659107 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fe4801098b0 con 0x7fe4800f9b70 2024-02-13T00:52:50.992 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:51.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.025+0000 7f1b79b21700 1 -- 172.21.15.162:0/937247544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b740fd490 msgr2=0x7f1b740f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:51.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.025+0000 7f1b79b21700 1 --2- 172.21.15.162:0/937247544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b740fd490 0x7f1b740f6cc0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 rx=0x7f1b680047b0 tx=0x7f1b68009cb0).stop 2024-02-13T00:52:51.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.025+0000 7f1b79b21700 1 -- 172.21.15.162:0/937247544 shutdown_connections 2024-02-13T00:52:51.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.025+0000 7f1b79b21700 1 --2- 172.21.15.162:0/937247544 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b740f73a0 0x7f1b741039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.025+0000 7f1b79b21700 1 --2- 172.21.15.162:0/937247544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b740fcb30 0x7f1b740fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.025+0000 7f1b79b21700 1 --2- 172.21.15.162:0/937247544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b740fd490 0x7f1b740f6cc0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.025+0000 7f1b79b21700 1 -- 172.21.15.162:0/937247544 >> 172.21.15.162:0/937247544 conn(0x7f1b7406eb90 msgr2=0x7f1b740f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:51.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.025+0000 7f1b79b21700 1 -- 172.21.15.162:0/937247544 shutdown_connections 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.026+0000 7f1b79b21700 1 -- 172.21.15.162:0/937247544 wait complete. 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.026+0000 7f1b79b21700 1 librados: starting msgr at 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.026+0000 7f1b79b21700 1 librados: starting objecter 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.026+0000 7f1b79b21700 1 Processor -- start 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.026+0000 7f1b79b21700 1 -- start start 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.026+0000 7f1b79b21700 1 librados: setting wanted keys 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.026+0000 7f1b79b21700 1 librados: calling monclient init 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b79b21700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b740f73a0 0x7f1b7410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b79b21700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b740fcb30 0x7f1b7410d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b79b21700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b740fd490 0x7f1b74107170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b79b21700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b74107660 con 0x7f1b740f73a0 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b79b21700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b741077d0 con 0x7f1b740fd490 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b79b21700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b74107990 con 0x7f1b740fcb30 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b72ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b740fcb30 0x7f1b7410d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:51.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b72ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b740fcb30 0x7f1b7410d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:42144/0 (socket says 172.21.15.162:42144) 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b72ffd700 1 -- 172.21.15.162:0/319899983 learned_addr learned my addr 172.21.15.162:0/319899983 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b737fe700 1 --2- 172.21.15.162:0/319899983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b740f73a0 0x7f1b7410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b73fff700 1 --2- 172.21.15.162:0/319899983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b740fd490 0x7f1b74107170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b72ffd700 1 -- 172.21.15.162:0/319899983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b740fd490 msgr2=0x7f1b74107170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b72ffd700 1 --2- 172.21.15.162:0/319899983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b740fd490 0x7f1b74107170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b72ffd700 1 -- 172.21.15.162:0/319899983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b740f73a0 msgr2=0x7f1b7410cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b72ffd700 1 --2- 172.21.15.162:0/319899983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b740f73a0 0x7f1b7410cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b72ffd700 1 -- 172.21.15.162:0/319899983 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c0097c0 con 0x7f1b740fcb30 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b737fe700 1 --2- 172.21.15.162:0/319899983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b740f73a0 0x7f1b7410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b72ffd700 1 --2- 172.21.15.162:0/319899983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b740fcb30 0x7f1b7410d4c0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 rx=0x7f1b680044e0 tx=0x7f1b68007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b70ff9700 10 client.?.objecter ms_handle_connect 0x7f1b740fcb30 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b70ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.027+0000 7f1b70ff9700 1 -- 172.21.15.162:0/319899983 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b68018070 con 0x7f1b740fcb30 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.028+0000 7f1b79b21700 1 -- 172.21.15.162:0/319899983 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b68005040 con 0x7f1b740fcb30 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.028+0000 7f1b70ff9700 1 -- 172.21.15.162:0/319899983 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b680093b0 con 0x7f1b740fcb30 2024-02-13T00:52:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.028+0000 7f1b70ff9700 1 -- 172.21.15.162:0/319899983 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6801dd90 con 0x7f1b740fcb30 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.028+0000 7f1b79b21700 10 client.38123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.028+0000 7f1b79b21700 1 -- 172.21.15.162:0/319899983 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b74107eb0 con 0x7f1b740fcb30 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.028+0000 7f1b79b21700 1 librados: init done 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.029+0000 7f1b79b21700 1 -- 172.21.15.162:0/319899983 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b740475e0 con 0x7f1b740fcb30 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.029+0000 7f1b70ff9700 1 -- 172.21.15.162:0/319899983 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f1b6801d420 con 0x7f1b740fcb30 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.029+0000 7f1b70ff9700 1 --2- 172.21.15.162:0/319899983 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1b5c062c90 0x7f1b5c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.029+0000 7f1b70ff9700 1 -- 172.21.15.162:0/319899983 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(735..735 src has 449..735) v4 ==== 8616+0+0 (secure 0 0 0) 0x7f1b6807e890 con 0x7f1b740fcb30 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.029+0000 7f1b70ff9700 10 client.38123.objecter ms_dispatch 0x7f1b74001550 osd_map(735..735 src has 449..735) v4 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.029+0000 7f1b70ff9700 3 client.38123.objecter handle_osd_map got epochs [735,735] > 0 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.029+0000 7f1b70ff9700 3 client.38123.objecter handle_osd_map decoding full epoch 735 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.029+0000 7f1b737fe700 1 --2- 172.21.15.162:0/319899983 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1b5c062c90 0x7f1b5c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.030+0000 7f1b70ff9700 20 client.38123.objecter dump_active .. 0 homeless 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.030+0000 7f1b70ff9700 10 client.38123.objecter ms_handle_connect 0x7f1b5c062c90 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.030+0000 7f1b737fe700 1 --2- 172.21.15.162:0/319899983 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1b5c062c90 0x7f1b5c065150 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 rx=0x7f1b640103c0 tx=0x7f1b6400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:51.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.034+0000 7f1b70ff9700 1 -- 172.21.15.162:0/319899983 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1b68051ad0 con 0x7f1b740fcb30 2024-02-13T00:52:51.083 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:51.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:51.099 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:51.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:52:51.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.205+0000 7f1b79b21700 1 -- 172.21.15.162:0/319899983 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b74108750 con 0x7f1b5c062c90 2024-02-13T00:52:51.217 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:51.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.219+0000 7f1b70ff9700 1 -- 172.21.15.162:0/319899983 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+507288 (secure 0 0 0) 0x7f1b6400f040 con 0x7f1b5c062c90 2024-02-13T00:52:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:52:51.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 10 librados: watch_flush enter 2024-02-13T00:52:51.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 10 librados: watch_flush exit 2024-02-13T00:52:51.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 20 client.38123.objecter shutdown clearing up homeless session... 2024-02-13T00:52:51.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 10 client.38123.objecter successfully canceled tick 2024-02-13T00:52:51.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 1 -- 172.21.15.162:0/319899983 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1b5c062c90 msgr2=0x7f1b5c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:51.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 1 --2- 172.21.15.162:0/319899983 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1b5c062c90 0x7f1b5c065150 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 rx=0x7f1b640103c0 tx=0x7f1b6400b040).stop 2024-02-13T00:52:51.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 1 -- 172.21.15.162:0/319899983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b740fcb30 msgr2=0x7f1b7410d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:51.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 1 --2- 172.21.15.162:0/319899983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b740fcb30 0x7f1b7410d4c0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 rx=0x7f1b680044e0 tx=0x7f1b68007f90).stop 2024-02-13T00:52:51.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 1 -- 172.21.15.162:0/319899983 shutdown_connections 2024-02-13T00:52:51.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 1 --2- 172.21.15.162:0/319899983 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1b5c062c90 0x7f1b5c065150 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 1 --2- 172.21.15.162:0/319899983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b740fd490 0x7f1b74107170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 1 --2- 172.21.15.162:0/319899983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b740f73a0 0x7f1b7410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 1 --2- 172.21.15.162:0/319899983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b740fcb30 0x7f1b7410d4c0 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.222+0000 7f1b79b21700 1 -- 172.21.15.162:0/319899983 >> 172.21.15.162:0/319899983 conn(0x7f1b7406eb90 msgr2=0x7f1b740fa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:51.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.223+0000 7f1b79b21700 1 -- 172.21.15.162:0/319899983 shutdown_connections 2024-02-13T00:52:51.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.223+0000 7f1b79b21700 1 -- 172.21.15.162:0/319899983 wait complete. 2024-02-13T00:52:51.235 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.223+0000 7f1b79b21700 1 librados: shutdown 2024-02-13T00:52:51.235 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:52:51.237 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:52:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:51.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:52:51.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:51.368 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:51.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:50.376904+0000 mon.a (mon.0) 3527 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:51.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:50.528553+0000 mgr.y (mgr.14544) 610 : cluster [DBG] pgmap v1113: 204 pgs: 204 active+clean; 3.9 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 KiB/s rd, 7.4 KiB/s wr, 2 op/s 2024-02-13T00:52:51.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:50.989450+0000 mon.c (mon.2) 500 : audit [INF] from='client.? 172.21.15.162:0/3324659107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:51.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:50.989790+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:51.572 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.568+0000 7fe4767fc700 1 -- 172.21.15.162:0/3324659107 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v736) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe47c023030 con 0x7fe4800f9b70 2024-02-13T00:52:51.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.570+0000 7fe487836700 10 librados: watch_flush enter 2024-02-13T00:52:51.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.570+0000 7fe487836700 10 librados: watch_flush exit 2024-02-13T00:52:51.574 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.570+0000 7fe487836700 20 client.38117.objecter shutdown clearing up homeless session... 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.570+0000 7fe487836700 10 client.38117.objecter successfully canceled tick 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 -- 172.21.15.162:0/3324659107 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe46c062c90 msgr2=0x7fe46c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 --2- 172.21.15.162:0/3324659107 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe46c062c90 0x7fe46c065150 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7fe4700092f0 tx=0x7fe470009cb0).stop 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 -- 172.21.15.162:0/3324659107 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4800f9b70 msgr2=0x7fe48010e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 --2- 172.21.15.162:0/3324659107 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4800f9b70 0x7fe48010e720 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7fe47c009eb0 tx=0x7fe47c00b040).stop 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 -- 172.21.15.162:0/3324659107 shutdown_connections 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 --2- 172.21.15.162:0/3324659107 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe46c062c90 0x7fe46c065150 unknown :-1 s=CLOSED pgs=3017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 --2- 172.21.15.162:0/3324659107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4800fa440 0x7fe4801083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 --2- 172.21.15.162:0/3324659107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4800f8500 0x7fe48010e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 --2- 172.21.15.162:0/3324659107 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe4800f9b70 0x7fe48010e720 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 -- 172.21.15.162:0/3324659107 >> 172.21.15.162:0/3324659107 conn(0x7fe4800f38c0 msgr2=0x7fe4800fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 -- 172.21.15.162:0/3324659107 shutdown_connections 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 -- 172.21.15.162:0/3324659107 wait complete. 2024-02-13T00:52:51.575 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:51.571+0000 7fe487836700 1 librados: shutdown 2024-02-13T00:52:51.576 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:52:51.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:50.376904+0000 mon.a (mon.0) 3527 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:51.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:50.528553+0000 mgr.y (mgr.14544) 610 : cluster [DBG] pgmap v1113: 204 pgs: 204 active+clean; 3.9 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 KiB/s rd, 7.4 KiB/s wr, 2 op/s 2024-02-13T00:52:51.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:50.989450+0000 mon.c (mon.2) 500 : audit [INF] from='client.? 172.21.15.162:0/3324659107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:51.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:50.989790+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:51.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:50.376904+0000 mon.a (mon.0) 3527 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:52:51.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:50.528553+0000 mgr.y (mgr.14544) 610 : cluster [DBG] pgmap v1113: 204 pgs: 204 active+clean; 3.9 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 KiB/s rd, 7.4 KiB/s wr, 2 op/s 2024-02-13T00:52:51.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:50.989450+0000 mon.c (mon.2) 500 : audit [INF] from='client.? 172.21.15.162:0/3324659107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:51.581 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:50.989790+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:52:52.083 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":1113,"stamp":"2024-02-13T00:52:50.527925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4071594,"num_objects":245,"num_object_clones":1,"num_object_copies":735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":2414,"num_read_kb":2122,"num_write":960,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":863,"num_bytes_recovered":4021117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1151,"ondisk_log_size":1151,"up":612,"acting":612,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":612,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1252712,"kb_used_data":26952,"kb_used_omap":0,"kb_used_meta":1225728,"kb_avail":373727896,"statfs":{"total":383980142592,"available":382697365504,"internally_reserved":0,"allocated":27598848,"data_stored":17185503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1255145472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":65704,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.975525"},"pg_stats":[{"pgid":"249.1f","version":"0'0","reported_seq":14,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557456+0000","last_change":"2024-02-13T00:52:37.955543+0000","last_active":"2024-02-13T00:52:41.557456+0000","last_peered":"2024-02-13T00:52:41.557456+0000","last_clean":"2024-02-13T00:52:41.557456+0000","last_became_active":"2024-02-13T00:52:37.955212+0000","last_became_peered":"2024-02-13T00:52:37.955212+0000","last_unstale":"2024-02-13T00:52:41.557456+0000","last_undegraded":"2024-02-13T00:52:41.557456+0000","last_fullsized":"2024-02-13T00:52:41.557456+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"249.1e","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.600100+0000","last_change":"2024-02-13T00:52:37.951574+0000","last_active":"2024-02-13T00:52:41.600100+0000","last_peered":"2024-02-13T00:52:41.600100+0000","last_clean":"2024-02-13T00:52:41.600100+0000","last_became_active":"2024-02-13T00:52:37.950325+0000","last_became_peered":"2024-02-13T00:52:37.950325+0000","last_unstale":"2024-02-13T00:52:41.600100+0000","last_undegraded":"2024-02-13T00:52:41.600100+0000","last_fullsized":"2024-02-13T00:52:41.600100+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.1d","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597883+0000","last_change":"2024-02-13T00:52:37.951328+0000","last_active":"2024-02-13T00:52:41.597883+0000","last_peered":"2024-02-13T00:52:41.597883+0000","last_clean":"2024-02-13T00:52:41.597883+0000","last_became_active":"2024-02-13T00:52:37.950910+0000","last_became_peered":"2024-02-13T00:52:37.950910+0000","last_unstale":"2024-02-13T00:52:41.597883+0000","last_undegraded":"2024-02-13T00:52:41.597883+0000","last_fullsized":"2024-02-13T00:52:41.597883+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.1c","version":"0'0","reported_seq":14,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.555263+0000","last_change":"2024-02-13T00:52:37.584703+0000","last_active":"2024-02-13T00:52:41.555263+0000","last_peered":"2024-02-13T00:52:41.555263+0000","last_clean":"2024-02-13T00:52:41.555263+0000","last_became_active":"2024-02-13T00:52:37.584430+0000","last_became_peered":"2024-02-13T00:52:37.584430+0000","last_unstale":"2024-02-13T00:52:41.555263+0000","last_undegraded":"2024-02-13T00:52:41.555263+0000","last_fullsized":"2024-02-13T00:52:41.555263+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"249.1b","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597183+0000","last_change":"2024-02-13T00:52:37.950745+0000","last_active":"2024-02-13T00:52:41.597183+0000","last_peered":"2024-02-13T00:52:41.597183+0000","last_clean":"2024-02-13T00:52:41.597183+0000","last_became_active":"2024-02-13T00:52:37.950443+0000","last_became_peered":"2024-02-13T00:52:37.950443+0000","last_unstale":"2024-02-13T00:52:41.597183+0000","last_undegraded":"2024-02-13T00:52:41.597183+0000","last_fullsized":"2024-02-13T00:52:41.597183+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.1a","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597089+0000","last_change":"2024-02-13T00:52:37.586415+0000","last_active":"2024-02-13T00:52:41.597089+0000","last_peered":"2024-02-13T00:52:41.597089+0000","last_clean":"2024-02-13T00:52:41.597089+0000","last_became_active":"2024-02-13T00:52:37.586132+0000","last_became_peered":"2024-02-13T00:52:37.586132+0000","last_unstale":"2024-02-13T00:52:41.597089+0000","last_undegraded":"2024-02-13T00:52:41.597089+0000","last_fullsized":"2024-02-13T00:52:41.597089+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.19","version":"0'0","reported_seq":14,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.554419+0000","last_change":"2024-02-13T00:52:37.955220+0000","last_active":"2024-02-13T00:52:41.554419+0000","last_peered":"2024-02-13T00:52:41.554419+0000","last_clean":"2024-02-13T00:52:41.554419+0000","last_became_active":"2024-02-13T00:52:37.954934+0000","last_became_peered":"2024-02-13T00:52:37.954934+0000","last_unstale":"2024-02-13T00:52:41.554419+0000","last_undegraded":"2024-02-13T00:52:41.554419+0000","last_fullsized":"2024-02-13T00:52:41.554419+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"249.18","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.596731+0000","last_change":"2024-02-13T00:52:37.949578+0000","last_active":"2024-02-13T00:52:41.596731+0000","last_peered":"2024-02-13T00:52:41.596731+0000","last_clean":"2024-02-13T00:52:41.596731+0000","last_became_active":"2024-02-13T00:52:37.949087+0000","last_became_peered":"2024-02-13T00:52:37.949087+0000","last_unstale":"2024-02-13T00:52:41.596731+0000","last_undegraded":"2024-02-13T00:52:41.596731+0000","last_fullsized":"2024-02-13T00:52:41.596731+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.f","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.600693+0000","last_change":"2024-02-13T00:52:37.951019+0000","last_active":"2024-02-13T00:52:41.600693+0000","last_peered":"2024-02-13T00:52:41.600693+0000","last_clean":"2024-02-13T00:52:41.600693+0000","last_became_active":"2024-02-13T00:52:37.950750+0000","last_became_peered":"2024-02-13T00:52:37.950750+0000","last_unstale":"2024-02-13T00:52:41.600693+0000","last_undegraded":"2024-02-13T00:52:41.600693+0000","last_fullsized":"2024-02-13T00:52:41.600693+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.e","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.284974+0000","last_change":"2024-02-13T00:52:37.953710+0000","last_active":"2024-02-13T00:52:42.284974+0000","last_peered":"2024-02-13T00:52:42.284974+0000","last_clean":"2024-02-13T00:52:42.284974+0000","last_became_active":"2024-02-13T00:52:37.953411+0000","last_became_peered":"2024-02-13T00:52:37.953411+0000","last_unstale":"2024-02-13T00:52:42.284974+0000","last_undegraded":"2024-02-13T00:52:42.284974+0000","last_fullsized":"2024-02-13T00:52:42.284974+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.d","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283667+0000","last_change":"2024-02-13T00:52:37.584319+0000","last_active":"2024-02-13T00:52:42.283667+0000","last_peered":"2024-02-13T00:52:42.283667+0000","last_clean":"2024-02-13T00:52:42.283667+0000","last_became_active":"2024-02-13T00:52:37.584105+0000","last_became_peered":"2024-02-13T00:52:37.584105+0000","last_unstale":"2024-02-13T00:52:42.283667+0000","last_undegraded":"2024-02-13T00:52:42.283667+0000","last_fullsized":"2024-02-13T00:52:42.283667+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.c","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.284880+0000","last_change":"2024-02-13T00:52:37.952965+0000","last_active":"2024-02-13T00:52:42.284880+0000","last_peered":"2024-02-13T00:52:42.284880+0000","last_clean":"2024-02-13T00:52:42.284880+0000","last_became_active":"2024-02-13T00:52:37.952661+0000","last_became_peered":"2024-02-13T00:52:37.952661+0000","last_unstale":"2024-02-13T00:52:42.284880+0000","last_undegraded":"2024-02-13T00:52:42.284880+0000","last_fullsized":"2024-02-13T00:52:42.284880+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.b","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286291+0000","last_change":"2024-02-13T00:52:37.949853+0000","last_active":"2024-02-13T00:52:42.286291+0000","last_peered":"2024-02-13T00:52:42.286291+0000","last_clean":"2024-02-13T00:52:42.286291+0000","last_became_active":"2024-02-13T00:52:37.949542+0000","last_became_peered":"2024-02-13T00:52:37.949542+0000","last_unstale":"2024-02-13T00:52:42.286291+0000","last_undegraded":"2024-02-13T00:52:42.286291+0000","last_fullsized":"2024-02-13T00:52:42.286291+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.a","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285163+0000","last_change":"2024-02-13T00:52:37.583736+0000","last_active":"2024-02-13T00:52:42.285163+0000","last_peered":"2024-02-13T00:52:42.285163+0000","last_clean":"2024-02-13T00:52:42.285163+0000","last_became_active":"2024-02-13T00:52:37.583524+0000","last_became_peered":"2024-02-13T00:52:37.583524+0000","last_unstale":"2024-02-13T00:52:42.285163+0000","last_undegraded":"2024-02-13T00:52:42.285163+0000","last_fullsized":"2024-02-13T00:52:42.285163+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.9","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286479+0000","last_change":"2024-02-13T00:52:37.949835+0000","last_active":"2024-02-13T00:52:42.286479+0000","last_peered":"2024-02-13T00:52:42.286479+0000","last_clean":"2024-02-13T00:52:42.286479+0000","last_became_active":"2024-02-13T00:52:37.949000+0000","last_became_peered":"2024-02-13T00:52:37.949000+0000","last_unstale":"2024-02-13T00:52:42.286479+0000","last_undegraded":"2024-02-13T00:52:42.286479+0000","last_fullsized":"2024-02-13T00:52:42.286479+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.8","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599277+0000","last_change":"2024-02-13T00:52:37.586944+0000","last_active":"2024-02-13T00:52:41.599277+0000","last_peered":"2024-02-13T00:52:41.599277+0000","last_clean":"2024-02-13T00:52:41.599277+0000","last_became_active":"2024-02-13T00:52:37.586652+0000","last_became_peered":"2024-02-13T00:52:37.586652+0000","last_unstale":"2024-02-13T00:52:41.599277+0000","last_undegraded":"2024-02-13T00:52:41.599277+0000","last_fullsized":"2024-02-13T00:52:41.599277+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.7","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285273+0000","last_change":"2024-02-13T00:52:37.953125+0000","last_active":"2024-02-13T00:52:42.285273+0000","last_peered":"2024-02-13T00:52:42.285273+0000","last_clean":"2024-02-13T00:52:42.285273+0000","last_became_active":"2024-02-13T00:52:37.952793+0000","last_became_peered":"2024-02-13T00:52:37.952793+0000","last_unstale":"2024-02-13T00:52:42.285273+0000","last_undegraded":"2024-02-13T00:52:42.285273+0000","last_fullsized":"2024-02-13T00:52:42.285273+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.6","version":"735'10","reported_seq":23,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-02-13T00:52:48.577889+0000","last_change":"2024-02-13T00:52:37.951302+0000","last_active":"2024-02-13T00:52:48.577889+0000","last_peered":"2024-02-13T00:52:48.577889+0000","last_clean":"2024-02-13T00:52:48.577889+0000","last_became_active":"2024-02-13T00:52:37.950157+0000","last_became_peered":"2024-02-13T00:52:37.950157+0000","last_unstale":"2024-02-13T00:52:48.577889+0000","last_undegraded":"2024-02-13T00:52:48.577889+0000","last_fullsized":"2024-02-13T00:52:48.577889+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65704,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.5","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598332+0000","last_change":"2024-02-13T00:52:37.951170+0000","last_active":"2024-02-13T00:52:41.598332+0000","last_peered":"2024-02-13T00:52:41.598332+0000","last_clean":"2024-02-13T00:52:41.598332+0000","last_became_active":"2024-02-13T00:52:37.950522+0000","last_became_peered":"2024-02-13T00:52:37.950522+0000","last_unstale":"2024-02-13T00:52:41.598332+0000","last_undegraded":"2024-02-13T00:52:41.598332+0000","last_fullsized":"2024-02-13T00:52:41.598332+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.4","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598567+0000","last_change":"2024-02-13T00:52:37.950887+0000","last_active":"2024-02-13T00:52:41.598567+0000","last_peered":"2024-02-13T00:52:41.598567+0000","last_clean":"2024-02-13T00:52:41.598567+0000","last_became_active":"2024-02-13T00:52:37.950437+0000","last_became_peered":"2024-02-13T00:52:37.950437+0000","last_unstale":"2024-02-13T00:52:41.598567+0000","last_undegraded":"2024-02-13T00:52:41.598567+0000","last_fullsized":"2024-02-13T00:52:41.598567+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.3","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283343+0000","last_change":"2024-02-13T00:52:37.949869+0000","last_active":"2024-02-13T00:52:42.283343+0000","last_peered":"2024-02-13T00:52:42.283343+0000","last_clean":"2024-02-13T00:52:42.283343+0000","last_became_active":"2024-02-13T00:52:37.949536+0000","last_became_peered":"2024-02-13T00:52:37.949536+0000","last_unstale":"2024-02-13T00:52:42.283343+0000","last_undegraded":"2024-02-13T00:52:42.283343+0000","last_fullsized":"2024-02-13T00:52:42.283343+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.2","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283991+0000","last_change":"2024-02-13T00:52:37.583963+0000","last_active":"2024-02-13T00:52:42.283991+0000","last_peered":"2024-02-13T00:52:42.283991+0000","last_clean":"2024-02-13T00:52:42.283991+0000","last_became_active":"2024-02-13T00:52:37.583661+0000","last_became_peered":"2024-02-13T00:52:37.583661+0000","last_unstale":"2024-02-13T00:52:42.283991+0000","last_undegraded":"2024-02-13T00:52:42.283991+0000","last_fullsized":"2024-02-13T00:52:42.283991+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.1","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.596895+0000","last_change":"2024-02-13T00:52:37.948366+0000","last_active":"2024-02-13T00:52:41.596895+0000","last_peered":"2024-02-13T00:52:41.596895+0000","last_clean":"2024-02-13T00:52:41.596895+0000","last_became_active":"2024-02-13T00:52:37.947838+0000","last_became_peered":"2024-02-13T00:52:37.947838+0000","last_unstale":"2024-02-13T00:52:41.596895+0000","last_undegraded":"2024-02-13T00:52:41.596895+0000","last_fullsized":"2024-02-13T00:52:41.596895+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.0","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599866+0000","last_change":"2024-02-13T00:52:37.949359+0000","last_active":"2024-02-13T00:52:41.599866+0000","last_peered":"2024-02-13T00:52:41.599866+0000","last_clean":"2024-02-13T00:52:41.599866+0000","last_became_active":"2024-02-13T00:52:37.948671+0000","last_became_peered":"2024-02-13T00:52:37.948671+0000","last_unstale":"2024-02-13T00:52:41.599866+0000","last_undegraded":"2024-02-13T00:52:41.599866+0000","last_fullsized":"2024-02-13T00:52:41.599866+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.14","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597161+0000","last_change":"2024-02-13T00:52:37.950718+0000","last_active":"2024-02-13T00:52:41.597161+0000","last_peered":"2024-02-13T00:52:41.597161+0000","last_clean":"2024-02-13T00:52:41.597161+0000","last_became_active":"2024-02-13T00:52:37.950049+0000","last_became_peered":"2024-02-13T00:52:37.950049+0000","last_unstale":"2024-02-13T00:52:41.597161+0000","last_undegraded":"2024-02-13T00:52:41.597161+0000","last_fullsized":"2024-02-13T00:52:41.597161+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.7","version":"679'17","reported_seq":37,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.600837+0000","last_change":"2024-02-13T00:52:31.270066+0000","last_active":"2024-02-13T00:52:41.600837+0000","last_peered":"2024-02-13T00:52:41.600837+0000","last_clean":"2024-02-13T00:52:41.600837+0000","last_became_active":"2024-02-13T00:52:30.594798+0000","last_became_peered":"2024-02-13T00:52:30.594798+0000","last_unstale":"2024-02-13T00:52:41.600837+0000","last_undegraded":"2024-02-13T00:52:41.600837+0000","last_fullsized":"2024-02-13T00:52:41.600837+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.15","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283548+0000","last_change":"2024-02-13T00:52:37.954457+0000","last_active":"2024-02-13T00:52:42.283548+0000","last_peered":"2024-02-13T00:52:42.283548+0000","last_clean":"2024-02-13T00:52:42.283548+0000","last_became_active":"2024-02-13T00:52:37.954152+0000","last_became_peered":"2024-02-13T00:52:37.954152+0000","last_unstale":"2024-02-13T00:52:42.283548+0000","last_undegraded":"2024-02-13T00:52:42.283548+0000","last_fullsized":"2024-02-13T00:52:42.283548+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.6","version":"650'1","reported_seq":159,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285099+0000","last_change":"2024-02-13T00:52:07.570115+0000","last_active":"2024-02-13T00:52:42.285099+0000","last_peered":"2024-02-13T00:52:42.285099+0000","last_clean":"2024-02-13T00:52:42.285099+0000","last_became_active":"2024-02-13T00:50:14.552756+0000","last_became_peered":"2024-02-13T00:50:14.552756+0000","last_unstale":"2024-02-13T00:52:42.285099+0000","last_undegraded":"2024-02-13T00:52:42.285099+0000","last_fullsized":"2024-02-13T00:52:42.285099+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.16","version":"0'0","reported_seq":14,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557726+0000","last_change":"2024-02-13T00:52:37.954767+0000","last_active":"2024-02-13T00:52:41.557726+0000","last_peered":"2024-02-13T00:52:41.557726+0000","last_clean":"2024-02-13T00:52:41.557726+0000","last_became_active":"2024-02-13T00:52:37.954464+0000","last_became_peered":"2024-02-13T00:52:37.954464+0000","last_unstale":"2024-02-13T00:52:41.557726+0000","last_undegraded":"2024-02-13T00:52:41.557726+0000","last_fullsized":"2024-02-13T00:52:41.557726+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.5","version":"668'15","reported_seq":202,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283980+0000","last_change":"2024-02-13T00:52:34.332875+0000","last_active":"2024-02-13T00:52:42.283980+0000","last_peered":"2024-02-13T00:52:42.283980+0000","last_clean":"2024-02-13T00:52:42.283980+0000","last_became_active":"2024-02-13T00:52:30.595299+0000","last_became_peered":"2024-02-13T00:52:30.595299+0000","last_unstale":"2024-02-13T00:52:42.283980+0000","last_undegraded":"2024-02-13T00:52:42.283980+0000","last_fullsized":"2024-02-13T00:52:42.283980+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":24,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.17","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.600271+0000","last_change":"2024-02-13T00:52:37.950931+0000","last_active":"2024-02-13T00:52:41.600271+0000","last_peered":"2024-02-13T00:52:41.600271+0000","last_clean":"2024-02-13T00:52:41.600271+0000","last_became_active":"2024-02-13T00:52:37.950391+0000","last_became_peered":"2024-02-13T00:52:37.950391+0000","last_unstale":"2024-02-13T00:52:41.600271+0000","last_undegraded":"2024-02-13T00:52:41.600271+0000","last_fullsized":"2024-02-13T00:52:41.600271+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.4","version":"708'10","reported_seq":186,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597944+0000","last_change":"2024-02-13T00:52:07.557573+0000","last_active":"2024-02-13T00:52:41.597944+0000","last_peered":"2024-02-13T00:52:41.597944+0000","last_clean":"2024-02-13T00:52:41.597944+0000","last_became_active":"2024-02-13T00:51:41.580904+0000","last_became_peered":"2024-02-13T00:51:41.580904+0000","last_unstale":"2024-02-13T00:52:41.597944+0000","last_undegraded":"2024-02-13T00:52:41.597944+0000","last_fullsized":"2024-02-13T00:52:41.597944+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.10","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.281893+0000","last_change":"2024-02-13T00:52:37.952952+0000","last_active":"2024-02-13T00:52:42.281893+0000","last_peered":"2024-02-13T00:52:42.281893+0000","last_clean":"2024-02-13T00:52:42.281893+0000","last_became_active":"2024-02-13T00:52:37.952675+0000","last_became_peered":"2024-02-13T00:52:37.952675+0000","last_unstale":"2024-02-13T00:52:42.281893+0000","last_undegraded":"2024-02-13T00:52:42.281893+0000","last_fullsized":"2024-02-13T00:52:42.281893+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.3","version":"716'9","reported_seq":196,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.282710+0000","last_change":"2024-02-13T00:52:33.861253+0000","last_active":"2024-02-13T00:52:42.282710+0000","last_peered":"2024-02-13T00:52:42.282710+0000","last_clean":"2024-02-13T00:52:42.282710+0000","last_became_active":"2024-02-13T00:52:30.600633+0000","last_became_peered":"2024-02-13T00:52:30.600633+0000","last_unstale":"2024-02-13T00:52:42.282710+0000","last_undegraded":"2024-02-13T00:52:42.282710+0000","last_fullsized":"2024-02-13T00:52:42.282710+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000010,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":3910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4000010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.11","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.282046+0000","last_change":"2024-02-13T00:52:37.591564+0000","last_active":"2024-02-13T00:52:42.282046+0000","last_peered":"2024-02-13T00:52:42.282046+0000","last_clean":"2024-02-13T00:52:42.282046+0000","last_became_active":"2024-02-13T00:52:37.591342+0000","last_became_peered":"2024-02-13T00:52:37.591342+0000","last_unstale":"2024-02-13T00:52:42.282046+0000","last_undegraded":"2024-02-13T00:52:42.282046+0000","last_fullsized":"2024-02-13T00:52:42.282046+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.2","version":"660'15","reported_seq":202,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283002+0000","last_change":"2024-02-13T00:52:07.572797+0000","last_active":"2024-02-13T00:52:42.283002+0000","last_peered":"2024-02-13T00:52:42.283002+0000","last_clean":"2024-02-13T00:52:42.283002+0000","last_became_active":"2024-02-13T00:51:41.569720+0000","last_became_peered":"2024-02-13T00:51:41.569720+0000","last_unstale":"2024-02-13T00:52:42.283002+0000","last_undegraded":"2024-02-13T00:52:42.283002+0000","last_fullsized":"2024-02-13T00:52:42.283002+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.12","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.282311+0000","last_change":"2024-02-13T00:52:37.949878+0000","last_active":"2024-02-13T00:52:42.282311+0000","last_peered":"2024-02-13T00:52:42.282311+0000","last_clean":"2024-02-13T00:52:42.282311+0000","last_became_active":"2024-02-13T00:52:37.949660+0000","last_became_peered":"2024-02-13T00:52:37.949660+0000","last_unstale":"2024-02-13T00:52:42.282311+0000","last_undegraded":"2024-02-13T00:52:42.282311+0000","last_fullsized":"2024-02-13T00:52:42.282311+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.1","version":"0'0","reported_seq":172,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.554694+0000","last_change":"2024-02-13T00:52:30.605200+0000","last_active":"2024-02-13T00:52:41.554694+0000","last_peered":"2024-02-13T00:52:41.554694+0000","last_clean":"2024-02-13T00:52:41.554694+0000","last_became_active":"2024-02-13T00:52:30.603757+0000","last_became_peered":"2024-02-13T00:52:30.603757+0000","last_unstale":"2024-02-13T00:52:41.554694+0000","last_undegraded":"2024-02-13T00:52:41.554694+0000","last_fullsized":"2024-02-13T00:52:41.554694+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"249.13","version":"0'0","reported_seq":13,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.282526+0000","last_change":"2024-02-13T00:52:37.952830+0000","last_active":"2024-02-13T00:52:42.282526+0000","last_peered":"2024-02-13T00:52:42.282526+0000","last_clean":"2024-02-13T00:52:42.282526+0000","last_became_active":"2024-02-13T00:52:37.952510+0000","last_became_peered":"2024-02-13T00:52:37.952510+0000","last_unstale":"2024-02-13T00:52:42.282526+0000","last_undegraded":"2024-02-13T00:52:42.282526+0000","last_fullsized":"2024-02-13T00:52:42.282526+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":729,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:36.564517+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.564517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.0","version":"687'3","reported_seq":180,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.282245+0000","last_change":"2024-02-13T00:52:31.907499+0000","last_active":"2024-02-13T00:52:42.282245+0000","last_peered":"2024-02-13T00:52:42.282245+0000","last_clean":"2024-02-13T00:52:42.282245+0000","last_became_active":"2024-02-13T00:52:30.580474+0000","last_became_peered":"2024-02-13T00:52:30.580474+0000","last_unstale":"2024-02-13T00:52:42.282245+0000","last_undegraded":"2024-02-13T00:52:42.282245+0000","last_fullsized":"2024-02-13T00:52:42.282245+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":646,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597753+0000","last_change":"2024-02-13T00:51:41.586442+0000","last_active":"2024-02-13T00:52:41.597753+0000","last_peered":"2024-02-13T00:52:41.597753+0000","last_clean":"2024-02-13T00:52:41.597753+0000","last_became_active":"2024-02-13T00:51:41.585448+0000","last_became_peered":"2024-02-13T00:51:41.585448+0000","last_unstale":"2024-02-13T00:52:41.597753+0000","last_undegraded":"2024-02-13T00:52:41.597753+0000","last_fullsized":"2024-02-13T00:52:41.597753+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:32.100607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:50:32.100607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":568,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283521+0000","last_change":"2024-02-13T00:51:41.586812+0000","last_active":"2024-02-13T00:52:42.283521+0000","last_peered":"2024-02-13T00:52:42.283521+0000","last_clean":"2024-02-13T00:52:42.283521+0000","last_became_active":"2024-02-13T00:51:41.584445+0000","last_became_peered":"2024-02-13T00:51:41.584445+0000","last_unstale":"2024-02-13T00:52:42.283521+0000","last_undegraded":"2024-02-13T00:52:42.283521+0000","last_fullsized":"2024-02-13T00:52:42.283521+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:39.963625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:39.963625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":448,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.554795+0000","last_change":"2024-02-13T00:52:30.605704+0000","last_active":"2024-02-13T00:52:41.554795+0000","last_peered":"2024-02-13T00:52:41.554795+0000","last_clean":"2024-02-13T00:52:41.554795+0000","last_became_active":"2024-02-13T00:52:30.604888+0000","last_became_peered":"2024-02-13T00:52:30.604888+0000","last_unstale":"2024-02-13T00:52:41.554795+0000","last_undegraded":"2024-02-13T00:52:41.554795+0000","last_fullsized":"2024-02-13T00:52:41.554795+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:15.530059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:15.530059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":524,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.554258+0000","last_change":"2024-02-13T00:51:41.591561+0000","last_active":"2024-02-13T00:52:41.554258+0000","last_peered":"2024-02-13T00:52:41.554258+0000","last_clean":"2024-02-13T00:52:41.554258+0000","last_became_active":"2024-02-13T00:51:41.588863+0000","last_became_peered":"2024-02-13T00:51:41.588863+0000","last_unstale":"2024-02-13T00:52:41.554258+0000","last_undegraded":"2024-02-13T00:52:41.554258+0000","last_fullsized":"2024-02-13T00:52:41.554258+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:16.677431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:16.677431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":424,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285492+0000","last_change":"2024-02-13T00:52:30.601175+0000","last_active":"2024-02-13T00:52:42.285492+0000","last_peered":"2024-02-13T00:52:42.285492+0000","last_clean":"2024-02-13T00:52:42.285492+0000","last_became_active":"2024-02-13T00:52:30.600551+0000","last_became_peered":"2024-02-13T00:52:30.600551+0000","last_unstale":"2024-02-13T00:52:42.285492+0000","last_undegraded":"2024-02-13T00:52:42.285492+0000","last_fullsized":"2024-02-13T00:52:42.285492+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.469431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.469431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":21,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.282685+0000","last_change":"2024-02-13T00:52:30.571356+0000","last_active":"2024-02-13T00:52:42.282685+0000","last_peered":"2024-02-13T00:52:42.282685+0000","last_clean":"2024-02-13T00:52:42.282685+0000","last_became_active":"2024-02-13T00:52:30.571021+0000","last_became_peered":"2024-02-13T00:52:30.571021+0000","last_unstale":"2024-02-13T00:52:42.282685+0000","last_undegraded":"2024-02-13T00:52:42.282685+0000","last_fullsized":"2024-02-13T00:52:42.282685+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:54.542201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:54.542201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":433,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.281482+0000","last_change":"2024-02-13T00:51:40.927523+0000","last_active":"2024-02-13T00:52:42.281482+0000","last_peered":"2024-02-13T00:52:42.281482+0000","last_clean":"2024-02-13T00:52:42.281482+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:52:42.281482+0000","last_undegraded":"2024-02-13T00:52:42.281482+0000","last_fullsized":"2024-02-13T00:52:42.281482+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:40.927454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:40.927454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":696,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283782+0000","last_change":"2024-02-13T00:51:41.571583+0000","last_active":"2024-02-13T00:52:42.283782+0000","last_peered":"2024-02-13T00:52:42.283782+0000","last_clean":"2024-02-13T00:52:42.283782+0000","last_became_active":"2024-02-13T00:51:41.570041+0000","last_became_peered":"2024-02-13T00:51:41.570041+0000","last_unstale":"2024-02-13T00:52:42.283782+0000","last_undegraded":"2024-02-13T00:52:42.283782+0000","last_fullsized":"2024-02-13T00:52:42.283782+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":415,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286845+0000","last_change":"2024-02-13T00:49:48.070260+0000","last_active":"2024-02-13T00:52:42.286845+0000","last_peered":"2024-02-13T00:52:42.286845+0000","last_clean":"2024-02-13T00:52:42.286845+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:52:42.286845+0000","last_undegraded":"2024-02-13T00:52:42.286845+0000","last_fullsized":"2024-02-13T00:52:42.286845+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:48.070195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:49:48.070195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":515,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557906+0000","last_change":"2024-02-13T00:51:41.569866+0000","last_active":"2024-02-13T00:52:41.557906+0000","last_peered":"2024-02-13T00:52:41.557906+0000","last_clean":"2024-02-13T00:52:41.557906+0000","last_became_active":"2024-02-13T00:51:41.567986+0000","last_became_peered":"2024-02-13T00:51:41.567986+0000","last_unstale":"2024-02-13T00:52:41.557906+0000","last_undegraded":"2024-02-13T00:52:41.557906+0000","last_fullsized":"2024-02-13T00:52:41.557906+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:14.521241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:14.521241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":665,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599776+0000","last_change":"2024-02-13T00:52:30.593903+0000","last_active":"2024-02-13T00:52:41.599776+0000","last_peered":"2024-02-13T00:52:41.599776+0000","last_clean":"2024-02-13T00:52:41.599776+0000","last_became_active":"2024-02-13T00:52:30.593515+0000","last_became_peered":"2024-02-13T00:52:30.593515+0000","last_unstale":"2024-02-13T00:52:41.599776+0000","last_undegraded":"2024-02-13T00:52:41.599776+0000","last_fullsized":"2024-02-13T00:52:41.599776+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:49.069014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:49.069014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"728'22","reported_seq":605,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-02-13T00:52:44.384803+0000","last_change":"2024-02-13T00:52:35.583266+0000","last_active":"2024-02-13T00:52:44.384803+0000","last_peered":"2024-02-13T00:52:44.384803+0000","last_clean":"2024-02-13T00:52:44.384803+0000","last_became_active":"2024-02-13T00:52:35.581849+0000","last_became_peered":"2024-02-13T00:52:35.581849+0000","last_unstale":"2024-02-13T00:52:44.384803+0000","last_undegraded":"2024-02-13T00:52:44.384803+0000","last_fullsized":"2024-02-13T00:52:44.384803+0000","mapping_epoch":727,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:45:53.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.115260+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":812,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283373+0000","last_change":"2024-02-13T00:52:32.169701+0000","last_active":"2024-02-13T00:52:42.283373+0000","last_peered":"2024-02-13T00:52:42.283373+0000","last_clean":"2024-02-13T00:52:42.283373+0000","last_became_active":"2024-02-13T00:52:30.577711+0000","last_became_peered":"2024-02-13T00:52:30.577711+0000","last_unstale":"2024-02-13T00:52:42.283373+0000","last_undegraded":"2024-02-13T00:52:42.283373+0000","last_fullsized":"2024-02-13T00:52:42.283373+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:47:03.491647+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":733,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285292+0000","last_change":"2024-02-13T00:51:41.571304+0000","last_active":"2024-02-13T00:52:42.285292+0000","last_peered":"2024-02-13T00:52:42.285292+0000","last_clean":"2024-02-13T00:52:42.285292+0000","last_became_active":"2024-02-13T00:51:41.569518+0000","last_became_peered":"2024-02-13T00:51:41.569518+0000","last_unstale":"2024-02-13T00:52:42.285292+0000","last_undegraded":"2024-02-13T00:52:42.285292+0000","last_fullsized":"2024-02-13T00:52:42.285292+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:23.379773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:23.379773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":630,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.281354+0000","last_change":"2024-02-13T00:47:34.332791+0000","last_active":"2024-02-13T00:52:42.281354+0000","last_peered":"2024-02-13T00:52:42.281354+0000","last_clean":"2024-02-13T00:52:42.281354+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:52:42.281354+0000","last_undegraded":"2024-02-13T00:52:42.281354+0000","last_fullsized":"2024-02-13T00:52:42.281354+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:34.332704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:34.332704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":682,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285241+0000","last_change":"2024-02-13T00:52:30.190029+0000","last_active":"2024-02-13T00:52:42.285241+0000","last_peered":"2024-02-13T00:52:42.285241+0000","last_clean":"2024-02-13T00:52:42.285241+0000","last_became_active":"2024-02-13T00:51:41.573640+0000","last_became_peered":"2024-02-13T00:51:41.573640+0000","last_unstale":"2024-02-13T00:52:42.285241+0000","last_undegraded":"2024-02-13T00:52:42.285241+0000","last_fullsized":"2024-02-13T00:52:42.285241+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:30.189925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:30.189925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":756,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598385+0000","last_change":"2024-02-13T00:52:31.041126+0000","last_active":"2024-02-13T00:52:41.598385+0000","last_peered":"2024-02-13T00:52:41.598385+0000","last_clean":"2024-02-13T00:52:41.598385+0000","last_became_active":"2024-02-13T00:52:30.591236+0000","last_became_peered":"2024-02-13T00:52:30.591236+0000","last_unstale":"2024-02-13T00:52:41.598385+0000","last_undegraded":"2024-02-13T00:52:41.598385+0000","last_fullsized":"2024-02-13T00:52:41.598385+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'21","last_scrub_stamp":"2024-02-13T00:52:28.988459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:28.988459+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":725,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557731+0000","last_change":"2024-02-13T00:52:30.604523+0000","last_active":"2024-02-13T00:52:41.557731+0000","last_peered":"2024-02-13T00:52:41.557731+0000","last_clean":"2024-02-13T00:52:41.557731+0000","last_became_active":"2024-02-13T00:52:30.603634+0000","last_became_peered":"2024-02-13T00:52:30.603634+0000","last_unstale":"2024-02-13T00:52:41.557731+0000","last_undegraded":"2024-02-13T00:52:41.557731+0000","last_fullsized":"2024-02-13T00:52:41.557731+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:24.823107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.823107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":599,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597879+0000","last_change":"2024-02-13T00:52:30.587164+0000","last_active":"2024-02-13T00:52:41.597879+0000","last_peered":"2024-02-13T00:52:41.597879+0000","last_clean":"2024-02-13T00:52:41.597879+0000","last_became_active":"2024-02-13T00:52:30.586877+0000","last_became_peered":"2024-02-13T00:52:30.586877+0000","last_unstale":"2024-02-13T00:52:41.597879+0000","last_undegraded":"2024-02-13T00:52:41.597879+0000","last_fullsized":"2024-02-13T00:52:41.597879+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.788783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.788783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":722,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599496+0000","last_change":"2024-02-13T00:51:41.586644+0000","last_active":"2024-02-13T00:52:41.599496+0000","last_peered":"2024-02-13T00:52:41.599496+0000","last_clean":"2024-02-13T00:52:41.599496+0000","last_became_active":"2024-02-13T00:51:41.585885+0000","last_became_peered":"2024-02-13T00:51:41.585885+0000","last_unstale":"2024-02-13T00:52:41.599496+0000","last_undegraded":"2024-02-13T00:52:41.599496+0000","last_fullsized":"2024-02-13T00:52:41.599496+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.837810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.837810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":348,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597242+0000","last_change":"2024-02-13T00:52:30.588176+0000","last_active":"2024-02-13T00:52:41.597242+0000","last_peered":"2024-02-13T00:52:41.597242+0000","last_clean":"2024-02-13T00:52:41.597242+0000","last_became_active":"2024-02-13T00:52:30.587089+0000","last_became_peered":"2024-02-13T00:52:30.587089+0000","last_unstale":"2024-02-13T00:52:41.597242+0000","last_undegraded":"2024-02-13T00:52:41.597242+0000","last_fullsized":"2024-02-13T00:52:41.597242+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:34.184091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:34.184091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":554,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.284754+0000","last_change":"2024-02-13T00:51:43.825729+0000","last_active":"2024-02-13T00:52:42.284754+0000","last_peered":"2024-02-13T00:52:42.284754+0000","last_clean":"2024-02-13T00:52:42.284754+0000","last_became_active":"2024-02-13T00:51:41.573050+0000","last_became_peered":"2024-02-13T00:51:41.573050+0000","last_unstale":"2024-02-13T00:52:42.284754+0000","last_undegraded":"2024-02-13T00:52:42.284754+0000","last_fullsized":"2024-02-13T00:52:42.284754+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'51","last_scrub_stamp":"2024-02-13T00:51:33.100282+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:51:33.100282+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597534+0000","last_change":"2024-02-13T00:52:30.594905+0000","last_active":"2024-02-13T00:52:41.597534+0000","last_peered":"2024-02-13T00:52:41.597534+0000","last_clean":"2024-02-13T00:52:41.597534+0000","last_became_active":"2024-02-13T00:52:30.594543+0000","last_became_peered":"2024-02-13T00:52:30.594543+0000","last_unstale":"2024-02-13T00:52:41.597534+0000","last_undegraded":"2024-02-13T00:52:41.597534+0000","last_fullsized":"2024-02-13T00:52:41.597534+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:51.052139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:51:51.052139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":458,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598103+0000","last_change":"2024-02-13T00:51:41.587744+0000","last_active":"2024-02-13T00:52:41.598103+0000","last_peered":"2024-02-13T00:52:41.598103+0000","last_clean":"2024-02-13T00:52:41.598103+0000","last_became_active":"2024-02-13T00:51:41.582908+0000","last_became_peered":"2024-02-13T00:51:41.582908+0000","last_unstale":"2024-02-13T00:52:41.598103+0000","last_undegraded":"2024-02-13T00:52:41.598103+0000","last_fullsized":"2024-02-13T00:52:41.598103+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:14.244950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:51:14.244950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":560,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283675+0000","last_change":"2024-02-13T00:51:41.586157+0000","last_active":"2024-02-13T00:52:42.283675+0000","last_peered":"2024-02-13T00:52:42.283675+0000","last_clean":"2024-02-13T00:52:42.283675+0000","last_became_active":"2024-02-13T00:51:41.583542+0000","last_became_peered":"2024-02-13T00:51:41.583542+0000","last_unstale":"2024-02-13T00:52:42.283675+0000","last_undegraded":"2024-02-13T00:52:42.283675+0000","last_fullsized":"2024-02-13T00:52:42.283675+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:15.285536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:15.285536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":661,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557549+0000","last_change":"2024-02-13T00:51:35.505365+0000","last_active":"2024-02-13T00:52:41.557549+0000","last_peered":"2024-02-13T00:52:41.557549+0000","last_clean":"2024-02-13T00:52:41.557549+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:52:41.557549+0000","last_undegraded":"2024-02-13T00:52:41.557549+0000","last_fullsized":"2024-02-13T00:52:41.557549+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:35.505303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:35.505303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"467'31","reported_seq":520,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557600+0000","last_change":"2024-02-13T00:52:31.370996+0000","last_active":"2024-02-13T00:52:41.557600+0000","last_peered":"2024-02-13T00:52:41.557600+0000","last_clean":"2024-02-13T00:52:41.557600+0000","last_became_active":"2024-02-13T00:52:30.589869+0000","last_became_peered":"2024-02-13T00:52:30.589869+0000","last_unstale":"2024-02-13T00:52:41.557600+0000","last_undegraded":"2024-02-13T00:52:41.557600+0000","last_fullsized":"2024-02-13T00:52:41.557600+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'31","last_scrub_stamp":"2024-02-13T00:48:31.782729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.782729+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":532,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.282946+0000","last_change":"2024-02-13T00:52:33.259627+0000","last_active":"2024-02-13T00:52:42.282946+0000","last_peered":"2024-02-13T00:52:42.282946+0000","last_clean":"2024-02-13T00:52:42.282946+0000","last_became_active":"2024-02-13T00:52:30.596010+0000","last_became_peered":"2024-02-13T00:52:30.596010+0000","last_unstale":"2024-02-13T00:52:42.282946+0000","last_undegraded":"2024-02-13T00:52:42.282946+0000","last_fullsized":"2024-02-13T00:52:42.282946+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:52:25.149943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:52:25.149943+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":723,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557273+0000","last_change":"2024-02-13T00:52:30.589321+0000","last_active":"2024-02-13T00:52:41.557273+0000","last_peered":"2024-02-13T00:52:41.557273+0000","last_clean":"2024-02-13T00:52:41.557273+0000","last_became_active":"2024-02-13T00:52:30.588539+0000","last_became_peered":"2024-02-13T00:52:30.588539+0000","last_unstale":"2024-02-13T00:52:41.557273+0000","last_undegraded":"2024-02-13T00:52:41.557273+0000","last_fullsized":"2024-02-13T00:52:41.557273+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.793634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":709,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286562+0000","last_change":"2024-02-13T00:52:30.600848+0000","last_active":"2024-02-13T00:52:42.286562+0000","last_peered":"2024-02-13T00:52:42.286562+0000","last_clean":"2024-02-13T00:52:42.286562+0000","last_became_active":"2024-02-13T00:52:30.600493+0000","last_became_peered":"2024-02-13T00:52:30.600493+0000","last_unstale":"2024-02-13T00:52:42.286562+0000","last_undegraded":"2024-02-13T00:52:42.286562+0000","last_fullsized":"2024-02-13T00:52:42.286562+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:38.996288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:38.996288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":374,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598261+0000","last_change":"2024-02-13T00:52:30.597329+0000","last_active":"2024-02-13T00:52:41.598261+0000","last_peered":"2024-02-13T00:52:41.598261+0000","last_clean":"2024-02-13T00:52:41.598261+0000","last_became_active":"2024-02-13T00:52:30.595363+0000","last_became_peered":"2024-02-13T00:52:30.595363+0000","last_unstale":"2024-02-13T00:52:41.598261+0000","last_undegraded":"2024-02-13T00:52:41.598261+0000","last_fullsized":"2024-02-13T00:52:41.598261+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.925604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.925604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":463,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285026+0000","last_change":"2024-02-13T00:52:32.337869+0000","last_active":"2024-02-13T00:52:42.285026+0000","last_peered":"2024-02-13T00:52:42.285026+0000","last_clean":"2024-02-13T00:52:42.285026+0000","last_became_active":"2024-02-13T00:52:30.584756+0000","last_became_peered":"2024-02-13T00:52:30.584756+0000","last_unstale":"2024-02-13T00:52:42.285026+0000","last_undegraded":"2024-02-13T00:52:42.285026+0000","last_fullsized":"2024-02-13T00:52:42.285026+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:49:03.294499+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:49:03.294499+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":734,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.556102+0000","last_change":"2024-02-13T00:51:41.573970+0000","last_active":"2024-02-13T00:52:41.556102+0000","last_peered":"2024-02-13T00:52:41.556102+0000","last_clean":"2024-02-13T00:52:41.556102+0000","last_became_active":"2024-02-13T00:51:41.569525+0000","last_became_peered":"2024-02-13T00:51:41.569525+0000","last_unstale":"2024-02-13T00:52:41.556102+0000","last_undegraded":"2024-02-13T00:52:41.556102+0000","last_fullsized":"2024-02-13T00:52:41.556102+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:59.111828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:59.111828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"726'19","reported_seq":526,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-02-13T00:52:44.384599+0000","last_change":"2024-02-13T00:52:33.561456+0000","last_active":"2024-02-13T00:52:44.384599+0000","last_peered":"2024-02-13T00:52:44.384599+0000","last_clean":"2024-02-13T00:52:44.384599+0000","last_became_active":"2024-02-13T00:52:33.561139+0000","last_became_peered":"2024-02-13T00:52:33.561139+0000","last_unstale":"2024-02-13T00:52:44.384599+0000","last_undegraded":"2024-02-13T00:52:44.384599+0000","last_fullsized":"2024-02-13T00:52:44.384599+0000","mapping_epoch":725,"log_start":"404'11","ondisk_log_start":"404'11","created":60,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'16","last_scrub_stamp":"2024-02-13T00:52:22.988904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:52:22.988904+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":723,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.558070+0000","last_change":"2024-02-13T00:52:31.529160+0000","last_active":"2024-02-13T00:52:41.558070+0000","last_peered":"2024-02-13T00:52:41.558070+0000","last_clean":"2024-02-13T00:52:41.558070+0000","last_became_active":"2024-02-13T00:52:30.602716+0000","last_became_peered":"2024-02-13T00:52:30.602716+0000","last_unstale":"2024-02-13T00:52:41.558070+0000","last_undegraded":"2024-02-13T00:52:41.558070+0000","last_fullsized":"2024-02-13T00:52:41.558070+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:51:05.644558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:05.644558+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":524,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.558048+0000","last_change":"2024-02-13T00:52:30.605308+0000","last_active":"2024-02-13T00:52:41.558048+0000","last_peered":"2024-02-13T00:52:41.558048+0000","last_clean":"2024-02-13T00:52:41.558048+0000","last_became_active":"2024-02-13T00:52:30.604615+0000","last_became_peered":"2024-02-13T00:52:30.604615+0000","last_unstale":"2024-02-13T00:52:41.558048+0000","last_undegraded":"2024-02-13T00:52:41.558048+0000","last_fullsized":"2024-02-13T00:52:41.558048+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:26.830270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:26.830270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"467'27","reported_seq":864,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.558024+0000","last_change":"2024-02-13T00:52:32.788019+0000","last_active":"2024-02-13T00:52:41.558024+0000","last_peered":"2024-02-13T00:52:41.558024+0000","last_clean":"2024-02-13T00:52:41.558024+0000","last_became_active":"2024-02-13T00:52:30.603369+0000","last_became_peered":"2024-02-13T00:52:30.603369+0000","last_unstale":"2024-02-13T00:52:41.558024+0000","last_undegraded":"2024-02-13T00:52:41.558024+0000","last_fullsized":"2024-02-13T00:52:41.558024+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:48:54.043088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.043088+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":770,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.555068+0000","last_change":"2024-02-13T00:51:43.912749+0000","last_active":"2024-02-13T00:52:41.555068+0000","last_peered":"2024-02-13T00:52:41.555068+0000","last_clean":"2024-02-13T00:52:41.555068+0000","last_became_active":"2024-02-13T00:51:41.588865+0000","last_became_peered":"2024-02-13T00:51:41.588865+0000","last_unstale":"2024-02-13T00:52:41.555068+0000","last_undegraded":"2024-02-13T00:52:41.555068+0000","last_fullsized":"2024-02-13T00:52:41.555068+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:49:37.053398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.053398+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"727'19","reported_seq":531,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-02-13T00:52:44.384480+0000","last_change":"2024-02-13T00:52:34.567820+0000","last_active":"2024-02-13T00:52:44.384480+0000","last_peered":"2024-02-13T00:52:44.384480+0000","last_clean":"2024-02-13T00:52:44.384480+0000","last_became_active":"2024-02-13T00:52:34.566362+0000","last_became_peered":"2024-02-13T00:52:34.566362+0000","last_unstale":"2024-02-13T00:52:44.384480+0000","last_undegraded":"2024-02-13T00:52:44.384480+0000","last_fullsized":"2024-02-13T00:52:44.384480+0000","mapping_epoch":726,"log_start":"467'11","ondisk_log_start":"467'11","created":60,"last_epoch_clean":727,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:47:49.687727+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":347,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557295+0000","last_change":"2024-02-13T00:50:19.734411+0000","last_active":"2024-02-13T00:52:41.557295+0000","last_peered":"2024-02-13T00:52:41.557295+0000","last_clean":"2024-02-13T00:52:41.557295+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:52:41.557295+0000","last_undegraded":"2024-02-13T00:52:41.557295+0000","last_fullsized":"2024-02-13T00:52:41.557295+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:19.734320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:50:19.734320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":656,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.596498+0000","last_change":"2024-02-13T00:51:11.239012+0000","last_active":"2024-02-13T00:52:41.596498+0000","last_peered":"2024-02-13T00:52:41.596498+0000","last_clean":"2024-02-13T00:52:41.596498+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:52:41.596498+0000","last_undegraded":"2024-02-13T00:52:41.596498+0000","last_fullsized":"2024-02-13T00:52:41.596498+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.238914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.238914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":464,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286225+0000","last_change":"2024-02-13T00:52:32.853997+0000","last_active":"2024-02-13T00:52:42.286225+0000","last_peered":"2024-02-13T00:52:42.286225+0000","last_clean":"2024-02-13T00:52:42.286225+0000","last_became_active":"2024-02-13T00:52:30.593631+0000","last_became_peered":"2024-02-13T00:52:30.593631+0000","last_unstale":"2024-02-13T00:52:42.286225+0000","last_undegraded":"2024-02-13T00:52:42.286225+0000","last_fullsized":"2024-02-13T00:52:42.286225+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:57.386254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:57.386254+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":548,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599482+0000","last_change":"2024-02-13T00:51:53.090015+0000","last_active":"2024-02-13T00:52:41.599482+0000","last_peered":"2024-02-13T00:52:41.599482+0000","last_clean":"2024-02-13T00:52:41.599482+0000","last_became_active":"2024-02-13T00:51:41.582606+0000","last_became_peered":"2024-02-13T00:51:41.582606+0000","last_unstale":"2024-02-13T00:52:41.599482+0000","last_undegraded":"2024-02-13T00:52:41.599482+0000","last_fullsized":"2024-02-13T00:52:41.599482+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:53.089946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:51:53.089946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"728'19","reported_seq":732,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-02-13T00:52:44.384812+0000","last_change":"2024-02-13T00:52:35.610840+0000","last_active":"2024-02-13T00:52:44.384812+0000","last_peered":"2024-02-13T00:52:44.384812+0000","last_clean":"2024-02-13T00:52:44.384812+0000","last_became_active":"2024-02-13T00:52:35.610249+0000","last_became_peered":"2024-02-13T00:52:35.610249+0000","last_unstale":"2024-02-13T00:52:44.384812+0000","last_undegraded":"2024-02-13T00:52:44.384812+0000","last_fullsized":"2024-02-13T00:52:44.384812+0000","mapping_epoch":727,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'16","last_scrub_stamp":"2024-02-13T00:49:45.051539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:49:45.051539+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":442,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283874+0000","last_change":"2024-02-13T00:51:41.570899+0000","last_active":"2024-02-13T00:52:42.283874+0000","last_peered":"2024-02-13T00:52:42.283874+0000","last_clean":"2024-02-13T00:52:42.283874+0000","last_became_active":"2024-02-13T00:51:41.568351+0000","last_became_peered":"2024-02-13T00:51:41.568351+0000","last_unstale":"2024-02-13T00:52:42.283874+0000","last_undegraded":"2024-02-13T00:52:42.283874+0000","last_fullsized":"2024-02-13T00:52:42.283874+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:31.463442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.463442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":626,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597291+0000","last_change":"2024-02-13T00:52:30.595757+0000","last_active":"2024-02-13T00:52:41.597291+0000","last_peered":"2024-02-13T00:52:41.597291+0000","last_clean":"2024-02-13T00:52:41.597291+0000","last_became_active":"2024-02-13T00:52:30.594189+0000","last_became_peered":"2024-02-13T00:52:30.594189+0000","last_unstale":"2024-02-13T00:52:41.597291+0000","last_undegraded":"2024-02-13T00:52:41.597291+0000","last_fullsized":"2024-02-13T00:52:41.597291+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:00.881431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:00.881431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"726'11","reported_seq":778,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-02-13T00:52:44.384479+0000","last_change":"2024-02-13T00:52:31.384662+0000","last_active":"2024-02-13T00:52:44.384479+0000","last_peered":"2024-02-13T00:52:44.384479+0000","last_clean":"2024-02-13T00:52:44.384479+0000","last_became_active":"2024-02-13T00:52:30.582279+0000","last_became_peered":"2024-02-13T00:52:30.582279+0000","last_unstale":"2024-02-13T00:52:44.384479+0000","last_undegraded":"2024-02-13T00:52:44.384479+0000","last_fullsized":"2024-02-13T00:52:44.384479+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.949406+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":732,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283027+0000","last_change":"2024-02-13T00:52:31.298166+0000","last_active":"2024-02-13T00:52:42.283027+0000","last_peered":"2024-02-13T00:52:42.283027+0000","last_clean":"2024-02-13T00:52:42.283027+0000","last_became_active":"2024-02-13T00:52:30.579027+0000","last_became_peered":"2024-02-13T00:52:30.579027+0000","last_unstale":"2024-02-13T00:52:42.283027+0000","last_undegraded":"2024-02-13T00:52:42.283027+0000","last_fullsized":"2024-02-13T00:52:42.283027+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'34","last_scrub_stamp":"2024-02-13T00:49:46.101874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.101874+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":442,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286780+0000","last_change":"2024-02-13T00:51:41.572212+0000","last_active":"2024-02-13T00:52:42.286780+0000","last_peered":"2024-02-13T00:52:42.286780+0000","last_clean":"2024-02-13T00:52:42.286780+0000","last_became_active":"2024-02-13T00:51:41.569996+0000","last_became_peered":"2024-02-13T00:51:41.569996+0000","last_unstale":"2024-02-13T00:52:42.286780+0000","last_undegraded":"2024-02-13T00:52:42.286780+0000","last_fullsized":"2024-02-13T00:52:42.286780+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:47.075741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.075741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":442,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286252+0000","last_change":"2024-02-13T00:51:41.572697+0000","last_active":"2024-02-13T00:52:42.286252+0000","last_peered":"2024-02-13T00:52:42.286252+0000","last_clean":"2024-02-13T00:52:42.286252+0000","last_became_active":"2024-02-13T00:51:41.569841+0000","last_became_peered":"2024-02-13T00:51:41.569841+0000","last_unstale":"2024-02-13T00:52:42.286252+0000","last_undegraded":"2024-02-13T00:52:42.286252+0000","last_fullsized":"2024-02-13T00:52:42.286252+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:17.268189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:17.268189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":467,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599379+0000","last_change":"2024-02-13T00:51:41.585096+0000","last_active":"2024-02-13T00:52:41.599379+0000","last_peered":"2024-02-13T00:52:41.599379+0000","last_clean":"2024-02-13T00:52:41.599379+0000","last_became_active":"2024-02-13T00:51:41.582942+0000","last_became_peered":"2024-02-13T00:51:41.582942+0000","last_unstale":"2024-02-13T00:52:41.599379+0000","last_undegraded":"2024-02-13T00:52:41.599379+0000","last_fullsized":"2024-02-13T00:52:41.599379+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:36.841508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:36.841508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":517,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597639+0000","last_change":"2024-02-13T00:52:30.595853+0000","last_active":"2024-02-13T00:52:41.597639+0000","last_peered":"2024-02-13T00:52:41.597639+0000","last_clean":"2024-02-13T00:52:41.597639+0000","last_became_active":"2024-02-13T00:52:30.594613+0000","last_became_peered":"2024-02-13T00:52:30.594613+0000","last_unstale":"2024-02-13T00:52:41.597639+0000","last_undegraded":"2024-02-13T00:52:41.597639+0000","last_fullsized":"2024-02-13T00:52:41.597639+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:18.751081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:18.751081+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"467'42","reported_seq":691,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599497+0000","last_change":"2024-02-13T00:51:42.397324+0000","last_active":"2024-02-13T00:52:41.599497+0000","last_peered":"2024-02-13T00:52:41.599497+0000","last_clean":"2024-02-13T00:52:41.599497+0000","last_became_active":"2024-02-13T00:51:41.582068+0000","last_became_peered":"2024-02-13T00:51:41.582068+0000","last_unstale":"2024-02-13T00:52:41.599497+0000","last_undegraded":"2024-02-13T00:52:41.599497+0000","last_fullsized":"2024-02-13T00:52:41.599497+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'42","last_scrub_stamp":"2024-02-13T00:48:24.675687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.675687+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":739,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557911+0000","last_change":"2024-02-13T00:52:30.582723+0000","last_active":"2024-02-13T00:52:41.557911+0000","last_peered":"2024-02-13T00:52:41.557911+0000","last_clean":"2024-02-13T00:52:41.557911+0000","last_became_active":"2024-02-13T00:52:30.582296+0000","last_became_peered":"2024-02-13T00:52:30.582296+0000","last_unstale":"2024-02-13T00:52:41.557911+0000","last_undegraded":"2024-02-13T00:52:41.557911+0000","last_fullsized":"2024-02-13T00:52:41.557911+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:29.412336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:52:29.412336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":681,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283514+0000","last_change":"2024-02-13T00:52:30.595054+0000","last_active":"2024-02-13T00:52:42.283514+0000","last_peered":"2024-02-13T00:52:42.283514+0000","last_clean":"2024-02-13T00:52:42.283514+0000","last_became_active":"2024-02-13T00:52:30.593405+0000","last_became_peered":"2024-02-13T00:52:30.593405+0000","last_unstale":"2024-02-13T00:52:42.283514+0000","last_undegraded":"2024-02-13T00:52:42.283514+0000","last_fullsized":"2024-02-13T00:52:42.283514+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":664,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285002+0000","last_change":"2024-02-13T00:52:30.586680+0000","last_active":"2024-02-13T00:52:42.285002+0000","last_peered":"2024-02-13T00:52:42.285002+0000","last_clean":"2024-02-13T00:52:42.285002+0000","last_became_active":"2024-02-13T00:52:30.586041+0000","last_became_peered":"2024-02-13T00:52:30.586041+0000","last_unstale":"2024-02-13T00:52:42.285002+0000","last_undegraded":"2024-02-13T00:52:42.285002+0000","last_fullsized":"2024-02-13T00:52:42.285002+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_clean_scrub_stamp":"2024-02-13T00:48:35.434645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":433,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.284776+0000","last_change":"2024-02-13T00:51:41.574980+0000","last_active":"2024-02-13T00:52:42.284776+0000","last_peered":"2024-02-13T00:52:42.284776+0000","last_clean":"2024-02-13T00:52:42.284776+0000","last_became_active":"2024-02-13T00:51:41.573763+0000","last_became_peered":"2024-02-13T00:51:41.573763+0000","last_unstale":"2024-02-13T00:52:42.284776+0000","last_undegraded":"2024-02-13T00:52:42.284776+0000","last_fullsized":"2024-02-13T00:52:42.284776+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:33.204969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:33.204969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":857,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598200+0000","last_change":"2024-02-13T00:52:26.954476+0000","last_active":"2024-02-13T00:52:41.598200+0000","last_peered":"2024-02-13T00:52:41.598200+0000","last_clean":"2024-02-13T00:52:41.598200+0000","last_became_active":"2024-02-13T00:51:41.587256+0000","last_became_peered":"2024-02-13T00:51:41.587256+0000","last_unstale":"2024-02-13T00:52:41.598200+0000","last_undegraded":"2024-02-13T00:52:41.598200+0000","last_fullsized":"2024-02-13T00:52:41.598200+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'44","last_scrub_stamp":"2024-02-13T00:52:26.954384+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:52:26.954384+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":557,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599713+0000","last_change":"2024-02-13T00:52:30.596233+0000","last_active":"2024-02-13T00:52:41.599713+0000","last_peered":"2024-02-13T00:52:41.599713+0000","last_clean":"2024-02-13T00:52:41.599713+0000","last_became_active":"2024-02-13T00:52:30.595886+0000","last_became_peered":"2024-02-13T00:52:30.595886+0000","last_unstale":"2024-02-13T00:52:41.599713+0000","last_undegraded":"2024-02-13T00:52:41.599713+0000","last_fullsized":"2024-02-13T00:52:41.599713+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:25.001219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:25.001219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":746,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.555116+0000","last_change":"2024-02-13T00:51:41.589718+0000","last_active":"2024-02-13T00:52:41.555116+0000","last_peered":"2024-02-13T00:52:41.555116+0000","last_clean":"2024-02-13T00:52:41.555116+0000","last_became_active":"2024-02-13T00:51:41.589295+0000","last_became_peered":"2024-02-13T00:51:41.589295+0000","last_unstale":"2024-02-13T00:52:41.555116+0000","last_undegraded":"2024-02-13T00:52:41.555116+0000","last_fullsized":"2024-02-13T00:52:41.555116+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.726660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.726660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":359,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557825+0000","last_change":"2024-02-13T00:52:30.582714+0000","last_active":"2024-02-13T00:52:41.557825+0000","last_peered":"2024-02-13T00:52:41.557825+0000","last_clean":"2024-02-13T00:52:41.557825+0000","last_became_active":"2024-02-13T00:52:30.582272+0000","last_became_peered":"2024-02-13T00:52:30.582272+0000","last_unstale":"2024-02-13T00:52:41.557825+0000","last_undegraded":"2024-02-13T00:52:41.557825+0000","last_fullsized":"2024-02-13T00:52:41.557825+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:55.050411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.050411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":374,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.558193+0000","last_change":"2024-02-13T00:52:30.605419+0000","last_active":"2024-02-13T00:52:41.558193+0000","last_peered":"2024-02-13T00:52:41.558193+0000","last_clean":"2024-02-13T00:52:41.558193+0000","last_became_active":"2024-02-13T00:52:30.604937+0000","last_became_peered":"2024-02-13T00:52:30.604937+0000","last_unstale":"2024-02-13T00:52:41.558193+0000","last_undegraded":"2024-02-13T00:52:41.558193+0000","last_fullsized":"2024-02-13T00:52:41.558193+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:43.903553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:43.903553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":846,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.601020+0000","last_change":"2024-02-13T00:52:23.951273+0000","last_active":"2024-02-13T00:52:41.601020+0000","last_peered":"2024-02-13T00:52:41.601020+0000","last_clean":"2024-02-13T00:52:41.601020+0000","last_became_active":"2024-02-13T00:51:41.588905+0000","last_became_peered":"2024-02-13T00:51:41.588905+0000","last_unstale":"2024-02-13T00:52:41.601020+0000","last_undegraded":"2024-02-13T00:52:41.601020+0000","last_fullsized":"2024-02-13T00:52:41.601020+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:52:23.951206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:23.951206+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":447,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.554472+0000","last_change":"2024-02-13T00:51:53.574669+0000","last_active":"2024-02-13T00:52:41.554472+0000","last_peered":"2024-02-13T00:52:41.554472+0000","last_clean":"2024-02-13T00:52:41.554472+0000","last_became_active":"2024-02-13T00:51:41.588531+0000","last_became_peered":"2024-02-13T00:51:41.588531+0000","last_unstale":"2024-02-13T00:52:41.554472+0000","last_undegraded":"2024-02-13T00:52:41.554472+0000","last_fullsized":"2024-02-13T00:52:41.554472+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:53.574605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:53.574605+0000","last_clean_scrub_stamp":"2024-02-13T00:51:53.574605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":733,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599311+0000","last_change":"2024-02-13T00:52:30.596032+0000","last_active":"2024-02-13T00:52:41.599311+0000","last_peered":"2024-02-13T00:52:41.599311+0000","last_clean":"2024-02-13T00:52:41.599311+0000","last_became_active":"2024-02-13T00:52:30.595531+0000","last_became_peered":"2024-02-13T00:52:30.595531+0000","last_unstale":"2024-02-13T00:52:41.599311+0000","last_undegraded":"2024-02-13T00:52:41.599311+0000","last_fullsized":"2024-02-13T00:52:41.599311+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:46.119431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:51:46.119431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":737,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557358+0000","last_change":"2024-02-13T00:52:30.589500+0000","last_active":"2024-02-13T00:52:41.557358+0000","last_peered":"2024-02-13T00:52:41.557358+0000","last_clean":"2024-02-13T00:52:41.557358+0000","last_became_active":"2024-02-13T00:52:30.588991+0000","last_became_peered":"2024-02-13T00:52:30.588991+0000","last_unstale":"2024-02-13T00:52:41.557358+0000","last_undegraded":"2024-02-13T00:52:41.557358+0000","last_fullsized":"2024-02-13T00:52:41.557358+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:16.535961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:51:16.535961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":584,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286744+0000","last_change":"2024-02-13T00:52:30.603205+0000","last_active":"2024-02-13T00:52:42.286744+0000","last_peered":"2024-02-13T00:52:42.286744+0000","last_clean":"2024-02-13T00:52:42.286744+0000","last_became_active":"2024-02-13T00:52:30.600547+0000","last_became_peered":"2024-02-13T00:52:30.600547+0000","last_unstale":"2024-02-13T00:52:42.286744+0000","last_undegraded":"2024-02-13T00:52:42.286744+0000","last_fullsized":"2024-02-13T00:52:42.286744+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:23.124083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:23.124083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":389,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598206+0000","last_change":"2024-02-13T00:52:30.017699+0000","last_active":"2024-02-13T00:52:41.598206+0000","last_peered":"2024-02-13T00:52:41.598206+0000","last_clean":"2024-02-13T00:52:41.598206+0000","last_became_active":"2024-02-13T00:51:41.580760+0000","last_became_peered":"2024-02-13T00:51:41.580760+0000","last_unstale":"2024-02-13T00:52:41.598206+0000","last_undegraded":"2024-02-13T00:52:41.598206+0000","last_fullsized":"2024-02-13T00:52:41.598206+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:30.017618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:52:30.017618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":524,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286842+0000","last_change":"2024-02-13T00:52:34.694390+0000","last_active":"2024-02-13T00:52:42.286842+0000","last_peered":"2024-02-13T00:52:42.286842+0000","last_clean":"2024-02-13T00:52:42.286842+0000","last_became_active":"2024-02-13T00:52:30.596789+0000","last_became_peered":"2024-02-13T00:52:30.596789+0000","last_unstale":"2024-02-13T00:52:42.286842+0000","last_undegraded":"2024-02-13T00:52:42.286842+0000","last_fullsized":"2024-02-13T00:52:42.286842+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":727,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:46:57.381540+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:46:57.381540+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":689,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597902+0000","last_change":"2024-02-13T00:52:36.042288+0000","last_active":"2024-02-13T00:52:41.597902+0000","last_peered":"2024-02-13T00:52:41.597902+0000","last_clean":"2024-02-13T00:52:41.597902+0000","last_became_active":"2024-02-13T00:51:41.582126+0000","last_became_peered":"2024-02-13T00:51:41.582126+0000","last_unstale":"2024-02-13T00:52:41.597902+0000","last_undegraded":"2024-02-13T00:52:41.597902+0000","last_fullsized":"2024-02-13T00:52:41.597902+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.042221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.042221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":681,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283161+0000","last_change":"2024-02-13T00:51:41.574139+0000","last_active":"2024-02-13T00:52:42.283161+0000","last_peered":"2024-02-13T00:52:42.283161+0000","last_clean":"2024-02-13T00:52:42.283161+0000","last_became_active":"2024-02-13T00:51:41.571758+0000","last_became_peered":"2024-02-13T00:51:41.571758+0000","last_unstale":"2024-02-13T00:52:42.283161+0000","last_undegraded":"2024-02-13T00:52:42.283161+0000","last_fullsized":"2024-02-13T00:52:42.283161+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:04.493841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:04.493841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":554,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283315+0000","last_change":"2024-02-13T00:52:30.580452+0000","last_active":"2024-02-13T00:52:42.283315+0000","last_peered":"2024-02-13T00:52:42.283315+0000","last_clean":"2024-02-13T00:52:42.283315+0000","last_became_active":"2024-02-13T00:52:30.580091+0000","last_became_peered":"2024-02-13T00:52:30.580091+0000","last_unstale":"2024-02-13T00:52:42.283315+0000","last_undegraded":"2024-02-13T00:52:42.283315+0000","last_fullsized":"2024-02-13T00:52:42.283315+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:25.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:48:25.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":594,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557930+0000","last_change":"2024-02-13T00:52:30.604915+0000","last_active":"2024-02-13T00:52:41.557930+0000","last_peered":"2024-02-13T00:52:41.557930+0000","last_clean":"2024-02-13T00:52:41.557930+0000","last_became_active":"2024-02-13T00:52:30.603107+0000","last_became_peered":"2024-02-13T00:52:30.603107+0000","last_unstale":"2024-02-13T00:52:41.557930+0000","last_undegraded":"2024-02-13T00:52:41.557930+0000","last_fullsized":"2024-02-13T00:52:41.557930+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:19.521319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:51:19.521319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":674,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.601250+0000","last_change":"2024-02-13T00:52:30.596982+0000","last_active":"2024-02-13T00:52:41.601250+0000","last_peered":"2024-02-13T00:52:41.601250+0000","last_clean":"2024-02-13T00:52:41.601250+0000","last_became_active":"2024-02-13T00:52:30.596402+0000","last_became_peered":"2024-02-13T00:52:30.596402+0000","last_unstale":"2024-02-13T00:52:41.601250+0000","last_undegraded":"2024-02-13T00:52:41.601250+0000","last_fullsized":"2024-02-13T00:52:41.601250+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:42.854585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:42.854585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":420,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557567+0000","last_change":"2024-02-13T00:52:30.581391+0000","last_active":"2024-02-13T00:52:41.557567+0000","last_peered":"2024-02-13T00:52:41.557567+0000","last_clean":"2024-02-13T00:52:41.557567+0000","last_became_active":"2024-02-13T00:52:30.581048+0000","last_became_peered":"2024-02-13T00:52:30.581048+0000","last_unstale":"2024-02-13T00:52:41.557567+0000","last_undegraded":"2024-02-13T00:52:41.557567+0000","last_fullsized":"2024-02-13T00:52:41.557567+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:53.013103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.013103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"467'23","reported_seq":790,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.558074+0000","last_change":"2024-02-13T00:51:42.290298+0000","last_active":"2024-02-13T00:52:41.558074+0000","last_peered":"2024-02-13T00:52:41.558074+0000","last_clean":"2024-02-13T00:52:41.558074+0000","last_became_active":"2024-02-13T00:51:41.569273+0000","last_became_peered":"2024-02-13T00:51:41.569273+0000","last_unstale":"2024-02-13T00:52:41.558074+0000","last_undegraded":"2024-02-13T00:52:41.558074+0000","last_fullsized":"2024-02-13T00:52:41.558074+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'23","last_scrub_stamp":"2024-02-13T00:49:07.192902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:07.192902+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":630,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598065+0000","last_change":"2024-02-13T00:49:46.851558+0000","last_active":"2024-02-13T00:52:41.598065+0000","last_peered":"2024-02-13T00:52:41.598065+0000","last_clean":"2024-02-13T00:52:41.598065+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:52:41.598065+0000","last_undegraded":"2024-02-13T00:52:41.598065+0000","last_fullsized":"2024-02-13T00:52:41.598065+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.851486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.851486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":1647,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-02-13T00:52:49.229169+0000","last_change":"2024-02-13T00:51:38.527164+0000","last_active":"2024-02-13T00:52:49.229169+0000","last_peered":"2024-02-13T00:52:49.229169+0000","last_clean":"2024-02-13T00:52:49.229169+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:52:49.229169+0000","last_undegraded":"2024-02-13T00:52:49.229169+0000","last_fullsized":"2024-02-13T00:52:49.229169+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:51:38.527084+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:51:38.527084+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":896,"num_read_kb":891,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":446,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286584+0000","last_change":"2024-02-13T00:51:41.571352+0000","last_active":"2024-02-13T00:52:42.286584+0000","last_peered":"2024-02-13T00:52:42.286584+0000","last_clean":"2024-02-13T00:52:42.286584+0000","last_became_active":"2024-02-13T00:51:41.570686+0000","last_became_peered":"2024-02-13T00:51:41.570686+0000","last_unstale":"2024-02-13T00:52:42.286584+0000","last_undegraded":"2024-02-13T00:52:42.286584+0000","last_fullsized":"2024-02-13T00:52:42.286584+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":519,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557025+0000","last_change":"2024-02-13T00:52:30.604046+0000","last_active":"2024-02-13T00:52:41.557025+0000","last_peered":"2024-02-13T00:52:41.557025+0000","last_clean":"2024-02-13T00:52:41.557025+0000","last_became_active":"2024-02-13T00:52:30.603400+0000","last_became_peered":"2024-02-13T00:52:30.603400+0000","last_unstale":"2024-02-13T00:52:41.557025+0000","last_undegraded":"2024-02-13T00:52:41.557025+0000","last_fullsized":"2024-02-13T00:52:41.557025+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:37.545088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:37.545088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":21,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598648+0000","last_change":"2024-02-13T00:52:30.575376+0000","last_active":"2024-02-13T00:52:41.598648+0000","last_peered":"2024-02-13T00:52:41.598648+0000","last_clean":"2024-02-13T00:52:41.598648+0000","last_became_active":"2024-02-13T00:52:30.575153+0000","last_became_peered":"2024-02-13T00:52:30.575153+0000","last_unstale":"2024-02-13T00:52:41.598648+0000","last_undegraded":"2024-02-13T00:52:41.598648+0000","last_fullsized":"2024-02-13T00:52:41.598648+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:26.113119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:52:26.113119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"701'67","reported_seq":78,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-02-13T00:52:44.384379+0000","last_change":"2024-02-13T00:52:09.459680+0000","last_active":"2024-02-13T00:52:44.384379+0000","last_peered":"2024-02-13T00:52:44.384379+0000","last_clean":"2024-02-13T00:52:44.384379+0000","last_became_active":"2024-02-13T00:52:07.565910+0000","last_became_peered":"2024-02-13T00:52:07.565910+0000","last_unstale":"2024-02-13T00:52:44.384379+0000","last_undegraded":"2024-02-13T00:52:44.384379+0000","last_fullsized":"2024-02-13T00:52:44.384379+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'65","last_scrub_stamp":"2024-02-13T00:49:47.928548+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.928548+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":77,"num_read_kb":61,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":2864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":646,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.555831+0000","last_change":"2024-02-13T00:50:37.777963+0000","last_active":"2024-02-13T00:52:41.555831+0000","last_peered":"2024-02-13T00:52:41.555831+0000","last_clean":"2024-02-13T00:52:41.555831+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:52:41.555831+0000","last_undegraded":"2024-02-13T00:52:41.555831+0000","last_fullsized":"2024-02-13T00:52:41.555831+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:37.777891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:37.777891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":273,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.600258+0000","last_change":"2024-02-13T00:52:30.598078+0000","last_active":"2024-02-13T00:52:41.600258+0000","last_peered":"2024-02-13T00:52:41.600258+0000","last_clean":"2024-02-13T00:52:41.600258+0000","last_became_active":"2024-02-13T00:52:30.595326+0000","last_became_peered":"2024-02-13T00:52:30.595326+0000","last_unstale":"2024-02-13T00:52:41.600258+0000","last_undegraded":"2024-02-13T00:52:41.600258+0000","last_fullsized":"2024-02-13T00:52:41.600258+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":424,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285058+0000","last_change":"2024-02-13T00:48:17.340891+0000","last_active":"2024-02-13T00:52:42.285058+0000","last_peered":"2024-02-13T00:52:42.285058+0000","last_clean":"2024-02-13T00:52:42.285058+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:52:42.285058+0000","last_undegraded":"2024-02-13T00:52:42.285058+0000","last_fullsized":"2024-02-13T00:52:42.285058+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:17.340797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:17.340797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":932,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-02-13T00:52:44.384634+0000","last_change":"2024-02-13T00:48:32.669477+0000","last_active":"2024-02-13T00:52:44.384634+0000","last_peered":"2024-02-13T00:52:44.384634+0000","last_clean":"2024-02-13T00:52:44.384634+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:52:44.384634+0000","last_undegraded":"2024-02-13T00:52:44.384634+0000","last_fullsized":"2024-02-13T00:52:44.384634+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14","last_scrub_stamp":"2024-02-13T00:48:32.669406+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.669406+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":487,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.556475+0000","last_change":"2024-02-13T00:52:33.373486+0000","last_active":"2024-02-13T00:52:41.556475+0000","last_peered":"2024-02-13T00:52:41.556475+0000","last_clean":"2024-02-13T00:52:41.556475+0000","last_became_active":"2024-02-13T00:52:30.605408+0000","last_became_peered":"2024-02-13T00:52:30.605408+0000","last_unstale":"2024-02-13T00:52:41.556475+0000","last_undegraded":"2024-02-13T00:52:41.556475+0000","last_fullsized":"2024-02-13T00:52:41.556475+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:48:20.928158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:20.928158+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":721,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557072+0000","last_change":"2024-02-13T00:51:41.570719+0000","last_active":"2024-02-13T00:52:41.557072+0000","last_peered":"2024-02-13T00:52:41.557072+0000","last_clean":"2024-02-13T00:52:41.557072+0000","last_became_active":"2024-02-13T00:51:41.567277+0000","last_became_peered":"2024-02-13T00:51:41.567277+0000","last_unstale":"2024-02-13T00:52:41.557072+0000","last_undegraded":"2024-02-13T00:52:41.557072+0000","last_fullsized":"2024-02-13T00:52:41.557072+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:33.040516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:49:33.040516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":731,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597674+0000","last_change":"2024-02-13T00:52:30.572639+0000","last_active":"2024-02-13T00:52:41.597674+0000","last_peered":"2024-02-13T00:52:41.597674+0000","last_clean":"2024-02-13T00:52:41.597674+0000","last_became_active":"2024-02-13T00:52:30.572144+0000","last_became_peered":"2024-02-13T00:52:30.572144+0000","last_unstale":"2024-02-13T00:52:41.597674+0000","last_undegraded":"2024-02-13T00:52:41.597674+0000","last_fullsized":"2024-02-13T00:52:41.597674+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:38.051468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:38.051468+0000","last_clean_scrub_stamp":"2024-02-13T00:50:38.051468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":670,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.554955+0000","last_change":"2024-02-13T00:51:41.589341+0000","last_active":"2024-02-13T00:52:41.554955+0000","last_peered":"2024-02-13T00:52:41.554955+0000","last_clean":"2024-02-13T00:52:41.554955+0000","last_became_active":"2024-02-13T00:51:41.588483+0000","last_became_peered":"2024-02-13T00:51:41.588483+0000","last_unstale":"2024-02-13T00:52:41.554955+0000","last_undegraded":"2024-02-13T00:52:41.554955+0000","last_fullsized":"2024-02-13T00:52:41.554955+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:08.606262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:51:08.606262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":449,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599678+0000","last_change":"2024-02-13T00:52:30.572465+0000","last_active":"2024-02-13T00:52:41.599678+0000","last_peered":"2024-02-13T00:52:41.599678+0000","last_clean":"2024-02-13T00:52:41.599678+0000","last_became_active":"2024-02-13T00:52:30.571718+0000","last_became_peered":"2024-02-13T00:52:30.571718+0000","last_unstale":"2024-02-13T00:52:41.599678+0000","last_undegraded":"2024-02-13T00:52:41.599678+0000","last_fullsized":"2024-02-13T00:52:41.599678+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:48.089417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:51:48.089417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":807,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.554951+0000","last_change":"2024-02-13T00:52:32.017951+0000","last_active":"2024-02-13T00:52:41.554951+0000","last_peered":"2024-02-13T00:52:41.554951+0000","last_clean":"2024-02-13T00:52:41.554951+0000","last_became_active":"2024-02-13T00:52:30.581394+0000","last_became_peered":"2024-02-13T00:52:30.581394+0000","last_unstale":"2024-02-13T00:52:41.554951+0000","last_undegraded":"2024-02-13T00:52:41.554951+0000","last_fullsized":"2024-02-13T00:52:41.554951+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'30","last_scrub_stamp":"2024-02-13T00:49:31.072862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:31.072862+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":741,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.601147+0000","last_change":"2024-02-13T00:52:31.527837+0000","last_active":"2024-02-13T00:52:41.601147+0000","last_peered":"2024-02-13T00:52:41.601147+0000","last_clean":"2024-02-13T00:52:41.601147+0000","last_became_active":"2024-02-13T00:52:30.595453+0000","last_became_peered":"2024-02-13T00:52:30.595453+0000","last_unstale":"2024-02-13T00:52:41.601147+0000","last_undegraded":"2024-02-13T00:52:41.601147+0000","last_fullsized":"2024-02-13T00:52:41.601147+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:48:58.861325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:58.861325+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":518,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.555992+0000","last_change":"2024-02-13T00:52:34.361735+0000","last_active":"2024-02-13T00:52:41.555992+0000","last_peered":"2024-02-13T00:52:41.555992+0000","last_clean":"2024-02-13T00:52:41.555992+0000","last_became_active":"2024-02-13T00:51:41.568308+0000","last_became_peered":"2024-02-13T00:51:41.568308+0000","last_unstale":"2024-02-13T00:52:41.555992+0000","last_undegraded":"2024-02-13T00:52:41.555992+0000","last_fullsized":"2024-02-13T00:52:41.555992+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:06.895612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:47:06.895612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":674,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.601245+0000","last_change":"2024-02-13T00:52:30.598253+0000","last_active":"2024-02-13T00:52:41.601245+0000","last_peered":"2024-02-13T00:52:41.601245+0000","last_clean":"2024-02-13T00:52:41.601245+0000","last_became_active":"2024-02-13T00:52:30.597320+0000","last_became_peered":"2024-02-13T00:52:30.597320+0000","last_unstale":"2024-02-13T00:52:41.601245+0000","last_undegraded":"2024-02-13T00:52:41.601245+0000","last_fullsized":"2024-02-13T00:52:41.601245+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.880001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.880001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"467'27","reported_seq":867,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.601152+0000","last_change":"2024-02-13T00:51:42.198644+0000","last_active":"2024-02-13T00:52:41.601152+0000","last_peered":"2024-02-13T00:52:41.601152+0000","last_clean":"2024-02-13T00:52:41.601152+0000","last_became_active":"2024-02-13T00:51:41.580479+0000","last_became_peered":"2024-02-13T00:51:41.580479+0000","last_unstale":"2024-02-13T00:52:41.601152+0000","last_undegraded":"2024-02-13T00:52:41.601152+0000","last_fullsized":"2024-02-13T00:52:41.601152+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:35.834669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.834669+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":428,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.554433+0000","last_change":"2024-02-13T00:52:30.581921+0000","last_active":"2024-02-13T00:52:41.554433+0000","last_peered":"2024-02-13T00:52:41.554433+0000","last_clean":"2024-02-13T00:52:41.554433+0000","last_became_active":"2024-02-13T00:52:30.581023+0000","last_became_peered":"2024-02-13T00:52:30.581023+0000","last_unstale":"2024-02-13T00:52:41.554433+0000","last_undegraded":"2024-02-13T00:52:41.554433+0000","last_fullsized":"2024-02-13T00:52:41.554433+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:48.521656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:51:48.521656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":447,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598069+0000","last_change":"2024-02-13T00:52:30.595830+0000","last_active":"2024-02-13T00:52:41.598069+0000","last_peered":"2024-02-13T00:52:41.598069+0000","last_clean":"2024-02-13T00:52:41.598069+0000","last_became_active":"2024-02-13T00:52:30.594958+0000","last_became_peered":"2024-02-13T00:52:30.594958+0000","last_unstale":"2024-02-13T00:52:41.598069+0000","last_undegraded":"2024-02-13T00:52:41.598069+0000","last_fullsized":"2024-02-13T00:52:41.598069+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:36.152699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:36.152699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":433,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286681+0000","last_change":"2024-02-13T00:52:30.601489+0000","last_active":"2024-02-13T00:52:42.286681+0000","last_peered":"2024-02-13T00:52:42.286681+0000","last_clean":"2024-02-13T00:52:42.286681+0000","last_became_active":"2024-02-13T00:52:30.600883+0000","last_became_peered":"2024-02-13T00:52:30.600883+0000","last_unstale":"2024-02-13T00:52:42.286681+0000","last_undegraded":"2024-02-13T00:52:42.286681+0000","last_fullsized":"2024-02-13T00:52:42.286681+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:07.304576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:07.304576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":607,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.558148+0000","last_change":"2024-02-13T00:51:52.536631+0000","last_active":"2024-02-13T00:52:41.558148+0000","last_peered":"2024-02-13T00:52:41.558148+0000","last_clean":"2024-02-13T00:52:41.558148+0000","last_became_active":"2024-02-13T00:51:41.571280+0000","last_became_peered":"2024-02-13T00:51:41.571280+0000","last_unstale":"2024-02-13T00:52:41.558148+0000","last_undegraded":"2024-02-13T00:52:41.558148+0000","last_fullsized":"2024-02-13T00:52:41.558148+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'18","last_scrub_stamp":"2024-02-13T00:51:52.536524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:51:52.536524+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":431,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286277+0000","last_change":"2024-02-13T00:52:30.575760+0000","last_active":"2024-02-13T00:52:42.286277+0000","last_peered":"2024-02-13T00:52:42.286277+0000","last_clean":"2024-02-13T00:52:42.286277+0000","last_became_active":"2024-02-13T00:52:30.575381+0000","last_became_peered":"2024-02-13T00:52:30.575381+0000","last_unstale":"2024-02-13T00:52:42.286277+0000","last_undegraded":"2024-02-13T00:52:42.286277+0000","last_fullsized":"2024-02-13T00:52:42.286277+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:20.093279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:52:20.093279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":451,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286636+0000","last_change":"2024-02-13T00:52:30.597684+0000","last_active":"2024-02-13T00:52:42.286636+0000","last_peered":"2024-02-13T00:52:42.286636+0000","last_clean":"2024-02-13T00:52:42.286636+0000","last_became_active":"2024-02-13T00:52:30.596322+0000","last_became_peered":"2024-02-13T00:52:30.596322+0000","last_unstale":"2024-02-13T00:52:42.286636+0000","last_undegraded":"2024-02-13T00:52:42.286636+0000","last_fullsized":"2024-02-13T00:52:42.286636+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:40.384428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:40.384428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":524,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598420+0000","last_change":"2024-02-13T00:52:30.597677+0000","last_active":"2024-02-13T00:52:41.598420+0000","last_peered":"2024-02-13T00:52:41.598420+0000","last_clean":"2024-02-13T00:52:41.598420+0000","last_became_active":"2024-02-13T00:52:30.595811+0000","last_became_peered":"2024-02-13T00:52:30.595811+0000","last_unstale":"2024-02-13T00:52:41.598420+0000","last_undegraded":"2024-02-13T00:52:41.598420+0000","last_fullsized":"2024-02-13T00:52:41.598420+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:19.286611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:19.286611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":691,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557265+0000","last_change":"2024-02-13T00:51:43.571860+0000","last_active":"2024-02-13T00:52:41.557265+0000","last_peered":"2024-02-13T00:52:41.557265+0000","last_clean":"2024-02-13T00:52:41.557265+0000","last_became_active":"2024-02-13T00:51:41.569584+0000","last_became_peered":"2024-02-13T00:51:41.569584+0000","last_unstale":"2024-02-13T00:52:41.557265+0000","last_undegraded":"2024-02-13T00:52:41.557265+0000","last_fullsized":"2024-02-13T00:52:41.557265+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'39","last_scrub_stamp":"2024-02-13T00:49:06.154931+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:49:06.154931+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":694,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598137+0000","last_change":"2024-02-13T00:52:30.595096+0000","last_active":"2024-02-13T00:52:41.598137+0000","last_peered":"2024-02-13T00:52:41.598137+0000","last_clean":"2024-02-13T00:52:41.598137+0000","last_became_active":"2024-02-13T00:52:30.594229+0000","last_became_peered":"2024-02-13T00:52:30.594229+0000","last_unstale":"2024-02-13T00:52:41.598137+0000","last_undegraded":"2024-02-13T00:52:41.598137+0000","last_fullsized":"2024-02-13T00:52:41.598137+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.916152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"726'10","reported_seq":465,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-02-13T00:52:44.384364+0000","last_change":"2024-02-13T00:52:32.256014+0000","last_active":"2024-02-13T00:52:44.384364+0000","last_peered":"2024-02-13T00:52:44.384364+0000","last_clean":"2024-02-13T00:52:44.384364+0000","last_became_active":"2024-02-13T00:52:30.603380+0000","last_became_peered":"2024-02-13T00:52:30.603380+0000","last_unstale":"2024-02-13T00:52:44.384364+0000","last_undegraded":"2024-02-13T00:52:44.384364+0000","last_fullsized":"2024-02-13T00:52:44.384364+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9","last_scrub_stamp":"2024-02-13T00:47:37.012934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:37.012934+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":731,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.600884+0000","last_change":"2024-02-13T00:52:30.598029+0000","last_active":"2024-02-13T00:52:41.600884+0000","last_peered":"2024-02-13T00:52:41.600884+0000","last_clean":"2024-02-13T00:52:41.600884+0000","last_became_active":"2024-02-13T00:52:30.596877+0000","last_became_peered":"2024-02-13T00:52:30.596877+0000","last_unstale":"2024-02-13T00:52:41.600884+0000","last_undegraded":"2024-02-13T00:52:41.600884+0000","last_fullsized":"2024-02-13T00:52:41.600884+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:26.647367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:26.647367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":393,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599529+0000","last_change":"2024-02-13T00:51:41.587438+0000","last_active":"2024-02-13T00:52:41.599529+0000","last_peered":"2024-02-13T00:52:41.599529+0000","last_clean":"2024-02-13T00:52:41.599529+0000","last_became_active":"2024-02-13T00:51:41.583454+0000","last_became_peered":"2024-02-13T00:51:41.583454+0000","last_unstale":"2024-02-13T00:52:41.599529+0000","last_undegraded":"2024-02-13T00:52:41.599529+0000","last_fullsized":"2024-02-13T00:52:41.599529+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.849687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.849687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"467'22","reported_seq":718,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286142+0000","last_change":"2024-02-13T00:51:44.862046+0000","last_active":"2024-02-13T00:52:42.286142+0000","last_peered":"2024-02-13T00:52:42.286142+0000","last_clean":"2024-02-13T00:52:42.286142+0000","last_became_active":"2024-02-13T00:51:41.584471+0000","last_became_peered":"2024-02-13T00:51:41.584471+0000","last_unstale":"2024-02-13T00:52:42.286142+0000","last_undegraded":"2024-02-13T00:52:42.286142+0000","last_fullsized":"2024-02-13T00:52:42.286142+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:47:35.369616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:35.369616+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":746,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598259+0000","last_change":"2024-02-13T00:52:30.598076+0000","last_active":"2024-02-13T00:52:41.598259+0000","last_peered":"2024-02-13T00:52:41.598259+0000","last_clean":"2024-02-13T00:52:41.598259+0000","last_became_active":"2024-02-13T00:52:30.593985+0000","last_became_peered":"2024-02-13T00:52:30.593985+0000","last_unstale":"2024-02-13T00:52:41.598259+0000","last_undegraded":"2024-02-13T00:52:41.598259+0000","last_fullsized":"2024-02-13T00:52:41.598259+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:05.887665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:05.887665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"726'16","reported_seq":621,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-02-13T00:52:44.384403+0000","last_change":"2024-02-13T00:52:31.620125+0000","last_active":"2024-02-13T00:52:44.384403+0000","last_peered":"2024-02-13T00:52:44.384403+0000","last_clean":"2024-02-13T00:52:44.384403+0000","last_became_active":"2024-02-13T00:52:30.603133+0000","last_became_peered":"2024-02-13T00:52:30.603133+0000","last_unstale":"2024-02-13T00:52:44.384403+0000","last_undegraded":"2024-02-13T00:52:44.384403+0000","last_fullsized":"2024-02-13T00:52:44.384403+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:47:07.942668+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:47:07.942668+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":370,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.601032+0000","last_change":"2024-02-13T00:52:30.596098+0000","last_active":"2024-02-13T00:52:41.601032+0000","last_peered":"2024-02-13T00:52:41.601032+0000","last_clean":"2024-02-13T00:52:41.601032+0000","last_became_active":"2024-02-13T00:52:30.594426+0000","last_became_peered":"2024-02-13T00:52:30.594426+0000","last_unstale":"2024-02-13T00:52:41.601032+0000","last_undegraded":"2024-02-13T00:52:41.601032+0000","last_fullsized":"2024-02-13T00:52:41.601032+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:27.933639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:27.933639+0000","last_clean_scrub_stamp":"2024-02-13T00:52:27.933639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":589,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.281392+0000","last_change":"2024-02-13T00:51:34.127332+0000","last_active":"2024-02-13T00:52:42.281392+0000","last_peered":"2024-02-13T00:52:42.281392+0000","last_clean":"2024-02-13T00:52:42.281392+0000","last_became_active":"2024-02-13T00:46:01.449694+0000","last_became_peered":"2024-02-13T00:46:01.449694+0000","last_unstale":"2024-02-13T00:52:42.281392+0000","last_undegraded":"2024-02-13T00:52:42.281392+0000","last_fullsized":"2024-02-13T00:52:42.281392+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:34.127252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:34.127252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":787,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.556238+0000","last_change":"2024-02-13T00:49:38.007273+0000","last_active":"2024-02-13T00:52:41.556238+0000","last_peered":"2024-02-13T00:52:41.556238+0000","last_clean":"2024-02-13T00:52:41.556238+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:52:41.556238+0000","last_undegraded":"2024-02-13T00:52:41.556238+0000","last_fullsized":"2024-02-13T00:52:41.556238+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'19","last_scrub_stamp":"2024-02-13T00:49:38.007204+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:49:38.007204+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":442,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283950+0000","last_change":"2024-02-13T00:51:41.571972+0000","last_active":"2024-02-13T00:52:42.283950+0000","last_peered":"2024-02-13T00:52:42.283950+0000","last_clean":"2024-02-13T00:52:42.283950+0000","last_became_active":"2024-02-13T00:51:41.570640+0000","last_became_peered":"2024-02-13T00:51:41.570640+0000","last_unstale":"2024-02-13T00:52:42.283950+0000","last_undegraded":"2024-02-13T00:52:42.283950+0000","last_fullsized":"2024-02-13T00:52:42.283950+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:13.316727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:51:13.316727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":506,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597990+0000","last_change":"2024-02-13T00:52:30.594111+0000","last_active":"2024-02-13T00:52:41.597990+0000","last_peered":"2024-02-13T00:52:41.597990+0000","last_clean":"2024-02-13T00:52:41.597990+0000","last_became_active":"2024-02-13T00:52:30.593800+0000","last_became_peered":"2024-02-13T00:52:30.593800+0000","last_unstale":"2024-02-13T00:52:41.597990+0000","last_undegraded":"2024-02-13T00:52:41.597990+0000","last_fullsized":"2024-02-13T00:52:41.597990+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:24.916456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:52:24.916456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":695,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.596276+0000","last_change":"2024-02-13T00:48:27.659266+0000","last_active":"2024-02-13T00:52:41.596276+0000","last_peered":"2024-02-13T00:52:41.596276+0000","last_clean":"2024-02-13T00:52:41.596276+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:52:41.596276+0000","last_undegraded":"2024-02-13T00:52:41.596276+0000","last_fullsized":"2024-02-13T00:52:41.596276+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:27.659197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:48:27.659197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":436,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286510+0000","last_change":"2024-02-13T00:52:39.216071+0000","last_active":"2024-02-13T00:52:42.286510+0000","last_peered":"2024-02-13T00:52:42.286510+0000","last_clean":"2024-02-13T00:52:42.286510+0000","last_became_active":"2024-02-13T00:51:41.570732+0000","last_became_peered":"2024-02-13T00:51:41.570732+0000","last_unstale":"2024-02-13T00:52:42.286510+0000","last_undegraded":"2024-02-13T00:52:42.286510+0000","last_fullsized":"2024-02-13T00:52:42.286510+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:40.233185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:45:40.233185+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":481,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286424+0000","last_change":"2024-02-13T00:52:33.431600+0000","last_active":"2024-02-13T00:52:42.286424+0000","last_peered":"2024-02-13T00:52:42.286424+0000","last_clean":"2024-02-13T00:52:42.286424+0000","last_became_active":"2024-02-13T00:52:30.594405+0000","last_became_peered":"2024-02-13T00:52:30.594405+0000","last_unstale":"2024-02-13T00:52:42.286424+0000","last_undegraded":"2024-02-13T00:52:42.286424+0000","last_fullsized":"2024-02-13T00:52:42.286424+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:48:34.458264+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:48:34.458264+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":712,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557506+0000","last_change":"2024-02-13T00:51:47.538426+0000","last_active":"2024-02-13T00:52:41.557506+0000","last_peered":"2024-02-13T00:52:41.557506+0000","last_clean":"2024-02-13T00:52:41.557506+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:52:41.557506+0000","last_undegraded":"2024-02-13T00:52:41.557506+0000","last_fullsized":"2024-02-13T00:52:41.557506+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:47.538354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:51:47.538354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":735,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.556317+0000","last_change":"2024-02-13T00:51:41.569851+0000","last_active":"2024-02-13T00:52:41.556317+0000","last_peered":"2024-02-13T00:52:41.556317+0000","last_clean":"2024-02-13T00:52:41.556317+0000","last_became_active":"2024-02-13T00:51:41.568200+0000","last_became_peered":"2024-02-13T00:51:41.568200+0000","last_unstale":"2024-02-13T00:52:41.556317+0000","last_undegraded":"2024-02-13T00:52:41.556317+0000","last_fullsized":"2024-02-13T00:52:41.556317+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:36.787604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:50:36.787604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":510,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599366+0000","last_change":"2024-02-13T00:52:30.588849+0000","last_active":"2024-02-13T00:52:41.599366+0000","last_peered":"2024-02-13T00:52:41.599366+0000","last_clean":"2024-02-13T00:52:41.599366+0000","last_became_active":"2024-02-13T00:52:30.588561+0000","last_became_peered":"2024-02-13T00:52:30.588561+0000","last_unstale":"2024-02-13T00:52:41.599366+0000","last_undegraded":"2024-02-13T00:52:41.599366+0000","last_fullsized":"2024-02-13T00:52:41.599366+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:29.681414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:29.681414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":442,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283841+0000","last_change":"2024-02-13T00:51:41.586630+0000","last_active":"2024-02-13T00:52:42.283841+0000","last_peered":"2024-02-13T00:52:42.283841+0000","last_clean":"2024-02-13T00:52:42.283841+0000","last_became_active":"2024-02-13T00:51:41.584153+0000","last_became_peered":"2024-02-13T00:51:41.584153+0000","last_unstale":"2024-02-13T00:52:42.283841+0000","last_undegraded":"2024-02-13T00:52:42.283841+0000","last_fullsized":"2024-02-13T00:52:42.283841+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:02.260045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:02.260045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":554,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.554557+0000","last_change":"2024-02-13T00:52:33.431814+0000","last_active":"2024-02-13T00:52:41.554557+0000","last_peered":"2024-02-13T00:52:41.554557+0000","last_clean":"2024-02-13T00:52:41.554557+0000","last_became_active":"2024-02-13T00:52:30.606082+0000","last_became_peered":"2024-02-13T00:52:30.606082+0000","last_unstale":"2024-02-13T00:52:41.554557+0000","last_undegraded":"2024-02-13T00:52:41.554557+0000","last_fullsized":"2024-02-13T00:52:41.554557+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'8","last_scrub_stamp":"2024-02-13T00:50:32.774820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:50:32.774820+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":659,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.281447+0000","last_change":"2024-02-13T00:48:32.441279+0000","last_active":"2024-02-13T00:52:42.281447+0000","last_peered":"2024-02-13T00:52:42.281447+0000","last_clean":"2024-02-13T00:52:42.281447+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:52:42.281447+0000","last_undegraded":"2024-02-13T00:52:42.281447+0000","last_fullsized":"2024-02-13T00:52:42.281447+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.441170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.441170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":442,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285262+0000","last_change":"2024-02-13T00:52:30.579288+0000","last_active":"2024-02-13T00:52:42.285262+0000","last_peered":"2024-02-13T00:52:42.285262+0000","last_clean":"2024-02-13T00:52:42.285262+0000","last_became_active":"2024-02-13T00:52:30.578921+0000","last_became_peered":"2024-02-13T00:52:30.578921+0000","last_unstale":"2024-02-13T00:52:42.285262+0000","last_undegraded":"2024-02-13T00:52:42.285262+0000","last_fullsized":"2024-02-13T00:52:42.285262+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:28.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:28.353347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":439,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.555550+0000","last_change":"2024-02-13T00:52:30.603553+0000","last_active":"2024-02-13T00:52:41.555550+0000","last_peered":"2024-02-13T00:52:41.555550+0000","last_clean":"2024-02-13T00:52:41.555550+0000","last_became_active":"2024-02-13T00:52:30.602714+0000","last_became_peered":"2024-02-13T00:52:30.602714+0000","last_unstale":"2024-02-13T00:52:41.555550+0000","last_undegraded":"2024-02-13T00:52:41.555550+0000","last_fullsized":"2024-02-13T00:52:41.555550+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.052640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.052640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":589,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283267+0000","last_change":"2024-02-13T00:52:30.593895+0000","last_active":"2024-02-13T00:52:42.283267+0000","last_peered":"2024-02-13T00:52:42.283267+0000","last_clean":"2024-02-13T00:52:42.283267+0000","last_became_active":"2024-02-13T00:52:30.593558+0000","last_became_peered":"2024-02-13T00:52:30.593558+0000","last_unstale":"2024-02-13T00:52:42.283267+0000","last_undegraded":"2024-02-13T00:52:42.283267+0000","last_fullsized":"2024-02-13T00:52:42.283267+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:54.114116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:54.114116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"467'25","reported_seq":510,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285224+0000","last_change":"2024-02-13T00:51:43.826108+0000","last_active":"2024-02-13T00:52:42.285224+0000","last_peered":"2024-02-13T00:52:42.285224+0000","last_clean":"2024-02-13T00:52:42.285224+0000","last_became_active":"2024-02-13T00:51:41.585940+0000","last_became_peered":"2024-02-13T00:51:41.585940+0000","last_unstale":"2024-02-13T00:52:42.285224+0000","last_undegraded":"2024-02-13T00:52:42.285224+0000","last_fullsized":"2024-02-13T00:52:42.285224+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:47:36.368863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.368863+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":433,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.281493+0000","last_change":"2024-02-13T00:46:59.388809+0000","last_active":"2024-02-13T00:52:42.281493+0000","last_peered":"2024-02-13T00:52:42.281493+0000","last_clean":"2024-02-13T00:52:42.281493+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:52:42.281493+0000","last_undegraded":"2024-02-13T00:52:42.281493+0000","last_fullsized":"2024-02-13T00:52:42.281493+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:59.388745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:59.388745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":442,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283358+0000","last_change":"2024-02-13T00:52:30.578502+0000","last_active":"2024-02-13T00:52:42.283358+0000","last_peered":"2024-02-13T00:52:42.283358+0000","last_clean":"2024-02-13T00:52:42.283358+0000","last_became_active":"2024-02-13T00:52:30.577297+0000","last_became_peered":"2024-02-13T00:52:30.577297+0000","last_unstale":"2024-02-13T00:52:42.283358+0000","last_undegraded":"2024-02-13T00:52:42.283358+0000","last_fullsized":"2024-02-13T00:52:42.283358+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:08.345652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:51:08.345652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":642,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.597980+0000","last_change":"2024-02-13T00:52:30.572207+0000","last_active":"2024-02-13T00:52:41.597980+0000","last_peered":"2024-02-13T00:52:41.597980+0000","last_clean":"2024-02-13T00:52:41.597980+0000","last_became_active":"2024-02-13T00:52:30.569895+0000","last_became_peered":"2024-02-13T00:52:30.569895+0000","last_unstale":"2024-02-13T00:52:41.597980+0000","last_undegraded":"2024-02-13T00:52:41.597980+0000","last_fullsized":"2024-02-13T00:52:41.597980+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:06.264572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:51:06.264572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":556,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285255+0000","last_change":"2024-02-13T00:51:41.571094+0000","last_active":"2024-02-13T00:52:42.285255+0000","last_peered":"2024-02-13T00:52:42.285255+0000","last_clean":"2024-02-13T00:52:42.285255+0000","last_became_active":"2024-02-13T00:51:41.569716+0000","last_became_peered":"2024-02-13T00:51:41.569716+0000","last_unstale":"2024-02-13T00:52:42.285255+0000","last_undegraded":"2024-02-13T00:52:42.285255+0000","last_fullsized":"2024-02-13T00:52:42.285255+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:35.101298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.101298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":840,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.601426+0000","last_change":"2024-02-13T00:51:44.341669+0000","last_active":"2024-02-13T00:52:41.601426+0000","last_peered":"2024-02-13T00:52:41.601426+0000","last_clean":"2024-02-13T00:52:41.601426+0000","last_became_active":"2024-02-13T00:51:41.589397+0000","last_became_peered":"2024-02-13T00:51:41.589397+0000","last_unstale":"2024-02-13T00:52:41.601426+0000","last_undegraded":"2024-02-13T00:52:41.601426+0000","last_fullsized":"2024-02-13T00:52:41.601426+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'32","last_scrub_stamp":"2024-02-13T00:48:53.841797+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.841797+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":679,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.282434+0000","last_change":"2024-02-13T00:52:30.578860+0000","last_active":"2024-02-13T00:52:42.282434+0000","last_peered":"2024-02-13T00:52:42.282434+0000","last_clean":"2024-02-13T00:52:42.282434+0000","last_became_active":"2024-02-13T00:52:30.578094+0000","last_became_peered":"2024-02-13T00:52:30.578094+0000","last_unstale":"2024-02-13T00:52:42.282434+0000","last_undegraded":"2024-02-13T00:52:42.282434+0000","last_fullsized":"2024-02-13T00:52:42.282434+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.297918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.297918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":551,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283313+0000","last_change":"2024-02-13T00:52:32.509795+0000","last_active":"2024-02-13T00:52:42.283313+0000","last_peered":"2024-02-13T00:52:42.283313+0000","last_clean":"2024-02-13T00:52:42.283313+0000","last_became_active":"2024-02-13T00:52:30.585316+0000","last_became_peered":"2024-02-13T00:52:30.585316+0000","last_unstale":"2024-02-13T00:52:42.283313+0000","last_undegraded":"2024-02-13T00:52:42.283313+0000","last_fullsized":"2024-02-13T00:52:42.283313+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:48:33.460247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:33.460247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":679,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.558121+0000","last_change":"2024-02-13T00:51:41.571960+0000","last_active":"2024-02-13T00:52:41.558121+0000","last_peered":"2024-02-13T00:52:41.558121+0000","last_clean":"2024-02-13T00:52:41.558121+0000","last_became_active":"2024-02-13T00:51:41.569703+0000","last_became_peered":"2024-02-13T00:51:41.569703+0000","last_unstale":"2024-02-13T00:52:41.558121+0000","last_undegraded":"2024-02-13T00:52:41.558121+0000","last_fullsized":"2024-02-13T00:52:41.558121+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:39.570602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:39.570602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":464,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.281456+0000","last_change":"2024-02-13T00:48:55.424021+0000","last_active":"2024-02-13T00:52:42.281456+0000","last_peered":"2024-02-13T00:52:42.281456+0000","last_clean":"2024-02-13T00:52:42.281456+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:52:42.281456+0000","last_undegraded":"2024-02-13T00:52:42.281456+0000","last_fullsized":"2024-02-13T00:52:42.281456+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:55.423952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.423952+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":442,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285495+0000","last_change":"2024-02-13T00:51:41.586115+0000","last_active":"2024-02-13T00:52:42.285495+0000","last_peered":"2024-02-13T00:52:42.285495+0000","last_clean":"2024-02-13T00:52:42.285495+0000","last_became_active":"2024-02-13T00:51:41.584919+0000","last_became_peered":"2024-02-13T00:51:41.584919+0000","last_unstale":"2024-02-13T00:52:42.285495+0000","last_undegraded":"2024-02-13T00:52:42.285495+0000","last_fullsized":"2024-02-13T00:52:42.285495+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:56.339885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.339885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":715,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557377+0000","last_change":"2024-02-13T00:52:30.603945+0000","last_active":"2024-02-13T00:52:41.557377+0000","last_peered":"2024-02-13T00:52:41.557377+0000","last_clean":"2024-02-13T00:52:41.557377+0000","last_became_active":"2024-02-13T00:52:30.602705+0000","last_became_peered":"2024-02-13T00:52:30.602705+0000","last_unstale":"2024-02-13T00:52:41.557377+0000","last_undegraded":"2024-02-13T00:52:41.557377+0000","last_fullsized":"2024-02-13T00:52:41.557377+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.928651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.928651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":429,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285014+0000","last_change":"2024-02-13T00:52:24.173739+0000","last_active":"2024-02-13T00:52:42.285014+0000","last_peered":"2024-02-13T00:52:42.285014+0000","last_clean":"2024-02-13T00:52:42.285014+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:52:42.285014+0000","last_undegraded":"2024-02-13T00:52:42.285014+0000","last_fullsized":"2024-02-13T00:52:42.285014+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:24.173651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:24.173651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":510,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557157+0000","last_change":"2024-02-13T00:52:32.307501+0000","last_active":"2024-02-13T00:52:41.557157+0000","last_peered":"2024-02-13T00:52:41.557157+0000","last_clean":"2024-02-13T00:52:41.557157+0000","last_became_active":"2024-02-13T00:52:30.603658+0000","last_became_peered":"2024-02-13T00:52:30.603658+0000","last_unstale":"2024-02-13T00:52:41.557157+0000","last_undegraded":"2024-02-13T00:52:41.557157+0000","last_fullsized":"2024-02-13T00:52:41.557157+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:30.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:30.086856+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":21,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.282798+0000","last_change":"2024-02-13T00:52:30.578696+0000","last_active":"2024-02-13T00:52:42.282798+0000","last_peered":"2024-02-13T00:52:42.282798+0000","last_clean":"2024-02-13T00:52:42.282798+0000","last_became_active":"2024-02-13T00:52:30.578352+0000","last_became_peered":"2024-02-13T00:52:30.578352+0000","last_unstale":"2024-02-13T00:52:42.282798+0000","last_undegraded":"2024-02-13T00:52:42.282798+0000","last_fullsized":"2024-02-13T00:52:42.282798+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:04.282529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:04.282529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":433,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.285454+0000","last_change":"2024-02-13T00:52:30.594940+0000","last_active":"2024-02-13T00:52:42.285454+0000","last_peered":"2024-02-13T00:52:42.285454+0000","last_clean":"2024-02-13T00:52:42.285454+0000","last_became_active":"2024-02-13T00:52:30.594309+0000","last_became_peered":"2024-02-13T00:52:30.594309+0000","last_unstale":"2024-02-13T00:52:42.285454+0000","last_undegraded":"2024-02-13T00:52:42.285454+0000","last_fullsized":"2024-02-13T00:52:42.285454+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:42.329367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:42.329367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":356,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286178+0000","last_change":"2024-02-13T00:52:30.597508+0000","last_active":"2024-02-13T00:52:42.286178+0000","last_peered":"2024-02-13T00:52:42.286178+0000","last_clean":"2024-02-13T00:52:42.286178+0000","last_became_active":"2024-02-13T00:52:30.595882+0000","last_became_peered":"2024-02-13T00:52:30.595882+0000","last_unstale":"2024-02-13T00:52:42.286178+0000","last_undegraded":"2024-02-13T00:52:42.286178+0000","last_fullsized":"2024-02-13T00:52:42.286178+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:23.430396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:23.430396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":614,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.557792+0000","last_change":"2024-02-13T00:52:32.857449+0000","last_active":"2024-02-13T00:52:41.557792+0000","last_peered":"2024-02-13T00:52:41.557792+0000","last_clean":"2024-02-13T00:52:41.557792+0000","last_became_active":"2024-02-13T00:52:30.604045+0000","last_became_peered":"2024-02-13T00:52:30.604045+0000","last_unstale":"2024-02-13T00:52:41.557792+0000","last_undegraded":"2024-02-13T00:52:41.557792+0000","last_fullsized":"2024-02-13T00:52:41.557792+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:52:21.312574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:21.312574+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":451,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286579+0000","last_change":"2024-02-13T00:52:30.596393+0000","last_active":"2024-02-13T00:52:42.286579+0000","last_peered":"2024-02-13T00:52:42.286579+0000","last_clean":"2024-02-13T00:52:42.286579+0000","last_became_active":"2024-02-13T00:52:30.596074+0000","last_became_peered":"2024-02-13T00:52:30.596074+0000","last_unstale":"2024-02-13T00:52:42.286579+0000","last_undegraded":"2024-02-13T00:52:42.286579+0000","last_fullsized":"2024-02-13T00:52:42.286579+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:18.264738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:18.264738+0000","last_clean_scrub_stamp":"2024-02-13T00:51:18.264738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":424,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.281643+0000","last_change":"2024-02-13T00:46:01.461188+0000","last_active":"2024-02-13T00:52:42.281643+0000","last_peered":"2024-02-13T00:52:42.281643+0000","last_clean":"2024-02-13T00:52:42.281643+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:52:42.281643+0000","last_undegraded":"2024-02-13T00:52:42.281643+0000","last_fullsized":"2024-02-13T00:52:42.281643+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.238152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.238152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":674,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.596333+0000","last_change":"2024-02-13T00:51:35.190514+0000","last_active":"2024-02-13T00:52:41.596333+0000","last_peered":"2024-02-13T00:52:41.596333+0000","last_clean":"2024-02-13T00:52:41.596333+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:52:41.596333+0000","last_undegraded":"2024-02-13T00:52:41.596333+0000","last_fullsized":"2024-02-13T00:52:41.596333+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:35.190419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:35.190419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":602,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598025+0000","last_change":"2024-02-13T00:52:31.275396+0000","last_active":"2024-02-13T00:52:41.598025+0000","last_peered":"2024-02-13T00:52:41.598025+0000","last_clean":"2024-02-13T00:52:41.598025+0000","last_became_active":"2024-02-13T00:52:30.592405+0000","last_became_peered":"2024-02-13T00:52:30.592405+0000","last_unstale":"2024-02-13T00:52:41.598025+0000","last_undegraded":"2024-02-13T00:52:41.598025+0000","last_fullsized":"2024-02-13T00:52:41.598025+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:52:16.976052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:16.976052+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":517,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283506+0000","last_change":"2024-02-13T00:51:41.571773+0000","last_active":"2024-02-13T00:52:42.283506+0000","last_peered":"2024-02-13T00:52:42.283506+0000","last_clean":"2024-02-13T00:52:42.283506+0000","last_became_active":"2024-02-13T00:51:41.569507+0000","last_became_peered":"2024-02-13T00:51:41.569507+0000","last_unstale":"2024-02-13T00:52:42.283506+0000","last_undegraded":"2024-02-13T00:52:42.283506+0000","last_fullsized":"2024-02-13T00:52:42.283506+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:43.067615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:43.067615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":711,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.599627+0000","last_change":"2024-02-13T00:52:35.023058+0000","last_active":"2024-02-13T00:52:41.599627+0000","last_peered":"2024-02-13T00:52:41.599627+0000","last_clean":"2024-02-13T00:52:41.599627+0000","last_became_active":"2024-02-13T00:51:41.582536+0000","last_became_peered":"2024-02-13T00:51:41.582536+0000","last_unstale":"2024-02-13T00:52:41.599627+0000","last_undegraded":"2024-02-13T00:52:41.599627+0000","last_fullsized":"2024-02-13T00:52:41.599627+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:35.022992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:52:35.022992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":617,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.598510+0000","last_change":"2024-02-13T00:51:41.587948+0000","last_active":"2024-02-13T00:52:41.598510+0000","last_peered":"2024-02-13T00:52:41.598510+0000","last_clean":"2024-02-13T00:52:41.598510+0000","last_became_active":"2024-02-13T00:51:41.583578+0000","last_became_peered":"2024-02-13T00:51:41.583578+0000","last_unstale":"2024-02-13T00:52:41.598510+0000","last_undegraded":"2024-02-13T00:52:41.598510+0000","last_fullsized":"2024-02-13T00:52:41.598510+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:15.846461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:15.846461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":633,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.555852+0000","last_change":"2024-02-13T00:52:32.781431+0000","last_active":"2024-02-13T00:52:41.555852+0000","last_peered":"2024-02-13T00:52:41.555852+0000","last_clean":"2024-02-13T00:52:41.555852+0000","last_became_active":"2024-02-13T00:52:30.604370+0000","last_became_peered":"2024-02-13T00:52:30.604370+0000","last_unstale":"2024-02-13T00:52:41.555852+0000","last_undegraded":"2024-02-13T00:52:41.555852+0000","last_fullsized":"2024-02-13T00:52:41.555852+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'16","last_scrub_stamp":"2024-02-13T00:51:49.491168+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:51:49.491168+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":442,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286724+0000","last_change":"2024-02-13T00:51:41.586572+0000","last_active":"2024-02-13T00:52:42.286724+0000","last_peered":"2024-02-13T00:52:42.286724+0000","last_clean":"2024-02-13T00:52:42.286724+0000","last_became_active":"2024-02-13T00:51:41.583922+0000","last_became_peered":"2024-02-13T00:51:41.583922+0000","last_unstale":"2024-02-13T00:52:42.286724+0000","last_undegraded":"2024-02-13T00:52:42.286724+0000","last_fullsized":"2024-02-13T00:52:42.286724+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.284014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.284014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":660,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.555162+0000","last_change":"2024-02-13T00:51:41.589710+0000","last_active":"2024-02-13T00:52:41.555162+0000","last_peered":"2024-02-13T00:52:41.555162+0000","last_clean":"2024-02-13T00:52:41.555162+0000","last_became_active":"2024-02-13T00:51:41.589029+0000","last_became_peered":"2024-02-13T00:51:41.589029+0000","last_unstale":"2024-02-13T00:52:41.555162+0000","last_undegraded":"2024-02-13T00:52:41.555162+0000","last_fullsized":"2024-02-13T00:52:41.555162+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:01.133709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:49:01.133709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":420,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.283862+0000","last_change":"2024-02-13T00:52:38.236729+0000","last_active":"2024-02-13T00:52:42.283862+0000","last_peered":"2024-02-13T00:52:42.283862+0000","last_clean":"2024-02-13T00:52:42.283862+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:52:42.283862+0000","last_undegraded":"2024-02-13T00:52:42.283862+0000","last_fullsized":"2024-02-13T00:52:42.283862+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:38.236655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:38.236655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":499,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.284024+0000","last_change":"2024-02-13T00:52:32.083766+0000","last_active":"2024-02-13T00:52:42.284024+0000","last_peered":"2024-02-13T00:52:42.284024+0000","last_clean":"2024-02-13T00:52:42.284024+0000","last_became_active":"2024-02-13T00:52:30.576217+0000","last_became_peered":"2024-02-13T00:52:30.576217+0000","last_unstale":"2024-02-13T00:52:42.284024+0000","last_undegraded":"2024-02-13T00:52:42.284024+0000","last_fullsized":"2024-02-13T00:52:42.284024+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:51:10.298233+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:51:10.298233+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":528,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.601298+0000","last_change":"2024-02-13T00:52:30.598152+0000","last_active":"2024-02-13T00:52:41.601298+0000","last_peered":"2024-02-13T00:52:41.601298+0000","last_clean":"2024-02-13T00:52:41.601298+0000","last_became_active":"2024-02-13T00:52:30.597277+0000","last_became_peered":"2024-02-13T00:52:30.597277+0000","last_unstale":"2024-02-13T00:52:41.601298+0000","last_undegraded":"2024-02-13T00:52:41.601298+0000","last_fullsized":"2024-02-13T00:52:41.601298+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.711486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.711486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":368,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.555747+0000","last_change":"2024-02-13T00:52:30.603770+0000","last_active":"2024-02-13T00:52:41.555747+0000","last_peered":"2024-02-13T00:52:41.555747+0000","last_clean":"2024-02-13T00:52:41.555747+0000","last_became_active":"2024-02-13T00:52:30.602788+0000","last_became_peered":"2024-02-13T00:52:30.602788+0000","last_unstale":"2024-02-13T00:52:41.555747+0000","last_undegraded":"2024-02-13T00:52:41.555747+0000","last_fullsized":"2024-02-13T00:52:41.555747+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:53.825971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:53.825971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":682,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.555336+0000","last_change":"2024-02-13T00:51:36.522278+0000","last_active":"2024-02-13T00:52:41.555336+0000","last_peered":"2024-02-13T00:52:41.555336+0000","last_clean":"2024-02-13T00:52:41.555336+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:52:41.555336+0000","last_undegraded":"2024-02-13T00:52:41.555336+0000","last_fullsized":"2024-02-13T00:52:41.555336+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:51:36.522208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:36.522208+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":609,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.596276+0000","last_change":"2024-02-13T00:47:44.757362+0000","last_active":"2024-02-13T00:52:41.596276+0000","last_peered":"2024-02-13T00:52:41.596276+0000","last_clean":"2024-02-13T00:52:41.596276+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:52:41.596276+0000","last_undegraded":"2024-02-13T00:52:41.596276+0000","last_fullsized":"2024-02-13T00:52:41.596276+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'50","last_scrub_stamp":"2024-02-13T00:47:44.757262+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:47:44.757262+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":719,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:41.601107+0000","last_change":"2024-02-13T00:52:30.596601+0000","last_active":"2024-02-13T00:52:41.601107+0000","last_peered":"2024-02-13T00:52:41.601107+0000","last_clean":"2024-02-13T00:52:41.601107+0000","last_became_active":"2024-02-13T00:52:30.595533+0000","last_became_peered":"2024-02-13T00:52:30.595533+0000","last_unstale":"2024-02-13T00:52:41.601107+0000","last_undegraded":"2024-02-13T00:52:41.601107+0000","last_fullsized":"2024-02-13T00:52:41.601107+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:54.817999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.817999+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":433,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-02-13T00:52:42.286686+0000","last_change":"2024-02-13T00:51:41.573986+0000","last_active":"2024-02-13T00:52:42.286686+0000","last_peered":"2024-02-13T00:52:42.286686+0000","last_clean":"2024-02-13T00:52:42.286686+0000","last_became_active":"2024-02-13T00:51:41.571183+0000","last_became_peered":"2024-02-13T00:51:41.571183+0000","last_unstale":"2024-02-13T00:52:42.286686+0000","last_undegraded":"2024-02-13T00:52:42.286686+0000","last_fullsized":"2024-02-13T00:52:42.286686+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:03.383666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:03.383666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":249,"num_pg":32,"stat_sum":{"num_bytes":65704,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":197112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":4},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":4000089,"num_objects":17,"num_object_clones":1,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":16,"num_read_kb":4,"num_write":37,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4000081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6193152,"data_stored":6193152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":24,"acting":24,"num_store_stats":5},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1491,"num_read_kb":1216,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":780,"num_bytes_recovered":13820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":11436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":972,"ondisk_log_size":972,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":896,"num_read_kb":891,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266872,"num_pgs":147,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316772,"kb_used_data":7260,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428380,"statfs":{"total":95995035648,"available":95670661120,"internally_reserved":0,"allocated":7434240,"data_stored":4831012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.495,"15min":0.63300000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.66600000000000004,"5min":0.76700000000000002,"15min":10.682},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.502,"15min":0.745},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":25.504000000000001},"last":0.52200000000000002}]},{"osd":1,"last update":"Tue Feb 13 00:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.57699999999999996},"min":{"1min":0.26500000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":13.347},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.67000000000000004},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69299999999999995,"5min":0.77300000000000002,"15min":13.526999999999999},"last":0.54400000000000004}]},{"osd":2,"last update":"Tue Feb 13 00:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.52300000000000002,"15min":0.66100000000000003},"min":{"1min":0.46000000000000002,"5min":0.34799999999999998,"15min":0.26600000000000001},"max":{"1min":0.871,"5min":0.871,"15min":14.359},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53900000000000003,"15min":0.74299999999999999},"min":{"1min":0.32000000000000001,"5min":0.30399999999999999,"15min":0.26400000000000001},"max":{"1min":0.70299999999999996,"5min":0.81599999999999995,"15min":21.497},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Feb 13 00:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.71299999999999997},"min":{"1min":0.39700000000000002,"5min":0.34899999999999998,"15min":0.29699999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":12.997},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53800000000000003,"15min":0.81999999999999995},"min":{"1min":0.36599999999999999,"5min":0.35199999999999998,"15min":0.318},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":21.622},"last":0.51600000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.44,"15min":0.44},"min":{"1min":0.32900000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.70499999999999996,"5min":0.76000000000000001,"15min":2.452},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.46800000000000003,"15min":0.45000000000000001},"min":{"1min":0.31900000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":1.0780000000000001},"last":0.41299999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51000000000000001,"15min":0.50900000000000001},"min":{"1min":0.35599999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71199999999999997,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.434},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51500000000000001,"15min":0.52000000000000002},"min":{"1min":0.33500000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.747,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45500000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.38,"5min":0.25,"15min":0.25},"max":{"1min":0.67700000000000005,"5min":0.81299999999999994,"15min":0.95299999999999996},"last":0.5},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.374,"5min":0.30399999999999999,"15min":0.27500000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":1.0940000000000001},"last":0.50600000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528510,"num_pgs":157,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304404,"kb_used_data":7180,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93440748,"statfs":{"total":95995035648,"available":95683325952,"internally_reserved":0,"allocated":7352320,"data_stored":4757290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.504,"15min":0.53900000000000003},"min":{"1min":0.36199999999999999,"5min":0.316,"15min":0.19700000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":8.1340000000000003},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.495,"5min":0.502,"15min":0.61699999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.221},"max":{"1min":0.71599999999999997,"5min":0.72799999999999998,"15min":11.367000000000001},"last":0.48399999999999999}]},{"osd":1,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.505,"15min":0.88400000000000001},"min":{"1min":0.33300000000000002,"5min":0.30099999999999999,"15min":0.255},"max":{"1min":0.89300000000000002,"5min":1.1080000000000001,"15min":54.002000000000002},"last":0.499},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51200000000000001,"15min":0.67800000000000005},"min":{"1min":0.30599999999999999,"5min":0.27200000000000002,"15min":0.23499999999999999},"max":{"1min":0.73599999999999999,"5min":0.95199999999999996,"15min":12.403},"last":0.63200000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52000000000000002,"15min":0.71199999999999997},"min":{"1min":0.32400000000000001,"5min":0.31900000000000001,"15min":0.25600000000000001},"max":{"1min":0.80000000000000004,"5min":1.587,"15min":12.477},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.5,"15min":0.67900000000000005},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.67100000000000004,"5min":1.1899999999999999,"15min":11.702},"last":0.51400000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56999999999999995,"15min":0.67300000000000004},"min":{"1min":0.32700000000000001,"5min":0.31900000000000001,"15min":0.29199999999999998},"max":{"1min":0.79400000000000004,"5min":5.931,"15min":8.0519999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53900000000000003,"15min":0.77200000000000002},"min":{"1min":0.38200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":19.081},"last":0.57899999999999996}]},{"osd":4,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42299999999999999,"15min":0.42099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.24199999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":1.0649999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46100000000000002,"15min":0.44500000000000001},"min":{"1min":0.28799999999999998,"5min":0.252,"15min":0.221},"max":{"1min":0.66600000000000004,"5min":1.008,"15min":1.008},"last":0.64400000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.49399999999999999,"15min":0.51100000000000001},"min":{"1min":0.29299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66200000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.314,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73099999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.66600000000000004}]},{"osd":7,"last update":"Tue Feb 13 00:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.45700000000000002,"15min":0.47499999999999998},"min":{"1min":0.30199999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61599999999999999,"5min":0.72299999999999998,"15min":1.173},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51500000000000001,"15min":0.52300000000000002},"min":{"1min":0.318,"5min":0.28299999999999997,"15min":0.26900000000000002},"max":{"1min":0.76100000000000001,"5min":0.80700000000000005,"15min":1.262},"last":0.67600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575063,"num_pgs":154,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316740,"kb_used_data":7228,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428412,"statfs":{"total":95995035648,"available":95670693888,"internally_reserved":0,"allocated":7401472,"data_stored":4781795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34000000000000002,"15min":0.95299999999999996},"min":{"1min":0.23899999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":61.856000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.67100000000000004},"min":{"1min":0.224,"5min":0.19900000000000001,"15min":0.16500000000000001},"max":{"1min":0.74399999999999999,"5min":0.77600000000000002,"15min":61.908999999999999},"last":0.377}]},{"osd":2,"last update":"Tue Feb 13 00:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34100000000000003,"15min":0.84399999999999997},"min":{"1min":0.24399999999999999,"5min":0.22900000000000001,"15min":0.20599999999999999},"max":{"1min":0.46800000000000003,"5min":0.57599999999999996,"15min":32.161000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35299999999999998,"15min":0.82199999999999995},"min":{"1min":0.26000000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.51000000000000001,"5min":0.55900000000000005,"15min":29.693000000000001},"last":0.44800000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.36699999999999999,"15min":1.2509999999999999},"min":{"1min":0.27500000000000002,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":131.23099999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.38400000000000001,"15min":0.84599999999999997},"min":{"1min":0.29099999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.753,"5min":0.753,"15min":49.055},"last":0.41499999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.52200000000000002,"15min":0.748},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.27900000000000003},"max":{"1min":0.65200000000000002,"5min":0.77800000000000002,"15min":16.547000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57399999999999995,"15min":0.71099999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.24199999999999999},"max":{"1min":0.68200000000000005,"5min":0.85199999999999998,"15min":14.731999999999999},"last":0.81799999999999995}]},{"osd":5,"last update":"Tue Feb 13 00:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58199999999999996,"15min":0.57199999999999995},"min":{"1min":0.41299999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84699999999999998,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59799999999999998,"15min":0.58699999999999997},"min":{"1min":0.377,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76100000000000001,"5min":1.831,"15min":1.831},"last":0.84699999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57099999999999995,"15min":0.71099999999999997},"min":{"1min":0.39000000000000001,"5min":0.32300000000000001,"15min":0.29099999999999998},"max":{"1min":0.878,"5min":1.9279999999999999,"15min":14.772},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55200000000000005,"15min":0.91200000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77300000000000002,"5min":1.2709999999999999,"15min":26.600000000000001},"last":0.80700000000000005}]},{"osd":7,"last update":"Tue Feb 13 00:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.57599999999999996,"15min":0.72099999999999997},"min":{"1min":0.38300000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":1.8680000000000001,"15min":14.794},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57199999999999995,"15min":0.92000000000000004},"min":{"1min":0.38200000000000001,"5min":0.33000000000000002,"15min":0.32200000000000001},"max":{"1min":0.80400000000000005,"5min":1.246,"15min":26.693999999999999},"last":0.83899999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738587,"num_pgs":154,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314796,"kb_used_data":5284,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430356,"statfs":{"total":95995035648,"available":95672684544,"internally_reserved":0,"allocated":5410816,"data_stored":2815406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33900000000000002,"15min":0.622},"min":{"1min":0.22500000000000001,"5min":0.191,"15min":0.189},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":30.184000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.36499999999999999,"15min":0.45600000000000002},"min":{"1min":0.20699999999999999,"5min":0.20499999999999999,"15min":0.187},"max":{"1min":1.0109999999999999,"5min":1.0780000000000001,"15min":5.6959999999999997},"last":0.26700000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.373,"15min":0.49399999999999999},"min":{"1min":0.248,"5min":0.24399999999999999,"15min":0.216},"max":{"1min":0.72499999999999998,"5min":1.137,"15min":7.0839999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.377,"15min":0.67200000000000004},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.185},"max":{"1min":0.61099999999999999,"5min":2.0129999999999999,"15min":24.459},"last":0.32800000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:52:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.36899999999999999,"15min":0.48099999999999998},"min":{"1min":0.25900000000000001,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":8.718},"last":0.317},{"interface":"front","average":{"1min":0.438,"5min":0.40699999999999997,"15min":0.52600000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.222},"max":{"1min":1.097,"5min":1.1120000000000001,"15min":7.1139999999999999},"last":0.29799999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.80500000000000005},"min":{"1min":0.34899999999999998,"5min":0.29899999999999999,"15min":0.24399999999999999},"max":{"1min":0.75800000000000001,"5min":0.874,"15min":24.381},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57399999999999995,"15min":0.67100000000000004},"min":{"1min":0.39600000000000002,"5min":0.31,"15min":0.28699999999999998},"max":{"1min":0.78700000000000003,"5min":0.88200000000000001,"15min":8.6950000000000003},"last":0.44500000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.68799999999999994,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54800000000000004,"15min":0.53600000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71399999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.46999999999999997}]},{"osd":6,"last update":"Tue Feb 13 00:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.55000000000000004,"15min":0.63800000000000001},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.31},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":8.6129999999999995},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.66000000000000003},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.32400000000000001},"max":{"1min":1.411,"5min":1.411,"15min":14.263},"last":0.46000000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55900000000000005,"15min":0.66500000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.317},"max":{"1min":0.76600000000000001,"5min":0.80800000000000005,"15min":8.6679999999999993},"last":0.497},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58699999999999997,"15min":0.67000000000000004},"min":{"1min":0.40899999999999997,"5min":0.40400000000000003,"15min":0.35899999999999999},"max":{"1min":0.89600000000000002,"5min":1.6910000000000001,"15min":8.1609999999999996},"last":0.42299999999999999}]}]},{"osd":2,"up_from":0,"seq":77309411538,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182604,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":133143986376,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":2057289334842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":3218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2023424,"data_stored":2023424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2056192,"data_stored":2056192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2064384,"data_stored":2064384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":249,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":65704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":249,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":65704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":249,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":249,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":65704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:52:52.092 INFO:tasks.thrashosds.thrasher:Setting 2.1d to [0, 7, 4, 2, 6, 1] 2024-02-13T00:52:52.092 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.1d', '0', '7', '4', '2', '6', '1'] 2024-02-13T00:52:52.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pg-upmap-items 2.1d 0 7 4 2 6 1 2024-02-13T00:52:52.522 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:52.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:52.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:52.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:52:52.654 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:51.206272+0000 mgr.y (mgr.14544) 611 : audit [DBG] from='client.38123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:52:52.655 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:51.560738+0000 mon.a (mon.0) 3529 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:52:52.655 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:51.566208+0000 mon.a (mon.0) 3530 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:52:52.655 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:51.566281+0000 mon.a (mon.0) 3531 : cluster [DBG] osdmap e736: 8 total, 8 up, 4 in 2024-02-13T00:52:52.655 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:51.206272+0000 mgr.y (mgr.14544) 611 : audit [DBG] from='client.38123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:52:52.655 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:51.560738+0000 mon.a (mon.0) 3529 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:52:52.655 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:51.566208+0000 mon.a (mon.0) 3530 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:52:52.655 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:51.566281+0000 mon.a (mon.0) 3531 : cluster [DBG] osdmap e736: 8 total, 8 up, 4 in 2024-02-13T00:52:52.656 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:51.206272+0000 mgr.y (mgr.14544) 611 : audit [DBG] from='client.38123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:52:52.656 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:51.560738+0000 mon.a (mon.0) 3529 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:52:52.656 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:51.566208+0000 mon.a (mon.0) 3530 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:52:52.656 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:51.566281+0000 mon.a (mon.0) 3531 : cluster [DBG] osdmap e736: 8 total, 8 up, 4 in 2024-02-13T00:52:52.657 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:52.678 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:52:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:52.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:52:52.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:52.809 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:53.258 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:53.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:53.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:53.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:53.390 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:53.579 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:52.529025+0000 mgr.y (mgr.14544) 612 : cluster [DBG] pgmap v1115: 204 pgs: 204 active+clean; 3.9 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 6.6 KiB/s wr, 2 op/s 2024-02-13T00:52:53.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:52.529025+0000 mgr.y (mgr.14544) 612 : cluster [DBG] pgmap v1115: 204 pgs: 204 active+clean; 3.9 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 6.6 KiB/s wr, 2 op/s 2024-02-13T00:52:53.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:52.529025+0000 mgr.y (mgr.14544) 612 : cluster [DBG] pgmap v1115: 204 pgs: 204 active+clean; 3.9 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 6.6 KiB/s wr, 2 op/s 2024-02-13T00:52:53.938 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:53.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:53.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:54.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:54.070 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:55.219 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:55.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:55.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:55.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:52:55.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:55.352 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:55.502 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:55.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:55.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:55.636 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:55.657 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:55.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:52:55.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.710+0000 7faf16619700 1 -- 172.21.15.162:0/1276765560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf100fcb30 msgr2=0x7faf100fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.710+0000 7faf16619700 1 --2- 172.21.15.162:0/1276765560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf100fcb30 0x7faf100fcf10 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7faf000047b0 tx=0x7faf00009cb0).stop 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.710+0000 7faf16619700 1 -- 172.21.15.162:0/1276765560 shutdown_connections 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.710+0000 7faf16619700 1 --2- 172.21.15.162:0/1276765560 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf100f73a0 0x7faf101039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.710+0000 7faf16619700 1 --2- 172.21.15.162:0/1276765560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf100fcb30 0x7faf100fcf10 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.710+0000 7faf16619700 1 --2- 172.21.15.162:0/1276765560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf100fd490 0x7faf100f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.710+0000 7faf16619700 1 -- 172.21.15.162:0/1276765560 >> 172.21.15.162:0/1276765560 conn(0x7faf1006eb90 msgr2=0x7faf100f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.710+0000 7faf16619700 1 -- 172.21.15.162:0/1276765560 shutdown_connections 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.710+0000 7faf16619700 1 -- 172.21.15.162:0/1276765560 wait complete. 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.711+0000 7faf16619700 1 librados: starting msgr at 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.711+0000 7faf16619700 1 librados: starting objecter 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.711+0000 7faf16619700 1 Processor -- start 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.711+0000 7faf16619700 1 -- start start 2024-02-13T00:52:55.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.711+0000 7faf16619700 1 librados: setting wanted keys 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.711+0000 7faf16619700 1 librados: calling monclient init 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.711+0000 7faf16619700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf100f73a0 0x7faf1010e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.711+0000 7faf16619700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf100fcb30 0x7faf1010e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.711+0000 7faf16619700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf100fd490 0x7faf10108240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.711+0000 7faf16619700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf10108730 con 0x7faf100fd490 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.711+0000 7faf16619700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf101088a0 con 0x7faf100f73a0 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.711+0000 7faf16619700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf10108a60 con 0x7faf100fcb30 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf14bb6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf100fd490 0x7faf10108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf0f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf100fcb30 0x7faf1010e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf14bb6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf100fd490 0x7faf10108240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43136/0 (socket says 172.21.15.162:43136) 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf0f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf100fcb30 0x7faf1010e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:51596/0 (socket says 172.21.15.162:51596) 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf14bb6700 1 -- 172.21.15.162:0/2189710663 learned_addr learned my addr 172.21.15.162:0/2189710663 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf0ffff700 1 --2- 172.21.15.162:0/2189710663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf100f73a0 0x7faf1010e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:55.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf14bb6700 1 -- 172.21.15.162:0/2189710663 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf100fcb30 msgr2=0x7faf1010e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf14bb6700 1 --2- 172.21.15.162:0/2189710663 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faf100fcb30 0x7faf1010e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf14bb6700 1 -- 172.21.15.162:0/2189710663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf100f73a0 msgr2=0x7faf1010e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf14bb6700 1 --2- 172.21.15.162:0/2189710663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf100f73a0 0x7faf1010e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf14bb6700 1 -- 172.21.15.162:0/2189710663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf00005040 con 0x7faf100fd490 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf14bb6700 1 --2- 172.21.15.162:0/2189710663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf100fd490 0x7faf10108240 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7faf0800eaf0 tx=0x7faf0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf0d7fa700 10 client.?.objecter ms_handle_connect 0x7faf100fd490 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf0d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf08009860 con 0x7faf100fd490 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf0800f040 con 0x7faf100fd490 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf08007de0 con 0x7faf100fd490 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf10108ce0 con 0x7faf100fd490 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf16619700 10 client.28032.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf10114ed0 con 0x7faf100fd490 2024-02-13T00:52:55.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.712+0000 7faf16619700 1 librados: init done 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.713+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7faf10106090 con 0x7faf100fd490 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.713+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7faf080092e0 con 0x7faf100fd490 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf0d7fa700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6800/2866534053 conn(0x7faef8062c90 0x7faef8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(737..737 src has 449..737) v4 ==== 8582+0+0 (secure 0 0 0) 0x7faf0807d060 con 0x7faf100fd490 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 osd_map(737..737 src has 449..737) v4 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf0d7fa700 3 client.28032.objecter handle_osd_map got epochs [737,737] > 0 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf0d7fa700 3 client.28032.objecter handle_osd_map decoding full epoch 737 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf0ffff700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6800/2866534053 conn(0x7faef8062c90 0x7faef8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf0d7fa700 20 client.28032.objecter dump_active .. 0 homeless 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v737) v1 ==== 54+0+16 (secure 0 0 0) 0x7faf08084070 con 0x7faf100fd490 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf16619700 10 client.28032.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 1 2024-02-13T00:52:55.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf0d7fa700 10 client.28032.objecter ms_handle_connect 0x7faef8062c90 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf0ffff700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6800/2866534053 conn(0x7faef8062c90 0x7faef8065150 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 rx=0x7faf00005010 tx=0x7faf00007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6801/2103743338 conn(0x7faf100ffe10 0x7faf101001f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10109720 osd=0 3 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10109720 osd=0 3 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10109720 osd=0 3 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 1 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf10109720 osd=0 3 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 0 1 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10109720 osd=0 4 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf16619700 10 client.28032.objecter _send_command 1 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf101006e0 con 0x7faf100ffe10 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.714+0000 7faf0f7fe700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6801/2103743338 conn(0x7faf100ffe10 0x7faf101001f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.715+0000 7faf0d7fa700 10 client.28032.objecter ms_handle_connect 0x7faf100ffe10 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.715+0000 7faf0f7fe700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6801/2103743338 conn(0x7faf100ffe10 0x7faf101001f0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.715+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faf101006e0 con 0x7faf100ffe10 2024-02-13T00:52:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.715+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:52:55.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.715+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 1 = system:0 2024-02-13T00:52:55.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.715+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf10109720 osd=0 4 2024-02-13T00:52:55.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.715+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 0 1 2024-02-13T00:52:55.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.730+0000 7faf16619700 10 client.28032.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:55.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.730+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 2 2024-02-13T00:52:55.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.730+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10109720 osd=0 3 2024-02-13T00:52:55.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.730+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10109720 osd=0 3 2024-02-13T00:52:55.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.730+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10109720 osd=0 3 2024-02-13T00:52:55.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.730+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 2 2024-02-13T00:52:55.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.730+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf10109720 osd=0 3 2024-02-13T00:52:55.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.730+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 0 2 2024-02-13T00:52:55.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.730+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10109720 osd=0 4 2024-02-13T00:52:55.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.730+0000 7faf16619700 10 client.28032.objecter _send_command 2 2024-02-13T00:52:55.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.730+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf101008a0 con 0x7faf100ffe10 2024-02-13T00:52:55.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faf101008a0 con 0x7faf100ffe10 2024-02-13T00:52:55.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:55.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf10109720 osd=0 4 2024-02-13T00:52:55.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 0 2 2024-02-13T00:52:55.735 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:52:55.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf16619700 10 client.28032.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:55.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 3 2024-02-13T00:52:55.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6805/150599780 conn(0x7faf10028590 0x7faf10119080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:55.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf100619f0 osd=1 3 2024-02-13T00:52:55.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf100619f0 osd=1 3 2024-02-13T00:52:55.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf100619f0 osd=1 3 2024-02-13T00:52:55.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 3 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf100619f0 osd=1 3 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 1 3 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf100619f0 osd=1 4 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf16619700 10 client.28032.objecter _send_command 3 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.731+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf10061b60 con 0x7faf10028590 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.732+0000 7faf0ffff700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6805/150599780 conn(0x7faf10028590 0x7faf10119080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.732+0000 7faf0d7fa700 10 client.28032.objecter ms_handle_connect 0x7faf10028590 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.732+0000 7faf0ffff700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6805/150599780 conn(0x7faf10028590 0x7faf10119080 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.732+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faf10061b60 con 0x7faf10028590 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.732+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.732+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 3 = system:0 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.732+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf100619f0 osd=1 4 2024-02-13T00:52:55.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.732+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 1 3 2024-02-13T00:52:55.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.747+0000 7faf16619700 10 client.28032.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:55.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.747+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 4 2024-02-13T00:52:55.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.747+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf100619f0 osd=1 3 2024-02-13T00:52:55.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.747+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf100619f0 osd=1 3 2024-02-13T00:52:55.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.747+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf100619f0 osd=1 3 2024-02-13T00:52:55.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.747+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 4 2024-02-13T00:52:55.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.747+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf100619f0 osd=1 3 2024-02-13T00:52:55.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.747+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 1 4 2024-02-13T00:52:55.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.747+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf100619f0 osd=1 4 2024-02-13T00:52:55.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.747+0000 7faf16619700 10 client.28032.objecter _send_command 4 2024-02-13T00:52:55.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.747+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf1005a0e0 con 0x7faf10028590 2024-02-13T00:52:55.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.748+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faf1005a0e0 con 0x7faf10028590 2024-02-13T00:52:55.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.748+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:55.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.748+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.748+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf100619f0 osd=1 4 2024-02-13T00:52:55.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.748+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 1 4 2024-02-13T00:52:55.752 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:52:55.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.748+0000 7faf16619700 10 client.28032.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:55.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.748+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 5 2024-02-13T00:52:55.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6809/3689719376 conn(0x7faf10059570 0x7faf1011c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:55.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10059400 osd=2 3 2024-02-13T00:52:55.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10059400 osd=2 3 2024-02-13T00:52:55.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10059400 osd=2 3 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 5 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf10059400 osd=2 3 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 2 5 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10059400 osd=2 4 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf16619700 10 client.28032.objecter _send_command 5 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf10028270 con 0x7faf10059570 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf0f7fe700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6809/3689719376 conn(0x7faf10059570 0x7faf1011c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf0d7fa700 10 client.28032.objecter ms_handle_connect 0x7faf10059570 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.749+0000 7faf0f7fe700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6809/3689719376 conn(0x7faf10059570 0x7faf1011c700 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.750+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faf10028270 con 0x7faf10059570 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.750+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.750+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 5 = system:0 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.750+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf10059400 osd=2 4 2024-02-13T00:52:55.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.750+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 2 5 2024-02-13T00:52:55.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:55.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.765+0000 7faf16619700 10 client.28032.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:55.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.765+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 6 2024-02-13T00:52:55.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.765+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10059400 osd=2 3 2024-02-13T00:52:55.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.765+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10059400 osd=2 3 2024-02-13T00:52:55.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.765+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10059400 osd=2 3 2024-02-13T00:52:55.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.765+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 6 2024-02-13T00:52:55.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.765+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf10059400 osd=2 3 2024-02-13T00:52:55.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.765+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 2 6 2024-02-13T00:52:55.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.765+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10059400 osd=2 4 2024-02-13T00:52:55.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.765+0000 7faf16619700 10 client.28032.objecter _send_command 6 2024-02-13T00:52:55.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.765+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf10027650 con 0x7faf10059570 2024-02-13T00:52:55.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faf10027650 con 0x7faf10059570 2024-02-13T00:52:55.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:55.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf10059400 osd=2 4 2024-02-13T00:52:55.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 2 6 2024-02-13T00:52:55.771 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf16619700 10 client.28032.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 7 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6813/604971011 conn(0x7faf1011dd70 0x7faf10120160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10119d40 osd=3 3 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10119d40 osd=3 3 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10119d40 osd=3 3 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 7 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf10119d40 osd=3 3 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 3 7 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10119d40 osd=3 4 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf16619700 10 client.28032.objecter _send_command 7 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf1011a3c0 con 0x7faf1011dd70 2024-02-13T00:52:55.772 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf14bb6700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6813/604971011 conn(0x7faf1011dd70 0x7faf10120160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:55.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf0d7fa700 10 client.28032.objecter ms_handle_connect 0x7faf1011dd70 2024-02-13T00:52:55.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.767+0000 7faf14bb6700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6813/604971011 conn(0x7faf1011dd70 0x7faf10120160 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:55.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.768+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faf1011a3c0 con 0x7faf1011dd70 2024-02-13T00:52:55.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.768+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:52:55.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.768+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 7 = system:0 2024-02-13T00:52:55.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.768+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf10119d40 osd=3 4 2024-02-13T00:52:55.773 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.768+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 3 7 2024-02-13T00:52:55.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:54.529526+0000 mgr.y (mgr.14544) 613 : cluster [DBG] pgmap v1116: 204 pgs: 204 active+clean; 3.9 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 6.6 KiB/s wr, 1 op/s 2024-02-13T00:52:55.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:54.655231+0000 mon.a (mon.0) 3532 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:55.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:54.655556+0000 mon.a (mon.0) 3533 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-130"}]: dispatch 2024-02-13T00:52:55.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:55.557175+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-130"}]': finished 2024-02-13T00:52:55.787 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:55.557243+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e737: 8 total, 8 up, 4 in 2024-02-13T00:52:55.788 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:54.529526+0000 mgr.y (mgr.14544) 613 : cluster [DBG] pgmap v1116: 204 pgs: 204 active+clean; 3.9 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 6.6 KiB/s wr, 1 op/s 2024-02-13T00:52:55.788 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:54.655231+0000 mon.a (mon.0) 3532 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:55.788 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:54.655556+0000 mon.a (mon.0) 3533 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-130"}]: dispatch 2024-02-13T00:52:55.788 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:55.557175+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-130"}]': finished 2024-02-13T00:52:55.788 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:55.557243+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e737: 8 total, 8 up, 4 in 2024-02-13T00:52:55.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.785+0000 7faf16619700 10 client.28032.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:55.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.785+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 8 2024-02-13T00:52:55.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.785+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10119d40 osd=3 3 2024-02-13T00:52:55.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.785+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10119d40 osd=3 3 2024-02-13T00:52:55.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.785+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10119d40 osd=3 3 2024-02-13T00:52:55.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.785+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 8 2024-02-13T00:52:55.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.785+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf10119d40 osd=3 3 2024-02-13T00:52:55.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.785+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 3 8 2024-02-13T00:52:55.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.785+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10119d40 osd=3 4 2024-02-13T00:52:55.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.785+0000 7faf16619700 10 client.28032.objecter _send_command 8 2024-02-13T00:52:55.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.785+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf101197b0 con 0x7faf1011dd70 2024-02-13T00:52:55.790 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:55.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faf101197b0 con 0x7faf1011dd70 2024-02-13T00:52:55.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:55.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf10119d40 osd=3 4 2024-02-13T00:52:55.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 3 8 2024-02-13T00:52:55.791 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:52:55.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf16619700 10 client.28032.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:55.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 9 2024-02-13T00:52:55.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6800/3344263388 conn(0x7faf1011d8d0 0x7faf101237e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:55.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf101197b0 osd=4 3 2024-02-13T00:52:55.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf101197b0 osd=4 3 2024-02-13T00:52:55.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf101197b0 osd=4 3 2024-02-13T00:52:55.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 9 2024-02-13T00:52:55.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf101197b0 osd=4 3 2024-02-13T00:52:55.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 4 9 2024-02-13T00:52:55.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf101197b0 osd=4 4 2024-02-13T00:52:55.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf16619700 10 client.28032.objecter _send_command 9 2024-02-13T00:52:55.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.787+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf101199b0 con 0x7faf1011d8d0 2024-02-13T00:52:55.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.788+0000 7faf0ffff700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6800/3344263388 conn(0x7faf1011d8d0 0x7faf101237e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:55.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.788+0000 7faf0d7fa700 10 client.28032.objecter ms_handle_connect 0x7faf1011d8d0 2024-02-13T00:52:55.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.788+0000 7faf0ffff700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6800/3344263388 conn(0x7faf1011d8d0 0x7faf101237e0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:55.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.789+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faf101199b0 con 0x7faf1011d8d0 2024-02-13T00:52:55.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.789+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:52:55.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.789+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 9 = system:0 2024-02-13T00:52:55.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.789+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf101197b0 osd=4 4 2024-02-13T00:52:55.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.789+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 4 9 2024-02-13T00:52:55.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.810+0000 7faf16619700 10 client.28032.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:55.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.810+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 10 2024-02-13T00:52:55.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.810+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf101197b0 osd=4 3 2024-02-13T00:52:55.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.810+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf101197b0 osd=4 3 2024-02-13T00:52:55.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.810+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf101197b0 osd=4 3 2024-02-13T00:52:55.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.810+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 10 2024-02-13T00:52:55.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.810+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf101197b0 osd=4 3 2024-02-13T00:52:55.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.810+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 4 10 2024-02-13T00:52:55.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.810+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf101197b0 osd=4 4 2024-02-13T00:52:55.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.810+0000 7faf16619700 10 client.28032.objecter _send_command 10 2024-02-13T00:52:55.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.810+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf10119bd0 con 0x7faf1011d8d0 2024-02-13T00:52:55.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faf10119bd0 con 0x7faf1011d8d0 2024-02-13T00:52:55.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:55.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf101197b0 osd=4 4 2024-02-13T00:52:55.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 4 10 2024-02-13T00:52:55.816 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf16619700 10 client.28032.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 11 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6804/2277917372 conn(0x7faf10124220 0x7faf10124680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf101199b0 osd=5 3 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf101199b0 osd=5 3 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf101199b0 osd=5 3 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 11 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf101199b0 osd=5 3 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 5 11 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf101199b0 osd=5 4 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf16619700 10 client.28032.objecter _send_command 11 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf10119bd0 con 0x7faf10124220 2024-02-13T00:52:55.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf0f7fe700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6804/2277917372 conn(0x7faf10124220 0x7faf10124680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:55.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf0d7fa700 10 client.28032.objecter ms_handle_connect 0x7faf10124220 2024-02-13T00:52:55.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.812+0000 7faf0f7fe700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6804/2277917372 conn(0x7faf10124220 0x7faf10124680 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:55.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.813+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faf10119bd0 con 0x7faf10124220 2024-02-13T00:52:55.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.813+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:52:55.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.813+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 11 = system:0 2024-02-13T00:52:55.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.813+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf101199b0 osd=5 4 2024-02-13T00:52:55.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.813+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 5 11 2024-02-13T00:52:55.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.827+0000 7faf16619700 10 client.28032.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:55.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.827+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 12 2024-02-13T00:52:55.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.827+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf101199b0 osd=5 3 2024-02-13T00:52:55.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.827+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf101199b0 osd=5 3 2024-02-13T00:52:55.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.827+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf101199b0 osd=5 3 2024-02-13T00:52:55.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.827+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 12 2024-02-13T00:52:55.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.827+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf101199b0 osd=5 3 2024-02-13T00:52:55.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.827+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 5 12 2024-02-13T00:52:55.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.827+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf101199b0 osd=5 4 2024-02-13T00:52:55.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.827+0000 7faf16619700 10 client.28032.objecter _send_command 12 2024-02-13T00:52:55.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.827+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf10119bd0 con 0x7faf10124220 2024-02-13T00:52:55.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.829+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faf10119bd0 con 0x7faf10124220 2024-02-13T00:52:55.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.829+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:55.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.829+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.829+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf101199b0 osd=5 4 2024-02-13T00:52:55.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.829+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 5 12 2024-02-13T00:52:55.833 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:52:55.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.829+0000 7faf16619700 10 client.28032.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:55.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.829+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 13 2024-02-13T00:52:55.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6808/1506640341 conn(0x7faf10125140 0x7faf101255a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:55.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10119bd0 osd=6 3 2024-02-13T00:52:55.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10119bd0 osd=6 3 2024-02-13T00:52:55.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10119bd0 osd=6 3 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 13 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf10119bd0 osd=6 3 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 6 13 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10119bd0 osd=6 4 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf16619700 10 client.28032.objecter _send_command 13 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf10125be0 con 0x7faf10125140 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf14bb6700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6808/1506640341 conn(0x7faf10125140 0x7faf101255a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf0d7fa700 10 client.28032.objecter ms_handle_connect 0x7faf10125140 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.830+0000 7faf14bb6700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6808/1506640341 conn(0x7faf10125140 0x7faf101255a0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.831+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faf10125be0 con 0x7faf10125140 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.831+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.831+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 13 = system:0 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.831+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf10119bd0 osd=6 4 2024-02-13T00:52:55.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.831+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 6 13 2024-02-13T00:52:55.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.845+0000 7faf16619700 10 client.28032.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:55.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.845+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 14 2024-02-13T00:52:55.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.845+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10119bd0 osd=6 3 2024-02-13T00:52:55.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.845+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10119bd0 osd=6 3 2024-02-13T00:52:55.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.845+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10119bd0 osd=6 3 2024-02-13T00:52:55.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.845+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 14 2024-02-13T00:52:55.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.845+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf10119bd0 osd=6 3 2024-02-13T00:52:55.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.845+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 6 14 2024-02-13T00:52:55.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.845+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10119bd0 osd=6 4 2024-02-13T00:52:55.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.845+0000 7faf16619700 10 client.28032.objecter _send_command 14 2024-02-13T00:52:55.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.845+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf10125be0 con 0x7faf10125140 2024-02-13T00:52:55.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faf10125be0 con 0x7faf10125140 2024-02-13T00:52:55.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:55.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf10119bd0 osd=6 4 2024-02-13T00:52:55.850 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 6 14 2024-02-13T00:52:55.851 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:52:55.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf16619700 10 client.28032.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:52:55.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 15 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6812/1964192772 conn(0x7faf10125e30 0x7faf1012c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10125be0 osd=7 3 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10125be0 osd=7 3 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10125be0 osd=7 3 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 15 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf10125be0 osd=7 3 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 7 15 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10125be0 osd=7 4 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf16619700 10 client.28032.objecter _send_command 15 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf1012f7e0 con 0x7faf10125e30 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf0ffff700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6812/1964192772 conn(0x7faf10125e30 0x7faf1012c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf0d7fa700 10 client.28032.objecter ms_handle_connect 0x7faf10125e30 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.847+0000 7faf0ffff700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6812/1964192772 conn(0x7faf10125e30 0x7faf1012c9e0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.848+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7faf1012f7e0 con 0x7faf10125e30 2024-02-13T00:52:55.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.848+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:52:55.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.848+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 15 = system:0 2024-02-13T00:52:55.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.848+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf10125be0 osd=7 4 2024-02-13T00:52:55.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.848+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 7 15 2024-02-13T00:52:55.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.863+0000 7faf16619700 10 client.28032.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:52:55.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.863+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign -1 16 2024-02-13T00:52:55.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.863+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10125be0 osd=7 3 2024-02-13T00:52:55.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.863+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10125be0 osd=7 3 2024-02-13T00:52:55.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.863+0000 7faf16619700 20 client.28032.objecter _get_session s=0x7faf10125be0 osd=7 3 2024-02-13T00:52:55.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.863+0000 7faf16619700 15 client.28032.objecter _session_command_op_remove -1 16 2024-02-13T00:52:55.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.863+0000 7faf16619700 20 client.28032.objecter get_session s=0x7faf10125be0 osd=7 3 2024-02-13T00:52:55.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.863+0000 7faf16619700 15 client.28032.objecter _session_command_op_assign 7 16 2024-02-13T00:52:55.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.863+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10125be0 osd=7 4 2024-02-13T00:52:55.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.863+0000 7faf16619700 10 client.28032.objecter _send_command 16 2024-02-13T00:52:55.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.863+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faf1012f7e0 con 0x7faf10125e30 2024-02-13T00:52:55.868 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:55.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf0d7fa700 1 -- 172.21.15.162:0/2189710663 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7faf1012f7e0 con 0x7faf10125e30 2024-02-13T00:52:55.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf0d7fa700 10 client.28032.objecter ms_dispatch 0x7faf10001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:52:55.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf0d7fa700 10 client.28032.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf0d7fa700 20 client.28032.objecter put_session s=0x7faf10125be0 osd=7 4 2024-02-13T00:52:55.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf0d7fa700 15 client.28032.objecter _session_command_op_remove 7 16 2024-02-13T00:52:55.869 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 10 librados: watch_flush enter 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 10 librados: watch_flush exit 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 10 client.28032.objecter close_session for osd.0 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6801/2103743338 conn(0x7faf100ffe10 msgr2=0x7faf101001f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6801/2103743338 conn(0x7faf100ffe10 0x7faf101001f0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10109720 osd=0 1 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 10 client.28032.objecter close_session for osd.1 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6805/150599780 conn(0x7faf10028590 msgr2=0x7faf10119080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6805/150599780 conn(0x7faf10028590 0x7faf10119080 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf100619f0 osd=1 1 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 10 client.28032.objecter close_session for osd.2 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6809/3689719376 conn(0x7faf10059570 msgr2=0x7faf1011c700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6809/3689719376 conn(0x7faf10059570 0x7faf1011c700 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10059400 osd=2 1 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 10 client.28032.objecter close_session for osd.3 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6813/604971011 conn(0x7faf1011dd70 msgr2=0x7faf10120160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6813/604971011 conn(0x7faf1011dd70 0x7faf10120160 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10119d40 osd=3 1 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 10 client.28032.objecter close_session for osd.4 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6800/3344263388 conn(0x7faf1011d8d0 msgr2=0x7faf101237e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6800/3344263388 conn(0x7faf1011d8d0 0x7faf101237e0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf101197b0 osd=4 1 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 10 client.28032.objecter close_session for osd.5 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6804/2277917372 conn(0x7faf10124220 msgr2=0x7faf10124680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6804/2277917372 conn(0x7faf10124220 0x7faf10124680 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf101199b0 osd=5 1 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 10 client.28032.objecter close_session for osd.6 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6808/1506640341 conn(0x7faf10125140 msgr2=0x7faf101255a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6808/1506640341 conn(0x7faf10125140 0x7faf101255a0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10119bd0 osd=6 1 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 10 client.28032.objecter close_session for osd.7 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6812/1964192772 conn(0x7faf10125e30 msgr2=0x7faf1012c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.181:6812/1964192772 conn(0x7faf10125e30 0x7faf1012c9e0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 20 client.28032.objecter put_session s=0x7faf10125be0 osd=7 1 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 20 client.28032.objecter shutdown clearing up homeless session... 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 10 client.28032.objecter successfully canceled tick 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6800/2866534053 conn(0x7faef8062c90 msgr2=0x7faef8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> v2:172.21.15.162:6800/2866534053 conn(0x7faef8062c90 0x7faef8065150 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 rx=0x7faf00005010 tx=0x7faf00007f90).stop 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf100fd490 msgr2=0x7faf10108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf16619700 1 --2- 172.21.15.162:0/2189710663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf100fd490 0x7faf10108240 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7faf0800eaf0 tx=0x7faf0800b040).stop 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.865+0000 7faf14bb6700 1 -- 172.21.15.162:0/2189710663 reap_dead start 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.866+0000 7faf14bb6700 1 -- 172.21.15.162:0/2189710663 reap_dead start 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.866+0000 7faf14bb6700 1 -- 172.21.15.162:0/2189710663 reap_dead start 2024-02-13T00:52:55.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.866+0000 7faf14bb6700 1 -- 172.21.15.162:0/2189710663 reap_dead start 2024-02-13T00:52:55.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.866+0000 7faf14bb6700 1 -- 172.21.15.162:0/2189710663 reap_dead start 2024-02-13T00:52:55.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.866+0000 7faf14bb6700 1 -- 172.21.15.162:0/2189710663 reap_dead start 2024-02-13T00:52:55.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.866+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 shutdown_connections 2024-02-13T00:52:55.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.866+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 >> 172.21.15.162:0/2189710663 conn(0x7faf1006eb90 msgr2=0x7faf10060e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:52:55.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.866+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 shutdown_connections 2024-02-13T00:52:55.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.866+0000 7faf16619700 1 -- 172.21.15.162:0/2189710663 wait complete. 2024-02-13T00:52:55.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:52:55.866+0000 7faf16619700 1 librados: shutdown 2024-02-13T00:52:55.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:55.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:55.874 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.874 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.874 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.874 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.875 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.875 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.875 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.875 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:52:55.928 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:54.529526+0000 mgr.y (mgr.14544) 613 : cluster [DBG] pgmap v1116: 204 pgs: 204 active+clean; 3.9 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 6.6 KiB/s wr, 1 op/s 2024-02-13T00:52:55.928 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:54.655231+0000 mon.a (mon.0) 3532 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:52:55.928 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:54.655556+0000 mon.a (mon.0) 3533 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-130"}]: dispatch 2024-02-13T00:52:55.928 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:55.557175+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-130"}]': finished 2024-02-13T00:52:55.928 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:55.557243+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e737: 8 total, 8 up, 4 in 2024-02-13T00:52:55.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:52:55.993 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:52:55.993 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:52:56.003 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:56.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:52:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:52:56.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:52:56.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:56.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:52:56.422 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:52:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:52:56] "GET /metrics HTTP/1.1" 200 189881 "" "Prometheus/2.33.4" 2024-02-13T00:52:56.457 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:56.486 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:55.714829+0000 mon.a (mon.0) 3536 : audit [DBG] from='client.? 172.21.15.162:0/2189710663' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:56.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:55.925104+0000 mgr.y (mgr.14544) 614 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:55.714829+0000 mon.a (mon.0) 3536 : audit [DBG] from='client.? 172.21.15.162:0/2189710663' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:56.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:55.925104+0000 mgr.y (mgr.14544) 614 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:56.834 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:56.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:56.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:56.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-13T00:52:56.890 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:55.714829+0000 mon.a (mon.0) 3536 : audit [DBG] from='client.? 172.21.15.162:0/2189710663' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:52:56.890 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:55.925104+0000 mgr.y (mgr.14544) 614 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:52:56.915 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:56.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:52:56.965 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:57.132 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:52:57.567 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:52:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:57.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:52:57.591 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:52:57.625 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:56.529964+0000 mgr.y (mgr.14544) 615 : cluster [DBG] pgmap v1118: 204 pgs: 204 active+clean; 3.9 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-02-13T00:52:57.625 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:56.583279+0000 mon.a (mon.0) 3537 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:52:57.625 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:56.596534+0000 mon.a (mon.0) 3538 : cluster [DBG] osdmap e738: 8 total, 8 up, 4 in 2024-02-13T00:52:57.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:57.702 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:57.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:56.529964+0000 mgr.y (mgr.14544) 615 : cluster [DBG] pgmap v1118: 204 pgs: 204 active+clean; 3.9 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-02-13T00:52:57.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:56.583279+0000 mon.a (mon.0) 3537 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:52:57.805 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:56.596534+0000 mon.a (mon.0) 3538 : cluster [DBG] osdmap e738: 8 total, 8 up, 4 in 2024-02-13T00:52:57.805 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:56.529964+0000 mgr.y (mgr.14544) 615 : cluster [DBG] pgmap v1118: 204 pgs: 204 active+clean; 3.9 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-02-13T00:52:57.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:56.583279+0000 mon.a (mon.0) 3537 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:52:57.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:56.596534+0000 mon.a (mon.0) 3538 : cluster [DBG] osdmap e738: 8 total, 8 up, 4 in 2024-02-13T00:52:57.951 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:57.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:58.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:58.084 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:58.102 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:58.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:58.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:52:58.232 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:58.581 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:52:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:58.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:52:58.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:52:58.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:57.600503+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e739: 8 total, 8 up, 4 in 2024-02-13T00:52:58.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:57.601877+0000 mon.a (mon.0) 3540 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:58.709 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:57.600503+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e739: 8 total, 8 up, 4 in 2024-02-13T00:52:58.709 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:57.601877+0000 mon.a (mon.0) 3540 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:58.712 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:58.730 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:52:58.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:58.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:52:58.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:57.600503+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e739: 8 total, 8 up, 4 in 2024-02-13T00:52:58.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:57.601877+0000 mon.a (mon.0) 3540 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:52:58.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:52:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:52:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:52:58.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:52:58.860 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:59.008 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:52:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:59.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:52:59.112 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:52:59.126 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:52:59.126 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:52:59.141 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:59.341 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:52:59.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:59.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:52:59.444 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:52:59.471 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:59.503 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:52:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:58.530564+0000 mgr.y (mgr.14544) 616 : cluster [DBG] pgmap v1121: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:52:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:58.606589+0000 mon.a (mon.0) 3541 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:58.606739+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e740: 8 total, 8 up, 4 in 2024-02-13T00:52:59.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:52:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:58.647325+0000 mon.a (mon.0) 3543 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:59.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-13T00:52:59.820 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:52:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:59.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:52:59.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:58.530564+0000 mgr.y (mgr.14544) 616 : cluster [DBG] pgmap v1121: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:52:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:58.606589+0000 mon.a (mon.0) 3541 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:58.606739+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e740: 8 total, 8 up, 4 in 2024-02-13T00:52:59.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:52:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:58.647325+0000 mon.a (mon.0) 3543 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:59.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:58.530564+0000 mgr.y (mgr.14544) 616 : cluster [DBG] pgmap v1121: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:52:59.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:58.606589+0000 mon.a (mon.0) 3541 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:52:59.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:58.606739+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e740: 8 total, 8 up, 4 in 2024-02-13T00:52:59.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:52:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:58.647325+0000 mon.a (mon.0) 3543 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:52:59.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:52:59.950 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:52:59.970 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:52:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:52:59.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:00.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:00.104 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:00.130 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:00.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:00.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:00.134 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:53:00.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 -- 172.21.15.162:0/4083390699 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b40f8500 msgr2=0x7f64b4104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:00.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 --2- 172.21.15.162:0/4083390699 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b40f8500 0x7f64b4104bd0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 rx=0x7f64ac009be0 tx=0x7f64ac004d10).stop 2024-02-13T00:53:00.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 -- 172.21.15.162:0/4083390699 shutdown_connections 2024-02-13T00:53:00.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 --2- 172.21.15.162:0/4083390699 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b40f8500 0x7f64b4104bd0 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:00.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 --2- 172.21.15.162:0/4083390699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b40f9b70 0x7f64b40f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:00.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 --2- 172.21.15.162:0/4083390699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64b40fa440 0x7f64b40f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:00.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 -- 172.21.15.162:0/4083390699 >> 172.21.15.162:0/4083390699 conn(0x7f64b40f38c0 msgr2=0x7f64b40f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:00.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 -- 172.21.15.162:0/4083390699 shutdown_connections 2024-02-13T00:53:00.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 -- 172.21.15.162:0/4083390699 wait complete. 2024-02-13T00:53:00.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 librados: starting msgr at 2024-02-13T00:53:00.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 librados: starting objecter 2024-02-13T00:53:00.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 Processor -- start 2024-02-13T00:53:00.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 -- start start 2024-02-13T00:53:00.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 librados: setting wanted keys 2024-02-13T00:53:00.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.142+0000 7f64ba61f700 1 librados: calling monclient init 2024-02-13T00:53:00.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64ba61f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b40f8500 0x7f64b410dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:00.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64ba61f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64b40f9b70 0x7f64b410e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:00.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64ba61f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b40fa440 0x7f64b4108180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:00.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64ba61f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b4108670 con 0x7f64b40f8500 2024-02-13T00:53:00.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64ba61f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b41087e0 con 0x7f64b40f9b70 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64ba61f700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b41089a0 con 0x7f64b40fa440 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b8bbc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b40fa440 0x7f64b4108180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b3fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b40f8500 0x7f64b410dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b8bbc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b40fa440 0x7f64b4108180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:51612/0 (socket says 172.21.15.162:51612) 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b8bbc700 1 -- 172.21.15.162:0/3972216991 learned_addr learned my addr 172.21.15.162:0/3972216991 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b3fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b40f8500 0x7f64b410dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43148/0 (socket says 172.21.15.162:43148) 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b37fe700 1 --2- 172.21.15.162:0/3972216991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64b40f9b70 0x7f64b410e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b8bbc700 1 -- 172.21.15.162:0/3972216991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64b40f9b70 msgr2=0x7f64b410e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b8bbc700 1 --2- 172.21.15.162:0/3972216991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64b40f9b70 0x7f64b410e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b8bbc700 1 -- 172.21.15.162:0/3972216991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b40f8500 msgr2=0x7f64b410dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b8bbc700 1 --2- 172.21.15.162:0/3972216991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b40f8500 0x7f64b410dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b8bbc700 1 -- 172.21.15.162:0/3972216991 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ac0097c0 con 0x7f64b40fa440 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b3fff700 1 --2- 172.21.15.162:0/3972216991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b40f8500 0x7f64b410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:00.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b8bbc700 1 --2- 172.21.15.162:0/3972216991 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b40fa440 0x7f64b4108180 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 rx=0x7f64ac004a30 tx=0x7f64ac004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b17fa700 10 client.?.objecter ms_handle_connect 0x7f64b40fa440 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b17fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b17fa700 1 -- 172.21.15.162:0/3972216991 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00e040 con 0x7f64b40fa440 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b17fa700 1 -- 172.21.15.162:0/3972216991 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f64ac018040 con 0x7f64b40fa440 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64ba61f700 1 -- 172.21.15.162:0/3972216991 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b4108c20 con 0x7f64b40fa440 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.143+0000 7f64b17fa700 1 -- 172.21.15.162:0/3972216991 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac01c3f0 con 0x7f64b40fa440 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.144+0000 7f64ba61f700 10 client.38135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.144+0000 7f64ba61f700 1 -- 172.21.15.162:0/3972216991 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b4114e10 con 0x7f64b40fa440 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.144+0000 7f64ba61f700 1 librados: init done 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.144+0000 7f64ba61f700 1 -- 172.21.15.162:0/3972216991 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b40475e0 con 0x7f64b40fa440 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.145+0000 7f64b17fa700 1 -- 172.21.15.162:0/3972216991 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f64ac007b20 con 0x7f64b40fa440 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.145+0000 7f64b17fa700 1 --2- 172.21.15.162:0/3972216991 >> v2:172.21.15.162:6800/2866534053 conn(0x7f649c062c90 0x7f649c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.145+0000 7f64b17fa700 1 -- 172.21.15.162:0/3972216991 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(741..741 src has 449..741) v4 ==== 8590+0+0 (secure 0 0 0) 0x7f64ac07e440 con 0x7f64b40fa440 2024-02-13T00:53:00.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.145+0000 7f64b17fa700 10 client.38135.objecter ms_dispatch 0x7f64b4001550 osd_map(741..741 src has 449..741) v4 2024-02-13T00:53:00.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.145+0000 7f64b17fa700 3 client.38135.objecter handle_osd_map got epochs [741,741] > 0 2024-02-13T00:53:00.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.145+0000 7f64b17fa700 3 client.38135.objecter handle_osd_map decoding full epoch 741 2024-02-13T00:53:00.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.145+0000 7f64b3fff700 1 --2- 172.21.15.162:0/3972216991 >> v2:172.21.15.162:6800/2866534053 conn(0x7f649c062c90 0x7f649c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:00.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.145+0000 7f64b17fa700 20 client.38135.objecter dump_active .. 0 homeless 2024-02-13T00:53:00.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.146+0000 7f64b17fa700 10 client.38135.objecter ms_handle_connect 0x7f649c062c90 2024-02-13T00:53:00.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.146+0000 7f64b3fff700 1 --2- 172.21.15.162:0/3972216991 >> v2:172.21.15.162:6800/2866534053 conn(0x7f649c062c90 0x7f649c065150 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 rx=0x7f64a8007bf0 tx=0x7f64a800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:00.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.150+0000 7f64b17fa700 1 -- 172.21.15.162:0/3972216991 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f64ac0517b0 con 0x7f64b40fa440 2024-02-13T00:53:00.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:00.261 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:00.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.341+0000 7f64ba61f700 1 -- 172.21.15.162:0/3972216991 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "2.1d", "id": [0, 7, 4, 2, 6, 1]} v 0) v1 -- 0x7f64b4109660 con 0x7f64b40fa440 2024-02-13T00:53:00.460 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:00.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:00.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:00.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:00.592 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:00.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.615+0000 7f64b17fa700 1 -- 172.21.15.162:0/3972216991 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "2.1d", "id": [0, 7, 4, 2, 6, 1]}]=0 set 2.1d pg_upmap_items mapping to [0->7,4->2,6->1] v742) v1 ==== 159+0+0 (secure 0 0 0) 0x7f64ac0210b0 con 0x7f64b40fa440 2024-02-13T00:53:00.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.622+0000 7f64ba61f700 10 librados: watch_flush enter 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 10 librados: watch_flush exit 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 20 client.38135.objecter shutdown clearing up homeless session... 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 10 client.38135.objecter successfully canceled tick 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 -- 172.21.15.162:0/3972216991 >> v2:172.21.15.162:6800/2866534053 conn(0x7f649c062c90 msgr2=0x7f649c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 --2- 172.21.15.162:0/3972216991 >> v2:172.21.15.162:6800/2866534053 conn(0x7f649c062c90 0x7f649c065150 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 rx=0x7f64a8007bf0 tx=0x7f64a800b040).stop 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 -- 172.21.15.162:0/3972216991 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b40fa440 msgr2=0x7f64b4108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 --2- 172.21.15.162:0/3972216991 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b40fa440 0x7f64b4108180 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 rx=0x7f64ac004a30 tx=0x7f64ac004d10).stop 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 -- 172.21.15.162:0/3972216991 shutdown_connections 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 --2- 172.21.15.162:0/3972216991 >> v2:172.21.15.162:6800/2866534053 conn(0x7f649c062c90 0x7f649c065150 unknown :-1 s=CLOSED pgs=3020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 --2- 172.21.15.162:0/3972216991 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b40fa440 0x7f64b4108180 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 --2- 172.21.15.162:0/3972216991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b40f8500 0x7f64b410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 --2- 172.21.15.162:0/3972216991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64b40f9b70 0x7f64b410e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 -- 172.21.15.162:0/3972216991 >> 172.21.15.162:0/3972216991 conn(0x7f64b40f38c0 msgr2=0x7f64b40f4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:00.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 -- 172.21.15.162:0/3972216991 shutdown_connections 2024-02-13T00:53:00.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 -- 172.21.15.162:0/3972216991 wait complete. 2024-02-13T00:53:00.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:00.623+0000 7f64ba61f700 1 librados: shutdown 2024-02-13T00:53:00.628 INFO:teuthology.orchestra.run.smithi162.stderr:set 2.1d pg_upmap_items mapping to [0->7,4->2,6->1] 2024-02-13T00:53:00.973 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:59.609970+0000 mon.a (mon.0) 3544 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:53:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:52:59.610042+0000 mon.a (mon.0) 3545 : cluster [DBG] osdmap e741: 8 total, 8 up, 4 in 2024-02-13T00:53:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:52:59.613135+0000 mon.a (mon.0) 3546 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-132"}]: dispatch 2024-02-13T00:53:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:00.342518+0000 mon.c (mon.2) 501 : audit [INF] from='client.? 172.21.15.162:0/3972216991' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1d", "id": [0, 7, 4, 2, 6, 1]}]: dispatch 2024-02-13T00:53:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:00.342880+0000 mon.a (mon.0) 3547 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1d", "id": [0, 7, 4, 2, 6, 1]}]: dispatch 2024-02-13T00:53:01.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:59.609970+0000 mon.a (mon.0) 3544 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:53:01.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:52:59.610042+0000 mon.a (mon.0) 3545 : cluster [DBG] osdmap e741: 8 total, 8 up, 4 in 2024-02-13T00:53:01.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:52:59.613135+0000 mon.a (mon.0) 3546 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-132"}]: dispatch 2024-02-13T00:53:01.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:00.342518+0000 mon.c (mon.2) 501 : audit [INF] from='client.? 172.21.15.162:0/3972216991' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1d", "id": [0, 7, 4, 2, 6, 1]}]: dispatch 2024-02-13T00:53:01.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:00.342880+0000 mon.a (mon.0) 3547 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1d", "id": [0, 7, 4, 2, 6, 1]}]: dispatch 2024-02-13T00:53:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:59.609970+0000 mon.a (mon.0) 3544 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:53:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:52:59.610042+0000 mon.a (mon.0) 3545 : cluster [DBG] osdmap e741: 8 total, 8 up, 4 in 2024-02-13T00:53:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:52:59.613135+0000 mon.a (mon.0) 3546 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-132"}]: dispatch 2024-02-13T00:53:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:00.342518+0000 mon.c (mon.2) 501 : audit [INF] from='client.? 172.21.15.162:0/3972216991' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1d", "id": [0, 7, 4, 2, 6, 1]}]: dispatch 2024-02-13T00:53:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:00.342880+0000 mon.a (mon.0) 3547 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1d", "id": [0, 7, 4, 2, 6, 1]}]: dispatch 2024-02-13T00:53:01.441 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:01.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:01.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:01.576 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:01.597 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:01.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:01.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:01.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:01.728 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:01.766 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:01.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:01.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:01.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:00.531559+0000 mgr.y (mgr.14544) 617 : cluster [DBG] pgmap v1124: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:01.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:00.615589+0000 mon.a (mon.0) 3548 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-132"}]': finished 2024-02-13T00:53:01.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:00.615663+0000 mon.a (mon.0) 3549 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.1d", "id": [0, 7, 4, 2, 6, 1]}]': finished 2024-02-13T00:53:01.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:00.615722+0000 mon.a (mon.0) 3550 : cluster [DBG] osdmap e742: 8 total, 8 up, 4 in 2024-02-13T00:53:01.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:00.616694+0000 mon.a (mon.0) 3551 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-132", "mode": "writeback"}]: dispatch 2024-02-13T00:53:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:00.531559+0000 mgr.y (mgr.14544) 617 : cluster [DBG] pgmap v1124: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:00.615589+0000 mon.a (mon.0) 3548 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-132"}]': finished 2024-02-13T00:53:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:00.615663+0000 mon.a (mon.0) 3549 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.1d", "id": [0, 7, 4, 2, 6, 1]}]': finished 2024-02-13T00:53:01.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:00.615722+0000 mon.a (mon.0) 3550 : cluster [DBG] osdmap e742: 8 total, 8 up, 4 in 2024-02-13T00:53:01.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:00.616694+0000 mon.a (mon.0) 3551 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-132", "mode": "writeback"}]: dispatch 2024-02-13T00:53:01.869 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:01.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:00.531559+0000 mgr.y (mgr.14544) 617 : cluster [DBG] pgmap v1124: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:01.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:00.615589+0000 mon.a (mon.0) 3548 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-132"}]': finished 2024-02-13T00:53:01.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:00.615663+0000 mon.a (mon.0) 3549 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.1d", "id": [0, 7, 4, 2, 6, 1]}]': finished 2024-02-13T00:53:01.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:00.615722+0000 mon.a (mon.0) 3550 : cluster [DBG] osdmap e742: 8 total, 8 up, 4 in 2024-02-13T00:53:01.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:00.616694+0000 mon.a (mon.0) 3551 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-132", "mode": "writeback"}]: dispatch 2024-02-13T00:53:01.897 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:02.141 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:53:02.142 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:53:02.145 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:02.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:02.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:02.276 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:02.399 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:02.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:02.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:02.533 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:02.731 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:02.732 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:01.612541+0000 mon.a (mon.0) 3552 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:53:02.732 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:01.621627+0000 mon.a (mon.0) 3553 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-132", "mode": "writeback"}]': finished 2024-02-13T00:53:02.733 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:01.621704+0000 mon.a (mon.0) 3554 : cluster [DBG] osdmap e743: 8 total, 8 up, 4 in 2024-02-13T00:53:02.733 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:01.622750+0000 mon.a (mon.0) 3555 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:53:02.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:01.612541+0000 mon.a (mon.0) 3552 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:53:02.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:01.621627+0000 mon.a (mon.0) 3553 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-132", "mode": "writeback"}]': finished 2024-02-13T00:53:02.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:01.621704+0000 mon.a (mon.0) 3554 : cluster [DBG] osdmap e743: 8 total, 8 up, 4 in 2024-02-13T00:53:02.733 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:01.622750+0000 mon.a (mon.0) 3555 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:53:02.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:02.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:01.612541+0000 mon.a (mon.0) 3552 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:53:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:01.621627+0000 mon.a (mon.0) 3553 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-132", "mode": "writeback"}]': finished 2024-02-13T00:53:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:01.621704+0000 mon.a (mon.0) 3554 : cluster [DBG] osdmap e743: 8 total, 8 up, 4 in 2024-02-13T00:53:02.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:01.622750+0000 mon.a (mon.0) 3555 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:53:02.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:02.862 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:02.900 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:02.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:02.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-13T00:53:02.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 -- 172.21.15.162:0/495640902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72fc0fcc40 msgr2=0x7f72fc0f6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:02.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 --2- 172.21.15.162:0/495640902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72fc0fcc40 0x7f72fc0f6470 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 rx=0x7f72f0004830 tx=0x7f72f000c040).stop 2024-02-13T00:53:02.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 -- 172.21.15.162:0/495640902 shutdown_connections 2024-02-13T00:53:02.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 --2- 172.21.15.162:0/495640902 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72fc0f6b50 0x7f72fc103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:02.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 --2- 172.21.15.162:0/495640902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72fc0fc2e0 0x7f72fc0fc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:02.975 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 --2- 172.21.15.162:0/495640902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72fc0fcc40 0x7f72fc0f6470 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 -- 172.21.15.162:0/495640902 >> 172.21.15.162:0/495640902 conn(0x7f72fc070060 msgr2=0x7f72fc0f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 -- 172.21.15.162:0/495640902 shutdown_connections 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 -- 172.21.15.162:0/495640902 wait complete. 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 librados: starting msgr at 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 librados: starting objecter 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 Processor -- start 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 -- start start 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 librados: setting wanted keys 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.970+0000 7f730261c700 1 librados: calling monclient init 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f730261c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72fc0f6b50 0x7f72fc10beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f730261c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72fc0fc2e0 0x7f72fc10c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f730261c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72fc0fcc40 0x7f72fc106050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:02.976 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f730261c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc106540 con 0x7f72fc0fcc40 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f730261c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc1066b0 con 0x7f72fc0f6b50 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f730261c700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc106870 con 0x7f72fc0fc2e0 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72fb7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72fc0fc2e0 0x7f72fc10c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f7300bb9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72fc0fcc40 0x7f72fc106050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72fb7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72fc0fc2e0 0x7f72fc10c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:51624/0 (socket says 172.21.15.162:51624) 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72fb7fe700 1 -- 172.21.15.162:0/3539695139 learned_addr learned my addr 172.21.15.162:0/3539695139 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72fbfff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72fc0f6b50 0x7f72fc10beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72fb7fe700 1 -- 172.21.15.162:0/3539695139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72fc0f6b50 msgr2=0x7f72fc10beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72fb7fe700 1 --2- 172.21.15.162:0/3539695139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72fc0f6b50 0x7f72fc10beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72fb7fe700 1 -- 172.21.15.162:0/3539695139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72fc0fcc40 msgr2=0x7f72fc106050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72fb7fe700 1 --2- 172.21.15.162:0/3539695139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72fc0fcc40 0x7f72fc106050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72fb7fe700 1 -- 172.21.15.162:0/3539695139 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f40097c0 con 0x7f72fc0fc2e0 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f7300bb9700 1 --2- 172.21.15.162:0/3539695139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72fc0fcc40 0x7f72fc106050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72fb7fe700 1 --2- 172.21.15.162:0/3539695139 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72fc0fc2e0 0x7f72fc10c3a0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7f72f000af30 tx=0x7f72f000c040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:02.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72f97fa700 10 client.?.objecter ms_handle_connect 0x7f72fc0fc2e0 2024-02-13T00:53:02.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72f97fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:02.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f001a070 con 0x7f72fc0fc2e0 2024-02-13T00:53:02.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f0005040 con 0x7f72fc0fc2e0 2024-02-13T00:53:02.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72f00138e0 con 0x7f72fc0fc2e0 2024-02-13T00:53:02.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.971+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f001f6b0 con 0x7f72fc0fc2e0 2024-02-13T00:53:02.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.972+0000 7f730261c700 10 client.38141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:02.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.972+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc106d90 con 0x7f72fc0fc2e0 2024-02-13T00:53:02.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.972+0000 7f730261c700 1 librados: init done 2024-02-13T00:53:02.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.972+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f72fc05f960 con 0x7f72fc0fc2e0 2024-02-13T00:53:02.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f72f001fae0 con 0x7f72fc0fc2e0 2024-02-13T00:53:02.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f72f97fa700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6800/2866534053 conn(0x7f72e4062c90 0x7f72e4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:02.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(744..744 src has 449..744) v4 ==== 8627+0+0 (secure 0 0 0) 0x7f72f007e880 con 0x7f72fc0fc2e0 2024-02-13T00:53:02.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 osd_map(744..744 src has 449..744) v4 2024-02-13T00:53:02.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f72f97fa700 3 client.38141.objecter handle_osd_map got epochs [744,744] > 0 2024-02-13T00:53:02.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f72f97fa700 3 client.38141.objecter handle_osd_map decoding full epoch 744 2024-02-13T00:53:02.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f72fbfff700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6800/2866534053 conn(0x7f72e4062c90 0x7f72e4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:02.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f72f97fa700 20 client.38141.objecter dump_active .. 0 homeless 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v744) v1 ==== 54+0+16 (secure 0 0 0) 0x7f72f007ebe0 con 0x7f72fc0fc2e0 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f730261c700 10 client.38141.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 1 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f72f97fa700 10 client.38141.objecter ms_handle_connect 0x7f72e4062c90 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f72fbfff700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6800/2866534053 conn(0x7f72e4062c90 0x7f72e4065150 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f72ec004c30 tx=0x7f72ec00d040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6801/2103743338 conn(0x7f72fc0fd550 0x7f72fc0fd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.973+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc0475e0 osd=0 3 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc0475e0 osd=0 3 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc0475e0 osd=0 3 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 1 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc0475e0 osd=0 3 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 0 1 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc0475e0 osd=0 4 2024-02-13T00:53:02.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f730261c700 10 client.38141.objecter _send_command 1 2024-02-13T00:53:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72fc104f40 con 0x7f72fc0fd550 2024-02-13T00:53:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f7300bb9700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6801/2103743338 conn(0x7f72fc0fd550 0x7f72fc0fd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f72f97fa700 10 client.38141.objecter ms_handle_connect 0x7f72fc0fd550 2024-02-13T00:53:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f7300bb9700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6801/2103743338 conn(0x7f72fc0fd550 0x7f72fc0fd930 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f72fc104f40 con 0x7f72fc0fd550 2024-02-13T00:53:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:53:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f72f97fa700 10 client.38141.objecter _finish_command 1 = system:0 2024-02-13T00:53:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc0475e0 osd=0 4 2024-02-13T00:53:02.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.974+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 0 1 2024-02-13T00:53:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.989+0000 7f730261c700 10 client.38141.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.989+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 2 2024-02-13T00:53:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.989+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc0475e0 osd=0 3 2024-02-13T00:53:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.989+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc0475e0 osd=0 3 2024-02-13T00:53:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.989+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc0475e0 osd=0 3 2024-02-13T00:53:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.989+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 2 2024-02-13T00:53:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.989+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc0475e0 osd=0 3 2024-02-13T00:53:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.989+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 0 2 2024-02-13T00:53:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.989+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc0475e0 osd=0 4 2024-02-13T00:53:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.989+0000 7f730261c700 10 client.38141.objecter _send_command 2 2024-02-13T00:53:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.989+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f72fc027410 con 0x7f72fc0fd550 2024-02-13T00:53:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f72fc027410 con 0x7f72fc0fd550 2024-02-13T00:53:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f72f97fa700 10 client.38141.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc0475e0 osd=0 4 2024-02-13T00:53:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 0 2 2024-02-13T00:53:02.995 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:53:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f730261c700 10 client.38141.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 3 2024-02-13T00:53:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6805/150599780 conn(0x7f72fc059990 0x7f72fc059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc028760 osd=1 3 2024-02-13T00:53:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc028760 osd=1 3 2024-02-13T00:53:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc028760 osd=1 3 2024-02-13T00:53:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 3 2024-02-13T00:53:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc028760 osd=1 3 2024-02-13T00:53:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 1 3 2024-02-13T00:53:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc028760 osd=1 4 2024-02-13T00:53:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f730261c700 10 client.38141.objecter _send_command 3 2024-02-13T00:53:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72fc119460 con 0x7f72fc059990 2024-02-13T00:53:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f72fbfff700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6805/150599780 conn(0x7f72fc059990 0x7f72fc059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f72f97fa700 10 client.38141.objecter ms_handle_connect 0x7f72fc059990 2024-02-13T00:53:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.991+0000 7f72fbfff700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6805/150599780 conn(0x7f72fc059990 0x7f72fc059d70 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.992+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f72fc119460 con 0x7f72fc059990 2024-02-13T00:53:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.992+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:53:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.992+0000 7f72f97fa700 10 client.38141.objecter _finish_command 3 = system:0 2024-02-13T00:53:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.992+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc028760 osd=1 4 2024-02-13T00:53:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:02.992+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 1 3 2024-02-13T00:53:03.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.006+0000 7f730261c700 10 client.38141.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.006+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 4 2024-02-13T00:53:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.006+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc028760 osd=1 3 2024-02-13T00:53:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.006+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc028760 osd=1 3 2024-02-13T00:53:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.006+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc028760 osd=1 3 2024-02-13T00:53:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.006+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 4 2024-02-13T00:53:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.006+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc028760 osd=1 3 2024-02-13T00:53:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.006+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 1 4 2024-02-13T00:53:03.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.006+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc028760 osd=1 4 2024-02-13T00:53:03.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.006+0000 7f730261c700 10 client.38141.objecter _send_command 4 2024-02-13T00:53:03.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.006+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f72fc059400 con 0x7f72fc059990 2024-02-13T00:53:03.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f72fc059400 con 0x7f72fc059990 2024-02-13T00:53:03.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:03.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f72f97fa700 10 client.38141.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc028760 osd=1 4 2024-02-13T00:53:03.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 1 4 2024-02-13T00:53:03.012 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:53:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f730261c700 10 client.38141.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 5 2024-02-13T00:53:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6809/3689719376 conn(0x7f72fc0282c0 0x7f72fc117e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc059810 osd=2 3 2024-02-13T00:53:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc059810 osd=2 3 2024-02-13T00:53:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc059810 osd=2 3 2024-02-13T00:53:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 5 2024-02-13T00:53:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc059810 osd=2 3 2024-02-13T00:53:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 2 5 2024-02-13T00:53:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc059810 osd=2 4 2024-02-13T00:53:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f730261c700 10 client.38141.objecter _send_command 5 2024-02-13T00:53:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72fc118370 con 0x7f72fc0282c0 2024-02-13T00:53:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f72fb7fe700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6809/3689719376 conn(0x7f72fc0282c0 0x7f72fc117e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f72f97fa700 10 client.38141.objecter ms_handle_connect 0x7f72fc0282c0 2024-02-13T00:53:03.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.008+0000 7f72fb7fe700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6809/3689719376 conn(0x7f72fc0282c0 0x7f72fc117e80 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:03.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.009+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f72fc118370 con 0x7f72fc0282c0 2024-02-13T00:53:03.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.009+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:53:03.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.009+0000 7f72f97fa700 10 client.38141.objecter _finish_command 5 = system:0 2024-02-13T00:53:03.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.009+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc059810 osd=2 4 2024-02-13T00:53:03.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.009+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 2 5 2024-02-13T00:53:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.023+0000 7f730261c700 10 client.38141.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.023+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 6 2024-02-13T00:53:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.023+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc059810 osd=2 3 2024-02-13T00:53:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.023+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc059810 osd=2 3 2024-02-13T00:53:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.023+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc059810 osd=2 3 2024-02-13T00:53:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.023+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 6 2024-02-13T00:53:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.023+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc059810 osd=2 3 2024-02-13T00:53:03.027 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.023+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 2 6 2024-02-13T00:53:03.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.023+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc059810 osd=2 4 2024-02-13T00:53:03.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.023+0000 7f730261c700 10 client.38141.objecter _send_command 6 2024-02-13T00:53:03.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.023+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f72fc027820 con 0x7f72fc0282c0 2024-02-13T00:53:03.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f72fc027820 con 0x7f72fc0282c0 2024-02-13T00:53:03.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:03.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f72f97fa700 10 client.38141.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.028 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc059810 osd=2 4 2024-02-13T00:53:03.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 2 6 2024-02-13T00:53:03.029 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:53:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f730261c700 10 client.38141.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 7 2024-02-13T00:53:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6813/604971011 conn(0x7f72fc11b5e0 0x7f72fc11d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc118d10 osd=3 3 2024-02-13T00:53:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc118d10 osd=3 3 2024-02-13T00:53:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc118d10 osd=3 3 2024-02-13T00:53:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 7 2024-02-13T00:53:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc118d10 osd=3 3 2024-02-13T00:53:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 3 7 2024-02-13T00:53:03.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc118d10 osd=3 4 2024-02-13T00:53:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f730261c700 10 client.38141.objecter _send_command 7 2024-02-13T00:53:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72fc120620 con 0x7f72fc11b5e0 2024-02-13T00:53:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f7300bb9700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6813/604971011 conn(0x7f72fc11b5e0 0x7f72fc11d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f72f97fa700 10 client.38141.objecter ms_handle_connect 0x7f72fc11b5e0 2024-02-13T00:53:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.025+0000 7f7300bb9700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6813/604971011 conn(0x7f72fc11b5e0 0x7f72fc11d9d0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.026+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f72fc120620 con 0x7f72fc11b5e0 2024-02-13T00:53:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.026+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:53:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.026+0000 7f72f97fa700 10 client.38141.objecter _finish_command 7 = system:0 2024-02-13T00:53:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.026+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc118d10 osd=3 4 2024-02-13T00:53:03.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.026+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 3 7 2024-02-13T00:53:03.031 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.040+0000 7f730261c700 10 client.38141.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:03.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.040+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 8 2024-02-13T00:53:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.040+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc118d10 osd=3 3 2024-02-13T00:53:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.040+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc118d10 osd=3 3 2024-02-13T00:53:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.040+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc118d10 osd=3 3 2024-02-13T00:53:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.040+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 8 2024-02-13T00:53:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.040+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc118d10 osd=3 3 2024-02-13T00:53:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.040+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 3 8 2024-02-13T00:53:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.040+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc118d10 osd=3 4 2024-02-13T00:53:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.040+0000 7f730261c700 10 client.38141.objecter _send_command 8 2024-02-13T00:53:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.040+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f72fc118780 con 0x7f72fc11b5e0 2024-02-13T00:53:03.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f72fc118780 con 0x7f72fc11b5e0 2024-02-13T00:53:03.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:03.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f72f97fa700 10 client.38141.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc118d10 osd=3 4 2024-02-13T00:53:03.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 3 8 2024-02-13T00:53:03.045 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:53:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f730261c700 10 client.38141.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 9 2024-02-13T00:53:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6800/3344263388 conn(0x7f72fc1179e0 0x7f72fc11f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc118540 osd=4 3 2024-02-13T00:53:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc118540 osd=4 3 2024-02-13T00:53:03.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc118540 osd=4 3 2024-02-13T00:53:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 9 2024-02-13T00:53:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc118540 osd=4 3 2024-02-13T00:53:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 4 9 2024-02-13T00:53:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc118540 osd=4 4 2024-02-13T00:53:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f730261c700 10 client.38141.objecter _send_command 9 2024-02-13T00:53:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.041+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72fc1186b0 con 0x7f72fc1179e0 2024-02-13T00:53:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.042+0000 7f72fbfff700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6800/3344263388 conn(0x7f72fc1179e0 0x7f72fc11f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.042+0000 7f72f97fa700 10 client.38141.objecter ms_handle_connect 0x7f72fc1179e0 2024-02-13T00:53:03.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.042+0000 7f72fbfff700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6800/3344263388 conn(0x7f72fc1179e0 0x7f72fc11f040 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:03.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.043+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f72fc1186b0 con 0x7f72fc1179e0 2024-02-13T00:53:03.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.043+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:53:03.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.043+0000 7f72f97fa700 10 client.38141.objecter _finish_command 9 = system:0 2024-02-13T00:53:03.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.043+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc118540 osd=4 4 2024-02-13T00:53:03.048 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.043+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 4 9 2024-02-13T00:53:03.050 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:03.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:03.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.058+0000 7f730261c700 10 client.38141.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.058+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 10 2024-02-13T00:53:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.058+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc118540 osd=4 3 2024-02-13T00:53:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.058+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc118540 osd=4 3 2024-02-13T00:53:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.058+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc118540 osd=4 3 2024-02-13T00:53:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.058+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 10 2024-02-13T00:53:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.058+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc118540 osd=4 3 2024-02-13T00:53:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.058+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 4 10 2024-02-13T00:53:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.058+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc118540 osd=4 4 2024-02-13T00:53:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.058+0000 7f730261c700 10 client.38141.objecter _send_command 10 2024-02-13T00:53:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.058+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f72fc1186b0 con 0x7f72fc1179e0 2024-02-13T00:53:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f72fc1186b0 con 0x7f72fc1179e0 2024-02-13T00:53:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f72f97fa700 10 client.38141.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc118540 osd=4 4 2024-02-13T00:53:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 4 10 2024-02-13T00:53:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:53:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f730261c700 10 client.38141.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 11 2024-02-13T00:53:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6804/2277917372 conn(0x7f72fc11fb10 0x7f72fc11ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc1186b0 osd=5 3 2024-02-13T00:53:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc1186b0 osd=5 3 2024-02-13T00:53:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc1186b0 osd=5 3 2024-02-13T00:53:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 11 2024-02-13T00:53:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc1186b0 osd=5 3 2024-02-13T00:53:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 5 11 2024-02-13T00:53:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc1186b0 osd=5 4 2024-02-13T00:53:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f730261c700 10 client.38141.objecter _send_command 11 2024-02-13T00:53:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72fc12b3a0 con 0x7f72fc11fb10 2024-02-13T00:53:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f72fb7fe700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6804/2277917372 conn(0x7f72fc11fb10 0x7f72fc11ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f72f97fa700 10 client.38141.objecter ms_handle_connect 0x7f72fc11fb10 2024-02-13T00:53:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.065+0000 7f72fb7fe700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6804/2277917372 conn(0x7f72fc11fb10 0x7f72fc11ff50 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.066+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f72fc12b3a0 con 0x7f72fc11fb10 2024-02-13T00:53:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.066+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:53:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.066+0000 7f72f97fa700 10 client.38141.objecter _finish_command 11 = system:0 2024-02-13T00:53:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.066+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc1186b0 osd=5 4 2024-02-13T00:53:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.066+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 5 11 2024-02-13T00:53:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.081+0000 7f730261c700 10 client.38141.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.081+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 12 2024-02-13T00:53:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.081+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc1186b0 osd=5 3 2024-02-13T00:53:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.081+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc1186b0 osd=5 3 2024-02-13T00:53:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.081+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc1186b0 osd=5 3 2024-02-13T00:53:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.081+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 12 2024-02-13T00:53:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.081+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc1186b0 osd=5 3 2024-02-13T00:53:03.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.081+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 5 12 2024-02-13T00:53:03.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.081+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc1186b0 osd=5 4 2024-02-13T00:53:03.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.081+0000 7f730261c700 10 client.38141.objecter _send_command 12 2024-02-13T00:53:03.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.081+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f72fc128c40 con 0x7f72fc11fb10 2024-02-13T00:53:03.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.082+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f72fc128c40 con 0x7f72fc11fb10 2024-02-13T00:53:03.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.082+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:03.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.082+0000 7f72f97fa700 10 client.38141.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.082+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc1186b0 osd=5 4 2024-02-13T00:53:03.086 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.082+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 5 12 2024-02-13T00:53:03.087 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:53:03.087 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f730261c700 10 client.38141.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 13 2024-02-13T00:53:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6808/1506640341 conn(0x7f72fc129320 0x7f72fc129760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc128c40 osd=6 3 2024-02-13T00:53:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc128c40 osd=6 3 2024-02-13T00:53:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc128c40 osd=6 3 2024-02-13T00:53:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 13 2024-02-13T00:53:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc128c40 osd=6 3 2024-02-13T00:53:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 6 13 2024-02-13T00:53:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc128c40 osd=6 4 2024-02-13T00:53:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f730261c700 10 client.38141.objecter _send_command 13 2024-02-13T00:53:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72fc12e6f0 con 0x7f72fc129320 2024-02-13T00:53:03.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f7300bb9700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6808/1506640341 conn(0x7f72fc129320 0x7f72fc129760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:03.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f72f97fa700 10 client.38141.objecter ms_handle_connect 0x7f72fc129320 2024-02-13T00:53:03.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.083+0000 7f7300bb9700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6808/1506640341 conn(0x7f72fc129320 0x7f72fc129760 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:03.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.084+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f72fc12e6f0 con 0x7f72fc129320 2024-02-13T00:53:03.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.084+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:53:03.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.084+0000 7f72f97fa700 10 client.38141.objecter _finish_command 13 = system:0 2024-02-13T00:53:03.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.084+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc128c40 osd=6 4 2024-02-13T00:53:03.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.084+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 6 13 2024-02-13T00:53:03.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.098+0000 7f730261c700 10 client.38141.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.098+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 14 2024-02-13T00:53:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.098+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc128c40 osd=6 3 2024-02-13T00:53:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.098+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc128c40 osd=6 3 2024-02-13T00:53:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.098+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc128c40 osd=6 3 2024-02-13T00:53:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.098+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 14 2024-02-13T00:53:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.098+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc128c40 osd=6 3 2024-02-13T00:53:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.098+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 6 14 2024-02-13T00:53:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.098+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc128c40 osd=6 4 2024-02-13T00:53:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.098+0000 7f730261c700 10 client.38141.objecter _send_command 14 2024-02-13T00:53:03.102 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.098+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f72fc129d40 con 0x7f72fc129320 2024-02-13T00:53:03.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f72fc129d40 con 0x7f72fc129320 2024-02-13T00:53:03.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:03.103 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f72f97fa700 10 client.38141.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc128c40 osd=6 4 2024-02-13T00:53:03.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 6 14 2024-02-13T00:53:03.104 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:53:03.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f730261c700 10 client.38141.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:03.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 15 2024-02-13T00:53:03.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6812/1964192772 conn(0x7f72fc12a3f0 0x7f72fc12a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:03.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc129d40 osd=7 3 2024-02-13T00:53:03.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc129d40 osd=7 3 2024-02-13T00:53:03.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc129d40 osd=7 3 2024-02-13T00:53:03.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 15 2024-02-13T00:53:03.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc129d40 osd=7 3 2024-02-13T00:53:03.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 7 15 2024-02-13T00:53:03.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc129d40 osd=7 4 2024-02-13T00:53:03.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f730261c700 10 client.38141.objecter _send_command 15 2024-02-13T00:53:03.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72fc12ae70 con 0x7f72fc12a3f0 2024-02-13T00:53:03.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.100+0000 7f72fbfff700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6812/1964192772 conn(0x7f72fc12a3f0 0x7f72fc12a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:03.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.101+0000 7f72f97fa700 10 client.38141.objecter ms_handle_connect 0x7f72fc12a3f0 2024-02-13T00:53:03.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.101+0000 7f72fbfff700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6812/1964192772 conn(0x7f72fc12a3f0 0x7f72fc12a850 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:03.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.101+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f72fc12ae70 con 0x7f72fc12a3f0 2024-02-13T00:53:03.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.101+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:53:03.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.101+0000 7f72f97fa700 10 client.38141.objecter _finish_command 15 = system:0 2024-02-13T00:53:03.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.101+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc129d40 osd=7 4 2024-02-13T00:53:03.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.101+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 7 15 2024-02-13T00:53:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.115+0000 7f730261c700 10 client.38141.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.115+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign -1 16 2024-02-13T00:53:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.115+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc129d40 osd=7 3 2024-02-13T00:53:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.115+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc129d40 osd=7 3 2024-02-13T00:53:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.115+0000 7f730261c700 20 client.38141.objecter _get_session s=0x7f72fc129d40 osd=7 3 2024-02-13T00:53:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.115+0000 7f730261c700 15 client.38141.objecter _session_command_op_remove -1 16 2024-02-13T00:53:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.115+0000 7f730261c700 20 client.38141.objecter get_session s=0x7f72fc129d40 osd=7 3 2024-02-13T00:53:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.115+0000 7f730261c700 15 client.38141.objecter _session_command_op_assign 7 16 2024-02-13T00:53:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.115+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc129d40 osd=7 4 2024-02-13T00:53:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.115+0000 7f730261c700 10 client.38141.objecter _send_command 16 2024-02-13T00:53:03.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.115+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f72fc12ae70 con 0x7f72fc12a3f0 2024-02-13T00:53:03.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f72f97fa700 1 -- 172.21.15.162:0/3539695139 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f72fc12ae70 con 0x7f72fc12a3f0 2024-02-13T00:53:03.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f72f97fa700 10 client.38141.objecter ms_dispatch 0x7f72fc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:03.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f72f97fa700 10 client.38141.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f72f97fa700 20 client.38141.objecter put_session s=0x7f72fc129d40 osd=7 4 2024-02-13T00:53:03.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f72f97fa700 15 client.38141.objecter _session_command_op_remove 7 16 2024-02-13T00:53:03.121 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:53:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 10 librados: watch_flush enter 2024-02-13T00:53:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 10 librados: watch_flush exit 2024-02-13T00:53:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 10 client.38141.objecter close_session for osd.0 2024-02-13T00:53:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6801/2103743338 conn(0x7f72fc0fd550 msgr2=0x7f72fc0fd930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6801/2103743338 conn(0x7f72fc0fd550 0x7f72fc0fd930 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc0475e0 osd=0 1 2024-02-13T00:53:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 10 client.38141.objecter close_session for osd.1 2024-02-13T00:53:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6805/150599780 conn(0x7f72fc059990 msgr2=0x7f72fc059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6805/150599780 conn(0x7f72fc059990 0x7f72fc059d70 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc028760 osd=1 1 2024-02-13T00:53:03.123 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 10 client.38141.objecter close_session for osd.2 2024-02-13T00:53:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6809/3689719376 conn(0x7f72fc0282c0 msgr2=0x7f72fc117e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6809/3689719376 conn(0x7f72fc0282c0 0x7f72fc117e80 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc059810 osd=2 1 2024-02-13T00:53:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 10 client.38141.objecter close_session for osd.3 2024-02-13T00:53:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6813/604971011 conn(0x7f72fc11b5e0 msgr2=0x7f72fc11d9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f7300bb9700 1 -- 172.21.15.162:0/3539695139 reap_dead start 2024-02-13T00:53:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6813/604971011 conn(0x7f72fc11b5e0 0x7f72fc11d9d0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc118d10 osd=3 1 2024-02-13T00:53:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 10 client.38141.objecter close_session for osd.4 2024-02-13T00:53:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6800/3344263388 conn(0x7f72fc1179e0 msgr2=0x7f72fc11f040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.124 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6800/3344263388 conn(0x7f72fc1179e0 0x7f72fc11f040 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc118540 osd=4 1 2024-02-13T00:53:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 10 client.38141.objecter close_session for osd.5 2024-02-13T00:53:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6804/2277917372 conn(0x7f72fc11fb10 msgr2=0x7f72fc11ff50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6804/2277917372 conn(0x7f72fc11fb10 0x7f72fc11ff50 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc1186b0 osd=5 1 2024-02-13T00:53:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 10 client.38141.objecter close_session for osd.6 2024-02-13T00:53:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6808/1506640341 conn(0x7f72fc129320 msgr2=0x7f72fc129760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6808/1506640341 conn(0x7f72fc129320 0x7f72fc129760 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc128c40 osd=6 1 2024-02-13T00:53:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 10 client.38141.objecter close_session for osd.7 2024-02-13T00:53:03.125 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6812/1964192772 conn(0x7f72fc12a3f0 msgr2=0x7f72fc12a850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.181:6812/1964192772 conn(0x7f72fc12a3f0 0x7f72fc12a850 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 20 client.38141.objecter put_session s=0x7f72fc129d40 osd=7 1 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 20 client.38141.objecter shutdown clearing up homeless session... 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.117+0000 7f730261c700 10 client.38141.objecter successfully canceled tick 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.118+0000 7f7300bb9700 1 -- 172.21.15.162:0/3539695139 reap_dead start 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.118+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6800/2866534053 conn(0x7f72e4062c90 msgr2=0x7f72e4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.118+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6800/2866534053 conn(0x7f72e4062c90 0x7f72e4065150 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f72ec004c30 tx=0x7f72ec00d040).stop 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.118+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72fc0fc2e0 msgr2=0x7f72fc10c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.118+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72fc0fc2e0 0x7f72fc10c3a0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7f72f000af30 tx=0x7f72f000c040).stop 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.118+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 shutdown_connections 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.118+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72fc0fc2e0 0x7f72fc10c3a0 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.118+0000 7f730261c700 1 --2- 172.21.15.162:0/3539695139 >> v2:172.21.15.162:6800/2866534053 conn(0x7f72e4062c90 0x7f72e4065150 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.126 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.118+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 >> 172.21.15.162:0/3539695139 conn(0x7f72fc070060 msgr2=0x7f72fc0f9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.118+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 shutdown_connections 2024-02-13T00:53:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.118+0000 7f730261c700 1 -- 172.21.15.162:0/3539695139 wait complete. 2024-02-13T00:53:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.118+0000 7f730261c700 1 librados: shutdown 2024-02-13T00:53:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:53:03.153 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:03.181 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:03.201 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:03.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:03.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:03.233 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:53:03.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:03.331 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:03.453 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:03.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:03.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:03.589 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:03.712 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:03.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:02.531986+0000 mgr.y (mgr.14544) 618 : cluster [DBG] pgmap v1127: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-13T00:53:03.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:02.632257+0000 mon.a (mon.0) 3556 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:53:03.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:02.632344+0000 mon.a (mon.0) 3557 : cluster [DBG] osdmap e744: 8 total, 8 up, 4 in 2024-02-13T00:53:03.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:02.634132+0000 mon.a (mon.0) 3558 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:53:03.714 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:02.974107+0000 mon.c (mon.2) 502 : audit [DBG] from='client.? 172.21.15.162:0/3539695139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:03.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:03.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.784+0000 7f53595f2700 1 -- 172.21.15.162:0/3167962349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53540fd4b0 msgr2=0x7f53540f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.784+0000 7f53595f2700 1 --2- 172.21.15.162:0/3167962349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53540fd4b0 0x7f53540f7370 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 rx=0x7f53480047b0 tx=0x7f5348009cb0).stop 2024-02-13T00:53:03.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.784+0000 7f53595f2700 1 -- 172.21.15.162:0/3167962349 shutdown_connections 2024-02-13T00:53:03.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.784+0000 7f53595f2700 1 --2- 172.21.15.162:0/3167962349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53540f7a50 0x7f53541039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.784+0000 7f53595f2700 1 --2- 172.21.15.162:0/3167962349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53540fcb50 0x7f53540fcf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.784+0000 7f53595f2700 1 --2- 172.21.15.162:0/3167962349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53540fd4b0 0x7f53540f7370 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.784+0000 7f53595f2700 1 -- 172.21.15.162:0/3167962349 >> 172.21.15.162:0/3167962349 conn(0x7f53540700d0 msgr2=0x7f53540f4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:03.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.784+0000 7f53595f2700 1 -- 172.21.15.162:0/3167962349 shutdown_connections 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.784+0000 7f53595f2700 1 -- 172.21.15.162:0/3167962349 wait complete. 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53595f2700 1 librados: starting msgr at 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53595f2700 1 librados: starting objecter 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53595f2700 1 Processor -- start 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53595f2700 1 -- start start 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53595f2700 1 librados: setting wanted keys 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53595f2700 1 librados: calling monclient init 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53595f2700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53540f7a50 0x7f535410d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53595f2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53540fcb50 0x7f535410d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53595f2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53540fd4b0 0x7f5354107260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53595f2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5354107750 con 0x7f53540fd4b0 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53595f2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53541078c0 con 0x7f53540fcb50 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53595f2700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5354107a80 con 0x7f53540f7a50 2024-02-13T00:53:03.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53537fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53540fd4b0 0x7f5354107260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f5352ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53540f7a50 0x7f535410d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53537fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53540fd4b0 0x7f5354107260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43182/0 (socket says 172.21.15.162:43182) 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.785+0000 7f53537fe700 1 -- 172.21.15.162:0/1947056122 learned_addr learned my addr 172.21.15.162:0/1947056122 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f53527fc700 1 --2- 172.21.15.162:0/1947056122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53540fcb50 0x7f535410d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f53537fe700 1 -- 172.21.15.162:0/1947056122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53540f7a50 msgr2=0x7f535410d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f53537fe700 1 --2- 172.21.15.162:0/1947056122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53540f7a50 0x7f535410d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f53537fe700 1 -- 172.21.15.162:0/1947056122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53540fcb50 msgr2=0x7f535410d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f53537fe700 1 --2- 172.21.15.162:0/1947056122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53540fcb50 0x7f535410d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f53537fe700 1 -- 172.21.15.162:0/1947056122 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c0097c0 con 0x7f53540fd4b0 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f5352ffd700 1 --2- 172.21.15.162:0/1947056122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53540f7a50 0x7f535410d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f53537fe700 1 --2- 172.21.15.162:0/1947056122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53540fd4b0 0x7f5354107260 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7f534c001df0 tx=0x7f534c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f533bfff700 10 client.?.objecter ms_handle_connect 0x7f53540fd4b0 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f533bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f533bfff700 1 -- 172.21.15.162:0/1947056122 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c009a10 con 0x7f53540fd4b0 2024-02-13T00:53:03.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f53595f2700 1 -- 172.21.15.162:0/1947056122 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5348005040 con 0x7f53540fd4b0 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f53595f2700 10 client.28050.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f53595f2700 1 -- 172.21.15.162:0/1947056122 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53541080c0 con 0x7f53540fd4b0 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f53595f2700 1 librados: init done 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f533bfff700 1 -- 172.21.15.162:0/1947056122 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f534c0095a0 con 0x7f53540fd4b0 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.786+0000 7f533bfff700 1 -- 172.21.15.162:0/1947056122 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c007ce0 con 0x7f53540fd4b0 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.787+0000 7f53595f2700 1 -- 172.21.15.162:0/1947056122 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5340004f50 con 0x7f53540fd4b0 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.787+0000 7f533bfff700 1 -- 172.21.15.162:0/1947056122 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f534c00fd40 con 0x7f53540fd4b0 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.788+0000 7f533bfff700 1 --2- 172.21.15.162:0/1947056122 >> v2:172.21.15.162:6800/2866534053 conn(0x7f533c062c90 0x7f533c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.788+0000 7f533bfff700 1 -- 172.21.15.162:0/1947056122 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(745..745 src has 449..745) v4 ==== 8627+0+0 (secure 0 0 0) 0x7f534c07db50 con 0x7f53540fd4b0 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.788+0000 7f533bfff700 10 client.28050.objecter ms_dispatch 0x7f5354001550 osd_map(745..745 src has 449..745) v4 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.788+0000 7f533bfff700 3 client.28050.objecter handle_osd_map got epochs [745,745] > 0 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.788+0000 7f533bfff700 3 client.28050.objecter handle_osd_map decoding full epoch 745 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.788+0000 7f5352ffd700 1 --2- 172.21.15.162:0/1947056122 >> v2:172.21.15.162:6800/2866534053 conn(0x7f533c062c90 0x7f533c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:03.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.788+0000 7f533bfff700 20 client.28050.objecter dump_active .. 0 homeless 2024-02-13T00:53:03.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.788+0000 7f533bfff700 10 client.28050.objecter ms_handle_connect 0x7f533c062c90 2024-02-13T00:53:03.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.788+0000 7f5352ffd700 1 --2- 172.21.15.162:0/1947056122 >> v2:172.21.15.162:6800/2866534053 conn(0x7f533c062c90 0x7f533c065150 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f53440103c0 tx=0x7f534400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:03.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.792+0000 7f533bfff700 1 -- 172.21.15.162:0/1947056122 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f534c050de0 con 0x7f53540fd4b0 2024-02-13T00:53:03.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:03.845 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:03.865 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:03.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:03.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:03.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:03.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:02.531986+0000 mgr.y (mgr.14544) 618 : cluster [DBG] pgmap v1127: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-13T00:53:03.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:02.632257+0000 mon.a (mon.0) 3556 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:53:03.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:02.632344+0000 mon.a (mon.0) 3557 : cluster [DBG] osdmap e744: 8 total, 8 up, 4 in 2024-02-13T00:53:03.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:02.634132+0000 mon.a (mon.0) 3558 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:53:03.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:02.974107+0000 mon.c (mon.2) 502 : audit [DBG] from='client.? 172.21.15.162:0/3539695139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:03.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:02.531986+0000 mgr.y (mgr.14544) 618 : cluster [DBG] pgmap v1127: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-13T00:53:03.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:02.632257+0000 mon.a (mon.0) 3556 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:53:03.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:02.632344+0000 mon.a (mon.0) 3557 : cluster [DBG] osdmap e744: 8 total, 8 up, 4 in 2024-02-13T00:53:03.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:02.634132+0000 mon.a (mon.0) 3558 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:53:03.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:02.974107+0000 mon.c (mon.2) 502 : audit [DBG] from='client.? 172.21.15.162:0/3539695139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:03.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:03.982+0000 7f53595f2700 1 -- 172.21.15.162:0/1947056122 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f5340005200 con 0x7f53540fd4b0 2024-02-13T00:53:03.996 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:04.023 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:04.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:04.127 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:04.153 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:04.253 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:04.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:04.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:04.389 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:04.589 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:04.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:04.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.631+0000 7f533bfff700 1 -- 172.21.15.162:0/1947056122 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v746) v1 ==== 91+0+0 (secure 0 0 0) 0x7f534c0509d0 con 0x7f53540fd4b0 2024-02-13T00:53:04.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 10 librados: watch_flush enter 2024-02-13T00:53:04.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 10 librados: watch_flush exit 2024-02-13T00:53:04.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 20 client.28050.objecter shutdown clearing up homeless session... 2024-02-13T00:53:04.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 10 client.28050.objecter successfully canceled tick 2024-02-13T00:53:04.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 -- 172.21.15.162:0/1947056122 >> v2:172.21.15.162:6800/2866534053 conn(0x7f533c062c90 msgr2=0x7f533c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:04.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 --2- 172.21.15.162:0/1947056122 >> v2:172.21.15.162:6800/2866534053 conn(0x7f533c062c90 0x7f533c065150 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f53440103c0 tx=0x7f534400b040).stop 2024-02-13T00:53:04.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 -- 172.21.15.162:0/1947056122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53540fd4b0 msgr2=0x7f5354107260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:04.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 --2- 172.21.15.162:0/1947056122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53540fd4b0 0x7f5354107260 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7f534c001df0 tx=0x7f534c004d10).stop 2024-02-13T00:53:04.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 -- 172.21.15.162:0/1947056122 shutdown_connections 2024-02-13T00:53:04.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 --2- 172.21.15.162:0/1947056122 >> v2:172.21.15.162:6800/2866534053 conn(0x7f533c062c90 0x7f533c065150 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:04.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 --2- 172.21.15.162:0/1947056122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53540fd4b0 0x7f5354107260 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:04.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 --2- 172.21.15.162:0/1947056122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53540f7a50 0x7f535410d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:04.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 --2- 172.21.15.162:0/1947056122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53540fcb50 0x7f535410d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:04.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 -- 172.21.15.162:0/1947056122 >> 172.21.15.162:0/1947056122 conn(0x7f53540700d0 msgr2=0x7f53540fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:04.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 -- 172.21.15.162:0/1947056122 shutdown_connections 2024-02-13T00:53:04.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 -- 172.21.15.162:0/1947056122 wait complete. 2024-02-13T00:53:04.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:04.634+0000 7f53595f2700 1 librados: shutdown 2024-02-13T00:53:04.639 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:53:04.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:04.718 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:03.630917+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:53:04.718 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:03.630990+0000 mon.a (mon.0) 3560 : cluster [DBG] osdmap e745: 8 total, 8 up, 4 in 2024-02-13T00:53:04.718 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:03.632022+0000 mon.a (mon.0) 3561 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:53:04.718 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:03.983198+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? 172.21.15.162:0/1947056122' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:53:04.718 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:04.343033+0000 mon.a (mon.0) 3563 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:04.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:04.628022+0000 mon.a (mon.0) 3564 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:53:04.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:04.628045+0000 mon.a (mon.0) 3565 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:53:04.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:04.632102+0000 mon.a (mon.0) 3566 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:53:04.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:04.632178+0000 mon.a (mon.0) 3567 : audit [INF] from='client.? 172.21.15.162:0/1947056122' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:53:04.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:04.632215+0000 mon.a (mon.0) 3568 : cluster [DBG] osdmap e746: 8 total, 8 up, 4 in 2024-02-13T00:53:04.719 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:04.633230+0000 mon.a (mon.0) 3569 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:53:04.719 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:03.630917+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:53:04.719 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:03.630990+0000 mon.a (mon.0) 3560 : cluster [DBG] osdmap e745: 8 total, 8 up, 4 in 2024-02-13T00:53:04.719 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:03.632022+0000 mon.a (mon.0) 3561 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:53:04.719 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:03.983198+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? 172.21.15.162:0/1947056122' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:53:04.719 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:04.343033+0000 mon.a (mon.0) 3563 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:04.719 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:04.628022+0000 mon.a (mon.0) 3564 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:53:04.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:04.628045+0000 mon.a (mon.0) 3565 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:53:04.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:04.632102+0000 mon.a (mon.0) 3566 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:53:04.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:04.632178+0000 mon.a (mon.0) 3567 : audit [INF] from='client.? 172.21.15.162:0/1947056122' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:53:04.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:04.632215+0000 mon.a (mon.0) 3568 : cluster [DBG] osdmap e746: 8 total, 8 up, 4 in 2024-02-13T00:53:04.720 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:04.633230+0000 mon.a (mon.0) 3569 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:53:04.721 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:04.870 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:04.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:04.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:04.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:03.630917+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:53:04.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:03.630990+0000 mon.a (mon.0) 3560 : cluster [DBG] osdmap e745: 8 total, 8 up, 4 in 2024-02-13T00:53:04.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:03.632022+0000 mon.a (mon.0) 3561 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:53:04.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:03.983198+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? 172.21.15.162:0/1947056122' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:53:04.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:04.343033+0000 mon.a (mon.0) 3563 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:04.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:04.628022+0000 mon.a (mon.0) 3564 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:53:04.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:04.628045+0000 mon.a (mon.0) 3565 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:53:04.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:04.632102+0000 mon.a (mon.0) 3566 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:53:04.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:04.632178+0000 mon.a (mon.0) 3567 : audit [INF] from='client.? 172.21.15.162:0/1947056122' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:53:04.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:04.632215+0000 mon.a (mon.0) 3568 : cluster [DBG] osdmap e746: 8 total, 8 up, 4 in 2024-02-13T00:53:04.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:04.633230+0000 mon.a (mon.0) 3569 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:53:04.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:05.002 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:05.022 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:05.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:05.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:05.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:05.136 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:53:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:53:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:53:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:53:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.25162.0:3128 184.0 e300/296 184:00adb854:test-rados-api-smithi162-84057-38::foo:head v 300'2, mlcod=0'0)", 2024-02-13T00:53:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:15.383766+0000", 2024-02-13T00:53:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 529.74745132400005, 2024-02-13T00:53:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013839536, 2024-02-13T00:53:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.383766+0000", 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.383766+0000", 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.383765+0000", 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002561158 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386326+0000", 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7059e-05 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386344+0000", 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.259e-06 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386352+0000", 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014103499999999999 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386493+0000", 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2059999999999999e-05 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.386515+0000", 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011046402 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.397561+0000", 2024-02-13T00:53:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4487000000000003e-05 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:15.397606+0000", 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013839536 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(28.c REQUEST e379)", 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:45:34.516696+0000", 2024-02-13T00:53:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 450.61452145200002, 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003757631, 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516696+0000", 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516696+0000", 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516695+0000", 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1699999999999997e-06 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516703+0000", 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0827000000000001e-05 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516734+0000", 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5639000000000001e-05 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516750+0000", 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011122500000000001 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516861+0000", 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6915999999999999e-05 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516878+0000", 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035454750000000002 2024-02-13T00:53:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.520424+0000", 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0348e-05 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.520434+0000", 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9860999999999998e-05 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.520454+0000", 2024-02-13T00:53:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003757631 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(6.17 REQUEST e438)", 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:47:02.781512+0000", 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 362.34970544700002, 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043540109999999996, 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781512+0000", 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781512+0000", 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781512+0000", 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4500000000000003e-06 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781517+0000", 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4729999999999999e-05 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781542+0000", 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3523000000000001e-05 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781555+0000", 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010741799999999999 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781663+0000", 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9942999999999999e-05 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781683+0000", 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041526920000000004 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.785835+0000", 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2148e-05 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.785848+0000", 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8539999999999999e-05 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.785866+0000", 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043540109999999996 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(28.17 REQUEST e612)", 2024-02-13T00:53:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:50:26.821965+0000", 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 158.30925275199999, 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049237200000000004, 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821965+0000", 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821965+0000", 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821964+0000", 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1099999999999999e-06 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821970+0000", 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6339999999999999e-05 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821997+0000", 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4263e-05 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.822011+0000", 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012683799999999999 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.822138+0000", 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8040000000000002e-06 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.822146+0000", 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046927059999999996 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.826838+0000", 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9426e-05 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.826858+0000", 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0803000000000003e-05 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.826889+0000", 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049237200000000004 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.5s1 639/599 ECSubWrite(tid=706, reqid=client.27753.0:474, at_version=0'0, trim_to=0'0, roll_forward_to=639'6))", 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:04.536955+0000", 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 120.59426240000001, 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012597979999999999, 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536955+0000", 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536955+0000", 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536955+0000", 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2790000000000004e-06 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536964+0000", 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8669999999999996e-06 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536971+0000", 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1420000000000004e-06 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536978+0000", 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010723200000000001 2024-02-13T00:53:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.537085+0000", 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0047999999999999e-05 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.537095+0000", 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011055310000000001 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.538201+0000", 2024-02-13T00:53:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.396e-05 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.538215+0000", 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012597979999999999 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(6.e REQUEST e640)", 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:05.640107+0000", 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 119.49111087199999, 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00283475, 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640107+0000", 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640107+0000", 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640106+0000", 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4880000000000001e-06 2024-02-13T00:53:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640109+0000", 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.359e-05 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640123+0000", 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6920000000000002e-06 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640130+0000", 2024-02-13T00:53:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0739000000000003e-05 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640181+0000", 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8570000000000007e-06 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640191+0000", 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026984040000000002 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.642889+0000", 2024-02-13T00:53:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6985000000000001e-05 2024-02-13T00:53:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.642906+0000", 2024-02-13T00:53:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5238999999999999e-05 2024-02-13T00:53:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.642941+0000", 2024-02-13T00:53:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00283475 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.7s1 660/599 ECSubWrite(tid=1084, reqid=osd.1.0:1084, at_version=660'16, trim_to=0'0, roll_forward_to=656'12))", 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:25.619279+0000", 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 99.511938885000006, 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028506830000000001, 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619279+0000", 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619279+0000", 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619278+0000", 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8071e-05 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619296+0000", 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2529e-05 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619309+0000", 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0450000000000003e-06 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619316+0000", 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4156000000000001e-05 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619370+0000", 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2890000000000003e-06 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619379+0000", 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027309360000000002 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.622110+0000", 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9262999999999998e-05 2024-02-13T00:53:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.622129+0000", 2024-02-13T00:53:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028506830000000001 2024-02-13T00:53:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.7s1 660/599 ECSubWrite(tid=1085, reqid=osd.1.0:1084, at_version=0'0, trim_to=0'0, roll_forward_to=660'16))", 2024-02-13T00:53:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:25.624225+0000", 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 99.506992621999999, 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011742090000000001, 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624225+0000", 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624225+0000", 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624224+0000", 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.024e-05 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624235+0000", 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6730000000000004e-06 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624244+0000", 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6660000000000002e-06 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624251+0000", 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8631000000000002e-05 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624309+0000", 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4010000000000003e-06 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624319+0000", 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010623919999999999 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.625381+0000", 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7849000000000001e-05 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.625399+0000", 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011742090000000001 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.1.0:1092 243.f e676/675 243:f2a1c75f:test-rados-api-smithi162-84057-119::foo:head v 676'1, mlcod=0'0)", 2024-02-13T00:53:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:41.579543+0000", 2024-02-13T00:53:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 83.551674477000006, 2024-02-13T00:53:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081382700000000004, 2024-02-13T00:53:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579543+0000", 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579543+0000", 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579542+0000", 2024-02-13T00:53:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9680000000000001e-06 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579552+0000", 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2031999999999999e-05 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579564+0000", 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6990000000000003e-06 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579572+0000", 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9949e-05 2024-02-13T00:53:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579612+0000", 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8391000000000001e-05 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579630+0000", 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00069848999999999996 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580329+0000", 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8271000000000001e-05 2024-02-13T00:53:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580357+0000", 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081382700000000004 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.27753.0:1101 243.f e676/675 243:f2a1c75f:test-rados-api-smithi162-84057-119::foo:head v 676'2, mlcod=0'0)", 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:41.580331+0000", 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 83.550886605000002, 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00087110900000000001, 2024-02-13T00:53:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580331+0000", 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580331+0000", 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580330+0000", 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2383e-05 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580342+0000", 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0047000000000001e-05 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580353+0000", 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6449999999999994e-06 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580361+0000", 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6291999999999997e-05 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580397+0000", 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8377e-05 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580416+0000", 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00075487600000000003 2024-02-13T00:53:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.581171+0000", 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1349000000000001e-05 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.581202+0000", 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00087110900000000001 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.d epoch 677/676 rep_tid 1908 entries 677'14 (0'0) error 5:b4812045:::notify.4:head by client.24329.0:2087 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:44.375642+0000", 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.755575856999997, 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065348800000000003, 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375642+0000", 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375642+0000", 2024-02-13T00:53:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375641+0000", 2024-02-13T00:53:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2651e-05 2024-02-13T00:53:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375654+0000", 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7756000000000001e-05 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375671+0000", 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4800000000000001e-06 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375680+0000", 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010102000000000001 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375781+0000", 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00051420100000000005 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376295+0000", 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065348800000000003 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.e epoch 677/676 rep_tid 1907 entries 677'14 (0'0) error 5:7759931f:::notify.3:head by client.24329.0:2086 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:53:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:44.375694+0000", 2024-02-13T00:53:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.755523315000005, 2024-02-13T00:53:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081520500000000005, 2024-02-13T00:53:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375694+0000", 2024-02-13T00:53:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375694+0000", 2024-02-13T00:53:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375694+0000", 2024-02-13T00:53:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7979999999999996e-06 2024-02-13T00:53:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375699+0000", 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2609999999999996e-06 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375706+0000", 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9020000000000004e-06 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375712+0000", 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014652799999999999 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375858+0000", 2024-02-13T00:53:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000651028 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376509+0000", 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081520500000000005 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.e epoch 677/676 rep_tid 1911 entries 677'15 (0'0) error 5:7759931f:::notify.3:head by client.24329.0:2091 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:44.376063+0000", 2024-02-13T00:53:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.755155016000003, 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00085006100000000005, 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376063+0000", 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376063+0000", 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376062+0000", 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9030000000000003e-06 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376067+0000", 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5970000000000003e-06 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376075+0000", 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0220000000000001e-06 2024-02-13T00:53:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376081+0000", 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000341861 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376423+0000", 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049005800000000003 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376913+0000", 2024-02-13T00:53:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00085006100000000005 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s2 697/675 ECSubWrite(tid=779, reqid=osd.1.13:208, at_version=697'2, trim_to=0'0, roll_forward_to=0'0))", 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:04.576118+0000", 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 60.555099069999997, 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012258690000000001, 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576118+0000", 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576118+0000", 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576118+0000", 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7736999999999999e-05 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576145+0000", 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0842000000000001e-05 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576156+0000", 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7569999999999993e-06 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576165+0000", 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061519500000000002 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576780+0000", 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9189999999999994e-06 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576790+0000", 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00053753199999999996 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.577327+0000", 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6843000000000001e-05 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.577344+0000", 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012258690000000001 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s2 697/675 ECSubWrite(tid=781, reqid=osd.1.13:209, at_version=697'4, trim_to=0'0, roll_forward_to=697'2))", 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:04.580507+0000", 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 60.550710183, 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061816799999999999, 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580507+0000", 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580507+0000", 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580507+0000", 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4573e-05 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580531+0000", 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0698e-05 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580542+0000", 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2680000000000002e-06 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580547+0000", 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1179e-05 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580588+0000", 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1452e-05 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580600+0000", 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050946100000000001 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.581109+0000", 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6378000000000001e-05 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.581126+0000", 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061816799999999999 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s2 697/675 ECSubWrite(tid=783, reqid=osd.1.13:210, at_version=697'6, trim_to=0'0, roll_forward_to=697'4))", 2024-02-13T00:53:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:04.586217+0000", 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 60.545000547000001, 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057734300000000002, 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586217+0000", 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586217+0000", 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:53:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586216+0000", 2024-02-13T00:53:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9509000000000002e-05 2024-02-13T00:53:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586246+0000", 2024-02-13T00:53:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1206000000000001e-05 2024-02-13T00:53:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586257+0000", 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7780000000000002e-06 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586263+0000", 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1049000000000005e-05 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586325+0000", 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7629999999999995e-06 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586334+0000", 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00043614599999999999 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586770+0000", 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3896e-05 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586794+0000", 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057734300000000002 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s2 700/675 ECSubWrite(tid=788, reqid=osd.6.0:788, at_version=700'8, trim_to=0'0, roll_forward_to=697'6))", 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:07.568941+0000", 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 57.562276974, 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062311400000000002, 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.568941+0000", 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.568941+0000", 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999905 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.568931+0000", 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7778999999999999e-05 2024-02-13T00:53:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.568969+0000", 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5094e-05 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.568984+0000", 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4279999999999996e-06 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.568992+0000", 2024-02-13T00:53:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7242000000000004e-05 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.569059+0000", 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1119e-05 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.569070+0000", 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046841800000000001 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.569539+0000", 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4779000000000001e-05 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.569564+0000", 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062311400000000002 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s2 700/675 ECSubWrite(tid=789, reqid=osd.6.0:788, at_version=0'0, trim_to=0'0, roll_forward_to=700'8))", 2024-02-13T00:53:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:07.574196+0000", 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 57.557021311, 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058833900000000001, 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574196+0000", 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574196+0000", 2024-02-13T00:53:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574196+0000", 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.204e-05 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574208+0000", 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.275e-05 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574220+0000", 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7740000000000004e-06 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574228+0000", 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014377300000000001 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574372+0000", 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1268e-05 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574383+0000", 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00037631200000000001 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574759+0000", 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5117e-05 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574785+0000", 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058833900000000001 2024-02-13T00:53:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s2 716/675 ECSubWrite(tid=792, reqid=osd.6.45:144, at_version=716'9, trim_to=0'0, roll_forward_to=700'8))", 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:23.590917+0000", 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.540300180000003, 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071739589999999997, 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.590917+0000", 2024-02-13T00:53:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.590917+0000", 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.590917+0000", 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012856810000000001 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.592202+0000", 2024-02-13T00:53:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6427e-05 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.592219+0000", 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1756000000000001e-05 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.592231+0000", 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121882 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.592352+0000", 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2697000000000001e-05 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.592365+0000", 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056990820000000003 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.598064+0000", 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7070999999999999e-05 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.598091+0000", 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071739589999999997 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(4.b REQUEST e721)", 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:28.978959+0000", 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 36.152258316999998, 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0060215579999999998, 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:53:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.978959+0000", 2024-02-13T00:53:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.978959+0000", 2024-02-13T00:53:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.978959+0000", 2024-02-13T00:53:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6009999999999997e-06 2024-02-13T00:53:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.978963+0000", 2024-02-13T00:53:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2576e-05 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.978976+0000", 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8340000000000001e-06 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.978983+0000", 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5028000000000003e-05 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.979028+0000", 2024-02-13T00:53:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4430000000000002e-06 2024-02-13T00:53:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:53:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.979033+0000", 2024-02-13T00:53:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058818209999999998 2024-02-13T00:53:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:05.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.984915+0000", 2024-02-13T00:53:05.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4091999999999998e-05 2024-02-13T00:53:05.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:05.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.984949+0000", 2024-02-13T00:53:05.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1514999999999999e-05 2024-02-13T00:53:05.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:05.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:05.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:05.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.984981+0000", 2024-02-13T00:53:05.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0060215579999999998 2024-02-13T00:53:05.225 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.225 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.225 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.225 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:05.225 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:05.225 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:53:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:05.226 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:05.226 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:05.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:05.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:05.356 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:05.375 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:05.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:05.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:05.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:05.507 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:05.524 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:05.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:05.628 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:05.655 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:04.532315+0000 mgr.y (mgr.14544) 619 : cluster [DBG] pgmap v1130: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:53:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:05.555361+0000 mon.a (mon.0) 3570 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:53:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:05.560851+0000 mon.a (mon.0) 3571 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-13T00:53:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:05.560917+0000 mon.a (mon.0) 3572 : cluster [DBG] osdmap e747: 8 total, 8 up, 4 in 2024-02-13T00:53:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:05.563902+0000 mon.c (mon.2) 503 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.a", "id": [6, 7]}]: dispatch 2024-02-13T00:53:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:05.564248+0000 mon.a (mon.0) 3573 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-13T00:53:05.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:05.564548+0000 mon.a (mon.0) 3574 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.a", "id": [6, 7]}]: dispatch 2024-02-13T00:53:05.843 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-13T00:53:05.853 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:05.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:05.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:05.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:05.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:04.532315+0000 mgr.y (mgr.14544) 619 : cluster [DBG] pgmap v1130: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:53:05.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:05.555361+0000 mon.a (mon.0) 3570 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:53:05.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:05.560851+0000 mon.a (mon.0) 3571 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-13T00:53:05.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:05.560917+0000 mon.a (mon.0) 3572 : cluster [DBG] osdmap e747: 8 total, 8 up, 4 in 2024-02-13T00:53:05.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:05.563902+0000 mon.c (mon.2) 503 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.a", "id": [6, 7]}]: dispatch 2024-02-13T00:53:05.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:05.564248+0000 mon.a (mon.0) 3573 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-13T00:53:05.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:05.564548+0000 mon.a (mon.0) 3574 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.a", "id": [6, 7]}]: dispatch 2024-02-13T00:53:05.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:04.532315+0000 mgr.y (mgr.14544) 619 : cluster [DBG] pgmap v1130: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:53:05.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:05.555361+0000 mon.a (mon.0) 3570 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:53:05.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:05.560851+0000 mon.a (mon.0) 3571 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-13T00:53:05.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:05.560917+0000 mon.a (mon.0) 3572 : cluster [DBG] osdmap e747: 8 total, 8 up, 4 in 2024-02-13T00:53:05.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:05.563902+0000 mon.c (mon.2) 503 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.a", "id": [6, 7]}]: dispatch 2024-02-13T00:53:05.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:05.564248+0000 mon.a (mon.0) 3573 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-13T00:53:05.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:05.564548+0000 mon.a (mon.0) 3574 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.a", "id": [6, 7]}]: dispatch 2024-02-13T00:53:05.985 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:06.004 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:06.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:06.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:06.118 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:53:06.134 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:06.152 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:06.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:06.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:06.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:06.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:53:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:53:06.282 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:06.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:53:06] "GET /metrics HTTP/1.1" 200 189968 "" "Prometheus/2.33.4" 2024-02-13T00:53:06.405 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:06.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:06.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:06.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:53:06.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:06.536 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:06.555 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:06.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:06.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:06.659 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:06.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:06.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:06.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:06.730 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:06.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:05.935159+0000 mgr.y (mgr.14544) 620 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:06.020938+0000 osd.0 (osd.0) 293 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:53:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:06.023446+0000 osd.0 (osd.0) 294 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:53:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:06.564862+0000 mon.a (mon.0) 3575 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-13T00:53:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:06.564942+0000 mon.a (mon.0) 3576 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.a", "id": [6, 7]}]': finished 2024-02-13T00:53:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:06.565008+0000 mon.a (mon.0) 3577 : cluster [DBG] osdmap e748: 8 total, 8 up, 4 in 2024-02-13T00:53:06.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:06.566472+0000 mon.a (mon.0) 3578 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-13T00:53:06.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:05.935159+0000 mgr.y (mgr.14544) 620 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:06.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:06.020938+0000 osd.0 (osd.0) 293 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:53:06.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:06.023446+0000 osd.0 (osd.0) 294 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:53:06.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:06.564862+0000 mon.a (mon.0) 3575 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-13T00:53:06.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:06.564942+0000 mon.a (mon.0) 3576 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.a", "id": [6, 7]}]': finished 2024-02-13T00:53:06.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:06.565008+0000 mon.a (mon.0) 3577 : cluster [DBG] osdmap e748: 8 total, 8 up, 4 in 2024-02-13T00:53:06.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:06.566472+0000 mon.a (mon.0) 3578 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-13T00:53:06.839 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:07.036 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:07.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:05.935159+0000 mgr.y (mgr.14544) 620 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:06.020938+0000 osd.0 (osd.0) 293 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:53:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:06.023446+0000 osd.0 (osd.0) 294 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:53:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:06.564862+0000 mon.a (mon.0) 3575 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-13T00:53:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:06.564942+0000 mon.a (mon.0) 3576 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.a", "id": [6, 7]}]': finished 2024-02-13T00:53:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:06.565008+0000 mon.a (mon.0) 3577 : cluster [DBG] osdmap e748: 8 total, 8 up, 4 in 2024-02-13T00:53:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:06.566472+0000 mon.a (mon.0) 3578 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-13T00:53:07.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:07.169 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:07.266 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:07.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:07.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:53:07.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:07.411 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:07.569 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:07.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:07.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:07.705 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:06.532684+0000 mgr.y (mgr.14544) 621 : cluster [DBG] pgmap v1133: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:53:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:07.564424+0000 mon.a (mon.0) 3579 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-13T00:53:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:07.564506+0000 mon.a (mon.0) 3580 : cluster [DBG] osdmap e749: 8 total, 8 up, 4 in 2024-02-13T00:53:07.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:07.603013+0000 mon.a (mon.0) 3581 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:06.532684+0000 mgr.y (mgr.14544) 621 : cluster [DBG] pgmap v1133: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:53:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:07.564424+0000 mon.a (mon.0) 3579 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-13T00:53:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:07.564506+0000 mon.a (mon.0) 3580 : cluster [DBG] osdmap e749: 8 total, 8 up, 4 in 2024-02-13T00:53:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:07.603013+0000 mon.a (mon.0) 3581 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:07.906 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:07.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:07.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:08.009 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:08.037 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:08.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:06.532684+0000 mgr.y (mgr.14544) 621 : cluster [DBG] pgmap v1133: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:53:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:07.564424+0000 mon.a (mon.0) 3579 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-132","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-13T00:53:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:07.564506+0000 mon.a (mon.0) 3580 : cluster [DBG] osdmap e749: 8 total, 8 up, 4 in 2024-02-13T00:53:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:07.603013+0000 mon.a (mon.0) 3581 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:08.090 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:08.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:08.125 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:08.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:08.223 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:08.296 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:08.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:08.384 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 6, 7] out_osds: [3, 2, 5, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-13T00:53:08.385 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:53:08.385 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.meta 2024-02-13T00:53:08.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:53:08.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:08.412 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:53:08.412 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:53:08.412 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:53:08.412 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:53:08.455 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:08.578 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:08.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:08.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:08.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:08.709 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:53:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:53:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:53:08.712 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:08.786 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:08.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:08.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:08.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.787+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/1003815563 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e180f73a0 msgr2=0x7f6e181039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.787+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/1003815563 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e180f73a0 0x7f6e181039a0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 rx=0x7f6e10009be0 tx=0x7f6e10004d10).stop 2024-02-13T00:53:08.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.787+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/1003815563 shutdown_connections 2024-02-13T00:53:08.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.787+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/1003815563 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e180f73a0 0x7f6e181039a0 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.787+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/1003815563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e180fcb30 0x7f6e180fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.787+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/1003815563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e180fd490 0x7f6e180f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.787+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/1003815563 >> 172.21.15.162:0/1003815563 conn(0x7f6e1806eb90 msgr2=0x7f6e180f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:08.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.787+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/1003815563 shutdown_connections 2024-02-13T00:53:08.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.787+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/1003815563 wait complete. 2024-02-13T00:53:08.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1e8a0700 1 librados: starting msgr at 2024-02-13T00:53:08.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1e8a0700 1 librados: starting objecter 2024-02-13T00:53:08.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1e8a0700 1 Processor -- start 2024-02-13T00:53:08.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1e8a0700 1 -- start start 2024-02-13T00:53:08.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1e8a0700 1 librados: setting wanted keys 2024-02-13T00:53:08.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1e8a0700 1 librados: calling monclient init 2024-02-13T00:53:08.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1e8a0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e180f73a0 0x7f6e18109080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:08.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1e8a0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e180fcb30 0x7f6e18109570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:08.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1e8a0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e180fd490 0x7f6e18109b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:08.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1e8a0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e181142b0 con 0x7f6e180fcb30 2024-02-13T00:53:08.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1e8a0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e18114420 con 0x7f6e180fd490 2024-02-13T00:53:08.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1e8a0700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e181145e0 con 0x7f6e180f73a0 2024-02-13T00:53:08.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e177fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e180fcb30 0x7f6e18109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e17fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e180f73a0 0x7f6e18109080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e177fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e180fcb30 0x7f6e18109570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:49652/0 (socket says 172.21.15.162:49652) 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e177fe700 1 -- 172.21.15.162:0/3373533275 learned_addr learned my addr 172.21.15.162:0/3373533275 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.788+0000 7f6e1ce3d700 1 --2- 172.21.15.162:0/3373533275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e180fd490 0x7f6e18109b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e177fe700 1 -- 172.21.15.162:0/3373533275 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e180f73a0 msgr2=0x7f6e18109080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e177fe700 1 --2- 172.21.15.162:0/3373533275 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e180f73a0 0x7f6e18109080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e177fe700 1 -- 172.21.15.162:0/3373533275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e180fd490 msgr2=0x7f6e18109b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e177fe700 1 --2- 172.21.15.162:0/3373533275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e180fd490 0x7f6e18109b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e177fe700 1 -- 172.21.15.162:0/3373533275 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e00005040 con 0x7f6e180fcb30 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e17fff700 1 --2- 172.21.15.162:0/3373533275 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e180f73a0 0x7f6e18109080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e177fe700 1 --2- 172.21.15.162:0/3373533275 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e180fcb30 0x7f6e18109570 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 rx=0x7f6e080041d0 tx=0x7f6e08009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e157fa700 10 client.?.objecter ms_handle_connect 0x7f6e180fcb30 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e157fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e08007b40 con 0x7f6e180fcb30 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e100097c0 con 0x7f6e180fcb30 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e0800f040 con 0x7f6e180fcb30 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e08014730 con 0x7f6e180fcb30 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e1e8a0700 10 client.28056.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e18114c70 con 0x7f6e180fcb30 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.789+0000 7f6e1e8a0700 1 librados: init done 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.790+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6e18106090 con 0x7f6e180fcb30 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.790+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6e08009110 con 0x7f6e180fcb30 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e157fa700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6e04062c90 0x7f6e04065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(750..750 src has 449..750) v4 ==== 8682+0+0 (secure 0 0 0) 0x7f6e0807d8b0 con 0x7f6e180fcb30 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 osd_map(750..750 src has 449..750) v4 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e157fa700 3 client.28056.objecter handle_osd_map got epochs [750,750] > 0 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e157fa700 3 client.28056.objecter handle_osd_map decoding full epoch 750 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e17fff700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6e04062c90 0x7f6e04065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:08.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e157fa700 20 client.28056.objecter dump_active .. 0 homeless 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v750) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6e0807e0a0 con 0x7f6e180fcb30 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e157fa700 10 client.28056.objecter ms_handle_connect 0x7f6e04062c90 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e17fff700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6e04062c90 0x7f6e04065150 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 rx=0x7f6e00009760 tx=0x7f6e0000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 1 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6e18059990 0x7f6e18059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e180614a0 osd=0 3 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e180614a0 osd=0 3 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e180614a0 osd=0 3 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 1 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e180614a0 osd=0 3 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 0 1 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e180614a0 osd=0 4 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 1 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e1805fb80 con 0x7f6e18059990 2024-02-13T00:53:08.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.791+0000 7f6e1ce3d700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6e18059990 0x7f6e18059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:08.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.792+0000 7f6e157fa700 10 client.28056.objecter ms_handle_connect 0x7f6e18059990 2024-02-13T00:53:08.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.792+0000 7f6e1ce3d700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6e18059990 0x7f6e18059d70 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:08.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.792+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6e1805fb80 con 0x7f6e18059990 2024-02-13T00:53:08.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.792+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:53:08.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.792+0000 7f6e157fa700 10 client.28056.objecter _finish_command 1 = system:0 2024-02-13T00:53:08.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.792+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e180614a0 osd=0 4 2024-02-13T00:53:08.799 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.792+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 0 1 2024-02-13T00:53:08.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.808+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:08.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.808+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 2 2024-02-13T00:53:08.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.808+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e180614a0 osd=0 3 2024-02-13T00:53:08.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.808+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e180614a0 osd=0 3 2024-02-13T00:53:08.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.808+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e180614a0 osd=0 3 2024-02-13T00:53:08.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.808+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 2 2024-02-13T00:53:08.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.808+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e180614a0 osd=0 3 2024-02-13T00:53:08.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.808+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 0 2 2024-02-13T00:53:08.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.808+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e180614a0 osd=0 4 2024-02-13T00:53:08.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.808+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 2 2024-02-13T00:53:08.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.808+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e18028590 con 0x7f6e18059990 2024-02-13T00:53:08.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6e18028590 con 0x7f6e18059990 2024-02-13T00:53:08.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:08.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e157fa700 10 client.28056.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e180614a0 osd=0 4 2024-02-13T00:53:08.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 0 2 2024-02-13T00:53:08.814 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:53:08.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:08.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 3 2024-02-13T00:53:08.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6805/150599780 conn(0x7f6e18059570 0x7f6e18116f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:08.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18059400 osd=1 3 2024-02-13T00:53:08.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18059400 osd=1 3 2024-02-13T00:53:08.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18059400 osd=1 3 2024-02-13T00:53:08.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 3 2024-02-13T00:53:08.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e18059400 osd=1 3 2024-02-13T00:53:08.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 1 3 2024-02-13T00:53:08.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18059400 osd=1 4 2024-02-13T00:53:08.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 3 2024-02-13T00:53:08.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e18108e80 con 0x7f6e18059570 2024-02-13T00:53:08.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.810+0000 7f6e17fff700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6805/150599780 conn(0x7f6e18059570 0x7f6e18116f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:08.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.811+0000 7f6e157fa700 10 client.28056.objecter ms_handle_connect 0x7f6e18059570 2024-02-13T00:53:08.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.811+0000 7f6e17fff700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6805/150599780 conn(0x7f6e18059570 0x7f6e18116f00 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:08.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.811+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6e18108e80 con 0x7f6e18059570 2024-02-13T00:53:08.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.811+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:53:08.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.811+0000 7f6e157fa700 10 client.28056.objecter _finish_command 3 = system:0 2024-02-13T00:53:08.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.812+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e18059400 osd=1 4 2024-02-13T00:53:08.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.812+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 1 3 2024-02-13T00:53:08.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.835+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:08.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.835+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 4 2024-02-13T00:53:08.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.835+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18059400 osd=1 3 2024-02-13T00:53:08.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.835+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18059400 osd=1 3 2024-02-13T00:53:08.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.835+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18059400 osd=1 3 2024-02-13T00:53:08.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.835+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 4 2024-02-13T00:53:08.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.835+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e18059400 osd=1 3 2024-02-13T00:53:08.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.835+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 1 4 2024-02-13T00:53:08.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.835+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18059400 osd=1 4 2024-02-13T00:53:08.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.835+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 4 2024-02-13T00:53:08.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.835+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e180fe810 con 0x7f6e18059570 2024-02-13T00:53:08.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6e180fe810 con 0x7f6e18059570 2024-02-13T00:53:08.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:08.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e157fa700 10 client.28056.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.841 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:53:08.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e18059400 osd=1 4 2024-02-13T00:53:08.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 1 4 2024-02-13T00:53:08.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:08.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 5 2024-02-13T00:53:08.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6e180fdca0 0x7f6e18118570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:08.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e180fdb30 osd=2 3 2024-02-13T00:53:08.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e180fdb30 osd=2 3 2024-02-13T00:53:08.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e180fdb30 osd=2 3 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 5 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e180fdb30 osd=2 3 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 2 5 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e180fdb30 osd=2 4 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 5 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e180280f0 con 0x7f6e180fdca0 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.837+0000 7f6e177fe700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6e180fdca0 0x7f6e18118570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.838+0000 7f6e157fa700 10 client.28056.objecter ms_handle_connect 0x7f6e180fdca0 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.838+0000 7f6e177fe700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6e180fdca0 0x7f6e18118570 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.838+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6e180280f0 con 0x7f6e180fdca0 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.838+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.838+0000 7f6e157fa700 10 client.28056.objecter _finish_command 5 = system:0 2024-02-13T00:53:08.843 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.838+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e180fdb30 osd=2 4 2024-02-13T00:53:08.844 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.838+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 2 5 2024-02-13T00:53:08.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.852+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:08.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.852+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 6 2024-02-13T00:53:08.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.852+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e180fdb30 osd=2 3 2024-02-13T00:53:08.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.852+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e180fdb30 osd=2 3 2024-02-13T00:53:08.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.852+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e180fdb30 osd=2 3 2024-02-13T00:53:08.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.852+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 6 2024-02-13T00:53:08.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.852+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e180fdb30 osd=2 3 2024-02-13T00:53:08.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.852+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 2 6 2024-02-13T00:53:08.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.852+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e180fdb30 osd=2 4 2024-02-13T00:53:08.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.852+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 6 2024-02-13T00:53:08.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.852+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e18118a60 con 0x7f6e180fdca0 2024-02-13T00:53:08.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6e18118a60 con 0x7f6e180fdca0 2024-02-13T00:53:08.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:08.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e157fa700 10 client.28056.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e180fdb30 osd=2 4 2024-02-13T00:53:08.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 2 6 2024-02-13T00:53:08.858 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:53:08.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:08.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 7 2024-02-13T00:53:08.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6813/604971011 conn(0x7f6e1811fa80 0x7f6e18121e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:08.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18119400 osd=3 3 2024-02-13T00:53:08.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18119400 osd=3 3 2024-02-13T00:53:08.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18119400 osd=3 3 2024-02-13T00:53:08.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 7 2024-02-13T00:53:08.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e18119400 osd=3 3 2024-02-13T00:53:08.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 3 7 2024-02-13T00:53:08.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18119400 osd=3 4 2024-02-13T00:53:08.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 7 2024-02-13T00:53:08.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e18124ac0 con 0x7f6e1811fa80 2024-02-13T00:53:08.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.854+0000 7f6e1ce3d700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6813/604971011 conn(0x7f6e1811fa80 0x7f6e18121e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:08.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.855+0000 7f6e157fa700 10 client.28056.objecter ms_handle_connect 0x7f6e1811fa80 2024-02-13T00:53:08.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.855+0000 7f6e1ce3d700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6813/604971011 conn(0x7f6e1811fa80 0x7f6e18121e70 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:08.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.855+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6e18124ac0 con 0x7f6e1811fa80 2024-02-13T00:53:08.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.855+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:53:08.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.855+0000 7f6e157fa700 10 client.28056.objecter _finish_command 7 = system:0 2024-02-13T00:53:08.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.855+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e18119400 osd=3 4 2024-02-13T00:53:08.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.855+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 3 7 2024-02-13T00:53:08.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.869+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:08.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.869+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 8 2024-02-13T00:53:08.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.869+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18119400 osd=3 3 2024-02-13T00:53:08.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.869+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18119400 osd=3 3 2024-02-13T00:53:08.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.869+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18119400 osd=3 3 2024-02-13T00:53:08.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.869+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 8 2024-02-13T00:53:08.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.869+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e18119400 osd=3 3 2024-02-13T00:53:08.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.869+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 3 8 2024-02-13T00:53:08.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.869+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18119400 osd=3 4 2024-02-13T00:53:08.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.869+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 8 2024-02-13T00:53:08.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.869+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e18118e10 con 0x7f6e1811fa80 2024-02-13T00:53:08.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6e18118e10 con 0x7f6e1811fa80 2024-02-13T00:53:08.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:08.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e157fa700 10 client.28056.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e18119400 osd=3 4 2024-02-13T00:53:08.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 3 8 2024-02-13T00:53:08.875 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:53:08.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:08.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 9 2024-02-13T00:53:08.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6e18118fe0 0x7f6e181234e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:08.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18118e10 osd=4 3 2024-02-13T00:53:08.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18118e10 osd=4 3 2024-02-13T00:53:08.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18118e10 osd=4 3 2024-02-13T00:53:08.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 9 2024-02-13T00:53:08.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e18118e10 osd=4 3 2024-02-13T00:53:08.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 4 9 2024-02-13T00:53:08.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18118e10 osd=4 4 2024-02-13T00:53:08.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 9 2024-02-13T00:53:08.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e18118100 con 0x7f6e18118fe0 2024-02-13T00:53:08.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.871+0000 7f6e17fff700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6e18118fe0 0x7f6e181234e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:08.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.872+0000 7f6e157fa700 10 client.28056.objecter ms_handle_connect 0x7f6e18118fe0 2024-02-13T00:53:08.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.872+0000 7f6e17fff700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6e18118fe0 0x7f6e181234e0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:08.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.872+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6e18118100 con 0x7f6e18118fe0 2024-02-13T00:53:08.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.872+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:53:08.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.872+0000 7f6e157fa700 10 client.28056.objecter _finish_command 9 = system:0 2024-02-13T00:53:08.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.872+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e18118e10 osd=4 4 2024-02-13T00:53:08.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.872+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 4 9 2024-02-13T00:53:08.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:08.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.887+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:08.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.887+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 10 2024-02-13T00:53:08.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.887+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18118e10 osd=4 3 2024-02-13T00:53:08.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.887+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18118e10 osd=4 3 2024-02-13T00:53:08.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.887+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18118e10 osd=4 3 2024-02-13T00:53:08.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.887+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 10 2024-02-13T00:53:08.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.887+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e18118e10 osd=4 3 2024-02-13T00:53:08.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.887+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 4 10 2024-02-13T00:53:08.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.887+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18118e10 osd=4 4 2024-02-13T00:53:08.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.887+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 10 2024-02-13T00:53:08.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.887+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e18118100 con 0x7f6e18118fe0 2024-02-13T00:53:08.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6e18118100 con 0x7f6e18118fe0 2024-02-13T00:53:08.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:08.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e157fa700 10 client.28056.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e18118e10 osd=4 4 2024-02-13T00:53:08.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 4 10 2024-02-13T00:53:08.893 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:53:08.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:08.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 11 2024-02-13T00:53:08.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6e18123ee0 0x7f6e18124320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:08.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18118350 osd=5 3 2024-02-13T00:53:08.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18118350 osd=5 3 2024-02-13T00:53:08.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18118350 osd=5 3 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 11 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e18118350 osd=5 3 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 5 11 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18118350 osd=5 4 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 11 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.889+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e18124840 con 0x7f6e18123ee0 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.890+0000 7f6e177fe700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6e18123ee0 0x7f6e18124320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.890+0000 7f6e157fa700 10 client.28056.objecter ms_handle_connect 0x7f6e18123ee0 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.890+0000 7f6e177fe700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6e18123ee0 0x7f6e18124320 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.891+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6e18124840 con 0x7f6e18123ee0 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.891+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.891+0000 7f6e157fa700 10 client.28056.objecter _finish_command 11 = system:0 2024-02-13T00:53:08.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.891+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e18118350 osd=5 4 2024-02-13T00:53:08.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.891+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 5 11 2024-02-13T00:53:08.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.906+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:08.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.906+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 12 2024-02-13T00:53:08.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.906+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18118350 osd=5 3 2024-02-13T00:53:08.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.906+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18118350 osd=5 3 2024-02-13T00:53:08.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.906+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18118350 osd=5 3 2024-02-13T00:53:08.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.906+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 12 2024-02-13T00:53:08.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.906+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e18118350 osd=5 3 2024-02-13T00:53:08.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.906+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 5 12 2024-02-13T00:53:08.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.906+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18118350 osd=5 4 2024-02-13T00:53:08.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.906+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 12 2024-02-13T00:53:08.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.906+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e18124840 con 0x7f6e18123ee0 2024-02-13T00:53:08.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6e18124840 con 0x7f6e18123ee0 2024-02-13T00:53:08.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:08.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e157fa700 10 client.28056.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e18118350 osd=5 4 2024-02-13T00:53:08.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 5 12 2024-02-13T00:53:08.912 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:53:08.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:08.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 13 2024-02-13T00:53:08.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6e18129160 0x7f6e1812b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:08.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18124840 osd=6 3 2024-02-13T00:53:08.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18124840 osd=6 3 2024-02-13T00:53:08.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18124840 osd=6 3 2024-02-13T00:53:08.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 13 2024-02-13T00:53:08.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e18124840 osd=6 3 2024-02-13T00:53:08.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 6 13 2024-02-13T00:53:08.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18124840 osd=6 4 2024-02-13T00:53:08.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 13 2024-02-13T00:53:08.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e1812e3b0 con 0x7f6e18129160 2024-02-13T00:53:08.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.908+0000 7f6e1ce3d700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6e18129160 0x7f6e1812b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:08.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.909+0000 7f6e157fa700 10 client.28056.objecter ms_handle_connect 0x7f6e18129160 2024-02-13T00:53:08.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.909+0000 7f6e1ce3d700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6e18129160 0x7f6e1812b5b0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:08.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.909+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6e1812e3b0 con 0x7f6e18129160 2024-02-13T00:53:08.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.909+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:53:08.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.909+0000 7f6e157fa700 10 client.28056.objecter _finish_command 13 = system:0 2024-02-13T00:53:08.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.909+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e18124840 osd=6 4 2024-02-13T00:53:08.914 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.909+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 6 13 2024-02-13T00:53:08.917 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:08.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.924+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:08.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.924+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 14 2024-02-13T00:53:08.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.924+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18124840 osd=6 3 2024-02-13T00:53:08.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.924+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18124840 osd=6 3 2024-02-13T00:53:08.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.924+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e18124840 osd=6 3 2024-02-13T00:53:08.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.924+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 14 2024-02-13T00:53:08.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.924+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e18124840 osd=6 3 2024-02-13T00:53:08.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.924+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 6 14 2024-02-13T00:53:08.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.924+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18124840 osd=6 4 2024-02-13T00:53:08.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.924+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 14 2024-02-13T00:53:08.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.924+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e1812e3b0 con 0x7f6e18129160 2024-02-13T00:53:08.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6e1812e3b0 con 0x7f6e18129160 2024-02-13T00:53:08.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:08.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e157fa700 10 client.28056.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e18124840 osd=6 4 2024-02-13T00:53:08.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 6 14 2024-02-13T00:53:08.930 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:53:08.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:08.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 15 2024-02-13T00:53:08.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6e1812d6b0 0x7f6e1812db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:08.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e1812e3b0 osd=7 3 2024-02-13T00:53:08.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e1812e3b0 osd=7 3 2024-02-13T00:53:08.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e1812e3b0 osd=7 3 2024-02-13T00:53:08.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 15 2024-02-13T00:53:08.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e1812e3b0 osd=7 3 2024-02-13T00:53:08.931 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 7 15 2024-02-13T00:53:08.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e1812e3b0 osd=7 4 2024-02-13T00:53:08.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 15 2024-02-13T00:53:08.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.926+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e1812e160 con 0x7f6e1812d6b0 2024-02-13T00:53:08.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.927+0000 7f6e17fff700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6e1812d6b0 0x7f6e1812db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:08.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.927+0000 7f6e17fff700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6e1812d6b0 0x7f6e1812db10 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:08.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.927+0000 7f6e157fa700 10 client.28056.objecter ms_handle_connect 0x7f6e1812d6b0 2024-02-13T00:53:08.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.928+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6e1812e160 con 0x7f6e1812d6b0 2024-02-13T00:53:08.932 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.928+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:53:08.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.928+0000 7f6e157fa700 10 client.28056.objecter _finish_command 15 = system:0 2024-02-13T00:53:08.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.928+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e1812e3b0 osd=7 4 2024-02-13T00:53:08.933 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.928+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 7 15 2024-02-13T00:53:08.936 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:08.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:08.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.945+0000 7f6e1e8a0700 10 client.28056.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:08.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.945+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign -1 16 2024-02-13T00:53:08.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.945+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e1812e3b0 osd=7 3 2024-02-13T00:53:08.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.945+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e1812e3b0 osd=7 3 2024-02-13T00:53:08.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.945+0000 7f6e1e8a0700 20 client.28056.objecter _get_session s=0x7f6e1812e3b0 osd=7 3 2024-02-13T00:53:08.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.945+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_remove -1 16 2024-02-13T00:53:08.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.945+0000 7f6e1e8a0700 20 client.28056.objecter get_session s=0x7f6e1812e3b0 osd=7 3 2024-02-13T00:53:08.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.945+0000 7f6e1e8a0700 15 client.28056.objecter _session_command_op_assign 7 16 2024-02-13T00:53:08.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.945+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e1812e3b0 osd=7 4 2024-02-13T00:53:08.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.945+0000 7f6e1e8a0700 10 client.28056.objecter _send_command 16 2024-02-13T00:53:08.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.945+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6e1812e160 con 0x7f6e1812d6b0 2024-02-13T00:53:08.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.947+0000 7f6e157fa700 1 -- 172.21.15.162:0/3373533275 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6e1812e160 con 0x7f6e1812d6b0 2024-02-13T00:53:08.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.947+0000 7f6e157fa700 10 client.28056.objecter ms_dispatch 0x7f6e18001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:08.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.947+0000 7f6e157fa700 10 client.28056.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.947+0000 7f6e157fa700 20 client.28056.objecter put_session s=0x7f6e1812e3b0 osd=7 4 2024-02-13T00:53:08.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.947+0000 7f6e157fa700 15 client.28056.objecter _session_command_op_remove 7 16 2024-02-13T00:53:08.951 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:53:08.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 10 librados: watch_flush enter 2024-02-13T00:53:08.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 10 librados: watch_flush exit 2024-02-13T00:53:08.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 10 client.28056.objecter close_session for osd.0 2024-02-13T00:53:08.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6e18059990 msgr2=0x7f6e18059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6e18059990 0x7f6e18059d70 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e180614a0 osd=0 1 2024-02-13T00:53:08.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 10 client.28056.objecter close_session for osd.1 2024-02-13T00:53:08.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6805/150599780 conn(0x7f6e18059570 msgr2=0x7f6e18116f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6805/150599780 conn(0x7f6e18059570 0x7f6e18116f00 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18059400 osd=1 1 2024-02-13T00:53:08.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 10 client.28056.objecter close_session for osd.2 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6e180fdca0 msgr2=0x7f6e18118570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6e180fdca0 0x7f6e18118570 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e180fdb30 osd=2 1 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 10 client.28056.objecter close_session for osd.3 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1ce3d700 1 -- 172.21.15.162:0/3373533275 reap_dead start 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6813/604971011 conn(0x7f6e1811fa80 msgr2=0x7f6e18121e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6813/604971011 conn(0x7f6e1811fa80 0x7f6e18121e70 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18119400 osd=3 1 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 10 client.28056.objecter close_session for osd.4 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6e18118fe0 msgr2=0x7f6e181234e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6e18118fe0 0x7f6e181234e0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18118e10 osd=4 1 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 10 client.28056.objecter close_session for osd.5 2024-02-13T00:53:08.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6e18123ee0 msgr2=0x7f6e18124320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6e18123ee0 0x7f6e18124320 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18118350 osd=5 1 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 10 client.28056.objecter close_session for osd.6 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6e18129160 msgr2=0x7f6e1812b5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6e18129160 0x7f6e1812b5b0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e18124840 osd=6 1 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 10 client.28056.objecter close_session for osd.7 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6e1812d6b0 msgr2=0x7f6e1812db10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6e1812d6b0 0x7f6e1812db10 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 20 client.28056.objecter put_session s=0x7f6e1812e3b0 osd=7 1 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 20 client.28056.objecter shutdown clearing up homeless session... 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 10 client.28056.objecter successfully canceled tick 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1ce3d700 1 -- 172.21.15.162:0/3373533275 reap_dead start 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6e04062c90 msgr2=0x7f6e04065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6e04062c90 0x7f6e04065150 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 rx=0x7f6e00009760 tx=0x7f6e0000b040).stop 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e180fcb30 msgr2=0x7f6e18109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e180fcb30 0x7f6e18109570 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 rx=0x7f6e080041d0 tx=0x7f6e08009f60).stop 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 shutdown_connections 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e180fcb30 0x7f6e18109570 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 --2- 172.21.15.162:0/3373533275 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6e04062c90 0x7f6e04065150 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 >> 172.21.15.162:0/3373533275 conn(0x7f6e1806eb90 msgr2=0x7f6e180fab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 shutdown_connections 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 -- 172.21.15.162:0/3373533275 wait complete. 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:08.948+0000 7f6e1e8a0700 1 librados: shutdown 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.956 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.957 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.957 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:53:08.957 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:53:09.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:09.069 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:09.142 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:09.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:09.153 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-13T00:53:09.246 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:09.272 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:09.291 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:09.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:09.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:09.422 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:09.463 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:53:09.620 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:09.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:09.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:09.626 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:09.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:09.751 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:09.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:08.069454+0000 osd.0 (osd.0) 295 : cluster [DBG] 5.e scrub starts 2024-02-13T00:53:09.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:08.072912+0000 osd.0 (osd.0) 296 : cluster [DBG] 5.e scrub ok 2024-02-13T00:53:09.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:08.371406+0000 osd.6 (osd.6) 235 : cluster [DBG] 28.b scrub starts 2024-02-13T00:53:09.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:08.374983+0000 osd.6 (osd.6) 236 : cluster [DBG] 28.b scrub ok 2024-02-13T00:53:09.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:08.533081+0000 mgr.y (mgr.14544) 622 : cluster [DBG] pgmap v1136: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:53:09.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:08.570240+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:53:09.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:08.570302+0000 mon.a (mon.0) 3583 : cluster [DBG] osdmap e750: 8 total, 8 up, 4 in 2024-02-13T00:53:09.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:08.572404+0000 mon.a (mon.0) 3584 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132"}]: dispatch 2024-02-13T00:53:09.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:08.791332+0000 mon.a (mon.0) 3585 : audit [DBG] from='client.? 172.21.15.162:0/3373533275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:09.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:08.069454+0000 osd.0 (osd.0) 295 : cluster [DBG] 5.e scrub starts 2024-02-13T00:53:09.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:08.072912+0000 osd.0 (osd.0) 296 : cluster [DBG] 5.e scrub ok 2024-02-13T00:53:09.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:08.371406+0000 osd.6 (osd.6) 235 : cluster [DBG] 28.b scrub starts 2024-02-13T00:53:09.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:08.374983+0000 osd.6 (osd.6) 236 : cluster [DBG] 28.b scrub ok 2024-02-13T00:53:09.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:08.533081+0000 mgr.y (mgr.14544) 622 : cluster [DBG] pgmap v1136: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:53:09.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:08.570240+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:53:09.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:08.570302+0000 mon.a (mon.0) 3583 : cluster [DBG] osdmap e750: 8 total, 8 up, 4 in 2024-02-13T00:53:09.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:08.572404+0000 mon.a (mon.0) 3584 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132"}]: dispatch 2024-02-13T00:53:09.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:08.791332+0000 mon.a (mon.0) 3585 : audit [DBG] from='client.? 172.21.15.162:0/3373533275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:09.770 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:09.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:09.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:09.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:08.069454+0000 osd.0 (osd.0) 295 : cluster [DBG] 5.e scrub starts 2024-02-13T00:53:09.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:08.072912+0000 osd.0 (osd.0) 296 : cluster [DBG] 5.e scrub ok 2024-02-13T00:53:09.899 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:08.371406+0000 osd.6 (osd.6) 235 : cluster [DBG] 28.b scrub starts 2024-02-13T00:53:09.899 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:08.374983+0000 osd.6 (osd.6) 236 : cluster [DBG] 28.b scrub ok 2024-02-13T00:53:09.899 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:08.533081+0000 mgr.y (mgr.14544) 622 : cluster [DBG] pgmap v1136: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:53:09.899 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:08.570240+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:53:09.899 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:08.570302+0000 mon.a (mon.0) 3583 : cluster [DBG] osdmap e750: 8 total, 8 up, 4 in 2024-02-13T00:53:09.899 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:08.572404+0000 mon.a (mon.0) 3584 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132"}]: dispatch 2024-02-13T00:53:09.899 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:08.791332+0000 mon.a (mon.0) 3585 : audit [DBG] from='client.? 172.21.15.162:0/3373533275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:09.900 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:09.928 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:09.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:10.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:10.060 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:10.183 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:10.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:10.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:10.313 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:10.445 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:10.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:10.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:10.583 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:10.752 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:09.071684+0000 osd.0 (osd.0) 297 : cluster [DBG] 2.1e scrub starts 2024-02-13T00:53:10.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:09.074565+0000 osd.0 (osd.0) 298 : cluster [DBG] 2.1e scrub ok 2024-02-13T00:53:10.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:09.187056+0000 osd.1 (osd.1) 307 : cluster [DBG] 6.b scrub starts 2024-02-13T00:53:10.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:09.190019+0000 osd.1 (osd.1) 308 : cluster [DBG] 6.b scrub ok 2024-02-13T00:53:10.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:09.342015+0000 osd.6 (osd.6) 237 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:53:10.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:09.344920+0000 osd.6 (osd.6) 238 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:53:10.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:09.569124+0000 mon.a (mon.0) 3586 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132"}]': finished 2024-02-13T00:53:10.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:09.569503+0000 mon.a (mon.0) 3587 : cluster [DBG] osdmap e751: 8 total, 8 up, 4 in 2024-02-13T00:53:10.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:09.588312+0000 mon.a (mon.0) 3588 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:10.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:09.588651+0000 mon.a (mon.0) 3589 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132"}]: dispatch 2024-02-13T00:53:10.785 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:10.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:10.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:10.889 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:10.915 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:11.013 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:11.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:11.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:09.071684+0000 osd.0 (osd.0) 297 : cluster [DBG] 2.1e scrub starts 2024-02-13T00:53:11.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:09.074565+0000 osd.0 (osd.0) 298 : cluster [DBG] 2.1e scrub ok 2024-02-13T00:53:11.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:09.187056+0000 osd.1 (osd.1) 307 : cluster [DBG] 6.b scrub starts 2024-02-13T00:53:11.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:09.190019+0000 osd.1 (osd.1) 308 : cluster [DBG] 6.b scrub ok 2024-02-13T00:53:11.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:09.342015+0000 osd.6 (osd.6) 237 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:53:11.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:09.344920+0000 osd.6 (osd.6) 238 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:53:11.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:09.569124+0000 mon.a (mon.0) 3586 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132"}]': finished 2024-02-13T00:53:11.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:09.569503+0000 mon.a (mon.0) 3587 : cluster [DBG] osdmap e751: 8 total, 8 up, 4 in 2024-02-13T00:53:11.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:09.588312+0000 mon.a (mon.0) 3588 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:11.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:09.588651+0000 mon.a (mon.0) 3589 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132"}]: dispatch 2024-02-13T00:53:11.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:09.071684+0000 osd.0 (osd.0) 297 : cluster [DBG] 2.1e scrub starts 2024-02-13T00:53:11.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:09.074565+0000 osd.0 (osd.0) 298 : cluster [DBG] 2.1e scrub ok 2024-02-13T00:53:11.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:09.187056+0000 osd.1 (osd.1) 307 : cluster [DBG] 6.b scrub starts 2024-02-13T00:53:11.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:09.190019+0000 osd.1 (osd.1) 308 : cluster [DBG] 6.b scrub ok 2024-02-13T00:53:11.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:09.342015+0000 osd.6 (osd.6) 237 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:53:11.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:09.344920+0000 osd.6 (osd.6) 238 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:53:11.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:09.569124+0000 mon.a (mon.0) 3586 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132"}]': finished 2024-02-13T00:53:11.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:09.569503+0000 mon.a (mon.0) 3587 : cluster [DBG] osdmap e751: 8 total, 8 up, 4 in 2024-02-13T00:53:11.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:09.588312+0000 mon.a (mon.0) 3588 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:11.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:09.588651+0000 mon.a (mon.0) 3589 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-132"}]: dispatch 2024-02-13T00:53:11.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:11.144 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:11.267 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:11.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:11.289 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:53:11.290 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:53:11.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:11.400 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:11.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:11.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:11.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:11.633 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:11.656 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:11.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:11.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:11.788 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:10.214668+0000 osd.1 (osd.1) 309 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:53:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:10.218137+0000 osd.1 (osd.1) 310 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:53:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:10.297375+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:53:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:10.300839+0000 osd.6 (osd.6) 240 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:53:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:10.533924+0000 mgr.y (mgr.14544) 623 : cluster [DBG] pgmap v1139: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-13T00:53:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:10.589468+0000 mon.a (mon.0) 3590 : cluster [DBG] osdmap e752: 8 total, 8 up, 4 in 2024-02-13T00:53:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:10.214668+0000 osd.1 (osd.1) 309 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:53:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:10.218137+0000 osd.1 (osd.1) 310 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:53:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:10.297375+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:53:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:10.300839+0000 osd.6 (osd.6) 240 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:53:11.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:10.533924+0000 mgr.y (mgr.14544) 623 : cluster [DBG] pgmap v1139: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-13T00:53:11.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:10.589468+0000 mon.a (mon.0) 3590 : cluster [DBG] osdmap e752: 8 total, 8 up, 4 in 2024-02-13T00:53:11.966 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:10.214668+0000 osd.1 (osd.1) 309 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:53:11.966 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:10.218137+0000 osd.1 (osd.1) 310 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:53:11.966 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:10.297375+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:53:11.967 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:10.300839+0000 osd.6 (osd.6) 240 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:53:11.967 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:10.533924+0000 mgr.y (mgr.14544) 623 : cluster [DBG] pgmap v1139: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-13T00:53:11.967 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:10.589468+0000 mon.a (mon.0) 3590 : cluster [DBG] osdmap e752: 8 total, 8 up, 4 in 2024-02-13T00:53:11.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-13T00:53:11.986 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:11.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:11.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:12.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:12.115 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:12.134 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:12.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:12.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:53:12.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:12.253 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:53:12.272 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:12.470 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:12.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:12.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:12.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:12.602 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:12.629 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:12.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:12.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:12.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:12.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:11.592951+0000 mon.a (mon.0) 3591 : cluster [DBG] osdmap e753: 8 total, 8 up, 4 in 2024-02-13T00:53:12.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:11.594493+0000 mon.a (mon.0) 3592 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:53:12.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:11.952338+0000 mon.c (mon.2) 504 : audit [INF] from='client.? 172.21.15.162:0/3190834127' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:12.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:11.952651+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:12.760 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:12.779 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:12.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:12.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 -- 172.21.15.162:0/1113961266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67900f8500 msgr2=0x7f6790104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:12.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 --2- 172.21.15.162:0/1113961266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67900f8500 0x7f6790104bd0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7f6788009c30 tx=0x7f6788004c50).stop 2024-02-13T00:53:12.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 -- 172.21.15.162:0/1113961266 shutdown_connections 2024-02-13T00:53:12.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 --2- 172.21.15.162:0/1113961266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67900f8500 0x7f6790104bd0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:12.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 --2- 172.21.15.162:0/1113961266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67900f9b70 0x7f67900f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:12.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 --2- 172.21.15.162:0/1113961266 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67900fa440 0x7f67900f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 -- 172.21.15.162:0/1113961266 >> 172.21.15.162:0/1113961266 conn(0x7f67900f38c0 msgr2=0x7f67900f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 -- 172.21.15.162:0/1113961266 shutdown_connections 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 -- 172.21.15.162:0/1113961266 wait complete. 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 librados: starting msgr at 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 librados: starting objecter 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 Processor -- start 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 -- start start 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 librados: setting wanted keys 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.764+0000 7f6795f59700 1 librados: calling monclient init 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f6795f59700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67900f8500 0x7f679010e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f6795f59700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67900f9b70 0x7f679010e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f6795f59700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67900fa440 0x7f67901082f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f6795f59700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67901087e0 con 0x7f67900fa440 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f6795f59700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6790108950 con 0x7f67900f8500 2024-02-13T00:53:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f6795f59700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6790108b10 con 0x7f67900f9b70 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678effd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67900f9b70 0x7f679010e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67900fa440 0x7f67901082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678effd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67900f9b70 0x7f679010e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:39066/0 (socket says 172.21.15.162:39066) 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678effd700 1 -- 172.21.15.162:0/3190834127 learned_addr learned my addr 172.21.15.162:0/3190834127 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678f7fe700 1 --2- 172.21.15.162:0/3190834127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67900f8500 0x7f679010e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678effd700 1 -- 172.21.15.162:0/3190834127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67900f8500 msgr2=0x7f679010e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678effd700 1 --2- 172.21.15.162:0/3190834127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67900f8500 0x7f679010e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678effd700 1 -- 172.21.15.162:0/3190834127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67900fa440 msgr2=0x7f67901082f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678effd700 1 --2- 172.21.15.162:0/3190834127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67900fa440 0x7f67901082f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678effd700 1 -- 172.21.15.162:0/3190834127 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6780005040 con 0x7f67900f9b70 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678ffff700 1 --2- 172.21.15.162:0/3190834127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67900fa440 0x7f67901082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678effd700 1 --2- 172.21.15.162:0/3190834127 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67900f9b70 0x7f679010e640 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 rx=0x7f678400e970 tx=0x7f678400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:12.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678cff9700 10 client.?.objecter ms_handle_connect 0x7f67900f9b70 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678cff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678cff9700 1 -- 172.21.15.162:0/3190834127 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784004660 con 0x7f67900f9b70 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678cff9700 1 -- 172.21.15.162:0/3190834127 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f678400f040 con 0x7f67900f9b70 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f6795f59700 1 -- 172.21.15.162:0/3190834127 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67880097c0 con 0x7f67900f9b70 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f678cff9700 1 -- 172.21.15.162:0/3190834127 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784014790 con 0x7f67900f9b70 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f6795f59700 10 client.38159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f6795f59700 1 -- 172.21.15.162:0/3190834127 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6790114f80 con 0x7f67900f9b70 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.765+0000 7f6795f59700 1 librados: init done 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.766+0000 7f6795f59700 1 -- 172.21.15.162:0/3190834127 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67900475e0 con 0x7f67900f9b70 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.767+0000 7f678cff9700 1 -- 172.21.15.162:0/3190834127 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f67840090d0 con 0x7f67900f9b70 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.767+0000 7f678cff9700 1 --2- 172.21.15.162:0/3190834127 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6778062c90 0x7f6778065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.767+0000 7f678cff9700 1 -- 172.21.15.162:0/3190834127 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(753..753 src has 449..753) v4 ==== 8648+0+0 (secure 0 0 0) 0x7f678407cf10 con 0x7f67900f9b70 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.767+0000 7f678cff9700 10 client.38159.objecter ms_dispatch 0x7f6790001550 osd_map(753..753 src has 449..753) v4 2024-02-13T00:53:12.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.767+0000 7f678cff9700 3 client.38159.objecter handle_osd_map got epochs [753,753] > 0 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.767+0000 7f678cff9700 3 client.38159.objecter handle_osd_map decoding full epoch 753 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.767+0000 7f678f7fe700 1 --2- 172.21.15.162:0/3190834127 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6778062c90 0x7f6778065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.767+0000 7f678cff9700 20 client.38159.objecter dump_active .. 0 homeless 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.767+0000 7f678f7fe700 1 --2- 172.21.15.162:0/3190834127 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6778062c90 0x7f6778065150 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 rx=0x7f6780004870 tx=0x7f678000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.767+0000 7f678cff9700 10 client.38159.objecter ms_handle_connect 0x7f6778062c90 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.771+0000 7f678cff9700 1 -- 172.21.15.162:0/3190834127 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f678404af60 con 0x7f67900f9b70 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:11.951+0000 7f6795f59700 1 -- 172.21.15.162:0/3190834127 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f67901097d0 con 0x7f67900f9b70 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.602+0000 7f678cff9700 1 -- 172.21.15.162:0/3190834127 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v754) v1 ==== 101+0+0 (secure 0 0 0) 0x7f6784019070 con 0x7f67900f9b70 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.606+0000 7f6795f59700 10 librados: watch_flush enter 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.608+0000 7f6795f59700 10 librados: watch_flush exit 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.608+0000 7f6795f59700 20 client.38159.objecter shutdown clearing up homeless session... 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.608+0000 7f6795f59700 10 client.38159.objecter successfully canceled tick 2024-02-13T00:53:12.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.609+0000 7f6795f59700 1 -- 172.21.15.162:0/3190834127 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6778062c90 msgr2=0x7f6778065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.609+0000 7f6795f59700 1 --2- 172.21.15.162:0/3190834127 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6778062c90 0x7f6778065150 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 rx=0x7f6780004870 tx=0x7f678000b040).stop 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.609+0000 7f6795f59700 1 -- 172.21.15.162:0/3190834127 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67900f9b70 msgr2=0x7f679010e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.609+0000 7f6795f59700 1 --2- 172.21.15.162:0/3190834127 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67900f9b70 0x7f679010e640 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 rx=0x7f678400e970 tx=0x7f678400b040).stop 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.609+0000 7f6795f59700 1 -- 172.21.15.162:0/3190834127 shutdown_connections 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.609+0000 7f6795f59700 1 --2- 172.21.15.162:0/3190834127 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6778062c90 0x7f6778065150 unknown :-1 s=CLOSED pgs=3032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.609+0000 7f6795f59700 1 --2- 172.21.15.162:0/3190834127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67900fa440 0x7f67901082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.609+0000 7f6795f59700 1 --2- 172.21.15.162:0/3190834127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67900f8500 0x7f679010e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.609+0000 7f6795f59700 1 --2- 172.21.15.162:0/3190834127 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67900f9b70 0x7f679010e640 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.609+0000 7f6795f59700 1 -- 172.21.15.162:0/3190834127 >> 172.21.15.162:0/3190834127 conn(0x7f67900f38c0 msgr2=0x7f67900f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.609+0000 7f6795f59700 1 -- 172.21.15.162:0/3190834127 shutdown_connections 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.610+0000 7f6795f59700 1 -- 172.21.15.162:0/3190834127 wait complete. 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:12.610+0000 7f6795f59700 1 librados: shutdown 2024-02-13T00:53:12.842 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:53:12.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:12.910 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:13.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:11.592951+0000 mon.a (mon.0) 3591 : cluster [DBG] osdmap e753: 8 total, 8 up, 4 in 2024-02-13T00:53:13.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:11.594493+0000 mon.a (mon.0) 3592 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:53:13.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:11.952338+0000 mon.c (mon.2) 504 : audit [INF] from='client.? 172.21.15.162:0/3190834127' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:13.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:11.952651+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:13.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:11.592951+0000 mon.a (mon.0) 3591 : cluster [DBG] osdmap e753: 8 total, 8 up, 4 in 2024-02-13T00:53:13.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:11.594493+0000 mon.a (mon.0) 3592 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:53:13.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:11.952338+0000 mon.c (mon.2) 504 : audit [INF] from='client.? 172.21.15.162:0/3190834127' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:13.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:11.952651+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:13.108 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:13.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:13.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:13.241 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:13.857 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:13.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:12.196586+0000 osd.1 (osd.1) 311 : cluster [DBG] 5.0 scrub starts 2024-02-13T00:53:13.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:12.200058+0000 osd.1 (osd.1) 312 : cluster [DBG] 5.0 scrub ok 2024-02-13T00:53:13.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:12.534328+0000 mgr.y (mgr.14544) 624 : cluster [DBG] pgmap v1142: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:53:13.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:12.591390+0000 mon.a (mon.0) 3594 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:53:13.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:12.603178+0000 mon.a (mon.0) 3595 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:53:13.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:12.603267+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:53:13.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:12.603320+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e754: 8 total, 8 up, 4 in 2024-02-13T00:53:13.866 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:12.623758+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:53:13.866 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:13.133999+0000 osd.0 (osd.0) 299 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:13.866 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:13.134205+0000 osd.0 (osd.0) 300 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:13.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.885+0000 7f6db99de700 1 -- 172.21.15.162:0/3770173437 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6db4104660 msgr2=0x7f6db4106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:13.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.885+0000 7f6db99de700 1 --2- 172.21.15.162:0/3770173437 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6db4104660 0x7f6db4106a50 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 rx=0x7f6dac009c10 tx=0x7f6dac004d10).stop 2024-02-13T00:53:13.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.885+0000 7f6db99de700 1 -- 172.21.15.162:0/3770173437 shutdown_connections 2024-02-13T00:53:13.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.885+0000 7f6db99de700 1 --2- 172.21.15.162:0/3770173437 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6db4104660 0x7f6db4106a50 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:13.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.885+0000 7f6db99de700 1 --2- 172.21.15.162:0/3770173437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6db40f6b00 0x7f6db40f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:13.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.885+0000 7f6db99de700 1 --2- 172.21.15.162:0/3770173437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6db40f73d0 0x7f6db4104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.886+0000 7f6db99de700 1 -- 172.21.15.162:0/3770173437 >> 172.21.15.162:0/3770173437 conn(0x7f6db406eb90 msgr2=0x7f6db40f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.886+0000 7f6db99de700 1 -- 172.21.15.162:0/3770173437 shutdown_connections 2024-02-13T00:53:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.886+0000 7f6db99de700 1 -- 172.21.15.162:0/3770173437 wait complete. 2024-02-13T00:53:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.886+0000 7f6db99de700 1 librados: starting msgr at 2024-02-13T00:53:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.886+0000 7f6db99de700 1 librados: starting objecter 2024-02-13T00:53:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.886+0000 7f6db99de700 1 Processor -- start 2024-02-13T00:53:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.886+0000 7f6db99de700 1 -- start start 2024-02-13T00:53:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.886+0000 7f6db99de700 1 librados: setting wanted keys 2024-02-13T00:53:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.886+0000 7f6db99de700 1 librados: calling monclient init 2024-02-13T00:53:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db99de700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6db40f6b00 0x7f6db4110150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db99de700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6db40f73d0 0x7f6db4110640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:13.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db99de700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6db4104660 0x7f6db410a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db99de700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db410a7e0 con 0x7f6db40f73d0 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db99de700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db410a950 con 0x7f6db40f6b00 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db99de700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db410ab10 con 0x7f6db4104660 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db2ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6db40f6b00 0x7f6db4110150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db2ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6db40f6b00 0x7f6db4110150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:48902/0 (socket says 172.21.15.162:48902) 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db2ffd700 1 -- 172.21.15.162:0/1134466092 learned_addr learned my addr 172.21.15.162:0/1134466092 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db37fe700 1 --2- 172.21.15.162:0/1134466092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6db4104660 0x7f6db410a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db27fc700 1 --2- 172.21.15.162:0/1134466092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6db40f73d0 0x7f6db4110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db37fe700 1 -- 172.21.15.162:0/1134466092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6db40f6b00 msgr2=0x7f6db4110150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db37fe700 1 --2- 172.21.15.162:0/1134466092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6db40f6b00 0x7f6db4110150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db37fe700 1 -- 172.21.15.162:0/1134466092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6db40f73d0 msgr2=0x7f6db4110640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db37fe700 1 --2- 172.21.15.162:0/1134466092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6db40f73d0 0x7f6db4110640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db37fe700 1 -- 172.21.15.162:0/1134466092 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da8005040 con 0x7f6db4104660 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db2ffd700 1 --2- 172.21.15.162:0/1134466092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6db40f6b00 0x7f6db4110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db27fc700 1 --2- 172.21.15.162:0/1134466092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6db40f73d0 0x7f6db4110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:13.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db37fe700 1 --2- 172.21.15.162:0/1134466092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6db4104660 0x7f6db410a2f0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 rx=0x7f6dac00b6c0 tx=0x7f6dac004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6d9bfff700 10 client.?.objecter ms_handle_connect 0x7f6db4104660 2024-02-13T00:53:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6d9bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6d9bfff700 1 -- 172.21.15.162:0/1134466092 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac00e040 con 0x7f6db4104660 2024-02-13T00:53:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6d9bfff700 1 -- 172.21.15.162:0/1134466092 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6dac018040 con 0x7f6db4104660 2024-02-13T00:53:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6db99de700 1 -- 172.21.15.162:0/1134466092 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dac0097c0 con 0x7f6db4104660 2024-02-13T00:53:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.887+0000 7f6d9bfff700 1 -- 172.21.15.162:0/1134466092 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac013e50 con 0x7f6db4104660 2024-02-13T00:53:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.888+0000 7f6db99de700 10 client.38165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.888+0000 7f6db99de700 1 -- 172.21.15.162:0/1134466092 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db410b020 con 0x7f6db4104660 2024-02-13T00:53:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.888+0000 7f6db99de700 1 librados: init done 2024-02-13T00:53:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.889+0000 7f6db99de700 1 -- 172.21.15.162:0/1134466092 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db40475e0 con 0x7f6db4104660 2024-02-13T00:53:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.889+0000 7f6d9bfff700 1 -- 172.21.15.162:0/1134466092 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6dac007bf0 con 0x7f6db4104660 2024-02-13T00:53:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.889+0000 7f6d9bfff700 1 --2- 172.21.15.162:0/1134466092 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d9c062c90 0x7f6d9c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.890+0000 7f6d9bfff700 1 -- 172.21.15.162:0/1134466092 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(755..755 src has 449..755) v4 ==== 8656+0+0 (secure 0 0 0) 0x7f6dac07df90 con 0x7f6db4104660 2024-02-13T00:53:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.890+0000 7f6db2ffd700 1 --2- 172.21.15.162:0/1134466092 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d9c062c90 0x7f6d9c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.890+0000 7f6d9bfff700 10 client.38165.objecter ms_dispatch 0x7f6db4001550 osd_map(755..755 src has 449..755) v4 2024-02-13T00:53:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.890+0000 7f6d9bfff700 3 client.38165.objecter handle_osd_map got epochs [755,755] > 0 2024-02-13T00:53:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.890+0000 7f6d9bfff700 3 client.38165.objecter handle_osd_map decoding full epoch 755 2024-02-13T00:53:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.890+0000 7f6d9bfff700 20 client.38165.objecter dump_active .. 0 homeless 2024-02-13T00:53:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.890+0000 7f6d9bfff700 10 client.38165.objecter ms_handle_connect 0x7f6d9c062c90 2024-02-13T00:53:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.890+0000 7f6db2ffd700 1 --2- 172.21.15.162:0/1134466092 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d9c062c90 0x7f6d9c065150 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 rx=0x7f6da4004fd0 tx=0x7f6da400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:13.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:13.894+0000 7f6d9bfff700 1 -- 172.21.15.162:0/1134466092 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6dac0512c0 con 0x7f6db4104660 2024-02-13T00:53:14.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.060+0000 7f6db99de700 1 -- 172.21.15.162:0/1134466092 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6db410b8e0 con 0x7f6d9c062c90 2024-02-13T00:53:14.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.073+0000 7f6d9bfff700 1 -- 172.21.15.162:0/1134466092 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+504101 (secure 0 0 0) 0x7f6db410b8e0 con 0x7f6d9c062c90 2024-02-13T00:53:14.083 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:53:14.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:12.196586+0000 osd.1 (osd.1) 311 : cluster [DBG] 5.0 scrub starts 2024-02-13T00:53:14.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:12.200058+0000 osd.1 (osd.1) 312 : cluster [DBG] 5.0 scrub ok 2024-02-13T00:53:14.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:12.534328+0000 mgr.y (mgr.14544) 624 : cluster [DBG] pgmap v1142: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:53:14.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:12.591390+0000 mon.a (mon.0) 3594 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:53:14.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:12.603178+0000 mon.a (mon.0) 3595 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:53:14.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:12.603267+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:53:14.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:12.603320+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e754: 8 total, 8 up, 4 in 2024-02-13T00:53:14.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:12.623758+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:53:14.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:13.133999+0000 osd.0 (osd.0) 299 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:14.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:13.134205+0000 osd.0 (osd.0) 300 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:14.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:12.196586+0000 osd.1 (osd.1) 311 : cluster [DBG] 5.0 scrub starts 2024-02-13T00:53:14.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:12.200058+0000 osd.1 (osd.1) 312 : cluster [DBG] 5.0 scrub ok 2024-02-13T00:53:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:12.534328+0000 mgr.y (mgr.14544) 624 : cluster [DBG] pgmap v1142: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:53:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:12.591390+0000 mon.a (mon.0) 3594 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:53:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:12.603178+0000 mon.a (mon.0) 3595 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:53:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:12.603267+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:53:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:12.603320+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e754: 8 total, 8 up, 4 in 2024-02-13T00:53:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:12.623758+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:53:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:13.133999+0000 osd.0 (osd.0) 299 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:13.134205+0000 osd.0 (osd.0) 300 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:14.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 10 librados: watch_flush enter 2024-02-13T00:53:14.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 10 librados: watch_flush exit 2024-02-13T00:53:14.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 20 client.38165.objecter shutdown clearing up homeless session... 2024-02-13T00:53:14.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 10 client.38165.objecter successfully canceled tick 2024-02-13T00:53:14.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 -- 172.21.15.162:0/1134466092 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d9c062c90 msgr2=0x7f6d9c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:14.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 --2- 172.21.15.162:0/1134466092 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d9c062c90 0x7f6d9c065150 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 rx=0x7f6da4004fd0 tx=0x7f6da400b040).stop 2024-02-13T00:53:14.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 -- 172.21.15.162:0/1134466092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6db4104660 msgr2=0x7f6db410a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:14.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 --2- 172.21.15.162:0/1134466092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6db4104660 0x7f6db410a2f0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 rx=0x7f6dac00b6c0 tx=0x7f6dac004d10).stop 2024-02-13T00:53:14.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 -- 172.21.15.162:0/1134466092 shutdown_connections 2024-02-13T00:53:14.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 --2- 172.21.15.162:0/1134466092 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d9c062c90 0x7f6d9c065150 unknown :-1 s=CLOSED pgs=3033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:14.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 --2- 172.21.15.162:0/1134466092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6db4104660 0x7f6db410a2f0 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:14.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 --2- 172.21.15.162:0/1134466092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6db40f6b00 0x7f6db4110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:14.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 --2- 172.21.15.162:0/1134466092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6db40f73d0 0x7f6db4110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:14.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 -- 172.21.15.162:0/1134466092 >> 172.21.15.162:0/1134466092 conn(0x7f6db406eb90 msgr2=0x7f6db4070810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:14.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 -- 172.21.15.162:0/1134466092 shutdown_connections 2024-02-13T00:53:14.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 -- 172.21.15.162:0/1134466092 wait complete. 2024-02-13T00:53:14.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:14.077+0000 7f6db99de700 1 librados: shutdown 2024-02-13T00:53:14.090 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:53:14.149 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:53:14.149 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:53:14.708 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:13.605715+0000 mon.a (mon.0) 3599 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:53:14.708 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:13.605802+0000 mon.a (mon.0) 3600 : cluster [DBG] osdmap e755: 8 total, 8 up, 4 in 2024-02-13T00:53:14.708 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:13.606626+0000 mon.a (mon.0) 3601 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-134"}]: dispatch 2024-02-13T00:53:14.708 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:14.061038+0000 mgr.y (mgr.14544) 625 : audit [DBG] from='client.38165 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:53:14.708 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:14.065864+0000 osd.0 (osd.0) 301 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:53:14.708 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:14.066776+0000 osd.0 (osd.0) 302 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:53:14.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:13.605715+0000 mon.a (mon.0) 3599 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:53:14.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:13.605802+0000 mon.a (mon.0) 3600 : cluster [DBG] osdmap e755: 8 total, 8 up, 4 in 2024-02-13T00:53:14.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:13.606626+0000 mon.a (mon.0) 3601 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-134"}]: dispatch 2024-02-13T00:53:14.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:14.061038+0000 mgr.y (mgr.14544) 625 : audit [DBG] from='client.38165 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:53:14.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:14.065864+0000 osd.0 (osd.0) 301 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:53:14.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:14.066776+0000 osd.0 (osd.0) 302 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:53:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:13.605715+0000 mon.a (mon.0) 3599 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:53:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:13.605802+0000 mon.a (mon.0) 3600 : cluster [DBG] osdmap e755: 8 total, 8 up, 4 in 2024-02-13T00:53:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:13.606626+0000 mon.a (mon.0) 3601 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-134"}]: dispatch 2024-02-13T00:53:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:14.061038+0000 mgr.y (mgr.14544) 625 : audit [DBG] from='client.38165 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:53:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:14.065864+0000 osd.0 (osd.0) 301 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:53:14.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:14.066776+0000 osd.0 (osd.0) 302 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:53:14.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-13T00:53:15.050 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":1144,"stamp":"2024-02-13T00:53:13.604151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4006050,"num_objects":262,"num_object_clones":1,"num_object_copies":786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":2437,"num_read_kb":2144,"num_write":970,"num_write_kb":4007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":863,"num_bytes_recovered":4021117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1161,"ondisk_log_size":1161,"up":516,"acting":516,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":612,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1253756,"kb_used_data":27996,"kb_used_omap":0,"kb_used_meta":1225728,"kb_avail":373726852,"statfs":{"total":383980142592,"available":382696296448,"internally_reserved":0,"allocated":28667904,"data_stored":17824351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1255145472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.037060"},"pg_stats":[{"pgid":"251.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"251.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.7","version":"740'19","reported_seq":54,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571895+0000","last_change":"2024-02-13T00:52:31.270066+0000","last_active":"2024-02-13T00:53:08.571895+0000","last_peered":"2024-02-13T00:53:08.571895+0000","last_clean":"2024-02-13T00:53:08.571895+0000","last_became_active":"2024-02-13T00:52:30.594798+0000","last_became_peered":"2024-02-13T00:52:30.594798+0000","last_unstale":"2024-02-13T00:53:08.571895+0000","last_undegraded":"2024-02-13T00:53:08.571895+0000","last_fullsized":"2024-02-13T00:53:08.571895+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"251.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.6","version":"740'2","reported_seq":176,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111501+0000","last_change":"2024-02-13T00:52:07.570115+0000","last_active":"2024-02-13T00:53:09.111501+0000","last_peered":"2024-02-13T00:53:09.111501+0000","last_clean":"2024-02-13T00:53:09.111501+0000","last_became_active":"2024-02-13T00:50:14.552756+0000","last_became_peered":"2024-02-13T00:50:14.552756+0000","last_unstale":"2024-02-13T00:53:09.111501+0000","last_undegraded":"2024-02-13T00:53:09.111501+0000","last_fullsized":"2024-02-13T00:53:09.111501+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"251.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.5","version":"740'18","reported_seq":221,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111602+0000","last_change":"2024-02-13T00:52:34.332875+0000","last_active":"2024-02-13T00:53:09.111602+0000","last_peered":"2024-02-13T00:53:09.111602+0000","last_clean":"2024-02-13T00:53:09.111602+0000","last_became_active":"2024-02-13T00:52:30.595299+0000","last_became_peered":"2024-02-13T00:52:30.595299+0000","last_unstale":"2024-02-13T00:53:09.111602+0000","last_undegraded":"2024-02-13T00:53:09.111602+0000","last_fullsized":"2024-02-13T00:53:09.111602+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":24,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"251.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.4","version":"740'12","reported_seq":203,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.572347+0000","last_change":"2024-02-13T00:52:07.557573+0000","last_active":"2024-02-13T00:53:08.572347+0000","last_peered":"2024-02-13T00:53:08.572347+0000","last_clean":"2024-02-13T00:53:08.572347+0000","last_became_active":"2024-02-13T00:51:41.580904+0000","last_became_peered":"2024-02-13T00:51:41.580904+0000","last_unstale":"2024-02-13T00:53:08.572347+0000","last_undegraded":"2024-02-13T00:53:08.572347+0000","last_fullsized":"2024-02-13T00:53:08.572347+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"251.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.3","version":"740'12","reported_seq":215,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109740+0000","last_change":"2024-02-13T00:52:33.861253+0000","last_active":"2024-02-13T00:53:09.109740+0000","last_peered":"2024-02-13T00:53:09.109740+0000","last_clean":"2024-02-13T00:53:09.109740+0000","last_became_active":"2024-02-13T00:52:30.600633+0000","last_became_peered":"2024-02-13T00:52:30.600633+0000","last_unstale":"2024-02-13T00:53:09.109740+0000","last_undegraded":"2024-02-13T00:53:09.109740+0000","last_fullsized":"2024-02-13T00:53:09.109740+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000034,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":7,"num_write_kb":3913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4000010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"251.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.2","version":"740'19","reported_seq":226,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109572+0000","last_change":"2024-02-13T00:52:07.572797+0000","last_active":"2024-02-13T00:53:09.109572+0000","last_peered":"2024-02-13T00:53:09.109572+0000","last_clean":"2024-02-13T00:53:09.109572+0000","last_became_active":"2024-02-13T00:51:41.569720+0000","last_became_peered":"2024-02-13T00:51:41.569720+0000","last_unstale":"2024-02-13T00:53:09.109572+0000","last_undegraded":"2024-02-13T00:53:09.109572+0000","last_fullsized":"2024-02-13T00:53:09.109572+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"251.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.1","version":"740'3","reported_seq":190,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.068961+0000","last_change":"2024-02-13T00:52:30.605200+0000","last_active":"2024-02-13T00:53:09.068961+0000","last_peered":"2024-02-13T00:53:09.068961+0000","last_clean":"2024-02-13T00:53:09.068961+0000","last_became_active":"2024-02-13T00:52:30.603757+0000","last_became_peered":"2024-02-13T00:52:30.603757+0000","last_unstale":"2024-02-13T00:53:09.068961+0000","last_undegraded":"2024-02-13T00:53:09.068961+0000","last_fullsized":"2024-02-13T00:53:09.068961+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"251.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-13T00:53:11.585339+0000","last_change":"2024-02-13T00:53:11.585339+0000","last_active":"2024-02-13T00:53:11.585339+0000","last_peered":"2024-02-13T00:53:11.585339+0000","last_clean":"2024-02-13T00:53:11.585339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-13T00:53:11.585339+0000","last_undegraded":"2024-02-13T00:53:11.585339+0000","last_fullsized":"2024-02-13T00:53:11.585339+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:11.585339+0000","last_clean_scrub_stamp":"2024-02-13T00:53:11.585339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.0","version":"740'5","reported_seq":198,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109334+0000","last_change":"2024-02-13T00:52:31.907499+0000","last_active":"2024-02-13T00:53:09.109334+0000","last_peered":"2024-02-13T00:53:09.109334+0000","last_clean":"2024-02-13T00:53:09.109334+0000","last_became_active":"2024-02-13T00:52:30.580474+0000","last_became_peered":"2024-02-13T00:52:30.580474+0000","last_unstale":"2024-02-13T00:53:09.109334+0000","last_undegraded":"2024-02-13T00:53:09.109334+0000","last_fullsized":"2024-02-13T00:53:09.109334+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":599,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:13.529162+0000","last_clean_scrub_stamp":"2024-02-13T00:50:13.529162+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":661,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.569739+0000","last_change":"2024-02-13T00:51:41.586442+0000","last_active":"2024-02-13T00:53:08.569739+0000","last_peered":"2024-02-13T00:53:08.569739+0000","last_clean":"2024-02-13T00:53:08.569739+0000","last_became_active":"2024-02-13T00:51:41.585448+0000","last_became_peered":"2024-02-13T00:51:41.585448+0000","last_unstale":"2024-02-13T00:53:08.569739+0000","last_undegraded":"2024-02-13T00:53:08.569739+0000","last_fullsized":"2024-02-13T00:53:08.569739+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:32.100607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:50:32.100607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":584,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109826+0000","last_change":"2024-02-13T00:51:41.586812+0000","last_active":"2024-02-13T00:53:09.109826+0000","last_peered":"2024-02-13T00:53:09.109826+0000","last_clean":"2024-02-13T00:53:09.109826+0000","last_became_active":"2024-02-13T00:51:41.584445+0000","last_became_peered":"2024-02-13T00:51:41.584445+0000","last_unstale":"2024-02-13T00:53:09.109826+0000","last_undegraded":"2024-02-13T00:53:09.109826+0000","last_fullsized":"2024-02-13T00:53:09.109826+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:39.963625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:39.963625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":463,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.068973+0000","last_change":"2024-02-13T00:52:30.605704+0000","last_active":"2024-02-13T00:53:09.068973+0000","last_peered":"2024-02-13T00:53:09.068973+0000","last_clean":"2024-02-13T00:53:09.068973+0000","last_became_active":"2024-02-13T00:52:30.604888+0000","last_became_peered":"2024-02-13T00:52:30.604888+0000","last_unstale":"2024-02-13T00:53:09.068973+0000","last_undegraded":"2024-02-13T00:53:09.068973+0000","last_fullsized":"2024-02-13T00:53:09.068973+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:15.530059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:15.530059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":539,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.068523+0000","last_change":"2024-02-13T00:51:41.591561+0000","last_active":"2024-02-13T00:53:09.068523+0000","last_peered":"2024-02-13T00:53:09.068523+0000","last_clean":"2024-02-13T00:53:09.068523+0000","last_became_active":"2024-02-13T00:51:41.588863+0000","last_became_peered":"2024-02-13T00:51:41.588863+0000","last_unstale":"2024-02-13T00:53:09.068523+0000","last_undegraded":"2024-02-13T00:53:09.068523+0000","last_fullsized":"2024-02-13T00:53:09.068523+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:16.677431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:16.677431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":440,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110448+0000","last_change":"2024-02-13T00:52:30.601175+0000","last_active":"2024-02-13T00:53:09.110448+0000","last_peered":"2024-02-13T00:53:09.110448+0000","last_clean":"2024-02-13T00:53:09.110448+0000","last_became_active":"2024-02-13T00:52:30.600551+0000","last_became_peered":"2024-02-13T00:52:30.600551+0000","last_unstale":"2024-02-13T00:53:09.110448+0000","last_undegraded":"2024-02-13T00:53:09.110448+0000","last_fullsized":"2024-02-13T00:53:09.110448+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.469431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.469431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":37,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109422+0000","last_change":"2024-02-13T00:52:30.571356+0000","last_active":"2024-02-13T00:53:09.109422+0000","last_peered":"2024-02-13T00:53:09.109422+0000","last_clean":"2024-02-13T00:53:09.109422+0000","last_became_active":"2024-02-13T00:52:30.571021+0000","last_became_peered":"2024-02-13T00:52:30.571021+0000","last_unstale":"2024-02-13T00:53:09.109422+0000","last_undegraded":"2024-02-13T00:53:09.109422+0000","last_fullsized":"2024-02-13T00:53:09.109422+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:54.542201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:54.542201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":449,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111974+0000","last_change":"2024-02-13T00:51:40.927523+0000","last_active":"2024-02-13T00:53:09.111974+0000","last_peered":"2024-02-13T00:53:09.111974+0000","last_clean":"2024-02-13T00:53:09.111974+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:53:09.111974+0000","last_undegraded":"2024-02-13T00:53:09.111974+0000","last_fullsized":"2024-02-13T00:53:09.111974+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:40.927454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:40.927454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":712,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110466+0000","last_change":"2024-02-13T00:51:41.571583+0000","last_active":"2024-02-13T00:53:09.110466+0000","last_peered":"2024-02-13T00:53:09.110466+0000","last_clean":"2024-02-13T00:53:09.110466+0000","last_became_active":"2024-02-13T00:51:41.570041+0000","last_became_peered":"2024-02-13T00:51:41.570041+0000","last_unstale":"2024-02-13T00:53:09.110466+0000","last_undegraded":"2024-02-13T00:53:09.110466+0000","last_fullsized":"2024-02-13T00:53:09.110466+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":431,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111975+0000","last_change":"2024-02-13T00:49:48.070260+0000","last_active":"2024-02-13T00:53:09.111975+0000","last_peered":"2024-02-13T00:53:09.111975+0000","last_clean":"2024-02-13T00:53:09.111975+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:53:09.111975+0000","last_undegraded":"2024-02-13T00:53:09.111975+0000","last_fullsized":"2024-02-13T00:53:09.111975+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:48.070195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:49:48.070195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":530,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070727+0000","last_change":"2024-02-13T00:51:41.569866+0000","last_active":"2024-02-13T00:53:09.070727+0000","last_peered":"2024-02-13T00:53:09.070727+0000","last_clean":"2024-02-13T00:53:09.070727+0000","last_became_active":"2024-02-13T00:51:41.567986+0000","last_became_peered":"2024-02-13T00:51:41.567986+0000","last_unstale":"2024-02-13T00:53:09.070727+0000","last_undegraded":"2024-02-13T00:53:09.070727+0000","last_fullsized":"2024-02-13T00:53:09.070727+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:14.521241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:14.521241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":680,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570333+0000","last_change":"2024-02-13T00:52:30.593903+0000","last_active":"2024-02-13T00:53:08.570333+0000","last_peered":"2024-02-13T00:53:08.570333+0000","last_clean":"2024-02-13T00:53:08.570333+0000","last_became_active":"2024-02-13T00:52:30.593515+0000","last_became_peered":"2024-02-13T00:52:30.593515+0000","last_unstale":"2024-02-13T00:53:08.570333+0000","last_undegraded":"2024-02-13T00:53:08.570333+0000","last_fullsized":"2024-02-13T00:53:08.570333+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:49.069014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:49.069014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"728'22","reported_seq":625,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.389879+0000","last_change":"2024-02-13T00:52:35.583266+0000","last_active":"2024-02-13T00:53:09.389879+0000","last_peered":"2024-02-13T00:53:09.389879+0000","last_clean":"2024-02-13T00:53:09.389879+0000","last_became_active":"2024-02-13T00:52:35.581849+0000","last_became_peered":"2024-02-13T00:52:35.581849+0000","last_unstale":"2024-02-13T00:53:09.389879+0000","last_undegraded":"2024-02-13T00:53:09.389879+0000","last_fullsized":"2024-02-13T00:53:09.389879+0000","mapping_epoch":727,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'14","last_scrub_stamp":"2024-02-13T00:45:53.115260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:45:53.115260+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":828,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109631+0000","last_change":"2024-02-13T00:52:32.169701+0000","last_active":"2024-02-13T00:53:09.109631+0000","last_peered":"2024-02-13T00:53:09.109631+0000","last_clean":"2024-02-13T00:53:09.109631+0000","last_became_active":"2024-02-13T00:52:30.577711+0000","last_became_peered":"2024-02-13T00:52:30.577711+0000","last_unstale":"2024-02-13T00:53:09.109631+0000","last_undegraded":"2024-02-13T00:53:09.109631+0000","last_fullsized":"2024-02-13T00:53:09.109631+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'36","last_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:47:03.491647+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":749,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111757+0000","last_change":"2024-02-13T00:51:41.571304+0000","last_active":"2024-02-13T00:53:09.111757+0000","last_peered":"2024-02-13T00:53:09.111757+0000","last_clean":"2024-02-13T00:53:09.111757+0000","last_became_active":"2024-02-13T00:51:41.569518+0000","last_became_peered":"2024-02-13T00:51:41.569518+0000","last_unstale":"2024-02-13T00:53:09.111757+0000","last_undegraded":"2024-02-13T00:53:09.111757+0000","last_fullsized":"2024-02-13T00:53:09.111757+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:23.379773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:23.379773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":646,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109776+0000","last_change":"2024-02-13T00:47:34.332791+0000","last_active":"2024-02-13T00:53:09.109776+0000","last_peered":"2024-02-13T00:53:09.109776+0000","last_clean":"2024-02-13T00:53:09.109776+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:53:09.109776+0000","last_undegraded":"2024-02-13T00:53:09.109776+0000","last_fullsized":"2024-02-13T00:53:09.109776+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:34.332704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:34.332704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":698,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110346+0000","last_change":"2024-02-13T00:52:30.190029+0000","last_active":"2024-02-13T00:53:09.110346+0000","last_peered":"2024-02-13T00:53:09.110346+0000","last_clean":"2024-02-13T00:53:09.110346+0000","last_became_active":"2024-02-13T00:51:41.573640+0000","last_became_peered":"2024-02-13T00:51:41.573640+0000","last_unstale":"2024-02-13T00:53:09.110346+0000","last_undegraded":"2024-02-13T00:53:09.110346+0000","last_fullsized":"2024-02-13T00:53:09.110346+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:30.189925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:30.189925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":771,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571006+0000","last_change":"2024-02-13T00:52:31.041126+0000","last_active":"2024-02-13T00:53:08.571006+0000","last_peered":"2024-02-13T00:53:08.571006+0000","last_clean":"2024-02-13T00:53:08.571006+0000","last_became_active":"2024-02-13T00:52:30.591236+0000","last_became_peered":"2024-02-13T00:52:30.591236+0000","last_unstale":"2024-02-13T00:53:08.571006+0000","last_undegraded":"2024-02-13T00:53:08.571006+0000","last_fullsized":"2024-02-13T00:53:08.571006+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'21","last_scrub_stamp":"2024-02-13T00:52:28.988459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:28.988459+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":740,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070783+0000","last_change":"2024-02-13T00:52:30.604523+0000","last_active":"2024-02-13T00:53:09.070783+0000","last_peered":"2024-02-13T00:53:09.070783+0000","last_clean":"2024-02-13T00:53:09.070783+0000","last_became_active":"2024-02-13T00:52:30.603634+0000","last_became_peered":"2024-02-13T00:52:30.603634+0000","last_unstale":"2024-02-13T00:53:09.070783+0000","last_undegraded":"2024-02-13T00:53:09.070783+0000","last_fullsized":"2024-02-13T00:53:09.070783+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:24.823107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.823107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":614,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571354+0000","last_change":"2024-02-13T00:52:30.587164+0000","last_active":"2024-02-13T00:53:08.571354+0000","last_peered":"2024-02-13T00:53:08.571354+0000","last_clean":"2024-02-13T00:53:08.571354+0000","last_became_active":"2024-02-13T00:52:30.586877+0000","last_became_peered":"2024-02-13T00:52:30.586877+0000","last_unstale":"2024-02-13T00:53:08.571354+0000","last_undegraded":"2024-02-13T00:53:08.571354+0000","last_fullsized":"2024-02-13T00:53:08.571354+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.788783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.788783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":737,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570202+0000","last_change":"2024-02-13T00:51:41.586644+0000","last_active":"2024-02-13T00:53:08.570202+0000","last_peered":"2024-02-13T00:53:08.570202+0000","last_clean":"2024-02-13T00:53:08.570202+0000","last_became_active":"2024-02-13T00:51:41.585885+0000","last_became_peered":"2024-02-13T00:51:41.585885+0000","last_unstale":"2024-02-13T00:53:08.570202+0000","last_undegraded":"2024-02-13T00:53:08.570202+0000","last_fullsized":"2024-02-13T00:53:08.570202+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.837810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.837810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":363,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570992+0000","last_change":"2024-02-13T00:52:30.588176+0000","last_active":"2024-02-13T00:53:08.570992+0000","last_peered":"2024-02-13T00:53:08.570992+0000","last_clean":"2024-02-13T00:53:08.570992+0000","last_became_active":"2024-02-13T00:52:30.587089+0000","last_became_peered":"2024-02-13T00:52:30.587089+0000","last_unstale":"2024-02-13T00:53:08.570992+0000","last_undegraded":"2024-02-13T00:53:08.570992+0000","last_fullsized":"2024-02-13T00:53:08.570992+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:34.184091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:34.184091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":570,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110170+0000","last_change":"2024-02-13T00:51:43.825729+0000","last_active":"2024-02-13T00:53:09.110170+0000","last_peered":"2024-02-13T00:53:09.110170+0000","last_clean":"2024-02-13T00:53:09.110170+0000","last_became_active":"2024-02-13T00:51:41.573050+0000","last_became_peered":"2024-02-13T00:51:41.573050+0000","last_unstale":"2024-02-13T00:53:09.110170+0000","last_undegraded":"2024-02-13T00:53:09.110170+0000","last_fullsized":"2024-02-13T00:53:09.110170+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'51","last_scrub_stamp":"2024-02-13T00:51:33.100282+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:51:33.100282+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":36,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.572317+0000","last_change":"2024-02-13T00:52:30.594905+0000","last_active":"2024-02-13T00:53:08.572317+0000","last_peered":"2024-02-13T00:53:08.572317+0000","last_clean":"2024-02-13T00:53:08.572317+0000","last_became_active":"2024-02-13T00:52:30.594543+0000","last_became_peered":"2024-02-13T00:52:30.594543+0000","last_unstale":"2024-02-13T00:53:08.572317+0000","last_undegraded":"2024-02-13T00:53:08.572317+0000","last_fullsized":"2024-02-13T00:53:08.572317+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:51.052139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:51:51.052139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":473,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570649+0000","last_change":"2024-02-13T00:51:41.587744+0000","last_active":"2024-02-13T00:53:08.570649+0000","last_peered":"2024-02-13T00:53:08.570649+0000","last_clean":"2024-02-13T00:53:08.570649+0000","last_became_active":"2024-02-13T00:51:41.582908+0000","last_became_peered":"2024-02-13T00:51:41.582908+0000","last_unstale":"2024-02-13T00:53:08.570649+0000","last_undegraded":"2024-02-13T00:53:08.570649+0000","last_fullsized":"2024-02-13T00:53:08.570649+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:14.244950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:51:14.244950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":576,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109725+0000","last_change":"2024-02-13T00:51:41.586157+0000","last_active":"2024-02-13T00:53:09.109725+0000","last_peered":"2024-02-13T00:53:09.109725+0000","last_clean":"2024-02-13T00:53:09.109725+0000","last_became_active":"2024-02-13T00:51:41.583542+0000","last_became_peered":"2024-02-13T00:51:41.583542+0000","last_unstale":"2024-02-13T00:53:09.109725+0000","last_undegraded":"2024-02-13T00:53:09.109725+0000","last_fullsized":"2024-02-13T00:53:09.109725+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:15.285536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:15.285536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":676,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070503+0000","last_change":"2024-02-13T00:51:35.505365+0000","last_active":"2024-02-13T00:53:09.070503+0000","last_peered":"2024-02-13T00:53:09.070503+0000","last_clean":"2024-02-13T00:53:09.070503+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:53:09.070503+0000","last_undegraded":"2024-02-13T00:53:09.070503+0000","last_fullsized":"2024-02-13T00:53:09.070503+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:35.505303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:35.505303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"467'31","reported_seq":535,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070597+0000","last_change":"2024-02-13T00:52:31.370996+0000","last_active":"2024-02-13T00:53:09.070597+0000","last_peered":"2024-02-13T00:53:09.070597+0000","last_clean":"2024-02-13T00:53:09.070597+0000","last_became_active":"2024-02-13T00:52:30.589869+0000","last_became_peered":"2024-02-13T00:52:30.589869+0000","last_unstale":"2024-02-13T00:53:09.070597+0000","last_undegraded":"2024-02-13T00:53:09.070597+0000","last_fullsized":"2024-02-13T00:53:09.070597+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'31","last_scrub_stamp":"2024-02-13T00:48:31.782729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.782729+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":548,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110098+0000","last_change":"2024-02-13T00:52:33.259627+0000","last_active":"2024-02-13T00:53:09.110098+0000","last_peered":"2024-02-13T00:53:09.110098+0000","last_clean":"2024-02-13T00:53:09.110098+0000","last_became_active":"2024-02-13T00:52:30.596010+0000","last_became_peered":"2024-02-13T00:52:30.596010+0000","last_unstale":"2024-02-13T00:53:09.110098+0000","last_undegraded":"2024-02-13T00:53:09.110098+0000","last_fullsized":"2024-02-13T00:53:09.110098+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:52:25.149943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:52:25.149943+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":738,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070468+0000","last_change":"2024-02-13T00:52:30.589321+0000","last_active":"2024-02-13T00:53:09.070468+0000","last_peered":"2024-02-13T00:53:09.070468+0000","last_clean":"2024-02-13T00:53:09.070468+0000","last_became_active":"2024-02-13T00:52:30.588539+0000","last_became_peered":"2024-02-13T00:52:30.588539+0000","last_unstale":"2024-02-13T00:53:09.070468+0000","last_undegraded":"2024-02-13T00:53:09.070468+0000","last_fullsized":"2024-02-13T00:53:09.070468+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.793634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":725,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111717+0000","last_change":"2024-02-13T00:52:30.600848+0000","last_active":"2024-02-13T00:53:09.111717+0000","last_peered":"2024-02-13T00:53:09.111717+0000","last_clean":"2024-02-13T00:53:09.111717+0000","last_became_active":"2024-02-13T00:52:30.600493+0000","last_became_peered":"2024-02-13T00:52:30.600493+0000","last_unstale":"2024-02-13T00:53:09.111717+0000","last_undegraded":"2024-02-13T00:53:09.111717+0000","last_fullsized":"2024-02-13T00:53:09.111717+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:38.996288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:38.996288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":389,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.572603+0000","last_change":"2024-02-13T00:52:30.597329+0000","last_active":"2024-02-13T00:53:08.572603+0000","last_peered":"2024-02-13T00:53:08.572603+0000","last_clean":"2024-02-13T00:53:08.572603+0000","last_became_active":"2024-02-13T00:52:30.595363+0000","last_became_peered":"2024-02-13T00:52:30.595363+0000","last_unstale":"2024-02-13T00:53:08.572603+0000","last_undegraded":"2024-02-13T00:53:08.572603+0000","last_fullsized":"2024-02-13T00:53:08.572603+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.925604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.925604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":479,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110655+0000","last_change":"2024-02-13T00:52:32.337869+0000","last_active":"2024-02-13T00:53:09.110655+0000","last_peered":"2024-02-13T00:53:09.110655+0000","last_clean":"2024-02-13T00:53:09.110655+0000","last_became_active":"2024-02-13T00:52:30.584756+0000","last_became_peered":"2024-02-13T00:52:30.584756+0000","last_unstale":"2024-02-13T00:53:09.110655+0000","last_undegraded":"2024-02-13T00:53:09.110655+0000","last_fullsized":"2024-02-13T00:53:09.110655+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:49:03.294499+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:49:03.294499+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":749,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069019+0000","last_change":"2024-02-13T00:51:41.573970+0000","last_active":"2024-02-13T00:53:09.069019+0000","last_peered":"2024-02-13T00:53:09.069019+0000","last_clean":"2024-02-13T00:53:09.069019+0000","last_became_active":"2024-02-13T00:51:41.569525+0000","last_became_peered":"2024-02-13T00:51:41.569525+0000","last_unstale":"2024-02-13T00:53:09.069019+0000","last_undegraded":"2024-02-13T00:53:09.069019+0000","last_fullsized":"2024-02-13T00:53:09.069019+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:59.111828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:59.111828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"726'19","reported_seq":545,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.389624+0000","last_change":"2024-02-13T00:52:33.561456+0000","last_active":"2024-02-13T00:53:09.389624+0000","last_peered":"2024-02-13T00:53:09.389624+0000","last_clean":"2024-02-13T00:53:09.389624+0000","last_became_active":"2024-02-13T00:52:33.561139+0000","last_became_peered":"2024-02-13T00:52:33.561139+0000","last_unstale":"2024-02-13T00:53:09.389624+0000","last_undegraded":"2024-02-13T00:53:09.389624+0000","last_fullsized":"2024-02-13T00:53:09.389624+0000","mapping_epoch":725,"log_start":"404'11","ondisk_log_start":"404'11","created":60,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'16","last_scrub_stamp":"2024-02-13T00:52:22.988904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:52:22.988904+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":738,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.071461+0000","last_change":"2024-02-13T00:52:31.529160+0000","last_active":"2024-02-13T00:53:09.071461+0000","last_peered":"2024-02-13T00:53:09.071461+0000","last_clean":"2024-02-13T00:53:09.071461+0000","last_became_active":"2024-02-13T00:52:30.602716+0000","last_became_peered":"2024-02-13T00:52:30.602716+0000","last_unstale":"2024-02-13T00:53:09.071461+0000","last_undegraded":"2024-02-13T00:53:09.071461+0000","last_fullsized":"2024-02-13T00:53:09.071461+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:51:05.644558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:05.644558+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":539,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070687+0000","last_change":"2024-02-13T00:52:30.605308+0000","last_active":"2024-02-13T00:53:09.070687+0000","last_peered":"2024-02-13T00:53:09.070687+0000","last_clean":"2024-02-13T00:53:09.070687+0000","last_became_active":"2024-02-13T00:52:30.604615+0000","last_became_peered":"2024-02-13T00:52:30.604615+0000","last_unstale":"2024-02-13T00:53:09.070687+0000","last_undegraded":"2024-02-13T00:53:09.070687+0000","last_fullsized":"2024-02-13T00:53:09.070687+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:26.830270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:26.830270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"467'27","reported_seq":879,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070701+0000","last_change":"2024-02-13T00:52:32.788019+0000","last_active":"2024-02-13T00:53:09.070701+0000","last_peered":"2024-02-13T00:53:09.070701+0000","last_clean":"2024-02-13T00:53:09.070701+0000","last_became_active":"2024-02-13T00:52:30.603369+0000","last_became_peered":"2024-02-13T00:52:30.603369+0000","last_unstale":"2024-02-13T00:53:09.070701+0000","last_undegraded":"2024-02-13T00:53:09.070701+0000","last_fullsized":"2024-02-13T00:53:09.070701+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:48:54.043088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.043088+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":785,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069263+0000","last_change":"2024-02-13T00:51:43.912749+0000","last_active":"2024-02-13T00:53:09.069263+0000","last_peered":"2024-02-13T00:53:09.069263+0000","last_clean":"2024-02-13T00:53:09.069263+0000","last_became_active":"2024-02-13T00:51:41.588865+0000","last_became_peered":"2024-02-13T00:51:41.588865+0000","last_unstale":"2024-02-13T00:53:09.069263+0000","last_undegraded":"2024-02-13T00:53:09.069263+0000","last_fullsized":"2024-02-13T00:53:09.069263+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:49:37.053398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.053398+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"727'19","reported_seq":555,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.389634+0000","last_change":"2024-02-13T00:53:08.072994+0000","last_active":"2024-02-13T00:53:09.389634+0000","last_peered":"2024-02-13T00:53:09.389634+0000","last_clean":"2024-02-13T00:53:09.389634+0000","last_became_active":"2024-02-13T00:52:34.566362+0000","last_became_peered":"2024-02-13T00:52:34.566362+0000","last_unstale":"2024-02-13T00:53:09.389634+0000","last_undegraded":"2024-02-13T00:53:09.389634+0000","last_fullsized":"2024-02-13T00:53:09.389634+0000","mapping_epoch":726,"log_start":"467'11","ondisk_log_start":"467'11","created":60,"last_epoch_clean":727,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'19","last_scrub_stamp":"2024-02-13T00:53:08.072916+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:53:08.072916+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":362,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070357+0000","last_change":"2024-02-13T00:50:19.734411+0000","last_active":"2024-02-13T00:53:09.070357+0000","last_peered":"2024-02-13T00:53:09.070357+0000","last_clean":"2024-02-13T00:53:09.070357+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:53:09.070357+0000","last_undegraded":"2024-02-13T00:53:09.070357+0000","last_fullsized":"2024-02-13T00:53:09.070357+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:19.734320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:50:19.734320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":671,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571912+0000","last_change":"2024-02-13T00:51:11.239012+0000","last_active":"2024-02-13T00:53:08.571912+0000","last_peered":"2024-02-13T00:53:08.571912+0000","last_clean":"2024-02-13T00:53:08.571912+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:53:08.571912+0000","last_undegraded":"2024-02-13T00:53:08.571912+0000","last_fullsized":"2024-02-13T00:53:08.571912+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.238914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.238914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":480,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111887+0000","last_change":"2024-02-13T00:52:32.853997+0000","last_active":"2024-02-13T00:53:09.111887+0000","last_peered":"2024-02-13T00:53:09.111887+0000","last_clean":"2024-02-13T00:53:09.111887+0000","last_became_active":"2024-02-13T00:52:30.593631+0000","last_became_peered":"2024-02-13T00:52:30.593631+0000","last_unstale":"2024-02-13T00:53:09.111887+0000","last_undegraded":"2024-02-13T00:53:09.111887+0000","last_fullsized":"2024-02-13T00:53:09.111887+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:57.386254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:57.386254+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":563,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570081+0000","last_change":"2024-02-13T00:51:53.090015+0000","last_active":"2024-02-13T00:53:08.570081+0000","last_peered":"2024-02-13T00:53:08.570081+0000","last_clean":"2024-02-13T00:53:08.570081+0000","last_became_active":"2024-02-13T00:51:41.582606+0000","last_became_peered":"2024-02-13T00:51:41.582606+0000","last_unstale":"2024-02-13T00:53:08.570081+0000","last_undegraded":"2024-02-13T00:53:08.570081+0000","last_fullsized":"2024-02-13T00:53:08.570081+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:53.089946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:51:53.089946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"728'19","reported_seq":752,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.389953+0000","last_change":"2024-02-13T00:52:35.610840+0000","last_active":"2024-02-13T00:53:09.389953+0000","last_peered":"2024-02-13T00:53:09.389953+0000","last_clean":"2024-02-13T00:53:09.389953+0000","last_became_active":"2024-02-13T00:52:35.610249+0000","last_became_peered":"2024-02-13T00:52:35.610249+0000","last_unstale":"2024-02-13T00:53:09.389953+0000","last_undegraded":"2024-02-13T00:53:09.389953+0000","last_fullsized":"2024-02-13T00:53:09.389953+0000","mapping_epoch":727,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'16","last_scrub_stamp":"2024-02-13T00:49:45.051539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:49:45.051539+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":458,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110482+0000","last_change":"2024-02-13T00:51:41.570899+0000","last_active":"2024-02-13T00:53:09.110482+0000","last_peered":"2024-02-13T00:53:09.110482+0000","last_clean":"2024-02-13T00:53:09.110482+0000","last_became_active":"2024-02-13T00:51:41.568351+0000","last_became_peered":"2024-02-13T00:51:41.568351+0000","last_unstale":"2024-02-13T00:53:09.110482+0000","last_undegraded":"2024-02-13T00:53:09.110482+0000","last_fullsized":"2024-02-13T00:53:09.110482+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:31.463442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.463442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":641,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.569450+0000","last_change":"2024-02-13T00:52:30.595757+0000","last_active":"2024-02-13T00:53:08.569450+0000","last_peered":"2024-02-13T00:53:08.569450+0000","last_clean":"2024-02-13T00:53:08.569450+0000","last_became_active":"2024-02-13T00:52:30.594189+0000","last_became_peered":"2024-02-13T00:52:30.594189+0000","last_unstale":"2024-02-13T00:53:08.569450+0000","last_undegraded":"2024-02-13T00:53:08.569450+0000","last_fullsized":"2024-02-13T00:53:08.569450+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:00.881431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:00.881431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"726'11","reported_seq":797,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.389667+0000","last_change":"2024-02-13T00:52:31.384662+0000","last_active":"2024-02-13T00:53:09.389667+0000","last_peered":"2024-02-13T00:53:09.389667+0000","last_clean":"2024-02-13T00:53:09.389667+0000","last_became_active":"2024-02-13T00:52:30.582279+0000","last_became_peered":"2024-02-13T00:52:30.582279+0000","last_unstale":"2024-02-13T00:53:09.389667+0000","last_undegraded":"2024-02-13T00:53:09.389667+0000","last_fullsized":"2024-02-13T00:53:09.389667+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:47:02.949406+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":748,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109464+0000","last_change":"2024-02-13T00:52:31.298166+0000","last_active":"2024-02-13T00:53:09.109464+0000","last_peered":"2024-02-13T00:53:09.109464+0000","last_clean":"2024-02-13T00:53:09.109464+0000","last_became_active":"2024-02-13T00:52:30.579027+0000","last_became_peered":"2024-02-13T00:52:30.579027+0000","last_unstale":"2024-02-13T00:53:09.109464+0000","last_undegraded":"2024-02-13T00:53:09.109464+0000","last_fullsized":"2024-02-13T00:53:09.109464+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'34","last_scrub_stamp":"2024-02-13T00:49:46.101874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.101874+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":458,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111896+0000","last_change":"2024-02-13T00:51:41.572212+0000","last_active":"2024-02-13T00:53:09.111896+0000","last_peered":"2024-02-13T00:53:09.111896+0000","last_clean":"2024-02-13T00:53:09.111896+0000","last_became_active":"2024-02-13T00:51:41.569996+0000","last_became_peered":"2024-02-13T00:51:41.569996+0000","last_unstale":"2024-02-13T00:53:09.111896+0000","last_undegraded":"2024-02-13T00:53:09.111896+0000","last_fullsized":"2024-02-13T00:53:09.111896+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:47.075741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.075741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":458,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110456+0000","last_change":"2024-02-13T00:51:41.572697+0000","last_active":"2024-02-13T00:53:09.110456+0000","last_peered":"2024-02-13T00:53:09.110456+0000","last_clean":"2024-02-13T00:53:09.110456+0000","last_became_active":"2024-02-13T00:51:41.569841+0000","last_became_peered":"2024-02-13T00:51:41.569841+0000","last_unstale":"2024-02-13T00:53:09.110456+0000","last_undegraded":"2024-02-13T00:53:09.110456+0000","last_fullsized":"2024-02-13T00:53:09.110456+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:17.268189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:17.268189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":482,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.569925+0000","last_change":"2024-02-13T00:51:41.585096+0000","last_active":"2024-02-13T00:53:08.569925+0000","last_peered":"2024-02-13T00:53:08.569925+0000","last_clean":"2024-02-13T00:53:08.569925+0000","last_became_active":"2024-02-13T00:51:41.582942+0000","last_became_peered":"2024-02-13T00:51:41.582942+0000","last_unstale":"2024-02-13T00:53:08.569925+0000","last_undegraded":"2024-02-13T00:53:08.569925+0000","last_fullsized":"2024-02-13T00:53:08.569925+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:36.841508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:36.841508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":532,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.569658+0000","last_change":"2024-02-13T00:52:30.595853+0000","last_active":"2024-02-13T00:53:08.569658+0000","last_peered":"2024-02-13T00:53:08.569658+0000","last_clean":"2024-02-13T00:53:08.569658+0000","last_became_active":"2024-02-13T00:52:30.594613+0000","last_became_peered":"2024-02-13T00:52:30.594613+0000","last_unstale":"2024-02-13T00:53:08.569658+0000","last_undegraded":"2024-02-13T00:53:08.569658+0000","last_fullsized":"2024-02-13T00:53:08.569658+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:18.751081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:18.751081+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"467'42","reported_seq":706,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570010+0000","last_change":"2024-02-13T00:51:42.397324+0000","last_active":"2024-02-13T00:53:08.570010+0000","last_peered":"2024-02-13T00:53:08.570010+0000","last_clean":"2024-02-13T00:53:08.570010+0000","last_became_active":"2024-02-13T00:51:41.582068+0000","last_became_peered":"2024-02-13T00:51:41.582068+0000","last_unstale":"2024-02-13T00:53:08.570010+0000","last_undegraded":"2024-02-13T00:53:08.570010+0000","last_fullsized":"2024-02-13T00:53:08.570010+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'42","last_scrub_stamp":"2024-02-13T00:48:24.675687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:24.675687+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":754,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.071364+0000","last_change":"2024-02-13T00:52:30.582723+0000","last_active":"2024-02-13T00:53:09.071364+0000","last_peered":"2024-02-13T00:53:09.071364+0000","last_clean":"2024-02-13T00:53:09.071364+0000","last_became_active":"2024-02-13T00:52:30.582296+0000","last_became_peered":"2024-02-13T00:52:30.582296+0000","last_unstale":"2024-02-13T00:53:09.071364+0000","last_undegraded":"2024-02-13T00:53:09.071364+0000","last_fullsized":"2024-02-13T00:53:09.071364+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:29.412336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:52:29.412336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":697,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109784+0000","last_change":"2024-02-13T00:52:30.595054+0000","last_active":"2024-02-13T00:53:09.109784+0000","last_peered":"2024-02-13T00:53:09.109784+0000","last_clean":"2024-02-13T00:53:09.109784+0000","last_became_active":"2024-02-13T00:52:30.593405+0000","last_became_peered":"2024-02-13T00:52:30.593405+0000","last_unstale":"2024-02-13T00:53:09.109784+0000","last_undegraded":"2024-02-13T00:53:09.109784+0000","last_fullsized":"2024-02-13T00:53:09.109784+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":680,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110247+0000","last_change":"2024-02-13T00:52:30.586680+0000","last_active":"2024-02-13T00:53:09.110247+0000","last_peered":"2024-02-13T00:53:09.110247+0000","last_clean":"2024-02-13T00:53:09.110247+0000","last_became_active":"2024-02-13T00:52:30.586041+0000","last_became_peered":"2024-02-13T00:52:30.586041+0000","last_unstale":"2024-02-13T00:53:09.110247+0000","last_undegraded":"2024-02-13T00:53:09.110247+0000","last_fullsized":"2024-02-13T00:53:09.110247+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_clean_scrub_stamp":"2024-02-13T00:48:35.434645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":454,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.345020+0000","last_change":"2024-02-13T00:53:09.344998+0000","last_active":"2024-02-13T00:53:09.345020+0000","last_peered":"2024-02-13T00:53:09.345020+0000","last_clean":"2024-02-13T00:53:09.345020+0000","last_became_active":"2024-02-13T00:51:41.573763+0000","last_became_peered":"2024-02-13T00:51:41.573763+0000","last_unstale":"2024-02-13T00:53:09.345020+0000","last_undegraded":"2024-02-13T00:53:09.345020+0000","last_fullsized":"2024-02-13T00:53:09.345020+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:09.344923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:53:09.344923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":872,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570792+0000","last_change":"2024-02-13T00:52:26.954476+0000","last_active":"2024-02-13T00:53:08.570792+0000","last_peered":"2024-02-13T00:53:08.570792+0000","last_clean":"2024-02-13T00:53:08.570792+0000","last_became_active":"2024-02-13T00:51:41.587256+0000","last_became_peered":"2024-02-13T00:51:41.587256+0000","last_unstale":"2024-02-13T00:53:08.570792+0000","last_undegraded":"2024-02-13T00:53:08.570792+0000","last_fullsized":"2024-02-13T00:53:08.570792+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'44","last_scrub_stamp":"2024-02-13T00:52:26.954384+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:52:26.954384+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":572,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570963+0000","last_change":"2024-02-13T00:52:30.596233+0000","last_active":"2024-02-13T00:53:08.570963+0000","last_peered":"2024-02-13T00:53:08.570963+0000","last_clean":"2024-02-13T00:53:08.570963+0000","last_became_active":"2024-02-13T00:52:30.595886+0000","last_became_peered":"2024-02-13T00:52:30.595886+0000","last_unstale":"2024-02-13T00:53:08.570963+0000","last_undegraded":"2024-02-13T00:53:08.570963+0000","last_fullsized":"2024-02-13T00:53:08.570963+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:25.001219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:25.001219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":761,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069271+0000","last_change":"2024-02-13T00:51:41.589718+0000","last_active":"2024-02-13T00:53:09.069271+0000","last_peered":"2024-02-13T00:53:09.069271+0000","last_clean":"2024-02-13T00:53:09.069271+0000","last_became_active":"2024-02-13T00:51:41.589295+0000","last_became_peered":"2024-02-13T00:51:41.589295+0000","last_unstale":"2024-02-13T00:53:09.069271+0000","last_undegraded":"2024-02-13T00:53:09.069271+0000","last_fullsized":"2024-02-13T00:53:09.069271+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.726660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.726660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":374,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070610+0000","last_change":"2024-02-13T00:52:30.582714+0000","last_active":"2024-02-13T00:53:09.070610+0000","last_peered":"2024-02-13T00:53:09.070610+0000","last_clean":"2024-02-13T00:53:09.070610+0000","last_became_active":"2024-02-13T00:52:30.582272+0000","last_became_peered":"2024-02-13T00:52:30.582272+0000","last_unstale":"2024-02-13T00:53:09.070610+0000","last_undegraded":"2024-02-13T00:53:09.070610+0000","last_fullsized":"2024-02-13T00:53:09.070610+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:55.050411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.050411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":389,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.071459+0000","last_change":"2024-02-13T00:52:30.605419+0000","last_active":"2024-02-13T00:53:09.071459+0000","last_peered":"2024-02-13T00:53:09.071459+0000","last_clean":"2024-02-13T00:53:09.071459+0000","last_became_active":"2024-02-13T00:52:30.604937+0000","last_became_peered":"2024-02-13T00:52:30.604937+0000","last_unstale":"2024-02-13T00:53:09.071459+0000","last_undegraded":"2024-02-13T00:53:09.071459+0000","last_fullsized":"2024-02-13T00:53:09.071459+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:43.903553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:43.903553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":861,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571453+0000","last_change":"2024-02-13T00:52:23.951273+0000","last_active":"2024-02-13T00:53:08.571453+0000","last_peered":"2024-02-13T00:53:08.571453+0000","last_clean":"2024-02-13T00:53:08.571453+0000","last_became_active":"2024-02-13T00:51:41.588905+0000","last_became_peered":"2024-02-13T00:51:41.588905+0000","last_unstale":"2024-02-13T00:53:08.571453+0000","last_undegraded":"2024-02-13T00:53:08.571453+0000","last_fullsized":"2024-02-13T00:53:08.571453+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:52:23.951206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:23.951206+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.068912+0000","last_change":"2024-02-13T00:51:53.574669+0000","last_active":"2024-02-13T00:53:09.068912+0000","last_peered":"2024-02-13T00:53:09.068912+0000","last_clean":"2024-02-13T00:53:09.068912+0000","last_became_active":"2024-02-13T00:51:41.588531+0000","last_became_peered":"2024-02-13T00:51:41.588531+0000","last_unstale":"2024-02-13T00:53:09.068912+0000","last_undegraded":"2024-02-13T00:53:09.068912+0000","last_fullsized":"2024-02-13T00:53:09.068912+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:53.574605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:53.574605+0000","last_clean_scrub_stamp":"2024-02-13T00:51:53.574605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":748,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.569849+0000","last_change":"2024-02-13T00:52:30.596032+0000","last_active":"2024-02-13T00:53:08.569849+0000","last_peered":"2024-02-13T00:53:08.569849+0000","last_clean":"2024-02-13T00:53:08.569849+0000","last_became_active":"2024-02-13T00:52:30.595531+0000","last_became_peered":"2024-02-13T00:52:30.595531+0000","last_unstale":"2024-02-13T00:53:08.569849+0000","last_undegraded":"2024-02-13T00:53:08.569849+0000","last_fullsized":"2024-02-13T00:53:08.569849+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:46.119431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:51:46.119431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":752,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070479+0000","last_change":"2024-02-13T00:52:30.589500+0000","last_active":"2024-02-13T00:53:09.070479+0000","last_peered":"2024-02-13T00:53:09.070479+0000","last_clean":"2024-02-13T00:53:09.070479+0000","last_became_active":"2024-02-13T00:52:30.588991+0000","last_became_peered":"2024-02-13T00:52:30.588991+0000","last_unstale":"2024-02-13T00:53:09.070479+0000","last_undegraded":"2024-02-13T00:53:09.070479+0000","last_fullsized":"2024-02-13T00:53:09.070479+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:16.535961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:51:16.535961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":600,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.112089+0000","last_change":"2024-02-13T00:52:30.603205+0000","last_active":"2024-02-13T00:53:09.112089+0000","last_peered":"2024-02-13T00:53:09.112089+0000","last_clean":"2024-02-13T00:53:09.112089+0000","last_became_active":"2024-02-13T00:52:30.600547+0000","last_became_peered":"2024-02-13T00:52:30.600547+0000","last_unstale":"2024-02-13T00:53:09.112089+0000","last_undegraded":"2024-02-13T00:53:09.112089+0000","last_fullsized":"2024-02-13T00:53:09.112089+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:23.124083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:23.124083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":404,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.572517+0000","last_change":"2024-02-13T00:52:30.017699+0000","last_active":"2024-02-13T00:53:08.572517+0000","last_peered":"2024-02-13T00:53:08.572517+0000","last_clean":"2024-02-13T00:53:08.572517+0000","last_became_active":"2024-02-13T00:51:41.580760+0000","last_became_peered":"2024-02-13T00:51:41.580760+0000","last_unstale":"2024-02-13T00:53:08.572517+0000","last_undegraded":"2024-02-13T00:53:08.572517+0000","last_fullsized":"2024-02-13T00:53:08.572517+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:30.017618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:52:30.017618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":540,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.112075+0000","last_change":"2024-02-13T00:52:34.694390+0000","last_active":"2024-02-13T00:53:09.112075+0000","last_peered":"2024-02-13T00:53:09.112075+0000","last_clean":"2024-02-13T00:53:09.112075+0000","last_became_active":"2024-02-13T00:52:30.596789+0000","last_became_peered":"2024-02-13T00:52:30.596789+0000","last_unstale":"2024-02-13T00:53:09.112075+0000","last_undegraded":"2024-02-13T00:53:09.112075+0000","last_fullsized":"2024-02-13T00:53:09.112075+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":727,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'21","last_scrub_stamp":"2024-02-13T00:46:57.381540+0000","last_deep_scrub":"157'21","last_deep_scrub_stamp":"2024-02-13T00:41:52.415907+0000","last_clean_scrub_stamp":"2024-02-13T00:46:57.381540+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":704,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.569793+0000","last_change":"2024-02-13T00:52:36.042288+0000","last_active":"2024-02-13T00:53:08.569793+0000","last_peered":"2024-02-13T00:53:08.569793+0000","last_clean":"2024-02-13T00:53:08.569793+0000","last_became_active":"2024-02-13T00:51:41.582126+0000","last_became_peered":"2024-02-13T00:51:41.582126+0000","last_unstale":"2024-02-13T00:53:08.569793+0000","last_undegraded":"2024-02-13T00:53:08.569793+0000","last_fullsized":"2024-02-13T00:53:08.569793+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.042221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.042221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":697,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110260+0000","last_change":"2024-02-13T00:51:41.574139+0000","last_active":"2024-02-13T00:53:09.110260+0000","last_peered":"2024-02-13T00:53:09.110260+0000","last_clean":"2024-02-13T00:53:09.110260+0000","last_became_active":"2024-02-13T00:51:41.571758+0000","last_became_peered":"2024-02-13T00:51:41.571758+0000","last_unstale":"2024-02-13T00:53:09.110260+0000","last_undegraded":"2024-02-13T00:53:09.110260+0000","last_fullsized":"2024-02-13T00:53:09.110260+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:04.493841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:47:04.493841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":570,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109666+0000","last_change":"2024-02-13T00:52:30.580452+0000","last_active":"2024-02-13T00:53:09.109666+0000","last_peered":"2024-02-13T00:53:09.109666+0000","last_clean":"2024-02-13T00:53:09.109666+0000","last_became_active":"2024-02-13T00:52:30.580091+0000","last_became_peered":"2024-02-13T00:52:30.580091+0000","last_unstale":"2024-02-13T00:53:09.109666+0000","last_undegraded":"2024-02-13T00:53:09.109666+0000","last_fullsized":"2024-02-13T00:53:09.109666+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:25.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:48:25.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":609,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070607+0000","last_change":"2024-02-13T00:52:30.604915+0000","last_active":"2024-02-13T00:53:09.070607+0000","last_peered":"2024-02-13T00:53:09.070607+0000","last_clean":"2024-02-13T00:53:09.070607+0000","last_became_active":"2024-02-13T00:52:30.603107+0000","last_became_peered":"2024-02-13T00:52:30.603107+0000","last_unstale":"2024-02-13T00:53:09.070607+0000","last_undegraded":"2024-02-13T00:53:09.070607+0000","last_fullsized":"2024-02-13T00:53:09.070607+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:19.521319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:51:19.521319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":689,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.572473+0000","last_change":"2024-02-13T00:52:30.596982+0000","last_active":"2024-02-13T00:53:08.572473+0000","last_peered":"2024-02-13T00:53:08.572473+0000","last_clean":"2024-02-13T00:53:08.572473+0000","last_became_active":"2024-02-13T00:52:30.596402+0000","last_became_peered":"2024-02-13T00:52:30.596402+0000","last_unstale":"2024-02-13T00:53:08.572473+0000","last_undegraded":"2024-02-13T00:53:08.572473+0000","last_fullsized":"2024-02-13T00:53:08.572473+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:42.854585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:42.854585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":435,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070484+0000","last_change":"2024-02-13T00:52:30.581391+0000","last_active":"2024-02-13T00:53:09.070484+0000","last_peered":"2024-02-13T00:53:09.070484+0000","last_clean":"2024-02-13T00:53:09.070484+0000","last_became_active":"2024-02-13T00:52:30.581048+0000","last_became_peered":"2024-02-13T00:52:30.581048+0000","last_unstale":"2024-02-13T00:53:09.070484+0000","last_undegraded":"2024-02-13T00:53:09.070484+0000","last_fullsized":"2024-02-13T00:53:09.070484+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:53.013103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.013103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"467'23","reported_seq":805,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070745+0000","last_change":"2024-02-13T00:51:42.290298+0000","last_active":"2024-02-13T00:53:09.070745+0000","last_peered":"2024-02-13T00:53:09.070745+0000","last_clean":"2024-02-13T00:53:09.070745+0000","last_became_active":"2024-02-13T00:51:41.569273+0000","last_became_peered":"2024-02-13T00:51:41.569273+0000","last_unstale":"2024-02-13T00:53:09.070745+0000","last_undegraded":"2024-02-13T00:53:09.070745+0000","last_fullsized":"2024-02-13T00:53:09.070745+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'23","last_scrub_stamp":"2024-02-13T00:49:07.192902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:07.192902+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":645,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570338+0000","last_change":"2024-02-13T00:49:46.851558+0000","last_active":"2024-02-13T00:53:08.570338+0000","last_peered":"2024-02-13T00:53:08.570338+0000","last_clean":"2024-02-13T00:53:08.570338+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:53:08.570338+0000","last_undegraded":"2024-02-13T00:53:08.570338+0000","last_fullsized":"2024-02-13T00:53:08.570338+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.851486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.851486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":1682,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.568781+0000","last_change":"2024-02-13T00:51:38.527164+0000","last_active":"2024-02-13T00:53:09.568781+0000","last_peered":"2024-02-13T00:53:09.568781+0000","last_clean":"2024-02-13T00:53:09.568781+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:53:09.568781+0000","last_undegraded":"2024-02-13T00:53:09.568781+0000","last_fullsized":"2024-02-13T00:53:09.568781+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:51:38.527084+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:51:38.527084+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":916,"num_read_kb":911,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":462,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.112008+0000","last_change":"2024-02-13T00:51:41.571352+0000","last_active":"2024-02-13T00:53:09.112008+0000","last_peered":"2024-02-13T00:53:09.112008+0000","last_clean":"2024-02-13T00:53:09.112008+0000","last_became_active":"2024-02-13T00:51:41.570686+0000","last_became_peered":"2024-02-13T00:51:41.570686+0000","last_unstale":"2024-02-13T00:53:09.112008+0000","last_undegraded":"2024-02-13T00:53:09.112008+0000","last_fullsized":"2024-02-13T00:53:09.112008+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":534,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070207+0000","last_change":"2024-02-13T00:52:30.604046+0000","last_active":"2024-02-13T00:53:09.070207+0000","last_peered":"2024-02-13T00:53:09.070207+0000","last_clean":"2024-02-13T00:53:09.070207+0000","last_became_active":"2024-02-13T00:52:30.603400+0000","last_became_peered":"2024-02-13T00:52:30.603400+0000","last_unstale":"2024-02-13T00:53:09.070207+0000","last_undegraded":"2024-02-13T00:53:09.070207+0000","last_fullsized":"2024-02-13T00:53:09.070207+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:37.545088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:37.545088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570259+0000","last_change":"2024-02-13T00:52:30.575376+0000","last_active":"2024-02-13T00:53:08.570259+0000","last_peered":"2024-02-13T00:53:08.570259+0000","last_clean":"2024-02-13T00:53:08.570259+0000","last_became_active":"2024-02-13T00:52:30.575153+0000","last_became_peered":"2024-02-13T00:52:30.575153+0000","last_unstale":"2024-02-13T00:53:08.570259+0000","last_undegraded":"2024-02-13T00:53:08.570259+0000","last_fullsized":"2024-02-13T00:53:08.570259+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:26.113119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:52:26.113119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"701'67","reported_seq":97,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.389630+0000","last_change":"2024-02-13T00:52:09.459680+0000","last_active":"2024-02-13T00:53:09.389630+0000","last_peered":"2024-02-13T00:53:09.389630+0000","last_clean":"2024-02-13T00:53:09.389630+0000","last_became_active":"2024-02-13T00:52:07.565910+0000","last_became_peered":"2024-02-13T00:52:07.565910+0000","last_unstale":"2024-02-13T00:53:09.389630+0000","last_undegraded":"2024-02-13T00:53:09.389630+0000","last_fullsized":"2024-02-13T00:53:09.389630+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'65","last_scrub_stamp":"2024-02-13T00:49:47.928548+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.928548+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":77,"num_read_kb":61,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":2864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":661,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.068909+0000","last_change":"2024-02-13T00:50:37.777963+0000","last_active":"2024-02-13T00:53:09.068909+0000","last_peered":"2024-02-13T00:53:09.068909+0000","last_clean":"2024-02-13T00:53:09.068909+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:53:09.068909+0000","last_undegraded":"2024-02-13T00:53:09.068909+0000","last_fullsized":"2024-02-13T00:53:09.068909+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:37.777891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:37.777891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":288,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571069+0000","last_change":"2024-02-13T00:52:30.598078+0000","last_active":"2024-02-13T00:53:08.571069+0000","last_peered":"2024-02-13T00:53:08.571069+0000","last_clean":"2024-02-13T00:53:08.571069+0000","last_became_active":"2024-02-13T00:52:30.595326+0000","last_became_peered":"2024-02-13T00:52:30.595326+0000","last_unstale":"2024-02-13T00:53:08.571069+0000","last_undegraded":"2024-02-13T00:53:08.571069+0000","last_fullsized":"2024-02-13T00:53:08.571069+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":440,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110224+0000","last_change":"2024-02-13T00:48:17.340891+0000","last_active":"2024-02-13T00:53:09.110224+0000","last_peered":"2024-02-13T00:53:09.110224+0000","last_clean":"2024-02-13T00:53:09.110224+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:53:09.110224+0000","last_undegraded":"2024-02-13T00:53:09.110224+0000","last_fullsized":"2024-02-13T00:53:09.110224+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:17.340797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:17.340797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":951,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.389872+0000","last_change":"2024-02-13T00:48:32.669477+0000","last_active":"2024-02-13T00:53:09.389872+0000","last_peered":"2024-02-13T00:53:09.389872+0000","last_clean":"2024-02-13T00:53:09.389872+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:53:09.389872+0000","last_undegraded":"2024-02-13T00:53:09.389872+0000","last_fullsized":"2024-02-13T00:53:09.389872+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14","last_scrub_stamp":"2024-02-13T00:48:32.669406+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.669406+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":502,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069315+0000","last_change":"2024-02-13T00:52:33.373486+0000","last_active":"2024-02-13T00:53:09.069315+0000","last_peered":"2024-02-13T00:53:09.069315+0000","last_clean":"2024-02-13T00:53:09.069315+0000","last_became_active":"2024-02-13T00:52:30.605408+0000","last_became_peered":"2024-02-13T00:52:30.605408+0000","last_unstale":"2024-02-13T00:53:09.069315+0000","last_undegraded":"2024-02-13T00:53:09.069315+0000","last_fullsized":"2024-02-13T00:53:09.069315+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:48:20.928158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:20.928158+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":736,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070287+0000","last_change":"2024-02-13T00:51:41.570719+0000","last_active":"2024-02-13T00:53:09.070287+0000","last_peered":"2024-02-13T00:53:09.070287+0000","last_clean":"2024-02-13T00:53:09.070287+0000","last_became_active":"2024-02-13T00:51:41.567277+0000","last_became_peered":"2024-02-13T00:51:41.567277+0000","last_unstale":"2024-02-13T00:53:09.070287+0000","last_undegraded":"2024-02-13T00:53:09.070287+0000","last_fullsized":"2024-02-13T00:53:09.070287+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:33.040516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:49:33.040516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":746,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.569717+0000","last_change":"2024-02-13T00:52:30.572639+0000","last_active":"2024-02-13T00:53:08.569717+0000","last_peered":"2024-02-13T00:53:08.569717+0000","last_clean":"2024-02-13T00:53:08.569717+0000","last_became_active":"2024-02-13T00:52:30.572144+0000","last_became_peered":"2024-02-13T00:52:30.572144+0000","last_unstale":"2024-02-13T00:53:08.569717+0000","last_undegraded":"2024-02-13T00:53:08.569717+0000","last_fullsized":"2024-02-13T00:53:08.569717+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:38.051468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:38.051468+0000","last_clean_scrub_stamp":"2024-02-13T00:50:38.051468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":685,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069158+0000","last_change":"2024-02-13T00:51:41.589341+0000","last_active":"2024-02-13T00:53:09.069158+0000","last_peered":"2024-02-13T00:53:09.069158+0000","last_clean":"2024-02-13T00:53:09.069158+0000","last_became_active":"2024-02-13T00:51:41.588483+0000","last_became_peered":"2024-02-13T00:51:41.588483+0000","last_unstale":"2024-02-13T00:53:09.069158+0000","last_undegraded":"2024-02-13T00:53:09.069158+0000","last_fullsized":"2024-02-13T00:53:09.069158+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:08.606262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:51:08.606262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":464,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570258+0000","last_change":"2024-02-13T00:52:30.572465+0000","last_active":"2024-02-13T00:53:08.570258+0000","last_peered":"2024-02-13T00:53:08.570258+0000","last_clean":"2024-02-13T00:53:08.570258+0000","last_became_active":"2024-02-13T00:52:30.571718+0000","last_became_peered":"2024-02-13T00:52:30.571718+0000","last_unstale":"2024-02-13T00:53:08.570258+0000","last_undegraded":"2024-02-13T00:53:08.570258+0000","last_fullsized":"2024-02-13T00:53:08.570258+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:48.089417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:51:48.089417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":822,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069155+0000","last_change":"2024-02-13T00:52:32.017951+0000","last_active":"2024-02-13T00:53:09.069155+0000","last_peered":"2024-02-13T00:53:09.069155+0000","last_clean":"2024-02-13T00:53:09.069155+0000","last_became_active":"2024-02-13T00:52:30.581394+0000","last_became_peered":"2024-02-13T00:52:30.581394+0000","last_unstale":"2024-02-13T00:53:09.069155+0000","last_undegraded":"2024-02-13T00:53:09.069155+0000","last_fullsized":"2024-02-13T00:53:09.069155+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'30","last_scrub_stamp":"2024-02-13T00:49:31.072862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:31.072862+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":756,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.572389+0000","last_change":"2024-02-13T00:52:31.527837+0000","last_active":"2024-02-13T00:53:08.572389+0000","last_peered":"2024-02-13T00:53:08.572389+0000","last_clean":"2024-02-13T00:53:08.572389+0000","last_became_active":"2024-02-13T00:52:30.595453+0000","last_became_peered":"2024-02-13T00:52:30.595453+0000","last_unstale":"2024-02-13T00:53:08.572389+0000","last_undegraded":"2024-02-13T00:53:08.572389+0000","last_fullsized":"2024-02-13T00:53:08.572389+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:48:58.861325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:58.861325+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":538,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.190113+0000","last_change":"2024-02-13T00:53:09.190093+0000","last_active":"2024-02-13T00:53:09.190113+0000","last_peered":"2024-02-13T00:53:09.190113+0000","last_clean":"2024-02-13T00:53:09.190113+0000","last_became_active":"2024-02-13T00:51:41.568308+0000","last_became_peered":"2024-02-13T00:51:41.568308+0000","last_unstale":"2024-02-13T00:53:09.190113+0000","last_undegraded":"2024-02-13T00:53:09.190113+0000","last_fullsized":"2024-02-13T00:53:09.190113+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:09.190024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:53:09.190024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":689,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571622+0000","last_change":"2024-02-13T00:52:30.598253+0000","last_active":"2024-02-13T00:53:08.571622+0000","last_peered":"2024-02-13T00:53:08.571622+0000","last_clean":"2024-02-13T00:53:08.571622+0000","last_became_active":"2024-02-13T00:52:30.597320+0000","last_became_peered":"2024-02-13T00:52:30.597320+0000","last_unstale":"2024-02-13T00:53:08.571622+0000","last_undegraded":"2024-02-13T00:53:08.571622+0000","last_fullsized":"2024-02-13T00:53:08.571622+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.880001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.880001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"467'27","reported_seq":882,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571540+0000","last_change":"2024-02-13T00:51:42.198644+0000","last_active":"2024-02-13T00:53:08.571540+0000","last_peered":"2024-02-13T00:53:08.571540+0000","last_clean":"2024-02-13T00:53:08.571540+0000","last_became_active":"2024-02-13T00:51:41.580479+0000","last_became_peered":"2024-02-13T00:51:41.580479+0000","last_unstale":"2024-02-13T00:53:08.571540+0000","last_undegraded":"2024-02-13T00:53:08.571540+0000","last_fullsized":"2024-02-13T00:53:08.571540+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:35.834669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.834669+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":443,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.068815+0000","last_change":"2024-02-13T00:52:30.581921+0000","last_active":"2024-02-13T00:53:09.068815+0000","last_peered":"2024-02-13T00:53:09.068815+0000","last_clean":"2024-02-13T00:53:09.068815+0000","last_became_active":"2024-02-13T00:52:30.581023+0000","last_became_peered":"2024-02-13T00:52:30.581023+0000","last_unstale":"2024-02-13T00:53:09.068815+0000","last_undegraded":"2024-02-13T00:53:09.068815+0000","last_fullsized":"2024-02-13T00:53:09.068815+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:48.521656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:51:48.521656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":462,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.572530+0000","last_change":"2024-02-13T00:52:30.595830+0000","last_active":"2024-02-13T00:53:08.572530+0000","last_peered":"2024-02-13T00:53:08.572530+0000","last_clean":"2024-02-13T00:53:08.572530+0000","last_became_active":"2024-02-13T00:52:30.594958+0000","last_became_peered":"2024-02-13T00:52:30.594958+0000","last_unstale":"2024-02-13T00:53:08.572530+0000","last_undegraded":"2024-02-13T00:53:08.572530+0000","last_fullsized":"2024-02-13T00:53:08.572530+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:36.152699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:36.152699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":449,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111860+0000","last_change":"2024-02-13T00:52:30.601489+0000","last_active":"2024-02-13T00:53:09.111860+0000","last_peered":"2024-02-13T00:53:09.111860+0000","last_clean":"2024-02-13T00:53:09.111860+0000","last_became_active":"2024-02-13T00:52:30.600883+0000","last_became_peered":"2024-02-13T00:52:30.600883+0000","last_unstale":"2024-02-13T00:53:09.111860+0000","last_undegraded":"2024-02-13T00:53:09.111860+0000","last_fullsized":"2024-02-13T00:53:09.111860+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:07.304576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:07.304576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":622,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.071386+0000","last_change":"2024-02-13T00:51:52.536631+0000","last_active":"2024-02-13T00:53:09.071386+0000","last_peered":"2024-02-13T00:53:09.071386+0000","last_clean":"2024-02-13T00:53:09.071386+0000","last_became_active":"2024-02-13T00:51:41.571280+0000","last_became_peered":"2024-02-13T00:51:41.571280+0000","last_unstale":"2024-02-13T00:53:09.071386+0000","last_undegraded":"2024-02-13T00:53:09.071386+0000","last_fullsized":"2024-02-13T00:53:09.071386+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'18","last_scrub_stamp":"2024-02-13T00:51:52.536524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:51:52.536524+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":447,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109953+0000","last_change":"2024-02-13T00:52:30.575760+0000","last_active":"2024-02-13T00:53:09.109953+0000","last_peered":"2024-02-13T00:53:09.109953+0000","last_clean":"2024-02-13T00:53:09.109953+0000","last_became_active":"2024-02-13T00:52:30.575381+0000","last_became_peered":"2024-02-13T00:52:30.575381+0000","last_unstale":"2024-02-13T00:53:09.109953+0000","last_undegraded":"2024-02-13T00:53:09.109953+0000","last_fullsized":"2024-02-13T00:53:09.109953+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:20.093279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:52:20.093279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":467,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.112029+0000","last_change":"2024-02-13T00:52:30.597684+0000","last_active":"2024-02-13T00:53:09.112029+0000","last_peered":"2024-02-13T00:53:09.112029+0000","last_clean":"2024-02-13T00:53:09.112029+0000","last_became_active":"2024-02-13T00:52:30.596322+0000","last_became_peered":"2024-02-13T00:52:30.596322+0000","last_unstale":"2024-02-13T00:53:09.112029+0000","last_undegraded":"2024-02-13T00:53:09.112029+0000","last_fullsized":"2024-02-13T00:53:09.112029+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:40.384428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:40.384428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":539,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.572674+0000","last_change":"2024-02-13T00:52:30.597677+0000","last_active":"2024-02-13T00:53:08.572674+0000","last_peered":"2024-02-13T00:53:08.572674+0000","last_clean":"2024-02-13T00:53:08.572674+0000","last_became_active":"2024-02-13T00:52:30.595811+0000","last_became_peered":"2024-02-13T00:52:30.595811+0000","last_unstale":"2024-02-13T00:53:08.572674+0000","last_undegraded":"2024-02-13T00:53:08.572674+0000","last_fullsized":"2024-02-13T00:53:08.572674+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:19.286611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:19.286611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":706,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070382+0000","last_change":"2024-02-13T00:51:43.571860+0000","last_active":"2024-02-13T00:53:09.070382+0000","last_peered":"2024-02-13T00:53:09.070382+0000","last_clean":"2024-02-13T00:53:09.070382+0000","last_became_active":"2024-02-13T00:51:41.569584+0000","last_became_peered":"2024-02-13T00:51:41.569584+0000","last_unstale":"2024-02-13T00:53:09.070382+0000","last_undegraded":"2024-02-13T00:53:09.070382+0000","last_fullsized":"2024-02-13T00:53:09.070382+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'39","last_scrub_stamp":"2024-02-13T00:49:06.154931+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:49:06.154931+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":709,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571730+0000","last_change":"2024-02-13T00:52:30.595096+0000","last_active":"2024-02-13T00:53:08.571730+0000","last_peered":"2024-02-13T00:53:08.571730+0000","last_clean":"2024-02-13T00:53:08.571730+0000","last_became_active":"2024-02-13T00:52:30.594229+0000","last_became_peered":"2024-02-13T00:52:30.594229+0000","last_unstale":"2024-02-13T00:53:08.571730+0000","last_undegraded":"2024-02-13T00:53:08.571730+0000","last_fullsized":"2024-02-13T00:53:08.571730+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.916152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"726'10","reported_seq":484,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.389670+0000","last_change":"2024-02-13T00:52:32.256014+0000","last_active":"2024-02-13T00:53:09.389670+0000","last_peered":"2024-02-13T00:53:09.389670+0000","last_clean":"2024-02-13T00:53:09.389670+0000","last_became_active":"2024-02-13T00:52:30.603380+0000","last_became_peered":"2024-02-13T00:52:30.603380+0000","last_unstale":"2024-02-13T00:53:09.389670+0000","last_undegraded":"2024-02-13T00:53:09.389670+0000","last_fullsized":"2024-02-13T00:53:09.389670+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9","last_scrub_stamp":"2024-02-13T00:47:37.012934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:47:37.012934+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":746,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571314+0000","last_change":"2024-02-13T00:52:30.598029+0000","last_active":"2024-02-13T00:53:08.571314+0000","last_peered":"2024-02-13T00:53:08.571314+0000","last_clean":"2024-02-13T00:53:08.571314+0000","last_became_active":"2024-02-13T00:52:30.596877+0000","last_became_peered":"2024-02-13T00:52:30.596877+0000","last_unstale":"2024-02-13T00:53:08.571314+0000","last_undegraded":"2024-02-13T00:53:08.571314+0000","last_fullsized":"2024-02-13T00:53:08.571314+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:26.647367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:26.647367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":408,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570661+0000","last_change":"2024-02-13T00:51:41.587438+0000","last_active":"2024-02-13T00:53:08.570661+0000","last_peered":"2024-02-13T00:53:08.570661+0000","last_clean":"2024-02-13T00:53:08.570661+0000","last_became_active":"2024-02-13T00:51:41.583454+0000","last_became_peered":"2024-02-13T00:51:41.583454+0000","last_unstale":"2024-02-13T00:53:08.570661+0000","last_undegraded":"2024-02-13T00:53:08.570661+0000","last_fullsized":"2024-02-13T00:53:08.570661+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.849687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.849687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"467'22","reported_seq":734,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111785+0000","last_change":"2024-02-13T00:51:44.862046+0000","last_active":"2024-02-13T00:53:09.111785+0000","last_peered":"2024-02-13T00:53:09.111785+0000","last_clean":"2024-02-13T00:53:09.111785+0000","last_became_active":"2024-02-13T00:51:41.584471+0000","last_became_peered":"2024-02-13T00:51:41.584471+0000","last_unstale":"2024-02-13T00:53:09.111785+0000","last_undegraded":"2024-02-13T00:53:09.111785+0000","last_fullsized":"2024-02-13T00:53:09.111785+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-02-13T00:47:35.369616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:35.369616+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":761,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570869+0000","last_change":"2024-02-13T00:52:30.598076+0000","last_active":"2024-02-13T00:53:08.570869+0000","last_peered":"2024-02-13T00:53:08.570869+0000","last_clean":"2024-02-13T00:53:08.570869+0000","last_became_active":"2024-02-13T00:52:30.593985+0000","last_became_peered":"2024-02-13T00:52:30.593985+0000","last_unstale":"2024-02-13T00:53:08.570869+0000","last_undegraded":"2024-02-13T00:53:08.570869+0000","last_fullsized":"2024-02-13T00:53:08.570869+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:05.887665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:05.887665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"726'16","reported_seq":645,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-02-13T00:53:10.218214+0000","last_change":"2024-02-13T00:53:10.218214+0000","last_active":"2024-02-13T00:53:10.218214+0000","last_peered":"2024-02-13T00:53:10.218214+0000","last_clean":"2024-02-13T00:53:10.218214+0000","last_became_active":"2024-02-13T00:52:30.603133+0000","last_became_peered":"2024-02-13T00:52:30.603133+0000","last_unstale":"2024-02-13T00:53:10.218214+0000","last_undegraded":"2024-02-13T00:53:10.218214+0000","last_fullsized":"2024-02-13T00:53:10.218214+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'16","last_scrub_stamp":"2024-02-13T00:53:10.218143+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:53:10.218143+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":385,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.572307+0000","last_change":"2024-02-13T00:52:30.596098+0000","last_active":"2024-02-13T00:53:08.572307+0000","last_peered":"2024-02-13T00:53:08.572307+0000","last_clean":"2024-02-13T00:53:08.572307+0000","last_became_active":"2024-02-13T00:52:30.594426+0000","last_became_peered":"2024-02-13T00:52:30.594426+0000","last_unstale":"2024-02-13T00:53:08.572307+0000","last_undegraded":"2024-02-13T00:53:08.572307+0000","last_fullsized":"2024-02-13T00:53:08.572307+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:27.933639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:27.933639+0000","last_clean_scrub_stamp":"2024-02-13T00:52:27.933639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":266,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069096+0000","last_change":"2024-02-13T00:53:07.596914+0000","last_active":"2024-02-13T00:53:09.069096+0000","last_peered":"2024-02-13T00:53:09.069096+0000","last_clean":"2024-02-13T00:53:09.069096+0000","last_became_active":"2024-02-13T00:53:07.596544+0000","last_became_peered":"2024-02-13T00:53:07.596544+0000","last_unstale":"2024-02-13T00:53:09.069096+0000","last_undegraded":"2024-02-13T00:53:09.069096+0000","last_fullsized":"2024-02-13T00:53:09.069096+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:34.127252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:34.127252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":802,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069182+0000","last_change":"2024-02-13T00:49:38.007273+0000","last_active":"2024-02-13T00:53:09.069182+0000","last_peered":"2024-02-13T00:53:09.069182+0000","last_clean":"2024-02-13T00:53:09.069182+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:53:09.069182+0000","last_undegraded":"2024-02-13T00:53:09.069182+0000","last_fullsized":"2024-02-13T00:53:09.069182+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'19","last_scrub_stamp":"2024-02-13T00:49:38.007204+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:49:38.007204+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":458,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111481+0000","last_change":"2024-02-13T00:51:41.571972+0000","last_active":"2024-02-13T00:53:09.111481+0000","last_peered":"2024-02-13T00:53:09.111481+0000","last_clean":"2024-02-13T00:53:09.111481+0000","last_became_active":"2024-02-13T00:51:41.570640+0000","last_became_peered":"2024-02-13T00:51:41.570640+0000","last_unstale":"2024-02-13T00:53:09.111481+0000","last_undegraded":"2024-02-13T00:53:09.111481+0000","last_fullsized":"2024-02-13T00:53:09.111481+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:13.316727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:51:13.316727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":521,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570575+0000","last_change":"2024-02-13T00:52:30.594111+0000","last_active":"2024-02-13T00:53:08.570575+0000","last_peered":"2024-02-13T00:53:08.570575+0000","last_clean":"2024-02-13T00:53:08.570575+0000","last_became_active":"2024-02-13T00:52:30.593800+0000","last_became_peered":"2024-02-13T00:52:30.593800+0000","last_unstale":"2024-02-13T00:53:08.570575+0000","last_undegraded":"2024-02-13T00:53:08.570575+0000","last_fullsized":"2024-02-13T00:53:08.570575+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:24.916456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:52:24.916456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":710,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.569656+0000","last_change":"2024-02-13T00:48:27.659266+0000","last_active":"2024-02-13T00:53:08.569656+0000","last_peered":"2024-02-13T00:53:08.569656+0000","last_clean":"2024-02-13T00:53:08.569656+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:53:08.569656+0000","last_undegraded":"2024-02-13T00:53:08.569656+0000","last_fullsized":"2024-02-13T00:53:08.569656+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:27.659197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:48:27.659197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":457,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110544+0000","last_change":"2024-02-13T00:53:08.375070+0000","last_active":"2024-02-13T00:53:09.110544+0000","last_peered":"2024-02-13T00:53:09.110544+0000","last_clean":"2024-02-13T00:53:09.110544+0000","last_became_active":"2024-02-13T00:51:41.570732+0000","last_became_peered":"2024-02-13T00:51:41.570732+0000","last_unstale":"2024-02-13T00:53:09.110544+0000","last_undegraded":"2024-02-13T00:53:09.110544+0000","last_fullsized":"2024-02-13T00:53:09.110544+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:08.374988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:53:08.374988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":497,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110552+0000","last_change":"2024-02-13T00:52:33.431600+0000","last_active":"2024-02-13T00:53:09.110552+0000","last_peered":"2024-02-13T00:53:09.110552+0000","last_clean":"2024-02-13T00:53:09.110552+0000","last_became_active":"2024-02-13T00:52:30.594405+0000","last_became_peered":"2024-02-13T00:52:30.594405+0000","last_unstale":"2024-02-13T00:53:09.110552+0000","last_undegraded":"2024-02-13T00:53:09.110552+0000","last_fullsized":"2024-02-13T00:53:09.110552+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:48:34.458264+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:48:34.458264+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":727,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070503+0000","last_change":"2024-02-13T00:51:47.538426+0000","last_active":"2024-02-13T00:53:09.070503+0000","last_peered":"2024-02-13T00:53:09.070503+0000","last_clean":"2024-02-13T00:53:09.070503+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:53:09.070503+0000","last_undegraded":"2024-02-13T00:53:09.070503+0000","last_fullsized":"2024-02-13T00:53:09.070503+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:47.538354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:51:47.538354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":750,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069250+0000","last_change":"2024-02-13T00:51:41.569851+0000","last_active":"2024-02-13T00:53:09.069250+0000","last_peered":"2024-02-13T00:53:09.069250+0000","last_clean":"2024-02-13T00:53:09.069250+0000","last_became_active":"2024-02-13T00:51:41.568200+0000","last_became_peered":"2024-02-13T00:51:41.568200+0000","last_unstale":"2024-02-13T00:53:09.069250+0000","last_undegraded":"2024-02-13T00:53:09.069250+0000","last_fullsized":"2024-02-13T00:53:09.069250+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:36.787604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:50:36.787604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":525,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.569930+0000","last_change":"2024-02-13T00:52:30.588849+0000","last_active":"2024-02-13T00:53:08.569930+0000","last_peered":"2024-02-13T00:53:08.569930+0000","last_clean":"2024-02-13T00:53:08.569930+0000","last_became_active":"2024-02-13T00:52:30.588561+0000","last_became_peered":"2024-02-13T00:52:30.588561+0000","last_unstale":"2024-02-13T00:53:08.569930+0000","last_undegraded":"2024-02-13T00:53:08.569930+0000","last_fullsized":"2024-02-13T00:53:08.569930+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:29.681414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:29.681414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":458,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110154+0000","last_change":"2024-02-13T00:51:41.586630+0000","last_active":"2024-02-13T00:53:09.110154+0000","last_peered":"2024-02-13T00:53:09.110154+0000","last_clean":"2024-02-13T00:53:09.110154+0000","last_became_active":"2024-02-13T00:51:41.584153+0000","last_became_peered":"2024-02-13T00:51:41.584153+0000","last_unstale":"2024-02-13T00:53:09.110154+0000","last_undegraded":"2024-02-13T00:53:09.110154+0000","last_fullsized":"2024-02-13T00:53:09.110154+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:02.260045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:02.260045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":569,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069147+0000","last_change":"2024-02-13T00:52:33.431814+0000","last_active":"2024-02-13T00:53:09.069147+0000","last_peered":"2024-02-13T00:53:09.069147+0000","last_clean":"2024-02-13T00:53:09.069147+0000","last_became_active":"2024-02-13T00:52:30.606082+0000","last_became_peered":"2024-02-13T00:52:30.606082+0000","last_unstale":"2024-02-13T00:53:09.069147+0000","last_undegraded":"2024-02-13T00:53:09.069147+0000","last_fullsized":"2024-02-13T00:53:09.069147+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'8","last_scrub_stamp":"2024-02-13T00:50:32.774820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:50:32.774820+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":675,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111670+0000","last_change":"2024-02-13T00:48:32.441279+0000","last_active":"2024-02-13T00:53:09.111670+0000","last_peered":"2024-02-13T00:53:09.111670+0000","last_clean":"2024-02-13T00:53:09.111670+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:53:09.111670+0000","last_undegraded":"2024-02-13T00:53:09.111670+0000","last_fullsized":"2024-02-13T00:53:09.111670+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.441170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.441170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":458,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109557+0000","last_change":"2024-02-13T00:52:30.579288+0000","last_active":"2024-02-13T00:53:09.109557+0000","last_peered":"2024-02-13T00:53:09.109557+0000","last_clean":"2024-02-13T00:53:09.109557+0000","last_became_active":"2024-02-13T00:52:30.578921+0000","last_became_peered":"2024-02-13T00:52:30.578921+0000","last_unstale":"2024-02-13T00:53:09.109557+0000","last_undegraded":"2024-02-13T00:53:09.109557+0000","last_fullsized":"2024-02-13T00:53:09.109557+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:28.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:28.353347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":454,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.068750+0000","last_change":"2024-02-13T00:52:30.603553+0000","last_active":"2024-02-13T00:53:09.068750+0000","last_peered":"2024-02-13T00:53:09.068750+0000","last_clean":"2024-02-13T00:53:09.068750+0000","last_became_active":"2024-02-13T00:52:30.602714+0000","last_became_peered":"2024-02-13T00:52:30.602714+0000","last_unstale":"2024-02-13T00:53:09.068750+0000","last_undegraded":"2024-02-13T00:53:09.068750+0000","last_fullsized":"2024-02-13T00:53:09.068750+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.052640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.052640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":605,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109772+0000","last_change":"2024-02-13T00:52:30.593895+0000","last_active":"2024-02-13T00:53:09.109772+0000","last_peered":"2024-02-13T00:53:09.109772+0000","last_clean":"2024-02-13T00:53:09.109772+0000","last_became_active":"2024-02-13T00:52:30.593558+0000","last_became_peered":"2024-02-13T00:52:30.593558+0000","last_unstale":"2024-02-13T00:53:09.109772+0000","last_undegraded":"2024-02-13T00:53:09.109772+0000","last_fullsized":"2024-02-13T00:53:09.109772+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:54.114116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:54.114116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"467'25","reported_seq":526,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109778+0000","last_change":"2024-02-13T00:51:43.826108+0000","last_active":"2024-02-13T00:53:09.109778+0000","last_peered":"2024-02-13T00:53:09.109778+0000","last_clean":"2024-02-13T00:53:09.109778+0000","last_became_active":"2024-02-13T00:51:41.585940+0000","last_became_peered":"2024-02-13T00:51:41.585940+0000","last_unstale":"2024-02-13T00:53:09.109778+0000","last_undegraded":"2024-02-13T00:53:09.109778+0000","last_fullsized":"2024-02-13T00:53:09.109778+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:47:36.368863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.368863+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":449,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111823+0000","last_change":"2024-02-13T00:46:59.388809+0000","last_active":"2024-02-13T00:53:09.111823+0000","last_peered":"2024-02-13T00:53:09.111823+0000","last_clean":"2024-02-13T00:53:09.111823+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:53:09.111823+0000","last_undegraded":"2024-02-13T00:53:09.111823+0000","last_fullsized":"2024-02-13T00:53:09.111823+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:59.388745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:59.388745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":458,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109943+0000","last_change":"2024-02-13T00:52:30.578502+0000","last_active":"2024-02-13T00:53:09.109943+0000","last_peered":"2024-02-13T00:53:09.109943+0000","last_clean":"2024-02-13T00:53:09.109943+0000","last_became_active":"2024-02-13T00:52:30.577297+0000","last_became_peered":"2024-02-13T00:52:30.577297+0000","last_unstale":"2024-02-13T00:53:09.109943+0000","last_undegraded":"2024-02-13T00:53:09.109943+0000","last_fullsized":"2024-02-13T00:53:09.109943+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:08.345652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:51:08.345652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":657,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570497+0000","last_change":"2024-02-13T00:52:30.572207+0000","last_active":"2024-02-13T00:53:08.570497+0000","last_peered":"2024-02-13T00:53:08.570497+0000","last_clean":"2024-02-13T00:53:08.570497+0000","last_became_active":"2024-02-13T00:52:30.569895+0000","last_became_peered":"2024-02-13T00:52:30.569895+0000","last_unstale":"2024-02-13T00:53:08.570497+0000","last_undegraded":"2024-02-13T00:53:08.570497+0000","last_fullsized":"2024-02-13T00:53:08.570497+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:06.264572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:51:06.264572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":572,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111675+0000","last_change":"2024-02-13T00:51:41.571094+0000","last_active":"2024-02-13T00:53:09.111675+0000","last_peered":"2024-02-13T00:53:09.111675+0000","last_clean":"2024-02-13T00:53:09.111675+0000","last_became_active":"2024-02-13T00:51:41.569716+0000","last_became_peered":"2024-02-13T00:51:41.569716+0000","last_unstale":"2024-02-13T00:53:09.111675+0000","last_undegraded":"2024-02-13T00:53:09.111675+0000","last_fullsized":"2024-02-13T00:53:09.111675+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:35.101298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.101298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":855,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571825+0000","last_change":"2024-02-13T00:51:44.341669+0000","last_active":"2024-02-13T00:53:08.571825+0000","last_peered":"2024-02-13T00:53:08.571825+0000","last_clean":"2024-02-13T00:53:08.571825+0000","last_became_active":"2024-02-13T00:51:41.589397+0000","last_became_peered":"2024-02-13T00:51:41.589397+0000","last_unstale":"2024-02-13T00:53:08.571825+0000","last_undegraded":"2024-02-13T00:53:08.571825+0000","last_fullsized":"2024-02-13T00:53:08.571825+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'32","last_scrub_stamp":"2024-02-13T00:48:53.841797+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:48:53.841797+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":695,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109322+0000","last_change":"2024-02-13T00:52:30.578860+0000","last_active":"2024-02-13T00:53:09.109322+0000","last_peered":"2024-02-13T00:53:09.109322+0000","last_clean":"2024-02-13T00:53:09.109322+0000","last_became_active":"2024-02-13T00:52:30.578094+0000","last_became_peered":"2024-02-13T00:52:30.578094+0000","last_unstale":"2024-02-13T00:53:09.109322+0000","last_undegraded":"2024-02-13T00:53:09.109322+0000","last_fullsized":"2024-02-13T00:53:09.109322+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:32.297918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:32.297918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":567,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110227+0000","last_change":"2024-02-13T00:52:32.509795+0000","last_active":"2024-02-13T00:53:09.110227+0000","last_peered":"2024-02-13T00:53:09.110227+0000","last_clean":"2024-02-13T00:53:09.110227+0000","last_became_active":"2024-02-13T00:52:30.585316+0000","last_became_peered":"2024-02-13T00:52:30.585316+0000","last_unstale":"2024-02-13T00:53:09.110227+0000","last_undegraded":"2024-02-13T00:53:09.110227+0000","last_fullsized":"2024-02-13T00:53:09.110227+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:48:33.460247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:33.460247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":694,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.071390+0000","last_change":"2024-02-13T00:51:41.571960+0000","last_active":"2024-02-13T00:53:09.071390+0000","last_peered":"2024-02-13T00:53:09.071390+0000","last_clean":"2024-02-13T00:53:09.071390+0000","last_became_active":"2024-02-13T00:51:41.569703+0000","last_became_peered":"2024-02-13T00:51:41.569703+0000","last_unstale":"2024-02-13T00:53:09.071390+0000","last_undegraded":"2024-02-13T00:53:09.071390+0000","last_fullsized":"2024-02-13T00:53:09.071390+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:39.570602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:39.570602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":480,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111944+0000","last_change":"2024-02-13T00:48:55.424021+0000","last_active":"2024-02-13T00:53:09.111944+0000","last_peered":"2024-02-13T00:53:09.111944+0000","last_clean":"2024-02-13T00:53:09.111944+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:53:09.111944+0000","last_undegraded":"2024-02-13T00:53:09.111944+0000","last_fullsized":"2024-02-13T00:53:09.111944+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:55.423952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.423952+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":458,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109858+0000","last_change":"2024-02-13T00:51:41.586115+0000","last_active":"2024-02-13T00:53:09.109858+0000","last_peered":"2024-02-13T00:53:09.109858+0000","last_clean":"2024-02-13T00:53:09.109858+0000","last_became_active":"2024-02-13T00:51:41.584919+0000","last_became_peered":"2024-02-13T00:51:41.584919+0000","last_unstale":"2024-02-13T00:53:09.109858+0000","last_undegraded":"2024-02-13T00:53:09.109858+0000","last_fullsized":"2024-02-13T00:53:09.109858+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:46:56.339885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:46:56.339885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":730,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070417+0000","last_change":"2024-02-13T00:52:30.603945+0000","last_active":"2024-02-13T00:53:09.070417+0000","last_peered":"2024-02-13T00:53:09.070417+0000","last_clean":"2024-02-13T00:53:09.070417+0000","last_became_active":"2024-02-13T00:52:30.602705+0000","last_became_peered":"2024-02-13T00:52:30.602705+0000","last_unstale":"2024-02-13T00:53:09.070417+0000","last_undegraded":"2024-02-13T00:53:09.070417+0000","last_fullsized":"2024-02-13T00:53:09.070417+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.928651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.928651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":445,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110740+0000","last_change":"2024-02-13T00:52:24.173739+0000","last_active":"2024-02-13T00:53:09.110740+0000","last_peered":"2024-02-13T00:53:09.110740+0000","last_clean":"2024-02-13T00:53:09.110740+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:53:09.110740+0000","last_undegraded":"2024-02-13T00:53:09.110740+0000","last_fullsized":"2024-02-13T00:53:09.110740+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:24.173651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:24.173651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":525,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.070371+0000","last_change":"2024-02-13T00:52:32.307501+0000","last_active":"2024-02-13T00:53:09.070371+0000","last_peered":"2024-02-13T00:53:09.070371+0000","last_clean":"2024-02-13T00:53:09.070371+0000","last_became_active":"2024-02-13T00:52:30.603658+0000","last_became_peered":"2024-02-13T00:52:30.603658+0000","last_unstale":"2024-02-13T00:53:09.070371+0000","last_undegraded":"2024-02-13T00:53:09.070371+0000","last_fullsized":"2024-02-13T00:53:09.070371+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:30.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:30.086856+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":37,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.109392+0000","last_change":"2024-02-13T00:52:30.578696+0000","last_active":"2024-02-13T00:53:09.109392+0000","last_peered":"2024-02-13T00:53:09.109392+0000","last_clean":"2024-02-13T00:53:09.109392+0000","last_became_active":"2024-02-13T00:52:30.578352+0000","last_became_peered":"2024-02-13T00:52:30.578352+0000","last_unstale":"2024-02-13T00:53:09.109392+0000","last_undegraded":"2024-02-13T00:53:09.109392+0000","last_fullsized":"2024-02-13T00:53:09.109392+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:04.282529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:04.282529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":449,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111834+0000","last_change":"2024-02-13T00:52:30.594940+0000","last_active":"2024-02-13T00:53:09.111834+0000","last_peered":"2024-02-13T00:53:09.111834+0000","last_clean":"2024-02-13T00:53:09.111834+0000","last_became_active":"2024-02-13T00:52:30.594309+0000","last_became_peered":"2024-02-13T00:52:30.594309+0000","last_unstale":"2024-02-13T00:53:09.111834+0000","last_undegraded":"2024-02-13T00:53:09.111834+0000","last_fullsized":"2024-02-13T00:53:09.111834+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:42.329367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:42.329367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":372,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111819+0000","last_change":"2024-02-13T00:52:30.597508+0000","last_active":"2024-02-13T00:53:09.111819+0000","last_peered":"2024-02-13T00:53:09.111819+0000","last_clean":"2024-02-13T00:53:09.111819+0000","last_became_active":"2024-02-13T00:52:30.595882+0000","last_became_peered":"2024-02-13T00:52:30.595882+0000","last_unstale":"2024-02-13T00:53:09.111819+0000","last_undegraded":"2024-02-13T00:53:09.111819+0000","last_fullsized":"2024-02-13T00:53:09.111819+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:23.430396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:23.430396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":629,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.071409+0000","last_change":"2024-02-13T00:52:32.857449+0000","last_active":"2024-02-13T00:53:09.071409+0000","last_peered":"2024-02-13T00:53:09.071409+0000","last_clean":"2024-02-13T00:53:09.071409+0000","last_became_active":"2024-02-13T00:52:30.604045+0000","last_became_peered":"2024-02-13T00:52:30.604045+0000","last_unstale":"2024-02-13T00:53:09.071409+0000","last_undegraded":"2024-02-13T00:53:09.071409+0000","last_fullsized":"2024-02-13T00:53:09.071409+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:52:21.312574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:21.312574+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":467,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110630+0000","last_change":"2024-02-13T00:52:30.596393+0000","last_active":"2024-02-13T00:53:09.110630+0000","last_peered":"2024-02-13T00:53:09.110630+0000","last_clean":"2024-02-13T00:53:09.110630+0000","last_became_active":"2024-02-13T00:52:30.596074+0000","last_became_peered":"2024-02-13T00:52:30.596074+0000","last_unstale":"2024-02-13T00:53:09.110630+0000","last_undegraded":"2024-02-13T00:53:09.110630+0000","last_fullsized":"2024-02-13T00:53:09.110630+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:18.264738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:18.264738+0000","last_clean_scrub_stamp":"2024-02-13T00:51:18.264738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":440,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.111769+0000","last_change":"2024-02-13T00:46:01.461188+0000","last_active":"2024-02-13T00:53:09.111769+0000","last_peered":"2024-02-13T00:53:09.111769+0000","last_clean":"2024-02-13T00:53:09.111769+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:53:09.111769+0000","last_undegraded":"2024-02-13T00:53:09.111769+0000","last_fullsized":"2024-02-13T00:53:09.111769+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:45:42.238152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:45:42.238152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":689,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571589+0000","last_change":"2024-02-13T00:51:35.190514+0000","last_active":"2024-02-13T00:53:08.571589+0000","last_peered":"2024-02-13T00:53:08.571589+0000","last_clean":"2024-02-13T00:53:08.571589+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:53:08.571589+0000","last_undegraded":"2024-02-13T00:53:08.571589+0000","last_fullsized":"2024-02-13T00:53:08.571589+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:35.190419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:35.190419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":617,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571508+0000","last_change":"2024-02-13T00:52:31.275396+0000","last_active":"2024-02-13T00:53:08.571508+0000","last_peered":"2024-02-13T00:53:08.571508+0000","last_clean":"2024-02-13T00:53:08.571508+0000","last_became_active":"2024-02-13T00:52:30.592405+0000","last_became_peered":"2024-02-13T00:52:30.592405+0000","last_unstale":"2024-02-13T00:53:08.571508+0000","last_undegraded":"2024-02-13T00:53:08.571508+0000","last_fullsized":"2024-02-13T00:53:08.571508+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:52:16.976052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:16.976052+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":533,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110308+0000","last_change":"2024-02-13T00:51:41.571773+0000","last_active":"2024-02-13T00:53:09.110308+0000","last_peered":"2024-02-13T00:53:09.110308+0000","last_clean":"2024-02-13T00:53:09.110308+0000","last_became_active":"2024-02-13T00:51:41.569507+0000","last_became_peered":"2024-02-13T00:51:41.569507+0000","last_unstale":"2024-02-13T00:53:09.110308+0000","last_undegraded":"2024-02-13T00:53:09.110308+0000","last_fullsized":"2024-02-13T00:53:09.110308+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:43.067615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:43.067615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":726,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570199+0000","last_change":"2024-02-13T00:52:35.023058+0000","last_active":"2024-02-13T00:53:08.570199+0000","last_peered":"2024-02-13T00:53:08.570199+0000","last_clean":"2024-02-13T00:53:08.570199+0000","last_became_active":"2024-02-13T00:51:41.582536+0000","last_became_peered":"2024-02-13T00:51:41.582536+0000","last_unstale":"2024-02-13T00:53:08.570199+0000","last_undegraded":"2024-02-13T00:53:08.570199+0000","last_fullsized":"2024-02-13T00:53:08.570199+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:35.022992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:52:35.022992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":632,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.571225+0000","last_change":"2024-02-13T00:51:41.587948+0000","last_active":"2024-02-13T00:53:08.571225+0000","last_peered":"2024-02-13T00:53:08.571225+0000","last_clean":"2024-02-13T00:53:08.571225+0000","last_became_active":"2024-02-13T00:51:41.583578+0000","last_became_peered":"2024-02-13T00:51:41.583578+0000","last_unstale":"2024-02-13T00:53:08.571225+0000","last_undegraded":"2024-02-13T00:53:08.571225+0000","last_fullsized":"2024-02-13T00:53:08.571225+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:15.846461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:15.846461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":648,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069409+0000","last_change":"2024-02-13T00:52:32.781431+0000","last_active":"2024-02-13T00:53:09.069409+0000","last_peered":"2024-02-13T00:53:09.069409+0000","last_clean":"2024-02-13T00:53:09.069409+0000","last_became_active":"2024-02-13T00:52:30.604370+0000","last_became_peered":"2024-02-13T00:52:30.604370+0000","last_unstale":"2024-02-13T00:53:09.069409+0000","last_undegraded":"2024-02-13T00:53:09.069409+0000","last_fullsized":"2024-02-13T00:53:09.069409+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'16","last_scrub_stamp":"2024-02-13T00:51:49.491168+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:51:49.491168+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":17,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069587+0000","last_change":"2024-02-13T00:53:01.780340+0000","last_active":"2024-02-13T00:53:09.069587+0000","last_peered":"2024-02-13T00:53:09.069587+0000","last_clean":"2024-02-13T00:53:09.069587+0000","last_became_active":"2024-02-13T00:53:01.780063+0000","last_became_peered":"2024-02-13T00:53:01.780063+0000","last_unstale":"2024-02-13T00:53:09.069587+0000","last_undegraded":"2024-02-13T00:53:09.069587+0000","last_fullsized":"2024-02-13T00:53:09.069587+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:47.284014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:47:47.284014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":675,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069383+0000","last_change":"2024-02-13T00:51:41.589710+0000","last_active":"2024-02-13T00:53:09.069383+0000","last_peered":"2024-02-13T00:53:09.069383+0000","last_clean":"2024-02-13T00:53:09.069383+0000","last_became_active":"2024-02-13T00:51:41.589029+0000","last_became_peered":"2024-02-13T00:51:41.589029+0000","last_unstale":"2024-02-13T00:53:09.069383+0000","last_undegraded":"2024-02-13T00:53:09.069383+0000","last_fullsized":"2024-02-13T00:53:09.069383+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:01.133709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:49:01.133709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":436,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110237+0000","last_change":"2024-02-13T00:52:38.236729+0000","last_active":"2024-02-13T00:53:09.110237+0000","last_peered":"2024-02-13T00:53:09.110237+0000","last_clean":"2024-02-13T00:53:09.110237+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:53:09.110237+0000","last_undegraded":"2024-02-13T00:53:09.110237+0000","last_fullsized":"2024-02-13T00:53:09.110237+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:38.236655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:38.236655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":515,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110314+0000","last_change":"2024-02-13T00:52:32.083766+0000","last_active":"2024-02-13T00:53:09.110314+0000","last_peered":"2024-02-13T00:53:09.110314+0000","last_clean":"2024-02-13T00:53:09.110314+0000","last_became_active":"2024-02-13T00:52:30.576217+0000","last_became_peered":"2024-02-13T00:52:30.576217+0000","last_unstale":"2024-02-13T00:53:09.110314+0000","last_undegraded":"2024-02-13T00:53:09.110314+0000","last_fullsized":"2024-02-13T00:53:09.110314+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:51:10.298233+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:51:10.298233+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":548,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.074659+0000","last_change":"2024-02-13T00:53:09.074634+0000","last_active":"2024-02-13T00:53:09.074659+0000","last_peered":"2024-02-13T00:53:09.074659+0000","last_clean":"2024-02-13T00:53:09.074659+0000","last_became_active":"2024-02-13T00:52:30.597277+0000","last_became_peered":"2024-02-13T00:52:30.597277+0000","last_unstale":"2024-02-13T00:53:09.074659+0000","last_undegraded":"2024-02-13T00:53:09.074659+0000","last_fullsized":"2024-02-13T00:53:09.074659+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:09.074568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:53:09.074568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":383,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.068862+0000","last_change":"2024-02-13T00:52:30.603770+0000","last_active":"2024-02-13T00:53:09.068862+0000","last_peered":"2024-02-13T00:53:09.068862+0000","last_clean":"2024-02-13T00:53:09.068862+0000","last_became_active":"2024-02-13T00:52:30.602788+0000","last_became_peered":"2024-02-13T00:52:30.602788+0000","last_unstale":"2024-02-13T00:53:09.068862+0000","last_undegraded":"2024-02-13T00:53:09.068862+0000","last_fullsized":"2024-02-13T00:53:09.068862+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:47:53.825971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:47:53.825971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":697,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.069481+0000","last_change":"2024-02-13T00:51:36.522278+0000","last_active":"2024-02-13T00:53:09.069481+0000","last_peered":"2024-02-13T00:53:09.069481+0000","last_clean":"2024-02-13T00:53:09.069481+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:53:09.069481+0000","last_undegraded":"2024-02-13T00:53:09.069481+0000","last_fullsized":"2024-02-13T00:53:09.069481+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:51:36.522208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:36.522208+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":629,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.570068+0000","last_change":"2024-02-13T00:53:06.023522+0000","last_active":"2024-02-13T00:53:08.570068+0000","last_peered":"2024-02-13T00:53:08.570068+0000","last_clean":"2024-02-13T00:53:08.570068+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:53:08.570068+0000","last_undegraded":"2024-02-13T00:53:08.570068+0000","last_fullsized":"2024-02-13T00:53:08.570068+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'62","last_scrub_stamp":"2024-02-13T00:53:06.023450+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:53:06.023450+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":734,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:08.572383+0000","last_change":"2024-02-13T00:52:30.596601+0000","last_active":"2024-02-13T00:53:08.572383+0000","last_peered":"2024-02-13T00:53:08.572383+0000","last_clean":"2024-02-13T00:53:08.572383+0000","last_became_active":"2024-02-13T00:52:30.595533+0000","last_became_peered":"2024-02-13T00:52:30.595533+0000","last_unstale":"2024-02-13T00:53:08.572383+0000","last_undegraded":"2024-02-13T00:53:08.572383+0000","last_fullsized":"2024-02-13T00:53:08.572383+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:54.817999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.817999+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":449,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-02-13T00:53:09.110646+0000","last_change":"2024-02-13T00:51:41.573986+0000","last_active":"2024-02-13T00:53:09.110646+0000","last_peered":"2024-02-13T00:53:09.110646+0000","last_clean":"2024-02-13T00:53:09.110646+0000","last_became_active":"2024-02-13T00:51:41.571183+0000","last_became_peered":"2024-02-13T00:51:41.571183+0000","last_unstale":"2024-02-13T00:53:09.110646+0000","last_undegraded":"2024-02-13T00:53:09.110646+0000","last_fullsized":"2024-02-13T00:53:09.110646+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:03.383666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:03.383666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":251,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":4000249,"num_objects":37,"num_object_clones":1,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":19,"num_read_kb":6,"num_write":57,"num_write_kb":3959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":4000081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6438912,"data_stored":6438912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":24,"acting":24,"num_store_stats":5},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1491,"num_read_kb":1216,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":780,"num_bytes_recovered":13820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":11436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":972,"ondisk_log_size":972,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":916,"num_read_kb":911,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266876,"num_pgs":145,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317032,"kb_used_data":7520,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428120,"statfs":{"total":95995035648,"available":95670394880,"internally_reserved":0,"allocated":7700480,"data_stored":4991017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.499,"15min":0.63300000000000001},"min":{"1min":0.30399999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.67600000000000005,"5min":0.76700000000000002,"15min":10.682},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50900000000000001,"15min":0.74399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.71599999999999997,"5min":0.77400000000000002,"15min":25.504000000000001},"last":0.52200000000000002}]},{"osd":1,"last update":"Tue Feb 13 00:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.57599999999999996},"min":{"1min":0.26600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64700000000000002,"5min":0.83499999999999996,"15min":13.347},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.66400000000000003},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67600000000000005,"5min":0.77300000000000002,"15min":13.526999999999999},"last":0.42799999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52500000000000002,"15min":0.65800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.26600000000000001},"max":{"1min":0.66300000000000003,"5min":0.871,"15min":14.359},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54300000000000004,"15min":0.73999999999999999},"min":{"1min":0.34899999999999998,"5min":0.32000000000000001,"15min":0.26400000000000001},"max":{"1min":0.67500000000000004,"5min":0.81599999999999995,"15min":21.497},"last":0.5}]},{"osd":3,"last update":"Tue Feb 13 00:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.70799999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.29699999999999999},"max":{"1min":0.69999999999999996,"5min":0.89400000000000002,"15min":12.997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54000000000000004,"15min":0.81899999999999995},"min":{"1min":0.42899999999999999,"5min":0.35199999999999998,"15min":0.318},"max":{"1min":0.628,"5min":0.79300000000000004,"15min":21.622},"last":0.44900000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.44500000000000001,"15min":0.438},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.23000000000000001},"max":{"1min":0.60099999999999998,"5min":0.76000000000000001,"15min":2.452},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.47799999999999998,"15min":0.44900000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.245},"max":{"1min":0.66400000000000003,"5min":0.83199999999999996,"15min":1.0780000000000001},"last":0.437}]},{"osd":5,"last update":"Tue Feb 13 00:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51000000000000001,"15min":0.50900000000000001},"min":{"1min":0.35599999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71199999999999997,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51500000000000001,"15min":0.52000000000000002},"min":{"1min":0.33500000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.747,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.47199999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51000000000000001,"15min":0.49399999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.25},"max":{"1min":0.71299999999999997,"5min":0.81299999999999994,"15min":0.95299999999999996},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51500000000000001,"15min":0.49399999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.27500000000000002},"max":{"1min":0.58299999999999996,"5min":0.85499999999999998,"15min":1.0940000000000001},"last":0.38400000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528514,"num_pgs":160,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304708,"kb_used_data":7484,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93440444,"statfs":{"total":95995035648,"available":95683014656,"internally_reserved":0,"allocated":7663616,"data_stored":4957083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.504,"15min":0.53900000000000003},"min":{"1min":0.36199999999999999,"5min":0.316,"15min":0.19700000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":8.1340000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.502,"15min":0.61699999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.221},"max":{"1min":0.71599999999999997,"5min":0.72799999999999998,"15min":11.367000000000001},"last":0.54400000000000004}]},{"osd":1,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.505,"15min":0.88400000000000001},"min":{"1min":0.33300000000000002,"5min":0.30099999999999999,"15min":0.255},"max":{"1min":0.89300000000000002,"5min":1.1080000000000001,"15min":54.002000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51200000000000001,"15min":0.67800000000000005},"min":{"1min":0.30599999999999999,"5min":0.27200000000000002,"15min":0.23499999999999999},"max":{"1min":0.73599999999999999,"5min":0.95199999999999996,"15min":12.403},"last":0.58999999999999997}]},{"osd":2,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52000000000000002,"15min":0.71199999999999997},"min":{"1min":0.32400000000000001,"5min":0.31900000000000001,"15min":0.25600000000000001},"max":{"1min":0.80000000000000004,"5min":1.587,"15min":12.477},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.504,"5min":0.5,"15min":0.67900000000000005},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.67100000000000004,"5min":1.1899999999999999,"15min":11.702},"last":0.501}]},{"osd":3,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56999999999999995,"15min":0.67300000000000004},"min":{"1min":0.32700000000000001,"5min":0.31900000000000001,"15min":0.29199999999999998},"max":{"1min":0.79400000000000004,"5min":5.931,"15min":8.0519999999999996},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53900000000000003,"15min":0.77200000000000002},"min":{"1min":0.38200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":19.081},"last":0.48299999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42299999999999999,"15min":0.42099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.24199999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":1.0649999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46100000000000002,"15min":0.44500000000000001},"min":{"1min":0.28799999999999998,"5min":0.252,"15min":0.221},"max":{"1min":0.66600000000000004,"5min":1.008,"15min":1.008},"last":0.307}]},{"osd":5,"last update":"Tue Feb 13 00:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.49399999999999999,"15min":0.51100000000000001},"min":{"1min":0.29299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66200000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.314,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73099999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.35699999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:53:03 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47399999999999998,"15min":0.47499999999999998},"min":{"1min":0.30399999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":1.173},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53000000000000003,"15min":0.52100000000000002},"min":{"1min":0.312,"5min":0.30099999999999999,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":0.80700000000000005,"15min":1.262},"last":0.41599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575067,"num_pgs":153,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316976,"kb_used_data":7464,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428176,"statfs":{"total":95995035648,"available":95670452224,"internally_reserved":0,"allocated":7643136,"data_stored":4917224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34000000000000002,"15min":0.95299999999999996},"min":{"1min":0.23899999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":61.856000000000002},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.67100000000000004},"min":{"1min":0.224,"5min":0.19900000000000001,"15min":0.16500000000000001},"max":{"1min":0.74399999999999999,"5min":0.77600000000000002,"15min":61.908999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.33900000000000002,"15min":0.83699999999999997},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":0.629,"5min":0.629,"15min":32.161000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.41899999999999998,"15min":0.83799999999999997},"min":{"1min":0.251,"5min":0.245,"15min":0.187},"max":{"1min":5.1319999999999997,"5min":5.1319999999999997,"15min":29.693000000000001},"last":0.316}]},{"osd":3,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.36699999999999999,"15min":1.2509999999999999},"min":{"1min":0.27500000000000002,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":131.23099999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.38400000000000001,"15min":0.84599999999999997},"min":{"1min":0.29099999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.753,"5min":0.753,"15min":49.055},"last":0.28799999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53200000000000003,"15min":0.749},"min":{"1min":0.374,"5min":0.35999999999999999,"15min":0.27900000000000003},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":16.547000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.58299999999999996,"15min":0.71499999999999997},"min":{"1min":0.36099999999999999,"5min":0.34300000000000003,"15min":0.24199999999999999},"max":{"1min":0.83599999999999997,"5min":0.85199999999999998,"15min":14.731999999999999},"last":0.39600000000000002}]},{"osd":5,"last update":"Tue Feb 13 00:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58199999999999996,"15min":0.57199999999999995},"min":{"1min":0.41299999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84699999999999998,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59799999999999998,"15min":0.58699999999999997},"min":{"1min":0.377,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76100000000000001,"5min":1.831,"15min":1.831},"last":0.52000000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57099999999999995,"15min":0.71099999999999997},"min":{"1min":0.39000000000000001,"5min":0.32300000000000001,"15min":0.29099999999999998},"max":{"1min":0.878,"5min":1.9279999999999999,"15min":14.772},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55200000000000005,"15min":0.91200000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77300000000000002,"5min":1.2709999999999999,"15min":26.600000000000001},"last":0.51100000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59199999999999997,"15min":0.71999999999999997},"min":{"1min":0.41699999999999998,"5min":0.38200000000000001,"15min":0.29499999999999998},"max":{"1min":0.94399999999999995,"5min":1.8680000000000001,"15min":14.794},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57899999999999996,"15min":0.91600000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.32200000000000001},"max":{"1min":0.83899999999999997,"5min":1.246,"15min":26.693999999999999},"last":0.53700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738591,"num_pgs":154,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315040,"kb_used_data":5528,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430112,"statfs":{"total":95995035648,"available":95672434688,"internally_reserved":0,"allocated":5660672,"data_stored":2959027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33900000000000002,"15min":0.622},"min":{"1min":0.22500000000000001,"5min":0.191,"15min":0.189},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":30.184000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.36499999999999999,"15min":0.45600000000000002},"min":{"1min":0.20699999999999999,"5min":0.20499999999999999,"15min":0.187},"max":{"1min":1.0109999999999999,"5min":1.0780000000000001,"15min":5.6959999999999997},"last":0.34599999999999997}]},{"osd":2,"last update":"Tue Feb 13 00:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.373,"15min":0.49399999999999999},"min":{"1min":0.248,"5min":0.24399999999999999,"15min":0.216},"max":{"1min":0.72499999999999998,"5min":1.137,"15min":7.0839999999999996},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.377,"15min":0.67200000000000004},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.185},"max":{"1min":0.61099999999999999,"5min":2.0129999999999999,"15min":24.459},"last":0.249}]},{"osd":3,"last update":"Tue Feb 13 00:52:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.36899999999999999,"15min":0.48099999999999998},"min":{"1min":0.25900000000000001,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":8.718},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.40699999999999997,"15min":0.52600000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.222},"max":{"1min":1.097,"5min":1.1120000000000001,"15min":7.1139999999999999},"last":0.30099999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55200000000000005,"15min":0.80800000000000005},"min":{"1min":0.379,"5min":0.34899999999999998,"15min":0.24399999999999999},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":24.381},"last":0.62},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59399999999999997,"15min":0.68000000000000005},"min":{"1min":0.40799999999999997,"5min":0.39600000000000002,"15min":0.28699999999999998},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":8.6950000000000003},"last":0.60199999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.68799999999999994,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54800000000000004,"15min":0.53600000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71399999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.63400000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.55000000000000004,"15min":0.63800000000000001},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.31},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":8.6129999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.66000000000000003},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.32400000000000001},"max":{"1min":1.411,"5min":1.411,"15min":14.263},"last":0.61099999999999999}]},{"osd":7,"last update":"Tue Feb 13 00:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57399999999999995,"15min":0.66900000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.317},"max":{"1min":2.496,"5min":2.496,"15min":8.6679999999999993},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58499999999999996,"15min":0.67000000000000004},"min":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.35899999999999999},"max":{"1min":1.375,"5min":1.375,"15min":8.1609999999999996},"last":0.58499999999999996}]}]},{"osd":2,"up_from":0,"seq":77309411542,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182608,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":133143986380,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":2057289334847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":3218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2068480,"data_stored":2068480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2117632,"data_stored":2117632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2142208,"data_stored":2142208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:53:15.058 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool set default.rgw.meta pgp_num 42 2024-02-13T00:53:15.197 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:53:15.548 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:15.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:15.575 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:15.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:53:15.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:15.680 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:15.700 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:15.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:15.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:15.723 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:14.240662+0000 osd.1 (osd.1) 313 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:53:15.723 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:14.241637+0000 osd.1 (osd.1) 314 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:53:15.723 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:14.534923+0000 mgr.y (mgr.14544) 626 : cluster [DBG] pgmap v1145: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:53:15.723 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:14.612862+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-134"}]': finished 2024-02-13T00:53:15.723 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:14.612928+0000 mon.a (mon.0) 3603 : cluster [DBG] osdmap e756: 8 total, 8 up, 4 in 2024-02-13T00:53:15.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:14.614562+0000 mon.a (mon.0) 3604 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-134", "mode": "readproxy"}]: dispatch 2024-02-13T00:53:15.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:15.559706+0000 mon.a (mon.0) 3605 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:53:15.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:15.564408+0000 mon.a (mon.0) 3606 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-134", "mode": "readproxy"}]': finished 2024-02-13T00:53:15.724 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:15.564461+0000 mon.a (mon.0) 3607 : cluster [DBG] osdmap e757: 8 total, 8 up, 4 in 2024-02-13T00:53:15.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:15.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:14.240662+0000 osd.1 (osd.1) 313 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:53:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:14.241637+0000 osd.1 (osd.1) 314 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:53:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:14.534923+0000 mgr.y (mgr.14544) 626 : cluster [DBG] pgmap v1145: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:53:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:14.612862+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-134"}]': finished 2024-02-13T00:53:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:14.612928+0000 mon.a (mon.0) 3603 : cluster [DBG] osdmap e756: 8 total, 8 up, 4 in 2024-02-13T00:53:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:14.614562+0000 mon.a (mon.0) 3604 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-134", "mode": "readproxy"}]: dispatch 2024-02-13T00:53:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:15.559706+0000 mon.a (mon.0) 3605 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:53:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:15.564408+0000 mon.a (mon.0) 3606 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-134", "mode": "readproxy"}]': finished 2024-02-13T00:53:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:15.564461+0000 mon.a (mon.0) 3607 : cluster [DBG] osdmap e757: 8 total, 8 up, 4 in 2024-02-13T00:53:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:14.240662+0000 osd.1 (osd.1) 313 : cluster [DBG] 5.7 scrub starts 2024-02-13T00:53:15.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:14.241637+0000 osd.1 (osd.1) 314 : cluster [DBG] 5.7 scrub ok 2024-02-13T00:53:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:14.534923+0000 mgr.y (mgr.14544) 626 : cluster [DBG] pgmap v1145: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:53:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:14.612862+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-134"}]': finished 2024-02-13T00:53:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:14.612928+0000 mon.a (mon.0) 3603 : cluster [DBG] osdmap e756: 8 total, 8 up, 4 in 2024-02-13T00:53:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:14.614562+0000 mon.a (mon.0) 3604 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-134", "mode": "readproxy"}]: dispatch 2024-02-13T00:53:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:15.559706+0000 mon.a (mon.0) 3605 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:53:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:15.564408+0000 mon.a (mon.0) 3606 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-134", "mode": "readproxy"}]': finished 2024-02-13T00:53:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:15.564461+0000 mon.a (mon.0) 3607 : cluster [DBG] osdmap e757: 8 total, 8 up, 4 in 2024-02-13T00:53:15.831 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:16.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:53:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:53:16.180 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:16.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:16.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:16.312 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:16.439 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:16.439 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:53:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:53:16] "GET /metrics HTTP/1.1" 200 189968 "" "Prometheus/2.33.4" 2024-02-13T00:53:16.562 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:16.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:16.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:16.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:15.945643+0000 mgr.y (mgr.14544) 627 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:16.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:15.945643+0000 mgr.y (mgr.14544) 627 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:16.694 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:16.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:15.945643+0000 mgr.y (mgr.14544) 627 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:16.817 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:16.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:16.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:16.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:16.951 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:16.972 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:16.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:16.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:17.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:17.086 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:53:17.086 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:53:17.086 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:53:17.086 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:53:17.105 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:17.123 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:17.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:17.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:17.254 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:17.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.911+0000 7f6d562e3700 1 -- 172.21.15.162:0/1848061219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d500fcbb0 msgr2=0x7f6d500fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.911+0000 7f6d562e3700 1 --2- 172.21.15.162:0/1848061219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d500fcbb0 0x7f6d500fcf90 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 rx=0x7f6d40004870 tx=0x7f6d40009cb0).stop 2024-02-13T00:53:17.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.911+0000 7f6d562e3700 1 -- 172.21.15.162:0/1848061219 shutdown_connections 2024-02-13T00:53:17.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.911+0000 7f6d562e3700 1 --2- 172.21.15.162:0/1848061219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d500f7990 0x7f6d501039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.911+0000 7f6d562e3700 1 --2- 172.21.15.162:0/1848061219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d500fcbb0 0x7f6d500fcf90 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.911+0000 7f6d562e3700 1 --2- 172.21.15.162:0/1848061219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d500fd510 0x7f6d500f72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.911+0000 7f6d562e3700 1 -- 172.21.15.162:0/1848061219 >> 172.21.15.162:0/1848061219 conn(0x7f6d50070080 msgr2=0x7f6d500f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:17.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.911+0000 7f6d562e3700 1 -- 172.21.15.162:0/1848061219 shutdown_connections 2024-02-13T00:53:17.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.911+0000 7f6d562e3700 1 -- 172.21.15.162:0/1848061219 wait complete. 2024-02-13T00:53:17.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.911+0000 7f6d562e3700 1 librados: starting msgr at 2024-02-13T00:53:17.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.911+0000 7f6d562e3700 1 librados: starting objecter 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d562e3700 1 Processor -- start 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d562e3700 1 -- start start 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d562e3700 1 librados: setting wanted keys 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d562e3700 1 librados: calling monclient init 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d562e3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d500f7990 0x7f6d5010d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d562e3700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d500fcbb0 0x7f6d5010d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d562e3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d500fd510 0x7f6d50107340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d562e3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d50107830 con 0x7f6d500f7990 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d562e3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d501079a0 con 0x7f6d500fd510 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d562e3700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d50107b60 con 0x7f6d500fcbb0 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d4ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d500f7990 0x7f6d5010d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d4f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d500fcbb0 0x7f6d5010d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d4ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d500f7990 0x7f6d5010d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:51368/0 (socket says 172.21.15.162:51368) 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d4f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d500fcbb0 0x7f6d5010d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:36344/0 (socket says 172.21.15.162:36344) 2024-02-13T00:53:17.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d4ffff700 1 -- 172.21.15.162:0/3332518596 learned_addr learned my addr 172.21.15.162:0/3332518596 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d54880700 1 --2- 172.21.15.162:0/3332518596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d500fd510 0x7f6d50107340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d4ffff700 1 -- 172.21.15.162:0/3332518596 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d500fcbb0 msgr2=0x7f6d5010d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d4ffff700 1 --2- 172.21.15.162:0/3332518596 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d500fcbb0 0x7f6d5010d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d4ffff700 1 -- 172.21.15.162:0/3332518596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d500fd510 msgr2=0x7f6d50107340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d4ffff700 1 --2- 172.21.15.162:0/3332518596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d500fd510 0x7f6d50107340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d4ffff700 1 -- 172.21.15.162:0/3332518596 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d44005040 con 0x7f6d500f7990 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d54880700 1 --2- 172.21.15.162:0/3332518596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d500fd510 0x7f6d50107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.912+0000 7f6d4f7fe700 1 --2- 172.21.15.162:0/3332518596 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6d500fcbb0 0x7f6d5010d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.913+0000 7f6d4ffff700 1 --2- 172.21.15.162:0/3332518596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d500f7990 0x7f6d5010d1a0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 rx=0x7f6d40009eb0 tx=0x7f6d4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.913+0000 7f6d4d775700 10 client.?.objecter ms_handle_connect 0x7f6d500f7990 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.913+0000 7f6d4d775700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.913+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4000e070 con 0x7f6d500f7990 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.913+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d40005040 con 0x7f6d500f7990 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.913+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d40013da0 con 0x7f6d500f7990 2024-02-13T00:53:17.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.913+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d40019070 con 0x7f6d500f7990 2024-02-13T00:53:17.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.913+0000 7f6d562e3700 10 client.28074.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:17.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.913+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d50108000 con 0x7f6d500f7990 2024-02-13T00:53:17.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.913+0000 7f6d562e3700 1 librados: init done 2024-02-13T00:53:17.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.914+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6d501081f0 con 0x7f6d500f7990 2024-02-13T00:53:17.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.914+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6d400283f0 con 0x7f6d500f7990 2024-02-13T00:53:17.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.914+0000 7f6d4d775700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d38062c90 0x7f6d38065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:17.352 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.914+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(757..757 src has 449..757) v4 ==== 8656+0+0 (secure 0 0 0) 0x7f6d4007e010 con 0x7f6d500f7990 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.914+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 osd_map(757..757 src has 449..757) v4 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.914+0000 7f6d4d775700 3 client.28074.objecter handle_osd_map got epochs [757,757] > 0 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.914+0000 7f6d4d775700 3 client.28074.objecter handle_osd_map decoding full epoch 757 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.914+0000 7f6d4f7fe700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d38062c90 0x7f6d38065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d4d775700 20 client.28074.objecter dump_active .. 0 homeless 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v757) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6d40085070 con 0x7f6d500f7990 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d562e3700 10 client.28074.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 1 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6d50028590 0x7f6d500fd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d5005fb80 osd=0 3 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d5005fb80 osd=0 3 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d5005fb80 osd=0 3 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 1 2024-02-13T00:53:17.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d5005fb80 osd=0 3 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 0 1 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d5005fb80 osd=0 4 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d562e3700 10 client.28074.objecter _send_command 1 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d500475e0 con 0x7f6d50028590 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d4d775700 10 client.28074.objecter ms_handle_connect 0x7f6d38062c90 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d4f7fe700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d38062c90 0x7f6d38065150 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 rx=0x7f6d50108ae0 tx=0x7f6d4400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d54880700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6d50028590 0x7f6d500fd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d4d775700 10 client.28074.objecter ms_handle_connect 0x7f6d50028590 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.915+0000 7f6d54880700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6d50028590 0x7f6d500fd950 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.916+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6d500475e0 con 0x7f6d50028590 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.916+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.916+0000 7f6d4d775700 10 client.28074.objecter _finish_command 1 = system:0 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.916+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d5005fb80 osd=0 4 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.916+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 0 1 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.930+0000 7f6d562e3700 10 client.28074.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:17.354 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.930+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 2 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.930+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d5005fb80 osd=0 3 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.930+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d5005fb80 osd=0 3 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.930+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d5005fb80 osd=0 3 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.930+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 2 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.930+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d5005fb80 osd=0 3 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.930+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 0 2 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.930+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d5005fb80 osd=0 4 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.930+0000 7f6d562e3700 10 client.28074.objecter _send_command 2 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.930+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d501060e0 con 0x7f6d50028590 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6d501060e0 con 0x7f6d50028590 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d4d775700 10 client.28074.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d5005fb80 osd=0 4 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 0 2 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d562e3700 10 client.28074.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:17.355 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 3 2024-02-13T00:53:17.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6805/150599780 conn(0x7f6d500f6820 0x7f6d500f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:17.356 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:53:17.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d500f61a0 osd=1 3 2024-02-13T00:53:17.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d500f61a0 osd=1 3 2024-02-13T00:53:17.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d500f61a0 osd=1 3 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 3 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d500f61a0 osd=1 3 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 1 3 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d500f61a0 osd=1 4 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d562e3700 10 client.28074.objecter _send_command 3 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d500f7120 con 0x7f6d500f6820 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.932+0000 7f6d4ffff700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6805/150599780 conn(0x7f6d500f6820 0x7f6d500f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.933+0000 7f6d4d775700 10 client.28074.objecter ms_handle_connect 0x7f6d500f6820 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.933+0000 7f6d4ffff700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6805/150599780 conn(0x7f6d500f6820 0x7f6d500f6c30 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.933+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6d500f7120 con 0x7f6d500f6820 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.933+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.933+0000 7f6d4d775700 10 client.28074.objecter _finish_command 3 = system:0 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.933+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d500f61a0 osd=1 4 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.933+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 1 3 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.948+0000 7f6d562e3700 10 client.28074.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:17.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.948+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 4 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.948+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d500f61a0 osd=1 3 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.948+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d500f61a0 osd=1 3 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.948+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d500f61a0 osd=1 3 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.948+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 4 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.948+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d500f61a0 osd=1 3 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.948+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 1 4 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.948+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d500f61a0 osd=1 4 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.948+0000 7f6d562e3700 10 client.28074.objecter _send_command 4 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.948+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d500fefc0 con 0x7f6d500f6820 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6d500fefc0 con 0x7f6d500f6820 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d4d775700 10 client.28074.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d500f61a0 osd=1 4 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 1 4 2024-02-13T00:53:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d562e3700 10 client.28074.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 5 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6d500280f0 0x7f6d50115fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d50059640 osd=2 3 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50059640 osd=2 3 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d50059640 osd=2 3 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 5 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d50059640 osd=2 3 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 2 5 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50059640 osd=2 4 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d562e3700 10 client.28074.objecter _send_command 5 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d501164d0 con 0x7f6d500280f0 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.949+0000 7f6d4f7fe700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6d500280f0 0x7f6d50115fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.950+0000 7f6d4d775700 10 client.28074.objecter ms_handle_connect 0x7f6d500280f0 2024-02-13T00:53:17.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.950+0000 7f6d4f7fe700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6d500280f0 0x7f6d50115fe0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.950+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6d501164d0 con 0x7f6d500280f0 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.950+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.950+0000 7f6d4d775700 10 client.28074.objecter _finish_command 5 = system:0 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.950+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d50059640 osd=2 4 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.950+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 2 5 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.965+0000 7f6d562e3700 10 client.28074.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.965+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 6 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.965+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d50059640 osd=2 3 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.965+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50059640 osd=2 3 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.965+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d50059640 osd=2 3 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.965+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 6 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.965+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d50059640 osd=2 3 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.965+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 2 6 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.965+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50059640 osd=2 4 2024-02-13T00:53:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.965+0000 7f6d562e3700 10 client.28074.objecter _send_command 6 2024-02-13T00:53:17.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.965+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d500276b0 con 0x7f6d500280f0 2024-02-13T00:53:17.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.966+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6d500276b0 con 0x7f6d500280f0 2024-02-13T00:53:17.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.966+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:17.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.966+0000 7f6d4d775700 10 client.28074.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.361 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:53:17.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.966+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d50059640 osd=2 4 2024-02-13T00:53:17.361 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.966+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 2 6 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.966+0000 7f6d562e3700 10 client.28074.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.966+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 7 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.966+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6813/604971011 conn(0x7f6d5011d7f0 0x7f6d5011fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.967+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d50116e70 osd=3 3 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.967+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50116e70 osd=3 3 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.967+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d50116e70 osd=3 3 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.967+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 7 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.967+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d50116e70 osd=3 3 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.967+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 3 7 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.967+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50116e70 osd=3 4 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.967+0000 7f6d562e3700 10 client.28074.objecter _send_command 7 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.967+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d50122960 con 0x7f6d5011d7f0 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.967+0000 7f6d54880700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6813/604971011 conn(0x7f6d5011d7f0 0x7f6d5011fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.967+0000 7f6d4d775700 10 client.28074.objecter ms_handle_connect 0x7f6d5011d7f0 2024-02-13T00:53:17.362 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.967+0000 7f6d54880700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6813/604971011 conn(0x7f6d5011d7f0 0x7f6d5011fbe0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.968+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6d50122960 con 0x7f6d5011d7f0 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.968+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.968+0000 7f6d4d775700 10 client.28074.objecter _finish_command 7 = system:0 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.968+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d50116e70 osd=3 4 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.968+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 3 7 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.983+0000 7f6d562e3700 10 client.28074.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.983+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 8 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.983+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d50116e70 osd=3 3 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.983+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50116e70 osd=3 3 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.983+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d50116e70 osd=3 3 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.983+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 8 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.983+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d50116e70 osd=3 3 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.983+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 3 8 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.983+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50116e70 osd=3 4 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.983+0000 7f6d562e3700 10 client.28074.objecter _send_command 8 2024-02-13T00:53:17.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.983+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d501168e0 con 0x7f6d5011d7f0 2024-02-13T00:53:17.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.984+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6d501168e0 con 0x7f6d5011d7f0 2024-02-13T00:53:17.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.984+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:17.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.984+0000 7f6d4d775700 10 client.28074.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.984+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d50116e70 osd=3 4 2024-02-13T00:53:17.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.984+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 3 8 2024-02-13T00:53:17.364 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:53:17.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d562e3700 10 client.28074.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 9 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6d500feb20 0x7f6d501201a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d501168e0 osd=4 3 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d501168e0 osd=4 3 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d501168e0 osd=4 3 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 9 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d501168e0 osd=4 3 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 4 9 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d501168e0 osd=4 4 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d562e3700 10 client.28074.objecter _send_command 9 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d50116bf0 con 0x7f6d500feb20 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d4ffff700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6d500feb20 0x7f6d501201a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d4d775700 10 client.28074.objecter ms_handle_connect 0x7f6d500feb20 2024-02-13T00:53:17.365 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.985+0000 7f6d4ffff700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6d500feb20 0x7f6d501201a0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:17.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.986+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6d50116bf0 con 0x7f6d500feb20 2024-02-13T00:53:17.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.986+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:53:17.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.986+0000 7f6d4d775700 10 client.28074.objecter _finish_command 9 = system:0 2024-02-13T00:53:17.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.986+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d501168e0 osd=4 4 2024-02-13T00:53:17.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:16.986+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 4 9 2024-02-13T00:53:17.366 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:53:17.366 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.000+0000 7f6d562e3700 10 client.28074.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.000+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 10 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.000+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d501168e0 osd=4 3 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.000+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d501168e0 osd=4 3 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.000+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d501168e0 osd=4 3 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.000+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 10 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.000+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d501168e0 osd=4 3 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.000+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 4 10 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.000+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d501168e0 osd=4 4 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.000+0000 7f6d562e3700 10 client.28074.objecter _send_command 10 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.000+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d50116bb0 con 0x7f6d500feb20 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6d50116bb0 con 0x7f6d500feb20 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d4d775700 10 client.28074.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d501168e0 osd=4 4 2024-02-13T00:53:17.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 4 10 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d562e3700 10 client.28074.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 11 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6d50120ba0 0x7f6d50121000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d50116bb0 osd=5 3 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50116bb0 osd=5 3 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d50116bb0 osd=5 3 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 11 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d50116bb0 osd=5 3 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 5 11 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50116bb0 osd=5 4 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d562e3700 10 client.28074.objecter _send_command 11 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d50129300 con 0x7f6d50120ba0 2024-02-13T00:53:17.368 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d4f7fe700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6d50120ba0 0x7f6d50121000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d4d775700 10 client.28074.objecter ms_handle_connect 0x7f6d50120ba0 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.002+0000 7f6d4f7fe700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6d50120ba0 0x7f6d50121000 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.003+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6d50129300 con 0x7f6d50120ba0 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.003+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.003+0000 7f6d4d775700 10 client.28074.objecter _finish_command 11 = system:0 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.003+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d50116bb0 osd=5 4 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.003+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 5 11 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.018+0000 7f6d562e3700 10 client.28074.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.018+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 12 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.018+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d50116bb0 osd=5 3 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.018+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50116bb0 osd=5 3 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.018+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d50116bb0 osd=5 3 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.018+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 12 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.018+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d50116bb0 osd=5 3 2024-02-13T00:53:17.369 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.018+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 5 12 2024-02-13T00:53:17.370 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.018+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50116bb0 osd=5 4 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.018+0000 7f6d562e3700 10 client.28074.objecter _send_command 12 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.018+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d501175c0 con 0x7f6d50120ba0 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6d501175c0 con 0x7f6d50120ba0 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d4d775700 10 client.28074.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d50116bb0 osd=5 4 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 5 12 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d562e3700 10 client.28074.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 13 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6d50117c40 0x7f6d50118080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d501175c0 osd=6 3 2024-02-13T00:53:17.371 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d501175c0 osd=6 3 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d501175c0 osd=6 3 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 13 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d501175c0 osd=6 3 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 6 13 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d501175c0 osd=6 4 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d562e3700 10 client.28074.objecter _send_command 13 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.020+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d5012e5d0 con 0x7f6d50117c40 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.021+0000 7f6d54880700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6d50117c40 0x7f6d50118080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.021+0000 7f6d54880700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6d50117c40 0x7f6d50118080 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.021+0000 7f6d4d775700 10 client.28074.objecter ms_handle_connect 0x7f6d50117c40 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.022+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6d5012e5d0 con 0x7f6d50117c40 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.022+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.022+0000 7f6d4d775700 10 client.28074.objecter _finish_command 13 = system:0 2024-02-13T00:53:17.372 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.022+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d501175c0 osd=6 4 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.022+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 6 13 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.036+0000 7f6d562e3700 10 client.28074.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.036+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 14 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.036+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d501175c0 osd=6 3 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.036+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d501175c0 osd=6 3 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.036+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d501175c0 osd=6 3 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.036+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 14 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.036+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d501175c0 osd=6 3 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.036+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 6 14 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.036+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d501175c0 osd=6 4 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.036+0000 7f6d562e3700 10 client.28074.objecter _send_command 14 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.036+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d5012e5d0 con 0x7f6d50117c40 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6d5012e5d0 con 0x7f6d50117c40 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d4d775700 10 client.28074.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d501175c0 osd=6 4 2024-02-13T00:53:17.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 6 14 2024-02-13T00:53:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:17.374 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:53:17.374 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:17.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d562e3700 10 client.28074.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 15 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6d50118bd0 0x7f6d50119030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d5012e5d0 osd=7 3 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d5012e5d0 osd=7 3 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d5012e5d0 osd=7 3 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 15 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d5012e5d0 osd=7 3 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 7 15 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d5012e5d0 osd=7 4 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d562e3700 10 client.28074.objecter _send_command 15 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d50119650 con 0x7f6d50118bd0 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.039+0000 7f6d4ffff700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6d50118bd0 0x7f6d50119030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.040+0000 7f6d4d775700 10 client.28074.objecter ms_handle_connect 0x7f6d50118bd0 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.040+0000 7f6d4ffff700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6d50118bd0 0x7f6d50119030 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:17.375 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.040+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6d4004fa90 con 0x7f6d50118bd0 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.040+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.040+0000 7f6d4d775700 10 client.28074.objecter _finish_command 15 = system:0 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.040+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d5012e5d0 osd=7 4 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.040+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 7 15 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.055+0000 7f6d562e3700 10 client.28074.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.055+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign -1 16 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.055+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d5012e5d0 osd=7 3 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.055+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d5012e5d0 osd=7 3 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.055+0000 7f6d562e3700 20 client.28074.objecter _get_session s=0x7f6d5012e5d0 osd=7 3 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.055+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_remove -1 16 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.055+0000 7f6d562e3700 20 client.28074.objecter get_session s=0x7f6d5012e5d0 osd=7 3 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.055+0000 7f6d562e3700 15 client.28074.objecter _session_command_op_assign 7 16 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.055+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d5012e5d0 osd=7 4 2024-02-13T00:53:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.055+0000 7f6d562e3700 10 client.28074.objecter _send_command 16 2024-02-13T00:53:17.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.055+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d50130800 con 0x7f6d50118bd0 2024-02-13T00:53:17.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.056+0000 7f6d4d775700 1 -- 172.21.15.162:0/3332518596 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6d50130800 con 0x7f6d50118bd0 2024-02-13T00:53:17.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.056+0000 7f6d4d775700 10 client.28074.objecter ms_dispatch 0x7f6d50001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:17.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.056+0000 7f6d4d775700 10 client.28074.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.056+0000 7f6d4d775700 20 client.28074.objecter put_session s=0x7f6d5012e5d0 osd=7 4 2024-02-13T00:53:17.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.056+0000 7f6d4d775700 15 client.28074.objecter _session_command_op_remove 7 16 2024-02-13T00:53:17.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:17.377 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:53:17.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.056+0000 7f6d562e3700 10 librados: watch_flush enter 2024-02-13T00:53:17.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.056+0000 7f6d562e3700 10 librados: watch_flush exit 2024-02-13T00:53:17.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.056+0000 7f6d562e3700 10 client.28074.objecter close_session for osd.0 2024-02-13T00:53:17.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.056+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6d50028590 msgr2=0x7f6d500fd950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.056+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6801/2103743338 conn(0x7f6d50028590 0x7f6d500fd950 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d5005fb80 osd=0 1 2024-02-13T00:53:17.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 10 client.28074.objecter close_session for osd.1 2024-02-13T00:53:17.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6805/150599780 conn(0x7f6d500f6820 msgr2=0x7f6d500f6c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6805/150599780 conn(0x7f6d500f6820 0x7f6d500f6c30 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d500f61a0 osd=1 1 2024-02-13T00:53:17.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 10 client.28074.objecter close_session for osd.2 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6d500280f0 msgr2=0x7f6d50115fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6809/3689719376 conn(0x7f6d500280f0 0x7f6d50115fe0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50059640 osd=2 1 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 10 client.28074.objecter close_session for osd.3 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6813/604971011 conn(0x7f6d5011d7f0 msgr2=0x7f6d5011fbe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6813/604971011 conn(0x7f6d5011d7f0 0x7f6d5011fbe0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d54880700 1 -- 172.21.15.162:0/3332518596 reap_dead start 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50116e70 osd=3 1 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 10 client.28074.objecter close_session for osd.4 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6d500feb20 msgr2=0x7f6d501201a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6800/3344263388 conn(0x7f6d500feb20 0x7f6d501201a0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d501168e0 osd=4 1 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 10 client.28074.objecter close_session for osd.5 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6d50120ba0 msgr2=0x7f6d50121000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6804/2277917372 conn(0x7f6d50120ba0 0x7f6d50121000 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d50116bb0 osd=5 1 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 10 client.28074.objecter close_session for osd.6 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6d50117c40 msgr2=0x7f6d50118080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6808/1506640341 conn(0x7f6d50117c40 0x7f6d50118080 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d501175c0 osd=6 1 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 10 client.28074.objecter close_session for osd.7 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6d50118bd0 msgr2=0x7f6d50119030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.181:6812/1964192772 conn(0x7f6d50118bd0 0x7f6d50119030 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 20 client.28074.objecter put_session s=0x7f6d5012e5d0 osd=7 1 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 20 client.28074.objecter shutdown clearing up homeless session... 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 10 client.28074.objecter successfully canceled tick 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d54880700 1 -- 172.21.15.162:0/3332518596 reap_dead start 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d38062c90 msgr2=0x7f6d38065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d38062c90 0x7f6d38065150 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 rx=0x7f6d50108ae0 tx=0x7f6d4400b040).stop 2024-02-13T00:53:17.380 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d500f7990 msgr2=0x7f6d5010d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d500f7990 0x7f6d5010d1a0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 rx=0x7f6d40009eb0 tx=0x7f6d4000b040).stop 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 shutdown_connections 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d500f7990 0x7f6d5010d1a0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 --2- 172.21.15.162:0/3332518596 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6d38062c90 0x7f6d38065150 unknown :-1 s=CLOSED pgs=3034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 >> 172.21.15.162:0/3332518596 conn(0x7f6d50070080 msgr2=0x7f6d500fa9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 shutdown_connections 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 -- 172.21.15.162:0/3332518596 wait complete. 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:17.057+0000 7f6d562e3700 1 librados: shutdown 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.381 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:53:17.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:17.504 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:17.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-13T00:53:17.852 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:17.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:17.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:17.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:16.535353+0000 mgr.y (mgr.14544) 628 : cluster [DBG] pgmap v1148: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:53:17.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:16.915387+0000 mon.a (mon.0) 3608 : audit [DBG] from='client.? 172.21.15.162:0/3332518596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:17.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:17.005979+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.11 deep-scrub starts 2024-02-13T00:53:17.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:17.006800+0000 osd.0 (osd.0) 304 : cluster [DBG] 6.11 deep-scrub ok 2024-02-13T00:53:17.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:17.983 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:16.535353+0000 mgr.y (mgr.14544) 628 : cluster [DBG] pgmap v1148: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:53:17.983 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:16.915387+0000 mon.a (mon.0) 3608 : audit [DBG] from='client.? 172.21.15.162:0/3332518596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:17.983 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:17.005979+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.11 deep-scrub starts 2024-02-13T00:53:17.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:17.006800+0000 osd.0 (osd.0) 304 : cluster [DBG] 6.11 deep-scrub ok 2024-02-13T00:53:17.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:16.535353+0000 mgr.y (mgr.14544) 628 : cluster [DBG] pgmap v1148: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-02-13T00:53:17.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:16.915387+0000 mon.a (mon.0) 3608 : audit [DBG] from='client.? 172.21.15.162:0/3332518596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:17.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:17.005979+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.11 deep-scrub starts 2024-02-13T00:53:17.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:17.006800+0000 osd.0 (osd.0) 304 : cluster [DBG] 6.11 deep-scrub ok 2024-02-13T00:53:17.986 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:18.085 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:18.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:18.116 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:53:18.190 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:18.217 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:18.317 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:18.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:18.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:18.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:18.449 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:18.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:53:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:53:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:53:18.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:17.395682+0000 osd.6 (osd.6) 241 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:53:18.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:17.397012+0000 osd.6 (osd.6) 242 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:53:19.007 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:17.395682+0000 osd.6 (osd.6) 241 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:53:19.007 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:17.397012+0000 osd.6 (osd.6) 242 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:53:19.007 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:17.395682+0000 osd.6 (osd.6) 241 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:53:19.007 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:17.397012+0000 osd.6 (osd.6) 242 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:53:19.298 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:19.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:19.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:19.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:19.433 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:19.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.529+0000 7f4bede9b700 1 -- 172.21.15.162:0/3494337807 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4be80fcb30 msgr2=0x7f4be80fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:19.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.529+0000 7f4bede9b700 1 --2- 172.21.15.162:0/3494337807 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4be80fcb30 0x7f4be80fcf10 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 rx=0x7f4bdc0047b0 tx=0x7f4bdc009cb0).stop 2024-02-13T00:53:19.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.529+0000 7f4bede9b700 1 -- 172.21.15.162:0/3494337807 shutdown_connections 2024-02-13T00:53:19.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.529+0000 7f4bede9b700 1 --2- 172.21.15.162:0/3494337807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4be80f73a0 0x7f4be81039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:19.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.529+0000 7f4bede9b700 1 --2- 172.21.15.162:0/3494337807 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4be80fcb30 0x7f4be80fcf10 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:19.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.529+0000 7f4bede9b700 1 --2- 172.21.15.162:0/3494337807 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be80fd490 0x7f4be80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:19.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.529+0000 7f4bede9b700 1 -- 172.21.15.162:0/3494337807 >> 172.21.15.162:0/3494337807 conn(0x7f4be806eb90 msgr2=0x7f4be80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:19.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.529+0000 7f4bede9b700 1 -- 172.21.15.162:0/3494337807 shutdown_connections 2024-02-13T00:53:19.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.529+0000 7f4bede9b700 1 -- 172.21.15.162:0/3494337807 wait complete. 2024-02-13T00:53:19.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.529+0000 7f4bede9b700 1 librados: starting msgr at 2024-02-13T00:53:19.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.529+0000 7f4bede9b700 1 librados: starting objecter 2024-02-13T00:53:19.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.529+0000 7f4bede9b700 1 Processor -- start 2024-02-13T00:53:19.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4bede9b700 1 -- start start 2024-02-13T00:53:19.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4bede9b700 1 librados: setting wanted keys 2024-02-13T00:53:19.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4bede9b700 1 librados: calling monclient init 2024-02-13T00:53:19.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4bede9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be80f73a0 0x7f4be81102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:19.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4bede9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4be80fcb30 0x7f4be8110790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:19.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4bede9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4be80fd490 0x7f4be810a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4bede9b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be810a930 con 0x7f4be80fd490 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4bede9b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be810aaa0 con 0x7f4be80f73a0 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4bede9b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be810ac60 con 0x7f4be80fcb30 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be7fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4be80fd490 0x7f4be810a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be6ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4be80fcb30 0x7f4be8110790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be7fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4be80fd490 0x7f4be810a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:51386/0 (socket says 172.21.15.162:51386) 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be7fff700 1 -- 172.21.15.162:0/2635571417 learned_addr learned my addr 172.21.15.162:0/2635571417 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be77fe700 1 --2- 172.21.15.162:0/2635571417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be80f73a0 0x7f4be81102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be6ffd700 1 -- 172.21.15.162:0/2635571417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be80f73a0 msgr2=0x7f4be81102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be6ffd700 1 --2- 172.21.15.162:0/2635571417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be80f73a0 0x7f4be81102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be6ffd700 1 -- 172.21.15.162:0/2635571417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4be80fd490 msgr2=0x7f4be810a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be6ffd700 1 --2- 172.21.15.162:0/2635571417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4be80fd490 0x7f4be810a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be6ffd700 1 -- 172.21.15.162:0/2635571417 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be00097c0 con 0x7f4be80fcb30 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be7fff700 1 --2- 172.21.15.162:0/2635571417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4be80fd490 0x7f4be810a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be6ffd700 1 --2- 172.21.15.162:0/2635571417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4be80fcb30 0x7f4be8110790 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 rx=0x7f4bd800c810 tx=0x7f4bd8007e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:19.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be4ff9700 10 client.?.objecter ms_handle_connect 0x7f4be80fcb30 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.530+0000 7f4be4ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.531+0000 7f4be4ff9700 1 -- 172.21.15.162:0/2635571417 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd800e4a0 con 0x7f4be80fcb30 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.531+0000 7f4be4ff9700 1 -- 172.21.15.162:0/2635571417 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4bd800ead0 con 0x7f4be80fcb30 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.531+0000 7f4be4ff9700 1 -- 172.21.15.162:0/2635571417 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8012a40 con 0x7f4be80fcb30 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.531+0000 7f4bede9b700 1 -- 172.21.15.162:0/2635571417 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc005040 con 0x7f4be80fcb30 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.531+0000 7f4bede9b700 10 client.38177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.531+0000 7f4bede9b700 1 -- 172.21.15.162:0/2635571417 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be810b2a0 con 0x7f4be80fcb30 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.531+0000 7f4bede9b700 1 librados: init done 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.532+0000 7f4bede9b700 1 -- 172.21.15.162:0/2635571417 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be80475e0 con 0x7f4be80fcb30 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.532+0000 7f4be4ff9700 1 -- 172.21.15.162:0/2635571417 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4bd800ec40 con 0x7f4be80fcb30 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.532+0000 7f4be4ff9700 1 --2- 172.21.15.162:0/2635571417 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4bd0062c90 0x7f4bd0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.533+0000 7f4be77fe700 1 --2- 172.21.15.162:0/2635571417 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4bd0062c90 0x7f4bd0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.533+0000 7f4be4ff9700 1 -- 172.21.15.162:0/2635571417 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(757..757 src has 449..757) v4 ==== 8656+0+0 (secure 0 0 0) 0x7f4bd807cfc0 con 0x7f4be80fcb30 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.533+0000 7f4be4ff9700 10 client.38177.objecter ms_dispatch 0x7f4be8001550 osd_map(757..757 src has 449..757) v4 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.533+0000 7f4be4ff9700 3 client.38177.objecter handle_osd_map got epochs [757,757] > 0 2024-02-13T00:53:19.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.533+0000 7f4be4ff9700 3 client.38177.objecter handle_osd_map decoding full epoch 757 2024-02-13T00:53:19.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.533+0000 7f4be4ff9700 20 client.38177.objecter dump_active .. 0 homeless 2024-02-13T00:53:19.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.533+0000 7f4be4ff9700 10 client.38177.objecter ms_handle_connect 0x7f4bd0062c90 2024-02-13T00:53:19.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.533+0000 7f4be77fe700 1 --2- 172.21.15.162:0/2635571417 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4bd0062c90 0x7f4bd0065150 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 rx=0x7f4bdc004e20 tx=0x7f4bdc007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:19.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.537+0000 7f4be4ff9700 1 -- 172.21.15.162:0/2635571417 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4bd8050240 con 0x7f4be80fcb30 2024-02-13T00:53:19.669 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:18.535790+0000 mgr.y (mgr.14544) 629 : cluster [DBG] pgmap v1149: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 690 B/s rd, 172 B/s wr, 0 op/s 2024-02-13T00:53:19.669 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:19.083392+0000 osd.0 (osd.0) 305 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:53:19.669 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:19.085312+0000 osd.0 (osd.0) 306 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:53:19.669 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:19.343100+0000 mon.a (mon.0) 3609 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:19.680 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:19.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:19.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:19.707+0000 7f4bede9b700 1 -- 172.21.15.162:0/2635571417 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "42"} v 0) v1 -- 0x7f4be805fb80 con 0x7f4be80fcb30 2024-02-13T00:53:19.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:19.813 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:19.911 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:19.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:19.925 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:53:19.925 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:53:19.926 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-02-13T00:53:19.926 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-02-13T00:53:19.926 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:53:20.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:20.045 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:18.535790+0000 mgr.y (mgr.14544) 629 : cluster [DBG] pgmap v1149: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 690 B/s rd, 172 B/s wr, 0 op/s 2024-02-13T00:53:20.045 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:19.083392+0000 osd.0 (osd.0) 305 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:53:20.045 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:19.085312+0000 osd.0 (osd.0) 306 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:53:20.045 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:19.343100+0000 mon.a (mon.0) 3609 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:20.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:18.535790+0000 mgr.y (mgr.14544) 629 : cluster [DBG] pgmap v1149: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 690 B/s rd, 172 B/s wr, 0 op/s 2024-02-13T00:53:20.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:19.083392+0000 osd.0 (osd.0) 305 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:53:20.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:19.085312+0000 osd.0 (osd.0) 306 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:53:20.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:19.343100+0000 mon.a (mon.0) 3609 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:20.048 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:20.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:53:20.497 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:20.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:20.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-13T00:53:20.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:20.673 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:20.693 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:20.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:20.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.733+0000 7f4be4ff9700 1 -- 172.21.15.162:0/2635571417 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "42"}]=0 set pool 6 pgp_num to 42 v758) v1 ==== 143+0+0 (secure 0 0 0) 0x7f4bd8018070 con 0x7f4be80fcb30 2024-02-13T00:53:20.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.735+0000 7f4bede9b700 10 librados: watch_flush enter 2024-02-13T00:53:20.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.735+0000 7f4bede9b700 10 librados: watch_flush exit 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.735+0000 7f4bede9b700 20 client.38177.objecter shutdown clearing up homeless session... 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.735+0000 7f4bede9b700 10 client.38177.objecter successfully canceled tick 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.735+0000 7f4bede9b700 1 -- 172.21.15.162:0/2635571417 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4bd0062c90 msgr2=0x7f4bd0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.735+0000 7f4bede9b700 1 --2- 172.21.15.162:0/2635571417 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4bd0062c90 0x7f4bd0065150 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 rx=0x7f4bdc004e20 tx=0x7f4bdc007f90).stop 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.736+0000 7f4bede9b700 1 -- 172.21.15.162:0/2635571417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4be80fcb30 msgr2=0x7f4be8110790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.736+0000 7f4bede9b700 1 --2- 172.21.15.162:0/2635571417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4be80fcb30 0x7f4be8110790 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 rx=0x7f4bd800c810 tx=0x7f4bd8007e30).stop 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.736+0000 7f4bede9b700 1 -- 172.21.15.162:0/2635571417 shutdown_connections 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.736+0000 7f4bede9b700 1 --2- 172.21.15.162:0/2635571417 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4bd0062c90 0x7f4bd0065150 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.736+0000 7f4bede9b700 1 --2- 172.21.15.162:0/2635571417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4be80fd490 0x7f4be810a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.736+0000 7f4bede9b700 1 --2- 172.21.15.162:0/2635571417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be80f73a0 0x7f4be81102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.736+0000 7f4bede9b700 1 --2- 172.21.15.162:0/2635571417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4be80fcb30 0x7f4be8110790 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.736+0000 7f4bede9b700 1 -- 172.21.15.162:0/2635571417 >> 172.21.15.162:0/2635571417 conn(0x7f4be806eb90 msgr2=0x7f4be8060e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.736+0000 7f4bede9b700 1 -- 172.21.15.162:0/2635571417 shutdown_connections 2024-02-13T00:53:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.736+0000 7f4bede9b700 1 -- 172.21.15.162:0/2635571417 wait complete. 2024-02-13T00:53:20.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:20.736+0000 7f4bede9b700 1 librados: shutdown 2024-02-13T00:53:20.742 INFO:teuthology.orchestra.run.smithi162.stderr:set pool 6 pgp_num to 42 2024-02-13T00:53:20.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:19.371307+0000 osd.1 (osd.1) 315 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:53:20.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:19.374782+0000 osd.1 (osd.1) 316 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:53:20.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:19.709205+0000 mon.c (mon.2) 505 : audit [INF] from='client.? 172.21.15.162:0/2635571417' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "42"}]: dispatch 2024-02-13T00:53:20.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:19.709509+0000 mon.a (mon.0) 3610 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "42"}]: dispatch 2024-02-13T00:53:20.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:20.404842+0000 osd.6 (osd.6) 243 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:53:20.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:20.407728+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:53:20.770 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:19.371307+0000 osd.1 (osd.1) 315 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:53:20.770 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:19.374782+0000 osd.1 (osd.1) 316 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:53:20.770 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:19.709205+0000 mon.c (mon.2) 505 : audit [INF] from='client.? 172.21.15.162:0/2635571417' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "42"}]: dispatch 2024-02-13T00:53:20.770 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:19.709509+0000 mon.a (mon.0) 3610 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "42"}]: dispatch 2024-02-13T00:53:20.770 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:20.404842+0000 osd.6 (osd.6) 243 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:53:20.771 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:20.407728+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:53:20.796 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:20.823 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:20.843 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:20.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:19.371307+0000 osd.1 (osd.1) 315 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:53:20.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:19.374782+0000 osd.1 (osd.1) 316 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:53:20.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:19.709205+0000 mon.c (mon.2) 505 : audit [INF] from='client.? 172.21.15.162:0/2635571417' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "42"}]: dispatch 2024-02-13T00:53:20.846 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:19.709509+0000 mon.a (mon.0) 3610 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "42"}]: dispatch 2024-02-13T00:53:20.846 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:20.404842+0000 osd.6 (osd.6) 243 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:53:20.846 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:20.407728+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:53:20.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:20.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:20.919 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:53:20.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:20.972 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:21.171 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:21.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:21.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:21.301 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:21.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:21.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:21.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:21.556 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:21.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.677+0000 7f2662ff5700 1 -- 172.21.15.162:0/1867493104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f265c0fd490 msgr2=0x7f265c0f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:21.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.677+0000 7f2662ff5700 1 --2- 172.21.15.162:0/1867493104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f265c0fd490 0x7f265c0f6cc0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 rx=0x7f2648004870 tx=0x7f2648009cb0).stop 2024-02-13T00:53:21.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.677+0000 7f2662ff5700 1 -- 172.21.15.162:0/1867493104 shutdown_connections 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.677+0000 7f2662ff5700 1 --2- 172.21.15.162:0/1867493104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f265c0f73a0 0x7f265c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.677+0000 7f2662ff5700 1 --2- 172.21.15.162:0/1867493104 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f265c0fcb30 0x7f265c0fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.677+0000 7f2662ff5700 1 --2- 172.21.15.162:0/1867493104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f265c0fd490 0x7f265c0f6cc0 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.677+0000 7f2662ff5700 1 -- 172.21.15.162:0/1867493104 >> 172.21.15.162:0/1867493104 conn(0x7f265c06eb90 msgr2=0x7f265c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.677+0000 7f2662ff5700 1 -- 172.21.15.162:0/1867493104 shutdown_connections 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.677+0000 7f2662ff5700 1 -- 172.21.15.162:0/1867493104 wait complete. 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.677+0000 7f2662ff5700 1 librados: starting msgr at 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.677+0000 7f2662ff5700 1 librados: starting objecter 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2662ff5700 1 Processor -- start 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2662ff5700 1 -- start start 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2662ff5700 1 librados: setting wanted keys 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2662ff5700 1 librados: calling monclient init 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2662ff5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f265c0f73a0 0x7f265c0f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:21.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2662ff5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f265c0fcb30 0x7f265c0f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2662ff5700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f265c0fd490 0x7f265c0f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2662ff5700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c116620 con 0x7f265c0f73a0 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2662ff5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c116790 con 0x7f265c0fcb30 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2662ff5700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265c116950 con 0x7f265c0fd490 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2661592700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f265c0fd490 0x7f265c0f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2660d91700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f265c0f73a0 0x7f265c0f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2661592700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f265c0fd490 0x7f265c0f9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:36388/0 (socket says 172.21.15.162:36388) 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2661592700 1 -- 172.21.15.162:0/597229652 learned_addr learned my addr 172.21.15.162:0/597229652 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f265bfff700 1 --2- 172.21.15.162:0/597229652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f265c0fcb30 0x7f265c0f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2660d91700 1 -- 172.21.15.162:0/597229652 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f265c0fd490 msgr2=0x7f265c0f9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2660d91700 1 --2- 172.21.15.162:0/597229652 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f265c0fd490 0x7f265c0f9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2660d91700 1 -- 172.21.15.162:0/597229652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f265c0fcb30 msgr2=0x7f265c0f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2660d91700 1 --2- 172.21.15.162:0/597229652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f265c0fcb30 0x7f265c0f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2660d91700 1 -- 172.21.15.162:0/597229652 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2650005040 con 0x7f265c0f73a0 2024-02-13T00:53:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2661592700 1 --2- 172.21.15.162:0/597229652 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f265c0fd490 0x7f265c0f9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:21.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2660d91700 1 --2- 172.21.15.162:0/597229652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f265c0f73a0 0x7f265c0f8930 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 rx=0x7f2650004920 tx=0x7f2650009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:21.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2659ffb700 10 client.?.objecter ms_handle_connect 0x7f265c0f73a0 2024-02-13T00:53:21.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2659ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:21.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2659ffb700 1 -- 172.21.15.162:0/597229652 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265000e030 con 0x7f265c0f73a0 2024-02-13T00:53:21.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2659ffb700 1 -- 172.21.15.162:0/597229652 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f265000f040 con 0x7f265c0f73a0 2024-02-13T00:53:21.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2662ff5700 1 -- 172.21.15.162:0/597229652 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2648005040 con 0x7f265c0f73a0 2024-02-13T00:53:21.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.678+0000 7f2659ffb700 1 -- 172.21.15.162:0/597229652 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650014850 con 0x7f265c0f73a0 2024-02-13T00:53:21.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.679+0000 7f2662ff5700 10 client.28086.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:21.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.679+0000 7f2662ff5700 1 -- 172.21.15.162:0/597229652 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c116fe0 con 0x7f265c0f73a0 2024-02-13T00:53:21.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.679+0000 7f2662ff5700 1 librados: init done 2024-02-13T00:53:21.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.680+0000 7f2662ff5700 1 -- 172.21.15.162:0/597229652 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265c105fe0 con 0x7f265c0f73a0 2024-02-13T00:53:21.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.680+0000 7f2659ffb700 1 -- 172.21.15.162:0/597229652 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f2650010500 con 0x7f265c0f73a0 2024-02-13T00:53:21.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.680+0000 7f2659ffb700 1 --2- 172.21.15.162:0/597229652 >> v2:172.21.15.162:6800/2866534053 conn(0x7f264c062c90 0x7f264c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:21.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.680+0000 7f2659ffb700 1 -- 172.21.15.162:0/597229652 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(758..758 src has 449..758) v4 ==== 8656+0+0 (secure 0 0 0) 0x7f265007d450 con 0x7f265c0f73a0 2024-02-13T00:53:21.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.680+0000 7f2659ffb700 10 client.28086.objecter ms_dispatch 0x7f265c001550 osd_map(758..758 src has 449..758) v4 2024-02-13T00:53:21.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.680+0000 7f2659ffb700 3 client.28086.objecter handle_osd_map got epochs [758,758] > 0 2024-02-13T00:53:21.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.680+0000 7f2659ffb700 3 client.28086.objecter handle_osd_map decoding full epoch 758 2024-02-13T00:53:21.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.680+0000 7f265bfff700 1 --2- 172.21.15.162:0/597229652 >> v2:172.21.15.162:6800/2866534053 conn(0x7f264c062c90 0x7f264c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:21.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.681+0000 7f2659ffb700 20 client.28086.objecter dump_active .. 0 homeless 2024-02-13T00:53:21.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.681+0000 7f2659ffb700 10 client.28086.objecter ms_handle_connect 0x7f264c062c90 2024-02-13T00:53:21.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.681+0000 7f265bfff700 1 --2- 172.21.15.162:0/597229652 >> v2:172.21.15.162:6800/2866534053 conn(0x7f264c062c90 0x7f264c065150 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 rx=0x7f2648004ee0 tx=0x7f264800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:21.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.685+0000 7f2659ffb700 1 -- 172.21.15.162:0/597229652 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2650050750 con 0x7f265c0f73a0 2024-02-13T00:53:21.691 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:21.755 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:21.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:21.858 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:21.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:21.863+0000 7f2662ff5700 1 -- 172.21.15.162:0/597229652 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f265c0f9e10 con 0x7f265c0f73a0 2024-02-13T00:53:21.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:20.248081+0000 osd.1 (osd.1) 317 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:21.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:20.248231+0000 osd.1 (osd.1) 318 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:21.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:20.536535+0000 mgr.y (mgr.14544) 630 : cluster [DBG] pgmap v1150: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.4 KiB/s rd, 147 B/s wr, 1 op/s 2024-02-13T00:53:21.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:20.734155+0000 mon.a (mon.0) 3611 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "42"}]': finished 2024-02-13T00:53:21.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:20.734211+0000 mon.a (mon.0) 3612 : cluster [DBG] osdmap e758: 8 total, 8 up, 4 in 2024-02-13T00:53:21.886 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:20.248081+0000 osd.1 (osd.1) 317 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:20.248231+0000 osd.1 (osd.1) 318 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:20.536535+0000 mgr.y (mgr.14544) 630 : cluster [DBG] pgmap v1150: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.4 KiB/s rd, 147 B/s wr, 1 op/s 2024-02-13T00:53:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:20.734155+0000 mon.a (mon.0) 3611 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "42"}]': finished 2024-02-13T00:53:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:20.734211+0000 mon.a (mon.0) 3612 : cluster [DBG] osdmap e758: 8 total, 8 up, 4 in 2024-02-13T00:53:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:20.248081+0000 osd.1 (osd.1) 317 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:20.248231+0000 osd.1 (osd.1) 318 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:20.536535+0000 mgr.y (mgr.14544) 630 : cluster [DBG] pgmap v1150: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.4 KiB/s rd, 147 B/s wr, 1 op/s 2024-02-13T00:53:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:20.734155+0000 mon.a (mon.0) 3611 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "42"}]': finished 2024-02-13T00:53:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:20.734211+0000 mon.a (mon.0) 3612 : cluster [DBG] osdmap e758: 8 total, 8 up, 4 in 2024-02-13T00:53:22.435 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:22.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:22.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:22.539 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:22.568 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:22.620 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:22.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:22.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:22.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.724+0000 7f2659ffb700 1 -- 172.21.15.162:0/597229652 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v759) v1 ==== 87+0+0 (secure 0 0 0) 0x7f2650050340 con 0x7f265c0f73a0 2024-02-13T00:53:22.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 10 librados: watch_flush enter 2024-02-13T00:53:22.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 10 librados: watch_flush exit 2024-02-13T00:53:22.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 20 client.28086.objecter shutdown clearing up homeless session... 2024-02-13T00:53:22.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 10 client.28086.objecter successfully canceled tick 2024-02-13T00:53:22.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 -- 172.21.15.162:0/597229652 >> v2:172.21.15.162:6800/2866534053 conn(0x7f264c062c90 msgr2=0x7f264c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:22.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 --2- 172.21.15.162:0/597229652 >> v2:172.21.15.162:6800/2866534053 conn(0x7f264c062c90 0x7f264c065150 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 rx=0x7f2648004ee0 tx=0x7f264800b040).stop 2024-02-13T00:53:22.732 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 -- 172.21.15.162:0/597229652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f265c0f73a0 msgr2=0x7f265c0f8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:22.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 --2- 172.21.15.162:0/597229652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f265c0f73a0 0x7f265c0f8930 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 rx=0x7f2650004920 tx=0x7f2650009cb0).stop 2024-02-13T00:53:22.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 -- 172.21.15.162:0/597229652 shutdown_connections 2024-02-13T00:53:22.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 --2- 172.21.15.162:0/597229652 >> v2:172.21.15.162:6800/2866534053 conn(0x7f264c062c90 0x7f264c065150 unknown :-1 s=CLOSED pgs=3036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:22.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 --2- 172.21.15.162:0/597229652 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f265c0fd490 0x7f265c0f9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:22.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 --2- 172.21.15.162:0/597229652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f265c0f73a0 0x7f265c0f8930 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:22.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 --2- 172.21.15.162:0/597229652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f265c0fcb30 0x7f265c0f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:22.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 -- 172.21.15.162:0/597229652 >> 172.21.15.162:0/597229652 conn(0x7f265c06eb90 msgr2=0x7f265c070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:22.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 -- 172.21.15.162:0/597229652 shutdown_connections 2024-02-13T00:53:22.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 -- 172.21.15.162:0/597229652 wait complete. 2024-02-13T00:53:22.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:22.727+0000 7f2662ff5700 1 librados: shutdown 2024-02-13T00:53:22.733 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:53:22.739 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:53:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:53:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:53:22.739 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.25162.0:4733 191.6 e335/323 191:602f83fe:::foo:head v 335'11, mlcod=333'9)", 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:44:50.443481+0000", 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 512.29163503100006, 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.024653034000000001, 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443481+0000", 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443481+0000", 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443481+0000", 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7215999999999999e-05 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443498+0000", 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1439999999999997e-06 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443505+0000", 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.409e-06 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443511+0000", 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039672500000000001 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443908+0000", 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9888999999999999e-05 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.443928+0000", 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.024168541000000002 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.468097+0000", 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7477e-05 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:44:50.468134+0000", 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.024653034000000001 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.1.0:1092 243.f e676/675 243:f2a1c75f:test-rados-api-smithi162-84057-119::foo:head v 676'1, mlcod=0'0)", 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:41.582582+0000", 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 101.152534258, 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003452798, 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.744 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582582+0000", 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582582+0000", 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582582+0000", 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1119999999999998e-06 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582586+0000", 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7359999999999998e-06 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582591+0000", 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.165e-06 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582599+0000", 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032407200000000001 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582923+0000", 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4011000000000003e-05 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582967+0000", 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030438359999999999 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:22.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.586010+0000", 2024-02-13T00:53:22.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4156999999999999e-05 2024-02-13T00:53:22.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.586035+0000", 2024-02-13T00:53:22.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003452798 2024-02-13T00:53:22.748 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.748 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.748 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.27753.0:1101 243.f e676/675 243:f2a1c75f:test-rados-api-smithi162-84057-119::foo:head v 676'2, mlcod=0'0)", 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:41.582604+0000", 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 101.152512347, 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034472249999999999, 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582604+0000", 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582604+0000", 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582603+0000", 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4819999999999997e-06 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582611+0000", 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6959999999999996e-06 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582616+0000", 2024-02-13T00:53:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8519999999999997e-06 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582619+0000", 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049247699999999995 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.583112+0000", 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6790999999999999e-05 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.583129+0000", 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029083709999999999 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.586037+0000", 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3876999999999999e-05 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.586051+0000", 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034472249999999999 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.752 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.14 epoch 677/675 rep_tid 1095 entries 677'15 (0'0) error 5:2b04a3e9:::notify.6:head by client.24329.0:2089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:44.375644+0000", 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 98.359472300999997, 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001229675, 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375644+0000", 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375644+0000", 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375643+0000", 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8799999999999997e-06 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375652+0000", 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2441999999999999e-05 2024-02-13T00:53:22.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375684+0000", 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9670000000000007e-06 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375693+0000", 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00038377499999999999 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376077+0000", 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00079641200000000003 2024-02-13T00:53:22.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376873+0000", 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001229675 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.e epoch 677/676 rep_tid 1907 entries 677'14 (0'0) error 5:7759931f:::notify.3:head by client.24329.0:2086 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:44.375693+0000", 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 98.359423015000004, 2024-02-13T00:53:22.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00107599, 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375693+0000", 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375693+0000", 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375692+0000", 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9560000000000003e-06 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375701+0000", 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4450999999999999e-05 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375716+0000", 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9150000000000002e-06 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375725+0000", 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050547700000000005 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376230+0000", 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00053894800000000003 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376769+0000", 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00107599 2024-02-13T00:53:22.759 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.9 epoch 677/676 rep_tid 759 entries 677'17 (0'0) error 5:93e5b521:::notify.7:head by client.24329.0:2090 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:44.375736+0000", 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 98.359380107000007, 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00092890799999999999, 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375736+0000", 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375736+0000", 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375735+0000", 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0960000000000003e-06 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375742+0000", 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7240000000000004e-06 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375752+0000", 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9700000000000002e-06 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375759+0000", 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018977600000000001 2024-02-13T00:53:22.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375949+0000", 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00071591199999999997 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376665+0000", 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00092890799999999999 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.763 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.d epoch 677/676 rep_tid 1908 entries 677'14 (0'0) error 5:b4812045:::notify.4:head by client.24329.0:2087 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:44.375740+0000", 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 98.359376405999996, 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00091101599999999995, 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375740+0000", 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375740+0000", 2024-02-13T00:53:22.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375739+0000", 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5940000000000001e-06 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375745+0000", 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5799999999999992e-06 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375753+0000", 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2639999999999997e-06 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375760+0000", 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9428e-05 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375829+0000", 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00082167500000000001 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376651+0000", 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00091101599999999995 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(4.4 epoch 677/675 rep_tid 1909 entries 677'66 (0'0) error 4:22d26bf9:::data_loggenerations_metadata:head by client.24329.0:2088 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:44.375773+0000", 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 98.359343092000003, 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011080160000000001, 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.767 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375773+0000", 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375773+0000", 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375773+0000", 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5809999999999996e-06 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375778+0000", 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6899999999999998e-06 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375787+0000", 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2740000000000004e-06 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375793+0000", 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00038068299999999999 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376174+0000", 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00070712300000000004 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376881+0000", 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011080160000000001 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.770 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s1 697/675 ECSubWrite(tid=779, reqid=osd.1.13:208, at_version=697'2, trim_to=0'0, roll_forward_to=0'0))", 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:04.576142+0000", 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 78.158974138999994, 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001836847, 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576142+0000", 2024-02-13T00:53:22.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576142+0000", 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576141+0000", 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3881000000000001e-05 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576165+0000", 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8133000000000001e-05 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576183+0000", 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6520000000000006e-06 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576190+0000", 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9348000000000001e-05 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576260+0000", 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.815e-06 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576270+0000", 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016926160000000001 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.577962+0000", 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6589e-05 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.577979+0000", 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001836847 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(234.3s1 698/675 ECSubRead(tid=785, to_read={234:c8594d0f:test-rados-api-smithi162-84057-123::foo:head=0,4096,0}, subchunks={234:c8594d0f:test-rados-api-smithi162-84057-123::foo:head=[0,1]}, attrs_to_read=))", 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:05.566388+0000", 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 77.168728436999999, 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020959070000000002, 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:22.775 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.566388+0000", 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.566388+0000", 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.566387+0000", 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8300000000000008e-06 2024-02-13T00:53:22.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.566397+0000", 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1646e-05 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.566409+0000", 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0070000000000003e-06 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.566418+0000", 2024-02-13T00:53:22.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001849334 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.568267+0000", 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021640599999999999 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.568484+0000", 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020959070000000002 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(234.3s1 698/675 ECSubRead(tid=786, to_read={234:c8594d0f:test-rados-api-smithi162-84057-123::foo:9=0,4096,0}, subchunks={234:c8594d0f:test-rados-api-smithi162-84057-123::foo:9=[0,1]}, attrs_to_read=))", 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:05.578253+0000", 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 77.156862646999997, 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039044069999999999, 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578253+0000", 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578253+0000", 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578253+0000", 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.465e-06 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578262+0000", 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0709000000000001e-05 2024-02-13T00:53:22.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578283+0000", 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0839999999999998e-06 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578292+0000", 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6171999999999997e-05 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578348+0000", 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038093770000000001 2024-02-13T00:53:22.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.582158+0000", 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039044069999999999 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.1.0:1106 245.13 e700/691 245:c8594d0f:test-rados-api-smithi162-84057-123::foo:9 v 700'11, mlcod=697'8)", 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:07.565418+0000", 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 75.169698069000006, 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00082021100000000005, 2024-02-13T00:53:22.782 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.565418+0000", 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.565418+0000", 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.565417+0000", 2024-02-13T00:53:22.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5659999999999999e-05 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.565432+0000", 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7000000000000003e-06 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.565442+0000", 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.858e-06 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.565450+0000", 2024-02-13T00:53:22.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9712000000000001e-05 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.565500+0000", 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2152000000000001e-05 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.565522+0000", 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067894299999999999 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.566201+0000", 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7468000000000001e-05 2024-02-13T00:53:22.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.566238+0000", 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00082021100000000005 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s1 716/675 ECSubWrite(tid=792, reqid=osd.6.45:144, at_version=716'9, trim_to=0'0, roll_forward_to=700'8))", 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:23.591714+0000", 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 59.143402524000003, 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0065409589999999998, 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:53:22.786 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.591714+0000", 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.591714+0000", 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.591713+0000", 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014055579999999999 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.593119+0000", 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4260999999999999e-05 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.593133+0000", 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1175000000000001e-05 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.593144+0000", 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056209999999999995 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.593706+0000", 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1963999999999999e-05 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.593718+0000", 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045167849999999997 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.598235+0000", 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9380000000000001e-05 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.598254+0000", 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0065409589999999998 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(2.b REQUEST e717)", 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:25.143446+0000", 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 57.591670121999996, 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027255790000000001, 2024-02-13T00:53:22.790 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143446+0000", 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143446+0000", 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967296 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143446+0000", 2024-02-13T00:53:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0850000000000001e-06 2024-02-13T00:53:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143449+0000", 2024-02-13T00:53:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1029000000000001e-05 2024-02-13T00:53:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143460+0000", 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.337e-06 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143468+0000", 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2531000000000002e-05 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143521+0000", 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8110000000000003e-06 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143529+0000", 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025945439999999998 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.146123+0000", 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6027000000000001e-05 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.146139+0000", 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2398999999999999e-05 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.146172+0000", 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027255790000000001 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.27753.0:2002 249.6 e735/729 249:602f83fe:::foo:head v 735'3, mlcod=735'1)", 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:44.566424+0000", 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.168692126000003, 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00088128000000000002, 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.795 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:44.566424+0000", 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:44.566424+0000", 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999971 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:44.566421+0000", 2024-02-13T00:53:22.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7607999999999994e-05 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:44.566489+0000", 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.809e-05 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:44.566517+0000", 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.378e-05 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:44.566531+0000", 2024-02-13T00:53:22.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015067999999999999 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:44.566681+0000", 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5751999999999998e-05 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:44.566727+0000", 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00054336599999999999 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:44.567270+0000", 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4737000000000003e-05 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:44.567305+0000", 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00088128000000000002 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.27753.0:2004 249.6 e735/729 249:602f83fe:::foo:head v 735'5, mlcod=735'2)", 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:45.570845+0000", 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.164271032000002, 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00076311499999999997, 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:22.799 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:45.570845+0000", 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:45.570845+0000", 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:45.570844+0000", 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8781000000000001e-05 2024-02-13T00:53:22.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:45.570873+0000", 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5119999999999997e-06 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:45.570883+0000", 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0549999999999996e-06 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:45.570889+0000", 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024622799999999997 2024-02-13T00:53:22.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:45.571135+0000", 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9309000000000001e-05 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:45.571154+0000", 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042461000000000002 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:45.571579+0000", 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9410000000000001e-05 2024-02-13T00:53:22.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:45.571608+0000", 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00076311499999999997 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.27753.0:2008 249.6 e735/729 249:602f83fe:::foo:head v 735'7, mlcod=735'5)", 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:47.574926+0000", 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 35.160189598000002, 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00079194900000000002, 2024-02-13T00:53:22.803 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:47.574926+0000", 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:47.574926+0000", 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:47.574924+0000", 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0445999999999997e-05 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:47.574985+0000", 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5925999999999999e-05 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:47.575011+0000", 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2476e-05 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:47.575023+0000", 2024-02-13T00:53:22.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013043700000000001 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:47.575154+0000", 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3187e-05 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:47.575177+0000", 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050983699999999996 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:47.575687+0000", 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1659999999999998e-05 2024-02-13T00:53:22.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:47.575718+0000", 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00079194900000000002 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.27753.0:2014 249.6 e735/729 249:602f83fe:::foo:head v 735'11, mlcod=735'9)", 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:50.582517+0000", 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 32.152598924999999, 2024-02-13T00:53:22.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.023893649999999999, 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582517+0000", 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582517+0000", 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999928 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582510+0000", 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8892999999999999e-05 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582539+0000", 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2396999999999999e-05 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582561+0000", 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7968999999999999e-05 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582589+0000", 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6641999999999997e-05 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582646+0000", 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1164000000000002e-05 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582677+0000", 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.023693683 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.606371+0000", 2024-02-13T00:53:22.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9833999999999998e-05 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.606411+0000", 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.023893649999999999 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.27753.0:2018 249.6 e736/729 249:602f83fe:::foo:head v 736'14, mlcod=735'12)", 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:52.639275+0000", 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 30.095840891999998, 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00075351699999999999, 2024-02-13T00:53:22.811 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:52.639275+0000", 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:52.639275+0000", 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999933 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:52.639268+0000", 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9989999999999999e-05 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:52.639298+0000", 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0179000000000001e-05 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:52.639319+0000", 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9540000000000004e-06 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:52.639326+0000", 2024-02-13T00:53:22.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8857999999999997e-05 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:52.639395+0000", 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1721999999999999e-05 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:52.639427+0000", 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056488399999999998 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:52.639992+0000", 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6732e-05 2024-02-13T00:53:22.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:52.640029+0000", 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00075351699999999999 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.0.0:1950 249.6 e737/729 249:60000000:.ceph-internal::hit_set_249.6_archive_2024-02-13 00%3a52%3a51.636360Z_2024-02-13 00%3a52%3a54.643103Z:head v 737'22, has_updated_hit_set_history, mlcod=736'17)", 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:55.565530+0000", 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 27.169586177999999, 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00076383899999999999, 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:22.815 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:55.565530+0000", 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:55.565530+0000", 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:53:22.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:55.565530+0000", 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.054e-06 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:55.565536+0000", 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.093e-05 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:55.565547+0000", 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5529999999999998e-06 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:53:22.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:55.565553+0000", 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4752e-05 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:55.565608+0000", 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0959e-05 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:55.565639+0000", 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00060899299999999995 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:55.566248+0000", 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5911999999999997e-05 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:55.566294+0000", 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00076383899999999999 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:53:22.819 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:53:22.820 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:21.865567+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? 172.21.15.162:0/597229652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:53:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:22.458864+0000 osd.6 (osd.6) 245 : cluster [DBG] 4.5 deep-scrub starts 2024-02-13T00:53:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:22.462510+0000 osd.6 (osd.6) 246 : cluster [DBG] 4.5 deep-scrub ok 2024-02-13T00:53:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:21.865567+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? 172.21.15.162:0/597229652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:53:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:22.458864+0000 osd.6 (osd.6) 245 : cluster [DBG] 4.5 deep-scrub starts 2024-02-13T00:53:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:22.462510+0000 osd.6 (osd.6) 246 : cluster [DBG] 4.5 deep-scrub ok 2024-02-13T00:53:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:21.865567+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? 172.21.15.162:0/597229652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:53:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:22.458864+0000 osd.6 (osd.6) 245 : cluster [DBG] 4.5 deep-scrub starts 2024-02-13T00:53:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:22.462510+0000 osd.6 (osd.6) 246 : cluster [DBG] 4.5 deep-scrub ok 2024-02-13T00:53:23.114 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:23.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:23.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:23.247 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:23.345 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:23.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:23.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:23.475 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:23.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-13T00:53:23.759 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:53:23.973 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:22.536940+0000 mgr.y (mgr.14544) 631 : cluster [DBG] pgmap v1152: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 129 B/s wr, 1 op/s 2024-02-13T00:53:23.973 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:22.717308+0000 mon.a (mon.0) 3614 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:53:23.974 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:22.724668+0000 mon.a (mon.0) 3615 : audit [INF] from='client.? 172.21.15.162:0/597229652' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:53:23.974 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:22.724737+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e759: 8 total, 8 up, 4 in 2024-02-13T00:53:23.974 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:22.826174+0000 osd.4 (osd.4) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:23.974 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:22.826495+0000 osd.4 (osd.4) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:22.536940+0000 mgr.y (mgr.14544) 631 : cluster [DBG] pgmap v1152: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 129 B/s wr, 1 op/s 2024-02-13T00:53:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:22.717308+0000 mon.a (mon.0) 3614 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:53:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:22.724668+0000 mon.a (mon.0) 3615 : audit [INF] from='client.? 172.21.15.162:0/597229652' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:53:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:22.724737+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e759: 8 total, 8 up, 4 in 2024-02-13T00:53:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:22.826174+0000 osd.4 (osd.4) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:24.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:22.826495+0000 osd.4 (osd.4) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:24.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:22.536940+0000 mgr.y (mgr.14544) 631 : cluster [DBG] pgmap v1152: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 129 B/s wr, 1 op/s 2024-02-13T00:53:24.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:22.717308+0000 mon.a (mon.0) 3614 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:53:24.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:22.724668+0000 mon.a (mon.0) 3615 : audit [INF] from='client.? 172.21.15.162:0/597229652' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:53:24.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:22.724737+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e759: 8 total, 8 up, 4 in 2024-02-13T00:53:24.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:22.826174+0000 osd.4 (osd.4) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:24.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:22.826495+0000 osd.4 (osd.4) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:24.425 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:24.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:24.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:24.556 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:24.655 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:24.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:24.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:24.792 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:24.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 -- 172.21.15.162:0/212622875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f105c0f6b00 msgr2=0x7f105c0f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:24.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 --2- 172.21.15.162:0/212622875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f105c0f6b00 0x7f105c0f6ee0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7f10580047e0 tx=0x7f1058009cb0).stop 2024-02-13T00:53:24.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 -- 172.21.15.162:0/212622875 shutdown_connections 2024-02-13T00:53:24.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 --2- 172.21.15.162:0/212622875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f105c104660 0x7f105c106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:24.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 --2- 172.21.15.162:0/212622875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f105c0f6b00 0x7f105c0f6ee0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:24.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 --2- 172.21.15.162:0/212622875 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f105c0f73d0 0x7f105c104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:24.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 -- 172.21.15.162:0/212622875 >> 172.21.15.162:0/212622875 conn(0x7f105c06eb90 msgr2=0x7f105c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:24.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 -- 172.21.15.162:0/212622875 shutdown_connections 2024-02-13T00:53:24.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 -- 172.21.15.162:0/212622875 wait complete. 2024-02-13T00:53:24.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 librados: starting msgr at 2024-02-13T00:53:24.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 librados: starting objecter 2024-02-13T00:53:24.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 Processor -- start 2024-02-13T00:53:24.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 -- start start 2024-02-13T00:53:24.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 librados: setting wanted keys 2024-02-13T00:53:24.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.879+0000 7f10646aa700 1 librados: calling monclient init 2024-02-13T00:53:24.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10646aa700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f105c0f6b00 0x7f105c110390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10646aa700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f105c0f73d0 0x7f105c110880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10646aa700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f105c104660 0x7f105c10a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10646aa700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c10aa60 con 0x7f105c104660 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10646aa700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c10abd0 con 0x7f105c0f73d0 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10646aa700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105c10ad40 con 0x7f105c0f6b00 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f1062c47700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f105c104660 0x7f105c10a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f1062446700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f105c0f6b00 0x7f105c110390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f1062c47700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f105c104660 0x7f105c10a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:42444/0 (socket says 172.21.15.162:42444) 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f1062c47700 1 -- 172.21.15.162:0/3085455857 learned_addr learned my addr 172.21.15.162:0/3085455857 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f1061c45700 1 --2- 172.21.15.162:0/3085455857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f105c0f73d0 0x7f105c110880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f1062c47700 1 -- 172.21.15.162:0/3085455857 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f105c0f6b00 msgr2=0x7f105c110390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f1062c47700 1 --2- 172.21.15.162:0/3085455857 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f105c0f6b00 0x7f105c110390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f1062c47700 1 -- 172.21.15.162:0/3085455857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f105c0f73d0 msgr2=0x7f105c110880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:24.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f1062c47700 1 --2- 172.21.15.162:0/3085455857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f105c0f73d0 0x7f105c110880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:24.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f1062c47700 1 -- 172.21.15.162:0/3085455857 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c005040 con 0x7f105c104660 2024-02-13T00:53:24.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f1062446700 1 --2- 172.21.15.162:0/3085455857 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f105c0f6b00 0x7f105c110390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:24.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f1062c47700 1 --2- 172.21.15.162:0/3085455857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f105c104660 0x7f105c10a570 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7f105400ec20 tx=0x7f105400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:24.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10537fe700 10 client.?.objecter ms_handle_connect 0x7f105c104660 2024-02-13T00:53:24.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10537fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:24.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054009950 con 0x7f105c104660 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f105400f040 con 0x7f105c104660 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1058005040 con 0x7f105c104660 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054007df0 con 0x7f105c104660 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.880+0000 7f10646aa700 10 client.28092.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.881+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f105c10b350 con 0x7f105c104660 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.881+0000 7f10646aa700 1 librados: init done 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.881+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f105c109370 con 0x7f105c104660 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.882+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f10540092e0 con 0x7f105c104660 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.882+0000 7f10537fe700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1048062c90 0x7f1048065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f1062446700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1048062c90 0x7f1048065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(760..760 src has 449..760) v4 ==== 8656+0+0 (secure 0 0 0) 0x7f105407d120 con 0x7f105c104660 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 osd_map(760..760 src has 449..760) v4 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10537fe700 3 client.28092.objecter handle_osd_map got epochs [760,760] > 0 2024-02-13T00:53:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10537fe700 3 client.28092.objecter handle_osd_map decoding full epoch 760 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10537fe700 20 client.28092.objecter dump_active .. 0 homeless 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v760) v1 ==== 54+0+16 (secure 0 0 0) 0x7f1054084070 con 0x7f105c104660 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f1062446700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1048062c90 0x7f1048065150 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 rx=0x7f1058009da0 tx=0x7f1058007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10537fe700 10 client.28092.objecter ms_handle_connect 0x7f1048062c90 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10646aa700 10 client.28092.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 1 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6801/2103743338 conn(0x7f105c028590 0x7f105c0fe130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c10ba50 osd=0 3 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c10ba50 osd=0 3 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c10ba50 osd=0 3 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 1 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c10ba50 osd=0 3 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 0 1 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c10ba50 osd=0 4 2024-02-13T00:53:24.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10646aa700 10 client.28092.objecter _send_command 1 2024-02-13T00:53:24.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f105c05a0e0 con 0x7f105c028590 2024-02-13T00:53:24.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f1061c45700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6801/2103743338 conn(0x7f105c028590 0x7f105c0fe130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:24.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f10537fe700 10 client.28092.objecter ms_handle_connect 0x7f105c028590 2024-02-13T00:53:24.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.883+0000 7f1061c45700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6801/2103743338 conn(0x7f105c028590 0x7f105c0fe130 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:24.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.884+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f105c05a0e0 con 0x7f105c028590 2024-02-13T00:53:24.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.884+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:53:24.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.884+0000 7f10537fe700 10 client.28092.objecter _finish_command 1 = system:0 2024-02-13T00:53:24.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.884+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c10ba50 osd=0 4 2024-02-13T00:53:24.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.884+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 0 1 2024-02-13T00:53:24.891 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:24.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:24.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:24.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.900+0000 7f10646aa700 10 client.28092.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:24.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.900+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 2 2024-02-13T00:53:24.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.901+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c10ba50 osd=0 3 2024-02-13T00:53:24.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.901+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c10ba50 osd=0 3 2024-02-13T00:53:24.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.901+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c10ba50 osd=0 3 2024-02-13T00:53:24.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.901+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 2 2024-02-13T00:53:24.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.901+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c10ba50 osd=0 3 2024-02-13T00:53:24.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.901+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 0 2 2024-02-13T00:53:24.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.901+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c10ba50 osd=0 4 2024-02-13T00:53:24.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.901+0000 7f10646aa700 10 client.28092.objecter _send_command 2 2024-02-13T00:53:24.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.901+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f105c0fe620 con 0x7f105c028590 2024-02-13T00:53:24.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f105c0fe620 con 0x7f105c028590 2024-02-13T00:53:24.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:24.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10537fe700 10 client.28092.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:24.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c10ba50 osd=0 4 2024-02-13T00:53:24.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 0 2 2024-02-13T00:53:24.907 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:53:24.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10646aa700 10 client.28092.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:24.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 3 2024-02-13T00:53:24.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6805/150599780 conn(0x7f105c0597b0 0x7f105c11b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:24.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c059640 osd=1 3 2024-02-13T00:53:24.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c059640 osd=1 3 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c059640 osd=1 3 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 3 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c059640 osd=1 3 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 1 3 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c059640 osd=1 4 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10646aa700 10 client.28092.objecter _send_command 3 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.902+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f105c0280f0 con 0x7f105c0597b0 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.903+0000 7f1062446700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6805/150599780 conn(0x7f105c0597b0 0x7f105c11b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.903+0000 7f10537fe700 10 client.28092.objecter ms_handle_connect 0x7f105c0597b0 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.903+0000 7f1062446700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6805/150599780 conn(0x7f105c0597b0 0x7f105c11b220 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.904+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f105c0280f0 con 0x7f105c0597b0 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.904+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.904+0000 7f10537fe700 10 client.28092.objecter _finish_command 3 = system:0 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.904+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c059640 osd=1 4 2024-02-13T00:53:24.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.904+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 1 3 2024-02-13T00:53:24.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.919+0000 7f10646aa700 10 client.28092.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:24.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.919+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 4 2024-02-13T00:53:24.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.919+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c059640 osd=1 3 2024-02-13T00:53:24.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.919+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c059640 osd=1 3 2024-02-13T00:53:24.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.919+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c059640 osd=1 3 2024-02-13T00:53:24.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.919+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 4 2024-02-13T00:53:24.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.919+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c059640 osd=1 3 2024-02-13T00:53:24.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.919+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 1 4 2024-02-13T00:53:24.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.919+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c059640 osd=1 4 2024-02-13T00:53:24.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.919+0000 7f10646aa700 10 client.28092.objecter _send_command 4 2024-02-13T00:53:24.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.919+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f105c11b710 con 0x7f105c0597b0 2024-02-13T00:53:24.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f105c11b710 con 0x7f105c0597b0 2024-02-13T00:53:24.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:24.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10537fe700 10 client.28092.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:24.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c059640 osd=1 4 2024-02-13T00:53:24.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 1 4 2024-02-13T00:53:24.926 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:53:24.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10646aa700 10 client.28092.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:24.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 5 2024-02-13T00:53:24.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6809/3689719376 conn(0x7f105c0277c0 0x7f105c0ff4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:24.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c027650 osd=2 3 2024-02-13T00:53:24.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c027650 osd=2 3 2024-02-13T00:53:24.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c027650 osd=2 3 2024-02-13T00:53:24.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 5 2024-02-13T00:53:24.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c027650 osd=2 3 2024-02-13T00:53:24.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 2 5 2024-02-13T00:53:24.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c027650 osd=2 4 2024-02-13T00:53:24.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10646aa700 10 client.28092.objecter _send_command 5 2024-02-13T00:53:24.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f105c121200 con 0x7f105c0277c0 2024-02-13T00:53:24.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.921+0000 7f1061c45700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6809/3689719376 conn(0x7f105c0277c0 0x7f105c0ff4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:24.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.922+0000 7f10537fe700 10 client.28092.objecter ms_handle_connect 0x7f105c0277c0 2024-02-13T00:53:24.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.922+0000 7f1061c45700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6809/3689719376 conn(0x7f105c0277c0 0x7f105c0ff4d0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:24.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.922+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f105c121200 con 0x7f105c0277c0 2024-02-13T00:53:24.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.922+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:53:24.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.922+0000 7f10537fe700 10 client.28092.objecter _finish_command 5 = system:0 2024-02-13T00:53:24.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.922+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c027650 osd=2 4 2024-02-13T00:53:24.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.922+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 2 5 2024-02-13T00:53:24.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.937+0000 7f10646aa700 10 client.28092.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:24.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.937+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 6 2024-02-13T00:53:24.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.937+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c027650 osd=2 3 2024-02-13T00:53:24.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.937+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c027650 osd=2 3 2024-02-13T00:53:24.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.937+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c027650 osd=2 3 2024-02-13T00:53:24.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.937+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 6 2024-02-13T00:53:24.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.937+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c027650 osd=2 3 2024-02-13T00:53:24.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.937+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 2 6 2024-02-13T00:53:24.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.937+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c027650 osd=2 4 2024-02-13T00:53:24.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.937+0000 7f10646aa700 10 client.28092.objecter _send_command 6 2024-02-13T00:53:24.942 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.937+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f105c0fea30 con 0x7f105c0277c0 2024-02-13T00:53:24.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f105c0fea30 con 0x7f105c0277c0 2024-02-13T00:53:24.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:24.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10537fe700 10 client.28092.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:24.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c027650 osd=2 4 2024-02-13T00:53:24.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 2 6 2024-02-13T00:53:24.944 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:53:24.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10646aa700 10 client.28092.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:24.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 7 2024-02-13T00:53:24.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6813/604971011 conn(0x7f105c11fc20 0x7f105c120060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c11c080 osd=3 3 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11c080 osd=3 3 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c11c080 osd=3 3 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 7 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c11c080 osd=3 3 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 3 7 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11c080 osd=3 4 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10646aa700 10 client.28092.objecter _send_command 7 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f105c11c700 con 0x7f105c11fc20 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f1062c47700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6813/604971011 conn(0x7f105c11fc20 0x7f105c120060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f10537fe700 10 client.28092.objecter ms_handle_connect 0x7f105c11fc20 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.939+0000 7f1062c47700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6813/604971011 conn(0x7f105c11fc20 0x7f105c120060 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.940+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f105c11c700 con 0x7f105c11fc20 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.940+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:53:24.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.940+0000 7f10537fe700 10 client.28092.objecter _finish_command 7 = system:0 2024-02-13T00:53:24.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.940+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c11c080 osd=3 4 2024-02-13T00:53:24.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.940+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 3 7 2024-02-13T00:53:24.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.954+0000 7f10646aa700 10 client.28092.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:24.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.954+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 8 2024-02-13T00:53:24.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.954+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c11c080 osd=3 3 2024-02-13T00:53:24.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.954+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11c080 osd=3 3 2024-02-13T00:53:24.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.954+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c11c080 osd=3 3 2024-02-13T00:53:24.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.954+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 8 2024-02-13T00:53:24.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.954+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c11c080 osd=3 3 2024-02-13T00:53:24.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.954+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 3 8 2024-02-13T00:53:24.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.954+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11c080 osd=3 4 2024-02-13T00:53:24.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.954+0000 7f10646aa700 10 client.28092.objecter _send_command 8 2024-02-13T00:53:24.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.954+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f105c11c920 con 0x7f105c11fc20 2024-02-13T00:53:24.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f105c11c920 con 0x7f105c11fc20 2024-02-13T00:53:24.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:24.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10537fe700 10 client.28092.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:24.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c11c080 osd=3 4 2024-02-13T00:53:24.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 3 8 2024-02-13T00:53:24.961 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10646aa700 10 client.28092.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 9 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6800/3344263388 conn(0x7f105c123380 0x7f105c125770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c11c920 osd=4 3 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11c920 osd=4 3 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c11c920 osd=4 3 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 9 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c11c920 osd=4 3 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 4 9 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11c920 osd=4 4 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10646aa700 10 client.28092.objecter _send_command 9 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f105c11bdc0 con 0x7f105c123380 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.956+0000 7f1062446700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6800/3344263388 conn(0x7f105c123380 0x7f105c125770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.957+0000 7f10537fe700 10 client.28092.objecter ms_handle_connect 0x7f105c123380 2024-02-13T00:53:24.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.957+0000 7f1062446700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6800/3344263388 conn(0x7f105c123380 0x7f105c125770 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:24.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.957+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f105c11bdc0 con 0x7f105c123380 2024-02-13T00:53:24.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.957+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:53:24.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.957+0000 7f10537fe700 10 client.28092.objecter _finish_command 9 = system:0 2024-02-13T00:53:24.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.957+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c11c920 osd=4 4 2024-02-13T00:53:24.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.957+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 4 9 2024-02-13T00:53:24.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.972+0000 7f10646aa700 10 client.28092.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:24.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.972+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 10 2024-02-13T00:53:24.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.972+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c11c920 osd=4 3 2024-02-13T00:53:24.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.972+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11c920 osd=4 3 2024-02-13T00:53:24.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.972+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c11c920 osd=4 3 2024-02-13T00:53:24.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.972+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 10 2024-02-13T00:53:24.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.972+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c11c920 osd=4 3 2024-02-13T00:53:24.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.972+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 4 10 2024-02-13T00:53:24.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.972+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11c920 osd=4 4 2024-02-13T00:53:24.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.972+0000 7f10646aa700 10 client.28092.objecter _send_command 10 2024-02-13T00:53:24.977 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.972+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f105c11bdc0 con 0x7f105c123380 2024-02-13T00:53:24.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f105c11bdc0 con 0x7f105c123380 2024-02-13T00:53:24.978 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:24.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10537fe700 10 client.28092.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:24.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c11c920 osd=4 4 2024-02-13T00:53:24.979 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 4 10 2024-02-13T00:53:24.979 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:53:24.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10646aa700 10 client.28092.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:24.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 11 2024-02-13T00:53:24.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6804/2277917372 conn(0x7f105c126170 0x7f105c128560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:24.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c11bdc0 osd=5 3 2024-02-13T00:53:24.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11bdc0 osd=5 3 2024-02-13T00:53:24.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c11bdc0 osd=5 3 2024-02-13T00:53:24.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 11 2024-02-13T00:53:24.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c11bdc0 osd=5 3 2024-02-13T00:53:24.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 5 11 2024-02-13T00:53:24.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11bdc0 osd=5 4 2024-02-13T00:53:24.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10646aa700 10 client.28092.objecter _send_command 11 2024-02-13T00:53:24.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f105c12b320 con 0x7f105c126170 2024-02-13T00:53:24.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.974+0000 7f1061c45700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6804/2277917372 conn(0x7f105c126170 0x7f105c128560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:24.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.975+0000 7f10537fe700 10 client.28092.objecter ms_handle_connect 0x7f105c126170 2024-02-13T00:53:24.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.975+0000 7f1061c45700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6804/2277917372 conn(0x7f105c126170 0x7f105c128560 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:24.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.975+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f105c12b320 con 0x7f105c126170 2024-02-13T00:53:24.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.975+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:53:24.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.975+0000 7f10537fe700 10 client.28092.objecter _finish_command 11 = system:0 2024-02-13T00:53:24.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.975+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c11bdc0 osd=5 4 2024-02-13T00:53:24.981 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.975+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 5 11 2024-02-13T00:53:24.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.990+0000 7f10646aa700 10 client.28092.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:24.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.990+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 12 2024-02-13T00:53:24.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.990+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c11bdc0 osd=5 3 2024-02-13T00:53:24.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.990+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11bdc0 osd=5 3 2024-02-13T00:53:24.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.990+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c11bdc0 osd=5 3 2024-02-13T00:53:24.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.990+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 12 2024-02-13T00:53:24.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.990+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c11bdc0 osd=5 3 2024-02-13T00:53:24.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.990+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 5 12 2024-02-13T00:53:24.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.990+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11bdc0 osd=5 4 2024-02-13T00:53:24.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.990+0000 7f10646aa700 10 client.28092.objecter _send_command 12 2024-02-13T00:53:24.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.990+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f105c12b320 con 0x7f105c126170 2024-02-13T00:53:24.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:24.998 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:53:24.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.992+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f105c12b320 con 0x7f105c126170 2024-02-13T00:53:24.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.992+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.992+0000 7f10537fe700 10 client.28092.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.992+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c11bdc0 osd=5 4 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.992+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 5 12 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10646aa700 10 client.28092.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 13 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6808/1506640341 conn(0x7f105c12a640 0x7f105c12aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c12b320 osd=6 3 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c12b320 osd=6 3 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c12b320 osd=6 3 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 13 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c12b320 osd=6 3 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 6 13 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c12b320 osd=6 4 2024-02-13T00:53:24.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10646aa700 10 client.28092.objecter _send_command 13 2024-02-13T00:53:25.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f105c12b110 con 0x7f105c12a640 2024-02-13T00:53:25.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f1062c47700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6808/1506640341 conn(0x7f105c12a640 0x7f105c12aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:25.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f1062c47700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6808/1506640341 conn(0x7f105c12a640 0x7f105c12aaa0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:25.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.993+0000 7f10537fe700 10 client.28092.objecter ms_handle_connect 0x7f105c12a640 2024-02-13T00:53:25.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.994+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f105c12b110 con 0x7f105c12a640 2024-02-13T00:53:25.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.994+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:53:25.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.994+0000 7f10537fe700 10 client.28092.objecter _finish_command 13 = system:0 2024-02-13T00:53:25.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.994+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c12b320 osd=6 4 2024-02-13T00:53:25.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:24.994+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 6 13 2024-02-13T00:53:25.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:23.727312+0000 mon.a (mon.0) 3617 : cluster [DBG] osdmap e760: 8 total, 8 up, 4 in 2024-02-13T00:53:25.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.011+0000 7f10646aa700 10 client.28092.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:25.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.011+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 14 2024-02-13T00:53:25.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.011+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c12b320 osd=6 3 2024-02-13T00:53:25.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.011+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c12b320 osd=6 3 2024-02-13T00:53:25.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.011+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c12b320 osd=6 3 2024-02-13T00:53:25.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.011+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 14 2024-02-13T00:53:25.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.011+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c12b320 osd=6 3 2024-02-13T00:53:25.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.011+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 6 14 2024-02-13T00:53:25.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.011+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c12b320 osd=6 4 2024-02-13T00:53:25.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.011+0000 7f10646aa700 10 client.28092.objecter _send_command 14 2024-02-13T00:53:25.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.011+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f105c12b110 con 0x7f105c12a640 2024-02-13T00:53:25.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f105c12b110 con 0x7f105c12a640 2024-02-13T00:53:25.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:25.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10537fe700 10 client.28092.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:25.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c12b320 osd=6 4 2024-02-13T00:53:25.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 6 14 2024-02-13T00:53:25.018 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:53:25.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10646aa700 10 client.28092.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:25.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 15 2024-02-13T00:53:25.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6812/1964192772 conn(0x7f105c12db20 0x7f105c12df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:25.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c12b110 osd=7 3 2024-02-13T00:53:25.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c12b110 osd=7 3 2024-02-13T00:53:25.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c12b110 osd=7 3 2024-02-13T00:53:25.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 15 2024-02-13T00:53:25.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c12b110 osd=7 3 2024-02-13T00:53:25.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 7 15 2024-02-13T00:53:25.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c12b110 osd=7 4 2024-02-13T00:53:25.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10646aa700 10 client.28092.objecter _send_command 15 2024-02-13T00:53:25.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f105c131910 con 0x7f105c12db20 2024-02-13T00:53:25.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.013+0000 7f1062446700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6812/1964192772 conn(0x7f105c12db20 0x7f105c12df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:25.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.014+0000 7f10537fe700 10 client.28092.objecter ms_handle_connect 0x7f105c12db20 2024-02-13T00:53:25.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.014+0000 7f1062446700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6812/1964192772 conn(0x7f105c12db20 0x7f105c12df80 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:25.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.014+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f105c131910 con 0x7f105c12db20 2024-02-13T00:53:25.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.014+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:53:25.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.014+0000 7f10537fe700 10 client.28092.objecter _finish_command 15 = system:0 2024-02-13T00:53:25.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.014+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c12b110 osd=7 4 2024-02-13T00:53:25.020 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.014+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 7 15 2024-02-13T00:53:25.023 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:23.727312+0000 mon.a (mon.0) 3617 : cluster [DBG] osdmap e760: 8 total, 8 up, 4 in 2024-02-13T00:53:25.023 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:23.727312+0000 mon.a (mon.0) 3617 : cluster [DBG] osdmap e760: 8 total, 8 up, 4 in 2024-02-13T00:53:25.026 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:25.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.030+0000 7f10646aa700 10 client.28092.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:25.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.030+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign -1 16 2024-02-13T00:53:25.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.030+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c12b110 osd=7 3 2024-02-13T00:53:25.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.030+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c12b110 osd=7 3 2024-02-13T00:53:25.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.030+0000 7f10646aa700 20 client.28092.objecter _get_session s=0x7f105c12b110 osd=7 3 2024-02-13T00:53:25.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.030+0000 7f10646aa700 15 client.28092.objecter _session_command_op_remove -1 16 2024-02-13T00:53:25.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.030+0000 7f10646aa700 20 client.28092.objecter get_session s=0x7f105c12b110 osd=7 3 2024-02-13T00:53:25.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.030+0000 7f10646aa700 15 client.28092.objecter _session_command_op_assign 7 16 2024-02-13T00:53:25.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.030+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c12b110 osd=7 4 2024-02-13T00:53:25.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.030+0000 7f10646aa700 10 client.28092.objecter _send_command 16 2024-02-13T00:53:25.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.030+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f105c128b20 con 0x7f105c12db20 2024-02-13T00:53:25.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10537fe700 1 -- 172.21.15.162:0/3085455857 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f105c128b20 con 0x7f105c12db20 2024-02-13T00:53:25.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10537fe700 10 client.28092.objecter ms_dispatch 0x7f105c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:25.037 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10537fe700 10 client.28092.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:25.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10537fe700 20 client.28092.objecter put_session s=0x7f105c12b110 osd=7 4 2024-02-13T00:53:25.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10537fe700 15 client.28092.objecter _session_command_op_remove 7 16 2024-02-13T00:53:25.038 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:53:25.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 10 librados: watch_flush enter 2024-02-13T00:53:25.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 10 librados: watch_flush exit 2024-02-13T00:53:25.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 10 client.28092.objecter close_session for osd.0 2024-02-13T00:53:25.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6801/2103743338 conn(0x7f105c028590 msgr2=0x7f105c0fe130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:25.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6801/2103743338 conn(0x7f105c028590 0x7f105c0fe130 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:25.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c10ba50 osd=0 1 2024-02-13T00:53:25.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 10 client.28092.objecter close_session for osd.1 2024-02-13T00:53:25.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6805/150599780 conn(0x7f105c0597b0 msgr2=0x7f105c11b220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:25.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6805/150599780 conn(0x7f105c0597b0 0x7f105c11b220 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c059640 osd=1 1 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 10 client.28092.objecter close_session for osd.2 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6809/3689719376 conn(0x7f105c0277c0 msgr2=0x7f105c0ff4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6809/3689719376 conn(0x7f105c0277c0 0x7f105c0ff4d0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c027650 osd=2 1 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 10 client.28092.objecter close_session for osd.3 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6813/604971011 conn(0x7f105c11fc20 msgr2=0x7f105c120060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6813/604971011 conn(0x7f105c11fc20 0x7f105c120060 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11c080 osd=3 1 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 10 client.28092.objecter close_session for osd.4 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6800/3344263388 conn(0x7f105c123380 msgr2=0x7f105c125770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6800/3344263388 conn(0x7f105c123380 0x7f105c125770 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11c920 osd=4 1 2024-02-13T00:53:25.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 10 client.28092.objecter close_session for osd.5 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6804/2277917372 conn(0x7f105c126170 msgr2=0x7f105c128560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6804/2277917372 conn(0x7f105c126170 0x7f105c128560 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c11bdc0 osd=5 1 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 10 client.28092.objecter close_session for osd.6 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6808/1506640341 conn(0x7f105c12a640 msgr2=0x7f105c12aaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6808/1506640341 conn(0x7f105c12a640 0x7f105c12aaa0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c12b320 osd=6 1 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 10 client.28092.objecter close_session for osd.7 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6812/1964192772 conn(0x7f105c12db20 msgr2=0x7f105c12df80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.181:6812/1964192772 conn(0x7f105c12db20 0x7f105c12df80 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 20 client.28092.objecter put_session s=0x7f105c12b110 osd=7 1 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 20 client.28092.objecter shutdown clearing up homeless session... 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f10646aa700 10 client.28092.objecter successfully canceled tick 2024-02-13T00:53:25.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f1062c47700 1 -- 172.21.15.162:0/3085455857 reap_dead start 2024-02-13T00:53:25.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f1062c47700 1 -- 172.21.15.162:0/3085455857 reap_dead start 2024-02-13T00:53:25.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f1062c47700 1 -- 172.21.15.162:0/3085455857 reap_dead start 2024-02-13T00:53:25.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f1062c47700 1 -- 172.21.15.162:0/3085455857 reap_dead start 2024-02-13T00:53:25.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.033+0000 7f1062c47700 1 -- 172.21.15.162:0/3085455857 reap_dead start 2024-02-13T00:53:25.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.034+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1048062c90 msgr2=0x7f1048065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:25.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.034+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1048062c90 0x7f1048065150 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 rx=0x7f1058009da0 tx=0x7f1058007f90).stop 2024-02-13T00:53:25.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.034+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f105c104660 msgr2=0x7f105c10a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:25.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.034+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f105c104660 0x7f105c10a570 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7f105400ec20 tx=0x7f105400b040).stop 2024-02-13T00:53:25.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.034+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 shutdown_connections 2024-02-13T00:53:25.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.034+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f105c104660 0x7f105c10a570 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:25.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.034+0000 7f10646aa700 1 --2- 172.21.15.162:0/3085455857 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1048062c90 0x7f1048065150 unknown :-1 s=CLOSED pgs=3037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:25.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.034+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 >> 172.21.15.162:0/3085455857 conn(0x7f105c06eb90 msgr2=0x7f105c06fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:25.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.034+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 shutdown_connections 2024-02-13T00:53:25.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.034+0000 7f10646aa700 1 -- 172.21.15.162:0/3085455857 wait complete. 2024-02-13T00:53:25.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:25.034+0000 7f10646aa700 1 librados: shutdown 2024-02-13T00:53:25.043 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:53:25.043 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:53:25.043 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:53:25.043 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:53:25.043 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:53:25.043 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:53:25.043 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:53:25.043 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:53:25.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:25.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:25.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:25.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:25.179 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:25.253 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:25.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:25.357 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:25.385 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:25.413 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:25.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:25.516 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:25.540 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:53:25.541 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:53:25.545 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:25.564 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:25.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:25.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:25.696 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:24.537520+0000 mgr.y (mgr.14544) 632 : cluster [DBG] pgmap v1155: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:53:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:24.814070+0000 mon.c (mon.2) 506 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:53:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:24.884331+0000 mon.a (mon.0) 3618 : audit [DBG] from='client.? 172.21.15.162:0/3085455857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:25.073493+0000 mon.a (mon.0) 3619 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:25.395854+0000 mon.c (mon.2) 507 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:53:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:25.396456+0000 mon.c (mon.2) 508 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:53:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:25.400950+0000 mon.a (mon.0) 3620 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:25.571652+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e761: 8 total, 8 up, 4 in 2024-02-13T00:53:25.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:25.689579+0000 mon.a (mon.0) 3622 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:25.894 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:25.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:25.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:25.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:26.026 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:26.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:24.537520+0000 mgr.y (mgr.14544) 632 : cluster [DBG] pgmap v1155: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:53:26.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:24.814070+0000 mon.c (mon.2) 506 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:53:26.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:24.884331+0000 mon.a (mon.0) 3618 : audit [DBG] from='client.? 172.21.15.162:0/3085455857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:26.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:25.073493+0000 mon.a (mon.0) 3619 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:26.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:25.395854+0000 mon.c (mon.2) 507 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:53:26.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:25.396456+0000 mon.c (mon.2) 508 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:53:26.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:25.400950+0000 mon.a (mon.0) 3620 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:25.571652+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e761: 8 total, 8 up, 4 in 2024-02-13T00:53:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:25.689579+0000 mon.a (mon.0) 3622 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:26.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:24.537520+0000 mgr.y (mgr.14544) 632 : cluster [DBG] pgmap v1155: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:53:26.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:24.814070+0000 mon.c (mon.2) 506 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:53:26.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:24.884331+0000 mon.a (mon.0) 3618 : audit [DBG] from='client.? 172.21.15.162:0/3085455857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:26.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:25.073493+0000 mon.a (mon.0) 3619 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:26.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:25.395854+0000 mon.c (mon.2) 507 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:53:26.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:25.396456+0000 mon.c (mon.2) 508 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:53:26.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:25.400950+0000 mon.a (mon.0) 3620 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:26.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:25.571652+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e761: 8 total, 8 up, 4 in 2024-02-13T00:53:26.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:25.689579+0000 mon.a (mon.0) 3622 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:26.200 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:53:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:53:26.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-13T00:53:26.224 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:26.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:26.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:26.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:53:26.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:26.353 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:26.373 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:26.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:26.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:26.486 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:53:26.504 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:26.523 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:26.525 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:26.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:26.579 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:53:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:53:26] "GET /metrics HTTP/1.1" 200 190088 "" "Prometheus/2.33.4" 2024-02-13T00:53:26.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:26.652 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:26.671 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:26.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:26.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:26.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:26.781 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:25.948975+0000 mgr.y (mgr.14544) 633 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:26.781 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:26.578796+0000 mon.a (mon.0) 3623 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:53:26.781 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:26.578850+0000 mon.a (mon.0) 3624 : cluster [DBG] osdmap e762: 8 total, 8 up, 4 in 2024-02-13T00:53:26.781 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:26.579286+0000 mon.a (mon.0) 3625 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134"}]: dispatch 2024-02-13T00:53:26.805 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:26.826 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:26.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:26.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:26.953 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:25.948975+0000 mgr.y (mgr.14544) 633 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:26.954 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:26.578796+0000 mon.a (mon.0) 3623 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:53:26.954 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:26.578850+0000 mon.a (mon.0) 3624 : cluster [DBG] osdmap e762: 8 total, 8 up, 4 in 2024-02-13T00:53:26.954 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:26.579286+0000 mon.a (mon.0) 3625 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134"}]: dispatch 2024-02-13T00:53:26.954 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:25.948975+0000 mgr.y (mgr.14544) 633 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:26.954 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:26.578796+0000 mon.a (mon.0) 3623 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:53:26.954 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:26.578850+0000 mon.a (mon.0) 3624 : cluster [DBG] osdmap e762: 8 total, 8 up, 4 in 2024-02-13T00:53:26.954 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:26.579286+0000 mon.a (mon.0) 3625 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134"}]: dispatch 2024-02-13T00:53:26.956 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:27.080 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:27.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:27.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:27.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:27.194 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 6, 7] out_osds: [3, 2, 5, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-13T00:53:27.194 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:53:27.194 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-13T00:53:27.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-13T00:53:27.210 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:27.409 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:27.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:27.473 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:27.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:27.540 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:27.788 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:27.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:27.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:27.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:53:27.919 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:26.537985+0000 mgr.y (mgr.14544) 634 : cluster [DBG] pgmap v1157: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 881 B/s rd, 0 op/s 2024-02-13T00:53:27.919 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:27.576754+0000 mon.a (mon.0) 3626 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:53:27.919 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:27.580680+0000 mon.a (mon.0) 3627 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134"}]': finished 2024-02-13T00:53:27.919 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:27.580758+0000 mon.a (mon.0) 3628 : cluster [DBG] osdmap e763: 8 total, 8 up, 4 in 2024-02-13T00:53:27.919 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:27.595847+0000 mon.a (mon.0) 3629 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:27.919 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:27.596154+0000 mon.a (mon.0) 3630 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134"}]: dispatch 2024-02-13T00:53:27.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:26.537985+0000 mgr.y (mgr.14544) 634 : cluster [DBG] pgmap v1157: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 881 B/s rd, 0 op/s 2024-02-13T00:53:27.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:27.576754+0000 mon.a (mon.0) 3626 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:53:27.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:27.580680+0000 mon.a (mon.0) 3627 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134"}]': finished 2024-02-13T00:53:27.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:27.580758+0000 mon.a (mon.0) 3628 : cluster [DBG] osdmap e763: 8 total, 8 up, 4 in 2024-02-13T00:53:27.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:27.595847+0000 mon.a (mon.0) 3629 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:27.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:27.596154+0000 mon.a (mon.0) 3630 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134"}]: dispatch 2024-02-13T00:53:27.921 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:28.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:28.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:28.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:26.537985+0000 mgr.y (mgr.14544) 634 : cluster [DBG] pgmap v1157: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 881 B/s rd, 0 op/s 2024-02-13T00:53:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:27.576754+0000 mon.a (mon.0) 3626 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:53:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:27.580680+0000 mon.a (mon.0) 3627 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134"}]': finished 2024-02-13T00:53:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:27.580758+0000 mon.a (mon.0) 3628 : cluster [DBG] osdmap e763: 8 total, 8 up, 4 in 2024-02-13T00:53:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:27.595847+0000 mon.a (mon.0) 3629 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:27.596154+0000 mon.a (mon.0) 3630 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-134"}]: dispatch 2024-02-13T00:53:28.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:28.148 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:28.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:28.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:28.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:28.413 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:53:28.413 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:53:28.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:28.478 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:28.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.539+0000 7fb13decf700 1 -- 172.21.15.162:0/4273848020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1380fcb30 msgr2=0x7fb1380fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:28.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.539+0000 7fb13decf700 1 --2- 172.21.15.162:0/4273848020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1380fcb30 0x7fb1380fcf10 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 rx=0x7fb128004870 tx=0x7fb128009cb0).stop 2024-02-13T00:53:28.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.539+0000 7fb13decf700 1 -- 172.21.15.162:0/4273848020 shutdown_connections 2024-02-13T00:53:28.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.539+0000 7fb13decf700 1 --2- 172.21.15.162:0/4273848020 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1380f73a0 0x7fb1381039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.539+0000 7fb13decf700 1 --2- 172.21.15.162:0/4273848020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1380fcb30 0x7fb1380fcf10 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.539+0000 7fb13decf700 1 --2- 172.21.15.162:0/4273848020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1380fd490 0x7fb1380f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.539+0000 7fb13decf700 1 -- 172.21.15.162:0/4273848020 >> 172.21.15.162:0/4273848020 conn(0x7fb13806eb90 msgr2=0x7fb1380f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.539+0000 7fb13decf700 1 -- 172.21.15.162:0/4273848020 shutdown_connections 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.539+0000 7fb13decf700 1 -- 172.21.15.162:0/4273848020 wait complete. 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.540+0000 7fb13decf700 1 librados: starting msgr at 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.540+0000 7fb13decf700 1 librados: starting objecter 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.540+0000 7fb13decf700 1 Processor -- start 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.540+0000 7fb13decf700 1 -- start start 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.540+0000 7fb13decf700 1 librados: setting wanted keys 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.540+0000 7fb13decf700 1 librados: calling monclient init 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.540+0000 7fb13decf700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1380f73a0 0x7fb13810d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.540+0000 7fb13decf700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1380fcb30 0x7fb13810d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:28.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.540+0000 7fb13decf700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1380fd490 0x7fb1381072e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.540+0000 7fb13decf700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1381077d0 con 0x7fb1380f73a0 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.540+0000 7fb13decf700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb138107940 con 0x7fb1380fcb30 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb13decf700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb138107b00 con 0x7fb1380fd490 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb1377fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1380f73a0 0x7fb13810d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb137fff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1380fd490 0x7fb1381072e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb1377fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1380f73a0 0x7fb13810d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:42472/0 (socket says 172.21.15.162:42472) 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb1377fe700 1 -- 172.21.15.162:0/3931237217 learned_addr learned my addr 172.21.15.162:0/3931237217 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb136ffd700 1 --2- 172.21.15.162:0/3931237217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1380fcb30 0x7fb13810d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb1377fe700 1 -- 172.21.15.162:0/3931237217 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1380fd490 msgr2=0x7fb1381072e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb1377fe700 1 --2- 172.21.15.162:0/3931237217 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1380fd490 0x7fb1381072e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb1377fe700 1 -- 172.21.15.162:0/3931237217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1380fcb30 msgr2=0x7fb13810d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb1377fe700 1 --2- 172.21.15.162:0/3931237217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1380fcb30 0x7fb13810d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb1377fe700 1 -- 172.21.15.162:0/3931237217 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1300097c0 con 0x7fb1380f73a0 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb137fff700 1 --2- 172.21.15.162:0/3931237217 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1380fd490 0x7fb1381072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb136ffd700 1 --2- 172.21.15.162:0/3931237217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1380fcb30 0x7fb13810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:53:28.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb1377fe700 1 --2- 172.21.15.162:0/3931237217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1380f73a0 0x7fb13810d0f0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 rx=0x7fb128009eb0 tx=0x7fb12800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:28.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb134ff9700 10 client.?.objecter ms_handle_connect 0x7fb1380f73a0 2024-02-13T00:53:28.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb134ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:28.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb134ff9700 1 -- 172.21.15.162:0/3931237217 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12800e070 con 0x7fb1380f73a0 2024-02-13T00:53:28.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb134ff9700 1 -- 172.21.15.162:0/3931237217 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb128013da0 con 0x7fb1380f73a0 2024-02-13T00:53:28.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb134ff9700 1 -- 172.21.15.162:0/3931237217 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128019070 con 0x7fb1380f73a0 2024-02-13T00:53:28.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb13decf700 1 -- 172.21.15.162:0/3931237217 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb128005040 con 0x7fb1380f73a0 2024-02-13T00:53:28.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb13decf700 10 client.28098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:28.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb13decf700 1 -- 172.21.15.162:0/3931237217 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb138107fa0 con 0x7fb1380f73a0 2024-02-13T00:53:28.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.541+0000 7fb13decf700 1 librados: init done 2024-02-13T00:53:28.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.542+0000 7fb13decf700 1 -- 172.21.15.162:0/3931237217 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1381081c0 con 0x7fb1380f73a0 2024-02-13T00:53:28.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.542+0000 7fb134ff9700 1 -- 172.21.15.162:0/3931237217 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fb12801e420 con 0x7fb1380f73a0 2024-02-13T00:53:28.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.543+0000 7fb134ff9700 1 --2- 172.21.15.162:0/3931237217 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb120062c90 0x7fb120065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:28.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.543+0000 7fb136ffd700 1 --2- 172.21.15.162:0/3931237217 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb120062c90 0x7fb120065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:28.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.543+0000 7fb134ff9700 1 -- 172.21.15.162:0/3931237217 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(763..763 src has 449..763) v4 ==== 8648+0+0 (secure 0 0 0) 0x7fb12807e030 con 0x7fb1380f73a0 2024-02-13T00:53:28.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.543+0000 7fb134ff9700 10 client.28098.objecter ms_dispatch 0x7fb138001550 osd_map(763..763 src has 449..763) v4 2024-02-13T00:53:28.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.543+0000 7fb134ff9700 3 client.28098.objecter handle_osd_map got epochs [763,763] > 0 2024-02-13T00:53:28.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.543+0000 7fb134ff9700 3 client.28098.objecter handle_osd_map decoding full epoch 763 2024-02-13T00:53:28.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.543+0000 7fb134ff9700 20 client.28098.objecter dump_active .. 0 homeless 2024-02-13T00:53:28.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.543+0000 7fb134ff9700 10 client.28098.objecter ms_handle_connect 0x7fb120062c90 2024-02-13T00:53:28.552 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.543+0000 7fb136ffd700 1 --2- 172.21.15.162:0/3931237217 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb120062c90 0x7fb120065150 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 rx=0x7fb138108a30 tx=0x7fb12c008040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:28.553 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:28.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:28.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.550+0000 7fb134ff9700 1 -- 172.21.15.162:0/3931237217 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb1280512b0 con 0x7fb1380f73a0 2024-02-13T00:53:28.594 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:28.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:28.687 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:28.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:28.756+0000 7fb13decf700 1 -- 172.21.15.162:0/3931237217 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fb1381087a0 con 0x7fb1380f73a0 2024-02-13T00:53:28.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:53:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:53:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:53:28.903 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:28.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:28.942 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:27.971272+0000 osd.2 (osd.2) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:28.943 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:27.971465+0000 osd.2 (osd.2) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:28.943 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:28.584859+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e764: 8 total, 8 up, 4 in 2024-02-13T00:53:28.943 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:27.971272+0000 osd.2 (osd.2) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:28.943 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:27.971465+0000 osd.2 (osd.2) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:28.943 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:28.584859+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e764: 8 total, 8 up, 4 in 2024-02-13T00:53:29.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:29.034 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:29.083 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:27.971272+0000 osd.2 (osd.2) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:29.083 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:27.971465+0000 osd.2 (osd.2) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:29.083 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:28.584859+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e764: 8 total, 8 up, 4 in 2024-02-13T00:53:29.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-13T00:53:29.369 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:53:29.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.583+0000 7fb134ff9700 1 -- 172.21.15.162:0/3931237217 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v765) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb128023030 con 0x7fb1380f73a0 2024-02-13T00:53:29.591 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.586+0000 7fb13decf700 10 librados: watch_flush enter 2024-02-13T00:53:29.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.587+0000 7fb13decf700 10 librados: watch_flush exit 2024-02-13T00:53:29.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.587+0000 7fb13decf700 20 client.28098.objecter shutdown clearing up homeless session... 2024-02-13T00:53:29.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.587+0000 7fb13decf700 10 client.28098.objecter successfully canceled tick 2024-02-13T00:53:29.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.587+0000 7fb13decf700 1 -- 172.21.15.162:0/3931237217 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb120062c90 msgr2=0x7fb120065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:29.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.587+0000 7fb13decf700 1 --2- 172.21.15.162:0/3931237217 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb120062c90 0x7fb120065150 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 rx=0x7fb138108a30 tx=0x7fb12c008040).stop 2024-02-13T00:53:29.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.587+0000 7fb13decf700 1 -- 172.21.15.162:0/3931237217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1380f73a0 msgr2=0x7fb13810d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:29.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.587+0000 7fb13decf700 1 --2- 172.21.15.162:0/3931237217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1380f73a0 0x7fb13810d0f0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 rx=0x7fb128009eb0 tx=0x7fb12800b040).stop 2024-02-13T00:53:29.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.588+0000 7fb13decf700 1 -- 172.21.15.162:0/3931237217 shutdown_connections 2024-02-13T00:53:29.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.588+0000 7fb13decf700 1 --2- 172.21.15.162:0/3931237217 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb120062c90 0x7fb120065150 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:29.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.588+0000 7fb13decf700 1 --2- 172.21.15.162:0/3931237217 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb1380fd490 0x7fb1381072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:29.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.588+0000 7fb13decf700 1 --2- 172.21.15.162:0/3931237217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1380f73a0 0x7fb13810d0f0 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:29.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.588+0000 7fb13decf700 1 --2- 172.21.15.162:0/3931237217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1380fcb30 0x7fb13810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:29.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.588+0000 7fb13decf700 1 -- 172.21.15.162:0/3931237217 >> 172.21.15.162:0/3931237217 conn(0x7fb13806eb90 msgr2=0x7fb1380fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:29.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.589+0000 7fb13decf700 1 -- 172.21.15.162:0/3931237217 shutdown_connections 2024-02-13T00:53:29.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.589+0000 7fb13decf700 1 -- 172.21.15.162:0/3931237217 wait complete. 2024-02-13T00:53:29.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:29.589+0000 7fb13decf700 1 librados: shutdown 2024-02-13T00:53:29.598 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:53:29.884 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:29.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:27.700200+0000 osd.6 (osd.6) 247 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:29.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:27.700419+0000 osd.6 (osd.6) 248 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:29.886 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:28.538349+0000 mgr.y (mgr.14544) 635 : cluster [DBG] pgmap v1160: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:53:29.887 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:28.838779+0000 mon.a (mon.0) 3632 : audit [INF] from='client.? 172.21.15.162:0/3931237217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:29.887 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:29.579988+0000 mon.a (mon.0) 3633 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:53:29.887 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:29.584037+0000 mon.a (mon.0) 3634 : audit [INF] from='client.? 172.21.15.162:0/3931237217' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:53:29.887 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:29.584100+0000 mon.a (mon.0) 3635 : cluster [DBG] osdmap e765: 8 total, 8 up, 4 in 2024-02-13T00:53:29.887 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:29.588132+0000 mon.a (mon.0) 3636 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:53:29.887 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:27.700200+0000 osd.6 (osd.6) 247 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:29.887 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:27.700419+0000 osd.6 (osd.6) 248 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:29.887 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:28.538349+0000 mgr.y (mgr.14544) 635 : cluster [DBG] pgmap v1160: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:53:29.888 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:28.838779+0000 mon.a (mon.0) 3632 : audit [INF] from='client.? 172.21.15.162:0/3931237217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:29.888 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:29.579988+0000 mon.a (mon.0) 3633 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:53:29.888 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:29.584037+0000 mon.a (mon.0) 3634 : audit [INF] from='client.? 172.21.15.162:0/3931237217' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:53:29.888 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:29.584100+0000 mon.a (mon.0) 3635 : cluster [DBG] osdmap e765: 8 total, 8 up, 4 in 2024-02-13T00:53:29.888 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:29.588132+0000 mon.a (mon.0) 3636 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:53:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:29.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:29.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:30.017 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:27.700200+0000 osd.6 (osd.6) 247 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:27.700419+0000 osd.6 (osd.6) 248 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:28.538349+0000 mgr.y (mgr.14544) 635 : cluster [DBG] pgmap v1160: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:53:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:28.838779+0000 mon.a (mon.0) 3632 : audit [INF] from='client.? 172.21.15.162:0/3931237217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:29.579988+0000 mon.a (mon.0) 3633 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:53:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:29.584037+0000 mon.a (mon.0) 3634 : audit [INF] from='client.? 172.21.15.162:0/3931237217' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:53:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:29.584100+0000 mon.a (mon.0) 3635 : cluster [DBG] osdmap e765: 8 total, 8 up, 4 in 2024-02-13T00:53:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:29.588132+0000 mon.a (mon.0) 3636 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:53:30.971 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:29.473173+0000 osd.5 (osd.5) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:30.971 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:29.473348+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:30.971 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:30.588968+0000 mon.a (mon.0) 3637 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:53:30.972 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:30.589064+0000 mon.a (mon.0) 3638 : cluster [DBG] osdmap e766: 8 total, 8 up, 4 in 2024-02-13T00:53:30.972 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:30.665466+0000 mon.a (mon.0) 3639 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:53:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:29.473173+0000 osd.5 (osd.5) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:29.473348+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:30.588968+0000 mon.a (mon.0) 3637 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:53:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:30.589064+0000 mon.a (mon.0) 3638 : cluster [DBG] osdmap e766: 8 total, 8 up, 4 in 2024-02-13T00:53:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:30.665466+0000 mon.a (mon.0) 3639 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:53:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:29.473173+0000 osd.5 (osd.5) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:29.473348+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:30.588968+0000 mon.a (mon.0) 3637 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:53:31.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:30.589064+0000 mon.a (mon.0) 3638 : cluster [DBG] osdmap e766: 8 total, 8 up, 4 in 2024-02-13T00:53:31.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:30.665466+0000 mon.a (mon.0) 3639 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:53:31.225 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-02-13T00:53:31.225 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:53:31.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.797+0000 7fc3bc966700 1 -- 172.21.15.162:0/2931700436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fd490 msgr2=0x7fc3b80f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.797+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2931700436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fd490 0x7fc3b80f6cc0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 rx=0x7fc3a8004870 tx=0x7fc3a8009cb0).stop 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.797+0000 7fc3bc966700 1 -- 172.21.15.162:0/2931700436 shutdown_connections 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.797+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2931700436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3b80f73a0 0x7fc3b81039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.797+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2931700436 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80fcb30 0x7fc3b80fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.797+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2931700436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fd490 0x7fc3b80f6cc0 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.797+0000 7fc3bc966700 1 -- 172.21.15.162:0/2931700436 >> 172.21.15.162:0/2931700436 conn(0x7fc3b806eb90 msgr2=0x7fc3b80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.797+0000 7fc3bc966700 1 -- 172.21.15.162:0/2931700436 shutdown_connections 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.797+0000 7fc3bc966700 1 -- 172.21.15.162:0/2931700436 wait complete. 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3bc966700 1 librados: starting msgr at 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3bc966700 1 librados: starting objecter 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3bc966700 1 Processor -- start 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3bc966700 1 -- start start 2024-02-13T00:53:31.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3bc966700 1 librados: setting wanted keys 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3bc966700 1 librados: calling monclient init 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3bc966700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3b80f73a0 0x7fc3b8109210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3bc966700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fcb30 0x7fc3b8109700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3bc966700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80fd490 0x7fc3b8109d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3bc966700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b8114440 con 0x7fc3b80fcb30 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3bc966700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b81145b0 con 0x7fc3b80f73a0 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3bc966700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b8114770 con 0x7fc3b80fd490 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3b5d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fcb30 0x7fc3b8109700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3b6d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80fd490 0x7fc3b8109d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3b5d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fcb30 0x7fc3b8109700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:42496/0 (socket says 172.21.15.162:42496) 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3b6d9d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80fd490 0x7fc3b8109d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:52422/0 (socket says 172.21.15.162:52422) 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3b5d9b700 1 -- 172.21.15.162:0/2076715840 learned_addr learned my addr 172.21.15.162:0/2076715840 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.798+0000 7fc3b659c700 1 --2- 172.21.15.162:0/2076715840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3b80f73a0 0x7fc3b8109210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:31.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3b6d9d700 1 -- 172.21.15.162:0/2076715840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3b80f73a0 msgr2=0x7fc3b8109210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3b6d9d700 1 --2- 172.21.15.162:0/2076715840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3b80f73a0 0x7fc3b8109210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3b6d9d700 1 -- 172.21.15.162:0/2076715840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fcb30 msgr2=0x7fc3b8109700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3b6d9d700 1 --2- 172.21.15.162:0/2076715840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fcb30 0x7fc3b8109700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3b6d9d700 1 -- 172.21.15.162:0/2076715840 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac005040 con 0x7fc3b80fd490 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3b5d9b700 1 --2- 172.21.15.162:0/2076715840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3b80fcb30 0x7fc3b8109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3b659c700 1 --2- 172.21.15.162:0/2076715840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3b80f73a0 0x7fc3b8109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3b6d9d700 1 --2- 172.21.15.162:0/2076715840 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80fd490 0x7fc3b8109d30 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 rx=0x7fc3b000eb70 tx=0x7fc3b000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3a77fe700 10 client.?.objecter ms_handle_connect 0x7fc3b80fd490 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3a77fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0009870 con 0x7fc3b80fd490 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3b000f040 con 0x7fc3b80fd490 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a8005040 con 0x7fc3b80fd490 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0007e40 con 0x7fc3b80fd490 2024-02-13T00:53:31.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3bc966700 10 client.38201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b8114dd0 con 0x7fc3b80fd490 2024-02-13T00:53:31.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.799+0000 7fc3bc966700 1 librados: init done 2024-02-13T00:53:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.800+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc3b8105fe0 con 0x7fc3b80fd490 2024-02-13T00:53:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.800+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc3b0004190 con 0x7fc3b80fd490 2024-02-13T00:53:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3a77fe700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc3a0062c90 0x7fc3a0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:31.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(767..767 src has 449..767) v4 ==== 8656+0+0 (secure 0 0 0) 0x7fc3b007d110 con 0x7fc3b80fd490 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 osd_map(767..767 src has 449..767) v4 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3a77fe700 3 client.38201.objecter handle_osd_map got epochs [767,767] > 0 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3a77fe700 3 client.38201.objecter handle_osd_map decoding full epoch 767 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3b659c700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc3a0062c90 0x7fc3a0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3a77fe700 20 client.38201.objecter dump_active .. 0 homeless 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v767) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc3b0084070 con 0x7fc3b80fd490 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3a77fe700 10 client.38201.objecter ms_handle_connect 0x7fc3a0062c90 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3b659c700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc3a0062c90 0x7fc3a0065150 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 rx=0x7fc3ac007fd0 tx=0x7fc3ac009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3bc966700 10 client.38201.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 1 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc3b8028590 0x7fc3b80fd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8061550 osd=0 3 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8061550 osd=0 3 2024-02-13T00:53:31.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8061550 osd=0 3 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 1 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b8061550 osd=0 3 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 0 1 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8061550 osd=0 4 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3bc966700 10 client.38201.objecter _send_command 1 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b805a0e0 con 0x7fc3b8028590 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.801+0000 7fc3b5d9b700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc3b8028590 0x7fc3b80fd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.802+0000 7fc3a77fe700 10 client.38201.objecter ms_handle_connect 0x7fc3b8028590 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.802+0000 7fc3b5d9b700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc3b8028590 0x7fc3b80fd890 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.802+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b805a0e0 con 0x7fc3b8028590 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.802+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.802+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 1 = system:0 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.802+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b8061550 osd=0 4 2024-02-13T00:53:31.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.802+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 0 1 2024-02-13T00:53:31.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.817+0000 7fc3bc966700 10 client.38201.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:31.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.817+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 2 2024-02-13T00:53:31.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.817+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8061550 osd=0 3 2024-02-13T00:53:31.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.817+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8061550 osd=0 3 2024-02-13T00:53:31.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.817+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8061550 osd=0 3 2024-02-13T00:53:31.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.817+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 2 2024-02-13T00:53:31.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.817+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b8061550 osd=0 3 2024-02-13T00:53:31.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.817+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 0 2 2024-02-13T00:53:31.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.817+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8061550 osd=0 4 2024-02-13T00:53:31.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.817+0000 7fc3bc966700 10 client.38201.objecter _send_command 2 2024-02-13T00:53:31.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.817+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b80fdd80 con 0x7fc3b8028590 2024-02-13T00:53:31.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b80fdd80 con 0x7fc3b8028590 2024-02-13T00:53:31.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:31.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b8061550 osd=0 4 2024-02-13T00:53:31.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 0 2 2024-02-13T00:53:31.822 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:53:31.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3bc966700 10 client.38201.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:31.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 3 2024-02-13T00:53:31.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6805/150599780 conn(0x7fc3b8027be0 0x7fc3b8027fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:31.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b80ff070 osd=1 3 2024-02-13T00:53:31.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b80ff070 osd=1 3 2024-02-13T00:53:31.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b80ff070 osd=1 3 2024-02-13T00:53:31.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 3 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b80ff070 osd=1 3 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 1 3 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b80ff070 osd=1 4 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3bc966700 10 client.38201.objecter _send_command 3 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.818+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b811b770 con 0x7fc3b8027be0 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.819+0000 7fc3b659c700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6805/150599780 conn(0x7fc3b8027be0 0x7fc3b8027fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.819+0000 7fc3a77fe700 10 client.38201.objecter ms_handle_connect 0x7fc3b8027be0 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.819+0000 7fc3b659c700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6805/150599780 conn(0x7fc3b8027be0 0x7fc3b8027fc0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.819+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b811b770 con 0x7fc3b8027be0 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.819+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.819+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 3 = system:0 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.819+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b80ff070 osd=1 4 2024-02-13T00:53:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.819+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 1 3 2024-02-13T00:53:31.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.834+0000 7fc3bc966700 10 client.38201.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:31.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.834+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 4 2024-02-13T00:53:31.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.834+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b80ff070 osd=1 3 2024-02-13T00:53:31.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.834+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b80ff070 osd=1 3 2024-02-13T00:53:31.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.834+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b80ff070 osd=1 3 2024-02-13T00:53:31.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.834+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 4 2024-02-13T00:53:31.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.834+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b80ff070 osd=1 3 2024-02-13T00:53:31.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.834+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 1 4 2024-02-13T00:53:31.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.834+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b80ff070 osd=1 4 2024-02-13T00:53:31.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.834+0000 7fc3bc966700 10 client.38201.objecter _send_command 4 2024-02-13T00:53:31.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.834+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b8027650 con 0x7fc3b8027be0 2024-02-13T00:53:31.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b8027650 con 0x7fc3b8027be0 2024-02-13T00:53:31.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:31.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b80ff070 osd=1 4 2024-02-13T00:53:31.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 1 4 2024-02-13T00:53:31.840 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:53:31.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3bc966700 10 client.38201.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:31.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 5 2024-02-13T00:53:31.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc3b80febd0 0x7fc3b811a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:31.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8027a60 osd=2 3 2024-02-13T00:53:31.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8027a60 osd=2 3 2024-02-13T00:53:31.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8027a60 osd=2 3 2024-02-13T00:53:31.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 5 2024-02-13T00:53:31.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b8027a60 osd=2 3 2024-02-13T00:53:31.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 2 5 2024-02-13T00:53:31.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8027a60 osd=2 4 2024-02-13T00:53:31.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3bc966700 10 client.38201.objecter _send_command 5 2024-02-13T00:53:31.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b811a680 con 0x7fc3b80febd0 2024-02-13T00:53:31.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3b5d9b700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc3b80febd0 0x7fc3b811a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:31.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3a77fe700 10 client.38201.objecter ms_handle_connect 0x7fc3b80febd0 2024-02-13T00:53:31.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.836+0000 7fc3b5d9b700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc3b80febd0 0x7fc3b811a190 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:31.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.837+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b811a680 con 0x7fc3b80febd0 2024-02-13T00:53:31.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.837+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:53:31.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.837+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 5 = system:0 2024-02-13T00:53:31.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.837+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b8027a60 osd=2 4 2024-02-13T00:53:31.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.837+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 2 5 2024-02-13T00:53:31.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.851+0000 7fc3bc966700 10 client.38201.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:31.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.851+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 6 2024-02-13T00:53:31.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.851+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8027a60 osd=2 3 2024-02-13T00:53:31.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.851+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8027a60 osd=2 3 2024-02-13T00:53:31.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.851+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8027a60 osd=2 3 2024-02-13T00:53:31.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.851+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 6 2024-02-13T00:53:31.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.851+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b8027a60 osd=2 3 2024-02-13T00:53:31.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.851+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 2 6 2024-02-13T00:53:31.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.851+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8027a60 osd=2 4 2024-02-13T00:53:31.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.851+0000 7fc3bc966700 10 client.38201.objecter _send_command 6 2024-02-13T00:53:31.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.851+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b80fe130 con 0x7fc3b80febd0 2024-02-13T00:53:31.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b80fe130 con 0x7fc3b80febd0 2024-02-13T00:53:31.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:31.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b8027a60 osd=2 4 2024-02-13T00:53:31.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 2 6 2024-02-13T00:53:31.857 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:53:31.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3bc966700 10 client.38201.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:31.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 7 2024-02-13T00:53:31.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6813/604971011 conn(0x7fc3b811d8f0 0x7fc3b811fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:31.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b811b020 osd=3 3 2024-02-13T00:53:31.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b811b020 osd=3 3 2024-02-13T00:53:31.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b811b020 osd=3 3 2024-02-13T00:53:31.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 7 2024-02-13T00:53:31.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b811b020 osd=3 3 2024-02-13T00:53:31.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 3 7 2024-02-13T00:53:31.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b811b020 osd=3 4 2024-02-13T00:53:31.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3bc966700 10 client.38201.objecter _send_command 7 2024-02-13T00:53:31.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b8122930 con 0x7fc3b811d8f0 2024-02-13T00:53:31.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3b6d9d700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6813/604971011 conn(0x7fc3b811d8f0 0x7fc3b811fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:31.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3a77fe700 10 client.38201.objecter ms_handle_connect 0x7fc3b811d8f0 2024-02-13T00:53:31.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.853+0000 7fc3b6d9d700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6813/604971011 conn(0x7fc3b811d8f0 0x7fc3b811fce0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:31.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.854+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b8122930 con 0x7fc3b811d8f0 2024-02-13T00:53:31.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.854+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:53:31.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.854+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 7 = system:0 2024-02-13T00:53:31.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.854+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b811b020 osd=3 4 2024-02-13T00:53:31.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.854+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 3 7 2024-02-13T00:53:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.869+0000 7fc3bc966700 10 client.38201.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.869+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 8 2024-02-13T00:53:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.869+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b811b020 osd=3 3 2024-02-13T00:53:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.869+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b811b020 osd=3 3 2024-02-13T00:53:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.869+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b811b020 osd=3 3 2024-02-13T00:53:31.872 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.869+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 8 2024-02-13T00:53:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.869+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b811b020 osd=3 3 2024-02-13T00:53:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.869+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 3 8 2024-02-13T00:53:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.869+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b811b020 osd=3 4 2024-02-13T00:53:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.869+0000 7fc3bc966700 10 client.38201.objecter _send_command 8 2024-02-13T00:53:31.873 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.869+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b811aa90 con 0x7fc3b811d8f0 2024-02-13T00:53:31.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b811aa90 con 0x7fc3b811d8f0 2024-02-13T00:53:31.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:31.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b811b020 osd=3 4 2024-02-13T00:53:31.874 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 3 8 2024-02-13T00:53:31.874 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:53:31.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3bc966700 10 client.38201.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:31.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 9 2024-02-13T00:53:31.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc3b811ac30 0x7fc3b8121350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:31.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b811aa90 osd=4 3 2024-02-13T00:53:31.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b811aa90 osd=4 3 2024-02-13T00:53:31.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b811aa90 osd=4 3 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 9 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b811aa90 osd=4 3 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 4 9 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b811aa90 osd=4 4 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3bc966700 10 client.38201.objecter _send_command 9 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.870+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b8119ed0 con 0x7fc3b811ac30 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.871+0000 7fc3b659c700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc3b811ac30 0x7fc3b8121350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.871+0000 7fc3b659c700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc3b811ac30 0x7fc3b8121350 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.871+0000 7fc3a77fe700 10 client.38201.objecter ms_handle_connect 0x7fc3b811ac30 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.872+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b8119ed0 con 0x7fc3b811ac30 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.872+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.872+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 9 = system:0 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.872+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b811aa90 osd=4 4 2024-02-13T00:53:31.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.872+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 4 9 2024-02-13T00:53:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.886+0000 7fc3bc966700 10 client.38201.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.886+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 10 2024-02-13T00:53:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.886+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b811aa90 osd=4 3 2024-02-13T00:53:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.886+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b811aa90 osd=4 3 2024-02-13T00:53:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.886+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b811aa90 osd=4 3 2024-02-13T00:53:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.886+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 10 2024-02-13T00:53:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.886+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b811aa90 osd=4 3 2024-02-13T00:53:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.886+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 4 10 2024-02-13T00:53:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.886+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b811aa90 osd=4 4 2024-02-13T00:53:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.886+0000 7fc3bc966700 10 client.38201.objecter _send_command 10 2024-02-13T00:53:31.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.886+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b8119ed0 con 0x7fc3b811ac30 2024-02-13T00:53:31.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b8119ed0 con 0x7fc3b811ac30 2024-02-13T00:53:31.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:31.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b811aa90 osd=4 4 2024-02-13T00:53:31.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 4 10 2024-02-13T00:53:31.892 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:53:31.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3bc966700 10 client.38201.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:31.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 11 2024-02-13T00:53:31.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc3b8121d50 0x7fc3b8122170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:31.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8119ed0 osd=5 3 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8119ed0 osd=5 3 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8119ed0 osd=5 3 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 11 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b8119ed0 osd=5 3 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 5 11 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8119ed0 osd=5 4 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3bc966700 10 client.38201.objecter _send_command 11 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.888+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b8122660 con 0x7fc3b8121d50 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.889+0000 7fc3b5d9b700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc3b8121d50 0x7fc3b8122170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.889+0000 7fc3b5d9b700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc3b8121d50 0x7fc3b8122170 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.889+0000 7fc3a77fe700 10 client.38201.objecter ms_handle_connect 0x7fc3b8121d50 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.890+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b8122660 con 0x7fc3b8121d50 2024-02-13T00:53:31.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.890+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:53:31.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.890+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 11 = system:0 2024-02-13T00:53:31.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.890+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b8119ed0 osd=5 4 2024-02-13T00:53:31.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.890+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 5 11 2024-02-13T00:53:31.900 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-13T00:53:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.904+0000 7fc3bc966700 10 client.38201.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.904+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 12 2024-02-13T00:53:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.904+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8119ed0 osd=5 3 2024-02-13T00:53:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.904+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8119ed0 osd=5 3 2024-02-13T00:53:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.904+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8119ed0 osd=5 3 2024-02-13T00:53:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.904+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 12 2024-02-13T00:53:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.904+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b8119ed0 osd=5 3 2024-02-13T00:53:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.904+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 5 12 2024-02-13T00:53:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.904+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8119ed0 osd=5 4 2024-02-13T00:53:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.904+0000 7fc3bc966700 10 client.38201.objecter _send_command 12 2024-02-13T00:53:31.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.904+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b8122660 con 0x7fc3b8121d50 2024-02-13T00:53:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b8122660 con 0x7fc3b8121d50 2024-02-13T00:53:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b8119ed0 osd=5 4 2024-02-13T00:53:31.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 5 12 2024-02-13T00:53:31.910 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:53:31.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3bc966700 10 client.38201.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:31.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 13 2024-02-13T00:53:31.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc3b8129420 0x7fc3b812b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:31.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8122660 osd=6 3 2024-02-13T00:53:31.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8122660 osd=6 3 2024-02-13T00:53:31.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8122660 osd=6 3 2024-02-13T00:53:31.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 13 2024-02-13T00:53:31.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b8122660 osd=6 3 2024-02-13T00:53:31.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 6 13 2024-02-13T00:53:31.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8122660 osd=6 4 2024-02-13T00:53:31.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3bc966700 10 client.38201.objecter _send_command 13 2024-02-13T00:53:31.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.906+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b812e5e0 con 0x7fc3b8129420 2024-02-13T00:53:31.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.907+0000 7fc3b6d9d700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc3b8129420 0x7fc3b812b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:31.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.907+0000 7fc3a77fe700 10 client.38201.objecter ms_handle_connect 0x7fc3b8129420 2024-02-13T00:53:31.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.907+0000 7fc3b6d9d700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc3b8129420 0x7fc3b812b810 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:31.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.908+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b812e5e0 con 0x7fc3b8129420 2024-02-13T00:53:31.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.908+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:53:31.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.908+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 13 = system:0 2024-02-13T00:53:31.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.908+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b8122660 osd=6 4 2024-02-13T00:53:31.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.908+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 6 13 2024-02-13T00:53:31.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.922+0000 7fc3bc966700 10 client.38201.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:31.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.922+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 14 2024-02-13T00:53:31.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.922+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8122660 osd=6 3 2024-02-13T00:53:31.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.922+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8122660 osd=6 3 2024-02-13T00:53:31.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.922+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b8122660 osd=6 3 2024-02-13T00:53:31.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.922+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 14 2024-02-13T00:53:31.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.922+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b8122660 osd=6 3 2024-02-13T00:53:31.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.922+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 6 14 2024-02-13T00:53:31.926 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.922+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8122660 osd=6 4 2024-02-13T00:53:31.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.922+0000 7fc3bc966700 10 client.38201.objecter _send_command 14 2024-02-13T00:53:31.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.922+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b812e5e0 con 0x7fc3b8129420 2024-02-13T00:53:31.927 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b812e5e0 con 0x7fc3b8129420 2024-02-13T00:53:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b8122660 osd=6 4 2024-02-13T00:53:31.928 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 6 14 2024-02-13T00:53:31.928 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:53:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3bc966700 10 client.38201.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 15 2024-02-13T00:53:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc3b812d8e0 0x7fc3b812dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b812e5e0 osd=7 3 2024-02-13T00:53:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b812e5e0 osd=7 3 2024-02-13T00:53:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b812e5e0 osd=7 3 2024-02-13T00:53:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 15 2024-02-13T00:53:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b812e5e0 osd=7 3 2024-02-13T00:53:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 7 15 2024-02-13T00:53:31.929 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b812e5e0 osd=7 4 2024-02-13T00:53:31.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3bc966700 10 client.38201.objecter _send_command 15 2024-02-13T00:53:31.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.924+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3b812e3d0 con 0x7fc3b812d8e0 2024-02-13T00:53:31.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.925+0000 7fc3b659c700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc3b812d8e0 0x7fc3b812dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:31.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.925+0000 7fc3a77fe700 10 client.38201.objecter ms_handle_connect 0x7fc3b812d8e0 2024-02-13T00:53:31.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.925+0000 7fc3b659c700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc3b812d8e0 0x7fc3b812dd60 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:31.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.926+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc3b812e3d0 con 0x7fc3b812d8e0 2024-02-13T00:53:31.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.926+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:53:31.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.926+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 15 = system:0 2024-02-13T00:53:31.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.926+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b812e5e0 osd=7 4 2024-02-13T00:53:31.930 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.926+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 7 15 2024-02-13T00:53:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.940+0000 7fc3bc966700 10 client.38201.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.940+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign -1 16 2024-02-13T00:53:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.940+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b812e5e0 osd=7 3 2024-02-13T00:53:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.940+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b812e5e0 osd=7 3 2024-02-13T00:53:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.940+0000 7fc3bc966700 20 client.38201.objecter _get_session s=0x7fc3b812e5e0 osd=7 3 2024-02-13T00:53:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.940+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_remove -1 16 2024-02-13T00:53:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.940+0000 7fc3bc966700 20 client.38201.objecter get_session s=0x7fc3b812e5e0 osd=7 3 2024-02-13T00:53:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.940+0000 7fc3bc966700 15 client.38201.objecter _session_command_op_assign 7 16 2024-02-13T00:53:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.940+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b812e5e0 osd=7 4 2024-02-13T00:53:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.940+0000 7fc3bc966700 10 client.38201.objecter _send_command 16 2024-02-13T00:53:31.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.940+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc3b812e3d0 con 0x7fc3b812d8e0 2024-02-13T00:53:31.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3a77fe700 1 -- 172.21.15.162:0/2076715840 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc3b812e3d0 con 0x7fc3b812d8e0 2024-02-13T00:53:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3a77fe700 10 client.38201.objecter ms_dispatch 0x7fc3b8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3a77fe700 10 client.38201.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3a77fe700 20 client.38201.objecter put_session s=0x7fc3b812e5e0 osd=7 4 2024-02-13T00:53:31.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3a77fe700 15 client.38201.objecter _session_command_op_remove 7 16 2024-02-13T00:53:31.946 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:53:31.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 10 librados: watch_flush enter 2024-02-13T00:53:31.947 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 10 librados: watch_flush exit 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 10 client.38201.objecter close_session for osd.0 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc3b8028590 msgr2=0x7fc3b80fd890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc3b8028590 0x7fc3b80fd890 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8061550 osd=0 1 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 10 client.38201.objecter close_session for osd.1 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6805/150599780 conn(0x7fc3b8027be0 msgr2=0x7fc3b8027fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6805/150599780 conn(0x7fc3b8027be0 0x7fc3b8027fc0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b80ff070 osd=1 1 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 10 client.38201.objecter close_session for osd.2 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc3b80febd0 msgr2=0x7fc3b811a190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc3b80febd0 0x7fc3b811a190 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8027a60 osd=2 1 2024-02-13T00:53:31.948 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 10 client.38201.objecter close_session for osd.3 2024-02-13T00:53:31.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6813/604971011 conn(0x7fc3b811d8f0 msgr2=0x7fc3b811fce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6813/604971011 conn(0x7fc3b811d8f0 0x7fc3b811fce0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b811b020 osd=3 1 2024-02-13T00:53:31.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 10 client.38201.objecter close_session for osd.4 2024-02-13T00:53:31.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc3b811ac30 msgr2=0x7fc3b8121350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc3b811ac30 0x7fc3b8121350 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b811aa90 osd=4 1 2024-02-13T00:53:31.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 10 client.38201.objecter close_session for osd.5 2024-02-13T00:53:31.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc3b8121d50 msgr2=0x7fc3b8122170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc3b8121d50 0x7fc3b8122170 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8119ed0 osd=5 1 2024-02-13T00:53:31.949 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 10 client.38201.objecter close_session for osd.6 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc3b8129420 msgr2=0x7fc3b812b810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc3b8129420 0x7fc3b812b810 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b8122660 osd=6 1 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 10 client.38201.objecter close_session for osd.7 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc3b812d8e0 msgr2=0x7fc3b812dd60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc3b812d8e0 0x7fc3b812dd60 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 20 client.38201.objecter put_session s=0x7fc3b812e5e0 osd=7 1 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 20 client.38201.objecter shutdown clearing up homeless session... 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.942+0000 7fc3bc966700 10 client.38201.objecter successfully canceled tick 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3b6d9d700 1 -- 172.21.15.162:0/2076715840 reap_dead start 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3b6d9d700 1 -- 172.21.15.162:0/2076715840 reap_dead start 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3b6d9d700 1 -- 172.21.15.162:0/2076715840 reap_dead start 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3b6d9d700 1 -- 172.21.15.162:0/2076715840 reap_dead start 2024-02-13T00:53:31.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3b6d9d700 1 -- 172.21.15.162:0/2076715840 reap_dead start 2024-02-13T00:53:31.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc3a0062c90 msgr2=0x7fc3a0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc3a0062c90 0x7fc3a0065150 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 rx=0x7fc3ac007fd0 tx=0x7fc3ac009cb0).stop 2024-02-13T00:53:31.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80fd490 msgr2=0x7fc3b8109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:31.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80fd490 0x7fc3b8109d30 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 rx=0x7fc3b000eb70 tx=0x7fc3b000b040).stop 2024-02-13T00:53:31.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 shutdown_connections 2024-02-13T00:53:31.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc3b80fd490 0x7fc3b8109d30 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3bc966700 1 --2- 172.21.15.162:0/2076715840 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc3a0062c90 0x7fc3a0065150 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:31.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 >> 172.21.15.162:0/2076715840 conn(0x7fc3b806eb90 msgr2=0x7fc3b80fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:31.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 shutdown_connections 2024-02-13T00:53:31.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3bc966700 1 -- 172.21.15.162:0/2076715840 wait complete. 2024-02-13T00:53:31.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:31.943+0000 7fc3bc966700 1 librados: shutdown 2024-02-13T00:53:31.952 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.952 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.952 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.952 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.952 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.952 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.952 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:53:31.952 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:53:32.205 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:30.539389+0000 mgr.y (mgr.14544) 636 : cluster [DBG] pgmap v1163: 204 pgs: 32 creating+peering, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:53:32.205 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:31.594599+0000 mon.a (mon.0) 3640 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:53:32.205 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:31.594675+0000 mon.a (mon.0) 3641 : cluster [DBG] osdmap e767: 8 total, 8 up, 4 in 2024-02-13T00:53:32.205 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:31.597208+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-136"}]: dispatch 2024-02-13T00:53:32.205 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:31.801671+0000 mon.c (mon.2) 509 : audit [DBG] from='client.? 172.21.15.162:0/2076715840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:32.205 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:30.539389+0000 mgr.y (mgr.14544) 636 : cluster [DBG] pgmap v1163: 204 pgs: 32 creating+peering, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:53:32.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:31.594599+0000 mon.a (mon.0) 3640 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:53:32.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:31.594675+0000 mon.a (mon.0) 3641 : cluster [DBG] osdmap e767: 8 total, 8 up, 4 in 2024-02-13T00:53:32.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:31.597208+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-136"}]: dispatch 2024-02-13T00:53:32.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:31.801671+0000 mon.c (mon.2) 509 : audit [DBG] from='client.? 172.21.15.162:0/2076715840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:32.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:30.539389+0000 mgr.y (mgr.14544) 636 : cluster [DBG] pgmap v1163: 204 pgs: 32 creating+peering, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:53:32.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:31.594599+0000 mon.a (mon.0) 3640 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:53:32.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:31.594675+0000 mon.a (mon.0) 3641 : cluster [DBG] osdmap e767: 8 total, 8 up, 4 in 2024-02-13T00:53:32.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:31.597208+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-136"}]: dispatch 2024-02-13T00:53:32.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:31.801671+0000 mon.c (mon.2) 509 : audit [DBG] from='client.? 172.21.15.162:0/2076715840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:32.968 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:32.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:32.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:32.982 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:33.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:33.110 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:33.131 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:33.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:33.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:53:33.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:33.269 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:33.418 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:33.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:33.522 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:33.554 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:33.575 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:33.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:33.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:33.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:33.707 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:33.726 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:33.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:33.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:33.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.798+0000 7f69e1534700 1 -- 172.21.15.162:0/2151909061 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69dc102270 msgr2=0x7f69dc104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.798+0000 7f69e1534700 1 --2- 172.21.15.162:0/2151909061 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69dc102270 0x7f69dc104660 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7f69d4009c30 tx=0x7f69d4004c50).stop 2024-02-13T00:53:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.798+0000 7f69e1534700 1 -- 172.21.15.162:0/2151909061 shutdown_connections 2024-02-13T00:53:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.798+0000 7f69e1534700 1 --2- 172.21.15.162:0/2151909061 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69dc102270 0x7f69dc104660 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.798+0000 7f69e1534700 1 --2- 172.21.15.162:0/2151909061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69dc0f8d10 0x7f69dc0f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.798+0000 7f69e1534700 1 --2- 172.21.15.162:0/2151909061 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69dc0f95e0 0x7f69dc101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.798+0000 7f69e1534700 1 -- 172.21.15.162:0/2151909061 >> 172.21.15.162:0/2151909061 conn(0x7f69dc06eb90 msgr2=0x7f69dc0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.798+0000 7f69e1534700 1 -- 172.21.15.162:0/2151909061 shutdown_connections 2024-02-13T00:53:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.798+0000 7f69e1534700 1 -- 172.21.15.162:0/2151909061 wait complete. 2024-02-13T00:53:33.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.798+0000 7f69e1534700 1 librados: starting msgr at 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.798+0000 7f69e1534700 1 librados: starting objecter 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69e1534700 1 Processor -- start 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69e1534700 1 -- start start 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69e1534700 1 librados: setting wanted keys 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69e1534700 1 librados: calling monclient init 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69e1534700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69dc0f8d10 0x7f69dc10e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69e1534700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69dc0f95e0 0x7f69dc10e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69e1534700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69dc102270 0x7f69dc1082b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69e1534700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc1087a0 con 0x7f69dc102270 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69e1534700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc108910 con 0x7f69dc0f8d10 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69e1534700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc108a80 con 0x7f69dc0f95e0 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69da7fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69dc0f95e0 0x7f69dc10e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69db7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69dc102270 0x7f69dc1082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69da7fc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69dc0f95e0 0x7f69dc10e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:52440/0 (socket says 172.21.15.162:52440) 2024-02-13T00:53:33.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69da7fc700 1 -- 172.21.15.162:0/284420930 learned_addr learned my addr 172.21.15.162:0/284420930 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69daffd700 1 --2- 172.21.15.162:0/284420930 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69dc0f8d10 0x7f69dc10e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69db7fe700 1 -- 172.21.15.162:0/284420930 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69dc0f95e0 msgr2=0x7f69dc10e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69db7fe700 1 --2- 172.21.15.162:0/284420930 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69dc0f95e0 0x7f69dc10e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69db7fe700 1 -- 172.21.15.162:0/284420930 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69dc0f8d10 msgr2=0x7f69dc10e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69db7fe700 1 --2- 172.21.15.162:0/284420930 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69dc0f8d10 0x7f69dc10e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69db7fe700 1 -- 172.21.15.162:0/284420930 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d0005040 con 0x7f69dc102270 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.799+0000 7f69daffd700 1 --2- 172.21.15.162:0/284420930 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69dc0f8d10 0x7f69dc10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.800+0000 7f69db7fe700 1 --2- 172.21.15.162:0/284420930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69dc102270 0x7f69dc1082b0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f69d4004e10 tx=0x7f69d4004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.800+0000 7f69c3fff700 10 client.?.objecter ms_handle_connect 0x7f69dc102270 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.800+0000 7f69c3fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.800+0000 7f69c3fff700 1 -- 172.21.15.162:0/284420930 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d400e040 con 0x7f69dc102270 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.800+0000 7f69e1534700 1 -- 172.21.15.162:0/284420930 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d40097c0 con 0x7f69dc102270 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.800+0000 7f69c3fff700 1 -- 172.21.15.162:0/284420930 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69d4018040 con 0x7f69dc102270 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.800+0000 7f69c3fff700 1 -- 172.21.15.162:0/284420930 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d401c3f0 con 0x7f69dc102270 2024-02-13T00:53:33.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.800+0000 7f69e1534700 10 client.28110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.800+0000 7f69e1534700 1 -- 172.21.15.162:0/284420930 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc108f70 con 0x7f69dc102270 2024-02-13T00:53:33.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.800+0000 7f69e1534700 1 librados: init done 2024-02-13T00:53:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.801+0000 7f69e1534700 1 -- 172.21.15.162:0/284420930 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69dc0475e0 con 0x7f69dc102270 2024-02-13T00:53:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.801+0000 7f69c3fff700 1 -- 172.21.15.162:0/284420930 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f69d4007bf0 con 0x7f69dc102270 2024-02-13T00:53:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.801+0000 7f69c3fff700 1 --2- 172.21.15.162:0/284420930 >> v2:172.21.15.162:6800/2866534053 conn(0x7f69c4062c90 0x7f69c4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.801+0000 7f69c3fff700 1 -- 172.21.15.162:0/284420930 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(769..769 src has 449..769) v4 ==== 8656+0+0 (secure 0 0 0) 0x7f69d407e540 con 0x7f69dc102270 2024-02-13T00:53:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.801+0000 7f69c3fff700 10 client.28110.objecter ms_dispatch 0x7f69dc001550 osd_map(769..769 src has 449..769) v4 2024-02-13T00:53:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.801+0000 7f69c3fff700 3 client.28110.objecter handle_osd_map got epochs [769,769] > 0 2024-02-13T00:53:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.801+0000 7f69c3fff700 3 client.28110.objecter handle_osd_map decoding full epoch 769 2024-02-13T00:53:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.801+0000 7f69daffd700 1 --2- 172.21.15.162:0/284420930 >> v2:172.21.15.162:6800/2866534053 conn(0x7f69c4062c90 0x7f69c4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.802+0000 7f69c3fff700 20 client.28110.objecter dump_active .. 0 homeless 2024-02-13T00:53:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.802+0000 7f69c3fff700 10 client.28110.objecter ms_handle_connect 0x7f69c4062c90 2024-02-13T00:53:33.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.802+0000 7f69daffd700 1 --2- 172.21.15.162:0/284420930 >> v2:172.21.15.162:6800/2866534053 conn(0x7f69c4062c90 0x7f69c4065150 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 rx=0x7f69d0009f20 tx=0x7f69d0009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:33.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.806+0000 7f69c3fff700 1 -- 172.21.15.162:0/284420930 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f69d40517c0 con 0x7f69dc102270 2024-02-13T00:53:33.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:33.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:32.540064+0000 mgr.y (mgr.14544) 637 : cluster [DBG] pgmap v1166: 204 pgs: 32 creating+peering, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:53:33.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:32.593652+0000 mon.a (mon.0) 3643 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-136"}]': finished 2024-02-13T00:53:33.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:32.593722+0000 mon.a (mon.0) 3644 : cluster [DBG] osdmap e768: 8 total, 8 up, 4 in 2024-02-13T00:53:33.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:32.595451+0000 mon.a (mon.0) 3645 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-136", "mode": "writeback"}]: dispatch 2024-02-13T00:53:33.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:32.540064+0000 mgr.y (mgr.14544) 637 : cluster [DBG] pgmap v1166: 204 pgs: 32 creating+peering, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:53:33.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:32.593652+0000 mon.a (mon.0) 3643 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-136"}]': finished 2024-02-13T00:53:33.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:32.593722+0000 mon.a (mon.0) 3644 : cluster [DBG] osdmap e768: 8 total, 8 up, 4 in 2024-02-13T00:53:33.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:32.595451+0000 mon.a (mon.0) 3645 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-136", "mode": "writeback"}]: dispatch 2024-02-13T00:53:33.858 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:33.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.979+0000 7f69e1534700 1 -- 172.21.15.162:0/284420930 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f69dc109820 con 0x7f69dc102270 2024-02-13T00:53:33.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.980+0000 7f69c3fff700 1 -- 172.21.15.162:0/284420930 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v769) v1 ==== 81+0+49776 (secure 0 0 0) 0x7f69d40513b0 con 0x7f69dc102270 2024-02-13T00:53:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:53:33.985 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:53:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: "epoch": 769, 2024-02-13T00:53:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: "fsid": "d4ee01fc-ca06-11ee-95b9-87774f69a715", 2024-02-13T00:53:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: "created": "2024-02-13T00:31:51.797017+0000", 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "modified": "2024-02-13T00:53:33.587890+0000", 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "last_up_change": "2024-02-13T00:48:03.474219+0000", 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "last_in_change": "2024-02-13T00:52:29.558263+0000", 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_num": 5806080, 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_set": [ 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "nodeep-scrub", 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "noscrub", 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "pglog_hardlimit", 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "purged_snapdirs", 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_deletes", 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "sortbitwise" 2024-02-13T00:53:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_version": 109, 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "full_ratio": 0.94999998807907104, 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_snapshot": "", 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_max": 252, 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "max_osd": 8, 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "require_min_compat_client": "luminous", 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "min_compat_client": "luminous", 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "require_osd_release": "pacific", 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "pools": [ 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 1, 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "device_health_metrics", 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:35:34.545390+0000", 2024-02-13T00:53:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 1, 2024-02-13T00:53:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 1, 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 1, 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 1, 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 1, 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "71", 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:53:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:53:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:53:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:53:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:53:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:53:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:53:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:53:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:53:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:53:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:53:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:53:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:53:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:53:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_max": 32, 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_min": 1 2024-02-13T00:53:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "mgr_devicehealth": {} 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 2, 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": ".rgw.root", 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.459980+0000", 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:53:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:53:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "58", 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:53:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:53:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:53:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 3, 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "iscsi", 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:39.740219+0000", 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:53:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 3, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 3, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 3, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 3, 2024-02-13T00:53:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 3, 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "102", 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:53:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:53:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:53:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "rbd": {} 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 4, 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.log", 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:41.489292+0000", 2024-02-13T00:53:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:53:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "60", 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:53:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:53:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:53:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 5, 2024-02-13T00:53:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.control", 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:43.466304+0000", 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:53:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 22, 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 22, 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 22, 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 22, 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 22, 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "5.16", 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 544, 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 543, 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 543, 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "544", 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "544", 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "542", 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:53:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:53:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:53:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 6, 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "default.rgw.meta", 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:37:45.542442+0000", 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:53:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 42, 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 42, 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 42, 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 42, 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 42, 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:53:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "758", 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "246", 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:53:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:53:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:53:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "options": { 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_bias": 4 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": { 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "rgw": {} 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 28, 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:41:00.190777+0000", 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 1, 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool", 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:53:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:53:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "108", 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "0", 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "0", 2024-02-13T00:53:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 0, 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 0, 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:53:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:53:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 234, 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-101", 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:50:12.777962+0000", 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8193, 2024-02-13T00:53:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 3, 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 1, 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 8, 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 8, 2024-02-13T00:53:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 8, 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 8, 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 8, 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "768", 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "768", 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "768", 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "768", 2024-02-13T00:53:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 10, 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 767, 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [ 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: 252 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": -1, 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": 252, 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": 252, 2024-02-13T00:53:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "none", 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi162-84057-101", 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:53:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 8192, 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 252, 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_name": "test-rados-api-smithi162-84057-136", 2024-02-13T00:53:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "create_time": "2024-02-13T00:53:28.838018+0000", 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "flags": 8201, 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "type": 1, 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 3, 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "min_size": 2, 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_rule": 0, 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_count": 0, 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_target": 0, 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_barrier": 0, 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "object_hash": 2, 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_autoscale_mode": "off", 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num": 32, 2024-02-13T00:53:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num": 32, 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_placement_num_target": 32, 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_target": 32, 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_num_pending": 32, 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "last_pg_merge_meta": { 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "source_pgid": "0.0", 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "ready_epoch": 0, 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_started": 0, 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "last_epoch_clean": 0, 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "source_version": "0'0", 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "target_version": "0'0" 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "last_change": "769", 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend": "768", 2024-02-13T00:53:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_prenautilus": "768", 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "last_force_op_resend_preluminous": "768", 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "auid": 0, 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_mode": "selfmanaged", 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_seq": 10, 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "snap_epoch": 767, 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "pool_snaps": [], 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps": "[]", 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_bytes": 0, 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "quota_max_objects": 0, 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "tiers": [], 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "tier_of": 234, 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "read_tier": -1, 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "write_tier": -1, 2024-02-13T00:53:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_mode": "writeback", 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_bytes": 0, 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "target_max_objects": 0, 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_flush_age": 0, 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "cache_min_evict_age": 0, 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profile": "", 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_params": { 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "none" 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_period": 0, 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_count": 0, 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "use_gmt_hitset": true, 2024-02-13T00:53:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "min_read_recency_for_promote": 0, 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "min_write_recency_for_promote": 0, 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_grade_decay_rate": 0, 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "hit_set_search_last_n": 0, 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "grade_table": [], 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "stripe_width": 0, 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "expected_num_objects": 0, 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "fast_read": false, 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "options": {}, 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "application_metadata": {} 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:53:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99", 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 8, 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 765, 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6801", 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6802", 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6804", 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6803", 2024-02-13T00:53:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2103743338 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6801/2103743338", 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6802/2103743338", 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6804/2103743338", 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6803/2103743338", 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "39e9d2ef-9aa4-45bb-8a47-b0e0ec222886", 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 13, 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 765, 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6805", 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6806", 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6808", 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6807", 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 150599780 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6805/150599780", 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6806/150599780", 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6808/150599780", 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6807/150599780", 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e", 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 18, 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 333, 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:53:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6809", 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6810", 2024-02-13T00:53:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6812", 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6811", 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3689719376 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6809/3689719376", 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6810/3689719376", 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6812/3689719376", 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6811/3689719376", 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:53:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "2e6b9f0e-7a18-4cf8-9331-1a973fadc001", 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 25, 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 255, 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:53:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6813", 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6814", 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6816", 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.162:6815", 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 604971011 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.162:6813/604971011", 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.162:6814/604971011", 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.162:6816/604971011", 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.162:6815/604971011", 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:53:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3", 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:53:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 31, 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 81, 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6800", 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:53:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6801", 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6803", 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:53:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6802", 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 3344263388 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6800/3344263388", 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6801/3344263388", 2024-02-13T00:53:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6803/3344263388", 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6802/3344263388", 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "6aa2bd68-7f4b-4973-ac49-7fbc57c1a723", 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 0, 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 0, 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 39, 2024-02-13T00:53:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 436, 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 479, 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 399, 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 437, 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6804", 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6805", 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6807", 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6806", 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 2277917372 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6804/2277917372", 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6805/2277917372", 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2277917372", 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6806/2277917372", 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "791ba814-5284-432e-addc-d6b7f5f3d474", 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:53:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 1, 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 45, 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 765, 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6808", 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:53:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6809", 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6811", 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6810", 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1506640341 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6808/1506640341", 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6809/1506640341", 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6811/1506640341", 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6810/1506640341", 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:53:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "uuid": "1f369c65-e663-4eb2-87a5-f2900f15b35b", 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "up": 1, 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "in": 1, 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "weight": 1, 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_affinity": 0, 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_begin": 0, 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "last_clean_end": 0, 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "up_from": 53, 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "up_thru": 215, 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "down_at": 0, 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "lost_at": 0, 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addrs": { 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6812", 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addrs": { 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6813", 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addrs": { 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6815", 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addrs": { 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "addrvec": [ 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "type": "v2", 2024-02-13T00:53:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "addr": "172.21.15.181:6814", 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "nonce": 1964192772 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "public_addr": "172.21.15.181:6812/1964192772", 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "cluster_addr": "172.21.15.181:6813/1964192772", 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_back_addr": "172.21.15.181:6815/1964192772", 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "heartbeat_front_addr": "172.21.15.181:6814/1964192772", 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "state": [ 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "exists", 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "up" 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_xinfo": [ 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 0, 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:53:13.134208+0000", 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 1, 2024-02-13T00:53:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:53:20.248234+0000", 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 2, 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:53:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:53:27.971469+0000", 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 3, 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:52:34.637591+0000", 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 4, 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:53:22.826498+0000", 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 5, 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 65536, 2024-02-13T00:53:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:53:29.473350+0000", 2024-02-13T00:53:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 436 2024-02-13T00:53:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 6, 2024-02-13T00:53:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:53:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:53:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:53:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:53:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:53:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:53:27.700421+0000", 2024-02-13T00:53:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:53:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: "osd": 7, 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: "down_stamp": "0.000000", 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_probability": 0, 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: "laggy_interval": 0, 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: "features": 4540138314316775423, 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: "old_weight": 0, 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: "last_purged_snaps_scrub": "2024-02-13T00:52:40.814498+0000", 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: "dead_epoch": 0 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap": [ 2024-02-13T00:53:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "4.b", 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: 3, 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: 6, 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: 5 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "6.23", 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: "osds": [ 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: 3, 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: 6, 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: 7 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.074 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_upmap_items": [ 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "2.f", 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 7, 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 6 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "2.1a", 2024-02-13T00:53:34.075 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:53:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:53:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 1 2024-02-13T00:53:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "2.1d", 2024-02-13T00:53:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:53:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:53:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 7 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 1 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "5.5", 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 0, 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 6 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "6.6", 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 0 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "pgid": "28.a", 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "mappings": [ 2024-02-13T00:53:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "from": 6, 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "to": 7 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "pg_temp": [], 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "primary_temp": [], 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "blocklist": { 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/1909610488": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2370739667": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/1502590187": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/4121673848": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:53:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2412382714": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3041562701": "2024-02-14T00:32:47.469889+0000", 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:6800/2775563314": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/3135632190": "2024-02-14T00:32:24.380625+0000", 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/197307636": "2024-02-14T00:38:04.142895+0000", 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "172.21.15.162:0/2653247729": "2024-02-14T00:32:24.380625+0000" 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "range_blocklist": {}, 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "erasure_code_profiles": { 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "default": { 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:53:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van" 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "testprofile-test-rados-api-smithi162-84057-101": { 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-device-class": "", 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-failure-domain": "osd", 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "crush-root": "default", 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "k": "2", 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "m": "1", 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "plugin": "jerasure", 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "ruleset-failure-domain": "osd", 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "technique": "reed_sol_van", 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "w": "8" 2024-02-13T00:53:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "removed_snaps_queue": [ 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 24, 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 9, 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 25, 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 60, 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:53:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 78, 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 85, 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 94, 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:53:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 95, 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 4, 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 4 2024-02-13T00:53:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 107, 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 113, 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 1, 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 3, 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 5, 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 7, 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 125, 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 10, 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 2 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 133, 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 12, 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 154, 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 14, 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 166, 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 17, 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 176, 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 20, 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 237, 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 2, 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 238, 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 4, 2024-02-13T00:53:34.095 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 241, 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 6, 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.096 INFO:teuthology.orchestra.run.smithi162.stdout: "pool": 245, 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: "snaps": [ 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: "begin": 9, 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: "length": 1 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: "new_removed_snaps": [], 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: "new_purged_snaps": [], 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: "crush_node_flags": {}, 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: "device_class_flags": {}, 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode": { 2024-02-13T00:53:34.097 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_enabled": false, 2024-02-13T00:53:34.098 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_bucket_count": 0, 2024-02-13T00:53:34.098 INFO:teuthology.orchestra.run.smithi162.stdout: "degraded_stretch_mode": 0, 2024-02-13T00:53:34.098 INFO:teuthology.orchestra.run.smithi162.stdout: "recovering_stretch_mode": 0, 2024-02-13T00:53:34.098 INFO:teuthology.orchestra.run.smithi162.stdout: "stretch_mode_bucket": 0 2024-02-13T00:53:34.098 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:34.098 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:53:34.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:32.540064+0000 mgr.y (mgr.14544) 637 : cluster [DBG] pgmap v1166: 204 pgs: 32 creating+peering, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:53:34.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:32.593652+0000 mon.a (mon.0) 3643 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-84057-101", "overlaypool": "test-rados-api-smithi162-84057-136"}]': finished 2024-02-13T00:53:34.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:32.593722+0000 mon.a (mon.0) 3644 : cluster [DBG] osdmap e768: 8 total, 8 up, 4 in 2024-02-13T00:53:34.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:32.595451+0000 mon.a (mon.0) 3645 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-136", "mode": "writeback"}]: dispatch 2024-02-13T00:53:34.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.982+0000 7f69e1534700 10 librados: watch_flush enter 2024-02-13T00:53:34.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.982+0000 7f69e1534700 10 librados: watch_flush exit 2024-02-13T00:53:34.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.982+0000 7f69e1534700 20 client.28110.objecter shutdown clearing up homeless session... 2024-02-13T00:53:34.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.982+0000 7f69e1534700 10 client.28110.objecter successfully canceled tick 2024-02-13T00:53:34.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.982+0000 7f69e1534700 1 -- 172.21.15.162:0/284420930 >> v2:172.21.15.162:6800/2866534053 conn(0x7f69c4062c90 msgr2=0x7f69c4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:34.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.982+0000 7f69e1534700 1 --2- 172.21.15.162:0/284420930 >> v2:172.21.15.162:6800/2866534053 conn(0x7f69c4062c90 0x7f69c4065150 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 rx=0x7f69d0009f20 tx=0x7f69d0009cb0).stop 2024-02-13T00:53:34.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.983+0000 7f69e1534700 1 -- 172.21.15.162:0/284420930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69dc102270 msgr2=0x7f69dc1082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:34.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.983+0000 7f69e1534700 1 --2- 172.21.15.162:0/284420930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69dc102270 0x7f69dc1082b0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f69d4004e10 tx=0x7f69d4004c50).stop 2024-02-13T00:53:34.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.983+0000 7f69e1534700 1 -- 172.21.15.162:0/284420930 shutdown_connections 2024-02-13T00:53:34.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.983+0000 7f69e1534700 1 --2- 172.21.15.162:0/284420930 >> v2:172.21.15.162:6800/2866534053 conn(0x7f69c4062c90 0x7f69c4065150 unknown :-1 s=CLOSED pgs=3040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:34.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.983+0000 7f69e1534700 1 --2- 172.21.15.162:0/284420930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69dc102270 0x7f69dc1082b0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:34.100 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.983+0000 7f69e1534700 1 --2- 172.21.15.162:0/284420930 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69dc0f8d10 0x7f69dc10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:34.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.983+0000 7f69e1534700 1 --2- 172.21.15.162:0/284420930 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69dc0f95e0 0x7f69dc10e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:34.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.983+0000 7f69e1534700 1 -- 172.21.15.162:0/284420930 >> 172.21.15.162:0/284420930 conn(0x7f69dc06eb90 msgr2=0x7f69dc102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:34.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.983+0000 7f69e1534700 1 -- 172.21.15.162:0/284420930 shutdown_connections 2024-02-13T00:53:34.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.983+0000 7f69e1534700 1 -- 172.21.15.162:0/284420930 wait complete. 2024-02-13T00:53:34.101 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:33.983+0000 7f69e1534700 1 librados: shutdown 2024-02-13T00:53:34.448 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:34.923 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:33.588126+0000 mon.a (mon.0) 3646 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:53:34.924 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:33.591743+0000 mon.a (mon.0) 3647 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-136", "mode": "writeback"}]': finished 2024-02-13T00:53:34.924 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:33.591813+0000 mon.a (mon.0) 3648 : cluster [DBG] osdmap e769: 8 total, 8 up, 4 in 2024-02-13T00:53:34.924 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:33.632129+0000 mon.a (mon.0) 3649 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:53:34.924 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:33.980557+0000 mon.a (mon.0) 3650 : audit [DBG] from='client.? 172.21.15.162:0/284420930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:53:34.924 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:34.343740+0000 mon.a (mon.0) 3651 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:34.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:33.588126+0000 mon.a (mon.0) 3646 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:53:34.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:33.591743+0000 mon.a (mon.0) 3647 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-136", "mode": "writeback"}]': finished 2024-02-13T00:53:34.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:33.591813+0000 mon.a (mon.0) 3648 : cluster [DBG] osdmap e769: 8 total, 8 up, 4 in 2024-02-13T00:53:34.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:33.632129+0000 mon.a (mon.0) 3649 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:53:34.925 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:33.980557+0000 mon.a (mon.0) 3650 : audit [DBG] from='client.? 172.21.15.162:0/284420930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:53:34.925 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:34.343740+0000 mon.a (mon.0) 3651 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:34.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:53:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:33.588126+0000 mon.a (mon.0) 3646 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-13T00:53:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:33.591743+0000 mon.a (mon.0) 3647 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-84057-136", "mode": "writeback"}]': finished 2024-02-13T00:53:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:33.591813+0000 mon.a (mon.0) 3648 : cluster [DBG] osdmap e769: 8 total, 8 up, 4 in 2024-02-13T00:53:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:33.632129+0000 mon.a (mon.0) 3649 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-13T00:53:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:33.980557+0000 mon.a (mon.0) 3650 : audit [DBG] from='client.? 172.21.15.162:0/284420930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-13T00:53:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:34.343740+0000 mon.a (mon.0) 3651 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:35.509 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:35.510 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:53:35.510 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:53:35.510 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-13T00:53:35.510 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:53:35.510 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:35.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:35.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:35.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:35.642 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:35.716 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:35.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:35.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:35.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:34.540703+0000 mgr.y (mgr.14544) 638 : cluster [DBG] pgmap v1169: 204 pgs: 32 creating+peering, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:53:35.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:34.602796+0000 mon.a (mon.0) 3652 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:53:35.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:34.602859+0000 mon.a (mon.0) 3653 : cluster [DBG] osdmap e770: 8 total, 8 up, 4 in 2024-02-13T00:53:35.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:34.605857+0000 mon.a (mon.0) 3654 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:53:35.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:35.571862+0000 mon.a (mon.0) 3655 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:53:35.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:35.571929+0000 mon.a (mon.0) 3656 : cluster [DBG] osdmap e771: 8 total, 8 up, 4 in 2024-02-13T00:53:35.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:35.575098+0000 mon.a (mon.0) 3657 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:53:35.774 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:34.540703+0000 mgr.y (mgr.14544) 638 : cluster [DBG] pgmap v1169: 204 pgs: 32 creating+peering, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:53:35.774 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:34.602796+0000 mon.a (mon.0) 3652 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:53:35.774 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:34.602859+0000 mon.a (mon.0) 3653 : cluster [DBG] osdmap e770: 8 total, 8 up, 4 in 2024-02-13T00:53:35.774 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:34.605857+0000 mon.a (mon.0) 3654 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:53:35.774 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:35.571862+0000 mon.a (mon.0) 3655 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:53:35.774 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:35.571929+0000 mon.a (mon.0) 3656 : cluster [DBG] osdmap e771: 8 total, 8 up, 4 in 2024-02-13T00:53:35.774 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:35.575098+0000 mon.a (mon.0) 3657 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:53:35.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:35.848 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:35.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:34.540703+0000 mgr.y (mgr.14544) 638 : cluster [DBG] pgmap v1169: 204 pgs: 32 creating+peering, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:53:35.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:34.602796+0000 mon.a (mon.0) 3652 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_count","val": "2"}]': finished 2024-02-13T00:53:35.964 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:34.602859+0000 mon.a (mon.0) 3653 : cluster [DBG] osdmap e770: 8 total, 8 up, 4 in 2024-02-13T00:53:35.964 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:34.605857+0000 mon.a (mon.0) 3654 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-13T00:53:35.964 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:35.571862+0000 mon.a (mon.0) 3655 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_period","val": "600"}]': finished 2024-02-13T00:53:35.964 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:35.571929+0000 mon.a (mon.0) 3656 : cluster [DBG] osdmap e771: 8 total, 8 up, 4 in 2024-02-13T00:53:35.964 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:35.575098+0000 mon.a (mon.0) 3657 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-13T00:53:35.972 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:35.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:35.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:36.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:36.103 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:36.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:53:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:53:36.351 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:36.357 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:53:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:53:36] "GET /metrics HTTP/1.1" 200 190202 "" "Prometheus/2.33.4" 2024-02-13T00:53:36.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:36.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:36.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:36.491 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:36.513 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:36.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:36.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:36.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:36.645 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:36.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:35.960067+0000 mgr.y (mgr.14544) 639 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:36.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:36.568682+0000 mon.a (mon.0) 3658 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:53:36.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:36.572857+0000 mon.a (mon.0) 3659 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:53:36.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:36.572925+0000 mon.a (mon.0) 3660 : cluster [DBG] osdmap e772: 8 total, 8 up, 4 in 2024-02-13T00:53:36.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:36.574421+0000 mon.a (mon.0) 3661 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:53:36.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-13T00:53:36.896 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:35.960067+0000 mgr.y (mgr.14544) 639 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:36.896 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:36.568682+0000 mon.a (mon.0) 3658 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:53:36.896 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:36.572857+0000 mon.a (mon.0) 3659 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:53:36.896 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:36.572925+0000 mon.a (mon.0) 3660 : cluster [DBG] osdmap e772: 8 total, 8 up, 4 in 2024-02-13T00:53:36.896 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:36.574421+0000 mon.a (mon.0) 3661 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:53:36.897 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:35.960067+0000 mgr.y (mgr.14544) 639 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:36.897 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:36.568682+0000 mon.a (mon.0) 3658 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-13T00:53:36.897 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:36.572857+0000 mon.a (mon.0) 3659 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "hit_set_type","val": "bloom"}]': finished 2024-02-13T00:53:36.897 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:36.572925+0000 mon.a (mon.0) 3660 : cluster [DBG] osdmap e772: 8 total, 8 up, 4 in 2024-02-13T00:53:36.897 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:36.574421+0000 mon.a (mon.0) 3661 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-13T00:53:37.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:37.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:37.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:37.431 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:37.453 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:37.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:37.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:37.587 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:36.541308+0000 mgr.y (mgr.14544) 640 : cluster [DBG] pgmap v1172: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-13T00:53:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:37.576105+0000 mon.a (mon.0) 3662 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-13T00:53:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:37.576179+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e773: 8 total, 8 up, 4 in 2024-02-13T00:53:37.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:37.580140+0000 mon.a (mon.0) 3664 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-13T00:53:37.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:36.541308+0000 mgr.y (mgr.14544) 640 : cluster [DBG] pgmap v1172: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-13T00:53:37.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:37.576105+0000 mon.a (mon.0) 3662 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-13T00:53:37.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:37.576179+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e773: 8 total, 8 up, 4 in 2024-02-13T00:53:37.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:37.580140+0000 mon.a (mon.0) 3664 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-13T00:53:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:36.541308+0000 mgr.y (mgr.14544) 640 : cluster [DBG] pgmap v1172: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-13T00:53:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:37.576105+0000 mon.a (mon.0) 3662 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-13T00:53:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:37.576179+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e773: 8 total, 8 up, 4 in 2024-02-13T00:53:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:37.580140+0000 mon.a (mon.0) 3664 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-13T00:53:37.837 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:37.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:37.855 INFO:tasks.thrashosds.thrasher:j is {'epoch': 769, 'fsid': 'd4ee01fc-ca06-11ee-95b9-87774f69a715', 'created': '2024-02-13T00:31:51.797017+0000', 'modified': '2024-02-13T00:53:33.587890+0000', 'last_up_change': '2024-02-13T00:48:03.474219+0000', 'last_in_change': '2024-02-13T00:52:29.558263+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 109, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 252, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-13T00:35:34.545390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-13T00:37:39.459980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-13T00:37:39.740219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-13T00:37:41.489292+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-13T00:37:43.466304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 544, 'last_epoch_started': 543, 'last_epoch_clean': 543, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '544', 'last_force_op_resend_preluminous': '542', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-13T00:37:45.542442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '758', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '246', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 28, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-13T00:41:00.190777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 234, 'pool_name': 'test-rados-api-smithi162-84057-101', 'create_time': '2024-02-13T00:50:12.777962+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '768', 'last_force_op_resend': '768', 'last_force_op_resend_prenautilus': '768', 'last_force_op_resend_preluminous': '768', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 10, 'snap_epoch': 767, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [252], 'tier_of': -1, 'read_tier': 252, 'write_tier': 252, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi162-84057-101', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 252, 'pool_name': 'test-rados-api-smithi162-84057-136', 'create_time': '2024-02-13T00:53:28.838018+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '769', 'last_force_op_resend': '768', 'last_force_op_resend_prenautilus': '768', 'last_force_op_resend_preluminous': '768', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 10, 'snap_epoch': 767, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 234, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 765, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6801', 'nonce': 2103743338}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 2103743338}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 2103743338}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6803', 'nonce': 2103743338}]}, 'public_addr': '172.21.15.162:6801/2103743338', 'cluster_addr': '172.21.15.162:6802/2103743338', 'heartbeat_back_addr': '172.21.15.162:6804/2103743338', 'heartbeat_front_addr': '172.21.15.162:6803/2103743338', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '39e9d2ef-9aa4-45bb-8a47-b0e0ec222886', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 765, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6805', 'nonce': 150599780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 150599780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 150599780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6807', 'nonce': 150599780}]}, 'public_addr': '172.21.15.162:6805/150599780', 'cluster_addr': '172.21.15.162:6806/150599780', 'heartbeat_back_addr': '172.21.15.162:6808/150599780', 'heartbeat_front_addr': '172.21.15.162:6807/150599780', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 333, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6809', 'nonce': 3689719376}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3689719376}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3689719376}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6811', 'nonce': 3689719376}]}, 'public_addr': '172.21.15.162:6809/3689719376', 'cluster_addr': '172.21.15.162:6810/3689719376', 'heartbeat_back_addr': '172.21.15.162:6812/3689719376', 'heartbeat_front_addr': '172.21.15.162:6811/3689719376', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2e6b9f0e-7a18-4cf8-9331-1a973fadc001', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 255, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6813', 'nonce': 604971011}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 604971011}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 604971011}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6815', 'nonce': 604971011}]}, 'public_addr': '172.21.15.162:6813/604971011', 'cluster_addr': '172.21.15.162:6814/604971011', 'heartbeat_back_addr': '172.21.15.162:6816/604971011', 'heartbeat_front_addr': '172.21.15.162:6815/604971011', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 3344263388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6801', 'nonce': 3344263388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6803', 'nonce': 3344263388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 3344263388}]}, 'public_addr': '172.21.15.181:6800/3344263388', 'cluster_addr': '172.21.15.181:6801/3344263388', 'heartbeat_back_addr': '172.21.15.181:6803/3344263388', 'heartbeat_front_addr': '172.21.15.181:6802/3344263388', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6aa2bd68-7f4b-4973-ac49-7fbc57c1a723', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 436, 'up_from': 479, 'up_thru': 399, 'down_at': 437, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2277917372}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6805', 'nonce': 2277917372}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6807', 'nonce': 2277917372}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2277917372}]}, 'public_addr': '172.21.15.181:6804/2277917372', 'cluster_addr': '172.21.15.181:6805/2277917372', 'heartbeat_back_addr': '172.21.15.181:6807/2277917372', 'heartbeat_front_addr': '172.21.15.181:6806/2277917372', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '791ba814-5284-432e-addc-d6b7f5f3d474', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 765, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 1506640341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6809', 'nonce': 1506640341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6811', 'nonce': 1506640341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 1506640341}]}, 'public_addr': '172.21.15.181:6808/1506640341', 'cluster_addr': '172.21.15.181:6809/1506640341', 'heartbeat_back_addr': '172.21.15.181:6811/1506640341', 'heartbeat_front_addr': '172.21.15.181:6810/1506640341', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f369c65-e663-4eb2-87a5-f2900f15b35b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 215, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 1964192772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6813', 'nonce': 1964192772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6815', 'nonce': 1964192772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 1964192772}]}, 'public_addr': '172.21.15.181:6812/1964192772', 'cluster_addr': '172.21.15.181:6813/1964192772', 'heartbeat_back_addr': '172.21.15.181:6815/1964192772', 'heartbeat_front_addr': '172.21.15.181:6814/1964192772', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:53:13.134208+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:53:20.248234+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:53:27.971469+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:52:34.637591+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:53:22.826498+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-13T00:53:29.473350+0000', 'dead_epoch': 436}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:53:27.700421+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-13T00:52:40.814498+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.b', 'osds': [3, 6, 5]}, {'pgid': '6.23', 'osds': [3, 6, 7]}], 'pg_upmap_items': [{'pgid': '2.f', 'mappings': [{'from': 7, 'to': 6}]}, {'pgid': '2.1a', 'mappings': [{'from': 6, 'to': 1}]}, {'pgid': '2.1d', 'mappings': [{'from': 0, 'to': 7}, {'from': 6, 'to': 1}]}, {'pgid': '5.5', 'mappings': [{'from': 0, 'to': 6}]}, {'pgid': '6.6', 'mappings': [{'from': 6, 'to': 0}]}, {'pgid': '28.a', 'mappings': [{'from': 6, 'to': 7}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.162:6800/1909610488': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2370739667': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/1502590187': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/4121673848': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:0/2412382714': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/3041562701': '2024-02-14T00:32:47.469889+0000', '172.21.15.162:6800/2775563314': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/3135632190': '2024-02-14T00:32:24.380625+0000', '172.21.15.162:0/197307636': '2024-02-14T00:38:04.142895+0000', '172.21.15.162:0/2653247729': '2024-02-14T00:32:24.380625+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi162-84057-101': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 24, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 60, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 78, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 94, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 113, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 166, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 176, 'snaps': [{'begin': 20, 'length': 1}]}, {'pool': 237, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 238, 'snaps': [{'begin': 4, 'length': 1}]}, {'pool': 241, 'snaps': [{'begin': 6, 'length': 1}]}, {'pool': 245, 'snaps': [{'begin': 9, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-13T00:53:37.856 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 6.23 2024-02-13T00:53:37.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd rm-pg-upmap 6.23 2024-02-13T00:53:37.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:37.973 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:37.995 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:37.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:37.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:38.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:38.126 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:38.223 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:38.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:38.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:38.358 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:38.379 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:38.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:38.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:38.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:38.512 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:38.534 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:38.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:38.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:38.666 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:38.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:53:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:53:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:53:39.714 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:39.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:39.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:39.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:38.541915+0000 mgr.y (mgr.14544) 641 : cluster [DBG] pgmap v1175: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-13T00:53:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:38.575390+0000 mon.a (mon.0) 3665 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "target_max_objects","val": "1"}]': finished 2024-02-13T00:53:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:38.575458+0000 mon.a (mon.0) 3666 : cluster [DBG] osdmap e774: 8 total, 8 up, 4 in 2024-02-13T00:53:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:38.541915+0000 mgr.y (mgr.14544) 641 : cluster [DBG] pgmap v1175: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-13T00:53:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:38.575390+0000 mon.a (mon.0) 3665 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "target_max_objects","val": "1"}]': finished 2024-02-13T00:53:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:38.575458+0000 mon.a (mon.0) 3666 : cluster [DBG] osdmap e774: 8 total, 8 up, 4 in 2024-02-13T00:53:39.849 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:39.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.939+0000 7f7c8efde700 1 -- 172.21.15.162:0/2134223590 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c880f8d10 msgr2=0x7f7c880f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:39.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.939+0000 7f7c8efde700 1 --2- 172.21.15.162:0/2134223590 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c880f8d10 0x7f7c880f90f0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 rx=0x7f7c7c004870 tx=0x7f7c7c009cb0).stop 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.939+0000 7f7c8efde700 1 -- 172.21.15.162:0/2134223590 shutdown_connections 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.939+0000 7f7c8efde700 1 --2- 172.21.15.162:0/2134223590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c88102270 0x7f7c88104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.939+0000 7f7c8efde700 1 --2- 172.21.15.162:0/2134223590 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c880f8d10 0x7f7c880f90f0 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.939+0000 7f7c8efde700 1 --2- 172.21.15.162:0/2134223590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c880f95e0 0x7f7c88101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.939+0000 7f7c8efde700 1 -- 172.21.15.162:0/2134223590 >> 172.21.15.162:0/2134223590 conn(0x7f7c8806eb90 msgr2=0x7f7c880f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.939+0000 7f7c8efde700 1 -- 172.21.15.162:0/2134223590 shutdown_connections 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.939+0000 7f7c8efde700 1 -- 172.21.15.162:0/2134223590 wait complete. 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.939+0000 7f7c8efde700 1 librados: starting msgr at 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.939+0000 7f7c8efde700 1 librados: starting objecter 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.939+0000 7f7c8efde700 1 Processor -- start 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8efde700 1 -- start start 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8efde700 1 librados: setting wanted keys 2024-02-13T00:53:39.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8efde700 1 librados: calling monclient init 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8efde700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c880f8d10 0x7f7c8810e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8efde700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c880f95e0 0x7f7c8810e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8efde700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c88102270 0x7f7c88108330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8efde700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c88108820 con 0x7f7c880f95e0 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8efde700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c88108990 con 0x7f7c88102270 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8efde700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c88108b50 con 0x7f7c880f8d10 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8cd7a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c880f8d10 0x7f7c8810e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8cd7a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c880f8d10 0x7f7c8810e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:54346/0 (socket says 172.21.15.162:54346) 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8cd7a700 1 -- 172.21.15.162:0/196225305 learned_addr learned my addr 172.21.15.162:0/196225305 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c87fff700 1 --2- 172.21.15.162:0/196225305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c880f95e0 0x7f7c8810e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8d57b700 1 --2- 172.21.15.162:0/196225305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c88102270 0x7f7c88108330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8cd7a700 1 -- 172.21.15.162:0/196225305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c88102270 msgr2=0x7f7c88108330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8cd7a700 1 --2- 172.21.15.162:0/196225305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c88102270 0x7f7c88108330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:39.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8cd7a700 1 -- 172.21.15.162:0/196225305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c880f95e0 msgr2=0x7f7c8810e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8cd7a700 1 --2- 172.21.15.162:0/196225305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c880f95e0 0x7f7c8810e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8cd7a700 1 -- 172.21.15.162:0/196225305 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c800097c0 con 0x7f7c880f8d10 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c8cd7a700 1 --2- 172.21.15.162:0/196225305 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c880f8d10 0x7f7c8810e190 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 rx=0x7f7c7c009eb0 tx=0x7f7c7c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c85ffb700 10 client.?.objecter ms_handle_connect 0x7f7c880f8d10 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c85ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.940+0000 7f7c85ffb700 1 -- 172.21.15.162:0/196225305 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c00e070 con 0x7f7c880f8d10 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.941+0000 7f7c8efde700 1 -- 172.21.15.162:0/196225305 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7c005040 con 0x7f7c880f8d10 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.941+0000 7f7c85ffb700 1 -- 172.21.15.162:0/196225305 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c7c013da0 con 0x7f7c880f8d10 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.941+0000 7f7c85ffb700 1 -- 172.21.15.162:0/196225305 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c019070 con 0x7f7c880f8d10 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.941+0000 7f7c8efde700 10 client.38219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.941+0000 7f7c8efde700 1 -- 172.21.15.162:0/196225305 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c88108ff0 con 0x7f7c880f8d10 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.941+0000 7f7c8efde700 1 librados: init done 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.942+0000 7f7c8efde700 1 -- 172.21.15.162:0/196225305 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c880475e0 con 0x7f7c880f8d10 2024-02-13T00:53:39.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.942+0000 7f7c85ffb700 1 -- 172.21.15.162:0/196225305 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f7c7c01e420 con 0x7f7c880f8d10 2024-02-13T00:53:39.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.943+0000 7f7c85ffb700 1 --2- 172.21.15.162:0/196225305 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7c78062c90 0x7f7c78065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:39.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.943+0000 7f7c85ffb700 1 -- 172.21.15.162:0/196225305 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(774..774 src has 449..774) v4 ==== 8682+0+0 (secure 0 0 0) 0x7f7c7c07e2e0 con 0x7f7c880f8d10 2024-02-13T00:53:39.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.943+0000 7f7c85ffb700 10 client.38219.objecter ms_dispatch 0x7f7c88001550 osd_map(774..774 src has 449..774) v4 2024-02-13T00:53:39.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.943+0000 7f7c85ffb700 3 client.38219.objecter handle_osd_map got epochs [774,774] > 0 2024-02-13T00:53:39.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.943+0000 7f7c85ffb700 3 client.38219.objecter handle_osd_map decoding full epoch 774 2024-02-13T00:53:39.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.943+0000 7f7c87fff700 1 --2- 172.21.15.162:0/196225305 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7c78062c90 0x7f7c78065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:39.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.943+0000 7f7c85ffb700 20 client.38219.objecter dump_active .. 0 homeless 2024-02-13T00:53:39.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.943+0000 7f7c85ffb700 10 client.38219.objecter ms_handle_connect 0x7f7c78062c90 2024-02-13T00:53:39.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.943+0000 7f7c87fff700 1 --2- 172.21.15.162:0/196225305 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7c78062c90 0x7f7c78065150 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 rx=0x7f7c88109ad0 tx=0x7f7c7400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:39.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.947+0000 7f7c85ffb700 1 -- 172.21.15.162:0/196225305 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7c7c0515c0 con 0x7f7c880f8d10 2024-02-13T00:53:39.979 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:39.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:40.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:38.541915+0000 mgr.y (mgr.14544) 641 : cluster [DBG] pgmap v1175: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-13T00:53:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:38.575390+0000 mon.a (mon.0) 3665 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-136","var": "target_max_objects","val": "1"}]': finished 2024-02-13T00:53:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:38.575458+0000 mon.a (mon.0) 3666 : cluster [DBG] osdmap e774: 8 total, 8 up, 4 in 2024-02-13T00:53:40.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:40.114 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:40.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.809+0000 7fda897ff700 1 -- 172.21.15.162:0/1498673740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda840fdaa0 msgr2=0x7fda840fde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.809+0000 7fda897ff700 1 --2- 172.21.15.162:0/1498673740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda840fdaa0 0x7fda840fde80 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7fda780047b0 tx=0x7fda78009cb0).stop 2024-02-13T00:53:40.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.809+0000 7fda897ff700 1 -- 172.21.15.162:0/1498673740 shutdown_connections 2024-02-13T00:53:40.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.809+0000 7fda897ff700 1 --2- 172.21.15.162:0/1498673740 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda840f8310 0x7fda84104910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.809+0000 7fda897ff700 1 --2- 172.21.15.162:0/1498673740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda840fdaa0 0x7fda840fde80 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.809+0000 7fda897ff700 1 --2- 172.21.15.162:0/1498673740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda840fe400 0x7fda840f7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.809+0000 7fda897ff700 1 -- 172.21.15.162:0/1498673740 >> 172.21.15.162:0/1498673740 conn(0x7fda840f36d0 msgr2=0x7fda840f5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.809+0000 7fda897ff700 1 -- 172.21.15.162:0/1498673740 shutdown_connections 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.809+0000 7fda897ff700 1 -- 172.21.15.162:0/1498673740 wait complete. 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.809+0000 7fda897ff700 1 librados: starting msgr at 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.809+0000 7fda897ff700 1 librados: starting objecter 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.809+0000 7fda897ff700 1 Processor -- start 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda897ff700 1 -- start start 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda897ff700 1 librados: setting wanted keys 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda897ff700 1 librados: calling monclient init 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda897ff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda840f8310 0x7fda8410dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda897ff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda840fdaa0 0x7fda8410e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda897ff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda840fe400 0x7fda84108080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda897ff700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda84108570 con 0x7fda840fdaa0 2024-02-13T00:53:40.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda897ff700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda841086e0 con 0x7fda840fe400 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda897ff700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda841088a0 con 0x7fda840f8310 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda82ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda840f8310 0x7fda8410dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda82ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda840f8310 0x7fda8410dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:54328/0 (socket says 172.21.15.162:54328) 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda82ffd700 1 -- 172.21.15.162:0/3434816234 learned_addr learned my addr 172.21.15.162:0/3434816234 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda827fc700 1 --2- 172.21.15.162:0/3434816234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda840fdaa0 0x7fda8410e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda837fe700 1 --2- 172.21.15.162:0/3434816234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda840fe400 0x7fda84108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda82ffd700 1 -- 172.21.15.162:0/3434816234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda840fe400 msgr2=0x7fda84108080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda82ffd700 1 --2- 172.21.15.162:0/3434816234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda840fe400 0x7fda84108080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda82ffd700 1 -- 172.21.15.162:0/3434816234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda840fdaa0 msgr2=0x7fda8410e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda82ffd700 1 --2- 172.21.15.162:0/3434816234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda840fdaa0 0x7fda8410e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda82ffd700 1 -- 172.21.15.162:0/3434816234 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c0097c0 con 0x7fda840f8310 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda827fc700 1 --2- 172.21.15.162:0/3434816234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda840fdaa0 0x7fda8410e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda837fe700 1 --2- 172.21.15.162:0/3434816234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda840fe400 0x7fda84108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:53:40.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda82ffd700 1 --2- 172.21.15.162:0/3434816234 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda840f8310 0x7fda8410dee0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 rx=0x7fda780044e0 tx=0x7fda78007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda6bfff700 10 client.?.objecter ms_handle_connect 0x7fda840f8310 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.810+0000 7fda6bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.811+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda78018070 con 0x7fda840f8310 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.811+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda780093b0 con 0x7fda840f8310 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.811+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7801de40 con 0x7fda840f8310 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.811+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda78005040 con 0x7fda840f8310 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.811+0000 7fda897ff700 10 client.38213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.811+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda84108dc0 con 0x7fda840f8310 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.811+0000 7fda897ff700 1 librados: init done 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.811+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fda8405f960 con 0x7fda840f8310 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.812+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fda78013ad0 con 0x7fda840f8310 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.812+0000 7fda6bfff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6800/2866534053 conn(0x7fda6c062c90 0x7fda6c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:40.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.812+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(774..774 src has 449..774) v4 ==== 8682+0+0 (secure 0 0 0) 0x7fda7807e990 con 0x7fda840f8310 2024-02-13T00:53:40.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.812+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 osd_map(774..774 src has 449..774) v4 2024-02-13T00:53:40.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.812+0000 7fda6bfff700 3 client.38213.objecter handle_osd_map got epochs [774,774] > 0 2024-02-13T00:53:40.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.812+0000 7fda6bfff700 3 client.38213.objecter handle_osd_map decoding full epoch 774 2024-02-13T00:53:40.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.812+0000 7fda827fc700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6800/2866534053 conn(0x7fda6c062c90 0x7fda6c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:40.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.812+0000 7fda6bfff700 20 client.38213.objecter dump_active .. 0 homeless 2024-02-13T00:53:40.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v774) v1 ==== 54+0+16 (secure 0 0 0) 0x7fda78083220 con 0x7fda840f8310 2024-02-13T00:53:40.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda6bfff700 10 client.38213.objecter ms_handle_connect 0x7fda6c062c90 2024-02-13T00:53:40.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda827fc700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6800/2866534053 conn(0x7fda6c062c90 0x7fda6c065150 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 rx=0x7fda84109820 tx=0x7fda7400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:53:40.133 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:53:40.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda897ff700 10 client.38213.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:40.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 1 2024-02-13T00:53:40.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6801/2103743338 conn(0x7fda840fed70 0x7fda840ff150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:40.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda840475e0 osd=0 3 2024-02-13T00:53:40.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda840475e0 osd=0 3 2024-02-13T00:53:40.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda840475e0 osd=0 3 2024-02-13T00:53:40.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 1 2024-02-13T00:53:40.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda840475e0 osd=0 3 2024-02-13T00:53:40.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 0 1 2024-02-13T00:53:40.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda840475e0 osd=0 4 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda897ff700 10 client.38213.objecter _send_command 1 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda840ff640 con 0x7fda840fed70 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda837fe700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6801/2103743338 conn(0x7fda840fed70 0x7fda840ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda837fe700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6801/2103743338 conn(0x7fda840fed70 0x7fda840ff150 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.813+0000 7fda6bfff700 10 client.38213.objecter ms_handle_connect 0x7fda840fed70 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.814+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fda840ff640 con 0x7fda840fed70 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.814+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.814+0000 7fda6bfff700 10 client.38213.objecter _finish_command 1 = system:0 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.814+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda840475e0 osd=0 4 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.814+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 0 1 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.829+0000 7fda897ff700 10 client.38213.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.829+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 2 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.829+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda840475e0 osd=0 3 2024-02-13T00:53:40.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.829+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda840475e0 osd=0 3 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.829+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda840475e0 osd=0 3 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.829+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 2 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.829+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda840475e0 osd=0 3 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.829+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 0 2 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.829+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda840475e0 osd=0 4 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.829+0000 7fda897ff700 10 client.38213.objecter _send_command 2 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.829+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda840ff7b0 con 0x7fda840fed70 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fda840ff7b0 con 0x7fda840fed70 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda6bfff700 10 client.38213.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda840475e0 osd=0 4 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 0 2 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda897ff700 10 client.38213.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:40.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 3 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6805/150599780 conn(0x7fda84028590 0x7fda84118d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda840f5cd0 osd=1 3 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda840f5cd0 osd=1 3 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda840f5cd0 osd=1 3 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 3 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda840f5cd0 osd=1 3 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 1 3 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda840f5cd0 osd=1 4 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda897ff700 10 client.38213.objecter _send_command 3 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda840f5e40 con 0x7fda84028590 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.831+0000 7fda82ffd700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6805/150599780 conn(0x7fda84028590 0x7fda84118d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.832+0000 7fda6bfff700 10 client.38213.objecter ms_handle_connect 0x7fda84028590 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.832+0000 7fda82ffd700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6805/150599780 conn(0x7fda84028590 0x7fda84118d30 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:40.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.832+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fda840f5e40 con 0x7fda84028590 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.832+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.832+0000 7fda6bfff700 10 client.38213.objecter _finish_command 3 = system:0 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.832+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda840f5cd0 osd=1 4 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.832+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 1 3 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.848+0000 7fda897ff700 10 client.38213.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.848+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 4 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.848+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda840f5cd0 osd=1 3 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.848+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda840f5cd0 osd=1 3 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.848+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda840f5cd0 osd=1 3 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.848+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 4 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.848+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda840f5cd0 osd=1 3 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.848+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 1 4 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.848+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda840f5cd0 osd=1 4 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.848+0000 7fda897ff700 10 client.38213.objecter _send_command 4 2024-02-13T00:53:40.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.848+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda8405a0e0 con 0x7fda84028590 2024-02-13T00:53:40.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.850+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fda8405a0e0 con 0x7fda84028590 2024-02-13T00:53:40.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.850+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:40.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.850+0000 7fda6bfff700 10 client.38213.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.850+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda840f5cd0 osd=1 4 2024-02-13T00:53:40.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.850+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 1 4 2024-02-13T00:53:40.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda897ff700 10 client.38213.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:40.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 5 2024-02-13T00:53:40.139 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6809/3689719376 conn(0x7fda84059570 0x7fda8411c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84059400 osd=2 3 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84059400 osd=2 3 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84059400 osd=2 3 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 5 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda84059400 osd=2 3 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 2 5 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84059400 osd=2 4 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda897ff700 10 client.38213.objecter _send_command 5 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda840280f0 con 0x7fda84059570 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda827fc700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6809/3689719376 conn(0x7fda84059570 0x7fda8411c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda6bfff700 10 client.38213.objecter ms_handle_connect 0x7fda84059570 2024-02-13T00:53:40.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.851+0000 7fda827fc700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6809/3689719376 conn(0x7fda84059570 0x7fda8411c3b0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.852+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fda840280f0 con 0x7fda84059570 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.852+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.852+0000 7fda6bfff700 10 client.38213.objecter _finish_command 5 = system:0 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.852+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda84059400 osd=2 4 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.852+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 2 5 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.875+0000 7fda897ff700 10 client.38213.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.875+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 6 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.875+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84059400 osd=2 3 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.875+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84059400 osd=2 3 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.875+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84059400 osd=2 3 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.875+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 6 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.875+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda84059400 osd=2 3 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.875+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 2 6 2024-02-13T00:53:40.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.875+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84059400 osd=2 4 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.875+0000 7fda897ff700 10 client.38213.objecter _send_command 6 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.875+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda8411c8a0 con 0x7fda84059570 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fda8411c8a0 con 0x7fda84059570 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda6bfff700 10 client.38213.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda84059400 osd=2 4 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 2 6 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda897ff700 10 client.38213.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 7 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6813/604971011 conn(0x7fda840277c0 0x7fda8411dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84027650 osd=3 3 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84027650 osd=3 3 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84027650 osd=3 3 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 7 2024-02-13T00:53:40.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda84027650 osd=3 3 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 3 7 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84027650 osd=3 4 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda897ff700 10 client.38213.objecter _send_command 7 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda8411a0e0 con 0x7fda840277c0 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda837fe700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6813/604971011 conn(0x7fda840277c0 0x7fda8411dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda6bfff700 10 client.38213.objecter ms_handle_connect 0x7fda840277c0 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.877+0000 7fda837fe700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6813/604971011 conn(0x7fda840277c0 0x7fda8411dc20 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.878+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fda8411a0e0 con 0x7fda840277c0 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.878+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.878+0000 7fda6bfff700 10 client.38213.objecter _finish_command 7 = system:0 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.878+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda84027650 osd=3 4 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.878+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 3 7 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.892+0000 7fda897ff700 10 client.38213.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:40.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.892+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 8 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.892+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84027650 osd=3 3 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.892+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84027650 osd=3 3 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.892+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84027650 osd=3 3 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.892+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 8 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.892+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda84027650 osd=3 3 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.892+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 3 8 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.892+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84027650 osd=3 4 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.892+0000 7fda897ff700 10 client.38213.objecter _send_command 8 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.892+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda84119460 con 0x7fda840277c0 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fda84119460 con 0x7fda840277c0 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda6bfff700 10 client.38213.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda84027650 osd=3 4 2024-02-13T00:53:40.144 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 3 8 2024-02-13T00:53:40.145 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:53:40.145 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:53:40.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda897ff700 10 client.38213.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:40.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 9 2024-02-13T00:53:40.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6800/3344263388 conn(0x7fda8411edf0 0x7fda84123250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:40.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84119460 osd=4 3 2024-02-13T00:53:40.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84119460 osd=4 3 2024-02-13T00:53:40.145 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84119460 osd=4 3 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 9 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda84119460 osd=4 3 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 4 9 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84119460 osd=4 4 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda897ff700 10 client.38213.objecter _send_command 9 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.894+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda841196c0 con 0x7fda8411edf0 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.895+0000 7fda82ffd700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6800/3344263388 conn(0x7fda8411edf0 0x7fda84123250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.895+0000 7fda82ffd700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6800/3344263388 conn(0x7fda8411edf0 0x7fda84123250 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.895+0000 7fda6bfff700 10 client.38213.objecter ms_handle_connect 0x7fda8411edf0 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.896+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fda841196c0 con 0x7fda8411edf0 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.896+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.896+0000 7fda6bfff700 10 client.38213.objecter _finish_command 9 = system:0 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.896+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda84119460 osd=4 4 2024-02-13T00:53:40.146 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.896+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 4 9 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.910+0000 7fda897ff700 10 client.38213.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.910+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 10 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.910+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84119460 osd=4 3 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.910+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84119460 osd=4 3 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.910+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84119460 osd=4 3 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.910+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 10 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.910+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda84119460 osd=4 3 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.910+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 4 10 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.910+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84119460 osd=4 4 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.910+0000 7fda897ff700 10 client.38213.objecter _send_command 10 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.910+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda841196c0 con 0x7fda8411edf0 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.912+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fda841196c0 con 0x7fda8411edf0 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda6bfff700 10 client.38213.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda84119460 osd=4 4 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 4 10 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda897ff700 10 client.38213.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 11 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6804/2277917372 conn(0x7fda84123c50 0x7fda84126040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda841196c0 osd=5 3 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda841196c0 osd=5 3 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda841196c0 osd=5 3 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 11 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda841196c0 osd=5 3 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 5 11 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda841196c0 osd=5 4 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda897ff700 10 client.38213.objecter _send_command 11 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda84128e10 con 0x7fda84123c50 2024-02-13T00:53:40.148 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda827fc700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6804/2277917372 conn(0x7fda84123c50 0x7fda84126040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:40.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda6bfff700 10 client.38213.objecter ms_handle_connect 0x7fda84123c50 2024-02-13T00:53:40.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.913+0000 7fda827fc700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6804/2277917372 conn(0x7fda84123c50 0x7fda84126040 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:40.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.914+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fda84128e10 con 0x7fda84123c50 2024-02-13T00:53:40.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.914+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:53:40.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.914+0000 7fda6bfff700 10 client.38213.objecter _finish_command 11 = system:0 2024-02-13T00:53:40.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.914+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda841196c0 osd=5 4 2024-02-13T00:53:40.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.914+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 5 11 2024-02-13T00:53:40.149 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:53:40.149 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.929+0000 7fda897ff700 10 client.38213.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.929+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 12 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.929+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda841196c0 osd=5 3 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.929+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda841196c0 osd=5 3 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.929+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda841196c0 osd=5 3 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.929+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 12 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.929+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda841196c0 osd=5 3 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.929+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 5 12 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.929+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda841196c0 osd=5 4 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.929+0000 7fda897ff700 10 client.38213.objecter _send_command 12 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.929+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda84128e10 con 0x7fda84123c50 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.930+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fda84128e10 con 0x7fda84123c50 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.930+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:40.150 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.930+0000 7fda6bfff700 10 client.38213.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.930+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda841196c0 osd=5 4 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.930+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 5 12 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda897ff700 10 client.38213.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 13 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6808/1506640341 conn(0x7fda84128150 0x7fda841285b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84128e10 osd=6 3 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84128e10 osd=6 3 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84128e10 osd=6 3 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 13 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda84128e10 osd=6 3 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 6 13 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84128e10 osd=6 4 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda897ff700 10 client.38213.objecter _send_command 13 2024-02-13T00:53:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda84128c20 con 0x7fda84128150 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda837fe700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6808/1506640341 conn(0x7fda84128150 0x7fda841285b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda6bfff700 10 client.38213.objecter ms_handle_connect 0x7fda84128150 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.931+0000 7fda837fe700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6808/1506640341 conn(0x7fda84128150 0x7fda841285b0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.932+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fda7c00f040 con 0x7fda84128150 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.932+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.932+0000 7fda6bfff700 10 client.38213.objecter _finish_command 13 = system:0 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.932+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda84128e10 osd=6 4 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.932+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 6 13 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.948+0000 7fda897ff700 10 client.38213.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.948+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 14 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.948+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84128e10 osd=6 3 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.948+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84128e10 osd=6 3 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.948+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda84128e10 osd=6 3 2024-02-13T00:53:40.152 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.948+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 14 2024-02-13T00:53:40.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.948+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda84128e10 osd=6 3 2024-02-13T00:53:40.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.948+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 6 14 2024-02-13T00:53:40.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.948+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84128e10 osd=6 4 2024-02-13T00:53:40.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.948+0000 7fda897ff700 10 client.38213.objecter _send_command 14 2024-02-13T00:53:40.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.948+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda8412b0c0 con 0x7fda84128150 2024-02-13T00:53:40.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fda8412b0c0 con 0x7fda84128150 2024-02-13T00:53:40.153 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:40.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda6bfff700 10 client.38213.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda84128e10 osd=6 4 2024-02-13T00:53:40.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 6 14 2024-02-13T00:53:40.154 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:53:40.155 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:53:40.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda897ff700 10 client.38213.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:40.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 15 2024-02-13T00:53:40.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6812/1964192772 conn(0x7fda8412b770 0x7fda8412bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:40.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda8412b0c0 osd=7 3 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda8412b0c0 osd=7 3 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda8412b0c0 osd=7 3 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 15 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda8412b0c0 osd=7 3 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 7 15 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda8412b0c0 osd=7 4 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda897ff700 10 client.38213.objecter _send_command 15 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.950+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda8412f480 con 0x7fda8412b770 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.951+0000 7fda82ffd700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6812/1964192772 conn(0x7fda8412b770 0x7fda8412bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.951+0000 7fda82ffd700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6812/1964192772 conn(0x7fda8412b770 0x7fda8412bbb0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.951+0000 7fda6bfff700 10 client.38213.objecter ms_handle_connect 0x7fda8412b770 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.952+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fda8412f480 con 0x7fda8412b770 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.952+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.952+0000 7fda6bfff700 10 client.38213.objecter _finish_command 15 = system:0 2024-02-13T00:53:40.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.952+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda8412b0c0 osd=7 4 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.952+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 7 15 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.966+0000 7fda897ff700 10 client.38213.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.966+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign -1 16 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.966+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda8412b0c0 osd=7 3 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.966+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda8412b0c0 osd=7 3 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.966+0000 7fda897ff700 20 client.38213.objecter _get_session s=0x7fda8412b0c0 osd=7 3 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.966+0000 7fda897ff700 15 client.38213.objecter _session_command_op_remove -1 16 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.966+0000 7fda897ff700 20 client.38213.objecter get_session s=0x7fda8412b0c0 osd=7 3 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.966+0000 7fda897ff700 15 client.38213.objecter _session_command_op_assign 7 16 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.966+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda8412b0c0 osd=7 4 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.966+0000 7fda897ff700 10 client.38213.objecter _send_command 16 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.966+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fda8412e300 con 0x7fda8412b770 2024-02-13T00:53:40.157 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda6bfff700 1 -- 172.21.15.162:0/3434816234 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fda8412e300 con 0x7fda8412b770 2024-02-13T00:53:40.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda6bfff700 10 client.38213.objecter ms_dispatch 0x7fda84001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:53:40.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda6bfff700 10 client.38213.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda6bfff700 20 client.38213.objecter put_session s=0x7fda8412b0c0 osd=7 4 2024-02-13T00:53:40.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda6bfff700 15 client.38213.objecter _session_command_op_remove 7 16 2024-02-13T00:53:40.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 10 librados: watch_flush enter 2024-02-13T00:53:40.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 10 librados: watch_flush exit 2024-02-13T00:53:40.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 10 client.38213.objecter close_session for osd.0 2024-02-13T00:53:40.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6801/2103743338 conn(0x7fda840fed70 msgr2=0x7fda840ff150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6801/2103743338 conn(0x7fda840fed70 0x7fda840ff150 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda840475e0 osd=0 1 2024-02-13T00:53:40.158 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 10 client.38213.objecter close_session for osd.1 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6805/150599780 conn(0x7fda84028590 msgr2=0x7fda84118d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6805/150599780 conn(0x7fda84028590 0x7fda84118d30 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda840f5cd0 osd=1 1 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 10 client.38213.objecter close_session for osd.2 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6809/3689719376 conn(0x7fda84059570 msgr2=0x7fda8411c3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6809/3689719376 conn(0x7fda84059570 0x7fda8411c3b0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84059400 osd=2 1 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 10 client.38213.objecter close_session for osd.3 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6813/604971011 conn(0x7fda840277c0 msgr2=0x7fda8411dc20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6813/604971011 conn(0x7fda840277c0 0x7fda8411dc20 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda837fe700 1 -- 172.21.15.162:0/3434816234 reap_dead start 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84027650 osd=3 1 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 10 client.38213.objecter close_session for osd.4 2024-02-13T00:53:40.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6800/3344263388 conn(0x7fda8411edf0 msgr2=0x7fda84123250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6800/3344263388 conn(0x7fda8411edf0 0x7fda84123250 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84119460 osd=4 1 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 10 client.38213.objecter close_session for osd.5 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6804/2277917372 conn(0x7fda84123c50 msgr2=0x7fda84126040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6804/2277917372 conn(0x7fda84123c50 0x7fda84126040 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda841196c0 osd=5 1 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 10 client.38213.objecter close_session for osd.6 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.968+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6808/1506640341 conn(0x7fda84128150 msgr2=0x7fda841285b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6808/1506640341 conn(0x7fda84128150 0x7fda841285b0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda84128e10 osd=6 1 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 10 client.38213.objecter close_session for osd.7 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6812/1964192772 conn(0x7fda8412b770 msgr2=0x7fda8412bbb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.160 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.181:6812/1964192772 conn(0x7fda8412b770 0x7fda8412bbb0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 20 client.38213.objecter put_session s=0x7fda8412b0c0 osd=7 1 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 20 client.38213.objecter shutdown clearing up homeless session... 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 10 client.38213.objecter successfully canceled tick 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6800/2866534053 conn(0x7fda6c062c90 msgr2=0x7fda6c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> v2:172.21.15.162:6800/2866534053 conn(0x7fda6c062c90 0x7fda6c065150 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 rx=0x7fda84109820 tx=0x7fda7400b040).stop 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda837fe700 1 -- 172.21.15.162:0/3434816234 reap_dead start 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda840f8310 msgr2=0x7fda8410dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 1 --2- 172.21.15.162:0/3434816234 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda840f8310 0x7fda8410dee0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 rx=0x7fda780044e0 tx=0x7fda78007f90).stop 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda837fe700 1 -- 172.21.15.162:0/3434816234 reap_dead start 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 shutdown_connections 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 >> 172.21.15.162:0/3434816234 conn(0x7fda840f36d0 msgr2=0x7fda840f5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 shutdown_connections 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 1 -- 172.21.15.162:0/3434816234 wait complete. 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:39.969+0000 7fda897ff700 1 librados: shutdown 2024-02-13T00:53:40.161 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.162 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.162 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.162 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.162 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.162 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.162 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.162 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:53:40.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.138+0000 7f7c8efde700 1 -- 172.21.15.162:0/196225305 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f7c88107170 con 0x7f7c880f8d10 2024-02-13T00:53:40.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:40.163 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:40.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:40.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:40.289 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:40.289 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:40.293 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:40.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:40.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:40.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:40.578 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.574+0000 7f7c85ffb700 1 -- 172.21.15.162:0/196225305 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v775) v1 ==== 91+0+0 (secure 0 0 0) 0x7f7c7c023070 con 0x7f7c880f8d10 2024-02-13T00:53:40.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 10 librados: watch_flush enter 2024-02-13T00:53:40.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 10 librados: watch_flush exit 2024-02-13T00:53:40.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 20 client.38219.objecter shutdown clearing up homeless session... 2024-02-13T00:53:40.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 10 client.38219.objecter successfully canceled tick 2024-02-13T00:53:40.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 1 -- 172.21.15.162:0/196225305 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7c78062c90 msgr2=0x7f7c78065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 1 --2- 172.21.15.162:0/196225305 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7c78062c90 0x7f7c78065150 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 rx=0x7f7c88109ad0 tx=0x7f7c7400b040).stop 2024-02-13T00:53:40.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 1 -- 172.21.15.162:0/196225305 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c880f8d10 msgr2=0x7f7c8810e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:40.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 1 --2- 172.21.15.162:0/196225305 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c880f8d10 0x7f7c8810e190 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 rx=0x7f7c7c009eb0 tx=0x7f7c7c00b040).stop 2024-02-13T00:53:40.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 1 -- 172.21.15.162:0/196225305 shutdown_connections 2024-02-13T00:53:40.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 1 --2- 172.21.15.162:0/196225305 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7c78062c90 0x7f7c78065150 unknown :-1 s=CLOSED pgs=3042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 1 --2- 172.21.15.162:0/196225305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c88102270 0x7f7c88108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 1 --2- 172.21.15.162:0/196225305 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7c880f8d10 0x7f7c8810e190 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 1 --2- 172.21.15.162:0/196225305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c880f95e0 0x7f7c8810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:40.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.576+0000 7f7c8efde700 1 -- 172.21.15.162:0/196225305 >> 172.21.15.162:0/196225305 conn(0x7f7c8806eb90 msgr2=0x7f7c88102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:40.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.577+0000 7f7c8efde700 1 -- 172.21.15.162:0/196225305 shutdown_connections 2024-02-13T00:53:40.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.577+0000 7f7c8efde700 1 -- 172.21.15.162:0/196225305 wait complete. 2024-02-13T00:53:40.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:40.577+0000 7f7c8efde700 1 librados: shutdown 2024-02-13T00:53:40.582 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:53:40.587 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:40.608 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:40.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:40.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:40.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:40.739 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:40.890 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:39.813342+0000 mon.c (mon.2) 510 : audit [DBG] from='client.? 172.21.15.162:0/3434816234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:40.890 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:40.139113+0000 mon.c (mon.2) 511 : audit [INF] from='client.? 172.21.15.162:0/196225305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:53:40.890 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:40.139403+0000 mon.a (mon.0) 3667 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:53:40.890 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:40.568481+0000 mon.a (mon.0) 3668 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:53:40.891 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:40.573329+0000 mon.a (mon.0) 3669 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:53:40.891 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:40.573403+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e775: 8 total, 8 up, 4 in 2024-02-13T00:53:40.891 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:39.813342+0000 mon.c (mon.2) 510 : audit [DBG] from='client.? 172.21.15.162:0/3434816234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:40.891 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:40.139113+0000 mon.c (mon.2) 511 : audit [INF] from='client.? 172.21.15.162:0/196225305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:53:40.891 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:40.139403+0000 mon.a (mon.0) 3667 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:53:40.891 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:40.568481+0000 mon.a (mon.0) 3668 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:53:40.892 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:40.573329+0000 mon.a (mon.0) 3669 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:53:40.892 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:40.573403+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e775: 8 total, 8 up, 4 in 2024-02-13T00:53:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:39.813342+0000 mon.c (mon.2) 510 : audit [DBG] from='client.? 172.21.15.162:0/3434816234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:40.139113+0000 mon.c (mon.2) 511 : audit [INF] from='client.? 172.21.15.162:0/196225305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:53:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:40.139403+0000 mon.a (mon.0) 3667 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:53:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:40.568481+0000 mon.a (mon.0) 3668 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:53:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:40.573329+0000 mon.a (mon.0) 3669 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:53:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:40.573403+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e775: 8 total, 8 up, 4 in 2024-02-13T00:53:41.408 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:41.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:41.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:41.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:41.544 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:41.567 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:41.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:41.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:41.687 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:40.542727+0000 mgr.y (mgr.14544) 642 : cluster [DBG] pgmap v1177: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:41.687 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:40.574615+0000 mon.a (mon.0) 3671 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-13T00:53:41.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:40.542727+0000 mgr.y (mgr.14544) 642 : cluster [DBG] pgmap v1177: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:41.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:40.574615+0000 mon.a (mon.0) 3671 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-13T00:53:41.699 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:41.718 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:41.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:41.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:40.542727+0000 mgr.y (mgr.14544) 642 : cluster [DBG] pgmap v1177: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:41.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:40.574615+0000 mon.a (mon.0) 3671 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-13T00:53:41.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:41.850 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:42.599 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:42.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:42.703 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:42.731 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:42.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:53:43.078 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:43.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:43.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:43.210 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:42.492903+0000 osd.6 (osd.6) 249 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:53:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:42.494003+0000 osd.6 (osd.6) 250 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:53:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:42.497983+0000 osd.1 (osd.1) 319 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:53:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:42.501146+0000 osd.1 (osd.1) 320 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:53:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:42.543120+0000 mgr.y (mgr.14544) 643 : cluster [DBG] pgmap v1179: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 857 B/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:43.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:42.492903+0000 osd.6 (osd.6) 249 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:53:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:42.494003+0000 osd.6 (osd.6) 250 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:53:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:42.497983+0000 osd.1 (osd.1) 319 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:53:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:42.501146+0000 osd.1 (osd.1) 320 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:53:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:42.543120+0000 mgr.y (mgr.14544) 643 : cluster [DBG] pgmap v1179: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 857 B/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:42.492903+0000 osd.6 (osd.6) 249 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:53:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:42.494003+0000 osd.6 (osd.6) 250 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:53:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:42.497983+0000 osd.1 (osd.1) 319 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:53:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:42.501146+0000 osd.1 (osd.1) 320 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:53:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:42.543120+0000 mgr.y (mgr.14544) 643 : cluster [DBG] pgmap v1179: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 857 B/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:44.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:53:44.259 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:44.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:44.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:44.384 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:53:44.384 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:53:44.393 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:44.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:43.502557+0000 osd.1 (osd.1) 321 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:53:44.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:43.505786+0000 osd.1 (osd.1) 322 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:53:44.918 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:43.502557+0000 osd.1 (osd.1) 321 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:53:44.918 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:43.505786+0000 osd.1 (osd.1) 322 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:53:44.941 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:44.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:44.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:43.502557+0000 osd.1 (osd.1) 321 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:53:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:43.505786+0000 osd.1 (osd.1) 322 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:53:45.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:45.077 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:45.098 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:45.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:45.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:45.233 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.244+0000 7fed86850700 1 -- 172.21.15.162:0/3878609076 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed8006ba80 msgr2=0x7fed80104090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.244+0000 7fed86850700 1 --2- 172.21.15.162:0/3878609076 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed8006ba80 0x7fed80104090 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 rx=0x7fed74004870 tx=0x7fed74009cb0).stop 2024-02-13T00:53:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.244+0000 7fed86850700 1 -- 172.21.15.162:0/3878609076 shutdown_connections 2024-02-13T00:53:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.244+0000 7fed86850700 1 --2- 172.21.15.162:0/3878609076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed80104580 0x7fed80106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.244+0000 7fed86850700 1 --2- 172.21.15.162:0/3878609076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed8006b1b0 0x7fed8006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.244+0000 7fed86850700 1 --2- 172.21.15.162:0/3878609076 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed8006ba80 0x7fed80104090 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.244+0000 7fed86850700 1 -- 172.21.15.162:0/3878609076 >> 172.21.15.162:0/3878609076 conn(0x7fed800f38c0 msgr2=0x7fed800f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.244+0000 7fed86850700 1 -- 172.21.15.162:0/3878609076 shutdown_connections 2024-02-13T00:53:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.244+0000 7fed86850700 1 -- 172.21.15.162:0/3878609076 wait complete. 2024-02-13T00:53:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed86850700 1 librados: starting msgr at 2024-02-13T00:53:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed86850700 1 librados: starting objecter 2024-02-13T00:53:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed86850700 1 Processor -- start 2024-02-13T00:53:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed86850700 1 -- start start 2024-02-13T00:53:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed86850700 1 librados: setting wanted keys 2024-02-13T00:53:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed86850700 1 librados: calling monclient init 2024-02-13T00:53:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed86850700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed8006b1b0 0x7fed80110620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed86850700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed8006ba80 0x7fed80110b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed86850700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed80104580 0x7fed8010a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed86850700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8010acf0 con 0x7fed8006ba80 2024-02-13T00:53:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed86850700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8010ae60 con 0x7fed8006b1b0 2024-02-13T00:53:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed86850700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8010b020 con 0x7fed80104580 2024-02-13T00:53:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed7f7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed8006ba80 0x7fed80110b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed7f7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed8006ba80 0x7fed80110b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60380/0 (socket says 172.21.15.162:60380) 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed7f7fe700 1 -- 172.21.15.162:0/3258030835 learned_addr learned my addr 172.21.15.162:0/3258030835 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed84ded700 1 --2- 172.21.15.162:0/3258030835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed80104580 0x7fed8010a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.245+0000 7fed7ffff700 1 --2- 172.21.15.162:0/3258030835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed8006b1b0 0x7fed80110620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed7f7fe700 1 -- 172.21.15.162:0/3258030835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed80104580 msgr2=0x7fed8010a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed7f7fe700 1 --2- 172.21.15.162:0/3258030835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed80104580 0x7fed8010a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed7f7fe700 1 -- 172.21.15.162:0/3258030835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed8006b1b0 msgr2=0x7fed80110620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed7f7fe700 1 --2- 172.21.15.162:0/3258030835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed8006b1b0 0x7fed80110620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed7f7fe700 1 -- 172.21.15.162:0/3258030835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed70005040 con 0x7fed8006ba80 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed84ded700 1 --2- 172.21.15.162:0/3258030835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed80104580 0x7fed8010a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed7ffff700 1 --2- 172.21.15.162:0/3258030835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed8006b1b0 0x7fed80110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed7f7fe700 1 --2- 172.21.15.162:0/3258030835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed8006ba80 0x7fed80110b30 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7fed74004ee0 tx=0x7fed7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed7d7fa700 10 client.?.objecter ms_handle_connect 0x7fed8006ba80 2024-02-13T00:53:45.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed7d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:45.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed7d7fa700 1 -- 172.21.15.162:0/3258030835 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7400e030 con 0x7fed8006ba80 2024-02-13T00:53:45.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed7d7fa700 1 -- 172.21.15.162:0/3258030835 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed74004160 con 0x7fed8006ba80 2024-02-13T00:53:45.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed86850700 1 -- 172.21.15.162:0/3258030835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed74005040 con 0x7fed8006ba80 2024-02-13T00:53:45.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed7d7fa700 1 -- 172.21.15.162:0/3258030835 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74019070 con 0x7fed8006ba80 2024-02-13T00:53:45.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed86850700 10 client.28128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:45.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.246+0000 7fed86850700 1 -- 172.21.15.162:0/3258030835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8010b5a0 con 0x7fed8006ba80 2024-02-13T00:53:45.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.247+0000 7fed86850700 1 librados: init done 2024-02-13T00:53:45.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.248+0000 7fed86850700 1 -- 172.21.15.162:0/3258030835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed800475e0 con 0x7fed8006ba80 2024-02-13T00:53:45.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.248+0000 7fed7d7fa700 1 -- 172.21.15.162:0/3258030835 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fed740090d0 con 0x7fed8006ba80 2024-02-13T00:53:45.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.248+0000 7fed7d7fa700 1 --2- 172.21.15.162:0/3258030835 >> v2:172.21.15.162:6800/2866534053 conn(0x7fed68062c90 0x7fed68065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:45.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.248+0000 7fed7d7fa700 1 -- 172.21.15.162:0/3258030835 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(775..775 src has 449..775) v4 ==== 8682+0+0 (secure 0 0 0) 0x7fed7407e0d0 con 0x7fed8006ba80 2024-02-13T00:53:45.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.248+0000 7fed7d7fa700 10 client.28128.objecter ms_dispatch 0x7fed80001550 osd_map(775..775 src has 449..775) v4 2024-02-13T00:53:45.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.248+0000 7fed7d7fa700 3 client.28128.objecter handle_osd_map got epochs [775,775] > 0 2024-02-13T00:53:45.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.248+0000 7fed7d7fa700 3 client.28128.objecter handle_osd_map decoding full epoch 775 2024-02-13T00:53:45.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.248+0000 7fed7ffff700 1 --2- 172.21.15.162:0/3258030835 >> v2:172.21.15.162:6800/2866534053 conn(0x7fed68062c90 0x7fed68065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:45.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.248+0000 7fed7d7fa700 20 client.28128.objecter dump_active .. 0 homeless 2024-02-13T00:53:45.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.249+0000 7fed7d7fa700 10 client.28128.objecter ms_handle_connect 0x7fed68062c90 2024-02-13T00:53:45.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.249+0000 7fed7ffff700 1 --2- 172.21.15.162:0/3258030835 >> v2:172.21.15.162:6800/2866534053 conn(0x7fed68062c90 0x7fed68065150 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 rx=0x7fed80105f60 tx=0x7fed7000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:45.255 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:45.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:45.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.253+0000 7fed7d7fa700 1 -- 172.21.15.162:0/3258030835 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fed740513b0 con 0x7fed8006ba80 2024-02-13T00:53:45.358 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:45.385 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:45.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.437+0000 7fed86850700 1 -- 172.21.15.162:0/3258030835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd rm-pg-upmap", "pgid": "6.23"} v 0) v1 -- 0x7fed80109470 con 0x7fed8006ba80 2024-02-13T00:53:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.599+0000 7fed7d7fa700 1 -- 172.21.15.162:0/3258030835 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap", "pgid": "6.23"}]=0 clear 6.23 pg_upmap mapping v776) v1 ==== 106+0+0 (secure 0 0 0) 0x7fed74050fa0 con 0x7fed8006ba80 2024-02-13T00:53:45.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.604+0000 7fed86850700 10 librados: watch_flush enter 2024-02-13T00:53:45.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.604+0000 7fed86850700 10 librados: watch_flush exit 2024-02-13T00:53:45.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.604+0000 7fed86850700 20 client.28128.objecter shutdown clearing up homeless session... 2024-02-13T00:53:45.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.604+0000 7fed86850700 10 client.28128.objecter successfully canceled tick 2024-02-13T00:53:45.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 -- 172.21.15.162:0/3258030835 >> v2:172.21.15.162:6800/2866534053 conn(0x7fed68062c90 msgr2=0x7fed68065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 --2- 172.21.15.162:0/3258030835 >> v2:172.21.15.162:6800/2866534053 conn(0x7fed68062c90 0x7fed68065150 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 rx=0x7fed80105f60 tx=0x7fed7000b040).stop 2024-02-13T00:53:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 -- 172.21.15.162:0/3258030835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed8006ba80 msgr2=0x7fed80110b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 --2- 172.21.15.162:0/3258030835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed8006ba80 0x7fed80110b30 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7fed74004ee0 tx=0x7fed7400b040).stop 2024-02-13T00:53:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 -- 172.21.15.162:0/3258030835 shutdown_connections 2024-02-13T00:53:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 --2- 172.21.15.162:0/3258030835 >> v2:172.21.15.162:6800/2866534053 conn(0x7fed68062c90 0x7fed68065150 unknown :-1 s=CLOSED pgs=3043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 --2- 172.21.15.162:0/3258030835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed80104580 0x7fed8010a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 --2- 172.21.15.162:0/3258030835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed8006b1b0 0x7fed80110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 --2- 172.21.15.162:0/3258030835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed8006ba80 0x7fed80110b30 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 -- 172.21.15.162:0/3258030835 >> 172.21.15.162:0/3258030835 conn(0x7fed800f38c0 msgr2=0x7fed800f4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 -- 172.21.15.162:0/3258030835 shutdown_connections 2024-02-13T00:53:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 -- 172.21.15.162:0/3258030835 wait complete. 2024-02-13T00:53:45.610 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:45.606+0000 7fed86850700 1 librados: shutdown 2024-02-13T00:53:45.611 INFO:teuthology.orchestra.run.smithi162.stderr:clear 6.23 pg_upmap mapping 2024-02-13T00:53:45.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-13T00:53:45.967 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:44.543598+0000 mgr.y (mgr.14544) 644 : cluster [DBG] pgmap v1180: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 734 B/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:45.967 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:45.107698+0000 osd.0 (osd.0) 307 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:53:45.967 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:45.110578+0000 osd.0 (osd.0) 308 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:53:45.968 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:45.438206+0000 mon.a (mon.0) 3672 : audit [INF] from='client.? 172.21.15.162:0/3258030835' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.23"}]: dispatch 2024-02-13T00:53:46.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:44.543598+0000 mgr.y (mgr.14544) 644 : cluster [DBG] pgmap v1180: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 734 B/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:46.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:45.107698+0000 osd.0 (osd.0) 307 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:53:46.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:45.110578+0000 osd.0 (osd.0) 308 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:53:46.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:45.438206+0000 mon.a (mon.0) 3672 : audit [INF] from='client.? 172.21.15.162:0/3258030835' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.23"}]: dispatch 2024-02-13T00:53:46.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:44.543598+0000 mgr.y (mgr.14544) 644 : cluster [DBG] pgmap v1180: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 734 B/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:46.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:45.107698+0000 osd.0 (osd.0) 307 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:53:46.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:45.110578+0000 osd.0 (osd.0) 308 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:53:46.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:45.438206+0000 mon.a (mon.0) 3672 : audit [INF] from='client.? 172.21.15.162:0/3258030835' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.23"}]: dispatch 2024-02-13T00:53:46.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:46.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:46.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:46.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:53:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:53:46.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:46.363 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:46.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:53:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:53:46] "GET /metrics HTTP/1.1" 200 190202 "" "Prometheus/2.33.4" 2024-02-13T00:53:46.848 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:46.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:46.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:46.867 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:46.888 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:46.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:46.978 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:45.598513+0000 mon.a (mon.0) 3673 : audit [INF] from='client.? 172.21.15.162:0/3258030835' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.23"}]': finished 2024-02-13T00:53:46.978 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:45.598571+0000 mon.a (mon.0) 3674 : cluster [DBG] osdmap e776: 8 total, 8 up, 4 in 2024-02-13T00:53:46.978 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:45.723929+0000 osd.7 (osd.7) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:46.978 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:45.724099+0000 osd.7 (osd.7) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:46.978 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:45.964238+0000 mgr.y (mgr.14544) 645 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:46.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:45.598513+0000 mon.a (mon.0) 3673 : audit [INF] from='client.? 172.21.15.162:0/3258030835' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.23"}]': finished 2024-02-13T00:53:46.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:45.598571+0000 mon.a (mon.0) 3674 : cluster [DBG] osdmap e776: 8 total, 8 up, 4 in 2024-02-13T00:53:46.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:45.723929+0000 osd.7 (osd.7) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:46.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:45.724099+0000 osd.7 (osd.7) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:46.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:45.964238+0000 mgr.y (mgr.14544) 645 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:46.980 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:45.598513+0000 mon.a (mon.0) 3673 : audit [INF] from='client.? 172.21.15.162:0/3258030835' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.23"}]': finished 2024-02-13T00:53:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:45.598571+0000 mon.a (mon.0) 3674 : cluster [DBG] osdmap e776: 8 total, 8 up, 4 in 2024-02-13T00:53:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:45.723929+0000 osd.7 (osd.7) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:53:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:45.724099+0000 osd.7 (osd.7) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:53:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:45.964238+0000 mgr.y (mgr.14544) 645 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:47.129 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:47.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:47.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:47.260 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:47.384 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:47.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:47.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:47.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:47.520 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:47.542 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:47.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:47.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:47.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:47.676 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:47.696 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:47.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:47.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:46.544140+0000 mgr.y (mgr.14544) 646 : cluster [DBG] pgmap v1182: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:47.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:46.610466+0000 mon.a (mon.0) 3675 : cluster [DBG] osdmap e777: 8 total, 8 up, 4 in 2024-02-13T00:53:47.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:47.825 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:46.544140+0000 mgr.y (mgr.14544) 646 : cluster [DBG] pgmap v1182: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:47.825 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:46.610466+0000 mon.a (mon.0) 3675 : cluster [DBG] osdmap e777: 8 total, 8 up, 4 in 2024-02-13T00:53:47.825 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:46.544140+0000 mgr.y (mgr.14544) 646 : cluster [DBG] pgmap v1182: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:47.825 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:46.610466+0000 mon.a (mon.0) 3675 : cluster [DBG] osdmap e777: 8 total, 8 up, 4 in 2024-02-13T00:53:47.827 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:47.950 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:47.950 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:47.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:48.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:48.084 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:48.105 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:48.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:48.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:48.238 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:48.258 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:48.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:48.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:48.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:48.391 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:48.410 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:48.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:48.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:48.544 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:48.566 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:48.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:48.671 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:48.698 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:48.795 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:53:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:53:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:53:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:47.397196+0000 osd.6 (osd.6) 251 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:53:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:47.401914+0000 osd.6 (osd.6) 252 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:53:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:47.474306+0000 osd.1 (osd.1) 323 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:53:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:47.479729+0000 osd.1 (osd.1) 324 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:53:48.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:48.589008+0000 mon.a (mon.0) 3676 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:48.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:47.397196+0000 osd.6 (osd.6) 251 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:53:48.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:47.401914+0000 osd.6 (osd.6) 252 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:53:48.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:47.474306+0000 osd.1 (osd.1) 323 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:53:48.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:47.479729+0000 osd.1 (osd.1) 324 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:53:48.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:48.589008+0000 mon.a (mon.0) 3676 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:47.397196+0000 osd.6 (osd.6) 251 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:53:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:47.401914+0000 osd.6 (osd.6) 252 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:53:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:47.474306+0000 osd.1 (osd.1) 323 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:53:48.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:47.479729+0000 osd.1 (osd.1) 324 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:53:48.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:48.589008+0000 mon.a (mon.0) 3676 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:49.447 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:49.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:49.551 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:49.583 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:49.604 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:49.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:49.683 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:48.406998+0000 osd.6 (osd.6) 253 : cluster [DBG] 4.8 scrub starts 2024-02-13T00:53:49.683 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:48.410460+0000 osd.6 (osd.6) 254 : cluster [DBG] 4.8 scrub ok 2024-02-13T00:53:49.683 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:48.544567+0000 mgr.y (mgr.14544) 647 : cluster [DBG] pgmap v1184: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 642 B/s rd, 0 op/s 2024-02-13T00:53:49.683 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:48.616752+0000 mon.a (mon.0) 3677 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:53:49.683 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:48.616828+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e778: 8 total, 8 up, 4 in 2024-02-13T00:53:49.683 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:48.617381+0000 mon.a (mon.0) 3679 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136"}]: dispatch 2024-02-13T00:53:49.683 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:49.060937+0000 osd.0 (osd.0) 309 : cluster [DBG] 6.10 scrub starts 2024-02-13T00:53:49.683 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:49.062639+0000 osd.0 (osd.0) 310 : cluster [DBG] 6.10 scrub ok 2024-02-13T00:53:49.683 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:49.344892+0000 mon.a (mon.0) 3680 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:49.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:48.406998+0000 osd.6 (osd.6) 253 : cluster [DBG] 4.8 scrub starts 2024-02-13T00:53:49.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:48.410460+0000 osd.6 (osd.6) 254 : cluster [DBG] 4.8 scrub ok 2024-02-13T00:53:49.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:48.544567+0000 mgr.y (mgr.14544) 647 : cluster [DBG] pgmap v1184: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 642 B/s rd, 0 op/s 2024-02-13T00:53:49.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:48.616752+0000 mon.a (mon.0) 3677 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:53:49.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:48.616828+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e778: 8 total, 8 up, 4 in 2024-02-13T00:53:49.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:48.617381+0000 mon.a (mon.0) 3679 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136"}]: dispatch 2024-02-13T00:53:49.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:49.060937+0000 osd.0 (osd.0) 309 : cluster [DBG] 6.10 scrub starts 2024-02-13T00:53:49.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:49.062639+0000 osd.0 (osd.0) 310 : cluster [DBG] 6.10 scrub ok 2024-02-13T00:53:49.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:49.344892+0000 mon.a (mon.0) 3680 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:49.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:49.739 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:49.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:49.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:49.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:50.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:48.406998+0000 osd.6 (osd.6) 253 : cluster [DBG] 4.8 scrub starts 2024-02-13T00:53:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:48.410460+0000 osd.6 (osd.6) 254 : cluster [DBG] 4.8 scrub ok 2024-02-13T00:53:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:48.544567+0000 mgr.y (mgr.14544) 647 : cluster [DBG] pgmap v1184: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 642 B/s rd, 0 op/s 2024-02-13T00:53:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:48.616752+0000 mon.a (mon.0) 3677 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:53:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:48.616828+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e778: 8 total, 8 up, 4 in 2024-02-13T00:53:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:48.617381+0000 mon.a (mon.0) 3679 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136"}]: dispatch 2024-02-13T00:53:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:49.060937+0000 osd.0 (osd.0) 309 : cluster [DBG] 6.10 scrub starts 2024-02-13T00:53:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:49.062639+0000 osd.0 (osd.0) 310 : cluster [DBG] 6.10 scrub ok 2024-02-13T00:53:50.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:49.344892+0000 mon.a (mon.0) 3680 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:53:50.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:50.124 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:50.144 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:50.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:50.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:50.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:50.280 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:50.300 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:50.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:50.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:50.431 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:50.484 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:50.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:50.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.476+0000 7f30a696e700 1 -- 172.21.15.162:0/3884273375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a006b210 msgr2=0x7f30a006b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.476+0000 7f30a696e700 1 --2- 172.21.15.162:0/3884273375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a006b210 0x7f30a006b5f0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 rx=0x7f30900048a0 tx=0x7f3090009cb0).stop 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.476+0000 7f30a696e700 1 -- 172.21.15.162:0/3884273375 shutdown_connections 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.476+0000 7f30a696e700 1 --2- 172.21.15.162:0/3884273375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a0108cf0 0x7f30a010b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.476+0000 7f30a696e700 1 --2- 172.21.15.162:0/3884273375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a006b210 0x7f30a006b5f0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.476+0000 7f30a696e700 1 --2- 172.21.15.162:0/3884273375 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a006bae0 0x7f30a0108800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.476+0000 7f30a696e700 1 -- 172.21.15.162:0/3884273375 >> 172.21.15.162:0/3884273375 conn(0x7f30a00f38c0 msgr2=0x7f30a00f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.476+0000 7f30a696e700 1 -- 172.21.15.162:0/3884273375 shutdown_connections 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.476+0000 7f30a696e700 1 -- 172.21.15.162:0/3884273375 wait complete. 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.477+0000 7f30a696e700 1 librados: starting msgr at 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.477+0000 7f30a696e700 1 librados: starting objecter 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.477+0000 7f30a696e700 1 Processor -- start 2024-02-13T00:53:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.477+0000 7f30a696e700 1 -- start start 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.477+0000 7f30a696e700 1 librados: setting wanted keys 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.477+0000 7f30a696e700 1 librados: calling monclient init 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f30a696e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a006b210 0x7f30a0114940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f30a696e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a006bae0 0x7f30a0114e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f30a696e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a0108cf0 0x7f30a010eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f30a696e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a010efd0 con 0x7f30a0108cf0 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f30a696e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a010f110 con 0x7f30a006b210 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f30a696e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a010f2d0 con 0x7f30a006bae0 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f30a4f0b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a0108cf0 0x7f30a010eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f309f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a006bae0 0x7f30a0114e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f30a4f0b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a0108cf0 0x7f30a010eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60390/0 (socket says 172.21.15.162:60390) 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f309f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a006bae0 0x7f30a0114e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:51528/0 (socket says 172.21.15.162:51528) 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f30a4f0b700 1 -- 172.21.15.162:0/1596972258 learned_addr learned my addr 172.21.15.162:0/1596972258 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:50.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f309ffff700 1 --2- 172.21.15.162:0/1596972258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a006b210 0x7f30a0114940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f309f7fe700 1 -- 172.21.15.162:0/1596972258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a006b210 msgr2=0x7f30a0114940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f309f7fe700 1 --2- 172.21.15.162:0/1596972258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30a006b210 0x7f30a0114940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f309f7fe700 1 -- 172.21.15.162:0/1596972258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a0108cf0 msgr2=0x7f30a010eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f309f7fe700 1 --2- 172.21.15.162:0/1596972258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a0108cf0 0x7f30a010eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f309f7fe700 1 -- 172.21.15.162:0/1596972258 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3094005040 con 0x7f30a006bae0 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f30a4f0b700 1 --2- 172.21.15.162:0/1596972258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a0108cf0 0x7f30a010eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f309f7fe700 1 --2- 172.21.15.162:0/1596972258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a006bae0 0x7f30a0114e30 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 rx=0x7f3094004920 tx=0x7f3094009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f309d7fa700 10 client.?.objecter ms_handle_connect 0x7f30a006bae0 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f309d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.478+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309400e030 con 0x7f30a006bae0 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.479+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3090005040 con 0x7f30a006bae0 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.479+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f309400f040 con 0x7f30a006bae0 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.479+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094014850 con 0x7f30a006bae0 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.479+0000 7f30a696e700 10 client.38231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.479+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a010f910 con 0x7f30a006bae0 2024-02-13T00:53:50.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.479+0000 7f30a696e700 1 librados: init done 2024-02-13T00:53:50.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.479+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f30a00475e0 con 0x7f30a006bae0 2024-02-13T00:53:50.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f3094010500 con 0x7f30a006bae0 2024-02-13T00:53:50.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f309d7fa700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3088062c90 0x7f3088065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(779..779 src has 449..779) v4 ==== 8615+0+0 (secure 0 0 0) 0x7f309407de50 con 0x7f30a006bae0 2024-02-13T00:53:50.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 osd_map(779..779 src has 449..779) v4 2024-02-13T00:53:50.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f309d7fa700 3 client.38231.objecter handle_osd_map got epochs [779,779] > 0 2024-02-13T00:53:50.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f309d7fa700 3 client.38231.objecter handle_osd_map decoding full epoch 779 2024-02-13T00:53:50.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f309ffff700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3088062c90 0x7f3088065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f309d7fa700 20 client.38231.objecter dump_active .. 0 homeless 2024-02-13T00:53:50.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v779) v1 ==== 54+0+16 (secure 0 0 0) 0x7f309407e2f0 con 0x7f30a006bae0 2024-02-13T00:53:50.624 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:53:50.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f30a696e700 10 client.38231.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:50.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 1 2024-02-13T00:53:50.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f309d7fa700 10 client.38231.objecter ms_handle_connect 0x7f3088062c90 2024-02-13T00:53:50.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f309ffff700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3088062c90 0x7f3088065150 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 rx=0x7f3090004870 tx=0x7f309000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:50.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.481+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6801/2103743338 conn(0x7f30a0059990 0x7f30a0059d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.482+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a010ffc0 osd=0 3 2024-02-13T00:53:50.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.482+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a010ffc0 osd=0 3 2024-02-13T00:53:50.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.482+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a010ffc0 osd=0 3 2024-02-13T00:53:50.624 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.482+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 1 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.482+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a010ffc0 osd=0 3 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.482+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 0 1 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.482+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a010ffc0 osd=0 4 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.482+0000 7f30a696e700 10 client.38231.objecter _send_command 1 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.482+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30a010d830 con 0x7f30a0059990 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.482+0000 7f30a4f0b700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6801/2103743338 conn(0x7f30a0059990 0x7f30a0059d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.482+0000 7f309d7fa700 10 client.38231.objecter ms_handle_connect 0x7f30a0059990 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.482+0000 7f30a4f0b700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6801/2103743338 conn(0x7f30a0059990 0x7f30a0059d70 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.483+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30a010d830 con 0x7f30a0059990 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.483+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.483+0000 7f309d7fa700 10 client.38231.objecter _finish_command 1 = system:0 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.483+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a010ffc0 osd=0 4 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.483+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 0 1 2024-02-13T00:53:50.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.497+0000 7f30a696e700 10 client.38231.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.497+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 2 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.497+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a010ffc0 osd=0 3 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.497+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a010ffc0 osd=0 3 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.497+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a010ffc0 osd=0 3 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.497+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 2 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.497+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a010ffc0 osd=0 3 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.497+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 0 2 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.497+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a010ffc0 osd=0 4 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.497+0000 7f30a696e700 10 client.38231.objecter _send_command 2 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.497+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f30a0028590 con 0x7f30a0059990 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f30a0028590 con 0x7f30a0059990 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f309d7fa700 10 client.38231.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a010ffc0 osd=0 4 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 0 2 2024-02-13T00:53:50.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f30a696e700 10 client.38231.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 3 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6805/150599780 conn(0x7f30a0059570 0x7f30a011f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a0059400 osd=1 3 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0059400 osd=1 3 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a0059400 osd=1 3 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 3 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a0059400 osd=1 3 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 1 3 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0059400 osd=1 4 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f30a696e700 10 client.38231.objecter _send_command 3 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.499+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30a00fa780 con 0x7f30a0059570 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.500+0000 7f309ffff700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6805/150599780 conn(0x7f30a0059570 0x7f30a011f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.500+0000 7f309d7fa700 10 client.38231.objecter ms_handle_connect 0x7f30a0059570 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.500+0000 7f309ffff700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6805/150599780 conn(0x7f30a0059570 0x7f30a011f7c0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.500+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30a00fa780 con 0x7f30a0059570 2024-02-13T00:53:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.500+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:53:50.628 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:53:50.628 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:53:50.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.500+0000 7f309d7fa700 10 client.38231.objecter _finish_command 3 = system:0 2024-02-13T00:53:50.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.500+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a0059400 osd=1 4 2024-02-13T00:53:50.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.500+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 1 3 2024-02-13T00:53:50.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.515+0000 7f30a696e700 10 client.38231.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:50.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.515+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 4 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.515+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a0059400 osd=1 3 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.515+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0059400 osd=1 3 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.515+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a0059400 osd=1 3 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.515+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 4 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.515+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a0059400 osd=1 3 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.515+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 1 4 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.515+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0059400 osd=1 4 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.515+0000 7f30a696e700 10 client.38231.objecter _send_command 4 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.515+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f30a011fcb0 con 0x7f30a0059570 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f30a011fcb0 con 0x7f30a0059570 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f309d7fa700 10 client.38231.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a0059400 osd=1 4 2024-02-13T00:53:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 1 4 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f30a696e700 10 client.38231.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 5 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6809/3689719376 conn(0x7f30a00f9e50 0x7f30a0123010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a00f9ce0 osd=2 3 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a00f9ce0 osd=2 3 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a00f9ce0 osd=2 3 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 5 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a00f9ce0 osd=2 3 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 2 5 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a00f9ce0 osd=2 4 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f30a696e700 10 client.38231.objecter _send_command 5 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30a00280f0 con 0x7f30a00f9e50 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f309f7fe700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6809/3689719376 conn(0x7f30a00f9e50 0x7f30a0123010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f309d7fa700 10 client.38231.objecter ms_handle_connect 0x7f30a00f9e50 2024-02-13T00:53:50.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.517+0000 7f309f7fe700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6809/3689719376 conn(0x7f30a00f9e50 0x7f30a0123010 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.518+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30a00280f0 con 0x7f30a00f9e50 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.518+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.518+0000 7f309d7fa700 10 client.38231.objecter _finish_command 5 = system:0 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.518+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a00f9ce0 osd=2 4 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.518+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 2 5 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.532+0000 7f30a696e700 10 client.38231.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.532+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 6 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.532+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a00f9ce0 osd=2 3 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.532+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a00f9ce0 osd=2 3 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.532+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a00f9ce0 osd=2 3 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.532+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 6 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.532+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a00f9ce0 osd=2 3 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.532+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 2 6 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.532+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a00f9ce0 osd=2 4 2024-02-13T00:53:50.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.532+0000 7f30a696e700 10 client.38231.objecter _send_command 6 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.532+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f30a0123500 con 0x7f30a00f9e50 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f30a0123500 con 0x7f30a00f9e50 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f309d7fa700 10 client.38231.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a00f9ce0 osd=2 4 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 2 6 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a696e700 10 client.38231.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 7 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6813/604971011 conn(0x7f30a00277c0 0x7f30a0126800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a0027650 osd=3 3 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0027650 osd=3 3 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a0027650 osd=3 3 2024-02-13T00:53:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 7 2024-02-13T00:53:50.633 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:53:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a0027650 osd=3 3 2024-02-13T00:53:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 3 7 2024-02-13T00:53:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0027650 osd=3 4 2024-02-13T00:53:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a696e700 10 client.38231.objecter _send_command 7 2024-02-13T00:53:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30a0120b00 con 0x7f30a00277c0 2024-02-13T00:53:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a4f0b700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6813/604971011 conn(0x7f30a00277c0 0x7f30a0126800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f309d7fa700 10 client.38231.objecter ms_handle_connect 0x7f30a00277c0 2024-02-13T00:53:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.534+0000 7f30a4f0b700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6813/604971011 conn(0x7f30a00277c0 0x7f30a0126800 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.535+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30a0120b00 con 0x7f30a00277c0 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.535+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.535+0000 7f309d7fa700 10 client.38231.objecter _finish_command 7 = system:0 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.535+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a0027650 osd=3 4 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.535+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 3 7 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.550+0000 7f30a696e700 10 client.38231.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.550+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 8 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.550+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a0027650 osd=3 3 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.550+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0027650 osd=3 3 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.550+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a0027650 osd=3 3 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.550+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 8 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.550+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a0027650 osd=3 3 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.550+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 3 8 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.550+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0027650 osd=3 4 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.550+0000 7f30a696e700 10 client.38231.objecter _send_command 8 2024-02-13T00:53:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.550+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f30a0126cf0 con 0x7f30a00277c0 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f30a0126cf0 con 0x7f30a00277c0 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f309d7fa700 10 client.38231.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a0027650 osd=3 4 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 3 8 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f30a696e700 10 client.38231.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 9 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6800/3344263388 conn(0x7f30a0127400 0x7f30a01297f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a0126cf0 osd=4 3 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0126cf0 osd=4 3 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a0126cf0 osd=4 3 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 9 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a0126cf0 osd=4 3 2024-02-13T00:53:50.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 4 9 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0126cf0 osd=4 4 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f30a696e700 10 client.38231.objecter _send_command 9 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.551+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30a012c4b0 con 0x7f30a0127400 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.552+0000 7f309ffff700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6800/3344263388 conn(0x7f30a0127400 0x7f30a01297f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.552+0000 7f309ffff700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6800/3344263388 conn(0x7f30a0127400 0x7f30a01297f0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.552+0000 7f309d7fa700 10 client.38231.objecter ms_handle_connect 0x7f30a0127400 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.553+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30a012c4b0 con 0x7f30a0127400 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.553+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.553+0000 7f309d7fa700 10 client.38231.objecter _finish_command 9 = system:0 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.553+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a0126cf0 osd=4 4 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.553+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 4 9 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.568+0000 7f30a696e700 10 client.38231.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.568+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 10 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.568+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a0126cf0 osd=4 3 2024-02-13T00:53:50.636 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.568+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0126cf0 osd=4 3 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.568+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a0126cf0 osd=4 3 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.568+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 10 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.568+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a0126cf0 osd=4 3 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.568+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 4 10 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.568+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0126cf0 osd=4 4 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.568+0000 7f30a696e700 10 client.38231.objecter _send_command 10 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.568+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f30a012c4b0 con 0x7f30a0127400 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f30a012c4b0 con 0x7f30a0127400 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f309d7fa700 10 client.38231.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a0126cf0 osd=4 4 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:53:50.637 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:53:50.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 4 10 2024-02-13T00:53:50.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f30a696e700 10 client.38231.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:50.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 11 2024-02-13T00:53:50.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6804/2277917372 conn(0x7f30a012a230 0x7f30a012a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a012c4b0 osd=5 3 2024-02-13T00:53:50.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012c4b0 osd=5 3 2024-02-13T00:53:50.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a012c4b0 osd=5 3 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 11 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a012c4b0 osd=5 3 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 5 11 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012c4b0 osd=5 4 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f30a696e700 10 client.38231.objecter _send_command 11 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.570+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30a012f890 con 0x7f30a012a230 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.571+0000 7f309f7fe700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6804/2277917372 conn(0x7f30a012a230 0x7f30a012a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.571+0000 7f309d7fa700 10 client.38231.objecter ms_handle_connect 0x7f30a012a230 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.571+0000 7f309f7fe700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6804/2277917372 conn(0x7f30a012a230 0x7f30a012a690 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.572+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30a012f890 con 0x7f30a012a230 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.572+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.572+0000 7f309d7fa700 10 client.38231.objecter _finish_command 11 = system:0 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.572+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a012c4b0 osd=5 4 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.572+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 5 11 2024-02-13T00:53:50.639 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.593+0000 7f30a696e700 10 client.38231.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.593+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 12 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.593+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a012c4b0 osd=5 3 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.593+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012c4b0 osd=5 3 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.593+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a012c4b0 osd=5 3 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.593+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 12 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.593+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a012c4b0 osd=5 3 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.593+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 5 12 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.593+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012c4b0 osd=5 4 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.593+0000 7f30a696e700 10 client.38231.objecter _send_command 12 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.593+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f30a012ac40 con 0x7f30a012a230 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.595+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f30a012ac40 con 0x7f30a012a230 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.595+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.595+0000 7f309d7fa700 10 client.38231.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.595+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a012c4b0 osd=5 4 2024-02-13T00:53:50.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.595+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 5 12 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a696e700 10 client.38231.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 13 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6808/1506640341 conn(0x7f30a012b350 0x7f30a012b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a012ac40 osd=6 3 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012ac40 osd=6 3 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a012ac40 osd=6 3 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 13 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a012ac40 osd=6 3 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 6 13 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012ac40 osd=6 4 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a696e700 10 client.38231.objecter _send_command 13 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30a012bcf0 con 0x7f30a012b350 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.596+0000 7f30a4f0b700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6808/1506640341 conn(0x7f30a012b350 0x7f30a012b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.597+0000 7f30a4f0b700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6808/1506640341 conn(0x7f30a012b350 0x7f30a012b790 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.597+0000 7f309d7fa700 10 client.38231.objecter ms_handle_connect 0x7f30a012b350 2024-02-13T00:53:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.598+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30a012bcf0 con 0x7f30a012b350 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.598+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.598+0000 7f309d7fa700 10 client.38231.objecter _finish_command 13 = system:0 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.598+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a012ac40 osd=6 4 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.598+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 6 13 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.618+0000 7f30a696e700 10 client.38231.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.618+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 14 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.618+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a012ac40 osd=6 3 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.618+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012ac40 osd=6 3 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.618+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a012ac40 osd=6 3 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.618+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 14 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.618+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a012ac40 osd=6 3 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.618+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 6 14 2024-02-13T00:53:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.618+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012ac40 osd=6 4 2024-02-13T00:53:50.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.618+0000 7f30a696e700 10 client.38231.objecter _send_command 14 2024-02-13T00:53:50.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.618+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f30a012bcf0 con 0x7f30a012b350 2024-02-13T00:53:50.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.619+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f30a012bcf0 con 0x7f30a012b350 2024-02-13T00:53:50.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.619+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:50.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.619+0000 7f309d7fa700 10 client.38231.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.619+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a012ac40 osd=6 4 2024-02-13T00:53:50.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.619+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 6 14 2024-02-13T00:53:50.643 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:53:50.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f30a696e700 10 client.38231.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:53:50.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 15 2024-02-13T00:53:50.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6812/1964192772 conn(0x7f30a0130c30 0x7f30a0133020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a012bcf0 osd=7 3 2024-02-13T00:53:50.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012bcf0 osd=7 3 2024-02-13T00:53:50.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a012bcf0 osd=7 3 2024-02-13T00:53:50.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 15 2024-02-13T00:53:50.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a012bcf0 osd=7 3 2024-02-13T00:53:50.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 7 15 2024-02-13T00:53:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012bcf0 osd=7 4 2024-02-13T00:53:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f30a696e700 10 client.38231.objecter _send_command 15 2024-02-13T00:53:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30a0135c70 con 0x7f30a0130c30 2024-02-13T00:53:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f309ffff700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6812/1964192772 conn(0x7f30a0130c30 0x7f30a0133020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f309d7fa700 10 client.38231.objecter ms_handle_connect 0x7f30a0130c30 2024-02-13T00:53:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.620+0000 7f309ffff700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6812/1964192772 conn(0x7f30a0130c30 0x7f30a0133020 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.621+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30a0135c70 con 0x7f30a0130c30 2024-02-13T00:53:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.621+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:53:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.621+0000 7f309d7fa700 10 client.38231.objecter _finish_command 15 = system:0 2024-02-13T00:53:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.621+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a012bcf0 osd=7 4 2024-02-13T00:53:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.621+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 7 15 2024-02-13T00:53:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.642+0000 7f30a696e700 10 client.38231.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:53:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.642+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign -1 16 2024-02-13T00:53:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.642+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a012bcf0 osd=7 3 2024-02-13T00:53:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.642+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012bcf0 osd=7 3 2024-02-13T00:53:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.642+0000 7f30a696e700 20 client.38231.objecter _get_session s=0x7f30a012bcf0 osd=7 3 2024-02-13T00:53:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.642+0000 7f30a696e700 15 client.38231.objecter _session_command_op_remove -1 16 2024-02-13T00:53:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.642+0000 7f30a696e700 20 client.38231.objecter get_session s=0x7f30a012bcf0 osd=7 3 2024-02-13T00:53:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.642+0000 7f30a696e700 15 client.38231.objecter _session_command_op_assign 7 16 2024-02-13T00:53:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.642+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012bcf0 osd=7 4 2024-02-13T00:53:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.642+0000 7f30a696e700 10 client.38231.objecter _send_command 16 2024-02-13T00:53:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.642+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f30a0133510 con 0x7f30a0130c30 2024-02-13T00:53:50.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.643+0000 7f309d7fa700 1 -- 172.21.15.162:0/1596972258 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f30a0133510 con 0x7f30a0130c30 2024-02-13T00:53:50.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.643+0000 7f309d7fa700 10 client.38231.objecter ms_dispatch 0x7f30a0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:53:50.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.643+0000 7f309d7fa700 10 client.38231.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.643+0000 7f309d7fa700 20 client.38231.objecter put_session s=0x7f30a012bcf0 osd=7 4 2024-02-13T00:53:50.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.643+0000 7f309d7fa700 15 client.38231.objecter _session_command_op_remove 7 16 2024-02-13T00:53:50.647 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:53:50.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 10 librados: watch_flush enter 2024-02-13T00:53:50.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 10 librados: watch_flush exit 2024-02-13T00:53:50.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 10 client.38231.objecter close_session for osd.0 2024-02-13T00:53:50.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6801/2103743338 conn(0x7f30a0059990 msgr2=0x7f30a0059d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6801/2103743338 conn(0x7f30a0059990 0x7f30a0059d70 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a010ffc0 osd=0 1 2024-02-13T00:53:50.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 10 client.38231.objecter close_session for osd.1 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6805/150599780 conn(0x7f30a0059570 msgr2=0x7f30a011f7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6805/150599780 conn(0x7f30a0059570 0x7f30a011f7c0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0059400 osd=1 1 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 10 client.38231.objecter close_session for osd.2 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6809/3689719376 conn(0x7f30a00f9e50 msgr2=0x7f30a0123010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6809/3689719376 conn(0x7f30a00f9e50 0x7f30a0123010 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a00f9ce0 osd=2 1 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 10 client.38231.objecter close_session for osd.3 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6813/604971011 conn(0x7f30a00277c0 msgr2=0x7f30a0126800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6813/604971011 conn(0x7f30a00277c0 0x7f30a0126800 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0027650 osd=3 1 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a4f0b700 1 -- 172.21.15.162:0/1596972258 reap_dead start 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 10 client.38231.objecter close_session for osd.4 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6800/3344263388 conn(0x7f30a0127400 msgr2=0x7f30a01297f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6800/3344263388 conn(0x7f30a0127400 0x7f30a01297f0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a4f0b700 1 -- 172.21.15.162:0/1596972258 reap_dead start 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a0126cf0 osd=4 1 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 10 client.38231.objecter close_session for osd.5 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6804/2277917372 conn(0x7f30a012a230 msgr2=0x7f30a012a690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6804/2277917372 conn(0x7f30a012a230 0x7f30a012a690 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012c4b0 osd=5 1 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 10 client.38231.objecter close_session for osd.6 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6808/1506640341 conn(0x7f30a012b350 msgr2=0x7f30a012b790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6808/1506640341 conn(0x7f30a012b350 0x7f30a012b790 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012ac40 osd=6 1 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 10 client.38231.objecter close_session for osd.7 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6812/1964192772 conn(0x7f30a0130c30 msgr2=0x7f30a0133020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.181:6812/1964192772 conn(0x7f30a0130c30 0x7f30a0133020 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 20 client.38231.objecter put_session s=0x7f30a012bcf0 osd=7 1 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 20 client.38231.objecter shutdown clearing up homeless session... 2024-02-13T00:53:50.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 10 client.38231.objecter successfully canceled tick 2024-02-13T00:53:50.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3088062c90 msgr2=0x7f3088065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3088062c90 0x7f3088065150 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 rx=0x7f3090004870 tx=0x7f309000b040).stop 2024-02-13T00:53:50.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a006bae0 msgr2=0x7f30a0114e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a696e700 1 --2- 172.21.15.162:0/1596972258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a006bae0 0x7f30a0114e30 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 rx=0x7f3094004920 tx=0x7f3094009cb0).stop 2024-02-13T00:53:50.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.644+0000 7f30a4f0b700 1 -- 172.21.15.162:0/1596972258 reap_dead start 2024-02-13T00:53:50.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.645+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 shutdown_connections 2024-02-13T00:53:50.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.645+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 >> 172.21.15.162:0/1596972258 conn(0x7f30a00f38c0 msgr2=0x7f30a00f44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:50.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.645+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 shutdown_connections 2024-02-13T00:53:50.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.645+0000 7f30a696e700 1 -- 172.21.15.162:0/1596972258 wait complete. 2024-02-13T00:53:50.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.646+0000 7f30a696e700 1 librados: shutdown 2024-02-13T00:53:50.652 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.652 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.652 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.652 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.652 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.652 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.652 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.652 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:53:50.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.601+0000 7f17b5d87700 1 -- 172.21.15.162:0/713269474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17b00f73a0 msgr2=0x7f17b01039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.601+0000 7f17b5d87700 1 --2- 172.21.15.162:0/713269474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17b00f73a0 0x7f17b01039a0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 rx=0x7f17a40047b0 tx=0x7f17a4009cb0).stop 2024-02-13T00:53:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.601+0000 7f17b5d87700 1 -- 172.21.15.162:0/713269474 shutdown_connections 2024-02-13T00:53:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.601+0000 7f17b5d87700 1 --2- 172.21.15.162:0/713269474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17b00f73a0 0x7f17b01039a0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.601+0000 7f17b5d87700 1 --2- 172.21.15.162:0/713269474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b00fcb30 0x7f17b00fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:53:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.601+0000 7f17b5d87700 1 --2- 172.21.15.162:0/713269474 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17b00fd490 0x7f17b00f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.601+0000 7f17b5d87700 1 -- 172.21.15.162:0/713269474 >> 172.21.15.162:0/713269474 conn(0x7f17b006eb90 msgr2=0x7f17b00f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.601+0000 7f17b5d87700 1 -- 172.21.15.162:0/713269474 shutdown_connections 2024-02-13T00:53:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.601+0000 7f17b5d87700 1 -- 172.21.15.162:0/713269474 wait complete. 2024-02-13T00:53:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.602+0000 7f17b5d87700 1 librados: starting msgr at 2024-02-13T00:53:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.602+0000 7f17b5d87700 1 librados: starting objecter 2024-02-13T00:53:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.602+0000 7f17b5d87700 1 Processor -- start 2024-02-13T00:53:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.602+0000 7f17b5d87700 1 -- start start 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.602+0000 7f17b5d87700 1 librados: setting wanted keys 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.602+0000 7f17b5d87700 1 librados: calling monclient init 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17b5d87700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b00f73a0 0x7f17b0113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17b5d87700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17b00fcb30 0x7f17b0113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17b5d87700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17b00fd490 0x7f17b0102600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17b5d87700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b0102af0 con 0x7f17b00fd490 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17b5d87700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b0102c60 con 0x7f17b00f73a0 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17b5d87700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b0102e20 con 0x7f17b00fcb30 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17aeffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17b00fcb30 0x7f17b0113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17aeffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17b00fcb30 0x7f17b0113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:51538/0 (socket says 172.21.15.162:51538) 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17aeffd700 1 -- 172.21.15.162:0/2962608416 learned_addr learned my addr 172.21.15.162:0/2962608416 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17af7fe700 1 --2- 172.21.15.162:0/2962608416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b00f73a0 0x7f17b0113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17affff700 1 --2- 172.21.15.162:0/2962608416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17b00fd490 0x7f17b0102600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17aeffd700 1 -- 172.21.15.162:0/2962608416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b00f73a0 msgr2=0x7f17b0113570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17aeffd700 1 --2- 172.21.15.162:0/2962608416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b00f73a0 0x7f17b0113570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17aeffd700 1 -- 172.21.15.162:0/2962608416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17b00fd490 msgr2=0x7f17b0102600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17aeffd700 1 --2- 172.21.15.162:0/2962608416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17b00fd490 0x7f17b0102600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17aeffd700 1 -- 172.21.15.162:0/2962608416 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a4005040 con 0x7f17b00fcb30 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.603+0000 7f17af7fe700 1 --2- 172.21.15.162:0/2962608416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b00f73a0 0x7f17b0113570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.604+0000 7f17aeffd700 1 --2- 172.21.15.162:0/2962608416 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17b00fcb30 0x7f17b0113a80 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7f17a000e970 tx=0x7f17a000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.604+0000 7f17acff9700 10 client.?.objecter ms_handle_connect 0x7f17b00fcb30 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.604+0000 7f17acff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.604+0000 7f17acff9700 1 -- 172.21.15.162:0/2962608416 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a00046e0 con 0x7f17b00fcb30 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.604+0000 7f17acff9700 1 -- 172.21.15.162:0/2962608416 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17a000f040 con 0x7f17b00fcb30 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.604+0000 7f17acff9700 1 -- 172.21.15.162:0/2962608416 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a000c820 con 0x7f17b00fcb30 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.604+0000 7f17b5d87700 1 -- 172.21.15.162:0/2962608416 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b0103100 con 0x7f17b00fcb30 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.604+0000 7f17b5d87700 10 client.38237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:51.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.604+0000 7f17b5d87700 1 -- 172.21.15.162:0/2962608416 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b00627a0 con 0x7f17b00fcb30 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.604+0000 7f17b5d87700 1 librados: init done 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.605+0000 7f17b5d87700 1 -- 172.21.15.162:0/2962608416 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b01071d0 con 0x7f17b00fcb30 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.606+0000 7f17acff9700 1 -- 172.21.15.162:0/2962608416 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f17a000cc50 con 0x7f17b00fcb30 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.607+0000 7f17acff9700 1 --2- 172.21.15.162:0/2962608416 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1798062c90 0x7f1798065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.607+0000 7f17acff9700 1 -- 172.21.15.162:0/2962608416 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(779..779 src has 449..779) v4 ==== 8615+0+0 (secure 0 0 0) 0x7f17a007cf80 con 0x7f17b00fcb30 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.607+0000 7f17acff9700 10 client.38237.objecter ms_dispatch 0x7f17b0001550 osd_map(779..779 src has 449..779) v4 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.607+0000 7f17acff9700 3 client.38237.objecter handle_osd_map got epochs [779,779] > 0 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.607+0000 7f17acff9700 3 client.38237.objecter handle_osd_map decoding full epoch 779 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.607+0000 7f17acff9700 20 client.38237.objecter dump_active .. 0 homeless 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.607+0000 7f17af7fe700 1 --2- 172.21.15.162:0/2962608416 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1798062c90 0x7f1798065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.608+0000 7f17af7fe700 1 --2- 172.21.15.162:0/2962608416 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1798062c90 0x7f1798065150 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 rx=0x7f17a8006fd0 tx=0x7f17a8008040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.608+0000 7f17acff9700 10 client.38237.objecter ms_handle_connect 0x7f1798062c90 2024-02-13T00:53:51.001 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.612+0000 7f17acff9700 1 -- 172.21.15.162:0/2962608416 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f17a00abf60 con 0x7f17b00fcb30 2024-02-13T00:53:51.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:50.811+0000 7f17b5d87700 1 -- 172.21.15.162:0/2962608416 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f17b005fc80 con 0x7f17b00fcb30 2024-02-13T00:53:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:49.465043+0000 osd.1 (osd.1) 325 : cluster [DBG] 2.d scrub starts 2024-02-13T00:53:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:49.468025+0000 osd.1 (osd.1) 326 : cluster [DBG] 2.d scrub ok 2024-02-13T00:53:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:49.618533+0000 mon.a (mon.0) 3681 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136"}]': finished 2024-02-13T00:53:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:49.618573+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e779: 8 total, 8 up, 4 in 2024-02-13T00:53:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:49.636732+0000 mon.a (mon.0) 3683 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:49.637046+0000 mon.a (mon.0) 3684 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136"}]: dispatch 2024-02-13T00:53:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:50.042913+0000 osd.0 (osd.0) 311 : cluster [DBG] 5.3 scrub starts 2024-02-13T00:53:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:50.046451+0000 osd.0 (osd.0) 312 : cluster [DBG] 5.3 scrub ok 2024-02-13T00:53:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:50.480926+0000 mon.c (mon.2) 512 : audit [DBG] from='client.? 172.21.15.162:0/1596972258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:49.465043+0000 osd.1 (osd.1) 325 : cluster [DBG] 2.d scrub starts 2024-02-13T00:53:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:49.468025+0000 osd.1 (osd.1) 326 : cluster [DBG] 2.d scrub ok 2024-02-13T00:53:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:49.618533+0000 mon.a (mon.0) 3681 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136"}]': finished 2024-02-13T00:53:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:49.618573+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e779: 8 total, 8 up, 4 in 2024-02-13T00:53:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:49.636732+0000 mon.a (mon.0) 3683 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:49.637046+0000 mon.a (mon.0) 3684 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136"}]: dispatch 2024-02-13T00:53:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:50.042913+0000 osd.0 (osd.0) 311 : cluster [DBG] 5.3 scrub starts 2024-02-13T00:53:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:50.046451+0000 osd.0 (osd.0) 312 : cluster [DBG] 5.3 scrub ok 2024-02-13T00:53:51.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:50.480926+0000 mon.c (mon.2) 512 : audit [DBG] from='client.? 172.21.15.162:0/1596972258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:49.465043+0000 osd.1 (osd.1) 325 : cluster [DBG] 2.d scrub starts 2024-02-13T00:53:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:49.468025+0000 osd.1 (osd.1) 326 : cluster [DBG] 2.d scrub ok 2024-02-13T00:53:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:49.618533+0000 mon.a (mon.0) 3681 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136"}]': finished 2024-02-13T00:53:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:49.618573+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e779: 8 total, 8 up, 4 in 2024-02-13T00:53:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:49.636732+0000 mon.a (mon.0) 3683 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:49.637046+0000 mon.a (mon.0) 3684 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-136"}]: dispatch 2024-02-13T00:53:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:50.042913+0000 osd.0 (osd.0) 311 : cluster [DBG] 5.3 scrub starts 2024-02-13T00:53:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:50.046451+0000 osd.0 (osd.0) 312 : cluster [DBG] 5.3 scrub ok 2024-02-13T00:53:51.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:50.480926+0000 mon.c (mon.2) 512 : audit [DBG] from='client.? 172.21.15.162:0/1596972258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:53:51.082 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:51.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:51.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:51.215 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:51.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.680+0000 7f17acff9700 1 -- 172.21.15.162:0/2962608416 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v781) v1 ==== 101+0+0 (secure 0 0 0) 0x7f17a0019070 con 0x7f17b00fcb30 2024-02-13T00:53:51.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.684+0000 7f17b5d87700 10 librados: watch_flush enter 2024-02-13T00:53:51.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.684+0000 7f17b5d87700 10 librados: watch_flush exit 2024-02-13T00:53:51.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.684+0000 7f17b5d87700 20 client.38237.objecter shutdown clearing up homeless session... 2024-02-13T00:53:51.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.684+0000 7f17b5d87700 10 client.38237.objecter successfully canceled tick 2024-02-13T00:53:51.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.685+0000 7f17b5d87700 1 -- 172.21.15.162:0/2962608416 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1798062c90 msgr2=0x7f1798065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:51.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.685+0000 7f17b5d87700 1 --2- 172.21.15.162:0/2962608416 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1798062c90 0x7f1798065150 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 rx=0x7f17a8006fd0 tx=0x7f17a8008040).stop 2024-02-13T00:53:51.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.685+0000 7f17b5d87700 1 -- 172.21.15.162:0/2962608416 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17b00fcb30 msgr2=0x7f17b0113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:51.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.685+0000 7f17b5d87700 1 --2- 172.21.15.162:0/2962608416 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17b00fcb30 0x7f17b0113a80 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7f17a000e970 tx=0x7f17a000b040).stop 2024-02-13T00:53:51.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.686+0000 7f17b5d87700 1 -- 172.21.15.162:0/2962608416 shutdown_connections 2024-02-13T00:53:51.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.686+0000 7f17b5d87700 1 --2- 172.21.15.162:0/2962608416 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1798062c90 0x7f1798065150 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:51.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.686+0000 7f17b5d87700 1 --2- 172.21.15.162:0/2962608416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17b00fd490 0x7f17b0102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:51.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.686+0000 7f17b5d87700 1 --2- 172.21.15.162:0/2962608416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17b00f73a0 0x7f17b0113570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:51.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.686+0000 7f17b5d87700 1 --2- 172.21.15.162:0/2962608416 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17b00fcb30 0x7f17b0113a80 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:51.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.686+0000 7f17b5d87700 1 -- 172.21.15.162:0/2962608416 >> 172.21.15.162:0/2962608416 conn(0x7f17b006eb90 msgr2=0x7f17b006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:51.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.686+0000 7f17b5d87700 1 -- 172.21.15.162:0/2962608416 shutdown_connections 2024-02-13T00:53:51.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.686+0000 7f17b5d87700 1 -- 172.21.15.162:0/2962608416 wait complete. 2024-02-13T00:53:51.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:51.686+0000 7f17b5d87700 1 librados: shutdown 2024-02-13T00:53:51.694 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:53:51.860 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 6, 7] out_osds: [3, 2, 5, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-13T00:53:51.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-13T00:53:52.035 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:50.446153+0000 osd.1 (osd.1) 327 : cluster [DBG] 4.d scrub starts 2024-02-13T00:53:52.035 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:50.449718+0000 osd.1 (osd.1) 328 : cluster [DBG] 4.d scrub ok 2024-02-13T00:53:52.035 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:50.545354+0000 mgr.y (mgr.14544) 648 : cluster [DBG] pgmap v1187: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:50.620610+0000 mon.a (mon.0) 3685 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-13T00:53:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:50.672939+0000 mon.a (mon.0) 3686 : cluster [DBG] osdmap e780: 8 total, 8 up, 4 in 2024-02-13T00:53:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:50.812071+0000 mon.c (mon.2) 513 : audit [INF] from='client.? 172.21.15.162:0/2962608416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:52.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:50.931411+0000 mon.a (mon.0) 3687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:52.036 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:50.446153+0000 osd.1 (osd.1) 327 : cluster [DBG] 4.d scrub starts 2024-02-13T00:53:52.036 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:50.449718+0000 osd.1 (osd.1) 328 : cluster [DBG] 4.d scrub ok 2024-02-13T00:53:52.036 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:50.545354+0000 mgr.y (mgr.14544) 648 : cluster [DBG] pgmap v1187: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:52.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:50.620610+0000 mon.a (mon.0) 3685 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-13T00:53:52.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:50.672939+0000 mon.a (mon.0) 3686 : cluster [DBG] osdmap e780: 8 total, 8 up, 4 in 2024-02-13T00:53:52.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:50.812071+0000 mon.c (mon.2) 513 : audit [INF] from='client.? 172.21.15.162:0/2962608416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:52.040 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:50.931411+0000 mon.a (mon.0) 3687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:50.446153+0000 osd.1 (osd.1) 327 : cluster [DBG] 4.d scrub starts 2024-02-13T00:53:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:50.449718+0000 osd.1 (osd.1) 328 : cluster [DBG] 4.d scrub ok 2024-02-13T00:53:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:50.545354+0000 mgr.y (mgr.14544) 648 : cluster [DBG] pgmap v1187: 204 pgs: 204 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-13T00:53:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:50.620610+0000 mon.a (mon.0) 3685 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-13T00:53:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:50.672939+0000 mon.a (mon.0) 3686 : cluster [DBG] osdmap e780: 8 total, 8 up, 4 in 2024-02-13T00:53:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:50.812071+0000 mon.c (mon.2) 513 : audit [INF] from='client.? 172.21.15.162:0/2962608416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:52.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:50.931411+0000 mon.a (mon.0) 3687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:53:52.807 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:51.670160+0000 mon.a (mon.0) 3688 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:53:52.807 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:51.680186+0000 mon.a (mon.0) 3689 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:53:52.807 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:51.680270+0000 mon.a (mon.0) 3690 : cluster [DBG] osdmap e781: 8 total, 8 up, 4 in 2024-02-13T00:53:52.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:51.681492+0000 mon.a (mon.0) 3691 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:53:52.808 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:51.670160+0000 mon.a (mon.0) 3688 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:53:52.808 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:51.680186+0000 mon.a (mon.0) 3689 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:53:52.808 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:51.680270+0000 mon.a (mon.0) 3690 : cluster [DBG] osdmap e781: 8 total, 8 up, 4 in 2024-02-13T00:53:52.808 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:51.681492+0000 mon.a (mon.0) 3691 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:53:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:51.670160+0000 mon.a (mon.0) 3688 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:53:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:51.680186+0000 mon.a (mon.0) 3689 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:53:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:51.680270+0000 mon.a (mon.0) 3690 : cluster [DBG] osdmap e781: 8 total, 8 up, 4 in 2024-02-13T00:53:53.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:51.681492+0000 mon.a (mon.0) 3691 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:53:53.966 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:53.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:52.545995+0000 mgr.y (mgr.14544) 649 : cluster [DBG] pgmap v1190: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 255 B/s rd, 0 op/s 2024-02-13T00:53:53.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:52.685703+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:53:53.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:52.685824+0000 mon.a (mon.0) 3693 : cluster [DBG] osdmap e782: 8 total, 8 up, 4 in 2024-02-13T00:53:53.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:52.720281+0000 mon.a (mon.0) 3694 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:53:53.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:53.020309+0000 osd.0 (osd.0) 313 : cluster [DBG] 2.1f scrub starts 2024-02-13T00:53:53.968 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:53.023270+0000 osd.0 (osd.0) 314 : cluster [DBG] 2.1f scrub ok 2024-02-13T00:53:53.969 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:52.545995+0000 mgr.y (mgr.14544) 649 : cluster [DBG] pgmap v1190: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 255 B/s rd, 0 op/s 2024-02-13T00:53:53.969 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:52.685703+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:53:53.969 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:52.685824+0000 mon.a (mon.0) 3693 : cluster [DBG] osdmap e782: 8 total, 8 up, 4 in 2024-02-13T00:53:53.969 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:52.720281+0000 mon.a (mon.0) 3694 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:53:53.969 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:53.020309+0000 osd.0 (osd.0) 313 : cluster [DBG] 2.1f scrub starts 2024-02-13T00:53:53.969 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:53.023270+0000 osd.0 (osd.0) 314 : cluster [DBG] 2.1f scrub ok 2024-02-13T00:53:53.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:53.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:52.545995+0000 mgr.y (mgr.14544) 649 : cluster [DBG] pgmap v1190: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 255 B/s rd, 0 op/s 2024-02-13T00:53:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:52.685703+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:53:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:52.685824+0000 mon.a (mon.0) 3693 : cluster [DBG] osdmap e782: 8 total, 8 up, 4 in 2024-02-13T00:53:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:52.720281+0000 mon.a (mon.0) 3694 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-13T00:53:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:53.020309+0000 osd.0 (osd.0) 313 : cluster [DBG] 2.1f scrub starts 2024-02-13T00:53:54.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:53.023270+0000 osd.0 (osd.0) 314 : cluster [DBG] 2.1f scrub ok 2024-02-13T00:53:54.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:54.099 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:54.127 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:54.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:54.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:54.231 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27057.0:1 190.5s2 190:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e319)", 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:34.404917+0000", 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 559.84564205599997, 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.331389991, 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27057", 2024-02-13T00:53:54.257 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/4276929747", 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404917+0000", 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404917+0000", 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404915+0000", 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7039999999999997e-06 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404922+0000", 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0203999999999999e-05 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404932+0000", 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9459999999999998e-06 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.404940+0000", 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022950620000000001 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.407235+0000", 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5379999999999998e-06 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:34.407238+0000", 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3289484380000001 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:35.736187+0000", 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011990899999999999 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:35.736307+0000", 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.331389991 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24329.0:1153 5.3 5:c609908c:::notify.5:head [watch ping cookie 94514175194752 gen 1] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:39.296113+0000", 2024-02-13T00:53:54.261 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 554.95444572600002, 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223829202, 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24329", 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1153 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296113+0000", 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296113+0000", 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296110+0000", 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3057000000000001e-05 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296123+0000", 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0857999999999999e-05 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296144+0000", 2024-02-13T00:53:54.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1107999999999999e-05 2024-02-13T00:53:54.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296155+0000", 2024-02-13T00:53:54.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112284 2024-02-13T00:53:54.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296268+0000", 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3674e-05 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296281+0000", 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.197784131 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.494065+0000", 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034288400000000001 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-02-13T00:53:54.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.494408+0000", 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025363974000000001 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.519772+0000", 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5661999999999994e-05 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.519858+0000", 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4176000000000002e-05 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.519942+0000", 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223829202 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24329.0:1162 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94514175194752 gen 2] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:39.297175+0000", 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 554.95338378400004, 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22574032099999999, 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24329", 2024-02-13T00:53:54.267 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:53:54.268 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1162 2024-02-13T00:53:54.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.268 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297175+0000", 2024-02-13T00:53:54.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297175+0000", 2024-02-13T00:53:54.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297174+0000", 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4430000000000002e-06 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297179+0000", 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0529e-05 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297190+0000", 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2519999999999999e-06 2024-02-13T00:53:54.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297195+0000", 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2176999999999999e-05 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297247+0000", 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0348e-05 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297258+0000", 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19719434299999999 2024-02-13T00:53:54.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.494452+0000", 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1651999999999999e-05 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.494514+0000", 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025459631 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.519973+0000", 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0639000000000002e-05 2024-02-13T00:53:54.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.520014+0000", 2024-02-13T00:53:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002901299 2024-02-13T00:53:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.522915+0000", 2024-02-13T00:53:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22574032099999999 2024-02-13T00:53:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.272 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25162.0:4733 191.6 e335/323 191:602f83fe:::foo:head v 335'11, mlcod=333'9)", 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:50.443597+0000", 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 543.80696211700001, 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051008379, 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443597+0000", 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443597+0000", 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443596+0000", 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2199999999999992e-06 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443604+0000", 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3690000000000003e-06 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443611+0000", 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4969999999999998e-06 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443617+0000", 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7141999999999999e-05 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443714+0000", 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9049999999999999e-05 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.443733+0000", 2024-02-13T00:53:54.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050843781999999997 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.494577+0000", 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7654e-05 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.494605+0000", 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051008379 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24748.0:1139 18.4 18:20000000::::head [pgnls start_epoch 348] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e348)", 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:45:03.458723+0000", 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 530.79183520799995, 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94893485399999999, 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24748", 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2831164779", 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1139 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458723+0000", 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458723+0000", 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458723+0000", 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9849999999999997e-06 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458728+0000", 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1950000000000003e-06 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458736+0000", 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4649999999999996e-06 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458740+0000", 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9974999999999993e-05 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458810+0000", 2024-02-13T00:53:54.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0842999999999999e-05 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:03.458821+0000", 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94869083399999998 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:04.407512+0000", 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0757999999999998e-05 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:04.407553+0000", 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105543 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:04.407658+0000", 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94893485399999999 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.37256.0:1 197.2s1 197:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-02-13T00:53:54.281 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:45:05.418265+0000", 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 528.83229411900004, 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.004095116, 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.37256", 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/1003782898", 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418265+0000", 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418265+0000", 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418262+0000", 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0108999999999999e-05 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418272+0000", 2024-02-13T00:53:54.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1304000000000001e-05 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418283+0000", 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7669999999999992e-06 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.418292+0000", 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00073741699999999998 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.419030+0000", 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9870000000000001e-06 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:05.419034+0000", 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99819390399999997 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.417227+0000", 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013077100000000001 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.417358+0000", 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00046316100000000002 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.417821+0000", 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041701299999999999 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.418238+0000", 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040563220000000002 2024-02-13T00:53:54.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.422295+0000", 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4969999999999996e-05 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:06.422360+0000", 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.004095116 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.287 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:246 234.4s0 234:2d3d2007:test-rados-api-smithi162-84057-107::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e616)", 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:50:30.519484+0000", 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 203.73107498900001, 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065207086999999997, 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 246 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.519484+0000", 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.519484+0000", 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.519481+0000", 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3487e-05 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.519495+0000", 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9641e-05 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.519515+0000", 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0331e-05 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.519525+0000", 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001370551 2024-02-13T00:53:54.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.520895+0000", 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1079999999999999e-06 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.520905+0000", 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0087229360000000006 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.529627+0000", 2024-02-13T00:53:54.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013905800000000001 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.529767+0000", 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041805599999999998 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.530185+0000", 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000699291 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.530884+0000", 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053739826999999997 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.584624+0000", 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7084999999999999e-05 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:50:30.584691+0000", 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065207086999999997 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.293 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDECSubOpWrite(234.5s2 639/599 ECSubWrite(tid=706, reqid=client.27753.0:474, at_version=0'0, trim_to=0'0, roll_forward_to=639'6))", 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:51:04.536649+0000", 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 169.713909441, 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041735039999999998, 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:04.536649+0000", 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:04.536649+0000", 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:04.536649+0000", 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1159999999999999e-06 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:04.536657+0000", 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.438e-06 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:04.536663+0000", 2024-02-13T00:53:54.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5810000000000002e-06 2024-02-13T00:53:54.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:04.536670+0000", 2024-02-13T00:53:54.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7972999999999997e-05 2024-02-13T00:53:54.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:04.536708+0000", 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0285000000000001e-05 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:04.536718+0000", 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00097805200000000009 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:04.537696+0000", 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031264140000000001 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:04.540823+0000", 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041735039999999998 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:581 239.10 239:08000000::::head [pgnls start_epoch 644] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e644)", 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:51:09.544434+0000", 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 164.70612516, 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029813349999999999, 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.298 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 581 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:09.544434+0000", 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:09.544434+0000", 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:53:54.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:09.544432+0000", 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0463e-05 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:09.544442+0000", 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6444999999999999e-05 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:09.544459+0000", 2024-02-13T00:53:54.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7789999999999998e-06 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:09.544465+0000", 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4283000000000003e-05 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:09.544510+0000", 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6090000000000003e-06 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:09.544514+0000", 2024-02-13T00:53:54.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002767795 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:09.547282+0000", 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2833999999999998e-05 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:09.547315+0000", 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9889000000000005e-05 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:09.547415+0000", 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029813349999999999 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:819 241.4 241:20000000:%01:::head [pgnls start_epoch 660] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e660)", 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:51:25.628444+0000", 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 148.62211476799999, 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036944809999999999, 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 819 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:25.628444+0000", 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:25.628444+0000", 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:53:54.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:25.628442+0000", 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7430000000000008e-06 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:25.628451+0000", 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2741999999999999e-05 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:25.628464+0000", 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4559999999999997e-06 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:25.628470+0000", 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4298999999999997e-05 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:25.628514+0000", 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0329999999999997e-06 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:25.628520+0000", 2024-02-13T00:53:54.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035306130000000002 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:25.632051+0000", 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3402999999999999e-05 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:25.632074+0000", 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3934000000000001e-05 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:25.632138+0000", 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036944809999999999 2024-02-13T00:53:54.307 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:853 241.4 241:20000000::::head [pgnls start_epoch 662] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e662)", 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:51:27.627856+0000", 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 146.62270231100001, 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035213340000000001, 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 853 2024-02-13T00:53:54.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:27.627856+0000", 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:27.627856+0000", 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:27.627855+0000", 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7490000000000004e-06 2024-02-13T00:53:54.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:27.627863+0000", 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5119999999999999e-05 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:27.627878+0000", 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9229999999999999e-06 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:27.627884+0000", 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5741999999999997e-05 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:27.627930+0000", 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2860000000000001e-06 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:27.627934+0000", 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033310940000000002 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:27.631265+0000", 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0667000000000003e-05 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:27.631296+0000", 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1562000000000003e-05 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:27.631378+0000", 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035213340000000001 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.312 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDScrubReserve(28.7 REQUEST e674)", 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:51:39.563100+0000", 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 134.687458354, 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042823280000000002, 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:39.563100+0000", 2024-02-13T00:53:54.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:39.563100+0000", 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967296 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:39.563100+0000", 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.875e-06 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:39.563103+0000", 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2644e-05 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:39.563116+0000", 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6379999999999998e-06 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:39.563122+0000", 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8575000000000002e-05 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:39.563171+0000", 2024-02-13T00:53:54.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7619999999999995e-06 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:39.563179+0000", 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041646529999999999 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:39.567343+0000", 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.393e-05 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:39.567357+0000", 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5488999999999999e-05 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:39.567383+0000", 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042823280000000002 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27753.0:1794 247.b e716/712 247:d7155152:test-rados-api-smithi162-84057-127::foo:head v 716'2, mlcod=0'0)", 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:52:23.563854+0000", 2024-02-13T00:53:54.317 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 90.686704774999995, 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015499628, 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.563854+0000", 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.563854+0000", 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:53:54.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.563853+0000", 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010149708 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.574003+0000", 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4008999999999999e-05 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.574017+0000", 2024-02-13T00:53:54.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5529999999999998e-06 2024-02-13T00:53:54.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.574026+0000", 2024-02-13T00:53:54.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8074e-05 2024-02-13T00:53:54.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.574085+0000", 2024-02-13T00:53:54.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8695e-05 2024-02-13T00:53:54.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.574103+0000", 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052183780000000001 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.579322+0000", 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1928999999999999e-05 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.579354+0000", 2024-02-13T00:53:54.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015499628 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24329.0:2170 5.e 5:7759931f:::notify.3:head [watch ping cookie 94514175192448 gen 7] snapc 0=[] ondisk+write+known_if_redirected e716)", 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:52:29.381980+0000", 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 84.868578514000006, 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029792070000000002, 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.322 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24329", 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2170 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:29.381980+0000", 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:29.381980+0000", 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:53:54.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:29.381978+0000", 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1939e-05 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:29.381990+0000", 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0534e-05 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:29.382010+0000", 2024-02-13T00:53:54.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4939999999999992e-06 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:29.382020+0000", 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027620790000000002 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:29.384782+0000", 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7953000000000004e-05 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:29.384860+0000", 2024-02-13T00:53:54.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9624999999999996e-05 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:29.384959+0000", 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029792070000000002 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(234.7s1 723/722 [PushReplyOp(234:f8cedb5b:test-rados-api-smithi162-84057-107::bam:head)])", 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:52:30.825260+0000", 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.425298851999997, 2024-02-13T00:53:54.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00415191, 2024-02-13T00:53:54.327 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.327 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:53:54.327 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:30.825260+0000", 2024-02-13T00:53:54.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:30.825260+0000", 2024-02-13T00:53:54.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:30.825260+0000", 2024-02-13T00:53:54.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:30.825260+0000", 2024-02-13T00:53:54.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9830000000000008e-06 2024-02-13T00:53:54.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:30.825270+0000", 2024-02-13T00:53:54.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8829999999999999e-06 2024-02-13T00:53:54.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:30.825274+0000", 2024-02-13T00:53:54.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038597380000000001 2024-02-13T00:53:54.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:30.829133+0000", 2024-02-13T00:53:54.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027830599999999997 2024-02-13T00:53:54.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:30.829412+0000", 2024-02-13T00:53:54.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00415191 2024-02-13T00:53:54.332 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.333 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.333 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.333 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:2014 249.6 249:602f83fe:::foo:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected e735)", 2024-02-13T00:53:54.333 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:52:50.581705+0000", 2024-02-13T00:53:54.333 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 63.668854035000003, 2024-02-13T00:53:54.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053646311000000002, 2024-02-13T00:53:54.334 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.334 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:53:54.334 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.334 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:53:54.334 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:53:54.334 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2014 2024-02-13T00:53:54.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.335 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.581705+0000", 2024-02-13T00:53:54.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.581705+0000", 2024-02-13T00:53:54.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999962 2024-02-13T00:53:54.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.581701+0000", 2024-02-13T00:53:54.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0141000000000002e-05 2024-02-13T00:53:54.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.581731+0000", 2024-02-13T00:53:54.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3372000000000003e-05 2024-02-13T00:53:54.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.581784+0000", 2024-02-13T00:53:54.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2758000000000001e-05 2024-02-13T00:53:54.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.581797+0000", 2024-02-13T00:53:54.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0819000000000001e-05 2024-02-13T00:53:54.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.581868+0000", 2024-02-13T00:53:54.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9425000000000006e-05 2024-02-13T00:53:54.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.581947+0000", 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000170576 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.582118+0000", 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024336878999999999 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.606455+0000", 2024-02-13T00:53:54.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019728018999999999 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.626183+0000", 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0090905919999999998 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.635273+0000", 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8049000000000001e-05 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.635312+0000", 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9421000000000001e-05 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.635351+0000", 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053646311000000002 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:2145 250.12 250:4915ab7d:test-rados-api-smithi162-84057-133::foo0:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e749)", 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:53:07.567617+0000", 2024-02-13T00:53:54.342 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 46.682941423999999, 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014346619, 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2145 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:07.567617+0000", 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:07.567617+0000", 2024-02-13T00:53:54.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:53:54.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:07.567615+0000", 2024-02-13T00:53:54.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2268e-05 2024-02-13T00:53:54.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:07.567627+0000", 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2943999999999999e-05 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:07.567650+0000", 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9790000000000005e-06 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:07.567660+0000", 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5856999999999997e-05 2024-02-13T00:53:54.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:07.567716+0000", 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7479999999999996e-06 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:07.567723+0000", 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077299730000000002 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:07.575453+0000", 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0065111830000000002 2024-02-13T00:53:54.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:07.581964+0000", 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014346619 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:2737 252.4 252:26bf2024:test-rados-api-smithi162-84057-137::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e769)", 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:53:33.609523+0000", 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.641035754000001, 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042136220000000002, 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.347 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2737 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.609523+0000", 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.609523+0000", 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.609522+0000", 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.328e-06 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.609529+0000", 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0081999999999999e-05 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.609539+0000", 2024-02-13T00:53:54.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2089999999999996e-06 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.609544+0000", 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6257999999999997e-05 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.609580+0000", 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014212 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.609723+0000", 2024-02-13T00:53:54.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00133085 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.611053+0000", 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9320999999999998e-05 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.611103+0000", 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000367012 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.611470+0000", 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6579999999999998e-05 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.611516+0000", 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002220194 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.613737+0000", 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042136220000000002 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.352 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:2739 252.13 252:ce570790:test-rados-api-smithi162-84057-137::bam:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e769)", 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:53:33.616723+0000", 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.633835675, 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041631849999999998, 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2739 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.353 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.616723+0000", 2024-02-13T00:53:54.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.616723+0000", 2024-02-13T00:53:54.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:53:54.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.616722+0000", 2024-02-13T00:53:54.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7829999999999998e-06 2024-02-13T00:53:54.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.616727+0000", 2024-02-13T00:53:54.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0224000000000001e-05 2024-02-13T00:53:54.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.616737+0000", 2024-02-13T00:53:54.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4859999999999999e-06 2024-02-13T00:53:54.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.616742+0000", 2024-02-13T00:53:54.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7345999999999998e-05 2024-02-13T00:53:54.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.616779+0000", 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013996599999999999 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.616919+0000", 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001248288 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.618167+0000", 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0893000000000002e-05 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.618218+0000", 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040916599999999998 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.618627+0000", 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2068999999999999e-05 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.618679+0000", 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022068510000000001 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.620886+0000", 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041631849999999998 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(osd.0.0:1970 252.13 e773/765 ondisk, result = 0)", 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:53:38.577719+0000", 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.672839960999999, 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011851419999999999, 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:53:54.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:38.577719+0000", 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:38.577719+0000", 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:38.577717+0000", 2024-02-13T00:53:54.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.311e-05 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:38.577730+0000", 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.149e-06 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:38.577740+0000", 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8530000000000003e-06 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:53:54.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:38.577746+0000", 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001139139 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:38.578886+0000", 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8180000000000004e-06 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:38.578893+0000", 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0494e-05 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:53:54.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:53:54.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:38.578904+0000", 2024-02-13T00:53:54.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011851419999999999 2024-02-13T00:53:54.363 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.363 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.363 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.363 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:53:54.363 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:53:54.363 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:53:54.364 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:54.365 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:54.365 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:54.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:54.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:54.497 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:54.517 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:54.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:54.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:54.652 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:53.430557+0000 osd.1 (osd.1) 329 : cluster [DBG] 5.c scrub starts 2024-02-13T00:53:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:53.433512+0000 osd.1 (osd.1) 330 : cluster [DBG] 5.c scrub ok 2024-02-13T00:53:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:53.693500+0000 mon.a (mon.0) 3695 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:53:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:53.693542+0000 mon.a (mon.0) 3696 : cluster [DBG] osdmap e783: 8 total, 8 up, 4 in 2024-02-13T00:53:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:53.702939+0000 mon.a (mon.0) 3697 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138"}]: dispatch 2024-02-13T00:53:54.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:54.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:54.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:54.963 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:53.430557+0000 osd.1 (osd.1) 329 : cluster [DBG] 5.c scrub starts 2024-02-13T00:53:54.963 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:53.433512+0000 osd.1 (osd.1) 330 : cluster [DBG] 5.c scrub ok 2024-02-13T00:53:54.963 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:53.693500+0000 mon.a (mon.0) 3695 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:53:54.963 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:53.693542+0000 mon.a (mon.0) 3696 : cluster [DBG] osdmap e783: 8 total, 8 up, 4 in 2024-02-13T00:53:54.963 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:53.702939+0000 mon.a (mon.0) 3697 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138"}]: dispatch 2024-02-13T00:53:54.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:53.430557+0000 osd.1 (osd.1) 329 : cluster [DBG] 5.c scrub starts 2024-02-13T00:53:54.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:53.433512+0000 osd.1 (osd.1) 330 : cluster [DBG] 5.c scrub ok 2024-02-13T00:53:54.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:53.693500+0000 mon.a (mon.0) 3695 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-13T00:53:54.964 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:53.693542+0000 mon.a (mon.0) 3696 : cluster [DBG] osdmap e783: 8 total, 8 up, 4 in 2024-02-13T00:53:54.964 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:53.702939+0000 mon.a (mon.0) 3697 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138"}]: dispatch 2024-02-13T00:53:54.983 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:55.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-13T00:53:55.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-13T00:53:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:54.546327+0000 mgr.y (mgr.14544) 650 : cluster [DBG] pgmap v1193: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:53:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:54.699071+0000 mon.a (mon.0) 3698 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138"}]': finished 2024-02-13T00:53:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:54.699108+0000 mon.a (mon.0) 3699 : cluster [DBG] osdmap e784: 8 total, 8 up, 4 in 2024-02-13T00:53:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:54.717667+0000 mon.a (mon.0) 3700 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:54.718009+0000 mon.a (mon.0) 3701 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138"}]: dispatch 2024-02-13T00:53:56.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:55.014818+0000 osd.0 (osd.0) 315 : cluster [DBG] 28.19 scrub starts 2024-02-13T00:53:56.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:55.015815+0000 osd.0 (osd.0) 316 : cluster [DBG] 28.19 scrub ok 2024-02-13T00:53:56.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:55.582831+0000 mon.a (mon.0) 3702 : cluster [DBG] osdmap e785: 8 total, 8 up, 4 in 2024-02-13T00:53:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:54.546327+0000 mgr.y (mgr.14544) 650 : cluster [DBG] pgmap v1193: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:53:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:54.699071+0000 mon.a (mon.0) 3698 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138"}]': finished 2024-02-13T00:53:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:54.699108+0000 mon.a (mon.0) 3699 : cluster [DBG] osdmap e784: 8 total, 8 up, 4 in 2024-02-13T00:53:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:54.717667+0000 mon.a (mon.0) 3700 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:54.718009+0000 mon.a (mon.0) 3701 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138"}]: dispatch 2024-02-13T00:53:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:55.014818+0000 osd.0 (osd.0) 315 : cluster [DBG] 28.19 scrub starts 2024-02-13T00:53:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:55.015815+0000 osd.0 (osd.0) 316 : cluster [DBG] 28.19 scrub ok 2024-02-13T00:53:56.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:55.582831+0000 mon.a (mon.0) 3702 : cluster [DBG] osdmap e785: 8 total, 8 up, 4 in 2024-02-13T00:53:56.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:53:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:53:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:54.546327+0000 mgr.y (mgr.14544) 650 : cluster [DBG] pgmap v1193: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:53:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:54.699071+0000 mon.a (mon.0) 3698 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138"}]': finished 2024-02-13T00:53:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:54.699108+0000 mon.a (mon.0) 3699 : cluster [DBG] osdmap e784: 8 total, 8 up, 4 in 2024-02-13T00:53:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:54.717667+0000 mon.a (mon.0) 3700 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:56.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:54.718009+0000 mon.a (mon.0) 3701 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-138"}]: dispatch 2024-02-13T00:53:56.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:55.014818+0000 osd.0 (osd.0) 315 : cluster [DBG] 28.19 scrub starts 2024-02-13T00:53:56.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:55.015815+0000 osd.0 (osd.0) 316 : cluster [DBG] 28.19 scrub ok 2024-02-13T00:53:56.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:55.582831+0000 mon.a (mon.0) 3702 : cluster [DBG] osdmap e785: 8 total, 8 up, 4 in 2024-02-13T00:53:56.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-13T00:53:56.551 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:53:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:53:56] "GET /metrics HTTP/1.1" 200 190231 "" "Prometheus/2.33.4" 2024-02-13T00:53:56.814 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:56.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.826+0000 7fe53868a700 1 -- 172.21.15.162:0/862728027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5300fcc40 msgr2=0x7fe5300fd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:56.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.826+0000 7fe53868a700 1 --2- 172.21.15.162:0/862728027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5300fcc40 0x7fe5300fd020 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7fe52c004870 tx=0x7fe52c009cb0).stop 2024-02-13T00:53:56.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.826+0000 7fe53868a700 1 -- 172.21.15.162:0/862728027 shutdown_connections 2024-02-13T00:53:56.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.826+0000 7fe53868a700 1 --2- 172.21.15.162:0/862728027 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5300f7ca0 0x7fe530103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:53:56.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.826+0000 7fe53868a700 1 --2- 172.21.15.162:0/862728027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5300fcc40 0x7fe5300fd020 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:56.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.826+0000 7fe53868a700 1 --2- 172.21.15.162:0/862728027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5300fd5a0 0x7fe5300f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:56.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.826+0000 7fe53868a700 1 -- 172.21.15.162:0/862728027 >> 172.21.15.162:0/862728027 conn(0x7fe53006c410 msgr2=0x7fe53006c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:56.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.826+0000 7fe53868a700 1 -- 172.21.15.162:0/862728027 shutdown_connections 2024-02-13T00:53:56.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.826+0000 7fe53868a700 1 -- 172.21.15.162:0/862728027 wait complete. 2024-02-13T00:53:56.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe53868a700 1 librados: starting msgr at 2024-02-13T00:53:56.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe53868a700 1 librados: starting objecter 2024-02-13T00:53:56.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe53868a700 1 Processor -- start 2024-02-13T00:53:56.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe53868a700 1 -- start start 2024-02-13T00:53:56.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe53868a700 1 librados: setting wanted keys 2024-02-13T00:53:56.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe53868a700 1 librados: calling monclient init 2024-02-13T00:53:56.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe53868a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5300f7ca0 0x7fe53010e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:56.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe53868a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5300fcc40 0x7fe53010e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:56.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe53868a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5300fd5a0 0x7fe5301084a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:56.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe53868a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe530108990 con 0x7fe5300fcc40 2024-02-13T00:53:56.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe53868a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe530108b00 con 0x7fe5300f7ca0 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe53868a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe530108cc0 con 0x7fe5300fd5a0 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe535c25700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5300fcc40 0x7fe53010e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe535c25700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5300fcc40 0x7fe53010e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45656/0 (socket says 172.21.15.162:45656) 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe535c25700 1 -- 172.21.15.162:0/1114078172 learned_addr learned my addr 172.21.15.162:0/1114078172 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe536426700 1 --2- 172.21.15.162:0/1114078172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5300f7ca0 0x7fe53010e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.827+0000 7fe536c27700 1 --2- 172.21.15.162:0/1114078172 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5300fd5a0 0x7fe5301084a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe535c25700 1 -- 172.21.15.162:0/1114078172 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5300fd5a0 msgr2=0x7fe5301084a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe535c25700 1 --2- 172.21.15.162:0/1114078172 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5300fd5a0 0x7fe5301084a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe535c25700 1 -- 172.21.15.162:0/1114078172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5300f7ca0 msgr2=0x7fe53010e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe535c25700 1 --2- 172.21.15.162:0/1114078172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5300f7ca0 0x7fe53010e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe535c25700 1 -- 172.21.15.162:0/1114078172 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c005040 con 0x7fe5300fcc40 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe535c25700 1 --2- 172.21.15.162:0/1114078172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5300fcc40 0x7fe53010e7f0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7fe52000e970 tx=0x7fe52000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe5277fe700 10 client.?.objecter ms_handle_connect 0x7fe5300fcc40 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe5277fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:53:56.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe5277fe700 1 -- 172.21.15.162:0/1114078172 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52000ee60 con 0x7fe5300fcc40 2024-02-13T00:53:56.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe5277fe700 1 -- 172.21.15.162:0/1114078172 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe52000f040 con 0x7fe5300fcc40 2024-02-13T00:53:56.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe5277fe700 1 -- 172.21.15.162:0/1114078172 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe520007ea0 con 0x7fe5300fcc40 2024-02-13T00:53:56.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe53868a700 1 -- 172.21.15.162:0/1114078172 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe530108fa0 con 0x7fe5300fcc40 2024-02-13T00:53:56.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe53868a700 10 client.28146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:53:56.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe53868a700 1 -- 172.21.15.162:0/1114078172 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe530115130 con 0x7fe5300fcc40 2024-02-13T00:53:56.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.828+0000 7fe53868a700 1 librados: init done 2024-02-13T00:53:56.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.829+0000 7fe53868a700 1 -- 172.21.15.162:0/1114078172 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53005f9a0 con 0x7fe5300fcc40 2024-02-13T00:53:56.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:56.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.830+0000 7fe5277fe700 1 -- 172.21.15.162:0/1114078172 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe52000cb10 con 0x7fe5300fcc40 2024-02-13T00:53:56.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.831+0000 7fe5277fe700 1 --2- 172.21.15.162:0/1114078172 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe51c062c90 0x7fe51c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:53:56.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.831+0000 7fe5277fe700 1 -- 172.21.15.162:0/1114078172 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(786..786 src has 449..786) v4 ==== 8615+0+0 (secure 0 0 0) 0x7fe52007d1d0 con 0x7fe5300fcc40 2024-02-13T00:53:56.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.831+0000 7fe5277fe700 10 client.28146.objecter ms_dispatch 0x7fe530001550 osd_map(786..786 src has 449..786) v4 2024-02-13T00:53:56.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.831+0000 7fe5277fe700 3 client.28146.objecter handle_osd_map got epochs [786,786] > 0 2024-02-13T00:53:56.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.831+0000 7fe5277fe700 3 client.28146.objecter handle_osd_map decoding full epoch 786 2024-02-13T00:53:56.839 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.831+0000 7fe5277fe700 20 client.28146.objecter dump_active .. 0 homeless 2024-02-13T00:53:56.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:56.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:56.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.836+0000 7fe5277fe700 1 -- 172.21.15.162:0/1114078172 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe52004af60 con 0x7fe5300fcc40 2024-02-13T00:53:56.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.840+0000 7fe536426700 1 --2- 172.21.15.162:0/1114078172 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe51c062c90 0x7fe51c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:53:56.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.840+0000 7fe536426700 1 --2- 172.21.15.162:0/1114078172 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe51c062c90 0x7fe51c065150 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 rx=0x7fe52c0043d0 tx=0x7fe52c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:53:56.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:56.840+0000 7fe5277fe700 10 client.28146.objecter ms_handle_connect 0x7fe51c062c90 2024-02-13T00:53:56.860 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:56.860 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:53:56.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:56.974 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:57.038 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.034+0000 7fe53868a700 1 -- 172.21.15.162:0/1114078172 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7fe5300475e0 con 0x7fe5300fcc40 2024-02-13T00:53:57.167 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:55.972865+0000 mgr.y (mgr.14544) 651 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:57.168 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:56.469624+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:53:57.168 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:56.472554+0000 osd.6 (osd.6) 256 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:53:57.168 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:56.584788+0000 mon.a (mon.0) 3703 : cluster [DBG] osdmap e786: 8 total, 8 up, 4 in 2024-02-13T00:53:57.168 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:56.586694+0000 mon.a (mon.0) 3704 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:53:57.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:55.972865+0000 mgr.y (mgr.14544) 651 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:57.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:56.469624+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:53:57.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:56.472554+0000 osd.6 (osd.6) 256 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:53:57.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:56.584788+0000 mon.a (mon.0) 3703 : cluster [DBG] osdmap e786: 8 total, 8 up, 4 in 2024-02-13T00:53:57.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:56.586694+0000 mon.a (mon.0) 3704 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:53:57.172 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:57.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:57.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:53:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:55.972865+0000 mgr.y (mgr.14544) 651 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:53:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:56.469624+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:53:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:56.472554+0000 osd.6 (osd.6) 256 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:53:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:56.584788+0000 mon.a (mon.0) 3703 : cluster [DBG] osdmap e786: 8 total, 8 up, 4 in 2024-02-13T00:53:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:56.586694+0000 mon.a (mon.0) 3704 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:53:57.304 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:57.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:53:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:57.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:53:57.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:57.559 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:57.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.586+0000 7fe5277fe700 1 -- 172.21.15.162:0/1114078172 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v787) v1 ==== 113+0+0 (secure 0 0 0) 0x7fe520019070 con 0x7fe5300fcc40 2024-02-13T00:53:57.592 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 10 librados: watch_flush enter 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 10 librados: watch_flush exit 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 20 client.28146.objecter shutdown clearing up homeless session... 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 10 client.28146.objecter successfully canceled tick 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 -- 172.21.15.162:0/1114078172 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe51c062c90 msgr2=0x7fe51c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 --2- 172.21.15.162:0/1114078172 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe51c062c90 0x7fe51c065150 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 rx=0x7fe52c0043d0 tx=0x7fe52c00b040).stop 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 -- 172.21.15.162:0/1114078172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5300fcc40 msgr2=0x7fe53010e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 --2- 172.21.15.162:0/1114078172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5300fcc40 0x7fe53010e7f0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7fe52000e970 tx=0x7fe52000b040).stop 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 -- 172.21.15.162:0/1114078172 shutdown_connections 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 --2- 172.21.15.162:0/1114078172 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe51c062c90 0x7fe51c065150 unknown :-1 s=CLOSED pgs=3046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 --2- 172.21.15.162:0/1114078172 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5300fd5a0 0x7fe5301084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 --2- 172.21.15.162:0/1114078172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5300f7ca0 0x7fe53010e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 --2- 172.21.15.162:0/1114078172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5300fcc40 0x7fe53010e7f0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 -- 172.21.15.162:0/1114078172 >> 172.21.15.162:0/1114078172 conn(0x7fe53006c410 msgr2=0x7fe5300fb2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:53:57.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 -- 172.21.15.162:0/1114078172 shutdown_connections 2024-02-13T00:53:57.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 -- 172.21.15.162:0/1114078172 wait complete. 2024-02-13T00:53:57.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:53:57.589+0000 7fe53868a700 1 librados: shutdown 2024-02-13T00:53:57.594 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.0 to 1 (10000) 2024-02-13T00:53:57.806 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:57.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:57.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:53:57.943 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:57.963 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:53:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:57.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:53:58.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:56.546686+0000 mgr.y (mgr.14544) 652 : cluster [DBG] pgmap v1196: 172 pgs: 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:53:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:57.035871+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? 172.21.15.162:0/1114078172' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:53:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:57.457500+0000 osd.6 (osd.6) 257 : cluster [DBG] 6.a scrub starts 2024-02-13T00:53:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:57.460501+0000 osd.6 (osd.6) 258 : cluster [DBG] 6.a scrub ok 2024-02-13T00:53:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:57.584818+0000 mon.a (mon.0) 3706 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:53:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:57.584939+0000 mon.a (mon.0) 3707 : audit [INF] from='client.? 172.21.15.162:0/1114078172' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:53:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:57.584983+0000 mon.a (mon.0) 3708 : cluster [DBG] osdmap e787: 8 total, 8 up, 4 in 2024-02-13T00:53:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:57.627946+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:58.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:57.628228+0000 mon.a (mon.0) 3710 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-140"}]: dispatch 2024-02-13T00:53:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:56.546686+0000 mgr.y (mgr.14544) 652 : cluster [DBG] pgmap v1196: 172 pgs: 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:53:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:57.035871+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? 172.21.15.162:0/1114078172' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:53:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:57.457500+0000 osd.6 (osd.6) 257 : cluster [DBG] 6.a scrub starts 2024-02-13T00:53:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:57.460501+0000 osd.6 (osd.6) 258 : cluster [DBG] 6.a scrub ok 2024-02-13T00:53:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:57.584818+0000 mon.a (mon.0) 3706 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:53:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:57.584939+0000 mon.a (mon.0) 3707 : audit [INF] from='client.? 172.21.15.162:0/1114078172' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:53:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:57.584983+0000 mon.a (mon.0) 3708 : cluster [DBG] osdmap e787: 8 total, 8 up, 4 in 2024-02-13T00:53:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:57.627946+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:57.628228+0000 mon.a (mon.0) 3710 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-140"}]: dispatch 2024-02-13T00:53:58.099 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:58.119 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:58.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:56.546686+0000 mgr.y (mgr.14544) 652 : cluster [DBG] pgmap v1196: 172 pgs: 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:53:58.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:57.035871+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? 172.21.15.162:0/1114078172' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:53:58.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:57.457500+0000 osd.6 (osd.6) 257 : cluster [DBG] 6.a scrub starts 2024-02-13T00:53:58.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:57.460501+0000 osd.6 (osd.6) 258 : cluster [DBG] 6.a scrub ok 2024-02-13T00:53:58.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:57.584818+0000 mon.a (mon.0) 3706 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:53:58.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:57.584939+0000 mon.a (mon.0) 3707 : audit [INF] from='client.? 172.21.15.162:0/1114078172' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:53:58.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:57.584983+0000 mon.a (mon.0) 3708 : cluster [DBG] osdmap e787: 8 total, 8 up, 4 in 2024-02-13T00:53:58.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:57.627946+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:53:58.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:57.628228+0000 mon.a (mon.0) 3710 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-140"}]: dispatch 2024-02-13T00:53:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:58.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:58.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:53:58.252 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:58.271 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:53:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:58.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:53:58.376 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:58.405 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:58.458 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:58.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:58.567 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:58.598 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:58.622 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:58.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:58.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:58.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:53:58.752 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:58.772 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:53:58.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:58.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:53:58.875 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:53:58.902 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:59.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:53:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:59.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:53:59.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 1 1 2024-02-13T00:53:59.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:53:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:53:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:53:59.103 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:59.131 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:59.149 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:59.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:59.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:59.279 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:59.297 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:59.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:59.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:59.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:53:59.428 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:59.447 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:53:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:59.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:53:59.551 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:53:59.578 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:59.600 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:53:59.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:59.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:53:59.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:53:59.732 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:53:59.880 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:53:59.882 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:58.547430+0000 mgr.y (mgr.14544) 653 : cluster [DBG] pgmap v1199: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:53:59.882 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:53:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:58.586288+0000 mon.a (mon.0) 3711 : cluster [DBG] osdmap e788: 8 total, 8 up, 4 in 2024-02-13T00:53:59.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:58.547430+0000 mgr.y (mgr.14544) 653 : cluster [DBG] pgmap v1199: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:53:59.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:53:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:58.586288+0000 mon.a (mon.0) 3711 : cluster [DBG] osdmap e788: 8 total, 8 up, 4 in 2024-02-13T00:53:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:53:59.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:53:59.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:54:00.011 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:58.547430+0000 mgr.y (mgr.14544) 653 : cluster [DBG] pgmap v1199: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-13T00:54:00.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:53:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:58.586288+0000 mon.a (mon.0) 3711 : cluster [DBG] osdmap e788: 8 total, 8 up, 4 in 2024-02-13T00:54:00.803 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:53:59.590209+0000 mon.a (mon.0) 3712 : cluster [DBG] osdmap e789: 8 total, 8 up, 4 in 2024-02-13T00:54:00.803 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:53:59.591753+0000 mon.a (mon.0) 3713 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:54:00.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:53:59.590209+0000 mon.a (mon.0) 3712 : cluster [DBG] osdmap e789: 8 total, 8 up, 4 in 2024-02-13T00:54:00.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:53:59.591753+0000 mon.a (mon.0) 3713 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:54:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:53:59.590209+0000 mon.a (mon.0) 3712 : cluster [DBG] osdmap e789: 8 total, 8 up, 4 in 2024-02-13T00:54:01.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:53:59.591753+0000 mon.a (mon.0) 3713 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:54:01.932 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:54:01.932 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:54:01.932 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-13T00:54:01.932 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:54:01.958 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:00.548317+0000 mgr.y (mgr.14544) 654 : cluster [DBG] pgmap v1202: 204 pgs: 32 creating+peering, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:54:01.958 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:00.602340+0000 mon.a (mon.0) 3714 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:54:01.958 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:00.602465+0000 mon.a (mon.0) 3715 : cluster [DBG] osdmap e790: 8 total, 8 up, 4 in 2024-02-13T00:54:01.958 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:00.646183+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:54:01.958 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:00.646533+0000 mon.a (mon.0) 3717 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-142"}]: dispatch 2024-02-13T00:54:01.958 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:00.978599+0000 osd.0 (osd.0) 317 : cluster [DBG] 2.f scrub starts 2024-02-13T00:54:01.958 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:00.982070+0000 osd.0 (osd.0) 318 : cluster [DBG] 2.f scrub ok 2024-02-13T00:54:01.958 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:00.548317+0000 mgr.y (mgr.14544) 654 : cluster [DBG] pgmap v1202: 204 pgs: 32 creating+peering, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:54:01.959 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:00.602340+0000 mon.a (mon.0) 3714 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:54:01.959 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:00.602465+0000 mon.a (mon.0) 3715 : cluster [DBG] osdmap e790: 8 total, 8 up, 4 in 2024-02-13T00:54:01.959 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:00.646183+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:54:01.959 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:00.646533+0000 mon.a (mon.0) 3717 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-142"}]: dispatch 2024-02-13T00:54:01.959 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:00.978599+0000 osd.0 (osd.0) 317 : cluster [DBG] 2.f scrub starts 2024-02-13T00:54:01.959 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:00.982070+0000 osd.0 (osd.0) 318 : cluster [DBG] 2.f scrub ok 2024-02-13T00:54:02.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:00.548317+0000 mgr.y (mgr.14544) 654 : cluster [DBG] pgmap v1202: 204 pgs: 32 creating+peering, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:54:02.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:00.602340+0000 mon.a (mon.0) 3714 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:54:02.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:00.602465+0000 mon.a (mon.0) 3715 : cluster [DBG] osdmap e790: 8 total, 8 up, 4 in 2024-02-13T00:54:02.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:00.646183+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:54:02.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:00.646533+0000 mon.a (mon.0) 3717 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-142"}]: dispatch 2024-02-13T00:54:02.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:00.978599+0000 osd.0 (osd.0) 317 : cluster [DBG] 2.f scrub starts 2024-02-13T00:54:02.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:00.982070+0000 osd.0 (osd.0) 318 : cluster [DBG] 2.f scrub ok 2024-02-13T00:54:02.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.409+0000 7fac028d1700 1 -- 172.21.15.162:0/1447290385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fabfc0fe580 msgr2=0x7fabfc0f7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:02.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.409+0000 7fac028d1700 1 --2- 172.21.15.162:0/1447290385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fabfc0fe580 0x7fabfc0f7db0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7fabec004870 tx=0x7fabec009cb0).stop 2024-02-13T00:54:02.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.409+0000 7fac028d1700 1 -- 172.21.15.162:0/1447290385 shutdown_connections 2024-02-13T00:54:02.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.409+0000 7fac028d1700 1 --2- 172.21.15.162:0/1447290385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fabfc0f8490 0x7fabfc104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:02.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.409+0000 7fac028d1700 1 --2- 172.21.15.162:0/1447290385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabfc0fdc20 0x7fabfc0fe000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:02.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.409+0000 7fac028d1700 1 --2- 172.21.15.162:0/1447290385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fabfc0fe580 0x7fabfc0f7db0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.409+0000 7fac028d1700 1 -- 172.21.15.162:0/1447290385 >> 172.21.15.162:0/1447290385 conn(0x7fabfc0f3850 msgr2=0x7fabfc0f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.409+0000 7fac028d1700 1 -- 172.21.15.162:0/1447290385 shutdown_connections 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.409+0000 7fac028d1700 1 -- 172.21.15.162:0/1447290385 wait complete. 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac028d1700 1 librados: starting msgr at 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac028d1700 1 librados: starting objecter 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac028d1700 1 Processor -- start 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac028d1700 1 -- start start 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac028d1700 1 librados: setting wanted keys 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac028d1700 1 librados: calling monclient init 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac028d1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fabfc0f8490 0x7fabfc10e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac028d1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fabfc0fdc20 0x7fabfc10e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac028d1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabfc0fe580 0x7fabfc1082b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:02.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac028d1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc1087a0 con 0x7fabfc0f8490 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac028d1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc108910 con 0x7fabfc0fe580 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac028d1700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc108ad0 con 0x7fabfc0fdc20 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fabfbfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fabfc0f8490 0x7fabfc10e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fabfb7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fabfc0fdc20 0x7fabfc10e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fabfbfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fabfc0f8490 0x7fabfc10e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45664/0 (socket says 172.21.15.162:45664) 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fabfb7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fabfc0fdc20 0x7fabfc10e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:52206/0 (socket says 172.21.15.162:52206) 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fabfbfff700 1 -- 172.21.15.162:0/2387240454 learned_addr learned my addr 172.21.15.162:0/2387240454 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.410+0000 7fac00e6e700 1 --2- 172.21.15.162:0/2387240454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabfc0fe580 0x7fabfc1082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fabfb7fe700 1 -- 172.21.15.162:0/2387240454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabfc0fe580 msgr2=0x7fabfc1082b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fabfb7fe700 1 --2- 172.21.15.162:0/2387240454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabfc0fe580 0x7fabfc1082b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fabfb7fe700 1 -- 172.21.15.162:0/2387240454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fabfc0f8490 msgr2=0x7fabfc10e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fabfb7fe700 1 --2- 172.21.15.162:0/2387240454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fabfc0f8490 0x7fabfc10e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fabfb7fe700 1 -- 172.21.15.162:0/2387240454 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf40097c0 con 0x7fabfc0fdc20 2024-02-13T00:54:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fabfbfff700 1 --2- 172.21.15.162:0/2387240454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fabfc0f8490 0x7fabfc10e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:54:02.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fabfb7fe700 1 --2- 172.21.15.162:0/2387240454 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fabfc0fdc20 0x7fabfc10e600 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 rx=0x7fabec004870 tx=0x7fabec00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:02.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fabf97fa700 10 client.?.objecter ms_handle_connect 0x7fabfc0fdc20 2024-02-13T00:54:02.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fabf97fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:02.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fabf97fa700 1 -- 172.21.15.162:0/2387240454 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec00e070 con 0x7fabfc0fdc20 2024-02-13T00:54:02.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fabf97fa700 1 -- 172.21.15.162:0/2387240454 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabec013da0 con 0x7fabfc0fdc20 2024-02-13T00:54:02.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fac028d1700 1 -- 172.21.15.162:0/2387240454 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabec005040 con 0x7fabfc0fdc20 2024-02-13T00:54:02.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fabf97fa700 1 -- 172.21.15.162:0/2387240454 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec019070 con 0x7fabfc0fdc20 2024-02-13T00:54:02.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fac028d1700 10 client.38246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:02.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fac028d1700 1 -- 172.21.15.162:0/2387240454 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabfc109040 con 0x7fabfc0fdc20 2024-02-13T00:54:02.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.411+0000 7fac028d1700 1 librados: init done 2024-02-13T00:54:02.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.412+0000 7fac028d1700 1 -- 172.21.15.162:0/2387240454 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabfc0607b0 con 0x7fabfc0fdc20 2024-02-13T00:54:02.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.412+0000 7fabf97fa700 1 -- 172.21.15.162:0/2387240454 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fabec01e420 con 0x7fabfc0fdc20 2024-02-13T00:54:02.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.413+0000 7fabf97fa700 1 --2- 172.21.15.162:0/2387240454 >> v2:172.21.15.162:6800/2866534053 conn(0x7fabe4062c90 0x7fabe4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:02.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.413+0000 7fabf97fa700 1 -- 172.21.15.162:0/2387240454 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(791..791 src has 449..791) v4 ==== 8242+0+0 (secure 0 0 0) 0x7fabec07dcb0 con 0x7fabfc0fdc20 2024-02-13T00:54:02.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.413+0000 7fabf97fa700 10 client.38246.objecter ms_dispatch 0x7fabfc001550 osd_map(791..791 src has 449..791) v4 2024-02-13T00:54:02.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.413+0000 7fabf97fa700 3 client.38246.objecter handle_osd_map got epochs [791,791] > 0 2024-02-13T00:54:02.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.413+0000 7fabf97fa700 3 client.38246.objecter handle_osd_map decoding full epoch 791 2024-02-13T00:54:02.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.413+0000 7fabfbfff700 1 --2- 172.21.15.162:0/2387240454 >> v2:172.21.15.162:6800/2866534053 conn(0x7fabe4062c90 0x7fabe4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:02.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.413+0000 7fabf97fa700 20 client.38246.objecter dump_active .. 0 homeless 2024-02-13T00:54:02.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.413+0000 7fabf97fa700 10 client.38246.objecter ms_handle_connect 0x7fabe4062c90 2024-02-13T00:54:02.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.413+0000 7fabfbfff700 1 --2- 172.21.15.162:0/2387240454 >> v2:172.21.15.162:6800/2866534053 conn(0x7fabe4062c90 0x7fabe4065150 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 rx=0x7fabf0004fd0 tx=0x7fabf000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:02.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.417+0000 7fabf97fa700 1 -- 172.21.15.162:0/2387240454 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fabec051130 con 0x7fabfc0fdc20 2024-02-13T00:54:02.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.593+0000 7fac028d1700 1 -- 172.21.15.162:0/2387240454 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fabfc107110 con 0x7fabfc0fdc20 2024-02-13T00:54:02.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.605+0000 7fabf97fa700 1 -- 172.21.15.162:0/2387240454 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v792) v1 ==== 87+0+0 (secure 0 0 0) 0x7fabec050d20 con 0x7fabfc0fdc20 2024-02-13T00:54:02.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.610+0000 7fac028d1700 10 librados: watch_flush enter 2024-02-13T00:54:02.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.610+0000 7fac028d1700 10 librados: watch_flush exit 2024-02-13T00:54:02.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.610+0000 7fac028d1700 20 client.38246.objecter shutdown clearing up homeless session... 2024-02-13T00:54:02.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.610+0000 7fac028d1700 10 client.38246.objecter successfully canceled tick 2024-02-13T00:54:02.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.610+0000 7fac028d1700 1 -- 172.21.15.162:0/2387240454 >> v2:172.21.15.162:6800/2866534053 conn(0x7fabe4062c90 msgr2=0x7fabe4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:02.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.610+0000 7fac028d1700 1 --2- 172.21.15.162:0/2387240454 >> v2:172.21.15.162:6800/2866534053 conn(0x7fabe4062c90 0x7fabe4065150 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 rx=0x7fabf0004fd0 tx=0x7fabf000b040).stop 2024-02-13T00:54:02.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.610+0000 7fac028d1700 1 -- 172.21.15.162:0/2387240454 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fabfc0fdc20 msgr2=0x7fabfc10e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:02.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.610+0000 7fac028d1700 1 --2- 172.21.15.162:0/2387240454 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fabfc0fdc20 0x7fabfc10e600 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 rx=0x7fabec004870 tx=0x7fabec00b040).stop 2024-02-13T00:54:02.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.614+0000 7fac028d1700 1 -- 172.21.15.162:0/2387240454 shutdown_connections 2024-02-13T00:54:02.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.614+0000 7fac028d1700 1 --2- 172.21.15.162:0/2387240454 >> v2:172.21.15.162:6800/2866534053 conn(0x7fabe4062c90 0x7fabe4065150 unknown :-1 s=CLOSED pgs=3047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:02.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.614+0000 7fac028d1700 1 --2- 172.21.15.162:0/2387240454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabfc0fe580 0x7fabfc1082b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:02.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.614+0000 7fac028d1700 1 --2- 172.21.15.162:0/2387240454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fabfc0f8490 0x7fabfc10e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:02.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.614+0000 7fac028d1700 1 --2- 172.21.15.162:0/2387240454 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fabfc0fdc20 0x7fabfc10e600 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:02.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.614+0000 7fac028d1700 1 -- 172.21.15.162:0/2387240454 >> 172.21.15.162:0/2387240454 conn(0x7fabfc0f3850 msgr2=0x7fabfc0fba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:02.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.614+0000 7fac028d1700 1 -- 172.21.15.162:0/2387240454 shutdown_connections 2024-02-13T00:54:02.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.614+0000 7fac028d1700 1 -- 172.21.15.162:0/2387240454 wait complete. 2024-02-13T00:54:02.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.614+0000 7fac028d1700 1 librados: shutdown 2024-02-13T00:54:02.623 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is set 2024-02-13T00:54:02.663 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:54:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:02.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:54:02.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:54:02.801 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:02.822 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:54:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:02.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:54:02.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:54:02.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:01.480234+0000 osd.3 (osd.3) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:02.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:01.480390+0000 osd.3 (osd.3) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:02.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:01.603534+0000 mon.a (mon.0) 3718 : cluster [DBG] osdmap e791: 8 total, 8 up, 4 in 2024-02-13T00:54:02.931 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:02.594124+0000 mon.c (mon.2) 514 : audit [INF] from='client.? 172.21.15.162:0/2387240454' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:54:02.932 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:02.594400+0000 mon.a (mon.0) 3719 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:54:02.932 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:02.598813+0000 mon.a (mon.0) 3720 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:54:02.932 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:02.605109+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:54:02.932 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:02.605171+0000 mon.a (mon.0) 3722 : cluster [DBG] osdmap e792: 8 total, 8 up, 4 in 2024-02-13T00:54:02.932 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:02.606850+0000 mon.a (mon.0) 3723 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:54:02.932 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:02.768262+0000 mon.a (mon.0) 3724 : audit [DBG] from='client.? 172.21.15.162:0/3293767545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:54:02.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:01.480234+0000 osd.3 (osd.3) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:02.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:01.480390+0000 osd.3 (osd.3) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:02.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:01.603534+0000 mon.a (mon.0) 3718 : cluster [DBG] osdmap e791: 8 total, 8 up, 4 in 2024-02-13T00:54:02.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:02.594124+0000 mon.c (mon.2) 514 : audit [INF] from='client.? 172.21.15.162:0/2387240454' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:54:02.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:02.594400+0000 mon.a (mon.0) 3719 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:54:02.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:02.598813+0000 mon.a (mon.0) 3720 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:54:02.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:02.605109+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:54:02.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:02.605171+0000 mon.a (mon.0) 3722 : cluster [DBG] osdmap e792: 8 total, 8 up, 4 in 2024-02-13T00:54:02.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:02.606850+0000 mon.a (mon.0) 3723 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:54:02.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:02.768262+0000 mon.a (mon.0) 3724 : audit [DBG] from='client.? 172.21.15.162:0/3293767545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:54:02.956 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:03.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.764+0000 7f997c381700 1 -- 172.21.15.162:0/3710842860 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99740fcb10 msgr2=0x7f99740fcef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.764+0000 7f997c381700 1 --2- 172.21.15.162:0/3710842860 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99740fcb10 0x7f99740fcef0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 rx=0x7f9964009c30 tx=0x7f9964004c50).stop 2024-02-13T00:54:03.052 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.764+0000 7f997c381700 1 -- 172.21.15.162:0/3710842860 shutdown_connections 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.764+0000 7f997c381700 1 --2- 172.21.15.162:0/3710842860 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99740f7380 0x7f9974103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.764+0000 7f997c381700 1 --2- 172.21.15.162:0/3710842860 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99740fcb10 0x7f99740fcef0 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.764+0000 7f997c381700 1 --2- 172.21.15.162:0/3710842860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99740fd470 0x7f99740f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.764+0000 7f997c381700 1 -- 172.21.15.162:0/3710842860 >> 172.21.15.162:0/3710842860 conn(0x7f99740f2740 msgr2=0x7f99740f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.764+0000 7f997c381700 1 -- 172.21.15.162:0/3710842860 shutdown_connections 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.764+0000 7f997c381700 1 -- 172.21.15.162:0/3710842860 wait complete. 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997c381700 1 librados: starting msgr at 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997c381700 1 librados: starting objecter 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997c381700 1 Processor -- start 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997c381700 1 -- start start 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997c381700 1 librados: setting wanted keys 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997c381700 1 librados: calling monclient init 2024-02-13T00:54:03.053 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997c381700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99740f7380 0x7f997410d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997c381700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99740fcb10 0x7f997410d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997c381700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99740fd470 0x7f9974107260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997c381700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9974107750 con 0x7f99740f7380 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997c381700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99741078c0 con 0x7f99740fcb10 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997c381700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9974107a80 con 0x7f99740fd470 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997a11d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99740f7380 0x7f997410d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997a11d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99740f7380 0x7f997410d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45690/0 (socket says 172.21.15.162:45690) 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997a91e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99740fd470 0x7f9974107260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997a11d700 1 -- 172.21.15.162:0/3293767545 learned_addr learned my addr 172.21.15.162:0/3293767545 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997991c700 1 --2- 172.21.15.162:0/3293767545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99740fcb10 0x7f997410d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997a11d700 1 -- 172.21.15.162:0/3293767545 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99740fd470 msgr2=0x7f9974107260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997a11d700 1 --2- 172.21.15.162:0/3293767545 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99740fd470 0x7f9974107260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997a11d700 1 -- 172.21.15.162:0/3293767545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99740fcb10 msgr2=0x7f997410d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997a11d700 1 --2- 172.21.15.162:0/3293767545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99740fcb10 0x7f997410d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997a11d700 1 -- 172.21.15.162:0/3293767545 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9968005040 con 0x7f99740f7380 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997991c700 1 --2- 172.21.15.162:0/3293767545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99740fcb10 0x7f997410d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.765+0000 7f997a91e700 1 --2- 172.21.15.162:0/3293767545 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99740fd470 0x7f9974107260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.766+0000 7f997a11d700 1 --2- 172.21.15.162:0/3293767545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99740f7380 0x7f997410d0c0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7f9964004970 tx=0x7f9964004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.766+0000 7f99737fe700 10 client.?.objecter ms_handle_connect 0x7f99740f7380 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.766+0000 7f99737fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.766+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996400e040 con 0x7f99740f7380 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.766+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99640097c0 con 0x7f99740f7380 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.766+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9964018040 con 0x7f99740f7380 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.766+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964013e50 con 0x7f99740f7380 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.766+0000 7f997c381700 10 client.28158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.766+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9974107fa0 con 0x7f99740f7380 2024-02-13T00:54:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.766+0000 7f997c381700 1 librados: init done 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.767+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f99740607d0 con 0x7f99740f7380 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.767+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f9964007bf0 con 0x7f99740f7380 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f99737fe700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6800/2866534053 conn(0x7f995c062c90 0x7f995c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(792..792 src has 449..792) v4 ==== 8615+0+0 (secure 0 0 0) 0x7f996407ec20 con 0x7f99740f7380 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 osd_map(792..792 src has 449..792) v4 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f99737fe700 3 client.28158.objecter handle_osd_map got epochs [792,792] > 0 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f99737fe700 3 client.28158.objecter handle_osd_map decoding full epoch 792 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997991c700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6800/2866534053 conn(0x7f995c062c90 0x7f995c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f99737fe700 20 client.28158.objecter dump_active .. 0 homeless 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v792) v1 ==== 54+0+16 (secure 0 0 0) 0x7f996407f0c0 con 0x7f99740f7380 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f99737fe700 10 client.28158.objecter ms_handle_connect 0x7f995c062c90 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997991c700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6800/2866534053 conn(0x7f995c062c90 0x7f995c065150 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 rx=0x7f9968009f20 tx=0x7f9968009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:03.056 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997c381700 10 client.28158.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:03.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 1 2024-02-13T00:54:03.057 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:54:03.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9974028590 0x7f99740fd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:03.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f99740475e0 osd=0 3 2024-02-13T00:54:03.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f99740475e0 osd=0 3 2024-02-13T00:54:03.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f99740475e0 osd=0 3 2024-02-13T00:54:03.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 1 2024-02-13T00:54:03.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f99740475e0 osd=0 3 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 0 1 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f99740475e0 osd=0 4 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997c381700 10 client.28158.objecter _send_command 1 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9974106070 con 0x7f9974028590 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.768+0000 7f997a91e700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9974028590 0x7f99740fd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.769+0000 7f99737fe700 10 client.28158.objecter ms_handle_connect 0x7f9974028590 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.769+0000 7f997a91e700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9974028590 0x7f99740fd8b0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.769+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9974106070 con 0x7f9974028590 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.769+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.769+0000 7f99737fe700 10 client.28158.objecter _finish_command 1 = system:0 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.769+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f99740475e0 osd=0 4 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.769+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 0 1 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.784+0000 7f997c381700 10 client.28158.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:54:03.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.784+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 2 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.784+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f99740475e0 osd=0 3 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.784+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f99740475e0 osd=0 3 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.784+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f99740475e0 osd=0 3 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.784+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 2 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.784+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f99740475e0 osd=0 3 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.784+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 0 2 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.784+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f99740475e0 osd=0 4 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.784+0000 7f997c381700 10 client.28158.objecter _send_command 2 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.784+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f997405a0e0 con 0x7f9974028590 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f997405a0e0 con 0x7f9974028590 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f99737fe700 10 client.28158.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f99740475e0 osd=0 4 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 0 2 2024-02-13T00:54:03.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f997c381700 10 client.28158.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:03.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 3 2024-02-13T00:54:03.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6805/150599780 conn(0x7f9974059570 0x7f99740fdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:03.060 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:54:03.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974059400 osd=1 3 2024-02-13T00:54:03.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974059400 osd=1 3 2024-02-13T00:54:03.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974059400 osd=1 3 2024-02-13T00:54:03.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 3 2024-02-13T00:54:03.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f9974059400 osd=1 3 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 1 3 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974059400 osd=1 4 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f997c381700 10 client.28158.objecter _send_command 3 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.785+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9974061c80 con 0x7f9974059570 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.786+0000 7f997a11d700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6805/150599780 conn(0x7f9974059570 0x7f99740fdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.786+0000 7f99737fe700 10 client.28158.objecter ms_handle_connect 0x7f9974059570 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.786+0000 7f997a11d700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6805/150599780 conn(0x7f9974059570 0x7f99740fdda0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.786+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9974061c80 con 0x7f9974059570 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.786+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.786+0000 7f99737fe700 10 client.28158.objecter _finish_command 3 = system:0 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.786+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f9974059400 osd=1 4 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.786+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 1 3 2024-02-13T00:54:03.061 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.801+0000 7f997c381700 10 client.28158.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.801+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 4 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.801+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974059400 osd=1 3 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.801+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974059400 osd=1 3 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.801+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974059400 osd=1 3 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.801+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 4 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.801+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f9974059400 osd=1 3 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.801+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 1 4 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.801+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974059400 osd=1 4 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.801+0000 7f997c381700 10 client.28158.objecter _send_command 4 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.801+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9974061ea0 con 0x7f9974059570 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9974061ea0 con 0x7f9974059570 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f99737fe700 10 client.28158.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f9974059400 osd=1 4 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 1 4 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f997c381700 10 client.28158.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 5 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6809/3689719376 conn(0x7f997411a120 0x7f997411c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974061420 osd=2 3 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974061420 osd=2 3 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974061420 osd=2 3 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 5 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f9974061420 osd=2 3 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 2 5 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974061420 osd=2 4 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f997c381700 10 client.28158.objecter _send_command 5 2024-02-13T00:54:03.063 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.802+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99740fef70 con 0x7f997411a120 2024-02-13T00:54:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:54:03.064 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:54:03.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.803+0000 7f997991c700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6809/3689719376 conn(0x7f997411a120 0x7f997411c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:03.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.803+0000 7f99737fe700 10 client.28158.objecter ms_handle_connect 0x7f997411a120 2024-02-13T00:54:03.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.803+0000 7f997991c700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6809/3689719376 conn(0x7f997411a120 0x7f997411c510 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:03.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.803+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f99740fef70 con 0x7f997411a120 2024-02-13T00:54:03.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.803+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:54:03.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.803+0000 7f99737fe700 10 client.28158.objecter _finish_command 5 = system:0 2024-02-13T00:54:03.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.803+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f9974061420 osd=2 4 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.803+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 2 5 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.817+0000 7f997c381700 10 client.28158.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.817+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 6 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.817+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974061420 osd=2 3 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.817+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974061420 osd=2 3 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.817+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974061420 osd=2 3 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.817+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 6 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.817+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f9974061420 osd=2 3 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.817+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 2 6 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.817+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974061420 osd=2 4 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.817+0000 7f997c381700 10 client.28158.objecter _send_command 6 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.817+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f99740fe4d0 con 0x7f997411a120 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f99740fe4d0 con 0x7f997411a120 2024-02-13T00:54:03.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f99737fe700 10 client.28158.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f9974061420 osd=2 4 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 2 6 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997c381700 10 client.28158.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 7 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6813/604971011 conn(0x7f997411db80 0x7f997411ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974027be0 osd=3 3 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027be0 osd=3 3 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974027be0 osd=3 3 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 7 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f9974027be0 osd=3 3 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 3 7 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027be0 osd=3 4 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997c381700 10 client.28158.objecter _send_command 7 2024-02-13T00:54:03.066 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9974028260 con 0x7f997411db80 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997a91e700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6813/604971011 conn(0x7f997411db80 0x7f997411ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f99737fe700 10 client.28158.objecter ms_handle_connect 0x7f997411db80 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.819+0000 7f997a91e700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6813/604971011 conn(0x7f997411db80 0x7f997411ff70 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.820+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9974028260 con 0x7f997411db80 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.820+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.820+0000 7f99737fe700 10 client.28158.objecter _finish_command 7 = system:0 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.820+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f9974027be0 osd=3 4 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.820+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 3 7 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.834+0000 7f997c381700 10 client.28158.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.834+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 8 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.834+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974027be0 osd=3 3 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.834+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027be0 osd=3 3 2024-02-13T00:54:03.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.834+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974027be0 osd=3 3 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.834+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 8 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.834+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f9974027be0 osd=3 3 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.834+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 3 8 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.834+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027be0 osd=3 4 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.834+0000 7f997c381700 10 client.28158.objecter _send_command 8 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.834+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9974027650 con 0x7f997411db80 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9974027650 con 0x7f997411db80 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f99737fe700 10 client.28158.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f9974027be0 osd=3 4 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 3 8 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997c381700 10 client.28158.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 9 2024-02-13T00:54:03.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6800/3344263388 conn(0x7f997411d6e0 0x7f99741235f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:03.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974027650 osd=4 3 2024-02-13T00:54:03.069 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:54:03.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027650 osd=4 3 2024-02-13T00:54:03.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974027650 osd=4 3 2024-02-13T00:54:03.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 9 2024-02-13T00:54:03.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f9974027650 osd=4 3 2024-02-13T00:54:03.069 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 4 9 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027650 osd=4 4 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997c381700 10 client.28158.objecter _send_command 9 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9974027820 con 0x7f997411d6e0 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.836+0000 7f997a11d700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6800/3344263388 conn(0x7f997411d6e0 0x7f99741235f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.837+0000 7f99737fe700 10 client.28158.objecter ms_handle_connect 0x7f997411d6e0 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.837+0000 7f997a11d700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6800/3344263388 conn(0x7f997411d6e0 0x7f99741235f0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.837+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9974027820 con 0x7f997411d6e0 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.837+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.837+0000 7f99737fe700 10 client.28158.objecter _finish_command 9 = system:0 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.837+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f9974027650 osd=4 4 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.837+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 4 9 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.852+0000 7f997c381700 10 client.28158.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:54:03.070 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.852+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 10 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.852+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974027650 osd=4 3 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.852+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027650 osd=4 3 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.852+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974027650 osd=4 3 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.852+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 10 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.852+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f9974027650 osd=4 3 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.852+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 4 10 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.852+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027650 osd=4 4 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.852+0000 7f997c381700 10 client.28158.objecter _send_command 10 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.852+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9974027820 con 0x7f997411d6e0 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.853+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f9974027820 con 0x7f997411d6e0 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.853+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.853+0000 7f99737fe700 10 client.28158.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.853+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f9974027650 osd=4 4 2024-02-13T00:54:03.071 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.853+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 4 10 2024-02-13T00:54:03.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997c381700 10 client.28158.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:03.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 11 2024-02-13T00:54:03.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6804/2277917372 conn(0x7f9974123ff0 0x7f99741263e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:03.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974027820 osd=5 3 2024-02-13T00:54:03.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027820 osd=5 3 2024-02-13T00:54:03.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974027820 osd=5 3 2024-02-13T00:54:03.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 11 2024-02-13T00:54:03.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f9974027820 osd=5 3 2024-02-13T00:54:03.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 5 11 2024-02-13T00:54:03.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027820 osd=5 4 2024-02-13T00:54:03.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997c381700 10 client.28158.objecter _send_command 11 2024-02-13T00:54:03.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99741290c0 con 0x7f9974123ff0 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997991c700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6804/2277917372 conn(0x7f9974123ff0 0x7f99741263e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f997991c700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6804/2277917372 conn(0x7f9974123ff0 0x7f99741263e0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.854+0000 7f99737fe700 10 client.28158.objecter ms_handle_connect 0x7f9974123ff0 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.855+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f99741290c0 con 0x7f9974123ff0 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.855+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.855+0000 7f99737fe700 10 client.28158.objecter _finish_command 11 = system:0 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.855+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f9974027820 osd=5 4 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.855+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 5 11 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.871+0000 7f997c381700 10 client.28158.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.871+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 12 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.871+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974027820 osd=5 3 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.871+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027820 osd=5 3 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.871+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f9974027820 osd=5 3 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.871+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 12 2024-02-13T00:54:03.073 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.871+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f9974027820 osd=5 3 2024-02-13T00:54:03.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.871+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 5 12 2024-02-13T00:54:03.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.871+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027820 osd=5 4 2024-02-13T00:54:03.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.871+0000 7f997c381700 10 client.28158.objecter _send_command 12 2024-02-13T00:54:03.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.871+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f99741290c0 con 0x7f9974123ff0 2024-02-13T00:54:03.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f99741290c0 con 0x7f9974123ff0 2024-02-13T00:54:03.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:54:03.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f99737fe700 10 client.28158.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f9974027820 osd=5 4 2024-02-13T00:54:03.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 5 12 2024-02-13T00:54:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:54:03.074 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:54:03.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997c381700 10 client.28158.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:03.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 13 2024-02-13T00:54:03.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9974126e20 0x7f9974127280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:03.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f99741290c0 osd=6 3 2024-02-13T00:54:03.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f99741290c0 osd=6 3 2024-02-13T00:54:03.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f99741290c0 osd=6 3 2024-02-13T00:54:03.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 13 2024-02-13T00:54:03.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f99741290c0 osd=6 3 2024-02-13T00:54:03.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 6 13 2024-02-13T00:54:03.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f99741290c0 osd=6 4 2024-02-13T00:54:03.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997c381700 10 client.28158.objecter _send_command 13 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9974128ed0 con 0x7f9974126e20 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.873+0000 7f997a91e700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9974126e20 0x7f9974127280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.874+0000 7f99737fe700 10 client.28158.objecter ms_handle_connect 0x7f9974126e20 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.874+0000 7f997a91e700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9974126e20 0x7f9974127280 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.874+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f996c021370 con 0x7f9974126e20 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.874+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.874+0000 7f99737fe700 10 client.28158.objecter _finish_command 13 = system:0 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.874+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f99741290c0 osd=6 4 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.874+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 6 13 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.888+0000 7f997c381700 10 client.28158.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.888+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 14 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.888+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f99741290c0 osd=6 3 2024-02-13T00:54:03.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.888+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f99741290c0 osd=6 3 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.888+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f99741290c0 osd=6 3 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.888+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 14 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.888+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f99741290c0 osd=6 3 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.888+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 6 14 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.888+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f99741290c0 osd=6 4 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.888+0000 7f997c381700 10 client.28158.objecter _send_command 14 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.888+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f997412b370 con 0x7f9974126e20 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f997412b370 con 0x7f9974126e20 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f99737fe700 10 client.28158.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f99741290c0 osd=6 4 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 6 14 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997c381700 10 client.28158.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:03.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 15 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6812/1964192772 conn(0x7f997412ba30 0x7f997412be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f997412b370 osd=7 3 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f997412b370 osd=7 3 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f997412b370 osd=7 3 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 15 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f997412b370 osd=7 3 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 7 15 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f997412b370 osd=7 4 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997c381700 10 client.28158.objecter _send_command 15 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f997412f730 con 0x7f997412ba30 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997a11d700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6812/1964192772 conn(0x7f997412ba30 0x7f997412be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f99737fe700 10 client.28158.objecter ms_handle_connect 0x7f997412ba30 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.891+0000 7f997a11d700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6812/1964192772 conn(0x7f997412ba30 0x7f997412be70 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:03.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.892+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f997412f730 con 0x7f997412ba30 2024-02-13T00:54:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.892+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:54:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.892+0000 7f99737fe700 10 client.28158.objecter _finish_command 15 = system:0 2024-02-13T00:54:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.892+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f997412b370 osd=7 4 2024-02-13T00:54:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.892+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 7 15 2024-02-13T00:54:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.906+0000 7f997c381700 10 client.28158.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-13T00:54:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.906+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign -1 16 2024-02-13T00:54:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.906+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f997412b370 osd=7 3 2024-02-13T00:54:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.906+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f997412b370 osd=7 3 2024-02-13T00:54:03.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.906+0000 7f997c381700 20 client.28158.objecter _get_session s=0x7f997412b370 osd=7 3 2024-02-13T00:54:03.079 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:54:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.906+0000 7f997c381700 15 client.28158.objecter _session_command_op_remove -1 16 2024-02-13T00:54:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.906+0000 7f997c381700 20 client.28158.objecter get_session s=0x7f997412b370 osd=7 3 2024-02-13T00:54:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.906+0000 7f997c381700 15 client.28158.objecter _session_command_op_assign 7 16 2024-02-13T00:54:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.906+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f997412b370 osd=7 4 2024-02-13T00:54:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.906+0000 7f997c381700 10 client.28158.objecter _send_command 16 2024-02-13T00:54:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.906+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f997412e5b0 con 0x7f997412ba30 2024-02-13T00:54:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f99737fe700 1 -- 172.21.15.162:0/3293767545 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f997412e5b0 con 0x7f997412ba30 2024-02-13T00:54:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f99737fe700 10 client.28158.objecter ms_dispatch 0x7f9974001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-13T00:54:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f99737fe700 10 client.28158.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f99737fe700 20 client.28158.objecter put_session s=0x7f997412b370 osd=7 4 2024-02-13T00:54:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f99737fe700 15 client.28158.objecter _session_command_op_remove 7 16 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 10 librados: watch_flush enter 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 10 librados: watch_flush exit 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 10 client.28158.objecter close_session for osd.0 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9974028590 msgr2=0x7f99740fd8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6801/2103743338 conn(0x7f9974028590 0x7f99740fd8b0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f99740475e0 osd=0 1 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 10 client.28158.objecter close_session for osd.1 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6805/150599780 conn(0x7f9974059570 msgr2=0x7f99740fdda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6805/150599780 conn(0x7f9974059570 0x7f99740fdda0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974059400 osd=1 1 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 10 client.28158.objecter close_session for osd.2 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6809/3689719376 conn(0x7f997411a120 msgr2=0x7f997411c510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6809/3689719376 conn(0x7f997411a120 0x7f997411c510 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974061420 osd=2 1 2024-02-13T00:54:03.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 10 client.28158.objecter close_session for osd.3 2024-02-13T00:54:03.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6813/604971011 conn(0x7f997411db80 msgr2=0x7f997411ff70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997a91e700 1 -- 172.21.15.162:0/3293767545 reap_dead start 2024-02-13T00:54:03.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6813/604971011 conn(0x7f997411db80 0x7f997411ff70 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027be0 osd=3 1 2024-02-13T00:54:03.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 10 client.28158.objecter close_session for osd.4 2024-02-13T00:54:03.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6800/3344263388 conn(0x7f997411d6e0 msgr2=0x7f99741235f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6800/3344263388 conn(0x7f997411d6e0 0x7f99741235f0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027650 osd=4 1 2024-02-13T00:54:03.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 10 client.28158.objecter close_session for osd.5 2024-02-13T00:54:03.082 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6804/2277917372 conn(0x7f9974123ff0 msgr2=0x7f99741263e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6804/2277917372 conn(0x7f9974123ff0 0x7f99741263e0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f9974027820 osd=5 1 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 10 client.28158.objecter close_session for osd.6 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9974126e20 msgr2=0x7f9974127280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6808/1506640341 conn(0x7f9974126e20 0x7f9974127280 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f99741290c0 osd=6 1 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 10 client.28158.objecter close_session for osd.7 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6812/1964192772 conn(0x7f997412ba30 msgr2=0x7f997412be70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.181:6812/1964192772 conn(0x7f997412ba30 0x7f997412be70 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 20 client.28158.objecter put_session s=0x7f997412b370 osd=7 1 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 20 client.28158.objecter shutdown clearing up homeless session... 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 10 client.28158.objecter successfully canceled tick 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997a91e700 1 -- 172.21.15.162:0/3293767545 reap_dead start 2024-02-13T00:54:03.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6800/2866534053 conn(0x7f995c062c90 msgr2=0x7f995c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6800/2866534053 conn(0x7f995c062c90 0x7f995c065150 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 rx=0x7f9968009f20 tx=0x7f9968009cb0).stop 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99740f7380 msgr2=0x7f997410d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99740f7380 0x7f997410d0c0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7f9964004970 tx=0x7f9964004c50).stop 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 shutdown_connections 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99740f7380 0x7f997410d0c0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 --2- 172.21.15.162:0/3293767545 >> v2:172.21.15.162:6800/2866534053 conn(0x7f995c062c90 0x7f995c065150 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 >> 172.21.15.162:0/3293767545 conn(0x7f99740f2740 msgr2=0x7f99740fa900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 shutdown_connections 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.908+0000 7f997c381700 1 -- 172.21.15.162:0/3293767545 wait complete. 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:02.909+0000 7f997c381700 1 librados: shutdown 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.084 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.085 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.085 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.085 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.085 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.085 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-13T00:54:03.205 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:54:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:03.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:54:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:01.480234+0000 osd.3 (osd.3) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:01.480390+0000 osd.3 (osd.3) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:01.603534+0000 mon.a (mon.0) 3718 : cluster [DBG] osdmap e791: 8 total, 8 up, 4 in 2024-02-13T00:54:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:02.594124+0000 mon.c (mon.2) 514 : audit [INF] from='client.? 172.21.15.162:0/2387240454' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:54:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:02.594400+0000 mon.a (mon.0) 3719 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-13T00:54:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:02.598813+0000 mon.a (mon.0) 3720 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:54:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:02.605109+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-13T00:54:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:02.605171+0000 mon.a (mon.0) 3722 : cluster [DBG] osdmap e792: 8 total, 8 up, 4 in 2024-02-13T00:54:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:02.606850+0000 mon.a (mon.0) 3723 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-13T00:54:03.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:02.768262+0000 mon.a (mon.0) 3724 : audit [DBG] from='client.? 172.21.15.162:0/3293767545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:54:03.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:54:03.341 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:03.362 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:54:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:03.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:54:03.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7 2024-02-13T00:54:03.497 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:03.517 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:54:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:03.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7' 2024-02-13T00:54:03.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:54:03.649 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:03.668 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:54:03.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:03.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:54:03.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-13T00:54:03.723 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:03.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:54:03.798 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:03.976 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:02.548710+0000 mgr.y (mgr.14544) 655 : cluster [DBG] pgmap v1205: 172 pgs: 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:54:03.976 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:03.604876+0000 mon.a (mon.0) 3725 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:54:03.976 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:03.604982+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e793: 8 total, 8 up, 4 in 2024-02-13T00:54:03.976 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:03.608487+0000 mon.a (mon.0) 3727 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-101","var": "dedup_tier","val": "test-rados-api-smithi162-84057-144"}]: dispatch 2024-02-13T00:54:03.976 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:02.548710+0000 mgr.y (mgr.14544) 655 : cluster [DBG] pgmap v1205: 172 pgs: 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:54:03.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:03.604876+0000 mon.a (mon.0) 3725 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:54:03.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:03.604982+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e793: 8 total, 8 up, 4 in 2024-02-13T00:54:03.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:03.608487+0000 mon.a (mon.0) 3727 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-101","var": "dedup_tier","val": "test-rados-api-smithi162-84057-144"}]: dispatch 2024-02-13T00:54:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:02.548710+0000 mgr.y (mgr.14544) 655 : cluster [DBG] pgmap v1205: 172 pgs: 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-13T00:54:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:03.604876+0000 mon.a (mon.0) 3725 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-84057-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-13T00:54:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:03.604982+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e793: 8 total, 8 up, 4 in 2024-02-13T00:54:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:03.608487+0000 mon.a (mon.0) 3727 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-84057-101","var": "dedup_tier","val": "test-rados-api-smithi162-84057-144"}]: dispatch 2024-02-13T00:54:04.753 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:54:04.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:04.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:54:04.795 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:54:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:54:04+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-02-13T00:54:04.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2 2024-02-13T00:54:04.885 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:04.984 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:54:04.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:04.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2' 2024-02-13T00:54:05.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:54:05.119 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:05.568 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:54:05.570 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:04.346336+0000 mon.a (mon.0) 3728 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:54:05.570 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:04.549239+0000 mgr.y (mgr.14544) 656 : cluster [DBG] pgmap v1208: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:54:05.570 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:04.597528+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:54:05.570 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:04.597843+0000 mon.a (mon.0) 3730 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-144"}]: dispatch 2024-02-13T00:54:05.571 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:04.605613+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e794: 8 total, 8 up, 4 in 2024-02-13T00:54:05.571 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:04.346336+0000 mon.a (mon.0) 3728 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:54:05.571 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:04.549239+0000 mgr.y (mgr.14544) 656 : cluster [DBG] pgmap v1208: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:54:05.571 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:04.597528+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:54:05.571 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:04.597843+0000 mon.a (mon.0) 3730 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-144"}]: dispatch 2024-02-13T00:54:05.571 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:04.605613+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e794: 8 total, 8 up, 4 in 2024-02-13T00:54:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:05.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:54:05.596 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:05.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:54:05.702 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:04.346336+0000 mon.a (mon.0) 3728 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:54:05.702 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:04.549239+0000 mgr.y (mgr.14544) 656 : cluster [DBG] pgmap v1208: 204 pgs: 32 unknown, 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:54:05.702 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:04.597528+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:54:05.703 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:04.597843+0000 mon.a (mon.0) 3730 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-84057-101", "tierpool": "test-rados-api-smithi162-84057-144"}]: dispatch 2024-02-13T00:54:05.703 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:04.605613+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e794: 8 total, 8 up, 4 in 2024-02-13T00:54:05.705 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:05.726 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:54:05.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:05.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:54:05.830 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:54:05.859 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:06.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:54:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:54:06.411 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-13T00:54:06.440 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:54:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:54:06] "GET /metrics HTTP/1.1" 200 190323 "" "Prometheus/2.33.4" 2024-02-13T00:54:06.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:54:06.712 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:05.585997+0000 mon.a (mon.0) 3732 : cluster [DBG] osdmap e795: 8 total, 8 up, 4 in 2024-02-13T00:54:06.712 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:05.983929+0000 mgr.y (mgr.14544) 657 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:06.712 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:05.585997+0000 mon.a (mon.0) 3732 : cluster [DBG] osdmap e795: 8 total, 8 up, 4 in 2024-02-13T00:54:06.712 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:05.983929+0000 mgr.y (mgr.14544) 657 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:06.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:06.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:54:06.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-13T00:54:06.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.806+0000 7fbe6da38700 1 -- 172.21.15.162:0/2042254149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe680f68c0 msgr2=0x7fbe68104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:06.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.806+0000 7fbe6da38700 1 --2- 172.21.15.162:0/2042254149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe680f68c0 0x7fbe68104ac0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 rx=0x7fbe60009be0 tx=0x7fbe60004d10).stop 2024-02-13T00:54:06.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.806+0000 7fbe6da38700 1 -- 172.21.15.162:0/2042254149 shutdown_connections 2024-02-13T00:54:06.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.806+0000 7fbe6da38700 1 --2- 172.21.15.162:0/2042254149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe680f68c0 0x7fbe68104ac0 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:06.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.806+0000 7fbe6da38700 1 --2- 172.21.15.162:0/2042254149 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbe680fdcb0 0x7fbe680fe090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:06.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.806+0000 7fbe6da38700 1 --2- 172.21.15.162:0/2042254149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe680fe610 0x7fbe680f61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:06.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.806+0000 7fbe6da38700 1 -- 172.21.15.162:0/2042254149 >> 172.21.15.162:0/2042254149 conn(0x7fbe68070080 msgr2=0x7fbe680f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:06.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.806+0000 7fbe6da38700 1 -- 172.21.15.162:0/2042254149 shutdown_connections 2024-02-13T00:54:06.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.806+0000 7fbe6da38700 1 -- 172.21.15.162:0/2042254149 wait complete. 2024-02-13T00:54:06.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.806+0000 7fbe6da38700 1 librados: starting msgr at 2024-02-13T00:54:06.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.806+0000 7fbe6da38700 1 librados: starting objecter 2024-02-13T00:54:06.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe6da38700 1 Processor -- start 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe6da38700 1 -- start start 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe6da38700 1 librados: setting wanted keys 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe6da38700 1 librados: calling monclient init 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe6da38700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe680f68c0 0x7fbe680f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe6da38700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe680fdcb0 0x7fbe680f79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe6da38700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbe680fe610 0x7fbe680f7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe6da38700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe681165b0 con 0x7fbe680f68c0 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe6da38700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68116720 con 0x7fbe680fdcb0 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe6da38700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe681168e0 con 0x7fbe680fe610 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe66ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe680f68c0 0x7fbe680f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe66ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe680f68c0 0x7fbe680f74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:56400/0 (socket says 172.21.15.162:56400) 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe66ffd700 1 -- 172.21.15.162:0/4182503271 learned_addr learned my addr 172.21.15.162:0/4182503271 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:06.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe677fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbe680fe610 0x7fbe680f7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe667fc700 1 --2- 172.21.15.162:0/4182503271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe680fdcb0 0x7fbe680f79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe66ffd700 1 -- 172.21.15.162:0/4182503271 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbe680fe610 msgr2=0x7fbe680f7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe66ffd700 1 --2- 172.21.15.162:0/4182503271 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbe680fe610 0x7fbe680f7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe66ffd700 1 -- 172.21.15.162:0/4182503271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe680fdcb0 msgr2=0x7fbe680f79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe66ffd700 1 --2- 172.21.15.162:0/4182503271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe680fdcb0 0x7fbe680f79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe66ffd700 1 -- 172.21.15.162:0/4182503271 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe58005040 con 0x7fbe680f68c0 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.807+0000 7fbe677fe700 1 --2- 172.21.15.162:0/4182503271 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbe680fe610 0x7fbe680f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.808+0000 7fbe66ffd700 1 --2- 172.21.15.162:0/4182503271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe680f68c0 0x7fbe680f74b0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 rx=0x7fbe580048f0 tx=0x7fbe58009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.808+0000 7fbe4ffff700 10 client.?.objecter ms_handle_connect 0x7fbe680f68c0 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.808+0000 7fbe4ffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.808+0000 7fbe4ffff700 1 -- 172.21.15.162:0/4182503271 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58007b00 con 0x7fbe680f68c0 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.808+0000 7fbe6da38700 1 -- 172.21.15.162:0/4182503271 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe600097c0 con 0x7fbe680f68c0 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.808+0000 7fbe4ffff700 1 -- 172.21.15.162:0/4182503271 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe5800f040 con 0x7fbe680f68c0 2024-02-13T00:54:06.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.808+0000 7fbe4ffff700 1 -- 172.21.15.162:0/4182503271 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58014990 con 0x7fbe680f68c0 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.808+0000 7fbe6da38700 10 client.28164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.809+0000 7fbe6da38700 1 -- 172.21.15.162:0/4182503271 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe68116f70 con 0x7fbe680f68c0 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.809+0000 7fbe6da38700 1 librados: init done 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.809+0000 7fbe4ffff700 1 -- 172.21.15.162:0/4182503271 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fbe58010500 con 0x7fbe680f68c0 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.809+0000 7fbe6da38700 1 -- 172.21.15.162:0/4182503271 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6805fb80 con 0x7fbe680f68c0 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.810+0000 7fbe4ffff700 1 --2- 172.21.15.162:0/4182503271 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbe50062c90 0x7fbe50065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.810+0000 7fbe4ffff700 1 -- 172.21.15.162:0/4182503271 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(796..796 src has 449..796) v4 ==== 7823+0+0 (secure 0 0 0) 0x7fbe5807f1e0 con 0x7fbe680f68c0 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.810+0000 7fbe4ffff700 10 client.28164.objecter ms_dispatch 0x7fbe68001550 osd_map(796..796 src has 449..796) v4 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.810+0000 7fbe4ffff700 3 client.28164.objecter handle_osd_map got epochs [796,796] > 0 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.810+0000 7fbe4ffff700 3 client.28164.objecter handle_osd_map decoding full epoch 796 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.810+0000 7fbe667fc700 1 --2- 172.21.15.162:0/4182503271 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbe50062c90 0x7fbe50065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.810+0000 7fbe4ffff700 20 client.28164.objecter dump_active .. 0 homeless 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.810+0000 7fbe4ffff700 10 client.28164.objecter ms_handle_connect 0x7fbe50062c90 2024-02-13T00:54:06.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.810+0000 7fbe667fc700 1 --2- 172.21.15.162:0/4182503271 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbe50062c90 0x7fbe50065150 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 rx=0x7fbe680f8c50 tx=0x7fbe5c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:06.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5 2024-02-13T00:54:06.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:06.815+0000 7fbe4ffff700 1 -- 172.21.15.162:0/4182503271 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fbe58050090 con 0x7fbe680f68c0 2024-02-13T00:54:06.846 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:05.585997+0000 mon.a (mon.0) 3732 : cluster [DBG] osdmap e795: 8 total, 8 up, 4 in 2024-02-13T00:54:06.846 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:05.983929+0000 mgr.y (mgr.14544) 657 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:06.849 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:06.870 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:54:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:06.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5' 2024-02-13T00:54:06.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:54:07.002 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:07.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.012+0000 7fbe6da38700 1 -- 172.21.15.162:0/4182503271 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fbe680f8a20 con 0x7fbe680f68c0 2024-02-13T00:54:07.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:54:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:07.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:54:07.218 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:07.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6 2024-02-13T00:54:07.338 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:07.358 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:54:07.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:07.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6' 2024-02-13T00:54:07.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:54:07.490 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:07.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.595+0000 7fbe4ffff700 1 -- 172.21.15.162:0/4182503271 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v797) v1 ==== 113+0+0 (secure 0 0 0) 0x7fbe5804fc80 con 0x7fbe680f68c0 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 10 librados: watch_flush enter 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 10 librados: watch_flush exit 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 20 client.28164.objecter shutdown clearing up homeless session... 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 10 client.28164.objecter successfully canceled tick 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 1 -- 172.21.15.162:0/4182503271 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbe50062c90 msgr2=0x7fbe50065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 1 --2- 172.21.15.162:0/4182503271 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbe50062c90 0x7fbe50065150 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 rx=0x7fbe680f8c50 tx=0x7fbe5c00b040).stop 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 1 -- 172.21.15.162:0/4182503271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe680f68c0 msgr2=0x7fbe680f74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 1 --2- 172.21.15.162:0/4182503271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe680f68c0 0x7fbe680f74b0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 rx=0x7fbe580048f0 tx=0x7fbe58009cb0).stop 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 1 -- 172.21.15.162:0/4182503271 shutdown_connections 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 1 --2- 172.21.15.162:0/4182503271 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbe50062c90 0x7fbe50065150 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 1 --2- 172.21.15.162:0/4182503271 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbe680fe610 0x7fbe680f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 1 --2- 172.21.15.162:0/4182503271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe680f68c0 0x7fbe680f74b0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:07.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 1 --2- 172.21.15.162:0/4182503271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbe680fdcb0 0x7fbe680f79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:07.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.597+0000 7fbe6da38700 1 -- 172.21.15.162:0/4182503271 >> 172.21.15.162:0/4182503271 conn(0x7fbe68070080 msgr2=0x7fbe6806d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:07.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.599+0000 7fbe6da38700 1 -- 172.21.15.162:0/4182503271 shutdown_connections 2024-02-13T00:54:07.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.599+0000 7fbe6da38700 1 -- 172.21.15.162:0/4182503271 wait complete. 2024-02-13T00:54:07.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:07.599+0000 7fbe6da38700 1 librados: shutdown 2024-02-13T00:54:07.604 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.1 to 1 (10000) 2024-02-13T00:54:07.627 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:06.549632+0000 mgr.y (mgr.14544) 658 : cluster [DBG] pgmap v1211: 172 pgs: 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-13T00:54:07.627 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:06.593135+0000 mon.a (mon.0) 3733 : cluster [DBG] osdmap e796: 8 total, 8 up, 4 in 2024-02-13T00:54:07.627 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:06.594760+0000 mon.a (mon.0) 3734 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:54:07.627 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:07.013673+0000 mon.a (mon.0) 3735 : audit [INF] from='client.? 172.21.15.162:0/4182503271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:54:07.627 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:06.549632+0000 mgr.y (mgr.14544) 658 : cluster [DBG] pgmap v1211: 172 pgs: 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-13T00:54:07.628 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:06.593135+0000 mon.a (mon.0) 3733 : cluster [DBG] osdmap e796: 8 total, 8 up, 4 in 2024-02-13T00:54:07.628 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:06.594760+0000 mon.a (mon.0) 3734 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:54:07.628 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:07.013673+0000 mon.a (mon.0) 3735 : audit [INF] from='client.? 172.21.15.162:0/4182503271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:54:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:06.549632+0000 mgr.y (mgr.14544) 658 : cluster [DBG] pgmap v1211: 172 pgs: 172 active+clean; 3.8 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-13T00:54:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:06.593135+0000 mon.a (mon.0) 3733 : cluster [DBG] osdmap e796: 8 total, 8 up, 4 in 2024-02-13T00:54:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:06.594760+0000 mon.a (mon.0) 3734 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:54:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:07.013673+0000 mon.a (mon.0) 3735 : audit [INF] from='client.? 172.21.15.162:0/4182503271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:54:08.239 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:54:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:08.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:54:08.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0 2024-02-13T00:54:08.370 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:08.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:54:08.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:08.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0' 2024-02-13T00:54:08.547 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4 2024-02-13T00:54:08.579 INFO:teuthology.orchestra.run.smithi181.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:08.599 INFO:teuthology.orchestra.run.smithi181.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:54:08.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:08.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4' 2024-02-13T00:54:08.616 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9010 ms) 2024-02-13T00:54:08.616 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-02-13T00:54:08.616 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785563,0 2024-02-13T00:54:08.616 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: first is 1707785563 2024-02-13T00:54:08.616 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785563,0 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785563,0 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785563,1707785565,1707785566,0 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785563,1707785565,1707785566,0 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785563,1707785565,1707785566,0 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785563,1707785565,1707785566,1707785568,1707785569,0 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785563,1707785565,1707785566,1707785568,1707785569,0 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785563,1707785565,1707785566,1707785568,1707785569,0 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785563,1707785565,1707785566,1707785568,1707785569,1707785571,1707785572,0 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785563,1707785565,1707785566,1707785568,1707785569,1707785571,1707785572,0 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785563,1707785565,1707785566,1707785568,1707785569,1707785571,1707785572,0 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1707785566,1707785568,1707785569,1707785571,1707785572,1707785574,1707785575,0 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: first now 1707785566, trimmed 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21027 ms) 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-02-13T00:54:08.617 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: foo0 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13994 ms) 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17993 ms) 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22095 ms) 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4904 ms) 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3004 ms) 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3016 ms) 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3984 ms) 2024-02-13T00:54:08.618 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (232050 ms total) 2024-02-13T00:54:08.619 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 2024-02-13T00:54:08.619 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-02-13T00:54:08.619 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [==========] 72 tests from 4 test suites ran. (793901 ms total) 2024-02-13T00:54:08.619 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ PASSED ] 72 tests. 2024-02-13T00:54:08.619 INFO:tasks.workunit.client.0.smithi162.stderr:+ exit 0 2024-02-13T00:54:08.619 INFO:tasks.workunit.client.0.smithi162.stderr:+ cleanup 2024-02-13T00:54:08.619 INFO:tasks.workunit.client.0.smithi162.stderr:+ pkill -P 83967 2024-02-13T00:54:08.659 INFO:tasks.workunit.client.0.smithi162.stderr:+ true 2024-02-13T00:54:08.659 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-13T00:54:08.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-13T00:54:08.688 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-02-13T00:54:08.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-13T00:54:08.703 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3 2024-02-13T00:54:08.755 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:08.828 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:54:08.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:08.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3' 2024-02-13T00:54:08.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:07.596044+0000 mon.a (mon.0) 3736 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:54:08.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:07.596109+0000 mon.a (mon.0) 3737 : audit [INF] from='client.? 172.21.15.162:0/4182503271' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:54:08.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:07.596157+0000 mon.a (mon.0) 3738 : cluster [DBG] osdmap e797: 8 total, 8 up, 4 in 2024-02-13T00:54:08.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:07.596704+0000 mon.a (mon.0) 3739 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:54:08.873 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:07.596044+0000 mon.a (mon.0) 3736 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:54:08.873 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:07.596109+0000 mon.a (mon.0) 3737 : audit [INF] from='client.? 172.21.15.162:0/4182503271' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:54:08.873 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:07.596157+0000 mon.a (mon.0) 3738 : cluster [DBG] osdmap e797: 8 total, 8 up, 4 in 2024-02-13T00:54:08.873 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:07.596704+0000 mon.a (mon.0) 3739 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:54:08.933 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1 2024-02-13T00:54:08.960 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-13T00:54:09.028 DEBUG:teuthology.parallel:result is None 2024-02-13T00:54:09.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-02-13T00:54:09.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:54:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:54:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:54:09.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:07.596044+0000 mon.a (mon.0) 3736 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:54:09.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:07.596109+0000 mon.a (mon.0) 3737 : audit [INF] from='client.? 172.21.15.162:0/4182503271' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:54:09.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:07.596157+0000 mon.a (mon.0) 3738 : cluster [DBG] osdmap e797: 8 total, 8 up, 4 in 2024-02-13T00:54:09.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:07.596704+0000 mon.a (mon.0) 3739 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-101"}]: dispatch 2024-02-13T00:54:09.099 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-02-13T00:54:09.099 DEBUG:teuthology.orchestra.run.smithi162:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-02-13T00:54:09.204 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-02-13T00:54:09.205 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-02-13T00:54:09.215 INFO:tasks.thrashosds:joining thrashosds 2024-02-13T00:54:09.410 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:54:09.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-13T00:54:09.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1' 2024-02-13T00:54:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:08.550186+0000 mgr.y (mgr.14544) 659 : cluster [DBG] pgmap v1214: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:54:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:08.594165+0000 mon.a (mon.0) 3740 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:54:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:08.609278+0000 mon.a (mon.0) 3741 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:54:09.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:08.609364+0000 mon.a (mon.0) 3742 : cluster [DBG] osdmap e798: 8 total, 8 up, 4 in 2024-02-13T00:54:09.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:08.550186+0000 mgr.y (mgr.14544) 659 : cluster [DBG] pgmap v1214: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:54:09.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:08.594165+0000 mon.a (mon.0) 3740 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:54:09.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:08.609278+0000 mon.a (mon.0) 3741 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:54:09.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:08.609364+0000 mon.a (mon.0) 3742 : cluster [DBG] osdmap e798: 8 total, 8 up, 4 in 2024-02-13T00:54:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:08.550186+0000 mgr.y (mgr.14544) 659 : cluster [DBG] pgmap v1214: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:54:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:08.594165+0000 mon.a (mon.0) 3740 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T00:54:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:08.609278+0000 mon.a (mon.0) 3741 : audit [INF] from='client.? 172.21.15.162:0/2331396316' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-84057-101"}]': finished 2024-02-13T00:54:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:08.609364+0000 mon.a (mon.0) 3742 : cluster [DBG] osdmap e798: 8 total, 8 up, 4 in 2024-02-13T00:54:10.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 6 1 2024-02-13T00:54:10.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 -- 172.21.15.162:0/3201472682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18180fa440 msgr2=0x7f18180f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:10.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 --2- 172.21.15.162:0/3201472682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18180fa440 0x7f18180f7e20 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 rx=0x7f1808004870 tx=0x7f1808009cb0).stop 2024-02-13T00:54:10.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 -- 172.21.15.162:0/3201472682 shutdown_connections 2024-02-13T00:54:10.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 --2- 172.21.15.162:0/3201472682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18180f8500 0x7f1818104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:10.702 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 --2- 172.21.15.162:0/3201472682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18180f9b70 0x7f18180f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 --2- 172.21.15.162:0/3201472682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18180fa440 0x7f18180f7e20 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 -- 172.21.15.162:0/3201472682 >> 172.21.15.162:0/3201472682 conn(0x7f18180f38c0 msgr2=0x7f18180f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 -- 172.21.15.162:0/3201472682 shutdown_connections 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 -- 172.21.15.162:0/3201472682 wait complete. 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 librados: starting msgr at 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 librados: starting objecter 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 Processor -- start 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 -- start start 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 librados: setting wanted keys 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.695+0000 7f181f69e700 1 librados: calling monclient init 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181f69e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18180f8500 0x7f181810e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181f69e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18180f9b70 0x7f181810e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181f69e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18180fa440 0x7f1818108430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:10.703 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181f69e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1818108920 con 0x7f18180f9b70 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181f69e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1818108a90 con 0x7f18180fa440 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181f69e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1818108c50 con 0x7f18180f8500 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181d43a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18180f8500 0x7f181810e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181cc39700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18180f9b70 0x7f181810e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181d43a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18180f8500 0x7f181810e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:45416/0 (socket says 172.21.15.162:45416) 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181d43a700 1 -- 172.21.15.162:0/2296838154 learned_addr learned my addr 172.21.15.162:0/2296838154 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181dc3b700 1 --2- 172.21.15.162:0/2296838154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18180fa440 0x7f1818108430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181cc39700 1 -- 172.21.15.162:0/2296838154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18180f8500 msgr2=0x7f181810e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181cc39700 1 --2- 172.21.15.162:0/2296838154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18180f8500 0x7f181810e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181cc39700 1 -- 172.21.15.162:0/2296838154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18180fa440 msgr2=0x7f1818108430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181cc39700 1 --2- 172.21.15.162:0/2296838154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18180fa440 0x7f1818108430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181cc39700 1 -- 172.21.15.162:0/2296838154 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18100097c0 con 0x7f18180f9b70 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181d43a700 1 --2- 172.21.15.162:0/2296838154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18180f8500 0x7f181810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:54:10.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181dc3b700 1 --2- 172.21.15.162:0/2296838154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18180fa440 0x7f1818108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:54:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181cc39700 1 --2- 172.21.15.162:0/2296838154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18180f9b70 0x7f181810e730 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7f1808004870 tx=0x7f180800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f180e7fc700 10 client.?.objecter ms_handle_connect 0x7f18180f9b70 2024-02-13T00:54:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f180e7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f180e7fc700 1 -- 172.21.15.162:0/2296838154 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180800e070 con 0x7f18180f9b70 2024-02-13T00:54:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f180e7fc700 1 -- 172.21.15.162:0/2296838154 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1808013da0 con 0x7f18180f9b70 2024-02-13T00:54:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181f69e700 1 -- 172.21.15.162:0/2296838154 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1808005040 con 0x7f18180f9b70 2024-02-13T00:54:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f180e7fc700 1 -- 172.21.15.162:0/2296838154 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1808019070 con 0x7f18180f9b70 2024-02-13T00:54:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181f69e700 10 client.28170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181f69e700 1 -- 172.21.15.162:0/2296838154 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1818109140 con 0x7f18180f9b70 2024-02-13T00:54:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.696+0000 7f181f69e700 1 librados: init done 2024-02-13T00:54:10.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.697+0000 7f1803fff700 1 -- 172.21.15.162:0/2296838154 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f8004f20 con 0x7f18180f9b70 2024-02-13T00:54:10.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.698+0000 7f180e7fc700 1 -- 172.21.15.162:0/2296838154 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f180801e420 con 0x7f18180f9b70 2024-02-13T00:54:10.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.698+0000 7f180e7fc700 1 --2- 172.21.15.162:0/2296838154 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1804062c90 0x7f1804065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:10.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.698+0000 7f180e7fc700 1 -- 172.21.15.162:0/2296838154 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(799..799 src has 449..799) v4 ==== 7422+0+0 (secure 0 0 0) 0x7f180807d900 con 0x7f18180f9b70 2024-02-13T00:54:10.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.698+0000 7f180e7fc700 10 client.28170.objecter ms_dispatch 0x7f1818001550 osd_map(799..799 src has 449..799) v4 2024-02-13T00:54:10.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.698+0000 7f180e7fc700 3 client.28170.objecter handle_osd_map got epochs [799,799] > 0 2024-02-13T00:54:10.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.698+0000 7f180e7fc700 3 client.28170.objecter handle_osd_map decoding full epoch 799 2024-02-13T00:54:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.698+0000 7f181d43a700 1 --2- 172.21.15.162:0/2296838154 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1804062c90 0x7f1804065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.698+0000 7f180e7fc700 20 client.28170.objecter dump_active .. 0 homeless 2024-02-13T00:54:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.699+0000 7f180e7fc700 10 client.28170.objecter ms_handle_connect 0x7f1804062c90 2024-02-13T00:54:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.699+0000 7f181d43a700 1 --2- 172.21.15.162:0/2296838154 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1804062c90 0x7f1804065150 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 rx=0x7f1814009fd0 tx=0x7f181400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.703+0000 7f180e7fc700 1 -- 172.21.15.162:0/2296838154 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1808051050 con 0x7f18180f9b70 2024-02-13T00:54:10.805 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-13T00:54:10.806 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-13T00:54:10.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:10.880+0000 7f1803fff700 1 -- 172.21.15.162:0/2296838154 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f17f80051d0 con 0x7f18180f9b70 2024-02-13T00:54:11.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-13T00:54:11.574 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:11.574 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:11.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.610+0000 7f180e7fc700 1 -- 172.21.15.162:0/2296838154 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v800) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1808050c40 con 0x7f18180f9b70 2024-02-13T00:54:11.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.616+0000 7f1803fff700 10 librados: watch_flush enter 2024-02-13T00:54:11.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.616+0000 7f1803fff700 10 librados: watch_flush exit 2024-02-13T00:54:11.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.616+0000 7f1803fff700 20 client.28170.objecter shutdown clearing up homeless session... 2024-02-13T00:54:11.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.616+0000 7f1803fff700 10 client.28170.objecter successfully canceled tick 2024-02-13T00:54:11.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.616+0000 7f1803fff700 1 -- 172.21.15.162:0/2296838154 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1804062c90 msgr2=0x7f1804065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:11.621 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.616+0000 7f1803fff700 1 --2- 172.21.15.162:0/2296838154 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1804062c90 0x7f1804065150 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 rx=0x7f1814009fd0 tx=0x7f181400b040).stop 2024-02-13T00:54:11.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.616+0000 7f1803fff700 1 -- 172.21.15.162:0/2296838154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18180f9b70 msgr2=0x7f181810e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:11.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.616+0000 7f1803fff700 1 --2- 172.21.15.162:0/2296838154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18180f9b70 0x7f181810e730 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7f1808004870 tx=0x7f180800b040).stop 2024-02-13T00:54:11.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.617+0000 7f1803fff700 1 -- 172.21.15.162:0/2296838154 shutdown_connections 2024-02-13T00:54:11.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.617+0000 7f1803fff700 1 --2- 172.21.15.162:0/2296838154 >> v2:172.21.15.162:6800/2866534053 conn(0x7f1804062c90 0x7f1804065150 unknown :-1 s=CLOSED pgs=3050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:11.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.617+0000 7f1803fff700 1 --2- 172.21.15.162:0/2296838154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18180fa440 0x7f1818108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:11.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.617+0000 7f1803fff700 1 --2- 172.21.15.162:0/2296838154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f18180f8500 0x7f181810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:11.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.617+0000 7f1803fff700 1 --2- 172.21.15.162:0/2296838154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18180f9b70 0x7f181810e730 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:11.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.617+0000 7f1803fff700 1 -- 172.21.15.162:0/2296838154 >> 172.21.15.162:0/2296838154 conn(0x7f18180f38c0 msgr2=0x7f18180f4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:11.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.617+0000 7f1803fff700 1 -- 172.21.15.162:0/2296838154 shutdown_connections 2024-02-13T00:54:11.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.617+0000 7f1803fff700 1 -- 172.21.15.162:0/2296838154 wait complete. 2024-02-13T00:54:11.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:11.617+0000 7f1803fff700 1 librados: shutdown 2024-02-13T00:54:11.623 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is set 2024-02-13T00:54:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:10.550957+0000 mgr.y (mgr.14544) 660 : cluster [DBG] pgmap v1216: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:54:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:10.584874+0000 mon.a (mon.0) 3743 : cluster [DBG] osdmap e799: 8 total, 8 up, 4 in 2024-02-13T00:54:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:10.882729+0000 mon.a (mon.0) 3744 : audit [INF] from='client.? 172.21.15.162:0/2296838154' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:54:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:10.550957+0000 mgr.y (mgr.14544) 660 : cluster [DBG] pgmap v1216: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:54:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:10.584874+0000 mon.a (mon.0) 3743 : cluster [DBG] osdmap e799: 8 total, 8 up, 4 in 2024-02-13T00:54:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:10.882729+0000 mon.a (mon.0) 3744 : audit [INF] from='client.? 172.21.15.162:0/2296838154' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:54:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:10.550957+0000 mgr.y (mgr.14544) 660 : cluster [DBG] pgmap v1216: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail 2024-02-13T00:54:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:10.584874+0000 mon.a (mon.0) 3743 : cluster [DBG] osdmap e799: 8 total, 8 up, 4 in 2024-02-13T00:54:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:10.882729+0000 mon.a (mon.0) 3744 : audit [INF] from='client.? 172.21.15.162:0/2296838154' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:54:12.913 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:11.606125+0000 mon.a (mon.0) 3745 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:54:12.913 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:11.611969+0000 mon.a (mon.0) 3746 : audit [INF] from='client.? 172.21.15.162:0/2296838154' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:54:12.914 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:11.612031+0000 mon.a (mon.0) 3747 : cluster [DBG] osdmap e800: 8 total, 8 up, 4 in 2024-02-13T00:54:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:11.606125+0000 mon.a (mon.0) 3745 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:54:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:11.611969+0000 mon.a (mon.0) 3746 : audit [INF] from='client.? 172.21.15.162:0/2296838154' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:54:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:11.612031+0000 mon.a (mon.0) 3747 : cluster [DBG] osdmap e800: 8 total, 8 up, 4 in 2024-02-13T00:54:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:11.606125+0000 mon.a (mon.0) 3745 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:54:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:11.611969+0000 mon.a (mon.0) 3746 : audit [INF] from='client.? 172.21.15.162:0/2296838154' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-13T00:54:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:11.612031+0000 mon.a (mon.0) 3747 : cluster [DBG] osdmap e800: 8 total, 8 up, 4 in 2024-02-13T00:54:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:12.551329+0000 mgr.y (mgr.14544) 661 : cluster [DBG] pgmap v1219: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:54:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:12.551329+0000 mgr.y (mgr.14544) 661 : cluster [DBG] pgmap v1219: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:54:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:12.551329+0000 mgr.y (mgr.14544) 661 : cluster [DBG] pgmap v1219: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:54:14.714 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.705+0000 7f4ee624a700 1 -- 172.21.15.162:0/663878194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ee00fd490 msgr2=0x7f4ee00f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:14.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.705+0000 7f4ee624a700 1 --2- 172.21.15.162:0/663878194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ee00fd490 0x7f4ee00f6cc0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7f4ed40047b0 tx=0x7f4ed4009cb0).stop 2024-02-13T00:54:14.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.705+0000 7f4ee624a700 1 -- 172.21.15.162:0/663878194 shutdown_connections 2024-02-13T00:54:14.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.705+0000 7f4ee624a700 1 --2- 172.21.15.162:0/663878194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ee00f73a0 0x7f4ee01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:14.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.705+0000 7f4ee624a700 1 --2- 172.21.15.162:0/663878194 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ee00fcb30 0x7f4ee00fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:14.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.705+0000 7f4ee624a700 1 --2- 172.21.15.162:0/663878194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ee00fd490 0x7f4ee00f6cc0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:14.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.705+0000 7f4ee624a700 1 -- 172.21.15.162:0/663878194 >> 172.21.15.162:0/663878194 conn(0x7f4ee006eb90 msgr2=0x7f4ee00f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:14.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.705+0000 7f4ee624a700 1 -- 172.21.15.162:0/663878194 shutdown_connections 2024-02-13T00:54:14.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.705+0000 7f4ee624a700 1 -- 172.21.15.162:0/663878194 wait complete. 2024-02-13T00:54:14.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.706+0000 7f4ee624a700 1 librados: starting msgr at 2024-02-13T00:54:14.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.706+0000 7f4ee624a700 1 librados: starting objecter 2024-02-13T00:54:14.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.706+0000 7f4ee624a700 1 Processor -- start 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.706+0000 7f4ee624a700 1 -- start start 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.706+0000 7f4ee624a700 1 librados: setting wanted keys 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.706+0000 7f4ee624a700 1 librados: calling monclient init 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.706+0000 7f4ee624a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ee00f73a0 0x7f4ee010bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.706+0000 7f4ee624a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ee00fcb30 0x7f4ee010c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.706+0000 7f4ee624a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ee00fd490 0x7f4ee0106170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.706+0000 7f4ee624a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee0106660 con 0x7f4ee00fcb30 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.706+0000 7f4ee624a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee01067d0 con 0x7f4ee00f73a0 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.706+0000 7f4ee624a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee0106990 con 0x7f4ee00fd490 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edeffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ee00fcb30 0x7f4ee010c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edffff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ee00fd490 0x7f4ee0106170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edeffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ee00fcb30 0x7f4ee010c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:52476/0 (socket says 172.21.15.162:52476) 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edffff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ee00fd490 0x7f4ee0106170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:60008/0 (socket says 172.21.15.162:60008) 2024-02-13T00:54:14.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edeffd700 1 -- 172.21.15.162:0/208632912 learned_addr learned my addr 172.21.15.162:0/208632912 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edf7fe700 1 --2- 172.21.15.162:0/208632912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ee00f73a0 0x7f4ee010bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edeffd700 1 -- 172.21.15.162:0/208632912 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ee00fd490 msgr2=0x7f4ee0106170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edeffd700 1 --2- 172.21.15.162:0/208632912 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ee00fd490 0x7f4ee0106170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edeffd700 1 -- 172.21.15.162:0/208632912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ee00f73a0 msgr2=0x7f4ee010bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edeffd700 1 --2- 172.21.15.162:0/208632912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ee00f73a0 0x7f4ee010bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edeffd700 1 -- 172.21.15.162:0/208632912 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed0005040 con 0x7f4ee00fcb30 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edffff700 1 --2- 172.21.15.162:0/208632912 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ee00fd490 0x7f4ee0106170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edeffd700 1 --2- 172.21.15.162:0/208632912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ee00fcb30 0x7f4ee010c4c0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7f4ed40114c0 tx=0x7f4ed4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edcff9700 10 client.?.objecter ms_handle_connect 0x7f4ee00fcb30 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edcff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edcff9700 1 -- 172.21.15.162:0/208632912 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed40118f0 con 0x7f4ee00fcb30 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4ee624a700 1 -- 172.21.15.162:0/208632912 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed4005040 con 0x7f4ee00fcb30 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edcff9700 1 -- 172.21.15.162:0/208632912 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ed4011ca0 con 0x7f4ee00fcb30 2024-02-13T00:54:14.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4edcff9700 1 -- 172.21.15.162:0/208632912 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed400f6a0 con 0x7f4ee00fcb30 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4ee624a700 10 client.28176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4ee624a700 1 -- 172.21.15.162:0/208632912 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee0106eb0 con 0x7f4ee00fcb30 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.707+0000 7f4ee624a700 1 librados: init done 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.708+0000 7f4ee624a700 1 -- 172.21.15.162:0/208632912 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee00475e0 con 0x7f4ee00fcb30 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.709+0000 7f4edcff9700 1 -- 172.21.15.162:0/208632912 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f4ed400fa90 con 0x7f4ee00fcb30 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.709+0000 7f4edcff9700 1 --2- 172.21.15.162:0/208632912 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4ec8062c90 0x7f4ec8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.709+0000 7f4edcff9700 1 -- 172.21.15.162:0/208632912 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(800..800 src has 449..800) v4 ==== 7422+0+0 (secure 0 0 0) 0x7f4ed4080a80 con 0x7f4ee00fcb30 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.709+0000 7f4edcff9700 10 client.28176.objecter ms_dispatch 0x7f4ee0001550 osd_map(800..800 src has 449..800) v4 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.709+0000 7f4edcff9700 3 client.28176.objecter handle_osd_map got epochs [800,800] > 0 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.709+0000 7f4edcff9700 3 client.28176.objecter handle_osd_map decoding full epoch 800 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.709+0000 7f4edf7fe700 1 --2- 172.21.15.162:0/208632912 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4ec8062c90 0x7f4ec8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.709+0000 7f4edcff9700 20 client.28176.objecter dump_active .. 0 homeless 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.710+0000 7f4edcff9700 10 client.28176.objecter ms_handle_connect 0x7f4ec8062c90 2024-02-13T00:54:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.709+0000 7f4edf7fe700 1 --2- 172.21.15.162:0/208632912 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4ec8062c90 0x7f4ec8065150 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 rx=0x7f4ed0004870 tx=0x7f4ed000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:14.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.713+0000 7f4edcff9700 1 -- 172.21.15.162:0/208632912 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4ed4054210 con 0x7f4ee00fcb30 2024-02-13T00:54:14.732 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:14.777 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-13T00:54:14.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:14.908+0000 7f4ee624a700 1 -- 172.21.15.162:0/208632912 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f4ee0107650 con 0x7f4ee00fcb30 2024-02-13T00:54:15.590 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.585+0000 7f4edcff9700 1 -- 172.21.15.162:0/208632912 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v801) v1 ==== 113+0+0 (secure 0 0 0) 0x7f4ed4053e00 con 0x7f4ee00fcb30 2024-02-13T00:54:15.592 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.587+0000 7f4ee624a700 10 librados: watch_flush enter 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.587+0000 7f4ee624a700 10 librados: watch_flush exit 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.587+0000 7f4ee624a700 20 client.28176.objecter shutdown clearing up homeless session... 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.587+0000 7f4ee624a700 10 client.28176.objecter successfully canceled tick 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.587+0000 7f4ee624a700 1 -- 172.21.15.162:0/208632912 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4ec8062c90 msgr2=0x7f4ec8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.587+0000 7f4ee624a700 1 --2- 172.21.15.162:0/208632912 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4ec8062c90 0x7f4ec8065150 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 rx=0x7f4ed0004870 tx=0x7f4ed000b040).stop 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.587+0000 7f4ee624a700 1 -- 172.21.15.162:0/208632912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ee00fcb30 msgr2=0x7f4ee010c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.588+0000 7f4ee624a700 1 --2- 172.21.15.162:0/208632912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ee00fcb30 0x7f4ee010c4c0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7f4ed40114c0 tx=0x7f4ed4009cb0).stop 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.588+0000 7f4ee624a700 1 -- 172.21.15.162:0/208632912 shutdown_connections 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.588+0000 7f4ee624a700 1 --2- 172.21.15.162:0/208632912 >> v2:172.21.15.162:6800/2866534053 conn(0x7f4ec8062c90 0x7f4ec8065150 unknown :-1 s=CLOSED pgs=3051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.588+0000 7f4ee624a700 1 --2- 172.21.15.162:0/208632912 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ee00fd490 0x7f4ee0106170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.588+0000 7f4ee624a700 1 --2- 172.21.15.162:0/208632912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ee00f73a0 0x7f4ee010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.588+0000 7f4ee624a700 1 --2- 172.21.15.162:0/208632912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ee00fcb30 0x7f4ee010c4c0 secure :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 rx=0x7f4ed40114c0 tx=0x7f4ed4009cb0).stop 2024-02-13T00:54:15.593 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.588+0000 7f4ee624a700 1 -- 172.21.15.162:0/208632912 >> 172.21.15.162:0/208632912 conn(0x7f4ee006eb90 msgr2=0x7f4ee00fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:15.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.588+0000 7f4ee624a700 1 -- 172.21.15.162:0/208632912 shutdown_connections 2024-02-13T00:54:15.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.588+0000 7f4ee624a700 1 -- 172.21.15.162:0/208632912 wait complete. 2024-02-13T00:54:15.594 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:15.588+0000 7f4ee624a700 1 librados: shutdown 2024-02-13T00:54:15.594 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.6 to 1 (10000) 2024-02-13T00:54:15.998 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:14.551867+0000 mgr.y (mgr.14544) 662 : cluster [DBG] pgmap v1220: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:15.998 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:14.910472+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? 172.21.15.162:0/208632912' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:54:15.998 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:15.586274+0000 mon.a (mon.0) 3749 : audit [INF] from='client.? 172.21.15.162:0/208632912' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:54:15.998 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:15.586339+0000 mon.a (mon.0) 3750 : cluster [DBG] osdmap e801: 8 total, 8 up, 4 in 2024-02-13T00:54:16.056 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:14.551867+0000 mgr.y (mgr.14544) 662 : cluster [DBG] pgmap v1220: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:16.056 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:14.910472+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? 172.21.15.162:0/208632912' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:54:16.056 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:15.586274+0000 mon.a (mon.0) 3749 : audit [INF] from='client.? 172.21.15.162:0/208632912' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:54:16.056 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:15.586339+0000 mon.a (mon.0) 3750 : cluster [DBG] osdmap e801: 8 total, 8 up, 4 in 2024-02-13T00:54:16.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:14.551867+0000 mgr.y (mgr.14544) 662 : cluster [DBG] pgmap v1220: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:16.057 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:14.910472+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? 172.21.15.162:0/208632912' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:54:16.057 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:15.586274+0000 mon.a (mon.0) 3749 : audit [INF] from='client.? 172.21.15.162:0/208632912' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:54:16.057 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:15.586339+0000 mon.a (mon.0) 3750 : cluster [DBG] osdmap e801: 8 total, 8 up, 4 in 2024-02-13T00:54:16.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:54:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:54:16.330 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:54:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:54:16] "GET /metrics HTTP/1.1" 200 190323 "" "Prometheus/2.33.4" 2024-02-13T00:54:16.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:15.994979+0000 mgr.y (mgr.14544) 663 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:16.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:15.994979+0000 mgr.y (mgr.14544) 663 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:15.994979+0000 mgr.y (mgr.14544) 663 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:17.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.273+0000 7fc186915700 1 -- 172.21.15.162:0/144428524 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1800f9b70 msgr2=0x7fc1800f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.273+0000 7fc186915700 1 --2- 172.21.15.162:0/144428524 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1800f9b70 0x7fc1800f9f50 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 rx=0x7fc1700047b0 tx=0x7fc170009cb0).stop 2024-02-13T00:54:17.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.273+0000 7fc186915700 1 -- 172.21.15.162:0/144428524 shutdown_connections 2024-02-13T00:54:17.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.273+0000 7fc186915700 1 --2- 172.21.15.162:0/144428524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1800f8500 0x7fc180104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.273+0000 7fc186915700 1 --2- 172.21.15.162:0/144428524 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1800f9b70 0x7fc1800f9f50 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.273+0000 7fc186915700 1 --2- 172.21.15.162:0/144428524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1800fa440 0x7fc1800f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.274+0000 7fc186915700 1 -- 172.21.15.162:0/144428524 >> 172.21.15.162:0/144428524 conn(0x7fc1800f38c0 msgr2=0x7fc1800f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:17.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.274+0000 7fc186915700 1 -- 172.21.15.162:0/144428524 shutdown_connections 2024-02-13T00:54:17.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.274+0000 7fc186915700 1 -- 172.21.15.162:0/144428524 wait complete. 2024-02-13T00:54:17.384 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.274+0000 7fc186915700 1 librados: starting msgr at 2024-02-13T00:54:17.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.274+0000 7fc186915700 1 librados: starting objecter 2024-02-13T00:54:17.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.274+0000 7fc186915700 1 Processor -- start 2024-02-13T00:54:17.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc186915700 1 -- start start 2024-02-13T00:54:17.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc186915700 1 librados: setting wanted keys 2024-02-13T00:54:17.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc186915700 1 librados: calling monclient init 2024-02-13T00:54:17.385 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc186915700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1800f8500 0x7fc18006b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc186915700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1800f9b70 0x7fc18006b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc186915700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1800fa440 0x7fc18006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc186915700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc180116720 con 0x7fc1800fa440 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc186915700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc180116890 con 0x7fc1800f9b70 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc186915700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc180116a50 con 0x7fc1800f8500 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17ffff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1800f8500 0x7fc18006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc184eb2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1800fa440 0x7fc18006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17ffff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1800f8500 0x7fc18006b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:60036/0 (socket says 172.21.15.162:60036) 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17ffff700 1 -- 172.21.15.162:0/1788206591 learned_addr learned my addr 172.21.15.162:0/1788206591 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17ffff700 1 -- 172.21.15.162:0/1788206591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1800f9b70 msgr2=0x7fc18006b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17f7fe700 1 --2- 172.21.15.162:0/1788206591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1800f9b70 0x7fc18006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17ffff700 1 --2- 172.21.15.162:0/1788206591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1800f9b70 0x7fc18006b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17ffff700 1 -- 172.21.15.162:0/1788206591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1800fa440 msgr2=0x7fc18006bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17ffff700 1 --2- 172.21.15.162:0/1788206591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1800fa440 0x7fc18006bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.386 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17ffff700 1 -- 172.21.15.162:0/1788206591 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc168005040 con 0x7fc1800f8500 2024-02-13T00:54:17.387 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc184eb2700 1 --2- 172.21.15.162:0/1788206591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1800fa440 0x7fc18006bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:54:17.387 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17ffff700 1 --2- 172.21.15.162:0/1788206591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1800f8500 0x7fc18006b170 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 rx=0x7fc1700044e0 tx=0x7fc170007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:17.387 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17d7fa700 10 client.?.objecter ms_handle_connect 0x7fc1800f8500 2024-02-13T00:54:17.387 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17d7fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:17.387 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.275+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc170018070 con 0x7fc1800f8500 2024-02-13T00:54:17.387 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.276+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc170005040 con 0x7fc1800f8500 2024-02-13T00:54:17.387 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.276+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1700093b0 con 0x7fc1800f8500 2024-02-13T00:54:17.387 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.276+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17001de40 con 0x7fc1800f8500 2024-02-13T00:54:17.387 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.276+0000 7fc186915700 10 client.38276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:17.387 INFO:teuthology.orchestra.run.smithi162.stdout:osd.0: {} 2024-02-13T00:54:17.388 INFO:teuthology.orchestra.run.smithi162.stdout:osd.1: {} 2024-02-13T00:54:17.388 INFO:teuthology.orchestra.run.smithi162.stdout:osd.2: {} 2024-02-13T00:54:17.388 INFO:teuthology.orchestra.run.smithi162.stdout:osd.3: {} 2024-02-13T00:54:17.388 INFO:teuthology.orchestra.run.smithi162.stdout:osd.4: {} 2024-02-13T00:54:17.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.276+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc180116f70 con 0x7fc1800f8500 2024-02-13T00:54:17.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.276+0000 7fc186915700 1 librados: init done 2024-02-13T00:54:17.388 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.276+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc1800475e0 con 0x7fc1800f8500 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.277+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc170013ad0 con 0x7fc1800f8500 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.277+0000 7fc17d7fa700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc16c062c90 0x7fc16c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.277+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(801..801 src has 449..801) v4 ==== 7422+0+0 (secure 0 0 0) 0x7fc17007e3c0 con 0x7fc1800f8500 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.277+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 osd_map(801..801 src has 449..801) v4 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.277+0000 7fc17d7fa700 3 client.38276.objecter handle_osd_map got epochs [801,801] > 0 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.277+0000 7fc17d7fa700 3 client.38276.objecter handle_osd_map decoding full epoch 801 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.277+0000 7fc17f7fe700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc16c062c90 0x7fc16c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.277+0000 7fc17d7fa700 20 client.38276.objecter dump_active .. 0 homeless 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.277+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v801) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc170085070 con 0x7fc1800f8500 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc17d7fa700 10 client.38276.objecter ms_handle_connect 0x7fc16c062c90 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc17f7fe700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc16c062c90 0x7fc16c065150 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 rx=0x7fc18006c910 tx=0x7fc168009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc186915700 10 client.38276.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 1 2024-02-13T00:54:17.389 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc180059990 0x7fc180059df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc1801072c0 osd=0 3 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc1801072c0 osd=0 3 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc1801072c0 osd=0 3 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 1 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc1801072c0 osd=0 3 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 0 1 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc1801072c0 osd=0 4 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc186915700 10 client.38276.objecter _send_command 1 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc18006c650 con 0x7fc180059990 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc184eb2700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc180059990 0x7fc180059df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc17d7fa700 10 client.38276.objecter ms_handle_connect 0x7fc180059990 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.278+0000 7fc184eb2700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc180059990 0x7fc180059df0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.279+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc18006c650 con 0x7fc180059990 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.279+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.279+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 1 = system:0 2024-02-13T00:54:17.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.279+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc1801072c0 osd=0 4 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.279+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 0 1 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.296+0000 7fc186915700 10 client.38276.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.296+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 2 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.296+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc1801072c0 osd=0 3 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.296+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc1801072c0 osd=0 3 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.296+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc1801072c0 osd=0 3 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.296+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 2 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.296+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc1801072c0 osd=0 3 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.296+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 0 2 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.296+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc1801072c0 osd=0 4 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.296+0000 7fc186915700 10 client.38276.objecter _send_command 2 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.296+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc180028590 con 0x7fc180059990 2024-02-13T00:54:17.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc180028590 con 0x7fc180059990 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc1801072c0 osd=0 4 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 0 2 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc186915700 10 client.38276.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 3 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6805/150599780 conn(0x7fc180059570 0x7fc18011b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc180059400 osd=1 3 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc180059400 osd=1 3 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc180059400 osd=1 3 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 3 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc180059400 osd=1 3 2024-02-13T00:54:17.392 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 1 3 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc180059400 osd=1 4 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc186915700 10 client.38276.objecter _send_command 3 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.298+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.162:6805/150599780 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc1800ff740 con 0x7fc180059570 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.299+0000 7fc17ffff700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6805/150599780 conn(0x7fc180059570 0x7fc18011b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.299+0000 7fc17d7fa700 10 client.38276.objecter ms_handle_connect 0x7fc180059570 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.299+0000 7fc17ffff700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6805/150599780 conn(0x7fc180059570 0x7fc18011b120 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.300+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc1800ff740 con 0x7fc180059570 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.300+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 3: 0 ) v1 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.300+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 3 = system:0 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.300+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc180059400 osd=1 4 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.300+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 1 3 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.315+0000 7fc186915700 10 client.38276.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:54:17.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.315+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 4 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.315+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc180059400 osd=1 3 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.315+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc180059400 osd=1 3 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.315+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc180059400 osd=1 3 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.315+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 4 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.315+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc180059400 osd=1 3 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.315+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 1 4 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.315+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc180059400 osd=1 4 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.315+0000 7fc186915700 10 client.38276.objecter _send_command 4 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.315+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.162:6805/150599780 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc18011b610 con 0x7fc180059570 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc18011b610 con 0x7fc180059570 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc180059400 osd=1 4 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 1 4 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc186915700 10 client.38276.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 5 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc1800fee10 0x7fc18011e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc1800feca0 osd=2 3 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc1800feca0 osd=2 3 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc1800feca0 osd=2 3 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 5 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc1800feca0 osd=2 3 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 2 5 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc1800feca0 osd=2 4 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc186915700 10 client.38276.objecter _send_command 5 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.162:6809/3689719376 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc1800280f0 con 0x7fc1800fee10 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.317+0000 7fc17f7fe700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc1800fee10 0x7fc18011e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:17.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.318+0000 7fc17d7fa700 10 client.38276.objecter ms_handle_connect 0x7fc1800fee10 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.318+0000 7fc17f7fe700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc1800fee10 0x7fc18011e970 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.318+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc1800280f0 con 0x7fc1800fee10 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.318+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 5: 0 ) v1 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.318+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 5 = system:0 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.318+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc1800feca0 osd=2 4 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.318+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 2 5 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.333+0000 7fc186915700 10 client.38276.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.333+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 6 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.333+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc1800feca0 osd=2 3 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.333+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc1800feca0 osd=2 3 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.333+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc1800feca0 osd=2 3 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.333+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 6 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.333+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc1800feca0 osd=2 3 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.333+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 2 6 2024-02-13T00:54:17.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.333+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc1800feca0 osd=2 4 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.333+0000 7fc186915700 10 client.38276.objecter _send_command 6 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.333+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.162:6809/3689719376 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc18011ee60 con 0x7fc1800fee10 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.334+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc18011ee60 con 0x7fc1800fee10 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.334+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.334+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.334+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc1800feca0 osd=2 4 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.334+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 2 6 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc186915700 10 client.38276.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 7 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6813/604971011 conn(0x7fc1800277c0 0x7fc1801221f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc180027650 osd=3 3 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc180027650 osd=3 3 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc180027650 osd=3 3 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 7 2024-02-13T00:54:17.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc180027650 osd=3 3 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 3 7 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc180027650 osd=3 4 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc186915700 10 client.38276.objecter _send_command 7 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.162:6813/604971011 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc18011c650 con 0x7fc1800277c0 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc184eb2700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6813/604971011 conn(0x7fc1800277c0 0x7fc1801221f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc17d7fa700 10 client.38276.objecter ms_handle_connect 0x7fc1800277c0 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.335+0000 7fc184eb2700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6813/604971011 conn(0x7fc1800277c0 0x7fc1801221f0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.336+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc18011c650 con 0x7fc1800277c0 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.336+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 7: 0 ) v1 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.336+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 7 = system:0 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.336+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc180027650 osd=3 4 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.336+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 3 7 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.350+0000 7fc186915700 10 client.38276.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.350+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 8 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.350+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc180027650 osd=3 3 2024-02-13T00:54:17.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.350+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc180027650 osd=3 3 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.350+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc180027650 osd=3 3 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.350+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 8 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.350+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc180027650 osd=3 3 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.350+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 3 8 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.350+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc180027650 osd=3 4 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.350+0000 7fc186915700 10 client.38276.objecter _send_command 8 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.350+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.162:6813/604971011 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc18011ba20 con 0x7fc1800277c0 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc18011ba20 con 0x7fc1800277c0 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc180027650 osd=3 4 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 3 8 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc186915700 10 client.38276.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 9 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc18011fce0 0x7fc180125870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:17.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18011ba20 osd=4 3 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18011ba20 osd=4 3 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18011ba20 osd=4 3 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 9 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc18011ba20 osd=4 3 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 4 9 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18011ba20 osd=4 4 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc186915700 10 client.38276.objecter _send_command 9 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.181:6800/3344263388 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc18011bce0 con 0x7fc18011fce0 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.352+0000 7fc17ffff700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc18011fce0 0x7fc180125870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.353+0000 7fc17d7fa700 10 client.38276.objecter ms_handle_connect 0x7fc18011fce0 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.353+0000 7fc17ffff700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc18011fce0 0x7fc180125870 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.353+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc18011bce0 con 0x7fc18011fce0 2024-02-13T00:54:17.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.353+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 9: 0 ) v1 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.353+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 9 = system:0 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.353+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc18011ba20 osd=4 4 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.353+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 4 9 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.369+0000 7fc186915700 10 client.38276.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.369+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 10 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.369+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18011ba20 osd=4 3 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.369+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18011ba20 osd=4 3 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.369+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18011ba20 osd=4 3 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.369+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 10 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.369+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc18011ba20 osd=4 3 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.369+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 4 10 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.369+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18011ba20 osd=4 4 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.369+0000 7fc186915700 10 client.38276.objecter _send_command 10 2024-02-13T00:54:17.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.369+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.181:6800/3344263388 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc18011bce0 con 0x7fc18011fce0 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.370+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc18011bce0 con 0x7fc18011fce0 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.370+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.370+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.370+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc18011ba20 osd=4 4 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.370+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 4 10 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc186915700 10 client.38276.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 11 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc180126270 0x7fc180128660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18011bce0 osd=5 3 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18011bce0 osd=5 3 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18011bce0 osd=5 3 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 11 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc18011bce0 osd=5 3 2024-02-13T00:54:17.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 5 11 2024-02-13T00:54:17.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18011bce0 osd=5 4 2024-02-13T00:54:17.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc186915700 10 client.38276.objecter _send_command 11 2024-02-13T00:54:17.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.181:6804/2277917372 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc18012b3f0 con 0x7fc180126270 2024-02-13T00:54:17.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc17f7fe700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc180126270 0x7fc180128660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:17.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc17f7fe700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc180126270 0x7fc180128660 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:17.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.371+0000 7fc17d7fa700 10 client.38276.objecter ms_handle_connect 0x7fc180126270 2024-02-13T00:54:17.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.372+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc18012b3f0 con 0x7fc180126270 2024-02-13T00:54:17.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.372+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 11: 0 ) v1 2024-02-13T00:54:17.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.372+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 11 = system:0 2024-02-13T00:54:17.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.372+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc18011bce0 osd=5 4 2024-02-13T00:54:17.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.372+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 5 11 2024-02-13T00:54:17.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.387+0000 7fc186915700 10 client.38276.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:54:17.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.387+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 12 2024-02-13T00:54:17.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.387+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18011bce0 osd=5 3 2024-02-13T00:54:17.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.387+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18011bce0 osd=5 3 2024-02-13T00:54:17.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.387+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18011bce0 osd=5 3 2024-02-13T00:54:17.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.387+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 12 2024-02-13T00:54:17.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.387+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc18011bce0 osd=5 3 2024-02-13T00:54:17.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.387+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 5 12 2024-02-13T00:54:17.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.387+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18011bce0 osd=5 4 2024-02-13T00:54:17.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.387+0000 7fc186915700 10 client.38276.objecter _send_command 12 2024-02-13T00:54:17.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.387+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.181:6804/2277917372 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc18012b3f0 con 0x7fc180126270 2024-02-13T00:54:17.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc18012b3f0 con 0x7fc180126270 2024-02-13T00:54:17.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:54:17.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc18011bce0 osd=5 4 2024-02-13T00:54:17.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 5 12 2024-02-13T00:54:17.405 INFO:teuthology.orchestra.run.smithi162.stdout:osd.5: {} 2024-02-13T00:54:17.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc186915700 10 client.38276.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:17.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 13 2024-02-13T00:54:17.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc18012a750 0x7fc18012ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:17.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18012b3f0 osd=6 3 2024-02-13T00:54:17.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18012b3f0 osd=6 3 2024-02-13T00:54:17.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18012b3f0 osd=6 3 2024-02-13T00:54:17.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 13 2024-02-13T00:54:17.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc18012b3f0 osd=6 3 2024-02-13T00:54:17.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 6 13 2024-02-13T00:54:17.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18012b3f0 osd=6 4 2024-02-13T00:54:17.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc186915700 10 client.38276.objecter _send_command 13 2024-02-13T00:54:17.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.389+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.181:6808/1506640341 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc18012b200 con 0x7fc18012a750 2024-02-13T00:54:17.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.390+0000 7fc184eb2700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc18012a750 0x7fc18012ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:17.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.390+0000 7fc17d7fa700 10 client.38276.objecter ms_handle_connect 0x7fc18012a750 2024-02-13T00:54:17.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.390+0000 7fc184eb2700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc18012a750 0x7fc18012ab90 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:17.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.391+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc17800f040 con 0x7fc18012a750 2024-02-13T00:54:17.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.391+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 13: 0 ) v1 2024-02-13T00:54:17.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.391+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 13 = system:0 2024-02-13T00:54:17.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.391+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc18012b3f0 osd=6 4 2024-02-13T00:54:17.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.391+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 6 13 2024-02-13T00:54:17.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.406+0000 7fc186915700 10 client.38276.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:54:17.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.406+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 14 2024-02-13T00:54:17.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.406+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18012b3f0 osd=6 3 2024-02-13T00:54:17.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.406+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18012b3f0 osd=6 3 2024-02-13T00:54:17.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.406+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18012b3f0 osd=6 3 2024-02-13T00:54:17.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.406+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 14 2024-02-13T00:54:17.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.406+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc18012b3f0 osd=6 3 2024-02-13T00:54:17.411 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.406+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 6 14 2024-02-13T00:54:17.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.406+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18012b3f0 osd=6 4 2024-02-13T00:54:17.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.406+0000 7fc186915700 10 client.38276.objecter _send_command 14 2024-02-13T00:54:17.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.406+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.181:6808/1506640341 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc18012d6a0 con 0x7fc18012a750 2024-02-13T00:54:17.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.408+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc18012d6a0 con 0x7fc18012a750 2024-02-13T00:54:17.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.408+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:54:17.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.408+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.408+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc18012b3f0 osd=6 4 2024-02-13T00:54:17.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.408+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 6 14 2024-02-13T00:54:17.414 INFO:teuthology.orchestra.run.smithi162.stdout:osd.6: {} 2024-02-13T00:54:17.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc186915700 10 client.38276.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:54:17.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 15 2024-02-13T00:54:17.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc18012dd50 0x7fc18012e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:17.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18012d6a0 osd=7 3 2024-02-13T00:54:17.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18012d6a0 osd=7 3 2024-02-13T00:54:17.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18012d6a0 osd=7 3 2024-02-13T00:54:17.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 15 2024-02-13T00:54:17.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc18012d6a0 osd=7 3 2024-02-13T00:54:17.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 7 15 2024-02-13T00:54:17.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18012d6a0 osd=7 4 2024-02-13T00:54:17.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc186915700 10 client.38276.objecter _send_command 15 2024-02-13T00:54:17.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.181:6812/1964192772 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc180131a60 con 0x7fc18012dd50 2024-02-13T00:54:17.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc17ffff700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc18012dd50 0x7fc18012e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:17.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc17ffff700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc18012dd50 0x7fc18012e1b0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:17.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.409+0000 7fc17d7fa700 10 client.38276.objecter ms_handle_connect 0x7fc18012dd50 2024-02-13T00:54:17.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.413+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc180131a60 con 0x7fc18012dd50 2024-02-13T00:54:17.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.413+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 15: 0 ) v1 2024-02-13T00:54:17.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.413+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 15 = system:0 2024-02-13T00:54:17.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.413+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc18012d6a0 osd=7 4 2024-02-13T00:54:17.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.413+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 7 15 2024-02-13T00:54:17.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.427+0000 7fc186915700 10 client.38276.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-13T00:54:17.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.427+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign -1 16 2024-02-13T00:54:17.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.428+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18012d6a0 osd=7 3 2024-02-13T00:54:17.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.428+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18012d6a0 osd=7 3 2024-02-13T00:54:17.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.428+0000 7fc186915700 20 client.38276.objecter _get_session s=0x7fc18012d6a0 osd=7 3 2024-02-13T00:54:17.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.428+0000 7fc186915700 15 client.38276.objecter _session_command_op_remove -1 16 2024-02-13T00:54:17.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.428+0000 7fc186915700 20 client.38276.objecter get_session s=0x7fc18012d6a0 osd=7 3 2024-02-13T00:54:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.428+0000 7fc186915700 15 client.38276.objecter _session_command_op_assign 7 16 2024-02-13T00:54:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.428+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18012d6a0 osd=7 4 2024-02-13T00:54:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.428+0000 7fc186915700 10 client.38276.objecter _send_command 16 2024-02-13T00:54:17.433 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.428+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 --> v2:172.21.15.181:6812/1964192772 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc1801308e0 con 0x7fc18012dd50 2024-02-13T00:54:17.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.429+0000 7fc17d7fa700 1 -- 172.21.15.162:0/1788206591 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fc1801308e0 con 0x7fc18012dd50 2024-02-13T00:54:17.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.429+0000 7fc17d7fa700 10 client.38276.objecter ms_dispatch 0x7fc180001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-13T00:54:17.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.429+0000 7fc17d7fa700 10 client.38276.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.429+0000 7fc17d7fa700 20 client.38276.objecter put_session s=0x7fc18012d6a0 osd=7 4 2024-02-13T00:54:17.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.429+0000 7fc17d7fa700 15 client.38276.objecter _session_command_op_remove 7 16 2024-02-13T00:54:17.435 INFO:teuthology.orchestra.run.smithi162.stdout:osd.7: {} 2024-02-13T00:54:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 10 librados: watch_flush enter 2024-02-13T00:54:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 10 librados: watch_flush exit 2024-02-13T00:54:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 10 client.38276.objecter close_session for osd.0 2024-02-13T00:54:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc180059990 msgr2=0x7fc180059df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6801/2103743338 conn(0x7fc180059990 0x7fc180059df0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc1801072c0 osd=0 1 2024-02-13T00:54:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 10 client.38276.objecter close_session for osd.1 2024-02-13T00:54:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6805/150599780 conn(0x7fc180059570 msgr2=0x7fc18011b120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6805/150599780 conn(0x7fc180059570 0x7fc18011b120 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc180059400 osd=1 1 2024-02-13T00:54:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 10 client.38276.objecter close_session for osd.2 2024-02-13T00:54:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc1800fee10 msgr2=0x7fc18011e970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6809/3689719376 conn(0x7fc1800fee10 0x7fc18011e970 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc1800feca0 osd=2 1 2024-02-13T00:54:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 10 client.38276.objecter close_session for osd.3 2024-02-13T00:54:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6813/604971011 conn(0x7fc1800277c0 msgr2=0x7fc1801221f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6813/604971011 conn(0x7fc1800277c0 0x7fc1801221f0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc180027650 osd=3 1 2024-02-13T00:54:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 10 client.38276.objecter close_session for osd.4 2024-02-13T00:54:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc18011fce0 msgr2=0x7fc180125870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.437 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6800/3344263388 conn(0x7fc18011fce0 0x7fc180125870 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18011ba20 osd=4 1 2024-02-13T00:54:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 10 client.38276.objecter close_session for osd.5 2024-02-13T00:54:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc184eb2700 1 -- 172.21.15.162:0/1788206591 reap_dead start 2024-02-13T00:54:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc180126270 msgr2=0x7fc180128660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6804/2277917372 conn(0x7fc180126270 0x7fc180128660 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc184eb2700 1 -- 172.21.15.162:0/1788206591 reap_dead start 2024-02-13T00:54:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18011bce0 osd=5 1 2024-02-13T00:54:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 10 client.38276.objecter close_session for osd.6 2024-02-13T00:54:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc18012a750 msgr2=0x7fc18012ab90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6808/1506640341 conn(0x7fc18012a750 0x7fc18012ab90 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18012b3f0 osd=6 1 2024-02-13T00:54:17.438 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 10 client.38276.objecter close_session for osd.7 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc18012dd50 msgr2=0x7fc18012e1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc18012dd50 0x7fc18012e1b0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc184eb2700 1 -- 172.21.15.162:0/1788206591 reap_dead start 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 20 client.38276.objecter put_session s=0x7fc18012d6a0 osd=7 1 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 20 client.38276.objecter shutdown clearing up homeless session... 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 10 client.38276.objecter successfully canceled tick 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc16c062c90 msgr2=0x7fc16c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc16c062c90 0x7fc16c065150 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 rx=0x7fc18006c910 tx=0x7fc168009cb0).stop 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1800f8500 msgr2=0x7fc18006b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1800f8500 0x7fc18006b170 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 rx=0x7fc1700044e0 tx=0x7fc170007f90).stop 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 shutdown_connections 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.181:6812/1964192772 conn(0x7fc18012dd50 0x7fc18012e1b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.439 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1800f8500 0x7fc18006b170 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 --2- 172.21.15.162:0/1788206591 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc16c062c90 0x7fc16c065150 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 >> 172.21.15.162:0/1788206591 conn(0x7fc1800f38c0 msgr2=0x7fc1800f54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 shutdown_connections 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 -- 172.21.15.162:0/1788206591 wait complete. 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:17.430+0000 7fc186915700 1 librados: shutdown 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.440 INFO:teuthology.orchestra.run.smithi162.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-13T00:54:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:16.552466+0000 mgr.y (mgr.14544) 664 : cluster [DBG] pgmap v1222: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:17.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:17.279242+0000 mon.c (mon.2) 515 : audit [DBG] from='client.? 172.21.15.162:0/1788206591' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:54:17.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:16.552466+0000 mgr.y (mgr.14544) 664 : cluster [DBG] pgmap v1222: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:17.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:17.279242+0000 mon.c (mon.2) 515 : audit [DBG] from='client.? 172.21.15.162:0/1788206591' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:54:17.952 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:54:17.952 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:54:17.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:54:17.952 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-13T00:54:17.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14526.0:404 3.0 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e322)", 2024-02-13T00:54:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:38.696356+0000", 2024-02-13T00:54:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 578.71893895000005, 2024-02-13T00:54:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.80817416500000006, 2024-02-13T00:54:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:54:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14526", 2024-02-13T00:54:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.181:0/3978688658", 2024-02-13T00:54:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 404 2024-02-13T00:54:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.954 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:17.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:17.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:38.696356+0000", 2024-02-13T00:54:17.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:17.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:17.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:38.696356+0000", 2024-02-13T00:54:17.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:54:17.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:38.696354+0000", 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4250000000000001e-05 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:38.696368+0000", 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8759000000000001e-05 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:38.696387+0000", 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8650000000000007e-06 2024-02-13T00:54:17.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:38.696396+0000", 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9967000000000002e-05 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:38.696476+0000", 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.825e-06 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:54:17.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:38.696486+0000", 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.80654857300000005 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.503034+0000", 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000125802 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.503160+0000", 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001370437 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.504530+0000", 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.80817416500000006 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24329.0:1156 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94514175199360] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:39.296214+0000", 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 578.11908085200002, 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23397575300000001, 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:17.958 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24329", 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1156 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296214+0000", 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296214+0000", 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296214+0000", 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7270000000000001e-06 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296217+0000", 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7069999999999997e-06 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296223+0000", 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8659999999999997e-06 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296228+0000", 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000290616 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296519+0000", 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9189999999999994e-06 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.296529+0000", 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.205591095 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.502120+0000", 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00097887100000000008 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.503099+0000", 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026920178999999999 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.530019+0000", 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0661999999999994e-05 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.530109+0000", 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0629000000000003e-05 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.530190+0000", 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23397575300000001 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.963 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24329.0:1164 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94514175199360 gen 1] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:39.297223+0000", 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 578.11807221000004, 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23925315, 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24329", 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1164 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297223+0000", 2024-02-13T00:54:17.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297223+0000", 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297222+0000", 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5719999999999996e-06 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297227+0000", 2024-02-13T00:54:17.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9919999999999996e-06 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297236+0000", 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8600000000000001e-06 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297241+0000", 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2057000000000002e-05 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297273+0000", 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.198e-06 2024-02-13T00:54:17.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.297278+0000", 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20585395400000001 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.503132+0000", 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00084476899999999999 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for degraded object", 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.503977+0000", 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026240475999999999 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.530217+0000", 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0269999999999997e-05 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.530277+0000", 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061988110000000003 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:39.536476+0000", 2024-02-13T00:54:17.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23925315 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.37208.0:1 193.1s0 193:90ca3134:::oid_FlushAsyncPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e333)", 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:48.402768+0000", 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 569.01252695200003, 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.033695375, 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.37208", 2024-02-13T00:54:17.969 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3180918714", 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:48.402768+0000", 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:48.402768+0000", 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:48.402767+0000", 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0050000000000001e-06 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:48.402773+0000", 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7970000000000006e-06 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:48.402781+0000", 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.37e-06 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:48.402788+0000", 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5393999999999999e-05 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:48.402843+0000", 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8809999999999998e-06 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:48.402847+0000", 2024-02-13T00:54:17.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0211628610000001 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.424010+0000", 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131199 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.424141+0000", 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000406303 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:54:17.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.424547+0000", 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089238499999999999 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.425440+0000", 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010958383 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.436398+0000", 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5387000000000003e-05 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:17.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.436464+0000", 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.033695375 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14526.0:414 3.0 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e333)", 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:44:49.519387+0000", 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 567.89590829400004, 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.923429316, 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14526", 2024-02-13T00:54:17.975 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.181:0/3978688658", 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 414 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.519387+0000", 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.519387+0000", 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999962 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.519383+0000", 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0426000000000001e-05 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.519404+0000", 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2276000000000001e-05 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.519436+0000", 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5889e-05 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.519452+0000", 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000173136 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.519625+0000", 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3867e-05 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:49.519639+0000", 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.92285215399999998 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.442491+0000", 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016731100000000001 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.442658+0000", 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015790300000000001 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:44:50.442816+0000", 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.923429316 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:17.979 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.37301.0:1 202.5s1 202:b207418b:::oid_StatRemovePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:45:19.453005+0000", 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 537.96229012000003, 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.180058576, 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.37301", 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/3815326616", 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:17.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:19.453005+0000", 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:19.453005+0000", 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:19.453003+0000", 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5279999999999998e-06 2024-02-13T00:54:17.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:19.453010+0000", 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1164e-05 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:19.453021+0000", 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7479999999999993e-06 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:17.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:19.453029+0000", 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8600999999999998e-05 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:19.453078+0000", 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8779999999999998e-06 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:19.453081+0000", 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.17607449 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:20.629155+0000", 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015961999999999999 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:20.629315+0000", 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049075099999999999 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:20.629805+0000", 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051578099999999999 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:54:17.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:20.630321+0000", 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002675256 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:20.632996+0000", 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7138000000000007e-05 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:45:20.633064+0000", 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.180058576 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(4.14 403/401 [PushReplyOp(4:2d09e641:::obj_delete_at_hint.0000000088:head)])", 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:46:03.032030+0000", 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 494.38326552400002, 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089903727000000003, 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.032030+0000", 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:17.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.032030+0000", 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.032029+0000", 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9030000000000001e-06 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.032035+0000", 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6390000000000004e-06 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.032044+0000", 2024-02-13T00:54:17.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9810000000000003e-06 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.032049+0000", 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089737376999999993 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.121787+0000", 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000146663 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.121933+0000", 2024-02-13T00:54:17.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089903727000000003 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(4.14 404/401 [PushReplyOp(4:2dbd613a:::obj_delete_at_hint.0000000090:head)])", 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:46:03.471219+0000", 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 493.94407625500003, 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17573397499999999, 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.471219+0000", 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:17.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.471219+0000", 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.471218+0000", 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6920000000000002e-06 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.471226+0000", 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2906e-05 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.471238+0000", 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1940000000000001e-06 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.471246+0000", 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17546252500000001 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.646708+0000", 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000244639 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:46:03.646953+0000", 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17573397499999999 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25162.0:7082 219.1d 219:b9c3e6cd:test-rados-api-smithi162-84057-76::foo:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected e475)", 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:47:57.475847+0000", 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 379.939448196, 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26145807300000001, 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:17.992 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25162", 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7082 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.475847+0000", 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.475847+0000", 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.475845+0000", 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7629999999999995e-06 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.475855+0000", 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5560999999999999e-05 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.475870+0000", 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0550000000000005e-06 2024-02-13T00:54:17.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.475879+0000", 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0105000000000001e-05 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.475939+0000", 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6759999999999996e-06 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.475948+0000", 2024-02-13T00:54:17.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25886138400000003 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.734809+0000", 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013296400000000001 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.734942+0000", 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015625799999999999 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,6", 2024-02-13T00:54:17.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.735099+0000", 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013621939999999999 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.736461+0000", 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000439627 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.736901+0000", 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031737399999999999 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.737218+0000", 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8319000000000003e-05 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.737266+0000", 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8798999999999999e-05 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:47:57.737305+0000", 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26145807300000001 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:17.998 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:475 239.15 239:ab1ab8e8:test-rados-api-smithi162-84057-111::foo:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected e642)", 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:51:07.534846+0000", 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.880448841, 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088640260000000005, 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 475 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:17.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.534846+0000", 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.534846+0000", 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.534843+0000", 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3064000000000001e-05 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.534857+0000", 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8725000000000001e-05 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.534875+0000", 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4349999999999997e-06 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.534884+0000", 2024-02-13T00:54:18.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6863e-05 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.534951+0000", 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2407000000000001e-05 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.534973+0000", 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055403120000000004 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.540513+0000", 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021913799999999999 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.540732+0000", 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017683309999999999 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.542501+0000", 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9442000000000002e-05 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:18.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.542590+0000", 2024-02-13T00:54:18.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015991000000000001 2024-02-13T00:54:18.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,6", 2024-02-13T00:54:18.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.542750+0000", 2024-02-13T00:54:18.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035747099999999999 2024-02-13T00:54:18.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:54:18.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.543108+0000", 2024-02-13T00:54:18.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011938999999999999 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.543227+0000", 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00043918000000000001 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.543666+0000", 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5874999999999999e-05 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.543692+0000", 2024-02-13T00:54:18.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8329e-05 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:07.543710+0000", 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088640260000000005 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:1101 243.f 243:f2a1c75f:test-rados-api-smithi162-84057-119::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e676)", 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:51:41.536041+0000", 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 155.879254538, 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057862569000000003, 2024-02-13T00:54:18.006 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1101 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.536041+0000", 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.536041+0000", 2024-02-13T00:54:18.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.536039+0000", 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6320000000000008e-06 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.536047+0000", 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0386e-05 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:18.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.536057+0000", 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5889999999999997e-06 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.536065+0000", 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00046410199999999998 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.536529+0000", 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9508e-05 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:54:18.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.536548+0000", 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0096733199999999991 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.546222+0000", 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2630000000000003e-06 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.546230+0000", 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028260817000000001 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.574491+0000", 2024-02-13T00:54:18.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021817000000000001 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.574709+0000", 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045908759999999998 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.579300+0000", 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104768 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.579404+0000", 2024-02-13T00:54:18.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013933000000000001 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,7", 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.579544+0000", 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043057859999999998 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.583850+0000", 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0691000000000001e-05 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:54:18.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.583890+0000", 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0099271040000000008 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.593817+0000", 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9874999999999999e-05 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.593867+0000", 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5984e-05 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:18.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:51:41.593903+0000", 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057862569000000003 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:1524 245.13 245:c8594d0f:test-rados-api-smithi162-84057-123::foo:head [writefull 0~1] snapc 0=[] ondisk+write+known_if_redirected e695)", 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:52:02.542263+0000", 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 134.87303167799999, 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092796019999999996, 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:54:18.014 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1524 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.542263+0000", 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.542263+0000", 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.542261+0000", 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.188e-05 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.542273+0000", 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2773999999999999e-05 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.542296+0000", 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0582e-05 2024-02-13T00:54:18.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.542307+0000", 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8431999999999999e-05 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.542365+0000", 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.34e-06 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.542372+0000", 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043059630000000003 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.546678+0000", 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000204966 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.546883+0000", 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029810269999999998 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.549864+0000", 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4517e-05 2024-02-13T00:54:18.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.549919+0000", 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6932e-05 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,7", 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.550006+0000", 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030653199999999999 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.550312+0000", 2024-02-13T00:54:18.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037661699999999999 2024-02-13T00:54:18.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:54:18.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.550689+0000", 2024-02-13T00:54:18.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00080382800000000003 2024-02-13T00:54:18.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:54:18.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.551493+0000", 2024-02-13T00:54:18.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8969e-05 2024-02-13T00:54:18.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.551522+0000", 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.143e-05 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:02.551543+0000", 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092796019999999996 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.021 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:1528 245.13 245:c8594d0f:test-rados-api-smithi162-84057-123::foo:head [writefull 0~1] snapc 9=[9,8] ondisk+write+known_if_redirected e697)", 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:52:04.544185+0000", 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.87111052500001, 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0098529599999999991, 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1528 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.544185+0000", 2024-02-13T00:54:18.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.544185+0000", 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999976 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.544182+0000", 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2891000000000001e-05 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.544195+0000", 2024-02-13T00:54:18.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8508e-05 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.544214+0000", 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3729999999999993e-06 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.544222+0000", 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1749000000000002e-05 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.544274+0000", 2024-02-13T00:54:18.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7520000000000004e-06 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.544279+0000", 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0073666089999999997 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.551645+0000", 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109817 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.551755+0000", 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00092456099999999996 2024-02-13T00:54:18.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,7", 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.552680+0000", 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011205189999999999 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.553800+0000", 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.642e-05 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.553837+0000", 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013239600000000001 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.553969+0000", 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9824999999999999e-05 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.554009+0000", 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8724000000000001e-05 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:04.554038+0000", 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0098529599999999991 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:18.027 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27753.0:1794 247.b e716/712 247:d7155152:test-rados-api-smithi162-84057-127::foo:head v 716'2, mlcod=0'0)", 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:52:23.563898+0000", 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 113.851397535, 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0087750750000000002, 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.563898+0000", 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:18.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.563898+0000", 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.999999 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.563897+0000", 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051663849999999999 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.569063+0000", 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3040999999999999e-05 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.569076+0000", 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3350000000000003e-06 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.569083+0000", 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.295e-05 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.569135+0000", 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7523000000000001e-05 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:18.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.569153+0000", 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034800199999999999 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.572633+0000", 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9641999999999998e-05 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:23.572673+0000", 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0087750750000000002 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27753.0:2014 249.6 e735/729 249:602f83fe:::foo:head v 735'11, mlcod=735'9)", 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:52:50.582259+0000", 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 86.833035744, 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.052869559000000003, 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.582259+0000", 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.582259+0000", 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999986 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.582258+0000", 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7789000000000001e-05 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.582286+0000", 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8995999999999999e-05 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.582305+0000", 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6070000000000008e-06 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.582313+0000", 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0120999999999997e-05 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.582363+0000", 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8676000000000001e-05 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.582392+0000", 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.052687527999999997 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.635080+0000", 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9305000000000003e-05 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:52:50.635129+0000", 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.052869559000000003 2024-02-13T00:54:18.035 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.036 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:18.036 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.036 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:2736 252.a 252:56541f48:test-rados-api-smithi162-84057-137::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e769)", 2024-02-13T00:54:18.036 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:53:33.594577+0000", 2024-02-13T00:54:18.036 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.820718370000002, 2024-02-13T00:54:18.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014719394, 2024-02-13T00:54:18.036 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:18.036 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:54:18.036 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:18.036 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 2736 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.594577+0000", 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.594577+0000", 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.594575+0000", 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2459999999999997e-06 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.594582+0000", 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1361e-05 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.594594+0000", 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5379999999999999e-06 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.594600+0000", 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2061999999999997e-05 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.594652+0000", 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7559999999999999e-06 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op must wait for map", 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.594657+0000", 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030137670000000001 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.597671+0000", 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000220192 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for blocked object", 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.597891+0000", 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0086046539999999998 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.606496+0000", 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0979e-05 2024-02-13T00:54:18.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.606557+0000", 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000338742 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.606895+0000", 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7528000000000003e-05 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.606943+0000", 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023533360000000001 2024-02-13T00:54:18.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:33.609296+0000", 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014719394 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDECSubOpWrite(234.0s2 782/722 ECSubWrite(tid=853, reqid=client.27753.0:2911, at_version=782'6, trim_to=0'0, roll_forward_to=740'5))", 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:53:52.697899+0000", 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 24.717396526999998, 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074502750000000001, 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:18.042 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.697899+0000", 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.697899+0000", 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:18.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.697898+0000", 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1377999999999997e-05 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.697930+0000", 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3319999999999998e-06 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.697939+0000", 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3430000000000002e-06 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.697948+0000", 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00054679499999999999 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.698495+0000", 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1596999999999999e-05 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.698507+0000", 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0068246110000000004 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.705331+0000", 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7507999999999999e-05 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.705349+0000", 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074502750000000001 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDECSubOpWrite(234.0s2 782/722 ECSubWrite(tid=854, reqid=client.27753.0:2911, at_version=0'0, trim_to=0'0, roll_forward_to=782'6))", 2024-02-13T00:54:18.046 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:53:52.705931+0000", 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 24.709363840000002, 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011776272000000001, 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.705931+0000", 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.705931+0000", 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:18.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.705931+0000", 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0083e-05 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.705941+0000", 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3550000000000005e-06 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.705949+0000", 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7360000000000007e-06 2024-02-13T00:54:18.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.705958+0000", 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011272039999999999 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.707085+0000", 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.187e-05 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_started", 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.707097+0000", 2024-02-13T00:54:18.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01059191 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_committed", 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.717689+0000", 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8430000000000001e-05 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:53:52.717708+0000", 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011776272000000001 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27753.0:3122 255.10 255:0ff09505:test-rados-api-smithi162-84057-143::foo:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected e790)", 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:54:00.603621+0000", 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.811673965000001, 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018852965999999999, 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27753", 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3122 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:18.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.603621+0000", 2024-02-13T00:54:18.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:18.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:18.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.603621+0000", 2024-02-13T00:54:18.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999981 2024-02-13T00:54:18.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:18.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.603619+0000", 2024-02-13T00:54:18.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4280000000000002e-06 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.603629+0000", 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3288000000000001e-05 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.603642+0000", 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2110000000000004e-06 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:18.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.603651+0000", 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010957300000000001 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.604747+0000", 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1959999999999999e-06 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.604752+0000", 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011079494 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.615832+0000", 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012809200000000001 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.615960+0000", 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015091599999999999 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,7", 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.616111+0000", 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082970600000000004 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.616940+0000", 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044702450000000003 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.621411+0000", 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00098442800000000008 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.622395+0000", 2024-02-13T00:54:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8789000000000003e-05 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.622444+0000", 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0369e-05 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:00.622474+0000", 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018852965999999999 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14526.0:980 3.0 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e801)", 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-13T00:54:17.370900+0000", 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.044395108000000003, 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000228421, 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14526", 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.181:0/3978688658", 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 980 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-13T00:54:18.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:17.370900+0000", 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:17.370900+0000", 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4294967295.9999971 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:17.370897+0000", 2024-02-13T00:54:18.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8074e-05 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:17.370915+0000", 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3357e-05 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:17.370939+0000", 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3278999999999999e-05 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:17.370952+0000", 2024-02-13T00:54:18.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2240000000000001e-05 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:17.371004+0000", 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3615000000000007e-05 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:17.371068+0000", 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0769e-05 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-13T00:54:18.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-13T00:54:18.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-13T00:54:17.371128+0000", 2024-02-13T00:54:18.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000228421 2024-02-13T00:54:18.062 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.062 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:18.062 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.062 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-13T00:54:18.062 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-13T00:54:18.062 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:54:18.063 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:16.552466+0000 mgr.y (mgr.14544) 664 : cluster [DBG] pgmap v1222: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:18.063 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:17.279242+0000 mon.c (mon.2) 515 : audit [DBG] from='client.? 172.21.15.162:0/1788206591' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-13T00:54:18.357 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:18.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 7 1 2024-02-13T00:54:19.044 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:54:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:54:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:54:19.904 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:18.552961+0000 mgr.y (mgr.14544) 665 : cluster [DBG] pgmap v1223: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 642 B/s rd, 0 op/s 2024-02-13T00:54:19.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:18.552961+0000 mgr.y (mgr.14544) 665 : cluster [DBG] pgmap v1223: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 642 B/s rd, 0 op/s 2024-02-13T00:54:20.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:18.552961+0000 mgr.y (mgr.14544) 665 : cluster [DBG] pgmap v1223: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 642 B/s rd, 0 op/s 2024-02-13T00:54:20.393 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-13T00:54:21.071 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:21.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:20.553708+0000 mgr.y (mgr.14544) 666 : cluster [DBG] pgmap v1224: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:54:21.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:20.553708+0000 mgr.y (mgr.14544) 666 : cluster [DBG] pgmap v1224: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:54:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:20.553708+0000 mgr.y (mgr.14544) 666 : cluster [DBG] pgmap v1224: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:54:22.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.912+0000 7fe3f578e700 1 -- 172.21.15.162:0/1504210430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00fcb40 msgr2=0x7fe3f00fcf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:22.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.912+0000 7fe3f578e700 1 --2- 172.21.15.162:0/1504210430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00fcb40 0x7fe3f00fcf20 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7fe3d8004870 tx=0x7fe3d8009cb0).stop 2024-02-13T00:54:22.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.912+0000 7fe3f578e700 1 -- 172.21.15.162:0/1504210430 shutdown_connections 2024-02-13T00:54:22.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.912+0000 7fe3f578e700 1 --2- 172.21.15.162:0/1504210430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f00f73b0 0x7fe3f01039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:54:22.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.912+0000 7fe3f578e700 1 --2- 172.21.15.162:0/1504210430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00fcb40 0x7fe3f00fcf20 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:22.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.912+0000 7fe3f578e700 1 --2- 172.21.15.162:0/1504210430 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00fd4a0 0x7fe3f00f6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.912+0000 7fe3f578e700 1 -- 172.21.15.162:0/1504210430 >> 172.21.15.162:0/1504210430 conn(0x7fe3f00f2750 msgr2=0x7fe3f00f4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.912+0000 7fe3f578e700 1 -- 172.21.15.162:0/1504210430 shutdown_connections 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.912+0000 7fe3f578e700 1 -- 172.21.15.162:0/1504210430 wait complete. 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3f578e700 1 librados: starting msgr at 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3f578e700 1 librados: starting objecter 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3f578e700 1 Processor -- start 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3f578e700 1 -- start start 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3f578e700 1 librados: setting wanted keys 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3f578e700 1 librados: calling monclient init 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3f578e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00f73b0 0x7fe3f010e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3f578e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00fcb40 0x7fe3f010e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3f578e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f00fd4a0 0x7fe3f01083c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3f578e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f01088b0 con 0x7fe3f00fcb40 2024-02-13T00:54:22.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3f578e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f0108a20 con 0x7fe3f00fd4a0 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3f578e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f0108be0 con 0x7fe3f00f73b0 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3eeffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00f73b0 0x7fe3f010e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3ee7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00fcb40 0x7fe3f010e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3ee7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00fcb40 0x7fe3f010e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:52498/0 (socket says 172.21.15.162:52498) 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3eeffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00f73b0 0x7fe3f010e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:60050/0 (socket says 172.21.15.162:60050) 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.913+0000 7fe3ee7fc700 1 -- 172.21.15.162:0/2786619743 learned_addr learned my addr 172.21.15.162:0/2786619743 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3ef7fe700 1 --2- 172.21.15.162:0/2786619743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f00fd4a0 0x7fe3f01083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3eeffd700 1 -- 172.21.15.162:0/2786619743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f00fd4a0 msgr2=0x7fe3f01083c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3eeffd700 1 --2- 172.21.15.162:0/2786619743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f00fd4a0 0x7fe3f01083c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3eeffd700 1 -- 172.21.15.162:0/2786619743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00fcb40 msgr2=0x7fe3f010e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3eeffd700 1 --2- 172.21.15.162:0/2786619743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00fcb40 0x7fe3f010e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3eeffd700 1 -- 172.21.15.162:0/2786619743 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e0005040 con 0x7fe3f00f73b0 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3ee7fc700 1 --2- 172.21.15.162:0/2786619743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00fcb40 0x7fe3f010e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3ef7fe700 1 --2- 172.21.15.162:0/2786619743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f00fd4a0 0x7fe3f01083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3eeffd700 1 --2- 172.21.15.162:0/2786619743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00f73b0 0x7fe3f010e220 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 rx=0x7fe3d8004870 tx=0x7fe3d800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:22.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3e7fff700 10 client.?.objecter ms_handle_connect 0x7fe3f00f73b0 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3e7fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3e7fff700 1 -- 172.21.15.162:0/2786619743 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d800e070 con 0x7fe3f00f73b0 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3e7fff700 1 -- 172.21.15.162:0/2786619743 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3d8013da0 con 0x7fe3f00f73b0 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3f578e700 1 -- 172.21.15.162:0/2786619743 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d8005040 con 0x7fe3f00f73b0 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3e7fff700 1 -- 172.21.15.162:0/2786619743 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8019070 con 0x7fe3f00f73b0 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3f578e700 10 client.38282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3f578e700 1 -- 172.21.15.162:0/2786619743 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f01090d0 con 0x7fe3f00f73b0 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.914+0000 7fe3f578e700 1 librados: init done 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.915+0000 7fe3f578e700 1 -- 172.21.15.162:0/2786619743 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f0060790 con 0x7fe3f00f73b0 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.915+0000 7fe3e7fff700 1 -- 172.21.15.162:0/2786619743 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe3d801e420 con 0x7fe3f00f73b0 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.916+0000 7fe3e7fff700 1 --2- 172.21.15.162:0/2786619743 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe3dc062c90 0x7fe3dc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.916+0000 7fe3e7fff700 1 -- 172.21.15.162:0/2786619743 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(801..801 src has 449..801) v4 ==== 7422+0+0 (secure 0 0 0) 0x7fe3d807d860 con 0x7fe3f00f73b0 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.916+0000 7fe3e7fff700 10 client.38282.objecter ms_dispatch 0x7fe3f0001550 osd_map(801..801 src has 449..801) v4 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.916+0000 7fe3e7fff700 3 client.38282.objecter handle_osd_map got epochs [801,801] > 0 2024-02-13T00:54:22.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.916+0000 7fe3e7fff700 3 client.38282.objecter handle_osd_map decoding full epoch 801 2024-02-13T00:54:22.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.916+0000 7fe3ee7fc700 1 --2- 172.21.15.162:0/2786619743 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe3dc062c90 0x7fe3dc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:22.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.916+0000 7fe3e7fff700 20 client.38282.objecter dump_active .. 0 homeless 2024-02-13T00:54:22.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.916+0000 7fe3e7fff700 10 client.38282.objecter ms_handle_connect 0x7fe3dc062c90 2024-02-13T00:54:22.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.916+0000 7fe3ee7fc700 1 --2- 172.21.15.162:0/2786619743 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe3dc062c90 0x7fe3dc065150 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 rx=0x7fe3f0109b60 tx=0x7fe3e000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:22.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:21.920+0000 7fe3e7fff700 1 -- 172.21.15.162:0/2786619743 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe3d80510d0 con 0x7fe3f00f73b0 2024-02-13T00:54:22.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.096+0000 7fe3f578e700 1 -- 172.21.15.162:0/2786619743 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fe3f01098a0 con 0x7fe3f00f73b0 2024-02-13T00:54:22.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.635+0000 7fe3e7fff700 1 -- 172.21.15.162:0/2786619743 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v802) v1 ==== 91+0+0 (secure 0 0 0) 0x7fe3d8050cc0 con 0x7fe3f00f73b0 2024-02-13T00:54:22.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 10 librados: watch_flush enter 2024-02-13T00:54:22.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 10 librados: watch_flush exit 2024-02-13T00:54:22.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 20 client.38282.objecter shutdown clearing up homeless session... 2024-02-13T00:54:22.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 10 client.38282.objecter successfully canceled tick 2024-02-13T00:54:22.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 -- 172.21.15.162:0/2786619743 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe3dc062c90 msgr2=0x7fe3dc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 --2- 172.21.15.162:0/2786619743 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe3dc062c90 0x7fe3dc065150 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 rx=0x7fe3f0109b60 tx=0x7fe3e000b040).stop 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 -- 172.21.15.162:0/2786619743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00f73b0 msgr2=0x7fe3f010e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 --2- 172.21.15.162:0/2786619743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00f73b0 0x7fe3f010e220 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 rx=0x7fe3d8004870 tx=0x7fe3d800b040).stop 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 -- 172.21.15.162:0/2786619743 shutdown_connections 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 --2- 172.21.15.162:0/2786619743 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe3dc062c90 0x7fe3dc065150 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 --2- 172.21.15.162:0/2786619743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3f00fd4a0 0x7fe3f01083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 --2- 172.21.15.162:0/2786619743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00f73b0 0x7fe3f010e220 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 --2- 172.21.15.162:0/2786619743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00fcb40 0x7fe3f010e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 -- 172.21.15.162:0/2786619743 >> 172.21.15.162:0/2786619743 conn(0x7fe3f00f2750 msgr2=0x7fe3f0061100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 -- 172.21.15.162:0/2786619743 shutdown_connections 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 -- 172.21.15.162:0/2786619743 wait complete. 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:22.637+0000 7fe3f578e700 1 librados: shutdown 2024-02-13T00:54:22.701 INFO:teuthology.orchestra.run.smithi162.stderr:noscrub is unset 2024-02-13T00:54:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:22.097867+0000 mon.c (mon.2) 516 : audit [INF] from='client.? 172.21.15.162:0/2786619743' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:54:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:22.098245+0000 mon.a (mon.0) 3751 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:54:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:22.097867+0000 mon.c (mon.2) 516 : audit [INF] from='client.? 172.21.15.162:0/2786619743' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:54:22.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:22.098245+0000 mon.a (mon.0) 3751 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:54:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:22.097867+0000 mon.c (mon.2) 516 : audit [INF] from='client.? 172.21.15.162:0/2786619743' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:54:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:22.098245+0000 mon.a (mon.0) 3751 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-13T00:54:23.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-13T00:54:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:22.554139+0000 mgr.y (mgr.14544) 667 : cluster [DBG] pgmap v1225: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:54:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:22.627630+0000 mon.a (mon.0) 3752 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:54:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:22.634608+0000 mon.a (mon.0) 3753 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:54:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:22.634686+0000 mon.a (mon.0) 3754 : cluster [DBG] osdmap e802: 8 total, 8 up, 4 in 2024-02-13T00:54:24.046 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:22.554139+0000 mgr.y (mgr.14544) 667 : cluster [DBG] pgmap v1225: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:54:24.046 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:22.627630+0000 mon.a (mon.0) 3752 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:54:24.046 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:22.634608+0000 mon.a (mon.0) 3753 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:54:24.046 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:22.634686+0000 mon.a (mon.0) 3754 : cluster [DBG] osdmap e802: 8 total, 8 up, 4 in 2024-02-13T00:54:24.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:22.554139+0000 mgr.y (mgr.14544) 667 : cluster [DBG] pgmap v1225: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:54:24.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:22.627630+0000 mon.a (mon.0) 3752 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-13T00:54:24.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:22.634608+0000 mon.a (mon.0) 3753 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-13T00:54:24.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:22.634686+0000 mon.a (mon.0) 3754 : cluster [DBG] osdmap e802: 8 total, 8 up, 4 in 2024-02-13T00:54:24.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.304+0000 7fba353b2700 1 -- 172.21.15.162:0/2700222109 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba30102230 msgr2=0x7fba30104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:24.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.304+0000 7fba353b2700 1 --2- 172.21.15.162:0/2700222109 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba30102230 0x7fba30104620 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 rx=0x7fba28009be0 tx=0x7fba28004d10).stop 2024-02-13T00:54:24.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.304+0000 7fba353b2700 1 -- 172.21.15.162:0/2700222109 shutdown_connections 2024-02-13T00:54:24.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.304+0000 7fba353b2700 1 --2- 172.21.15.162:0/2700222109 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba30102230 0x7fba30104620 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:24.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.304+0000 7fba353b2700 1 --2- 172.21.15.162:0/2700222109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba300f8cd0 0x7fba300f90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:24.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.304+0000 7fba353b2700 1 --2- 172.21.15.162:0/2700222109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba300f95a0 0x7fba30101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:24.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.304+0000 7fba353b2700 1 -- 172.21.15.162:0/2700222109 >> 172.21.15.162:0/2700222109 conn(0x7fba300f2760 msgr2=0x7fba300f4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:24.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.304+0000 7fba353b2700 1 -- 172.21.15.162:0/2700222109 shutdown_connections 2024-02-13T00:54:24.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.304+0000 7fba353b2700 1 -- 172.21.15.162:0/2700222109 wait complete. 2024-02-13T00:54:24.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.305+0000 7fba353b2700 1 librados: starting msgr at 2024-02-13T00:54:24.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.305+0000 7fba353b2700 1 librados: starting objecter 2024-02-13T00:54:24.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.305+0000 7fba353b2700 1 Processor -- start 2024-02-13T00:54:24.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.305+0000 7fba353b2700 1 -- start start 2024-02-13T00:54:24.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.305+0000 7fba353b2700 1 librados: setting wanted keys 2024-02-13T00:54:24.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.305+0000 7fba353b2700 1 librados: calling monclient init 2024-02-13T00:54:24.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba353b2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba300f8cd0 0x7fba3010e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba353b2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba300f95a0 0x7fba3010e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba353b2700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba30102230 0x7fba30108320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba353b2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba30108810 con 0x7fba300f95a0 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba353b2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba30108980 con 0x7fba300f8cd0 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba353b2700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba30108b40 con 0x7fba30102230 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba2e7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba300f95a0 0x7fba3010e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba2e7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba300f95a0 0x7fba3010e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43790/0 (socket says 172.21.15.162:43790) 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba2e7fc700 1 -- 172.21.15.162:0/3391761395 learned_addr learned my addr 172.21.15.162:0/3391761395 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba2f7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba30102230 0x7fba30108320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba2effd700 1 --2- 172.21.15.162:0/3391761395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba300f8cd0 0x7fba3010e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba2e7fc700 1 -- 172.21.15.162:0/3391761395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba30102230 msgr2=0x7fba30108320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba2e7fc700 1 --2- 172.21.15.162:0/3391761395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba30102230 0x7fba30108320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba2e7fc700 1 -- 172.21.15.162:0/3391761395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba300f8cd0 msgr2=0x7fba3010e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba2e7fc700 1 --2- 172.21.15.162:0/3391761395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba300f8cd0 0x7fba3010e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba2e7fc700 1 -- 172.21.15.162:0/3391761395 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20005040 con 0x7fba300f95a0 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba2f7fe700 1 --2- 172.21.15.162:0/3391761395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba30102230 0x7fba30108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba2e7fc700 1 --2- 172.21.15.162:0/3391761395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba300f95a0 0x7fba3010e670 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 rx=0x7fba2400e990 tx=0x7fba2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba17fff700 10 client.?.objecter ms_handle_connect 0x7fba300f95a0 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.306+0000 7fba17fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.307+0000 7fba17fff700 1 -- 172.21.15.162:0/3391761395 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba24004680 con 0x7fba300f95a0 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.307+0000 7fba353b2700 1 -- 172.21.15.162:0/3391761395 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba280097c0 con 0x7fba300f95a0 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.307+0000 7fba17fff700 1 -- 172.21.15.162:0/3391761395 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba2400f040 con 0x7fba300f95a0 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.307+0000 7fba17fff700 1 -- 172.21.15.162:0/3391761395 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba240146b0 con 0x7fba300f95a0 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.307+0000 7fba353b2700 10 client.28194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.307+0000 7fba353b2700 1 -- 172.21.15.162:0/3391761395 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba30109070 con 0x7fba300f95a0 2024-02-13T00:54:24.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.307+0000 7fba353b2700 1 librados: init done 2024-02-13T00:54:24.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.308+0000 7fba353b2700 1 -- 172.21.15.162:0/3391761395 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba30060770 con 0x7fba300f95a0 2024-02-13T00:54:24.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.308+0000 7fba17fff700 1 -- 172.21.15.162:0/3391761395 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fba24014ae0 con 0x7fba300f95a0 2024-02-13T00:54:24.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.309+0000 7fba17fff700 1 --2- 172.21.15.162:0/3391761395 >> v2:172.21.15.162:6800/2866534053 conn(0x7fba18062c90 0x7fba18065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:24.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.309+0000 7fba17fff700 1 -- 172.21.15.162:0/3391761395 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(802..802 src has 449..802) v4 ==== 7422+0+0 (secure 0 0 0) 0x7fba2407c9e0 con 0x7fba300f95a0 2024-02-13T00:54:24.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.309+0000 7fba17fff700 10 client.28194.objecter ms_dispatch 0x7fba30001550 osd_map(802..802 src has 449..802) v4 2024-02-13T00:54:24.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.309+0000 7fba17fff700 3 client.28194.objecter handle_osd_map got epochs [802,802] > 0 2024-02-13T00:54:24.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.309+0000 7fba17fff700 3 client.28194.objecter handle_osd_map decoding full epoch 802 2024-02-13T00:54:24.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.309+0000 7fba2effd700 1 --2- 172.21.15.162:0/3391761395 >> v2:172.21.15.162:6800/2866534053 conn(0x7fba18062c90 0x7fba18065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:24.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.309+0000 7fba17fff700 20 client.28194.objecter dump_active .. 0 homeless 2024-02-13T00:54:24.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.309+0000 7fba17fff700 10 client.28194.objecter ms_handle_connect 0x7fba18062c90 2024-02-13T00:54:24.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.309+0000 7fba2effd700 1 --2- 172.21.15.162:0/3391761395 >> v2:172.21.15.162:6800/2866534053 conn(0x7fba18062c90 0x7fba18065150 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 rx=0x7fba20009350 tx=0x7fba2000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:24.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.315+0000 7fba17fff700 1 -- 172.21.15.162:0/3391761395 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fba24049f60 con 0x7fba300f95a0 2024-02-13T00:54:24.345 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:24.348 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:24.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.512+0000 7fba353b2700 1 -- 172.21.15.162:0/3391761395 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7fba301071c0 con 0x7fba300f95a0 2024-02-13T00:54:24.640 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.636+0000 7fba17fff700 1 -- 172.21.15.162:0/3391761395 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v803) v1 ==== 113+0+0 (secure 0 0 0) 0x7fba240501e0 con 0x7fba300f95a0 2024-02-13T00:54:24.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 10 librados: watch_flush enter 2024-02-13T00:54:24.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 10 librados: watch_flush exit 2024-02-13T00:54:24.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 20 client.28194.objecter shutdown clearing up homeless session... 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 10 client.28194.objecter successfully canceled tick 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 1 -- 172.21.15.162:0/3391761395 >> v2:172.21.15.162:6800/2866534053 conn(0x7fba18062c90 msgr2=0x7fba18065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 1 --2- 172.21.15.162:0/3391761395 >> v2:172.21.15.162:6800/2866534053 conn(0x7fba18062c90 0x7fba18065150 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 rx=0x7fba20009350 tx=0x7fba2000b040).stop 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 1 -- 172.21.15.162:0/3391761395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba300f95a0 msgr2=0x7fba3010e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 1 --2- 172.21.15.162:0/3391761395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba300f95a0 0x7fba3010e670 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 rx=0x7fba2400e990 tx=0x7fba2400b040).stop 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 1 -- 172.21.15.162:0/3391761395 shutdown_connections 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 1 --2- 172.21.15.162:0/3391761395 >> v2:172.21.15.162:6800/2866534053 conn(0x7fba18062c90 0x7fba18065150 unknown :-1 s=CLOSED pgs=3054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 1 --2- 172.21.15.162:0/3391761395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba30102230 0x7fba30108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 1 --2- 172.21.15.162:0/3391761395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba300f8cd0 0x7fba3010e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 1 --2- 172.21.15.162:0/3391761395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba300f95a0 0x7fba3010e670 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.638+0000 7fba353b2700 1 -- 172.21.15.162:0/3391761395 >> 172.21.15.162:0/3391761395 conn(0x7fba300f2760 msgr2=0x7fba300f3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.639+0000 7fba353b2700 1 -- 172.21.15.162:0/3391761395 shutdown_connections 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.639+0000 7fba353b2700 1 -- 172.21.15.162:0/3391761395 wait complete. 2024-02-13T00:54:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:24.639+0000 7fba353b2700 1 librados: shutdown 2024-02-13T00:54:24.645 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.7 to 1 (10000) 2024-02-13T00:54:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:23.236655+0000 osd.1 (osd.1) 331 : cluster [DBG] 28.1e scrub starts 2024-02-13T00:54:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:23.237602+0000 osd.1 (osd.1) 332 : cluster [DBG] 28.1e scrub ok 2024-02-13T00:54:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:23.583449+0000 osd.6 (osd.6) 259 : cluster [DBG] 4.15 scrub starts 2024-02-13T00:54:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:23.587098+0000 osd.6 (osd.6) 260 : cluster [DBG] 4.15 scrub ok 2024-02-13T00:54:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:24.513874+0000 mon.a (mon.0) 3755 : audit [INF] from='client.? 172.21.15.162:0/3391761395' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-13T00:54:25.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:23.236655+0000 osd.1 (osd.1) 331 : cluster [DBG] 28.1e scrub starts 2024-02-13T00:54:25.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:23.237602+0000 osd.1 (osd.1) 332 : cluster [DBG] 28.1e scrub ok 2024-02-13T00:54:25.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:23.583449+0000 osd.6 (osd.6) 259 : cluster [DBG] 4.15 scrub starts 2024-02-13T00:54:25.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:23.587098+0000 osd.6 (osd.6) 260 : cluster [DBG] 4.15 scrub ok 2024-02-13T00:54:25.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:24.513874+0000 mon.a (mon.0) 3755 : audit [INF] from='client.? 172.21.15.162:0/3391761395' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-13T00:54:25.071 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:23.236655+0000 osd.1 (osd.1) 331 : cluster [DBG] 28.1e scrub starts 2024-02-13T00:54:25.071 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:23.237602+0000 osd.1 (osd.1) 332 : cluster [DBG] 28.1e scrub ok 2024-02-13T00:54:25.071 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:23.583449+0000 osd.6 (osd.6) 259 : cluster [DBG] 4.15 scrub starts 2024-02-13T00:54:25.072 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:23.587098+0000 osd.6 (osd.6) 260 : cluster [DBG] 4.15 scrub ok 2024-02-13T00:54:25.072 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:24.513874+0000 mon.a (mon.0) 3755 : audit [INF] from='client.? 172.21.15.162:0/3391761395' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-13T00:54:25.430 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-13T00:54:25.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:54:26.000 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:24.554465+0000 mgr.y (mgr.14544) 668 : cluster [DBG] pgmap v1227: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 570 B/s rd, 0 op/s 2024-02-13T00:54:26.000 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:24.636968+0000 mon.a (mon.0) 3756 : audit [INF] from='client.? 172.21.15.162:0/3391761395' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-13T00:54:26.000 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:24.637013+0000 mon.a (mon.0) 3757 : cluster [DBG] osdmap e803: 8 total, 8 up, 4 in 2024-02-13T00:54:26.000 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:25.409288+0000 mon.c (mon.2) 517 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:54:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:24.554465+0000 mgr.y (mgr.14544) 668 : cluster [DBG] pgmap v1227: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 570 B/s rd, 0 op/s 2024-02-13T00:54:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:24.636968+0000 mon.a (mon.0) 3756 : audit [INF] from='client.? 172.21.15.162:0/3391761395' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-13T00:54:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:24.637013+0000 mon.a (mon.0) 3757 : cluster [DBG] osdmap e803: 8 total, 8 up, 4 in 2024-02-13T00:54:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:25.409288+0000 mon.c (mon.2) 517 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:54:26.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:24.554465+0000 mgr.y (mgr.14544) 668 : cluster [DBG] pgmap v1227: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 570 B/s rd, 0 op/s 2024-02-13T00:54:26.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:24.636968+0000 mon.a (mon.0) 3756 : audit [INF] from='client.? 172.21.15.162:0/3391761395' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-13T00:54:26.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:24.637013+0000 mon.a (mon.0) 3757 : cluster [DBG] osdmap e803: 8 total, 8 up, 4 in 2024-02-13T00:54:26.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:25.409288+0000 mon.c (mon.2) 517 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:54:26.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:54:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:54:26.418 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:54:26] "GET /metrics HTTP/1.1" 200 185414 "" "Prometheus/2.33.4" 2024-02-13T00:54:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:25.246287+0000 osd.1 (osd.1) 333 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:54:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:25.247060+0000 osd.1 (osd.1) 334 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:54:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:25.676789+0000 mon.a (mon.0) 3758 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:54:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:25.956978+0000 mon.c (mon.2) 518 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:54:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:25.957624+0000 mon.c (mon.2) 519 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:54:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:25.962209+0000 mon.a (mon.0) 3759 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:54:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:25.996351+0000 mgr.y (mgr.14544) 669 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:25.246287+0000 osd.1 (osd.1) 333 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:54:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:25.247060+0000 osd.1 (osd.1) 334 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:54:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:25.676789+0000 mon.a (mon.0) 3758 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:54:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:25.956978+0000 mon.c (mon.2) 518 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:54:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:25.957624+0000 mon.c (mon.2) 519 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:54:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:25.962209+0000 mon.a (mon.0) 3759 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:54:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:25.996351+0000 mgr.y (mgr.14544) 669 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:27.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:25.246287+0000 osd.1 (osd.1) 333 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:54:27.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:25.247060+0000 osd.1 (osd.1) 334 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:54:27.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:25.676789+0000 mon.a (mon.0) 3758 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:54:27.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:25.956978+0000 mon.c (mon.2) 518 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:54:27.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:25.957624+0000 mon.c (mon.2) 519 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:54:27.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:25.962209+0000 mon.a (mon.0) 3759 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:54:27.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:25.996351+0000 mgr.y (mgr.14544) 669 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:28.047 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:26.555206+0000 mgr.y (mgr.14544) 670 : cluster [DBG] pgmap v1229: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:54:28.048 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:27.593867+0000 osd.6 (osd.6) 261 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:54:28.048 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:27.596891+0000 osd.6 (osd.6) 262 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:54:28.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:26.555206+0000 mgr.y (mgr.14544) 670 : cluster [DBG] pgmap v1229: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:54:28.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:27.593867+0000 osd.6 (osd.6) 261 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:54:28.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:27.596891+0000 osd.6 (osd.6) 262 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:54:28.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.245+0000 7f6e37c81700 1 -- 172.21.15.162:0/1440994484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e300fd490 msgr2=0x7f6e300f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.245+0000 7f6e37c81700 1 --2- 172.21.15.162:0/1440994484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e300fd490 0x7f6e300f6cc0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 rx=0x7f6e20004870 tx=0x7f6e20009cb0).stop 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.245+0000 7f6e37c81700 1 -- 172.21.15.162:0/1440994484 shutdown_connections 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.245+0000 7f6e37c81700 1 --2- 172.21.15.162:0/1440994484 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e300f73a0 0x7f6e301039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.245+0000 7f6e37c81700 1 --2- 172.21.15.162:0/1440994484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e300fcb30 0x7f6e300fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.245+0000 7f6e37c81700 1 --2- 172.21.15.162:0/1440994484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e300fd490 0x7f6e300f6cc0 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.245+0000 7f6e37c81700 1 -- 172.21.15.162:0/1440994484 >> 172.21.15.162:0/1440994484 conn(0x7f6e3006eb90 msgr2=0x7f6e300f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.245+0000 7f6e37c81700 1 -- 172.21.15.162:0/1440994484 shutdown_connections 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.245+0000 7f6e37c81700 1 -- 172.21.15.162:0/1440994484 wait complete. 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.245+0000 7f6e37c81700 1 librados: starting msgr at 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.245+0000 7f6e37c81700 1 librados: starting objecter 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.245+0000 7f6e37c81700 1 Processor -- start 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e37c81700 1 -- start start 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e37c81700 1 librados: setting wanted keys 2024-02-13T00:54:28.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e37c81700 1 librados: calling monclient init 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e37c81700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e300f73a0 0x7f6e30109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e37c81700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e300fcb30 0x7f6e30109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e37c81700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e300fd490 0x7f6e30109c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e37c81700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e301143a0 con 0x7f6e300fd490 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e37c81700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e30114510 con 0x7f6e300fcb30 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e37c81700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e301146d0 con 0x7f6e300f73a0 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e35a1d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e300f73a0 0x7f6e30109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e3621e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e300fd490 0x7f6e30109c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e35a1d700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e300f73a0 0x7f6e30109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:38922/0 (socket says 172.21.15.162:38922) 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e35a1d700 1 -- 172.21.15.162:0/2260070016 learned_addr learned my addr 172.21.15.162:0/2260070016 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e3521c700 1 --2- 172.21.15.162:0/2260070016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e300fcb30 0x7f6e30109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e35a1d700 1 -- 172.21.15.162:0/2260070016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e300fcb30 msgr2=0x7f6e30109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e35a1d700 1 --2- 172.21.15.162:0/2260070016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e300fcb30 0x7f6e30109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e35a1d700 1 -- 172.21.15.162:0/2260070016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e300fd490 msgr2=0x7f6e30109c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:28.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e35a1d700 1 --2- 172.21.15.162:0/2260070016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e300fd490 0x7f6e30109c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e35a1d700 1 -- 172.21.15.162:0/2260070016 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e280097c0 con 0x7f6e300f73a0 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e3621e700 1 --2- 172.21.15.162:0/2260070016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e300fd490 0x7f6e30109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e35a1d700 1 --2- 172.21.15.162:0/2260070016 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e300f73a0 0x7f6e30109170 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 rx=0x7f6e2c0041d0 tx=0x7f6e2c009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e26ffd700 10 client.?.objecter ms_handle_connect 0x7f6e300f73a0 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.246+0000 7f6e26ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.247+0000 7f6e26ffd700 1 -- 172.21.15.162:0/2260070016 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c007b40 con 0x7f6e300f73a0 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.247+0000 7f6e37c81700 1 -- 172.21.15.162:0/2260070016 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e20005040 con 0x7f6e300f73a0 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.247+0000 7f6e26ffd700 1 -- 172.21.15.162:0/2260070016 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e2c00f040 con 0x7f6e300f73a0 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.247+0000 7f6e26ffd700 1 -- 172.21.15.162:0/2260070016 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c014730 con 0x7f6e300f73a0 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.247+0000 7f6e37c81700 10 client.38294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.247+0000 7f6e37c81700 1 -- 172.21.15.162:0/2260070016 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e30114de0 con 0x7f6e300f73a0 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.247+0000 7f6e37c81700 1 librados: init done 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.248+0000 7f6e37c81700 1 -- 172.21.15.162:0/2260070016 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e30106020 con 0x7f6e300f73a0 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.248+0000 7f6e26ffd700 1 -- 172.21.15.162:0/2260070016 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6e2c009110 con 0x7f6e300f73a0 2024-02-13T00:54:28.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.248+0000 7f6e26ffd700 1 --2- 172.21.15.162:0/2260070016 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6e1c062c90 0x7f6e1c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:28.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.248+0000 7f6e26ffd700 1 -- 172.21.15.162:0/2260070016 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(803..803 src has 449..803) v4 ==== 7422+0+0 (secure 0 0 0) 0x7f6e2c07cf40 con 0x7f6e300f73a0 2024-02-13T00:54:28.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.248+0000 7f6e26ffd700 10 client.38294.objecter ms_dispatch 0x7f6e30001550 osd_map(803..803 src has 449..803) v4 2024-02-13T00:54:28.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.248+0000 7f6e26ffd700 3 client.38294.objecter handle_osd_map got epochs [803,803] > 0 2024-02-13T00:54:28.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.248+0000 7f6e26ffd700 3 client.38294.objecter handle_osd_map decoding full epoch 803 2024-02-13T00:54:28.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.248+0000 7f6e3521c700 1 --2- 172.21.15.162:0/2260070016 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6e1c062c90 0x7f6e1c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:28.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.248+0000 7f6e26ffd700 20 client.38294.objecter dump_active .. 0 homeless 2024-02-13T00:54:28.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.249+0000 7f6e26ffd700 10 client.38294.objecter ms_handle_connect 0x7f6e1c062c90 2024-02-13T00:54:28.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.249+0000 7f6e3521c700 1 --2- 172.21.15.162:0/2260070016 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6e1c062c90 0x7f6e1c065150 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 rx=0x7f6e20009a00 tx=0x7f6e2000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:28.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.253+0000 7f6e26ffd700 1 -- 172.21.15.162:0/2260070016 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6e2c0506d0 con 0x7f6e300f73a0 2024-02-13T00:54:28.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:26.555206+0000 mgr.y (mgr.14544) 670 : cluster [DBG] pgmap v1229: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:54:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:27.593867+0000 osd.6 (osd.6) 261 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:54:28.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:27.596891+0000 osd.6 (osd.6) 262 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:54:28.436 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.432+0000 7f6e37c81700 1 -- 172.21.15.162:0/2260070016 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f6e3005fb80 con 0x7f6e300f73a0 2024-02-13T00:54:28.480 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:54:28.480 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:54:28.480 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-13T00:54:28.480 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:54:28.487 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:28.973 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:54:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:54:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:54:28.980 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.975+0000 7f6e26ffd700 1 -- 172.21.15.162:0/2260070016 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v804) v1 ==== 101+0+0 (secure 0 0 0) 0x7f6e2c00d360 con 0x7f6e300f73a0 2024-02-13T00:54:28.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.978+0000 7f6e37c81700 10 librados: watch_flush enter 2024-02-13T00:54:28.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.980+0000 7f6e37c81700 10 librados: watch_flush exit 2024-02-13T00:54:28.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.980+0000 7f6e37c81700 20 client.38294.objecter shutdown clearing up homeless session... 2024-02-13T00:54:28.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.980+0000 7f6e37c81700 10 client.38294.objecter successfully canceled tick 2024-02-13T00:54:28.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.980+0000 7f6e37c81700 1 -- 172.21.15.162:0/2260070016 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6e1c062c90 msgr2=0x7f6e1c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.980+0000 7f6e37c81700 1 --2- 172.21.15.162:0/2260070016 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6e1c062c90 0x7f6e1c065150 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 rx=0x7f6e20009a00 tx=0x7f6e2000b040).stop 2024-02-13T00:54:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.980+0000 7f6e37c81700 1 -- 172.21.15.162:0/2260070016 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e300f73a0 msgr2=0x7f6e30109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.980+0000 7f6e37c81700 1 --2- 172.21.15.162:0/2260070016 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e300f73a0 0x7f6e30109170 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 rx=0x7f6e2c0041d0 tx=0x7f6e2c009f60).stop 2024-02-13T00:54:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.980+0000 7f6e37c81700 1 -- 172.21.15.162:0/2260070016 shutdown_connections 2024-02-13T00:54:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.980+0000 7f6e37c81700 1 --2- 172.21.15.162:0/2260070016 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6e1c062c90 0x7f6e1c065150 unknown :-1 s=CLOSED pgs=3055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.980+0000 7f6e37c81700 1 --2- 172.21.15.162:0/2260070016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e300fd490 0x7f6e30109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.980+0000 7f6e37c81700 1 --2- 172.21.15.162:0/2260070016 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e300f73a0 0x7f6e30109170 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.980+0000 7f6e37c81700 1 --2- 172.21.15.162:0/2260070016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e300fcb30 0x7f6e30109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.981+0000 7f6e37c81700 1 -- 172.21.15.162:0/2260070016 >> 172.21.15.162:0/2260070016 conn(0x7f6e3006eb90 msgr2=0x7f6e300fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.981+0000 7f6e37c81700 1 -- 172.21.15.162:0/2260070016 shutdown_connections 2024-02-13T00:54:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.981+0000 7f6e37c81700 1 -- 172.21.15.162:0/2260070016 wait complete. 2024-02-13T00:54:28.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:28.981+0000 7f6e37c81700 1 librados: shutdown 2024-02-13T00:54:28.986 INFO:teuthology.orchestra.run.smithi162.stderr:nodeep-scrub is unset 2024-02-13T00:54:29.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:28.433930+0000 mon.c (mon.2) 520 : audit [INF] from='client.? 172.21.15.162:0/2260070016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:54:29.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:28.434252+0000 mon.a (mon.0) 3760 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:54:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:28.433930+0000 mon.c (mon.2) 520 : audit [INF] from='client.? 172.21.15.162:0/2260070016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:54:29.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:28.434252+0000 mon.a (mon.0) 3760 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:54:29.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:28.433930+0000 mon.c (mon.2) 520 : audit [INF] from='client.? 172.21.15.162:0/2260070016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:54:29.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:28.434252+0000 mon.a (mon.0) 3760 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-13T00:54:29.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-13T00:54:30.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:28.555686+0000 mgr.y (mgr.14544) 671 : cluster [DBG] pgmap v1230: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:54:30.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:28.968674+0000 mon.a (mon.0) 3761 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:54:30.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:28.976250+0000 mon.a (mon.0) 3762 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:54:30.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:28.976295+0000 mon.a (mon.0) 3763 : cluster [DBG] osdmap e804: 8 total, 8 up, 4 in 2024-02-13T00:54:30.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:29.164567+0000 osd.0 (osd.0) 319 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:54:30.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:29.167961+0000 osd.0 (osd.0) 320 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:54:30.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:28.555686+0000 mgr.y (mgr.14544) 671 : cluster [DBG] pgmap v1230: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:54:30.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:28.968674+0000 mon.a (mon.0) 3761 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:54:30.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:28.976250+0000 mon.a (mon.0) 3762 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:54:30.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:28.976295+0000 mon.a (mon.0) 3763 : cluster [DBG] osdmap e804: 8 total, 8 up, 4 in 2024-02-13T00:54:30.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:29.164567+0000 osd.0 (osd.0) 319 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:54:30.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:29.167961+0000 osd.0 (osd.0) 320 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:54:30.318 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:28.555686+0000 mgr.y (mgr.14544) 671 : cluster [DBG] pgmap v1230: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:54:30.318 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:28.968674+0000 mon.a (mon.0) 3761 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-13T00:54:30.318 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:28.976250+0000 mon.a (mon.0) 3762 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-13T00:54:30.318 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:28.976295+0000 mon.a (mon.0) 3763 : cluster [DBG] osdmap e804: 8 total, 8 up, 4 in 2024-02-13T00:54:30.318 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:29.164567+0000 osd.0 (osd.0) 319 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:54:30.318 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:29.167961+0000 osd.0 (osd.0) 320 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:54:31.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.174+0000 7fc113928700 1 -- 172.21.15.162:0/2679284690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c0f8500 msgr2=0x7fc10c104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:31.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.174+0000 7fc113928700 1 --2- 172.21.15.162:0/2679284690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c0f8500 0x7fc10c104bd0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 rx=0x7fc104009be0 tx=0x7fc104004d10).stop 2024-02-13T00:54:31.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.174+0000 7fc113928700 1 -- 172.21.15.162:0/2679284690 shutdown_connections 2024-02-13T00:54:31.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.174+0000 7fc113928700 1 --2- 172.21.15.162:0/2679284690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c0f8500 0x7fc10c104bd0 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:31.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.174+0000 7fc113928700 1 --2- 172.21.15.162:0/2679284690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc10c0f9b70 0x7fc10c0f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:31.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.174+0000 7fc113928700 1 --2- 172.21.15.162:0/2679284690 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc10c0fa440 0x7fc10c0f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:31.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.174+0000 7fc113928700 1 -- 172.21.15.162:0/2679284690 >> 172.21.15.162:0/2679284690 conn(0x7fc10c0f38c0 msgr2=0x7fc10c0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:31.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.174+0000 7fc113928700 1 -- 172.21.15.162:0/2679284690 shutdown_connections 2024-02-13T00:54:31.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.174+0000 7fc113928700 1 -- 172.21.15.162:0/2679284690 wait complete. 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.174+0000 7fc113928700 1 librados: starting msgr at 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.174+0000 7fc113928700 1 librados: starting objecter 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc113928700 1 Processor -- start 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc113928700 1 -- start start 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc113928700 1 librados: setting wanted keys 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc113928700 1 librados: calling monclient init 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc113928700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc10c0f8500 0x7fc10c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc113928700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c0f9b70 0x7fc10c06b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc113928700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc10c0fa440 0x7fc10c06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc113928700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c116520 con 0x7fc10c0f9b70 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc113928700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c116690 con 0x7fc10c0fa440 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc113928700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c116850 con 0x7fc10c0f8500 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc110ec3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c0f9b70 0x7fc10c06b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc1116c4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc10c0f8500 0x7fc10c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc110ec3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c0f9b70 0x7fc10c06b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43830/0 (socket says 172.21.15.162:43830) 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc110ec3700 1 -- 172.21.15.162:0/4093184993 learned_addr learned my addr 172.21.15.162:0/4093184993 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc1116c4700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc10c0f8500 0x7fc10c06b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:38934/0 (socket says 172.21.15.162:38934) 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc111ec5700 1 --2- 172.21.15.162:0/4093184993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc10c0fa440 0x7fc10c06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc110ec3700 1 -- 172.21.15.162:0/4093184993 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc10c0f8500 msgr2=0x7fc10c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc110ec3700 1 --2- 172.21.15.162:0/4093184993 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc10c0f8500 0x7fc10c06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc110ec3700 1 -- 172.21.15.162:0/4093184993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc10c0fa440 msgr2=0x7fc10c06bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc110ec3700 1 --2- 172.21.15.162:0/4093184993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc10c0fa440 0x7fc10c06bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc110ec3700 1 -- 172.21.15.162:0/4093184993 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1040097c0 con 0x7fc10c0f9b70 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc111ec5700 1 --2- 172.21.15.162:0/4093184993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc10c0fa440 0x7fc10c06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc1116c4700 1 --2- 172.21.15.162:0/4093184993 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc10c0f8500 0x7fc10c06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc110ec3700 1 --2- 172.21.15.162:0/4093184993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c0f9b70 0x7fc10c06b660 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7fc10800e970 tx=0x7fc10800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc1027fc700 10 client.?.objecter ms_handle_connect 0x7fc10c0f9b70 2024-02-13T00:54:31.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.175+0000 7fc1027fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.176+0000 7fc1027fc700 1 -- 172.21.15.162:0/4093184993 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10800ee60 con 0x7fc10c0f9b70 2024-02-13T00:54:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.176+0000 7fc1027fc700 1 -- 172.21.15.162:0/4093184993 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc10800f040 con 0x7fc10c0f9b70 2024-02-13T00:54:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.176+0000 7fc113928700 1 -- 172.21.15.162:0/4093184993 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10c116b30 con 0x7fc10c0f9b70 2024-02-13T00:54:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.176+0000 7fc1027fc700 1 -- 172.21.15.162:0/4093184993 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc108007ea0 con 0x7fc10c0f9b70 2024-02-13T00:54:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.176+0000 7fc113928700 10 client.28206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.176+0000 7fc113928700 1 -- 172.21.15.162:0/4093184993 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10c117070 con 0x7fc10c0f9b70 2024-02-13T00:54:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.176+0000 7fc113928700 1 librados: init done 2024-02-13T00:54:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.177+0000 7fc113928700 1 -- 172.21.15.162:0/4093184993 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10c0475e0 con 0x7fc10c0f9b70 2024-02-13T00:54:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.177+0000 7fc1027fc700 1 -- 172.21.15.162:0/4093184993 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc10800cb10 con 0x7fc10c0f9b70 2024-02-13T00:54:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.177+0000 7fc1027fc700 1 --2- 172.21.15.162:0/4093184993 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0f8062c90 0x7fc0f8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.177+0000 7fc1027fc700 1 -- 172.21.15.162:0/4093184993 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(805..805 src has 449..805) v4 ==== 7422+0+0 (secure 0 0 0) 0x7fc10807ca90 con 0x7fc10c0f9b70 2024-02-13T00:54:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.177+0000 7fc1027fc700 10 client.28206.objecter ms_dispatch 0x7fc10c001550 osd_map(805..805 src has 449..805) v4 2024-02-13T00:54:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.177+0000 7fc1027fc700 3 client.28206.objecter handle_osd_map got epochs [805,805] > 0 2024-02-13T00:54:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.177+0000 7fc1027fc700 3 client.28206.objecter handle_osd_map decoding full epoch 805 2024-02-13T00:54:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.177+0000 7fc1116c4700 1 --2- 172.21.15.162:0/4093184993 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0f8062c90 0x7fc0f8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.177+0000 7fc1027fc700 20 client.28206.objecter dump_active .. 0 homeless 2024-02-13T00:54:31.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.178+0000 7fc1027fc700 10 client.28206.objecter ms_handle_connect 0x7fc0f8062c90 2024-02-13T00:54:31.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.178+0000 7fc1116c4700 1 --2- 172.21.15.162:0/4093184993 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0f8062c90 0x7fc0f8065150 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 rx=0x7fc0fc0103c0 tx=0x7fc0fc00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:31.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.182+0000 7fc1027fc700 1 -- 172.21.15.162:0/4093184993 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc10804af60 con 0x7fc10c0f9b70 2024-02-13T00:54:31.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:29.487639+0000 osd.2 (osd.2) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:31.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:29.487966+0000 osd.2 (osd.2) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:31.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:29.983519+0000 mon.a (mon.0) 3764 : cluster [DBG] osdmap e805: 8 total, 8 up, 4 in 2024-02-13T00:54:31.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:29.487639+0000 osd.2 (osd.2) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:31.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:29.487966+0000 osd.2 (osd.2) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:31.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:29.983519+0000 mon.a (mon.0) 3764 : cluster [DBG] osdmap e805: 8 total, 8 up, 4 in 2024-02-13T00:54:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:29.487639+0000 osd.2 (osd.2) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:29.487966+0000 osd.2 (osd.2) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:31.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:29.983519+0000 mon.a (mon.0) 3764 : cluster [DBG] osdmap e805: 8 total, 8 up, 4 in 2024-02-13T00:54:31.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.348+0000 7fc113928700 1 -- 172.21.15.162:0/4093184993 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc10c06c6b0 con 0x7fc0f8062c90 2024-02-13T00:54:31.363 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.359+0000 7fc1027fc700 1 -- 172.21.15.162:0/4093184993 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+413671 (secure 0 0 0) 0x7fc10c06c6b0 con 0x7fc0f8062c90 2024-02-13T00:54:31.369 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.362+0000 7fc113928700 10 librados: watch_flush enter 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.362+0000 7fc113928700 10 librados: watch_flush exit 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.362+0000 7fc113928700 20 client.28206.objecter shutdown clearing up homeless session... 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.362+0000 7fc113928700 10 client.28206.objecter successfully canceled tick 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.362+0000 7fc113928700 1 -- 172.21.15.162:0/4093184993 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0f8062c90 msgr2=0x7fc0f8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.362+0000 7fc113928700 1 --2- 172.21.15.162:0/4093184993 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0f8062c90 0x7fc0f8065150 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 rx=0x7fc0fc0103c0 tx=0x7fc0fc00b040).stop 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.362+0000 7fc113928700 1 -- 172.21.15.162:0/4093184993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c0f9b70 msgr2=0x7fc10c06b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.362+0000 7fc113928700 1 --2- 172.21.15.162:0/4093184993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c0f9b70 0x7fc10c06b660 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7fc10800e970 tx=0x7fc10800b040).stop 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.363+0000 7fc113928700 1 -- 172.21.15.162:0/4093184993 shutdown_connections 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.363+0000 7fc113928700 1 --2- 172.21.15.162:0/4093184993 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc0f8062c90 0x7fc0f8065150 unknown :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.363+0000 7fc113928700 1 --2- 172.21.15.162:0/4093184993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc10c0fa440 0x7fc10c06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.363+0000 7fc113928700 1 --2- 172.21.15.162:0/4093184993 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc10c0f8500 0x7fc10c06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:31.373 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.363+0000 7fc113928700 1 --2- 172.21.15.162:0/4093184993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc10c0f9b70 0x7fc10c06b660 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:31.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.363+0000 7fc113928700 1 -- 172.21.15.162:0/4093184993 >> 172.21.15.162:0/4093184993 conn(0x7fc10c0f38c0 msgr2=0x7fc10c0fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:31.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.363+0000 7fc113928700 1 -- 172.21.15.162:0/4093184993 shutdown_connections 2024-02-13T00:54:31.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.363+0000 7fc113928700 1 -- 172.21.15.162:0/4093184993 wait complete. 2024-02-13T00:54:31.374 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:31.363+0000 7fc113928700 1 librados: shutdown 2024-02-13T00:54:31.374 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:54:32.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:30.556599+0000 mgr.y (mgr.14544) 672 : cluster [DBG] pgmap v1233: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:54:32.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:30.556599+0000 mgr.y (mgr.14544) 672 : cluster [DBG] pgmap v1233: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:54:32.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:30.556599+0000 mgr.y (mgr.14544) 672 : cluster [DBG] pgmap v1233: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:54:32.572 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":1233,"stamp":"2024-02-13T00:54:30.555915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":2497,"num_read_kb":2217,"num_write":913,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":835,"num_bytes_recovered":21036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1071,"ondisk_log_size":1071,"up":492,"acting":492,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1250868,"kb_used_data":25108,"kb_used_omap":0,"kb_used_meta":1225728,"kb_avail":373729740,"statfs":{"total":383980142592,"available":382699253760,"internally_reserved":0,"allocated":25710592,"data_stored":13300519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1255145472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001628"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":713,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978378+0000","last_change":"2024-02-13T00:51:41.586442+0000","last_active":"2024-02-13T00:54:28.978378+0000","last_peered":"2024-02-13T00:54:28.978378+0000","last_clean":"2024-02-13T00:54:28.978378+0000","last_became_active":"2024-02-13T00:51:41.585448+0000","last_became_peered":"2024-02-13T00:51:41.585448+0000","last_unstale":"2024-02-13T00:54:28.978378+0000","last_undegraded":"2024-02-13T00:54:28.978378+0000","last_fullsized":"2024-02-13T00:54:28.978378+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:32.100607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:50:32.100607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":635,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532465+0000","last_change":"2024-02-13T00:51:41.586812+0000","last_active":"2024-02-13T00:54:25.532465+0000","last_peered":"2024-02-13T00:54:25.532465+0000","last_clean":"2024-02-13T00:54:25.532465+0000","last_became_active":"2024-02-13T00:51:41.584445+0000","last_became_peered":"2024-02-13T00:51:41.584445+0000","last_unstale":"2024-02-13T00:54:25.532465+0000","last_undegraded":"2024-02-13T00:54:25.532465+0000","last_fullsized":"2024-02-13T00:54:25.532465+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:39.963625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:39.963625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":513,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.058806+0000","last_change":"2024-02-13T00:52:30.605704+0000","last_active":"2024-02-13T00:54:29.058806+0000","last_peered":"2024-02-13T00:54:29.058806+0000","last_clean":"2024-02-13T00:54:29.058806+0000","last_became_active":"2024-02-13T00:52:30.604888+0000","last_became_peered":"2024-02-13T00:52:30.604888+0000","last_unstale":"2024-02-13T00:54:29.058806+0000","last_undegraded":"2024-02-13T00:54:29.058806+0000","last_fullsized":"2024-02-13T00:54:29.058806+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:15.530059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:15.530059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":589,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.058753+0000","last_change":"2024-02-13T00:51:41.591561+0000","last_active":"2024-02-13T00:54:29.058753+0000","last_peered":"2024-02-13T00:54:29.058753+0000","last_clean":"2024-02-13T00:54:29.058753+0000","last_became_active":"2024-02-13T00:51:41.588863+0000","last_became_peered":"2024-02-13T00:51:41.588863+0000","last_unstale":"2024-02-13T00:54:29.058753+0000","last_undegraded":"2024-02-13T00:54:29.058753+0000","last_fullsized":"2024-02-13T00:54:29.058753+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:16.677431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:16.677431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":491,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533219+0000","last_change":"2024-02-13T00:52:30.601175+0000","last_active":"2024-02-13T00:54:25.533219+0000","last_peered":"2024-02-13T00:54:25.533219+0000","last_clean":"2024-02-13T00:54:25.533219+0000","last_became_active":"2024-02-13T00:52:30.600551+0000","last_became_peered":"2024-02-13T00:52:30.600551+0000","last_unstale":"2024-02-13T00:54:25.533219+0000","last_undegraded":"2024-02-13T00:54:25.533219+0000","last_fullsized":"2024-02-13T00:54:25.533219+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.469431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.469431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":88,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532614+0000","last_change":"2024-02-13T00:52:30.571356+0000","last_active":"2024-02-13T00:54:25.532614+0000","last_peered":"2024-02-13T00:54:25.532614+0000","last_clean":"2024-02-13T00:54:25.532614+0000","last_became_active":"2024-02-13T00:52:30.571021+0000","last_became_peered":"2024-02-13T00:52:30.571021+0000","last_unstale":"2024-02-13T00:54:25.532614+0000","last_undegraded":"2024-02-13T00:54:25.532614+0000","last_fullsized":"2024-02-13T00:54:25.532614+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:54.542201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:54.542201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":500,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.531731+0000","last_change":"2024-02-13T00:51:40.927523+0000","last_active":"2024-02-13T00:54:25.531731+0000","last_peered":"2024-02-13T00:54:25.531731+0000","last_clean":"2024-02-13T00:54:25.531731+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:54:25.531731+0000","last_undegraded":"2024-02-13T00:54:25.531731+0000","last_fullsized":"2024-02-13T00:54:25.531731+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:40.927454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:40.927454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":763,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533405+0000","last_change":"2024-02-13T00:51:41.571583+0000","last_active":"2024-02-13T00:54:25.533405+0000","last_peered":"2024-02-13T00:54:25.533405+0000","last_clean":"2024-02-13T00:54:25.533405+0000","last_became_active":"2024-02-13T00:51:41.570041+0000","last_became_peered":"2024-02-13T00:51:41.570041+0000","last_unstale":"2024-02-13T00:54:25.533405+0000","last_undegraded":"2024-02-13T00:54:25.533405+0000","last_fullsized":"2024-02-13T00:54:25.533405+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":482,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.534551+0000","last_change":"2024-02-13T00:49:48.070260+0000","last_active":"2024-02-13T00:54:25.534551+0000","last_peered":"2024-02-13T00:54:25.534551+0000","last_clean":"2024-02-13T00:54:25.534551+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:54:25.534551+0000","last_undegraded":"2024-02-13T00:54:25.534551+0000","last_fullsized":"2024-02-13T00:54:25.534551+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:48.070195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:49:48.070195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":580,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060162+0000","last_change":"2024-02-13T00:51:41.569866+0000","last_active":"2024-02-13T00:54:29.060162+0000","last_peered":"2024-02-13T00:54:29.060162+0000","last_clean":"2024-02-13T00:54:29.060162+0000","last_became_active":"2024-02-13T00:51:41.567986+0000","last_became_peered":"2024-02-13T00:51:41.567986+0000","last_unstale":"2024-02-13T00:54:29.060162+0000","last_undegraded":"2024-02-13T00:54:29.060162+0000","last_fullsized":"2024-02-13T00:54:29.060162+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:14.521241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:14.521241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":732,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978418+0000","last_change":"2024-02-13T00:52:30.593903+0000","last_active":"2024-02-13T00:54:28.978418+0000","last_peered":"2024-02-13T00:54:28.978418+0000","last_clean":"2024-02-13T00:54:28.978418+0000","last_became_active":"2024-02-13T00:52:30.593515+0000","last_became_peered":"2024-02-13T00:52:30.593515+0000","last_unstale":"2024-02-13T00:54:28.978418+0000","last_undegraded":"2024-02-13T00:54:28.978418+0000","last_fullsized":"2024-02-13T00:54:28.978418+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:49.069014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:49.069014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"728'22","reported_seq":697,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121466+0000","last_change":"2024-02-13T00:53:10.300922+0000","last_active":"2024-02-13T00:54:29.121466+0000","last_peered":"2024-02-13T00:54:29.121466+0000","last_clean":"2024-02-13T00:54:29.121466+0000","last_became_active":"2024-02-13T00:52:35.581849+0000","last_became_peered":"2024-02-13T00:52:35.581849+0000","last_unstale":"2024-02-13T00:54:29.121466+0000","last_undegraded":"2024-02-13T00:54:29.121466+0000","last_fullsized":"2024-02-13T00:54:29.121466+0000","mapping_epoch":727,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"728'22","last_scrub_stamp":"2024-02-13T00:53:10.300843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:53:10.300843+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":884,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532343+0000","last_change":"2024-02-13T00:53:48.410561+0000","last_active":"2024-02-13T00:54:25.532343+0000","last_peered":"2024-02-13T00:54:25.532343+0000","last_clean":"2024-02-13T00:54:25.532343+0000","last_became_active":"2024-02-13T00:52:30.577711+0000","last_became_peered":"2024-02-13T00:52:30.577711+0000","last_unstale":"2024-02-13T00:54:25.532343+0000","last_undegraded":"2024-02-13T00:54:25.532343+0000","last_fullsized":"2024-02-13T00:54:25.532343+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'48","last_scrub_stamp":"2024-02-13T00:53:48.410464+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:53:48.410464+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":800,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.534232+0000","last_change":"2024-02-13T00:51:41.571304+0000","last_active":"2024-02-13T00:54:25.534232+0000","last_peered":"2024-02-13T00:54:25.534232+0000","last_clean":"2024-02-13T00:54:25.534232+0000","last_became_active":"2024-02-13T00:51:41.569518+0000","last_became_peered":"2024-02-13T00:51:41.569518+0000","last_unstale":"2024-02-13T00:54:25.534232+0000","last_undegraded":"2024-02-13T00:54:25.534232+0000","last_fullsized":"2024-02-13T00:54:25.534232+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:23.379773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:23.379773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":702,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.531665+0000","last_change":"2024-02-13T00:53:57.460642+0000","last_active":"2024-02-13T00:54:25.531665+0000","last_peered":"2024-02-13T00:54:25.531665+0000","last_clean":"2024-02-13T00:54:25.531665+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:54:25.531665+0000","last_undegraded":"2024-02-13T00:54:25.531665+0000","last_fullsized":"2024-02-13T00:54:25.531665+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:57.460508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:53:57.460508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":749,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533123+0000","last_change":"2024-02-13T00:52:30.190029+0000","last_active":"2024-02-13T00:54:25.533123+0000","last_peered":"2024-02-13T00:54:25.533123+0000","last_clean":"2024-02-13T00:54:25.533123+0000","last_became_active":"2024-02-13T00:51:41.573640+0000","last_became_peered":"2024-02-13T00:51:41.573640+0000","last_unstale":"2024-02-13T00:54:25.533123+0000","last_undegraded":"2024-02-13T00:54:25.533123+0000","last_fullsized":"2024-02-13T00:54:25.533123+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:30.189925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:30.189925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":823,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979199+0000","last_change":"2024-02-13T00:52:31.041126+0000","last_active":"2024-02-13T00:54:28.979199+0000","last_peered":"2024-02-13T00:54:28.979199+0000","last_clean":"2024-02-13T00:54:28.979199+0000","last_became_active":"2024-02-13T00:52:30.591236+0000","last_became_peered":"2024-02-13T00:52:30.591236+0000","last_unstale":"2024-02-13T00:54:28.979199+0000","last_undegraded":"2024-02-13T00:54:28.979199+0000","last_fullsized":"2024-02-13T00:54:28.979199+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'21","last_scrub_stamp":"2024-02-13T00:52:28.988459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:28.988459+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":795,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060150+0000","last_change":"2024-02-13T00:53:49.468107+0000","last_active":"2024-02-13T00:54:29.060150+0000","last_peered":"2024-02-13T00:54:29.060150+0000","last_clean":"2024-02-13T00:54:29.060150+0000","last_became_active":"2024-02-13T00:52:30.603634+0000","last_became_peered":"2024-02-13T00:52:30.603634+0000","last_unstale":"2024-02-13T00:54:29.060150+0000","last_undegraded":"2024-02-13T00:54:29.060150+0000","last_fullsized":"2024-02-13T00:54:29.060150+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:49.468037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:53:49.468037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":666,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979060+0000","last_change":"2024-02-13T00:52:30.587164+0000","last_active":"2024-02-13T00:54:28.979060+0000","last_peered":"2024-02-13T00:54:28.979060+0000","last_clean":"2024-02-13T00:54:28.979060+0000","last_became_active":"2024-02-13T00:52:30.586877+0000","last_became_peered":"2024-02-13T00:52:30.586877+0000","last_unstale":"2024-02-13T00:54:28.979060+0000","last_undegraded":"2024-02-13T00:54:28.979060+0000","last_fullsized":"2024-02-13T00:54:28.979060+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.788783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.788783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":789,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978688+0000","last_change":"2024-02-13T00:51:41.586644+0000","last_active":"2024-02-13T00:54:28.978688+0000","last_peered":"2024-02-13T00:54:28.978688+0000","last_clean":"2024-02-13T00:54:28.978688+0000","last_became_active":"2024-02-13T00:51:41.585885+0000","last_became_peered":"2024-02-13T00:51:41.585885+0000","last_unstale":"2024-02-13T00:54:28.978688+0000","last_undegraded":"2024-02-13T00:54:28.978688+0000","last_fullsized":"2024-02-13T00:54:28.978688+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.837810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.837810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":415,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978654+0000","last_change":"2024-02-13T00:52:30.588176+0000","last_active":"2024-02-13T00:54:28.978654+0000","last_peered":"2024-02-13T00:54:28.978654+0000","last_clean":"2024-02-13T00:54:28.978654+0000","last_became_active":"2024-02-13T00:52:30.587089+0000","last_became_peered":"2024-02-13T00:52:30.587089+0000","last_unstale":"2024-02-13T00:54:28.978654+0000","last_undegraded":"2024-02-13T00:54:28.978654+0000","last_fullsized":"2024-02-13T00:54:28.978654+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:34.184091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:34.184091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":621,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533019+0000","last_change":"2024-02-13T00:51:43.825729+0000","last_active":"2024-02-13T00:54:25.533019+0000","last_peered":"2024-02-13T00:54:25.533019+0000","last_clean":"2024-02-13T00:54:25.533019+0000","last_became_active":"2024-02-13T00:51:41.573050+0000","last_became_peered":"2024-02-13T00:51:41.573050+0000","last_unstale":"2024-02-13T00:54:25.533019+0000","last_undegraded":"2024-02-13T00:54:25.533019+0000","last_fullsized":"2024-02-13T00:54:25.533019+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'51","last_scrub_stamp":"2024-02-13T00:51:33.100282+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:51:33.100282+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":88,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979241+0000","last_change":"2024-02-13T00:52:30.594905+0000","last_active":"2024-02-13T00:54:28.979241+0000","last_peered":"2024-02-13T00:54:28.979241+0000","last_clean":"2024-02-13T00:54:28.979241+0000","last_became_active":"2024-02-13T00:52:30.594543+0000","last_became_peered":"2024-02-13T00:52:30.594543+0000","last_unstale":"2024-02-13T00:54:28.979241+0000","last_undegraded":"2024-02-13T00:54:28.979241+0000","last_fullsized":"2024-02-13T00:54:28.979241+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:51.052139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:51:51.052139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":525,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978803+0000","last_change":"2024-02-13T00:51:41.587744+0000","last_active":"2024-02-13T00:54:28.978803+0000","last_peered":"2024-02-13T00:54:28.978803+0000","last_clean":"2024-02-13T00:54:28.978803+0000","last_became_active":"2024-02-13T00:51:41.582908+0000","last_became_peered":"2024-02-13T00:51:41.582908+0000","last_unstale":"2024-02-13T00:54:28.978803+0000","last_undegraded":"2024-02-13T00:54:28.978803+0000","last_fullsized":"2024-02-13T00:54:28.978803+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:14.244950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:51:14.244950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":627,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532428+0000","last_change":"2024-02-13T00:51:41.586157+0000","last_active":"2024-02-13T00:54:25.532428+0000","last_peered":"2024-02-13T00:54:25.532428+0000","last_clean":"2024-02-13T00:54:25.532428+0000","last_became_active":"2024-02-13T00:51:41.583542+0000","last_became_peered":"2024-02-13T00:51:41.583542+0000","last_unstale":"2024-02-13T00:54:25.532428+0000","last_undegraded":"2024-02-13T00:54:25.532428+0000","last_fullsized":"2024-02-13T00:54:25.532428+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:15.285536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:15.285536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":726,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059966+0000","last_change":"2024-02-13T00:51:35.505365+0000","last_active":"2024-02-13T00:54:29.059966+0000","last_peered":"2024-02-13T00:54:29.059966+0000","last_clean":"2024-02-13T00:54:29.059966+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:54:29.059966+0000","last_undegraded":"2024-02-13T00:54:29.059966+0000","last_fullsized":"2024-02-13T00:54:29.059966+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:35.505303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:35.505303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"467'31","reported_seq":590,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060058+0000","last_change":"2024-02-13T00:53:50.449803+0000","last_active":"2024-02-13T00:54:29.060058+0000","last_peered":"2024-02-13T00:54:29.060058+0000","last_clean":"2024-02-13T00:54:29.060058+0000","last_became_active":"2024-02-13T00:52:30.589869+0000","last_became_peered":"2024-02-13T00:52:30.589869+0000","last_unstale":"2024-02-13T00:54:29.060058+0000","last_undegraded":"2024-02-13T00:54:29.060058+0000","last_fullsized":"2024-02-13T00:54:29.060058+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'31","last_scrub_stamp":"2024-02-13T00:53:50.449723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:53:50.449723+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":599,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532915+0000","last_change":"2024-02-13T00:52:33.259627+0000","last_active":"2024-02-13T00:54:25.532915+0000","last_peered":"2024-02-13T00:54:25.532915+0000","last_clean":"2024-02-13T00:54:25.532915+0000","last_became_active":"2024-02-13T00:52:30.596010+0000","last_became_peered":"2024-02-13T00:52:30.596010+0000","last_unstale":"2024-02-13T00:54:25.532915+0000","last_undegraded":"2024-02-13T00:54:25.532915+0000","last_fullsized":"2024-02-13T00:54:25.532915+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:52:25.149943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:52:25.149943+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":793,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060095+0000","last_change":"2024-02-13T00:53:53.433619+0000","last_active":"2024-02-13T00:54:29.060095+0000","last_peered":"2024-02-13T00:54:29.060095+0000","last_clean":"2024-02-13T00:54:29.060095+0000","last_became_active":"2024-02-13T00:52:30.588539+0000","last_became_peered":"2024-02-13T00:52:30.588539+0000","last_unstale":"2024-02-13T00:54:29.060095+0000","last_undegraded":"2024-02-13T00:54:29.060095+0000","last_fullsized":"2024-02-13T00:54:29.060095+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:53.433519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_clean_scrub_stamp":"2024-02-13T00:53:53.433519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":776,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.534423+0000","last_change":"2024-02-13T00:52:30.600848+0000","last_active":"2024-02-13T00:54:25.534423+0000","last_peered":"2024-02-13T00:54:25.534423+0000","last_clean":"2024-02-13T00:54:25.534423+0000","last_became_active":"2024-02-13T00:52:30.600493+0000","last_became_peered":"2024-02-13T00:52:30.600493+0000","last_unstale":"2024-02-13T00:54:25.534423+0000","last_undegraded":"2024-02-13T00:54:25.534423+0000","last_fullsized":"2024-02-13T00:54:25.534423+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:38.996288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:38.996288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":441,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979325+0000","last_change":"2024-02-13T00:52:30.597329+0000","last_active":"2024-02-13T00:54:28.979325+0000","last_peered":"2024-02-13T00:54:28.979325+0000","last_clean":"2024-02-13T00:54:28.979325+0000","last_became_active":"2024-02-13T00:52:30.595363+0000","last_became_peered":"2024-02-13T00:52:30.595363+0000","last_unstale":"2024-02-13T00:54:28.979325+0000","last_undegraded":"2024-02-13T00:54:28.979325+0000","last_fullsized":"2024-02-13T00:54:28.979325+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.925604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.925604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":530,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533571+0000","last_change":"2024-02-13T00:52:32.337869+0000","last_active":"2024-02-13T00:54:25.533571+0000","last_peered":"2024-02-13T00:54:25.533571+0000","last_clean":"2024-02-13T00:54:25.533571+0000","last_became_active":"2024-02-13T00:52:30.584756+0000","last_became_peered":"2024-02-13T00:52:30.584756+0000","last_unstale":"2024-02-13T00:54:25.533571+0000","last_undegraded":"2024-02-13T00:54:25.533571+0000","last_fullsized":"2024-02-13T00:54:25.533571+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:49:03.294499+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:49:03.294499+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":799,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059287+0000","last_change":"2024-02-13T00:51:41.573970+0000","last_active":"2024-02-13T00:54:29.059287+0000","last_peered":"2024-02-13T00:54:29.059287+0000","last_clean":"2024-02-13T00:54:29.059287+0000","last_became_active":"2024-02-13T00:51:41.569525+0000","last_became_peered":"2024-02-13T00:51:41.569525+0000","last_unstale":"2024-02-13T00:54:29.059287+0000","last_undegraded":"2024-02-13T00:54:29.059287+0000","last_fullsized":"2024-02-13T00:54:29.059287+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:59.111828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:59.111828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"726'19","reported_seq":613,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.403392+0000","last_change":"2024-02-13T00:52:33.561456+0000","last_active":"2024-02-13T00:54:29.403392+0000","last_peered":"2024-02-13T00:54:29.403392+0000","last_clean":"2024-02-13T00:54:29.403392+0000","last_became_active":"2024-02-13T00:52:33.561139+0000","last_became_peered":"2024-02-13T00:52:33.561139+0000","last_unstale":"2024-02-13T00:54:29.403392+0000","last_undegraded":"2024-02-13T00:54:29.403392+0000","last_fullsized":"2024-02-13T00:54:29.403392+0000","mapping_epoch":725,"log_start":"404'11","ondisk_log_start":"404'11","created":60,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'16","last_scrub_stamp":"2024-02-13T00:52:22.988904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:52:22.988904+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":788,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060314+0000","last_change":"2024-02-13T00:52:31.529160+0000","last_active":"2024-02-13T00:54:29.060314+0000","last_peered":"2024-02-13T00:54:29.060314+0000","last_clean":"2024-02-13T00:54:29.060314+0000","last_became_active":"2024-02-13T00:52:30.602716+0000","last_became_peered":"2024-02-13T00:52:30.602716+0000","last_unstale":"2024-02-13T00:54:29.060314+0000","last_undegraded":"2024-02-13T00:54:29.060314+0000","last_fullsized":"2024-02-13T00:54:29.060314+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:51:05.644558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:05.644558+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":589,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060598+0000","last_change":"2024-02-13T00:52:30.605308+0000","last_active":"2024-02-13T00:54:29.060598+0000","last_peered":"2024-02-13T00:54:29.060598+0000","last_clean":"2024-02-13T00:54:29.060598+0000","last_became_active":"2024-02-13T00:52:30.604615+0000","last_became_peered":"2024-02-13T00:52:30.604615+0000","last_unstale":"2024-02-13T00:54:29.060598+0000","last_undegraded":"2024-02-13T00:54:29.060598+0000","last_fullsized":"2024-02-13T00:54:29.060598+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:26.830270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:26.830270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"467'27","reported_seq":929,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060518+0000","last_change":"2024-02-13T00:52:32.788019+0000","last_active":"2024-02-13T00:54:29.060518+0000","last_peered":"2024-02-13T00:54:29.060518+0000","last_clean":"2024-02-13T00:54:29.060518+0000","last_became_active":"2024-02-13T00:52:30.603369+0000","last_became_peered":"2024-02-13T00:52:30.603369+0000","last_unstale":"2024-02-13T00:54:29.060518+0000","last_undegraded":"2024-02-13T00:54:29.060518+0000","last_fullsized":"2024-02-13T00:54:29.060518+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:48:54.043088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.043088+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":835,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059067+0000","last_change":"2024-02-13T00:51:43.912749+0000","last_active":"2024-02-13T00:54:29.059067+0000","last_peered":"2024-02-13T00:54:29.059067+0000","last_clean":"2024-02-13T00:54:29.059067+0000","last_became_active":"2024-02-13T00:51:41.588865+0000","last_became_peered":"2024-02-13T00:51:41.588865+0000","last_unstale":"2024-02-13T00:54:29.059067+0000","last_undegraded":"2024-02-13T00:54:29.059067+0000","last_fullsized":"2024-02-13T00:54:29.059067+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:49:37.053398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.053398+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"727'19","reported_seq":623,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.403330+0000","last_change":"2024-02-13T00:53:08.072994+0000","last_active":"2024-02-13T00:54:29.403330+0000","last_peered":"2024-02-13T00:54:29.403330+0000","last_clean":"2024-02-13T00:54:29.403330+0000","last_became_active":"2024-02-13T00:52:34.566362+0000","last_became_peered":"2024-02-13T00:52:34.566362+0000","last_unstale":"2024-02-13T00:54:29.403330+0000","last_undegraded":"2024-02-13T00:54:29.403330+0000","last_fullsized":"2024-02-13T00:54:29.403330+0000","mapping_epoch":726,"log_start":"467'11","ondisk_log_start":"467'11","created":60,"last_epoch_clean":727,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'19","last_scrub_stamp":"2024-02-13T00:53:08.072916+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:53:08.072916+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":412,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059791+0000","last_change":"2024-02-13T00:50:19.734411+0000","last_active":"2024-02-13T00:54:29.059791+0000","last_peered":"2024-02-13T00:54:29.059791+0000","last_clean":"2024-02-13T00:54:29.059791+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:54:29.059791+0000","last_undegraded":"2024-02-13T00:54:29.059791+0000","last_fullsized":"2024-02-13T00:54:29.059791+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:19.734320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:50:19.734320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":723,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.980164+0000","last_change":"2024-02-13T00:51:11.239012+0000","last_active":"2024-02-13T00:54:28.980164+0000","last_peered":"2024-02-13T00:54:28.980164+0000","last_clean":"2024-02-13T00:54:28.980164+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:54:28.980164+0000","last_undegraded":"2024-02-13T00:54:28.980164+0000","last_fullsized":"2024-02-13T00:54:28.980164+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.238914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.238914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":531,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.534358+0000","last_change":"2024-02-13T00:52:32.853997+0000","last_active":"2024-02-13T00:54:25.534358+0000","last_peered":"2024-02-13T00:54:25.534358+0000","last_clean":"2024-02-13T00:54:25.534358+0000","last_became_active":"2024-02-13T00:52:30.593631+0000","last_became_peered":"2024-02-13T00:52:30.593631+0000","last_unstale":"2024-02-13T00:54:25.534358+0000","last_undegraded":"2024-02-13T00:54:25.534358+0000","last_fullsized":"2024-02-13T00:54:25.534358+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:57.386254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:57.386254+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":615,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978258+0000","last_change":"2024-02-13T00:51:53.090015+0000","last_active":"2024-02-13T00:54:28.978258+0000","last_peered":"2024-02-13T00:54:28.978258+0000","last_clean":"2024-02-13T00:54:28.978258+0000","last_became_active":"2024-02-13T00:51:41.582606+0000","last_became_peered":"2024-02-13T00:51:41.582606+0000","last_unstale":"2024-02-13T00:54:28.978258+0000","last_undegraded":"2024-02-13T00:54:28.978258+0000","last_fullsized":"2024-02-13T00:54:28.978258+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:53.089946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:51:53.089946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"728'19","reported_seq":819,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122486+0000","last_change":"2024-02-13T00:52:35.610840+0000","last_active":"2024-02-13T00:54:29.122486+0000","last_peered":"2024-02-13T00:54:29.122486+0000","last_clean":"2024-02-13T00:54:29.122486+0000","last_became_active":"2024-02-13T00:52:35.610249+0000","last_became_peered":"2024-02-13T00:52:35.610249+0000","last_unstale":"2024-02-13T00:54:29.122486+0000","last_undegraded":"2024-02-13T00:54:29.122486+0000","last_fullsized":"2024-02-13T00:54:29.122486+0000","mapping_epoch":727,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'16","last_scrub_stamp":"2024-02-13T00:49:45.051539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:49:45.051539+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":509,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533456+0000","last_change":"2024-02-13T00:51:41.570899+0000","last_active":"2024-02-13T00:54:25.533456+0000","last_peered":"2024-02-13T00:54:25.533456+0000","last_clean":"2024-02-13T00:54:25.533456+0000","last_became_active":"2024-02-13T00:51:41.568351+0000","last_became_peered":"2024-02-13T00:51:41.568351+0000","last_unstale":"2024-02-13T00:54:25.533456+0000","last_undegraded":"2024-02-13T00:54:25.533456+0000","last_fullsized":"2024-02-13T00:54:25.533456+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:31.463442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.463442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":698,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978082+0000","last_change":"2024-02-13T00:53:55.015887+0000","last_active":"2024-02-13T00:54:28.978082+0000","last_peered":"2024-02-13T00:54:28.978082+0000","last_clean":"2024-02-13T00:54:28.978082+0000","last_became_active":"2024-02-13T00:52:30.594189+0000","last_became_peered":"2024-02-13T00:52:30.594189+0000","last_unstale":"2024-02-13T00:54:28.978082+0000","last_undegraded":"2024-02-13T00:54:28.978082+0000","last_fullsized":"2024-02-13T00:54:28.978082+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:55.015820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:53:55.015820+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"726'11","reported_seq":868,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.403325+0000","last_change":"2024-02-13T00:53:12.200136+0000","last_active":"2024-02-13T00:54:29.403325+0000","last_peered":"2024-02-13T00:54:29.403325+0000","last_clean":"2024-02-13T00:54:29.403325+0000","last_became_active":"2024-02-13T00:52:30.582279+0000","last_became_peered":"2024-02-13T00:52:30.582279+0000","last_unstale":"2024-02-13T00:54:29.403325+0000","last_undegraded":"2024-02-13T00:54:29.403325+0000","last_fullsized":"2024-02-13T00:54:29.403325+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'11","last_scrub_stamp":"2024-02-13T00:53:12.200063+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:53:12.200063+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":799,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532609+0000","last_change":"2024-02-13T00:52:31.298166+0000","last_active":"2024-02-13T00:54:25.532609+0000","last_peered":"2024-02-13T00:54:25.532609+0000","last_clean":"2024-02-13T00:54:25.532609+0000","last_became_active":"2024-02-13T00:52:30.579027+0000","last_became_peered":"2024-02-13T00:52:30.579027+0000","last_unstale":"2024-02-13T00:54:25.532609+0000","last_undegraded":"2024-02-13T00:54:25.532609+0000","last_fullsized":"2024-02-13T00:54:25.532609+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'34","last_scrub_stamp":"2024-02-13T00:49:46.101874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.101874+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":509,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.534516+0000","last_change":"2024-02-13T00:51:41.572212+0000","last_active":"2024-02-13T00:54:25.534516+0000","last_peered":"2024-02-13T00:54:25.534516+0000","last_clean":"2024-02-13T00:54:25.534516+0000","last_became_active":"2024-02-13T00:51:41.569996+0000","last_became_peered":"2024-02-13T00:51:41.569996+0000","last_unstale":"2024-02-13T00:54:25.534516+0000","last_undegraded":"2024-02-13T00:54:25.534516+0000","last_fullsized":"2024-02-13T00:54:25.534516+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:47.075741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.075741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":509,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533199+0000","last_change":"2024-02-13T00:51:41.572697+0000","last_active":"2024-02-13T00:54:25.533199+0000","last_peered":"2024-02-13T00:54:25.533199+0000","last_clean":"2024-02-13T00:54:25.533199+0000","last_became_active":"2024-02-13T00:51:41.569841+0000","last_became_peered":"2024-02-13T00:51:41.569841+0000","last_unstale":"2024-02-13T00:54:25.533199+0000","last_undegraded":"2024-02-13T00:54:25.533199+0000","last_fullsized":"2024-02-13T00:54:25.533199+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:17.268189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:17.268189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":534,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978212+0000","last_change":"2024-02-13T00:51:41.585096+0000","last_active":"2024-02-13T00:54:28.978212+0000","last_peered":"2024-02-13T00:54:28.978212+0000","last_clean":"2024-02-13T00:54:28.978212+0000","last_became_active":"2024-02-13T00:51:41.582942+0000","last_became_peered":"2024-02-13T00:51:41.582942+0000","last_unstale":"2024-02-13T00:54:28.978212+0000","last_undegraded":"2024-02-13T00:54:28.978212+0000","last_fullsized":"2024-02-13T00:54:28.978212+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:36.841508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:36.841508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":589,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978367+0000","last_change":"2024-02-13T00:53:14.066845+0000","last_active":"2024-02-13T00:54:28.978367+0000","last_peered":"2024-02-13T00:54:28.978367+0000","last_clean":"2024-02-13T00:54:28.978367+0000","last_became_active":"2024-02-13T00:52:30.594613+0000","last_became_peered":"2024-02-13T00:52:30.594613+0000","last_unstale":"2024-02-13T00:54:28.978367+0000","last_undegraded":"2024-02-13T00:54:28.978367+0000","last_fullsized":"2024-02-13T00:54:28.978367+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:14.066781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:53:14.066781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"467'42","reported_seq":763,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978229+0000","last_change":"2024-02-13T00:53:19.085386+0000","last_active":"2024-02-13T00:54:28.978229+0000","last_peered":"2024-02-13T00:54:28.978229+0000","last_clean":"2024-02-13T00:54:28.978229+0000","last_became_active":"2024-02-13T00:51:41.582068+0000","last_became_peered":"2024-02-13T00:51:41.582068+0000","last_unstale":"2024-02-13T00:54:28.978229+0000","last_undegraded":"2024-02-13T00:54:28.978229+0000","last_fullsized":"2024-02-13T00:54:28.978229+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'42","last_scrub_stamp":"2024-02-13T00:53:19.085318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:53:19.085318+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":804,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060209+0000","last_change":"2024-02-13T00:52:30.582723+0000","last_active":"2024-02-13T00:54:29.060209+0000","last_peered":"2024-02-13T00:54:29.060209+0000","last_clean":"2024-02-13T00:54:29.060209+0000","last_became_active":"2024-02-13T00:52:30.582296+0000","last_became_peered":"2024-02-13T00:52:30.582296+0000","last_unstale":"2024-02-13T00:54:29.060209+0000","last_undegraded":"2024-02-13T00:54:29.060209+0000","last_fullsized":"2024-02-13T00:54:29.060209+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:29.412336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:52:29.412336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":748,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532838+0000","last_change":"2024-02-13T00:52:30.595054+0000","last_active":"2024-02-13T00:54:25.532838+0000","last_peered":"2024-02-13T00:54:25.532838+0000","last_clean":"2024-02-13T00:54:25.532838+0000","last_became_active":"2024-02-13T00:52:30.593405+0000","last_became_peered":"2024-02-13T00:52:30.593405+0000","last_unstale":"2024-02-13T00:54:25.532838+0000","last_undegraded":"2024-02-13T00:54:25.532838+0000","last_fullsized":"2024-02-13T00:54:25.532838+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":731,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533036+0000","last_change":"2024-02-13T00:52:30.586680+0000","last_active":"2024-02-13T00:54:25.533036+0000","last_peered":"2024-02-13T00:54:25.533036+0000","last_clean":"2024-02-13T00:54:25.533036+0000","last_became_active":"2024-02-13T00:52:30.586041+0000","last_became_peered":"2024-02-13T00:52:30.586041+0000","last_unstale":"2024-02-13T00:54:25.533036+0000","last_undegraded":"2024-02-13T00:54:25.533036+0000","last_fullsized":"2024-02-13T00:54:25.533036+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_clean_scrub_stamp":"2024-02-13T00:48:35.434645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":505,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533038+0000","last_change":"2024-02-13T00:53:09.344998+0000","last_active":"2024-02-13T00:54:25.533038+0000","last_peered":"2024-02-13T00:54:25.533038+0000","last_clean":"2024-02-13T00:54:25.533038+0000","last_became_active":"2024-02-13T00:51:41.573763+0000","last_became_peered":"2024-02-13T00:51:41.573763+0000","last_unstale":"2024-02-13T00:54:25.533038+0000","last_undegraded":"2024-02-13T00:54:25.533038+0000","last_fullsized":"2024-02-13T00:54:25.533038+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:09.344923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:53:09.344923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":924,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978943+0000","last_change":"2024-02-13T00:52:26.954476+0000","last_active":"2024-02-13T00:54:28.978943+0000","last_peered":"2024-02-13T00:54:28.978943+0000","last_clean":"2024-02-13T00:54:28.978943+0000","last_became_active":"2024-02-13T00:51:41.587256+0000","last_became_peered":"2024-02-13T00:51:41.587256+0000","last_unstale":"2024-02-13T00:54:28.978943+0000","last_undegraded":"2024-02-13T00:54:28.978943+0000","last_fullsized":"2024-02-13T00:54:28.978943+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'44","last_scrub_stamp":"2024-02-13T00:52:26.954384+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:52:26.954384+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":624,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979258+0000","last_change":"2024-02-13T00:52:30.596233+0000","last_active":"2024-02-13T00:54:28.979258+0000","last_peered":"2024-02-13T00:54:28.979258+0000","last_clean":"2024-02-13T00:54:28.979258+0000","last_became_active":"2024-02-13T00:52:30.595886+0000","last_became_peered":"2024-02-13T00:52:30.595886+0000","last_unstale":"2024-02-13T00:54:28.979258+0000","last_undegraded":"2024-02-13T00:54:28.979258+0000","last_fullsized":"2024-02-13T00:54:28.979258+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:25.001219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:25.001219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":811,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059068+0000","last_change":"2024-02-13T00:51:41.589718+0000","last_active":"2024-02-13T00:54:29.059068+0000","last_peered":"2024-02-13T00:54:29.059068+0000","last_clean":"2024-02-13T00:54:29.059068+0000","last_became_active":"2024-02-13T00:51:41.589295+0000","last_became_peered":"2024-02-13T00:51:41.589295+0000","last_unstale":"2024-02-13T00:54:29.059068+0000","last_undegraded":"2024-02-13T00:54:29.059068+0000","last_fullsized":"2024-02-13T00:54:29.059068+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.726660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.726660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":429,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060020+0000","last_change":"2024-02-13T00:54:23.237671+0000","last_active":"2024-02-13T00:54:29.060020+0000","last_peered":"2024-02-13T00:54:29.060020+0000","last_clean":"2024-02-13T00:54:29.060020+0000","last_became_active":"2024-02-13T00:52:30.582272+0000","last_became_peered":"2024-02-13T00:52:30.582272+0000","last_unstale":"2024-02-13T00:54:29.060020+0000","last_undegraded":"2024-02-13T00:54:29.060020+0000","last_fullsized":"2024-02-13T00:54:29.060020+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:54:23.237607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:54:23.237607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":444,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.061035+0000","last_change":"2024-02-13T00:53:14.241706+0000","last_active":"2024-02-13T00:54:29.061035+0000","last_peered":"2024-02-13T00:54:29.061035+0000","last_clean":"2024-02-13T00:54:29.061035+0000","last_became_active":"2024-02-13T00:52:30.604937+0000","last_became_peered":"2024-02-13T00:52:30.604937+0000","last_unstale":"2024-02-13T00:54:29.061035+0000","last_undegraded":"2024-02-13T00:54:29.061035+0000","last_fullsized":"2024-02-13T00:54:29.061035+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:14.241642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:53:14.241642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":913,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979972+0000","last_change":"2024-02-13T00:52:23.951273+0000","last_active":"2024-02-13T00:54:28.979972+0000","last_peered":"2024-02-13T00:54:28.979972+0000","last_clean":"2024-02-13T00:54:28.979972+0000","last_became_active":"2024-02-13T00:51:41.588905+0000","last_became_peered":"2024-02-13T00:51:41.588905+0000","last_unstale":"2024-02-13T00:54:28.979972+0000","last_undegraded":"2024-02-13T00:54:28.979972+0000","last_fullsized":"2024-02-13T00:54:28.979972+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:52:23.951206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:23.951206+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":512,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059053+0000","last_change":"2024-02-13T00:51:53.574669+0000","last_active":"2024-02-13T00:54:29.059053+0000","last_peered":"2024-02-13T00:54:29.059053+0000","last_clean":"2024-02-13T00:54:29.059053+0000","last_became_active":"2024-02-13T00:51:41.588531+0000","last_became_peered":"2024-02-13T00:51:41.588531+0000","last_unstale":"2024-02-13T00:54:29.059053+0000","last_undegraded":"2024-02-13T00:54:29.059053+0000","last_fullsized":"2024-02-13T00:54:29.059053+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:53.574605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:53.574605+0000","last_clean_scrub_stamp":"2024-02-13T00:51:53.574605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":800,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978469+0000","last_change":"2024-02-13T00:52:30.596032+0000","last_active":"2024-02-13T00:54:28.978469+0000","last_peered":"2024-02-13T00:54:28.978469+0000","last_clean":"2024-02-13T00:54:28.978469+0000","last_became_active":"2024-02-13T00:52:30.595531+0000","last_became_peered":"2024-02-13T00:52:30.595531+0000","last_unstale":"2024-02-13T00:54:28.978469+0000","last_undegraded":"2024-02-13T00:54:28.978469+0000","last_fullsized":"2024-02-13T00:54:28.978469+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:46.119431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:51:46.119431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":802,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060144+0000","last_change":"2024-02-13T00:52:30.589500+0000","last_active":"2024-02-13T00:54:29.060144+0000","last_peered":"2024-02-13T00:54:29.060144+0000","last_clean":"2024-02-13T00:54:29.060144+0000","last_became_active":"2024-02-13T00:52:30.588991+0000","last_became_peered":"2024-02-13T00:52:30.588991+0000","last_unstale":"2024-02-13T00:54:29.060144+0000","last_undegraded":"2024-02-13T00:54:29.060144+0000","last_fullsized":"2024-02-13T00:54:29.060144+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:16.535961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:51:16.535961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":651,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.534204+0000","last_change":"2024-02-13T00:52:30.603205+0000","last_active":"2024-02-13T00:54:25.534204+0000","last_peered":"2024-02-13T00:54:25.534204+0000","last_clean":"2024-02-13T00:54:25.534204+0000","last_became_active":"2024-02-13T00:52:30.600547+0000","last_became_peered":"2024-02-13T00:52:30.600547+0000","last_unstale":"2024-02-13T00:54:25.534204+0000","last_undegraded":"2024-02-13T00:54:25.534204+0000","last_fullsized":"2024-02-13T00:54:25.534204+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:23.124083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:23.124083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":456,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979044+0000","last_change":"2024-02-13T00:52:30.017699+0000","last_active":"2024-02-13T00:54:28.979044+0000","last_peered":"2024-02-13T00:54:28.979044+0000","last_clean":"2024-02-13T00:54:28.979044+0000","last_became_active":"2024-02-13T00:51:41.580760+0000","last_became_peered":"2024-02-13T00:51:41.580760+0000","last_unstale":"2024-02-13T00:54:28.979044+0000","last_undegraded":"2024-02-13T00:54:28.979044+0000","last_fullsized":"2024-02-13T00:54:28.979044+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:30.017618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:52:30.017618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":596,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.534286+0000","last_change":"2024-02-13T00:53:22.462609+0000","last_active":"2024-02-13T00:54:25.534286+0000","last_peered":"2024-02-13T00:54:25.534286+0000","last_clean":"2024-02-13T00:54:25.534286+0000","last_became_active":"2024-02-13T00:52:30.596789+0000","last_became_peered":"2024-02-13T00:52:30.596789+0000","last_unstale":"2024-02-13T00:54:25.534286+0000","last_undegraded":"2024-02-13T00:54:25.534286+0000","last_fullsized":"2024-02-13T00:54:25.534286+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":727,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'33","last_scrub_stamp":"2024-02-13T00:53:22.462526+0000","last_deep_scrub":"467'33","last_deep_scrub_stamp":"2024-02-13T00:53:22.462526+0000","last_clean_scrub_stamp":"2024-02-13T00:53:22.462526+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":756,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978165+0000","last_change":"2024-02-13T00:52:36.042288+0000","last_active":"2024-02-13T00:54:28.978165+0000","last_peered":"2024-02-13T00:54:28.978165+0000","last_clean":"2024-02-13T00:54:28.978165+0000","last_became_active":"2024-02-13T00:51:41.582126+0000","last_became_peered":"2024-02-13T00:51:41.582126+0000","last_unstale":"2024-02-13T00:54:28.978165+0000","last_undegraded":"2024-02-13T00:54:28.978165+0000","last_fullsized":"2024-02-13T00:54:28.978165+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.042221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.042221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":753,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532914+0000","last_change":"2024-02-13T00:53:56.472636+0000","last_active":"2024-02-13T00:54:25.532914+0000","last_peered":"2024-02-13T00:54:25.532914+0000","last_clean":"2024-02-13T00:54:25.532914+0000","last_became_active":"2024-02-13T00:51:41.571758+0000","last_became_peered":"2024-02-13T00:51:41.571758+0000","last_unstale":"2024-02-13T00:54:25.532914+0000","last_undegraded":"2024-02-13T00:54:25.532914+0000","last_fullsized":"2024-02-13T00:54:25.532914+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:56.472560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:53:56.472560+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":621,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532801+0000","last_change":"2024-02-13T00:52:30.580452+0000","last_active":"2024-02-13T00:54:25.532801+0000","last_peered":"2024-02-13T00:54:25.532801+0000","last_clean":"2024-02-13T00:54:25.532801+0000","last_became_active":"2024-02-13T00:52:30.580091+0000","last_became_peered":"2024-02-13T00:52:30.580091+0000","last_unstale":"2024-02-13T00:54:25.532801+0000","last_undegraded":"2024-02-13T00:54:25.532801+0000","last_fullsized":"2024-02-13T00:54:25.532801+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:25.359548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:48:25.359548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":659,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060299+0000","last_change":"2024-02-13T00:52:30.604915+0000","last_active":"2024-02-13T00:54:29.060299+0000","last_peered":"2024-02-13T00:54:29.060299+0000","last_clean":"2024-02-13T00:54:29.060299+0000","last_became_active":"2024-02-13T00:52:30.603107+0000","last_became_peered":"2024-02-13T00:52:30.603107+0000","last_unstale":"2024-02-13T00:54:29.060299+0000","last_undegraded":"2024-02-13T00:54:29.060299+0000","last_fullsized":"2024-02-13T00:54:29.060299+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:19.521319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:51:19.521319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":741,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979833+0000","last_change":"2024-02-13T00:52:30.596982+0000","last_active":"2024-02-13T00:54:28.979833+0000","last_peered":"2024-02-13T00:54:28.979833+0000","last_clean":"2024-02-13T00:54:28.979833+0000","last_became_active":"2024-02-13T00:52:30.596402+0000","last_became_peered":"2024-02-13T00:52:30.596402+0000","last_unstale":"2024-02-13T00:54:28.979833+0000","last_undegraded":"2024-02-13T00:54:28.979833+0000","last_fullsized":"2024-02-13T00:54:28.979833+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:42.854585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:42.854585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":490,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059930+0000","last_change":"2024-02-13T00:54:25.247133+0000","last_active":"2024-02-13T00:54:29.059930+0000","last_peered":"2024-02-13T00:54:29.059930+0000","last_clean":"2024-02-13T00:54:29.059930+0000","last_became_active":"2024-02-13T00:52:30.581048+0000","last_became_peered":"2024-02-13T00:52:30.581048+0000","last_unstale":"2024-02-13T00:54:29.059930+0000","last_undegraded":"2024-02-13T00:54:29.059930+0000","last_fullsized":"2024-02-13T00:54:29.059930+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:54:25.247064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:54:25.247064+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"467'23","reported_seq":855,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060679+0000","last_change":"2024-02-13T00:51:42.290298+0000","last_active":"2024-02-13T00:54:29.060679+0000","last_peered":"2024-02-13T00:54:29.060679+0000","last_clean":"2024-02-13T00:54:29.060679+0000","last_became_active":"2024-02-13T00:51:41.569273+0000","last_became_peered":"2024-02-13T00:51:41.569273+0000","last_unstale":"2024-02-13T00:54:29.060679+0000","last_undegraded":"2024-02-13T00:54:29.060679+0000","last_fullsized":"2024-02-13T00:54:29.060679+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'23","last_scrub_stamp":"2024-02-13T00:49:07.192902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:07.192902+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":697,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978444+0000","last_change":"2024-02-13T00:49:46.851558+0000","last_active":"2024-02-13T00:54:28.978444+0000","last_peered":"2024-02-13T00:54:28.978444+0000","last_clean":"2024-02-13T00:54:28.978444+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:54:28.978444+0000","last_undegraded":"2024-02-13T00:54:28.978444+0000","last_fullsized":"2024-02-13T00:54:28.978444+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.851486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.851486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":1811,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.393327+0000","last_change":"2024-02-13T00:51:38.527164+0000","last_active":"2024-02-13T00:54:29.393327+0000","last_peered":"2024-02-13T00:54:29.393327+0000","last_clean":"2024-02-13T00:54:29.393327+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:54:29.393327+0000","last_undegraded":"2024-02-13T00:54:29.393327+0000","last_fullsized":"2024-02-13T00:54:29.393327+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:51:38.527084+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:51:38.527084+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":995,"num_read_kb":990,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":513,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533691+0000","last_change":"2024-02-13T00:51:41.571352+0000","last_active":"2024-02-13T00:54:25.533691+0000","last_peered":"2024-02-13T00:54:25.533691+0000","last_clean":"2024-02-13T00:54:25.533691+0000","last_became_active":"2024-02-13T00:51:41.570686+0000","last_became_peered":"2024-02-13T00:51:41.570686+0000","last_unstale":"2024-02-13T00:54:25.533691+0000","last_undegraded":"2024-02-13T00:54:25.533691+0000","last_fullsized":"2024-02-13T00:54:25.533691+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":584,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059897+0000","last_change":"2024-02-13T00:52:30.604046+0000","last_active":"2024-02-13T00:54:29.059897+0000","last_peered":"2024-02-13T00:54:29.059897+0000","last_clean":"2024-02-13T00:54:29.059897+0000","last_became_active":"2024-02-13T00:52:30.603400+0000","last_became_peered":"2024-02-13T00:52:30.603400+0000","last_unstale":"2024-02-13T00:54:29.059897+0000","last_undegraded":"2024-02-13T00:54:29.059897+0000","last_fullsized":"2024-02-13T00:54:29.059897+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:37.545088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:37.545088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":88,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978918+0000","last_change":"2024-02-13T00:52:30.575376+0000","last_active":"2024-02-13T00:54:28.978918+0000","last_peered":"2024-02-13T00:54:28.978918+0000","last_clean":"2024-02-13T00:54:28.978918+0000","last_became_active":"2024-02-13T00:52:30.575153+0000","last_became_peered":"2024-02-13T00:52:30.575153+0000","last_unstale":"2024-02-13T00:54:28.978918+0000","last_undegraded":"2024-02-13T00:54:28.978918+0000","last_fullsized":"2024-02-13T00:54:28.978918+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:26.113119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:52:26.113119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"701'67","reported_seq":163,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.403448+0000","last_change":"2024-02-13T00:52:09.459680+0000","last_active":"2024-02-13T00:54:29.403448+0000","last_peered":"2024-02-13T00:54:29.403448+0000","last_clean":"2024-02-13T00:54:29.403448+0000","last_became_active":"2024-02-13T00:52:07.565910+0000","last_became_peered":"2024-02-13T00:52:07.565910+0000","last_unstale":"2024-02-13T00:54:29.403448+0000","last_undegraded":"2024-02-13T00:54:29.403448+0000","last_fullsized":"2024-02-13T00:54:29.403448+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'65","last_scrub_stamp":"2024-02-13T00:49:47.928548+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.928548+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":77,"num_read_kb":61,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":2864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":711,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059203+0000","last_change":"2024-02-13T00:50:37.777963+0000","last_active":"2024-02-13T00:54:29.059203+0000","last_peered":"2024-02-13T00:54:29.059203+0000","last_clean":"2024-02-13T00:54:29.059203+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:54:29.059203+0000","last_undegraded":"2024-02-13T00:54:29.059203+0000","last_fullsized":"2024-02-13T00:54:29.059203+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:37.777891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:37.777891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":340,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979808+0000","last_change":"2024-02-13T00:52:30.598078+0000","last_active":"2024-02-13T00:54:28.979808+0000","last_peered":"2024-02-13T00:54:28.979808+0000","last_clean":"2024-02-13T00:54:28.979808+0000","last_became_active":"2024-02-13T00:52:30.595326+0000","last_became_peered":"2024-02-13T00:52:30.595326+0000","last_unstale":"2024-02-13T00:54:28.979808+0000","last_undegraded":"2024-02-13T00:54:28.979808+0000","last_fullsized":"2024-02-13T00:54:28.979808+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":491,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533110+0000","last_change":"2024-02-13T00:48:17.340891+0000","last_active":"2024-02-13T00:54:25.533110+0000","last_peered":"2024-02-13T00:54:25.533110+0000","last_clean":"2024-02-13T00:54:25.533110+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:54:25.533110+0000","last_undegraded":"2024-02-13T00:54:25.533110+0000","last_fullsized":"2024-02-13T00:54:25.533110+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:17.340797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:17.340797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":1024,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.403449+0000","last_change":"2024-02-13T00:53:50.046537+0000","last_active":"2024-02-13T00:54:29.403449+0000","last_peered":"2024-02-13T00:54:29.403449+0000","last_clean":"2024-02-13T00:54:29.403449+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:54:29.403449+0000","last_undegraded":"2024-02-13T00:54:29.403449+0000","last_fullsized":"2024-02-13T00:54:29.403449+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14","last_scrub_stamp":"2024-02-13T00:53:50.046455+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:53:50.046455+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":557,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059435+0000","last_change":"2024-02-13T00:53:47.479840+0000","last_active":"2024-02-13T00:54:29.059435+0000","last_peered":"2024-02-13T00:54:29.059435+0000","last_clean":"2024-02-13T00:54:29.059435+0000","last_became_active":"2024-02-13T00:52:30.605408+0000","last_became_peered":"2024-02-13T00:52:30.605408+0000","last_unstale":"2024-02-13T00:54:29.059435+0000","last_undegraded":"2024-02-13T00:54:29.059435+0000","last_fullsized":"2024-02-13T00:54:29.059435+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:53:47.479733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:53:47.479733+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":786,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059910+0000","last_change":"2024-02-13T00:51:41.570719+0000","last_active":"2024-02-13T00:54:29.059910+0000","last_peered":"2024-02-13T00:54:29.059910+0000","last_clean":"2024-02-13T00:54:29.059910+0000","last_became_active":"2024-02-13T00:51:41.567277+0000","last_became_peered":"2024-02-13T00:51:41.567277+0000","last_unstale":"2024-02-13T00:54:29.059910+0000","last_undegraded":"2024-02-13T00:54:29.059910+0000","last_fullsized":"2024-02-13T00:54:29.059910+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:33.040516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:49:33.040516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":798,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978115+0000","last_change":"2024-02-13T00:52:30.572639+0000","last_active":"2024-02-13T00:54:28.978115+0000","last_peered":"2024-02-13T00:54:28.978115+0000","last_clean":"2024-02-13T00:54:28.978115+0000","last_became_active":"2024-02-13T00:52:30.572144+0000","last_became_peered":"2024-02-13T00:52:30.572144+0000","last_unstale":"2024-02-13T00:54:28.978115+0000","last_undegraded":"2024-02-13T00:54:28.978115+0000","last_fullsized":"2024-02-13T00:54:28.978115+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:38.051468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:38.051468+0000","last_clean_scrub_stamp":"2024-02-13T00:50:38.051468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":735,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.058957+0000","last_change":"2024-02-13T00:51:41.589341+0000","last_active":"2024-02-13T00:54:29.058957+0000","last_peered":"2024-02-13T00:54:29.058957+0000","last_clean":"2024-02-13T00:54:29.058957+0000","last_became_active":"2024-02-13T00:51:41.588483+0000","last_became_peered":"2024-02-13T00:51:41.588483+0000","last_unstale":"2024-02-13T00:54:29.058957+0000","last_undegraded":"2024-02-13T00:54:29.058957+0000","last_fullsized":"2024-02-13T00:54:29.058957+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:08.606262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:51:08.606262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":516,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978337+0000","last_change":"2024-02-13T00:52:30.572465+0000","last_active":"2024-02-13T00:54:28.978337+0000","last_peered":"2024-02-13T00:54:28.978337+0000","last_clean":"2024-02-13T00:54:28.978337+0000","last_became_active":"2024-02-13T00:52:30.571718+0000","last_became_peered":"2024-02-13T00:52:30.571718+0000","last_unstale":"2024-02-13T00:54:28.978337+0000","last_undegraded":"2024-02-13T00:54:28.978337+0000","last_fullsized":"2024-02-13T00:54:28.978337+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:48.089417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:51:48.089417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":872,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.058961+0000","last_change":"2024-02-13T00:52:32.017951+0000","last_active":"2024-02-13T00:54:29.058961+0000","last_peered":"2024-02-13T00:54:29.058961+0000","last_clean":"2024-02-13T00:54:29.058961+0000","last_became_active":"2024-02-13T00:52:30.581394+0000","last_became_peered":"2024-02-13T00:52:30.581394+0000","last_unstale":"2024-02-13T00:54:29.058961+0000","last_undegraded":"2024-02-13T00:54:29.058961+0000","last_fullsized":"2024-02-13T00:54:29.058961+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'30","last_scrub_stamp":"2024-02-13T00:49:31.072862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:31.072862+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":813,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979772+0000","last_change":"2024-02-13T00:54:00.982156+0000","last_active":"2024-02-13T00:54:28.979772+0000","last_peered":"2024-02-13T00:54:28.979772+0000","last_clean":"2024-02-13T00:54:28.979772+0000","last_became_active":"2024-02-13T00:52:30.595453+0000","last_became_peered":"2024-02-13T00:52:30.595453+0000","last_unstale":"2024-02-13T00:54:28.979772+0000","last_undegraded":"2024-02-13T00:54:28.979772+0000","last_fullsized":"2024-02-13T00:54:28.979772+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:54:00.982074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:54:00.982074+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":588,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059739+0000","last_change":"2024-02-13T00:53:09.190093+0000","last_active":"2024-02-13T00:54:29.059739+0000","last_peered":"2024-02-13T00:54:29.059739+0000","last_clean":"2024-02-13T00:54:29.059739+0000","last_became_active":"2024-02-13T00:51:41.568308+0000","last_became_peered":"2024-02-13T00:51:41.568308+0000","last_unstale":"2024-02-13T00:54:29.059739+0000","last_undegraded":"2024-02-13T00:54:29.059739+0000","last_fullsized":"2024-02-13T00:54:29.059739+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:09.190024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:53:09.190024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":741,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.980018+0000","last_change":"2024-02-13T00:52:30.598253+0000","last_active":"2024-02-13T00:54:28.980018+0000","last_peered":"2024-02-13T00:54:28.980018+0000","last_clean":"2024-02-13T00:54:28.980018+0000","last_became_active":"2024-02-13T00:52:30.597320+0000","last_became_peered":"2024-02-13T00:52:30.597320+0000","last_unstale":"2024-02-13T00:54:28.980018+0000","last_undegraded":"2024-02-13T00:54:28.980018+0000","last_fullsized":"2024-02-13T00:54:28.980018+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.880001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.880001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"467'27","reported_seq":934,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.980043+0000","last_change":"2024-02-13T00:51:42.198644+0000","last_active":"2024-02-13T00:54:28.980043+0000","last_peered":"2024-02-13T00:54:28.980043+0000","last_clean":"2024-02-13T00:54:28.980043+0000","last_became_active":"2024-02-13T00:51:41.580479+0000","last_became_peered":"2024-02-13T00:51:41.580479+0000","last_unstale":"2024-02-13T00:54:28.980043+0000","last_undegraded":"2024-02-13T00:54:28.980043+0000","last_fullsized":"2024-02-13T00:54:28.980043+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:35.834669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.834669+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":493,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059030+0000","last_change":"2024-02-13T00:52:30.581921+0000","last_active":"2024-02-13T00:54:29.059030+0000","last_peered":"2024-02-13T00:54:29.059030+0000","last_clean":"2024-02-13T00:54:29.059030+0000","last_became_active":"2024-02-13T00:52:30.581023+0000","last_became_peered":"2024-02-13T00:52:30.581023+0000","last_unstale":"2024-02-13T00:54:29.059030+0000","last_undegraded":"2024-02-13T00:54:29.059030+0000","last_fullsized":"2024-02-13T00:54:29.059030+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:48.521656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:51:48.521656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":514,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979189+0000","last_change":"2024-02-13T00:52:30.595830+0000","last_active":"2024-02-13T00:54:28.979189+0000","last_peered":"2024-02-13T00:54:28.979189+0000","last_clean":"2024-02-13T00:54:28.979189+0000","last_became_active":"2024-02-13T00:52:30.594958+0000","last_became_peered":"2024-02-13T00:52:30.594958+0000","last_unstale":"2024-02-13T00:54:28.979189+0000","last_undegraded":"2024-02-13T00:54:28.979189+0000","last_fullsized":"2024-02-13T00:54:28.979189+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:36.152699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:36.152699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":500,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.534462+0000","last_change":"2024-02-13T00:52:30.601489+0000","last_active":"2024-02-13T00:54:25.534462+0000","last_peered":"2024-02-13T00:54:25.534462+0000","last_clean":"2024-02-13T00:54:25.534462+0000","last_became_active":"2024-02-13T00:52:30.600883+0000","last_became_peered":"2024-02-13T00:52:30.600883+0000","last_unstale":"2024-02-13T00:54:25.534462+0000","last_undegraded":"2024-02-13T00:54:25.534462+0000","last_fullsized":"2024-02-13T00:54:25.534462+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:07.304576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:07.304576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":672,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060904+0000","last_change":"2024-02-13T00:51:52.536631+0000","last_active":"2024-02-13T00:54:29.060904+0000","last_peered":"2024-02-13T00:54:29.060904+0000","last_clean":"2024-02-13T00:54:29.060904+0000","last_became_active":"2024-02-13T00:51:41.571280+0000","last_became_peered":"2024-02-13T00:51:41.571280+0000","last_unstale":"2024-02-13T00:54:29.060904+0000","last_undegraded":"2024-02-13T00:54:29.060904+0000","last_fullsized":"2024-02-13T00:54:29.060904+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'18","last_scrub_stamp":"2024-02-13T00:51:52.536524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:51:52.536524+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":498,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532909+0000","last_change":"2024-02-13T00:52:30.575760+0000","last_active":"2024-02-13T00:54:25.532909+0000","last_peered":"2024-02-13T00:54:25.532909+0000","last_clean":"2024-02-13T00:54:25.532909+0000","last_became_active":"2024-02-13T00:52:30.575381+0000","last_became_peered":"2024-02-13T00:52:30.575381+0000","last_unstale":"2024-02-13T00:54:25.532909+0000","last_undegraded":"2024-02-13T00:54:25.532909+0000","last_fullsized":"2024-02-13T00:54:25.532909+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:20.093279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:52:20.093279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":518,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533695+0000","last_change":"2024-02-13T00:52:30.597684+0000","last_active":"2024-02-13T00:54:25.533695+0000","last_peered":"2024-02-13T00:54:25.533695+0000","last_clean":"2024-02-13T00:54:25.533695+0000","last_became_active":"2024-02-13T00:52:30.596322+0000","last_became_peered":"2024-02-13T00:52:30.596322+0000","last_unstale":"2024-02-13T00:54:25.533695+0000","last_undegraded":"2024-02-13T00:54:25.533695+0000","last_fullsized":"2024-02-13T00:54:25.533695+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:40.384428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:40.384428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":591,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979488+0000","last_change":"2024-02-13T00:52:30.597677+0000","last_active":"2024-02-13T00:54:28.979488+0000","last_peered":"2024-02-13T00:54:28.979488+0000","last_clean":"2024-02-13T00:54:28.979488+0000","last_became_active":"2024-02-13T00:52:30.595811+0000","last_became_peered":"2024-02-13T00:52:30.595811+0000","last_unstale":"2024-02-13T00:54:28.979488+0000","last_undegraded":"2024-02-13T00:54:28.979488+0000","last_fullsized":"2024-02-13T00:54:28.979488+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:19.286611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:19.286611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":756,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060039+0000","last_change":"2024-02-13T00:51:43.571860+0000","last_active":"2024-02-13T00:54:29.060039+0000","last_peered":"2024-02-13T00:54:29.060039+0000","last_clean":"2024-02-13T00:54:29.060039+0000","last_became_active":"2024-02-13T00:51:41.569584+0000","last_became_peered":"2024-02-13T00:51:41.569584+0000","last_unstale":"2024-02-13T00:54:29.060039+0000","last_undegraded":"2024-02-13T00:54:29.060039+0000","last_fullsized":"2024-02-13T00:54:29.060039+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'39","last_scrub_stamp":"2024-02-13T00:49:06.154931+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:49:06.154931+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":761,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979426+0000","last_change":"2024-02-13T00:52:30.595096+0000","last_active":"2024-02-13T00:54:28.979426+0000","last_peered":"2024-02-13T00:54:28.979426+0000","last_clean":"2024-02-13T00:54:28.979426+0000","last_became_active":"2024-02-13T00:52:30.594229+0000","last_became_peered":"2024-02-13T00:52:30.594229+0000","last_unstale":"2024-02-13T00:54:28.979426+0000","last_undegraded":"2024-02-13T00:54:28.979426+0000","last_fullsized":"2024-02-13T00:54:28.979426+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.916152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"726'10","reported_seq":555,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.403488+0000","last_change":"2024-02-13T00:53:19.374855+0000","last_active":"2024-02-13T00:54:29.403488+0000","last_peered":"2024-02-13T00:54:29.403488+0000","last_clean":"2024-02-13T00:54:29.403488+0000","last_became_active":"2024-02-13T00:52:30.603380+0000","last_became_peered":"2024-02-13T00:52:30.603380+0000","last_unstale":"2024-02-13T00:54:29.403488+0000","last_undegraded":"2024-02-13T00:54:29.403488+0000","last_fullsized":"2024-02-13T00:54:29.403488+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'10","last_scrub_stamp":"2024-02-13T00:53:19.374786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:53:19.374786+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":803,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979950+0000","last_change":"2024-02-13T00:53:45.110649+0000","last_active":"2024-02-13T00:54:28.979950+0000","last_peered":"2024-02-13T00:54:28.979950+0000","last_clean":"2024-02-13T00:54:28.979950+0000","last_became_active":"2024-02-13T00:52:30.596877+0000","last_became_peered":"2024-02-13T00:52:30.596877+0000","last_unstale":"2024-02-13T00:54:28.979950+0000","last_undegraded":"2024-02-13T00:54:28.979950+0000","last_fullsized":"2024-02-13T00:54:28.979950+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:45.110582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:53:45.110582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":460,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979114+0000","last_change":"2024-02-13T00:51:41.587438+0000","last_active":"2024-02-13T00:54:28.979114+0000","last_peered":"2024-02-13T00:54:28.979114+0000","last_clean":"2024-02-13T00:54:28.979114+0000","last_became_active":"2024-02-13T00:51:41.583454+0000","last_became_peered":"2024-02-13T00:51:41.583454+0000","last_unstale":"2024-02-13T00:54:28.979114+0000","last_undegraded":"2024-02-13T00:54:28.979114+0000","last_fullsized":"2024-02-13T00:54:28.979114+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.849687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.849687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"467'22","reported_seq":790,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533510+0000","last_change":"2024-02-13T00:54:23.587189+0000","last_active":"2024-02-13T00:54:25.533510+0000","last_peered":"2024-02-13T00:54:25.533510+0000","last_clean":"2024-02-13T00:54:25.533510+0000","last_became_active":"2024-02-13T00:51:41.584471+0000","last_became_peered":"2024-02-13T00:51:41.584471+0000","last_unstale":"2024-02-13T00:54:25.533510+0000","last_undegraded":"2024-02-13T00:54:25.533510+0000","last_fullsized":"2024-02-13T00:54:25.533510+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'22","last_scrub_stamp":"2024-02-13T00:54:23.587102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:54:23.587102+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":813,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979021+0000","last_change":"2024-02-13T00:52:30.598076+0000","last_active":"2024-02-13T00:54:28.979021+0000","last_peered":"2024-02-13T00:54:28.979021+0000","last_clean":"2024-02-13T00:54:28.979021+0000","last_became_active":"2024-02-13T00:52:30.593985+0000","last_became_peered":"2024-02-13T00:52:30.593985+0000","last_unstale":"2024-02-13T00:54:28.979021+0000","last_undegraded":"2024-02-13T00:54:28.979021+0000","last_fullsized":"2024-02-13T00:54:28.979021+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:05.887665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:05.887665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"726'16","reported_seq":711,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.403466+0000","last_change":"2024-02-13T00:53:10.218214+0000","last_active":"2024-02-13T00:54:29.403466+0000","last_peered":"2024-02-13T00:54:29.403466+0000","last_clean":"2024-02-13T00:54:29.403466+0000","last_became_active":"2024-02-13T00:52:30.603133+0000","last_became_peered":"2024-02-13T00:52:30.603133+0000","last_unstale":"2024-02-13T00:54:29.403466+0000","last_undegraded":"2024-02-13T00:54:29.403466+0000","last_fullsized":"2024-02-13T00:54:29.403466+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'16","last_scrub_stamp":"2024-02-13T00:53:10.218143+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:53:10.218143+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":437,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979718+0000","last_change":"2024-02-13T00:52:30.596098+0000","last_active":"2024-02-13T00:54:28.979718+0000","last_peered":"2024-02-13T00:54:28.979718+0000","last_clean":"2024-02-13T00:54:28.979718+0000","last_became_active":"2024-02-13T00:52:30.594426+0000","last_became_peered":"2024-02-13T00:52:30.594426+0000","last_unstale":"2024-02-13T00:54:28.979718+0000","last_undegraded":"2024-02-13T00:54:28.979718+0000","last_fullsized":"2024-02-13T00:54:28.979718+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:27.933639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:27.933639+0000","last_clean_scrub_stamp":"2024-02-13T00:52:27.933639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":316,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059331+0000","last_change":"2024-02-13T00:53:07.596914+0000","last_active":"2024-02-13T00:54:29.059331+0000","last_peered":"2024-02-13T00:54:29.059331+0000","last_clean":"2024-02-13T00:54:29.059331+0000","last_became_active":"2024-02-13T00:53:07.596544+0000","last_became_peered":"2024-02-13T00:53:07.596544+0000","last_unstale":"2024-02-13T00:54:29.059331+0000","last_undegraded":"2024-02-13T00:54:29.059331+0000","last_fullsized":"2024-02-13T00:54:29.059331+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:34.127252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:34.127252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":852,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059340+0000","last_change":"2024-02-13T00:49:38.007273+0000","last_active":"2024-02-13T00:54:29.059340+0000","last_peered":"2024-02-13T00:54:29.059340+0000","last_clean":"2024-02-13T00:54:29.059340+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:54:29.059340+0000","last_undegraded":"2024-02-13T00:54:29.059340+0000","last_fullsized":"2024-02-13T00:54:29.059340+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'19","last_scrub_stamp":"2024-02-13T00:49:38.007204+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:49:38.007204+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":509,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533487+0000","last_change":"2024-02-13T00:51:41.571972+0000","last_active":"2024-02-13T00:54:25.533487+0000","last_peered":"2024-02-13T00:54:25.533487+0000","last_clean":"2024-02-13T00:54:25.533487+0000","last_became_active":"2024-02-13T00:51:41.570640+0000","last_became_peered":"2024-02-13T00:51:41.570640+0000","last_unstale":"2024-02-13T00:54:25.533487+0000","last_undegraded":"2024-02-13T00:54:25.533487+0000","last_fullsized":"2024-02-13T00:54:25.533487+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:13.316727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:51:13.316727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":573,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978722+0000","last_change":"2024-02-13T00:52:30.594111+0000","last_active":"2024-02-13T00:54:28.978722+0000","last_peered":"2024-02-13T00:54:28.978722+0000","last_clean":"2024-02-13T00:54:28.978722+0000","last_became_active":"2024-02-13T00:52:30.593800+0000","last_became_peered":"2024-02-13T00:52:30.593800+0000","last_unstale":"2024-02-13T00:54:28.978722+0000","last_undegraded":"2024-02-13T00:54:28.978722+0000","last_fullsized":"2024-02-13T00:54:28.978722+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:24.916456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:52:24.916456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":767,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978120+0000","last_change":"2024-02-13T00:53:49.062752+0000","last_active":"2024-02-13T00:54:28.978120+0000","last_peered":"2024-02-13T00:54:28.978120+0000","last_clean":"2024-02-13T00:54:28.978120+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:54:28.978120+0000","last_undegraded":"2024-02-13T00:54:28.978120+0000","last_fullsized":"2024-02-13T00:54:28.978120+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:49.062647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:53:49.062647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":508,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533300+0000","last_change":"2024-02-13T00:53:08.375070+0000","last_active":"2024-02-13T00:54:25.533300+0000","last_peered":"2024-02-13T00:54:25.533300+0000","last_clean":"2024-02-13T00:54:25.533300+0000","last_became_active":"2024-02-13T00:51:41.570732+0000","last_became_peered":"2024-02-13T00:51:41.570732+0000","last_unstale":"2024-02-13T00:54:25.533300+0000","last_undegraded":"2024-02-13T00:54:25.533300+0000","last_fullsized":"2024-02-13T00:54:25.533300+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:08.374988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:53:08.374988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":548,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533282+0000","last_change":"2024-02-13T00:52:33.431600+0000","last_active":"2024-02-13T00:54:25.533282+0000","last_peered":"2024-02-13T00:54:25.533282+0000","last_clean":"2024-02-13T00:54:25.533282+0000","last_became_active":"2024-02-13T00:52:30.594405+0000","last_became_peered":"2024-02-13T00:52:30.594405+0000","last_unstale":"2024-02-13T00:54:25.533282+0000","last_undegraded":"2024-02-13T00:54:25.533282+0000","last_fullsized":"2024-02-13T00:54:25.533282+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:48:34.458264+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:48:34.458264+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":777,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059953+0000","last_change":"2024-02-13T00:51:47.538426+0000","last_active":"2024-02-13T00:54:29.059953+0000","last_peered":"2024-02-13T00:54:29.059953+0000","last_clean":"2024-02-13T00:54:29.059953+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:54:29.059953+0000","last_undegraded":"2024-02-13T00:54:29.059953+0000","last_fullsized":"2024-02-13T00:54:29.059953+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:47.538354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:51:47.538354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":800,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059392+0000","last_change":"2024-02-13T00:51:41.569851+0000","last_active":"2024-02-13T00:54:29.059392+0000","last_peered":"2024-02-13T00:54:29.059392+0000","last_clean":"2024-02-13T00:54:29.059392+0000","last_became_active":"2024-02-13T00:51:41.568200+0000","last_became_peered":"2024-02-13T00:51:41.568200+0000","last_unstale":"2024-02-13T00:54:29.059392+0000","last_undegraded":"2024-02-13T00:54:29.059392+0000","last_fullsized":"2024-02-13T00:54:29.059392+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:36.787604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:50:36.787604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":582,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978535+0000","last_change":"2024-02-13T00:53:17.006874+0000","last_active":"2024-02-13T00:54:28.978535+0000","last_peered":"2024-02-13T00:54:28.978535+0000","last_clean":"2024-02-13T00:54:28.978535+0000","last_became_active":"2024-02-13T00:52:30.588561+0000","last_became_peered":"2024-02-13T00:52:30.588561+0000","last_unstale":"2024-02-13T00:54:28.978535+0000","last_undegraded":"2024-02-13T00:54:28.978535+0000","last_fullsized":"2024-02-13T00:54:28.978535+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:17.006804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:17.006804+0000","last_clean_scrub_stamp":"2024-02-13T00:53:17.006804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":509,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532620+0000","last_change":"2024-02-13T00:51:41.586630+0000","last_active":"2024-02-13T00:54:25.532620+0000","last_peered":"2024-02-13T00:54:25.532620+0000","last_clean":"2024-02-13T00:54:25.532620+0000","last_became_active":"2024-02-13T00:51:41.584153+0000","last_became_peered":"2024-02-13T00:51:41.584153+0000","last_unstale":"2024-02-13T00:54:25.532620+0000","last_undegraded":"2024-02-13T00:54:25.532620+0000","last_fullsized":"2024-02-13T00:54:25.532620+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:02.260045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:02.260045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":619,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059222+0000","last_change":"2024-02-13T00:52:33.431814+0000","last_active":"2024-02-13T00:54:29.059222+0000","last_peered":"2024-02-13T00:54:29.059222+0000","last_clean":"2024-02-13T00:54:29.059222+0000","last_became_active":"2024-02-13T00:52:30.606082+0000","last_became_peered":"2024-02-13T00:52:30.606082+0000","last_unstale":"2024-02-13T00:54:29.059222+0000","last_undegraded":"2024-02-13T00:54:29.059222+0000","last_fullsized":"2024-02-13T00:54:29.059222+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'8","last_scrub_stamp":"2024-02-13T00:50:32.774820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:50:32.774820+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":726,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.531879+0000","last_change":"2024-02-13T00:48:32.441279+0000","last_active":"2024-02-13T00:54:25.531879+0000","last_peered":"2024-02-13T00:54:25.531879+0000","last_clean":"2024-02-13T00:54:25.531879+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:54:25.531879+0000","last_undegraded":"2024-02-13T00:54:25.531879+0000","last_fullsized":"2024-02-13T00:54:25.531879+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.441170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.441170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":509,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532674+0000","last_change":"2024-02-13T00:52:30.579288+0000","last_active":"2024-02-13T00:54:25.532674+0000","last_peered":"2024-02-13T00:54:25.532674+0000","last_clean":"2024-02-13T00:54:25.532674+0000","last_became_active":"2024-02-13T00:52:30.578921+0000","last_became_peered":"2024-02-13T00:52:30.578921+0000","last_unstale":"2024-02-13T00:54:25.532674+0000","last_undegraded":"2024-02-13T00:54:25.532674+0000","last_fullsized":"2024-02-13T00:54:25.532674+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:28.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:28.353347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":504,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059063+0000","last_change":"2024-02-13T00:52:30.603553+0000","last_active":"2024-02-13T00:54:29.059063+0000","last_peered":"2024-02-13T00:54:29.059063+0000","last_clean":"2024-02-13T00:54:29.059063+0000","last_became_active":"2024-02-13T00:52:30.602714+0000","last_became_peered":"2024-02-13T00:52:30.602714+0000","last_unstale":"2024-02-13T00:54:29.059063+0000","last_undegraded":"2024-02-13T00:54:29.059063+0000","last_fullsized":"2024-02-13T00:54:29.059063+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.052640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.052640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":656,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532761+0000","last_change":"2024-02-13T00:52:30.593895+0000","last_active":"2024-02-13T00:54:25.532761+0000","last_peered":"2024-02-13T00:54:25.532761+0000","last_clean":"2024-02-13T00:54:25.532761+0000","last_became_active":"2024-02-13T00:52:30.593558+0000","last_became_peered":"2024-02-13T00:52:30.593558+0000","last_unstale":"2024-02-13T00:54:25.532761+0000","last_undegraded":"2024-02-13T00:54:25.532761+0000","last_fullsized":"2024-02-13T00:54:25.532761+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:54.114116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:54.114116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"467'25","reported_seq":577,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532789+0000","last_change":"2024-02-13T00:51:43.826108+0000","last_active":"2024-02-13T00:54:25.532789+0000","last_peered":"2024-02-13T00:54:25.532789+0000","last_clean":"2024-02-13T00:54:25.532789+0000","last_became_active":"2024-02-13T00:51:41.585940+0000","last_became_peered":"2024-02-13T00:51:41.585940+0000","last_unstale":"2024-02-13T00:54:25.532789+0000","last_undegraded":"2024-02-13T00:54:25.532789+0000","last_fullsized":"2024-02-13T00:54:25.532789+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-13T00:47:36.368863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:47:36.368863+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":505,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.531947+0000","last_change":"2024-02-13T00:53:42.494098+0000","last_active":"2024-02-13T00:54:25.531947+0000","last_peered":"2024-02-13T00:54:25.531947+0000","last_clean":"2024-02-13T00:54:25.531947+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:54:25.531947+0000","last_undegraded":"2024-02-13T00:54:25.531947+0000","last_fullsized":"2024-02-13T00:54:25.531947+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:42.494009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:53:42.494009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":509,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532385+0000","last_change":"2024-02-13T00:52:30.578502+0000","last_active":"2024-02-13T00:54:25.532385+0000","last_peered":"2024-02-13T00:54:25.532385+0000","last_clean":"2024-02-13T00:54:25.532385+0000","last_became_active":"2024-02-13T00:52:30.577297+0000","last_became_peered":"2024-02-13T00:52:30.577297+0000","last_unstale":"2024-02-13T00:54:25.532385+0000","last_undegraded":"2024-02-13T00:54:25.532385+0000","last_fullsized":"2024-02-13T00:54:25.532385+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:08.345652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:51:08.345652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":709,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978665+0000","last_change":"2024-02-13T00:52:30.572207+0000","last_active":"2024-02-13T00:54:28.978665+0000","last_peered":"2024-02-13T00:54:28.978665+0000","last_clean":"2024-02-13T00:54:28.978665+0000","last_became_active":"2024-02-13T00:52:30.569895+0000","last_became_peered":"2024-02-13T00:52:30.569895+0000","last_unstale":"2024-02-13T00:54:28.978665+0000","last_undegraded":"2024-02-13T00:54:28.978665+0000","last_fullsized":"2024-02-13T00:54:28.978665+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:06.264572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:51:06.264572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":623,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.534185+0000","last_change":"2024-02-13T00:51:41.571094+0000","last_active":"2024-02-13T00:54:25.534185+0000","last_peered":"2024-02-13T00:54:25.534185+0000","last_clean":"2024-02-13T00:54:25.534185+0000","last_became_active":"2024-02-13T00:51:41.569716+0000","last_became_peered":"2024-02-13T00:51:41.569716+0000","last_unstale":"2024-02-13T00:54:25.534185+0000","last_undegraded":"2024-02-13T00:54:25.534185+0000","last_fullsized":"2024-02-13T00:54:25.534185+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:35.101298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.101298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":912,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.168056+0000","last_change":"2024-02-13T00:54:29.168034+0000","last_active":"2024-02-13T00:54:29.168056+0000","last_peered":"2024-02-13T00:54:29.168056+0000","last_clean":"2024-02-13T00:54:29.168056+0000","last_became_active":"2024-02-13T00:51:41.589397+0000","last_became_peered":"2024-02-13T00:51:41.589397+0000","last_unstale":"2024-02-13T00:54:29.168056+0000","last_undegraded":"2024-02-13T00:54:29.168056+0000","last_fullsized":"2024-02-13T00:54:29.168056+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'32","last_scrub_stamp":"2024-02-13T00:54:29.167966+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:54:29.167966+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":751,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532174+0000","last_change":"2024-02-13T00:53:20.407812+0000","last_active":"2024-02-13T00:54:25.532174+0000","last_peered":"2024-02-13T00:54:25.532174+0000","last_clean":"2024-02-13T00:54:25.532174+0000","last_became_active":"2024-02-13T00:52:30.578094+0000","last_became_peered":"2024-02-13T00:52:30.578094+0000","last_unstale":"2024-02-13T00:54:25.532174+0000","last_undegraded":"2024-02-13T00:54:25.532174+0000","last_fullsized":"2024-02-13T00:54:25.532174+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:20.407731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:53:20.407731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":618,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533133+0000","last_change":"2024-02-13T00:52:32.509795+0000","last_active":"2024-02-13T00:54:25.533133+0000","last_peered":"2024-02-13T00:54:25.533133+0000","last_clean":"2024-02-13T00:54:25.533133+0000","last_became_active":"2024-02-13T00:52:30.585316+0000","last_became_peered":"2024-02-13T00:52:30.585316+0000","last_unstale":"2024-02-13T00:54:25.533133+0000","last_undegraded":"2024-02-13T00:54:25.533133+0000","last_fullsized":"2024-02-13T00:54:25.533133+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:48:33.460247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:33.460247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":744,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060818+0000","last_change":"2024-02-13T00:51:41.571960+0000","last_active":"2024-02-13T00:54:29.060818+0000","last_peered":"2024-02-13T00:54:29.060818+0000","last_clean":"2024-02-13T00:54:29.060818+0000","last_became_active":"2024-02-13T00:51:41.569703+0000","last_became_peered":"2024-02-13T00:51:41.569703+0000","last_unstale":"2024-02-13T00:54:29.060818+0000","last_undegraded":"2024-02-13T00:54:29.060818+0000","last_fullsized":"2024-02-13T00:54:29.060818+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:39.570602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:39.570602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":531,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.531817+0000","last_change":"2024-02-13T00:48:55.424021+0000","last_active":"2024-02-13T00:54:25.531817+0000","last_peered":"2024-02-13T00:54:25.531817+0000","last_clean":"2024-02-13T00:54:25.531817+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:54:25.531817+0000","last_undegraded":"2024-02-13T00:54:25.531817+0000","last_fullsized":"2024-02-13T00:54:25.531817+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:55.423952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.423952+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":514,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532853+0000","last_change":"2024-02-13T00:53:47.402044+0000","last_active":"2024-02-13T00:54:25.532853+0000","last_peered":"2024-02-13T00:54:25.532853+0000","last_clean":"2024-02-13T00:54:25.532853+0000","last_became_active":"2024-02-13T00:51:41.584919+0000","last_became_peered":"2024-02-13T00:51:41.584919+0000","last_unstale":"2024-02-13T00:54:25.532853+0000","last_undegraded":"2024-02-13T00:54:25.532853+0000","last_fullsized":"2024-02-13T00:54:25.532853+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:47.401921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:53:47.401921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":780,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059877+0000","last_change":"2024-02-13T00:52:30.603945+0000","last_active":"2024-02-13T00:54:29.059877+0000","last_peered":"2024-02-13T00:54:29.059877+0000","last_clean":"2024-02-13T00:54:29.059877+0000","last_became_active":"2024-02-13T00:52:30.602705+0000","last_became_peered":"2024-02-13T00:52:30.602705+0000","last_unstale":"2024-02-13T00:54:29.059877+0000","last_undegraded":"2024-02-13T00:54:29.059877+0000","last_fullsized":"2024-02-13T00:54:29.059877+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.928651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.928651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":496,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533598+0000","last_change":"2024-02-13T00:52:24.173739+0000","last_active":"2024-02-13T00:54:25.533598+0000","last_peered":"2024-02-13T00:54:25.533598+0000","last_clean":"2024-02-13T00:54:25.533598+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:54:25.533598+0000","last_undegraded":"2024-02-13T00:54:25.533598+0000","last_fullsized":"2024-02-13T00:54:25.533598+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:24.173651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:24.173651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":575,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060009+0000","last_change":"2024-02-13T00:52:32.307501+0000","last_active":"2024-02-13T00:54:29.060009+0000","last_peered":"2024-02-13T00:54:29.060009+0000","last_clean":"2024-02-13T00:54:29.060009+0000","last_became_active":"2024-02-13T00:52:30.603658+0000","last_became_peered":"2024-02-13T00:52:30.603658+0000","last_unstale":"2024-02-13T00:54:29.060009+0000","last_undegraded":"2024-02-13T00:54:29.060009+0000","last_fullsized":"2024-02-13T00:54:29.060009+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:30.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:30.086856+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":88,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532663+0000","last_change":"2024-02-13T00:52:30.578696+0000","last_active":"2024-02-13T00:54:25.532663+0000","last_peered":"2024-02-13T00:54:25.532663+0000","last_clean":"2024-02-13T00:54:25.532663+0000","last_became_active":"2024-02-13T00:52:30.578352+0000","last_became_peered":"2024-02-13T00:52:30.578352+0000","last_unstale":"2024-02-13T00:54:25.532663+0000","last_undegraded":"2024-02-13T00:54:25.532663+0000","last_fullsized":"2024-02-13T00:54:25.532663+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:04.282529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:04.282529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":505,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:27.597022+0000","last_change":"2024-02-13T00:54:27.596986+0000","last_active":"2024-02-13T00:54:27.597022+0000","last_peered":"2024-02-13T00:54:27.597022+0000","last_clean":"2024-02-13T00:54:27.597022+0000","last_became_active":"2024-02-13T00:52:30.594309+0000","last_became_peered":"2024-02-13T00:52:30.594309+0000","last_unstale":"2024-02-13T00:54:27.597022+0000","last_undegraded":"2024-02-13T00:54:27.597022+0000","last_fullsized":"2024-02-13T00:54:27.597022+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:54:27.596895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:54:27.596895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":423,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533487+0000","last_change":"2024-02-13T00:52:30.597508+0000","last_active":"2024-02-13T00:54:25.533487+0000","last_peered":"2024-02-13T00:54:25.533487+0000","last_clean":"2024-02-13T00:54:25.533487+0000","last_became_active":"2024-02-13T00:52:30.595882+0000","last_became_peered":"2024-02-13T00:52:30.595882+0000","last_unstale":"2024-02-13T00:54:25.533487+0000","last_undegraded":"2024-02-13T00:54:25.533487+0000","last_fullsized":"2024-02-13T00:54:25.533487+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:23.430396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:23.430396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":679,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060178+0000","last_change":"2024-02-13T00:52:32.857449+0000","last_active":"2024-02-13T00:54:29.060178+0000","last_peered":"2024-02-13T00:54:29.060178+0000","last_clean":"2024-02-13T00:54:29.060178+0000","last_became_active":"2024-02-13T00:52:30.604045+0000","last_became_peered":"2024-02-13T00:52:30.604045+0000","last_unstale":"2024-02-13T00:54:29.060178+0000","last_undegraded":"2024-02-13T00:54:29.060178+0000","last_fullsized":"2024-02-13T00:54:29.060178+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:52:21.312574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:21.312574+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":518,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533387+0000","last_change":"2024-02-13T00:52:30.596393+0000","last_active":"2024-02-13T00:54:25.533387+0000","last_peered":"2024-02-13T00:54:25.533387+0000","last_clean":"2024-02-13T00:54:25.533387+0000","last_became_active":"2024-02-13T00:52:30.596074+0000","last_became_peered":"2024-02-13T00:52:30.596074+0000","last_unstale":"2024-02-13T00:54:25.533387+0000","last_undegraded":"2024-02-13T00:54:25.533387+0000","last_fullsized":"2024-02-13T00:54:25.533387+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:18.264738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:18.264738+0000","last_clean_scrub_stamp":"2024-02-13T00:51:18.264738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":496,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532075+0000","last_change":"2024-02-13T00:53:17.397082+0000","last_active":"2024-02-13T00:54:25.532075+0000","last_peered":"2024-02-13T00:54:25.532075+0000","last_clean":"2024-02-13T00:54:25.532075+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:54:25.532075+0000","last_undegraded":"2024-02-13T00:54:25.532075+0000","last_fullsized":"2024-02-13T00:54:25.532075+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:17.397016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:53:17.397016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":741,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979312+0000","last_change":"2024-02-13T00:51:35.190514+0000","last_active":"2024-02-13T00:54:28.979312+0000","last_peered":"2024-02-13T00:54:28.979312+0000","last_clean":"2024-02-13T00:54:28.979312+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:54:28.979312+0000","last_undegraded":"2024-02-13T00:54:28.979312+0000","last_fullsized":"2024-02-13T00:54:28.979312+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:35.190419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:35.190419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":669,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979215+0000","last_change":"2024-02-13T00:52:31.275396+0000","last_active":"2024-02-13T00:54:28.979215+0000","last_peered":"2024-02-13T00:54:28.979215+0000","last_clean":"2024-02-13T00:54:28.979215+0000","last_became_active":"2024-02-13T00:52:30.592405+0000","last_became_peered":"2024-02-13T00:52:30.592405+0000","last_unstale":"2024-02-13T00:54:28.979215+0000","last_undegraded":"2024-02-13T00:54:28.979215+0000","last_fullsized":"2024-02-13T00:54:28.979215+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:52:16.976052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:16.976052+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":584,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533261+0000","last_change":"2024-02-13T00:51:41.571773+0000","last_active":"2024-02-13T00:54:25.533261+0000","last_peered":"2024-02-13T00:54:25.533261+0000","last_clean":"2024-02-13T00:54:25.533261+0000","last_became_active":"2024-02-13T00:51:41.569507+0000","last_became_peered":"2024-02-13T00:51:41.569507+0000","last_unstale":"2024-02-13T00:54:25.533261+0000","last_undegraded":"2024-02-13T00:54:25.533261+0000","last_fullsized":"2024-02-13T00:54:25.533261+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:43.067615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:43.067615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":778,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978343+0000","last_change":"2024-02-13T00:52:35.023058+0000","last_active":"2024-02-13T00:54:28.978343+0000","last_peered":"2024-02-13T00:54:28.978343+0000","last_clean":"2024-02-13T00:54:28.978343+0000","last_became_active":"2024-02-13T00:51:41.582536+0000","last_became_peered":"2024-02-13T00:51:41.582536+0000","last_unstale":"2024-02-13T00:54:28.978343+0000","last_undegraded":"2024-02-13T00:54:28.978343+0000","last_fullsized":"2024-02-13T00:54:28.978343+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:35.022992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:52:35.022992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":684,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979650+0000","last_change":"2024-02-13T00:51:41.587948+0000","last_active":"2024-02-13T00:54:28.979650+0000","last_peered":"2024-02-13T00:54:28.979650+0000","last_clean":"2024-02-13T00:54:28.979650+0000","last_became_active":"2024-02-13T00:51:41.583578+0000","last_became_peered":"2024-02-13T00:51:41.583578+0000","last_unstale":"2024-02-13T00:54:28.979650+0000","last_undegraded":"2024-02-13T00:54:28.979650+0000","last_fullsized":"2024-02-13T00:54:28.979650+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:15.846461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:15.846461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":698,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059609+0000","last_change":"2024-02-13T00:52:32.781431+0000","last_active":"2024-02-13T00:54:29.059609+0000","last_peered":"2024-02-13T00:54:29.059609+0000","last_clean":"2024-02-13T00:54:29.059609+0000","last_became_active":"2024-02-13T00:52:30.604370+0000","last_became_peered":"2024-02-13T00:52:30.604370+0000","last_unstale":"2024-02-13T00:54:29.059609+0000","last_undegraded":"2024-02-13T00:54:29.059609+0000","last_fullsized":"2024-02-13T00:54:29.059609+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'16","last_scrub_stamp":"2024-02-13T00:51:49.491168+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:51:49.491168+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":72,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059533+0000","last_change":"2024-02-13T00:53:42.501211+0000","last_active":"2024-02-13T00:54:29.059533+0000","last_peered":"2024-02-13T00:54:29.059533+0000","last_clean":"2024-02-13T00:54:29.059533+0000","last_became_active":"2024-02-13T00:53:01.780063+0000","last_became_peered":"2024-02-13T00:53:01.780063+0000","last_unstale":"2024-02-13T00:54:29.059533+0000","last_undegraded":"2024-02-13T00:54:29.059533+0000","last_fullsized":"2024-02-13T00:54:29.059533+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:42.501149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:53:42.501149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":725,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059153+0000","last_change":"2024-02-13T00:51:41.589710+0000","last_active":"2024-02-13T00:54:29.059153+0000","last_peered":"2024-02-13T00:54:29.059153+0000","last_clean":"2024-02-13T00:54:29.059153+0000","last_became_active":"2024-02-13T00:51:41.589029+0000","last_became_peered":"2024-02-13T00:51:41.589029+0000","last_unstale":"2024-02-13T00:54:29.059153+0000","last_undegraded":"2024-02-13T00:54:29.059153+0000","last_fullsized":"2024-02-13T00:54:29.059153+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:01.133709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:49:01.133709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":487,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532568+0000","last_change":"2024-02-13T00:52:38.236729+0000","last_active":"2024-02-13T00:54:25.532568+0000","last_peered":"2024-02-13T00:54:25.532568+0000","last_clean":"2024-02-13T00:54:25.532568+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:54:25.532568+0000","last_undegraded":"2024-02-13T00:54:25.532568+0000","last_fullsized":"2024-02-13T00:54:25.532568+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:38.236655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:38.236655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":566,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.532719+0000","last_change":"2024-02-13T00:52:32.083766+0000","last_active":"2024-02-13T00:54:25.532719+0000","last_peered":"2024-02-13T00:54:25.532719+0000","last_clean":"2024-02-13T00:54:25.532719+0000","last_became_active":"2024-02-13T00:52:30.576217+0000","last_became_peered":"2024-02-13T00:52:30.576217+0000","last_unstale":"2024-02-13T00:54:25.532719+0000","last_undegraded":"2024-02-13T00:54:25.532719+0000","last_fullsized":"2024-02-13T00:54:25.532719+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:51:10.298233+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:51:10.298233+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":600,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.980072+0000","last_change":"2024-02-13T00:53:09.074634+0000","last_active":"2024-02-13T00:54:28.980072+0000","last_peered":"2024-02-13T00:54:28.980072+0000","last_clean":"2024-02-13T00:54:28.980072+0000","last_became_active":"2024-02-13T00:52:30.597277+0000","last_became_peered":"2024-02-13T00:52:30.597277+0000","last_unstale":"2024-02-13T00:54:28.980072+0000","last_undegraded":"2024-02-13T00:54:28.980072+0000","last_fullsized":"2024-02-13T00:54:28.980072+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:09.074568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:53:09.074568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":438,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059162+0000","last_change":"2024-02-13T00:53:43.505855+0000","last_active":"2024-02-13T00:54:29.059162+0000","last_peered":"2024-02-13T00:54:29.059162+0000","last_clean":"2024-02-13T00:54:29.059162+0000","last_became_active":"2024-02-13T00:52:30.602788+0000","last_became_peered":"2024-02-13T00:52:30.602788+0000","last_unstale":"2024-02-13T00:54:29.059162+0000","last_undegraded":"2024-02-13T00:54:29.059162+0000","last_fullsized":"2024-02-13T00:54:29.059162+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:43.505792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:53:43.505792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":747,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059241+0000","last_change":"2024-02-13T00:51:36.522278+0000","last_active":"2024-02-13T00:54:29.059241+0000","last_peered":"2024-02-13T00:54:29.059241+0000","last_clean":"2024-02-13T00:54:29.059241+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:54:29.059241+0000","last_undegraded":"2024-02-13T00:54:29.059241+0000","last_fullsized":"2024-02-13T00:54:29.059241+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:51:36.522208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:36.522208+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":681,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978615+0000","last_change":"2024-02-13T00:53:06.023522+0000","last_active":"2024-02-13T00:54:28.978615+0000","last_peered":"2024-02-13T00:54:28.978615+0000","last_clean":"2024-02-13T00:54:28.978615+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:54:28.978615+0000","last_undegraded":"2024-02-13T00:54:28.978615+0000","last_fullsized":"2024-02-13T00:54:28.978615+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'62","last_scrub_stamp":"2024-02-13T00:53:06.023450+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:53:06.023450+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":791,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979780+0000","last_change":"2024-02-13T00:53:53.023337+0000","last_active":"2024-02-13T00:54:28.979780+0000","last_peered":"2024-02-13T00:54:28.979780+0000","last_clean":"2024-02-13T00:54:28.979780+0000","last_became_active":"2024-02-13T00:52:30.595533+0000","last_became_peered":"2024-02-13T00:52:30.595533+0000","last_unstale":"2024-02-13T00:54:28.979780+0000","last_undegraded":"2024-02-13T00:54:28.979780+0000","last_fullsized":"2024-02-13T00:54:28.979780+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:53.023274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:53:53.023274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":500,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-13T00:54:25.533413+0000","last_change":"2024-02-13T00:51:41.573986+0000","last_active":"2024-02-13T00:54:25.533413+0000","last_peered":"2024-02-13T00:54:25.533413+0000","last_clean":"2024-02-13T00:54:25.533413+0000","last_became_active":"2024-02-13T00:51:41.571183+0000","last_became_peered":"2024-02-13T00:51:41.571183+0000","last_unstale":"2024-02-13T00:54:25.533413+0000","last_undegraded":"2024-02-13T00:54:25.533413+0000","last_fullsized":"2024-02-13T00:54:25.533413+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:03.383666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:03.383666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1491,"num_read_kb":1216,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":780,"num_bytes_recovered":13820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":11436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":972,"ondisk_log_size":972,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":995,"num_read_kb":990,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266892,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319880,"kb_used_data":6272,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93425272,"statfs":{"total":95995035648,"available":95667478528,"internally_reserved":0,"allocated":6422528,"data_stored":3323280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.497,"15min":0.63200000000000001},"min":{"1min":0.27000000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.71499999999999997,"5min":0.76700000000000002,"15min":10.682},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.50900000000000001,"15min":0.74099999999999999},"min":{"1min":0.32600000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.68700000000000006,"5min":0.77400000000000002,"15min":25.504000000000001},"last":0.58899999999999997}]},{"osd":1,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.50800000000000001,"15min":0.57599999999999996},"min":{"1min":0.27000000000000002,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.68200000000000005,"5min":0.83499999999999996,"15min":13.347},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51500000000000001,"15min":0.65500000000000003},"min":{"1min":0.33900000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":13.526999999999999},"last":0.69799999999999995}]},{"osd":2,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52300000000000002,"15min":0.65600000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.26600000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":14.359},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53600000000000003,"15min":0.73499999999999999},"min":{"1min":0.36299999999999999,"5min":0.32000000000000001,"15min":0.26400000000000001},"max":{"1min":0.75,"5min":0.81599999999999995,"15min":21.497},"last":0.57699999999999996}]},{"osd":3,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51200000000000001,"15min":0.70199999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61099999999999999,"5min":0.89400000000000002,"15min":12.997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.81799999999999995},"min":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":0.79300000000000004,"15min":21.622},"last":0.67600000000000005}]},{"osd":4,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.442,"15min":0.435},"min":{"1min":0.28899999999999998,"5min":0.26500000000000001,"15min":0.23000000000000001},"max":{"1min":0.69299999999999995,"5min":0.76000000000000001,"15min":2.452},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.47299999999999998,"15min":0.44700000000000001},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.245},"max":{"1min":0.60099999999999998,"5min":0.83199999999999996,"15min":1.0780000000000001},"last":0.377}]},{"osd":5,"last update":"Tue Feb 13 00:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48799999999999999,"15min":0.502},"min":{"1min":0.35399999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.5,"15min":0.51500000000000001},"min":{"1min":0.31,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.64600000000000002,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.40799999999999997}]},{"osd":6,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.49399999999999999},"min":{"1min":0.38,"5min":0.31,"15min":0.25},"max":{"1min":0.66500000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.51200000000000001,"15min":0.49099999999999999},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.27500000000000002},"max":{"1min":0.79300000000000004,"5min":0.85499999999999998,"15min":1.0940000000000001},"last":0.438}]}]},{"osd":6,"up_from":45,"seq":193273528530,"num_pgs":128,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307592,"kb_used_data":6272,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437560,"statfs":{"total":95995035648,"available":95680061440,"internally_reserved":0,"allocated":6422528,"data_stored":3323454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.498,"15min":0.53300000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.83099999999999996,"15min":8.1340000000000003},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.505,"15min":0.62},"min":{"1min":0.33900000000000002,"5min":0.27700000000000002,"15min":0.221},"max":{"1min":0.872,"5min":0.872,"15min":11.367000000000001},"last":0.626}]},{"osd":1,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.496,"15min":0.88200000000000001},"min":{"1min":0.32800000000000001,"5min":0.30099999999999999,"15min":0.255},"max":{"1min":0.81499999999999995,"5min":0.89300000000000002,"15min":54.002000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.51100000000000001,"15min":0.67900000000000005},"min":{"1min":0.27900000000000003,"5min":0.27200000000000002,"15min":0.23499999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":12.403},"last":0.58599999999999997}]},{"osd":2,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.501,"15min":0.71199999999999997},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.25600000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":12.477},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48999999999999999,"15min":0.67900000000000005},"min":{"1min":0.33800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":11.702},"last":0.46999999999999997}]},{"osd":3,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51200000000000001,"15min":0.67100000000000004},"min":{"1min":0.35399999999999998,"5min":0.31900000000000001,"15min":0.29199999999999998},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":8.0519999999999996},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.54000000000000004,"15min":0.77100000000000002},"min":{"1min":0.41799999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.74099999999999999,"5min":0.76000000000000001,"15min":19.081},"last":0.65400000000000003}]},{"osd":4,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42399999999999999,"15min":0.42299999999999999},"min":{"1min":0.30399999999999999,"5min":0.26600000000000001,"15min":0.24199999999999999},"max":{"1min":0.65100000000000002,"5min":0.70599999999999996,"15min":0.72199999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46500000000000002,"15min":0.44900000000000001},"min":{"1min":0.307,"5min":0.252,"15min":0.221},"max":{"1min":0.64400000000000002,"5min":1.008,"15min":1.008},"last":0.66400000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.48999999999999999,"15min":0.51000000000000001},"min":{"1min":0.38200000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.441},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.499,"15min":0.52100000000000002},"min":{"1min":0.36599999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68000000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.67900000000000005}]},{"osd":7,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47999999999999998,"15min":0.47299999999999998},"min":{"1min":0.34300000000000003,"5min":0.30199999999999999,"15min":0.27200000000000002},"max":{"1min":0.65600000000000003,"5min":0.74399999999999999,"15min":0.91600000000000004},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53200000000000003,"15min":0.51900000000000002},"min":{"1min":0.35199999999999998,"5min":0.312,"15min":0.26900000000000002},"max":{"1min":0.70399999999999996,"5min":0.76100000000000001,"15min":0.80700000000000005},"last":0.64000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":125,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319908,"kb_used_data":6300,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93425244,"statfs":{"total":95995035648,"available":95667449856,"internally_reserved":0,"allocated":6451200,"data_stored":3330879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.95199999999999996},"min":{"1min":0.23300000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.61799999999999999,"5min":0.80600000000000005,"15min":61.856000000000002},"last":0.504},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.34100000000000003,"15min":0.66800000000000004},"min":{"1min":0.246,"5min":0.19900000000000001,"15min":0.16500000000000001},"max":{"1min":0.44600000000000001,"5min":0.77600000000000002,"15min":61.908999999999999},"last":0.29699999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.83599999999999997},"min":{"1min":0.27000000000000002,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":0.49199999999999999,"5min":0.629,"15min":32.161000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41699999999999998,"15min":0.83599999999999997},"min":{"1min":0.26600000000000001,"5min":0.245,"15min":0.187},"max":{"1min":0.55800000000000005,"5min":5.1319999999999997,"15min":29.693000000000001},"last":0.36299999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":1.2470000000000001},"min":{"1min":0.28499999999999998,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":0.53600000000000003,"5min":0.75700000000000001,"15min":131.23099999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.39100000000000001,"15min":0.84199999999999997},"min":{"1min":0.28799999999999998,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":0.65500000000000003,"5min":0.753,"15min":49.055},"last":0.39800000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:53:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.53200000000000003,"15min":0.74399999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.85799999999999998,"15min":16.547000000000001},"last":0.749},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58399999999999996,"15min":0.71199999999999997},"min":{"1min":0.39600000000000002,"5min":0.34300000000000003,"15min":0.24199999999999999},"max":{"1min":0.70899999999999996,"5min":0.85199999999999998,"15min":14.731999999999999},"last":0.72599999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58599999999999997,"15min":0.57399999999999995},"min":{"1min":0.41599999999999998,"5min":0.39600000000000002,"15min":0.39400000000000002},"max":{"1min":0.78000000000000003,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.45500000000000002,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":0.745,"5min":1.831,"15min":1.831},"last":0.86099999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57699999999999996,"15min":0.70799999999999996},"min":{"1min":0.44600000000000001,"5min":0.38600000000000001,"15min":0.29099999999999998},"max":{"1min":0.65500000000000003,"5min":1.9279999999999999,"15min":14.772},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55800000000000005,"15min":0.90900000000000003},"min":{"1min":0.44400000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80700000000000005,"5min":1.2709999999999999,"15min":26.600000000000001},"last":0.70699999999999996}]},{"osd":7,"last update":"Tue Feb 13 00:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59399999999999997,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.38200000000000001,"15min":0.29499999999999998},"max":{"1min":0.80900000000000005,"5min":1.8680000000000001,"15min":14.794},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57699999999999996,"15min":0.91200000000000003},"min":{"1min":0.44900000000000001,"5min":0.373,"15min":0.32200000000000001},"max":{"1min":0.75900000000000001,"5min":1.246,"15min":26.693999999999999},"last":0.73399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738607,"num_pgs":123,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303488,"kb_used_data":6264,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441664,"statfs":{"total":95995035648,"available":95684263936,"internally_reserved":0,"allocated":6414336,"data_stored":3322906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34699999999999998,"15min":0.61899999999999999},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.189},"max":{"1min":0.499,"5min":0.82699999999999996,"15min":30.184000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35999999999999999,"15min":0.45200000000000001},"min":{"1min":0.248,"5min":0.20499999999999999,"15min":0.187},"max":{"1min":0.60499999999999998,"5min":1.0109999999999999,"15min":5.6959999999999997},"last":0.40899999999999997}]},{"osd":2,"last update":"Tue Feb 13 00:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.36699999999999999,"15min":0.48699999999999999},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.216},"max":{"1min":0.53100000000000003,"5min":1.0509999999999999,"15min":7.0839999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38600000000000001,"15min":0.67000000000000004},"min":{"1min":0.27500000000000002,"5min":0.23300000000000001,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":2.0129999999999999,"15min":24.459},"last":0.42199999999999999}]},{"osd":3,"last update":"Tue Feb 13 00:54:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.372,"15min":0.47899999999999998},"min":{"1min":0.28000000000000003,"5min":0.214,"15min":0.214},"max":{"1min":0.51100000000000001,"5min":1.0640000000000001,"15min":8.718},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.52500000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.222},"max":{"1min":1.119,"5min":1.119,"15min":7.1139999999999999},"last":0.46400000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:54:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55600000000000005,"15min":0.80700000000000005},"min":{"1min":0.39700000000000002,"5min":0.34899999999999998,"15min":0.24399999999999999},"max":{"1min":0.65900000000000003,"5min":1.4710000000000001,"15min":24.381},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.68200000000000005},"min":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.28699999999999998},"max":{"1min":0.76000000000000001,"5min":2.6339999999999999,"15min":8.6950000000000003},"last":0.628}]},{"osd":5,"last update":"Tue Feb 13 00:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53500000000000003,"15min":0.50700000000000001},"min":{"1min":0.42199999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.64100000000000001,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":0.68000000000000005}]},{"osd":6,"last update":"Tue Feb 13 00:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55800000000000005,"15min":0.64000000000000001},"min":{"1min":0.39100000000000001,"5min":0.35999999999999999,"15min":0.31},"max":{"1min":0.80400000000000005,"5min":2.5960000000000001,"15min":8.6129999999999995},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56799999999999995,"15min":0.66100000000000003},"min":{"1min":0.44500000000000001,"5min":0.34000000000000002,"15min":0.32400000000000001},"max":{"1min":0.84199999999999997,"5min":1.411,"15min":14.263},"last":0.65800000000000003}]},{"osd":7,"last update":"Tue Feb 13 00:54:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57899999999999996,"15min":0.66800000000000004},"min":{"1min":0.38700000000000001,"5min":0.33900000000000002,"15min":0.317},"max":{"1min":0.82599999999999996,"5min":2.496,"15min":8.6679999999999993},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58199999999999996,"15min":0.66900000000000004},"min":{"1min":0.49299999999999999,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.754,"5min":1.375,"15min":8.1609999999999996},"last":0.61399999999999999}]}]},{"osd":2,"up_from":0,"seq":77309411558,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182624,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":133143986396,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":2057289334862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":3218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:54:32.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:54:32.581 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:33.072 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:31.213436+0000 osd.1 (osd.1) 335 : cluster [DBG] 4.7 deep-scrub starts 2024-02-13T00:54:33.072 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:31.218060+0000 osd.1 (osd.1) 336 : cluster [DBG] 4.7 deep-scrub ok 2024-02-13T00:54:33.072 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:31.349028+0000 mgr.y (mgr.14544) 673 : audit [DBG] from='client.28206 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:54:33.072 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:32.207170+0000 osd.0 (osd.0) 321 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:54:33.072 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:32.210118+0000 osd.0 (osd.0) 322 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:54:33.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:31.213436+0000 osd.1 (osd.1) 335 : cluster [DBG] 4.7 deep-scrub starts 2024-02-13T00:54:33.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:31.218060+0000 osd.1 (osd.1) 336 : cluster [DBG] 4.7 deep-scrub ok 2024-02-13T00:54:33.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:31.349028+0000 mgr.y (mgr.14544) 673 : audit [DBG] from='client.28206 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:54:33.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:32.207170+0000 osd.0 (osd.0) 321 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:54:33.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:32.210118+0000 osd.0 (osd.0) 322 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:54:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:31.213436+0000 osd.1 (osd.1) 335 : cluster [DBG] 4.7 deep-scrub starts 2024-02-13T00:54:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:31.218060+0000 osd.1 (osd.1) 336 : cluster [DBG] 4.7 deep-scrub ok 2024-02-13T00:54:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:31.349028+0000 mgr.y (mgr.14544) 673 : audit [DBG] from='client.28206 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:54:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:32.207170+0000 osd.0 (osd.0) 321 : cluster [DBG] 2.13 scrub starts 2024-02-13T00:54:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:32.210118+0000 osd.0 (osd.0) 322 : cluster [DBG] 2.13 scrub ok 2024-02-13T00:54:33.358 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:34.146 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:32.556989+0000 mgr.y (mgr.14544) 674 : cluster [DBG] pgmap v1234: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:54:34.146 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:32.691582+0000 osd.6 (osd.6) 263 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:54:34.146 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:32.695170+0000 osd.6 (osd.6) 264 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:54:34.146 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:32.556989+0000 mgr.y (mgr.14544) 674 : cluster [DBG] pgmap v1234: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:54:34.146 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:32.691582+0000 osd.6 (osd.6) 263 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:54:34.147 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:32.695170+0000 osd.6 (osd.6) 264 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:54:34.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:32.556989+0000 mgr.y (mgr.14544) 674 : cluster [DBG] pgmap v1234: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:54:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:32.691582+0000 osd.6 (osd.6) 263 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:54:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:32.695170+0000 osd.6 (osd.6) 264 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:54:35.130 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:54:35.131 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:54:35.131 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:54:35.131 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:54:35.131 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:54:35.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:33.672303+0000 osd.6 (osd.6) 265 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:54:35.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:33.675266+0000 osd.6 (osd.6) 266 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:54:35.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:34.232943+0000 osd.1 (osd.1) 337 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:54:35.457 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:34.236147+0000 osd.1 (osd.1) 338 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:54:35.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:34.345331+0000 mon.a (mon.0) 3765 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:54:35.458 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:34.557526+0000 mgr.y (mgr.14544) 675 : cluster [DBG] pgmap v1235: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:54:35.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:33.672303+0000 osd.6 (osd.6) 265 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:54:35.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:33.675266+0000 osd.6 (osd.6) 266 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:54:35.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:34.232943+0000 osd.1 (osd.1) 337 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:54:35.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:34.236147+0000 osd.1 (osd.1) 338 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:54:35.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:34.345331+0000 mon.a (mon.0) 3765 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:54:35.458 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:34.557526+0000 mgr.y (mgr.14544) 675 : cluster [DBG] pgmap v1235: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:54:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:33.672303+0000 osd.6 (osd.6) 265 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:54:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:33.675266+0000 osd.6 (osd.6) 266 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:54:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:34.232943+0000 osd.1 (osd.1) 337 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:54:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:34.236147+0000 osd.1 (osd.1) 338 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:54:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:34.345331+0000 mon.a (mon.0) 3765 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:54:35.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:34.557526+0000 mgr.y (mgr.14544) 675 : cluster [DBG] pgmap v1235: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:54:35.907 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-13T00:54:36.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:54:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:54:36.559 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:54:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:54:36] "GET /metrics HTTP/1.1" 200 185427 "" "Prometheus/2.33.4" 2024-02-13T00:54:36.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 -- 172.21.15.162:0/3004258175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200f8d10 msgr2=0x7fda200f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:36.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 --2- 172.21.15.162:0/3004258175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200f8d10 0x7fda200f90f0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 rx=0x7fda100047b0 tx=0x7fda10009cb0).stop 2024-02-13T00:54:36.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 -- 172.21.15.162:0/3004258175 shutdown_connections 2024-02-13T00:54:36.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 --2- 172.21.15.162:0/3004258175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda20102270 0x7fda20104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:36.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 --2- 172.21.15.162:0/3004258175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200f8d10 0x7fda200f90f0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:36.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 --2- 172.21.15.162:0/3004258175 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200f95e0 0x7fda20101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:36.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 -- 172.21.15.162:0/3004258175 >> 172.21.15.162:0/3004258175 conn(0x7fda2006eb90 msgr2=0x7fda200f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:36.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 -- 172.21.15.162:0/3004258175 shutdown_connections 2024-02-13T00:54:36.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 -- 172.21.15.162:0/3004258175 wait complete. 2024-02-13T00:54:36.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 librados: starting msgr at 2024-02-13T00:54:36.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 librados: starting objecter 2024-02-13T00:54:36.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 Processor -- start 2024-02-13T00:54:36.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 -- start start 2024-02-13T00:54:36.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 librados: setting wanted keys 2024-02-13T00:54:36.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.785+0000 7fda2862c700 1 librados: calling monclient init 2024-02-13T00:54:36.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda2862c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200f8d10 0x7fda200f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:36.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda2862c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200f95e0 0x7fda200f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:36.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda2862c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda20102270 0x7fda200f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:36.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda2862c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201166e0 con 0x7fda200f95e0 2024-02-13T00:54:36.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda2862c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda20116850 con 0x7fda20102270 2024-02-13T00:54:36.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda2862c700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda20116a10 con 0x7fda200f8d10 2024-02-13T00:54:36.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda25bc7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200f95e0 0x7fda200f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:36.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda25bc7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200f95e0 0x7fda200f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:40142/0 (socket says 172.21.15.162:40142) 2024-02-13T00:54:36.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda25bc7700 1 -- 172.21.15.162:0/1627831835 learned_addr learned my addr 172.21.15.162:0/1627831835 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda263c8700 1 --2- 172.21.15.162:0/1627831835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200f8d10 0x7fda200f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda26bc9700 1 --2- 172.21.15.162:0/1627831835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda20102270 0x7fda200f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda25bc7700 1 -- 172.21.15.162:0/1627831835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200f8d10 msgr2=0x7fda200f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda25bc7700 1 --2- 172.21.15.162:0/1627831835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200f8d10 0x7fda200f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda25bc7700 1 -- 172.21.15.162:0/1627831835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda20102270 msgr2=0x7fda200f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda25bc7700 1 --2- 172.21.15.162:0/1627831835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda20102270 0x7fda200f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda25bc7700 1 -- 172.21.15.162:0/1627831835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c005040 con 0x7fda200f95e0 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda26bc9700 1 --2- 172.21.15.162:0/1627831835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda20102270 0x7fda200f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda25bc7700 1 --2- 172.21.15.162:0/1627831835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200f95e0 0x7fda200f71b0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 rx=0x7fda1c009400 tx=0x7fda1c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda177fe700 10 client.?.objecter ms_handle_connect 0x7fda200f95e0 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda177fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda177fe700 1 -- 172.21.15.162:0/1627831835 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c007a40 con 0x7fda200f95e0 2024-02-13T00:54:36.793 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda2862c700 1 -- 172.21.15.162:0/1627831835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda10005040 con 0x7fda200f95e0 2024-02-13T00:54:36.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.786+0000 7fda177fe700 1 -- 172.21.15.162:0/1627831835 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda1c00f040 con 0x7fda200f95e0 2024-02-13T00:54:36.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.787+0000 7fda177fe700 1 -- 172.21.15.162:0/1627831835 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c014aa0 con 0x7fda200f95e0 2024-02-13T00:54:36.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.787+0000 7fda2862c700 10 client.28212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:36.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.787+0000 7fda2862c700 1 -- 172.21.15.162:0/1627831835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda201170a0 con 0x7fda200f95e0 2024-02-13T00:54:36.794 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.787+0000 7fda2862c700 1 librados: init done 2024-02-13T00:54:36.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.788+0000 7fda177fe700 1 -- 172.21.15.162:0/1627831835 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fda1c00e630 con 0x7fda200f95e0 2024-02-13T00:54:36.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.788+0000 7fda2862c700 1 -- 172.21.15.162:0/1627831835 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda20107170 con 0x7fda200f95e0 2024-02-13T00:54:36.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.788+0000 7fda177fe700 1 --2- 172.21.15.162:0/1627831835 >> v2:172.21.15.162:6800/2866534053 conn(0x7fda0c062c90 0x7fda0c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:36.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.788+0000 7fda177fe700 1 -- 172.21.15.162:0/1627831835 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(806..806 src has 449..806) v4 ==== 7422+0+0 (secure 0 0 0) 0x7fda1c07cc10 con 0x7fda200f95e0 2024-02-13T00:54:36.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.788+0000 7fda177fe700 10 client.28212.objecter ms_dispatch 0x7fda20001550 osd_map(806..806 src has 449..806) v4 2024-02-13T00:54:36.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.788+0000 7fda177fe700 3 client.28212.objecter handle_osd_map got epochs [806,806] > 0 2024-02-13T00:54:36.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.788+0000 7fda177fe700 3 client.28212.objecter handle_osd_map decoding full epoch 806 2024-02-13T00:54:36.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.788+0000 7fda263c8700 1 --2- 172.21.15.162:0/1627831835 >> v2:172.21.15.162:6800/2866534053 conn(0x7fda0c062c90 0x7fda0c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:36.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.789+0000 7fda177fe700 20 client.28212.objecter dump_active .. 0 homeless 2024-02-13T00:54:36.795 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.789+0000 7fda177fe700 10 client.28212.objecter ms_handle_connect 0x7fda0c062c90 2024-02-13T00:54:36.796 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.789+0000 7fda263c8700 1 --2- 172.21.15.162:0/1627831835 >> v2:172.21.15.162:6800/2866534053 conn(0x7fda0c062c90 0x7fda0c065150 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 rx=0x7fda10005010 tx=0x7fda10009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:36.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.793+0000 7fda177fe700 1 -- 172.21.15.162:0/1627831835 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fda1c049f60 con 0x7fda200f95e0 2024-02-13T00:54:36.828 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:36.829 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:35.634684+0000 osd.5 (osd.5) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:36.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:35.634892+0000 osd.5 (osd.5) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:36.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:36.006541+0000 mgr.y (mgr.14544) 676 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:36.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:36.215434+0000 osd.0 (osd.0) 323 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:54:36.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:36.216325+0000 osd.0 (osd.0) 324 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:54:36.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:35.634684+0000 osd.5 (osd.5) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:36.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:35.634892+0000 osd.5 (osd.5) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:36.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:36.006541+0000 mgr.y (mgr.14544) 676 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:36.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:36.215434+0000 osd.0 (osd.0) 323 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:54:36.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:36.216325+0000 osd.0 (osd.0) 324 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:54:36.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.960+0000 7fda2862c700 1 -- 172.21.15.162:0/1627831835 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fda200591c0 con 0x7fda0c062c90 2024-02-13T00:54:36.974 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.971+0000 7fda177fe700 1 -- 172.21.15.162:0/1627831835 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+413715 (secure 0 0 0) 0x7fda200591c0 con 0x7fda0c062c90 2024-02-13T00:54:36.982 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:54:36.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 10 librados: watch_flush enter 2024-02-13T00:54:36.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 10 librados: watch_flush exit 2024-02-13T00:54:36.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 20 client.28212.objecter shutdown clearing up homeless session... 2024-02-13T00:54:36.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 10 client.28212.objecter successfully canceled tick 2024-02-13T00:54:36.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 -- 172.21.15.162:0/1627831835 >> v2:172.21.15.162:6800/2866534053 conn(0x7fda0c062c90 msgr2=0x7fda0c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:36.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 --2- 172.21.15.162:0/1627831835 >> v2:172.21.15.162:6800/2866534053 conn(0x7fda0c062c90 0x7fda0c065150 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 rx=0x7fda10005010 tx=0x7fda10009cb0).stop 2024-02-13T00:54:36.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 -- 172.21.15.162:0/1627831835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200f95e0 msgr2=0x7fda200f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:36.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 --2- 172.21.15.162:0/1627831835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200f95e0 0x7fda200f71b0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 rx=0x7fda1c009400 tx=0x7fda1c009cb0).stop 2024-02-13T00:54:36.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 -- 172.21.15.162:0/1627831835 shutdown_connections 2024-02-13T00:54:36.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 --2- 172.21.15.162:0/1627831835 >> v2:172.21.15.162:6800/2866534053 conn(0x7fda0c062c90 0x7fda0c065150 unknown :-1 s=CLOSED pgs=3057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:36.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 --2- 172.21.15.162:0/1627831835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda20102270 0x7fda200f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:36.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 --2- 172.21.15.162:0/1627831835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200f8d10 0x7fda200f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:36.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 --2- 172.21.15.162:0/1627831835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200f95e0 0x7fda200f71b0 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:36.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 -- 172.21.15.162:0/1627831835 >> 172.21.15.162:0/1627831835 conn(0x7fda2006eb90 msgr2=0x7fda20102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:36.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 -- 172.21.15.162:0/1627831835 shutdown_connections 2024-02-13T00:54:36.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 -- 172.21.15.162:0/1627831835 wait complete. 2024-02-13T00:54:36.987 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:36.974+0000 7fda2862c700 1 librados: shutdown 2024-02-13T00:54:36.987 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:54:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:35.634684+0000 osd.5 (osd.5) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:35.634892+0000 osd.5 (osd.5) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:36.006541+0000 mgr.y (mgr.14544) 676 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:36.215434+0000 osd.0 (osd.0) 323 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:54:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:36.216325+0000 osd.0 (osd.0) 324 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:54:37.789 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:36.558321+0000 mgr.y (mgr.14544) 677 : cluster [DBG] pgmap v1236: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:54:37.789 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:36.603358+0000 mon.a (mon.0) 3766 : cluster [DBG] osdmap e806: 8 total, 8 up, 4 in 2024-02-13T00:54:37.790 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:36.961186+0000 mgr.y (mgr.14544) 678 : audit [DBG] from='client.28212 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:54:37.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:36.558321+0000 mgr.y (mgr.14544) 677 : cluster [DBG] pgmap v1236: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:54:37.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:36.603358+0000 mon.a (mon.0) 3766 : cluster [DBG] osdmap e806: 8 total, 8 up, 4 in 2024-02-13T00:54:37.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:36.961186+0000 mgr.y (mgr.14544) 678 : audit [DBG] from='client.28212 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:54:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:36.558321+0000 mgr.y (mgr.14544) 677 : cluster [DBG] pgmap v1236: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:54:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:36.603358+0000 mon.a (mon.0) 3766 : cluster [DBG] osdmap e806: 8 total, 8 up, 4 in 2024-02-13T00:54:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:36.961186+0000 mgr.y (mgr.14544) 678 : audit [DBG] from='client.28212 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:54:38.615 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":1237,"stamp":"2024-02-13T00:54:36.600487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":2502,"num_read_kb":2222,"num_write":913,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":835,"num_bytes_recovered":21036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1077,"ondisk_log_size":1077,"up":492,"acting":492,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1255128,"kb_used_data":25144,"kb_used_omap":0,"kb_used_meta":1229952,"kb_avail":373725480,"statfs":{"total":383980142592,"available":382694891520,"internally_reserved":0,"allocated":25747456,"data_stored":13323511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1259470848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.626152"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":713,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978378+0000","last_change":"2024-02-13T00:51:41.586442+0000","last_active":"2024-02-13T00:54:28.978378+0000","last_peered":"2024-02-13T00:54:28.978378+0000","last_clean":"2024-02-13T00:54:28.978378+0000","last_became_active":"2024-02-13T00:51:41.585448+0000","last_became_peered":"2024-02-13T00:51:41.585448+0000","last_unstale":"2024-02-13T00:54:28.978378+0000","last_undegraded":"2024-02-13T00:54:28.978378+0000","last_fullsized":"2024-02-13T00:54:28.978378+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:32.100607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:50:32.100607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":636,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121222+0000","last_change":"2024-02-13T00:51:41.586812+0000","last_active":"2024-02-13T00:54:29.121222+0000","last_peered":"2024-02-13T00:54:29.121222+0000","last_clean":"2024-02-13T00:54:29.121222+0000","last_became_active":"2024-02-13T00:51:41.584445+0000","last_became_peered":"2024-02-13T00:51:41.584445+0000","last_unstale":"2024-02-13T00:54:29.121222+0000","last_undegraded":"2024-02-13T00:54:29.121222+0000","last_fullsized":"2024-02-13T00:54:29.121222+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:39.963625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:39.963625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":513,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.058806+0000","last_change":"2024-02-13T00:52:30.605704+0000","last_active":"2024-02-13T00:54:29.058806+0000","last_peered":"2024-02-13T00:54:29.058806+0000","last_clean":"2024-02-13T00:54:29.058806+0000","last_became_active":"2024-02-13T00:52:30.604888+0000","last_became_peered":"2024-02-13T00:52:30.604888+0000","last_unstale":"2024-02-13T00:54:29.058806+0000","last_undegraded":"2024-02-13T00:54:29.058806+0000","last_fullsized":"2024-02-13T00:54:29.058806+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:15.530059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:15.530059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":589,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.058753+0000","last_change":"2024-02-13T00:51:41.591561+0000","last_active":"2024-02-13T00:54:29.058753+0000","last_peered":"2024-02-13T00:54:29.058753+0000","last_clean":"2024-02-13T00:54:29.058753+0000","last_became_active":"2024-02-13T00:51:41.588863+0000","last_became_peered":"2024-02-13T00:51:41.588863+0000","last_unstale":"2024-02-13T00:54:29.058753+0000","last_undegraded":"2024-02-13T00:54:29.058753+0000","last_fullsized":"2024-02-13T00:54:29.058753+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:16.677431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:16.677431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":492,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122098+0000","last_change":"2024-02-13T00:52:30.601175+0000","last_active":"2024-02-13T00:54:29.122098+0000","last_peered":"2024-02-13T00:54:29.122098+0000","last_clean":"2024-02-13T00:54:29.122098+0000","last_became_active":"2024-02-13T00:52:30.600551+0000","last_became_peered":"2024-02-13T00:52:30.600551+0000","last_unstale":"2024-02-13T00:54:29.122098+0000","last_undegraded":"2024-02-13T00:54:29.122098+0000","last_fullsized":"2024-02-13T00:54:29.122098+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.469431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.469431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":89,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121176+0000","last_change":"2024-02-13T00:52:30.571356+0000","last_active":"2024-02-13T00:54:29.121176+0000","last_peered":"2024-02-13T00:54:29.121176+0000","last_clean":"2024-02-13T00:54:29.121176+0000","last_became_active":"2024-02-13T00:52:30.571021+0000","last_became_peered":"2024-02-13T00:52:30.571021+0000","last_unstale":"2024-02-13T00:54:29.121176+0000","last_undegraded":"2024-02-13T00:54:29.121176+0000","last_fullsized":"2024-02-13T00:54:29.121176+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:54.542201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:54.542201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":501,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122485+0000","last_change":"2024-02-13T00:51:40.927523+0000","last_active":"2024-02-13T00:54:29.122485+0000","last_peered":"2024-02-13T00:54:29.122485+0000","last_clean":"2024-02-13T00:54:29.122485+0000","last_became_active":"2024-02-13T00:46:01.459447+0000","last_became_peered":"2024-02-13T00:46:01.459447+0000","last_unstale":"2024-02-13T00:54:29.122485+0000","last_undegraded":"2024-02-13T00:54:29.122485+0000","last_fullsized":"2024-02-13T00:54:29.122485+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:40.927454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:40.927454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":764,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122926+0000","last_change":"2024-02-13T00:51:41.571583+0000","last_active":"2024-02-13T00:54:29.122926+0000","last_peered":"2024-02-13T00:54:29.122926+0000","last_clean":"2024-02-13T00:54:29.122926+0000","last_became_active":"2024-02-13T00:51:41.570041+0000","last_became_peered":"2024-02-13T00:51:41.570041+0000","last_unstale":"2024-02-13T00:54:29.122926+0000","last_undegraded":"2024-02-13T00:54:29.122926+0000","last_fullsized":"2024-02-13T00:54:29.122926+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:43:04.249704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":483,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.123124+0000","last_change":"2024-02-13T00:49:48.070260+0000","last_active":"2024-02-13T00:54:29.123124+0000","last_peered":"2024-02-13T00:54:29.123124+0000","last_clean":"2024-02-13T00:54:29.123124+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:54:29.123124+0000","last_undegraded":"2024-02-13T00:54:29.123124+0000","last_fullsized":"2024-02-13T00:54:29.123124+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:48.070195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:49:48.070195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":580,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060162+0000","last_change":"2024-02-13T00:51:41.569866+0000","last_active":"2024-02-13T00:54:29.060162+0000","last_peered":"2024-02-13T00:54:29.060162+0000","last_clean":"2024-02-13T00:54:29.060162+0000","last_became_active":"2024-02-13T00:51:41.567986+0000","last_became_peered":"2024-02-13T00:51:41.567986+0000","last_unstale":"2024-02-13T00:54:29.060162+0000","last_undegraded":"2024-02-13T00:54:29.060162+0000","last_fullsized":"2024-02-13T00:54:29.060162+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:14.521241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:14.521241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":732,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978418+0000","last_change":"2024-02-13T00:52:30.593903+0000","last_active":"2024-02-13T00:54:28.978418+0000","last_peered":"2024-02-13T00:54:28.978418+0000","last_clean":"2024-02-13T00:54:28.978418+0000","last_became_active":"2024-02-13T00:52:30.593515+0000","last_became_peered":"2024-02-13T00:52:30.593515+0000","last_unstale":"2024-02-13T00:54:28.978418+0000","last_undegraded":"2024-02-13T00:54:28.978418+0000","last_fullsized":"2024-02-13T00:54:28.978418+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:49.069014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:49.069014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"728'22","reported_seq":699,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:30.277293+0000","last_change":"2024-02-13T00:53:10.300922+0000","last_active":"2024-02-13T00:54:30.277293+0000","last_peered":"2024-02-13T00:54:30.277293+0000","last_clean":"2024-02-13T00:54:30.277293+0000","last_became_active":"2024-02-13T00:52:35.581849+0000","last_became_peered":"2024-02-13T00:52:35.581849+0000","last_unstale":"2024-02-13T00:54:30.277293+0000","last_undegraded":"2024-02-13T00:54:30.277293+0000","last_fullsized":"2024-02-13T00:54:30.277293+0000","mapping_epoch":727,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"728'22","last_scrub_stamp":"2024-02-13T00:53:10.300843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:53:10.300843+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":885,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121092+0000","last_change":"2024-02-13T00:53:48.410561+0000","last_active":"2024-02-13T00:54:29.121092+0000","last_peered":"2024-02-13T00:54:29.121092+0000","last_clean":"2024-02-13T00:54:29.121092+0000","last_became_active":"2024-02-13T00:52:30.577711+0000","last_became_peered":"2024-02-13T00:52:30.577711+0000","last_unstale":"2024-02-13T00:54:29.121092+0000","last_undegraded":"2024-02-13T00:54:29.121092+0000","last_fullsized":"2024-02-13T00:54:29.121092+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'48","last_scrub_stamp":"2024-02-13T00:53:48.410464+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:53:48.410464+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":801,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122741+0000","last_change":"2024-02-13T00:51:41.571304+0000","last_active":"2024-02-13T00:54:29.122741+0000","last_peered":"2024-02-13T00:54:29.122741+0000","last_clean":"2024-02-13T00:54:29.122741+0000","last_became_active":"2024-02-13T00:51:41.569518+0000","last_became_peered":"2024-02-13T00:51:41.569518+0000","last_unstale":"2024-02-13T00:54:29.122741+0000","last_undegraded":"2024-02-13T00:54:29.122741+0000","last_fullsized":"2024-02-13T00:54:29.122741+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:23.379773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:23.379773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":703,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121527+0000","last_change":"2024-02-13T00:53:57.460642+0000","last_active":"2024-02-13T00:54:29.121527+0000","last_peered":"2024-02-13T00:54:29.121527+0000","last_clean":"2024-02-13T00:54:29.121527+0000","last_became_active":"2024-02-13T00:46:01.451454+0000","last_became_peered":"2024-02-13T00:46:01.451454+0000","last_unstale":"2024-02-13T00:54:29.121527+0000","last_undegraded":"2024-02-13T00:54:29.121527+0000","last_fullsized":"2024-02-13T00:54:29.121527+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:57.460508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:53:57.460508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":750,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122205+0000","last_change":"2024-02-13T00:52:30.190029+0000","last_active":"2024-02-13T00:54:29.122205+0000","last_peered":"2024-02-13T00:54:29.122205+0000","last_clean":"2024-02-13T00:54:29.122205+0000","last_became_active":"2024-02-13T00:51:41.573640+0000","last_became_peered":"2024-02-13T00:51:41.573640+0000","last_unstale":"2024-02-13T00:54:29.122205+0000","last_undegraded":"2024-02-13T00:54:29.122205+0000","last_fullsized":"2024-02-13T00:54:29.122205+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:30.189925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:30.189925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":823,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979199+0000","last_change":"2024-02-13T00:52:31.041126+0000","last_active":"2024-02-13T00:54:28.979199+0000","last_peered":"2024-02-13T00:54:28.979199+0000","last_clean":"2024-02-13T00:54:28.979199+0000","last_became_active":"2024-02-13T00:52:30.591236+0000","last_became_peered":"2024-02-13T00:52:30.591236+0000","last_unstale":"2024-02-13T00:54:28.979199+0000","last_undegraded":"2024-02-13T00:54:28.979199+0000","last_fullsized":"2024-02-13T00:54:28.979199+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'21","last_scrub_stamp":"2024-02-13T00:52:28.988459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:28.988459+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":795,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060150+0000","last_change":"2024-02-13T00:53:49.468107+0000","last_active":"2024-02-13T00:54:29.060150+0000","last_peered":"2024-02-13T00:54:29.060150+0000","last_clean":"2024-02-13T00:54:29.060150+0000","last_became_active":"2024-02-13T00:52:30.603634+0000","last_became_peered":"2024-02-13T00:52:30.603634+0000","last_unstale":"2024-02-13T00:54:29.060150+0000","last_undegraded":"2024-02-13T00:54:29.060150+0000","last_fullsized":"2024-02-13T00:54:29.060150+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:49.468037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:53:49.468037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":666,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979060+0000","last_change":"2024-02-13T00:52:30.587164+0000","last_active":"2024-02-13T00:54:28.979060+0000","last_peered":"2024-02-13T00:54:28.979060+0000","last_clean":"2024-02-13T00:54:28.979060+0000","last_became_active":"2024-02-13T00:52:30.586877+0000","last_became_peered":"2024-02-13T00:52:30.586877+0000","last_unstale":"2024-02-13T00:54:28.979060+0000","last_undegraded":"2024-02-13T00:54:28.979060+0000","last_fullsized":"2024-02-13T00:54:28.979060+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:39.788783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:49:39.788783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":789,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978688+0000","last_change":"2024-02-13T00:51:41.586644+0000","last_active":"2024-02-13T00:54:28.978688+0000","last_peered":"2024-02-13T00:54:28.978688+0000","last_clean":"2024-02-13T00:54:28.978688+0000","last_became_active":"2024-02-13T00:51:41.585885+0000","last_became_peered":"2024-02-13T00:51:41.585885+0000","last_unstale":"2024-02-13T00:54:28.978688+0000","last_undegraded":"2024-02-13T00:54:28.978688+0000","last_fullsized":"2024-02-13T00:54:28.978688+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:41.837810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:49:41.837810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":415,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978654+0000","last_change":"2024-02-13T00:52:30.588176+0000","last_active":"2024-02-13T00:54:28.978654+0000","last_peered":"2024-02-13T00:54:28.978654+0000","last_clean":"2024-02-13T00:54:28.978654+0000","last_became_active":"2024-02-13T00:52:30.587089+0000","last_became_peered":"2024-02-13T00:52:30.587089+0000","last_unstale":"2024-02-13T00:54:28.978654+0000","last_undegraded":"2024-02-13T00:54:28.978654+0000","last_fullsized":"2024-02-13T00:54:28.978654+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:34.184091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:34.184091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":622,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121836+0000","last_change":"2024-02-13T00:51:43.825729+0000","last_active":"2024-02-13T00:54:29.121836+0000","last_peered":"2024-02-13T00:54:29.121836+0000","last_clean":"2024-02-13T00:54:29.121836+0000","last_became_active":"2024-02-13T00:51:41.573050+0000","last_became_peered":"2024-02-13T00:51:41.573050+0000","last_unstale":"2024-02-13T00:54:29.121836+0000","last_undegraded":"2024-02-13T00:54:29.121836+0000","last_fullsized":"2024-02-13T00:54:29.121836+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'51","last_scrub_stamp":"2024-02-13T00:51:33.100282+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:51:33.100282+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":88,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979241+0000","last_change":"2024-02-13T00:52:30.594905+0000","last_active":"2024-02-13T00:54:28.979241+0000","last_peered":"2024-02-13T00:54:28.979241+0000","last_clean":"2024-02-13T00:54:28.979241+0000","last_became_active":"2024-02-13T00:52:30.594543+0000","last_became_peered":"2024-02-13T00:52:30.594543+0000","last_unstale":"2024-02-13T00:54:28.979241+0000","last_undegraded":"2024-02-13T00:54:28.979241+0000","last_fullsized":"2024-02-13T00:54:28.979241+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:51.052139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:51:51.052139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":525,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978803+0000","last_change":"2024-02-13T00:51:41.587744+0000","last_active":"2024-02-13T00:54:28.978803+0000","last_peered":"2024-02-13T00:54:28.978803+0000","last_clean":"2024-02-13T00:54:28.978803+0000","last_became_active":"2024-02-13T00:51:41.582908+0000","last_became_peered":"2024-02-13T00:51:41.582908+0000","last_unstale":"2024-02-13T00:54:28.978803+0000","last_undegraded":"2024-02-13T00:54:28.978803+0000","last_fullsized":"2024-02-13T00:54:28.978803+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:14.244950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:51:14.244950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":628,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121209+0000","last_change":"2024-02-13T00:51:41.586157+0000","last_active":"2024-02-13T00:54:29.121209+0000","last_peered":"2024-02-13T00:54:29.121209+0000","last_clean":"2024-02-13T00:54:29.121209+0000","last_became_active":"2024-02-13T00:51:41.583542+0000","last_became_peered":"2024-02-13T00:51:41.583542+0000","last_unstale":"2024-02-13T00:54:29.121209+0000","last_undegraded":"2024-02-13T00:54:29.121209+0000","last_fullsized":"2024-02-13T00:54:29.121209+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:15.285536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:15.285536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":726,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059966+0000","last_change":"2024-02-13T00:51:35.505365+0000","last_active":"2024-02-13T00:54:29.059966+0000","last_peered":"2024-02-13T00:54:29.059966+0000","last_clean":"2024-02-13T00:54:29.059966+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:54:29.059966+0000","last_undegraded":"2024-02-13T00:54:29.059966+0000","last_fullsized":"2024-02-13T00:54:29.059966+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:35.505303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:35.505303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"467'31","reported_seq":590,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060058+0000","last_change":"2024-02-13T00:53:50.449803+0000","last_active":"2024-02-13T00:54:29.060058+0000","last_peered":"2024-02-13T00:54:29.060058+0000","last_clean":"2024-02-13T00:54:29.060058+0000","last_became_active":"2024-02-13T00:52:30.589869+0000","last_became_peered":"2024-02-13T00:52:30.589869+0000","last_unstale":"2024-02-13T00:54:29.060058+0000","last_undegraded":"2024-02-13T00:54:29.060058+0000","last_fullsized":"2024-02-13T00:54:29.060058+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'31","last_scrub_stamp":"2024-02-13T00:53:50.449723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:53:50.449723+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":600,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121598+0000","last_change":"2024-02-13T00:52:33.259627+0000","last_active":"2024-02-13T00:54:29.121598+0000","last_peered":"2024-02-13T00:54:29.121598+0000","last_clean":"2024-02-13T00:54:29.121598+0000","last_became_active":"2024-02-13T00:52:30.596010+0000","last_became_peered":"2024-02-13T00:52:30.596010+0000","last_unstale":"2024-02-13T00:54:29.121598+0000","last_undegraded":"2024-02-13T00:54:29.121598+0000","last_fullsized":"2024-02-13T00:54:29.121598+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:52:25.149943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:52:25.149943+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":793,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060095+0000","last_change":"2024-02-13T00:53:53.433619+0000","last_active":"2024-02-13T00:54:29.060095+0000","last_peered":"2024-02-13T00:54:29.060095+0000","last_clean":"2024-02-13T00:54:29.060095+0000","last_became_active":"2024-02-13T00:52:30.588539+0000","last_became_peered":"2024-02-13T00:52:30.588539+0000","last_unstale":"2024-02-13T00:54:29.060095+0000","last_undegraded":"2024-02-13T00:54:29.060095+0000","last_fullsized":"2024-02-13T00:54:29.060095+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:53.433519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_clean_scrub_stamp":"2024-02-13T00:53:53.433519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":777,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122778+0000","last_change":"2024-02-13T00:52:30.600848+0000","last_active":"2024-02-13T00:54:29.122778+0000","last_peered":"2024-02-13T00:54:29.122778+0000","last_clean":"2024-02-13T00:54:29.122778+0000","last_became_active":"2024-02-13T00:52:30.600493+0000","last_became_peered":"2024-02-13T00:52:30.600493+0000","last_unstale":"2024-02-13T00:54:29.122778+0000","last_undegraded":"2024-02-13T00:54:29.122778+0000","last_fullsized":"2024-02-13T00:54:29.122778+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:38.996288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:38.996288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":441,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979325+0000","last_change":"2024-02-13T00:52:30.597329+0000","last_active":"2024-02-13T00:54:28.979325+0000","last_peered":"2024-02-13T00:54:28.979325+0000","last_clean":"2024-02-13T00:54:28.979325+0000","last_became_active":"2024-02-13T00:52:30.595363+0000","last_became_peered":"2024-02-13T00:52:30.595363+0000","last_unstale":"2024-02-13T00:54:28.979325+0000","last_undegraded":"2024-02-13T00:54:28.979325+0000","last_fullsized":"2024-02-13T00:54:28.979325+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:22.925604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:22.925604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":531,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.123125+0000","last_change":"2024-02-13T00:52:32.337869+0000","last_active":"2024-02-13T00:54:29.123125+0000","last_peered":"2024-02-13T00:54:29.123125+0000","last_clean":"2024-02-13T00:54:29.123125+0000","last_became_active":"2024-02-13T00:52:30.584756+0000","last_became_peered":"2024-02-13T00:52:30.584756+0000","last_unstale":"2024-02-13T00:54:29.123125+0000","last_undegraded":"2024-02-13T00:54:29.123125+0000","last_fullsized":"2024-02-13T00:54:29.123125+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:49:03.294499+0000","last_deep_scrub":"85'15","last_deep_scrub_stamp":"2024-02-13T00:39:28.305179+0000","last_clean_scrub_stamp":"2024-02-13T00:49:03.294499+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":799,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059287+0000","last_change":"2024-02-13T00:51:41.573970+0000","last_active":"2024-02-13T00:54:29.059287+0000","last_peered":"2024-02-13T00:54:29.059287+0000","last_clean":"2024-02-13T00:54:29.059287+0000","last_became_active":"2024-02-13T00:51:41.569525+0000","last_became_peered":"2024-02-13T00:51:41.569525+0000","last_unstale":"2024-02-13T00:54:29.059287+0000","last_undegraded":"2024-02-13T00:54:29.059287+0000","last_fullsized":"2024-02-13T00:54:29.059287+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:59.111828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:48:59.111828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"726'19","reported_seq":614,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:30.276276+0000","last_change":"2024-02-13T00:52:33.561456+0000","last_active":"2024-02-13T00:54:30.276276+0000","last_peered":"2024-02-13T00:54:30.276276+0000","last_clean":"2024-02-13T00:54:30.276276+0000","last_became_active":"2024-02-13T00:52:33.561139+0000","last_became_peered":"2024-02-13T00:52:33.561139+0000","last_unstale":"2024-02-13T00:54:30.276276+0000","last_undegraded":"2024-02-13T00:54:30.276276+0000","last_fullsized":"2024-02-13T00:54:30.276276+0000","mapping_epoch":725,"log_start":"404'11","ondisk_log_start":"404'11","created":60,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'16","last_scrub_stamp":"2024-02-13T00:52:22.988904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:52:22.988904+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":788,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060314+0000","last_change":"2024-02-13T00:52:31.529160+0000","last_active":"2024-02-13T00:54:29.060314+0000","last_peered":"2024-02-13T00:54:29.060314+0000","last_clean":"2024-02-13T00:54:29.060314+0000","last_became_active":"2024-02-13T00:52:30.602716+0000","last_became_peered":"2024-02-13T00:52:30.602716+0000","last_unstale":"2024-02-13T00:54:29.060314+0000","last_undegraded":"2024-02-13T00:54:29.060314+0000","last_fullsized":"2024-02-13T00:54:29.060314+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:51:05.644558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:05.644558+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":589,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060598+0000","last_change":"2024-02-13T00:52:30.605308+0000","last_active":"2024-02-13T00:54:29.060598+0000","last_peered":"2024-02-13T00:54:29.060598+0000","last_clean":"2024-02-13T00:54:29.060598+0000","last_became_active":"2024-02-13T00:52:30.604615+0000","last_became_peered":"2024-02-13T00:52:30.604615+0000","last_unstale":"2024-02-13T00:54:29.060598+0000","last_undegraded":"2024-02-13T00:54:29.060598+0000","last_fullsized":"2024-02-13T00:54:29.060598+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:26.830270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:26.830270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"467'27","reported_seq":929,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060518+0000","last_change":"2024-02-13T00:52:32.788019+0000","last_active":"2024-02-13T00:54:29.060518+0000","last_peered":"2024-02-13T00:54:29.060518+0000","last_clean":"2024-02-13T00:54:29.060518+0000","last_became_active":"2024-02-13T00:52:30.603369+0000","last_became_peered":"2024-02-13T00:52:30.603369+0000","last_unstale":"2024-02-13T00:54:29.060518+0000","last_undegraded":"2024-02-13T00:54:29.060518+0000","last_fullsized":"2024-02-13T00:54:29.060518+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:48:54.043088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:54.043088+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":835,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059067+0000","last_change":"2024-02-13T00:51:43.912749+0000","last_active":"2024-02-13T00:54:29.059067+0000","last_peered":"2024-02-13T00:54:29.059067+0000","last_clean":"2024-02-13T00:54:29.059067+0000","last_became_active":"2024-02-13T00:51:41.588865+0000","last_became_peered":"2024-02-13T00:51:41.588865+0000","last_unstale":"2024-02-13T00:54:29.059067+0000","last_undegraded":"2024-02-13T00:54:29.059067+0000","last_fullsized":"2024-02-13T00:54:29.059067+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:49:37.053398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:37.053398+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"727'19","reported_seq":624,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:30.276743+0000","last_change":"2024-02-13T00:53:08.072994+0000","last_active":"2024-02-13T00:54:30.276743+0000","last_peered":"2024-02-13T00:54:30.276743+0000","last_clean":"2024-02-13T00:54:30.276743+0000","last_became_active":"2024-02-13T00:52:34.566362+0000","last_became_peered":"2024-02-13T00:52:34.566362+0000","last_unstale":"2024-02-13T00:54:30.276743+0000","last_undegraded":"2024-02-13T00:54:30.276743+0000","last_fullsized":"2024-02-13T00:54:30.276743+0000","mapping_epoch":726,"log_start":"467'11","ondisk_log_start":"467'11","created":60,"last_epoch_clean":727,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'19","last_scrub_stamp":"2024-02-13T00:53:08.072916+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:53:08.072916+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":412,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059791+0000","last_change":"2024-02-13T00:50:19.734411+0000","last_active":"2024-02-13T00:54:29.059791+0000","last_peered":"2024-02-13T00:54:29.059791+0000","last_clean":"2024-02-13T00:54:29.059791+0000","last_became_active":"2024-02-13T00:46:01.453101+0000","last_became_peered":"2024-02-13T00:46:01.453101+0000","last_unstale":"2024-02-13T00:54:29.059791+0000","last_undegraded":"2024-02-13T00:54:29.059791+0000","last_fullsized":"2024-02-13T00:54:29.059791+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:19.734320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:50:19.734320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":723,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.980164+0000","last_change":"2024-02-13T00:51:11.239012+0000","last_active":"2024-02-13T00:54:28.980164+0000","last_peered":"2024-02-13T00:54:28.980164+0000","last_clean":"2024-02-13T00:54:28.980164+0000","last_became_active":"2024-02-13T00:46:01.455536+0000","last_became_peered":"2024-02-13T00:46:01.455536+0000","last_unstale":"2024-02-13T00:54:28.980164+0000","last_undegraded":"2024-02-13T00:54:28.980164+0000","last_fullsized":"2024-02-13T00:54:28.980164+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:11.238914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:51:11.238914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":532,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122950+0000","last_change":"2024-02-13T00:52:32.853997+0000","last_active":"2024-02-13T00:54:29.122950+0000","last_peered":"2024-02-13T00:54:29.122950+0000","last_clean":"2024-02-13T00:54:29.122950+0000","last_became_active":"2024-02-13T00:52:30.593631+0000","last_became_peered":"2024-02-13T00:52:30.593631+0000","last_unstale":"2024-02-13T00:54:29.122950+0000","last_undegraded":"2024-02-13T00:54:29.122950+0000","last_fullsized":"2024-02-13T00:54:29.122950+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:57.386254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:57.386254+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":615,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978258+0000","last_change":"2024-02-13T00:51:53.090015+0000","last_active":"2024-02-13T00:54:28.978258+0000","last_peered":"2024-02-13T00:54:28.978258+0000","last_clean":"2024-02-13T00:54:28.978258+0000","last_became_active":"2024-02-13T00:51:41.582606+0000","last_became_peered":"2024-02-13T00:51:41.582606+0000","last_unstale":"2024-02-13T00:54:28.978258+0000","last_undegraded":"2024-02-13T00:54:28.978258+0000","last_fullsized":"2024-02-13T00:54:28.978258+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:53.089946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:51:53.089946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"728'19","reported_seq":821,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:30.276612+0000","last_change":"2024-02-13T00:52:35.610840+0000","last_active":"2024-02-13T00:54:30.276612+0000","last_peered":"2024-02-13T00:54:30.276612+0000","last_clean":"2024-02-13T00:54:30.276612+0000","last_became_active":"2024-02-13T00:52:35.610249+0000","last_became_peered":"2024-02-13T00:52:35.610249+0000","last_unstale":"2024-02-13T00:54:30.276612+0000","last_undegraded":"2024-02-13T00:54:30.276612+0000","last_fullsized":"2024-02-13T00:54:30.276612+0000","mapping_epoch":727,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'16","last_scrub_stamp":"2024-02-13T00:49:45.051539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:49:45.051539+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":510,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122950+0000","last_change":"2024-02-13T00:51:41.570899+0000","last_active":"2024-02-13T00:54:29.122950+0000","last_peered":"2024-02-13T00:54:29.122950+0000","last_clean":"2024-02-13T00:54:29.122950+0000","last_became_active":"2024-02-13T00:51:41.568351+0000","last_became_peered":"2024-02-13T00:51:41.568351+0000","last_unstale":"2024-02-13T00:54:29.122950+0000","last_undegraded":"2024-02-13T00:54:29.122950+0000","last_fullsized":"2024-02-13T00:54:29.122950+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:31.463442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:31.463442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":698,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978082+0000","last_change":"2024-02-13T00:53:55.015887+0000","last_active":"2024-02-13T00:54:28.978082+0000","last_peered":"2024-02-13T00:54:28.978082+0000","last_clean":"2024-02-13T00:54:28.978082+0000","last_became_active":"2024-02-13T00:52:30.594189+0000","last_became_peered":"2024-02-13T00:52:30.594189+0000","last_unstale":"2024-02-13T00:54:28.978082+0000","last_undegraded":"2024-02-13T00:54:28.978082+0000","last_fullsized":"2024-02-13T00:54:28.978082+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:55.015820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:53:55.015820+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"726'11","reported_seq":869,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.983103+0000","last_change":"2024-02-13T00:53:12.200136+0000","last_active":"2024-02-13T00:54:29.983103+0000","last_peered":"2024-02-13T00:54:29.983103+0000","last_clean":"2024-02-13T00:54:29.983103+0000","last_became_active":"2024-02-13T00:52:30.582279+0000","last_became_peered":"2024-02-13T00:52:30.582279+0000","last_unstale":"2024-02-13T00:54:29.983103+0000","last_undegraded":"2024-02-13T00:54:29.983103+0000","last_fullsized":"2024-02-13T00:54:29.983103+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'11","last_scrub_stamp":"2024-02-13T00:53:12.200063+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:53:12.200063+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":800,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121365+0000","last_change":"2024-02-13T00:52:31.298166+0000","last_active":"2024-02-13T00:54:29.121365+0000","last_peered":"2024-02-13T00:54:29.121365+0000","last_clean":"2024-02-13T00:54:29.121365+0000","last_became_active":"2024-02-13T00:52:30.579027+0000","last_became_peered":"2024-02-13T00:52:30.579027+0000","last_unstale":"2024-02-13T00:54:29.121365+0000","last_undegraded":"2024-02-13T00:54:29.121365+0000","last_fullsized":"2024-02-13T00:54:29.121365+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'34","last_scrub_stamp":"2024-02-13T00:49:46.101874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.101874+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":510,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.123007+0000","last_change":"2024-02-13T00:51:41.572212+0000","last_active":"2024-02-13T00:54:29.123007+0000","last_peered":"2024-02-13T00:54:29.123007+0000","last_clean":"2024-02-13T00:54:29.123007+0000","last_became_active":"2024-02-13T00:51:41.569996+0000","last_became_peered":"2024-02-13T00:51:41.569996+0000","last_unstale":"2024-02-13T00:54:29.123007+0000","last_undegraded":"2024-02-13T00:54:29.123007+0000","last_fullsized":"2024-02-13T00:54:29.123007+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:47.075741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.075741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":510,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122195+0000","last_change":"2024-02-13T00:51:41.572697+0000","last_active":"2024-02-13T00:54:29.122195+0000","last_peered":"2024-02-13T00:54:29.122195+0000","last_clean":"2024-02-13T00:54:29.122195+0000","last_became_active":"2024-02-13T00:51:41.569841+0000","last_became_peered":"2024-02-13T00:51:41.569841+0000","last_unstale":"2024-02-13T00:54:29.122195+0000","last_undegraded":"2024-02-13T00:54:29.122195+0000","last_fullsized":"2024-02-13T00:54:29.122195+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:17.268189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:17.268189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":534,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978212+0000","last_change":"2024-02-13T00:51:41.585096+0000","last_active":"2024-02-13T00:54:28.978212+0000","last_peered":"2024-02-13T00:54:28.978212+0000","last_clean":"2024-02-13T00:54:28.978212+0000","last_became_active":"2024-02-13T00:51:41.582942+0000","last_became_peered":"2024-02-13T00:51:41.582942+0000","last_unstale":"2024-02-13T00:54:28.978212+0000","last_undegraded":"2024-02-13T00:54:28.978212+0000","last_fullsized":"2024-02-13T00:54:28.978212+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:36.841508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:36.841508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":589,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978367+0000","last_change":"2024-02-13T00:53:14.066845+0000","last_active":"2024-02-13T00:54:28.978367+0000","last_peered":"2024-02-13T00:54:28.978367+0000","last_clean":"2024-02-13T00:54:28.978367+0000","last_became_active":"2024-02-13T00:52:30.594613+0000","last_became_peered":"2024-02-13T00:52:30.594613+0000","last_unstale":"2024-02-13T00:54:28.978367+0000","last_undegraded":"2024-02-13T00:54:28.978367+0000","last_fullsized":"2024-02-13T00:54:28.978367+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:14.066781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:53:14.066781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"467'42","reported_seq":763,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978229+0000","last_change":"2024-02-13T00:53:19.085386+0000","last_active":"2024-02-13T00:54:28.978229+0000","last_peered":"2024-02-13T00:54:28.978229+0000","last_clean":"2024-02-13T00:54:28.978229+0000","last_became_active":"2024-02-13T00:51:41.582068+0000","last_became_peered":"2024-02-13T00:51:41.582068+0000","last_unstale":"2024-02-13T00:54:28.978229+0000","last_undegraded":"2024-02-13T00:54:28.978229+0000","last_fullsized":"2024-02-13T00:54:28.978229+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'42","last_scrub_stamp":"2024-02-13T00:53:19.085318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:53:19.085318+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":804,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060209+0000","last_change":"2024-02-13T00:52:30.582723+0000","last_active":"2024-02-13T00:54:29.060209+0000","last_peered":"2024-02-13T00:54:29.060209+0000","last_clean":"2024-02-13T00:54:29.060209+0000","last_became_active":"2024-02-13T00:52:30.582296+0000","last_became_peered":"2024-02-13T00:52:30.582296+0000","last_unstale":"2024-02-13T00:54:29.060209+0000","last_undegraded":"2024-02-13T00:54:29.060209+0000","last_fullsized":"2024-02-13T00:54:29.060209+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:29.412336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:52:29.412336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":749,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121539+0000","last_change":"2024-02-13T00:52:30.595054+0000","last_active":"2024-02-13T00:54:29.121539+0000","last_peered":"2024-02-13T00:54:29.121539+0000","last_clean":"2024-02-13T00:54:29.121539+0000","last_became_active":"2024-02-13T00:52:30.593405+0000","last_became_peered":"2024-02-13T00:52:30.593405+0000","last_unstale":"2024-02-13T00:54:29.121539+0000","last_undegraded":"2024-02-13T00:54:29.121539+0000","last_fullsized":"2024-02-13T00:54:29.121539+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:41:02.817243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:41:02.817243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":732,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121752+0000","last_change":"2024-02-13T00:52:30.586680+0000","last_active":"2024-02-13T00:54:29.121752+0000","last_peered":"2024-02-13T00:54:29.121752+0000","last_clean":"2024-02-13T00:54:29.121752+0000","last_became_active":"2024-02-13T00:52:30.586041+0000","last_became_peered":"2024-02-13T00:52:30.586041+0000","last_unstale":"2024-02-13T00:54:29.121752+0000","last_undegraded":"2024-02-13T00:54:29.121752+0000","last_fullsized":"2024-02-13T00:54:29.121752+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_clean_scrub_stamp":"2024-02-13T00:48:35.434645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":506,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121884+0000","last_change":"2024-02-13T00:53:09.344998+0000","last_active":"2024-02-13T00:54:29.121884+0000","last_peered":"2024-02-13T00:54:29.121884+0000","last_clean":"2024-02-13T00:54:29.121884+0000","last_became_active":"2024-02-13T00:51:41.573763+0000","last_became_peered":"2024-02-13T00:51:41.573763+0000","last_unstale":"2024-02-13T00:54:29.121884+0000","last_undegraded":"2024-02-13T00:54:29.121884+0000","last_fullsized":"2024-02-13T00:54:29.121884+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:09.344923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:53:09.344923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":924,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978943+0000","last_change":"2024-02-13T00:52:26.954476+0000","last_active":"2024-02-13T00:54:28.978943+0000","last_peered":"2024-02-13T00:54:28.978943+0000","last_clean":"2024-02-13T00:54:28.978943+0000","last_became_active":"2024-02-13T00:51:41.587256+0000","last_became_peered":"2024-02-13T00:51:41.587256+0000","last_unstale":"2024-02-13T00:54:28.978943+0000","last_undegraded":"2024-02-13T00:54:28.978943+0000","last_fullsized":"2024-02-13T00:54:28.978943+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'44","last_scrub_stamp":"2024-02-13T00:52:26.954384+0000","last_deep_scrub":"80'34","last_deep_scrub_stamp":"2024-02-13T00:42:20.688132+0000","last_clean_scrub_stamp":"2024-02-13T00:52:26.954384+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":624,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979258+0000","last_change":"2024-02-13T00:52:30.596233+0000","last_active":"2024-02-13T00:54:28.979258+0000","last_peered":"2024-02-13T00:54:28.979258+0000","last_clean":"2024-02-13T00:54:28.979258+0000","last_became_active":"2024-02-13T00:52:30.595886+0000","last_became_peered":"2024-02-13T00:52:30.595886+0000","last_unstale":"2024-02-13T00:54:28.979258+0000","last_undegraded":"2024-02-13T00:54:28.979258+0000","last_fullsized":"2024-02-13T00:54:28.979258+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:25.001219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:25.001219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":811,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059068+0000","last_change":"2024-02-13T00:51:41.589718+0000","last_active":"2024-02-13T00:54:29.059068+0000","last_peered":"2024-02-13T00:54:29.059068+0000","last_clean":"2024-02-13T00:54:29.059068+0000","last_became_active":"2024-02-13T00:51:41.589295+0000","last_became_peered":"2024-02-13T00:51:41.589295+0000","last_unstale":"2024-02-13T00:54:29.059068+0000","last_undegraded":"2024-02-13T00:54:29.059068+0000","last_fullsized":"2024-02-13T00:54:29.059068+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.726660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.726660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":429,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060020+0000","last_change":"2024-02-13T00:54:23.237671+0000","last_active":"2024-02-13T00:54:29.060020+0000","last_peered":"2024-02-13T00:54:29.060020+0000","last_clean":"2024-02-13T00:54:29.060020+0000","last_became_active":"2024-02-13T00:52:30.582272+0000","last_became_peered":"2024-02-13T00:52:30.582272+0000","last_unstale":"2024-02-13T00:54:29.060020+0000","last_undegraded":"2024-02-13T00:54:29.060020+0000","last_fullsized":"2024-02-13T00:54:29.060020+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:54:23.237607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:54:23.237607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":444,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.061035+0000","last_change":"2024-02-13T00:53:14.241706+0000","last_active":"2024-02-13T00:54:29.061035+0000","last_peered":"2024-02-13T00:54:29.061035+0000","last_clean":"2024-02-13T00:54:29.061035+0000","last_became_active":"2024-02-13T00:52:30.604937+0000","last_became_peered":"2024-02-13T00:52:30.604937+0000","last_unstale":"2024-02-13T00:54:29.061035+0000","last_undegraded":"2024-02-13T00:54:29.061035+0000","last_fullsized":"2024-02-13T00:54:29.061035+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:14.241642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:53:14.241642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":913,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979972+0000","last_change":"2024-02-13T00:52:23.951273+0000","last_active":"2024-02-13T00:54:28.979972+0000","last_peered":"2024-02-13T00:54:28.979972+0000","last_clean":"2024-02-13T00:54:28.979972+0000","last_became_active":"2024-02-13T00:51:41.588905+0000","last_became_peered":"2024-02-13T00:51:41.588905+0000","last_unstale":"2024-02-13T00:54:28.979972+0000","last_undegraded":"2024-02-13T00:54:28.979972+0000","last_fullsized":"2024-02-13T00:54:28.979972+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:52:23.951206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:23.951206+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":512,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059053+0000","last_change":"2024-02-13T00:51:53.574669+0000","last_active":"2024-02-13T00:54:29.059053+0000","last_peered":"2024-02-13T00:54:29.059053+0000","last_clean":"2024-02-13T00:54:29.059053+0000","last_became_active":"2024-02-13T00:51:41.588531+0000","last_became_peered":"2024-02-13T00:51:41.588531+0000","last_unstale":"2024-02-13T00:54:29.059053+0000","last_undegraded":"2024-02-13T00:54:29.059053+0000","last_fullsized":"2024-02-13T00:54:29.059053+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:53.574605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:53.574605+0000","last_clean_scrub_stamp":"2024-02-13T00:51:53.574605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":800,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978469+0000","last_change":"2024-02-13T00:52:30.596032+0000","last_active":"2024-02-13T00:54:28.978469+0000","last_peered":"2024-02-13T00:54:28.978469+0000","last_clean":"2024-02-13T00:54:28.978469+0000","last_became_active":"2024-02-13T00:52:30.595531+0000","last_became_peered":"2024-02-13T00:52:30.595531+0000","last_unstale":"2024-02-13T00:54:28.978469+0000","last_undegraded":"2024-02-13T00:54:28.978469+0000","last_fullsized":"2024-02-13T00:54:28.978469+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:46.119431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:51:46.119431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":802,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060144+0000","last_change":"2024-02-13T00:52:30.589500+0000","last_active":"2024-02-13T00:54:29.060144+0000","last_peered":"2024-02-13T00:54:29.060144+0000","last_clean":"2024-02-13T00:54:29.060144+0000","last_became_active":"2024-02-13T00:52:30.588991+0000","last_became_peered":"2024-02-13T00:52:30.588991+0000","last_unstale":"2024-02-13T00:54:29.060144+0000","last_undegraded":"2024-02-13T00:54:29.060144+0000","last_fullsized":"2024-02-13T00:54:29.060144+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:16.535961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:51:16.535961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":652,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.123024+0000","last_change":"2024-02-13T00:52:30.603205+0000","last_active":"2024-02-13T00:54:29.123024+0000","last_peered":"2024-02-13T00:54:29.123024+0000","last_clean":"2024-02-13T00:54:29.123024+0000","last_became_active":"2024-02-13T00:52:30.600547+0000","last_became_peered":"2024-02-13T00:52:30.600547+0000","last_unstale":"2024-02-13T00:54:29.123024+0000","last_undegraded":"2024-02-13T00:54:29.123024+0000","last_fullsized":"2024-02-13T00:54:29.123024+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:23.124083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:23.124083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":456,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979044+0000","last_change":"2024-02-13T00:52:30.017699+0000","last_active":"2024-02-13T00:54:28.979044+0000","last_peered":"2024-02-13T00:54:28.979044+0000","last_clean":"2024-02-13T00:54:28.979044+0000","last_became_active":"2024-02-13T00:51:41.580760+0000","last_became_peered":"2024-02-13T00:51:41.580760+0000","last_unstale":"2024-02-13T00:54:28.979044+0000","last_undegraded":"2024-02-13T00:54:28.979044+0000","last_fullsized":"2024-02-13T00:54:28.979044+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:30.017618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:52:30.017618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":597,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122855+0000","last_change":"2024-02-13T00:53:22.462609+0000","last_active":"2024-02-13T00:54:29.122855+0000","last_peered":"2024-02-13T00:54:29.122855+0000","last_clean":"2024-02-13T00:54:29.122855+0000","last_became_active":"2024-02-13T00:52:30.596789+0000","last_became_peered":"2024-02-13T00:52:30.596789+0000","last_unstale":"2024-02-13T00:54:29.122855+0000","last_undegraded":"2024-02-13T00:54:29.122855+0000","last_fullsized":"2024-02-13T00:54:29.122855+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":727,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'33","last_scrub_stamp":"2024-02-13T00:53:22.462526+0000","last_deep_scrub":"467'33","last_deep_scrub_stamp":"2024-02-13T00:53:22.462526+0000","last_clean_scrub_stamp":"2024-02-13T00:53:22.462526+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":756,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978165+0000","last_change":"2024-02-13T00:52:36.042288+0000","last_active":"2024-02-13T00:54:28.978165+0000","last_peered":"2024-02-13T00:54:28.978165+0000","last_clean":"2024-02-13T00:54:28.978165+0000","last_became_active":"2024-02-13T00:51:41.582126+0000","last_became_peered":"2024-02-13T00:51:41.582126+0000","last_unstale":"2024-02-13T00:54:28.978165+0000","last_undegraded":"2024-02-13T00:54:28.978165+0000","last_fullsized":"2024-02-13T00:54:28.978165+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:36.042221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:41.255785+0000","last_clean_scrub_stamp":"2024-02-13T00:52:36.042221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":754,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121607+0000","last_change":"2024-02-13T00:53:56.472636+0000","last_active":"2024-02-13T00:54:29.121607+0000","last_peered":"2024-02-13T00:54:29.121607+0000","last_clean":"2024-02-13T00:54:29.121607+0000","last_became_active":"2024-02-13T00:51:41.571758+0000","last_became_peered":"2024-02-13T00:51:41.571758+0000","last_unstale":"2024-02-13T00:54:29.121607+0000","last_undegraded":"2024-02-13T00:54:29.121607+0000","last_fullsized":"2024-02-13T00:54:29.121607+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:56.472560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:53:56.472560+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":627,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:33.675346+0000","last_change":"2024-02-13T00:54:33.675346+0000","last_active":"2024-02-13T00:54:33.675346+0000","last_peered":"2024-02-13T00:54:33.675346+0000","last_clean":"2024-02-13T00:54:33.675346+0000","last_became_active":"2024-02-13T00:52:30.580091+0000","last_became_peered":"2024-02-13T00:52:30.580091+0000","last_unstale":"2024-02-13T00:54:33.675346+0000","last_undegraded":"2024-02-13T00:54:33.675346+0000","last_fullsized":"2024-02-13T00:54:33.675346+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:54:33.675270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:54:33.675270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":659,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060299+0000","last_change":"2024-02-13T00:52:30.604915+0000","last_active":"2024-02-13T00:54:29.060299+0000","last_peered":"2024-02-13T00:54:29.060299+0000","last_clean":"2024-02-13T00:54:29.060299+0000","last_became_active":"2024-02-13T00:52:30.603107+0000","last_became_peered":"2024-02-13T00:52:30.603107+0000","last_unstale":"2024-02-13T00:54:29.060299+0000","last_undegraded":"2024-02-13T00:54:29.060299+0000","last_fullsized":"2024-02-13T00:54:29.060299+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:19.521319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:51:19.521319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":741,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979833+0000","last_change":"2024-02-13T00:52:30.596982+0000","last_active":"2024-02-13T00:54:28.979833+0000","last_peered":"2024-02-13T00:54:28.979833+0000","last_clean":"2024-02-13T00:54:28.979833+0000","last_became_active":"2024-02-13T00:52:30.596402+0000","last_became_peered":"2024-02-13T00:52:30.596402+0000","last_unstale":"2024-02-13T00:54:28.979833+0000","last_undegraded":"2024-02-13T00:54:28.979833+0000","last_fullsized":"2024-02-13T00:54:28.979833+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:42.854585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:42.854585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":490,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059930+0000","last_change":"2024-02-13T00:54:25.247133+0000","last_active":"2024-02-13T00:54:29.059930+0000","last_peered":"2024-02-13T00:54:29.059930+0000","last_clean":"2024-02-13T00:54:29.059930+0000","last_became_active":"2024-02-13T00:52:30.581048+0000","last_became_peered":"2024-02-13T00:52:30.581048+0000","last_unstale":"2024-02-13T00:54:29.059930+0000","last_undegraded":"2024-02-13T00:54:29.059930+0000","last_fullsized":"2024-02-13T00:54:29.059930+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:54:25.247064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:54:25.247064+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"805'29","reported_seq":866,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:31.218128+0000","last_change":"2024-02-13T00:54:31.218128+0000","last_active":"2024-02-13T00:54:31.218128+0000","last_peered":"2024-02-13T00:54:31.218128+0000","last_clean":"2024-02-13T00:54:31.218128+0000","last_became_active":"2024-02-13T00:51:41.569273+0000","last_became_peered":"2024-02-13T00:51:41.569273+0000","last_unstale":"2024-02-13T00:54:31.218128+0000","last_undegraded":"2024-02-13T00:54:31.218128+0000","last_fullsized":"2024-02-13T00:54:31.218128+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"805'29","last_scrub_stamp":"2024-02-13T00:54:31.218066+0000","last_deep_scrub":"805'29","last_deep_scrub_stamp":"2024-02-13T00:54:31.218066+0000","last_clean_scrub_stamp":"2024-02-13T00:54:31.218066+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":697,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978444+0000","last_change":"2024-02-13T00:49:46.851558+0000","last_active":"2024-02-13T00:54:28.978444+0000","last_peered":"2024-02-13T00:54:28.978444+0000","last_clean":"2024-02-13T00:54:28.978444+0000","last_became_active":"2024-02-13T00:44:50.425300+0000","last_became_peered":"2024-02-13T00:44:50.425300+0000","last_unstale":"2024-02-13T00:54:28.978444+0000","last_undegraded":"2024-02-13T00:54:28.978444+0000","last_fullsized":"2024-02-13T00:54:28.978444+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.851486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.851486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":1816,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:33.400342+0000","last_change":"2024-02-13T00:51:38.527164+0000","last_active":"2024-02-13T00:54:33.400342+0000","last_peered":"2024-02-13T00:54:33.400342+0000","last_clean":"2024-02-13T00:54:33.400342+0000","last_became_active":"2024-02-13T00:44:50.442214+0000","last_became_peered":"2024-02-13T00:44:50.442214+0000","last_unstale":"2024-02-13T00:54:33.400342+0000","last_undegraded":"2024-02-13T00:54:33.400342+0000","last_fullsized":"2024-02-13T00:54:33.400342+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:51:38.527084+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:45:54.075194+0000","last_clean_scrub_stamp":"2024-02-13T00:51:38.527084+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1000,"num_read_kb":995,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":514,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122494+0000","last_change":"2024-02-13T00:51:41.571352+0000","last_active":"2024-02-13T00:54:29.122494+0000","last_peered":"2024-02-13T00:54:29.122494+0000","last_clean":"2024-02-13T00:54:29.122494+0000","last_became_active":"2024-02-13T00:51:41.570686+0000","last_became_peered":"2024-02-13T00:51:41.570686+0000","last_unstale":"2024-02-13T00:54:29.122494+0000","last_undegraded":"2024-02-13T00:54:29.122494+0000","last_fullsized":"2024-02-13T00:54:29.122494+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:40:51.198070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:40:51.198070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":584,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059897+0000","last_change":"2024-02-13T00:52:30.604046+0000","last_active":"2024-02-13T00:54:29.059897+0000","last_peered":"2024-02-13T00:54:29.059897+0000","last_clean":"2024-02-13T00:54:29.059897+0000","last_became_active":"2024-02-13T00:52:30.603400+0000","last_became_peered":"2024-02-13T00:52:30.603400+0000","last_unstale":"2024-02-13T00:54:29.059897+0000","last_undegraded":"2024-02-13T00:54:29.059897+0000","last_fullsized":"2024-02-13T00:54:29.059897+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:37.545088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:37.545088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":88,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978918+0000","last_change":"2024-02-13T00:52:30.575376+0000","last_active":"2024-02-13T00:54:28.978918+0000","last_peered":"2024-02-13T00:54:28.978918+0000","last_clean":"2024-02-13T00:54:28.978918+0000","last_became_active":"2024-02-13T00:52:30.575153+0000","last_became_peered":"2024-02-13T00:52:30.575153+0000","last_unstale":"2024-02-13T00:54:28.978918+0000","last_undegraded":"2024-02-13T00:54:28.978918+0000","last_fullsized":"2024-02-13T00:54:28.978918+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:26.113119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:52:26.113119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"701'67","reported_seq":164,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.985400+0000","last_change":"2024-02-13T00:52:09.459680+0000","last_active":"2024-02-13T00:54:29.985400+0000","last_peered":"2024-02-13T00:54:29.985400+0000","last_clean":"2024-02-13T00:54:29.985400+0000","last_became_active":"2024-02-13T00:52:07.565910+0000","last_became_peered":"2024-02-13T00:52:07.565910+0000","last_unstale":"2024-02-13T00:54:29.985400+0000","last_undegraded":"2024-02-13T00:54:29.985400+0000","last_fullsized":"2024-02-13T00:54:29.985400+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'65","last_scrub_stamp":"2024-02-13T00:49:47.928548+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:49:47.928548+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":77,"num_read_kb":61,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":2864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":711,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059203+0000","last_change":"2024-02-13T00:50:37.777963+0000","last_active":"2024-02-13T00:54:29.059203+0000","last_peered":"2024-02-13T00:54:29.059203+0000","last_clean":"2024-02-13T00:54:29.059203+0000","last_became_active":"2024-02-13T00:46:01.452240+0000","last_became_peered":"2024-02-13T00:46:01.452240+0000","last_unstale":"2024-02-13T00:54:29.059203+0000","last_undegraded":"2024-02-13T00:54:29.059203+0000","last_fullsized":"2024-02-13T00:54:29.059203+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:37.777891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:50:37.777891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":340,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979808+0000","last_change":"2024-02-13T00:52:30.598078+0000","last_active":"2024-02-13T00:54:28.979808+0000","last_peered":"2024-02-13T00:54:28.979808+0000","last_clean":"2024-02-13T00:54:28.979808+0000","last_became_active":"2024-02-13T00:52:30.595326+0000","last_became_peered":"2024-02-13T00:52:30.595326+0000","last_unstale":"2024-02-13T00:54:28.979808+0000","last_undegraded":"2024-02-13T00:54:28.979808+0000","last_fullsized":"2024-02-13T00:54:28.979808+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:42:25.457290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:42:25.457290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":492,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121975+0000","last_change":"2024-02-13T00:48:17.340891+0000","last_active":"2024-02-13T00:54:29.121975+0000","last_peered":"2024-02-13T00:54:29.121975+0000","last_clean":"2024-02-13T00:54:29.121975+0000","last_became_active":"2024-02-13T00:44:50.439048+0000","last_became_peered":"2024-02-13T00:44:50.439048+0000","last_unstale":"2024-02-13T00:54:29.121975+0000","last_undegraded":"2024-02-13T00:54:29.121975+0000","last_fullsized":"2024-02-13T00:54:29.121975+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:17.340797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:48:17.340797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"404'14","reported_seq":1025,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:30.276786+0000","last_change":"2024-02-13T00:53:50.046537+0000","last_active":"2024-02-13T00:54:30.276786+0000","last_peered":"2024-02-13T00:54:30.276786+0000","last_clean":"2024-02-13T00:54:30.276786+0000","last_became_active":"2024-02-13T00:46:01.455323+0000","last_became_peered":"2024-02-13T00:46:01.455323+0000","last_unstale":"2024-02-13T00:54:30.276786+0000","last_undegraded":"2024-02-13T00:54:30.276786+0000","last_fullsized":"2024-02-13T00:54:30.276786+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14","last_scrub_stamp":"2024-02-13T00:53:50.046455+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:53:50.046455+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":557,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059435+0000","last_change":"2024-02-13T00:53:47.479840+0000","last_active":"2024-02-13T00:54:29.059435+0000","last_peered":"2024-02-13T00:54:29.059435+0000","last_clean":"2024-02-13T00:54:29.059435+0000","last_became_active":"2024-02-13T00:52:30.605408+0000","last_became_peered":"2024-02-13T00:52:30.605408+0000","last_unstale":"2024-02-13T00:54:29.059435+0000","last_undegraded":"2024-02-13T00:54:29.059435+0000","last_fullsized":"2024-02-13T00:54:29.059435+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:53:47.479733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:53:47.479733+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":786,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059910+0000","last_change":"2024-02-13T00:51:41.570719+0000","last_active":"2024-02-13T00:54:29.059910+0000","last_peered":"2024-02-13T00:54:29.059910+0000","last_clean":"2024-02-13T00:54:29.059910+0000","last_became_active":"2024-02-13T00:51:41.567277+0000","last_became_peered":"2024-02-13T00:51:41.567277+0000","last_unstale":"2024-02-13T00:54:29.059910+0000","last_undegraded":"2024-02-13T00:54:29.059910+0000","last_fullsized":"2024-02-13T00:54:29.059910+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:33.040516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:49:33.040516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":798,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978115+0000","last_change":"2024-02-13T00:52:30.572639+0000","last_active":"2024-02-13T00:54:28.978115+0000","last_peered":"2024-02-13T00:54:28.978115+0000","last_clean":"2024-02-13T00:54:28.978115+0000","last_became_active":"2024-02-13T00:52:30.572144+0000","last_became_peered":"2024-02-13T00:52:30.572144+0000","last_unstale":"2024-02-13T00:54:28.978115+0000","last_undegraded":"2024-02-13T00:54:28.978115+0000","last_fullsized":"2024-02-13T00:54:28.978115+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:38.051468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:50:38.051468+0000","last_clean_scrub_stamp":"2024-02-13T00:50:38.051468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":735,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.058957+0000","last_change":"2024-02-13T00:51:41.589341+0000","last_active":"2024-02-13T00:54:29.058957+0000","last_peered":"2024-02-13T00:54:29.058957+0000","last_clean":"2024-02-13T00:54:29.058957+0000","last_became_active":"2024-02-13T00:51:41.588483+0000","last_became_peered":"2024-02-13T00:51:41.588483+0000","last_unstale":"2024-02-13T00:54:29.058957+0000","last_undegraded":"2024-02-13T00:54:29.058957+0000","last_fullsized":"2024-02-13T00:54:29.058957+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:08.606262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:51:08.606262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":516,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978337+0000","last_change":"2024-02-13T00:52:30.572465+0000","last_active":"2024-02-13T00:54:28.978337+0000","last_peered":"2024-02-13T00:54:28.978337+0000","last_clean":"2024-02-13T00:54:28.978337+0000","last_became_active":"2024-02-13T00:52:30.571718+0000","last_became_peered":"2024-02-13T00:52:30.571718+0000","last_unstale":"2024-02-13T00:54:28.978337+0000","last_undegraded":"2024-02-13T00:54:28.978337+0000","last_fullsized":"2024-02-13T00:54:28.978337+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:48.089417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:51:48.089417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":872,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.058961+0000","last_change":"2024-02-13T00:52:32.017951+0000","last_active":"2024-02-13T00:54:29.058961+0000","last_peered":"2024-02-13T00:54:29.058961+0000","last_clean":"2024-02-13T00:54:29.058961+0000","last_became_active":"2024-02-13T00:52:30.581394+0000","last_became_peered":"2024-02-13T00:52:30.581394+0000","last_unstale":"2024-02-13T00:54:29.058961+0000","last_undegraded":"2024-02-13T00:54:29.058961+0000","last_fullsized":"2024-02-13T00:54:29.058961+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'30","last_scrub_stamp":"2024-02-13T00:49:31.072862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:31.072862+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":813,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979772+0000","last_change":"2024-02-13T00:54:00.982156+0000","last_active":"2024-02-13T00:54:28.979772+0000","last_peered":"2024-02-13T00:54:28.979772+0000","last_clean":"2024-02-13T00:54:28.979772+0000","last_became_active":"2024-02-13T00:52:30.595453+0000","last_became_peered":"2024-02-13T00:52:30.595453+0000","last_unstale":"2024-02-13T00:54:28.979772+0000","last_undegraded":"2024-02-13T00:54:28.979772+0000","last_fullsized":"2024-02-13T00:54:28.979772+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:54:00.982074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:54:00.982074+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":588,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059739+0000","last_change":"2024-02-13T00:53:09.190093+0000","last_active":"2024-02-13T00:54:29.059739+0000","last_peered":"2024-02-13T00:54:29.059739+0000","last_clean":"2024-02-13T00:54:29.059739+0000","last_became_active":"2024-02-13T00:51:41.568308+0000","last_became_peered":"2024-02-13T00:51:41.568308+0000","last_unstale":"2024-02-13T00:54:29.059739+0000","last_undegraded":"2024-02-13T00:54:29.059739+0000","last_fullsized":"2024-02-13T00:54:29.059739+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:09.190024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:53:09.190024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":741,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.980018+0000","last_change":"2024-02-13T00:52:30.598253+0000","last_active":"2024-02-13T00:54:28.980018+0000","last_peered":"2024-02-13T00:54:28.980018+0000","last_clean":"2024-02-13T00:54:28.980018+0000","last_became_active":"2024-02-13T00:52:30.597320+0000","last_became_peered":"2024-02-13T00:52:30.597320+0000","last_unstale":"2024-02-13T00:54:28.980018+0000","last_undegraded":"2024-02-13T00:54:28.980018+0000","last_fullsized":"2024-02-13T00:54:28.980018+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:21.880001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:21.880001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"467'27","reported_seq":934,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.980043+0000","last_change":"2024-02-13T00:51:42.198644+0000","last_active":"2024-02-13T00:54:28.980043+0000","last_peered":"2024-02-13T00:54:28.980043+0000","last_clean":"2024-02-13T00:54:28.980043+0000","last_became_active":"2024-02-13T00:51:41.580479+0000","last_became_peered":"2024-02-13T00:51:41.580479+0000","last_unstale":"2024-02-13T00:54:28.980043+0000","last_undegraded":"2024-02-13T00:54:28.980043+0000","last_fullsized":"2024-02-13T00:54:28.980043+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:35.834669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.834669+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":493,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059030+0000","last_change":"2024-02-13T00:52:30.581921+0000","last_active":"2024-02-13T00:54:29.059030+0000","last_peered":"2024-02-13T00:54:29.059030+0000","last_clean":"2024-02-13T00:54:29.059030+0000","last_became_active":"2024-02-13T00:52:30.581023+0000","last_became_peered":"2024-02-13T00:52:30.581023+0000","last_unstale":"2024-02-13T00:54:29.059030+0000","last_undegraded":"2024-02-13T00:54:29.059030+0000","last_fullsized":"2024-02-13T00:54:29.059030+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:48.521656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:51:48.521656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":514,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979189+0000","last_change":"2024-02-13T00:52:30.595830+0000","last_active":"2024-02-13T00:54:28.979189+0000","last_peered":"2024-02-13T00:54:28.979189+0000","last_clean":"2024-02-13T00:54:28.979189+0000","last_became_active":"2024-02-13T00:52:30.594958+0000","last_became_peered":"2024-02-13T00:52:30.594958+0000","last_unstale":"2024-02-13T00:54:28.979189+0000","last_undegraded":"2024-02-13T00:54:28.979189+0000","last_fullsized":"2024-02-13T00:54:28.979189+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:36.152699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:36.152699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":501,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.123009+0000","last_change":"2024-02-13T00:52:30.601489+0000","last_active":"2024-02-13T00:54:29.123009+0000","last_peered":"2024-02-13T00:54:29.123009+0000","last_clean":"2024-02-13T00:54:29.123009+0000","last_became_active":"2024-02-13T00:52:30.600883+0000","last_became_peered":"2024-02-13T00:52:30.600883+0000","last_unstale":"2024-02-13T00:54:29.123009+0000","last_undegraded":"2024-02-13T00:54:29.123009+0000","last_fullsized":"2024-02-13T00:54:29.123009+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:07.304576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:07.304576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":672,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060904+0000","last_change":"2024-02-13T00:51:52.536631+0000","last_active":"2024-02-13T00:54:29.060904+0000","last_peered":"2024-02-13T00:54:29.060904+0000","last_clean":"2024-02-13T00:54:29.060904+0000","last_became_active":"2024-02-13T00:51:41.571280+0000","last_became_peered":"2024-02-13T00:51:41.571280+0000","last_unstale":"2024-02-13T00:54:29.060904+0000","last_undegraded":"2024-02-13T00:54:29.060904+0000","last_fullsized":"2024-02-13T00:54:29.060904+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'18","last_scrub_stamp":"2024-02-13T00:51:52.536524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:51:52.536524+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":499,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121768+0000","last_change":"2024-02-13T00:52:30.575760+0000","last_active":"2024-02-13T00:54:29.121768+0000","last_peered":"2024-02-13T00:54:29.121768+0000","last_clean":"2024-02-13T00:54:29.121768+0000","last_became_active":"2024-02-13T00:52:30.575381+0000","last_became_peered":"2024-02-13T00:52:30.575381+0000","last_unstale":"2024-02-13T00:54:29.121768+0000","last_undegraded":"2024-02-13T00:54:29.121768+0000","last_fullsized":"2024-02-13T00:54:29.121768+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:20.093279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:52:20.093279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":519,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122738+0000","last_change":"2024-02-13T00:52:30.597684+0000","last_active":"2024-02-13T00:54:29.122738+0000","last_peered":"2024-02-13T00:54:29.122738+0000","last_clean":"2024-02-13T00:54:29.122738+0000","last_became_active":"2024-02-13T00:52:30.596322+0000","last_became_peered":"2024-02-13T00:52:30.596322+0000","last_unstale":"2024-02-13T00:54:29.122738+0000","last_undegraded":"2024-02-13T00:54:29.122738+0000","last_fullsized":"2024-02-13T00:54:29.122738+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:40.384428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:50:40.384428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":591,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979488+0000","last_change":"2024-02-13T00:52:30.597677+0000","last_active":"2024-02-13T00:54:28.979488+0000","last_peered":"2024-02-13T00:54:28.979488+0000","last_clean":"2024-02-13T00:54:28.979488+0000","last_became_active":"2024-02-13T00:52:30.595811+0000","last_became_peered":"2024-02-13T00:52:30.595811+0000","last_unstale":"2024-02-13T00:54:28.979488+0000","last_undegraded":"2024-02-13T00:54:28.979488+0000","last_fullsized":"2024-02-13T00:54:28.979488+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:19.286611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:19.286611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":756,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060039+0000","last_change":"2024-02-13T00:51:43.571860+0000","last_active":"2024-02-13T00:54:29.060039+0000","last_peered":"2024-02-13T00:54:29.060039+0000","last_clean":"2024-02-13T00:54:29.060039+0000","last_became_active":"2024-02-13T00:51:41.569584+0000","last_became_peered":"2024-02-13T00:51:41.569584+0000","last_unstale":"2024-02-13T00:54:29.060039+0000","last_undegraded":"2024-02-13T00:54:29.060039+0000","last_fullsized":"2024-02-13T00:54:29.060039+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'39","last_scrub_stamp":"2024-02-13T00:49:06.154931+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:49:06.154931+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":761,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979426+0000","last_change":"2024-02-13T00:52:30.595096+0000","last_active":"2024-02-13T00:54:28.979426+0000","last_peered":"2024-02-13T00:54:28.979426+0000","last_clean":"2024-02-13T00:54:28.979426+0000","last_became_active":"2024-02-13T00:52:30.594229+0000","last_became_peered":"2024-02-13T00:52:30.594229+0000","last_unstale":"2024-02-13T00:54:28.979426+0000","last_undegraded":"2024-02-13T00:54:28.979426+0000","last_fullsized":"2024-02-13T00:54:28.979426+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.916152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.916152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"726'10","reported_seq":556,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.985926+0000","last_change":"2024-02-13T00:53:19.374855+0000","last_active":"2024-02-13T00:54:29.985926+0000","last_peered":"2024-02-13T00:54:29.985926+0000","last_clean":"2024-02-13T00:54:29.985926+0000","last_became_active":"2024-02-13T00:52:30.603380+0000","last_became_peered":"2024-02-13T00:52:30.603380+0000","last_unstale":"2024-02-13T00:54:29.985926+0000","last_undegraded":"2024-02-13T00:54:29.985926+0000","last_fullsized":"2024-02-13T00:54:29.985926+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'10","last_scrub_stamp":"2024-02-13T00:53:19.374786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:53:19.374786+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":803,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979950+0000","last_change":"2024-02-13T00:53:45.110649+0000","last_active":"2024-02-13T00:54:28.979950+0000","last_peered":"2024-02-13T00:54:28.979950+0000","last_clean":"2024-02-13T00:54:28.979950+0000","last_became_active":"2024-02-13T00:52:30.596877+0000","last_became_peered":"2024-02-13T00:52:30.596877+0000","last_unstale":"2024-02-13T00:54:28.979950+0000","last_undegraded":"2024-02-13T00:54:28.979950+0000","last_fullsized":"2024-02-13T00:54:28.979950+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:45.110582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:53:45.110582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":460,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979114+0000","last_change":"2024-02-13T00:51:41.587438+0000","last_active":"2024-02-13T00:54:28.979114+0000","last_peered":"2024-02-13T00:54:28.979114+0000","last_clean":"2024-02-13T00:54:28.979114+0000","last_became_active":"2024-02-13T00:51:41.583454+0000","last_became_peered":"2024-02-13T00:51:41.583454+0000","last_unstale":"2024-02-13T00:54:28.979114+0000","last_undegraded":"2024-02-13T00:54:28.979114+0000","last_fullsized":"2024-02-13T00:54:28.979114+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:14.849687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:14.849687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"467'22","reported_seq":791,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122094+0000","last_change":"2024-02-13T00:54:23.587189+0000","last_active":"2024-02-13T00:54:29.122094+0000","last_peered":"2024-02-13T00:54:29.122094+0000","last_clean":"2024-02-13T00:54:29.122094+0000","last_became_active":"2024-02-13T00:51:41.584471+0000","last_became_peered":"2024-02-13T00:51:41.584471+0000","last_unstale":"2024-02-13T00:54:29.122094+0000","last_undegraded":"2024-02-13T00:54:29.122094+0000","last_fullsized":"2024-02-13T00:54:29.122094+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'22","last_scrub_stamp":"2024-02-13T00:54:23.587102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:54:23.587102+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":818,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:32.210195+0000","last_change":"2024-02-13T00:54:32.210195+0000","last_active":"2024-02-13T00:54:32.210195+0000","last_peered":"2024-02-13T00:54:32.210195+0000","last_clean":"2024-02-13T00:54:32.210195+0000","last_became_active":"2024-02-13T00:52:30.593985+0000","last_became_peered":"2024-02-13T00:52:30.593985+0000","last_unstale":"2024-02-13T00:54:32.210195+0000","last_undegraded":"2024-02-13T00:54:32.210195+0000","last_fullsized":"2024-02-13T00:54:32.210195+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:54:32.210121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:54:32.210121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"726'16","reported_seq":712,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.985387+0000","last_change":"2024-02-13T00:53:10.218214+0000","last_active":"2024-02-13T00:54:29.985387+0000","last_peered":"2024-02-13T00:54:29.985387+0000","last_clean":"2024-02-13T00:54:29.985387+0000","last_became_active":"2024-02-13T00:52:30.603133+0000","last_became_peered":"2024-02-13T00:52:30.603133+0000","last_unstale":"2024-02-13T00:54:29.985387+0000","last_undegraded":"2024-02-13T00:54:29.985387+0000","last_fullsized":"2024-02-13T00:54:29.985387+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'16","last_scrub_stamp":"2024-02-13T00:53:10.218143+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:53:10.218143+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":437,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979718+0000","last_change":"2024-02-13T00:52:30.596098+0000","last_active":"2024-02-13T00:54:28.979718+0000","last_peered":"2024-02-13T00:54:28.979718+0000","last_clean":"2024-02-13T00:54:28.979718+0000","last_became_active":"2024-02-13T00:52:30.594426+0000","last_became_peered":"2024-02-13T00:52:30.594426+0000","last_unstale":"2024-02-13T00:54:28.979718+0000","last_undegraded":"2024-02-13T00:54:28.979718+0000","last_fullsized":"2024-02-13T00:54:28.979718+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:27.933639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:27.933639+0000","last_clean_scrub_stamp":"2024-02-13T00:52:27.933639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":316,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059331+0000","last_change":"2024-02-13T00:53:07.596914+0000","last_active":"2024-02-13T00:54:29.059331+0000","last_peered":"2024-02-13T00:54:29.059331+0000","last_clean":"2024-02-13T00:54:29.059331+0000","last_became_active":"2024-02-13T00:53:07.596544+0000","last_became_peered":"2024-02-13T00:53:07.596544+0000","last_unstale":"2024-02-13T00:54:29.059331+0000","last_undegraded":"2024-02-13T00:54:29.059331+0000","last_fullsized":"2024-02-13T00:54:29.059331+0000","mapping_epoch":748,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":749,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:34.127252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:34.127252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":852,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059340+0000","last_change":"2024-02-13T00:49:38.007273+0000","last_active":"2024-02-13T00:54:29.059340+0000","last_peered":"2024-02-13T00:54:29.059340+0000","last_clean":"2024-02-13T00:54:29.059340+0000","last_became_active":"2024-02-13T00:44:50.441885+0000","last_became_peered":"2024-02-13T00:44:50.441885+0000","last_unstale":"2024-02-13T00:54:29.059340+0000","last_undegraded":"2024-02-13T00:54:29.059340+0000","last_fullsized":"2024-02-13T00:54:29.059340+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'19","last_scrub_stamp":"2024-02-13T00:49:38.007204+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:49:38.007204+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":510,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.123043+0000","last_change":"2024-02-13T00:51:41.571972+0000","last_active":"2024-02-13T00:54:29.123043+0000","last_peered":"2024-02-13T00:54:29.123043+0000","last_clean":"2024-02-13T00:54:29.123043+0000","last_became_active":"2024-02-13T00:51:41.570640+0000","last_became_peered":"2024-02-13T00:51:41.570640+0000","last_unstale":"2024-02-13T00:54:29.123043+0000","last_undegraded":"2024-02-13T00:54:29.123043+0000","last_fullsized":"2024-02-13T00:54:29.123043+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:13.316727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:51:13.316727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":573,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978722+0000","last_change":"2024-02-13T00:52:30.594111+0000","last_active":"2024-02-13T00:54:28.978722+0000","last_peered":"2024-02-13T00:54:28.978722+0000","last_clean":"2024-02-13T00:54:28.978722+0000","last_became_active":"2024-02-13T00:52:30.593800+0000","last_became_peered":"2024-02-13T00:52:30.593800+0000","last_unstale":"2024-02-13T00:54:28.978722+0000","last_undegraded":"2024-02-13T00:54:28.978722+0000","last_fullsized":"2024-02-13T00:54:28.978722+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:24.916456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:52:24.916456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":767,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978120+0000","last_change":"2024-02-13T00:53:49.062752+0000","last_active":"2024-02-13T00:54:28.978120+0000","last_peered":"2024-02-13T00:54:28.978120+0000","last_clean":"2024-02-13T00:54:28.978120+0000","last_became_active":"2024-02-13T00:46:01.445500+0000","last_became_peered":"2024-02-13T00:46:01.445500+0000","last_unstale":"2024-02-13T00:54:28.978120+0000","last_undegraded":"2024-02-13T00:54:28.978120+0000","last_fullsized":"2024-02-13T00:54:28.978120+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:49.062647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.413326+0000","last_clean_scrub_stamp":"2024-02-13T00:53:49.062647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":509,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122343+0000","last_change":"2024-02-13T00:53:08.375070+0000","last_active":"2024-02-13T00:54:29.122343+0000","last_peered":"2024-02-13T00:54:29.122343+0000","last_clean":"2024-02-13T00:54:29.122343+0000","last_became_active":"2024-02-13T00:51:41.570732+0000","last_became_peered":"2024-02-13T00:51:41.570732+0000","last_unstale":"2024-02-13T00:54:29.122343+0000","last_undegraded":"2024-02-13T00:54:29.122343+0000","last_fullsized":"2024-02-13T00:54:29.122343+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:08.374988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:53:08.374988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":549,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122323+0000","last_change":"2024-02-13T00:52:33.431600+0000","last_active":"2024-02-13T00:54:29.122323+0000","last_peered":"2024-02-13T00:54:29.122323+0000","last_clean":"2024-02-13T00:54:29.122323+0000","last_became_active":"2024-02-13T00:52:30.594405+0000","last_became_peered":"2024-02-13T00:52:30.594405+0000","last_unstale":"2024-02-13T00:54:29.122323+0000","last_undegraded":"2024-02-13T00:54:29.122323+0000","last_fullsized":"2024-02-13T00:54:29.122323+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:48:34.458264+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:48:34.458264+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":777,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059953+0000","last_change":"2024-02-13T00:51:47.538426+0000","last_active":"2024-02-13T00:54:29.059953+0000","last_peered":"2024-02-13T00:54:29.059953+0000","last_clean":"2024-02-13T00:54:29.059953+0000","last_became_active":"2024-02-13T00:46:01.456564+0000","last_became_peered":"2024-02-13T00:46:01.456564+0000","last_unstale":"2024-02-13T00:54:29.059953+0000","last_undegraded":"2024-02-13T00:54:29.059953+0000","last_fullsized":"2024-02-13T00:54:29.059953+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:47.538354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:51:47.538354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":800,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059392+0000","last_change":"2024-02-13T00:51:41.569851+0000","last_active":"2024-02-13T00:54:29.059392+0000","last_peered":"2024-02-13T00:54:29.059392+0000","last_clean":"2024-02-13T00:54:29.059392+0000","last_became_active":"2024-02-13T00:51:41.568200+0000","last_became_peered":"2024-02-13T00:51:41.568200+0000","last_unstale":"2024-02-13T00:54:29.059392+0000","last_undegraded":"2024-02-13T00:54:29.059392+0000","last_fullsized":"2024-02-13T00:54:29.059392+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:36.787604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:58.512053+0000","last_clean_scrub_stamp":"2024-02-13T00:50:36.787604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":582,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978535+0000","last_change":"2024-02-13T00:53:17.006874+0000","last_active":"2024-02-13T00:54:28.978535+0000","last_peered":"2024-02-13T00:54:28.978535+0000","last_clean":"2024-02-13T00:54:28.978535+0000","last_became_active":"2024-02-13T00:52:30.588561+0000","last_became_peered":"2024-02-13T00:52:30.588561+0000","last_unstale":"2024-02-13T00:54:28.978535+0000","last_undegraded":"2024-02-13T00:54:28.978535+0000","last_fullsized":"2024-02-13T00:54:28.978535+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:17.006804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:17.006804+0000","last_clean_scrub_stamp":"2024-02-13T00:53:17.006804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":510,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121336+0000","last_change":"2024-02-13T00:51:41.586630+0000","last_active":"2024-02-13T00:54:29.121336+0000","last_peered":"2024-02-13T00:54:29.121336+0000","last_clean":"2024-02-13T00:54:29.121336+0000","last_became_active":"2024-02-13T00:51:41.584153+0000","last_became_peered":"2024-02-13T00:51:41.584153+0000","last_unstale":"2024-02-13T00:54:29.121336+0000","last_undegraded":"2024-02-13T00:54:29.121336+0000","last_fullsized":"2024-02-13T00:54:29.121336+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:02.260045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:02.260045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":619,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059222+0000","last_change":"2024-02-13T00:52:33.431814+0000","last_active":"2024-02-13T00:54:29.059222+0000","last_peered":"2024-02-13T00:54:29.059222+0000","last_clean":"2024-02-13T00:54:29.059222+0000","last_became_active":"2024-02-13T00:52:30.606082+0000","last_became_peered":"2024-02-13T00:52:30.606082+0000","last_unstale":"2024-02-13T00:54:29.059222+0000","last_undegraded":"2024-02-13T00:54:29.059222+0000","last_fullsized":"2024-02-13T00:54:29.059222+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'8","last_scrub_stamp":"2024-02-13T00:50:32.774820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:50:32.774820+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":727,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122449+0000","last_change":"2024-02-13T00:48:32.441279+0000","last_active":"2024-02-13T00:54:29.122449+0000","last_peered":"2024-02-13T00:54:29.122449+0000","last_clean":"2024-02-13T00:54:29.122449+0000","last_became_active":"2024-02-13T00:46:01.461298+0000","last_became_peered":"2024-02-13T00:46:01.461298+0000","last_unstale":"2024-02-13T00:54:29.122449+0000","last_undegraded":"2024-02-13T00:54:29.122449+0000","last_fullsized":"2024-02-13T00:54:29.122449+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:32.441170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:48:32.441170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":510,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121475+0000","last_change":"2024-02-13T00:52:30.579288+0000","last_active":"2024-02-13T00:54:29.121475+0000","last_peered":"2024-02-13T00:54:29.121475+0000","last_clean":"2024-02-13T00:54:29.121475+0000","last_became_active":"2024-02-13T00:52:30.578921+0000","last_became_peered":"2024-02-13T00:52:30.578921+0000","last_unstale":"2024-02-13T00:54:29.121475+0000","last_undegraded":"2024-02-13T00:54:29.121475+0000","last_fullsized":"2024-02-13T00:54:29.121475+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:48:28.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:48:28.353347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":504,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059063+0000","last_change":"2024-02-13T00:52:30.603553+0000","last_active":"2024-02-13T00:54:29.059063+0000","last_peered":"2024-02-13T00:54:29.059063+0000","last_clean":"2024-02-13T00:54:29.059063+0000","last_became_active":"2024-02-13T00:52:30.602714+0000","last_became_peered":"2024-02-13T00:52:30.602714+0000","last_unstale":"2024-02-13T00:54:29.059063+0000","last_undegraded":"2024-02-13T00:54:29.059063+0000","last_fullsized":"2024-02-13T00:54:29.059063+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:29.052640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:49:29.052640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":657,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121587+0000","last_change":"2024-02-13T00:52:30.593895+0000","last_active":"2024-02-13T00:54:29.121587+0000","last_peered":"2024-02-13T00:54:29.121587+0000","last_clean":"2024-02-13T00:54:29.121587+0000","last_became_active":"2024-02-13T00:52:30.593558+0000","last_became_peered":"2024-02-13T00:52:30.593558+0000","last_unstale":"2024-02-13T00:54:29.121587+0000","last_undegraded":"2024-02-13T00:54:29.121587+0000","last_fullsized":"2024-02-13T00:54:29.121587+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:54.114116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:54.114116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"467'25","reported_seq":583,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:32.695307+0000","last_change":"2024-02-13T00:54:32.695307+0000","last_active":"2024-02-13T00:54:32.695307+0000","last_peered":"2024-02-13T00:54:32.695307+0000","last_clean":"2024-02-13T00:54:32.695307+0000","last_became_active":"2024-02-13T00:51:41.585940+0000","last_became_peered":"2024-02-13T00:51:41.585940+0000","last_unstale":"2024-02-13T00:54:32.695307+0000","last_undegraded":"2024-02-13T00:54:32.695307+0000","last_fullsized":"2024-02-13T00:54:32.695307+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:54:32.695181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:54:32.695181+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":506,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122902+0000","last_change":"2024-02-13T00:53:42.494098+0000","last_active":"2024-02-13T00:54:29.122902+0000","last_peered":"2024-02-13T00:54:29.122902+0000","last_clean":"2024-02-13T00:54:29.122902+0000","last_became_active":"2024-02-13T00:46:01.458710+0000","last_became_peered":"2024-02-13T00:46:01.458710+0000","last_unstale":"2024-02-13T00:54:29.122902+0000","last_undegraded":"2024-02-13T00:54:29.122902+0000","last_fullsized":"2024-02-13T00:54:29.122902+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:42.494009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:53:42.494009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":510,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121123+0000","last_change":"2024-02-13T00:52:30.578502+0000","last_active":"2024-02-13T00:54:29.121123+0000","last_peered":"2024-02-13T00:54:29.121123+0000","last_clean":"2024-02-13T00:54:29.121123+0000","last_became_active":"2024-02-13T00:52:30.577297+0000","last_became_peered":"2024-02-13T00:52:30.577297+0000","last_unstale":"2024-02-13T00:54:29.121123+0000","last_undegraded":"2024-02-13T00:54:29.121123+0000","last_fullsized":"2024-02-13T00:54:29.121123+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:08.345652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:51:08.345652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":709,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978665+0000","last_change":"2024-02-13T00:52:30.572207+0000","last_active":"2024-02-13T00:54:28.978665+0000","last_peered":"2024-02-13T00:54:28.978665+0000","last_clean":"2024-02-13T00:54:28.978665+0000","last_became_active":"2024-02-13T00:52:30.569895+0000","last_became_peered":"2024-02-13T00:52:30.569895+0000","last_unstale":"2024-02-13T00:54:28.978665+0000","last_undegraded":"2024-02-13T00:54:28.978665+0000","last_fullsized":"2024-02-13T00:54:28.978665+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:06.264572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:51:06.264572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":624,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122384+0000","last_change":"2024-02-13T00:51:41.571094+0000","last_active":"2024-02-13T00:54:29.122384+0000","last_peered":"2024-02-13T00:54:29.122384+0000","last_clean":"2024-02-13T00:54:29.122384+0000","last_became_active":"2024-02-13T00:51:41.569716+0000","last_became_peered":"2024-02-13T00:51:41.569716+0000","last_unstale":"2024-02-13T00:54:29.122384+0000","last_undegraded":"2024-02-13T00:54:29.122384+0000","last_fullsized":"2024-02-13T00:54:29.122384+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:35.101298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:49:35.101298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":912,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.168056+0000","last_change":"2024-02-13T00:54:29.168034+0000","last_active":"2024-02-13T00:54:29.168056+0000","last_peered":"2024-02-13T00:54:29.168056+0000","last_clean":"2024-02-13T00:54:29.168056+0000","last_became_active":"2024-02-13T00:51:41.589397+0000","last_became_peered":"2024-02-13T00:51:41.589397+0000","last_unstale":"2024-02-13T00:54:29.168056+0000","last_undegraded":"2024-02-13T00:54:29.168056+0000","last_fullsized":"2024-02-13T00:54:29.168056+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'32","last_scrub_stamp":"2024-02-13T00:54:29.167966+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:54:29.167966+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":752,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.120827+0000","last_change":"2024-02-13T00:53:20.407812+0000","last_active":"2024-02-13T00:54:29.120827+0000","last_peered":"2024-02-13T00:54:29.120827+0000","last_clean":"2024-02-13T00:54:29.120827+0000","last_became_active":"2024-02-13T00:52:30.578094+0000","last_became_peered":"2024-02-13T00:52:30.578094+0000","last_unstale":"2024-02-13T00:54:29.120827+0000","last_undegraded":"2024-02-13T00:54:29.120827+0000","last_fullsized":"2024-02-13T00:54:29.120827+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:20.407731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:53:20.407731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":619,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122158+0000","last_change":"2024-02-13T00:52:32.509795+0000","last_active":"2024-02-13T00:54:29.122158+0000","last_peered":"2024-02-13T00:54:29.122158+0000","last_clean":"2024-02-13T00:54:29.122158+0000","last_became_active":"2024-02-13T00:52:30.585316+0000","last_became_peered":"2024-02-13T00:52:30.585316+0000","last_unstale":"2024-02-13T00:54:29.122158+0000","last_undegraded":"2024-02-13T00:54:29.122158+0000","last_fullsized":"2024-02-13T00:54:29.122158+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:48:33.460247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:48:33.460247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":744,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060818+0000","last_change":"2024-02-13T00:51:41.571960+0000","last_active":"2024-02-13T00:54:29.060818+0000","last_peered":"2024-02-13T00:54:29.060818+0000","last_clean":"2024-02-13T00:54:29.060818+0000","last_became_active":"2024-02-13T00:51:41.569703+0000","last_became_peered":"2024-02-13T00:51:41.569703+0000","last_unstale":"2024-02-13T00:54:29.060818+0000","last_undegraded":"2024-02-13T00:54:29.060818+0000","last_fullsized":"2024-02-13T00:54:29.060818+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:39.570602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:39.570602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":532,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.123118+0000","last_change":"2024-02-13T00:48:55.424021+0000","last_active":"2024-02-13T00:54:29.123118+0000","last_peered":"2024-02-13T00:54:29.123118+0000","last_clean":"2024-02-13T00:54:29.123118+0000","last_became_active":"2024-02-13T00:46:01.459590+0000","last_became_peered":"2024-02-13T00:46:01.459590+0000","last_unstale":"2024-02-13T00:54:29.123118+0000","last_undegraded":"2024-02-13T00:54:29.123118+0000","last_fullsized":"2024-02-13T00:54:29.123118+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:48:55.423952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:48:55.423952+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":515,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121705+0000","last_change":"2024-02-13T00:53:47.402044+0000","last_active":"2024-02-13T00:54:29.121705+0000","last_peered":"2024-02-13T00:54:29.121705+0000","last_clean":"2024-02-13T00:54:29.121705+0000","last_became_active":"2024-02-13T00:51:41.584919+0000","last_became_peered":"2024-02-13T00:51:41.584919+0000","last_unstale":"2024-02-13T00:54:29.121705+0000","last_undegraded":"2024-02-13T00:54:29.121705+0000","last_fullsized":"2024-02-13T00:54:29.121705+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:47.401921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:53:47.401921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":780,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059877+0000","last_change":"2024-02-13T00:52:30.603945+0000","last_active":"2024-02-13T00:54:29.059877+0000","last_peered":"2024-02-13T00:54:29.059877+0000","last_clean":"2024-02-13T00:54:29.059877+0000","last_became_active":"2024-02-13T00:52:30.602705+0000","last_became_peered":"2024-02-13T00:52:30.602705+0000","last_unstale":"2024-02-13T00:54:29.059877+0000","last_undegraded":"2024-02-13T00:54:29.059877+0000","last_fullsized":"2024-02-13T00:54:29.059877+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:46.928651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:49:46.928651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":497,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.123129+0000","last_change":"2024-02-13T00:52:24.173739+0000","last_active":"2024-02-13T00:54:29.123129+0000","last_peered":"2024-02-13T00:54:29.123129+0000","last_clean":"2024-02-13T00:54:29.123129+0000","last_became_active":"2024-02-13T00:44:50.435886+0000","last_became_peered":"2024-02-13T00:44:50.435886+0000","last_unstale":"2024-02-13T00:54:29.123129+0000","last_undegraded":"2024-02-13T00:54:29.123129+0000","last_fullsized":"2024-02-13T00:54:29.123129+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:24.173651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:24.173651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":575,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060009+0000","last_change":"2024-02-13T00:52:32.307501+0000","last_active":"2024-02-13T00:54:29.060009+0000","last_peered":"2024-02-13T00:54:29.060009+0000","last_clean":"2024-02-13T00:54:29.060009+0000","last_became_active":"2024-02-13T00:52:30.603658+0000","last_became_peered":"2024-02-13T00:52:30.603658+0000","last_unstale":"2024-02-13T00:54:29.060009+0000","last_undegraded":"2024-02-13T00:54:29.060009+0000","last_fullsized":"2024-02-13T00:54:29.060009+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:49:30.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:49:30.086856+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":89,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121295+0000","last_change":"2024-02-13T00:52:30.578696+0000","last_active":"2024-02-13T00:54:29.121295+0000","last_peered":"2024-02-13T00:54:29.121295+0000","last_clean":"2024-02-13T00:54:29.121295+0000","last_became_active":"2024-02-13T00:52:30.578352+0000","last_became_peered":"2024-02-13T00:52:30.578352+0000","last_unstale":"2024-02-13T00:54:29.121295+0000","last_undegraded":"2024-02-13T00:54:29.121295+0000","last_fullsized":"2024-02-13T00:54:29.121295+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:04.282529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:04.282529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":506,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122875+0000","last_change":"2024-02-13T00:54:27.596986+0000","last_active":"2024-02-13T00:54:29.122875+0000","last_peered":"2024-02-13T00:54:29.122875+0000","last_clean":"2024-02-13T00:54:29.122875+0000","last_became_active":"2024-02-13T00:52:30.594309+0000","last_became_peered":"2024-02-13T00:52:30.594309+0000","last_unstale":"2024-02-13T00:54:29.122875+0000","last_undegraded":"2024-02-13T00:54:29.122875+0000","last_fullsized":"2024-02-13T00:54:29.122875+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:54:27.596895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:54:27.596895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":424,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122193+0000","last_change":"2024-02-13T00:52:30.597508+0000","last_active":"2024-02-13T00:54:29.122193+0000","last_peered":"2024-02-13T00:54:29.122193+0000","last_clean":"2024-02-13T00:54:29.122193+0000","last_became_active":"2024-02-13T00:52:30.595882+0000","last_became_peered":"2024-02-13T00:52:30.595882+0000","last_unstale":"2024-02-13T00:54:29.122193+0000","last_undegraded":"2024-02-13T00:54:29.122193+0000","last_fullsized":"2024-02-13T00:54:29.122193+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:23.430396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:23.430396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":679,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.060178+0000","last_change":"2024-02-13T00:52:32.857449+0000","last_active":"2024-02-13T00:54:29.060178+0000","last_peered":"2024-02-13T00:54:29.060178+0000","last_clean":"2024-02-13T00:54:29.060178+0000","last_became_active":"2024-02-13T00:52:30.604045+0000","last_became_peered":"2024-02-13T00:52:30.604045+0000","last_unstale":"2024-02-13T00:54:29.060178+0000","last_undegraded":"2024-02-13T00:54:29.060178+0000","last_fullsized":"2024-02-13T00:54:29.060178+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:52:21.312574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:21.312574+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":519,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122443+0000","last_change":"2024-02-13T00:52:30.596393+0000","last_active":"2024-02-13T00:54:29.122443+0000","last_peered":"2024-02-13T00:54:29.122443+0000","last_clean":"2024-02-13T00:54:29.122443+0000","last_became_active":"2024-02-13T00:52:30.596074+0000","last_became_peered":"2024-02-13T00:52:30.596074+0000","last_unstale":"2024-02-13T00:54:29.122443+0000","last_undegraded":"2024-02-13T00:54:29.122443+0000","last_fullsized":"2024-02-13T00:54:29.122443+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:18.264738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:18.264738+0000","last_clean_scrub_stamp":"2024-02-13T00:51:18.264738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":497,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122818+0000","last_change":"2024-02-13T00:53:17.397082+0000","last_active":"2024-02-13T00:54:29.122818+0000","last_peered":"2024-02-13T00:54:29.122818+0000","last_clean":"2024-02-13T00:54:29.122818+0000","last_became_active":"2024-02-13T00:46:01.459963+0000","last_became_peered":"2024-02-13T00:46:01.459963+0000","last_unstale":"2024-02-13T00:54:29.122818+0000","last_undegraded":"2024-02-13T00:54:29.122818+0000","last_fullsized":"2024-02-13T00:54:29.122818+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:17.397016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:53:17.397016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":741,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979312+0000","last_change":"2024-02-13T00:51:35.190514+0000","last_active":"2024-02-13T00:54:28.979312+0000","last_peered":"2024-02-13T00:54:28.979312+0000","last_clean":"2024-02-13T00:54:28.979312+0000","last_became_active":"2024-02-13T00:46:01.443423+0000","last_became_peered":"2024-02-13T00:46:01.443423+0000","last_unstale":"2024-02-13T00:54:28.979312+0000","last_undegraded":"2024-02-13T00:54:28.979312+0000","last_fullsized":"2024-02-13T00:54:28.979312+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:35.190419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:35.190419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":669,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979215+0000","last_change":"2024-02-13T00:52:31.275396+0000","last_active":"2024-02-13T00:54:28.979215+0000","last_peered":"2024-02-13T00:54:28.979215+0000","last_clean":"2024-02-13T00:54:28.979215+0000","last_became_active":"2024-02-13T00:52:30.592405+0000","last_became_peered":"2024-02-13T00:52:30.592405+0000","last_unstale":"2024-02-13T00:54:28.979215+0000","last_undegraded":"2024-02-13T00:54:28.979215+0000","last_fullsized":"2024-02-13T00:54:28.979215+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:52:16.976052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:52:16.976052+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":585,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122352+0000","last_change":"2024-02-13T00:51:41.571773+0000","last_active":"2024-02-13T00:54:29.122352+0000","last_peered":"2024-02-13T00:54:29.122352+0000","last_clean":"2024-02-13T00:54:29.122352+0000","last_became_active":"2024-02-13T00:51:41.569507+0000","last_became_peered":"2024-02-13T00:51:41.569507+0000","last_unstale":"2024-02-13T00:54:29.122352+0000","last_undegraded":"2024-02-13T00:54:29.122352+0000","last_fullsized":"2024-02-13T00:54:29.122352+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:49:43.067615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:49:43.067615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":778,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978343+0000","last_change":"2024-02-13T00:52:35.023058+0000","last_active":"2024-02-13T00:54:28.978343+0000","last_peered":"2024-02-13T00:54:28.978343+0000","last_clean":"2024-02-13T00:54:28.978343+0000","last_became_active":"2024-02-13T00:51:41.582536+0000","last_became_peered":"2024-02-13T00:51:41.582536+0000","last_unstale":"2024-02-13T00:54:28.978343+0000","last_undegraded":"2024-02-13T00:54:28.978343+0000","last_fullsized":"2024-02-13T00:54:28.978343+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:35.022992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:52:35.022992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":684,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979650+0000","last_change":"2024-02-13T00:51:41.587948+0000","last_active":"2024-02-13T00:54:28.979650+0000","last_peered":"2024-02-13T00:54:28.979650+0000","last_clean":"2024-02-13T00:54:28.979650+0000","last_became_active":"2024-02-13T00:51:41.583578+0000","last_became_peered":"2024-02-13T00:51:41.583578+0000","last_unstale":"2024-02-13T00:54:28.979650+0000","last_undegraded":"2024-02-13T00:54:28.979650+0000","last_fullsized":"2024-02-13T00:54:28.979650+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:50:15.846461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:50:15.846461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":698,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059609+0000","last_change":"2024-02-13T00:52:32.781431+0000","last_active":"2024-02-13T00:54:29.059609+0000","last_peered":"2024-02-13T00:54:29.059609+0000","last_clean":"2024-02-13T00:54:29.059609+0000","last_became_active":"2024-02-13T00:52:30.604370+0000","last_became_peered":"2024-02-13T00:52:30.604370+0000","last_unstale":"2024-02-13T00:54:29.059609+0000","last_undegraded":"2024-02-13T00:54:29.059609+0000","last_fullsized":"2024-02-13T00:54:29.059609+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'16","last_scrub_stamp":"2024-02-13T00:51:49.491168+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:51:49.491168+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":72,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059533+0000","last_change":"2024-02-13T00:53:42.501211+0000","last_active":"2024-02-13T00:54:29.059533+0000","last_peered":"2024-02-13T00:54:29.059533+0000","last_clean":"2024-02-13T00:54:29.059533+0000","last_became_active":"2024-02-13T00:53:01.780063+0000","last_became_peered":"2024-02-13T00:53:01.780063+0000","last_unstale":"2024-02-13T00:54:29.059533+0000","last_undegraded":"2024-02-13T00:54:29.059533+0000","last_fullsized":"2024-02-13T00:54:29.059533+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:42.501149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:53:42.501149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":730,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-13T00:54:34.236217+0000","last_change":"2024-02-13T00:54:34.236217+0000","last_active":"2024-02-13T00:54:34.236217+0000","last_peered":"2024-02-13T00:54:34.236217+0000","last_clean":"2024-02-13T00:54:34.236217+0000","last_became_active":"2024-02-13T00:51:41.589029+0000","last_became_peered":"2024-02-13T00:51:41.589029+0000","last_unstale":"2024-02-13T00:54:34.236217+0000","last_undegraded":"2024-02-13T00:54:34.236217+0000","last_fullsized":"2024-02-13T00:54:34.236217+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:54:34.236151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:54:34.236151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":488,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121392+0000","last_change":"2024-02-13T00:52:38.236729+0000","last_active":"2024-02-13T00:54:29.121392+0000","last_peered":"2024-02-13T00:54:29.121392+0000","last_clean":"2024-02-13T00:54:29.121392+0000","last_became_active":"2024-02-13T00:44:39.506484+0000","last_became_peered":"2024-02-13T00:44:39.506484+0000","last_unstale":"2024-02-13T00:54:29.121392+0000","last_undegraded":"2024-02-13T00:54:29.121392+0000","last_fullsized":"2024-02-13T00:54:29.121392+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:52:38.236655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:52:38.236655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":567,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.121463+0000","last_change":"2024-02-13T00:52:32.083766+0000","last_active":"2024-02-13T00:54:29.121463+0000","last_peered":"2024-02-13T00:54:29.121463+0000","last_clean":"2024-02-13T00:54:29.121463+0000","last_became_active":"2024-02-13T00:52:30.576217+0000","last_became_peered":"2024-02-13T00:52:30.576217+0000","last_unstale":"2024-02-13T00:54:29.121463+0000","last_undegraded":"2024-02-13T00:54:29.121463+0000","last_fullsized":"2024-02-13T00:54:29.121463+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:51:10.298233+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:51:10.298233+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":600,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.980072+0000","last_change":"2024-02-13T00:53:09.074634+0000","last_active":"2024-02-13T00:54:28.980072+0000","last_peered":"2024-02-13T00:54:28.980072+0000","last_clean":"2024-02-13T00:54:28.980072+0000","last_became_active":"2024-02-13T00:52:30.597277+0000","last_became_peered":"2024-02-13T00:52:30.597277+0000","last_unstale":"2024-02-13T00:54:28.980072+0000","last_undegraded":"2024-02-13T00:54:28.980072+0000","last_fullsized":"2024-02-13T00:54:28.980072+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:09.074568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:53:09.074568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":438,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059162+0000","last_change":"2024-02-13T00:53:43.505855+0000","last_active":"2024-02-13T00:54:29.059162+0000","last_peered":"2024-02-13T00:54:29.059162+0000","last_clean":"2024-02-13T00:54:29.059162+0000","last_became_active":"2024-02-13T00:52:30.602788+0000","last_became_peered":"2024-02-13T00:52:30.602788+0000","last_unstale":"2024-02-13T00:54:29.059162+0000","last_undegraded":"2024-02-13T00:54:29.059162+0000","last_fullsized":"2024-02-13T00:54:29.059162+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:43.505792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:53:43.505792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":747,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.059241+0000","last_change":"2024-02-13T00:51:36.522278+0000","last_active":"2024-02-13T00:54:29.059241+0000","last_peered":"2024-02-13T00:54:29.059241+0000","last_clean":"2024-02-13T00:54:29.059241+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:54:29.059241+0000","last_undegraded":"2024-02-13T00:54:29.059241+0000","last_fullsized":"2024-02-13T00:54:29.059241+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:51:36.522208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:51:36.522208+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":681,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.978615+0000","last_change":"2024-02-13T00:53:06.023522+0000","last_active":"2024-02-13T00:54:28.978615+0000","last_peered":"2024-02-13T00:54:28.978615+0000","last_clean":"2024-02-13T00:54:28.978615+0000","last_became_active":"2024-02-13T00:46:01.445791+0000","last_became_peered":"2024-02-13T00:46:01.445791+0000","last_unstale":"2024-02-13T00:54:28.978615+0000","last_undegraded":"2024-02-13T00:54:28.978615+0000","last_fullsized":"2024-02-13T00:54:28.978615+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'62","last_scrub_stamp":"2024-02-13T00:53:06.023450+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:53:06.023450+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":791,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:28.979780+0000","last_change":"2024-02-13T00:53:53.023337+0000","last_active":"2024-02-13T00:54:28.979780+0000","last_peered":"2024-02-13T00:54:28.979780+0000","last_clean":"2024-02-13T00:54:28.979780+0000","last_became_active":"2024-02-13T00:52:30.595533+0000","last_became_peered":"2024-02-13T00:52:30.595533+0000","last_unstale":"2024-02-13T00:54:28.979780+0000","last_undegraded":"2024-02-13T00:54:28.979780+0000","last_fullsized":"2024-02-13T00:54:28.979780+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:53:53.023274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:03.464398+0000","last_clean_scrub_stamp":"2024-02-13T00:53:53.023274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":501,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-13T00:54:29.122691+0000","last_change":"2024-02-13T00:51:41.573986+0000","last_active":"2024-02-13T00:54:29.122691+0000","last_peered":"2024-02-13T00:54:29.122691+0000","last_clean":"2024-02-13T00:54:29.122691+0000","last_became_active":"2024-02-13T00:51:41.571183+0000","last_became_peered":"2024-02-13T00:51:41.571183+0000","last_unstale":"2024-02-13T00:54:29.122691+0000","last_undegraded":"2024-02-13T00:54:29.122691+0000","last_fullsized":"2024-02-13T00:54:29.122691+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:51:03.383666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:51:03.383666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1491,"num_read_kb":1216,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":780,"num_bytes_recovered":13820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":11436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":978,"ondisk_log_size":978,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1000,"num_read_kb":995,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266893,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319892,"kb_used_data":6284,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93425260,"statfs":{"total":95995035648,"available":95667466240,"internally_reserved":0,"allocated":6434816,"data_stored":3330944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.497,"15min":0.63200000000000001},"min":{"1min":0.27000000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.71499999999999997,"5min":0.76700000000000002,"15min":10.682},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.50900000000000001,"15min":0.74099999999999999},"min":{"1min":0.32600000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.68700000000000006,"5min":0.77400000000000002,"15min":25.504000000000001},"last":0.40000000000000002}]},{"osd":1,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.50800000000000001,"15min":0.57599999999999996},"min":{"1min":0.27000000000000002,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.68200000000000005,"5min":0.83499999999999996,"15min":13.347},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51500000000000001,"15min":0.65500000000000003},"min":{"1min":0.33900000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":13.526999999999999},"last":0.29399999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52300000000000002,"15min":0.65600000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.26600000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":14.359},"last":0.372},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53600000000000003,"15min":0.73499999999999999},"min":{"1min":0.36299999999999999,"5min":0.32000000000000001,"15min":0.26400000000000001},"max":{"1min":0.75,"5min":0.81599999999999995,"15min":21.497},"last":0.46800000000000003}]},{"osd":3,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51200000000000001,"15min":0.70199999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61099999999999999,"5min":0.89400000000000002,"15min":12.997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.81799999999999995},"min":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":0.79300000000000004,"15min":21.622},"last":0.435}]},{"osd":4,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.442,"15min":0.435},"min":{"1min":0.28899999999999998,"5min":0.26500000000000001,"15min":0.23000000000000001},"max":{"1min":0.69299999999999995,"5min":0.76000000000000001,"15min":2.452},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.47299999999999998,"15min":0.44700000000000001},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.245},"max":{"1min":0.60099999999999998,"5min":0.83199999999999996,"15min":1.0780000000000001},"last":0.48199999999999998}]},{"osd":5,"last update":"Tue Feb 13 00:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48799999999999999,"15min":0.502},"min":{"1min":0.35399999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.5,"15min":0.51500000000000001},"min":{"1min":0.31,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.64600000000000002,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.51600000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.49399999999999999},"min":{"1min":0.38,"5min":0.31,"15min":0.25},"max":{"1min":0.66500000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.497,"5min":0.51200000000000001,"15min":0.49099999999999999},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.27500000000000002},"max":{"1min":0.79300000000000004,"5min":0.85499999999999998,"15min":1.0940000000000001},"last":0.61599999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528531,"num_pgs":128,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307604,"kb_used_data":6284,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437548,"statfs":{"total":95995035648,"available":95680049152,"internally_reserved":0,"allocated":6434816,"data_stored":3331118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.498,"15min":0.53300000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.83099999999999996,"15min":8.1340000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.505,"15min":0.62},"min":{"1min":0.33900000000000002,"5min":0.27700000000000002,"15min":0.221},"max":{"1min":0.872,"5min":0.872,"15min":11.367000000000001},"last":0.44800000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.496,"15min":0.88200000000000001},"min":{"1min":0.32800000000000001,"5min":0.30099999999999999,"15min":0.255},"max":{"1min":0.81499999999999995,"5min":0.89300000000000002,"15min":54.002000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.51100000000000001,"15min":0.67900000000000005},"min":{"1min":0.27900000000000003,"5min":0.27200000000000002,"15min":0.23499999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":12.403},"last":0.55400000000000005}]},{"osd":2,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.501,"15min":0.71199999999999997},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.25600000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":12.477},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48999999999999999,"15min":0.67900000000000005},"min":{"1min":0.33800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":11.702},"last":0.57499999999999996}]},{"osd":3,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51200000000000001,"15min":0.67100000000000004},"min":{"1min":0.35399999999999998,"5min":0.31900000000000001,"15min":0.29199999999999998},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":8.0519999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.54000000000000004,"15min":0.77100000000000002},"min":{"1min":0.41799999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.74099999999999999,"5min":0.76000000000000001,"15min":19.081},"last":0.49099999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42399999999999999,"15min":0.42299999999999999},"min":{"1min":0.30399999999999999,"5min":0.26600000000000001,"15min":0.24199999999999999},"max":{"1min":0.65100000000000002,"5min":0.70599999999999996,"15min":0.72199999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46500000000000002,"15min":0.44900000000000001},"min":{"1min":0.307,"5min":0.252,"15min":0.221},"max":{"1min":0.64400000000000002,"5min":1.008,"15min":1.008},"last":0.625}]},{"osd":5,"last update":"Tue Feb 13 00:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.48999999999999999,"15min":0.51000000000000001},"min":{"1min":0.38200000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.499,"15min":0.52100000000000002},"min":{"1min":0.36599999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68000000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.60799999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47999999999999998,"15min":0.47299999999999998},"min":{"1min":0.34300000000000003,"5min":0.30199999999999999,"15min":0.27200000000000002},"max":{"1min":0.65600000000000003,"5min":0.74399999999999999,"15min":0.91600000000000004},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53200000000000003,"15min":0.51900000000000002},"min":{"1min":0.35199999999999998,"5min":0.312,"15min":0.26900000000000002},"max":{"1min":0.70399999999999996,"5min":0.76100000000000001,"15min":0.80700000000000005},"last":0.65500000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":125,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":324132,"kb_used_data":6300,"kb_used_omap":0,"kb_used_meta":317824,"kb_avail":93421020,"statfs":{"total":95995035648,"available":95663124480,"internally_reserved":0,"allocated":6451200,"data_stored":3330879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":325451776},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.95199999999999996},"min":{"1min":0.23300000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.61799999999999999,"5min":0.80600000000000005,"15min":61.856000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.34100000000000003,"15min":0.66800000000000004},"min":{"1min":0.246,"5min":0.19900000000000001,"15min":0.16500000000000001},"max":{"1min":0.44600000000000001,"5min":0.77600000000000002,"15min":61.908999999999999},"last":0.28599999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.83599999999999997},"min":{"1min":0.27000000000000002,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":0.49199999999999999,"5min":0.629,"15min":32.161000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41699999999999998,"15min":0.83599999999999997},"min":{"1min":0.26600000000000001,"5min":0.245,"15min":0.187},"max":{"1min":0.55800000000000005,"5min":5.1319999999999997,"15min":29.693000000000001},"last":0.378}]},{"osd":3,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":1.2470000000000001},"min":{"1min":0.28499999999999998,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":0.53600000000000003,"5min":0.75700000000000001,"15min":131.23099999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.39100000000000001,"15min":0.84199999999999997},"min":{"1min":0.28799999999999998,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":0.65500000000000003,"5min":0.753,"15min":49.055},"last":0.35399999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:53:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.53200000000000003,"15min":0.74399999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.85799999999999998,"15min":16.547000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58399999999999996,"15min":0.71199999999999997},"min":{"1min":0.39600000000000002,"5min":0.34300000000000003,"15min":0.24199999999999999},"max":{"1min":0.70899999999999996,"5min":0.85199999999999998,"15min":14.731999999999999},"last":0.68200000000000005}]},{"osd":5,"last update":"Tue Feb 13 00:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58599999999999997,"15min":0.57399999999999995},"min":{"1min":0.41599999999999998,"5min":0.39600000000000002,"15min":0.39400000000000002},"max":{"1min":0.78000000000000003,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.45500000000000002,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":0.745,"5min":1.831,"15min":1.831},"last":0.53100000000000003}]},{"osd":6,"last update":"Tue Feb 13 00:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57699999999999996,"15min":0.70799999999999996},"min":{"1min":0.44600000000000001,"5min":0.38600000000000001,"15min":0.29099999999999998},"max":{"1min":0.65500000000000003,"5min":1.9279999999999999,"15min":14.772},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55800000000000005,"15min":0.90900000000000003},"min":{"1min":0.44400000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80700000000000005,"5min":1.2709999999999999,"15min":26.600000000000001},"last":0.71499999999999997}]},{"osd":7,"last update":"Tue Feb 13 00:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59399999999999997,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.38200000000000001,"15min":0.29499999999999998},"max":{"1min":0.80900000000000005,"5min":1.8680000000000001,"15min":14.794},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57699999999999996,"15min":0.91200000000000003},"min":{"1min":0.44900000000000001,"5min":0.373,"15min":0.32200000000000001},"max":{"1min":0.75900000000000001,"5min":1.246,"15min":26.693999999999999},"last":0.64900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738608,"num_pgs":123,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303500,"kb_used_data":6276,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441652,"statfs":{"total":95995035648,"available":95684251648,"internally_reserved":0,"allocated":6426624,"data_stored":3330570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34699999999999998,"15min":0.61899999999999999},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.189},"max":{"1min":0.499,"5min":0.82699999999999996,"15min":30.184000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35999999999999999,"15min":0.45200000000000001},"min":{"1min":0.248,"5min":0.20499999999999999,"15min":0.187},"max":{"1min":0.60499999999999998,"5min":1.0109999999999999,"15min":5.6959999999999997},"last":0.377}]},{"osd":2,"last update":"Tue Feb 13 00:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.36699999999999999,"15min":0.48699999999999999},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.216},"max":{"1min":0.53100000000000003,"5min":1.0509999999999999,"15min":7.0839999999999996},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38600000000000001,"15min":0.67000000000000004},"min":{"1min":0.27500000000000002,"5min":0.23300000000000001,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":2.0129999999999999,"15min":24.459},"last":0.33800000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:54:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.372,"15min":0.47899999999999998},"min":{"1min":0.28000000000000003,"5min":0.214,"15min":0.214},"max":{"1min":0.51100000000000001,"5min":1.0640000000000001,"15min":8.718},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.52500000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.222},"max":{"1min":1.119,"5min":1.119,"15min":7.1139999999999999},"last":0.35499999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:54:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55600000000000005,"15min":0.80700000000000005},"min":{"1min":0.39700000000000002,"5min":0.34899999999999998,"15min":0.24399999999999999},"max":{"1min":0.65900000000000003,"5min":1.4710000000000001,"15min":24.381},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.68200000000000005},"min":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.28699999999999998},"max":{"1min":0.76000000000000001,"5min":2.6339999999999999,"15min":8.6950000000000003},"last":0.505}]},{"osd":5,"last update":"Tue Feb 13 00:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53500000000000003,"15min":0.50700000000000001},"min":{"1min":0.42199999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.64100000000000001,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":0.42699999999999999}]},{"osd":6,"last update":"Tue Feb 13 00:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55800000000000005,"15min":0.64000000000000001},"min":{"1min":0.39100000000000001,"5min":0.35999999999999999,"15min":0.31},"max":{"1min":0.80400000000000005,"5min":2.5960000000000001,"15min":8.6129999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56799999999999995,"15min":0.66100000000000003},"min":{"1min":0.44500000000000001,"5min":0.34000000000000002,"15min":0.32400000000000001},"max":{"1min":0.84199999999999997,"5min":1.411,"15min":14.263},"last":0.46300000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:54:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57899999999999996,"15min":0.66800000000000004},"min":{"1min":0.38700000000000001,"5min":0.33900000000000002,"15min":0.317},"max":{"1min":0.82599999999999996,"5min":2.496,"15min":8.6679999999999993},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58199999999999996,"15min":0.66900000000000004},"min":{"1min":0.49299999999999999,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.754,"5min":1.375,"15min":8.1609999999999996},"last":0.49199999999999999}]}]},{"osd":2,"up_from":0,"seq":77309411559,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":0,"seq":107374182625,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":133143986397,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":2057289334863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":3218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:54:38.622 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-13T00:54:38.622 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-13T00:54:38.622 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-02-13T00:54:38.622 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-13T00:54:38.622 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-13T00:54:38.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-02-13T00:54:38.896 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:54:39.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:54:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:54:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:54:39.128 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:54:39.129 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:54:39.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:54:39.129 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [] 2024-02-13T00:54:39.129 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:54:39.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:38.213622+0000 osd.1 (osd.1) 339 : cluster [DBG] 4.f deep-scrub starts 2024-02-13T00:54:39.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:38.218258+0000 osd.1 (osd.1) 340 : cluster [DBG] 4.f deep-scrub ok 2024-02-13T00:54:39.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:38.558918+0000 mgr.y (mgr.14544) 679 : cluster [DBG] pgmap v1238: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:54:39.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:39.197973+0000 osd.0 (osd.0) 325 : cluster [DBG] 28.1f scrub starts 2024-02-13T00:54:39.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:39.201655+0000 osd.0 (osd.0) 326 : cluster [DBG] 28.1f scrub ok 2024-02-13T00:54:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:38.213622+0000 osd.1 (osd.1) 339 : cluster [DBG] 4.f deep-scrub starts 2024-02-13T00:54:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:38.218258+0000 osd.1 (osd.1) 340 : cluster [DBG] 4.f deep-scrub ok 2024-02-13T00:54:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:38.558918+0000 mgr.y (mgr.14544) 679 : cluster [DBG] pgmap v1238: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:54:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:39.197973+0000 osd.0 (osd.0) 325 : cluster [DBG] 28.1f scrub starts 2024-02-13T00:54:39.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:39.201655+0000 osd.0 (osd.0) 326 : cluster [DBG] 28.1f scrub ok 2024-02-13T00:54:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:38.213622+0000 osd.1 (osd.1) 339 : cluster [DBG] 4.f deep-scrub starts 2024-02-13T00:54:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:38.218258+0000 osd.1 (osd.1) 340 : cluster [DBG] 4.f deep-scrub ok 2024-02-13T00:54:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:38.558918+0000 mgr.y (mgr.14544) 679 : cluster [DBG] pgmap v1238: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:54:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:39.197973+0000 osd.0 (osd.0) 325 : cluster [DBG] 28.1f scrub starts 2024-02-13T00:54:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:39.201655+0000 osd.0 (osd.0) 326 : cluster [DBG] 28.1f scrub ok 2024-02-13T00:54:39.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-13T00:54:40.223 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:40.725 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:39.237736+0000 osd.1 (osd.1) 341 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:54:40.725 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:39.241236+0000 osd.1 (osd.1) 342 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:54:40.725 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:40.594051+0000 mon.a (mon.0) 3767 : cluster [DBG] osdmap e807: 8 total, 8 up, 4 in 2024-02-13T00:54:40.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:39.237736+0000 osd.1 (osd.1) 341 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:54:40.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:39.241236+0000 osd.1 (osd.1) 342 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:54:40.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:40.594051+0000 mon.a (mon.0) 3767 : cluster [DBG] osdmap e807: 8 total, 8 up, 4 in 2024-02-13T00:54:40.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:39.237736+0000 osd.1 (osd.1) 341 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:54:40.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:39.241236+0000 osd.1 (osd.1) 342 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:54:40.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:40.594051+0000 mon.a (mon.0) 3767 : cluster [DBG] osdmap e807: 8 total, 8 up, 4 in 2024-02-13T00:54:40.800 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:54:40.800 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "" 2024-02-13T00:54:40.800 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:54:41.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:40.559588+0000 mgr.y (mgr.14544) 680 : cluster [DBG] pgmap v1239: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:54:41.755 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:40.559588+0000 mgr.y (mgr.14544) 680 : cluster [DBG] pgmap v1239: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:54:41.794 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:40.559588+0000 mgr.y (mgr.14544) 680 : cluster [DBG] pgmap v1239: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:54:42.027 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:54:42.027 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:54:42.027 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-13T00:54:42.027 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:54:42.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-13T00:54:43.012 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:43.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:42.517211+0000 osd.0 (osd.0) 327 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:42.517433+0000 osd.0 (osd.0) 328 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:43.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:42.517211+0000 osd.0 (osd.0) 327 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:43.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:42.517433+0000 osd.0 (osd.0) 328 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:43.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:42.517211+0000 osd.0 (osd.0) 327 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:43.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:42.517433+0000 osd.0 (osd.0) 328 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:42.560203+0000 mgr.y (mgr.14544) 681 : cluster [DBG] pgmap v1241: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:54:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:42.626064+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e808: 8 total, 8 up, 4 in 2024-02-13T00:54:43.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:42.560203+0000 mgr.y (mgr.14544) 681 : cluster [DBG] pgmap v1241: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:54:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:42.626064+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e808: 8 total, 8 up, 4 in 2024-02-13T00:54:44.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:42.560203+0000 mgr.y (mgr.14544) 681 : cluster [DBG] pgmap v1241: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:54:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:42.626064+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e808: 8 total, 8 up, 4 in 2024-02-13T00:54:44.943 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:54:44.943 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-13T00:54:44.943 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-13T00:54:44.944 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-13T00:54:44.944 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:54:45.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-13T00:54:46.016 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:43.734929+0000 osd.6 (osd.6) 267 : cluster [DBG] 2.e scrub starts 2024-02-13T00:54:46.016 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:43.737812+0000 osd.6 (osd.6) 268 : cluster [DBG] 2.e scrub ok 2024-02-13T00:54:46.016 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:43.783338+0000 osd.6 (osd.6) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:46.016 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:43.783585+0000 osd.6 (osd.6) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:46.016 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:44.560643+0000 mgr.y (mgr.14544) 682 : cluster [DBG] pgmap v1243: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 643 B/s rd, 0 op/s 2024-02-13T00:54:46.016 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:44.632644+0000 mon.a (mon.0) 3769 : cluster [DBG] osdmap e809: 8 total, 8 up, 4 in 2024-02-13T00:54:46.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:43.734929+0000 osd.6 (osd.6) 267 : cluster [DBG] 2.e scrub starts 2024-02-13T00:54:46.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:43.737812+0000 osd.6 (osd.6) 268 : cluster [DBG] 2.e scrub ok 2024-02-13T00:54:46.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:43.783338+0000 osd.6 (osd.6) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:46.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:43.783585+0000 osd.6 (osd.6) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:46.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:44.560643+0000 mgr.y (mgr.14544) 682 : cluster [DBG] pgmap v1243: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 643 B/s rd, 0 op/s 2024-02-13T00:54:46.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:44.632644+0000 mon.a (mon.0) 3769 : cluster [DBG] osdmap e809: 8 total, 8 up, 4 in 2024-02-13T00:54:46.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:43.734929+0000 osd.6 (osd.6) 267 : cluster [DBG] 2.e scrub starts 2024-02-13T00:54:46.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:43.737812+0000 osd.6 (osd.6) 268 : cluster [DBG] 2.e scrub ok 2024-02-13T00:54:46.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:43.783338+0000 osd.6 (osd.6) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:46.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:43.783585+0000 osd.6 (osd.6) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:46.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:44.560643+0000 mgr.y (mgr.14544) 682 : cluster [DBG] pgmap v1243: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 643 B/s rd, 0 op/s 2024-02-13T00:54:46.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:44.632644+0000 mon.a (mon.0) 3769 : cluster [DBG] osdmap e809: 8 total, 8 up, 4 in 2024-02-13T00:54:46.087 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:46.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:54:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:54:46.477 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-13T00:54:46.477 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-02-13T00:54:46.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd in 5 2024-02-13T00:54:46.503 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:54:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:54:46] "GET /metrics HTTP/1.1" 200 185427 "" "Prometheus/2.33.4" 2024-02-13T00:54:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:45.245855+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.a scrub starts 2024-02-13T00:54:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:45.248822+0000 osd.1 (osd.1) 344 : cluster [DBG] 2.a scrub ok 2024-02-13T00:54:46.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:46.012557+0000 mgr.y (mgr.14544) 683 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:46.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:45.245855+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.a scrub starts 2024-02-13T00:54:46.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:45.248822+0000 osd.1 (osd.1) 344 : cluster [DBG] 2.a scrub ok 2024-02-13T00:54:46.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:46.012557+0000 mgr.y (mgr.14544) 683 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:45.245855+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.a scrub starts 2024-02-13T00:54:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:45.248822+0000 osd.1 (osd.1) 344 : cluster [DBG] 2.a scrub ok 2024-02-13T00:54:47.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:46.012557+0000 mgr.y (mgr.14544) 683 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:47.398 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:47.933 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:46.249673+0000 osd.1 (osd.1) 345 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:54:47.933 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:46.253368+0000 osd.1 (osd.1) 346 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:54:47.934 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:46.561432+0000 mgr.y (mgr.14544) 684 : cluster [DBG] pgmap v1245: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:47.934 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:47.260356+0000 osd.0 (osd.0) 329 : cluster [DBG] 6.9 deep-scrub starts 2024-02-13T00:54:47.934 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:47.261541+0000 osd.0 (osd.0) 330 : cluster [DBG] 6.9 deep-scrub ok 2024-02-13T00:54:47.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:46.249673+0000 osd.1 (osd.1) 345 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:54:47.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:46.253368+0000 osd.1 (osd.1) 346 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:54:47.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:46.561432+0000 mgr.y (mgr.14544) 684 : cluster [DBG] pgmap v1245: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:47.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:47.260356+0000 osd.0 (osd.0) 329 : cluster [DBG] 6.9 deep-scrub starts 2024-02-13T00:54:47.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:47.261541+0000 osd.0 (osd.0) 330 : cluster [DBG] 6.9 deep-scrub ok 2024-02-13T00:54:48.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:46.249673+0000 osd.1 (osd.1) 345 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:54:48.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:46.253368+0000 osd.1 (osd.1) 346 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:54:48.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:46.561432+0000 mgr.y (mgr.14544) 684 : cluster [DBG] pgmap v1245: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:48.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:47.260356+0000 osd.0 (osd.0) 329 : cluster [DBG] 6.9 deep-scrub starts 2024-02-13T00:54:48.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:47.261541+0000 osd.0 (osd.0) 330 : cluster [DBG] 6.9 deep-scrub ok 2024-02-13T00:54:48.131 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-13T00:54:48.132 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-13T00:54:48.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-13T00:54:48.132 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [] 2024-02-13T00:54:48.132 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-13T00:54:48.795 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:46.698970+0000 osd.6 (osd.6) 271 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:54:48.795 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:46.702468+0000 osd.6 (osd.6) 272 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:54:48.795 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:47.413790+0000 osd.1 (osd.1) 347 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:48.795 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:47.414101+0000 osd.1 (osd.1) 348 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:48.796 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:47.647592+0000 mon.a (mon.0) 3770 : cluster [DBG] osdmap e810: 8 total, 8 up, 4 in 2024-02-13T00:54:48.796 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:48.066194+0000 osd.4 (osd.4) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:48.796 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:48.066566+0000 osd.4 (osd.4) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:46.698970+0000 osd.6 (osd.6) 271 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:54:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:46.702468+0000 osd.6 (osd.6) 272 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:54:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:47.413790+0000 osd.1 (osd.1) 347 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:47.414101+0000 osd.1 (osd.1) 348 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:47.647592+0000 mon.a (mon.0) 3770 : cluster [DBG] osdmap e810: 8 total, 8 up, 4 in 2024-02-13T00:54:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:48.066194+0000 osd.4 (osd.4) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:48.066566+0000 osd.4 (osd.4) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:49.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:54:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:54:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:54:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:46.698970+0000 osd.6 (osd.6) 271 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:54:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:46.702468+0000 osd.6 (osd.6) 272 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:54:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:47.413790+0000 osd.1 (osd.1) 347 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:47.414101+0000 osd.1 (osd.1) 348 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:47.647592+0000 mon.a (mon.0) 3770 : cluster [DBG] osdmap e810: 8 total, 8 up, 4 in 2024-02-13T00:54:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:48.066194+0000 osd.4 (osd.4) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:54:49.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:48.066566+0000 osd.4 (osd.4) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:54:49.881 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:48.561904+0000 mgr.y (mgr.14544) 685 : cluster [DBG] pgmap v1247: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:49.882 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:48.648408+0000 mon.a (mon.0) 3771 : cluster [DBG] osdmap e811: 8 total, 8 up, 4 in 2024-02-13T00:54:49.882 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:49.612413+0000 osd.6 (osd.6) 273 : cluster [DBG] 28.1a scrub starts 2024-02-13T00:54:49.882 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:49.615320+0000 osd.6 (osd.6) 274 : cluster [DBG] 28.1a scrub ok 2024-02-13T00:54:49.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:48.561904+0000 mgr.y (mgr.14544) 685 : cluster [DBG] pgmap v1247: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:49.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:48.648408+0000 mon.a (mon.0) 3771 : cluster [DBG] osdmap e811: 8 total, 8 up, 4 in 2024-02-13T00:54:49.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:49.612413+0000 osd.6 (osd.6) 273 : cluster [DBG] 28.1a scrub starts 2024-02-13T00:54:49.882 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:49.615320+0000 osd.6 (osd.6) 274 : cluster [DBG] 28.1a scrub ok 2024-02-13T00:54:49.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-13T00:54:49.937 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:48.561904+0000 mgr.y (mgr.14544) 685 : cluster [DBG] pgmap v1247: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:49.937 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:48.648408+0000 mon.a (mon.0) 3771 : cluster [DBG] osdmap e811: 8 total, 8 up, 4 in 2024-02-13T00:54:49.937 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:49.612413+0000 osd.6 (osd.6) 273 : cluster [DBG] 28.1a scrub starts 2024-02-13T00:54:49.937 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:49.615320+0000 osd.6 (osd.6) 274 : cluster [DBG] 28.1a scrub ok 2024-02-13T00:54:50.183 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:54:50.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.508+0000 7fcd01164700 1 -- 172.21.15.162:0/3554945262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccfc0fd490 msgr2=0x7fccfc0f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:50.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.508+0000 7fcd01164700 1 --2- 172.21.15.162:0/3554945262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccfc0fd490 0x7fccfc0f6cc0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7fccec004870 tx=0x7fccec009cb0).stop 2024-02-13T00:54:50.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.508+0000 7fcd01164700 1 -- 172.21.15.162:0/3554945262 shutdown_connections 2024-02-13T00:54:50.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.508+0000 7fcd01164700 1 --2- 172.21.15.162:0/3554945262 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccfc0f73a0 0x7fccfc1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:50.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.508+0000 7fcd01164700 1 --2- 172.21.15.162:0/3554945262 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fccfc0fcb30 0x7fccfc0fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:50.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.508+0000 7fcd01164700 1 --2- 172.21.15.162:0/3554945262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccfc0fd490 0x7fccfc0f6cc0 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:50.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.508+0000 7fcd01164700 1 -- 172.21.15.162:0/3554945262 >> 172.21.15.162:0/3554945262 conn(0x7fccfc06eb90 msgr2=0x7fccfc0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:50.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.508+0000 7fcd01164700 1 -- 172.21.15.162:0/3554945262 shutdown_connections 2024-02-13T00:54:50.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.508+0000 7fcd01164700 1 -- 172.21.15.162:0/3554945262 wait complete. 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.509+0000 7fcd01164700 1 librados: starting msgr at 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.509+0000 7fcd01164700 1 librados: starting objecter 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.509+0000 7fcd01164700 1 Processor -- start 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.509+0000 7fcd01164700 1 -- start start 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.509+0000 7fcd01164700 1 librados: setting wanted keys 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.509+0000 7fcd01164700 1 librados: calling monclient init 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.509+0000 7fcd01164700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccfc0f73a0 0x7fccfc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.509+0000 7fcd01164700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccfc0fcb30 0x7fccfc1096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.509+0000 7fcd01164700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fccfc0fd490 0x7fccfc109c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.509+0000 7fcd01164700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc1143a0 con 0x7fccfc0f73a0 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.509+0000 7fcd01164700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc114510 con 0x7fccfc0fd490 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.509+0000 7fcd01164700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc1146d0 con 0x7fccfc0fcb30 2024-02-13T00:54:50.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fccfad9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccfc0f73a0 0x7fccfc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fccfa59c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccfc0fcb30 0x7fccfc1096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fccfad9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccfc0f73a0 0x7fccfc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:51380/0 (socket says 172.21.15.162:51380) 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fccfad9d700 1 -- 172.21.15.162:0/2125003064 learned_addr learned my addr 172.21.15.162:0/2125003064 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fccfb59e700 1 --2- 172.21.15.162:0/2125003064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fccfc0fd490 0x7fccfc109c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fccfad9d700 1 -- 172.21.15.162:0/2125003064 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccfc0fcb30 msgr2=0x7fccfc1096a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fccfad9d700 1 --2- 172.21.15.162:0/2125003064 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccfc0fcb30 0x7fccfc1096a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fccfad9d700 1 -- 172.21.15.162:0/2125003064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fccfc0fd490 msgr2=0x7fccfc109c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fccfad9d700 1 --2- 172.21.15.162:0/2125003064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fccfc0fd490 0x7fccfc109c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fccfad9d700 1 -- 172.21.15.162:0/2125003064 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf40097c0 con 0x7fccfc0f73a0 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fccfa59c700 1 --2- 172.21.15.162:0/2125003064 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccfc0fcb30 0x7fccfc1096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fccfad9d700 1 --2- 172.21.15.162:0/2125003064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccfc0f73a0 0x7fccfc1091b0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7fccf00041d0 tx=0x7fccf0009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fcce3fff700 10 client.?.objecter ms_handle_connect 0x7fccfc0f73a0 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fcce3fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:50.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fcce3fff700 1 -- 172.21.15.162:0/2125003064 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0007b40 con 0x7fccfc0f73a0 2024-02-13T00:54:50.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fcce3fff700 1 -- 172.21.15.162:0/2125003064 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccf000f040 con 0x7fccfc0f73a0 2024-02-13T00:54:50.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fcd01164700 1 -- 172.21.15.162:0/2125003064 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccec005040 con 0x7fccfc0f73a0 2024-02-13T00:54:50.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fcce3fff700 1 -- 172.21.15.162:0/2125003064 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0014810 con 0x7fccfc0f73a0 2024-02-13T00:54:50.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fcd01164700 10 client.28218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:50.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fcd01164700 1 -- 172.21.15.162:0/2125003064 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccfc114d90 con 0x7fccfc0f73a0 2024-02-13T00:54:50.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.510+0000 7fcd01164700 1 librados: init done 2024-02-13T00:54:50.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.511+0000 7fcd01164700 1 -- 172.21.15.162:0/2125003064 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccfc0475e0 con 0x7fccfc0f73a0 2024-02-13T00:54:50.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.512+0000 7fcce3fff700 1 -- 172.21.15.162:0/2125003064 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fccf0009110 con 0x7fccfc0f73a0 2024-02-13T00:54:50.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.512+0000 7fcce3fff700 1 --2- 172.21.15.162:0/2125003064 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcce4062c90 0x7fcce4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:50.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.512+0000 7fcce3fff700 1 -- 172.21.15.162:0/2125003064 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(811..811 src has 449..811) v4 ==== 7422+0+0 (secure 0 0 0) 0x7fccf007cf20 con 0x7fccfc0f73a0 2024-02-13T00:54:50.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.512+0000 7fcce3fff700 10 client.28218.objecter ms_dispatch 0x7fccfc001550 osd_map(811..811 src has 449..811) v4 2024-02-13T00:54:50.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.512+0000 7fcce3fff700 3 client.28218.objecter handle_osd_map got epochs [811,811] > 0 2024-02-13T00:54:50.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.512+0000 7fcce3fff700 3 client.28218.objecter handle_osd_map decoding full epoch 811 2024-02-13T00:54:50.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.512+0000 7fccfa59c700 1 --2- 172.21.15.162:0/2125003064 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcce4062c90 0x7fcce4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:50.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.512+0000 7fcce3fff700 20 client.28218.objecter dump_active .. 0 homeless 2024-02-13T00:54:50.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.512+0000 7fcce3fff700 10 client.28218.objecter ms_handle_connect 0x7fcce4062c90 2024-02-13T00:54:50.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.512+0000 7fccfa59c700 1 --2- 172.21.15.162:0/2125003064 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcce4062c90 0x7fcce4065150 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 rx=0x7fccec0043d0 tx=0x7fccec00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:50.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.516+0000 7fcce3fff700 1 -- 172.21.15.162:0/2125003064 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fccf00506b0 con 0x7fccfc0f73a0 2024-02-13T00:54:50.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:50.715+0000 7fcd01164700 1 -- 172.21.15.162:0/2125003064 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd in", "ids": ["5"]} v 0) v1 -- 0x7fccfc106020 con 0x7fccfc0f73a0 2024-02-13T00:54:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:49.312914+0000 osd.1 (osd.1) 349 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:54:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:49.316651+0000 osd.1 (osd.1) 350 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:54:51.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:50.339975+0000 osd.0 (osd.0) 331 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:54:51.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:50.341105+0000 osd.0 (osd.0) 332 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:54:51.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:50.582324+0000 osd.6 (osd.6) 275 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:54:51.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:50.585212+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:54:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:49.312914+0000 osd.1 (osd.1) 349 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:54:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:49.316651+0000 osd.1 (osd.1) 350 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:54:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:50.339975+0000 osd.0 (osd.0) 331 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:54:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:50.341105+0000 osd.0 (osd.0) 332 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:54:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:50.582324+0000 osd.6 (osd.6) 275 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:54:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:50.585212+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:54:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:54:50.716+0000 7f5f6486f700 -1 mon.a@0(leader).osd e811 definitely_dead 0 2024-02-13T00:54:51.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:49.312914+0000 osd.1 (osd.1) 349 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:54:51.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:49.316651+0000 osd.1 (osd.1) 350 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:54:51.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:50.339975+0000 osd.0 (osd.0) 331 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:54:51.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:50.341105+0000 osd.0 (osd.0) 332 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:54:51.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:50.582324+0000 osd.6 (osd.6) 275 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:54:51.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:50.585212+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:54:51.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.714+0000 7fcce3fff700 1 -- 172.21.15.162:0/2125003064 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["5"]}]=0 marked in osd.5. v812) v1 ==== 85+0+0 (secure 0 0 0) 0x7fccfc106020 con 0x7fccfc0f73a0 2024-02-13T00:54:51.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.717+0000 7fcd01164700 10 librados: watch_flush enter 2024-02-13T00:54:51.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.718+0000 7fcd01164700 10 librados: watch_flush exit 2024-02-13T00:54:51.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.718+0000 7fcd01164700 20 client.28218.objecter shutdown clearing up homeless session... 2024-02-13T00:54:51.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.718+0000 7fcd01164700 10 client.28218.objecter successfully canceled tick 2024-02-13T00:54:51.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.718+0000 7fcd01164700 1 -- 172.21.15.162:0/2125003064 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcce4062c90 msgr2=0x7fcce4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:51.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.718+0000 7fcd01164700 1 --2- 172.21.15.162:0/2125003064 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcce4062c90 0x7fcce4065150 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 rx=0x7fccec0043d0 tx=0x7fccec00b040).stop 2024-02-13T00:54:51.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.718+0000 7fcd01164700 1 -- 172.21.15.162:0/2125003064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccfc0f73a0 msgr2=0x7fccfc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:51.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.718+0000 7fcd01164700 1 --2- 172.21.15.162:0/2125003064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccfc0f73a0 0x7fccfc1091b0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7fccf00041d0 tx=0x7fccf0009f60).stop 2024-02-13T00:54:51.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.722+0000 7fcd01164700 1 -- 172.21.15.162:0/2125003064 shutdown_connections 2024-02-13T00:54:51.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.722+0000 7fcd01164700 1 --2- 172.21.15.162:0/2125003064 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcce4062c90 0x7fcce4065150 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:51.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.722+0000 7fcd01164700 1 --2- 172.21.15.162:0/2125003064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fccfc0fd490 0x7fccfc109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:51.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.722+0000 7fcd01164700 1 --2- 172.21.15.162:0/2125003064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccfc0f73a0 0x7fccfc1091b0 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:51.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.722+0000 7fcd01164700 1 --2- 172.21.15.162:0/2125003064 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccfc0fcb30 0x7fccfc1096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:51.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.722+0000 7fcd01164700 1 -- 172.21.15.162:0/2125003064 >> 172.21.15.162:0/2125003064 conn(0x7fccfc06eb90 msgr2=0x7fccfc0fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:51.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.722+0000 7fcd01164700 1 -- 172.21.15.162:0/2125003064 shutdown_connections 2024-02-13T00:54:51.726 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.722+0000 7fcd01164700 1 -- 172.21.15.162:0/2125003064 wait complete. 2024-02-13T00:54:51.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:51.722+0000 7fcd01164700 1 librados: shutdown 2024-02-13T00:54:51.734 INFO:teuthology.orchestra.run.smithi162.stderr:marked in osd.5. 2024-02-13T00:54:51.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:50.562594+0000 mgr.y (mgr.14544) 686 : cluster [DBG] pgmap v1249: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:54:51.976 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:50.716923+0000 mon.a (mon.0) 3772 : audit [DBG] from='client.? 172.21.15.162:0/2125003064' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-13T00:54:52.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:50.562594+0000 mgr.y (mgr.14544) 686 : cluster [DBG] pgmap v1249: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:54:52.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:50.716923+0000 mon.a (mon.0) 3772 : audit [DBG] from='client.? 172.21.15.162:0/2125003064' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-13T00:54:52.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:50.562594+0000 mgr.y (mgr.14544) 686 : cluster [DBG] pgmap v1249: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:54:52.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:50.716923+0000 mon.a (mon.0) 3772 : audit [DBG] from='client.? 172.21.15.162:0/2125003064' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-13T00:54:52.246 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:54:52.246 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:54:52.246 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:54:52.246 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:54:52.553 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-02-13T00:54:52.553 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-13T00:54:52.553 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-02-13T00:54:52.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd in 4 2024-02-13T00:54:52.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:51.710509+0000 mon.a (mon.0) 3773 : audit [INF] from='client.? 172.21.15.162:0/2125003064' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-02-13T00:54:52.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:51.710592+0000 mon.a (mon.0) 3774 : cluster [DBG] osdmap e812: 8 total, 8 up, 5 in 2024-02-13T00:54:52.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:52.295913+0000 osd.0 (osd.0) 333 : cluster [DBG] 4.16 deep-scrub starts 2024-02-13T00:54:52.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:52.301463+0000 osd.0 (osd.0) 334 : cluster [DBG] 4.16 deep-scrub ok 2024-02-13T00:54:52.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:52.494505+0000 osd.6 (osd.6) 277 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:54:52.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:52.497436+0000 osd.6 (osd.6) 278 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:54:52.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:51.710509+0000 mon.a (mon.0) 3773 : audit [INF] from='client.? 172.21.15.162:0/2125003064' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-02-13T00:54:52.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:51.710592+0000 mon.a (mon.0) 3774 : cluster [DBG] osdmap e812: 8 total, 8 up, 5 in 2024-02-13T00:54:52.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:52.295913+0000 osd.0 (osd.0) 333 : cluster [DBG] 4.16 deep-scrub starts 2024-02-13T00:54:52.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:52.301463+0000 osd.0 (osd.0) 334 : cluster [DBG] 4.16 deep-scrub ok 2024-02-13T00:54:52.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:52.494505+0000 osd.6 (osd.6) 277 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:54:52.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:52.497436+0000 osd.6 (osd.6) 278 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:54:52.847 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:52.985 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:51.710509+0000 mon.a (mon.0) 3773 : audit [INF] from='client.? 172.21.15.162:0/2125003064' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-02-13T00:54:52.986 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:51.710592+0000 mon.a (mon.0) 3774 : cluster [DBG] osdmap e812: 8 total, 8 up, 5 in 2024-02-13T00:54:52.986 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:52.295913+0000 osd.0 (osd.0) 333 : cluster [DBG] 4.16 deep-scrub starts 2024-02-13T00:54:52.986 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:52.301463+0000 osd.0 (osd.0) 334 : cluster [DBG] 4.16 deep-scrub ok 2024-02-13T00:54:52.986 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:52.494505+0000 osd.6 (osd.6) 277 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:54:52.986 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:52.497436+0000 osd.6 (osd.6) 278 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:54:53.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-13T00:54:53.418 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:54:53.725 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:52.562979+0000 mgr.y (mgr.14544) 687 : cluster [DBG] pgmap v1251: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:53.725 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:52.716582+0000 mon.a (mon.0) 3775 : cluster [DBG] osdmap e813: 8 total, 8 up, 5 in 2024-02-13T00:54:53.944 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:52.562979+0000 mgr.y (mgr.14544) 687 : cluster [DBG] pgmap v1251: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:53.944 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:52.716582+0000 mon.a (mon.0) 3775 : cluster [DBG] osdmap e813: 8 total, 8 up, 5 in 2024-02-13T00:54:54.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:52.562979+0000 mgr.y (mgr.14544) 687 : cluster [DBG] pgmap v1251: 164 pgs: 164 active+clean; 5.7 KiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:54:54.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:52.716582+0000 mon.a (mon.0) 3775 : cluster [DBG] osdmap e813: 8 total, 8 up, 5 in 2024-02-13T00:54:54.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.694+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1137851423 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19c0f95e0 msgr2=0x7ff19c101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:54.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.694+0000 7ff1a30c0700 1 --2- 172.21.15.162:0/1137851423 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19c0f95e0 0x7ff19c101d80 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 rx=0x7ff1880047b0 tx=0x7ff188009cb0).stop 2024-02-13T00:54:54.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.694+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1137851423 shutdown_connections 2024-02-13T00:54:54.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.694+0000 7ff1a30c0700 1 --2- 172.21.15.162:0/1137851423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff19c102270 0x7ff19c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:54.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.694+0000 7ff1a30c0700 1 --2- 172.21.15.162:0/1137851423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff19c0f8d10 0x7ff19c0f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.694+0000 7ff1a30c0700 1 --2- 172.21.15.162:0/1137851423 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19c0f95e0 0x7ff19c101d80 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.694+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1137851423 >> 172.21.15.162:0/1137851423 conn(0x7ff19c06eb90 msgr2=0x7ff19c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.694+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1137851423 shutdown_connections 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.694+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1137851423 wait complete. 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.694+0000 7ff1a30c0700 1 librados: starting msgr at 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.694+0000 7ff1a30c0700 1 librados: starting objecter 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a30c0700 1 Processor -- start 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a30c0700 1 -- start start 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a30c0700 1 librados: setting wanted keys 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a30c0700 1 librados: calling monclient init 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a30c0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19c0f8d10 0x7ff19c10e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a30c0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff19c0f95e0 0x7ff19c10e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:54.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a30c0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff19c102270 0x7ff19c108450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a30c0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c108940 con 0x7ff19c0f95e0 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a30c0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c108ab0 con 0x7ff19c102270 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a30c0700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c108c20 con 0x7ff19c0f8d10 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a0e5c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19c0f8d10 0x7ff19c10e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a0e5c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19c0f8d10 0x7ff19c10e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:35602/0 (socket says 172.21.15.162:35602) 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a0e5c700 1 -- 172.21.15.162:0/1138034949 learned_addr learned my addr 172.21.15.162:0/1138034949 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff19bfff700 1 --2- 172.21.15.162:0/1138034949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff19c0f95e0 0x7ff19c10e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a0e5c700 1 -- 172.21.15.162:0/1138034949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff19c102270 msgr2=0x7ff19c108450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a0e5c700 1 --2- 172.21.15.162:0/1138034949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff19c102270 0x7ff19c108450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a0e5c700 1 -- 172.21.15.162:0/1138034949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff19c0f95e0 msgr2=0x7ff19c10e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a0e5c700 1 --2- 172.21.15.162:0/1138034949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff19c0f95e0 0x7ff19c10e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.695+0000 7ff1a0e5c700 1 -- 172.21.15.162:0/1138034949 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1940097c0 con 0x7ff19c0f8d10 2024-02-13T00:54:54.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.696+0000 7ff1a0e5c700 1 --2- 172.21.15.162:0/1138034949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19c0f8d10 0x7ff19c10e270 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 rx=0x7ff19000e970 tx=0x7ff19000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.696+0000 7ff199ffb700 10 client.?.objecter ms_handle_connect 0x7ff19c0f8d10 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.696+0000 7ff199ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.696+0000 7ff199ffb700 1 -- 172.21.15.162:0/1138034949 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19000ee60 con 0x7ff19c0f8d10 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.696+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1138034949 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff188005040 con 0x7ff19c0f8d10 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.696+0000 7ff199ffb700 1 -- 172.21.15.162:0/1138034949 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff19000f040 con 0x7ff19c0f8d10 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.696+0000 7ff199ffb700 1 -- 172.21.15.162:0/1138034949 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190007ea0 con 0x7ff19c0f8d10 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.696+0000 7ff1a30c0700 10 client.38318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.696+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1138034949 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19c1092b0 con 0x7ff19c0f8d10 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.696+0000 7ff1a30c0700 1 librados: init done 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.697+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1138034949 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19c1071b0 con 0x7ff19c0f8d10 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.698+0000 7ff199ffb700 1 -- 172.21.15.162:0/1138034949 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff19000cb10 con 0x7ff19c0f8d10 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.698+0000 7ff199ffb700 1 --2- 172.21.15.162:0/1138034949 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff18c062c90 0x7ff18c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.698+0000 7ff199ffb700 1 -- 172.21.15.162:0/1138034949 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(814..814 src has 449..814) v4 ==== 7521+0+0 (secure 0 0 0) 0x7ff19007cbb0 con 0x7ff19c0f8d10 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.698+0000 7ff199ffb700 10 client.38318.objecter ms_dispatch 0x7ff19c001550 osd_map(814..814 src has 449..814) v4 2024-02-13T00:54:54.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.698+0000 7ff199ffb700 3 client.38318.objecter handle_osd_map got epochs [814,814] > 0 2024-02-13T00:54:54.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.698+0000 7ff199ffb700 3 client.38318.objecter handle_osd_map decoding full epoch 814 2024-02-13T00:54:54.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.698+0000 7ff19bfff700 1 --2- 172.21.15.162:0/1138034949 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff18c062c90 0x7ff18c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:54.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.698+0000 7ff199ffb700 20 client.38318.objecter dump_active .. 0 homeless 2024-02-13T00:54:54.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.698+0000 7ff199ffb700 10 client.38318.objecter ms_handle_connect 0x7ff18c062c90 2024-02-13T00:54:54.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.698+0000 7ff19bfff700 1 --2- 172.21.15.162:0/1138034949 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff18c062c90 0x7ff18c065150 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 rx=0x7ff188005010 tx=0x7ff188009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:54.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.706+0000 7ff199ffb700 1 -- 172.21.15.162:0/1138034949 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff19004af60 con 0x7ff19c0f8d10 2024-02-13T00:54:54.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:53.717085+0000 mon.a (mon.0) 3776 : cluster [DBG] osdmap e814: 8 total, 8 up, 5 in 2024-02-13T00:54:54.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:53.720930+0000 osd.0 (osd.0) 335 : cluster [DBG] 5.d starting backfill to osd.5 from (0'0,0'0] MAX to 726'19 2024-02-13T00:54:54.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:53.721451+0000 osd.0 (osd.0) 336 : cluster [DBG] 5.f starting backfill to osd.5 from (0'0,0'0] MAX to 728'19 2024-02-13T00:54:54.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:53.721952+0000 osd.0 (osd.0) 337 : cluster [DBG] 5.e starting backfill to osd.5 from (0'0,0'0] MAX to 727'19 2024-02-13T00:54:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:53.717085+0000 mon.a (mon.0) 3776 : cluster [DBG] osdmap e814: 8 total, 8 up, 5 in 2024-02-13T00:54:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:53.720930+0000 osd.0 (osd.0) 335 : cluster [DBG] 5.d starting backfill to osd.5 from (0'0,0'0] MAX to 726'19 2024-02-13T00:54:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:53.721451+0000 osd.0 (osd.0) 336 : cluster [DBG] 5.f starting backfill to osd.5 from (0'0,0'0] MAX to 728'19 2024-02-13T00:54:54.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:53.721952+0000 osd.0 (osd.0) 337 : cluster [DBG] 5.e starting backfill to osd.5 from (0'0,0'0] MAX to 727'19 2024-02-13T00:54:54.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:54.893+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1138034949 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd in", "ids": ["4"]} v 0) v1 -- 0x7ff19c109930 con 0x7ff19c0f8d10 2024-02-13T00:54:55.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:54:54.894+0000 7f5f6486f700 -1 mon.a@0(leader).osd e814 definitely_dead 0 2024-02-13T00:54:55.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:53.717085+0000 mon.a (mon.0) 3776 : cluster [DBG] osdmap e814: 8 total, 8 up, 5 in 2024-02-13T00:54:55.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:53.720930+0000 osd.0 (osd.0) 335 : cluster [DBG] 5.d starting backfill to osd.5 from (0'0,0'0] MAX to 726'19 2024-02-13T00:54:55.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:53.721451+0000 osd.0 (osd.0) 336 : cluster [DBG] 5.f starting backfill to osd.5 from (0'0,0'0] MAX to 728'19 2024-02-13T00:54:55.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:53.721952+0000 osd.0 (osd.0) 337 : cluster [DBG] 5.e starting backfill to osd.5 from (0'0,0'0] MAX to 727'19 2024-02-13T00:54:55.265 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:54:55.265 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:54:55.265 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-02-13T00:54:55.265 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-02-13T00:54:55.266 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:54:55.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.600+0000 7ff199ffb700 1 -- 172.21.15.162:0/1138034949 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["4"]}]=0 marked in osd.4. v815) v1 ==== 85+0+0 (secure 0 0 0) 0x7ff190019020 con 0x7ff19c0f8d10 2024-02-13T00:54:55.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.604+0000 7ff1a30c0700 10 librados: watch_flush enter 2024-02-13T00:54:55.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.604+0000 7ff1a30c0700 10 librados: watch_flush exit 2024-02-13T00:54:55.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.604+0000 7ff1a30c0700 20 client.38318.objecter shutdown clearing up homeless session... 2024-02-13T00:54:55.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.604+0000 7ff1a30c0700 10 client.38318.objecter successfully canceled tick 2024-02-13T00:54:55.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.605+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1138034949 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff18c062c90 msgr2=0x7ff18c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:55.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.605+0000 7ff1a30c0700 1 --2- 172.21.15.162:0/1138034949 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff18c062c90 0x7ff18c065150 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 rx=0x7ff188005010 tx=0x7ff188009cb0).stop 2024-02-13T00:54:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.605+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1138034949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19c0f8d10 msgr2=0x7ff19c10e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.605+0000 7ff1a30c0700 1 --2- 172.21.15.162:0/1138034949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19c0f8d10 0x7ff19c10e270 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 rx=0x7ff19000e970 tx=0x7ff19000b040).stop 2024-02-13T00:54:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.605+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1138034949 shutdown_connections 2024-02-13T00:54:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.605+0000 7ff1a30c0700 1 --2- 172.21.15.162:0/1138034949 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff18c062c90 0x7ff18c065150 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.605+0000 7ff1a30c0700 1 --2- 172.21.15.162:0/1138034949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff19c102270 0x7ff19c108450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:54:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.605+0000 7ff1a30c0700 1 --2- 172.21.15.162:0/1138034949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19c0f8d10 0x7ff19c10e270 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.605+0000 7ff1a30c0700 1 --2- 172.21.15.162:0/1138034949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff19c0f95e0 0x7ff19c10e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:55.609 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.605+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1138034949 >> 172.21.15.162:0/1138034949 conn(0x7ff19c06eb90 msgr2=0x7ff19c102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.607+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1138034949 shutdown_connections 2024-02-13T00:54:55.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.607+0000 7ff1a30c0700 1 -- 172.21.15.162:0/1138034949 wait complete. 2024-02-13T00:54:55.617 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:55.609+0000 7ff1a30c0700 1 librados: shutdown 2024-02-13T00:54:55.624 INFO:teuthology.orchestra.run.smithi162.stderr:marked in osd.4. 2024-02-13T00:54:55.986 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:54.563389+0000 mgr.y (mgr.14544) 688 : cluster [DBG] pgmap v1254: 164 pgs: 2 active+recovering, 162 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 519 B/s rd, 0 op/s; 6/675 objects misplaced (0.889%) 2024-02-13T00:54:55.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:54.894527+0000 mon.c (mon.2) 521 : audit [DBG] from='client.? 172.21.15.162:0/1138034949' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-13T00:54:55.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:54.894851+0000 mon.a (mon.0) 3777 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-13T00:54:55.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:55.598135+0000 mon.a (mon.0) 3778 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-13T00:54:55.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:55.598217+0000 mon.a (mon.0) 3779 : cluster [DBG] osdmap e815: 8 total, 8 up, 6 in 2024-02-13T00:54:56.006 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-13T00:54:56.017 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:54.563389+0000 mgr.y (mgr.14544) 688 : cluster [DBG] pgmap v1254: 164 pgs: 2 active+recovering, 162 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 519 B/s rd, 0 op/s; 6/675 objects misplaced (0.889%) 2024-02-13T00:54:56.017 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:54.894527+0000 mon.c (mon.2) 521 : audit [DBG] from='client.? 172.21.15.162:0/1138034949' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-13T00:54:56.017 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:54.894851+0000 mon.a (mon.0) 3777 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-13T00:54:56.017 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:55.598135+0000 mon.a (mon.0) 3778 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-13T00:54:56.017 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:55.598217+0000 mon.a (mon.0) 3779 : cluster [DBG] osdmap e815: 8 total, 8 up, 6 in 2024-02-13T00:54:56.017 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:54.563389+0000 mgr.y (mgr.14544) 688 : cluster [DBG] pgmap v1254: 164 pgs: 2 active+recovering, 162 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 519 B/s rd, 0 op/s; 6/675 objects misplaced (0.889%) 2024-02-13T00:54:56.018 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:54.894527+0000 mon.c (mon.2) 521 : audit [DBG] from='client.? 172.21.15.162:0/1138034949' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-13T00:54:56.018 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:54.894851+0000 mon.a (mon.0) 3777 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-13T00:54:56.018 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:55.598135+0000 mon.a (mon.0) 3778 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-13T00:54:56.018 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:55.598217+0000 mon.a (mon.0) 3779 : cluster [DBG] osdmap e815: 8 total, 8 up, 6 in 2024-02-13T00:54:56.271 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:54:56.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:54:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:54:56.329 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:54:56] "GET /metrics HTTP/1.1" 200 185430 "" "Prometheus/2.33.4" 2024-02-13T00:54:56.399 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-02-13T00:54:56.399 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-13T00:54:56.399 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-02-13T00:54:56.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd in 3 2024-02-13T00:54:56.694 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:54:56.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:56.023151+0000 mgr.y (mgr.14544) 689 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:56.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:56.209950+0000 osd.0 (osd.0) 338 : cluster [DBG] 28.e scrub starts 2024-02-13T00:54:56.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:56.212880+0000 osd.0 (osd.0) 339 : cluster [DBG] 28.e scrub ok 2024-02-13T00:54:56.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:56.283001+0000 osd.5 (osd.5) 13 : cluster [DBG] 28.18 scrub starts 2024-02-13T00:54:56.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:56.284022+0000 osd.5 (osd.5) 14 : cluster [DBG] 28.18 scrub ok 2024-02-13T00:54:56.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:56.524710+0000 osd.6 (osd.6) 279 : cluster [DBG] 6.c scrub starts 2024-02-13T00:54:56.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:56.527596+0000 osd.6 (osd.6) 280 : cluster [DBG] 6.c scrub ok 2024-02-13T00:54:56.740 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:56.600722+0000 mon.a (mon.0) 3780 : cluster [DBG] osdmap e816: 8 total, 8 up, 6 in 2024-02-13T00:54:57.006 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:56.023151+0000 mgr.y (mgr.14544) 689 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:57.007 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:56.209950+0000 osd.0 (osd.0) 338 : cluster [DBG] 28.e scrub starts 2024-02-13T00:54:57.007 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:56.212880+0000 osd.0 (osd.0) 339 : cluster [DBG] 28.e scrub ok 2024-02-13T00:54:57.007 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:56.283001+0000 osd.5 (osd.5) 13 : cluster [DBG] 28.18 scrub starts 2024-02-13T00:54:57.007 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:56.284022+0000 osd.5 (osd.5) 14 : cluster [DBG] 28.18 scrub ok 2024-02-13T00:54:57.007 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:56.524710+0000 osd.6 (osd.6) 279 : cluster [DBG] 6.c scrub starts 2024-02-13T00:54:57.007 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:56.527596+0000 osd.6 (osd.6) 280 : cluster [DBG] 6.c scrub ok 2024-02-13T00:54:57.007 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:56.600722+0000 mon.a (mon.0) 3780 : cluster [DBG] osdmap e816: 8 total, 8 up, 6 in 2024-02-13T00:54:57.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:56.023151+0000 mgr.y (mgr.14544) 689 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:54:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:56.209950+0000 osd.0 (osd.0) 338 : cluster [DBG] 28.e scrub starts 2024-02-13T00:54:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:56.212880+0000 osd.0 (osd.0) 339 : cluster [DBG] 28.e scrub ok 2024-02-13T00:54:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:56.283001+0000 osd.5 (osd.5) 13 : cluster [DBG] 28.18 scrub starts 2024-02-13T00:54:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:56.284022+0000 osd.5 (osd.5) 14 : cluster [DBG] 28.18 scrub ok 2024-02-13T00:54:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:56.524710+0000 osd.6 (osd.6) 279 : cluster [DBG] 6.c scrub starts 2024-02-13T00:54:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:56.527596+0000 osd.6 (osd.6) 280 : cluster [DBG] 6.c scrub ok 2024-02-13T00:54:57.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:56.600722+0000 mon.a (mon.0) 3780 : cluster [DBG] osdmap e816: 8 total, 8 up, 6 in 2024-02-13T00:54:57.974 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:56.563760+0000 mgr.y (mgr.14544) 690 : cluster [DBG] pgmap v1256: 164 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 4 active+recovering, 157 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 632 B/s rd, 0 op/s; 12/675 objects misplaced (1.778%); 406 B/s, 18 objects/s recovering 2024-02-13T00:54:57.974 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:56.601204+0000 osd.0 (osd.0) 340 : cluster [DBG] 5.f starting backfill to osd.4 from (0'0,0'0] MAX to 814'21 2024-02-13T00:54:57.974 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:56.601281+0000 osd.0 (osd.0) 341 : cluster [INF] 5.f continuing backfill to osd.5 from (363'14,814'21] MIN to 814'21 2024-02-13T00:54:57.974 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:56.726369+0000 mon.a (mon.0) 3781 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-13T00:54:57.974 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:57.601039+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e817: 8 total, 8 up, 6 in 2024-02-13T00:54:57.975 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:57.603708+0000 osd.6 (osd.6) 281 : cluster [DBG] 5.9 starting backfill to osd.4 from (0'0,0'0] MAX to 728'22 2024-02-13T00:54:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:56.563760+0000 mgr.y (mgr.14544) 690 : cluster [DBG] pgmap v1256: 164 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 4 active+recovering, 157 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 632 B/s rd, 0 op/s; 12/675 objects misplaced (1.778%); 406 B/s, 18 objects/s recovering 2024-02-13T00:54:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:56.601204+0000 osd.0 (osd.0) 340 : cluster [DBG] 5.f starting backfill to osd.4 from (0'0,0'0] MAX to 814'21 2024-02-13T00:54:58.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:56.601281+0000 osd.0 (osd.0) 341 : cluster [INF] 5.f continuing backfill to osd.5 from (363'14,814'21] MIN to 814'21 2024-02-13T00:54:58.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:56.726369+0000 mon.a (mon.0) 3781 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-13T00:54:58.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:57.601039+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e817: 8 total, 8 up, 6 in 2024-02-13T00:54:58.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:57.603708+0000 osd.6 (osd.6) 281 : cluster [DBG] 5.9 starting backfill to osd.4 from (0'0,0'0] MAX to 728'22 2024-02-13T00:54:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:56.563760+0000 mgr.y (mgr.14544) 690 : cluster [DBG] pgmap v1256: 164 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 4 active+recovering, 157 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 632 B/s rd, 0 op/s; 12/675 objects misplaced (1.778%); 406 B/s, 18 objects/s recovering 2024-02-13T00:54:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:56.601204+0000 osd.0 (osd.0) 340 : cluster [DBG] 5.f starting backfill to osd.4 from (0'0,0'0] MAX to 814'21 2024-02-13T00:54:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:56.601281+0000 osd.0 (osd.0) 341 : cluster [INF] 5.f continuing backfill to osd.5 from (363'14,814'21] MIN to 814'21 2024-02-13T00:54:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:56.726369+0000 mon.a (mon.0) 3781 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-13T00:54:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:57.601039+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e817: 8 total, 8 up, 6 in 2024-02-13T00:54:58.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:57.603708+0000 osd.6 (osd.6) 281 : cluster [DBG] 5.9 starting backfill to osd.4 from (0'0,0'0] MAX to 728'22 2024-02-13T00:54:58.250 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:54:58.250 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:54:58.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:54:58.250 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:54:58.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.250 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(28.c REQUEST e379)", 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:45:34.516696+0000", 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 563.72890724800004, 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003757631, 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516696+0000", 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516696+0000", 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516695+0000", 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1699999999999997e-06 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516703+0000", 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0827000000000001e-05 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516734+0000", 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5639000000000001e-05 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516750+0000", 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011122500000000001 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516861+0000", 2024-02-13T00:54:58.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6915999999999999e-05 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.516878+0000", 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035454750000000002 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.520424+0000", 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0348e-05 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.520434+0000", 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9860999999999998e-05 2024-02-13T00:54:58.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:34.520454+0000", 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003757631 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(6.17 REQUEST e438)", 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:47:02.781512+0000", 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 475.46409124299998, 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043540109999999996, 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781512+0000", 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781512+0000", 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781512+0000", 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4500000000000003e-06 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781517+0000", 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4729999999999999e-05 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781542+0000", 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3523000000000001e-05 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781555+0000", 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010741799999999999 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781663+0000", 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9942999999999999e-05 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.781683+0000", 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041526920000000004 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.785835+0000", 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2148e-05 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.785848+0000", 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8539999999999999e-05 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:47:02.785866+0000", 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043540109999999996 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.259 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.260 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(28.17 REQUEST e612)", 2024-02-13T00:54:58.260 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:50:26.821965+0000", 2024-02-13T00:54:58.260 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 271.42363854799999, 2024-02-13T00:54:58.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049237200000000004, 2024-02-13T00:54:58.260 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.260 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:54:58.260 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821965+0000", 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821965+0000", 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821964+0000", 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1099999999999999e-06 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821970+0000", 2024-02-13T00:54:58.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6339999999999999e-05 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.821997+0000", 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4263e-05 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.822011+0000", 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012683799999999999 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.822138+0000", 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8040000000000002e-06 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.822146+0000", 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046927059999999996 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.826838+0000", 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9426e-05 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.826858+0000", 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0803000000000003e-05 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:50:26.826889+0000", 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049237200000000004 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.264 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.5s1 639/599 ECSubWrite(tid=706, reqid=client.27753.0:474, at_version=0'0, trim_to=0'0, roll_forward_to=639'6))", 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:04.536955+0000", 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 233.70864819600001, 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012597979999999999, 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536955+0000", 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536955+0000", 2024-02-13T00:54:58.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536955+0000", 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2790000000000004e-06 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536964+0000", 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8669999999999996e-06 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536971+0000", 2024-02-13T00:54:58.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1420000000000004e-06 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.536978+0000", 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010723200000000001 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.537085+0000", 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0047999999999999e-05 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.537095+0000", 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011055310000000001 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.538201+0000", 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.396e-05 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:04.538215+0000", 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012597979999999999 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.268 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(6.e REQUEST e640)", 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:05.640107+0000", 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 232.605496668, 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00283475, 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640107+0000", 2024-02-13T00:54:58.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640107+0000", 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640106+0000", 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4880000000000001e-06 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640109+0000", 2024-02-13T00:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.359e-05 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640123+0000", 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6920000000000002e-06 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640130+0000", 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0739000000000003e-05 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640181+0000", 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8570000000000007e-06 2024-02-13T00:54:58.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.640191+0000", 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026984040000000002 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.642889+0000", 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6985000000000001e-05 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.642906+0000", 2024-02-13T00:54:58.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5238999999999999e-05 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:05.642941+0000", 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00283475 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.7s1 660/599 ECSubWrite(tid=1084, reqid=osd.1.0:1084, at_version=660'16, trim_to=0'0, roll_forward_to=656'12))", 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:25.619279+0000", 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 212.626324681, 2024-02-13T00:54:58.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028506830000000001, 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619279+0000", 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619279+0000", 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619278+0000", 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8071e-05 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619296+0000", 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2529e-05 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619309+0000", 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0450000000000003e-06 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619316+0000", 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4156000000000001e-05 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619370+0000", 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2890000000000003e-06 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.619379+0000", 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027309360000000002 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.622110+0000", 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9262999999999998e-05 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.622129+0000", 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028506830000000001 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.7s1 660/599 ECSubWrite(tid=1085, reqid=osd.1.0:1084, at_version=0'0, trim_to=0'0, roll_forward_to=660'16))", 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:25.624225+0000", 2024-02-13T00:54:58.277 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 212.62137841800001, 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011742090000000001, 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624225+0000", 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624225+0000", 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624224+0000", 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.024e-05 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624235+0000", 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6730000000000004e-06 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624244+0000", 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6660000000000002e-06 2024-02-13T00:54:58.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624251+0000", 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8631000000000002e-05 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624309+0000", 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4010000000000003e-06 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.624319+0000", 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010623919999999999 2024-02-13T00:54:58.280 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.625381+0000", 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7849000000000001e-05 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:25.625399+0000", 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011742090000000001 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.1.0:1092 243.f e676/675 243:f2a1c75f:test-rados-api-smithi162-84057-119::foo:head v 676'1, mlcod=0'0)", 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:41.579543+0000", 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 196.666060273, 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081382700000000004, 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579543+0000", 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579543+0000", 2024-02-13T00:54:58.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579542+0000", 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9680000000000001e-06 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579552+0000", 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2031999999999999e-05 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579564+0000", 2024-02-13T00:54:58.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6990000000000003e-06 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579572+0000", 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9949e-05 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579612+0000", 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8391000000000001e-05 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:54:58.284 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.579630+0000", 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00069848999999999996 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580329+0000", 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8271000000000001e-05 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580357+0000", 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081382700000000004 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.27753.0:1101 243.f e676/675 243:f2a1c75f:test-rados-api-smithi162-84057-119::foo:head v 676'2, mlcod=0'0)", 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:41.580331+0000", 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 196.66527240100001, 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00087110900000000001, 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580331+0000", 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580331+0000", 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580330+0000", 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2383e-05 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580342+0000", 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0047000000000001e-05 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580353+0000", 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6449999999999994e-06 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580361+0000", 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6291999999999997e-05 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580397+0000", 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8377e-05 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.580416+0000", 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00075487600000000003 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.581171+0000", 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1349000000000001e-05 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.581202+0000", 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00087110900000000001 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.d epoch 677/676 rep_tid 1908 entries 677'14 (0'0) error 5:b4812045:::notify.4:head by client.24329.0:2087 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:44.375642+0000", 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 193.86996165299999, 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065348800000000003, 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375642+0000", 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375642+0000", 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375641+0000", 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2651e-05 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375654+0000", 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7756000000000001e-05 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375671+0000", 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4800000000000001e-06 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375680+0000", 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010102000000000001 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375781+0000", 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00051420100000000005 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376295+0000", 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065348800000000003 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.e epoch 677/676 rep_tid 1907 entries 677'14 (0'0) error 5:7759931f:::notify.3:head by client.24329.0:2086 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:44.375694+0000", 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 193.869909111, 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081520500000000005, 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.293 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375694+0000", 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375694+0000", 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375694+0000", 2024-02-13T00:54:58.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7979999999999996e-06 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375699+0000", 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2609999999999996e-06 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375706+0000", 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9020000000000004e-06 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375712+0000", 2024-02-13T00:54:58.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014652799999999999 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375858+0000", 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000651028 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376509+0000", 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081520500000000005 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.e epoch 677/676 rep_tid 1911 entries 677'15 (0'0) error 5:7759931f:::notify.3:head by client.24329.0:2091 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:44.376063+0000", 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 193.869540812, 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00085006100000000005, 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376063+0000", 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376063+0000", 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376062+0000", 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9030000000000003e-06 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376067+0000", 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5970000000000003e-06 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.298 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376075+0000", 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0220000000000001e-06 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376081+0000", 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000341861 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376423+0000", 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049005800000000003 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376913+0000", 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00085006100000000005 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s2 697/675 ECSubWrite(tid=779, reqid=osd.1.13:208, at_version=697'2, trim_to=0'0, roll_forward_to=0'0))", 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:04.576118+0000", 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 173.669484866, 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012258690000000001, 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.300 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576118+0000", 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576118+0000", 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576118+0000", 2024-02-13T00:54:58.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7736999999999999e-05 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576145+0000", 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0842000000000001e-05 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576156+0000", 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7569999999999993e-06 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576165+0000", 2024-02-13T00:54:58.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061519500000000002 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576780+0000", 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9189999999999994e-06 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576790+0000", 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00053753199999999996 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.577327+0000", 2024-02-13T00:54:58.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6843000000000001e-05 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.577344+0000", 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012258690000000001 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s2 697/675 ECSubWrite(tid=781, reqid=osd.1.13:209, at_version=697'4, trim_to=0'0, roll_forward_to=697'2))", 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:04.580507+0000", 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 173.665095979, 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061816799999999999, 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.304 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580507+0000", 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580507+0000", 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580507+0000", 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4573e-05 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580531+0000", 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0698e-05 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580542+0000", 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2680000000000002e-06 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580547+0000", 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1179e-05 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580588+0000", 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1452e-05 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.580600+0000", 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050946100000000001 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.581109+0000", 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6378000000000001e-05 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.581126+0000", 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061816799999999999 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.308 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s2 697/675 ECSubWrite(tid=783, reqid=osd.1.13:210, at_version=697'6, trim_to=0'0, roll_forward_to=697'4))", 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:04.586217+0000", 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 173.65938634299999, 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057734300000000002, 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586217+0000", 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586217+0000", 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586216+0000", 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9509000000000002e-05 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586246+0000", 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1206000000000001e-05 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586257+0000", 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7780000000000002e-06 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586263+0000", 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1049000000000005e-05 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586325+0000", 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7629999999999995e-06 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586334+0000", 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00043614599999999999 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586770+0000", 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3896e-05 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.586794+0000", 2024-02-13T00:54:58.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057734300000000002 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s2 700/675 ECSubWrite(tid=788, reqid=osd.6.0:788, at_version=700'8, trim_to=0'0, roll_forward_to=697'6))", 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:07.568941+0000", 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 170.67666277000001, 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062311400000000002, 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.568941+0000", 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.568941+0000", 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999905 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.568931+0000", 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7778999999999999e-05 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.568969+0000", 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5094e-05 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.568984+0000", 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4279999999999996e-06 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.568992+0000", 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7242000000000004e-05 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.569059+0000", 2024-02-13T00:54:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1119e-05 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.569070+0000", 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046841800000000001 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.569539+0000", 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4779000000000001e-05 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.569564+0000", 2024-02-13T00:54:58.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062311400000000002 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s2 700/675 ECSubWrite(tid=789, reqid=osd.6.0:788, at_version=0'0, trim_to=0'0, roll_forward_to=700'8))", 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:07.574196+0000", 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 170.67140710699999, 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058833900000000001, 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574196+0000", 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574196+0000", 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574196+0000", 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.204e-05 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574208+0000", 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.275e-05 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574220+0000", 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7740000000000004e-06 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574228+0000", 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014377300000000001 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574372+0000", 2024-02-13T00:54:58.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1268e-05 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574383+0000", 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00037631200000000001 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574759+0000", 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5117e-05 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:07.574785+0000", 2024-02-13T00:54:58.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058833900000000001 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s2 716/675 ECSubWrite(tid=792, reqid=osd.6.45:144, at_version=716'9, trim_to=0'0, roll_forward_to=700'8))", 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:23.590917+0000", 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 154.654685976, 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071739589999999997, 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.590917+0000", 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.590917+0000", 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.590917+0000", 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012856810000000001 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.592202+0000", 2024-02-13T00:54:58.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6427e-05 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.592219+0000", 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1756000000000001e-05 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.592231+0000", 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121882 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.592352+0000", 2024-02-13T00:54:58.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2697000000000001e-05 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.592365+0000", 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056990820000000003 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.598064+0000", 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7070999999999999e-05 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.598091+0000", 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071739589999999997 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(4.b REQUEST e721)", 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:28.978959+0000", 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 149.26664411300001, 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0060215579999999998, 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.978959+0000", 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.978959+0000", 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.978959+0000", 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6009999999999997e-06 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.978963+0000", 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2576e-05 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.978976+0000", 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8340000000000001e-06 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.978983+0000", 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5028000000000003e-05 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.979028+0000", 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4430000000000002e-06 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.979033+0000", 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058818209999999998 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.984915+0000", 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4091999999999998e-05 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.984949+0000", 2024-02-13T00:54:58.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1514999999999999e-05 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:28.984981+0000", 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0060215579999999998 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.1a 816/815 [PushOp(4:5909e671:gc::gc.27:head, version: 64'9, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:5909e671:gc::gc.27:head@64'9, size: 110, copy_subset: [0~110], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:110, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:56.638671+0000", 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.606932069, 2024-02-13T00:54:58.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00068126000000000005, 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:56.638671+0000", 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:56.638671+0000", 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:54:58.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:56.638671+0000", 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.454e-05 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:56.638685+0000", 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3576e-05 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:56.638699+0000", 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1168e-05 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:56.638710+0000", 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042383699999999999 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:56.639134+0000", 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1400000000000006e-06 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:56.639143+0000", 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020933799999999999 2024-02-13T00:54:58.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:54:58.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:54:58.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:54:58.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:56.639353+0000", 2024-02-13T00:54:58.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00068126000000000005 2024-02-13T00:54:58.333 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.333 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.333 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.333 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:54:58.333 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:54:58.333 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:54:58.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.670+0000 7f3acdd4c700 1 -- 172.21.15.162:0/36121067 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ac80fcb30 msgr2=0x7f3ac80fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:58.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.670+0000 7f3acdd4c700 1 --2- 172.21.15.162:0/36121067 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ac80fcb30 0x7f3ac80fcf10 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f3ab8004870 tx=0x7f3ab8009cb0).stop 2024-02-13T00:54:58.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.670+0000 7f3acdd4c700 1 -- 172.21.15.162:0/36121067 shutdown_connections 2024-02-13T00:54:58.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.670+0000 7f3acdd4c700 1 --2- 172.21.15.162:0/36121067 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ac80f73a0 0x7f3ac81039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:58.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.670+0000 7f3acdd4c700 1 --2- 172.21.15.162:0/36121067 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ac80fcb30 0x7f3ac80fcf10 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:58.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.670+0000 7f3acdd4c700 1 --2- 172.21.15.162:0/36121067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ac80fd490 0x7f3ac80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:58.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.670+0000 7f3acdd4c700 1 -- 172.21.15.162:0/36121067 >> 172.21.15.162:0/36121067 conn(0x7f3ac806eb90 msgr2=0x7f3ac80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:58.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.670+0000 7f3acdd4c700 1 -- 172.21.15.162:0/36121067 shutdown_connections 2024-02-13T00:54:58.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.670+0000 7f3acdd4c700 1 -- 172.21.15.162:0/36121067 wait complete. 2024-02-13T00:54:58.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.670+0000 7f3acdd4c700 1 librados: starting msgr at 2024-02-13T00:54:58.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.670+0000 7f3acdd4c700 1 librados: starting objecter 2024-02-13T00:54:58.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3acdd4c700 1 Processor -- start 2024-02-13T00:54:58.677 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3acdd4c700 1 -- start start 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3acdd4c700 1 librados: setting wanted keys 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3acdd4c700 1 librados: calling monclient init 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3acdd4c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ac80f73a0 0x7f3ac81092e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3acdd4c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ac80fcb30 0x7f3ac81097d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3acdd4c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ac80fd490 0x7f3ac8109db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3acdd4c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac81144c0 con 0x7f3ac80f73a0 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3acdd4c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac8114630 con 0x7f3ac80fd490 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3acdd4c700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac81147f0 con 0x7f3ac80fcb30 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3ac77fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ac80f73a0 0x7f3ac81092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3abeffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ac80fcb30 0x7f3ac81097d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3ac77fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ac80f73a0 0x7f3ac81092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:33506/0 (socket says 172.21.15.162:33506) 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.671+0000 7f3ac77fe700 1 -- 172.21.15.162:0/1959600648 learned_addr learned my addr 172.21.15.162:0/1959600648 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3ac7fff700 1 --2- 172.21.15.162:0/1959600648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ac80fd490 0x7f3ac8109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3ac77fe700 1 -- 172.21.15.162:0/1959600648 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ac80fcb30 msgr2=0x7f3ac81097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3ac77fe700 1 --2- 172.21.15.162:0/1959600648 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ac80fcb30 0x7f3ac81097d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3ac77fe700 1 -- 172.21.15.162:0/1959600648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ac80fd490 msgr2=0x7f3ac8109db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3ac77fe700 1 --2- 172.21.15.162:0/1959600648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ac80fd490 0x7f3ac8109db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3ac77fe700 1 -- 172.21.15.162:0/1959600648 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab0005040 con 0x7f3ac80f73a0 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3ac77fe700 1 --2- 172.21.15.162:0/1959600648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ac80f73a0 0x7f3ac81092e0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 rx=0x7f3ab80043a0 tx=0x7f3ab8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3ac57fa700 10 client.?.objecter ms_handle_connect 0x7f3ac80f73a0 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3ac57fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3ac57fa700 1 -- 172.21.15.162:0/1959600648 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab800e030 con 0x7f3ac80f73a0 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3ac57fa700 1 -- 172.21.15.162:0/1959600648 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ab8013da0 con 0x7f3ac80f73a0 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3acdd4c700 1 -- 172.21.15.162:0/1959600648 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab8005040 con 0x7f3ac80f73a0 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3ac57fa700 1 -- 172.21.15.162:0/1959600648 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8018070 con 0x7f3ac80f73a0 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3acdd4c700 10 client.28230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3acdd4c700 1 -- 172.21.15.162:0/1959600648 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac8107560 con 0x7f3ac80f73a0 2024-02-13T00:54:58.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.672+0000 7f3acdd4c700 1 librados: init done 2024-02-13T00:54:58.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.673+0000 7f3acdd4c700 1 -- 172.21.15.162:0/1959600648 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac80475e0 con 0x7f3ac80f73a0 2024-02-13T00:54:58.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.678+0000 7f3ac57fa700 1 -- 172.21.15.162:0/1959600648 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f3ab801d420 con 0x7f3ac80f73a0 2024-02-13T00:54:58.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.679+0000 7f3ac57fa700 1 --2- 172.21.15.162:0/1959600648 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ab4062c90 0x7f3ab4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:54:58.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.679+0000 7f3ac57fa700 1 -- 172.21.15.162:0/1959600648 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(817..817 src has 449..817) v4 ==== 7521+0+0 (secure 0 0 0) 0x7f3ab807e560 con 0x7f3ac80f73a0 2024-02-13T00:54:58.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.679+0000 7f3ac57fa700 10 client.28230.objecter ms_dispatch 0x7f3ac8001550 osd_map(817..817 src has 449..817) v4 2024-02-13T00:54:58.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.679+0000 7f3ac57fa700 3 client.28230.objecter handle_osd_map got epochs [817,817] > 0 2024-02-13T00:54:58.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.679+0000 7f3ac57fa700 3 client.28230.objecter handle_osd_map decoding full epoch 817 2024-02-13T00:54:58.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.679+0000 7f3ac57fa700 20 client.28230.objecter dump_active .. 0 homeless 2024-02-13T00:54:58.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.679+0000 7f3ac57fa700 1 -- 172.21.15.162:0/1959600648 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3ab807e950 con 0x7f3ac80f73a0 2024-02-13T00:54:58.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.679+0000 7f3abeffd700 1 --2- 172.21.15.162:0/1959600648 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ab4062c90 0x7f3ab4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:54:58.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.680+0000 7f3ac57fa700 10 client.28230.objecter ms_handle_connect 0x7f3ab4062c90 2024-02-13T00:54:58.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.680+0000 7f3abeffd700 1 --2- 172.21.15.162:0/1959600648 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ab4062c90 0x7f3ab4065150 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 rx=0x7f3ac80617b0 tx=0x7f3ab000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:54:58.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:58.864+0000 7f3acdd4c700 1 -- 172.21.15.162:0/1959600648 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd in", "ids": ["3"]} v 0) v1 -- 0x7f3ac8106030 con 0x7f3ac80f73a0 2024-02-13T00:54:59.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:57.603561+0000 osd.5 (osd.5) 15 : cluster [DBG] 5.e starting backfill to osd.4 from (0'0,0'0] MAX to 814'21 2024-02-13T00:54:59.008 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:54:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:54:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:54:59.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-13T00:54:59.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:57.603561+0000 osd.5 (osd.5) 15 : cluster [DBG] 5.e starting backfill to osd.4 from (0'0,0'0] MAX to 814'21 2024-02-13T00:54:59.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:57.603561+0000 osd.5 (osd.5) 15 : cluster [DBG] 5.e starting backfill to osd.4 from (0'0,0'0] MAX to 814'21 2024-02-13T00:54:59.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:54:58.864+0000 7f5f6486f700 -1 mon.a@0(leader).osd e817 definitely_dead 0 2024-02-13T00:54:59.316 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:54:59.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.750+0000 7f3ac57fa700 1 -- 172.21.15.162:0/1959600648 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["3"]}]=0 marked in osd.3. v818) v1 ==== 85+0+0 (secure 0 0 0) 0x7f3ab8051cf0 con 0x7f3ac80f73a0 2024-02-13T00:54:59.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.752+0000 7f3acdd4c700 10 librados: watch_flush enter 2024-02-13T00:54:59.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.753+0000 7f3acdd4c700 10 librados: watch_flush exit 2024-02-13T00:54:59.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.753+0000 7f3acdd4c700 20 client.28230.objecter shutdown clearing up homeless session... 2024-02-13T00:54:59.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.753+0000 7f3acdd4c700 10 client.28230.objecter successfully canceled tick 2024-02-13T00:54:59.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.753+0000 7f3acdd4c700 1 -- 172.21.15.162:0/1959600648 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ab4062c90 msgr2=0x7f3ab4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:59.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.753+0000 7f3acdd4c700 1 --2- 172.21.15.162:0/1959600648 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ab4062c90 0x7f3ab4065150 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 rx=0x7f3ac80617b0 tx=0x7f3ab000b040).stop 2024-02-13T00:54:59.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.753+0000 7f3acdd4c700 1 -- 172.21.15.162:0/1959600648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ac80f73a0 msgr2=0x7f3ac81092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:54:59.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.753+0000 7f3acdd4c700 1 --2- 172.21.15.162:0/1959600648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ac80f73a0 0x7f3ac81092e0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 rx=0x7f3ab80043a0 tx=0x7f3ab8009cb0).stop 2024-02-13T00:54:59.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.756+0000 7f3acdd4c700 1 -- 172.21.15.162:0/1959600648 shutdown_connections 2024-02-13T00:54:59.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.756+0000 7f3acdd4c700 1 --2- 172.21.15.162:0/1959600648 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3ab4062c90 0x7f3ab4065150 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:59.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.756+0000 7f3acdd4c700 1 --2- 172.21.15.162:0/1959600648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ac80fd490 0x7f3ac8109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:59.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.756+0000 7f3acdd4c700 1 --2- 172.21.15.162:0/1959600648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ac80f73a0 0x7f3ac81092e0 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:59.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.756+0000 7f3acdd4c700 1 --2- 172.21.15.162:0/1959600648 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ac80fcb30 0x7f3ac81097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:54:59.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.756+0000 7f3acdd4c700 1 -- 172.21.15.162:0/1959600648 >> 172.21.15.162:0/1959600648 conn(0x7f3ac806eb90 msgr2=0x7f3ac80fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:54:59.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.757+0000 7f3acdd4c700 1 -- 172.21.15.162:0/1959600648 shutdown_connections 2024-02-13T00:54:59.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.757+0000 7f3acdd4c700 1 -- 172.21.15.162:0/1959600648 wait complete. 2024-02-13T00:54:59.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:54:59.757+0000 7f3acdd4c700 1 librados: shutdown 2024-02-13T00:54:59.785 INFO:teuthology.orchestra.run.smithi162.stderr:marked in osd.3. 2024-02-13T00:54:59.807 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:58.564328+0000 mgr.y (mgr.14544) 691 : cluster [DBG] pgmap v1259: 164 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+recovering, 160 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s; 5/675 objects misplaced (0.741%); 663 B/s, 23 objects/s recovering 2024-02-13T00:54:59.807 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:58.866339+0000 mon.a (mon.0) 3783 : audit [DBG] from='client.? 172.21.15.162:0/1959600648' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-13T00:54:59.807 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:59.459097+0000 osd.6 (osd.6) 282 : cluster [DBG] 28.1b scrub starts 2024-02-13T00:54:59.807 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:54:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:59.462479+0000 osd.6 (osd.6) 283 : cluster [DBG] 28.1b scrub ok 2024-02-13T00:55:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:58.564328+0000 mgr.y (mgr.14544) 691 : cluster [DBG] pgmap v1259: 164 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+recovering, 160 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s; 5/675 objects misplaced (0.741%); 663 B/s, 23 objects/s recovering 2024-02-13T00:55:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:58.866339+0000 mon.a (mon.0) 3783 : audit [DBG] from='client.? 172.21.15.162:0/1959600648' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-13T00:55:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:59.459097+0000 osd.6 (osd.6) 282 : cluster [DBG] 28.1b scrub starts 2024-02-13T00:55:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:54:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:59.462479+0000 osd.6 (osd.6) 283 : cluster [DBG] 28.1b scrub ok 2024-02-13T00:55:00.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:58.564328+0000 mgr.y (mgr.14544) 691 : cluster [DBG] pgmap v1259: 164 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+recovering, 160 active+clean; 5.7 KiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s; 5/675 objects misplaced (0.741%); 663 B/s, 23 objects/s recovering 2024-02-13T00:55:00.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:58.866339+0000 mon.a (mon.0) 3783 : audit [DBG] from='client.? 172.21.15.162:0/1959600648' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-13T00:55:00.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:59.459097+0000 osd.6 (osd.6) 282 : cluster [DBG] 28.1b scrub starts 2024-02-13T00:55:00.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:54:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:59.462479+0000 osd.6 (osd.6) 283 : cluster [DBG] 28.1b scrub ok 2024-02-13T00:55:00.633 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-02-13T00:55:00.633 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-13T00:55:00.633 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-02-13T00:55:00.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd in 2 2024-02-13T00:55:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:54:59.751399+0000 mon.a (mon.0) 3784 : audit [INF] from='client.? 172.21.15.162:0/1959600648' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-13T00:55:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:54:59.751498+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2024-02-13T00:55:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:00.169994+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:55:00.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:00.172867+0000 osd.5 (osd.5) 17 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:55:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:00.225403+0000 osd.0 (osd.0) 342 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:55:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:00.228359+0000 osd.0 (osd.0) 343 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:55:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:00.503126+0000 osd.6 (osd.6) 284 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:55:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:00.506638+0000 osd.6 (osd.6) 285 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:55:00.796 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:00.600598+0000 mon.a (mon.0) 3786 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2024-02-13T00:55:00.960 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:01.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:54:59.751399+0000 mon.a (mon.0) 3784 : audit [INF] from='client.? 172.21.15.162:0/1959600648' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-13T00:55:01.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:54:59.751498+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2024-02-13T00:55:01.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:00.169994+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:55:01.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:00.172867+0000 osd.5 (osd.5) 17 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:55:01.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:00.225403+0000 osd.0 (osd.0) 342 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:55:01.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:00.228359+0000 osd.0 (osd.0) 343 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:55:01.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:00.503126+0000 osd.6 (osd.6) 284 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:55:01.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:00.506638+0000 osd.6 (osd.6) 285 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:55:01.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:00.600598+0000 mon.a (mon.0) 3786 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2024-02-13T00:55:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:54:59.751399+0000 mon.a (mon.0) 3784 : audit [INF] from='client.? 172.21.15.162:0/1959600648' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-13T00:55:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:54:59.751498+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2024-02-13T00:55:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:00.169994+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:55:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:00.172867+0000 osd.5 (osd.5) 17 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:55:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:00.225403+0000 osd.0 (osd.0) 342 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:55:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:00.228359+0000 osd.0 (osd.0) 343 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:55:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:00.503126+0000 osd.6 (osd.6) 284 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:55:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:00.506638+0000 osd.6 (osd.6) 285 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:55:01.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:00.600598+0000 mon.a (mon.0) 3786 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2024-02-13T00:55:01.117 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:55:01.117 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:55:01.117 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:55:01.117 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:55:01.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-13T00:55:02.016 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:00.565032+0000 mgr.y (mgr.14544) 692 : cluster [DBG] pgmap v1261: 164 pgs: 67 peering, 1 active+remapped+backfill_wait, 1 active+remapped, 95 active+clean; 4.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3/474 objects misplaced (0.633%) 2024-02-13T00:55:02.016 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:00.749125+0000 mon.a (mon.0) 3787 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-13T00:55:02.016 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:00.565032+0000 mgr.y (mgr.14544) 692 : cluster [DBG] pgmap v1261: 164 pgs: 67 peering, 1 active+remapped+backfill_wait, 1 active+remapped, 95 active+clean; 4.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3/474 objects misplaced (0.633%) 2024-02-13T00:55:02.017 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:00.749125+0000 mon.a (mon.0) 3787 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-13T00:55:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:00.565032+0000 mgr.y (mgr.14544) 692 : cluster [DBG] pgmap v1261: 164 pgs: 67 peering, 1 active+remapped+backfill_wait, 1 active+remapped, 95 active+clean; 4.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3/474 objects misplaced (0.633%) 2024-02-13T00:55:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:00.749125+0000 mon.a (mon.0) 3787 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-13T00:55:02.160 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:55:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.987+0000 7f07ff871700 1 -- 172.21.15.162:0/204456903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f80fd590 msgr2=0x7f07f80f75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.987+0000 7f07ff871700 1 --2- 172.21.15.162:0/204456903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f80fd590 0x7f07f80f75a0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 rx=0x7f07e8004870 tx=0x7f07e8009cb0).stop 2024-02-13T00:55:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.988+0000 7f07ff871700 1 -- 172.21.15.162:0/204456903 shutdown_connections 2024-02-13T00:55:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.988+0000 7f07ff871700 1 --2- 172.21.15.162:0/204456903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07f80f7c80 0x7f07f8103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.988+0000 7f07ff871700 1 --2- 172.21.15.162:0/204456903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07f80fcc30 0x7f07f80fd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.988+0000 7f07ff871700 1 --2- 172.21.15.162:0/204456903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f80fd590 0x7f07f80f75a0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.988+0000 7f07ff871700 1 -- 172.21.15.162:0/204456903 >> 172.21.15.162:0/204456903 conn(0x7f07f806c470 msgr2=0x7f07f806c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.988+0000 7f07ff871700 1 -- 172.21.15.162:0/204456903 shutdown_connections 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.988+0000 7f07ff871700 1 -- 172.21.15.162:0/204456903 wait complete. 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.988+0000 7f07ff871700 1 librados: starting msgr at 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.988+0000 7f07ff871700 1 librados: starting objecter 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.988+0000 7f07ff871700 1 Processor -- start 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07ff871700 1 -- start start 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07ff871700 1 librados: setting wanted keys 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07ff871700 1 librados: calling monclient init 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07ff871700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07f80f7c80 0x7f07f8063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07ff871700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07f80fcc30 0x7f07f8063900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07ff871700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f80fd590 0x7f07f811c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07ff871700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f8064100 con 0x7f07f80fd590 2024-02-13T00:55:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07ff871700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f811ce50 con 0x7f07f80f7c80 2024-02-13T00:55:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07ff871700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f8104040 con 0x7f07f80fcc30 2024-02-13T00:55:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07fd60d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07f80f7c80 0x7f07f8063410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07fd60d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07f80f7c80 0x7f07f8063410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:53718/0 (socket says 172.21.15.162:53718) 2024-02-13T00:55:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07fd60d700 1 -- 172.21.15.162:0/667241080 learned_addr learned my addr 172.21.15.162:0/667241080 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07fce0c700 1 --2- 172.21.15.162:0/667241080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07f80fcc30 0x7f07f8063900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07fde0e700 1 --2- 172.21.15.162:0/667241080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f80fd590 0x7f07f811c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07fde0e700 1 -- 172.21.15.162:0/667241080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07f80fcc30 msgr2=0x7f07f8063900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07fde0e700 1 --2- 172.21.15.162:0/667241080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07f80fcc30 0x7f07f8063900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07fde0e700 1 -- 172.21.15.162:0/667241080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07f80f7c80 msgr2=0x7f07f8063410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07fde0e700 1 --2- 172.21.15.162:0/667241080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07f80f7c80 0x7f07f8063410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07fde0e700 1 -- 172.21.15.162:0/667241080 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e8005040 con 0x7f07f80fd590 2024-02-13T00:55:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07fce0c700 1 --2- 172.21.15.162:0/667241080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07f80fcc30 0x7f07f8063900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.989+0000 7f07fd60d700 1 --2- 172.21.15.162:0/667241080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07f80f7c80 0x7f07f8063410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.990+0000 7f07fde0e700 1 --2- 172.21.15.162:0/667241080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f80fd590 0x7f07f811c930 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7f07f000d8d0 tx=0x7f07f0004db0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.990+0000 7f07ee7fc700 10 client.?.objecter ms_handle_connect 0x7f07f80fd590 2024-02-13T00:55:02.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.990+0000 7f07ee7fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.990+0000 7f07ee7fc700 1 -- 172.21.15.162:0/667241080 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f0004020 con 0x7f07f80fd590 2024-02-13T00:55:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.990+0000 7f07ff871700 1 -- 172.21.15.162:0/667241080 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f8104320 con 0x7f07f80fd590 2024-02-13T00:55:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.990+0000 7f07ff871700 10 client.28236.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.990+0000 7f07ff871700 1 -- 172.21.15.162:0/667241080 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f8104830 con 0x7f07f80fd590 2024-02-13T00:55:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.990+0000 7f07ff871700 1 librados: init done 2024-02-13T00:55:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.990+0000 7f07ee7fc700 1 -- 172.21.15.162:0/667241080 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07f000f040 con 0x7f07f80fd590 2024-02-13T00:55:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.990+0000 7f07ee7fc700 1 -- 172.21.15.162:0/667241080 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f0003430 con 0x7f07f80fd590 2024-02-13T00:55:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.991+0000 7f07ff871700 1 -- 172.21.15.162:0/667241080 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07dc004f50 con 0x7f07f80fd590 2024-02-13T00:55:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.992+0000 7f07ee7fc700 1 -- 172.21.15.162:0/667241080 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f07f0004180 con 0x7f07f80fd590 2024-02-13T00:55:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.992+0000 7f07ee7fc700 1 --2- 172.21.15.162:0/667241080 >> v2:172.21.15.162:6800/2866534053 conn(0x7f07e4062c90 0x7f07e4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.992+0000 7f07ee7fc700 1 -- 172.21.15.162:0/667241080 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(819..819 src has 449..819) v4 ==== 7426+0+0 (secure 0 0 0) 0x7f07f007c810 con 0x7f07f80fd590 2024-02-13T00:55:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.992+0000 7f07ee7fc700 10 client.28236.objecter ms_dispatch 0x7f07f8001550 osd_map(819..819 src has 449..819) v4 2024-02-13T00:55:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.992+0000 7f07ee7fc700 3 client.28236.objecter handle_osd_map got epochs [819,819] > 0 2024-02-13T00:55:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.992+0000 7f07ee7fc700 3 client.28236.objecter handle_osd_map decoding full epoch 819 2024-02-13T00:55:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.992+0000 7f07fd60d700 1 --2- 172.21.15.162:0/667241080 >> v2:172.21.15.162:6800/2866534053 conn(0x7f07e4062c90 0x7f07e4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.993+0000 7f07ee7fc700 20 client.28236.objecter dump_active .. 0 homeless 2024-02-13T00:55:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.993+0000 7f07ee7fc700 10 client.28236.objecter ms_handle_connect 0x7f07e4062c90 2024-02-13T00:55:02.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.993+0000 7f07fd60d700 1 --2- 172.21.15.162:0/667241080 >> v2:172.21.15.162:6800/2866534053 conn(0x7f07e4062c90 0x7f07e4065150 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 rx=0x7f07f4004fd0 tx=0x7f07f400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:03.002 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:02.997+0000 7f07ee7fc700 1 -- 172.21.15.162:0/667241080 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f07f0048f60 con 0x7f07f80fd590 2024-02-13T00:55:03.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.180+0000 7f07ff871700 1 -- 172.21.15.162:0/667241080 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd in", "ids": ["2"]} v 0) v1 -- 0x7f07dc005780 con 0x7f07f80fd590 2024-02-13T00:55:03.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:55:03.180+0000 7f5f6486f700 -1 mon.a@0(leader).osd e819 definitely_dead 0 2024-02-13T00:55:03.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.767+0000 7f07ee7fc700 1 -- 172.21.15.162:0/667241080 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["2"]}]=0 marked in osd.2. v820) v1 ==== 85+0+0 (secure 0 0 0) 0x7f07f004ff90 con 0x7f07f80fd590 2024-02-13T00:55:03.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.770+0000 7f07ff871700 10 librados: watch_flush enter 2024-02-13T00:55:03.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.771+0000 7f07ff871700 10 librados: watch_flush exit 2024-02-13T00:55:03.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.771+0000 7f07ff871700 20 client.28236.objecter shutdown clearing up homeless session... 2024-02-13T00:55:03.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.771+0000 7f07ff871700 10 client.28236.objecter successfully canceled tick 2024-02-13T00:55:03.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.772+0000 7f07ff871700 1 -- 172.21.15.162:0/667241080 >> v2:172.21.15.162:6800/2866534053 conn(0x7f07e4062c90 msgr2=0x7f07e4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:03.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.772+0000 7f07ff871700 1 --2- 172.21.15.162:0/667241080 >> v2:172.21.15.162:6800/2866534053 conn(0x7f07e4062c90 0x7f07e4065150 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 rx=0x7f07f4004fd0 tx=0x7f07f400b040).stop 2024-02-13T00:55:03.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.772+0000 7f07ff871700 1 -- 172.21.15.162:0/667241080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f80fd590 msgr2=0x7f07f811c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:03.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.772+0000 7f07ff871700 1 --2- 172.21.15.162:0/667241080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f80fd590 0x7f07f811c930 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7f07f000d8d0 tx=0x7f07f0004db0).stop 2024-02-13T00:55:03.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.775+0000 7f07ff871700 1 -- 172.21.15.162:0/667241080 shutdown_connections 2024-02-13T00:55:03.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.775+0000 7f07ff871700 1 --2- 172.21.15.162:0/667241080 >> v2:172.21.15.162:6800/2866534053 conn(0x7f07e4062c90 0x7f07e4065150 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:03.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.775+0000 7f07ff871700 1 --2- 172.21.15.162:0/667241080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f80fd590 0x7f07f811c930 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:03.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.775+0000 7f07ff871700 1 --2- 172.21.15.162:0/667241080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07f80f7c80 0x7f07f8063410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:03.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.775+0000 7f07ff871700 1 --2- 172.21.15.162:0/667241080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07f80fcc30 0x7f07f8063900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:03.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.775+0000 7f07ff871700 1 -- 172.21.15.162:0/667241080 >> 172.21.15.162:0/667241080 conn(0x7f07f806c470 msgr2=0x7f07f80fa8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:03.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.783+0000 7f07ff871700 1 -- 172.21.15.162:0/667241080 shutdown_connections 2024-02-13T00:55:03.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.785+0000 7f07ff871700 1 -- 172.21.15.162:0/667241080 wait complete. 2024-02-13T00:55:03.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:03.786+0000 7f07ff871700 1 librados: shutdown 2024-02-13T00:55:03.794 INFO:teuthology.orchestra.run.smithi162.stderr:marked in osd.2. 2024-02-13T00:55:04.039 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:55:04.040 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:55:04.040 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-02-13T00:55:04.040 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-02-13T00:55:04.040 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:55:04.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:02.565449+0000 mgr.y (mgr.14544) 693 : cluster [DBG] pgmap v1263: 164 pgs: 67 peering, 1 active+remapped+backfill_wait, 1 active+remapped, 95 active+clean; 4.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3/474 objects misplaced (0.633%) 2024-02-13T00:55:04.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:03.182233+0000 mon.a (mon.0) 3788 : audit [DBG] from='client.? 172.21.15.162:0/667241080' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-13T00:55:04.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:02.565449+0000 mgr.y (mgr.14544) 693 : cluster [DBG] pgmap v1263: 164 pgs: 67 peering, 1 active+remapped+backfill_wait, 1 active+remapped, 95 active+clean; 4.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3/474 objects misplaced (0.633%) 2024-02-13T00:55:04.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:03.182233+0000 mon.a (mon.0) 3788 : audit [DBG] from='client.? 172.21.15.162:0/667241080' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-13T00:55:04.073 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:02.565449+0000 mgr.y (mgr.14544) 693 : cluster [DBG] pgmap v1263: 164 pgs: 67 peering, 1 active+remapped+backfill_wait, 1 active+remapped, 95 active+clean; 4.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3/474 objects misplaced (0.633%) 2024-02-13T00:55:04.073 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:03.182233+0000 mon.a (mon.0) 3788 : audit [DBG] from='client.? 172.21.15.162:0/667241080' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-13T00:55:04.499 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-02-13T00:55:04.499 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.control 2024-02-13T00:55:04.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd pool set default.rgw.control pgp_num 22 2024-02-13T00:55:04.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-13T00:55:04.796 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:04.960 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:55:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:03.764759+0000 mon.a (mon.0) 3789 : audit [INF] from='client.? 172.21.15.162:0/667241080' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-13T00:55:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:03.764829+0000 mon.a (mon.0) 3790 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-02-13T00:55:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:04.765565+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-02-13T00:55:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:03.764759+0000 mon.a (mon.0) 3789 : audit [INF] from='client.? 172.21.15.162:0/667241080' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-13T00:55:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:03.764829+0000 mon.a (mon.0) 3790 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-02-13T00:55:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:04.765565+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-02-13T00:55:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:03.764759+0000 mon.a (mon.0) 3789 : audit [INF] from='client.? 172.21.15.162:0/667241080' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-13T00:55:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:03.764829+0000 mon.a (mon.0) 3790 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-02-13T00:55:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:04.765565+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-02-13T00:55:06.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:04.565905+0000 mgr.y (mgr.14544) 694 : cluster [DBG] pgmap v1265: 164 pgs: 67 peering, 1 active+remapped, 96 active+clean; 4.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2/474 objects misplaced (0.422%) 2024-02-13T00:55:06.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:05.470106+0000 mon.a (mon.0) 3792 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:06.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:05.767600+0000 mon.a (mon.0) 3793 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-02-13T00:55:06.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:04.565905+0000 mgr.y (mgr.14544) 694 : cluster [DBG] pgmap v1265: 164 pgs: 67 peering, 1 active+remapped, 96 active+clean; 4.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2/474 objects misplaced (0.422%) 2024-02-13T00:55:06.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:05.470106+0000 mon.a (mon.0) 3792 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:06.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:05.767600+0000 mon.a (mon.0) 3793 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-02-13T00:55:06.263 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:55:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:55:06.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:04.565905+0000 mgr.y (mgr.14544) 694 : cluster [DBG] pgmap v1265: 164 pgs: 67 peering, 1 active+remapped, 96 active+clean; 4.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2/474 objects misplaced (0.422%) 2024-02-13T00:55:06.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:05.470106+0000 mon.a (mon.0) 3792 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:06.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:05.767600+0000 mon.a (mon.0) 3793 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-02-13T00:55:06.421 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:55:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:55:06] "GET /metrics HTTP/1.1" 200 185508 "" "Prometheus/2.33.4" 2024-02-13T00:55:06.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.679+0000 7fc94732a700 1 -- 172.21.15.162:0/1979104124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9400fd490 msgr2=0x7fc9400f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:06.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.679+0000 7fc94732a700 1 --2- 172.21.15.162:0/1979104124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9400fd490 0x7fc9400f6cc0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 rx=0x7fc930004870 tx=0x7fc930009cb0).stop 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.679+0000 7fc94732a700 1 -- 172.21.15.162:0/1979104124 shutdown_connections 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.679+0000 7fc94732a700 1 --2- 172.21.15.162:0/1979104124 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9400f73a0 0x7fc9401039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.679+0000 7fc94732a700 1 --2- 172.21.15.162:0/1979104124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9400fcb30 0x7fc9400fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.679+0000 7fc94732a700 1 --2- 172.21.15.162:0/1979104124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9400fd490 0x7fc9400f6cc0 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.679+0000 7fc94732a700 1 -- 172.21.15.162:0/1979104124 >> 172.21.15.162:0/1979104124 conn(0x7fc94006eb90 msgr2=0x7fc9400f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.679+0000 7fc94732a700 1 -- 172.21.15.162:0/1979104124 shutdown_connections 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.679+0000 7fc94732a700 1 -- 172.21.15.162:0/1979104124 wait complete. 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.680+0000 7fc94732a700 1 librados: starting msgr at 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.680+0000 7fc94732a700 1 librados: starting objecter 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.680+0000 7fc94732a700 1 Processor -- start 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.680+0000 7fc94732a700 1 -- start start 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.680+0000 7fc94732a700 1 librados: setting wanted keys 2024-02-13T00:55:06.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.680+0000 7fc94732a700 1 librados: calling monclient init 2024-02-13T00:55:06.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.680+0000 7fc94732a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9400f73a0 0x7fc94010e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:06.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc94732a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9400fcb30 0x7fc94010e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:06.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc94732a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9400fd490 0x7fc940108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:06.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc94732a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc940108e00 con 0x7fc9400fd490 2024-02-13T00:55:06.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc94732a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc940108f70 con 0x7fc9400fcb30 2024-02-13T00:55:06.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc94732a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc940109130 con 0x7fc9400f73a0 2024-02-13T00:55:06.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9450c6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9400f73a0 0x7fc94010e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:06.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9458c7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9400fd490 0x7fc940108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:06.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9450c6700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9400f73a0 0x7fc94010e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:59302/0 (socket says 172.21.15.162:59302) 2024-02-13T00:55:06.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9458c7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9400fd490 0x7fc940108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:37048/0 (socket says 172.21.15.162:37048) 2024-02-13T00:55:06.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9450c6700 1 -- 172.21.15.162:0/3782423831 learned_addr learned my addr 172.21.15.162:0/3782423831 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9448c5700 1 --2- 172.21.15.162:0/3782423831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9400fcb30 0x7fc94010e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9450c6700 1 -- 172.21.15.162:0/3782423831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9400fcb30 msgr2=0x7fc94010e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9450c6700 1 --2- 172.21.15.162:0/3782423831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9400fcb30 0x7fc94010e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9450c6700 1 -- 172.21.15.162:0/3782423831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9400fd490 msgr2=0x7fc940108910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9450c6700 1 --2- 172.21.15.162:0/3782423831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9400fd490 0x7fc940108910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9450c6700 1 -- 172.21.15.162:0/3782423831 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9380097c0 con 0x7fc9400f73a0 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9458c7700 1 --2- 172.21.15.162:0/3782423831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9400fd490 0x7fc940108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9450c6700 1 --2- 172.21.15.162:0/3782423831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9400f73a0 0x7fc94010e160 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7fc93c00e970 tx=0x7fc93c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9367fc700 10 client.?.objecter ms_handle_connect 0x7fc9400f73a0 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9367fc700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9367fc700 1 -- 172.21.15.162:0/3782423831 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c004660 con 0x7fc9400f73a0 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9367fc700 1 -- 172.21.15.162:0/3782423831 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc93c00f040 con 0x7fc9400f73a0 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc94732a700 1 -- 172.21.15.162:0/3782423831 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc930005040 con 0x7fc9400f73a0 2024-02-13T00:55:06.689 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc9367fc700 1 -- 172.21.15.162:0/3782423831 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c014790 con 0x7fc9400f73a0 2024-02-13T00:55:06.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc94732a700 10 client.38336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:06.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc94732a700 1 -- 172.21.15.162:0/3782423831 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc940114f90 con 0x7fc9400f73a0 2024-02-13T00:55:06.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.681+0000 7fc94732a700 1 librados: init done 2024-02-13T00:55:06.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.682+0000 7fc94732a700 1 -- 172.21.15.162:0/3782423831 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc940106020 con 0x7fc9400f73a0 2024-02-13T00:55:06.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.683+0000 7fc9367fc700 1 -- 172.21.15.162:0/3782423831 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fc93c0090d0 con 0x7fc9400f73a0 2024-02-13T00:55:06.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.683+0000 7fc9367fc700 1 --2- 172.21.15.162:0/3782423831 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc92c062c90 0x7fc92c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:06.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.683+0000 7fc9367fc700 1 -- 172.21.15.162:0/3782423831 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(822..822 src has 449..822) v4 ==== 7459+0+0 (secure 0 0 0) 0x7fc93c07cac0 con 0x7fc9400f73a0 2024-02-13T00:55:06.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.683+0000 7fc9367fc700 10 client.38336.objecter ms_dispatch 0x7fc940001550 osd_map(822..822 src has 449..822) v4 2024-02-13T00:55:06.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.683+0000 7fc9367fc700 3 client.38336.objecter handle_osd_map got epochs [822,822] > 0 2024-02-13T00:55:06.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.683+0000 7fc9367fc700 3 client.38336.objecter handle_osd_map decoding full epoch 822 2024-02-13T00:55:06.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.683+0000 7fc9448c5700 1 --2- 172.21.15.162:0/3782423831 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc92c062c90 0x7fc92c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:06.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.683+0000 7fc9367fc700 20 client.38336.objecter dump_active .. 0 homeless 2024-02-13T00:55:06.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.683+0000 7fc9367fc700 10 client.38336.objecter ms_handle_connect 0x7fc92c062c90 2024-02-13T00:55:06.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.683+0000 7fc9448c5700 1 --2- 172.21.15.162:0/3782423831 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc92c062c90 0x7fc92c065150 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 rx=0x7fc930004ee0 tx=0x7fc93000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:06.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.687+0000 7fc9367fc700 1 -- 172.21.15.162:0/3782423831 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc93c04af60 con 0x7fc9400f73a0 2024-02-13T00:55:06.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:06.855+0000 7fc94732a700 1 -- 172.21.15.162:0/3782423831 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"} v 0) v1 -- 0x7fc94005fb80 con 0x7fc9400f73a0 2024-02-13T00:55:06.871 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:55:06.872 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:55:06.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:55:06.872 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:55:06.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.872 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.27243.0:1 206.7s0 206:fbfde494:::oid_MultiWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-02-13T00:55:06.872 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:45:40.407348+0000", 2024-02-13T00:55:06.872 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 566.45983588599995, 2024-02-13T00:55:06.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.007028646, 2024-02-13T00:55:06.872 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.872 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:55:06.872 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:06.872 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.27243", 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/879748064", 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407348+0000", 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407348+0000", 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:55:06.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407346+0000", 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.246e-06 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407352+0000", 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7770000000000008e-06 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407361+0000", 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.96e-06 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407369+0000", 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3953000000000003e-05 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407423+0000", 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7290000000000002e-06 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-02-13T00:55:06.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:40.407427+0000", 2024-02-13T00:55:06.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0034671980000001 2024-02-13T00:55:06.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:41.410894+0000", 2024-02-13T00:55:06.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015416900000000001 2024-02-13T00:55:06.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:41.411048+0000", 2024-02-13T00:55:06.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000348709 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:41.411397+0000", 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000485553 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:41.411882+0000", 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002441473 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:41.414324+0000", 2024-02-13T00:55:06.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2499e-05 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:45:41.414376+0000", 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.007028646 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25162.0:5962 61.1e 61:78ef1149:test-rados-api-smithi162-84057-58::foo-chunk:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected e403)", 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:46:02.459220+0000", 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 544.40796354500003, 2024-02-13T00:55:06.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.028868371, 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25162", 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/868737482", 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 5962 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459220+0000", 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459220+0000", 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459219+0000", 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6969999999999998e-06 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459224+0000", 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2309999999999999e-06 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459232+0000", 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9619999999999996e-06 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459239+0000", 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011526 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459354+0000", 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013904400000000001 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459493+0000", 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000156098 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.459649+0000", 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019810800000000001 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.461630+0000", 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058996099999999996 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.462220+0000", 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025775693999999998 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.487996+0000", 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2039000000000001e-05 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.488048+0000", 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0672e-05 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:02.488088+0000", 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.028868371 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.884 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.0 404/401 [PushOp(4:02683654:reshard::reshard.0000000010:head, version: 64'10, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:02683654:reshard::reshard.0000000010:head@64'10, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:46:03.967764+0000", 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 542.89941971999997, 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084933228, 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:03.967764+0000", 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:03.967764+0000", 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:03.967763+0000", 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0074999999999999e-05 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:03.967773+0000", 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2969999999999999e-05 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:03.967786+0000", 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4960000000000002e-06 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:03.967796+0000", 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084775084000000001 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.052571+0000", 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1409999999999998e-06 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.052575+0000", 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000122024 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.052697+0000", 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084933228 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.888 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.0 404/401 [PushOp(4:013d6c33:gc::gc.30:head, version: 64'14, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:013d6c33:gc::gc.30:head@64'14, size: 110, copy_subset: [0~110], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:110, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:46:04.053490+0000", 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 542.81369333299995, 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085623471000000007, 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.053490+0000", 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.053490+0000", 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.053490+0000", 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0045e-05 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.053500+0000", 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1745e-05 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.053511+0000", 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3350000000000003e-06 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.053518+0000", 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085375106000000006 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.138893+0000", 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.75e-06 2024-02-13T00:55:06.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.138899+0000", 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021510799999999999 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.139114+0000", 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085623471000000007 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.0 404/401 [PushOp(4:03bbb29a:::obj_delete_at_hint.0000000026:head, version: 64'16, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:03bbb29a:::obj_delete_at_hint.0000000026:head@64'16, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:46:04.140167+0000", 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 542.72701656599997, 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085170970999999998, 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.140167+0000", 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.140167+0000", 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.140166+0000", 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5435999999999999e-05 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.140182+0000", 2024-02-13T00:55:06.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3654e-05 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.140195+0000", 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.579e-06 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.140202+0000", 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085002941999999998 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.225205+0000", 2024-02-13T00:55:06.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0469999999999996e-06 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.225212+0000", 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012621999999999999 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.225338+0000", 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085170970999999998 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.896 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.0 404/401 [PushOp(4:0705c68b:::obj_delete_at_hint.0000000028:head, version: 64'18, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:0705c68b:::obj_delete_at_hint.0000000028:head@64'18, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:46:04.226305+0000", 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 542.64087888999995, 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085333043999999997, 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.226305+0000", 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.226305+0000", 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.226304+0000", 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3427999999999999e-05 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.226317+0000", 2024-02-13T00:55:06.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3625000000000001e-05 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.226331+0000", 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5089999999999999e-06 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.226337+0000", 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085176106000000001 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.311514+0000", 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.058e-06 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.311521+0000", 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117083 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.311638+0000", 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085333043999999997 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.900 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.0 404/401 [PushOp(4:0667bb13:gc::gc.13:head, version: 64'20, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:0667bb13:gc::gc.13:head@64'20, size: 110, copy_subset: [0~110], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:110, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:46:04.323918+0000", 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 542.54326576999995, 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074098921999999998, 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.323918+0000", 2024-02-13T00:55:06.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.323918+0000", 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.323917+0000", 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1127e-05 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.323928+0000", 2024-02-13T00:55:06.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3138000000000001e-05 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.323951+0000", 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6029999999999994e-06 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.323961+0000", 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.073866717999999998 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.397828+0000", 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5600000000000004e-06 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.397832+0000", 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018444499999999999 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:46:04.398017+0000", 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074098921999999998 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.904 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.27753.0:1794 247.b 247:d7155152:test-rados-api-smithi162-84057-127::foo:head [writefull 0~4000008] snapc 0=[] ondisk+write+known_if_redirected e716)", 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:23.554482+0000", 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 163.31270185899999, 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025332342000000001, 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.27753", 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1794 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.905 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.554482+0000", 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.554482+0000", 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999976 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.554479+0000", 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041669009999999998 2024-02-13T00:55:06.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.558646+0000", 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3184e-05 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.558670+0000", 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8910000000000001e-06 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.558678+0000", 2024-02-13T00:55:06.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7270999999999998e-05 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.558776+0000", 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0689999999999998e-06 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.558782+0000", 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003251641 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.562033+0000", 2024-02-13T00:55:06.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024787700000000003 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.562281+0000", 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014715430000000001 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.563753+0000", 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3579000000000003e-05 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.563806+0000", 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010464899999999999 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1", 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.563911+0000", 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0055428379999999996 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.569454+0000", 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035170010000000001 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:55:06.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.572971+0000", 2024-02-13T00:55:06.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0067756439999999999 2024-02-13T00:55:06.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-02-13T00:55:06.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.579747+0000", 2024-02-13T00:55:06.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8028000000000003e-05 2024-02-13T00:55:06.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.579785+0000", 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9485999999999999e-05 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.579814+0000", 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025332342000000001 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.912 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.27753.0:1795 247.b 247:d7155152:test-rados-api-smithi162-84057-127::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected e716)", 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:23.580066+0000", 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 163.287117728, 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019068933, 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.27753", 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/2331396316", 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1795 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.580066+0000", 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.580066+0000", 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.580064+0000", 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9689999999999997e-06 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.580071+0000", 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0336999999999999e-05 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.580081+0000", 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2220000000000001e-06 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.580087+0000", 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1928999999999999e-05 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.580119+0000", 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8492000000000007e-05 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.580197+0000", 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.018725636 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.598923+0000", 2024-02-13T00:55:06.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000132903 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.599056+0000", 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9175e-05 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.599135+0000", 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019068933 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPushReply(4.18 723/722 [PushReplyOp(4:1e85421f:gc::gc.1:head)])", 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:30.652050+0000", 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 156.215133786, 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.038765903999999997, 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.652050+0000", 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.652050+0000", 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.652049+0000", 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1609999999999997e-06 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.652056+0000", 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2323e-05 2024-02-13T00:55:06.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.652068+0000", 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1100000000000002e-06 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.652073+0000", 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.038613028000000001 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.690686+0000", 2024-02-13T00:55:06.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000129386 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.690816+0000", 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.038765903999999997 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPushReply(4.18 723/722 [PushReplyOp(4:1fdc4f8e:lc::lc.10:head)])", 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:30.863400+0000", 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 156.00378375299999, 2024-02-13T00:55:06.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.089697361000000003, 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.863400+0000", 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.863400+0000", 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.863399+0000", 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4029999999999998e-06 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.863404+0000", 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5019999999999998e-06 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.863414+0000", 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3780000000000003e-06 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.863418+0000", 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.089561534999999998 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.952980+0000", 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117269 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:30.953097+0000", 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.089697361000000003 2024-02-13T00:55:06.924 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:2444 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94514175197056 gen 8] snapc 0=[] ondisk+write+known_if_redirected e808)", 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:49.407472+0000", 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.459711244000001, 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00076260499999999997, 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:55:06.925 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2444 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407472+0000", 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407472+0000", 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999957 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407468+0000", 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9476000000000001e-05 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407487+0000", 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1272999999999997e-05 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407519+0000", 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3557000000000001e-05 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407532+0000", 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000452335 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407985+0000", 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000115431 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.408100+0000", 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000134833 2024-02-13T00:55:06.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.408235+0000", 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00076260499999999997 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:2445 5.f 5:f95f44c2:::notify.0:head [watch ping cookie 94514175188992 gen 5] snapc 0=[] ondisk+write+known_if_redirected e808)", 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:49.407552+0000", 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.459631138999999, 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067977900000000004, 2024-02-13T00:55:06.929 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2445 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407552+0000", 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407552+0000", 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407551+0000", 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6760000000000004e-06 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407557+0000", 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1370000000000008e-06 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407566+0000", 2024-02-13T00:55:06.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6200000000000001e-06 2024-02-13T00:55:06.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407573+0000", 2024-02-13T00:55:06.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00041183799999999998 2024-02-13T00:55:06.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.407985+0000", 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000118825 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.408103+0000", 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012871599999999999 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:49.408232+0000", 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067977900000000004 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.0.0:1980 4.16 e812/675 4:6a988a9b:reshard::reshard.0000000007:head v 812'29, mlcod=0'0)", 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:52.299528+0000", 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 14.567655974999999, 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000607399, 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.299528+0000", 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.299528+0000", 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.299527+0000", 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.454e-06 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.299532+0000", 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0599999999999996e-06 2024-02-13T00:55:06.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.299538+0000", 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.777e-06 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.299541+0000", 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015542799999999999 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.299697+0000", 2024-02-13T00:55:06.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9701e-05 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.299717+0000", 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039566699999999999 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.300112+0000", 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2784000000000001e-05 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.300135+0000", 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000607399 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.f epoch 814/813 rep_tid 1987 entries 814'20 (0'0) error 5:f95f44c2:::notify.0:head by client.24329.0:2460 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:54.409791+0000", 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 12.457392252, 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00073930999999999997, 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:06.938 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409791+0000", 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409791+0000", 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409791+0000", 2024-02-13T00:55:06.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4840000000000003e-06 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409796+0000", 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3219999999999993e-06 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409805+0000", 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1059999999999996e-06 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.940 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409811+0000", 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4605999999999997e-05 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409885+0000", 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00064514800000000001 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410531+0000", 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00073930999999999997 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.941 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.3 epoch 814/812 rep_tid 1991 entries 814'15 (0'0) error 5:c609908c:::notify.5:head by client.24329.0:2457 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:54.410096+0000", 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 12.457087178, 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062509299999999996, 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410096+0000", 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410096+0000", 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410096+0000", 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4389999999999999e-06 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410101+0000", 2024-02-13T00:55:06.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5380000000000002e-06 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410110+0000", 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1240000000000002e-06 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410116+0000", 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1199000000000006e-05 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410207+0000", 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00051421500000000005 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410721+0000", 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062509299999999996 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:2478 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94514175197056 gen 9] snapc 0=[] ondisk+write+known_if_redirected e817)", 2024-02-13T00:55:06.945 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:59.410297+0000", 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.4568868999999998, 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001844928, 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2478 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410297+0000", 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410297+0000", 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410296+0000", 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6969999999999998e-06 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410301+0000", 2024-02-13T00:55:06.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2459999999999999e-05 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410314+0000", 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.763e-06 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410322+0000", 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2839000000000001e-05 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410365+0000", 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7188999999999995e-05 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410452+0000", 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016898799999999999 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.412142+0000", 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001844928 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.949 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:2480 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94514175197056 gen 10] snapc 0=[] ondisk+write+known_if_redirected e817)", 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:59.410331+0000", 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.4568526400000001, 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019765410000000001, 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2480 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.950 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410331+0000", 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410331+0000", 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.951 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410330+0000", 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3529999999999999e-06 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410334+0000", 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4766999999999999e-05 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410349+0000", 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9050000000000002e-06 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410354+0000", 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00051143100000000002 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410866+0000", 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9648999999999996e-05 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410946+0000", 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013619649999999999 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.953 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.412307+0000", 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019765410000000001 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:2494 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94514175197056 gen 11] snapc 0=[] ondisk+write+known_if_redirected e820)", 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:55:04.411286+0000", 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.455898033, 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00092112699999999999, 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:06.954 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2494 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411286+0000", 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411286+0000", 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:06.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411285+0000", 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3630000000000002e-06 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411290+0000", 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0397999999999999e-05 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411300+0000", 2024-02-13T00:55:06.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0390000000000004e-06 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411305+0000", 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6748999999999997e-05 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411342+0000", 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4118e-05 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411386+0000", 2024-02-13T00:55:06.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00082040599999999998 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.412207+0000", 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00092112699999999999 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPull(4.d e821/820 cost 1049576)", 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:55:04.813829+0000", 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.0533546, 2024-02-13T00:55:06.958 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030802099999999999, 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.813829+0000", 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.813829+0000", 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.813829+0000", 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1533e-05 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.813840+0000", 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4389999999999999e-06 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.813846+0000", 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9510000000000003e-06 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.960 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.813852+0000", 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4605999999999995e-05 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.813946+0000", 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029629299999999999 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.816909+0000", 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030802099999999999 2024-02-13T00:55:06.961 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.962 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.962 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.962 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:06.962 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:06.962 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:55:06.962 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:05.770017+0000 osd.0 (osd.0) 344 : cluster [DBG] 5.d starting backfill to osd.2 from (0'0,0'0] MAX to 817'22 2024-02-13T00:55:06.962 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:05.899468+0000 osd.7 (osd.7) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:55:06.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:05.899705+0000 osd.7 (osd.7) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:55:06.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:06.024075+0000 mgr.y (mgr.14544) 695 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:07.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:05.770017+0000 osd.0 (osd.0) 344 : cluster [DBG] 5.d starting backfill to osd.2 from (0'0,0'0] MAX to 817'22 2024-02-13T00:55:07.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:05.899468+0000 osd.7 (osd.7) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:55:07.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:05.899705+0000 osd.7 (osd.7) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:55:07.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:06.024075+0000 mgr.y (mgr.14544) 695 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:05.770017+0000 osd.0 (osd.0) 344 : cluster [DBG] 5.d starting backfill to osd.2 from (0'0,0'0] MAX to 817'22 2024-02-13T00:55:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:05.899468+0000 osd.7 (osd.7) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:55:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:05.899705+0000 osd.7 (osd.7) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:55:07.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:06.024075+0000 mgr.y (mgr.14544) 695 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:07.547 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-13T00:55:07.813 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:55:07.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.816+0000 7fc9367fc700 1 -- 172.21.15.162:0/3782423831 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]=0 set pool 5 pgp_num to 22 v824) v1 ==== 146+0+0 (secure 0 0 0) 0x7fc93c07ce60 con 0x7fc9400f73a0 2024-02-13T00:55:07.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.818+0000 7fc94732a700 10 librados: watch_flush enter 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.818+0000 7fc94732a700 10 librados: watch_flush exit 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.818+0000 7fc94732a700 20 client.38336.objecter shutdown clearing up homeless session... 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.818+0000 7fc94732a700 10 client.38336.objecter successfully canceled tick 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 -- 172.21.15.162:0/3782423831 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc92c062c90 msgr2=0x7fc92c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 --2- 172.21.15.162:0/3782423831 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc92c062c90 0x7fc92c065150 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 rx=0x7fc930004ee0 tx=0x7fc93000b040).stop 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 -- 172.21.15.162:0/3782423831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9400f73a0 msgr2=0x7fc94010e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 --2- 172.21.15.162:0/3782423831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9400f73a0 0x7fc94010e160 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7fc93c00e970 tx=0x7fc93c00b040).stop 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 -- 172.21.15.162:0/3782423831 shutdown_connections 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 --2- 172.21.15.162:0/3782423831 >> v2:172.21.15.162:6800/2866534053 conn(0x7fc92c062c90 0x7fc92c065150 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 --2- 172.21.15.162:0/3782423831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9400fd490 0x7fc940108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 --2- 172.21.15.162:0/3782423831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9400f73a0 0x7fc94010e160 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 --2- 172.21.15.162:0/3782423831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9400fcb30 0x7fc94010e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:07.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 -- 172.21.15.162:0/3782423831 >> 172.21.15.162:0/3782423831 conn(0x7fc94006eb90 msgr2=0x7fc94006f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:07.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 -- 172.21.15.162:0/3782423831 shutdown_connections 2024-02-13T00:55:07.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 -- 172.21.15.162:0/3782423831 wait complete. 2024-02-13T00:55:07.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:07.819+0000 7fc94732a700 1 librados: shutdown 2024-02-13T00:55:07.825 INFO:teuthology.orchestra.run.smithi162.stderr:set pool 5 pgp_num to 22 2024-02-13T00:55:08.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:06.566293+0000 mgr.y (mgr.14544) 696 : cluster [DBG] pgmap v1268: 164 pgs: 1 remapped+peering, 3 active+recovering, 1 active+remapped, 159 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 68 KiB/s rd, 1.5 KiB/s wr, 130 op/s; 26/675 objects misplaced (3.852%); 1.2 KiB/s, 63 objects/s recovering 2024-02-13T00:55:08.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:06.809604+0000 mon.a (mon.0) 3794 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-02-13T00:55:08.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:06.857160+0000 mon.c (mon.2) 522 : audit [INF] from='client.? 172.21.15.162:0/3782423831' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-13T00:55:08.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:06.857474+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-13T00:55:08.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:07.462904+0000 osd.6 (osd.6) 286 : cluster [DBG] 4.e scrub starts 2024-02-13T00:55:08.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:07.466495+0000 osd.6 (osd.6) 287 : cluster [DBG] 4.e scrub ok 2024-02-13T00:55:08.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:06.566293+0000 mgr.y (mgr.14544) 696 : cluster [DBG] pgmap v1268: 164 pgs: 1 remapped+peering, 3 active+recovering, 1 active+remapped, 159 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 68 KiB/s rd, 1.5 KiB/s wr, 130 op/s; 26/675 objects misplaced (3.852%); 1.2 KiB/s, 63 objects/s recovering 2024-02-13T00:55:08.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:06.809604+0000 mon.a (mon.0) 3794 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-02-13T00:55:08.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:06.857160+0000 mon.c (mon.2) 522 : audit [INF] from='client.? 172.21.15.162:0/3782423831' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-13T00:55:08.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:06.857474+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-13T00:55:08.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:07.462904+0000 osd.6 (osd.6) 286 : cluster [DBG] 4.e scrub starts 2024-02-13T00:55:08.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:07.466495+0000 osd.6 (osd.6) 287 : cluster [DBG] 4.e scrub ok 2024-02-13T00:55:08.083 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:06.566293+0000 mgr.y (mgr.14544) 696 : cluster [DBG] pgmap v1268: 164 pgs: 1 remapped+peering, 3 active+recovering, 1 active+remapped, 159 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 68 KiB/s rd, 1.5 KiB/s wr, 130 op/s; 26/675 objects misplaced (3.852%); 1.2 KiB/s, 63 objects/s recovering 2024-02-13T00:55:08.083 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:06.809604+0000 mon.a (mon.0) 3794 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-02-13T00:55:08.083 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:06.857160+0000 mon.c (mon.2) 522 : audit [INF] from='client.? 172.21.15.162:0/3782423831' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-13T00:55:08.083 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:06.857474+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]: dispatch 2024-02-13T00:55:08.083 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:07.462904+0000 osd.6 (osd.6) 286 : cluster [DBG] 4.e scrub starts 2024-02-13T00:55:08.083 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:07.466495+0000 osd.6 (osd.6) 287 : cluster [DBG] 4.e scrub ok 2024-02-13T00:55:08.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-02-13T00:55:08.794 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:55:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:55:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:55:08.823 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:07.797549+0000 osd.2 (osd.2) 153 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:55:08.823 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:07.797549+0000 osd.2 (osd.2) 153 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:55:08.823 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:07.798452+0000 osd.2 (osd.2) 154 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:55:08.823 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:07.813565+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]': finished 2024-02-13T00:55:08.824 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:07.813619+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-02-13T00:55:08.824 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:08.092381+0000 osd.0 (osd.0) 345 : cluster [DBG] 6.0 deep-scrub starts 2024-02-13T00:55:08.824 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:08.094222+0000 osd.0 (osd.0) 346 : cluster [DBG] 6.0 deep-scrub ok 2024-02-13T00:55:08.824 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:08.512117+0000 osd.6 (osd.6) 288 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:55:08.824 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:08.513798+0000 osd.6 (osd.6) 289 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:55:08.876 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:07.798452+0000 osd.2 (osd.2) 154 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:55:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:07.813565+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]': finished 2024-02-13T00:55:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:07.813619+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-02-13T00:55:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:08.092381+0000 osd.0 (osd.0) 345 : cluster [DBG] 6.0 deep-scrub starts 2024-02-13T00:55:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:08.094222+0000 osd.0 (osd.0) 346 : cluster [DBG] 6.0 deep-scrub ok 2024-02-13T00:55:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:08.512117+0000 osd.6 (osd.6) 288 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:55:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:08.513798+0000 osd.6 (osd.6) 289 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:55:09.094 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:07.797549+0000 osd.2 (osd.2) 153 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:55:09.094 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:07.798452+0000 osd.2 (osd.2) 154 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:55:09.094 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:07.813565+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "22"}]': finished 2024-02-13T00:55:09.094 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:07.813619+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-02-13T00:55:09.094 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:08.092381+0000 osd.0 (osd.0) 345 : cluster [DBG] 6.0 deep-scrub starts 2024-02-13T00:55:09.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:08.094222+0000 osd.0 (osd.0) 346 : cluster [DBG] 6.0 deep-scrub ok 2024-02-13T00:55:09.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:08.512117+0000 osd.6 (osd.6) 288 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:55:09.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:08.513798+0000 osd.6 (osd.6) 289 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:55:09.617 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:55:09.618 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:55:09.618 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:55:09.618 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:55:09.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:08.566918+0000 mgr.y (mgr.14544) 697 : cluster [DBG] pgmap v1271: 164 pgs: 1 remapped+peering, 3 active+recovering, 1 active+remapped, 159 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 101 KiB/s rd, 2.2 KiB/s wr, 194 op/s; 26/675 objects misplaced (3.852%); 1.8 KiB/s, 93 objects/s recovering 2024-02-13T00:55:09.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:08.698137+0000 osd.3 (osd.3) 167 : cluster [DBG] 4.c deep-scrub starts 2024-02-13T00:55:09.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:08.701873+0000 osd.3 (osd.3) 168 : cluster [DBG] 4.c deep-scrub ok 2024-02-13T00:55:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:08.566918+0000 mgr.y (mgr.14544) 697 : cluster [DBG] pgmap v1271: 164 pgs: 1 remapped+peering, 3 active+recovering, 1 active+remapped, 159 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 101 KiB/s rd, 2.2 KiB/s wr, 194 op/s; 26/675 objects misplaced (3.852%); 1.8 KiB/s, 93 objects/s recovering 2024-02-13T00:55:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:08.698137+0000 osd.3 (osd.3) 167 : cluster [DBG] 4.c deep-scrub starts 2024-02-13T00:55:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:08.701873+0000 osd.3 (osd.3) 168 : cluster [DBG] 4.c deep-scrub ok 2024-02-13T00:55:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:08.566918+0000 mgr.y (mgr.14544) 697 : cluster [DBG] pgmap v1271: 164 pgs: 1 remapped+peering, 3 active+recovering, 1 active+remapped, 159 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 101 KiB/s rd, 2.2 KiB/s wr, 194 op/s; 26/675 objects misplaced (3.852%); 1.8 KiB/s, 93 objects/s recovering 2024-02-13T00:55:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:08.698137+0000 osd.3 (osd.3) 167 : cluster [DBG] 4.c deep-scrub starts 2024-02-13T00:55:10.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:08.701873+0000 osd.3 (osd.3) 168 : cluster [DBG] 4.c deep-scrub ok 2024-02-13T00:55:10.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-13T00:55:10.552 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:55:10.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 -- 172.21.15.162:0/704601260 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f00fd470 msgr2=0x7f87f00f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:10.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 --2- 172.21.15.162:0/704601260 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f00fd470 0x7f87f00f6ca0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 rx=0x7f87dc0047b0 tx=0x7f87dc009cb0).stop 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 -- 172.21.15.162:0/704601260 shutdown_connections 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 --2- 172.21.15.162:0/704601260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00f7380 0x7f87f0103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 --2- 172.21.15.162:0/704601260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f00fcb10 0x7f87f00fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 --2- 172.21.15.162:0/704601260 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f00fd470 0x7f87f00f6ca0 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 -- 172.21.15.162:0/704601260 >> 172.21.15.162:0/704601260 conn(0x7f87f00f2720 msgr2=0x7f87f00f4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 -- 172.21.15.162:0/704601260 shutdown_connections 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 -- 172.21.15.162:0/704601260 wait complete. 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 librados: starting msgr at 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 librados: starting objecter 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 Processor -- start 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 -- start start 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 librados: setting wanted keys 2024-02-13T00:55:10.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.680+0000 7f87f6b71700 1 librados: calling monclient init 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87f6b71700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f00f7380 0x7f87f0106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87f6b71700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fcb10 0x7f87f01074a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87f6b71700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f00fd470 0x7f87f0107990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87f6b71700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0116630 con 0x7f87f00fcb10 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87f6b71700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0116770 con 0x7f87f00f7380 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87f6b71700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f01168e0 con 0x7f87f00fd470 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87effff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fcb10 0x7f87f01074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87f510e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f00fd470 0x7f87f0107990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87effff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fcb10 0x7f87f01074a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:37060/0 (socket says 172.21.15.162:37060) 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87effff700 1 -- 172.21.15.162:0/603181949 learned_addr learned my addr 172.21.15.162:0/603181949 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87f490d700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f00f7380 0x7f87f0106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87effff700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f00fd470 msgr2=0x7f87f0107990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87effff700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f00fd470 0x7f87f0107990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:10.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87effff700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f00f7380 msgr2=0x7f87f0106fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87effff700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f00f7380 0x7f87f0106fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87effff700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e80097c0 con 0x7f87f00fcb10 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f00fd470 0x7f87f0107990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87effff700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fcb10 0x7f87f01074a0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7f87dc0044e0 tx=0x7f87dc007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87edffb700 10 client.?.objecter ms_handle_connect 0x7f87f00fcb10 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87edffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc018070 con 0x7f87f00fcb10 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87dc005040 con 0x7f87f00fcb10 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87dc0093b0 con 0x7f87f00fcb10 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.681+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc01dd90 con 0x7f87f00fcb10 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.682+0000 7f87f6b71700 10 client.28248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.682+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f0116dd0 con 0x7f87f00fcb10 2024-02-13T00:55:10.688 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.682+0000 7f87f6b71700 1 librados: init done 2024-02-13T00:55:10.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.683+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87f00607d0 con 0x7f87f00fcb10 2024-02-13T00:55:10.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.683+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f87dc01d420 con 0x7f87f00fcb10 2024-02-13T00:55:10.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.683+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> v2:172.21.15.162:6800/2866534053 conn(0x7f87e0062c90 0x7f87e0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:10.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.683+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(825..825 src has 449..825) v4 ==== 7426+0+0 (secure 0 0 0) 0x7f87dc07e4a0 con 0x7f87f00fcb10 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.683+0000 7f87edffb700 10 client.28248.objecter ms_dispatch 0x7f87f0001550 osd_map(825..825 src has 449..825) v4 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.683+0000 7f87edffb700 3 client.28248.objecter handle_osd_map got epochs [825,825] > 0 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.683+0000 7f87edffb700 3 client.28248.objecter handle_osd_map decoding full epoch 825 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87f490d700 1 --2- 172.21.15.162:0/603181949 >> v2:172.21.15.162:6800/2866534053 conn(0x7f87e0062c90 0x7f87e0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87edffb700 20 client.28248.objecter dump_active .. 0 homeless 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1267 (secure 0 0 0) 0x7f87dc085090 con 0x7f87f00fcb10 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87edffb700 10 client.28248.objecter ms_handle_connect 0x7f87e0062c90 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87f490d700 1 --2- 172.21.15.162:0/603181949 >> v2:172.21.15.162:6800/2866534053 conn(0x7f87e0062c90 0x7f87e0065150 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 rx=0x7f87e4004e60 tx=0x7f87e400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87f6b71700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fd870 0x7f87f00fdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0061ad0 con 0x7f87f00fd870 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fd870 0x7f87f00fdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87f510e700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87f00475e0 con 0x7f87f00fd870 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87edffb700 10 client.28248.objecter ms_handle_connect 0x7f87f00fd870 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87edffb700 10 client.28248.objecter resend_mon_ops 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fd870 0x7f87f00fdc50 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7f87e8011460 tx=0x7f87e8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:10.691 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.684+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e80079b0 con 0x7f87f00fd870 2024-02-13T00:55:10.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.685+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.0 v2:172.21.15.162:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+8167 (secure 0 0 0) 0x7f87e800f030 con 0x7f87f00fd870 2024-02-13T00:55:10.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.685+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fd870 msgr2=0x7f87f00fdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:10.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.685+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fd870 0x7f87f00fdc50 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7f87e8011460 tx=0x7f87e8004d10).stop 2024-02-13T00:55:10.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.685+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fd870 msgr2=0x7f87f00fdc50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-13T00:55:10.692 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.685+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fd870 0x7f87f00fdc50 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:10.695 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.691+0000 7f87f6b71700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fe140 0x7f87f011b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:10.695 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.691+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f00008d0 con 0x7f87f00fe140 2024-02-13T00:55:10.695 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.692+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fe140 0x7f87f011b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:10.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.692+0000 7f87f510e700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7f87f00fe520 con 0x7f87f00fe140 2024-02-13T00:55:10.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.692+0000 7f87edffb700 10 client.28248.objecter ms_handle_connect 0x7f87f00fe140 2024-02-13T00:55:10.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.692+0000 7f87edffb700 10 client.28248.objecter resend_mon_ops 2024-02-13T00:55:10.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.692+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fe140 0x7f87f011b010 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 rx=0x7f87e8011460 tx=0x7f87e801a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:10.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.692+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e8016de0 con 0x7f87f00fe140 2024-02-13T00:55:10.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.693+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.0 v2:172.21.15.162:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '600' (not observed, change may require restart) ) v1 ==== 85+0+3 (secure 0 0 0) 0x7f87e80077a0 con 0x7f87f00fe140 2024-02-13T00:55:10.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.693+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fe140 msgr2=0x7f87f011b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:10.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.693+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fe140 0x7f87f011b010 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 rx=0x7f87e8011460 tx=0x7f87e801a000).stop 2024-02-13T00:55:10.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.693+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fe140 msgr2=0x7f87f011b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-13T00:55:10.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.693+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fe140 0x7f87f011b010 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:10.697 INFO:teuthology.orchestra.run.smithi162.stdout:mon.a: {} 2024-02-13T00:55:10.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.693+0000 7f87f6b71700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0027410 0x7f87f0027820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:10.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.693+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f00fe870 con 0x7f87f0027410 2024-02-13T00:55:10.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.694+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0027410 0x7f87f0027820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:10.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.694+0000 7f87f510e700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87f0027d60 con 0x7f87f0027410 2024-02-13T00:55:10.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.694+0000 7f87edffb700 10 client.28248.objecter ms_handle_connect 0x7f87f0027410 2024-02-13T00:55:10.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.694+0000 7f87edffb700 10 client.28248.objecter resend_mon_ops 2024-02-13T00:55:10.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.694+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0027410 0x7f87f0027820 secure :-1 s=READY pgs=4862 cs=0 l=1 rev1=1 rx=0x7f87e8000f80 tx=0x7f87e8023000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:10.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.694+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e802f410 con 0x7f87f0027410 2024-02-13T00:55:10.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.695+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+8167 (secure 0 0 0) 0x7f87e8033070 con 0x7f87f0027410 2024-02-13T00:55:10.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.695+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0027410 msgr2=0x7f87f0027820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:10.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.695+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0027410 0x7f87f0027820 secure :-1 s=READY pgs=4862 cs=0 l=1 rev1=1 rx=0x7f87e8000f80 tx=0x7f87e8023000).stop 2024-02-13T00:55:10.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.695+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0027410 msgr2=0x7f87f0027820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-13T00:55:10.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.695+0000 7f87f510e700 1 -- 172.21.15.162:0/603181949 reap_dead start 2024-02-13T00:55:10.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.695+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0027410 0x7f87f0027820 unknown :-1 s=CLOSED pgs=4862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.703+0000 7f87f6b71700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0107990 0x7f87f0027f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.703+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0107100 con 0x7f87f0107990 2024-02-13T00:55:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.703+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0107990 0x7f87f0027f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.703+0000 7f87f510e700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7f87f0107240 con 0x7f87f0107990 2024-02-13T00:55:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.703+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0107990 0x7f87f0027f30 secure :-1 s=READY pgs=4863 cs=0 l=1 rev1=1 rx=0x7f87e8001400 tx=0x7f87e8031070).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.703+0000 7f87edffb700 10 client.28248.objecter ms_handle_connect 0x7f87f0107990 2024-02-13T00:55:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.703+0000 7f87edffb700 10 client.28248.objecter resend_mon_ops 2024-02-13T00:55:10.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.703+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e8030980 con 0x7f87f0107990 2024-02-13T00:55:10.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.1 v2:172.21.15.181:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '600' (not observed, change may require restart) ) v1 ==== 85+0+3 (secure 0 0 0) 0x7f87e8023d50 con 0x7f87f0107990 2024-02-13T00:55:10.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0107990 msgr2=0x7f87f0027f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:10.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0107990 0x7f87f0027f30 secure :-1 s=READY pgs=4863 cs=0 l=1 rev1=1 rx=0x7f87e8001400 tx=0x7f87e8031070).stop 2024-02-13T00:55:10.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0107990 msgr2=0x7f87f0027f30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-13T00:55:10.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87f0107990 0x7f87f0027f30 unknown :-1 s=CLOSED pgs=4863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:10.709 INFO:teuthology.orchestra.run.smithi162.stdout:mon.b: {} 2024-02-13T00:55:10.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87f6b71700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0028510 0x7f87f011d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:10.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0120160 con 0x7f87f0028510 2024-02-13T00:55:10.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0028510 0x7f87f011d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:10.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87f510e700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87f01202a0 con 0x7f87f0028510 2024-02-13T00:55:10.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87edffb700 10 client.28248.objecter ms_handle_connect 0x7f87f0028510 2024-02-13T00:55:10.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87edffb700 10 client.28248.objecter resend_mon_ops 2024-02-13T00:55:10.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0028510 0x7f87f011d510 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 rx=0x7f87e80013d0 tx=0x7f87e803e070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:10.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.705+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e8049070 con 0x7f87f0028510 2024-02-13T00:55:10.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.706+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.2 v2:172.21.15.162:3301/0 2 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+8167 (secure 0 0 0) 0x7f87e8030ae0 con 0x7f87f0028510 2024-02-13T00:55:10.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.706+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0028510 msgr2=0x7f87f011d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:10.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.706+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0028510 0x7f87f011d510 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 rx=0x7f87e80013d0 tx=0x7f87e803e070).stop 2024-02-13T00:55:10.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.706+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0028510 msgr2=0x7f87f011d510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-13T00:55:10.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.706+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0028510 0x7f87f011d510 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:10.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.712+0000 7f87f6b71700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0120560 0x7f87f011fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:10.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.712+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f011ffa0 con 0x7f87f0120560 2024-02-13T00:55:10.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.712+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0120560 0x7f87f011fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:10.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.712+0000 7f87f510e700 1 -- 172.21.15.162:0/603181949 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7f87f0121f20 con 0x7f87f0120560 2024-02-13T00:55:10.716 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.712+0000 7f87edffb700 10 client.28248.objecter ms_handle_connect 0x7f87f0120560 2024-02-13T00:55:10.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.712+0000 7f87edffb700 10 client.28248.objecter resend_mon_ops 2024-02-13T00:55:10.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.712+0000 7f87f510e700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0120560 0x7f87f011fa10 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 rx=0x7f87e8023fd0 tx=0x7f87e8049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:10.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.713+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e8044ab0 con 0x7f87f0120560 2024-02-13T00:55:10.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 <== mon.2 v2:172.21.15.162:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '600' (not observed, change may require restart) ) v1 ==== 85+0+3 (secure 0 0 0) 0x7f87e8044ab0 con 0x7f87f0120560 2024-02-13T00:55:10.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0120560 msgr2=0x7f87f011fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:10.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0120560 0x7f87f011fa10 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 rx=0x7f87e8023fd0 tx=0x7f87e8049000).stop 2024-02-13T00:55:10.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87edffb700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0120560 msgr2=0x7f87f011fa10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-13T00:55:10.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87edffb700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f0120560 0x7f87f011fa10 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:10.718 INFO:teuthology.orchestra.run.smithi162.stdout:mon.c: {} 2024-02-13T00:55:10.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 10 librados: watch_flush enter 2024-02-13T00:55:10.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 10 librados: watch_flush exit 2024-02-13T00:55:10.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 20 client.28248.objecter shutdown clearing up homeless session... 2024-02-13T00:55:10.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 10 client.28248.objecter successfully canceled tick 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 >> v2:172.21.15.162:6800/2866534053 conn(0x7f87e0062c90 msgr2=0x7f87e0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 1 --2- 172.21.15.162:0/603181949 >> v2:172.21.15.162:6800/2866534053 conn(0x7f87e0062c90 0x7f87e0065150 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 rx=0x7f87e4004e60 tx=0x7f87e400b040).stop 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fcb10 msgr2=0x7f87f01074a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 1 --2- 172.21.15.162:0/603181949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f00fcb10 0x7f87f01074a0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7f87dc0044e0 tx=0x7f87dc007f90).stop 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f510e700 1 -- 172.21.15.162:0/603181949 reap_dead start 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 shutdown_connections 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 >> 172.21.15.162:0/603181949 conn(0x7f87f00f2720 msgr2=0x7f87f00fa8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 shutdown_connections 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 1 -- 172.21.15.162:0/603181949 wait complete. 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:10.714+0000 7f87f6b71700 1 librados: shutdown 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-13T00:55:10.720 INFO:teuthology.orchestra.run.smithi162.stderr:mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-13T00:55:11.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-13T00:55:11.732 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:09.709233+0000 osd.3 (osd.3) 169 : cluster [DBG] 28.d scrub starts 2024-02-13T00:55:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:09.712451+0000 osd.3 (osd.3) 170 : cluster [DBG] 28.d scrub ok 2024-02-13T00:55:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:10.567641+0000 mgr.y (mgr.14544) 698 : cluster [DBG] pgmap v1272: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 1.5 KiB/s wr, 135 op/s; 2/675 objects misplaced (0.296%); 1.3 KiB/s, 67 objects/s recovering 2024-02-13T00:55:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:10.602838+0000 mon.a (mon.0) 3798 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-02-13T00:55:11.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:10.684451+0000 mon.a (mon.0) 3799 : audit [DBG] from='client.? 172.21.15.162:0/603181949' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T00:55:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:09.709233+0000 osd.3 (osd.3) 169 : cluster [DBG] 28.d scrub starts 2024-02-13T00:55:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:09.712451+0000 osd.3 (osd.3) 170 : cluster [DBG] 28.d scrub ok 2024-02-13T00:55:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:10.567641+0000 mgr.y (mgr.14544) 698 : cluster [DBG] pgmap v1272: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 1.5 KiB/s wr, 135 op/s; 2/675 objects misplaced (0.296%); 1.3 KiB/s, 67 objects/s recovering 2024-02-13T00:55:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:10.602838+0000 mon.a (mon.0) 3798 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-02-13T00:55:11.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:10.684451+0000 mon.a (mon.0) 3799 : audit [DBG] from='client.? 172.21.15.162:0/603181949' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T00:55:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:09.709233+0000 osd.3 (osd.3) 169 : cluster [DBG] 28.d scrub starts 2024-02-13T00:55:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:09.712451+0000 osd.3 (osd.3) 170 : cluster [DBG] 28.d scrub ok 2024-02-13T00:55:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:10.567641+0000 mgr.y (mgr.14544) 698 : cluster [DBG] pgmap v1272: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 1.5 KiB/s wr, 135 op/s; 2/675 objects misplaced (0.296%); 1.3 KiB/s, 67 objects/s recovering 2024-02-13T00:55:11.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:10.602838+0000 mon.a (mon.0) 3798 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-02-13T00:55:11.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:10.684451+0000 mon.a (mon.0) 3799 : audit [DBG] from='client.? 172.21.15.162:0/603181949' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T00:55:12.593 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:55:12.593 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:55:12.593 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-02-13T00:55:12.594 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-02-13T00:55:12.594 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:55:12.912 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:10.697126+0000 osd.3 (osd.3) 171 : cluster [DBG] 28.14 deep-scrub starts 2024-02-13T00:55:12.912 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:10.700539+0000 osd.3 (osd.3) 172 : cluster [DBG] 28.14 deep-scrub ok 2024-02-13T00:55:12.912 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:11.838166+0000 osd.2 (osd.2) 155 : cluster [DBG] 5.a scrub starts 2024-02-13T00:55:12.912 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:11.838961+0000 osd.2 (osd.2) 156 : cluster [DBG] 5.a scrub ok 2024-02-13T00:55:13.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:10.697126+0000 osd.3 (osd.3) 171 : cluster [DBG] 28.14 deep-scrub starts 2024-02-13T00:55:13.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:10.700539+0000 osd.3 (osd.3) 172 : cluster [DBG] 28.14 deep-scrub ok 2024-02-13T00:55:13.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:11.838166+0000 osd.2 (osd.2) 155 : cluster [DBG] 5.a scrub starts 2024-02-13T00:55:13.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:11.838961+0000 osd.2 (osd.2) 156 : cluster [DBG] 5.a scrub ok 2024-02-13T00:55:13.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:10.697126+0000 osd.3 (osd.3) 171 : cluster [DBG] 28.14 deep-scrub starts 2024-02-13T00:55:13.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:10.700539+0000 osd.3 (osd.3) 172 : cluster [DBG] 28.14 deep-scrub ok 2024-02-13T00:55:13.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:11.838166+0000 osd.2 (osd.2) 155 : cluster [DBG] 5.a scrub starts 2024-02-13T00:55:13.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:11.838961+0000 osd.2 (osd.2) 156 : cluster [DBG] 5.a scrub ok 2024-02-13T00:55:13.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-13T00:55:13.533 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:55:13.595 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.590+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2875433837 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80fd490 msgr2=0x7fbdc80f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.590+0000 7fbdcd2d0700 1 --2- 172.21.15.162:0/2875433837 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80fd490 0x7fbdc80f6cc0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 rx=0x7fbdb8004870 tx=0x7fbdb8009cb0).stop 2024-02-13T00:55:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.590+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2875433837 shutdown_connections 2024-02-13T00:55:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.590+0000 7fbdcd2d0700 1 --2- 172.21.15.162:0/2875433837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdc80f73a0 0x7fbdc81039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.590+0000 7fbdcd2d0700 1 --2- 172.21.15.162:0/2875433837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdc80fcb30 0x7fbdc80fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.590+0000 7fbdcd2d0700 1 --2- 172.21.15.162:0/2875433837 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80fd490 0x7fbdc80f6cc0 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.590+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2875433837 >> 172.21.15.162:0/2875433837 conn(0x7fbdc806eb90 msgr2=0x7fbdc80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.590+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2875433837 shutdown_connections 2024-02-13T00:55:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.590+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2875433837 wait complete. 2024-02-13T00:55:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.591+0000 7fbdcd2d0700 1 librados: starting msgr at 2024-02-13T00:55:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.591+0000 7fbdcd2d0700 1 librados: starting objecter 2024-02-13T00:55:13.596 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.591+0000 7fbdcd2d0700 1 Processor -- start 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.591+0000 7fbdcd2d0700 1 -- start start 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.591+0000 7fbdcd2d0700 1 librados: setting wanted keys 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.591+0000 7fbdcd2d0700 1 librados: calling monclient init 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdcd2d0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdc80f73a0 0x7fbdc8109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdcd2d0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdc80fcb30 0x7fbdc8109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdcd2d0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80fd490 0x7fbdc8109bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdcd2d0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8114330 con 0x7fbdc80fcb30 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdcd2d0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc81144a0 con 0x7fbdc80f73a0 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdcd2d0700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8114660 con 0x7fbdc80fd490 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc67fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdc80fcb30 0x7fbdc8109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc77fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80fd490 0x7fbdc8109bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc6ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdc80f73a0 0x7fbdc8109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc77fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80fd490 0x7fbdc8109bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:59366/0 (socket says 172.21.15.162:59366) 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc67fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdc80fcb30 0x7fbdc8109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:37098/0 (socket says 172.21.15.162:37098) 2024-02-13T00:55:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc77fe700 1 -- 172.21.15.162:0/2841665738 learned_addr learned my addr 172.21.15.162:0/2841665738 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc6ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdc80f73a0 0x7fbdc8109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.162:39114/0 (socket says 172.21.15.162:39114) 2024-02-13T00:55:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc77fe700 1 -- 172.21.15.162:0/2841665738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdc80f73a0 msgr2=0x7fbdc8109170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc77fe700 1 --2- 172.21.15.162:0/2841665738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdc80f73a0 0x7fbdc8109170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc77fe700 1 -- 172.21.15.162:0/2841665738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdc80fcb30 msgr2=0x7fbdc8109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc77fe700 1 --2- 172.21.15.162:0/2841665738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdc80fcb30 0x7fbdc8109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc77fe700 1 -- 172.21.15.162:0/2841665738 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc00097c0 con 0x7fbdc80fd490 2024-02-13T00:55:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc67fc700 1 --2- 172.21.15.162:0/2841665738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdc80fcb30 0x7fbdc8109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdc77fe700 1 --2- 172.21.15.162:0/2841665738 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80fd490 0x7fbdc8109bb0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 rx=0x7fbdc0001df0 tx=0x7fbdc0004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdaffff700 10 client.?.objecter ms_handle_connect 0x7fbdc80fd490 2024-02-13T00:55:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdaffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:13.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdaffff700 1 -- 172.21.15.162:0/2841665738 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0009a10 con 0x7fbdc80fd490 2024-02-13T00:55:13.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdaffff700 1 -- 172.21.15.162:0/2841665738 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdc00090d0 con 0x7fbdc80fd490 2024-02-13T00:55:13.599 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2841665738 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb8005040 con 0x7fbdc80fd490 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdaffff700 1 -- 172.21.15.162:0/2841665738 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0007d30 con 0x7fbdc80fd490 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdcd2d0700 10 client.38354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2841665738 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc8114d20 con 0x7fbdc80fd490 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.592+0000 7fbdcd2d0700 1 librados: init done 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.593+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2841665738 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc8106020 con 0x7fbdc80fd490 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.594+0000 7fbdaffff700 1 -- 172.21.15.162:0/2841665738 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fbdc0015630 con 0x7fbdc80fd490 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.594+0000 7fbdaffff700 1 --2- 172.21.15.162:0/2841665738 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbdb0062c90 0x7fbdb0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.594+0000 7fbdaffff700 1 -- 172.21.15.162:0/2841665738 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(825..825 src has 449..825) v4 ==== 7426+0+0 (secure 0 0 0) 0x7fbdc007d7b0 con 0x7fbdc80fd490 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.594+0000 7fbdaffff700 10 client.38354.objecter ms_dispatch 0x7fbdc8001550 osd_map(825..825 src has 449..825) v4 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.594+0000 7fbdaffff700 3 client.38354.objecter handle_osd_map got epochs [825,825] > 0 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.594+0000 7fbdaffff700 3 client.38354.objecter handle_osd_map decoding full epoch 825 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.594+0000 7fbdc6ffd700 1 --2- 172.21.15.162:0/2841665738 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbdb0062c90 0x7fbdb0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.594+0000 7fbdaffff700 20 client.38354.objecter dump_active .. 0 homeless 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.594+0000 7fbdaffff700 10 client.38354.objecter ms_handle_connect 0x7fbdb0062c90 2024-02-13T00:55:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.594+0000 7fbdc6ffd700 1 --2- 172.21.15.162:0/2841665738 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbdb0062c90 0x7fbdb0065150 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 rx=0x7fbdbc004d20 tx=0x7fbdbc009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:13.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.598+0000 7fbdaffff700 1 -- 172.21.15.162:0/2841665738 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fbdc0050ed0 con 0x7fbdc80fd490 2024-02-13T00:55:13.611 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:12.503884+0000 osd.6 (osd.6) 290 : cluster [DBG] 28.8 scrub starts 2024-02-13T00:55:13.611 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:12.506868+0000 osd.6 (osd.6) 291 : cluster [DBG] 28.8 scrub ok 2024-02-13T00:55:13.611 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:12.568045+0000 mgr.y (mgr.14544) 699 : cluster [DBG] pgmap v1274: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%); 98 B/s, 3 objects/s recovering 2024-02-13T00:55:13.611 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:12.610048+0000 osd.3 (osd.3) 173 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:55:13.611 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:12.613517+0000 osd.3 (osd.3) 174 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:55:13.611 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:13.020401+0000 osd.1 (osd.1) 351 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:55:13.612 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:13.023336+0000 osd.1 (osd.1) 352 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:55:13.612 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:13.182600+0000 osd.5 (osd.5) 18 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:55:13.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:13.776+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2841665738 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7fbdc805fb80 con 0x7fbdc80fd490 2024-02-13T00:55:13.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:12.503884+0000 osd.6 (osd.6) 290 : cluster [DBG] 28.8 scrub starts 2024-02-13T00:55:13.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:12.506868+0000 osd.6 (osd.6) 291 : cluster [DBG] 28.8 scrub ok 2024-02-13T00:55:13.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:12.568045+0000 mgr.y (mgr.14544) 699 : cluster [DBG] pgmap v1274: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%); 98 B/s, 3 objects/s recovering 2024-02-13T00:55:13.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:12.610048+0000 osd.3 (osd.3) 173 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:55:13.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:12.613517+0000 osd.3 (osd.3) 174 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:55:13.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:13.020401+0000 osd.1 (osd.1) 351 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:55:13.873 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:13.023336+0000 osd.1 (osd.1) 352 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:55:13.874 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:13.182600+0000 osd.5 (osd.5) 18 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:55:13.874 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:13.186194+0000 osd.5 (osd.5) 19 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:55:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:13.186194+0000 osd.5 (osd.5) 19 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:55:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:12.503884+0000 osd.6 (osd.6) 290 : cluster [DBG] 28.8 scrub starts 2024-02-13T00:55:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:12.506868+0000 osd.6 (osd.6) 291 : cluster [DBG] 28.8 scrub ok 2024-02-13T00:55:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:12.568045+0000 mgr.y (mgr.14544) 699 : cluster [DBG] pgmap v1274: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%); 98 B/s, 3 objects/s recovering 2024-02-13T00:55:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:12.610048+0000 osd.3 (osd.3) 173 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:55:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:12.613517+0000 osd.3 (osd.3) 174 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:55:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:13.020401+0000 osd.1 (osd.1) 351 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:55:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:13.023336+0000 osd.1 (osd.1) 352 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:55:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:13.182600+0000 osd.5 (osd.5) 18 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:55:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:13.186194+0000 osd.5 (osd.5) 19 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:55:14.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.612+0000 7fbdaffff700 1 -- 172.21.15.162:0/2841665738 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v826) v1 ==== 113+0+0 (secure 0 0 0) 0x7fbdc004be60 con 0x7fbdc80fd490 2024-02-13T00:55:14.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 10 librados: watch_flush enter 2024-02-13T00:55:14.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 10 librados: watch_flush exit 2024-02-13T00:55:14.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 20 client.38354.objecter shutdown clearing up homeless session... 2024-02-13T00:55:14.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 10 client.38354.objecter successfully canceled tick 2024-02-13T00:55:14.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2841665738 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbdb0062c90 msgr2=0x7fbdb0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:14.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 1 --2- 172.21.15.162:0/2841665738 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbdb0062c90 0x7fbdb0065150 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 rx=0x7fbdbc004d20 tx=0x7fbdbc009040).stop 2024-02-13T00:55:14.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2841665738 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80fd490 msgr2=0x7fbdc8109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:14.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 1 --2- 172.21.15.162:0/2841665738 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80fd490 0x7fbdc8109bb0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 rx=0x7fbdc0001df0 tx=0x7fbdc0004c50).stop 2024-02-13T00:55:14.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2841665738 shutdown_connections 2024-02-13T00:55:14.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 1 --2- 172.21.15.162:0/2841665738 >> v2:172.21.15.162:6800/2866534053 conn(0x7fbdb0062c90 0x7fbdb0065150 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:14.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 1 --2- 172.21.15.162:0/2841665738 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80fd490 0x7fbdc8109bb0 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:14.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 1 --2- 172.21.15.162:0/2841665738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdc80f73a0 0x7fbdc8109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:14.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 1 --2- 172.21.15.162:0/2841665738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdc80fcb30 0x7fbdc8109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:14.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2841665738 >> 172.21.15.162:0/2841665738 conn(0x7fbdc806eb90 msgr2=0x7fbdc80700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:14.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2841665738 shutdown_connections 2024-02-13T00:55:14.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.615+0000 7fbdcd2d0700 1 -- 172.21.15.162:0/2841665738 wait complete. 2024-02-13T00:55:14.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:14.616+0000 7fbdcd2d0700 1 librados: shutdown 2024-02-13T00:55:14.620 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.0 to 1 (10000) 2024-02-13T00:55:14.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:13.581065+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:55:14.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:13.584308+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:55:14.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:13.777104+0000 mon.c (mon.2) 523 : audit [INF] from='client.? 172.21.15.162:0/2841665738' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:55:14.909 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:13.777378+0000 mon.a (mon.0) 3800 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:55:14.963 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:13.581065+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:55:14.963 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:13.584308+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:55:14.963 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:13.777104+0000 mon.c (mon.2) 523 : audit [INF] from='client.? 172.21.15.162:0/2841665738' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:55:14.963 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:13.777378+0000 mon.a (mon.0) 3800 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:55:14.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:13.581065+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:55:14.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:13.584308+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:55:14.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:13.777104+0000 mon.c (mon.2) 523 : audit [INF] from='client.? 172.21.15.162:0/2841665738' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:55:14.963 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:13.777378+0000 mon.a (mon.0) 3800 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:55:15.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-13T00:55:15.461 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:55:15.461 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:55:15.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:55:15.461 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:55:15.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.461 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.1.0:1092 243.f e676/675 243:f2a1c75f:test-rados-api-smithi162-84057-119::foo:head v 676'1, mlcod=0'0)", 2024-02-13T00:55:15.461 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:41.582582+0000", 2024-02-13T00:55:15.461 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 213.87391277699999, 2024-02-13T00:55:15.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003452798, 2024-02-13T00:55:15.461 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.461 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582582+0000", 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582582+0000", 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582582+0000", 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1119999999999998e-06 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582586+0000", 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7359999999999998e-06 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582591+0000", 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.165e-06 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582599+0000", 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032407200000000001 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582923+0000", 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4011000000000003e-05 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582967+0000", 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030438359999999999 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.586010+0000", 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4156999999999999e-05 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.586035+0000", 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003452798 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.27753.0:1101 243.f e676/675 243:f2a1c75f:test-rados-api-smithi162-84057-119::foo:head v 676'2, mlcod=0'0)", 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:41.582604+0000", 2024-02-13T00:55:15.465 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 213.87389086600001, 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034472249999999999, 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582604+0000", 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582604+0000", 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:15.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582603+0000", 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4819999999999997e-06 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582611+0000", 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6959999999999996e-06 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582616+0000", 2024-02-13T00:55:15.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8519999999999997e-06 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.582619+0000", 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049247699999999995 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.583112+0000", 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6790999999999999e-05 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:15.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.583129+0000", 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029083709999999999 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.586037+0000", 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3876999999999999e-05 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:41.586051+0000", 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034472249999999999 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.14 epoch 677/675 rep_tid 1095 entries 677'15 (0'0) error 5:2b04a3e9:::notify.6:head by client.24329.0:2089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:51:44.375644+0000", 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 211.08085081999999, 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001229675, 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375644+0000", 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375644+0000", 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375643+0000", 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8799999999999997e-06 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375652+0000", 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2441999999999999e-05 2024-02-13T00:55:15.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375684+0000", 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9670000000000007e-06 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.375693+0000", 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00038377499999999999 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376077+0000", 2024-02-13T00:55:15.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00079641200000000003 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:51:44.376873+0000", 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001229675 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s1 697/675 ECSubWrite(tid=779, reqid=osd.1.13:208, at_version=697'2, trim_to=0'0, roll_forward_to=0'0))", 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:04.576142+0000", 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 190.88035265799999, 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001836847, 2024-02-13T00:55:15.473 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576142+0000", 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576142+0000", 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576141+0000", 2024-02-13T00:55:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3881000000000001e-05 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576165+0000", 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8133000000000001e-05 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576183+0000", 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6520000000000006e-06 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576190+0000", 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9348000000000001e-05 2024-02-13T00:55:15.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576260+0000", 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.815e-06 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.576270+0000", 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016926160000000001 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.577962+0000", 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6589e-05 2024-02-13T00:55:15.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:04.577979+0000", 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001836847 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(234.3s1 698/675 ECSubRead(tid=785, to_read={234:c8594d0f:test-rados-api-smithi162-84057-123::foo:head=0,4096,0}, subchunks={234:c8594d0f:test-rados-api-smithi162-84057-123::foo:head=[0,1]}, attrs_to_read=))", 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:05.566388+0000", 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 189.89010695600001, 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020959070000000002, 2024-02-13T00:55:15.477 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.566388+0000", 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.566388+0000", 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.566387+0000", 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8300000000000008e-06 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.566397+0000", 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1646e-05 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.566409+0000", 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0070000000000003e-06 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.566418+0000", 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001849334 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.568267+0000", 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021640599999999999 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.568484+0000", 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020959070000000002 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(234.3s1 698/675 ECSubRead(tid=786, to_read={234:c8594d0f:test-rados-api-smithi162-84057-123::foo:9=0,4096,0}, subchunks={234:c8594d0f:test-rados-api-smithi162-84057-123::foo:9=[0,1]}, attrs_to_read=))", 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:05.578253+0000", 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 189.87824116600001, 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039044069999999999, 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578253+0000", 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578253+0000", 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578253+0000", 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.465e-06 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578262+0000", 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0709000000000001e-05 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578283+0000", 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0839999999999998e-06 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578292+0000", 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6171999999999997e-05 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.578348+0000", 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038093770000000001 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:05.582158+0000", 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039044069999999999 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.3s1 716/675 ECSubWrite(tid=792, reqid=osd.6.45:144, at_version=716'9, trim_to=0'0, roll_forward_to=700'8))", 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:23.591714+0000", 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 171.86478104299999, 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0065409589999999998, 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:15.484 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.591714+0000", 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.591714+0000", 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.591713+0000", 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014055579999999999 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.593119+0000", 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4260999999999999e-05 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.593133+0000", 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1175000000000001e-05 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.593144+0000", 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056209999999999995 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.593706+0000", 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1963999999999999e-05 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.593718+0000", 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045167849999999997 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.598235+0000", 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9380000000000001e-05 2024-02-13T00:55:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:23.598254+0000", 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0065409589999999998 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(2.b REQUEST e717)", 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:25.143446+0000", 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 170.31304864099999, 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027255790000000001, 2024-02-13T00:55:15.488 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143446+0000", 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143446+0000", 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967296 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143446+0000", 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0850000000000001e-06 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143449+0000", 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1029000000000001e-05 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143460+0000", 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.337e-06 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143468+0000", 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2531000000000002e-05 2024-02-13T00:55:15.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143521+0000", 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8110000000000003e-06 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.143529+0000", 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025945439999999998 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.146123+0000", 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6027000000000001e-05 2024-02-13T00:55:15.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.146139+0000", 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2398999999999999e-05 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:25.146172+0000", 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027255790000000001 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.27753.0:2014 249.6 e735/729 249:602f83fe:::foo:head v 735'11, mlcod=735'9)", 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:52:50.582517+0000", 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 144.87397744399999, 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.023893649999999999, 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582517+0000", 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582517+0000", 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999928 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582510+0000", 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8892999999999999e-05 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582539+0000", 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2396999999999999e-05 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582561+0000", 2024-02-13T00:55:15.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7968999999999999e-05 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582589+0000", 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6641999999999997e-05 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582646+0000", 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1164000000000002e-05 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.582677+0000", 2024-02-13T00:55:15.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.023693683 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.606371+0000", 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9833999999999998e-05 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:52:50.606411+0000", 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.023893649999999999 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.0.0:1971 252.4 e774/765 252:26bf2024:test-rados-api-smithi162-84057-137::bar:head v 774'2, mlcod=0'0)", 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:53:38.580805+0000", 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 96.875689821999998, 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015436169999999999, 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:38.580805+0000", 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:38.580805+0000", 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:38.580803+0000", 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1090999999999999e-05 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:38.580814+0000", 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.047e-05 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:38.580825+0000", 2024-02-13T00:55:15.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0520000000000004e-06 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:38.580832+0000", 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00091538100000000005 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:38.581747+0000", 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1787999999999999e-05 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:38.581769+0000", 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00054725900000000003 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:38.582316+0000", 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1962999999999999e-05 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:38.582348+0000", 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015436169999999999 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.500 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.0s0 782/722 ECSubWrite(tid=853, reqid=client.27753.0:2911, at_version=782'6, trim_to=0'0, roll_forward_to=740'5))", 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:53:52.691405+0000", 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 82.765089599000007, 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013084069999999999, 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.691405+0000", 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.691405+0000", 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.691405+0000", 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9669999999999999e-05 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.691434+0000", 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4099999999999997e-06 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.691444+0000", 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6000000000000007e-06 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.691452+0000", 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081601000000000004 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.692268+0000", 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.311e-05 2024-02-13T00:55:15.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.692281+0000", 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00040327400000000002 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.692685+0000", 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8603e-05 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.692713+0000", 2024-02-13T00:55:15.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013084069999999999 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.0s0 782/722 ECSubWrite(tid=854, reqid=client.27753.0:2911, at_version=0'0, trim_to=0'0, roll_forward_to=782'6))", 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:53:52.707482+0000", 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 82.749012819000001, 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011541819999999999, 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.707482+0000", 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.707482+0000", 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.707481+0000", 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0129000000000001e-05 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.707491+0000", 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2049999999999999e-06 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.707498+0000", 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4250000000000001e-06 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.707505+0000", 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000285414 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.707790+0000", 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8430000000000001e-05 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.707819+0000", 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00080071500000000002 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.708620+0000", 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6307000000000002e-05 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:52.708636+0000", 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011541819999999999 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.0s0 783/722 ECSubWrite(tid=857, reqid=client.27753.0:2913, at_version=0'0, trim_to=0'0, roll_forward_to=783'7))", 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:53:53.700354+0000", 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 81.756140259999995, 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017322080000000001, 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:53.700354+0000", 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:53.700354+0000", 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:53.700353+0000", 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4425e-05 2024-02-13T00:55:15.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:53.700368+0000", 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0895e-05 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:53.700378+0000", 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5959999999999998e-06 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:53.700386+0000", 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00088817300000000002 2024-02-13T00:55:15.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:53.701274+0000", 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0242e-05 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:53.701284+0000", 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00077592799999999995 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:55:15.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:53.702060+0000", 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6146999999999999e-05 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:53.702087+0000", 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017322080000000001 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(234.2s2 787/675 ECSubWrite(tid=861, reqid=client.27753.0:3017, at_version=0'0, trim_to=0'0, roll_forward_to=787'22))", 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:53:57.606669+0000", 2024-02-13T00:55:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 77.849825713000001, 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011975110000000001, 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:57.606669+0000", 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:57.606669+0000", 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:15.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:57.606668+0000", 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3042e-05 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:57.606681+0000", 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5468000000000002e-05 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:57.606696+0000", 2024-02-13T00:55:15.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.542e-06 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:57.606704+0000", 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6058999999999999e-05 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:57.606770+0000", 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2087e-05 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:57.606782+0000", 2024-02-13T00:55:15.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001059983 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:57.607842+0000", 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4267999999999999e-05 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:53:57.607866+0000", 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011975110000000001 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.27753.0:3122 255.10 e790/789 255:0ff09505:test-rados-api-smithi162-84057-143::foo:head v 790'1, mlcod=0'0)", 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:00.618019+0000", 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 74.838475555000002, 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039705870000000002, 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:00.618019+0000", 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:00.618019+0000", 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:00.618019+0000", 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.22e-06 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:00.618024+0000", 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7700000000000004e-06 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:00.618031+0000", 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2359999999999998e-06 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:00.618038+0000", 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026425 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:00.618302+0000", 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0367e-05 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:00.618323+0000", 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003636252 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:00.621959+0000", 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0811e-05 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:00.621990+0000", 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039705870000000002 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.521 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(5.6 REQUEST e803)", 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:25.243113+0000", 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 50.213381077999998, 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032175490000000001, 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:25.243113+0000", 2024-02-13T00:55:15.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:25.243113+0000", 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:25.243113+0000", 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1569999999999997e-06 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:25.243119+0000", 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1302999999999999e-05 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:25.243140+0000", 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3326000000000001e-05 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:25.243154+0000", 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114704 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:25.243268+0000", 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2319999999999998e-06 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:25.243275+0000", 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030182899999999999 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:25.246293+0000", 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4132e-05 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:25.246307+0000", 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3975e-05 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:25.246331+0000", 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032175490000000001 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDScrubReserve(4.1e REQUEST e804)", 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:29.160973+0000", 2024-02-13T00:55:15.526 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 46.295521399000002, 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032939409999999999, 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:29.160973+0000", 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:29.160973+0000", 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:29.160973+0000", 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8470000000000001e-06 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:29.160978+0000", 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0772e-05 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:29.160999+0000", 2024-02-13T00:55:15.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6447e-05 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:29.161016+0000", 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000123643 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:29.161139+0000", 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9990000000000002e-06 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:29.161146+0000", 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003083812 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:29.164230+0000", 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4654000000000001e-05 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:29.164245+0000", 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2399999999999999e-05 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:29.164267+0000", 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032939409999999999 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPull(2.b e813/812 cost 1049576)", 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:53.119755+0000", 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.336740047999999, 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027337849999999999, 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:15.531 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119755+0000", 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119755+0000", 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119754+0000", 2024-02-13T00:55:15.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4644000000000001e-05 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119769+0000", 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4676e-05 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119783+0000", 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1689999999999997e-06 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119789+0000", 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8303000000000002e-05 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119838+0000", 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026506519999999999 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.122488+0000", 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027337849999999999 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.14 epoch 817/815 rep_tid 1155 entries 817'17 (0'0) error 5:2b04a3e9:::notify.6:head by client.24329.0:2477 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:59.410646+0000", 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.045848361000001, 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00116052, 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410646+0000", 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410646+0000", 2024-02-13T00:55:15.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410645+0000", 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.046e-05 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410656+0000", 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9069e-05 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410675+0000", 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7829999999999993e-06 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410685+0000", 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8622999999999994e-05 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410783+0000", 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010234599999999999 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.411807+0000", 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00116052 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.d epoch 817/815 rep_tid 1 entries 817'22 (0'0) error 5:b4812045:::notify.4:head by client.24329.0:2474 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:59.410655+0000", 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.045839693000001, 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011691889999999999, 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:15.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410655+0000", 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410655+0000", 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410654+0000", 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8200000000000003e-06 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410663+0000", 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3169999999999999e-05 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410676+0000", 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5299999999999999e-06 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410684+0000", 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9796000000000001e-05 2024-02-13T00:55:15.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410743+0000", 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001080751 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.411824+0000", 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011691889999999999 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:15.541 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:55:15.714 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:14.568668+0000 mgr.y (mgr.14544) 700 : cluster [DBG] pgmap v1275: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 659 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%); 75 B/s, 2 objects/s recovering 2024-02-13T00:55:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:14.594968+0000 osd.3 (osd.3) 177 : cluster [DBG] 28.c scrub starts 2024-02-13T00:55:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:14.597916+0000 osd.3 (osd.3) 178 : cluster [DBG] 28.c scrub ok 2024-02-13T00:55:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:14.613086+0000 mon.a (mon.0) 3801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:55:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:14.613183+0000 mon.a (mon.0) 3802 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-02-13T00:55:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:14.941133+0000 osd.0 (osd.0) 347 : cluster [DBG] 28.16 scrub starts 2024-02-13T00:55:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:14.944134+0000 osd.0 (osd.0) 348 : cluster [DBG] 28.16 scrub ok 2024-02-13T00:55:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:15.493170+0000 mon.a (mon.0) 3803 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:15.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:15.608751+0000 mon.a (mon.0) 3804 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-02-13T00:55:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:14.568668+0000 mgr.y (mgr.14544) 700 : cluster [DBG] pgmap v1275: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 659 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%); 75 B/s, 2 objects/s recovering 2024-02-13T00:55:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:14.594968+0000 osd.3 (osd.3) 177 : cluster [DBG] 28.c scrub starts 2024-02-13T00:55:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:14.597916+0000 osd.3 (osd.3) 178 : cluster [DBG] 28.c scrub ok 2024-02-13T00:55:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:14.613086+0000 mon.a (mon.0) 3801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:55:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:14.613183+0000 mon.a (mon.0) 3802 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-02-13T00:55:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:14.941133+0000 osd.0 (osd.0) 347 : cluster [DBG] 28.16 scrub starts 2024-02-13T00:55:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:14.944134+0000 osd.0 (osd.0) 348 : cluster [DBG] 28.16 scrub ok 2024-02-13T00:55:15.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:15.493170+0000 mon.a (mon.0) 3803 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:15.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:15.608751+0000 mon.a (mon.0) 3804 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-02-13T00:55:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:14.568668+0000 mgr.y (mgr.14544) 700 : cluster [DBG] pgmap v1275: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 659 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%); 75 B/s, 2 objects/s recovering 2024-02-13T00:55:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:14.594968+0000 osd.3 (osd.3) 177 : cluster [DBG] 28.c scrub starts 2024-02-13T00:55:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:14.597916+0000 osd.3 (osd.3) 178 : cluster [DBG] 28.c scrub ok 2024-02-13T00:55:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:14.613086+0000 mon.a (mon.0) 3801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:55:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:14.613183+0000 mon.a (mon.0) 3802 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-02-13T00:55:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:14.941133+0000 osd.0 (osd.0) 347 : cluster [DBG] 28.16 scrub starts 2024-02-13T00:55:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:14.944134+0000 osd.0 (osd.0) 348 : cluster [DBG] 28.16 scrub ok 2024-02-13T00:55:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:15.493170+0000 mon.a (mon.0) 3803 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:15.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:15.608751+0000 mon.a (mon.0) 3804 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-02-13T00:55:16.140 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:55:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:55:16.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-13T00:55:16.296 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:55:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:55:16] "GET /metrics HTTP/1.1" 200 185508 "" "Prometheus/2.33.4" 2024-02-13T00:55:16.427 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:55:17.028 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:16.034799+0000 mgr.y (mgr.14544) 701 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:17.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:16.034799+0000 mgr.y (mgr.14544) 701 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:16.034799+0000 mgr.y (mgr.14544) 701 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:17.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.597+0000 7f9aca9ef700 1 -- 172.21.15.162:0/1496684775 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ac40fdcf0 msgr2=0x7f9ac40fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:17.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.597+0000 7f9aca9ef700 1 --2- 172.21.15.162:0/1496684775 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ac40fdcf0 0x7f9ac40fe0d0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 rx=0x7f9aac004870 tx=0x7f9aac009cb0).stop 2024-02-13T00:55:17.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.597+0000 7f9aca9ef700 1 -- 172.21.15.162:0/1496684775 shutdown_connections 2024-02-13T00:55:17.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.597+0000 7f9aca9ef700 1 --2- 172.21.15.162:0/1496684775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ac40f68e0 0x7f9ac4104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:17.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.597+0000 7f9aca9ef700 1 --2- 172.21.15.162:0/1496684775 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ac40fdcf0 0x7f9ac40fe0d0 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:17.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.597+0000 7f9aca9ef700 1 --2- 172.21.15.162:0/1496684775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ac40fe5c0 0x7f9ac40f6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:17.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.597+0000 7f9aca9ef700 1 -- 172.21.15.162:0/1496684775 >> 172.21.15.162:0/1496684775 conn(0x7f9ac4070080 msgr2=0x7f9ac40f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:17.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.597+0000 7f9aca9ef700 1 -- 172.21.15.162:0/1496684775 shutdown_connections 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.597+0000 7f9aca9ef700 1 -- 172.21.15.162:0/1496684775 wait complete. 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.597+0000 7f9aca9ef700 1 librados: starting msgr at 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.597+0000 7f9aca9ef700 1 librados: starting objecter 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.597+0000 7f9aca9ef700 1 Processor -- start 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9aca9ef700 1 -- start start 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9aca9ef700 1 librados: setting wanted keys 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9aca9ef700 1 librados: calling monclient init 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9aca9ef700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ac40f68e0 0x7f9ac40f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9aca9ef700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ac40fdcf0 0x7f9ac40f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9aca9ef700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ac40fe5c0 0x7f9ac40f7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9aca9ef700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac41166c0 con 0x7f9ac40f68e0 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9aca9ef700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac4116830 con 0x7f9ac40fdcf0 2024-02-13T00:55:17.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9aca9ef700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac41169f0 con 0x7f9ac40fe5c0 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9ac8f8c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ac40fe5c0 0x7f9ac40f7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9ac3fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ac40f68e0 0x7f9ac40f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9ac8f8c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ac40fe5c0 0x7f9ac40f7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:55596/0 (socket says 172.21.15.162:55596) 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9ac8f8c700 1 -- 172.21.15.162:0/2523969514 learned_addr learned my addr 172.21.15.162:0/2523969514 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9ac37fe700 1 --2- 172.21.15.162:0/2523969514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ac40fdcf0 0x7f9ac40f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9ac8f8c700 1 -- 172.21.15.162:0/2523969514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ac40fdcf0 msgr2=0x7f9ac40f79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9ac8f8c700 1 --2- 172.21.15.162:0/2523969514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ac40fdcf0 0x7f9ac40f79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9ac8f8c700 1 -- 172.21.15.162:0/2523969514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ac40f68e0 msgr2=0x7f9ac40f74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9ac8f8c700 1 --2- 172.21.15.162:0/2523969514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ac40f68e0 0x7f9ac40f74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9ac8f8c700 1 -- 172.21.15.162:0/2523969514 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab4005040 con 0x7f9ac40fe5c0 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.598+0000 7f9ac3fff700 1 --2- 172.21.15.162:0/2523969514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ac40f68e0 0x7f9ac40f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.599+0000 7f9ac17fa700 10 client.?.objecter ms_handle_connect 0x7f9ac40fe5c0 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.599+0000 7f9ac17fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:17.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.599+0000 7f9ac8f8c700 1 --2- 172.21.15.162:0/2523969514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ac40fe5c0 0x7f9ac40f7eb0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 rx=0x7f9abc00d970 tx=0x7f9abc004e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:17.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.599+0000 7f9ac17fa700 1 -- 172.21.15.162:0/2523969514 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc004020 con 0x7f9ac40fe5c0 2024-02-13T00:55:17.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.599+0000 7f9aca9ef700 1 -- 172.21.15.162:0/2523969514 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aac005040 con 0x7f9ac40fe5c0 2024-02-13T00:55:17.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.599+0000 7f9ac17fa700 1 -- 172.21.15.162:0/2523969514 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9abc00f040 con 0x7f9ac40fe5c0 2024-02-13T00:55:17.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.599+0000 7f9ac17fa700 1 -- 172.21.15.162:0/2523969514 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc0036c0 con 0x7f9ac40fe5c0 2024-02-13T00:55:17.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.599+0000 7f9aca9ef700 10 client.38360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:17.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.599+0000 7f9aca9ef700 1 -- 172.21.15.162:0/2523969514 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac4117100 con 0x7f9ac40fe5c0 2024-02-13T00:55:17.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.599+0000 7f9aca9ef700 1 librados: init done 2024-02-13T00:55:17.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.600+0000 7f9aca9ef700 1 -- 172.21.15.162:0/2523969514 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac405fb80 con 0x7f9ac40fe5c0 2024-02-13T00:55:17.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.600+0000 7f9ac17fa700 1 -- 172.21.15.162:0/2523969514 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f9abc003af0 con 0x7f9ac40fe5c0 2024-02-13T00:55:17.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.600+0000 7f9ac17fa700 1 --2- 172.21.15.162:0/2523969514 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9ab0062c90 0x7f9ab0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:17.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.601+0000 7f9ac17fa700 1 -- 172.21.15.162:0/2523969514 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(827..827 src has 449..827) v4 ==== 7426+0+0 (secure 0 0 0) 0x7f9abc07c950 con 0x7f9ac40fe5c0 2024-02-13T00:55:17.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.601+0000 7f9ac17fa700 10 client.38360.objecter ms_dispatch 0x7f9ac4001550 osd_map(827..827 src has 449..827) v4 2024-02-13T00:55:17.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.601+0000 7f9ac17fa700 3 client.38360.objecter handle_osd_map got epochs [827,827] > 0 2024-02-13T00:55:17.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.601+0000 7f9ac17fa700 3 client.38360.objecter handle_osd_map decoding full epoch 827 2024-02-13T00:55:17.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.601+0000 7f9ac3fff700 1 --2- 172.21.15.162:0/2523969514 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9ab0062c90 0x7f9ab0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:17.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.601+0000 7f9ac17fa700 20 client.38360.objecter dump_active .. 0 homeless 2024-02-13T00:55:17.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.601+0000 7f9ac17fa700 10 client.38360.objecter ms_handle_connect 0x7f9ab0062c90 2024-02-13T00:55:17.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.601+0000 7f9ac3fff700 1 --2- 172.21.15.162:0/2523969514 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9ab0062c90 0x7f9ab0065150 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 rx=0x7f9aac009bc0 tx=0x7f9aac00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:17.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.605+0000 7f9ac17fa700 1 -- 172.21.15.162:0/2523969514 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9abc048f60 con 0x7f9ac40fe5c0 2024-02-13T00:55:17.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:16.569119+0000 mgr.y (mgr.14544) 702 : cluster [DBG] pgmap v1278: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:17.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:16.820241+0000 osd.2 (osd.2) 157 : cluster [DBG] 28.6 scrub starts 2024-02-13T00:55:17.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:16.823151+0000 osd.2 (osd.2) 158 : cluster [DBG] 28.6 scrub ok 2024-02-13T00:55:17.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:16.945211+0000 osd.0 (osd.0) 349 : cluster [DBG] 28.12 scrub starts 2024-02-13T00:55:17.792 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:17.788+0000 7f9aca9ef700 1 -- 172.21.15.162:0/2523969514 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f9ac40475e0 con 0x7f9ac40fe5c0 2024-02-13T00:55:18.041 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:16.569119+0000 mgr.y (mgr.14544) 702 : cluster [DBG] pgmap v1278: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:18.042 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:16.820241+0000 osd.2 (osd.2) 157 : cluster [DBG] 28.6 scrub starts 2024-02-13T00:55:18.042 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:16.823151+0000 osd.2 (osd.2) 158 : cluster [DBG] 28.6 scrub ok 2024-02-13T00:55:18.042 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:16.945211+0000 osd.0 (osd.0) 349 : cluster [DBG] 28.12 scrub starts 2024-02-13T00:55:18.042 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:16.946110+0000 osd.0 (osd.0) 350 : cluster [DBG] 28.12 scrub ok 2024-02-13T00:55:18.042 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:17.013150+0000 osd.1 (osd.1) 353 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:55:18.042 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:17.016616+0000 osd.1 (osd.1) 354 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:55:18.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:16.946110+0000 osd.0 (osd.0) 350 : cluster [DBG] 28.12 scrub ok 2024-02-13T00:55:18.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:17.013150+0000 osd.1 (osd.1) 353 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:55:18.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:17.016616+0000 osd.1 (osd.1) 354 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:55:18.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:16.569119+0000 mgr.y (mgr.14544) 702 : cluster [DBG] pgmap v1278: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:18.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:16.820241+0000 osd.2 (osd.2) 157 : cluster [DBG] 28.6 scrub starts 2024-02-13T00:55:18.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:16.823151+0000 osd.2 (osd.2) 158 : cluster [DBG] 28.6 scrub ok 2024-02-13T00:55:18.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:16.945211+0000 osd.0 (osd.0) 349 : cluster [DBG] 28.12 scrub starts 2024-02-13T00:55:18.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:16.946110+0000 osd.0 (osd.0) 350 : cluster [DBG] 28.12 scrub ok 2024-02-13T00:55:18.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:17.013150+0000 osd.1 (osd.1) 353 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:55:18.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:17.016616+0000 osd.1 (osd.1) 354 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:55:18.589 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:55:18.589 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:55:18.589 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-02-13T00:55:18.589 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:55:18.609 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:55:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:55:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:55:18.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.634+0000 7f9ac17fa700 1 -- 172.21.15.162:0/2523969514 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v829) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9abc0500d0 con 0x7f9ac40fe5c0 2024-02-13T00:55:18.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 10 librados: watch_flush enter 2024-02-13T00:55:18.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 10 librados: watch_flush exit 2024-02-13T00:55:18.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 20 client.38360.objecter shutdown clearing up homeless session... 2024-02-13T00:55:18.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 10 client.38360.objecter successfully canceled tick 2024-02-13T00:55:18.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 -- 172.21.15.162:0/2523969514 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9ab0062c90 msgr2=0x7f9ab0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:18.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 --2- 172.21.15.162:0/2523969514 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9ab0062c90 0x7f9ab0065150 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 rx=0x7f9aac009bc0 tx=0x7f9aac00b040).stop 2024-02-13T00:55:18.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 -- 172.21.15.162:0/2523969514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ac40fe5c0 msgr2=0x7f9ac40f7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:18.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 --2- 172.21.15.162:0/2523969514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ac40fe5c0 0x7f9ac40f7eb0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 rx=0x7f9abc00d970 tx=0x7f9abc004e60).stop 2024-02-13T00:55:18.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 -- 172.21.15.162:0/2523969514 shutdown_connections 2024-02-13T00:55:18.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 --2- 172.21.15.162:0/2523969514 >> v2:172.21.15.162:6800/2866534053 conn(0x7f9ab0062c90 0x7f9ab0065150 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:18.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 --2- 172.21.15.162:0/2523969514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9ac40fe5c0 0x7f9ac40f7eb0 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:18.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 --2- 172.21.15.162:0/2523969514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ac40f68e0 0x7f9ac40f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:18.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 --2- 172.21.15.162:0/2523969514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ac40fdcf0 0x7f9ac40f79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:18.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 -- 172.21.15.162:0/2523969514 >> 172.21.15.162:0/2523969514 conn(0x7f9ac4070080 msgr2=0x7f9ac40fb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:18.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 -- 172.21.15.162:0/2523969514 shutdown_connections 2024-02-13T00:55:18.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 -- 172.21.15.162:0/2523969514 wait complete. 2024-02-13T00:55:18.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:18.637+0000 7f9aca9ef700 1 librados: shutdown 2024-02-13T00:55:18.643 INFO:teuthology.orchestra.run.smithi162.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-13T00:55:18.925 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:17.406570+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:55:18.926 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:17.406712+0000 osd.3 (osd.3) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:55:18.926 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:17.626577+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-02-13T00:55:18.926 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:17.789606+0000 mon.c (mon.2) 524 : audit [INF] from='client.? 172.21.15.162:0/2523969514' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:55:18.926 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:17.789978+0000 mon.a (mon.0) 3806 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:55:18.926 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:17.917030+0000 osd.0 (osd.0) 351 : cluster [DBG] 28.2 scrub starts 2024-02-13T00:55:18.926 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:17.917792+0000 osd.0 (osd.0) 352 : cluster [DBG] 28.2 scrub ok 2024-02-13T00:55:18.926 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:18.537402+0000 osd.3 (osd.3) 181 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:55:18.926 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:18.540759+0000 osd.3 (osd.3) 182 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:55:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:17.406570+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:55:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:17.406712+0000 osd.3 (osd.3) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:55:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:17.626577+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-02-13T00:55:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:17.789606+0000 mon.c (mon.2) 524 : audit [INF] from='client.? 172.21.15.162:0/2523969514' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:55:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:17.789978+0000 mon.a (mon.0) 3806 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:55:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:17.917030+0000 osd.0 (osd.0) 351 : cluster [DBG] 28.2 scrub starts 2024-02-13T00:55:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:17.917792+0000 osd.0 (osd.0) 352 : cluster [DBG] 28.2 scrub ok 2024-02-13T00:55:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:18.537402+0000 osd.3 (osd.3) 181 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:55:19.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:18.540759+0000 osd.3 (osd.3) 182 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:55:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:17.406570+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:55:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:17.406712+0000 osd.3 (osd.3) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:55:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:17.626577+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-02-13T00:55:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:17.789606+0000 mon.c (mon.2) 524 : audit [INF] from='client.? 172.21.15.162:0/2523969514' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:55:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:17.789978+0000 mon.a (mon.0) 3806 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-13T00:55:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:17.917030+0000 osd.0 (osd.0) 351 : cluster [DBG] 28.2 scrub starts 2024-02-13T00:55:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:17.917792+0000 osd.0 (osd.0) 352 : cluster [DBG] 28.2 scrub ok 2024-02-13T00:55:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:18.537402+0000 osd.3 (osd.3) 181 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:55:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:18.540759+0000 osd.3 (osd.3) 182 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:55:19.285 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-13T00:55:19.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 1 1 2024-02-13T00:55:19.555 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:55:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:19.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:18.517820+0000 osd.6 (osd.6) 292 : cluster [DBG] 2.7 deep-scrub starts 2024-02-13T00:55:19.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:18.518695+0000 osd.6 (osd.6) 293 : cluster [DBG] 2.7 deep-scrub ok 2024-02-13T00:55:19.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:18.569908+0000 mgr.y (mgr.14544) 703 : cluster [DBG] pgmap v1280: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:19.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:18.633692+0000 mon.a (mon.0) 3807 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:55:19.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:18.633792+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-02-13T00:55:19.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:18.945753+0000 osd.0 (osd.0) 353 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:55:19.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:18.946848+0000 osd.0 (osd.0) 354 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:55:19.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:18.517820+0000 osd.6 (osd.6) 292 : cluster [DBG] 2.7 deep-scrub starts 2024-02-13T00:55:19.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:18.518695+0000 osd.6 (osd.6) 293 : cluster [DBG] 2.7 deep-scrub ok 2024-02-13T00:55:19.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:18.569908+0000 mgr.y (mgr.14544) 703 : cluster [DBG] pgmap v1280: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:19.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:18.633692+0000 mon.a (mon.0) 3807 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:55:19.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:18.633792+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-02-13T00:55:19.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:18.945753+0000 osd.0 (osd.0) 353 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:55:19.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:18.946848+0000 osd.0 (osd.0) 354 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:55:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:18.517820+0000 osd.6 (osd.6) 292 : cluster [DBG] 2.7 deep-scrub starts 2024-02-13T00:55:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:18.518695+0000 osd.6 (osd.6) 293 : cluster [DBG] 2.7 deep-scrub ok 2024-02-13T00:55:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:18.569908+0000 mgr.y (mgr.14544) 703 : cluster [DBG] pgmap v1280: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:18.633692+0000 mon.a (mon.0) 3807 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-13T00:55:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:18.633792+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-02-13T00:55:20.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:18.945753+0000 osd.0 (osd.0) 353 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:55:20.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:18.946848+0000 osd.0 (osd.0) 354 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:55:20.926 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:20.009285+0000 osd.1 (osd.1) 355 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:55:20.926 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:20.012189+0000 osd.1 (osd.1) 356 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:55:20.926 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:20.506172+0000 mon.a (mon.0) 3809 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:20.969 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:20.009285+0000 osd.1 (osd.1) 355 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:55:20.970 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:20.012189+0000 osd.1 (osd.1) 356 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:55:20.970 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:20.506172+0000 mon.a (mon.0) 3809 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:20.970 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:20.009285+0000 osd.1 (osd.1) 355 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:55:20.970 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:20.012189+0000 osd.1 (osd.1) 356 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:55:20.970 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:20.506172+0000 mon.a (mon.0) 3809 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:21.473 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:55:21.473 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-02-13T00:55:21.473 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-02-13T00:55:21.473 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-02-13T00:55:21.473 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:55:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:20.570533+0000 mgr.y (mgr.14544) 704 : cluster [DBG] pgmap v1282: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:20.791090+0000 osd.2 (osd.2) 159 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:55:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:20.792880+0000 osd.2 (osd.2) 160 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:55:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:20.962176+0000 osd.0 (osd.0) 355 : cluster [DBG] 28.10 scrub starts 2024-02-13T00:55:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:20.962989+0000 osd.0 (osd.0) 356 : cluster [DBG] 28.10 scrub ok 2024-02-13T00:55:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:21.222510+0000 osd.5 (osd.5) 20 : cluster [DBG] 28.3 deep-scrub starts 2024-02-13T00:55:21.795 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:21.226046+0000 osd.5 (osd.5) 21 : cluster [DBG] 28.3 deep-scrub ok 2024-02-13T00:55:21.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.900+0000 7fd7defc9700 1 -- 172.21.15.162:0/1092009707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7d80f9b70 msgr2=0x7fd7d80f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:21.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.900+0000 7fd7defc9700 1 --2- 172.21.15.162:0/1092009707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7d80f9b70 0x7fd7d80f9f50 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7fd7cc004870 tx=0x7fd7cc009cb0).stop 2024-02-13T00:55:21.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.900+0000 7fd7defc9700 1 -- 172.21.15.162:0/1092009707 shutdown_connections 2024-02-13T00:55:21.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.900+0000 7fd7defc9700 1 --2- 172.21.15.162:0/1092009707 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7d80f8500 0x7fd7d8104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:21.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.900+0000 7fd7defc9700 1 --2- 172.21.15.162:0/1092009707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7d80f9b70 0x7fd7d80f9f50 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:21.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.900+0000 7fd7defc9700 1 --2- 172.21.15.162:0/1092009707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7d80fa440 0x7fd7d80f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:21.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.900+0000 7fd7defc9700 1 -- 172.21.15.162:0/1092009707 >> 172.21.15.162:0/1092009707 conn(0x7fd7d80f38c0 msgr2=0x7fd7d80f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:21.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.900+0000 7fd7defc9700 1 -- 172.21.15.162:0/1092009707 shutdown_connections 2024-02-13T00:55:21.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.900+0000 7fd7defc9700 1 -- 172.21.15.162:0/1092009707 wait complete. 2024-02-13T00:55:21.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7defc9700 1 librados: starting msgr at 2024-02-13T00:55:21.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7defc9700 1 librados: starting objecter 2024-02-13T00:55:21.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7defc9700 1 Processor -- start 2024-02-13T00:55:21.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7defc9700 1 -- start start 2024-02-13T00:55:21.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7defc9700 1 librados: setting wanted keys 2024-02-13T00:55:21.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7defc9700 1 librados: calling monclient init 2024-02-13T00:55:21.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7defc9700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7d80f8500 0x7fd7d810e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:21.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7defc9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7d80f9b70 0x7fd7d810e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:21.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7defc9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7d80fa440 0x7fd7d8108460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:21.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7defc9700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d8108950 con 0x7fd7d80f9b70 2024-02-13T00:55:21.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7defc9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d8108ac0 con 0x7fd7d80fa440 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7defc9700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d8108c80 con 0x7fd7d80f8500 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7dcd65700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7d80f8500 0x7fd7d810e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7d7fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7d80f9b70 0x7fd7d810e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7dcd65700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7d80f8500 0x7fd7d810e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:55616/0 (socket says 172.21.15.162:55616) 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7d7fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7d80f9b70 0x7fd7d810e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:33878/0 (socket says 172.21.15.162:33878) 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7dcd65700 1 -- 172.21.15.162:0/3498377403 learned_addr learned my addr 172.21.15.162:0/3498377403 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7dd566700 1 --2- 172.21.15.162:0/3498377403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7d80fa440 0x7fd7d8108460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7dcd65700 1 -- 172.21.15.162:0/3498377403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7d80fa440 msgr2=0x7fd7d8108460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7dcd65700 1 --2- 172.21.15.162:0/3498377403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7d80fa440 0x7fd7d8108460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7dcd65700 1 -- 172.21.15.162:0/3498377403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7d80f9b70 msgr2=0x7fd7d810e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7dcd65700 1 --2- 172.21.15.162:0/3498377403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7d80f9b70 0x7fd7d810e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7dcd65700 1 -- 172.21.15.162:0/3498377403 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d00097c0 con 0x7fd7d80f8500 2024-02-13T00:55:21.908 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7d7fff700 1 --2- 172.21.15.162:0/3498377403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7d80f9b70 0x7fd7d810e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7dcd65700 1 --2- 172.21.15.162:0/3498377403 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7d80f8500 0x7fd7d810e2c0 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 rx=0x7fd7cc0043a0 tx=0x7fd7cc00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.901+0000 7fd7d5ffb700 10 client.?.objecter ms_handle_connect 0x7fd7d80f8500 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.902+0000 7fd7d5ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.902+0000 7fd7d5ffb700 1 -- 172.21.15.162:0/3498377403 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc00e070 con 0x7fd7d80f8500 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.902+0000 7fd7d5ffb700 1 -- 172.21.15.162:0/3498377403 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7cc009710 con 0x7fd7d80f8500 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.902+0000 7fd7defc9700 1 -- 172.21.15.162:0/3498377403 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7cc005040 con 0x7fd7d80f8500 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.902+0000 7fd7d5ffb700 1 -- 172.21.15.162:0/3498377403 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc019070 con 0x7fd7d80f8500 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.902+0000 7fd7defc9700 10 client.38366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.902+0000 7fd7defc9700 1 -- 172.21.15.162:0/3498377403 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d81091f0 con 0x7fd7d80f8500 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.902+0000 7fd7defc9700 1 librados: init done 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.903+0000 7fd7defc9700 1 -- 172.21.15.162:0/3498377403 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d80475e0 con 0x7fd7d80f8500 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.903+0000 7fd7d5ffb700 1 -- 172.21.15.162:0/3498377403 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fd7cc0090d0 con 0x7fd7d80f8500 2024-02-13T00:55:21.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.903+0000 7fd7d5ffb700 1 --2- 172.21.15.162:0/3498377403 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd7c8062c90 0x7fd7c8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:21.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.904+0000 7fd7d5ffb700 1 -- 172.21.15.162:0/3498377403 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(829..829 src has 449..829) v4 ==== 7426+0+0 (secure 0 0 0) 0x7fd7cc07d890 con 0x7fd7d80f8500 2024-02-13T00:55:21.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.904+0000 7fd7d5ffb700 10 client.38366.objecter ms_dispatch 0x7fd7d8001550 osd_map(829..829 src has 449..829) v4 2024-02-13T00:55:21.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.904+0000 7fd7d5ffb700 3 client.38366.objecter handle_osd_map got epochs [829,829] > 0 2024-02-13T00:55:21.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.904+0000 7fd7d5ffb700 3 client.38366.objecter handle_osd_map decoding full epoch 829 2024-02-13T00:55:21.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.904+0000 7fd7d7fff700 1 --2- 172.21.15.162:0/3498377403 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd7c8062c90 0x7fd7c8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:21.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.904+0000 7fd7d5ffb700 20 client.38366.objecter dump_active .. 0 homeless 2024-02-13T00:55:21.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.904+0000 7fd7d5ffb700 10 client.38366.objecter ms_handle_connect 0x7fd7c8062c90 2024-02-13T00:55:21.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.904+0000 7fd7d7fff700 1 --2- 172.21.15.162:0/3498377403 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd7c8062c90 0x7fd7c8065150 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 rx=0x7fd7d8109c00 tx=0x7fd7c400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:21.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:21.908+0000 7fd7d5ffb700 1 -- 172.21.15.162:0/3498377403 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd7cc051080 con 0x7fd7d80f8500 2024-02-13T00:55:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:20.570533+0000 mgr.y (mgr.14544) 704 : cluster [DBG] pgmap v1282: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:20.791090+0000 osd.2 (osd.2) 159 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:55:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:20.792880+0000 osd.2 (osd.2) 160 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:55:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:20.962176+0000 osd.0 (osd.0) 355 : cluster [DBG] 28.10 scrub starts 2024-02-13T00:55:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:20.962989+0000 osd.0 (osd.0) 356 : cluster [DBG] 28.10 scrub ok 2024-02-13T00:55:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:21.222510+0000 osd.5 (osd.5) 20 : cluster [DBG] 28.3 deep-scrub starts 2024-02-13T00:55:22.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:21.226046+0000 osd.5 (osd.5) 21 : cluster [DBG] 28.3 deep-scrub ok 2024-02-13T00:55:22.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:20.570533+0000 mgr.y (mgr.14544) 704 : cluster [DBG] pgmap v1282: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:20.791090+0000 osd.2 (osd.2) 159 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:55:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:20.792880+0000 osd.2 (osd.2) 160 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:55:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:20.962176+0000 osd.0 (osd.0) 355 : cluster [DBG] 28.10 scrub starts 2024-02-13T00:55:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:20.962989+0000 osd.0 (osd.0) 356 : cluster [DBG] 28.10 scrub ok 2024-02-13T00:55:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:21.222510+0000 osd.5 (osd.5) 20 : cluster [DBG] 28.3 deep-scrub starts 2024-02-13T00:55:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:21.226046+0000 osd.5 (osd.5) 21 : cluster [DBG] 28.3 deep-scrub ok 2024-02-13T00:55:22.090 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.086+0000 7fd7defc9700 1 -- 172.21.15.162:0/3498377403 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fd7d8109940 con 0x7fd7d80f8500 2024-02-13T00:55:22.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-13T00:55:22.513 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T00:55:22.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.655+0000 7fd7d5ffb700 1 -- 172.21.15.162:0/3498377403 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v830) v1 ==== 113+0+0 (secure 0 0 0) 0x7fd7cc050c70 con 0x7fd7d80f8500 2024-02-13T00:55:22.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 10 librados: watch_flush enter 2024-02-13T00:55:22.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 10 librados: watch_flush exit 2024-02-13T00:55:22.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 20 client.38366.objecter shutdown clearing up homeless session... 2024-02-13T00:55:22.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 10 client.38366.objecter successfully canceled tick 2024-02-13T00:55:22.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 -- 172.21.15.162:0/3498377403 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd7c8062c90 msgr2=0x7fd7c8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:22.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 --2- 172.21.15.162:0/3498377403 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd7c8062c90 0x7fd7c8065150 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 rx=0x7fd7d8109c00 tx=0x7fd7c400b040).stop 2024-02-13T00:55:22.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 -- 172.21.15.162:0/3498377403 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7d80f8500 msgr2=0x7fd7d810e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:22.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 --2- 172.21.15.162:0/3498377403 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7d80f8500 0x7fd7d810e2c0 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 rx=0x7fd7cc0043a0 tx=0x7fd7cc00b040).stop 2024-02-13T00:55:22.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 -- 172.21.15.162:0/3498377403 shutdown_connections 2024-02-13T00:55:22.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 --2- 172.21.15.162:0/3498377403 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd7c8062c90 0x7fd7c8065150 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:22.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 --2- 172.21.15.162:0/3498377403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7d80fa440 0x7fd7d8108460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:22.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 --2- 172.21.15.162:0/3498377403 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7d80f8500 0x7fd7d810e2c0 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:22.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 --2- 172.21.15.162:0/3498377403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7d80f9b70 0x7fd7d810e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:22.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 -- 172.21.15.162:0/3498377403 >> 172.21.15.162:0/3498377403 conn(0x7fd7d80f38c0 msgr2=0x7fd7d80fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:22.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 -- 172.21.15.162:0/3498377403 shutdown_connections 2024-02-13T00:55:22.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 -- 172.21.15.162:0/3498377403 wait complete. 2024-02-13T00:55:22.663 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:22.658+0000 7fd7defc9700 1 librados: shutdown 2024-02-13T00:55:22.663 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.1 to 1 (10000) 2024-02-13T00:55:23.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:21.800935+0000 osd.2 (osd.2) 161 : cluster [DBG] 28.5 scrub starts 2024-02-13T00:55:23.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:21.802363+0000 osd.2 (osd.2) 162 : cluster [DBG] 28.5 scrub ok 2024-02-13T00:55:23.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:21.981865+0000 osd.1 (osd.1) 357 : cluster [DBG] 4.4 scrub starts 2024-02-13T00:55:23.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:21.985499+0000 osd.1 (osd.1) 358 : cluster [DBG] 4.4 scrub ok 2024-02-13T00:55:23.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:22.087533+0000 mon.c (mon.2) 525 : audit [INF] from='client.? 172.21.15.162:0/3498377403' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:55:23.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:22.087825+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:55:23.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:22.495340+0000 osd.6 (osd.6) 294 : cluster [DBG] 6.27 scrub starts 2024-02-13T00:55:23.015 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:22.496914+0000 osd.6 (osd.6) 295 : cluster [DBG] 6.27 scrub ok 2024-02-13T00:55:23.069 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:21.800935+0000 osd.2 (osd.2) 161 : cluster [DBG] 28.5 scrub starts 2024-02-13T00:55:23.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:21.802363+0000 osd.2 (osd.2) 162 : cluster [DBG] 28.5 scrub ok 2024-02-13T00:55:23.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:21.981865+0000 osd.1 (osd.1) 357 : cluster [DBG] 4.4 scrub starts 2024-02-13T00:55:23.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:21.985499+0000 osd.1 (osd.1) 358 : cluster [DBG] 4.4 scrub ok 2024-02-13T00:55:23.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:22.087533+0000 mon.c (mon.2) 525 : audit [INF] from='client.? 172.21.15.162:0/3498377403' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:55:23.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:22.087825+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:55:23.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:22.495340+0000 osd.6 (osd.6) 294 : cluster [DBG] 6.27 scrub starts 2024-02-13T00:55:23.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:22.496914+0000 osd.6 (osd.6) 295 : cluster [DBG] 6.27 scrub ok 2024-02-13T00:55:23.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:21.800935+0000 osd.2 (osd.2) 161 : cluster [DBG] 28.5 scrub starts 2024-02-13T00:55:23.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:21.802363+0000 osd.2 (osd.2) 162 : cluster [DBG] 28.5 scrub ok 2024-02-13T00:55:23.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:21.981865+0000 osd.1 (osd.1) 357 : cluster [DBG] 4.4 scrub starts 2024-02-13T00:55:23.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:21.985499+0000 osd.1 (osd.1) 358 : cluster [DBG] 4.4 scrub ok 2024-02-13T00:55:23.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:22.087533+0000 mon.c (mon.2) 525 : audit [INF] from='client.? 172.21.15.162:0/3498377403' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:55:23.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:22.087825+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:55:23.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:22.495340+0000 osd.6 (osd.6) 294 : cluster [DBG] 6.27 scrub starts 2024-02-13T00:55:23.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:22.496914+0000 osd.6 (osd.6) 295 : cluster [DBG] 6.27 scrub ok 2024-02-13T00:55:23.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 1 1 2024-02-13T00:55:23.664 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:22.571136+0000 mgr.y (mgr.14544) 705 : cluster [DBG] pgmap v1283: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 735 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:23.665 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:22.654734+0000 mon.a (mon.0) 3811 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:55:23.665 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:22.654831+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-02-13T00:55:23.665 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:23.028215+0000 osd.1 (osd.1) 359 : cluster [DBG] 6.1d deep-scrub starts 2024-02-13T00:55:23.665 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:23.031294+0000 osd.1 (osd.1) 360 : cluster [DBG] 6.1d deep-scrub ok 2024-02-13T00:55:23.665 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:23.185830+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.d scrub starts 2024-02-13T00:55:23.665 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:23.188912+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.d scrub ok 2024-02-13T00:55:23.710 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:23.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:22.571136+0000 mgr.y (mgr.14544) 705 : cluster [DBG] pgmap v1283: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 735 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:23.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:22.654734+0000 mon.a (mon.0) 3811 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:55:23.939 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:22.654831+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-02-13T00:55:23.940 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:23.028215+0000 osd.1 (osd.1) 359 : cluster [DBG] 6.1d deep-scrub starts 2024-02-13T00:55:23.940 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:23.031294+0000 osd.1 (osd.1) 360 : cluster [DBG] 6.1d deep-scrub ok 2024-02-13T00:55:23.940 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:23.185830+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.d scrub starts 2024-02-13T00:55:23.940 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:23.188912+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.d scrub ok 2024-02-13T00:55:24.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:22.571136+0000 mgr.y (mgr.14544) 705 : cluster [DBG] pgmap v1283: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 735 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:24.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:22.654734+0000 mon.a (mon.0) 3811 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:55:24.026 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:22.654831+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-02-13T00:55:24.027 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:23.028215+0000 osd.1 (osd.1) 359 : cluster [DBG] 6.1d deep-scrub starts 2024-02-13T00:55:24.027 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:23.031294+0000 osd.1 (osd.1) 360 : cluster [DBG] 6.1d deep-scrub ok 2024-02-13T00:55:24.027 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:23.185830+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.d scrub starts 2024-02-13T00:55:24.027 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:23.188912+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.d scrub ok 2024-02-13T00:55:24.326 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(6.e 813/812 [PushOp(6:76044a46:users.keys::QNCTCPBB37LWF51SCA76:head, version: 64'1, data_included: [0~13], data_size: 13, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:76044a46:users.keys::QNCTCPBB37LWF51SCA76:head@64'1, size: 13, copy_subset: [0~13], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:13, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:52.757331+0000", 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.564513281, 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000762621, 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.757331+0000", 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.757331+0000", 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.757330+0000", 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0260000000000007e-06 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.757338+0000", 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6639999999999993e-06 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.757348+0000", 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0642000000000001e-05 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.757359+0000", 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00053056900000000005 2024-02-13T00:55:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.757889+0000", 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7240000000000003e-06 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.757896+0000", 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019735299999999999 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:52.758093+0000", 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000762621 2024-02-13T00:55:24.330 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.1 813/812 [PushOp(4:856b9b26:reshard::reshard.0000000002:head, version: 467'22, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:856b9b26:reshard::reshard.0000000002:head@467'22, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:53.119293+0000", 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.202550658, 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049505800000000004, 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119293+0000", 2024-02-13T00:55:24.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119293+0000", 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119293+0000", 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3828999999999999e-05 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119307+0000", 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3774999999999999e-05 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119320+0000", 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0009e-05 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119330+0000", 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8019e-05 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119398+0000", 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1610000000000003e-06 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119407+0000", 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00038175600000000002 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.119788+0000", 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049505800000000004 2024-02-13T00:55:24.334 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(2.b 813/812 [PushOp(2:d6b4bb0f:::zone_info.65231d82-160b-42d4-a07d-27856f74fab0:head, version: 57'1, data_included: [0~873], data_size: 873, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d6b4bb0f:::zone_info.65231d82-160b-42d4-a07d-27856f74fab0:head@57'1, size: 873, copy_subset: [0~873], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:873, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:53.122524+0000", 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.199319536000001, 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050201399999999995, 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.122524+0000", 2024-02-13T00:55:24.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.122524+0000", 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.122524+0000", 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3033e-05 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.122537+0000", 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8448999999999999e-05 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.122555+0000", 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0473e-05 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.122566+0000", 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4986000000000004e-05 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.122631+0000", 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4959999999999999e-06 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.122638+0000", 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00038829899999999999 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.123026+0000", 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050201399999999995 2024-02-13T00:55:24.338 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.1 813/812 [PushOp(4:80b6fe23:reshard::reshard.0000000011:head, version: 467'24, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:80b6fe23:reshard::reshard.0000000011:head@467'24, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:53.291870+0000", 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.029973501000001, 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050332399999999998, 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.291870+0000", 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.291870+0000", 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.291870+0000", 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5006e-05 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.291885+0000", 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.349e-05 2024-02-13T00:55:24.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.291898+0000", 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5400000000000001e-06 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.291908+0000", 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4746000000000007e-05 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.291972+0000", 2024-02-13T00:55:24.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6279999999999996e-06 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.291980+0000", 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039372099999999998 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.292374+0000", 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050332399999999998 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.1 814/812 [PushOp(4:8411e274:::obj_delete_at_hint.0000000073:head, version: 467'30, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8411e274:::obj_delete_at_hint.0000000073:head@467'30, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:53.807583+0000", 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 30.514261422000001, 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049342600000000004, 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.807583+0000", 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.807583+0000", 2024-02-13T00:55:24.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:24.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.807582+0000", 2024-02-13T00:55:24.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1131000000000001e-05 2024-02-13T00:55:24.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.807593+0000", 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4384e-05 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.807607+0000", 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0733999999999999e-05 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.807618+0000", 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2181000000000007e-05 2024-02-13T00:55:24.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.807680+0000", 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8600000000000001e-06 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.807685+0000", 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039101300000000001 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:53.808076+0000", 2024-02-13T00:55:24.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049342600000000004 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:2459 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94514175191296 gen 3] snapc 0=[] ondisk+write+known_if_redirected e814)", 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:54.409781+0000", 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.912062619, 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001596283, 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:55:24.347 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2459 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409781+0000", 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409781+0000", 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999981 2024-02-13T00:55:24.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409780+0000", 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8860000000000001e-06 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409785+0000", 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0799999999999995e-06 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409795+0000", 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0909999999999998e-06 2024-02-13T00:55:24.349 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409801+0000", 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6499000000000002e-05 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409857+0000", 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1590999999999999e-05 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409929+0000", 2024-02-13T00:55:24.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001448852 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.411378+0000", 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001596283 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.0 epoch 814/812 rep_tid 1154 entries 814'12 (0'0) error 5:5bf5b680:::notify.1:head by client.24329.0:2458 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:54.409880+0000", 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.911963728, 2024-02-13T00:55:24.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056193900000000001, 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409880+0000", 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409880+0000", 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:24.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409879+0000", 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2515e-05 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409892+0000", 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4524e-05 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409906+0000", 2024-02-13T00:55:24.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5520000000000005e-06 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409915+0000", 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6338e-05 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.409991+0000", 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00045082400000000001 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410442+0000", 2024-02-13T00:55:24.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056193900000000001 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.e epoch 814/813 rep_tid 1985 entries 814'20 (0'0) error 5:7759931f:::notify.3:head by client.24329.0:2455 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:54.410006+0000", 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.911837441999999, 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00064983599999999999, 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410006+0000", 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410006+0000", 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410006+0000", 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2679999999999994e-06 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410014+0000", 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3713999999999999e-05 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410028+0000", 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2979999999999997e-06 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410037+0000", 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023446299999999999 2024-02-13T00:55:24.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410272+0000", 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00038474400000000001 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410656+0000", 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00064983599999999999 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.358 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.d epoch 814/813 rep_tid 1986 entries 814'20 (0'0) error 5:b4812045:::notify.4:head by client.24329.0:2456 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:54.410051+0000", 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.911792573, 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00060074100000000001, 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410051+0000", 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410051+0000", 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410051+0000", 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8829999999999997e-06 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410056+0000", 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3280000000000006e-06 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410064+0000", 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1840000000000004e-06 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410070+0000", 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013172600000000001 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410202+0000", 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044989299999999998 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410652+0000", 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00060074100000000001 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.f epoch 814/813 rep_tid 1987 entries 814'20 (0'0) error 5:f95f44c2:::notify.0:head by client.24329.0:2460 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:54.410081+0000", 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.911762831000001, 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057493400000000004, 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:24.362 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410081+0000", 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410081+0000", 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410081+0000", 2024-02-13T00:55:24.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4839999999999998e-06 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410085+0000", 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8800000000000008e-06 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410093+0000", 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2489999999999998e-06 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.364 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410099+0000", 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2237999999999996e-05 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410162+0000", 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049441800000000005 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410656+0000", 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057493400000000004 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.365 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.d epoch 814/813 rep_tid 1989 entries 814'21 (0'0) error 5:b4812045:::notify.4:head by client.24329.0:2462 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:54.410113+0000", 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.911731292999999, 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00064148500000000002, 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410113+0000", 2024-02-13T00:55:24.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410113+0000", 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410112+0000", 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7969999999999997e-06 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410118+0000", 2024-02-13T00:55:24.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3699999999999995e-06 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410126+0000", 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1580000000000002e-06 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410133+0000", 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016191799999999999 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410295+0000", 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00045959700000000001 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410754+0000", 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00064148500000000002 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.e epoch 814/813 rep_tid 1988 entries 814'21 (0'0) error 5:7759931f:::notify.3:head by client.24329.0:2461 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:54.410139+0000", 2024-02-13T00:55:24.369 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.911705057999999, 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061055199999999999, 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410139+0000", 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410139+0000", 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:24.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410139+0000", 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9369999999999997e-06 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410143+0000", 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8499999999999996e-06 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410149+0000", 2024-02-13T00:55:24.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8250000000000001e-06 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410155+0000", 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022827599999999999 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410383+0000", 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036596699999999997 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410749+0000", 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061055199999999999 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.f epoch 814/813 rep_tid 1990 entries 814'21 (0'0) error 5:f95f44c2:::notify.0:head by client.24329.0:2463 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:54.410165+0000", 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.911679267, 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049483299999999997, 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410165+0000", 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410165+0000", 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410164+0000", 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4580000000000001e-06 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410170+0000", 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3549999999999999e-06 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410177+0000", 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8669999999999999e-06 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410183+0000", 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1119999999999995e-05 2024-02-13T00:55:24.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410274+0000", 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000385363 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410659+0000", 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049483299999999997 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.3 epoch 814/812 rep_tid 1991 entries 814'15 (0'0) error 5:c609908c:::notify.5:head by client.24329.0:2457 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:54.410189+0000", 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.911654598999998, 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057263300000000002, 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410189+0000", 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410189+0000", 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410189+0000", 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.67e-06 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410193+0000", 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.455e-06 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410199+0000", 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4179999999999996e-06 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410205+0000", 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000237507 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410442+0000", 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031988600000000002 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:54.410762+0000", 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057263300000000002 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(4.b 814/812 [PushOp(4:d4f18cce:::obj_delete_at_hint.0000000107:head, version: 467'21, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:d4f18cce:::obj_delete_at_hint.0000000107:head@467'21, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:55.358747+0000", 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.963096729, 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050278400000000004, 2024-02-13T00:55:24.380 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:55.358747+0000", 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:55.358747+0000", 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999986 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:55.358746+0000", 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7679e-05 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:55.358773+0000", 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7119000000000002e-05 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:55.358810+0000", 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.95e-05 2024-02-13T00:55:24.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:55.358830+0000", 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022091299999999999 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:55.359051+0000", 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8741000000000001e-05 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:55.359080+0000", 2024-02-13T00:55:24.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017047199999999999 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:55.359250+0000", 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050278400000000004 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:2473 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94514175192448 gen 13] snapc 0=[] ondisk+write+known_if_redirected e817)", 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:59.410311+0000", 2024-02-13T00:55:24.384 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.911533315, 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018139269999999999, 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2473 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410311+0000", 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410311+0000", 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410310+0000", 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.049e-06 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410316+0000", 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0719e-05 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410326+0000", 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2829999999999998e-06 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410333+0000", 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5426999999999995e-05 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410398+0000", 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8027000000000004e-05 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410476+0000", 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001648341 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.412125+0000", 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018139269999999999 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:2474 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94514175193600 gen 14] snapc 0=[] ondisk+write+known_if_redirected e817)", 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:59.410345+0000", 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.911498723000001, 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016975860000000001, 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2474 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.389 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410345+0000", 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410345+0000", 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410345+0000", 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0680000000000001e-06 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410348+0000", 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2764e-05 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410360+0000", 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8180000000000003e-06 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410365+0000", 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6542000000000003e-05 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410412+0000", 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2169000000000006e-05 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410494+0000", 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015487879999999999 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.412043+0000", 2024-02-13T00:55:24.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016975860000000001 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:2479 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94514175192448 gen 14] snapc 0=[] ondisk+write+known_if_redirected e817)", 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:59.410376+0000", 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.911468312, 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001935988, 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:55:24.393 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2479 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410376+0000", 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410376+0000", 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.9999995 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410375+0000", 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1351e-05 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410386+0000", 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6610000000000006e-06 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410395+0000", 2024-02-13T00:55:24.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5860000000000005e-06 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410404+0000", 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050053800000000002 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410904+0000", 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0192000000000007e-05 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:24.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410994+0000", 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013171630000000001 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.412312+0000", 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001935988 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "pg_update_log_missing(5.f epoch 817/815 rep_tid 1992 entries 817'22 (0'0) error 5:f95f44c2:::notify.0:head by client.24329.0:2475 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:54:59.410793+0000", 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.911051117, 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00082162499999999998, 2024-02-13T00:55:24.397 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410793+0000", 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410793+0000", 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410792+0000", 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0597e-05 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410802+0000", 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2843000000000001e-05 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410815+0000", 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6160000000000002e-06 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410824+0000", 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3635999999999997e-05 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.410908+0000", 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00070690300000000005 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:54:59.411614+0000", 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00082162499999999998 2024-02-13T00:55:24.400 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24329.0:2493 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94514175192448 gen 15] snapc 0=[] ondisk+write+known_if_redirected e820)", 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-02-13T00:55:04.411349+0000", 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.910495363999999, 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001053114, 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24329", 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.162:0/3408777347", 2024-02-13T00:55:24.401 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 2493 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411349+0000", 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411349+0000", 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4294967295.999999 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411348+0000", 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0590000000000002e-06 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411353+0000", 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8679999999999994e-06 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411363+0000", 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7230000000000004e-06 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411367+0000", 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0083000000000002e-05 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411407+0000", 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0179000000000006e-05 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.411488+0000", 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00091413599999999996 2024-02-13T00:55:24.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-02-13T00:55:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-02-13T00:55:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-02-13T00:55:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-02-13T00:55:04.412402+0000", 2024-02-13T00:55:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001053114 2024-02-13T00:55:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-02-13T00:55:24.405 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-02-13T00:55:24.405 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-02-13T00:55:24.977 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:23.788637+0000 osd.2 (osd.2) 163 : cluster [DBG] 6.29 scrub starts 2024-02-13T00:55:24.977 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:23.791548+0000 osd.2 (osd.2) 164 : cluster [DBG] 6.29 scrub ok 2024-02-13T00:55:24.977 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:24.007861+0000 osd.0 (osd.0) 357 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:55:24.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:24.008637+0000 osd.0 (osd.0) 358 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:55:24.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:24.012408+0000 osd.1 (osd.1) 361 : cluster [DBG] 6.20 scrub starts 2024-02-13T00:55:24.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:24.015300+0000 osd.1 (osd.1) 362 : cluster [DBG] 6.20 scrub ok 2024-02-13T00:55:25.008 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:23.788637+0000 osd.2 (osd.2) 163 : cluster [DBG] 6.29 scrub starts 2024-02-13T00:55:25.008 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:23.791548+0000 osd.2 (osd.2) 164 : cluster [DBG] 6.29 scrub ok 2024-02-13T00:55:25.008 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:24.007861+0000 osd.0 (osd.0) 357 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:55:25.008 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:24.008637+0000 osd.0 (osd.0) 358 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:55:25.008 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:24.012408+0000 osd.1 (osd.1) 361 : cluster [DBG] 6.20 scrub starts 2024-02-13T00:55:25.009 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:24.015300+0000 osd.1 (osd.1) 362 : cluster [DBG] 6.20 scrub ok 2024-02-13T00:55:25.009 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:23.788637+0000 osd.2 (osd.2) 163 : cluster [DBG] 6.29 scrub starts 2024-02-13T00:55:25.009 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:23.791548+0000 osd.2 (osd.2) 164 : cluster [DBG] 6.29 scrub ok 2024-02-13T00:55:25.009 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:24.007861+0000 osd.0 (osd.0) 357 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:55:25.009 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:24.008637+0000 osd.0 (osd.0) 358 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:55:25.009 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:24.012408+0000 osd.1 (osd.1) 361 : cluster [DBG] 6.20 scrub starts 2024-02-13T00:55:25.009 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:24.015300+0000 osd.1 (osd.1) 362 : cluster [DBG] 6.20 scrub ok 2024-02-13T00:55:25.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.503+0000 7f82b9234700 1 -- 172.21.15.162:0/2403550039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82b40f73a0 msgr2=0x7f82b41039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.503+0000 7f82b9234700 1 --2- 172.21.15.162:0/2403550039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82b40f73a0 0x7f82b41039a0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7f82ac009c30 tx=0x7f82ac004c50).stop 2024-02-13T00:55:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.503+0000 7f82b9234700 1 -- 172.21.15.162:0/2403550039 shutdown_connections 2024-02-13T00:55:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.503+0000 7f82b9234700 1 --2- 172.21.15.162:0/2403550039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82b40f73a0 0x7f82b41039a0 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.503+0000 7f82b9234700 1 --2- 172.21.15.162:0/2403550039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82b40fcb30 0x7f82b40fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.503+0000 7f82b9234700 1 --2- 172.21.15.162:0/2403550039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82b40fd490 0x7f82b40f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.503+0000 7f82b9234700 1 -- 172.21.15.162:0/2403550039 >> 172.21.15.162:0/2403550039 conn(0x7f82b406eb90 msgr2=0x7f82b40f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.503+0000 7f82b9234700 1 -- 172.21.15.162:0/2403550039 shutdown_connections 2024-02-13T00:55:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.503+0000 7f82b9234700 1 -- 172.21.15.162:0/2403550039 wait complete. 2024-02-13T00:55:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.503+0000 7f82b9234700 1 librados: starting msgr at 2024-02-13T00:55:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.503+0000 7f82b9234700 1 librados: starting objecter 2024-02-13T00:55:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.503+0000 7f82b9234700 1 Processor -- start 2024-02-13T00:55:25.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b9234700 1 -- start start 2024-02-13T00:55:25.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b9234700 1 librados: setting wanted keys 2024-02-13T00:55:25.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b9234700 1 librados: calling monclient init 2024-02-13T00:55:25.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b9234700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82b40f73a0 0x7f82b40f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:25.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b9234700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82b40fcb30 0x7f82b40f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:25.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b9234700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82b40fd490 0x7f82b40f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:25.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b9234700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b4116400 con 0x7f82b40f73a0 2024-02-13T00:55:25.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b9234700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b4116570 con 0x7f82b40fd490 2024-02-13T00:55:25.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b9234700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b4116730 con 0x7f82b40fcb30 2024-02-13T00:55:25.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b2d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82b40f73a0 0x7f82b40f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:25.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b259c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82b40fcb30 0x7f82b40f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:25.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b2d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82b40f73a0 0x7f82b40f8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:40002/0 (socket says 172.21.15.162:40002) 2024-02-13T00:55:25.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b2d9d700 1 -- 172.21.15.162:0/2403611963 learned_addr learned my addr 172.21.15.162:0/2403611963 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:25.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b359e700 1 --2- 172.21.15.162:0/2403611963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82b40fd490 0x7f82b40f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:25.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b2d9d700 1 -- 172.21.15.162:0/2403611963 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82b40fcb30 msgr2=0x7f82b40f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:25.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b2d9d700 1 --2- 172.21.15.162:0/2403611963 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82b40fcb30 0x7f82b40f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:25.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b2d9d700 1 -- 172.21.15.162:0/2403611963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82b40fd490 msgr2=0x7f82b40f9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b2d9d700 1 --2- 172.21.15.162:0/2403611963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82b40fd490 0x7f82b40f9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b2d9d700 1 -- 172.21.15.162:0/2403611963 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a8005040 con 0x7f82b40f73a0 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b259c700 1 --2- 172.21.15.162:0/2403611963 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82b40fcb30 0x7f82b40f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f82b2d9d700 1 --2- 172.21.15.162:0/2403611963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82b40f73a0 0x7f82b40f8930 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7f82a8009400 tx=0x7f82a8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f829bfff700 10 client.?.objecter ms_handle_connect 0x7f82b40f73a0 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.504+0000 7f829bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.505+0000 7f829bfff700 1 -- 172.21.15.162:0/2403611963 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8007a40 con 0x7f82b40f73a0 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.505+0000 7f82b9234700 1 -- 172.21.15.162:0/2403611963 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac0097c0 con 0x7f82b40f73a0 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.505+0000 7f829bfff700 1 -- 172.21.15.162:0/2403611963 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82a800f040 con 0x7f82b40f73a0 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.505+0000 7f829bfff700 1 -- 172.21.15.162:0/2403611963 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8014aa0 con 0x7f82b40f73a0 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.505+0000 7f82b9234700 10 client.28278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.505+0000 7f82b9234700 1 -- 172.21.15.162:0/2403611963 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b4116e40 con 0x7f82b40f73a0 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.505+0000 7f82b9234700 1 librados: init done 2024-02-13T00:55:25.512 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.506+0000 7f82b9234700 1 -- 172.21.15.162:0/2403611963 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b40475e0 con 0x7f82b40f73a0 2024-02-13T00:55:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.506+0000 7f829bfff700 1 -- 172.21.15.162:0/2403611963 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f82a800e630 con 0x7f82b40f73a0 2024-02-13T00:55:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.506+0000 7f829bfff700 1 --2- 172.21.15.162:0/2403611963 >> v2:172.21.15.162:6800/2866534053 conn(0x7f829c062c90 0x7f829c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.506+0000 7f829bfff700 1 -- 172.21.15.162:0/2403611963 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(830..830 src has 449..830) v4 ==== 7426+0+0 (secure 0 0 0) 0x7f82a807cb60 con 0x7f82b40f73a0 2024-02-13T00:55:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.506+0000 7f829bfff700 10 client.28278.objecter ms_dispatch 0x7f82b4001550 osd_map(830..830 src has 449..830) v4 2024-02-13T00:55:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.506+0000 7f829bfff700 3 client.28278.objecter handle_osd_map got epochs [830,830] > 0 2024-02-13T00:55:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.506+0000 7f829bfff700 3 client.28278.objecter handle_osd_map decoding full epoch 830 2024-02-13T00:55:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.506+0000 7f82b259c700 1 --2- 172.21.15.162:0/2403611963 >> v2:172.21.15.162:6800/2866534053 conn(0x7f829c062c90 0x7f829c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.507+0000 7f829bfff700 20 client.28278.objecter dump_active .. 0 homeless 2024-02-13T00:55:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.507+0000 7f829bfff700 10 client.28278.objecter ms_handle_connect 0x7f829c062c90 2024-02-13T00:55:25.513 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.507+0000 7f82b259c700 1 --2- 172.21.15.162:0/2403611963 >> v2:172.21.15.162:6800/2866534053 conn(0x7f829c062c90 0x7f829c065150 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 rx=0x7f82b40fa0d0 tx=0x7f82a400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:25.515 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.512+0000 7f829bfff700 1 -- 172.21.15.162:0/2403611963 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f82a8049f60 con 0x7f82b40f73a0 2024-02-13T00:55:25.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:25.701+0000 7f82b9234700 1 -- 172.21.15.162:0/2403611963 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f82b40f9e10 con 0x7f82b40f73a0 2024-02-13T00:55:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:24.571700+0000 mgr.y (mgr.14544) 706 : cluster [DBG] pgmap v1285: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 737 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:25.255098+0000 osd.5 (osd.5) 24 : cluster [DBG] 28.17 scrub starts 2024-02-13T00:55:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:25.259004+0000 osd.5 (osd.5) 25 : cluster [DBG] 28.17 scrub ok 2024-02-13T00:55:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:25.538502+0000 osd.3 (osd.3) 183 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:55:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:25.541499+0000 osd.3 (osd.3) 184 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:55:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:25.549683+0000 osd.6 (osd.6) 296 : cluster [DBG] 6.23 scrub starts 2024-02-13T00:55:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:25.552646+0000 osd.6 (osd.6) 297 : cluster [DBG] 6.23 scrub ok 2024-02-13T00:55:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:25.607896+0000 mon.a (mon.0) 3813 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-02-13T00:55:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:24.571700+0000 mgr.y (mgr.14544) 706 : cluster [DBG] pgmap v1285: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 737 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:25.255098+0000 osd.5 (osd.5) 24 : cluster [DBG] 28.17 scrub starts 2024-02-13T00:55:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:25.259004+0000 osd.5 (osd.5) 25 : cluster [DBG] 28.17 scrub ok 2024-02-13T00:55:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:25.538502+0000 osd.3 (osd.3) 183 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:55:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:25.541499+0000 osd.3 (osd.3) 184 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:55:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:25.549683+0000 osd.6 (osd.6) 296 : cluster [DBG] 6.23 scrub starts 2024-02-13T00:55:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:25.552646+0000 osd.6 (osd.6) 297 : cluster [DBG] 6.23 scrub ok 2024-02-13T00:55:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:25.607896+0000 mon.a (mon.0) 3813 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-02-13T00:55:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:24.571700+0000 mgr.y (mgr.14544) 706 : cluster [DBG] pgmap v1285: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 737 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:25.255098+0000 osd.5 (osd.5) 24 : cluster [DBG] 28.17 scrub starts 2024-02-13T00:55:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:25.259004+0000 osd.5 (osd.5) 25 : cluster [DBG] 28.17 scrub ok 2024-02-13T00:55:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:25.538502+0000 osd.3 (osd.3) 183 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:55:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:25.541499+0000 osd.3 (osd.3) 184 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:55:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:25.549683+0000 osd.6 (osd.6) 296 : cluster [DBG] 6.23 scrub starts 2024-02-13T00:55:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:25.552646+0000 osd.6 (osd.6) 297 : cluster [DBG] 6.23 scrub ok 2024-02-13T00:55:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:25.607896+0000 mon.a (mon.0) 3813 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-02-13T00:55:26.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:55:26.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:55:26] "GET /metrics HTTP/1.1" 200 185509 "" "Prometheus/2.33.4" 2024-02-13T00:55:26.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.615+0000 7f829bfff700 1 -- 172.21.15.162:0/2403611963 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v832) v1 ==== 133+0+0 (secure 0 0 0) 0x7f82a80502e0 con 0x7f82b40f73a0 2024-02-13T00:55:26.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.617+0000 7f82b9234700 10 librados: watch_flush enter 2024-02-13T00:55:26.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.617+0000 7f82b9234700 10 librados: watch_flush exit 2024-02-13T00:55:26.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.617+0000 7f82b9234700 20 client.28278.objecter shutdown clearing up homeless session... 2024-02-13T00:55:26.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.617+0000 7f82b9234700 10 client.28278.objecter successfully canceled tick 2024-02-13T00:55:26.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 -- 172.21.15.162:0/2403611963 >> v2:172.21.15.162:6800/2866534053 conn(0x7f829c062c90 msgr2=0x7f829c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:26.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 --2- 172.21.15.162:0/2403611963 >> v2:172.21.15.162:6800/2866534053 conn(0x7f829c062c90 0x7f829c065150 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 rx=0x7f82b40fa0d0 tx=0x7f82a400b040).stop 2024-02-13T00:55:26.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 -- 172.21.15.162:0/2403611963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82b40f73a0 msgr2=0x7f82b40f8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:26.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 --2- 172.21.15.162:0/2403611963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82b40f73a0 0x7f82b40f8930 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7f82a8009400 tx=0x7f82a8009cb0).stop 2024-02-13T00:55:26.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 -- 172.21.15.162:0/2403611963 shutdown_connections 2024-02-13T00:55:26.622 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 --2- 172.21.15.162:0/2403611963 >> v2:172.21.15.162:6800/2866534053 conn(0x7f829c062c90 0x7f829c065150 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:26.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 --2- 172.21.15.162:0/2403611963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82b40fd490 0x7f82b40f9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:26.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 --2- 172.21.15.162:0/2403611963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82b40f73a0 0x7f82b40f8930 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:26.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 --2- 172.21.15.162:0/2403611963 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82b40fcb30 0x7f82b40f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:26.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 -- 172.21.15.162:0/2403611963 >> 172.21.15.162:0/2403611963 conn(0x7f82b406eb90 msgr2=0x7f82b4060e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:26.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 -- 172.21.15.162:0/2403611963 shutdown_connections 2024-02-13T00:55:26.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 -- 172.21.15.162:0/2403611963 wait complete. 2024-02-13T00:55:26.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:26.618+0000 7f82b9234700 1 librados: shutdown 2024-02-13T00:55:26.623 INFO:teuthology.orchestra.run.smithi162.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-02-13T00:55:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:25.703325+0000 mon.a (mon.0) 3814 : audit [INF] from='client.? 172.21.15.162:0/2403611963' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:55:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:25.748228+0000 osd.2 (osd.2) 165 : cluster [DBG] 6.21 scrub starts 2024-02-13T00:55:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:25.751440+0000 osd.2 (osd.2) 166 : cluster [DBG] 6.21 scrub ok 2024-02-13T00:55:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:25.968992+0000 mon.c (mon.2) 526 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:55:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:26.042470+0000 mgr.y (mgr.14544) 707 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:26.242816+0000 osd.5 (osd.5) 26 : cluster [DBG] 2.2 deep-scrub starts 2024-02-13T00:55:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:26.245718+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.2 deep-scrub ok 2024-02-13T00:55:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:26.264011+0000 mon.c (mon.2) 527 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:55:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:26.264673+0000 mon.c (mon.2) 528 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:55:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:26.271646+0000 mon.a (mon.0) 3815 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:27.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:26.616084+0000 mon.a (mon.0) 3816 : audit [INF] from='client.? 172.21.15.162:0/2403611963' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:55:27.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:26.616156+0000 mon.a (mon.0) 3817 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-02-13T00:55:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:25.703325+0000 mon.a (mon.0) 3814 : audit [INF] from='client.? 172.21.15.162:0/2403611963' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:55:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:25.748228+0000 osd.2 (osd.2) 165 : cluster [DBG] 6.21 scrub starts 2024-02-13T00:55:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:25.751440+0000 osd.2 (osd.2) 166 : cluster [DBG] 6.21 scrub ok 2024-02-13T00:55:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:25.968992+0000 mon.c (mon.2) 526 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:55:27.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:26.042470+0000 mgr.y (mgr.14544) 707 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:27.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:26.242816+0000 osd.5 (osd.5) 26 : cluster [DBG] 2.2 deep-scrub starts 2024-02-13T00:55:27.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:26.245718+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.2 deep-scrub ok 2024-02-13T00:55:27.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:26.264011+0000 mon.c (mon.2) 527 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:55:27.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:26.264673+0000 mon.c (mon.2) 528 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:55:27.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:26.271646+0000 mon.a (mon.0) 3815 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:27.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:26.616084+0000 mon.a (mon.0) 3816 : audit [INF] from='client.? 172.21.15.162:0/2403611963' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:55:27.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:26.616156+0000 mon.a (mon.0) 3817 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-02-13T00:55:27.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:25.703325+0000 mon.a (mon.0) 3814 : audit [INF] from='client.? 172.21.15.162:0/2403611963' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-13T00:55:27.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:25.748228+0000 osd.2 (osd.2) 165 : cluster [DBG] 6.21 scrub starts 2024-02-13T00:55:27.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:25.751440+0000 osd.2 (osd.2) 166 : cluster [DBG] 6.21 scrub ok 2024-02-13T00:55:27.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:25.968992+0000 mon.c (mon.2) 526 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:55:27.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:26.042470+0000 mgr.y (mgr.14544) 707 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:27.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:26.242816+0000 osd.5 (osd.5) 26 : cluster [DBG] 2.2 deep-scrub starts 2024-02-13T00:55:27.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:26.245718+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.2 deep-scrub ok 2024-02-13T00:55:27.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:26.264011+0000 mon.c (mon.2) 527 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:55:27.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:26.264673+0000 mon.c (mon.2) 528 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:55:27.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:26.271646+0000 mon.a (mon.0) 3815 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:27.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:26.616084+0000 mon.a (mon.0) 3816 : audit [INF] from='client.? 172.21.15.162:0/2403611963' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-13T00:55:27.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:26.616156+0000 mon.a (mon.0) 3817 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-02-13T00:55:27.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 2 1 2024-02-13T00:55:27.891 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:26.572152+0000 mgr.y (mgr.14544) 708 : cluster [DBG] pgmap v1287: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:26.998275+0000 osd.1 (osd.1) 363 : cluster [DBG] 6.1 scrub starts 2024-02-13T00:55:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:27.001355+0000 osd.1 (osd.1) 364 : cluster [DBG] 6.1 scrub ok 2024-02-13T00:55:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:26.572152+0000 mgr.y (mgr.14544) 708 : cluster [DBG] pgmap v1287: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:26.998275+0000 osd.1 (osd.1) 363 : cluster [DBG] 6.1 scrub starts 2024-02-13T00:55:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:27.001355+0000 osd.1 (osd.1) 364 : cluster [DBG] 6.1 scrub ok 2024-02-13T00:55:28.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:26.572152+0000 mgr.y (mgr.14544) 708 : cluster [DBG] pgmap v1287: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:28.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:26.998275+0000 osd.1 (osd.1) 363 : cluster [DBG] 6.1 scrub starts 2024-02-13T00:55:28.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:27.001355+0000 osd.1 (osd.1) 364 : cluster [DBG] 6.1 scrub ok 2024-02-13T00:55:29.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:55:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:55:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:55:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:27.768560+0000 osd.2 (osd.2) 167 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:55:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:27.771587+0000 osd.2 (osd.2) 168 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:55:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:27.999963+0000 osd.0 (osd.0) 359 : cluster [DBG] 4.3 deep-scrub starts 2024-02-13T00:55:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:28.004395+0000 osd.0 (osd.0) 360 : cluster [DBG] 4.3 deep-scrub ok 2024-02-13T00:55:29.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:27.768560+0000 osd.2 (osd.2) 167 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:55:29.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:27.771587+0000 osd.2 (osd.2) 168 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:55:29.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:27.999963+0000 osd.0 (osd.0) 359 : cluster [DBG] 4.3 deep-scrub starts 2024-02-13T00:55:29.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:28.004395+0000 osd.0 (osd.0) 360 : cluster [DBG] 4.3 deep-scrub ok 2024-02-13T00:55:29.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:27.768560+0000 osd.2 (osd.2) 167 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:55:29.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:27.771587+0000 osd.2 (osd.2) 168 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:55:29.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:27.999963+0000 osd.0 (osd.0) 359 : cluster [DBG] 4.3 deep-scrub starts 2024-02-13T00:55:29.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:28.004395+0000 osd.0 (osd.0) 360 : cluster [DBG] 4.3 deep-scrub ok 2024-02-13T00:55:29.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.855+0000 7fb2942cb700 1 -- 172.21.15.162:0/784008299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb28c0fd490 msgr2=0x7fb28c0f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:29.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.855+0000 7fb2942cb700 1 --2- 172.21.15.162:0/784008299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb28c0fd490 0x7fb28c0f6cc0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 rx=0x7fb288004870 tx=0x7fb288009cb0).stop 2024-02-13T00:55:29.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.855+0000 7fb2942cb700 1 -- 172.21.15.162:0/784008299 shutdown_connections 2024-02-13T00:55:29.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.855+0000 7fb2942cb700 1 --2- 172.21.15.162:0/784008299 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb28c0f73a0 0x7fb28c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:29.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.855+0000 7fb2942cb700 1 --2- 172.21.15.162:0/784008299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb28c0fcb30 0x7fb28c0fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:29.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.855+0000 7fb2942cb700 1 --2- 172.21.15.162:0/784008299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb28c0fd490 0x7fb28c0f6cc0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:29.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.855+0000 7fb2942cb700 1 -- 172.21.15.162:0/784008299 >> 172.21.15.162:0/784008299 conn(0x7fb28c06eb90 msgr2=0x7fb28c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:29.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.855+0000 7fb2942cb700 1 -- 172.21.15.162:0/784008299 shutdown_connections 2024-02-13T00:55:29.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.855+0000 7fb2942cb700 1 -- 172.21.15.162:0/784008299 wait complete. 2024-02-13T00:55:29.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb2942cb700 1 librados: starting msgr at 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb2942cb700 1 librados: starting objecter 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb2942cb700 1 Processor -- start 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb2942cb700 1 -- start start 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb2942cb700 1 librados: setting wanted keys 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb2942cb700 1 librados: calling monclient init 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb2942cb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb28c0f73a0 0x7fb28c10e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb2942cb700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb28c0fcb30 0x7fb28c10e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb2942cb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb28c0fd490 0x7fb28c1081d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb2942cb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c1086c0 con 0x7fb28c0fd490 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb2942cb700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c108830 con 0x7fb28c0f73a0 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb2942cb700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c1089f0 con 0x7fb28c0fcb30 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb291866700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb28c0fcb30 0x7fb28c10e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb292868700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb28c0fd490 0x7fb28c1081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb291866700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb28c0fcb30 0x7fb28c10e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:34490/0 (socket says 172.21.15.162:34490) 2024-02-13T00:55:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb291866700 1 -- 172.21.15.162:0/3321310586 learned_addr learned my addr 172.21.15.162:0/3321310586 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb292067700 1 --2- 172.21.15.162:0/3321310586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb28c0f73a0 0x7fb28c10e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb291866700 1 -- 172.21.15.162:0/3321310586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb28c0f73a0 msgr2=0x7fb28c10e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb291866700 1 --2- 172.21.15.162:0/3321310586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb28c0f73a0 0x7fb28c10e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb291866700 1 -- 172.21.15.162:0/3321310586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb28c0fd490 msgr2=0x7fb28c1081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb291866700 1 --2- 172.21.15.162:0/3321310586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb28c0fd490 0x7fb28c1081d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb291866700 1 -- 172.21.15.162:0/3321310586 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2840097c0 con 0x7fb28c0fcb30 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb292868700 1 --2- 172.21.15.162:0/3321310586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb28c0fd490 0x7fb28c1081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.856+0000 7fb291866700 1 --2- 172.21.15.162:0/3321310586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb28c0fcb30 0x7fb28c10e520 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 rx=0x7fb288009eb0 tx=0x7fb28800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.857+0000 7fb283779700 10 client.?.objecter ms_handle_connect 0x7fb28c0fcb30 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.857+0000 7fb283779700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.857+0000 7fb283779700 1 -- 172.21.15.162:0/3321310586 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28800e070 con 0x7fb28c0fcb30 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.857+0000 7fb283779700 1 -- 172.21.15.162:0/3321310586 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb288013da0 con 0x7fb28c0fcb30 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.857+0000 7fb2942cb700 1 -- 172.21.15.162:0/3321310586 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb288005040 con 0x7fb28c0fcb30 2024-02-13T00:55:29.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.857+0000 7fb283779700 1 -- 172.21.15.162:0/3321310586 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb288019070 con 0x7fb28c0fcb30 2024-02-13T00:55:29.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.857+0000 7fb2942cb700 10 client.38378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:29.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.857+0000 7fb2942cb700 1 -- 172.21.15.162:0/3321310586 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb28c108f60 con 0x7fb28c0fcb30 2024-02-13T00:55:29.864 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.857+0000 7fb2942cb700 1 librados: init done 2024-02-13T00:55:29.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.858+0000 7fb2942cb700 1 -- 172.21.15.162:0/3321310586 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28c0475e0 con 0x7fb28c0fcb30 2024-02-13T00:55:29.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.858+0000 7fb283779700 1 -- 172.21.15.162:0/3321310586 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fb28801e420 con 0x7fb28c0fcb30 2024-02-13T00:55:29.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.858+0000 7fb283779700 1 --2- 172.21.15.162:0/3321310586 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb278062c90 0x7fb278065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:29.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.859+0000 7fb283779700 1 -- 172.21.15.162:0/3321310586 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(833..833 src has 449..833) v4 ==== 7393+0+0 (secure 0 0 0) 0x7fb28807db10 con 0x7fb28c0fcb30 2024-02-13T00:55:29.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.859+0000 7fb283779700 10 client.38378.objecter ms_dispatch 0x7fb28c001550 osd_map(833..833 src has 449..833) v4 2024-02-13T00:55:29.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.859+0000 7fb283779700 3 client.38378.objecter handle_osd_map got epochs [833,833] > 0 2024-02-13T00:55:29.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.859+0000 7fb283779700 3 client.38378.objecter handle_osd_map decoding full epoch 833 2024-02-13T00:55:29.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.859+0000 7fb292067700 1 --2- 172.21.15.162:0/3321310586 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb278062c90 0x7fb278065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:29.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.859+0000 7fb283779700 20 client.38378.objecter dump_active .. 0 homeless 2024-02-13T00:55:29.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.859+0000 7fb283779700 10 client.38378.objecter ms_handle_connect 0x7fb278062c90 2024-02-13T00:55:29.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.859+0000 7fb292067700 1 --2- 172.21.15.162:0/3321310586 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb278062c90 0x7fb278065150 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 rx=0x7fb27c004b60 tx=0x7fb27c009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:29.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:29.863+0000 7fb283779700 1 -- 172.21.15.162:0/3321310586 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb2880512b0 con 0x7fb28c0fcb30 2024-02-13T00:55:30.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.038+0000 7fb2942cb700 1 -- 172.21.15.162:0/3321310586 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fb28c109730 con 0x7fb28c0fcb30 2024-02-13T00:55:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:28.572819+0000 mgr.y (mgr.14544) 709 : cluster [DBG] pgmap v1289: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:28.955842+0000 osd.0 (osd.0) 361 : cluster [DBG] 5.f scrub starts 2024-02-13T00:55:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:28.959316+0000 osd.0 (osd.0) 362 : cluster [DBG] 5.f scrub ok 2024-02-13T00:55:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:29.197427+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:55:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:29.200451+0000 osd.5 (osd.5) 29 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:55:30.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:29.619531+0000 osd.6 (osd.6) 298 : cluster [DBG] 28.f scrub starts 2024-02-13T00:55:30.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:29.622297+0000 osd.6 (osd.6) 299 : cluster [DBG] 28.f scrub ok 2024-02-13T00:55:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:28.572819+0000 mgr.y (mgr.14544) 709 : cluster [DBG] pgmap v1289: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:28.955842+0000 osd.0 (osd.0) 361 : cluster [DBG] 5.f scrub starts 2024-02-13T00:55:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:28.959316+0000 osd.0 (osd.0) 362 : cluster [DBG] 5.f scrub ok 2024-02-13T00:55:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:29.197427+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:55:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:29.200451+0000 osd.5 (osd.5) 29 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:55:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:29.619531+0000 osd.6 (osd.6) 298 : cluster [DBG] 28.f scrub starts 2024-02-13T00:55:30.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:29.622297+0000 osd.6 (osd.6) 299 : cluster [DBG] 28.f scrub ok 2024-02-13T00:55:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:28.572819+0000 mgr.y (mgr.14544) 709 : cluster [DBG] pgmap v1289: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:28.955842+0000 osd.0 (osd.0) 361 : cluster [DBG] 5.f scrub starts 2024-02-13T00:55:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:28.959316+0000 osd.0 (osd.0) 362 : cluster [DBG] 5.f scrub ok 2024-02-13T00:55:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:29.197427+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:55:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:29.200451+0000 osd.5 (osd.5) 29 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:55:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:29.619531+0000 osd.6 (osd.6) 298 : cluster [DBG] 28.f scrub starts 2024-02-13T00:55:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:29.622297+0000 osd.6 (osd.6) 299 : cluster [DBG] 28.f scrub ok 2024-02-13T00:55:30.734 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.730+0000 7fb283779700 1 -- 172.21.15.162:0/3321310586 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v834) v1 ==== 113+0+0 (secure 0 0 0) 0x7fb288023020 con 0x7fb28c0fcb30 2024-02-13T00:55:30.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 10 librados: watch_flush enter 2024-02-13T00:55:30.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 10 librados: watch_flush exit 2024-02-13T00:55:30.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 20 client.38378.objecter shutdown clearing up homeless session... 2024-02-13T00:55:30.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 10 client.38378.objecter successfully canceled tick 2024-02-13T00:55:30.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 -- 172.21.15.162:0/3321310586 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb278062c90 msgr2=0x7fb278065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:30.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 --2- 172.21.15.162:0/3321310586 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb278062c90 0x7fb278065150 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 rx=0x7fb27c004b60 tx=0x7fb27c009040).stop 2024-02-13T00:55:30.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 -- 172.21.15.162:0/3321310586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb28c0fcb30 msgr2=0x7fb28c10e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:30.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 --2- 172.21.15.162:0/3321310586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb28c0fcb30 0x7fb28c10e520 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 rx=0x7fb288009eb0 tx=0x7fb28800b040).stop 2024-02-13T00:55:30.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 -- 172.21.15.162:0/3321310586 shutdown_connections 2024-02-13T00:55:30.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 --2- 172.21.15.162:0/3321310586 >> v2:172.21.15.162:6800/2866534053 conn(0x7fb278062c90 0x7fb278065150 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:30.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 --2- 172.21.15.162:0/3321310586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb28c0fd490 0x7fb28c1081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:30.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 --2- 172.21.15.162:0/3321310586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb28c0f73a0 0x7fb28c10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:30.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 --2- 172.21.15.162:0/3321310586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb28c0fcb30 0x7fb28c10e520 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:30.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 -- 172.21.15.162:0/3321310586 >> 172.21.15.162:0/3321310586 conn(0x7fb28c06eb90 msgr2=0x7fb28c060e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:30.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 -- 172.21.15.162:0/3321310586 shutdown_connections 2024-02-13T00:55:30.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 -- 172.21.15.162:0/3321310586 wait complete. 2024-02-13T00:55:30.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:30.732+0000 7fb2942cb700 1 librados: shutdown 2024-02-13T00:55:30.740 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.2 to 1 (10000) 2024-02-13T00:55:31.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:29.723588+0000 mon.a (mon.0) 3818 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-02-13T00:55:31.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:29.838631+0000 osd.2 (osd.2) 169 : cluster [DBG] 6.f deep-scrub starts 2024-02-13T00:55:31.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:29.843119+0000 osd.2 (osd.2) 170 : cluster [DBG] 6.f deep-scrub ok 2024-02-13T00:55:31.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:30.039700+0000 mon.c (mon.2) 529 : audit [INF] from='client.? 172.21.15.162:0/3321310586' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-13T00:55:31.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:30.039953+0000 mon.a (mon.0) 3819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-13T00:55:31.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:30.618976+0000 osd.6 (osd.6) 300 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:55:31.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:30.622017+0000 osd.6 (osd.6) 301 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:55:31.026 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:29.723588+0000 mon.a (mon.0) 3818 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-02-13T00:55:31.027 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:29.838631+0000 osd.2 (osd.2) 169 : cluster [DBG] 6.f deep-scrub starts 2024-02-13T00:55:31.027 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:29.843119+0000 osd.2 (osd.2) 170 : cluster [DBG] 6.f deep-scrub ok 2024-02-13T00:55:31.027 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:30.039700+0000 mon.c (mon.2) 529 : audit [INF] from='client.? 172.21.15.162:0/3321310586' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-13T00:55:31.027 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:30.039953+0000 mon.a (mon.0) 3819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-13T00:55:31.027 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:30.618976+0000 osd.6 (osd.6) 300 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:55:31.027 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:30.622017+0000 osd.6 (osd.6) 301 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:55:31.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:29.723588+0000 mon.a (mon.0) 3818 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-02-13T00:55:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:29.838631+0000 osd.2 (osd.2) 169 : cluster [DBG] 6.f deep-scrub starts 2024-02-13T00:55:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:29.843119+0000 osd.2 (osd.2) 170 : cluster [DBG] 6.f deep-scrub ok 2024-02-13T00:55:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:30.039700+0000 mon.c (mon.2) 529 : audit [INF] from='client.? 172.21.15.162:0/3321310586' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-13T00:55:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:30.039953+0000 mon.a (mon.0) 3819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-13T00:55:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:30.618976+0000 osd.6 (osd.6) 300 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:55:31.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:30.622017+0000 osd.6 (osd.6) 301 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:55:31.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 2 1 2024-02-13T00:55:31.737 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:30.573497+0000 mgr.y (mgr.14544) 710 : cluster [DBG] pgmap v1291: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:30.730039+0000 mon.a (mon.0) 3820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-13T00:55:32.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:30.730110+0000 mon.a (mon.0) 3821 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-02-13T00:55:32.046 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:30.573497+0000 mgr.y (mgr.14544) 710 : cluster [DBG] pgmap v1291: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:32.046 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:30.730039+0000 mon.a (mon.0) 3820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-13T00:55:32.046 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:30.730110+0000 mon.a (mon.0) 3821 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-02-13T00:55:32.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:30.573497+0000 mgr.y (mgr.14544) 710 : cluster [DBG] pgmap v1291: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:32.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:30.730039+0000 mon.a (mon.0) 3820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-13T00:55:32.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:30.730110+0000 mon.a (mon.0) 3821 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-02-13T00:55:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:31.053190+0000 osd.1 (osd.1) 365 : cluster [DBG] 5.12 scrub starts 2024-02-13T00:55:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:31.056299+0000 osd.1 (osd.1) 366 : cluster [DBG] 5.12 scrub ok 2024-02-13T00:55:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:31.651086+0000 osd.6 (osd.6) 302 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:55:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:31.654016+0000 osd.6 (osd.6) 303 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:55:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:32.274870+0000 osd.5 (osd.5) 30 : cluster [DBG] 28.a scrub starts 2024-02-13T00:55:33.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:32.277838+0000 osd.5 (osd.5) 31 : cluster [DBG] 28.a scrub ok 2024-02-13T00:55:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:31.053190+0000 osd.1 (osd.1) 365 : cluster [DBG] 5.12 scrub starts 2024-02-13T00:55:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:31.056299+0000 osd.1 (osd.1) 366 : cluster [DBG] 5.12 scrub ok 2024-02-13T00:55:33.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:31.651086+0000 osd.6 (osd.6) 302 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:55:33.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:31.654016+0000 osd.6 (osd.6) 303 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:55:33.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:32.274870+0000 osd.5 (osd.5) 30 : cluster [DBG] 28.a scrub starts 2024-02-13T00:55:33.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:32.277838+0000 osd.5 (osd.5) 31 : cluster [DBG] 28.a scrub ok 2024-02-13T00:55:33.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:31.053190+0000 osd.1 (osd.1) 365 : cluster [DBG] 5.12 scrub starts 2024-02-13T00:55:33.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:31.056299+0000 osd.1 (osd.1) 366 : cluster [DBG] 5.12 scrub ok 2024-02-13T00:55:33.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:31.651086+0000 osd.6 (osd.6) 302 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:55:33.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:31.654016+0000 osd.6 (osd.6) 303 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:55:33.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:32.274870+0000 osd.5 (osd.5) 30 : cluster [DBG] 28.a scrub starts 2024-02-13T00:55:33.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:32.277838+0000 osd.5 (osd.5) 31 : cluster [DBG] 28.a scrub ok 2024-02-13T00:55:33.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 -- 172.21.15.162:0/2018404594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bdc0f8500 msgr2=0x7f5bdc104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:33.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 --2- 172.21.15.162:0/2018404594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bdc0f8500 0x7f5bdc104bd0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 rx=0x7f5bd4009c30 tx=0x7f5bd4004c50).stop 2024-02-13T00:55:33.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 -- 172.21.15.162:0/2018404594 shutdown_connections 2024-02-13T00:55:33.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 --2- 172.21.15.162:0/2018404594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bdc0f8500 0x7f5bdc104bd0 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:33.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 --2- 172.21.15.162:0/2018404594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bdc0f9b70 0x7f5bdc0f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:33.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 --2- 172.21.15.162:0/2018404594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc0fa440 0x7f5bdc0f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:33.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 -- 172.21.15.162:0/2018404594 >> 172.21.15.162:0/2018404594 conn(0x7f5bdc0f38c0 msgr2=0x7f5bdc0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:33.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 -- 172.21.15.162:0/2018404594 shutdown_connections 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 -- 172.21.15.162:0/2018404594 wait complete. 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 librados: starting msgr at 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 librados: starting objecter 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 Processor -- start 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 -- start start 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 librados: setting wanted keys 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.727+0000 7f5be44bb700 1 librados: calling monclient init 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be44bb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bdc0f8500 0x7f5bdc10e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be44bb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc0f9b70 0x7f5bdc10e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be44bb700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bdc0fa440 0x7f5bdc1081d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be44bb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc1086c0 con 0x7f5bdc0f8500 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be44bb700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc108830 con 0x7f5bdc0f9b70 2024-02-13T00:55:33.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be44bb700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc1089f0 con 0x7f5bdc0fa440 2024-02-13T00:55:33.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be2257700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bdc0f8500 0x7f5bdc10e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:33.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be2a58700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bdc0fa440 0x7f5bdc1081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:33.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be2257700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bdc0f8500 0x7f5bdc10e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:40046/0 (socket says 172.21.15.162:40046) 2024-02-13T00:55:33.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be2257700 1 -- 172.21.15.162:0/505142694 learned_addr learned my addr 172.21.15.162:0/505142694 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:33.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be1a56700 1 --2- 172.21.15.162:0/505142694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc0f9b70 0x7f5bdc10e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:33.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be2a58700 1 -- 172.21.15.162:0/505142694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc0f9b70 msgr2=0x7f5bdc10e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:33.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be2a58700 1 --2- 172.21.15.162:0/505142694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc0f9b70 0x7f5bdc10e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:33.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be2a58700 1 -- 172.21.15.162:0/505142694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bdc0f8500 msgr2=0x7f5bdc10e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:33.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be2a58700 1 --2- 172.21.15.162:0/505142694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bdc0f8500 0x7f5bdc10e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:33.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be2a58700 1 -- 172.21.15.162:0/505142694 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd8005040 con 0x7f5bdc0fa440 2024-02-13T00:55:33.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be2257700 1 --2- 172.21.15.162:0/505142694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bdc0f8500 0x7f5bdc10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:33.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be2a58700 1 --2- 172.21.15.162:0/505142694 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bdc0fa440 0x7f5bdc1081d0 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 rx=0x7f5bd4009c30 tx=0x7f5bd4004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5bd37fe700 10 client.?.objecter ms_handle_connect 0x7f5bdc0fa440 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5bd37fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5bd37fe700 1 -- 172.21.15.162:0/505142694 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd400e040 con 0x7f5bdc0fa440 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5bd37fe700 1 -- 172.21.15.162:0/505142694 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bd4018040 con 0x7f5bdc0fa440 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be44bb700 1 -- 172.21.15.162:0/505142694 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd40097c0 con 0x7f5bdc0fa440 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5bd37fe700 1 -- 172.21.15.162:0/505142694 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4003b90 con 0x7f5bdc0fa440 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be44bb700 10 client.38384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be44bb700 1 -- 172.21.15.162:0/505142694 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bdc108eb0 con 0x7f5bdc0fa440 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.728+0000 7f5be44bb700 1 librados: init done 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.729+0000 7f5be44bb700 1 -- 172.21.15.162:0/505142694 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bdc0475e0 con 0x7f5bdc0fa440 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.730+0000 7f5bd37fe700 1 -- 172.21.15.162:0/505142694 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f5bd4007bf0 con 0x7f5bdc0fa440 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.730+0000 7f5bd37fe700 1 --2- 172.21.15.162:0/505142694 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5bc8062c90 0x7f5bc8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:33.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.730+0000 7f5bd37fe700 1 -- 172.21.15.162:0/505142694 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(834..834 src has 449..834) v4 ==== 7393+0+0 (secure 0 0 0) 0x7f5bd407e030 con 0x7f5bdc0fa440 2024-02-13T00:55:33.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.730+0000 7f5bd37fe700 10 client.38384.objecter ms_dispatch 0x7f5bdc001550 osd_map(834..834 src has 449..834) v4 2024-02-13T00:55:33.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.730+0000 7f5bd37fe700 3 client.38384.objecter handle_osd_map got epochs [834,834] > 0 2024-02-13T00:55:33.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.730+0000 7f5bd37fe700 3 client.38384.objecter handle_osd_map decoding full epoch 834 2024-02-13T00:55:33.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.730+0000 7f5be2257700 1 --2- 172.21.15.162:0/505142694 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5bc8062c90 0x7f5bc8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:33.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.730+0000 7f5bd37fe700 20 client.38384.objecter dump_active .. 0 homeless 2024-02-13T00:55:33.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.730+0000 7f5bd37fe700 10 client.38384.objecter ms_handle_connect 0x7f5bc8062c90 2024-02-13T00:55:33.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.730+0000 7f5be2257700 1 --2- 172.21.15.162:0/505142694 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5bc8062c90 0x7f5bc8065150 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 rx=0x7f5bd8004d40 tx=0x7f5bd800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:33.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.735+0000 7f5bd37fe700 1 -- 172.21.15.162:0/505142694 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5bd40517d0 con 0x7f5bdc0fa440 2024-02-13T00:55:33.919 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:33.915+0000 7f5be44bb700 1 -- 172.21.15.162:0/505142694 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f5bdc1072c0 con 0x7f5bdc0fa440 2024-02-13T00:55:34.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:32.573898+0000 mgr.y (mgr.14544) 711 : cluster [DBG] pgmap v1293: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:34.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:32.660288+0000 osd.6 (osd.6) 304 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:55:34.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:32.663295+0000 osd.6 (osd.6) 305 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:55:34.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:32.755436+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.b scrub starts 2024-02-13T00:55:34.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:32.756289+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.b scrub ok 2024-02-13T00:55:34.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:33.047167+0000 osd.1 (osd.1) 367 : cluster [DBG] 28.11 scrub starts 2024-02-13T00:55:34.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:33.050115+0000 osd.1 (osd.1) 368 : cluster [DBG] 28.11 scrub ok 2024-02-13T00:55:34.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:33.487994+0000 osd.3 (osd.3) 185 : cluster [DBG] 4.1b scrub starts 2024-02-13T00:55:34.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:33.491630+0000 osd.3 (osd.3) 186 : cluster [DBG] 4.1b scrub ok 2024-02-13T00:55:34.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:32.573898+0000 mgr.y (mgr.14544) 711 : cluster [DBG] pgmap v1293: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:34.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:32.660288+0000 osd.6 (osd.6) 304 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:55:34.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:32.663295+0000 osd.6 (osd.6) 305 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:55:34.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:32.755436+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.b scrub starts 2024-02-13T00:55:34.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:32.756289+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.b scrub ok 2024-02-13T00:55:34.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:33.047167+0000 osd.1 (osd.1) 367 : cluster [DBG] 28.11 scrub starts 2024-02-13T00:55:34.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:33.050115+0000 osd.1 (osd.1) 368 : cluster [DBG] 28.11 scrub ok 2024-02-13T00:55:34.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:33.487994+0000 osd.3 (osd.3) 185 : cluster [DBG] 4.1b scrub starts 2024-02-13T00:55:34.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:33.491630+0000 osd.3 (osd.3) 186 : cluster [DBG] 4.1b scrub ok 2024-02-13T00:55:34.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:32.573898+0000 mgr.y (mgr.14544) 711 : cluster [DBG] pgmap v1293: 164 pgs: 1 active+remapped, 163 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/675 objects misplaced (0.296%) 2024-02-13T00:55:34.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:32.660288+0000 osd.6 (osd.6) 304 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:55:34.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:32.663295+0000 osd.6 (osd.6) 305 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:55:34.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:32.755436+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.b scrub starts 2024-02-13T00:55:34.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:32.756289+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.b scrub ok 2024-02-13T00:55:34.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:33.047167+0000 osd.1 (osd.1) 367 : cluster [DBG] 28.11 scrub starts 2024-02-13T00:55:34.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:33.050115+0000 osd.1 (osd.1) 368 : cluster [DBG] 28.11 scrub ok 2024-02-13T00:55:34.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:33.487994+0000 osd.3 (osd.3) 185 : cluster [DBG] 4.1b scrub starts 2024-02-13T00:55:34.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:33.491630+0000 osd.3 (osd.3) 186 : cluster [DBG] 4.1b scrub ok 2024-02-13T00:55:34.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.744+0000 7f5bd37fe700 1 -- 172.21.15.162:0/505142694 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v835) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5bd40513c0 con 0x7f5bdc0fa440 2024-02-13T00:55:34.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 10 librados: watch_flush enter 2024-02-13T00:55:34.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 10 librados: watch_flush exit 2024-02-13T00:55:34.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 20 client.38384.objecter shutdown clearing up homeless session... 2024-02-13T00:55:34.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 10 client.38384.objecter successfully canceled tick 2024-02-13T00:55:34.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 -- 172.21.15.162:0/505142694 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5bc8062c90 msgr2=0x7f5bc8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:34.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 --2- 172.21.15.162:0/505142694 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5bc8062c90 0x7f5bc8065150 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 rx=0x7f5bd8004d40 tx=0x7f5bd800b040).stop 2024-02-13T00:55:34.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 -- 172.21.15.162:0/505142694 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bdc0fa440 msgr2=0x7f5bdc1081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:34.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 --2- 172.21.15.162:0/505142694 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bdc0fa440 0x7f5bdc1081d0 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 rx=0x7f5bd4009c30 tx=0x7f5bd4004c50).stop 2024-02-13T00:55:34.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 -- 172.21.15.162:0/505142694 shutdown_connections 2024-02-13T00:55:34.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 --2- 172.21.15.162:0/505142694 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5bc8062c90 0x7f5bc8065150 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:34.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 --2- 172.21.15.162:0/505142694 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bdc0fa440 0x7f5bdc1081d0 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:34.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 --2- 172.21.15.162:0/505142694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bdc0f8500 0x7f5bdc10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:34.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 --2- 172.21.15.162:0/505142694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bdc0f9b70 0x7f5bdc10e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:34.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 -- 172.21.15.162:0/505142694 >> 172.21.15.162:0/505142694 conn(0x7f5bdc0f38c0 msgr2=0x7f5bdc0f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:34.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 -- 172.21.15.162:0/505142694 shutdown_connections 2024-02-13T00:55:34.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 -- 172.21.15.162:0/505142694 wait complete. 2024-02-13T00:55:34.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:34.747+0000 7f5be44bb700 1 librados: shutdown 2024-02-13T00:55:34.752 INFO:teuthology.orchestra.run.smithi162.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-02-13T00:55:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:33.916714+0000 mon.c (mon.2) 530 : audit [INF] from='client.? 172.21.15.162:0/505142694' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-13T00:55:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:33.917088+0000 mon.a (mon.0) 3822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-13T00:55:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:34.025016+0000 osd.1 (osd.1) 369 : cluster [DBG] 6.4 scrub starts 2024-02-13T00:55:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:34.027862+0000 osd.1 (osd.1) 370 : cluster [DBG] 6.4 scrub ok 2024-02-13T00:55:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:34.036004+0000 osd.0 (osd.0) 363 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:55:35.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:34.036869+0000 osd.0 (osd.0) 364 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:55:35.068 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:33.916714+0000 mon.c (mon.2) 530 : audit [INF] from='client.? 172.21.15.162:0/505142694' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-13T00:55:35.068 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:33.917088+0000 mon.a (mon.0) 3822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-13T00:55:35.068 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:34.025016+0000 osd.1 (osd.1) 369 : cluster [DBG] 6.4 scrub starts 2024-02-13T00:55:35.068 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:34.027862+0000 osd.1 (osd.1) 370 : cluster [DBG] 6.4 scrub ok 2024-02-13T00:55:35.068 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:34.036004+0000 osd.0 (osd.0) 363 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:55:35.069 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:34.036869+0000 osd.0 (osd.0) 364 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:55:35.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:33.916714+0000 mon.c (mon.2) 530 : audit [INF] from='client.? 172.21.15.162:0/505142694' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-13T00:55:35.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:33.917088+0000 mon.a (mon.0) 3822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-13T00:55:35.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:34.025016+0000 osd.1 (osd.1) 369 : cluster [DBG] 6.4 scrub starts 2024-02-13T00:55:35.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:34.027862+0000 osd.1 (osd.1) 370 : cluster [DBG] 6.4 scrub ok 2024-02-13T00:55:35.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:34.036004+0000 osd.0 (osd.0) 363 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:55:35.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:34.036869+0000 osd.0 (osd.0) 364 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:55:35.558 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 3 1 2024-02-13T00:55:35.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:34.574389+0000 mgr.y (mgr.14544) 712 : cluster [DBG] pgmap v1294: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 643 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:55:35.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:34.742328+0000 mon.a (mon.0) 3823 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-13T00:55:35.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:34.742422+0000 mon.a (mon.0) 3824 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-02-13T00:55:35.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:34.768010+0000 osd.2 (osd.2) 173 : cluster [DBG] 2.c scrub starts 2024-02-13T00:55:35.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:34.769257+0000 osd.2 (osd.2) 174 : cluster [DBG] 2.c scrub ok 2024-02-13T00:55:35.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:34.574389+0000 mgr.y (mgr.14544) 712 : cluster [DBG] pgmap v1294: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 643 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:55:35.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:34.742328+0000 mon.a (mon.0) 3823 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-13T00:55:35.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:34.742422+0000 mon.a (mon.0) 3824 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-02-13T00:55:35.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:34.768010+0000 osd.2 (osd.2) 173 : cluster [DBG] 2.c scrub starts 2024-02-13T00:55:35.832 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:34.769257+0000 osd.2 (osd.2) 174 : cluster [DBG] 2.c scrub ok 2024-02-13T00:55:35.854 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:34.574389+0000 mgr.y (mgr.14544) 712 : cluster [DBG] pgmap v1294: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 643 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:55:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:34.742328+0000 mon.a (mon.0) 3823 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-13T00:55:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:34.742422+0000 mon.a (mon.0) 3824 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-02-13T00:55:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:34.768010+0000 osd.2 (osd.2) 173 : cluster [DBG] 2.c scrub starts 2024-02-13T00:55:36.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:34.769257+0000 osd.2 (osd.2) 174 : cluster [DBG] 2.c scrub ok 2024-02-13T00:55:36.379 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:55:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:55:36] "GET /metrics HTTP/1.1" 200 185497 "" "Prometheus/2.33.4" 2024-02-13T00:55:36.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:55:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:55:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:36.053141+0000 mgr.y (mgr.14544) 713 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:36.053259+0000 osd.0 (osd.0) 365 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:55:37.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:36.056719+0000 osd.0 (osd.0) 366 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:55:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:36.053141+0000 mgr.y (mgr.14544) 713 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:36.053259+0000 osd.0 (osd.0) 365 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:55:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:36.056719+0000 osd.0 (osd.0) 366 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:55:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:36.053141+0000 mgr.y (mgr.14544) 713 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:36.053259+0000 osd.0 (osd.0) 365 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:55:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:36.056719+0000 osd.0 (osd.0) 366 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:55:37.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1200064840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bfc0f9b70 msgr2=0x7f0bfc0f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:37.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 --2- 172.21.15.162:0/1200064840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bfc0f9b70 0x7f0bfc0f9f50 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7f0bec004870 tx=0x7f0bec009cb0).stop 2024-02-13T00:55:37.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1200064840 shutdown_connections 2024-02-13T00:55:37.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 --2- 172.21.15.162:0/1200064840 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bfc0f8500 0x7f0bfc104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 --2- 172.21.15.162:0/1200064840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bfc0f9b70 0x7f0bfc0f9f50 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 --2- 172.21.15.162:0/1200064840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bfc0fa440 0x7f0bfc0f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1200064840 >> 172.21.15.162:0/1200064840 conn(0x7f0bfc0f38c0 msgr2=0x7f0bfc0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1200064840 shutdown_connections 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1200064840 wait complete. 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 librados: starting msgr at 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 librados: starting objecter 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 Processor -- start 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 -- start start 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 librados: setting wanted keys 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.715+0000 7f0c03fdb700 1 librados: calling monclient init 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c03fdb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bfc0f8500 0x7f0bfc06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:37.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c03fdb700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bfc0f9b70 0x7f0bfc06b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c03fdb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bfc0fa440 0x7f0bfc06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c03fdb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc1166c0 con 0x7f0bfc0f8500 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c03fdb700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc116830 con 0x7f0bfc0fa440 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c03fdb700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc1169f0 con 0x7f0bfc0f9b70 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c01d77700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bfc0f8500 0x7f0bfc06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c01d77700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bfc0f8500 0x7f0bfc06b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60202/0 (socket says 172.21.15.162:60202) 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c01d77700 1 -- 172.21.15.162:0/1753023192 learned_addr learned my addr 172.21.15.162:0/1753023192 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c01576700 1 --2- 172.21.15.162:0/1753023192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bfc0f9b70 0x7f0bfc06b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c02578700 1 --2- 172.21.15.162:0/1753023192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bfc0fa440 0x7f0bfc06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c01d77700 1 -- 172.21.15.162:0/1753023192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bfc0f9b70 msgr2=0x7f0bfc06b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c01d77700 1 --2- 172.21.15.162:0/1753023192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bfc0f9b70 0x7f0bfc06b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c01d77700 1 -- 172.21.15.162:0/1753023192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bfc0fa440 msgr2=0x7f0bfc06bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:37.766 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c01d77700 1 --2- 172.21.15.162:0/1753023192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bfc0fa440 0x7f0bfc06bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c01d77700 1 -- 172.21.15.162:0/1753023192 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf40097c0 con 0x7f0bfc0f8500 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c02578700 1 --2- 172.21.15.162:0/1753023192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bfc0fa440 0x7f0bfc06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c01576700 1 --2- 172.21.15.162:0/1753023192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bfc0f9b70 0x7f0bfc06b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c01d77700 1 --2- 172.21.15.162:0/1753023192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bfc0f8500 0x7f0bfc06b170 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 rx=0x7f0bec009eb0 tx=0x7f0bec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0bf2ffd700 10 client.?.objecter ms_handle_connect 0x7f0bfc0f8500 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0bf2ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0bf2ffd700 1 -- 172.21.15.162:0/1753023192 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec00e070 con 0x7f0bfc0f8500 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0bf2ffd700 1 -- 172.21.15.162:0/1753023192 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bec013da0 con 0x7f0bfc0f8500 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1753023192 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bec005040 con 0x7f0bfc0f8500 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0bf2ffd700 1 -- 172.21.15.162:0/1753023192 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec019070 con 0x7f0bfc0f8500 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c03fdb700 10 client.28296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1753023192 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bfc116fc0 con 0x7f0bfc0f8500 2024-02-13T00:55:37.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.716+0000 7f0c03fdb700 1 librados: init done 2024-02-13T00:55:37.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.717+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1753023192 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bfc0475e0 con 0x7f0bfc0f8500 2024-02-13T00:55:37.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.718+0000 7f0bf2ffd700 1 -- 172.21.15.162:0/1753023192 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0bec01e420 con 0x7f0bfc0f8500 2024-02-13T00:55:37.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.718+0000 7f0bf2ffd700 1 --2- 172.21.15.162:0/1753023192 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0be8062c90 0x7f0be8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:37.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.718+0000 7f0bf2ffd700 1 -- 172.21.15.162:0/1753023192 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(836..836 src has 449..836) v4 ==== 7393+0+0 (secure 0 0 0) 0x7f0bec07db00 con 0x7f0bfc0f8500 2024-02-13T00:55:37.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.718+0000 7f0bf2ffd700 10 client.28296.objecter ms_dispatch 0x7f0bfc001550 osd_map(836..836 src has 449..836) v4 2024-02-13T00:55:37.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.718+0000 7f0bf2ffd700 3 client.28296.objecter handle_osd_map got epochs [836,836] > 0 2024-02-13T00:55:37.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.718+0000 7f0bf2ffd700 3 client.28296.objecter handle_osd_map decoding full epoch 836 2024-02-13T00:55:37.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.718+0000 7f0c01576700 1 --2- 172.21.15.162:0/1753023192 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0be8062c90 0x7f0be8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:37.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.718+0000 7f0bf2ffd700 20 client.28296.objecter dump_active .. 0 homeless 2024-02-13T00:55:37.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.718+0000 7f0c01576700 1 --2- 172.21.15.162:0/1753023192 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0be8062c90 0x7f0be8065150 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 rx=0x7f0bfc06c910 tx=0x7f0bf800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:37.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.718+0000 7f0bf2ffd700 10 client.28296.objecter ms_handle_connect 0x7f0be8062c90 2024-02-13T00:55:37.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.722+0000 7f0bf2ffd700 1 -- 172.21.15.162:0/1753023192 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0bec0512a0 con 0x7f0bfc0f8500 2024-02-13T00:55:37.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:37.902+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1753023192 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f0bfc06c650 con 0x7f0bfc0f8500 2024-02-13T00:55:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:36.575316+0000 mgr.y (mgr.14544) 714 : cluster [DBG] pgmap v1296: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:55:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:36.591383+0000 osd.2 (osd.2) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:55:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:36.591717+0000 osd.2 (osd.2) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:55:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:36.761201+0000 mon.a (mon.0) 3825 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-02-13T00:55:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:36.985622+0000 osd.1 (osd.1) 371 : cluster [DBG] 6.e scrub starts 2024-02-13T00:55:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:36.987284+0000 osd.1 (osd.1) 372 : cluster [DBG] 6.e scrub ok 2024-02-13T00:55:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:37.042723+0000 osd.0 (osd.0) 367 : cluster [DBG] 1.0 deep-scrub starts 2024-02-13T00:55:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:37.044399+0000 osd.0 (osd.0) 368 : cluster [DBG] 1.0 deep-scrub ok 2024-02-13T00:55:38.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:37.379916+0000 osd.5 (osd.5) 32 : cluster [DBG] 28.9 scrub starts 2024-02-13T00:55:38.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:37.383008+0000 osd.5 (osd.5) 33 : cluster [DBG] 28.9 scrub ok 2024-02-13T00:55:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:36.575316+0000 mgr.y (mgr.14544) 714 : cluster [DBG] pgmap v1296: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:55:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:36.591383+0000 osd.2 (osd.2) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:55:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:36.591717+0000 osd.2 (osd.2) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:55:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:36.761201+0000 mon.a (mon.0) 3825 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-02-13T00:55:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:36.985622+0000 osd.1 (osd.1) 371 : cluster [DBG] 6.e scrub starts 2024-02-13T00:55:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:36.987284+0000 osd.1 (osd.1) 372 : cluster [DBG] 6.e scrub ok 2024-02-13T00:55:38.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:37.042723+0000 osd.0 (osd.0) 367 : cluster [DBG] 1.0 deep-scrub starts 2024-02-13T00:55:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:37.044399+0000 osd.0 (osd.0) 368 : cluster [DBG] 1.0 deep-scrub ok 2024-02-13T00:55:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:37.379916+0000 osd.5 (osd.5) 32 : cluster [DBG] 28.9 scrub starts 2024-02-13T00:55:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:37.383008+0000 osd.5 (osd.5) 33 : cluster [DBG] 28.9 scrub ok 2024-02-13T00:55:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:36.575316+0000 mgr.y (mgr.14544) 714 : cluster [DBG] pgmap v1296: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:55:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:36.591383+0000 osd.2 (osd.2) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:55:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:36.591717+0000 osd.2 (osd.2) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:55:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:36.761201+0000 mon.a (mon.0) 3825 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-02-13T00:55:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:36.985622+0000 osd.1 (osd.1) 371 : cluster [DBG] 6.e scrub starts 2024-02-13T00:55:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:36.987284+0000 osd.1 (osd.1) 372 : cluster [DBG] 6.e scrub ok 2024-02-13T00:55:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:37.042723+0000 osd.0 (osd.0) 367 : cluster [DBG] 1.0 deep-scrub starts 2024-02-13T00:55:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:37.044399+0000 osd.0 (osd.0) 368 : cluster [DBG] 1.0 deep-scrub ok 2024-02-13T00:55:38.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:37.379916+0000 osd.5 (osd.5) 32 : cluster [DBG] 28.9 scrub starts 2024-02-13T00:55:38.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:37.383008+0000 osd.5 (osd.5) 33 : cluster [DBG] 28.9 scrub ok 2024-02-13T00:55:38.775 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.771+0000 7f0bf2ffd700 1 -- 172.21.15.162:0/1753023192 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v837) v1 ==== 113+0+0 (secure 0 0 0) 0x7f0bec023020 con 0x7f0bfc0f8500 2024-02-13T00:55:38.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 10 librados: watch_flush enter 2024-02-13T00:55:38.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 10 librados: watch_flush exit 2024-02-13T00:55:38.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 20 client.28296.objecter shutdown clearing up homeless session... 2024-02-13T00:55:38.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 10 client.28296.objecter successfully canceled tick 2024-02-13T00:55:38.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1753023192 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0be8062c90 msgr2=0x7f0be8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:38.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 --2- 172.21.15.162:0/1753023192 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0be8062c90 0x7f0be8065150 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 rx=0x7f0bfc06c910 tx=0x7f0bf800b040).stop 2024-02-13T00:55:38.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1753023192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bfc0f8500 msgr2=0x7f0bfc06b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:38.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 --2- 172.21.15.162:0/1753023192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bfc0f8500 0x7f0bfc06b170 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 rx=0x7f0bec009eb0 tx=0x7f0bec00b040).stop 2024-02-13T00:55:38.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1753023192 shutdown_connections 2024-02-13T00:55:38.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 --2- 172.21.15.162:0/1753023192 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0be8062c90 0x7f0be8065150 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:38.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 --2- 172.21.15.162:0/1753023192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bfc0fa440 0x7f0bfc06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:38.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 --2- 172.21.15.162:0/1753023192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bfc0f8500 0x7f0bfc06b170 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:38.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 --2- 172.21.15.162:0/1753023192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bfc0f9b70 0x7f0bfc06b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:38.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1753023192 >> 172.21.15.162:0/1753023192 conn(0x7f0bfc0f38c0 msgr2=0x7f0bfc0fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:38.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1753023192 shutdown_connections 2024-02-13T00:55:38.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 -- 172.21.15.162:0/1753023192 wait complete. 2024-02-13T00:55:38.779 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:38.774+0000 7f0c03fdb700 1 librados: shutdown 2024-02-13T00:55:38.780 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.3 to 1 (10000) 2024-02-13T00:55:39.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:55:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:55:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:55:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:37.707204+0000 osd.6 (osd.6) 306 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:55:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:37.710291+0000 osd.6 (osd.6) 307 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:55:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:37.737963+0000 osd.2 (osd.2) 177 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:55:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:37.740887+0000 osd.2 (osd.2) 178 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:55:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:37.903122+0000 mon.a (mon.0) 3826 : audit [INF] from='client.? 172.21.15.162:0/1753023192' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-13T00:55:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:38.003180+0000 osd.1 (osd.1) 373 : cluster [DBG] 6.26 scrub starts 2024-02-13T00:55:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:38.004075+0000 osd.1 (osd.1) 374 : cluster [DBG] 6.26 scrub ok 2024-02-13T00:55:39.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:38.018886+0000 osd.0 (osd.0) 369 : cluster [DBG] 2.1f deep-scrub starts 2024-02-13T00:55:39.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:38.021909+0000 osd.0 (osd.0) 370 : cluster [DBG] 2.1f deep-scrub ok 2024-02-13T00:55:39.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:37.707204+0000 osd.6 (osd.6) 306 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:55:39.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:37.710291+0000 osd.6 (osd.6) 307 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:55:39.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:37.737963+0000 osd.2 (osd.2) 177 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:55:39.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:37.740887+0000 osd.2 (osd.2) 178 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:55:39.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:37.903122+0000 mon.a (mon.0) 3826 : audit [INF] from='client.? 172.21.15.162:0/1753023192' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-13T00:55:39.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:38.003180+0000 osd.1 (osd.1) 373 : cluster [DBG] 6.26 scrub starts 2024-02-13T00:55:39.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:38.004075+0000 osd.1 (osd.1) 374 : cluster [DBG] 6.26 scrub ok 2024-02-13T00:55:39.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:38.018886+0000 osd.0 (osd.0) 369 : cluster [DBG] 2.1f deep-scrub starts 2024-02-13T00:55:39.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:38.021909+0000 osd.0 (osd.0) 370 : cluster [DBG] 2.1f deep-scrub ok 2024-02-13T00:55:39.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:37.707204+0000 osd.6 (osd.6) 306 : cluster [DBG] 6.3 scrub starts 2024-02-13T00:55:39.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:37.710291+0000 osd.6 (osd.6) 307 : cluster [DBG] 6.3 scrub ok 2024-02-13T00:55:39.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:37.737963+0000 osd.2 (osd.2) 177 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:55:39.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:37.740887+0000 osd.2 (osd.2) 178 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:55:39.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:37.903122+0000 mon.a (mon.0) 3826 : audit [INF] from='client.? 172.21.15.162:0/1753023192' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-13T00:55:39.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:38.003180+0000 osd.1 (osd.1) 373 : cluster [DBG] 6.26 scrub starts 2024-02-13T00:55:39.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:38.004075+0000 osd.1 (osd.1) 374 : cluster [DBG] 6.26 scrub ok 2024-02-13T00:55:39.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:38.018886+0000 osd.0 (osd.0) 369 : cluster [DBG] 2.1f deep-scrub starts 2024-02-13T00:55:39.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:38 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:38.021909+0000 osd.0 (osd.0) 370 : cluster [DBG] 2.1f deep-scrub ok 2024-02-13T00:55:39.583 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-13T00:55:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:38.576000+0000 mgr.y (mgr.14544) 715 : cluster [DBG] pgmap v1298: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 652 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:55:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:38.709295+0000 osd.6 (osd.6) 308 : cluster [DBG] 4.18 scrub starts 2024-02-13T00:55:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:38.713020+0000 osd.6 (osd.6) 309 : cluster [DBG] 4.18 scrub ok 2024-02-13T00:55:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:38.771759+0000 mon.a (mon.0) 3827 : audit [INF] from='client.? 172.21.15.162:0/1753023192' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-13T00:55:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:38.771881+0000 mon.a (mon.0) 3828 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-02-13T00:55:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:38.985340+0000 osd.1 (osd.1) 375 : cluster [DBG] 28.15 scrub starts 2024-02-13T00:55:39.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:38.989205+0000 osd.1 (osd.1) 376 : cluster [DBG] 28.15 scrub ok 2024-02-13T00:55:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:38.576000+0000 mgr.y (mgr.14544) 715 : cluster [DBG] pgmap v1298: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 652 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:55:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:38.709295+0000 osd.6 (osd.6) 308 : cluster [DBG] 4.18 scrub starts 2024-02-13T00:55:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:38.713020+0000 osd.6 (osd.6) 309 : cluster [DBG] 4.18 scrub ok 2024-02-13T00:55:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:38.771759+0000 mon.a (mon.0) 3827 : audit [INF] from='client.? 172.21.15.162:0/1753023192' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-13T00:55:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:38.771881+0000 mon.a (mon.0) 3828 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-02-13T00:55:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:38.985340+0000 osd.1 (osd.1) 375 : cluster [DBG] 28.15 scrub starts 2024-02-13T00:55:39.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:38.989205+0000 osd.1 (osd.1) 376 : cluster [DBG] 28.15 scrub ok 2024-02-13T00:55:39.858 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:38.576000+0000 mgr.y (mgr.14544) 715 : cluster [DBG] pgmap v1298: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 652 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:55:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:38.709295+0000 osd.6 (osd.6) 308 : cluster [DBG] 4.18 scrub starts 2024-02-13T00:55:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:38.713020+0000 osd.6 (osd.6) 309 : cluster [DBG] 4.18 scrub ok 2024-02-13T00:55:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:38.771759+0000 mon.a (mon.0) 3827 : audit [INF] from='client.? 172.21.15.162:0/1753023192' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-13T00:55:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:38.771881+0000 mon.a (mon.0) 3828 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-02-13T00:55:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:38.985340+0000 osd.1 (osd.1) 375 : cluster [DBG] 28.15 scrub starts 2024-02-13T00:55:40.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:38.989205+0000 osd.1 (osd.1) 376 : cluster [DBG] 28.15 scrub ok 2024-02-13T00:55:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:39.668245+0000 osd.2 (osd.2) 179 : cluster [DBG] 6.28 deep-scrub starts 2024-02-13T00:55:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:39.671159+0000 osd.2 (osd.2) 180 : cluster [DBG] 6.28 deep-scrub ok 2024-02-13T00:55:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:40.012342+0000 osd.1 (osd.1) 377 : cluster [DBG] 3.0 deep-scrub starts 2024-02-13T00:55:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:40.016269+0000 osd.1 (osd.1) 378 : cluster [DBG] 3.0 deep-scrub ok 2024-02-13T00:55:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:40.312950+0000 osd.5 (osd.5) 34 : cluster [DBG] 28.1d scrub starts 2024-02-13T00:55:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:40.316233+0000 osd.5 (osd.5) 35 : cluster [DBG] 28.1d scrub ok 2024-02-13T00:55:41.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:40.522333+0000 mon.a (mon.0) 3829 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:41.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:39.668245+0000 osd.2 (osd.2) 179 : cluster [DBG] 6.28 deep-scrub starts 2024-02-13T00:55:41.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:39.671159+0000 osd.2 (osd.2) 180 : cluster [DBG] 6.28 deep-scrub ok 2024-02-13T00:55:41.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:40.012342+0000 osd.1 (osd.1) 377 : cluster [DBG] 3.0 deep-scrub starts 2024-02-13T00:55:41.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:40.016269+0000 osd.1 (osd.1) 378 : cluster [DBG] 3.0 deep-scrub ok 2024-02-13T00:55:41.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:40.312950+0000 osd.5 (osd.5) 34 : cluster [DBG] 28.1d scrub starts 2024-02-13T00:55:41.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:40.316233+0000 osd.5 (osd.5) 35 : cluster [DBG] 28.1d scrub ok 2024-02-13T00:55:41.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:40.522333+0000 mon.a (mon.0) 3829 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:39.668245+0000 osd.2 (osd.2) 179 : cluster [DBG] 6.28 deep-scrub starts 2024-02-13T00:55:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:39.671159+0000 osd.2 (osd.2) 180 : cluster [DBG] 6.28 deep-scrub ok 2024-02-13T00:55:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:40.012342+0000 osd.1 (osd.1) 377 : cluster [DBG] 3.0 deep-scrub starts 2024-02-13T00:55:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:40.016269+0000 osd.1 (osd.1) 378 : cluster [DBG] 3.0 deep-scrub ok 2024-02-13T00:55:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:40.312950+0000 osd.5 (osd.5) 34 : cluster [DBG] 28.1d scrub starts 2024-02-13T00:55:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:40.316233+0000 osd.5 (osd.5) 35 : cluster [DBG] 28.1d scrub ok 2024-02-13T00:55:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:40.522333+0000 mon.a (mon.0) 3829 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:55:41.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.690+0000 7fd7f19ae700 1 -- 172.21.15.162:0/2144088818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7ec0fcbb0 msgr2=0x7fd7ec0fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:41.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.690+0000 7fd7f19ae700 1 --2- 172.21.15.162:0/2144088818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7ec0fcbb0 0x7fd7ec0fcf90 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 rx=0x7fd7e00047b0 tx=0x7fd7e0009cb0).stop 2024-02-13T00:55:41.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.690+0000 7fd7f19ae700 1 -- 172.21.15.162:0/2144088818 shutdown_connections 2024-02-13T00:55:41.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.690+0000 7fd7f19ae700 1 --2- 172.21.15.162:0/2144088818 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7ec0f7a20 0x7fd7ec103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:41.696 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.690+0000 7fd7f19ae700 1 --2- 172.21.15.162:0/2144088818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7ec0fcbb0 0x7fd7ec0fcf90 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:41.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.690+0000 7fd7f19ae700 1 --2- 172.21.15.162:0/2144088818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7ec0fd510 0x7fd7ec0f7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:41.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.690+0000 7fd7f19ae700 1 -- 172.21.15.162:0/2144088818 >> 172.21.15.162:0/2144088818 conn(0x7fd7ec070100 msgr2=0x7fd7ec0f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:41.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.690+0000 7fd7f19ae700 1 -- 172.21.15.162:0/2144088818 shutdown_connections 2024-02-13T00:55:41.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.690+0000 7fd7f19ae700 1 -- 172.21.15.162:0/2144088818 wait complete. 2024-02-13T00:55:41.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.690+0000 7fd7f19ae700 1 librados: starting msgr at 2024-02-13T00:55:41.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7f19ae700 1 librados: starting objecter 2024-02-13T00:55:41.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7f19ae700 1 Processor -- start 2024-02-13T00:55:41.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7f19ae700 1 -- start start 2024-02-13T00:55:41.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7f19ae700 1 librados: setting wanted keys 2024-02-13T00:55:41.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7f19ae700 1 librados: calling monclient init 2024-02-13T00:55:41.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7f19ae700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7ec0f7a20 0x7fd7ec10d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:41.697 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7f19ae700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7ec0fcbb0 0x7fd7ec10d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7f19ae700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7ec0fd510 0x7fd7ec107250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7f19ae700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec107740 con 0x7fd7ec0fd510 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7f19ae700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec1078b0 con 0x7fd7ec0fcbb0 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7f19ae700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec107a70 con 0x7fd7ec0f7a20 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7eaffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7ec0f7a20 0x7fd7ec10d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7eb7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7ec0fd510 0x7fd7ec107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7eb7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7ec0fd510 0x7fd7ec107250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:60234/0 (socket says 172.21.15.162:60234) 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7eaffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7ec0f7a20 0x7fd7ec10d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:55290/0 (socket says 172.21.15.162:55290) 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.691+0000 7fd7eb7fe700 1 -- 172.21.15.162:0/3497063986 learned_addr learned my addr 172.21.15.162:0/3497063986 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7ea7fc700 1 --2- 172.21.15.162:0/3497063986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7ec0fcbb0 0x7fd7ec10d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7eb7fe700 1 -- 172.21.15.162:0/3497063986 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7ec0f7a20 msgr2=0x7fd7ec10d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7eb7fe700 1 --2- 172.21.15.162:0/3497063986 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7ec0f7a20 0x7fd7ec10d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7eb7fe700 1 -- 172.21.15.162:0/3497063986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7ec0fcbb0 msgr2=0x7fd7ec10d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:41.698 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7eb7fe700 1 --2- 172.21.15.162:0/3497063986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7ec0fcbb0 0x7fd7ec10d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7eb7fe700 1 -- 172.21.15.162:0/3497063986 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e40097c0 con 0x7fd7ec0fd510 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7eaffd700 1 --2- 172.21.15.162:0/3497063986 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7ec0f7a20 0x7fd7ec10d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7ea7fc700 1 --2- 172.21.15.162:0/3497063986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7ec0fcbb0 0x7fd7ec10d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7eb7fe700 1 --2- 172.21.15.162:0/3497063986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7ec0fd510 0x7fd7ec107250 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 rx=0x7fd7e4011f00 tx=0x7fd7e4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7d3fff700 10 client.?.objecter ms_handle_connect 0x7fd7ec0fd510 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7d3fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7d3fff700 1 -- 172.21.15.162:0/3497063986 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4009a10 con 0x7fd7ec0fd510 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7f19ae700 1 -- 172.21.15.162:0/3497063986 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e0005040 con 0x7fd7ec0fd510 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7d3fff700 1 -- 172.21.15.162:0/3497063986 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7e40095e0 con 0x7fd7ec0fd510 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7d3fff700 1 -- 172.21.15.162:0/3497063986 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4013460 con 0x7fd7ec0fd510 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7f19ae700 10 client.28302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7f19ae700 1 -- 172.21.15.162:0/3497063986 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ec108020 con 0x7fd7ec0fd510 2024-02-13T00:55:41.699 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.692+0000 7fd7f19ae700 1 librados: init done 2024-02-13T00:55:41.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.693+0000 7fd7f19ae700 1 -- 172.21.15.162:0/3497063986 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7ec05f980 con 0x7fd7ec0fd510 2024-02-13T00:55:41.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.693+0000 7fd7d3fff700 1 -- 172.21.15.162:0/3497063986 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fd7e4009110 con 0x7fd7ec0fd510 2024-02-13T00:55:41.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.694+0000 7fd7d3fff700 1 --2- 172.21.15.162:0/3497063986 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd7d4062c90 0x7fd7d4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:41.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.694+0000 7fd7d3fff700 1 -- 172.21.15.162:0/3497063986 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(837..837 src has 449..837) v4 ==== 7393+0+0 (secure 0 0 0) 0x7fd7e407d1f0 con 0x7fd7ec0fd510 2024-02-13T00:55:41.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.694+0000 7fd7d3fff700 10 client.28302.objecter ms_dispatch 0x7fd7ec001550 osd_map(837..837 src has 449..837) v4 2024-02-13T00:55:41.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.694+0000 7fd7d3fff700 3 client.28302.objecter handle_osd_map got epochs [837,837] > 0 2024-02-13T00:55:41.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.694+0000 7fd7d3fff700 3 client.28302.objecter handle_osd_map decoding full epoch 837 2024-02-13T00:55:41.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.694+0000 7fd7eaffd700 1 --2- 172.21.15.162:0/3497063986 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd7d4062c90 0x7fd7d4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:41.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.694+0000 7fd7d3fff700 20 client.28302.objecter dump_active .. 0 homeless 2024-02-13T00:55:41.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.694+0000 7fd7d3fff700 10 client.28302.objecter ms_handle_connect 0x7fd7d4062c90 2024-02-13T00:55:41.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.694+0000 7fd7eaffd700 1 --2- 172.21.15.162:0/3497063986 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd7d4062c90 0x7fd7d4065150 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 rx=0x7fd7e0005010 tx=0x7fd7e0009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:41.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.698+0000 7fd7d3fff700 1 -- 172.21.15.162:0/3497063986 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd7e4050990 con 0x7fd7ec0fd510 2024-02-13T00:55:41.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:41.876+0000 7fd7f19ae700 1 -- 172.21.15.162:0/3497063986 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) v1 -- 0x7fd7ec106110 con 0x7fd7ec0fd510 2024-02-13T00:55:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:40.576669+0000 mgr.y (mgr.14544) 716 : cluster [DBG] pgmap v1300: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:55:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:41.448250+0000 osd.3 (osd.3) 187 : cluster [DBG] 28.1c scrub starts 2024-02-13T00:55:42.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:41.451894+0000 osd.3 (osd.3) 188 : cluster [DBG] 28.1c scrub ok 2024-02-13T00:55:42.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:40.576669+0000 mgr.y (mgr.14544) 716 : cluster [DBG] pgmap v1300: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:55:42.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:41.448250+0000 osd.3 (osd.3) 187 : cluster [DBG] 28.1c scrub starts 2024-02-13T00:55:42.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:41.451894+0000 osd.3 (osd.3) 188 : cluster [DBG] 28.1c scrub ok 2024-02-13T00:55:42.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:40.576669+0000 mgr.y (mgr.14544) 716 : cluster [DBG] pgmap v1300: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-13T00:55:42.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:41.448250+0000 osd.3 (osd.3) 187 : cluster [DBG] 28.1c scrub starts 2024-02-13T00:55:42.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:41.451894+0000 osd.3 (osd.3) 188 : cluster [DBG] 28.1c scrub ok 2024-02-13T00:55:42.782 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.779+0000 7fd7d3fff700 1 -- 172.21.15.162:0/3497063986 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v838) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd7e4050580 con 0x7fd7ec0fd510 2024-02-13T00:55:42.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.781+0000 7fd7f19ae700 10 librados: watch_flush enter 2024-02-13T00:55:42.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.781+0000 7fd7f19ae700 10 librados: watch_flush exit 2024-02-13T00:55:42.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.781+0000 7fd7f19ae700 20 client.28302.objecter shutdown clearing up homeless session... 2024-02-13T00:55:42.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.781+0000 7fd7f19ae700 10 client.28302.objecter successfully canceled tick 2024-02-13T00:55:42.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.783+0000 7fd7f19ae700 1 -- 172.21.15.162:0/3497063986 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd7d4062c90 msgr2=0x7fd7d4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:42.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.783+0000 7fd7f19ae700 1 --2- 172.21.15.162:0/3497063986 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd7d4062c90 0x7fd7d4065150 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 rx=0x7fd7e0005010 tx=0x7fd7e0009cb0).stop 2024-02-13T00:55:42.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.783+0000 7fd7f19ae700 1 -- 172.21.15.162:0/3497063986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7ec0fd510 msgr2=0x7fd7ec107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:42.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.783+0000 7fd7f19ae700 1 --2- 172.21.15.162:0/3497063986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7ec0fd510 0x7fd7ec107250 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 rx=0x7fd7e4011f00 tx=0x7fd7e4004d10).stop 2024-02-13T00:55:42.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.783+0000 7fd7f19ae700 1 -- 172.21.15.162:0/3497063986 shutdown_connections 2024-02-13T00:55:42.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.783+0000 7fd7f19ae700 1 --2- 172.21.15.162:0/3497063986 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd7d4062c90 0x7fd7d4065150 unknown :-1 s=CLOSED pgs=3071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:42.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.783+0000 7fd7f19ae700 1 --2- 172.21.15.162:0/3497063986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7ec0fd510 0x7fd7ec107250 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:42.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.783+0000 7fd7f19ae700 1 --2- 172.21.15.162:0/3497063986 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7ec0f7a20 0x7fd7ec10d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:42.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.783+0000 7fd7f19ae700 1 --2- 172.21.15.162:0/3497063986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7ec0fcbb0 0x7fd7ec10d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:42.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.783+0000 7fd7f19ae700 1 -- 172.21.15.162:0/3497063986 >> 172.21.15.162:0/3497063986 conn(0x7fd7ec070100 msgr2=0x7fd7ec0fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:42.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.783+0000 7fd7f19ae700 1 -- 172.21.15.162:0/3497063986 shutdown_connections 2024-02-13T00:55:42.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.783+0000 7fd7f19ae700 1 -- 172.21.15.162:0/3497063986 wait complete. 2024-02-13T00:55:42.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:42.784+0000 7fd7f19ae700 1 librados: shutdown 2024-02-13T00:55:42.789 INFO:teuthology.orchestra.run.smithi162.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-13T00:55:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:41.877315+0000 mon.a (mon.0) 3830 : audit [INF] from='client.? 172.21.15.162:0/3497063986' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-13T00:55:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:42.043566+0000 osd.1 (osd.1) 379 : cluster [DBG] 28.1 scrub starts 2024-02-13T00:55:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:42.047037+0000 osd.1 (osd.1) 380 : cluster [DBG] 28.1 scrub ok 2024-02-13T00:55:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:42.456611+0000 osd.3 (osd.3) 189 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:55:43.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:42.458073+0000 osd.3 (osd.3) 190 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:55:43.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:41.877315+0000 mon.a (mon.0) 3830 : audit [INF] from='client.? 172.21.15.162:0/3497063986' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-13T00:55:43.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:42.043566+0000 osd.1 (osd.1) 379 : cluster [DBG] 28.1 scrub starts 2024-02-13T00:55:43.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:42.047037+0000 osd.1 (osd.1) 380 : cluster [DBG] 28.1 scrub ok 2024-02-13T00:55:43.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:42.456611+0000 osd.3 (osd.3) 189 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:55:43.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:42.458073+0000 osd.3 (osd.3) 190 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:55:43.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:41.877315+0000 mon.a (mon.0) 3830 : audit [INF] from='client.? 172.21.15.162:0/3497063986' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-13T00:55:43.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:42.043566+0000 osd.1 (osd.1) 379 : cluster [DBG] 28.1 scrub starts 2024-02-13T00:55:43.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:42.047037+0000 osd.1 (osd.1) 380 : cluster [DBG] 28.1 scrub ok 2024-02-13T00:55:43.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:42.456611+0000 osd.3 (osd.3) 189 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:55:43.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:42.458073+0000 osd.3 (osd.3) 190 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:55:43.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 4 1 2024-02-13T00:55:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:42.577149+0000 mgr.y (mgr.14544) 717 : cluster [DBG] pgmap v1301: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:55:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:42.779850+0000 mon.a (mon.0) 3831 : audit [INF] from='client.? 172.21.15.162:0/3497063986' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-13T00:55:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:42.779905+0000 mon.a (mon.0) 3832 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-02-13T00:55:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:42.938646+0000 osd.0 (osd.0) 371 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:55:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:42.941840+0000 osd.0 (osd.0) 372 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:55:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:43.362722+0000 osd.0 (osd.0) 373 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:55:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:43.363082+0000 osd.0 (osd.0) 374 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:55:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:43.433163+0000 osd.3 (osd.3) 191 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:55:43.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:43.437040+0000 osd.3 (osd.3) 192 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:55:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:42.577149+0000 mgr.y (mgr.14544) 717 : cluster [DBG] pgmap v1301: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:55:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:42.779850+0000 mon.a (mon.0) 3831 : audit [INF] from='client.? 172.21.15.162:0/3497063986' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-13T00:55:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:42.779905+0000 mon.a (mon.0) 3832 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-02-13T00:55:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:42.938646+0000 osd.0 (osd.0) 371 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:55:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:42.941840+0000 osd.0 (osd.0) 372 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:55:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:43.362722+0000 osd.0 (osd.0) 373 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:55:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:43.363082+0000 osd.0 (osd.0) 374 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:55:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:43.433163+0000 osd.3 (osd.3) 191 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:55:43.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:43.437040+0000 osd.3 (osd.3) 192 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:55:43.842 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:44.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:42.577149+0000 mgr.y (mgr.14544) 717 : cluster [DBG] pgmap v1301: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:55:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:42.779850+0000 mon.a (mon.0) 3831 : audit [INF] from='client.? 172.21.15.162:0/3497063986' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-13T00:55:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:42.779905+0000 mon.a (mon.0) 3832 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-02-13T00:55:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:42.938646+0000 osd.0 (osd.0) 371 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:55:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:42.941840+0000 osd.0 (osd.0) 372 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:55:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:43.362722+0000 osd.0 (osd.0) 373 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:55:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:43.363082+0000 osd.0 (osd.0) 374 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:55:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:43.433163+0000 osd.3 (osd.3) 191 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:55:44.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:43.437040+0000 osd.3 (osd.3) 192 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:55:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:43.663455+0000 osd.2 (osd.2) 181 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:55:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:43.664755+0000 osd.2 (osd.2) 182 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:55:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:43.710915+0000 osd.6 (osd.6) 310 : cluster [DBG] 4.a scrub starts 2024-02-13T00:55:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:43.714699+0000 osd.6 (osd.6) 311 : cluster [DBG] 4.a scrub ok 2024-02-13T00:55:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:43.788878+0000 mon.a (mon.0) 3833 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-02-13T00:55:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:44.040255+0000 osd.1 (osd.1) 381 : cluster [DBG] 28.7 scrub starts 2024-02-13T00:55:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:44.042236+0000 osd.1 (osd.1) 382 : cluster [DBG] 28.7 scrub ok 2024-02-13T00:55:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:44.260792+0000 osd.5 (osd.5) 36 : cluster [DBG] 2.b scrub starts 2024-02-13T00:55:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:44.264715+0000 osd.5 (osd.5) 37 : cluster [DBG] 2.b scrub ok 2024-02-13T00:55:45.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:44.455753+0000 osd.3 (osd.3) 193 : cluster [DBG] 5.13 scrub starts 2024-02-13T00:55:45.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:44.458664+0000 osd.3 (osd.3) 194 : cluster [DBG] 5.13 scrub ok 2024-02-13T00:55:45.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:43.663455+0000 osd.2 (osd.2) 181 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:55:45.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:43.664755+0000 osd.2 (osd.2) 182 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:55:45.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:43.710915+0000 osd.6 (osd.6) 310 : cluster [DBG] 4.a scrub starts 2024-02-13T00:55:45.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:43.714699+0000 osd.6 (osd.6) 311 : cluster [DBG] 4.a scrub ok 2024-02-13T00:55:45.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:43.788878+0000 mon.a (mon.0) 3833 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-02-13T00:55:45.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:44.040255+0000 osd.1 (osd.1) 381 : cluster [DBG] 28.7 scrub starts 2024-02-13T00:55:45.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:44.042236+0000 osd.1 (osd.1) 382 : cluster [DBG] 28.7 scrub ok 2024-02-13T00:55:45.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:44.260792+0000 osd.5 (osd.5) 36 : cluster [DBG] 2.b scrub starts 2024-02-13T00:55:45.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:44.264715+0000 osd.5 (osd.5) 37 : cluster [DBG] 2.b scrub ok 2024-02-13T00:55:45.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:44.455753+0000 osd.3 (osd.3) 193 : cluster [DBG] 5.13 scrub starts 2024-02-13T00:55:45.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:44.458664+0000 osd.3 (osd.3) 194 : cluster [DBG] 5.13 scrub ok 2024-02-13T00:55:45.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:43.663455+0000 osd.2 (osd.2) 181 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:55:45.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:43.664755+0000 osd.2 (osd.2) 182 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:55:45.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:43.710915+0000 osd.6 (osd.6) 310 : cluster [DBG] 4.a scrub starts 2024-02-13T00:55:45.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:43.714699+0000 osd.6 (osd.6) 311 : cluster [DBG] 4.a scrub ok 2024-02-13T00:55:45.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:43.788878+0000 mon.a (mon.0) 3833 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-02-13T00:55:45.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:44.040255+0000 osd.1 (osd.1) 381 : cluster [DBG] 28.7 scrub starts 2024-02-13T00:55:45.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:44.042236+0000 osd.1 (osd.1) 382 : cluster [DBG] 28.7 scrub ok 2024-02-13T00:55:45.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:44.260792+0000 osd.5 (osd.5) 36 : cluster [DBG] 2.b scrub starts 2024-02-13T00:55:45.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:44.264715+0000 osd.5 (osd.5) 37 : cluster [DBG] 2.b scrub ok 2024-02-13T00:55:45.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:44.455753+0000 osd.3 (osd.3) 193 : cluster [DBG] 5.13 scrub starts 2024-02-13T00:55:45.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:44.458664+0000 osd.3 (osd.3) 194 : cluster [DBG] 5.13 scrub ok 2024-02-13T00:55:45.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.672+0000 7fde45d9f700 1 -- 172.21.15.162:0/2172150178 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fde400f95e0 msgr2=0x7fde40101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:45.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.672+0000 7fde45d9f700 1 --2- 172.21.15.162:0/2172150178 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fde400f95e0 0x7fde40101d80 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 rx=0x7fde340047b0 tx=0x7fde34009cb0).stop 2024-02-13T00:55:45.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.672+0000 7fde45d9f700 1 -- 172.21.15.162:0/2172150178 shutdown_connections 2024-02-13T00:55:45.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.672+0000 7fde45d9f700 1 --2- 172.21.15.162:0/2172150178 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde40102270 0x7fde40104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:45.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.672+0000 7fde45d9f700 1 --2- 172.21.15.162:0/2172150178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde400f8d10 0x7fde400f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:45.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.672+0000 7fde45d9f700 1 --2- 172.21.15.162:0/2172150178 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fde400f95e0 0x7fde40101d80 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:45.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.672+0000 7fde45d9f700 1 -- 172.21.15.162:0/2172150178 >> 172.21.15.162:0/2172150178 conn(0x7fde4006eb90 msgr2=0x7fde400f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.672+0000 7fde45d9f700 1 -- 172.21.15.162:0/2172150178 shutdown_connections 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.672+0000 7fde45d9f700 1 -- 172.21.15.162:0/2172150178 wait complete. 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.673+0000 7fde45d9f700 1 librados: starting msgr at 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.673+0000 7fde45d9f700 1 librados: starting objecter 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.673+0000 7fde45d9f700 1 Processor -- start 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.673+0000 7fde45d9f700 1 -- start start 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.673+0000 7fde45d9f700 1 librados: setting wanted keys 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.673+0000 7fde45d9f700 1 librados: calling monclient init 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde45d9f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde400f8d10 0x7fde400f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde45d9f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde400f95e0 0x7fde400f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde45d9f700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fde40102270 0x7fde400f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde45d9f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde401167f0 con 0x7fde400f8d10 2024-02-13T00:55:45.680 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde45d9f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde40116960 con 0x7fde400f95e0 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde45d9f700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde40116b20 con 0x7fde40102270 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3f7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde400f8d10 0x7fde400f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3ffff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fde40102270 0x7fde400f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3f7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde400f8d10 0x7fde400f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:55910/0 (socket says 172.21.15.162:55910) 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3ffff700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fde40102270 0x7fde400f76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:32992/0 (socket says 172.21.15.162:32992) 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3f7fe700 1 -- 172.21.15.162:0/1127889046 learned_addr learned my addr 172.21.15.162:0/1127889046 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3effd700 1 --2- 172.21.15.162:0/1127889046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde400f95e0 0x7fde400f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3f7fe700 1 -- 172.21.15.162:0/1127889046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fde40102270 msgr2=0x7fde400f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3f7fe700 1 --2- 172.21.15.162:0/1127889046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fde40102270 0x7fde400f76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3f7fe700 1 -- 172.21.15.162:0/1127889046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde400f95e0 msgr2=0x7fde400f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3f7fe700 1 --2- 172.21.15.162:0/1127889046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde400f95e0 0x7fde400f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3f7fe700 1 -- 172.21.15.162:0/1127889046 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde30005040 con 0x7fde400f8d10 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3ffff700 1 --2- 172.21.15.162:0/1127889046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fde40102270 0x7fde400f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:45.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3f7fe700 1 --2- 172.21.15.162:0/1127889046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde400f8d10 0x7fde400f6cc0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7fde30009400 tx=0x7fde30009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:45.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3cff9700 10 client.?.objecter ms_handle_connect 0x7fde400f8d10 2024-02-13T00:55:45.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3cff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:45.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3cff9700 1 -- 172.21.15.162:0/1127889046 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde30007a40 con 0x7fde400f8d10 2024-02-13T00:55:45.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde45d9f700 1 -- 172.21.15.162:0/1127889046 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde34005040 con 0x7fde400f8d10 2024-02-13T00:55:45.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3cff9700 1 -- 172.21.15.162:0/1127889046 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde3000f040 con 0x7fde400f8d10 2024-02-13T00:55:45.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde3cff9700 1 -- 172.21.15.162:0/1127889046 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde30014a70 con 0x7fde400f8d10 2024-02-13T00:55:45.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde45d9f700 10 client.28308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:45.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde45d9f700 1 -- 172.21.15.162:0/1127889046 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde401171b0 con 0x7fde400f8d10 2024-02-13T00:55:45.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.674+0000 7fde45d9f700 1 librados: init done 2024-02-13T00:55:45.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.675+0000 7fde45d9f700 1 -- 172.21.15.162:0/1127889046 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde401071b0 con 0x7fde400f8d10 2024-02-13T00:55:45.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.676+0000 7fde3cff9700 1 -- 172.21.15.162:0/1127889046 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fde30010530 con 0x7fde400f8d10 2024-02-13T00:55:45.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.676+0000 7fde3cff9700 1 --2- 172.21.15.162:0/1127889046 >> v2:172.21.15.162:6800/2866534053 conn(0x7fde28062c90 0x7fde28065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:45.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.676+0000 7fde3cff9700 1 -- 172.21.15.162:0/1127889046 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(839..839 src has 449..839) v4 ==== 7393+0+0 (secure 0 0 0) 0x7fde3007cab0 con 0x7fde400f8d10 2024-02-13T00:55:45.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.676+0000 7fde3cff9700 10 client.28308.objecter ms_dispatch 0x7fde40001550 osd_map(839..839 src has 449..839) v4 2024-02-13T00:55:45.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.676+0000 7fde3cff9700 3 client.28308.objecter handle_osd_map got epochs [839,839] > 0 2024-02-13T00:55:45.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.676+0000 7fde3cff9700 3 client.28308.objecter handle_osd_map decoding full epoch 839 2024-02-13T00:55:45.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.676+0000 7fde3effd700 1 --2- 172.21.15.162:0/1127889046 >> v2:172.21.15.162:6800/2866534053 conn(0x7fde28062c90 0x7fde28065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:45.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.676+0000 7fde3cff9700 20 client.28308.objecter dump_active .. 0 homeless 2024-02-13T00:55:45.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.676+0000 7fde3cff9700 10 client.28308.objecter ms_handle_connect 0x7fde28062c90 2024-02-13T00:55:45.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.676+0000 7fde3effd700 1 --2- 172.21.15.162:0/1127889046 >> v2:172.21.15.162:6800/2866534053 conn(0x7fde28062c90 0x7fde28065150 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 rx=0x7fde34005010 tx=0x7fde34009cb0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:45.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.680+0000 7fde3cff9700 1 -- 172.21.15.162:0/1127889046 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fde30049f60 con 0x7fde400f8d10 2024-02-13T00:55:45.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:45.858+0000 7fde45d9f700 1 -- 172.21.15.162:0/1127889046 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fde4005fb80 con 0x7fde400f8d10 2024-02-13T00:55:46.067 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:44.577526+0000 mgr.y (mgr.14544) 718 : cluster [DBG] pgmap v1304: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:55:46.067 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:44.700129+0000 osd.6 (osd.6) 312 : cluster [DBG] 6.25 scrub starts 2024-02-13T00:55:46.067 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:44.703660+0000 osd.6 (osd.6) 313 : cluster [DBG] 6.25 scrub ok 2024-02-13T00:55:46.067 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:45.254778+0000 osd.5 (osd.5) 38 : cluster [DBG] 5.e scrub starts 2024-02-13T00:55:46.067 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:45.258361+0000 osd.5 (osd.5) 39 : cluster [DBG] 5.e scrub ok 2024-02-13T00:55:46.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:44.577526+0000 mgr.y (mgr.14544) 718 : cluster [DBG] pgmap v1304: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:55:46.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:44.700129+0000 osd.6 (osd.6) 312 : cluster [DBG] 6.25 scrub starts 2024-02-13T00:55:46.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:44.703660+0000 osd.6 (osd.6) 313 : cluster [DBG] 6.25 scrub ok 2024-02-13T00:55:46.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:45.254778+0000 osd.5 (osd.5) 38 : cluster [DBG] 5.e scrub starts 2024-02-13T00:55:46.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:45.258361+0000 osd.5 (osd.5) 39 : cluster [DBG] 5.e scrub ok 2024-02-13T00:55:46.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:44.577526+0000 mgr.y (mgr.14544) 718 : cluster [DBG] pgmap v1304: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:55:46.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:44.700129+0000 osd.6 (osd.6) 312 : cluster [DBG] 6.25 scrub starts 2024-02-13T00:55:46.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:44.703660+0000 osd.6 (osd.6) 313 : cluster [DBG] 6.25 scrub ok 2024-02-13T00:55:46.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:45.254778+0000 osd.5 (osd.5) 38 : cluster [DBG] 5.e scrub starts 2024-02-13T00:55:46.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:45.258361+0000 osd.5 (osd.5) 39 : cluster [DBG] 5.e scrub ok 2024-02-13T00:55:46.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:55:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:55:46.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:55:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:55:46] "GET /metrics HTTP/1.1" 200 185497 "" "Prometheus/2.33.4" 2024-02-13T00:55:46.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.799+0000 7fde3cff9700 1 -- 172.21.15.162:0/1127889046 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v840) v1 ==== 113+0+0 (secure 0 0 0) 0x7fde30050250 con 0x7fde400f8d10 2024-02-13T00:55:46.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.802+0000 7fde45d9f700 10 librados: watch_flush enter 2024-02-13T00:55:46.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.802+0000 7fde45d9f700 10 librados: watch_flush exit 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.802+0000 7fde45d9f700 20 client.28308.objecter shutdown clearing up homeless session... 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.802+0000 7fde45d9f700 10 client.28308.objecter successfully canceled tick 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.802+0000 7fde45d9f700 1 -- 172.21.15.162:0/1127889046 >> v2:172.21.15.162:6800/2866534053 conn(0x7fde28062c90 msgr2=0x7fde28065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.802+0000 7fde45d9f700 1 --2- 172.21.15.162:0/1127889046 >> v2:172.21.15.162:6800/2866534053 conn(0x7fde28062c90 0x7fde28065150 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 rx=0x7fde34005010 tx=0x7fde34009cb0).stop 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.802+0000 7fde45d9f700 1 -- 172.21.15.162:0/1127889046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde400f8d10 msgr2=0x7fde400f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.802+0000 7fde45d9f700 1 --2- 172.21.15.162:0/1127889046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde400f8d10 0x7fde400f6cc0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7fde30009400 tx=0x7fde30009cb0).stop 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.803+0000 7fde45d9f700 1 -- 172.21.15.162:0/1127889046 shutdown_connections 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.803+0000 7fde45d9f700 1 --2- 172.21.15.162:0/1127889046 >> v2:172.21.15.162:6800/2866534053 conn(0x7fde28062c90 0x7fde28065150 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.803+0000 7fde45d9f700 1 --2- 172.21.15.162:0/1127889046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fde40102270 0x7fde400f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.803+0000 7fde45d9f700 1 --2- 172.21.15.162:0/1127889046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde400f8d10 0x7fde400f6cc0 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.803+0000 7fde45d9f700 1 --2- 172.21.15.162:0/1127889046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde400f95e0 0x7fde400f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.803+0000 7fde45d9f700 1 -- 172.21.15.162:0/1127889046 >> 172.21.15.162:0/1127889046 conn(0x7fde4006eb90 msgr2=0x7fde40102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.803+0000 7fde45d9f700 1 -- 172.21.15.162:0/1127889046 shutdown_connections 2024-02-13T00:55:46.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.803+0000 7fde45d9f700 1 -- 172.21.15.162:0/1127889046 wait complete. 2024-02-13T00:55:46.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:46.803+0000 7fde45d9f700 1 librados: shutdown 2024-02-13T00:55:46.808 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.4 to 1 (10000) 2024-02-13T00:55:47.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:45.859800+0000 mon.a (mon.0) 3834 : audit [INF] from='client.? 172.21.15.162:0/1127889046' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:55:47.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:46.020269+0000 osd.1 (osd.1) 383 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:55:47.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:46.023263+0000 osd.1 (osd.1) 384 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:55:47.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:46.064080+0000 mgr.y (mgr.14544) 719 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:47.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:45.859800+0000 mon.a (mon.0) 3834 : audit [INF] from='client.? 172.21.15.162:0/1127889046' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:55:47.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:46.020269+0000 osd.1 (osd.1) 383 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:55:47.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:46.023263+0000 osd.1 (osd.1) 384 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:55:47.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:46.064080+0000 mgr.y (mgr.14544) 719 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:45.859800+0000 mon.a (mon.0) 3834 : audit [INF] from='client.? 172.21.15.162:0/1127889046' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:55:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:46.020269+0000 osd.1 (osd.1) 383 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:55:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:46.023263+0000 osd.1 (osd.1) 384 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:55:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:46.064080+0000 mgr.y (mgr.14544) 719 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:47.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 4 1 2024-02-13T00:55:47.822 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:48.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:46.578325+0000 mgr.y (mgr.14544) 720 : cluster [DBG] pgmap v1305: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:55:48.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:46.799182+0000 mon.a (mon.0) 3835 : audit [INF] from='client.? 172.21.15.162:0/1127889046' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-13T00:55:48.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:46.799270+0000 mon.a (mon.0) 3836 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-02-13T00:55:48.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:47.044674+0000 osd.1 (osd.1) 385 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:55:48.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:47.047754+0000 osd.1 (osd.1) 386 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:55:48.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:46.578325+0000 mgr.y (mgr.14544) 720 : cluster [DBG] pgmap v1305: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:55:48.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:46.799182+0000 mon.a (mon.0) 3835 : audit [INF] from='client.? 172.21.15.162:0/1127889046' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-13T00:55:48.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:46.799270+0000 mon.a (mon.0) 3836 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-02-13T00:55:48.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:47.044674+0000 osd.1 (osd.1) 385 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:55:48.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:47.047754+0000 osd.1 (osd.1) 386 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:55:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:46.578325+0000 mgr.y (mgr.14544) 720 : cluster [DBG] pgmap v1305: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:55:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:46.799182+0000 mon.a (mon.0) 3835 : audit [INF] from='client.? 172.21.15.162:0/1127889046' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-13T00:55:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:46.799270+0000 mon.a (mon.0) 3836 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-02-13T00:55:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:47.044674+0000 osd.1 (osd.1) 385 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:55:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:47.047754+0000 osd.1 (osd.1) 386 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:55:48.810 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:55:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:55:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:55:49.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:47.754676+0000 osd.6 (osd.6) 314 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:55:49.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:47.757603+0000 osd.6 (osd.6) 315 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:55:49.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:47.966121+0000 osd.0 (osd.0) 375 : cluster [DBG] 6.10 deep-scrub starts 2024-02-13T00:55:49.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:47.967063+0000 osd.0 (osd.0) 376 : cluster [DBG] 6.10 deep-scrub ok 2024-02-13T00:55:49.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:47.754676+0000 osd.6 (osd.6) 314 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:55:49.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:47.757603+0000 osd.6 (osd.6) 315 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:55:49.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:47.966121+0000 osd.0 (osd.0) 375 : cluster [DBG] 6.10 deep-scrub starts 2024-02-13T00:55:49.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:47.967063+0000 osd.0 (osd.0) 376 : cluster [DBG] 6.10 deep-scrub ok 2024-02-13T00:55:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:47.754676+0000 osd.6 (osd.6) 314 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:55:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:47.757603+0000 osd.6 (osd.6) 315 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:55:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:47.966121+0000 osd.0 (osd.0) 375 : cluster [DBG] 6.10 deep-scrub starts 2024-02-13T00:55:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:47.967063+0000 osd.0 (osd.0) 376 : cluster [DBG] 6.10 deep-scrub ok 2024-02-13T00:55:49.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.797+0000 7fae67b8e700 1 -- 172.21.15.162:0/2218375850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae600f7380 msgr2=0x7fae60103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:49.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.797+0000 7fae67b8e700 1 --2- 172.21.15.162:0/2218375850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae600f7380 0x7fae60103980 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7fae58009c30 tx=0x7fae58004c50).stop 2024-02-13T00:55:49.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 -- 172.21.15.162:0/2218375850 shutdown_connections 2024-02-13T00:55:49.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 --2- 172.21.15.162:0/2218375850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae600f7380 0x7fae60103980 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:49.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 --2- 172.21.15.162:0/2218375850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae600fcb10 0x7fae600fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:49.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 --2- 172.21.15.162:0/2218375850 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae600fd470 0x7fae600f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:49.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 -- 172.21.15.162:0/2218375850 >> 172.21.15.162:0/2218375850 conn(0x7fae600f2740 msgr2=0x7fae600f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:49.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 -- 172.21.15.162:0/2218375850 shutdown_connections 2024-02-13T00:55:49.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 -- 172.21.15.162:0/2218375850 wait complete. 2024-02-13T00:55:49.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 librados: starting msgr at 2024-02-13T00:55:49.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 librados: starting objecter 2024-02-13T00:55:49.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 Processor -- start 2024-02-13T00:55:49.805 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 -- start start 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 librados: setting wanted keys 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.798+0000 7fae67b8e700 1 librados: calling monclient init 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae67b8e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae600f7380 0x7fae6010ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae67b8e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae600fcb10 0x7fae6010d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae67b8e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae600fd470 0x7fae60107010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae67b8e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae60107500 con 0x7fae600f7380 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae67b8e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae60107670 con 0x7fae600fcb10 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae67b8e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae60107830 con 0x7fae600fd470 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae6592a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae600f7380 0x7fae6010ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae6612b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae600fd470 0x7fae60107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae6592a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae600f7380 0x7fae6010ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:55938/0 (socket says 172.21.15.162:55938) 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae6592a700 1 -- 172.21.15.162:0/2761520278 learned_addr learned my addr 172.21.15.162:0/2761520278 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae65129700 1 --2- 172.21.15.162:0/2761520278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae600fcb10 0x7fae6010d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:49.806 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae6612b700 1 -- 172.21.15.162:0/2761520278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae600fcb10 msgr2=0x7fae6010d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae6612b700 1 --2- 172.21.15.162:0/2761520278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae600fcb10 0x7fae6010d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae6612b700 1 -- 172.21.15.162:0/2761520278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae600f7380 msgr2=0x7fae6010ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae6612b700 1 --2- 172.21.15.162:0/2761520278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae600f7380 0x7fae6010ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae6612b700 1 -- 172.21.15.162:0/2761520278 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae50005040 con 0x7fae600fd470 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae6592a700 1 --2- 172.21.15.162:0/2761520278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae600f7380 0x7fae6010ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae6612b700 1 --2- 172.21.15.162:0/2761520278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae600fd470 0x7fae60107010 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 rx=0x7fae58009c30 tx=0x7fae58004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae56ffd700 10 client.?.objecter ms_handle_connect 0x7fae600fd470 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae56ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae56ffd700 1 -- 172.21.15.162:0/2761520278 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5800e040 con 0x7fae600fd470 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae67b8e700 1 -- 172.21.15.162:0/2761520278 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae580097c0 con 0x7fae600fd470 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae56ffd700 1 -- 172.21.15.162:0/2761520278 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae58007bf0 con 0x7fae600fd470 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae56ffd700 1 -- 172.21.15.162:0/2761520278 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae58019c90 con 0x7fae600fd470 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae67b8e700 10 client.38408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:49.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae67b8e700 1 -- 172.21.15.162:0/2761520278 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae60107d50 con 0x7fae600fd470 2024-02-13T00:55:49.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.799+0000 7fae67b8e700 1 librados: init done 2024-02-13T00:55:49.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.800+0000 7fae67b8e700 1 -- 172.21.15.162:0/2761520278 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae600607d0 con 0x7fae600fd470 2024-02-13T00:55:49.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.801+0000 7fae56ffd700 1 -- 172.21.15.162:0/2761520278 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fae58004020 con 0x7fae600fd470 2024-02-13T00:55:49.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.801+0000 7fae56ffd700 1 --2- 172.21.15.162:0/2761520278 >> v2:172.21.15.162:6800/2866534053 conn(0x7fae4c062c90 0x7fae4c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:49.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.801+0000 7fae56ffd700 1 -- 172.21.15.162:0/2761520278 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(840..840 src has 449..840) v4 ==== 7393+0+0 (secure 0 0 0) 0x7fae58082330 con 0x7fae600fd470 2024-02-13T00:55:49.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.801+0000 7fae56ffd700 10 client.38408.objecter ms_dispatch 0x7fae60001550 osd_map(840..840 src has 449..840) v4 2024-02-13T00:55:49.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.801+0000 7fae56ffd700 3 client.38408.objecter handle_osd_map got epochs [840,840] > 0 2024-02-13T00:55:49.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.801+0000 7fae56ffd700 3 client.38408.objecter handle_osd_map decoding full epoch 840 2024-02-13T00:55:49.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.801+0000 7fae6592a700 1 --2- 172.21.15.162:0/2761520278 >> v2:172.21.15.162:6800/2866534053 conn(0x7fae4c062c90 0x7fae4c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:49.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.801+0000 7fae56ffd700 20 client.38408.objecter dump_active .. 0 homeless 2024-02-13T00:55:49.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.802+0000 7fae56ffd700 10 client.38408.objecter ms_handle_connect 0x7fae4c062c90 2024-02-13T00:55:49.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.802+0000 7fae6592a700 1 --2- 172.21.15.162:0/2761520278 >> v2:172.21.15.162:6800/2866534053 conn(0x7fae4c062c90 0x7fae4c065150 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 rx=0x7fae5c0041d0 tx=0x7fae5c00a0f0).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:49.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.806+0000 7fae56ffd700 1 -- 172.21.15.162:0/2761520278 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fae58055ad0 con 0x7fae600fd470 2024-02-13T00:55:49.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:49.986+0000 7fae67b8e700 1 -- 172.21.15.162:0/2761520278 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fae601084a0 con 0x7fae600fd470 2024-02-13T00:55:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:48.068183+0000 osd.1 (osd.1) 387 : cluster [DBG] 2.6 deep-scrub starts 2024-02-13T00:55:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:48.069071+0000 osd.1 (osd.1) 388 : cluster [DBG] 2.6 deep-scrub ok 2024-02-13T00:55:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:48.578758+0000 mgr.y (mgr.14544) 721 : cluster [DBG] pgmap v1307: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:55:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:48.803621+0000 osd.6 (osd.6) 316 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:55:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:48.806515+0000 osd.6 (osd.6) 317 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:55:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:49.019109+0000 osd.1 (osd.1) 389 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:55:50.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:49.022795+0000 osd.1 (osd.1) 390 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:55:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:48.068183+0000 osd.1 (osd.1) 387 : cluster [DBG] 2.6 deep-scrub starts 2024-02-13T00:55:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:48.069071+0000 osd.1 (osd.1) 388 : cluster [DBG] 2.6 deep-scrub ok 2024-02-13T00:55:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:48.578758+0000 mgr.y (mgr.14544) 721 : cluster [DBG] pgmap v1307: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:55:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:48.803621+0000 osd.6 (osd.6) 316 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:55:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:48.806515+0000 osd.6 (osd.6) 317 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:55:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:49.019109+0000 osd.1 (osd.1) 389 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:55:50.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:49.022795+0000 osd.1 (osd.1) 390 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:55:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:48.068183+0000 osd.1 (osd.1) 387 : cluster [DBG] 2.6 deep-scrub starts 2024-02-13T00:55:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:48.069071+0000 osd.1 (osd.1) 388 : cluster [DBG] 2.6 deep-scrub ok 2024-02-13T00:55:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:48.578758+0000 mgr.y (mgr.14544) 721 : cluster [DBG] pgmap v1307: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:55:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:48.803621+0000 osd.6 (osd.6) 316 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:55:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:48.806515+0000 osd.6 (osd.6) 317 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:55:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:49.019109+0000 osd.1 (osd.1) 389 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:55:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:49.022795+0000 osd.1 (osd.1) 390 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:55:50.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.615+0000 7fae56ffd700 1 -- 172.21.15.162:0/2761520278 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v841) v1 ==== 133+0+0 (secure 0 0 0) 0x7fae580556c0 con 0x7fae600fd470 2024-02-13T00:55:50.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.623+0000 7fae67b8e700 10 librados: watch_flush enter 2024-02-13T00:55:50.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.624+0000 7fae67b8e700 10 librados: watch_flush exit 2024-02-13T00:55:50.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.624+0000 7fae67b8e700 20 client.38408.objecter shutdown clearing up homeless session... 2024-02-13T00:55:50.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.624+0000 7fae67b8e700 10 client.38408.objecter successfully canceled tick 2024-02-13T00:55:50.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.624+0000 7fae67b8e700 1 -- 172.21.15.162:0/2761520278 >> v2:172.21.15.162:6800/2866534053 conn(0x7fae4c062c90 msgr2=0x7fae4c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.624+0000 7fae67b8e700 1 --2- 172.21.15.162:0/2761520278 >> v2:172.21.15.162:6800/2866534053 conn(0x7fae4c062c90 0x7fae4c065150 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 rx=0x7fae5c0041d0 tx=0x7fae5c00a0f0).stop 2024-02-13T00:55:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.624+0000 7fae67b8e700 1 -- 172.21.15.162:0/2761520278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae600fd470 msgr2=0x7fae60107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:50.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.624+0000 7fae67b8e700 1 --2- 172.21.15.162:0/2761520278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae600fd470 0x7fae60107010 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 rx=0x7fae58009c30 tx=0x7fae58004c50).stop 2024-02-13T00:55:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.628+0000 7fae67b8e700 1 -- 172.21.15.162:0/2761520278 shutdown_connections 2024-02-13T00:55:50.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.628+0000 7fae67b8e700 1 --2- 172.21.15.162:0/2761520278 >> v2:172.21.15.162:6800/2866534053 conn(0x7fae4c062c90 0x7fae4c065150 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.628+0000 7fae67b8e700 1 --2- 172.21.15.162:0/2761520278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae600fd470 0x7fae60107010 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.628+0000 7fae67b8e700 1 --2- 172.21.15.162:0/2761520278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae600f7380 0x7fae6010ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.628+0000 7fae67b8e700 1 --2- 172.21.15.162:0/2761520278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae600fcb10 0x7fae6010d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.628+0000 7fae67b8e700 1 -- 172.21.15.162:0/2761520278 >> 172.21.15.162:0/2761520278 conn(0x7fae600f2740 msgr2=0x7fae600f3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.628+0000 7fae67b8e700 1 -- 172.21.15.162:0/2761520278 shutdown_connections 2024-02-13T00:55:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.628+0000 7fae67b8e700 1 -- 172.21.15.162:0/2761520278 wait complete. 2024-02-13T00:55:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:50.628+0000 7fae67b8e700 1 librados: shutdown 2024-02-13T00:55:50.633 INFO:teuthology.orchestra.run.smithi162.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-02-13T00:55:50.980 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:49.979651+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:55:50.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:49.982597+0000 osd.1 (osd.1) 392 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:55:50.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:49.988318+0000 mon.c (mon.2) 531 : audit [INF] from='client.? 172.21.15.162:0/2761520278' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:55:50.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:49.988636+0000 mon.a (mon.0) 3837 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:55:50.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:50.616060+0000 mon.a (mon.0) 3838 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-13T00:55:50.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:50.616147+0000 mon.a (mon.0) 3839 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-02-13T00:55:50.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:49.979651+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:55:50.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:49.982597+0000 osd.1 (osd.1) 392 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:55:50.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:49.988318+0000 mon.c (mon.2) 531 : audit [INF] from='client.? 172.21.15.162:0/2761520278' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:55:50.981 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:49.988636+0000 mon.a (mon.0) 3837 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:55:50.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:50.616060+0000 mon.a (mon.0) 3838 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-13T00:55:50.982 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:50.616147+0000 mon.a (mon.0) 3839 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-02-13T00:55:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:49.979651+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:55:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:49.982597+0000 osd.1 (osd.1) 392 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:55:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:49.988318+0000 mon.c (mon.2) 531 : audit [INF] from='client.? 172.21.15.162:0/2761520278' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:55:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:49.988636+0000 mon.a (mon.0) 3837 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-13T00:55:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:50.616060+0000 mon.a (mon.0) 3838 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-13T00:55:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:50.616147+0000 mon.a (mon.0) 3839 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-02-13T00:55:51.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 6 1 2024-02-13T00:55:51.663 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:52.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:50.579543+0000 mgr.y (mgr.14544) 722 : cluster [DBG] pgmap v1308: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:55:52.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:50.871339+0000 osd.6 (osd.6) 318 : cluster [DBG] 28.4 deep-scrub starts 2024-02-13T00:55:52.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:50.874218+0000 osd.6 (osd.6) 319 : cluster [DBG] 28.4 deep-scrub ok 2024-02-13T00:55:52.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:51.622882+0000 mon.a (mon.0) 3840 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-02-13T00:55:52.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:50.579543+0000 mgr.y (mgr.14544) 722 : cluster [DBG] pgmap v1308: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:55:52.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:50.871339+0000 osd.6 (osd.6) 318 : cluster [DBG] 28.4 deep-scrub starts 2024-02-13T00:55:52.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:50.874218+0000 osd.6 (osd.6) 319 : cluster [DBG] 28.4 deep-scrub ok 2024-02-13T00:55:52.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:51.622882+0000 mon.a (mon.0) 3840 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-02-13T00:55:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:50.579543+0000 mgr.y (mgr.14544) 722 : cluster [DBG] pgmap v1308: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:55:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:50.871339+0000 osd.6 (osd.6) 318 : cluster [DBG] 28.4 deep-scrub starts 2024-02-13T00:55:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:50.874218+0000 osd.6 (osd.6) 319 : cluster [DBG] 28.4 deep-scrub ok 2024-02-13T00:55:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:51.622882+0000 mon.a (mon.0) 3840 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-02-13T00:55:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:51.720817+0000 osd.2 (osd.2) 183 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:55:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:51.723721+0000 osd.2 (osd.2) 184 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:55:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:51.963467+0000 osd.1 (osd.1) 393 : cluster [DBG] 2.d scrub starts 2024-02-13T00:55:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:51.966484+0000 osd.1 (osd.1) 394 : cluster [DBG] 2.d scrub ok 2024-02-13T00:55:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:52.007636+0000 osd.0 (osd.0) 377 : cluster [DBG] 5.3 scrub starts 2024-02-13T00:55:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:52.011196+0000 osd.0 (osd.0) 378 : cluster [DBG] 5.3 scrub ok 2024-02-13T00:55:53.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:52.212746+0000 osd.5 (osd.5) 40 : cluster [DBG] 5.7 deep-scrub starts 2024-02-13T00:55:53.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:52.215709+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.7 deep-scrub ok 2024-02-13T00:55:53.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:52.586504+0000 osd.3 (osd.3) 195 : cluster [DBG] 4.b scrub starts 2024-02-13T00:55:53.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:52.590266+0000 osd.3 (osd.3) 196 : cluster [DBG] 4.b scrub ok 2024-02-13T00:55:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:51.720817+0000 osd.2 (osd.2) 183 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:55:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:51.723721+0000 osd.2 (osd.2) 184 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:55:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:51.963467+0000 osd.1 (osd.1) 393 : cluster [DBG] 2.d scrub starts 2024-02-13T00:55:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:51.966484+0000 osd.1 (osd.1) 394 : cluster [DBG] 2.d scrub ok 2024-02-13T00:55:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:52.007636+0000 osd.0 (osd.0) 377 : cluster [DBG] 5.3 scrub starts 2024-02-13T00:55:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:52.011196+0000 osd.0 (osd.0) 378 : cluster [DBG] 5.3 scrub ok 2024-02-13T00:55:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:52.212746+0000 osd.5 (osd.5) 40 : cluster [DBG] 5.7 deep-scrub starts 2024-02-13T00:55:53.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:52.215709+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.7 deep-scrub ok 2024-02-13T00:55:53.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:52.586504+0000 osd.3 (osd.3) 195 : cluster [DBG] 4.b scrub starts 2024-02-13T00:55:53.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:52.590266+0000 osd.3 (osd.3) 196 : cluster [DBG] 4.b scrub ok 2024-02-13T00:55:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:51.720817+0000 osd.2 (osd.2) 183 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:55:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:51.723721+0000 osd.2 (osd.2) 184 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:55:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:51.963467+0000 osd.1 (osd.1) 393 : cluster [DBG] 2.d scrub starts 2024-02-13T00:55:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:51.966484+0000 osd.1 (osd.1) 394 : cluster [DBG] 2.d scrub ok 2024-02-13T00:55:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:52.007636+0000 osd.0 (osd.0) 377 : cluster [DBG] 5.3 scrub starts 2024-02-13T00:55:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:52.011196+0000 osd.0 (osd.0) 378 : cluster [DBG] 5.3 scrub ok 2024-02-13T00:55:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:52.212746+0000 osd.5 (osd.5) 40 : cluster [DBG] 5.7 deep-scrub starts 2024-02-13T00:55:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:52.215709+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.7 deep-scrub ok 2024-02-13T00:55:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:52.586504+0000 osd.3 (osd.3) 195 : cluster [DBG] 4.b scrub starts 2024-02-13T00:55:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:52.590266+0000 osd.3 (osd.3) 196 : cluster [DBG] 4.b scrub ok 2024-02-13T00:55:53.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.870+0000 7f3137d6a700 1 -- 172.21.15.162:0/291543775 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31300fcb30 msgr2=0x7f31300fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:53.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.870+0000 7f3137d6a700 1 --2- 172.21.15.162:0/291543775 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31300fcb30 0x7f31300fcf10 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 rx=0x7f3124004870 tx=0x7f3124009cb0).stop 2024-02-13T00:55:53.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.870+0000 7f3137d6a700 1 -- 172.21.15.162:0/291543775 shutdown_connections 2024-02-13T00:55:53.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.870+0000 7f3137d6a700 1 --2- 172.21.15.162:0/291543775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31300f73a0 0x7f31301039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:53.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.870+0000 7f3137d6a700 1 --2- 172.21.15.162:0/291543775 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31300fcb30 0x7f31300fcf10 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:53.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.870+0000 7f3137d6a700 1 --2- 172.21.15.162:0/291543775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31300fd490 0x7f31300f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:53.877 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.870+0000 7f3137d6a700 1 -- 172.21.15.162:0/291543775 >> 172.21.15.162:0/291543775 conn(0x7f313006eb90 msgr2=0x7f31300f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.870+0000 7f3137d6a700 1 -- 172.21.15.162:0/291543775 shutdown_connections 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 -- 172.21.15.162:0/291543775 wait complete. 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 librados: starting msgr at 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 librados: starting objecter 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 Processor -- start 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 -- start start 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 librados: setting wanted keys 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 librados: calling monclient init 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31300f73a0 0x7f31301092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31300fcb30 0x7f31301097c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31300fd490 0x7f3130109d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3130114490 con 0x7f31300fd490 2024-02-13T00:55:53.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3130114600 con 0x7f31300fcb30 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3137d6a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31301147c0 con 0x7f31300f73a0 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3135b06700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31300f73a0 0x7f31301092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3136307700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31300fd490 0x7f3130109d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3136307700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31300fd490 0x7f3130109d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:55962/0 (socket says 172.21.15.162:55962) 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3135b06700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31300f73a0 0x7f31301092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:33010/0 (socket says 172.21.15.162:33010) 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.871+0000 7f3136307700 1 -- 172.21.15.162:0/847958880 learned_addr learned my addr 172.21.15.162:0/847958880 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3135305700 1 --2- 172.21.15.162:0/847958880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31300fcb30 0x7f31301097c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3136307700 1 -- 172.21.15.162:0/847958880 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31300f73a0 msgr2=0x7f31301092d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3136307700 1 --2- 172.21.15.162:0/847958880 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31300f73a0 0x7f31301092d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3136307700 1 -- 172.21.15.162:0/847958880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31300fcb30 msgr2=0x7f31301097c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3136307700 1 --2- 172.21.15.162:0/847958880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31300fcb30 0x7f31301097c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3136307700 1 -- 172.21.15.162:0/847958880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c005040 con 0x7f31300fd490 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3136307700 1 --2- 172.21.15.162:0/847958880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31300fd490 0x7f3130109d10 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f312800d970 tx=0x7f3128004e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3122ffd700 10 client.?.objecter ms_handle_connect 0x7f31300fd490 2024-02-13T00:55:53.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3122ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:53.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3122ffd700 1 -- 172.21.15.162:0/847958880 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3128004020 con 0x7f31300fd490 2024-02-13T00:55:53.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3137d6a700 1 -- 172.21.15.162:0/847958880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3124005040 con 0x7f31300fd490 2024-02-13T00:55:53.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3122ffd700 1 -- 172.21.15.162:0/847958880 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f312800f040 con 0x7f31300fd490 2024-02-13T00:55:53.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3122ffd700 1 -- 172.21.15.162:0/847958880 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31280036c0 con 0x7f31300fd490 2024-02-13T00:55:53.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3137d6a700 10 client.28320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:53.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3137d6a700 1 -- 172.21.15.162:0/847958880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3130114e30 con 0x7f31300fd490 2024-02-13T00:55:53.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.872+0000 7f3137d6a700 1 librados: init done 2024-02-13T00:55:53.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.873+0000 7f3137d6a700 1 -- 172.21.15.162:0/847958880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3130105fe0 con 0x7f31300fd490 2024-02-13T00:55:53.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.873+0000 7f3122ffd700 1 -- 172.21.15.162:0/847958880 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f3128003af0 con 0x7f31300fd490 2024-02-13T00:55:53.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.874+0000 7f3122ffd700 1 --2- 172.21.15.162:0/847958880 >> v2:172.21.15.162:6800/2866534053 conn(0x7f311c062c90 0x7f311c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:53.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.874+0000 7f3135b06700 1 --2- 172.21.15.162:0/847958880 >> v2:172.21.15.162:6800/2866534053 conn(0x7f311c062c90 0x7f311c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:53.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.874+0000 7f3122ffd700 1 -- 172.21.15.162:0/847958880 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(842..842 src has 449..842) v4 ==== 7393+0+0 (secure 0 0 0) 0x7f312807c960 con 0x7f31300fd490 2024-02-13T00:55:53.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.874+0000 7f3122ffd700 10 client.28320.objecter ms_dispatch 0x7f3130001550 osd_map(842..842 src has 449..842) v4 2024-02-13T00:55:53.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.874+0000 7f3122ffd700 3 client.28320.objecter handle_osd_map got epochs [842,842] > 0 2024-02-13T00:55:53.881 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.874+0000 7f3122ffd700 3 client.28320.objecter handle_osd_map decoding full epoch 842 2024-02-13T00:55:53.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.874+0000 7f3122ffd700 20 client.28320.objecter dump_active .. 0 homeless 2024-02-13T00:55:53.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.874+0000 7f3122ffd700 10 client.28320.objecter ms_handle_connect 0x7f311c062c90 2024-02-13T00:55:53.882 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.874+0000 7f3135b06700 1 --2- 172.21.15.162:0/847958880 >> v2:172.21.15.162:6800/2866534053 conn(0x7f311c062c90 0x7f311c065150 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 rx=0x7f31240043d0 tx=0x7f312400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:53.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:53.878+0000 7f3122ffd700 1 -- 172.21.15.162:0/847958880 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3128048f60 con 0x7f31300fd490 2024-02-13T00:55:54.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.056+0000 7f3137d6a700 1 -- 172.21.15.162:0/847958880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f313005fb80 con 0x7f31300fd490 2024-02-13T00:55:54.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:52.579922+0000 mgr.y (mgr.14544) 723 : cluster [DBG] pgmap v1311: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:55:54.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:52.735569+0000 osd.2 (osd.2) 185 : cluster [DBG] 5.d scrub starts 2024-02-13T00:55:54.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:52.737225+0000 osd.2 (osd.2) 186 : cluster [DBG] 5.d scrub ok 2024-02-13T00:55:54.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:53.366357+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:55:54.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:53.369819+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:55:54.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:53.586297+0000 osd.3 (osd.3) 197 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:55:54.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:53.589812+0000 osd.3 (osd.3) 198 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:55:54.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:52.579922+0000 mgr.y (mgr.14544) 723 : cluster [DBG] pgmap v1311: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:55:54.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:52.735569+0000 osd.2 (osd.2) 185 : cluster [DBG] 5.d scrub starts 2024-02-13T00:55:54.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:52.737225+0000 osd.2 (osd.2) 186 : cluster [DBG] 5.d scrub ok 2024-02-13T00:55:54.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:53.366357+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:55:54.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:53.369819+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:55:54.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:53.586297+0000 osd.3 (osd.3) 197 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:55:54.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:53.589812+0000 osd.3 (osd.3) 198 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:55:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:52.579922+0000 mgr.y (mgr.14544) 723 : cluster [DBG] pgmap v1311: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:55:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:52.735569+0000 osd.2 (osd.2) 185 : cluster [DBG] 5.d scrub starts 2024-02-13T00:55:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:52.737225+0000 osd.2 (osd.2) 186 : cluster [DBG] 5.d scrub ok 2024-02-13T00:55:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:53.366357+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:55:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:53.369819+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:55:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:53.586297+0000 osd.3 (osd.3) 197 : cluster [DBG] 4.19 scrub starts 2024-02-13T00:55:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:53.589812+0000 osd.3 (osd.3) 198 : cluster [DBG] 4.19 scrub ok 2024-02-13T00:55:54.837 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.832+0000 7f3122ffd700 1 -- 172.21.15.162:0/847958880 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v843) v1 ==== 113+0+0 (secure 0 0 0) 0x7f3128050100 con 0x7f31300fd490 2024-02-13T00:55:54.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.835+0000 7f3137d6a700 10 librados: watch_flush enter 2024-02-13T00:55:54.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.835+0000 7f3137d6a700 10 librados: watch_flush exit 2024-02-13T00:55:54.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.835+0000 7f3137d6a700 20 client.28320.objecter shutdown clearing up homeless session... 2024-02-13T00:55:54.840 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.835+0000 7f3137d6a700 10 client.28320.objecter successfully canceled tick 2024-02-13T00:55:54.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.836+0000 7f3137d6a700 1 -- 172.21.15.162:0/847958880 >> v2:172.21.15.162:6800/2866534053 conn(0x7f311c062c90 msgr2=0x7f311c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:54.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.836+0000 7f3137d6a700 1 --2- 172.21.15.162:0/847958880 >> v2:172.21.15.162:6800/2866534053 conn(0x7f311c062c90 0x7f311c065150 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 rx=0x7f31240043d0 tx=0x7f312400b040).stop 2024-02-13T00:55:54.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.836+0000 7f3137d6a700 1 -- 172.21.15.162:0/847958880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31300fd490 msgr2=0x7f3130109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:54.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.836+0000 7f3137d6a700 1 --2- 172.21.15.162:0/847958880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31300fd490 0x7f3130109d10 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f312800d970 tx=0x7f3128004e60).stop 2024-02-13T00:55:54.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.836+0000 7f3137d6a700 1 -- 172.21.15.162:0/847958880 shutdown_connections 2024-02-13T00:55:54.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.836+0000 7f3137d6a700 1 --2- 172.21.15.162:0/847958880 >> v2:172.21.15.162:6800/2866534053 conn(0x7f311c062c90 0x7f311c065150 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:54.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.836+0000 7f3137d6a700 1 --2- 172.21.15.162:0/847958880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31300fd490 0x7f3130109d10 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:54.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.836+0000 7f3137d6a700 1 --2- 172.21.15.162:0/847958880 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31300f73a0 0x7f31301092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:54.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.836+0000 7f3137d6a700 1 --2- 172.21.15.162:0/847958880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31300fcb30 0x7f31301097c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:54.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.836+0000 7f3137d6a700 1 -- 172.21.15.162:0/847958880 >> 172.21.15.162:0/847958880 conn(0x7f313006eb90 msgr2=0x7f31300fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:54.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.837+0000 7f3137d6a700 1 -- 172.21.15.162:0/847958880 shutdown_connections 2024-02-13T00:55:54.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.837+0000 7f3137d6a700 1 -- 172.21.15.162:0/847958880 wait complete. 2024-02-13T00:55:54.842 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:54.837+0000 7f3137d6a700 1 librados: shutdown 2024-02-13T00:55:54.842 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.6 to 1 (10000) 2024-02-13T00:55:55.255 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:53.728647+0000 osd.2 (osd.2) 187 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:55:55.255 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:53.729893+0000 osd.2 (osd.2) 188 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:55:55.255 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:53.930906+0000 osd.1 (osd.1) 395 : cluster [DBG] 5.c scrub starts 2024-02-13T00:55:55.255 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:53.933985+0000 osd.1 (osd.1) 396 : cluster [DBG] 5.c scrub ok 2024-02-13T00:55:55.255 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:53.940163+0000 osd.0 (osd.0) 379 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:55:55.256 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:53.943594+0000 osd.0 (osd.0) 380 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:55:55.256 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:54.058234+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? 172.21.15.162:0/847958880' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:55:55.256 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:54.190703+0000 osd.5 (osd.5) 42 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:55:55.256 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:54.194237+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:55:55.256 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:54.372633+0000 osd.4 (osd.4) 67 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:55:55.256 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:54.374052+0000 osd.4 (osd.4) 68 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:55:55.256 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:54.600197+0000 osd.3 (osd.3) 199 : cluster [DBG] 6.b scrub starts 2024-02-13T00:55:55.256 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:54.603525+0000 osd.3 (osd.3) 200 : cluster [DBG] 6.b scrub ok 2024-02-13T00:55:55.256 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:53.728647+0000 osd.2 (osd.2) 187 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:55:55.256 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:53.729893+0000 osd.2 (osd.2) 188 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:55:55.256 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:53.930906+0000 osd.1 (osd.1) 395 : cluster [DBG] 5.c scrub starts 2024-02-13T00:55:55.257 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:53.933985+0000 osd.1 (osd.1) 396 : cluster [DBG] 5.c scrub ok 2024-02-13T00:55:55.257 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:53.940163+0000 osd.0 (osd.0) 379 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:55:55.257 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:53.943594+0000 osd.0 (osd.0) 380 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:55:55.257 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:54.058234+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? 172.21.15.162:0/847958880' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:55:55.257 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:54.190703+0000 osd.5 (osd.5) 42 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:55:55.257 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:54.194237+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:55:55.257 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:54.372633+0000 osd.4 (osd.4) 67 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:55:55.257 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:54.374052+0000 osd.4 (osd.4) 68 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:55:55.257 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:54.600197+0000 osd.3 (osd.3) 199 : cluster [DBG] 6.b scrub starts 2024-02-13T00:55:55.257 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:54.603525+0000 osd.3 (osd.3) 200 : cluster [DBG] 6.b scrub ok 2024-02-13T00:55:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:53.728647+0000 osd.2 (osd.2) 187 : cluster [DBG] 5.4 scrub starts 2024-02-13T00:55:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:53.729893+0000 osd.2 (osd.2) 188 : cluster [DBG] 5.4 scrub ok 2024-02-13T00:55:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:53.930906+0000 osd.1 (osd.1) 395 : cluster [DBG] 5.c scrub starts 2024-02-13T00:55:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:53.933985+0000 osd.1 (osd.1) 396 : cluster [DBG] 5.c scrub ok 2024-02-13T00:55:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:53.940163+0000 osd.0 (osd.0) 379 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:55:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:53.943594+0000 osd.0 (osd.0) 380 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:55:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:54.058234+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? 172.21.15.162:0/847958880' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:55:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:54.190703+0000 osd.5 (osd.5) 42 : cluster [DBG] 5.15 scrub starts 2024-02-13T00:55:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:54.194237+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.15 scrub ok 2024-02-13T00:55:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:54.372633+0000 osd.4 (osd.4) 67 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:55:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:54.374052+0000 osd.4 (osd.4) 68 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:55:55.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:54.600197+0000 osd.3 (osd.3) 199 : cluster [DBG] 6.b scrub starts 2024-02-13T00:55:55.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:54.603525+0000 osd.3 (osd.3) 200 : cluster [DBG] 6.b scrub ok 2024-02-13T00:55:55.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 6 1 2024-02-13T00:55:55.847 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:54.580348+0000 mgr.y (mgr.14544) 724 : cluster [DBG] pgmap v1312: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 657 B/s rd, 0 op/s 2024-02-13T00:55:55.848 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:54.834481+0000 mon.a (mon.0) 3842 : audit [INF] from='client.? 172.21.15.162:0/847958880' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:55:55.848 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:54.834526+0000 mon.a (mon.0) 3843 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-02-13T00:55:55.848 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:55.602336+0000 osd.3 (osd.3) 201 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:55:55.848 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:55.606698+0000 osd.3 (osd.3) 202 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:55:55.895 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:55:56.164 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:54.580348+0000 mgr.y (mgr.14544) 724 : cluster [DBG] pgmap v1312: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 657 B/s rd, 0 op/s 2024-02-13T00:55:56.164 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:54.834481+0000 mon.a (mon.0) 3842 : audit [INF] from='client.? 172.21.15.162:0/847958880' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:55:56.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:54.834526+0000 mon.a (mon.0) 3843 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-02-13T00:55:56.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:55.602336+0000 osd.3 (osd.3) 201 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:55:56.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:55.606698+0000 osd.3 (osd.3) 202 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:55:56.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:55:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:55:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:54.580348+0000 mgr.y (mgr.14544) 724 : cluster [DBG] pgmap v1312: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 657 B/s rd, 0 op/s 2024-02-13T00:55:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:54.834481+0000 mon.a (mon.0) 3842 : audit [INF] from='client.? 172.21.15.162:0/847958880' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:55:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:54.834526+0000 mon.a (mon.0) 3843 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-02-13T00:55:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:55.602336+0000 osd.3 (osd.3) 201 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:55:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:55.606698+0000 osd.3 (osd.3) 202 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:55:56.457 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:55:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:55:56] "GET /metrics HTTP/1.1" 200 185510 "" "Prometheus/2.33.4" 2024-02-13T00:55:57.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:55.893745+0000 osd.0 (osd.0) 381 : cluster [DBG] 28.1f scrub starts 2024-02-13T00:55:57.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:55.897897+0000 osd.0 (osd.0) 382 : cluster [DBG] 28.1f scrub ok 2024-02-13T00:55:57.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:55.952700+0000 osd.1 (osd.1) 397 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:55:57.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:55.954711+0000 osd.1 (osd.1) 398 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:55:57.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:56.074360+0000 mgr.y (mgr.14544) 725 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:57.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:55.893745+0000 osd.0 (osd.0) 381 : cluster [DBG] 28.1f scrub starts 2024-02-13T00:55:57.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:55.897897+0000 osd.0 (osd.0) 382 : cluster [DBG] 28.1f scrub ok 2024-02-13T00:55:57.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:55.952700+0000 osd.1 (osd.1) 397 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:55:57.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:55.954711+0000 osd.1 (osd.1) 398 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:55:57.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:56.074360+0000 mgr.y (mgr.14544) 725 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:55.893745+0000 osd.0 (osd.0) 381 : cluster [DBG] 28.1f scrub starts 2024-02-13T00:55:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:55.897897+0000 osd.0 (osd.0) 382 : cluster [DBG] 28.1f scrub ok 2024-02-13T00:55:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:55.952700+0000 osd.1 (osd.1) 397 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:55:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:55.954711+0000 osd.1 (osd.1) 398 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:55:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:56.074360+0000 mgr.y (mgr.14544) 725 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:55:58.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.108+0000 7f0bc1833700 1 -- 172.21.15.162:0/2210663458 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bbc0fd470 msgr2=0x7f0bbc0f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:58.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.108+0000 7f0bc1833700 1 --2- 172.21.15.162:0/2210663458 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bbc0fd470 0x7f0bbc0f6ca0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 rx=0x7f0bb00047b0 tx=0x7f0bb0009cb0).stop 2024-02-13T00:55:58.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 -- 172.21.15.162:0/2210663458 shutdown_connections 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 --2- 172.21.15.162:0/2210663458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bbc0f7380 0x7f0bbc103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 --2- 172.21.15.162:0/2210663458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bbc0fcb10 0x7f0bbc0fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 --2- 172.21.15.162:0/2210663458 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bbc0fd470 0x7f0bbc0f6ca0 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 -- 172.21.15.162:0/2210663458 >> 172.21.15.162:0/2210663458 conn(0x7f0bbc0f2720 msgr2=0x7f0bbc0f4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 -- 172.21.15.162:0/2210663458 shutdown_connections 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 -- 172.21.15.162:0/2210663458 wait complete. 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 librados: starting msgr at 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 librados: starting objecter 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 Processor -- start 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 -- start start 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 librados: setting wanted keys 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.109+0000 7f0bc1833700 1 librados: calling monclient init 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bc1833700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bbc0f7380 0x7f0bbc10d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bc1833700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bbc0fcb10 0x7f0bbc10d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bc1833700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bbc0fd470 0x7f0bbc1071e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bc1833700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc1076d0 con 0x7f0bbc0fcb10 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bc1833700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc107840 con 0x7f0bbc0f7380 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bc1833700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc107a00 con 0x7f0bbc0fd470 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bbb7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bbc0fd470 0x7f0bbc1071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bba7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bbc0fcb10 0x7f0bbc10d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bbb7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bbc0fd470 0x7f0bbc1071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:57830/0 (socket says 172.21.15.162:57830) 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bbb7fe700 1 -- 172.21.15.162:0/4156810367 learned_addr learned my addr 172.21.15.162:0/4156810367 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bbaffd700 1 --2- 172.21.15.162:0/4156810367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bbc0f7380 0x7f0bbc10d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bbb7fe700 1 -- 172.21.15.162:0/4156810367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bbc0f7380 msgr2=0x7f0bbc10d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bbb7fe700 1 --2- 172.21.15.162:0/4156810367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bbc0f7380 0x7f0bbc10d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bbb7fe700 1 -- 172.21.15.162:0/4156810367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bbc0fcb10 msgr2=0x7f0bbc10d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:58.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bbb7fe700 1 --2- 172.21.15.162:0/4156810367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bbc0fcb10 0x7f0bbc10d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:58.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bbb7fe700 1 -- 172.21.15.162:0/4156810367 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac005040 con 0x7f0bbc0fd470 2024-02-13T00:55:58.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bba7fc700 1 --2- 172.21.15.162:0/4156810367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bbc0fcb10 0x7f0bbc10d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:55:58.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bbb7fe700 1 --2- 172.21.15.162:0/4156810367 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bbc0fd470 0x7f0bbc1071e0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 rx=0x7f0bb400d970 tx=0x7f0bb4004e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:58.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0ba3fff700 10 client.?.objecter ms_handle_connect 0x7f0bbc0fd470 2024-02-13T00:55:58.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0ba3fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:55:58.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0ba3fff700 1 -- 172.21.15.162:0/4156810367 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4004020 con 0x7f0bbc0fd470 2024-02-13T00:55:58.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0ba3fff700 1 -- 172.21.15.162:0/4156810367 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bb400f040 con 0x7f0bbc0fd470 2024-02-13T00:55:58.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0ba3fff700 1 -- 172.21.15.162:0/4156810367 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb40036c0 con 0x7f0bbc0fd470 2024-02-13T00:55:58.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bc1833700 1 -- 172.21.15.162:0/4156810367 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb0005040 con 0x7f0bbc0fd470 2024-02-13T00:55:58.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bc1833700 10 client.38420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:55:58.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bc1833700 1 -- 172.21.15.162:0/4156810367 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc107ff0 con 0x7f0bbc0fd470 2024-02-13T00:55:58.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.110+0000 7f0bc1833700 1 librados: init done 2024-02-13T00:55:58.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.111+0000 7f0bc1833700 1 -- 172.21.15.162:0/4156810367 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc0607f0 con 0x7f0bbc0fd470 2024-02-13T00:55:58.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.112+0000 7f0ba3fff700 1 -- 172.21.15.162:0/4156810367 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0bb4003af0 con 0x7f0bbc0fd470 2024-02-13T00:55:58.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.112+0000 7f0ba3fff700 1 --2- 172.21.15.162:0/4156810367 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0ba4062c90 0x7f0ba4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:55:58.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.112+0000 7f0ba3fff700 1 -- 172.21.15.162:0/4156810367 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(843..843 src has 449..843) v4 ==== 7393+0+0 (secure 0 0 0) 0x7f0bb407c960 con 0x7f0bbc0fd470 2024-02-13T00:55:58.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.112+0000 7f0ba3fff700 10 client.38420.objecter ms_dispatch 0x7f0bbc001550 osd_map(843..843 src has 449..843) v4 2024-02-13T00:55:58.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.112+0000 7f0ba3fff700 3 client.38420.objecter handle_osd_map got epochs [843,843] > 0 2024-02-13T00:55:58.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.112+0000 7f0ba3fff700 3 client.38420.objecter handle_osd_map decoding full epoch 843 2024-02-13T00:55:58.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.112+0000 7f0bbaffd700 1 --2- 172.21.15.162:0/4156810367 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0ba4062c90 0x7f0ba4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:55:58.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.112+0000 7f0ba3fff700 20 client.38420.objecter dump_active .. 0 homeless 2024-02-13T00:55:58.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.113+0000 7f0ba3fff700 10 client.38420.objecter ms_handle_connect 0x7f0ba4062c90 2024-02-13T00:55:58.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.113+0000 7f0bbaffd700 1 --2- 172.21.15.162:0/4156810367 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0ba4062c90 0x7f0ba4065150 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 rx=0x7f0bac004870 tx=0x7f0bac00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:55:58.121 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.116+0000 7f0ba3fff700 1 -- 172.21.15.162:0/4156810367 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0bb4048f60 con 0x7f0bbc0fd470 2024-02-13T00:55:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:56.581310+0000 mgr.y (mgr.14544) 726 : cluster [DBG] pgmap v1314: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:55:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:56.631100+0000 osd.3 (osd.3) 203 : cluster [DBG] 4.0 deep-scrub starts 2024-02-13T00:55:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:56.638635+0000 osd.3 (osd.3) 204 : cluster [DBG] 4.0 deep-scrub ok 2024-02-13T00:55:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:56.945271+0000 osd.1 (osd.1) 399 : cluster [DBG] 4.f scrub starts 2024-02-13T00:55:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:56.946782+0000 osd.1 (osd.1) 400 : cluster [DBG] 4.f scrub ok 2024-02-13T00:55:58.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.292+0000 7f0bc1833700 1 -- 172.21.15.162:0/4156810367 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f0bbc1086c0 con 0x7f0bbc0fd470 2024-02-13T00:55:58.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:56.581310+0000 mgr.y (mgr.14544) 726 : cluster [DBG] pgmap v1314: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:55:58.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:56.631100+0000 osd.3 (osd.3) 203 : cluster [DBG] 4.0 deep-scrub starts 2024-02-13T00:55:58.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:56.638635+0000 osd.3 (osd.3) 204 : cluster [DBG] 4.0 deep-scrub ok 2024-02-13T00:55:58.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:56.945271+0000 osd.1 (osd.1) 399 : cluster [DBG] 4.f scrub starts 2024-02-13T00:55:58.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:56.946782+0000 osd.1 (osd.1) 400 : cluster [DBG] 4.f scrub ok 2024-02-13T00:55:58.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:56.581310+0000 mgr.y (mgr.14544) 726 : cluster [DBG] pgmap v1314: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:55:58.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:56.631100+0000 osd.3 (osd.3) 203 : cluster [DBG] 4.0 deep-scrub starts 2024-02-13T00:55:58.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:56.638635+0000 osd.3 (osd.3) 204 : cluster [DBG] 4.0 deep-scrub ok 2024-02-13T00:55:58.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:56.945271+0000 osd.1 (osd.1) 399 : cluster [DBG] 4.f scrub starts 2024-02-13T00:55:58.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:56.946782+0000 osd.1 (osd.1) 400 : cluster [DBG] 4.f scrub ok 2024-02-13T00:55:58.858 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.853+0000 7f0ba3fff700 1 -- 172.21.15.162:0/4156810367 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v844) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0bb4050100 con 0x7f0bbc0fd470 2024-02-13T00:55:58.859 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:55:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:55:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:55:58.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.855+0000 7f0bc1833700 10 librados: watch_flush enter 2024-02-13T00:55:58.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 10 librados: watch_flush exit 2024-02-13T00:55:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 20 client.38420.objecter shutdown clearing up homeless session... 2024-02-13T00:55:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 10 client.38420.objecter successfully canceled tick 2024-02-13T00:55:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 -- 172.21.15.162:0/4156810367 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0ba4062c90 msgr2=0x7f0ba4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 --2- 172.21.15.162:0/4156810367 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0ba4062c90 0x7f0ba4065150 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 rx=0x7f0bac004870 tx=0x7f0bac00b040).stop 2024-02-13T00:55:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 -- 172.21.15.162:0/4156810367 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bbc0fd470 msgr2=0x7f0bbc1071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:55:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 --2- 172.21.15.162:0/4156810367 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bbc0fd470 0x7f0bbc1071e0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 rx=0x7f0bb400d970 tx=0x7f0bb4004e60).stop 2024-02-13T00:55:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 -- 172.21.15.162:0/4156810367 shutdown_connections 2024-02-13T00:55:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 --2- 172.21.15.162:0/4156810367 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0ba4062c90 0x7f0ba4065150 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 --2- 172.21.15.162:0/4156810367 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bbc0fd470 0x7f0bbc1071e0 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 --2- 172.21.15.162:0/4156810367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bbc0f7380 0x7f0bbc10d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 --2- 172.21.15.162:0/4156810367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bbc0fcb10 0x7f0bbc10d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:55:58.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 -- 172.21.15.162:0/4156810367 >> 172.21.15.162:0/4156810367 conn(0x7f0bbc0f2720 msgr2=0x7f0bbc0fa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:55:58.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 -- 172.21.15.162:0/4156810367 shutdown_connections 2024-02-13T00:55:58.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 -- 172.21.15.162:0/4156810367 wait complete. 2024-02-13T00:55:58.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:55:58.856+0000 7f0bc1833700 1 librados: shutdown 2024-02-13T00:55:58.862 INFO:teuthology.orchestra.run.smithi162.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-02-13T00:55:59.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:57.805819+0000 osd.6 (osd.6) 320 : cluster [DBG] 28.0 scrub starts 2024-02-13T00:55:59.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:57.808885+0000 osd.6 (osd.6) 321 : cluster [DBG] 28.0 scrub ok 2024-02-13T00:55:59.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:57.947510+0000 osd.0 (osd.0) 383 : cluster [DBG] 2.13 deep-scrub starts 2024-02-13T00:55:59.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:57.950756+0000 osd.0 (osd.0) 384 : cluster [DBG] 2.13 deep-scrub ok 2024-02-13T00:55:59.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:58.294022+0000 mon.c (mon.2) 532 : audit [INF] from='client.? 172.21.15.162:0/4156810367' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:55:59.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:58.294360+0000 mon.a (mon.0) 3844 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:55:59.281 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:57.805819+0000 osd.6 (osd.6) 320 : cluster [DBG] 28.0 scrub starts 2024-02-13T00:55:59.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:57.808885+0000 osd.6 (osd.6) 321 : cluster [DBG] 28.0 scrub ok 2024-02-13T00:55:59.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:57.947510+0000 osd.0 (osd.0) 383 : cluster [DBG] 2.13 deep-scrub starts 2024-02-13T00:55:59.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:57.950756+0000 osd.0 (osd.0) 384 : cluster [DBG] 2.13 deep-scrub ok 2024-02-13T00:55:59.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:58.294022+0000 mon.c (mon.2) 532 : audit [INF] from='client.? 172.21.15.162:0/4156810367' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:55:59.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:58.294360+0000 mon.a (mon.0) 3844 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:55:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:57.805819+0000 osd.6 (osd.6) 320 : cluster [DBG] 28.0 scrub starts 2024-02-13T00:55:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:57.808885+0000 osd.6 (osd.6) 321 : cluster [DBG] 28.0 scrub ok 2024-02-13T00:55:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:57.947510+0000 osd.0 (osd.0) 383 : cluster [DBG] 2.13 deep-scrub starts 2024-02-13T00:55:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:57.950756+0000 osd.0 (osd.0) 384 : cluster [DBG] 2.13 deep-scrub ok 2024-02-13T00:55:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:58.294022+0000 mon.c (mon.2) 532 : audit [INF] from='client.? 172.21.15.162:0/4156810367' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:55:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:58.294360+0000 mon.a (mon.0) 3844 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-13T00:55:59.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 7 1 2024-02-13T00:55:59.992 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:58.581814+0000 mgr.y (mgr.14544) 727 : cluster [DBG] pgmap v1315: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-02-13T00:56:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:55:58.852195+0000 mon.a (mon.0) 3845 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:56:00.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:55:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:55:58.852281+0000 mon.a (mon.0) 3846 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-02-13T00:56:00.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:58.581814+0000 mgr.y (mgr.14544) 727 : cluster [DBG] pgmap v1315: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-02-13T00:56:00.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:55:58.852195+0000 mon.a (mon.0) 3845 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:56:00.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:55:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:55:58.852281+0000 mon.a (mon.0) 3846 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-02-13T00:56:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:58.581814+0000 mgr.y (mgr.14544) 727 : cluster [DBG] pgmap v1315: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-02-13T00:56:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:55:58.852195+0000 mon.a (mon.0) 3845 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-13T00:56:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:55:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:55:58.852281+0000 mon.a (mon.0) 3846 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-02-13T00:56:01.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.693+0000 7f2fd18dc700 1 -- 172.21.15.162:0/839060219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc0fcb10 msgr2=0x7f2fcc0fcef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:01.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.693+0000 7f2fd18dc700 1 --2- 172.21.15.162:0/839060219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc0fcb10 0x7f2fcc0fcef0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 rx=0x7f2fc0004870 tx=0x7f2fc0009cb0).stop 2024-02-13T00:56:01.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.693+0000 7f2fd18dc700 1 -- 172.21.15.162:0/839060219 shutdown_connections 2024-02-13T00:56:01.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.693+0000 7f2fd18dc700 1 --2- 172.21.15.162:0/839060219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fcc0f7380 0x7f2fcc103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:01.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.693+0000 7f2fd18dc700 1 --2- 172.21.15.162:0/839060219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc0fcb10 0x7f2fcc0fcef0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:01.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.693+0000 7f2fd18dc700 1 --2- 172.21.15.162:0/839060219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc0fd470 0x7f2fcc0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:01.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.693+0000 7f2fd18dc700 1 -- 172.21.15.162:0/839060219 >> 172.21.15.162:0/839060219 conn(0x7f2fcc0f2740 msgr2=0x7f2fcc0f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:01.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.693+0000 7f2fd18dc700 1 -- 172.21.15.162:0/839060219 shutdown_connections 2024-02-13T00:56:01.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.693+0000 7f2fd18dc700 1 -- 172.21.15.162:0/839060219 wait complete. 2024-02-13T00:56:01.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fd18dc700 1 librados: starting msgr at 2024-02-13T00:56:01.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fd18dc700 1 librados: starting objecter 2024-02-13T00:56:01.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fd18dc700 1 Processor -- start 2024-02-13T00:56:01.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fd18dc700 1 -- start start 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fd18dc700 1 librados: setting wanted keys 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fd18dc700 1 librados: calling monclient init 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fd18dc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fcc0f7380 0x7f2fcc10d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fd18dc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc0fcb10 0x7f2fcc10d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fd18dc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc0fd470 0x7f2fcc1072a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fd18dc700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc107790 con 0x7f2fcc0fcb10 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fd18dc700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc107900 con 0x7f2fcc0f7380 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fd18dc700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc107ac0 con 0x7f2fcc0fd470 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fca7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc0fcb10 0x7f2fcc10d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fca7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc0fcb10 0x7f2fcc10d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:55192/0 (socket says 172.21.15.162:55192) 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fcb7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc0fd470 0x7f2fcc1072a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fca7fc700 1 -- 172.21.15.162:0/1508556316 learned_addr learned my addr 172.21.15.162:0/1508556316 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:01.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.694+0000 7f2fcaffd700 1 --2- 172.21.15.162:0/1508556316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fcc0f7380 0x7f2fcc10d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fca7fc700 1 -- 172.21.15.162:0/1508556316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc0fd470 msgr2=0x7f2fcc1072a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fca7fc700 1 --2- 172.21.15.162:0/1508556316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc0fd470 0x7f2fcc1072a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fca7fc700 1 -- 172.21.15.162:0/1508556316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fcc0f7380 msgr2=0x7f2fcc10d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fca7fc700 1 --2- 172.21.15.162:0/1508556316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fcc0f7380 0x7f2fcc10d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fca7fc700 1 -- 172.21.15.162:0/1508556316 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc005040 con 0x7f2fcc0fcb10 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fca7fc700 1 --2- 172.21.15.162:0/1508556316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc0fcb10 0x7f2fcc10d550 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 rx=0x7f2fbc0048f0 tx=0x7f2fbc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fb3fff700 10 client.?.objecter ms_handle_connect 0x7f2fcc0fcb10 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fb3fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fb3fff700 1 -- 172.21.15.162:0/1508556316 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc007b00 con 0x7f2fcc0fcb10 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fb3fff700 1 -- 172.21.15.162:0/1508556316 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fbc00f040 con 0x7f2fcc0fcb10 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fd18dc700 1 -- 172.21.15.162:0/1508556316 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc0005040 con 0x7f2fcc0fcb10 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fb3fff700 1 -- 172.21.15.162:0/1508556316 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc014990 con 0x7f2fcc0fcb10 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fd18dc700 10 client.28332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:01.710 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fd18dc700 1 -- 172.21.15.162:0/1508556316 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc108100 con 0x7f2fcc0fcb10 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.695+0000 7f2fd18dc700 1 librados: init done 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.696+0000 7f2fd18dc700 1 -- 172.21.15.162:0/1508556316 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc0607b0 con 0x7f2fcc0fcb10 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.696+0000 7f2fb3fff700 1 -- 172.21.15.162:0/1508556316 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f2fbc010500 con 0x7f2fcc0fcb10 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.697+0000 7f2fb3fff700 1 --2- 172.21.15.162:0/1508556316 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2fb4062c90 0x7f2fb4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.697+0000 7f2fb3fff700 1 -- 172.21.15.162:0/1508556316 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(845..845 src has 449..845) v4 ==== 7393+0+0 (secure 0 0 0) 0x7f2fbc07ca70 con 0x7f2fcc0fcb10 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.697+0000 7f2fb3fff700 10 client.28332.objecter ms_dispatch 0x7f2fcc001550 osd_map(845..845 src has 449..845) v4 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.697+0000 7f2fb3fff700 3 client.28332.objecter handle_osd_map got epochs [845,845] > 0 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.697+0000 7f2fb3fff700 3 client.28332.objecter handle_osd_map decoding full epoch 845 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.697+0000 7f2fcaffd700 1 --2- 172.21.15.162:0/1508556316 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2fb4062c90 0x7f2fb4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.697+0000 7f2fb3fff700 20 client.28332.objecter dump_active .. 0 homeless 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.697+0000 7f2fb3fff700 10 client.28332.objecter ms_handle_connect 0x7f2fb4062c90 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.697+0000 7f2fcaffd700 1 --2- 172.21.15.162:0/1508556316 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2fb4062c90 0x7f2fb4065150 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 rx=0x7f2fc00043d0 tx=0x7f2fc000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:01.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.701+0000 7f2fb3fff700 1 -- 172.21.15.162:0/1508556316 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2fbc049f60 con 0x7f2fcc0fcb10 2024-02-13T00:56:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:00.582521+0000 mgr.y (mgr.14544) 728 : cluster [DBG] pgmap v1317: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:56:01.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:00.618197+0000 mon.a (mon.0) 3847 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-02-13T00:56:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:00.582521+0000 mgr.y (mgr.14544) 728 : cluster [DBG] pgmap v1317: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:56:01.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:00.618197+0000 mon.a (mon.0) 3847 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-02-13T00:56:01.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:01.875+0000 7f2fd18dc700 1 -- 172.21.15.162:0/1508556316 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f2fcc0475e0 con 0x7f2fcc0fcb10 2024-02-13T00:56:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:00.582521+0000 mgr.y (mgr.14544) 728 : cluster [DBG] pgmap v1317: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:56:02.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:00.618197+0000 mon.a (mon.0) 3847 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-02-13T00:56:02.627 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.622+0000 7f2fb3fff700 1 -- 172.21.15.162:0/1508556316 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v846) v1 ==== 113+0+0 (secure 0 0 0) 0x7f2fbc050210 con 0x7f2fcc0fcb10 2024-02-13T00:56:02.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 10 librados: watch_flush enter 2024-02-13T00:56:02.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 10 librados: watch_flush exit 2024-02-13T00:56:02.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 20 client.28332.objecter shutdown clearing up homeless session... 2024-02-13T00:56:02.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 10 client.28332.objecter successfully canceled tick 2024-02-13T00:56:02.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 1 -- 172.21.15.162:0/1508556316 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2fb4062c90 msgr2=0x7f2fb4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:02.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 1 --2- 172.21.15.162:0/1508556316 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2fb4062c90 0x7f2fb4065150 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 rx=0x7f2fc00043d0 tx=0x7f2fc000b040).stop 2024-02-13T00:56:02.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 1 -- 172.21.15.162:0/1508556316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc0fcb10 msgr2=0x7f2fcc10d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:02.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 1 --2- 172.21.15.162:0/1508556316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc0fcb10 0x7f2fcc10d550 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 rx=0x7f2fbc0048f0 tx=0x7f2fbc009cb0).stop 2024-02-13T00:56:02.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 1 -- 172.21.15.162:0/1508556316 shutdown_connections 2024-02-13T00:56:02.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 1 --2- 172.21.15.162:0/1508556316 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2fb4062c90 0x7f2fb4065150 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:02.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 1 --2- 172.21.15.162:0/1508556316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc0fd470 0x7f2fcc1072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:02.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 1 --2- 172.21.15.162:0/1508556316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fcc0f7380 0x7f2fcc10d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:02.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 1 --2- 172.21.15.162:0/1508556316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc0fcb10 0x7f2fcc10d550 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:02.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.624+0000 7f2fd18dc700 1 -- 172.21.15.162:0/1508556316 >> 172.21.15.162:0/1508556316 conn(0x7f2fcc0f2740 msgr2=0x7f2fcc0fa940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:02.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.625+0000 7f2fd18dc700 1 -- 172.21.15.162:0/1508556316 shutdown_connections 2024-02-13T00:56:02.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.625+0000 7f2fd18dc700 1 -- 172.21.15.162:0/1508556316 wait complete. 2024-02-13T00:56:02.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:02.625+0000 7f2fd18dc700 1 librados: shutdown 2024-02-13T00:56:02.630 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.7 to 1 (10000) 2024-02-13T00:56:03.044 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:00.685803+0000 osd.3 (osd.3) 205 : cluster [DBG] 2.1e scrub starts 2024-02-13T00:56:03.044 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:00.689100+0000 osd.3 (osd.3) 206 : cluster [DBG] 2.1e scrub ok 2024-02-13T00:56:03.044 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:01.708629+0000 osd.2 (osd.2) 189 : cluster [DBG] 28.13 scrub starts 2024-02-13T00:56:03.044 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:01.710305+0000 osd.2 (osd.2) 190 : cluster [DBG] 28.13 scrub ok 2024-02-13T00:56:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:01.877209+0000 mon.a (mon.0) 3848 : audit [INF] from='client.? 172.21.15.162:0/1508556316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-13T00:56:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:02.273525+0000 osd.5 (osd.5) 44 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:02.273694+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:03.045 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:00.685803+0000 osd.3 (osd.3) 205 : cluster [DBG] 2.1e scrub starts 2024-02-13T00:56:03.045 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:00.689100+0000 osd.3 (osd.3) 206 : cluster [DBG] 2.1e scrub ok 2024-02-13T00:56:03.045 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:01.708629+0000 osd.2 (osd.2) 189 : cluster [DBG] 28.13 scrub starts 2024-02-13T00:56:03.045 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:01.710305+0000 osd.2 (osd.2) 190 : cluster [DBG] 28.13 scrub ok 2024-02-13T00:56:03.045 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:01.877209+0000 mon.a (mon.0) 3848 : audit [INF] from='client.? 172.21.15.162:0/1508556316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-13T00:56:03.046 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:02.273525+0000 osd.5 (osd.5) 44 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:03.046 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:02.273694+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:03.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:00.685803+0000 osd.3 (osd.3) 205 : cluster [DBG] 2.1e scrub starts 2024-02-13T00:56:03.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:00.689100+0000 osd.3 (osd.3) 206 : cluster [DBG] 2.1e scrub ok 2024-02-13T00:56:03.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:01.708629+0000 osd.2 (osd.2) 189 : cluster [DBG] 28.13 scrub starts 2024-02-13T00:56:03.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:01.710305+0000 osd.2 (osd.2) 190 : cluster [DBG] 28.13 scrub ok 2024-02-13T00:56:03.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:01.877209+0000 mon.a (mon.0) 3848 : audit [INF] from='client.? 172.21.15.162:0/1508556316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-13T00:56:03.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:02.273525+0000 osd.5 (osd.5) 44 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:03.047 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:02.273694+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:03.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-13T00:56:03.671 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:03.897 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:02.582996+0000 mgr.y (mgr.14544) 729 : cluster [DBG] pgmap v1319: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:56:03.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:02.622073+0000 mon.a (mon.0) 3849 : audit [INF] from='client.? 172.21.15.162:0/1508556316' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-13T00:56:03.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:02.622152+0000 mon.a (mon.0) 3850 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-02-13T00:56:03.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:02.729886+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:56:03.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:02.733033+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:56:03.898 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:02.582996+0000 mgr.y (mgr.14544) 729 : cluster [DBG] pgmap v1319: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:56:03.898 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:02.622073+0000 mon.a (mon.0) 3849 : audit [INF] from='client.? 172.21.15.162:0/1508556316' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-13T00:56:03.898 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:02.622152+0000 mon.a (mon.0) 3850 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-02-13T00:56:03.898 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:02.729886+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:56:03.898 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:02.733033+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:56:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:02.582996+0000 mgr.y (mgr.14544) 729 : cluster [DBG] pgmap v1319: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-13T00:56:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:02.622073+0000 mon.a (mon.0) 3849 : audit [INF] from='client.? 172.21.15.162:0/1508556316' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-13T00:56:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:02.622152+0000 mon.a (mon.0) 3850 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-02-13T00:56:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:02.729886+0000 osd.2 (osd.2) 191 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:56:04.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:02.733033+0000 osd.2 (osd.2) 192 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:56:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:02.685533+0000 osd.3 (osd.3) 207 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:56:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:02.690632+0000 osd.3 (osd.3) 208 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:56:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:02.917597+0000 osd.6 (osd.6) 322 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:56:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:02.921041+0000 osd.6 (osd.6) 323 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:56:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:03.985620+0000 osd.0 (osd.0) 385 : cluster [DBG] 6.9 scrub starts 2024-02-13T00:56:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:03.986435+0000 osd.0 (osd.0) 386 : cluster [DBG] 6.9 scrub ok 2024-02-13T00:56:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:04.217943+0000 osd.5 (osd.5) 46 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:56:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:04.218905+0000 osd.5 (osd.5) 47 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:56:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:04.366868+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:56:05.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:04.369775+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:56:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:02.685533+0000 osd.3 (osd.3) 207 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:56:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:02.690632+0000 osd.3 (osd.3) 208 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:56:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:02.917597+0000 osd.6 (osd.6) 322 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:56:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:02.921041+0000 osd.6 (osd.6) 323 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:56:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:03.985620+0000 osd.0 (osd.0) 385 : cluster [DBG] 6.9 scrub starts 2024-02-13T00:56:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:03.986435+0000 osd.0 (osd.0) 386 : cluster [DBG] 6.9 scrub ok 2024-02-13T00:56:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:04.217943+0000 osd.5 (osd.5) 46 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:56:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:04.218905+0000 osd.5 (osd.5) 47 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:56:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:04.366868+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:56:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:04.369775+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:56:05.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:02.685533+0000 osd.3 (osd.3) 207 : cluster [DBG] 6.11 scrub starts 2024-02-13T00:56:05.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:02.690632+0000 osd.3 (osd.3) 208 : cluster [DBG] 6.11 scrub ok 2024-02-13T00:56:05.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:02.917597+0000 osd.6 (osd.6) 322 : cluster [DBG] 5.9 scrub starts 2024-02-13T00:56:05.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:02.921041+0000 osd.6 (osd.6) 323 : cluster [DBG] 5.9 scrub ok 2024-02-13T00:56:05.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:03.985620+0000 osd.0 (osd.0) 385 : cluster [DBG] 6.9 scrub starts 2024-02-13T00:56:05.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:03.986435+0000 osd.0 (osd.0) 386 : cluster [DBG] 6.9 scrub ok 2024-02-13T00:56:05.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:04.217943+0000 osd.5 (osd.5) 46 : cluster [DBG] 2.18 scrub starts 2024-02-13T00:56:05.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:04.218905+0000 osd.5 (osd.5) 47 : cluster [DBG] 2.18 scrub ok 2024-02-13T00:56:05.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:04.366868+0000 osd.4 (osd.4) 69 : cluster [DBG] 5.1 scrub starts 2024-02-13T00:56:05.082 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:04.369775+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.1 scrub ok 2024-02-13T00:56:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.699+0000 7f6cd90f1700 1 -- 172.21.15.162:0/2084463245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cd40f8cd0 msgr2=0x7f6cd40f90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.699+0000 7f6cd90f1700 1 --2- 172.21.15.162:0/2084463245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cd40f8cd0 0x7f6cd40f90b0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7f6cc4004870 tx=0x7f6cc4009cb0).stop 2024-02-13T00:56:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.699+0000 7f6cd90f1700 1 -- 172.21.15.162:0/2084463245 shutdown_connections 2024-02-13T00:56:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.699+0000 7f6cd90f1700 1 --2- 172.21.15.162:0/2084463245 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cd4102230 0x7f6cd4104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.699+0000 7f6cd90f1700 1 --2- 172.21.15.162:0/2084463245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cd40f8cd0 0x7f6cd40f90b0 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.699+0000 7f6cd90f1700 1 --2- 172.21.15.162:0/2084463245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cd40f95a0 0x7f6cd4101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.699+0000 7f6cd90f1700 1 -- 172.21.15.162:0/2084463245 >> 172.21.15.162:0/2084463245 conn(0x7f6cd40f2740 msgr2=0x7f6cd40f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.699+0000 7f6cd90f1700 1 -- 172.21.15.162:0/2084463245 shutdown_connections 2024-02-13T00:56:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.699+0000 7f6cd90f1700 1 -- 172.21.15.162:0/2084463245 wait complete. 2024-02-13T00:56:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd90f1700 1 librados: starting msgr at 2024-02-13T00:56:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd90f1700 1 librados: starting objecter 2024-02-13T00:56:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd90f1700 1 Processor -- start 2024-02-13T00:56:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd90f1700 1 -- start start 2024-02-13T00:56:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd90f1700 1 librados: setting wanted keys 2024-02-13T00:56:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd90f1700 1 librados: calling monclient init 2024-02-13T00:56:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd90f1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cd40f8cd0 0x7f6cd410e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:05.708 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd90f1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cd40f95a0 0x7f6cd410e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:05.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd90f1700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cd4102230 0x7f6cd41082f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:05.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd90f1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd41087e0 con 0x7f6cd40f8cd0 2024-02-13T00:56:05.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd90f1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd4108950 con 0x7f6cd40f95a0 2024-02-13T00:56:05.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd90f1700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd4108b10 con 0x7f6cd4102230 2024-02-13T00:56:05.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd2d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cd40f8cd0 0x7f6cd410e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:05.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd2d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cd40f8cd0 0x7f6cd410e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:40082/0 (socket says 172.21.15.162:40082) 2024-02-13T00:56:05.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd2d9d700 1 -- 172.21.15.162:0/1235046875 learned_addr learned my addr 172.21.15.162:0/1235046875 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:05.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.700+0000 7f6cd359e700 1 --2- 172.21.15.162:0/1235046875 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cd4102230 0x7f6cd41082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:05.709 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd259c700 1 --2- 172.21.15.162:0/1235046875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cd40f95a0 0x7f6cd410e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd2d9d700 1 -- 172.21.15.162:0/1235046875 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cd4102230 msgr2=0x7f6cd41082f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd2d9d700 1 --2- 172.21.15.162:0/1235046875 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cd4102230 0x7f6cd41082f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd2d9d700 1 -- 172.21.15.162:0/1235046875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cd40f95a0 msgr2=0x7f6cd410e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd2d9d700 1 --2- 172.21.15.162:0/1235046875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cd40f95a0 0x7f6cd410e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd2d9d700 1 -- 172.21.15.162:0/1235046875 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc4005040 con 0x7f6cd40f8cd0 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd259c700 1 --2- 172.21.15.162:0/1235046875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cd40f95a0 0x7f6cd410e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd359e700 1 --2- 172.21.15.162:0/1235046875 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cd4102230 0x7f6cd41082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd2d9d700 1 --2- 172.21.15.162:0/1235046875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cd40f8cd0 0x7f6cd410e150 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7f6cc4009eb0 tx=0x7f6cc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cbbfff700 10 client.?.objecter ms_handle_connect 0x7f6cd40f8cd0 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cbbfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cbbfff700 1 -- 172.21.15.162:0/1235046875 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc400e070 con 0x7f6cd40f8cd0 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd90f1700 1 -- 172.21.15.162:0/1235046875 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd4108d90 con 0x7f6cd40f8cd0 2024-02-13T00:56:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cbbfff700 1 -- 172.21.15.162:0/1235046875 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6cc4013da0 con 0x7f6cd40f8cd0 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cbbfff700 1 -- 172.21.15.162:0/1235046875 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4019070 con 0x7f6cd40f8cd0 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd90f1700 10 client.28338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd90f1700 1 -- 172.21.15.162:0/1235046875 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd4114f80 con 0x7f6cd40f8cd0 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.701+0000 7f6cd90f1700 1 librados: init done 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.702+0000 7f6cd90f1700 1 -- 172.21.15.162:0/1235046875 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd40607b0 con 0x7f6cd40f8cd0 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.702+0000 7f6cbbfff700 1 -- 172.21.15.162:0/1235046875 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6cc401e420 con 0x7f6cd40f8cd0 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.703+0000 7f6cbbfff700 1 --2- 172.21.15.162:0/1235046875 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6cbc062c90 0x7f6cbc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.703+0000 7f6cbbfff700 1 -- 172.21.15.162:0/1235046875 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(847..847 src has 449..847) v4 ==== 7393+0+0 (secure 0 0 0) 0x7f6cc407db00 con 0x7f6cd40f8cd0 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.703+0000 7f6cbbfff700 10 client.28338.objecter ms_dispatch 0x7f6cd4001550 osd_map(847..847 src has 449..847) v4 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.703+0000 7f6cbbfff700 3 client.28338.objecter handle_osd_map got epochs [847,847] > 0 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.703+0000 7f6cbbfff700 3 client.28338.objecter handle_osd_map decoding full epoch 847 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.703+0000 7f6cd259c700 1 --2- 172.21.15.162:0/1235046875 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6cbc062c90 0x7f6cbc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:05.712 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.703+0000 7f6cbbfff700 20 client.28338.objecter dump_active .. 0 homeless 2024-02-13T00:56:05.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.703+0000 7f6cbbfff700 10 client.28338.objecter ms_handle_connect 0x7f6cbc062c90 2024-02-13T00:56:05.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.703+0000 7f6cd259c700 1 --2- 172.21.15.162:0/1235046875 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6cbc062c90 0x7f6cbc065150 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 rx=0x7f6cd4109a90 tx=0x7f6cc8009e00).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:05.713 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.707+0000 7f6cbbfff700 1 -- 172.21.15.162:0/1235046875 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6cc40512a0 con 0x7f6cd40f8cd0 2024-02-13T00:56:05.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:03.931838+0000 osd.6 (osd.6) 324 : cluster [DBG] 28.b scrub starts 2024-02-13T00:56:05.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:03.935463+0000 osd.6 (osd.6) 325 : cluster [DBG] 28.b scrub ok 2024-02-13T00:56:05.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:04.583410+0000 mgr.y (mgr.14544) 730 : cluster [DBG] pgmap v1321: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:56:05.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:04.732078+0000 osd.3 (osd.3) 209 : cluster [DBG] 5.0 scrub starts 2024-02-13T00:56:05.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:04.735721+0000 osd.3 (osd.3) 210 : cluster [DBG] 5.0 scrub ok 2024-02-13T00:56:05.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:04.774468+0000 osd.2 (osd.2) 193 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:56:05.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:04.777358+0000 osd.2 (osd.2) 194 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:56:05.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:05.249397+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.a deep-scrub starts 2024-02-13T00:56:05.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:05.252290+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.a deep-scrub ok 2024-02-13T00:56:05.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:05.622217+0000 mon.a (mon.0) 3851 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-02-13T00:56:05.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:03.931838+0000 osd.6 (osd.6) 324 : cluster [DBG] 28.b scrub starts 2024-02-13T00:56:05.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:03.935463+0000 osd.6 (osd.6) 325 : cluster [DBG] 28.b scrub ok 2024-02-13T00:56:05.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:04.583410+0000 mgr.y (mgr.14544) 730 : cluster [DBG] pgmap v1321: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:56:05.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:04.732078+0000 osd.3 (osd.3) 209 : cluster [DBG] 5.0 scrub starts 2024-02-13T00:56:05.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:04.735721+0000 osd.3 (osd.3) 210 : cluster [DBG] 5.0 scrub ok 2024-02-13T00:56:05.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:04.774468+0000 osd.2 (osd.2) 193 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:56:05.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:04.777358+0000 osd.2 (osd.2) 194 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:56:05.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:05.249397+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.a deep-scrub starts 2024-02-13T00:56:05.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:05.252290+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.a deep-scrub ok 2024-02-13T00:56:05.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:05.622217+0000 mon.a (mon.0) 3851 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-02-13T00:56:05.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:05.887+0000 7f6cd90f1700 1 -- 172.21.15.162:0/1235046875 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f6cd4107130 con 0x7f6cd40f8cd0 2024-02-13T00:56:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:03.931838+0000 osd.6 (osd.6) 324 : cluster [DBG] 28.b scrub starts 2024-02-13T00:56:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:03.935463+0000 osd.6 (osd.6) 325 : cluster [DBG] 28.b scrub ok 2024-02-13T00:56:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:04.583410+0000 mgr.y (mgr.14544) 730 : cluster [DBG] pgmap v1321: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:56:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:04.732078+0000 osd.3 (osd.3) 209 : cluster [DBG] 5.0 scrub starts 2024-02-13T00:56:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:04.735721+0000 osd.3 (osd.3) 210 : cluster [DBG] 5.0 scrub ok 2024-02-13T00:56:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:04.774468+0000 osd.2 (osd.2) 193 : cluster [DBG] 2.1 scrub starts 2024-02-13T00:56:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:04.777358+0000 osd.2 (osd.2) 194 : cluster [DBG] 2.1 scrub ok 2024-02-13T00:56:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:05.249397+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.a deep-scrub starts 2024-02-13T00:56:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:05.252290+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.a deep-scrub ok 2024-02-13T00:56:06.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:05.622217+0000 mon.a (mon.0) 3851 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-02-13T00:56:06.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:56:06.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:56:06] "GET /metrics HTTP/1.1" 200 185524 "" "Prometheus/2.33.4" 2024-02-13T00:56:06.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.645+0000 7f6cbbfff700 1 -- 172.21.15.162:0/1235046875 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v848) v1 ==== 133+0+0 (secure 0 0 0) 0x7f6cc4023020 con 0x7f6cd40f8cd0 2024-02-13T00:56:06.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.649+0000 7f6cd90f1700 10 librados: watch_flush enter 2024-02-13T00:56:06.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 10 librados: watch_flush exit 2024-02-13T00:56:06.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 20 client.28338.objecter shutdown clearing up homeless session... 2024-02-13T00:56:06.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 10 client.28338.objecter successfully canceled tick 2024-02-13T00:56:06.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 1 -- 172.21.15.162:0/1235046875 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6cbc062c90 msgr2=0x7f6cbc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:06.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 1 --2- 172.21.15.162:0/1235046875 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6cbc062c90 0x7f6cbc065150 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 rx=0x7f6cd4109a90 tx=0x7f6cc8009e00).stop 2024-02-13T00:56:06.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 1 -- 172.21.15.162:0/1235046875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cd40f8cd0 msgr2=0x7f6cd410e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:06.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 1 --2- 172.21.15.162:0/1235046875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cd40f8cd0 0x7f6cd410e150 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7f6cc4009eb0 tx=0x7f6cc400b040).stop 2024-02-13T00:56:06.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 1 -- 172.21.15.162:0/1235046875 shutdown_connections 2024-02-13T00:56:06.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 1 --2- 172.21.15.162:0/1235046875 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6cbc062c90 0x7f6cbc065150 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:06.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 1 --2- 172.21.15.162:0/1235046875 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cd4102230 0x7f6cd41082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:06.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 1 --2- 172.21.15.162:0/1235046875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cd40f8cd0 0x7f6cd410e150 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:06.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 1 --2- 172.21.15.162:0/1235046875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cd40f95a0 0x7f6cd410e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:06.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.651+0000 7f6cd90f1700 1 -- 172.21.15.162:0/1235046875 >> 172.21.15.162:0/1235046875 conn(0x7f6cd40f2740 msgr2=0x7f6cd40f3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:06.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.652+0000 7f6cd90f1700 1 -- 172.21.15.162:0/1235046875 shutdown_connections 2024-02-13T00:56:06.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.652+0000 7f6cd90f1700 1 -- 172.21.15.162:0/1235046875 wait complete. 2024-02-13T00:56:06.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:06.655+0000 7f6cd90f1700 1 librados: shutdown 2024-02-13T00:56:06.662 INFO:teuthology.orchestra.run.smithi162.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-13T00:56:07.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:05.402538+0000 osd.6 (osd.6) 326 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:07.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:05.402773+0000 osd.6 (osd.6) 327 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:07.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:05.755391+0000 osd.3 (osd.3) 211 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:56:07.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:05.757834+0000 osd.3 (osd.3) 212 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:56:07.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:05.888407+0000 mon.a (mon.0) 3852 : audit [INF] from='client.? 172.21.15.162:0/1235046875' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-13T00:56:07.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.085538+0000 mgr.y (mgr.14544) 731 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:07.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.574942+0000 mon.c (mon.2) 533 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]: dispatch 2024-02-13T00:56:07.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.575218+0000 mon.c (mon.2) 534 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]: dispatch 2024-02-13T00:56:07.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.575405+0000 mon.a (mon.0) 3853 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]: dispatch 2024-02-13T00:56:07.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.575495+0000 mon.c (mon.2) 535 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-13T00:56:07.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.575781+0000 mon.c (mon.2) 536 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]}]: dispatch 2024-02-13T00:56:07.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.576022+0000 mon.a (mon.0) 3854 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]: dispatch 2024-02-13T00:56:07.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.576425+0000 mon.a (mon.0) 3855 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-13T00:56:07.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.576749+0000 mon.a (mon.0) 3856 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]}]: dispatch 2024-02-13T00:56:07.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:05.402538+0000 osd.6 (osd.6) 326 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:07.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:05.402773+0000 osd.6 (osd.6) 327 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:07.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:05.755391+0000 osd.3 (osd.3) 211 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:56:07.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:05.757834+0000 osd.3 (osd.3) 212 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:56:07.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:05.888407+0000 mon.a (mon.0) 3852 : audit [INF] from='client.? 172.21.15.162:0/1235046875' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-13T00:56:07.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.085538+0000 mgr.y (mgr.14544) 731 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:07.037 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.574942+0000 mon.c (mon.2) 533 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]: dispatch 2024-02-13T00:56:07.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.575218+0000 mon.c (mon.2) 534 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]: dispatch 2024-02-13T00:56:07.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.575405+0000 mon.a (mon.0) 3853 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]: dispatch 2024-02-13T00:56:07.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.575495+0000 mon.c (mon.2) 535 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-13T00:56:07.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.575781+0000 mon.c (mon.2) 536 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]}]: dispatch 2024-02-13T00:56:07.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.576022+0000 mon.a (mon.0) 3854 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]: dispatch 2024-02-13T00:56:07.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.576425+0000 mon.a (mon.0) 3855 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-13T00:56:07.038 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.576749+0000 mon.a (mon.0) 3856 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]}]: dispatch 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:05.402538+0000 osd.6 (osd.6) 326 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:05.402773+0000 osd.6 (osd.6) 327 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:05.755391+0000 osd.3 (osd.3) 211 : cluster [DBG] 6.1f scrub starts 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:05.757834+0000 osd.3 (osd.3) 212 : cluster [DBG] 6.1f scrub ok 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:05.888407+0000 mon.a (mon.0) 3852 : audit [INF] from='client.? 172.21.15.162:0/1235046875' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.085538+0000 mgr.y (mgr.14544) 731 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.574942+0000 mon.c (mon.2) 533 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]: dispatch 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.575218+0000 mon.c (mon.2) 534 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]: dispatch 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.575405+0000 mon.a (mon.0) 3853 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]: dispatch 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.575495+0000 mon.c (mon.2) 535 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.575781+0000 mon.c (mon.2) 536 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]}]: dispatch 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.576022+0000 mon.a (mon.0) 3854 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]: dispatch 2024-02-13T00:56:07.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.576425+0000 mon.a (mon.0) 3855 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-13T00:56:07.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.576749+0000 mon.a (mon.0) 3856 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]}]: dispatch 2024-02-13T00:56:07.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd reweight 5 1 2024-02-13T00:56:07.777 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:06.584013+0000 mgr.y (mgr.14544) 732 : cluster [DBG] pgmap v1323: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:56:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.645240+0000 mon.a (mon.0) 3857 : audit [INF] from='client.? 172.21.15.162:0/1235046875' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-13T00:56:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.645422+0000 mon.a (mon.0) 3858 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]': finished 2024-02-13T00:56:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.645532+0000 mon.a (mon.0) 3859 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]': finished 2024-02-13T00:56:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.645626+0000 mon.a (mon.0) 3860 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-02-13T00:56:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:06.645715+0000 mon.a (mon.0) 3861 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]}]': finished 2024-02-13T00:56:07.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:06.645799+0000 mon.a (mon.0) 3862 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-02-13T00:56:07.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:06.845855+0000 osd.4 (osd.4) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:07.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:06.846184+0000 osd.4 (osd.4) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:07.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:06.912403+0000 osd.6 (osd.6) 328 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:56:07.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:06.915446+0000 osd.6 (osd.6) 329 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:56:07.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:06.951268+0000 osd.0 (osd.0) 387 : cluster [DBG] 28.e scrub starts 2024-02-13T00:56:07.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:06.954232+0000 osd.0 (osd.0) 388 : cluster [DBG] 28.e scrub ok 2024-02-13T00:56:07.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:07.220391+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.24 scrub starts 2024-02-13T00:56:07.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:07.223396+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.24 scrub ok 2024-02-13T00:56:07.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:07.643604+0000 mon.a (mon.0) 3863 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-02-13T00:56:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:06.584013+0000 mgr.y (mgr.14544) 732 : cluster [DBG] pgmap v1323: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:56:07.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.645240+0000 mon.a (mon.0) 3857 : audit [INF] from='client.? 172.21.15.162:0/1235046875' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.645422+0000 mon.a (mon.0) 3858 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]': finished 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.645532+0000 mon.a (mon.0) 3859 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]': finished 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.645626+0000 mon.a (mon.0) 3860 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:06.645715+0000 mon.a (mon.0) 3861 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]}]': finished 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:06.645799+0000 mon.a (mon.0) 3862 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:06.845855+0000 osd.4 (osd.4) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:06.846184+0000 osd.4 (osd.4) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:06.912403+0000 osd.6 (osd.6) 328 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:06.915446+0000 osd.6 (osd.6) 329 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:06.951268+0000 osd.0 (osd.0) 387 : cluster [DBG] 28.e scrub starts 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:06.954232+0000 osd.0 (osd.0) 388 : cluster [DBG] 28.e scrub ok 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:07.220391+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.24 scrub starts 2024-02-13T00:56:07.832 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:07.223396+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.24 scrub ok 2024-02-13T00:56:07.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:07.643604+0000 mon.a (mon.0) 3863 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-02-13T00:56:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:06.584013+0000 mgr.y (mgr.14544) 732 : cluster [DBG] pgmap v1323: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:56:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.645240+0000 mon.a (mon.0) 3857 : audit [INF] from='client.? 172.21.15.162:0/1235046875' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-13T00:56:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.645422+0000 mon.a (mon.0) 3858 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 2]}]': finished 2024-02-13T00:56:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.645532+0000 mon.a (mon.0) 3859 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]': finished 2024-02-13T00:56:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.645626+0000 mon.a (mon.0) 3860 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-02-13T00:56:08.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:06.645715+0000 mon.a (mon.0) 3861 : audit [INF] from='mgr.14544 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]}]': finished 2024-02-13T00:56:08.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:06.645799+0000 mon.a (mon.0) 3862 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-02-13T00:56:08.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:06.845855+0000 osd.4 (osd.4) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:08.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:06.846184+0000 osd.4 (osd.4) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:08.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:06.912403+0000 osd.6 (osd.6) 328 : cluster [DBG] 5.2 scrub starts 2024-02-13T00:56:08.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:06.915446+0000 osd.6 (osd.6) 329 : cluster [DBG] 5.2 scrub ok 2024-02-13T00:56:08.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:06.951268+0000 osd.0 (osd.0) 387 : cluster [DBG] 28.e scrub starts 2024-02-13T00:56:08.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:06.954232+0000 osd.0 (osd.0) 388 : cluster [DBG] 28.e scrub ok 2024-02-13T00:56:08.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:07.220391+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.24 scrub starts 2024-02-13T00:56:08.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:07.223396+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.24 scrub ok 2024-02-13T00:56:08.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:07.643604+0000 mon.a (mon.0) 3863 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-02-13T00:56:09.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:07.119366+0000 osd.1 (osd.1) 401 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:09.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:07.119609+0000 osd.1 (osd.1) 402 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:09.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:56:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:56:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:56:09.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:07.119366+0000 osd.1 (osd.1) 401 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:09.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:07.119609+0000 osd.1 (osd.1) 402 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:07.119366+0000 osd.1 (osd.1) 401 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:09.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:07.119609+0000 osd.1 (osd.1) 402 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:09.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 -- 172.21.15.162:0/2783436389 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17fc0f9b70 msgr2=0x7f17fc0f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:09.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 --2- 172.21.15.162:0/2783436389 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17fc0f9b70 0x7f17fc0f9f50 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 rx=0x7f17f80047b0 tx=0x7f17f8009cb0).stop 2024-02-13T00:56:09.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 -- 172.21.15.162:0/2783436389 shutdown_connections 2024-02-13T00:56:09.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 --2- 172.21.15.162:0/2783436389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17fc0f8500 0x7f17fc104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:09.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 --2- 172.21.15.162:0/2783436389 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17fc0f9b70 0x7f17fc0f9f50 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:09.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 --2- 172.21.15.162:0/2783436389 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17fc0fa440 0x7f17fc0f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:09.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 -- 172.21.15.162:0/2783436389 >> 172.21.15.162:0/2783436389 conn(0x7f17fc0f38c0 msgr2=0x7f17fc0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:09.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 -- 172.21.15.162:0/2783436389 shutdown_connections 2024-02-13T00:56:09.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 -- 172.21.15.162:0/2783436389 wait complete. 2024-02-13T00:56:09.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 librados: starting msgr at 2024-02-13T00:56:09.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 librados: starting objecter 2024-02-13T00:56:09.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 Processor -- start 2024-02-13T00:56:09.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 -- start start 2024-02-13T00:56:09.654 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 librados: setting wanted keys 2024-02-13T00:56:09.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.647+0000 7f1804788700 1 librados: calling monclient init 2024-02-13T00:56:09.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1804788700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17fc0f8500 0x7f17fc10e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:09.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1804788700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17fc0f9b70 0x7f17fc10e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:09.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1804788700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17fc0fa440 0x7f17fc108390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:09.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1804788700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc108880 con 0x7f17fc0fa440 2024-02-13T00:56:09.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1804788700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc1089f0 con 0x7f17fc0f8500 2024-02-13T00:56:09.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1804788700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc108bb0 con 0x7f17fc0f9b70 2024-02-13T00:56:09.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1801d23700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17fc0f9b70 0x7f17fc10e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:09.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1802d25700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17fc0fa440 0x7f17fc108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:09.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1801d23700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17fc0f9b70 0x7f17fc10e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:57188/0 (socket says 172.21.15.162:57188) 2024-02-13T00:56:09.655 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1801d23700 1 -- 172.21.15.162:0/1067951258 learned_addr learned my addr 172.21.15.162:0/1067951258 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:09.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1802524700 1 --2- 172.21.15.162:0/1067951258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17fc0f8500 0x7f17fc10e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:09.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1801d23700 1 -- 172.21.15.162:0/1067951258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17fc0f8500 msgr2=0x7f17fc10e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:09.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1801d23700 1 --2- 172.21.15.162:0/1067951258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17fc0f8500 0x7f17fc10e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:09.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1801d23700 1 -- 172.21.15.162:0/1067951258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17fc0fa440 msgr2=0x7f17fc108390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:09.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1801d23700 1 --2- 172.21.15.162:0/1067951258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17fc0fa440 0x7f17fc108390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:09.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1801d23700 1 -- 172.21.15.162:0/1067951258 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f40097c0 con 0x7f17fc0f9b70 2024-02-13T00:56:09.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1802d25700 1 --2- 172.21.15.162:0/1067951258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17fc0fa440 0x7f17fc108390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:09.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f17f37fe700 10 client.?.objecter ms_handle_connect 0x7f17fc0f9b70 2024-02-13T00:56:09.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f17f37fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:09.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1801d23700 1 --2- 172.21.15.162:0/1067951258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17fc0f9b70 0x7f17fc10e6e0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 rx=0x7f17ec0027f0 tx=0x7f17ec002b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:09.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f17f37fe700 1 -- 172.21.15.162:0/1067951258 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec007b40 con 0x7f17fc0f9b70 2024-02-13T00:56:09.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f17f37fe700 1 -- 172.21.15.162:0/1067951258 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17ec00f040 con 0x7f17fc0f9b70 2024-02-13T00:56:09.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1804788700 1 -- 172.21.15.162:0/1067951258 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f8005040 con 0x7f17fc0f9b70 2024-02-13T00:56:09.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f17f37fe700 1 -- 172.21.15.162:0/1067951258 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec0146f0 con 0x7f17fc0f9b70 2024-02-13T00:56:09.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1804788700 10 client.38438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:09.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1804788700 1 -- 172.21.15.162:0/1067951258 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17fc1091f0 con 0x7f17fc0f9b70 2024-02-13T00:56:09.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.648+0000 7f1804788700 1 librados: init done 2024-02-13T00:56:09.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.649+0000 7f1804788700 1 -- 172.21.15.162:0/1067951258 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17fc0475e0 con 0x7f17fc0f9b70 2024-02-13T00:56:09.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.650+0000 7f17f37fe700 1 -- 172.21.15.162:0/1067951258 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f17ec009110 con 0x7f17fc0f9b70 2024-02-13T00:56:09.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.650+0000 7f17f37fe700 1 --2- 172.21.15.162:0/1067951258 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17e8062c90 0x7f17e8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:09.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.650+0000 7f17f37fe700 1 -- 172.21.15.162:0/1067951258 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(849..849 src has 449..849) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f17ec07ccf0 con 0x7f17fc0f9b70 2024-02-13T00:56:09.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.650+0000 7f17f37fe700 10 client.38438.objecter ms_dispatch 0x7f17fc001550 osd_map(849..849 src has 449..849) v4 2024-02-13T00:56:09.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.650+0000 7f17f37fe700 3 client.38438.objecter handle_osd_map got epochs [849,849] > 0 2024-02-13T00:56:09.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.650+0000 7f17f37fe700 3 client.38438.objecter handle_osd_map decoding full epoch 849 2024-02-13T00:56:09.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.650+0000 7f1802524700 1 --2- 172.21.15.162:0/1067951258 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17e8062c90 0x7f17e8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:09.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.650+0000 7f17f37fe700 20 client.38438.objecter dump_active .. 0 homeless 2024-02-13T00:56:09.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.651+0000 7f17f37fe700 10 client.38438.objecter ms_handle_connect 0x7f17e8062c90 2024-02-13T00:56:09.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.651+0000 7f1802524700 1 --2- 172.21.15.162:0/1067951258 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17e8062c90 0x7f17e8065150 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 rx=0x7f17f8005010 tx=0x7f17f8007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:09.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.655+0000 7f17f37fe700 1 -- 172.21.15.162:0/1067951258 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f17ec081070 con 0x7f17fc0f9b70 2024-02-13T00:56:09.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:09.832+0000 7f1804788700 1 -- 172.21.15.162:0/1067951258 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f17fc109870 con 0x7f17fc0f9b70 2024-02-13T00:56:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:08.584425+0000 mgr.y (mgr.14544) 733 : cluster [DBG] pgmap v1326: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-02-13T00:56:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:08.760496+0000 osd.3 (osd.3) 213 : cluster [DBG] 6.6 scrub starts 2024-02-13T00:56:10.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:08.763520+0000 osd.3 (osd.3) 214 : cluster [DBG] 6.6 scrub ok 2024-02-13T00:56:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:08.584425+0000 mgr.y (mgr.14544) 733 : cluster [DBG] pgmap v1326: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-02-13T00:56:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:08.760496+0000 osd.3 (osd.3) 213 : cluster [DBG] 6.6 scrub starts 2024-02-13T00:56:10.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:08.763520+0000 osd.3 (osd.3) 214 : cluster [DBG] 6.6 scrub ok 2024-02-13T00:56:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:08.584425+0000 mgr.y (mgr.14544) 733 : cluster [DBG] pgmap v1326: 164 pgs: 164 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-02-13T00:56:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:08.760496+0000 osd.3 (osd.3) 213 : cluster [DBG] 6.6 scrub starts 2024-02-13T00:56:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:08.763520+0000 osd.3 (osd.3) 214 : cluster [DBG] 6.6 scrub ok 2024-02-13T00:56:10.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.735+0000 7f17f37fe700 1 -- 172.21.15.162:0/1067951258 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v850) v1 ==== 113+0+0 (secure 0 0 0) 0x7f17ec0503c0 con 0x7f17fc0f9b70 2024-02-13T00:56:10.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 10 librados: watch_flush enter 2024-02-13T00:56:10.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 10 librados: watch_flush exit 2024-02-13T00:56:10.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 20 client.38438.objecter shutdown clearing up homeless session... 2024-02-13T00:56:10.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 10 client.38438.objecter successfully canceled tick 2024-02-13T00:56:10.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 -- 172.21.15.162:0/1067951258 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17e8062c90 msgr2=0x7f17e8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:10.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 --2- 172.21.15.162:0/1067951258 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17e8062c90 0x7f17e8065150 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 rx=0x7f17f8005010 tx=0x7f17f8007f90).stop 2024-02-13T00:56:10.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 -- 172.21.15.162:0/1067951258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17fc0f9b70 msgr2=0x7f17fc10e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:10.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 --2- 172.21.15.162:0/1067951258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17fc0f9b70 0x7f17fc10e6e0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 rx=0x7f17ec0027f0 tx=0x7f17ec002b00).stop 2024-02-13T00:56:10.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 -- 172.21.15.162:0/1067951258 shutdown_connections 2024-02-13T00:56:10.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 --2- 172.21.15.162:0/1067951258 >> v2:172.21.15.162:6800/2866534053 conn(0x7f17e8062c90 0x7f17e8065150 unknown :-1 s=CLOSED pgs=3078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:10.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 --2- 172.21.15.162:0/1067951258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17fc0fa440 0x7f17fc108390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:10.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 --2- 172.21.15.162:0/1067951258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17fc0f8500 0x7f17fc10e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:10.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 --2- 172.21.15.162:0/1067951258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17fc0f9b70 0x7f17fc10e6e0 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:10.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 -- 172.21.15.162:0/1067951258 >> 172.21.15.162:0/1067951258 conn(0x7f17fc0f38c0 msgr2=0x7f17fc0fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:10.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 -- 172.21.15.162:0/1067951258 shutdown_connections 2024-02-13T00:56:10.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 -- 172.21.15.162:0/1067951258 wait complete. 2024-02-13T00:56:10.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:10.738+0000 7f1804788700 1 librados: shutdown 2024-02-13T00:56:10.744 INFO:teuthology.orchestra.run.smithi162.stderr:reweighted osd.5 to 1 (10000) 2024-02-13T00:56:11.043 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:09.833748+0000 mon.c (mon.2) 537 : audit [INF] from='client.? 172.21.15.162:0/1067951258' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-13T00:56:11.043 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:09.834050+0000 mon.a (mon.0) 3864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-13T00:56:11.043 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:10.355010+0000 osd.4 (osd.4) 73 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:56:11.043 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:10.357942+0000 osd.4 (osd.4) 74 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:56:11.044 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:09.833748+0000 mon.c (mon.2) 537 : audit [INF] from='client.? 172.21.15.162:0/1067951258' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-13T00:56:11.044 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:09.834050+0000 mon.a (mon.0) 3864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-13T00:56:11.044 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:10.355010+0000 osd.4 (osd.4) 73 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:56:11.044 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:10.357942+0000 osd.4 (osd.4) 74 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:56:11.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:09.833748+0000 mon.c (mon.2) 537 : audit [INF] from='client.? 172.21.15.162:0/1067951258' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-13T00:56:11.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:09.834050+0000 mon.a (mon.0) 3864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-13T00:56:11.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:10.355010+0000 osd.4 (osd.4) 73 : cluster [DBG] 6.1a scrub starts 2024-02-13T00:56:11.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:10.357942+0000 osd.4 (osd.4) 74 : cluster [DBG] 6.1a scrub ok 2024-02-13T00:56:11.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 5 1 2024-02-13T00:56:11.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:10.585273+0000 mgr.y (mgr.14544) 734 : cluster [DBG] pgmap v1327: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-02-13T00:56:11.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:10.733915+0000 mon.a (mon.0) 3865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-13T00:56:11.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:10.733996+0000 mon.a (mon.0) 3866 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-02-13T00:56:11.737 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:10.769090+0000 osd.2 (osd.2) 195 : cluster [DBG] 4.11 deep-scrub starts 2024-02-13T00:56:11.737 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:10.788357+0000 osd.2 (osd.2) 196 : cluster [DBG] 4.11 deep-scrub ok 2024-02-13T00:56:11.737 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:10.847429+0000 osd.7 (osd.7) 120 : cluster [DBG] 6.5 scrub starts 2024-02-13T00:56:11.737 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:10.848349+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.5 scrub ok 2024-02-13T00:56:11.737 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:10.908838+0000 osd.1 (osd.1) 403 : cluster [DBG] 2.a scrub starts 2024-02-13T00:56:11.737 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:10.911722+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.a scrub ok 2024-02-13T00:56:11.737 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:11.196578+0000 osd.5 (osd.5) 52 : cluster [DBG] 4.8 scrub starts 2024-02-13T00:56:11.737 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:11.198782+0000 osd.5 (osd.5) 53 : cluster [DBG] 4.8 scrub ok 2024-02-13T00:56:11.756 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:12.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:10.585273+0000 mgr.y (mgr.14544) 734 : cluster [DBG] pgmap v1327: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-02-13T00:56:12.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:10.733915+0000 mon.a (mon.0) 3865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-13T00:56:12.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:10.733996+0000 mon.a (mon.0) 3866 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-02-13T00:56:12.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:10.769090+0000 osd.2 (osd.2) 195 : cluster [DBG] 4.11 deep-scrub starts 2024-02-13T00:56:12.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:10.788357+0000 osd.2 (osd.2) 196 : cluster [DBG] 4.11 deep-scrub ok 2024-02-13T00:56:12.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:10.847429+0000 osd.7 (osd.7) 120 : cluster [DBG] 6.5 scrub starts 2024-02-13T00:56:12.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:10.848349+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.5 scrub ok 2024-02-13T00:56:12.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:10.908838+0000 osd.1 (osd.1) 403 : cluster [DBG] 2.a scrub starts 2024-02-13T00:56:12.014 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:10.911722+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.a scrub ok 2024-02-13T00:56:12.015 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:11.196578+0000 osd.5 (osd.5) 52 : cluster [DBG] 4.8 scrub starts 2024-02-13T00:56:12.015 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:11.198782+0000 osd.5 (osd.5) 53 : cluster [DBG] 4.8 scrub ok 2024-02-13T00:56:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:10.585273+0000 mgr.y (mgr.14544) 734 : cluster [DBG] pgmap v1327: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-02-13T00:56:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:10.733915+0000 mon.a (mon.0) 3865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-13T00:56:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:10.733996+0000 mon.a (mon.0) 3866 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-02-13T00:56:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:10.769090+0000 osd.2 (osd.2) 195 : cluster [DBG] 4.11 deep-scrub starts 2024-02-13T00:56:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:10.788357+0000 osd.2 (osd.2) 196 : cluster [DBG] 4.11 deep-scrub ok 2024-02-13T00:56:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:10.847429+0000 osd.7 (osd.7) 120 : cluster [DBG] 6.5 scrub starts 2024-02-13T00:56:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:10.848349+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.5 scrub ok 2024-02-13T00:56:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:10.908838+0000 osd.1 (osd.1) 403 : cluster [DBG] 2.a scrub starts 2024-02-13T00:56:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:10.911722+0000 osd.1 (osd.1) 404 : cluster [DBG] 2.a scrub ok 2024-02-13T00:56:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:11.196578+0000 osd.5 (osd.5) 52 : cluster [DBG] 4.8 scrub starts 2024-02-13T00:56:12.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:11.198782+0000 osd.5 (osd.5) 53 : cluster [DBG] 4.8 scrub ok 2024-02-13T00:56:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:11.815369+0000 osd.2 (osd.2) 197 : cluster [DBG] 28.1b scrub starts 2024-02-13T00:56:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:11.817596+0000 osd.2 (osd.2) 198 : cluster [DBG] 28.1b scrub ok 2024-02-13T00:56:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:11.885783+0000 osd.7 (osd.7) 122 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:56:13.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:11.886751+0000 osd.7 (osd.7) 123 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:56:13.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:11.815369+0000 osd.2 (osd.2) 197 : cluster [DBG] 28.1b scrub starts 2024-02-13T00:56:13.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:11.817596+0000 osd.2 (osd.2) 198 : cluster [DBG] 28.1b scrub ok 2024-02-13T00:56:13.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:11.885783+0000 osd.7 (osd.7) 122 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:56:13.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:11.886751+0000 osd.7 (osd.7) 123 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:56:13.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:11.815369+0000 osd.2 (osd.2) 197 : cluster [DBG] 28.1b scrub starts 2024-02-13T00:56:13.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:11.817596+0000 osd.2 (osd.2) 198 : cluster [DBG] 28.1b scrub ok 2024-02-13T00:56:13.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:11.885783+0000 osd.7 (osd.7) 122 : cluster [DBG] 2.5 scrub starts 2024-02-13T00:56:13.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:11.886751+0000 osd.7 (osd.7) 123 : cluster [DBG] 2.5 scrub ok 2024-02-13T00:56:13.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.704+0000 7ff19450e700 1 -- 172.21.15.162:0/546895967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff18c0f9b70 msgr2=0x7ff18c0f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:13.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.704+0000 7ff19450e700 1 --2- 172.21.15.162:0/546895967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff18c0f9b70 0x7ff18c0f9f50 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 rx=0x7ff17c004870 tx=0x7ff17c009cb0).stop 2024-02-13T00:56:13.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.704+0000 7ff19450e700 1 -- 172.21.15.162:0/546895967 shutdown_connections 2024-02-13T00:56:13.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.704+0000 7ff19450e700 1 --2- 172.21.15.162:0/546895967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18c0f8500 0x7ff18c104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:56:13.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.704+0000 7ff19450e700 1 --2- 172.21.15.162:0/546895967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff18c0f9b70 0x7ff18c0f9f50 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:13.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.704+0000 7ff19450e700 1 --2- 172.21.15.162:0/546895967 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff18c0fa440 0x7ff18c0f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:13.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.704+0000 7ff19450e700 1 -- 172.21.15.162:0/546895967 >> 172.21.15.162:0/546895967 conn(0x7ff18c0f38c0 msgr2=0x7ff18c0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:13.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 -- 172.21.15.162:0/546895967 shutdown_connections 2024-02-13T00:56:13.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 -- 172.21.15.162:0/546895967 wait complete. 2024-02-13T00:56:13.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 librados: starting msgr at 2024-02-13T00:56:13.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 librados: starting objecter 2024-02-13T00:56:13.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 Processor -- start 2024-02-13T00:56:13.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 -- start start 2024-02-13T00:56:13.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 librados: setting wanted keys 2024-02-13T00:56:13.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 librados: calling monclient init 2024-02-13T00:56:13.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff18c0f8500 0x7ff18c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:13.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff18c0f9b70 0x7ff18c06b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:13.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18c0fa440 0x7ff18c06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:13.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c1167f0 con 0x7ff18c0f9b70 2024-02-13T00:56:13.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c116960 con 0x7ff18c0fa440 2024-02-13T00:56:13.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff19450e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c116b20 con 0x7ff18c0f8500 2024-02-13T00:56:13.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff1922aa700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff18c0f8500 0x7ff18c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:13.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff1922aa700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff18c0f8500 0x7ff18c06b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:57196/0 (socket says 172.21.15.162:57196) 2024-02-13T00:56:13.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff1922aa700 1 -- 172.21.15.162:0/3359679711 learned_addr learned my addr 172.21.15.162:0/3359679711 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:13.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.705+0000 7ff191aa9700 1 --2- 172.21.15.162:0/3359679711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff18c0f9b70 0x7ff18c06b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:13.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff192aab700 1 --2- 172.21.15.162:0/3359679711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18c0fa440 0x7ff18c06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:13.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff1922aa700 1 -- 172.21.15.162:0/3359679711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18c0fa440 msgr2=0x7ff18c06bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:13.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff1922aa700 1 --2- 172.21.15.162:0/3359679711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18c0fa440 0x7ff18c06bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:13.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff1922aa700 1 -- 172.21.15.162:0/3359679711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff18c0f9b70 msgr2=0x7ff18c06b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:13.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff1922aa700 1 --2- 172.21.15.162:0/3359679711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff18c0f9b70 0x7ff18c06b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:13.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff1922aa700 1 -- 172.21.15.162:0/3359679711 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff188005040 con 0x7ff18c0f8500 2024-02-13T00:56:13.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff1922aa700 1 --2- 172.21.15.162:0/3359679711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff18c0f8500 0x7ff18c06b170 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 rx=0x7ff17c009eb0 tx=0x7ff17c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:13.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff1837fe700 10 client.?.objecter ms_handle_connect 0x7ff18c0f8500 2024-02-13T00:56:13.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff1837fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:13.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff1837fe700 1 -- 172.21.15.162:0/3359679711 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c00e070 con 0x7ff18c0f8500 2024-02-13T00:56:13.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff1837fe700 1 -- 172.21.15.162:0/3359679711 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff17c013da0 con 0x7ff18c0f8500 2024-02-13T00:56:13.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff19450e700 1 -- 172.21.15.162:0/3359679711 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff17c005040 con 0x7ff18c0f8500 2024-02-13T00:56:13.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff1837fe700 1 -- 172.21.15.162:0/3359679711 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c019070 con 0x7ff18c0f8500 2024-02-13T00:56:13.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff19450e700 10 client.38444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:13.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff19450e700 1 -- 172.21.15.162:0/3359679711 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18c117070 con 0x7ff18c0f8500 2024-02-13T00:56:13.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.706+0000 7ff19450e700 1 librados: init done 2024-02-13T00:56:13.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.707+0000 7ff19450e700 1 -- 172.21.15.162:0/3359679711 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18c0475e0 con 0x7ff18c0f8500 2024-02-13T00:56:13.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.707+0000 7ff1837fe700 1 -- 172.21.15.162:0/3359679711 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7ff17c01e420 con 0x7ff18c0f8500 2024-02-13T00:56:13.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.708+0000 7ff1837fe700 1 --2- 172.21.15.162:0/3359679711 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff178062c90 0x7ff178065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:13.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.708+0000 7ff191aa9700 1 --2- 172.21.15.162:0/3359679711 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff178062c90 0x7ff178065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:13.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.708+0000 7ff1837fe700 1 -- 172.21.15.162:0/3359679711 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(850..850 src has 449..850) v4 ==== 7509+0+0 (secure 0 0 0) 0x7ff17c07dac0 con 0x7ff18c0f8500 2024-02-13T00:56:13.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.708+0000 7ff1837fe700 10 client.38444.objecter ms_dispatch 0x7ff18c001550 osd_map(850..850 src has 449..850) v4 2024-02-13T00:56:13.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.708+0000 7ff1837fe700 3 client.38444.objecter handle_osd_map got epochs [850,850] > 0 2024-02-13T00:56:13.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.708+0000 7ff1837fe700 3 client.38444.objecter handle_osd_map decoding full epoch 850 2024-02-13T00:56:13.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.708+0000 7ff1837fe700 20 client.38444.objecter dump_active .. 0 homeless 2024-02-13T00:56:13.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.708+0000 7ff1837fe700 10 client.38444.objecter ms_handle_connect 0x7ff178062c90 2024-02-13T00:56:13.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.708+0000 7ff191aa9700 1 --2- 172.21.15.162:0/3359679711 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff178062c90 0x7ff178065150 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 rx=0x7ff18c06c910 tx=0x7ff18800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:13.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.712+0000 7ff1837fe700 1 -- 172.21.15.162:0/3359679711 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff17c0512d0 con 0x7ff18c0f8500 2024-02-13T00:56:13.895 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:13.891+0000 7ff19450e700 1 -- 172.21.15.162:0/3359679711 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) v1 -- 0x7ff18c06c650 con 0x7ff18c0f8500 2024-02-13T00:56:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:12.585835+0000 mgr.y (mgr.14544) 735 : cluster [DBG] pgmap v1329: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 1 objects/s recovering 2024-02-13T00:56:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:12.714838+0000 osd.3 (osd.3) 215 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:56:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:12.718016+0000 osd.3 (osd.3) 216 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:56:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:12.922612+0000 osd.1 (osd.1) 405 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:56:14.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:12.926209+0000 osd.1 (osd.1) 406 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:56:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:12.585835+0000 mgr.y (mgr.14544) 735 : cluster [DBG] pgmap v1329: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 1 objects/s recovering 2024-02-13T00:56:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:12.714838+0000 osd.3 (osd.3) 215 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:56:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:12.718016+0000 osd.3 (osd.3) 216 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:56:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:12.922612+0000 osd.1 (osd.1) 405 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:56:14.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:12.926209+0000 osd.1 (osd.1) 406 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:56:14.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:12.585835+0000 mgr.y (mgr.14544) 735 : cluster [DBG] pgmap v1329: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 1 objects/s recovering 2024-02-13T00:56:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:12.714838+0000 osd.3 (osd.3) 215 : cluster [DBG] 2.19 scrub starts 2024-02-13T00:56:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:12.718016+0000 osd.3 (osd.3) 216 : cluster [DBG] 2.19 scrub ok 2024-02-13T00:56:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:12.922612+0000 osd.1 (osd.1) 405 : cluster [DBG] 4.12 scrub starts 2024-02-13T00:56:14.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:12.926209+0000 osd.1 (osd.1) 406 : cluster [DBG] 4.12 scrub ok 2024-02-13T00:56:14.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.749+0000 7ff1837fe700 1 -- 172.21.15.162:0/3359679711 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v851) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff17c023020 con 0x7ff18c0f8500 2024-02-13T00:56:14.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 10 librados: watch_flush enter 2024-02-13T00:56:14.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 10 librados: watch_flush exit 2024-02-13T00:56:14.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 20 client.38444.objecter shutdown clearing up homeless session... 2024-02-13T00:56:14.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 10 client.38444.objecter successfully canceled tick 2024-02-13T00:56:14.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 -- 172.21.15.162:0/3359679711 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff178062c90 msgr2=0x7ff178065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:14.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 --2- 172.21.15.162:0/3359679711 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff178062c90 0x7ff178065150 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 rx=0x7ff18c06c910 tx=0x7ff18800b040).stop 2024-02-13T00:56:14.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 -- 172.21.15.162:0/3359679711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff18c0f8500 msgr2=0x7ff18c06b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:14.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 --2- 172.21.15.162:0/3359679711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff18c0f8500 0x7ff18c06b170 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 rx=0x7ff17c009eb0 tx=0x7ff17c00b040).stop 2024-02-13T00:56:14.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 -- 172.21.15.162:0/3359679711 shutdown_connections 2024-02-13T00:56:14.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 --2- 172.21.15.162:0/3359679711 >> v2:172.21.15.162:6800/2866534053 conn(0x7ff178062c90 0x7ff178065150 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:14.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 --2- 172.21.15.162:0/3359679711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff18c0fa440 0x7ff18c06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:14.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 --2- 172.21.15.162:0/3359679711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff18c0f8500 0x7ff18c06b170 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:14.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 --2- 172.21.15.162:0/3359679711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff18c0f9b70 0x7ff18c06b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:14.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 -- 172.21.15.162:0/3359679711 >> 172.21.15.162:0/3359679711 conn(0x7ff18c0f38c0 msgr2=0x7ff18c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:14.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 -- 172.21.15.162:0/3359679711 shutdown_connections 2024-02-13T00:56:14.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 -- 172.21.15.162:0/3359679711 wait complete. 2024-02-13T00:56:14.756 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:14.751+0000 7ff19450e700 1 librados: shutdown 2024-02-13T00:56:14.757 INFO:teuthology.orchestra.run.smithi162.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-02-13T00:56:15.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:13.677083+0000 osd.3 (osd.3) 217 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:56:15.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:13.680169+0000 osd.3 (osd.3) 218 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:56:15.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:13.892966+0000 mon.c (mon.2) 538 : audit [INF] from='client.? 172.21.15.162:0/3359679711' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-13T00:56:15.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:13.893296+0000 mon.a (mon.0) 3867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-13T00:56:15.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:13.677083+0000 osd.3 (osd.3) 217 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:56:15.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:13.680169+0000 osd.3 (osd.3) 218 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:56:15.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:13.892966+0000 mon.c (mon.2) 538 : audit [INF] from='client.? 172.21.15.162:0/3359679711' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-13T00:56:15.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:13.893296+0000 mon.a (mon.0) 3867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-13T00:56:15.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:13.677083+0000 osd.3 (osd.3) 217 : cluster [DBG] 3.2 scrub starts 2024-02-13T00:56:15.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:13.680169+0000 osd.3 (osd.3) 218 : cluster [DBG] 3.2 scrub ok 2024-02-13T00:56:15.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:13.892966+0000 mon.c (mon.2) 538 : audit [INF] from='client.? 172.21.15.162:0/3359679711' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-13T00:56:15.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:13.893296+0000 mon.a (mon.0) 3867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-13T00:56:15.491 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-02-13T00:56:15.491 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-02-13T00:56:15.492 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-02-13T00:56:15.492 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-02-13T00:56:15.492 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-13T00:56:15.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-13T00:56:15.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:14.586329+0000 mgr.y (mgr.14544) 736 : cluster [DBG] pgmap v1330: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:56:15.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:14.632673+0000 osd.3 (osd.3) 219 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:56:15.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:14.636323+0000 osd.3 (osd.3) 220 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:56:15.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:14.749208+0000 mon.a (mon.0) 3868 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-13T00:56:15.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:14.749298+0000 mon.a (mon.0) 3869 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-02-13T00:56:15.751 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:14.951270+0000 osd.0 (osd.0) 389 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:56:15.752 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:14.954751+0000 osd.0 (osd.0) 390 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:56:15.752 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:15.026334+0000 osd.6 (osd.6) 330 : cluster [DBG] 4.5 scrub starts 2024-02-13T00:56:15.752 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:15.029924+0000 osd.6 (osd.6) 331 : cluster [DBG] 4.5 scrub ok 2024-02-13T00:56:15.752 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:15.302494+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.d deep-scrub starts 2024-02-13T00:56:15.772 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:16.027 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:15.309238+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.d deep-scrub ok 2024-02-13T00:56:16.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:14.586329+0000 mgr.y (mgr.14544) 736 : cluster [DBG] pgmap v1330: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:56:16.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:14.632673+0000 osd.3 (osd.3) 219 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:56:16.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:14.636323+0000 osd.3 (osd.3) 220 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:56:16.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:14.749208+0000 mon.a (mon.0) 3868 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-13T00:56:16.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:14.749298+0000 mon.a (mon.0) 3869 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-02-13T00:56:16.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:14.951270+0000 osd.0 (osd.0) 389 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:56:16.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:14.954751+0000 osd.0 (osd.0) 390 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:56:16.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:15.026334+0000 osd.6 (osd.6) 330 : cluster [DBG] 4.5 scrub starts 2024-02-13T00:56:16.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:15.029924+0000 osd.6 (osd.6) 331 : cluster [DBG] 4.5 scrub ok 2024-02-13T00:56:16.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:15.302494+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.d deep-scrub starts 2024-02-13T00:56:16.028 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:15.309238+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.d deep-scrub ok 2024-02-13T00:56:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:14.586329+0000 mgr.y (mgr.14544) 736 : cluster [DBG] pgmap v1330: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:56:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:14.632673+0000 osd.3 (osd.3) 219 : cluster [DBG] 4.14 scrub starts 2024-02-13T00:56:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:14.636323+0000 osd.3 (osd.3) 220 : cluster [DBG] 4.14 scrub ok 2024-02-13T00:56:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:14.749208+0000 mon.a (mon.0) 3868 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-13T00:56:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:14.749298+0000 mon.a (mon.0) 3869 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-02-13T00:56:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:14.951270+0000 osd.0 (osd.0) 389 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:56:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:14.954751+0000 osd.0 (osd.0) 390 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:56:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:15.026334+0000 osd.6 (osd.6) 330 : cluster [DBG] 4.5 scrub starts 2024-02-13T00:56:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:15.029924+0000 osd.6 (osd.6) 331 : cluster [DBG] 4.5 scrub ok 2024-02-13T00:56:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:15.302494+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.d deep-scrub starts 2024-02-13T00:56:16.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:15.309238+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.d deep-scrub ok 2024-02-13T00:56:16.316 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:56:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:56:16] "GET /metrics HTTP/1.1" 200 185524 "" "Prometheus/2.33.4" 2024-02-13T00:56:16.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:56:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:56:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:15.782290+0000 osd.2 (osd.2) 199 : cluster [DBG] 2.e scrub starts 2024-02-13T00:56:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:15.785243+0000 osd.2 (osd.2) 200 : cluster [DBG] 2.e scrub ok 2024-02-13T00:56:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:16.044944+0000 osd.6 (osd.6) 332 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:56:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:16.045868+0000 osd.6 (osd.6) 333 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:56:17.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:16.096488+0000 mgr.y (mgr.14544) 737 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:17.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:15.782290+0000 osd.2 (osd.2) 199 : cluster [DBG] 2.e scrub starts 2024-02-13T00:56:17.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:15.785243+0000 osd.2 (osd.2) 200 : cluster [DBG] 2.e scrub ok 2024-02-13T00:56:17.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:16.044944+0000 osd.6 (osd.6) 332 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:56:17.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:16.045868+0000 osd.6 (osd.6) 333 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:56:17.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:16.096488+0000 mgr.y (mgr.14544) 737 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:15.782290+0000 osd.2 (osd.2) 199 : cluster [DBG] 2.e scrub starts 2024-02-13T00:56:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:15.785243+0000 osd.2 (osd.2) 200 : cluster [DBG] 2.e scrub ok 2024-02-13T00:56:17.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:16.044944+0000 osd.6 (osd.6) 332 : cluster [DBG] 2.17 scrub starts 2024-02-13T00:56:17.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:16.045868+0000 osd.6 (osd.6) 333 : cluster [DBG] 2.17 scrub ok 2024-02-13T00:56:17.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:16.096488+0000 mgr.y (mgr.14544) 737 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:17.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.712+0000 7f0300040700 1 -- 172.21.15.162:0/1800661843 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02f8102270 msgr2=0x7f02f8104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:17.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.712+0000 7f0300040700 1 --2- 172.21.15.162:0/1800661843 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02f8102270 0x7f02f8104660 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7f02f0009be0 tx=0x7f02f0004d10).stop 2024-02-13T00:56:17.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 -- 172.21.15.162:0/1800661843 shutdown_connections 2024-02-13T00:56:17.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 --2- 172.21.15.162:0/1800661843 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02f8102270 0x7f02f8104660 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:17.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 --2- 172.21.15.162:0/1800661843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02f80f8d10 0x7f02f80f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:56:17.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 --2- 172.21.15.162:0/1800661843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f80f95e0 0x7f02f8101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:17.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 -- 172.21.15.162:0/1800661843 >> 172.21.15.162:0/1800661843 conn(0x7f02f806eb90 msgr2=0x7f02f80f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:17.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 -- 172.21.15.162:0/1800661843 shutdown_connections 2024-02-13T00:56:17.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 -- 172.21.15.162:0/1800661843 wait complete. 2024-02-13T00:56:17.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 librados: starting msgr at 2024-02-13T00:56:17.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 librados: starting objecter 2024-02-13T00:56:17.719 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 Processor -- start 2024-02-13T00:56:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 -- start start 2024-02-13T00:56:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 librados: setting wanted keys 2024-02-13T00:56:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.713+0000 7f0300040700 1 librados: calling monclient init 2024-02-13T00:56:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f0300040700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02f80f8d10 0x7f02f80f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f0300040700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f80f95e0 0x7f02f80f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f0300040700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02f8102270 0x7f02f80f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f0300040700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f81166a0 con 0x7f02f80f95e0 2024-02-13T00:56:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f0300040700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f8116810 con 0x7f02f80f8d10 2024-02-13T00:56:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f0300040700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f81169d0 con 0x7f02f8102270 2024-02-13T00:56:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02fd5db700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f80f95e0 0x7f02f80f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02fe5dd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02f8102270 0x7f02f80f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02fd5db700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f80f95e0 0x7f02f80f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43668/0 (socket says 172.21.15.162:43668) 2024-02-13T00:56:17.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02fd5db700 1 -- 172.21.15.162:0/3720856433 learned_addr learned my addr 172.21.15.162:0/3720856433 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:17.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02fdddc700 1 --2- 172.21.15.162:0/3720856433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02f80f8d10 0x7f02f80f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:17.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02fd5db700 1 -- 172.21.15.162:0/3720856433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02f8102270 msgr2=0x7f02f80f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:17.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02fd5db700 1 --2- 172.21.15.162:0/3720856433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02f8102270 0x7f02f80f76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:17.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02fd5db700 1 -- 172.21.15.162:0/3720856433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02f80f8d10 msgr2=0x7f02f80f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:17.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02fd5db700 1 --2- 172.21.15.162:0/3720856433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02f80f8d10 0x7f02f80f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:17.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02fd5db700 1 -- 172.21.15.162:0/3720856433 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f4005040 con 0x7f02f80f95e0 2024-02-13T00:56:17.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02fe5dd700 1 --2- 172.21.15.162:0/3720856433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02f8102270 0x7f02f80f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:17.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02fd5db700 1 --2- 172.21.15.162:0/3720856433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f80f95e0 0x7f02f80f71b0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7f02f40048f0 tx=0x7f02f4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:17.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02eeffd700 10 client.?.objecter ms_handle_connect 0x7f02f80f95e0 2024-02-13T00:56:17.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02eeffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:17.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02eeffd700 1 -- 172.21.15.162:0/3720856433 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4007b00 con 0x7f02f80f95e0 2024-02-13T00:56:17.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f0300040700 1 -- 172.21.15.162:0/3720856433 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f00097c0 con 0x7f02f80f95e0 2024-02-13T00:56:17.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02eeffd700 1 -- 172.21.15.162:0/3720856433 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02f400f040 con 0x7f02f80f95e0 2024-02-13T00:56:17.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f02eeffd700 1 -- 172.21.15.162:0/3720856433 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4014a00 con 0x7f02f80f95e0 2024-02-13T00:56:17.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f0300040700 10 client.28356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:17.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f0300040700 1 -- 172.21.15.162:0/3720856433 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f8116fe0 con 0x7f02f80f95e0 2024-02-13T00:56:17.722 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.714+0000 7f0300040700 1 librados: init done 2024-02-13T00:56:17.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.715+0000 7f0300040700 1 -- 172.21.15.162:0/3720856433 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f80475e0 con 0x7f02f80f95e0 2024-02-13T00:56:17.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.716+0000 7f02eeffd700 1 -- 172.21.15.162:0/3720856433 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f02f400e580 con 0x7f02f80f95e0 2024-02-13T00:56:17.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.716+0000 7f02eeffd700 1 --2- 172.21.15.162:0/3720856433 >> v2:172.21.15.162:6800/2866534053 conn(0x7f02e4062c90 0x7f02e4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:17.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.716+0000 7f02fdddc700 1 --2- 172.21.15.162:0/3720856433 >> v2:172.21.15.162:6800/2866534053 conn(0x7f02e4062c90 0x7f02e4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:17.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.716+0000 7f02eeffd700 1 -- 172.21.15.162:0/3720856433 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(851..851 src has 449..851) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f02f407cb10 con 0x7f02f80f95e0 2024-02-13T00:56:17.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.716+0000 7f02eeffd700 10 client.28356.objecter ms_dispatch 0x7f02f8001550 osd_map(851..851 src has 449..851) v4 2024-02-13T00:56:17.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.716+0000 7f02eeffd700 3 client.28356.objecter handle_osd_map got epochs [851,851] > 0 2024-02-13T00:56:17.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.716+0000 7f02eeffd700 3 client.28356.objecter handle_osd_map decoding full epoch 851 2024-02-13T00:56:17.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.716+0000 7f02eeffd700 20 client.28356.objecter dump_active .. 0 homeless 2024-02-13T00:56:17.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.717+0000 7f02eeffd700 10 client.28356.objecter ms_handle_connect 0x7f02e4062c90 2024-02-13T00:56:17.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.717+0000 7f02fdddc700 1 --2- 172.21.15.162:0/3720856433 >> v2:172.21.15.162:6800/2866534053 conn(0x7f02e4062c90 0x7f02e4065150 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 rx=0x7f02e800b910 tx=0x7f02e8008040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:17.725 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.721+0000 7f02eeffd700 1 -- 172.21.15.162:0/3720856433 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f02f4049f60 con 0x7f02f80f95e0 2024-02-13T00:56:17.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.885+0000 7f0300040700 1 -- 172.21.15.162:0/3720856433 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02f805fb80 con 0x7f02f80f95e0 2024-02-13T00:56:17.889 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.886+0000 7f02eeffd700 1 -- 172.21.15.162:0/3720856433 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v851) v1 ==== 74+0+22252 (secure 0 0 0) 0x7f02f4050240 con 0x7f02f80f95e0 2024-02-13T00:56:17.890 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:56:17.890 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":851,"fsid":"d4ee01fc-ca06-11ee-95b9-87774f69a715","created":"2024-02-13T00:31:51.797017+0000","modified":"2024-02-13T00:56:14.740778+0000","last_up_change":"2024-02-13T00:48:03.474219+0000","last_in_change":"2024-02-13T00:55:03.759603+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":110,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":256,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-13T00:35:34.545390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-13T00:37:39.459980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-13T00:37:39.740219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-13T00:37:41.489292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-13T00:37:43.466304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":22,"pg_placement_num_target":22,"pg_num_target":22,"pg_num_pending":22,"last_pg_merge_meta":{"source_pgid":"5.16","ready_epoch":544,"last_epoch_started":543,"last_epoch_clean":543,"source_version":"0'0","target_version":"0'0"},"last_change":"824","last_force_op_resend":"0","last_force_op_resend_prenautilus":"544","last_force_op_resend_preluminous":"542","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-13T00:37:45.542442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":42,"pg_placement_num":42,"pg_placement_num_target":42,"pg_num_target":42,"pg_num_pending":42,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"758","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"246","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":28,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-02-13T00:41:00.190777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":832,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6801","nonce":2103743338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2103743338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2103743338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6803","nonce":2103743338}]},"public_addr":"172.21.15.162:6801/2103743338","cluster_addr":"172.21.15.162:6802/2103743338","heartbeat_back_addr":"172.21.15.162:6804/2103743338","heartbeat_front_addr":"172.21.15.162:6803/2103743338","state":["exists","up"]},{"osd":1,"uuid":"39e9d2ef-9aa4-45bb-8a47-b0e0ec222886","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":848,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6805","nonce":150599780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":150599780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":150599780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6807","nonce":150599780}]},"public_addr":"172.21.15.162:6805/150599780","cluster_addr":"172.21.15.162:6806/150599780","heartbeat_back_addr":"172.21.15.162:6808/150599780","heartbeat_front_addr":"172.21.15.162:6807/150599780","state":["exists","up"]},{"osd":2,"uuid":"86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":848,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6809","nonce":3689719376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3689719376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3689719376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6811","nonce":3689719376}]},"public_addr":"172.21.15.162:6809/3689719376","cluster_addr":"172.21.15.162:6810/3689719376","heartbeat_back_addr":"172.21.15.162:6812/3689719376","heartbeat_front_addr":"172.21.15.162:6811/3689719376","state":["exists","up"]},{"osd":3,"uuid":"2e6b9f0e-7a18-4cf8-9331-1a973fadc001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":848,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6813","nonce":604971011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":604971011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":604971011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6815","nonce":604971011}]},"public_addr":"172.21.15.162:6813/604971011","cluster_addr":"172.21.15.162:6814/604971011","heartbeat_back_addr":"172.21.15.162:6816/604971011","heartbeat_front_addr":"172.21.15.162:6815/604971011","state":["exists","up"]},{"osd":4,"uuid":"c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":841,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3344263388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":3344263388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":3344263388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3344263388}]},"public_addr":"172.21.15.181:6800/3344263388","cluster_addr":"172.21.15.181:6801/3344263388","heartbeat_back_addr":"172.21.15.181:6803/3344263388","heartbeat_front_addr":"172.21.15.181:6802/3344263388","state":["exists","up"]},{"osd":5,"uuid":"6aa2bd68-7f4b-4973-ac49-7fbc57c1a723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":436,"up_from":479,"up_thru":833,"down_at":437,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2277917372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":2277917372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":2277917372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2277917372}]},"public_addr":"172.21.15.181:6804/2277917372","cluster_addr":"172.21.15.181:6805/2277917372","heartbeat_back_addr":"172.21.15.181:6807/2277917372","heartbeat_front_addr":"172.21.15.181:6806/2277917372","state":["exists","up"]},{"osd":6,"uuid":"791ba814-5284-432e-addc-d6b7f5f3d474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":848,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1506640341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1506640341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1506640341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1506640341}]},"public_addr":"172.21.15.181:6808/1506640341","cluster_addr":"172.21.15.181:6809/1506640341","heartbeat_back_addr":"172.21.15.181:6811/1506640341","heartbeat_front_addr":"172.21.15.181:6810/1506640341","state":["exists","up"]},{"osd":7,"uuid":"1f369c65-e663-4eb2-87a5-f2900f15b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":848,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1964192772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":1964192772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":1964192772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1964192772}]},"public_addr":"172.21.15.181:6812/1964192772","cluster_addr":"172.21.15.181:6813/1964192772","heartbeat_back_addr":"172.21.15.181:6815/1964192772","heartbeat_front_addr":"172.21.15.181:6814/1964192772","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:55:43.363086+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:56:07.119612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:55:36.591721+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:55:17.406714+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:56:06.846186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:56:02.273697+0000","dead_epoch":436},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:56:05.402776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:55:05.899708+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.b","osds":[3,6,5]}],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":7,"to":6}]},{"pgid":"2.1a","mappings":[{"from":6,"to":1}]},{"pgid":"2.1d","mappings":[{"from":0,"to":7},{"from":6,"to":1}]},{"pgid":"4.11","mappings":[{"from":3,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"5.5","mappings":[{"from":0,"to":6}]},{"pgid":"28.7","mappings":[{"from":6,"to":4}]},{"pgid":"28.a","mappings":[{"from":6,"to":7}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.162:6800/1909610488":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/2370739667":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/1502590187":"2024-02-14T00:32:47.469889+0000","172.21.15.162:6800/4121673848":"2024-02-14T00:32:47.469889+0000","172.21.15.162:0/2412382714":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/3041562701":"2024-02-14T00:32:47.469889+0000","172.21.15.162:6800/2775563314":"2024-02-14T00:32:24.380625+0000","172.21.15.162:0/3135632190":"2024-02-14T00:32:24.380625+0000","172.21.15.162:0/197307636":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/2653247729":"2024-02-14T00:32:24.380625+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":24,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":25,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":60,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":78,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":85,"snaps":[{"begin":2,"length":1}]},{"pool":94,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":95,"snaps":[{"begin":4,"length":4}]},{"pool":107,"snaps":[{"begin":2,"length":1}]},{"pool":113,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":125,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":133,"snaps":[{"begin":12,"length":1}]},{"pool":154,"snaps":[{"begin":14,"length":1}]},{"pool":166,"snaps":[{"begin":17,"length":1}]},{"pool":176,"snaps":[{"begin":20,"length":1}]},{"pool":237,"snaps":[{"begin":2,"length":1}]},{"pool":238,"snaps":[{"begin":4,"length":1}]},{"pool":241,"snaps":[{"begin":6,"length":1}]},{"pool":245,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-13T00:56:17.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.887+0000 7f0300040700 10 librados: watch_flush enter 2024-02-13T00:56:17.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.887+0000 7f0300040700 10 librados: watch_flush exit 2024-02-13T00:56:17.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 20 client.28356.objecter shutdown clearing up homeless session... 2024-02-13T00:56:17.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 10 client.28356.objecter successfully canceled tick 2024-02-13T00:56:17.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 -- 172.21.15.162:0/3720856433 >> v2:172.21.15.162:6800/2866534053 conn(0x7f02e4062c90 msgr2=0x7f02e4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:17.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 --2- 172.21.15.162:0/3720856433 >> v2:172.21.15.162:6800/2866534053 conn(0x7f02e4062c90 0x7f02e4065150 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 rx=0x7f02e800b910 tx=0x7f02e8008040).stop 2024-02-13T00:56:17.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 -- 172.21.15.162:0/3720856433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f80f95e0 msgr2=0x7f02f80f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:17.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 --2- 172.21.15.162:0/3720856433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f80f95e0 0x7f02f80f71b0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7f02f40048f0 tx=0x7f02f4009cb0).stop 2024-02-13T00:56:17.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 -- 172.21.15.162:0/3720856433 shutdown_connections 2024-02-13T00:56:17.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 --2- 172.21.15.162:0/3720856433 >> v2:172.21.15.162:6800/2866534053 conn(0x7f02e4062c90 0x7f02e4065150 unknown :-1 s=CLOSED pgs=3080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:17.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 --2- 172.21.15.162:0/3720856433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02f8102270 0x7f02f80f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:17.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 --2- 172.21.15.162:0/3720856433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02f80f8d10 0x7f02f80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:17.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 --2- 172.21.15.162:0/3720856433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02f80f95e0 0x7f02f80f71b0 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:17.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 -- 172.21.15.162:0/3720856433 >> 172.21.15.162:0/3720856433 conn(0x7f02f806eb90 msgr2=0x7f02f8102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:17.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 -- 172.21.15.162:0/3720856433 shutdown_connections 2024-02-13T00:56:17.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 -- 172.21.15.162:0/3720856433 wait complete. 2024-02-13T00:56:17.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:17.888+0000 7f0300040700 1 librados: shutdown 2024-02-13T00:56:18.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:16.587133+0000 mgr.y (mgr.14544) 738 : cluster [DBG] pgmap v1332: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:56:18.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:16.587133+0000 mgr.y (mgr.14544) 738 : cluster [DBG] pgmap v1332: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:56:18.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:16.587133+0000 mgr.y (mgr.14544) 738 : cluster [DBG] pgmap v1332: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-13T00:56:18.634 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-13T00:56:18.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-13T00:56:18.907 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:19.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:56:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:56:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:56:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:17.662013+0000 osd.3 (osd.3) 221 : cluster [DBG] 28.18 scrub starts 2024-02-13T00:56:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:17.665199+0000 osd.3 (osd.3) 222 : cluster [DBG] 28.18 scrub ok 2024-02-13T00:56:19.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:17.886164+0000 mon.a (mon.0) 3870 : audit [DBG] from='client.? 172.21.15.162:0/3720856433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:56:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:17.662013+0000 osd.3 (osd.3) 221 : cluster [DBG] 28.18 scrub starts 2024-02-13T00:56:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:17.665199+0000 osd.3 (osd.3) 222 : cluster [DBG] 28.18 scrub ok 2024-02-13T00:56:19.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:17.886164+0000 mon.a (mon.0) 3870 : audit [DBG] from='client.? 172.21.15.162:0/3720856433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:56:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:17.662013+0000 osd.3 (osd.3) 221 : cluster [DBG] 28.18 scrub starts 2024-02-13T00:56:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:17.665199+0000 osd.3 (osd.3) 222 : cluster [DBG] 28.18 scrub ok 2024-02-13T00:56:19.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:17.886164+0000 mon.a (mon.0) 3870 : audit [DBG] from='client.? 172.21.15.162:0/3720856433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:56:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:18.102574+0000 osd.7 (osd.7) 124 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:56:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:18.105477+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:56:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:18.587529+0000 mgr.y (mgr.14544) 739 : cluster [DBG] pgmap v1333: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:56:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:18.710085+0000 osd.3 (osd.3) 223 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:56:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:18.713172+0000 osd.3 (osd.3) 224 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:56:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:19.078689+0000 osd.7 (osd.7) 126 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:56:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:19.082160+0000 osd.7 (osd.7) 127 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:56:20.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:19.144199+0000 osd.5 (osd.5) 54 : cluster [DBG] 28.19 scrub starts 2024-02-13T00:56:20.046 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:19.147778+0000 osd.5 (osd.5) 55 : cluster [DBG] 28.19 scrub ok 2024-02-13T00:56:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:18.102574+0000 osd.7 (osd.7) 124 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:56:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:18.105477+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:56:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:18.587529+0000 mgr.y (mgr.14544) 739 : cluster [DBG] pgmap v1333: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:56:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:18.710085+0000 osd.3 (osd.3) 223 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:56:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:18.713172+0000 osd.3 (osd.3) 224 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:56:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:19.078689+0000 osd.7 (osd.7) 126 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:56:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:19.082160+0000 osd.7 (osd.7) 127 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:56:20.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:19.144199+0000 osd.5 (osd.5) 54 : cluster [DBG] 28.19 scrub starts 2024-02-13T00:56:20.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:19.147778+0000 osd.5 (osd.5) 55 : cluster [DBG] 28.19 scrub ok 2024-02-13T00:56:20.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:18.102574+0000 osd.7 (osd.7) 124 : cluster [DBG] 2.1c scrub starts 2024-02-13T00:56:20.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:18.105477+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.1c scrub ok 2024-02-13T00:56:20.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:18.587529+0000 mgr.y (mgr.14544) 739 : cluster [DBG] pgmap v1333: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:56:20.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:18.710085+0000 osd.3 (osd.3) 223 : cluster [DBG] 6.13 scrub starts 2024-02-13T00:56:20.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:18.713172+0000 osd.3 (osd.3) 224 : cluster [DBG] 6.13 scrub ok 2024-02-13T00:56:20.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:19.078689+0000 osd.7 (osd.7) 126 : cluster [DBG] 6.1c scrub starts 2024-02-13T00:56:20.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:19.082160+0000 osd.7 (osd.7) 127 : cluster [DBG] 6.1c scrub ok 2024-02-13T00:56:20.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:19.144199+0000 osd.5 (osd.5) 54 : cluster [DBG] 28.19 scrub starts 2024-02-13T00:56:20.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:19.147778+0000 osd.5 (osd.5) 55 : cluster [DBG] 28.19 scrub ok 2024-02-13T00:56:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:19.818816+0000 osd.2 (osd.2) 201 : cluster [DBG] 5.a scrub starts 2024-02-13T00:56:20.831 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:19.819792+0000 osd.2 (osd.2) 202 : cluster [DBG] 5.a scrub ok 2024-02-13T00:56:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:19.818816+0000 osd.2 (osd.2) 201 : cluster [DBG] 5.a scrub starts 2024-02-13T00:56:20.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:19.819792+0000 osd.2 (osd.2) 202 : cluster [DBG] 5.a scrub ok 2024-02-13T00:56:20.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.888+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3207269967 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a380fcb50 msgr2=0x7f0a380fcf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:20.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.888+0000 7f0a3cb73700 1 --2- 172.21.15.162:0/3207269967 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a380fcb50 0x7f0a380fcf30 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 rx=0x7f0a2c004870 tx=0x7f0a2c009cb0).stop 2024-02-13T00:56:20.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.888+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3207269967 shutdown_connections 2024-02-13T00:56:20.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.888+0000 7f0a3cb73700 1 --2- 172.21.15.162:0/3207269967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a380f73c0 0x7f0a381039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:20.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.888+0000 7f0a3cb73700 1 --2- 172.21.15.162:0/3207269967 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a380fcb50 0x7f0a380fcf30 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:20.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.888+0000 7f0a3cb73700 1 --2- 172.21.15.162:0/3207269967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a380fd4b0 0x7f0a380f6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:20.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.888+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3207269967 >> 172.21.15.162:0/3207269967 conn(0x7f0a380f2760 msgr2=0x7f0a380f4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:20.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.888+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3207269967 shutdown_connections 2024-02-13T00:56:20.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.888+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3207269967 wait complete. 2024-02-13T00:56:20.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.889+0000 7f0a3cb73700 1 librados: starting msgr at 2024-02-13T00:56:20.893 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.889+0000 7f0a3cb73700 1 librados: starting objecter 2024-02-13T00:56:20.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.889+0000 7f0a3cb73700 1 Processor -- start 2024-02-13T00:56:20.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.889+0000 7f0a3cb73700 1 -- start start 2024-02-13T00:56:20.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.889+0000 7f0a3cb73700 1 librados: setting wanted keys 2024-02-13T00:56:20.894 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.889+0000 7f0a3cb73700 1 librados: calling monclient init 2024-02-13T00:56:20.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3cb73700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a380f73c0 0x7f0a3810d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3cb73700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a380fcb50 0x7f0a3810d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3cb73700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a380fd4b0 0x7f0a381071d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3cb73700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a381076c0 con 0x7f0a380f73c0 2024-02-13T00:56:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3cb73700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a38107830 con 0x7f0a380fd4b0 2024-02-13T00:56:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3cb73700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a381079f0 con 0x7f0a380fcb50 2024-02-13T00:56:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a380f73c0 0x7f0a3810d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a35d9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a380fcb50 0x7f0a3810d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3659c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a380f73c0 0x7f0a3810d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43674/0 (socket says 172.21.15.162:43674) 2024-02-13T00:56:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3659c700 1 -- 172.21.15.162:0/3633120206 learned_addr learned my addr 172.21.15.162:0/3633120206 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a36d9d700 1 --2- 172.21.15.162:0/3633120206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a380fd4b0 0x7f0a381071d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3659c700 1 -- 172.21.15.162:0/3633120206 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a380fcb50 msgr2=0x7f0a3810d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:20.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3659c700 1 --2- 172.21.15.162:0/3633120206 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a380fcb50 0x7f0a3810d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:20.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3659c700 1 -- 172.21.15.162:0/3633120206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a380fd4b0 msgr2=0x7f0a381071d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:20.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3659c700 1 --2- 172.21.15.162:0/3633120206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a380fd4b0 0x7f0a381071d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:20.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3659c700 1 -- 172.21.15.162:0/3633120206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a28005040 con 0x7f0a380f73c0 2024-02-13T00:56:20.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a36d9d700 1 --2- 172.21.15.162:0/3633120206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a380fd4b0 0x7f0a381071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:56:20.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a35d9b700 1 --2- 172.21.15.162:0/3633120206 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a380fcb50 0x7f0a3810d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:20.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3659c700 1 --2- 172.21.15.162:0/3633120206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a380f73c0 0x7f0a3810d030 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f0a2c0043a0 tx=0x7f0a2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:20.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a277fe700 10 client.?.objecter ms_handle_connect 0x7f0a380f73c0 2024-02-13T00:56:20.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a277fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:20.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a277fe700 1 -- 172.21.15.162:0/3633120206 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c00e030 con 0x7f0a380f73c0 2024-02-13T00:56:20.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3633120206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2c005040 con 0x7f0a380f73c0 2024-02-13T00:56:20.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a277fe700 1 -- 172.21.15.162:0/3633120206 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a2c009710 con 0x7f0a380f73c0 2024-02-13T00:56:20.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a277fe700 1 -- 172.21.15.162:0/3633120206 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c01b070 con 0x7f0a380f73c0 2024-02-13T00:56:20.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3cb73700 10 client.28362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:20.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3633120206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a38107f60 con 0x7f0a380f73c0 2024-02-13T00:56:20.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.890+0000 7f0a3cb73700 1 librados: init done 2024-02-13T00:56:20.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.891+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3633120206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a38060770 con 0x7f0a380f73c0 2024-02-13T00:56:20.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.892+0000 7f0a277fe700 1 -- 172.21.15.162:0/3633120206 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f0a2c020420 con 0x7f0a380f73c0 2024-02-13T00:56:20.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.893+0000 7f0a277fe700 1 --2- 172.21.15.162:0/3633120206 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0a20062c90 0x7f0a20065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:20.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.893+0000 7f0a277fe700 1 -- 172.21.15.162:0/3633120206 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(851..851 src has 449..851) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f0a2c080ae0 con 0x7f0a380f73c0 2024-02-13T00:56:20.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.893+0000 7f0a277fe700 10 client.28362.objecter ms_dispatch 0x7f0a38001550 osd_map(851..851 src has 449..851) v4 2024-02-13T00:56:20.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.893+0000 7f0a277fe700 3 client.28362.objecter handle_osd_map got epochs [851,851] > 0 2024-02-13T00:56:20.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.893+0000 7f0a277fe700 3 client.28362.objecter handle_osd_map decoding full epoch 851 2024-02-13T00:56:20.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.893+0000 7f0a35d9b700 1 --2- 172.21.15.162:0/3633120206 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0a20062c90 0x7f0a20065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:20.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.893+0000 7f0a277fe700 20 client.28362.objecter dump_active .. 0 homeless 2024-02-13T00:56:20.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.893+0000 7f0a277fe700 10 client.28362.objecter ms_handle_connect 0x7f0a20062c90 2024-02-13T00:56:20.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.893+0000 7f0a35d9b700 1 --2- 172.21.15.162:0/3633120206 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0a20062c90 0x7f0a20065150 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 rx=0x7f0a38108970 tx=0x7f0a2800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:20.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:20.896+0000 7f0a277fe700 1 -- 172.21.15.162:0/3633120206 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0a2c0542b0 con 0x7f0a380f73c0 2024-02-13T00:56:21.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:19.818816+0000 osd.2 (osd.2) 201 : cluster [DBG] 5.a scrub starts 2024-02-13T00:56:21.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:19.819792+0000 osd.2 (osd.2) 202 : cluster [DBG] 5.a scrub ok 2024-02-13T00:56:21.064 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.060+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3633120206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a381086b0 con 0x7f0a380f73c0 2024-02-13T00:56:21.065 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.061+0000 7f0a277fe700 1 -- 172.21.15.162:0/3633120206 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v851) v1 ==== 74+0+22252 (secure 0 0 0) 0x7f0a2c053ea0 con 0x7f0a380f73c0 2024-02-13T00:56:21.066 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:56:21.066 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":851,"fsid":"d4ee01fc-ca06-11ee-95b9-87774f69a715","created":"2024-02-13T00:31:51.797017+0000","modified":"2024-02-13T00:56:14.740778+0000","last_up_change":"2024-02-13T00:48:03.474219+0000","last_in_change":"2024-02-13T00:55:03.759603+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":110,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":256,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-13T00:35:34.545390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-13T00:37:39.459980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-13T00:37:39.740219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-13T00:37:41.489292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-13T00:37:43.466304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":22,"pg_placement_num_target":22,"pg_num_target":22,"pg_num_pending":22,"last_pg_merge_meta":{"source_pgid":"5.16","ready_epoch":544,"last_epoch_started":543,"last_epoch_clean":543,"source_version":"0'0","target_version":"0'0"},"last_change":"824","last_force_op_resend":"0","last_force_op_resend_prenautilus":"544","last_force_op_resend_preluminous":"542","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-13T00:37:45.542442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":42,"pg_placement_num":42,"pg_placement_num_target":42,"pg_num_target":42,"pg_num_pending":42,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"758","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"246","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":28,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-02-13T00:41:00.190777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"9e18d75f-bc12-42e0-bd4a-1a51fc9a4a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":832,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6801","nonce":2103743338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2103743338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2103743338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6803","nonce":2103743338}]},"public_addr":"172.21.15.162:6801/2103743338","cluster_addr":"172.21.15.162:6802/2103743338","heartbeat_back_addr":"172.21.15.162:6804/2103743338","heartbeat_front_addr":"172.21.15.162:6803/2103743338","state":["exists","up"]},{"osd":1,"uuid":"39e9d2ef-9aa4-45bb-8a47-b0e0ec222886","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":848,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6805","nonce":150599780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":150599780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":150599780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6807","nonce":150599780}]},"public_addr":"172.21.15.162:6805/150599780","cluster_addr":"172.21.15.162:6806/150599780","heartbeat_back_addr":"172.21.15.162:6808/150599780","heartbeat_front_addr":"172.21.15.162:6807/150599780","state":["exists","up"]},{"osd":2,"uuid":"86e57e40-c2e8-4a1a-b1ae-7a5a0ad6db2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":848,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6809","nonce":3689719376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3689719376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3689719376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6811","nonce":3689719376}]},"public_addr":"172.21.15.162:6809/3689719376","cluster_addr":"172.21.15.162:6810/3689719376","heartbeat_back_addr":"172.21.15.162:6812/3689719376","heartbeat_front_addr":"172.21.15.162:6811/3689719376","state":["exists","up"]},{"osd":3,"uuid":"2e6b9f0e-7a18-4cf8-9331-1a973fadc001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":848,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6813","nonce":604971011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":604971011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":604971011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6815","nonce":604971011}]},"public_addr":"172.21.15.162:6813/604971011","cluster_addr":"172.21.15.162:6814/604971011","heartbeat_back_addr":"172.21.15.162:6816/604971011","heartbeat_front_addr":"172.21.15.162:6815/604971011","state":["exists","up"]},{"osd":4,"uuid":"c0fc47c2-0e9a-4f31-aba9-94d2760bd6f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":841,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3344263388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":3344263388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":3344263388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3344263388}]},"public_addr":"172.21.15.181:6800/3344263388","cluster_addr":"172.21.15.181:6801/3344263388","heartbeat_back_addr":"172.21.15.181:6803/3344263388","heartbeat_front_addr":"172.21.15.181:6802/3344263388","state":["exists","up"]},{"osd":5,"uuid":"6aa2bd68-7f4b-4973-ac49-7fbc57c1a723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":436,"up_from":479,"up_thru":833,"down_at":437,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2277917372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":2277917372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":2277917372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2277917372}]},"public_addr":"172.21.15.181:6804/2277917372","cluster_addr":"172.21.15.181:6805/2277917372","heartbeat_back_addr":"172.21.15.181:6807/2277917372","heartbeat_front_addr":"172.21.15.181:6806/2277917372","state":["exists","up"]},{"osd":6,"uuid":"791ba814-5284-432e-addc-d6b7f5f3d474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":848,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1506640341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1506640341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1506640341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1506640341}]},"public_addr":"172.21.15.181:6808/1506640341","cluster_addr":"172.21.15.181:6809/1506640341","heartbeat_back_addr":"172.21.15.181:6811/1506640341","heartbeat_front_addr":"172.21.15.181:6810/1506640341","state":["exists","up"]},{"osd":7,"uuid":"1f369c65-e663-4eb2-87a5-f2900f15b35b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":848,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1964192772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":1964192772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":1964192772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1964192772}]},"public_addr":"172.21.15.181:6812/1964192772","cluster_addr":"172.21.15.181:6813/1964192772","heartbeat_back_addr":"172.21.15.181:6815/1964192772","heartbeat_front_addr":"172.21.15.181:6814/1964192772","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:55:43.363086+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:56:07.119612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:55:36.591721+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:55:17.406714+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:56:06.846186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:56:02.273697+0000","dead_epoch":436},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:56:05.402776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T00:55:05.899708+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.b","osds":[3,6,5]}],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":7,"to":6}]},{"pgid":"2.1a","mappings":[{"from":6,"to":1}]},{"pgid":"2.1d","mappings":[{"from":0,"to":7},{"from":6,"to":1}]},{"pgid":"4.11","mappings":[{"from":3,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"5.5","mappings":[{"from":0,"to":6}]},{"pgid":"28.7","mappings":[{"from":6,"to":4}]},{"pgid":"28.a","mappings":[{"from":6,"to":7}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.162:6800/1909610488":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/2370739667":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/1502590187":"2024-02-14T00:32:47.469889+0000","172.21.15.162:6800/4121673848":"2024-02-14T00:32:47.469889+0000","172.21.15.162:0/2412382714":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/3041562701":"2024-02-14T00:32:47.469889+0000","172.21.15.162:6800/2775563314":"2024-02-14T00:32:24.380625+0000","172.21.15.162:0/3135632190":"2024-02-14T00:32:24.380625+0000","172.21.15.162:0/197307636":"2024-02-14T00:38:04.142895+0000","172.21.15.162:0/2653247729":"2024-02-14T00:32:24.380625+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":24,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":25,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":60,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":78,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":85,"snaps":[{"begin":2,"length":1}]},{"pool":94,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":95,"snaps":[{"begin":4,"length":4}]},{"pool":107,"snaps":[{"begin":2,"length":1}]},{"pool":113,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":125,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":133,"snaps":[{"begin":12,"length":1}]},{"pool":154,"snaps":[{"begin":14,"length":1}]},{"pool":166,"snaps":[{"begin":17,"length":1}]},{"pool":176,"snaps":[{"begin":20,"length":1}]},{"pool":237,"snaps":[{"begin":2,"length":1}]},{"pool":238,"snaps":[{"begin":4,"length":1}]},{"pool":241,"snaps":[{"begin":6,"length":1}]},{"pool":245,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-13T00:56:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 10 librados: watch_flush enter 2024-02-13T00:56:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 10 librados: watch_flush exit 2024-02-13T00:56:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 20 client.28362.objecter shutdown clearing up homeless session... 2024-02-13T00:56:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 10 client.28362.objecter successfully canceled tick 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3633120206 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0a20062c90 msgr2=0x7f0a20065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 1 --2- 172.21.15.162:0/3633120206 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0a20062c90 0x7f0a20065150 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 rx=0x7f0a38108970 tx=0x7f0a2800b040).stop 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3633120206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a380f73c0 msgr2=0x7f0a3810d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 1 --2- 172.21.15.162:0/3633120206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a380f73c0 0x7f0a3810d030 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f0a2c0043a0 tx=0x7f0a2c00b040).stop 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3633120206 shutdown_connections 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 1 --2- 172.21.15.162:0/3633120206 >> v2:172.21.15.162:6800/2866534053 conn(0x7f0a20062c90 0x7f0a20065150 unknown :-1 s=CLOSED pgs=3081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 1 --2- 172.21.15.162:0/3633120206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a380fd4b0 0x7f0a381071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 1 --2- 172.21.15.162:0/3633120206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a380f73c0 0x7f0a3810d030 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 1 --2- 172.21.15.162:0/3633120206 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a380fcb50 0x7f0a3810d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3633120206 >> 172.21.15.162:0/3633120206 conn(0x7f0a380f2760 msgr2=0x7f0a380fa840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3633120206 shutdown_connections 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.063+0000 7f0a3cb73700 1 -- 172.21.15.162:0/3633120206 wait complete. 2024-02-13T00:56:21.068 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:21.064+0000 7f0a3cb73700 1 librados: shutdown 2024-02-13T00:56:21.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-13T00:56:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:20.588290+0000 mgr.y (mgr.14544) 740 : cluster [DBG] pgmap v1334: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:56:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:20.720960+0000 osd.3 (osd.3) 225 : cluster [DBG] 28.14 scrub starts 2024-02-13T00:56:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:20.723864+0000 osd.3 (osd.3) 226 : cluster [DBG] 28.14 scrub ok 2024-02-13T00:56:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:21.061828+0000 mon.a (mon.0) 3871 : audit [DBG] from='client.? 172.21.15.162:0/3633120206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:56:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:21.128595+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:56:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:21.131497+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:56:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:21.178884+0000 osd.6 (osd.6) 334 : cluster [DBG] 6.22 deep-scrub starts 2024-02-13T00:56:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:21.181882+0000 osd.6 (osd.6) 335 : cluster [DBG] 6.22 deep-scrub ok 2024-02-13T00:56:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:21.214891+0000 osd.4 (osd.4) 77 : cluster [DBG] 2.f scrub starts 2024-02-13T00:56:22.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:21.216363+0000 osd.4 (osd.4) 78 : cluster [DBG] 2.f scrub ok 2024-02-13T00:56:22.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:20.588290+0000 mgr.y (mgr.14544) 740 : cluster [DBG] pgmap v1334: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:56:22.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:20.720960+0000 osd.3 (osd.3) 225 : cluster [DBG] 28.14 scrub starts 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:20.723864+0000 osd.3 (osd.3) 226 : cluster [DBG] 28.14 scrub ok 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:21.061828+0000 mon.a (mon.0) 3871 : audit [DBG] from='client.? 172.21.15.162:0/3633120206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:21.128595+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:21.131497+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:21.178884+0000 osd.6 (osd.6) 334 : cluster [DBG] 6.22 deep-scrub starts 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:21.181882+0000 osd.6 (osd.6) 335 : cluster [DBG] 6.22 deep-scrub ok 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:21.214891+0000 osd.4 (osd.4) 77 : cluster [DBG] 2.f scrub starts 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:21.216363+0000 osd.4 (osd.4) 78 : cluster [DBG] 2.f scrub ok 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:20.588290+0000 mgr.y (mgr.14544) 740 : cluster [DBG] pgmap v1334: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:20.720960+0000 osd.3 (osd.3) 225 : cluster [DBG] 28.14 scrub starts 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:20.723864+0000 osd.3 (osd.3) 226 : cluster [DBG] 28.14 scrub ok 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:21.061828+0000 mon.a (mon.0) 3871 : audit [DBG] from='client.? 172.21.15.162:0/3633120206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T00:56:22.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:21.128595+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.3 scrub starts 2024-02-13T00:56:22.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:21.131497+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.3 scrub ok 2024-02-13T00:56:22.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:21.178884+0000 osd.6 (osd.6) 334 : cluster [DBG] 6.22 deep-scrub starts 2024-02-13T00:56:22.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:21.181882+0000 osd.6 (osd.6) 335 : cluster [DBG] 6.22 deep-scrub ok 2024-02-13T00:56:22.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:21.214891+0000 osd.4 (osd.4) 77 : cluster [DBG] 2.f scrub starts 2024-02-13T00:56:22.082 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:21.216363+0000 osd.4 (osd.4) 78 : cluster [DBG] 2.f scrub ok 2024-02-13T00:56:22.087 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:21.714079+0000 osd.3 (osd.3) 227 : cluster [DBG] 6.c scrub starts 2024-02-13T00:56:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:21.717104+0000 osd.3 (osd.3) 228 : cluster [DBG] 6.c scrub ok 2024-02-13T00:56:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:22.219499+0000 osd.4 (osd.4) 79 : cluster [DBG] 28.1e scrub starts 2024-02-13T00:56:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:22.222688+0000 osd.4 (osd.4) 80 : cluster [DBG] 28.1e scrub ok 2024-02-13T00:56:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:22.225836+0000 osd.6 (osd.6) 336 : cluster [DBG] 28.1a scrub starts 2024-02-13T00:56:22.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:22.228788+0000 osd.6 (osd.6) 337 : cluster [DBG] 28.1a scrub ok 2024-02-13T00:56:22.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:21.714079+0000 osd.3 (osd.3) 227 : cluster [DBG] 6.c scrub starts 2024-02-13T00:56:22.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:21.717104+0000 osd.3 (osd.3) 228 : cluster [DBG] 6.c scrub ok 2024-02-13T00:56:22.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:22.219499+0000 osd.4 (osd.4) 79 : cluster [DBG] 28.1e scrub starts 2024-02-13T00:56:22.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:22.222688+0000 osd.4 (osd.4) 80 : cluster [DBG] 28.1e scrub ok 2024-02-13T00:56:22.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:22.225836+0000 osd.6 (osd.6) 336 : cluster [DBG] 28.1a scrub starts 2024-02-13T00:56:22.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:22.228788+0000 osd.6 (osd.6) 337 : cluster [DBG] 28.1a scrub ok 2024-02-13T00:56:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:21.714079+0000 osd.3 (osd.3) 227 : cluster [DBG] 6.c scrub starts 2024-02-13T00:56:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:21.717104+0000 osd.3 (osd.3) 228 : cluster [DBG] 6.c scrub ok 2024-02-13T00:56:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:22.219499+0000 osd.4 (osd.4) 79 : cluster [DBG] 28.1e scrub starts 2024-02-13T00:56:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:22.222688+0000 osd.4 (osd.4) 80 : cluster [DBG] 28.1e scrub ok 2024-02-13T00:56:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:22.225836+0000 osd.6 (osd.6) 336 : cluster [DBG] 28.1a scrub starts 2024-02-13T00:56:23.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:22.228788+0000 osd.6 (osd.6) 337 : cluster [DBG] 28.1a scrub ok 2024-02-13T00:56:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:22.588969+0000 mgr.y (mgr.14544) 741 : cluster [DBG] pgmap v1335: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:22.665478+0000 osd.3 (osd.3) 229 : cluster [DBG] 28.c scrub starts 2024-02-13T00:56:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:22.668758+0000 osd.3 (osd.3) 230 : cluster [DBG] 28.c scrub ok 2024-02-13T00:56:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:23.231370+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:56:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:23.234301+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:56:24.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:22.588969+0000 mgr.y (mgr.14544) 741 : cluster [DBG] pgmap v1335: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:24.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:22.665478+0000 osd.3 (osd.3) 229 : cluster [DBG] 28.c scrub starts 2024-02-13T00:56:24.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:22.668758+0000 osd.3 (osd.3) 230 : cluster [DBG] 28.c scrub ok 2024-02-13T00:56:24.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:23.231370+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:56:24.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:23.234301+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:56:24.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:22.588969+0000 mgr.y (mgr.14544) 741 : cluster [DBG] pgmap v1335: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:24.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:22.665478+0000 osd.3 (osd.3) 229 : cluster [DBG] 28.c scrub starts 2024-02-13T00:56:24.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:22.668758+0000 osd.3 (osd.3) 230 : cluster [DBG] 28.c scrub ok 2024-02-13T00:56:24.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:23.231370+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.11 scrub starts 2024-02-13T00:56:24.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:23 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:23.234301+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.11 scrub ok 2024-02-13T00:56:24.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.184+0000 7f84826f3700 1 -- 172.21.15.162:0/3378068150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f847c0f8500 msgr2=0x7f847c104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:24.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.184+0000 7f84826f3700 1 --2- 172.21.15.162:0/3378068150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f847c0f8500 0x7f847c104bd0 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 rx=0x7f8474009c30 tx=0x7f8474004c50).stop 2024-02-13T00:56:24.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.184+0000 7f84826f3700 1 -- 172.21.15.162:0/3378068150 shutdown_connections 2024-02-13T00:56:24.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.184+0000 7f84826f3700 1 --2- 172.21.15.162:0/3378068150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f847c0f8500 0x7f847c104bd0 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:24.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.184+0000 7f84826f3700 1 --2- 172.21.15.162:0/3378068150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f847c0f9b70 0x7f847c0f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:56:24.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.184+0000 7f84826f3700 1 --2- 172.21.15.162:0/3378068150 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f847c0fa440 0x7f847c0f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:24.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.184+0000 7f84826f3700 1 -- 172.21.15.162:0/3378068150 >> 172.21.15.162:0/3378068150 conn(0x7f847c0f38c0 msgr2=0x7f847c0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:24.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.184+0000 7f84826f3700 1 -- 172.21.15.162:0/3378068150 shutdown_connections 2024-02-13T00:56:24.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.184+0000 7f84826f3700 1 -- 172.21.15.162:0/3378068150 wait complete. 2024-02-13T00:56:24.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f84826f3700 1 librados: starting msgr at 2024-02-13T00:56:24.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f84826f3700 1 librados: starting objecter 2024-02-13T00:56:24.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f84826f3700 1 Processor -- start 2024-02-13T00:56:24.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f84826f3700 1 -- start start 2024-02-13T00:56:24.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f84826f3700 1 librados: setting wanted keys 2024-02-13T00:56:24.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f84826f3700 1 librados: calling monclient init 2024-02-13T00:56:24.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f84826f3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f847c0f8500 0x7f847c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:24.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f84826f3700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f847c0f9b70 0x7f847c06b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:24.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f84826f3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f847c0fa440 0x7f847c06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f84826f3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c1165a0 con 0x7f847c0f8500 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f84826f3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c116710 con 0x7f847c0fa440 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f84826f3700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c1168d0 con 0x7f847c0f9b70 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f847b7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f847c0f9b70 0x7f847c06b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f847bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f847c0f8500 0x7f847c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f847b7fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f847c0f9b70 0x7f847c06b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:32784/0 (socket says 172.21.15.162:32784) 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f847bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f847c0f8500 0x7f847c06b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43688/0 (socket says 172.21.15.162:43688) 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.185+0000 7f847b7fe700 1 -- 172.21.15.162:0/1215199341 learned_addr learned my addr 172.21.15.162:0/1215199341 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f8480c90700 1 --2- 172.21.15.162:0/1215199341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f847c0fa440 0x7f847c06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f847bfff700 1 -- 172.21.15.162:0/1215199341 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f847c0f9b70 msgr2=0x7f847c06b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f847bfff700 1 --2- 172.21.15.162:0/1215199341 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f847c0f9b70 0x7f847c06b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f847bfff700 1 -- 172.21.15.162:0/1215199341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f847c0fa440 msgr2=0x7f847c06bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f847bfff700 1 --2- 172.21.15.162:0/1215199341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f847c0fa440 0x7f847c06bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f847bfff700 1 -- 172.21.15.162:0/1215199341 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8470005040 con 0x7f847c0f8500 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f847bfff700 1 --2- 172.21.15.162:0/1215199341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f847c0f8500 0x7f847c06b170 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7f846c00b4e0 tx=0x7f846c009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f84797fa700 10 client.?.objecter ms_handle_connect 0x7f847c0f8500 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f84797fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f84797fa700 1 -- 172.21.15.162:0/1215199341 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c00be30 con 0x7f847c0f8500 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f84797fa700 1 -- 172.21.15.162:0/1215199341 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f846c0027f0 con 0x7f847c0f8500 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f84826f3700 1 -- 172.21.15.162:0/1215199341 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84740097c0 con 0x7f847c0f8500 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f84797fa700 1 -- 172.21.15.162:0/1215199341 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c012950 con 0x7f847c0f8500 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f84826f3700 10 client.28368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f84826f3700 1 -- 172.21.15.162:0/1215199341 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c116f90 con 0x7f847c0f8500 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.186+0000 7f84826f3700 1 librados: init done 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.187+0000 7f84826f3700 10 client.28368.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.187+0000 7f84826f3700 15 client.28368.objecter _session_command_op_assign -1 1 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.187+0000 7f84826f3700 20 client.28368.objecter _get_session osd=-1 returning homeless 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.187+0000 7f84826f3700 10 client.28368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.187+0000 7f84826f3700 1 -- 172.21.15.162:0/1215199341 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f847c0475e0 con 0x7f847c0f8500 2024-02-13T00:56:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.187+0000 7f84797fa700 1 -- 172.21.15.162:0/1215199341 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f846c002320 con 0x7f847c0f8500 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 1 --2- 172.21.15.162:0/1215199341 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8464062c90 0x7f8464065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 1 -- 172.21.15.162:0/1215199341 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(851..851 src has 449..851) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f846c07ce00 con 0x7f847c0f8500 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 10 client.28368.objecter ms_dispatch 0x7f847c001550 osd_map(851..851 src has 449..851) v4 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 3 client.28368.objecter handle_osd_map got epochs [851,851] > 0 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 3 client.28368.objecter handle_osd_map decoding full epoch 851 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f847b7fe700 1 --2- 172.21.15.162:0/1215199341 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8464062c90 0x7f8464065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 10 client.28368.objecter checking command 1 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 1 --2- 172.21.15.162:0/1215199341 >> v2:172.21.15.162:6801/2103743338 conn(0x7f846406e060 0x7f8464070500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 20 client.28368.objecter _get_session s=0x7f846406d9b0 osd=0 3 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 20 client.28368.objecter put_session s=0x7f846406d9b0 osd=0 3 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 15 client.28368.objecter _session_command_op_remove -1 1 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 20 client.28368.objecter _get_session s=0x7f846406d9b0 osd=0 3 2024-02-13T00:56:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 20 client.28368.objecter get_session s=0x7f846406d9b0 osd=0 3 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 15 client.28368.objecter _session_command_op_assign 0 1 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 20 client.28368.objecter put_session s=0x7f846406d9b0 osd=0 4 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 10 client.28368.objecter _send_command 1 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 1 -- 172.21.15.162:0/1215199341 --> v2:172.21.15.162:6801/2103743338 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8464070b40 con 0x7f846406e060 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 20 client.28368.objecter dump_active .. 0 homeless 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 1 -- 172.21.15.162:0/1215199341 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=851) v2 ==== 24+0+0 (secure 0 0 0) 0x7f846c07f420 con 0x7f847c0f8500 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f8480c90700 1 --2- 172.21.15.162:0/1215199341 >> v2:172.21.15.162:6801/2103743338 conn(0x7f846406e060 0x7f8464070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 10 client.28368.objecter ms_handle_connect 0x7f8464062c90 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f847b7fe700 1 --2- 172.21.15.162:0/1215199341 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8464062c90 0x7f8464065150 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 rx=0x7f847c06c910 tx=0x7f847000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f84797fa700 10 client.28368.objecter ms_handle_connect 0x7f846406e060 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.188+0000 7f8480c90700 1 --2- 172.21.15.162:0/1215199341 >> v2:172.21.15.162:6801/2103743338 conn(0x7f846406e060 0x7f8464070500 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.189+0000 7f84797fa700 1 -- 172.21.15.162:0/1215199341 <== osd.0 v2:172.21.15.162:6801/2103743338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8464070b40 con 0x7f846406e060 2024-02-13T00:56:24.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.189+0000 7f84797fa700 10 client.28368.objecter ms_dispatch 0x7f847c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:56:24.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.189+0000 7f84797fa700 10 client.28368.objecter _finish_command 1 = system:0 2024-02-13T00:56:24.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.189+0000 7f84797fa700 20 client.28368.objecter put_session s=0x7f846406d9b0 osd=0 4 2024-02-13T00:56:24.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.189+0000 7f84797fa700 15 client.28368.objecter _session_command_op_remove 0 1 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 10 client.28368.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 15 client.28368.objecter _session_command_op_assign -1 2 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 20 client.28368.objecter _get_session s=0x7f846406d9b0 osd=0 3 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 20 client.28368.objecter put_session s=0x7f846406d9b0 osd=0 3 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 20 client.28368.objecter _get_session s=0x7f846406d9b0 osd=0 3 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 15 client.28368.objecter _session_command_op_remove -1 2 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 20 client.28368.objecter get_session s=0x7f846406d9b0 osd=0 3 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 15 client.28368.objecter _session_command_op_assign 0 2 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 20 client.28368.objecter put_session s=0x7f846406d9b0 osd=0 4 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 10 client.28368.objecter _send_command 2 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 1 -- 172.21.15.162:0/1215199341 --> v2:172.21.15.162:6801/2103743338 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f847c1072c0 con 0x7f846406e060 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84797fa700 1 -- 172.21.15.162:0/1215199341 <== osd.0 v2:172.21.15.162:6801/2103743338 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f847c1072c0 con 0x7f846406e060 2024-02-13T00:56:24.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84797fa700 10 client.28368.objecter ms_dispatch 0x7f847c001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:56:24.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84797fa700 10 client.28368.objecter _finish_command 2 = system:0 2024-02-13T00:56:24.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84797fa700 20 client.28368.objecter put_session s=0x7f846406d9b0 osd=0 4 2024-02-13T00:56:24.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84797fa700 15 client.28368.objecter _session_command_op_remove 0 2 2024-02-13T00:56:24.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 10 librados: watch_flush enter 2024-02-13T00:56:24.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 10 librados: watch_flush exit 2024-02-13T00:56:24.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 10 client.28368.objecter close_session for osd.0 2024-02-13T00:56:24.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.203+0000 7f84826f3700 1 -- 172.21.15.162:0/1215199341 >> v2:172.21.15.162:6801/2103743338 conn(0x7f846406e060 msgr2=0x7f8464070500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:24.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 1 --2- 172.21.15.162:0/1215199341 >> v2:172.21.15.162:6801/2103743338 conn(0x7f846406e060 0x7f8464070500 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:24.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 20 client.28368.objecter put_session s=0x7f846406d9b0 osd=0 1 2024-02-13T00:56:24.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 20 client.28368.objecter shutdown clearing up homeless session... 2024-02-13T00:56:24.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 10 client.28368.objecter successfully canceled tick 2024-02-13T00:56:24.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 1 -- 172.21.15.162:0/1215199341 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8464062c90 msgr2=0x7f8464065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:24.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 1 --2- 172.21.15.162:0/1215199341 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8464062c90 0x7f8464065150 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 rx=0x7f847c06c910 tx=0x7f847000b040).stop 2024-02-13T00:56:24.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 1 -- 172.21.15.162:0/1215199341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f847c0f8500 msgr2=0x7f847c06b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:24.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 1 --2- 172.21.15.162:0/1215199341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f847c0f8500 0x7f847c06b170 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7f846c00b4e0 tx=0x7f846c009060).stop 2024-02-13T00:56:24.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f8480c90700 1 -- 172.21.15.162:0/1215199341 reap_dead start 2024-02-13T00:56:24.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 1 -- 172.21.15.162:0/1215199341 shutdown_connections 2024-02-13T00:56:24.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 1 -- 172.21.15.162:0/1215199341 >> 172.21.15.162:0/1215199341 conn(0x7f847c0f38c0 msgr2=0x7f847c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:24.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 1 -- 172.21.15.162:0/1215199341 shutdown_connections 2024-02-13T00:56:24.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 1 -- 172.21.15.162:0/1215199341 wait complete. 2024-02-13T00:56:24.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:24.204+0000 7f84826f3700 1 librados: shutdown 2024-02-13T00:56:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:24.106351+0000 osd.5 (osd.5) 58 : cluster [DBG] 4.15 deep-scrub starts 2024-02-13T00:56:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:24.112089+0000 osd.5 (osd.5) 59 : cluster [DBG] 4.15 deep-scrub ok 2024-02-13T00:56:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:24.238741+0000 osd.6 (osd.6) 340 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:56:24.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:24.240797+0000 osd.6 (osd.6) 341 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:56:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:24.106351+0000 osd.5 (osd.5) 58 : cluster [DBG] 4.15 deep-scrub starts 2024-02-13T00:56:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:24.112089+0000 osd.5 (osd.5) 59 : cluster [DBG] 4.15 deep-scrub ok 2024-02-13T00:56:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:24.238741+0000 osd.6 (osd.6) 340 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:56:24.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:24 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:24.240797+0000 osd.6 (osd.6) 341 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:56:24.922 INFO:teuthology.orchestra.run.smithi162.stdout:34359738630 2024-02-13T00:56:24.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-13T00:56:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:24.106351+0000 osd.5 (osd.5) 58 : cluster [DBG] 4.15 deep-scrub starts 2024-02-13T00:56:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:24.112089+0000 osd.5 (osd.5) 59 : cluster [DBG] 4.15 deep-scrub ok 2024-02-13T00:56:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:24.238741+0000 osd.6 (osd.6) 340 : cluster [DBG] 4.1f scrub starts 2024-02-13T00:56:25.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:24.240797+0000 osd.6 (osd.6) 341 : cluster [DBG] 4.1f scrub ok 2024-02-13T00:56:25.195 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:24.589693+0000 mgr.y (mgr.14544) 742 : cluster [DBG] pgmap v1336: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:24.884156+0000 osd.1 (osd.1) 407 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:56:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:24.887703+0000 osd.1 (osd.1) 408 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:56:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:25.086761+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:56:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:25.089708+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:56:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:25.124883+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:56:26.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:25.127794+0000 osd.5 (osd.5) 61 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:56:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:24.589693+0000 mgr.y (mgr.14544) 742 : cluster [DBG] pgmap v1336: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:24.884156+0000 osd.1 (osd.1) 407 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:56:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:24.887703+0000 osd.1 (osd.1) 408 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:56:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:25.086761+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:56:26.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:25.089708+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:56:26.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:25.124883+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:56:26.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:25.127794+0000 osd.5 (osd.5) 61 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:56:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:24.589693+0000 mgr.y (mgr.14544) 742 : cluster [DBG] pgmap v1336: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:24.884156+0000 osd.1 (osd.1) 407 : cluster [DBG] 2.9 scrub starts 2024-02-13T00:56:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:24.887703+0000 osd.1 (osd.1) 408 : cluster [DBG] 2.9 scrub ok 2024-02-13T00:56:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:25.086761+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-02-13T00:56:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:25.089708+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-02-13T00:56:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:25.124883+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.12 scrub starts 2024-02-13T00:56:26.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:25.127794+0000 osd.5 (osd.5) 61 : cluster [DBG] 2.12 scrub ok 2024-02-13T00:56:26.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:56:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:56:26.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:56:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:56:26] "GET /metrics HTTP/1.1" 200 185532 "" "Prometheus/2.33.4" 2024-02-13T00:56:26.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:26.098966+0000 mgr.y (mgr.14544) 743 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:26.910 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:26.278927+0000 mon.c (mon.2) 539 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:56:26.910 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:26.559958+0000 mon.c (mon.2) 540 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:56:26.910 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:26.560635+0000 mon.c (mon.2) 541 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:56:26.910 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:26.565546+0000 mon.a (mon.0) 3872 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:56:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:26.098966+0000 mgr.y (mgr.14544) 743 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:26.278927+0000 mon.c (mon.2) 539 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:56:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:26.559958+0000 mon.c (mon.2) 540 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:56:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:26.560635+0000 mon.c (mon.2) 541 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:56:26.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:26 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:26.565546+0000 mon.a (mon.0) 3872 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:56:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:26.098966+0000 mgr.y (mgr.14544) 743 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:26.278927+0000 mon.c (mon.2) 539 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:56:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:26.559958+0000 mon.c (mon.2) 540 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:56:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:26.560635+0000 mon.c (mon.2) 541 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:56:27.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:26.565546+0000 mon.a (mon.0) 3872 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:56:27.168 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.156+0000 7f5d3d410700 1 -- 172.21.15.162:0/339004503 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d380fa440 msgr2=0x7f5d380f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.156+0000 7f5d3d410700 1 --2- 172.21.15.162:0/339004503 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d380fa440 0x7f5d380f7e20 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 rx=0x7f5d28004870 tx=0x7f5d28009cb0).stop 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.156+0000 7f5d3d410700 1 -- 172.21.15.162:0/339004503 shutdown_connections 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.156+0000 7f5d3d410700 1 --2- 172.21.15.162:0/339004503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d380f8500 0x7f5d38104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.156+0000 7f5d3d410700 1 --2- 172.21.15.162:0/339004503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d380f9b70 0x7f5d380f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.156+0000 7f5d3d410700 1 --2- 172.21.15.162:0/339004503 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d380fa440 0x7f5d380f7e20 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.156+0000 7f5d3d410700 1 -- 172.21.15.162:0/339004503 >> 172.21.15.162:0/339004503 conn(0x7f5d380f38c0 msgr2=0x7f5d380f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.156+0000 7f5d3d410700 1 -- 172.21.15.162:0/339004503 shutdown_connections 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.156+0000 7f5d3d410700 1 -- 172.21.15.162:0/339004503 wait complete. 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.156+0000 7f5d3d410700 1 librados: starting msgr at 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.156+0000 7f5d3d410700 1 librados: starting objecter 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.156+0000 7f5d3d410700 1 Processor -- start 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d3d410700 1 -- start start 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d3d410700 1 librados: setting wanted keys 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d3d410700 1 librados: calling monclient init 2024-02-13T00:56:27.169 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d3d410700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d380f8500 0x7f5d3810e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d3d410700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d380f9b70 0x7f5d3810e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d3d410700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d380fa440 0x7f5d381083d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d3d410700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d381088c0 con 0x7f5d380f8500 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d3d410700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d38108a30 con 0x7f5d380f9b70 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d3d410700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d38108bf0 con 0x7f5d380fa440 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d36ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d380f8500 0x7f5d3810e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d36ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d380f8500 0x7f5d3810e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45174/0 (socket says 172.21.15.162:45174) 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d36ffd700 1 -- 172.21.15.162:0/962214718 learned_addr learned my addr 172.21.15.162:0/962214718 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d377fe700 1 --2- 172.21.15.162:0/962214718 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d380fa440 0x7f5d381083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d367fc700 1 --2- 172.21.15.162:0/962214718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d380f9b70 0x7f5d3810e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d36ffd700 1 -- 172.21.15.162:0/962214718 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d380fa440 msgr2=0x7f5d381083d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d36ffd700 1 --2- 172.21.15.162:0/962214718 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d380fa440 0x7f5d381083d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d36ffd700 1 -- 172.21.15.162:0/962214718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d380f9b70 msgr2=0x7f5d3810e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:27.170 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d36ffd700 1 --2- 172.21.15.162:0/962214718 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d380f9b70 0x7f5d3810e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d36ffd700 1 -- 172.21.15.162:0/962214718 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d28005040 con 0x7f5d380f8500 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d36ffd700 1 --2- 172.21.15.162:0/962214718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d380f8500 0x7f5d3810e230 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7f5d2c00e970 tx=0x7f5d2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d1ffff700 10 client.?.objecter ms_handle_connect 0x7f5d380f8500 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d1ffff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d1ffff700 1 -- 172.21.15.162:0/962214718 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c004660 con 0x7f5d380f8500 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.157+0000 7f5d3d410700 1 -- 172.21.15.162:0/962214718 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d38108ed0 con 0x7f5d380f8500 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.158+0000 7f5d1ffff700 1 -- 172.21.15.162:0/962214718 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d2c00f040 con 0x7f5d380f8500 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.158+0000 7f5d1ffff700 1 -- 172.21.15.162:0/962214718 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c014790 con 0x7f5d380f8500 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.158+0000 7f5d3d410700 10 client.28374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.158+0000 7f5d3d410700 1 -- 172.21.15.162:0/962214718 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d38109250 con 0x7f5d380f8500 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.158+0000 7f5d3d410700 1 librados: init done 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.158+0000 7f5d3d410700 10 client.28374.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.158+0000 7f5d3d410700 15 client.28374.objecter _session_command_op_assign -1 1 2024-02-13T00:56:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.158+0000 7f5d3d410700 20 client.28374.objecter _get_session osd=-1 returning homeless 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.158+0000 7f5d3d410700 10 client.28374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.158+0000 7f5d3d410700 1 -- 172.21.15.162:0/962214718 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5d380475e0 con 0x7f5d380f8500 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 1 -- 172.21.15.162:0/962214718 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f5d2c0090d0 con 0x7f5d380f8500 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 1 --2- 172.21.15.162:0/962214718 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5d20062c90 0x7f5d20065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 1 -- 172.21.15.162:0/962214718 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(851..851 src has 449..851) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f5d2c07d0b0 con 0x7f5d380f8500 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 10 client.28374.objecter ms_dispatch 0x7f5d38001550 osd_map(851..851 src has 449..851) v4 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 3 client.28374.objecter handle_osd_map got epochs [851,851] > 0 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 3 client.28374.objecter handle_osd_map decoding full epoch 851 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d367fc700 1 --2- 172.21.15.162:0/962214718 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5d20062c90 0x7f5d20065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 10 client.28374.objecter checking command 1 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 1 --2- 172.21.15.162:0/962214718 >> v2:172.21.15.162:6805/150599780 conn(0x7f5d2006e060 0x7f5d20070500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 20 client.28374.objecter _get_session s=0x7f5d2006d9b0 osd=1 3 2024-02-13T00:56:27.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 20 client.28374.objecter put_session s=0x7f5d2006d9b0 osd=1 3 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 15 client.28374.objecter _session_command_op_remove -1 1 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 20 client.28374.objecter _get_session s=0x7f5d2006d9b0 osd=1 3 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 20 client.28374.objecter get_session s=0x7f5d2006d9b0 osd=1 3 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 15 client.28374.objecter _session_command_op_assign 1 1 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 20 client.28374.objecter put_session s=0x7f5d2006d9b0 osd=1 4 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 10 client.28374.objecter _send_command 1 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 1 -- 172.21.15.162:0/962214718 --> v2:172.21.15.162:6805/150599780 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d20070b40 con 0x7f5d2006e060 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 20 client.28374.objecter dump_active .. 0 homeless 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.159+0000 7f5d1ffff700 1 -- 172.21.15.162:0/962214718 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=851) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d2c07d460 con 0x7f5d380f8500 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.160+0000 7f5d377fe700 1 --2- 172.21.15.162:0/962214718 >> v2:172.21.15.162:6805/150599780 conn(0x7f5d2006e060 0x7f5d20070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.160+0000 7f5d1ffff700 10 client.28374.objecter ms_handle_connect 0x7f5d20062c90 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.160+0000 7f5d367fc700 1 --2- 172.21.15.162:0/962214718 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5d20062c90 0x7f5d20065150 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 rx=0x7f5d28004ee0 tx=0x7f5d2800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.160+0000 7f5d1ffff700 10 client.28374.objecter ms_handle_connect 0x7f5d2006e060 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.160+0000 7f5d377fe700 1 --2- 172.21.15.162:0/962214718 >> v2:172.21.15.162:6805/150599780 conn(0x7f5d2006e060 0x7f5d20070500 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.160+0000 7f5d1ffff700 1 -- 172.21.15.162:0/962214718 <== osd.1 v2:172.21.15.162:6805/150599780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5d20070b40 con 0x7f5d2006e060 2024-02-13T00:56:27.173 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.160+0000 7f5d1ffff700 10 client.28374.objecter ms_dispatch 0x7f5d38001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:56:27.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.160+0000 7f5d1ffff700 10 client.28374.objecter _finish_command 1 = system:0 2024-02-13T00:56:27.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.160+0000 7f5d1ffff700 20 client.28374.objecter put_session s=0x7f5d2006d9b0 osd=1 4 2024-02-13T00:56:27.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.160+0000 7f5d1ffff700 15 client.28374.objecter _session_command_op_remove 1 1 2024-02-13T00:56:27.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 10 client.28374.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:56:27.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 15 client.28374.objecter _session_command_op_assign -1 2 2024-02-13T00:56:27.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 20 client.28374.objecter _get_session s=0x7f5d2006d9b0 osd=1 3 2024-02-13T00:56:27.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 20 client.28374.objecter put_session s=0x7f5d2006d9b0 osd=1 3 2024-02-13T00:56:27.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 20 client.28374.objecter _get_session s=0x7f5d2006d9b0 osd=1 3 2024-02-13T00:56:27.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 15 client.28374.objecter _session_command_op_remove -1 2 2024-02-13T00:56:27.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 20 client.28374.objecter get_session s=0x7f5d2006d9b0 osd=1 3 2024-02-13T00:56:27.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 15 client.28374.objecter _session_command_op_assign 1 2 2024-02-13T00:56:27.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 20 client.28374.objecter put_session s=0x7f5d2006d9b0 osd=1 4 2024-02-13T00:56:27.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 10 client.28374.objecter _send_command 2 2024-02-13T00:56:27.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 1 -- 172.21.15.162:0/962214718 --> v2:172.21.15.162:6805/150599780 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5d38107250 con 0x7f5d2006e060 2024-02-13T00:56:27.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d1ffff700 1 -- 172.21.15.162:0/962214718 <== osd.1 v2:172.21.15.162:6805/150599780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5d38107250 con 0x7f5d2006e060 2024-02-13T00:56:27.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d1ffff700 10 client.28374.objecter ms_dispatch 0x7f5d38001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:56:27.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d1ffff700 10 client.28374.objecter _finish_command 2 = system:0 2024-02-13T00:56:27.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d1ffff700 20 client.28374.objecter put_session s=0x7f5d2006d9b0 osd=1 4 2024-02-13T00:56:27.180 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d1ffff700 15 client.28374.objecter _session_command_op_remove 1 2 2024-02-13T00:56:27.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 10 librados: watch_flush enter 2024-02-13T00:56:27.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 10 librados: watch_flush exit 2024-02-13T00:56:27.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 10 client.28374.objecter close_session for osd.1 2024-02-13T00:56:27.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 1 -- 172.21.15.162:0/962214718 >> v2:172.21.15.162:6805/150599780 conn(0x7f5d2006e060 msgr2=0x7f5d20070500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:27.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 1 --2- 172.21.15.162:0/962214718 >> v2:172.21.15.162:6805/150599780 conn(0x7f5d2006e060 0x7f5d20070500 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:27.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 20 client.28374.objecter put_session s=0x7f5d2006d9b0 osd=1 1 2024-02-13T00:56:27.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 20 client.28374.objecter shutdown clearing up homeless session... 2024-02-13T00:56:27.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 10 client.28374.objecter successfully canceled tick 2024-02-13T00:56:27.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 1 -- 172.21.15.162:0/962214718 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5d20062c90 msgr2=0x7f5d20065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:27.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 1 --2- 172.21.15.162:0/962214718 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5d20062c90 0x7f5d20065150 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 rx=0x7f5d28004ee0 tx=0x7f5d2800b040).stop 2024-02-13T00:56:27.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 1 -- 172.21.15.162:0/962214718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d380f8500 msgr2=0x7f5d3810e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:27.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 1 --2- 172.21.15.162:0/962214718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d380f8500 0x7f5d3810e230 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7f5d2c00e970 tx=0x7f5d2c00b040).stop 2024-02-13T00:56:27.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d377fe700 1 -- 172.21.15.162:0/962214718 reap_dead start 2024-02-13T00:56:27.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 1 -- 172.21.15.162:0/962214718 shutdown_connections 2024-02-13T00:56:27.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 1 -- 172.21.15.162:0/962214718 >> 172.21.15.162:0/962214718 conn(0x7f5d380f38c0 msgr2=0x7f5d380fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:27.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 1 -- 172.21.15.162:0/962214718 shutdown_connections 2024-02-13T00:56:27.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.175+0000 7f5d3d410700 1 -- 172.21.15.162:0/962214718 wait complete. 2024-02-13T00:56:27.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:27.176+0000 7f5d3d410700 1 librados: shutdown 2024-02-13T00:56:27.997 INFO:teuthology.orchestra.run.smithi162.stdout:55834575107 2024-02-13T00:56:27.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-13T00:56:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:26.590538+0000 mgr.y (mgr.14544) 744 : cluster [DBG] pgmap v1337: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:56:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:26.849065+0000 osd.0 (osd.0) 391 : cluster [DBG] 28.2 deep-scrub starts 2024-02-13T00:56:28.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:27 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:26.849961+0000 osd.0 (osd.0) 392 : cluster [DBG] 28.2 deep-scrub ok 2024-02-13T00:56:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:26.590538+0000 mgr.y (mgr.14544) 744 : cluster [DBG] pgmap v1337: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:56:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:26.849065+0000 osd.0 (osd.0) 391 : cluster [DBG] 28.2 deep-scrub starts 2024-02-13T00:56:28.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:26.849961+0000 osd.0 (osd.0) 392 : cluster [DBG] 28.2 deep-scrub ok 2024-02-13T00:56:28.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:26.590538+0000 mgr.y (mgr.14544) 744 : cluster [DBG] pgmap v1337: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-13T00:56:28.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:26.849065+0000 osd.0 (osd.0) 391 : cluster [DBG] 28.2 deep-scrub starts 2024-02-13T00:56:28.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:26.849961+0000 osd.0 (osd.0) 392 : cluster [DBG] 28.2 deep-scrub ok 2024-02-13T00:56:28.267 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:28.942 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:28.010415+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:56:28.942 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:28.013335+0000 osd.5 (osd.5) 63 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:56:28.942 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:28.010415+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:56:28.942 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:28 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:28.013335+0000 osd.5 (osd.5) 63 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:56:29.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:28.010415+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.19 scrub starts 2024-02-13T00:56:29.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:28.013335+0000 osd.5 (osd.5) 63 : cluster [DBG] 6.19 scrub ok 2024-02-13T00:56:29.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:56:28 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:56:28] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:56:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:28.118306+0000 osd.7 (osd.7) 128 : cluster [DBG] 28.6 scrub starts 2024-02-13T00:56:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:28.119489+0000 osd.7 (osd.7) 129 : cluster [DBG] 28.6 scrub ok 2024-02-13T00:56:30.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:28.591107+0000 mgr.y (mgr.14544) 745 : cluster [DBG] pgmap v1338: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:30.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:28.118306+0000 osd.7 (osd.7) 128 : cluster [DBG] 28.6 scrub starts 2024-02-13T00:56:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:28.119489+0000 osd.7 (osd.7) 129 : cluster [DBG] 28.6 scrub ok 2024-02-13T00:56:30.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:28.591107+0000 mgr.y (mgr.14544) 745 : cluster [DBG] pgmap v1338: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:30.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:28.118306+0000 osd.7 (osd.7) 128 : cluster [DBG] 28.6 scrub starts 2024-02-13T00:56:30.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:28.119489+0000 osd.7 (osd.7) 129 : cluster [DBG] 28.6 scrub ok 2024-02-13T00:56:30.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:29 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:28.591107+0000 mgr.y (mgr.14544) 745 : cluster [DBG] pgmap v1338: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:30.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 -- 172.21.15.162:0/761472709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57500fe570 msgr2=0x7f57500f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:30.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 --2- 172.21.15.162:0/761472709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57500fe570 0x7f57500f7da0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 rx=0x7f57400047b0 tx=0x7f5740009cb0).stop 2024-02-13T00:56:30.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 -- 172.21.15.162:0/761472709 shutdown_connections 2024-02-13T00:56:30.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 --2- 172.21.15.162:0/761472709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57500f8480 0x7f5750104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:30.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 --2- 172.21.15.162:0/761472709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57500fdc10 0x7f57500fdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:30.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 --2- 172.21.15.162:0/761472709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57500fe570 0x7f57500f7da0 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 -- 172.21.15.162:0/761472709 >> 172.21.15.162:0/761472709 conn(0x7f57500f3800 msgr2=0x7f57500f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 -- 172.21.15.162:0/761472709 shutdown_connections 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 -- 172.21.15.162:0/761472709 wait complete. 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 librados: starting msgr at 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 librados: starting objecter 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 Processor -- start 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 -- start start 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 librados: setting wanted keys 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.358+0000 7f5757b74700 1 librados: calling monclient init 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5757b74700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57500f8480 0x7f575010e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5757b74700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57500fdc10 0x7f575010e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5757b74700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57500fe570 0x7f5750108370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5757b74700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5750108860 con 0x7f57500fe570 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5757b74700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57501089d0 con 0x7f57500fdc10 2024-02-13T00:56:30.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5757b74700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5750108b90 con 0x7f57500f8480 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5755910700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57500f8480 0x7f575010e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5755910700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57500f8480 0x7f575010e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48420/0 (socket says 172.21.15.162:48420) 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5755910700 1 -- 172.21.15.162:0/1870596680 learned_addr learned my addr 172.21.15.162:0/1870596680 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5756111700 1 --2- 172.21.15.162:0/1870596680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57500fe570 0x7f5750108370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f575510f700 1 --2- 172.21.15.162:0/1870596680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57500fdc10 0x7f575010e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5755910700 1 -- 172.21.15.162:0/1870596680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57500fdc10 msgr2=0x7f575010e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5755910700 1 --2- 172.21.15.162:0/1870596680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57500fdc10 0x7f575010e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5755910700 1 -- 172.21.15.162:0/1870596680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57500fe570 msgr2=0x7f5750108370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5755910700 1 --2- 172.21.15.162:0/1870596680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57500fe570 0x7f5750108370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5755910700 1 -- 172.21.15.162:0/1870596680 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c005040 con 0x7f57500f8480 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5756111700 1 --2- 172.21.15.162:0/1870596680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57500fe570 0x7f5750108370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5755910700 1 --2- 172.21.15.162:0/1870596680 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57500f8480 0x7f575010e1d0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 rx=0x7f574c009400 tx=0x7f574c009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:30.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5746ffd700 10 client.?.objecter ms_handle_connect 0x7f57500f8480 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5746ffd700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5746ffd700 1 -- 172.21.15.162:0/1870596680 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c007a40 con 0x7f57500f8480 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5746ffd700 1 -- 172.21.15.162:0/1870596680 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f574c00f040 con 0x7f57500f8480 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5757b74700 1 -- 172.21.15.162:0/1870596680 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5740005040 con 0x7f57500f8480 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5746ffd700 1 -- 172.21.15.162:0/1870596680 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c014a70 con 0x7f57500f8480 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5757b74700 10 client.38474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5757b74700 1 -- 172.21.15.162:0/1870596680 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57501091d0 con 0x7f57500f8480 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.359+0000 7f5757b74700 1 librados: init done 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.360+0000 7f5757b74700 10 client.38474.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.360+0000 7f5757b74700 15 client.38474.objecter _session_command_op_assign -1 1 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.360+0000 7f5757b74700 20 client.38474.objecter _get_session osd=-1 returning homeless 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.360+0000 7f5757b74700 10 client.38474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.360+0000 7f5757b74700 1 -- 172.21.15.162:0/1870596680 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f575005fb20 con 0x7f57500f8480 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.361+0000 7f5746ffd700 1 -- 172.21.15.162:0/1870596680 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f574c010530 con 0x7f57500f8480 2024-02-13T00:56:30.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.361+0000 7f5746ffd700 1 --2- 172.21.15.162:0/1870596680 >> v2:172.21.15.162:6800/2866534053 conn(0x7f573c062c90 0x7f573c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.361+0000 7f5746ffd700 1 -- 172.21.15.162:0/1870596680 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(851..851 src has 449..851) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f574c07cfa0 con 0x7f57500f8480 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.361+0000 7f5746ffd700 10 client.38474.objecter ms_dispatch 0x7f5750001550 osd_map(851..851 src has 449..851) v4 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.361+0000 7f5746ffd700 3 client.38474.objecter handle_osd_map got epochs [851,851] > 0 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.361+0000 7f5746ffd700 3 client.38474.objecter handle_osd_map decoding full epoch 851 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f575510f700 1 --2- 172.21.15.162:0/1870596680 >> v2:172.21.15.162:6800/2866534053 conn(0x7f573c062c90 0x7f573c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 10 client.38474.objecter checking command 1 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 1 --2- 172.21.15.162:0/1870596680 >> v2:172.21.15.162:6809/3689719376 conn(0x7f573c06e060 0x7f573c070500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 20 client.38474.objecter _get_session s=0x7f573c06d9b0 osd=2 3 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 20 client.38474.objecter put_session s=0x7f573c06d9b0 osd=2 3 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 15 client.38474.objecter _session_command_op_remove -1 1 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 20 client.38474.objecter _get_session s=0x7f573c06d9b0 osd=2 3 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 20 client.38474.objecter get_session s=0x7f573c06d9b0 osd=2 3 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 15 client.38474.objecter _session_command_op_assign 2 1 2024-02-13T00:56:30.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 20 client.38474.objecter put_session s=0x7f573c06d9b0 osd=2 4 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 10 client.38474.objecter _send_command 1 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 1 -- 172.21.15.162:0/1870596680 --> v2:172.21.15.162:6809/3689719376 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f573c070b40 con 0x7f573c06e060 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 20 client.38474.objecter dump_active .. 0 homeless 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 1 -- 172.21.15.162:0/1870596680 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=851) v2 ==== 24+0+0 (secure 0 0 0) 0x7f574c07d350 con 0x7f57500f8480 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5756111700 1 --2- 172.21.15.162:0/1870596680 >> v2:172.21.15.162:6809/3689719376 conn(0x7f573c06e060 0x7f573c070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 10 client.38474.objecter ms_handle_connect 0x7f573c062c90 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f575510f700 1 --2- 172.21.15.162:0/1870596680 >> v2:172.21.15.162:6800/2866534053 conn(0x7f573c062c90 0x7f573c065150 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 rx=0x7f5750109b10 tx=0x7f5740007f90).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5746ffd700 10 client.38474.objecter ms_handle_connect 0x7f573c06e060 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.362+0000 7f5756111700 1 --2- 172.21.15.162:0/1870596680 >> v2:172.21.15.162:6809/3689719376 conn(0x7f573c06e060 0x7f573c070500 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.363+0000 7f5746ffd700 1 -- 172.21.15.162:0/1870596680 <== osd.2 v2:172.21.15.162:6809/3689719376 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f573c070b40 con 0x7f573c06e060 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.363+0000 7f5746ffd700 10 client.38474.objecter ms_dispatch 0x7f5750001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.363+0000 7f5746ffd700 10 client.38474.objecter _finish_command 1 = system:0 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.363+0000 7f5746ffd700 20 client.38474.objecter put_session s=0x7f573c06d9b0 osd=2 4 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.363+0000 7f5746ffd700 15 client.38474.objecter _session_command_op_remove 2 1 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.376+0000 7f5757b74700 10 client.38474.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:56:30.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.376+0000 7f5757b74700 15 client.38474.objecter _session_command_op_assign -1 2 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.376+0000 7f5757b74700 20 client.38474.objecter _get_session s=0x7f573c06d9b0 osd=2 3 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.376+0000 7f5757b74700 20 client.38474.objecter put_session s=0x7f573c06d9b0 osd=2 3 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.376+0000 7f5757b74700 20 client.38474.objecter _get_session s=0x7f573c06d9b0 osd=2 3 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.376+0000 7f5757b74700 15 client.38474.objecter _session_command_op_remove -1 2 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.376+0000 7f5757b74700 20 client.38474.objecter get_session s=0x7f573c06d9b0 osd=2 3 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.376+0000 7f5757b74700 15 client.38474.objecter _session_command_op_assign 2 2 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.376+0000 7f5757b74700 20 client.38474.objecter put_session s=0x7f573c06d9b0 osd=2 4 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.376+0000 7f5757b74700 10 client.38474.objecter _send_command 2 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.376+0000 7f5757b74700 1 -- 172.21.15.162:0/1870596680 --> v2:172.21.15.162:6809/3689719376 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f57500475e0 con 0x7f573c06e060 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5746ffd700 1 -- 172.21.15.162:0/1870596680 <== osd.2 v2:172.21.15.162:6809/3689719376 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f57500475e0 con 0x7f573c06e060 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5746ffd700 10 client.38474.objecter ms_dispatch 0x7f5750001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5746ffd700 10 client.38474.objecter _finish_command 2 = system:0 2024-02-13T00:56:30.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5746ffd700 20 client.38474.objecter put_session s=0x7f573c06d9b0 osd=2 4 2024-02-13T00:56:30.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5746ffd700 15 client.38474.objecter _session_command_op_remove 2 2 2024-02-13T00:56:30.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 10 librados: watch_flush enter 2024-02-13T00:56:30.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 10 librados: watch_flush exit 2024-02-13T00:56:30.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 10 client.38474.objecter close_session for osd.2 2024-02-13T00:56:30.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 1 -- 172.21.15.162:0/1870596680 >> v2:172.21.15.162:6809/3689719376 conn(0x7f573c06e060 msgr2=0x7f573c070500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:30.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 1 --2- 172.21.15.162:0/1870596680 >> v2:172.21.15.162:6809/3689719376 conn(0x7f573c06e060 0x7f573c070500 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:30.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 20 client.38474.objecter put_session s=0x7f573c06d9b0 osd=2 1 2024-02-13T00:56:30.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 20 client.38474.objecter shutdown clearing up homeless session... 2024-02-13T00:56:30.407 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 10 client.38474.objecter successfully canceled tick 2024-02-13T00:56:30.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 1 -- 172.21.15.162:0/1870596680 >> v2:172.21.15.162:6800/2866534053 conn(0x7f573c062c90 msgr2=0x7f573c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:30.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 1 --2- 172.21.15.162:0/1870596680 >> v2:172.21.15.162:6800/2866534053 conn(0x7f573c062c90 0x7f573c065150 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 rx=0x7f5750109b10 tx=0x7f5740007f90).stop 2024-02-13T00:56:30.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 1 -- 172.21.15.162:0/1870596680 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57500f8480 msgr2=0x7f575010e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:30.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 1 --2- 172.21.15.162:0/1870596680 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57500f8480 0x7f575010e1d0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 rx=0x7f574c009400 tx=0x7f574c009cb0).stop 2024-02-13T00:56:30.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5756111700 1 -- 172.21.15.162:0/1870596680 reap_dead start 2024-02-13T00:56:30.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 1 -- 172.21.15.162:0/1870596680 shutdown_connections 2024-02-13T00:56:30.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 1 -- 172.21.15.162:0/1870596680 >> 172.21.15.162:0/1870596680 conn(0x7f57500f3800 msgr2=0x7f57500fb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:30.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 1 -- 172.21.15.162:0/1870596680 shutdown_connections 2024-02-13T00:56:30.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 1 -- 172.21.15.162:0/1870596680 wait complete. 2024-02-13T00:56:30.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:30.377+0000 7f5757b74700 1 librados: shutdown 2024-02-13T00:56:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:29.862089+0000 osd.7 (osd.7) 130 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:31.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:29.862340+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:29.862089+0000 osd.7 (osd.7) 130 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:31.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:29.862340+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:31.161 INFO:teuthology.orchestra.run.smithi162.stdout:77309411583 2024-02-13T00:56:31.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-13T00:56:31.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:29.862089+0000 osd.7 (osd.7) 130 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:31.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:30 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:29.862340+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:31.435 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:32.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:30.591912+0000 mgr.y (mgr.14544) 746 : cluster [DBG] pgmap v1339: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:32.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:30.801474+0000 mon.a (mon.0) 3873 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-02-13T00:56:32.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:31.002092+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:56:32.019 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:31.004984+0000 osd.5 (osd.5) 65 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:56:32.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:30.591912+0000 mgr.y (mgr.14544) 746 : cluster [DBG] pgmap v1339: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:32.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:30.801474+0000 mon.a (mon.0) 3873 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-02-13T00:56:32.019 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:31.002092+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:56:32.020 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:31 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:31.004984+0000 osd.5 (osd.5) 65 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:56:32.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:30.591912+0000 mgr.y (mgr.14544) 746 : cluster [DBG] pgmap v1339: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:32.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:30.801474+0000 mon.a (mon.0) 3873 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-02-13T00:56:32.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:31.002092+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.16 scrub starts 2024-02-13T00:56:32.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:31 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:31.004984+0000 osd.5 (osd.5) 65 : cluster [DBG] 2.16 scrub ok 2024-02-13T00:56:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:31.707320+0000 osd.0 (osd.0) 393 : cluster [DBG] 6.0 scrub starts 2024-02-13T00:56:33.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:31.708096+0000 osd.0 (osd.0) 394 : cluster [DBG] 6.0 scrub ok 2024-02-13T00:56:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:31.707320+0000 osd.0 (osd.0) 393 : cluster [DBG] 6.0 scrub starts 2024-02-13T00:56:33.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:32 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:31.708096+0000 osd.0 (osd.0) 394 : cluster [DBG] 6.0 scrub ok 2024-02-13T00:56:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:31.707320+0000 osd.0 (osd.0) 393 : cluster [DBG] 6.0 scrub starts 2024-02-13T00:56:33.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:32 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:31.708096+0000 osd.0 (osd.0) 394 : cluster [DBG] 6.0 scrub ok 2024-02-13T00:56:33.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.495+0000 7fad547f0700 1 -- 172.21.15.162:0/979585117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad4c0f84c0 msgr2=0x7fad4c104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:33.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.495+0000 7fad547f0700 1 --2- 172.21.15.162:0/979585117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad4c0f84c0 0x7fad4c104ac0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 rx=0x7fad44009be0 tx=0x7fad44004d10).stop 2024-02-13T00:56:33.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.495+0000 7fad547f0700 1 -- 172.21.15.162:0/979585117 shutdown_connections 2024-02-13T00:56:33.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.495+0000 7fad547f0700 1 --2- 172.21.15.162:0/979585117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad4c0f84c0 0x7fad4c104ac0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:33.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.495+0000 7fad547f0700 1 --2- 172.21.15.162:0/979585117 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad4c0fdc50 0x7fad4c0fe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:33.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.495+0000 7fad547f0700 1 --2- 172.21.15.162:0/979585117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c0fe5b0 0x7fad4c0f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:33.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.495+0000 7fad547f0700 1 -- 172.21.15.162:0/979585117 >> 172.21.15.162:0/979585117 conn(0x7fad4c0f38a0 msgr2=0x7fad4c0f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:33.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.495+0000 7fad547f0700 1 -- 172.21.15.162:0/979585117 shutdown_connections 2024-02-13T00:56:33.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.495+0000 7fad547f0700 1 -- 172.21.15.162:0/979585117 wait complete. 2024-02-13T00:56:33.501 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad547f0700 1 librados: starting msgr at 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad547f0700 1 librados: starting objecter 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad547f0700 1 Processor -- start 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad547f0700 1 -- start start 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad547f0700 1 librados: setting wanted keys 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad547f0700 1 librados: calling monclient init 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad547f0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad4c0f84c0 0x7fad4c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad547f0700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad4c0fdc50 0x7fad4c06b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad547f0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c0fe5b0 0x7fad4c06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad547f0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c116450 con 0x7fad4c0f84c0 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad547f0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c1165c0 con 0x7fad4c0fe5b0 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad547f0700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c116780 con 0x7fad4c0fdc50 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad5258c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad4c0f84c0 0x7fad4c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad51d8b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad4c0fdc50 0x7fad4c06b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:33.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad51d8b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad4c0fdc50 0x7fad4c06b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:48436/0 (socket says 172.21.15.162:48436) 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad5258c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad4c0f84c0 0x7fad4c06b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45200/0 (socket says 172.21.15.162:45200) 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad51d8b700 1 -- 172.21.15.162:0/1318908204 learned_addr learned my addr 172.21.15.162:0/1318908204 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad52d8d700 1 --2- 172.21.15.162:0/1318908204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c0fe5b0 0x7fad4c06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad51d8b700 1 -- 172.21.15.162:0/1318908204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c0fe5b0 msgr2=0x7fad4c06bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad51d8b700 1 --2- 172.21.15.162:0/1318908204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c0fe5b0 0x7fad4c06bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad51d8b700 1 -- 172.21.15.162:0/1318908204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad4c0f84c0 msgr2=0x7fad4c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.496+0000 7fad51d8b700 1 --2- 172.21.15.162:0/1318908204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad4c0f84c0 0x7fad4c06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad51d8b700 1 -- 172.21.15.162:0/1318908204 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad40005040 con 0x7fad4c0fdc50 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad5258c700 1 --2- 172.21.15.162:0/1318908204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad4c0f84c0 0x7fad4c06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad52d8d700 1 --2- 172.21.15.162:0/1318908204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad4c0fe5b0 0x7fad4c06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad51d8b700 1 --2- 172.21.15.162:0/1318908204 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad4c0fdc50 0x7fad4c06b660 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 rx=0x7fad480041d0 tx=0x7fad48009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad3f7fe700 10 client.?.objecter ms_handle_connect 0x7fad4c0fdc50 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad3f7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad3f7fe700 1 -- 172.21.15.162:0/1318908204 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48007b40 con 0x7fad4c0fdc50 2024-02-13T00:56:33.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad547f0700 1 -- 172.21.15.162:0/1318908204 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad440097c0 con 0x7fad4c0fdc50 2024-02-13T00:56:33.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad3f7fe700 1 -- 172.21.15.162:0/1318908204 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad4800f040 con 0x7fad4c0fdc50 2024-02-13T00:56:33.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad3f7fe700 1 -- 172.21.15.162:0/1318908204 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48014770 con 0x7fad4c0fdc50 2024-02-13T00:56:33.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad547f0700 10 client.38480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:33.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad547f0700 1 -- 172.21.15.162:0/1318908204 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c116e90 con 0x7fad4c0fdc50 2024-02-13T00:56:33.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.497+0000 7fad547f0700 1 librados: init done 2024-02-13T00:56:33.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.498+0000 7fad547f0700 10 client.38480.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:56:33.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.498+0000 7fad547f0700 15 client.38480.objecter _session_command_op_assign -1 1 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.498+0000 7fad547f0700 20 client.38480.objecter _get_session osd=-1 returning homeless 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.498+0000 7fad547f0700 10 client.38480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.498+0000 7fad547f0700 1 -- 172.21.15.162:0/1318908204 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fad4c0475e0 con 0x7fad4c0fdc50 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.498+0000 7fad3f7fe700 1 -- 172.21.15.162:0/1318908204 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fad48009110 con 0x7fad4c0fdc50 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.498+0000 7fad3f7fe700 1 --2- 172.21.15.162:0/1318908204 >> v2:172.21.15.162:6800/2866534053 conn(0x7fad38062c90 0x7fad38065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 1 -- 172.21.15.162:0/1318908204 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(852..852 src has 449..852) v4 ==== 7509+0+0 (secure 0 0 0) 0x7fad4807d050 con 0x7fad4c0fdc50 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad5258c700 1 --2- 172.21.15.162:0/1318908204 >> v2:172.21.15.162:6800/2866534053 conn(0x7fad38062c90 0x7fad38065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 10 client.38480.objecter ms_dispatch 0x7fad4c001550 osd_map(852..852 src has 449..852) v4 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 3 client.38480.objecter handle_osd_map got epochs [852,852] > 0 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 3 client.38480.objecter handle_osd_map decoding full epoch 852 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 10 client.38480.objecter checking command 1 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 1 --2- 172.21.15.162:0/1318908204 >> v2:172.21.15.162:6813/604971011 conn(0x7fad3806e060 0x7fad38070500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 20 client.38480.objecter _get_session s=0x7fad3806d9b0 osd=3 3 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 20 client.38480.objecter put_session s=0x7fad3806d9b0 osd=3 3 2024-02-13T00:56:33.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 15 client.38480.objecter _session_command_op_remove -1 1 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 20 client.38480.objecter _get_session s=0x7fad3806d9b0 osd=3 3 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 20 client.38480.objecter get_session s=0x7fad3806d9b0 osd=3 3 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 15 client.38480.objecter _session_command_op_assign 3 1 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 20 client.38480.objecter put_session s=0x7fad3806d9b0 osd=3 4 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 10 client.38480.objecter _send_command 1 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 1 -- 172.21.15.162:0/1318908204 --> v2:172.21.15.162:6813/604971011 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad38070b40 con 0x7fad3806e060 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 20 client.38480.objecter dump_active .. 0 homeless 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 1 -- 172.21.15.162:0/1318908204 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=852) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad4807f670 con 0x7fad4c0fdc50 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad52d8d700 1 --2- 172.21.15.162:0/1318908204 >> v2:172.21.15.162:6813/604971011 conn(0x7fad3806e060 0x7fad38070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 10 client.38480.objecter ms_handle_connect 0x7fad38062c90 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad5258c700 1 --2- 172.21.15.162:0/1318908204 >> v2:172.21.15.162:6800/2866534053 conn(0x7fad38062c90 0x7fad38065150 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 rx=0x7fad40009760 tx=0x7fad4000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad3f7fe700 10 client.38480.objecter ms_handle_connect 0x7fad3806e060 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.499+0000 7fad52d8d700 1 --2- 172.21.15.162:0/1318908204 >> v2:172.21.15.162:6813/604971011 conn(0x7fad3806e060 0x7fad38070500 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.500+0000 7fad3f7fe700 1 -- 172.21.15.162:0/1318908204 <== osd.3 v2:172.21.15.162:6813/604971011 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fad38070b40 con 0x7fad3806e060 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.500+0000 7fad3f7fe700 10 client.38480.objecter ms_dispatch 0x7fad4c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:56:33.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.500+0000 7fad3f7fe700 10 client.38480.objecter _finish_command 1 = system:0 2024-02-13T00:56:33.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.500+0000 7fad3f7fe700 20 client.38480.objecter put_session s=0x7fad3806d9b0 osd=3 4 2024-02-13T00:56:33.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.500+0000 7fad3f7fe700 15 client.38480.objecter _session_command_op_remove 3 1 2024-02-13T00:56:33.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 10 client.38480.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:56:33.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 15 client.38480.objecter _session_command_op_assign -1 2 2024-02-13T00:56:33.517 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 20 client.38480.objecter _get_session s=0x7fad3806d9b0 osd=3 3 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 20 client.38480.objecter put_session s=0x7fad3806d9b0 osd=3 3 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 20 client.38480.objecter _get_session s=0x7fad3806d9b0 osd=3 3 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 15 client.38480.objecter _session_command_op_remove -1 2 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 20 client.38480.objecter get_session s=0x7fad3806d9b0 osd=3 3 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 15 client.38480.objecter _session_command_op_assign 3 2 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 20 client.38480.objecter put_session s=0x7fad3806d9b0 osd=3 4 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 10 client.38480.objecter _send_command 2 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 1 -- 172.21.15.162:0/1318908204 --> v2:172.21.15.162:6813/604971011 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fad4c1071b0 con 0x7fad3806e060 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad3f7fe700 1 -- 172.21.15.162:0/1318908204 <== osd.3 v2:172.21.15.162:6813/604971011 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fad4c1071b0 con 0x7fad3806e060 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad3f7fe700 10 client.38480.objecter ms_dispatch 0x7fad4c001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad3f7fe700 10 client.38480.objecter _finish_command 2 = system:0 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad3f7fe700 20 client.38480.objecter put_session s=0x7fad3806d9b0 osd=3 4 2024-02-13T00:56:33.518 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad3f7fe700 15 client.38480.objecter _session_command_op_remove 3 2 2024-02-13T00:56:33.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 10 librados: watch_flush enter 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 10 librados: watch_flush exit 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 10 client.38480.objecter close_session for osd.3 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 1 -- 172.21.15.162:0/1318908204 >> v2:172.21.15.162:6813/604971011 conn(0x7fad3806e060 msgr2=0x7fad38070500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 1 --2- 172.21.15.162:0/1318908204 >> v2:172.21.15.162:6813/604971011 conn(0x7fad3806e060 0x7fad38070500 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 20 client.38480.objecter put_session s=0x7fad3806d9b0 osd=3 1 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 20 client.38480.objecter shutdown clearing up homeless session... 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 10 client.38480.objecter successfully canceled tick 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 1 -- 172.21.15.162:0/1318908204 >> v2:172.21.15.162:6800/2866534053 conn(0x7fad38062c90 msgr2=0x7fad38065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 1 --2- 172.21.15.162:0/1318908204 >> v2:172.21.15.162:6800/2866534053 conn(0x7fad38062c90 0x7fad38065150 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 rx=0x7fad40009760 tx=0x7fad4000b040).stop 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 1 -- 172.21.15.162:0/1318908204 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad4c0fdc50 msgr2=0x7fad4c06b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 1 --2- 172.21.15.162:0/1318908204 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad4c0fdc50 0x7fad4c06b660 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 rx=0x7fad480041d0 tx=0x7fad48009f60).stop 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad52d8d700 1 -- 172.21.15.162:0/1318908204 reap_dead start 2024-02-13T00:56:33.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 1 -- 172.21.15.162:0/1318908204 shutdown_connections 2024-02-13T00:56:33.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 1 -- 172.21.15.162:0/1318908204 >> 172.21.15.162:0/1318908204 conn(0x7fad4c0f38a0 msgr2=0x7fad4c0f50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:33.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.514+0000 7fad547f0700 1 -- 172.21.15.162:0/1318908204 shutdown_connections 2024-02-13T00:56:33.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.515+0000 7fad547f0700 1 -- 172.21.15.162:0/1318908204 wait complete. 2024-02-13T00:56:33.521 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:33.515+0000 7fad547f0700 1 librados: shutdown 2024-02-13T00:56:34.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:32.592279+0000 mgr.y (mgr.14544) 747 : cluster [DBG] pgmap v1341: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:34.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:32.721511+0000 osd.0 (osd.0) 395 : cluster [DBG] 28.16 scrub starts 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:32.724536+0000 osd.0 (osd.0) 396 : cluster [DBG] 28.16 scrub ok 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:32.957406+0000 osd.1 (osd.1) 409 : cluster [DBG] 4.4 scrub starts 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:32.960996+0000 osd.1 (osd.1) 410 : cluster [DBG] 4.4 scrub ok 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:33.072085+0000 osd.5 (osd.5) 66 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:33.075258+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:33.240406+0000 osd.6 (osd.6) 342 : cluster [DBG] 28.f deep-scrub starts 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:33.243013+0000 osd.6 (osd.6) 343 : cluster [DBG] 28.f deep-scrub ok 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:32.592279+0000 mgr.y (mgr.14544) 747 : cluster [DBG] pgmap v1341: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:32.721511+0000 osd.0 (osd.0) 395 : cluster [DBG] 28.16 scrub starts 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:32.724536+0000 osd.0 (osd.0) 396 : cluster [DBG] 28.16 scrub ok 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:32.957406+0000 osd.1 (osd.1) 409 : cluster [DBG] 4.4 scrub starts 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:32.960996+0000 osd.1 (osd.1) 410 : cluster [DBG] 4.4 scrub ok 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:33.072085+0000 osd.5 (osd.5) 66 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:56:34.079 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:33.075258+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:56:34.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:33.240406+0000 osd.6 (osd.6) 342 : cluster [DBG] 28.f deep-scrub starts 2024-02-13T00:56:34.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:33 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:33.243013+0000 osd.6 (osd.6) 343 : cluster [DBG] 28.f deep-scrub ok 2024-02-13T00:56:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:32.592279+0000 mgr.y (mgr.14544) 747 : cluster [DBG] pgmap v1341: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:32.721511+0000 osd.0 (osd.0) 395 : cluster [DBG] 28.16 scrub starts 2024-02-13T00:56:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:32.724536+0000 osd.0 (osd.0) 396 : cluster [DBG] 28.16 scrub ok 2024-02-13T00:56:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:32.957406+0000 osd.1 (osd.1) 409 : cluster [DBG] 4.4 scrub starts 2024-02-13T00:56:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:32.960996+0000 osd.1 (osd.1) 410 : cluster [DBG] 4.4 scrub ok 2024-02-13T00:56:34.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:33.072085+0000 osd.5 (osd.5) 66 : cluster [DBG] 2.2 scrub starts 2024-02-13T00:56:34.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:33.075258+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.2 scrub ok 2024-02-13T00:56:34.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:33.240406+0000 osd.6 (osd.6) 342 : cluster [DBG] 28.f deep-scrub starts 2024-02-13T00:56:34.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:33 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:33.243013+0000 osd.6 (osd.6) 343 : cluster [DBG] 28.f deep-scrub ok 2024-02-13T00:56:34.415 INFO:teuthology.orchestra.run.smithi162.stdout:107374182649 2024-02-13T00:56:34.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-13T00:56:34.692 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:35.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:33.154633+0000 osd.7 (osd.7) 132 : cluster [DBG] 6.29 scrub starts 2024-02-13T00:56:35.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:33.157749+0000 osd.7 (osd.7) 133 : cluster [DBG] 6.29 scrub ok 2024-02-13T00:56:35.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:33.740708+0000 osd.0 (osd.0) 397 : cluster [DBG] 28.12 scrub starts 2024-02-13T00:56:35.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:33.741769+0000 osd.0 (osd.0) 398 : cluster [DBG] 28.12 scrub ok 2024-02-13T00:56:35.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:34.264149+0000 osd.6 (osd.6) 344 : cluster [DBG] 28.8 scrub starts 2024-02-13T00:56:35.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:34.267239+0000 osd.6 (osd.6) 345 : cluster [DBG] 28.8 scrub ok 2024-02-13T00:56:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:33.154633+0000 osd.7 (osd.7) 132 : cluster [DBG] 6.29 scrub starts 2024-02-13T00:56:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:33.157749+0000 osd.7 (osd.7) 133 : cluster [DBG] 6.29 scrub ok 2024-02-13T00:56:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:33.740708+0000 osd.0 (osd.0) 397 : cluster [DBG] 28.12 scrub starts 2024-02-13T00:56:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:33.741769+0000 osd.0 (osd.0) 398 : cluster [DBG] 28.12 scrub ok 2024-02-13T00:56:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:34.264149+0000 osd.6 (osd.6) 344 : cluster [DBG] 28.8 scrub starts 2024-02-13T00:56:35.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:34 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:34.267239+0000 osd.6 (osd.6) 345 : cluster [DBG] 28.8 scrub ok 2024-02-13T00:56:35.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:33.154633+0000 osd.7 (osd.7) 132 : cluster [DBG] 6.29 scrub starts 2024-02-13T00:56:35.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:33.157749+0000 osd.7 (osd.7) 133 : cluster [DBG] 6.29 scrub ok 2024-02-13T00:56:35.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:33.740708+0000 osd.0 (osd.0) 397 : cluster [DBG] 28.12 scrub starts 2024-02-13T00:56:35.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:33.741769+0000 osd.0 (osd.0) 398 : cluster [DBG] 28.12 scrub ok 2024-02-13T00:56:35.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:34.264149+0000 osd.6 (osd.6) 344 : cluster [DBG] 28.8 scrub starts 2024-02-13T00:56:35.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:34 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:34.267239+0000 osd.6 (osd.6) 345 : cluster [DBG] 28.8 scrub ok 2024-02-13T00:56:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:34.592924+0000 mgr.y (mgr.14544) 748 : cluster [DBG] pgmap v1342: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:36.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:35.628613+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-02-13T00:56:36.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:34.592924+0000 mgr.y (mgr.14544) 748 : cluster [DBG] pgmap v1342: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:36.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:35.628613+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-02-13T00:56:36.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:34.592924+0000 mgr.y (mgr.14544) 748 : cluster [DBG] pgmap v1342: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:36.103 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:35 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:35.628613+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-02-13T00:56:36.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.523+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2483711134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9a40f8500 msgr2=0x7fa9a4104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:36.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.523+0000 7fa9a8fe8700 1 --2- 172.21.15.162:0/2483711134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9a40f8500 0x7fa9a4104bd0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7fa99c009be0 tx=0x7fa99c004d10).stop 2024-02-13T00:56:36.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.523+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2483711134 shutdown_connections 2024-02-13T00:56:36.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.523+0000 7fa9a8fe8700 1 --2- 172.21.15.162:0/2483711134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9a40f8500 0x7fa9a4104bd0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:36.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.523+0000 7fa9a8fe8700 1 --2- 172.21.15.162:0/2483711134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa9a40f9b70 0x7fa9a40f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:36.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.523+0000 7fa9a8fe8700 1 --2- 172.21.15.162:0/2483711134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a40fa440 0x7fa9a40f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.523+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2483711134 >> 172.21.15.162:0/2483711134 conn(0x7fa9a40f38c0 msgr2=0x7fa9a40f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.523+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2483711134 shutdown_connections 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.523+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2483711134 wait complete. 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.523+0000 7fa9a8fe8700 1 librados: starting msgr at 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.523+0000 7fa9a8fe8700 1 librados: starting objecter 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a8fe8700 1 Processor -- start 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a8fe8700 1 -- start start 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a8fe8700 1 librados: setting wanted keys 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a8fe8700 1 librados: calling monclient init 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a8fe8700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa9a40f8500 0x7fa9a410e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a8fe8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a40f9b70 0x7fa9a410e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a8fe8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9a40fa440 0x7fa9a4108240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a8fe8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a4108730 con 0x7fa9a40fa440 2024-02-13T00:56:36.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a8fe8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a41088a0 con 0x7fa9a40f9b70 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a8fe8700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a4108a60 con 0x7fa9a40f8500 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a259c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa9a40f8500 0x7fa9a410e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a2d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9a40fa440 0x7fa9a4108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a259c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa9a40f8500 0x7fa9a410e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:43870/0 (socket says 172.21.15.162:43870) 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a2d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9a40fa440 0x7fa9a4108240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:33274/0 (socket says 172.21.15.162:33274) 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a259c700 1 -- 172.21.15.162:0/2864319716 learned_addr learned my addr 172.21.15.162:0/2864319716 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a1d9b700 1 --2- 172.21.15.162:0/2864319716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a40f9b70 0x7fa9a410e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a259c700 1 -- 172.21.15.162:0/2864319716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a40f9b70 msgr2=0x7fa9a410e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a259c700 1 --2- 172.21.15.162:0/2864319716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a40f9b70 0x7fa9a410e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a259c700 1 -- 172.21.15.162:0/2864319716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9a40fa440 msgr2=0x7fa9a4108240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a259c700 1 --2- 172.21.15.162:0/2864319716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9a40fa440 0x7fa9a4108240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a259c700 1 -- 172.21.15.162:0/2864319716 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa998005040 con 0x7fa9a40f8500 2024-02-13T00:56:36.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a2d9d700 1 --2- 172.21.15.162:0/2864319716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9a40fa440 0x7fa9a4108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:36.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.524+0000 7fa9a1d9b700 1 --2- 172.21.15.162:0/2864319716 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a40f9b70 0x7fa9a410e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:56:36.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9a259c700 1 --2- 172.21.15.162:0/2864319716 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa9a40f8500 0x7fa9a410e0a0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7fa9980048f0 tx=0x7fa998009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:36.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9937fe700 10 client.?.objecter ms_handle_connect 0x7fa9a40f8500 2024-02-13T00:56:36.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9937fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:36.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9937fe700 1 -- 172.21.15.162:0/2864319716 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998007b00 con 0x7fa9a40f8500 2024-02-13T00:56:36.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9937fe700 1 -- 172.21.15.162:0/2864319716 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa99800f040 con 0x7fa9a40f8500 2024-02-13T00:56:36.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9937fe700 1 -- 172.21.15.162:0/2864319716 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998014a00 con 0x7fa9a40f8500 2024-02-13T00:56:36.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2864319716 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c0097c0 con 0x7fa9a40f8500 2024-02-13T00:56:36.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9a8fe8700 10 client.38486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:36.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2864319716 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a41090a0 con 0x7fa9a40f8500 2024-02-13T00:56:36.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9a8fe8700 1 librados: init done 2024-02-13T00:56:36.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9a8fe8700 10 client.38486.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:56:36.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9a8fe8700 15 client.38486.objecter _session_command_op_assign -1 1 2024-02-13T00:56:36.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9a8fe8700 20 client.38486.objecter _get_session osd=-1 returning homeless 2024-02-13T00:56:36.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9a8fe8700 10 client.38486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:36.533 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.525+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2864319716 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa9a40475e0 con 0x7fa9a40f8500 2024-02-13T00:56:36.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.526+0000 7fa9937fe700 1 -- 172.21.15.162:0/2864319716 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fa99800e580 con 0x7fa9a40f8500 2024-02-13T00:56:36.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.526+0000 7fa9937fe700 1 --2- 172.21.15.162:0/2864319716 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa98c062c90 0x7fa98c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:36.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 1 -- 172.21.15.162:0/2864319716 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(853..853 src has 449..853) v4 ==== 7509+0+0 (secure 0 0 0) 0x7fa99807caa0 con 0x7fa9a40f8500 2024-02-13T00:56:36.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 10 client.38486.objecter ms_dispatch 0x7fa9a4001550 osd_map(853..853 src has 449..853) v4 2024-02-13T00:56:36.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 3 client.38486.objecter handle_osd_map got epochs [853,853] > 0 2024-02-13T00:56:36.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 3 client.38486.objecter handle_osd_map decoding full epoch 853 2024-02-13T00:56:36.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9a1d9b700 1 --2- 172.21.15.162:0/2864319716 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa98c062c90 0x7fa98c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:36.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 10 client.38486.objecter checking command 1 2024-02-13T00:56:36.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 1 --2- 172.21.15.162:0/2864319716 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa98c06e0e0 0x7fa98c070580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:36.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 20 client.38486.objecter _get_session s=0x7fa98c06da30 osd=4 3 2024-02-13T00:56:36.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 20 client.38486.objecter put_session s=0x7fa98c06da30 osd=4 3 2024-02-13T00:56:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 15 client.38486.objecter _session_command_op_remove -1 1 2024-02-13T00:56:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 20 client.38486.objecter _get_session s=0x7fa98c06da30 osd=4 3 2024-02-13T00:56:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 20 client.38486.objecter get_session s=0x7fa98c06da30 osd=4 3 2024-02-13T00:56:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 15 client.38486.objecter _session_command_op_assign 4 1 2024-02-13T00:56:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 20 client.38486.objecter put_session s=0x7fa98c06da30 osd=4 4 2024-02-13T00:56:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 10 client.38486.objecter _send_command 1 2024-02-13T00:56:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 1 -- 172.21.15.162:0/2864319716 --> v2:172.21.15.181:6800/3344263388 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa98c070bc0 con 0x7fa98c06e0e0 2024-02-13T00:56:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 20 client.38486.objecter dump_active .. 0 homeless 2024-02-13T00:56:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 1 -- 172.21.15.162:0/2864319716 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=853) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa998081220 con 0x7fa9a40f8500 2024-02-13T00:56:36.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 10 client.38486.objecter ms_handle_connect 0x7fa98c062c90 2024-02-13T00:56:36.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9a1d9b700 1 --2- 172.21.15.162:0/2864319716 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa98c062c90 0x7fa98c065150 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 rx=0x7fa9a41099e0 tx=0x7fa99400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:36.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9a2d9d700 1 --2- 172.21.15.162:0/2864319716 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa98c06e0e0 0x7fa98c070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:36.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9937fe700 10 client.38486.objecter ms_handle_connect 0x7fa98c06e0e0 2024-02-13T00:56:36.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.527+0000 7fa9a2d9d700 1 --2- 172.21.15.162:0/2864319716 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa98c06e0e0 0x7fa98c070580 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:36.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.528+0000 7fa9937fe700 1 -- 172.21.15.162:0/2864319716 <== osd.4 v2:172.21.15.181:6800/3344263388 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa98c070bc0 con 0x7fa98c06e0e0 2024-02-13T00:56:36.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.528+0000 7fa9937fe700 10 client.38486.objecter ms_dispatch 0x7fa9a4001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:56:36.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.528+0000 7fa9937fe700 10 client.38486.objecter _finish_command 1 = system:0 2024-02-13T00:56:36.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.528+0000 7fa9937fe700 20 client.38486.objecter put_session s=0x7fa98c06da30 osd=4 4 2024-02-13T00:56:36.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.528+0000 7fa9937fe700 15 client.38486.objecter _session_command_op_remove 4 1 2024-02-13T00:56:36.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:56:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:56:36.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 10 client.38486.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:56:36.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 15 client.38486.objecter _session_command_op_assign -1 2 2024-02-13T00:56:36.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 20 client.38486.objecter _get_session s=0x7fa98c06da30 osd=4 3 2024-02-13T00:56:36.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 20 client.38486.objecter put_session s=0x7fa98c06da30 osd=4 3 2024-02-13T00:56:36.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 20 client.38486.objecter _get_session s=0x7fa98c06da30 osd=4 3 2024-02-13T00:56:36.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 15 client.38486.objecter _session_command_op_remove -1 2 2024-02-13T00:56:36.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 20 client.38486.objecter get_session s=0x7fa98c06da30 osd=4 3 2024-02-13T00:56:36.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 15 client.38486.objecter _session_command_op_assign 4 2 2024-02-13T00:56:36.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 20 client.38486.objecter put_session s=0x7fa98c06da30 osd=4 4 2024-02-13T00:56:36.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 10 client.38486.objecter _send_command 2 2024-02-13T00:56:36.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2864319716 --> v2:172.21.15.181:6800/3344263388 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa9a41072c0 con 0x7fa98c06e0e0 2024-02-13T00:56:36.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9937fe700 1 -- 172.21.15.162:0/2864319716 <== osd.4 v2:172.21.15.181:6800/3344263388 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa9a41072c0 con 0x7fa98c06e0e0 2024-02-13T00:56:36.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9937fe700 10 client.38486.objecter ms_dispatch 0x7fa9a4001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:56:36.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9937fe700 10 client.38486.objecter _finish_command 2 = system:0 2024-02-13T00:56:36.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9937fe700 20 client.38486.objecter put_session s=0x7fa98c06da30 osd=4 4 2024-02-13T00:56:36.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9937fe700 15 client.38486.objecter _session_command_op_remove 4 2 2024-02-13T00:56:36.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 10 librados: watch_flush enter 2024-02-13T00:56:36.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 10 librados: watch_flush exit 2024-02-13T00:56:36.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 10 client.38486.objecter close_session for osd.4 2024-02-13T00:56:36.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2864319716 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa98c06e0e0 msgr2=0x7fa98c070580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:36.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 1 --2- 172.21.15.162:0/2864319716 >> v2:172.21.15.181:6800/3344263388 conn(0x7fa98c06e0e0 0x7fa98c070580 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:36.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 20 client.38486.objecter put_session s=0x7fa98c06da30 osd=4 1 2024-02-13T00:56:36.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 20 client.38486.objecter shutdown clearing up homeless session... 2024-02-13T00:56:36.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 10 client.38486.objecter successfully canceled tick 2024-02-13T00:56:36.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2864319716 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa98c062c90 msgr2=0x7fa98c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:36.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 1 --2- 172.21.15.162:0/2864319716 >> v2:172.21.15.162:6800/2866534053 conn(0x7fa98c062c90 0x7fa98c065150 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 rx=0x7fa9a41099e0 tx=0x7fa99400b040).stop 2024-02-13T00:56:36.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2864319716 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa9a40f8500 msgr2=0x7fa9a410e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:36.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.542+0000 7fa9a8fe8700 1 --2- 172.21.15.162:0/2864319716 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa9a40f8500 0x7fa9a410e0a0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7fa9980048f0 tx=0x7fa998009cb0).stop 2024-02-13T00:56:36.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.543+0000 7fa9a2d9d700 1 -- 172.21.15.162:0/2864319716 reap_dead start 2024-02-13T00:56:36.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.543+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2864319716 shutdown_connections 2024-02-13T00:56:36.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.543+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2864319716 >> 172.21.15.162:0/2864319716 conn(0x7fa9a40f38c0 msgr2=0x7fa9a40f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:36.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.543+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2864319716 shutdown_connections 2024-02-13T00:56:36.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.543+0000 7fa9a8fe8700 1 -- 172.21.15.162:0/2864319716 wait complete. 2024-02-13T00:56:36.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:36.543+0000 7fa9a8fe8700 1 librados: shutdown 2024-02-13T00:56:36.569 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:56:36] "GET /metrics HTTP/1.1" 200 185528 "" "Prometheus/2.33.4" 2024-02-13T00:56:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:35.975059+0000 osd.1 (osd.1) 411 : cluster [DBG] 6.4 scrub starts 2024-02-13T00:56:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:35.978095+0000 osd.1 (osd.1) 412 : cluster [DBG] 6.4 scrub ok 2024-02-13T00:56:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:36.021620+0000 osd.4 (osd.4) 83 : cluster [DBG] 6.2 scrub starts 2024-02-13T00:56:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:36.024529+0000 osd.4 (osd.4) 84 : cluster [DBG] 6.2 scrub ok 2024-02-13T00:56:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:36.100166+0000 mgr.y (mgr.14544) 749 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:36.213573+0000 osd.6 (osd.6) 346 : cluster [DBG] 6.27 scrub starts 2024-02-13T00:56:37.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:36.214948+0000 osd.6 (osd.6) 347 : cluster [DBG] 6.27 scrub ok 2024-02-13T00:56:37.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:35.975059+0000 osd.1 (osd.1) 411 : cluster [DBG] 6.4 scrub starts 2024-02-13T00:56:37.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:35.978095+0000 osd.1 (osd.1) 412 : cluster [DBG] 6.4 scrub ok 2024-02-13T00:56:37.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:36.021620+0000 osd.4 (osd.4) 83 : cluster [DBG] 6.2 scrub starts 2024-02-13T00:56:37.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:36.024529+0000 osd.4 (osd.4) 84 : cluster [DBG] 6.2 scrub ok 2024-02-13T00:56:37.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:36.100166+0000 mgr.y (mgr.14544) 749 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:37.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:36.213573+0000 osd.6 (osd.6) 346 : cluster [DBG] 6.27 scrub starts 2024-02-13T00:56:37.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:36 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:36.214948+0000 osd.6 (osd.6) 347 : cluster [DBG] 6.27 scrub ok 2024-02-13T00:56:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:35.975059+0000 osd.1 (osd.1) 411 : cluster [DBG] 6.4 scrub starts 2024-02-13T00:56:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:35.978095+0000 osd.1 (osd.1) 412 : cluster [DBG] 6.4 scrub ok 2024-02-13T00:56:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:36.021620+0000 osd.4 (osd.4) 83 : cluster [DBG] 6.2 scrub starts 2024-02-13T00:56:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:36.024529+0000 osd.4 (osd.4) 84 : cluster [DBG] 6.2 scrub ok 2024-02-13T00:56:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:36.100166+0000 mgr.y (mgr.14544) 749 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:36.213573+0000 osd.6 (osd.6) 346 : cluster [DBG] 6.27 scrub starts 2024-02-13T00:56:37.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:36 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:36.214948+0000 osd.6 (osd.6) 347 : cluster [DBG] 6.27 scrub ok 2024-02-13T00:56:37.334 INFO:teuthology.orchestra.run.smithi162.stdout:133143986422 2024-02-13T00:56:37.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-13T00:56:37.606 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:38.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:36.593380+0000 mgr.y (mgr.14544) 750 : cluster [DBG] pgmap v1344: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:38.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:37.068300+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.c scrub starts 2024-02-13T00:56:38.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:37.071984+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.c scrub ok 2024-02-13T00:56:38.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:37.180903+0000 osd.6 (osd.6) 348 : cluster [DBG] 6.23 scrub starts 2024-02-13T00:56:38.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:37.183849+0000 osd.6 (osd.6) 349 : cluster [DBG] 6.23 scrub ok 2024-02-13T00:56:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:36.593380+0000 mgr.y (mgr.14544) 750 : cluster [DBG] pgmap v1344: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:37.068300+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.c scrub starts 2024-02-13T00:56:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:37.071984+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.c scrub ok 2024-02-13T00:56:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:37.180903+0000 osd.6 (osd.6) 348 : cluster [DBG] 6.23 scrub starts 2024-02-13T00:56:38.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:37 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:37.183849+0000 osd.6 (osd.6) 349 : cluster [DBG] 6.23 scrub ok 2024-02-13T00:56:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:36.593380+0000 mgr.y (mgr.14544) 750 : cluster [DBG] pgmap v1344: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:37.068300+0000 osd.4 (osd.4) 85 : cluster [DBG] 4.c scrub starts 2024-02-13T00:56:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:37.071984+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.c scrub ok 2024-02-13T00:56:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:37.180903+0000 osd.6 (osd.6) 348 : cluster [DBG] 6.23 scrub starts 2024-02-13T00:56:38.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:37 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:37.183849+0000 osd.6 (osd.6) 349 : cluster [DBG] 6.23 scrub ok 2024-02-13T00:56:39.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:56:38 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:56:38] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:56:39.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.416+0000 7fd760833700 1 -- 172.21.15.162:0/964328607 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd75c0fcb30 msgr2=0x7fd75c0fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:39.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.416+0000 7fd760833700 1 --2- 172.21.15.162:0/964328607 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd75c0fcb30 0x7fd75c0fcf10 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 rx=0x7fd74c0047b0 tx=0x7fd74c009cb0).stop 2024-02-13T00:56:39.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.416+0000 7fd760833700 1 -- 172.21.15.162:0/964328607 shutdown_connections 2024-02-13T00:56:39.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.416+0000 7fd760833700 1 --2- 172.21.15.162:0/964328607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd75c0f73a0 0x7fd75c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:39.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.416+0000 7fd760833700 1 --2- 172.21.15.162:0/964328607 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd75c0fcb30 0x7fd75c0fcf10 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:39.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.416+0000 7fd760833700 1 --2- 172.21.15.162:0/964328607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd75c0fd490 0x7fd75c0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:39.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.416+0000 7fd760833700 1 -- 172.21.15.162:0/964328607 >> 172.21.15.162:0/964328607 conn(0x7fd75c06eb90 msgr2=0x7fd75c0f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:39.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.416+0000 7fd760833700 1 -- 172.21.15.162:0/964328607 shutdown_connections 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.416+0000 7fd760833700 1 -- 172.21.15.162:0/964328607 wait complete. 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd760833700 1 librados: starting msgr at 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd760833700 1 librados: starting objecter 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd760833700 1 Processor -- start 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd760833700 1 -- start start 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd760833700 1 librados: setting wanted keys 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd760833700 1 librados: calling monclient init 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd760833700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd75c0f73a0 0x7fd75c10e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd760833700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd75c0fcb30 0x7fd75c10e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd760833700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd75c0fd490 0x7fd75c108310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd760833700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c108800 con 0x7fd75c0fcb30 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd760833700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c108970 con 0x7fd75c0fd490 2024-02-13T00:56:39.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd760833700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c108b30 con 0x7fd75c0f73a0 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd75a59c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd75c0f73a0 0x7fd75c10e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd759d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd75c0fcb30 0x7fd75c10e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd75a59c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd75c0f73a0 0x7fd75c10e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:43886/0 (socket says 172.21.15.162:43886) 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd759d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd75c0fcb30 0x7fd75c10e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:33300/0 (socket says 172.21.15.162:33300) 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd75a59c700 1 -- 172.21.15.162:0/2419947335 learned_addr learned my addr 172.21.15.162:0/2419947335 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.417+0000 7fd75ad9d700 1 --2- 172.21.15.162:0/2419947335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd75c0fd490 0x7fd75c108310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd75a59c700 1 -- 172.21.15.162:0/2419947335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd75c0fd490 msgr2=0x7fd75c108310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd75a59c700 1 --2- 172.21.15.162:0/2419947335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd75c0fd490 0x7fd75c108310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd75a59c700 1 -- 172.21.15.162:0/2419947335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd75c0fcb30 msgr2=0x7fd75c10e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd75a59c700 1 --2- 172.21.15.162:0/2419947335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd75c0fcb30 0x7fd75c10e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd75a59c700 1 -- 172.21.15.162:0/2419947335 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7540097c0 con 0x7fd75c0f73a0 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd759d9b700 1 --2- 172.21.15.162:0/2419947335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd75c0fcb30 0x7fd75c10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:39.453 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd75a59c700 1 --2- 172.21.15.162:0/2419947335 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd75c0f73a0 0x7fd75c10e170 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 rx=0x7fd74c0044e0 tx=0x7fd74c007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd74b7fe700 10 client.?.objecter ms_handle_connect 0x7fd75c0f73a0 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd74b7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd74b7fe700 1 -- 172.21.15.162:0/2419947335 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c018070 con 0x7fd75c0f73a0 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd74b7fe700 1 -- 172.21.15.162:0/2419947335 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd74c0093b0 con 0x7fd75c0f73a0 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd760833700 1 -- 172.21.15.162:0/2419947335 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74c005040 con 0x7fd75c0f73a0 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd74b7fe700 1 -- 172.21.15.162:0/2419947335 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c01dd90 con 0x7fd75c0f73a0 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd760833700 10 client.38492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd760833700 1 -- 172.21.15.162:0/2419947335 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75c108fd0 con 0x7fd75c0f73a0 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.418+0000 7fd760833700 1 librados: init done 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.419+0000 7fd760833700 10 client.38492.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.419+0000 7fd760833700 15 client.38492.objecter _session_command_op_assign -1 1 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.419+0000 7fd760833700 20 client.38492.objecter _get_session osd=-1 returning homeless 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.419+0000 7fd760833700 10 client.38492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.419+0000 7fd760833700 1 -- 172.21.15.162:0/2419947335 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd75c0475e0 con 0x7fd75c0f73a0 2024-02-13T00:56:39.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.419+0000 7fd74b7fe700 1 -- 172.21.15.162:0/2419947335 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fd74c01d420 con 0x7fd75c0f73a0 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 1 --2- 172.21.15.162:0/2419947335 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd744062c90 0x7fd744065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 1 -- 172.21.15.162:0/2419947335 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(853..853 src has 449..853) v4 ==== 7509+0+0 (secure 0 0 0) 0x7fd74c07e400 con 0x7fd75c0f73a0 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 10 client.38492.objecter ms_dispatch 0x7fd75c001550 osd_map(853..853 src has 449..853) v4 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 3 client.38492.objecter handle_osd_map got epochs [853,853] > 0 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 3 client.38492.objecter handle_osd_map decoding full epoch 853 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd759d9b700 1 --2- 172.21.15.162:0/2419947335 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd744062c90 0x7fd744065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 10 client.38492.objecter checking command 1 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 1 --2- 172.21.15.162:0/2419947335 >> v2:172.21.15.181:6804/2277917372 conn(0x7fd74406e060 0x7fd744070500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 20 client.38492.objecter _get_session s=0x7fd74406d9b0 osd=5 3 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 20 client.38492.objecter put_session s=0x7fd74406d9b0 osd=5 3 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 15 client.38492.objecter _session_command_op_remove -1 1 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 20 client.38492.objecter _get_session s=0x7fd74406d9b0 osd=5 3 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 20 client.38492.objecter get_session s=0x7fd74406d9b0 osd=5 3 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 15 client.38492.objecter _session_command_op_assign 5 1 2024-02-13T00:56:39.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 20 client.38492.objecter put_session s=0x7fd74406d9b0 osd=5 4 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 10 client.38492.objecter _send_command 1 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 1 -- 172.21.15.162:0/2419947335 --> v2:172.21.15.181:6804/2277917372 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd744070b40 con 0x7fd74406e060 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 20 client.38492.objecter dump_active .. 0 homeless 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 1 -- 172.21.15.162:0/2419947335 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=853) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd74c080a20 con 0x7fd75c0f73a0 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd759d9b700 1 --2- 172.21.15.162:0/2419947335 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd744062c90 0x7fd744065150 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 rx=0x7fd75c109ab0 tx=0x7fd75000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 10 client.38492.objecter ms_handle_connect 0x7fd744062c90 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd75ad9d700 1 --2- 172.21.15.162:0/2419947335 >> v2:172.21.15.181:6804/2277917372 conn(0x7fd74406e060 0x7fd744070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd74b7fe700 10 client.38492.objecter ms_handle_connect 0x7fd74406e060 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.420+0000 7fd75ad9d700 1 --2- 172.21.15.162:0/2419947335 >> v2:172.21.15.181:6804/2277917372 conn(0x7fd74406e060 0x7fd744070500 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.421+0000 7fd74b7fe700 1 -- 172.21.15.162:0/2419947335 <== osd.5 v2:172.21.15.181:6804/2277917372 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd744070b40 con 0x7fd74406e060 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.421+0000 7fd74b7fe700 10 client.38492.objecter ms_dispatch 0x7fd75c001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.421+0000 7fd74b7fe700 10 client.38492.objecter _finish_command 1 = system:0 2024-02-13T00:56:39.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.421+0000 7fd74b7fe700 20 client.38492.objecter put_session s=0x7fd74406d9b0 osd=5 4 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.421+0000 7fd74b7fe700 15 client.38492.objecter _session_command_op_remove 5 1 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd760833700 10 client.38492.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd760833700 15 client.38492.objecter _session_command_op_assign -1 2 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd760833700 20 client.38492.objecter _get_session s=0x7fd74406d9b0 osd=5 3 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd760833700 20 client.38492.objecter put_session s=0x7fd74406d9b0 osd=5 3 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd760833700 20 client.38492.objecter _get_session s=0x7fd74406d9b0 osd=5 3 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd760833700 15 client.38492.objecter _session_command_op_remove -1 2 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd760833700 20 client.38492.objecter get_session s=0x7fd74406d9b0 osd=5 3 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd760833700 15 client.38492.objecter _session_command_op_assign 5 2 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd760833700 20 client.38492.objecter put_session s=0x7fd74406d9b0 osd=5 4 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd760833700 10 client.38492.objecter _send_command 2 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd760833700 1 -- 172.21.15.162:0/2419947335 --> v2:172.21.15.181:6804/2277917372 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd75c106090 con 0x7fd74406e060 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd74b7fe700 1 -- 172.21.15.162:0/2419947335 <== osd.5 v2:172.21.15.181:6804/2277917372 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+13 (crc 0 0 0) 0x7fd75c106090 con 0x7fd74406e060 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd74b7fe700 10 client.38492.objecter ms_dispatch 0x7fd75c001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.436+0000 7fd74b7fe700 10 client.38492.objecter _finish_command 2 = system:0 2024-02-13T00:56:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd74b7fe700 20 client.38492.objecter put_session s=0x7fd74406d9b0 osd=5 4 2024-02-13T00:56:39.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd74b7fe700 15 client.38492.objecter _session_command_op_remove 5 2 2024-02-13T00:56:39.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 10 librados: watch_flush enter 2024-02-13T00:56:39.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 10 librados: watch_flush exit 2024-02-13T00:56:39.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 10 client.38492.objecter close_session for osd.5 2024-02-13T00:56:39.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 1 -- 172.21.15.162:0/2419947335 >> v2:172.21.15.181:6804/2277917372 conn(0x7fd74406e060 msgr2=0x7fd744070500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:39.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 1 --2- 172.21.15.162:0/2419947335 >> v2:172.21.15.181:6804/2277917372 conn(0x7fd74406e060 0x7fd744070500 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:39.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 20 client.38492.objecter put_session s=0x7fd74406d9b0 osd=5 1 2024-02-13T00:56:39.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 20 client.38492.objecter shutdown clearing up homeless session... 2024-02-13T00:56:39.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 10 client.38492.objecter successfully canceled tick 2024-02-13T00:56:39.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 1 -- 172.21.15.162:0/2419947335 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd744062c90 msgr2=0x7fd744065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:39.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 1 --2- 172.21.15.162:0/2419947335 >> v2:172.21.15.162:6800/2866534053 conn(0x7fd744062c90 0x7fd744065150 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 rx=0x7fd75c109ab0 tx=0x7fd75000b040).stop 2024-02-13T00:56:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 1 -- 172.21.15.162:0/2419947335 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd75c0f73a0 msgr2=0x7fd75c10e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 1 --2- 172.21.15.162:0/2419947335 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd75c0f73a0 0x7fd75c10e170 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 rx=0x7fd74c0044e0 tx=0x7fd74c007f90).stop 2024-02-13T00:56:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd75ad9d700 1 -- 172.21.15.162:0/2419947335 reap_dead start 2024-02-13T00:56:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 1 -- 172.21.15.162:0/2419947335 shutdown_connections 2024-02-13T00:56:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 1 -- 172.21.15.162:0/2419947335 >> 172.21.15.162:0/2419947335 conn(0x7fd75c06eb90 msgr2=0x7fd75c06ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 1 -- 172.21.15.162:0/2419947335 shutdown_connections 2024-02-13T00:56:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 1 -- 172.21.15.162:0/2419947335 wait complete. 2024-02-13T00:56:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:39.437+0000 7fd760833700 1 librados: shutdown 2024-02-13T00:56:40.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:38.593843+0000 mgr.y (mgr.14544) 751 : cluster [DBG] pgmap v1345: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:56:40.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:39 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:38.593843+0000 mgr.y (mgr.14544) 751 : cluster [DBG] pgmap v1345: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:56:40.129 INFO:teuthology.orchestra.run.smithi162.stdout:2057289334889 2024-02-13T00:56:40.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-13T00:56:40.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:39 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:38.593843+0000 mgr.y (mgr.14544) 751 : cluster [DBG] pgmap v1345: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:56:40.401 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:41.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:39.824798+0000 osd.0 (osd.0) 399 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:56:41.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:39.825795+0000 osd.0 (osd.0) 400 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:56:41.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:39.976093+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:56:41.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:39.979360+0000 osd.1 (osd.1) 414 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:56:41.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:40.082677+0000 osd.7 (osd.7) 134 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:56:41.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:40.085565+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:56:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:39.824798+0000 osd.0 (osd.0) 399 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:56:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:39.825795+0000 osd.0 (osd.0) 400 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:56:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:39.976093+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:56:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:39.979360+0000 osd.1 (osd.1) 414 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:56:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:40.082677+0000 osd.7 (osd.7) 134 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:56:41.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:40.085565+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:56:41.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:39.824798+0000 osd.0 (osd.0) 399 : cluster [DBG] 6.18 scrub starts 2024-02-13T00:56:41.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:39.825795+0000 osd.0 (osd.0) 400 : cluster [DBG] 6.18 scrub ok 2024-02-13T00:56:41.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:39.976093+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.1a scrub starts 2024-02-13T00:56:41.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:39.979360+0000 osd.1 (osd.1) 414 : cluster [DBG] 2.1a scrub ok 2024-02-13T00:56:41.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:40.082677+0000 osd.7 (osd.7) 134 : cluster [DBG] 6.12 scrub starts 2024-02-13T00:56:41.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:40.085565+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.12 scrub ok 2024-02-13T00:56:42.076 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:40.594798+0000 mgr.y (mgr.14544) 752 : cluster [DBG] pgmap v1346: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:56:42.076 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:41.066258+0000 osd.5 (osd.5) 68 : cluster [DBG] 28.3 scrub starts 2024-02-13T00:56:42.076 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:41.069736+0000 osd.5 (osd.5) 69 : cluster [DBG] 28.3 scrub ok 2024-02-13T00:56:42.076 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:41.117393+0000 osd.4 (osd.4) 87 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:56:42.076 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:41.121156+0000 osd.4 (osd.4) 88 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:56:42.076 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:41.121450+0000 osd.6 (osd.6) 350 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:56:42.076 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:41.122375+0000 osd.6 (osd.6) 351 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:56:42.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:40.594798+0000 mgr.y (mgr.14544) 752 : cluster [DBG] pgmap v1346: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:56:42.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:41.066258+0000 osd.5 (osd.5) 68 : cluster [DBG] 28.3 scrub starts 2024-02-13T00:56:42.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:41.069736+0000 osd.5 (osd.5) 69 : cluster [DBG] 28.3 scrub ok 2024-02-13T00:56:42.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:41.117393+0000 osd.4 (osd.4) 87 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:56:42.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:41.121156+0000 osd.4 (osd.4) 88 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:56:42.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:41.121450+0000 osd.6 (osd.6) 350 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:56:42.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:41.122375+0000 osd.6 (osd.6) 351 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:56:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:40.594798+0000 mgr.y (mgr.14544) 752 : cluster [DBG] pgmap v1346: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-13T00:56:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:41.066258+0000 osd.5 (osd.5) 68 : cluster [DBG] 28.3 scrub starts 2024-02-13T00:56:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:41.069736+0000 osd.5 (osd.5) 69 : cluster [DBG] 28.3 scrub ok 2024-02-13T00:56:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:41.117393+0000 osd.4 (osd.4) 87 : cluster [DBG] 4.9 scrub starts 2024-02-13T00:56:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:41.121156+0000 osd.4 (osd.4) 88 : cluster [DBG] 4.9 scrub ok 2024-02-13T00:56:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:41.121450+0000 osd.6 (osd.6) 350 : cluster [DBG] 2.7 scrub starts 2024-02-13T00:56:42.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:41 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:41.122375+0000 osd.6 (osd.6) 351 : cluster [DBG] 2.7 scrub ok 2024-02-13T00:56:42.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.294+0000 7f717ad95700 1 -- 172.21.15.162:0/1145216754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71740fa440 msgr2=0x7f71740f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:42.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.294+0000 7f717ad95700 1 --2- 172.21.15.162:0/1145216754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71740fa440 0x7f71740f7e20 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 rx=0x7f71600047b0 tx=0x7f7160009cb0).stop 2024-02-13T00:56:42.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.294+0000 7f717ad95700 1 -- 172.21.15.162:0/1145216754 shutdown_connections 2024-02-13T00:56:42.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.294+0000 7f717ad95700 1 --2- 172.21.15.162:0/1145216754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71740f8500 0x7f7174104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:42.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.294+0000 7f717ad95700 1 --2- 172.21.15.162:0/1145216754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71740f9b70 0x7f71740f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:42.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.294+0000 7f717ad95700 1 --2- 172.21.15.162:0/1145216754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71740fa440 0x7f71740f7e20 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:42.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.294+0000 7f717ad95700 1 -- 172.21.15.162:0/1145216754 >> 172.21.15.162:0/1145216754 conn(0x7f71740f38c0 msgr2=0x7f71740f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:42.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.294+0000 7f717ad95700 1 -- 172.21.15.162:0/1145216754 shutdown_connections 2024-02-13T00:56:42.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.294+0000 7f717ad95700 1 -- 172.21.15.162:0/1145216754 wait complete. 2024-02-13T00:56:42.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.295+0000 7f717ad95700 1 librados: starting msgr at 2024-02-13T00:56:42.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.295+0000 7f717ad95700 1 librados: starting objecter 2024-02-13T00:56:42.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.295+0000 7f717ad95700 1 Processor -- start 2024-02-13T00:56:42.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.295+0000 7f717ad95700 1 -- start start 2024-02-13T00:56:42.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.295+0000 7f717ad95700 1 librados: setting wanted keys 2024-02-13T00:56:42.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.295+0000 7f717ad95700 1 librados: calling monclient init 2024-02-13T00:56:42.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.295+0000 7f717ad95700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71740f8500 0x7f717406b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:42.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.295+0000 7f717ad95700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71740f9b70 0x7f717406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:42.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.295+0000 7f717ad95700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71740fa440 0x7f717406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:42.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.295+0000 7f717ad95700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7174116710 con 0x7f71740f9b70 2024-02-13T00:56:42.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.295+0000 7f717ad95700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7174116880 con 0x7f71740fa440 2024-02-13T00:56:42.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f717ad95700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7174116a40 con 0x7f71740f8500 2024-02-13T00:56:42.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7173fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71740f9b70 0x7f717406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:42.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7173fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71740f9b70 0x7f717406b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:33324/0 (socket says 172.21.15.162:33324) 2024-02-13T00:56:42.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7173fff700 1 -- 172.21.15.162:0/3460226437 learned_addr learned my addr 172.21.15.162:0/3460226437 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:42.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7179332700 1 --2- 172.21.15.162:0/3460226437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71740fa440 0x7f717406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:42.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7178b31700 1 --2- 172.21.15.162:0/3460226437 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71740f8500 0x7f717406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7173fff700 1 -- 172.21.15.162:0/3460226437 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71740f8500 msgr2=0x7f717406b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7173fff700 1 --2- 172.21.15.162:0/3460226437 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71740f8500 0x7f717406b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7173fff700 1 -- 172.21.15.162:0/3460226437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71740fa440 msgr2=0x7f717406bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7173fff700 1 --2- 172.21.15.162:0/3460226437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71740fa440 0x7f717406bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7173fff700 1 -- 172.21.15.162:0/3460226437 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7168005040 con 0x7f71740f9b70 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7173fff700 1 --2- 172.21.15.162:0/3460226437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71740f9b70 0x7f717406b660 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7f71600044e0 tx=0x7f7160007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7171ffb700 10 client.?.objecter ms_handle_connect 0x7f71740f9b70 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7171ffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7171ffb700 1 -- 172.21.15.162:0/3460226437 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7160018070 con 0x7f71740f9b70 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7171ffb700 1 -- 172.21.15.162:0/3460226437 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71600093b0 con 0x7f71740f9b70 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f717ad95700 1 -- 172.21.15.162:0/3460226437 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7160005040 con 0x7f71740f9b70 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f7171ffb700 1 -- 172.21.15.162:0/3460226437 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716001dd90 con 0x7f71740f9b70 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f717ad95700 10 client.28404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:42.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.296+0000 7f717ad95700 1 -- 172.21.15.162:0/3460226437 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7174116f60 con 0x7f71740f9b70 2024-02-13T00:56:42.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.297+0000 7f717ad95700 1 librados: init done 2024-02-13T00:56:42.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.297+0000 7f717ad95700 10 client.28404.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:56:42.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.297+0000 7f717ad95700 15 client.28404.objecter _session_command_op_assign -1 1 2024-02-13T00:56:42.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.297+0000 7f717ad95700 20 client.28404.objecter _get_session osd=-1 returning homeless 2024-02-13T00:56:42.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.297+0000 7f717ad95700 10 client.28404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:42.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.297+0000 7f717ad95700 1 -- 172.21.15.162:0/3460226437 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71740475e0 con 0x7f71740f9b70 2024-02-13T00:56:42.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 1 -- 172.21.15.162:0/3460226437 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f716001d420 con 0x7f71740f9b70 2024-02-13T00:56:42.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 1 --2- 172.21.15.162:0/3460226437 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7164062c90 0x7f7164065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:42.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 1 -- 172.21.15.162:0/3460226437 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(853..853 src has 449..853) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f716007e7a0 con 0x7f71740f9b70 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 10 client.28404.objecter ms_dispatch 0x7f7174001550 osd_map(853..853 src has 449..853) v4 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 3 client.28404.objecter handle_osd_map got epochs [853,853] > 0 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 3 client.28404.objecter handle_osd_map decoding full epoch 853 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7178b31700 1 --2- 172.21.15.162:0/3460226437 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7164062c90 0x7f7164065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 10 client.28404.objecter checking command 1 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 1 --2- 172.21.15.162:0/3460226437 >> v2:172.21.15.181:6808/1506640341 conn(0x7f716406e060 0x7f7164070500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 20 client.28404.objecter _get_session s=0x7f716406d9b0 osd=6 3 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 20 client.28404.objecter put_session s=0x7f716406d9b0 osd=6 3 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 15 client.28404.objecter _session_command_op_remove -1 1 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 20 client.28404.objecter _get_session s=0x7f716406d9b0 osd=6 3 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 20 client.28404.objecter get_session s=0x7f716406d9b0 osd=6 3 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 15 client.28404.objecter _session_command_op_assign 6 1 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 20 client.28404.objecter put_session s=0x7f716406d9b0 osd=6 4 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 10 client.28404.objecter _send_command 1 2024-02-13T00:56:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 1 -- 172.21.15.162:0/3460226437 --> v2:172.21.15.181:6808/1506640341 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7164070b40 con 0x7f716406e060 2024-02-13T00:56:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 20 client.28404.objecter dump_active .. 0 homeless 2024-02-13T00:56:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.298+0000 7f7171ffb700 1 -- 172.21.15.162:0/3460226437 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=853) v2 ==== 24+0+0 (secure 0 0 0) 0x7f716007eb50 con 0x7f71740f9b70 2024-02-13T00:56:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.299+0000 7f7171ffb700 10 client.28404.objecter ms_handle_connect 0x7f7164062c90 2024-02-13T00:56:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.299+0000 7f7178b31700 1 --2- 172.21.15.162:0/3460226437 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7164062c90 0x7f7164065150 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 rx=0x7f7168004870 tx=0x7f716800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.299+0000 7f7179332700 1 --2- 172.21.15.162:0/3460226437 >> v2:172.21.15.181:6808/1506640341 conn(0x7f716406e060 0x7f7164070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.299+0000 7f7171ffb700 10 client.28404.objecter ms_handle_connect 0x7f716406e060 2024-02-13T00:56:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.299+0000 7f7179332700 1 --2- 172.21.15.162:0/3460226437 >> v2:172.21.15.181:6808/1506640341 conn(0x7f716406e060 0x7f7164070500 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.300+0000 7f7171ffb700 1 -- 172.21.15.162:0/3460226437 <== osd.6 v2:172.21.15.181:6808/1506640341 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7164070b40 con 0x7f716406e060 2024-02-13T00:56:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.300+0000 7f7171ffb700 10 client.28404.objecter ms_dispatch 0x7f7174001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:56:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.300+0000 7f7171ffb700 10 client.28404.objecter _finish_command 1 = system:0 2024-02-13T00:56:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.300+0000 7f7171ffb700 20 client.28404.objecter put_session s=0x7f716406d9b0 osd=6 4 2024-02-13T00:56:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.300+0000 7f7171ffb700 15 client.28404.objecter _session_command_op_remove 6 1 2024-02-13T00:56:42.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f717ad95700 10 client.28404.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:56:42.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f717ad95700 15 client.28404.objecter _session_command_op_assign -1 2 2024-02-13T00:56:42.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f717ad95700 20 client.28404.objecter _get_session s=0x7f716406d9b0 osd=6 3 2024-02-13T00:56:42.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f717ad95700 20 client.28404.objecter put_session s=0x7f716406d9b0 osd=6 3 2024-02-13T00:56:42.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f717ad95700 20 client.28404.objecter _get_session s=0x7f716406d9b0 osd=6 3 2024-02-13T00:56:42.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f717ad95700 15 client.28404.objecter _session_command_op_remove -1 2 2024-02-13T00:56:42.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f717ad95700 20 client.28404.objecter get_session s=0x7f716406d9b0 osd=6 3 2024-02-13T00:56:42.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f717ad95700 15 client.28404.objecter _session_command_op_assign 6 2 2024-02-13T00:56:42.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f717ad95700 20 client.28404.objecter put_session s=0x7f716406d9b0 osd=6 4 2024-02-13T00:56:42.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f717ad95700 10 client.28404.objecter _send_command 2 2024-02-13T00:56:42.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f717ad95700 1 -- 172.21.15.162:0/3460226437 --> v2:172.21.15.181:6808/1506640341 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7174107280 con 0x7f716406e060 2024-02-13T00:56:42.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f7171ffb700 1 -- 172.21.15.162:0/3460226437 <== osd.6 v2:172.21.15.181:6808/1506640341 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7174107280 con 0x7f716406e060 2024-02-13T00:56:42.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f7171ffb700 10 client.28404.objecter ms_dispatch 0x7f7174001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:56:42.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.314+0000 7f7171ffb700 10 client.28404.objecter _finish_command 2 = system:0 2024-02-13T00:56:42.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f7171ffb700 20 client.28404.objecter put_session s=0x7f716406d9b0 osd=6 4 2024-02-13T00:56:42.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f7171ffb700 15 client.28404.objecter _session_command_op_remove 6 2 2024-02-13T00:56:42.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 10 librados: watch_flush enter 2024-02-13T00:56:42.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 10 librados: watch_flush exit 2024-02-13T00:56:42.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 10 client.28404.objecter close_session for osd.6 2024-02-13T00:56:42.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 1 -- 172.21.15.162:0/3460226437 >> v2:172.21.15.181:6808/1506640341 conn(0x7f716406e060 msgr2=0x7f7164070500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:42.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 1 --2- 172.21.15.162:0/3460226437 >> v2:172.21.15.181:6808/1506640341 conn(0x7f716406e060 0x7f7164070500 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:42.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 20 client.28404.objecter put_session s=0x7f716406d9b0 osd=6 1 2024-02-13T00:56:42.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 20 client.28404.objecter shutdown clearing up homeless session... 2024-02-13T00:56:42.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 10 client.28404.objecter successfully canceled tick 2024-02-13T00:56:42.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 1 -- 172.21.15.162:0/3460226437 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7164062c90 msgr2=0x7f7164065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:42.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 1 --2- 172.21.15.162:0/3460226437 >> v2:172.21.15.162:6800/2866534053 conn(0x7f7164062c90 0x7f7164065150 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 rx=0x7f7168004870 tx=0x7f716800b040).stop 2024-02-13T00:56:42.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 1 -- 172.21.15.162:0/3460226437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71740f9b70 msgr2=0x7f717406b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:42.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 1 --2- 172.21.15.162:0/3460226437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71740f9b70 0x7f717406b660 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7f71600044e0 tx=0x7f7160007f90).stop 2024-02-13T00:56:42.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f7179332700 1 -- 172.21.15.162:0/3460226437 reap_dead start 2024-02-13T00:56:42.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 1 -- 172.21.15.162:0/3460226437 shutdown_connections 2024-02-13T00:56:42.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 1 -- 172.21.15.162:0/3460226437 >> 172.21.15.162:0/3460226437 conn(0x7f71740f38c0 msgr2=0x7f71740f4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:42.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 1 -- 172.21.15.162:0/3460226437 shutdown_connections 2024-02-13T00:56:42.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 1 -- 172.21.15.162:0/3460226437 wait complete. 2024-02-13T00:56:42.323 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:42.315+0000 7f717ad95700 1 librados: shutdown 2024-02-13T00:56:43.077 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:41.807704+0000 osd.0 (osd.0) 401 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:56:43.078 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:41.808907+0000 osd.0 (osd.0) 402 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:56:43.078 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:42.019461+0000 osd.5 (osd.5) 70 : cluster [DBG] 6.d scrub starts 2024-02-13T00:56:43.078 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:42.022696+0000 osd.5 (osd.5) 71 : cluster [DBG] 6.d scrub ok 2024-02-13T00:56:43.078 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:42.108108+0000 osd.4 (osd.4) 89 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:56:43.078 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:42.111778+0000 osd.4 (osd.4) 90 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:56:43.078 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:42.389751+0000 osd.3 (osd.3) 231 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:43.078 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:42.389921+0000 osd.3 (osd.3) 232 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:43.078 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:42.545856+0000 osd.3 (osd.3) 233 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:56:43.078 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:42.549356+0000 osd.3 (osd.3) 234 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:56:43.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:41.807704+0000 osd.0 (osd.0) 401 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:56:43.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:41.808907+0000 osd.0 (osd.0) 402 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:56:43.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:42.019461+0000 osd.5 (osd.5) 70 : cluster [DBG] 6.d scrub starts 2024-02-13T00:56:43.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:42.022696+0000 osd.5 (osd.5) 71 : cluster [DBG] 6.d scrub ok 2024-02-13T00:56:43.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:42.108108+0000 osd.4 (osd.4) 89 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:56:43.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:42.111778+0000 osd.4 (osd.4) 90 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:56:43.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:42.389751+0000 osd.3 (osd.3) 231 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:43.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:42.389921+0000 osd.3 (osd.3) 232 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:43.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:42.545856+0000 osd.3 (osd.3) 233 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:56:43.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:42 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:42.549356+0000 osd.3 (osd.3) 234 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:56:43.097 INFO:teuthology.orchestra.run.smithi162.stdout:193273528557 2024-02-13T00:56:43.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-13T00:56:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:41.807704+0000 osd.0 (osd.0) 401 : cluster [DBG] 3.1 scrub starts 2024-02-13T00:56:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:41.808907+0000 osd.0 (osd.0) 402 : cluster [DBG] 3.1 scrub ok 2024-02-13T00:56:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:42.019461+0000 osd.5 (osd.5) 70 : cluster [DBG] 6.d scrub starts 2024-02-13T00:56:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:42.022696+0000 osd.5 (osd.5) 71 : cluster [DBG] 6.d scrub ok 2024-02-13T00:56:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:42.108108+0000 osd.4 (osd.4) 89 : cluster [DBG] 4.13 scrub starts 2024-02-13T00:56:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:42.111778+0000 osd.4 (osd.4) 90 : cluster [DBG] 4.13 scrub ok 2024-02-13T00:56:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:42.389751+0000 osd.3 (osd.3) 231 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:42.389921+0000 osd.3 (osd.3) 232 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:42.545856+0000 osd.3 (osd.3) 233 : cluster [DBG] 4.10 scrub starts 2024-02-13T00:56:43.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:42 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:42.549356+0000 osd.3 (osd.3) 234 : cluster [DBG] 4.10 scrub ok 2024-02-13T00:56:43.367 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:44.253 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:42.595244+0000 mgr.y (mgr.14544) 753 : cluster [DBG] pgmap v1347: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:44.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:42.843453+0000 mon.a (mon.0) 3875 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-02-13T00:56:44.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:42.922287+0000 osd.2 (osd.2) 203 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:56:44.254 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:42.924715+0000 osd.2 (osd.2) 204 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:56:44.254 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:42.595244+0000 mgr.y (mgr.14544) 753 : cluster [DBG] pgmap v1347: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:44.254 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:42.843453+0000 mon.a (mon.0) 3875 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-02-13T00:56:44.254 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:42.922287+0000 osd.2 (osd.2) 203 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:56:44.254 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:42.924715+0000 osd.2 (osd.2) 204 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:56:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:42.595244+0000 mgr.y (mgr.14544) 753 : cluster [DBG] pgmap v1347: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:56:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:42.843453+0000 mon.a (mon.0) 3875 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-02-13T00:56:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:42.922287+0000 osd.2 (osd.2) 203 : cluster [DBG] 4.1c scrub starts 2024-02-13T00:56:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:43 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:42.924715+0000 osd.2 (osd.2) 204 : cluster [DBG] 4.1c scrub ok 2024-02-13T00:56:44.943 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:43.773723+0000 osd.0 (osd.0) 403 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:56:44.943 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:43.777203+0000 osd.0 (osd.0) 404 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:56:44.943 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:43.887817+0000 osd.2 (osd.2) 205 : cluster [DBG] 5.b scrub starts 2024-02-13T00:56:44.943 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:43.888810+0000 osd.2 (osd.2) 206 : cluster [DBG] 5.b scrub ok 2024-02-13T00:56:44.943 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:44.029094+0000 osd.7 (osd.7) 136 : cluster [DBG] 28.d scrub starts 2024-02-13T00:56:44.943 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:44.030608+0000 osd.7 (osd.7) 137 : cluster [DBG] 28.d scrub ok 2024-02-13T00:56:44.943 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:44.166646+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.e scrub starts 2024-02-13T00:56:44.943 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:44.170171+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.e scrub ok 2024-02-13T00:56:44.944 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:43.773723+0000 osd.0 (osd.0) 403 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:56:44.944 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:43.777203+0000 osd.0 (osd.0) 404 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:56:44.944 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:43.887817+0000 osd.2 (osd.2) 205 : cluster [DBG] 5.b scrub starts 2024-02-13T00:56:44.944 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:43.888810+0000 osd.2 (osd.2) 206 : cluster [DBG] 5.b scrub ok 2024-02-13T00:56:44.944 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:44.029094+0000 osd.7 (osd.7) 136 : cluster [DBG] 28.d scrub starts 2024-02-13T00:56:44.944 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:44.030608+0000 osd.7 (osd.7) 137 : cluster [DBG] 28.d scrub ok 2024-02-13T00:56:44.944 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:44.166646+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.e scrub starts 2024-02-13T00:56:44.944 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:44 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:44.170171+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.e scrub ok 2024-02-13T00:56:45.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.195+0000 7f76ea080700 1 -- 172.21.15.162:0/3668071020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76e40fd490 msgr2=0x7f76e40f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:45.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.195+0000 7f76ea080700 1 --2- 172.21.15.162:0/3668071020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76e40fd490 0x7f76e40f6cc0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 rx=0x7f76d4004870 tx=0x7f76d4009cb0).stop 2024-02-13T00:56:45.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.195+0000 7f76ea080700 1 -- 172.21.15.162:0/3668071020 shutdown_connections 2024-02-13T00:56:45.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.195+0000 7f76ea080700 1 --2- 172.21.15.162:0/3668071020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76e40f73a0 0x7f76e41039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:45.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.195+0000 7f76ea080700 1 --2- 172.21.15.162:0/3668071020 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76e40fcb30 0x7f76e40fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:45.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.195+0000 7f76ea080700 1 --2- 172.21.15.162:0/3668071020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76e40fd490 0x7f76e40f6cc0 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:45.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.195+0000 7f76ea080700 1 -- 172.21.15.162:0/3668071020 >> 172.21.15.162:0/3668071020 conn(0x7f76e406eb90 msgr2=0x7f76e40f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:45.242 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.195+0000 7f76ea080700 1 -- 172.21.15.162:0/3668071020 shutdown_connections 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.195+0000 7f76ea080700 1 -- 172.21.15.162:0/3668071020 wait complete. 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.196+0000 7f76ea080700 1 librados: starting msgr at 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.196+0000 7f76ea080700 1 librados: starting objecter 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.196+0000 7f76ea080700 1 Processor -- start 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.196+0000 7f76ea080700 1 -- start start 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.196+0000 7f76ea080700 1 librados: setting wanted keys 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.196+0000 7f76ea080700 1 librados: calling monclient init 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76ea080700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76e40f73a0 0x7f76e4109130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76ea080700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76e40fcb30 0x7f76e4109620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76ea080700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76e40fd490 0x7f76e4109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76ea080700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e4114310 con 0x7f76e40fcb30 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76ea080700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e4114480 con 0x7f76e40fd490 2024-02-13T00:56:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76ea080700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e4114640 con 0x7f76e40f73a0 2024-02-13T00:56:45.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e37fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76e40f73a0 0x7f76e4109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:45.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76e40fcb30 0x7f76e4109620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:45.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e37fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76e40f73a0 0x7f76e4109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:36924/0 (socket says 172.21.15.162:36924) 2024-02-13T00:56:45.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76e40fcb30 0x7f76e4109620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:51322/0 (socket says 172.21.15.162:51322) 2024-02-13T00:56:45.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e37fe700 1 -- 172.21.15.162:0/1839366116 learned_addr learned my addr 172.21.15.162:0/1839366116 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:45.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e3fff700 1 --2- 172.21.15.162:0/1839366116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76e40fd490 0x7f76e4109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:45.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e2ffd700 1 -- 172.21.15.162:0/1839366116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76e40f73a0 msgr2=0x7f76e4109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:45.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e2ffd700 1 --2- 172.21.15.162:0/1839366116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76e40f73a0 0x7f76e4109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:45.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e2ffd700 1 -- 172.21.15.162:0/1839366116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76e40fd490 msgr2=0x7f76e4109c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:45.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e2ffd700 1 --2- 172.21.15.162:0/1839366116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76e40fd490 0x7f76e4109c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:45.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e2ffd700 1 -- 172.21.15.162:0/1839366116 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d4005040 con 0x7f76e40fcb30 2024-02-13T00:56:45.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e3fff700 1 --2- 172.21.15.162:0/1839366116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76e40fd490 0x7f76e4109c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e37fe700 1 --2- 172.21.15.162:0/1839366116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76e40f73a0 0x7f76e4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e2ffd700 1 --2- 172.21.15.162:0/1839366116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76e40fcb30 0x7f76e4109620 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7f76d40043d0 tx=0x7f76d4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e0ff9700 10 client.?.objecter ms_handle_connect 0x7f76e40fcb30 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e0ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e0ff9700 1 -- 172.21.15.162:0/1839366116 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d400e030 con 0x7f76e40fcb30 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76ea080700 1 -- 172.21.15.162:0/1839366116 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e41148c0 con 0x7f76e40fcb30 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e0ff9700 1 -- 172.21.15.162:0/1839366116 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76d4013d60 con 0x7f76e40fcb30 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76e0ff9700 1 -- 172.21.15.162:0/1839366116 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d4018070 con 0x7f76e40fcb30 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76ea080700 10 client.28410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76ea080700 1 -- 172.21.15.162:0/1839366116 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e4114d40 con 0x7f76e40fcb30 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.197+0000 7f76ea080700 1 librados: init done 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76ea080700 10 client.28410.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76ea080700 15 client.28410.objecter _session_command_op_assign -1 1 2024-02-13T00:56:45.245 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76ea080700 20 client.28410.objecter _get_session osd=-1 returning homeless 2024-02-13T00:56:45.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76ea080700 10 client.28410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:45.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76ea080700 1 -- 172.21.15.162:0/1839366116 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f76e40475e0 con 0x7f76e40fcb30 2024-02-13T00:56:45.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 1 -- 172.21.15.162:0/1839366116 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f76d401d420 con 0x7f76e40fcb30 2024-02-13T00:56:45.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 1 --2- 172.21.15.162:0/1839366116 >> v2:172.21.15.162:6800/2866534053 conn(0x7f76cc062c90 0x7f76cc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:45.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 1 -- 172.21.15.162:0/1839366116 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(854..854 src has 449..854) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f76d407e300 con 0x7f76e40fcb30 2024-02-13T00:56:45.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 10 client.28410.objecter ms_dispatch 0x7f76e4001550 osd_map(854..854 src has 449..854) v4 2024-02-13T00:56:45.246 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 3 client.28410.objecter handle_osd_map got epochs [854,854] > 0 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 3 client.28410.objecter handle_osd_map decoding full epoch 854 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e37fe700 1 --2- 172.21.15.162:0/1839366116 >> v2:172.21.15.162:6800/2866534053 conn(0x7f76cc062c90 0x7f76cc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 10 client.28410.objecter checking command 1 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 1 --2- 172.21.15.162:0/1839366116 >> v2:172.21.15.181:6812/1964192772 conn(0x7f76cc06e060 0x7f76cc070500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 20 client.28410.objecter _get_session s=0x7f76cc06d9b0 osd=7 3 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 20 client.28410.objecter put_session s=0x7f76cc06d9b0 osd=7 3 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 15 client.28410.objecter _session_command_op_remove -1 1 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 20 client.28410.objecter _get_session s=0x7f76cc06d9b0 osd=7 3 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 20 client.28410.objecter get_session s=0x7f76cc06d9b0 osd=7 3 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 15 client.28410.objecter _session_command_op_assign 7 1 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 20 client.28410.objecter put_session s=0x7f76cc06d9b0 osd=7 4 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 10 client.28410.objecter _send_command 1 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 1 -- 172.21.15.162:0/1839366116 --> v2:172.21.15.181:6812/1964192772 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f76cc070b40 con 0x7f76cc06e060 2024-02-13T00:56:45.247 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 20 client.28410.objecter dump_active .. 0 homeless 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.199+0000 7f76e0ff9700 1 -- 172.21.15.162:0/1839366116 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=854) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76d4080920 con 0x7f76e40fcb30 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.200+0000 7f76e0ff9700 10 client.28410.objecter ms_handle_connect 0x7f76cc062c90 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.200+0000 7f76e37fe700 1 --2- 172.21.15.162:0/1839366116 >> v2:172.21.15.162:6800/2866534053 conn(0x7f76cc062c90 0x7f76cc065150 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 rx=0x7f76d8009fd0 tx=0x7f76d800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.200+0000 7f76e3fff700 1 --2- 172.21.15.162:0/1839366116 >> v2:172.21.15.181:6812/1964192772 conn(0x7f76cc06e060 0x7f76cc070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.200+0000 7f76e3fff700 1 --2- 172.21.15.162:0/1839366116 >> v2:172.21.15.181:6812/1964192772 conn(0x7f76cc06e060 0x7f76cc070500 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.200+0000 7f76e0ff9700 10 client.28410.objecter ms_handle_connect 0x7f76cc06e060 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.201+0000 7f76e0ff9700 1 -- 172.21.15.162:0/1839366116 <== osd.7 v2:172.21.15.181:6812/1964192772 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f76cc070b40 con 0x7f76cc06e060 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.201+0000 7f76e0ff9700 10 client.28410.objecter ms_dispatch 0x7f76e4001550 command_reply(tid 1: 0 ) v1 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.201+0000 7f76e0ff9700 10 client.28410.objecter _finish_command 1 = system:0 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.201+0000 7f76e0ff9700 20 client.28410.objecter put_session s=0x7f76cc06d9b0 osd=7 4 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.201+0000 7f76e0ff9700 15 client.28410.objecter _session_command_op_remove 7 1 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 10 client.28410.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-13T00:56:45.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 15 client.28410.objecter _session_command_op_assign -1 2 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 20 client.28410.objecter _get_session s=0x7f76cc06d9b0 osd=7 3 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 20 client.28410.objecter put_session s=0x7f76cc06d9b0 osd=7 3 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 20 client.28410.objecter _get_session s=0x7f76cc06d9b0 osd=7 3 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 15 client.28410.objecter _session_command_op_remove -1 2 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 20 client.28410.objecter get_session s=0x7f76cc06d9b0 osd=7 3 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 15 client.28410.objecter _session_command_op_assign 7 2 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 20 client.28410.objecter put_session s=0x7f76cc06d9b0 osd=7 4 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 10 client.28410.objecter _send_command 2 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 1 -- 172.21.15.162:0/1839366116 --> v2:172.21.15.181:6812/1964192772 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f76e4106090 con 0x7f76cc06e060 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76e0ff9700 1 -- 172.21.15.162:0/1839366116 <== osd.7 v2:172.21.15.181:6812/1964192772 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f76e4106090 con 0x7f76cc06e060 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76e0ff9700 10 client.28410.objecter ms_dispatch 0x7f76e4001550 command_reply(tid 2: 0 ) v1 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76e0ff9700 10 client.28410.objecter _finish_command 2 = system:0 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76e0ff9700 20 client.28410.objecter put_session s=0x7f76cc06d9b0 osd=7 4 2024-02-13T00:56:45.249 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76e0ff9700 15 client.28410.objecter _session_command_op_remove 7 2 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 10 librados: watch_flush enter 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 10 librados: watch_flush exit 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 10 client.28410.objecter close_session for osd.7 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 1 -- 172.21.15.162:0/1839366116 >> v2:172.21.15.181:6812/1964192772 conn(0x7f76cc06e060 msgr2=0x7f76cc070500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 1 --2- 172.21.15.162:0/1839366116 >> v2:172.21.15.181:6812/1964192772 conn(0x7f76cc06e060 0x7f76cc070500 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 20 client.28410.objecter put_session s=0x7f76cc06d9b0 osd=7 1 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 20 client.28410.objecter shutdown clearing up homeless session... 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 10 client.28410.objecter successfully canceled tick 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 1 -- 172.21.15.162:0/1839366116 >> v2:172.21.15.162:6800/2866534053 conn(0x7f76cc062c90 msgr2=0x7f76cc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 1 --2- 172.21.15.162:0/1839366116 >> v2:172.21.15.162:6800/2866534053 conn(0x7f76cc062c90 0x7f76cc065150 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 rx=0x7f76d8009fd0 tx=0x7f76d800b040).stop 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 1 -- 172.21.15.162:0/1839366116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76e40fcb30 msgr2=0x7f76e4109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 1 --2- 172.21.15.162:0/1839366116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76e40fcb30 0x7f76e4109620 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7f76d40043d0 tx=0x7f76d4009cb0).stop 2024-02-13T00:56:45.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76e3fff700 1 -- 172.21.15.162:0/1839366116 reap_dead start 2024-02-13T00:56:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.215+0000 7f76ea080700 1 -- 172.21.15.162:0/1839366116 shutdown_connections 2024-02-13T00:56:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.216+0000 7f76ea080700 1 -- 172.21.15.162:0/1839366116 >> 172.21.15.162:0/1839366116 conn(0x7f76e406eb90 msgr2=0x7f76e4070000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.216+0000 7f76ea080700 1 -- 172.21.15.162:0/1839366116 shutdown_connections 2024-02-13T00:56:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.216+0000 7f76ea080700 1 -- 172.21.15.162:0/1839366116 wait complete. 2024-02-13T00:56:45.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:45.216+0000 7f76ea080700 1 librados: shutdown 2024-02-13T00:56:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:43.773723+0000 osd.0 (osd.0) 403 : cluster [DBG] 4.6 scrub starts 2024-02-13T00:56:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:43.777203+0000 osd.0 (osd.0) 404 : cluster [DBG] 4.6 scrub ok 2024-02-13T00:56:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:43.887817+0000 osd.2 (osd.2) 205 : cluster [DBG] 5.b scrub starts 2024-02-13T00:56:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:43.888810+0000 osd.2 (osd.2) 206 : cluster [DBG] 5.b scrub ok 2024-02-13T00:56:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:44.029094+0000 osd.7 (osd.7) 136 : cluster [DBG] 28.d scrub starts 2024-02-13T00:56:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:44.030608+0000 osd.7 (osd.7) 137 : cluster [DBG] 28.d scrub ok 2024-02-13T00:56:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:44.166646+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.e scrub starts 2024-02-13T00:56:45.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:44 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:44.170171+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.e scrub ok 2024-02-13T00:56:45.954 INFO:teuthology.orchestra.run.smithi162.stdout:227633266919 2024-02-13T00:56:45.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-13T00:56:46.114 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:44.595754+0000 mgr.y (mgr.14544) 754 : cluster [DBG] pgmap v1349: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2024-02-13T00:56:46.114 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:44.991606+0000 osd.7 (osd.7) 138 : cluster [DBG] 6.21 scrub starts 2024-02-13T00:56:46.114 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:44.994668+0000 osd.7 (osd.7) 139 : cluster [DBG] 6.21 scrub ok 2024-02-13T00:56:46.114 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:45 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:45.634396+0000 mon.a (mon.0) 3876 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-02-13T00:56:46.167 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:44.595754+0000 mgr.y (mgr.14544) 754 : cluster [DBG] pgmap v1349: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2024-02-13T00:56:46.168 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:44.991606+0000 osd.7 (osd.7) 138 : cluster [DBG] 6.21 scrub starts 2024-02-13T00:56:46.168 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:44.994668+0000 osd.7 (osd.7) 139 : cluster [DBG] 6.21 scrub ok 2024-02-13T00:56:46.168 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:45.634396+0000 mon.a (mon.0) 3876 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-02-13T00:56:46.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:44.595754+0000 mgr.y (mgr.14544) 754 : cluster [DBG] pgmap v1349: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2024-02-13T00:56:46.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:44.991606+0000 osd.7 (osd.7) 138 : cluster [DBG] 6.21 scrub starts 2024-02-13T00:56:46.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:44.994668+0000 osd.7 (osd.7) 139 : cluster [DBG] 6.21 scrub ok 2024-02-13T00:56:46.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:45 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:45.634396+0000 mon.a (mon.0) 3876 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-02-13T00:56:46.225 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:46.483 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:56:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:56:46] "GET /metrics HTTP/1.1" 200 185528 "" "Prometheus/2.33.4" 2024-02-13T00:56:46.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:56:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:56:47.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:45.966092+0000 osd.7 (osd.7) 140 : cluster [DBG] 28.17 scrub starts 2024-02-13T00:56:47.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:45.968341+0000 osd.7 (osd.7) 141 : cluster [DBG] 28.17 scrub ok 2024-02-13T00:56:47.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:46.111025+0000 mgr.y (mgr.14544) 755 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:47.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:46.152809+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:56:47.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:46.156462+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:56:47.162 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:45.966092+0000 osd.7 (osd.7) 140 : cluster [DBG] 28.17 scrub starts 2024-02-13T00:56:47.162 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:45.968341+0000 osd.7 (osd.7) 141 : cluster [DBG] 28.17 scrub ok 2024-02-13T00:56:47.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:46.111025+0000 mgr.y (mgr.14544) 755 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:47.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:46.152809+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:56:47.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:46 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:46.156462+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:56:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:45.966092+0000 osd.7 (osd.7) 140 : cluster [DBG] 28.17 scrub starts 2024-02-13T00:56:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:45.968341+0000 osd.7 (osd.7) 141 : cluster [DBG] 28.17 scrub ok 2024-02-13T00:56:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:46.111025+0000 mgr.y (mgr.14544) 755 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:46.152809+0000 osd.4 (osd.4) 93 : cluster [DBG] 4.1 scrub starts 2024-02-13T00:56:47.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:46.156462+0000 osd.4 (osd.4) 94 : cluster [DBG] 4.1 scrub ok 2024-02-13T00:56:48.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.049+0000 7f503937b700 1 -- 172.21.15.162:0/3164751235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50340fcb10 msgr2=0x7f50340fcef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:48.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.049+0000 7f503937b700 1 --2- 172.21.15.162:0/3164751235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50340fcb10 0x7f50340fcef0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 rx=0x7f502c009be0 tx=0x7f502c004d10).stop 2024-02-13T00:56:48.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.049+0000 7f503937b700 1 -- 172.21.15.162:0/3164751235 shutdown_connections 2024-02-13T00:56:48.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.049+0000 7f503937b700 1 --2- 172.21.15.162:0/3164751235 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50340f7380 0x7f5034103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:48.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.049+0000 7f503937b700 1 --2- 172.21.15.162:0/3164751235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50340fcb10 0x7f50340fcef0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:48.054 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.049+0000 7f503937b700 1 --2- 172.21.15.162:0/3164751235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50340fd470 0x7f50340f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:48.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.049+0000 7f503937b700 1 -- 172.21.15.162:0/3164751235 >> 172.21.15.162:0/3164751235 conn(0x7f50340f2720 msgr2=0x7f50340f4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:48.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.049+0000 7f503937b700 1 -- 172.21.15.162:0/3164751235 shutdown_connections 2024-02-13T00:56:48.055 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.049+0000 7f503937b700 1 -- 172.21.15.162:0/3164751235 wait complete. 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.050+0000 7f503937b700 1 librados: starting msgr at 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.050+0000 7f503937b700 1 librados: starting objecter 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.050+0000 7f503937b700 1 Processor -- start 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.050+0000 7f503937b700 1 -- start start 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.050+0000 7f503937b700 1 librados: setting wanted keys 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.050+0000 7f503937b700 1 librados: calling monclient init 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f503937b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50340f7380 0x7f503410d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f503937b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50340fcb10 0x7f503410d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f503937b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50340fd470 0x7f50341072d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f503937b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50341077c0 con 0x7f50340fcb10 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f503937b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5034107930 con 0x7f50340f7380 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f503937b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5034107af0 con 0x7f50340fd470 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f50337fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50340fd470 0x7f50341072d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:48.057 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f50327fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50340fcb10 0x7f503410d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f50337fe700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50340fd470 0x7f50341072d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:36940/0 (socket says 172.21.15.162:36940) 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f50337fe700 1 -- 172.21.15.162:0/23256833 learned_addr learned my addr 172.21.15.162:0/23256833 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f5032ffd700 1 --2- 172.21.15.162:0/23256833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50340f7380 0x7f503410d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f50327fc700 1 -- 172.21.15.162:0/23256833 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50340fd470 msgr2=0x7f50341072d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f50327fc700 1 --2- 172.21.15.162:0/23256833 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50340fd470 0x7f50341072d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f50327fc700 1 -- 172.21.15.162:0/23256833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50340f7380 msgr2=0x7f503410d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f50327fc700 1 --2- 172.21.15.162:0/23256833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50340f7380 0x7f503410d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f50327fc700 1 -- 172.21.15.162:0/23256833 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5024005040 con 0x7f50340fcb10 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f50337fe700 1 --2- 172.21.15.162:0/23256833 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50340fd470 0x7f50341072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f50327fc700 1 --2- 172.21.15.162:0/23256833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50340fcb10 0x7f503410d620 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7f501c002780 tx=0x7f501c009e40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f502bfff700 10 client.?.objecter ms_handle_connect 0x7f50340fcb10 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f502bfff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f502bfff700 1 -- 172.21.15.162:0/23256833 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c007b40 con 0x7f50340fcb10 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f503937b700 1 -- 172.21.15.162:0/23256833 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502c0097c0 con 0x7f50340fcb10 2024-02-13T00:56:48.058 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f502bfff700 1 -- 172.21.15.162:0/23256833 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f501c00f040 con 0x7f50340fcb10 2024-02-13T00:56:48.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.051+0000 7f502bfff700 1 -- 172.21.15.162:0/23256833 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c014850 con 0x7f50340fcb10 2024-02-13T00:56:48.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.052+0000 7f503937b700 10 client.28416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:48.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.052+0000 7f503937b700 1 -- 172.21.15.162:0/23256833 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5034108130 con 0x7f50340fcb10 2024-02-13T00:56:48.059 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.052+0000 7f503937b700 1 librados: init done 2024-02-13T00:56:48.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.052+0000 7f503937b700 1 -- 172.21.15.162:0/23256833 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50340607d0 con 0x7f50340fcb10 2024-02-13T00:56:48.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.053+0000 7f502bfff700 1 -- 172.21.15.162:0/23256833 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f501c007ca0 con 0x7f50340fcb10 2024-02-13T00:56:48.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.053+0000 7f502bfff700 1 --2- 172.21.15.162:0/23256833 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5020062c90 0x7f5020065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:48.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.053+0000 7f502bfff700 1 -- 172.21.15.162:0/23256833 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(855..855 src has 449..855) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f501c07cf10 con 0x7f50340fcb10 2024-02-13T00:56:48.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.053+0000 7f502bfff700 10 client.28416.objecter ms_dispatch 0x7f5034001550 osd_map(855..855 src has 449..855) v4 2024-02-13T00:56:48.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.053+0000 7f502bfff700 3 client.28416.objecter handle_osd_map got epochs [855,855] > 0 2024-02-13T00:56:48.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.053+0000 7f502bfff700 3 client.28416.objecter handle_osd_map decoding full epoch 855 2024-02-13T00:56:48.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.053+0000 7f5032ffd700 1 --2- 172.21.15.162:0/23256833 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5020062c90 0x7f5020065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:48.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.054+0000 7f502bfff700 20 client.28416.objecter dump_active .. 0 homeless 2024-02-13T00:56:48.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.054+0000 7f5032ffd700 1 --2- 172.21.15.162:0/23256833 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5020062c90 0x7f5020065150 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 rx=0x7f502c00b3b0 tx=0x7f502c004d10).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:48.060 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.054+0000 7f502bfff700 10 client.28416.objecter ms_handle_connect 0x7f5020062c90 2024-02-13T00:56:48.062 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.058+0000 7f502bfff700 1 -- 172.21.15.162:0/23256833 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f501c050640 con 0x7f50340fcb10 2024-02-13T00:56:48.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.220+0000 7f503937b700 1 -- 172.21.15.162:0/23256833 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f50340475e0 con 0x7f50340fcb10 2024-02-13T00:56:48.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.221+0000 7f502bfff700 1 -- 172.21.15.162:0/23256833 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f501c050230 con 0x7f50340fcb10 2024-02-13T00:56:48.226 INFO:teuthology.orchestra.run.smithi162.stdout:34359738635 2024-02-13T00:56:48.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.222+0000 7f503937b700 10 librados: watch_flush enter 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.222+0000 7f503937b700 10 librados: watch_flush exit 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.222+0000 7f503937b700 20 client.28416.objecter shutdown clearing up homeless session... 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.222+0000 7f503937b700 10 client.28416.objecter successfully canceled tick 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.222+0000 7f503937b700 1 -- 172.21.15.162:0/23256833 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5020062c90 msgr2=0x7f5020065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.222+0000 7f503937b700 1 --2- 172.21.15.162:0/23256833 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5020062c90 0x7f5020065150 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 rx=0x7f502c00b3b0 tx=0x7f502c004d10).stop 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.222+0000 7f503937b700 1 -- 172.21.15.162:0/23256833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50340fcb10 msgr2=0x7f503410d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.223+0000 7f503937b700 1 --2- 172.21.15.162:0/23256833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50340fcb10 0x7f503410d620 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7f501c002780 tx=0x7f501c009e40).stop 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.223+0000 7f503937b700 1 -- 172.21.15.162:0/23256833 shutdown_connections 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.223+0000 7f503937b700 1 --2- 172.21.15.162:0/23256833 >> v2:172.21.15.162:6800/2866534053 conn(0x7f5020062c90 0x7f5020065150 unknown :-1 s=CLOSED pgs=3090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.223+0000 7f503937b700 1 --2- 172.21.15.162:0/23256833 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50340fd470 0x7f50341072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.223+0000 7f503937b700 1 --2- 172.21.15.162:0/23256833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50340f7380 0x7f503410d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.223+0000 7f503937b700 1 --2- 172.21.15.162:0/23256833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50340fcb10 0x7f503410d620 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:48.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.223+0000 7f503937b700 1 -- 172.21.15.162:0/23256833 >> 172.21.15.162:0/23256833 conn(0x7f50340f2720 msgr2=0x7f50340fa900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:48.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.223+0000 7f503937b700 1 -- 172.21.15.162:0/23256833 shutdown_connections 2024-02-13T00:56:48.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.223+0000 7f503937b700 1 -- 172.21.15.162:0/23256833 wait complete. 2024-02-13T00:56:48.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:48.223+0000 7f503937b700 1 librados: shutdown 2024-02-13T00:56:48.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:46.596085+0000 mgr.y (mgr.14544) 756 : cluster [DBG] pgmap v1351: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:48.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:46.847669+0000 osd.2 (osd.2) 207 : cluster [DBG] 6.f scrub starts 2024-02-13T00:56:48.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:46.850985+0000 osd.2 (osd.2) 208 : cluster [DBG] 6.f scrub ok 2024-02-13T00:56:48.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:46.596085+0000 mgr.y (mgr.14544) 756 : cluster [DBG] pgmap v1351: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:48.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:46.847669+0000 osd.2 (osd.2) 207 : cluster [DBG] 6.f scrub starts 2024-02-13T00:56:48.247 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:47 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:46.850985+0000 osd.2 (osd.2) 208 : cluster [DBG] 6.f scrub ok 2024-02-13T00:56:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:46.596085+0000 mgr.y (mgr.14544) 756 : cluster [DBG] pgmap v1351: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:46.847669+0000 osd.2 (osd.2) 207 : cluster [DBG] 6.f scrub starts 2024-02-13T00:56:48.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:47 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:46.850985+0000 osd.2 (osd.2) 208 : cluster [DBG] 6.f scrub ok 2024-02-13T00:56:48.866 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:56:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:56:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:56:48.946 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738630 got 34359738635 for osd.0 2024-02-13T00:56:48.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-13T00:56:49.214 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:49.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:48 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:48.222900+0000 mon.a (mon.0) 3877 : audit [DBG] from='client.? 172.21.15.162:0/23256833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-13T00:56:49.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:48.222900+0000 mon.a (mon.0) 3877 : audit [DBG] from='client.? 172.21.15.162:0/23256833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-13T00:56:49.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:48 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:48.222900+0000 mon.a (mon.0) 3877 : audit [DBG] from='client.? 172.21.15.162:0/23256833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-13T00:56:50.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:48.067359+0000 osd.1 (osd.1) 415 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:56:50.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:48.070660+0000 osd.1 (osd.1) 416 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:56:50.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:48.596643+0000 mgr.y (mgr.14544) 757 : cluster [DBG] pgmap v1352: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:56:50.061 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:48.849025+0000 osd.2 (osd.2) 209 : cluster [DBG] 2.c scrub starts 2024-02-13T00:56:50.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:48.849910+0000 osd.2 (osd.2) 210 : cluster [DBG] 2.c scrub ok 2024-02-13T00:56:50.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:48.912250+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:56:50.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:48.913213+0000 osd.7 (osd.7) 143 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:56:50.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:49.187861+0000 osd.4 (osd.4) 95 : cluster [DBG] 6.e scrub starts 2024-02-13T00:56:50.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:49.191334+0000 osd.4 (osd.4) 96 : cluster [DBG] 6.e scrub ok 2024-02-13T00:56:50.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:48.067359+0000 osd.1 (osd.1) 415 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:56:50.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:48.070660+0000 osd.1 (osd.1) 416 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:56:50.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:48.596643+0000 mgr.y (mgr.14544) 757 : cluster [DBG] pgmap v1352: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:56:50.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:48.849025+0000 osd.2 (osd.2) 209 : cluster [DBG] 2.c scrub starts 2024-02-13T00:56:50.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:48.849910+0000 osd.2 (osd.2) 210 : cluster [DBG] 2.c scrub ok 2024-02-13T00:56:50.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:48.912250+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:56:50.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:48.913213+0000 osd.7 (osd.7) 143 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:56:50.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:49.187861+0000 osd.4 (osd.4) 95 : cluster [DBG] 6.e scrub starts 2024-02-13T00:56:50.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:49 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:49.191334+0000 osd.4 (osd.4) 96 : cluster [DBG] 6.e scrub ok 2024-02-13T00:56:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:48.067359+0000 osd.1 (osd.1) 415 : cluster [DBG] 6.1d scrub starts 2024-02-13T00:56:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:48.070660+0000 osd.1 (osd.1) 416 : cluster [DBG] 6.1d scrub ok 2024-02-13T00:56:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:48.596643+0000 mgr.y (mgr.14544) 757 : cluster [DBG] pgmap v1352: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:56:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:48.849025+0000 osd.2 (osd.2) 209 : cluster [DBG] 2.c scrub starts 2024-02-13T00:56:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:48.849910+0000 osd.2 (osd.2) 210 : cluster [DBG] 2.c scrub ok 2024-02-13T00:56:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:48.912250+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.8 scrub starts 2024-02-13T00:56:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:48.913213+0000 osd.7 (osd.7) 143 : cluster [DBG] 6.8 scrub ok 2024-02-13T00:56:50.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:49.187861+0000 osd.4 (osd.4) 95 : cluster [DBG] 6.e scrub starts 2024-02-13T00:56:50.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:49 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:49.191334+0000 osd.4 (osd.4) 96 : cluster [DBG] 6.e scrub ok 2024-02-13T00:56:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:49.838744+0000 osd.2 (osd.2) 211 : cluster [DBG] 6.28 scrub starts 2024-02-13T00:56:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:49.841866+0000 osd.2 (osd.2) 212 : cluster [DBG] 6.28 scrub ok 2024-02-13T00:56:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:50.203001+0000 osd.4 (osd.4) 97 : cluster [DBG] 28.1c deep-scrub starts 2024-02-13T00:56:51.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:50.206132+0000 osd.4 (osd.4) 98 : cluster [DBG] 28.1c deep-scrub ok 2024-02-13T00:56:51.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:49.838744+0000 osd.2 (osd.2) 211 : cluster [DBG] 6.28 scrub starts 2024-02-13T00:56:51.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:49.841866+0000 osd.2 (osd.2) 212 : cluster [DBG] 6.28 scrub ok 2024-02-13T00:56:51.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:50.203001+0000 osd.4 (osd.4) 97 : cluster [DBG] 28.1c deep-scrub starts 2024-02-13T00:56:51.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:50.206132+0000 osd.4 (osd.4) 98 : cluster [DBG] 28.1c deep-scrub ok 2024-02-13T00:56:51.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.107+0000 7f30726ed700 1 -- 172.21.15.162:0/651477886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f306c0f9b70 msgr2=0x7f306c0f9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:51.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.107+0000 7f30726ed700 1 --2- 172.21.15.162:0/651477886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f306c0f9b70 0x7f306c0f9f50 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 rx=0x7f3054004870 tx=0x7f3054009cb0).stop 2024-02-13T00:56:51.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.107+0000 7f30726ed700 1 -- 172.21.15.162:0/651477886 shutdown_connections 2024-02-13T00:56:51.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.107+0000 7f30726ed700 1 --2- 172.21.15.162:0/651477886 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f306c0f8500 0x7f306c104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:51.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.107+0000 7f30726ed700 1 --2- 172.21.15.162:0/651477886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f306c0f9b70 0x7f306c0f9f50 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:51.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.107+0000 7f30726ed700 1 --2- 172.21.15.162:0/651477886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f306c0fa440 0x7f306c0f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:51.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.107+0000 7f30726ed700 1 -- 172.21.15.162:0/651477886 >> 172.21.15.162:0/651477886 conn(0x7f306c0f38c0 msgr2=0x7f306c0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:51.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.107+0000 7f30726ed700 1 -- 172.21.15.162:0/651477886 shutdown_connections 2024-02-13T00:56:51.112 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.107+0000 7f30726ed700 1 -- 172.21.15.162:0/651477886 wait complete. 2024-02-13T00:56:51.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.107+0000 7f30726ed700 1 librados: starting msgr at 2024-02-13T00:56:51.114 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.107+0000 7f30726ed700 1 librados: starting objecter 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.107+0000 7f30726ed700 1 Processor -- start 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f30726ed700 1 -- start start 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f30726ed700 1 librados: setting wanted keys 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f30726ed700 1 librados: calling monclient init 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f30726ed700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f306c0f8500 0x7f306c10e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f30726ed700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f306c0f9b70 0x7f306c10e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f30726ed700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f306c0fa440 0x7f306c1083b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f30726ed700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c1088a0 con 0x7f306c0f9b70 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f30726ed700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c108a10 con 0x7f306c0f8500 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f30726ed700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306c108bd0 con 0x7f306c0fa440 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f306b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f306c0f9b70 0x7f306c10e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f3070c8a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f306c0fa440 0x7f306c1083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:51.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f306b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f306c0f9b70 0x7f306c10e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:51350/0 (socket says 172.21.15.162:51350) 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f3070c8a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f306c0fa440 0x7f306c1083b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:36966/0 (socket says 172.21.15.162:36966) 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f306b7fe700 1 -- 172.21.15.162:0/854517899 learned_addr learned my addr 172.21.15.162:0/854517899 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f306bfff700 1 --2- 172.21.15.162:0/854517899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f306c0f8500 0x7f306c10e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f306b7fe700 1 -- 172.21.15.162:0/854517899 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f306c0fa440 msgr2=0x7f306c1083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f306b7fe700 1 --2- 172.21.15.162:0/854517899 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f306c0fa440 0x7f306c1083b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f306b7fe700 1 -- 172.21.15.162:0/854517899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f306c0f8500 msgr2=0x7f306c10e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f306b7fe700 1 --2- 172.21.15.162:0/854517899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f306c0f8500 0x7f306c10e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f306b7fe700 1 -- 172.21.15.162:0/854517899 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30640097c0 con 0x7f306c0f9b70 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f306bfff700 1 --2- 172.21.15.162:0/854517899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f306c0f8500 0x7f306c10e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.108+0000 7f3070c8a700 1 --2- 172.21.15.162:0/854517899 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f306c0fa440 0x7f306c1083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.109+0000 7f306b7fe700 1 --2- 172.21.15.162:0/854517899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f306c0f9b70 0x7f306c10e700 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f305c0041d0 tx=0x7f305c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.109+0000 7f30697fa700 10 client.?.objecter ms_handle_connect 0x7f306c0f9b70 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.109+0000 7f30697fa700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:51.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.109+0000 7f30697fa700 1 -- 172.21.15.162:0/854517899 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c007b40 con 0x7f306c0f9b70 2024-02-13T00:56:51.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.109+0000 7f30726ed700 1 -- 172.21.15.162:0/854517899 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3054005040 con 0x7f306c0f9b70 2024-02-13T00:56:51.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.109+0000 7f30697fa700 1 -- 172.21.15.162:0/854517899 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f305c00f040 con 0x7f306c0f9b70 2024-02-13T00:56:51.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.109+0000 7f30697fa700 1 -- 172.21.15.162:0/854517899 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c014770 con 0x7f306c0f9b70 2024-02-13T00:56:51.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.109+0000 7f30726ed700 10 client.28422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:51.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.109+0000 7f30726ed700 1 -- 172.21.15.162:0/854517899 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306c115040 con 0x7f306c0f9b70 2024-02-13T00:56:51.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.109+0000 7f30726ed700 1 librados: init done 2024-02-13T00:56:51.117 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.110+0000 7f30726ed700 1 -- 172.21.15.162:0/854517899 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306c0475e0 con 0x7f306c0f9b70 2024-02-13T00:56:51.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.110+0000 7f30697fa700 1 -- 172.21.15.162:0/854517899 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f305c009110 con 0x7f306c0f9b70 2024-02-13T00:56:51.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.111+0000 7f30697fa700 1 --2- 172.21.15.162:0/854517899 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3058062c90 0x7f3058065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:51.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.111+0000 7f306bfff700 1 --2- 172.21.15.162:0/854517899 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3058062c90 0x7f3058065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:51.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.111+0000 7f30697fa700 1 -- 172.21.15.162:0/854517899 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(855..855 src has 449..855) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f305c07d050 con 0x7f306c0f9b70 2024-02-13T00:56:51.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.111+0000 7f30697fa700 10 client.28422.objecter ms_dispatch 0x7f306c001550 osd_map(855..855 src has 449..855) v4 2024-02-13T00:56:51.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.111+0000 7f30697fa700 3 client.28422.objecter handle_osd_map got epochs [855,855] > 0 2024-02-13T00:56:51.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.111+0000 7f30697fa700 3 client.28422.objecter handle_osd_map decoding full epoch 855 2024-02-13T00:56:51.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.111+0000 7f30697fa700 20 client.28422.objecter dump_active .. 0 homeless 2024-02-13T00:56:51.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.111+0000 7f30697fa700 10 client.28422.objecter ms_handle_connect 0x7f3058062c90 2024-02-13T00:56:51.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.111+0000 7f306bfff700 1 --2- 172.21.15.162:0/854517899 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3058062c90 0x7f3058065150 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 rx=0x7f30540043d0 tx=0x7f305400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:51.120 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.116+0000 7f30697fa700 1 -- 172.21.15.162:0/854517899 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f305c050780 con 0x7f306c0f9b70 2024-02-13T00:56:51.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.281+0000 7f30726ed700 1 -- 172.21.15.162:0/854517899 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f306c109890 con 0x7f306c0f9b70 2024-02-13T00:56:51.285 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.282+0000 7f30697fa700 1 -- 172.21.15.162:0/854517899 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f305c050370 con 0x7f306c0f9b70 2024-02-13T00:56:51.286 INFO:teuthology.orchestra.run.smithi162.stdout:55834575112 2024-02-13T00:56:51.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.283+0000 7f30726ed700 10 librados: watch_flush enter 2024-02-13T00:56:51.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.283+0000 7f30726ed700 10 librados: watch_flush exit 2024-02-13T00:56:51.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.283+0000 7f30726ed700 20 client.28422.objecter shutdown clearing up homeless session... 2024-02-13T00:56:51.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.283+0000 7f30726ed700 10 client.28422.objecter successfully canceled tick 2024-02-13T00:56:51.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.283+0000 7f30726ed700 1 -- 172.21.15.162:0/854517899 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3058062c90 msgr2=0x7f3058065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:51.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.283+0000 7f30726ed700 1 --2- 172.21.15.162:0/854517899 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3058062c90 0x7f3058065150 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 rx=0x7f30540043d0 tx=0x7f305400b040).stop 2024-02-13T00:56:51.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.283+0000 7f30726ed700 1 -- 172.21.15.162:0/854517899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f306c0f9b70 msgr2=0x7f306c10e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:51.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.283+0000 7f30726ed700 1 --2- 172.21.15.162:0/854517899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f306c0f9b70 0x7f306c10e700 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f305c0041d0 tx=0x7f305c009f60).stop 2024-02-13T00:56:51.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.284+0000 7f30726ed700 1 -- 172.21.15.162:0/854517899 shutdown_connections 2024-02-13T00:56:51.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.284+0000 7f30726ed700 1 --2- 172.21.15.162:0/854517899 >> v2:172.21.15.162:6800/2866534053 conn(0x7f3058062c90 0x7f3058065150 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:51.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.284+0000 7f30726ed700 1 --2- 172.21.15.162:0/854517899 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f306c0fa440 0x7f306c1083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:51.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.284+0000 7f30726ed700 1 --2- 172.21.15.162:0/854517899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f306c0f8500 0x7f306c10e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:51.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.284+0000 7f30726ed700 1 --2- 172.21.15.162:0/854517899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f306c0f9b70 0x7f306c10e700 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:51.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.284+0000 7f30726ed700 1 -- 172.21.15.162:0/854517899 >> 172.21.15.162:0/854517899 conn(0x7f306c0f38c0 msgr2=0x7f306c0fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:51.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.284+0000 7f30726ed700 1 -- 172.21.15.162:0/854517899 shutdown_connections 2024-02-13T00:56:51.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.284+0000 7f30726ed700 1 -- 172.21.15.162:0/854517899 wait complete. 2024-02-13T00:56:51.289 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:51.284+0000 7f30726ed700 1 librados: shutdown 2024-02-13T00:56:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:49.838744+0000 osd.2 (osd.2) 211 : cluster [DBG] 6.28 scrub starts 2024-02-13T00:56:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:49.841866+0000 osd.2 (osd.2) 212 : cluster [DBG] 6.28 scrub ok 2024-02-13T00:56:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:50.203001+0000 osd.4 (osd.4) 97 : cluster [DBG] 28.1c deep-scrub starts 2024-02-13T00:56:51.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:50 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:50.206132+0000 osd.4 (osd.4) 98 : cluster [DBG] 28.1c deep-scrub ok 2024-02-13T00:56:51.976 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:50.597506+0000 mgr.y (mgr.14544) 758 : cluster [DBG] pgmap v1353: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:51.976 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:50.690545+0000 osd.3 (osd.3) 235 : cluster [DBG] 5.13 scrub starts 2024-02-13T00:56:51.977 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:50.693433+0000 osd.3 (osd.3) 236 : cluster [DBG] 5.13 scrub ok 2024-02-13T00:56:51.977 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:50.885240+0000 osd.0 (osd.0) 405 : cluster [DBG] 4.3 scrub starts 2024-02-13T00:56:51.977 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:50.889046+0000 osd.0 (osd.0) 406 : cluster [DBG] 4.3 scrub ok 2024-02-13T00:56:51.977 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:51.080898+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:56:51.977 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:51.083792+0000 osd.6 (osd.6) 353 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:56:51.977 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:51.229192+0000 osd.4 (osd.4) 99 : cluster [DBG] 28.10 scrub starts 2024-02-13T00:56:51.977 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:51.232090+0000 osd.4 (osd.4) 100 : cluster [DBG] 28.10 scrub ok 2024-02-13T00:56:51.977 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:51.282745+0000 mon.a (mon.0) 3878 : audit [DBG] from='client.? 172.21.15.162:0/854517899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-13T00:56:51.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:50.597506+0000 mgr.y (mgr.14544) 758 : cluster [DBG] pgmap v1353: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:51.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:50.690545+0000 osd.3 (osd.3) 235 : cluster [DBG] 5.13 scrub starts 2024-02-13T00:56:51.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:50.693433+0000 osd.3 (osd.3) 236 : cluster [DBG] 5.13 scrub ok 2024-02-13T00:56:51.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:50.885240+0000 osd.0 (osd.0) 405 : cluster [DBG] 4.3 scrub starts 2024-02-13T00:56:51.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:50.889046+0000 osd.0 (osd.0) 406 : cluster [DBG] 4.3 scrub ok 2024-02-13T00:56:51.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:51.080898+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:56:51.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:51.083792+0000 osd.6 (osd.6) 353 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:56:51.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:51.229192+0000 osd.4 (osd.4) 99 : cluster [DBG] 28.10 scrub starts 2024-02-13T00:56:51.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:51.232090+0000 osd.4 (osd.4) 100 : cluster [DBG] 28.10 scrub ok 2024-02-13T00:56:51.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:51.282745+0000 mon.a (mon.0) 3878 : audit [DBG] from='client.? 172.21.15.162:0/854517899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-13T00:56:51.996 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575107 got 55834575112 for osd.1 2024-02-13T00:56:51.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-13T00:56:52.265 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:50.597506+0000 mgr.y (mgr.14544) 758 : cluster [DBG] pgmap v1353: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:50.690545+0000 osd.3 (osd.3) 235 : cluster [DBG] 5.13 scrub starts 2024-02-13T00:56:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:50.693433+0000 osd.3 (osd.3) 236 : cluster [DBG] 5.13 scrub ok 2024-02-13T00:56:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:50.885240+0000 osd.0 (osd.0) 405 : cluster [DBG] 4.3 scrub starts 2024-02-13T00:56:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:50.889046+0000 osd.0 (osd.0) 406 : cluster [DBG] 4.3 scrub ok 2024-02-13T00:56:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:51.080898+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.11 scrub starts 2024-02-13T00:56:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:51.083792+0000 osd.6 (osd.6) 353 : cluster [DBG] 2.11 scrub ok 2024-02-13T00:56:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:51.229192+0000 osd.4 (osd.4) 99 : cluster [DBG] 28.10 scrub starts 2024-02-13T00:56:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:51.232090+0000 osd.4 (osd.4) 100 : cluster [DBG] 28.10 scrub ok 2024-02-13T00:56:52.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:51 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:51.282745+0000 mon.a (mon.0) 3878 : audit [DBG] from='client.? 172.21.15.162:0/854517899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-13T00:56:53.245 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:51.745357+0000 osd.2 (osd.2) 213 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:53.245 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:51.745664+0000 osd.2 (osd.2) 214 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:53.245 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:51.842356+0000 osd.7 (osd.7) 144 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:56:53.245 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:51.843382+0000 osd.7 (osd.7) 145 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:56:53.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:51.877112+0000 mon.a (mon.0) 3879 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-02-13T00:56:53.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:51.745357+0000 osd.2 (osd.2) 213 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:53.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:51.745664+0000 osd.2 (osd.2) 214 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:53.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:51.842356+0000 osd.7 (osd.7) 144 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:56:53.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:51.843382+0000 osd.7 (osd.7) 145 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:56:53.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:52 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:51.877112+0000 mon.a (mon.0) 3879 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-02-13T00:56:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:51.745357+0000 osd.2 (osd.2) 213 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:56:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:51.745664+0000 osd.2 (osd.2) 214 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:56:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:51.842356+0000 osd.7 (osd.7) 144 : cluster [DBG] 1.0 scrub starts 2024-02-13T00:56:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:51.843382+0000 osd.7 (osd.7) 145 : cluster [DBG] 1.0 scrub ok 2024-02-13T00:56:53.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:52 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:51.877112+0000 mon.a (mon.0) 3879 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-02-13T00:56:53.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:52.597927+0000 mgr.y (mgr.14544) 759 : cluster [DBG] pgmap v1355: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:53.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:52.697612+0000 osd.3 (osd.3) 237 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:56:53.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:52.701203+0000 osd.3 (osd.3) 238 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:56:53.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:52.823169+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:56:53.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:52.824138+0000 osd.7 (osd.7) 147 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:56:53.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:53.223305+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.18 scrub starts 2024-02-13T00:56:53.909 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:53.226976+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.18 scrub ok 2024-02-13T00:56:53.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:52.597927+0000 mgr.y (mgr.14544) 759 : cluster [DBG] pgmap v1355: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:53.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:52.697612+0000 osd.3 (osd.3) 237 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:56:53.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:52.701203+0000 osd.3 (osd.3) 238 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:56:53.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:52.823169+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:56:53.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:52.824138+0000 osd.7 (osd.7) 147 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:56:53.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:53.223305+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.18 scrub starts 2024-02-13T00:56:53.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:53 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:53.226976+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.18 scrub ok 2024-02-13T00:56:54.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.156+0000 7f8647012700 1 -- 172.21.15.162:0/1055098489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86400f73a0 msgr2=0x7f86401039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:54.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.156+0000 7f8647012700 1 --2- 172.21.15.162:0/1055098489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86400f73a0 0x7f86401039a0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 rx=0x7f8638009c30 tx=0x7f8638004c50).stop 2024-02-13T00:56:54.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.156+0000 7f8647012700 1 -- 172.21.15.162:0/1055098489 shutdown_connections 2024-02-13T00:56:54.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.156+0000 7f8647012700 1 --2- 172.21.15.162:0/1055098489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86400f73a0 0x7f86401039a0 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:54.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.156+0000 7f8647012700 1 --2- 172.21.15.162:0/1055098489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86400fcb30 0x7f86400fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:54.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.156+0000 7f8647012700 1 --2- 172.21.15.162:0/1055098489 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86400fd490 0x7f86400f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:54.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.156+0000 7f8647012700 1 -- 172.21.15.162:0/1055098489 >> 172.21.15.162:0/1055098489 conn(0x7f864006eb90 msgr2=0x7f86400f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:54.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.156+0000 7f8647012700 1 -- 172.21.15.162:0/1055098489 shutdown_connections 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.156+0000 7f8647012700 1 -- 172.21.15.162:0/1055098489 wait complete. 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8647012700 1 librados: starting msgr at 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8647012700 1 librados: starting objecter 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8647012700 1 Processor -- start 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8647012700 1 -- start start 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8647012700 1 librados: setting wanted keys 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8647012700 1 librados: calling monclient init 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8647012700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86400f73a0 0x7f864010ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8647012700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86400fcb30 0x7f864010d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8647012700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86400fd490 0x7f8640107010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8647012700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8640107500 con 0x7f86400fcb30 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8647012700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8640107670 con 0x7f86400fd490 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8647012700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8640107830 con 0x7f86400f73a0 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8644dae700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86400f73a0 0x7f864010ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f863ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86400fcb30 0x7f864010d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8644dae700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86400f73a0 0x7f864010ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:36998/0 (socket says 172.21.15.162:36998) 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f8644dae700 1 -- 172.21.15.162:0/466276849 learned_addr learned my addr 172.21.15.162:0/466276849 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.157+0000 7f863ffff700 1 --2- 172.21.15.162:0/466276849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86400fcb30 0x7f864010d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:51372/0 (socket says 172.21.15.162:51372) 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f8644dae700 1 -- 172.21.15.162:0/466276849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86400fd490 msgr2=0x7f8640107010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f8644dae700 1 --2- 172.21.15.162:0/466276849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86400fd490 0x7f8640107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f8644dae700 1 -- 172.21.15.162:0/466276849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86400fcb30 msgr2=0x7f864010d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f8644dae700 1 --2- 172.21.15.162:0/466276849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86400fcb30 0x7f864010d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f8644dae700 1 -- 172.21.15.162:0/466276849 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c005040 con 0x7f86400f73a0 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f863ffff700 1 --2- 172.21.15.162:0/466276849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86400fcb30 0x7f864010d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f8644dae700 1 --2- 172.21.15.162:0/466276849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86400f73a0 0x7f864010ce70 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 rx=0x7f86340041d0 tx=0x7f8634009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f863dffb700 10 client.?.objecter ms_handle_connect 0x7f86400f73a0 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f863dffb700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f863dffb700 1 -- 172.21.15.162:0/466276849 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8634007b40 con 0x7f86400f73a0 2024-02-13T00:56:54.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f863dffb700 1 -- 172.21.15.162:0/466276849 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f863400f040 con 0x7f86400f73a0 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f8647012700 1 -- 172.21.15.162:0/466276849 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86380097c0 con 0x7f86400f73a0 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f863dffb700 1 -- 172.21.15.162:0/466276849 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8634014810 con 0x7f86400f73a0 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f8647012700 10 client.38522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f8647012700 1 -- 172.21.15.162:0/466276849 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8640107e70 con 0x7f86400f73a0 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.158+0000 7f8647012700 1 librados: init done 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.159+0000 7f8647012700 1 -- 172.21.15.162:0/466276849 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8640106090 con 0x7f86400f73a0 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.159+0000 7f863dffb700 1 -- 172.21.15.162:0/466276849 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f8634009110 con 0x7f86400f73a0 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.160+0000 7f863dffb700 1 --2- 172.21.15.162:0/466276849 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8630062c90 0x7f8630065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.160+0000 7f863dffb700 1 -- 172.21.15.162:0/466276849 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(856..856 src has 449..856) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f863407d040 con 0x7f86400f73a0 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.160+0000 7f863dffb700 10 client.38522.objecter ms_dispatch 0x7f8640001550 osd_map(856..856 src has 449..856) v4 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.160+0000 7f863dffb700 3 client.38522.objecter handle_osd_map got epochs [856,856] > 0 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.160+0000 7f863dffb700 3 client.38522.objecter handle_osd_map decoding full epoch 856 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.160+0000 7f863ffff700 1 --2- 172.21.15.162:0/466276849 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8630062c90 0x7f8630065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.160+0000 7f863dffb700 20 client.38522.objecter dump_active .. 0 homeless 2024-02-13T00:56:54.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.160+0000 7f863dffb700 10 client.38522.objecter ms_handle_connect 0x7f8630062c90 2024-02-13T00:56:54.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.160+0000 7f863ffff700 1 --2- 172.21.15.162:0/466276849 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8630062c90 0x7f8630065150 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 rx=0x7f86401087b0 tx=0x7f862c00b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:54.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.164+0000 7f863dffb700 1 -- 172.21.15.162:0/466276849 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8634050770 con 0x7f86400f73a0 2024-02-13T00:56:54.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:52.597927+0000 mgr.y (mgr.14544) 759 : cluster [DBG] pgmap v1355: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:52.697612+0000 osd.3 (osd.3) 237 : cluster [DBG] 4.1d scrub starts 2024-02-13T00:56:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:52.701203+0000 osd.3 (osd.3) 238 : cluster [DBG] 4.1d scrub ok 2024-02-13T00:56:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:52.823169+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.8 scrub starts 2024-02-13T00:56:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:52.824138+0000 osd.7 (osd.7) 147 : cluster [DBG] 2.8 scrub ok 2024-02-13T00:56:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:53.223305+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.18 scrub starts 2024-02-13T00:56:54.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:53 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:53.226976+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.18 scrub ok 2024-02-13T00:56:54.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.329+0000 7f8647012700 1 -- 172.21.15.162:0/466276849 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f864005fc10 con 0x7f86400f73a0 2024-02-13T00:56:54.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.329+0000 7f863dffb700 1 -- 172.21.15.162:0/466276849 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8634050360 con 0x7f86400f73a0 2024-02-13T00:56:54.333 INFO:teuthology.orchestra.run.smithi162.stdout:77309411587 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.331+0000 7f8647012700 10 librados: watch_flush enter 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 10 librados: watch_flush exit 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 20 client.38522.objecter shutdown clearing up homeless session... 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 10 client.38522.objecter successfully canceled tick 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 -- 172.21.15.162:0/466276849 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8630062c90 msgr2=0x7f8630065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 --2- 172.21.15.162:0/466276849 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8630062c90 0x7f8630065150 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 rx=0x7f86401087b0 tx=0x7f862c00b040).stop 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 -- 172.21.15.162:0/466276849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86400f73a0 msgr2=0x7f864010ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 --2- 172.21.15.162:0/466276849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86400f73a0 0x7f864010ce70 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 rx=0x7f86340041d0 tx=0x7f8634009f60).stop 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 -- 172.21.15.162:0/466276849 shutdown_connections 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 --2- 172.21.15.162:0/466276849 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8630062c90 0x7f8630065150 unknown :-1 s=CLOSED pgs=3092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 --2- 172.21.15.162:0/466276849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86400fd490 0x7f8640107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 --2- 172.21.15.162:0/466276849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86400f73a0 0x7f864010ce70 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:54.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 --2- 172.21.15.162:0/466276849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86400fcb30 0x7f864010d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:54.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 -- 172.21.15.162:0/466276849 >> 172.21.15.162:0/466276849 conn(0x7f864006eb90 msgr2=0x7f864006fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:54.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 -- 172.21.15.162:0/466276849 shutdown_connections 2024-02-13T00:56:54.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 -- 172.21.15.162:0/466276849 wait complete. 2024-02-13T00:56:54.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:54.332+0000 7f8647012700 1 librados: shutdown 2024-02-13T00:56:55.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:54.330640+0000 mon.c (mon.2) 542 : audit [DBG] from='client.? 172.21.15.162:0/466276849' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-13T00:56:55.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:54 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:54.330640+0000 mon.c (mon.2) 542 : audit [DBG] from='client.? 172.21.15.162:0/466276849' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-13T00:56:55.196 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411583 got 77309411587 for osd.2 2024-02-13T00:56:55.196 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-13T00:56:55.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:54.330640+0000 mon.c (mon.2) 542 : audit [DBG] from='client.? 172.21.15.162:0/466276849' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-13T00:56:55.468 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:55.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:54.090312+0000 osd.1 (osd.1) 417 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:56:55.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:54.093395+0000 osd.1 (osd.1) 418 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:56:55.984 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:54.598355+0000 mgr.y (mgr.14544) 760 : cluster [DBG] pgmap v1356: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2024-02-13T00:56:55.985 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:54.680614+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:56:55.985 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:54.683926+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:56:55.985 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:55.200852+0000 osd.4 (osd.4) 103 : cluster [DBG] 4.1b scrub starts 2024-02-13T00:56:55.985 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:55.204412+0000 osd.4 (osd.4) 104 : cluster [DBG] 4.1b scrub ok 2024-02-13T00:56:55.985 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:55.635030+0000 mon.a (mon.0) 3880 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-02-13T00:56:55.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:54.090312+0000 osd.1 (osd.1) 417 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:56:55.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:54.093395+0000 osd.1 (osd.1) 418 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:56:55.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:54.598355+0000 mgr.y (mgr.14544) 760 : cluster [DBG] pgmap v1356: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2024-02-13T00:56:55.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:54.680614+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:56:55.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:54.683926+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:56:55.986 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:55.200852+0000 osd.4 (osd.4) 103 : cluster [DBG] 4.1b scrub starts 2024-02-13T00:56:55.986 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:55.204412+0000 osd.4 (osd.4) 104 : cluster [DBG] 4.1b scrub ok 2024-02-13T00:56:55.986 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:55 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:55.635030+0000 mon.a (mon.0) 3880 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-02-13T00:56:56.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:56:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:56:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:54.090312+0000 osd.1 (osd.1) 417 : cluster [DBG] 2.4 scrub starts 2024-02-13T00:56:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:54.093395+0000 osd.1 (osd.1) 418 : cluster [DBG] 2.4 scrub ok 2024-02-13T00:56:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:54.598355+0000 mgr.y (mgr.14544) 760 : cluster [DBG] pgmap v1356: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2024-02-13T00:56:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:54.680614+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.1b scrub starts 2024-02-13T00:56:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:54.683926+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.1b scrub ok 2024-02-13T00:56:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:55.200852+0000 osd.4 (osd.4) 103 : cluster [DBG] 4.1b scrub starts 2024-02-13T00:56:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:55.204412+0000 osd.4 (osd.4) 104 : cluster [DBG] 4.1b scrub ok 2024-02-13T00:56:56.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:55.635030+0000 mon.a (mon.0) 3880 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-02-13T00:56:56.330 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:56:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:56:56] "GET /metrics HTTP/1.1" 200 185535 "" "Prometheus/2.33.4" 2024-02-13T00:56:57.100 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:55.840929+0000 osd.5 (osd.5) 72 : cluster [DBG] 28.a deep-scrub starts 2024-02-13T00:56:57.100 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:55.843968+0000 osd.5 (osd.5) 73 : cluster [DBG] 28.a deep-scrub ok 2024-02-13T00:56:57.100 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:55.862999+0000 osd.7 (osd.7) 148 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:56:57.100 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:55.863918+0000 osd.7 (osd.7) 149 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:56:57.100 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:56.116749+0000 mgr.y (mgr.14544) 761 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:57.101 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:55.840929+0000 osd.5 (osd.5) 72 : cluster [DBG] 28.a deep-scrub starts 2024-02-13T00:56:57.101 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:55.843968+0000 osd.5 (osd.5) 73 : cluster [DBG] 28.a deep-scrub ok 2024-02-13T00:56:57.101 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:55.862999+0000 osd.7 (osd.7) 148 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:56:57.101 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:55.863918+0000 osd.7 (osd.7) 149 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:56:57.101 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:56 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:56.116749+0000 mgr.y (mgr.14544) 761 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:55.840929+0000 osd.5 (osd.5) 72 : cluster [DBG] 28.a deep-scrub starts 2024-02-13T00:56:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:55.843968+0000 osd.5 (osd.5) 73 : cluster [DBG] 28.a deep-scrub ok 2024-02-13T00:56:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:55.862999+0000 osd.7 (osd.7) 148 : cluster [DBG] 2.0 scrub starts 2024-02-13T00:56:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:55.863918+0000 osd.7 (osd.7) 149 : cluster [DBG] 2.0 scrub ok 2024-02-13T00:56:57.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:56.116749+0000 mgr.y (mgr.14544) 761 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:56:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.327+0000 7fdf50903700 1 -- 172.21.15.162:0/2992772390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf4c0f6950 msgr2=0x7fdf4c104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.327+0000 7fdf50903700 1 --2- 172.21.15.162:0/2992772390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf4c0f6950 0x7fdf4c104af0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 rx=0x7fdf44009be0 tx=0x7fdf44004d10).stop 2024-02-13T00:56:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.328+0000 7fdf50903700 1 -- 172.21.15.162:0/2992772390 shutdown_connections 2024-02-13T00:56:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.328+0000 7fdf50903700 1 --2- 172.21.15.162:0/2992772390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf4c0f6950 0x7fdf4c104af0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.328+0000 7fdf50903700 1 --2- 172.21.15.162:0/2992772390 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf4c0fdce0 0x7fdf4c0fe0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.328+0000 7fdf50903700 1 --2- 172.21.15.162:0/2992772390 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf4c0fe640 0x7fdf4c0f6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.328+0000 7fdf50903700 1 -- 172.21.15.162:0/2992772390 >> 172.21.15.162:0/2992772390 conn(0x7fdf4c070060 msgr2=0x7fdf4c0f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.328+0000 7fdf50903700 1 -- 172.21.15.162:0/2992772390 shutdown_connections 2024-02-13T00:56:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.328+0000 7fdf50903700 1 -- 172.21.15.162:0/2992772390 wait complete. 2024-02-13T00:56:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.328+0000 7fdf50903700 1 librados: starting msgr at 2024-02-13T00:56:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.328+0000 7fdf50903700 1 librados: starting objecter 2024-02-13T00:56:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf50903700 1 Processor -- start 2024-02-13T00:56:57.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf50903700 1 -- start start 2024-02-13T00:56:57.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf50903700 1 librados: setting wanted keys 2024-02-13T00:56:57.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf50903700 1 librados: calling monclient init 2024-02-13T00:56:57.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf50903700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf4c0f6950 0x7fdf4c10e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:57.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf50903700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf4c0fdce0 0x7fdf4c10e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:57.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf50903700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf4c0fe640 0x7fdf4c1081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:57.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf50903700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c1086b0 con 0x7fdf4c0fe640 2024-02-13T00:56:57.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf50903700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c108820 con 0x7fdf4c0f6950 2024-02-13T00:56:57.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf50903700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c1089e0 con 0x7fdf4c0fdce0 2024-02-13T00:56:57.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf49d9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf4c0fdce0 0x7fdf4c10e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:57.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf4ad9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf4c0fe640 0x7fdf4c1081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:57.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf49d9b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf4c0fdce0 0x7fdf4c10e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:55854/0 (socket says 172.21.15.162:55854) 2024-02-13T00:56:57.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf49d9b700 1 -- 172.21.15.162:0/2056230756 learned_addr learned my addr 172.21.15.162:0/2056230756 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:56:57.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf4ad9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf4c0fe640 0x7fdf4c1081c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34140/0 (socket says 172.21.15.162:34140) 2024-02-13T00:56:57.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf4a59c700 1 --2- 172.21.15.162:0/2056230756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf4c0f6950 0x7fdf4c10e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:57.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf4ad9d700 1 -- 172.21.15.162:0/2056230756 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf4c0fdce0 msgr2=0x7fdf4c10e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:57.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf4ad9d700 1 --2- 172.21.15.162:0/2056230756 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf4c0fdce0 0x7fdf4c10e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:57.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf4ad9d700 1 -- 172.21.15.162:0/2056230756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf4c0f6950 msgr2=0x7fdf4c10e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:57.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf4ad9d700 1 --2- 172.21.15.162:0/2056230756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf4c0f6950 0x7fdf4c10e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:57.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf4ad9d700 1 -- 172.21.15.162:0/2056230756 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf34005040 con 0x7fdf4c0fe640 2024-02-13T00:56:57.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf4a59c700 1 --2- 172.21.15.162:0/2056230756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf4c0f6950 0x7fdf4c10e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:56:57.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf49d9b700 1 --2- 172.21.15.162:0/2056230756 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf4c0fdce0 0x7fdf4c10e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:56:57.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf4ad9d700 1 --2- 172.21.15.162:0/2056230756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf4c0fe640 0x7fdf4c1081c0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 rx=0x7fdf4400b6d0 tx=0x7fdf44004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:57.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf437fe700 10 client.?.objecter ms_handle_connect 0x7fdf4c0fe640 2024-02-13T00:56:57.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.329+0000 7fdf437fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:56:57.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.330+0000 7fdf437fe700 1 -- 172.21.15.162:0/2056230756 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4400e040 con 0x7fdf4c0fe640 2024-02-13T00:56:57.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.330+0000 7fdf437fe700 1 -- 172.21.15.162:0/2056230756 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf44018040 con 0x7fdf4c0fe640 2024-02-13T00:56:57.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.330+0000 7fdf50903700 1 -- 172.21.15.162:0/2056230756 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf440097c0 con 0x7fdf4c0fe640 2024-02-13T00:56:57.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.330+0000 7fdf437fe700 1 -- 172.21.15.162:0/2056230756 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4401c3f0 con 0x7fdf4c0fe640 2024-02-13T00:56:57.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.330+0000 7fdf50903700 10 client.28434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:56:57.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.330+0000 7fdf50903700 1 -- 172.21.15.162:0/2056230756 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4c108e70 con 0x7fdf4c0fe640 2024-02-13T00:56:57.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.330+0000 7fdf50903700 1 librados: init done 2024-02-13T00:56:57.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.331+0000 7fdf50903700 1 -- 172.21.15.162:0/2056230756 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4c05f960 con 0x7fdf4c0fe640 2024-02-13T00:56:57.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.331+0000 7fdf437fe700 1 -- 172.21.15.162:0/2056230756 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fdf44013420 con 0x7fdf4c0fe640 2024-02-13T00:56:57.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.331+0000 7fdf437fe700 1 --2- 172.21.15.162:0/2056230756 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf38062c90 0x7fdf38065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:56:57.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.332+0000 7fdf4a59c700 1 --2- 172.21.15.162:0/2056230756 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf38062c90 0x7fdf38065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:56:57.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.332+0000 7fdf437fe700 1 -- 172.21.15.162:0/2056230756 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(857..857 src has 449..857) v4 ==== 7509+0+0 (secure 0 0 0) 0x7fdf4407db70 con 0x7fdf4c0fe640 2024-02-13T00:56:57.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.332+0000 7fdf437fe700 10 client.28434.objecter ms_dispatch 0x7fdf4c001550 osd_map(857..857 src has 449..857) v4 2024-02-13T00:56:57.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.332+0000 7fdf437fe700 3 client.28434.objecter handle_osd_map got epochs [857,857] > 0 2024-02-13T00:56:57.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.332+0000 7fdf437fe700 3 client.28434.objecter handle_osd_map decoding full epoch 857 2024-02-13T00:56:57.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.332+0000 7fdf437fe700 20 client.28434.objecter dump_active .. 0 homeless 2024-02-13T00:56:57.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.332+0000 7fdf437fe700 10 client.28434.objecter ms_handle_connect 0x7fdf38062c90 2024-02-13T00:56:57.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.332+0000 7fdf4a59c700 1 --2- 172.21.15.162:0/2056230756 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf38062c90 0x7fdf38065150 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 rx=0x7fdf3c009250 tx=0x7fdf3c009d60).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:56:57.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.336+0000 7fdf437fe700 1 -- 172.21.15.162:0/2056230756 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fdf44051350 con 0x7fdf4c0fe640 2024-02-13T00:56:57.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.500+0000 7fdf50903700 1 -- 172.21.15.162:0/2056230756 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fdf4c109060 con 0x7fdf4c0fe640 2024-02-13T00:56:57.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.500+0000 7fdf437fe700 1 -- 172.21.15.162:0/2056230756 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fdf44021020 con 0x7fdf4c0fe640 2024-02-13T00:56:57.504 INFO:teuthology.orchestra.run.smithi162.stdout:107374182654 2024-02-13T00:56:57.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 10 librados: watch_flush enter 2024-02-13T00:56:57.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 10 librados: watch_flush exit 2024-02-13T00:56:57.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 20 client.28434.objecter shutdown clearing up homeless session... 2024-02-13T00:56:57.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 10 client.28434.objecter successfully canceled tick 2024-02-13T00:56:57.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 -- 172.21.15.162:0/2056230756 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf38062c90 msgr2=0x7fdf38065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:57.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 --2- 172.21.15.162:0/2056230756 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf38062c90 0x7fdf38065150 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 rx=0x7fdf3c009250 tx=0x7fdf3c009d60).stop 2024-02-13T00:56:57.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 -- 172.21.15.162:0/2056230756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf4c0fe640 msgr2=0x7fdf4c1081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:56:57.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 --2- 172.21.15.162:0/2056230756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf4c0fe640 0x7fdf4c1081c0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 rx=0x7fdf4400b6d0 tx=0x7fdf44004d10).stop 2024-02-13T00:56:57.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 -- 172.21.15.162:0/2056230756 shutdown_connections 2024-02-13T00:56:57.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 --2- 172.21.15.162:0/2056230756 >> v2:172.21.15.162:6800/2866534053 conn(0x7fdf38062c90 0x7fdf38065150 unknown :-1 s=CLOSED pgs=3093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:57.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 --2- 172.21.15.162:0/2056230756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf4c0fe640 0x7fdf4c1081c0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:57.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 --2- 172.21.15.162:0/2056230756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf4c0f6950 0x7fdf4c10e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:57.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 --2- 172.21.15.162:0/2056230756 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf4c0fdce0 0x7fdf4c10e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:56:57.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 -- 172.21.15.162:0/2056230756 >> 172.21.15.162:0/2056230756 conn(0x7fdf4c070060 msgr2=0x7fdf4c06d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:56:57.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 -- 172.21.15.162:0/2056230756 shutdown_connections 2024-02-13T00:56:57.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 -- 172.21.15.162:0/2056230756 wait complete. 2024-02-13T00:56:57.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:56:57.502+0000 7fdf50903700 1 librados: shutdown 2024-02-13T00:56:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:56.598809+0000 mgr.y (mgr.14544) 762 : cluster [DBG] pgmap v1358: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:56.723471+0000 osd.3 (osd.3) 241 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:56:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:56.726539+0000 osd.3 (osd.3) 242 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:56:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:56.896376+0000 osd.2 (osd.2) 215 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:56:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:56.899327+0000 osd.2 (osd.2) 216 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:56:58.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:56:57.501604+0000 mon.a (mon.0) 3881 : audit [DBG] from='client.? 172.21.15.162:0/2056230756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-13T00:56:58.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:56.598809+0000 mgr.y (mgr.14544) 762 : cluster [DBG] pgmap v1358: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:58.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:56.723471+0000 osd.3 (osd.3) 241 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:56:58.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:56.726539+0000 osd.3 (osd.3) 242 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:56:58.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:56.896376+0000 osd.2 (osd.2) 215 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:56:58.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:56.899327+0000 osd.2 (osd.2) 216 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:56:58.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:56:57.501604+0000 mon.a (mon.0) 3881 : audit [DBG] from='client.? 172.21.15.162:0/2056230756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-13T00:56:58.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:56.598809+0000 mgr.y (mgr.14544) 762 : cluster [DBG] pgmap v1358: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:56:58.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:56.723471+0000 osd.3 (osd.3) 241 : cluster [DBG] 2.1b scrub starts 2024-02-13T00:56:58.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:56.726539+0000 osd.3 (osd.3) 242 : cluster [DBG] 2.1b scrub ok 2024-02-13T00:56:58.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:56.896376+0000 osd.2 (osd.2) 215 : cluster [DBG] 2.10 scrub starts 2024-02-13T00:56:58.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:56.899327+0000 osd.2 (osd.2) 216 : cluster [DBG] 2.10 scrub ok 2024-02-13T00:56:58.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:57 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:56:57.501604+0000 mon.a (mon.0) 3881 : audit [DBG] from='client.? 172.21.15.162:0/2056230756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-13T00:56:58.406 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182649 got 107374182654 for osd.3 2024-02-13T00:56:58.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-13T00:56:58.678 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:56:58.917 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:56:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:56:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:56:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:57.140873+0000 osd.1 (osd.1) 419 : cluster [DBG] 6.1 deep-scrub starts 2024-02-13T00:56:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:57.143846+0000 osd.1 (osd.1) 420 : cluster [DBG] 6.1 deep-scrub ok 2024-02-13T00:56:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:57.770592+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:56:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:57.773616+0000 osd.5 (osd.5) 75 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:56:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:57.795583+0000 osd.0 (osd.0) 407 : cluster [DBG] 6.10 scrub starts 2024-02-13T00:56:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:57.796476+0000 osd.0 (osd.0) 408 : cluster [DBG] 6.10 scrub ok 2024-02-13T00:56:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:58.241618+0000 osd.4 (osd.4) 105 : cluster [DBG] 5.e scrub starts 2024-02-13T00:56:59.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:58.245114+0000 osd.4 (osd.4) 106 : cluster [DBG] 5.e scrub ok 2024-02-13T00:56:59.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:57.140873+0000 osd.1 (osd.1) 419 : cluster [DBG] 6.1 deep-scrub starts 2024-02-13T00:56:59.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:57.143846+0000 osd.1 (osd.1) 420 : cluster [DBG] 6.1 deep-scrub ok 2024-02-13T00:56:59.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:57.770592+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:56:59.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:57.773616+0000 osd.5 (osd.5) 75 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:56:59.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:57.795583+0000 osd.0 (osd.0) 407 : cluster [DBG] 6.10 scrub starts 2024-02-13T00:56:59.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:57.796476+0000 osd.0 (osd.0) 408 : cluster [DBG] 6.10 scrub ok 2024-02-13T00:56:59.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:58.241618+0000 osd.4 (osd.4) 105 : cluster [DBG] 5.e scrub starts 2024-02-13T00:56:59.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:58.245114+0000 osd.4 (osd.4) 106 : cluster [DBG] 5.e scrub ok 2024-02-13T00:56:59.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:57.140873+0000 osd.1 (osd.1) 419 : cluster [DBG] 6.1 deep-scrub starts 2024-02-13T00:56:59.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:57.143846+0000 osd.1 (osd.1) 420 : cluster [DBG] 6.1 deep-scrub ok 2024-02-13T00:56:59.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:57.770592+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.7 scrub starts 2024-02-13T00:56:59.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:57.773616+0000 osd.5 (osd.5) 75 : cluster [DBG] 6.7 scrub ok 2024-02-13T00:56:59.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:57.795583+0000 osd.0 (osd.0) 407 : cluster [DBG] 6.10 scrub starts 2024-02-13T00:56:59.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:57.796476+0000 osd.0 (osd.0) 408 : cluster [DBG] 6.10 scrub ok 2024-02-13T00:56:59.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:58.241618+0000 osd.4 (osd.4) 105 : cluster [DBG] 5.e scrub starts 2024-02-13T00:56:59.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:58 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:58.245114+0000 osd.4 (osd.4) 106 : cluster [DBG] 5.e scrub ok 2024-02-13T00:57:00.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:58.599254+0000 mgr.y (mgr.14544) 763 : cluster [DBG] pgmap v1359: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:57:00.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:58.771703+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.f scrub starts 2024-02-13T00:57:00.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:58.775173+0000 osd.5 (osd.5) 77 : cluster [DBG] 5.f scrub ok 2024-02-13T00:57:00.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:58.828556+0000 osd.7 (osd.7) 150 : cluster [DBG] 28.5 scrub starts 2024-02-13T00:57:00.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:56:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:58.829719+0000 osd.7 (osd.7) 151 : cluster [DBG] 28.5 scrub ok 2024-02-13T00:57:00.206 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:58.599254+0000 mgr.y (mgr.14544) 763 : cluster [DBG] pgmap v1359: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:57:00.206 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:58.771703+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.f scrub starts 2024-02-13T00:57:00.207 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:58.775173+0000 osd.5 (osd.5) 77 : cluster [DBG] 5.f scrub ok 2024-02-13T00:57:00.207 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:58.828556+0000 osd.7 (osd.7) 150 : cluster [DBG] 28.5 scrub starts 2024-02-13T00:57:00.207 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:56:59 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:58.829719+0000 osd.7 (osd.7) 151 : cluster [DBG] 28.5 scrub ok 2024-02-13T00:57:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:58.599254+0000 mgr.y (mgr.14544) 763 : cluster [DBG] pgmap v1359: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:57:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:58.771703+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.f scrub starts 2024-02-13T00:57:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:58.775173+0000 osd.5 (osd.5) 77 : cluster [DBG] 5.f scrub ok 2024-02-13T00:57:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:58.828556+0000 osd.7 (osd.7) 150 : cluster [DBG] 28.5 scrub starts 2024-02-13T00:57:00.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:56:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:58.829719+0000 osd.7 (osd.7) 151 : cluster [DBG] 28.5 scrub ok 2024-02-13T00:57:00.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.760+0000 7f265925c700 1 -- 172.21.15.162:0/1962176831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26540fe580 msgr2=0x7f26540f7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:00.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.760+0000 7f265925c700 1 --2- 172.21.15.162:0/1962176831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26540fe580 0x7f26540f7db0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 rx=0x7f263c0047b0 tx=0x7f263c009cb0).stop 2024-02-13T00:57:00.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.760+0000 7f265925c700 1 -- 172.21.15.162:0/1962176831 shutdown_connections 2024-02-13T00:57:00.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.760+0000 7f265925c700 1 --2- 172.21.15.162:0/1962176831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26540f8490 0x7f2654104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:00.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.760+0000 7f265925c700 1 --2- 172.21.15.162:0/1962176831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26540fdc20 0x7f26540fe000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:00.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.760+0000 7f265925c700 1 --2- 172.21.15.162:0/1962176831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26540fe580 0x7f26540f7db0 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:00.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.760+0000 7f265925c700 1 -- 172.21.15.162:0/1962176831 >> 172.21.15.162:0/1962176831 conn(0x7f26540f3850 msgr2=0x7f26540f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:00.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.760+0000 7f265925c700 1 -- 172.21.15.162:0/1962176831 shutdown_connections 2024-02-13T00:57:00.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.760+0000 7f265925c700 1 -- 172.21.15.162:0/1962176831 wait complete. 2024-02-13T00:57:00.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.760+0000 7f265925c700 1 librados: starting msgr at 2024-02-13T00:57:00.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.760+0000 7f265925c700 1 librados: starting objecter 2024-02-13T00:57:00.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265925c700 1 Processor -- start 2024-02-13T00:57:00.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265925c700 1 -- start start 2024-02-13T00:57:00.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265925c700 1 librados: setting wanted keys 2024-02-13T00:57:00.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265925c700 1 librados: calling monclient init 2024-02-13T00:57:00.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265925c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26540f8490 0x7f265410e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:00.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265925c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26540fdc20 0x7f265410e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:00.767 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265925c700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26540fe580 0x7f2654108340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:00.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265925c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2654108830 con 0x7f26540fdc20 2024-02-13T00:57:00.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265925c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26541089a0 con 0x7f26540f8490 2024-02-13T00:57:00.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265925c700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2654108b60 con 0x7f26540fe580 2024-02-13T00:57:00.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f264bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26540fdc20 0x7f265410e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:00.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265359e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26540fe580 0x7f2654108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:00.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f264bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26540fdc20 0x7f265410e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34160/0 (socket says 172.21.15.162:34160) 2024-02-13T00:57:00.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f264bfff700 1 -- 172.21.15.162:0/124022183 learned_addr learned my addr 172.21.15.162:0/124022183 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:57:00.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f2652d9d700 1 --2- 172.21.15.162:0/124022183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26540f8490 0x7f265410e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:00.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265359e700 1 -- 172.21.15.162:0/124022183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26540f8490 msgr2=0x7f265410e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:00.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265359e700 1 --2- 172.21.15.162:0/124022183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26540f8490 0x7f265410e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:00.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265359e700 1 -- 172.21.15.162:0/124022183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26540fdc20 msgr2=0x7f265410e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:00.768 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265359e700 1 --2- 172.21.15.162:0/124022183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26540fdc20 0x7f265410e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f265359e700 1 -- 172.21.15.162:0/124022183 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c005040 con 0x7f26540fe580 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.761+0000 7f264bfff700 1 --2- 172.21.15.162:0/124022183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26540fdc20 0x7f265410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.762+0000 7f265359e700 1 --2- 172.21.15.162:0/124022183 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26540fe580 0x7f2654108340 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 rx=0x7f264c00eb70 tx=0x7f264c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.762+0000 7f2650d99700 10 client.?.objecter ms_handle_connect 0x7f26540fe580 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.762+0000 7f2650d99700 10 client.?.objecter resend_mon_ops 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.762+0000 7f2650d99700 1 -- 172.21.15.162:0/124022183 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c00ee80 con 0x7f26540fe580 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.762+0000 7f265925c700 1 -- 172.21.15.162:0/124022183 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2654108de0 con 0x7f26540fe580 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.762+0000 7f2650d99700 1 -- 172.21.15.162:0/124022183 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f264c00f040 con 0x7f26540fe580 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.762+0000 7f2650d99700 1 -- 172.21.15.162:0/124022183 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c007d60 con 0x7f26540fe580 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.762+0000 7f265925c700 10 client.38534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.762+0000 7f265925c700 1 -- 172.21.15.162:0/124022183 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2654114f30 con 0x7f26540fe580 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.762+0000 7f265925c700 1 librados: init done 2024-02-13T00:57:00.769 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.763+0000 7f265925c700 1 -- 172.21.15.162:0/124022183 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26540607d0 con 0x7f26540fe580 2024-02-13T00:57:00.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.763+0000 7f2650d99700 1 -- 172.21.15.162:0/124022183 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f264c0092e0 con 0x7f26540fe580 2024-02-13T00:57:00.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.763+0000 7f2650d99700 1 --2- 172.21.15.162:0/124022183 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2640062c90 0x7f2640065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:00.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.764+0000 7f2650d99700 1 -- 172.21.15.162:0/124022183 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(857..857 src has 449..857) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f264c07cca0 con 0x7f26540fe580 2024-02-13T00:57:00.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.764+0000 7f2650d99700 10 client.38534.objecter ms_dispatch 0x7f2654001550 osd_map(857..857 src has 449..857) v4 2024-02-13T00:57:00.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.764+0000 7f2650d99700 3 client.38534.objecter handle_osd_map got epochs [857,857] > 0 2024-02-13T00:57:00.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.764+0000 7f2650d99700 3 client.38534.objecter handle_osd_map decoding full epoch 857 2024-02-13T00:57:00.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.764+0000 7f2652d9d700 1 --2- 172.21.15.162:0/124022183 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2640062c90 0x7f2640065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:00.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.764+0000 7f2650d99700 20 client.38534.objecter dump_active .. 0 homeless 2024-02-13T00:57:00.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.764+0000 7f2652d9d700 1 --2- 172.21.15.162:0/124022183 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2640062c90 0x7f2640065150 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 rx=0x7f2644004de0 tx=0x7f264400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:00.770 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.764+0000 7f2650d99700 10 client.38534.objecter ms_handle_connect 0x7f2640062c90 2024-02-13T00:57:00.771 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.768+0000 7f2650d99700 1 -- 172.21.15.162:0/124022183 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f264c049f60 con 0x7f26540fe580 2024-02-13T00:57:00.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.933+0000 7f265925c700 1 -- 172.21.15.162:0/124022183 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f26540475e0 con 0x7f26540fe580 2024-02-13T00:57:00.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.934+0000 7f2650d99700 1 -- 172.21.15.162:0/124022183 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f264c050440 con 0x7f26540fe580 2024-02-13T00:57:00.938 INFO:teuthology.orchestra.run.smithi162.stdout:133143986427 2024-02-13T00:57:00.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 10 librados: watch_flush enter 2024-02-13T00:57:00.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 10 librados: watch_flush exit 2024-02-13T00:57:00.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 20 client.38534.objecter shutdown clearing up homeless session... 2024-02-13T00:57:00.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 10 client.38534.objecter successfully canceled tick 2024-02-13T00:57:00.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 -- 172.21.15.162:0/124022183 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2640062c90 msgr2=0x7f2640065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:00.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 --2- 172.21.15.162:0/124022183 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2640062c90 0x7f2640065150 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 rx=0x7f2644004de0 tx=0x7f264400b040).stop 2024-02-13T00:57:00.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 -- 172.21.15.162:0/124022183 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26540fe580 msgr2=0x7f2654108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:00.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 --2- 172.21.15.162:0/124022183 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26540fe580 0x7f2654108340 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 rx=0x7f264c00eb70 tx=0x7f264c00b040).stop 2024-02-13T00:57:00.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 -- 172.21.15.162:0/124022183 shutdown_connections 2024-02-13T00:57:00.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 --2- 172.21.15.162:0/124022183 >> v2:172.21.15.162:6800/2866534053 conn(0x7f2640062c90 0x7f2640065150 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:00.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 --2- 172.21.15.162:0/124022183 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26540fe580 0x7f2654108340 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:00.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 --2- 172.21.15.162:0/124022183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26540f8490 0x7f265410e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:00.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 --2- 172.21.15.162:0/124022183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26540fdc20 0x7f265410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:00.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 -- 172.21.15.162:0/124022183 >> 172.21.15.162:0/124022183 conn(0x7f26540f3850 msgr2=0x7f26540fba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:00.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 -- 172.21.15.162:0/124022183 shutdown_connections 2024-02-13T00:57:00.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 -- 172.21.15.162:0/124022183 wait complete. 2024-02-13T00:57:00.941 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:00.936+0000 7f265925c700 1 librados: shutdown 2024-02-13T00:57:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:59.110226+0000 osd.1 (osd.1) 421 : cluster [DBG] 28.11 scrub starts 2024-02-13T00:57:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:59.113236+0000 osd.1 (osd.1) 422 : cluster [DBG] 28.11 scrub ok 2024-02-13T00:57:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:59.759654+0000 osd.5 (osd.5) 78 : cluster [DBG] 28.9 deep-scrub starts 2024-02-13T00:57:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:59.762790+0000 osd.5 (osd.5) 79 : cluster [DBG] 28.9 deep-scrub ok 2024-02-13T00:57:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:59.975983+0000 osd.6 (osd.6) 354 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:57:01.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:56:59.978872+0000 osd.6 (osd.6) 355 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:57:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:59.110226+0000 osd.1 (osd.1) 421 : cluster [DBG] 28.11 scrub starts 2024-02-13T00:57:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:59.113236+0000 osd.1 (osd.1) 422 : cluster [DBG] 28.11 scrub ok 2024-02-13T00:57:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:59.759654+0000 osd.5 (osd.5) 78 : cluster [DBG] 28.9 deep-scrub starts 2024-02-13T00:57:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:59.762790+0000 osd.5 (osd.5) 79 : cluster [DBG] 28.9 deep-scrub ok 2024-02-13T00:57:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:59.975983+0000 osd.6 (osd.6) 354 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:57:01.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:56:59.978872+0000 osd.6 (osd.6) 355 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:57:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:59.110226+0000 osd.1 (osd.1) 421 : cluster [DBG] 28.11 scrub starts 2024-02-13T00:57:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:59.113236+0000 osd.1 (osd.1) 422 : cluster [DBG] 28.11 scrub ok 2024-02-13T00:57:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:59.759654+0000 osd.5 (osd.5) 78 : cluster [DBG] 28.9 deep-scrub starts 2024-02-13T00:57:01.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:59.762790+0000 osd.5 (osd.5) 79 : cluster [DBG] 28.9 deep-scrub ok 2024-02-13T00:57:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:59.975983+0000 osd.6 (osd.6) 354 : cluster [DBG] 2.14 scrub starts 2024-02-13T00:57:01.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:56:59.978872+0000 osd.6 (osd.6) 355 : cluster [DBG] 2.14 scrub ok 2024-02-13T00:57:01.768 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986422 got 133143986427 for osd.4 2024-02-13T00:57:01.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-13T00:57:02.040 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:57:02.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:00.128036+0000 osd.1 (osd.1) 423 : cluster [DBG] 6.26 scrub starts 2024-02-13T00:57:02.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:00.128941+0000 osd.1 (osd.1) 424 : cluster [DBG] 6.26 scrub ok 2024-02-13T00:57:02.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:00.600155+0000 mgr.y (mgr.14544) 764 : cluster [DBG] pgmap v1360: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:57:02.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:00.934933+0000 mon.c (mon.2) 543 : audit [DBG] from='client.? 172.21.15.162:0/124022183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-13T00:57:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:00.128036+0000 osd.1 (osd.1) 423 : cluster [DBG] 6.26 scrub starts 2024-02-13T00:57:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:00.128941+0000 osd.1 (osd.1) 424 : cluster [DBG] 6.26 scrub ok 2024-02-13T00:57:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:00.600155+0000 mgr.y (mgr.14544) 764 : cluster [DBG] pgmap v1360: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:57:02.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:01 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:00.934933+0000 mon.c (mon.2) 543 : audit [DBG] from='client.? 172.21.15.162:0/124022183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-13T00:57:02.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:00.128036+0000 osd.1 (osd.1) 423 : cluster [DBG] 6.26 scrub starts 2024-02-13T00:57:02.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:00.128941+0000 osd.1 (osd.1) 424 : cluster [DBG] 6.26 scrub ok 2024-02-13T00:57:02.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:00.600155+0000 mgr.y (mgr.14544) 764 : cluster [DBG] pgmap v1360: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-13T00:57:02.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:00.934933+0000 mon.c (mon.2) 543 : audit [DBG] from='client.? 172.21.15.162:0/124022183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-13T00:57:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:01.688206+0000 osd.5 (osd.5) 80 : cluster [DBG] 28.1d scrub starts 2024-02-13T00:57:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:01.691805+0000 osd.5 (osd.5) 81 : cluster [DBG] 28.1d scrub ok 2024-02-13T00:57:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:01.822139+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:57:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:01.823296+0000 osd.7 (osd.7) 153 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:57:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:01.847214+0000 osd.0 (osd.0) 409 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:57:03.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:01.850681+0000 osd.0 (osd.0) 410 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:57:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:01.688206+0000 osd.5 (osd.5) 80 : cluster [DBG] 28.1d scrub starts 2024-02-13T00:57:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:01.691805+0000 osd.5 (osd.5) 81 : cluster [DBG] 28.1d scrub ok 2024-02-13T00:57:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:01.822139+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:57:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:01.823296+0000 osd.7 (osd.7) 153 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:57:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:01.847214+0000 osd.0 (osd.0) 409 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:57:03.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:01.850681+0000 osd.0 (osd.0) 410 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:57:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:01.688206+0000 osd.5 (osd.5) 80 : cluster [DBG] 28.1d scrub starts 2024-02-13T00:57:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:01.691805+0000 osd.5 (osd.5) 81 : cluster [DBG] 28.1d scrub ok 2024-02-13T00:57:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:01.822139+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.15 scrub starts 2024-02-13T00:57:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:01.823296+0000 osd.7 (osd.7) 153 : cluster [DBG] 6.15 scrub ok 2024-02-13T00:57:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:01.847214+0000 osd.0 (osd.0) 409 : cluster [DBG] 4.1e scrub starts 2024-02-13T00:57:03.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:02 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:01.850681+0000 osd.0 (osd.0) 410 : cluster [DBG] 4.1e scrub ok 2024-02-13T00:57:03.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.819+0000 7f6c9329a700 1 -- 172.21.15.162:0/3769912501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c8c0fde50 msgr2=0x7f6c8c0fe230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:03.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.819+0000 7f6c9329a700 1 --2- 172.21.15.162:0/3769912501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c8c0fde50 0x7f6c8c0fe230 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 rx=0x7f6c88004870 tx=0x7f6c88009cb0).stop 2024-02-13T00:57:03.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.819+0000 7f6c9329a700 1 -- 172.21.15.162:0/3769912501 shutdown_connections 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.819+0000 7f6c9329a700 1 --2- 172.21.15.162:0/3769912501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c8c0f6880 0x7f6c8c104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.819+0000 7f6c9329a700 1 --2- 172.21.15.162:0/3769912501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c8c0fde50 0x7f6c8c0fe230 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.819+0000 7f6c9329a700 1 --2- 172.21.15.162:0/3769912501 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6c8c0fe7b0 0x7f6c8c0f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.819+0000 7f6c9329a700 1 -- 172.21.15.162:0/3769912501 >> 172.21.15.162:0/3769912501 conn(0x7f6c8c070080 msgr2=0x7f6c8c0f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 -- 172.21.15.162:0/3769912501 shutdown_connections 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 -- 172.21.15.162:0/3769912501 wait complete. 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 librados: starting msgr at 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 librados: starting objecter 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 Processor -- start 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 -- start start 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 librados: setting wanted keys 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 librados: calling monclient init 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c8c0f6880 0x7f6c8c0f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:03.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c8c0fde50 0x7f6c8c0f7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6c8c0fe7b0 0x7f6c8c0f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c116930 con 0x7f6c8c0fde50 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c116aa0 con 0x7f6c8c0f6880 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c9329a700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c116c60 con 0x7f6c8c0fe7b0 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.820+0000 7f6c90835700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c8c0fde50 0x7f6c8c0f7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c90835700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c8c0fde50 0x7f6c8c0f7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34184/0 (socket says 172.21.15.162:34184) 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c90835700 1 -- 172.21.15.162:0/1928278524 learned_addr learned my addr 172.21.15.162:0/1928278524 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c91837700 1 --2- 172.21.15.162:0/1928278524 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6c8c0fe7b0 0x7f6c8c0f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c91036700 1 --2- 172.21.15.162:0/1928278524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c8c0f6880 0x7f6c8c0f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c90835700 1 -- 172.21.15.162:0/1928278524 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6c8c0fe7b0 msgr2=0x7f6c8c0f7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c90835700 1 --2- 172.21.15.162:0/1928278524 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6c8c0fe7b0 0x7f6c8c0f7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c90835700 1 -- 172.21.15.162:0/1928278524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c8c0f6880 msgr2=0x7f6c8c0f7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c90835700 1 --2- 172.21.15.162:0/1928278524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c8c0f6880 0x7f6c8c0f7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c90835700 1 -- 172.21.15.162:0/1928278524 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c88005040 con 0x7f6c8c0fde50 2024-02-13T00:57:03.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c91837700 1 --2- 172.21.15.162:0/1928278524 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6c8c0fe7b0 0x7f6c8c0f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:57:03.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c90835700 1 --2- 172.21.15.162:0/1928278524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c8c0fde50 0x7f6c8c0f7960 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7f6c7c00e970 tx=0x7f6c7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:03.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c82777700 10 client.?.objecter ms_handle_connect 0x7f6c8c0fde50 2024-02-13T00:57:03.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c82777700 10 client.?.objecter resend_mon_ops 2024-02-13T00:57:03.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c82777700 1 -- 172.21.15.162:0/1928278524 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c00ee60 con 0x7f6c8c0fde50 2024-02-13T00:57:03.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c82777700 1 -- 172.21.15.162:0/1928278524 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c7c00f040 con 0x7f6c8c0fde50 2024-02-13T00:57:03.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c9329a700 1 -- 172.21.15.162:0/1928278524 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c116f40 con 0x7f6c8c0fde50 2024-02-13T00:57:03.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c82777700 1 -- 172.21.15.162:0/1928278524 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c007ea0 con 0x7f6c8c0fde50 2024-02-13T00:57:03.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c9329a700 10 client.28446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:57:03.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c9329a700 1 -- 172.21.15.162:0/1928278524 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c1173f0 con 0x7f6c8c0fde50 2024-02-13T00:57:03.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.821+0000 7f6c9329a700 1 librados: init done 2024-02-13T00:57:03.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.822+0000 7f6c9329a700 1 -- 172.21.15.162:0/1928278524 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8c05fb80 con 0x7f6c8c0fde50 2024-02-13T00:57:03.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.822+0000 7f6c82777700 1 -- 172.21.15.162:0/1928278524 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f6c7c00cb10 con 0x7f6c8c0fde50 2024-02-13T00:57:03.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.823+0000 7f6c82777700 1 --2- 172.21.15.162:0/1928278524 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6c78062c90 0x7f6c78065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:03.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.823+0000 7f6c91036700 1 --2- 172.21.15.162:0/1928278524 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6c78062c90 0x7f6c78065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:03.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.823+0000 7f6c82777700 1 -- 172.21.15.162:0/1928278524 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(857..857 src has 449..857) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f6c7c07cb90 con 0x7f6c8c0fde50 2024-02-13T00:57:03.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.823+0000 7f6c82777700 10 client.28446.objecter ms_dispatch 0x7f6c8c001550 osd_map(857..857 src has 449..857) v4 2024-02-13T00:57:03.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.823+0000 7f6c82777700 3 client.28446.objecter handle_osd_map got epochs [857,857] > 0 2024-02-13T00:57:03.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.823+0000 7f6c82777700 3 client.28446.objecter handle_osd_map decoding full epoch 857 2024-02-13T00:57:03.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.823+0000 7f6c82777700 20 client.28446.objecter dump_active .. 0 homeless 2024-02-13T00:57:03.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.823+0000 7f6c82777700 10 client.28446.objecter ms_handle_connect 0x7f6c78062c90 2024-02-13T00:57:03.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.823+0000 7f6c91036700 1 --2- 172.21.15.162:0/1928278524 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6c78062c90 0x7f6c78065150 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 rx=0x7f6c88004ee0 tx=0x7f6c8800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:03.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.827+0000 7f6c82777700 1 -- 172.21.15.162:0/1928278524 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6c7c050370 con 0x7f6c8c0fde50 2024-02-13T00:57:03.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.992+0000 7f6c9329a700 1 -- 172.21.15.162:0/1928278524 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6c8c060420 con 0x7f6c8c0fde50 2024-02-13T00:57:03.997 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.993+0000 7f6c82777700 1 -- 172.21.15.162:0/1928278524 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+14 (secure 0 0 0) 0x7f6c7c04ff60 con 0x7f6c8c0fde50 2024-02-13T00:57:03.997 INFO:teuthology.orchestra.run.smithi162.stdout:2057289334893 2024-02-13T00:57:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 10 librados: watch_flush enter 2024-02-13T00:57:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 10 librados: watch_flush exit 2024-02-13T00:57:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 20 client.28446.objecter shutdown clearing up homeless session... 2024-02-13T00:57:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 10 client.28446.objecter successfully canceled tick 2024-02-13T00:57:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 -- 172.21.15.162:0/1928278524 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6c78062c90 msgr2=0x7f6c78065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 --2- 172.21.15.162:0/1928278524 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6c78062c90 0x7f6c78065150 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 rx=0x7f6c88004ee0 tx=0x7f6c8800b040).stop 2024-02-13T00:57:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 -- 172.21.15.162:0/1928278524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c8c0fde50 msgr2=0x7f6c8c0f7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 --2- 172.21.15.162:0/1928278524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c8c0fde50 0x7f6c8c0f7960 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7f6c7c00e970 tx=0x7f6c7c00b040).stop 2024-02-13T00:57:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 -- 172.21.15.162:0/1928278524 shutdown_connections 2024-02-13T00:57:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 --2- 172.21.15.162:0/1928278524 >> v2:172.21.15.162:6800/2866534053 conn(0x7f6c78062c90 0x7f6c78065150 unknown :-1 s=CLOSED pgs=3095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 --2- 172.21.15.162:0/1928278524 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6c8c0fe7b0 0x7f6c8c0f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:03.999 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 --2- 172.21.15.162:0/1928278524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c8c0f6880 0x7f6c8c0f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:04.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 --2- 172.21.15.162:0/1928278524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c8c0fde50 0x7f6c8c0f7960 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:04.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 -- 172.21.15.162:0/1928278524 >> 172.21.15.162:0/1928278524 conn(0x7f6c8c070080 msgr2=0x7f6c8c0fbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:04.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 -- 172.21.15.162:0/1928278524 shutdown_connections 2024-02-13T00:57:04.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 -- 172.21.15.162:0/1928278524 wait complete. 2024-02-13T00:57:04.000 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:03.995+0000 7f6c9329a700 1 librados: shutdown 2024-02-13T00:57:04.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:02.600567+0000 mgr.y (mgr.14544) 765 : cluster [DBG] pgmap v1361: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:57:04.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:02.709099+0000 osd.3 (osd.3) 243 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:57:04.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:02.710626+0000 osd.3 (osd.3) 244 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:57:04.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:02.600567+0000 mgr.y (mgr.14544) 765 : cluster [DBG] pgmap v1361: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:57:04.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:02.709099+0000 osd.3 (osd.3) 243 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:57:04.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:02.710626+0000 osd.3 (osd.3) 244 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:57:04.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:02.600567+0000 mgr.y (mgr.14544) 765 : cluster [DBG] pgmap v1361: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:57:04.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:02.709099+0000 osd.3 (osd.3) 243 : cluster [DBG] 4.17 scrub starts 2024-02-13T00:57:04.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:03 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:02.710626+0000 osd.3 (osd.3) 244 : cluster [DBG] 4.17 scrub ok 2024-02-13T00:57:04.734 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2057289334889 got 2057289334893 for osd.5 2024-02-13T00:57:04.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-13T00:57:05.011 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:57:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:03.108927+0000 osd.1 (osd.1) 425 : cluster [DBG] 5.12 deep-scrub starts 2024-02-13T00:57:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:03.112015+0000 osd.1 (osd.1) 426 : cluster [DBG] 5.12 deep-scrub ok 2024-02-13T00:57:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:03.782628+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:57:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:03.783570+0000 osd.7 (osd.7) 155 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:57:05.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:03.993930+0000 mon.a (mon.0) 3882 : audit [DBG] from='client.? 172.21.15.162:0/1928278524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-13T00:57:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:03.108927+0000 osd.1 (osd.1) 425 : cluster [DBG] 5.12 deep-scrub starts 2024-02-13T00:57:05.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:03.112015+0000 osd.1 (osd.1) 426 : cluster [DBG] 5.12 deep-scrub ok 2024-02-13T00:57:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:03.782628+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:57:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:03.783570+0000 osd.7 (osd.7) 155 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:57:05.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:04 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:03.993930+0000 mon.a (mon.0) 3882 : audit [DBG] from='client.? 172.21.15.162:0/1928278524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-13T00:57:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:03.108927+0000 osd.1 (osd.1) 425 : cluster [DBG] 5.12 deep-scrub starts 2024-02-13T00:57:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:03.112015+0000 osd.1 (osd.1) 426 : cluster [DBG] 5.12 deep-scrub ok 2024-02-13T00:57:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:03.782628+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub starts 2024-02-13T00:57:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:03.783570+0000 osd.7 (osd.7) 155 : cluster [DBG] 2.1d scrub ok 2024-02-13T00:57:05.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:03.993930+0000 mon.a (mon.0) 3882 : audit [DBG] from='client.? 172.21.15.162:0/1928278524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-13T00:57:06.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:57:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:57:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:04.601070+0000 mgr.y (mgr.14544) 766 : cluster [DBG] pgmap v1362: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:57:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:04.927136+0000 osd.0 (osd.0) 411 : cluster [DBG] 2.1f deep-scrub starts 2024-02-13T00:57:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:04.930144+0000 osd.0 (osd.0) 412 : cluster [DBG] 2.1f deep-scrub ok 2024-02-13T00:57:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:04.936586+0000 osd.6 (osd.6) 356 : cluster [DBG] 4.a scrub starts 2024-02-13T00:57:06.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:04.940302+0000 osd.6 (osd.6) 357 : cluster [DBG] 4.a scrub ok 2024-02-13T00:57:06.330 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:57:06] "GET /metrics HTTP/1.1" 200 185524 "" "Prometheus/2.33.4" 2024-02-13T00:57:06.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:04.601070+0000 mgr.y (mgr.14544) 766 : cluster [DBG] pgmap v1362: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:57:06.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:04.927136+0000 osd.0 (osd.0) 411 : cluster [DBG] 2.1f deep-scrub starts 2024-02-13T00:57:06.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:04.930144+0000 osd.0 (osd.0) 412 : cluster [DBG] 2.1f deep-scrub ok 2024-02-13T00:57:06.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:04.936586+0000 osd.6 (osd.6) 356 : cluster [DBG] 4.a scrub starts 2024-02-13T00:57:06.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:04.940302+0000 osd.6 (osd.6) 357 : cluster [DBG] 4.a scrub ok 2024-02-13T00:57:06.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:04.601070+0000 mgr.y (mgr.14544) 766 : cluster [DBG] pgmap v1362: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:57:06.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:04.927136+0000 osd.0 (osd.0) 411 : cluster [DBG] 2.1f deep-scrub starts 2024-02-13T00:57:06.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:04.930144+0000 osd.0 (osd.0) 412 : cluster [DBG] 2.1f deep-scrub ok 2024-02-13T00:57:06.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:04.936586+0000 osd.6 (osd.6) 356 : cluster [DBG] 4.a scrub starts 2024-02-13T00:57:06.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:05 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:04.940302+0000 osd.6 (osd.6) 357 : cluster [DBG] 4.a scrub ok 2024-02-13T00:57:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.019+0000 7fcdca00b700 1 -- 172.21.15.162:0/636806206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc40f73a0 msgr2=0x7fcdc41039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.019+0000 7fcdca00b700 1 --2- 172.21.15.162:0/636806206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc40f73a0 0x7fcdc41039a0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 rx=0x7fcdbc009c30 tx=0x7fcdbc004c50).stop 2024-02-13T00:57:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.019+0000 7fcdca00b700 1 -- 172.21.15.162:0/636806206 shutdown_connections 2024-02-13T00:57:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.019+0000 7fcdca00b700 1 --2- 172.21.15.162:0/636806206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc40f73a0 0x7fcdc41039a0 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.019+0000 7fcdca00b700 1 --2- 172.21.15.162:0/636806206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdc40fcb30 0x7fcdc40fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.019+0000 7fcdca00b700 1 --2- 172.21.15.162:0/636806206 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdc40fd490 0x7fcdc40f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.019+0000 7fcdca00b700 1 -- 172.21.15.162:0/636806206 >> 172.21.15.162:0/636806206 conn(0x7fcdc406eb90 msgr2=0x7fcdc40f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.020+0000 7fcdca00b700 1 -- 172.21.15.162:0/636806206 shutdown_connections 2024-02-13T00:57:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.020+0000 7fcdca00b700 1 -- 172.21.15.162:0/636806206 wait complete. 2024-02-13T00:57:07.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.020+0000 7fcdca00b700 1 librados: starting msgr at 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.020+0000 7fcdca00b700 1 librados: starting objecter 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.020+0000 7fcdca00b700 1 Processor -- start 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdca00b700 1 -- start start 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdca00b700 1 librados: setting wanted keys 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdca00b700 1 librados: calling monclient init 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdca00b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdc40f73a0 0x7fcdc410ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdca00b700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdc40fcb30 0x7fcdc410d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdca00b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc40fd490 0x7fcdc4107010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdca00b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4107500 con 0x7fcdc40fd490 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdca00b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4107670 con 0x7fcdc40f73a0 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdca00b700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4107830 con 0x7fcdc40fcb30 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc2ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdc40fcb30 0x7fcdc410d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:07.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc3fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc40fd490 0x7fcdc4107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc2ffd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdc40fcb30 0x7fcdc410d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:38266/0 (socket says 172.21.15.162:38266) 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc2ffd700 1 -- 172.21.15.162:0/4137760259 learned_addr learned my addr 172.21.15.162:0/4137760259 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc37fe700 1 --2- 172.21.15.162:0/4137760259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdc40f73a0 0x7fcdc410ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc2ffd700 1 -- 172.21.15.162:0/4137760259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdc40f73a0 msgr2=0x7fcdc410ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc2ffd700 1 --2- 172.21.15.162:0/4137760259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdc40f73a0 0x7fcdc410ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc2ffd700 1 -- 172.21.15.162:0/4137760259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc40fd490 msgr2=0x7fcdc4107010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc2ffd700 1 --2- 172.21.15.162:0/4137760259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc40fd490 0x7fcdc4107010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc2ffd700 1 -- 172.21.15.162:0/4137760259 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb8005040 con 0x7fcdc40fcb30 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc3fff700 1 --2- 172.21.15.162:0/4137760259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc40fd490 0x7fcdc4107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc2ffd700 1 --2- 172.21.15.162:0/4137760259 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdc40fcb30 0x7fcdc410d360 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 rx=0x7fcdb8009400 tx=0x7fcdb8009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc0ff9700 10 client.?.objecter ms_handle_connect 0x7fcdc40fcb30 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc0ff9700 10 client.?.objecter resend_mon_ops 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.021+0000 7fcdc0ff9700 1 -- 172.21.15.162:0/4137760259 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb8007a40 con 0x7fcdc40fcb30 2024-02-13T00:57:07.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.022+0000 7fcdc0ff9700 1 -- 172.21.15.162:0/4137760259 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdb800f040 con 0x7fcdc40fcb30 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.022+0000 7fcdca00b700 1 -- 172.21.15.162:0/4137760259 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdbc0097c0 con 0x7fcdc40fcb30 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.022+0000 7fcdc0ff9700 1 -- 172.21.15.162:0/4137760259 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb8014a70 con 0x7fcdc40fcb30 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.022+0000 7fcdca00b700 10 client.38546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.022+0000 7fcdca00b700 1 -- 172.21.15.162:0/4137760259 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc4107e70 con 0x7fcdc40fcb30 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.022+0000 7fcdca00b700 1 librados: init done 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.023+0000 7fcdca00b700 1 -- 172.21.15.162:0/4137760259 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc40475e0 con 0x7fcdc40fcb30 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.023+0000 7fcdc0ff9700 1 -- 172.21.15.162:0/4137760259 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fcdb8010530 con 0x7fcdc40fcb30 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.023+0000 7fcdc0ff9700 1 --2- 172.21.15.162:0/4137760259 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcdac062c90 0x7fcdac065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.024+0000 7fcdc37fe700 1 --2- 172.21.15.162:0/4137760259 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcdac062c90 0x7fcdac065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.024+0000 7fcdc0ff9700 1 -- 172.21.15.162:0/4137760259 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(857..857 src has 449..857) v4 ==== 7509+0+0 (secure 0 0 0) 0x7fcdb807cb40 con 0x7fcdc40fcb30 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.024+0000 7fcdc0ff9700 10 client.38546.objecter ms_dispatch 0x7fcdc4001550 osd_map(857..857 src has 449..857) v4 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.024+0000 7fcdc0ff9700 3 client.38546.objecter handle_osd_map got epochs [857,857] > 0 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.024+0000 7fcdc0ff9700 3 client.38546.objecter handle_osd_map decoding full epoch 857 2024-02-13T00:57:07.034 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.024+0000 7fcdc0ff9700 20 client.38546.objecter dump_active .. 0 homeless 2024-02-13T00:57:07.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.024+0000 7fcdc0ff9700 10 client.38546.objecter ms_handle_connect 0x7fcdac062c90 2024-02-13T00:57:07.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.024+0000 7fcdc37fe700 1 --2- 172.21.15.162:0/4137760259 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcdac062c90 0x7fcdac065150 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 rx=0x7fcdb4004dc0 tx=0x7fcdb400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:07.035 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.028+0000 7fcdc0ff9700 1 -- 172.21.15.162:0/4137760259 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fcdb8049f60 con 0x7fcdc40fcb30 2024-02-13T00:57:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:05.653221+0000 osd.3 (osd.3) 245 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:57:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:05.656715+0000 osd.3 (osd.3) 246 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:57:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:05.970409+0000 osd.2 (osd.2) 217 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:57:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:05.973302+0000 osd.2 (osd.2) 218 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:57:07.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:06.127587+0000 mgr.y (mgr.14544) 767 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:57:07.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:05.653221+0000 osd.3 (osd.3) 245 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:57:07.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:05.656715+0000 osd.3 (osd.3) 246 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:57:07.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:05.970409+0000 osd.2 (osd.2) 217 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:57:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:05.973302+0000 osd.2 (osd.2) 218 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:57:07.081 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:06 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:06.127587+0000 mgr.y (mgr.14544) 767 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:57:07.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.197+0000 7fcdca00b700 1 -- 172.21.15.162:0/4137760259 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fcdc405fb80 con 0x7fcdc40fcb30 2024-02-13T00:57:07.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.197+0000 7fcdc0ff9700 1 -- 172.21.15.162:0/4137760259 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fcdb8050270 con 0x7fcdc40fcb30 2024-02-13T00:57:07.201 INFO:teuthology.orchestra.run.smithi162.stdout:193273528562 2024-02-13T00:57:07.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 10 librados: watch_flush enter 2024-02-13T00:57:07.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 10 librados: watch_flush exit 2024-02-13T00:57:07.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 20 client.38546.objecter shutdown clearing up homeless session... 2024-02-13T00:57:07.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 10 client.38546.objecter successfully canceled tick 2024-02-13T00:57:07.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 -- 172.21.15.162:0/4137760259 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcdac062c90 msgr2=0x7fcdac065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:07.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 --2- 172.21.15.162:0/4137760259 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcdac062c90 0x7fcdac065150 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 rx=0x7fcdb4004dc0 tx=0x7fcdb400b040).stop 2024-02-13T00:57:07.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 -- 172.21.15.162:0/4137760259 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdc40fcb30 msgr2=0x7fcdc410d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:07.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 --2- 172.21.15.162:0/4137760259 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdc40fcb30 0x7fcdc410d360 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 rx=0x7fcdb8009400 tx=0x7fcdb8009cb0).stop 2024-02-13T00:57:07.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 -- 172.21.15.162:0/4137760259 shutdown_connections 2024-02-13T00:57:07.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 --2- 172.21.15.162:0/4137760259 >> v2:172.21.15.162:6800/2866534053 conn(0x7fcdac062c90 0x7fcdac065150 unknown :-1 s=CLOSED pgs=3096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:07.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 --2- 172.21.15.162:0/4137760259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc40fd490 0x7fcdc4107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:07.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 --2- 172.21.15.162:0/4137760259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdc40f73a0 0x7fcdc410ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:07.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 --2- 172.21.15.162:0/4137760259 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcdc40fcb30 0x7fcdc410d360 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:07.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 -- 172.21.15.162:0/4137760259 >> 172.21.15.162:0/4137760259 conn(0x7fcdc406eb90 msgr2=0x7fcdc406fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:07.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 -- 172.21.15.162:0/4137760259 shutdown_connections 2024-02-13T00:57:07.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 -- 172.21.15.162:0/4137760259 wait complete. 2024-02-13T00:57:07.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:07.199+0000 7fcdca00b700 1 librados: shutdown 2024-02-13T00:57:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:05.653221+0000 osd.3 (osd.3) 245 : cluster [DBG] 4.0 scrub starts 2024-02-13T00:57:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:05.656715+0000 osd.3 (osd.3) 246 : cluster [DBG] 4.0 scrub ok 2024-02-13T00:57:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:05.970409+0000 osd.2 (osd.2) 217 : cluster [DBG] 5.8 scrub starts 2024-02-13T00:57:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:05.973302+0000 osd.2 (osd.2) 218 : cluster [DBG] 5.8 scrub ok 2024-02-13T00:57:07.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:06.127587+0000 mgr.y (mgr.14544) 767 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:57:07.945 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528557 got 193273528562 for osd.6 2024-02-13T00:57:07.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-13T00:57:08.218 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:57:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:06.155374+0000 osd.1 (osd.1) 427 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:57:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:06.158473+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:57:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:06.601807+0000 mgr.y (mgr.14544) 768 : cluster [DBG] pgmap v1363: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-02-13T00:57:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:07.198223+0000 mon.c (mon.2) 544 : audit [DBG] from='client.? 172.21.15.162:0/4137760259' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-13T00:57:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:07.312780+0000 osd.4 (osd.4) 107 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:57:08.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:07.314178+0000 osd.4 (osd.4) 108 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:57:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:06.155374+0000 osd.1 (osd.1) 427 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:57:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:06.158473+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:57:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:06.601807+0000 mgr.y (mgr.14544) 768 : cluster [DBG] pgmap v1363: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-02-13T00:57:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:07.198223+0000 mon.c (mon.2) 544 : audit [DBG] from='client.? 172.21.15.162:0/4137760259' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-13T00:57:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:07.312780+0000 osd.4 (osd.4) 107 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:57:08.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:07.314178+0000 osd.4 (osd.4) 108 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:57:08.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:06.155374+0000 osd.1 (osd.1) 427 : cluster [DBG] 2.15 scrub starts 2024-02-13T00:57:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:06.158473+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.15 scrub ok 2024-02-13T00:57:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:06.601807+0000 mgr.y (mgr.14544) 768 : cluster [DBG] pgmap v1363: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-02-13T00:57:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:07.198223+0000 mon.c (mon.2) 544 : audit [DBG] from='client.? 172.21.15.162:0/4137760259' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-13T00:57:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:07.312780+0000 osd.4 (osd.4) 107 : cluster [DBG] 6.17 scrub starts 2024-02-13T00:57:08.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:07 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:07.314178+0000 osd.4 (osd.4) 108 : cluster [DBG] 6.17 scrub ok 2024-02-13T00:57:08.952 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:57:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:57:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:57:09.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:07.141440+0000 osd.1 (osd.1) 429 : cluster [DBG] 28.7 scrub starts 2024-02-13T00:57:09.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:07.144021+0000 osd.1 (osd.1) 430 : cluster [DBG] 28.7 scrub ok 2024-02-13T00:57:09.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:07.782851+0000 osd.7 (osd.7) 156 : cluster [DBG] 6.3 deep-scrub starts 2024-02-13T00:57:09.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:07.785893+0000 osd.7 (osd.7) 157 : cluster [DBG] 6.3 deep-scrub ok 2024-02-13T00:57:09.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:07.897371+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:57:09.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:07.900306+0000 osd.2 (osd.2) 220 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:57:09.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:07.141440+0000 osd.1 (osd.1) 429 : cluster [DBG] 28.7 scrub starts 2024-02-13T00:57:09.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:07.144021+0000 osd.1 (osd.1) 430 : cluster [DBG] 28.7 scrub ok 2024-02-13T00:57:09.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:07.782851+0000 osd.7 (osd.7) 156 : cluster [DBG] 6.3 deep-scrub starts 2024-02-13T00:57:09.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:07.785893+0000 osd.7 (osd.7) 157 : cluster [DBG] 6.3 deep-scrub ok 2024-02-13T00:57:09.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:07.897371+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:57:09.081 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:08 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:07.900306+0000 osd.2 (osd.2) 220 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:57:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:07.141440+0000 osd.1 (osd.1) 429 : cluster [DBG] 28.7 scrub starts 2024-02-13T00:57:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:07.144021+0000 osd.1 (osd.1) 430 : cluster [DBG] 28.7 scrub ok 2024-02-13T00:57:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:07.782851+0000 osd.7 (osd.7) 156 : cluster [DBG] 6.3 deep-scrub starts 2024-02-13T00:57:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:07.785893+0000 osd.7 (osd.7) 157 : cluster [DBG] 6.3 deep-scrub ok 2024-02-13T00:57:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:07.897371+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.6 scrub starts 2024-02-13T00:57:09.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:07.900306+0000 osd.2 (osd.2) 220 : cluster [DBG] 5.6 scrub ok 2024-02-13T00:57:10.025 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:08.156164+0000 osd.1 (osd.1) 431 : cluster [DBG] 6.20 scrub starts 2024-02-13T00:57:10.025 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:08.159078+0000 osd.1 (osd.1) 432 : cluster [DBG] 6.20 scrub ok 2024-02-13T00:57:10.025 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:08.602264+0000 mgr.y (mgr.14544) 769 : cluster [DBG] pgmap v1364: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:57:10.025 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:08.691644+0000 osd.3 (osd.3) 247 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:57:10.025 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:08.695160+0000 osd.3 (osd.3) 248 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:57:10.025 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:08.805633+0000 osd.7 (osd.7) 158 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:57:10.025 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:08.806502+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:57:10.026 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:08.894075+0000 osd.2 (osd.2) 221 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:57:10.026 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:08.895195+0000 osd.2 (osd.2) 222 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:57:10.026 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:08.995515+0000 osd.0 (osd.0) 413 : cluster [DBG] 28.1f deep-scrub starts 2024-02-13T00:57:10.026 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:08.999790+0000 osd.0 (osd.0) 414 : cluster [DBG] 28.1f deep-scrub ok 2024-02-13T00:57:10.026 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:09.318088+0000 osd.4 (osd.4) 109 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:57:10.026 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:09.321661+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:57:10.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:08.156164+0000 osd.1 (osd.1) 431 : cluster [DBG] 6.20 scrub starts 2024-02-13T00:57:10.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:08.159078+0000 osd.1 (osd.1) 432 : cluster [DBG] 6.20 scrub ok 2024-02-13T00:57:10.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:08.602264+0000 mgr.y (mgr.14544) 769 : cluster [DBG] pgmap v1364: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:57:10.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:08.691644+0000 osd.3 (osd.3) 247 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:57:10.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:08.695160+0000 osd.3 (osd.3) 248 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:57:10.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:08.805633+0000 osd.7 (osd.7) 158 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:57:10.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:08.806502+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:57:10.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:08.894075+0000 osd.2 (osd.2) 221 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:57:10.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:08.895195+0000 osd.2 (osd.2) 222 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:57:10.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:08.995515+0000 osd.0 (osd.0) 413 : cluster [DBG] 28.1f deep-scrub starts 2024-02-13T00:57:10.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:08.999790+0000 osd.0 (osd.0) 414 : cluster [DBG] 28.1f deep-scrub ok 2024-02-13T00:57:10.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:09.318088+0000 osd.4 (osd.4) 109 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:57:10.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:09 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:09.321661+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:57:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:08.156164+0000 osd.1 (osd.1) 431 : cluster [DBG] 6.20 scrub starts 2024-02-13T00:57:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:08.159078+0000 osd.1 (osd.1) 432 : cluster [DBG] 6.20 scrub ok 2024-02-13T00:57:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:08.602264+0000 mgr.y (mgr.14544) 769 : cluster [DBG] pgmap v1364: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-13T00:57:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:08.691644+0000 osd.3 (osd.3) 247 : cluster [DBG] 5.14 scrub starts 2024-02-13T00:57:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:08.695160+0000 osd.3 (osd.3) 248 : cluster [DBG] 5.14 scrub ok 2024-02-13T00:57:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:08.805633+0000 osd.7 (osd.7) 158 : cluster [DBG] 5.10 scrub starts 2024-02-13T00:57:10.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:08.806502+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.10 scrub ok 2024-02-13T00:57:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:08.894075+0000 osd.2 (osd.2) 221 : cluster [DBG] 6.14 scrub starts 2024-02-13T00:57:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:08.895195+0000 osd.2 (osd.2) 222 : cluster [DBG] 6.14 scrub ok 2024-02-13T00:57:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:08.995515+0000 osd.0 (osd.0) 413 : cluster [DBG] 28.1f deep-scrub starts 2024-02-13T00:57:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:08.999790+0000 osd.0 (osd.0) 414 : cluster [DBG] 28.1f deep-scrub ok 2024-02-13T00:57:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:09.318088+0000 osd.4 (osd.4) 109 : cluster [DBG] 4.1a scrub starts 2024-02-13T00:57:10.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:09.321661+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.1a scrub ok 2024-02-13T00:57:10.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.306+0000 7f833567e700 1 -- 172.21.15.162:0/2952307294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83300fdc20 msgr2=0x7f83300fe000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:10.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.306+0000 7f833567e700 1 --2- 172.21.15.162:0/2952307294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83300fdc20 0x7f83300fe000 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7f8324004870 tx=0x7f8324009cb0).stop 2024-02-13T00:57:10.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.306+0000 7f833567e700 1 -- 172.21.15.162:0/2952307294 shutdown_connections 2024-02-13T00:57:10.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.306+0000 7f833567e700 1 --2- 172.21.15.162:0/2952307294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83300f8490 0x7f8330104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:10.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.306+0000 7f833567e700 1 --2- 172.21.15.162:0/2952307294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83300fdc20 0x7f83300fe000 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:10.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.306+0000 7f833567e700 1 --2- 172.21.15.162:0/2952307294 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83300fe580 0x7f83300f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:10.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.306+0000 7f833567e700 1 -- 172.21.15.162:0/2952307294 >> 172.21.15.162:0/2952307294 conn(0x7f83300f3850 msgr2=0x7f83300f5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.306+0000 7f833567e700 1 -- 172.21.15.162:0/2952307294 shutdown_connections 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.306+0000 7f833567e700 1 -- 172.21.15.162:0/2952307294 wait complete. 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f833567e700 1 librados: starting msgr at 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f833567e700 1 librados: starting objecter 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f833567e700 1 Processor -- start 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f833567e700 1 -- start start 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f833567e700 1 librados: setting wanted keys 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f833567e700 1 librados: calling monclient init 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f833567e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83300f8490 0x7f833006b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f833567e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83300fdc20 0x7f833006b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f833567e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83300fe580 0x7f833006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f833567e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8330116610 con 0x7f83300fdc20 2024-02-13T00:57:10.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f833567e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8330116780 con 0x7f83300fe580 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f833567e700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8330116940 con 0x7f83300f8490 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f832effd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83300f8490 0x7f833006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f832e7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83300fdc20 0x7f833006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f832effd700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83300f8490 0x7f833006b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:38282/0 (socket says 172.21.15.162:38282) 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f832effd700 1 -- 172.21.15.162:0/3892055198 learned_addr learned my addr 172.21.15.162:0/3892055198 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f832f7fe700 1 --2- 172.21.15.162:0/3892055198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83300fe580 0x7f833006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f832effd700 1 -- 172.21.15.162:0/3892055198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83300fe580 msgr2=0x7f833006bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f832effd700 1 --2- 172.21.15.162:0/3892055198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83300fe580 0x7f833006bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f832effd700 1 -- 172.21.15.162:0/3892055198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83300fdc20 msgr2=0x7f833006b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f832effd700 1 --2- 172.21.15.162:0/3892055198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83300fdc20 0x7f833006b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f832effd700 1 -- 172.21.15.162:0/3892055198 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8320005040 con 0x7f83300f8490 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.307+0000 7f832e7fc700 1 --2- 172.21.15.162:0/3892055198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83300fdc20 0x7f833006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.308+0000 7f832effd700 1 --2- 172.21.15.162:0/3892055198 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83300f8490 0x7f833006b170 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 rx=0x7f8324009eb0 tx=0x7f832400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:10.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.308+0000 7f8317fff700 10 client.?.objecter ms_handle_connect 0x7f83300f8490 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.308+0000 7f8317fff700 10 client.?.objecter resend_mon_ops 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.308+0000 7f8317fff700 1 -- 172.21.15.162:0/3892055198 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832400e070 con 0x7f83300f8490 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.308+0000 7f8317fff700 1 -- 172.21.15.162:0/3892055198 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8324013da0 con 0x7f83300f8490 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.308+0000 7f833567e700 1 -- 172.21.15.162:0/3892055198 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8324005040 con 0x7f83300f8490 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.308+0000 7f8317fff700 1 -- 172.21.15.162:0/3892055198 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8324019070 con 0x7f83300f8490 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.308+0000 7f833567e700 10 client.38552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.308+0000 7f833567e700 1 -- 172.21.15.162:0/3892055198 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8330116f10 con 0x7f83300f8490 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.308+0000 7f833567e700 1 librados: init done 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.309+0000 7f8317fff700 1 -- 172.21.15.162:0/3892055198 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f832401e420 con 0x7f83300f8490 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.309+0000 7f833567e700 1 -- 172.21.15.162:0/3892055198 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83300607b0 con 0x7f83300f8490 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.309+0000 7f8317fff700 1 --2- 172.21.15.162:0/3892055198 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8318062c90 0x7f8318065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.310+0000 7f8317fff700 1 -- 172.21.15.162:0/3892055198 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(858..858 src has 449..858) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f832407db80 con 0x7f83300f8490 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.310+0000 7f8317fff700 10 client.38552.objecter ms_dispatch 0x7f8330001550 osd_map(858..858 src has 449..858) v4 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.310+0000 7f8317fff700 3 client.38552.objecter handle_osd_map got epochs [858,858] > 0 2024-02-13T00:57:10.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.310+0000 7f8317fff700 3 client.38552.objecter handle_osd_map decoding full epoch 858 2024-02-13T00:57:10.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.310+0000 7f832e7fc700 1 --2- 172.21.15.162:0/3892055198 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8318062c90 0x7f8318065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:10.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.310+0000 7f8317fff700 20 client.38552.objecter dump_active .. 0 homeless 2024-02-13T00:57:10.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.310+0000 7f8317fff700 10 client.38552.objecter ms_handle_connect 0x7f8318062c90 2024-02-13T00:57:10.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.310+0000 7f832e7fc700 1 --2- 172.21.15.162:0/3892055198 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8318062c90 0x7f8318065150 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 rx=0x7f833006c910 tx=0x7f832000b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:10.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.314+0000 7f8317fff700 1 -- 172.21.15.162:0/3892055198 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f83240512d0 con 0x7f83300f8490 2024-02-13T00:57:10.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.476+0000 7f833567e700 1 -- 172.21.15.162:0/3892055198 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f833006c650 con 0x7f83300f8490 2024-02-13T00:57:10.480 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.476+0000 7f8317fff700 1 -- 172.21.15.162:0/3892055198 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8324023020 con 0x7f83300f8490 2024-02-13T00:57:10.480 INFO:teuthology.orchestra.run.smithi162.stdout:227633266924 2024-02-13T00:57:10.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 10 librados: watch_flush enter 2024-02-13T00:57:10.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 10 librados: watch_flush exit 2024-02-13T00:57:10.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 20 client.38552.objecter shutdown clearing up homeless session... 2024-02-13T00:57:10.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 10 client.38552.objecter successfully canceled tick 2024-02-13T00:57:10.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 -- 172.21.15.162:0/3892055198 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8318062c90 msgr2=0x7f8318065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:10.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 --2- 172.21.15.162:0/3892055198 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8318062c90 0x7f8318065150 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 rx=0x7f833006c910 tx=0x7f832000b040).stop 2024-02-13T00:57:10.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 -- 172.21.15.162:0/3892055198 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83300f8490 msgr2=0x7f833006b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:10.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 --2- 172.21.15.162:0/3892055198 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83300f8490 0x7f833006b170 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 rx=0x7f8324009eb0 tx=0x7f832400b040).stop 2024-02-13T00:57:10.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 -- 172.21.15.162:0/3892055198 shutdown_connections 2024-02-13T00:57:10.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 --2- 172.21.15.162:0/3892055198 >> v2:172.21.15.162:6800/2866534053 conn(0x7f8318062c90 0x7f8318065150 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:10.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 --2- 172.21.15.162:0/3892055198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83300fe580 0x7f833006bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:10.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 --2- 172.21.15.162:0/3892055198 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83300f8490 0x7f833006b170 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:10.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 --2- 172.21.15.162:0/3892055198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83300fdc20 0x7f833006b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:10.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 -- 172.21.15.162:0/3892055198 >> 172.21.15.162:0/3892055198 conn(0x7f83300f3850 msgr2=0x7f83300fb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:10.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 -- 172.21.15.162:0/3892055198 shutdown_connections 2024-02-13T00:57:10.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 -- 172.21.15.162:0/3892055198 wait complete. 2024-02-13T00:57:10.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:10.478+0000 7f833567e700 1 librados: shutdown 2024-02-13T00:57:11.201 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:09.623564+0000 osd.0 (osd.0) 415 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:57:11.201 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:09.623756+0000 osd.0 (osd.0) 416 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:57:11.201 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:09.957356+0000 mon.a (mon.0) 3883 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2024-02-13T00:57:11.201 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:10.477493+0000 mon.c (mon.2) 545 : audit [DBG] from='client.? 172.21.15.162:0/3892055198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-13T00:57:11.201 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:10.639649+0000 mon.a (mon.0) 3884 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2024-02-13T00:57:11.201 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:09.623564+0000 osd.0 (osd.0) 415 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:57:11.202 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:09.623756+0000 osd.0 (osd.0) 416 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:57:11.202 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:09.957356+0000 mon.a (mon.0) 3883 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2024-02-13T00:57:11.202 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:10.477493+0000 mon.c (mon.2) 545 : audit [DBG] from='client.? 172.21.15.162:0/3892055198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-13T00:57:11.202 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:10.639649+0000 mon.a (mon.0) 3884 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2024-02-13T00:57:11.222 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266919 got 227633266924 for osd.7 2024-02-13T00:57:11.222 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-13T00:57:11.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:57:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:09.623564+0000 osd.0 (osd.0) 415 : cluster [DBG] purged_snaps scrub starts 2024-02-13T00:57:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:09.623756+0000 osd.0 (osd.0) 416 : cluster [DBG] purged_snaps scrub ok 2024-02-13T00:57:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:09.957356+0000 mon.a (mon.0) 3883 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2024-02-13T00:57:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:10.477493+0000 mon.c (mon.2) 545 : audit [DBG] from='client.? 172.21.15.162:0/3892055198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-13T00:57:11.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:10.639649+0000 mon.a (mon.0) 3884 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2024-02-13T00:57:11.494 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:57:12.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:10.154860+0000 osd.1 (osd.1) 433 : cluster [DBG] 5.c scrub starts 2024-02-13T00:57:12.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:10.157831+0000 osd.1 (osd.1) 434 : cluster [DBG] 5.c scrub ok 2024-02-13T00:57:12.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:10.603045+0000 mgr.y (mgr.14544) 770 : cluster [DBG] pgmap v1366: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:57:12.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:10.808470+0000 osd.7 (osd.7) 160 : cluster [DBG] 2.b scrub starts 2024-02-13T00:57:12.031 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:10.812609+0000 osd.7 (osd.7) 161 : cluster [DBG] 2.b scrub ok 2024-02-13T00:57:12.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:10.154860+0000 osd.1 (osd.1) 433 : cluster [DBG] 5.c scrub starts 2024-02-13T00:57:12.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:10.157831+0000 osd.1 (osd.1) 434 : cluster [DBG] 5.c scrub ok 2024-02-13T00:57:12.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:10.603045+0000 mgr.y (mgr.14544) 770 : cluster [DBG] pgmap v1366: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:57:12.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:10.808470+0000 osd.7 (osd.7) 160 : cluster [DBG] 2.b scrub starts 2024-02-13T00:57:12.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:11 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:10.812609+0000 osd.7 (osd.7) 161 : cluster [DBG] 2.b scrub ok 2024-02-13T00:57:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:10.154860+0000 osd.1 (osd.1) 433 : cluster [DBG] 5.c scrub starts 2024-02-13T00:57:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:10.157831+0000 osd.1 (osd.1) 434 : cluster [DBG] 5.c scrub ok 2024-02-13T00:57:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:10.603045+0000 mgr.y (mgr.14544) 770 : cluster [DBG] pgmap v1366: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-13T00:57:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:10.808470+0000 osd.7 (osd.7) 160 : cluster [DBG] 2.b scrub starts 2024-02-13T00:57:12.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:10.812609+0000 osd.7 (osd.7) 161 : cluster [DBG] 2.b scrub ok 2024-02-13T00:57:13.134 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:11.132557+0000 osd.1 (osd.1) 435 : cluster [DBG] 3.0 scrub starts 2024-02-13T00:57:13.135 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:11.136151+0000 osd.1 (osd.1) 436 : cluster [DBG] 3.0 scrub ok 2024-02-13T00:57:13.135 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:11.659912+0000 osd.3 (osd.3) 249 : cluster [DBG] 4.b scrub starts 2024-02-13T00:57:13.135 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:11.663422+0000 osd.3 (osd.3) 250 : cluster [DBG] 4.b scrub ok 2024-02-13T00:57:13.135 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:11.819596+0000 osd.6 (osd.6) 358 : cluster [DBG] 28.b scrub starts 2024-02-13T00:57:13.135 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:11.823288+0000 osd.6 (osd.6) 359 : cluster [DBG] 28.b scrub ok 2024-02-13T00:57:13.135 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:11.995098+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.d scrub starts 2024-02-13T00:57:13.135 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:11.996857+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.d scrub ok 2024-02-13T00:57:13.135 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:11.132557+0000 osd.1 (osd.1) 435 : cluster [DBG] 3.0 scrub starts 2024-02-13T00:57:13.135 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:11.136151+0000 osd.1 (osd.1) 436 : cluster [DBG] 3.0 scrub ok 2024-02-13T00:57:13.135 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:11.659912+0000 osd.3 (osd.3) 249 : cluster [DBG] 4.b scrub starts 2024-02-13T00:57:13.136 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:11.663422+0000 osd.3 (osd.3) 250 : cluster [DBG] 4.b scrub ok 2024-02-13T00:57:13.136 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:11.819596+0000 osd.6 (osd.6) 358 : cluster [DBG] 28.b scrub starts 2024-02-13T00:57:13.136 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:11.823288+0000 osd.6 (osd.6) 359 : cluster [DBG] 28.b scrub ok 2024-02-13T00:57:13.136 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:11.995098+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.d scrub starts 2024-02-13T00:57:13.136 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:12 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:11.996857+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.d scrub ok 2024-02-13T00:57:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:11.132557+0000 osd.1 (osd.1) 435 : cluster [DBG] 3.0 scrub starts 2024-02-13T00:57:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:11.136151+0000 osd.1 (osd.1) 436 : cluster [DBG] 3.0 scrub ok 2024-02-13T00:57:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:11.659912+0000 osd.3 (osd.3) 249 : cluster [DBG] 4.b scrub starts 2024-02-13T00:57:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:11.663422+0000 osd.3 (osd.3) 250 : cluster [DBG] 4.b scrub ok 2024-02-13T00:57:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:11.819596+0000 osd.6 (osd.6) 358 : cluster [DBG] 28.b scrub starts 2024-02-13T00:57:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:11.823288+0000 osd.6 (osd.6) 359 : cluster [DBG] 28.b scrub ok 2024-02-13T00:57:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:11.995098+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.d scrub starts 2024-02-13T00:57:13.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:11.996857+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.d scrub ok 2024-02-13T00:57:13.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.423+0000 7fe5e4371700 1 -- 172.21.15.162:0/866467800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5dc0fa440 msgr2=0x7fe5dc0f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:13.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.423+0000 7fe5e4371700 1 --2- 172.21.15.162:0/866467800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5dc0fa440 0x7fe5dc0f7e20 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 rx=0x7fe5d0004870 tx=0x7fe5d0009cb0).stop 2024-02-13T00:57:13.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 -- 172.21.15.162:0/866467800 shutdown_connections 2024-02-13T00:57:13.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 --2- 172.21.15.162:0/866467800 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5dc0f8500 0x7fe5dc104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:13.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 --2- 172.21.15.162:0/866467800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5dc0f9b70 0x7fe5dc0f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:13.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 --2- 172.21.15.162:0/866467800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5dc0fa440 0x7fe5dc0f7e20 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:13.441 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 -- 172.21.15.162:0/866467800 >> 172.21.15.162:0/866467800 conn(0x7fe5dc0f38c0 msgr2=0x7fe5dc0f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:13.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 -- 172.21.15.162:0/866467800 shutdown_connections 2024-02-13T00:57:13.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 -- 172.21.15.162:0/866467800 wait complete. 2024-02-13T00:57:13.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 librados: starting msgr at 2024-02-13T00:57:13.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 librados: starting objecter 2024-02-13T00:57:13.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 Processor -- start 2024-02-13T00:57:13.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 -- start start 2024-02-13T00:57:13.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 librados: setting wanted keys 2024-02-13T00:57:13.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 librados: calling monclient init 2024-02-13T00:57:13.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5dc0f8500 0x7fe5dc10e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:13.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5dc0f9b70 0x7fe5dc10e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:13.442 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5dc0fa440 0x7fe5dc108430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc108920 con 0x7fe5dc0f8500 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc108a90 con 0x7fe5dc0f9b70 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e4371700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc108c50 con 0x7fe5dc0fa440 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e210d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5dc0f8500 0x7fe5dc10e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.424+0000 7fe5e290e700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5dc0fa440 0x7fe5dc108430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e210d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5dc0f8500 0x7fe5dc10e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:56700/0 (socket says 172.21.15.162:56700) 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e210d700 1 -- 172.21.15.162:0/3510208893 learned_addr learned my addr 172.21.15.162:0/3510208893 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e190c700 1 --2- 172.21.15.162:0/3510208893 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5dc0f9b70 0x7fe5dc10e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e290e700 1 -- 172.21.15.162:0/3510208893 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5dc0f9b70 msgr2=0x7fe5dc10e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e290e700 1 --2- 172.21.15.162:0/3510208893 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5dc0f9b70 0x7fe5dc10e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e290e700 1 -- 172.21.15.162:0/3510208893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5dc0f8500 msgr2=0x7fe5dc10e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e290e700 1 --2- 172.21.15.162:0/3510208893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5dc0f8500 0x7fe5dc10e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e290e700 1 -- 172.21.15.162:0/3510208893 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d40097c0 con 0x7fe5dc0fa440 2024-02-13T00:57:13.443 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e210d700 1 --2- 172.21.15.162:0/3510208893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5dc0f8500 0x7fe5dc10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:57:13.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e290e700 1 --2- 172.21.15.162:0/3510208893 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5dc0fa440 0x7fe5dc108430 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 rx=0x7fe5d4001cb0 tx=0x7fe5d4004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:13.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5cf7fe700 10 client.?.objecter ms_handle_connect 0x7fe5dc0fa440 2024-02-13T00:57:13.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5cf7fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:57:13.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5cf7fe700 1 -- 172.21.15.162:0/3510208893 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d40079b0 con 0x7fe5dc0fa440 2024-02-13T00:57:13.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5cf7fe700 1 -- 172.21.15.162:0/3510208893 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5d40090d0 con 0x7fe5dc0fa440 2024-02-13T00:57:13.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e4371700 1 -- 172.21.15.162:0/3510208893 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d0005040 con 0x7fe5dc0fa440 2024-02-13T00:57:13.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5cf7fe700 1 -- 172.21.15.162:0/3510208893 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d400fc20 con 0x7fe5dc0fa440 2024-02-13T00:57:13.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e4371700 10 client.38558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:57:13.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e4371700 1 -- 172.21.15.162:0/3510208893 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5dc109290 con 0x7fe5dc0fa440 2024-02-13T00:57:13.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.425+0000 7fe5e4371700 1 librados: init done 2024-02-13T00:57:13.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.426+0000 7fe5e4371700 1 -- 172.21.15.162:0/3510208893 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5dc0475e0 con 0x7fe5dc0fa440 2024-02-13T00:57:13.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.426+0000 7fe5cf7fe700 1 -- 172.21.15.162:0/3510208893 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fe5d400fd80 con 0x7fe5dc0fa440 2024-02-13T00:57:13.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.427+0000 7fe5cf7fe700 1 --2- 172.21.15.162:0/3510208893 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe5c8062c90 0x7fe5c8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:13.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.427+0000 7fe5cf7fe700 1 -- 172.21.15.162:0/3510208893 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(859..859 src has 449..859) v4 ==== 7509+0+0 (secure 0 0 0) 0x7fe5d407d780 con 0x7fe5dc0fa440 2024-02-13T00:57:13.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.427+0000 7fe5cf7fe700 10 client.38558.objecter ms_dispatch 0x7fe5dc001550 osd_map(859..859 src has 449..859) v4 2024-02-13T00:57:13.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.427+0000 7fe5cf7fe700 3 client.38558.objecter handle_osd_map got epochs [859,859] > 0 2024-02-13T00:57:13.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.427+0000 7fe5cf7fe700 3 client.38558.objecter handle_osd_map decoding full epoch 859 2024-02-13T00:57:13.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.427+0000 7fe5e210d700 1 --2- 172.21.15.162:0/3510208893 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe5c8062c90 0x7fe5c8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:13.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.427+0000 7fe5cf7fe700 20 client.38558.objecter dump_active .. 0 homeless 2024-02-13T00:57:13.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.427+0000 7fe5cf7fe700 10 client.38558.objecter ms_handle_connect 0x7fe5c8062c90 2024-02-13T00:57:13.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.427+0000 7fe5e210d700 1 --2- 172.21.15.162:0/3510208893 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe5c8062c90 0x7fe5c8065150 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 rx=0x7fe5d8004fd0 tx=0x7fe5d800b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:13.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.431+0000 7fe5cf7fe700 1 -- 172.21.15.162:0/3510208893 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe5d4050eb0 con 0x7fe5dc0fa440 2024-02-13T00:57:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.593+0000 7fe5e4371700 1 -- 172.21.15.162:0/3510208893 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe5dc1098c0 con 0x7fe5c8062c90 2024-02-13T00:57:13.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.605+0000 7fe5cf7fe700 1 -- 172.21.15.162:0/3510208893 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+432231 (secure 0 0 0) 0x7fe5dc1098c0 con 0x7fe5c8062c90 2024-02-13T00:57:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:57:13.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 10 librados: watch_flush enter 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 10 librados: watch_flush exit 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 20 client.38558.objecter shutdown clearing up homeless session... 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 10 client.38558.objecter successfully canceled tick 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 -- 172.21.15.162:0/3510208893 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe5c8062c90 msgr2=0x7fe5c8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 --2- 172.21.15.162:0/3510208893 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe5c8062c90 0x7fe5c8065150 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 rx=0x7fe5d8004fd0 tx=0x7fe5d800b040).stop 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 -- 172.21.15.162:0/3510208893 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5dc0fa440 msgr2=0x7fe5dc108430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 --2- 172.21.15.162:0/3510208893 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5dc0fa440 0x7fe5dc108430 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 rx=0x7fe5d4001cb0 tx=0x7fe5d4004c50).stop 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 -- 172.21.15.162:0/3510208893 shutdown_connections 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 --2- 172.21.15.162:0/3510208893 >> v2:172.21.15.162:6800/2866534053 conn(0x7fe5c8062c90 0x7fe5c8065150 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 --2- 172.21.15.162:0/3510208893 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5dc0fa440 0x7fe5dc108430 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 --2- 172.21.15.162:0/3510208893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5dc0f8500 0x7fe5dc10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 --2- 172.21.15.162:0/3510208893 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5dc0f9b70 0x7fe5dc10e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 -- 172.21.15.162:0/3510208893 >> 172.21.15.162:0/3510208893 conn(0x7fe5dc0f38c0 msgr2=0x7fe5dc0f4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 -- 172.21.15.162:0/3510208893 shutdown_connections 2024-02-13T00:57:13.619 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 -- 172.21.15.162:0/3510208893 wait complete. 2024-02-13T00:57:13.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:13.608+0000 7fe5e4371700 1 librados: shutdown 2024-02-13T00:57:13.620 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:57:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:12.603425+0000 mgr.y (mgr.14544) 771 : cluster [DBG] pgmap v1368: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:57:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:12.813006+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.a scrub starts 2024-02-13T00:57:14.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:12.815889+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.a scrub ok 2024-02-13T00:57:14.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:12.603425+0000 mgr.y (mgr.14544) 771 : cluster [DBG] pgmap v1368: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:57:14.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:12.813006+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.a scrub starts 2024-02-13T00:57:14.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:12.815889+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.a scrub ok 2024-02-13T00:57:14.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:12.603425+0000 mgr.y (mgr.14544) 771 : cluster [DBG] pgmap v1368: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:57:14.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:12.813006+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.a scrub starts 2024-02-13T00:57:14.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:13 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:12.815889+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.a scrub ok 2024-02-13T00:57:14.473 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":1368,"stamp":"2024-02-13T00:57:12.603229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":2651,"num_read_kb":2371,"num_write":913,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":23794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1148,"ondisk_log_size":1148,"up":492,"acting":492,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2466500,"kb_used_data":54788,"kb_used_omap":42,"kb_used_meta":2411605,"kb_avail":747494716,"statfs":{"total":767960285184,"available":765434589184,"internally_reserved":0,"allocated":56102912,"data_stored":29880146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43373,"internal_metadata":2469484179},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002419"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":606,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797747+0000","last_change":"2024-02-13T00:56:33.157883+0000","last_active":"2024-02-13T00:56:55.797747+0000","last_peered":"2024-02-13T00:56:55.797747+0000","last_clean":"2024-02-13T00:56:55.797747+0000","last_became_active":"2024-02-13T00:56:07.836922+0000","last_became_peered":"2024-02-13T00:56:07.836922+0000","last_unstale":"2024-02-13T00:56:55.797747+0000","last_undegraded":"2024-02-13T00:56:55.797747+0000","last_fullsized":"2024-02-13T00:56:55.797747+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:33.157756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:56:33.157756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":57,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.795585+0000","last_change":"2024-02-13T00:56:49.841938+0000","last_active":"2024-02-13T00:56:55.795585+0000","last_peered":"2024-02-13T00:56:55.795585+0000","last_clean":"2024-02-13T00:56:55.795585+0000","last_became_active":"2024-02-13T00:55:04.781199+0000","last_became_peered":"2024-02-13T00:55:04.781199+0000","last_unstale":"2024-02-13T00:56:55.795585+0000","last_undegraded":"2024-02-13T00:56:55.795585+0000","last_fullsized":"2024-02-13T00:56:55.795585+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:49.841870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:39.671163+0000","last_clean_scrub_stamp":"2024-02-13T00:56:49.841870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":559,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798227+0000","last_change":"2024-02-13T00:56:44.994751+0000","last_active":"2024-02-13T00:56:55.798227+0000","last_peered":"2024-02-13T00:56:55.798227+0000","last_clean":"2024-02-13T00:56:55.798227+0000","last_became_active":"2024-02-13T00:56:07.653631+0000","last_became_peered":"2024-02-13T00:56:07.653631+0000","last_unstale":"2024-02-13T00:56:55.798227+0000","last_undegraded":"2024-02-13T00:56:55.798227+0000","last_fullsized":"2024-02-13T00:56:55.798227+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:44.994674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:44.994674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":661,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:08.159194+0000","last_change":"2024-02-13T00:57:08.159171+0000","last_active":"2024-02-13T00:57:08.159194+0000","last_peered":"2024-02-13T00:57:08.159194+0000","last_clean":"2024-02-13T00:57:08.159194+0000","last_became_active":"2024-02-13T00:55:04.796268+0000","last_became_peered":"2024-02-13T00:55:04.796268+0000","last_unstale":"2024-02-13T00:57:08.159194+0000","last_undegraded":"2024-02-13T00:57:08.159194+0000","last_fullsized":"2024-02-13T00:57:08.159194+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:08.159082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:57:08.159082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":572,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797269+0000","last_change":"2024-02-13T00:56:37.183946+0000","last_active":"2024-02-13T00:56:55.797269+0000","last_peered":"2024-02-13T00:56:55.797269+0000","last_clean":"2024-02-13T00:56:55.797269+0000","last_became_active":"2024-02-13T00:54:56.636553+0000","last_became_peered":"2024-02-13T00:54:56.636553+0000","last_unstale":"2024-02-13T00:56:55.797269+0000","last_undegraded":"2024-02-13T00:56:55.797269+0000","last_fullsized":"2024-02-13T00:56:55.797269+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:37.183853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:37.183853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":164,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796490+0000","last_change":"2024-02-13T00:56:21.181975+0000","last_active":"2024-02-13T00:56:55.796490+0000","last_peered":"2024-02-13T00:56:55.796490+0000","last_clean":"2024-02-13T00:56:55.796490+0000","last_became_active":"2024-02-13T00:55:00.638229+0000","last_became_peered":"2024-02-13T00:55:00.638229+0000","last_unstale":"2024-02-13T00:56:55.796490+0000","last_undegraded":"2024-02-13T00:56:55.796490+0000","last_fullsized":"2024-02-13T00:56:55.796490+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:21.181886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:21.181886+0000","last_clean_scrub_stamp":"2024-02-13T00:56:21.181886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":576,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797491+0000","last_change":"2024-02-13T00:55:44.703735+0000","last_active":"2024-02-13T00:56:55.797491+0000","last_peered":"2024-02-13T00:56:55.797491+0000","last_clean":"2024-02-13T00:56:55.797491+0000","last_became_active":"2024-02-13T00:55:04.793521+0000","last_became_peered":"2024-02-13T00:55:04.793521+0000","last_unstale":"2024-02-13T00:56:55.797491+0000","last_undegraded":"2024-02-13T00:56:55.797491+0000","last_fullsized":"2024-02-13T00:56:55.797491+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:44.703665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:55:44.703665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":77,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.879815+0000","last_change":"2024-02-13T00:56:07.223483+0000","last_active":"2024-02-13T00:56:51.879815+0000","last_peered":"2024-02-13T00:56:51.879815+0000","last_clean":"2024-02-13T00:56:51.879815+0000","last_became_active":"2024-02-13T00:55:04.789308+0000","last_became_peered":"2024-02-13T00:55:04.789308+0000","last_unstale":"2024-02-13T00:56:51.879815+0000","last_undegraded":"2024-02-13T00:56:51.879815+0000","last_fullsized":"2024-02-13T00:56:51.879815+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:07.223400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:56:07.223400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":545,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.799581+0000","last_change":"2024-02-13T00:56:36.215066+0000","last_active":"2024-02-13T00:56:55.799581+0000","last_peered":"2024-02-13T00:56:55.799581+0000","last_clean":"2024-02-13T00:56:55.799581+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:56:55.799581+0000","last_undegraded":"2024-02-13T00:56:55.799581+0000","last_fullsized":"2024-02-13T00:56:55.799581+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:36.214957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:56:36.214957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":652,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:00.129035+0000","last_change":"2024-02-13T00:57:00.129012+0000","last_active":"2024-02-13T00:57:00.129035+0000","last_peered":"2024-02-13T00:57:00.129035+0000","last_clean":"2024-02-13T00:57:00.129035+0000","last_became_active":"2024-02-13T00:54:52.736324+0000","last_became_peered":"2024-02-13T00:54:52.736324+0000","last_unstale":"2024-02-13T00:57:00.129035+0000","last_undegraded":"2024-02-13T00:57:00.129035+0000","last_fullsized":"2024-02-13T00:57:00.129035+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:00.128944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:57:00.128944+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":775,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796306+0000","last_change":"2024-02-13T00:56:51.232178+0000","last_active":"2024-02-13T00:56:55.796306+0000","last_peered":"2024-02-13T00:56:55.796306+0000","last_clean":"2024-02-13T00:56:55.796306+0000","last_became_active":"2024-02-13T00:55:51.633619+0000","last_became_peered":"2024-02-13T00:55:51.633619+0000","last_unstale":"2024-02-13T00:56:55.796306+0000","last_undegraded":"2024-02-13T00:56:55.796306+0000","last_fullsized":"2024-02-13T00:56:55.796306+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:51.232094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:51.232094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"849'26","reported_seq":531,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.432806+0000","last_change":"2024-02-13T00:56:07.653898+0000","last_active":"2024-02-13T00:57:09.432806+0000","last_peered":"2024-02-13T00:57:09.432806+0000","last_clean":"2024-02-13T00:57:09.432806+0000","last_became_active":"2024-02-13T00:56:07.653175+0000","last_became_peered":"2024-02-13T00:56:07.653175+0000","last_unstale":"2024-02-13T00:57:09.432806+0000","last_undegraded":"2024-02-13T00:57:09.432806+0000","last_fullsized":"2024-02-13T00:57:09.432806+0000","mapping_epoch":848,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"820'25","last_scrub_stamp":"2024-02-13T00:56:02.921047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:56:02.921047+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":79,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.879303+0000","last_change":"2024-02-13T00:56:11.198869+0000","last_active":"2024-02-13T00:56:51.879303+0000","last_peered":"2024-02-13T00:56:51.879303+0000","last_clean":"2024-02-13T00:56:51.879303+0000","last_became_active":"2024-02-13T00:54:52.742726+0000","last_became_peered":"2024-02-13T00:54:52.742726+0000","last_unstale":"2024-02-13T00:56:51.879303+0000","last_undegraded":"2024-02-13T00:56:51.879303+0000","last_fullsized":"2024-02-13T00:56:51.879303+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'48","last_scrub_stamp":"2024-02-13T00:56:11.198787+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:56:11.198787+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":52,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796429+0000","last_change":"2024-02-13T00:56:15.785316+0000","last_active":"2024-02-13T00:56:55.796429+0000","last_peered":"2024-02-13T00:56:55.796429+0000","last_clean":"2024-02-13T00:56:55.796429+0000","last_became_active":"2024-02-13T00:55:04.799211+0000","last_became_peered":"2024-02-13T00:55:04.799211+0000","last_unstale":"2024-02-13T00:56:55.796429+0000","last_undegraded":"2024-02-13T00:56:55.796429+0000","last_fullsized":"2024-02-13T00:56:55.796429+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:15.785248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:15.785248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":59,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.879514+0000","last_change":"2024-02-13T00:56:05.252369+0000","last_active":"2024-02-13T00:56:51.879514+0000","last_peered":"2024-02-13T00:56:51.879514+0000","last_clean":"2024-02-13T00:56:51.879514+0000","last_became_active":"2024-02-13T00:54:52.751474+0000","last_became_peered":"2024-02-13T00:54:52.751474+0000","last_unstale":"2024-02-13T00:56:51.879514+0000","last_undegraded":"2024-02-13T00:56:51.879514+0000","last_fullsized":"2024-02-13T00:56:51.879514+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:05.252294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:05.252294+0000","last_clean_scrub_stamp":"2024-02-13T00:56:05.252294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":52,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.795932+0000","last_change":"2024-02-13T00:56:01.710378+0000","last_active":"2024-02-13T00:56:55.795932+0000","last_peered":"2024-02-13T00:56:55.795932+0000","last_clean":"2024-02-13T00:56:55.795932+0000","last_became_active":"2024-02-13T00:55:04.790490+0000","last_became_peered":"2024-02-13T00:55:04.790490+0000","last_unstale":"2024-02-13T00:56:55.795932+0000","last_undegraded":"2024-02-13T00:56:55.795932+0000","last_fullsized":"2024-02-13T00:56:55.795932+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:01.710308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:01.710308+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":76,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.637029+0000","last_change":"2024-02-13T00:55:52.590347+0000","last_active":"2024-02-13T00:56:55.637029+0000","last_peered":"2024-02-13T00:56:55.637029+0000","last_clean":"2024-02-13T00:56:55.637029+0000","last_became_active":"2024-02-13T00:55:00.638928+0000","last_became_peered":"2024-02-13T00:55:00.638928+0000","last_unstale":"2024-02-13T00:56:55.637029+0000","last_undegraded":"2024-02-13T00:56:55.637029+0000","last_fullsized":"2024-02-13T00:56:55.637029+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'21","last_scrub_stamp":"2024-02-13T00:55:52.590270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:55:52.590270+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":871,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.635459+0000","last_change":"2024-02-13T00:55:51.966565+0000","last_active":"2024-02-13T00:56:55.635459+0000","last_peered":"2024-02-13T00:56:55.635459+0000","last_clean":"2024-02-13T00:56:55.635459+0000","last_became_active":"2024-02-13T00:55:00.635786+0000","last_became_peered":"2024-02-13T00:55:00.635786+0000","last_unstale":"2024-02-13T00:56:55.635459+0000","last_undegraded":"2024-02-13T00:56:55.635459+0000","last_fullsized":"2024-02-13T00:56:55.635459+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:51.966488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:55:51.966488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":57,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.795930+0000","last_change":"2024-02-13T00:56:19.819857+0000","last_active":"2024-02-13T00:56:55.795930+0000","last_peered":"2024-02-13T00:56:55.795930+0000","last_clean":"2024-02-13T00:56:55.795930+0000","last_became_active":"2024-02-13T00:55:04.801366+0000","last_became_peered":"2024-02-13T00:55:04.801366+0000","last_unstale":"2024-02-13T00:56:55.795930+0000","last_undegraded":"2024-02-13T00:56:55.795930+0000","last_fullsized":"2024-02-13T00:56:55.795930+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:19.819795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:56:19.819795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":870,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796324+0000","last_change":"2024-02-13T00:56:03.986502+0000","last_active":"2024-02-13T00:56:55.796324+0000","last_peered":"2024-02-13T00:56:55.796324+0000","last_clean":"2024-02-13T00:56:55.796324+0000","last_became_active":"2024-02-13T00:55:04.815877+0000","last_became_peered":"2024-02-13T00:55:04.815877+0000","last_unstale":"2024-02-13T00:56:55.796324+0000","last_undegraded":"2024-02-13T00:56:55.796324+0000","last_fullsized":"2024-02-13T00:56:55.796324+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:03.986440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:54:47.261547+0000","last_clean_scrub_stamp":"2024-02-13T00:56:03.986440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":496,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.795840+0000","last_change":"2024-02-13T00:56:33.741869+0000","last_active":"2024-02-13T00:56:55.795840+0000","last_peered":"2024-02-13T00:56:55.795840+0000","last_clean":"2024-02-13T00:56:55.795840+0000","last_became_active":"2024-02-13T00:55:00.640016+0000","last_became_peered":"2024-02-13T00:55:00.640016+0000","last_unstale":"2024-02-13T00:56:55.795840+0000","last_undegraded":"2024-02-13T00:56:55.795840+0000","last_fullsized":"2024-02-13T00:56:55.795840+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:33.741779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:33.741779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":684,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:04.940413+0000","last_change":"2024-02-13T00:57:04.940383+0000","last_active":"2024-02-13T00:57:04.940413+0000","last_peered":"2024-02-13T00:57:04.940413+0000","last_clean":"2024-02-13T00:57:04.940413+0000","last_became_active":"2024-02-13T00:51:41.573050+0000","last_became_peered":"2024-02-13T00:51:41.573050+0000","last_unstale":"2024-02-13T00:57:04.940413+0000","last_undegraded":"2024-02-13T00:57:04.940413+0000","last_fullsized":"2024-02-13T00:57:04.940413+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'51","last_scrub_stamp":"2024-02-13T00:57:04.940306+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:57:04.940306+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":57,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796255+0000","last_change":"2024-02-13T00:56:48.850014+0000","last_active":"2024-02-13T00:56:55.796255+0000","last_peered":"2024-02-13T00:56:55.796255+0000","last_clean":"2024-02-13T00:56:55.796255+0000","last_became_active":"2024-02-13T00:55:04.808497+0000","last_became_peered":"2024-02-13T00:55:04.808497+0000","last_unstale":"2024-02-13T00:56:55.796255+0000","last_undegraded":"2024-02-13T00:56:55.796255+0000","last_fullsized":"2024-02-13T00:56:55.796255+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:48.849919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:48.849919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":57,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796076+0000","last_change":"2024-02-13T00:56:43.888878+0000","last_active":"2024-02-13T00:56:55.796076+0000","last_peered":"2024-02-13T00:56:55.796076+0000","last_clean":"2024-02-13T00:56:55.796076+0000","last_became_active":"2024-02-13T00:55:04.808497+0000","last_became_peered":"2024-02-13T00:55:04.808497+0000","last_unstale":"2024-02-13T00:56:55.796076+0000","last_undegraded":"2024-02-13T00:56:55.796076+0000","last_fullsized":"2024-02-13T00:56:55.796076+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:43.888814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:56:43.888814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":521,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798769+0000","last_change":"2024-02-13T00:56:48.913299+0000","last_active":"2024-02-13T00:56:55.798769+0000","last_peered":"2024-02-13T00:56:55.798769+0000","last_clean":"2024-02-13T00:56:55.798769+0000","last_became_active":"2024-02-13T00:56:07.654355+0000","last_became_peered":"2024-02-13T00:56:07.654355+0000","last_unstale":"2024-02-13T00:56:55.798769+0000","last_undegraded":"2024-02-13T00:56:55.798769+0000","last_fullsized":"2024-02-13T00:56:55.798769+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:48.913221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:48.913221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":784,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.635258+0000","last_change":"2024-02-13T00:55:38.989290+0000","last_active":"2024-02-13T00:56:55.635258+0000","last_peered":"2024-02-13T00:56:55.635258+0000","last_clean":"2024-02-13T00:56:55.635258+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:56:55.635258+0000","last_undegraded":"2024-02-13T00:56:55.635258+0000","last_fullsized":"2024-02-13T00:56:55.635258+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:38.989213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:55:38.989213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"851'39","reported_seq":641,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796922+0000","last_change":"2024-02-13T00:56:15.309324+0000","last_active":"2024-02-13T00:56:55.796922+0000","last_peered":"2024-02-13T00:56:55.796922+0000","last_clean":"2024-02-13T00:56:55.796922+0000","last_became_active":"2024-02-13T00:55:51.631136+0000","last_became_peered":"2024-02-13T00:55:51.631136+0000","last_unstale":"2024-02-13T00:56:55.796922+0000","last_undegraded":"2024-02-13T00:56:55.796922+0000","last_fullsized":"2024-02-13T00:56:55.796922+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'39","last_scrub_stamp":"2024-02-13T00:56:15.309243+0000","last_deep_scrub":"851'39","last_deep_scrub_stamp":"2024-02-13T00:56:15.309243+0000","last_clean_scrub_stamp":"2024-02-13T00:56:15.309243+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":466,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797471+0000","last_change":"2024-02-13T00:56:07.657974+0000","last_active":"2024-02-13T00:56:55.797471+0000","last_peered":"2024-02-13T00:56:55.797471+0000","last_clean":"2024-02-13T00:56:55.797471+0000","last_became_active":"2024-02-13T00:56:07.657609+0000","last_became_peered":"2024-02-13T00:56:07.657609+0000","last_unstale":"2024-02-13T00:56:55.797471+0000","last_undegraded":"2024-02-13T00:56:55.797471+0000","last_fullsized":"2024-02-13T00:56:55.797471+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:55:44.264719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:55:44.264719+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":865,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.157900+0000","last_change":"2024-02-13T00:57:10.157900+0000","last_active":"2024-02-13T00:57:10.157900+0000","last_peered":"2024-02-13T00:57:10.157900+0000","last_clean":"2024-02-13T00:57:10.157900+0000","last_became_active":"2024-02-13T00:54:56.630042+0000","last_became_peered":"2024-02-13T00:54:56.630042+0000","last_unstale":"2024-02-13T00:57:10.157900+0000","last_undegraded":"2024-02-13T00:57:10.157900+0000","last_fullsized":"2024-02-13T00:57:10.157900+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:10.157837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_clean_scrub_stamp":"2024-02-13T00:57:10.157837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":59,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796534+0000","last_change":"2024-02-13T00:56:46.851061+0000","last_active":"2024-02-13T00:56:55.796534+0000","last_peered":"2024-02-13T00:56:55.796534+0000","last_clean":"2024-02-13T00:56:55.796534+0000","last_became_active":"2024-02-13T00:55:04.806188+0000","last_became_peered":"2024-02-13T00:55:04.806188+0000","last_unstale":"2024-02-13T00:56:55.796534+0000","last_undegraded":"2024-02-13T00:56:55.796534+0000","last_fullsized":"2024-02-13T00:56:55.796534+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:46.850991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:29.843131+0000","last_clean_scrub_stamp":"2024-02-13T00:56:46.850991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":67,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.637213+0000","last_change":"2024-02-13T00:56:20.723933+0000","last_active":"2024-02-13T00:56:55.637213+0000","last_peered":"2024-02-13T00:56:55.637213+0000","last_clean":"2024-02-13T00:56:55.637213+0000","last_became_active":"2024-02-13T00:55:04.802114+0000","last_became_peered":"2024-02-13T00:55:04.802114+0000","last_unstale":"2024-02-13T00:56:55.637213+0000","last_undegraded":"2024-02-13T00:56:55.637213+0000","last_fullsized":"2024-02-13T00:56:55.637213+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:20.723868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:10.700548+0000","last_clean_scrub_stamp":"2024-02-13T00:56:20.723868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":574,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796774+0000","last_change":"2024-02-13T00:56:37.072073+0000","last_active":"2024-02-13T00:56:55.796774+0000","last_peered":"2024-02-13T00:56:55.796774+0000","last_clean":"2024-02-13T00:56:55.796774+0000","last_became_active":"2024-02-13T00:55:51.635737+0000","last_became_peered":"2024-02-13T00:55:51.635737+0000","last_unstale":"2024-02-13T00:56:55.796774+0000","last_undegraded":"2024-02-13T00:56:55.796774+0000","last_fullsized":"2024-02-13T00:56:55.796774+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:56:37.071990+0000","last_deep_scrub":"467'25","last_deep_scrub_stamp":"2024-02-13T00:55:08.701877+0000","last_clean_scrub_stamp":"2024-02-13T00:56:37.071990+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":880,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.634317+0000","last_change":"2024-02-13T00:56:10.911812+0000","last_active":"2024-02-13T00:56:55.634317+0000","last_peered":"2024-02-13T00:56:55.634317+0000","last_clean":"2024-02-13T00:56:55.634317+0000","last_became_active":"2024-02-13T00:55:00.636125+0000","last_became_peered":"2024-02-13T00:55:00.636125+0000","last_unstale":"2024-02-13T00:56:55.634317+0000","last_undegraded":"2024-02-13T00:56:55.634317+0000","last_fullsized":"2024-02-13T00:56:55.634317+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:10.911727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:10.911727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"824'24","reported_seq":732,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.432257+0000","last_change":"2024-02-13T00:55:52.737311+0000","last_active":"2024-02-13T00:57:09.432257+0000","last_peered":"2024-02-13T00:57:09.432257+0000","last_clean":"2024-02-13T00:57:09.432257+0000","last_became_active":"2024-02-13T00:55:07.818223+0000","last_became_peered":"2024-02-13T00:55:07.818223+0000","last_unstale":"2024-02-13T00:57:09.432257+0000","last_undegraded":"2024-02-13T00:57:09.432257+0000","last_fullsized":"2024-02-13T00:57:09.432257+0000","mapping_epoch":823,"log_start":"404'11","ondisk_log_start":"404'11","created":60,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"824'24","last_scrub_stamp":"2024-02-13T00:55:52.737230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:55:52.737230+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":844,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798910+0000","last_change":"2024-02-13T00:56:49.191428+0000","last_active":"2024-02-13T00:56:55.798910+0000","last_peered":"2024-02-13T00:56:55.798910+0000","last_clean":"2024-02-13T00:56:55.798910+0000","last_became_active":"2024-02-13T00:55:51.631073+0000","last_became_peered":"2024-02-13T00:55:51.631073+0000","last_unstale":"2024-02-13T00:56:55.798910+0000","last_undegraded":"2024-02-13T00:56:55.798910+0000","last_fullsized":"2024-02-13T00:56:55.798910+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:56:49.191338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:49.191338+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":529,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796817+0000","last_change":"2024-02-13T00:56:45.968431+0000","last_active":"2024-02-13T00:56:55.796817+0000","last_peered":"2024-02-13T00:56:55.796817+0000","last_clean":"2024-02-13T00:56:55.796817+0000","last_became_active":"2024-02-13T00:56:07.657754+0000","last_became_peered":"2024-02-13T00:56:07.657754+0000","last_unstale":"2024-02-13T00:56:55.796817+0000","last_undegraded":"2024-02-13T00:56:55.796817+0000","last_fullsized":"2024-02-13T00:56:55.796817+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:45.968347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:45.968347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"806'34","reported_seq":1074,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.634343+0000","last_change":"2024-02-13T00:55:56.946858+0000","last_active":"2024-02-13T00:56:55.634343+0000","last_peered":"2024-02-13T00:56:55.634343+0000","last_clean":"2024-02-13T00:56:55.634343+0000","last_became_active":"2024-02-13T00:55:00.635264+0000","last_became_peered":"2024-02-13T00:55:00.635264+0000","last_unstale":"2024-02-13T00:56:55.634343+0000","last_undegraded":"2024-02-13T00:56:55.634343+0000","last_fullsized":"2024-02-13T00:56:55.634343+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"806'34","last_scrub_stamp":"2024-02-13T00:55:56.946788+0000","last_deep_scrub":"806'34","last_deep_scrub_stamp":"2024-02-13T00:54:38.218263+0000","last_clean_scrub_stamp":"2024-02-13T00:55:56.946788+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":911,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.634880+0000","last_change":"2024-02-13T00:56:24.887795+0000","last_active":"2024-02-13T00:56:55.634880+0000","last_peered":"2024-02-13T00:56:55.634880+0000","last_clean":"2024-02-13T00:56:55.634880+0000","last_became_active":"2024-02-13T00:55:00.638261+0000","last_became_peered":"2024-02-13T00:55:00.638261+0000","last_unstale":"2024-02-13T00:56:55.634880+0000","last_undegraded":"2024-02-13T00:56:55.634880+0000","last_fullsized":"2024-02-13T00:56:55.634880+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:56:24.887707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:24.887707+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"842'25","reported_seq":710,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.432802+0000","last_change":"2024-02-13T00:56:58.245207+0000","last_active":"2024-02-13T00:57:09.432802+0000","last_peered":"2024-02-13T00:57:09.432802+0000","last_clean":"2024-02-13T00:57:09.432802+0000","last_became_active":"2024-02-13T00:55:51.633596+0000","last_became_peered":"2024-02-13T00:55:51.633596+0000","last_unstale":"2024-02-13T00:57:09.432802+0000","last_undegraded":"2024-02-13T00:57:09.432802+0000","last_fullsized":"2024-02-13T00:57:09.432802+0000","mapping_epoch":841,"log_start":"467'11","ondisk_log_start":"467'11","created":60,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"842'25","last_scrub_stamp":"2024-02-13T00:56:58.245119+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:56:58.245119+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":64,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.880270+0000","last_change":"2024-02-13T00:56:42.022798+0000","last_active":"2024-02-13T00:56:51.880270+0000","last_peered":"2024-02-13T00:56:51.880270+0000","last_clean":"2024-02-13T00:56:51.880270+0000","last_became_active":"2024-02-13T00:54:52.746904+0000","last_became_peered":"2024-02-13T00:54:52.746904+0000","last_unstale":"2024-02-13T00:56:51.880270+0000","last_undegraded":"2024-02-13T00:56:51.880270+0000","last_fullsized":"2024-02-13T00:56:51.880270+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:42.022702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:56:42.022702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":804,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797367+0000","last_change":"2024-02-13T00:56:32.724624+0000","last_active":"2024-02-13T00:56:55.797367+0000","last_peered":"2024-02-13T00:56:55.797367+0000","last_clean":"2024-02-13T00:56:55.797367+0000","last_became_active":"2024-02-13T00:55:04.815387+0000","last_became_peered":"2024-02-13T00:55:04.815387+0000","last_unstale":"2024-02-13T00:56:55.797367+0000","last_undegraded":"2024-02-13T00:56:55.797367+0000","last_fullsized":"2024-02-13T00:56:55.797367+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:32.724540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:56:32.724540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":598,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798600+0000","last_change":"2024-02-13T00:56:44.170249+0000","last_active":"2024-02-13T00:56:55.798600+0000","last_peered":"2024-02-13T00:56:55.798600+0000","last_clean":"2024-02-13T00:56:55.798600+0000","last_became_active":"2024-02-13T00:55:51.635856+0000","last_became_peered":"2024-02-13T00:55:51.635856+0000","last_unstale":"2024-02-13T00:56:55.798600+0000","last_undegraded":"2024-02-13T00:56:55.798600+0000","last_fullsized":"2024-02-13T00:56:55.798600+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:56:44.170175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:44.170175+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":503,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798882+0000","last_change":"2024-02-13T00:56:52.824236+0000","last_active":"2024-02-13T00:56:55.798882+0000","last_peered":"2024-02-13T00:56:55.798882+0000","last_clean":"2024-02-13T00:56:55.798882+0000","last_became_active":"2024-02-13T00:56:07.837024+0000","last_became_peered":"2024-02-13T00:56:07.837024+0000","last_unstale":"2024-02-13T00:56:55.798882+0000","last_undegraded":"2024-02-13T00:56:55.798882+0000","last_fullsized":"2024-02-13T00:56:55.798882+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:52.824142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:52.824142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"834'23","reported_seq":822,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:59.430687+0000","last_change":"2024-02-13T00:56:58.775254+0000","last_active":"2024-02-13T00:56:59.430687+0000","last_peered":"2024-02-13T00:56:59.430687+0000","last_clean":"2024-02-13T00:56:59.430687+0000","last_became_active":"2024-02-13T00:55:30.734460+0000","last_became_peered":"2024-02-13T00:55:30.734460+0000","last_unstale":"2024-02-13T00:56:59.430687+0000","last_undegraded":"2024-02-13T00:56:59.430687+0000","last_fullsized":"2024-02-13T00:56:59.430687+0000","mapping_epoch":833,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"834'23","last_scrub_stamp":"2024-02-13T00:56:58.775177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:56:58.775177+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":53,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.636914+0000","last_change":"2024-02-13T00:56:21.717193+0000","last_active":"2024-02-13T00:56:55.636914+0000","last_peered":"2024-02-13T00:56:55.636914+0000","last_clean":"2024-02-13T00:56:55.636914+0000","last_became_active":"2024-02-13T00:55:00.637174+0000","last_became_peered":"2024-02-13T00:55:00.637174+0000","last_unstale":"2024-02-13T00:56:55.636914+0000","last_undegraded":"2024-02-13T00:56:55.636914+0000","last_fullsized":"2024-02-13T00:56:55.636914+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:21.717108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:21.717108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":68,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.879348+0000","last_change":"2024-02-13T00:56:19.147899+0000","last_active":"2024-02-13T00:56:51.879348+0000","last_peered":"2024-02-13T00:56:51.879348+0000","last_clean":"2024-02-13T00:56:51.879348+0000","last_became_active":"2024-02-13T00:54:56.630273+0000","last_became_peered":"2024-02-13T00:54:56.630273+0000","last_unstale":"2024-02-13T00:56:51.879348+0000","last_undegraded":"2024-02-13T00:56:51.879348+0000","last_fullsized":"2024-02-13T00:56:51.879348+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:19.147801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:19.147801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"820'14","reported_seq":83,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.432288+0000","last_change":"2024-02-13T00:56:04.735803+0000","last_active":"2024-02-13T00:57:09.432288+0000","last_peered":"2024-02-13T00:57:09.432288+0000","last_clean":"2024-02-13T00:57:09.432288+0000","last_became_active":"2024-02-13T00:55:00.647621+0000","last_became_peered":"2024-02-13T00:55:00.647621+0000","last_unstale":"2024-02-13T00:57:09.432288+0000","last_undegraded":"2024-02-13T00:57:09.432288+0000","last_fullsized":"2024-02-13T00:57:09.432288+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"820'14","last_scrub_stamp":"2024-02-13T00:56:04.735726+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:56:04.735726+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":63,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.799186+0000","last_change":"2024-02-13T00:56:46.156551+0000","last_active":"2024-02-13T00:56:55.799186+0000","last_peered":"2024-02-13T00:56:55.799186+0000","last_clean":"2024-02-13T00:56:55.799186+0000","last_became_active":"2024-02-13T00:55:51.635323+0000","last_became_peered":"2024-02-13T00:55:51.635323+0000","last_unstale":"2024-02-13T00:56:55.799186+0000","last_undegraded":"2024-02-13T00:56:55.799186+0000","last_fullsized":"2024-02-13T00:56:55.799186+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'34","last_scrub_stamp":"2024-02-13T00:56:46.156468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:46.156468+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":590,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.799339+0000","last_change":"2024-02-13T00:56:41.122457+0000","last_active":"2024-02-13T00:56:55.799339+0000","last_peered":"2024-02-13T00:56:55.799339+0000","last_clean":"2024-02-13T00:56:55.799339+0000","last_became_active":"2024-02-13T00:55:04.781853+0000","last_became_peered":"2024-02-13T00:55:04.781853+0000","last_unstale":"2024-02-13T00:56:55.799339+0000","last_undegraded":"2024-02-13T00:56:55.799339+0000","last_fullsized":"2024-02-13T00:56:55.799339+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:41.122380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:18.518704+0000","last_clean_scrub_stamp":"2024-02-13T00:56:41.122380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":552,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:07.786025+0000","last_change":"2024-02-13T00:57:07.785996+0000","last_active":"2024-02-13T00:57:07.786025+0000","last_peered":"2024-02-13T00:57:07.786025+0000","last_clean":"2024-02-13T00:57:07.786025+0000","last_became_active":"2024-02-13T00:56:07.649968+0000","last_became_peered":"2024-02-13T00:56:07.649968+0000","last_unstale":"2024-02-13T00:57:07.786025+0000","last_undegraded":"2024-02-13T00:57:07.786025+0000","last_fullsized":"2024-02-13T00:57:07.786025+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:07.785898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:57:07.785898+0000","last_clean_scrub_stamp":"2024-02-13T00:57:07.785898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":53,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.633858+0000","last_change":"2024-02-13T00:56:17.665320+0000","last_active":"2024-02-13T00:56:55.633858+0000","last_peered":"2024-02-13T00:56:55.633858+0000","last_clean":"2024-02-13T00:56:55.633858+0000","last_became_active":"2024-02-13T00:55:00.654433+0000","last_became_peered":"2024-02-13T00:55:00.654433+0000","last_unstale":"2024-02-13T00:56:55.633858+0000","last_undegraded":"2024-02-13T00:56:55.633858+0000","last_fullsized":"2024-02-13T00:56:55.633858+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:17.665207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:17.665207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":641,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.799308+0000","last_change":"2024-02-13T00:56:04.369856+0000","last_active":"2024-02-13T00:56:55.799308+0000","last_peered":"2024-02-13T00:56:55.799308+0000","last_clean":"2024-02-13T00:56:55.799308+0000","last_became_active":"2024-02-13T00:55:51.634823+0000","last_became_peered":"2024-02-13T00:55:51.634823+0000","last_unstale":"2024-02-13T00:56:55.799308+0000","last_undegraded":"2024-02-13T00:56:55.799308+0000","last_fullsized":"2024-02-13T00:56:55.799308+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:04.369780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:56:04.369780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"843'52","reported_seq":90,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:05.656877+0000","last_change":"2024-02-13T00:57:05.656835+0000","last_active":"2024-02-13T00:57:05.656877+0000","last_peered":"2024-02-13T00:57:05.656877+0000","last_clean":"2024-02-13T00:57:05.656877+0000","last_became_active":"2024-02-13T00:55:00.654881+0000","last_became_peered":"2024-02-13T00:55:00.654881+0000","last_unstale":"2024-02-13T00:57:05.656877+0000","last_undegraded":"2024-02-13T00:57:05.656877+0000","last_fullsized":"2024-02-13T00:57:05.656877+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'52","last_scrub_stamp":"2024-02-13T00:57:05.656720+0000","last_deep_scrub":"843'52","last_deep_scrub_stamp":"2024-02-13T00:55:56.638643+0000","last_clean_scrub_stamp":"2024-02-13T00:57:05.656720+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":871,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.634363+0000","last_change":"2024-02-13T00:55:48.069136+0000","last_active":"2024-02-13T00:56:55.634363+0000","last_peered":"2024-02-13T00:56:55.634363+0000","last_clean":"2024-02-13T00:56:55.634363+0000","last_became_active":"2024-02-13T00:54:56.623499+0000","last_became_peered":"2024-02-13T00:54:56.623499+0000","last_unstale":"2024-02-13T00:56:55.634363+0000","last_undegraded":"2024-02-13T00:56:55.634363+0000","last_fullsized":"2024-02-13T00:56:55.634363+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:48.069074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:48.069074+0000","last_clean_scrub_stamp":"2024-02-13T00:55:48.069074+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":801,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798156+0000","last_change":"2024-02-13T00:56:36.024603+0000","last_active":"2024-02-13T00:56:55.798156+0000","last_peered":"2024-02-13T00:56:55.798156+0000","last_clean":"2024-02-13T00:56:55.798156+0000","last_became_active":"2024-02-13T00:55:51.632395+0000","last_became_peered":"2024-02-13T00:55:51.632395+0000","last_unstale":"2024-02-13T00:56:55.798156+0000","last_undegraded":"2024-02-13T00:56:55.798156+0000","last_fullsized":"2024-02-13T00:56:55.798156+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:36.024532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:36.024532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":52,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796013+0000","last_change":"2024-02-13T00:56:11.817669+0000","last_active":"2024-02-13T00:56:55.796013+0000","last_peered":"2024-02-13T00:56:55.796013+0000","last_clean":"2024-02-13T00:56:55.796013+0000","last_became_active":"2024-02-13T00:55:04.790891+0000","last_became_peered":"2024-02-13T00:55:04.790891+0000","last_unstale":"2024-02-13T00:56:55.796013+0000","last_undegraded":"2024-02-13T00:56:55.796013+0000","last_fullsized":"2024-02-13T00:56:55.796013+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:11.817599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_clean_scrub_stamp":"2024-02-13T00:56:11.817599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":572,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796958+0000","last_change":"2024-02-13T00:56:06.915576+0000","last_active":"2024-02-13T00:56:55.796958+0000","last_peered":"2024-02-13T00:56:55.796958+0000","last_clean":"2024-02-13T00:56:55.796958+0000","last_became_active":"2024-02-13T00:54:52.757890+0000","last_became_peered":"2024-02-13T00:54:52.757890+0000","last_unstale":"2024-02-13T00:56:55.796958+0000","last_undegraded":"2024-02-13T00:56:55.796958+0000","last_fullsized":"2024-02-13T00:56:55.796958+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:06.915454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:56:06.915454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":1016,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.795990+0000","last_change":"2024-02-13T00:56:50.889120+0000","last_active":"2024-02-13T00:56:55.795990+0000","last_peered":"2024-02-13T00:56:55.795990+0000","last_clean":"2024-02-13T00:56:55.795990+0000","last_became_active":"2024-02-13T00:54:52.756864+0000","last_became_peered":"2024-02-13T00:54:52.756864+0000","last_unstale":"2024-02-13T00:56:55.795990+0000","last_undegraded":"2024-02-13T00:56:55.795990+0000","last_fullsized":"2024-02-13T00:56:55.795990+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":814,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'44","last_scrub_stamp":"2024-02-13T00:56:50.889053+0000","last_deep_scrub":"467'44","last_deep_scrub_stamp":"2024-02-13T00:55:28.004399+0000","last_clean_scrub_stamp":"2024-02-13T00:56:50.889053+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":661,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798855+0000","last_change":"2024-02-13T00:56:11.886835+0000","last_active":"2024-02-13T00:56:55.798855+0000","last_peered":"2024-02-13T00:56:55.798855+0000","last_clean":"2024-02-13T00:56:55.798855+0000","last_became_active":"2024-02-13T00:56:07.837531+0000","last_became_peered":"2024-02-13T00:56:07.837531+0000","last_unstale":"2024-02-13T00:56:55.798855+0000","last_undegraded":"2024-02-13T00:56:55.798855+0000","last_fullsized":"2024-02-13T00:56:55.798855+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:11.886755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:11.886755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":883,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:57.143955+0000","last_change":"2024-02-13T00:56:57.143932+0000","last_active":"2024-02-13T00:56:57.143955+0000","last_peered":"2024-02-13T00:56:57.143955+0000","last_clean":"2024-02-13T00:56:57.143955+0000","last_became_active":"2024-02-13T00:55:04.803301+0000","last_became_peered":"2024-02-13T00:55:04.803301+0000","last_unstale":"2024-02-13T00:56:57.143955+0000","last_undegraded":"2024-02-13T00:56:57.143955+0000","last_fullsized":"2024-02-13T00:56:57.143955+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:57.143854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:57.143854+0000","last_clean_scrub_stamp":"2024-02-13T00:56:57.143854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":45,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798837+0000","last_change":"2024-02-13T00:56:22.222784+0000","last_active":"2024-02-13T00:56:55.798837+0000","last_peered":"2024-02-13T00:56:55.798837+0000","last_clean":"2024-02-13T00:56:55.798837+0000","last_became_active":"2024-02-13T00:55:51.629972+0000","last_became_peered":"2024-02-13T00:55:51.629972+0000","last_unstale":"2024-02-13T00:56:55.798837+0000","last_undegraded":"2024-02-13T00:56:55.798837+0000","last_fullsized":"2024-02-13T00:56:55.798837+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:22.222692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:22.222692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":59,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.880647+0000","last_change":"2024-02-13T00:55:52.215783+0000","last_active":"2024-02-13T00:56:51.880647+0000","last_peered":"2024-02-13T00:56:51.880647+0000","last_clean":"2024-02-13T00:56:51.880647+0000","last_became_active":"2024-02-13T00:54:52.749099+0000","last_became_peered":"2024-02-13T00:54:52.749099+0000","last_unstale":"2024-02-13T00:56:51.880647+0000","last_undegraded":"2024-02-13T00:56:51.880647+0000","last_fullsized":"2024-02-13T00:56:51.880647+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:52.215713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:52.215713+0000","last_clean_scrub_stamp":"2024-02-13T00:55:52.215713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":999,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797129+0000","last_change":"2024-02-13T00:56:43.777275+0000","last_active":"2024-02-13T00:56:55.797129+0000","last_peered":"2024-02-13T00:56:55.797129+0000","last_clean":"2024-02-13T00:56:55.797129+0000","last_became_active":"2024-02-13T00:55:04.818088+0000","last_became_peered":"2024-02-13T00:55:04.818088+0000","last_unstale":"2024-02-13T00:56:55.797129+0000","last_undegraded":"2024-02-13T00:56:55.797129+0000","last_fullsized":"2024-02-13T00:56:55.797129+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:56:43.777208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:43.777208+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":400,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.864082+0000","last_change":"2024-02-13T00:56:55.864043+0000","last_active":"2024-02-13T00:56:55.864082+0000","last_peered":"2024-02-13T00:56:55.864082+0000","last_clean":"2024-02-13T00:56:55.864082+0000","last_became_active":"2024-02-13T00:56:07.836974+0000","last_became_peered":"2024-02-13T00:56:07.836974+0000","last_unstale":"2024-02-13T00:56:55.864082+0000","last_undegraded":"2024-02-13T00:56:55.864082+0000","last_fullsized":"2024-02-13T00:56:55.864082+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:55.863926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:53.574605+0000","last_clean_scrub_stamp":"2024-02-13T00:56:55.863926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":890,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796008+0000","last_change":"2024-02-13T00:56:41.808990+0000","last_active":"2024-02-13T00:56:55.796008+0000","last_peered":"2024-02-13T00:56:55.796008+0000","last_clean":"2024-02-13T00:56:55.796008+0000","last_became_active":"2024-02-13T00:55:00.640461+0000","last_became_peered":"2024-02-13T00:55:00.640461+0000","last_unstale":"2024-02-13T00:56:55.796008+0000","last_undegraded":"2024-02-13T00:56:55.796008+0000","last_fullsized":"2024-02-13T00:56:55.796008+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:41.808911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:41.808911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":883,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.636672+0000","last_change":"2024-02-13T00:56:35.978194+0000","last_active":"2024-02-13T00:56:55.636672+0000","last_peered":"2024-02-13T00:56:55.636672+0000","last_clean":"2024-02-13T00:56:55.636672+0000","last_became_active":"2024-02-13T00:55:00.650492+0000","last_became_peered":"2024-02-13T00:55:00.650492+0000","last_unstale":"2024-02-13T00:56:55.636672+0000","last_undegraded":"2024-02-13T00:56:55.636672+0000","last_fullsized":"2024-02-13T00:56:55.636672+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:35.978100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:56:35.978100+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":64,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:01.691896+0000","last_change":"2024-02-13T00:57:01.691896+0000","last_active":"2024-02-13T00:57:01.691896+0000","last_peered":"2024-02-13T00:57:01.691896+0000","last_clean":"2024-02-13T00:57:01.691896+0000","last_became_active":"2024-02-13T00:54:52.746829+0000","last_became_peered":"2024-02-13T00:54:52.746829+0000","last_unstale":"2024-02-13T00:57:01.691896+0000","last_undegraded":"2024-02-13T00:57:01.691896+0000","last_fullsized":"2024-02-13T00:57:01.691896+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:01.691809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:57:01.691809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":339,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797240+0000","last_change":"2024-02-13T00:56:07.658699+0000","last_active":"2024-02-13T00:56:55.797240+0000","last_peered":"2024-02-13T00:56:55.797240+0000","last_clean":"2024-02-13T00:56:55.797240+0000","last_became_active":"2024-02-13T00:56:07.657907+0000","last_became_peered":"2024-02-13T00:56:07.657907+0000","last_unstale":"2024-02-13T00:56:55.797240+0000","last_undegraded":"2024-02-13T00:56:55.797240+0000","last_fullsized":"2024-02-13T00:56:55.797240+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:53.729897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:55:53.729897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":679,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797761+0000","last_change":"2024-02-13T00:56:15.030027+0000","last_active":"2024-02-13T00:56:55.797761+0000","last_peered":"2024-02-13T00:56:55.797761+0000","last_clean":"2024-02-13T00:56:55.797761+0000","last_became_active":"2024-02-13T00:54:56.633712+0000","last_became_peered":"2024-02-13T00:54:56.633712+0000","last_unstale":"2024-02-13T00:56:55.797761+0000","last_undegraded":"2024-02-13T00:56:55.797761+0000","last_fullsized":"2024-02-13T00:56:55.797761+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'33","last_scrub_stamp":"2024-02-13T00:56:15.029929+0000","last_deep_scrub":"467'33","last_deep_scrub_stamp":"2024-02-13T00:53:22.462526+0000","last_clean_scrub_stamp":"2024-02-13T00:56:15.029929+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":644,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.799172+0000","last_change":"2024-02-13T00:56:51.843465+0000","last_active":"2024-02-13T00:56:55.799172+0000","last_peered":"2024-02-13T00:56:55.799172+0000","last_clean":"2024-02-13T00:56:55.799172+0000","last_became_active":"2024-02-13T00:56:07.837752+0000","last_became_peered":"2024-02-13T00:56:07.837752+0000","last_unstale":"2024-02-13T00:56:55.799172+0000","last_undegraded":"2024-02-13T00:56:55.799172+0000","last_fullsized":"2024-02-13T00:56:55.799172+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:51.843386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:37.044403+0000","last_clean_scrub_stamp":"2024-02-13T00:56:51.843386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.879672+0000","last_change":"2024-02-13T00:56:21.131572+0000","last_active":"2024-02-13T00:56:51.879672+0000","last_peered":"2024-02-13T00:56:51.879672+0000","last_clean":"2024-02-13T00:56:51.879672+0000","last_became_active":"2024-02-13T00:55:04.781120+0000","last_became_peered":"2024-02-13T00:55:04.781120+0000","last_unstale":"2024-02-13T00:56:51.879672+0000","last_undegraded":"2024-02-13T00:56:51.879672+0000","last_fullsized":"2024-02-13T00:56:51.879672+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:21.131500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:21.131500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":53,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.636483+0000","last_change":"2024-02-13T00:56:13.680249+0000","last_active":"2024-02-13T00:56:55.636483+0000","last_peered":"2024-02-13T00:56:55.636483+0000","last_clean":"2024-02-13T00:56:55.636483+0000","last_became_active":"2024-02-13T00:55:00.640413+0000","last_became_peered":"2024-02-13T00:55:00.640413+0000","last_unstale":"2024-02-13T00:56:55.636483+0000","last_undegraded":"2024-02-13T00:56:55.636483+0000","last_fullsized":"2024-02-13T00:56:55.636483+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:13.680172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:56:13.680172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":82,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:57.773718+0000","last_change":"2024-02-13T00:56:57.773718+0000","last_active":"2024-02-13T00:56:57.773718+0000","last_peered":"2024-02-13T00:56:57.773718+0000","last_clean":"2024-02-13T00:56:57.773718+0000","last_became_active":"2024-02-13T00:55:00.622363+0000","last_became_peered":"2024-02-13T00:55:00.622363+0000","last_unstale":"2024-02-13T00:56:57.773718+0000","last_undegraded":"2024-02-13T00:56:57.773718+0000","last_fullsized":"2024-02-13T00:56:57.773718+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:57.773620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:56:57.773620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":827,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:08.999882+0000","last_change":"2024-02-13T00:57:08.999858+0000","last_active":"2024-02-13T00:57:08.999882+0000","last_peered":"2024-02-13T00:57:08.999882+0000","last_clean":"2024-02-13T00:57:08.999882+0000","last_became_active":"2024-02-13T00:54:56.635321+0000","last_became_peered":"2024-02-13T00:54:56.635321+0000","last_unstale":"2024-02-13T00:57:08.999882+0000","last_undegraded":"2024-02-13T00:57:08.999882+0000","last_fullsized":"2024-02-13T00:57:08.999882+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:08.999794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:57:08.999794+0000","last_clean_scrub_stamp":"2024-02-13T00:57:08.999794+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":57,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:07.900418+0000","last_change":"2024-02-13T00:57:07.900382+0000","last_active":"2024-02-13T00:57:07.900418+0000","last_peered":"2024-02-13T00:57:07.900418+0000","last_clean":"2024-02-13T00:57:07.900418+0000","last_became_active":"2024-02-13T00:55:04.781222+0000","last_became_peered":"2024-02-13T00:55:04.781222+0000","last_unstale":"2024-02-13T00:57:07.900418+0000","last_undegraded":"2024-02-13T00:57:07.900418+0000","last_fullsized":"2024-02-13T00:57:07.900418+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:07.900310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:57:07.900310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"805'29","reported_seq":947,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.634474+0000","last_change":"2024-02-13T00:55:55.954797+0000","last_active":"2024-02-13T00:56:55.634474+0000","last_peered":"2024-02-13T00:56:55.634474+0000","last_clean":"2024-02-13T00:56:55.634474+0000","last_became_active":"2024-02-13T00:54:52.743094+0000","last_became_peered":"2024-02-13T00:54:52.743094+0000","last_unstale":"2024-02-13T00:56:55.634474+0000","last_undegraded":"2024-02-13T00:56:55.634474+0000","last_fullsized":"2024-02-13T00:56:55.634474+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":814,"parent":"0.0","parent_split_bits":0,"last_scrub":"805'29","last_scrub_stamp":"2024-02-13T00:55:55.954715+0000","last_deep_scrub":"805'29","last_deep_scrub_stamp":"2024-02-13T00:54:31.218066+0000","last_clean_scrub_stamp":"2024-02-13T00:55:55.954715+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.795697+0000","last_change":"2024-02-13T00:56:04.777422+0000","last_active":"2024-02-13T00:56:55.795697+0000","last_peered":"2024-02-13T00:56:55.795697+0000","last_clean":"2024-02-13T00:56:55.795697+0000","last_became_active":"2024-02-13T00:55:04.808209+0000","last_became_peered":"2024-02-13T00:55:04.808209+0000","last_unstale":"2024-02-13T00:56:55.795697+0000","last_undegraded":"2024-02-13T00:56:55.795697+0000","last_fullsized":"2024-02-13T00:56:55.795697+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:04.777361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:56:04.777361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":2042,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957105+0000","last_change":"2024-02-13T00:55:40.016374+0000","last_active":"2024-02-13T00:57:09.957105+0000","last_peered":"2024-02-13T00:57:09.957105+0000","last_clean":"2024-02-13T00:57:09.957105+0000","last_became_active":"2024-02-13T00:55:04.803993+0000","last_became_peered":"2024-02-13T00:55:04.803993+0000","last_unstale":"2024-02-13T00:57:09.957105+0000","last_undegraded":"2024-02-13T00:57:09.957105+0000","last_fullsized":"2024-02-13T00:57:09.957105+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:55:40.016277+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:55:40.016277+0000","last_clean_scrub_stamp":"2024-02-13T00:55:40.016277+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1152,"num_read_kb":1147,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":563,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796462+0000","last_change":"2024-02-13T00:56:10.848413+0000","last_active":"2024-02-13T00:56:55.796462+0000","last_peered":"2024-02-13T00:56:55.796462+0000","last_clean":"2024-02-13T00:56:55.796462+0000","last_became_active":"2024-02-13T00:56:07.654155+0000","last_became_peered":"2024-02-13T00:56:07.654155+0000","last_unstale":"2024-02-13T00:56:55.796462+0000","last_undegraded":"2024-02-13T00:56:55.796462+0000","last_fullsized":"2024-02-13T00:56:55.796462+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:10.848353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:10.848353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":627,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796881+0000","last_change":"2024-02-13T00:56:50.206280+0000","last_active":"2024-02-13T00:56:55.796881+0000","last_peered":"2024-02-13T00:56:55.796881+0000","last_clean":"2024-02-13T00:56:55.796881+0000","last_became_active":"2024-02-13T00:55:51.633728+0000","last_became_peered":"2024-02-13T00:55:51.633728+0000","last_unstale":"2024-02-13T00:56:55.796881+0000","last_undegraded":"2024-02-13T00:56:55.796881+0000","last_fullsized":"2024-02-13T00:56:55.796881+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:50.206142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:50.206142+0000","last_clean_scrub_stamp":"2024-02-13T00:56:50.206142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":437,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797270+0000","last_change":"2024-02-13T00:55:48.806610+0000","last_active":"2024-02-13T00:56:55.797270+0000","last_peered":"2024-02-13T00:56:55.797270+0000","last_clean":"2024-02-13T00:56:55.797270+0000","last_became_active":"2024-02-13T00:54:56.633237+0000","last_became_peered":"2024-02-13T00:54:56.633237+0000","last_unstale":"2024-02-13T00:56:55.797270+0000","last_undegraded":"2024-02-13T00:56:55.797270+0000","last_fullsized":"2024-02-13T00:56:55.797270+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:48.806530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:55:48.806530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"821'69","reported_seq":325,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.432284+0000","last_change":"2024-02-13T00:56:32.961075+0000","last_active":"2024-02-13T00:57:09.432284+0000","last_peered":"2024-02-13T00:57:09.432284+0000","last_clean":"2024-02-13T00:57:09.432284+0000","last_became_active":"2024-02-13T00:55:04.805061+0000","last_became_peered":"2024-02-13T00:55:04.805061+0000","last_unstale":"2024-02-13T00:57:09.432284+0000","last_undegraded":"2024-02-13T00:57:09.432284+0000","last_fullsized":"2024-02-13T00:57:09.432284+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":822,"parent":"0.0","parent_split_bits":0,"last_scrub":"821'69","last_scrub_stamp":"2024-02-13T00:56:32.961000+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:56:32.961000+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":77,"num_read_kb":61,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":3580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":64,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.879758+0000","last_change":"2024-02-13T00:56:33.075349+0000","last_active":"2024-02-13T00:56:51.879758+0000","last_peered":"2024-02-13T00:56:51.879758+0000","last_clean":"2024-02-13T00:56:51.879758+0000","last_became_active":"2024-02-13T00:54:52.751283+0000","last_became_peered":"2024-02-13T00:54:52.751283+0000","last_unstale":"2024-02-13T00:56:51.879758+0000","last_undegraded":"2024-02-13T00:56:51.879758+0000","last_fullsized":"2024-02-13T00:56:51.879758+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:33.075264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:26.245724+0000","last_clean_scrub_stamp":"2024-02-13T00:56:33.075264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":53,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.636865+0000","last_change":"2024-02-13T00:56:08.763587+0000","last_active":"2024-02-13T00:56:55.636865+0000","last_peered":"2024-02-13T00:56:55.636865+0000","last_clean":"2024-02-13T00:56:55.636865+0000","last_became_active":"2024-02-13T00:55:00.651759+0000","last_became_peered":"2024-02-13T00:55:00.651759+0000","last_unstale":"2024-02-13T00:56:55.636865+0000","last_undegraded":"2024-02-13T00:56:55.636865+0000","last_fullsized":"2024-02-13T00:56:55.636865+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:08.763528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:08.763528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":572,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796988+0000","last_change":"2024-02-13T00:56:22.228885+0000","last_active":"2024-02-13T00:56:55.796988+0000","last_peered":"2024-02-13T00:56:55.796988+0000","last_clean":"2024-02-13T00:56:55.796988+0000","last_became_active":"2024-02-13T00:55:04.786126+0000","last_became_peered":"2024-02-13T00:55:04.786126+0000","last_unstale":"2024-02-13T00:56:55.796988+0000","last_undegraded":"2024-02-13T00:56:55.796988+0000","last_fullsized":"2024-02-13T00:56:55.796988+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:22.228792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:22.228792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"814'15","reported_seq":1127,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.432240+0000","last_change":"2024-02-13T00:55:52.011279+0000","last_active":"2024-02-13T00:57:09.432240+0000","last_peered":"2024-02-13T00:57:09.432240+0000","last_clean":"2024-02-13T00:57:09.432240+0000","last_became_active":"2024-02-13T00:54:52.756936+0000","last_became_peered":"2024-02-13T00:54:52.756936+0000","last_unstale":"2024-02-13T00:57:09.432240+0000","last_undegraded":"2024-02-13T00:57:09.432240+0000","last_fullsized":"2024-02-13T00:57:09.432240+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":814,"parent":"0.0","parent_split_bits":0,"last_scrub":"814'15","last_scrub_stamp":"2024-02-13T00:55:52.011200+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:55:52.011200+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":661,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.634586+0000","last_change":"2024-02-13T00:55:49.022870+0000","last_active":"2024-02-13T00:56:55.634586+0000","last_peered":"2024-02-13T00:56:55.634586+0000","last_clean":"2024-02-13T00:56:55.634586+0000","last_became_active":"2024-02-13T00:54:56.623791+0000","last_became_peered":"2024-02-13T00:54:56.623791+0000","last_unstale":"2024-02-13T00:56:55.634586+0000","last_undegraded":"2024-02-13T00:56:55.634586+0000","last_fullsized":"2024-02-13T00:56:55.634586+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:55:49.022799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:55:49.022799+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":849,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.636355+0000","last_change":"2024-02-13T00:56:54.093470+0000","last_active":"2024-02-13T00:56:55.636355+0000","last_peered":"2024-02-13T00:56:55.636355+0000","last_clean":"2024-02-13T00:56:55.636355+0000","last_became_active":"2024-02-13T00:51:41.567277+0000","last_became_peered":"2024-02-13T00:51:41.567277+0000","last_unstale":"2024-02-13T00:56:55.636355+0000","last_undegraded":"2024-02-13T00:56:55.636355+0000","last_fullsized":"2024-02-13T00:56:55.636355+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:54.093399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:56:54.093399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":888,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796926+0000","last_change":"2024-02-13T00:56:31.708170+0000","last_active":"2024-02-13T00:56:55.796926+0000","last_peered":"2024-02-13T00:56:55.796926+0000","last_clean":"2024-02-13T00:56:55.796926+0000","last_became_active":"2024-02-13T00:55:04.811884+0000","last_became_peered":"2024-02-13T00:55:04.811884+0000","last_unstale":"2024-02-13T00:56:55.796926+0000","last_undegraded":"2024-02-13T00:56:55.796926+0000","last_fullsized":"2024-02-13T00:56:55.796926+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:31.708099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:08.094228+0000","last_clean_scrub_stamp":"2024-02-13T00:56:31.708099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":807,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:59.113337+0000","last_change":"2024-02-13T00:56:59.113316+0000","last_active":"2024-02-13T00:56:59.113337+0000","last_peered":"2024-02-13T00:56:59.113337+0000","last_clean":"2024-02-13T00:56:59.113337+0000","last_became_active":"2024-02-13T00:55:04.802927+0000","last_became_peered":"2024-02-13T00:55:04.802927+0000","last_unstale":"2024-02-13T00:56:59.113337+0000","last_undegraded":"2024-02-13T00:56:59.113337+0000","last_fullsized":"2024-02-13T00:56:59.113337+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:59.113239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:56:59.113239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":57,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:05.973421+0000","last_change":"2024-02-13T00:57:05.973384+0000","last_active":"2024-02-13T00:57:05.973421+0000","last_peered":"2024-02-13T00:57:05.973421+0000","last_clean":"2024-02-13T00:57:05.973421+0000","last_became_active":"2024-02-13T00:55:04.808568+0000","last_became_peered":"2024-02-13T00:55:04.808568+0000","last_unstale":"2024-02-13T00:57:05.973421+0000","last_undegraded":"2024-02-13T00:57:05.973421+0000","last_fullsized":"2024-02-13T00:57:05.973421+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:05.973305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:57:05.973305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":920,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.799391+0000","last_change":"2024-02-13T00:56:41.121270+0000","last_active":"2024-02-13T00:56:55.799391+0000","last_peered":"2024-02-13T00:56:55.799391+0000","last_clean":"2024-02-13T00:56:55.799391+0000","last_became_active":"2024-02-13T00:55:51.634260+0000","last_became_peered":"2024-02-13T00:55:51.634260+0000","last_unstale":"2024-02-13T00:56:55.799391+0000","last_undegraded":"2024-02-13T00:56:55.799391+0000","last_fullsized":"2024-02-13T00:56:55.799391+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'30","last_scrub_stamp":"2024-02-13T00:56:41.121161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:41.121161+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":871,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797631+0000","last_change":"2024-02-13T00:56:21.216438+0000","last_active":"2024-02-13T00:56:55.797631+0000","last_peered":"2024-02-13T00:56:55.797631+0000","last_clean":"2024-02-13T00:56:55.797631+0000","last_became_active":"2024-02-13T00:55:51.635608+0000","last_became_peered":"2024-02-13T00:55:51.635608+0000","last_unstale":"2024-02-13T00:56:55.797631+0000","last_undegraded":"2024-02-13T00:56:55.797631+0000","last_fullsized":"2024-02-13T00:56:55.797631+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:56:21.216367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:21.216367+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":53,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.636785+0000","last_change":"2024-02-13T00:55:54.603621+0000","last_active":"2024-02-13T00:56:55.636785+0000","last_peered":"2024-02-13T00:56:55.636785+0000","last_clean":"2024-02-13T00:56:55.636785+0000","last_became_active":"2024-02-13T00:55:00.647645+0000","last_became_peered":"2024-02-13T00:55:00.647645+0000","last_unstale":"2024-02-13T00:56:55.636785+0000","last_undegraded":"2024-02-13T00:56:55.636785+0000","last_fullsized":"2024-02-13T00:56:55.636785+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:54.603530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:55:54.603530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":813,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797222+0000","last_change":"2024-02-13T00:56:06.954325+0000","last_active":"2024-02-13T00:56:55.797222+0000","last_peered":"2024-02-13T00:56:55.797222+0000","last_clean":"2024-02-13T00:56:55.797222+0000","last_became_active":"2024-02-13T00:54:52.755872+0000","last_became_peered":"2024-02-13T00:54:52.755872+0000","last_unstale":"2024-02-13T00:56:55.797222+0000","last_undegraded":"2024-02-13T00:56:55.797222+0000","last_fullsized":"2024-02-13T00:56:55.797222+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:06.954236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:06.954236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"812'33","reported_seq":1026,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797180+0000","last_change":"2024-02-13T00:56:14.954843+0000","last_active":"2024-02-13T00:56:55.797180+0000","last_peered":"2024-02-13T00:56:55.797180+0000","last_clean":"2024-02-13T00:56:55.797180+0000","last_became_active":"2024-02-13T00:55:00.642447+0000","last_became_peered":"2024-02-13T00:55:00.642447+0000","last_unstale":"2024-02-13T00:56:55.797180+0000","last_undegraded":"2024-02-13T00:56:55.797180+0000","last_fullsized":"2024-02-13T00:56:55.797180+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"812'33","last_scrub_stamp":"2024-02-13T00:56:14.954755+0000","last_deep_scrub":"812'33","last_deep_scrub_stamp":"2024-02-13T00:54:52.301469+0000","last_clean_scrub_stamp":"2024-02-13T00:56:14.954755+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":57,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:56.899444+0000","last_change":"2024-02-13T00:56:56.899420+0000","last_active":"2024-02-13T00:56:56.899444+0000","last_peered":"2024-02-13T00:56:56.899444+0000","last_clean":"2024-02-13T00:56:56.899444+0000","last_became_active":"2024-02-13T00:55:04.808145+0000","last_became_peered":"2024-02-13T00:55:04.808145+0000","last_unstale":"2024-02-13T00:56:56.899444+0000","last_undegraded":"2024-02-13T00:56:56.899444+0000","last_fullsized":"2024-02-13T00:56:56.899444+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:56.899331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:56:56.899331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":57,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:08.895286+0000","last_change":"2024-02-13T00:57:08.895263+0000","last_active":"2024-02-13T00:57:08.895286+0000","last_peered":"2024-02-13T00:57:08.895286+0000","last_clean":"2024-02-13T00:57:08.895286+0000","last_became_active":"2024-02-13T00:55:04.781860+0000","last_became_peered":"2024-02-13T00:55:04.781860+0000","last_unstale":"2024-02-13T00:57:08.895286+0000","last_undegraded":"2024-02-13T00:57:08.895286+0000","last_fullsized":"2024-02-13T00:57:08.895286+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:08.895198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:57:08.895198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":581,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.799248+0000","last_change":"2024-02-13T00:56:33.243096+0000","last_active":"2024-02-13T00:56:55.799248+0000","last_peered":"2024-02-13T00:56:55.799248+0000","last_clean":"2024-02-13T00:56:55.799248+0000","last_became_active":"2024-02-13T00:55:04.793577+0000","last_became_peered":"2024-02-13T00:55:04.793577+0000","last_unstale":"2024-02-13T00:56:55.799248+0000","last_undegraded":"2024-02-13T00:56:55.799248+0000","last_fullsized":"2024-02-13T00:56:55.799248+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:33.243018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:33.243018+0000","last_clean_scrub_stamp":"2024-02-13T00:56:33.243018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":87,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:02.710744+0000","last_change":"2024-02-13T00:57:02.710707+0000","last_active":"2024-02-13T00:57:02.710744+0000","last_peered":"2024-02-13T00:57:02.710744+0000","last_clean":"2024-02-13T00:57:02.710744+0000","last_became_active":"2024-02-13T00:56:07.837031+0000","last_became_peered":"2024-02-13T00:56:07.837031+0000","last_unstale":"2024-02-13T00:57:02.710744+0000","last_undegraded":"2024-02-13T00:57:02.710744+0000","last_fullsized":"2024-02-13T00:57:02.710744+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'18","last_scrub_stamp":"2024-02-13T00:57:02.710631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:57:02.710631+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":579,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796981+0000","last_change":"2024-02-13T00:56:51.083897+0000","last_active":"2024-02-13T00:56:55.796981+0000","last_peered":"2024-02-13T00:56:55.796981+0000","last_clean":"2024-02-13T00:56:55.796981+0000","last_became_active":"2024-02-13T00:54:56.632379+0000","last_became_peered":"2024-02-13T00:54:56.632379+0000","last_unstale":"2024-02-13T00:56:55.796981+0000","last_undegraded":"2024-02-13T00:56:55.796981+0000","last_fullsized":"2024-02-13T00:56:55.796981+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:51.083797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:56:51.083797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":565,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:01.823412+0000","last_change":"2024-02-13T00:57:01.823383+0000","last_active":"2024-02-13T00:57:01.823412+0000","last_peered":"2024-02-13T00:57:01.823412+0000","last_clean":"2024-02-13T00:57:01.823412+0000","last_became_active":"2024-02-13T00:56:07.653415+0000","last_became_peered":"2024-02-13T00:56:07.653415+0000","last_unstale":"2024-02-13T00:57:01.823412+0000","last_undegraded":"2024-02-13T00:57:01.823412+0000","last_fullsized":"2024-02-13T00:57:01.823412+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:01.823300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:57:01.823300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":58,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.637082+0000","last_change":"2024-02-13T00:56:22.668873+0000","last_active":"2024-02-13T00:56:55.637082+0000","last_peered":"2024-02-13T00:56:55.637082+0000","last_clean":"2024-02-13T00:56:55.637082+0000","last_became_active":"2024-02-13T00:55:00.654451+0000","last_became_peered":"2024-02-13T00:55:00.654451+0000","last_unstale":"2024-02-13T00:56:55.637082+0000","last_undegraded":"2024-02-13T00:56:55.637082+0000","last_fullsized":"2024-02-13T00:56:55.637082+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:22.668785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:22.668785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":69,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.637001+0000","last_change":"2024-02-13T00:56:14.636419+0000","last_active":"2024-02-13T00:56:55.637001+0000","last_peered":"2024-02-13T00:56:55.637001+0000","last_clean":"2024-02-13T00:56:55.637001+0000","last_became_active":"2024-02-13T00:55:00.646426+0000","last_became_peered":"2024-02-13T00:55:00.646426+0000","last_unstale":"2024-02-13T00:56:55.637001+0000","last_undegraded":"2024-02-13T00:56:55.637001+0000","last_fullsized":"2024-02-13T00:56:55.637001+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'39","last_scrub_stamp":"2024-02-13T00:56:14.636328+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:56:14.636328+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":68,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.879484+0000","last_change":"2024-02-13T00:56:25.127870+0000","last_active":"2024-02-13T00:56:51.879484+0000","last_peered":"2024-02-13T00:56:51.879484+0000","last_clean":"2024-02-13T00:56:51.879484+0000","last_became_active":"2024-02-13T00:55:00.622671+0000","last_became_peered":"2024-02-13T00:55:00.622671+0000","last_unstale":"2024-02-13T00:56:51.879484+0000","last_undegraded":"2024-02-13T00:56:51.879484+0000","last_fullsized":"2024-02-13T00:56:51.879484+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:25.127798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:25.127798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"814'11","reported_seq":90,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:59.430691+0000","last_change":"2024-02-13T00:55:54.194316+0000","last_active":"2024-02-13T00:56:59.430691+0000","last_peered":"2024-02-13T00:56:59.430691+0000","last_clean":"2024-02-13T00:56:59.430691+0000","last_became_active":"2024-02-13T00:54:52.746048+0000","last_became_peered":"2024-02-13T00:54:52.746048+0000","last_unstale":"2024-02-13T00:56:59.430691+0000","last_undegraded":"2024-02-13T00:56:59.430691+0000","last_fullsized":"2024-02-13T00:56:59.430691+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"814'11","last_scrub_stamp":"2024-02-13T00:55:54.194242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:55:54.194242+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":888,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797062+0000","last_change":"2024-02-13T00:55:42.941918+0000","last_active":"2024-02-13T00:56:55.797062+0000","last_peered":"2024-02-13T00:56:55.797062+0000","last_clean":"2024-02-13T00:56:55.797062+0000","last_became_active":"2024-02-13T00:55:00.638952+0000","last_became_peered":"2024-02-13T00:55:00.638952+0000","last_unstale":"2024-02-13T00:56:55.797062+0000","last_undegraded":"2024-02-13T00:56:55.797062+0000","last_fullsized":"2024-02-13T00:56:55.797062+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:42.941846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:55:42.941846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":353,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796584+0000","last_change":"2024-02-13T00:56:44.030693+0000","last_active":"2024-02-13T00:56:55.796584+0000","last_peered":"2024-02-13T00:56:55.796584+0000","last_clean":"2024-02-13T00:56:55.796584+0000","last_became_active":"2024-02-13T00:56:07.656128+0000","last_became_peered":"2024-02-13T00:56:07.656128+0000","last_unstale":"2024-02-13T00:56:55.796584+0000","last_undegraded":"2024-02-13T00:56:55.796584+0000","last_fullsized":"2024-02-13T00:56:55.796584+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:44.030613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:44.030613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"851'27","reported_seq":97,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.880109+0000","last_change":"2024-02-13T00:56:24.112171+0000","last_active":"2024-02-13T00:56:51.880109+0000","last_peered":"2024-02-13T00:56:51.880109+0000","last_clean":"2024-02-13T00:56:51.880109+0000","last_became_active":"2024-02-13T00:55:00.622624+0000","last_became_peered":"2024-02-13T00:55:00.622624+0000","last_unstale":"2024-02-13T00:56:51.880109+0000","last_undegraded":"2024-02-13T00:56:51.880109+0000","last_fullsized":"2024-02-13T00:56:51.880109+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'27","last_scrub_stamp":"2024-02-13T00:56:24.112097+0000","last_deep_scrub":"851'27","last_deep_scrub_stamp":"2024-02-13T00:56:24.112097+0000","last_clean_scrub_stamp":"2024-02-13T00:56:24.112097+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":30,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":903,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796066+0000","last_change":"2024-02-13T00:55:57.950855+0000","last_active":"2024-02-13T00:56:55.796066+0000","last_peered":"2024-02-13T00:56:55.796066+0000","last_clean":"2024-02-13T00:56:55.796066+0000","last_became_active":"2024-02-13T00:55:00.638516+0000","last_became_peered":"2024-02-13T00:55:00.638516+0000","last_unstale":"2024-02-13T00:56:55.796066+0000","last_undegraded":"2024-02-13T00:56:55.796066+0000","last_fullsized":"2024-02-13T00:56:55.796066+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:57.950760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:57.950760+0000","last_clean_scrub_stamp":"2024-02-13T00:55:57.950760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"821'19","reported_seq":102,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.432318+0000","last_change":"2024-02-13T00:57:08.695233+0000","last_active":"2024-02-13T00:57:09.432318+0000","last_peered":"2024-02-13T00:57:09.432318+0000","last_clean":"2024-02-13T00:57:09.432318+0000","last_became_active":"2024-02-13T00:55:04.810233+0000","last_became_peered":"2024-02-13T00:55:04.810233+0000","last_unstale":"2024-02-13T00:57:09.432318+0000","last_undegraded":"2024-02-13T00:57:09.432318+0000","last_fullsized":"2024-02-13T00:57:09.432318+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"821'19","last_scrub_stamp":"2024-02-13T00:57:08.695165+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:57:08.695165+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":50,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:07.314282+0000","last_change":"2024-02-13T00:57:07.314254+0000","last_active":"2024-02-13T00:57:07.314282+0000","last_peered":"2024-02-13T00:57:07.314282+0000","last_clean":"2024-02-13T00:57:07.314282+0000","last_became_active":"2024-02-13T00:55:51.629259+0000","last_became_peered":"2024-02-13T00:55:51.629259+0000","last_unstale":"2024-02-13T00:57:07.314282+0000","last_undegraded":"2024-02-13T00:57:07.314282+0000","last_fullsized":"2024-02-13T00:57:07.314282+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:07.314182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:27.933639+0000","last_clean_scrub_stamp":"2024-02-13T00:57:07.314182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":73,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.844080+0000","last_change":"2024-02-13T00:56:55.844080+0000","last_active":"2024-02-13T00:56:55.844080+0000","last_peered":"2024-02-13T00:56:55.844080+0000","last_clean":"2024-02-13T00:56:55.844080+0000","last_became_active":"2024-02-13T00:55:00.623176+0000","last_became_peered":"2024-02-13T00:55:00.623176+0000","last_unstale":"2024-02-13T00:56:55.844080+0000","last_undegraded":"2024-02-13T00:56:55.844080+0000","last_fullsized":"2024-02-13T00:56:55.844080+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:55.843976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:55.843976+0000","last_clean_scrub_stamp":"2024-02-13T00:56:55.843976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":934,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.634410+0000","last_change":"2024-02-13T00:56:12.926290+0000","last_active":"2024-02-13T00:56:55.634410+0000","last_peered":"2024-02-13T00:56:55.634410+0000","last_clean":"2024-02-13T00:56:55.634410+0000","last_became_active":"2024-02-13T00:55:04.802878+0000","last_became_peered":"2024-02-13T00:55:04.802878+0000","last_unstale":"2024-02-13T00:56:55.634410+0000","last_undegraded":"2024-02-13T00:56:55.634410+0000","last_fullsized":"2024-02-13T00:56:55.634410+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'19","last_scrub_stamp":"2024-02-13T00:56:12.926213+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:56:12.926213+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":590,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:59.978989+0000","last_change":"2024-02-13T00:56:59.978959+0000","last_active":"2024-02-13T00:56:59.978989+0000","last_peered":"2024-02-13T00:56:59.978989+0000","last_clean":"2024-02-13T00:56:59.978989+0000","last_became_active":"2024-02-13T00:55:00.638271+0000","last_became_peered":"2024-02-13T00:55:00.638271+0000","last_unstale":"2024-02-13T00:56:59.978989+0000","last_undegraded":"2024-02-13T00:56:59.978989+0000","last_fullsized":"2024-02-13T00:56:59.978989+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:59.978875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:56:59.978875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":58,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.637070+0000","last_change":"2024-02-13T00:56:50.693498+0000","last_active":"2024-02-13T00:56:55.637070+0000","last_peered":"2024-02-13T00:56:55.637070+0000","last_clean":"2024-02-13T00:56:55.637070+0000","last_became_active":"2024-02-13T00:55:00.655453+0000","last_became_peered":"2024-02-13T00:55:00.655453+0000","last_unstale":"2024-02-13T00:56:55.637070+0000","last_undegraded":"2024-02-13T00:56:55.637070+0000","last_fullsized":"2024-02-13T00:56:55.637070+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:50.693436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:56:50.693436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":839,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:57.796565+0000","last_change":"2024-02-13T00:56:57.796543+0000","last_active":"2024-02-13T00:56:57.796565+0000","last_peered":"2024-02-13T00:56:57.796565+0000","last_clean":"2024-02-13T00:56:57.796565+0000","last_became_active":"2024-02-13T00:54:52.752154+0000","last_became_peered":"2024-02-13T00:54:52.752154+0000","last_unstale":"2024-02-13T00:56:57.796565+0000","last_undegraded":"2024-02-13T00:56:57.796565+0000","last_fullsized":"2024-02-13T00:56:57.796565+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:57.796480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:47.967067+0000","last_clean_scrub_stamp":"2024-02-13T00:56:57.796480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":566,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797503+0000","last_change":"2024-02-13T00:56:03.935576+0000","last_active":"2024-02-13T00:56:55.797503+0000","last_peered":"2024-02-13T00:56:55.797503+0000","last_clean":"2024-02-13T00:56:55.797503+0000","last_became_active":"2024-02-13T00:51:41.570732+0000","last_became_peered":"2024-02-13T00:51:41.570732+0000","last_unstale":"2024-02-13T00:56:55.797503+0000","last_undegraded":"2024-02-13T00:56:55.797503+0000","last_fullsized":"2024-02-13T00:56:55.797503+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:03.935473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:03.935473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":613,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796415+0000","last_change":"2024-02-13T00:56:42.111865+0000","last_active":"2024-02-13T00:56:55.796415+0000","last_peered":"2024-02-13T00:56:55.796415+0000","last_clean":"2024-02-13T00:56:55.796415+0000","last_became_active":"2024-02-13T00:55:51.633265+0000","last_became_peered":"2024-02-13T00:55:51.633265+0000","last_unstale":"2024-02-13T00:56:55.796415+0000","last_undegraded":"2024-02-13T00:56:55.796415+0000","last_fullsized":"2024-02-13T00:56:55.796415+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:56:42.111782+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:56:42.111782+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":849,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:06.158574+0000","last_change":"2024-02-13T00:57:06.158548+0000","last_active":"2024-02-13T00:57:06.158574+0000","last_peered":"2024-02-13T00:57:06.158574+0000","last_clean":"2024-02-13T00:57:06.158574+0000","last_became_active":"2024-02-13T00:54:52.737403+0000","last_became_peered":"2024-02-13T00:54:52.737403+0000","last_unstale":"2024-02-13T00:57:06.158574+0000","last_undegraded":"2024-02-13T00:57:06.158574+0000","last_fullsized":"2024-02-13T00:57:06.158574+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:06.158476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:57:06.158476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":881,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:03.112157+0000","last_change":"2024-02-13T00:57:03.112124+0000","last_active":"2024-02-13T00:57:03.112157+0000","last_peered":"2024-02-13T00:57:03.112157+0000","last_clean":"2024-02-13T00:57:03.112157+0000","last_became_active":"2024-02-13T00:55:00.636526+0000","last_became_peered":"2024-02-13T00:55:00.636526+0000","last_unstale":"2024-02-13T00:57:03.112157+0000","last_undegraded":"2024-02-13T00:57:03.112157+0000","last_fullsized":"2024-02-13T00:57:03.112157+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:03.112023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:57:03.112023+0000","last_clean_scrub_stamp":"2024-02-13T00:57:03.112023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.634262+0000","last_change":"2024-02-13T00:56:02.690748+0000","last_active":"2024-02-13T00:56:55.634262+0000","last_peered":"2024-02-13T00:56:55.634262+0000","last_clean":"2024-02-13T00:56:55.634262+0000","last_became_active":"2024-02-13T00:55:00.655473+0000","last_became_peered":"2024-02-13T00:55:00.655473+0000","last_unstale":"2024-02-13T00:56:55.634262+0000","last_undegraded":"2024-02-13T00:56:55.634262+0000","last_fullsized":"2024-02-13T00:56:55.634262+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:02.690640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:17.006804+0000","last_clean_scrub_stamp":"2024-02-13T00:56:02.690640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":590,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798585+0000","last_change":"2024-02-13T00:56:34.267342+0000","last_active":"2024-02-13T00:56:55.798585+0000","last_peered":"2024-02-13T00:56:55.798585+0000","last_clean":"2024-02-13T00:56:55.798585+0000","last_became_active":"2024-02-13T00:55:00.638786+0000","last_became_peered":"2024-02-13T00:55:00.638786+0000","last_unstale":"2024-02-13T00:56:55.798585+0000","last_undegraded":"2024-02-13T00:56:55.798585+0000","last_fullsized":"2024-02-13T00:56:55.798585+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:34.267245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:34.267245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":64,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.634039+0000","last_change":"2024-02-13T00:56:42.549442+0000","last_active":"2024-02-13T00:56:55.634039+0000","last_peered":"2024-02-13T00:56:55.634039+0000","last_clean":"2024-02-13T00:56:55.634039+0000","last_became_active":"2024-02-13T00:55:00.647721+0000","last_became_peered":"2024-02-13T00:55:00.647721+0000","last_unstale":"2024-02-13T00:56:55.634039+0000","last_undegraded":"2024-02-13T00:56:55.634039+0000","last_fullsized":"2024-02-13T00:56:55.634039+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'8","last_scrub_stamp":"2024-02-13T00:56:42.549360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:42.549360+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":68,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.880316+0000","last_change":"2024-02-13T00:56:31.005069+0000","last_active":"2024-02-13T00:56:51.880316+0000","last_peered":"2024-02-13T00:56:51.880316+0000","last_clean":"2024-02-13T00:56:51.880316+0000","last_became_active":"2024-02-13T00:55:04.781288+0000","last_became_peered":"2024-02-13T00:55:04.781288+0000","last_unstale":"2024-02-13T00:56:51.880316+0000","last_undegraded":"2024-02-13T00:56:51.880316+0000","last_fullsized":"2024-02-13T00:56:51.880316+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:31.004988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:56:31.004988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":581,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796693+0000","last_change":"2024-02-13T00:56:23.234395+0000","last_active":"2024-02-13T00:56:55.796693+0000","last_peered":"2024-02-13T00:56:55.796693+0000","last_clean":"2024-02-13T00:56:55.796693+0000","last_became_active":"2024-02-13T00:54:56.632657+0000","last_became_peered":"2024-02-13T00:54:56.632657+0000","last_unstale":"2024-02-13T00:56:55.796693+0000","last_undegraded":"2024-02-13T00:56:55.796693+0000","last_fullsized":"2024-02-13T00:56:55.796693+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:23.234306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:56:23.234306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":541,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798099+0000","last_change":"2024-02-13T00:56:40.085654+0000","last_active":"2024-02-13T00:56:55.798099+0000","last_peered":"2024-02-13T00:56:55.798099+0000","last_clean":"2024-02-13T00:56:55.798099+0000","last_became_active":"2024-02-13T00:56:07.653436+0000","last_became_peered":"2024-02-13T00:56:07.653436+0000","last_unstale":"2024-02-13T00:56:55.798099+0000","last_undegraded":"2024-02-13T00:56:55.798099+0000","last_fullsized":"2024-02-13T00:56:55.798099+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:40.085569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:56:40.085569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":64,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:59.762883+0000","last_change":"2024-02-13T00:56:59.762883+0000","last_active":"2024-02-13T00:56:59.762883+0000","last_peered":"2024-02-13T00:56:59.762883+0000","last_clean":"2024-02-13T00:56:59.762883+0000","last_became_active":"2024-02-13T00:54:52.746519+0000","last_became_peered":"2024-02-13T00:54:52.746519+0000","last_unstale":"2024-02-13T00:56:59.762883+0000","last_undegraded":"2024-02-13T00:56:59.762883+0000","last_fullsized":"2024-02-13T00:56:59.762883+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:59.762800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:59.762800+0000","last_clean_scrub_stamp":"2024-02-13T00:56:59.762800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"850'31","reported_seq":44,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.795522+0000","last_change":"2024-02-13T00:56:10.788469+0000","last_active":"2024-02-13T00:56:55.795522+0000","last_peered":"2024-02-13T00:56:55.795522+0000","last_clean":"2024-02-13T00:56:55.795522+0000","last_became_active":"2024-02-13T00:56:07.653569+0000","last_became_peered":"2024-02-13T00:56:07.653569+0000","last_unstale":"2024-02-13T00:56:55.795522+0000","last_undegraded":"2024-02-13T00:56:55.795522+0000","last_fullsized":"2024-02-13T00:56:55.795522+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"850'31","last_scrub_stamp":"2024-02-13T00:56:10.788366+0000","last_deep_scrub":"850'31","last_deep_scrub_stamp":"2024-02-13T00:56:10.788366+0000","last_clean_scrub_stamp":"2024-02-13T00:56:10.788366+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":581,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.799480+0000","last_change":"2024-02-13T00:56:16.045968+0000","last_active":"2024-02-13T00:56:55.799480+0000","last_peered":"2024-02-13T00:56:55.799480+0000","last_clean":"2024-02-13T00:56:55.799480+0000","last_became_active":"2024-02-13T00:55:04.781662+0000","last_became_peered":"2024-02-13T00:55:04.781662+0000","last_unstale":"2024-02-13T00:56:55.799480+0000","last_undegraded":"2024-02-13T00:56:55.799480+0000","last_fullsized":"2024-02-13T00:56:55.799480+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:16.045873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:16.045873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":550,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:08.806606+0000","last_change":"2024-02-13T00:57:08.806580+0000","last_active":"2024-02-13T00:57:08.806606+0000","last_peered":"2024-02-13T00:57:08.806606+0000","last_clean":"2024-02-13T00:57:08.806606+0000","last_became_active":"2024-02-13T00:56:07.650266+0000","last_became_peered":"2024-02-13T00:56:07.650266+0000","last_unstale":"2024-02-13T00:57:08.806606+0000","last_undegraded":"2024-02-13T00:57:08.806606+0000","last_fullsized":"2024-02-13T00:57:08.806606+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:08.806509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:57:08.806509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":58,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.637113+0000","last_change":"2024-02-13T00:56:18.713246+0000","last_active":"2024-02-13T00:56:55.637113+0000","last_peered":"2024-02-13T00:56:55.637113+0000","last_clean":"2024-02-13T00:56:55.637113+0000","last_became_active":"2024-02-13T00:55:00.655926+0000","last_became_peered":"2024-02-13T00:55:00.655926+0000","last_unstale":"2024-02-13T00:56:55.637113+0000","last_undegraded":"2024-02-13T00:56:55.637113+0000","last_fullsized":"2024-02-13T00:56:55.637113+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:18.713176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:56:18.713176+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":517,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796700+0000","last_change":"2024-02-13T00:56:28.119573+0000","last_active":"2024-02-13T00:56:55.796700+0000","last_peered":"2024-02-13T00:56:55.796700+0000","last_clean":"2024-02-13T00:56:55.796700+0000","last_became_active":"2024-02-13T00:56:07.651694+0000","last_became_peered":"2024-02-13T00:56:07.651694+0000","last_unstale":"2024-02-13T00:56:55.796700+0000","last_undegraded":"2024-02-13T00:56:55.796700+0000","last_fullsized":"2024-02-13T00:56:55.796700+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:28.119494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:28.119494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":998,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:01.850792+0000","last_change":"2024-02-13T00:57:01.850757+0000","last_active":"2024-02-13T00:57:01.850792+0000","last_peered":"2024-02-13T00:57:01.850792+0000","last_clean":"2024-02-13T00:57:01.850792+0000","last_became_active":"2024-02-13T00:55:00.638505+0000","last_became_peered":"2024-02-13T00:55:00.638505+0000","last_unstale":"2024-02-13T00:57:01.850792+0000","last_undegraded":"2024-02-13T00:57:01.850792+0000","last_fullsized":"2024-02-13T00:57:01.850792+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'32","last_scrub_stamp":"2024-02-13T00:57:01.850685+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:57:01.850685+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":68,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.879300+0000","last_change":"2024-02-13T00:56:04.218997+0000","last_active":"2024-02-13T00:56:51.879300+0000","last_peered":"2024-02-13T00:56:51.879300+0000","last_clean":"2024-02-13T00:56:51.879300+0000","last_became_active":"2024-02-13T00:54:56.630920+0000","last_became_peered":"2024-02-13T00:54:56.630920+0000","last_unstale":"2024-02-13T00:56:51.879300+0000","last_undegraded":"2024-02-13T00:56:51.879300+0000","last_fullsized":"2024-02-13T00:56:51.879300+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:04.218912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:04.218912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":495,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797131+0000","last_change":"2024-02-13T00:56:19.082248+0000","last_active":"2024-02-13T00:56:55.797131+0000","last_peered":"2024-02-13T00:56:55.797131+0000","last_clean":"2024-02-13T00:56:55.797131+0000","last_became_active":"2024-02-13T00:56:07.658268+0000","last_became_peered":"2024-02-13T00:56:07.658268+0000","last_unstale":"2024-02-13T00:56:55.797131+0000","last_undegraded":"2024-02-13T00:56:55.797131+0000","last_fullsized":"2024-02-13T00:56:55.797131+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:56:19.082165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:19.082165+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":816,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:07.144113+0000","last_change":"2024-02-13T00:57:07.144092+0000","last_active":"2024-02-13T00:57:07.144113+0000","last_peered":"2024-02-13T00:57:07.144113+0000","last_clean":"2024-02-13T00:57:07.144113+0000","last_became_active":"2024-02-13T00:56:07.650605+0000","last_became_peered":"2024-02-13T00:56:07.650605+0000","last_unstale":"2024-02-13T00:57:07.144113+0000","last_undegraded":"2024-02-13T00:57:07.144113+0000","last_fullsized":"2024-02-13T00:57:07.144113+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:07.144025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:57:07.144025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":640,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796072+0000","last_change":"2024-02-13T00:56:24.240884+0000","last_active":"2024-02-13T00:56:55.796072+0000","last_peered":"2024-02-13T00:56:55.796072+0000","last_clean":"2024-02-13T00:56:55.796072+0000","last_became_active":"2024-02-13T00:56:07.657465+0000","last_became_peered":"2024-02-13T00:56:07.657465+0000","last_unstale":"2024-02-13T00:56:55.796072+0000","last_undegraded":"2024-02-13T00:56:55.796072+0000","last_fullsized":"2024-02-13T00:56:55.796072+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:56:24.240803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:24.240803+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":53,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.634184+0000","last_change":"2024-02-13T00:56:12.718088+0000","last_active":"2024-02-13T00:56:55.634184+0000","last_peered":"2024-02-13T00:56:55.634184+0000","last_clean":"2024-02-13T00:56:55.634184+0000","last_became_active":"2024-02-13T00:55:00.655998+0000","last_became_peered":"2024-02-13T00:55:00.655998+0000","last_unstale":"2024-02-13T00:56:55.634184+0000","last_undegraded":"2024-02-13T00:56:55.634184+0000","last_fullsized":"2024-02-13T00:56:55.634184+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:12.718020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:12.718020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":861,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.635042+0000","last_change":"2024-02-13T00:56:48.070734+0000","last_active":"2024-02-13T00:56:55.635042+0000","last_peered":"2024-02-13T00:56:55.635042+0000","last_clean":"2024-02-13T00:56:55.635042+0000","last_became_active":"2024-02-13T00:54:56.623506+0000","last_became_peered":"2024-02-13T00:54:56.623506+0000","last_unstale":"2024-02-13T00:56:55.635042+0000","last_undegraded":"2024-02-13T00:56:55.635042+0000","last_fullsized":"2024-02-13T00:56:55.635042+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:48.070668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:23.031298+0000","last_clean_scrub_stamp":"2024-02-13T00:56:48.070668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":563,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797166+0000","last_change":"2024-02-13T00:55:50.874302+0000","last_active":"2024-02-13T00:56:55.797166+0000","last_peered":"2024-02-13T00:56:55.797166+0000","last_clean":"2024-02-13T00:56:55.797166+0000","last_became_active":"2024-02-13T00:55:04.793613+0000","last_became_peered":"2024-02-13T00:55:04.793613+0000","last_unstale":"2024-02-13T00:56:55.797166+0000","last_undegraded":"2024-02-13T00:56:55.797166+0000","last_fullsized":"2024-02-13T00:56:55.797166+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:50.874222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:50.874222+0000","last_clean_scrub_stamp":"2024-02-13T00:55:50.874222+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":73,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796176+0000","last_change":"2024-02-13T00:56:42.924806+0000","last_active":"2024-02-13T00:56:55.796176+0000","last_peered":"2024-02-13T00:56:55.796176+0000","last_clean":"2024-02-13T00:56:55.796176+0000","last_became_active":"2024-02-13T00:55:04.797172+0000","last_became_peered":"2024-02-13T00:55:04.797172+0000","last_unstale":"2024-02-13T00:56:55.796176+0000","last_undegraded":"2024-02-13T00:56:55.796176+0000","last_fullsized":"2024-02-13T00:56:55.796176+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:56:42.924722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:42.924722+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":97,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.635600+0000","last_change":"2024-02-13T00:56:39.979443+0000","last_active":"2024-02-13T00:56:55.635600+0000","last_peered":"2024-02-13T00:56:55.635600+0000","last_clean":"2024-02-13T00:56:55.635600+0000","last_became_active":"2024-02-13T00:54:56.623377+0000","last_became_peered":"2024-02-13T00:54:56.623377+0000","last_unstale":"2024-02-13T00:56:55.635600+0000","last_undegraded":"2024-02-13T00:56:55.635600+0000","last_fullsized":"2024-02-13T00:56:55.635600+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:39.979364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:39.979364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":558,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798482+0000","last_change":"2024-02-13T00:56:25.089788+0000","last_active":"2024-02-13T00:56:55.798482+0000","last_peered":"2024-02-13T00:56:55.798482+0000","last_clean":"2024-02-13T00:56:55.798482+0000","last_became_active":"2024-02-13T00:55:51.633676+0000","last_became_peered":"2024-02-13T00:55:51.633676+0000","last_unstale":"2024-02-13T00:56:55.798482+0000","last_undegraded":"2024-02-13T00:56:55.798482+0000","last_fullsized":"2024-02-13T00:56:55.798482+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:25.089713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:25.089713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":473,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:58.829821+0000","last_change":"2024-02-13T00:56:58.829795+0000","last_active":"2024-02-13T00:56:58.829821+0000","last_peered":"2024-02-13T00:56:58.829821+0000","last_clean":"2024-02-13T00:56:58.829821+0000","last_became_active":"2024-02-13T00:56:07.656780+0000","last_became_peered":"2024-02-13T00:56:07.656780+0000","last_unstale":"2024-02-13T00:56:58.829821+0000","last_undegraded":"2024-02-13T00:56:58.829821+0000","last_fullsized":"2024-02-13T00:56:58.829821+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:58.829723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:58.829723+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":72,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.637250+0000","last_change":"2024-02-13T00:56:52.701271+0000","last_active":"2024-02-13T00:56:55.637250+0000","last_peered":"2024-02-13T00:56:55.637250+0000","last_clean":"2024-02-13T00:56:55.637250+0000","last_became_active":"2024-02-13T00:55:00.648023+0000","last_became_peered":"2024-02-13T00:55:00.648023+0000","last_unstale":"2024-02-13T00:56:55.637250+0000","last_undegraded":"2024-02-13T00:56:55.637250+0000","last_fullsized":"2024-02-13T00:56:55.637250+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:56:52.701207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:52.701207+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":58,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:56.726651+0000","last_change":"2024-02-13T00:56:56.726615+0000","last_active":"2024-02-13T00:56:56.726651+0000","last_peered":"2024-02-13T00:56:56.726651+0000","last_clean":"2024-02-13T00:56:56.726651+0000","last_became_active":"2024-02-13T00:55:00.645985+0000","last_became_peered":"2024-02-13T00:55:00.645985+0000","last_unstale":"2024-02-13T00:56:56.726651+0000","last_undegraded":"2024-02-13T00:56:56.726651+0000","last_fullsized":"2024-02-13T00:56:56.726651+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:56.726543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:18.264738+0000","last_clean_scrub_stamp":"2024-02-13T00:56:56.726543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":53,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.636823+0000","last_change":"2024-02-13T00:56:05.757898+0000","last_active":"2024-02-13T00:56:55.636823+0000","last_peered":"2024-02-13T00:56:55.636823+0000","last_clean":"2024-02-13T00:56:55.636823+0000","last_became_active":"2024-02-13T00:55:00.637834+0000","last_became_peered":"2024-02-13T00:55:00.637834+0000","last_unstale":"2024-02-13T00:56:55.636823+0000","last_undegraded":"2024-02-13T00:56:55.636823+0000","last_fullsized":"2024-02-13T00:56:55.636823+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:05.757837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:05.757837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":822,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796018+0000","last_change":"2024-02-13T00:56:26.850032+0000","last_active":"2024-02-13T00:56:55.796018+0000","last_peered":"2024-02-13T00:56:55.796018+0000","last_clean":"2024-02-13T00:56:55.796018+0000","last_became_active":"2024-02-13T00:55:04.814928+0000","last_became_peered":"2024-02-13T00:55:04.814928+0000","last_unstale":"2024-02-13T00:56:55.796018+0000","last_undegraded":"2024-02-13T00:56:55.796018+0000","last_fullsized":"2024-02-13T00:56:55.796018+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:26.849966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:26.849966+0000","last_clean_scrub_stamp":"2024-02-13T00:56:26.849966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":740,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.321787+0000","last_change":"2024-02-13T00:57:09.321755+0000","last_active":"2024-02-13T00:57:09.321787+0000","last_peered":"2024-02-13T00:57:09.321787+0000","last_clean":"2024-02-13T00:57:09.321787+0000","last_became_active":"2024-02-13T00:55:51.632600+0000","last_became_peered":"2024-02-13T00:55:51.632600+0000","last_unstale":"2024-02-13T00:57:09.321787+0000","last_undegraded":"2024-02-13T00:57:09.321787+0000","last_fullsized":"2024-02-13T00:57:09.321787+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:57:09.321666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:57:09.321666+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":478,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797344+0000","last_change":"2024-02-13T00:56:18.105575+0000","last_active":"2024-02-13T00:56:55.797344+0000","last_peered":"2024-02-13T00:56:55.797344+0000","last_clean":"2024-02-13T00:56:55.797344+0000","last_became_active":"2024-02-13T00:56:07.657519+0000","last_became_peered":"2024-02-13T00:56:07.657519+0000","last_unstale":"2024-02-13T00:56:55.797344+0000","last_undegraded":"2024-02-13T00:56:55.797344+0000","last_fullsized":"2024-02-13T00:56:55.797344+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:18.105481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:18.105481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":841,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.797337+0000","last_change":"2024-02-13T00:56:39.825890+0000","last_active":"2024-02-13T00:56:55.797337+0000","last_peered":"2024-02-13T00:56:55.797337+0000","last_clean":"2024-02-13T00:56:55.797337+0000","last_became_active":"2024-02-13T00:51:41.582536+0000","last_became_peered":"2024-02-13T00:51:41.582536+0000","last_unstale":"2024-02-13T00:56:55.797337+0000","last_undegraded":"2024-02-13T00:56:55.797337+0000","last_fullsized":"2024-02-13T00:56:55.797337+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:39.825802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:56:39.825802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":64,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.879794+0000","last_change":"2024-02-13T00:56:41.069838+0000","last_active":"2024-02-13T00:56:51.879794+0000","last_peered":"2024-02-13T00:56:51.879794+0000","last_clean":"2024-02-13T00:56:51.879794+0000","last_became_active":"2024-02-13T00:54:52.751112+0000","last_became_peered":"2024-02-13T00:54:52.751112+0000","last_unstale":"2024-02-13T00:56:51.879794+0000","last_undegraded":"2024-02-13T00:56:51.879794+0000","last_fullsized":"2024-02-13T00:56:51.879794+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:41.069740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:21.226050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:41.069740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":741,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796237+0000","last_change":"2024-02-13T00:56:55.204514+0000","last_active":"2024-02-13T00:56:55.796237+0000","last_peered":"2024-02-13T00:56:55.796237+0000","last_clean":"2024-02-13T00:56:55.796237+0000","last_became_active":"2024-02-13T00:55:51.633205+0000","last_became_peered":"2024-02-13T00:55:51.633205+0000","last_unstale":"2024-02-13T00:56:55.796237+0000","last_undegraded":"2024-02-13T00:56:55.796237+0000","last_fullsized":"2024-02-13T00:56:55.796237+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'16","last_scrub_stamp":"2024-02-13T00:56:55.204417+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:56:55.204417+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":403,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:03.783692+0000","last_change":"2024-02-13T00:57:03.783659+0000","last_active":"2024-02-13T00:57:03.783692+0000","last_peered":"2024-02-13T00:57:03.783692+0000","last_clean":"2024-02-13T00:57:03.783692+0000","last_became_active":"2024-02-13T00:56:07.837581+0000","last_became_peered":"2024-02-13T00:56:07.837581+0000","last_unstale":"2024-02-13T00:57:03.783692+0000","last_undegraded":"2024-02-13T00:57:03.783692+0000","last_fullsized":"2024-02-13T00:57:03.783692+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:03.783575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:57:03.783575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":68,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-13T00:56:51.879444+0000","last_change":"2024-02-13T00:56:28.013442+0000","last_active":"2024-02-13T00:56:51.879444+0000","last_peered":"2024-02-13T00:56:51.879444+0000","last_clean":"2024-02-13T00:56:51.879444+0000","last_became_active":"2024-02-13T00:55:00.635557+0000","last_became_peered":"2024-02-13T00:55:00.635557+0000","last_unstale":"2024-02-13T00:56:51.879444+0000","last_undegraded":"2024-02-13T00:56:51.879444+0000","last_fullsized":"2024-02-13T00:56:51.879444+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:28.013343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:56:28.013343+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":554,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798667+0000","last_change":"2024-02-13T00:55:57.809002+0000","last_active":"2024-02-13T00:56:55.798667+0000","last_peered":"2024-02-13T00:56:55.798667+0000","last_clean":"2024-02-13T00:56:55.798667+0000","last_became_active":"2024-02-13T00:55:00.637611+0000","last_became_peered":"2024-02-13T00:55:00.637611+0000","last_unstale":"2024-02-13T00:56:55.798667+0000","last_undegraded":"2024-02-13T00:56:55.798667+0000","last_fullsized":"2024-02-13T00:56:55.798667+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:57.808891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:55:57.808891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":610,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.796071+0000","last_change":"2024-02-13T00:56:53.227063+0000","last_active":"2024-02-13T00:56:55.796071+0000","last_peered":"2024-02-13T00:56:55.796071+0000","last_clean":"2024-02-13T00:56:55.796071+0000","last_became_active":"2024-02-13T00:55:51.635798+0000","last_became_peered":"2024-02-13T00:55:51.635798+0000","last_unstale":"2024-02-13T00:56:55.796071+0000","last_undegraded":"2024-02-13T00:56:55.796071+0000","last_fullsized":"2024-02-13T00:56:55.796071+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:56:53.226982+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:56:53.226982+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":53,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.636844+0000","last_change":"2024-02-13T00:56:00.689172+0000","last_active":"2024-02-13T00:56:55.636844+0000","last_peered":"2024-02-13T00:56:55.636844+0000","last_clean":"2024-02-13T00:56:55.636844+0000","last_became_active":"2024-02-13T00:55:00.655317+0000","last_became_peered":"2024-02-13T00:55:00.655317+0000","last_unstale":"2024-02-13T00:56:55.636844+0000","last_undegraded":"2024-02-13T00:56:55.636844+0000","last_fullsized":"2024-02-13T00:56:55.636844+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:00.689104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:56:00.689104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":45,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.798242+0000","last_change":"2024-02-13T00:56:10.358031+0000","last_active":"2024-02-13T00:56:55.798242+0000","last_peered":"2024-02-13T00:56:55.798242+0000","last_clean":"2024-02-13T00:56:55.798242+0000","last_became_active":"2024-02-13T00:55:51.631587+0000","last_became_peered":"2024-02-13T00:55:51.631587+0000","last_unstale":"2024-02-13T00:56:55.798242+0000","last_undegraded":"2024-02-13T00:56:55.798242+0000","last_fullsized":"2024-02-13T00:56:55.798242+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:10.357948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:10.357948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":805,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.635037+0000","last_change":"2024-02-13T00:55:42.047116+0000","last_active":"2024-02-13T00:56:55.635037+0000","last_peered":"2024-02-13T00:56:55.635037+0000","last_clean":"2024-02-13T00:56:55.635037+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:56:55.635037+0000","last_undegraded":"2024-02-13T00:56:55.635037+0000","last_fullsized":"2024-02-13T00:56:55.635037+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:55:42.047043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:55:42.047043+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":99,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.634104+0000","last_change":"2024-02-13T00:55:53.589892+0000","last_active":"2024-02-13T00:56:55.634104+0000","last_peered":"2024-02-13T00:56:55.634104+0000","last_clean":"2024-02-13T00:56:55.634104+0000","last_became_active":"2024-02-13T00:55:04.810397+0000","last_became_peered":"2024-02-13T00:55:04.810397+0000","last_unstale":"2024-02-13T00:56:55.634104+0000","last_undegraded":"2024-02-13T00:56:55.634104+0000","last_fullsized":"2024-02-13T00:56:55.634104+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":822,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'62","last_scrub_stamp":"2024-02-13T00:55:53.589818+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:55:53.589818+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":863,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:57:04.930351+0000","last_change":"2024-02-13T00:57:04.930293+0000","last_active":"2024-02-13T00:57:04.930351+0000","last_peered":"2024-02-13T00:57:04.930351+0000","last_clean":"2024-02-13T00:57:04.930351+0000","last_became_active":"2024-02-13T00:54:56.636095+0000","last_became_peered":"2024-02-13T00:54:56.636095+0000","last_unstale":"2024-02-13T00:57:04.930351+0000","last_undegraded":"2024-02-13T00:57:04.930351+0000","last_fullsized":"2024-02-13T00:57:04.930351+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:04.930154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:57:04.930154+0000","last_clean_scrub_stamp":"2024-02-13T00:57:04.930154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":58,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-13T00:56:55.636862+0000","last_change":"2024-02-13T00:56:54.684002+0000","last_active":"2024-02-13T00:56:55.636862+0000","last_peered":"2024-02-13T00:56:55.636862+0000","last_clean":"2024-02-13T00:56:55.636862+0000","last_became_active":"2024-02-13T00:55:00.645669+0000","last_became_peered":"2024-02-13T00:55:00.645669+0000","last_unstale":"2024-02-13T00:56:55.636862+0000","last_undegraded":"2024-02-13T00:56:55.636862+0000","last_fullsized":"2024-02-13T00:56:55.636862+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:54.683930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:54.683930+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1488,"num_read_kb":1213,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1018,"num_bytes_recovered":18210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1022,"ondisk_log_size":1022,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1152,"num_read_kb":1147,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":3215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266925,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304064,"kb_used_data":6840,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441088,"statfs":{"total":95995035648,"available":95683674112,"internally_reserved":0,"allocated":7004160,"data_stored":3734728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.50900000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":10.682},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.499,"15min":0.73799999999999999},"min":{"1min":0.29599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.751,"5min":0.77400000000000002,"15min":25.504000000000001},"last":0.91000000000000003}]},{"osd":1,"last update":"Tue Feb 13 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.52100000000000002,"15min":0.58499999999999996},"min":{"1min":0.441,"5min":0.255,"15min":0.255},"max":{"1min":0.77600000000000002,"5min":2.661,"15min":13.347},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51500000000000001,"15min":0.65900000000000003},"min":{"1min":0.36099999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69999999999999996,"5min":1.141,"15min":13.526999999999999},"last":0.87}]},{"osd":2,"last update":"Tue Feb 13 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51000000000000001,"15min":0.64800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26600000000000001},"max":{"1min":0.80100000000000005,"5min":0.90100000000000002,"15min":14.359},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.52500000000000002,"15min":0.73499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32000000000000001,"15min":0.27400000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":21.497},"last":0.89900000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51000000000000001,"15min":0.69399999999999995},"min":{"1min":0.40200000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76700000000000002,"5min":0.89400000000000002,"15min":12.997},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.53400000000000003,"15min":0.80400000000000005},"min":{"1min":0.372,"5min":0.35399999999999998,"15min":0.33200000000000002},"max":{"1min":0.72999999999999998,"5min":0.79300000000000004,"15min":21.622},"last":0.86299999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44600000000000001,"15min":0.436},"min":{"1min":0.32000000000000001,"5min":0.27300000000000002,"15min":0.23000000000000001},"max":{"1min":0.56100000000000005,"5min":0.70499999999999996,"15min":2.452},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.47499999999999998,"15min":0.45400000000000001},"min":{"1min":0.33500000000000002,"5min":0.29499999999999998,"15min":0.245},"max":{"1min":0.64300000000000002,"5min":0.83199999999999996,"15min":1.0780000000000001},"last":0.502}]},{"osd":5,"last update":"Tue Feb 13 00:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48199999999999998,"15min":0.499},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66700000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.49099999999999999,"15min":0.51200000000000001},"min":{"1min":0.41399999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72299999999999998,"5min":0.747,"15min":0.79300000000000004},"last":0.55400000000000005}]},{"osd":6,"last update":"Tue Feb 13 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51300000000000001,"15min":0.497},"min":{"1min":0.40500000000000003,"5min":0.31,"15min":0.25},"max":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.81299999999999994},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51500000000000001,"15min":0.496},"min":{"1min":0.38800000000000001,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.65000000000000002,"5min":0.85499999999999998,"15min":1.0940000000000001},"last":0.60299999999999998}]}]},{"osd":6,"up_from":45,"seq":193273528563,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312256,"kb_used_data":6840,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432896,"statfs":{"total":95995035648,"available":95675285504,"internally_reserved":0,"allocated":7004160,"data_stored":3733504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.504,"15min":0.53700000000000003},"min":{"1min":0.27500000000000002,"5min":0.19800000000000001,"15min":0.19700000000000001},"max":{"1min":0.78800000000000003,"5min":1.0609999999999999,"15min":8.1340000000000003},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.499,"5min":0.501,"15min":0.61799999999999999},"min":{"1min":0.34300000000000003,"5min":0.27400000000000002,"15min":0.221},"max":{"1min":0.63500000000000001,"5min":0.872,"15min":11.367000000000001},"last":0.54100000000000004}]},{"osd":1,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.67800000000000005},"min":{"1min":0.32500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.03,"5min":1.03,"15min":24.209},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.63800000000000001},"min":{"1min":0.30399999999999999,"5min":0.27900000000000003,"15min":0.23499999999999999},"max":{"1min":0.68799999999999994,"5min":0.79900000000000004,"15min":12.403},"last":0.58299999999999996}]},{"osd":2,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51100000000000001,"15min":0.67000000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.25600000000000001},"max":{"1min":0.71199999999999997,"5min":0.83799999999999997,"15min":12.477},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49399999999999999,"15min":0.66000000000000003},"min":{"1min":0.38800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.60399999999999998,"5min":1.032,"15min":11.702},"last":0.55800000000000005}]},{"osd":3,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52100000000000002,"15min":0.65000000000000002},"min":{"1min":0.36099999999999999,"5min":0.309,"15min":0.29199999999999998},"max":{"1min":1.165,"5min":1.165,"15min":8.0519999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56899999999999995,"15min":0.69799999999999995},"min":{"1min":0.41199999999999998,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":0.66500000000000004,"5min":3.0219999999999998,"15min":13.257},"last":0.60899999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.48699999999999999,"15min":0.439},"min":{"1min":0.318,"5min":0.26600000000000001,"15min":0.249},"max":{"1min":0.62,"5min":5.3840000000000003,"15min":5.3840000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.505,"15min":0.45900000000000002},"min":{"1min":0.39200000000000002,"5min":0.27500000000000002,"15min":0.221},"max":{"1min":0.69399999999999995,"5min":1.927,"15min":1.927},"last":0.45200000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.49299999999999999,"15min":0.50700000000000001},"min":{"1min":0.313,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71899999999999997,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.51800000000000002},"min":{"1min":0.32500000000000001,"5min":0.30199999999999999,"15min":0.27300000000000002},"max":{"1min":0.69599999999999995,"5min":0.876,"15min":0.876},"last":0.41299999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:57:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.497,"15min":0.47399999999999998},"min":{"1min":0.41799999999999998,"5min":0.30399999999999999,"15min":0.27200000000000002},"max":{"1min":0.72999999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.439},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53700000000000003,"15min":0.52100000000000002},"min":{"1min":0.32900000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.80700000000000005},"last":0.49299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575116,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304116,"kb_used_data":6892,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441036,"statfs":{"total":95995035648,"available":95683620864,"internally_reserved":0,"allocated":7057408,"data_stored":3742508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36199999999999999,"15min":0.80000000000000004},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.187},"max":{"1min":0.52300000000000002,"5min":1.657,"15min":61.856000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.33700000000000002,"15min":0.64000000000000001},"min":{"1min":0.26100000000000001,"5min":0.224,"15min":0.16500000000000001},"max":{"1min":0.48699999999999999,"5min":0.74399999999999999,"15min":61.908999999999999},"last":0.307}]},{"osd":2,"last update":"Tue Feb 13 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33600000000000002,"15min":0.83299999999999996},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":0.52700000000000002,"5min":0.629,"15min":32.161000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.435,"15min":0.83499999999999996},"min":{"1min":0.255,"5min":0.23799999999999999,"15min":0.187},"max":{"1min":0.51000000000000001,"5min":5.1319999999999997,"15min":29.693000000000001},"last":0.32400000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.379,"15min":1.093},"min":{"1min":0.27300000000000002,"5min":0.26900000000000002,"15min":0.219},"max":{"1min":0.57899999999999996,"5min":0.75700000000000001,"15min":131.23099999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.39200000000000002,"15min":0.77800000000000002},"min":{"1min":0.29599999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":0.54700000000000004,"5min":0.753,"15min":49.055},"last":0.36299999999999999}]},{"osd":4,"last update":"Tue Feb 13 00:57:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.57299999999999995,"5min":0.35399999999999998,"15min":0.27900000000000003},"max":{"1min":0.88,"5min":0.88,"15min":7.2089999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.63,"15min":0.63800000000000001},"min":{"1min":0.63400000000000001,"5min":0.36099999999999999,"15min":0.24199999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":10.050000000000001},"last":0.65800000000000003}]},{"osd":5,"last update":"Tue Feb 13 00:56:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58999999999999997,"15min":0.57899999999999996},"min":{"1min":0.42499999999999999,"5min":0.41299999999999998,"15min":0.39400000000000002},"max":{"1min":0.92600000000000005,"5min":0.96699999999999997,"15min":1.7889999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.33400000000000002},"max":{"1min":0.85799999999999998,"5min":0.86099999999999999,"15min":1.831},"last":0.751}]},{"osd":6,"last update":"Tue Feb 13 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.58199999999999996,"15min":0.66000000000000003},"min":{"1min":0.504,"5min":0.33000000000000002,"15min":0.29099999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":14.772},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.58499999999999996,"15min":0.78000000000000003},"min":{"1min":0.50900000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.871,"5min":0.871,"15min":26.600000000000001},"last":0.76900000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.59199999999999997,"15min":0.72199999999999998},"min":{"1min":0.44400000000000001,"5min":0.38300000000000001,"15min":0.29499999999999998},"max":{"1min":0.84799999999999998,"5min":0.94399999999999995,"15min":14.794},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.623,"5min":0.58099999999999996,"15min":0.91300000000000003},"min":{"1min":0.375,"5min":0.373,"15min":0.32200000000000001},"max":{"1min":0.87,"5min":0.87,"15min":26.693999999999999},"last":0.71999999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738640,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308160,"kb_used_data":6840,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436992,"statfs":{"total":95995035648,"available":95679479808,"internally_reserved":0,"allocated":7004160,"data_stored":3733181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.38700000000000001,"15min":0.624},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.40400000000000003,"5min":5.5250000000000004,"15min":30.184000000000001},"last":0.497},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.42599999999999999,"15min":0.44600000000000001},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20300000000000001},"max":{"1min":0.46700000000000003,"5min":3.125,"15min":5.6959999999999997},"last":0.44800000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.39700000000000002,"15min":0.48699999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.216},"max":{"1min":2.286,"5min":2.286,"15min":7.0839999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.40600000000000003,"15min":0.67500000000000004},"min":{"1min":0.26600000000000001,"5min":0.23300000000000001,"15min":0.20899999999999999},"max":{"1min":0.48399999999999999,"5min":2.1419999999999999,"15min":24.459},"last":0.46000000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:56:31 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.377,"15min":0.47199999999999998},"min":{"1min":0.28599999999999998,"5min":0.214,"15min":0.214},"max":{"1min":0.50800000000000001,"5min":1.0640000000000001,"15min":8.718},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.53300000000000003},"min":{"1min":0.27800000000000002,"5min":0.27700000000000002,"15min":0.222},"max":{"1min":2.4209999999999998,"5min":3.0419999999999998,"15min":7.1139999999999999},"last":0.53000000000000003}]},{"osd":4,"last update":"Tue Feb 13 00:57:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56299999999999994,"15min":0.65100000000000002},"min":{"1min":0.41199999999999998,"5min":0.31,"15min":0.29599999999999999},"max":{"1min":0.73299999999999998,"5min":1.552,"15min":9.2400000000000002},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59299999999999997,"15min":0.66400000000000003},"min":{"1min":0.47799999999999998,"5min":0.34399999999999997,"15min":0.28699999999999998},"max":{"1min":0.67600000000000005,"5min":2.6339999999999999,"15min":8.6950000000000003},"last":0.63100000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54500000000000004,"15min":0.51700000000000002},"min":{"1min":0.42599999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.79000000000000004,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58399999999999996,"15min":0.55400000000000005},"min":{"1min":0.41599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73299999999999998,"5min":3.3450000000000002,"15min":3.3450000000000002},"last":0.72299999999999998}]},{"osd":6,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58699999999999997,"15min":0.64200000000000002},"min":{"1min":0.41699999999999998,"5min":0.36899999999999999,"15min":0.32900000000000001},"max":{"1min":0.69599999999999995,"5min":3.399,"15min":8.6129999999999995},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.65300000000000002,"15min":0.67400000000000004},"min":{"1min":0.44400000000000001,"5min":0.38200000000000001,"15min":0.32400000000000001},"max":{"1min":0.74099999999999999,"5min":7.2969999999999997,"15min":14.263},"last":0.76800000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:57:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59099999999999997,"15min":0.62},"min":{"1min":0.46200000000000002,"5min":0.33900000000000002,"15min":0.317},"max":{"1min":0.745,"5min":3.375,"15min":7.7720000000000002},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60699999999999998,"15min":0.65300000000000002},"min":{"1min":0.441,"5min":0.377,"15min":0.35899999999999999},"max":{"1min":0.75600000000000001,"5min":3.101,"15min":8.1609999999999996},"last":0.74399999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411591,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312252,"kb_used_data":6836,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432900,"statfs":{"total":95995035648,"available":95675289600,"internally_reserved":0,"allocated":7000064,"data_stored":3733663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.33400000000000002,"15min":0.496},"min":{"1min":0.248,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.47499999999999998,"5min":0.55200000000000005,"15min":9.7769999999999992},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.33700000000000002,"15min":0.52500000000000002},"min":{"1min":0.19600000000000001,"5min":0.191,"15min":0.17499999999999999},"max":{"1min":0.46999999999999997,"5min":0.68799999999999994,"15min":11.351000000000001},"last":0.25900000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36899999999999999,"15min":0.48999999999999999},"min":{"1min":0.248,"5min":0.219,"15min":0.17199999999999999},"max":{"1min":0.496,"5min":1.843,"15min":10.141},"last":0.311},{"interface":"front","average":{"1min":0.38,"5min":0.36599999999999999,"15min":0.49099999999999999},"min":{"1min":0.27300000000000002,"5min":0.221,"15min":0.21299999999999999},"max":{"1min":0.48699999999999999,"5min":0.65400000000000003,"15min":9.0809999999999995},"last":0.28199999999999997}]},{"osd":3,"last update":"Tue Feb 13 00:56:32 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40200000000000002,"15min":0.51200000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.54200000000000004,"5min":1.875,"15min":11.807},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.54500000000000004},"min":{"1min":0.29199999999999998,"5min":0.255,"15min":0.223},"max":{"1min":0.51000000000000001,"5min":0.65300000000000002,"15min":16.981000000000002},"last":0.35199999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.29099999999999998},"max":{"1min":0.85399999999999998,"5min":2.4630000000000001,"15min":6.1420000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.623,"5min":0.60499999999999998,"15min":0.64400000000000002},"min":{"1min":0.439,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86799999999999999,"5min":1.99,"15min":8.9640000000000004},"last":0.68400000000000005}]},{"osd":5,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54300000000000004,"15min":0.55500000000000005},"min":{"1min":0.37,"5min":0.309,"15min":0.309},"max":{"1min":0.79600000000000004,"5min":0.87,"15min":0.87},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58699999999999997,"15min":0.57899999999999996},"min":{"1min":0.40500000000000003,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.63200000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58199999999999996,"15min":0.71899999999999997},"min":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.317},"max":{"1min":0.84999999999999998,"5min":2.1930000000000001,"15min":13.382},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59499999999999997,"15min":0.70499999999999996},"min":{"1min":0.47499999999999998,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.74399999999999999,"5min":3.3300000000000001,"15min":9.5670000000000002},"last":0.78500000000000003}]},{"osd":7,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58099999999999996,"15min":0.71699999999999997},"min":{"1min":0.38,"5min":0.35499999999999998,"15min":0.33800000000000002},"max":{"1min":0.78400000000000003,"5min":3.1970000000000001,"15min":9.6099999999999994},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59199999999999997,"15min":0.63800000000000001},"min":{"1min":0.46000000000000002,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.82899999999999996,"5min":1.927,"15min":6.1829999999999998},"last":0.66800000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182657,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304084,"kb_used_data":6860,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441068,"statfs":{"total":95995035648,"available":95683653632,"internally_reserved":0,"allocated":7024640,"data_stored":3734130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.372,"15min":0.72399999999999998},"min":{"1min":0.214,"5min":0.186,"15min":0.186},"max":{"1min":0.51400000000000001,"5min":3.9550000000000001,"15min":42.459000000000003},"last":0.372},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.376,"15min":0.746},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55300000000000005,"5min":1.9450000000000001,"15min":48.088999999999999},"last":0.38300000000000001}]},{"osd":1,"last update":"Tue Feb 13 00:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.39600000000000002,"15min":0.67200000000000004},"min":{"1min":0.247,"5min":0.23499999999999999,"15min":0.22},"max":{"1min":0.60899999999999999,"5min":1.97,"15min":47.999000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.40699999999999997,"15min":0.74199999999999999},"min":{"1min":0.23100000000000001,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":0.54000000000000004,"5min":3.7749999999999999,"15min":32.973999999999997},"last":0.35899999999999999}]},{"osd":2,"last update":"Tue Feb 13 00:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41099999999999998,"15min":0.70699999999999996},"min":{"1min":0.28399999999999997,"5min":0.23999999999999999,"15min":0.214},"max":{"1min":0.60499999999999998,"5min":1.9279999999999999,"15min":29.661999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.42399999999999999,"15min":0.67600000000000005},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23100000000000001},"max":{"1min":0.54800000000000004,"5min":4.218,"15min":35.256999999999998},"last":0.40699999999999997}]},{"osd":4,"last update":"Tue Feb 13 00:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.59399999999999997,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.314},"max":{"1min":0.69299999999999995,"5min":4.1289999999999996,"15min":8.968},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.58299999999999996,"15min":0.60899999999999999},"min":{"1min":0.41299999999999998,"5min":0.33600000000000002,"15min":0.27800000000000002},"max":{"1min":0.73999999999999999,"5min":1.8899999999999999,"15min":5.9649999999999999},"last":0.59399999999999997}]},{"osd":5,"last update":"Tue Feb 13 00:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.64300000000000002,"15min":0.57799999999999996},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.90200000000000002,"5min":4.077,"15min":4.077},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58099999999999996,"15min":0.57899999999999996},"min":{"1min":0.40300000000000002,"5min":0.35999999999999999,"15min":0.32200000000000001},"max":{"1min":0.80600000000000005,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.68999999999999995}]},{"osd":6,"last update":"Tue Feb 13 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.57299999999999995,"15min":0.80000000000000004},"min":{"1min":0.41299999999999998,"5min":0.34699999999999998,"15min":0.28799999999999998},"max":{"1min":0.81200000000000006,"5min":1.2210000000000001,"15min":31.457999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.66300000000000003,"15min":0.72699999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.35999999999999999},"max":{"1min":0.75700000000000001,"5min":4.1150000000000002,"15min":13.603999999999999},"last":0.76000000000000001}]},{"osd":7,"last update":"Tue Feb 13 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.68000000000000005,"15min":0.88900000000000001},"min":{"1min":0.437,"5min":0.38700000000000001,"15min":0.376},"max":{"1min":0.85499999999999998,"5min":4.1870000000000003,"15min":35.234000000000002},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.62,"15min":0.66800000000000004},"min":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.379},"max":{"1min":0.72399999999999998,"5min":3.1080000000000001,"15min":5.9320000000000004},"last":0.61299999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986429,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316360,"kb_used_data":6848,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428792,"statfs":{"total":95995035648,"available":95671083008,"internally_reserved":0,"allocated":7012352,"data_stored":3734216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50700000000000001,"15min":0.59099999999999997},"min":{"1min":0.34899999999999998,"5min":0.27100000000000002,"15min":0.21199999999999999},"max":{"1min":0.65100000000000002,"5min":1.9470000000000001,"15min":9.3399999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.56699999999999995,"15min":0.623},"min":{"1min":0.314,"5min":0.25700000000000001,"15min":0.20799999999999999},"max":{"1min":0.72899999999999998,"5min":4.3849999999999998,"15min":13.833},"last":0.59099999999999997}]},{"osd":1,"last update":"Tue Feb 13 00:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.746},"min":{"1min":0.34200000000000003,"5min":0.28899999999999998,"15min":0.25},"max":{"1min":0.71099999999999997,"5min":0.83199999999999996,"15min":53.188000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.53600000000000003,"15min":0.70299999999999996},"min":{"1min":0.40899999999999997,"5min":0.27500000000000002,"15min":0.25600000000000001},"max":{"1min":0.68400000000000005,"5min":2.1629999999999998,"15min":44.793999999999997},"last":0.51100000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55800000000000005,"15min":0.60399999999999998},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.26900000000000002},"max":{"1min":0.66600000000000004,"5min":2.1930000000000001,"15min":8.6829999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58499999999999996,"15min":0.64900000000000002},"min":{"1min":0.40000000000000002,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":0.68899999999999995,"5min":4.3310000000000004,"15min":18.911000000000001},"last":0.56399999999999995}]},{"osd":3,"last update":"Tue Feb 13 00:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59099999999999997,"15min":0.66800000000000004},"min":{"1min":0.35999999999999999,"5min":0.313,"15min":0.30599999999999999},"max":{"1min":0.67200000000000004,"5min":4.2450000000000001,"15min":23.5},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53900000000000003,"15min":0.71599999999999997},"min":{"1min":0.46300000000000002,"5min":0.36699999999999999,"15min":0.30599999999999999},"max":{"1min":0.75800000000000001,"5min":0.83799999999999997,"15min":36.152999999999999},"last":0.5}]},{"osd":5,"last update":"Tue Feb 13 00:56:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53300000000000003,"15min":0.48499999999999999},"min":{"1min":0.36899999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72899999999999998,"5min":4.306,"15min":4.306},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52000000000000002,"15min":0.47099999999999997},"min":{"1min":0.34899999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73999999999999999,"5min":2.367,"15min":2.367},"last":0.40300000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49399999999999999,"15min":0.46400000000000002},"min":{"1min":0.32500000000000001,"5min":0.27800000000000002,"15min":0.25700000000000001},"max":{"1min":0.68899999999999995,"5min":0.81399999999999995,"15min":3.153},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55000000000000004,"15min":0.50900000000000001},"min":{"1min":0.40400000000000003,"5min":0.29099999999999998,"15min":0.254},"max":{"1min":0.73099999999999998,"5min":2.48,"15min":4.9390000000000001},"last":0.55200000000000005}]},{"osd":7,"last update":"Tue Feb 13 00:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56599999999999995,"15min":0.51500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.29399999999999998},"max":{"1min":0.76400000000000001,"5min":3.024,"15min":3.024},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.61099999999999999,"15min":0.53000000000000003},"min":{"1min":0.38600000000000001,"5min":0.29299999999999998,"15min":0.28100000000000003},"max":{"1min":0.80700000000000005,"5min":4.3650000000000002,"15min":4.3650000000000002},"last":0.51900000000000002}]}]},{"osd":5,"up_from":479,"seq":2057289334895,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305208,"kb_used_data":6832,"kb_used_omap":42,"kb_used_meta":298325,"kb_avail":93439944,"statfs":{"total":95995035648,"available":95682502656,"internally_reserved":0,"allocated":6995968,"data_stored":3734216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43373,"internal_metadata":305485459},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.48799999999999999,"15min":0.47399999999999998},"min":{"1min":0.371,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.498},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.51200000000000001,"15min":0.48499999999999999},"min":{"1min":0.39000000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.88300000000000001},"last":0.56599999999999995}]},{"osd":1,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53700000000000003,"15min":0.52900000000000003},"min":{"1min":0.441,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.93100000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.55100000000000005,"15min":0.52800000000000002},"min":{"1min":0.46000000000000002,"5min":0.32300000000000001,"15min":0.28599999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.85199999999999998},"last":0.60399999999999998}]},{"osd":2,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.51900000000000002,"15min":0.51200000000000001},"min":{"1min":0.39400000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.76600000000000001,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.499},"min":{"1min":0.33600000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.44800000000000001}]},{"osd":3,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.5},"min":{"1min":0.378,"5min":0.29299999999999998,"15min":0.248},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.54300000000000004,"15min":0.53900000000000003},"min":{"1min":0.41799999999999998,"5min":0.33300000000000002,"15min":0.32700000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.64100000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.32600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.623,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.52300000000000002},"min":{"1min":0.40600000000000003,"5min":0.33700000000000002,"15min":0.32300000000000001},"max":{"1min":0.70199999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.67800000000000005}]},{"osd":6,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47199999999999998,"15min":0.45100000000000001},"min":{"1min":0.34399999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65900000000000003,"5min":0.80800000000000005,"15min":1.2390000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.48399999999999999,"15min":0.47099999999999997},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.248},"max":{"1min":0.70199999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55600000000000005}]},{"osd":7,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.50900000000000001,"15min":0.5},"min":{"1min":0.34200000000000003,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":0.72499999999999998,"5min":0.82699999999999996,"15min":0.97299999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.57699999999999996,"15min":0.55200000000000005},"min":{"1min":0.38800000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":0.88800000000000001,"15min":0.89400000000000002},"last":0.68899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:57:14.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T00:57:14.758 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:57:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:13.137451+0000 osd.1 (osd.1) 437 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:57:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:13.139338+0000 osd.1 (osd.1) 438 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:57:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:13.594507+0000 mgr.y (mgr.14544) 772 : audit [DBG] from='client.38558 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:57:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:13.787428+0000 osd.5 (osd.5) 84 : cluster [DBG] 5.7 deep-scrub starts 2024-02-13T00:57:15.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:13.790303+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.7 deep-scrub ok 2024-02-13T00:57:15.304 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:13.137451+0000 osd.1 (osd.1) 437 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:57:15.304 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:13.139338+0000 osd.1 (osd.1) 438 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:57:15.304 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:13.594507+0000 mgr.y (mgr.14544) 772 : audit [DBG] from='client.38558 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:57:15.305 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:13.787428+0000 osd.5 (osd.5) 84 : cluster [DBG] 5.7 deep-scrub starts 2024-02-13T00:57:15.305 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:13.790303+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.7 deep-scrub ok 2024-02-13T00:57:15.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:13.137451+0000 osd.1 (osd.1) 437 : cluster [DBG] 4.7 scrub starts 2024-02-13T00:57:15.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:13.139338+0000 osd.1 (osd.1) 438 : cluster [DBG] 4.7 scrub ok 2024-02-13T00:57:15.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:13.594507+0000 mgr.y (mgr.14544) 772 : audit [DBG] from='client.38558 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:57:15.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:13.787428+0000 osd.5 (osd.5) 84 : cluster [DBG] 5.7 deep-scrub starts 2024-02-13T00:57:15.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:14 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:13.790303+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.7 deep-scrub ok 2024-02-13T00:57:16.242 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:14.603893+0000 mgr.y (mgr.14544) 773 : cluster [DBG] pgmap v1369: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:57:16.242 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:14.824901+0000 osd.7 (osd.7) 162 : cluster [DBG] 6.5 scrub starts 2024-02-13T00:57:16.242 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:14.826018+0000 osd.7 (osd.7) 163 : cluster [DBG] 6.5 scrub ok 2024-02-13T00:57:16.242 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:14.920008+0000 osd.2 (osd.2) 225 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:57:16.242 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:14.923842+0000 osd.2 (osd.2) 226 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:57:16.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:15.577049+0000 osd.3 (osd.3) 251 : cluster [DBG] 6.6 deep-scrub starts 2024-02-13T00:57:16.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:15.579977+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.6 deep-scrub ok 2024-02-13T00:57:16.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:14.603893+0000 mgr.y (mgr.14544) 773 : cluster [DBG] pgmap v1369: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:57:16.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:14.824901+0000 osd.7 (osd.7) 162 : cluster [DBG] 6.5 scrub starts 2024-02-13T00:57:16.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:14.826018+0000 osd.7 (osd.7) 163 : cluster [DBG] 6.5 scrub ok 2024-02-13T00:57:16.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:14.920008+0000 osd.2 (osd.2) 225 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:57:16.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:14.923842+0000 osd.2 (osd.2) 226 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:57:16.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:15.577049+0000 osd.3 (osd.3) 251 : cluster [DBG] 6.6 deep-scrub starts 2024-02-13T00:57:16.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:15 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:15.579977+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.6 deep-scrub ok 2024-02-13T00:57:16.244 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:57:16] "GET /metrics HTTP/1.1" 200 185524 "" "Prometheus/2.33.4" 2024-02-13T00:57:16.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:57:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug there is no tcmu-runner data available 2024-02-13T00:57:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:14.603893+0000 mgr.y (mgr.14544) 773 : cluster [DBG] pgmap v1369: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-13T00:57:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:14.824901+0000 osd.7 (osd.7) 162 : cluster [DBG] 6.5 scrub starts 2024-02-13T00:57:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:14.826018+0000 osd.7 (osd.7) 163 : cluster [DBG] 6.5 scrub ok 2024-02-13T00:57:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:14.920008+0000 osd.2 (osd.2) 225 : cluster [DBG] 4.11 scrub starts 2024-02-13T00:57:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:14.923842+0000 osd.2 (osd.2) 226 : cluster [DBG] 4.11 scrub ok 2024-02-13T00:57:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:15.577049+0000 osd.3 (osd.3) 251 : cluster [DBG] 6.6 deep-scrub starts 2024-02-13T00:57:16.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:15.579977+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.6 deep-scrub ok 2024-02-13T00:57:16.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.487+0000 7fee28afc700 1 -- 172.21.15.162:0/313273559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee240f73a0 msgr2=0x7fee241039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:16.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.487+0000 7fee28afc700 1 --2- 172.21.15.162:0/313273559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee240f73a0 0x7fee241039a0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7fee1c009be0 tx=0x7fee1c004d10).stop 2024-02-13T00:57:16.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 -- 172.21.15.162:0/313273559 shutdown_connections 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 --2- 172.21.15.162:0/313273559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee240f73a0 0x7fee241039a0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 --2- 172.21.15.162:0/313273559 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee240fcb30 0x7fee240fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 --2- 172.21.15.162:0/313273559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee240fd490 0x7fee240f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 -- 172.21.15.162:0/313273559 >> 172.21.15.162:0/313273559 conn(0x7fee2406eb90 msgr2=0x7fee240f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 -- 172.21.15.162:0/313273559 shutdown_connections 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 -- 172.21.15.162:0/313273559 wait complete. 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 librados: starting msgr at 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 librados: starting objecter 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 Processor -- start 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 -- start start 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 librados: setting wanted keys 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.488+0000 7fee28afc700 1 librados: calling monclient init 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee28afc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee240f73a0 0x7fee2410df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee28afc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee240fcb30 0x7fee2410e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:16.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee28afc700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee240fd490 0x7fee241080c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee28afc700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee241085b0 con 0x7fee240fcb30 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee28afc700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee24108720 con 0x7fee240f73a0 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee28afc700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee241088e0 con 0x7fee240fd490 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee21d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee240fcb30 0x7fee2410e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee21d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee240fcb30 0x7fee2410e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43028/0 (socket says 172.21.15.162:43028) 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee21d9b700 1 -- 172.21.15.162:0/4090752857 learned_addr learned my addr 172.21.15.162:0/4090752857 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee22d9d700 1 --2- 172.21.15.162:0/4090752857 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee240fd490 0x7fee241080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee2259c700 1 --2- 172.21.15.162:0/4090752857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee240f73a0 0x7fee2410df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee22d9d700 1 -- 172.21.15.162:0/4090752857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee240f73a0 msgr2=0x7fee2410df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee22d9d700 1 --2- 172.21.15.162:0/4090752857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee240f73a0 0x7fee2410df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee22d9d700 1 -- 172.21.15.162:0/4090752857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee240fcb30 msgr2=0x7fee2410e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee22d9d700 1 --2- 172.21.15.162:0/4090752857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee240fcb30 0x7fee2410e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee22d9d700 1 -- 172.21.15.162:0/4090752857 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee14005040 con 0x7fee240fd490 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee21d9b700 1 --2- 172.21.15.162:0/4090752857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee240fcb30 0x7fee2410e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-13T00:57:16.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee22d9d700 1 --2- 172.21.15.162:0/4090752857 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee240fd490 0x7fee241080c0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 rx=0x7fee1c00b6d0 tx=0x7fee1c004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:16.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee137fe700 10 client.?.objecter ms_handle_connect 0x7fee240fd490 2024-02-13T00:57:16.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee137fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:57:16.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee137fe700 1 -- 172.21.15.162:0/4090752857 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c00e040 con 0x7fee240fd490 2024-02-13T00:57:16.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee28afc700 1 -- 172.21.15.162:0/4090752857 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c0097c0 con 0x7fee240fd490 2024-02-13T00:57:16.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee137fe700 1 -- 172.21.15.162:0/4090752857 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee1c018040 con 0x7fee240fd490 2024-02-13T00:57:16.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.489+0000 7fee137fe700 1 -- 172.21.15.162:0/4090752857 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c013e90 con 0x7fee240fd490 2024-02-13T00:57:16.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.490+0000 7fee28afc700 10 client.38564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:57:16.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.490+0000 7fee28afc700 1 -- 172.21.15.162:0/4090752857 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee24108df0 con 0x7fee240fd490 2024-02-13T00:57:16.496 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.490+0000 7fee28afc700 1 librados: init done 2024-02-13T00:57:16.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.490+0000 7fee28afc700 1 -- 172.21.15.162:0/4090752857 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee240475e0 con 0x7fee240fd490 2024-02-13T00:57:16.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.491+0000 7fee137fe700 1 -- 172.21.15.162:0/4090752857 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7fee1c013590 con 0x7fee240fd490 2024-02-13T00:57:16.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.491+0000 7fee137fe700 1 --2- 172.21.15.162:0/4090752857 >> v2:172.21.15.162:6800/2866534053 conn(0x7fee0c062c90 0x7fee0c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:16.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.491+0000 7fee137fe700 1 -- 172.21.15.162:0/4090752857 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(859..859 src has 449..859) v4 ==== 7509+0+0 (secure 0 0 0) 0x7fee1c07db00 con 0x7fee240fd490 2024-02-13T00:57:16.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.491+0000 7fee137fe700 10 client.38564.objecter ms_dispatch 0x7fee24001550 osd_map(859..859 src has 449..859) v4 2024-02-13T00:57:16.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.491+0000 7fee137fe700 3 client.38564.objecter handle_osd_map got epochs [859,859] > 0 2024-02-13T00:57:16.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.491+0000 7fee137fe700 3 client.38564.objecter handle_osd_map decoding full epoch 859 2024-02-13T00:57:16.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.491+0000 7fee2259c700 1 --2- 172.21.15.162:0/4090752857 >> v2:172.21.15.162:6800/2866534053 conn(0x7fee0c062c90 0x7fee0c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:16.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.491+0000 7fee137fe700 20 client.38564.objecter dump_active .. 0 homeless 2024-02-13T00:57:16.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.491+0000 7fee137fe700 10 client.38564.objecter ms_handle_connect 0x7fee0c062c90 2024-02-13T00:57:16.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.491+0000 7fee2259c700 1 --2- 172.21.15.162:0/4090752857 >> v2:172.21.15.162:6800/2866534053 conn(0x7fee0c062c90 0x7fee0c065150 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 rx=0x7fee14009350 tx=0x7fee1400b040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:16.499 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.495+0000 7fee137fe700 1 -- 172.21.15.162:0/4090752857 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fee1c0512e0 con 0x7fee240fd490 2024-02-13T00:57:16.665 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.661+0000 7fee28afc700 1 -- 172.21.15.162:0/4090752857 --> v2:172.21.15.162:6800/2866534053 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee2405fb80 con 0x7fee0c062c90 2024-02-13T00:57:16.676 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.673+0000 7fee137fe700 1 -- 172.21.15.162:0/4090752857 <== mgr.14544 v2:172.21.15.162:6800/2866534053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+432150 (secure 0 0 0) 0x7fee2405fb80 con 0x7fee0c062c90 2024-02-13T00:57:16.682 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-13T00:57:16.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 10 librados: watch_flush enter 2024-02-13T00:57:16.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 10 librados: watch_flush exit 2024-02-13T00:57:16.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 20 client.38564.objecter shutdown clearing up homeless session... 2024-02-13T00:57:16.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 10 client.38564.objecter successfully canceled tick 2024-02-13T00:57:16.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 1 -- 172.21.15.162:0/4090752857 >> v2:172.21.15.162:6800/2866534053 conn(0x7fee0c062c90 msgr2=0x7fee0c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 1 --2- 172.21.15.162:0/4090752857 >> v2:172.21.15.162:6800/2866534053 conn(0x7fee0c062c90 0x7fee0c065150 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 rx=0x7fee14009350 tx=0x7fee1400b040).stop 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 1 -- 172.21.15.162:0/4090752857 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee240fd490 msgr2=0x7fee241080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 1 --2- 172.21.15.162:0/4090752857 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee240fd490 0x7fee241080c0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 rx=0x7fee1c00b6d0 tx=0x7fee1c004d10).stop 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 1 -- 172.21.15.162:0/4090752857 shutdown_connections 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 1 --2- 172.21.15.162:0/4090752857 >> v2:172.21.15.162:6800/2866534053 conn(0x7fee0c062c90 0x7fee0c065150 unknown :-1 s=CLOSED pgs=3099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 1 --2- 172.21.15.162:0/4090752857 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee240fd490 0x7fee241080c0 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 1 --2- 172.21.15.162:0/4090752857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee240f73a0 0x7fee2410df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 1 --2- 172.21.15.162:0/4090752857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee240fcb30 0x7fee2410e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 1 -- 172.21.15.162:0/4090752857 >> 172.21.15.162:0/4090752857 conn(0x7fee2406eb90 msgr2=0x7fee24070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.676+0000 7fee28afc700 1 -- 172.21.15.162:0/4090752857 shutdown_connections 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.677+0000 7fee28afc700 1 -- 172.21.15.162:0/4090752857 wait complete. 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:16.677+0000 7fee28afc700 1 librados: shutdown 2024-02-13T00:57:16.687 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-02-13T00:57:17.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:15.951268+0000 osd.0 (osd.0) 417 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:57:17.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:15.954384+0000 osd.0 (osd.0) 418 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:57:17.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:16.138652+0000 mgr.y (mgr.14544) 774 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:57:17.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:16.545372+0000 osd.3 (osd.3) 253 : cluster [DBG] 6.b scrub starts 2024-02-13T00:57:17.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:16.548665+0000 osd.3 (osd.3) 254 : cluster [DBG] 6.b scrub ok 2024-02-13T00:57:17.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:15.951268+0000 osd.0 (osd.0) 417 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:57:17.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:15.954384+0000 osd.0 (osd.0) 418 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:57:17.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:16.138652+0000 mgr.y (mgr.14544) 774 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:57:17.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:16.545372+0000 osd.3 (osd.3) 253 : cluster [DBG] 6.b scrub starts 2024-02-13T00:57:17.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:16.548665+0000 osd.3 (osd.3) 254 : cluster [DBG] 6.b scrub ok 2024-02-13T00:57:17.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:15.951268+0000 osd.0 (osd.0) 417 : cluster [DBG] 6.16 scrub starts 2024-02-13T00:57:17.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:15.954384+0000 osd.0 (osd.0) 418 : cluster [DBG] 6.16 scrub ok 2024-02-13T00:57:17.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:16.138652+0000 mgr.y (mgr.14544) 774 : audit [DBG] from='client.14520 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-13T00:57:17.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:16.545372+0000 osd.3 (osd.3) 253 : cluster [DBG] 6.b scrub starts 2024-02-13T00:57:17.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:16 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:16.548665+0000 osd.3 (osd.3) 254 : cluster [DBG] 6.b scrub ok 2024-02-13T00:57:17.439 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":1370,"stamp":"2024-02-13T00:57:16.604076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":2656,"num_read_kb":2376,"num_write":913,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":23794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1148,"ondisk_log_size":1148,"up":492,"acting":492,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2466680,"kb_used_data":54968,"kb_used_omap":42,"kb_used_meta":2411605,"kb_avail":747494536,"statfs":{"total":767960285184,"available":765434404864,"internally_reserved":0,"allocated":56287232,"data_stored":29996011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43373,"internal_metadata":2469484179},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002075"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":607,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184418+0000","last_change":"2024-02-13T00:56:33.157883+0000","last_active":"2024-02-13T00:57:10.184418+0000","last_peered":"2024-02-13T00:57:10.184418+0000","last_clean":"2024-02-13T00:57:10.184418+0000","last_became_active":"2024-02-13T00:56:07.836922+0000","last_became_peered":"2024-02-13T00:56:07.836922+0000","last_unstale":"2024-02-13T00:57:10.184418+0000","last_undegraded":"2024-02-13T00:57:10.184418+0000","last_fullsized":"2024-02-13T00:57:10.184418+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:33.157756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:18.359604+0000","last_clean_scrub_stamp":"2024-02-13T00:56:33.157756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":58,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.183613+0000","last_change":"2024-02-13T00:56:49.841938+0000","last_active":"2024-02-13T00:57:10.183613+0000","last_peered":"2024-02-13T00:57:10.183613+0000","last_clean":"2024-02-13T00:57:10.183613+0000","last_became_active":"2024-02-13T00:55:04.781199+0000","last_became_peered":"2024-02-13T00:55:04.781199+0000","last_unstale":"2024-02-13T00:57:10.183613+0000","last_undegraded":"2024-02-13T00:57:10.183613+0000","last_fullsized":"2024-02-13T00:57:10.183613+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:49.841870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:39.671163+0000","last_clean_scrub_stamp":"2024-02-13T00:56:49.841870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":560,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184790+0000","last_change":"2024-02-13T00:56:44.994751+0000","last_active":"2024-02-13T00:57:10.184790+0000","last_peered":"2024-02-13T00:57:10.184790+0000","last_clean":"2024-02-13T00:57:10.184790+0000","last_became_active":"2024-02-13T00:56:07.653631+0000","last_became_peered":"2024-02-13T00:56:07.653631+0000","last_unstale":"2024-02-13T00:57:10.184790+0000","last_undegraded":"2024-02-13T00:57:10.184790+0000","last_fullsized":"2024-02-13T00:57:10.184790+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:44.994674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:44.994674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":662,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.956582+0000","last_change":"2024-02-13T00:57:08.159171+0000","last_active":"2024-02-13T00:57:09.956582+0000","last_peered":"2024-02-13T00:57:09.956582+0000","last_clean":"2024-02-13T00:57:09.956582+0000","last_became_active":"2024-02-13T00:55:04.796268+0000","last_became_peered":"2024-02-13T00:55:04.796268+0000","last_unstale":"2024-02-13T00:57:09.956582+0000","last_undegraded":"2024-02-13T00:57:09.956582+0000","last_fullsized":"2024-02-13T00:57:09.956582+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:08.159082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:57:08.159082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":573,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185266+0000","last_change":"2024-02-13T00:56:37.183946+0000","last_active":"2024-02-13T00:57:10.185266+0000","last_peered":"2024-02-13T00:57:10.185266+0000","last_clean":"2024-02-13T00:57:10.185266+0000","last_became_active":"2024-02-13T00:54:56.636553+0000","last_became_peered":"2024-02-13T00:54:56.636553+0000","last_unstale":"2024-02-13T00:57:10.185266+0000","last_undegraded":"2024-02-13T00:57:10.185266+0000","last_fullsized":"2024-02-13T00:57:10.185266+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:37.183853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:37.183853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":165,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185109+0000","last_change":"2024-02-13T00:56:21.181975+0000","last_active":"2024-02-13T00:57:10.185109+0000","last_peered":"2024-02-13T00:57:10.185109+0000","last_clean":"2024-02-13T00:57:10.185109+0000","last_became_active":"2024-02-13T00:55:00.638229+0000","last_became_peered":"2024-02-13T00:55:00.638229+0000","last_unstale":"2024-02-13T00:57:10.185109+0000","last_undegraded":"2024-02-13T00:57:10.185109+0000","last_fullsized":"2024-02-13T00:57:10.185109+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:21.181886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:21.181886+0000","last_clean_scrub_stamp":"2024-02-13T00:56:21.181886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":577,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.186138+0000","last_change":"2024-02-13T00:55:44.703735+0000","last_active":"2024-02-13T00:57:10.186138+0000","last_peered":"2024-02-13T00:57:10.186138+0000","last_clean":"2024-02-13T00:57:10.186138+0000","last_became_active":"2024-02-13T00:55:04.793521+0000","last_became_peered":"2024-02-13T00:55:04.793521+0000","last_unstale":"2024-02-13T00:57:10.186138+0000","last_undegraded":"2024-02-13T00:57:10.186138+0000","last_fullsized":"2024-02-13T00:57:10.186138+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:44.703665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:55:44.703665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":79,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959373+0000","last_change":"2024-02-13T00:56:07.223483+0000","last_active":"2024-02-13T00:57:09.959373+0000","last_peered":"2024-02-13T00:57:09.959373+0000","last_clean":"2024-02-13T00:57:09.959373+0000","last_became_active":"2024-02-13T00:55:04.789308+0000","last_became_peered":"2024-02-13T00:55:04.789308+0000","last_unstale":"2024-02-13T00:57:09.959373+0000","last_undegraded":"2024-02-13T00:57:09.959373+0000","last_fullsized":"2024-02-13T00:57:09.959373+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:07.223400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:56:07.223400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":546,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.186467+0000","last_change":"2024-02-13T00:56:36.215066+0000","last_active":"2024-02-13T00:57:10.186467+0000","last_peered":"2024-02-13T00:57:10.186467+0000","last_clean":"2024-02-13T00:57:10.186467+0000","last_became_active":"2024-02-13T00:44:39.524971+0000","last_became_peered":"2024-02-13T00:44:39.524971+0000","last_unstale":"2024-02-13T00:57:10.186467+0000","last_undegraded":"2024-02-13T00:57:10.186467+0000","last_fullsized":"2024-02-13T00:57:10.186467+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:36.214957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:56:36.214957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":653,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957986+0000","last_change":"2024-02-13T00:57:00.129012+0000","last_active":"2024-02-13T00:57:09.957986+0000","last_peered":"2024-02-13T00:57:09.957986+0000","last_clean":"2024-02-13T00:57:09.957986+0000","last_became_active":"2024-02-13T00:54:52.736324+0000","last_became_peered":"2024-02-13T00:54:52.736324+0000","last_unstale":"2024-02-13T00:57:09.957986+0000","last_undegraded":"2024-02-13T00:57:09.957986+0000","last_fullsized":"2024-02-13T00:57:09.957986+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:00.128944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:57:00.128944+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":776,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957616+0000","last_change":"2024-02-13T00:56:51.232178+0000","last_active":"2024-02-13T00:57:09.957616+0000","last_peered":"2024-02-13T00:57:09.957616+0000","last_clean":"2024-02-13T00:57:09.957616+0000","last_became_active":"2024-02-13T00:55:51.633619+0000","last_became_peered":"2024-02-13T00:55:51.633619+0000","last_unstale":"2024-02-13T00:57:09.957616+0000","last_undegraded":"2024-02-13T00:57:09.957616+0000","last_fullsized":"2024-02-13T00:57:09.957616+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:51.232094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:51.232094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"849'26","reported_seq":533,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.809698+0000","last_change":"2024-02-13T00:56:07.653898+0000","last_active":"2024-02-13T00:57:10.809698+0000","last_peered":"2024-02-13T00:57:10.809698+0000","last_clean":"2024-02-13T00:57:10.809698+0000","last_became_active":"2024-02-13T00:56:07.653175+0000","last_became_peered":"2024-02-13T00:56:07.653175+0000","last_unstale":"2024-02-13T00:57:10.809698+0000","last_undegraded":"2024-02-13T00:57:10.809698+0000","last_fullsized":"2024-02-13T00:57:10.809698+0000","mapping_epoch":848,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"820'25","last_scrub_stamp":"2024-02-13T00:56:02.921047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:56:02.921047+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"467'48","reported_seq":81,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958639+0000","last_change":"2024-02-13T00:56:11.198869+0000","last_active":"2024-02-13T00:57:09.958639+0000","last_peered":"2024-02-13T00:57:09.958639+0000","last_clean":"2024-02-13T00:57:09.958639+0000","last_became_active":"2024-02-13T00:54:52.742726+0000","last_became_peered":"2024-02-13T00:54:52.742726+0000","last_unstale":"2024-02-13T00:57:09.958639+0000","last_undegraded":"2024-02-13T00:57:09.958639+0000","last_fullsized":"2024-02-13T00:57:09.958639+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'48","last_scrub_stamp":"2024-02-13T00:56:11.198787+0000","last_deep_scrub":"80'36","last_deep_scrub_stamp":"2024-02-13T00:47:03.491647+0000","last_clean_scrub_stamp":"2024-02-13T00:56:11.198787+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":53,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184579+0000","last_change":"2024-02-13T00:56:15.785316+0000","last_active":"2024-02-13T00:57:10.184579+0000","last_peered":"2024-02-13T00:57:10.184579+0000","last_clean":"2024-02-13T00:57:10.184579+0000","last_became_active":"2024-02-13T00:55:04.799211+0000","last_became_peered":"2024-02-13T00:55:04.799211+0000","last_unstale":"2024-02-13T00:57:10.184579+0000","last_undegraded":"2024-02-13T00:57:10.184579+0000","last_fullsized":"2024-02-13T00:57:10.184579+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:15.785248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:15.785248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":66,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:12.815995+0000","last_change":"2024-02-13T00:57:12.815995+0000","last_active":"2024-02-13T00:57:12.815995+0000","last_peered":"2024-02-13T00:57:12.815995+0000","last_clean":"2024-02-13T00:57:12.815995+0000","last_became_active":"2024-02-13T00:54:52.751474+0000","last_became_peered":"2024-02-13T00:54:52.751474+0000","last_unstale":"2024-02-13T00:57:12.815995+0000","last_undegraded":"2024-02-13T00:57:12.815995+0000","last_fullsized":"2024-02-13T00:57:12.815995+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:12.815893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:05.252294+0000","last_clean_scrub_stamp":"2024-02-13T00:57:12.815893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":53,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184076+0000","last_change":"2024-02-13T00:56:01.710378+0000","last_active":"2024-02-13T00:57:10.184076+0000","last_peered":"2024-02-13T00:57:10.184076+0000","last_clean":"2024-02-13T00:57:10.184076+0000","last_became_active":"2024-02-13T00:55:04.790490+0000","last_became_peered":"2024-02-13T00:55:04.790490+0000","last_unstale":"2024-02-13T00:57:10.184076+0000","last_undegraded":"2024-02-13T00:57:10.184076+0000","last_fullsized":"2024-02-13T00:57:10.184076+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:01.710308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:01.710308+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"467'21","reported_seq":82,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:11.663512+0000","last_change":"2024-02-13T00:57:11.663512+0000","last_active":"2024-02-13T00:57:11.663512+0000","last_peered":"2024-02-13T00:57:11.663512+0000","last_clean":"2024-02-13T00:57:11.663512+0000","last_became_active":"2024-02-13T00:55:00.638928+0000","last_became_peered":"2024-02-13T00:55:00.638928+0000","last_unstale":"2024-02-13T00:57:11.663512+0000","last_undegraded":"2024-02-13T00:57:11.663512+0000","last_fullsized":"2024-02-13T00:57:11.663512+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'21","last_scrub_stamp":"2024-02-13T00:57:11.663427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:57:11.663427+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":872,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958191+0000","last_change":"2024-02-13T00:55:51.966565+0000","last_active":"2024-02-13T00:57:09.958191+0000","last_peered":"2024-02-13T00:57:09.958191+0000","last_clean":"2024-02-13T00:57:09.958191+0000","last_became_active":"2024-02-13T00:55:00.635786+0000","last_became_peered":"2024-02-13T00:55:00.635786+0000","last_unstale":"2024-02-13T00:57:09.958191+0000","last_undegraded":"2024-02-13T00:57:09.958191+0000","last_fullsized":"2024-02-13T00:57:09.958191+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:51.966488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:55:51.966488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":58,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.183957+0000","last_change":"2024-02-13T00:56:19.819857+0000","last_active":"2024-02-13T00:57:10.183957+0000","last_peered":"2024-02-13T00:57:10.183957+0000","last_clean":"2024-02-13T00:57:10.183957+0000","last_became_active":"2024-02-13T00:55:04.801366+0000","last_became_peered":"2024-02-13T00:55:04.801366+0000","last_unstale":"2024-02-13T00:57:10.183957+0000","last_undegraded":"2024-02-13T00:57:10.183957+0000","last_fullsized":"2024-02-13T00:57:10.183957+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:19.819795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:56:19.819795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":871,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154414+0000","last_change":"2024-02-13T00:56:03.986502+0000","last_active":"2024-02-13T00:57:10.154414+0000","last_peered":"2024-02-13T00:57:10.154414+0000","last_clean":"2024-02-13T00:57:10.154414+0000","last_became_active":"2024-02-13T00:55:04.815877+0000","last_became_peered":"2024-02-13T00:55:04.815877+0000","last_unstale":"2024-02-13T00:57:10.154414+0000","last_undegraded":"2024-02-13T00:57:10.154414+0000","last_fullsized":"2024-02-13T00:57:10.154414+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:03.986440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:54:47.261547+0000","last_clean_scrub_stamp":"2024-02-13T00:56:03.986440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":497,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154192+0000","last_change":"2024-02-13T00:56:33.741869+0000","last_active":"2024-02-13T00:57:10.154192+0000","last_peered":"2024-02-13T00:57:10.154192+0000","last_clean":"2024-02-13T00:57:10.154192+0000","last_became_active":"2024-02-13T00:55:00.640016+0000","last_became_peered":"2024-02-13T00:55:00.640016+0000","last_unstale":"2024-02-13T00:57:10.154192+0000","last_undegraded":"2024-02-13T00:57:10.154192+0000","last_fullsized":"2024-02-13T00:57:10.154192+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:33.741779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:33.741779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"467'51","reported_seq":685,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185527+0000","last_change":"2024-02-13T00:57:04.940383+0000","last_active":"2024-02-13T00:57:10.185527+0000","last_peered":"2024-02-13T00:57:10.185527+0000","last_clean":"2024-02-13T00:57:10.185527+0000","last_became_active":"2024-02-13T00:51:41.573050+0000","last_became_peered":"2024-02-13T00:51:41.573050+0000","last_unstale":"2024-02-13T00:57:10.185527+0000","last_undegraded":"2024-02-13T00:57:10.185527+0000","last_fullsized":"2024-02-13T00:57:10.185527+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'51","last_scrub_stamp":"2024-02-13T00:57:04.940306+0000","last_deep_scrub":"91'35","last_deep_scrub_stamp":"2024-02-13T00:40:15.074034+0000","last_clean_scrub_stamp":"2024-02-13T00:57:04.940306+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":58,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184310+0000","last_change":"2024-02-13T00:56:48.850014+0000","last_active":"2024-02-13T00:57:10.184310+0000","last_peered":"2024-02-13T00:57:10.184310+0000","last_clean":"2024-02-13T00:57:10.184310+0000","last_became_active":"2024-02-13T00:55:04.808497+0000","last_became_peered":"2024-02-13T00:55:04.808497+0000","last_unstale":"2024-02-13T00:57:10.184310+0000","last_undegraded":"2024-02-13T00:57:10.184310+0000","last_fullsized":"2024-02-13T00:57:10.184310+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:48.849919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:48.849919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":58,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184231+0000","last_change":"2024-02-13T00:56:43.888878+0000","last_active":"2024-02-13T00:57:10.184231+0000","last_peered":"2024-02-13T00:57:10.184231+0000","last_clean":"2024-02-13T00:57:10.184231+0000","last_became_active":"2024-02-13T00:55:04.808497+0000","last_became_peered":"2024-02-13T00:55:04.808497+0000","last_unstale":"2024-02-13T00:57:10.184231+0000","last_undegraded":"2024-02-13T00:57:10.184231+0000","last_fullsized":"2024-02-13T00:57:10.184231+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:43.888814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:42.592672+0000","last_clean_scrub_stamp":"2024-02-13T00:56:43.888814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":522,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184811+0000","last_change":"2024-02-13T00:56:48.913299+0000","last_active":"2024-02-13T00:57:10.184811+0000","last_peered":"2024-02-13T00:57:10.184811+0000","last_clean":"2024-02-13T00:57:10.184811+0000","last_became_active":"2024-02-13T00:56:07.654355+0000","last_became_peered":"2024-02-13T00:56:07.654355+0000","last_unstale":"2024-02-13T00:57:10.184811+0000","last_undegraded":"2024-02-13T00:57:10.184811+0000","last_fullsized":"2024-02-13T00:57:10.184811+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:48.913221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:48.913221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":785,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958080+0000","last_change":"2024-02-13T00:55:38.989290+0000","last_active":"2024-02-13T00:57:09.958080+0000","last_peered":"2024-02-13T00:57:09.958080+0000","last_clean":"2024-02-13T00:57:09.958080+0000","last_became_active":"2024-02-13T00:44:39.501515+0000","last_became_peered":"2024-02-13T00:44:39.501515+0000","last_unstale":"2024-02-13T00:57:09.958080+0000","last_undegraded":"2024-02-13T00:57:09.958080+0000","last_fullsized":"2024-02-13T00:57:09.958080+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:38.989213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:55:38.989213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"851'39","reported_seq":642,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958477+0000","last_change":"2024-02-13T00:56:15.309324+0000","last_active":"2024-02-13T00:57:09.958477+0000","last_peered":"2024-02-13T00:57:09.958477+0000","last_clean":"2024-02-13T00:57:09.958477+0000","last_became_active":"2024-02-13T00:55:51.631136+0000","last_became_peered":"2024-02-13T00:55:51.631136+0000","last_unstale":"2024-02-13T00:57:09.958477+0000","last_undegraded":"2024-02-13T00:57:09.958477+0000","last_fullsized":"2024-02-13T00:57:09.958477+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'39","last_scrub_stamp":"2024-02-13T00:56:15.309243+0000","last_deep_scrub":"851'39","last_deep_scrub_stamp":"2024-02-13T00:56:15.309243+0000","last_clean_scrub_stamp":"2024-02-13T00:56:15.309243+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"57'1","reported_seq":472,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.812700+0000","last_change":"2024-02-13T00:57:10.812700+0000","last_active":"2024-02-13T00:57:10.812700+0000","last_peered":"2024-02-13T00:57:10.812700+0000","last_clean":"2024-02-13T00:57:10.812700+0000","last_became_active":"2024-02-13T00:56:07.657609+0000","last_became_peered":"2024-02-13T00:56:07.657609+0000","last_unstale":"2024-02-13T00:57:10.812700+0000","last_undegraded":"2024-02-13T00:57:10.812700+0000","last_fullsized":"2024-02-13T00:57:10.812700+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:57:10.812616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:57:10.812616+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":866,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.157920+0000","last_change":"2024-02-13T00:57:10.157900+0000","last_active":"2024-02-13T00:57:10.157920+0000","last_peered":"2024-02-13T00:57:10.157920+0000","last_clean":"2024-02-13T00:57:10.157920+0000","last_became_active":"2024-02-13T00:54:56.630042+0000","last_became_peered":"2024-02-13T00:54:56.630042+0000","last_unstale":"2024-02-13T00:57:10.157920+0000","last_undegraded":"2024-02-13T00:57:10.157920+0000","last_fullsized":"2024-02-13T00:57:10.157920+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:10.157837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:32.793634+0000","last_clean_scrub_stamp":"2024-02-13T00:57:10.157837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":60,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184270+0000","last_change":"2024-02-13T00:56:46.851061+0000","last_active":"2024-02-13T00:57:10.184270+0000","last_peered":"2024-02-13T00:57:10.184270+0000","last_clean":"2024-02-13T00:57:10.184270+0000","last_became_active":"2024-02-13T00:55:04.806188+0000","last_became_peered":"2024-02-13T00:55:04.806188+0000","last_unstale":"2024-02-13T00:57:10.184270+0000","last_undegraded":"2024-02-13T00:57:10.184270+0000","last_fullsized":"2024-02-13T00:57:10.184270+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:46.850991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:29.843131+0000","last_clean_scrub_stamp":"2024-02-13T00:56:46.850991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":68,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185527+0000","last_change":"2024-02-13T00:56:20.723933+0000","last_active":"2024-02-13T00:57:10.185527+0000","last_peered":"2024-02-13T00:57:10.185527+0000","last_clean":"2024-02-13T00:57:10.185527+0000","last_became_active":"2024-02-13T00:55:04.802114+0000","last_became_peered":"2024-02-13T00:55:04.802114+0000","last_unstale":"2024-02-13T00:57:10.185527+0000","last_undegraded":"2024-02-13T00:57:10.185527+0000","last_fullsized":"2024-02-13T00:57:10.185527+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:20.723868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:10.700548+0000","last_clean_scrub_stamp":"2024-02-13T00:56:20.723868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"467'25","reported_seq":575,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958115+0000","last_change":"2024-02-13T00:56:37.072073+0000","last_active":"2024-02-13T00:57:09.958115+0000","last_peered":"2024-02-13T00:57:09.958115+0000","last_clean":"2024-02-13T00:57:09.958115+0000","last_became_active":"2024-02-13T00:55:51.635737+0000","last_became_peered":"2024-02-13T00:55:51.635737+0000","last_unstale":"2024-02-13T00:57:09.958115+0000","last_undegraded":"2024-02-13T00:57:09.958115+0000","last_fullsized":"2024-02-13T00:57:09.958115+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:56:37.071990+0000","last_deep_scrub":"467'25","last_deep_scrub_stamp":"2024-02-13T00:55:08.701877+0000","last_clean_scrub_stamp":"2024-02-13T00:56:37.071990+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":881,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957102+0000","last_change":"2024-02-13T00:56:10.911812+0000","last_active":"2024-02-13T00:57:09.957102+0000","last_peered":"2024-02-13T00:57:09.957102+0000","last_clean":"2024-02-13T00:57:09.957102+0000","last_became_active":"2024-02-13T00:55:00.636125+0000","last_became_peered":"2024-02-13T00:55:00.636125+0000","last_unstale":"2024-02-13T00:57:09.957102+0000","last_undegraded":"2024-02-13T00:57:09.957102+0000","last_fullsized":"2024-02-13T00:57:09.957102+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:10.911727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:10.911727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"824'24","reported_seq":739,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:11.996977+0000","last_change":"2024-02-13T00:57:11.996943+0000","last_active":"2024-02-13T00:57:11.996977+0000","last_peered":"2024-02-13T00:57:11.996977+0000","last_clean":"2024-02-13T00:57:11.996977+0000","last_became_active":"2024-02-13T00:55:07.818223+0000","last_became_peered":"2024-02-13T00:55:07.818223+0000","last_unstale":"2024-02-13T00:57:11.996977+0000","last_undegraded":"2024-02-13T00:57:11.996977+0000","last_fullsized":"2024-02-13T00:57:11.996977+0000","mapping_epoch":823,"log_start":"404'11","ondisk_log_start":"404'11","created":60,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"824'24","last_scrub_stamp":"2024-02-13T00:57:11.996862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:57:11.996862+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"64'1","reported_seq":845,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958549+0000","last_change":"2024-02-13T00:56:49.191428+0000","last_active":"2024-02-13T00:57:09.958549+0000","last_peered":"2024-02-13T00:57:09.958549+0000","last_clean":"2024-02-13T00:57:09.958549+0000","last_became_active":"2024-02-13T00:55:51.631073+0000","last_became_peered":"2024-02-13T00:55:51.631073+0000","last_unstale":"2024-02-13T00:57:09.958549+0000","last_undegraded":"2024-02-13T00:57:09.958549+0000","last_fullsized":"2024-02-13T00:57:09.958549+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:56:49.191338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:49.191338+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":530,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185179+0000","last_change":"2024-02-13T00:56:45.968431+0000","last_active":"2024-02-13T00:57:10.185179+0000","last_peered":"2024-02-13T00:57:10.185179+0000","last_clean":"2024-02-13T00:57:10.185179+0000","last_became_active":"2024-02-13T00:56:07.657754+0000","last_became_peered":"2024-02-13T00:56:07.657754+0000","last_unstale":"2024-02-13T00:57:10.185179+0000","last_undegraded":"2024-02-13T00:57:10.185179+0000","last_fullsized":"2024-02-13T00:57:10.185179+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:45.968347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:45.968347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"806'34","reported_seq":1075,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958030+0000","last_change":"2024-02-13T00:55:56.946858+0000","last_active":"2024-02-13T00:57:09.958030+0000","last_peered":"2024-02-13T00:57:09.958030+0000","last_clean":"2024-02-13T00:57:09.958030+0000","last_became_active":"2024-02-13T00:55:00.635264+0000","last_became_peered":"2024-02-13T00:55:00.635264+0000","last_unstale":"2024-02-13T00:57:09.958030+0000","last_undegraded":"2024-02-13T00:57:09.958030+0000","last_fullsized":"2024-02-13T00:57:09.958030+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"806'34","last_scrub_stamp":"2024-02-13T00:55:56.946788+0000","last_deep_scrub":"806'34","last_deep_scrub_stamp":"2024-02-13T00:54:38.218263+0000","last_clean_scrub_stamp":"2024-02-13T00:55:56.946788+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"57'1","reported_seq":912,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.956816+0000","last_change":"2024-02-13T00:56:24.887795+0000","last_active":"2024-02-13T00:57:09.956816+0000","last_peered":"2024-02-13T00:57:09.956816+0000","last_clean":"2024-02-13T00:57:09.956816+0000","last_became_active":"2024-02-13T00:55:00.638261+0000","last_became_peered":"2024-02-13T00:55:00.638261+0000","last_unstale":"2024-02-13T00:57:09.956816+0000","last_undegraded":"2024-02-13T00:57:09.956816+0000","last_fullsized":"2024-02-13T00:57:09.956816+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2024-02-13T00:56:24.887707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:24.887707+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"842'25","reported_seq":712,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.812214+0000","last_change":"2024-02-13T00:56:58.245207+0000","last_active":"2024-02-13T00:57:10.812214+0000","last_peered":"2024-02-13T00:57:10.812214+0000","last_clean":"2024-02-13T00:57:10.812214+0000","last_became_active":"2024-02-13T00:55:51.633596+0000","last_became_peered":"2024-02-13T00:55:51.633596+0000","last_unstale":"2024-02-13T00:57:10.812214+0000","last_undegraded":"2024-02-13T00:57:10.812214+0000","last_fullsized":"2024-02-13T00:57:10.812214+0000","mapping_epoch":841,"log_start":"467'11","ondisk_log_start":"467'11","created":60,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"842'25","last_scrub_stamp":"2024-02-13T00:56:58.245119+0000","last_deep_scrub":"467'11","last_deep_scrub_stamp":"2024-02-13T00:47:49.687727+0000","last_clean_scrub_stamp":"2024-02-13T00:56:58.245119+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":66,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959787+0000","last_change":"2024-02-13T00:56:42.022798+0000","last_active":"2024-02-13T00:57:09.959787+0000","last_peered":"2024-02-13T00:57:09.959787+0000","last_clean":"2024-02-13T00:57:09.959787+0000","last_became_active":"2024-02-13T00:54:52.746904+0000","last_became_peered":"2024-02-13T00:54:52.746904+0000","last_unstale":"2024-02-13T00:57:09.959787+0000","last_undegraded":"2024-02-13T00:57:09.959787+0000","last_fullsized":"2024-02-13T00:57:09.959787+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:42.022702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:58.541144+0000","last_clean_scrub_stamp":"2024-02-13T00:56:42.022702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":805,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154898+0000","last_change":"2024-02-13T00:56:32.724624+0000","last_active":"2024-02-13T00:57:10.154898+0000","last_peered":"2024-02-13T00:57:10.154898+0000","last_clean":"2024-02-13T00:57:10.154898+0000","last_became_active":"2024-02-13T00:55:04.815387+0000","last_became_peered":"2024-02-13T00:55:04.815387+0000","last_unstale":"2024-02-13T00:57:10.154898+0000","last_undegraded":"2024-02-13T00:57:10.154898+0000","last_fullsized":"2024-02-13T00:57:10.154898+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:32.724540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:59.963616+0000","last_clean_scrub_stamp":"2024-02-13T00:56:32.724540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"467'25","reported_seq":599,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958584+0000","last_change":"2024-02-13T00:56:44.170249+0000","last_active":"2024-02-13T00:57:09.958584+0000","last_peered":"2024-02-13T00:57:09.958584+0000","last_clean":"2024-02-13T00:57:09.958584+0000","last_became_active":"2024-02-13T00:55:51.635856+0000","last_became_peered":"2024-02-13T00:55:51.635856+0000","last_unstale":"2024-02-13T00:57:09.958584+0000","last_undegraded":"2024-02-13T00:57:09.958584+0000","last_fullsized":"2024-02-13T00:57:09.958584+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:56:44.170175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:44.170175+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":504,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184203+0000","last_change":"2024-02-13T00:56:52.824236+0000","last_active":"2024-02-13T00:57:10.184203+0000","last_peered":"2024-02-13T00:57:10.184203+0000","last_clean":"2024-02-13T00:57:10.184203+0000","last_became_active":"2024-02-13T00:56:07.837024+0000","last_became_peered":"2024-02-13T00:56:07.837024+0000","last_unstale":"2024-02-13T00:57:10.184203+0000","last_undegraded":"2024-02-13T00:57:10.184203+0000","last_fullsized":"2024-02-13T00:57:10.184203+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:52.824142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:52.824142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"834'23","reported_seq":825,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959862+0000","last_change":"2024-02-13T00:56:58.775254+0000","last_active":"2024-02-13T00:57:09.959862+0000","last_peered":"2024-02-13T00:57:09.959862+0000","last_clean":"2024-02-13T00:57:09.959862+0000","last_became_active":"2024-02-13T00:55:30.734460+0000","last_became_peered":"2024-02-13T00:55:30.734460+0000","last_unstale":"2024-02-13T00:57:09.959862+0000","last_undegraded":"2024-02-13T00:57:09.959862+0000","last_fullsized":"2024-02-13T00:57:09.959862+0000","mapping_epoch":833,"log_start":"363'14","ondisk_log_start":"363'14","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"834'23","last_scrub_stamp":"2024-02-13T00:56:58.775177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:57.494653+0000","last_clean_scrub_stamp":"2024-02-13T00:56:58.775177+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":54,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185371+0000","last_change":"2024-02-13T00:56:21.717193+0000","last_active":"2024-02-13T00:57:10.185371+0000","last_peered":"2024-02-13T00:57:10.185371+0000","last_clean":"2024-02-13T00:57:10.185371+0000","last_became_active":"2024-02-13T00:55:00.637174+0000","last_became_peered":"2024-02-13T00:55:00.637174+0000","last_unstale":"2024-02-13T00:57:10.185371+0000","last_undegraded":"2024-02-13T00:57:10.185371+0000","last_fullsized":"2024-02-13T00:57:10.185371+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:21.717108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:21.717108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":70,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958764+0000","last_change":"2024-02-13T00:56:19.147899+0000","last_active":"2024-02-13T00:57:09.958764+0000","last_peered":"2024-02-13T00:57:09.958764+0000","last_clean":"2024-02-13T00:57:09.958764+0000","last_became_active":"2024-02-13T00:54:56.630273+0000","last_became_peered":"2024-02-13T00:54:56.630273+0000","last_unstale":"2024-02-13T00:57:09.958764+0000","last_undegraded":"2024-02-13T00:57:09.958764+0000","last_fullsized":"2024-02-13T00:57:09.958764+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:19.147801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:19.147801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"820'14","reported_seq":85,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.638738+0000","last_change":"2024-02-13T00:56:04.735803+0000","last_active":"2024-02-13T00:57:10.638738+0000","last_peered":"2024-02-13T00:57:10.638738+0000","last_clean":"2024-02-13T00:57:10.638738+0000","last_became_active":"2024-02-13T00:55:00.647621+0000","last_became_peered":"2024-02-13T00:55:00.647621+0000","last_unstale":"2024-02-13T00:57:10.638738+0000","last_undegraded":"2024-02-13T00:57:10.638738+0000","last_fullsized":"2024-02-13T00:57:10.638738+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"820'14","last_scrub_stamp":"2024-02-13T00:56:04.735726+0000","last_deep_scrub":"263'10","last_deep_scrub_stamp":"2024-02-13T00:47:02.949406+0000","last_clean_scrub_stamp":"2024-02-13T00:56:04.735726+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"467'34","reported_seq":64,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958022+0000","last_change":"2024-02-13T00:56:46.156551+0000","last_active":"2024-02-13T00:57:09.958022+0000","last_peered":"2024-02-13T00:57:09.958022+0000","last_clean":"2024-02-13T00:57:09.958022+0000","last_became_active":"2024-02-13T00:55:51.635323+0000","last_became_peered":"2024-02-13T00:55:51.635323+0000","last_unstale":"2024-02-13T00:57:09.958022+0000","last_undegraded":"2024-02-13T00:57:09.958022+0000","last_fullsized":"2024-02-13T00:57:09.958022+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'34","last_scrub_stamp":"2024-02-13T00:56:46.156468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:46.156468+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":591,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.186414+0000","last_change":"2024-02-13T00:56:41.122457+0000","last_active":"2024-02-13T00:57:10.186414+0000","last_peered":"2024-02-13T00:57:10.186414+0000","last_clean":"2024-02-13T00:57:10.186414+0000","last_became_active":"2024-02-13T00:55:04.781853+0000","last_became_peered":"2024-02-13T00:55:04.781853+0000","last_unstale":"2024-02-13T00:57:10.186414+0000","last_undegraded":"2024-02-13T00:57:10.186414+0000","last_fullsized":"2024-02-13T00:57:10.186414+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:41.122380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:18.518704+0000","last_clean_scrub_stamp":"2024-02-13T00:56:41.122380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":553,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.186046+0000","last_change":"2024-02-13T00:57:07.785996+0000","last_active":"2024-02-13T00:57:10.186046+0000","last_peered":"2024-02-13T00:57:10.186046+0000","last_clean":"2024-02-13T00:57:10.186046+0000","last_became_active":"2024-02-13T00:56:07.649968+0000","last_became_peered":"2024-02-13T00:56:07.649968+0000","last_unstale":"2024-02-13T00:57:10.186046+0000","last_undegraded":"2024-02-13T00:57:10.186046+0000","last_fullsized":"2024-02-13T00:57:10.186046+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:07.785898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:57:07.785898+0000","last_clean_scrub_stamp":"2024-02-13T00:57:07.785898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":54,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.183419+0000","last_change":"2024-02-13T00:56:17.665320+0000","last_active":"2024-02-13T00:57:10.183419+0000","last_peered":"2024-02-13T00:57:10.183419+0000","last_clean":"2024-02-13T00:57:10.183419+0000","last_became_active":"2024-02-13T00:55:00.654433+0000","last_became_peered":"2024-02-13T00:55:00.654433+0000","last_unstale":"2024-02-13T00:57:10.183419+0000","last_undegraded":"2024-02-13T00:57:10.183419+0000","last_fullsized":"2024-02-13T00:57:10.183419+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:17.665207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:17.665207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":642,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957771+0000","last_change":"2024-02-13T00:56:04.369856+0000","last_active":"2024-02-13T00:57:09.957771+0000","last_peered":"2024-02-13T00:57:09.957771+0000","last_clean":"2024-02-13T00:57:09.957771+0000","last_became_active":"2024-02-13T00:55:51.634823+0000","last_became_peered":"2024-02-13T00:55:51.634823+0000","last_unstale":"2024-02-13T00:57:09.957771+0000","last_undegraded":"2024-02-13T00:57:09.957771+0000","last_fullsized":"2024-02-13T00:57:09.957771+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:04.369780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:56:04.369780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"843'52","reported_seq":91,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.183374+0000","last_change":"2024-02-13T00:57:05.656835+0000","last_active":"2024-02-13T00:57:10.183374+0000","last_peered":"2024-02-13T00:57:10.183374+0000","last_clean":"2024-02-13T00:57:10.183374+0000","last_became_active":"2024-02-13T00:55:00.654881+0000","last_became_peered":"2024-02-13T00:55:00.654881+0000","last_unstale":"2024-02-13T00:57:10.183374+0000","last_undegraded":"2024-02-13T00:57:10.183374+0000","last_fullsized":"2024-02-13T00:57:10.183374+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'52","last_scrub_stamp":"2024-02-13T00:57:05.656720+0000","last_deep_scrub":"843'52","last_deep_scrub_stamp":"2024-02-13T00:55:56.638643+0000","last_clean_scrub_stamp":"2024-02-13T00:57:05.656720+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":872,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958037+0000","last_change":"2024-02-13T00:55:48.069136+0000","last_active":"2024-02-13T00:57:09.958037+0000","last_peered":"2024-02-13T00:57:09.958037+0000","last_clean":"2024-02-13T00:57:09.958037+0000","last_became_active":"2024-02-13T00:54:56.623499+0000","last_became_peered":"2024-02-13T00:54:56.623499+0000","last_unstale":"2024-02-13T00:57:09.958037+0000","last_undegraded":"2024-02-13T00:57:09.958037+0000","last_fullsized":"2024-02-13T00:57:09.958037+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:48.069074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:48.069074+0000","last_clean_scrub_stamp":"2024-02-13T00:55:48.069074+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":802,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958322+0000","last_change":"2024-02-13T00:56:36.024603+0000","last_active":"2024-02-13T00:57:09.958322+0000","last_peered":"2024-02-13T00:57:09.958322+0000","last_clean":"2024-02-13T00:57:09.958322+0000","last_became_active":"2024-02-13T00:55:51.632395+0000","last_became_peered":"2024-02-13T00:55:51.632395+0000","last_unstale":"2024-02-13T00:57:09.958322+0000","last_undegraded":"2024-02-13T00:57:09.958322+0000","last_fullsized":"2024-02-13T00:57:09.958322+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:36.024532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:36.024532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":53,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184153+0000","last_change":"2024-02-13T00:56:11.817669+0000","last_active":"2024-02-13T00:57:10.184153+0000","last_peered":"2024-02-13T00:57:10.184153+0000","last_clean":"2024-02-13T00:57:10.184153+0000","last_became_active":"2024-02-13T00:55:04.790891+0000","last_became_peered":"2024-02-13T00:55:04.790891+0000","last_unstale":"2024-02-13T00:57:10.184153+0000","last_undegraded":"2024-02-13T00:57:10.184153+0000","last_fullsized":"2024-02-13T00:57:10.184153+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:11.817599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:48:35.434645+0000","last_clean_scrub_stamp":"2024-02-13T00:56:11.817599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":573,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185616+0000","last_change":"2024-02-13T00:56:06.915576+0000","last_active":"2024-02-13T00:57:10.185616+0000","last_peered":"2024-02-13T00:57:10.185616+0000","last_clean":"2024-02-13T00:57:10.185616+0000","last_became_active":"2024-02-13T00:54:52.757890+0000","last_became_peered":"2024-02-13T00:54:52.757890+0000","last_unstale":"2024-02-13T00:57:10.185616+0000","last_undegraded":"2024-02-13T00:57:10.185616+0000","last_fullsized":"2024-02-13T00:57:10.185616+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:06.915454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:56:06.915454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"467'44","reported_seq":1017,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154392+0000","last_change":"2024-02-13T00:56:50.889120+0000","last_active":"2024-02-13T00:57:10.154392+0000","last_peered":"2024-02-13T00:57:10.154392+0000","last_clean":"2024-02-13T00:57:10.154392+0000","last_became_active":"2024-02-13T00:54:52.756864+0000","last_became_peered":"2024-02-13T00:54:52.756864+0000","last_unstale":"2024-02-13T00:57:10.154392+0000","last_undegraded":"2024-02-13T00:57:10.154392+0000","last_fullsized":"2024-02-13T00:57:10.154392+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":814,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'44","last_scrub_stamp":"2024-02-13T00:56:50.889053+0000","last_deep_scrub":"467'44","last_deep_scrub_stamp":"2024-02-13T00:55:28.004399+0000","last_clean_scrub_stamp":"2024-02-13T00:56:50.889053+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":662,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185169+0000","last_change":"2024-02-13T00:56:11.886835+0000","last_active":"2024-02-13T00:57:10.185169+0000","last_peered":"2024-02-13T00:57:10.185169+0000","last_clean":"2024-02-13T00:57:10.185169+0000","last_became_active":"2024-02-13T00:56:07.837531+0000","last_became_peered":"2024-02-13T00:56:07.837531+0000","last_unstale":"2024-02-13T00:57:10.185169+0000","last_undegraded":"2024-02-13T00:57:10.185169+0000","last_fullsized":"2024-02-13T00:57:10.185169+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:11.886755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:11.886755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":884,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957016+0000","last_change":"2024-02-13T00:56:57.143932+0000","last_active":"2024-02-13T00:57:09.957016+0000","last_peered":"2024-02-13T00:57:09.957016+0000","last_clean":"2024-02-13T00:57:09.957016+0000","last_became_active":"2024-02-13T00:55:04.803301+0000","last_became_peered":"2024-02-13T00:55:04.803301+0000","last_unstale":"2024-02-13T00:57:09.957016+0000","last_undegraded":"2024-02-13T00:57:09.957016+0000","last_fullsized":"2024-02-13T00:57:09.957016+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:57.143854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:57.143854+0000","last_clean_scrub_stamp":"2024-02-13T00:56:57.143854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":46,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958295+0000","last_change":"2024-02-13T00:56:22.222784+0000","last_active":"2024-02-13T00:57:09.958295+0000","last_peered":"2024-02-13T00:57:09.958295+0000","last_clean":"2024-02-13T00:57:09.958295+0000","last_became_active":"2024-02-13T00:55:51.629972+0000","last_became_peered":"2024-02-13T00:55:51.629972+0000","last_unstale":"2024-02-13T00:57:09.958295+0000","last_undegraded":"2024-02-13T00:57:09.958295+0000","last_fullsized":"2024-02-13T00:57:09.958295+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:22.222692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:22.222692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":61,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959905+0000","last_change":"2024-02-13T00:55:52.215783+0000","last_active":"2024-02-13T00:57:09.959905+0000","last_peered":"2024-02-13T00:57:09.959905+0000","last_clean":"2024-02-13T00:57:09.959905+0000","last_became_active":"2024-02-13T00:54:52.749099+0000","last_became_peered":"2024-02-13T00:54:52.749099+0000","last_unstale":"2024-02-13T00:57:09.959905+0000","last_undegraded":"2024-02-13T00:57:09.959905+0000","last_fullsized":"2024-02-13T00:57:09.959905+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:52.215713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:52.215713+0000","last_clean_scrub_stamp":"2024-02-13T00:55:52.215713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"467'24","reported_seq":1000,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154690+0000","last_change":"2024-02-13T00:56:43.777275+0000","last_active":"2024-02-13T00:57:10.154690+0000","last_peered":"2024-02-13T00:57:10.154690+0000","last_clean":"2024-02-13T00:57:10.154690+0000","last_became_active":"2024-02-13T00:55:04.818088+0000","last_became_peered":"2024-02-13T00:55:04.818088+0000","last_unstale":"2024-02-13T00:57:10.154690+0000","last_undegraded":"2024-02-13T00:57:10.154690+0000","last_fullsized":"2024-02-13T00:57:10.154690+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:56:43.777208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:43.777208+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":401,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184705+0000","last_change":"2024-02-13T00:56:55.864043+0000","last_active":"2024-02-13T00:57:10.184705+0000","last_peered":"2024-02-13T00:57:10.184705+0000","last_clean":"2024-02-13T00:57:10.184705+0000","last_became_active":"2024-02-13T00:56:07.836974+0000","last_became_peered":"2024-02-13T00:56:07.836974+0000","last_unstale":"2024-02-13T00:57:10.184705+0000","last_undegraded":"2024-02-13T00:57:10.184705+0000","last_fullsized":"2024-02-13T00:57:10.184705+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:55.863926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:53.574605+0000","last_clean_scrub_stamp":"2024-02-13T00:56:55.863926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":891,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154200+0000","last_change":"2024-02-13T00:56:41.808990+0000","last_active":"2024-02-13T00:57:10.154200+0000","last_peered":"2024-02-13T00:57:10.154200+0000","last_clean":"2024-02-13T00:57:10.154200+0000","last_became_active":"2024-02-13T00:55:00.640461+0000","last_became_peered":"2024-02-13T00:55:00.640461+0000","last_unstale":"2024-02-13T00:57:10.154200+0000","last_undegraded":"2024-02-13T00:57:10.154200+0000","last_fullsized":"2024-02-13T00:57:10.154200+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:41.808911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:41.808911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":884,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957426+0000","last_change":"2024-02-13T00:56:35.978194+0000","last_active":"2024-02-13T00:57:09.957426+0000","last_peered":"2024-02-13T00:57:09.957426+0000","last_clean":"2024-02-13T00:57:09.957426+0000","last_became_active":"2024-02-13T00:55:00.650492+0000","last_became_peered":"2024-02-13T00:55:00.650492+0000","last_unstale":"2024-02-13T00:57:09.957426+0000","last_undegraded":"2024-02-13T00:57:09.957426+0000","last_fullsized":"2024-02-13T00:57:09.957426+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:35.978100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:04.249704+0000","last_clean_scrub_stamp":"2024-02-13T00:56:35.978100+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":66,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959651+0000","last_change":"2024-02-13T00:57:01.691896+0000","last_active":"2024-02-13T00:57:09.959651+0000","last_peered":"2024-02-13T00:57:09.959651+0000","last_clean":"2024-02-13T00:57:09.959651+0000","last_became_active":"2024-02-13T00:54:52.746829+0000","last_became_peered":"2024-02-13T00:54:52.746829+0000","last_unstale":"2024-02-13T00:57:09.959651+0000","last_undegraded":"2024-02-13T00:57:09.959651+0000","last_fullsized":"2024-02-13T00:57:09.959651+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:01.691809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:57:01.691809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":340,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185543+0000","last_change":"2024-02-13T00:56:07.658699+0000","last_active":"2024-02-13T00:57:10.185543+0000","last_peered":"2024-02-13T00:57:10.185543+0000","last_clean":"2024-02-13T00:57:10.185543+0000","last_became_active":"2024-02-13T00:56:07.657907+0000","last_became_peered":"2024-02-13T00:56:07.657907+0000","last_unstale":"2024-02-13T00:57:10.185543+0000","last_undegraded":"2024-02-13T00:57:10.185543+0000","last_fullsized":"2024-02-13T00:57:10.185543+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:53.729897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:55:53.729897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"467'33","reported_seq":680,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.186278+0000","last_change":"2024-02-13T00:56:15.030027+0000","last_active":"2024-02-13T00:57:10.186278+0000","last_peered":"2024-02-13T00:57:10.186278+0000","last_clean":"2024-02-13T00:57:10.186278+0000","last_became_active":"2024-02-13T00:54:56.633712+0000","last_became_peered":"2024-02-13T00:54:56.633712+0000","last_unstale":"2024-02-13T00:57:10.186278+0000","last_undegraded":"2024-02-13T00:57:10.186278+0000","last_fullsized":"2024-02-13T00:57:10.186278+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'33","last_scrub_stamp":"2024-02-13T00:56:15.029929+0000","last_deep_scrub":"467'33","last_deep_scrub_stamp":"2024-02-13T00:53:22.462526+0000","last_clean_scrub_stamp":"2024-02-13T00:56:15.029929+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":645,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184449+0000","last_change":"2024-02-13T00:56:51.843465+0000","last_active":"2024-02-13T00:57:10.184449+0000","last_peered":"2024-02-13T00:57:10.184449+0000","last_clean":"2024-02-13T00:57:10.184449+0000","last_became_active":"2024-02-13T00:56:07.837752+0000","last_became_peered":"2024-02-13T00:56:07.837752+0000","last_unstale":"2024-02-13T00:57:10.184449+0000","last_undegraded":"2024-02-13T00:57:10.184449+0000","last_fullsized":"2024-02-13T00:57:10.184449+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:51.843386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:37.044403+0000","last_clean_scrub_stamp":"2024-02-13T00:56:51.843386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":79,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959161+0000","last_change":"2024-02-13T00:56:21.131572+0000","last_active":"2024-02-13T00:57:09.959161+0000","last_peered":"2024-02-13T00:57:09.959161+0000","last_clean":"2024-02-13T00:57:09.959161+0000","last_became_active":"2024-02-13T00:55:04.781120+0000","last_became_peered":"2024-02-13T00:55:04.781120+0000","last_unstale":"2024-02-13T00:57:09.959161+0000","last_undegraded":"2024-02-13T00:57:09.959161+0000","last_fullsized":"2024-02-13T00:57:09.959161+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:21.131500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:21.131500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":54,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185040+0000","last_change":"2024-02-13T00:56:13.680249+0000","last_active":"2024-02-13T00:57:10.185040+0000","last_peered":"2024-02-13T00:57:10.185040+0000","last_clean":"2024-02-13T00:57:10.185040+0000","last_became_active":"2024-02-13T00:55:00.640413+0000","last_became_peered":"2024-02-13T00:55:00.640413+0000","last_unstale":"2024-02-13T00:57:10.185040+0000","last_undegraded":"2024-02-13T00:57:10.185040+0000","last_fullsized":"2024-02-13T00:57:10.185040+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:13.680172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:04.221506+0000","last_clean_scrub_stamp":"2024-02-13T00:56:13.680172+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":84,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.960018+0000","last_change":"2024-02-13T00:56:57.773718+0000","last_active":"2024-02-13T00:57:09.960018+0000","last_peered":"2024-02-13T00:57:09.960018+0000","last_clean":"2024-02-13T00:57:09.960018+0000","last_became_active":"2024-02-13T00:55:00.622363+0000","last_became_peered":"2024-02-13T00:55:00.622363+0000","last_unstale":"2024-02-13T00:57:09.960018+0000","last_undegraded":"2024-02-13T00:57:09.960018+0000","last_fullsized":"2024-02-13T00:57:09.960018+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:57.773620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:05.596462+0000","last_clean_scrub_stamp":"2024-02-13T00:56:57.773620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":828,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.155059+0000","last_change":"2024-02-13T00:57:08.999858+0000","last_active":"2024-02-13T00:57:10.155059+0000","last_peered":"2024-02-13T00:57:10.155059+0000","last_clean":"2024-02-13T00:57:10.155059+0000","last_became_active":"2024-02-13T00:54:56.635321+0000","last_became_peered":"2024-02-13T00:54:56.635321+0000","last_unstale":"2024-02-13T00:57:10.155059+0000","last_undegraded":"2024-02-13T00:57:10.155059+0000","last_fullsized":"2024-02-13T00:57:10.155059+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:08.999794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:57:08.999794+0000","last_clean_scrub_stamp":"2024-02-13T00:57:08.999794+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":58,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184658+0000","last_change":"2024-02-13T00:57:07.900382+0000","last_active":"2024-02-13T00:57:10.184658+0000","last_peered":"2024-02-13T00:57:10.184658+0000","last_clean":"2024-02-13T00:57:10.184658+0000","last_became_active":"2024-02-13T00:55:04.781222+0000","last_became_peered":"2024-02-13T00:55:04.781222+0000","last_unstale":"2024-02-13T00:57:10.184658+0000","last_undegraded":"2024-02-13T00:57:10.184658+0000","last_fullsized":"2024-02-13T00:57:10.184658+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:07.900310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:57:07.900310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"805'29","reported_seq":953,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:13.139424+0000","last_change":"2024-02-13T00:57:13.139424+0000","last_active":"2024-02-13T00:57:13.139424+0000","last_peered":"2024-02-13T00:57:13.139424+0000","last_clean":"2024-02-13T00:57:13.139424+0000","last_became_active":"2024-02-13T00:54:52.743094+0000","last_became_peered":"2024-02-13T00:54:52.743094+0000","last_unstale":"2024-02-13T00:57:13.139424+0000","last_undegraded":"2024-02-13T00:57:13.139424+0000","last_fullsized":"2024-02-13T00:57:13.139424+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":814,"parent":"0.0","parent_split_bits":0,"last_scrub":"805'29","last_scrub_stamp":"2024-02-13T00:57:13.139343+0000","last_deep_scrub":"805'29","last_deep_scrub_stamp":"2024-02-13T00:54:31.218066+0000","last_clean_scrub_stamp":"2024-02-13T00:57:13.139343+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":36,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.183883+0000","last_change":"2024-02-13T00:56:04.777422+0000","last_active":"2024-02-13T00:57:10.183883+0000","last_peered":"2024-02-13T00:57:10.183883+0000","last_clean":"2024-02-13T00:57:10.183883+0000","last_became_active":"2024-02-13T00:55:04.808209+0000","last_became_peered":"2024-02-13T00:55:04.808209+0000","last_unstale":"2024-02-13T00:57:10.183883+0000","last_undegraded":"2024-02-13T00:57:10.183883+0000","last_fullsized":"2024-02-13T00:57:10.183883+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:04.777361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:40:57.469743+0000","last_clean_scrub_stamp":"2024-02-13T00:56:04.777361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"394'6","reported_seq":2053,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:14.030159+0000","last_change":"2024-02-13T00:57:11.136232+0000","last_active":"2024-02-13T00:57:14.030159+0000","last_peered":"2024-02-13T00:57:14.030159+0000","last_clean":"2024-02-13T00:57:14.030159+0000","last_became_active":"2024-02-13T00:55:04.803993+0000","last_became_peered":"2024-02-13T00:55:04.803993+0000","last_unstale":"2024-02-13T00:57:14.030159+0000","last_undegraded":"2024-02-13T00:57:14.030159+0000","last_fullsized":"2024-02-13T00:57:14.030159+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'6","last_scrub_stamp":"2024-02-13T00:57:11.136156+0000","last_deep_scrub":"394'6","last_deep_scrub_stamp":"2024-02-13T00:55:40.016277+0000","last_clean_scrub_stamp":"2024-02-13T00:57:11.136156+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1157,"num_read_kb":1152,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":569,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:14.826143+0000","last_change":"2024-02-13T00:57:14.826143+0000","last_active":"2024-02-13T00:57:14.826143+0000","last_peered":"2024-02-13T00:57:14.826143+0000","last_clean":"2024-02-13T00:57:14.826143+0000","last_became_active":"2024-02-13T00:56:07.654155+0000","last_became_peered":"2024-02-13T00:56:07.654155+0000","last_unstale":"2024-02-13T00:57:14.826143+0000","last_undegraded":"2024-02-13T00:57:14.826143+0000","last_fullsized":"2024-02-13T00:57:14.826143+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:14.826028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:57:14.826028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":628,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958110+0000","last_change":"2024-02-13T00:56:50.206280+0000","last_active":"2024-02-13T00:57:09.958110+0000","last_peered":"2024-02-13T00:57:09.958110+0000","last_clean":"2024-02-13T00:57:09.958110+0000","last_became_active":"2024-02-13T00:55:51.633728+0000","last_became_peered":"2024-02-13T00:55:51.633728+0000","last_unstale":"2024-02-13T00:57:09.958110+0000","last_undegraded":"2024-02-13T00:57:09.958110+0000","last_fullsized":"2024-02-13T00:57:09.958110+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:50.206142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:50.206142+0000","last_clean_scrub_stamp":"2024-02-13T00:56:50.206142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":438,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185955+0000","last_change":"2024-02-13T00:55:48.806610+0000","last_active":"2024-02-13T00:57:10.185955+0000","last_peered":"2024-02-13T00:57:10.185955+0000","last_clean":"2024-02-13T00:57:10.185955+0000","last_became_active":"2024-02-13T00:54:56.633237+0000","last_became_peered":"2024-02-13T00:54:56.633237+0000","last_unstale":"2024-02-13T00:57:10.185955+0000","last_undegraded":"2024-02-13T00:57:10.185955+0000","last_fullsized":"2024-02-13T00:57:10.185955+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:48.806530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:55:48.806530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"821'69","reported_seq":327,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.640520+0000","last_change":"2024-02-13T00:56:32.961075+0000","last_active":"2024-02-13T00:57:10.640520+0000","last_peered":"2024-02-13T00:57:10.640520+0000","last_clean":"2024-02-13T00:57:10.640520+0000","last_became_active":"2024-02-13T00:55:04.805061+0000","last_became_peered":"2024-02-13T00:55:04.805061+0000","last_unstale":"2024-02-13T00:57:10.640520+0000","last_undegraded":"2024-02-13T00:57:10.640520+0000","last_fullsized":"2024-02-13T00:57:10.640520+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":822,"parent":"0.0","parent_split_bits":0,"last_scrub":"821'69","last_scrub_stamp":"2024-02-13T00:56:32.961000+0000","last_deep_scrub":"263'48","last_deep_scrub_stamp":"2024-02-13T00:43:48.212428+0000","last_clean_scrub_stamp":"2024-02-13T00:56:32.961000+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":77,"num_read_kb":61,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":3580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":66,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959354+0000","last_change":"2024-02-13T00:56:33.075349+0000","last_active":"2024-02-13T00:57:09.959354+0000","last_peered":"2024-02-13T00:57:09.959354+0000","last_clean":"2024-02-13T00:57:09.959354+0000","last_became_active":"2024-02-13T00:54:52.751283+0000","last_became_peered":"2024-02-13T00:54:52.751283+0000","last_unstale":"2024-02-13T00:57:09.959354+0000","last_undegraded":"2024-02-13T00:57:09.959354+0000","last_fullsized":"2024-02-13T00:57:09.959354+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:33.075264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:26.245724+0000","last_clean_scrub_stamp":"2024-02-13T00:56:33.075264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":54,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185165+0000","last_change":"2024-02-13T00:56:08.763587+0000","last_active":"2024-02-13T00:57:10.185165+0000","last_peered":"2024-02-13T00:57:10.185165+0000","last_clean":"2024-02-13T00:57:10.185165+0000","last_became_active":"2024-02-13T00:55:00.651759+0000","last_became_peered":"2024-02-13T00:55:00.651759+0000","last_unstale":"2024-02-13T00:57:10.185165+0000","last_undegraded":"2024-02-13T00:57:10.185165+0000","last_fullsized":"2024-02-13T00:57:10.185165+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:08.763528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:08.763528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":573,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185701+0000","last_change":"2024-02-13T00:56:22.228885+0000","last_active":"2024-02-13T00:57:10.185701+0000","last_peered":"2024-02-13T00:57:10.185701+0000","last_clean":"2024-02-13T00:57:10.185701+0000","last_became_active":"2024-02-13T00:55:04.786126+0000","last_became_peered":"2024-02-13T00:55:04.786126+0000","last_unstale":"2024-02-13T00:57:10.185701+0000","last_undegraded":"2024-02-13T00:57:10.185701+0000","last_fullsized":"2024-02-13T00:57:10.185701+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:22.228792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:22.228792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"814'15","reported_seq":1129,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.899052+0000","last_change":"2024-02-13T00:55:52.011279+0000","last_active":"2024-02-13T00:57:10.899052+0000","last_peered":"2024-02-13T00:57:10.899052+0000","last_clean":"2024-02-13T00:57:10.899052+0000","last_became_active":"2024-02-13T00:54:52.756936+0000","last_became_peered":"2024-02-13T00:54:52.756936+0000","last_unstale":"2024-02-13T00:57:10.899052+0000","last_undegraded":"2024-02-13T00:57:10.899052+0000","last_fullsized":"2024-02-13T00:57:10.899052+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":814,"parent":"0.0","parent_split_bits":0,"last_scrub":"814'15","last_scrub_stamp":"2024-02-13T00:55:52.011200+0000","last_deep_scrub":"227'11","last_deep_scrub_stamp":"2024-02-13T00:43:02.423358+0000","last_clean_scrub_stamp":"2024-02-13T00:55:52.011200+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"467'26","reported_seq":662,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957212+0000","last_change":"2024-02-13T00:55:49.022870+0000","last_active":"2024-02-13T00:57:09.957212+0000","last_peered":"2024-02-13T00:57:09.957212+0000","last_clean":"2024-02-13T00:57:09.957212+0000","last_became_active":"2024-02-13T00:54:56.623791+0000","last_became_peered":"2024-02-13T00:54:56.623791+0000","last_unstale":"2024-02-13T00:57:09.957212+0000","last_undegraded":"2024-02-13T00:57:09.957212+0000","last_fullsized":"2024-02-13T00:57:09.957212+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:55:49.022799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:55:49.022799+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":850,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.956872+0000","last_change":"2024-02-13T00:56:54.093470+0000","last_active":"2024-02-13T00:57:09.956872+0000","last_peered":"2024-02-13T00:57:09.956872+0000","last_clean":"2024-02-13T00:57:09.956872+0000","last_became_active":"2024-02-13T00:51:41.567277+0000","last_became_peered":"2024-02-13T00:51:41.567277+0000","last_unstale":"2024-02-13T00:57:09.956872+0000","last_undegraded":"2024-02-13T00:57:09.956872+0000","last_fullsized":"2024-02-13T00:57:09.956872+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:54.093399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:49.617988+0000","last_clean_scrub_stamp":"2024-02-13T00:56:54.093399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":889,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154107+0000","last_change":"2024-02-13T00:56:31.708170+0000","last_active":"2024-02-13T00:57:10.154107+0000","last_peered":"2024-02-13T00:57:10.154107+0000","last_clean":"2024-02-13T00:57:10.154107+0000","last_became_active":"2024-02-13T00:55:04.811884+0000","last_became_peered":"2024-02-13T00:55:04.811884+0000","last_unstale":"2024-02-13T00:57:10.154107+0000","last_undegraded":"2024-02-13T00:57:10.154107+0000","last_fullsized":"2024-02-13T00:57:10.154107+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:31.708099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:08.094228+0000","last_clean_scrub_stamp":"2024-02-13T00:56:31.708099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":808,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.956933+0000","last_change":"2024-02-13T00:56:59.113316+0000","last_active":"2024-02-13T00:57:09.956933+0000","last_peered":"2024-02-13T00:57:09.956933+0000","last_clean":"2024-02-13T00:57:09.956933+0000","last_became_active":"2024-02-13T00:55:04.802927+0000","last_became_peered":"2024-02-13T00:55:04.802927+0000","last_unstale":"2024-02-13T00:57:09.956933+0000","last_undegraded":"2024-02-13T00:57:09.956933+0000","last_fullsized":"2024-02-13T00:57:09.956933+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:59.113239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:44:34.171185+0000","last_clean_scrub_stamp":"2024-02-13T00:56:59.113239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":58,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.183783+0000","last_change":"2024-02-13T00:57:05.973384+0000","last_active":"2024-02-13T00:57:10.183783+0000","last_peered":"2024-02-13T00:57:10.183783+0000","last_clean":"2024-02-13T00:57:10.183783+0000","last_became_active":"2024-02-13T00:55:04.808568+0000","last_became_peered":"2024-02-13T00:55:04.808568+0000","last_unstale":"2024-02-13T00:57:10.183783+0000","last_undegraded":"2024-02-13T00:57:10.183783+0000","last_fullsized":"2024-02-13T00:57:10.183783+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:05.973305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:57:05.973305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"467'30","reported_seq":921,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957625+0000","last_change":"2024-02-13T00:56:41.121270+0000","last_active":"2024-02-13T00:57:09.957625+0000","last_peered":"2024-02-13T00:57:09.957625+0000","last_clean":"2024-02-13T00:57:09.957625+0000","last_became_active":"2024-02-13T00:55:51.634260+0000","last_became_peered":"2024-02-13T00:55:51.634260+0000","last_unstale":"2024-02-13T00:57:09.957625+0000","last_undegraded":"2024-02-13T00:57:09.957625+0000","last_fullsized":"2024-02-13T00:57:09.957625+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'30","last_scrub_stamp":"2024-02-13T00:56:41.121161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:41.121161+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"57'2","reported_seq":872,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959096+0000","last_change":"2024-02-13T00:56:21.216438+0000","last_active":"2024-02-13T00:57:09.959096+0000","last_peered":"2024-02-13T00:57:09.959096+0000","last_clean":"2024-02-13T00:57:09.959096+0000","last_became_active":"2024-02-13T00:55:51.635608+0000","last_became_peered":"2024-02-13T00:55:51.635608+0000","last_unstale":"2024-02-13T00:57:09.959096+0000","last_undegraded":"2024-02-13T00:57:09.959096+0000","last_fullsized":"2024-02-13T00:57:09.959096+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2024-02-13T00:56:21.216367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:21.216367+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":54,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.183802+0000","last_change":"2024-02-13T00:55:54.603621+0000","last_active":"2024-02-13T00:57:10.183802+0000","last_peered":"2024-02-13T00:57:10.183802+0000","last_clean":"2024-02-13T00:57:10.183802+0000","last_became_active":"2024-02-13T00:55:00.647645+0000","last_became_peered":"2024-02-13T00:55:00.647645+0000","last_unstale":"2024-02-13T00:57:10.183802+0000","last_undegraded":"2024-02-13T00:57:10.183802+0000","last_fullsized":"2024-02-13T00:57:10.183802+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:54.603530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:45.660653+0000","last_clean_scrub_stamp":"2024-02-13T00:55:54.603530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":814,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154777+0000","last_change":"2024-02-13T00:56:06.954325+0000","last_active":"2024-02-13T00:57:10.154777+0000","last_peered":"2024-02-13T00:57:10.154777+0000","last_clean":"2024-02-13T00:57:10.154777+0000","last_became_active":"2024-02-13T00:54:52.755872+0000","last_became_peered":"2024-02-13T00:54:52.755872+0000","last_unstale":"2024-02-13T00:57:10.154777+0000","last_undegraded":"2024-02-13T00:57:10.154777+0000","last_fullsized":"2024-02-13T00:57:10.154777+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:06.954236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:06.954236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"812'33","reported_seq":1027,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154790+0000","last_change":"2024-02-13T00:56:14.954843+0000","last_active":"2024-02-13T00:57:10.154790+0000","last_peered":"2024-02-13T00:57:10.154790+0000","last_clean":"2024-02-13T00:57:10.154790+0000","last_became_active":"2024-02-13T00:55:00.642447+0000","last_became_peered":"2024-02-13T00:55:00.642447+0000","last_unstale":"2024-02-13T00:57:10.154790+0000","last_undegraded":"2024-02-13T00:57:10.154790+0000","last_fullsized":"2024-02-13T00:57:10.154790+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"812'33","last_scrub_stamp":"2024-02-13T00:56:14.954755+0000","last_deep_scrub":"812'33","last_deep_scrub_stamp":"2024-02-13T00:54:52.301469+0000","last_clean_scrub_stamp":"2024-02-13T00:56:14.954755+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":58,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.183695+0000","last_change":"2024-02-13T00:56:56.899420+0000","last_active":"2024-02-13T00:57:10.183695+0000","last_peered":"2024-02-13T00:57:10.183695+0000","last_clean":"2024-02-13T00:57:10.183695+0000","last_became_active":"2024-02-13T00:55:04.808145+0000","last_became_peered":"2024-02-13T00:55:04.808145+0000","last_unstale":"2024-02-13T00:57:10.183695+0000","last_undegraded":"2024-02-13T00:57:10.183695+0000","last_fullsized":"2024-02-13T00:57:10.183695+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:56.899331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:30.095307+0000","last_clean_scrub_stamp":"2024-02-13T00:56:56.899331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":58,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184087+0000","last_change":"2024-02-13T00:57:08.895263+0000","last_active":"2024-02-13T00:57:10.184087+0000","last_peered":"2024-02-13T00:57:10.184087+0000","last_clean":"2024-02-13T00:57:10.184087+0000","last_became_active":"2024-02-13T00:55:04.781860+0000","last_became_peered":"2024-02-13T00:55:04.781860+0000","last_unstale":"2024-02-13T00:57:10.184087+0000","last_undegraded":"2024-02-13T00:57:10.184087+0000","last_fullsized":"2024-02-13T00:57:10.184087+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:08.895198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:57:08.895198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":582,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.186350+0000","last_change":"2024-02-13T00:56:33.243096+0000","last_active":"2024-02-13T00:57:10.186350+0000","last_peered":"2024-02-13T00:57:10.186350+0000","last_clean":"2024-02-13T00:57:10.186350+0000","last_became_active":"2024-02-13T00:55:04.793577+0000","last_became_peered":"2024-02-13T00:55:04.793577+0000","last_unstale":"2024-02-13T00:57:10.186350+0000","last_undegraded":"2024-02-13T00:57:10.186350+0000","last_fullsized":"2024-02-13T00:57:10.186350+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:33.243018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:33.243018+0000","last_clean_scrub_stamp":"2024-02-13T00:56:33.243018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"467'18","reported_seq":88,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185141+0000","last_change":"2024-02-13T00:57:02.710707+0000","last_active":"2024-02-13T00:57:10.185141+0000","last_peered":"2024-02-13T00:57:10.185141+0000","last_clean":"2024-02-13T00:57:10.185141+0000","last_became_active":"2024-02-13T00:56:07.837031+0000","last_became_peered":"2024-02-13T00:56:07.837031+0000","last_unstale":"2024-02-13T00:57:10.185141+0000","last_undegraded":"2024-02-13T00:57:10.185141+0000","last_fullsized":"2024-02-13T00:57:10.185141+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'18","last_scrub_stamp":"2024-02-13T00:57:02.710631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:57:02.710631+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":580,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184883+0000","last_change":"2024-02-13T00:56:51.083897+0000","last_active":"2024-02-13T00:57:10.184883+0000","last_peered":"2024-02-13T00:57:10.184883+0000","last_clean":"2024-02-13T00:57:10.184883+0000","last_became_active":"2024-02-13T00:54:56.632379+0000","last_became_peered":"2024-02-13T00:54:56.632379+0000","last_unstale":"2024-02-13T00:57:10.184883+0000","last_undegraded":"2024-02-13T00:57:10.184883+0000","last_fullsized":"2024-02-13T00:57:10.184883+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:51.083797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:01.439513+0000","last_clean_scrub_stamp":"2024-02-13T00:56:51.083797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":566,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185621+0000","last_change":"2024-02-13T00:57:01.823383+0000","last_active":"2024-02-13T00:57:10.185621+0000","last_peered":"2024-02-13T00:57:10.185621+0000","last_clean":"2024-02-13T00:57:10.185621+0000","last_became_active":"2024-02-13T00:56:07.653415+0000","last_became_peered":"2024-02-13T00:56:07.653415+0000","last_unstale":"2024-02-13T00:57:10.185621+0000","last_undegraded":"2024-02-13T00:57:10.185621+0000","last_fullsized":"2024-02-13T00:57:10.185621+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:01.823300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:57:01.823300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":59,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185459+0000","last_change":"2024-02-13T00:56:22.668873+0000","last_active":"2024-02-13T00:57:10.185459+0000","last_peered":"2024-02-13T00:57:10.185459+0000","last_clean":"2024-02-13T00:57:10.185459+0000","last_became_active":"2024-02-13T00:55:00.654451+0000","last_became_peered":"2024-02-13T00:55:00.654451+0000","last_unstale":"2024-02-13T00:57:10.185459+0000","last_undegraded":"2024-02-13T00:57:10.185459+0000","last_fullsized":"2024-02-13T00:57:10.185459+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:22.668785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:22.668785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"467'39","reported_seq":70,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185417+0000","last_change":"2024-02-13T00:56:14.636419+0000","last_active":"2024-02-13T00:57:10.185417+0000","last_peered":"2024-02-13T00:57:10.185417+0000","last_clean":"2024-02-13T00:57:10.185417+0000","last_became_active":"2024-02-13T00:55:00.646426+0000","last_became_peered":"2024-02-13T00:55:00.646426+0000","last_unstale":"2024-02-13T00:57:10.185417+0000","last_undegraded":"2024-02-13T00:57:10.185417+0000","last_fullsized":"2024-02-13T00:57:10.185417+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'39","last_scrub_stamp":"2024-02-13T00:56:14.636328+0000","last_deep_scrub":"76'29","last_deep_scrub_stamp":"2024-02-13T00:38:46.169344+0000","last_clean_scrub_stamp":"2024-02-13T00:56:14.636328+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":70,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959029+0000","last_change":"2024-02-13T00:56:25.127870+0000","last_active":"2024-02-13T00:57:09.959029+0000","last_peered":"2024-02-13T00:57:09.959029+0000","last_clean":"2024-02-13T00:57:09.959029+0000","last_became_active":"2024-02-13T00:55:00.622671+0000","last_became_peered":"2024-02-13T00:55:00.622671+0000","last_unstale":"2024-02-13T00:57:09.959029+0000","last_undegraded":"2024-02-13T00:57:09.959029+0000","last_fullsized":"2024-02-13T00:57:09.959029+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:25.127798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:25.127798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"814'11","reported_seq":93,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959486+0000","last_change":"2024-02-13T00:55:54.194316+0000","last_active":"2024-02-13T00:57:09.959486+0000","last_peered":"2024-02-13T00:57:09.959486+0000","last_clean":"2024-02-13T00:57:09.959486+0000","last_became_active":"2024-02-13T00:54:52.746048+0000","last_became_peered":"2024-02-13T00:54:52.746048+0000","last_unstale":"2024-02-13T00:57:09.959486+0000","last_undegraded":"2024-02-13T00:57:09.959486+0000","last_fullsized":"2024-02-13T00:57:09.959486+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"814'11","last_scrub_stamp":"2024-02-13T00:55:54.194242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:55:54.194242+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":889,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154668+0000","last_change":"2024-02-13T00:55:42.941918+0000","last_active":"2024-02-13T00:57:10.154668+0000","last_peered":"2024-02-13T00:57:10.154668+0000","last_clean":"2024-02-13T00:57:10.154668+0000","last_became_active":"2024-02-13T00:55:00.638952+0000","last_became_peered":"2024-02-13T00:55:00.638952+0000","last_unstale":"2024-02-13T00:57:10.154668+0000","last_undegraded":"2024-02-13T00:57:10.154668+0000","last_fullsized":"2024-02-13T00:57:10.154668+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:42.941846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:55:42.941846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":354,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185342+0000","last_change":"2024-02-13T00:56:44.030693+0000","last_active":"2024-02-13T00:57:10.185342+0000","last_peered":"2024-02-13T00:57:10.185342+0000","last_clean":"2024-02-13T00:57:10.185342+0000","last_became_active":"2024-02-13T00:56:07.656128+0000","last_became_peered":"2024-02-13T00:56:07.656128+0000","last_unstale":"2024-02-13T00:57:10.185342+0000","last_undegraded":"2024-02-13T00:57:10.185342+0000","last_fullsized":"2024-02-13T00:57:10.185342+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:44.030613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:44.030613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"851'27","reported_seq":99,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959519+0000","last_change":"2024-02-13T00:56:24.112171+0000","last_active":"2024-02-13T00:57:09.959519+0000","last_peered":"2024-02-13T00:57:09.959519+0000","last_clean":"2024-02-13T00:57:09.959519+0000","last_became_active":"2024-02-13T00:55:00.622624+0000","last_became_peered":"2024-02-13T00:55:00.622624+0000","last_unstale":"2024-02-13T00:57:09.959519+0000","last_undegraded":"2024-02-13T00:57:09.959519+0000","last_fullsized":"2024-02-13T00:57:09.959519+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'27","last_scrub_stamp":"2024-02-13T00:56:24.112097+0000","last_deep_scrub":"851'27","last_deep_scrub_stamp":"2024-02-13T00:56:24.112097+0000","last_clean_scrub_stamp":"2024-02-13T00:56:24.112097+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":30,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":904,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154423+0000","last_change":"2024-02-13T00:55:57.950855+0000","last_active":"2024-02-13T00:57:10.154423+0000","last_peered":"2024-02-13T00:57:10.154423+0000","last_clean":"2024-02-13T00:57:10.154423+0000","last_became_active":"2024-02-13T00:55:00.638516+0000","last_became_peered":"2024-02-13T00:55:00.638516+0000","last_unstale":"2024-02-13T00:57:10.154423+0000","last_undegraded":"2024-02-13T00:57:10.154423+0000","last_fullsized":"2024-02-13T00:57:10.154423+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:57.950760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:57.950760+0000","last_clean_scrub_stamp":"2024-02-13T00:55:57.950760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"821'19","reported_seq":104,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.639883+0000","last_change":"2024-02-13T00:57:08.695233+0000","last_active":"2024-02-13T00:57:10.639883+0000","last_peered":"2024-02-13T00:57:10.639883+0000","last_clean":"2024-02-13T00:57:10.639883+0000","last_became_active":"2024-02-13T00:55:04.810233+0000","last_became_peered":"2024-02-13T00:55:04.810233+0000","last_unstale":"2024-02-13T00:57:10.639883+0000","last_undegraded":"2024-02-13T00:57:10.639883+0000","last_fullsized":"2024-02-13T00:57:10.639883+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"821'19","last_scrub_stamp":"2024-02-13T00:57:08.695165+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-02-13T00:39:22.269104+0000","last_clean_scrub_stamp":"2024-02-13T00:57:08.695165+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":51,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959253+0000","last_change":"2024-02-13T00:57:07.314254+0000","last_active":"2024-02-13T00:57:09.959253+0000","last_peered":"2024-02-13T00:57:09.959253+0000","last_clean":"2024-02-13T00:57:09.959253+0000","last_became_active":"2024-02-13T00:55:51.629259+0000","last_became_peered":"2024-02-13T00:55:51.629259+0000","last_unstale":"2024-02-13T00:57:09.959253+0000","last_undegraded":"2024-02-13T00:57:09.959253+0000","last_fullsized":"2024-02-13T00:57:09.959253+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:07.314182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:52:27.933639+0000","last_clean_scrub_stamp":"2024-02-13T00:57:07.314182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":75,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959275+0000","last_change":"2024-02-13T00:56:55.844080+0000","last_active":"2024-02-13T00:57:09.959275+0000","last_peered":"2024-02-13T00:57:09.959275+0000","last_clean":"2024-02-13T00:57:09.959275+0000","last_became_active":"2024-02-13T00:55:00.623176+0000","last_became_peered":"2024-02-13T00:55:00.623176+0000","last_unstale":"2024-02-13T00:57:09.959275+0000","last_undegraded":"2024-02-13T00:57:09.959275+0000","last_fullsized":"2024-02-13T00:57:09.959275+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:55.843976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:55.843976+0000","last_clean_scrub_stamp":"2024-02-13T00:56:55.843976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"467'19","reported_seq":935,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957123+0000","last_change":"2024-02-13T00:56:12.926290+0000","last_active":"2024-02-13T00:57:09.957123+0000","last_peered":"2024-02-13T00:57:09.957123+0000","last_clean":"2024-02-13T00:57:09.957123+0000","last_became_active":"2024-02-13T00:55:04.802878+0000","last_became_peered":"2024-02-13T00:55:04.802878+0000","last_unstale":"2024-02-13T00:57:09.957123+0000","last_undegraded":"2024-02-13T00:57:09.957123+0000","last_fullsized":"2024-02-13T00:57:09.957123+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'19","last_scrub_stamp":"2024-02-13T00:56:12.926213+0000","last_deep_scrub":"83'13","last_deep_scrub_stamp":"2024-02-13T00:39:12.348480+0000","last_clean_scrub_stamp":"2024-02-13T00:56:12.926213+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":591,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185605+0000","last_change":"2024-02-13T00:56:59.978959+0000","last_active":"2024-02-13T00:57:10.185605+0000","last_peered":"2024-02-13T00:57:10.185605+0000","last_clean":"2024-02-13T00:57:10.185605+0000","last_became_active":"2024-02-13T00:55:00.638271+0000","last_became_peered":"2024-02-13T00:55:00.638271+0000","last_unstale":"2024-02-13T00:57:10.185605+0000","last_undegraded":"2024-02-13T00:57:10.185605+0000","last_fullsized":"2024-02-13T00:57:10.185605+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:59.978875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:43:49.095888+0000","last_clean_scrub_stamp":"2024-02-13T00:56:59.978875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":59,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184269+0000","last_change":"2024-02-13T00:56:50.693498+0000","last_active":"2024-02-13T00:57:10.184269+0000","last_peered":"2024-02-13T00:57:10.184269+0000","last_clean":"2024-02-13T00:57:10.184269+0000","last_became_active":"2024-02-13T00:55:00.655453+0000","last_became_peered":"2024-02-13T00:55:00.655453+0000","last_unstale":"2024-02-13T00:57:10.184269+0000","last_undegraded":"2024-02-13T00:57:10.184269+0000","last_fullsized":"2024-02-13T00:57:10.184269+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:50.693436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:46:32.638349+0000","last_clean_scrub_stamp":"2024-02-13T00:56:50.693436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":840,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154057+0000","last_change":"2024-02-13T00:56:57.796543+0000","last_active":"2024-02-13T00:57:10.154057+0000","last_peered":"2024-02-13T00:57:10.154057+0000","last_clean":"2024-02-13T00:57:10.154057+0000","last_became_active":"2024-02-13T00:54:52.752154+0000","last_became_peered":"2024-02-13T00:54:52.752154+0000","last_unstale":"2024-02-13T00:57:10.154057+0000","last_undegraded":"2024-02-13T00:57:10.154057+0000","last_fullsized":"2024-02-13T00:57:10.154057+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:57.796480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:47.967067+0000","last_clean_scrub_stamp":"2024-02-13T00:56:57.796480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":572,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:11.823379+0000","last_change":"2024-02-13T00:57:11.823379+0000","last_active":"2024-02-13T00:57:11.823379+0000","last_peered":"2024-02-13T00:57:11.823379+0000","last_clean":"2024-02-13T00:57:11.823379+0000","last_became_active":"2024-02-13T00:51:41.570732+0000","last_became_peered":"2024-02-13T00:51:41.570732+0000","last_unstale":"2024-02-13T00:57:11.823379+0000","last_undegraded":"2024-02-13T00:57:11.823379+0000","last_fullsized":"2024-02-13T00:57:11.823379+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:11.823292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:57:11.823292+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"467'24","reported_seq":614,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957847+0000","last_change":"2024-02-13T00:56:42.111865+0000","last_active":"2024-02-13T00:57:09.957847+0000","last_peered":"2024-02-13T00:57:09.957847+0000","last_clean":"2024-02-13T00:57:09.957847+0000","last_became_active":"2024-02-13T00:55:51.633265+0000","last_became_peered":"2024-02-13T00:55:51.633265+0000","last_unstale":"2024-02-13T00:57:09.957847+0000","last_undegraded":"2024-02-13T00:57:09.957847+0000","last_fullsized":"2024-02-13T00:57:09.957847+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:56:42.111782+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-02-13T00:39:03.189237+0000","last_clean_scrub_stamp":"2024-02-13T00:56:42.111782+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":850,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958036+0000","last_change":"2024-02-13T00:57:06.158548+0000","last_active":"2024-02-13T00:57:09.958036+0000","last_peered":"2024-02-13T00:57:09.958036+0000","last_clean":"2024-02-13T00:57:09.958036+0000","last_became_active":"2024-02-13T00:54:52.737403+0000","last_became_peered":"2024-02-13T00:54:52.737403+0000","last_unstale":"2024-02-13T00:57:09.958036+0000","last_undegraded":"2024-02-13T00:57:09.958036+0000","last_fullsized":"2024-02-13T00:57:09.958036+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:06.158476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:45:53.097481+0000","last_clean_scrub_stamp":"2024-02-13T00:57:06.158476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":882,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957172+0000","last_change":"2024-02-13T00:57:03.112124+0000","last_active":"2024-02-13T00:57:09.957172+0000","last_peered":"2024-02-13T00:57:09.957172+0000","last_clean":"2024-02-13T00:57:09.957172+0000","last_became_active":"2024-02-13T00:55:00.636526+0000","last_became_peered":"2024-02-13T00:55:00.636526+0000","last_unstale":"2024-02-13T00:57:09.957172+0000","last_undegraded":"2024-02-13T00:57:09.957172+0000","last_fullsized":"2024-02-13T00:57:09.957172+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:03.112023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:57:03.112023+0000","last_clean_scrub_stamp":"2024-02-13T00:57:03.112023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":54,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184646+0000","last_change":"2024-02-13T00:56:02.690748+0000","last_active":"2024-02-13T00:57:10.184646+0000","last_peered":"2024-02-13T00:57:10.184646+0000","last_clean":"2024-02-13T00:57:10.184646+0000","last_became_active":"2024-02-13T00:55:00.655473+0000","last_became_peered":"2024-02-13T00:55:00.655473+0000","last_unstale":"2024-02-13T00:57:10.184646+0000","last_undegraded":"2024-02-13T00:57:10.184646+0000","last_fullsized":"2024-02-13T00:57:10.184646+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:02.690640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:53:17.006804+0000","last_clean_scrub_stamp":"2024-02-13T00:56:02.690640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":591,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184411+0000","last_change":"2024-02-13T00:56:34.267342+0000","last_active":"2024-02-13T00:57:10.184411+0000","last_peered":"2024-02-13T00:57:10.184411+0000","last_clean":"2024-02-13T00:57:10.184411+0000","last_became_active":"2024-02-13T00:55:00.638786+0000","last_became_peered":"2024-02-13T00:55:00.638786+0000","last_unstale":"2024-02-13T00:57:10.184411+0000","last_undegraded":"2024-02-13T00:57:10.184411+0000","last_fullsized":"2024-02-13T00:57:10.184411+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:34.267245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:34.267245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"467'8","reported_seq":65,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.183605+0000","last_change":"2024-02-13T00:56:42.549442+0000","last_active":"2024-02-13T00:57:10.183605+0000","last_peered":"2024-02-13T00:57:10.183605+0000","last_clean":"2024-02-13T00:57:10.183605+0000","last_became_active":"2024-02-13T00:55:00.647721+0000","last_became_peered":"2024-02-13T00:55:00.647721+0000","last_unstale":"2024-02-13T00:57:10.183605+0000","last_undegraded":"2024-02-13T00:57:10.183605+0000","last_fullsized":"2024-02-13T00:57:10.183605+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'8","last_scrub_stamp":"2024-02-13T00:56:42.549360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:42.549360+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":70,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959767+0000","last_change":"2024-02-13T00:56:31.005069+0000","last_active":"2024-02-13T00:57:09.959767+0000","last_peered":"2024-02-13T00:57:09.959767+0000","last_clean":"2024-02-13T00:57:09.959767+0000","last_became_active":"2024-02-13T00:55:04.781288+0000","last_became_peered":"2024-02-13T00:55:04.781288+0000","last_unstale":"2024-02-13T00:57:09.959767+0000","last_undegraded":"2024-02-13T00:57:09.959767+0000","last_fullsized":"2024-02-13T00:57:09.959767+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:31.004988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.499576+0000","last_clean_scrub_stamp":"2024-02-13T00:56:31.004988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":582,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184517+0000","last_change":"2024-02-13T00:56:23.234395+0000","last_active":"2024-02-13T00:57:10.184517+0000","last_peered":"2024-02-13T00:57:10.184517+0000","last_clean":"2024-02-13T00:57:10.184517+0000","last_became_active":"2024-02-13T00:54:56.632657+0000","last_became_peered":"2024-02-13T00:54:56.632657+0000","last_unstale":"2024-02-13T00:57:10.184517+0000","last_undegraded":"2024-02-13T00:57:10.184517+0000","last_fullsized":"2024-02-13T00:57:10.184517+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:23.234306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:56:23.234306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":542,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185342+0000","last_change":"2024-02-13T00:56:40.085654+0000","last_active":"2024-02-13T00:57:10.185342+0000","last_peered":"2024-02-13T00:57:10.185342+0000","last_clean":"2024-02-13T00:57:10.185342+0000","last_became_active":"2024-02-13T00:56:07.653436+0000","last_became_peered":"2024-02-13T00:56:07.653436+0000","last_unstale":"2024-02-13T00:57:10.185342+0000","last_undegraded":"2024-02-13T00:57:10.185342+0000","last_fullsized":"2024-02-13T00:57:10.185342+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:40.085569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:42:56.479450+0000","last_clean_scrub_stamp":"2024-02-13T00:56:40.085569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":66,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959104+0000","last_change":"2024-02-13T00:56:59.762883+0000","last_active":"2024-02-13T00:57:09.959104+0000","last_peered":"2024-02-13T00:57:09.959104+0000","last_clean":"2024-02-13T00:57:09.959104+0000","last_became_active":"2024-02-13T00:54:52.746519+0000","last_became_peered":"2024-02-13T00:54:52.746519+0000","last_unstale":"2024-02-13T00:57:09.959104+0000","last_undegraded":"2024-02-13T00:57:09.959104+0000","last_fullsized":"2024-02-13T00:57:09.959104+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:59.762800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:59.762800+0000","last_clean_scrub_stamp":"2024-02-13T00:56:59.762800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"850'31","reported_seq":50,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-02-13T00:57:14.923919+0000","last_change":"2024-02-13T00:57:14.923919+0000","last_active":"2024-02-13T00:57:14.923919+0000","last_peered":"2024-02-13T00:57:14.923919+0000","last_clean":"2024-02-13T00:57:14.923919+0000","last_became_active":"2024-02-13T00:56:07.653569+0000","last_became_peered":"2024-02-13T00:56:07.653569+0000","last_unstale":"2024-02-13T00:57:14.923919+0000","last_undegraded":"2024-02-13T00:57:14.923919+0000","last_fullsized":"2024-02-13T00:57:14.923919+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"850'31","last_scrub_stamp":"2024-02-13T00:57:14.923846+0000","last_deep_scrub":"850'31","last_deep_scrub_stamp":"2024-02-13T00:56:10.788366+0000","last_clean_scrub_stamp":"2024-02-13T00:57:14.923846+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":582,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.186345+0000","last_change":"2024-02-13T00:56:16.045968+0000","last_active":"2024-02-13T00:57:10.186345+0000","last_peered":"2024-02-13T00:57:10.186345+0000","last_clean":"2024-02-13T00:57:10.186345+0000","last_became_active":"2024-02-13T00:55:04.781662+0000","last_became_peered":"2024-02-13T00:55:04.781662+0000","last_unstale":"2024-02-13T00:57:10.186345+0000","last_undegraded":"2024-02-13T00:57:10.186345+0000","last_fullsized":"2024-02-13T00:57:10.186345+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:16.045873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:16.045873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":551,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184594+0000","last_change":"2024-02-13T00:57:08.806580+0000","last_active":"2024-02-13T00:57:10.184594+0000","last_peered":"2024-02-13T00:57:10.184594+0000","last_clean":"2024-02-13T00:57:10.184594+0000","last_became_active":"2024-02-13T00:56:07.650266+0000","last_became_peered":"2024-02-13T00:56:07.650266+0000","last_unstale":"2024-02-13T00:57:10.184594+0000","last_undegraded":"2024-02-13T00:57:10.184594+0000","last_fullsized":"2024-02-13T00:57:10.184594+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:08.806509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:44.215312+0000","last_clean_scrub_stamp":"2024-02-13T00:57:08.806509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":59,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184353+0000","last_change":"2024-02-13T00:56:18.713246+0000","last_active":"2024-02-13T00:57:10.184353+0000","last_peered":"2024-02-13T00:57:10.184353+0000","last_clean":"2024-02-13T00:57:10.184353+0000","last_became_active":"2024-02-13T00:55:00.655926+0000","last_became_peered":"2024-02-13T00:55:00.655926+0000","last_unstale":"2024-02-13T00:57:10.184353+0000","last_undegraded":"2024-02-13T00:57:10.184353+0000","last_fullsized":"2024-02-13T00:57:10.184353+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:18.713176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.243527+0000","last_clean_scrub_stamp":"2024-02-13T00:56:18.713176+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":518,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185200+0000","last_change":"2024-02-13T00:56:28.119573+0000","last_active":"2024-02-13T00:57:10.185200+0000","last_peered":"2024-02-13T00:57:10.185200+0000","last_clean":"2024-02-13T00:57:10.185200+0000","last_became_active":"2024-02-13T00:56:07.651694+0000","last_became_peered":"2024-02-13T00:56:07.651694+0000","last_unstale":"2024-02-13T00:57:10.185200+0000","last_undegraded":"2024-02-13T00:57:10.185200+0000","last_fullsized":"2024-02-13T00:57:10.185200+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:28.119494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:28.119494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"467'32","reported_seq":999,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154913+0000","last_change":"2024-02-13T00:57:01.850757+0000","last_active":"2024-02-13T00:57:10.154913+0000","last_peered":"2024-02-13T00:57:10.154913+0000","last_clean":"2024-02-13T00:57:10.154913+0000","last_became_active":"2024-02-13T00:55:00.638505+0000","last_became_peered":"2024-02-13T00:55:00.638505+0000","last_unstale":"2024-02-13T00:57:10.154913+0000","last_undegraded":"2024-02-13T00:57:10.154913+0000","last_fullsized":"2024-02-13T00:57:10.154913+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'32","last_scrub_stamp":"2024-02-13T00:57:01.850685+0000","last_deep_scrub":"79'22","last_deep_scrub_stamp":"2024-02-13T00:38:52.380369+0000","last_clean_scrub_stamp":"2024-02-13T00:57:01.850685+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":70,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958636+0000","last_change":"2024-02-13T00:56:04.218997+0000","last_active":"2024-02-13T00:57:09.958636+0000","last_peered":"2024-02-13T00:57:09.958636+0000","last_clean":"2024-02-13T00:57:09.958636+0000","last_became_active":"2024-02-13T00:54:56.630920+0000","last_became_peered":"2024-02-13T00:54:56.630920+0000","last_unstale":"2024-02-13T00:57:09.958636+0000","last_undegraded":"2024-02-13T00:57:09.958636+0000","last_fullsized":"2024-02-13T00:57:09.958636+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:04.218912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:04.218912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"64'1","reported_seq":496,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185653+0000","last_change":"2024-02-13T00:56:19.082248+0000","last_active":"2024-02-13T00:57:10.185653+0000","last_peered":"2024-02-13T00:57:10.185653+0000","last_clean":"2024-02-13T00:57:10.185653+0000","last_became_active":"2024-02-13T00:56:07.658268+0000","last_became_peered":"2024-02-13T00:56:07.658268+0000","last_unstale":"2024-02-13T00:57:10.185653+0000","last_undegraded":"2024-02-13T00:57:10.185653+0000","last_fullsized":"2024-02-13T00:57:10.185653+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-02-13T00:56:19.082165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:19.082165+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":817,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958136+0000","last_change":"2024-02-13T00:57:07.144092+0000","last_active":"2024-02-13T00:57:09.958136+0000","last_peered":"2024-02-13T00:57:09.958136+0000","last_clean":"2024-02-13T00:57:09.958136+0000","last_became_active":"2024-02-13T00:56:07.650605+0000","last_became_peered":"2024-02-13T00:56:07.650605+0000","last_unstale":"2024-02-13T00:57:09.958136+0000","last_undegraded":"2024-02-13T00:57:09.958136+0000","last_fullsized":"2024-02-13T00:57:09.958136+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:07.144025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:57:07.144025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"467'25","reported_seq":641,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.186458+0000","last_change":"2024-02-13T00:56:24.240884+0000","last_active":"2024-02-13T00:57:10.186458+0000","last_peered":"2024-02-13T00:57:10.186458+0000","last_clean":"2024-02-13T00:57:10.186458+0000","last_became_active":"2024-02-13T00:56:07.657465+0000","last_became_peered":"2024-02-13T00:56:07.657465+0000","last_unstale":"2024-02-13T00:57:10.186458+0000","last_undegraded":"2024-02-13T00:57:10.186458+0000","last_fullsized":"2024-02-13T00:57:10.186458+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'25","last_scrub_stamp":"2024-02-13T00:56:24.240803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:24.240803+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":54,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184601+0000","last_change":"2024-02-13T00:56:12.718088+0000","last_active":"2024-02-13T00:57:10.184601+0000","last_peered":"2024-02-13T00:57:10.184601+0000","last_clean":"2024-02-13T00:57:10.184601+0000","last_became_active":"2024-02-13T00:55:00.655998+0000","last_became_peered":"2024-02-13T00:55:00.655998+0000","last_unstale":"2024-02-13T00:57:10.184601+0000","last_undegraded":"2024-02-13T00:57:10.184601+0000","last_fullsized":"2024-02-13T00:57:10.184601+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:12.718020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:12.718020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":862,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957980+0000","last_change":"2024-02-13T00:56:48.070734+0000","last_active":"2024-02-13T00:57:09.957980+0000","last_peered":"2024-02-13T00:57:09.957980+0000","last_clean":"2024-02-13T00:57:09.957980+0000","last_became_active":"2024-02-13T00:54:56.623506+0000","last_became_peered":"2024-02-13T00:54:56.623506+0000","last_unstale":"2024-02-13T00:57:09.957980+0000","last_undegraded":"2024-02-13T00:57:09.957980+0000","last_fullsized":"2024-02-13T00:57:09.957980+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:48.070668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:23.031298+0000","last_clean_scrub_stamp":"2024-02-13T00:56:48.070668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":564,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185887+0000","last_change":"2024-02-13T00:55:50.874302+0000","last_active":"2024-02-13T00:57:10.185887+0000","last_peered":"2024-02-13T00:57:10.185887+0000","last_clean":"2024-02-13T00:57:10.185887+0000","last_became_active":"2024-02-13T00:55:04.793613+0000","last_became_peered":"2024-02-13T00:55:04.793613+0000","last_unstale":"2024-02-13T00:57:10.185887+0000","last_undegraded":"2024-02-13T00:57:10.185887+0000","last_fullsized":"2024-02-13T00:57:10.185887+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:50.874222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:50.874222+0000","last_clean_scrub_stamp":"2024-02-13T00:55:50.874222+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"467'27","reported_seq":74,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184226+0000","last_change":"2024-02-13T00:56:42.924806+0000","last_active":"2024-02-13T00:57:10.184226+0000","last_peered":"2024-02-13T00:57:10.184226+0000","last_clean":"2024-02-13T00:57:10.184226+0000","last_became_active":"2024-02-13T00:55:04.797172+0000","last_became_peered":"2024-02-13T00:55:04.797172+0000","last_unstale":"2024-02-13T00:57:10.184226+0000","last_undegraded":"2024-02-13T00:57:10.184226+0000","last_fullsized":"2024-02-13T00:57:10.184226+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:56:42.924722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:42.924722+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":98,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957017+0000","last_change":"2024-02-13T00:56:39.979443+0000","last_active":"2024-02-13T00:57:09.957017+0000","last_peered":"2024-02-13T00:57:09.957017+0000","last_clean":"2024-02-13T00:57:09.957017+0000","last_became_active":"2024-02-13T00:54:56.623377+0000","last_became_peered":"2024-02-13T00:54:56.623377+0000","last_unstale":"2024-02-13T00:57:09.957017+0000","last_undegraded":"2024-02-13T00:57:09.957017+0000","last_fullsized":"2024-02-13T00:57:09.957017+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:39.979364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:39.979364+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":559,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958744+0000","last_change":"2024-02-13T00:56:25.089788+0000","last_active":"2024-02-13T00:57:09.958744+0000","last_peered":"2024-02-13T00:57:09.958744+0000","last_clean":"2024-02-13T00:57:09.958744+0000","last_became_active":"2024-02-13T00:55:51.633676+0000","last_became_peered":"2024-02-13T00:55:51.633676+0000","last_unstale":"2024-02-13T00:57:09.958744+0000","last_undegraded":"2024-02-13T00:57:09.958744+0000","last_fullsized":"2024-02-13T00:57:09.958744+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:25.089713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:25.089713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":474,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185810+0000","last_change":"2024-02-13T00:56:58.829795+0000","last_active":"2024-02-13T00:57:10.185810+0000","last_peered":"2024-02-13T00:57:10.185810+0000","last_clean":"2024-02-13T00:57:10.185810+0000","last_became_active":"2024-02-13T00:56:07.656780+0000","last_became_peered":"2024-02-13T00:56:07.656780+0000","last_unstale":"2024-02-13T00:57:10.185810+0000","last_undegraded":"2024-02-13T00:57:10.185810+0000","last_fullsized":"2024-02-13T00:57:10.185810+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:58.829723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:56:58.829723+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"467'24","reported_seq":73,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184914+0000","last_change":"2024-02-13T00:56:52.701271+0000","last_active":"2024-02-13T00:57:10.184914+0000","last_peered":"2024-02-13T00:57:10.184914+0000","last_clean":"2024-02-13T00:57:10.184914+0000","last_became_active":"2024-02-13T00:55:00.648023+0000","last_became_peered":"2024-02-13T00:55:00.648023+0000","last_unstale":"2024-02-13T00:57:10.184914+0000","last_undegraded":"2024-02-13T00:57:10.184914+0000","last_fullsized":"2024-02-13T00:57:10.184914+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'24","last_scrub_stamp":"2024-02-13T00:56:52.701207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:52.701207+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":59,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184063+0000","last_change":"2024-02-13T00:56:56.726615+0000","last_active":"2024-02-13T00:57:10.184063+0000","last_peered":"2024-02-13T00:57:10.184063+0000","last_clean":"2024-02-13T00:57:10.184063+0000","last_became_active":"2024-02-13T00:55:00.645985+0000","last_became_peered":"2024-02-13T00:55:00.645985+0000","last_unstale":"2024-02-13T00:57:10.184063+0000","last_undegraded":"2024-02-13T00:57:10.184063+0000","last_fullsized":"2024-02-13T00:57:10.184063+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:56.726543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:51:18.264738+0000","last_clean_scrub_stamp":"2024-02-13T00:56:56.726543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":54,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185254+0000","last_change":"2024-02-13T00:56:05.757898+0000","last_active":"2024-02-13T00:57:10.185254+0000","last_peered":"2024-02-13T00:57:10.185254+0000","last_clean":"2024-02-13T00:57:10.185254+0000","last_became_active":"2024-02-13T00:55:00.637834+0000","last_became_peered":"2024-02-13T00:55:00.637834+0000","last_unstale":"2024-02-13T00:57:10.185254+0000","last_undegraded":"2024-02-13T00:57:10.185254+0000","last_fullsized":"2024-02-13T00:57:10.185254+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:05.757837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:05.757837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":823,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154349+0000","last_change":"2024-02-13T00:56:26.850032+0000","last_active":"2024-02-13T00:57:10.154349+0000","last_peered":"2024-02-13T00:57:10.154349+0000","last_clean":"2024-02-13T00:57:10.154349+0000","last_became_active":"2024-02-13T00:55:04.814928+0000","last_became_peered":"2024-02-13T00:55:04.814928+0000","last_unstale":"2024-02-13T00:57:10.154349+0000","last_undegraded":"2024-02-13T00:57:10.154349+0000","last_fullsized":"2024-02-13T00:57:10.154349+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:26.849966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:56:26.849966+0000","last_clean_scrub_stamp":"2024-02-13T00:56:26.849966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"467'27","reported_seq":741,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958298+0000","last_change":"2024-02-13T00:57:09.321755+0000","last_active":"2024-02-13T00:57:09.958298+0000","last_peered":"2024-02-13T00:57:09.958298+0000","last_clean":"2024-02-13T00:57:09.958298+0000","last_became_active":"2024-02-13T00:55:51.632600+0000","last_became_peered":"2024-02-13T00:55:51.632600+0000","last_unstale":"2024-02-13T00:57:09.958298+0000","last_undegraded":"2024-02-13T00:57:09.958298+0000","last_fullsized":"2024-02-13T00:57:09.958298+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'27","last_scrub_stamp":"2024-02-13T00:57:09.321666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:42.205050+0000","last_clean_scrub_stamp":"2024-02-13T00:57:09.321666+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":40,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":479,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185459+0000","last_change":"2024-02-13T00:56:18.105575+0000","last_active":"2024-02-13T00:57:10.185459+0000","last_peered":"2024-02-13T00:57:10.185459+0000","last_clean":"2024-02-13T00:57:10.185459+0000","last_became_active":"2024-02-13T00:56:07.657519+0000","last_became_peered":"2024-02-13T00:56:07.657519+0000","last_unstale":"2024-02-13T00:57:10.185459+0000","last_undegraded":"2024-02-13T00:57:10.185459+0000","last_fullsized":"2024-02-13T00:57:10.185459+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:18.105481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:40.201175+0000","last_clean_scrub_stamp":"2024-02-13T00:56:18.105481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":842,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154464+0000","last_change":"2024-02-13T00:56:39.825890+0000","last_active":"2024-02-13T00:57:10.154464+0000","last_peered":"2024-02-13T00:57:10.154464+0000","last_clean":"2024-02-13T00:57:10.154464+0000","last_became_active":"2024-02-13T00:51:41.582536+0000","last_became_peered":"2024-02-13T00:51:41.582536+0000","last_unstale":"2024-02-13T00:57:10.154464+0000","last_undegraded":"2024-02-13T00:57:10.154464+0000","last_fullsized":"2024-02-13T00:57:10.154464+0000","mapping_epoch":675,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":676,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:39.825802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:22.338042+0000","last_clean_scrub_stamp":"2024-02-13T00:56:39.825802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":66,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.959272+0000","last_change":"2024-02-13T00:56:41.069838+0000","last_active":"2024-02-13T00:57:09.959272+0000","last_peered":"2024-02-13T00:57:09.959272+0000","last_clean":"2024-02-13T00:57:09.959272+0000","last_became_active":"2024-02-13T00:54:52.751112+0000","last_became_peered":"2024-02-13T00:54:52.751112+0000","last_unstale":"2024-02-13T00:57:09.959272+0000","last_undegraded":"2024-02-13T00:57:09.959272+0000","last_fullsized":"2024-02-13T00:57:09.959272+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:41.069740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:55:21.226050+0000","last_clean_scrub_stamp":"2024-02-13T00:56:41.069740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"467'16","reported_seq":742,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958006+0000","last_change":"2024-02-13T00:56:55.204514+0000","last_active":"2024-02-13T00:57:09.958006+0000","last_peered":"2024-02-13T00:57:09.958006+0000","last_clean":"2024-02-13T00:57:09.958006+0000","last_became_active":"2024-02-13T00:55:51.633205+0000","last_became_peered":"2024-02-13T00:55:51.633205+0000","last_unstale":"2024-02-13T00:57:09.958006+0000","last_undegraded":"2024-02-13T00:57:09.958006+0000","last_fullsized":"2024-02-13T00:57:09.958006+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'16","last_scrub_stamp":"2024-02-13T00:56:55.204417+0000","last_deep_scrub":"85'14","last_deep_scrub_stamp":"2024-02-13T00:41:47.734904+0000","last_clean_scrub_stamp":"2024-02-13T00:56:55.204417+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":404,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185520+0000","last_change":"2024-02-13T00:57:03.783659+0000","last_active":"2024-02-13T00:57:10.185520+0000","last_peered":"2024-02-13T00:57:10.185520+0000","last_clean":"2024-02-13T00:57:10.185520+0000","last_became_active":"2024-02-13T00:56:07.837581+0000","last_became_peered":"2024-02-13T00:56:07.837581+0000","last_unstale":"2024-02-13T00:57:10.185520+0000","last_undegraded":"2024-02-13T00:57:10.185520+0000","last_fullsized":"2024-02-13T00:57:10.185520+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:03.783575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:21.165246+0000","last_clean_scrub_stamp":"2024-02-13T00:57:03.783575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":70,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958889+0000","last_change":"2024-02-13T00:56:28.013442+0000","last_active":"2024-02-13T00:57:09.958889+0000","last_peered":"2024-02-13T00:57:09.958889+0000","last_clean":"2024-02-13T00:57:09.958889+0000","last_became_active":"2024-02-13T00:55:00.635557+0000","last_became_peered":"2024-02-13T00:55:00.635557+0000","last_unstale":"2024-02-13T00:57:09.958889+0000","last_undegraded":"2024-02-13T00:57:09.958889+0000","last_fullsized":"2024-02-13T00:57:09.958889+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:28.013343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:38:53.600425+0000","last_clean_scrub_stamp":"2024-02-13T00:56:28.013343+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":555,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184446+0000","last_change":"2024-02-13T00:55:57.809002+0000","last_active":"2024-02-13T00:57:10.184446+0000","last_peered":"2024-02-13T00:57:10.184446+0000","last_clean":"2024-02-13T00:57:10.184446+0000","last_became_active":"2024-02-13T00:55:00.637611+0000","last_became_peered":"2024-02-13T00:55:00.637611+0000","last_unstale":"2024-02-13T00:57:10.184446+0000","last_undegraded":"2024-02-13T00:57:10.184446+0000","last_fullsized":"2024-02-13T00:57:10.184446+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:55:57.808891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:55:57.808891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"467'26","reported_seq":611,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957532+0000","last_change":"2024-02-13T00:56:53.227063+0000","last_active":"2024-02-13T00:57:09.957532+0000","last_peered":"2024-02-13T00:57:09.957532+0000","last_clean":"2024-02-13T00:57:09.957532+0000","last_became_active":"2024-02-13T00:55:51.635798+0000","last_became_peered":"2024-02-13T00:55:51.635798+0000","last_unstale":"2024-02-13T00:57:09.957532+0000","last_undegraded":"2024-02-13T00:57:09.957532+0000","last_fullsized":"2024-02-13T00:57:09.957532+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'26","last_scrub_stamp":"2024-02-13T00:56:53.226982+0000","last_deep_scrub":"90'20","last_deep_scrub_stamp":"2024-02-13T00:40:13.144451+0000","last_clean_scrub_stamp":"2024-02-13T00:56:53.226982+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":54,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.185104+0000","last_change":"2024-02-13T00:56:00.689172+0000","last_active":"2024-02-13T00:57:10.185104+0000","last_peered":"2024-02-13T00:57:10.185104+0000","last_clean":"2024-02-13T00:57:10.185104+0000","last_became_active":"2024-02-13T00:55:00.655317+0000","last_became_peered":"2024-02-13T00:55:00.655317+0000","last_unstale":"2024-02-13T00:57:10.185104+0000","last_undegraded":"2024-02-13T00:57:10.185104+0000","last_fullsized":"2024-02-13T00:57:10.185104+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:00.689104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:39:40.280305+0000","last_clean_scrub_stamp":"2024-02-13T00:56:00.689104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":46,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.958077+0000","last_change":"2024-02-13T00:56:10.358031+0000","last_active":"2024-02-13T00:57:09.958077+0000","last_peered":"2024-02-13T00:57:09.958077+0000","last_clean":"2024-02-13T00:57:09.958077+0000","last_became_active":"2024-02-13T00:55:51.631587+0000","last_became_peered":"2024-02-13T00:55:51.631587+0000","last_unstale":"2024-02-13T00:57:09.958077+0000","last_undegraded":"2024-02-13T00:57:09.958077+0000","last_fullsized":"2024-02-13T00:57:09.958077+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:10.357948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:10.357948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.1","version":"107'7","reported_seq":806,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:09.957157+0000","last_change":"2024-02-13T00:55:42.047116+0000","last_active":"2024-02-13T00:57:09.957157+0000","last_peered":"2024-02-13T00:57:09.957157+0000","last_clean":"2024-02-13T00:57:09.957157+0000","last_became_active":"2024-02-13T00:44:39.501209+0000","last_became_peered":"2024-02-13T00:44:39.501209+0000","last_unstale":"2024-02-13T00:57:09.957157+0000","last_undegraded":"2024-02-13T00:57:09.957157+0000","last_fullsized":"2024-02-13T00:57:09.957157+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'7","last_scrub_stamp":"2024-02-13T00:55:42.047043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:41:00.445926+0000","last_clean_scrub_stamp":"2024-02-13T00:55:42.047043+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"467'62","reported_seq":100,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.184553+0000","last_change":"2024-02-13T00:55:53.589892+0000","last_active":"2024-02-13T00:57:10.184553+0000","last_peered":"2024-02-13T00:57:10.184553+0000","last_clean":"2024-02-13T00:57:10.184553+0000","last_became_active":"2024-02-13T00:55:04.810397+0000","last_became_peered":"2024-02-13T00:55:04.810397+0000","last_unstale":"2024-02-13T00:57:10.184553+0000","last_undegraded":"2024-02-13T00:57:10.184553+0000","last_fullsized":"2024-02-13T00:57:10.184553+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":822,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'62","last_scrub_stamp":"2024-02-13T00:55:53.589818+0000","last_deep_scrub":"86'50","last_deep_scrub_stamp":"2024-02-13T00:39:53.118989+0000","last_clean_scrub_stamp":"2024-02-13T00:55:53.589818+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":864,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.154879+0000","last_change":"2024-02-13T00:57:04.930293+0000","last_active":"2024-02-13T00:57:10.154879+0000","last_peered":"2024-02-13T00:57:10.154879+0000","last_clean":"2024-02-13T00:57:10.154879+0000","last_became_active":"2024-02-13T00:54:56.636095+0000","last_became_peered":"2024-02-13T00:54:56.636095+0000","last_unstale":"2024-02-13T00:57:10.154879+0000","last_undegraded":"2024-02-13T00:57:10.154879+0000","last_fullsized":"2024-02-13T00:57:10.154879+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:57:04.930154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:57:04.930154+0000","last_clean_scrub_stamp":"2024-02-13T00:57:04.930154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":59,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-02-13T00:57:10.183881+0000","last_change":"2024-02-13T00:56:54.684002+0000","last_active":"2024-02-13T00:57:10.183881+0000","last_peered":"2024-02-13T00:57:10.183881+0000","last_clean":"2024-02-13T00:57:10.183881+0000","last_became_active":"2024-02-13T00:55:00.645669+0000","last_became_peered":"2024-02-13T00:55:00.645669+0000","last_unstale":"2024-02-13T00:57:10.183881+0000","last_undegraded":"2024-02-13T00:57:10.183881+0000","last_fullsized":"2024-02-13T00:57:10.183881+0000","mapping_epoch":818,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T00:56:54.683930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T00:37:46.228444+0000","last_clean_scrub_stamp":"2024-02-13T00:56:54.683930+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":126,"acting":126,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1488,"num_read_kb":1213,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1018,"num_bytes_recovered":18210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1022,"ondisk_log_size":1022,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1157,"num_read_kb":1152,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":3215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266926,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304088,"kb_used_data":6864,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441064,"statfs":{"total":95995035648,"available":95683649536,"internally_reserved":0,"allocated":7028736,"data_stored":3750180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.50900000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":10.682},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.499,"15min":0.73799999999999999},"min":{"1min":0.29599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.751,"5min":0.77400000000000002,"15min":25.504000000000001},"last":0.57699999999999996}]},{"osd":1,"last update":"Tue Feb 13 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.52100000000000002,"15min":0.58499999999999996},"min":{"1min":0.441,"5min":0.255,"15min":0.255},"max":{"1min":0.77600000000000002,"5min":2.661,"15min":13.347},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51500000000000001,"15min":0.65900000000000003},"min":{"1min":0.36099999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69999999999999996,"5min":1.141,"15min":13.526999999999999},"last":0.52400000000000002}]},{"osd":2,"last update":"Tue Feb 13 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51000000000000001,"15min":0.64800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26600000000000001},"max":{"1min":0.80100000000000005,"5min":0.90100000000000002,"15min":14.359},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.52500000000000002,"15min":0.73499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32000000000000001,"15min":0.27400000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":21.497},"last":0.498}]},{"osd":3,"last update":"Tue Feb 13 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51000000000000001,"15min":0.69399999999999995},"min":{"1min":0.40200000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76700000000000002,"5min":0.89400000000000002,"15min":12.997},"last":0.438},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.53400000000000003,"15min":0.80400000000000005},"min":{"1min":0.372,"5min":0.35399999999999998,"15min":0.33200000000000002},"max":{"1min":0.72999999999999998,"5min":0.79300000000000004,"15min":21.622},"last":0.58699999999999997}]},{"osd":4,"last update":"Tue Feb 13 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44600000000000001,"15min":0.436},"min":{"1min":0.32000000000000001,"5min":0.27300000000000002,"15min":0.23000000000000001},"max":{"1min":0.56100000000000005,"5min":0.70499999999999996,"15min":2.452},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.47499999999999998,"15min":0.45400000000000001},"min":{"1min":0.33500000000000002,"5min":0.29499999999999998,"15min":0.245},"max":{"1min":0.64300000000000002,"5min":0.83199999999999996,"15min":1.0780000000000001},"last":0.51000000000000001}]},{"osd":5,"last update":"Tue Feb 13 00:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48199999999999998,"15min":0.499},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66700000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.621},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.49099999999999999,"15min":0.51200000000000001},"min":{"1min":0.41399999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72299999999999998,"5min":0.747,"15min":0.79300000000000004},"last":0.65900000000000003}]},{"osd":6,"last update":"Tue Feb 13 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51300000000000001,"15min":0.497},"min":{"1min":0.40500000000000003,"5min":0.31,"15min":0.25},"max":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.81299999999999994},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51500000000000001,"15min":0.496},"min":{"1min":0.38800000000000001,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.65000000000000002,"5min":0.85499999999999998,"15min":1.0940000000000001},"last":0.59799999999999998}]}]},{"osd":6,"up_from":45,"seq":193273528564,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312280,"kb_used_data":6864,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432872,"statfs":{"total":95995035648,"available":95675260928,"internally_reserved":0,"allocated":7028736,"data_stored":3748956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.504,"15min":0.53700000000000003},"min":{"1min":0.27500000000000002,"5min":0.19800000000000001,"15min":0.19700000000000001},"max":{"1min":0.78800000000000003,"5min":1.0609999999999999,"15min":8.1340000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.501,"15min":0.61799999999999999},"min":{"1min":0.34300000000000003,"5min":0.27400000000000002,"15min":0.221},"max":{"1min":0.63500000000000001,"5min":0.872,"15min":11.367000000000001},"last":0.48899999999999999}]},{"osd":1,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51100000000000001,"15min":0.67800000000000005},"min":{"1min":0.32500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.03,"5min":1.03,"15min":24.209},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.63800000000000001},"min":{"1min":0.30399999999999999,"5min":0.27900000000000003,"15min":0.23499999999999999},"max":{"1min":0.68799999999999994,"5min":0.79900000000000004,"15min":12.403},"last":0.58999999999999997}]},{"osd":2,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51100000000000001,"15min":0.67000000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.25600000000000001},"max":{"1min":0.71199999999999997,"5min":0.83799999999999997,"15min":12.477},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49399999999999999,"15min":0.66000000000000003},"min":{"1min":0.38800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.60399999999999998,"5min":1.032,"15min":11.702},"last":0.439}]},{"osd":3,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52100000000000002,"15min":0.65000000000000002},"min":{"1min":0.36099999999999999,"5min":0.309,"15min":0.29199999999999998},"max":{"1min":1.165,"5min":1.165,"15min":8.0519999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56899999999999995,"15min":0.69799999999999995},"min":{"1min":0.41199999999999998,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":0.66500000000000004,"5min":3.0219999999999998,"15min":13.257},"last":0.55200000000000005}]},{"osd":4,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.48699999999999999,"15min":0.439},"min":{"1min":0.318,"5min":0.26600000000000001,"15min":0.249},"max":{"1min":0.62,"5min":5.3840000000000003,"15min":5.3840000000000003},"last":0.502},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.505,"15min":0.45900000000000002},"min":{"1min":0.39200000000000002,"5min":0.27500000000000002,"15min":0.221},"max":{"1min":0.69399999999999995,"5min":1.927,"15min":1.927},"last":0.46999999999999997}]},{"osd":5,"last update":"Tue Feb 13 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.49299999999999999,"15min":0.50700000000000001},"min":{"1min":0.313,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71899999999999997,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.51800000000000002},"min":{"1min":0.32500000000000001,"5min":0.30199999999999999,"15min":0.27300000000000002},"max":{"1min":0.69599999999999995,"5min":0.876,"15min":0.876},"last":0.41899999999999998}]},{"osd":7,"last update":"Tue Feb 13 00:57:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.497,"15min":0.47399999999999998},"min":{"1min":0.41799999999999998,"5min":0.30399999999999999,"15min":0.27200000000000002},"max":{"1min":0.72999999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53700000000000003,"15min":0.52100000000000002},"min":{"1min":0.32900000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.80700000000000005},"last":0.67000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575117,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304128,"kb_used_data":6904,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441024,"statfs":{"total":95995035648,"available":95683608576,"internally_reserved":0,"allocated":7069696,"data_stored":3750209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36199999999999999,"15min":0.80000000000000004},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.187},"max":{"1min":0.52300000000000002,"5min":1.657,"15min":61.856000000000002},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.33700000000000002,"15min":0.64000000000000001},"min":{"1min":0.26100000000000001,"5min":0.224,"15min":0.16500000000000001},"max":{"1min":0.48699999999999999,"5min":0.74399999999999999,"15min":61.908999999999999},"last":0.25700000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33600000000000002,"15min":0.83299999999999996},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.20599999999999999},"max":{"1min":0.52700000000000002,"5min":0.629,"15min":32.161000000000001},"last":0.249},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.435,"15min":0.83499999999999996},"min":{"1min":0.255,"5min":0.23799999999999999,"15min":0.187},"max":{"1min":0.51000000000000001,"5min":5.1319999999999997,"15min":29.693000000000001},"last":0.28999999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.379,"15min":1.093},"min":{"1min":0.27300000000000002,"5min":0.26900000000000002,"15min":0.219},"max":{"1min":0.57899999999999996,"5min":0.75700000000000001,"15min":131.23099999999999},"last":0.309},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.39200000000000002,"15min":0.77800000000000002},"min":{"1min":0.29599999999999999,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":0.54700000000000004,"5min":0.753,"15min":49.055},"last":0.26900000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:57:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.57299999999999995,"5min":0.35399999999999998,"15min":0.27900000000000003},"max":{"1min":0.88,"5min":0.88,"15min":7.2089999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.63,"15min":0.63800000000000001},"min":{"1min":0.63400000000000001,"5min":0.36099999999999999,"15min":0.24199999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":10.050000000000001},"last":0.73599999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:56:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58999999999999997,"15min":0.57899999999999996},"min":{"1min":0.42499999999999999,"5min":0.41299999999999998,"15min":0.39400000000000002},"max":{"1min":0.92600000000000005,"5min":0.96699999999999997,"15min":1.7889999999999999},"last":0.63},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.33400000000000002},"max":{"1min":0.85799999999999998,"5min":0.86099999999999999,"15min":1.831},"last":0.76100000000000001}]},{"osd":6,"last update":"Tue Feb 13 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.58199999999999996,"15min":0.66000000000000003},"min":{"1min":0.504,"5min":0.33000000000000002,"15min":0.29099999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":14.772},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.58499999999999996,"15min":0.78000000000000003},"min":{"1min":0.50900000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.871,"5min":0.871,"15min":26.600000000000001},"last":0.65300000000000002}]},{"osd":7,"last update":"Tue Feb 13 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.59199999999999997,"15min":0.72199999999999998},"min":{"1min":0.44400000000000001,"5min":0.38300000000000001,"15min":0.29499999999999998},"max":{"1min":0.84799999999999998,"5min":0.94399999999999995,"15min":14.794},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.58099999999999996,"15min":0.91300000000000003},"min":{"1min":0.375,"5min":0.373,"15min":0.32200000000000001},"max":{"1min":0.87,"5min":0.87,"15min":26.693999999999999},"last":0.67200000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738641,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308184,"kb_used_data":6864,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436968,"statfs":{"total":95995035648,"available":95679455232,"internally_reserved":0,"allocated":7028736,"data_stored":3748633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.38700000000000001,"15min":0.624},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.40400000000000003,"5min":5.5250000000000004,"15min":30.184000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.42599999999999999,"15min":0.44600000000000001},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20300000000000001},"max":{"1min":0.46700000000000003,"5min":3.125,"15min":5.6959999999999997},"last":0.38200000000000001}]},{"osd":2,"last update":"Tue Feb 13 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.39700000000000002,"15min":0.48699999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.216},"max":{"1min":2.286,"5min":2.286,"15min":7.0839999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.40600000000000003,"15min":0.67500000000000004},"min":{"1min":0.26600000000000001,"5min":0.23300000000000001,"15min":0.20899999999999999},"max":{"1min":0.48399999999999999,"5min":2.1419999999999999,"15min":24.459},"last":0.318}]},{"osd":3,"last update":"Tue Feb 13 00:56:31 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.377,"15min":0.47199999999999998},"min":{"1min":0.28599999999999998,"5min":0.214,"15min":0.214},"max":{"1min":0.50800000000000001,"5min":1.0640000000000001,"15min":8.718},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.53300000000000003},"min":{"1min":0.27800000000000002,"5min":0.27700000000000002,"15min":0.222},"max":{"1min":2.4209999999999998,"5min":3.0419999999999998,"15min":7.1139999999999999},"last":0.28999999999999998}]},{"osd":4,"last update":"Tue Feb 13 00:57:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56299999999999994,"15min":0.65100000000000002},"min":{"1min":0.41199999999999998,"5min":0.31,"15min":0.29599999999999999},"max":{"1min":0.73299999999999998,"5min":1.552,"15min":9.2400000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59299999999999997,"15min":0.66400000000000003},"min":{"1min":0.47799999999999998,"5min":0.34399999999999997,"15min":0.28699999999999998},"max":{"1min":0.67600000000000005,"5min":2.6339999999999999,"15min":8.6950000000000003},"last":0.71799999999999997}]},{"osd":5,"last update":"Tue Feb 13 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54500000000000004,"15min":0.51700000000000002},"min":{"1min":0.42599999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.79000000000000004,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58399999999999996,"15min":0.55400000000000005},"min":{"1min":0.41599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73299999999999998,"5min":3.3450000000000002,"15min":3.3450000000000002},"last":0.54800000000000004}]},{"osd":6,"last update":"Tue Feb 13 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58699999999999997,"15min":0.64200000000000002},"min":{"1min":0.41699999999999998,"5min":0.36899999999999999,"15min":0.32900000000000001},"max":{"1min":0.69599999999999995,"5min":3.399,"15min":8.6129999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.65300000000000002,"15min":0.67400000000000004},"min":{"1min":0.44400000000000001,"5min":0.38200000000000001,"15min":0.32400000000000001},"max":{"1min":0.74099999999999999,"5min":7.2969999999999997,"15min":14.263},"last":0.627}]},{"osd":7,"last update":"Tue Feb 13 00:57:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59099999999999997,"15min":0.62},"min":{"1min":0.46200000000000002,"5min":0.33900000000000002,"15min":0.317},"max":{"1min":0.745,"5min":3.375,"15min":7.7720000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60699999999999998,"15min":0.65300000000000002},"min":{"1min":0.441,"5min":0.377,"15min":0.35899999999999999},"max":{"1min":0.75600000000000001,"5min":3.101,"15min":8.1609999999999996},"last":0.60599999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411592,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312276,"kb_used_data":6860,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432876,"statfs":{"total":95995035648,"available":95675265024,"internally_reserved":0,"allocated":7024640,"data_stored":3749115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.33400000000000002,"15min":0.496},"min":{"1min":0.248,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.47499999999999998,"5min":0.55200000000000005,"15min":9.7769999999999992},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.33700000000000002,"15min":0.52500000000000002},"min":{"1min":0.19600000000000001,"5min":0.191,"15min":0.17499999999999999},"max":{"1min":0.46999999999999997,"5min":0.68799999999999994,"15min":11.351000000000001},"last":0.29099999999999998}]},{"osd":1,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36899999999999999,"15min":0.48999999999999999},"min":{"1min":0.248,"5min":0.219,"15min":0.17199999999999999},"max":{"1min":0.496,"5min":1.843,"15min":10.141},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.36599999999999999,"15min":0.49099999999999999},"min":{"1min":0.27300000000000002,"5min":0.221,"15min":0.21299999999999999},"max":{"1min":0.48699999999999999,"5min":0.65400000000000003,"15min":9.0809999999999995},"last":0.35099999999999998}]},{"osd":3,"last update":"Tue Feb 13 00:56:32 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40200000000000002,"15min":0.51200000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.54200000000000004,"5min":1.875,"15min":11.807},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.54500000000000004},"min":{"1min":0.29199999999999998,"5min":0.255,"15min":0.223},"max":{"1min":0.51000000000000001,"5min":0.65300000000000002,"15min":16.981000000000002},"last":0.33600000000000002}]},{"osd":4,"last update":"Tue Feb 13 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.29099999999999998},"max":{"1min":0.85399999999999998,"5min":2.4630000000000001,"15min":6.1420000000000003},"last":0.495},{"interface":"front","average":{"1min":0.623,"5min":0.60499999999999998,"15min":0.64400000000000002},"min":{"1min":0.439,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86799999999999999,"5min":1.99,"15min":8.9640000000000004},"last":0.42799999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54300000000000004,"15min":0.55500000000000005},"min":{"1min":0.37,"5min":0.309,"15min":0.309},"max":{"1min":0.79600000000000004,"5min":0.87,"15min":0.87},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58699999999999997,"15min":0.57899999999999996},"min":{"1min":0.40500000000000003,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.52300000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58199999999999996,"15min":0.71899999999999997},"min":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.317},"max":{"1min":0.84999999999999998,"5min":2.1930000000000001,"15min":13.382},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59499999999999997,"15min":0.70499999999999996},"min":{"1min":0.47499999999999998,"5min":0.34499999999999997,"15min":0.33100000000000002},"max":{"1min":0.74399999999999999,"5min":3.3300000000000001,"15min":9.5670000000000002},"last":0.435}]},{"osd":7,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58099999999999996,"15min":0.71699999999999997},"min":{"1min":0.38,"5min":0.35499999999999998,"15min":0.33800000000000002},"max":{"1min":0.78400000000000003,"5min":3.1970000000000001,"15min":9.6099999999999994},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59199999999999997,"15min":0.63800000000000001},"min":{"1min":0.46000000000000002,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.82899999999999996,"5min":1.927,"15min":6.1829999999999998},"last":0.46500000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182658,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304108,"kb_used_data":6884,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441044,"statfs":{"total":95995035648,"available":95683629056,"internally_reserved":0,"allocated":7049216,"data_stored":3749582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.372,"15min":0.72399999999999998},"min":{"1min":0.214,"5min":0.186,"15min":0.186},"max":{"1min":0.51400000000000001,"5min":3.9550000000000001,"15min":42.459000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.376,"15min":0.746},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55300000000000005,"5min":1.9450000000000001,"15min":48.088999999999999},"last":0.35299999999999998}]},{"osd":1,"last update":"Tue Feb 13 00:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.39600000000000002,"15min":0.67200000000000004},"min":{"1min":0.247,"5min":0.23499999999999999,"15min":0.22},"max":{"1min":0.60899999999999999,"5min":1.97,"15min":47.999000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.40699999999999997,"15min":0.74199999999999999},"min":{"1min":0.23100000000000001,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":0.54000000000000004,"5min":3.7749999999999999,"15min":32.973999999999997},"last":0.28100000000000003}]},{"osd":2,"last update":"Tue Feb 13 00:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41099999999999998,"15min":0.70699999999999996},"min":{"1min":0.28399999999999997,"5min":0.23999999999999999,"15min":0.214},"max":{"1min":0.60499999999999998,"5min":1.9279999999999999,"15min":29.661999999999999},"last":0.318},{"interface":"front","average":{"1min":0.378,"5min":0.42399999999999999,"15min":0.67600000000000005},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23100000000000001},"max":{"1min":0.54800000000000004,"5min":4.218,"15min":35.256999999999998},"last":0.32800000000000001}]},{"osd":4,"last update":"Tue Feb 13 00:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.59399999999999997,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.314},"max":{"1min":0.69299999999999995,"5min":4.1289999999999996,"15min":8.968},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.58299999999999996,"15min":0.60899999999999999},"min":{"1min":0.41299999999999998,"5min":0.33600000000000002,"15min":0.27800000000000002},"max":{"1min":0.73999999999999999,"5min":1.8899999999999999,"15min":5.9649999999999999},"last":0.42799999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.64300000000000002,"15min":0.57799999999999996},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.90200000000000002,"5min":4.077,"15min":4.077},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58099999999999996,"15min":0.57899999999999996},"min":{"1min":0.40300000000000002,"5min":0.35999999999999999,"15min":0.32200000000000001},"max":{"1min":0.80600000000000005,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.52100000000000002}]},{"osd":6,"last update":"Tue Feb 13 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.57299999999999995,"15min":0.80000000000000004},"min":{"1min":0.41299999999999998,"5min":0.34699999999999998,"15min":0.28799999999999998},"max":{"1min":0.81200000000000006,"5min":1.2210000000000001,"15min":31.457999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.66300000000000003,"15min":0.72699999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.35999999999999999},"max":{"1min":0.75700000000000001,"5min":4.1150000000000002,"15min":13.603999999999999},"last":0.495}]},{"osd":7,"last update":"Tue Feb 13 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.68000000000000005,"15min":0.88900000000000001},"min":{"1min":0.437,"5min":0.38700000000000001,"15min":0.376},"max":{"1min":0.85499999999999998,"5min":4.1870000000000003,"15min":35.234000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.62,"15min":0.66800000000000004},"min":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.379},"max":{"1min":0.72399999999999998,"5min":3.1080000000000001,"15min":5.9320000000000004},"last":0.54500000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986430,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316384,"kb_used_data":6872,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428768,"statfs":{"total":95995035648,"available":95671058432,"internally_reserved":0,"allocated":7036928,"data_stored":3749668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.28899999999999998,"5min":0.27100000000000002,"15min":0.21199999999999999},"max":{"1min":0.61399999999999999,"5min":1.9470000000000001,"15min":5.3319999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.32900000000000001,"5min":0.314,"15min":0.20799999999999999},"max":{"1min":0.67600000000000005,"5min":4.3849999999999998,"15min":11.532},"last":0.59999999999999998}]},{"osd":1,"last update":"Tue Feb 13 00:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.502,"15min":0.52700000000000002},"min":{"1min":0.35299999999999998,"5min":0.28899999999999998,"15min":0.25},"max":{"1min":0.60699999999999998,"5min":0.83199999999999996,"15min":7.5410000000000004},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53700000000000003,"15min":0.54300000000000004},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.25600000000000001},"max":{"1min":0.64800000000000002,"5min":2.1629999999999998,"15min":11.497999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Tue Feb 13 00:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.34599999999999997,"5min":0.30199999999999999,"15min":0.26900000000000002},"max":{"1min":0.68400000000000005,"5min":2.1930000000000001,"15min":4.2199999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58999999999999997,"15min":0.57099999999999995},"min":{"1min":0.42399999999999999,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":0.65900000000000003,"5min":4.3310000000000004,"15min":4.3310000000000004},"last":0.52500000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.59599999999999997,"15min":0.56699999999999995},"min":{"1min":0.371,"5min":0.313,"15min":0.30599999999999999},"max":{"1min":0.69899999999999995,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.53800000000000003,"15min":0.53900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.30599999999999999},"max":{"1min":0.67200000000000004,"5min":0.83799999999999997,"15min":3.9729999999999999},"last":0.61799999999999999}]},{"osd":5,"last update":"Tue Feb 13 00:56:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53300000000000003,"15min":0.48499999999999999},"min":{"1min":0.36899999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72899999999999998,"5min":4.306,"15min":4.306},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52000000000000002,"15min":0.47099999999999997},"min":{"1min":0.34899999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73999999999999999,"5min":2.367,"15min":2.367},"last":0.68200000000000005}]},{"osd":6,"last update":"Tue Feb 13 00:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49399999999999999,"15min":0.46400000000000002},"min":{"1min":0.32500000000000001,"5min":0.27800000000000002,"15min":0.25700000000000001},"max":{"1min":0.68899999999999995,"5min":0.81399999999999995,"15min":3.153},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55000000000000004,"15min":0.50900000000000001},"min":{"1min":0.40400000000000003,"5min":0.29099999999999998,"15min":0.254},"max":{"1min":0.73099999999999998,"5min":2.48,"15min":4.9390000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Tue Feb 13 00:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.29399999999999998},"max":{"1min":0.69099999999999995,"5min":3.024,"15min":3.024},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.623,"15min":0.53700000000000003},"min":{"1min":0.35999999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73699999999999999,"5min":4.3650000000000002,"15min":4.3650000000000002},"last":0.63100000000000001}]}]},{"osd":5,"up_from":479,"seq":2057289334896,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305232,"kb_used_data":6856,"kb_used_omap":42,"kb_used_meta":298325,"kb_avail":93439920,"statfs":{"total":95995035648,"available":95682478080,"internally_reserved":0,"allocated":7020544,"data_stored":3749668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43373,"internal_metadata":305485459},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.48799999999999999,"15min":0.47399999999999998},"min":{"1min":0.371,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.51200000000000001,"15min":0.48499999999999999},"min":{"1min":0.39000000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.88300000000000001},"last":0.62}]},{"osd":1,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53700000000000003,"15min":0.52900000000000003},"min":{"1min":0.441,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.93100000000000005},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.55100000000000005,"15min":0.52800000000000002},"min":{"1min":0.46000000000000002,"5min":0.32300000000000001,"15min":0.28599999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.85199999999999998},"last":0.71399999999999997}]},{"osd":2,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.51900000000000002,"15min":0.51200000000000001},"min":{"1min":0.39400000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.76600000000000001,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.499},"min":{"1min":0.33600000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52100000000000002}]},{"osd":3,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.5},"min":{"1min":0.378,"5min":0.29299999999999998,"15min":0.248},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.54300000000000004,"15min":0.53900000000000003},"min":{"1min":0.41799999999999998,"5min":0.33300000000000002,"15min":0.32700000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.56999999999999995}]},{"osd":4,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.32600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.623,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.52300000000000002},"min":{"1min":0.40600000000000003,"5min":0.33700000000000002,"15min":0.32300000000000001},"max":{"1min":0.70199999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55400000000000005}]},{"osd":6,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47199999999999998,"15min":0.45100000000000001},"min":{"1min":0.34399999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65900000000000003,"5min":0.80800000000000005,"15min":1.2390000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.48399999999999999,"15min":0.47099999999999997},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.248},"max":{"1min":0.70199999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.442}]},{"osd":7,"last update":"Tue Feb 13 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.50900000000000001,"15min":0.5},"min":{"1min":0.34200000000000003,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":0.72499999999999998,"5min":0.82699999999999996,"15min":0.97299999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.57699999999999996,"15min":0.55200000000000005},"min":{"1min":0.38800000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":0.88800000000000001,"15min":0.89400000000000002},"last":0.63800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T00:57:17.446 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-13T00:57:17.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-02-13T00:57:17.712 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T00:57:18.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:16.604620+0000 mgr.y (mgr.14544) 775 : cluster [DBG] pgmap v1370: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:57:18.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:16.662441+0000 mgr.y (mgr.14544) 776 : audit [DBG] from='client.38564 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:57:18.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:16.604620+0000 mgr.y (mgr.14544) 775 : cluster [DBG] pgmap v1370: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:57:18.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:16.662441+0000 mgr.y (mgr.14544) 776 : audit [DBG] from='client.38564 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:57:18.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:16.604620+0000 mgr.y (mgr.14544) 775 : cluster [DBG] pgmap v1370: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:57:18.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:17 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:16.662441+0000 mgr.y (mgr.14544) 776 : audit [DBG] from='client.38564 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T00:57:18.995 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:57:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x[42374]: ::ffff:172.21.15.181 - - [13/Feb/2024:00:57:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-13T00:57:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:17.082492+0000 osd.1 (osd.1) 439 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:57:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:17.086285+0000 osd.1 (osd.1) 440 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:57:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:17.855758+0000 osd.6 (osd.6) 360 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:57:19.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:17.858631+0000 osd.6 (osd.6) 361 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:57:19.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:17.082492+0000 osd.1 (osd.1) 439 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:57:19.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:17.086285+0000 osd.1 (osd.1) 440 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:57:19.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:17.855758+0000 osd.6 (osd.6) 360 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:57:19.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:17.858631+0000 osd.6 (osd.6) 361 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:57:19.302 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:17.082492+0000 osd.1 (osd.1) 439 : cluster [DBG] 4.2 scrub starts 2024-02-13T00:57:19.302 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:17.086285+0000 osd.1 (osd.1) 440 : cluster [DBG] 4.2 scrub ok 2024-02-13T00:57:19.302 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:17.855758+0000 osd.6 (osd.6) 360 : cluster [DBG] 5.5 scrub starts 2024-02-13T00:57:19.302 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:18 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:17.858631+0000 osd.6 (osd.6) 361 : cluster [DBG] 5.5 scrub ok 2024-02-13T00:57:19.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.557+0000 7f62f88a5700 1 -- 172.21.15.162:0/1500027987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62f40fdc50 msgr2=0x7f62f40fe030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:19.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.557+0000 7f62f88a5700 1 --2- 172.21.15.162:0/1500027987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62f40fdc50 0x7f62f40fe030 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 rx=0x7f62e8004870 tx=0x7f62e8009cb0).stop 2024-02-13T00:57:19.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.557+0000 7f62f88a5700 1 -- 172.21.15.162:0/1500027987 shutdown_connections 2024-02-13T00:57:19.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.557+0000 7f62f88a5700 1 --2- 172.21.15.162:0/1500027987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62f40f84c0 0x7f62f4104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:19.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.557+0000 7f62f88a5700 1 --2- 172.21.15.162:0/1500027987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62f40fdc50 0x7f62f40fe030 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:19.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.557+0000 7f62f88a5700 1 --2- 172.21.15.162:0/1500027987 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62f40fe5b0 0x7f62f40f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:19.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.557+0000 7f62f88a5700 1 -- 172.21.15.162:0/1500027987 >> 172.21.15.162:0/1500027987 conn(0x7f62f40f38a0 msgr2=0x7f62f40f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:19.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.557+0000 7f62f88a5700 1 -- 172.21.15.162:0/1500027987 shutdown_connections 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.557+0000 7f62f88a5700 1 -- 172.21.15.162:0/1500027987 wait complete. 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f88a5700 1 librados: starting msgr at 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f88a5700 1 librados: starting objecter 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f88a5700 1 Processor -- start 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f88a5700 1 -- start start 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f88a5700 1 librados: setting wanted keys 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f88a5700 1 librados: calling monclient init 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f88a5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62f40f84c0 0x7f62f406b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f88a5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62f40fdc50 0x7f62f406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f88a5700 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62f40fe5b0 0x7f62f406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f88a5700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41165c0 con 0x7f62f40f84c0 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f88a5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f4116730 con 0x7f62f40fdc50 2024-02-13T00:57:19.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f88a5700 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41168f0 con 0x7f62f40fe5b0 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f259c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62f40f84c0 0x7f62f406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f259c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62f40f84c0 0x7f62f406b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:43036/0 (socket says 172.21.15.162:43036) 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f259c700 1 -- 172.21.15.162:0/1900673744 learned_addr learned my addr 172.21.15.162:0/1900673744 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f2d9d700 1 --2- 172.21.15.162:0/1900673744 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62f40fe5b0 0x7f62f406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.558+0000 7f62f1d9b700 1 --2- 172.21.15.162:0/1900673744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62f40fdc50 0x7f62f406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62f259c700 1 -- 172.21.15.162:0/1900673744 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62f40fe5b0 msgr2=0x7f62f406bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62f259c700 1 --2- 172.21.15.162:0/1900673744 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62f40fe5b0 0x7f62f406bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62f259c700 1 -- 172.21.15.162:0/1900673744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62f40fdc50 msgr2=0x7f62f406b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62f259c700 1 --2- 172.21.15.162:0/1900673744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62f40fdc50 0x7f62f406b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62f259c700 1 -- 172.21.15.162:0/1900673744 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec0097c0 con 0x7f62f40f84c0 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62f1d9b700 1 --2- 172.21.15.162:0/1900673744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62f40fdc50 0x7f62f406b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62f259c700 1 --2- 172.21.15.162:0/1900673744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62f40f84c0 0x7f62f406b170 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7f62e80043a0 tx=0x7f62e8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62e37fe700 10 client.?.objecter ms_handle_connect 0x7f62f40f84c0 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62e37fe700 10 client.?.objecter resend_mon_ops 2024-02-13T00:57:19.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62e37fe700 1 -- 172.21.15.162:0/1900673744 <== mon.0 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e800e030 con 0x7f62f40f84c0 2024-02-13T00:57:19.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62f88a5700 1 -- 172.21.15.162:0/1900673744 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e8005040 con 0x7f62f40f84c0 2024-02-13T00:57:19.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62e37fe700 1 -- 172.21.15.162:0/1900673744 <== mon.0 v2:172.21.15.162:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62e8013da0 con 0x7f62f40f84c0 2024-02-13T00:57:19.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62e37fe700 1 -- 172.21.15.162:0/1900673744 <== mon.0 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e8018070 con 0x7f62f40f84c0 2024-02-13T00:57:19.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62f88a5700 10 client.28476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-13T00:57:19.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62f88a5700 1 -- 172.21.15.162:0/1900673744 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f4116e70 con 0x7f62f40f84c0 2024-02-13T00:57:19.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.559+0000 7f62f88a5700 1 librados: init done 2024-02-13T00:57:19.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.560+0000 7f62f88a5700 1 -- 172.21.15.162:0/1900673744 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f4107190 con 0x7f62f40f84c0 2024-02-13T00:57:19.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.561+0000 7f62e37fe700 1 -- 172.21.15.162:0/1900673744 <== mon.0 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) v1 ==== 77088+0+0 (secure 0 0 0) 0x7f62e801d420 con 0x7f62f40f84c0 2024-02-13T00:57:19.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.561+0000 7f62e37fe700 1 --2- 172.21.15.162:0/1900673744 >> v2:172.21.15.162:6800/2866534053 conn(0x7f62dc062c90 0x7f62dc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T00:57:19.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.561+0000 7f62e37fe700 1 -- 172.21.15.162:0/1900673744 <== mon.0 v2:172.21.15.162:3300/0 5 ==== osd_map(859..859 src has 449..859) v4 ==== 7509+0+0 (secure 0 0 0) 0x7f62e807e550 con 0x7f62f40f84c0 2024-02-13T00:57:19.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.561+0000 7f62e37fe700 10 client.28476.objecter ms_dispatch 0x7f62f4001550 osd_map(859..859 src has 449..859) v4 2024-02-13T00:57:19.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.561+0000 7f62e37fe700 3 client.28476.objecter handle_osd_map got epochs [859,859] > 0 2024-02-13T00:57:19.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.561+0000 7f62e37fe700 3 client.28476.objecter handle_osd_map decoding full epoch 859 2024-02-13T00:57:19.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.561+0000 7f62f1d9b700 1 --2- 172.21.15.162:0/1900673744 >> v2:172.21.15.162:6800/2866534053 conn(0x7f62dc062c90 0x7f62dc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T00:57:19.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.561+0000 7f62e37fe700 20 client.28476.objecter dump_active .. 0 homeless 2024-02-13T00:57:19.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.561+0000 7f62e37fe700 10 client.28476.objecter ms_handle_connect 0x7f62dc062c90 2024-02-13T00:57:19.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.561+0000 7f62f1d9b700 1 --2- 172.21.15.162:0/1900673744 >> v2:172.21.15.162:6800/2866534053 conn(0x7f62dc062c90 0x7f62dc065150 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 rx=0x7f62e4007fd0 tx=0x7f62e4009040).ready entity=mgr.14544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T00:57:19.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.564+0000 7f62e37fe700 1 -- 172.21.15.162:0/1900673744 <== mon.0 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f62e8051d20 con 0x7f62f40f84c0 2024-02-13T00:57:19.735 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.731+0000 7f62f88a5700 1 -- 172.21.15.162:0/1900673744 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) v1 -- 0x7f62f405fbe0 con 0x7f62f40f84c0 2024-02-13T00:57:19.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.736+0000 7f62e37fe700 1 -- 172.21.15.162:0/1900673744 <== mon.0 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v30) v1 ==== 118+0+0 (secure 0 0 0) 0x7f62e8051910 con 0x7f62f40f84c0 2024-02-13T00:57:19.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 10 librados: watch_flush enter 2024-02-13T00:57:19.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 10 librados: watch_flush exit 2024-02-13T00:57:19.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 20 client.28476.objecter shutdown clearing up homeless session... 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 10 client.28476.objecter successfully canceled tick 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 1 -- 172.21.15.162:0/1900673744 >> v2:172.21.15.162:6800/2866534053 conn(0x7f62dc062c90 msgr2=0x7f62dc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 1 --2- 172.21.15.162:0/1900673744 >> v2:172.21.15.162:6800/2866534053 conn(0x7f62dc062c90 0x7f62dc065150 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 rx=0x7f62e4007fd0 tx=0x7f62e4009040).stop 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 1 -- 172.21.15.162:0/1900673744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62f40f84c0 msgr2=0x7f62f406b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 1 --2- 172.21.15.162:0/1900673744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62f40f84c0 0x7f62f406b170 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7f62e80043a0 tx=0x7f62e8009cb0).stop 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 1 -- 172.21.15.162:0/1900673744 shutdown_connections 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 1 --2- 172.21.15.162:0/1900673744 >> v2:172.21.15.162:6800/2866534053 conn(0x7f62dc062c90 0x7f62dc065150 unknown :-1 s=CLOSED pgs=3100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 1 --2- 172.21.15.162:0/1900673744 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62f40fe5b0 0x7f62f406bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 1 --2- 172.21.15.162:0/1900673744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62f40f84c0 0x7f62f406b170 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 1 --2- 172.21.15.162:0/1900673744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62f40fdc50 0x7f62f406b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.738+0000 7f62f88a5700 1 -- 172.21.15.162:0/1900673744 >> 172.21.15.162:0/1900673744 conn(0x7f62f40f38a0 msgr2=0x7f62f40f5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.739+0000 7f62f88a5700 1 -- 172.21.15.162:0/1900673744 shutdown_connections 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.739+0000 7f62f88a5700 1 -- 172.21.15.162:0/1900673744 wait complete. 2024-02-13T00:57:19.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-02-13T00:57:19.739+0000 7f62f88a5700 1 librados: shutdown 2024-02-13T00:57:19.996 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:57:19] ENGINE Bus STOPPING 2024-02-13T00:57:19.996 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:57:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T00:57:19.996 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:57:19] ENGINE Bus STOPPED 2024-02-13T00:57:19.996 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:57:19] ENGINE Bus STARTING 2024-02-13T00:57:19.996 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:57:19] ENGINE Serving on http://:::9283 2024-02-13T00:57:19.996 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:57:19] ENGINE Bus STARTED 2024-02-13T00:57:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:18.074483+0000 osd.1 (osd.1) 441 : cluster [DBG] 28.15 scrub starts 2024-02-13T00:57:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:18.077466+0000 osd.1 (osd.1) 442 : cluster [DBG] 28.15 scrub ok 2024-02-13T00:57:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:18.605004+0000 mgr.y (mgr.14544) 777 : cluster [DBG] pgmap v1371: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:57:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:18.808105+0000 osd.6 (osd.6) 362 : cluster [DBG] 6.25 deep-scrub starts 2024-02-13T00:57:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:18.811430+0000 osd.6 (osd.6) 363 : cluster [DBG] 6.25 deep-scrub ok 2024-02-13T00:57:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:18.938670+0000 osd.0 (osd.0) 419 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:57:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: cluster 2024-02-13T00:57:18.942141+0000 osd.0 (osd.0) 420 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:57:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:19.732782+0000 mon.a (mon.0) 3885 : audit [INF] from='client.? 172.21.15.162:0/1900673744' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-13T00:57:20.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:19.737069+0000 mon.a (mon.0) 3886 : audit [INF] from='client.? 172.21.15.162:0/1900673744' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-13T00:57:20.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:19.749325+0000 mon.c (mon.2) 546 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:57:20.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:19.750663+0000 mon.c (mon.2) 547 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:57:20.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:19.751251+0000 mon.c (mon.2) 548 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:57:20.296 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: audit 2024-02-13T00:57:19.758193+0000 mon.a (mon.0) 3887 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:57:20.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:18.074483+0000 osd.1 (osd.1) 441 : cluster [DBG] 28.15 scrub starts 2024-02-13T00:57:20.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:18.077466+0000 osd.1 (osd.1) 442 : cluster [DBG] 28.15 scrub ok 2024-02-13T00:57:20.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:18.605004+0000 mgr.y (mgr.14544) 777 : cluster [DBG] pgmap v1371: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:57:20.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:18.808105+0000 osd.6 (osd.6) 362 : cluster [DBG] 6.25 deep-scrub starts 2024-02-13T00:57:20.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:18.811430+0000 osd.6 (osd.6) 363 : cluster [DBG] 6.25 deep-scrub ok 2024-02-13T00:57:20.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:18.938670+0000 osd.0 (osd.0) 419 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:57:20.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: cluster 2024-02-13T00:57:18.942141+0000 osd.0 (osd.0) 420 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:57:20.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:19.732782+0000 mon.a (mon.0) 3885 : audit [INF] from='client.? 172.21.15.162:0/1900673744' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-13T00:57:20.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:19.737069+0000 mon.a (mon.0) 3886 : audit [INF] from='client.? 172.21.15.162:0/1900673744' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-13T00:57:20.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:19.749325+0000 mon.c (mon.2) 546 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:57:20.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:19.750663+0000 mon.c (mon.2) 547 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:57:20.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:19.751251+0000 mon.c (mon.2) 548 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:57:20.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: audit 2024-02-13T00:57:19.758193+0000 mon.a (mon.0) 3887 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:57:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:18.074483+0000 osd.1 (osd.1) 441 : cluster [DBG] 28.15 scrub starts 2024-02-13T00:57:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:18.077466+0000 osd.1 (osd.1) 442 : cluster [DBG] 28.15 scrub ok 2024-02-13T00:57:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:18.605004+0000 mgr.y (mgr.14544) 777 : cluster [DBG] pgmap v1371: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-13T00:57:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:18.808105+0000 osd.6 (osd.6) 362 : cluster [DBG] 6.25 deep-scrub starts 2024-02-13T00:57:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:18.811430+0000 osd.6 (osd.6) 363 : cluster [DBG] 6.25 deep-scrub ok 2024-02-13T00:57:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:18.938670+0000 osd.0 (osd.0) 419 : cluster [DBG] 4.16 scrub starts 2024-02-13T00:57:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: cluster 2024-02-13T00:57:18.942141+0000 osd.0 (osd.0) 420 : cluster [DBG] 4.16 scrub ok 2024-02-13T00:57:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:19.732782+0000 mon.a (mon.0) 3885 : audit [INF] from='client.? 172.21.15.162:0/1900673744' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-13T00:57:20.331 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:19.737069+0000 mon.a (mon.0) 3886 : audit [INF] from='client.? 172.21.15.162:0/1900673744' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-13T00:57:20.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:19.749325+0000 mon.c (mon.2) 546 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T00:57:20.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:19.750663+0000 mon.c (mon.2) 547 : audit [DBG] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T00:57:20.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:19.751251+0000 mon.c (mon.2) 548 : audit [INF] from='mgr.14544 172.21.15.162:0/3226178310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T00:57:20.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:19 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: audit 2024-02-13T00:57:19.758193+0000 mon.a (mon.0) 3887 : audit [INF] from='mgr.14544 ' entity='mgr.y' 2024-02-13T00:57:20.570 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-13T00:57:20.581 INFO:tasks.cephadm:Teardown begin 2024-02-13T00:57:20.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T00:57:20.614 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T00:57:20.653 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-13T00:57:20.653 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-13T00:57:20.673 DEBUG:teuthology.orchestra.run.smithi181:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-13T00:57:20.714 INFO:tasks.cephadm:Stopping all daemons... 2024-02-13T00:57:20.714 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-13T00:57:20.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.a 2024-02-13T00:57:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:20 smithi162 systemd[1]: Stopping Ceph mon.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:57:20.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:20 smithi162 bash[260888]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon.a" found: no such container 2024-02-13T00:57:21.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:57:21.045+0000 7f5f6b07c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-13T00:57:21.330 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:21 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a[39606]: debug 2024-02-13T00:57:21.045+0000 7f5f6b07c700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-13T00:57:21.830 INFO:journalctl@ceph.mon.a.smithi162.stdout:Feb 13 00:57:21 smithi162 bash[260928]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-a 2024-02-13T00:57:21.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.a.service' 2024-02-13T00:57:21.962 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:57:21.962 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-13T00:57:21.962 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-13T00:57:21.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.c 2024-02-13T00:57:22.323 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:22 smithi162 systemd[1]: Stopping Ceph mon.c for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:57:22.323 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:22 smithi162 bash[261047]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon.c" found: no such container 2024-02-13T00:57:22.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:57:22.318+0000 7f7b627ab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-13T00:57:22.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c[46375]: debug 2024-02-13T00:57:22.318+0000 7f7b627ab700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-02-13T00:57:22.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:57:22] ENGINE Bus STOPPING 2024-02-13T00:57:22.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:57:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T00:57:22.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:57:22] ENGINE Bus STOPPED 2024-02-13T00:57:22.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:57:22] ENGINE Bus STARTING 2024-02-13T00:57:22.580 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:57:22] ENGINE Serving on http://:::9283 2024-02-13T00:57:22.581 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:22 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: [13/Feb/2024:00:57:22] ENGINE Bus STARTED 2024-02-13T00:57:23.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Feb 13 00:57:22 smithi162 bash[261085]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-c 2024-02-13T00:57:23.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.c.service' 2024-02-13T00:57:23.265 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:57:23.265 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-13T00:57:23.265 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-13T00:57:23.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.b 2024-02-13T00:57:23.591 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:23 smithi181 systemd[1]: Stopping Ceph mon.b for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:57:23.591 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:23 smithi181 bash[120437]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon.b" found: no such container 2024-02-13T00:57:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: debug 2024-02-13T00:57:23.586+0000 7f1ce59ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-13T00:57:24.045 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b[40909]: debug 2024-02-13T00:57:23.586+0000 7f1ce59ba700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-02-13T00:57:24.545 INFO:journalctl@ceph.mon.b.smithi181.stdout:Feb 13 00:57:24 smithi181 bash[120479]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mon-b 2024-02-13T00:57:24.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mon.b.service' 2024-02-13T00:57:24.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:57:24.825 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-13T00:57:24.825 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-13T00:57:24.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mgr.y 2024-02-13T00:57:25.198 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:24 smithi162 systemd[1]: Stopping Ceph mgr.y for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:57:25.198 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:24 smithi162 bash[261216]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr.y" found: no such container 2024-02-13T00:57:25.545 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:25 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y[39881]: debug 2024-02-13T00:57:25.193+0000 7f1777872700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-13T00:57:25.830 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Feb 13 00:57:25 smithi162 bash[261258]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-y 2024-02-13T00:57:26.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mgr.y.service' 2024-02-13T00:57:26.105 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:57:26.106 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-13T00:57:26.106 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-13T00:57:26.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mgr.x 2024-02-13T00:57:26.545 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:57:26 smithi181 systemd[1]: Stopping Ceph mgr.x for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:57:26.545 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:57:26 smithi181 bash[120601]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr.x" found: no such container 2024-02-13T00:57:27.045 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Feb 13 00:57:26 smithi181 bash[120642]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-mgr-x 2024-02-13T00:57:27.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@mgr.x.service' 2024-02-13T00:57:27.397 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:57:27.397 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-13T00:57:27.397 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-13T00:57:27.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.0 2024-02-13T00:57:27.689 INFO:journalctl@ceph.osd.0.smithi162.stdout:Feb 13 00:57:27 smithi162 systemd[1]: Stopping Ceph osd.0 for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:57:27.689 INFO:journalctl@ceph.osd.0.smithi162.stdout:Feb 13 00:57:27 smithi162 bash[261378]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.0" found: no such container 2024-02-13T00:57:28.080 INFO:journalctl@ceph.osd.0.smithi162.stdout:Feb 13 00:57:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-0[49887]: debug 2024-02-13T00:57:27.685+0000 7f239a545700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-13T00:57:28.080 INFO:journalctl@ceph.osd.0.smithi162.stdout:Feb 13 00:57:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-0[49887]: debug 2024-02-13T00:57:27.685+0000 7f239a545700 -1 osd.0 859 *** Got signal Terminated *** 2024-02-13T00:57:28.080 INFO:journalctl@ceph.osd.0.smithi162.stdout:Feb 13 00:57:27 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-0[49887]: debug 2024-02-13T00:57:27.685+0000 7f239a545700 -1 osd.0 859 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T00:57:30.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:57:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:57:30.148Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:57:30.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:57:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:57:30.148Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-02-13T00:57:33.580 INFO:journalctl@ceph.osd.0.smithi162.stdout:Feb 13 00:57:33 smithi162 bash[261418]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-0 2024-02-13T00:57:35.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.0.service' 2024-02-13T00:57:35.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:57:35.167 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-13T00:57:35.167 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-13T00:57:35.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.1 2024-02-13T00:57:35.515 INFO:journalctl@ceph.osd.1.smithi162.stdout:Feb 13 00:57:35 smithi162 systemd[1]: Stopping Ceph osd.1 for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:57:35.515 INFO:journalctl@ceph.osd.1.smithi162.stdout:Feb 13 00:57:35 smithi162 bash[261661]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.1" found: no such container 2024-02-13T00:57:35.830 INFO:journalctl@ceph.osd.1.smithi162.stdout:Feb 13 00:57:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-1[54323]: debug 2024-02-13T00:57:35.509+0000 7fe46455a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-13T00:57:35.830 INFO:journalctl@ceph.osd.1.smithi162.stdout:Feb 13 00:57:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-1[54323]: debug 2024-02-13T00:57:35.509+0000 7fe46455a700 -1 osd.1 859 *** Got signal Terminated *** 2024-02-13T00:57:35.830 INFO:journalctl@ceph.osd.1.smithi162.stdout:Feb 13 00:57:35 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-1[54323]: debug 2024-02-13T00:57:35.509+0000 7fe46455a700 -1 osd.1 859 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T00:57:40.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:57:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=error ts=2024-02-13T00:57:40.148Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-02-13T00:57:40.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:57:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:57:40.148Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:57:40.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:57:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:57:40.149Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-02-13T00:57:41.330 INFO:journalctl@ceph.osd.1.smithi162.stdout:Feb 13 00:57:41 smithi162 bash[261703]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-1 2024-02-13T00:57:42.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.1.service' 2024-02-13T00:57:43.025 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:57:43.025 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-13T00:57:43.025 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-13T00:57:43.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.2 2024-02-13T00:57:43.330 INFO:journalctl@ceph.osd.2.smithi162.stdout:Feb 13 00:57:43 smithi162 systemd[1]: Stopping Ceph osd.2 for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:57:43.330 INFO:journalctl@ceph.osd.2.smithi162.stdout:Feb 13 00:57:43 smithi162 bash[261938]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.2" found: no such container 2024-02-13T00:57:43.830 INFO:journalctl@ceph.osd.2.smithi162.stdout:Feb 13 00:57:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-2[58993]: debug 2024-02-13T00:57:43.325+0000 7f7a15bc2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-13T00:57:43.830 INFO:journalctl@ceph.osd.2.smithi162.stdout:Feb 13 00:57:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-2[58993]: debug 2024-02-13T00:57:43.325+0000 7f7a15bc2700 -1 osd.2 859 *** Got signal Terminated *** 2024-02-13T00:57:43.830 INFO:journalctl@ceph.osd.2.smithi162.stdout:Feb 13 00:57:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-2[58993]: debug 2024-02-13T00:57:43.325+0000 7f7a15bc2700 -1 osd.2 859 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T00:57:49.080 INFO:journalctl@ceph.osd.2.smithi162.stdout:Feb 13 00:57:48 smithi162 bash[261980]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-2 2024-02-13T00:57:50.380 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:57:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=error ts=2024-02-13T00:57:50.149Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:57:50.381 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:57:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:57:50.149Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:57:50.381 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:57:50 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:57:50.150Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-02-13T00:57:50.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.2.service' 2024-02-13T00:57:50.755 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:57:50.755 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-13T00:57:50.756 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-13T00:57:50.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.3 2024-02-13T00:57:51.080 INFO:journalctl@ceph.osd.3.smithi162.stdout:Feb 13 00:57:50 smithi162 systemd[1]: Stopping Ceph osd.3 for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:57:51.080 INFO:journalctl@ceph.osd.3.smithi162.stdout:Feb 13 00:57:50 smithi162 bash[262213]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.3" found: no such container 2024-02-13T00:57:51.580 INFO:journalctl@ceph.osd.3.smithi162.stdout:Feb 13 00:57:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-3[63463]: debug 2024-02-13T00:57:51.136+0000 7fab70887700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-13T00:57:51.580 INFO:journalctl@ceph.osd.3.smithi162.stdout:Feb 13 00:57:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-3[63463]: debug 2024-02-13T00:57:51.136+0000 7fab70887700 -1 osd.3 859 *** Got signal Terminated *** 2024-02-13T00:57:51.580 INFO:journalctl@ceph.osd.3.smithi162.stdout:Feb 13 00:57:51 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-3[63463]: debug 2024-02-13T00:57:51.136+0000 7fab70887700 -1 osd.3 859 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T00:57:54.795 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:57:54 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:57:54.469+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:57:55.795 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:57:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:57:55.505+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:57:56.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:57:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:57:55.886+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:57:56.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:57:55 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:57:55.915+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:57:56.830 INFO:journalctl@ceph.osd.3.smithi162.stdout:Feb 13 00:57:56 smithi162 bash[262251]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-3 2024-02-13T00:57:56.892 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:57:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:57:56.552+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:57:57.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:57:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:57:56.906+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:57:57.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:57:56 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:57:56.889+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:57:57.862 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:57:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:57:57.588+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:57:58.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:57:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:57:57.881+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:57:58.295 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:57:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:57:57.858+0000 7f54d205f700 -1 osd.4 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:31.432023+0000 front 2024-02-13T00:57:31.432051+0000 (oldest deadline 2024-02-13T00:57:57.331996+0000) 2024-02-13T00:57:58.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:57:57 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:57:57.935+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:57:58.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.3.service' 2024-02-13T00:57:58.425 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:57:58.426 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-13T00:57:58.426 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-13T00:57:58.426 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.4 2024-02-13T00:57:58.711 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:57:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:57:58.592+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:57:58.711 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:57:58 smithi181 systemd[1]: Stopping Ceph osd.4 for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:57:58.711 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:57:58 smithi181 bash[120777]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.4" found: no such container 2024-02-13T00:57:58.711 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:57:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:57:58.707+0000 7f54d48e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-13T00:57:58.969 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:57:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:57:58.707+0000 7f54d48e1700 -1 osd.4 859 *** Got signal Terminated *** 2024-02-13T00:57:58.969 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:57:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:57:58.707+0000 7f54d48e1700 -1 osd.4 859 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T00:57:58.969 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:57:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:57:58.872+0000 7f54d205f700 -1 osd.4 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:31.432023+0000 front 2024-02-13T00:57:31.432051+0000 (oldest deadline 2024-02-13T00:57:57.331996+0000) 2024-02-13T00:57:58.970 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:57:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:57:58.880+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:57:59.294 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:57:58 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:57:58.965+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:57:59.853 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:57:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:57:59.553+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:58:00.295 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:57:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:57:59.849+0000 7f54d205f700 -1 osd.4 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:31.432023+0000 front 2024-02-13T00:57:31.432051+0000 (oldest deadline 2024-02-13T00:57:57.331996+0000) 2024-02-13T00:58:00.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:57:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:57:59.998+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:00.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:57:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:57:59.850+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:00.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=error ts=2024-02-13T00:58:00.149Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-02-13T00:58:00.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:58:00.151Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:58:00.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:00 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:58:00.151Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-02-13T00:58:00.884 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:00.600+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:58:01.295 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:58:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:58:00.898+0000 7f54d205f700 -1 osd.4 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:31.432023+0000 front 2024-02-13T00:57:31.432051+0000 (oldest deadline 2024-02-13T00:57:57.331996+0000) 2024-02-13T00:58:01.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:00 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:00.880+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:01.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:01.003+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:01.911 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:01.604+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:58:02.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:02.034+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:02.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:02.034+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:02.296 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:01.923+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:02.296 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:58:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:58:01.907+0000 7f54d205f700 -1 osd.4 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:31.432023+0000 front 2024-02-13T00:57:31.432051+0000 (oldest deadline 2024-02-13T00:57:57.331996+0000) 2024-02-13T00:58:02.296 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:58:01 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:58:01.907+0000 7f54d205f700 -1 osd.4 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.533480+0000 front 2024-02-13T00:57:39.533424+0000 (oldest deadline 2024-02-13T00:58:01.833394+0000) 2024-02-13T00:58:02.926 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:02.645+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:58:03.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:02.951+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:03.295 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:58:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:58:02.922+0000 7f54d205f700 -1 osd.4 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:31.432023+0000 front 2024-02-13T00:57:31.432051+0000 (oldest deadline 2024-02-13T00:57:57.331996+0000) 2024-02-13T00:58:03.295 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:58:02 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4[45138]: debug 2024-02-13T00:58:02.922+0000 7f54d205f700 -1 osd.4 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.533480+0000 front 2024-02-13T00:57:39.533424+0000 (oldest deadline 2024-02-13T00:58:01.833394+0000) 2024-02-13T00:58:03.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:03.058+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:03.296 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:03.058+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:03.937 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:03.635+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:58:04.282 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:04.064+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:04.282 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:04.064+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:04.282 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:03.932+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:04.282 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:03 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:03.932+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:04.545 INFO:journalctl@ceph.osd.4.smithi181.stdout:Feb 13 00:58:04 smithi181 bash[120817]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-4 2024-02-13T00:58:04.958 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:04.661+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:58:05.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:05.054+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:05.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:05.054+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:05.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:04.954+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:05.296 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:04 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:04.954+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:06.045 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:05.692+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:58:06.045 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:05.692+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.802047+0000 front 2024-02-13T00:57:39.802198+0000 (oldest deadline 2024-02-13T00:58:05.102032+0000) 2024-02-13T00:58:06.045 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:05.928+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:06.045 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:05 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:05.928+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:06.113 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.4.service' 2024-02-13T00:58:06.154 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:58:06.154 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-13T00:58:06.155 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-13T00:58:06.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.5 2024-02-13T00:58:06.545 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:06 smithi181 systemd[1]: Stopping Ceph osd.5 for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:58:06.545 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:06 smithi181 bash[121049]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.5" found: no such container 2024-02-13T00:58:06.546 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:06.102+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:06.546 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:06.102+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:06.975 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:06.558+0000 7ff8c1e53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-13T00:58:06.975 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:06.558+0000 7ff8c1e53700 -1 osd.5 859 *** Got signal Terminated *** 2024-02-13T00:58:06.975 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:06.558+0000 7ff8c1e53700 -1 osd.5 859 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T00:58:06.976 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:06.648+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:58:06.976 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:06.648+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.802047+0000 front 2024-02-13T00:57:39.802198+0000 (oldest deadline 2024-02-13T00:58:05.102032+0000) 2024-02-13T00:58:07.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:06.971+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:07.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:06 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:06.971+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:07.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:07.052+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:07.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:07.052+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:07.979 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:07.624+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:58:07.980 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:07.624+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.802047+0000 front 2024-02-13T00:57:39.802198+0000 (oldest deadline 2024-02-13T00:58:05.102032+0000) 2024-02-13T00:58:08.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:07.974+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:08.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:07 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:07.974+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:08.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:08.028+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:08.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:08.028+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:08.994 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:08.669+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:58:08.995 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:08.669+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.802047+0000 front 2024-02-13T00:57:39.802198+0000 (oldest deadline 2024-02-13T00:58:05.102032+0000) 2024-02-13T00:58:09.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:08.991+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:09.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:08.991+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:09.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:08 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:08.991+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:09.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:09.029+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:09.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:09.029+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:10.024 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:09.714+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:58:10.024 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:09.714+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.802047+0000 front 2024-02-13T00:57:39.802198+0000 (oldest deadline 2024-02-13T00:58:05.102032+0000) 2024-02-13T00:58:10.024 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:09 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:09.714+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:46.703544+0000 front 2024-02-13T00:57:46.703507+0000 (oldest deadline 2024-02-13T00:58:09.003394+0000) 2024-02-13T00:58:10.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:10.020+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:10.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:10.020+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:10.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:10.020+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:10.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:10.027+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:10.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:10.027+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:10.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=error ts=2024-02-13T00:58:10.150Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-02-13T00:58:10.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:58:10.151Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:58:10.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:10 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:58:10.151Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-02-13T00:58:10.986 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:10.719+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.300493+0000 front 2024-02-13T00:57:32.300522+0000 (oldest deadline 2024-02-13T00:57:54.000508+0000) 2024-02-13T00:58:10.986 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:10.719+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.802047+0000 front 2024-02-13T00:57:39.802198+0000 (oldest deadline 2024-02-13T00:58:05.102032+0000) 2024-02-13T00:58:10.986 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5[93328]: debug 2024-02-13T00:58:10.719+0000 7ff8bb5c9700 -1 osd.5 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:46.703544+0000 front 2024-02-13T00:57:46.703507+0000 (oldest deadline 2024-02-13T00:58:09.003394+0000) 2024-02-13T00:58:11.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:10.982+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:11.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:10.982+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:11.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:10.991+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:11.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:10.991+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:11.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:10 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:10.991+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:12.295 INFO:journalctl@ceph.osd.5.smithi181.stdout:Feb 13 00:58:12 smithi181 bash[121088]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-5 2024-02-13T00:58:12.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:11.956+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:12.296 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:11 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:11.956+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:12.296 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:12.021+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:12.296 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:12.021+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:12.296 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:12.021+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:13.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:12.964+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:13.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:12 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:12.964+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:13.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:13.028+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:13.296 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:13.028+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:13.296 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:13.028+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:13.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.5.service' 2024-02-13T00:58:13.973 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:58:13.974 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-13T00:58:13.974 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-13T00:58:13.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.6 2024-02-13T00:58:14.287 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:13.990+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:14.287 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:13.990+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:14.287 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:13.990+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:14.288 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:13.941+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:14.288 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:13.941+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:14.288 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:13 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:13.941+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:47.846687+0000 front 2024-02-13T00:57:47.846613+0000 (oldest deadline 2024-02-13T00:58:13.146484+0000) 2024-02-13T00:58:14.288 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:14 smithi181 systemd[1]: Stopping Ceph osd.6 for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:58:14.288 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:14 smithi181 bash[121322]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.6" found: no such container 2024-02-13T00:58:14.545 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:14.283+0000 7f8a7a6ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-13T00:58:14.545 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:14.283+0000 7f8a7a6ee700 -1 osd.6 859 *** Got signal Terminated *** 2024-02-13T00:58:14.545 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:14.283+0000 7f8a7a6ee700 -1 osd.6 859 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T00:58:15.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:14.922+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:15.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:14.922+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:15.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:14.922+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:47.846687+0000 front 2024-02-13T00:57:47.846613+0000 (oldest deadline 2024-02-13T00:58:13.146484+0000) 2024-02-13T00:58:15.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:14.998+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:15.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:14.998+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:15.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:14 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:14.998+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:16.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:16.011+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:16.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:16.011+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:16.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:16.011+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:16.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:15.894+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:16.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:15.894+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:16.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:15 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:15.894+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:47.846687+0000 front 2024-02-13T00:57:47.846613+0000 (oldest deadline 2024-02-13T00:58:13.146484+0000) 2024-02-13T00:58:17.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:17.062+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:17.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:17.062+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:17.295 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:17.062+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:17.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:16.852+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:17.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:16.852+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:17.295 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:16 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:16.852+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:47.846687+0000 front 2024-02-13T00:57:47.846613+0000 (oldest deadline 2024-02-13T00:58:13.146484+0000) 2024-02-13T00:58:18.100 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:17.808+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:18.101 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:17.808+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:18.101 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:17 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:17.808+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:47.846687+0000 front 2024-02-13T00:57:47.846613+0000 (oldest deadline 2024-02-13T00:58:13.146484+0000) 2024-02-13T00:58:18.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:18.097+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:18.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:18.097+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:18.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:18.097+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:18.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:18.097+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-02-13T00:57:51.416403+0000 front 2024-02-13T00:57:51.416439+0000 (oldest deadline 2024-02-13T00:58:17.316376+0000) 2024-02-13T00:58:19.145 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:18.822+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.644916+0000 front 2024-02-13T00:57:32.644773+0000 (oldest deadline 2024-02-13T00:57:55.544896+0000) 2024-02-13T00:58:19.145 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:18.822+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:39.645872+0000 front 2024-02-13T00:57:39.645813+0000 (oldest deadline 2024-02-13T00:58:01.945712+0000) 2024-02-13T00:58:19.145 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:18 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6[54219]: debug 2024-02-13T00:58:18.822+0000 7f8a77e6c700 -1 osd.6 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:47.846687+0000 front 2024-02-13T00:57:47.846613+0000 (oldest deadline 2024-02-13T00:58:13.146484+0000) 2024-02-13T00:58:19.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:19.141+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:19.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:19.141+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:19.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:19.141+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:19.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:19 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:19.141+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-02-13T00:57:51.416403+0000 front 2024-02-13T00:57:51.416439+0000 (oldest deadline 2024-02-13T00:58:17.316376+0000) 2024-02-13T00:58:20.045 INFO:journalctl@ceph.osd.6.smithi181.stdout:Feb 13 00:58:19 smithi181 bash[121361]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-6 2024-02-13T00:58:20.477 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:20.187+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:20.477 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:20.187+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:20.477 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:20.187+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:20.478 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:20 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:20.187+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-02-13T00:57:51.416403+0000 front 2024-02-13T00:57:51.416439+0000 (oldest deadline 2024-02-13T00:58:17.316376+0000) 2024-02-13T00:58:20.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=error ts=2024-02-13T00:58:20.151Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:58:20.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:58:20.151Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:58:20.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:20 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:58:20.152Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-02-13T00:58:21.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:21.185+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:21.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:21.185+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:21.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:21.185+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:21.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:21.185+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-02-13T00:57:51.416403+0000 front 2024-02-13T00:57:51.416439+0000 (oldest deadline 2024-02-13T00:58:17.316376+0000) 2024-02-13T00:58:21.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.6.service' 2024-02-13T00:58:21.601 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:58:21.601 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-13T00:58:21.602 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-13T00:58:21.602 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.7 2024-02-13T00:58:21.918 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:21 smithi181 systemd[1]: Stopping Ceph osd.7 for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:58:21.918 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:21 smithi181 bash[121596]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd.7" found: no such container 2024-02-13T00:58:22.175 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:21.914+0000 7f92799c4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-13T00:58:22.176 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:21.914+0000 7f92799c4700 -1 osd.7 859 *** Got signal Terminated *** 2024-02-13T00:58:22.176 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:21 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:21.914+0000 7f92799c4700 -1 osd.7 859 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T00:58:22.176 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:22.172+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:22.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:22.172+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:22.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:22.172+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:22.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:22 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:22.172+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-02-13T00:57:51.416403+0000 front 2024-02-13T00:57:51.416439+0000 (oldest deadline 2024-02-13T00:58:17.316376+0000) 2024-02-13T00:58:23.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:23.173+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:23.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:23.173+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:23.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:23.173+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:23.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:23 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:23.173+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-02-13T00:57:51.416403+0000 front 2024-02-13T00:57:51.416439+0000 (oldest deadline 2024-02-13T00:58:17.316376+0000) 2024-02-13T00:58:24.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:24.210+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:24.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:24.210+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:24.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:24.210+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:24.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:24 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:24.210+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-02-13T00:57:51.416403+0000 front 2024-02-13T00:57:51.416439+0000 (oldest deadline 2024-02-13T00:58:17.316376+0000) 2024-02-13T00:58:25.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:25.186+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:25.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:25.186+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:25.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:25.186+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:25.546 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:25.186+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-02-13T00:57:51.416403+0000 front 2024-02-13T00:57:51.416439+0000 (oldest deadline 2024-02-13T00:58:17.316376+0000) 2024-02-13T00:58:25.546 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:25 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:25.186+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.181:6802 osd.4 since back 2024-02-13T00:58:02.517646+0000 front 2024-02-13T00:58:02.517679+0000 (oldest deadline 2024-02-13T00:58:24.817665+0000) 2024-02-13T00:58:26.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:26.140+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-02-13T00:57:32.313228+0000 front 2024-02-13T00:57:32.313015+0000 (oldest deadline 2024-02-13T00:57:55.212983+0000) 2024-02-13T00:58:26.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:26.140+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-02-13T00:57:37.514131+0000 front 2024-02-13T00:57:37.514145+0000 (oldest deadline 2024-02-13T00:58:03.414161+0000) 2024-02-13T00:58:26.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:26.140+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-02-13T00:57:45.715113+0000 front 2024-02-13T00:57:45.715103+0000 (oldest deadline 2024-02-13T00:58:08.615091+0000) 2024-02-13T00:58:26.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:26.140+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-02-13T00:57:51.416403+0000 front 2024-02-13T00:57:51.416439+0000 (oldest deadline 2024-02-13T00:58:17.316376+0000) 2024-02-13T00:58:26.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:26 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7[58697]: debug 2024-02-13T00:58:26.140+0000 7f9276941700 -1 osd.7 859 heartbeat_check: no reply from 172.21.15.181:6802 osd.4 since back 2024-02-13T00:58:02.517646+0000 front 2024-02-13T00:58:02.517679+0000 (oldest deadline 2024-02-13T00:58:24.817665+0000) 2024-02-13T00:58:27.545 INFO:journalctl@ceph.osd.7.smithi181.stdout:Feb 13 00:58:27 smithi181 bash[121636]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-osd-7 2024-02-13T00:58:29.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@osd.7.service' 2024-02-13T00:58:29.078 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:58:29.078 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-13T00:58:29.078 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-13T00:58:29.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@rgw.foo.a 2024-02-13T00:58:29.387 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:58:29 smithi162 systemd[1]: Stopping Ceph rgw.foo.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:58:29.387 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:58:29 smithi162 bash[262486]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-rgw.foo.a" found: no such container 2024-02-13T00:58:29.830 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:58:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-rgw-foo-a[66295]: debug 2024-02-13T00:58:29.382+0000 7fa24ebca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-13T00:58:29.830 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:58:29 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-rgw-foo-a[66295]: debug 2024-02-13T00:58:29.382+0000 7fa265c94880 -1 shutting down 2024-02-13T00:58:30.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=error ts=2024-02-13T00:58:30.152Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:58:30.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:58:30.152Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:58:30.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:30 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:58:30.153Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-02-13T00:58:39.747 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:58:39 smithi162 bash[262527]: time="2024-02-13T00:58:39Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-13T00:58:40.080 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Feb 13 00:58:39 smithi162 bash[262527]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-rgw-foo-a 2024-02-13T00:58:40.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@rgw.foo.a.service' 2024-02-13T00:58:40.344 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:58:40.345 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-13T00:58:40.345 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-13T00:58:40.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@prometheus.a 2024-02-13T00:58:40.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=error ts=2024-02-13T00:58:40.153Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:58:40.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:58:40.153Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:58:40.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:40 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:58:40.154Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-02-13T00:58:40.670 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 systemd[1]: Stopping Ceph prometheus.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:58:40.670 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 bash[121879]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-13T00:58:40.938 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:58:40.666Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-13T00:58:40.938 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:58:40.666Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-13T00:58:40.939 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:58:40.666Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-13T00:58:40.939 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:58:40.666Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-13T00:58:40.939 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:58:40.666Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-13T00:58:40.939 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:58:40.666Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-13T00:58:40.939 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:58:40.666Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-13T00:58:40.939 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:58:40.666Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-13T00:58:40.940 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:58:40.666Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-13T00:58:40.940 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:58:40.667Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-13T00:58:40.940 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:58:40.667Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-13T00:58:40.940 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:40 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a[64576]: ts=2024-02-13T00:58:40.667Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-13T00:58:41.295 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Feb 13 00:58:41 smithi181 bash[121919]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-prometheus-a 2024-02-13T00:58:41.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@prometheus.a.service' 2024-02-13T00:58:41.550 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T00:58:41.550 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-13T00:58:41.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-13T00:58:41.845 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:58:41.627Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-02-13T00:58:41.845 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:41 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=warn ts=2024-02-13T00:58:41.627Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-02-13T00:58:42.830 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:42 smithi162 systemd[1]: Stopping Ceph alertmanager.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:58:42.830 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:42 smithi162 bash[262962]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-13T00:58:43.292 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:43 smithi162 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a[74007]: level=info ts=2024-02-13T00:58:43.002Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-13T00:58:43.580 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:43 smithi162 bash[263003]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-alertmanager-a 2024-02-13T00:58:44.330 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:58:44 smithi162 systemd[1]: Stopping Ceph node-exporter.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:58:44.330 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:58:44 smithi162 bash[263105]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter.a" found: no such container 2024-02-13T00:58:44.331 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:43 smithi162 systemd[1]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@alertmanager.a.service: Succeeded. 2024-02-13T00:58:44.331 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Feb 13 00:58:43 smithi162 systemd[1]: Stopped Ceph alertmanager.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:58:44.881 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:58:44 smithi162 bash[263144]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-a 2024-02-13T00:58:45.164 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:58:45 smithi162 systemd[1]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-13T00:58:45.428 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:58:45 smithi162 systemd[1]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-13T00:58:45.428 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Feb 13 00:58:45 smithi162 systemd[1]: Stopped Ceph node-exporter.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:58:45.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-13T00:58:46.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:58:46 smithi181 systemd[1]: Stopping Ceph iscsi.iscsi.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:58:46.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:58:46 smithi181 bash[122276]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi.iscsi.a" found: no such container 2024-02-13T00:58:47.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:58:46 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a[62417]: debug Shutdown received 2024-02-13T00:58:57.125 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:58:56 smithi181 bash[122314]: time="2024-02-13T00:58:56Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-02-13T00:58:57.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:58:57 smithi181 bash[122314]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a 2024-02-13T00:58:57.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:58:57 smithi181 systemd[1]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-02-13T00:58:58.281 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:58:58 smithi181 systemd[1]: Stopping Ceph node-exporter.b for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:58:58.281 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:58:58 smithi181 bash[122388]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-iscsi-iscsi-a-tcmu 2024-02-13T00:58:58.281 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:58:58 smithi181 systemd[1]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-02-13T00:58:58.281 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Feb 13 00:58:58 smithi181 systemd[1]: Stopped Ceph iscsi.iscsi.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:58:58.545 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:58:58 smithi181 bash[122465]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter.b" found: no such container 2024-02-13T00:58:59.045 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:58:58 smithi181 bash[122505]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-node-exporter-b 2024-02-13T00:58:59.045 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:58:58 smithi181 systemd[1]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-13T00:58:59.545 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:58:59 smithi181 systemd[1]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-13T00:58:59.545 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Feb 13 00:58:59 smithi181 systemd[1]: Stopped Ceph node-exporter.b for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:58:59.545 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:58:59 smithi181 systemd[1]: Stopping Ceph grafana.a for d4ee01fc-ca06-11ee-95b9-87774f69a715... 2024-02-13T00:58:59.545 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:58:59 smithi181 bash[122639]: Error: no container with name or ID "ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana.a" found: no such container 2024-02-13T00:58:59.910 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:58:59 smithi181 ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a[64941]: t=2024-02-13T00:58:59+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-13T00:59:00.162 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:59:00 smithi181 bash[122678]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715-grafana-a 2024-02-13T00:59:00.432 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:59:00 smithi181 systemd[1]: ceph-d4ee01fc-ca06-11ee-95b9-87774f69a715@grafana.a.service: Succeeded. 2024-02-13T00:59:00.432 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Feb 13 00:59:00 smithi181 systemd[1]: Stopped Ceph grafana.a for d4ee01fc-ca06-11ee-95b9-87774f69a715. 2024-02-13T00:59:00.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T00:59:00.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T00:59:00.758 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-13T00:59:00.760 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557148/remote/ubuntu@smithi162.front.sepia.ceph.com/crash 2024-02-13T00:59:00.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/crash -- . 2024-02-13T00:59:00.807 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-13T00:59:00.807 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now 2024-02-13T00:59:00.809 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557148/remote/ubuntu@smithi181.front.sepia.ceph.com/crash 2024-02-13T00:59:00.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/crash -- . 2024-02-13T00:59:00.847 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/d4ee01fc-ca06-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-13T00:59:00.847 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now 2024-02-13T00:59:00.848 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-13T00:59:00.849 INFO:tasks.cephadm:Removing cluster... 2024-02-13T00:59:00.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 --force 2024-02-13T00:59:01.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4ee01fc-ca06-11ee-95b9-87774f69a715 --force 2024-02-13T00:59:02.533 INFO:tasks.cephadm:Removing cephadm ... 2024-02-13T00:59:02.533 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-13T00:59:02.556 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-13T00:59:02.575 INFO:tasks.cephadm:Teardown complete 2024-02-13T00:59:02.576 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 1508, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 238, in ceph_log if first_in_ceph_log('\[ERR\]|\[WRN\]|\[SEC\]', File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 229, in first_in_ceph_log r = ctx.ceph[cluster_name].bootstrap_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 447, in run r = RemoteProcess(client, args, check_status=check_status, hostname=name, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 65, in __init__ self.command = quote(args) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 257, in quote return ' '.join(_quote(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 255, in _quote yield pipes.quote(a) File "/usr/lib/python3.8/shlex.py", line 325, in quote if _find_unsafe(s) is None: TypeError: expected string or bytes-like object 2024-02-13T00:59:02.578 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-13T00:59:02.588 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 1508, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 238, in ceph_log if first_in_ceph_log('\[ERR\]|\[WRN\]|\[SEC\]', File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 229, in first_in_ceph_log r = ctx.ceph[cluster_name].bootstrap_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 447, in run r = RemoteProcess(client, args, check_status=check_status, hostname=name, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 65, in __init__ self.command = quote(args) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 257, in quote return ' '.join(_quote(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 255, in _quote yield pipes.quote(a) File "/usr/lib/python3.8/shlex.py", line 325, in quote if _find_unsafe(s) is None: TypeError: expected string or bytes-like object 2024-02-13T00:59:02.589 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-13T00:59:02.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-13T00:59:02.599 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-13T00:59:02.690 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-13T00:59:02.699 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-13T00:59:02.699 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T00:59:02.702 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T00:59:02.719 INFO:teuthology.orchestra.run.smithi181.stderr:bash: ntpq: command not found 2024-02-13T00:59:02.720 INFO:teuthology.orchestra.run.smithi162.stderr:bash: ntpq: command not found 2024-02-13T00:59:02.723 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T00:59:02.723 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-02-13T00:59:02.723 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +526us[ +526us] +/- 76ms 2024-02-13T00:59:02.723 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv02.front.sepia.ceph.com 3 6 377 60 -754us[ -803us] +/- 69ms 2024-02-13T00:59:02.723 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +342us[ +342us] +/- 71ms 2024-02-13T00:59:02.723 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T00:59:02.725 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T00:59:02.725 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-02-13T00:59:02.725 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 60 +470us[ +470us] +/- 76ms 2024-02-13T00:59:02.725 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv02.front.sepia.ceph.com 3 8 377 120 -760us[ -913us] +/- 68ms 2024-02-13T00:59:02.725 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +260us[ +260us] +/- 71ms 2024-02-13T00:59:02.725 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T00:59:02.726 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-13T00:59:02.735 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-13T00:59:02.736 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-13T00:59:02.745 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-13T00:59:02.865 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-13T00:59:02.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T00:59:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T00:59:02.983 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T00:59:03.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T00:59:03.014 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-13T00:59:03.024 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-13T00:59:03.033 INFO:teuthology.task.internal:Duration was 2422.433276 seconds 2024-02-13T00:59:03.034 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-13T00:59:03.043 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-13T00:59:03.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-13T00:59:03.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-13T00:59:03.090 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T00:59:03.100 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T00:59:03.339 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-13T00:59:03.339 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-02-13T00:59:03.340 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-13T00:59:03.362 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-02-13T00:59:03.362 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-13T00:59:03.385 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-13T00:59:03.386 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-13T00:59:03.405 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-13T00:59:03.514 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-13T00:59:03.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-13T00:59:03.516 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-13T00:59:03.798 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-13T00:59:03.809 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-13T00:59:03.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-13T00:59:03.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-13T00:59:03.865 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-13T00:59:03.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-13T00:59:03.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-13T00:59:03.904 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-02-13T00:59:03.932 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-02-13T00:59:03.963 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-13T00:59:03.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T00:59:03.980 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-13T00:59:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T00:59:04.020 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-13T00:59:04.031 INFO:teuthology.task.internal:Transferring archived files... 2024-02-13T00:59:04.032 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557148/remote/smithi162 2024-02-13T00:59:04.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-13T00:59:04.096 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557148/remote/smithi181 2024-02-13T00:59:04.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-13T00:59:04.144 INFO:teuthology.task.internal:Removing archive directory... 2024-02-13T00:59:04.145 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-13T00:59:04.147 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-13T00:59:04.192 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-13T00:59:04.204 INFO:teuthology.task.internal:Not uploading archives. 2024-02-13T00:59:04.204 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-13T00:59:04.213 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-13T00:59:04.213 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-13T00:59:04.216 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-13T00:59:04.231 INFO:teuthology.orchestra.run.smithi162.stdout: 393225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 13 00:59 /home/ubuntu/cephtest 2024-02-13T00:59:04.249 INFO:teuthology.orchestra.run.smithi181.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 13 00:59 /home/ubuntu/cephtest 2024-02-13T00:59:04.251 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-13T00:59:04.261 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-13T00:59:04.272 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: TypeError: expected string or bytes-like object 2024-02-13T00:59:04.272 INFO:teuthology.nuke:Checking targets against current locks 2024-02-13T00:59:04.299 DEBUG:teuthology.nuke:shortname: smithi162 2024-02-13T00:59:04.299 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T00:59:04.322 DEBUG:teuthology.nuke:shortname: smithi181 2024-02-13T00:59:04.323 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T00:59:04.337 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 00:09:36.879680', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEXH5Amm+A+g8MvSb9PRCKMns56HmLlUD/QFloAsBJ1Hzoj4z30vb+z498JHtCUNexF8/DZeVnn+d+sgeu0iHpo='} 2024-02-13T00:59:04.342 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 00:09:36.881062', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPk7kbpnJ5UoTYejzsFP9Z4RpDIbvfOfvJ+d3SOBX5z+BTz8zZrsDvSL/iAsFZ1zjIzk8ddi5qyqh+Soevp8eo0='} 2024-02-13T00:59:04.392 INFO:teuthology.orchestra.console.smithi181:Power off 2024-02-13T00:59:04.392 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-13T00:59:04.407 INFO:teuthology.orchestra.console.smithi162:Power off 2024-02-13T00:59:04.407 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-13T00:59:04.419 DEBUG:teuthology.orchestra.console.smithi181:power off output: Chassis Power Control: Down/Off 2024-02-13T00:59:04.420 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T00:59:04.435 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2024-02-13T00:59:04.436 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T00:59:04.444 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-02-13T00:59:04.458 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-02-13T00:59:08.445 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T00:59:08.458 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T00:59:08.556 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-02-13T00:59:08.568 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-02-13T00:59:12.560 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T00:59:12.569 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T00:59:12.670 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off 2024-02-13T00:59:12.678 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-02-13T00:59:12.771 INFO:teuthology.orchestra.console.smithi181:Power off completed 2024-02-13T00:59:12.779 INFO:teuthology.orchestra.console.smithi162:Power off completed 2024-02-13T00:59:12.883 INFO:teuthology.run:Summary data: description: rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} duration: 2422.4332761764526 failure_reason: expected string or bytes-like object flavor: default owner: scheduled_lflores@teuthology status: fail success: false 2024-02-13T00:59:12.884 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-13T00:59:12.970 INFO:teuthology.run:FAIL